2024-04-02T14:08:49.228 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-04-02T14:08:49.228 DEBUG:teuthology.run:Teuthology command: teuthology --name pdonnell-2024-04-02_11:52:43-fs-wip-batrick-testing-20240402.004512-distro-default-smithi --archive /home/teuthworker/archive/pdonnell-2024-04-02_11:52:43-fs-wip-batrick-testing-20240402.004512-distro-default-smithi/7635633 --verbose --owner scheduled_pdonnell@teuthology --description fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/postgres}} -- /home/teuthworker/archive/pdonnell-2024-04-02_11:52:43-fs-wip-batrick-testing-20240402.004512-distro-default-smithi/7635633/orig.config.yaml 2024-04-02T14:08:49.297 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-02T14:08:49.392 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/pdonnell-2024-04-02_11:52:43-fs-wip-batrick-testing-20240402.004512-distro-default-smithi/7635633 branch: wip-batrick-testing-20240402.004512 description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/postgres}} email: pdonnell@redhat.com first_in_suite: false job_id: '7635633' kernel: client: branch: testing kdb: true sha1: distro ktype: upstream last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi mgrmodules: sequential: - print: Enabling mgr modules - exec: mon.a: - ceph mgr module enable snap_schedule - ceph config set mgr mgr/snap_schedule/allow_m_granularity true - ceph config set mgr mgr/snap_schedule/dump_on_update true name: pdonnell-2024-04-02_11:52:43-fs-wip-batrick-testing-20240402.004512-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-batrick-testing-20240402.004512 ceph: cephfs: max_mds: 5 session_timeout: 300 standby_replay: true conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true mds_bal_replicate_threshold: 1 mds_export_ephemeral_random_max: 0.1 osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: cephsqlite lock renewal timeout: 900000 debug cephsqlite: 20 debug client: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 osd shutdown pgref assert: true osd_max_omap_entries_per_request: 10 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - filesystem is online with fewer MDS than max_mds - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - PG_AVAILABILITY - PG_DEGRADED - Reduced data availability - slow metadata IO - SLOW_OPS - slow request sha1: 191d963ac9de7322da8aff20b567b3fa458e2ef6 subvols: create: 2 subvol_options: --namespace-isolated ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs ceph-fuse: client.0: mntopts: - context=system_u:object_r:postgresql_db_t:s0 mount_subvol_num: 0 mountpoint: /tmp/cephfs check-counter: counters: mds: - mds.exported - mds.imported - min: 3 name: mds_cache.dir_update - min: 3 name: mds_cache.dir_update_receipt - mds.root_rsnaps - mds_server.req_mksnap_latency.avgcount - mds_server.req_rmsnap_latency.avgcount install: ceph: flavor: default sha1: 191d963ac9de7322da8aff20b567b3fa458e2ef6 kclient: client.0: mntopts: - context=system_u:object_r:postgresql_db_t:s0 mount_subvol_num: 1 mountpoint: /tmp/cephfs mntopts: - ms_mode=secure - nowsync syntax: v2 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-batrick-testing-20240402.004512 sha1: 191d963ac9de7322da8aff20b567b3fa458e2ef6 owner: scheduled_pdonnell@teuthology priority: 75 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - mds.a - mds.d - mds.g - mds.j - osd.0 - osd.3 - osd.6 - osd.9 - client.0 - - mon.b - mgr.y - mds.b - mds.e - mds.h - mds.k - osd.1 - osd.4 - osd.7 - osd.10 - - mon.c - mgr.z - mds.c - mds.f - mds.i - mds.l - osd.2 - osd.5 - osd.8 - osd.11 seed: 2763 sha1: 191d963ac9de7322da8aff20b567b3fa458e2ef6 sleep_before_teardown: 0 subset: 318/512 suite: fs suite_branch: wip-batrick-testing-20240402.004512 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_191d963ac9de7322da8aff20b567b3fa458e2ef6/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 191d963ac9de7322da8aff20b567b3fa458e2ef6 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi090.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJS93EoA4MChx/q67iYrMT1Obfm2I07Gk+q4/6Jmlvf6klUQ3J3AS7yZ+1/ts6xGTOkpkbdUpkV0fP816IS0OMk= smithi092.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJuCtV9TCEQvasOugmWQqmXiTqfkE60iZ0BdPbwHiSHnR8IrHgqtyalK87AAGA5CgQnquCASauo24il75OhQFCE= smithi177.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOhQVdK2fqKEdXq4G1AcWvfCJbdxAhzafRar5kO/1BGBcupxYJjyADu9+jEC99Q1L3EYwhf8AN8DNkr+Ql/uR6o= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - cephadm: roleless: false - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: mon.a: - ceph fs dump - ceph osd dump - fs.ready: timeout: 300 - sequential: - mgrmodules - kclient: null - exec: mon.a: - ceph fs set cephfs balance_automate false - ceph fs subvolumegroup pin cephfs qa random 0.10 - check-counter: {} - fwd_scrub: scrub_timeout: 900 sleep_between_iterations: 1 - exec: mon.a: - ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m - ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - date +%s > START_TIME - full_sequential_finally: - exec: mon.a: - date +%s > END_TIME - START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - background_exec: mon.a: - while sleep 13; do ceph tell mds.cephfs:0 flush journal; done - exec: client.0: - sudo ls -l /tmp/cephfs/ && sudo df -h /tmp/cephfs/ - sudo mkdir -m 755 --context=system_u:system_r:postgresql_t:s0 /tmp/cephfs/postgres && sudo chown postgres:postgres /tmp/cephfs/postgres - sudo -u postgres -- mkdir -m 700 --context=system_u:system_r:postgresql_t:s0 /tmp/cephfs/postgres/data - sudo semanage fcontext -a -t postgresql_db_t "/tmp/cephfs/postgres(/.*)?" - sudo grep -i postgresql /etc/selinux/targeted/contexts/files/file_contexts.local - sudo restorecon -R -v /tmp/cephfs/postgres - sudo ls -lZaR /tmp/cephfs/postgres/ - sudo mkdir -p /etc/systemd/system/postgresql.service.d/ && printf '[Service]\nEnvironment=PGDATA=/tmp/cephfs/postgres/data\nEnvironment=PGLOG=/tmp/cephfs/postgres/pgstartup.log\n' | sudo tee /etc/systemd/system/postgresql.service.d/env.conf - sudo -u postgres -- postgresql-setup --initdb - sudo ls -lZaR /tmp/cephfs/postgres/ - sudo systemctl start postgresql - sudo -u postgres -- pgbench -s 32 -i - sudo -u postgres -- pgbench -c 100 -j 4 --progress=5 --time=900 - sudo systemctl stop postgresql - sudo ls -lZaR /tmp/cephfs/postgres/ teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-04-02_11:52:43 tube: smithi user: pdonnell verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-02T14:08:49.392 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_191d963ac9de7322da8aff20b567b3fa458e2ef6/qa; will attempt to use it 2024-04-02T14:08:49.393 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_191d963ac9de7322da8aff20b567b3fa458e2ef6/qa/tasks 2024-04-02T14:08:49.393 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-02T14:08:49.394 INFO:teuthology.task.internal:Checking packages... 2024-04-02T14:08:49.417 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '191d963ac9de7322da8aff20b567b3fa458e2ef6' 2024-04-02T14:08:49.417 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-02T14:08:49.418 INFO:teuthology.packaging:ref: None 2024-04-02T14:08:49.418 INFO:teuthology.packaging:tag: None 2024-04-02T14:08:49.418 INFO:teuthology.packaging:branch: wip-batrick-testing-20240402.004512 2024-04-02T14:08:49.418 INFO:teuthology.packaging:sha1: 191d963ac9de7322da8aff20b567b3fa458e2ef6 2024-04-02T14:08:49.418 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-batrick-testing-20240402.004512 2024-04-02T14:08:49.560 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2609.g191d963a 2024-04-02T14:08:49.562 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-02T14:08:49.569 INFO:teuthology.task.internal:no buildpackages task found 2024-04-02T14:08:49.569 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-02T14:08:49.603 INFO:teuthology.task.internal:Saving configuration 2024-04-02T14:08:49.628 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-02T14:08:49.636 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-02T14:08:49.661 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi090.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-04-02_11:52:43-fs-wip-batrick-testing-20240402.004512-distro-default-smithi/7635633', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-02 14:02:37.469537', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJS93EoA4MChx/q67iYrMT1Obfm2I07Gk+q4/6Jmlvf6klUQ3J3AS7yZ+1/ts6xGTOkpkbdUpkV0fP816IS0OMk='} 2024-04-02T14:08:49.685 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi092.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-04-02_11:52:43-fs-wip-batrick-testing-20240402.004512-distro-default-smithi/7635633', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-02 14:02:37.468340', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJuCtV9TCEQvasOugmWQqmXiTqfkE60iZ0BdPbwHiSHnR8IrHgqtyalK87AAGA5CgQnquCASauo24il75OhQFCE='} 2024-04-02T14:08:49.706 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi177.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-04-02_11:52:43-fs-wip-batrick-testing-20240402.004512-distro-default-smithi/7635633', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-02 14:02:37.466927', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOhQVdK2fqKEdXq4G1AcWvfCJbdxAhzafRar5kO/1BGBcupxYJjyADu9+jEC99Q1L3EYwhf8AN8DNkr+Ql/uR6o='} 2024-04-02T14:08:49.707 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-02T14:08:49.713 INFO:teuthology.task.internal:roles: ubuntu@smithi090.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2024-04-02T14:08:49.713 INFO:teuthology.task.internal:roles: ubuntu@smithi092.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2024-04-02T14:08:49.714 INFO:teuthology.task.internal:roles: ubuntu@smithi177.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2024-04-02T14:08:49.714 INFO:teuthology.run_tasks:Running task console_log... 2024-04-02T14:08:49.845 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f10f5cc9160>, signals=[15]) 2024-04-02T14:08:49.846 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-02T14:08:49.852 INFO:teuthology.task.internal:Opening connections... 2024-04-02T14:08:49.852 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi090.front.sepia.ceph.com 2024-04-02T14:08:49.855 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-02T14:08:49.949 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi092.front.sepia.ceph.com 2024-04-02T14:08:49.950 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi092.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-02T14:08:50.037 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi177.front.sepia.ceph.com 2024-04-02T14:08:50.038 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi177.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-02T14:08:50.126 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-02T14:08:50.133 DEBUG:teuthology.orchestra.run.smithi090:> uname -m 2024-04-02T14:08:50.151 INFO:teuthology.orchestra.run.smithi090.stdout:x86_64 2024-04-02T14:08:50.151 DEBUG:teuthology.orchestra.run.smithi090:> cat /etc/os-release 2024-04-02T14:08:50.207 INFO:teuthology.orchestra.run.smithi090.stdout:NAME="CentOS Stream" 2024-04-02T14:08:50.207 INFO:teuthology.orchestra.run.smithi090.stdout:VERSION="9" 2024-04-02T14:08:50.207 INFO:teuthology.orchestra.run.smithi090.stdout:ID="centos" 2024-04-02T14:08:50.208 INFO:teuthology.orchestra.run.smithi090.stdout:ID_LIKE="rhel fedora" 2024-04-02T14:08:50.208 INFO:teuthology.orchestra.run.smithi090.stdout:VERSION_ID="9" 2024-04-02T14:08:50.208 INFO:teuthology.orchestra.run.smithi090.stdout:PLATFORM_ID="platform:el9" 2024-04-02T14:08:50.208 INFO:teuthology.orchestra.run.smithi090.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-02T14:08:50.208 INFO:teuthology.orchestra.run.smithi090.stdout:ANSI_COLOR="0;31" 2024-04-02T14:08:50.208 INFO:teuthology.orchestra.run.smithi090.stdout:LOGO="fedora-logo-icon" 2024-04-02T14:08:50.208 INFO:teuthology.orchestra.run.smithi090.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-02T14:08:50.208 INFO:teuthology.orchestra.run.smithi090.stdout:HOME_URL="https://centos.org/" 2024-04-02T14:08:50.208 INFO:teuthology.orchestra.run.smithi090.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-02T14:08:50.208 INFO:teuthology.orchestra.run.smithi090.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-02T14:08:50.208 INFO:teuthology.orchestra.run.smithi090.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-02T14:08:50.209 INFO:teuthology.lock.ops:Updating smithi090.front.sepia.ceph.com on lock server 2024-04-02T14:08:50.229 DEBUG:teuthology.orchestra.run.smithi092:> uname -m 2024-04-02T14:08:50.247 INFO:teuthology.orchestra.run.smithi092.stdout:x86_64 2024-04-02T14:08:50.247 DEBUG:teuthology.orchestra.run.smithi092:> cat /etc/os-release 2024-04-02T14:08:50.305 INFO:teuthology.orchestra.run.smithi092.stdout:NAME="CentOS Stream" 2024-04-02T14:08:50.305 INFO:teuthology.orchestra.run.smithi092.stdout:VERSION="9" 2024-04-02T14:08:50.305 INFO:teuthology.orchestra.run.smithi092.stdout:ID="centos" 2024-04-02T14:08:50.305 INFO:teuthology.orchestra.run.smithi092.stdout:ID_LIKE="rhel fedora" 2024-04-02T14:08:50.305 INFO:teuthology.orchestra.run.smithi092.stdout:VERSION_ID="9" 2024-04-02T14:08:50.305 INFO:teuthology.orchestra.run.smithi092.stdout:PLATFORM_ID="platform:el9" 2024-04-02T14:08:50.305 INFO:teuthology.orchestra.run.smithi092.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-02T14:08:50.305 INFO:teuthology.orchestra.run.smithi092.stdout:ANSI_COLOR="0;31" 2024-04-02T14:08:50.305 INFO:teuthology.orchestra.run.smithi092.stdout:LOGO="fedora-logo-icon" 2024-04-02T14:08:50.305 INFO:teuthology.orchestra.run.smithi092.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-02T14:08:50.306 INFO:teuthology.orchestra.run.smithi092.stdout:HOME_URL="https://centos.org/" 2024-04-02T14:08:50.306 INFO:teuthology.orchestra.run.smithi092.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-02T14:08:50.306 INFO:teuthology.orchestra.run.smithi092.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-02T14:08:50.306 INFO:teuthology.orchestra.run.smithi092.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-02T14:08:50.306 INFO:teuthology.lock.ops:Updating smithi092.front.sepia.ceph.com on lock server 2024-04-02T14:08:50.329 DEBUG:teuthology.orchestra.run.smithi177:> uname -m 2024-04-02T14:08:50.346 INFO:teuthology.orchestra.run.smithi177.stdout:x86_64 2024-04-02T14:08:50.347 DEBUG:teuthology.orchestra.run.smithi177:> cat /etc/os-release 2024-04-02T14:08:50.403 INFO:teuthology.orchestra.run.smithi177.stdout:NAME="CentOS Stream" 2024-04-02T14:08:50.403 INFO:teuthology.orchestra.run.smithi177.stdout:VERSION="9" 2024-04-02T14:08:50.403 INFO:teuthology.orchestra.run.smithi177.stdout:ID="centos" 2024-04-02T14:08:50.403 INFO:teuthology.orchestra.run.smithi177.stdout:ID_LIKE="rhel fedora" 2024-04-02T14:08:50.403 INFO:teuthology.orchestra.run.smithi177.stdout:VERSION_ID="9" 2024-04-02T14:08:50.403 INFO:teuthology.orchestra.run.smithi177.stdout:PLATFORM_ID="platform:el9" 2024-04-02T14:08:50.403 INFO:teuthology.orchestra.run.smithi177.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-02T14:08:50.403 INFO:teuthology.orchestra.run.smithi177.stdout:ANSI_COLOR="0;31" 2024-04-02T14:08:50.403 INFO:teuthology.orchestra.run.smithi177.stdout:LOGO="fedora-logo-icon" 2024-04-02T14:08:50.403 INFO:teuthology.orchestra.run.smithi177.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-02T14:08:50.403 INFO:teuthology.orchestra.run.smithi177.stdout:HOME_URL="https://centos.org/" 2024-04-02T14:08:50.403 INFO:teuthology.orchestra.run.smithi177.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-02T14:08:50.404 INFO:teuthology.orchestra.run.smithi177.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-02T14:08:50.404 INFO:teuthology.orchestra.run.smithi177.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-02T14:08:50.404 INFO:teuthology.lock.ops:Updating smithi177.front.sepia.ceph.com on lock server 2024-04-02T14:08:50.429 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-02T14:08:50.440 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-02T14:08:50.470 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-02T14:08:50.470 DEBUG:teuthology.orchestra.run.smithi090:> test '!' -e /home/ubuntu/cephtest 2024-04-02T14:08:50.472 DEBUG:teuthology.orchestra.run.smithi092:> test '!' -e /home/ubuntu/cephtest 2024-04-02T14:08:50.475 DEBUG:teuthology.orchestra.run.smithi177:> test '!' -e /home/ubuntu/cephtest 2024-04-02T14:08:50.490 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-02T14:08:50.503 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-02T14:08:50.504 DEBUG:teuthology.orchestra.run.smithi090:> test -z $(ls -A /var/lib/ceph) 2024-04-02T14:08:50.529 DEBUG:teuthology.orchestra.run.smithi092:> test -z $(ls -A /var/lib/ceph) 2024-04-02T14:08:50.532 DEBUG:teuthology.orchestra.run.smithi177:> test -z $(ls -A /var/lib/ceph) 2024-04-02T14:08:50.560 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-02T14:08:50.638 INFO:teuthology.run_tasks:Running task kernel... 2024-04-02T14:08:50.653 INFO:teuthology.task.kernel:normalize config orig: {'client': {'branch': 'testing'}, 'kdb': True, 'sha1': 'distro'} 2024-04-02T14:08:50.653 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'branch': 'testing'}} 2024-04-02T14:08:50.653 INFO:teuthology.task.kernel:config {'client.0': {'branch': 'testing'}}, timeout 300 2024-04-02T14:08:50.653 DEBUG:teuthology.orchestra.run.smithi090:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-02T14:08:50.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T14:08:50.670 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&ref=testing 2024-04-02T14:08:50.994 DEBUG:teuthology.task.kernel:sha1 for client.0 is 1df0c7e8a2354c42d738e5454ba83ea7332deac2 2024-04-02T14:08:50.994 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "1df0c7e8a2354c42d738e5454ba83ea7332deac2"... 2024-04-02T14:08:50.994 DEBUG:teuthology.orchestra.run.smithi090:> uname -r 2024-04-02T14:08:51.011 INFO:teuthology.orchestra.run.smithi090.stdout:5.14.0-361.el9.x86_64 2024-04-02T14:08:51.012 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-361.el9.x86_64 vs 1df0c7e8a2354c42d738e5454ba83ea7332deac2 2024-04-02T14:08:51.012 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2024-04-02T14:08:51.012 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum upgrade -y linux-firmware 2024-04-02T14:08:54.123 INFO:teuthology.orchestra.run.smithi090.stdout:CentOS-9 - AppStream 6.6 MB/s | 6.7 MB 00:01 2024-04-02T14:08:55.731 INFO:teuthology.orchestra.run.smithi090.stdout:CentOS-9 - BaseOS 3.9 MB/s | 2.0 MB 00:00 2024-04-02T14:09:01.905 INFO:teuthology.orchestra.run.smithi090.stdout:CentOS Stream 9 - BaseOS 1.4 MB/s | 8.0 MB 00:05 2024-04-02T14:09:07.248 INFO:teuthology.orchestra.run.smithi090.stdout:CentOS Stream 9 - AppStream 4.4 MB/s | 19 MB 00:04 2024-04-02T14:09:13.158 INFO:teuthology.orchestra.run.smithi090.stdout:CentOS Stream 9 - CRB 2.5 MB/s | 6.1 MB 00:02 2024-04-02T14:09:14.560 INFO:teuthology.orchestra.run.smithi090.stdout:CentOS Stream 9 - Extras packages 71 kB/s | 16 kB 00:00 2024-04-02T14:09:17.406 INFO:teuthology.orchestra.run.smithi090.stdout:Extra Packages for Enterprise Linux 7.6 MB/s | 21 MB 00:02 2024-04-02T14:09:22.496 INFO:teuthology.orchestra.run.smithi090.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-04-02T14:09:24.882 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-04-02T14:09:24.888 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-04-02T14:09:24.888 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repository Size 2024-04-02T14:09:24.888 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-04-02T14:09:24.888 INFO:teuthology.orchestra.run.smithi090.stdout:Upgrading: 2024-04-02T14:09:24.888 INFO:teuthology.orchestra.run.smithi090.stdout: linux-firmware noarch 20240219-143.el9 CentOS-BaseOS 375 M 2024-04-02T14:09:24.888 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-02T14:09:24.888 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-04-02T14:09:24.888 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-04-02T14:09:24.888 INFO:teuthology.orchestra.run.smithi090.stdout:Upgrade 1 Package 2024-04-02T14:09:24.888 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-02T14:09:24.889 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 375 M 2024-04-02T14:09:24.889 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2024-04-02T14:09:40.836 INFO:teuthology.orchestra.run.smithi090.stdout:linux-firmware-20240219-143.el9.noarch.rpm 24 MB/s | 375 MB 00:15 2024-04-02T14:09:40.836 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-04-02T14:09:40.836 INFO:teuthology.orchestra.run.smithi090.stdout:Total 24 MB/s | 375 MB 00:15 2024-04-02T14:09:40.836 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2024-04-02T14:09:40.954 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2024-04-02T14:09:40.954 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2024-04-02T14:09:43.362 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2024-04-02T14:09:43.362 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2024-04-02T14:09:50.468 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2024-04-02T14:09:50.680 INFO:teuthology.orchestra.run.smithi090.stdout: Upgrading : linux-firmware-20240219-143.el9.noarch 1/2 2024-04-02T14:09:50.772 INFO:teuthology.orchestra.run.smithi090.stdout: Cleanup : linux-firmware-20230814-139.el9.noarch 2/2 2024-04-02T14:09:52.266 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: linux-firmware-20230814-139.el9.noarch 2/2 2024-04-02T14:09:52.266 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : linux-firmware-20240219-143.el9.noarch 1/2 2024-04-02T14:09:52.649 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : linux-firmware-20230814-139.el9.noarch 2/2 2024-04-02T14:09:52.650 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-02T14:09:52.650 INFO:teuthology.orchestra.run.smithi090.stdout:Upgraded: 2024-04-02T14:09:52.650 INFO:teuthology.orchestra.run.smithi090.stdout: linux-firmware-20240219-143.el9.noarch 2024-04-02T14:09:52.650 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-02T14:09:52.650 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-04-02T14:09:53.007 INFO:teuthology.task.kernel:Downloading kernel 1df0c7e8a2354c42d738e5454ba83ea7332deac2 on client.0... 2024-04-02T14:09:53.008 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=1df0c7e8a2354c42d738e5454ba83ea7332deac2 2024-04-02T14:09:53.305 INFO:teuthology.task.kernel:fetching, builder baseurl is https://3.chacra.ceph.com/r/kernel/testing/1df0c7e8a2354c42d738e5454ba83ea7332deac2/centos/9/flavors/default/x86_64/ 2024-04-02T14:09:53.305 DEBUG:teuthology.orchestra.run.smithi090:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.8.0_g1df0c7e8a235-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://3.chacra.ceph.com/r/kernel/testing/1df0c7e8a2354c42d738e5454ba83ea7332deac2/centos/9/flavors/default/x86_64/ --input-file=- 2024-04-02T14:09:53.308 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi090.front.sepia.ceph.com to complete... 2024-04-02T14:10:06.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02 14:10:06 URL:https://3.chacra.ceph.com/r/kernel/testing/1df0c7e8a2354c42d738e5454ba83ea7332deac2/centos/9/flavors/default/x86_64/kernel-6.8.0_g1df0c7e8a235-1.x86_64.rpm [336317861/336317861] -> "/tmp/kernel.x86_64.rpm" [1] 2024-04-02T14:10:06.674 INFO:teuthology.orchestra.run.smithi090.stderr:FINISHED --2024-04-02 14:10:06-- 2024-04-02T14:10:06.674 INFO:teuthology.orchestra.run.smithi090.stderr:Total wall clock time: 13s 2024-04-02T14:10:06.674 INFO:teuthology.orchestra.run.smithi090.stderr:Downloaded: 1 files, 321M in 13s (24.6 MB/s) 2024-04-02T14:10:06.676 INFO:teuthology.task.kernel:Installing kernel 1df0c7e8a2354c42d738e5454ba83ea7332deac2 on client.0... 2024-04-02T14:10:06.677 DEBUG:teuthology.orchestra.run.smithi090:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2024-04-02T14:10:08.615 INFO:teuthology.orchestra.run.smithi090.stdout:Verifying... ######################################## 2024-04-02T14:10:08.686 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing... ######################################## 2024-04-02T14:10:08.765 INFO:teuthology.orchestra.run.smithi090.stdout:Updating / installing... 2024-04-02T14:10:16.284 INFO:teuthology.orchestra.run.smithi090.stdout:kernel-6.8.0_g1df0c7e8a235-1 ######################################## 2024-04-02T14:10:37.043 INFO:teuthology.orchestra.run.smithi090.stderr:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-04-02T14:11:00.648 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi090.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2024-04-02T14:11:00.649 DEBUG:teuthology.orchestra.run.smithi090:> rpm -qlp /tmp/kernel.x86_64.rpm 2024-04-02T14:11:00.739 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235 2024-04-02T14:11:00.739 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/System.map 2024-04-02T14:11:00.739 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/config 2024-04-02T14:11:00.739 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel 2024-04-02T14:11:00.739 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch 2024-04-02T14:11:00.740 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86 2024-04-02T14:11:00.740 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto 2024-04-02T14:11:00.740 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/blowfish-x86_64.ko 2024-04-02T14:11:00.740 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2024-04-02T14:11:00.740 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2024-04-02T14:11:00.740 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/camellia-x86_64.ko 2024-04-02T14:11:00.740 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2024-04-02T14:11:00.740 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2024-04-02T14:11:00.740 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/crc32-pclmul.ko 2024-04-02T14:11:00.740 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/crc32c-intel.ko 2024-04-02T14:11:00.740 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2024-04-02T14:11:00.740 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2024-04-02T14:11:00.740 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2024-04-02T14:11:00.741 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2024-04-02T14:11:00.741 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/serpent-avx2.ko 2024-04-02T14:11:00.741 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2024-04-02T14:11:00.741 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/sha1-ssse3.ko 2024-04-02T14:11:00.741 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/sha256-ssse3.ko 2024-04-02T14:11:00.741 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/sha512-ssse3.ko 2024-04-02T14:11:00.741 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2024-04-02T14:11:00.741 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2024-04-02T14:11:00.741 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/twofish-x86_64.ko 2024-04-02T14:11:00.741 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kernel 2024-04-02T14:11:00.741 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kernel/cpu 2024-04-02T14:11:00.741 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kernel/cpu/mce 2024-04-02T14:11:00.741 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2024-04-02T14:11:00.741 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kvm 2024-04-02T14:11:00.742 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kvm/kvm-amd.ko 2024-04-02T14:11:00.742 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kvm/kvm-intel.ko 2024-04-02T14:11:00.742 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kvm/kvm.ko 2024-04-02T14:11:00.742 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto 2024-04-02T14:11:00.742 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/ansi_cprng.ko 2024-04-02T14:11:00.742 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/anubis.ko 2024-04-02T14:11:00.742 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/arc4.ko 2024-04-02T14:11:00.742 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/async_tx 2024-04-02T14:11:00.742 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/async_tx/async_memcpy.ko 2024-04-02T14:11:00.742 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/async_tx/async_pq.ko 2024-04-02T14:11:00.742 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/async_tx/async_raid6_recov.ko 2024-04-02T14:11:00.742 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/async_tx/async_tx.ko 2024-04-02T14:11:00.742 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/async_tx/async_xor.ko 2024-04-02T14:11:00.743 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/async_tx/raid6test.ko 2024-04-02T14:11:00.743 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/authenc.ko 2024-04-02T14:11:00.743 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/authencesn.ko 2024-04-02T14:11:00.743 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/blake2b_generic.ko 2024-04-02T14:11:00.743 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/blowfish_common.ko 2024-04-02T14:11:00.743 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/blowfish_generic.ko 2024-04-02T14:11:00.743 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/camellia_generic.ko 2024-04-02T14:11:00.743 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/cast5_generic.ko 2024-04-02T14:11:00.743 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/cast6_generic.ko 2024-04-02T14:11:00.743 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/cast_common.ko 2024-04-02T14:11:00.743 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/ccm.ko 2024-04-02T14:11:00.743 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/cmac.ko 2024-04-02T14:11:00.743 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/crc32_generic.ko 2024-04-02T14:11:00.743 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/crypto_engine.ko 2024-04-02T14:11:00.744 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/crypto_user.ko 2024-04-02T14:11:00.744 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/cts.ko 2024-04-02T14:11:00.744 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/des_generic.ko 2024-04-02T14:11:00.744 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/dh_generic.ko 2024-04-02T14:11:00.744 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/echainiv.ko 2024-04-02T14:11:00.744 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/essiv.ko 2024-04-02T14:11:00.744 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/fcrypt.ko 2024-04-02T14:11:00.744 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/khazad.ko 2024-04-02T14:11:00.744 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/lz4.ko 2024-04-02T14:11:00.744 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/lz4hc.ko 2024-04-02T14:11:00.744 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/md4.ko 2024-04-02T14:11:00.744 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/michael_mic.ko 2024-04-02T14:11:00.744 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/pcbc.ko 2024-04-02T14:11:00.744 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/pcrypt.ko 2024-04-02T14:11:00.744 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/rmd160.ko 2024-04-02T14:11:00.745 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/seed.ko 2024-04-02T14:11:00.745 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/serpent_generic.ko 2024-04-02T14:11:00.745 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/tcrypt.ko 2024-04-02T14:11:00.745 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/tea.ko 2024-04-02T14:11:00.745 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/twofish_common.ko 2024-04-02T14:11:00.745 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/twofish_generic.ko 2024-04-02T14:11:00.745 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/vmac.ko 2024-04-02T14:11:00.745 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/wp512.ko 2024-04-02T14:11:00.745 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/xcbc.ko 2024-04-02T14:11:00.745 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/xor.ko 2024-04-02T14:11:00.745 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/xxhash_generic.ko 2024-04-02T14:11:00.745 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers 2024-04-02T14:11:00.745 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi 2024-04-02T14:11:00.745 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/acpi_ipmi.ko 2024-04-02T14:11:00.746 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/acpi_pad.ko 2024-04-02T14:11:00.746 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/custom_method.ko 2024-04-02T14:11:00.746 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/ec_sys.ko 2024-04-02T14:11:00.746 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/platform_profile.ko 2024-04-02T14:11:00.746 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/sbs.ko 2024-04-02T14:11:00.746 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/sbshc.ko 2024-04-02T14:11:00.746 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/video.ko 2024-04-02T14:11:00.746 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata 2024-04-02T14:11:00.746 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/acard-ahci.ko 2024-04-02T14:11:00.746 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/ahci_platform.ko 2024-04-02T14:11:00.746 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/ata_generic.ko 2024-04-02T14:11:00.746 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/libahci_platform.ko 2024-04-02T14:11:00.746 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_acpi.ko 2024-04-02T14:11:00.746 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_ali.ko 2024-04-02T14:11:00.747 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_amd.ko 2024-04-02T14:11:00.747 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_artop.ko 2024-04-02T14:11:00.747 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_atiixp.ko 2024-04-02T14:11:00.747 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_atp867x.ko 2024-04-02T14:11:00.747 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_cmd640.ko 2024-04-02T14:11:00.747 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_cmd64x.ko 2024-04-02T14:11:00.747 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_cypress.ko 2024-04-02T14:11:00.747 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_efar.ko 2024-04-02T14:11:00.747 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_hpt366.ko 2024-04-02T14:11:00.747 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_hpt37x.ko 2024-04-02T14:11:00.747 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_hpt3x2n.ko 2024-04-02T14:11:00.747 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_hpt3x3.ko 2024-04-02T14:11:00.747 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_it8213.ko 2024-04-02T14:11:00.747 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_it821x.ko 2024-04-02T14:11:00.747 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_jmicron.ko 2024-04-02T14:11:00.748 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_marvell.ko 2024-04-02T14:11:00.748 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_mpiix.ko 2024-04-02T14:11:00.748 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_netcell.ko 2024-04-02T14:11:00.748 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_ninja32.ko 2024-04-02T14:11:00.748 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_ns87410.ko 2024-04-02T14:11:00.748 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_ns87415.ko 2024-04-02T14:11:00.748 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_oldpiix.ko 2024-04-02T14:11:00.748 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_opti.ko 2024-04-02T14:11:00.748 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_optidma.ko 2024-04-02T14:11:00.748 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_pcmcia.ko 2024-04-02T14:11:00.748 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_pdc2027x.ko 2024-04-02T14:11:00.748 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_pdc202xx_old.ko 2024-04-02T14:11:00.748 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_piccolo.ko 2024-04-02T14:11:00.748 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_rdc.ko 2024-04-02T14:11:00.748 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_sch.ko 2024-04-02T14:11:00.749 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_serverworks.ko 2024-04-02T14:11:00.749 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_sil680.ko 2024-04-02T14:11:00.749 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_sis.ko 2024-04-02T14:11:00.749 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_sl82c105.ko 2024-04-02T14:11:00.749 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_triflex.ko 2024-04-02T14:11:00.749 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_via.ko 2024-04-02T14:11:00.749 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pdc_adma.ko 2024-04-02T14:11:00.749 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_inic162x.ko 2024-04-02T14:11:00.749 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_mv.ko 2024-04-02T14:11:00.749 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_nv.ko 2024-04-02T14:11:00.749 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_promise.ko 2024-04-02T14:11:00.749 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_qstor.ko 2024-04-02T14:11:00.749 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_sil.ko 2024-04-02T14:11:00.749 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_sil24.ko 2024-04-02T14:11:00.749 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_sis.ko 2024-04-02T14:11:00.749 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_svw.ko 2024-04-02T14:11:00.750 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_sx4.ko 2024-04-02T14:11:00.750 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_uli.ko 2024-04-02T14:11:00.750 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_via.ko 2024-04-02T14:11:00.750 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_vsc.ko 2024-04-02T14:11:00.750 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/auxdisplay 2024-04-02T14:11:00.750 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/auxdisplay/ks0108.ko 2024-04-02T14:11:00.750 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/base 2024-04-02T14:11:00.750 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/base/regmap 2024-04-02T14:11:00.750 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/base/regmap/regmap-i2c.ko 2024-04-02T14:11:00.750 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/base/regmap/regmap-mmio.ko 2024-04-02T14:11:00.750 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/bcma 2024-04-02T14:11:00.750 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/bcma/bcma.ko 2024-04-02T14:11:00.750 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block 2024-04-02T14:11:00.750 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/aoe 2024-04-02T14:11:00.751 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/aoe/aoe.ko 2024-04-02T14:11:00.751 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/brd.ko 2024-04-02T14:11:00.751 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/drbd 2024-04-02T14:11:00.751 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/drbd/drbd.ko 2024-04-02T14:11:00.751 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/floppy.ko 2024-04-02T14:11:00.751 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/loop.ko 2024-04-02T14:11:00.751 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/mtip32xx 2024-04-02T14:11:00.751 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/mtip32xx/mtip32xx.ko 2024-04-02T14:11:00.751 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/nbd.ko 2024-04-02T14:11:00.751 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/null_blk 2024-04-02T14:11:00.751 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/null_blk/null_blk.ko 2024-04-02T14:11:00.751 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/pktcdvd.ko 2024-04-02T14:11:00.751 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/rbd.ko 2024-04-02T14:11:00.751 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/virtio_blk.ko 2024-04-02T14:11:00.751 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/xen-blkback 2024-04-02T14:11:00.752 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/xen-blkback/xen-blkback.ko 2024-04-02T14:11:00.752 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/xen-blkfront.ko 2024-04-02T14:11:00.752 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/zram 2024-04-02T14:11:00.752 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/zram/zram.ko 2024-04-02T14:11:00.752 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char 2024-04-02T14:11:00.752 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/hangcheck-timer.ko 2024-04-02T14:11:00.752 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/hw_random 2024-04-02T14:11:00.752 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/hw_random/amd-rng.ko 2024-04-02T14:11:00.752 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/hw_random/intel-rng.ko 2024-04-02T14:11:00.752 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/hw_random/timeriomem-rng.ko 2024-04-02T14:11:00.752 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/hw_random/via-rng.ko 2024-04-02T14:11:00.752 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/hw_random/virtio-rng.ko 2024-04-02T14:11:00.752 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ipmi 2024-04-02T14:11:00.752 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ipmi/ipmi_devintf.ko 2024-04-02T14:11:00.752 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2024-04-02T14:11:00.753 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2024-04-02T14:11:00.753 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ipmi/ipmi_si.ko 2024-04-02T14:11:00.753 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ipmi/ipmi_ssif.ko 2024-04-02T14:11:00.753 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2024-04-02T14:11:00.753 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/lp.ko 2024-04-02T14:11:00.753 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/mwave 2024-04-02T14:11:00.753 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/mwave/mwave.ko 2024-04-02T14:11:00.753 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ppdev.ko 2024-04-02T14:11:00.753 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tlclk.ko 2024-04-02T14:11:00.753 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tpm 2024-04-02T14:11:00.753 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tpm/tpm.ko 2024-04-02T14:11:00.753 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tpm/tpm_atmel.ko 2024-04-02T14:11:00.753 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tpm/tpm_infineon.ko 2024-04-02T14:11:00.753 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tpm/tpm_nsc.ko 2024-04-02T14:11:00.754 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tpm/tpm_tis.ko 2024-04-02T14:11:00.754 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tpm/tpm_tis_core.ko 2024-04-02T14:11:00.754 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/uv_mmtimer.ko 2024-04-02T14:11:00.754 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/virtio_console.ko 2024-04-02T14:11:00.754 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/cpufreq 2024-04-02T14:11:00.754 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/cpufreq/acpi-cpufreq.ko 2024-04-02T14:11:00.754 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2024-04-02T14:11:00.754 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/cpufreq/p4-clockmod.ko 2024-04-02T14:11:00.754 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/cpufreq/pcc-cpufreq.ko 2024-04-02T14:11:00.754 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/cpufreq/powernow-k8.ko 2024-04-02T14:11:00.754 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/cpufreq/speedstep-lib.ko 2024-04-02T14:11:00.754 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto 2024-04-02T14:11:00.754 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/ccp 2024-04-02T14:11:00.755 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/ccp/ccp-crypto.ko 2024-04-02T14:11:00.755 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/ccp/ccp.ko 2024-04-02T14:11:00.755 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/intel 2024-04-02T14:11:00.755 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/intel/qat 2024-04-02T14:11:00.755 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/intel/qat/qat_common 2024-04-02T14:11:00.755 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2024-04-02T14:11:00.755 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/intel/qat/qat_dh895xcc 2024-04-02T14:11:00.755 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2024-04-02T14:11:00.755 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/padlock-aes.ko 2024-04-02T14:11:00.755 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/padlock-sha.ko 2024-04-02T14:11:00.755 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/virtio 2024-04-02T14:11:00.755 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/virtio/virtio_crypto.ko 2024-04-02T14:11:00.755 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/dca 2024-04-02T14:11:00.755 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/dca/dca.ko 2024-04-02T14:11:00.755 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/devfreq 2024-04-02T14:11:00.755 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/devfreq/governor_simpleondemand.ko 2024-04-02T14:11:00.756 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/dma 2024-04-02T14:11:00.756 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/dma/dw 2024-04-02T14:11:00.756 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/dma/dw/dw_dmac.ko 2024-04-02T14:11:00.756 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/dma/ioat 2024-04-02T14:11:00.756 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/dma/ioat/ioatdma.ko 2024-04-02T14:11:00.756 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac 2024-04-02T14:11:00.756 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/amd64_edac.ko 2024-04-02T14:11:00.756 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/e752x_edac.ko 2024-04-02T14:11:00.756 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/edac_mce_amd.ko 2024-04-02T14:11:00.756 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/i3000_edac.ko 2024-04-02T14:11:00.756 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/i3200_edac.ko 2024-04-02T14:11:00.756 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/i5100_edac.ko 2024-04-02T14:11:00.756 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/i5400_edac.ko 2024-04-02T14:11:00.756 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/i7300_edac.ko 2024-04-02T14:11:00.756 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/i7core_edac.ko 2024-04-02T14:11:00.756 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/i82975x_edac.ko 2024-04-02T14:11:00.757 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/ie31200_edac.ko 2024-04-02T14:11:00.757 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/sb_edac.ko 2024-04-02T14:11:00.757 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/x38_edac.ko 2024-04-02T14:11:00.757 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/firmware 2024-04-02T14:11:00.757 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/firmware/edd.ko 2024-04-02T14:11:00.757 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/firmware/iscsi_ibft.ko 2024-04-02T14:11:00.757 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/gpio 2024-04-02T14:11:00.757 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/gpio/gpio-ich.ko 2024-04-02T14:11:00.757 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/gpio/gpio-viperboard.ko 2024-04-02T14:11:00.757 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid 2024-04-02T14:11:00.757 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-appleir.ko 2024-04-02T14:11:00.757 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-aureal.ko 2024-04-02T14:11:00.757 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-axff.ko 2024-04-02T14:11:00.757 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-dr.ko 2024-04-02T14:11:00.757 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-elecom.ko 2024-04-02T14:11:00.758 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-elo.ko 2024-04-02T14:11:00.758 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-emsff.ko 2024-04-02T14:11:00.758 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-gaff.ko 2024-04-02T14:11:00.758 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-gt683r.ko 2024-04-02T14:11:00.758 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-gyration.ko 2024-04-02T14:11:00.758 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-holtek-kbd.ko 2024-04-02T14:11:00.758 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-holtek-mouse.ko 2024-04-02T14:11:00.758 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-holtekff.ko 2024-04-02T14:11:00.758 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-hyperv.ko 2024-04-02T14:11:00.758 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-icade.ko 2024-04-02T14:11:00.758 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-keytouch.ko 2024-04-02T14:11:00.758 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-kye.ko 2024-04-02T14:11:00.758 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-lcpower.ko 2024-04-02T14:11:00.759 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-led.ko 2024-04-02T14:11:00.759 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-lenovo.ko 2024-04-02T14:11:00.759 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-logitech-dj.ko 2024-04-02T14:11:00.759 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-logitech-hidpp.ko 2024-04-02T14:11:00.759 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-multitouch.ko 2024-04-02T14:11:00.759 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-ortek.ko 2024-04-02T14:11:00.759 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-penmount.ko 2024-04-02T14:11:00.759 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-petalynx.ko 2024-04-02T14:11:00.759 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-picolcd.ko 2024-04-02T14:11:00.759 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-pl.ko 2024-04-02T14:11:00.759 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-plantronics.ko 2024-04-02T14:11:00.759 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-primax.ko 2024-04-02T14:11:00.759 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-rmi.ko 2024-04-02T14:11:00.759 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-arvo.ko 2024-04-02T14:11:00.759 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-common.ko 2024-04-02T14:11:00.760 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-isku.ko 2024-04-02T14:11:00.760 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-kone.ko 2024-04-02T14:11:00.760 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-koneplus.ko 2024-04-02T14:11:00.760 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-konepure.ko 2024-04-02T14:11:00.760 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-kovaplus.ko 2024-04-02T14:11:00.760 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-lua.ko 2024-04-02T14:11:00.760 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-pyra.ko 2024-04-02T14:11:00.760 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-ryos.ko 2024-04-02T14:11:00.760 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-savu.ko 2024-04-02T14:11:00.760 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat.ko 2024-04-02T14:11:00.760 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-saitek.ko 2024-04-02T14:11:00.760 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-samsung.ko 2024-04-02T14:11:00.760 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-sensor-hub.ko 2024-04-02T14:11:00.760 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-sjoy.ko 2024-04-02T14:11:00.760 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-sony.ko 2024-04-02T14:11:00.761 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-speedlink.ko 2024-04-02T14:11:00.761 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-steelseries.ko 2024-04-02T14:11:00.761 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-sunplus.ko 2024-04-02T14:11:00.761 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-thrustmaster.ko 2024-04-02T14:11:00.761 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-tivo.ko 2024-04-02T14:11:00.761 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-tmff.ko 2024-04-02T14:11:00.761 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-topseed.ko 2024-04-02T14:11:00.761 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-twinhan.ko 2024-04-02T14:11:00.761 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-uclogic.ko 2024-04-02T14:11:00.761 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-waltop.ko 2024-04-02T14:11:00.761 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-wiimote.ko 2024-04-02T14:11:00.761 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-xinmo.ko 2024-04-02T14:11:00.761 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-zpff.ko 2024-04-02T14:11:00.762 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-zydacron.ko 2024-04-02T14:11:00.762 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/uhid.ko 2024-04-02T14:11:00.762 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/wacom.ko 2024-04-02T14:11:00.762 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hv 2024-04-02T14:11:00.762 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hv/hv_balloon.ko 2024-04-02T14:11:00.762 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hv/hv_utils.ko 2024-04-02T14:11:00.762 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hv/hv_vmbus.ko 2024-04-02T14:11:00.762 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon 2024-04-02T14:11:00.762 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/abituguru.ko 2024-04-02T14:11:00.762 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/abituguru3.ko 2024-04-02T14:11:00.762 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/acpi_power_meter.ko 2024-04-02T14:11:00.762 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ad7414.ko 2024-04-02T14:11:00.762 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ad7418.ko 2024-04-02T14:11:00.762 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adc128d818.ko 2024-04-02T14:11:00.763 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adm1025.ko 2024-04-02T14:11:00.763 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adm1026.ko 2024-04-02T14:11:00.763 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adm1029.ko 2024-04-02T14:11:00.763 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adm1031.ko 2024-04-02T14:11:00.763 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adm9240.ko 2024-04-02T14:11:00.763 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ads7828.ko 2024-04-02T14:11:00.763 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adt7410.ko 2024-04-02T14:11:00.763 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adt7411.ko 2024-04-02T14:11:00.763 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adt7462.ko 2024-04-02T14:11:00.763 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adt7470.ko 2024-04-02T14:11:00.763 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adt7475.ko 2024-04-02T14:11:00.763 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adt7x10.ko 2024-04-02T14:11:00.763 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/amc6821.ko 2024-04-02T14:11:00.763 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/applesmc.ko 2024-04-02T14:11:00.764 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/asb100.ko 2024-04-02T14:11:00.764 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/asc7621.ko 2024-04-02T14:11:00.764 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/asus_atk0110.ko 2024-04-02T14:11:00.764 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/atxp1.ko 2024-04-02T14:11:00.764 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/coretemp.ko 2024-04-02T14:11:00.764 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/dell-smm-hwmon.ko 2024-04-02T14:11:00.764 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/dme1737.ko 2024-04-02T14:11:00.764 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ds1621.ko 2024-04-02T14:11:00.764 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ds620.ko 2024-04-02T14:11:00.764 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/emc1403.ko 2024-04-02T14:11:00.764 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/emc6w201.ko 2024-04-02T14:11:00.764 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/f71805f.ko 2024-04-02T14:11:00.765 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/f71882fg.ko 2024-04-02T14:11:00.765 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/f75375s.ko 2024-04-02T14:11:00.765 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/fam15h_power.ko 2024-04-02T14:11:00.765 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/fschmd.ko 2024-04-02T14:11:00.765 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/g760a.ko 2024-04-02T14:11:00.765 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/g762.ko 2024-04-02T14:11:00.765 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/gl518sm.ko 2024-04-02T14:11:00.765 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/gl520sm.ko 2024-04-02T14:11:00.765 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/hwmon-vid.ko 2024-04-02T14:11:00.765 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/i5500_temp.ko 2024-04-02T14:11:00.765 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/i5k_amb.ko 2024-04-02T14:11:00.765 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ibmaem.ko 2024-04-02T14:11:00.765 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ibmpex.ko 2024-04-02T14:11:00.765 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ina209.ko 2024-04-02T14:11:00.766 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ina2xx.ko 2024-04-02T14:11:00.766 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/it87.ko 2024-04-02T14:11:00.766 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/k10temp.ko 2024-04-02T14:11:00.766 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/k8temp.ko 2024-04-02T14:11:00.766 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lineage-pem.ko 2024-04-02T14:11:00.766 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm63.ko 2024-04-02T14:11:00.766 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm73.ko 2024-04-02T14:11:00.766 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm75.ko 2024-04-02T14:11:00.766 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm77.ko 2024-04-02T14:11:00.766 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm78.ko 2024-04-02T14:11:00.766 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm80.ko 2024-04-02T14:11:00.766 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm83.ko 2024-04-02T14:11:00.766 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm85.ko 2024-04-02T14:11:00.767 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm87.ko 2024-04-02T14:11:00.767 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm90.ko 2024-04-02T14:11:00.767 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm92.ko 2024-04-02T14:11:00.767 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm93.ko 2024-04-02T14:11:00.767 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm95234.ko 2024-04-02T14:11:00.767 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm95241.ko 2024-04-02T14:11:00.767 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm95245.ko 2024-04-02T14:11:00.767 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ltc2945.ko 2024-04-02T14:11:00.767 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ltc4151.ko 2024-04-02T14:11:00.767 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ltc4215.ko 2024-04-02T14:11:00.767 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ltc4222.ko 2024-04-02T14:11:00.767 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ltc4245.ko 2024-04-02T14:11:00.767 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ltc4260.ko 2024-04-02T14:11:00.767 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ltc4261.ko 2024-04-02T14:11:00.767 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/max16065.ko 2024-04-02T14:11:00.767 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/max1619.ko 2024-04-02T14:11:00.767 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/max1668.ko 2024-04-02T14:11:00.768 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/max197.ko 2024-04-02T14:11:00.768 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/max6639.ko 2024-04-02T14:11:00.768 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/max6650.ko 2024-04-02T14:11:00.768 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/max6697.ko 2024-04-02T14:11:00.768 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/mcp3021.ko 2024-04-02T14:11:00.768 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/nct6683.ko 2024-04-02T14:11:00.768 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/nct6775-core.ko 2024-04-02T14:11:00.768 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/nct6775.ko 2024-04-02T14:11:00.768 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/nct7802.ko 2024-04-02T14:11:00.768 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pc87360.ko 2024-04-02T14:11:00.768 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pc87427.ko 2024-04-02T14:11:00.768 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pcf8591.ko 2024-04-02T14:11:00.768 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus 2024-04-02T14:11:00.768 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/adm1275.ko 2024-04-02T14:11:00.768 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/lm25066.ko 2024-04-02T14:11:00.768 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/ltc2978.ko 2024-04-02T14:11:00.768 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/max16064.ko 2024-04-02T14:11:00.769 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/max34440.ko 2024-04-02T14:11:00.769 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/max8688.ko 2024-04-02T14:11:00.769 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/pmbus.ko 2024-04-02T14:11:00.769 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2024-04-02T14:11:00.769 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/tps40422.ko 2024-04-02T14:11:00.769 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/ucd9000.ko 2024-04-02T14:11:00.769 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/ucd9200.ko 2024-04-02T14:11:00.769 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/zl6100.ko 2024-04-02T14:11:00.769 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/powr1220.ko 2024-04-02T14:11:00.769 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/sch5627.ko 2024-04-02T14:11:00.769 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/sch5636.ko 2024-04-02T14:11:00.769 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/sch56xx-common.ko 2024-04-02T14:11:00.769 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/sht15.ko 2024-04-02T14:11:00.769 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/sht21.ko 2024-04-02T14:11:00.769 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/shtc1.ko 2024-04-02T14:11:00.769 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/sis5595.ko 2024-04-02T14:11:00.770 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/smsc47b397.ko 2024-04-02T14:11:00.770 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/smsc47m1.ko 2024-04-02T14:11:00.770 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/smsc47m192.ko 2024-04-02T14:11:00.770 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/thmc50.ko 2024-04-02T14:11:00.770 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/tmp102.ko 2024-04-02T14:11:00.770 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/tmp103.ko 2024-04-02T14:11:00.770 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/tmp401.ko 2024-04-02T14:11:00.770 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/tmp421.ko 2024-04-02T14:11:00.770 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/via-cputemp.ko 2024-04-02T14:11:00.770 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/via686a.ko 2024-04-02T14:11:00.770 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/vt1211.ko 2024-04-02T14:11:00.770 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/vt8231.ko 2024-04-02T14:11:00.770 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83627ehf.ko 2024-04-02T14:11:00.770 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83627hf.ko 2024-04-02T14:11:00.771 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83781d.ko 2024-04-02T14:11:00.771 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83791d.ko 2024-04-02T14:11:00.771 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83792d.ko 2024-04-02T14:11:00.771 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83793.ko 2024-04-02T14:11:00.771 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83795.ko 2024-04-02T14:11:00.771 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83l785ts.ko 2024-04-02T14:11:00.771 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83l786ng.ko 2024-04-02T14:11:00.771 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c 2024-04-02T14:11:00.771 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/algos 2024-04-02T14:11:00.771 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2024-04-02T14:11:00.771 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2024-04-02T14:11:00.771 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses 2024-04-02T14:11:00.771 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2024-04-02T14:11:00.771 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-amd756.ko 2024-04-02T14:11:00.772 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-amd8111.ko 2024-04-02T14:11:00.772 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2024-04-02T14:11:00.772 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-designware-core.ko 2024-04-02T14:11:00.772 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2024-04-02T14:11:00.772 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2024-04-02T14:11:00.772 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2024-04-02T14:11:00.772 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-i801.ko 2024-04-02T14:11:00.772 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-isch.ko 2024-04-02T14:11:00.772 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-ismt.ko 2024-04-02T14:11:00.772 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2024-04-02T14:11:00.772 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-nforce2.ko 2024-04-02T14:11:00.772 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-parport.ko 2024-04-02T14:11:00.772 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2024-04-02T14:11:00.772 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-piix4.ko 2024-04-02T14:11:00.773 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-scmi.ko 2024-04-02T14:11:00.773 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-simtec.ko 2024-04-02T14:11:00.773 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-sis96x.ko 2024-04-02T14:11:00.773 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2024-04-02T14:11:00.773 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-via.ko 2024-04-02T14:11:00.773 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-viapro.ko 2024-04-02T14:11:00.773 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-viperboard.ko 2024-04-02T14:11:00.773 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/i2c-dev.ko 2024-04-02T14:11:00.773 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/i2c-mux.ko 2024-04-02T14:11:00.773 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/i2c-slave-eeprom.ko 2024-04-02T14:11:00.773 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/i2c-smbus.ko 2024-04-02T14:11:00.773 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/i2c-stub.ko 2024-04-02T14:11:00.773 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband 2024-04-02T14:11:00.773 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core 2024-04-02T14:11:00.773 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core/ib_cm.ko 2024-04-02T14:11:00.774 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core/ib_core.ko 2024-04-02T14:11:00.774 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core/ib_umad.ko 2024-04-02T14:11:00.774 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core/ib_uverbs.ko 2024-04-02T14:11:00.774 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core/iw_cm.ko 2024-04-02T14:11:00.774 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core/rdma_cm.ko 2024-04-02T14:11:00.774 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core/rdma_ucm.ko 2024-04-02T14:11:00.774 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw 2024-04-02T14:11:00.774 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/cxgb4 2024-04-02T14:11:00.774 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2024-04-02T14:11:00.774 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/mlx4 2024-04-02T14:11:00.774 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2024-04-02T14:11:00.774 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/mlx5 2024-04-02T14:11:00.774 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2024-04-02T14:11:00.774 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/mthca 2024-04-02T14:11:00.775 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2024-04-02T14:11:00.775 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp 2024-04-02T14:11:00.775 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/ipoib 2024-04-02T14:11:00.775 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2024-04-02T14:11:00.775 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/iser 2024-04-02T14:11:00.775 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2024-04-02T14:11:00.775 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/isert 2024-04-02T14:11:00.775 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2024-04-02T14:11:00.775 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/srp 2024-04-02T14:11:00.775 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2024-04-02T14:11:00.775 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/srpt 2024-04-02T14:11:00.775 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2024-04-02T14:11:00.775 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input 2024-04-02T14:11:00.775 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/gameport 2024-04-02T14:11:00.776 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/gameport/emu10k1-gp.ko 2024-04-02T14:11:00.776 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/gameport/fm801-gp.ko 2024-04-02T14:11:00.776 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/gameport/gameport.ko 2024-04-02T14:11:00.776 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/joydev.ko 2024-04-02T14:11:00.776 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/keyboard 2024-04-02T14:11:00.776 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/keyboard/gpio_keys.ko 2024-04-02T14:11:00.776 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse 2024-04-02T14:11:00.776 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/appletouch.ko 2024-04-02T14:11:00.776 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/bcm5974.ko 2024-04-02T14:11:00.776 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/cyapatp.ko 2024-04-02T14:11:00.776 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/elan_i2c.ko 2024-04-02T14:11:00.776 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/sermouse.ko 2024-04-02T14:11:00.776 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/synaptics_i2c.ko 2024-04-02T14:11:00.776 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/synaptics_usb.ko 2024-04-02T14:11:00.777 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/vsxxxaa.ko 2024-04-02T14:11:00.777 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/rmi4 2024-04-02T14:11:00.777 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/rmi4/rmi_core.ko 2024-04-02T14:11:00.777 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/serio 2024-04-02T14:11:00.777 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/serio/altera_ps2.ko 2024-04-02T14:11:00.777 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/serio/arc_ps2.ko 2024-04-02T14:11:00.777 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/serio/hyperv-keyboard.ko 2024-04-02T14:11:00.777 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/serio/serio_raw.ko 2024-04-02T14:11:00.777 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/sparse-keymap.ko 2024-04-02T14:11:00.777 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds 2024-04-02T14:11:00.777 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/leds-blinkm.ko 2024-04-02T14:11:00.777 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/leds-lm3530.ko 2024-04-02T14:11:00.777 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/leds-lp3944.ko 2024-04-02T14:11:00.777 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/leds-ss4200.ko 2024-04-02T14:11:00.778 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger 2024-04-02T14:11:00.778 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-audio.ko 2024-04-02T14:11:00.778 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2024-04-02T14:11:00.778 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-camera.ko 2024-04-02T14:11:00.778 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2024-04-02T14:11:00.778 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-gpio.ko 2024-04-02T14:11:00.778 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2024-04-02T14:11:00.778 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2024-04-02T14:11:00.778 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-timer.ko 2024-04-02T14:11:00.778 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-transient.ko 2024-04-02T14:11:00.778 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md 2024-04-02T14:11:00.778 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/bcache 2024-04-02T14:11:00.778 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/bcache/bcache.ko 2024-04-02T14:11:00.778 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-bio-prison.ko 2024-04-02T14:11:00.779 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-cache-smq.ko 2024-04-02T14:11:00.779 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-cache.ko 2024-04-02T14:11:00.779 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-crypt.ko 2024-04-02T14:11:00.779 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-delay.ko 2024-04-02T14:11:00.779 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-flakey.ko 2024-04-02T14:11:00.779 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-log-userspace.ko 2024-04-02T14:11:00.779 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-multipath.ko 2024-04-02T14:11:00.779 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-queue-length.ko 2024-04-02T14:11:00.779 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-raid.ko 2024-04-02T14:11:00.779 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-round-robin.ko 2024-04-02T14:11:00.779 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-service-time.ko 2024-04-02T14:11:00.779 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-switch.ko 2024-04-02T14:11:00.779 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-thin-pool.ko 2024-04-02T14:11:00.779 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-verity.ko 2024-04-02T14:11:00.779 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/persistent-data 2024-04-02T14:11:00.780 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2024-04-02T14:11:00.780 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/raid0.ko 2024-04-02T14:11:00.780 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/raid1.ko 2024-04-02T14:11:00.780 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/raid10.ko 2024-04-02T14:11:00.780 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/raid456.ko 2024-04-02T14:11:00.780 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message 2024-04-02T14:11:00.780 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion 2024-04-02T14:11:00.780 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion/mptbase.ko 2024-04-02T14:11:00.780 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion/mptctl.ko 2024-04-02T14:11:00.780 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion/mptfc.ko 2024-04-02T14:11:00.780 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion/mptlan.ko 2024-04-02T14:11:00.780 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion/mptsas.ko 2024-04-02T14:11:00.780 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion/mptscsih.ko 2024-04-02T14:11:00.780 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion/mptspi.ko 2024-04-02T14:11:00.781 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd 2024-04-02T14:11:00.781 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/lpc_ich.ko 2024-04-02T14:11:00.781 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/lpc_sch.ko 2024-04-02T14:11:00.781 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/mfd-core.ko 2024-04-02T14:11:00.781 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/sm501.ko 2024-04-02T14:11:00.781 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/viperboard.ko 2024-04-02T14:11:00.781 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/vx855.ko 2024-04-02T14:11:00.781 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/wl1273-core.ko 2024-04-02T14:11:00.781 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc 2024-04-02T14:11:00.781 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/altera-stapl 2024-04-02T14:11:00.781 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2024-04-02T14:11:00.781 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/apds9802als.ko 2024-04-02T14:11:00.781 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/apds990x.ko 2024-04-02T14:11:00.781 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/bh1770glc.ko 2024-04-02T14:11:00.782 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/cb710 2024-04-02T14:11:00.782 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/cb710/cb710.ko 2024-04-02T14:11:00.782 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/eeprom 2024-04-02T14:11:00.782 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/eeprom/at24.ko 2024-04-02T14:11:00.782 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2024-04-02T14:11:00.782 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/eeprom/max6875.ko 2024-04-02T14:11:00.782 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/enclosure.ko 2024-04-02T14:11:00.782 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/hpilo.ko 2024-04-02T14:11:00.782 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/isl29003.ko 2024-04-02T14:11:00.782 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/isl29020.ko 2024-04-02T14:11:00.782 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/lis3lv02d 2024-04-02T14:11:00.782 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2024-04-02T14:11:00.782 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2024-04-02T14:11:00.783 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/mei 2024-04-02T14:11:00.783 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/mei/mei-me.ko 2024-04-02T14:11:00.783 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/mei/mei-txe.ko 2024-04-02T14:11:00.783 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/mei/mei.ko 2024-04-02T14:11:00.783 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/sgi-gru 2024-04-02T14:11:00.783 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/sgi-gru/gru.ko 2024-04-02T14:11:00.783 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/sgi-xp 2024-04-02T14:11:00.783 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/sgi-xp/xp.ko 2024-04-02T14:11:00.783 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/sgi-xp/xpc.ko 2024-04-02T14:11:00.783 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/sgi-xp/xpnet.ko 2024-04-02T14:11:00.783 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/tifm_7xx1.ko 2024-04-02T14:11:00.783 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/tifm_core.ko 2024-04-02T14:11:00.783 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/tsl2550.ko 2024-04-02T14:11:00.783 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/vmw_balloon.ko 2024-04-02T14:11:00.784 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/vmw_vmci 2024-04-02T14:11:00.784 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2024-04-02T14:11:00.784 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net 2024-04-02T14:11:00.784 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/bonding 2024-04-02T14:11:00.784 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/bonding/bonding.ko 2024-04-02T14:11:00.784 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dsa 2024-04-02T14:11:00.784 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dsa/b53 2024-04-02T14:11:00.784 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dsa/b53/b53_common.ko 2024-04-02T14:11:00.784 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dsa/bcm-sf2.ko 2024-04-02T14:11:00.784 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dsa/mv88e6060.ko 2024-04-02T14:11:00.784 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dsa/mv88e6xxx 2024-04-02T14:11:00.784 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2024-04-02T14:11:00.784 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dummy.ko 2024-04-02T14:11:00.784 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/eql.ko 2024-04-02T14:11:00.785 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet 2024-04-02T14:11:00.785 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/3com 2024-04-02T14:11:00.785 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2024-04-02T14:11:00.785 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2024-04-02T14:11:00.785 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/3com/3c59x.ko 2024-04-02T14:11:00.785 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/3com/typhoon.ko 2024-04-02T14:11:00.785 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/8390 2024-04-02T14:11:00.785 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/8390/8390.ko 2024-04-02T14:11:00.785 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2024-04-02T14:11:00.785 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2024-04-02T14:11:00.785 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2024-04-02T14:11:00.785 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/adaptec 2024-04-02T14:11:00.785 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/adaptec/starfire.ko 2024-04-02T14:11:00.785 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/agere 2024-04-02T14:11:00.786 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/agere/et131x.ko 2024-04-02T14:11:00.786 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/alteon 2024-04-02T14:11:00.786 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/alteon/acenic.ko 2024-04-02T14:11:00.786 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/altera 2024-04-02T14:11:00.786 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/altera/altera_tse.ko 2024-04-02T14:11:00.786 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/amd 2024-04-02T14:11:00.786 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/amd/amd8111e.ko 2024-04-02T14:11:00.786 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2024-04-02T14:11:00.786 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/amd/pcnet32.ko 2024-04-02T14:11:00.786 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros 2024-04-02T14:11:00.786 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/alx 2024-04-02T14:11:00.786 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2024-04-02T14:11:00.786 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/atl1c 2024-04-02T14:11:00.786 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2024-04-02T14:11:00.787 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/atl1e 2024-04-02T14:11:00.787 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2024-04-02T14:11:00.787 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/atlx 2024-04-02T14:11:00.787 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2024-04-02T14:11:00.787 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2024-04-02T14:11:00.787 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom 2024-04-02T14:11:00.787 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/b44.ko 2024-04-02T14:11:00.787 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2024-04-02T14:11:00.787 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/bnx2x 2024-04-02T14:11:00.787 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2024-04-02T14:11:00.788 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/cnic.ko 2024-04-02T14:11:00.788 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/genet 2024-04-02T14:11:00.788 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2024-04-02T14:11:00.788 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/tg3.ko 2024-04-02T14:11:00.788 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/brocade 2024-04-02T14:11:00.788 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/brocade/bna 2024-04-02T14:11:00.788 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2024-04-02T14:11:00.788 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio 2024-04-02T14:11:00.788 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb 2024-04-02T14:11:00.788 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2024-04-02T14:11:00.788 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb3 2024-04-02T14:11:00.789 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2024-04-02T14:11:00.789 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb4 2024-04-02T14:11:00.789 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2024-04-02T14:11:00.789 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2024-04-02T14:11:00.789 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2024-04-02T14:11:00.789 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/libcxgb 2024-04-02T14:11:00.789 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2024-04-02T14:11:00.789 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/cisco 2024-04-02T14:11:00.789 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/cisco/enic 2024-04-02T14:11:00.789 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2024-04-02T14:11:00.789 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec 2024-04-02T14:11:00.789 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec/tulip 2024-04-02T14:11:00.789 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2024-04-02T14:11:00.789 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2024-04-02T14:11:00.789 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2024-04-02T14:11:00.790 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2024-04-02T14:11:00.790 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2024-04-02T14:11:00.790 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2024-04-02T14:11:00.790 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dlink 2024-04-02T14:11:00.790 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dlink/dl2k.ko 2024-04-02T14:11:00.790 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dlink/sundance.ko 2024-04-02T14:11:00.790 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dnet.ko 2024-04-02T14:11:00.790 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/emulex 2024-04-02T14:11:00.790 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/emulex/benet 2024-04-02T14:11:00.790 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2024-04-02T14:11:00.790 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/ethoc.ko 2024-04-02T14:11:00.790 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/fealnx.ko 2024-04-02T14:11:00.790 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel 2024-04-02T14:11:00.790 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/e100.ko 2024-04-02T14:11:00.791 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/e1000 2024-04-02T14:11:00.791 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2024-04-02T14:11:00.791 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/e1000e 2024-04-02T14:11:00.791 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2024-04-02T14:11:00.791 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/fm10k 2024-04-02T14:11:00.791 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2024-04-02T14:11:00.791 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/i40e 2024-04-02T14:11:00.791 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2024-04-02T14:11:00.791 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/iavf 2024-04-02T14:11:00.791 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2024-04-02T14:11:00.791 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/igb 2024-04-02T14:11:00.791 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/igb/igb.ko 2024-04-02T14:11:00.791 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/igbvf 2024-04-02T14:11:00.791 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2024-04-02T14:11:00.792 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/ixgbe 2024-04-02T14:11:00.792 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2024-04-02T14:11:00.792 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/ixgbevf 2024-04-02T14:11:00.792 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2024-04-02T14:11:00.792 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/jme.ko 2024-04-02T14:11:00.792 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/marvell 2024-04-02T14:11:00.792 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2024-04-02T14:11:00.792 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/marvell/skge.ko 2024-04-02T14:11:00.792 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/marvell/sky2.ko 2024-04-02T14:11:00.792 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/mellanox 2024-04-02T14:11:00.792 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/mellanox/mlx4 2024-04-02T14:11:00.792 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2024-04-02T14:11:00.792 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2024-04-02T14:11:00.792 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/mellanox/mlx5 2024-04-02T14:11:00.792 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/mellanox/mlx5/core 2024-04-02T14:11:00.793 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2024-04-02T14:11:00.793 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/micrel 2024-04-02T14:11:00.793 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2024-04-02T14:11:00.793 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/myricom 2024-04-02T14:11:00.793 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/myricom/myri10ge 2024-04-02T14:11:00.793 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2024-04-02T14:11:00.793 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/natsemi 2024-04-02T14:11:00.793 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2024-04-02T14:11:00.793 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2024-04-02T14:11:00.793 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/neterion 2024-04-02T14:11:00.793 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/neterion/s2io.ko 2024-04-02T14:11:00.793 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/nvidia 2024-04-02T14:11:00.793 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2024-04-02T14:11:00.793 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/packetengines 2024-04-02T14:11:00.794 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2024-04-02T14:11:00.794 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2024-04-02T14:11:00.794 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/qlogic 2024-04-02T14:11:00.794 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/qlogic/netxen 2024-04-02T14:11:00.794 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2024-04-02T14:11:00.794 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2024-04-02T14:11:00.794 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/qlogic/qlcnic 2024-04-02T14:11:00.794 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2024-04-02T14:11:00.794 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/rdc 2024-04-02T14:11:00.794 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/rdc/r6040.ko 2024-04-02T14:11:00.794 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/realtek 2024-04-02T14:11:00.794 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/realtek/8139cp.ko 2024-04-02T14:11:00.794 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/realtek/8139too.ko 2024-04-02T14:11:00.794 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/realtek/atp.ko 2024-04-02T14:11:00.795 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/realtek/r8169.ko 2024-04-02T14:11:00.795 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/rocker 2024-04-02T14:11:00.795 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/rocker/rocker.ko 2024-04-02T14:11:00.795 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sfc 2024-04-02T14:11:00.795 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sfc/sfc.ko 2024-04-02T14:11:00.795 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/silan 2024-04-02T14:11:00.795 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/silan/sc92031.ko 2024-04-02T14:11:00.795 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sis 2024-04-02T14:11:00.795 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sis/sis190.ko 2024-04-02T14:11:00.795 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sis/sis900.ko 2024-04-02T14:11:00.795 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/smsc 2024-04-02T14:11:00.795 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/smsc/epic100.ko 2024-04-02T14:11:00.795 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2024-04-02T14:11:00.795 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2024-04-02T14:11:00.795 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2024-04-02T14:11:00.795 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/stmicro 2024-04-02T14:11:00.796 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/stmicro/stmmac 2024-04-02T14:11:00.796 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2024-04-02T14:11:00.796 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2024-04-02T14:11:00.796 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sun 2024-04-02T14:11:00.796 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sun/cassini.ko 2024-04-02T14:11:00.796 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sun/niu.ko 2024-04-02T14:11:00.796 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sun/sungem.ko 2024-04-02T14:11:00.796 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sun/sunhme.ko 2024-04-02T14:11:00.796 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/tehuti 2024-04-02T14:11:00.796 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2024-04-02T14:11:00.796 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/ti 2024-04-02T14:11:00.796 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/ti/tlan.ko 2024-04-02T14:11:00.796 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/via 2024-04-02T14:11:00.796 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/via/via-rhine.ko 2024-04-02T14:11:00.796 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/via/via-velocity.ko 2024-04-02T14:11:00.797 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/wiznet 2024-04-02T14:11:00.797 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/wiznet/w5100.ko 2024-04-02T14:11:00.797 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/wiznet/w5300.ko 2024-04-02T14:11:00.797 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/xircom 2024-04-02T14:11:00.797 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2024-04-02T14:11:00.797 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/geneve.ko 2024-04-02T14:11:00.797 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/hyperv 2024-04-02T14:11:00.797 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/hyperv/hv_netvsc.ko 2024-04-02T14:11:00.797 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ieee802154 2024-04-02T14:11:00.797 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ieee802154/fakelb.ko 2024-04-02T14:11:00.797 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ifb.ko 2024-04-02T14:11:00.797 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ipvlan 2024-04-02T14:11:00.797 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ipvlan/ipvlan.ko 2024-04-02T14:11:00.797 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/macvlan.ko 2024-04-02T14:11:00.797 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/macvtap.ko 2024-04-02T14:11:00.797 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/mdio 2024-04-02T14:11:00.798 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/mdio.ko 2024-04-02T14:11:00.798 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2024-04-02T14:11:00.798 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/mdio/mdio-bitbang.ko 2024-04-02T14:11:00.798 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/mdio/mdio-regmap.ko 2024-04-02T14:11:00.798 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/mii.ko 2024-04-02T14:11:00.798 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/net_failover.ko 2024-04-02T14:11:00.798 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/netconsole.ko 2024-04-02T14:11:00.798 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/nlmon.ko 2024-04-02T14:11:00.798 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/pcs 2024-04-02T14:11:00.798 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/pcs/pcs-lynx.ko 2024-04-02T14:11:00.798 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/pcs/pcs_xpcs.ko 2024-04-02T14:11:00.798 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy 2024-04-02T14:11:00.798 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/amd.ko 2024-04-02T14:11:00.798 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/bcm-phy-lib.ko 2024-04-02T14:11:00.799 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/bcm-phy-ptp.ko 2024-04-02T14:11:00.799 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/bcm7xxx.ko 2024-04-02T14:11:00.799 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/bcm87xx.ko 2024-04-02T14:11:00.799 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/broadcom.ko 2024-04-02T14:11:00.799 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/cicada.ko 2024-04-02T14:11:00.799 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/davicom.ko 2024-04-02T14:11:00.799 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/dp83640.ko 2024-04-02T14:11:00.799 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/et1011c.ko 2024-04-02T14:11:00.799 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/icplus.ko 2024-04-02T14:11:00.799 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/lxt.ko 2024-04-02T14:11:00.799 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/marvell.ko 2024-04-02T14:11:00.799 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/micrel.ko 2024-04-02T14:11:00.799 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/national.ko 2024-04-02T14:11:00.799 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/phylink.ko 2024-04-02T14:11:00.799 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/qsemi.ko 2024-04-02T14:11:00.799 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/realtek.ko 2024-04-02T14:11:00.800 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/smsc.ko 2024-04-02T14:11:00.800 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/ste10Xp.ko 2024-04-02T14:11:00.800 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/vitesse.ko 2024-04-02T14:11:00.800 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp 2024-04-02T14:11:00.800 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/bsd_comp.ko 2024-04-02T14:11:00.800 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/ppp_async.ko 2024-04-02T14:11:00.800 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/ppp_deflate.ko 2024-04-02T14:11:00.800 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/ppp_generic.ko 2024-04-02T14:11:00.800 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/ppp_mppe.ko 2024-04-02T14:11:00.800 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/ppp_synctty.ko 2024-04-02T14:11:00.800 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/pppoe.ko 2024-04-02T14:11:00.800 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/pppox.ko 2024-04-02T14:11:00.800 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/pptp.ko 2024-04-02T14:11:00.800 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/slip 2024-04-02T14:11:00.801 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/slip/slhc.ko 2024-04-02T14:11:00.801 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/slip/slip.ko 2024-04-02T14:11:00.801 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/sungem_phy.ko 2024-04-02T14:11:00.801 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/tap.ko 2024-04-02T14:11:00.801 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/team 2024-04-02T14:11:00.801 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/team/team.ko 2024-04-02T14:11:00.801 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/team/team_mode_activebackup.ko 2024-04-02T14:11:00.801 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/team/team_mode_broadcast.ko 2024-04-02T14:11:00.801 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/team/team_mode_loadbalance.ko 2024-04-02T14:11:00.801 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/team/team_mode_random.ko 2024-04-02T14:11:00.801 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/team/team_mode_roundrobin.ko 2024-04-02T14:11:00.801 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/tun.ko 2024-04-02T14:11:00.801 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/veth.ko 2024-04-02T14:11:00.801 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/virtio_net.ko 2024-04-02T14:11:00.801 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/vmxnet3 2024-04-02T14:11:00.802 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/vmxnet3/vmxnet3.ko 2024-04-02T14:11:00.802 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/vxlan 2024-04-02T14:11:00.802 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/vxlan/vxlan.ko 2024-04-02T14:11:00.802 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/xen-netback 2024-04-02T14:11:00.802 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/xen-netback/xen-netback.ko 2024-04-02T14:11:00.802 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/xen-netfront.ko 2024-04-02T14:11:00.802 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ntb 2024-04-02T14:11:00.802 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ntb/ntb.ko 2024-04-02T14:11:00.802 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/nvme 2024-04-02T14:11:00.802 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/nvme/host 2024-04-02T14:11:00.802 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/nvme/host/nvme-core.ko 2024-04-02T14:11:00.802 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/nvme/host/nvme.ko 2024-04-02T14:11:00.802 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/parport 2024-04-02T14:11:00.802 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/parport/parport.ko 2024-04-02T14:11:00.802 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/parport/parport_cs.ko 2024-04-02T14:11:00.803 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/parport/parport_pc.ko 2024-04-02T14:11:00.803 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/parport/parport_serial.ko 2024-04-02T14:11:00.803 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pci 2024-04-02T14:11:00.803 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pci/hotplug 2024-04-02T14:11:00.803 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2024-04-02T14:11:00.803 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pci/pcie 2024-04-02T14:11:00.803 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pci/pcie/aer_inject.ko 2024-04-02T14:11:00.803 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pci/xen-pcifront.ko 2024-04-02T14:11:00.803 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pcmcia 2024-04-02T14:11:00.803 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pcmcia/i82092.ko 2024-04-02T14:11:00.803 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pcmcia/pd6729.ko 2024-04-02T14:11:00.803 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pcmcia/yenta_socket.ko 2024-04-02T14:11:00.803 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pinctrl 2024-04-02T14:11:00.803 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pinctrl/intel 2024-04-02T14:11:00.803 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2024-04-02T14:11:00.804 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform 2024-04-02T14:11:00.804 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86 2024-04-02T14:11:00.804 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/acer-wmi.ko 2024-04-02T14:11:00.804 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/acerhdf.ko 2024-04-02T14:11:00.804 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/apple-gmux.ko 2024-04-02T14:11:00.804 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/asus-laptop.ko 2024-04-02T14:11:00.804 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/asus-nb-wmi.ko 2024-04-02T14:11:00.804 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/asus-wmi.ko 2024-04-02T14:11:00.804 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/classmate-laptop.ko 2024-04-02T14:11:00.804 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/eeepc-laptop.ko 2024-04-02T14:11:00.804 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/eeepc-wmi.ko 2024-04-02T14:11:00.804 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/fujitsu-laptop.ko 2024-04-02T14:11:00.804 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/fujitsu-tablet.ko 2024-04-02T14:11:00.804 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/hdaps.ko 2024-04-02T14:11:00.804 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/intel 2024-04-02T14:11:00.804 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/intel/intel-rst.ko 2024-04-02T14:11:00.805 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/intel_ips.ko 2024-04-02T14:11:00.805 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/msi-wmi.ko 2024-04-02T14:11:00.805 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/mxm-wmi.ko 2024-04-02T14:11:00.805 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/panasonic-laptop.ko 2024-04-02T14:11:00.805 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/samsung-laptop.ko 2024-04-02T14:11:00.805 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/samsung-q10.ko 2024-04-02T14:11:00.805 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/topstar-laptop.ko 2024-04-02T14:11:00.805 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2024-04-02T14:11:00.805 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/toshiba_haps.ko 2024-04-02T14:11:00.805 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/wmi-bmof.ko 2024-04-02T14:11:00.805 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/wmi.ko 2024-04-02T14:11:00.805 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/powercap 2024-04-02T14:11:00.805 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/powercap/intel_rapl_common.ko 2024-04-02T14:11:00.805 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/powercap/intel_rapl_msr.ko 2024-04-02T14:11:00.805 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pps 2024-04-02T14:11:00.805 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pps/clients 2024-04-02T14:11:00.806 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pps/clients/pps-gpio.ko 2024-04-02T14:11:00.806 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pps/clients/pps-ldisc.ko 2024-04-02T14:11:00.806 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pps/clients/pps_parport.ko 2024-04-02T14:11:00.806 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pps/pps_core.ko 2024-04-02T14:11:00.806 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ptp 2024-04-02T14:11:00.806 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ptp/ptp.ko 2024-04-02T14:11:00.806 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ptp/ptp_kvm.ko 2024-04-02T14:11:00.806 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc 2024-04-02T14:11:00.806 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-bq32k.ko 2024-04-02T14:11:00.806 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds1286.ko 2024-04-02T14:11:00.806 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds1307.ko 2024-04-02T14:11:00.806 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds1374.ko 2024-04-02T14:11:00.806 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds1511.ko 2024-04-02T14:11:00.806 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds1553.ko 2024-04-02T14:11:00.807 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds1672.ko 2024-04-02T14:11:00.807 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds1742.ko 2024-04-02T14:11:00.807 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds2404.ko 2024-04-02T14:11:00.807 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds3232.ko 2024-04-02T14:11:00.807 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-em3027.ko 2024-04-02T14:11:00.807 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-fm3130.ko 2024-04-02T14:11:00.807 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-isl12022.ko 2024-04-02T14:11:00.807 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-isl1208.ko 2024-04-02T14:11:00.807 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-m41t80.ko 2024-04-02T14:11:00.807 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-m48t35.ko 2024-04-02T14:11:00.807 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-m48t59.ko 2024-04-02T14:11:00.807 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-max6900.ko 2024-04-02T14:11:00.807 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-msm6242.ko 2024-04-02T14:11:00.807 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-pcf2127.ko 2024-04-02T14:11:00.807 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-pcf85063.ko 2024-04-02T14:11:00.807 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-pcf8523.ko 2024-04-02T14:11:00.808 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-pcf8563.ko 2024-04-02T14:11:00.808 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-pcf8583.ko 2024-04-02T14:11:00.808 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-rp5c01.ko 2024-04-02T14:11:00.808 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-rs5c372.ko 2024-04-02T14:11:00.808 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-rv3029c2.ko 2024-04-02T14:11:00.808 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-rx8025.ko 2024-04-02T14:11:00.808 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-rx8581.ko 2024-04-02T14:11:00.808 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-stk17ta8.ko 2024-04-02T14:11:00.808 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-x1205.ko 2024-04-02T14:11:00.808 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi 2024-04-02T14:11:00.808 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/3w-9xxx.ko 2024-04-02T14:11:00.808 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/3w-sas.ko 2024-04-02T14:11:00.808 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/3w-xxxx.ko 2024-04-02T14:11:00.808 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/BusLogic.ko 2024-04-02T14:11:00.809 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/a100u2w.ko 2024-04-02T14:11:00.809 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/aacraid 2024-04-02T14:11:00.809 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/aacraid/aacraid.ko 2024-04-02T14:11:00.809 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/advansys.ko 2024-04-02T14:11:00.809 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/aic7xxx 2024-04-02T14:11:00.809 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2024-04-02T14:11:00.809 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2024-04-02T14:11:00.809 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/am53c974.ko 2024-04-02T14:11:00.809 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/arcmsr 2024-04-02T14:11:00.809 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/arcmsr/arcmsr.ko 2024-04-02T14:11:00.809 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/atp870u.ko 2024-04-02T14:11:00.809 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/be2iscsi 2024-04-02T14:11:00.809 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2024-04-02T14:11:00.809 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/bfa 2024-04-02T14:11:00.810 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/bfa/bfa.ko 2024-04-02T14:11:00.810 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/bnx2fc 2024-04-02T14:11:00.810 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2024-04-02T14:11:00.810 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/bnx2i 2024-04-02T14:11:00.810 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/bnx2i/bnx2i.ko 2024-04-02T14:11:00.810 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/ch.ko 2024-04-02T14:11:00.810 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/csiostor 2024-04-02T14:11:00.810 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/csiostor/csiostor.ko 2024-04-02T14:11:00.810 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/cxgbi 2024-04-02T14:11:00.813 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/cxgbi/cxgb3i 2024-04-02T14:11:00.813 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2024-04-02T14:11:00.813 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/cxgbi/cxgb4i 2024-04-02T14:11:00.813 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2024-04-02T14:11:00.813 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2024-04-02T14:11:00.813 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/dc395x.ko 2024-04-02T14:11:00.813 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/device_handler 2024-04-02T14:11:00.813 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2024-04-02T14:11:00.814 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2024-04-02T14:11:00.814 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2024-04-02T14:11:00.814 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2024-04-02T14:11:00.814 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/dmx3191d.ko 2024-04-02T14:11:00.814 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/esas2r 2024-04-02T14:11:00.814 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/esas2r/esas2r.ko 2024-04-02T14:11:00.814 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/esp_scsi.ko 2024-04-02T14:11:00.814 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/fcoe 2024-04-02T14:11:00.814 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/fcoe/fcoe.ko 2024-04-02T14:11:00.814 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/fcoe/libfcoe.ko 2024-04-02T14:11:00.814 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/fnic 2024-04-02T14:11:00.814 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/fnic/fnic.ko 2024-04-02T14:11:00.814 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/hpsa.ko 2024-04-02T14:11:00.814 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/hptiop.ko 2024-04-02T14:11:00.815 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/hv_storvsc.ko 2024-04-02T14:11:00.815 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/initio.ko 2024-04-02T14:11:00.815 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/ipr.ko 2024-04-02T14:11:00.815 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/ips.ko 2024-04-02T14:11:00.815 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/isci 2024-04-02T14:11:00.815 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/isci/isci.ko 2024-04-02T14:11:00.815 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2024-04-02T14:11:00.815 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/iscsi_tcp.ko 2024-04-02T14:11:00.815 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/libfc 2024-04-02T14:11:00.815 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/libfc/libfc.ko 2024-04-02T14:11:00.815 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/libiscsi.ko 2024-04-02T14:11:00.815 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/libiscsi_tcp.ko 2024-04-02T14:11:00.815 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/libsas 2024-04-02T14:11:00.815 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/libsas/libsas.ko 2024-04-02T14:11:00.815 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/lpfc 2024-04-02T14:11:00.816 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/lpfc/lpfc.ko 2024-04-02T14:11:00.816 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/megaraid 2024-04-02T14:11:00.816 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/megaraid.ko 2024-04-02T14:11:00.816 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2024-04-02T14:11:00.816 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2024-04-02T14:11:00.816 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2024-04-02T14:11:00.816 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/mpt3sas 2024-04-02T14:11:00.816 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2024-04-02T14:11:00.816 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/mvsas 2024-04-02T14:11:00.816 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/mvsas/mvsas.ko 2024-04-02T14:11:00.816 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/mvumi.ko 2024-04-02T14:11:00.816 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/pm8001 2024-04-02T14:11:00.816 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/pm8001/pm80xx.ko 2024-04-02T14:11:00.816 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/pmcraid.ko 2024-04-02T14:11:00.817 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/qla1280.ko 2024-04-02T14:11:00.817 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/qla2xxx 2024-04-02T14:11:00.817 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2024-04-02T14:11:00.817 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2024-04-02T14:11:00.817 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/qla4xxx 2024-04-02T14:11:00.817 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2024-04-02T14:11:00.817 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/raid_class.ko 2024-04-02T14:11:00.817 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/scsi_debug.ko 2024-04-02T14:11:00.817 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/scsi_transport_fc.ko 2024-04-02T14:11:00.817 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/scsi_transport_iscsi.ko 2024-04-02T14:11:00.817 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/scsi_transport_sas.ko 2024-04-02T14:11:00.817 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/scsi_transport_spi.ko 2024-04-02T14:11:00.817 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/scsi_transport_srp.ko 2024-04-02T14:11:00.817 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/ses.ko 2024-04-02T14:11:00.818 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/st.ko 2024-04-02T14:11:00.818 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/stex.ko 2024-04-02T14:11:00.818 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/sym53c8xx_2 2024-04-02T14:11:00.818 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2024-04-02T14:11:00.818 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/virtio_scsi.ko 2024-04-02T14:11:00.818 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/vmw_pvscsi.ko 2024-04-02T14:11:00.818 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/wd719x.ko 2024-04-02T14:11:00.818 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ssb 2024-04-02T14:11:00.818 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ssb/ssb.ko 2024-04-02T14:11:00.818 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target 2024-04-02T14:11:00.818 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/iscsi 2024-04-02T14:11:00.818 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2024-04-02T14:11:00.818 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/loopback 2024-04-02T14:11:00.818 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/loopback/tcm_loop.ko 2024-04-02T14:11:00.819 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/target_core_file.ko 2024-04-02T14:11:00.819 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/target_core_iblock.ko 2024-04-02T14:11:00.819 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/target_core_mod.ko 2024-04-02T14:11:00.819 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/target_core_pscsi.ko 2024-04-02T14:11:00.819 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/target_core_user.ko 2024-04-02T14:11:00.819 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/tcm_fc 2024-04-02T14:11:00.819 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/tcm_fc/tcm_fc.ko 2024-04-02T14:11:00.819 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal 2024-04-02T14:11:00.819 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel 2024-04-02T14:11:00.819 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal 2024-04-02T14:11:00.819 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2024-04-02T14:11:00.819 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2024-04-02T14:11:00.819 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2024-04-02T14:11:00.819 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2024-04-02T14:11:00.819 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2024-04-02T14:11:00.820 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2024-04-02T14:11:00.820 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2024-04-02T14:11:00.820 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2024-04-02T14:11:00.820 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2024-04-02T14:11:00.820 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2024-04-02T14:11:00.820 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_power_floor.ko 2024-04-02T14:11:00.820 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2024-04-02T14:11:00.820 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2024-04-02T14:11:00.820 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_hint.ko 2024-04-02T14:11:00.820 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_req.ko 2024-04-02T14:11:00.820 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2024-04-02T14:11:00.820 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2024-04-02T14:11:00.820 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2024-04-02T14:11:00.820 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty 2024-04-02T14:11:00.820 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/ipwireless 2024-04-02T14:11:00.821 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/ipwireless/ipwireless.ko 2024-04-02T14:11:00.821 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/n_gsm.ko 2024-04-02T14:11:00.821 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/n_hdlc.ko 2024-04-02T14:11:00.821 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/nozomi.ko 2024-04-02T14:11:00.821 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/serial 2024-04-02T14:11:00.821 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/serial/8250 2024-04-02T14:11:00.821 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/serial/8250/serial_cs.ko 2024-04-02T14:11:00.821 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/serial/arc_uart.ko 2024-04-02T14:11:00.821 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/serial/jsm 2024-04-02T14:11:00.821 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/serial/jsm/jsm.ko 2024-04-02T14:11:00.821 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ufs 2024-04-02T14:11:00.821 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ufs/core 2024-04-02T14:11:00.821 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ufs/core/ufshcd-core.ko 2024-04-02T14:11:00.821 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ufs/host 2024-04-02T14:11:00.821 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ufs/host/ufshcd-pci.ko 2024-04-02T14:11:00.822 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/uio 2024-04-02T14:11:00.822 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/uio/uio.ko 2024-04-02T14:11:00.822 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/uio/uio_aec.ko 2024-04-02T14:11:00.822 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/uio/uio_cif.ko 2024-04-02T14:11:00.822 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/uio/uio_pci_generic.ko 2024-04-02T14:11:00.822 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/uio/uio_sercos3.ko 2024-04-02T14:11:00.822 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb 2024-04-02T14:11:00.822 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/class 2024-04-02T14:11:00.822 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/class/cdc-acm.ko 2024-04-02T14:11:00.822 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/class/cdc-wdm.ko 2024-04-02T14:11:00.822 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/class/usblp.ko 2024-04-02T14:11:00.822 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/class/usbtmc.ko 2024-04-02T14:11:00.822 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/host 2024-04-02T14:11:00.822 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/host/sl811-hcd.ko 2024-04-02T14:11:00.822 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/image 2024-04-02T14:11:00.823 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/image/mdc800.ko 2024-04-02T14:11:00.823 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/image/microtek.ko 2024-04-02T14:11:00.823 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc 2024-04-02T14:11:00.823 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/adutux.ko 2024-04-02T14:11:00.823 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/appledisplay.ko 2024-04-02T14:11:00.823 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/emi26.ko 2024-04-02T14:11:00.823 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/emi62.ko 2024-04-02T14:11:00.823 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/ezusb.ko 2024-04-02T14:11:00.823 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/idmouse.ko 2024-04-02T14:11:00.823 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/iowarrior.ko 2024-04-02T14:11:00.823 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/isight_firmware.ko 2024-04-02T14:11:00.823 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/ldusb.ko 2024-04-02T14:11:00.823 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/legousbtower.ko 2024-04-02T14:11:00.823 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/sisusbvga 2024-04-02T14:11:00.823 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2024-04-02T14:11:00.823 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/trancevibrator.ko 2024-04-02T14:11:00.824 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/usb3503.ko 2024-04-02T14:11:00.824 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/usblcd.ko 2024-04-02T14:11:00.824 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/usbsevseg.ko 2024-04-02T14:11:00.824 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/uss720.ko 2024-04-02T14:11:00.824 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/yurex.ko 2024-04-02T14:11:00.824 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/phy 2024-04-02T14:11:00.824 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/phy/phy-generic.ko 2024-04-02T14:11:00.824 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial 2024-04-02T14:11:00.824 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/aircable.ko 2024-04-02T14:11:00.824 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ark3116.ko 2024-04-02T14:11:00.824 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/belkin_sa.ko 2024-04-02T14:11:00.824 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ch341.ko 2024-04-02T14:11:00.824 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/cp210x.ko 2024-04-02T14:11:00.824 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/cyberjack.ko 2024-04-02T14:11:00.824 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/cypress_m8.ko 2024-04-02T14:11:00.825 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/digi_acceleport.ko 2024-04-02T14:11:00.825 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/empeg.ko 2024-04-02T14:11:00.825 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ftdi_sio.ko 2024-04-02T14:11:00.825 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/garmin_gps.ko 2024-04-02T14:11:00.825 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/io_edgeport.ko 2024-04-02T14:11:00.825 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/io_ti.ko 2024-04-02T14:11:00.825 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ipaq.ko 2024-04-02T14:11:00.825 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ipw.ko 2024-04-02T14:11:00.825 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ir-usb.ko 2024-04-02T14:11:00.825 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/iuu_phoenix.ko 2024-04-02T14:11:00.825 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/keyspan.ko 2024-04-02T14:11:00.825 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/keyspan_pda.ko 2024-04-02T14:11:00.825 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/kl5kusb105.ko 2024-04-02T14:11:00.825 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/kobil_sct.ko 2024-04-02T14:11:00.825 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/mct_u232.ko 2024-04-02T14:11:00.826 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/mos7720.ko 2024-04-02T14:11:00.826 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/mos7840.ko 2024-04-02T14:11:00.826 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/navman.ko 2024-04-02T14:11:00.826 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/omninet.ko 2024-04-02T14:11:00.826 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/opticon.ko 2024-04-02T14:11:00.826 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/option.ko 2024-04-02T14:11:00.826 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/oti6858.ko 2024-04-02T14:11:00.826 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/pl2303.ko 2024-04-02T14:11:00.826 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/qcaux.ko 2024-04-02T14:11:00.826 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/qcserial.ko 2024-04-02T14:11:00.826 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/quatech2.ko 2024-04-02T14:11:00.826 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/safe_serial.ko 2024-04-02T14:11:00.826 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/sierra.ko 2024-04-02T14:11:00.826 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/spcp8x5.ko 2024-04-02T14:11:00.826 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ssu100.ko 2024-04-02T14:11:00.827 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/symbolserial.ko 2024-04-02T14:11:00.827 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2024-04-02T14:11:00.827 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/usb-serial-simple.ko 2024-04-02T14:11:00.827 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/usb_debug.ko 2024-04-02T14:11:00.827 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/usb_wwan.ko 2024-04-02T14:11:00.827 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/visor.ko 2024-04-02T14:11:00.827 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/whiteheat.ko 2024-04-02T14:11:00.827 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/xsens_mt.ko 2024-04-02T14:11:00.827 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage 2024-04-02T14:11:00.827 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/uas.ko 2024-04-02T14:11:00.827 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-alauda.ko 2024-04-02T14:11:00.827 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-cypress.ko 2024-04-02T14:11:00.827 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-datafab.ko 2024-04-02T14:11:00.827 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-eneub6250.ko 2024-04-02T14:11:00.827 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-freecom.ko 2024-04-02T14:11:00.828 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-isd200.ko 2024-04-02T14:11:00.828 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-jumpshot.ko 2024-04-02T14:11:00.828 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-karma.ko 2024-04-02T14:11:00.828 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-onetouch.ko 2024-04-02T14:11:00.828 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-realtek.ko 2024-04-02T14:11:00.828 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-sddr09.ko 2024-04-02T14:11:00.828 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-sddr55.ko 2024-04-02T14:11:00.828 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-usbat.ko 2024-04-02T14:11:00.828 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/usb-storage.ko 2024-04-02T14:11:00.828 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/usbip 2024-04-02T14:11:00.828 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/usbip/usbip-core.ko 2024-04-02T14:11:00.828 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/usbip/usbip-host.ko 2024-04-02T14:11:00.828 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/usbip/vhci-hcd.ko 2024-04-02T14:11:00.828 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vfio 2024-04-02T14:11:00.828 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vfio/pci 2024-04-02T14:11:00.829 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vfio/pci/vfio-pci-core.ko 2024-04-02T14:11:00.829 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vfio/pci/vfio-pci.ko 2024-04-02T14:11:00.829 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vfio/vfio.ko 2024-04-02T14:11:00.829 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vfio/vfio_iommu_type1.ko 2024-04-02T14:11:00.829 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vhost 2024-04-02T14:11:00.829 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vhost/vhost.ko 2024-04-02T14:11:00.829 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vhost/vhost_iotlb.ko 2024-04-02T14:11:00.829 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vhost/vhost_net.ko 2024-04-02T14:11:00.829 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vhost/vhost_scsi.ko 2024-04-02T14:11:00.829 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video 2024-04-02T14:11:00.829 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video/backlight 2024-04-02T14:11:00.829 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video/backlight/apple_bl.ko 2024-04-02T14:11:00.829 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video/backlight/lcd.ko 2024-04-02T14:11:00.829 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video/backlight/platform_lcd.ko 2024-04-02T14:11:00.829 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/virtio 2024-04-02T14:11:00.830 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/virtio/virtio.ko 2024-04-02T14:11:00.830 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/virtio/virtio_balloon.ko 2024-04-02T14:11:00.830 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/virtio/virtio_mmio.ko 2024-04-02T14:11:00.830 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/virtio/virtio_pci.ko 2024-04-02T14:11:00.830 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2024-04-02T14:11:00.830 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2024-04-02T14:11:00.830 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/virtio/virtio_ring.ko 2024-04-02T14:11:00.830 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1 2024-04-02T14:11:00.830 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/masters 2024-04-02T14:11:00.830 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/masters/ds2482.ko 2024-04-02T14:11:00.830 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/masters/ds2490.ko 2024-04-02T14:11:00.830 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves 2024-04-02T14:11:00.830 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2406.ko 2024-04-02T14:11:00.830 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2408.ko 2024-04-02T14:11:00.830 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2413.ko 2024-04-02T14:11:00.831 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2423.ko 2024-04-02T14:11:00.831 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2431.ko 2024-04-02T14:11:00.831 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2433.ko 2024-04-02T14:11:00.831 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2780.ko 2024-04-02T14:11:00.831 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2781.ko 2024-04-02T14:11:00.831 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds28e04.ko 2024-04-02T14:11:00.831 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_smem.ko 2024-04-02T14:11:00.831 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_therm.ko 2024-04-02T14:11:00.831 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/wire.ko 2024-04-02T14:11:00.831 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog 2024-04-02T14:11:00.831 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/alim1535_wdt.ko 2024-04-02T14:11:00.831 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/alim7101_wdt.ko 2024-04-02T14:11:00.831 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/f71808e_wdt.ko 2024-04-02T14:11:00.831 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/hpwdt.ko 2024-04-02T14:11:00.831 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/i6300esb.ko 2024-04-02T14:11:00.832 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/iTCO_vendor_support.ko 2024-04-02T14:11:00.832 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/iTCO_wdt.ko 2024-04-02T14:11:00.832 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/ib700wdt.ko 2024-04-02T14:11:00.832 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/ibmasr.ko 2024-04-02T14:11:00.832 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/ie6xx_wdt.ko 2024-04-02T14:11:00.832 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/it8712f_wdt.ko 2024-04-02T14:11:00.832 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/it87_wdt.ko 2024-04-02T14:11:00.832 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/machzwd.ko 2024-04-02T14:11:00.832 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/nv_tco.ko 2024-04-02T14:11:00.832 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/pcwd_pci.ko 2024-04-02T14:11:00.832 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/pcwd_usb.ko 2024-04-02T14:11:00.832 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2024-04-02T14:11:00.832 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/sch311x_wdt.ko 2024-04-02T14:11:00.832 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/softdog.ko 2024-04-02T14:11:00.832 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/sp5100_tco.ko 2024-04-02T14:11:00.833 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/via_wdt.ko 2024-04-02T14:11:00.833 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/w83627hf_wdt.ko 2024-04-02T14:11:00.833 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/w83877f_wdt.ko 2024-04-02T14:11:00.833 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/w83977f_wdt.ko 2024-04-02T14:11:00.833 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/wdt_pci.ko 2024-04-02T14:11:00.833 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/xen_wdt.ko 2024-04-02T14:11:00.833 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen 2024-04-02T14:11:00.833 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xen-acpi-processor.ko 2024-04-02T14:11:00.833 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xen-evtchn.ko 2024-04-02T14:11:00.833 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xen-gntalloc.ko 2024-04-02T14:11:00.833 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xen-gntdev.ko 2024-04-02T14:11:00.833 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xen-pciback 2024-04-02T14:11:00.833 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2024-04-02T14:11:00.833 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xen-privcmd.ko 2024-04-02T14:11:00.833 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xenfs 2024-04-02T14:11:00.834 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xenfs/xenfs.ko 2024-04-02T14:11:00.834 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs 2024-04-02T14:11:00.834 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/9p 2024-04-02T14:11:00.834 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/9p/9p.ko 2024-04-02T14:11:00.834 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/affs 2024-04-02T14:11:00.834 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/affs/affs.ko 2024-04-02T14:11:00.834 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/befs 2024-04-02T14:11:00.834 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/befs/befs.ko 2024-04-02T14:11:00.834 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/binfmt_misc.ko 2024-04-02T14:11:00.834 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/btrfs 2024-04-02T14:11:00.834 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/btrfs/btrfs.ko 2024-04-02T14:11:00.834 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/cachefiles 2024-04-02T14:11:00.834 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/cachefiles/cachefiles.ko 2024-04-02T14:11:00.834 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ceph 2024-04-02T14:11:00.835 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ceph/ceph.ko 2024-04-02T14:11:00.835 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/coda 2024-04-02T14:11:00.835 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/coda/coda.ko 2024-04-02T14:11:00.835 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/cramfs 2024-04-02T14:11:00.835 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/cramfs/cramfs.ko 2024-04-02T14:11:00.835 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/dlm 2024-04-02T14:11:00.835 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/dlm/dlm.ko 2024-04-02T14:11:00.835 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ecryptfs 2024-04-02T14:11:00.835 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ecryptfs/ecryptfs.ko 2024-04-02T14:11:00.835 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/f2fs 2024-04-02T14:11:00.835 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/f2fs/f2fs.ko 2024-04-02T14:11:00.835 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/fat 2024-04-02T14:11:00.835 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/fat/fat.ko 2024-04-02T14:11:00.835 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/fat/msdos.ko 2024-04-02T14:11:00.836 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/fat/vfat.ko 2024-04-02T14:11:00.836 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/fuse 2024-04-02T14:11:00.836 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/fuse/cuse.ko 2024-04-02T14:11:00.836 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/fuse/fuse.ko 2024-04-02T14:11:00.836 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/gfs2 2024-04-02T14:11:00.836 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/gfs2/gfs2.ko 2024-04-02T14:11:00.836 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/hfs 2024-04-02T14:11:00.836 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/hfs/hfs.ko 2024-04-02T14:11:00.836 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/hfsplus 2024-04-02T14:11:00.836 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/hfsplus/hfsplus.ko 2024-04-02T14:11:00.836 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/isofs 2024-04-02T14:11:00.836 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/isofs/isofs.ko 2024-04-02T14:11:00.836 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/jfs 2024-04-02T14:11:00.836 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/jfs/jfs.ko 2024-04-02T14:11:00.836 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/lockd 2024-04-02T14:11:00.837 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/lockd/lockd.ko 2024-04-02T14:11:00.837 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/minix 2024-04-02T14:11:00.837 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/minix/minix.ko 2024-04-02T14:11:00.837 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/netfs 2024-04-02T14:11:00.837 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/netfs/netfs.ko 2024-04-02T14:11:00.837 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs 2024-04-02T14:11:00.837 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/blocklayout 2024-04-02T14:11:00.837 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2024-04-02T14:11:00.837 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/filelayout 2024-04-02T14:11:00.837 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2024-04-02T14:11:00.837 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/flexfilelayout 2024-04-02T14:11:00.837 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2024-04-02T14:11:00.837 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/nfs.ko 2024-04-02T14:11:00.837 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/nfsv3.ko 2024-04-02T14:11:00.837 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/nfsv4.ko 2024-04-02T14:11:00.837 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs_common 2024-04-02T14:11:00.838 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs_common/grace.ko 2024-04-02T14:11:00.838 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs_common/nfs_acl.ko 2024-04-02T14:11:00.838 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfsd 2024-04-02T14:11:00.838 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfsd/nfsd.ko 2024-04-02T14:11:00.838 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nilfs2 2024-04-02T14:11:00.838 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nilfs2/nilfs2.ko 2024-04-02T14:11:00.838 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls 2024-04-02T14:11:00.838 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-celtic.ko 2024-04-02T14:11:00.838 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-centeuro.ko 2024-04-02T14:11:00.838 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-croatian.ko 2024-04-02T14:11:00.838 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-cyrillic.ko 2024-04-02T14:11:00.838 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-gaelic.ko 2024-04-02T14:11:00.838 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-greek.ko 2024-04-02T14:11:00.838 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-iceland.ko 2024-04-02T14:11:00.839 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-inuit.ko 2024-04-02T14:11:00.839 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-roman.ko 2024-04-02T14:11:00.839 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-romanian.ko 2024-04-02T14:11:00.839 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-turkish.ko 2024-04-02T14:11:00.839 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp1250.ko 2024-04-02T14:11:00.839 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp1251.ko 2024-04-02T14:11:00.839 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp1255.ko 2024-04-02T14:11:00.839 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp737.ko 2024-04-02T14:11:00.839 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp775.ko 2024-04-02T14:11:00.839 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp850.ko 2024-04-02T14:11:00.839 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp852.ko 2024-04-02T14:11:00.839 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp855.ko 2024-04-02T14:11:00.839 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp857.ko 2024-04-02T14:11:00.839 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp860.ko 2024-04-02T14:11:00.840 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp861.ko 2024-04-02T14:11:00.840 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp862.ko 2024-04-02T14:11:00.840 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp863.ko 2024-04-02T14:11:00.840 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp864.ko 2024-04-02T14:11:00.840 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp865.ko 2024-04-02T14:11:00.840 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp866.ko 2024-04-02T14:11:00.840 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp869.ko 2024-04-02T14:11:00.840 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp874.ko 2024-04-02T14:11:00.840 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp932.ko 2024-04-02T14:11:00.840 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp936.ko 2024-04-02T14:11:00.840 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp949.ko 2024-04-02T14:11:00.840 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp950.ko 2024-04-02T14:11:00.840 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_euc-jp.ko 2024-04-02T14:11:00.840 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-1.ko 2024-04-02T14:11:00.840 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-13.ko 2024-04-02T14:11:00.840 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-14.ko 2024-04-02T14:11:00.841 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-15.ko 2024-04-02T14:11:00.841 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-2.ko 2024-04-02T14:11:00.841 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-3.ko 2024-04-02T14:11:00.841 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-4.ko 2024-04-02T14:11:00.841 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-5.ko 2024-04-02T14:11:00.841 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-6.ko 2024-04-02T14:11:00.841 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-7.ko 2024-04-02T14:11:00.841 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-9.ko 2024-04-02T14:11:00.841 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_koi8-r.ko 2024-04-02T14:11:00.841 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_koi8-ru.ko 2024-04-02T14:11:00.841 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_koi8-u.ko 2024-04-02T14:11:00.841 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_ucs2_utils.ko 2024-04-02T14:11:00.841 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_utf8.ko 2024-04-02T14:11:00.841 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2 2024-04-02T14:11:00.841 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/cluster 2024-04-02T14:11:00.842 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2024-04-02T14:11:00.842 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/dlm 2024-04-02T14:11:00.842 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2024-04-02T14:11:00.842 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/dlmfs 2024-04-02T14:11:00.842 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2024-04-02T14:11:00.842 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/ocfs2.ko 2024-04-02T14:11:00.842 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2024-04-02T14:11:00.842 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/ocfs2_stack_user.ko 2024-04-02T14:11:00.842 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/ocfs2_stackglue.ko 2024-04-02T14:11:00.842 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/overlayfs 2024-04-02T14:11:00.842 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/overlayfs/overlay.ko 2024-04-02T14:11:00.842 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/pstore 2024-04-02T14:11:00.842 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/pstore/ramoops.ko 2024-04-02T14:11:00.842 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/reiserfs 2024-04-02T14:11:00.843 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/reiserfs/reiserfs.ko 2024-04-02T14:11:00.843 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/romfs 2024-04-02T14:11:00.843 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/romfs/romfs.ko 2024-04-02T14:11:00.843 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/smb 2024-04-02T14:11:00.843 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/smb/client 2024-04-02T14:11:00.843 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/smb/client/cifs.ko 2024-04-02T14:11:00.843 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/smb/common 2024-04-02T14:11:00.843 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/smb/common/cifs_arc4.ko 2024-04-02T14:11:00.843 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/smb/common/cifs_md4.ko 2024-04-02T14:11:00.843 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/squashfs 2024-04-02T14:11:00.843 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/squashfs/squashfs.ko 2024-04-02T14:11:00.843 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/sysv 2024-04-02T14:11:00.843 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/sysv/sysv.ko 2024-04-02T14:11:00.843 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/udf 2024-04-02T14:11:00.843 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/udf/udf.ko 2024-04-02T14:11:00.844 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ufs 2024-04-02T14:11:00.844 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ufs/ufs.ko 2024-04-02T14:11:00.844 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/xfs 2024-04-02T14:11:00.844 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/xfs/xfs.ko 2024-04-02T14:11:00.844 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/kernel 2024-04-02T14:11:00.844 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/kernel/rcu 2024-04-02T14:11:00.844 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/kernel/rcu/rcutorture.ko 2024-04-02T14:11:00.844 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/kernel/torture.ko 2024-04-02T14:11:00.844 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/kernel/trace 2024-04-02T14:11:00.844 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/kernel/trace/ring_buffer_benchmark.ko 2024-04-02T14:11:00.844 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib 2024-04-02T14:11:00.844 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/asn1_encoder.ko 2024-04-02T14:11:00.844 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/crc-itu-t.ko 2024-04-02T14:11:00.844 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/crc8.ko 2024-04-02T14:11:00.844 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/crypto 2024-04-02T14:11:00.844 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/crypto/libarc4.ko 2024-04-02T14:11:00.845 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/crypto/libdes.ko 2024-04-02T14:11:00.845 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/libcrc32c.ko 2024-04-02T14:11:00.845 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/lru_cache.ko 2024-04-02T14:11:00.845 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/lz4 2024-04-02T14:11:00.845 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/lz4/lz4_compress.ko 2024-04-02T14:11:00.845 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/lz4/lz4hc_compress.ko 2024-04-02T14:11:00.845 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/math 2024-04-02T14:11:00.845 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/math/cordic.ko 2024-04-02T14:11:00.845 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/raid6 2024-04-02T14:11:00.845 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/raid6/raid6_pq.ko 2024-04-02T14:11:00.845 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/reed_solomon 2024-04-02T14:11:00.845 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/reed_solomon/reed_solomon.ko 2024-04-02T14:11:00.845 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/ts_bm.ko 2024-04-02T14:11:00.845 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/ts_fsm.ko 2024-04-02T14:11:00.845 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/ts_kmp.ko 2024-04-02T14:11:00.845 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/zstd 2024-04-02T14:11:00.845 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/zstd/zstd_compress.ko 2024-04-02T14:11:00.846 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/mm 2024-04-02T14:11:00.846 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/mm/hwpoison-inject.ko 2024-04-02T14:11:00.846 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net 2024-04-02T14:11:00.846 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan 2024-04-02T14:11:00.846 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/6lowpan.ko 2024-04-02T14:11:00.846 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/nhc_dest.ko 2024-04-02T14:11:00.846 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/nhc_fragment.ko 2024-04-02T14:11:00.846 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/nhc_hop.ko 2024-04-02T14:11:00.846 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/nhc_ipv6.ko 2024-04-02T14:11:00.846 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/nhc_mobility.ko 2024-04-02T14:11:00.846 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/nhc_routing.ko 2024-04-02T14:11:00.846 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/nhc_udp.ko 2024-04-02T14:11:00.846 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/802 2024-04-02T14:11:00.846 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/802/garp.ko 2024-04-02T14:11:00.847 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/802/mrp.ko 2024-04-02T14:11:00.847 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/802/p8022.ko 2024-04-02T14:11:00.847 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/802/psnap.ko 2024-04-02T14:11:00.847 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/802/stp.ko 2024-04-02T14:11:00.847 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/8021q 2024-04-02T14:11:00.847 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/8021q/8021q.ko 2024-04-02T14:11:00.847 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/9p 2024-04-02T14:11:00.847 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/9p/9pnet.ko 2024-04-02T14:11:00.847 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/9p/9pnet_fd.ko 2024-04-02T14:11:00.847 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/9p/9pnet_rdma.ko 2024-04-02T14:11:00.847 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/9p/9pnet_virtio.ko 2024-04-02T14:11:00.847 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/batman-adv 2024-04-02T14:11:00.847 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/batman-adv/batman-adv.ko 2024-04-02T14:11:00.847 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge 2024-04-02T14:11:00.847 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/br_netfilter.ko 2024-04-02T14:11:00.848 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/bridge.ko 2024-04-02T14:11:00.848 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter 2024-04-02T14:11:00.848 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_802_3.ko 2024-04-02T14:11:00.848 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_among.ko 2024-04-02T14:11:00.848 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_arp.ko 2024-04-02T14:11:00.848 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_arpreply.ko 2024-04-02T14:11:00.848 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_dnat.ko 2024-04-02T14:11:00.848 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_ip.ko 2024-04-02T14:11:00.848 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_ip6.ko 2024-04-02T14:11:00.848 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_limit.ko 2024-04-02T14:11:00.848 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_log.ko 2024-04-02T14:11:00.848 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_mark.ko 2024-04-02T14:11:00.848 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_mark_m.ko 2024-04-02T14:11:00.848 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_nflog.ko 2024-04-02T14:11:00.848 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_pkttype.ko 2024-04-02T14:11:00.848 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_redirect.ko 2024-04-02T14:11:00.848 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_snat.ko 2024-04-02T14:11:00.849 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_stp.ko 2024-04-02T14:11:00.849 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_vlan.ko 2024-04-02T14:11:00.849 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebtable_broute.ko 2024-04-02T14:11:00.849 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebtable_filter.ko 2024-04-02T14:11:00.849 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebtable_nat.ko 2024-04-02T14:11:00.849 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebtables.ko 2024-04-02T14:11:00.849 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2024-04-02T14:11:00.849 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2024-04-02T14:11:00.849 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2024-04-02T14:11:00.849 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ceph 2024-04-02T14:11:00.849 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ceph/libceph.ko 2024-04-02T14:11:00.849 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/core 2024-04-02T14:11:00.849 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/core/failover.ko 2024-04-02T14:11:00.849 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/core/pktgen.ko 2024-04-02T14:11:00.850 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dccp 2024-04-02T14:11:00.850 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dccp/dccp.ko 2024-04-02T14:11:00.850 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dccp/dccp_diag.ko 2024-04-02T14:11:00.850 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dccp/dccp_ipv4.ko 2024-04-02T14:11:00.850 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dccp/dccp_ipv6.ko 2024-04-02T14:11:00.850 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dns_resolver 2024-04-02T14:11:00.850 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dns_resolver/dns_resolver.ko 2024-04-02T14:11:00.850 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dsa 2024-04-02T14:11:00.850 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dsa/dsa_core.ko 2024-04-02T14:11:00.850 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dsa/tag_brcm.ko 2024-04-02T14:11:00.850 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dsa/tag_dsa.ko 2024-04-02T14:11:00.850 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dsa/tag_none.ko 2024-04-02T14:11:00.850 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dsa/tag_trailer.ko 2024-04-02T14:11:00.850 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ieee802154 2024-04-02T14:11:00.850 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ieee802154/6lowpan 2024-04-02T14:11:00.851 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2024-04-02T14:11:00.851 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ieee802154/ieee802154.ko 2024-04-02T14:11:00.851 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ieee802154/ieee802154_socket.ko 2024-04-02T14:11:00.851 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4 2024-04-02T14:11:00.851 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/ah4.ko 2024-04-02T14:11:00.851 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/esp4.ko 2024-04-02T14:11:00.851 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/fou.ko 2024-04-02T14:11:00.851 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/gre.ko 2024-04-02T14:11:00.851 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/inet_diag.ko 2024-04-02T14:11:00.851 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/ip_gre.ko 2024-04-02T14:11:00.851 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/ip_tunnel.ko 2024-04-02T14:11:00.851 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/ip_vti.ko 2024-04-02T14:11:00.851 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/ipcomp.ko 2024-04-02T14:11:00.851 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/ipip.ko 2024-04-02T14:11:00.852 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter 2024-04-02T14:11:00.852 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/arp_tables.ko 2024-04-02T14:11:00.852 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/arpt_mangle.ko 2024-04-02T14:11:00.852 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/arptable_filter.ko 2024-04-02T14:11:00.852 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/ipt_ECN.ko 2024-04-02T14:11:00.852 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2024-04-02T14:11:00.852 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/ipt_ah.ko 2024-04-02T14:11:00.852 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2024-04-02T14:11:00.852 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/iptable_mangle.ko 2024-04-02T14:11:00.852 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/iptable_nat.ko 2024-04-02T14:11:00.852 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/iptable_raw.ko 2024-04-02T14:11:00.852 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/iptable_security.ko 2024-04-02T14:11:00.852 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2024-04-02T14:11:00.852 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2024-04-02T14:11:00.853 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2024-04-02T14:11:00.853 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2024-04-02T14:11:00.853 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2024-04-02T14:11:00.853 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2024-04-02T14:11:00.853 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2024-04-02T14:11:00.853 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2024-04-02T14:11:00.853 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2024-04-02T14:11:00.853 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2024-04-02T14:11:00.853 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_bic.ko 2024-04-02T14:11:00.853 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_dctcp.ko 2024-04-02T14:11:00.853 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_diag.ko 2024-04-02T14:11:00.853 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_highspeed.ko 2024-04-02T14:11:00.853 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_htcp.ko 2024-04-02T14:11:00.853 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_hybla.ko 2024-04-02T14:11:00.854 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_illinois.ko 2024-04-02T14:11:00.854 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_lp.ko 2024-04-02T14:11:00.854 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_scalable.ko 2024-04-02T14:11:00.854 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_vegas.ko 2024-04-02T14:11:00.854 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_veno.ko 2024-04-02T14:11:00.854 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_westwood.ko 2024-04-02T14:11:00.854 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_yeah.ko 2024-04-02T14:11:00.854 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tunnel4.ko 2024-04-02T14:11:00.854 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/udp_diag.ko 2024-04-02T14:11:00.854 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/udp_tunnel.ko 2024-04-02T14:11:00.854 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/xfrm4_tunnel.ko 2024-04-02T14:11:00.854 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6 2024-04-02T14:11:00.854 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/ah6.ko 2024-04-02T14:11:00.854 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/esp6.ko 2024-04-02T14:11:00.854 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/fou6.ko 2024-04-02T14:11:00.855 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/ip6_tunnel.ko 2024-04-02T14:11:00.855 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/ip6_udp_tunnel.ko 2024-04-02T14:11:00.855 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/ip6_vti.ko 2024-04-02T14:11:00.855 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/ipcomp6.ko 2024-04-02T14:11:00.855 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter 2024-04-02T14:11:00.855 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6_tables.ko 2024-04-02T14:11:00.855 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2024-04-02T14:11:00.855 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2024-04-02T14:11:00.855 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_ah.ko 2024-04-02T14:11:00.855 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2024-04-02T14:11:00.855 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_frag.ko 2024-04-02T14:11:00.855 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2024-04-02T14:11:00.855 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2024-04-02T14:11:00.855 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_mh.ko 2024-04-02T14:11:00.855 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2024-04-02T14:11:00.856 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_rt.ko 2024-04-02T14:11:00.856 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6table_filter.ko 2024-04-02T14:11:00.856 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2024-04-02T14:11:00.856 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6table_nat.ko 2024-04-02T14:11:00.856 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6table_raw.ko 2024-04-02T14:11:00.856 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6table_security.ko 2024-04-02T14:11:00.856 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2024-04-02T14:11:00.856 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2024-04-02T14:11:00.856 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2024-04-02T14:11:00.856 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2024-04-02T14:11:00.856 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2024-04-02T14:11:00.856 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2024-04-02T14:11:00.856 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2024-04-02T14:11:00.856 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2024-04-02T14:11:00.856 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/sit.ko 2024-04-02T14:11:00.857 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/tunnel6.ko 2024-04-02T14:11:00.857 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/xfrm6_tunnel.ko 2024-04-02T14:11:00.857 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/key 2024-04-02T14:11:00.857 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/key/af_key.ko 2024-04-02T14:11:00.857 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp 2024-04-02T14:11:00.857 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp/l2tp_core.ko 2024-04-02T14:11:00.857 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp/l2tp_debugfs.ko 2024-04-02T14:11:00.857 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp/l2tp_eth.ko 2024-04-02T14:11:00.857 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp/l2tp_ip.ko 2024-04-02T14:11:00.857 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp/l2tp_ip6.ko 2024-04-02T14:11:00.857 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp/l2tp_netlink.ko 2024-04-02T14:11:00.857 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp/l2tp_ppp.ko 2024-04-02T14:11:00.857 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/llc 2024-04-02T14:11:00.857 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/llc/llc.ko 2024-04-02T14:11:00.857 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/mac802154 2024-04-02T14:11:00.858 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/mac802154/mac802154.ko 2024-04-02T14:11:00.858 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/mpls 2024-04-02T14:11:00.858 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/mpls/mpls_gso.ko 2024-04-02T14:11:00.858 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter 2024-04-02T14:11:00.858 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset 2024-04-02T14:11:00.858 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set.ko 2024-04-02T14:11:00.858 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2024-04-02T14:11:00.858 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2024-04-02T14:11:00.858 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2024-04-02T14:11:00.858 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2024-04-02T14:11:00.858 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2024-04-02T14:11:00.858 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2024-04-02T14:11:00.858 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2024-04-02T14:11:00.858 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2024-04-02T14:11:00.858 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2024-04-02T14:11:00.858 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2024-04-02T14:11:00.859 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2024-04-02T14:11:00.859 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2024-04-02T14:11:00.859 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2024-04-02T14:11:00.859 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2024-04-02T14:11:00.859 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_list_set.ko 2024-04-02T14:11:00.859 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs 2024-04-02T14:11:00.859 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs.ko 2024-04-02T14:11:00.859 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2024-04-02T14:11:00.859 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2024-04-02T14:11:00.859 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2024-04-02T14:11:00.859 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2024-04-02T14:11:00.859 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2024-04-02T14:11:00.859 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2024-04-02T14:11:00.859 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2024-04-02T14:11:00.859 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2024-04-02T14:11:00.860 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2024-04-02T14:11:00.860 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2024-04-02T14:11:00.860 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2024-04-02T14:11:00.860 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2024-04-02T14:11:00.860 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2024-04-02T14:11:00.860 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conncount.ko 2024-04-02T14:11:00.860 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack.ko 2024-04-02T14:11:00.860 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_amanda.ko 2024-04-02T14:11:00.860 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_broadcast.ko 2024-04-02T14:11:00.860 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_ftp.ko 2024-04-02T14:11:00.860 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_h323.ko 2024-04-02T14:11:00.860 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_irc.ko 2024-04-02T14:11:00.860 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2024-04-02T14:11:00.860 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_netlink.ko 2024-04-02T14:11:00.860 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_pptp.ko 2024-04-02T14:11:00.860 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_sane.ko 2024-04-02T14:11:00.860 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_sip.ko 2024-04-02T14:11:00.861 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_snmp.ko 2024-04-02T14:11:00.861 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_tftp.ko 2024-04-02T14:11:00.861 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_dup_netdev.ko 2024-04-02T14:11:00.861 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_flow_table.ko 2024-04-02T14:11:00.861 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_flow_table_inet.ko 2024-04-02T14:11:00.861 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_log_syslog.ko 2024-04-02T14:11:00.861 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_nat.ko 2024-04-02T14:11:00.861 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_nat_amanda.ko 2024-04-02T14:11:00.861 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_nat_ftp.ko 2024-04-02T14:11:00.861 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_nat_irc.ko 2024-04-02T14:11:00.861 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_nat_sip.ko 2024-04-02T14:11:00.861 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_nat_tftp.ko 2024-04-02T14:11:00.861 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_synproxy_core.ko 2024-04-02T14:11:00.861 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_tables.ko 2024-04-02T14:11:00.861 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nfnetlink.ko 2024-04-02T14:11:00.861 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nfnetlink_acct.ko 2024-04-02T14:11:00.861 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nfnetlink_log.ko 2024-04-02T14:11:00.862 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nfnetlink_osf.ko 2024-04-02T14:11:00.862 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nfnetlink_queue.ko 2024-04-02T14:11:00.862 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_chain_nat.ko 2024-04-02T14:11:00.862 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_compat.ko 2024-04-02T14:11:00.862 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_ct.ko 2024-04-02T14:11:00.862 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_dup_netdev.ko 2024-04-02T14:11:00.862 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_fib.ko 2024-04-02T14:11:00.862 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_fwd_netdev.ko 2024-04-02T14:11:00.862 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_hash.ko 2024-04-02T14:11:00.862 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_limit.ko 2024-04-02T14:11:00.862 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_log.ko 2024-04-02T14:11:00.862 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_masq.ko 2024-04-02T14:11:00.862 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_nat.ko 2024-04-02T14:11:00.862 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_numgen.ko 2024-04-02T14:11:00.863 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_queue.ko 2024-04-02T14:11:00.863 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_quota.ko 2024-04-02T14:11:00.863 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_redir.ko 2024-04-02T14:11:00.863 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_reject.ko 2024-04-02T14:11:00.863 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_reject_inet.ko 2024-04-02T14:11:00.863 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_socket.ko 2024-04-02T14:11:00.863 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_tunnel.ko 2024-04-02T14:11:00.863 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_AUDIT.ko 2024-04-02T14:11:00.863 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_CHECKSUM.ko 2024-04-02T14:11:00.863 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_CLASSIFY.ko 2024-04-02T14:11:00.863 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_CONNSECMARK.ko 2024-04-02T14:11:00.863 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_CT.ko 2024-04-02T14:11:00.863 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_DSCP.ko 2024-04-02T14:11:00.863 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_HL.ko 2024-04-02T14:11:00.863 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_HMARK.ko 2024-04-02T14:11:00.864 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_IDLETIMER.ko 2024-04-02T14:11:00.864 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_LED.ko 2024-04-02T14:11:00.864 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_LOG.ko 2024-04-02T14:11:00.864 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_MASQUERADE.ko 2024-04-02T14:11:00.864 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_NETMAP.ko 2024-04-02T14:11:00.864 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_NFLOG.ko 2024-04-02T14:11:00.864 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_NFQUEUE.ko 2024-04-02T14:11:00.864 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_RATEEST.ko 2024-04-02T14:11:00.864 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_REDIRECT.ko 2024-04-02T14:11:00.864 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_SECMARK.ko 2024-04-02T14:11:00.864 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_TCPMSS.ko 2024-04-02T14:11:00.864 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2024-04-02T14:11:00.864 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_TEE.ko 2024-04-02T14:11:00.864 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_TPROXY.ko 2024-04-02T14:11:00.864 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_TRACE.ko 2024-04-02T14:11:00.865 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_addrtype.ko 2024-04-02T14:11:00.865 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_bpf.ko 2024-04-02T14:11:00.865 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_cgroup.ko 2024-04-02T14:11:00.865 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_cluster.ko 2024-04-02T14:11:00.865 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_comment.ko 2024-04-02T14:11:00.865 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_connbytes.ko 2024-04-02T14:11:00.865 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_connlabel.ko 2024-04-02T14:11:00.865 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_connlimit.ko 2024-04-02T14:11:00.865 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_connmark.ko 2024-04-02T14:11:00.865 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_conntrack.ko 2024-04-02T14:11:00.865 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_cpu.ko 2024-04-02T14:11:00.865 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_dccp.ko 2024-04-02T14:11:00.865 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_devgroup.ko 2024-04-02T14:11:00.865 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_dscp.ko 2024-04-02T14:11:00.865 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_ecn.ko 2024-04-02T14:11:00.865 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_esp.ko 2024-04-02T14:11:00.865 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_hashlimit.ko 2024-04-02T14:11:00.866 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_helper.ko 2024-04-02T14:11:00.866 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_hl.ko 2024-04-02T14:11:00.866 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_ipcomp.ko 2024-04-02T14:11:00.866 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_iprange.ko 2024-04-02T14:11:00.866 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_ipvs.ko 2024-04-02T14:11:00.866 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_l2tp.ko 2024-04-02T14:11:00.866 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_length.ko 2024-04-02T14:11:00.866 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_limit.ko 2024-04-02T14:11:00.866 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_mac.ko 2024-04-02T14:11:00.866 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_mark.ko 2024-04-02T14:11:00.866 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_multiport.ko 2024-04-02T14:11:00.866 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_nat.ko 2024-04-02T14:11:00.866 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_nfacct.ko 2024-04-02T14:11:00.866 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_osf.ko 2024-04-02T14:11:00.866 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_owner.ko 2024-04-02T14:11:00.867 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_physdev.ko 2024-04-02T14:11:00.867 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_pkttype.ko 2024-04-02T14:11:00.867 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_policy.ko 2024-04-02T14:11:00.867 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_quota.ko 2024-04-02T14:11:00.867 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_rateest.ko 2024-04-02T14:11:00.867 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_realm.ko 2024-04-02T14:11:00.867 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_recent.ko 2024-04-02T14:11:00.867 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_sctp.ko 2024-04-02T14:11:00.867 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_set.ko 2024-04-02T14:11:00.867 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_socket.ko 2024-04-02T14:11:00.867 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_state.ko 2024-04-02T14:11:00.867 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_statistic.ko 2024-04-02T14:11:00.867 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_string.ko 2024-04-02T14:11:00.867 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_tcpmss.ko 2024-04-02T14:11:00.867 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_time.ko 2024-04-02T14:11:00.868 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_u32.ko 2024-04-02T14:11:00.868 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netlink 2024-04-02T14:11:00.868 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netlink/netlink_diag.ko 2024-04-02T14:11:00.868 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/nsh 2024-04-02T14:11:00.868 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/nsh/nsh.ko 2024-04-02T14:11:00.868 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/openvswitch 2024-04-02T14:11:00.868 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/openvswitch/openvswitch.ko 2024-04-02T14:11:00.868 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/openvswitch/vport-geneve.ko 2024-04-02T14:11:00.868 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/openvswitch/vport-gre.ko 2024-04-02T14:11:00.868 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/openvswitch/vport-vxlan.ko 2024-04-02T14:11:00.868 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/packet 2024-04-02T14:11:00.868 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/packet/af_packet_diag.ko 2024-04-02T14:11:00.868 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/rds 2024-04-02T14:11:00.868 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/rds/rds.ko 2024-04-02T14:11:00.868 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/rds/rds_rdma.ko 2024-04-02T14:11:00.868 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/rds/rds_tcp.ko 2024-04-02T14:11:00.869 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched 2024-04-02T14:11:00.869 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_csum.ko 2024-04-02T14:11:00.869 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_gact.ko 2024-04-02T14:11:00.869 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_mirred.ko 2024-04-02T14:11:00.869 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_nat.ko 2024-04-02T14:11:00.869 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_pedit.ko 2024-04-02T14:11:00.869 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_police.ko 2024-04-02T14:11:00.869 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_simple.ko 2024-04-02T14:11:00.869 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_skbedit.ko 2024-04-02T14:11:00.869 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_vlan.ko 2024-04-02T14:11:00.869 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/cls_basic.ko 2024-04-02T14:11:00.869 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/cls_bpf.ko 2024-04-02T14:11:00.869 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/cls_flow.ko 2024-04-02T14:11:00.869 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/cls_fw.ko 2024-04-02T14:11:00.869 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/cls_route.ko 2024-04-02T14:11:00.869 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/cls_u32.ko 2024-04-02T14:11:00.870 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/em_cmp.ko 2024-04-02T14:11:00.870 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/em_ipset.ko 2024-04-02T14:11:00.870 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/em_meta.ko 2024-04-02T14:11:00.870 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/em_nbyte.ko 2024-04-02T14:11:00.870 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/em_text.ko 2024-04-02T14:11:00.870 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/em_u32.ko 2024-04-02T14:11:00.870 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_choke.ko 2024-04-02T14:11:00.870 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_codel.ko 2024-04-02T14:11:00.870 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_drr.ko 2024-04-02T14:11:00.870 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_fq.ko 2024-04-02T14:11:00.870 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_gred.ko 2024-04-02T14:11:00.870 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_hfsc.ko 2024-04-02T14:11:00.870 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_hhf.ko 2024-04-02T14:11:00.870 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_htb.ko 2024-04-02T14:11:00.871 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_ingress.ko 2024-04-02T14:11:00.871 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_mqprio.ko 2024-04-02T14:11:00.871 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_mqprio_lib.ko 2024-04-02T14:11:00.871 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_multiq.ko 2024-04-02T14:11:00.871 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_netem.ko 2024-04-02T14:11:00.871 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_pie.ko 2024-04-02T14:11:00.871 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_plug.ko 2024-04-02T14:11:00.871 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_prio.ko 2024-04-02T14:11:00.871 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_qfq.ko 2024-04-02T14:11:00.871 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_red.ko 2024-04-02T14:11:00.871 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_sfb.ko 2024-04-02T14:11:00.871 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_sfq.ko 2024-04-02T14:11:00.871 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_tbf.ko 2024-04-02T14:11:00.871 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_teql.ko 2024-04-02T14:11:00.872 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sctp 2024-04-02T14:11:00.872 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sctp/sctp.ko 2024-04-02T14:11:00.872 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sctp/sctp_diag.ko 2024-04-02T14:11:00.872 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sunrpc 2024-04-02T14:11:00.872 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sunrpc/auth_gss 2024-04-02T14:11:00.872 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2024-04-02T14:11:00.872 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2024-04-02T14:11:00.872 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sunrpc/sunrpc.ko 2024-04-02T14:11:00.872 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sunrpc/xprtrdma 2024-04-02T14:11:00.872 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2024-04-02T14:11:00.872 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/tipc 2024-04-02T14:11:00.872 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/tipc/diag.ko 2024-04-02T14:11:00.872 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/tipc/tipc.ko 2024-04-02T14:11:00.872 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/unix 2024-04-02T14:11:00.872 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/unix/unix_diag.ko 2024-04-02T14:11:00.872 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/vmw_vsock 2024-04-02T14:11:00.873 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2024-04-02T14:11:00.873 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2024-04-02T14:11:00.873 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/vmw_vsock/vsock.ko 2024-04-02T14:11:00.873 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/vmw_vsock/vsock_diag.ko 2024-04-02T14:11:00.873 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/vmw_vsock/vsock_loopback.ko 2024-04-02T14:11:00.873 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/xfrm 2024-04-02T14:11:00.873 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/xfrm/xfrm_ipcomp.ko 2024-04-02T14:11:00.873 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/security 2024-04-02T14:11:00.873 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/security/keys 2024-04-02T14:11:00.873 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/security/keys/encrypted-keys 2024-04-02T14:11:00.873 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2024-04-02T14:11:00.873 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/security/keys/trusted-keys 2024-04-02T14:11:00.873 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/security/keys/trusted-keys/trusted.ko 2024-04-02T14:11:00.873 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/virt 2024-04-02T14:11:00.873 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/virt/lib 2024-04-02T14:11:00.873 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/virt/lib/irqbypass.ko 2024-04-02T14:11:00.873 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.alias 2024-04-02T14:11:00.874 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.alias.bin 2024-04-02T14:11:00.874 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.builtin 2024-04-02T14:11:00.874 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.builtin.alias.bin 2024-04-02T14:11:00.874 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.builtin.bin 2024-04-02T14:11:00.874 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.builtin.modinfo 2024-04-02T14:11:00.874 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.dep 2024-04-02T14:11:00.874 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.dep.bin 2024-04-02T14:11:00.874 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.devname 2024-04-02T14:11:00.874 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.order 2024-04-02T14:11:00.874 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.softdep 2024-04-02T14:11:00.874 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.symbols 2024-04-02T14:11:00.874 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.symbols.bin 2024-04-02T14:11:00.874 INFO:teuthology.orchestra.run.smithi090.stdout:/lib/modules/6.8.0-g1df0c7e8a235/vmlinuz 2024-04-02T14:11:00.878 DEBUG:teuthology.task.kernel:get_image_version: 6.8.0-g1df0c7e8a235 2024-04-02T14:11:00.878 DEBUG:teuthology.orchestra.run.smithi090:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2024-04-02T14:11:00.904 INFO:teuthology.orchestra.run.smithi090.stdout:postinstall scriptlet (using /bin/sh): 2024-04-02T14:11:00.904 INFO:teuthology.orchestra.run.smithi090.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-04-02T14:11:00.904 INFO:teuthology.orchestra.run.smithi090.stdout: /usr/bin/kernel-install add 6.8.0-g1df0c7e8a235 /lib/modules/6.8.0-g1df0c7e8a235/vmlinuz 2024-04-02T14:11:00.904 INFO:teuthology.orchestra.run.smithi090.stdout:fi 2024-04-02T14:11:00.904 INFO:teuthology.orchestra.run.smithi090.stdout:for file in vmlinuz System.map config; do 2024-04-02T14:11:00.904 INFO:teuthology.orchestra.run.smithi090.stdout: if ! cmp --silent "/lib/modules/6.8.0-g1df0c7e8a235/${file}" "/boot/${file}-6.8.0-g1df0c7e8a235"; then 2024-04-02T14:11:00.904 INFO:teuthology.orchestra.run.smithi090.stdout: cp "/lib/modules/6.8.0-g1df0c7e8a235/${file}" "/boot/${file}-6.8.0-g1df0c7e8a235" 2024-04-02T14:11:00.904 INFO:teuthology.orchestra.run.smithi090.stdout: fi 2024-04-02T14:11:00.904 INFO:teuthology.orchestra.run.smithi090.stdout:done 2024-04-02T14:11:00.904 INFO:teuthology.orchestra.run.smithi090.stdout:preuninstall scriptlet (using /bin/sh): 2024-04-02T14:11:00.904 INFO:teuthology.orchestra.run.smithi090.stdout:if [ -x /sbin/new-kernel-pkg ]; then 2024-04-02T14:11:00.904 INFO:teuthology.orchestra.run.smithi090.stdout:new-kernel-pkg --remove 6.8.0-g1df0c7e8a235 --rminitrd --initrdfile=/boot/initramfs-6.8.0-g1df0c7e8a235.img 2024-04-02T14:11:00.904 INFO:teuthology.orchestra.run.smithi090.stdout:elif [ -x /usr/bin/kernel-install ]; then 2024-04-02T14:11:00.905 INFO:teuthology.orchestra.run.smithi090.stdout:kernel-install remove 6.8.0-g1df0c7e8a235 2024-04-02T14:11:00.905 INFO:teuthology.orchestra.run.smithi090.stdout:fi 2024-04-02T14:11:00.905 INFO:teuthology.orchestra.run.smithi090.stdout:postuninstall scriptlet (using /bin/sh): 2024-04-02T14:11:00.905 INFO:teuthology.orchestra.run.smithi090.stdout:if [ -x /sbin/update-bootloader ]; then 2024-04-02T14:11:00.905 INFO:teuthology.orchestra.run.smithi090.stdout:/sbin/update-bootloader --remove 6.8.0-g1df0c7e8a235 2024-04-02T14:11:00.905 INFO:teuthology.orchestra.run.smithi090.stdout:fi 2024-04-02T14:11:00.906 DEBUG:teuthology.orchestra.run.smithi090:> sudo rpm -qi grub2-tools 2024-04-02T14:11:00.984 INFO:teuthology.orchestra.run.smithi090.stdout:Name : grub2-tools 2024-04-02T14:11:00.985 INFO:teuthology.orchestra.run.smithi090.stdout:Epoch : 1 2024-04-02T14:11:00.985 INFO:teuthology.orchestra.run.smithi090.stdout:Version : 2.06 2024-04-02T14:11:00.985 INFO:teuthology.orchestra.run.smithi090.stdout:Release : 68.el9 2024-04-02T14:11:00.985 INFO:teuthology.orchestra.run.smithi090.stdout:Architecture: x86_64 2024-04-02T14:11:00.985 INFO:teuthology.orchestra.run.smithi090.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-04-02T14:11:00.985 INFO:teuthology.orchestra.run.smithi090.stdout:Group : Unspecified 2024-04-02T14:11:00.985 INFO:teuthology.orchestra.run.smithi090.stdout:Size : 8264254 2024-04-02T14:11:00.985 INFO:teuthology.orchestra.run.smithi090.stdout:License : GPLv3+ 2024-04-02T14:11:00.985 INFO:teuthology.orchestra.run.smithi090.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-04-02T14:11:00.985 INFO:teuthology.orchestra.run.smithi090.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-04-02T14:11:00.985 INFO:teuthology.orchestra.run.smithi090.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-04-02T14:11:00.985 INFO:teuthology.orchestra.run.smithi090.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-04-02T14:11:00.985 INFO:teuthology.orchestra.run.smithi090.stdout:Packager : builder@centos.org 2024-04-02T14:11:00.985 INFO:teuthology.orchestra.run.smithi090.stdout:Vendor : CentOS 2024-04-02T14:11:00.986 INFO:teuthology.orchestra.run.smithi090.stdout:URL : http://www.gnu.org/software/grub/ 2024-04-02T14:11:00.986 INFO:teuthology.orchestra.run.smithi090.stdout:Summary : Support tools for GRUB. 2024-04-02T14:11:00.986 INFO:teuthology.orchestra.run.smithi090.stdout:Description : 2024-04-02T14:11:00.986 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-02T14:11:00.986 INFO:teuthology.orchestra.run.smithi090.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-04-02T14:11:00.986 INFO:teuthology.orchestra.run.smithi090.stdout:customizable bootloader with modular architecture. It supports a rich 2024-04-02T14:11:00.986 INFO:teuthology.orchestra.run.smithi090.stdout:variety of kernel formats, file systems, computer architectures and 2024-04-02T14:11:00.986 INFO:teuthology.orchestra.run.smithi090.stdout:hardware devices. 2024-04-02T14:11:00.986 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-02T14:11:00.986 INFO:teuthology.orchestra.run.smithi090.stdout:This subpackage provides tools for support of all platforms. 2024-04-02T14:11:00.987 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-04-02T14:11:00.987 INFO:teuthology.task.kernel:Updating grub on smithi090 to boot 6.8.0-g1df0c7e8a235 2024-04-02T14:11:00.987 DEBUG:teuthology.orchestra.run.smithi090:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-04-02T14:11:01.609 INFO:teuthology.orchestra.run.smithi090.stderr:Generating grub configuration file ... 2024-04-02T14:11:03.526 INFO:teuthology.orchestra.run.smithi090.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-04-02T14:11:03.547 INFO:teuthology.orchestra.run.smithi090.stderr:done 2024-04-02T14:11:03.550 DEBUG:teuthology.orchestra.run.smithi090:> mktemp 2024-04-02T14:11:03.580 INFO:teuthology.orchestra.run.smithi090.stdout:/tmp/tmp.bEyP3ibV9S 2024-04-02T14:11:03.580 DEBUG:teuthology.orchestra.run.smithi090:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.bEyP3ibV9S 2024-04-02T14:11:03.656 DEBUG:teuthology.orchestra.run.smithi090:> sudo chmod 0666 /tmp/tmp.bEyP3ibV9S 2024-04-02T14:11:03.804 DEBUG:teuthology.orchestra.remote:smithi090:/tmp/tmp.bEyP3ibV9S is 6KB 2024-04-02T14:11:03.853 DEBUG:teuthology.orchestra.run.smithi090:> rm -fr /tmp/tmp.bEyP3ibV9S 2024-04-02T14:11:03.869 DEBUG:teuthology.orchestra.run.smithi090:> sudo /bin/ls /boot/loader/entries || true 2024-04-02T14:11:03.952 INFO:teuthology.orchestra.run.smithi090.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-04-02T14:11:03.952 INFO:teuthology.orchestra.run.smithi090.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-04-02T14:11:03.952 INFO:teuthology.orchestra.run.smithi090.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-04-02T14:11:03.953 INFO:teuthology.orchestra.run.smithi090.stdout:15949e3c88704300882482eddfd7baae-6.8.0-g1df0c7e8a235.conf 2024-04-02T14:11:03.956 DEBUG:teuthology.orchestra.run.smithi090:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-6.8.0-g1df0c7e8a235 2024-04-02T14:11:04.137 DEBUG:teuthology.orchestra.run.smithi090:> sudo shutdown -r now 2024-04-02T14:11:04.260 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2024-04-02T14:11:34.148 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2024-04-02T14:11:34.149 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-04-02T14:11:34.150 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-02T14:11:52.478 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.90 2024-04-02T14:12:01.479 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-04-02T14:12:01.480 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-02T14:12:04.542 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.90 2024-04-02T14:12:16.544 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-04-02T14:12:16.545 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-02T14:12:19.614 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.90 2024-04-02T14:12:34.615 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-04-02T14:12:34.616 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-02T14:13:34.618 DEBUG:teuthology.orchestra.remote:timed out 2024-04-02T14:13:52.620 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-04-02T14:13:52.621 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-02T14:13:53.081 DEBUG:teuthology.orchestra.run.smithi090:> true 2024-04-02T14:13:53.810 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-04-02T14:13:53.810 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "6.8.0-g1df0c7e8a235"... 2024-04-02T14:13:53.811 DEBUG:teuthology.orchestra.run.smithi090:> uname -r 2024-04-02T14:13:53.861 INFO:teuthology.orchestra.run.smithi090.stdout:6.8.0-g1df0c7e8a235 2024-04-02T14:13:53.861 DEBUG:teuthology.task.kernel:current kernel version is 6.8.0-g1df0c7e8a235 vs 6.8.0-g1df0c7e8a235 2024-04-02T14:13:53.861 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-02T14:13:53.861 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-04-02T14:13:54.863 DEBUG:teuthology.parallel:result is None 2024-04-02T14:13:54.863 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-02T14:13:54.871 INFO:teuthology.task.internal:Creating test directory... 2024-04-02T14:13:54.871 DEBUG:teuthology.orchestra.run.smithi090:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-02T14:13:54.879 DEBUG:teuthology.orchestra.run.smithi092:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-02T14:13:54.881 DEBUG:teuthology.orchestra.run.smithi177:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-02T14:13:54.928 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-02T14:13:54.934 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-02T14:13:54.954 INFO:teuthology.task.internal:Creating archive directory... 2024-04-02T14:13:54.954 DEBUG:teuthology.orchestra.run.smithi090:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-02T14:13:54.962 DEBUG:teuthology.orchestra.run.smithi092:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-02T14:13:54.964 DEBUG:teuthology.orchestra.run.smithi177:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-02T14:13:55.081 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-02T14:13:55.087 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-02T14:13:55.087 DEBUG:teuthology.orchestra.run.smithi092:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-02T14:13:55.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T14:13:55.104 DEBUG:teuthology.orchestra.run.smithi177:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-02T14:13:55.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T14:13:55.121 DEBUG:teuthology.orchestra.run.smithi090:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-02T14:13:55.128 DEBUG:teuthology.orchestra.run.smithi092:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-02T14:13:55.146 DEBUG:teuthology.orchestra.run.smithi177:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-02T14:13:55.189 INFO:teuthology.orchestra.run.smithi092.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-02T14:13:55.206 INFO:teuthology.orchestra.run.smithi092.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-02T14:13:55.213 INFO:teuthology.orchestra.run.smithi177.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-02T14:13:55.231 INFO:teuthology.orchestra.run.smithi177.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-02T14:13:55.300 INFO:teuthology.orchestra.run.smithi090.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-02T14:13:55.379 INFO:teuthology.orchestra.run.smithi090.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-02T14:13:55.384 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-02T14:13:55.391 INFO:teuthology.task.internal:Configuring sudo... 2024-04-02T14:13:55.391 DEBUG:teuthology.orchestra.run.smithi090:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-02T14:13:55.398 DEBUG:teuthology.orchestra.run.smithi092:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-02T14:13:55.400 DEBUG:teuthology.orchestra.run.smithi177:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-02T14:13:55.495 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-02T14:13:55.504 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-02T14:13:55.505 DEBUG:teuthology.orchestra.run.smithi090:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-02T14:13:55.512 DEBUG:teuthology.orchestra.run.smithi092:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-02T14:13:55.514 DEBUG:teuthology.orchestra.run.smithi177:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-02T14:13:55.559 DEBUG:teuthology.orchestra.run.smithi090:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-02T14:13:55.642 DEBUG:teuthology.orchestra.run.smithi090:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-02T14:13:55.749 DEBUG:teuthology.orchestra.run.smithi090:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-02T14:13:55.832 DEBUG:teuthology.orchestra.run.smithi090:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-02T14:13:55.929 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-04-02T14:13:55.929 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-02T14:13:56.024 DEBUG:teuthology.orchestra.run.smithi092:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-02T14:13:56.067 DEBUG:teuthology.orchestra.run.smithi092:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-02T14:13:56.126 DEBUG:teuthology.orchestra.run.smithi092:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-02T14:13:56.214 DEBUG:teuthology.orchestra.run.smithi092:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-02T14:13:56.257 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2024-04-02T14:13:56.257 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-02T14:13:56.329 DEBUG:teuthology.orchestra.run.smithi177:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-02T14:13:56.372 DEBUG:teuthology.orchestra.run.smithi177:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-02T14:13:56.426 DEBUG:teuthology.orchestra.run.smithi177:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-02T14:13:56.512 DEBUG:teuthology.orchestra.run.smithi177:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-02T14:13:56.556 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-04-02T14:13:56.556 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-02T14:13:56.627 DEBUG:teuthology.orchestra.run.smithi090:> sudo service rsyslog restart 2024-04-02T14:13:56.635 DEBUG:teuthology.orchestra.run.smithi092:> sudo service rsyslog restart 2024-04-02T14:13:56.637 DEBUG:teuthology.orchestra.run.smithi177:> sudo service rsyslog restart 2024-04-02T14:13:56.679 INFO:teuthology.orchestra.run.smithi092.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-02T14:13:56.711 INFO:teuthology.orchestra.run.smithi177.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-02T14:13:56.759 INFO:teuthology.orchestra.run.smithi090.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-02T14:13:57.258 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-02T14:13:57.265 INFO:teuthology.task.internal:Starting timer... 2024-04-02T14:13:57.265 INFO:teuthology.run_tasks:Running task pcp... 2024-04-02T14:13:57.317 INFO:teuthology.run_tasks:Running task selinux... 2024-04-02T14:13:57.348 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-04-02T14:13:57.349 DEBUG:teuthology.orchestra.run.smithi090:> sudo service auditd rotate 2024-04-02T14:13:57.475 INFO:teuthology.orchestra.run.smithi090.stdout:Rotating logs: 2024-04-02T14:13:57.479 DEBUG:teuthology.orchestra.run.smithi092:> sudo service auditd rotate 2024-04-02T14:13:57.527 INFO:teuthology.orchestra.run.smithi092.stdout:Rotating logs: 2024-04-02T14:13:57.529 DEBUG:teuthology.orchestra.run.smithi177:> sudo service auditd rotate 2024-04-02T14:13:57.583 INFO:teuthology.orchestra.run.smithi177.stdout:Rotating logs: 2024-04-02T14:13:57.585 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-02T14:13:57.585 DEBUG:teuthology.orchestra.run.smithi090:> /usr/sbin/getenforce 2024-04-02T14:13:57.641 INFO:teuthology.orchestra.run.smithi090.stdout:Permissive 2024-04-02T14:13:57.642 DEBUG:teuthology.orchestra.run.smithi092:> /usr/sbin/getenforce 2024-04-02T14:13:57.669 INFO:teuthology.orchestra.run.smithi092.stdout:Permissive 2024-04-02T14:13:57.669 DEBUG:teuthology.orchestra.run.smithi177:> /usr/sbin/getenforce 2024-04-02T14:13:57.695 INFO:teuthology.orchestra.run.smithi177.stdout:Permissive 2024-04-02T14:13:57.696 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi090.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi092.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi177.front.sepia.ceph.com': 'permissive'} 2024-04-02T14:13:57.696 DEBUG:teuthology.orchestra.run.smithi090:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-02T14:13:57.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T14:13:57.774 DEBUG:teuthology.orchestra.run.smithi092:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-02T14:13:57.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T14:13:57.804 DEBUG:teuthology.orchestra.run.smithi177:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-02T14:13:57.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T14:13:57.833 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-02T14:13:57.833 DEBUG:teuthology.orchestra.run.smithi090:> sudo /usr/sbin/setenforce permissive 2024-04-02T14:13:57.927 DEBUG:teuthology.orchestra.run.smithi092:> sudo /usr/sbin/setenforce permissive 2024-04-02T14:13:57.962 DEBUG:teuthology.orchestra.run.smithi177:> sudo /usr/sbin/setenforce permissive 2024-04-02T14:13:57.998 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-02T14:13:58.007 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-02T14:13:58.023 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-02T14:13:58.024 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi090.front.sepia.ceph.com,smithi092.front.sepia.ceph.com,smithi177.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-02T14:19:57.277 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi090.front.sepia.ceph.com'), Remote(name='ubuntu@smithi092.front.sepia.ceph.com'), Remote(name='ubuntu@smithi177.front.sepia.ceph.com')] 2024-04-02T14:19:57.278 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-04-02T14:19:57.279 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-02T14:19:57.421 DEBUG:teuthology.orchestra.run.smithi090:> true 2024-04-02T14:19:57.569 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-04-02T14:19:57.569 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi092.front.sepia.ceph.com' 2024-04-02T14:19:57.570 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi092.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-02T14:19:57.657 DEBUG:teuthology.orchestra.run.smithi092:> true 2024-04-02T14:19:57.682 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi092.front.sepia.ceph.com' 2024-04-02T14:19:57.683 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi177.front.sepia.ceph.com' 2024-04-02T14:19:57.683 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi177.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-02T14:19:57.773 DEBUG:teuthology.orchestra.run.smithi177:> true 2024-04-02T14:19:57.842 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi177.front.sepia.ceph.com' 2024-04-02T14:19:57.842 INFO:teuthology.run_tasks:Running task clock... 2024-04-02T14:19:57.853 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-02T14:19:57.853 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-02T14:19:57.854 DEBUG:teuthology.orchestra.run.smithi090:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-02T14:19:57.861 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-02T14:19:57.861 DEBUG:teuthology.orchestra.run.smithi092:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-02T14:19:57.863 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-02T14:19:57.863 DEBUG:teuthology.orchestra.run.smithi177:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-02T14:19:57.892 INFO:teuthology.orchestra.run.smithi092.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-02T14:19:57.910 INFO:teuthology.orchestra.run.smithi092.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-02T14:19:57.914 INFO:teuthology.orchestra.run.smithi177.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-02T14:19:57.931 INFO:teuthology.orchestra.run.smithi177.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-02T14:19:57.939 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: ntpd: command not found 2024-04-02T14:19:57.953 INFO:teuthology.orchestra.run.smithi092.stdout:506 Cannot talk to daemon 2024-04-02T14:19:57.956 INFO:teuthology.orchestra.run.smithi090.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-02T14:19:57.961 INFO:teuthology.orchestra.run.smithi177.stderr:sudo: ntpd: command not found 2024-04-02T14:19:57.969 INFO:teuthology.orchestra.run.smithi092.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-02T14:19:57.975 INFO:teuthology.orchestra.run.smithi177.stdout:506 Cannot talk to daemon 2024-04-02T14:19:57.985 INFO:teuthology.orchestra.run.smithi092.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-02T14:19:57.992 INFO:teuthology.orchestra.run.smithi177.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-02T14:19:58.006 INFO:teuthology.orchestra.run.smithi090.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-02T14:19:58.008 INFO:teuthology.orchestra.run.smithi177.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-02T14:19:58.029 INFO:teuthology.orchestra.run.smithi092.stderr:bash: line 1: ntpq: command not found 2024-04-02T14:19:58.033 INFO:teuthology.orchestra.run.smithi092.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-02T14:19:58.033 INFO:teuthology.orchestra.run.smithi092.stdout:=============================================================================== 2024-04-02T14:19:58.033 INFO:teuthology.orchestra.run.smithi092.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-02T14:19:58.033 INFO:teuthology.orchestra.run.smithi092.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-02T14:19:58.033 INFO:teuthology.orchestra.run.smithi092.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-02T14:19:58.033 INFO:teuthology.orchestra.run.smithi092.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-02T14:19:58.051 INFO:teuthology.orchestra.run.smithi177.stderr:bash: line 1: ntpq: command not found 2024-04-02T14:19:58.055 INFO:teuthology.orchestra.run.smithi177.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-02T14:19:58.055 INFO:teuthology.orchestra.run.smithi177.stdout:=============================================================================== 2024-04-02T14:19:58.055 INFO:teuthology.orchestra.run.smithi177.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-02T14:19:58.055 INFO:teuthology.orchestra.run.smithi177.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-02T14:19:58.055 INFO:teuthology.orchestra.run.smithi177.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-02T14:19:58.055 INFO:teuthology.orchestra.run.smithi177.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-02T14:19:58.103 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: ntpd: command not found 2024-04-02T14:19:58.144 INFO:teuthology.orchestra.run.smithi090.stdout:506 Cannot talk to daemon 2024-04-02T14:19:58.196 INFO:teuthology.orchestra.run.smithi090.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-02T14:19:58.245 INFO:teuthology.orchestra.run.smithi090.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-02T14:19:58.369 INFO:teuthology.orchestra.run.smithi090.stderr:bash: line 1: ntpq: command not found 2024-04-02T14:19:58.377 INFO:teuthology.orchestra.run.smithi090.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-02T14:19:58.377 INFO:teuthology.orchestra.run.smithi090.stdout:=============================================================================== 2024-04-02T14:19:58.377 INFO:teuthology.orchestra.run.smithi090.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-02T14:19:58.377 INFO:teuthology.orchestra.run.smithi090.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-02T14:19:58.377 INFO:teuthology.orchestra.run.smithi090.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-02T14:19:58.377 INFO:teuthology.orchestra.run.smithi090.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-02T14:19:58.379 INFO:teuthology.run_tasks:Running task install... 2024-04-02T14:19:58.390 DEBUG:teuthology.task.install:project ceph 2024-04-02T14:19:58.390 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '191d963ac9de7322da8aff20b567b3fa458e2ef6'}} 2024-04-02T14:19:58.390 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '191d963ac9de7322da8aff20b567b3fa458e2ef6'} 2024-04-02T14:19:58.390 INFO:teuthology.task.install:Using flavor: default 2024-04-02T14:19:58.396 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-04-02T14:19:58.396 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-04-02T14:19:58.396 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '191d963ac9de7322da8aff20b567b3fa458e2ef6', 'tag': None, 'wait_for_package': False} 2024-04-02T14:19:58.397 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=191d963ac9de7322da8aff20b567b3fa458e2ef6 2024-04-02T14:19:58.399 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '191d963ac9de7322da8aff20b567b3fa458e2ef6', 'tag': None, 'wait_for_package': False} 2024-04-02T14:19:58.399 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=191d963ac9de7322da8aff20b567b3fa458e2ef6 2024-04-02T14:19:58.400 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '191d963ac9de7322da8aff20b567b3fa458e2ef6', 'tag': None, 'wait_for_package': False} 2024-04-02T14:19:58.400 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=191d963ac9de7322da8aff20b567b3fa458e2ef6 2024-04-02T14:19:58.542 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240402.004512/191d963ac9de7322da8aff20b567b3fa458e2ef6/centos/9/flavors/default/ 2024-04-02T14:19:58.542 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2609.g191d963a 2024-04-02T14:19:58.553 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240402.004512/191d963ac9de7322da8aff20b567b3fa458e2ef6/centos/9/flavors/default/ 2024-04-02T14:19:58.553 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2609.g191d963a 2024-04-02T14:19:58.619 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240402.004512/191d963ac9de7322da8aff20b567b3fa458e2ef6/centos/9/flavors/default/ 2024-04-02T14:19:58.619 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2609.g191d963a 2024-04-02T14:19:58.668 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240402.004512/191d963ac9de7322da8aff20b567b3fa458e2ef6/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240402.004512/191d963ac9de7322da8aff20b567b3fa458e2ef6/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240402.004512/191d963ac9de7322da8aff20b567b3fa458e2ef6/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-02T14:19:58.668 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2024-04-02T14:19:58.669 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-02T14:19:58.698 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-02T14:19:58.698 DEBUG:teuthology.orchestra.run.smithi092:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/191d963ac9de7322da8aff20b567b3fa458e2ef6/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-02T14:19:58.720 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240402.004512/191d963ac9de7322da8aff20b567b3fa458e2ef6/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240402.004512/191d963ac9de7322da8aff20b567b3fa458e2ef6/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240402.004512/191d963ac9de7322da8aff20b567b3fa458e2ef6/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-02T14:19:58.720 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-04-02T14:19:58.721 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-02T14:19:58.728 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240402.004512/191d963ac9de7322da8aff20b567b3fa458e2ef6/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240402.004512/191d963ac9de7322da8aff20b567b3fa458e2ef6/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240402.004512/191d963ac9de7322da8aff20b567b3fa458e2ef6/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-02T14:19:58.728 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-04-02T14:19:58.728 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-02T14:19:58.760 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-02T14:19:58.760 DEBUG:teuthology.orchestra.run.smithi177:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/191d963ac9de7322da8aff20b567b3fa458e2ef6/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-02T14:19:58.767 DEBUG:teuthology.orchestra.run.smithi092:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-02T14:19:58.818 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-02T14:19:58.818 DEBUG:teuthology.orchestra.run.smithi090:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/191d963ac9de7322da8aff20b567b3fa458e2ef6/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-02T14:19:58.832 DEBUG:teuthology.orchestra.run.smithi177:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-02T14:19:58.848 DEBUG:teuthology.orchestra.run.smithi092:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-02T14:19:58.915 INFO:teuthology.orchestra.run.smithi092.stdout:check_obsoletes = 1 2024-04-02T14:19:58.917 DEBUG:teuthology.orchestra.run.smithi092:> sudo yum clean all 2024-04-02T14:19:58.917 DEBUG:teuthology.orchestra.run.smithi177:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-02T14:19:58.949 DEBUG:teuthology.orchestra.run.smithi090:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-02T14:19:58.985 INFO:teuthology.orchestra.run.smithi177.stdout:check_obsoletes = 1 2024-04-02T14:19:58.987 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum clean all 2024-04-02T14:19:59.081 DEBUG:teuthology.orchestra.run.smithi090:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-02T14:19:59.151 INFO:teuthology.orchestra.run.smithi092.stdout:74 files removed 2024-04-02T14:19:59.166 INFO:teuthology.orchestra.run.smithi090.stdout:check_obsoletes = 1 2024-04-02T14:19:59.171 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum clean all 2024-04-02T14:19:59.179 DEBUG:teuthology.orchestra.run.smithi092:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-02T14:19:59.225 INFO:teuthology.orchestra.run.smithi177.stdout:74 files removed 2024-04-02T14:19:59.253 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-02T14:19:59.523 INFO:teuthology.orchestra.run.smithi090.stdout:74 files removed 2024-04-02T14:19:59.562 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-02T14:19:59.953 INFO:teuthology.orchestra.run.smithi092.stdout:ceph packages for x86_64 172 kB/s | 83 kB 00:00 2024-04-02T14:19:59.968 INFO:teuthology.orchestra.run.smithi177.stdout:ceph packages for x86_64 195 kB/s | 83 kB 00:00 2024-04-02T14:20:00.285 INFO:teuthology.orchestra.run.smithi177.stdout:ceph noarch packages 42 kB/s | 12 kB 00:00 2024-04-02T14:20:00.378 INFO:teuthology.orchestra.run.smithi092.stdout:ceph noarch packages 30 kB/s | 12 kB 00:00 2024-04-02T14:20:00.529 INFO:teuthology.orchestra.run.smithi090.stdout:ceph packages for x86_64 179 kB/s | 83 kB 00:00 2024-04-02T14:20:00.669 INFO:teuthology.orchestra.run.smithi177.stdout:ceph source packages 5.4 kB/s | 1.9 kB 00:00 2024-04-02T14:20:00.804 INFO:teuthology.orchestra.run.smithi092.stdout:ceph source packages 4.8 kB/s | 1.9 kB 00:00 2024-04-02T14:20:01.246 INFO:teuthology.orchestra.run.smithi090.stdout:ceph noarch packages 19 kB/s | 12 kB 00:00 2024-04-02T14:20:01.246 INFO:teuthology.orchestra.run.smithi177.stdout:CentOS-9 - AppStream 12 MB/s | 6.7 MB 00:00 2024-04-02T14:20:01.373 INFO:teuthology.orchestra.run.smithi092.stdout:CentOS-9 - AppStream 12 MB/s | 6.7 MB 00:00 2024-04-02T14:20:01.822 INFO:teuthology.orchestra.run.smithi090.stdout:ceph source packages 3.8 kB/s | 1.9 kB 00:00 2024-04-02T14:20:02.780 INFO:teuthology.orchestra.run.smithi177.stdout:CentOS-9 - BaseOS 4.8 MB/s | 2.0 MB 00:00 2024-04-02T14:20:02.890 INFO:teuthology.orchestra.run.smithi092.stdout:CentOS-9 - BaseOS 5.0 MB/s | 2.0 MB 00:00 2024-04-02T14:20:03.259 INFO:teuthology.orchestra.run.smithi090.stdout:CentOS-9 - AppStream 4.9 MB/s | 6.7 MB 00:01 2024-04-02T14:20:04.339 INFO:teuthology.orchestra.run.smithi092.stdout:CentOS Stream 9 - BaseOS 7.1 MB/s | 8.0 MB 00:01 2024-04-02T14:20:04.928 INFO:teuthology.orchestra.run.smithi090.stdout:CentOS-9 - BaseOS 5.3 MB/s | 2.0 MB 00:00 2024-04-02T14:20:06.649 INFO:teuthology.orchestra.run.smithi090.stdout:CentOS Stream 9 - BaseOS 6.0 MB/s | 8.0 MB 00:01 2024-04-02T14:20:07.473 INFO:teuthology.orchestra.run.smithi092.stdout:CentOS Stream 9 - AppStream 9.2 MB/s | 19 MB 00:02 2024-04-02T14:20:09.592 INFO:teuthology.orchestra.run.smithi090.stdout:CentOS Stream 9 - AppStream 11 MB/s | 19 MB 00:01 2024-04-02T14:20:12.590 INFO:teuthology.orchestra.run.smithi092.stdout:CentOS Stream 9 - CRB 3.8 MB/s | 6.1 MB 00:01 2024-04-02T14:20:13.944 INFO:teuthology.orchestra.run.smithi090.stdout:CentOS Stream 9 - CRB 11 MB/s | 6.1 MB 00:00 2024-04-02T14:20:13.990 INFO:teuthology.orchestra.run.smithi092.stdout:CentOS Stream 9 - Extras packages 60 kB/s | 16 kB 00:00 2024-04-02T14:20:14.809 INFO:teuthology.orchestra.run.smithi092.stdout:Extra Packages for Enterprise Linux 30 MB/s | 21 MB 00:00 2024-04-02T14:20:14.903 INFO:teuthology.orchestra.run.smithi177.stdout:CentOS Stream 9 - BaseOS 697 kB/s | 8.0 MB 00:11 2024-04-02T14:20:15.495 INFO:teuthology.orchestra.run.smithi090.stdout:CentOS Stream 9 - Extras packages 70 kB/s | 16 kB 00:00 2024-04-02T14:20:17.820 INFO:teuthology.orchestra.run.smithi090.stdout:Extra Packages for Enterprise Linux 9.7 MB/s | 21 MB 00:02 2024-04-02T14:20:19.742 INFO:teuthology.orchestra.run.smithi092.stdout:lab-extras 27 kB/s | 1.7 kB 00:00 2024-04-02T14:20:19.770 INFO:teuthology.orchestra.run.smithi177.stdout:CentOS Stream 9 - AppStream 5.0 MB/s | 19 MB 00:03 2024-04-02T14:20:22.258 INFO:teuthology.orchestra.run.smithi092.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-02T14:20:22.259 INFO:teuthology.orchestra.run.smithi092.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-02T14:20:22.274 INFO:teuthology.orchestra.run.smithi092.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-04-02T14:20:22.275 INFO:teuthology.orchestra.run.smithi092.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-02T14:20:22.276 INFO:teuthology.orchestra.run.smithi092.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-04-02T14:20:22.277 INFO:teuthology.orchestra.run.smithi092.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-04-02T14:20:22.279 INFO:teuthology.orchestra.run.smithi092.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-02T14:20:22.282 INFO:teuthology.orchestra.run.smithi092.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-04-02T14:20:22.284 INFO:teuthology.orchestra.run.smithi092.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-04-02T14:20:22.366 INFO:teuthology.orchestra.run.smithi092.stdout:Dependencies resolved. 2024-04-02T14:20:22.378 INFO:teuthology.orchestra.run.smithi092.stdout:============================================================================================ 2024-04-02T14:20:22.379 INFO:teuthology.orchestra.run.smithi092.stdout: Package Arch Version Repository Size 2024-04-02T14:20:22.379 INFO:teuthology.orchestra.run.smithi092.stdout:============================================================================================ 2024-04-02T14:20:22.379 INFO:teuthology.orchestra.run.smithi092.stdout:Installing: 2024-04-02T14:20:22.379 INFO:teuthology.orchestra.run.smithi092.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-04-02T14:20:22.379 INFO:teuthology.orchestra.run.smithi092.stdout: ceph x86_64 2:19.0.0-2609.g191d963a.el9 ceph 6.6 k 2024-04-02T14:20:22.379 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-base x86_64 2:19.0.0-2609.g191d963a.el9 ceph 5.4 M 2024-04-02T14:20:22.379 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-fuse x86_64 2:19.0.0-2609.g191d963a.el9 ceph 890 k 2024-04-02T14:20:22.379 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2609.g191d963a.el9 ceph 145 k 2024-04-02T14:20:22.379 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mgr x86_64 2:19.0.0-2609.g191d963a.el9 ceph 1.5 M 2024-04-02T14:20:22.379 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2609.g191d963a.el9 ceph-noarch 141 k 2024-04-02T14:20:22.379 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2609.g191d963a.el9 ceph-noarch 3.5 M 2024-04-02T14:20:22.379 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2609.g191d963a.el9 ceph-noarch 7.4 M 2024-04-02T14:20:22.379 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mgr-rook noarch 2:19.0.0-2609.g191d963a.el9 ceph-noarch 49 k 2024-04-02T14:20:22.380 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-radosgw x86_64 2:19.0.0-2609.g191d963a.el9 ceph 9.8 M 2024-04-02T14:20:22.380 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-test x86_64 2:19.0.0-2609.g191d963a.el9 ceph 48 M 2024-04-02T14:20:22.380 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-volume noarch 2:19.0.0-2609.g191d963a.el9 ceph-noarch 264 k 2024-04-02T14:20:22.380 INFO:teuthology.orchestra.run.smithi092.stdout: cephadm noarch 2:19.0.0-2609.g191d963a.el9 ceph-noarch 762 k 2024-04-02T14:20:22.380 INFO:teuthology.orchestra.run.smithi092.stdout: cephfs-mirror x86_64 2:19.0.0-2609.g191d963a.el9 ceph 225 k 2024-04-02T14:20:22.380 INFO:teuthology.orchestra.run.smithi092.stdout: cephfs-shell x86_64 2:19.0.0-2609.g191d963a.el9 ceph 27 k 2024-04-02T14:20:22.380 INFO:teuthology.orchestra.run.smithi092.stdout: cephfs-top noarch 2:19.0.0-2609.g191d963a.el9 ceph-noarch 18 k 2024-04-02T14:20:22.380 INFO:teuthology.orchestra.run.smithi092.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-02T14:20:22.380 INFO:teuthology.orchestra.run.smithi092.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-04-02T14:20:22.380 INFO:teuthology.orchestra.run.smithi092.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-04-02T14:20:22.380 INFO:teuthology.orchestra.run.smithi092.stdout: libcephfs-devel x86_64 2:19.0.0-2609.g191d963a.el9 ceph 32 k 2024-04-02T14:20:22.380 INFO:teuthology.orchestra.run.smithi092.stdout: libcephfs2 x86_64 2:19.0.0-2609.g191d963a.el9 ceph 747 k 2024-04-02T14:20:22.380 INFO:teuthology.orchestra.run.smithi092.stdout: librados-devel x86_64 2:19.0.0-2609.g191d963a.el9 ceph 127 k 2024-04-02T14:20:22.380 INFO:teuthology.orchestra.run.smithi092.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-04-02T14:20:22.381 INFO:teuthology.orchestra.run.smithi092.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-02T14:20:22.381 INFO:teuthology.orchestra.run.smithi092.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-04-02T14:20:22.381 INFO:teuthology.orchestra.run.smithi092.stdout: postgresql x86_64 13.14-1.el9 CentOS-AppStream 1.5 M 2024-04-02T14:20:22.381 INFO:teuthology.orchestra.run.smithi092.stdout: postgresql-contrib x86_64 13.14-1.el9 CentOS-AppStream 817 k 2024-04-02T14:20:22.381 INFO:teuthology.orchestra.run.smithi092.stdout: postgresql-server x86_64 13.14-1.el9 CentOS-AppStream 5.7 M 2024-04-02T14:20:22.381 INFO:teuthology.orchestra.run.smithi092.stdout: python3-cephfs x86_64 2:19.0.0-2609.g191d963a.el9 ceph 161 k 2024-04-02T14:20:22.381 INFO:teuthology.orchestra.run.smithi092.stdout: python3-rados x86_64 2:19.0.0-2609.g191d963a.el9 ceph 320 k 2024-04-02T14:20:22.381 INFO:teuthology.orchestra.run.smithi092.stdout: python3-rbd x86_64 2:19.0.0-2609.g191d963a.el9 ceph 299 k 2024-04-02T14:20:22.381 INFO:teuthology.orchestra.run.smithi092.stdout: python3-rgw x86_64 2:19.0.0-2609.g191d963a.el9 ceph 99 k 2024-04-02T14:20:22.381 INFO:teuthology.orchestra.run.smithi092.stdout: rbd-fuse x86_64 2:19.0.0-2609.g191d963a.el9 ceph 85 k 2024-04-02T14:20:22.381 INFO:teuthology.orchestra.run.smithi092.stdout: rbd-mirror x86_64 2:19.0.0-2609.g191d963a.el9 ceph 3.1 M 2024-04-02T14:20:22.381 INFO:teuthology.orchestra.run.smithi092.stdout: rbd-nbd x86_64 2:19.0.0-2609.g191d963a.el9 ceph 170 k 2024-04-02T14:20:22.382 INFO:teuthology.orchestra.run.smithi092.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-02T14:20:22.382 INFO:teuthology.orchestra.run.smithi092.stdout:Upgrading: 2024-04-02T14:20:22.382 INFO:teuthology.orchestra.run.smithi092.stdout: NetworkManager x86_64 1:1.47.2-1.el9 CentOS-BaseOS 2.3 M 2024-04-02T14:20:22.382 INFO:teuthology.orchestra.run.smithi092.stdout: NetworkManager-libnm x86_64 1:1.47.2-1.el9 CentOS-BaseOS 1.8 M 2024-04-02T14:20:22.382 INFO:teuthology.orchestra.run.smithi092.stdout: NetworkManager-team x86_64 1:1.47.2-1.el9 CentOS-BaseOS 41 k 2024-04-02T14:20:22.382 INFO:teuthology.orchestra.run.smithi092.stdout: NetworkManager-tui x86_64 1:1.47.2-1.el9 CentOS-BaseOS 247 k 2024-04-02T14:20:22.382 INFO:teuthology.orchestra.run.smithi092.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-04-02T14:20:22.382 INFO:teuthology.orchestra.run.smithi092.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-04-02T14:20:22.382 INFO:teuthology.orchestra.run.smithi092.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-04-02T14:20:22.382 INFO:teuthology.orchestra.run.smithi092.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-04-02T14:20:22.382 INFO:teuthology.orchestra.run.smithi092.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-04-02T14:20:22.382 INFO:teuthology.orchestra.run.smithi092.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-04-02T14:20:22.382 INFO:teuthology.orchestra.run.smithi092.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-04-02T14:20:22.382 INFO:teuthology.orchestra.run.smithi092.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-04-02T14:20:22.383 INFO:teuthology.orchestra.run.smithi092.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-04-02T14:20:22.383 INFO:teuthology.orchestra.run.smithi092.stdout: librados2 x86_64 2:19.0.0-2609.g191d963a.el9 ceph 3.4 M 2024-04-02T14:20:22.383 INFO:teuthology.orchestra.run.smithi092.stdout: librbd1 x86_64 2:19.0.0-2609.g191d963a.el9 ceph 3.1 M 2024-04-02T14:20:22.383 INFO:teuthology.orchestra.run.smithi092.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-04-02T14:20:22.383 INFO:teuthology.orchestra.run.smithi092.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-04-02T14:20:22.383 INFO:teuthology.orchestra.run.smithi092.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-04-02T14:20:22.383 INFO:teuthology.orchestra.run.smithi092.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-04-02T14:20:22.383 INFO:teuthology.orchestra.run.smithi092.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-04-02T14:20:22.383 INFO:teuthology.orchestra.run.smithi092.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-04-02T14:20:22.383 INFO:teuthology.orchestra.run.smithi092.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-04-02T14:20:22.383 INFO:teuthology.orchestra.run.smithi092.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-04-02T14:20:22.383 INFO:teuthology.orchestra.run.smithi092.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-04-02T14:20:22.383 INFO:teuthology.orchestra.run.smithi092.stdout:Installing dependencies: 2024-04-02T14:20:22.383 INFO:teuthology.orchestra.run.smithi092.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-02T14:20:22.383 INFO:teuthology.orchestra.run.smithi092.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-04-02T14:20:22.383 INFO:teuthology.orchestra.run.smithi092.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-04-02T14:20:22.384 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-common x86_64 2:19.0.0-2609.g191d963a.el9 ceph 21 M 2024-04-02T14:20:22.384 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2609.g191d963a.el9 ceph-noarch 34 k 2024-04-02T14:20:22.384 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mds x86_64 2:19.0.0-2609.g191d963a.el9 ceph 2.4 M 2024-04-02T14:20:22.384 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2609.g191d963a.el9 ceph-noarch 249 k 2024-04-02T14:20:22.384 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mon x86_64 2:19.0.0-2609.g191d963a.el9 ceph 4.7 M 2024-04-02T14:20:22.384 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-osd x86_64 2:19.0.0-2609.g191d963a.el9 ceph 17 M 2024-04-02T14:20:22.384 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2609.g191d963a.el9 ceph-noarch 16 k 2024-04-02T14:20:22.384 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-selinux x86_64 2:19.0.0-2609.g191d963a.el9 ceph 25 k 2024-04-02T14:20:22.384 INFO:teuthology.orchestra.run.smithi092.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-04-02T14:20:22.384 INFO:teuthology.orchestra.run.smithi092.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-04-02T14:20:22.384 INFO:teuthology.orchestra.run.smithi092.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-04-02T14:20:22.384 INFO:teuthology.orchestra.run.smithi092.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-04-02T14:20:22.384 INFO:teuthology.orchestra.run.smithi092.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-02T14:20:22.384 INFO:teuthology.orchestra.run.smithi092.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-02T14:20:22.384 INFO:teuthology.orchestra.run.smithi092.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-04-02T14:20:22.385 INFO:teuthology.orchestra.run.smithi092.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-02T14:20:22.385 INFO:teuthology.orchestra.run.smithi092.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-02T14:20:22.385 INFO:teuthology.orchestra.run.smithi092.stdout: libcephsqlite x86_64 2:19.0.0-2609.g191d963a.el9 ceph 164 k 2024-04-02T14:20:22.385 INFO:teuthology.orchestra.run.smithi092.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-02T14:20:22.385 INFO:teuthology.orchestra.run.smithi092.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-02T14:20:22.385 INFO:teuthology.orchestra.run.smithi092.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-04-02T14:20:22.385 INFO:teuthology.orchestra.run.smithi092.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-02T14:20:22.385 INFO:teuthology.orchestra.run.smithi092.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-04-02T14:20:22.385 INFO:teuthology.orchestra.run.smithi092.stdout: libradosstriper1 x86_64 2:19.0.0-2609.g191d963a.el9 ceph 500 k 2024-04-02T14:20:22.385 INFO:teuthology.orchestra.run.smithi092.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-04-02T14:20:22.385 INFO:teuthology.orchestra.run.smithi092.stdout: librgw2 x86_64 2:19.0.0-2609.g191d963a.el9 ceph 5.1 M 2024-04-02T14:20:22.385 INFO:teuthology.orchestra.run.smithi092.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-02T14:20:22.385 INFO:teuthology.orchestra.run.smithi092.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-04-02T14:20:22.385 INFO:teuthology.orchestra.run.smithi092.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-04-02T14:20:22.386 INFO:teuthology.orchestra.run.smithi092.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-04-02T14:20:22.386 INFO:teuthology.orchestra.run.smithi092.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-02T14:20:22.386 INFO:teuthology.orchestra.run.smithi092.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-02T14:20:22.386 INFO:teuthology.orchestra.run.smithi092.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-04-02T14:20:22.386 INFO:teuthology.orchestra.run.smithi092.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-04-02T14:20:22.386 INFO:teuthology.orchestra.run.smithi092.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-04-02T14:20:22.386 INFO:teuthology.orchestra.run.smithi092.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-02T14:20:22.386 INFO:teuthology.orchestra.run.smithi092.stdout: postgresql-private-libs x86_64 13.14-1.el9 CentOS-AppStream 132 k 2024-04-02T14:20:22.386 INFO:teuthology.orchestra.run.smithi092.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-04-02T14:20:22.386 INFO:teuthology.orchestra.run.smithi092.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-02T14:20:22.386 INFO:teuthology.orchestra.run.smithi092.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-02T14:20:22.386 INFO:teuthology.orchestra.run.smithi092.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-04-02T14:20:22.386 INFO:teuthology.orchestra.run.smithi092.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-04-02T14:20:22.386 INFO:teuthology.orchestra.run.smithi092.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-02T14:20:22.386 INFO:teuthology.orchestra.run.smithi092.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-02T14:20:22.387 INFO:teuthology.orchestra.run.smithi092.stdout: python3-ceph-argparse x86_64 2:19.0.0-2609.g191d963a.el9 ceph 45 k 2024-04-02T14:20:22.387 INFO:teuthology.orchestra.run.smithi092.stdout: python3-ceph-common x86_64 2:19.0.0-2609.g191d963a.el9 ceph 134 k 2024-04-02T14:20:22.387 INFO:teuthology.orchestra.run.smithi092.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-02T14:20:22.387 INFO:teuthology.orchestra.run.smithi092.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-04-02T14:20:22.387 INFO:teuthology.orchestra.run.smithi092.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-02T14:20:22.387 INFO:teuthology.orchestra.run.smithi092.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-02T14:20:22.387 INFO:teuthology.orchestra.run.smithi092.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-02T14:20:22.387 INFO:teuthology.orchestra.run.smithi092.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-02T14:20:22.387 INFO:teuthology.orchestra.run.smithi092.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-04-02T14:20:22.387 INFO:teuthology.orchestra.run.smithi092.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-04-02T14:20:22.387 INFO:teuthology.orchestra.run.smithi092.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-04-02T14:20:22.387 INFO:teuthology.orchestra.run.smithi092.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-02T14:20:22.387 INFO:teuthology.orchestra.run.smithi092.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-02T14:20:22.387 INFO:teuthology.orchestra.run.smithi092.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-02T14:20:22.387 INFO:teuthology.orchestra.run.smithi092.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-02T14:20:22.388 INFO:teuthology.orchestra.run.smithi092.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-02T14:20:22.388 INFO:teuthology.orchestra.run.smithi092.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-02T14:20:22.388 INFO:teuthology.orchestra.run.smithi092.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-02T14:20:22.388 INFO:teuthology.orchestra.run.smithi092.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-02T14:20:22.388 INFO:teuthology.orchestra.run.smithi092.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-04-02T14:20:22.388 INFO:teuthology.orchestra.run.smithi092.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-04-02T14:20:22.388 INFO:teuthology.orchestra.run.smithi092.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-02T14:20:22.388 INFO:teuthology.orchestra.run.smithi092.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-02T14:20:22.388 INFO:teuthology.orchestra.run.smithi092.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-04-02T14:20:22.388 INFO:teuthology.orchestra.run.smithi092.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-04-02T14:20:22.388 INFO:teuthology.orchestra.run.smithi092.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-02T14:20:22.388 INFO:teuthology.orchestra.run.smithi092.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-02T14:20:22.388 INFO:teuthology.orchestra.run.smithi092.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-04-02T14:20:22.388 INFO:teuthology.orchestra.run.smithi092.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-04-02T14:20:22.388 INFO:teuthology.orchestra.run.smithi092.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-04-02T14:20:22.388 INFO:teuthology.orchestra.run.smithi092.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-04-02T14:20:22.389 INFO:teuthology.orchestra.run.smithi092.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-02T14:20:22.389 INFO:teuthology.orchestra.run.smithi092.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-04-02T14:20:22.389 INFO:teuthology.orchestra.run.smithi092.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-02T14:20:22.389 INFO:teuthology.orchestra.run.smithi092.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-04-02T14:20:22.389 INFO:teuthology.orchestra.run.smithi092.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-04-02T14:20:22.389 INFO:teuthology.orchestra.run.smithi092.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-02T14:20:22.389 INFO:teuthology.orchestra.run.smithi092.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-04-02T14:20:22.389 INFO:teuthology.orchestra.run.smithi092.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-04-02T14:20:22.389 INFO:teuthology.orchestra.run.smithi092.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-04-02T14:20:22.389 INFO:teuthology.orchestra.run.smithi092.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-02T14:20:22.389 INFO:teuthology.orchestra.run.smithi092.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-04-02T14:20:22.389 INFO:teuthology.orchestra.run.smithi092.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-02T14:20:22.389 INFO:teuthology.orchestra.run.smithi092.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-02T14:20:22.389 INFO:teuthology.orchestra.run.smithi092.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-04-02T14:20:22.389 INFO:teuthology.orchestra.run.smithi092.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-02T14:20:22.390 INFO:teuthology.orchestra.run.smithi092.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-02T14:20:22.390 INFO:teuthology.orchestra.run.smithi092.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-04-02T14:20:22.390 INFO:teuthology.orchestra.run.smithi092.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-02T14:20:22.390 INFO:teuthology.orchestra.run.smithi092.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-04-02T14:20:22.390 INFO:teuthology.orchestra.run.smithi092.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-02T14:20:22.390 INFO:teuthology.orchestra.run.smithi092.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-04-02T14:20:22.390 INFO:teuthology.orchestra.run.smithi092.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-04-02T14:20:22.390 INFO:teuthology.orchestra.run.smithi092.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-02T14:20:22.390 INFO:teuthology.orchestra.run.smithi092.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-02T14:20:22.390 INFO:teuthology.orchestra.run.smithi092.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-02T14:20:22.390 INFO:teuthology.orchestra.run.smithi092.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-02T14:20:22.390 INFO:teuthology.orchestra.run.smithi092.stdout: qatlib x86_64 24.02.0-1.el9 CentOS-AppStream 220 k 2024-04-02T14:20:22.390 INFO:teuthology.orchestra.run.smithi092.stdout: qatzip-libs x86_64 1.2.0-1.el9 CentOS-AppStream 46 k 2024-04-02T14:20:22.390 INFO:teuthology.orchestra.run.smithi092.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-02T14:20:22.391 INFO:teuthology.orchestra.run.smithi092.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-04-02T14:20:22.391 INFO:teuthology.orchestra.run.smithi092.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-02T14:20:22.391 INFO:teuthology.orchestra.run.smithi092.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-04-02T14:20:22.391 INFO:teuthology.orchestra.run.smithi092.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-04-02T14:20:22.391 INFO:teuthology.orchestra.run.smithi092.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-04-02T14:20:22.391 INFO:teuthology.orchestra.run.smithi092.stdout:Installing weak dependencies: 2024-04-02T14:20:22.391 INFO:teuthology.orchestra.run.smithi092.stdout: qatlib-service x86_64 24.02.0-1.el9 CentOS-AppStream 35 k 2024-04-02T14:20:22.391 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-04-02T14:20:22.391 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction Summary 2024-04-02T14:20:22.391 INFO:teuthology.orchestra.run.smithi092.stdout:============================================================================================ 2024-04-02T14:20:22.391 INFO:teuthology.orchestra.run.smithi092.stdout:Install 150 Packages 2024-04-02T14:20:22.391 INFO:teuthology.orchestra.run.smithi092.stdout:Upgrade 24 Packages 2024-04-02T14:20:22.391 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-04-02T14:20:22.392 INFO:teuthology.orchestra.run.smithi092.stdout:Total download size: 226 M 2024-04-02T14:20:22.392 INFO:teuthology.orchestra.run.smithi092.stdout:Downloading Packages: 2024-04-02T14:20:23.476 INFO:teuthology.orchestra.run.smithi092.stdout:(1/174): ceph-19.0.0-2609.g191d963a.el9.x86_64. 21 kB/s | 6.6 kB 00:00 2024-04-02T14:20:23.758 INFO:teuthology.orchestra.run.smithi090.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-04-02T14:20:23.912 INFO:teuthology.orchestra.run.smithi177.stdout:CentOS Stream 9 - CRB 10 MB/s | 6.1 MB 00:00 2024-04-02T14:20:24.143 INFO:teuthology.orchestra.run.smithi092.stdout:(2/174): ceph-fuse-19.0.0-2609.g191d963a.el9.x8 1.3 MB/s | 890 kB 00:00 2024-04-02T14:20:24.260 INFO:teuthology.orchestra.run.smithi092.stdout:(3/174): ceph-base-19.0.0-2609.g191d963a.el9.x8 4.9 MB/s | 5.4 MB 00:01 2024-04-02T14:20:24.318 INFO:teuthology.orchestra.run.smithi092.stdout:(4/174): ceph-immutable-object-cache-19.0.0-260 826 kB/s | 145 kB 00:00 2024-04-02T14:20:24.535 INFO:teuthology.orchestra.run.smithi092.stdout:(5/174): ceph-mds-19.0.0-2609.g191d963a.el9.x86 8.6 MB/s | 2.4 MB 00:00 2024-04-02T14:20:25.144 INFO:teuthology.orchestra.run.smithi092.stdout:(6/174): ceph-common-19.0.0-2609.g191d963a.el9. 10 MB/s | 21 MB 00:01 2024-04-02T14:20:25.210 INFO:teuthology.orchestra.run.smithi092.stdout:(7/174): ceph-mon-19.0.0-2609.g191d963a.el9.x86 7.0 MB/s | 4.7 MB 00:00 2024-04-02T14:20:25.244 INFO:teuthology.orchestra.run.smithi092.stdout:(8/174): ceph-mgr-19.0.0-2609.g191d963a.el9.x86 1.6 MB/s | 1.5 MB 00:00 2024-04-02T14:20:25.361 INFO:teuthology.orchestra.run.smithi092.stdout:(9/174): ceph-selinux-19.0.0-2609.g191d963a.el9 215 kB/s | 25 kB 00:00 2024-04-02T14:20:25.814 INFO:teuthology.orchestra.run.smithi177.stdout:CentOS Stream 9 - Extras packages 21 kB/s | 16 kB 00:00 2024-04-02T14:20:26.186 INFO:teuthology.orchestra.run.smithi092.stdout:(10/174): ceph-osd-19.0.0-2609.g191d963a.el9.x8 16 MB/s | 17 MB 00:01 2024-04-02T14:20:26.253 INFO:teuthology.orchestra.run.smithi092.stdout:(11/174): cephfs-mirror-19.0.0-2609.g191d963a.e 3.3 MB/s | 225 kB 00:00 2024-04-02T14:20:26.303 INFO:teuthology.orchestra.run.smithi092.stdout:(12/174): cephfs-shell-19.0.0-2609.g191d963a.el 544 kB/s | 27 kB 00:00 2024-04-02T14:20:26.337 INFO:teuthology.orchestra.run.smithi092.stdout:(13/174): libcephfs-devel-19.0.0-2609.g191d963a 948 kB/s | 32 kB 00:00 2024-04-02T14:20:26.446 INFO:teuthology.orchestra.run.smithi092.stdout:(14/174): ceph-radosgw-19.0.0-2609.g191d963a.el 8.0 MB/s | 9.8 MB 00:01 2024-04-02T14:20:26.487 INFO:teuthology.orchestra.run.smithi092.stdout:(15/174): libcephfs2-19.0.0-2609.g191d963a.el9. 4.9 MB/s | 747 kB 00:00 2024-04-02T14:20:26.514 INFO:teuthology.orchestra.run.smithi092.stdout:(16/174): libcephsqlite-19.0.0-2609.g191d963a.e 2.3 MB/s | 164 kB 00:00 2024-04-02T14:20:26.538 INFO:teuthology.orchestra.run.smithi092.stdout:(17/174): librados-devel-19.0.0-2609.g191d963a. 2.5 MB/s | 127 kB 00:00 2024-04-02T14:20:26.556 INFO:teuthology.orchestra.run.smithi090.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-02T14:20:26.558 INFO:teuthology.orchestra.run.smithi090.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-02T14:20:26.579 INFO:teuthology.orchestra.run.smithi090.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-04-02T14:20:26.580 INFO:teuthology.orchestra.run.smithi092.stdout:(18/174): libradosstriper1-19.0.0-2609.g191d963 7.5 MB/s | 500 kB 00:00 2024-04-02T14:20:26.580 INFO:teuthology.orchestra.run.smithi090.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-02T14:20:26.581 INFO:teuthology.orchestra.run.smithi090.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-04-02T14:20:26.583 INFO:teuthology.orchestra.run.smithi090.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-04-02T14:20:26.585 INFO:teuthology.orchestra.run.smithi090.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-02T14:20:26.589 INFO:teuthology.orchestra.run.smithi090.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-04-02T14:20:26.590 INFO:teuthology.orchestra.run.smithi090.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-04-02T14:20:26.655 INFO:teuthology.orchestra.run.smithi092.stdout:(19/174): python3-ceph-argparse-19.0.0-2609.g19 602 kB/s | 45 kB 00:00 2024-04-02T14:20:26.676 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-04-02T14:20:26.704 INFO:teuthology.orchestra.run.smithi090.stdout:============================================================================================ 2024-04-02T14:20:26.704 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repository Size 2024-04-02T14:20:26.704 INFO:teuthology.orchestra.run.smithi090.stdout:============================================================================================ 2024-04-02T14:20:26.704 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2024-04-02T14:20:26.705 INFO:teuthology.orchestra.run.smithi090.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-04-02T14:20:26.705 INFO:teuthology.orchestra.run.smithi090.stdout: ceph x86_64 2:19.0.0-2609.g191d963a.el9 ceph 6.6 k 2024-04-02T14:20:26.705 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-base x86_64 2:19.0.0-2609.g191d963a.el9 ceph 5.4 M 2024-04-02T14:20:26.705 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-fuse x86_64 2:19.0.0-2609.g191d963a.el9 ceph 890 k 2024-04-02T14:20:26.705 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2609.g191d963a.el9 ceph 145 k 2024-04-02T14:20:26.705 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr x86_64 2:19.0.0-2609.g191d963a.el9 ceph 1.5 M 2024-04-02T14:20:26.705 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2609.g191d963a.el9 ceph-noarch 141 k 2024-04-02T14:20:26.705 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2609.g191d963a.el9 ceph-noarch 3.5 M 2024-04-02T14:20:26.705 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2609.g191d963a.el9 ceph-noarch 7.4 M 2024-04-02T14:20:26.705 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-rook noarch 2:19.0.0-2609.g191d963a.el9 ceph-noarch 49 k 2024-04-02T14:20:26.705 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-radosgw x86_64 2:19.0.0-2609.g191d963a.el9 ceph 9.8 M 2024-04-02T14:20:26.705 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-test x86_64 2:19.0.0-2609.g191d963a.el9 ceph 48 M 2024-04-02T14:20:26.705 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-volume noarch 2:19.0.0-2609.g191d963a.el9 ceph-noarch 264 k 2024-04-02T14:20:26.705 INFO:teuthology.orchestra.run.smithi090.stdout: cephadm noarch 2:19.0.0-2609.g191d963a.el9 ceph-noarch 762 k 2024-04-02T14:20:26.706 INFO:teuthology.orchestra.run.smithi090.stdout: cephfs-mirror x86_64 2:19.0.0-2609.g191d963a.el9 ceph 225 k 2024-04-02T14:20:26.706 INFO:teuthology.orchestra.run.smithi090.stdout: cephfs-shell x86_64 2:19.0.0-2609.g191d963a.el9 ceph 27 k 2024-04-02T14:20:26.706 INFO:teuthology.orchestra.run.smithi090.stdout: cephfs-top noarch 2:19.0.0-2609.g191d963a.el9 ceph-noarch 18 k 2024-04-02T14:20:26.706 INFO:teuthology.orchestra.run.smithi090.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-02T14:20:26.706 INFO:teuthology.orchestra.run.smithi090.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-04-02T14:20:26.706 INFO:teuthology.orchestra.run.smithi090.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-04-02T14:20:26.706 INFO:teuthology.orchestra.run.smithi090.stdout: libcephfs-devel x86_64 2:19.0.0-2609.g191d963a.el9 ceph 32 k 2024-04-02T14:20:26.706 INFO:teuthology.orchestra.run.smithi090.stdout: libcephfs2 x86_64 2:19.0.0-2609.g191d963a.el9 ceph 747 k 2024-04-02T14:20:26.706 INFO:teuthology.orchestra.run.smithi090.stdout: librados-devel x86_64 2:19.0.0-2609.g191d963a.el9 ceph 127 k 2024-04-02T14:20:26.706 INFO:teuthology.orchestra.run.smithi090.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-04-02T14:20:26.706 INFO:teuthology.orchestra.run.smithi090.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-02T14:20:26.706 INFO:teuthology.orchestra.run.smithi090.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-04-02T14:20:26.706 INFO:teuthology.orchestra.run.smithi090.stdout: postgresql x86_64 13.14-1.el9 CentOS-AppStream 1.5 M 2024-04-02T14:20:26.707 INFO:teuthology.orchestra.run.smithi090.stdout: postgresql-contrib x86_64 13.14-1.el9 CentOS-AppStream 817 k 2024-04-02T14:20:26.707 INFO:teuthology.orchestra.run.smithi090.stdout: postgresql-server x86_64 13.14-1.el9 CentOS-AppStream 5.7 M 2024-04-02T14:20:26.707 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cephfs x86_64 2:19.0.0-2609.g191d963a.el9 ceph 161 k 2024-04-02T14:20:26.707 INFO:teuthology.orchestra.run.smithi090.stdout: python3-rados x86_64 2:19.0.0-2609.g191d963a.el9 ceph 320 k 2024-04-02T14:20:26.707 INFO:teuthology.orchestra.run.smithi090.stdout: python3-rbd x86_64 2:19.0.0-2609.g191d963a.el9 ceph 299 k 2024-04-02T14:20:26.707 INFO:teuthology.orchestra.run.smithi090.stdout: python3-rgw x86_64 2:19.0.0-2609.g191d963a.el9 ceph 99 k 2024-04-02T14:20:26.707 INFO:teuthology.orchestra.run.smithi090.stdout: rbd-fuse x86_64 2:19.0.0-2609.g191d963a.el9 ceph 85 k 2024-04-02T14:20:26.707 INFO:teuthology.orchestra.run.smithi090.stdout: rbd-mirror x86_64 2:19.0.0-2609.g191d963a.el9 ceph 3.1 M 2024-04-02T14:20:26.707 INFO:teuthology.orchestra.run.smithi090.stdout: rbd-nbd x86_64 2:19.0.0-2609.g191d963a.el9 ceph 170 k 2024-04-02T14:20:26.707 INFO:teuthology.orchestra.run.smithi090.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-02T14:20:26.707 INFO:teuthology.orchestra.run.smithi090.stdout:Upgrading: 2024-04-02T14:20:26.707 INFO:teuthology.orchestra.run.smithi090.stdout: NetworkManager x86_64 1:1.47.2-1.el9 CentOS-BaseOS 2.3 M 2024-04-02T14:20:26.707 INFO:teuthology.orchestra.run.smithi090.stdout: NetworkManager-libnm x86_64 1:1.47.2-1.el9 CentOS-BaseOS 1.8 M 2024-04-02T14:20:26.707 INFO:teuthology.orchestra.run.smithi090.stdout: NetworkManager-team x86_64 1:1.47.2-1.el9 CentOS-BaseOS 41 k 2024-04-02T14:20:26.708 INFO:teuthology.orchestra.run.smithi090.stdout: NetworkManager-tui x86_64 1:1.47.2-1.el9 CentOS-BaseOS 247 k 2024-04-02T14:20:26.708 INFO:teuthology.orchestra.run.smithi090.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-04-02T14:20:26.708 INFO:teuthology.orchestra.run.smithi090.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-04-02T14:20:26.708 INFO:teuthology.orchestra.run.smithi090.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-04-02T14:20:26.708 INFO:teuthology.orchestra.run.smithi090.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-04-02T14:20:26.708 INFO:teuthology.orchestra.run.smithi090.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-04-02T14:20:26.708 INFO:teuthology.orchestra.run.smithi090.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-04-02T14:20:26.708 INFO:teuthology.orchestra.run.smithi090.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-04-02T14:20:26.708 INFO:teuthology.orchestra.run.smithi090.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-04-02T14:20:26.708 INFO:teuthology.orchestra.run.smithi090.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-04-02T14:20:26.708 INFO:teuthology.orchestra.run.smithi090.stdout: librados2 x86_64 2:19.0.0-2609.g191d963a.el9 ceph 3.4 M 2024-04-02T14:20:26.708 INFO:teuthology.orchestra.run.smithi090.stdout: librbd1 x86_64 2:19.0.0-2609.g191d963a.el9 ceph 3.1 M 2024-04-02T14:20:26.708 INFO:teuthology.orchestra.run.smithi090.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-04-02T14:20:26.708 INFO:teuthology.orchestra.run.smithi090.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-04-02T14:20:26.709 INFO:teuthology.orchestra.run.smithi090.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-04-02T14:20:26.709 INFO:teuthology.orchestra.run.smithi090.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-04-02T14:20:26.709 INFO:teuthology.orchestra.run.smithi090.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-04-02T14:20:26.709 INFO:teuthology.orchestra.run.smithi090.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-04-02T14:20:26.709 INFO:teuthology.orchestra.run.smithi090.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-04-02T14:20:26.709 INFO:teuthology.orchestra.run.smithi090.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-04-02T14:20:26.709 INFO:teuthology.orchestra.run.smithi090.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-04-02T14:20:26.709 INFO:teuthology.orchestra.run.smithi090.stdout:Installing dependencies: 2024-04-02T14:20:26.709 INFO:teuthology.orchestra.run.smithi090.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-02T14:20:26.709 INFO:teuthology.orchestra.run.smithi090.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-04-02T14:20:26.709 INFO:teuthology.orchestra.run.smithi090.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-04-02T14:20:26.709 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-common x86_64 2:19.0.0-2609.g191d963a.el9 ceph 21 M 2024-04-02T14:20:26.709 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2609.g191d963a.el9 ceph-noarch 34 k 2024-04-02T14:20:26.709 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mds x86_64 2:19.0.0-2609.g191d963a.el9 ceph 2.4 M 2024-04-02T14:20:26.710 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2609.g191d963a.el9 ceph-noarch 249 k 2024-04-02T14:20:26.710 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mon x86_64 2:19.0.0-2609.g191d963a.el9 ceph 4.7 M 2024-04-02T14:20:26.710 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-osd x86_64 2:19.0.0-2609.g191d963a.el9 ceph 17 M 2024-04-02T14:20:26.710 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2609.g191d963a.el9 ceph-noarch 16 k 2024-04-02T14:20:26.710 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-selinux x86_64 2:19.0.0-2609.g191d963a.el9 ceph 25 k 2024-04-02T14:20:26.710 INFO:teuthology.orchestra.run.smithi090.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-04-02T14:20:26.710 INFO:teuthology.orchestra.run.smithi090.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-04-02T14:20:26.710 INFO:teuthology.orchestra.run.smithi090.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-04-02T14:20:26.710 INFO:teuthology.orchestra.run.smithi090.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-04-02T14:20:26.710 INFO:teuthology.orchestra.run.smithi090.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-02T14:20:26.710 INFO:teuthology.orchestra.run.smithi090.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-02T14:20:26.710 INFO:teuthology.orchestra.run.smithi090.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-04-02T14:20:26.710 INFO:teuthology.orchestra.run.smithi090.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-02T14:20:26.710 INFO:teuthology.orchestra.run.smithi090.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-02T14:20:26.711 INFO:teuthology.orchestra.run.smithi090.stdout: libcephsqlite x86_64 2:19.0.0-2609.g191d963a.el9 ceph 164 k 2024-04-02T14:20:26.711 INFO:teuthology.orchestra.run.smithi090.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-02T14:20:26.711 INFO:teuthology.orchestra.run.smithi090.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-02T14:20:26.711 INFO:teuthology.orchestra.run.smithi090.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-04-02T14:20:26.711 INFO:teuthology.orchestra.run.smithi090.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-02T14:20:26.711 INFO:teuthology.orchestra.run.smithi090.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-04-02T14:20:26.711 INFO:teuthology.orchestra.run.smithi090.stdout: libradosstriper1 x86_64 2:19.0.0-2609.g191d963a.el9 ceph 500 k 2024-04-02T14:20:26.711 INFO:teuthology.orchestra.run.smithi090.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-04-02T14:20:26.711 INFO:teuthology.orchestra.run.smithi090.stdout: librgw2 x86_64 2:19.0.0-2609.g191d963a.el9 ceph 5.1 M 2024-04-02T14:20:26.711 INFO:teuthology.orchestra.run.smithi090.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-02T14:20:26.711 INFO:teuthology.orchestra.run.smithi090.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-04-02T14:20:26.711 INFO:teuthology.orchestra.run.smithi090.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-04-02T14:20:26.711 INFO:teuthology.orchestra.run.smithi090.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-04-02T14:20:26.711 INFO:teuthology.orchestra.run.smithi090.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-02T14:20:26.712 INFO:teuthology.orchestra.run.smithi090.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-02T14:20:26.712 INFO:teuthology.orchestra.run.smithi090.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-04-02T14:20:26.712 INFO:teuthology.orchestra.run.smithi090.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-04-02T14:20:26.712 INFO:teuthology.orchestra.run.smithi090.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-04-02T14:20:26.712 INFO:teuthology.orchestra.run.smithi090.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-02T14:20:26.712 INFO:teuthology.orchestra.run.smithi090.stdout: postgresql-private-libs x86_64 13.14-1.el9 CentOS-AppStream 132 k 2024-04-02T14:20:26.712 INFO:teuthology.orchestra.run.smithi090.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-04-02T14:20:26.712 INFO:teuthology.orchestra.run.smithi090.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-02T14:20:26.712 INFO:teuthology.orchestra.run.smithi090.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-02T14:20:26.712 INFO:teuthology.orchestra.run.smithi090.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-04-02T14:20:26.712 INFO:teuthology.orchestra.run.smithi090.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-04-02T14:20:26.712 INFO:teuthology.orchestra.run.smithi090.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-02T14:20:26.712 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-02T14:20:26.712 INFO:teuthology.orchestra.run.smithi090.stdout: python3-ceph-argparse x86_64 2:19.0.0-2609.g191d963a.el9 ceph 45 k 2024-04-02T14:20:26.713 INFO:teuthology.orchestra.run.smithi090.stdout: python3-ceph-common x86_64 2:19.0.0-2609.g191d963a.el9 ceph 134 k 2024-04-02T14:20:26.713 INFO:teuthology.orchestra.run.smithi090.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-02T14:20:26.713 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-04-02T14:20:26.713 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-02T14:20:26.713 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-02T14:20:26.713 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-02T14:20:26.713 INFO:teuthology.orchestra.run.smithi090.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-02T14:20:26.713 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-04-02T14:20:26.713 INFO:teuthology.orchestra.run.smithi090.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-04-02T14:20:26.713 INFO:teuthology.orchestra.run.smithi090.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-04-02T14:20:26.713 INFO:teuthology.orchestra.run.smithi090.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-02T14:20:26.713 INFO:teuthology.orchestra.run.smithi090.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-02T14:20:26.713 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-02T14:20:26.713 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-02T14:20:26.714 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-02T14:20:26.714 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-02T14:20:26.714 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-02T14:20:26.714 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-02T14:20:26.714 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-04-02T14:20:26.714 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-04-02T14:20:26.714 INFO:teuthology.orchestra.run.smithi090.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-02T14:20:26.714 INFO:teuthology.orchestra.run.smithi090.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-02T14:20:26.714 INFO:teuthology.orchestra.run.smithi090.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-04-02T14:20:26.714 INFO:teuthology.orchestra.run.smithi090.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-04-02T14:20:26.714 INFO:teuthology.orchestra.run.smithi090.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-02T14:20:26.714 INFO:teuthology.orchestra.run.smithi090.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-02T14:20:26.714 INFO:teuthology.orchestra.run.smithi090.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-04-02T14:20:26.714 INFO:teuthology.orchestra.run.smithi090.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-04-02T14:20:26.714 INFO:teuthology.orchestra.run.smithi090.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-04-02T14:20:26.714 INFO:teuthology.orchestra.run.smithi090.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-04-02T14:20:26.715 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-02T14:20:26.715 INFO:teuthology.orchestra.run.smithi090.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-04-02T14:20:26.715 INFO:teuthology.orchestra.run.smithi090.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-02T14:20:26.715 INFO:teuthology.orchestra.run.smithi090.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-04-02T14:20:26.715 INFO:teuthology.orchestra.run.smithi090.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-04-02T14:20:26.715 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-02T14:20:26.715 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-04-02T14:20:26.715 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-04-02T14:20:26.715 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-04-02T14:20:26.715 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-02T14:20:26.715 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-04-02T14:20:26.715 INFO:teuthology.orchestra.run.smithi090.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-02T14:20:26.715 INFO:teuthology.orchestra.run.smithi090.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-02T14:20:26.715 INFO:teuthology.orchestra.run.smithi090.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-04-02T14:20:26.715 INFO:teuthology.orchestra.run.smithi090.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-02T14:20:26.716 INFO:teuthology.orchestra.run.smithi090.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-02T14:20:26.716 INFO:teuthology.orchestra.run.smithi090.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-04-02T14:20:26.716 INFO:teuthology.orchestra.run.smithi090.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-02T14:20:26.716 INFO:teuthology.orchestra.run.smithi090.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-04-02T14:20:26.716 INFO:teuthology.orchestra.run.smithi090.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-02T14:20:26.716 INFO:teuthology.orchestra.run.smithi090.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-04-02T14:20:26.716 INFO:teuthology.orchestra.run.smithi090.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-04-02T14:20:26.716 INFO:teuthology.orchestra.run.smithi090.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-02T14:20:26.716 INFO:teuthology.orchestra.run.smithi090.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-02T14:20:26.716 INFO:teuthology.orchestra.run.smithi090.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-02T14:20:26.716 INFO:teuthology.orchestra.run.smithi090.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-02T14:20:26.716 INFO:teuthology.orchestra.run.smithi090.stdout: qatlib x86_64 24.02.0-1.el9 CentOS-AppStream 220 k 2024-04-02T14:20:26.716 INFO:teuthology.orchestra.run.smithi090.stdout: qatzip-libs x86_64 1.2.0-1.el9 CentOS-AppStream 46 k 2024-04-02T14:20:26.716 INFO:teuthology.orchestra.run.smithi090.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-02T14:20:26.716 INFO:teuthology.orchestra.run.smithi090.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-04-02T14:20:26.716 INFO:teuthology.orchestra.run.smithi090.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-02T14:20:26.717 INFO:teuthology.orchestra.run.smithi090.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-04-02T14:20:26.717 INFO:teuthology.orchestra.run.smithi090.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-04-02T14:20:26.717 INFO:teuthology.orchestra.run.smithi090.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-04-02T14:20:26.717 INFO:teuthology.orchestra.run.smithi090.stdout:Installing weak dependencies: 2024-04-02T14:20:26.717 INFO:teuthology.orchestra.run.smithi090.stdout: qatlib-service x86_64 24.02.0-1.el9 CentOS-AppStream 35 k 2024-04-02T14:20:26.717 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-02T14:20:26.717 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-04-02T14:20:26.717 INFO:teuthology.orchestra.run.smithi090.stdout:============================================================================================ 2024-04-02T14:20:26.717 INFO:teuthology.orchestra.run.smithi090.stdout:Install 150 Packages 2024-04-02T14:20:26.717 INFO:teuthology.orchestra.run.smithi090.stdout:Upgrade 24 Packages 2024-04-02T14:20:26.717 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-02T14:20:26.718 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 226 M 2024-04-02T14:20:26.718 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2024-04-02T14:20:26.739 INFO:teuthology.orchestra.run.smithi092.stdout:(20/174): python3-ceph-common-19.0.0-2609.g191d 1.6 MB/s | 134 kB 00:00 2024-04-02T14:20:26.848 INFO:teuthology.orchestra.run.smithi092.stdout:(21/174): librgw2-19.0.0-2609.g191d963a.el9.x86 17 MB/s | 5.1 MB 00:00 2024-04-02T14:20:26.864 INFO:teuthology.orchestra.run.smithi092.stdout:(22/174): python3-cephfs-19.0.0-2609.g191d963a. 1.3 MB/s | 161 kB 00:00 2024-04-02T14:20:26.898 INFO:teuthology.orchestra.run.smithi092.stdout:(23/174): python3-rados-19.0.0-2609.g191d963a.e 6.3 MB/s | 320 kB 00:00 2024-04-02T14:20:26.923 INFO:teuthology.orchestra.run.smithi092.stdout:(24/174): python3-rbd-19.0.0-2609.g191d963a.el9 5.0 MB/s | 299 kB 00:00 2024-04-02T14:20:26.949 INFO:teuthology.orchestra.run.smithi092.stdout:(25/174): python3-rgw-19.0.0-2609.g191d963a.el9 1.9 MB/s | 99 kB 00:00 2024-04-02T14:20:26.974 INFO:teuthology.orchestra.run.smithi092.stdout:(26/174): rbd-fuse-19.0.0-2609.g191d963a.el9.x8 1.7 MB/s | 85 kB 00:00 2024-04-02T14:20:27.108 INFO:teuthology.orchestra.run.smithi092.stdout:(27/174): rbd-nbd-19.0.0-2609.g191d963a.el9.x86 1.2 MB/s | 170 kB 00:00 2024-04-02T14:20:27.175 INFO:teuthology.orchestra.run.smithi092.stdout:(28/174): rbd-mirror-19.0.0-2609.g191d963a.el9. 14 MB/s | 3.1 MB 00:00 2024-04-02T14:20:27.200 INFO:teuthology.orchestra.run.smithi092.stdout:(29/174): ceph-grafana-dashboards-19.0.0-2609.g 370 kB/s | 34 kB 00:00 2024-04-02T14:20:27.225 INFO:teuthology.orchestra.run.smithi092.stdout:(30/174): ceph-mgr-cephadm-19.0.0-2609.g191d963 2.8 MB/s | 141 kB 00:00 2024-04-02T14:20:27.634 INFO:teuthology.orchestra.run.smithi090.stdout:(1/174): ceph-19.0.0-2609.g191d963a.el9.x86_64. 16 kB/s | 6.6 kB 00:00 2024-04-02T14:20:27.725 INFO:teuthology.orchestra.run.smithi092.stdout:(31/174): ceph-mgr-dashboard-19.0.0-2609.g191d9 6.7 MB/s | 3.5 MB 00:00 2024-04-02T14:20:27.809 INFO:teuthology.orchestra.run.smithi092.stdout:(32/174): ceph-mgr-diskprediction-local-19.0.0- 13 MB/s | 7.4 MB 00:00 2024-04-02T14:20:27.834 INFO:teuthology.orchestra.run.smithi092.stdout:(33/174): ceph-mgr-modules-core-19.0.0-2609.g19 2.2 MB/s | 249 kB 00:00 2024-04-02T14:20:27.851 INFO:teuthology.orchestra.run.smithi092.stdout:(34/174): ceph-mgr-rook-19.0.0-2609.g191d963a.e 1.2 MB/s | 49 kB 00:00 2024-04-02T14:20:27.876 INFO:teuthology.orchestra.run.smithi092.stdout:(35/174): ceph-prometheus-alerts-19.0.0-2609.g1 391 kB/s | 16 kB 00:00 2024-04-02T14:20:27.901 INFO:teuthology.orchestra.run.smithi090.stdout:(2/174): ceph-fuse-19.0.0-2609.g191d963a.el9.x8 3.3 MB/s | 890 kB 00:00 2024-04-02T14:20:27.910 INFO:teuthology.orchestra.run.smithi092.stdout:(36/174): ceph-volume-19.0.0-2609.g191d963a.el9 4.4 MB/s | 264 kB 00:00 2024-04-02T14:20:27.952 INFO:teuthology.orchestra.run.smithi090.stdout:(3/174): ceph-immutable-object-cache-19.0.0-260 2.8 MB/s | 145 kB 00:00 2024-04-02T14:20:27.960 INFO:teuthology.orchestra.run.smithi092.stdout:(37/174): cephadm-19.0.0-2609.g191d963a.el9.noa 8.9 MB/s | 762 kB 00:00 2024-04-02T14:20:27.977 INFO:teuthology.orchestra.run.smithi092.stdout:(38/174): cephfs-top-19.0.0-2609.g191d963a.el9. 272 kB/s | 18 kB 00:00 2024-04-02T14:20:28.235 INFO:teuthology.orchestra.run.smithi090.stdout:(4/174): ceph-mds-19.0.0-2609.g191d963a.el9.x86 8.4 MB/s | 2.4 MB 00:00 2024-04-02T14:20:28.294 INFO:teuthology.orchestra.run.smithi092.stdout:(39/174): automake-1.16.2-8.el9.noarch.rpm 1.9 MB/s | 662 kB 00:00 2024-04-02T14:20:28.386 INFO:teuthology.orchestra.run.smithi092.stdout:(40/174): boost-program-options-1.75.0-8.el9.x8 1.1 MB/s | 106 kB 00:00 2024-04-02T14:20:28.394 INFO:teuthology.orchestra.run.smithi090.stdout:(5/174): ceph-mgr-19.0.0-2609.g191d963a.el9.x86 9.5 MB/s | 1.5 MB 00:00 2024-04-02T14:20:28.420 INFO:teuthology.orchestra.run.smithi092.stdout:(41/174): bison-3.7.4-5.el9.x86_64.rpm 2.0 MB/s | 920 kB 00:00 2024-04-02T14:20:28.462 INFO:teuthology.orchestra.run.smithi092.stdout:(42/174): flex-2.6.4-9.el9.x86_64.rpm 4.0 MB/s | 307 kB 00:00 2024-04-02T14:20:28.479 INFO:teuthology.orchestra.run.smithi092.stdout:(43/174): flexiblas-3.0.4-8.el9.x86_64.rpm 539 kB/s | 32 kB 00:00 2024-04-02T14:20:28.646 INFO:teuthology.orchestra.run.smithi092.stdout:(44/174): flexiblas-openblas-openmp-3.0.4-8.el9 103 kB/s | 17 kB 00:00 2024-04-02T14:20:28.746 INFO:teuthology.orchestra.run.smithi092.stdout:(45/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 11 MB/s | 3.0 MB 00:00 2024-04-02T14:20:28.763 INFO:teuthology.orchestra.run.smithi092.stdout:(46/174): libaio-devel-0.3.111-13.el9.x86_64.rp 87 kB/s | 10 kB 00:00 2024-04-02T14:20:28.803 INFO:teuthology.orchestra.run.smithi090.stdout:(6/174): ceph-mon-19.0.0-2609.g191d963a.el9.x86 12 MB/s | 4.7 MB 00:00 2024-04-02T14:20:28.813 INFO:teuthology.orchestra.run.smithi092.stdout:(47/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.1 MB/s | 159 kB 00:00 2024-04-02T14:20:28.839 INFO:teuthology.orchestra.run.smithi092.stdout:(48/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 587 kB/s | 44 kB 00:00 2024-04-02T14:20:28.881 INFO:teuthology.orchestra.run.smithi092.stdout:(49/174): librdkafka-1.6.1-102.el9.x86_64.rpm 9.6 MB/s | 661 kB 00:00 2024-04-02T14:20:28.906 INFO:teuthology.orchestra.run.smithi092.stdout:(50/174): libtool-2.4.6-45.el9.x86_64.rpm 8.5 MB/s | 578 kB 00:00 2024-04-02T14:20:28.923 INFO:teuthology.orchestra.run.smithi092.stdout:(51/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 1.1 MB/s | 46 kB 00:00 2024-04-02T14:20:28.948 INFO:teuthology.orchestra.run.smithi092.stdout:(52/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 6.6 MB/s | 282 kB 00:00 2024-04-02T14:20:28.973 INFO:teuthology.orchestra.run.smithi092.stdout:(53/174): lua-5.4.4-4.el9.x86_64.rpm 3.6 MB/s | 187 kB 00:00 2024-04-02T14:20:28.999 INFO:teuthology.orchestra.run.smithi092.stdout:(54/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 4.2 MB/s | 217 kB 00:00 2024-04-02T14:20:29.022 INFO:teuthology.orchestra.run.smithi177.stdout:Extra Packages for Enterprise Linux 6.7 MB/s | 21 MB 00:03 2024-04-02T14:20:29.024 INFO:teuthology.orchestra.run.smithi092.stdout:(55/174): openblas-0.3.21-2.el9.x86_64.rpm 683 kB/s | 34 kB 00:00 2024-04-02T14:20:29.158 INFO:teuthology.orchestra.run.smithi092.stdout:(56/174): openblas-openmp-0.3.21-2.el9.x86_64.r 30 MB/s | 4.7 MB 00:00 2024-04-02T14:20:29.200 INFO:teuthology.orchestra.run.smithi092.stdout:(57/174): postgresql-13.14-1.el9.x86_64.rpm 8.8 MB/s | 1.5 MB 00:00 2024-04-02T14:20:29.242 INFO:teuthology.orchestra.run.smithi092.stdout:(58/174): postgresql-contrib-13.14-1.el9.x86_64 9.8 MB/s | 817 kB 00:00 2024-04-02T14:20:29.267 INFO:teuthology.orchestra.run.smithi092.stdout:(59/174): postgresql-private-libs-13.14-1.el9.x 2.0 MB/s | 132 kB 00:00 2024-04-02T14:20:29.467 INFO:teuthology.orchestra.run.smithi092.stdout:(60/174): postgresql-server-13.14-1.el9.x86_64. 25 MB/s | 5.7 MB 00:00 2024-04-02T14:20:29.501 INFO:teuthology.orchestra.run.smithi092.stdout:(61/174): protobuf-3.14.0-13.el9.x86_64.rpm 4.3 MB/s | 1.0 MB 00:00 2024-04-02T14:20:29.534 INFO:teuthology.orchestra.run.smithi092.stdout:(62/174): python3-attrs-20.3.0-7.el9.noarch.rpm 1.2 MB/s | 78 kB 00:00 2024-04-02T14:20:29.760 INFO:teuthology.orchestra.run.smithi092.stdout:(63/174): python3-babel-2.9.1-2.el9.noarch.rpm 22 MB/s | 5.8 MB 00:00 2024-04-02T14:20:29.785 INFO:teuthology.orchestra.run.smithi092.stdout:(64/174): python3-jsonpatch-1.21-16.el9.noarch. 98 kB/s | 24 kB 00:00 2024-04-02T14:20:29.810 INFO:teuthology.orchestra.run.smithi092.stdout:(65/174): python3-jsonpointer-2.0-4.el9.noarch. 361 kB/s | 18 kB 00:00 2024-04-02T14:20:29.835 INFO:teuthology.orchestra.run.smithi092.stdout:(66/174): python3-mako-1.1.4-6.el9.noarch.rpm 2.8 MB/s | 144 kB 00:00 2024-04-02T14:20:29.852 INFO:teuthology.orchestra.run.smithi092.stdout:(67/174): python3-markupsafe-1.1.1-12.el9.x86_6 756 kB/s | 32 kB 00:00 2024-04-02T14:20:30.038 INFO:teuthology.orchestra.run.smithi092.stdout:(68/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 24 MB/s | 4.8 MB 00:00 2024-04-02T14:20:30.061 INFO:teuthology.orchestra.run.smithi090.stdout:(7/174): ceph-osd-19.0.0-2609.g191d963a.el9.x86 13 MB/s | 17 MB 00:01 2024-04-02T14:20:30.063 INFO:teuthology.orchestra.run.smithi092.stdout:(69/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.0 MB/s | 217 kB 00:00 2024-04-02T14:20:30.089 INFO:teuthology.orchestra.run.smithi092.stdout:(70/174): python3-oauthlib-3.1.1-5.el9.noarch.r 3.4 MB/s | 174 kB 00:00 2024-04-02T14:20:30.114 INFO:teuthology.orchestra.run.smithi092.stdout:(71/174): python3-packaging-20.9-5.el9.noarch.r 1.3 MB/s | 69 kB 00:00 2024-04-02T14:20:30.131 INFO:teuthology.orchestra.run.smithi092.stdout:(72/174): python3-prettytable-0.7.2-27.el9.noar 979 kB/s | 41 kB 00:00 2024-04-02T14:20:30.156 INFO:teuthology.orchestra.run.smithi092.stdout:(73/174): python3-protobuf-3.14.0-13.el9.noarch 5.5 MB/s | 238 kB 00:00 2024-04-02T14:20:30.182 INFO:teuthology.orchestra.run.smithi092.stdout:(74/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.6 MB/s | 132 kB 00:00 2024-04-02T14:20:30.195 INFO:teuthology.orchestra.run.smithi090.stdout:(8/174): ceph-base-19.0.0-2609.g191d963a.el9.x8 1.8 MB/s | 5.4 MB 00:02 2024-04-02T14:20:30.207 INFO:teuthology.orchestra.run.smithi092.stdout:(75/174): python3-pyasn1-modules-0.4.8-6.el9.no 4.1 MB/s | 211 kB 00:00 2024-04-02T14:20:30.232 INFO:teuthology.orchestra.run.smithi092.stdout:(76/174): python3-pytz-2021.1-5.el9.noarch.rpm 943 kB/s | 47 kB 00:00 2024-04-02T14:20:30.274 INFO:teuthology.orchestra.run.smithi092.stdout:(77/174): python3-requests-oauthlib-1.3.0-12.el 637 kB/s | 43 kB 00:00 2024-04-02T14:20:30.529 INFO:teuthology.orchestra.run.smithi090.stdout:(9/174): ceph-selinux-19.0.0-2609.g191d963a.el9 76 kB/s | 25 kB 00:00 2024-04-02T14:20:30.633 INFO:teuthology.orchestra.run.smithi092.stdout:(78/174): python3-toml-0.10.2-6.el9.noarch.rpm 108 kB/s | 39 kB 00:00 2024-04-02T14:20:30.730 INFO:teuthology.orchestra.run.smithi090.stdout:(10/174): ceph-radosgw-19.0.0-2609.g191d963a.el 15 MB/s | 9.8 MB 00:00 2024-04-02T14:20:30.779 INFO:teuthology.orchestra.run.smithi090.stdout:(11/174): cephfs-mirror-19.0.0-2609.g191d963a.e 4.5 MB/s | 225 kB 00:00 2024-04-02T14:20:30.830 INFO:teuthology.orchestra.run.smithi090.stdout:(12/174): cephfs-shell-19.0.0-2609.g191d963a.el 544 kB/s | 27 kB 00:00 2024-04-02T14:20:30.881 INFO:teuthology.orchestra.run.smithi090.stdout:(13/174): libcephfs-devel-19.0.0-2609.g191d963a 629 kB/s | 32 kB 00:00 2024-04-02T14:20:30.973 INFO:teuthology.orchestra.run.smithi090.stdout:(14/174): libcephfs2-19.0.0-2609.g191d963a.el9. 8.0 MB/s | 747 kB 00:00 2024-04-02T14:20:31.023 INFO:teuthology.orchestra.run.smithi090.stdout:(15/174): libcephsqlite-19.0.0-2609.g191d963a.e 3.3 MB/s | 164 kB 00:00 2024-04-02T14:20:31.034 INFO:teuthology.orchestra.run.smithi092.stdout:(79/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 20 MB/s | 16 MB 00:00 2024-04-02T14:20:31.059 INFO:teuthology.orchestra.run.smithi092.stdout:(80/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 96 kB/s | 41 kB 00:00 2024-04-02T14:20:31.082 INFO:teuthology.orchestra.run.smithi090.stdout:(16/174): librados-devel-19.0.0-2609.g191d963a. 2.1 MB/s | 127 kB 00:00 2024-04-02T14:20:31.084 INFO:teuthology.orchestra.run.smithi092.stdout:(81/174): qatlib-24.02.0-1.el9.x86_64.rpm 4.3 MB/s | 220 kB 00:00 2024-04-02T14:20:31.109 INFO:teuthology.orchestra.run.smithi092.stdout:(82/174): qatlib-service-24.02.0-1.el9.x86_64.r 699 kB/s | 35 kB 00:00 2024-04-02T14:20:31.135 INFO:teuthology.orchestra.run.smithi092.stdout:(83/174): qatzip-libs-1.2.0-1.el9.x86_64.rpm 926 kB/s | 46 kB 00:00 2024-04-02T14:20:31.158 INFO:teuthology.orchestra.run.smithi090.stdout:(17/174): libradosstriper1-19.0.0-2609.g191d963 6.6 MB/s | 500 kB 00:00 2024-04-02T14:20:31.160 INFO:teuthology.orchestra.run.smithi092.stdout:(84/174): socat-1.7.4.1-5.el9.x86_64.rpm 5.8 MB/s | 300 kB 00:00 2024-04-02T14:20:31.185 INFO:teuthology.orchestra.run.smithi092.stdout:(85/174): uuid-1.6.2-55.el9.x86_64.rpm 1.1 MB/s | 56 kB 00:00 2024-04-02T14:20:31.210 INFO:teuthology.orchestra.run.smithi092.stdout:(86/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 63 kB 00:00 2024-04-02T14:20:31.236 INFO:teuthology.orchestra.run.smithi092.stdout:(87/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 879 kB/s | 44 kB 00:00 2024-04-02T14:20:31.261 INFO:teuthology.orchestra.run.smithi092.stdout:(88/174): fuse-2.9.9-15.el9.x86_64.rpm 1.5 MB/s | 79 kB 00:00 2024-04-02T14:20:31.286 INFO:teuthology.orchestra.run.smithi092.stdout:(89/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2024-04-02T14:20:31.362 INFO:teuthology.orchestra.run.smithi092.stdout:(90/174): python3-cryptography-36.0.1-4.el9.x86 11 MB/s | 1.1 MB 00:00 2024-04-02T14:20:31.387 INFO:teuthology.orchestra.run.smithi092.stdout:(91/174): python3-ply-3.11-14.el9.noarch.rpm 1.0 MB/s | 103 kB 00:00 2024-04-02T14:20:31.421 INFO:teuthology.orchestra.run.smithi092.stdout:(92/174): python3-pycparser-2.20-6.el9.noarch.r 2.1 MB/s | 124 kB 00:00 2024-04-02T14:20:31.446 INFO:teuthology.orchestra.run.smithi092.stdout:(93/174): python3-urllib3-1.26.5-5.el9.noarch.r 3.1 MB/s | 187 kB 00:00 2024-04-02T14:20:31.566 INFO:teuthology.orchestra.run.smithi090.stdout:(18/174): librgw2-19.0.0-2609.g191d963a.el9.x86 13 MB/s | 5.1 MB 00:00 2024-04-02T14:20:31.571 INFO:teuthology.orchestra.run.smithi092.stdout:(94/174): jq-1.6-16.el9.x86_64.rpm 1.2 MB/s | 187 kB 00:00 2024-04-02T14:20:31.608 INFO:teuthology.orchestra.run.smithi090.stdout:(19/174): python3-ceph-argparse-19.0.0-2609.g19 1.1 MB/s | 45 kB 00:00 2024-04-02T14:20:31.613 INFO:teuthology.orchestra.run.smithi092.stdout:(95/174): libquadmath-11.4.1-3.el9.x86_64.rpm 4.6 MB/s | 195 kB 00:00 2024-04-02T14:20:31.658 INFO:teuthology.orchestra.run.smithi090.stdout:(20/174): python3-ceph-common-19.0.0-2609.g191d 2.6 MB/s | 134 kB 00:00 2024-04-02T14:20:31.672 INFO:teuthology.orchestra.run.smithi092.stdout:(96/174): libgfortran-11.4.1-3.el9.x86_64.rpm 3.5 MB/s | 803 kB 00:00 2024-04-02T14:20:31.689 INFO:teuthology.orchestra.run.smithi092.stdout:(97/174): python3-requests-2.25.1-8.el9.noarch. 1.6 MB/s | 125 kB 00:00 2024-04-02T14:20:31.709 INFO:teuthology.orchestra.run.smithi090.stdout:(21/174): python3-cephfs-19.0.0-2609.g191d963a. 3.2 MB/s | 161 kB 00:00 2024-04-02T14:20:31.739 INFO:teuthology.orchestra.run.smithi092.stdout:(98/174): elfutils-libelf-devel-0.190-2.el9.x86 321 kB/s | 21 kB 00:00 2024-04-02T14:20:31.759 INFO:teuthology.orchestra.run.smithi090.stdout:(22/174): python3-rados-19.0.0-2609.g191d963a.e 6.4 MB/s | 320 kB 00:00 2024-04-02T14:20:31.765 INFO:teuthology.orchestra.run.smithi092.stdout:(99/174): libuuid-devel-2.37.4-18.el9.x86_64.rp 338 kB/s | 25 kB 00:00 2024-04-02T14:20:31.818 INFO:teuthology.orchestra.run.smithi090.stdout:(23/174): python3-rbd-19.0.0-2609.g191d963a.el9 5.0 MB/s | 299 kB 00:00 2024-04-02T14:20:31.865 INFO:teuthology.orchestra.run.smithi092.stdout:(100/174): python3-devel-3.9.17-2.el9.x86_64.rp 2.4 MB/s | 245 kB 00:00 2024-04-02T14:20:31.868 INFO:teuthology.orchestra.run.smithi090.stdout:(24/174): python3-rgw-19.0.0-2609.g191d963a.el9 2.0 MB/s | 99 kB 00:00 2024-04-02T14:20:31.915 INFO:teuthology.orchestra.run.smithi092.stdout:(101/174): python3-jinja2-2.11.3-5.el9.noarch.r 4.9 MB/s | 248 kB 00:00 2024-04-02T14:20:31.919 INFO:teuthology.orchestra.run.smithi090.stdout:(25/174): rbd-fuse-19.0.0-2609.g191d963a.el9.x8 1.7 MB/s | 85 kB 00:00 2024-04-02T14:20:31.991 INFO:teuthology.orchestra.run.smithi092.stdout:(102/174): openssl-devel-3.0.7-27.el9.x86_64.rp 16 MB/s | 4.1 MB 00:00 2024-04-02T14:20:32.016 INFO:teuthology.orchestra.run.smithi092.stdout:(103/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 936 kB/s | 94 kB 00:00 2024-04-02T14:20:32.041 INFO:teuthology.orchestra.run.smithi092.stdout:(104/174): lua-devel-5.4.4-4.el9.x86_64.rpm 445 kB/s | 22 kB 00:00 2024-04-02T14:20:32.092 INFO:teuthology.orchestra.run.smithi092.stdout:(105/174): protobuf-compiler-3.14.0-13.el9.x86_ 11 MB/s | 863 kB 00:00 2024-04-02T14:20:32.153 INFO:teuthology.orchestra.run.smithi090.stdout:(26/174): rbd-mirror-19.0.0-2609.g191d963a.el9. 13 MB/s | 3.1 MB 00:00 2024-04-02T14:20:32.203 INFO:teuthology.orchestra.run.smithi090.stdout:(27/174): rbd-nbd-19.0.0-2609.g191d963a.el9.x86 3.4 MB/s | 170 kB 00:00 2024-04-02T14:20:32.253 INFO:teuthology.orchestra.run.smithi090.stdout:(28/174): ceph-grafana-dashboards-19.0.0-2609.g 690 kB/s | 34 kB 00:00 2024-04-02T14:20:32.303 INFO:teuthology.orchestra.run.smithi090.stdout:(29/174): ceph-mgr-cephadm-19.0.0-2609.g191d963 2.8 MB/s | 141 kB 00:00 2024-04-02T14:20:32.554 INFO:teuthology.orchestra.run.smithi090.stdout:(30/174): ceph-mgr-dashboard-19.0.0-2609.g191d9 14 MB/s | 3.5 MB 00:00 2024-04-02T14:20:32.801 INFO:teuthology.orchestra.run.smithi092.stdout:(106/174): abseil-cpp-20211102.0-3.el9.x86_64.r 722 kB/s | 548 kB 00:00 2024-04-02T14:20:32.842 INFO:teuthology.orchestra.run.smithi092.stdout:(107/174): gperftools-libs-2.9.1-2.el9.x86_64.r 411 kB/s | 309 kB 00:00 2024-04-02T14:20:32.884 INFO:teuthology.orchestra.run.smithi092.stdout:(108/174): grpc-data-1.46.7-10.el9.noarch.rpm 233 kB/s | 19 kB 00:00 2024-04-02T14:20:32.993 INFO:teuthology.orchestra.run.smithi092.stdout:(109/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 239 kB/s | 26 kB 00:00 2024-04-02T14:20:33.004 INFO:teuthology.orchestra.run.smithi090.stdout:(31/174): ceph-mgr-diskprediction-local-19.0.0- 16 MB/s | 7.4 MB 00:00 2024-04-02T14:20:33.054 INFO:teuthology.orchestra.run.smithi090.stdout:(32/174): ceph-mgr-modules-core-19.0.0-2609.g19 5.0 MB/s | 249 kB 00:00 2024-04-02T14:20:33.093 INFO:teuthology.orchestra.run.smithi092.stdout:(110/174): liboath-2.6.7-2.el9.x86_64.rpm 487 kB/s | 49 kB 00:00 2024-04-02T14:20:33.096 INFO:teuthology.orchestra.run.smithi090.stdout:(33/174): ceph-mgr-rook-19.0.0-2609.g191d963a.e 1.2 MB/s | 49 kB 00:00 2024-04-02T14:20:33.138 INFO:teuthology.orchestra.run.smithi090.stdout:(34/174): ceph-prometheus-alerts-19.0.0-2609.g1 403 kB/s | 16 kB 00:00 2024-04-02T14:20:33.197 INFO:teuthology.orchestra.run.smithi090.stdout:(35/174): ceph-volume-19.0.0-2609.g191d963a.el9 4.5 MB/s | 264 kB 00:00 2024-04-02T14:20:33.202 INFO:teuthology.orchestra.run.smithi092.stdout:(111/174): libunwind-1.6.2-1.el9.x86_64.rpm 621 kB/s | 67 kB 00:00 2024-04-02T14:20:33.289 INFO:teuthology.orchestra.run.smithi090.stdout:(36/174): cephadm-19.0.0-2609.g191d963a.el9.noa 8.2 MB/s | 762 kB 00:00 2024-04-02T14:20:33.319 INFO:teuthology.orchestra.run.smithi092.stdout:(112/174): luarocks-3.9.2-1.el9.noarch.rpm 1.3 MB/s | 151 kB 00:00 2024-04-02T14:20:33.332 INFO:teuthology.orchestra.run.smithi090.stdout:(37/174): cephfs-top-19.0.0-2609.g191d963a.el9. 434 kB/s | 18 kB 00:00 2024-04-02T14:20:33.377 INFO:teuthology.orchestra.run.smithi092.stdout:(113/174): libarrow-9.0.0-10.el9.x86_64.rpm 8.3 MB/s | 4.4 MB 00:00 2024-04-02T14:20:33.486 INFO:teuthology.orchestra.run.smithi092.stdout:(114/174): python3-asyncssh-2.13.2-5.el9.noarch 4.9 MB/s | 548 kB 00:00 2024-04-02T14:20:33.536 INFO:teuthology.orchestra.run.smithi092.stdout:(115/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 3.8 MB/s | 838 kB 00:00 2024-04-02T14:20:33.550 INFO:teuthology.orchestra.run.smithi090.stdout:(38/174): automake-1.16.2-8.el9.noarch.rpm 3.0 MB/s | 662 kB 00:00 2024-04-02T14:20:33.578 INFO:teuthology.orchestra.run.smithi092.stdout:(116/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 472 kB/s | 43 kB 00:00 2024-04-02T14:20:33.632 INFO:teuthology.orchestra.run.smithi092.stdout:(117/174): python3-cachetools-4.2.4-1.el9.noarc 338 kB/s | 32 kB 00:00 2024-04-02T14:20:33.633 INFO:teuthology.orchestra.run.smithi090.stdout:(39/174): bison-3.7.4-5.el9.x86_64.rpm 11 MB/s | 920 kB 00:00 2024-04-02T14:20:33.674 INFO:teuthology.orchestra.run.smithi092.stdout:(118/174): python3-certifi-2021.10.8-2.el9.noar 157 kB/s | 15 kB 00:00 2024-04-02T14:20:33.675 INFO:teuthology.orchestra.run.smithi090.stdout:(40/174): boost-program-options-1.75.0-8.el9.x8 2.5 MB/s | 106 kB 00:00 2024-04-02T14:20:33.717 INFO:teuthology.orchestra.run.smithi090.stdout:(41/174): flex-2.6.4-9.el9.x86_64.rpm 7.4 MB/s | 307 kB 00:00 2024-04-02T14:20:33.740 INFO:teuthology.orchestra.run.smithi092.stdout:(119/174): python3-cheroot-8.6.0-4.el9.noarch.r 1.5 MB/s | 172 kB 00:00 2024-04-02T14:20:33.751 INFO:teuthology.orchestra.run.smithi090.stdout:(42/174): flexiblas-3.0.4-8.el9.x86_64.rpm 971 kB/s | 32 kB 00:00 2024-04-02T14:20:33.774 INFO:teuthology.orchestra.run.smithi092.stdout:(120/174): python3-cherrypy-18.6.1-2.el9.noarch 3.5 MB/s | 358 kB 00:00 2024-04-02T14:20:33.875 INFO:teuthology.orchestra.run.smithi092.stdout:(121/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 2.2 MB/s | 303 kB 00:00 2024-04-02T14:20:33.900 INFO:teuthology.orchestra.run.smithi092.stdout:(122/174): python3-colorama-0.4.4-12.el9.noarch 281 kB/s | 35 kB 00:00 2024-04-02T14:20:33.926 INFO:teuthology.orchestra.run.smithi090.stdout:(43/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 17 MB/s | 3.0 MB 00:00 2024-04-02T14:20:33.968 INFO:teuthology.orchestra.run.smithi090.stdout:(44/174): flexiblas-openblas-openmp-3.0.4-8.el9 420 kB/s | 17 kB 00:00 2024-04-02T14:20:33.988 INFO:teuthology.orchestra.run.smithi177.stdout:lab-extras 39 kB/s | 1.7 kB 00:00 2024-04-02T14:20:33.992 INFO:teuthology.orchestra.run.smithi092.stdout:(123/174): python3-google-auth-2.28.2-1.el9.noa 1.7 MB/s | 203 kB 00:00 2024-04-02T14:20:34.027 INFO:teuthology.orchestra.run.smithi090.stdout:(45/174): libaio-devel-0.3.111-13.el9.x86_64.rp 174 kB/s | 10 kB 00:00 2024-04-02T14:20:34.094 INFO:teuthology.orchestra.run.smithi090.stdout:(46/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.4 MB/s | 159 kB 00:00 2024-04-02T14:20:34.211 INFO:teuthology.orchestra.run.smithi090.stdout:(47/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 380 kB/s | 44 kB 00:00 2024-04-02T14:20:34.287 INFO:teuthology.orchestra.run.smithi090.stdout:(48/174): librdkafka-1.6.1-102.el9.x86_64.rpm 8.7 MB/s | 661 kB 00:00 2024-04-02T14:20:34.337 INFO:teuthology.orchestra.run.smithi090.stdout:(49/174): libtool-2.4.6-45.el9.x86_64.rpm 11 MB/s | 578 kB 00:00 2024-04-02T14:20:34.379 INFO:teuthology.orchestra.run.smithi090.stdout:(50/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 1.1 MB/s | 46 kB 00:00 2024-04-02T14:20:34.426 INFO:teuthology.orchestra.run.smithi092.stdout:(124/174): ceph-test-19.0.0-2609.g191d963a.el9. 5.3 MB/s | 48 MB 00:09 2024-04-02T14:20:34.429 INFO:teuthology.orchestra.run.smithi090.stdout:(51/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 5.6 MB/s | 282 kB 00:00 2024-04-02T14:20:34.451 INFO:teuthology.orchestra.run.smithi092.stdout:(125/174): python3-grpcio-tools-1.46.7-10.el9.x 315 kB/s | 144 kB 00:00 2024-04-02T14:20:34.471 INFO:teuthology.orchestra.run.smithi090.stdout:(52/174): lua-5.4.4-4.el9.x86_64.rpm 4.4 MB/s | 187 kB 00:00 2024-04-02T14:20:34.501 INFO:teuthology.orchestra.run.smithi092.stdout:(126/174): python3-grpcio-1.46.7-10.el9.x86_64. 3.4 MB/s | 2.0 MB 00:00 2024-04-02T14:20:34.513 INFO:teuthology.orchestra.run.smithi090.stdout:(53/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 5.2 MB/s | 217 kB 00:00 2024-04-02T14:20:34.552 INFO:teuthology.orchestra.run.smithi092.stdout:(127/174): python3-jaraco-classes-3.2.1-5.el9.n 176 kB/s | 18 kB 00:00 2024-04-02T14:20:34.564 INFO:teuthology.orchestra.run.smithi090.stdout:(54/174): openblas-0.3.21-2.el9.x86_64.rpm 694 kB/s | 34 kB 00:00 2024-04-02T14:20:34.619 INFO:teuthology.orchestra.run.smithi092.stdout:(128/174): python3-jaraco-collections-3.0.0-8.e 197 kB/s | 23 kB 00:00 2024-04-02T14:20:34.635 INFO:teuthology.orchestra.run.smithi092.stdout:(129/174): python3-jaraco-8.2.1-3.el9.noarch.rp 51 kB/s | 11 kB 00:00 2024-04-02T14:20:34.669 INFO:teuthology.orchestra.run.smithi092.stdout:(130/174): python3-jaraco-functools-3.5.0-2.el9 166 kB/s | 19 kB 00:00 2024-04-02T14:20:34.728 INFO:teuthology.orchestra.run.smithi092.stdout:(131/174): python3-jaraco-text-3.2.0-6.el9.noar 181 kB/s | 20 kB 00:00 2024-04-02T14:20:34.761 INFO:teuthology.orchestra.run.smithi092.stdout:(132/174): python3-logutils-0.3.5-21.el9.noarch 505 kB/s | 46 kB 00:00 2024-04-02T14:20:34.820 INFO:teuthology.orchestra.run.smithi092.stdout:(133/174): python3-more-itertools-8.12.0-2.el9. 857 kB/s | 79 kB 00:00 2024-04-02T14:20:34.853 INFO:teuthology.orchestra.run.smithi092.stdout:(134/174): python3-natsort-7.1.1-5.el9.noarch.r 627 kB/s | 58 kB 00:00 2024-04-02T14:20:34.873 INFO:teuthology.orchestra.run.smithi090.stdout:(55/174): openblas-openmp-0.3.21-2.el9.x86_64.r 15 MB/s | 4.7 MB 00:00 2024-04-02T14:20:34.937 INFO:teuthology.orchestra.run.smithi092.stdout:(135/174): python3-pecan-1.4.2-3.el9.noarch.rpm 2.3 MB/s | 272 kB 00:00 2024-04-02T14:20:34.962 INFO:teuthology.orchestra.run.smithi092.stdout:(136/174): python3-portend-3.1.0-2.el9.noarch.r 151 kB/s | 16 kB 00:00 2024-04-02T14:20:35.015 INFO:teuthology.orchestra.run.smithi090.stdout:(56/174): postgresql-13.14-1.el9.x86_64.rpm 11 MB/s | 1.5 MB 00:00 2024-04-02T14:20:35.038 INFO:teuthology.orchestra.run.smithi092.stdout:(137/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 896 kB/s | 90 kB 00:00 2024-04-02T14:20:35.063 INFO:teuthology.orchestra.run.smithi092.stdout:(138/174): python3-pyperclip-1.8.2-2.el9.noarch 282 kB/s | 28 kB 00:00 2024-04-02T14:20:35.098 INFO:teuthology.orchestra.run.smithi090.stdout:(57/174): postgresql-contrib-13.14-1.el9.x86_64 9.7 MB/s | 817 kB 00:00 2024-04-02T14:20:35.138 INFO:teuthology.orchestra.run.smithi092.stdout:(139/174): python3-repoze-lru-0.7-16.el9.noarch 306 kB/s | 31 kB 00:00 2024-04-02T14:20:35.149 INFO:teuthology.orchestra.run.smithi090.stdout:(58/174): postgresql-private-libs-13.14-1.el9.x 2.6 MB/s | 132 kB 00:00 2024-04-02T14:20:35.180 INFO:teuthology.orchestra.run.smithi092.stdout:(140/174): python3-routes-2.5.1-5.el9.noarch.rp 1.6 MB/s | 188 kB 00:00 2024-04-02T14:20:35.306 INFO:teuthology.orchestra.run.smithi092.stdout:(141/174): python3-tempora-5.0.0-2.el9.noarch.r 286 kB/s | 36 kB 00:00 2024-04-02T14:20:35.389 INFO:teuthology.orchestra.run.smithi092.stdout:(142/174): python3-kubernetes-26.1.0-2.el9.noar 1.4 MB/s | 1.0 MB 00:00 2024-04-02T14:20:35.439 INFO:teuthology.orchestra.run.smithi092.stdout:(143/174): python3-typing-extensions-4.4.0-2.el 382 kB/s | 51 kB 00:00 2024-04-02T14:20:35.500 INFO:teuthology.orchestra.run.smithi090.stdout:(59/174): postgresql-server-13.14-1.el9.x86_64. 16 MB/s | 5.7 MB 00:00 2024-04-02T14:20:35.523 INFO:teuthology.orchestra.run.smithi092.stdout:(144/174): python3-webob-1.8.7-6.el9.noarch.rpm 1.7 MB/s | 230 kB 00:00 2024-04-02T14:20:35.565 INFO:teuthology.orchestra.run.smithi092.stdout:(145/174): python3-websocket-client-1.2.3-2.el9 716 kB/s | 90 kB 00:00 2024-04-02T14:20:35.673 INFO:teuthology.orchestra.run.smithi092.stdout:(146/174): python3-werkzeug-2.0.3-3.el9.1.noarc 2.8 MB/s | 427 kB 00:00 2024-04-02T14:20:35.675 INFO:teuthology.orchestra.run.smithi090.stdout:(60/174): protobuf-3.14.0-13.el9.x86_64.rpm 5.8 MB/s | 1.0 MB 00:00 2024-04-02T14:20:35.698 INFO:teuthology.orchestra.run.smithi092.stdout:(147/174): python3-zc-lockfile-2.0-10.el9.noarc 150 kB/s | 20 kB 00:00 2024-04-02T14:20:35.751 INFO:teuthology.orchestra.run.smithi090.stdout:(61/174): python3-attrs-20.3.0-7.el9.noarch.rpm 1.0 MB/s | 78 kB 00:00 2024-04-02T14:20:35.766 INFO:teuthology.orchestra.run.smithi092.stdout:(148/174): python3-rsa-4.9-2.el9.noarch.rpm 94 kB/s | 59 kB 00:00 2024-04-02T14:20:35.932 INFO:teuthology.orchestra.run.smithi092.stdout:(149/174): re2-20211101-3.el9.x86_64.rpm 740 kB/s | 192 kB 00:00 2024-04-02T14:20:36.016 INFO:teuthology.orchestra.run.smithi092.stdout:(150/174): thrift-0.15.0-2.el9.x86_64.rpm 5.0 MB/s | 1.6 MB 00:00 2024-04-02T14:20:36.191 INFO:teuthology.orchestra.run.smithi092.stdout:(151/174): NetworkManager-1.47.2-1.el9.x86_64.r 13 MB/s | 2.3 MB 00:00 2024-04-02T14:20:36.218 INFO:teuthology.orchestra.run.smithi090.stdout:(62/174): python3-babel-2.9.1-2.el9.noarch.rpm 12 MB/s | 5.8 MB 00:00 2024-04-02T14:20:36.260 INFO:teuthology.orchestra.run.smithi090.stdout:(63/174): python3-jsonpatch-1.21-16.el9.noarch. 595 kB/s | 24 kB 00:00 2024-04-02T14:20:36.267 INFO:teuthology.orchestra.run.smithi092.stdout:(152/174): librbd1-19.0.0-2609.g191d963a.el9.x8 9.4 MB/s | 3.1 MB 00:00 2024-04-02T14:20:36.294 INFO:teuthology.orchestra.run.smithi090.stdout:(64/174): python3-jsonpointer-2.0-4.el9.noarch. 557 kB/s | 18 kB 00:00 2024-04-02T14:20:36.334 INFO:teuthology.orchestra.run.smithi092.stdout:(153/174): NetworkManager-libnm-1.47.2-1.el9.x8 13 MB/s | 1.8 MB 00:00 2024-04-02T14:20:36.344 INFO:teuthology.orchestra.run.smithi090.stdout:(65/174): python3-mako-1.1.4-6.el9.noarch.rpm 2.8 MB/s | 144 kB 00:00 2024-04-02T14:20:36.359 INFO:teuthology.orchestra.run.smithi092.stdout:(154/174): NetworkManager-team-1.47.2-1.el9.x86 458 kB/s | 41 kB 00:00 2024-04-02T14:20:36.384 INFO:teuthology.orchestra.run.smithi092.stdout:(155/174): NetworkManager-tui-1.47.2-1.el9.x86_ 4.8 MB/s | 247 kB 00:00 2024-04-02T14:20:36.386 INFO:teuthology.orchestra.run.smithi090.stdout:(66/174): python3-markupsafe-1.1.1-12.el9.x86_6 774 kB/s | 32 kB 00:00 2024-04-02T14:20:36.410 INFO:teuthology.orchestra.run.smithi092.stdout:(156/174): xfsdump-3.1.12-4.el9.x86_64.rpm 6.3 MB/s | 321 kB 00:00 2024-04-02T14:20:36.469 INFO:teuthology.orchestra.run.smithi092.stdout:(157/174): acl-2.3.1-4.el9.x86_64.rpm 855 kB/s | 71 kB 00:00 2024-04-02T14:20:36.494 INFO:teuthology.orchestra.run.smithi092.stdout:(158/174): elfutils-debuginfod-client-0.190-2.e 437 kB/s | 36 kB 00:00 2024-04-02T14:20:36.523 INFO:teuthology.orchestra.run.smithi177.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-02T14:20:36.524 INFO:teuthology.orchestra.run.smithi177.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-02T14:20:36.535 INFO:teuthology.orchestra.run.smithi092.stdout:(159/174): elfutils-libelf-0.190-2.el9.x86_64.r 2.8 MB/s | 192 kB 00:00 2024-04-02T14:20:36.538 INFO:teuthology.orchestra.run.smithi177.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-04-02T14:20:36.539 INFO:teuthology.orchestra.run.smithi177.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-02T14:20:36.540 INFO:teuthology.orchestra.run.smithi177.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-04-02T14:20:36.542 INFO:teuthology.orchestra.run.smithi177.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-04-02T14:20:36.544 INFO:teuthology.orchestra.run.smithi177.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-02T14:20:36.546 INFO:teuthology.orchestra.run.smithi177.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-04-02T14:20:36.548 INFO:teuthology.orchestra.run.smithi177.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-04-02T14:20:36.578 INFO:teuthology.orchestra.run.smithi092.stdout:(160/174): elfutils-libs-0.190-2.el9.x86_64.rpm 3.0 MB/s | 256 kB 00:00 2024-04-02T14:20:36.611 INFO:teuthology.orchestra.run.smithi092.stdout:(161/174): libacl-2.3.1-4.el9.x86_64.rpm 306 kB/s | 23 kB 00:00 2024-04-02T14:20:36.627 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-04-02T14:20:36.636 INFO:teuthology.orchestra.run.smithi092.stdout:(162/174): libblkid-2.37.4-18.el9.x86_64.rpm 1.8 MB/s | 108 kB 00:00 2024-04-02T14:20:36.649 INFO:teuthology.orchestra.run.smithi177.stdout:============================================================================================ 2024-04-02T14:20:36.649 INFO:teuthology.orchestra.run.smithi177.stdout: Package Arch Version Repository Size 2024-04-02T14:20:36.649 INFO:teuthology.orchestra.run.smithi177.stdout:============================================================================================ 2024-04-02T14:20:36.649 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2024-04-02T14:20:36.650 INFO:teuthology.orchestra.run.smithi177.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-04-02T14:20:36.650 INFO:teuthology.orchestra.run.smithi177.stdout: ceph x86_64 2:19.0.0-2609.g191d963a.el9 ceph 6.6 k 2024-04-02T14:20:36.650 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-base x86_64 2:19.0.0-2609.g191d963a.el9 ceph 5.4 M 2024-04-02T14:20:36.650 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-fuse x86_64 2:19.0.0-2609.g191d963a.el9 ceph 890 k 2024-04-02T14:20:36.650 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2609.g191d963a.el9 ceph 145 k 2024-04-02T14:20:36.650 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr x86_64 2:19.0.0-2609.g191d963a.el9 ceph 1.5 M 2024-04-02T14:20:36.650 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2609.g191d963a.el9 ceph-noarch 141 k 2024-04-02T14:20:36.650 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2609.g191d963a.el9 ceph-noarch 3.5 M 2024-04-02T14:20:36.650 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2609.g191d963a.el9 ceph-noarch 7.4 M 2024-04-02T14:20:36.650 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr-rook noarch 2:19.0.0-2609.g191d963a.el9 ceph-noarch 49 k 2024-04-02T14:20:36.650 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-radosgw x86_64 2:19.0.0-2609.g191d963a.el9 ceph 9.8 M 2024-04-02T14:20:36.650 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-test x86_64 2:19.0.0-2609.g191d963a.el9 ceph 48 M 2024-04-02T14:20:36.650 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-volume noarch 2:19.0.0-2609.g191d963a.el9 ceph-noarch 264 k 2024-04-02T14:20:36.650 INFO:teuthology.orchestra.run.smithi177.stdout: cephadm noarch 2:19.0.0-2609.g191d963a.el9 ceph-noarch 762 k 2024-04-02T14:20:36.651 INFO:teuthology.orchestra.run.smithi177.stdout: cephfs-mirror x86_64 2:19.0.0-2609.g191d963a.el9 ceph 225 k 2024-04-02T14:20:36.651 INFO:teuthology.orchestra.run.smithi177.stdout: cephfs-shell x86_64 2:19.0.0-2609.g191d963a.el9 ceph 27 k 2024-04-02T14:20:36.651 INFO:teuthology.orchestra.run.smithi177.stdout: cephfs-top noarch 2:19.0.0-2609.g191d963a.el9 ceph-noarch 18 k 2024-04-02T14:20:36.651 INFO:teuthology.orchestra.run.smithi177.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-02T14:20:36.651 INFO:teuthology.orchestra.run.smithi177.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-04-02T14:20:36.651 INFO:teuthology.orchestra.run.smithi177.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-04-02T14:20:36.651 INFO:teuthology.orchestra.run.smithi177.stdout: libcephfs-devel x86_64 2:19.0.0-2609.g191d963a.el9 ceph 32 k 2024-04-02T14:20:36.651 INFO:teuthology.orchestra.run.smithi177.stdout: libcephfs2 x86_64 2:19.0.0-2609.g191d963a.el9 ceph 747 k 2024-04-02T14:20:36.651 INFO:teuthology.orchestra.run.smithi177.stdout: librados-devel x86_64 2:19.0.0-2609.g191d963a.el9 ceph 127 k 2024-04-02T14:20:36.651 INFO:teuthology.orchestra.run.smithi177.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-04-02T14:20:36.651 INFO:teuthology.orchestra.run.smithi177.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-02T14:20:36.651 INFO:teuthology.orchestra.run.smithi177.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-04-02T14:20:36.651 INFO:teuthology.orchestra.run.smithi177.stdout: postgresql x86_64 13.14-1.el9 CentOS-AppStream 1.5 M 2024-04-02T14:20:36.652 INFO:teuthology.orchestra.run.smithi177.stdout: postgresql-contrib x86_64 13.14-1.el9 CentOS-AppStream 817 k 2024-04-02T14:20:36.652 INFO:teuthology.orchestra.run.smithi177.stdout: postgresql-server x86_64 13.14-1.el9 CentOS-AppStream 5.7 M 2024-04-02T14:20:36.652 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cephfs x86_64 2:19.0.0-2609.g191d963a.el9 ceph 161 k 2024-04-02T14:20:36.652 INFO:teuthology.orchestra.run.smithi177.stdout: python3-rados x86_64 2:19.0.0-2609.g191d963a.el9 ceph 320 k 2024-04-02T14:20:36.652 INFO:teuthology.orchestra.run.smithi177.stdout: python3-rbd x86_64 2:19.0.0-2609.g191d963a.el9 ceph 299 k 2024-04-02T14:20:36.652 INFO:teuthology.orchestra.run.smithi177.stdout: python3-rgw x86_64 2:19.0.0-2609.g191d963a.el9 ceph 99 k 2024-04-02T14:20:36.652 INFO:teuthology.orchestra.run.smithi177.stdout: rbd-fuse x86_64 2:19.0.0-2609.g191d963a.el9 ceph 85 k 2024-04-02T14:20:36.652 INFO:teuthology.orchestra.run.smithi177.stdout: rbd-mirror x86_64 2:19.0.0-2609.g191d963a.el9 ceph 3.1 M 2024-04-02T14:20:36.652 INFO:teuthology.orchestra.run.smithi177.stdout: rbd-nbd x86_64 2:19.0.0-2609.g191d963a.el9 ceph 170 k 2024-04-02T14:20:36.652 INFO:teuthology.orchestra.run.smithi177.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-02T14:20:36.652 INFO:teuthology.orchestra.run.smithi177.stdout:Upgrading: 2024-04-02T14:20:36.652 INFO:teuthology.orchestra.run.smithi177.stdout: NetworkManager x86_64 1:1.47.2-1.el9 CentOS-BaseOS 2.3 M 2024-04-02T14:20:36.652 INFO:teuthology.orchestra.run.smithi177.stdout: NetworkManager-libnm x86_64 1:1.47.2-1.el9 CentOS-BaseOS 1.8 M 2024-04-02T14:20:36.653 INFO:teuthology.orchestra.run.smithi177.stdout: NetworkManager-team x86_64 1:1.47.2-1.el9 CentOS-BaseOS 41 k 2024-04-02T14:20:36.653 INFO:teuthology.orchestra.run.smithi177.stdout: NetworkManager-tui x86_64 1:1.47.2-1.el9 CentOS-BaseOS 247 k 2024-04-02T14:20:36.653 INFO:teuthology.orchestra.run.smithi177.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-04-02T14:20:36.653 INFO:teuthology.orchestra.run.smithi177.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-04-02T14:20:36.653 INFO:teuthology.orchestra.run.smithi177.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-04-02T14:20:36.653 INFO:teuthology.orchestra.run.smithi177.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-04-02T14:20:36.653 INFO:teuthology.orchestra.run.smithi177.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-04-02T14:20:36.653 INFO:teuthology.orchestra.run.smithi177.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-04-02T14:20:36.653 INFO:teuthology.orchestra.run.smithi177.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-04-02T14:20:36.653 INFO:teuthology.orchestra.run.smithi177.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-04-02T14:20:36.653 INFO:teuthology.orchestra.run.smithi177.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-04-02T14:20:36.653 INFO:teuthology.orchestra.run.smithi177.stdout: librados2 x86_64 2:19.0.0-2609.g191d963a.el9 ceph 3.4 M 2024-04-02T14:20:36.653 INFO:teuthology.orchestra.run.smithi177.stdout: librbd1 x86_64 2:19.0.0-2609.g191d963a.el9 ceph 3.1 M 2024-04-02T14:20:36.653 INFO:teuthology.orchestra.run.smithi177.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-04-02T14:20:36.653 INFO:teuthology.orchestra.run.smithi177.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-04-02T14:20:36.654 INFO:teuthology.orchestra.run.smithi177.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-04-02T14:20:36.654 INFO:teuthology.orchestra.run.smithi177.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-04-02T14:20:36.654 INFO:teuthology.orchestra.run.smithi177.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-04-02T14:20:36.654 INFO:teuthology.orchestra.run.smithi177.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-04-02T14:20:36.654 INFO:teuthology.orchestra.run.smithi177.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-04-02T14:20:36.654 INFO:teuthology.orchestra.run.smithi177.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-04-02T14:20:36.654 INFO:teuthology.orchestra.run.smithi177.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-04-02T14:20:36.654 INFO:teuthology.orchestra.run.smithi177.stdout:Installing dependencies: 2024-04-02T14:20:36.654 INFO:teuthology.orchestra.run.smithi177.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-02T14:20:36.654 INFO:teuthology.orchestra.run.smithi177.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-04-02T14:20:36.654 INFO:teuthology.orchestra.run.smithi177.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-04-02T14:20:36.654 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-common x86_64 2:19.0.0-2609.g191d963a.el9 ceph 21 M 2024-04-02T14:20:36.654 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2609.g191d963a.el9 ceph-noarch 34 k 2024-04-02T14:20:36.654 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mds x86_64 2:19.0.0-2609.g191d963a.el9 ceph 2.4 M 2024-04-02T14:20:36.655 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2609.g191d963a.el9 ceph-noarch 249 k 2024-04-02T14:20:36.655 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mon x86_64 2:19.0.0-2609.g191d963a.el9 ceph 4.7 M 2024-04-02T14:20:36.655 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-osd x86_64 2:19.0.0-2609.g191d963a.el9 ceph 17 M 2024-04-02T14:20:36.655 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2609.g191d963a.el9 ceph-noarch 16 k 2024-04-02T14:20:36.655 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-selinux x86_64 2:19.0.0-2609.g191d963a.el9 ceph 25 k 2024-04-02T14:20:36.655 INFO:teuthology.orchestra.run.smithi177.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-04-02T14:20:36.655 INFO:teuthology.orchestra.run.smithi177.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-04-02T14:20:36.655 INFO:teuthology.orchestra.run.smithi177.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-04-02T14:20:36.655 INFO:teuthology.orchestra.run.smithi177.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-04-02T14:20:36.655 INFO:teuthology.orchestra.run.smithi177.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-02T14:20:36.655 INFO:teuthology.orchestra.run.smithi177.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-02T14:20:36.655 INFO:teuthology.orchestra.run.smithi177.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-04-02T14:20:36.655 INFO:teuthology.orchestra.run.smithi177.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-02T14:20:36.655 INFO:teuthology.orchestra.run.smithi177.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-02T14:20:36.655 INFO:teuthology.orchestra.run.smithi177.stdout: libcephsqlite x86_64 2:19.0.0-2609.g191d963a.el9 ceph 164 k 2024-04-02T14:20:36.656 INFO:teuthology.orchestra.run.smithi177.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-02T14:20:36.656 INFO:teuthology.orchestra.run.smithi177.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-02T14:20:36.656 INFO:teuthology.orchestra.run.smithi177.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-04-02T14:20:36.656 INFO:teuthology.orchestra.run.smithi177.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-02T14:20:36.656 INFO:teuthology.orchestra.run.smithi177.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-04-02T14:20:36.656 INFO:teuthology.orchestra.run.smithi177.stdout: libradosstriper1 x86_64 2:19.0.0-2609.g191d963a.el9 ceph 500 k 2024-04-02T14:20:36.656 INFO:teuthology.orchestra.run.smithi177.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-04-02T14:20:36.656 INFO:teuthology.orchestra.run.smithi177.stdout: librgw2 x86_64 2:19.0.0-2609.g191d963a.el9 ceph 5.1 M 2024-04-02T14:20:36.656 INFO:teuthology.orchestra.run.smithi177.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-02T14:20:36.656 INFO:teuthology.orchestra.run.smithi177.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-04-02T14:20:36.656 INFO:teuthology.orchestra.run.smithi177.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-04-02T14:20:36.656 INFO:teuthology.orchestra.run.smithi177.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-04-02T14:20:36.656 INFO:teuthology.orchestra.run.smithi177.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-02T14:20:36.656 INFO:teuthology.orchestra.run.smithi177.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-02T14:20:36.657 INFO:teuthology.orchestra.run.smithi177.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-04-02T14:20:36.657 INFO:teuthology.orchestra.run.smithi177.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-04-02T14:20:36.657 INFO:teuthology.orchestra.run.smithi177.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-04-02T14:20:36.657 INFO:teuthology.orchestra.run.smithi177.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-02T14:20:36.657 INFO:teuthology.orchestra.run.smithi177.stdout: postgresql-private-libs x86_64 13.14-1.el9 CentOS-AppStream 132 k 2024-04-02T14:20:36.657 INFO:teuthology.orchestra.run.smithi177.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-04-02T14:20:36.657 INFO:teuthology.orchestra.run.smithi177.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-02T14:20:36.657 INFO:teuthology.orchestra.run.smithi177.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-02T14:20:36.657 INFO:teuthology.orchestra.run.smithi177.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-04-02T14:20:36.657 INFO:teuthology.orchestra.run.smithi177.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-04-02T14:20:36.657 INFO:teuthology.orchestra.run.smithi177.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-02T14:20:36.657 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-02T14:20:36.657 INFO:teuthology.orchestra.run.smithi177.stdout: python3-ceph-argparse x86_64 2:19.0.0-2609.g191d963a.el9 ceph 45 k 2024-04-02T14:20:36.657 INFO:teuthology.orchestra.run.smithi177.stdout: python3-ceph-common x86_64 2:19.0.0-2609.g191d963a.el9 ceph 134 k 2024-04-02T14:20:36.657 INFO:teuthology.orchestra.run.smithi177.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-02T14:20:36.658 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-04-02T14:20:36.658 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-02T14:20:36.658 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-02T14:20:36.658 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-02T14:20:36.658 INFO:teuthology.orchestra.run.smithi177.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-02T14:20:36.658 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-04-02T14:20:36.658 INFO:teuthology.orchestra.run.smithi177.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-04-02T14:20:36.658 INFO:teuthology.orchestra.run.smithi177.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-04-02T14:20:36.658 INFO:teuthology.orchestra.run.smithi177.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-02T14:20:36.658 INFO:teuthology.orchestra.run.smithi177.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-02T14:20:36.662 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-02T14:20:36.662 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-02T14:20:36.662 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-02T14:20:36.662 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-02T14:20:36.662 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-02T14:20:36.662 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-02T14:20:36.662 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-04-02T14:20:36.662 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-04-02T14:20:36.663 INFO:teuthology.orchestra.run.smithi177.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-02T14:20:36.663 INFO:teuthology.orchestra.run.smithi177.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-02T14:20:36.663 INFO:teuthology.orchestra.run.smithi177.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-04-02T14:20:36.663 INFO:teuthology.orchestra.run.smithi177.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-04-02T14:20:36.663 INFO:teuthology.orchestra.run.smithi177.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-02T14:20:36.663 INFO:teuthology.orchestra.run.smithi177.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-02T14:20:36.663 INFO:teuthology.orchestra.run.smithi177.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-04-02T14:20:36.663 INFO:teuthology.orchestra.run.smithi177.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-04-02T14:20:36.663 INFO:teuthology.orchestra.run.smithi177.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-04-02T14:20:36.663 INFO:teuthology.orchestra.run.smithi177.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-04-02T14:20:36.663 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-02T14:20:36.663 INFO:teuthology.orchestra.run.smithi177.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-04-02T14:20:36.663 INFO:teuthology.orchestra.run.smithi177.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-02T14:20:36.663 INFO:teuthology.orchestra.run.smithi177.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-04-02T14:20:36.663 INFO:teuthology.orchestra.run.smithi177.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-04-02T14:20:36.663 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-02T14:20:36.664 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-04-02T14:20:36.664 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-04-02T14:20:36.664 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-04-02T14:20:36.664 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-02T14:20:36.664 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-04-02T14:20:36.664 INFO:teuthology.orchestra.run.smithi177.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-02T14:20:36.664 INFO:teuthology.orchestra.run.smithi177.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-02T14:20:36.664 INFO:teuthology.orchestra.run.smithi177.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-04-02T14:20:36.664 INFO:teuthology.orchestra.run.smithi177.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-02T14:20:36.664 INFO:teuthology.orchestra.run.smithi177.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-02T14:20:36.664 INFO:teuthology.orchestra.run.smithi177.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-04-02T14:20:36.664 INFO:teuthology.orchestra.run.smithi177.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-02T14:20:36.664 INFO:teuthology.orchestra.run.smithi177.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-04-02T14:20:36.664 INFO:teuthology.orchestra.run.smithi177.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-02T14:20:36.664 INFO:teuthology.orchestra.run.smithi177.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-04-02T14:20:36.665 INFO:teuthology.orchestra.run.smithi177.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-04-02T14:20:36.665 INFO:teuthology.orchestra.run.smithi177.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-02T14:20:36.665 INFO:teuthology.orchestra.run.smithi177.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-02T14:20:36.665 INFO:teuthology.orchestra.run.smithi177.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-02T14:20:36.665 INFO:teuthology.orchestra.run.smithi177.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-02T14:20:36.665 INFO:teuthology.orchestra.run.smithi177.stdout: qatlib x86_64 24.02.0-1.el9 CentOS-AppStream 220 k 2024-04-02T14:20:36.665 INFO:teuthology.orchestra.run.smithi177.stdout: qatzip-libs x86_64 1.2.0-1.el9 CentOS-AppStream 46 k 2024-04-02T14:20:36.665 INFO:teuthology.orchestra.run.smithi177.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-02T14:20:36.665 INFO:teuthology.orchestra.run.smithi177.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-04-02T14:20:36.665 INFO:teuthology.orchestra.run.smithi177.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-02T14:20:36.665 INFO:teuthology.orchestra.run.smithi177.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-04-02T14:20:36.665 INFO:teuthology.orchestra.run.smithi177.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-04-02T14:20:36.665 INFO:teuthology.orchestra.run.smithi177.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-04-02T14:20:36.666 INFO:teuthology.orchestra.run.smithi177.stdout:Installing weak dependencies: 2024-04-02T14:20:36.666 INFO:teuthology.orchestra.run.smithi177.stdout: qatlib-service x86_64 24.02.0-1.el9 CentOS-AppStream 35 k 2024-04-02T14:20:36.666 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-02T14:20:36.666 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2024-04-02T14:20:36.666 INFO:teuthology.orchestra.run.smithi177.stdout:============================================================================================ 2024-04-02T14:20:36.666 INFO:teuthology.orchestra.run.smithi177.stdout:Install 150 Packages 2024-04-02T14:20:36.666 INFO:teuthology.orchestra.run.smithi177.stdout:Upgrade 24 Packages 2024-04-02T14:20:36.666 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-02T14:20:36.667 INFO:teuthology.orchestra.run.smithi092.stdout:(163/174): libfdisk-2.37.4-18.el9.x86_64.rpm 3.0 MB/s | 155 kB 00:00 2024-04-02T14:20:36.667 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 226 M 2024-04-02T14:20:36.667 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2024-04-02T14:20:36.670 INFO:teuthology.orchestra.run.smithi090.stdout:(67/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 17 MB/s | 4.8 MB 00:00 2024-04-02T14:20:36.687 INFO:teuthology.orchestra.run.smithi092.stdout:(164/174): libmount-2.37.4-18.el9.x86_64.rpm 2.6 MB/s | 136 kB 00:00 2024-04-02T14:20:36.712 INFO:teuthology.orchestra.run.smithi092.stdout:(165/174): libsmartcols-2.37.4-18.el9.x86_64.rp 1.2 MB/s | 64 kB 00:00 2024-04-02T14:20:36.737 INFO:teuthology.orchestra.run.smithi090.stdout:(68/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 3.2 MB/s | 217 kB 00:00 2024-04-02T14:20:36.737 INFO:teuthology.orchestra.run.smithi092.stdout:(166/174): libuuid-2.37.4-18.el9.x86_64.rpm 572 kB/s | 29 kB 00:00 2024-04-02T14:20:36.787 INFO:teuthology.orchestra.run.smithi090.stdout:(69/174): python3-oauthlib-3.1.1-5.el9.noarch.r 3.5 MB/s | 174 kB 00:00 2024-04-02T14:20:36.838 INFO:teuthology.orchestra.run.smithi090.stdout:(70/174): python3-packaging-20.9-5.el9.noarch.r 1.4 MB/s | 69 kB 00:00 2024-04-02T14:20:36.880 INFO:teuthology.orchestra.run.smithi090.stdout:(71/174): python3-prettytable-0.7.2-27.el9.noar 1.0 MB/s | 41 kB 00:00 2024-04-02T14:20:36.888 INFO:teuthology.orchestra.run.smithi092.stdout:(167/174): openssl-3.0.7-27.el9.x86_64.rpm 6.9 MB/s | 1.2 MB 00:00 2024-04-02T14:20:36.930 INFO:teuthology.orchestra.run.smithi090.stdout:(72/174): python3-protobuf-3.14.0-13.el9.noarch 4.7 MB/s | 238 kB 00:00 2024-04-02T14:20:36.938 INFO:teuthology.orchestra.run.smithi092.stdout:(168/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 11 MB/s | 2.2 MB 00:00 2024-04-02T14:20:36.972 INFO:teuthology.orchestra.run.smithi090.stdout:(73/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 3.2 MB/s | 132 kB 00:00 2024-04-02T14:20:37.005 INFO:teuthology.orchestra.run.smithi092.stdout:(169/174): util-linux-2.37.4-18.el9.x86_64.rpm 20 MB/s | 2.3 MB 00:00 2024-04-02T14:20:37.022 INFO:teuthology.orchestra.run.smithi090.stdout:(74/174): python3-pyasn1-modules-0.4.8-6.el9.no 4.2 MB/s | 211 kB 00:00 2024-04-02T14:20:37.030 INFO:teuthology.orchestra.run.smithi092.stdout:(170/174): util-linux-core-2.37.4-18.el9.x86_64 5.0 MB/s | 465 kB 00:00 2024-04-02T14:20:37.047 INFO:teuthology.orchestra.run.smithi092.stdout:(171/174): util-linux-user-2.37.4-18.el9.x86_64 744 kB/s | 31 kB 00:00 2024-04-02T14:20:37.065 INFO:teuthology.orchestra.run.smithi090.stdout:(75/174): python3-pytz-2021.1-5.el9.noarch.rpm 1.1 MB/s | 47 kB 00:00 2024-04-02T14:20:37.089 INFO:teuthology.orchestra.run.smithi092.stdout:(172/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 18 MB/s | 1.1 MB 00:00 2024-04-02T14:20:37.106 INFO:teuthology.orchestra.run.smithi090.stdout:(76/174): python3-requests-oauthlib-1.3.0-12.el 1.0 MB/s | 43 kB 00:00 2024-04-02T14:20:37.156 INFO:teuthology.orchestra.run.smithi092.stdout:(173/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 685 kB/s | 74 kB 00:00 2024-04-02T14:20:37.724 INFO:teuthology.orchestra.run.smithi090.stdout:(77/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 25 MB/s | 16 MB 00:00 2024-04-02T14:20:37.766 INFO:teuthology.orchestra.run.smithi090.stdout:(78/174): python3-toml-0.10.2-6.el9.noarch.rpm 955 kB/s | 39 kB 00:00 2024-04-02T14:20:37.808 INFO:teuthology.orchestra.run.smithi090.stdout:(79/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 1.0 MB/s | 41 kB 00:00 2024-04-02T14:20:37.898 INFO:teuthology.orchestra.run.smithi090.stdout:(80/174): qatlib-24.02.0-1.el9.x86_64.rpm 2.4 MB/s | 220 kB 00:00 2024-04-02T14:20:37.942 INFO:teuthology.orchestra.run.smithi090.stdout:(81/174): qatlib-service-24.02.0-1.el9.x86_64.r 818 kB/s | 35 kB 00:00 2024-04-02T14:20:37.981 INFO:teuthology.orchestra.run.smithi177.stdout:(1/174): ceph-19.0.0-2609.g191d963a.el9.x86_64. 14 kB/s | 6.6 kB 00:00 2024-04-02T14:20:37.985 INFO:teuthology.orchestra.run.smithi090.stdout:(82/174): qatzip-libs-1.2.0-1.el9.x86_64.rpm 1.1 MB/s | 46 kB 00:00 2024-04-02T14:20:38.035 INFO:teuthology.orchestra.run.smithi090.stdout:(83/174): socat-1.7.4.1-5.el9.x86_64.rpm 6.0 MB/s | 300 kB 00:00 2024-04-02T14:20:38.156 INFO:teuthology.orchestra.run.smithi177.stdout:(2/174): ceph-fuse-19.0.0-2609.g191d963a.el9.x8 5.0 MB/s | 890 kB 00:00 2024-04-02T14:20:38.207 INFO:teuthology.orchestra.run.smithi177.stdout:(3/174): ceph-immutable-object-cache-19.0.0-260 2.8 MB/s | 145 kB 00:00 2024-04-02T14:20:38.328 INFO:teuthology.orchestra.run.smithi090.stdout:(84/174): ceph-common-19.0.0-2609.g191d963a.el9 1.9 MB/s | 21 MB 00:11 2024-04-02T14:20:38.353 INFO:teuthology.orchestra.run.smithi090.stdout:(85/174): uuid-1.6.2-55.el9.x86_64.rpm 178 kB/s | 56 kB 00:00 2024-04-02T14:20:38.378 INFO:teuthology.orchestra.run.smithi090.stdout:(86/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.3 MB/s | 63 kB 00:00 2024-04-02T14:20:38.403 INFO:teuthology.orchestra.run.smithi090.stdout:(87/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 898 kB/s | 44 kB 00:00 2024-04-02T14:20:38.454 INFO:teuthology.orchestra.run.smithi090.stdout:(88/174): fuse-2.9.9-15.el9.x86_64.rpm 1.0 MB/s | 79 kB 00:00 2024-04-02T14:20:38.473 INFO:teuthology.orchestra.run.smithi177.stdout:(4/174): ceph-mds-19.0.0-2609.g191d963a.el9.x86 8.9 MB/s | 2.4 MB 00:00 2024-04-02T14:20:38.488 INFO:teuthology.orchestra.run.smithi090.stdout:(89/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 2.9 MB/s | 241 kB 00:00 2024-04-02T14:20:38.546 INFO:teuthology.orchestra.run.smithi090.stdout:(90/174): python3-cryptography-36.0.1-4.el9.x86 13 MB/s | 1.1 MB 00:00 2024-04-02T14:20:38.571 INFO:teuthology.orchestra.run.smithi090.stdout:(91/174): python3-ply-3.11-14.el9.noarch.rpm 1.4 MB/s | 103 kB 00:00 2024-04-02T14:20:38.590 INFO:teuthology.orchestra.run.smithi177.stdout:(5/174): ceph-base-19.0.0-2609.g191d963a.el9.x8 5.0 MB/s | 5.4 MB 00:01 2024-04-02T14:20:38.597 INFO:teuthology.orchestra.run.smithi090.stdout:(92/174): python3-pycparser-2.20-6.el9.noarch.r 2.5 MB/s | 124 kB 00:00 2024-04-02T14:20:38.631 INFO:teuthology.orchestra.run.smithi090.stdout:(93/174): python3-urllib3-1.26.5-5.el9.noarch.r 3.2 MB/s | 187 kB 00:00 2024-04-02T14:20:38.632 INFO:teuthology.orchestra.run.smithi177.stdout:(6/174): ceph-mgr-19.0.0-2609.g191d963a.el9.x86 9.4 MB/s | 1.5 MB 00:00 2024-04-02T14:20:38.823 INFO:teuthology.orchestra.run.smithi090.stdout:(94/174): jq-1.6-16.el9.x86_64.rpm 839 kB/s | 187 kB 00:00 2024-04-02T14:20:38.974 INFO:teuthology.orchestra.run.smithi177.stdout:(7/174): ceph-mon-19.0.0-2609.g191d963a.el9.x86 12 MB/s | 4.7 MB 00:00 2024-04-02T14:20:39.053 INFO:teuthology.orchestra.run.smithi090.stdout:(95/174): libgfortran-11.4.1-3.el9.x86_64.rpm 1.9 MB/s | 803 kB 00:00 2024-04-02T14:20:39.132 INFO:teuthology.orchestra.run.smithi090.stdout:(96/174): libquadmath-11.4.1-3.el9.x86_64.rpm 634 kB/s | 195 kB 00:00 2024-04-02T14:20:39.199 INFO:teuthology.orchestra.run.smithi090.stdout:(97/174): elfutils-libelf-devel-0.190-2.el9.x86 327 kB/s | 21 kB 00:00 2024-04-02T14:20:39.257 INFO:teuthology.orchestra.run.smithi090.stdout:(98/174): libuuid-devel-2.37.4-18.el9.x86_64.rp 442 kB/s | 25 kB 00:00 2024-04-02T14:20:39.339 INFO:teuthology.orchestra.run.smithi090.stdout:(99/174): python3-requests-2.25.1-8.el9.noarch. 438 kB/s | 125 kB 00:00 2024-04-02T14:20:39.425 INFO:teuthology.orchestra.run.smithi090.stdout:(100/174): python3-devel-3.9.17-2.el9.x86_64.rp 2.8 MB/s | 245 kB 00:00 2024-04-02T14:20:39.525 INFO:teuthology.orchestra.run.smithi090.stdout:(101/174): python3-jinja2-2.11.3-5.el9.noarch.r 2.4 MB/s | 248 kB 00:00 2024-04-02T14:20:39.609 INFO:teuthology.orchestra.run.smithi090.stdout:(102/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 1.1 MB/s | 94 kB 00:00 2024-04-02T14:20:39.734 INFO:teuthology.orchestra.run.smithi090.stdout:(103/174): openssl-devel-3.0.7-27.el9.x86_64.rp 8.7 MB/s | 4.1 MB 00:00 2024-04-02T14:20:39.840 INFO:teuthology.orchestra.run.smithi090.stdout:(104/174): lua-devel-5.4.4-4.el9.x86_64.rpm 97 kB/s | 22 kB 00:00 2024-04-02T14:20:39.900 INFO:teuthology.orchestra.run.smithi177.stdout:(8/174): ceph-radosgw-19.0.0-2609.g191d963a.el9 11 MB/s | 9.8 MB 00:00 2024-04-02T14:20:40.116 INFO:teuthology.orchestra.run.smithi177.stdout:(9/174): ceph-osd-19.0.0-2609.g191d963a.el9.x86 11 MB/s | 17 MB 00:01 2024-04-02T14:20:40.142 INFO:teuthology.orchestra.run.smithi177.stdout:(10/174): ceph-selinux-19.0.0-2609.g191d963a.el 104 kB/s | 25 kB 00:00 2024-04-02T14:20:40.203 INFO:teuthology.orchestra.run.smithi090.stdout:(105/174): protobuf-compiler-3.14.0-13.el9.x86_ 1.8 MB/s | 863 kB 00:00 2024-04-02T14:20:40.242 INFO:teuthology.orchestra.run.smithi177.stdout:(11/174): cephfs-mirror-19.0.0-2609.g191d963a.e 2.2 MB/s | 225 kB 00:00 2024-04-02T14:20:40.325 INFO:teuthology.orchestra.run.smithi177.stdout:(12/174): cephfs-shell-19.0.0-2609.g191d963a.el 325 kB/s | 27 kB 00:00 2024-04-02T14:20:40.386 INFO:teuthology.orchestra.run.smithi090.stdout:(106/174): abseil-cpp-20211102.0-3.el9.x86_64.r 1.0 MB/s | 548 kB 00:00 2024-04-02T14:20:40.417 INFO:teuthology.orchestra.run.smithi177.stdout:(13/174): libcephfs-devel-19.0.0-2609.g191d963a 347 kB/s | 32 kB 00:00 2024-04-02T14:20:40.470 INFO:teuthology.orchestra.run.smithi090.stdout:(107/174): grpc-data-1.46.7-10.el9.noarch.rpm 243 kB/s | 19 kB 00:00 2024-04-02T14:20:40.514 INFO:teuthology.orchestra.run.smithi092.stdout:(174/174): librados2-19.0.0-2609.g191d963a.el9. 732 kB/s | 3.4 MB 00:04 2024-04-02T14:20:40.518 INFO:teuthology.orchestra.run.smithi092.stdout:-------------------------------------------------------------------------------- 2024-04-02T14:20:40.518 INFO:teuthology.orchestra.run.smithi092.stdout:Total 12 MB/s | 226 MB 00:18 2024-04-02T14:20:40.543 INFO:teuthology.orchestra.run.smithi177.stdout:(14/174): libcephfs2-19.0.0-2609.g191d963a.el9. 5.8 MB/s | 747 kB 00:00 2024-04-02T14:20:40.643 INFO:teuthology.orchestra.run.smithi177.stdout:(15/174): libcephsqlite-19.0.0-2609.g191d963a.e 1.6 MB/s | 164 kB 00:00 2024-04-02T14:20:40.695 INFO:teuthology.orchestra.run.smithi090.stdout:(108/174): gperftools-libs-2.9.1-2.el9.x86_64.r 629 kB/s | 309 kB 00:00 2024-04-02T14:20:40.743 INFO:teuthology.orchestra.run.smithi177.stdout:(16/174): librados-devel-19.0.0-2609.g191d963a. 1.2 MB/s | 127 kB 00:00 2024-04-02T14:20:40.821 INFO:teuthology.orchestra.run.smithi090.stdout:(109/174): libarrow-9.0.0-10.el9.x86_64.rpm 13 MB/s | 4.4 MB 00:00 2024-04-02T14:20:40.846 INFO:teuthology.orchestra.run.smithi090.stdout:(110/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 173 kB/s | 26 kB 00:00 2024-04-02T14:20:40.860 INFO:teuthology.orchestra.run.smithi177.stdout:(17/174): libradosstriper1-19.0.0-2609.g191d963 4.2 MB/s | 500 kB 00:00 2024-04-02T14:20:40.895 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction check 2024-04-02T14:20:40.896 INFO:teuthology.orchestra.run.smithi090.stdout:(111/174): liboath-2.6.7-2.el9.x86_64.rpm 653 kB/s | 49 kB 00:00 2024-04-02T14:20:40.930 INFO:teuthology.orchestra.run.smithi090.stdout:(112/174): libunwind-1.6.2-1.el9.x86_64.rpm 812 kB/s | 67 kB 00:00 2024-04-02T14:20:40.980 INFO:teuthology.orchestra.run.smithi090.stdout:(113/174): luarocks-3.9.2-1.el9.noarch.rpm 1.8 MB/s | 151 kB 00:00 2024-04-02T14:20:41.072 INFO:teuthology.orchestra.run.smithi090.stdout:(114/174): python3-asyncssh-2.13.2-5.el9.noarch 5.9 MB/s | 548 kB 00:00 2024-04-02T14:20:41.123 INFO:teuthology.orchestra.run.smithi090.stdout:(115/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 4.3 MB/s | 838 kB 00:00 2024-04-02T14:20:41.148 INFO:teuthology.orchestra.run.smithi090.stdout:(116/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 583 kB/s | 43 kB 00:00 2024-04-02T14:20:41.164 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction check succeeded. 2024-04-02T14:20:41.164 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction test 2024-04-02T14:20:41.198 INFO:teuthology.orchestra.run.smithi090.stdout:(117/174): python3-cachetools-4.2.4-1.el9.noarc 432 kB/s | 32 kB 00:00 2024-04-02T14:20:41.224 INFO:teuthology.orchestra.run.smithi090.stdout:(118/174): python3-certifi-2021.10.8-2.el9.noar 200 kB/s | 15 kB 00:00 2024-04-02T14:20:41.282 INFO:teuthology.orchestra.run.smithi090.stdout:(119/174): python3-cheroot-8.6.0-4.el9.noarch.r 2.0 MB/s | 172 kB 00:00 2024-04-02T14:20:41.350 INFO:teuthology.orchestra.run.smithi090.stdout:(120/174): python3-cherrypy-18.6.1-2.el9.noarch 2.8 MB/s | 358 kB 00:00 2024-04-02T14:20:41.391 INFO:teuthology.orchestra.run.smithi090.stdout:(121/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 2.7 MB/s | 303 kB 00:00 2024-04-02T14:20:41.425 INFO:teuthology.orchestra.run.smithi090.stdout:(122/174): python3-colorama-0.4.4-12.el9.noarch 475 kB/s | 35 kB 00:00 2024-04-02T14:20:41.467 INFO:teuthology.orchestra.run.smithi090.stdout:(123/174): python3-google-auth-2.28.2-1.el9.noa 2.6 MB/s | 203 kB 00:00 2024-04-02T14:20:41.510 INFO:teuthology.orchestra.run.smithi177.stdout:(18/174): librgw2-19.0.0-2609.g191d963a.el9.x86 7.9 MB/s | 5.1 MB 00:00 2024-04-02T14:20:41.551 INFO:teuthology.orchestra.run.smithi090.stdout:(124/174): python3-grpcio-tools-1.46.7-10.el9.x 1.7 MB/s | 144 kB 00:00 2024-04-02T14:20:41.594 INFO:teuthology.orchestra.run.smithi177.stdout:(19/174): python3-ceph-argparse-19.0.0-2609.g19 543 kB/s | 45 kB 00:00 2024-04-02T14:20:41.626 INFO:teuthology.orchestra.run.smithi090.stdout:(125/174): python3-grpcio-1.46.7-10.el9.x86_64. 10 MB/s | 2.0 MB 00:00 2024-04-02T14:20:41.651 INFO:teuthology.orchestra.run.smithi090.stdout:(126/174): python3-jaraco-8.2.1-3.el9.noarch.rp 107 kB/s | 11 kB 00:00 2024-04-02T14:20:41.677 INFO:teuthology.orchestra.run.smithi177.stdout:(20/174): python3-ceph-common-19.0.0-2609.g191d 1.6 MB/s | 134 kB 00:00 2024-04-02T14:20:41.702 INFO:teuthology.orchestra.run.smithi090.stdout:(127/174): python3-jaraco-classes-3.2.1-5.el9.n 237 kB/s | 18 kB 00:00 2024-04-02T14:20:41.727 INFO:teuthology.orchestra.run.smithi090.stdout:(128/174): python3-jaraco-collections-3.0.0-8.e 310 kB/s | 23 kB 00:00 2024-04-02T14:20:41.779 INFO:teuthology.orchestra.run.smithi090.stdout:(129/174): python3-jaraco-functools-3.5.0-2.el9 257 kB/s | 19 kB 00:00 2024-04-02T14:20:41.804 INFO:teuthology.orchestra.run.smithi090.stdout:(130/174): python3-jaraco-text-3.2.0-6.el9.noar 259 kB/s | 20 kB 00:00 2024-04-02T14:20:41.869 INFO:teuthology.orchestra.run.smithi177.stdout:(21/174): ceph-common-19.0.0-2609.g191d963a.el9 4.7 MB/s | 21 MB 00:04 2024-04-02T14:20:41.894 INFO:teuthology.orchestra.run.smithi177.stdout:(22/174): python3-cephfs-19.0.0-2609.g191d963a. 744 kB/s | 161 kB 00:00 2024-04-02T14:20:41.922 INFO:teuthology.orchestra.run.smithi090.stdout:(131/174): python3-kubernetes-26.1.0-2.el9.noar 7.2 MB/s | 1.0 MB 00:00 2024-04-02T14:20:41.955 INFO:teuthology.orchestra.run.smithi090.stdout:(132/174): python3-logutils-0.3.5-21.el9.noarch 309 kB/s | 46 kB 00:00 2024-04-02T14:20:41.978 INFO:teuthology.orchestra.run.smithi177.stdout:(23/174): python3-rbd-19.0.0-2609.g191d963a.el9 3.5 MB/s | 299 kB 00:00 2024-04-02T14:20:41.997 INFO:teuthology.orchestra.run.smithi090.stdout:(133/174): python3-more-itertools-8.12.0-2.el9. 1.0 MB/s | 79 kB 00:00 2024-04-02T14:20:42.012 INFO:teuthology.orchestra.run.smithi177.stdout:(24/174): python3-rados-19.0.0-2609.g191d963a.e 2.2 MB/s | 320 kB 00:00 2024-04-02T14:20:42.039 INFO:teuthology.orchestra.run.smithi090.stdout:(134/174): python3-natsort-7.1.1-5.el9.noarch.r 695 kB/s | 58 kB 00:00 2024-04-02T14:20:42.045 INFO:teuthology.orchestra.run.smithi177.stdout:(25/174): python3-rgw-19.0.0-2609.g191d963a.el9 1.5 MB/s | 99 kB 00:00 2024-04-02T14:20:42.079 INFO:teuthology.orchestra.run.smithi177.stdout:(26/174): rbd-fuse-19.0.0-2609.g191d963a.el9.x8 1.2 MB/s | 85 kB 00:00 2024-04-02T14:20:42.081 INFO:teuthology.orchestra.run.smithi090.stdout:(135/174): python3-pecan-1.4.2-3.el9.noarch.rpm 3.2 MB/s | 272 kB 00:00 2024-04-02T14:20:42.106 INFO:teuthology.orchestra.run.smithi090.stdout:(136/174): python3-portend-3.1.0-2.el9.noarch.r 247 kB/s | 16 kB 00:00 2024-04-02T14:20:42.156 INFO:teuthology.orchestra.run.smithi090.stdout:(137/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.2 MB/s | 90 kB 00:00 2024-04-02T14:20:42.171 INFO:teuthology.orchestra.run.smithi177.stdout:(27/174): rbd-nbd-19.0.0-2609.g191d963a.el9.x86 1.8 MB/s | 170 kB 00:00 2024-04-02T14:20:42.191 INFO:teuthology.orchestra.run.smithi090.stdout:(138/174): python3-pyperclip-1.8.2-2.el9.noarch 336 kB/s | 28 kB 00:00 2024-04-02T14:20:42.233 INFO:teuthology.orchestra.run.smithi090.stdout:(139/174): python3-repoze-lru-0.7-16.el9.noarch 407 kB/s | 31 kB 00:00 2024-04-02T14:20:42.246 INFO:teuthology.orchestra.run.smithi177.stdout:(28/174): ceph-grafana-dashboards-19.0.0-2609.g 453 kB/s | 34 kB 00:00 2024-04-02T14:20:42.267 INFO:teuthology.orchestra.run.smithi090.stdout:(140/174): python3-routes-2.5.1-5.el9.noarch.rp 2.5 MB/s | 188 kB 00:00 2024-04-02T14:20:42.309 INFO:teuthology.orchestra.run.smithi090.stdout:(141/174): python3-rsa-4.9-2.el9.noarch.rpm 788 kB/s | 59 kB 00:00 2024-04-02T14:20:42.343 INFO:teuthology.orchestra.run.smithi090.stdout:(142/174): python3-tempora-5.0.0-2.el9.noarch.r 475 kB/s | 36 kB 00:00 2024-04-02T14:20:42.346 INFO:teuthology.orchestra.run.smithi177.stdout:(29/174): rbd-mirror-19.0.0-2609.g191d963a.el9. 10 MB/s | 3.1 MB 00:00 2024-04-02T14:20:42.372 INFO:teuthology.orchestra.run.smithi177.stdout:(30/174): ceph-mgr-cephadm-19.0.0-2609.g191d963 1.1 MB/s | 141 kB 00:00 2024-04-02T14:20:42.385 INFO:teuthology.orchestra.run.smithi090.stdout:(143/174): python3-typing-extensions-4.4.0-2.el 678 kB/s | 51 kB 00:00 2024-04-02T14:20:42.418 INFO:teuthology.orchestra.run.smithi090.stdout:(144/174): python3-webob-1.8.7-6.el9.noarch.rpm 3.0 MB/s | 230 kB 00:00 2024-04-02T14:20:42.494 INFO:teuthology.orchestra.run.smithi090.stdout:(145/174): python3-websocket-client-1.2.3-2.el9 829 kB/s | 90 kB 00:00 2024-04-02T14:20:42.536 INFO:teuthology.orchestra.run.smithi090.stdout:(146/174): python3-werkzeug-2.0.3-3.el9.1.noarc 3.6 MB/s | 427 kB 00:00 2024-04-02T14:20:42.570 INFO:teuthology.orchestra.run.smithi090.stdout:(147/174): python3-zc-lockfile-2.0-10.el9.noarc 268 kB/s | 20 kB 00:00 2024-04-02T14:20:42.620 INFO:teuthology.orchestra.run.smithi090.stdout:(148/174): re2-20211101-3.el9.x86_64.rpm 2.3 MB/s | 192 kB 00:00 2024-04-02T14:20:42.704 INFO:teuthology.orchestra.run.smithi090.stdout:(149/174): thrift-0.15.0-2.el9.x86_64.rpm 12 MB/s | 1.6 MB 00:00 2024-04-02T14:20:42.755 INFO:teuthology.orchestra.run.smithi177.stdout:(31/174): ceph-mgr-dashboard-19.0.0-2609.g191d9 8.7 MB/s | 3.5 MB 00:00 2024-04-02T14:20:42.855 INFO:teuthology.orchestra.run.smithi177.stdout:(32/174): ceph-mgr-modules-core-19.0.0-2609.g19 2.4 MB/s | 249 kB 00:00 2024-04-02T14:20:42.899 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction test succeeded. 2024-04-02T14:20:42.900 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction 2024-04-02T14:20:42.939 INFO:teuthology.orchestra.run.smithi177.stdout:(33/174): ceph-mgr-rook-19.0.0-2609.g191d963a.e 592 kB/s | 49 kB 00:00 2024-04-02T14:20:43.031 INFO:teuthology.orchestra.run.smithi177.stdout:(34/174): ceph-prometheus-alerts-19.0.0-2609.g1 180 kB/s | 16 kB 00:00 2024-04-02T14:20:43.156 INFO:teuthology.orchestra.run.smithi177.stdout:(35/174): ceph-volume-19.0.0-2609.g191d963a.el9 2.1 MB/s | 264 kB 00:00 2024-04-02T14:20:43.357 INFO:teuthology.orchestra.run.smithi177.stdout:(36/174): ceph-mgr-diskprediction-local-19.0.0- 7.5 MB/s | 7.4 MB 00:00 2024-04-02T14:20:43.362 INFO:teuthology.orchestra.run.smithi090.stdout:(150/174): librados2-19.0.0-2609.g191d963a.el9. 4.6 MB/s | 3.4 MB 00:00 2024-04-02T14:20:43.415 INFO:teuthology.orchestra.run.smithi177.stdout:(37/174): cephfs-top-19.0.0-2609.g191d963a.el9. 314 kB/s | 18 kB 00:00 2024-04-02T14:20:43.488 INFO:teuthology.orchestra.run.smithi090.stdout:(151/174): NetworkManager-1.47.2-1.el9.x86_64.r 18 MB/s | 2.3 MB 00:00 2024-04-02T14:20:43.566 INFO:teuthology.orchestra.run.smithi177.stdout:(38/174): automake-1.16.2-8.el9.noarch.rpm 4.3 MB/s | 662 kB 00:00 2024-04-02T14:20:43.596 INFO:teuthology.orchestra.run.smithi090.stdout:(152/174): NetworkManager-libnm-1.47.2-1.el9.x8 17 MB/s | 1.8 MB 00:00 2024-04-02T14:20:43.618 INFO:teuthology.orchestra.run.smithi177.stdout:(39/174): bison-3.7.4-5.el9.x86_64.rpm 18 MB/s | 920 kB 00:00 2024-04-02T14:20:43.647 INFO:teuthology.orchestra.run.smithi090.stdout:(153/174): NetworkManager-team-1.47.2-1.el9.x86 832 kB/s | 41 kB 00:00 2024-04-02T14:20:43.650 INFO:teuthology.orchestra.run.smithi177.stdout:(40/174): boost-program-options-1.75.0-8.el9.x8 3.1 MB/s | 106 kB 00:00 2024-04-02T14:20:43.689 INFO:teuthology.orchestra.run.smithi090.stdout:(154/174): NetworkManager-tui-1.47.2-1.el9.x86_ 5.9 MB/s | 247 kB 00:00 2024-04-02T14:20:43.692 INFO:teuthology.orchestra.run.smithi177.stdout:(41/174): flex-2.6.4-9.el9.x86_64.rpm 7.2 MB/s | 307 kB 00:00 2024-04-02T14:20:43.726 INFO:teuthology.orchestra.run.smithi177.stdout:(42/174): cephadm-19.0.0-2609.g191d963a.el9.noa 1.3 MB/s | 762 kB 00:00 2024-04-02T14:20:43.731 INFO:teuthology.orchestra.run.smithi090.stdout:(155/174): xfsdump-3.1.12-4.el9.x86_64.rpm 7.7 MB/s | 321 kB 00:00 2024-04-02T14:20:43.752 INFO:teuthology.orchestra.run.smithi177.stdout:(43/174): flexiblas-3.0.4-8.el9.x86_64.rpm 536 kB/s | 32 kB 00:00 2024-04-02T14:20:43.798 INFO:teuthology.orchestra.run.smithi090.stdout:(156/174): acl-2.3.1-4.el9.x86_64.rpm 1.1 MB/s | 71 kB 00:00 2024-04-02T14:20:43.827 INFO:teuthology.orchestra.run.smithi177.stdout:(44/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 30 MB/s | 3.0 MB 00:00 2024-04-02T14:20:43.852 INFO:teuthology.orchestra.run.smithi177.stdout:(45/174): flexiblas-openblas-openmp-3.0.4-8.el9 176 kB/s | 17 kB 00:00 2024-04-02T14:20:43.858 INFO:teuthology.orchestra.run.smithi090.stdout:(157/174): elfutils-debuginfod-client-0.190-2.e 616 kB/s | 36 kB 00:00 2024-04-02T14:20:43.877 INFO:teuthology.orchestra.run.smithi177.stdout:(46/174): libaio-devel-0.3.111-13.el9.x86_64.rp 201 kB/s | 10 kB 00:00 2024-04-02T14:20:43.894 INFO:teuthology.orchestra.run.smithi177.stdout:(47/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.7 MB/s | 159 kB 00:00 2024-04-02T14:20:43.920 INFO:teuthology.orchestra.run.smithi177.stdout:(48/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.0 MB/s | 44 kB 00:00 2024-04-02T14:20:43.945 INFO:teuthology.orchestra.run.smithi177.stdout:(49/174): librdkafka-1.6.1-102.el9.x86_64.rpm 13 MB/s | 661 kB 00:00 2024-04-02T14:20:43.950 INFO:teuthology.orchestra.run.smithi090.stdout:(158/174): elfutils-libelf-0.190-2.el9.x86_64.r 2.1 MB/s | 192 kB 00:00 2024-04-02T14:20:43.979 INFO:teuthology.orchestra.run.smithi177.stdout:(50/174): libtool-2.4.6-45.el9.x86_64.rpm 9.8 MB/s | 578 kB 00:00 2024-04-02T14:20:43.995 INFO:teuthology.orchestra.run.smithi177.stdout:(51/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 946 kB/s | 46 kB 00:00 2024-04-02T14:20:44.021 INFO:teuthology.orchestra.run.smithi177.stdout:(52/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 6.7 MB/s | 282 kB 00:00 2024-04-02T14:20:44.042 INFO:teuthology.orchestra.run.smithi090.stdout:(159/174): elfutils-libs-0.190-2.el9.x86_64.rpm 2.8 MB/s | 256 kB 00:00 2024-04-02T14:20:44.046 INFO:teuthology.orchestra.run.smithi177.stdout:(53/174): lua-5.4.4-4.el9.x86_64.rpm 3.7 MB/s | 187 kB 00:00 2024-04-02T14:20:44.071 INFO:teuthology.orchestra.run.smithi177.stdout:(54/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 4.3 MB/s | 217 kB 00:00 2024-04-02T14:20:44.088 INFO:teuthology.orchestra.run.smithi177.stdout:(55/174): openblas-0.3.21-2.el9.x86_64.rpm 828 kB/s | 34 kB 00:00 2024-04-02T14:20:44.092 INFO:teuthology.orchestra.run.smithi090.stdout:(160/174): libacl-2.3.1-4.el9.x86_64.rpm 469 kB/s | 23 kB 00:00 2024-04-02T14:20:44.168 INFO:teuthology.orchestra.run.smithi090.stdout:(161/174): libblkid-2.37.4-18.el9.x86_64.rpm 1.4 MB/s | 108 kB 00:00 2024-04-02T14:20:44.251 INFO:teuthology.orchestra.run.smithi090.stdout:(162/174): librbd1-19.0.0-2609.g191d963a.el9.x8 2.0 MB/s | 3.1 MB 00:01 2024-04-02T14:20:44.318 INFO:teuthology.orchestra.run.smithi090.stdout:(163/174): libfdisk-2.37.4-18.el9.x86_64.rpm 1.0 MB/s | 155 kB 00:00 2024-04-02T14:20:44.344 INFO:teuthology.orchestra.run.smithi090.stdout:(164/174): libmount-2.37.4-18.el9.x86_64.rpm 1.5 MB/s | 136 kB 00:00 2024-04-02T14:20:44.394 INFO:teuthology.orchestra.run.smithi090.stdout:(165/174): libsmartcols-2.37.4-18.el9.x86_64.rp 857 kB/s | 64 kB 00:00 2024-04-02T14:20:44.419 INFO:teuthology.orchestra.run.smithi090.stdout:(166/174): libuuid-2.37.4-18.el9.x86_64.rpm 386 kB/s | 29 kB 00:00 2024-04-02T14:20:44.506 INFO:teuthology.orchestra.run.smithi177.stdout:(56/174): ceph-test-19.0.0-2609.g191d963a.el9.x 11 MB/s | 48 MB 00:04 2024-04-02T14:20:44.581 INFO:teuthology.orchestra.run.smithi177.stdout:(57/174): openblas-openmp-0.3.21-2.el9.x86_64.r 9.3 MB/s | 4.7 MB 00:00 2024-04-02T14:20:44.614 INFO:teuthology.orchestra.run.smithi177.stdout:(58/174): postgresql-13.14-1.el9.x86_64.rpm 2.9 MB/s | 1.5 MB 00:00 2024-04-02T14:20:44.628 INFO:teuthology.orchestra.run.smithi090.stdout:(167/174): openssl-3.0.7-27.el9.x86_64.rpm 5.2 MB/s | 1.2 MB 00:00 2024-04-02T14:20:44.639 INFO:teuthology.orchestra.run.smithi177.stdout:(59/174): postgresql-contrib-13.14-1.el9.x86_64 6.1 MB/s | 817 kB 00:00 2024-04-02T14:20:44.664 INFO:teuthology.orchestra.run.smithi177.stdout:(60/174): postgresql-private-libs-13.14-1.el9.x 1.6 MB/s | 132 kB 00:00 2024-04-02T14:20:44.740 INFO:teuthology.orchestra.run.smithi177.stdout:(61/174): protobuf-3.14.0-13.el9.x86_64.rpm 10 MB/s | 1.0 MB 00:00 2024-04-02T14:20:44.799 INFO:teuthology.orchestra.run.smithi090.stdout:(168/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 5.7 MB/s | 2.2 MB 00:00 2024-04-02T14:20:44.824 INFO:teuthology.orchestra.run.smithi177.stdout:(62/174): postgresql-server-13.14-1.el9.x86_64. 31 MB/s | 5.7 MB 00:00 2024-04-02T14:20:44.840 INFO:teuthology.orchestra.run.smithi177.stdout:(63/174): python3-attrs-20.3.0-7.el9.noarch.rpm 447 kB/s | 78 kB 00:00 2024-04-02T14:20:44.912 INFO:teuthology.orchestra.run.smithi090.stdout:(169/174): util-linux-core-2.37.4-18.el9.x86_64 4.1 MB/s | 465 kB 00:00 2024-04-02T14:20:44.949 INFO:teuthology.orchestra.run.smithi177.stdout:(64/174): python3-babel-2.9.1-2.el9.noarch.rpm 28 MB/s | 5.8 MB 00:00 2024-04-02T14:20:44.966 INFO:teuthology.orchestra.run.smithi177.stdout:(65/174): python3-jsonpatch-1.21-16.el9.noarch. 175 kB/s | 24 kB 00:00 2024-04-02T14:20:44.982 INFO:teuthology.orchestra.run.smithi177.stdout:(66/174): python3-jsonpointer-2.0-4.el9.noarch. 130 kB/s | 18 kB 00:00 2024-04-02T14:20:45.008 INFO:teuthology.orchestra.run.smithi177.stdout:(67/174): python3-mako-1.1.4-6.el9.noarch.rpm 2.4 MB/s | 144 kB 00:00 2024-04-02T14:20:45.025 INFO:teuthology.orchestra.run.smithi177.stdout:(68/174): python3-markupsafe-1.1.1-12.el9.x86_6 760 kB/s | 32 kB 00:00 2024-04-02T14:20:45.133 INFO:teuthology.orchestra.run.smithi177.stdout:(69/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 32 MB/s | 4.8 MB 00:00 2024-04-02T14:20:45.159 INFO:teuthology.orchestra.run.smithi177.stdout:(70/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.4 MB/s | 217 kB 00:00 2024-04-02T14:20:45.179 INFO:teuthology.orchestra.run.smithi090.stdout:(170/174): util-linux-user-2.37.4-18.el9.x86_64 117 kB/s | 31 kB 00:00 2024-04-02T14:20:45.184 INFO:teuthology.orchestra.run.smithi177.stdout:(71/174): python3-oauthlib-3.1.1-5.el9.noarch.r 1.1 MB/s | 174 kB 00:00 2024-04-02T14:20:45.209 INFO:teuthology.orchestra.run.smithi177.stdout:(72/174): python3-packaging-20.9-5.el9.noarch.r 921 kB/s | 69 kB 00:00 2024-04-02T14:20:45.234 INFO:teuthology.orchestra.run.smithi177.stdout:(73/174): python3-prettytable-0.7.2-27.el9.noar 825 kB/s | 41 kB 00:00 2024-04-02T14:20:45.259 INFO:teuthology.orchestra.run.smithi177.stdout:(74/174): python3-protobuf-3.14.0-13.el9.noarch 3.1 MB/s | 238 kB 00:00 2024-04-02T14:20:45.271 INFO:teuthology.orchestra.run.smithi090.stdout:(171/174): util-linux-2.37.4-18.el9.x86_64.rpm 3.6 MB/s | 2.3 MB 00:00 2024-04-02T14:20:45.285 INFO:teuthology.orchestra.run.smithi177.stdout:(75/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-04-02T14:20:45.310 INFO:teuthology.orchestra.run.smithi177.stdout:(76/174): python3-pyasn1-modules-0.4.8-6.el9.no 4.1 MB/s | 211 kB 00:00 2024-04-02T14:20:45.335 INFO:teuthology.orchestra.run.smithi177.stdout:(77/174): python3-pytz-2021.1-5.el9.noarch.rpm 630 kB/s | 47 kB 00:00 2024-04-02T14:20:45.361 INFO:teuthology.orchestra.run.smithi177.stdout:(78/174): python3-requests-oauthlib-1.3.0-12.el 572 kB/s | 43 kB 00:00 2024-04-02T14:20:45.372 INFO:teuthology.orchestra.run.smithi090.stdout:(172/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 750 kB/s | 74 kB 00:00 2024-04-02T14:20:45.419 INFO:teuthology.orchestra.run.smithi177.stdout:(79/174): python3-toml-0.10.2-6.el9.noarch.rpm 463 kB/s | 39 kB 00:00 2024-04-02T14:20:45.430 INFO:teuthology.orchestra.run.smithi090.stdout:(173/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 4.3 MB/s | 1.1 MB 00:00 2024-04-02T14:20:45.744 INFO:teuthology.orchestra.run.smithi177.stdout:(80/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 38 MB/s | 16 MB 00:00 2024-04-02T14:20:45.770 INFO:teuthology.orchestra.run.smithi177.stdout:(81/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 101 kB/s | 41 kB 00:00 2024-04-02T14:20:45.795 INFO:teuthology.orchestra.run.smithi177.stdout:(82/174): qatlib-24.02.0-1.el9.x86_64.rpm 590 kB/s | 220 kB 00:00 2024-04-02T14:20:45.843 INFO:teuthology.orchestra.run.smithi177.stdout:(83/174): qatlib-service-24.02.0-1.el9.x86_64.r 358 kB/s | 35 kB 00:00 2024-04-02T14:20:45.870 INFO:teuthology.orchestra.run.smithi177.stdout:(84/174): qatzip-libs-1.2.0-1.el9.x86_64.rpm 616 kB/s | 46 kB 00:00 2024-04-02T14:20:45.895 INFO:teuthology.orchestra.run.smithi177.stdout:(85/174): socat-1.7.4.1-5.el9.x86_64.rpm 2.9 MB/s | 300 kB 00:00 2024-04-02T14:20:45.921 INFO:teuthology.orchestra.run.smithi177.stdout:(86/174): uuid-1.6.2-55.el9.x86_64.rpm 728 kB/s | 56 kB 00:00 2024-04-02T14:20:45.946 INFO:teuthology.orchestra.run.smithi177.stdout:(87/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 63 kB 00:00 2024-04-02T14:20:45.971 INFO:teuthology.orchestra.run.smithi177.stdout:(88/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 586 kB/s | 44 kB 00:00 2024-04-02T14:20:45.996 INFO:teuthology.orchestra.run.smithi177.stdout:(89/174): fuse-2.9.9-15.el9.x86_64.rpm 1.0 MB/s | 79 kB 00:00 2024-04-02T14:20:46.022 INFO:teuthology.orchestra.run.smithi177.stdout:(90/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2024-04-02T14:20:46.064 INFO:teuthology.orchestra.run.smithi177.stdout:(91/174): python3-cryptography-36.0.1-4.el9.x86 12 MB/s | 1.1 MB 00:00 2024-04-02T14:20:46.089 INFO:teuthology.orchestra.run.smithi177.stdout:(92/174): python3-ply-3.11-14.el9.noarch.rpm 1.1 MB/s | 103 kB 00:00 2024-04-02T14:20:46.114 INFO:teuthology.orchestra.run.smithi177.stdout:(93/174): python3-pycparser-2.20-6.el9.noarch.r 1.3 MB/s | 124 kB 00:00 2024-04-02T14:20:46.139 INFO:teuthology.orchestra.run.smithi177.stdout:(94/174): python3-urllib3-1.26.5-5.el9.noarch.r 3.7 MB/s | 187 kB 00:00 2024-04-02T14:20:46.539 INFO:teuthology.orchestra.run.smithi177.stdout:(95/174): jq-1.6-16.el9.x86_64.rpm 416 kB/s | 187 kB 00:00 2024-04-02T14:20:46.565 INFO:teuthology.orchestra.run.smithi177.stdout:(96/174): libquadmath-11.4.1-3.el9.x86_64.rpm 459 kB/s | 195 kB 00:00 2024-04-02T14:20:46.648 INFO:teuthology.orchestra.run.smithi177.stdout:(97/174): python3-requests-2.25.1-8.el9.noarch. 1.1 MB/s | 125 kB 00:00 2024-04-02T14:20:46.665 INFO:teuthology.orchestra.run.smithi177.stdout:(98/174): elfutils-libelf-devel-0.190-2.el9.x86 214 kB/s | 21 kB 00:00 2024-04-02T14:20:46.724 INFO:teuthology.orchestra.run.smithi177.stdout:(99/174): libuuid-devel-2.37.4-18.el9.x86_64.rp 340 kB/s | 25 kB 00:00 2024-04-02T14:20:46.799 INFO:teuthology.orchestra.run.smithi177.stdout:(100/174): libgfortran-11.4.1-3.el9.x86_64.rpm 1.1 MB/s | 803 kB 00:00 2024-04-02T14:20:46.849 INFO:teuthology.orchestra.run.smithi177.stdout:(101/174): python3-devel-3.9.17-2.el9.x86_64.rp 1.9 MB/s | 245 kB 00:00 2024-04-02T14:20:46.899 INFO:teuthology.orchestra.run.smithi177.stdout:(102/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 1.8 MB/s | 94 kB 00:00 2024-04-02T14:20:47.000 INFO:teuthology.orchestra.run.smithi177.stdout:(103/174): python3-jinja2-2.11.3-5.el9.noarch.r 1.2 MB/s | 248 kB 00:00 2024-04-02T14:20:47.017 INFO:teuthology.orchestra.run.smithi177.stdout:(104/174): lua-devel-5.4.4-4.el9.x86_64.rpm 191 kB/s | 22 kB 00:00 2024-04-02T14:20:47.125 INFO:teuthology.orchestra.run.smithi177.stdout:(105/174): protobuf-compiler-3.14.0-13.el9.x86_ 6.7 MB/s | 863 kB 00:00 2024-04-02T14:20:47.659 INFO:teuthology.orchestra.run.smithi177.stdout:(106/174): openssl-devel-3.0.7-27.el9.x86_64.rp 4.2 MB/s | 4.1 MB 00:00 2024-04-02T14:20:47.684 INFO:teuthology.orchestra.run.smithi177.stdout:(107/174): abseil-cpp-20211102.0-3.el9.x86_64.r 822 kB/s | 548 kB 00:00 2024-04-02T14:20:47.709 INFO:teuthology.orchestra.run.smithi177.stdout:(108/174): gperftools-libs-2.9.1-2.el9.x86_64.r 529 kB/s | 309 kB 00:00 2024-04-02T14:20:47.818 INFO:teuthology.orchestra.run.smithi177.stdout:(109/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 240 kB/s | 26 kB 00:00 2024-04-02T14:20:47.943 INFO:teuthology.orchestra.run.smithi177.stdout:(110/174): liboath-2.6.7-2.el9.x86_64.rpm 389 kB/s | 49 kB 00:00 2024-04-02T14:20:47.960 INFO:teuthology.orchestra.run.smithi177.stdout:(111/174): grpc-data-1.46.7-10.el9.noarch.rpm 65 kB/s | 19 kB 00:00 2024-04-02T14:20:48.077 INFO:teuthology.orchestra.run.smithi177.stdout:(112/174): libunwind-1.6.2-1.el9.x86_64.rpm 505 kB/s | 67 kB 00:00 2024-04-02T14:20:48.160 INFO:teuthology.orchestra.run.smithi177.stdout:(113/174): libarrow-9.0.0-10.el9.x86_64.rpm 9.3 MB/s | 4.4 MB 00:00 2024-04-02T14:20:48.277 INFO:teuthology.orchestra.run.smithi177.stdout:(114/174): python3-asyncssh-2.13.2-5.el9.noarch 4.6 MB/s | 548 kB 00:00 2024-04-02T14:20:48.319 INFO:teuthology.orchestra.run.smithi177.stdout:(115/174): luarocks-3.9.2-1.el9.noarch.rpm 420 kB/s | 151 kB 00:00 2024-04-02T14:20:48.345 INFO:teuthology.orchestra.run.smithi177.stdout:(116/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 3.1 MB/s | 838 kB 00:00 2024-04-02T14:20:48.378 INFO:teuthology.orchestra.run.smithi177.stdout:(117/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 432 kB/s | 43 kB 00:00 2024-04-02T14:20:48.437 INFO:teuthology.orchestra.run.smithi177.stdout:(118/174): python3-cachetools-4.2.4-1.el9.noarc 274 kB/s | 32 kB 00:00 2024-04-02T14:20:48.462 INFO:teuthology.orchestra.run.smithi177.stdout:(119/174): python3-certifi-2021.10.8-2.el9.noar 128 kB/s | 15 kB 00:00 2024-04-02T14:20:48.487 INFO:teuthology.orchestra.run.smithi177.stdout:(120/174): python3-cheroot-8.6.0-4.el9.noarch.r 1.5 MB/s | 172 kB 00:00 2024-04-02T14:20:48.563 INFO:teuthology.orchestra.run.smithi177.stdout:(121/174): python3-cherrypy-18.6.1-2.el9.noarch 2.8 MB/s | 358 kB 00:00 2024-04-02T14:20:48.588 INFO:teuthology.orchestra.run.smithi177.stdout:(122/174): python3-colorama-0.4.4-12.el9.noarch 352 kB/s | 35 kB 00:00 2024-04-02T14:20:48.672 INFO:teuthology.orchestra.run.smithi177.stdout:(123/174): python3-google-auth-2.28.2-1.el9.noa 1.8 MB/s | 203 kB 00:00 2024-04-02T14:20:48.697 INFO:teuthology.orchestra.run.smithi177.stdout:(124/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 1.3 MB/s | 303 kB 00:00 2024-04-02T14:20:48.739 INFO:teuthology.orchestra.run.smithi177.stdout:(125/174): python3-grpcio-1.46.7-10.el9.x86_64. 14 MB/s | 2.0 MB 00:00 2024-04-02T14:20:48.772 INFO:teuthology.orchestra.run.smithi177.stdout:(126/174): python3-grpcio-tools-1.46.7-10.el9.x 1.4 MB/s | 144 kB 00:00 2024-04-02T14:20:48.797 INFO:teuthology.orchestra.run.smithi177.stdout:(127/174): python3-jaraco-8.2.1-3.el9.noarch.rp 107 kB/s | 11 kB 00:00 2024-04-02T14:20:48.833 INFO:teuthology.orchestra.run.smithi177.stdout:(128/174): python3-jaraco-classes-3.2.1-5.el9.n 192 kB/s | 18 kB 00:00 2024-04-02T14:20:48.865 INFO:teuthology.orchestra.run.smithi177.stdout:(129/174): python3-jaraco-collections-3.0.0-8.e 252 kB/s | 23 kB 00:00 2024-04-02T14:20:48.890 INFO:teuthology.orchestra.run.smithi177.stdout:(130/174): python3-jaraco-functools-3.5.0-2.el9 210 kB/s | 19 kB 00:00 2024-04-02T14:20:48.932 INFO:teuthology.orchestra.run.smithi177.stdout:(131/174): python3-jaraco-text-3.2.0-6.el9.noar 196 kB/s | 20 kB 00:00 2024-04-02T14:20:48.990 INFO:teuthology.orchestra.run.smithi177.stdout:(132/174): python3-logutils-0.3.5-21.el9.noarch 462 kB/s | 46 kB 00:00 2024-04-02T14:20:49.024 INFO:teuthology.orchestra.run.smithi177.stdout:(133/174): python3-more-itertools-8.12.0-2.el9. 858 kB/s | 79 kB 00:00 2024-04-02T14:20:49.057 INFO:teuthology.orchestra.run.smithi177.stdout:(134/174): python3-kubernetes-26.1.0-2.el9.noar 5.3 MB/s | 1.0 MB 00:00 2024-04-02T14:20:49.091 INFO:teuthology.orchestra.run.smithi177.stdout:(135/174): python3-natsort-7.1.1-5.el9.noarch.r 574 kB/s | 58 kB 00:00 2024-04-02T14:20:49.125 INFO:teuthology.orchestra.run.smithi177.stdout:(136/174): python3-pecan-1.4.2-3.el9.noarch.rpm 2.6 MB/s | 272 kB 00:00 2024-04-02T14:20:49.150 INFO:teuthology.orchestra.run.smithi177.stdout:(137/174): python3-portend-3.1.0-2.el9.noarch.r 178 kB/s | 16 kB 00:00 2024-04-02T14:20:49.208 INFO:teuthology.orchestra.run.smithi177.stdout:(138/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 769 kB/s | 90 kB 00:00 2024-04-02T14:20:49.234 INFO:teuthology.orchestra.run.smithi177.stdout:(139/174): python3-pyperclip-1.8.2-2.el9.noarch 259 kB/s | 28 kB 00:00 2024-04-02T14:20:49.259 INFO:teuthology.orchestra.run.smithi177.stdout:(140/174): python3-repoze-lru-0.7-16.el9.noarch 283 kB/s | 31 kB 00:00 2024-04-02T14:20:49.343 INFO:teuthology.orchestra.run.smithi177.stdout:(141/174): python3-rsa-4.9-2.el9.noarch.rpm 544 kB/s | 59 kB 00:00 2024-04-02T14:20:49.368 INFO:teuthology.orchestra.run.smithi177.stdout:(142/174): python3-routes-2.5.1-5.el9.noarch.rp 1.2 MB/s | 188 kB 00:00 2024-04-02T14:20:49.393 INFO:teuthology.orchestra.run.smithi177.stdout:(143/174): python3-tempora-5.0.0-2.el9.noarch.r 267 kB/s | 36 kB 00:00 2024-04-02T14:20:49.435 INFO:teuthology.orchestra.run.smithi177.stdout:(144/174): python3-typing-extensions-4.4.0-2.el 554 kB/s | 51 kB 00:00 2024-04-02T14:20:49.485 INFO:teuthology.orchestra.run.smithi177.stdout:(145/174): python3-webob-1.8.7-6.el9.noarch.rpm 1.9 MB/s | 230 kB 00:00 2024-04-02T14:20:49.511 INFO:teuthology.orchestra.run.smithi177.stdout:(146/174): python3-websocket-client-1.2.3-2.el9 765 kB/s | 90 kB 00:00 2024-04-02T14:20:49.561 INFO:teuthology.orchestra.run.smithi177.stdout:(147/174): python3-werkzeug-2.0.3-3.el9.1.noarc 3.3 MB/s | 427 kB 00:00 2024-04-02T14:20:49.586 INFO:teuthology.orchestra.run.smithi177.stdout:(148/174): python3-zc-lockfile-2.0-10.el9.noarc 198 kB/s | 20 kB 00:00 2024-04-02T14:20:49.637 INFO:teuthology.orchestra.run.smithi177.stdout:(149/174): re2-20211101-3.el9.x86_64.rpm 1.5 MB/s | 192 kB 00:00 2024-04-02T14:20:49.695 INFO:teuthology.orchestra.run.smithi177.stdout:(150/174): thrift-0.15.0-2.el9.x86_64.rpm 12 MB/s | 1.6 MB 00:00 2024-04-02T14:20:49.864 INFO:teuthology.orchestra.run.smithi090.stdout:(174/174): ceph-test-19.0.0-2609.g191d963a.el9. 2.5 MB/s | 48 MB 00:19 2024-04-02T14:20:49.875 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-04-02T14:20:49.876 INFO:teuthology.orchestra.run.smithi090.stdout:Total 9.8 MB/s | 226 MB 00:23 2024-04-02T14:20:49.887 INFO:teuthology.orchestra.run.smithi177.stdout:(151/174): NetworkManager-1.47.2-1.el9.x86_64.r 14 MB/s | 2.3 MB 00:00 2024-04-02T14:20:49.987 INFO:teuthology.orchestra.run.smithi177.stdout:(152/174): NetworkManager-libnm-1.47.2-1.el9.x8 18 MB/s | 1.8 MB 00:00 2024-04-02T14:20:50.028 INFO:teuthology.orchestra.run.smithi092.stdout: Preparing : 1/1 2024-04-02T14:20:50.038 INFO:teuthology.orchestra.run.smithi177.stdout:(153/174): librados2-19.0.0-2609.g191d963a.el9. 7.5 MB/s | 3.4 MB 00:00 2024-04-02T14:20:50.071 INFO:teuthology.orchestra.run.smithi177.stdout:(154/174): NetworkManager-team-1.47.2-1.el9.x86 490 kB/s | 41 kB 00:00 2024-04-02T14:20:50.096 INFO:teuthology.orchestra.run.smithi177.stdout:(155/174): NetworkManager-tui-1.47.2-1.el9.x86_ 4.1 MB/s | 247 kB 00:00 2024-04-02T14:20:50.117 INFO:teuthology.orchestra.run.smithi092.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-04-02T14:20:50.122 INFO:teuthology.orchestra.run.smithi177.stdout:(156/174): xfsdump-3.1.12-4.el9.x86_64.rpm 6.3 MB/s | 321 kB 00:00 2024-04-02T14:20:50.153 INFO:teuthology.orchestra.run.smithi092.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-04-02T14:20:50.183 INFO:teuthology.orchestra.run.smithi092.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-04-02T14:20:50.192 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-04-02T14:20:50.232 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-04-02T14:20:50.286 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-04-02T14:20:50.305 INFO:teuthology.orchestra.run.smithi177.stdout:(157/174): elfutils-debuginfod-client-0.190-2.e 198 kB/s | 36 kB 00:00 2024-04-02T14:20:50.319 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-04-02T14:20:50.345 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-04-02T14:20:50.356 INFO:teuthology.orchestra.run.smithi177.stdout:(158/174): librbd1-19.0.0-2609.g191d963a.el9.x8 4.4 MB/s | 3.1 MB 00:00 2024-04-02T14:20:50.381 INFO:teuthology.orchestra.run.smithi177.stdout:(159/174): acl-2.3.1-4.el9.x86_64.rpm 251 kB/s | 71 kB 00:00 2024-04-02T14:20:50.387 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-04-02T14:20:50.481 INFO:teuthology.orchestra.run.smithi177.stdout:(160/174): libacl-2.3.1-4.el9.x86_64.rpm 231 kB/s | 23 kB 00:00 2024-04-02T14:20:50.573 INFO:teuthology.orchestra.run.smithi177.stdout:(161/174): elfutils-libelf-0.190-2.el9.x86_64.r 719 kB/s | 192 kB 00:00 2024-04-02T14:20:50.665 INFO:teuthology.orchestra.run.smithi177.stdout:(162/174): libblkid-2.37.4-18.el9.x86_64.rpm 590 kB/s | 108 kB 00:00 2024-04-02T14:20:50.690 INFO:teuthology.orchestra.run.smithi177.stdout:(163/174): libfdisk-2.37.4-18.el9.x86_64.rpm 1.3 MB/s | 155 kB 00:00 2024-04-02T14:20:50.707 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-04-02T14:20:50.762 INFO:teuthology.orchestra.run.smithi092.stdout: Upgrading : librados2-2:19.0.0-2609.g191d963a.el9.x86_64 10/198 2024-04-02T14:20:50.774 INFO:teuthology.orchestra.run.smithi177.stdout:(164/174): libmount-2.37.4-18.el9.x86_64.rpm 1.2 MB/s | 136 kB 00:00 2024-04-02T14:20:50.799 INFO:teuthology.orchestra.run.smithi177.stdout:(165/174): libsmartcols-2.37.4-18.el9.x86_64.rp 589 kB/s | 64 kB 00:00 2024-04-02T14:20:50.866 INFO:teuthology.orchestra.run.smithi177.stdout:(166/174): elfutils-libs-0.190-2.el9.x86_64.rpm 501 kB/s | 256 kB 00:00 2024-04-02T14:20:50.884 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: librados2-2:19.0.0-2609.g191d963a.el9.x86_64 10/198 2024-04-02T14:20:50.891 INFO:teuthology.orchestra.run.smithi177.stdout:(167/174): libuuid-2.37.4-18.el9.x86_64.rpm 245 kB/s | 29 kB 00:00 2024-04-02T14:20:50.914 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : libcephfs2-2:19.0.0-2609.g191d963a.el9.x86_64 11/198 2024-04-02T14:20:50.977 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2024-04-02T14:20:50.987 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: libcephfs2-2:19.0.0-2609.g191d963a.el9.x86_64 11/198 2024-04-02T14:20:51.018 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-rados-2:19.0.0-2609.g191d963a.el9.x86_64 12/198 2024-04-02T14:20:51.060 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 13/198 2024-04-02T14:20:51.075 INFO:teuthology.orchestra.run.smithi177.stdout:(168/174): openssl-3.0.7-27.el9.x86_64.rpm 4.4 MB/s | 1.2 MB 00:00 2024-04-02T14:20:51.200 INFO:teuthology.orchestra.run.smithi177.stdout:(169/174): util-linux-core-2.37.4-18.el9.x86_64 3.6 MB/s | 465 kB 00:00 2024-04-02T14:20:51.224 INFO:teuthology.orchestra.run.smithi092.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-04-02T14:20:51.274 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : postgresql-13.14-1.el9.x86_64 15/198 2024-04-02T14:20:51.283 INFO:teuthology.orchestra.run.smithi177.stdout:(170/174): util-linux-2.37.4-18.el9.x86_64.rpm 5.8 MB/s | 2.3 MB 00:00 2024-04-02T14:20:51.300 INFO:teuthology.orchestra.run.smithi177.stdout:(171/174): util-linux-user-2.37.4-18.el9.x86_64 310 kB/s | 31 kB 00:00 2024-04-02T14:20:51.302 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : libcephsqlite-2:19.0.0-2609.g191d963a.el9.x86_64 16/198 2024-04-02T14:20:51.343 INFO:teuthology.orchestra.run.smithi177.stdout:(172/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 4.5 MB/s | 2.2 MB 00:00 2024-04-02T14:20:51.350 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2024-04-02T14:20:51.350 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2024-04-02T14:20:51.389 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2609.g191d963a.el9.x86_64 16/198 2024-04-02T14:20:51.409 INFO:teuthology.orchestra.run.smithi177.stdout:(173/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 8.6 MB/s | 1.1 MB 00:00 2024-04-02T14:20:51.412 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : libradosstriper1-2:19.0.0-2609.g191d963a.el9.x86 17/198 2024-04-02T14:20:51.459 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2609.g191d963a.el9.x86 17/198 2024-04-02T14:20:51.460 INFO:teuthology.orchestra.run.smithi177.stdout:(174/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 468 kB/s | 74 kB 00:00 2024-04-02T14:20:51.463 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2024-04-02T14:20:51.464 INFO:teuthology.orchestra.run.smithi177.stdout:Total 15 MB/s | 226 MB 00:14 2024-04-02T14:20:51.499 INFO:teuthology.orchestra.run.smithi092.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-04-02T14:20:51.513 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 19/198 2024-04-02T14:20:51.529 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 19/198 2024-04-02T14:20:51.714 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 20/198 2024-04-02T14:20:51.746 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 20/198 2024-04-02T14:20:51.778 INFO:teuthology.orchestra.run.smithi092.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-04-02T14:20:51.837 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2024-04-02T14:20:51.882 INFO:teuthology.orchestra.run.smithi092.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-04-02T14:20:51.980 INFO:teuthology.orchestra.run.smithi092.stdout: Upgrading : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-04-02T14:20:52.024 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-04-02T14:20:52.097 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-04-02T14:20:52.107 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2024-04-02T14:20:52.107 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2024-04-02T14:20:52.166 INFO:teuthology.orchestra.run.smithi092.stdout: Upgrading : NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-04-02T14:20:52.369 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-04-02T14:20:52.472 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-04-02T14:20:52.558 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-04-02T14:20:52.596 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-04-02T14:20:52.664 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-04-02T14:20:52.700 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-04-02T14:20:52.776 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-04-02T14:20:52.829 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-04-02T14:20:52.867 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-04-02T14:20:52.931 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-04-02T14:20:52.966 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-04-02T14:20:53.009 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-04-02T14:20:53.041 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-04-02T14:20:53.069 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-04-02T14:20:53.139 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-04-02T14:20:53.174 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-ceph-common-2:19.0.0-2609.g191d963a.el9. 39/198 2024-04-02T14:20:53.217 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-ceph-argparse-2:19.0.0-2609.g191d963a.el 40/198 2024-04-02T14:20:53.246 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-cephfs-2:19.0.0-2609.g191d963a.el9.x86_6 41/198 2024-04-02T14:20:53.317 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-04-02T14:20:53.361 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-04-02T14:20:53.436 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-04-02T14:20:53.569 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-04-02T14:20:53.622 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-04-02T14:20:53.823 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2024-04-02T14:20:53.824 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2024-04-02T14:20:53.892 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2024-04-02T14:20:53.893 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2024-04-02T14:20:54.147 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-04-02T14:20:54.226 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-04-02T14:20:54.296 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-04-02T14:20:54.345 INFO:teuthology.orchestra.run.smithi092.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-04-02T14:20:54.355 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-04-02T14:20:54.385 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 51/198 2024-04-02T14:20:54.434 INFO:teuthology.orchestra.run.smithi092.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-04-02T14:20:54.472 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : librados-devel-2:19.0.0-2609.g191d963a.el9.x86_6 53/198 2024-04-02T14:20:54.501 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-04-02T14:20:54.733 INFO:teuthology.orchestra.run.smithi092.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-04-02T14:20:54.901 INFO:teuthology.orchestra.run.smithi092.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-04-02T14:20:54.962 INFO:teuthology.orchestra.run.smithi092.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-04-02T14:20:54.997 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-04-02T14:20:55.137 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-04-02T14:20:55.176 INFO:teuthology.orchestra.run.smithi092.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-04-02T14:20:55.248 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-04-02T14:20:55.319 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-04-02T14:20:55.351 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-04-02T14:20:55.388 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-04-02T14:20:55.435 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-04-02T14:20:55.471 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-04-02T14:20:55.513 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-04-02T14:20:55.555 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-04-02T14:20:55.689 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-04-02T14:20:55.743 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-04-02T14:20:55.773 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-04-02T14:20:55.814 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-04-02T14:20:55.917 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-04-02T14:20:55.965 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 74/198 2024-04-02T14:20:56.004 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-04-02T14:20:56.031 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-04-02T14:20:56.055 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-04-02T14:20:56.128 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-04-02T14:20:56.248 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-04-02T14:20:56.347 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-04-02T14:20:56.397 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-04-02T14:20:56.453 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-04-02T14:20:56.508 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-04-02T14:20:56.693 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-04-02T14:20:56.751 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-04-02T14:20:56.890 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-04-02T14:20:56.932 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 87/198 2024-04-02T14:20:56.964 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-04-02T14:20:56.995 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-04-02T14:20:57.023 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-04-02T14:20:57.057 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-04-02T14:20:57.096 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-04-02T14:20:57.128 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-04-02T14:20:57.158 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-04-02T14:20:57.238 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-04-02T14:20:57.276 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-04-02T14:20:57.310 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-04-02T14:20:57.351 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-04-02T14:20:57.385 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-04-02T14:20:57.475 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-04-02T14:20:57.555 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-04-02T14:20:57.608 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-04-02T14:20:57.639 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-04-02T14:20:57.686 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-04-02T14:20:57.783 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-04-02T14:20:57.900 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-04-02T14:20:58.635 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-04-02T14:20:58.688 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-04-02T14:20:58.722 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-04-02T14:20:58.770 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-04-02T14:20:58.856 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-04-02T14:20:58.889 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-04-02T14:20:59.066 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-04-02T14:20:59.095 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-04-02T14:20:59.180 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-04-02T14:20:59.819 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-04-02T14:20:59.925 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-04-02T14:21:00.672 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2024-04-02T14:21:00.780 INFO:teuthology.orchestra.run.smithi177.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-04-02T14:21:00.814 INFO:teuthology.orchestra.run.smithi177.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-04-02T14:21:00.853 INFO:teuthology.orchestra.run.smithi177.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-04-02T14:21:00.863 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-04-02T14:21:00.911 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-04-02T14:21:00.973 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-04-02T14:21:01.007 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-04-02T14:21:01.032 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-04-02T14:21:01.074 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-04-02T14:21:01.185 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-04-02T14:21:01.270 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-04-02T14:21:01.300 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-04-02T14:21:01.344 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-04-02T14:21:01.368 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-04-02T14:21:01.395 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-04-02T14:21:01.450 INFO:teuthology.orchestra.run.smithi177.stdout: Upgrading : librados2-2:19.0.0-2609.g191d963a.el9.x86_64 10/198 2024-04-02T14:21:01.486 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2024-04-02T14:21:01.570 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: librados2-2:19.0.0-2609.g191d963a.el9.x86_64 10/198 2024-04-02T14:21:01.601 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libcephfs2-2:19.0.0-2609.g191d963a.el9.x86_64 11/198 2024-04-02T14:21:01.624 INFO:teuthology.orchestra.run.smithi090.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-04-02T14:21:01.680 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-04-02T14:21:01.681 INFO:teuthology.orchestra.run.smithi090.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-04-02T14:21:01.692 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: libcephfs2-2:19.0.0-2609.g191d963a.el9.x86_64 11/198 2024-04-02T14:21:01.714 INFO:teuthology.orchestra.run.smithi092.stdout: Upgrading : librbd1-2:19.0.0-2609.g191d963a.el9.x86_64 124/198 2024-04-02T14:21:01.719 INFO:teuthology.orchestra.run.smithi090.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-04-02T14:21:01.721 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-rados-2:19.0.0-2609.g191d963a.el9.x86_64 12/198 2024-04-02T14:21:01.758 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 13/198 2024-04-02T14:21:01.762 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-04-02T14:21:01.805 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: librbd1-2:19.0.0-2609.g191d963a.el9.x86_64 124/198 2024-04-02T14:21:01.836 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-rbd-2:19.0.0-2609.g191d963a.el9.x86_64 125/198 2024-04-02T14:21:01.859 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-04-02T14:21:01.880 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-04-02T14:21:01.947 INFO:teuthology.orchestra.run.smithi177.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-04-02T14:21:01.955 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-04-02T14:21:02.003 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : postgresql-13.14-1.el9.x86_64 15/198 2024-04-02T14:21:02.006 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-04-02T14:21:02.031 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libcephsqlite-2:19.0.0-2609.g191d963a.el9.x86_64 16/198 2024-04-02T14:21:02.035 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-04-02T14:21:02.117 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2609.g191d963a.el9.x86_64 16/198 2024-04-02T14:21:02.140 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libradosstriper1-2:19.0.0-2609.g191d963a.el9.x86 17/198 2024-04-02T14:21:02.154 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-04-02T14:21:02.187 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2609.g191d963a.el9.x86 17/198 2024-04-02T14:21:02.227 INFO:teuthology.orchestra.run.smithi177.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-04-02T14:21:02.250 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 19/198 2024-04-02T14:21:02.265 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 19/198 2024-04-02T14:21:02.340 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-04-02T14:21:02.375 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : librgw2-2:19.0.0-2609.g191d963a.el9.x86_64 128/198 2024-04-02T14:21:02.441 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 20/198 2024-04-02T14:21:02.446 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: librgw2-2:19.0.0-2609.g191d963a.el9.x86_64 128/198 2024-04-02T14:21:02.469 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-rgw-2:19.0.0-2609.g191d963a.el9.x86_64 129/198 2024-04-02T14:21:02.472 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 20/198 2024-04-02T14:21:02.504 INFO:teuthology.orchestra.run.smithi177.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-04-02T14:21:02.581 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-04-02T14:21:02.607 INFO:teuthology.orchestra.run.smithi177.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-04-02T14:21:02.698 INFO:teuthology.orchestra.run.smithi090.stdout: Upgrading : librados2-2:19.0.0-2609.g191d963a.el9.x86_64 10/198 2024-04-02T14:21:02.732 INFO:teuthology.orchestra.run.smithi177.stdout: Upgrading : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-04-02T14:21:02.791 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-04-02T14:21:02.860 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: librados2-2:19.0.0-2609.g191d963a.el9.x86_64 10/198 2024-04-02T14:21:02.863 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-04-02T14:21:02.895 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libcephfs2-2:19.0.0-2609.g191d963a.el9.x86_64 11/198 2024-04-02T14:21:02.909 INFO:teuthology.orchestra.run.smithi177.stdout: Upgrading : NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-04-02T14:21:03.028 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: libcephfs2-2:19.0.0-2609.g191d963a.el9.x86_64 11/198 2024-04-02T14:21:03.069 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-rados-2:19.0.0-2609.g191d963a.el9.x86_64 12/198 2024-04-02T14:21:03.125 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-04-02T14:21:03.155 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 13/198 2024-04-02T14:21:03.232 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-04-02T14:21:03.318 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-04-02T14:21:03.358 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-04-02T14:21:03.384 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-04-02T14:21:03.420 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-04-02T14:21:03.473 INFO:teuthology.orchestra.run.smithi090.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-04-02T14:21:03.492 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-04-02T14:21:03.535 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : postgresql-13.14-1.el9.x86_64 15/198 2024-04-02T14:21:03.538 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-04-02T14:21:03.569 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libcephsqlite-2:19.0.0-2609.g191d963a.el9.x86_64 16/198 2024-04-02T14:21:03.577 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-04-02T14:21:03.640 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-04-02T14:21:03.685 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-04-02T14:21:03.724 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2609.g191d963a.el9.x86_64 16/198 2024-04-02T14:21:03.729 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-04-02T14:21:03.753 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libradosstriper1-2:19.0.0-2609.g191d963a.el9.x86 17/198 2024-04-02T14:21:03.761 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-04-02T14:21:03.789 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-04-02T14:21:03.859 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-04-02T14:21:03.894 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-ceph-common-2:19.0.0-2609.g191d963a.el9. 39/198 2024-04-02T14:21:03.909 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2609.g191d963a.el9.x86 17/198 2024-04-02T14:21:03.938 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-ceph-argparse-2:19.0.0-2609.g191d963a.el 40/198 2024-04-02T14:21:03.966 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-cephfs-2:19.0.0-2609.g191d963a.el9.x86_6 41/198 2024-04-02T14:21:03.972 INFO:teuthology.orchestra.run.smithi090.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-04-02T14:21:04.006 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 19/198 2024-04-02T14:21:04.035 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-04-02T14:21:04.065 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 19/198 2024-04-02T14:21:04.081 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-04-02T14:21:04.154 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-04-02T14:21:04.296 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-04-02T14:21:04.333 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 20/198 2024-04-02T14:21:04.381 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 20/198 2024-04-02T14:21:04.471 INFO:teuthology.orchestra.run.smithi090.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-04-02T14:21:04.535 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-04-02T14:21:04.813 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-common-2:19.0.0-2609.g191d963a.el9.x86_64 130/198 2024-04-02T14:21:04.849 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-common-2:19.0.0-2609.g191d963a.el9.x86_64 130/198 2024-04-02T14:21:04.898 INFO:teuthology.orchestra.run.smithi090.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-04-02T14:21:04.949 INFO:teuthology.orchestra.run.smithi090.stdout: Upgrading : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-04-02T14:21:05.047 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-04-02T14:21:05.063 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-04-02T14:21:05.145 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-04-02T14:21:05.217 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-04-02T14:21:05.273 INFO:teuthology.orchestra.run.smithi177.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-04-02T14:21:05.283 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-04-02T14:21:05.313 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 51/198 2024-04-02T14:21:05.328 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-04-02T14:21:05.354 INFO:teuthology.orchestra.run.smithi177.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-04-02T14:21:05.369 INFO:teuthology.orchestra.run.smithi090.stdout: Upgrading : NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-04-02T14:21:05.389 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-common-2:19.0.0-2609.g191d963a.el9.x86_64 130/198 2024-04-02T14:21:05.392 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : librados-devel-2:19.0.0-2609.g191d963a.el9.x86_6 53/198 2024-04-02T14:21:05.421 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-04-02T14:21:05.427 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-base-2:19.0.0-2609.g191d963a.el9.x86_64 131/198 2024-04-02T14:21:05.500 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-base-2:19.0.0-2609.g191d963a.el9.x86_64 131/198 2024-04-02T14:21:05.500 INFO:teuthology.orchestra.run.smithi092.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-02T14:21:05.500 INFO:teuthology.orchestra.run.smithi092.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-02T14:21:05.500 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-04-02T14:21:05.529 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-selinux-2:19.0.0-2609.g191d963a.el9.x86_64 132/198 2024-04-02T14:21:05.644 INFO:teuthology.orchestra.run.smithi177.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-04-02T14:21:05.746 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-04-02T14:21:05.826 INFO:teuthology.orchestra.run.smithi177.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-04-02T14:21:05.882 INFO:teuthology.orchestra.run.smithi177.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-04-02T14:21:05.926 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-04-02T14:21:05.947 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-04-02T14:21:06.068 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-04-02T14:21:06.075 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-04-02T14:21:06.101 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-04-02T14:21:06.129 INFO:teuthology.orchestra.run.smithi177.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-04-02T14:21:06.162 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-04-02T14:21:06.184 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-04-02T14:21:06.223 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-04-02T14:21:06.257 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-04-02T14:21:06.288 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-04-02T14:21:06.341 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-04-02T14:21:06.345 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-04-02T14:21:06.388 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-04-02T14:21:06.410 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-04-02T14:21:06.432 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-04-02T14:21:06.461 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-04-02T14:21:06.475 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-04-02T14:21:06.517 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-04-02T14:21:06.551 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-04-02T14:21:06.599 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-04-02T14:21:06.651 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-04-02T14:21:06.660 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-04-02T14:21:06.690 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-04-02T14:21:06.719 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-04-02T14:21:06.726 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-04-02T14:21:06.761 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-04-02T14:21:06.773 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-04-02T14:21:06.855 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-04-02T14:21:06.886 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 74/198 2024-04-02T14:21:06.894 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-04-02T14:21:06.926 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-04-02T14:21:06.933 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-ceph-common-2:19.0.0-2609.g191d963a.el9. 39/198 2024-04-02T14:21:06.952 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-04-02T14:21:06.977 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-04-02T14:21:06.988 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-ceph-argparse-2:19.0.0-2609.g191d963a.el 40/198 2024-04-02T14:21:07.024 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-cephfs-2:19.0.0-2609.g191d963a.el9.x86_6 41/198 2024-04-02T14:21:07.049 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-04-02T14:21:07.142 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-04-02T14:21:07.176 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-04-02T14:21:07.201 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-04-02T14:21:07.276 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-04-02T14:21:07.306 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-04-02T14:21:07.318 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-04-02T14:21:07.366 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-04-02T14:21:07.421 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-04-02T14:21:07.486 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-04-02T14:21:07.547 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-04-02T14:21:07.604 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-04-02T14:21:07.655 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-04-02T14:21:07.782 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-04-02T14:21:07.828 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 87/198 2024-04-02T14:21:07.877 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-04-02T14:21:07.908 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-04-02T14:21:07.936 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-04-02T14:21:07.978 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-04-02T14:21:08.018 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-04-02T14:21:08.049 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-04-02T14:21:08.079 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-04-02T14:21:08.155 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-04-02T14:21:08.190 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-04-02T14:21:08.223 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-04-02T14:21:08.232 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-04-02T14:21:08.264 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-04-02T14:21:08.307 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-04-02T14:21:08.328 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-04-02T14:21:08.395 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-04-02T14:21:08.478 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-04-02T14:21:08.508 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-04-02T14:21:08.521 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-04-02T14:21:08.553 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-04-02T14:21:08.556 INFO:teuthology.orchestra.run.smithi090.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-04-02T14:21:08.590 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-04-02T14:21:08.600 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-04-02T14:21:08.643 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 51/198 2024-04-02T14:21:08.697 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-04-02T14:21:08.708 INFO:teuthology.orchestra.run.smithi090.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-04-02T14:21:08.754 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-04-02T14:21:08.757 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : librados-devel-2:19.0.0-2609.g191d963a.el9.x86_6 53/198 2024-04-02T14:21:08.806 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-04-02T14:21:09.338 INFO:teuthology.orchestra.run.smithi090.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-04-02T14:21:09.473 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-04-02T14:21:09.518 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-04-02T14:21:09.551 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-04-02T14:21:09.579 INFO:teuthology.orchestra.run.smithi090.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-04-02T14:21:09.598 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-04-02T14:21:09.675 INFO:teuthology.orchestra.run.smithi090.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-04-02T14:21:09.678 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-04-02T14:21:09.718 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-04-02T14:21:09.745 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-04-02T14:21:09.895 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-04-02T14:21:09.925 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-04-02T14:21:09.981 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-04-02T14:21:10.002 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-04-02T14:21:10.039 INFO:teuthology.orchestra.run.smithi090.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-04-02T14:21:10.118 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-04-02T14:21:10.278 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-04-02T14:21:10.323 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-04-02T14:21:10.373 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-04-02T14:21:10.453 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-04-02T14:21:10.496 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-04-02T14:21:10.560 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-04-02T14:21:10.630 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-04-02T14:21:10.638 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-04-02T14:21:10.739 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-04-02T14:21:10.839 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-04-02T14:21:10.905 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-04-02T14:21:10.957 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-04-02T14:21:11.002 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-04-02T14:21:11.147 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-04-02T14:21:11.174 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 74/198 2024-04-02T14:21:11.227 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-04-02T14:21:11.266 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-04-02T14:21:11.290 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-04-02T14:21:11.509 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-04-02T14:21:11.681 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-04-02T14:21:11.823 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-04-02T14:21:11.878 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-04-02T14:21:11.940 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-04-02T14:21:11.967 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-04-02T14:21:12.025 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-04-02T14:21:12.051 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-04-02T14:21:12.056 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-04-02T14:21:12.087 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-04-02T14:21:12.115 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-04-02T14:21:12.302 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-04-02T14:21:12.367 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-04-02T14:21:12.424 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-04-02T14:21:12.461 INFO:teuthology.orchestra.run.smithi177.stdout: Upgrading : librbd1-2:19.0.0-2609.g191d963a.el9.x86_64 124/198 2024-04-02T14:21:12.551 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: librbd1-2:19.0.0-2609.g191d963a.el9.x86_64 124/198 2024-04-02T14:21:12.551 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-04-02T14:21:12.574 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-rbd-2:19.0.0-2609.g191d963a.el9.x86_64 125/198 2024-04-02T14:21:12.606 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 87/198 2024-04-02T14:21:12.610 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-04-02T14:21:12.647 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-04-02T14:21:12.686 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-04-02T14:21:12.743 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-04-02T14:21:12.791 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-04-02T14:21:12.858 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-04-02T14:21:12.916 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-04-02T14:21:12.965 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-04-02T14:21:13.069 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-04-02T14:21:13.093 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-04-02T14:21:13.097 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : librgw2-2:19.0.0-2609.g191d963a.el9.x86_64 128/198 2024-04-02T14:21:13.145 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-04-02T14:21:13.160 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: librgw2-2:19.0.0-2609.g191d963a.el9.x86_64 128/198 2024-04-02T14:21:13.183 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-rgw-2:19.0.0-2609.g191d963a.el9.x86_64 129/198 2024-04-02T14:21:13.190 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-04-02T14:21:13.255 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-04-02T14:21:13.301 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-04-02T14:21:13.431 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-04-02T14:21:13.544 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-04-02T14:21:13.597 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-04-02T14:21:13.682 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-04-02T14:21:13.751 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-04-02T14:21:13.904 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-04-02T14:21:13.966 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-04-02T14:21:14.939 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-04-02T14:21:15.000 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-04-02T14:21:15.042 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-04-02T14:21:15.113 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-04-02T14:21:15.234 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-04-02T14:21:15.270 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-04-02T14:21:15.382 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-common-2:19.0.0-2609.g191d963a.el9.x86_64 130/198 2024-04-02T14:21:15.429 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-common-2:19.0.0-2609.g191d963a.el9.x86_64 130/198 2024-04-02T14:21:15.502 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-04-02T14:21:15.533 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-04-02T14:21:15.647 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-04-02T14:21:15.964 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-common-2:19.0.0-2609.g191d963a.el9.x86_64 130/198 2024-04-02T14:21:15.999 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-base-2:19.0.0-2609.g191d963a.el9.x86_64 131/198 2024-04-02T14:21:16.063 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-base-2:19.0.0-2609.g191d963a.el9.x86_64 131/198 2024-04-02T14:21:16.063 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-02T14:21:16.064 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-02T14:21:16.064 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-02T14:21:16.092 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-selinux-2:19.0.0-2609.g191d963a.el9.x86_64 132/198 2024-04-02T14:21:16.445 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-04-02T14:21:16.630 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-04-02T14:21:17.369 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2609.g191d963a.el9.x86_64 132/198 2024-04-02T14:21:17.369 INFO:teuthology.orchestra.run.smithi092.stdout:skipping the directory /sys 2024-04-02T14:21:17.370 INFO:teuthology.orchestra.run.smithi092.stdout:skipping the directory /proc 2024-04-02T14:21:17.370 INFO:teuthology.orchestra.run.smithi092.stdout:skipping the directory /mnt 2024-04-02T14:21:17.370 INFO:teuthology.orchestra.run.smithi092.stdout:skipping the directory /var/tmp 2024-04-02T14:21:17.370 INFO:teuthology.orchestra.run.smithi092.stdout:skipping the directory /home 2024-04-02T14:21:17.370 INFO:teuthology.orchestra.run.smithi092.stdout:skipping the directory /root 2024-04-02T14:21:17.370 INFO:teuthology.orchestra.run.smithi092.stdout:skipping the directory /tmp 2024-04-02T14:21:17.370 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-04-02T14:21:17.539 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-volume-2:19.0.0-2609.g191d963a.el9.noarch 133/198 2024-04-02T14:21:17.555 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-volume-2:19.0.0-2609.g191d963a.el9.noarch 133/198 2024-04-02T14:21:17.556 INFO:teuthology.orchestra.run.smithi092.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-02T14:21:17.556 INFO:teuthology.orchestra.run.smithi092.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-02T14:21:17.556 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-04-02T14:21:18.170 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-04-02T14:21:18.304 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-04-02T14:21:18.342 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-04-02T14:21:18.382 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-04-02T14:21:18.424 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-04-02T14:21:18.820 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-04-02T14:21:18.853 INFO:teuthology.orchestra.run.smithi090.stdout: Upgrading : librbd1-2:19.0.0-2609.g191d963a.el9.x86_64 124/198 2024-04-02T14:21:18.973 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: librbd1-2:19.0.0-2609.g191d963a.el9.x86_64 124/198 2024-04-02T14:21:19.010 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-rbd-2:19.0.0-2609.g191d963a.el9.x86_64 125/198 2024-04-02T14:21:19.050 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-osd-2:19.0.0-2609.g191d963a.el9.x86_64 134/198 2024-04-02T14:21:19.075 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-04-02T14:21:19.075 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-osd-2:19.0.0-2609.g191d963a.el9.x86_64 134/198 2024-04-02T14:21:19.076 INFO:teuthology.orchestra.run.smithi092.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-02T14:21:19.076 INFO:teuthology.orchestra.run.smithi092.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-02T14:21:19.076 INFO:teuthology.orchestra.run.smithi092.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-02T14:21:19.076 INFO:teuthology.orchestra.run.smithi092.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-02T14:21:19.076 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-04-02T14:21:19.300 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-mds-2:19.0.0-2609.g191d963a.el9.x86_64 135/198 2024-04-02T14:21:19.323 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-mds-2:19.0.0-2609.g191d963a.el9.x86_64 135/198 2024-04-02T14:21:19.323 INFO:teuthology.orchestra.run.smithi092.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-02T14:21:19.323 INFO:teuthology.orchestra.run.smithi092.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-02T14:21:19.323 INFO:teuthology.orchestra.run.smithi092.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-02T14:21:19.323 INFO:teuthology.orchestra.run.smithi092.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-02T14:21:19.323 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-04-02T14:21:19.599 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-04-02T14:21:19.641 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : librgw2-2:19.0.0-2609.g191d963a.el9.x86_64 128/198 2024-04-02T14:21:19.742 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: librgw2-2:19.0.0-2609.g191d963a.el9.x86_64 128/198 2024-04-02T14:21:19.768 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-mon-2:19.0.0-2609.g191d963a.el9.x86_64 136/198 2024-04-02T14:21:19.769 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-rgw-2:19.0.0-2609.g191d963a.el9.x86_64 129/198 2024-04-02T14:21:19.790 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-mon-2:19.0.0-2609.g191d963a.el9.x86_64 136/198 2024-04-02T14:21:19.790 INFO:teuthology.orchestra.run.smithi092.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-02T14:21:19.790 INFO:teuthology.orchestra.run.smithi092.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-02T14:21:19.791 INFO:teuthology.orchestra.run.smithi092.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-02T14:21:19.791 INFO:teuthology.orchestra.run.smithi092.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-02T14:21:19.791 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-04-02T14:21:19.886 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-04-02T14:21:20.347 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: cephadm-2:19.0.0-2609.g191d963a.el9.noarch 138/198 2024-04-02T14:21:20.376 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : cephadm-2:19.0.0-2609.g191d963a.el9.noarch 138/198 2024-04-02T14:21:20.408 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2609.g191d963a.e 139/198 2024-04-02T14:21:20.469 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2609.g191d963a. 140/198 2024-04-02T14:21:20.499 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2609.g191d963a.el9.noa 141/198 2024-04-02T14:21:21.524 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2609.g191d963a.el9.noa 141/198 2024-04-02T14:21:21.559 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2609.g191d963a.el9.n 142/198 2024-04-02T14:21:22.411 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-common-2:19.0.0-2609.g191d963a.el9.x86_64 130/198 2024-04-02T14:21:22.451 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-common-2:19.0.0-2609.g191d963a.el9.x86_64 130/198 2024-04-02T14:21:22.516 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2609.g191d963a.el9.n 142/198 2024-04-02T14:21:22.551 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2609.g191 143/198 2024-04-02T14:21:22.650 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2609.g191 143/198 2024-04-02T14:21:22.806 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2609.g191d963a.el 144/198 2024-04-02T14:21:22.832 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-mgr-2:19.0.0-2609.g191d963a.el9.x86_64 145/198 2024-04-02T14:21:22.856 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2609.g191d963a.el9.x86_64 145/198 2024-04-02T14:21:22.856 INFO:teuthology.orchestra.run.smithi092.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-02T14:21:22.856 INFO:teuthology.orchestra.run.smithi092.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-02T14:21:22.856 INFO:teuthology.orchestra.run.smithi092.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-02T14:21:22.856 INFO:teuthology.orchestra.run.smithi092.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-02T14:21:22.856 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-04-02T14:21:22.904 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-mgr-rook-2:19.0.0-2609.g191d963a.el9.noarch 146/198 2024-04-02T14:21:22.915 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2609.g191d963a.el9.noarch 146/198 2024-04-02T14:21:22.968 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-2:19.0.0-2609.g191d963a.el9.x86_64 147/198 2024-04-02T14:21:23.084 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-common-2:19.0.0-2609.g191d963a.el9.x86_64 130/198 2024-04-02T14:21:23.138 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-base-2:19.0.0-2609.g191d963a.el9.x86_64 131/198 2024-04-02T14:21:23.315 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-base-2:19.0.0-2609.g191d963a.el9.x86_64 131/198 2024-04-02T14:21:23.315 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-02T14:21:23.315 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-02T14:21:23.315 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-02T14:21:23.356 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-selinux-2:19.0.0-2609.g191d963a.el9.x86_64 132/198 2024-04-02T14:21:23.839 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-04-02T14:21:23.872 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-radosgw-2:19.0.0-2609.g191d963a.el9.x86_64 149/198 2024-04-02T14:21:23.895 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2609.g191d963a.el9.x86_64 149/198 2024-04-02T14:21:23.895 INFO:teuthology.orchestra.run.smithi092.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-02T14:21:23.895 INFO:teuthology.orchestra.run.smithi092.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-02T14:21:23.895 INFO:teuthology.orchestra.run.smithi092.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-02T14:21:23.895 INFO:teuthology.orchestra.run.smithi092.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-02T14:21:23.895 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-04-02T14:21:23.939 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2609.g191d9 150/198 2024-04-02T14:21:23.961 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2609.g191d9 150/198 2024-04-02T14:21:23.962 INFO:teuthology.orchestra.run.smithi092.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-02T14:21:23.962 INFO:teuthology.orchestra.run.smithi092.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-02T14:21:23.962 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-04-02T14:21:24.007 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : cephfs-mirror-2:19.0.0-2609.g191d963a.el9.x86_64 151/198 2024-04-02T14:21:24.029 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2609.g191d963a.el9.x86_64 151/198 2024-04-02T14:21:24.029 INFO:teuthology.orchestra.run.smithi092.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-02T14:21:24.029 INFO:teuthology.orchestra.run.smithi092.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-02T14:21:24.029 INFO:teuthology.orchestra.run.smithi092.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-02T14:21:24.029 INFO:teuthology.orchestra.run.smithi092.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-02T14:21:24.029 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-04-02T14:21:24.332 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : rbd-mirror-2:19.0.0-2609.g191d963a.el9.x86_64 152/198 2024-04-02T14:21:24.355 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2609.g191d963a.el9.x86_64 152/198 2024-04-02T14:21:24.355 INFO:teuthology.orchestra.run.smithi092.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-02T14:21:24.355 INFO:teuthology.orchestra.run.smithi092.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-02T14:21:24.355 INFO:teuthology.orchestra.run.smithi092.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-02T14:21:24.355 INFO:teuthology.orchestra.run.smithi092.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-02T14:21:24.355 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-04-02T14:21:27.752 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2609.g191d963a.el9.x86_64 132/198 2024-04-02T14:21:27.752 INFO:teuthology.orchestra.run.smithi177.stdout:skipping the directory /sys 2024-04-02T14:21:27.752 INFO:teuthology.orchestra.run.smithi177.stdout:skipping the directory /proc 2024-04-02T14:21:27.752 INFO:teuthology.orchestra.run.smithi177.stdout:skipping the directory /mnt 2024-04-02T14:21:27.752 INFO:teuthology.orchestra.run.smithi177.stdout:skipping the directory /var/tmp 2024-04-02T14:21:27.753 INFO:teuthology.orchestra.run.smithi177.stdout:skipping the directory /home 2024-04-02T14:21:27.753 INFO:teuthology.orchestra.run.smithi177.stdout:skipping the directory /root 2024-04-02T14:21:27.753 INFO:teuthology.orchestra.run.smithi177.stdout:skipping the directory /tmp 2024-04-02T14:21:27.753 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-02T14:21:27.912 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-volume-2:19.0.0-2609.g191d963a.el9.noarch 133/198 2024-04-02T14:21:27.928 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-volume-2:19.0.0-2609.g191d963a.el9.noarch 133/198 2024-04-02T14:21:27.928 INFO:teuthology.orchestra.run.smithi177.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-02T14:21:27.928 INFO:teuthology.orchestra.run.smithi177.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-02T14:21:27.928 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-02T14:21:28.786 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-test-2:19.0.0-2609.g191d963a.el9.x86_64 153/198 2024-04-02T14:21:28.827 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : rbd-fuse-2:19.0.0-2609.g191d963a.el9.x86_64 154/198 2024-04-02T14:21:28.858 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : rbd-nbd-2:19.0.0-2609.g191d963a.el9.x86_64 155/198 2024-04-02T14:21:28.885 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-04-02T14:21:29.065 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : cephfs-shell-2:19.0.0-2609.g191d963a.el9.x86_64 157/198 2024-04-02T14:21:29.178 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 158/198 2024-04-02T14:21:29.226 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-fuse-2:19.0.0-2609.g191d963a.el9.x86_64 159/198 2024-04-02T14:21:29.265 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-04-02T14:21:29.318 INFO:teuthology.orchestra.run.smithi092.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-04-02T14:21:29.361 INFO:teuthology.orchestra.run.smithi092.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-04-02T14:21:29.394 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : libcephfs-devel-2:19.0.0-2609.g191d963a.el9.x86_ 163/198 2024-04-02T14:21:29.415 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-osd-2:19.0.0-2609.g191d963a.el9.x86_64 134/198 2024-04-02T14:21:29.426 INFO:teuthology.orchestra.run.smithi092.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-04-02T14:21:29.443 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-osd-2:19.0.0-2609.g191d963a.el9.x86_64 134/198 2024-04-02T14:21:29.443 INFO:teuthology.orchestra.run.smithi177.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-02T14:21:29.443 INFO:teuthology.orchestra.run.smithi177.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-02T14:21:29.443 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-02T14:21:29.443 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-02T14:21:29.443 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-02T14:21:29.455 INFO:teuthology.orchestra.run.smithi092.stdout: Upgrading : NetworkManager-team-1:1.47.2-1.el9.x86_64 165/198 2024-04-02T14:21:29.502 INFO:teuthology.orchestra.run.smithi092.stdout: Upgrading : NetworkManager-tui-1:1.47.2-1.el9.x86_64 166/198 2024-04-02T14:21:29.563 INFO:teuthology.orchestra.run.smithi092.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-04-02T14:21:29.604 INFO:teuthology.orchestra.run.smithi092.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-04-02T14:21:29.666 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-mds-2:19.0.0-2609.g191d963a.el9.x86_64 135/198 2024-04-02T14:21:29.690 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-mds-2:19.0.0-2609.g191d963a.el9.x86_64 135/198 2024-04-02T14:21:29.690 INFO:teuthology.orchestra.run.smithi177.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-02T14:21:29.690 INFO:teuthology.orchestra.run.smithi177.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-02T14:21:29.690 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-02T14:21:29.690 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-02T14:21:29.690 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-02T14:21:30.133 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-mon-2:19.0.0-2609.g191d963a.el9.x86_64 136/198 2024-04-02T14:21:30.156 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-mon-2:19.0.0-2609.g191d963a.el9.x86_64 136/198 2024-04-02T14:21:30.156 INFO:teuthology.orchestra.run.smithi177.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-02T14:21:30.156 INFO:teuthology.orchestra.run.smithi177.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-02T14:21:30.156 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-02T14:21:30.156 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-02T14:21:30.156 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-02T14:21:30.182 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 169/198 2024-04-02T14:21:30.218 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 169/198 2024-04-02T14:21:30.238 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 169/198 2024-04-02T14:21:30.252 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-04-02T14:21:30.645 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: cephadm-2:19.0.0-2609.g191d963a.el9.noarch 138/198 2024-04-02T14:21:30.675 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : cephadm-2:19.0.0-2609.g191d963a.el9.noarch 138/198 2024-04-02T14:21:30.707 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2609.g191d963a.e 139/198 2024-04-02T14:21:30.767 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2609.g191d963a. 140/198 2024-04-02T14:21:30.790 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2609.g191d963a.el9.noa 141/198 2024-04-02T14:21:31.049 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : cephfs-top-2:19.0.0-2609.g191d963a.el9.noarch 170/198 2024-04-02T14:21:31.122 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-04-02T14:21:31.162 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-04-02T14:21:31.243 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-04-02T14:21:31.275 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-04-02T14:21:31.308 INFO:teuthology.orchestra.run.smithi092.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-04-02T14:21:31.334 INFO:teuthology.orchestra.run.smithi092.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-04-02T14:21:31.429 INFO:teuthology.orchestra.run.smithi092.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-04-02T14:21:31.429 INFO:teuthology.orchestra.run.smithi092.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-04-02T14:21:31.448 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-04-02T14:21:31.481 INFO:teuthology.orchestra.run.smithi092.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-04-02T14:21:31.481 INFO:teuthology.orchestra.run.smithi092.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-04-02T14:21:31.555 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-04-02T14:21:31.580 INFO:teuthology.orchestra.run.smithi092.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-04-02T14:21:31.622 INFO:teuthology.orchestra.run.smithi092.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-04-02T14:21:31.647 INFO:teuthology.orchestra.run.smithi092.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-04-02T14:21:31.672 INFO:teuthology.orchestra.run.smithi092.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-04-02T14:21:31.705 INFO:teuthology.orchestra.run.smithi092.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-04-02T14:21:31.752 INFO:teuthology.orchestra.run.smithi092.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-04-02T14:21:31.793 INFO:teuthology.orchestra.run.smithi092.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-04-02T14:21:31.814 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2609.g191d963a.el9.noa 141/198 2024-04-02T14:21:31.818 INFO:teuthology.orchestra.run.smithi092.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-04-02T14:21:31.844 INFO:teuthology.orchestra.run.smithi092.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-04-02T14:21:31.850 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2609.g191d963a.el9.n 142/198 2024-04-02T14:21:31.869 INFO:teuthology.orchestra.run.smithi092.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-04-02T14:21:31.894 INFO:teuthology.orchestra.run.smithi092.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-04-02T14:21:31.901 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-04-02T14:21:31.901 INFO:teuthology.orchestra.run.smithi092.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-04-02T14:21:32.091 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-04-02T14:21:32.091 INFO:teuthology.orchestra.run.smithi092.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-04-02T14:21:32.155 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-04-02T14:21:32.187 INFO:teuthology.orchestra.run.smithi092.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-04-02T14:21:32.213 INFO:teuthology.orchestra.run.smithi092.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-04-02T14:21:32.239 INFO:teuthology.orchestra.run.smithi092.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-04-02T14:21:32.264 INFO:teuthology.orchestra.run.smithi092.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-04-02T14:21:32.329 INFO:teuthology.orchestra.run.smithi092.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-04-02T14:21:32.813 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2609.g191d963a.el9.n 142/198 2024-04-02T14:21:32.842 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2609.g191 143/198 2024-04-02T14:21:32.942 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2609.g191 143/198 2024-04-02T14:21:33.105 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2609.g191d963a.el 144/198 2024-04-02T14:21:33.135 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-mgr-2:19.0.0-2609.g191d963a.el9.x86_64 145/198 2024-04-02T14:21:33.158 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2609.g191d963a.el9.x86_64 145/198 2024-04-02T14:21:33.158 INFO:teuthology.orchestra.run.smithi177.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-02T14:21:33.158 INFO:teuthology.orchestra.run.smithi177.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-02T14:21:33.159 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-02T14:21:33.159 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-02T14:21:33.159 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-02T14:21:33.203 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-mgr-rook-2:19.0.0-2609.g191d963a.el9.noarch 146/198 2024-04-02T14:21:33.214 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2609.g191d963a.el9.noarch 146/198 2024-04-02T14:21:33.259 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-2:19.0.0-2609.g191d963a.el9.x86_64 147/198 2024-04-02T14:21:34.128 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-04-02T14:21:34.155 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-radosgw-2:19.0.0-2609.g191d963a.el9.x86_64 149/198 2024-04-02T14:21:34.177 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2609.g191d963a.el9.x86_64 149/198 2024-04-02T14:21:34.178 INFO:teuthology.orchestra.run.smithi177.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-02T14:21:34.178 INFO:teuthology.orchestra.run.smithi177.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-02T14:21:34.178 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-02T14:21:34.178 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-02T14:21:34.178 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-02T14:21:34.222 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2609.g191d9 150/198 2024-04-02T14:21:34.244 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2609.g191d9 150/198 2024-04-02T14:21:34.244 INFO:teuthology.orchestra.run.smithi177.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-02T14:21:34.244 INFO:teuthology.orchestra.run.smithi177.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-02T14:21:34.244 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-02T14:21:34.298 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : cephfs-mirror-2:19.0.0-2609.g191d963a.el9.x86_64 151/198 2024-04-02T14:21:34.320 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2609.g191d963a.el9.x86_64 151/198 2024-04-02T14:21:34.321 INFO:teuthology.orchestra.run.smithi177.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-02T14:21:34.321 INFO:teuthology.orchestra.run.smithi177.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-02T14:21:34.321 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-02T14:21:34.321 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-02T14:21:34.321 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-02T14:21:34.623 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : rbd-mirror-2:19.0.0-2609.g191d963a.el9.x86_64 152/198 2024-04-02T14:21:34.646 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2609.g191d963a.el9.x86_64 152/198 2024-04-02T14:21:34.646 INFO:teuthology.orchestra.run.smithi177.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-02T14:21:34.646 INFO:teuthology.orchestra.run.smithi177.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-02T14:21:34.646 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-02T14:21:34.646 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-02T14:21:34.646 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-02T14:21:37.778 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-04-02T14:21:37.779 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-2:19.0.0-2609.g191d963a.el9.x86_64 1/198 2024-04-02T14:21:37.779 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-base-2:19.0.0-2609.g191d963a.el9.x86_64 2/198 2024-04-02T14:21:37.779 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-common-2:19.0.0-2609.g191d963a.el9.x86_64 3/198 2024-04-02T14:21:37.779 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-fuse-2:19.0.0-2609.g191d963a.el9.x86_64 4/198 2024-04-02T14:21:37.779 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2609.g191d9 5/198 2024-04-02T14:21:37.779 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-mds-2:19.0.0-2609.g191d963a.el9.x86_64 6/198 2024-04-02T14:21:37.779 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-mgr-2:19.0.0-2609.g191d963a.el9.x86_64 7/198 2024-04-02T14:21:37.779 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-mon-2:19.0.0-2609.g191d963a.el9.x86_64 8/198 2024-04-02T14:21:37.779 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-osd-2:19.0.0-2609.g191d963a.el9.x86_64 9/198 2024-04-02T14:21:37.779 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-radosgw-2:19.0.0-2609.g191d963a.el9.x86_64 10/198 2024-04-02T14:21:37.779 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-selinux-2:19.0.0-2609.g191d963a.el9.x86_64 11/198 2024-04-02T14:21:37.779 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-test-2:19.0.0-2609.g191d963a.el9.x86_64 12/198 2024-04-02T14:21:37.779 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : cephfs-mirror-2:19.0.0-2609.g191d963a.el9.x86_64 13/198 2024-04-02T14:21:37.780 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : cephfs-shell-2:19.0.0-2609.g191d963a.el9.x86_64 14/198 2024-04-02T14:21:37.780 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libcephfs-devel-2:19.0.0-2609.g191d963a.el9.x86_ 15/198 2024-04-02T14:21:37.780 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libcephfs2-2:19.0.0-2609.g191d963a.el9.x86_64 16/198 2024-04-02T14:21:37.780 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libcephsqlite-2:19.0.0-2609.g191d963a.el9.x86_64 17/198 2024-04-02T14:21:37.780 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : librados-devel-2:19.0.0-2609.g191d963a.el9.x86_6 18/198 2024-04-02T14:21:37.780 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libradosstriper1-2:19.0.0-2609.g191d963a.el9.x86 19/198 2024-04-02T14:21:37.780 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : librgw2-2:19.0.0-2609.g191d963a.el9.x86_64 20/198 2024-04-02T14:21:37.780 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2609.g191d963a.el 21/198 2024-04-02T14:21:37.780 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-ceph-common-2:19.0.0-2609.g191d963a.el9. 22/198 2024-04-02T14:21:37.780 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-cephfs-2:19.0.0-2609.g191d963a.el9.x86_6 23/198 2024-04-02T14:21:37.780 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-rados-2:19.0.0-2609.g191d963a.el9.x86_64 24/198 2024-04-02T14:21:37.780 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-rbd-2:19.0.0-2609.g191d963a.el9.x86_64 25/198 2024-04-02T14:21:37.780 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-rgw-2:19.0.0-2609.g191d963a.el9.x86_64 26/198 2024-04-02T14:21:37.780 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : rbd-fuse-2:19.0.0-2609.g191d963a.el9.x86_64 27/198 2024-04-02T14:21:37.781 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : rbd-mirror-2:19.0.0-2609.g191d963a.el9.x86_64 28/198 2024-04-02T14:21:37.781 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : rbd-nbd-2:19.0.0-2609.g191d963a.el9.x86_64 29/198 2024-04-02T14:21:37.781 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2609.g191d963a. 30/198 2024-04-02T14:21:37.781 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2609.g191d963a.el9.noa 31/198 2024-04-02T14:21:37.781 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2609.g191d963a.el9.n 32/198 2024-04-02T14:21:37.781 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2609.g191 33/198 2024-04-02T14:21:37.781 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2609.g191d963a.el 34/198 2024-04-02T14:21:37.781 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2609.g191d963a.el9.noarch 35/198 2024-04-02T14:21:37.781 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2609.g191d963a.e 36/198 2024-04-02T14:21:37.781 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-volume-2:19.0.0-2609.g191d963a.el9.noarch 37/198 2024-04-02T14:21:37.781 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : cephadm-2:19.0.0-2609.g191d963a.el9.noarch 38/198 2024-04-02T14:21:37.781 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : cephfs-top-2:19.0.0-2609.g191d963a.el9.noarch 39/198 2024-04-02T14:21:37.781 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-04-02T14:21:37.781 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-04-02T14:21:37.782 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-04-02T14:21:37.782 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-04-02T14:21:37.782 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-04-02T14:21:37.782 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-04-02T14:21:37.782 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-04-02T14:21:37.782 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-04-02T14:21:37.782 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-04-02T14:21:37.782 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-04-02T14:21:37.782 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-04-02T14:21:37.782 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-04-02T14:21:37.782 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-04-02T14:21:37.782 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-04-02T14:21:37.782 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-04-02T14:21:37.782 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-04-02T14:21:37.783 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-04-02T14:21:37.783 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-04-02T14:21:37.783 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : postgresql-13.14-1.el9.x86_64 58/198 2024-04-02T14:21:37.783 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 59/198 2024-04-02T14:21:37.783 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 60/198 2024-04-02T14:21:37.783 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 61/198 2024-04-02T14:21:37.783 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 62/198 2024-04-02T14:21:37.783 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 63/198 2024-04-02T14:21:37.783 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 64/198 2024-04-02T14:21:37.783 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 65/198 2024-04-02T14:21:37.783 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 66/198 2024-04-02T14:21:37.783 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 67/198 2024-04-02T14:21:37.783 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 68/198 2024-04-02T14:21:37.783 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 69/198 2024-04-02T14:21:37.783 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 70/198 2024-04-02T14:21:37.784 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 71/198 2024-04-02T14:21:37.784 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 72/198 2024-04-02T14:21:37.784 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 73/198 2024-04-02T14:21:37.784 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 74/198 2024-04-02T14:21:37.784 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 75/198 2024-04-02T14:21:37.784 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 76/198 2024-04-02T14:21:37.784 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 77/198 2024-04-02T14:21:37.784 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 78/198 2024-04-02T14:21:37.784 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 79/198 2024-04-02T14:21:37.784 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 80/198 2024-04-02T14:21:37.784 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 81/198 2024-04-02T14:21:37.784 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 82/198 2024-04-02T14:21:37.784 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 83/198 2024-04-02T14:21:37.784 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 84/198 2024-04-02T14:21:37.784 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 85/198 2024-04-02T14:21:37.784 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 86/198 2024-04-02T14:21:37.785 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 87/198 2024-04-02T14:21:37.785 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 88/198 2024-04-02T14:21:37.785 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 89/198 2024-04-02T14:21:37.785 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 90/198 2024-04-02T14:21:37.785 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 91/198 2024-04-02T14:21:37.785 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-ply-3.11-14.el9.noarch 92/198 2024-04-02T14:21:37.785 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 93/198 2024-04-02T14:21:37.785 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 94/198 2024-04-02T14:21:37.785 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : jq-1.6-16.el9.x86_64 95/198 2024-04-02T14:21:37.785 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 96/198 2024-04-02T14:21:37.785 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 97/198 2024-04-02T14:21:37.785 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 98/198 2024-04-02T14:21:37.785 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 99/198 2024-04-02T14:21:37.785 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 100/198 2024-04-02T14:21:37.785 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 101/198 2024-04-02T14:21:37.786 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 102/198 2024-04-02T14:21:37.786 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 103/198 2024-04-02T14:21:37.786 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-04-02T14:21:37.786 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-04-02T14:21:37.786 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-04-02T14:21:37.786 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-04-02T14:21:37.786 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-04-02T14:21:37.786 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-04-02T14:21:37.786 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-04-02T14:21:37.786 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-04-02T14:21:37.786 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-04-02T14:21:37.787 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-04-02T14:21:37.787 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-04-02T14:21:37.787 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-04-02T14:21:37.787 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 116/198 2024-04-02T14:21:37.787 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-04-02T14:21:37.787 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-04-02T14:21:37.787 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-04-02T14:21:37.787 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-04-02T14:21:37.787 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-04-02T14:21:37.787 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-04-02T14:21:37.787 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-04-02T14:21:37.787 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 124/198 2024-04-02T14:21:37.788 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-04-02T14:21:37.788 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-04-02T14:21:37.788 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-04-02T14:21:37.788 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-04-02T14:21:37.788 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-04-02T14:21:37.788 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-04-02T14:21:37.788 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-04-02T14:21:37.788 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-04-02T14:21:37.788 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-04-02T14:21:37.788 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-04-02T14:21:37.788 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-04-02T14:21:37.788 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-04-02T14:21:37.788 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-04-02T14:21:37.788 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-04-02T14:21:37.789 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-04-02T14:21:37.789 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-04-02T14:21:37.789 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-04-02T14:21:37.789 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-04-02T14:21:37.789 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-04-02T14:21:37.789 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-04-02T14:21:37.789 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-04-02T14:21:37.789 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-04-02T14:21:37.789 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-04-02T14:21:37.789 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-04-02T14:21:37.789 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-04-02T14:21:37.789 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-04-02T14:21:37.789 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : librados2-2:19.0.0-2609.g191d963a.el9.x86_64 151/198 2024-04-02T14:21:37.789 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-04-02T14:21:37.789 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : librbd1-2:19.0.0-2609.g191d963a.el9.x86_64 153/198 2024-04-02T14:21:37.790 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-04-02T14:21:37.790 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : NetworkManager-1:1.47.2-1.el9.x86_64 155/198 2024-04-02T14:21:37.790 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 156/198 2024-04-02T14:21:37.790 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 157/198 2024-04-02T14:21:37.790 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 158/198 2024-04-02T14:21:37.790 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : NetworkManager-team-1:1.47.2-1.el9.x86_64 159/198 2024-04-02T14:21:37.790 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 160/198 2024-04-02T14:21:37.790 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : NetworkManager-tui-1:1.47.2-1.el9.x86_64 161/198 2024-04-02T14:21:37.790 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 162/198 2024-04-02T14:21:37.790 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 163/198 2024-04-02T14:21:37.790 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 164/198 2024-04-02T14:21:37.790 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-04-02T14:21:37.790 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-04-02T14:21:37.790 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-04-02T14:21:37.790 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-04-02T14:21:37.791 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-04-02T14:21:37.791 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-04-02T14:21:37.791 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-04-02T14:21:37.791 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-04-02T14:21:37.791 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-04-02T14:21:37.791 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-04-02T14:21:37.791 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-04-02T14:21:37.791 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-04-02T14:21:37.791 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-04-02T14:21:37.791 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-04-02T14:21:37.791 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-04-02T14:21:37.791 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-04-02T14:21:37.791 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-04-02T14:21:37.791 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-04-02T14:21:37.792 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-04-02T14:21:37.792 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-04-02T14:21:37.792 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-04-02T14:21:37.792 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-04-02T14:21:37.792 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-04-02T14:21:37.792 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-04-02T14:21:37.792 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-04-02T14:21:37.792 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-04-02T14:21:37.792 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-04-02T14:21:37.792 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-04-02T14:21:37.792 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-04-02T14:21:37.792 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-04-02T14:21:37.792 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-04-02T14:21:37.792 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-04-02T14:21:37.792 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-04-02T14:21:37.937 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2609.g191d963a.el9.x86_64 132/198 2024-04-02T14:21:37.938 INFO:teuthology.orchestra.run.smithi090.stdout:skipping the directory /sys 2024-04-02T14:21:37.938 INFO:teuthology.orchestra.run.smithi090.stdout:skipping the directory /proc 2024-04-02T14:21:37.938 INFO:teuthology.orchestra.run.smithi090.stdout:skipping the directory /mnt 2024-04-02T14:21:37.938 INFO:teuthology.orchestra.run.smithi090.stdout:skipping the directory /var/tmp 2024-04-02T14:21:37.938 INFO:teuthology.orchestra.run.smithi090.stdout:skipping the directory /home 2024-04-02T14:21:37.938 INFO:teuthology.orchestra.run.smithi090.stdout:skipping the directory /root 2024-04-02T14:21:37.938 INFO:teuthology.orchestra.run.smithi090.stdout:skipping the directory /tmp 2024-04-02T14:21:37.938 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-02T14:21:38.183 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-volume-2:19.0.0-2609.g191d963a.el9.noarch 133/198 2024-04-02T14:21:38.231 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-volume-2:19.0.0-2609.g191d963a.el9.noarch 133/198 2024-04-02T14:21:38.231 INFO:teuthology.orchestra.run.smithi090.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-02T14:21:38.231 INFO:teuthology.orchestra.run.smithi090.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-02T14:21:38.232 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-02T14:21:39.086 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-test-2:19.0.0-2609.g191d963a.el9.x86_64 153/198 2024-04-02T14:21:39.127 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : rbd-fuse-2:19.0.0-2609.g191d963a.el9.x86_64 154/198 2024-04-02T14:21:39.158 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : rbd-nbd-2:19.0.0-2609.g191d963a.el9.x86_64 155/198 2024-04-02T14:21:39.185 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-04-02T14:21:39.362 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : cephfs-shell-2:19.0.0-2609.g191d963a.el9.x86_64 157/198 2024-04-02T14:21:39.475 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 158/198 2024-04-02T14:21:39.515 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-fuse-2:19.0.0-2609.g191d963a.el9.x86_64 159/198 2024-04-02T14:21:39.555 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-04-02T14:21:39.608 INFO:teuthology.orchestra.run.smithi177.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-04-02T14:21:39.651 INFO:teuthology.orchestra.run.smithi177.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-04-02T14:21:39.675 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libcephfs-devel-2:19.0.0-2609.g191d963a.el9.x86_ 163/198 2024-04-02T14:21:39.724 INFO:teuthology.orchestra.run.smithi177.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-04-02T14:21:39.753 INFO:teuthology.orchestra.run.smithi177.stdout: Upgrading : NetworkManager-team-1:1.47.2-1.el9.x86_64 165/198 2024-04-02T14:21:39.799 INFO:teuthology.orchestra.run.smithi177.stdout: Upgrading : NetworkManager-tui-1:1.47.2-1.el9.x86_64 166/198 2024-04-02T14:21:39.869 INFO:teuthology.orchestra.run.smithi177.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-04-02T14:21:39.935 INFO:teuthology.orchestra.run.smithi177.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-04-02T14:21:39.945 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-osd-2:19.0.0-2609.g191d963a.el9.x86_64 134/198 2024-04-02T14:21:40.025 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-osd-2:19.0.0-2609.g191d963a.el9.x86_64 134/198 2024-04-02T14:21:40.025 INFO:teuthology.orchestra.run.smithi090.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-02T14:21:40.025 INFO:teuthology.orchestra.run.smithi090.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-02T14:21:40.025 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-02T14:21:40.025 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-02T14:21:40.025 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-02T14:21:40.287 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-mds-2:19.0.0-2609.g191d963a.el9.x86_64 135/198 2024-04-02T14:21:40.353 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-mds-2:19.0.0-2609.g191d963a.el9.x86_64 135/198 2024-04-02T14:21:40.353 INFO:teuthology.orchestra.run.smithi090.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-02T14:21:40.353 INFO:teuthology.orchestra.run.smithi090.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-02T14:21:40.353 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-02T14:21:40.354 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-02T14:21:40.354 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-02T14:21:40.467 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 169/198 2024-04-02T14:21:40.517 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 169/198 2024-04-02T14:21:40.536 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 169/198 2024-04-02T14:21:40.863 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-mon-2:19.0.0-2609.g191d963a.el9.x86_64 136/198 2024-04-02T14:21:40.932 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-mon-2:19.0.0-2609.g191d963a.el9.x86_64 136/198 2024-04-02T14:21:40.932 INFO:teuthology.orchestra.run.smithi090.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-02T14:21:40.932 INFO:teuthology.orchestra.run.smithi090.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-02T14:21:40.932 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-02T14:21:40.932 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-02T14:21:40.932 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-02T14:21:41.049 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-04-02T14:21:41.054 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-04-02T14:21:41.054 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-04-02T14:21:41.054 INFO:teuthology.orchestra.run.smithi092.stdout:Upgraded: 2024-04-02T14:21:41.054 INFO:teuthology.orchestra.run.smithi092.stdout: NetworkManager-1:1.47.2-1.el9.x86_64 2024-04-02T14:21:41.055 INFO:teuthology.orchestra.run.smithi092.stdout: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 2024-04-02T14:21:41.055 INFO:teuthology.orchestra.run.smithi092.stdout: NetworkManager-team-1:1.47.2-1.el9.x86_64 2024-04-02T14:21:41.055 INFO:teuthology.orchestra.run.smithi092.stdout: NetworkManager-tui-1:1.47.2-1.el9.x86_64 2024-04-02T14:21:41.055 INFO:teuthology.orchestra.run.smithi092.stdout: acl-2.3.1-4.el9.x86_64 2024-04-02T14:21:41.055 INFO:teuthology.orchestra.run.smithi092.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-04-02T14:21:41.055 INFO:teuthology.orchestra.run.smithi092.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-04-02T14:21:41.055 INFO:teuthology.orchestra.run.smithi092.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-04-02T14:21:41.055 INFO:teuthology.orchestra.run.smithi092.stdout: libacl-2.3.1-4.el9.x86_64 2024-04-02T14:21:41.055 INFO:teuthology.orchestra.run.smithi092.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-04-02T14:21:41.055 INFO:teuthology.orchestra.run.smithi092.stdout: libblkid-2.37.4-18.el9.x86_64 2024-04-02T14:21:41.055 INFO:teuthology.orchestra.run.smithi092.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-04-02T14:21:41.055 INFO:teuthology.orchestra.run.smithi092.stdout: libmount-2.37.4-18.el9.x86_64 2024-04-02T14:21:41.055 INFO:teuthology.orchestra.run.smithi092.stdout: librados2-2:19.0.0-2609.g191d963a.el9.x86_64 2024-04-02T14:21:41.055 INFO:teuthology.orchestra.run.smithi092.stdout: librbd1-2:19.0.0-2609.g191d963a.el9.x86_64 2024-04-02T14:21:41.056 INFO:teuthology.orchestra.run.smithi092.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-04-02T14:21:41.056 INFO:teuthology.orchestra.run.smithi092.stdout: libuuid-2.37.4-18.el9.x86_64 2024-04-02T14:21:41.056 INFO:teuthology.orchestra.run.smithi092.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-04-02T14:21:41.056 INFO:teuthology.orchestra.run.smithi092.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-04-02T14:21:41.056 INFO:teuthology.orchestra.run.smithi092.stdout: util-linux-2.37.4-18.el9.x86_64 2024-04-02T14:21:41.056 INFO:teuthology.orchestra.run.smithi092.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-04-02T14:21:41.056 INFO:teuthology.orchestra.run.smithi092.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-04-02T14:21:41.056 INFO:teuthology.orchestra.run.smithi092.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-04-02T14:21:41.056 INFO:teuthology.orchestra.run.smithi092.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-04-02T14:21:41.056 INFO:teuthology.orchestra.run.smithi092.stdout:Installed: 2024-04-02T14:21:41.056 INFO:teuthology.orchestra.run.smithi092.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-02T14:21:41.056 INFO:teuthology.orchestra.run.smithi092.stdout: automake-1.16.2-8.el9.noarch 2024-04-02T14:21:41.056 INFO:teuthology.orchestra.run.smithi092.stdout: bison-3.7.4-5.el9.x86_64 2024-04-02T14:21:41.056 INFO:teuthology.orchestra.run.smithi092.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-02T14:21:41.057 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-2:19.0.0-2609.g191d963a.el9.x86_64 2024-04-02T14:21:41.057 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-base-2:19.0.0-2609.g191d963a.el9.x86_64 2024-04-02T14:21:41.057 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-common-2:19.0.0-2609.g191d963a.el9.x86_64 2024-04-02T14:21:41.057 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-fuse-2:19.0.0-2609.g191d963a.el9.x86_64 2024-04-02T14:21:41.057 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-grafana-dashboards-2:19.0.0-2609.g191d963a.el9.noarch 2024-04-02T14:21:41.057 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-immutable-object-cache-2:19.0.0-2609.g191d963a.el9.x86_64 2024-04-02T14:21:41.057 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mds-2:19.0.0-2609.g191d963a.el9.x86_64 2024-04-02T14:21:41.057 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mgr-2:19.0.0-2609.g191d963a.el9.x86_64 2024-04-02T14:21:41.057 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mgr-cephadm-2:19.0.0-2609.g191d963a.el9.noarch 2024-04-02T14:21:41.057 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mgr-dashboard-2:19.0.0-2609.g191d963a.el9.noarch 2024-04-02T14:21:41.057 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2609.g191d963a.el9.noarch 2024-04-02T14:21:41.057 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mgr-modules-core-2:19.0.0-2609.g191d963a.el9.noarch 2024-04-02T14:21:41.057 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mgr-rook-2:19.0.0-2609.g191d963a.el9.noarch 2024-04-02T14:21:41.057 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mon-2:19.0.0-2609.g191d963a.el9.x86_64 2024-04-02T14:21:41.058 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-osd-2:19.0.0-2609.g191d963a.el9.x86_64 2024-04-02T14:21:41.058 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-prometheus-alerts-2:19.0.0-2609.g191d963a.el9.noarch 2024-04-02T14:21:41.058 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-radosgw-2:19.0.0-2609.g191d963a.el9.x86_64 2024-04-02T14:21:41.058 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-selinux-2:19.0.0-2609.g191d963a.el9.x86_64 2024-04-02T14:21:41.058 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-test-2:19.0.0-2609.g191d963a.el9.x86_64 2024-04-02T14:21:41.058 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-volume-2:19.0.0-2609.g191d963a.el9.noarch 2024-04-02T14:21:41.058 INFO:teuthology.orchestra.run.smithi092.stdout: cephadm-2:19.0.0-2609.g191d963a.el9.noarch 2024-04-02T14:21:41.058 INFO:teuthology.orchestra.run.smithi092.stdout: cephfs-mirror-2:19.0.0-2609.g191d963a.el9.x86_64 2024-04-02T14:21:41.058 INFO:teuthology.orchestra.run.smithi092.stdout: cephfs-shell-2:19.0.0-2609.g191d963a.el9.x86_64 2024-04-02T14:21:41.058 INFO:teuthology.orchestra.run.smithi092.stdout: cephfs-top-2:19.0.0-2609.g191d963a.el9.noarch 2024-04-02T14:21:41.058 INFO:teuthology.orchestra.run.smithi092.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-02T14:21:41.058 INFO:teuthology.orchestra.run.smithi092.stdout: flex-2.6.4-9.el9.x86_64 2024-04-02T14:21:41.058 INFO:teuthology.orchestra.run.smithi092.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-02T14:21:41.059 INFO:teuthology.orchestra.run.smithi092.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-02T14:21:41.059 INFO:teuthology.orchestra.run.smithi092.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-02T14:21:41.059 INFO:teuthology.orchestra.run.smithi092.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-02T14:21:41.059 INFO:teuthology.orchestra.run.smithi092.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-02T14:21:41.059 INFO:teuthology.orchestra.run.smithi092.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-02T14:21:41.059 INFO:teuthology.orchestra.run.smithi092.stdout: jq-1.6-16.el9.x86_64 2024-04-02T14:21:41.059 INFO:teuthology.orchestra.run.smithi092.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-02T14:21:41.059 INFO:teuthology.orchestra.run.smithi092.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-02T14:21:41.059 INFO:teuthology.orchestra.run.smithi092.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-02T14:21:41.059 INFO:teuthology.orchestra.run.smithi092.stdout: libcephfs-devel-2:19.0.0-2609.g191d963a.el9.x86_64 2024-04-02T14:21:41.059 INFO:teuthology.orchestra.run.smithi092.stdout: libcephfs2-2:19.0.0-2609.g191d963a.el9.x86_64 2024-04-02T14:21:41.059 INFO:teuthology.orchestra.run.smithi092.stdout: libcephsqlite-2:19.0.0-2609.g191d963a.el9.x86_64 2024-04-02T14:21:41.059 INFO:teuthology.orchestra.run.smithi092.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-02T14:21:41.059 INFO:teuthology.orchestra.run.smithi092.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-02T14:21:41.059 INFO:teuthology.orchestra.run.smithi092.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-02T14:21:41.060 INFO:teuthology.orchestra.run.smithi092.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-02T14:21:41.060 INFO:teuthology.orchestra.run.smithi092.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-02T14:21:41.060 INFO:teuthology.orchestra.run.smithi092.stdout: librados-devel-2:19.0.0-2609.g191d963a.el9.x86_64 2024-04-02T14:21:41.060 INFO:teuthology.orchestra.run.smithi092.stdout: libradosstriper1-2:19.0.0-2609.g191d963a.el9.x86_64 2024-04-02T14:21:41.060 INFO:teuthology.orchestra.run.smithi092.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-02T14:21:41.060 INFO:teuthology.orchestra.run.smithi092.stdout: librgw2-2:19.0.0-2609.g191d963a.el9.x86_64 2024-04-02T14:21:41.060 INFO:teuthology.orchestra.run.smithi092.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-02T14:21:41.060 INFO:teuthology.orchestra.run.smithi092.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-02T14:21:41.060 INFO:teuthology.orchestra.run.smithi092.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-02T14:21:41.060 INFO:teuthology.orchestra.run.smithi092.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-02T14:21:41.060 INFO:teuthology.orchestra.run.smithi092.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-02T14:21:41.060 INFO:teuthology.orchestra.run.smithi092.stdout: lua-5.4.4-4.el9.x86_64 2024-04-02T14:21:41.060 INFO:teuthology.orchestra.run.smithi092.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-02T14:21:41.060 INFO:teuthology.orchestra.run.smithi092.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-02T14:21:41.061 INFO:teuthology.orchestra.run.smithi092.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-04-02T14:21:41.061 INFO:teuthology.orchestra.run.smithi092.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-02T14:21:41.061 INFO:teuthology.orchestra.run.smithi092.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-02T14:21:41.061 INFO:teuthology.orchestra.run.smithi092.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-04-02T14:21:41.061 INFO:teuthology.orchestra.run.smithi092.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-02T14:21:41.061 INFO:teuthology.orchestra.run.smithi092.stdout: postgresql-13.14-1.el9.x86_64 2024-04-02T14:21:41.061 INFO:teuthology.orchestra.run.smithi092.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-02T14:21:41.061 INFO:teuthology.orchestra.run.smithi092.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-02T14:21:41.061 INFO:teuthology.orchestra.run.smithi092.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-02T14:21:41.061 INFO:teuthology.orchestra.run.smithi092.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-02T14:21:41.061 INFO:teuthology.orchestra.run.smithi092.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-02T14:21:41.061 INFO:teuthology.orchestra.run.smithi092.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-02T14:21:41.061 INFO:teuthology.orchestra.run.smithi092.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-02T14:21:41.061 INFO:teuthology.orchestra.run.smithi092.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-02T14:21:41.061 INFO:teuthology.orchestra.run.smithi092.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-02T14:21:41.062 INFO:teuthology.orchestra.run.smithi092.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-02T14:21:41.062 INFO:teuthology.orchestra.run.smithi092.stdout: python3-ceph-argparse-2:19.0.0-2609.g191d963a.el9.x86_64 2024-04-02T14:21:41.062 INFO:teuthology.orchestra.run.smithi092.stdout: python3-ceph-common-2:19.0.0-2609.g191d963a.el9.x86_64 2024-04-02T14:21:41.062 INFO:teuthology.orchestra.run.smithi092.stdout: python3-cephfs-2:19.0.0-2609.g191d963a.el9.x86_64 2024-04-02T14:21:41.062 INFO:teuthology.orchestra.run.smithi092.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-02T14:21:41.062 INFO:teuthology.orchestra.run.smithi092.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-02T14:21:41.062 INFO:teuthology.orchestra.run.smithi092.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-02T14:21:41.062 INFO:teuthology.orchestra.run.smithi092.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-02T14:21:41.062 INFO:teuthology.orchestra.run.smithi092.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-02T14:21:41.062 INFO:teuthology.orchestra.run.smithi092.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-02T14:21:41.062 INFO:teuthology.orchestra.run.smithi092.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-02T14:21:41.062 INFO:teuthology.orchestra.run.smithi092.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-04-02T14:21:41.062 INFO:teuthology.orchestra.run.smithi092.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-04-02T14:21:41.063 INFO:teuthology.orchestra.run.smithi092.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-02T14:21:41.063 INFO:teuthology.orchestra.run.smithi092.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-02T14:21:41.063 INFO:teuthology.orchestra.run.smithi092.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-02T14:21:41.063 INFO:teuthology.orchestra.run.smithi092.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-02T14:21:41.063 INFO:teuthology.orchestra.run.smithi092.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-02T14:21:41.063 INFO:teuthology.orchestra.run.smithi092.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-02T14:21:41.063 INFO:teuthology.orchestra.run.smithi092.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-02T14:21:41.063 INFO:teuthology.orchestra.run.smithi092.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-02T14:21:41.063 INFO:teuthology.orchestra.run.smithi092.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-02T14:21:41.063 INFO:teuthology.orchestra.run.smithi092.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-02T14:21:41.063 INFO:teuthology.orchestra.run.smithi092.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-02T14:21:41.063 INFO:teuthology.orchestra.run.smithi092.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-02T14:21:41.063 INFO:teuthology.orchestra.run.smithi092.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-02T14:21:41.064 INFO:teuthology.orchestra.run.smithi092.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-02T14:21:41.064 INFO:teuthology.orchestra.run.smithi092.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-02T14:21:41.064 INFO:teuthology.orchestra.run.smithi092.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-02T14:21:41.064 INFO:teuthology.orchestra.run.smithi092.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-02T14:21:41.064 INFO:teuthology.orchestra.run.smithi092.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-02T14:21:41.064 INFO:teuthology.orchestra.run.smithi092.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-02T14:21:41.064 INFO:teuthology.orchestra.run.smithi092.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-02T14:21:41.064 INFO:teuthology.orchestra.run.smithi092.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-02T14:21:41.064 INFO:teuthology.orchestra.run.smithi092.stdout: python3-ply-3.11-14.el9.noarch 2024-04-02T14:21:41.064 INFO:teuthology.orchestra.run.smithi092.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-02T14:21:41.064 INFO:teuthology.orchestra.run.smithi092.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-02T14:21:41.064 INFO:teuthology.orchestra.run.smithi092.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-02T14:21:41.065 INFO:teuthology.orchestra.run.smithi092.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-02T14:21:41.065 INFO:teuthology.orchestra.run.smithi092.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-02T14:21:41.065 INFO:teuthology.orchestra.run.smithi092.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-02T14:21:41.065 INFO:teuthology.orchestra.run.smithi092.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-02T14:21:41.065 INFO:teuthology.orchestra.run.smithi092.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-02T14:21:41.065 INFO:teuthology.orchestra.run.smithi092.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-02T14:21:41.065 INFO:teuthology.orchestra.run.smithi092.stdout: python3-rados-2:19.0.0-2609.g191d963a.el9.x86_64 2024-04-02T14:21:41.065 INFO:teuthology.orchestra.run.smithi092.stdout: python3-rbd-2:19.0.0-2609.g191d963a.el9.x86_64 2024-04-02T14:21:41.065 INFO:teuthology.orchestra.run.smithi092.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-02T14:21:41.065 INFO:teuthology.orchestra.run.smithi092.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-02T14:21:41.065 INFO:teuthology.orchestra.run.smithi092.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-02T14:21:41.065 INFO:teuthology.orchestra.run.smithi092.stdout: python3-rgw-2:19.0.0-2609.g191d963a.el9.x86_64 2024-04-02T14:21:41.065 INFO:teuthology.orchestra.run.smithi092.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-02T14:21:41.065 INFO:teuthology.orchestra.run.smithi092.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-02T14:21:41.066 INFO:teuthology.orchestra.run.smithi092.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-02T14:21:41.066 INFO:teuthology.orchestra.run.smithi092.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-02T14:21:41.066 INFO:teuthology.orchestra.run.smithi092.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-02T14:21:41.066 INFO:teuthology.orchestra.run.smithi092.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-02T14:21:41.066 INFO:teuthology.orchestra.run.smithi092.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-02T14:21:41.066 INFO:teuthology.orchestra.run.smithi092.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-02T14:21:41.066 INFO:teuthology.orchestra.run.smithi092.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-02T14:21:41.066 INFO:teuthology.orchestra.run.smithi092.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-02T14:21:41.066 INFO:teuthology.orchestra.run.smithi092.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-02T14:21:41.066 INFO:teuthology.orchestra.run.smithi092.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-02T14:21:41.066 INFO:teuthology.orchestra.run.smithi092.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-02T14:21:41.066 INFO:teuthology.orchestra.run.smithi092.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-02T14:21:41.066 INFO:teuthology.orchestra.run.smithi092.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-02T14:21:41.066 INFO:teuthology.orchestra.run.smithi092.stdout: rbd-fuse-2:19.0.0-2609.g191d963a.el9.x86_64 2024-04-02T14:21:41.067 INFO:teuthology.orchestra.run.smithi092.stdout: rbd-mirror-2:19.0.0-2609.g191d963a.el9.x86_64 2024-04-02T14:21:41.067 INFO:teuthology.orchestra.run.smithi092.stdout: rbd-nbd-2:19.0.0-2609.g191d963a.el9.x86_64 2024-04-02T14:21:41.067 INFO:teuthology.orchestra.run.smithi092.stdout: re2-1:20211101-3.el9.x86_64 2024-04-02T14:21:41.067 INFO:teuthology.orchestra.run.smithi092.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-02T14:21:41.067 INFO:teuthology.orchestra.run.smithi092.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-02T14:21:41.067 INFO:teuthology.orchestra.run.smithi092.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-02T14:21:41.067 INFO:teuthology.orchestra.run.smithi092.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-02T14:21:41.067 INFO:teuthology.orchestra.run.smithi092.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-02T14:21:41.067 INFO:teuthology.orchestra.run.smithi092.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-04-02T14:21:41.067 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-04-02T14:21:41.067 INFO:teuthology.orchestra.run.smithi092.stdout:Complete! 2024-04-02T14:21:41.347 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : cephfs-top-2:19.0.0-2609.g191d963a.el9.noarch 170/198 2024-04-02T14:21:41.436 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-04-02T14:21:41.476 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-04-02T14:21:41.517 DEBUG:teuthology.parallel:result is None 2024-04-02T14:21:41.556 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-04-02T14:21:41.592 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-04-02T14:21:41.617 INFO:teuthology.orchestra.run.smithi177.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-04-02T14:21:41.643 INFO:teuthology.orchestra.run.smithi177.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-04-02T14:21:41.669 INFO:teuthology.orchestra.run.smithi177.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-04-02T14:21:41.669 INFO:teuthology.orchestra.run.smithi177.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-04-02T14:21:41.683 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: cephadm-2:19.0.0-2609.g191d963a.el9.noarch 138/198 2024-04-02T14:21:41.727 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : cephadm-2:19.0.0-2609.g191d963a.el9.noarch 138/198 2024-04-02T14:21:41.740 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-04-02T14:21:41.770 INFO:teuthology.orchestra.run.smithi177.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-04-02T14:21:41.770 INFO:teuthology.orchestra.run.smithi177.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-04-02T14:21:41.778 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2609.g191d963a.e 139/198 2024-04-02T14:21:41.864 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-04-02T14:21:41.880 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2609.g191d963a. 140/198 2024-04-02T14:21:41.889 INFO:teuthology.orchestra.run.smithi177.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-04-02T14:21:41.915 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2609.g191d963a.el9.noa 141/198 2024-04-02T14:21:41.922 INFO:teuthology.orchestra.run.smithi177.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-04-02T14:21:41.947 INFO:teuthology.orchestra.run.smithi177.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-04-02T14:21:41.973 INFO:teuthology.orchestra.run.smithi177.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-04-02T14:21:42.005 INFO:teuthology.orchestra.run.smithi177.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-04-02T14:21:42.045 INFO:teuthology.orchestra.run.smithi177.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-04-02T14:21:42.077 INFO:teuthology.orchestra.run.smithi177.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-04-02T14:21:42.102 INFO:teuthology.orchestra.run.smithi177.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-04-02T14:21:42.127 INFO:teuthology.orchestra.run.smithi177.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-04-02T14:21:42.169 INFO:teuthology.orchestra.run.smithi177.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-04-02T14:21:42.195 INFO:teuthology.orchestra.run.smithi177.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-04-02T14:21:42.201 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-04-02T14:21:42.201 INFO:teuthology.orchestra.run.smithi177.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-04-02T14:21:42.381 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-04-02T14:21:42.381 INFO:teuthology.orchestra.run.smithi177.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-04-02T14:21:42.464 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-04-02T14:21:42.495 INFO:teuthology.orchestra.run.smithi177.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-04-02T14:21:42.522 INFO:teuthology.orchestra.run.smithi177.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-04-02T14:21:42.548 INFO:teuthology.orchestra.run.smithi177.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-04-02T14:21:42.573 INFO:teuthology.orchestra.run.smithi177.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-04-02T14:21:42.636 INFO:teuthology.orchestra.run.smithi177.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-04-02T14:21:43.204 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2609.g191d963a.el9.noa 141/198 2024-04-02T14:21:43.251 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2609.g191d963a.el9.n 142/198 2024-04-02T14:21:44.351 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2609.g191d963a.el9.n 142/198 2024-04-02T14:21:44.384 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2609.g191 143/198 2024-04-02T14:21:44.526 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2609.g191 143/198 2024-04-02T14:21:44.708 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2609.g191d963a.el 144/198 2024-04-02T14:21:44.738 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-mgr-2:19.0.0-2609.g191d963a.el9.x86_64 145/198 2024-04-02T14:21:44.805 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2609.g191d963a.el9.x86_64 145/198 2024-04-02T14:21:44.805 INFO:teuthology.orchestra.run.smithi090.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-02T14:21:44.805 INFO:teuthology.orchestra.run.smithi090.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-02T14:21:44.805 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-02T14:21:44.806 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-02T14:21:44.806 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-02T14:21:44.856 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-mgr-rook-2:19.0.0-2609.g191d963a.el9.noarch 146/198 2024-04-02T14:21:44.891 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2609.g191d963a.el9.noarch 146/198 2024-04-02T14:21:44.950 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-2:19.0.0-2609.g191d963a.el9.x86_64 147/198 2024-04-02T14:21:45.928 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-04-02T14:21:45.966 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-radosgw-2:19.0.0-2609.g191d963a.el9.x86_64 149/198 2024-04-02T14:21:46.051 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2609.g191d963a.el9.x86_64 149/198 2024-04-02T14:21:46.051 INFO:teuthology.orchestra.run.smithi090.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-02T14:21:46.051 INFO:teuthology.orchestra.run.smithi090.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-02T14:21:46.052 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-02T14:21:46.052 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-02T14:21:46.052 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-02T14:21:46.100 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2609.g191d9 150/198 2024-04-02T14:21:46.169 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2609.g191d9 150/198 2024-04-02T14:21:46.169 INFO:teuthology.orchestra.run.smithi090.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-02T14:21:46.170 INFO:teuthology.orchestra.run.smithi090.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-02T14:21:46.170 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-02T14:21:46.252 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : cephfs-mirror-2:19.0.0-2609.g191d963a.el9.x86_64 151/198 2024-04-02T14:21:46.336 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2609.g191d963a.el9.x86_64 151/198 2024-04-02T14:21:46.336 INFO:teuthology.orchestra.run.smithi090.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-02T14:21:46.336 INFO:teuthology.orchestra.run.smithi090.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-02T14:21:46.336 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-02T14:21:46.337 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-02T14:21:46.337 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-02T14:21:46.712 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : rbd-mirror-2:19.0.0-2609.g191d963a.el9.x86_64 152/198 2024-04-02T14:21:46.782 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2609.g191d963a.el9.x86_64 152/198 2024-04-02T14:21:46.782 INFO:teuthology.orchestra.run.smithi090.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-02T14:21:46.782 INFO:teuthology.orchestra.run.smithi090.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-02T14:21:46.782 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-02T14:21:46.782 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-02T14:21:46.782 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-02T14:21:48.191 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-04-02T14:21:48.191 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-2:19.0.0-2609.g191d963a.el9.x86_64 1/198 2024-04-02T14:21:48.191 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-base-2:19.0.0-2609.g191d963a.el9.x86_64 2/198 2024-04-02T14:21:48.191 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-common-2:19.0.0-2609.g191d963a.el9.x86_64 3/198 2024-04-02T14:21:48.191 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-fuse-2:19.0.0-2609.g191d963a.el9.x86_64 4/198 2024-04-02T14:21:48.191 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2609.g191d9 5/198 2024-04-02T14:21:48.192 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-mds-2:19.0.0-2609.g191d963a.el9.x86_64 6/198 2024-04-02T14:21:48.192 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-mgr-2:19.0.0-2609.g191d963a.el9.x86_64 7/198 2024-04-02T14:21:48.192 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-mon-2:19.0.0-2609.g191d963a.el9.x86_64 8/198 2024-04-02T14:21:48.192 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-osd-2:19.0.0-2609.g191d963a.el9.x86_64 9/198 2024-04-02T14:21:48.192 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-radosgw-2:19.0.0-2609.g191d963a.el9.x86_64 10/198 2024-04-02T14:21:48.192 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-selinux-2:19.0.0-2609.g191d963a.el9.x86_64 11/198 2024-04-02T14:21:48.192 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-test-2:19.0.0-2609.g191d963a.el9.x86_64 12/198 2024-04-02T14:21:48.192 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : cephfs-mirror-2:19.0.0-2609.g191d963a.el9.x86_64 13/198 2024-04-02T14:21:48.192 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : cephfs-shell-2:19.0.0-2609.g191d963a.el9.x86_64 14/198 2024-04-02T14:21:48.192 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libcephfs-devel-2:19.0.0-2609.g191d963a.el9.x86_ 15/198 2024-04-02T14:21:48.192 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libcephfs2-2:19.0.0-2609.g191d963a.el9.x86_64 16/198 2024-04-02T14:21:48.192 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libcephsqlite-2:19.0.0-2609.g191d963a.el9.x86_64 17/198 2024-04-02T14:21:48.192 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : librados-devel-2:19.0.0-2609.g191d963a.el9.x86_6 18/198 2024-04-02T14:21:48.192 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libradosstriper1-2:19.0.0-2609.g191d963a.el9.x86 19/198 2024-04-02T14:21:48.193 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : librgw2-2:19.0.0-2609.g191d963a.el9.x86_64 20/198 2024-04-02T14:21:48.193 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2609.g191d963a.el 21/198 2024-04-02T14:21:48.193 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-ceph-common-2:19.0.0-2609.g191d963a.el9. 22/198 2024-04-02T14:21:48.193 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-cephfs-2:19.0.0-2609.g191d963a.el9.x86_6 23/198 2024-04-02T14:21:48.193 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-rados-2:19.0.0-2609.g191d963a.el9.x86_64 24/198 2024-04-02T14:21:48.193 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-rbd-2:19.0.0-2609.g191d963a.el9.x86_64 25/198 2024-04-02T14:21:48.193 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-rgw-2:19.0.0-2609.g191d963a.el9.x86_64 26/198 2024-04-02T14:21:48.193 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : rbd-fuse-2:19.0.0-2609.g191d963a.el9.x86_64 27/198 2024-04-02T14:21:48.193 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : rbd-mirror-2:19.0.0-2609.g191d963a.el9.x86_64 28/198 2024-04-02T14:21:48.193 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : rbd-nbd-2:19.0.0-2609.g191d963a.el9.x86_64 29/198 2024-04-02T14:21:48.193 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2609.g191d963a. 30/198 2024-04-02T14:21:48.193 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2609.g191d963a.el9.noa 31/198 2024-04-02T14:21:48.193 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2609.g191d963a.el9.n 32/198 2024-04-02T14:21:48.193 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2609.g191 33/198 2024-04-02T14:21:48.194 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2609.g191d963a.el 34/198 2024-04-02T14:21:48.194 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2609.g191d963a.el9.noarch 35/198 2024-04-02T14:21:48.194 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2609.g191d963a.e 36/198 2024-04-02T14:21:48.194 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-volume-2:19.0.0-2609.g191d963a.el9.noarch 37/198 2024-04-02T14:21:48.194 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : cephadm-2:19.0.0-2609.g191d963a.el9.noarch 38/198 2024-04-02T14:21:48.194 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : cephfs-top-2:19.0.0-2609.g191d963a.el9.noarch 39/198 2024-04-02T14:21:48.194 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-04-02T14:21:48.194 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-04-02T14:21:48.194 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-04-02T14:21:48.194 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-04-02T14:21:48.194 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-04-02T14:21:48.194 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-04-02T14:21:48.194 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-04-02T14:21:48.194 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-04-02T14:21:48.194 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-04-02T14:21:48.195 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-04-02T14:21:48.195 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-04-02T14:21:48.195 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-04-02T14:21:48.195 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-04-02T14:21:48.195 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-04-02T14:21:48.195 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-04-02T14:21:48.195 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-04-02T14:21:48.195 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-04-02T14:21:48.195 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-04-02T14:21:48.195 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : postgresql-13.14-1.el9.x86_64 58/198 2024-04-02T14:21:48.195 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 59/198 2024-04-02T14:21:48.195 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 60/198 2024-04-02T14:21:48.195 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 61/198 2024-04-02T14:21:48.195 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 62/198 2024-04-02T14:21:48.196 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 63/198 2024-04-02T14:21:48.196 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 64/198 2024-04-02T14:21:48.196 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 65/198 2024-04-02T14:21:48.196 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 66/198 2024-04-02T14:21:48.196 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 67/198 2024-04-02T14:21:48.196 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 68/198 2024-04-02T14:21:48.196 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 69/198 2024-04-02T14:21:48.196 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 70/198 2024-04-02T14:21:48.196 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 71/198 2024-04-02T14:21:48.196 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 72/198 2024-04-02T14:21:48.196 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 73/198 2024-04-02T14:21:48.196 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 74/198 2024-04-02T14:21:48.196 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 75/198 2024-04-02T14:21:48.196 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 76/198 2024-04-02T14:21:48.196 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 77/198 2024-04-02T14:21:48.197 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 78/198 2024-04-02T14:21:48.197 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 79/198 2024-04-02T14:21:48.197 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 80/198 2024-04-02T14:21:48.197 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 81/198 2024-04-02T14:21:48.197 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 82/198 2024-04-02T14:21:48.197 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 83/198 2024-04-02T14:21:48.197 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 84/198 2024-04-02T14:21:48.197 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 85/198 2024-04-02T14:21:48.197 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 86/198 2024-04-02T14:21:48.197 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 87/198 2024-04-02T14:21:48.197 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 88/198 2024-04-02T14:21:48.197 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 89/198 2024-04-02T14:21:48.197 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 90/198 2024-04-02T14:21:48.197 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 91/198 2024-04-02T14:21:48.198 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-ply-3.11-14.el9.noarch 92/198 2024-04-02T14:21:48.198 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 93/198 2024-04-02T14:21:48.198 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 94/198 2024-04-02T14:21:48.198 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : jq-1.6-16.el9.x86_64 95/198 2024-04-02T14:21:48.198 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 96/198 2024-04-02T14:21:48.198 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 97/198 2024-04-02T14:21:48.198 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 98/198 2024-04-02T14:21:48.198 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 99/198 2024-04-02T14:21:48.198 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 100/198 2024-04-02T14:21:48.198 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 101/198 2024-04-02T14:21:48.198 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 102/198 2024-04-02T14:21:48.198 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 103/198 2024-04-02T14:21:48.198 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-04-02T14:21:48.198 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-04-02T14:21:48.199 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-04-02T14:21:48.199 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-04-02T14:21:48.199 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-04-02T14:21:48.199 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-04-02T14:21:48.199 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-04-02T14:21:48.199 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-04-02T14:21:48.199 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-04-02T14:21:48.199 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-04-02T14:21:48.199 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-04-02T14:21:48.199 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-04-02T14:21:48.199 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 116/198 2024-04-02T14:21:48.199 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-04-02T14:21:48.199 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-04-02T14:21:48.199 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-04-02T14:21:48.200 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-04-02T14:21:48.200 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-04-02T14:21:48.200 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-04-02T14:21:48.200 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-04-02T14:21:48.200 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 124/198 2024-04-02T14:21:48.200 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-04-02T14:21:48.200 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-04-02T14:21:48.200 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-04-02T14:21:48.200 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-04-02T14:21:48.200 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-04-02T14:21:48.200 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-04-02T14:21:48.200 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-04-02T14:21:48.200 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-04-02T14:21:48.200 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-04-02T14:21:48.200 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-04-02T14:21:48.201 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-04-02T14:21:48.201 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-04-02T14:21:48.201 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-04-02T14:21:48.201 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-04-02T14:21:48.201 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-04-02T14:21:48.201 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-04-02T14:21:48.201 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-04-02T14:21:48.201 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-04-02T14:21:48.201 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-04-02T14:21:48.201 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-04-02T14:21:48.201 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-04-02T14:21:48.201 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-04-02T14:21:48.201 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-04-02T14:21:48.202 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-04-02T14:21:48.202 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-04-02T14:21:48.202 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-04-02T14:21:48.202 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : librados2-2:19.0.0-2609.g191d963a.el9.x86_64 151/198 2024-04-02T14:21:48.202 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-04-02T14:21:48.202 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : librbd1-2:19.0.0-2609.g191d963a.el9.x86_64 153/198 2024-04-02T14:21:48.202 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-04-02T14:21:48.202 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : NetworkManager-1:1.47.2-1.el9.x86_64 155/198 2024-04-02T14:21:48.202 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 156/198 2024-04-02T14:21:48.202 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 157/198 2024-04-02T14:21:48.202 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 158/198 2024-04-02T14:21:48.202 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : NetworkManager-team-1:1.47.2-1.el9.x86_64 159/198 2024-04-02T14:21:48.202 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 160/198 2024-04-02T14:21:48.202 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : NetworkManager-tui-1:1.47.2-1.el9.x86_64 161/198 2024-04-02T14:21:48.203 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 162/198 2024-04-02T14:21:48.203 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 163/198 2024-04-02T14:21:48.203 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 164/198 2024-04-02T14:21:48.203 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-04-02T14:21:48.203 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-04-02T14:21:48.203 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-04-02T14:21:48.203 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-04-02T14:21:48.203 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-04-02T14:21:48.203 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-04-02T14:21:48.203 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-04-02T14:21:48.203 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-04-02T14:21:48.203 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-04-02T14:21:48.203 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-04-02T14:21:48.203 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-04-02T14:21:48.204 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-04-02T14:21:48.204 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-04-02T14:21:48.204 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-04-02T14:21:48.204 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-04-02T14:21:48.204 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-04-02T14:21:48.204 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-04-02T14:21:48.204 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-04-02T14:21:48.204 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-04-02T14:21:48.204 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-04-02T14:21:48.204 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-04-02T14:21:48.204 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-04-02T14:21:48.204 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-04-02T14:21:48.204 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-04-02T14:21:48.204 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-04-02T14:21:48.204 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-04-02T14:21:48.205 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-04-02T14:21:48.205 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-04-02T14:21:48.205 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-04-02T14:21:48.205 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-04-02T14:21:48.205 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-04-02T14:21:48.205 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-04-02T14:21:48.205 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-04-02T14:21:51.162 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-04-02T14:21:51.162 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-02T14:21:51.162 INFO:teuthology.orchestra.run.smithi177.stdout:Upgraded: 2024-04-02T14:21:51.163 INFO:teuthology.orchestra.run.smithi177.stdout: NetworkManager-1:1.47.2-1.el9.x86_64 2024-04-02T14:21:51.163 INFO:teuthology.orchestra.run.smithi177.stdout: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 2024-04-02T14:21:51.163 INFO:teuthology.orchestra.run.smithi177.stdout: NetworkManager-team-1:1.47.2-1.el9.x86_64 2024-04-02T14:21:51.163 INFO:teuthology.orchestra.run.smithi177.stdout: NetworkManager-tui-1:1.47.2-1.el9.x86_64 2024-04-02T14:21:51.163 INFO:teuthology.orchestra.run.smithi177.stdout: acl-2.3.1-4.el9.x86_64 2024-04-02T14:21:51.163 INFO:teuthology.orchestra.run.smithi177.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-04-02T14:21:51.163 INFO:teuthology.orchestra.run.smithi177.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-04-02T14:21:51.163 INFO:teuthology.orchestra.run.smithi177.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-04-02T14:21:51.163 INFO:teuthology.orchestra.run.smithi177.stdout: libacl-2.3.1-4.el9.x86_64 2024-04-02T14:21:51.163 INFO:teuthology.orchestra.run.smithi177.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-04-02T14:21:51.163 INFO:teuthology.orchestra.run.smithi177.stdout: libblkid-2.37.4-18.el9.x86_64 2024-04-02T14:21:51.163 INFO:teuthology.orchestra.run.smithi177.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-04-02T14:21:51.163 INFO:teuthology.orchestra.run.smithi177.stdout: libmount-2.37.4-18.el9.x86_64 2024-04-02T14:21:51.163 INFO:teuthology.orchestra.run.smithi177.stdout: librados2-2:19.0.0-2609.g191d963a.el9.x86_64 2024-04-02T14:21:51.164 INFO:teuthology.orchestra.run.smithi177.stdout: librbd1-2:19.0.0-2609.g191d963a.el9.x86_64 2024-04-02T14:21:51.164 INFO:teuthology.orchestra.run.smithi177.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-04-02T14:21:51.164 INFO:teuthology.orchestra.run.smithi177.stdout: libuuid-2.37.4-18.el9.x86_64 2024-04-02T14:21:51.164 INFO:teuthology.orchestra.run.smithi177.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-04-02T14:21:51.164 INFO:teuthology.orchestra.run.smithi177.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-04-02T14:21:51.164 INFO:teuthology.orchestra.run.smithi177.stdout: util-linux-2.37.4-18.el9.x86_64 2024-04-02T14:21:51.164 INFO:teuthology.orchestra.run.smithi177.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-04-02T14:21:51.164 INFO:teuthology.orchestra.run.smithi177.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-04-02T14:21:51.164 INFO:teuthology.orchestra.run.smithi177.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-04-02T14:21:51.164 INFO:teuthology.orchestra.run.smithi177.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-04-02T14:21:51.164 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2024-04-02T14:21:51.164 INFO:teuthology.orchestra.run.smithi177.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-02T14:21:51.164 INFO:teuthology.orchestra.run.smithi177.stdout: automake-1.16.2-8.el9.noarch 2024-04-02T14:21:51.164 INFO:teuthology.orchestra.run.smithi177.stdout: bison-3.7.4-5.el9.x86_64 2024-04-02T14:21:51.164 INFO:teuthology.orchestra.run.smithi177.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-02T14:21:51.165 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-2:19.0.0-2609.g191d963a.el9.x86_64 2024-04-02T14:21:51.165 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-base-2:19.0.0-2609.g191d963a.el9.x86_64 2024-04-02T14:21:51.165 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-common-2:19.0.0-2609.g191d963a.el9.x86_64 2024-04-02T14:21:51.165 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-fuse-2:19.0.0-2609.g191d963a.el9.x86_64 2024-04-02T14:21:51.165 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-grafana-dashboards-2:19.0.0-2609.g191d963a.el9.noarch 2024-04-02T14:21:51.165 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-immutable-object-cache-2:19.0.0-2609.g191d963a.el9.x86_64 2024-04-02T14:21:51.165 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mds-2:19.0.0-2609.g191d963a.el9.x86_64 2024-04-02T14:21:51.165 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr-2:19.0.0-2609.g191d963a.el9.x86_64 2024-04-02T14:21:51.165 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr-cephadm-2:19.0.0-2609.g191d963a.el9.noarch 2024-04-02T14:21:51.165 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr-dashboard-2:19.0.0-2609.g191d963a.el9.noarch 2024-04-02T14:21:51.165 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2609.g191d963a.el9.noarch 2024-04-02T14:21:51.165 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr-modules-core-2:19.0.0-2609.g191d963a.el9.noarch 2024-04-02T14:21:51.165 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr-rook-2:19.0.0-2609.g191d963a.el9.noarch 2024-04-02T14:21:51.165 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mon-2:19.0.0-2609.g191d963a.el9.x86_64 2024-04-02T14:21:51.166 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-osd-2:19.0.0-2609.g191d963a.el9.x86_64 2024-04-02T14:21:51.166 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-prometheus-alerts-2:19.0.0-2609.g191d963a.el9.noarch 2024-04-02T14:21:51.166 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-radosgw-2:19.0.0-2609.g191d963a.el9.x86_64 2024-04-02T14:21:51.166 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-selinux-2:19.0.0-2609.g191d963a.el9.x86_64 2024-04-02T14:21:51.166 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-test-2:19.0.0-2609.g191d963a.el9.x86_64 2024-04-02T14:21:51.166 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-volume-2:19.0.0-2609.g191d963a.el9.noarch 2024-04-02T14:21:51.166 INFO:teuthology.orchestra.run.smithi177.stdout: cephadm-2:19.0.0-2609.g191d963a.el9.noarch 2024-04-02T14:21:51.166 INFO:teuthology.orchestra.run.smithi177.stdout: cephfs-mirror-2:19.0.0-2609.g191d963a.el9.x86_64 2024-04-02T14:21:51.166 INFO:teuthology.orchestra.run.smithi177.stdout: cephfs-shell-2:19.0.0-2609.g191d963a.el9.x86_64 2024-04-02T14:21:51.166 INFO:teuthology.orchestra.run.smithi177.stdout: cephfs-top-2:19.0.0-2609.g191d963a.el9.noarch 2024-04-02T14:21:51.166 INFO:teuthology.orchestra.run.smithi177.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-02T14:21:51.166 INFO:teuthology.orchestra.run.smithi177.stdout: flex-2.6.4-9.el9.x86_64 2024-04-02T14:21:51.166 INFO:teuthology.orchestra.run.smithi177.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-02T14:21:51.166 INFO:teuthology.orchestra.run.smithi177.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-02T14:21:51.167 INFO:teuthology.orchestra.run.smithi177.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-02T14:21:51.167 INFO:teuthology.orchestra.run.smithi177.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-02T14:21:51.167 INFO:teuthology.orchestra.run.smithi177.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-02T14:21:51.167 INFO:teuthology.orchestra.run.smithi177.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-02T14:21:51.167 INFO:teuthology.orchestra.run.smithi177.stdout: jq-1.6-16.el9.x86_64 2024-04-02T14:21:51.167 INFO:teuthology.orchestra.run.smithi177.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-02T14:21:51.167 INFO:teuthology.orchestra.run.smithi177.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-02T14:21:51.167 INFO:teuthology.orchestra.run.smithi177.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-02T14:21:51.167 INFO:teuthology.orchestra.run.smithi177.stdout: libcephfs-devel-2:19.0.0-2609.g191d963a.el9.x86_64 2024-04-02T14:21:51.167 INFO:teuthology.orchestra.run.smithi177.stdout: libcephfs2-2:19.0.0-2609.g191d963a.el9.x86_64 2024-04-02T14:21:51.167 INFO:teuthology.orchestra.run.smithi177.stdout: libcephsqlite-2:19.0.0-2609.g191d963a.el9.x86_64 2024-04-02T14:21:51.167 INFO:teuthology.orchestra.run.smithi177.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-02T14:21:51.167 INFO:teuthology.orchestra.run.smithi177.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-02T14:21:51.167 INFO:teuthology.orchestra.run.smithi177.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-02T14:21:51.168 INFO:teuthology.orchestra.run.smithi177.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-02T14:21:51.168 INFO:teuthology.orchestra.run.smithi177.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-02T14:21:51.168 INFO:teuthology.orchestra.run.smithi177.stdout: librados-devel-2:19.0.0-2609.g191d963a.el9.x86_64 2024-04-02T14:21:51.168 INFO:teuthology.orchestra.run.smithi177.stdout: libradosstriper1-2:19.0.0-2609.g191d963a.el9.x86_64 2024-04-02T14:21:51.168 INFO:teuthology.orchestra.run.smithi177.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-02T14:21:51.168 INFO:teuthology.orchestra.run.smithi177.stdout: librgw2-2:19.0.0-2609.g191d963a.el9.x86_64 2024-04-02T14:21:51.168 INFO:teuthology.orchestra.run.smithi177.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-02T14:21:51.168 INFO:teuthology.orchestra.run.smithi177.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-02T14:21:51.168 INFO:teuthology.orchestra.run.smithi177.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-02T14:21:51.168 INFO:teuthology.orchestra.run.smithi177.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-02T14:21:51.168 INFO:teuthology.orchestra.run.smithi177.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-02T14:21:51.168 INFO:teuthology.orchestra.run.smithi177.stdout: lua-5.4.4-4.el9.x86_64 2024-04-02T14:21:51.168 INFO:teuthology.orchestra.run.smithi177.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-02T14:21:51.168 INFO:teuthology.orchestra.run.smithi177.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-02T14:21:51.169 INFO:teuthology.orchestra.run.smithi177.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-04-02T14:21:51.169 INFO:teuthology.orchestra.run.smithi177.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-02T14:21:51.169 INFO:teuthology.orchestra.run.smithi177.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-02T14:21:51.169 INFO:teuthology.orchestra.run.smithi177.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-04-02T14:21:51.169 INFO:teuthology.orchestra.run.smithi177.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-02T14:21:51.169 INFO:teuthology.orchestra.run.smithi177.stdout: postgresql-13.14-1.el9.x86_64 2024-04-02T14:21:51.169 INFO:teuthology.orchestra.run.smithi177.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-02T14:21:51.169 INFO:teuthology.orchestra.run.smithi177.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-02T14:21:51.169 INFO:teuthology.orchestra.run.smithi177.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-02T14:21:51.169 INFO:teuthology.orchestra.run.smithi177.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-02T14:21:51.169 INFO:teuthology.orchestra.run.smithi177.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-02T14:21:51.169 INFO:teuthology.orchestra.run.smithi177.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-02T14:21:51.169 INFO:teuthology.orchestra.run.smithi177.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-02T14:21:51.169 INFO:teuthology.orchestra.run.smithi177.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-02T14:21:51.169 INFO:teuthology.orchestra.run.smithi177.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-02T14:21:51.169 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-02T14:21:51.170 INFO:teuthology.orchestra.run.smithi177.stdout: python3-ceph-argparse-2:19.0.0-2609.g191d963a.el9.x86_64 2024-04-02T14:21:51.170 INFO:teuthology.orchestra.run.smithi177.stdout: python3-ceph-common-2:19.0.0-2609.g191d963a.el9.x86_64 2024-04-02T14:21:51.170 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cephfs-2:19.0.0-2609.g191d963a.el9.x86_64 2024-04-02T14:21:51.170 INFO:teuthology.orchestra.run.smithi177.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-02T14:21:51.170 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-02T14:21:51.170 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-02T14:21:51.170 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-02T14:21:51.170 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-02T14:21:51.170 INFO:teuthology.orchestra.run.smithi177.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-02T14:21:51.170 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-02T14:21:51.170 INFO:teuthology.orchestra.run.smithi177.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-04-02T14:21:51.170 INFO:teuthology.orchestra.run.smithi177.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-04-02T14:21:51.170 INFO:teuthology.orchestra.run.smithi177.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-02T14:21:51.170 INFO:teuthology.orchestra.run.smithi177.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-02T14:21:51.171 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-02T14:21:51.171 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-02T14:21:51.171 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-02T14:21:51.171 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-02T14:21:51.171 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-02T14:21:51.171 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-02T14:21:51.171 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-02T14:21:51.171 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-02T14:21:51.171 INFO:teuthology.orchestra.run.smithi177.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-02T14:21:51.171 INFO:teuthology.orchestra.run.smithi177.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-02T14:21:51.171 INFO:teuthology.orchestra.run.smithi177.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-02T14:21:51.171 INFO:teuthology.orchestra.run.smithi177.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-02T14:21:51.171 INFO:teuthology.orchestra.run.smithi177.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-02T14:21:51.171 INFO:teuthology.orchestra.run.smithi177.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-02T14:21:51.171 INFO:teuthology.orchestra.run.smithi177.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-02T14:21:51.172 INFO:teuthology.orchestra.run.smithi177.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-02T14:21:51.172 INFO:teuthology.orchestra.run.smithi177.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-02T14:21:51.172 INFO:teuthology.orchestra.run.smithi177.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-02T14:21:51.172 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-02T14:21:51.172 INFO:teuthology.orchestra.run.smithi177.stdout: python3-ply-3.11-14.el9.noarch 2024-04-02T14:21:51.172 INFO:teuthology.orchestra.run.smithi177.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-02T14:21:51.172 INFO:teuthology.orchestra.run.smithi177.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-02T14:21:51.172 INFO:teuthology.orchestra.run.smithi177.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-02T14:21:51.172 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-02T14:21:51.172 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-02T14:21:51.172 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-02T14:21:51.172 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-02T14:21:51.172 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-02T14:21:51.172 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-02T14:21:51.172 INFO:teuthology.orchestra.run.smithi177.stdout: python3-rados-2:19.0.0-2609.g191d963a.el9.x86_64 2024-04-02T14:21:51.173 INFO:teuthology.orchestra.run.smithi177.stdout: python3-rbd-2:19.0.0-2609.g191d963a.el9.x86_64 2024-04-02T14:21:51.173 INFO:teuthology.orchestra.run.smithi177.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-02T14:21:51.173 INFO:teuthology.orchestra.run.smithi177.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-02T14:21:51.173 INFO:teuthology.orchestra.run.smithi177.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-02T14:21:51.173 INFO:teuthology.orchestra.run.smithi177.stdout: python3-rgw-2:19.0.0-2609.g191d963a.el9.x86_64 2024-04-02T14:21:51.173 INFO:teuthology.orchestra.run.smithi177.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-02T14:21:51.173 INFO:teuthology.orchestra.run.smithi177.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-02T14:21:51.173 INFO:teuthology.orchestra.run.smithi177.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-02T14:21:51.173 INFO:teuthology.orchestra.run.smithi177.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-02T14:21:51.173 INFO:teuthology.orchestra.run.smithi177.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-02T14:21:51.173 INFO:teuthology.orchestra.run.smithi177.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-02T14:21:51.173 INFO:teuthology.orchestra.run.smithi177.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-02T14:21:51.173 INFO:teuthology.orchestra.run.smithi177.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-02T14:21:51.174 INFO:teuthology.orchestra.run.smithi177.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-02T14:21:51.174 INFO:teuthology.orchestra.run.smithi177.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-02T14:21:51.174 INFO:teuthology.orchestra.run.smithi177.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-02T14:21:51.174 INFO:teuthology.orchestra.run.smithi177.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-02T14:21:51.174 INFO:teuthology.orchestra.run.smithi177.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-02T14:21:51.174 INFO:teuthology.orchestra.run.smithi177.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-02T14:21:51.174 INFO:teuthology.orchestra.run.smithi177.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-02T14:21:51.174 INFO:teuthology.orchestra.run.smithi177.stdout: rbd-fuse-2:19.0.0-2609.g191d963a.el9.x86_64 2024-04-02T14:21:51.174 INFO:teuthology.orchestra.run.smithi177.stdout: rbd-mirror-2:19.0.0-2609.g191d963a.el9.x86_64 2024-04-02T14:21:51.174 INFO:teuthology.orchestra.run.smithi177.stdout: rbd-nbd-2:19.0.0-2609.g191d963a.el9.x86_64 2024-04-02T14:21:51.174 INFO:teuthology.orchestra.run.smithi177.stdout: re2-1:20211101-3.el9.x86_64 2024-04-02T14:21:51.174 INFO:teuthology.orchestra.run.smithi177.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-02T14:21:51.174 INFO:teuthology.orchestra.run.smithi177.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-02T14:21:51.174 INFO:teuthology.orchestra.run.smithi177.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-02T14:21:51.174 INFO:teuthology.orchestra.run.smithi177.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-02T14:21:51.175 INFO:teuthology.orchestra.run.smithi177.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-02T14:21:51.175 INFO:teuthology.orchestra.run.smithi177.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-04-02T14:21:51.175 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-02T14:21:51.175 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-04-02T14:21:51.642 DEBUG:teuthology.parallel:result is None 2024-04-02T14:21:51.657 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-test-2:19.0.0-2609.g191d963a.el9.x86_64 153/198 2024-04-02T14:21:51.714 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : rbd-fuse-2:19.0.0-2609.g191d963a.el9.x86_64 154/198 2024-04-02T14:21:51.756 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : rbd-nbd-2:19.0.0-2609.g191d963a.el9.x86_64 155/198 2024-04-02T14:21:51.795 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-04-02T14:21:52.067 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : cephfs-shell-2:19.0.0-2609.g191d963a.el9.x86_64 157/198 2024-04-02T14:21:52.193 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 158/198 2024-04-02T14:21:52.247 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-fuse-2:19.0.0-2609.g191d963a.el9.x86_64 159/198 2024-04-02T14:21:52.330 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-04-02T14:21:52.384 INFO:teuthology.orchestra.run.smithi090.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-04-02T14:21:52.429 INFO:teuthology.orchestra.run.smithi090.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-04-02T14:21:52.471 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libcephfs-devel-2:19.0.0-2609.g191d963a.el9.x86_ 163/198 2024-04-02T14:21:52.508 INFO:teuthology.orchestra.run.smithi090.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-04-02T14:21:52.560 INFO:teuthology.orchestra.run.smithi090.stdout: Upgrading : NetworkManager-team-1:1.47.2-1.el9.x86_64 165/198 2024-04-02T14:21:52.652 INFO:teuthology.orchestra.run.smithi090.stdout: Upgrading : NetworkManager-tui-1:1.47.2-1.el9.x86_64 166/198 2024-04-02T14:21:52.766 INFO:teuthology.orchestra.run.smithi090.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-04-02T14:21:52.802 INFO:teuthology.orchestra.run.smithi090.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-04-02T14:21:53.772 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 169/198 2024-04-02T14:21:53.827 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 169/198 2024-04-02T14:21:53.883 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 169/198 2024-04-02T14:21:55.219 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : cephfs-top-2:19.0.0-2609.g191d963a.el9.noarch 170/198 2024-04-02T14:21:55.319 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-04-02T14:21:55.377 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-04-02T14:21:55.528 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-04-02T14:21:55.574 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-04-02T14:21:55.599 INFO:teuthology.orchestra.run.smithi090.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-04-02T14:21:55.658 INFO:teuthology.orchestra.run.smithi090.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-04-02T14:21:55.702 INFO:teuthology.orchestra.run.smithi090.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-04-02T14:21:55.702 INFO:teuthology.orchestra.run.smithi090.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-04-02T14:21:55.854 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-04-02T14:21:55.896 INFO:teuthology.orchestra.run.smithi090.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-04-02T14:21:55.896 INFO:teuthology.orchestra.run.smithi090.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-04-02T14:21:56.059 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-04-02T14:21:56.113 INFO:teuthology.orchestra.run.smithi090.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-04-02T14:21:56.159 INFO:teuthology.orchestra.run.smithi090.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-04-02T14:21:56.234 INFO:teuthology.orchestra.run.smithi090.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-04-02T14:21:56.267 INFO:teuthology.orchestra.run.smithi090.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-04-02T14:21:56.340 INFO:teuthology.orchestra.run.smithi090.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-04-02T14:21:56.424 INFO:teuthology.orchestra.run.smithi090.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-04-02T14:21:56.473 INFO:teuthology.orchestra.run.smithi090.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-04-02T14:21:56.506 INFO:teuthology.orchestra.run.smithi090.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-04-02T14:21:56.540 INFO:teuthology.orchestra.run.smithi090.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-04-02T14:21:56.583 INFO:teuthology.orchestra.run.smithi090.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-04-02T14:21:56.617 INFO:teuthology.orchestra.run.smithi090.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-04-02T14:21:56.653 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-04-02T14:21:56.654 INFO:teuthology.orchestra.run.smithi090.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-04-02T14:21:56.992 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-04-02T14:21:56.992 INFO:teuthology.orchestra.run.smithi090.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-04-02T14:21:57.132 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-04-02T14:21:57.191 INFO:teuthology.orchestra.run.smithi090.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-04-02T14:21:57.220 INFO:teuthology.orchestra.run.smithi090.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-04-02T14:21:57.256 INFO:teuthology.orchestra.run.smithi090.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-04-02T14:21:57.305 INFO:teuthology.orchestra.run.smithi090.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-04-02T14:21:57.470 INFO:teuthology.orchestra.run.smithi090.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-04-02T14:21:57.728 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-04-02T14:21:57.729 INFO:teuthology.orchestra.run.smithi090.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2024-04-02T14:21:57.729 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-02T14:22:04.698 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-2:19.0.0-2609.g191d963a.el9.x86_64 1/198 2024-04-02T14:22:04.698 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-base-2:19.0.0-2609.g191d963a.el9.x86_64 2/198 2024-04-02T14:22:04.698 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-common-2:19.0.0-2609.g191d963a.el9.x86_64 3/198 2024-04-02T14:22:04.698 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-fuse-2:19.0.0-2609.g191d963a.el9.x86_64 4/198 2024-04-02T14:22:04.698 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2609.g191d9 5/198 2024-04-02T14:22:04.698 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-mds-2:19.0.0-2609.g191d963a.el9.x86_64 6/198 2024-04-02T14:22:04.698 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-mgr-2:19.0.0-2609.g191d963a.el9.x86_64 7/198 2024-04-02T14:22:04.698 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-mon-2:19.0.0-2609.g191d963a.el9.x86_64 8/198 2024-04-02T14:22:04.698 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-osd-2:19.0.0-2609.g191d963a.el9.x86_64 9/198 2024-04-02T14:22:04.699 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-radosgw-2:19.0.0-2609.g191d963a.el9.x86_64 10/198 2024-04-02T14:22:04.699 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-selinux-2:19.0.0-2609.g191d963a.el9.x86_64 11/198 2024-04-02T14:22:04.699 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-test-2:19.0.0-2609.g191d963a.el9.x86_64 12/198 2024-04-02T14:22:04.699 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : cephfs-mirror-2:19.0.0-2609.g191d963a.el9.x86_64 13/198 2024-04-02T14:22:04.699 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : cephfs-shell-2:19.0.0-2609.g191d963a.el9.x86_64 14/198 2024-04-02T14:22:04.699 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libcephfs-devel-2:19.0.0-2609.g191d963a.el9.x86_ 15/198 2024-04-02T14:22:04.699 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libcephfs2-2:19.0.0-2609.g191d963a.el9.x86_64 16/198 2024-04-02T14:22:04.699 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libcephsqlite-2:19.0.0-2609.g191d963a.el9.x86_64 17/198 2024-04-02T14:22:04.699 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : librados-devel-2:19.0.0-2609.g191d963a.el9.x86_6 18/198 2024-04-02T14:22:04.699 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libradosstriper1-2:19.0.0-2609.g191d963a.el9.x86 19/198 2024-04-02T14:22:04.699 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : librgw2-2:19.0.0-2609.g191d963a.el9.x86_64 20/198 2024-04-02T14:22:04.699 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2609.g191d963a.el 21/198 2024-04-02T14:22:04.699 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-ceph-common-2:19.0.0-2609.g191d963a.el9. 22/198 2024-04-02T14:22:04.699 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-cephfs-2:19.0.0-2609.g191d963a.el9.x86_6 23/198 2024-04-02T14:22:04.699 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-rados-2:19.0.0-2609.g191d963a.el9.x86_64 24/198 2024-04-02T14:22:04.700 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-rbd-2:19.0.0-2609.g191d963a.el9.x86_64 25/198 2024-04-02T14:22:04.700 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-rgw-2:19.0.0-2609.g191d963a.el9.x86_64 26/198 2024-04-02T14:22:04.700 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : rbd-fuse-2:19.0.0-2609.g191d963a.el9.x86_64 27/198 2024-04-02T14:22:04.700 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : rbd-mirror-2:19.0.0-2609.g191d963a.el9.x86_64 28/198 2024-04-02T14:22:04.700 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : rbd-nbd-2:19.0.0-2609.g191d963a.el9.x86_64 29/198 2024-04-02T14:22:04.715 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2609.g191d963a. 30/198 2024-04-02T14:22:04.715 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2609.g191d963a.el9.noa 31/198 2024-04-02T14:22:04.715 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2609.g191d963a.el9.n 32/198 2024-04-02T14:22:04.715 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2609.g191 33/198 2024-04-02T14:22:04.716 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2609.g191d963a.el 34/198 2024-04-02T14:22:04.716 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2609.g191d963a.el9.noarch 35/198 2024-04-02T14:22:04.716 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2609.g191d963a.e 36/198 2024-04-02T14:22:04.716 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-volume-2:19.0.0-2609.g191d963a.el9.noarch 37/198 2024-04-02T14:22:04.716 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : cephadm-2:19.0.0-2609.g191d963a.el9.noarch 38/198 2024-04-02T14:22:04.716 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : cephfs-top-2:19.0.0-2609.g191d963a.el9.noarch 39/198 2024-04-02T14:22:04.716 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-04-02T14:22:04.716 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-04-02T14:22:04.716 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-04-02T14:22:04.716 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-04-02T14:22:04.716 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-04-02T14:22:04.716 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-04-02T14:22:04.716 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-04-02T14:22:04.716 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-04-02T14:22:04.716 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-04-02T14:22:04.717 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-04-02T14:22:04.717 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-04-02T14:22:04.717 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-04-02T14:22:04.717 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-04-02T14:22:04.717 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-04-02T14:22:04.717 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-04-02T14:22:04.717 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-04-02T14:22:04.717 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-04-02T14:22:04.717 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-04-02T14:22:04.717 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : postgresql-13.14-1.el9.x86_64 58/198 2024-04-02T14:22:04.717 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 59/198 2024-04-02T14:22:04.717 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 60/198 2024-04-02T14:22:04.717 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 61/198 2024-04-02T14:22:04.717 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 62/198 2024-04-02T14:22:04.718 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 63/198 2024-04-02T14:22:04.718 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 64/198 2024-04-02T14:22:04.718 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 65/198 2024-04-02T14:22:04.718 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 66/198 2024-04-02T14:22:04.718 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 67/198 2024-04-02T14:22:04.718 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 68/198 2024-04-02T14:22:04.718 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 69/198 2024-04-02T14:22:04.718 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 70/198 2024-04-02T14:22:04.718 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 71/198 2024-04-02T14:22:04.718 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 72/198 2024-04-02T14:22:04.718 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 73/198 2024-04-02T14:22:04.718 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 74/198 2024-04-02T14:22:04.718 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 75/198 2024-04-02T14:22:04.718 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 76/198 2024-04-02T14:22:04.719 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 77/198 2024-04-02T14:22:04.719 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 78/198 2024-04-02T14:22:04.719 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 79/198 2024-04-02T14:22:04.719 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 80/198 2024-04-02T14:22:04.719 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 81/198 2024-04-02T14:22:04.719 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 82/198 2024-04-02T14:22:04.719 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 83/198 2024-04-02T14:22:04.719 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 84/198 2024-04-02T14:22:04.719 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 85/198 2024-04-02T14:22:04.719 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 86/198 2024-04-02T14:22:04.719 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 87/198 2024-04-02T14:22:04.719 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 88/198 2024-04-02T14:22:04.719 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 89/198 2024-04-02T14:22:04.719 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 90/198 2024-04-02T14:22:04.720 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 91/198 2024-04-02T14:22:04.720 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-ply-3.11-14.el9.noarch 92/198 2024-04-02T14:22:04.720 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 93/198 2024-04-02T14:22:04.720 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 94/198 2024-04-02T14:22:04.720 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : jq-1.6-16.el9.x86_64 95/198 2024-04-02T14:22:04.720 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 96/198 2024-04-02T14:22:04.720 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 97/198 2024-04-02T14:22:04.720 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 98/198 2024-04-02T14:22:04.720 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 99/198 2024-04-02T14:22:04.720 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 100/198 2024-04-02T14:22:04.720 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 101/198 2024-04-02T14:22:04.720 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 102/198 2024-04-02T14:22:04.720 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 103/198 2024-04-02T14:22:04.720 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-04-02T14:22:04.720 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-04-02T14:22:04.720 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-04-02T14:22:04.721 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-04-02T14:22:04.721 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-04-02T14:22:04.721 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-04-02T14:22:04.721 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-04-02T14:22:04.721 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-04-02T14:22:04.721 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-04-02T14:22:04.721 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-04-02T14:22:04.721 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-04-02T14:22:04.721 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-04-02T14:22:04.721 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 116/198 2024-04-02T14:22:04.721 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-04-02T14:22:04.721 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-04-02T14:22:04.721 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-04-02T14:22:04.721 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-04-02T14:22:04.721 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-04-02T14:22:04.722 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-04-02T14:22:04.722 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-04-02T14:22:04.722 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 124/198 2024-04-02T14:22:04.722 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-04-02T14:22:04.722 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-04-02T14:22:04.722 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-04-02T14:22:04.722 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-04-02T14:22:04.722 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-04-02T14:22:04.722 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-04-02T14:22:04.722 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-04-02T14:22:04.722 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-04-02T14:22:04.722 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-04-02T14:22:04.722 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-04-02T14:22:04.722 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-04-02T14:22:04.723 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-04-02T14:22:04.723 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-04-02T14:22:04.723 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-04-02T14:22:04.723 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-04-02T14:22:04.723 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-04-02T14:22:04.723 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-04-02T14:22:04.723 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-04-02T14:22:04.723 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-04-02T14:22:04.723 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-04-02T14:22:04.723 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-04-02T14:22:04.723 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-04-02T14:22:04.723 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-04-02T14:22:04.723 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-04-02T14:22:04.724 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-04-02T14:22:04.724 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-04-02T14:22:04.724 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : librados2-2:19.0.0-2609.g191d963a.el9.x86_64 151/198 2024-04-02T14:22:04.724 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-04-02T14:22:04.724 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : librbd1-2:19.0.0-2609.g191d963a.el9.x86_64 153/198 2024-04-02T14:22:04.724 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-04-02T14:22:04.724 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : NetworkManager-1:1.47.2-1.el9.x86_64 155/198 2024-04-02T14:22:04.724 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 156/198 2024-04-02T14:22:04.724 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 157/198 2024-04-02T14:22:04.724 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 158/198 2024-04-02T14:22:04.724 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : NetworkManager-team-1:1.47.2-1.el9.x86_64 159/198 2024-04-02T14:22:04.724 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 160/198 2024-04-02T14:22:04.724 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : NetworkManager-tui-1:1.47.2-1.el9.x86_64 161/198 2024-04-02T14:22:04.724 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 162/198 2024-04-02T14:22:04.724 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 163/198 2024-04-02T14:22:04.725 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 164/198 2024-04-02T14:22:04.725 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-04-02T14:22:04.725 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-04-02T14:22:04.725 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-04-02T14:22:04.725 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-04-02T14:22:04.725 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-04-02T14:22:04.725 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-04-02T14:22:04.725 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-04-02T14:22:04.725 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-04-02T14:22:04.725 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-04-02T14:22:04.725 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-04-02T14:22:04.725 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-04-02T14:22:04.725 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-04-02T14:22:04.725 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-04-02T14:22:04.725 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-04-02T14:22:04.725 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-04-02T14:22:04.726 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-04-02T14:22:04.726 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-04-02T14:22:04.726 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-04-02T14:22:04.726 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-04-02T14:22:04.726 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-04-02T14:22:04.726 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-04-02T14:22:04.726 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-04-02T14:22:04.726 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-04-02T14:22:04.726 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-04-02T14:22:04.726 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-04-02T14:22:04.726 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-04-02T14:22:04.726 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-04-02T14:22:04.726 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-04-02T14:22:04.726 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-04-02T14:22:04.727 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-04-02T14:22:04.727 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-04-02T14:22:04.727 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-04-02T14:22:04.727 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-04-02T14:22:07.317 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-04-02T14:22:07.317 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-02T14:22:07.317 INFO:teuthology.orchestra.run.smithi090.stdout:Upgraded: 2024-04-02T14:22:07.317 INFO:teuthology.orchestra.run.smithi090.stdout: NetworkManager-1:1.47.2-1.el9.x86_64 2024-04-02T14:22:07.317 INFO:teuthology.orchestra.run.smithi090.stdout: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 2024-04-02T14:22:07.317 INFO:teuthology.orchestra.run.smithi090.stdout: NetworkManager-team-1:1.47.2-1.el9.x86_64 2024-04-02T14:22:07.318 INFO:teuthology.orchestra.run.smithi090.stdout: NetworkManager-tui-1:1.47.2-1.el9.x86_64 2024-04-02T14:22:07.318 INFO:teuthology.orchestra.run.smithi090.stdout: acl-2.3.1-4.el9.x86_64 2024-04-02T14:22:07.318 INFO:teuthology.orchestra.run.smithi090.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-04-02T14:22:07.318 INFO:teuthology.orchestra.run.smithi090.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-04-02T14:22:07.318 INFO:teuthology.orchestra.run.smithi090.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-04-02T14:22:07.318 INFO:teuthology.orchestra.run.smithi090.stdout: libacl-2.3.1-4.el9.x86_64 2024-04-02T14:22:07.318 INFO:teuthology.orchestra.run.smithi090.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-04-02T14:22:07.318 INFO:teuthology.orchestra.run.smithi090.stdout: libblkid-2.37.4-18.el9.x86_64 2024-04-02T14:22:07.318 INFO:teuthology.orchestra.run.smithi090.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-04-02T14:22:07.318 INFO:teuthology.orchestra.run.smithi090.stdout: libmount-2.37.4-18.el9.x86_64 2024-04-02T14:22:07.318 INFO:teuthology.orchestra.run.smithi090.stdout: librados2-2:19.0.0-2609.g191d963a.el9.x86_64 2024-04-02T14:22:07.318 INFO:teuthology.orchestra.run.smithi090.stdout: librbd1-2:19.0.0-2609.g191d963a.el9.x86_64 2024-04-02T14:22:07.318 INFO:teuthology.orchestra.run.smithi090.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-04-02T14:22:07.318 INFO:teuthology.orchestra.run.smithi090.stdout: libuuid-2.37.4-18.el9.x86_64 2024-04-02T14:22:07.319 INFO:teuthology.orchestra.run.smithi090.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-04-02T14:22:07.319 INFO:teuthology.orchestra.run.smithi090.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-04-02T14:22:07.319 INFO:teuthology.orchestra.run.smithi090.stdout: util-linux-2.37.4-18.el9.x86_64 2024-04-02T14:22:07.319 INFO:teuthology.orchestra.run.smithi090.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-04-02T14:22:07.319 INFO:teuthology.orchestra.run.smithi090.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-04-02T14:22:07.319 INFO:teuthology.orchestra.run.smithi090.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-04-02T14:22:07.319 INFO:teuthology.orchestra.run.smithi090.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-04-02T14:22:07.319 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2024-04-02T14:22:07.319 INFO:teuthology.orchestra.run.smithi090.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-02T14:22:07.319 INFO:teuthology.orchestra.run.smithi090.stdout: automake-1.16.2-8.el9.noarch 2024-04-02T14:22:07.319 INFO:teuthology.orchestra.run.smithi090.stdout: bison-3.7.4-5.el9.x86_64 2024-04-02T14:22:07.319 INFO:teuthology.orchestra.run.smithi090.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-02T14:22:07.319 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-2:19.0.0-2609.g191d963a.el9.x86_64 2024-04-02T14:22:07.319 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-base-2:19.0.0-2609.g191d963a.el9.x86_64 2024-04-02T14:22:07.319 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-common-2:19.0.0-2609.g191d963a.el9.x86_64 2024-04-02T14:22:07.320 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-fuse-2:19.0.0-2609.g191d963a.el9.x86_64 2024-04-02T14:22:07.320 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-grafana-dashboards-2:19.0.0-2609.g191d963a.el9.noarch 2024-04-02T14:22:07.320 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-immutable-object-cache-2:19.0.0-2609.g191d963a.el9.x86_64 2024-04-02T14:22:07.320 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mds-2:19.0.0-2609.g191d963a.el9.x86_64 2024-04-02T14:22:07.320 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-2:19.0.0-2609.g191d963a.el9.x86_64 2024-04-02T14:22:07.320 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-cephadm-2:19.0.0-2609.g191d963a.el9.noarch 2024-04-02T14:22:07.320 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-dashboard-2:19.0.0-2609.g191d963a.el9.noarch 2024-04-02T14:22:07.320 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2609.g191d963a.el9.noarch 2024-04-02T14:22:07.320 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-modules-core-2:19.0.0-2609.g191d963a.el9.noarch 2024-04-02T14:22:07.320 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-rook-2:19.0.0-2609.g191d963a.el9.noarch 2024-04-02T14:22:07.320 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mon-2:19.0.0-2609.g191d963a.el9.x86_64 2024-04-02T14:22:07.320 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-osd-2:19.0.0-2609.g191d963a.el9.x86_64 2024-04-02T14:22:07.320 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-prometheus-alerts-2:19.0.0-2609.g191d963a.el9.noarch 2024-04-02T14:22:07.320 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-radosgw-2:19.0.0-2609.g191d963a.el9.x86_64 2024-04-02T14:22:07.321 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-selinux-2:19.0.0-2609.g191d963a.el9.x86_64 2024-04-02T14:22:07.321 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-test-2:19.0.0-2609.g191d963a.el9.x86_64 2024-04-02T14:22:07.321 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-volume-2:19.0.0-2609.g191d963a.el9.noarch 2024-04-02T14:22:07.321 INFO:teuthology.orchestra.run.smithi090.stdout: cephadm-2:19.0.0-2609.g191d963a.el9.noarch 2024-04-02T14:22:07.321 INFO:teuthology.orchestra.run.smithi090.stdout: cephfs-mirror-2:19.0.0-2609.g191d963a.el9.x86_64 2024-04-02T14:22:07.321 INFO:teuthology.orchestra.run.smithi090.stdout: cephfs-shell-2:19.0.0-2609.g191d963a.el9.x86_64 2024-04-02T14:22:07.321 INFO:teuthology.orchestra.run.smithi090.stdout: cephfs-top-2:19.0.0-2609.g191d963a.el9.noarch 2024-04-02T14:22:07.321 INFO:teuthology.orchestra.run.smithi090.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-02T14:22:07.321 INFO:teuthology.orchestra.run.smithi090.stdout: flex-2.6.4-9.el9.x86_64 2024-04-02T14:22:07.321 INFO:teuthology.orchestra.run.smithi090.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-02T14:22:07.321 INFO:teuthology.orchestra.run.smithi090.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-02T14:22:07.321 INFO:teuthology.orchestra.run.smithi090.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-02T14:22:07.321 INFO:teuthology.orchestra.run.smithi090.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-02T14:22:07.321 INFO:teuthology.orchestra.run.smithi090.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-02T14:22:07.322 INFO:teuthology.orchestra.run.smithi090.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-02T14:22:07.322 INFO:teuthology.orchestra.run.smithi090.stdout: jq-1.6-16.el9.x86_64 2024-04-02T14:22:07.322 INFO:teuthology.orchestra.run.smithi090.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-02T14:22:07.322 INFO:teuthology.orchestra.run.smithi090.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-02T14:22:07.322 INFO:teuthology.orchestra.run.smithi090.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-02T14:22:07.322 INFO:teuthology.orchestra.run.smithi090.stdout: libcephfs-devel-2:19.0.0-2609.g191d963a.el9.x86_64 2024-04-02T14:22:07.322 INFO:teuthology.orchestra.run.smithi090.stdout: libcephfs2-2:19.0.0-2609.g191d963a.el9.x86_64 2024-04-02T14:22:07.322 INFO:teuthology.orchestra.run.smithi090.stdout: libcephsqlite-2:19.0.0-2609.g191d963a.el9.x86_64 2024-04-02T14:22:07.322 INFO:teuthology.orchestra.run.smithi090.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-02T14:22:07.322 INFO:teuthology.orchestra.run.smithi090.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-02T14:22:07.322 INFO:teuthology.orchestra.run.smithi090.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-02T14:22:07.322 INFO:teuthology.orchestra.run.smithi090.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-02T14:22:07.322 INFO:teuthology.orchestra.run.smithi090.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-02T14:22:07.322 INFO:teuthology.orchestra.run.smithi090.stdout: librados-devel-2:19.0.0-2609.g191d963a.el9.x86_64 2024-04-02T14:22:07.323 INFO:teuthology.orchestra.run.smithi090.stdout: libradosstriper1-2:19.0.0-2609.g191d963a.el9.x86_64 2024-04-02T14:22:07.323 INFO:teuthology.orchestra.run.smithi090.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-02T14:22:07.323 INFO:teuthology.orchestra.run.smithi090.stdout: librgw2-2:19.0.0-2609.g191d963a.el9.x86_64 2024-04-02T14:22:07.323 INFO:teuthology.orchestra.run.smithi090.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-02T14:22:07.323 INFO:teuthology.orchestra.run.smithi090.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-02T14:22:07.323 INFO:teuthology.orchestra.run.smithi090.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-02T14:22:07.323 INFO:teuthology.orchestra.run.smithi090.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-02T14:22:07.323 INFO:teuthology.orchestra.run.smithi090.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-02T14:22:07.323 INFO:teuthology.orchestra.run.smithi090.stdout: lua-5.4.4-4.el9.x86_64 2024-04-02T14:22:07.323 INFO:teuthology.orchestra.run.smithi090.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-02T14:22:07.323 INFO:teuthology.orchestra.run.smithi090.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-02T14:22:07.323 INFO:teuthology.orchestra.run.smithi090.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-04-02T14:22:07.323 INFO:teuthology.orchestra.run.smithi090.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-02T14:22:07.323 INFO:teuthology.orchestra.run.smithi090.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-02T14:22:07.323 INFO:teuthology.orchestra.run.smithi090.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-04-02T14:22:07.323 INFO:teuthology.orchestra.run.smithi090.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-02T14:22:07.324 INFO:teuthology.orchestra.run.smithi090.stdout: postgresql-13.14-1.el9.x86_64 2024-04-02T14:22:07.324 INFO:teuthology.orchestra.run.smithi090.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-02T14:22:07.324 INFO:teuthology.orchestra.run.smithi090.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-02T14:22:07.324 INFO:teuthology.orchestra.run.smithi090.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-02T14:22:07.324 INFO:teuthology.orchestra.run.smithi090.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-02T14:22:07.324 INFO:teuthology.orchestra.run.smithi090.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-02T14:22:07.324 INFO:teuthology.orchestra.run.smithi090.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-02T14:22:07.324 INFO:teuthology.orchestra.run.smithi090.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-02T14:22:07.324 INFO:teuthology.orchestra.run.smithi090.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-02T14:22:07.324 INFO:teuthology.orchestra.run.smithi090.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-02T14:22:07.324 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-02T14:22:07.324 INFO:teuthology.orchestra.run.smithi090.stdout: python3-ceph-argparse-2:19.0.0-2609.g191d963a.el9.x86_64 2024-04-02T14:22:07.324 INFO:teuthology.orchestra.run.smithi090.stdout: python3-ceph-common-2:19.0.0-2609.g191d963a.el9.x86_64 2024-04-02T14:22:07.324 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cephfs-2:19.0.0-2609.g191d963a.el9.x86_64 2024-04-02T14:22:07.325 INFO:teuthology.orchestra.run.smithi090.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-02T14:22:07.325 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-02T14:22:07.325 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-02T14:22:07.325 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-02T14:22:07.325 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-02T14:22:07.325 INFO:teuthology.orchestra.run.smithi090.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-02T14:22:07.325 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-02T14:22:07.325 INFO:teuthology.orchestra.run.smithi090.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-04-02T14:22:07.325 INFO:teuthology.orchestra.run.smithi090.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-04-02T14:22:07.325 INFO:teuthology.orchestra.run.smithi090.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-02T14:22:07.325 INFO:teuthology.orchestra.run.smithi090.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-02T14:22:07.325 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-02T14:22:07.325 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-02T14:22:07.325 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-02T14:22:07.325 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-02T14:22:07.326 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-02T14:22:07.326 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-02T14:22:07.326 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-02T14:22:07.326 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-02T14:22:07.326 INFO:teuthology.orchestra.run.smithi090.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-02T14:22:07.326 INFO:teuthology.orchestra.run.smithi090.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-02T14:22:07.326 INFO:teuthology.orchestra.run.smithi090.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-02T14:22:07.326 INFO:teuthology.orchestra.run.smithi090.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-02T14:22:07.326 INFO:teuthology.orchestra.run.smithi090.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-02T14:22:07.326 INFO:teuthology.orchestra.run.smithi090.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-02T14:22:07.326 INFO:teuthology.orchestra.run.smithi090.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-02T14:22:07.326 INFO:teuthology.orchestra.run.smithi090.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-02T14:22:07.326 INFO:teuthology.orchestra.run.smithi090.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-02T14:22:07.326 INFO:teuthology.orchestra.run.smithi090.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-02T14:22:07.327 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-02T14:22:07.327 INFO:teuthology.orchestra.run.smithi090.stdout: python3-ply-3.11-14.el9.noarch 2024-04-02T14:22:07.327 INFO:teuthology.orchestra.run.smithi090.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-02T14:22:07.327 INFO:teuthology.orchestra.run.smithi090.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-02T14:22:07.327 INFO:teuthology.orchestra.run.smithi090.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-02T14:22:07.327 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-02T14:22:07.327 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-02T14:22:07.327 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-02T14:22:07.327 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-02T14:22:07.327 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-02T14:22:07.327 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-02T14:22:07.327 INFO:teuthology.orchestra.run.smithi090.stdout: python3-rados-2:19.0.0-2609.g191d963a.el9.x86_64 2024-04-02T14:22:07.327 INFO:teuthology.orchestra.run.smithi090.stdout: python3-rbd-2:19.0.0-2609.g191d963a.el9.x86_64 2024-04-02T14:22:07.327 INFO:teuthology.orchestra.run.smithi090.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-02T14:22:07.327 INFO:teuthology.orchestra.run.smithi090.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-02T14:22:07.327 INFO:teuthology.orchestra.run.smithi090.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-02T14:22:07.328 INFO:teuthology.orchestra.run.smithi090.stdout: python3-rgw-2:19.0.0-2609.g191d963a.el9.x86_64 2024-04-02T14:22:07.328 INFO:teuthology.orchestra.run.smithi090.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-02T14:22:07.328 INFO:teuthology.orchestra.run.smithi090.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-02T14:22:07.328 INFO:teuthology.orchestra.run.smithi090.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-02T14:22:07.328 INFO:teuthology.orchestra.run.smithi090.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-02T14:22:07.328 INFO:teuthology.orchestra.run.smithi090.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-02T14:22:07.328 INFO:teuthology.orchestra.run.smithi090.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-02T14:22:07.328 INFO:teuthology.orchestra.run.smithi090.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-02T14:22:07.328 INFO:teuthology.orchestra.run.smithi090.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-02T14:22:07.328 INFO:teuthology.orchestra.run.smithi090.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-02T14:22:07.328 INFO:teuthology.orchestra.run.smithi090.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-02T14:22:07.328 INFO:teuthology.orchestra.run.smithi090.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-02T14:22:07.328 INFO:teuthology.orchestra.run.smithi090.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-02T14:22:07.328 INFO:teuthology.orchestra.run.smithi090.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-02T14:22:07.328 INFO:teuthology.orchestra.run.smithi090.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-02T14:22:07.328 INFO:teuthology.orchestra.run.smithi090.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-02T14:22:07.329 INFO:teuthology.orchestra.run.smithi090.stdout: rbd-fuse-2:19.0.0-2609.g191d963a.el9.x86_64 2024-04-02T14:22:07.329 INFO:teuthology.orchestra.run.smithi090.stdout: rbd-mirror-2:19.0.0-2609.g191d963a.el9.x86_64 2024-04-02T14:22:07.329 INFO:teuthology.orchestra.run.smithi090.stdout: rbd-nbd-2:19.0.0-2609.g191d963a.el9.x86_64 2024-04-02T14:22:07.329 INFO:teuthology.orchestra.run.smithi090.stdout: re2-1:20211101-3.el9.x86_64 2024-04-02T14:22:07.329 INFO:teuthology.orchestra.run.smithi090.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-02T14:22:07.329 INFO:teuthology.orchestra.run.smithi090.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-02T14:22:07.329 INFO:teuthology.orchestra.run.smithi090.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-02T14:22:07.329 INFO:teuthology.orchestra.run.smithi090.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-02T14:22:07.329 INFO:teuthology.orchestra.run.smithi090.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-02T14:22:07.329 INFO:teuthology.orchestra.run.smithi090.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-04-02T14:22:07.329 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-02T14:22:07.330 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-04-02T14:22:07.986 DEBUG:teuthology.parallel:result is None 2024-04-02T14:22:07.987 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=191d963ac9de7322da8aff20b567b3fa458e2ef6 2024-04-02T14:22:08.216 DEBUG:teuthology.orchestra.run.smithi090:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-02T14:22:08.268 INFO:teuthology.orchestra.run.smithi090.stdout:19.0.0-2609.g191d963a.el9 2024-04-02T14:22:08.269 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2609.g191d963a.el9 2024-04-02T14:22:08.269 INFO:teuthology.task.install:The correct ceph version 19.0.0-2609.g191d963a is installed. 2024-04-02T14:22:08.271 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=191d963ac9de7322da8aff20b567b3fa458e2ef6 2024-04-02T14:22:08.421 DEBUG:teuthology.orchestra.run.smithi092:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-02T14:22:08.446 INFO:teuthology.orchestra.run.smithi092.stdout:19.0.0-2609.g191d963a.el9 2024-04-02T14:22:08.447 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2609.g191d963a.el9 2024-04-02T14:22:08.447 INFO:teuthology.task.install:The correct ceph version 19.0.0-2609.g191d963a is installed. 2024-04-02T14:22:08.449 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=191d963ac9de7322da8aff20b567b3fa458e2ef6 2024-04-02T14:22:08.592 DEBUG:teuthology.orchestra.run.smithi177:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-02T14:22:08.618 INFO:teuthology.orchestra.run.smithi177.stdout:19.0.0-2609.g191d963a.el9 2024-04-02T14:22:08.618 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2609.g191d963a.el9 2024-04-02T14:22:08.618 INFO:teuthology.task.install:The correct ceph version 19.0.0-2609.g191d963a is installed. 2024-04-02T14:22:08.620 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-04-02T14:22:08.620 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-04-02T14:22:08.621 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-02T14:22:08.696 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2024-04-02T14:22:08.696 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-02T14:22:08.730 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-04-02T14:22:08.730 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-02T14:22:08.765 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-04-02T14:22:08.765 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-04-02T14:22:08.765 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd of=/usr/bin/daemon-helper 2024-04-02T14:22:08.834 DEBUG:teuthology.orchestra.run.smithi090:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-02T14:22:08.954 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2024-04-02T14:22:08.954 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd of=/usr/bin/daemon-helper 2024-04-02T14:22:08.985 DEBUG:teuthology.orchestra.run.smithi092:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-02T14:22:09.053 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-04-02T14:22:09.053 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd of=/usr/bin/daemon-helper 2024-04-02T14:22:09.085 DEBUG:teuthology.orchestra.run.smithi177:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-02T14:22:09.153 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-04-02T14:22:09.153 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-04-02T14:22:09.153 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-02T14:22:09.239 DEBUG:teuthology.orchestra.run.smithi090:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-02T14:22:09.348 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2024-04-02T14:22:09.348 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-02T14:22:09.374 DEBUG:teuthology.orchestra.run.smithi092:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-02T14:22:09.441 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-04-02T14:22:09.441 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-02T14:22:09.468 DEBUG:teuthology.orchestra.run.smithi177:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-02T14:22:09.535 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-04-02T14:22:09.535 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-04-02T14:22:09.535 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd of=/usr/bin/stdin-killer 2024-04-02T14:22:09.606 DEBUG:teuthology.orchestra.run.smithi090:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-02T14:22:09.712 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2024-04-02T14:22:09.712 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd of=/usr/bin/stdin-killer 2024-04-02T14:22:09.739 DEBUG:teuthology.orchestra.run.smithi092:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-02T14:22:09.805 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-04-02T14:22:09.805 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd of=/usr/bin/stdin-killer 2024-04-02T14:22:09.832 DEBUG:teuthology.orchestra.run.smithi177:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-02T14:22:09.899 INFO:teuthology.run_tasks:Running task cephadm... 2024-04-02T14:22:10.000 INFO:tasks.cephadm:Config: {'roleless': False, 'cephfs': {'max_mds': 5, 'session_timeout': 300, 'standby_replay': True}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_bal_replicate_threshold': 1, 'mds_export_ephemeral_random_max': 0.1, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'cephsqlite lock renewal timeout': 900000, 'debug cephsqlite': 20, 'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180, 'osd shutdown pgref assert': True, 'osd_max_omap_entries_per_request': 10}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'slow metadata IO', 'SLOW_OPS', 'slow request'], 'sha1': '191d963ac9de7322da8aff20b567b3fa458e2ef6', 'subvols': {'create': 2, 'subvol_options': '--namespace-isolated'}} 2024-04-02T14:22:10.000 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:191d963ac9de7322da8aff20b567b3fa458e2ef6 2024-04-02T14:22:10.001 INFO:tasks.cephadm:Cluster fsid is 6449448a-f0fc-11ee-b647-cb9ed24678a4 2024-04-02T14:22:10.001 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-04-02T14:22:10.001 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.90', 'mon.b': '172.21.15.92', 'mon.c': '172.21.15.177'} 2024-04-02T14:22:10.001 INFO:tasks.cephadm:First mon is mon.a on smithi090 2024-04-02T14:22:10.001 INFO:tasks.cephadm:First mgr is x 2024-04-02T14:22:10.001 INFO:tasks.cephadm:Normalizing hostnames... 2024-04-02T14:22:10.002 DEBUG:teuthology.orchestra.run.smithi090:> sudo hostname $(hostname -s) 2024-04-02T14:22:10.099 DEBUG:teuthology.orchestra.run.smithi092:> sudo hostname $(hostname -s) 2024-04-02T14:22:10.125 DEBUG:teuthology.orchestra.run.smithi177:> sudo hostname $(hostname -s) 2024-04-02T14:22:10.152 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-04-02T14:22:10.153 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=191d963ac9de7322da8aff20b567b3fa458e2ef6 2024-04-02T14:22:10.371 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '191d963ac9de7322da8aff20b567b3fa458e2ef6', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/78154/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-2609-g191d963a', 'node_name': '172.21.5.35+adami05', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-2609.g191d963a'}, 'url': 'https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240402.004512/191d963ac9de7322da8aff20b567b3fa458e2ef6/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-04-02 02:04:04.615852', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-batrick-testing-20240402.004512', 'chacra_url': 'https://1.chacra.ceph.com/repos/ceph/wip-batrick-testing-20240402.004512/191d963ac9de7322da8aff20b567b3fa458e2ef6/centos/9/flavors/default/', 'archs': ['source', 'x86_64'], 'distro': 'centos'}] 2024-04-02T14:22:10.641 INFO:tasks.util.chacra:got chacra host 1.chacra.ceph.com, ref wip-batrick-testing-20240402.004512, sha1 191d963ac9de7322da8aff20b567b3fa458e2ef6 from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=191d963ac9de7322da8aff20b567b3fa458e2ef6 2024-04-02T14:22:10.642 INFO:tasks.cephadm:Discovered cachra url: https://1.chacra.ceph.com/binaries/ceph/wip-batrick-testing-20240402.004512/191d963ac9de7322da8aff20b567b3fa458e2ef6/centos/9/x86_64/flavors/default/cephadm 2024-04-02T14:22:10.643 DEBUG:teuthology.orchestra.run.smithi090:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/wip-batrick-testing-20240402.004512/191d963ac9de7322da8aff20b567b3fa458e2ef6/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-02T14:22:11.365 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r--. 1 ubuntu ubuntu 781237 Apr 2 14:22 /home/ubuntu/cephtest/cephadm 2024-04-02T14:22:11.365 DEBUG:teuthology.orchestra.run.smithi092:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/wip-batrick-testing-20240402.004512/191d963ac9de7322da8aff20b567b3fa458e2ef6/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-02T14:22:11.909 INFO:teuthology.orchestra.run.smithi092.stdout:-rw-r--r--. 1 ubuntu ubuntu 781237 Apr 2 14:22 /home/ubuntu/cephtest/cephadm 2024-04-02T14:22:11.910 DEBUG:teuthology.orchestra.run.smithi177:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/wip-batrick-testing-20240402.004512/191d963ac9de7322da8aff20b567b3fa458e2ef6/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-02T14:22:12.513 INFO:teuthology.orchestra.run.smithi177.stdout:-rw-r--r--. 1 ubuntu ubuntu 781237 Apr 2 14:22 /home/ubuntu/cephtest/cephadm 2024-04-02T14:22:12.513 DEBUG:teuthology.orchestra.run.smithi090:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-02T14:22:12.572 DEBUG:teuthology.orchestra.run.smithi092:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-02T14:22:12.590 DEBUG:teuthology.orchestra.run.smithi177:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-02T14:22:12.627 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:191d963ac9de7322da8aff20b567b3fa458e2ef6 on all hosts... 2024-04-02T14:22:12.627 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:191d963ac9de7322da8aff20b567b3fa458e2ef6 pull 2024-04-02T14:22:12.635 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:191d963ac9de7322da8aff20b567b3fa458e2ef6 pull 2024-04-02T14:22:12.637 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:191d963ac9de7322da8aff20b567b3fa458e2ef6 pull 2024-04-02T14:22:13.494 INFO:teuthology.orchestra.run.smithi092.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:191d963ac9de7322da8aff20b567b3fa458e2ef6... 2024-04-02T14:22:13.541 INFO:teuthology.orchestra.run.smithi177.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:191d963ac9de7322da8aff20b567b3fa458e2ef6... 2024-04-02T14:22:13.983 INFO:teuthology.orchestra.run.smithi090.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:191d963ac9de7322da8aff20b567b3fa458e2ef6... 2024-04-02T14:22:49.926 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-04-02T14:22:49.926 INFO:teuthology.orchestra.run.smithi177.stdout: "ceph_version": "ceph version 19.0.0-2609-g191d963a (191d963ac9de7322da8aff20b567b3fa458e2ef6) squid (dev)", 2024-04-02T14:22:49.926 INFO:teuthology.orchestra.run.smithi177.stdout: "image_id": "f7301e070545fc23d8e5b7b052a2c15e832a0e57102f1acad8e4f3583709d23f", 2024-04-02T14:22:49.926 INFO:teuthology.orchestra.run.smithi177.stdout: "repo_digests": [ 2024-04-02T14:22:49.926 INFO:teuthology.orchestra.run.smithi177.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534" 2024-04-02T14:22:49.927 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-04-02T14:22:49.927 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-04-02T14:22:54.983 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-04-02T14:22:54.983 INFO:teuthology.orchestra.run.smithi092.stdout: "ceph_version": "ceph version 19.0.0-2609-g191d963a (191d963ac9de7322da8aff20b567b3fa458e2ef6) squid (dev)", 2024-04-02T14:22:54.983 INFO:teuthology.orchestra.run.smithi092.stdout: "image_id": "f7301e070545fc23d8e5b7b052a2c15e832a0e57102f1acad8e4f3583709d23f", 2024-04-02T14:22:54.983 INFO:teuthology.orchestra.run.smithi092.stdout: "repo_digests": [ 2024-04-02T14:22:54.983 INFO:teuthology.orchestra.run.smithi092.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534" 2024-04-02T14:22:54.983 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-04-02T14:22:54.983 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-04-02T14:23:04.607 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-02T14:23:04.607 INFO:teuthology.orchestra.run.smithi090.stdout: "ceph_version": "ceph version 19.0.0-2609-g191d963a (191d963ac9de7322da8aff20b567b3fa458e2ef6) squid (dev)", 2024-04-02T14:23:04.607 INFO:teuthology.orchestra.run.smithi090.stdout: "image_id": "f7301e070545fc23d8e5b7b052a2c15e832a0e57102f1acad8e4f3583709d23f", 2024-04-02T14:23:04.607 INFO:teuthology.orchestra.run.smithi090.stdout: "repo_digests": [ 2024-04-02T14:23:04.607 INFO:teuthology.orchestra.run.smithi090.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534" 2024-04-02T14:23:04.607 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-02T14:23:04.608 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-02T14:23:04.642 DEBUG:teuthology.orchestra.run.smithi090:> sudo mkdir -p /etc/ceph 2024-04-02T14:23:04.725 DEBUG:teuthology.orchestra.run.smithi092:> sudo mkdir -p /etc/ceph 2024-04-02T14:23:04.761 DEBUG:teuthology.orchestra.run.smithi177:> sudo mkdir -p /etc/ceph 2024-04-02T14:23:04.798 DEBUG:teuthology.orchestra.run.smithi090:> sudo chmod 777 /etc/ceph 2024-04-02T14:23:04.889 DEBUG:teuthology.orchestra.run.smithi092:> sudo chmod 777 /etc/ceph 2024-04-02T14:23:04.920 DEBUG:teuthology.orchestra.run.smithi177:> sudo chmod 777 /etc/ceph 2024-04-02T14:23:04.953 INFO:tasks.cephadm:Writing seed config... 2024-04-02T14:23:04.954 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-04-02T14:23:04.954 INFO:tasks.cephadm: override: [client] debug client = 20 2024-04-02T14:23:04.954 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-04-02T14:23:04.955 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-04-02T14:23:04.955 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-04-02T14:23:04.955 INFO:tasks.cephadm: override: [global] ms die on skipped message = False 2024-04-02T14:23:04.955 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-04-02T14:23:04.955 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-04-02T14:23:04.955 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-04-02T14:23:04.955 INFO:tasks.cephadm: override: [mds] mds bal fragment size max = 10000 2024-04-02T14:23:04.955 INFO:tasks.cephadm: override: [mds] mds bal merge size = 5 2024-04-02T14:23:04.955 INFO:tasks.cephadm: override: [mds] mds bal split bits = 3 2024-04-02T14:23:04.955 INFO:tasks.cephadm: override: [mds] mds bal split size = 100 2024-04-02T14:23:04.955 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-04-02T14:23:04.955 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-04-02T14:23:04.955 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-04-02T14:23:04.955 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-04-02T14:23:04.955 INFO:tasks.cephadm: override: [mds] mds_bal_replicate_threshold = 1 2024-04-02T14:23:04.956 INFO:tasks.cephadm: override: [mds] mds_export_ephemeral_random_max = 0.1 2024-04-02T14:23:04.956 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-04-02T14:23:04.956 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-04-02T14:23:04.956 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-04-02T14:23:04.956 INFO:tasks.cephadm: override: [mgr] cephsqlite lock renewal timeout = 900000 2024-04-02T14:23:04.956 INFO:tasks.cephadm: override: [mgr] debug cephsqlite = 20 2024-04-02T14:23:04.956 INFO:tasks.cephadm: override: [mgr] debug client = 20 2024-04-02T14:23:04.956 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-04-02T14:23:04.956 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-04-02T14:23:04.956 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-04-02T14:23:04.956 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-04-02T14:23:04.956 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-04-02T14:23:04.956 INFO:tasks.cephadm: override: [mon] mon down mkfs grace = 300 2024-04-02T14:23:04.956 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-04-02T14:23:04.956 INFO:tasks.cephadm: override: [osd] bdev async discard = True 2024-04-02T14:23:04.956 INFO:tasks.cephadm: override: [osd] bdev enable discard = True 2024-04-02T14:23:04.957 INFO:tasks.cephadm: override: [osd] bluestore allocator = bitmap 2024-04-02T14:23:04.957 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-04-02T14:23:04.957 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-04-02T14:23:04.957 INFO:tasks.cephadm: override: [osd] debug bluefs = 1/20 2024-04-02T14:23:04.957 INFO:tasks.cephadm: override: [osd] debug bluestore = 1/20 2024-04-02T14:23:04.957 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-04-02T14:23:04.957 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-04-02T14:23:04.957 INFO:tasks.cephadm: override: [osd] debug rocksdb = 4/10 2024-04-02T14:23:04.957 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-04-02T14:23:04.957 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-04-02T14:23:04.957 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-04-02T14:23:04.957 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-04-02T14:23:04.957 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-04-02T14:23:04.957 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-04-02T14:23:04.957 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-04-02T14:23:04.958 INFO:tasks.cephadm: override: [osd] osd_max_omap_entries_per_request = 10 2024-04-02T14:23:04.958 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-04-02T14:23:04.958 DEBUG:teuthology.orchestra.run.smithi090:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-04-02T14:23:05.018 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 6449448a-f0fc-11ee-b647-cb9ed24678a4 ms die on skipped message = False [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bdev async discard = True bdev enable discard = True bluestore allocator = bitmap bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 1/20 debug bluestore = 1/20 debug ms = 1 debug osd = 20 debug rocksdb = 4/10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 osd_max_omap_entries_per_request = 10 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false cephsqlite lock renewal timeout = 900000 debug cephsqlite = 20 debug client = 20 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon down mkfs grace = 300 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds bal fragment size max = 10000 mds bal merge size = 5 mds bal split bits = 3 mds bal split size = 100 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True mds_bal_replicate_threshold = 1 mds_export_ephemeral_random_max = 0.1 osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-04-02T14:23:05.019 DEBUG:teuthology.orchestra.run.smithi090:mon.a> sudo journalctl -f -n 0 -u ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4@mon.a.service 2024-04-02T14:23:05.066 DEBUG:teuthology.orchestra.run.smithi090:mgr.x> sudo journalctl -f -n 0 -u ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4@mgr.x.service 2024-04-02T14:23:05.114 INFO:tasks.cephadm:Bootstrapping... 2024-04-02T14:23:05.115 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:191d963ac9de7322da8aff20b567b3fa458e2ef6 -v bootstrap --fsid 6449448a-f0fc-11ee-b647-cb9ed24678a4 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.90 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-04-02T14:23:05.388 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-04-02T14:23:05.389 INFO:teuthology.orchestra.run.smithi090.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:191d963ac9de7322da8aff20b567b3fa458e2ef6', '-v', 'bootstrap', '--fsid', '6449448a-f0fc-11ee-b647-cb9ed24678a4', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.90', '--skip-admin-label'] 2024-04-02T14:23:05.389 INFO:teuthology.orchestra.run.smithi090.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-04-02T14:23:05.389 INFO:teuthology.orchestra.run.smithi090.stdout:Verifying podman|docker is present... 2024-04-02T14:23:05.442 INFO:teuthology.orchestra.run.smithi090.stdout:/bin/podman: stdout 4.9.4-dev 2024-04-02T14:23:05.443 INFO:teuthology.orchestra.run.smithi090.stdout:Verifying lvm2 is present... 2024-04-02T14:23:05.443 INFO:teuthology.orchestra.run.smithi090.stdout:Verifying time synchronization is in place... 2024-04-02T14:23:05.459 INFO:teuthology.orchestra.run.smithi090.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-04-02T14:23:05.460 INFO:teuthology.orchestra.run.smithi090.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-04-02T14:23:05.476 INFO:teuthology.orchestra.run.smithi090.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-04-02T14:23:05.476 INFO:teuthology.orchestra.run.smithi090.stdout:systemctl: stdout inactive 2024-04-02T14:23:05.498 INFO:teuthology.orchestra.run.smithi090.stdout:systemctl: stdout enabled 2024-04-02T14:23:05.516 INFO:teuthology.orchestra.run.smithi090.stdout:systemctl: stdout active 2024-04-02T14:23:05.517 INFO:teuthology.orchestra.run.smithi090.stdout:Unit chronyd.service is enabled and running 2024-04-02T14:23:05.517 INFO:teuthology.orchestra.run.smithi090.stdout:Repeating the final host check... 2024-04-02T14:23:05.574 INFO:teuthology.orchestra.run.smithi090.stdout:/bin/podman: stdout 4.9.4-dev 2024-04-02T14:23:05.574 INFO:teuthology.orchestra.run.smithi090.stdout:podman (/bin/podman) version 4.9.4 is present 2024-04-02T14:23:05.575 INFO:teuthology.orchestra.run.smithi090.stdout:systemctl is present 2024-04-02T14:23:05.575 INFO:teuthology.orchestra.run.smithi090.stdout:lvcreate is present 2024-04-02T14:23:05.588 INFO:teuthology.orchestra.run.smithi090.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-04-02T14:23:05.588 INFO:teuthology.orchestra.run.smithi090.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-04-02T14:23:05.604 INFO:teuthology.orchestra.run.smithi090.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-04-02T14:23:05.605 INFO:teuthology.orchestra.run.smithi090.stdout:systemctl: stdout inactive 2024-04-02T14:23:05.627 INFO:teuthology.orchestra.run.smithi090.stdout:systemctl: stdout enabled 2024-04-02T14:23:05.646 INFO:teuthology.orchestra.run.smithi090.stdout:systemctl: stdout active 2024-04-02T14:23:05.647 INFO:teuthology.orchestra.run.smithi090.stdout:Unit chronyd.service is enabled and running 2024-04-02T14:23:05.647 INFO:teuthology.orchestra.run.smithi090.stdout:Host looks OK 2024-04-02T14:23:05.647 INFO:teuthology.orchestra.run.smithi090.stdout:Cluster fsid: 6449448a-f0fc-11ee-b647-cb9ed24678a4 2024-04-02T14:23:05.647 INFO:teuthology.orchestra.run.smithi090.stdout:Acquiring lock 140222041674416 on /run/cephadm/6449448a-f0fc-11ee-b647-cb9ed24678a4.lock 2024-04-02T14:23:05.648 INFO:teuthology.orchestra.run.smithi090.stdout:Lock 140222041674416 acquired on /run/cephadm/6449448a-f0fc-11ee-b647-cb9ed24678a4.lock 2024-04-02T14:23:05.649 INFO:teuthology.orchestra.run.smithi090.stdout:Verifying IP 172.21.15.90 port 3300 ... 2024-04-02T14:23:05.650 INFO:teuthology.orchestra.run.smithi090.stdout:Verifying IP 172.21.15.90 port 6789 ... 2024-04-02T14:23:05.650 INFO:teuthology.orchestra.run.smithi090.stdout:Base mon IP(s) is [172.21.15.90:3300, 172.21.15.90:6789], mon addrv is [v2:172.21.15.90:3300,v1:172.21.15.90:6789] 2024-04-02T14:23:05.664 INFO:teuthology.orchestra.run.smithi090.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.90 metric 100 2024-04-02T14:23:05.664 INFO:teuthology.orchestra.run.smithi090.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.90 metric 100 2024-04-02T14:23:05.676 INFO:teuthology.orchestra.run.smithi090.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-04-02T14:23:05.676 INFO:teuthology.orchestra.run.smithi090.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2024-04-02T14:23:05.676 INFO:teuthology.orchestra.run.smithi090.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 1024 expires 1791sec hoplimit 64 pref medium 2024-04-02T14:23:05.686 INFO:teuthology.orchestra.run.smithi090.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-04-02T14:23:05.687 INFO:teuthology.orchestra.run.smithi090.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-04-02T14:23:05.687 INFO:teuthology.orchestra.run.smithi090.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-04-02T14:23:05.687 INFO:teuthology.orchestra.run.smithi090.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-04-02T14:23:05.687 INFO:teuthology.orchestra.run.smithi090.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:fe8f:d767/64 scope link 2024-04-02T14:23:05.687 INFO:teuthology.orchestra.run.smithi090.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-04-02T14:23:05.689 INFO:teuthology.orchestra.run.smithi090.stdout:Mon IP `172.21.15.90` is in CIDR network `172.21.0.0/20` 2024-04-02T14:23:05.689 INFO:teuthology.orchestra.run.smithi090.stdout:Mon IP `172.21.15.90` is in CIDR network `172.21.0.0/20` 2024-04-02T14:23:05.689 INFO:teuthology.orchestra.run.smithi090.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-04-02T14:23:05.692 INFO:teuthology.orchestra.run.smithi090.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-04-02T14:23:05.694 INFO:teuthology.orchestra.run.smithi090.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:191d963ac9de7322da8aff20b567b3fa458e2ef6... 2024-04-02T14:23:06.328 INFO:teuthology.orchestra.run.smithi090.stdout:/bin/podman: stdout f7301e070545fc23d8e5b7b052a2c15e832a0e57102f1acad8e4f3583709d23f 2024-04-02T14:23:06.328 INFO:teuthology.orchestra.run.smithi090.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:191d963ac9de7322da8aff20b567b3fa458e2ef6... 2024-04-02T14:23:06.328 INFO:teuthology.orchestra.run.smithi090.stdout:/bin/podman: stderr Getting image source signatures 2024-04-02T14:23:06.328 INFO:teuthology.orchestra.run.smithi090.stdout:/bin/podman: stderr Copying blob sha256:db8c11fadded9ebce76f2e720ef87f32e8913958c61c526c89a5e4e2c41a8200 2024-04-02T14:23:06.328 INFO:teuthology.orchestra.run.smithi090.stdout:/bin/podman: stderr Copying blob sha256:11da15500eca1f826c328eb3265caa30b0a3730510e2a32023084d6fe9cee20c 2024-04-02T14:23:06.328 INFO:teuthology.orchestra.run.smithi090.stdout:/bin/podman: stderr Copying config sha256:f7301e070545fc23d8e5b7b052a2c15e832a0e57102f1acad8e4f3583709d23f 2024-04-02T14:23:06.328 INFO:teuthology.orchestra.run.smithi090.stdout:/bin/podman: stderr Writing manifest to image destination 2024-04-02T14:23:07.542 INFO:teuthology.orchestra.run.smithi090.stdout:ceph: stdout ceph version 19.0.0-2609-g191d963a (191d963ac9de7322da8aff20b567b3fa458e2ef6) squid (dev) 2024-04-02T14:23:07.543 INFO:teuthology.orchestra.run.smithi090.stdout:Ceph version: ceph version 19.0.0-2609-g191d963a (191d963ac9de7322da8aff20b567b3fa458e2ef6) squid (dev) 2024-04-02T14:23:07.543 INFO:teuthology.orchestra.run.smithi090.stdout:Extracting ceph user uid/gid from container image... 2024-04-02T14:23:08.708 INFO:teuthology.orchestra.run.smithi090.stdout:stat: stdout 167 167 2024-04-02T14:23:08.709 INFO:teuthology.orchestra.run.smithi090.stdout:Creating initial keys... 2024-04-02T14:23:09.747 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph-authtool: stdout AQDNFAxmuKomBxAA7DFb5d5NXBqneoDiWsyA6g== 2024-04-02T14:23:10.881 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph-authtool: stdout AQDOFAxmALk0DBAATQEcJuj7na2Gy5f0iKKNCw== 2024-04-02T14:23:12.019 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph-authtool: stdout AQDPFAxmDvmeFxAAjIxdJNI+qNGC5AgAQWxJUQ== 2024-04-02T14:23:12.021 INFO:teuthology.orchestra.run.smithi090.stdout:Creating initial monmap... 2024-04-02T14:23:13.177 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-02T14:23:13.177 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-04-02T14:23:13.178 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 6449448a-f0fc-11ee-b647-cb9ed24678a4 2024-04-02T14:23:13.178 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-02T14:23:13.178 INFO:teuthology.orchestra.run.smithi090.stdout:monmaptool for a [v2:172.21.15.90:3300,v1:172.21.15.90:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-02T14:23:13.178 INFO:teuthology.orchestra.run.smithi090.stdout:setting min_mon_release = quincy 2024-04-02T14:23:13.178 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/monmaptool: set fsid to 6449448a-f0fc-11ee-b647-cb9ed24678a4 2024-04-02T14:23:13.178 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-02T14:23:13.178 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-02T14:23:13.178 INFO:teuthology.orchestra.run.smithi090.stdout:Creating mon... 2024-04-02T14:23:14.323 INFO:teuthology.orchestra.run.smithi090.stdout:create mon.a on 2024-04-02T14:23:14.674 INFO:teuthology.orchestra.run.smithi090.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-04-02T14:23:14.898 INFO:teuthology.orchestra.run.smithi090.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-04-02T14:23:15.169 INFO:teuthology.orchestra.run.smithi090.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4.target → /etc/systemd/system/ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4.target. 2024-04-02T14:23:15.169 INFO:teuthology.orchestra.run.smithi090.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4.target → /etc/systemd/system/ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4.target. 2024-04-02T14:23:15.516 INFO:teuthology.orchestra.run.smithi090.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4@mon.a 2024-04-02T14:23:15.517 INFO:teuthology.orchestra.run.smithi090.stdout:systemctl: stderr Failed to reset failed state of unit ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4@mon.a.service: Unit ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4@mon.a.service not loaded. 2024-04-02T14:23:15.792 INFO:teuthology.orchestra.run.smithi090.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4.target.wants/ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4@mon.a.service → /etc/systemd/system/ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4@.service. 2024-04-02T14:23:16.440 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:16 smithi090 podman[31291]: 2024-04-02 14:23:16.172615639 +0000 UTC m=+0.131872500 container create f66c7b9b4bf4aaa3ddd47577eea49e857e8efef8edad115d1cfa2dc3556254fd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:191d963ac9de7322da8aff20b567b3fa458e2ef6, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mon-a, GIT_CLEAN=True, org.label-schema.build-date=20240326, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240402.004512-191d963, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-02T14:23:16.588 INFO:teuthology.orchestra.run.smithi090.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-02T14:23:16.588 INFO:teuthology.orchestra.run.smithi090.stdout:systemctl: stdout disabled 2024-04-02T14:23:16.609 INFO:teuthology.orchestra.run.smithi090.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-02T14:23:16.609 INFO:teuthology.orchestra.run.smithi090.stdout:systemctl: stdout inactive 2024-04-02T14:23:16.609 INFO:teuthology.orchestra.run.smithi090.stdout:firewalld.service is not enabled 2024-04-02T14:23:16.610 INFO:teuthology.orchestra.run.smithi090.stdout:Not possible to enable service . firewalld.service is not available 2024-04-02T14:23:16.611 INFO:teuthology.orchestra.run.smithi090.stdout:Waiting for mon to start... 2024-04-02T14:23:16.611 INFO:teuthology.orchestra.run.smithi090.stdout:Waiting for mon... 2024-04-02T14:23:16.754 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:16 smithi090 podman[31291]: 2024-04-02 14:23:16.445058522 +0000 UTC m=+0.404315389 container init f66c7b9b4bf4aaa3ddd47577eea49e857e8efef8edad115d1cfa2dc3556254fd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:191d963ac9de7322da8aff20b567b3fa458e2ef6, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mon-a, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240326, io.buildah.version=1.33.5, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240402.004512-191d963, org.label-schema.license=GPLv2) 2024-04-02T14:23:16.754 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:16 smithi090 podman[31291]: 2024-04-02 14:23:16.456346995 +0000 UTC m=+0.415603843 container start f66c7b9b4bf4aaa3ddd47577eea49e857e8efef8edad115d1cfa2dc3556254fd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:191d963ac9de7322da8aff20b567b3fa458e2ef6, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mon-a, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240326, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240402.004512-191d963) 2024-04-02T14:23:16.754 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:16 smithi090 ceph-mon[31310]: mkfs 6449448a-f0fc-11ee-b647-cb9ed24678a4 2024-04-02T14:23:16.754 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:16 smithi090 ceph-mon[31310]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-02T14:23:16.754 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:16 smithi090 bash[31291]: f66c7b9b4bf4aaa3ddd47577eea49e857e8efef8edad115d1cfa2dc3556254fd 2024-04-02T14:23:16.754 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:16 smithi090 systemd[1]: Started Ceph mon.a for 6449448a-f0fc-11ee-b647-cb9ed24678a4. 2024-04-02T14:23:17.689 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:17 smithi090 ceph-mon[31310]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-02T14:23:17.689 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:17 smithi090 ceph-mon[31310]: monmap epoch 1 2024-04-02T14:23:17.689 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:17 smithi090 ceph-mon[31310]: fsid 6449448a-f0fc-11ee-b647-cb9ed24678a4 2024-04-02T14:23:17.689 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:17 smithi090 ceph-mon[31310]: last_changed 2024-04-02T14:23:12.545436+0000 2024-04-02T14:23:17.689 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:17 smithi090 ceph-mon[31310]: created 2024-04-02T14:23:12.545436+0000 2024-04-02T14:23:17.689 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:17 smithi090 ceph-mon[31310]: min_mon_release 19 (squid) 2024-04-02T14:23:17.689 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:17 smithi090 ceph-mon[31310]: election_strategy: 1 2024-04-02T14:23:17.689 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:17 smithi090 ceph-mon[31310]: 0: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.a 2024-04-02T14:23:17.690 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:17 smithi090 ceph-mon[31310]: fsmap 2024-04-02T14:23:17.690 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:17 smithi090 ceph-mon[31310]: osdmap e1: 0 total, 0 up, 0 in 2024-04-02T14:23:17.690 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:17 smithi090 ceph-mon[31310]: mgrmap e1: no daemons active 2024-04-02T14:23:17.690 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:17 smithi090 ceph-mon[31310]: from='client.? 172.21.15.90:0/4290657369' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-04-02T14:23:18.113 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout cluster: 2024-04-02T14:23:18.113 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout id: 6449448a-f0fc-11ee-b647-cb9ed24678a4 2024-04-02T14:23:18.113 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-04-02T14:23:18.113 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout 2024-04-02T14:23:18.119 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout services: 2024-04-02T14:23:18.119 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.770474s) 2024-04-02T14:23:18.119 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-04-02T14:23:18.120 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-04-02T14:23:18.120 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout 2024-04-02T14:23:18.120 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout data: 2024-04-02T14:23:18.120 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-04-02T14:23:18.120 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-04-02T14:23:18.120 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-04-02T14:23:18.120 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout pgs: 2024-04-02T14:23:18.120 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout 2024-04-02T14:23:18.121 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:17.207+0000 7f54ab400700 1 Processor -- start 2024-04-02T14:23:18.121 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:17.208+0000 7f54ab400700 1 -- start start 2024-04-02T14:23:18.121 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:17.208+0000 7f54ab400700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f54a4108270 0x7f54a4108690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:23:18.121 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:17.208+0000 7f54ab400700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f54a4108c60 con 0x7f54a4108270 2024-04-02T14:23:18.121 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:17.209+0000 7f54a8a00700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f54a4108270 0x7f54a4108690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:23:18.121 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:17.209+0000 7f54a8a00700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f54a4108270 0x7f54a4108690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:50990/0 (socket says 172.21.15.90:50990) 2024-04-02T14:23:18.121 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:17.209+0000 7f54a8a00700 1 -- 172.21.15.90:0/165578979 learned_addr learned my addr 172.21.15.90:0/165578979 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:23:18.121 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:17.209+0000 7f54a8a00700 1 -- 172.21.15.90:0/165578979 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54a4109470 con 0x7f54a4108270 2024-04-02T14:23:18.121 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:17.210+0000 7f54a8a00700 1 --2- 172.21.15.90:0/165578979 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f54a4108270 0x7f54a4108690 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f54940098a0 tx=0x7f5494009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7210b32f048695d5 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:23:18.121 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:17.210+0000 7f54a3e00700 1 -- 172.21.15.90:0/165578979 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5494004160 con 0x7f54a4108270 2024-04-02T14:23:18.121 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:17.210+0000 7f54a3e00700 1 -- 172.21.15.90:0/165578979 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5494004300 con 0x7f54a4108270 2024-04-02T14:23:18.121 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:17.210+0000 7f54a3e00700 1 -- 172.21.15.90:0/165578979 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f54940036a0 con 0x7f54a4108270 2024-04-02T14:23:18.121 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:17.211+0000 7f54ab400700 1 -- 172.21.15.90:0/165578979 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f54a4108270 msgr2=0x7f54a4108690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:23:18.121 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:17.211+0000 7f54ab400700 1 --2- 172.21.15.90:0/165578979 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f54a4108270 0x7f54a4108690 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f54940098a0 tx=0x7f5494009bb0 comp rx=0 tx=0).stop 2024-04-02T14:23:18.122 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:17.211+0000 7f54ab400700 1 -- 172.21.15.90:0/165578979 shutdown_connections 2024-04-02T14:23:18.122 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:17.211+0000 7f54ab400700 1 --2- 172.21.15.90:0/165578979 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f54a4108270 0x7f54a4108690 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:23:18.122 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:17.211+0000 7f54ab400700 1 -- 172.21.15.90:0/165578979 >> 172.21.15.90:0/165578979 conn(0x7f54a4103830 msgr2=0x7f54a4105c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:23:18.122 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:17.211+0000 7f54ab400700 1 -- 172.21.15.90:0/165578979 shutdown_connections 2024-04-02T14:23:18.122 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:17.211+0000 7f54ab400700 1 -- 172.21.15.90:0/165578979 wait complete. 2024-04-02T14:23:18.122 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:17.212+0000 7f54ab400700 1 Processor -- start 2024-04-02T14:23:18.122 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:17.212+0000 7f54ab400700 1 -- start start 2024-04-02T14:23:18.122 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:17.213+0000 7f54ab400700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f54a4108270 0x7f54a407ca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:23:18.122 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:17.213+0000 7f54ab400700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f54a41099a0 con 0x7f54a4108270 2024-04-02T14:23:18.122 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:17.213+0000 7f54a8a00700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f54a4108270 0x7f54a407ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:23:18.122 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:17.213+0000 7f54a8a00700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f54a4108270 0x7f54a407ca70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:50992/0 (socket says 172.21.15.90:50992) 2024-04-02T14:23:18.122 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:17.213+0000 7f54a8a00700 1 -- 172.21.15.90:0/4290657369 learned_addr learned my addr 172.21.15.90:0/4290657369 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:23:18.122 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:17.214+0000 7f54a8a00700 1 -- 172.21.15.90:0/4290657369 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54a407cfb0 con 0x7f54a4108270 2024-04-02T14:23:18.123 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:17.214+0000 7f54a8a00700 1 --2- 172.21.15.90:0/4290657369 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f54a4108270 0x7f54a407ca70 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f5494000c00 tx=0x7f5494004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:23:18.123 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:17.214+0000 7f54a2000700 1 -- 172.21.15.90:0/4290657369 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5494003db0 con 0x7f54a4108270 2024-04-02T14:23:18.123 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:17.214+0000 7f54a2000700 1 -- 172.21.15.90:0/4290657369 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5494011630 con 0x7f54a4108270 2024-04-02T14:23:18.123 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:17.214+0000 7f54ab400700 1 -- 172.21.15.90:0/4290657369 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f54a407b1a0 con 0x7f54a4108270 2024-04-02T14:23:18.123 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:17.214+0000 7f54a2000700 1 -- 172.21.15.90:0/4290657369 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5494011950 con 0x7f54a4108270 2024-04-02T14:23:18.123 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:17.214+0000 7f54ab400700 1 -- 172.21.15.90:0/4290657369 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f54a407b6e0 con 0x7f54a4108270 2024-04-02T14:23:18.123 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:17.215+0000 7f54a2000700 1 -- 172.21.15.90:0/4290657369 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f5494011af0 con 0x7f54a4108270 2024-04-02T14:23:18.123 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:17.215+0000 7f54a2000700 1 -- 172.21.15.90:0/4290657369 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f549401c450 con 0x7f54a4108270 2024-04-02T14:23:18.123 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:17.215+0000 7f54ab400700 1 -- 172.21.15.90:0/4290657369 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f54a4063c20 con 0x7f54a4108270 2024-04-02T14:23:18.123 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:17.219+0000 7f54a2000700 1 -- 172.21.15.90:0/4290657369 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f5494032020 con 0x7f54a4108270 2024-04-02T14:23:18.123 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:17.297+0000 7f54ab400700 1 -- 172.21.15.90:0/4290657369 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "status"} v 0) -- 0x7f54a4195230 con 0x7f54a4108270 2024-04-02T14:23:18.123 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:17.298+0000 7f54a2000700 1 -- 172.21.15.90:0/4290657369 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) ==== 54+0+317 (secure 0 0 0) 0x7f5494004260 con 0x7f54a4108270 2024-04-02T14:23:18.123 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:17.299+0000 7f54ab400700 1 -- 172.21.15.90:0/4290657369 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f54a4108270 msgr2=0x7f54a407ca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:23:18.123 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:17.299+0000 7f54ab400700 1 --2- 172.21.15.90:0/4290657369 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f54a4108270 0x7f54a407ca70 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f5494000c00 tx=0x7f5494004750 comp rx=0 tx=0).stop 2024-04-02T14:23:18.124 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:17.300+0000 7f54ab400700 1 -- 172.21.15.90:0/4290657369 shutdown_connections 2024-04-02T14:23:18.124 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:17.300+0000 7f54ab400700 1 --2- 172.21.15.90:0/4290657369 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f54a4108270 0x7f54a407ca70 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:23:18.124 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:17.300+0000 7f54ab400700 1 -- 172.21.15.90:0/4290657369 >> 172.21.15.90:0/4290657369 conn(0x7f54a4103830 msgr2=0x7f54a4192670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:23:18.124 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:17.300+0000 7f54ab400700 1 -- 172.21.15.90:0/4290657369 shutdown_connections 2024-04-02T14:23:18.124 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:17.300+0000 7f54ab400700 1 -- 172.21.15.90:0/4290657369 wait complete. 2024-04-02T14:23:18.124 INFO:teuthology.orchestra.run.smithi090.stdout:mon is available 2024-04-02T14:23:18.124 INFO:teuthology.orchestra.run.smithi090.stdout:Assimilating anything we can from ceph.conf... 2024-04-02T14:23:18.939 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:18 smithi090 ceph-mon[31310]: from='client.? 172.21.15.90:0/2261876550' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-02T14:23:18.939 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:18 smithi090 ceph-mon[31310]: from='client.? 172.21.15.90:0/2261876550' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-04-02T14:23:19.401 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout 2024-04-02T14:23:19.402 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout [global] 2024-04-02T14:23:19.402 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout fsid = 6449448a-f0fc-11ee-b647-cb9ed24678a4 2024-04-02T14:23:19.402 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.90:3300,v1:172.21.15.90:6789] 2024-04-02T14:23:19.403 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-04-02T14:23:19.403 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-04-02T14:23:19.403 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-04-02T14:23:19.403 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-04-02T14:23:19.406 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout 2024-04-02T14:23:19.406 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout [mgr] 2024-04-02T14:23:19.406 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-04-02T14:23:19.406 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout 2024-04-02T14:23:19.406 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout [osd] 2024-04-02T14:23:19.406 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout bdev_async_discard = True 2024-04-02T14:23:19.406 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-04-02T14:23:19.406 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-04-02T14:23:19.407 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-04-02T14:23:19.407 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-04-02T14:23:19.407 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-04-02T14:23:19.407 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:18.667+0000 7f2a87200700 1 Processor -- start 2024-04-02T14:23:19.407 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:18.667+0000 7f2a87200700 1 -- start start 2024-04-02T14:23:19.407 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:18.667+0000 7f2a87200700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2a8010ef40 0x7f2a8010f360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:23:19.407 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:18.667+0000 7f2a87200700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f2a8010f930 con 0x7f2a8010ef40 2024-04-02T14:23:19.407 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:18.668+0000 7f2a7fe00700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2a8010ef40 0x7f2a8010f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:23:19.407 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:18.668+0000 7f2a7fe00700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2a8010ef40 0x7f2a8010f360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:51000/0 (socket says 172.21.15.90:51000) 2024-04-02T14:23:19.407 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:18.668+0000 7f2a7fe00700 1 -- 172.21.15.90:0/1017072995 learned_addr learned my addr 172.21.15.90:0/1017072995 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:23:19.407 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:18.668+0000 7f2a7fe00700 1 -- 172.21.15.90:0/1017072995 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a8010fab0 con 0x7f2a8010ef40 2024-04-02T14:23:19.407 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:18.669+0000 7f2a7fe00700 1 --2- 172.21.15.90:0/1017072995 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2a8010ef40 0x7f2a8010f360 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f2a700098a0 tx=0x7f2a70009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7676183eac8d00f8 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:23:19.407 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:18.669+0000 7f2a7ea00700 1 -- 172.21.15.90:0/1017072995 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2a70004160 con 0x7f2a8010ef40 2024-04-02T14:23:19.407 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:18.669+0000 7f2a7ea00700 1 -- 172.21.15.90:0/1017072995 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2a70004300 con 0x7f2a8010ef40 2024-04-02T14:23:19.407 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:18.669+0000 7f2a7ea00700 1 -- 172.21.15.90:0/1017072995 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2a700036a0 con 0x7f2a8010ef40 2024-04-02T14:23:19.408 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:18.670+0000 7f2a87200700 1 -- 172.21.15.90:0/1017072995 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2a8010ef40 msgr2=0x7f2a8010f360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:23:19.411 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:18.670+0000 7f2a87200700 1 --2- 172.21.15.90:0/1017072995 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2a8010ef40 0x7f2a8010f360 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f2a700098a0 tx=0x7f2a70009bb0 comp rx=0 tx=0).stop 2024-04-02T14:23:19.411 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:18.670+0000 7f2a87200700 1 -- 172.21.15.90:0/1017072995 shutdown_connections 2024-04-02T14:23:19.411 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:18.670+0000 7f2a87200700 1 --2- 172.21.15.90:0/1017072995 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2a8010ef40 0x7f2a8010f360 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:23:19.411 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:18.670+0000 7f2a87200700 1 -- 172.21.15.90:0/1017072995 >> 172.21.15.90:0/1017072995 conn(0x7f2a8007d9c0 msgr2=0x7f2a8007fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:23:19.411 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:18.670+0000 7f2a87200700 1 -- 172.21.15.90:0/1017072995 shutdown_connections 2024-04-02T14:23:19.412 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:18.670+0000 7f2a87200700 1 -- 172.21.15.90:0/1017072995 wait complete. 2024-04-02T14:23:19.412 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:18.671+0000 7f2a87200700 1 Processor -- start 2024-04-02T14:23:19.412 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:18.671+0000 7f2a87200700 1 -- start start 2024-04-02T14:23:19.412 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:18.671+0000 7f2a87200700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2a8010ef40 0x7f2a801a1ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:23:19.412 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:18.671+0000 7f2a87200700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f2a801120e0 con 0x7f2a8010ef40 2024-04-02T14:23:19.412 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:18.671+0000 7f2a7fe00700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2a8010ef40 0x7f2a801a1ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:23:19.412 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:18.672+0000 7f2a7fe00700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2a8010ef40 0x7f2a801a1ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:51014/0 (socket says 172.21.15.90:51014) 2024-04-02T14:23:19.412 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:18.672+0000 7f2a7fe00700 1 -- 172.21.15.90:0/2261876550 learned_addr learned my addr 172.21.15.90:0/2261876550 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:23:19.412 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:18.672+0000 7f2a7fe00700 1 -- 172.21.15.90:0/2261876550 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a801a2420 con 0x7f2a8010ef40 2024-04-02T14:23:19.412 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:18.672+0000 7f2a7fe00700 1 --2- 172.21.15.90:0/2261876550 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2a8010ef40 0x7f2a801a1ee0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f2a70000c00 tx=0x7f2a70004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:23:19.412 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:18.672+0000 7f2a7cc00700 1 -- 172.21.15.90:0/2261876550 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2a70003db0 con 0x7f2a8010ef40 2024-04-02T14:23:19.412 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:18.672+0000 7f2a7cc00700 1 -- 172.21.15.90:0/2261876550 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2a70011630 con 0x7f2a8010ef40 2024-04-02T14:23:19.412 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:18.672+0000 7f2a7cc00700 1 -- 172.21.15.90:0/2261876550 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2a70011950 con 0x7f2a8010ef40 2024-04-02T14:23:19.412 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:18.672+0000 7f2a87200700 1 -- 172.21.15.90:0/2261876550 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a801a2710 con 0x7f2a8010ef40 2024-04-02T14:23:19.413 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:18.673+0000 7f2a87200700 1 -- 172.21.15.90:0/2261876550 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a801a53a0 con 0x7f2a8010ef40 2024-04-02T14:23:19.413 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:18.673+0000 7f2a7cc00700 1 -- 172.21.15.90:0/2261876550 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f2a7001a460 con 0x7f2a8010ef40 2024-04-02T14:23:19.413 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:18.673+0000 7f2a7cc00700 1 -- 172.21.15.90:0/2261876550 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f2a7001b780 con 0x7f2a8010ef40 2024-04-02T14:23:19.413 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:18.674+0000 7f2a87200700 1 -- 172.21.15.90:0/2261876550 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2a80063c20 con 0x7f2a8010ef40 2024-04-02T14:23:19.413 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:18.677+0000 7f2a7cc00700 1 -- 172.21.15.90:0/2261876550 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f2a7001b980 con 0x7f2a8010ef40 2024-04-02T14:23:19.413 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:18.752+0000 7f2a87200700 1 -- 172.21.15.90:0/2261876550 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7f2a8002df40 con 0x7f2a8010ef40 2024-04-02T14:23:19.413 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:18.759+0000 7f2a7cc00700 1 -- 172.21.15.90:0/2261876550 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) ==== 70+0+456 (secure 0 0 0) 0x7f2a7001bb60 con 0x7f2a8010ef40 2024-04-02T14:23:19.413 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:18.761+0000 7f2a87200700 1 -- 172.21.15.90:0/2261876550 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2a8010ef40 msgr2=0x7f2a801a1ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:23:19.413 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:18.761+0000 7f2a87200700 1 --2- 172.21.15.90:0/2261876550 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2a8010ef40 0x7f2a801a1ee0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f2a70000c00 tx=0x7f2a70004750 comp rx=0 tx=0).stop 2024-04-02T14:23:19.413 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:18.761+0000 7f2a87200700 1 -- 172.21.15.90:0/2261876550 shutdown_connections 2024-04-02T14:23:19.413 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:18.761+0000 7f2a87200700 1 --2- 172.21.15.90:0/2261876550 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2a8010ef40 0x7f2a801a1ee0 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:23:19.413 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:18.761+0000 7f2a87200700 1 -- 172.21.15.90:0/2261876550 >> 172.21.15.90:0/2261876550 conn(0x7f2a8007d9c0 msgr2=0x7f2a8007fcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:23:19.413 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:18.761+0000 7f2a87200700 1 -- 172.21.15.90:0/2261876550 shutdown_connections 2024-04-02T14:23:19.413 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:18.761+0000 7f2a87200700 1 -- 172.21.15.90:0/2261876550 wait complete. 2024-04-02T14:23:19.413 INFO:teuthology.orchestra.run.smithi090.stdout:Generating new minimal ceph.conf... 2024-04-02T14:23:20.392 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:20 smithi090 ceph-mon[31310]: from='client.? 172.21.15.90:0/933582467' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:23:20.688 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:19.965+0000 7f63bce00700 1 Processor -- start 2024-04-02T14:23:20.688 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:19.966+0000 7f63bce00700 1 -- start start 2024-04-02T14:23:20.689 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:19.966+0000 7f63bce00700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f63b81082a0 0x7f63b81086c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:23:20.689 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:19.966+0000 7f63bce00700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f63b8108c90 con 0x7f63b81082a0 2024-04-02T14:23:20.689 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:19.966+0000 7f63b5e00700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f63b81082a0 0x7f63b81086c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:23:20.689 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:19.967+0000 7f63b5e00700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f63b81082a0 0x7f63b81086c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:51028/0 (socket says 172.21.15.90:51028) 2024-04-02T14:23:20.689 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:19.967+0000 7f63b5e00700 1 -- 172.21.15.90:0/2654654528 learned_addr learned my addr 172.21.15.90:0/2654654528 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:23:20.689 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:19.967+0000 7f63b5e00700 1 -- 172.21.15.90:0/2654654528 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63b8109400 con 0x7f63b81082a0 2024-04-02T14:23:20.689 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:19.967+0000 7f63b5e00700 1 --2- 172.21.15.90:0/2654654528 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f63b81082a0 0x7f63b81086c0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f63a8009e80 tx=0x7f63a800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eee9b091a4eb45c2 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:23:20.689 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:19.968+0000 7f63b5400700 1 -- 172.21.15.90:0/2654654528 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f63a8017070 con 0x7f63b81082a0 2024-04-02T14:23:20.689 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:19.968+0000 7f63b5400700 1 -- 172.21.15.90:0/2654654528 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f63a800b8c0 con 0x7f63b81082a0 2024-04-02T14:23:20.689 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:19.968+0000 7f63b5400700 1 -- 172.21.15.90:0/2654654528 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f63a80045c0 con 0x7f63b81082a0 2024-04-02T14:23:20.689 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:19.968+0000 7f63bce00700 1 -- 172.21.15.90:0/2654654528 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f63b81082a0 msgr2=0x7f63b81086c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:23:20.689 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:19.968+0000 7f63bce00700 1 --2- 172.21.15.90:0/2654654528 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f63b81082a0 0x7f63b81086c0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f63a8009e80 tx=0x7f63a800b330 comp rx=0 tx=0).stop 2024-04-02T14:23:20.689 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:19.969+0000 7f63bce00700 1 -- 172.21.15.90:0/2654654528 shutdown_connections 2024-04-02T14:23:20.690 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:19.969+0000 7f63bce00700 1 --2- 172.21.15.90:0/2654654528 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f63b81082a0 0x7f63b81086c0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:23:20.690 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:19.969+0000 7f63bce00700 1 -- 172.21.15.90:0/2654654528 >> 172.21.15.90:0/2654654528 conn(0x7f63b8103880 msgr2=0x7f63b8105ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:23:20.690 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:19.969+0000 7f63bce00700 1 -- 172.21.15.90:0/2654654528 shutdown_connections 2024-04-02T14:23:20.690 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:19.969+0000 7f63bce00700 1 -- 172.21.15.90:0/2654654528 wait complete. 2024-04-02T14:23:20.690 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:19.970+0000 7f63bce00700 1 Processor -- start 2024-04-02T14:23:20.690 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:19.970+0000 7f63bce00700 1 -- start start 2024-04-02T14:23:20.690 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:19.970+0000 7f63bce00700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f63b81082a0 0x7f63b819e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:23:20.690 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:19.970+0000 7f63bce00700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f63b810c7b0 con 0x7f63b81082a0 2024-04-02T14:23:20.690 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:19.970+0000 7f63b5e00700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f63b81082a0 0x7f63b819e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:23:20.690 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:19.970+0000 7f63b5e00700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f63b81082a0 0x7f63b819e6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:51040/0 (socket says 172.21.15.90:51040) 2024-04-02T14:23:20.690 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:19.970+0000 7f63b5e00700 1 -- 172.21.15.90:0/933582467 learned_addr learned my addr 172.21.15.90:0/933582467 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:23:20.690 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:19.971+0000 7f63b5e00700 1 -- 172.21.15.90:0/933582467 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63b819ebe0 con 0x7f63b81082a0 2024-04-02T14:23:20.690 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:19.971+0000 7f63b5e00700 1 --2- 172.21.15.90:0/933582467 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f63b81082a0 0x7f63b819e6a0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f63a8009530 tx=0x7f63a800bd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:23:20.691 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:19.971+0000 7f63af400700 1 -- 172.21.15.90:0/933582467 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f63a8020020 con 0x7f63b81082a0 2024-04-02T14:23:20.691 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:19.971+0000 7f63af400700 1 -- 172.21.15.90:0/933582467 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f63a8003b50 con 0x7f63b81082a0 2024-04-02T14:23:20.691 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:19.971+0000 7f63bce00700 1 -- 172.21.15.90:0/933582467 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63b819ee70 con 0x7f63b81082a0 2024-04-02T14:23:20.691 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:19.971+0000 7f63af400700 1 -- 172.21.15.90:0/933582467 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f63a8017040 con 0x7f63b81082a0 2024-04-02T14:23:20.691 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:19.971+0000 7f63bce00700 1 -- 172.21.15.90:0/933582467 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63b81a1b60 con 0x7f63b81082a0 2024-04-02T14:23:20.691 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:19.972+0000 7f63af400700 1 -- 172.21.15.90:0/933582467 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f63a8023020 con 0x7f63b81082a0 2024-04-02T14:23:20.691 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:19.972+0000 7f63af400700 1 -- 172.21.15.90:0/933582467 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f63a800f050 con 0x7f63b81082a0 2024-04-02T14:23:20.691 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:19.973+0000 7f63bce00700 1 -- 172.21.15.90:0/933582467 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f63b8063c20 con 0x7f63b81082a0 2024-04-02T14:23:20.691 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:19.975+0000 7f63af400700 1 -- 172.21.15.90:0/933582467 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f63a802a020 con 0x7f63b81082a0 2024-04-02T14:23:20.691 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:20.052+0000 7f63bce00700 1 -- 172.21.15.90:0/933582467 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7f63b802dd30 con 0x7f63b81082a0 2024-04-02T14:23:20.691 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:20.053+0000 7f63af400700 1 -- 172.21.15.90:0/933582467 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) ==== 76+0+175 (secure 0 0 0) 0x7f63a800b860 con 0x7f63b81082a0 2024-04-02T14:23:20.691 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:20.054+0000 7f63bce00700 1 -- 172.21.15.90:0/933582467 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f63b81082a0 msgr2=0x7f63b819e6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:23:20.691 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:20.054+0000 7f63bce00700 1 --2- 172.21.15.90:0/933582467 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f63b81082a0 0x7f63b819e6a0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f63a8009530 tx=0x7f63a800bd50 comp rx=0 tx=0).stop 2024-04-02T14:23:20.691 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:20.054+0000 7f63bce00700 1 -- 172.21.15.90:0/933582467 shutdown_connections 2024-04-02T14:23:20.692 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:20.054+0000 7f63bce00700 1 --2- 172.21.15.90:0/933582467 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f63b81082a0 0x7f63b819e6a0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:23:20.692 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:20.054+0000 7f63bce00700 1 -- 172.21.15.90:0/933582467 >> 172.21.15.90:0/933582467 conn(0x7f63b8103880 msgr2=0x7f63b8104470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:23:20.692 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:20.054+0000 7f63bce00700 1 -- 172.21.15.90:0/933582467 shutdown_connections 2024-04-02T14:23:20.692 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:20.054+0000 7f63bce00700 1 -- 172.21.15.90:0/933582467 wait complete. 2024-04-02T14:23:20.692 INFO:teuthology.orchestra.run.smithi090.stdout:Restarting the monitor... 2024-04-02T14:23:20.938 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:20 smithi090 systemd[1]: Stopping Ceph mon.a for 6449448a-f0fc-11ee-b647-cb9ed24678a4... 2024-04-02T14:23:21.679 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:21 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mon-a[31305]: 2024-04-02T14:23:21.196+0000 7fd8b5e00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-02T14:23:21.679 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:21 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mon-a[31305]: 2024-04-02T14:23:21.196+0000 7fd8b5e00700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-04-02T14:23:21.679 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:21 smithi090 podman[31551]: 2024-04-02 14:23:21.339976727 +0000 UTC m=+0.270761904 container died f66c7b9b4bf4aaa3ddd47577eea49e857e8efef8edad115d1cfa2dc3556254fd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:191d963ac9de7322da8aff20b567b3fa458e2ef6, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mon-a, RELEASE=wip-batrick-testing-20240402.004512-191d963, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240326, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-02T14:23:21.939 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:21 smithi090 podman[31551]: 2024-04-02 14:23:21.789301268 +0000 UTC m=+0.720086468 container cleanup f66c7b9b4bf4aaa3ddd47577eea49e857e8efef8edad115d1cfa2dc3556254fd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:191d963ac9de7322da8aff20b567b3fa458e2ef6, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mon-a, org.label-schema.build-date=20240326, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240402.004512-191d963, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True) 2024-04-02T14:23:21.939 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:21 smithi090 bash[31551]: ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mon-a 2024-04-02T14:23:22.433 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:22 smithi090 podman[31566]: 2024-04-02 14:23:22.040592453 +0000 UTC m=+0.682524211 container remove f66c7b9b4bf4aaa3ddd47577eea49e857e8efef8edad115d1cfa2dc3556254fd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:191d963ac9de7322da8aff20b567b3fa458e2ef6, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, org.label-schema.build-date=20240326, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240402.004512-191d963) 2024-04-02T14:23:22.806 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:22 smithi090 systemd[1]: ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4@mon.a.service: Deactivated successfully. 2024-04-02T14:23:22.807 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:22 smithi090 systemd[1]: Stopped Ceph mon.a for 6449448a-f0fc-11ee-b647-cb9ed24678a4. 2024-04-02T14:23:22.807 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:22 smithi090 systemd[1]: ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4@mon.a.service: Consumed 1.862s CPU time. 2024-04-02T14:23:22.807 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:22 smithi090 systemd[1]: Starting Ceph mon.a for 6449448a-f0fc-11ee-b647-cb9ed24678a4... 2024-04-02T14:23:23.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:22 smithi090 podman[31656]: 2024-04-02 14:23:22.708801137 +0000 UTC m=+0.031678596 image pull f7301e070545fc23d8e5b7b052a2c15e832a0e57102f1acad8e4f3583709d23f quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:191d963ac9de7322da8aff20b567b3fa458e2ef6 2024-04-02T14:23:28.439 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:27 smithi090 podman[31656]: 2024-04-02 14:23:27.986294354 +0000 UTC m=+5.309171850 container create c279381a76c0b9cad3ae8fc1060f16ff48ffbeccb15d933f53df963923db1eec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:191d963ac9de7322da8aff20b567b3fa458e2ef6, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mon-a, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240326, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, RELEASE=wip-batrick-testing-20240402.004512-191d963, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-04-02T14:23:34.607 INFO:teuthology.orchestra.run.smithi090.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-04-02T14:23:34.749 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 podman[31656]: 2024-04-02 14:23:34.495905389 +0000 UTC m=+11.818782918 container init c279381a76c0b9cad3ae8fc1060f16ff48ffbeccb15d933f53df963923db1eec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:191d963ac9de7322da8aff20b567b3fa458e2ef6, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mon-a, RELEASE=wip-batrick-testing-20240402.004512-191d963, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240326, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-04-02T14:23:34.749 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 podman[31656]: 2024-04-02 14:23:34.506650637 +0000 UTC m=+11.829528143 container start c279381a76c0b9cad3ae8fc1060f16ff48ffbeccb15d933f53df963923db1eec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:191d963ac9de7322da8aff20b567b3fa458e2ef6, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mon-a, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, RELEASE=wip-batrick-testing-20240402.004512-191d963, org.label-schema.build-date=20240326, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-04-02T14:23:34.749 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: set uid:gid to 167:167 (ceph:ceph) 2024-04-02T14:23:34.749 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: ceph version 19.0.0-2609-g191d963a (191d963ac9de7322da8aff20b567b3fa458e2ef6) squid (dev), process ceph-mon, pid 2 2024-04-02T14:23:34.749 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: pidfile_write: ignore empty --pid-file 2024-04-02T14:23:34.749 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: load: jerasure load: lrc 2024-04-02T14:23:34.749 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: RocksDB version: 7.9.2 2024-04-02T14:23:34.749 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: Git sha 0 2024-04-02T14:23:34.750 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: Compile date 2024-04-02 01:02:34 2024-04-02T14:23:34.750 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: DB SUMMARY 2024-04-02T14:23:34.750 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: DB Session ID: 36B45TQZ5IVT7VXC4KO2 2024-04-02T14:23:34.750 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: CURRENT file: CURRENT 2024-04-02T14:23:34.750 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: IDENTITY file: IDENTITY 2024-04-02T14:23:34.750 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-04-02T14:23:34.750 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-04-02T14:23:34.750 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 106552 ; 2024-04-02T14:23:34.750 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: Options.error_if_exists: 0 2024-04-02T14:23:34.750 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: Options.create_if_missing: 0 2024-04-02T14:23:34.750 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: Options.paranoid_checks: 1 2024-04-02T14:23:34.750 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-02T14:23:34.750 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-02T14:23:34.750 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-02T14:23:34.750 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: Options.env: 0x55d003fbdc60 2024-04-02T14:23:34.751 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: Options.fs: PosixFileSystem 2024-04-02T14:23:34.751 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: Options.info_log: 0x55d0065a1b20 2024-04-02T14:23:34.751 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: Options.max_file_opening_threads: 16 2024-04-02T14:23:34.751 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: Options.statistics: (nil) 2024-04-02T14:23:34.751 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: Options.use_fsync: 0 2024-04-02T14:23:34.751 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: Options.max_log_file_size: 0 2024-04-02T14:23:34.752 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-02T14:23:34.752 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-02T14:23:34.752 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: Options.keep_log_file_num: 1000 2024-04-02T14:23:34.752 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: Options.recycle_log_file_num: 0 2024-04-02T14:23:34.752 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: Options.allow_fallocate: 1 2024-04-02T14:23:34.752 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: Options.allow_mmap_reads: 0 2024-04-02T14:23:34.752 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: Options.allow_mmap_writes: 0 2024-04-02T14:23:34.752 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: Options.use_direct_reads: 0 2024-04-02T14:23:34.752 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-02T14:23:34.752 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: Options.create_missing_column_families: 0 2024-04-02T14:23:34.752 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: Options.db_log_dir: 2024-04-02T14:23:34.752 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: Options.wal_dir: 2024-04-02T14:23:34.752 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-02T14:23:34.752 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-02T14:23:34.753 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-02T14:23:34.753 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-02T14:23:34.753 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-02T14:23:34.753 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-02T14:23:34.753 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: Options.advise_random_on_open: 1 2024-04-02T14:23:34.753 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: Options.db_write_buffer_size: 0 2024-04-02T14:23:34.753 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: Options.write_buffer_manager: 0x55d0065721e0 2024-04-02T14:23:34.753 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-02T14:23:34.753 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-02T14:23:34.753 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-02T14:23:34.753 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: Options.rate_limiter: (nil) 2024-04-02T14:23:34.753 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-02T14:23:34.753 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: Options.wal_recovery_mode: 2 2024-04-02T14:23:34.753 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: Options.enable_thread_tracking: 0 2024-04-02T14:23:34.753 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: Options.enable_pipelined_write: 0 2024-04-02T14:23:34.754 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: Options.unordered_write: 0 2024-04-02T14:23:34.754 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-02T14:23:34.754 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-02T14:23:34.754 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-02T14:23:34.754 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-02T14:23:34.754 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: Options.row_cache: None 2024-04-02T14:23:34.754 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: Options.wal_filter: None 2024-04-02T14:23:34.754 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-02T14:23:34.754 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: Options.allow_ingest_behind: 0 2024-04-02T14:23:34.754 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: Options.two_write_queues: 0 2024-04-02T14:23:34.754 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: Options.manual_wal_flush: 0 2024-04-02T14:23:34.754 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: Options.wal_compression: 0 2024-04-02T14:23:34.754 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: Options.atomic_flush: 0 2024-04-02T14:23:34.754 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-02T14:23:34.755 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-02T14:23:34.755 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-02T14:23:34.755 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: Options.log_readahead_size: 0 2024-04-02T14:23:34.755 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-02T14:23:34.755 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: Options.best_efforts_recovery: 0 2024-04-02T14:23:34.755 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-02T14:23:34.755 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-02T14:23:34.755 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: Options.allow_data_in_errors: 0 2024-04-02T14:23:34.755 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: Options.db_host_id: __hostname__ 2024-04-02T14:23:34.755 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-02T14:23:34.755 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: Options.max_background_jobs: 2 2024-04-02T14:23:34.755 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: Options.max_background_compactions: -1 2024-04-02T14:23:34.755 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: Options.max_subcompactions: 1 2024-04-02T14:23:34.755 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-02T14:23:34.755 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-02T14:23:34.756 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-02T14:23:34.756 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: Options.max_total_wal_size: 0 2024-04-02T14:23:34.756 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-02T14:23:34.756 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-02T14:23:34.756 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-02T14:23:34.756 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-02T14:23:34.756 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: Options.max_open_files: -1 2024-04-02T14:23:34.756 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: Options.bytes_per_sync: 0 2024-04-02T14:23:34.756 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-02T14:23:34.756 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-02T14:23:34.756 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: Options.compaction_readahead_size: 0 2024-04-02T14:23:34.756 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: Options.max_background_flushes: -1 2024-04-02T14:23:34.756 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: Compression algorithms supported: 2024-04-02T14:23:34.756 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-02T14:23:34.756 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: kZSTD supported: 0 2024-04-02T14:23:34.757 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: kXpressCompression supported: 0 2024-04-02T14:23:34.757 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: kLZ4HCCompression supported: 1 2024-04-02T14:23:34.757 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: kZlibCompression supported: 1 2024-04-02T14:23:34.757 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: kSnappyCompression supported: 1 2024-04-02T14:23:34.758 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: kLZ4Compression supported: 1 2024-04-02T14:23:34.758 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: kBZip2Compression supported: 0 2024-04-02T14:23:34.758 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-02T14:23:34.758 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-02T14:23:34.758 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-04-02T14:23:34.758 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-02T14:23:34.758 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-02T14:23:34.758 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: Options.merge_operator: 2024-04-02T14:23:34.758 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: Options.compaction_filter: None 2024-04-02T14:23:34.759 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: Options.compaction_filter_factory: None 2024-04-02T14:23:34.759 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: Options.sst_partitioner_factory: None 2024-04-02T14:23:34.759 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-02T14:23:34.759 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-02T14:23:34.759 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55d0065a1d00) 2024-04-02T14:23:34.759 INFO:journalctl@ceph.mon.a.smithi090.stdout: cache_index_and_filter_blocks: 1 2024-04-02T14:23:34.759 INFO:journalctl@ceph.mon.a.smithi090.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-02T14:23:34.759 INFO:journalctl@ceph.mon.a.smithi090.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-02T14:23:34.759 INFO:journalctl@ceph.mon.a.smithi090.stdout: pin_top_level_index_and_filter: 1 2024-04-02T14:23:34.759 INFO:journalctl@ceph.mon.a.smithi090.stdout: index_type: 0 2024-04-02T14:23:34.759 INFO:journalctl@ceph.mon.a.smithi090.stdout: data_block_index_type: 0 2024-04-02T14:23:34.759 INFO:journalctl@ceph.mon.a.smithi090.stdout: index_shortening: 1 2024-04-02T14:23:34.759 INFO:journalctl@ceph.mon.a.smithi090.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-02T14:23:34.759 INFO:journalctl@ceph.mon.a.smithi090.stdout: checksum: 4 2024-04-02T14:23:34.759 INFO:journalctl@ceph.mon.a.smithi090.stdout: no_block_cache: 0 2024-04-02T14:23:34.760 INFO:journalctl@ceph.mon.a.smithi090.stdout: block_cache: 0x55d0065a5090 2024-04-02T14:23:34.760 INFO:journalctl@ceph.mon.a.smithi090.stdout: block_cache_name: BinnedLRUCache 2024-04-02T14:23:34.760 INFO:journalctl@ceph.mon.a.smithi090.stdout: block_cache_options: 2024-04-02T14:23:34.760 INFO:journalctl@ceph.mon.a.smithi090.stdout: capacity : 536870912 2024-04-02T14:23:34.760 INFO:journalctl@ceph.mon.a.smithi090.stdout: num_shard_bits : 4 2024-04-02T14:23:34.760 INFO:journalctl@ceph.mon.a.smithi090.stdout: strict_capacity_limit : 0 2024-04-02T14:23:34.760 INFO:journalctl@ceph.mon.a.smithi090.stdout: high_pri_pool_ratio: 0.000 2024-04-02T14:23:34.760 INFO:journalctl@ceph.mon.a.smithi090.stdout: block_cache_compressed: (nil) 2024-04-02T14:23:34.760 INFO:journalctl@ceph.mon.a.smithi090.stdout: persistent_cache: (nil) 2024-04-02T14:23:34.760 INFO:journalctl@ceph.mon.a.smithi090.stdout: block_size: 4096 2024-04-02T14:23:34.760 INFO:journalctl@ceph.mon.a.smithi090.stdout: block_size_deviation: 10 2024-04-02T14:23:34.760 INFO:journalctl@ceph.mon.a.smithi090.stdout: block_restart_interval: 16 2024-04-02T14:23:34.760 INFO:journalctl@ceph.mon.a.smithi090.stdout: index_block_restart_interval: 1 2024-04-02T14:23:34.760 INFO:journalctl@ceph.mon.a.smithi090.stdout: metadata_block_size: 4096 2024-04-02T14:23:34.760 INFO:journalctl@ceph.mon.a.smithi090.stdout: partition_filters: 0 2024-04-02T14:23:34.761 INFO:journalctl@ceph.mon.a.smithi090.stdout: use_delta_encoding: 1 2024-04-02T14:23:34.761 INFO:journalctl@ceph.mon.a.smithi090.stdout: filter_policy: bloomfilter 2024-04-02T14:23:34.761 INFO:journalctl@ceph.mon.a.smithi090.stdout: whole_key_filtering: 1 2024-04-02T14:23:34.761 INFO:journalctl@ceph.mon.a.smithi090.stdout: verify_compression: 0 2024-04-02T14:23:34.761 INFO:journalctl@ceph.mon.a.smithi090.stdout: read_amp_bytes_per_bit: 0 2024-04-02T14:23:34.761 INFO:journalctl@ceph.mon.a.smithi090.stdout: format_version: 5 2024-04-02T14:23:34.761 INFO:journalctl@ceph.mon.a.smithi090.stdout: enable_index_compression: 1 2024-04-02T14:23:34.761 INFO:journalctl@ceph.mon.a.smithi090.stdout: block_align: 0 2024-04-02T14:23:34.761 INFO:journalctl@ceph.mon.a.smithi090.stdout: max_auto_readahead_size: 262144 2024-04-02T14:23:34.761 INFO:journalctl@ceph.mon.a.smithi090.stdout: prepopulate_block_cache: 0 2024-04-02T14:23:34.761 INFO:journalctl@ceph.mon.a.smithi090.stdout: initial_auto_readahead_size: 8192 2024-04-02T14:23:34.761 INFO:journalctl@ceph.mon.a.smithi090.stdout: num_file_reads_for_auto_readahead: 2 2024-04-02T14:23:34.761 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: Options.write_buffer_size: 33554432 2024-04-02T14:23:34.761 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: Options.max_write_buffer_number: 2 2024-04-02T14:23:34.761 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: Options.compression: NoCompression 2024-04-02T14:23:34.761 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: Options.bottommost_compression: Disabled 2024-04-02T14:23:34.762 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: Options.prefix_extractor: nullptr 2024-04-02T14:23:34.762 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-02T14:23:34.762 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: Options.num_levels: 7 2024-04-02T14:23:34.762 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-02T14:23:34.762 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-02T14:23:34.762 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-02T14:23:34.762 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-02T14:23:34.762 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-02T14:23:34.762 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-02T14:23:34.762 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-02T14:23:34.762 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-02T14:23:34.762 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-02T14:23:34.762 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-02T14:23:34.762 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-02T14:23:34.763 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-02T14:23:34.763 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-02T14:23:34.763 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: Options.compression_opts.level: 32767 2024-04-02T14:23:34.763 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: Options.compression_opts.strategy: 0 2024-04-02T14:23:34.763 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-02T14:23:34.763 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-02T14:23:34.763 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-02T14:23:34.763 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-02T14:23:34.763 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: Options.compression_opts.enabled: false 2024-04-02T14:23:34.763 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-02T14:23:34.763 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-02T14:23:34.763 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-02T14:23:34.763 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-02T14:23:34.763 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: Options.target_file_size_base: 67108864 2024-04-02T14:23:34.763 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-02T14:23:34.763 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-02T14:23:34.764 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-02T14:23:34.764 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-02T14:23:34.764 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-02T14:23:34.764 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-02T14:23:34.764 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-02T14:23:34.764 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-02T14:23:34.764 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-02T14:23:34.764 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-02T14:23:34.764 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-02T14:23:34.764 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-02T14:23:34.764 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-02T14:23:34.764 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-02T14:23:34.764 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: Options.arena_block_size: 1048576 2024-04-02T14:23:34.764 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-02T14:23:34.764 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-02T14:23:34.765 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: Options.disable_auto_compactions: 0 2024-04-02T14:23:34.765 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-02T14:23:34.765 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-02T14:23:34.765 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-02T14:23:34.765 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-02T14:23:34.765 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-02T14:23:34.765 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-02T14:23:34.765 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-02T14:23:34.765 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-02T14:23:34.765 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-02T14:23:34.765 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-02T14:23:34.765 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-04-02T14:23:34.765 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: Options.inplace_update_support: 0 2024-04-02T14:23:34.765 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-02T14:23:34.765 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-02T14:23:34.766 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-02T14:23:34.766 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-02T14:23:34.766 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: Options.bloom_locality: 0 2024-04-02T14:23:34.766 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: Options.max_successive_merges: 0 2024-04-02T14:23:34.766 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-02T14:23:34.766 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: Options.paranoid_file_checks: 0 2024-04-02T14:23:34.766 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: Options.force_consistency_checks: 1 2024-04-02T14:23:34.766 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: Options.report_bg_io_stats: 0 2024-04-02T14:23:34.766 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: Options.ttl: 2592000 2024-04-02T14:23:34.766 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-02T14:23:34.766 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-02T14:23:34.766 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-02T14:23:34.766 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: Options.enable_blob_files: false 2024-04-02T14:23:34.766 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: Options.min_blob_size: 0 2024-04-02T14:23:34.767 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: Options.blob_file_size: 268435456 2024-04-02T14:23:34.767 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-02T14:23:34.767 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-02T14:23:34.767 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-02T14:23:34.767 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-02T14:23:34.767 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-02T14:23:34.767 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: Options.blob_file_starting_level: 0 2024-04-02T14:23:34.767 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-02T14:23:34.767 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-04-02T14:23:34.767 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-04-02T14:23:34.767 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: e5183a73-1d62-4318-952c-228144a4d36e 2024-04-02T14:23:34.767 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: EVENT_LOG_v1 {"time_micros": 1712067814564032, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-04-02T14:23:34.767 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-04-02T14:23:34.767 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: EVENT_LOG_v1 {"time_micros": 1712067814565775, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 101586, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 332, "table_properties": {"data_size": 99462, "index_size": 316, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 773, "raw_key_size": 14542, "raw_average_key_size": 49, "raw_value_size": 91545, "raw_average_value_size": 314, "num_data_blocks": 12, "num_entries": 291, "num_filter_entries": 291, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1712067814, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "e5183a73-1d62-4318-952c-228144a4d36e", "db_session_id": "36B45TQZ5IVT7VXC4KO2", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-04-02T14:23:34.767 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: EVENT_LOG_v1 {"time_micros": 1712067814566011, "job": 1, "event": "recovery_finished"} 2024-04-02T14:23:34.767 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-04-02T14:23:34.768 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-02T14:23:34.768 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55d006694000 2024-04-02T14:23:34.768 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: DB pointer 0x55d00667e000 2024-04-02T14:23:34.768 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-02T14:23:34.768 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-02T14:23:34.768 INFO:journalctl@ceph.mon.a.smithi090.stdout: ** DB Stats ** 2024-04-02T14:23:34.768 INFO:journalctl@ceph.mon.a.smithi090.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-02T14:23:34.768 INFO:journalctl@ceph.mon.a.smithi090.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-02T14:23:34.768 INFO:journalctl@ceph.mon.a.smithi090.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-02T14:23:34.768 INFO:journalctl@ceph.mon.a.smithi090.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-02T14:23:34.768 INFO:journalctl@ceph.mon.a.smithi090.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-02T14:23:34.768 INFO:journalctl@ceph.mon.a.smithi090.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-02T14:23:34.768 INFO:journalctl@ceph.mon.a.smithi090.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-02T14:23:34.768 INFO:journalctl@ceph.mon.a.smithi090.stdout: 2024-04-02T14:23:34.769 INFO:journalctl@ceph.mon.a.smithi090.stdout: ** Compaction Stats [default] ** 2024-04-02T14:23:34.769 INFO:journalctl@ceph.mon.a.smithi090.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-02T14:23:34.769 INFO:journalctl@ceph.mon.a.smithi090.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-02T14:23:34.769 INFO:journalctl@ceph.mon.a.smithi090.stdout: L0 2/0 101.07 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 87.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-02T14:23:34.769 INFO:journalctl@ceph.mon.a.smithi090.stdout: Sum 2/0 101.07 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 87.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-02T14:23:34.769 INFO:journalctl@ceph.mon.a.smithi090.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 87.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-02T14:23:34.769 INFO:journalctl@ceph.mon.a.smithi090.stdout: 2024-04-02T14:23:34.769 INFO:journalctl@ceph.mon.a.smithi090.stdout: ** Compaction Stats [default] ** 2024-04-02T14:23:34.769 INFO:journalctl@ceph.mon.a.smithi090.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-02T14:23:34.769 INFO:journalctl@ceph.mon.a.smithi090.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-02T14:23:34.769 INFO:journalctl@ceph.mon.a.smithi090.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 87.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-02T14:23:34.769 INFO:journalctl@ceph.mon.a.smithi090.stdout: 2024-04-02T14:23:34.769 INFO:journalctl@ceph.mon.a.smithi090.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-02T14:23:34.769 INFO:journalctl@ceph.mon.a.smithi090.stdout: 2024-04-02T14:23:34.769 INFO:journalctl@ceph.mon.a.smithi090.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-02T14:23:34.770 INFO:journalctl@ceph.mon.a.smithi090.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-02T14:23:34.770 INFO:journalctl@ceph.mon.a.smithi090.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-02T14:23:34.770 INFO:journalctl@ceph.mon.a.smithi090.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-02T14:23:34.770 INFO:journalctl@ceph.mon.a.smithi090.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-02T14:23:34.770 INFO:journalctl@ceph.mon.a.smithi090.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-02T14:23:34.770 INFO:journalctl@ceph.mon.a.smithi090.stdout: Cumulative compaction: 0.00 GB write, 15.93 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-02T14:23:34.770 INFO:journalctl@ceph.mon.a.smithi090.stdout: Interval compaction: 0.00 GB write, 15.93 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-02T14:23:34.770 INFO:journalctl@ceph.mon.a.smithi090.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-02T14:23:34.770 INFO:journalctl@ceph.mon.a.smithi090.stdout: Block cache BinnedLRUCache@0x55d0065a5090#2 capacity: 512.00 MB usage: 1.52 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1e-05 secs_since: 0 2024-04-02T14:23:34.770 INFO:journalctl@ceph.mon.a.smithi090.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,1.02 KB,0.000193715%) IndexBlock(2,0.50 KB,9.53674e-05%) Misc(1,0.00 KB,0%) 2024-04-02T14:23:34.770 INFO:journalctl@ceph.mon.a.smithi090.stdout: 2024-04-02T14:23:34.770 INFO:journalctl@ceph.mon.a.smithi090.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-02T14:23:34.770 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: starting mon.a rank 0 at public addrs [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] at bind addrs [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 6449448a-f0fc-11ee-b647-cb9ed24678a4 2024-04-02T14:23:34.770 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: mon.a@-1(???) e1 preinit fsid 6449448a-f0fc-11ee-b647-cb9ed24678a4 2024-04-02T14:23:34.770 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: mon.a@-1(???).mds e1 new map 2024-04-02T14:23:34.771 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: mon.a@-1(???).mds e1 print_map 2024-04-02T14:23:34.771 INFO:journalctl@ceph.mon.a.smithi090.stdout: e1 2024-04-02T14:23:34.771 INFO:journalctl@ceph.mon.a.smithi090.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-02T14:23:34.771 INFO:journalctl@ceph.mon.a.smithi090.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-02T14:23:34.771 INFO:journalctl@ceph.mon.a.smithi090.stdout: legacy client fscid: -1 2024-04-02T14:23:34.771 INFO:journalctl@ceph.mon.a.smithi090.stdout: 2024-04-02T14:23:34.771 INFO:journalctl@ceph.mon.a.smithi090.stdout: No filesystems configured 2024-04-02T14:23:34.771 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-02T14:23:34.771 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-02T14:23:34.771 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-02T14:23:34.771 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-02T14:23:34.771 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-04-02T14:23:34.771 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 bash[31656]: c279381a76c0b9cad3ae8fc1060f16ff48ffbeccb15d933f53df963923db1eec 2024-04-02T14:23:34.771 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 systemd[1]: Started Ceph mon.a for 6449448a-f0fc-11ee-b647-cb9ed24678a4. 2024-04-02T14:23:34.771 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-02T14:23:34.772 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: monmap epoch 1 2024-04-02T14:23:34.772 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: fsid 6449448a-f0fc-11ee-b647-cb9ed24678a4 2024-04-02T14:23:34.772 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: last_changed 2024-04-02T14:23:12.545436+0000 2024-04-02T14:23:34.772 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: created 2024-04-02T14:23:12.545436+0000 2024-04-02T14:23:34.772 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: min_mon_release 19 (squid) 2024-04-02T14:23:34.772 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: election_strategy: 1 2024-04-02T14:23:34.772 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: 0: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.a 2024-04-02T14:23:34.772 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: fsmap 2024-04-02T14:23:34.772 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: osdmap e1: 0 total, 0 up, 0 in 2024-04-02T14:23:34.772 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:34 smithi090 ceph-mon[31674]: mgrmap e1: no daemons active 2024-04-02T14:23:36.057 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:35.295+0000 7fe145a00700 1 Processor -- start 2024-04-02T14:23:36.057 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:35.295+0000 7fe145a00700 1 -- start start 2024-04-02T14:23:36.058 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:35.295+0000 7fe145a00700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe14007cd80 0x7fe14007b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:23:36.058 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:35.295+0000 7fe145a00700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fe14007b7b0 con 0x7fe14007cd80 2024-04-02T14:23:36.058 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:35.296+0000 7fe13ea00700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe14007cd80 0x7fe14007b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:23:36.058 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:35.296+0000 7fe13ea00700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe14007cd80 0x7fe14007b1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41226/0 (socket says 172.21.15.90:41226) 2024-04-02T14:23:36.058 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:35.296+0000 7fe13ea00700 1 -- 172.21.15.90:0/2881643616 learned_addr learned my addr 172.21.15.90:0/2881643616 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:23:36.058 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:35.297+0000 7fe13ea00700 1 -- 172.21.15.90:0/2881643616 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe14007b930 con 0x7fe14007cd80 2024-04-02T14:23:36.058 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:35.297+0000 7fe13ea00700 1 --2- 172.21.15.90:0/2881643616 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe14007cd80 0x7fe14007b1e0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fe1300098a0 tx=0x7fe130009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97496b324d163ad3 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:23:36.058 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:35.297+0000 7fe13d600700 1 -- 172.21.15.90:0/2881643616 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe130004160 con 0x7fe14007cd80 2024-04-02T14:23:36.059 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:35.298+0000 7fe13d600700 1 -- 172.21.15.90:0/2881643616 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fe130004300 con 0x7fe14007cd80 2024-04-02T14:23:36.059 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:35.298+0000 7fe13d600700 1 -- 172.21.15.90:0/2881643616 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe130011b50 con 0x7fe14007cd80 2024-04-02T14:23:36.059 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:35.298+0000 7fe145a00700 1 -- 172.21.15.90:0/2881643616 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe14007cd80 msgr2=0x7fe14007b1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:23:36.059 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:35.298+0000 7fe145a00700 1 --2- 172.21.15.90:0/2881643616 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe14007cd80 0x7fe14007b1e0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fe1300098a0 tx=0x7fe130009bb0 comp rx=0 tx=0).stop 2024-04-02T14:23:36.059 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:35.298+0000 7fe145a00700 1 -- 172.21.15.90:0/2881643616 shutdown_connections 2024-04-02T14:23:36.059 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:35.298+0000 7fe145a00700 1 --2- 172.21.15.90:0/2881643616 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe14007cd80 0x7fe14007b1e0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:23:36.059 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:35.298+0000 7fe145a00700 1 -- 172.21.15.90:0/2881643616 >> 172.21.15.90:0/2881643616 conn(0x7fe140103810 msgr2=0x7fe140105c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:23:36.059 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:35.299+0000 7fe145a00700 1 -- 172.21.15.90:0/2881643616 shutdown_connections 2024-04-02T14:23:36.059 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:35.299+0000 7fe145a00700 1 -- 172.21.15.90:0/2881643616 wait complete. 2024-04-02T14:23:36.059 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:35.300+0000 7fe145a00700 1 Processor -- start 2024-04-02T14:23:36.060 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:35.300+0000 7fe145a00700 1 -- start start 2024-04-02T14:23:36.060 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:35.300+0000 7fe145a00700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe14007cd80 0x7fe14019e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:23:36.060 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:35.300+0000 7fe145a00700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fe14010e860 con 0x7fe14007cd80 2024-04-02T14:23:36.060 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:35.300+0000 7fe13ea00700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe14007cd80 0x7fe14019e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:23:36.060 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:35.300+0000 7fe13ea00700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe14007cd80 0x7fe14019e3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41230/0 (socket says 172.21.15.90:41230) 2024-04-02T14:23:36.060 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:35.300+0000 7fe13ea00700 1 -- 172.21.15.90:0/1304128311 learned_addr learned my addr 172.21.15.90:0/1304128311 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:23:36.060 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:35.301+0000 7fe13ea00700 1 -- 172.21.15.90:0/1304128311 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe14019e910 con 0x7fe14007cd80 2024-04-02T14:23:36.060 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:35.301+0000 7fe13ea00700 1 --2- 172.21.15.90:0/1304128311 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe14007cd80 0x7fe14019e3d0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fe130011b60 tx=0x7fe130011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:23:36.060 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:35.302+0000 7fe137400700 1 -- 172.21.15.90:0/1304128311 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe130003b20 con 0x7fe14007cd80 2024-04-02T14:23:36.061 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:35.302+0000 7fe137400700 1 -- 172.21.15.90:0/1304128311 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fe130003cc0 con 0x7fe14007cd80 2024-04-02T14:23:36.061 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:35.302+0000 7fe145a00700 1 -- 172.21.15.90:0/1304128311 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe14019eba0 con 0x7fe14007cd80 2024-04-02T14:23:36.061 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:35.302+0000 7fe137400700 1 -- 172.21.15.90:0/1304128311 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe130024840 con 0x7fe14007cd80 2024-04-02T14:23:36.061 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:35.302+0000 7fe145a00700 1 -- 172.21.15.90:0/1304128311 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1401a1890 con 0x7fe14007cd80 2024-04-02T14:23:36.061 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:35.302+0000 7fe137400700 1 -- 172.21.15.90:0/1304128311 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fe130022020 con 0x7fe14007cd80 2024-04-02T14:23:36.061 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:35.302+0000 7fe137400700 1 -- 172.21.15.90:0/1304128311 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fe130011e00 con 0x7fe14007cd80 2024-04-02T14:23:36.061 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:35.303+0000 7fe145a00700 1 -- 172.21.15.90:0/1304128311 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe140063c20 con 0x7fe14007cd80 2024-04-02T14:23:36.061 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:35.306+0000 7fe137400700 1 -- 172.21.15.90:0/1304128311 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fe130043dd0 con 0x7fe14007cd80 2024-04-02T14:23:36.061 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:35.382+0000 7fe145a00700 1 -- 172.21.15.90:0/1304128311 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) -- 0x7fe1401a1a60 con 0x7fe14007cd80 2024-04-02T14:23:36.061 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:35.385+0000 7fe137400700 1 -- 172.21.15.90:0/1304128311 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) ==== 124+0+0 (secure 0 0 0) 0x7fe130032020 con 0x7fe14007cd80 2024-04-02T14:23:36.061 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:35.386+0000 7fe145a00700 1 -- 172.21.15.90:0/1304128311 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe14007cd80 msgr2=0x7fe14019e3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:23:36.061 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:35.386+0000 7fe145a00700 1 --2- 172.21.15.90:0/1304128311 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe14007cd80 0x7fe14019e3d0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fe130011b60 tx=0x7fe130011c40 comp rx=0 tx=0).stop 2024-04-02T14:23:36.062 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:35.386+0000 7fe145a00700 1 -- 172.21.15.90:0/1304128311 shutdown_connections 2024-04-02T14:23:36.062 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:35.386+0000 7fe145a00700 1 --2- 172.21.15.90:0/1304128311 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe14007cd80 0x7fe14019e3d0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:23:36.062 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:35.386+0000 7fe145a00700 1 -- 172.21.15.90:0/1304128311 >> 172.21.15.90:0/1304128311 conn(0x7fe140103810 msgr2=0x7fe140104370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:23:36.062 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:35.386+0000 7fe145a00700 1 -- 172.21.15.90:0/1304128311 shutdown_connections 2024-04-02T14:23:36.062 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:35.387+0000 7fe145a00700 1 -- 172.21.15.90:0/1304128311 wait complete. 2024-04-02T14:23:36.062 INFO:teuthology.orchestra.run.smithi090.stdout:Wrote config to /etc/ceph/ceph.conf 2024-04-02T14:23:36.078 INFO:teuthology.orchestra.run.smithi090.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-04-02T14:23:36.079 INFO:teuthology.orchestra.run.smithi090.stdout:Creating mgr... 2024-04-02T14:23:36.080 INFO:teuthology.orchestra.run.smithi090.stdout:Verifying port 0.0.0.0:9283 ... 2024-04-02T14:23:36.080 INFO:teuthology.orchestra.run.smithi090.stdout:Verifying port 0.0.0.0:8765 ... 2024-04-02T14:23:36.501 INFO:teuthology.orchestra.run.smithi090.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4@mgr.x 2024-04-02T14:23:36.501 INFO:teuthology.orchestra.run.smithi090.stdout:systemctl: stderr Failed to reset failed state of unit ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4@mgr.x.service: Unit ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4@mgr.x.service not loaded. 2024-04-02T14:23:36.688 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:36 smithi090 ceph-mon[31674]: from='client.? 172.21.15.90:0/1304128311' entity='client.admin' 2024-04-02T14:23:36.746 INFO:teuthology.orchestra.run.smithi090.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4.target.wants/ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4@mgr.x.service → /etc/systemd/system/ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4@.service. 2024-04-02T14:23:37.133 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 02 14:23:36 smithi090 systemd[1]: Starting Ceph mgr.x for 6449448a-f0fc-11ee-b647-cb9ed24678a4... 2024-04-02T14:23:37.436 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 02 14:23:37 smithi090 podman[31895]: 2024-04-02 14:23:37.131176314 +0000 UTC m=+0.124595938 container create 3c48d7f60a2ef43864ecbcab9b9a093ebc25d4c6aeca6d8313f5c0a7d172988d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:191d963ac9de7322da8aff20b567b3fa458e2ef6, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-x, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240402.004512-191d963, org.label-schema.build-date=20240326) 2024-04-02T14:23:37.436 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 02 14:23:37 smithi090 podman[31895]: 2024-04-02 14:23:37.03898645 +0000 UTC m=+0.032406071 image pull f7301e070545fc23d8e5b7b052a2c15e832a0e57102f1acad8e4f3583709d23f quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:191d963ac9de7322da8aff20b567b3fa458e2ef6 2024-04-02T14:23:37.437 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 02 14:23:37 smithi090 podman[31895]: 2024-04-02 14:23:37.352888257 +0000 UTC m=+0.346307904 container init 3c48d7f60a2ef43864ecbcab9b9a093ebc25d4c6aeca6d8313f5c0a7d172988d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:191d963ac9de7322da8aff20b567b3fa458e2ef6, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-x, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240402.004512-191d963, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240326) 2024-04-02T14:23:37.437 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 02 14:23:37 smithi090 podman[31895]: 2024-04-02 14:23:37.36141832 +0000 UTC m=+0.354837953 container start 3c48d7f60a2ef43864ecbcab9b9a093ebc25d4c6aeca6d8313f5c0a7d172988d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:191d963ac9de7322da8aff20b567b3fa458e2ef6, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-x, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240402.004512-191d963, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240326, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-04-02T14:23:37.437 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 02 14:23:37 smithi090 bash[31895]: 3c48d7f60a2ef43864ecbcab9b9a093ebc25d4c6aeca6d8313f5c0a7d172988d 2024-04-02T14:23:37.467 INFO:teuthology.orchestra.run.smithi090.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-02T14:23:37.467 INFO:teuthology.orchestra.run.smithi090.stdout:systemctl: stdout disabled 2024-04-02T14:23:37.483 INFO:teuthology.orchestra.run.smithi090.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-02T14:23:37.484 INFO:teuthology.orchestra.run.smithi090.stdout:systemctl: stdout inactive 2024-04-02T14:23:37.484 INFO:teuthology.orchestra.run.smithi090.stdout:firewalld.service is not enabled 2024-04-02T14:23:37.484 INFO:teuthology.orchestra.run.smithi090.stdout:Not possible to enable service . firewalld.service is not available 2024-04-02T14:23:37.504 INFO:teuthology.orchestra.run.smithi090.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-02T14:23:37.504 INFO:teuthology.orchestra.run.smithi090.stdout:systemctl: stdout disabled 2024-04-02T14:23:37.522 INFO:teuthology.orchestra.run.smithi090.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-02T14:23:37.522 INFO:teuthology.orchestra.run.smithi090.stdout:systemctl: stdout inactive 2024-04-02T14:23:37.522 INFO:teuthology.orchestra.run.smithi090.stdout:firewalld.service is not enabled 2024-04-02T14:23:37.522 INFO:teuthology.orchestra.run.smithi090.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-04-02T14:23:37.523 INFO:teuthology.orchestra.run.smithi090.stdout:Waiting for mgr to start... 2024-04-02T14:23:37.523 INFO:teuthology.orchestra.run.smithi090.stdout:Waiting for mgr... 2024-04-02T14:23:37.689 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 02 14:23:37 smithi090 systemd[1]: Started Ceph mgr.x for 6449448a-f0fc-11ee-b647-cb9ed24678a4. 2024-04-02T14:23:38.438 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:38 smithi090 ceph-mon[31674]: from='client.? 172.21.15.90:0/3852288182' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-02T14:23:38.439 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 02 14:23:38 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-x[31910]: 2024-04-02T14:23:38.154+0000 7fb9fd42b200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-02T14:23:38.439 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 02 14:23:38 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-x[31910]: 2024-04-02T14:23:38.280+0000 7fb9fd42b200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-02T14:23:38.877 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout 2024-04-02T14:23:38.877 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout { 2024-04-02T14:23:38.877 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "fsid": "6449448a-f0fc-11ee-b647-cb9ed24678a4", 2024-04-02T14:23:38.877 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "health": { 2024-04-02T14:23:38.877 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-02T14:23:38.877 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-02T14:23:38.877 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-02T14:23:38.877 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout }, 2024-04-02T14:23:38.877 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-02T14:23:38.877 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-02T14:23:38.877 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout 0 2024-04-02T14:23:38.878 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout ], 2024-04-02T14:23:38.878 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-02T14:23:38.878 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "a" 2024-04-02T14:23:38.878 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout ], 2024-04-02T14:23:38.878 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "quorum_age": 3, 2024-04-02T14:23:38.878 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-02T14:23:38.878 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-02T14:23:38.878 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-02T14:23:38.878 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-02T14:23:38.878 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout }, 2024-04-02T14:23:38.878 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-02T14:23:38.878 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-02T14:23:38.878 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-02T14:23:38.879 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-02T14:23:38.879 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-02T14:23:38.879 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-02T14:23:38.879 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-02T14:23:38.879 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-02T14:23:38.879 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout }, 2024-04-02T14:23:38.879 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-02T14:23:38.879 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-02T14:23:38.879 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-02T14:23:38.879 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-02T14:23:38.879 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-02T14:23:38.879 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-02T14:23:38.879 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-02T14:23:38.879 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-02T14:23:38.879 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-02T14:23:38.880 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout }, 2024-04-02T14:23:38.880 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-02T14:23:38.880 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-02T14:23:38.880 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-02T14:23:38.880 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-02T14:23:38.880 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout }, 2024-04-02T14:23:38.880 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-02T14:23:38.880 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-02T14:23:38.880 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-02T14:23:38.880 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-02T14:23:38.880 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-02T14:23:38.880 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-02T14:23:38.880 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "restful" 2024-04-02T14:23:38.880 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout ], 2024-04-02T14:23:38.880 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-02T14:23:38.880 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout }, 2024-04-02T14:23:38.881 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-02T14:23:38.881 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-02T14:23:38.881 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "modified": "2024-04-02T14:23:16.530776+0000", 2024-04-02T14:23:38.881 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-02T14:23:38.881 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout }, 2024-04-02T14:23:38.881 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-02T14:23:38.881 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout } 2024-04-02T14:23:38.881 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:38.085+0000 7f4bc0800700 1 Processor -- start 2024-04-02T14:23:38.881 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:38.085+0000 7f4bc0800700 1 -- start start 2024-04-02T14:23:38.881 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:38.085+0000 7f4bc0800700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4bb807cda0 0x7f4bb807b200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:23:38.881 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:38.085+0000 7f4bc0800700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f4bb807b7d0 con 0x7f4bb807cda0 2024-04-02T14:23:38.881 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:38.086+0000 7f4bbde00700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4bb807cda0 0x7f4bb807b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:23:38.881 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:38.086+0000 7f4bbde00700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4bb807cda0 0x7f4bb807b200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41256/0 (socket says 172.21.15.90:41256) 2024-04-02T14:23:38.881 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:38.086+0000 7f4bbde00700 1 -- 172.21.15.90:0/1296386680 learned_addr learned my addr 172.21.15.90:0/1296386680 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:23:38.882 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:38.087+0000 7f4bbde00700 1 -- 172.21.15.90:0/1296386680 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4bb807b950 con 0x7f4bb807cda0 2024-04-02T14:23:38.882 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:38.087+0000 7f4bbde00700 1 --2- 172.21.15.90:0/1296386680 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4bb807cda0 0x7f4bb807b200 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f4bac0098a0 tx=0x7f4bac009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88ffad426f947b7 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:23:38.882 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:38.087+0000 7f4bbca00700 1 -- 172.21.15.90:0/1296386680 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4bac004160 con 0x7f4bb807cda0 2024-04-02T14:23:38.882 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:38.087+0000 7f4bbca00700 1 -- 172.21.15.90:0/1296386680 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f4bac004300 con 0x7f4bb807cda0 2024-04-02T14:23:38.882 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:38.087+0000 7f4bbca00700 1 -- 172.21.15.90:0/1296386680 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4bac011b50 con 0x7f4bb807cda0 2024-04-02T14:23:38.882 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:38.088+0000 7f4bc0800700 1 -- 172.21.15.90:0/1296386680 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4bb807cda0 msgr2=0x7f4bb807b200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:23:38.882 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:38.088+0000 7f4bc0800700 1 --2- 172.21.15.90:0/1296386680 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4bb807cda0 0x7f4bb807b200 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f4bac0098a0 tx=0x7f4bac009bb0 comp rx=0 tx=0).stop 2024-04-02T14:23:38.882 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:38.088+0000 7f4bc0800700 1 -- 172.21.15.90:0/1296386680 shutdown_connections 2024-04-02T14:23:38.882 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:38.088+0000 7f4bc0800700 1 --2- 172.21.15.90:0/1296386680 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4bb807cda0 0x7f4bb807b200 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:23:38.882 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:38.088+0000 7f4bc0800700 1 -- 172.21.15.90:0/1296386680 >> 172.21.15.90:0/1296386680 conn(0x7f4bb8103830 msgr2=0x7f4bb8105c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:23:38.882 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:38.088+0000 7f4bc0800700 1 -- 172.21.15.90:0/1296386680 shutdown_connections 2024-04-02T14:23:38.882 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:38.088+0000 7f4bc0800700 1 -- 172.21.15.90:0/1296386680 wait complete. 2024-04-02T14:23:38.883 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:38.090+0000 7f4bc0800700 1 Processor -- start 2024-04-02T14:23:38.883 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:38.090+0000 7f4bc0800700 1 -- start start 2024-04-02T14:23:38.883 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:38.090+0000 7f4bc0800700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4bb807cda0 0x7f4bb819e440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:23:38.883 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:38.090+0000 7f4bc0800700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f4bb810e880 con 0x7f4bb807cda0 2024-04-02T14:23:38.883 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:38.090+0000 7f4bbde00700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4bb807cda0 0x7f4bb819e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:23:38.883 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:38.090+0000 7f4bbde00700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4bb807cda0 0x7f4bb819e440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41260/0 (socket says 172.21.15.90:41260) 2024-04-02T14:23:38.883 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:38.090+0000 7f4bbde00700 1 -- 172.21.15.90:0/3852288182 learned_addr learned my addr 172.21.15.90:0/3852288182 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:23:38.883 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:38.091+0000 7f4bbde00700 1 -- 172.21.15.90:0/3852288182 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4bb819e980 con 0x7f4bb807cda0 2024-04-02T14:23:38.883 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:38.091+0000 7f4bbde00700 1 --2- 172.21.15.90:0/3852288182 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4bb807cda0 0x7f4bb819e440 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f4bac011670 tx=0x7f4bac011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:23:38.883 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:38.091+0000 7f4bb2a00700 1 -- 172.21.15.90:0/3852288182 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4bac0039c0 con 0x7f4bb807cda0 2024-04-02T14:23:38.883 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:38.091+0000 7f4bb2a00700 1 -- 172.21.15.90:0/3852288182 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f4bac011e50 con 0x7f4bb807cda0 2024-04-02T14:23:38.883 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:38.091+0000 7f4bb2a00700 1 -- 172.21.15.90:0/3852288182 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4bac01a5c0 con 0x7f4bb807cda0 2024-04-02T14:23:38.883 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:38.091+0000 7f4bc0800700 1 -- 172.21.15.90:0/3852288182 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4bb819ec10 con 0x7f4bb807cda0 2024-04-02T14:23:38.883 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:38.091+0000 7f4bc0800700 1 -- 172.21.15.90:0/3852288182 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4bb81a1900 con 0x7f4bb807cda0 2024-04-02T14:23:38.884 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:38.092+0000 7f4bb2a00700 1 -- 172.21.15.90:0/3852288182 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f4bac01aaf0 con 0x7f4bb807cda0 2024-04-02T14:23:38.884 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:38.092+0000 7f4bb2a00700 1 -- 172.21.15.90:0/3852288182 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f4bac019700 con 0x7f4bb807cda0 2024-04-02T14:23:38.884 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:38.092+0000 7f4bc0800700 1 -- 172.21.15.90:0/3852288182 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4bb8063c20 con 0x7f4bb807cda0 2024-04-02T14:23:38.884 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:38.095+0000 7f4bb2a00700 1 -- 172.21.15.90:0/3852288182 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f4bac003b60 con 0x7f4bb807cda0 2024-04-02T14:23:38.884 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:38.175+0000 7f4bc0800700 1 -- 172.21.15.90:0/3852288182 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f4bb81a1ec0 con 0x7f4bb807cda0 2024-04-02T14:23:38.884 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:38.176+0000 7f4bb2a00700 1 -- 172.21.15.90:0/3852288182 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f4bac032030 con 0x7f4bb807cda0 2024-04-02T14:23:38.884 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:38.177+0000 7f4bc0800700 1 -- 172.21.15.90:0/3852288182 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4bb807cda0 msgr2=0x7f4bb819e440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:23:38.884 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:38.177+0000 7f4bc0800700 1 --2- 172.21.15.90:0/3852288182 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4bb807cda0 0x7f4bb819e440 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f4bac011670 tx=0x7f4bac011b60 comp rx=0 tx=0).stop 2024-04-02T14:23:38.884 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:38.177+0000 7f4bc0800700 1 -- 172.21.15.90:0/3852288182 shutdown_connections 2024-04-02T14:23:38.884 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:38.177+0000 7f4bc0800700 1 --2- 172.21.15.90:0/3852288182 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4bb807cda0 0x7f4bb819e440 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:23:38.884 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:38.177+0000 7f4bc0800700 1 -- 172.21.15.90:0/3852288182 >> 172.21.15.90:0/3852288182 conn(0x7f4bb8103830 msgr2=0x7f4bb8104390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:23:38.884 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:38.177+0000 7f4bc0800700 1 -- 172.21.15.90:0/3852288182 shutdown_connections 2024-04-02T14:23:38.884 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:38.177+0000 7f4bc0800700 1 -- 172.21.15.90:0/3852288182 wait complete. 2024-04-02T14:23:38.884 INFO:teuthology.orchestra.run.smithi090.stdout:mgr not available, waiting (1/15)... 2024-04-02T14:23:39.188 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 02 14:23:39 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-x[31910]: 2024-04-02T14:23:39.095+0000 7fb9fd42b200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-02T14:23:39.512 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 02 14:23:39 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-x[31910]: 2024-04-02T14:23:39.214+0000 7fb9fd42b200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-02T14:23:39.513 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 02 14:23:39 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-x[31910]: 2024-04-02T14:23:39.388+0000 7fb9fd42b200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-02T14:23:39.513 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 02 14:23:39 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-x[31910]: 2024-04-02T14:23:39.511+0000 7fb9fd42b200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-02T14:23:39.939 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 02 14:23:39 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-x[31910]: 2024-04-02T14:23:39.619+0000 7fb9fd42b200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-02T14:23:40.321 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 02 14:23:39 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-x[31910]: 2024-04-02T14:23:39.947+0000 7fb9fd42b200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-02T14:23:40.321 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 02 14:23:40 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-x[31910]: 2024-04-02T14:23:40.086+0000 7fb9fd42b200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-02T14:23:40.321 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 02 14:23:40 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-x[31910]: 2024-04-02T14:23:40.197+0000 7fb9fd42b200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-02T14:23:40.689 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 02 14:23:40 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-x[31910]: 2024-04-02T14:23:40.319+0000 7fb9fd42b200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-02T14:23:40.689 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 02 14:23:40 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-x[31910]: 2024-04-02T14:23:40.532+0000 7fb9fd42b200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-02T14:23:41.688 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:41 smithi090 ceph-mon[31674]: from='client.? 172.21.15.90:0/3407915758' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-02T14:23:42.189 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 02 14:23:41 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-x[31910]: 2024-04-02T14:23:41.859+0000 7fb9fd42b200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-02T14:23:42.189 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 02 14:23:42 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-x[31910]: 2024-04-02T14:23:42.033+0000 7fb9fd42b200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-02T14:23:42.220 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout 2024-04-02T14:23:42.220 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout { 2024-04-02T14:23:42.220 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "fsid": "6449448a-f0fc-11ee-b647-cb9ed24678a4", 2024-04-02T14:23:42.220 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "health": { 2024-04-02T14:23:42.220 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-02T14:23:42.220 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-02T14:23:42.220 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-02T14:23:42.220 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout }, 2024-04-02T14:23:42.220 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-02T14:23:42.221 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-02T14:23:42.221 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout 0 2024-04-02T14:23:42.221 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout ], 2024-04-02T14:23:42.221 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-02T14:23:42.221 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "a" 2024-04-02T14:23:42.221 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout ], 2024-04-02T14:23:42.221 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "quorum_age": 6, 2024-04-02T14:23:42.221 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-02T14:23:42.221 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-02T14:23:42.221 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-02T14:23:42.221 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-02T14:23:42.221 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout }, 2024-04-02T14:23:42.221 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-02T14:23:42.222 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-02T14:23:42.222 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-02T14:23:42.222 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-02T14:23:42.222 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-02T14:23:42.222 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-02T14:23:42.222 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-02T14:23:42.222 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-02T14:23:42.222 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout }, 2024-04-02T14:23:42.222 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-02T14:23:42.222 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-02T14:23:42.222 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-02T14:23:42.222 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-02T14:23:42.223 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-02T14:23:42.223 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-02T14:23:42.223 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-02T14:23:42.223 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-02T14:23:42.223 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-02T14:23:42.223 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout }, 2024-04-02T14:23:42.223 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-02T14:23:42.223 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-02T14:23:42.223 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-02T14:23:42.223 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-02T14:23:42.223 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout }, 2024-04-02T14:23:42.223 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-02T14:23:42.224 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-02T14:23:42.224 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-02T14:23:42.224 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-02T14:23:42.224 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-02T14:23:42.224 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-02T14:23:42.224 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "restful" 2024-04-02T14:23:42.224 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout ], 2024-04-02T14:23:42.224 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-02T14:23:42.224 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout }, 2024-04-02T14:23:42.224 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-02T14:23:42.224 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-02T14:23:42.224 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "modified": "2024-04-02T14:23:16.530776+0000", 2024-04-02T14:23:42.224 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-02T14:23:42.225 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout }, 2024-04-02T14:23:42.225 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-02T14:23:42.225 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout } 2024-04-02T14:23:42.225 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:41.366+0000 7f085ac00700 1 Processor -- start 2024-04-02T14:23:42.225 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:41.366+0000 7f085ac00700 1 -- start start 2024-04-02T14:23:42.225 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:41.366+0000 7f085ac00700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0854108ce0 0x7f085410b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:23:42.225 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:41.366+0000 7f085ac00700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f0854075ef0 con 0x7f0854108ce0 2024-04-02T14:23:42.225 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:41.367+0000 7f0853e00700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0854108ce0 0x7f085410b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:23:42.225 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:41.367+0000 7f0853e00700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0854108ce0 0x7f085410b0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41274/0 (socket says 172.21.15.90:41274) 2024-04-02T14:23:42.225 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:41.367+0000 7f0853e00700 1 -- 172.21.15.90:0/590442129 learned_addr learned my addr 172.21.15.90:0/590442129 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:23:42.225 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:41.367+0000 7f0853e00700 1 -- 172.21.15.90:0/590442129 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f085410b610 con 0x7f0854108ce0 2024-04-02T14:23:42.225 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:41.367+0000 7f0853e00700 1 --2- 172.21.15.90:0/590442129 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0854108ce0 0x7f085410b0d0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f08440098a0 tx=0x7f0844009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7d0433a349d2826 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:23:42.225 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:41.368+0000 7f0852a00700 1 -- 172.21.15.90:0/590442129 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0844004160 con 0x7f0854108ce0 2024-04-02T14:23:42.226 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:41.368+0000 7f0852a00700 1 -- 172.21.15.90:0/590442129 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f0844004300 con 0x7f0854108ce0 2024-04-02T14:23:42.226 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:41.368+0000 7f0852a00700 1 -- 172.21.15.90:0/590442129 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0844011b50 con 0x7f0854108ce0 2024-04-02T14:23:42.226 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:41.368+0000 7f085ac00700 1 -- 172.21.15.90:0/590442129 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0854108ce0 msgr2=0x7f085410b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:23:42.226 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:41.368+0000 7f085ac00700 1 --2- 172.21.15.90:0/590442129 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0854108ce0 0x7f085410b0d0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f08440098a0 tx=0x7f0844009bb0 comp rx=0 tx=0).stop 2024-04-02T14:23:42.226 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:41.369+0000 7f085ac00700 1 -- 172.21.15.90:0/590442129 shutdown_connections 2024-04-02T14:23:42.226 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:41.369+0000 7f085ac00700 1 --2- 172.21.15.90:0/590442129 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0854108ce0 0x7f085410b0d0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:23:42.226 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:41.369+0000 7f085ac00700 1 -- 172.21.15.90:0/590442129 >> 172.21.15.90:0/590442129 conn(0x7f0854102730 msgr2=0x7f0854104b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:23:42.226 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:41.369+0000 7f085ac00700 1 -- 172.21.15.90:0/590442129 shutdown_connections 2024-04-02T14:23:42.226 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:41.369+0000 7f085ac00700 1 -- 172.21.15.90:0/590442129 wait complete. 2024-04-02T14:23:42.226 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:41.370+0000 7f085ac00700 1 Processor -- start 2024-04-02T14:23:42.226 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:41.370+0000 7f085ac00700 1 -- start start 2024-04-02T14:23:42.226 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:41.370+0000 7f085ac00700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0854108ce0 0x7f0854108560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:23:42.227 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:41.370+0000 7f085ac00700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f085410c640 con 0x7f0854108ce0 2024-04-02T14:23:42.227 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:41.370+0000 7f0853e00700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0854108ce0 0x7f0854108560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:23:42.227 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:41.370+0000 7f0853e00700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0854108ce0 0x7f0854108560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41280/0 (socket says 172.21.15.90:41280) 2024-04-02T14:23:42.227 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:41.370+0000 7f0853e00700 1 -- 172.21.15.90:0/3407915758 learned_addr learned my addr 172.21.15.90:0/3407915758 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:23:42.227 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:41.371+0000 7f0853e00700 1 -- 172.21.15.90:0/3407915758 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0854108aa0 con 0x7f0854108ce0 2024-04-02T14:23:42.227 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:41.371+0000 7f0853e00700 1 --2- 172.21.15.90:0/3407915758 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0854108ce0 0x7f0854108560 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f0844011b60 tx=0x7f0844011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:23:42.227 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:41.371+0000 7f0850c00700 1 -- 172.21.15.90:0/3407915758 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0844003970 con 0x7f0854108ce0 2024-04-02T14:23:42.227 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:41.371+0000 7f0850c00700 1 -- 172.21.15.90:0/3407915758 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f0844003b10 con 0x7f0854108ce0 2024-04-02T14:23:42.227 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:41.371+0000 7f085ac00700 1 -- 172.21.15.90:0/3407915758 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0854106c90 con 0x7f0854108ce0 2024-04-02T14:23:42.227 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:41.371+0000 7f0850c00700 1 -- 172.21.15.90:0/3407915758 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0844024610 con 0x7f0854108ce0 2024-04-02T14:23:42.227 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:41.372+0000 7f085ac00700 1 -- 172.21.15.90:0/3407915758 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f08541071d0 con 0x7f0854108ce0 2024-04-02T14:23:42.227 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:41.372+0000 7f0850c00700 1 -- 172.21.15.90:0/3407915758 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f0844024c50 con 0x7f0854108ce0 2024-04-02T14:23:42.228 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:41.372+0000 7f0850c00700 1 -- 172.21.15.90:0/3407915758 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f0844011e00 con 0x7f0854108ce0 2024-04-02T14:23:42.228 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:41.373+0000 7f085ac00700 1 -- 172.21.15.90:0/3407915758 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0854063c20 con 0x7f0854108ce0 2024-04-02T14:23:42.228 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:41.375+0000 7f0850c00700 1 -- 172.21.15.90:0/3407915758 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f0844043dd0 con 0x7f0854108ce0 2024-04-02T14:23:42.228 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:41.453+0000 7f085ac00700 1 -- 172.21.15.90:0/3407915758 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f08541076e0 con 0x7f0854108ce0 2024-04-02T14:23:42.228 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:41.454+0000 7f0850c00700 1 -- 172.21.15.90:0/3407915758 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f0844032030 con 0x7f0854108ce0 2024-04-02T14:23:42.228 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:41.455+0000 7f085ac00700 1 -- 172.21.15.90:0/3407915758 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0854108ce0 msgr2=0x7f0854108560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:23:42.228 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:41.455+0000 7f085ac00700 1 --2- 172.21.15.90:0/3407915758 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0854108ce0 0x7f0854108560 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f0844011b60 tx=0x7f0844011c40 comp rx=0 tx=0).stop 2024-04-02T14:23:42.228 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:41.456+0000 7f085ac00700 1 -- 172.21.15.90:0/3407915758 shutdown_connections 2024-04-02T14:23:42.228 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:41.456+0000 7f085ac00700 1 --2- 172.21.15.90:0/3407915758 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0854108ce0 0x7f0854108560 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:23:42.228 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:41.456+0000 7f085ac00700 1 -- 172.21.15.90:0/3407915758 >> 172.21.15.90:0/3407915758 conn(0x7f0854102730 msgr2=0x7f0854103290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:23:42.228 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:41.456+0000 7f085ac00700 1 -- 172.21.15.90:0/3407915758 shutdown_connections 2024-04-02T14:23:42.228 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:41.456+0000 7f085ac00700 1 -- 172.21.15.90:0/3407915758 wait complete. 2024-04-02T14:23:42.228 INFO:teuthology.orchestra.run.smithi090.stdout:mgr not available, waiting (2/15)... 2024-04-02T14:23:42.817 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 02 14:23:42 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-x[31910]: 2024-04-02T14:23:42.446+0000 7fb9fd42b200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-02T14:23:42.817 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 02 14:23:42 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-x[31910]: 2024-04-02T14:23:42.563+0000 7fb9fd42b200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-02T14:23:42.817 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 02 14:23:42 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-x[31910]: 2024-04-02T14:23:42.815+0000 7fb9fd42b200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-02T14:23:43.188 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 02 14:23:43 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-x[31910]: 2024-04-02T14:23:43.056+0000 7fb9fd42b200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-02T14:23:43.688 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 02 14:23:43 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-x[31910]: 2024-04-02T14:23:43.425+0000 7fb9fd42b200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-02T14:23:44.042 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 02 14:23:43 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-x[31910]: 2024-04-02T14:23:43.736+0000 7fb9fd42b200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-02T14:23:44.043 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 02 14:23:44 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-x[31910]: 2024-04-02T14:23:44.040+0000 7fb9fd42b200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-02T14:23:45.102 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:44 smithi090 ceph-mon[31674]: from='client.? 172.21.15.90:0/2035259193' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-02T14:23:45.383 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 02 14:23:45 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-x[31910]: 2024-04-02T14:23:45.100+0000 7fb9fd42b200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-02T14:23:45.561 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout 2024-04-02T14:23:45.561 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout { 2024-04-02T14:23:45.562 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "fsid": "6449448a-f0fc-11ee-b647-cb9ed24678a4", 2024-04-02T14:23:45.562 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "health": { 2024-04-02T14:23:45.562 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-02T14:23:45.562 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-02T14:23:45.562 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-02T14:23:45.562 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout }, 2024-04-02T14:23:45.562 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-02T14:23:45.562 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-02T14:23:45.562 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout 0 2024-04-02T14:23:45.562 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout ], 2024-04-02T14:23:45.562 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-02T14:23:45.562 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "a" 2024-04-02T14:23:45.562 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout ], 2024-04-02T14:23:45.563 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "quorum_age": 10, 2024-04-02T14:23:45.563 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-02T14:23:45.563 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-02T14:23:45.563 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-02T14:23:45.563 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-02T14:23:45.563 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout }, 2024-04-02T14:23:45.563 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-02T14:23:45.563 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-02T14:23:45.563 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-02T14:23:45.563 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-02T14:23:45.563 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-02T14:23:45.563 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-02T14:23:45.563 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-02T14:23:45.563 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-02T14:23:45.563 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout }, 2024-04-02T14:23:45.564 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-02T14:23:45.564 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-02T14:23:45.564 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-02T14:23:45.564 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-02T14:23:45.564 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-02T14:23:45.564 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-02T14:23:45.564 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-02T14:23:45.564 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-02T14:23:45.564 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-02T14:23:45.571 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout }, 2024-04-02T14:23:45.571 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-02T14:23:45.571 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-02T14:23:45.571 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-02T14:23:45.572 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-02T14:23:45.572 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout }, 2024-04-02T14:23:45.572 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-02T14:23:45.572 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-02T14:23:45.572 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-02T14:23:45.572 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-02T14:23:45.572 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-02T14:23:45.572 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-02T14:23:45.572 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "restful" 2024-04-02T14:23:45.572 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout ], 2024-04-02T14:23:45.572 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-02T14:23:45.573 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout }, 2024-04-02T14:23:45.573 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-02T14:23:45.573 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-02T14:23:45.573 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "modified": "2024-04-02T14:23:16.530776+0000", 2024-04-02T14:23:45.573 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-02T14:23:45.573 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout }, 2024-04-02T14:23:45.573 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-02T14:23:45.573 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout } 2024-04-02T14:23:45.573 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:44.757+0000 7f6cc1c00700 1 Processor -- start 2024-04-02T14:23:45.573 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:44.757+0000 7f6cc1c00700 1 -- start start 2024-04-02T14:23:45.573 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:44.758+0000 7f6cc1c00700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6cbc07cd80 0x7f6cbc07b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:23:45.573 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:44.758+0000 7f6cc1c00700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f6cbc07b7b0 con 0x7f6cbc07cd80 2024-04-02T14:23:45.573 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:44.758+0000 7f6cbaa00700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6cbc07cd80 0x7f6cbc07b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:23:45.573 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:44.758+0000 7f6cbaa00700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6cbc07cd80 0x7f6cbc07b1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:56520/0 (socket says 172.21.15.90:56520) 2024-04-02T14:23:45.574 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:44.758+0000 7f6cbaa00700 1 -- 172.21.15.90:0/3752224275 learned_addr learned my addr 172.21.15.90:0/3752224275 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:23:45.574 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:44.759+0000 7f6cbaa00700 1 -- 172.21.15.90:0/3752224275 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6cbc07b930 con 0x7f6cbc07cd80 2024-04-02T14:23:45.574 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:44.759+0000 7f6cbaa00700 1 --2- 172.21.15.90:0/3752224275 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6cbc07cd80 0x7f6cbc07b1e0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f6cac0098a0 tx=0x7f6cac009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=acac2649815d6c0e server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:23:45.574 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:44.759+0000 7f6cb9600700 1 -- 172.21.15.90:0/3752224275 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6cac004160 con 0x7f6cbc07cd80 2024-04-02T14:23:45.574 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:44.759+0000 7f6cb9600700 1 -- 172.21.15.90:0/3752224275 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f6cac004300 con 0x7f6cbc07cd80 2024-04-02T14:23:45.574 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:44.759+0000 7f6cb9600700 1 -- 172.21.15.90:0/3752224275 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6cac011b50 con 0x7f6cbc07cd80 2024-04-02T14:23:45.574 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:44.760+0000 7f6cc1c00700 1 -- 172.21.15.90:0/3752224275 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6cbc07cd80 msgr2=0x7f6cbc07b1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:23:45.574 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:44.760+0000 7f6cc1c00700 1 --2- 172.21.15.90:0/3752224275 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6cbc07cd80 0x7f6cbc07b1e0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f6cac0098a0 tx=0x7f6cac009bb0 comp rx=0 tx=0).stop 2024-04-02T14:23:45.574 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:44.760+0000 7f6cc1c00700 1 -- 172.21.15.90:0/3752224275 shutdown_connections 2024-04-02T14:23:45.574 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:44.760+0000 7f6cc1c00700 1 --2- 172.21.15.90:0/3752224275 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6cbc07cd80 0x7f6cbc07b1e0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:23:45.574 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:44.760+0000 7f6cc1c00700 1 -- 172.21.15.90:0/3752224275 >> 172.21.15.90:0/3752224275 conn(0x7f6cbc103830 msgr2=0x7f6cbc105c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:23:45.574 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:44.760+0000 7f6cc1c00700 1 -- 172.21.15.90:0/3752224275 shutdown_connections 2024-04-02T14:23:45.575 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:44.760+0000 7f6cc1c00700 1 -- 172.21.15.90:0/3752224275 wait complete. 2024-04-02T14:23:45.575 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:44.761+0000 7f6cc1c00700 1 Processor -- start 2024-04-02T14:23:45.575 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:44.761+0000 7f6cc1c00700 1 -- start start 2024-04-02T14:23:45.575 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:44.761+0000 7f6cc1c00700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6cbc07cd80 0x7f6cbc19e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:23:45.575 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:44.762+0000 7f6cc1c00700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f6cbc10e8a0 con 0x7f6cbc07cd80 2024-04-02T14:23:45.575 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:44.762+0000 7f6cbaa00700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6cbc07cd80 0x7f6cbc19e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:23:45.575 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:44.762+0000 7f6cbaa00700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6cbc07cd80 0x7f6cbc19e410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:56522/0 (socket says 172.21.15.90:56522) 2024-04-02T14:23:45.575 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:44.762+0000 7f6cbaa00700 1 -- 172.21.15.90:0/2035259193 learned_addr learned my addr 172.21.15.90:0/2035259193 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:23:45.575 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:44.762+0000 7f6cbaa00700 1 -- 172.21.15.90:0/2035259193 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6cbc19e950 con 0x7f6cbc07cd80 2024-04-02T14:23:45.575 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:44.762+0000 7f6cbaa00700 1 --2- 172.21.15.90:0/2035259193 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6cbc07cd80 0x7f6cbc19e410 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f6cac011b60 tx=0x7f6cac011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:23:45.575 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:44.763+0000 7f6cb3400700 1 -- 172.21.15.90:0/2035259193 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6cac003b20 con 0x7f6cbc07cd80 2024-04-02T14:23:45.575 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:44.763+0000 7f6cb3400700 1 -- 172.21.15.90:0/2035259193 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f6cac003cc0 con 0x7f6cbc07cd80 2024-04-02T14:23:45.575 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:44.763+0000 7f6cc1c00700 1 -- 172.21.15.90:0/2035259193 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6cbc19ebe0 con 0x7f6cbc07cd80 2024-04-02T14:23:45.575 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:44.763+0000 7f6cb3400700 1 -- 172.21.15.90:0/2035259193 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6cac024840 con 0x7f6cbc07cd80 2024-04-02T14:23:45.576 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:44.763+0000 7f6cc1c00700 1 -- 172.21.15.90:0/2035259193 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6cbc1a18d0 con 0x7f6cbc07cd80 2024-04-02T14:23:45.576 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:44.763+0000 7f6cb3400700 1 -- 172.21.15.90:0/2035259193 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f6cac022020 con 0x7f6cbc07cd80 2024-04-02T14:23:45.576 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:44.763+0000 7f6cb3400700 1 -- 172.21.15.90:0/2035259193 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f6cac011e00 con 0x7f6cbc07cd80 2024-04-02T14:23:45.576 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:44.764+0000 7f6cc1c00700 1 -- 172.21.15.90:0/2035259193 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6cbc063c20 con 0x7f6cbc07cd80 2024-04-02T14:23:45.576 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:44.767+0000 7f6cb3400700 1 -- 172.21.15.90:0/2035259193 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f6cac043dd0 con 0x7f6cbc07cd80 2024-04-02T14:23:45.576 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:44.846+0000 7f6cc1c00700 1 -- 172.21.15.90:0/2035259193 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f6cbc1a1fb0 con 0x7f6cbc07cd80 2024-04-02T14:23:45.576 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:44.847+0000 7f6cb3400700 1 -- 172.21.15.90:0/2035259193 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1240 (secure 0 0 0) 0x7f6cac032030 con 0x7f6cbc07cd80 2024-04-02T14:23:45.576 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:44.848+0000 7f6cc1c00700 1 -- 172.21.15.90:0/2035259193 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6cbc07cd80 msgr2=0x7f6cbc19e410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:23:45.576 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:44.848+0000 7f6cc1c00700 1 --2- 172.21.15.90:0/2035259193 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6cbc07cd80 0x7f6cbc19e410 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f6cac011b60 tx=0x7f6cac011c40 comp rx=0 tx=0).stop 2024-04-02T14:23:45.576 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:44.848+0000 7f6cc1c00700 1 -- 172.21.15.90:0/2035259193 shutdown_connections 2024-04-02T14:23:45.576 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:44.848+0000 7f6cc1c00700 1 --2- 172.21.15.90:0/2035259193 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6cbc07cd80 0x7f6cbc19e410 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:23:45.576 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:44.848+0000 7f6cc1c00700 1 -- 172.21.15.90:0/2035259193 >> 172.21.15.90:0/2035259193 conn(0x7f6cbc103830 msgr2=0x7f6cbc104390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:23:45.576 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:44.848+0000 7f6cc1c00700 1 -- 172.21.15.90:0/2035259193 shutdown_connections 2024-04-02T14:23:45.577 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:44.848+0000 7f6cc1c00700 1 -- 172.21.15.90:0/2035259193 wait complete. 2024-04-02T14:23:45.577 INFO:teuthology.orchestra.run.smithi090.stdout:mgr not available, waiting (3/15)... 2024-04-02T14:23:46.190 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 02 14:23:45 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-x[31910]: 2024-04-02T14:23:45.798+0000 7fb9fd42b200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-02T14:23:46.190 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 02 14:23:45 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-x[31910]: 2024-04-02T14:23:45.924+0000 7fb9fd42b200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-02T14:23:46.688 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 02 14:23:46 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-x[31910]: 2024-04-02T14:23:46.201+0000 7fb9fd42b200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-02T14:23:46.689 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:46 smithi090 ceph-mon[31674]: Activating manager daemon x 2024-04-02T14:23:46.689 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:46 smithi090 ceph-mon[31674]: mgrmap e2: x(active, starting, since 0.0112251s) 2024-04-02T14:23:46.689 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:46 smithi090 ceph-mon[31674]: from='mgr.14100 172.21.15.90:0/3883459805' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-02T14:23:46.689 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:46 smithi090 ceph-mon[31674]: from='mgr.14100 172.21.15.90:0/3883459805' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-02T14:23:46.689 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:46 smithi090 ceph-mon[31674]: from='mgr.14100 172.21.15.90:0/3883459805' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-02T14:23:46.690 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:46 smithi090 ceph-mon[31674]: from='mgr.14100 172.21.15.90:0/3883459805' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-02T14:23:46.690 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:46 smithi090 ceph-mon[31674]: from='mgr.14100 172.21.15.90:0/3883459805' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-02T14:23:46.690 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:46 smithi090 ceph-mon[31674]: Manager daemon x is now available 2024-04-02T14:23:46.690 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:46 smithi090 ceph-mon[31674]: from='mgr.14100 172.21.15.90:0/3883459805' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-02T14:23:46.690 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:46 smithi090 ceph-mon[31674]: from='mgr.14100 172.21.15.90:0/3883459805' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-02T14:23:46.690 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:46 smithi090 ceph-mon[31674]: from='mgr.14100 172.21.15.90:0/3883459805' entity='mgr.x' 2024-04-02T14:23:46.690 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:46 smithi090 ceph-mon[31674]: from='mgr.14100 172.21.15.90:0/3883459805' entity='mgr.x' 2024-04-02T14:23:47.689 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:47 smithi090 ceph-mon[31674]: from='mgr.14100 172.21.15.90:0/3883459805' entity='mgr.x' 2024-04-02T14:23:47.689 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:47 smithi090 ceph-mon[31674]: mgrmap e3: x(active, since 1.02289s) 2024-04-02T14:23:49.150 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout 2024-04-02T14:23:49.151 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout { 2024-04-02T14:23:49.151 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "fsid": "6449448a-f0fc-11ee-b647-cb9ed24678a4", 2024-04-02T14:23:49.151 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "health": { 2024-04-02T14:23:49.151 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-02T14:23:49.151 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-02T14:23:49.151 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-02T14:23:49.151 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout }, 2024-04-02T14:23:49.151 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-02T14:23:49.151 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-02T14:23:49.151 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout 0 2024-04-02T14:23:49.151 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout ], 2024-04-02T14:23:49.151 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-02T14:23:49.151 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "a" 2024-04-02T14:23:49.151 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout ], 2024-04-02T14:23:49.152 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "quorum_age": 13, 2024-04-02T14:23:49.152 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-02T14:23:49.152 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-02T14:23:49.152 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-02T14:23:49.152 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-02T14:23:49.152 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout }, 2024-04-02T14:23:49.152 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-02T14:23:49.152 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-02T14:23:49.152 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-02T14:23:49.152 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-02T14:23:49.152 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-02T14:23:49.162 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-02T14:23:49.163 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-02T14:23:49.163 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-02T14:23:49.163 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout }, 2024-04-02T14:23:49.163 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-02T14:23:49.163 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-02T14:23:49.163 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-02T14:23:49.163 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-02T14:23:49.163 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-02T14:23:49.163 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-02T14:23:49.164 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-02T14:23:49.164 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-02T14:23:49.164 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-02T14:23:49.164 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout }, 2024-04-02T14:23:49.164 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-02T14:23:49.164 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-02T14:23:49.164 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-02T14:23:49.164 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-02T14:23:49.164 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout }, 2024-04-02T14:23:49.164 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-02T14:23:49.164 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-02T14:23:49.164 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-02T14:23:49.164 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-02T14:23:49.164 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-02T14:23:49.164 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-02T14:23:49.165 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "restful" 2024-04-02T14:23:49.165 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout ], 2024-04-02T14:23:49.165 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-02T14:23:49.165 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout }, 2024-04-02T14:23:49.165 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-02T14:23:49.165 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-02T14:23:49.165 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "modified": "2024-04-02T14:23:16.530776+0000", 2024-04-02T14:23:49.165 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-02T14:23:49.165 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout }, 2024-04-02T14:23:49.165 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-02T14:23:49.165 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout } 2024-04-02T14:23:49.165 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:48.149+0000 7fc475a00700 1 Processor -- start 2024-04-02T14:23:49.165 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:48.149+0000 7fc475a00700 1 -- start start 2024-04-02T14:23:49.165 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:48.150+0000 7fc475a00700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc47007ddf0 0x7fc47007c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:23:49.165 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:48.150+0000 7fc475a00700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fc47007c830 con 0x7fc47007ddf0 2024-04-02T14:23:49.166 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:48.150+0000 7fc46ea00700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc47007ddf0 0x7fc47007c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:23:49.166 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:48.150+0000 7fc46ea00700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc47007ddf0 0x7fc47007c2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:56568/0 (socket says 172.21.15.90:56568) 2024-04-02T14:23:49.166 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:48.150+0000 7fc46ea00700 1 -- 172.21.15.90:0/3932949059 learned_addr learned my addr 172.21.15.90:0/3932949059 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:23:49.166 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:48.151+0000 7fc46ea00700 1 -- 172.21.15.90:0/3932949059 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc47007c9b0 con 0x7fc47007ddf0 2024-04-02T14:23:49.166 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:48.151+0000 7fc46ea00700 1 --2- 172.21.15.90:0/3932949059 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc47007ddf0 0x7fc47007c2f0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fc4600098a0 tx=0x7fc460009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eead0687885f00a0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:23:49.166 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:48.151+0000 7fc46d600700 1 -- 172.21.15.90:0/3932949059 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc460004160 con 0x7fc47007ddf0 2024-04-02T14:23:49.166 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:48.151+0000 7fc46d600700 1 -- 172.21.15.90:0/3932949059 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fc460004300 con 0x7fc47007ddf0 2024-04-02T14:23:49.166 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:48.151+0000 7fc46d600700 1 -- 172.21.15.90:0/3932949059 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc460011b50 con 0x7fc47007ddf0 2024-04-02T14:23:49.166 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:48.152+0000 7fc475a00700 1 -- 172.21.15.90:0/3932949059 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc47007ddf0 msgr2=0x7fc47007c2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:23:49.166 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:48.152+0000 7fc475a00700 1 --2- 172.21.15.90:0/3932949059 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc47007ddf0 0x7fc47007c2f0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fc4600098a0 tx=0x7fc460009bb0 comp rx=0 tx=0).stop 2024-04-02T14:23:49.166 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:48.152+0000 7fc475a00700 1 -- 172.21.15.90:0/3932949059 shutdown_connections 2024-04-02T14:23:49.166 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:48.152+0000 7fc475a00700 1 --2- 172.21.15.90:0/3932949059 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc47007ddf0 0x7fc47007c2f0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:23:49.167 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:48.152+0000 7fc475a00700 1 -- 172.21.15.90:0/3932949059 >> 172.21.15.90:0/3932949059 conn(0x7fc470103880 msgr2=0x7fc470105ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:23:49.167 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:48.152+0000 7fc475a00700 1 -- 172.21.15.90:0/3932949059 shutdown_connections 2024-04-02T14:23:49.167 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:48.152+0000 7fc475a00700 1 -- 172.21.15.90:0/3932949059 wait complete. 2024-04-02T14:23:49.167 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:48.153+0000 7fc475a00700 1 Processor -- start 2024-04-02T14:23:49.167 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:48.153+0000 7fc475a00700 1 -- start start 2024-04-02T14:23:49.167 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:48.154+0000 7fc475a00700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc47007ddf0 0x7fc47019e690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:23:49.167 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:48.154+0000 7fc475a00700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fc47010e810 con 0x7fc47007ddf0 2024-04-02T14:23:49.167 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:48.154+0000 7fc46ea00700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc47007ddf0 0x7fc47019e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:23:49.167 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:48.154+0000 7fc46ea00700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc47007ddf0 0x7fc47019e690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:56572/0 (socket says 172.21.15.90:56572) 2024-04-02T14:23:49.167 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:48.154+0000 7fc46ea00700 1 -- 172.21.15.90:0/1364794115 learned_addr learned my addr 172.21.15.90:0/1364794115 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:23:49.167 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:48.154+0000 7fc46ea00700 1 -- 172.21.15.90:0/1364794115 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc47019ebd0 con 0x7fc47007ddf0 2024-04-02T14:23:49.167 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:48.154+0000 7fc46ea00700 1 --2- 172.21.15.90:0/1364794115 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc47007ddf0 0x7fc47019e690 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fc460011b60 tx=0x7fc460011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:23:49.167 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:48.155+0000 7fc467400700 1 -- 172.21.15.90:0/1364794115 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc460003b20 con 0x7fc47007ddf0 2024-04-02T14:23:49.167 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:48.155+0000 7fc467400700 1 -- 172.21.15.90:0/1364794115 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fc460003cc0 con 0x7fc47007ddf0 2024-04-02T14:23:49.167 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:48.155+0000 7fc475a00700 1 -- 172.21.15.90:0/1364794115 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc47019ee60 con 0x7fc47007ddf0 2024-04-02T14:23:49.168 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:48.155+0000 7fc475a00700 1 -- 172.21.15.90:0/1364794115 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4701a1b50 con 0x7fc47007ddf0 2024-04-02T14:23:49.168 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:48.155+0000 7fc467400700 1 -- 172.21.15.90:0/1364794115 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc460024840 con 0x7fc47007ddf0 2024-04-02T14:23:49.168 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:48.156+0000 7fc467400700 1 -- 172.21.15.90:0/1364794115 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 3) ==== 46441+0+0 (secure 0 0 0) 0x7fc460022020 con 0x7fc47007ddf0 2024-04-02T14:23:49.168 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:48.156+0000 7fc475a00700 1 -- 172.21.15.90:0/1364794115 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc470063c20 con 0x7fc47007ddf0 2024-04-02T14:23:49.168 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:48.156+0000 7fc467400700 1 --2- 172.21.15.90:0/1364794115 >> [v2:172.21.15.90:6800/2372398598,v1:172.21.15.90:6801/2372398598] conn(0x7fc454039490 0x7fc45403b950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:23:49.168 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:48.156+0000 7fc467400700 1 -- 172.21.15.90:0/1364794115 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fc460011e00 con 0x7fc47007ddf0 2024-04-02T14:23:49.168 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:48.157+0000 7fc46e000700 1 --2- 172.21.15.90:0/1364794115 >> [v2:172.21.15.90:6800/2372398598,v1:172.21.15.90:6801/2372398598] conn(0x7fc454039490 0x7fc45403b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:23:49.168 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:48.157+0000 7fc46e000700 1 --2- 172.21.15.90:0/1364794115 >> [v2:172.21.15.90:6800/2372398598,v1:172.21.15.90:6801/2372398598] conn(0x7fc454039490 0x7fc45403b950 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fc45c006fd0 tx=0x7fc45c006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:23:49.168 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:48.163+0000 7fc467400700 1 -- 172.21.15.90:0/1364794115 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7fc46000fdd0 con 0x7fc47007ddf0 2024-04-02T14:23:49.168 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:48.269+0000 7fc467400700 1 -- 172.21.15.90:0/1364794115 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mgrmap(e 4) ==== 46547+0+0 (secure 0 0 0) 0x7fc460024b20 con 0x7fc47007ddf0 2024-04-02T14:23:49.168 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:48.479+0000 7fc475a00700 1 -- 172.21.15.90:0/1364794115 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7fc47002dd30 con 0x7fc47007ddf0 2024-04-02T14:23:49.168 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:48.480+0000 7fc467400700 1 -- 172.21.15.90:0/1364794115 <== mon.0 v2:172.21.15.90:3300/0 8 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7fc46002c7f0 con 0x7fc47007ddf0 2024-04-02T14:23:49.168 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:48.482+0000 7fc475a00700 1 -- 172.21.15.90:0/1364794115 >> [v2:172.21.15.90:6800/2372398598,v1:172.21.15.90:6801/2372398598] conn(0x7fc454039490 msgr2=0x7fc45403b950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:23:49.169 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:48.482+0000 7fc475a00700 1 --2- 172.21.15.90:0/1364794115 >> [v2:172.21.15.90:6800/2372398598,v1:172.21.15.90:6801/2372398598] conn(0x7fc454039490 0x7fc45403b950 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fc45c006fd0 tx=0x7fc45c006e40 comp rx=0 tx=0).stop 2024-04-02T14:23:49.169 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:48.482+0000 7fc475a00700 1 -- 172.21.15.90:0/1364794115 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc47007ddf0 msgr2=0x7fc47019e690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:23:49.169 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:48.482+0000 7fc475a00700 1 --2- 172.21.15.90:0/1364794115 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc47007ddf0 0x7fc47019e690 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fc460011b60 tx=0x7fc460011c40 comp rx=0 tx=0).stop 2024-04-02T14:23:49.169 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:48.483+0000 7fc475a00700 1 -- 172.21.15.90:0/1364794115 shutdown_connections 2024-04-02T14:23:49.169 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:48.483+0000 7fc475a00700 1 --2- 172.21.15.90:0/1364794115 >> [v2:172.21.15.90:6800/2372398598,v1:172.21.15.90:6801/2372398598] conn(0x7fc454039490 0x7fc45403b950 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:23:49.169 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:48.483+0000 7fc475a00700 1 --2- 172.21.15.90:0/1364794115 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc47007ddf0 0x7fc47019e690 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:23:49.169 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:48.483+0000 7fc475a00700 1 -- 172.21.15.90:0/1364794115 >> 172.21.15.90:0/1364794115 conn(0x7fc470103880 msgr2=0x7fc470104570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:23:49.169 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:48.483+0000 7fc475a00700 1 -- 172.21.15.90:0/1364794115 shutdown_connections 2024-04-02T14:23:49.169 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:48.483+0000 7fc475a00700 1 -- 172.21.15.90:0/1364794115 wait complete. 2024-04-02T14:23:49.169 INFO:teuthology.orchestra.run.smithi090.stdout:mgr is available 2024-04-02T14:23:49.429 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:49 smithi090 ceph-mon[31674]: mgrmap e4: x(active, since 2s) 2024-04-02T14:23:49.429 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:49 smithi090 ceph-mon[31674]: from='client.? 172.21.15.90:0/1364794115' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-02T14:23:50.664 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:50 smithi090 ceph-mon[31674]: from='client.? 172.21.15.90:0/3986001099' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-02T14:23:50.673 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout 2024-04-02T14:23:50.673 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout [global] 2024-04-02T14:23:50.673 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout fsid = 6449448a-f0fc-11ee-b647-cb9ed24678a4 2024-04-02T14:23:50.673 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-04-02T14:23:50.673 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-04-02T14:23:50.673 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-04-02T14:23:50.676 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-04-02T14:23:50.676 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout 2024-04-02T14:23:50.676 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout [mgr] 2024-04-02T14:23:50.676 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-04-02T14:23:50.676 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout 2024-04-02T14:23:50.676 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout [osd] 2024-04-02T14:23:50.677 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout bdev_async_discard = True 2024-04-02T14:23:50.677 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-04-02T14:23:50.677 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-04-02T14:23:50.677 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-04-02T14:23:50.677 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-04-02T14:23:50.677 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-04-02T14:23:50.677 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:49.718+0000 7febbf200700 1 Processor -- start 2024-04-02T14:23:50.677 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:49.718+0000 7febbf200700 1 -- start start 2024-04-02T14:23:50.677 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:49.719+0000 7febbf200700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7febb810ef40 0x7febb810f360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:23:50.677 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:49.719+0000 7febbf200700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7febb810f930 con 0x7febb810ef40 2024-04-02T14:23:50.677 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:49.719+0000 7febb7e00700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7febb810ef40 0x7febb810f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:23:50.677 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:49.719+0000 7febb7e00700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7febb810ef40 0x7febb810f360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:56584/0 (socket says 172.21.15.90:56584) 2024-04-02T14:23:50.677 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:49.719+0000 7febb7e00700 1 -- 172.21.15.90:0/3818870384 learned_addr learned my addr 172.21.15.90:0/3818870384 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:23:50.677 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:49.719+0000 7febb7e00700 1 -- 172.21.15.90:0/3818870384 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7febb810fab0 con 0x7febb810ef40 2024-04-02T14:23:50.678 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:49.720+0000 7febb7e00700 1 --2- 172.21.15.90:0/3818870384 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7febb810ef40 0x7febb810f360 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7feba80098a0 tx=0x7feba8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7473f5b58042a6a9 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:23:50.678 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:49.720+0000 7febb6a00700 1 -- 172.21.15.90:0/3818870384 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7feba8004160 con 0x7febb810ef40 2024-04-02T14:23:50.678 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:49.720+0000 7febb6a00700 1 -- 172.21.15.90:0/3818870384 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7feba8004300 con 0x7febb810ef40 2024-04-02T14:23:50.678 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:49.720+0000 7febb6a00700 1 -- 172.21.15.90:0/3818870384 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7feba8011b50 con 0x7febb810ef40 2024-04-02T14:23:50.678 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:49.720+0000 7febbf200700 1 -- 172.21.15.90:0/3818870384 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7febb810ef40 msgr2=0x7febb810f360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:23:50.678 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:49.720+0000 7febbf200700 1 --2- 172.21.15.90:0/3818870384 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7febb810ef40 0x7febb810f360 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7feba80098a0 tx=0x7feba8009bb0 comp rx=0 tx=0).stop 2024-04-02T14:23:50.678 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:49.721+0000 7febbf200700 1 -- 172.21.15.90:0/3818870384 shutdown_connections 2024-04-02T14:23:50.678 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:49.721+0000 7febbf200700 1 --2- 172.21.15.90:0/3818870384 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7febb810ef40 0x7febb810f360 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:23:50.682 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:49.721+0000 7febbf200700 1 -- 172.21.15.90:0/3818870384 >> 172.21.15.90:0/3818870384 conn(0x7febb807d9c0 msgr2=0x7febb807fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:23:50.682 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:49.721+0000 7febbf200700 1 -- 172.21.15.90:0/3818870384 shutdown_connections 2024-04-02T14:23:50.682 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:49.721+0000 7febbf200700 1 -- 172.21.15.90:0/3818870384 wait complete. 2024-04-02T14:23:50.682 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:49.722+0000 7febbf200700 1 Processor -- start 2024-04-02T14:23:50.682 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:49.722+0000 7febbf200700 1 -- start start 2024-04-02T14:23:50.682 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:49.722+0000 7febbf200700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7febb810ef40 0x7febb81a27a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:23:50.682 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:49.722+0000 7febbf200700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7febb8112b70 con 0x7febb810ef40 2024-04-02T14:23:50.682 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:49.722+0000 7febb7e00700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7febb810ef40 0x7febb81a27a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:23:50.683 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:49.722+0000 7febb7e00700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7febb810ef40 0x7febb81a27a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:56600/0 (socket says 172.21.15.90:56600) 2024-04-02T14:23:50.683 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:49.722+0000 7febb7e00700 1 -- 172.21.15.90:0/3986001099 learned_addr learned my addr 172.21.15.90:0/3986001099 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:23:50.683 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:49.723+0000 7febb7e00700 1 -- 172.21.15.90:0/3986001099 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7febb81a2ce0 con 0x7febb810ef40 2024-04-02T14:23:50.683 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:49.723+0000 7febb7e00700 1 --2- 172.21.15.90:0/3986001099 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7febb810ef40 0x7febb81a27a0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7feba8011850 tx=0x7feba8011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:23:50.683 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:49.723+0000 7febb4c00700 1 -- 172.21.15.90:0/3986001099 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7feba8003af0 con 0x7febb810ef40 2024-04-02T14:23:50.683 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:49.723+0000 7febb4c00700 1 -- 172.21.15.90:0/3986001099 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7feba8020070 con 0x7febb810ef40 2024-04-02T14:23:50.683 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:49.723+0000 7febbf200700 1 -- 172.21.15.90:0/3986001099 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7febb81a2fd0 con 0x7febb810ef40 2024-04-02T14:23:50.683 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:49.723+0000 7febb4c00700 1 -- 172.21.15.90:0/3986001099 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7feba8019550 con 0x7febb810ef40 2024-04-02T14:23:50.683 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:49.724+0000 7febbf200700 1 -- 172.21.15.90:0/3986001099 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7febb81a5c60 con 0x7febb810ef40 2024-04-02T14:23:50.683 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:49.724+0000 7febb4c00700 1 -- 172.21.15.90:0/3986001099 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 4) ==== 46547+0+0 (secure 0 0 0) 0x7feba8019850 con 0x7febb810ef40 2024-04-02T14:23:50.683 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:49.725+0000 7febbf200700 1 -- 172.21.15.90:0/3986001099 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7febb8063c20 con 0x7febb810ef40 2024-04-02T14:23:50.683 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:49.725+0000 7febb4c00700 1 --2- 172.21.15.90:0/3986001099 >> [v2:172.21.15.90:6800/2372398598,v1:172.21.15.90:6801/2372398598] conn(0x7feb9c039560 0x7feb9c03ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:23:50.683 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:49.725+0000 7febb4c00700 1 -- 172.21.15.90:0/3986001099 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7feba8011e00 con 0x7febb810ef40 2024-04-02T14:23:50.683 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:49.725+0000 7febb7400700 1 --2- 172.21.15.90:0/3986001099 >> [v2:172.21.15.90:6800/2372398598,v1:172.21.15.90:6801/2372398598] conn(0x7feb9c039560 0x7feb9c03ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:23:50.684 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:49.726+0000 7febb7400700 1 --2- 172.21.15.90:0/3986001099 >> [v2:172.21.15.90:6800/2372398598,v1:172.21.15.90:6801/2372398598] conn(0x7feb9c039560 0x7feb9c03ba20 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7feba4006fd0 tx=0x7feba4006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:23:50.684 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:49.730+0000 7febb4c00700 1 -- 172.21.15.90:0/3986001099 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7feba800e260 con 0x7febb810ef40 2024-04-02T14:23:50.684 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:49.953+0000 7febbf200700 1 -- 172.21.15.90:0/3986001099 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7febb81a5e30 con 0x7febb810ef40 2024-04-02T14:23:50.684 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:49.956+0000 7febb4c00700 1 -- 172.21.15.90:0/3986001099 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) ==== 70+0+400 (secure 0 0 0) 0x7feba804d4e0 con 0x7febb810ef40 2024-04-02T14:23:50.684 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:49.958+0000 7febbf200700 1 -- 172.21.15.90:0/3986001099 >> [v2:172.21.15.90:6800/2372398598,v1:172.21.15.90:6801/2372398598] conn(0x7feb9c039560 msgr2=0x7feb9c03ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:23:50.684 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:49.958+0000 7febbf200700 1 --2- 172.21.15.90:0/3986001099 >> [v2:172.21.15.90:6800/2372398598,v1:172.21.15.90:6801/2372398598] conn(0x7feb9c039560 0x7feb9c03ba20 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7feba4006fd0 tx=0x7feba4006e40 comp rx=0 tx=0).stop 2024-04-02T14:23:50.684 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:49.958+0000 7febbf200700 1 -- 172.21.15.90:0/3986001099 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7febb810ef40 msgr2=0x7febb81a27a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:23:50.684 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:49.958+0000 7febbf200700 1 --2- 172.21.15.90:0/3986001099 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7febb810ef40 0x7febb81a27a0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7feba8011850 tx=0x7feba8011c10 comp rx=0 tx=0).stop 2024-04-02T14:23:50.684 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:49.959+0000 7febbf200700 1 -- 172.21.15.90:0/3986001099 shutdown_connections 2024-04-02T14:23:50.684 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:49.959+0000 7febbf200700 1 --2- 172.21.15.90:0/3986001099 >> [v2:172.21.15.90:6800/2372398598,v1:172.21.15.90:6801/2372398598] conn(0x7feb9c039560 0x7feb9c03ba20 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:23:50.684 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:49.959+0000 7febbf200700 1 --2- 172.21.15.90:0/3986001099 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7febb810ef40 0x7febb81a27a0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:23:50.684 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:49.959+0000 7febbf200700 1 -- 172.21.15.90:0/3986001099 >> 172.21.15.90:0/3986001099 conn(0x7febb807d9c0 msgr2=0x7febb807e4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:23:50.684 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:49.959+0000 7febbf200700 1 -- 172.21.15.90:0/3986001099 shutdown_connections 2024-04-02T14:23:50.684 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:49.959+0000 7febbf200700 1 -- 172.21.15.90:0/3986001099 wait complete. 2024-04-02T14:23:50.685 INFO:teuthology.orchestra.run.smithi090.stdout:Enabling cephadm module... 2024-04-02T14:23:51.939 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:51 smithi090 ceph-mon[31674]: from='client.? 172.21.15.90:0/2691787795' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-04-02T14:23:51.939 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 02 14:23:51 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-x[31910]: ignoring --setuser ceph since I am not root 2024-04-02T14:23:51.939 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 02 14:23:51 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-x[31910]: ignoring --setgroup ceph since I am not root 2024-04-02T14:23:52.266 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:51.248+0000 7ff3db200700 1 Processor -- start 2024-04-02T14:23:52.266 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:51.248+0000 7ff3db200700 1 -- start start 2024-04-02T14:23:52.266 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:51.248+0000 7ff3db200700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff3dc1080a0 0x7ff3dc1084c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:23:52.266 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:51.248+0000 7ff3db200700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7ff3dc108a00 con 0x7ff3dc1080a0 2024-04-02T14:23:52.266 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:51.249+0000 7ff3d9e00700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff3dc1080a0 0x7ff3dc1084c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:23:52.266 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:51.249+0000 7ff3d9e00700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff3dc1080a0 0x7ff3dc1084c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:56610/0 (socket says 172.21.15.90:56610) 2024-04-02T14:23:52.266 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:51.249+0000 7ff3d9e00700 1 -- 172.21.15.90:0/3989502132 learned_addr learned my addr 172.21.15.90:0/3989502132 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:23:52.266 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:51.249+0000 7ff3d9e00700 1 -- 172.21.15.90:0/3989502132 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3dc108b80 con 0x7ff3dc1080a0 2024-04-02T14:23:52.267 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:51.250+0000 7ff3d9e00700 1 --2- 172.21.15.90:0/3989502132 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff3dc1080a0 0x7ff3dc1084c0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7ff3cc0098a0 tx=0x7ff3cc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19b91d61298bbeb9 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:23:52.267 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:51.250+0000 7ff3d8a00700 1 -- 172.21.15.90:0/3989502132 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff3cc004160 con 0x7ff3dc1080a0 2024-04-02T14:23:52.267 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:51.250+0000 7ff3d8a00700 1 -- 172.21.15.90:0/3989502132 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7ff3cc004300 con 0x7ff3dc1080a0 2024-04-02T14:23:52.267 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:51.250+0000 7ff3d8a00700 1 -- 172.21.15.90:0/3989502132 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff3cc011b50 con 0x7ff3dc1080a0 2024-04-02T14:23:52.267 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:51.250+0000 7ff3db200700 1 -- 172.21.15.90:0/3989502132 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff3dc1080a0 msgr2=0x7ff3dc1084c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:23:52.267 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:51.250+0000 7ff3db200700 1 --2- 172.21.15.90:0/3989502132 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff3dc1080a0 0x7ff3dc1084c0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7ff3cc0098a0 tx=0x7ff3cc009bb0 comp rx=0 tx=0).stop 2024-04-02T14:23:52.267 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:51.251+0000 7ff3db200700 1 -- 172.21.15.90:0/3989502132 shutdown_connections 2024-04-02T14:23:52.267 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:51.251+0000 7ff3db200700 1 --2- 172.21.15.90:0/3989502132 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff3dc1080a0 0x7ff3dc1084c0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:23:52.267 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:51.251+0000 7ff3db200700 1 -- 172.21.15.90:0/3989502132 >> 172.21.15.90:0/3989502132 conn(0x7ff3dc103740 msgr2=0x7ff3dc105b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:23:52.267 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:51.251+0000 7ff3db200700 1 -- 172.21.15.90:0/3989502132 shutdown_connections 2024-04-02T14:23:52.267 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:51.251+0000 7ff3db200700 1 -- 172.21.15.90:0/3989502132 wait complete. 2024-04-02T14:23:52.267 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:51.252+0000 7ff3db200700 1 Processor -- start 2024-04-02T14:23:52.267 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:51.252+0000 7ff3db200700 1 -- start start 2024-04-02T14:23:52.267 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:51.252+0000 7ff3db200700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff3dc1080a0 0x7ff3dc19e570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:23:52.268 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:51.252+0000 7ff3db200700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7ff3dc10c510 con 0x7ff3dc1080a0 2024-04-02T14:23:52.268 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:51.253+0000 7ff3d9e00700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff3dc1080a0 0x7ff3dc19e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:23:52.268 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:51.253+0000 7ff3d9e00700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff3dc1080a0 0x7ff3dc19e570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:56618/0 (socket says 172.21.15.90:56618) 2024-04-02T14:23:52.268 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:51.253+0000 7ff3d9e00700 1 -- 172.21.15.90:0/2691787795 learned_addr learned my addr 172.21.15.90:0/2691787795 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:23:52.268 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:51.253+0000 7ff3d9e00700 1 -- 172.21.15.90:0/2691787795 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3dc19eab0 con 0x7ff3dc1080a0 2024-04-02T14:23:52.268 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:51.253+0000 7ff3d9e00700 1 --2- 172.21.15.90:0/2691787795 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff3dc1080a0 0x7ff3dc19e570 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7ff3cc011b60 tx=0x7ff3cc011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:23:52.268 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:51.253+0000 7ff3d2a00700 1 -- 172.21.15.90:0/2691787795 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff3cc003b20 con 0x7ff3dc1080a0 2024-04-02T14:23:52.268 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:51.254+0000 7ff3d2a00700 1 -- 172.21.15.90:0/2691787795 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7ff3cc003cc0 con 0x7ff3dc1080a0 2024-04-02T14:23:52.268 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:51.254+0000 7ff3db200700 1 -- 172.21.15.90:0/2691787795 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff3dc19ed40 con 0x7ff3dc1080a0 2024-04-02T14:23:52.268 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:51.254+0000 7ff3d2a00700 1 -- 172.21.15.90:0/2691787795 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff3cc024840 con 0x7ff3dc1080a0 2024-04-02T14:23:52.268 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:51.254+0000 7ff3db200700 1 -- 172.21.15.90:0/2691787795 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3dc1a19e0 con 0x7ff3dc1080a0 2024-04-02T14:23:52.268 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:51.255+0000 7ff3d2a00700 1 -- 172.21.15.90:0/2691787795 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 4) ==== 46547+0+0 (secure 0 0 0) 0x7ff3cc022020 con 0x7ff3dc1080a0 2024-04-02T14:23:52.268 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:51.255+0000 7ff3db200700 1 -- 172.21.15.90:0/2691787795 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff3dc063c20 con 0x7ff3dc1080a0 2024-04-02T14:23:52.268 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:51.255+0000 7ff3d2a00700 1 --2- 172.21.15.90:0/2691787795 >> [v2:172.21.15.90:6800/2372398598,v1:172.21.15.90:6801/2372398598] conn(0x7ff3c00395b0 0x7ff3c003ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:23:52.268 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:51.255+0000 7ff3d2a00700 1 -- 172.21.15.90:0/2691787795 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7ff3cc011e00 con 0x7ff3dc1080a0 2024-04-02T14:23:52.269 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:51.256+0000 7ff3d9400700 1 --2- 172.21.15.90:0/2691787795 >> [v2:172.21.15.90:6800/2372398598,v1:172.21.15.90:6801/2372398598] conn(0x7ff3c00395b0 0x7ff3c003ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:23:52.269 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:51.256+0000 7ff3d9400700 1 --2- 172.21.15.90:0/2691787795 >> [v2:172.21.15.90:6800/2372398598,v1:172.21.15.90:6801/2372398598] conn(0x7ff3c00395b0 0x7ff3c003ba70 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7ff3c8006fd0 tx=0x7ff3c8006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:23:52.269 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:51.261+0000 7ff3d2a00700 1 -- 172.21.15.90:0/2691787795 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7ff3cc04d190 con 0x7ff3dc1080a0 2024-04-02T14:23:52.269 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:51.540+0000 7ff3db200700 1 -- 172.21.15.90:0/2691787795 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) -- 0x7ff3dc02dd30 con 0x7ff3dc1080a0 2024-04-02T14:23:52.269 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:51.593+0000 7ff3d2a00700 1 -- 172.21.15.90:0/2691787795 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) ==== 86+0+0 (secure 0 0 0) 0x7ff3cc01a7b0 con 0x7ff3dc1080a0 2024-04-02T14:23:52.269 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:51.597+0000 7ff3d2a00700 1 -- 172.21.15.90:0/2691787795 <== mon.0 v2:172.21.15.90:3300/0 8 ==== mgrmap(e 5) ==== 46558+0+0 (secure 0 0 0) 0x7ff3cc00b3e0 con 0x7ff3dc1080a0 2024-04-02T14:23:52.269 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:51.598+0000 7ff3db200700 1 -- 172.21.15.90:0/2691787795 >> [v2:172.21.15.90:6800/2372398598,v1:172.21.15.90:6801/2372398598] conn(0x7ff3c00395b0 msgr2=0x7ff3c003ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:23:52.269 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:51.598+0000 7ff3db200700 1 --2- 172.21.15.90:0/2691787795 >> [v2:172.21.15.90:6800/2372398598,v1:172.21.15.90:6801/2372398598] conn(0x7ff3c00395b0 0x7ff3c003ba70 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7ff3c8006fd0 tx=0x7ff3c8006e40 comp rx=0 tx=0).stop 2024-04-02T14:23:52.269 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:51.598+0000 7ff3db200700 1 -- 172.21.15.90:0/2691787795 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff3dc1080a0 msgr2=0x7ff3dc19e570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:23:52.269 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:51.598+0000 7ff3db200700 1 --2- 172.21.15.90:0/2691787795 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff3dc1080a0 0x7ff3dc19e570 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7ff3cc011b60 tx=0x7ff3cc011c40 comp rx=0 tx=0).stop 2024-04-02T14:23:52.269 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:51.598+0000 7ff3db200700 1 -- 172.21.15.90:0/2691787795 shutdown_connections 2024-04-02T14:23:52.269 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:51.598+0000 7ff3db200700 1 --2- 172.21.15.90:0/2691787795 >> [v2:172.21.15.90:6800/2372398598,v1:172.21.15.90:6801/2372398598] conn(0x7ff3c00395b0 0x7ff3c003ba70 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:23:52.269 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:51.598+0000 7ff3db200700 1 --2- 172.21.15.90:0/2691787795 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff3dc1080a0 0x7ff3dc19e570 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:23:52.269 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:51.598+0000 7ff3db200700 1 -- 172.21.15.90:0/2691787795 >> 172.21.15.90:0/2691787795 conn(0x7ff3dc103740 msgr2=0x7ff3dc1042d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:23:52.269 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:51.599+0000 7ff3db200700 1 -- 172.21.15.90:0/2691787795 shutdown_connections 2024-04-02T14:23:52.270 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:51.599+0000 7ff3db200700 1 -- 172.21.15.90:0/2691787795 wait complete. 2024-04-02T14:23:52.500 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 02 14:23:52 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-x[31910]: 2024-04-02T14:23:52.375+0000 7effbcbe8200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-02T14:23:52.802 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:52 smithi090 ceph-mon[31674]: from='client.? 172.21.15.90:0/2691787795' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-04-02T14:23:52.803 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:52 smithi090 ceph-mon[31674]: mgrmap e5: x(active, since 5s) 2024-04-02T14:23:52.803 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 02 14:23:52 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-x[31910]: 2024-04-02T14:23:52.497+0000 7effbcbe8200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-02T14:23:53.438 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 02 14:23:53 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-x[31910]: 2024-04-02T14:23:53.324+0000 7effbcbe8200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-02T14:23:53.751 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:23:53 smithi090 ceph-mon[31674]: from='client.? 172.21.15.90:0/3316453596' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-02T14:23:53.752 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 02 14:23:53 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-x[31910]: 2024-04-02T14:23:53.447+0000 7effbcbe8200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-02T14:23:53.752 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 02 14:23:53 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-x[31910]: 2024-04-02T14:23:53.625+0000 7effbcbe8200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-02T14:23:53.752 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 02 14:23:53 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-x[31910]: 2024-04-02T14:23:53.749+0000 7effbcbe8200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-02T14:23:53.792 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout { 2024-04-02T14:23:53.792 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-04-02T14:23:53.792 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-02T14:23:53.792 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-04-02T14:23:53.793 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-04-02T14:23:53.793 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout } 2024-04-02T14:23:53.793 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:52.800+0000 7fb0b7200700 1 Processor -- start 2024-04-02T14:23:53.793 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:52.800+0000 7fb0b7200700 1 -- start start 2024-04-02T14:23:53.793 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:52.801+0000 7fb0b7200700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb0b01082a0 0x7fb0b01086c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:23:53.793 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:52.801+0000 7fb0b7200700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fb0b0108c00 con 0x7fb0b01082a0 2024-04-02T14:23:53.793 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:52.801+0000 7fb0afe00700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb0b01082a0 0x7fb0b01086c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:23:53.793 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:52.801+0000 7fb0afe00700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb0b01082a0 0x7fb0b01086c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:56652/0 (socket says 172.21.15.90:56652) 2024-04-02T14:23:53.793 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:52.801+0000 7fb0afe00700 1 -- 172.21.15.90:0/673632651 learned_addr learned my addr 172.21.15.90:0/673632651 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:23:53.793 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:52.801+0000 7fb0afe00700 1 -- 172.21.15.90:0/673632651 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0b0108d80 con 0x7fb0b01082a0 2024-04-02T14:23:53.798 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:52.802+0000 7fb0afe00700 1 --2- 172.21.15.90:0/673632651 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb0b01082a0 0x7fb0b01086c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fb0a00098a0 tx=0x7fb0a0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ab6f4c1f823bbc9 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:23:53.798 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:52.802+0000 7fb0aea00700 1 -- 172.21.15.90:0/673632651 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb0a0004160 con 0x7fb0b01082a0 2024-04-02T14:23:53.798 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:52.802+0000 7fb0aea00700 1 -- 172.21.15.90:0/673632651 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fb0a0004300 con 0x7fb0b01082a0 2024-04-02T14:23:53.799 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:52.802+0000 7fb0aea00700 1 -- 172.21.15.90:0/673632651 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb0a0011b50 con 0x7fb0b01082a0 2024-04-02T14:23:53.799 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:52.803+0000 7fb0b7200700 1 -- 172.21.15.90:0/673632651 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb0b01082a0 msgr2=0x7fb0b01086c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:23:53.799 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:52.803+0000 7fb0b7200700 1 --2- 172.21.15.90:0/673632651 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb0b01082a0 0x7fb0b01086c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fb0a00098a0 tx=0x7fb0a0009bb0 comp rx=0 tx=0).stop 2024-04-02T14:23:53.799 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:52.803+0000 7fb0b7200700 1 -- 172.21.15.90:0/673632651 shutdown_connections 2024-04-02T14:23:53.799 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:52.803+0000 7fb0b7200700 1 --2- 172.21.15.90:0/673632651 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb0b01082a0 0x7fb0b01086c0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:23:53.799 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:52.803+0000 7fb0b7200700 1 -- 172.21.15.90:0/673632651 >> 172.21.15.90:0/673632651 conn(0x7fb0b0103900 msgr2=0x7fb0b0105d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:23:53.799 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:52.803+0000 7fb0b7200700 1 -- 172.21.15.90:0/673632651 shutdown_connections 2024-04-02T14:23:53.799 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:52.803+0000 7fb0b7200700 1 -- 172.21.15.90:0/673632651 wait complete. 2024-04-02T14:23:53.799 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:52.804+0000 7fb0b7200700 1 Processor -- start 2024-04-02T14:23:53.799 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:52.804+0000 7fb0b7200700 1 -- start start 2024-04-02T14:23:53.799 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:52.804+0000 7fb0b7200700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb0b01082a0 0x7fb0b019c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:23:53.799 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:52.804+0000 7fb0b7200700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fb0b010c6c0 con 0x7fb0b01082a0 2024-04-02T14:23:53.799 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:52.805+0000 7fb0afe00700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb0b01082a0 0x7fb0b019c190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:23:53.800 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:52.805+0000 7fb0afe00700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb0b01082a0 0x7fb0b019c190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:56656/0 (socket says 172.21.15.90:56656) 2024-04-02T14:23:53.800 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:52.805+0000 7fb0afe00700 1 -- 172.21.15.90:0/3316453596 learned_addr learned my addr 172.21.15.90:0/3316453596 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:23:53.800 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:52.805+0000 7fb0afe00700 1 -- 172.21.15.90:0/3316453596 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0b019c6d0 con 0x7fb0b01082a0 2024-04-02T14:23:53.800 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:52.805+0000 7fb0afe00700 1 --2- 172.21.15.90:0/3316453596 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb0b01082a0 0x7fb0b019c190 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fb0a0011b60 tx=0x7fb0a0011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:23:53.800 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:52.805+0000 7fb0acc00700 1 -- 172.21.15.90:0/3316453596 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb0a0003b20 con 0x7fb0b01082a0 2024-04-02T14:23:53.800 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:52.806+0000 7fb0acc00700 1 -- 172.21.15.90:0/3316453596 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fb0a0003cc0 con 0x7fb0b01082a0 2024-04-02T14:23:53.800 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:52.806+0000 7fb0b7200700 1 -- 172.21.15.90:0/3316453596 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0b019c960 con 0x7fb0b01082a0 2024-04-02T14:23:53.800 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:52.806+0000 7fb0acc00700 1 -- 172.21.15.90:0/3316453596 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb0a0024840 con 0x7fb0b01082a0 2024-04-02T14:23:53.800 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:52.806+0000 7fb0b7200700 1 -- 172.21.15.90:0/3316453596 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0b019f650 con 0x7fb0b01082a0 2024-04-02T14:23:53.800 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:52.807+0000 7fb0acc00700 1 -- 172.21.15.90:0/3316453596 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 46558+0+0 (secure 0 0 0) 0x7fb0a0022020 con 0x7fb0b01082a0 2024-04-02T14:23:53.800 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:52.807+0000 7fb0b7200700 1 -- 172.21.15.90:0/3316453596 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb0b0063c20 con 0x7fb0b01082a0 2024-04-02T14:23:53.800 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:52.807+0000 7fb0acc00700 1 --2- 172.21.15.90:0/3316453596 >> [v2:172.21.15.90:6800/2372398598,v1:172.21.15.90:6801/2372398598] conn(0x7fb094039600 0x7fb09403bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:23:53.800 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:52.807+0000 7fb0acc00700 1 -- 172.21.15.90:0/3316453596 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fb0a0011e00 con 0x7fb0b01082a0 2024-04-02T14:23:53.800 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:52.807+0000 7fb0af400700 1 -- 172.21.15.90:0/3316453596 >> [v2:172.21.15.90:6800/2372398598,v1:172.21.15.90:6801/2372398598] conn(0x7fb094039600 msgr2=0x7fb09403bac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.90:6800/2372398598 2024-04-02T14:23:53.801 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:52.807+0000 7fb0af400700 1 --2- 172.21.15.90:0/3316453596 >> [v2:172.21.15.90:6800/2372398598,v1:172.21.15.90:6801/2372398598] conn(0x7fb094039600 0x7fb09403bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T14:23:53.801 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:52.813+0000 7fb0acc00700 1 -- 172.21.15.90:0/3316453596 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7fb0a004d1c0 con 0x7fb0b01082a0 2024-04-02T14:23:53.801 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:53.008+0000 7fb0af400700 1 -- 172.21.15.90:0/3316453596 >> [v2:172.21.15.90:6800/2372398598,v1:172.21.15.90:6801/2372398598] conn(0x7fb094039600 msgr2=0x7fb09403bac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.90:6800/2372398598 2024-04-02T14:23:53.801 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:53.008+0000 7fb0af400700 1 --2- 172.21.15.90:0/3316453596 >> [v2:172.21.15.90:6800/2372398598,v1:172.21.15.90:6801/2372398598] conn(0x7fb094039600 0x7fb09403bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T14:23:53.801 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:53.096+0000 7fb0b7200700 1 -- 172.21.15.90:0/3316453596 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7fb0b019f820 con 0x7fb0b01082a0 2024-04-02T14:23:53.801 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:53.097+0000 7fb0acc00700 1 -- 172.21.15.90:0/3316453596 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) ==== 56+0+88 (secure 0 0 0) 0x7fb0b019f820 con 0x7fb0b01082a0 2024-04-02T14:23:53.801 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:53.099+0000 7fb0b7200700 1 -- 172.21.15.90:0/3316453596 >> [v2:172.21.15.90:6800/2372398598,v1:172.21.15.90:6801/2372398598] conn(0x7fb094039600 msgr2=0x7fb09403bac0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T14:23:53.801 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:53.099+0000 7fb0b7200700 1 --2- 172.21.15.90:0/3316453596 >> [v2:172.21.15.90:6800/2372398598,v1:172.21.15.90:6801/2372398598] conn(0x7fb094039600 0x7fb09403bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:23:53.801 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:53.099+0000 7fb0b7200700 1 -- 172.21.15.90:0/3316453596 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb0b01082a0 msgr2=0x7fb0b019c190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:23:53.801 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:53.099+0000 7fb0b7200700 1 --2- 172.21.15.90:0/3316453596 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb0b01082a0 0x7fb0b019c190 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fb0a0011b60 tx=0x7fb0a0011c40 comp rx=0 tx=0).stop 2024-04-02T14:23:53.801 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:53.099+0000 7fb0b7200700 1 -- 172.21.15.90:0/3316453596 shutdown_connections 2024-04-02T14:23:53.801 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:53.099+0000 7fb0b7200700 1 --2- 172.21.15.90:0/3316453596 >> [v2:172.21.15.90:6800/2372398598,v1:172.21.15.90:6801/2372398598] conn(0x7fb094039600 0x7fb09403bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:23:53.801 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:53.099+0000 7fb0b7200700 1 --2- 172.21.15.90:0/3316453596 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb0b01082a0 0x7fb0b019c190 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:23:53.802 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:53.099+0000 7fb0b7200700 1 -- 172.21.15.90:0/3316453596 >> 172.21.15.90:0/3316453596 conn(0x7fb0b0103900 msgr2=0x7fb0b01043a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:23:53.802 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:53.100+0000 7fb0b7200700 1 -- 172.21.15.90:0/3316453596 shutdown_connections 2024-04-02T14:23:53.802 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:53.100+0000 7fb0b7200700 1 -- 172.21.15.90:0/3316453596 wait complete. 2024-04-02T14:23:53.802 INFO:teuthology.orchestra.run.smithi090.stdout:Waiting for the mgr to restart... 2024-04-02T14:23:53.802 INFO:teuthology.orchestra.run.smithi090.stdout:Waiting for mgr epoch 5... 2024-04-02T14:23:54.104 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 02 14:23:53 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-x[31910]: 2024-04-02T14:23:53.870+0000 7effbcbe8200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-02T14:23:54.377 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 02 14:23:54 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-x[31910]: 2024-04-02T14:23:54.231+0000 7effbcbe8200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-02T14:23:54.689 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 02 14:23:54 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-x[31910]: 2024-04-02T14:23:54.376+0000 7effbcbe8200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-02T14:23:54.689 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 02 14:23:54 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-x[31910]: 2024-04-02T14:23:54.488+0000 7effbcbe8200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-02T14:23:54.689 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 02 14:23:54 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-x[31910]: 2024-04-02T14:23:54.609+0000 7effbcbe8200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-02T14:23:55.188 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 02 14:23:54 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-x[31910]: 2024-04-02T14:23:54.825+0000 7effbcbe8200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-02T14:23:56.438 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 02 14:23:56 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-x[31910]: 2024-04-02T14:23:56.076+0000 7effbcbe8200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-02T14:23:56.439 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 02 14:23:56 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-x[31910]: 2024-04-02T14:23:56.248+0000 7effbcbe8200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-02T14:23:56.938 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 02 14:23:56 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-x[31910]: 2024-04-02T14:23:56.659+0000 7effbcbe8200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-02T14:23:56.939 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 02 14:23:56 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-x[31910]: 2024-04-02T14:23:56.776+0000 7effbcbe8200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-02T14:23:57.391 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 02 14:23:57 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-x[31910]: 2024-04-02T14:23:57.029+0000 7effbcbe8200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-02T14:23:57.391 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 02 14:23:57 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-x[31910]: 2024-04-02T14:23:57.270+0000 7effbcbe8200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-02T14:23:57.938 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 02 14:23:57 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-x[31910]: 2024-04-02T14:23:57.641+0000 7effbcbe8200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-02T14:23:58.257 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 02 14:23:57 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-x[31910]: 2024-04-02T14:23:57.954+0000 7effbcbe8200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-02T14:23:58.257 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 02 14:23:58 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-x[31910]: 2024-04-02T14:23:58.254+0000 7effbcbe8200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-02T14:23:59.688 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 02 14:23:59 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-x[31910]: 2024-04-02T14:23:59.264+0000 7effbcbe8200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-02T14:24:00.360 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 02 14:23:59 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-x[31910]: 2024-04-02T14:23:59.951+0000 7effbcbe8200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-02T14:24:00.360 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 02 14:24:00 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-x[31910]: 2024-04-02T14:24:00.080+0000 7effbcbe8200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-02T14:24:00.688 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 02 14:24:00 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-x[31910]: 2024-04-02T14:24:00.357+0000 7effbcbe8200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-02T14:24:00.689 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:24:00 smithi090 ceph-mon[31674]: Active manager daemon x restarted 2024-04-02T14:24:00.689 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:24:00 smithi090 ceph-mon[31674]: Activating manager daemon x 2024-04-02T14:24:00.689 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:24:00 smithi090 ceph-mon[31674]: osdmap e2: 0 total, 0 up, 0 in 2024-04-02T14:24:00.689 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:24:00 smithi090 ceph-mon[31674]: mgrmap e6: x(active, starting, since 0.00968259s) 2024-04-02T14:24:00.689 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:24:00 smithi090 ceph-mon[31674]: from='mgr.14120 172.21.15.90:0/2851514084' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-02T14:24:00.689 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:24:00 smithi090 ceph-mon[31674]: from='mgr.14120 172.21.15.90:0/2851514084' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-02T14:24:00.689 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:24:00 smithi090 ceph-mon[31674]: from='mgr.14120 172.21.15.90:0/2851514084' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-02T14:24:00.689 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:24:00 smithi090 ceph-mon[31674]: from='mgr.14120 172.21.15.90:0/2851514084' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-02T14:24:00.689 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:24:00 smithi090 ceph-mon[31674]: from='mgr.14120 172.21.15.90:0/2851514084' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-02T14:24:00.689 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:24:00 smithi090 ceph-mon[31674]: Manager daemon x is now available 2024-04-02T14:24:00.689 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:24:00 smithi090 ceph-mon[31674]: from='mgr.14120 172.21.15.90:0/2851514084' entity='mgr.x' 2024-04-02T14:24:01.689 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:24:01 smithi090 ceph-mon[31674]: Found migration_current of "None". Setting to last migration. 2024-04-02T14:24:01.689 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:24:01 smithi090 ceph-mon[31674]: from='mgr.14120 172.21.15.90:0/2851514084' entity='mgr.x' 2024-04-02T14:24:01.689 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:24:01 smithi090 ceph-mon[31674]: from='mgr.14120 172.21.15.90:0/2851514084' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T14:24:01.689 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:24:01 smithi090 ceph-mon[31674]: from='mgr.14120 172.21.15.90:0/2851514084' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T14:24:01.689 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:24:01 smithi090 ceph-mon[31674]: from='mgr.14120 172.21.15.90:0/2851514084' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-02T14:24:01.689 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:24:01 smithi090 ceph-mon[31674]: from='mgr.14120 172.21.15.90:0/2851514084' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-02T14:24:01.689 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:24:01 smithi090 ceph-mon[31674]: from='mgr.14120 172.21.15.90:0/2851514084' entity='mgr.x' 2024-04-02T14:24:01.689 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:24:01 smithi090 ceph-mon[31674]: from='mgr.14120 172.21.15.90:0/2851514084' entity='mgr.x' 2024-04-02T14:24:01.689 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:24:01 smithi090 ceph-mon[31674]: mgrmap e7: x(active, since 1.01638s) 2024-04-02T14:24:02.107 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout { 2024-04-02T14:24:02.107 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-04-02T14:24:02.107 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "initialized": true 2024-04-02T14:24:02.107 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout } 2024-04-02T14:24:02.107 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:54.374+0000 7fe72aa00700 1 Processor -- start 2024-04-02T14:24:02.115 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:54.375+0000 7fe72aa00700 1 -- start start 2024-04-02T14:24:02.115 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:54.375+0000 7fe72aa00700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe72407cdc0 0x7fe72407b220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:24:02.115 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:54.375+0000 7fe72aa00700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fe72407b7f0 con 0x7fe72407cdc0 2024-04-02T14:24:02.115 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:54.376+0000 7fe723e00700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe72407cdc0 0x7fe72407b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:24:02.115 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:54.376+0000 7fe723e00700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe72407cdc0 0x7fe72407b220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33088/0 (socket says 172.21.15.90:33088) 2024-04-02T14:24:02.115 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:54.376+0000 7fe723e00700 1 -- 172.21.15.90:0/262541402 learned_addr learned my addr 172.21.15.90:0/262541402 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:24:02.115 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:54.376+0000 7fe723e00700 1 -- 172.21.15.90:0/262541402 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe72407b970 con 0x7fe72407cdc0 2024-04-02T14:24:02.115 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:54.376+0000 7fe723e00700 1 --2- 172.21.15.90:0/262541402 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe72407cdc0 0x7fe72407b220 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fe7140098a0 tx=0x7fe714009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dec94b4167b67f7 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:24:02.116 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:54.377+0000 7fe722a00700 1 -- 172.21.15.90:0/262541402 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe714004160 con 0x7fe72407cdc0 2024-04-02T14:24:02.116 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:54.377+0000 7fe722a00700 1 -- 172.21.15.90:0/262541402 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fe714004300 con 0x7fe72407cdc0 2024-04-02T14:24:02.116 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:54.377+0000 7fe722a00700 1 -- 172.21.15.90:0/262541402 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe714011b50 con 0x7fe72407cdc0 2024-04-02T14:24:02.116 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:54.377+0000 7fe72aa00700 1 -- 172.21.15.90:0/262541402 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe72407cdc0 msgr2=0x7fe72407b220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:24:02.116 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:54.377+0000 7fe72aa00700 1 --2- 172.21.15.90:0/262541402 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe72407cdc0 0x7fe72407b220 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fe7140098a0 tx=0x7fe714009bb0 comp rx=0 tx=0).stop 2024-04-02T14:24:02.116 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:54.378+0000 7fe72aa00700 1 -- 172.21.15.90:0/262541402 shutdown_connections 2024-04-02T14:24:02.116 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:54.378+0000 7fe72aa00700 1 --2- 172.21.15.90:0/262541402 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe72407cdc0 0x7fe72407b220 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:24:02.116 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:54.378+0000 7fe72aa00700 1 -- 172.21.15.90:0/262541402 >> 172.21.15.90:0/262541402 conn(0x7fe724103830 msgr2=0x7fe724105c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:24:02.116 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:54.378+0000 7fe72aa00700 1 -- 172.21.15.90:0/262541402 shutdown_connections 2024-04-02T14:24:02.116 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:54.378+0000 7fe72aa00700 1 -- 172.21.15.90:0/262541402 wait complete. 2024-04-02T14:24:02.116 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:54.379+0000 7fe72aa00700 1 Processor -- start 2024-04-02T14:24:02.116 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:54.379+0000 7fe72aa00700 1 -- start start 2024-04-02T14:24:02.116 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:54.379+0000 7fe72aa00700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe72407cdc0 0x7fe72419e420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:24:02.117 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:54.379+0000 7fe72aa00700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fe72410e860 con 0x7fe72407cdc0 2024-04-02T14:24:02.117 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:54.379+0000 7fe723e00700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe72407cdc0 0x7fe72419e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:24:02.117 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:54.380+0000 7fe723e00700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe72407cdc0 0x7fe72419e420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33096/0 (socket says 172.21.15.90:33096) 2024-04-02T14:24:02.117 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:54.380+0000 7fe723e00700 1 -- 172.21.15.90:0/2296242491 learned_addr learned my addr 172.21.15.90:0/2296242491 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:24:02.117 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:54.380+0000 7fe723e00700 1 -- 172.21.15.90:0/2296242491 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe72419e960 con 0x7fe72407cdc0 2024-04-02T14:24:02.117 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:54.380+0000 7fe723e00700 1 --2- 172.21.15.90:0/2296242491 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe72407cdc0 0x7fe72419e420 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fe714000c00 tx=0x7fe714011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:24:02.117 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:54.380+0000 7fe720c00700 1 -- 172.21.15.90:0/2296242491 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe714003b20 con 0x7fe72407cdc0 2024-04-02T14:24:02.117 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:54.381+0000 7fe720c00700 1 -- 172.21.15.90:0/2296242491 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fe714003cc0 con 0x7fe72407cdc0 2024-04-02T14:24:02.117 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:54.381+0000 7fe72aa00700 1 -- 172.21.15.90:0/2296242491 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe72419ebf0 con 0x7fe72407cdc0 2024-04-02T14:24:02.117 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:54.381+0000 7fe720c00700 1 -- 172.21.15.90:0/2296242491 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe714024910 con 0x7fe72407cdc0 2024-04-02T14:24:02.117 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:54.381+0000 7fe72aa00700 1 -- 172.21.15.90:0/2296242491 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7241a18e0 con 0x7fe72407cdc0 2024-04-02T14:24:02.117 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:54.382+0000 7fe720c00700 1 -- 172.21.15.90:0/2296242491 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 46558+0+0 (secure 0 0 0) 0x7fe714022020 con 0x7fe72407cdc0 2024-04-02T14:24:02.117 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:54.382+0000 7fe720c00700 1 --2- 172.21.15.90:0/2296242491 >> [v2:172.21.15.90:6800/2372398598,v1:172.21.15.90:6801/2372398598] conn(0x7fe708039660 0x7fe70803bb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:24:02.117 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:54.382+0000 7fe720c00700 1 -- 172.21.15.90:0/2296242491 --> [v2:172.21.15.90:6800/2372398598,v1:172.21.15.90:6801/2372398598] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fe70803c230 con 0x7fe708039660 2024-04-02T14:24:02.117 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:54.382+0000 7fe720c00700 1 -- 172.21.15.90:0/2296242491 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fe714011e00 con 0x7fe72407cdc0 2024-04-02T14:24:02.118 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:54.382+0000 7fe723400700 1 -- 172.21.15.90:0/2296242491 >> [v2:172.21.15.90:6800/2372398598,v1:172.21.15.90:6801/2372398598] conn(0x7fe708039660 msgr2=0x7fe70803bb20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.90:6800/2372398598 2024-04-02T14:24:02.118 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:54.383+0000 7fe723400700 1 --2- 172.21.15.90:0/2296242491 >> [v2:172.21.15.90:6800/2372398598,v1:172.21.15.90:6801/2372398598] conn(0x7fe708039660 0x7fe70803bb20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T14:24:02.118 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:54.584+0000 7fe723400700 1 -- 172.21.15.90:0/2296242491 >> [v2:172.21.15.90:6800/2372398598,v1:172.21.15.90:6801/2372398598] conn(0x7fe708039660 msgr2=0x7fe70803bb20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.90:6800/2372398598 2024-04-02T14:24:02.118 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:54.584+0000 7fe723400700 1 --2- 172.21.15.90:0/2296242491 >> [v2:172.21.15.90:6800/2372398598,v1:172.21.15.90:6801/2372398598] conn(0x7fe708039660 0x7fe70803bb20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T14:24:02.118 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:54.985+0000 7fe723400700 1 -- 172.21.15.90:0/2296242491 >> [v2:172.21.15.90:6800/2372398598,v1:172.21.15.90:6801/2372398598] conn(0x7fe708039660 msgr2=0x7fe70803bb20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.90:6800/2372398598 2024-04-02T14:24:02.118 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:54.985+0000 7fe723400700 1 --2- 172.21.15.90:0/2296242491 >> [v2:172.21.15.90:6800/2372398598,v1:172.21.15.90:6801/2372398598] conn(0x7fe708039660 0x7fe70803bb20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-02T14:24:02.118 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:55.786+0000 7fe723400700 1 -- 172.21.15.90:0/2296242491 >> [v2:172.21.15.90:6800/2372398598,v1:172.21.15.90:6801/2372398598] conn(0x7fe708039660 msgr2=0x7fe70803bb20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.90:6800/2372398598 2024-04-02T14:24:02.118 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:55.786+0000 7fe723400700 1 --2- 172.21.15.90:0/2296242491 >> [v2:172.21.15.90:6800/2372398598,v1:172.21.15.90:6801/2372398598] conn(0x7fe708039660 0x7fe70803bb20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-02T14:24:02.118 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:57.388+0000 7fe723400700 1 -- 172.21.15.90:0/2296242491 >> [v2:172.21.15.90:6800/2372398598,v1:172.21.15.90:6801/2372398598] conn(0x7fe708039660 msgr2=0x7fe70803bb20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.90:6800/2372398598 2024-04-02T14:24:02.118 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:23:57.388+0000 7fe723400700 1 --2- 172.21.15.90:0/2296242491 >> [v2:172.21.15.90:6800/2372398598,v1:172.21.15.90:6801/2372398598] conn(0x7fe708039660 0x7fe70803bb20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-02T14:24:02.118 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:00.367+0000 7fe720c00700 1 -- 172.21.15.90:0/2296242491 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mgrmap(e 6) ==== 46325+0+0 (secure 0 0 0) 0x7fe71401acd0 con 0x7fe72407cdc0 2024-04-02T14:24:02.118 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:00.367+0000 7fe720c00700 1 -- 172.21.15.90:0/2296242491 >> [v2:172.21.15.90:6800/2372398598,v1:172.21.15.90:6801/2372398598] conn(0x7fe708039660 msgr2=0x7fe70803bb20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T14:24:02.118 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:00.367+0000 7fe720c00700 1 --2- 172.21.15.90:0/2296242491 >> [v2:172.21.15.90:6800/2372398598,v1:172.21.15.90:6801/2372398598] conn(0x7fe708039660 0x7fe70803bb20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:24:02.119 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:01.370+0000 7fe720c00700 1 -- 172.21.15.90:0/2296242491 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mgrmap(e 7) ==== 46452+0+0 (secure 0 0 0) 0x7fe71404ede0 con 0x7fe72407cdc0 2024-04-02T14:24:02.119 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:01.371+0000 7fe720c00700 1 --2- 172.21.15.90:0/2296242491 >> [v2:172.21.15.90:6800/129436436,v1:172.21.15.90:6801/129436436] conn(0x7fe70803cf50 0x7fe70803f340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:24:02.119 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:01.371+0000 7fe720c00700 1 -- 172.21.15.90:0/2296242491 --> [v2:172.21.15.90:6800/129436436,v1:172.21.15.90:6801/129436436] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fe714007da0 con 0x7fe70803cf50 2024-04-02T14:24:02.119 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:01.373+0000 7fe723400700 1 --2- 172.21.15.90:0/2296242491 >> [v2:172.21.15.90:6800/129436436,v1:172.21.15.90:6801/129436436] conn(0x7fe70803cf50 0x7fe70803f340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:24:02.119 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:01.376+0000 7fe723400700 1 --2- 172.21.15.90:0/2296242491 >> [v2:172.21.15.90:6800/129436436,v1:172.21.15.90:6801/129436436] conn(0x7fe70803cf50 0x7fe70803f340 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fe710003a10 tx=0x7fe7100092b0 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:24:02.119 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:01.377+0000 7fe720c00700 1 -- 172.21.15.90:0/2296242491 <== mgr.14120 v2:172.21.15.90:6800/129436436 1 ==== command_reply(tid 0: 0 ) ==== 8+0+7774 (secure 0 0 0) 0x7fe714007da0 con 0x7fe70803cf50 2024-04-02T14:24:02.119 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:01.384+0000 7fe72aa00700 1 -- 172.21.15.90:0/2296242491 --> [v2:172.21.15.90:6800/129436436,v1:172.21.15.90:6801/129436436] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7fe724063c20 con 0x7fe70803cf50 2024-04-02T14:24:02.119 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:01.385+0000 7fe720c00700 1 -- 172.21.15.90:0/2296242491 <== mgr.14120 v2:172.21.15.90:6800/129436436 2 ==== command_reply(tid 1: 0 ) ==== 8+0+51 (secure 0 0 0) 0x7fe724063c20 con 0x7fe70803cf50 2024-04-02T14:24:02.119 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:01.385+0000 7fe72aa00700 1 -- 172.21.15.90:0/2296242491 >> [v2:172.21.15.90:6800/129436436,v1:172.21.15.90:6801/129436436] conn(0x7fe70803cf50 msgr2=0x7fe70803f340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:24:02.119 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:01.385+0000 7fe72aa00700 1 --2- 172.21.15.90:0/2296242491 >> [v2:172.21.15.90:6800/129436436,v1:172.21.15.90:6801/129436436] conn(0x7fe70803cf50 0x7fe70803f340 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fe710003a10 tx=0x7fe7100092b0 comp rx=0 tx=0).stop 2024-04-02T14:24:02.119 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:01.385+0000 7fe72aa00700 1 -- 172.21.15.90:0/2296242491 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe72407cdc0 msgr2=0x7fe72419e420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:24:02.119 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:01.385+0000 7fe72aa00700 1 --2- 172.21.15.90:0/2296242491 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe72407cdc0 0x7fe72419e420 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fe714000c00 tx=0x7fe714011c10 comp rx=0 tx=0).stop 2024-04-02T14:24:02.119 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:01.385+0000 7fe72aa00700 1 -- 172.21.15.90:0/2296242491 shutdown_connections 2024-04-02T14:24:02.120 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:01.385+0000 7fe72aa00700 1 --2- 172.21.15.90:0/2296242491 >> [v2:172.21.15.90:6800/129436436,v1:172.21.15.90:6801/129436436] conn(0x7fe70803cf50 0x7fe70803f340 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:24:02.120 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:01.385+0000 7fe72aa00700 1 --2- 172.21.15.90:0/2296242491 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe72407cdc0 0x7fe72419e420 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:24:02.120 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:01.386+0000 7fe72aa00700 1 --2- 172.21.15.90:0/2296242491 >> [v2:172.21.15.90:6800/2372398598,v1:172.21.15.90:6801/2372398598] conn(0x7fe708039660 0x7fe70803bb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:24:02.120 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:01.386+0000 7fe72aa00700 1 -- 172.21.15.90:0/2296242491 >> 172.21.15.90:0/2296242491 conn(0x7fe724103830 msgr2=0x7fe724104340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:24:02.120 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:01.386+0000 7fe72aa00700 1 -- 172.21.15.90:0/2296242491 shutdown_connections 2024-04-02T14:24:02.120 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:01.386+0000 7fe72aa00700 1 -- 172.21.15.90:0/2296242491 wait complete. 2024-04-02T14:24:02.120 INFO:teuthology.orchestra.run.smithi090.stdout:mgr epoch 5 is available 2024-04-02T14:24:02.120 INFO:teuthology.orchestra.run.smithi090.stdout:Setting orchestrator backend to cephadm... 2024-04-02T14:24:02.688 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:24:02 smithi090 ceph-mon[31674]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-02T14:24:02.689 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:24:02 smithi090 ceph-mon[31674]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-02T14:24:02.689 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:24:02 smithi090 ceph-mon[31674]: [02/Apr/2024:14:24:01] ENGINE Bus STARTING 2024-04-02T14:24:02.689 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:24:02 smithi090 ceph-mon[31674]: from='mgr.14120 172.21.15.90:0/2851514084' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T14:24:03.641 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:24:03 smithi090 ceph-mon[31674]: [02/Apr/2024:14:24:01] ENGINE Serving on http://172.21.15.90:8765 2024-04-02T14:24:03.642 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:24:03 smithi090 ceph-mon[31674]: [02/Apr/2024:14:24:01] ENGINE Serving on https://172.21.15.90:7150 2024-04-02T14:24:03.642 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:24:03 smithi090 ceph-mon[31674]: [02/Apr/2024:14:24:01] ENGINE Bus STARTED 2024-04-02T14:24:03.642 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:24:03 smithi090 ceph-mon[31674]: mgrmap e8: x(active, since 2s) 2024-04-02T14:24:03.642 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:24:03 smithi090 ceph-mon[31674]: from='mgr.14120 172.21.15.90:0/2851514084' entity='mgr.x' 2024-04-02T14:24:03.642 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:24:03 smithi090 ceph-mon[31674]: from='mgr.14120 172.21.15.90:0/2851514084' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T14:24:03.656 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:02.715+0000 7f0fd9c00700 1 Processor -- start 2024-04-02T14:24:03.656 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:02.715+0000 7f0fd9c00700 1 -- start start 2024-04-02T14:24:03.656 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:02.715+0000 7f0fd9c00700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0fd407dd30 0x7f0fd407c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:24:03.656 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:02.715+0000 7f0fd9c00700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f0fd407e150 con 0x7f0fd407dd30 2024-04-02T14:24:03.657 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:02.715+0000 7f0fd3e00700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0fd407dd30 0x7f0fd407c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:24:03.657 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:02.716+0000 7f0fd3e00700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0fd407dd30 0x7f0fd407c2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33178/0 (socket says 172.21.15.90:33178) 2024-04-02T14:24:03.657 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:02.716+0000 7f0fd3e00700 1 -- 172.21.15.90:0/3571677922 learned_addr learned my addr 172.21.15.90:0/3571677922 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:24:03.657 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:02.716+0000 7f0fd3e00700 1 -- 172.21.15.90:0/3571677922 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0fd407c830 con 0x7f0fd407dd30 2024-04-02T14:24:03.657 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:02.716+0000 7f0fd3e00700 1 --2- 172.21.15.90:0/3571677922 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0fd407dd30 0x7f0fd407c2f0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f0fc40098a0 tx=0x7f0fc4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb08f60ea684a0f9 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:24:03.657 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:02.716+0000 7f0fd3400700 1 -- 172.21.15.90:0/3571677922 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0fc4004160 con 0x7f0fd407dd30 2024-04-02T14:24:03.657 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:02.716+0000 7f0fd3400700 1 -- 172.21.15.90:0/3571677922 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f0fc4004300 con 0x7f0fd407dd30 2024-04-02T14:24:03.657 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:02.716+0000 7f0fd3400700 1 -- 172.21.15.90:0/3571677922 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0fc4011b50 con 0x7f0fd407dd30 2024-04-02T14:24:03.657 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:02.717+0000 7f0fd9c00700 1 -- 172.21.15.90:0/3571677922 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0fd407dd30 msgr2=0x7f0fd407c2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:24:03.657 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:02.717+0000 7f0fd9c00700 1 --2- 172.21.15.90:0/3571677922 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0fd407dd30 0x7f0fd407c2f0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f0fc40098a0 tx=0x7f0fc4009bb0 comp rx=0 tx=0).stop 2024-04-02T14:24:03.657 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:02.717+0000 7f0fd9c00700 1 -- 172.21.15.90:0/3571677922 shutdown_connections 2024-04-02T14:24:03.657 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:02.717+0000 7f0fd9c00700 1 --2- 172.21.15.90:0/3571677922 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0fd407dd30 0x7f0fd407c2f0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:24:03.657 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:02.717+0000 7f0fd9c00700 1 -- 172.21.15.90:0/3571677922 >> 172.21.15.90:0/3571677922 conn(0x7f0fd4103740 msgr2=0x7f0fd4105b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:24:03.657 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:02.717+0000 7f0fd9c00700 1 -- 172.21.15.90:0/3571677922 shutdown_connections 2024-04-02T14:24:03.658 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:02.717+0000 7f0fd9c00700 1 -- 172.21.15.90:0/3571677922 wait complete. 2024-04-02T14:24:03.658 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:02.718+0000 7f0fd9c00700 1 Processor -- start 2024-04-02T14:24:03.658 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:02.718+0000 7f0fd9c00700 1 -- start start 2024-04-02T14:24:03.658 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:02.719+0000 7f0fd9c00700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0fd407dd30 0x7f0fd419e1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:24:03.658 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:02.719+0000 7f0fd9c00700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f0fd410e570 con 0x7f0fd407dd30 2024-04-02T14:24:03.658 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:02.719+0000 7f0fd3e00700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0fd407dd30 0x7f0fd419e1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:24:03.658 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:02.719+0000 7f0fd3e00700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0fd407dd30 0x7f0fd419e1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33180/0 (socket says 172.21.15.90:33180) 2024-04-02T14:24:03.658 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:02.719+0000 7f0fd3e00700 1 -- 172.21.15.90:0/3641309620 learned_addr learned my addr 172.21.15.90:0/3641309620 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:24:03.658 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:02.719+0000 7f0fd3e00700 1 -- 172.21.15.90:0/3641309620 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0fd419e710 con 0x7f0fd407dd30 2024-04-02T14:24:03.658 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:02.720+0000 7f0fd3e00700 1 --2- 172.21.15.90:0/3641309620 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0fd407dd30 0x7f0fd419e1d0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f0fc4011b60 tx=0x7f0fc4011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:24:03.658 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:02.720+0000 7f0fd1600700 1 -- 172.21.15.90:0/3641309620 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0fc4003b20 con 0x7f0fd407dd30 2024-04-02T14:24:03.658 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:02.720+0000 7f0fd1600700 1 -- 172.21.15.90:0/3641309620 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f0fc4003cc0 con 0x7f0fd407dd30 2024-04-02T14:24:03.658 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:02.720+0000 7f0fd9c00700 1 -- 172.21.15.90:0/3641309620 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0fd419e9a0 con 0x7f0fd407dd30 2024-04-02T14:24:03.658 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:02.720+0000 7f0fd1600700 1 -- 172.21.15.90:0/3641309620 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0fc4024840 con 0x7f0fd407dd30 2024-04-02T14:24:03.658 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:02.720+0000 7f0fd9c00700 1 -- 172.21.15.90:0/3641309620 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0fd41a1690 con 0x7f0fd407dd30 2024-04-02T14:24:03.659 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:02.721+0000 7f0fd1600700 1 -- 172.21.15.90:0/3641309620 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f0fc4022020 con 0x7f0fd407dd30 2024-04-02T14:24:03.659 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:02.721+0000 7f0fd9c00700 1 -- 172.21.15.90:0/3641309620 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0fd4063c20 con 0x7f0fd407dd30 2024-04-02T14:24:03.659 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:02.722+0000 7f0fd1600700 1 --2- 172.21.15.90:0/3641309620 >> [v2:172.21.15.90:6800/129436436,v1:172.21.15.90:6801/129436436] conn(0x7f0fb8039600 0x7f0fb803bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:24:03.659 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:02.722+0000 7f0fd1600700 1 -- 172.21.15.90:0/3641309620 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f0fc4011e00 con 0x7f0fd407dd30 2024-04-02T14:24:03.659 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:02.722+0000 7f0fcb400700 1 --2- 172.21.15.90:0/3641309620 >> [v2:172.21.15.90:6800/129436436,v1:172.21.15.90:6801/129436436] conn(0x7f0fb8039600 0x7f0fb803bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:24:03.659 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:02.722+0000 7f0fcb400700 1 --2- 172.21.15.90:0/3641309620 >> [v2:172.21.15.90:6800/129436436,v1:172.21.15.90:6801/129436436] conn(0x7f0fb8039600 0x7f0fb803bac0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f0fbc006fd0 tx=0x7f0fbc006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:24:03.659 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:02.728+0000 7f0fd1600700 1 -- 172.21.15.90:0/3641309620 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7f0fc404d2d0 con 0x7f0fd407dd30 2024-04-02T14:24:03.659 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:02.969+0000 7f0fd9c00700 1 -- 172.21.15.90:0/3641309620 --> [v2:172.21.15.90:6800/129436436,v1:172.21.15.90:6801/129436436] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) -- 0x7f0fd4104990 con 0x7f0fb8039600 2024-04-02T14:24:03.659 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:02.982+0000 7f0fd1600700 1 -- 172.21.15.90:0/3641309620 <== mgr.14120 v2:172.21.15.90:6800/129436436 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f0fd4104990 con 0x7f0fb8039600 2024-04-02T14:24:03.659 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:02.985+0000 7f0fd9c00700 1 -- 172.21.15.90:0/3641309620 >> [v2:172.21.15.90:6800/129436436,v1:172.21.15.90:6801/129436436] conn(0x7f0fb8039600 msgr2=0x7f0fb803bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:24:03.659 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:02.985+0000 7f0fd9c00700 1 --2- 172.21.15.90:0/3641309620 >> [v2:172.21.15.90:6800/129436436,v1:172.21.15.90:6801/129436436] conn(0x7f0fb8039600 0x7f0fb803bac0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f0fbc006fd0 tx=0x7f0fbc006e40 comp rx=0 tx=0).stop 2024-04-02T14:24:03.659 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:02.985+0000 7f0fd9c00700 1 -- 172.21.15.90:0/3641309620 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0fd407dd30 msgr2=0x7f0fd419e1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:24:03.659 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:02.985+0000 7f0fd9c00700 1 --2- 172.21.15.90:0/3641309620 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0fd407dd30 0x7f0fd419e1d0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f0fc4011b60 tx=0x7f0fc4011c40 comp rx=0 tx=0).stop 2024-04-02T14:24:03.659 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:02.985+0000 7f0fd9c00700 1 -- 172.21.15.90:0/3641309620 shutdown_connections 2024-04-02T14:24:03.659 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:02.985+0000 7f0fd9c00700 1 --2- 172.21.15.90:0/3641309620 >> [v2:172.21.15.90:6800/129436436,v1:172.21.15.90:6801/129436436] conn(0x7f0fb8039600 0x7f0fb803bac0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:24:03.660 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:02.985+0000 7f0fd9c00700 1 --2- 172.21.15.90:0/3641309620 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0fd407dd30 0x7f0fd419e1d0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:24:03.660 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:02.985+0000 7f0fd9c00700 1 -- 172.21.15.90:0/3641309620 >> 172.21.15.90:0/3641309620 conn(0x7f0fd4103740 msgr2=0x7f0fd4104280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:24:03.660 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:02.985+0000 7f0fd9c00700 1 -- 172.21.15.90:0/3641309620 shutdown_connections 2024-04-02T14:24:03.660 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:02.985+0000 7f0fd9c00700 1 -- 172.21.15.90:0/3641309620 wait complete. 2024-04-02T14:24:04.439 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:24:04 smithi090 ceph-mon[31674]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T14:24:05.198 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout value unchanged 2024-04-02T14:24:05.198 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:04.265+0000 7f5a65000700 1 Processor -- start 2024-04-02T14:24:05.198 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:04.265+0000 7f5a65000700 1 -- start start 2024-04-02T14:24:05.198 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:04.265+0000 7f5a65000700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5a60106b30 0x7f5a60106f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:24:05.198 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:04.266+0000 7f5a65000700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f5a60075e90 con 0x7f5a60106b30 2024-04-02T14:24:05.203 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:04.266+0000 7f5a5de00700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5a60106b30 0x7f5a60106f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:24:05.203 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:04.266+0000 7f5a5de00700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5a60106b30 0x7f5a60106f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38900/0 (socket says 172.21.15.90:38900) 2024-04-02T14:24:05.203 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:04.266+0000 7f5a5de00700 1 -- 172.21.15.90:0/1132240499 learned_addr learned my addr 172.21.15.90:0/1132240499 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:24:05.204 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:04.266+0000 7f5a5de00700 1 -- 172.21.15.90:0/1132240499 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a60107490 con 0x7f5a60106b30 2024-04-02T14:24:05.204 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:04.267+0000 7f5a5de00700 1 --2- 172.21.15.90:0/1132240499 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5a60106b30 0x7f5a60106f50 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f5a500098a0 tx=0x7f5a50009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=687798b165b6ab36 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:24:05.204 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:04.267+0000 7f5a5ca00700 1 -- 172.21.15.90:0/1132240499 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5a50004160 con 0x7f5a60106b30 2024-04-02T14:24:05.204 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:04.267+0000 7f5a5ca00700 1 -- 172.21.15.90:0/1132240499 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f5a50004300 con 0x7f5a60106b30 2024-04-02T14:24:05.204 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:04.267+0000 7f5a5ca00700 1 -- 172.21.15.90:0/1132240499 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5a50011b50 con 0x7f5a60106b30 2024-04-02T14:24:05.204 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:04.267+0000 7f5a65000700 1 -- 172.21.15.90:0/1132240499 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5a60106b30 msgr2=0x7f5a60106f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:24:05.204 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:04.268+0000 7f5a65000700 1 --2- 172.21.15.90:0/1132240499 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5a60106b30 0x7f5a60106f50 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f5a500098a0 tx=0x7f5a50009bb0 comp rx=0 tx=0).stop 2024-04-02T14:24:05.204 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:04.268+0000 7f5a65000700 1 -- 172.21.15.90:0/1132240499 shutdown_connections 2024-04-02T14:24:05.204 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:04.268+0000 7f5a65000700 1 --2- 172.21.15.90:0/1132240499 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5a60106b30 0x7f5a60106f50 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:24:05.204 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:04.268+0000 7f5a65000700 1 -- 172.21.15.90:0/1132240499 >> 172.21.15.90:0/1132240499 conn(0x7f5a60102730 msgr2=0x7f5a60104b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:24:05.204 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:04.268+0000 7f5a65000700 1 -- 172.21.15.90:0/1132240499 shutdown_connections 2024-04-02T14:24:05.204 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:04.268+0000 7f5a65000700 1 -- 172.21.15.90:0/1132240499 wait complete. 2024-04-02T14:24:05.204 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:04.269+0000 7f5a65000700 1 Processor -- start 2024-04-02T14:24:05.205 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:04.269+0000 7f5a65000700 1 -- start start 2024-04-02T14:24:05.205 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:04.269+0000 7f5a65000700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5a60106b30 0x7f5a6019e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:24:05.205 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:04.269+0000 7f5a65000700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f5a60108520 con 0x7f5a60106b30 2024-04-02T14:24:05.205 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:04.269+0000 7f5a5de00700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5a60106b30 0x7f5a6019e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:24:05.205 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:04.269+0000 7f5a5de00700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5a60106b30 0x7f5a6019e470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38914/0 (socket says 172.21.15.90:38914) 2024-04-02T14:24:05.205 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:04.269+0000 7f5a5de00700 1 -- 172.21.15.90:0/160137346 learned_addr learned my addr 172.21.15.90:0/160137346 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:24:05.205 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:04.270+0000 7f5a5de00700 1 -- 172.21.15.90:0/160137346 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a6019e9b0 con 0x7f5a60106b30 2024-04-02T14:24:05.205 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:04.270+0000 7f5a5de00700 1 --2- 172.21.15.90:0/160137346 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5a60106b30 0x7f5a6019e470 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f5a50011b60 tx=0x7f5a50011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:24:05.205 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:04.270+0000 7f5a56a00700 1 -- 172.21.15.90:0/160137346 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5a50003970 con 0x7f5a60106b30 2024-04-02T14:24:05.205 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:04.270+0000 7f5a56a00700 1 -- 172.21.15.90:0/160137346 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f5a50003b10 con 0x7f5a60106b30 2024-04-02T14:24:05.205 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:04.270+0000 7f5a65000700 1 -- 172.21.15.90:0/160137346 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a6019ec40 con 0x7f5a60106b30 2024-04-02T14:24:05.205 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:04.270+0000 7f5a56a00700 1 -- 172.21.15.90:0/160137346 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5a50024610 con 0x7f5a60106b30 2024-04-02T14:24:05.205 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:04.271+0000 7f5a65000700 1 -- 172.21.15.90:0/160137346 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a601a1930 con 0x7f5a60106b30 2024-04-02T14:24:05.205 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:04.272+0000 7f5a56a00700 1 -- 172.21.15.90:0/160137346 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f5a50024910 con 0x7f5a60106b30 2024-04-02T14:24:05.206 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:04.272+0000 7f5a65000700 1 -- 172.21.15.90:0/160137346 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5a60063c20 con 0x7f5a60106b30 2024-04-02T14:24:05.206 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:04.272+0000 7f5a56a00700 1 --2- 172.21.15.90:0/160137346 >> [v2:172.21.15.90:6800/129436436,v1:172.21.15.90:6801/129436436] conn(0x7f5a440395b0 0x7f5a4403ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:24:05.206 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:04.272+0000 7f5a56a00700 1 -- 172.21.15.90:0/160137346 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f5a50011e00 con 0x7f5a60106b30 2024-04-02T14:24:05.206 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:04.272+0000 7f5a5d400700 1 --2- 172.21.15.90:0/160137346 >> [v2:172.21.15.90:6800/129436436,v1:172.21.15.90:6801/129436436] conn(0x7f5a440395b0 0x7f5a4403ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:24:05.206 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:04.273+0000 7f5a5d400700 1 --2- 172.21.15.90:0/160137346 >> [v2:172.21.15.90:6800/129436436,v1:172.21.15.90:6801/129436436] conn(0x7f5a440395b0 0x7f5a4403ba70 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f5a4c006fd0 tx=0x7f5a4c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:24:05.206 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:04.278+0000 7f5a56a00700 1 -- 172.21.15.90:0/160137346 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7f5a500172b0 con 0x7f5a60106b30 2024-04-02T14:24:05.206 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:04.505+0000 7f5a65000700 1 -- 172.21.15.90:0/160137346 --> [v2:172.21.15.90:6800/129436436,v1:172.21.15.90:6801/129436436] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) -- 0x7f5a600008d0 con 0x7f5a440395b0 2024-04-02T14:24:05.206 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:04.507+0000 7f5a56a00700 1 -- 172.21.15.90:0/160137346 <== mgr.14120 v2:172.21.15.90:6800/129436436 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+16 (secure 0 0 0) 0x7f5a600008d0 con 0x7f5a440395b0 2024-04-02T14:24:05.206 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:04.510+0000 7f5a65000700 1 -- 172.21.15.90:0/160137346 >> [v2:172.21.15.90:6800/129436436,v1:172.21.15.90:6801/129436436] conn(0x7f5a440395b0 msgr2=0x7f5a4403ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:24:05.206 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:04.510+0000 7f5a65000700 1 --2- 172.21.15.90:0/160137346 >> [v2:172.21.15.90:6800/129436436,v1:172.21.15.90:6801/129436436] conn(0x7f5a440395b0 0x7f5a4403ba70 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f5a4c006fd0 tx=0x7f5a4c006e40 comp rx=0 tx=0).stop 2024-04-02T14:24:05.206 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:04.510+0000 7f5a65000700 1 -- 172.21.15.90:0/160137346 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5a60106b30 msgr2=0x7f5a6019e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:24:05.206 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:04.510+0000 7f5a65000700 1 --2- 172.21.15.90:0/160137346 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5a60106b30 0x7f5a6019e470 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f5a50011b60 tx=0x7f5a50011c40 comp rx=0 tx=0).stop 2024-04-02T14:24:05.206 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:04.510+0000 7f5a65000700 1 -- 172.21.15.90:0/160137346 shutdown_connections 2024-04-02T14:24:05.207 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:04.510+0000 7f5a65000700 1 --2- 172.21.15.90:0/160137346 >> [v2:172.21.15.90:6800/129436436,v1:172.21.15.90:6801/129436436] conn(0x7f5a440395b0 0x7f5a4403ba70 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:24:05.207 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:04.510+0000 7f5a65000700 1 --2- 172.21.15.90:0/160137346 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5a60106b30 0x7f5a6019e470 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:24:05.207 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:04.510+0000 7f5a65000700 1 -- 172.21.15.90:0/160137346 >> 172.21.15.90:0/160137346 conn(0x7f5a60102730 msgr2=0x7f5a601950e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:24:05.207 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:04.510+0000 7f5a65000700 1 -- 172.21.15.90:0/160137346 shutdown_connections 2024-04-02T14:24:05.207 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:04.510+0000 7f5a65000700 1 -- 172.21.15.90:0/160137346 wait complete. 2024-04-02T14:24:05.207 INFO:teuthology.orchestra.run.smithi090.stdout:Generating ssh key... 2024-04-02T14:24:05.483 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:24:05 smithi090 ceph-mon[31674]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T14:24:06.440 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 02 14:24:06 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-x[31910]: Generating public/private rsa key pair. 2024-04-02T14:24:06.440 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 02 14:24:06 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-x[31910]: Your identification has been saved in /tmp/tmp1vf7mekh/key. 2024-04-02T14:24:06.440 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 02 14:24:06 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-x[31910]: Your public key has been saved in /tmp/tmp1vf7mekh/key.pub. 2024-04-02T14:24:06.440 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 02 14:24:06 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-x[31910]: The key fingerprint is: 2024-04-02T14:24:06.440 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 02 14:24:06 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-x[31910]: SHA256:iYD1eWDpo9OIXHhxCyU1C+NVTdqstC613HYDlrRk4Ig ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4 2024-04-02T14:24:06.440 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 02 14:24:06 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-x[31910]: The key's randomart image is: 2024-04-02T14:24:06.440 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 02 14:24:06 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-x[31910]: +---[RSA 3072]----+ 2024-04-02T14:24:06.440 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 02 14:24:06 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-x[31910]: | =oBooo. | 2024-04-02T14:24:06.440 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 02 14:24:06 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-x[31910]: | +oOoB =. | 2024-04-02T14:24:06.440 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 02 14:24:06 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-x[31910]: | ..E==.= * | 2024-04-02T14:24:06.440 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 02 14:24:06 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-x[31910]: | . o.++ B o | 2024-04-02T14:24:06.441 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 02 14:24:06 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-x[31910]: | . + +..S = | 2024-04-02T14:24:06.441 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 02 14:24:06 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-x[31910]: | o + .+ + . | 2024-04-02T14:24:06.441 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 02 14:24:06 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-x[31910]: | .. + o o | 2024-04-02T14:24:06.441 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 02 14:24:06 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-x[31910]: | . . . . | 2024-04-02T14:24:06.441 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 02 14:24:06 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-x[31910]: | | 2024-04-02T14:24:06.441 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 02 14:24:06 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-x[31910]: +----[SHA256]-----+ 2024-04-02T14:24:07.040 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:05.766+0000 7ffb1ac00700 1 Processor -- start 2024-04-02T14:24:07.045 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:05.766+0000 7ffb1ac00700 1 -- start start 2024-04-02T14:24:07.045 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:05.766+0000 7ffb1ac00700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffb1407cd80 0x7ffb1407b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:24:07.046 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:05.766+0000 7ffb1ac00700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7ffb1407b7b0 con 0x7ffb1407cd80 2024-04-02T14:24:07.046 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:05.767+0000 7ffb13e00700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffb1407cd80 0x7ffb1407b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:24:07.046 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:05.767+0000 7ffb13e00700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffb1407cd80 0x7ffb1407b1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38926/0 (socket says 172.21.15.90:38926) 2024-04-02T14:24:07.046 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:05.767+0000 7ffb13e00700 1 -- 172.21.15.90:0/169651134 learned_addr learned my addr 172.21.15.90:0/169651134 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:24:07.046 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:05.768+0000 7ffb13e00700 1 -- 172.21.15.90:0/169651134 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb1407b930 con 0x7ffb1407cd80 2024-04-02T14:24:07.046 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:05.768+0000 7ffb13e00700 1 --2- 172.21.15.90:0/169651134 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffb1407cd80 0x7ffb1407b1e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7ffb04009e80 tx=0x7ffb0400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f36f3cc24a48049 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:24:07.046 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:05.768+0000 7ffb13400700 1 -- 172.21.15.90:0/169651134 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ffb04017070 con 0x7ffb1407cd80 2024-04-02T14:24:07.046 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:05.769+0000 7ffb13400700 1 -- 172.21.15.90:0/169651134 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7ffb0400b8c0 con 0x7ffb1407cd80 2024-04-02T14:24:07.046 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:05.769+0000 7ffb13400700 1 -- 172.21.15.90:0/169651134 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ffb040045c0 con 0x7ffb1407cd80 2024-04-02T14:24:07.046 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:05.769+0000 7ffb1ac00700 1 -- 172.21.15.90:0/169651134 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffb1407cd80 msgr2=0x7ffb1407b1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:24:07.046 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:05.769+0000 7ffb1ac00700 1 --2- 172.21.15.90:0/169651134 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffb1407cd80 0x7ffb1407b1e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7ffb04009e80 tx=0x7ffb0400b330 comp rx=0 tx=0).stop 2024-04-02T14:24:07.046 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:05.769+0000 7ffb1ac00700 1 -- 172.21.15.90:0/169651134 shutdown_connections 2024-04-02T14:24:07.047 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:05.769+0000 7ffb1ac00700 1 --2- 172.21.15.90:0/169651134 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffb1407cd80 0x7ffb1407b1e0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:24:07.047 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:05.769+0000 7ffb1ac00700 1 -- 172.21.15.90:0/169651134 >> 172.21.15.90:0/169651134 conn(0x7ffb14103870 msgr2=0x7ffb14105c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:24:07.047 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:05.769+0000 7ffb1ac00700 1 -- 172.21.15.90:0/169651134 shutdown_connections 2024-04-02T14:24:07.047 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:05.770+0000 7ffb1ac00700 1 -- 172.21.15.90:0/169651134 wait complete. 2024-04-02T14:24:07.047 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:05.770+0000 7ffb1ac00700 1 Processor -- start 2024-04-02T14:24:07.047 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:05.771+0000 7ffb1ac00700 1 -- start start 2024-04-02T14:24:07.047 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:05.771+0000 7ffb1ac00700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffb1407cd80 0x7ffb1419e370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:24:07.047 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:05.771+0000 7ffb1ac00700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7ffb1410e8a0 con 0x7ffb1407cd80 2024-04-02T14:24:07.047 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:05.771+0000 7ffb13e00700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffb1407cd80 0x7ffb1419e370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:24:07.047 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:05.771+0000 7ffb13e00700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffb1407cd80 0x7ffb1419e370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38938/0 (socket says 172.21.15.90:38938) 2024-04-02T14:24:07.047 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:05.771+0000 7ffb13e00700 1 -- 172.21.15.90:0/2551958438 learned_addr learned my addr 172.21.15.90:0/2551958438 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:24:07.047 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:05.772+0000 7ffb13e00700 1 -- 172.21.15.90:0/2551958438 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb1419e8b0 con 0x7ffb1407cd80 2024-04-02T14:24:07.047 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:05.772+0000 7ffb13e00700 1 --2- 172.21.15.90:0/2551958438 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffb1407cd80 0x7ffb1419e370 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7ffb04004350 tx=0x7ffb0400bd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:24:07.047 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:05.772+0000 7ffb11600700 1 -- 172.21.15.90:0/2551958438 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ffb04020020 con 0x7ffb1407cd80 2024-04-02T14:24:07.048 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:05.772+0000 7ffb1ac00700 1 -- 172.21.15.90:0/2551958438 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffb1419eb40 con 0x7ffb1407cd80 2024-04-02T14:24:07.048 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:05.772+0000 7ffb11600700 1 -- 172.21.15.90:0/2551958438 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7ffb04003c10 con 0x7ffb1407cd80 2024-04-02T14:24:07.048 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:05.772+0000 7ffb11600700 1 -- 172.21.15.90:0/2551958438 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ffb04017040 con 0x7ffb1407cd80 2024-04-02T14:24:07.048 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:05.772+0000 7ffb1ac00700 1 -- 172.21.15.90:0/2551958438 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffb141a1830 con 0x7ffb1407cd80 2024-04-02T14:24:07.048 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:05.774+0000 7ffb11600700 1 -- 172.21.15.90:0/2551958438 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7ffb04023020 con 0x7ffb1407cd80 2024-04-02T14:24:07.048 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:05.774+0000 7ffb1ac00700 1 -- 172.21.15.90:0/2551958438 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffb14063c20 con 0x7ffb1407cd80 2024-04-02T14:24:07.048 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:05.774+0000 7ffb11600700 1 --2- 172.21.15.90:0/2551958438 >> [v2:172.21.15.90:6800/129436436,v1:172.21.15.90:6801/129436436] conn(0x7ffaf80395b0 0x7ffaf803ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:24:07.048 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:05.774+0000 7ffb11600700 1 -- 172.21.15.90:0/2551958438 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7ffb0400f050 con 0x7ffb1407cd80 2024-04-02T14:24:07.048 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:05.775+0000 7ffb0b400700 1 --2- 172.21.15.90:0/2551958438 >> [v2:172.21.15.90:6800/129436436,v1:172.21.15.90:6801/129436436] conn(0x7ffaf80395b0 0x7ffaf803ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:24:07.048 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:05.775+0000 7ffb0b400700 1 --2- 172.21.15.90:0/2551958438 >> [v2:172.21.15.90:6800/129436436,v1:172.21.15.90:6801/129436436] conn(0x7ffaf80395b0 0x7ffaf803ba70 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7ffafc006fd0 tx=0x7ffafc006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:24:07.048 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:05.780+0000 7ffb11600700 1 -- 172.21.15.90:0/2551958438 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7ffb0402a020 con 0x7ffb1407cd80 2024-04-02T14:24:07.048 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:06.009+0000 7ffb1ac00700 1 -- 172.21.15.90:0/2551958438 --> [v2:172.21.15.90:6800/129436436,v1:172.21.15.90:6801/129436436] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) -- 0x7ffb14104ae0 con 0x7ffaf80395b0 2024-04-02T14:24:07.048 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:06.190+0000 7ffb11600700 1 -- 172.21.15.90:0/2551958438 <== mgr.14120 v2:172.21.15.90:6800/129436436 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7ffb14104ae0 con 0x7ffaf80395b0 2024-04-02T14:24:07.049 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:06.193+0000 7ffb1ac00700 1 -- 172.21.15.90:0/2551958438 >> [v2:172.21.15.90:6800/129436436,v1:172.21.15.90:6801/129436436] conn(0x7ffaf80395b0 msgr2=0x7ffaf803ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:24:07.049 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:06.194+0000 7ffb1ac00700 1 --2- 172.21.15.90:0/2551958438 >> [v2:172.21.15.90:6800/129436436,v1:172.21.15.90:6801/129436436] conn(0x7ffaf80395b0 0x7ffaf803ba70 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7ffafc006fd0 tx=0x7ffafc006e40 comp rx=0 tx=0).stop 2024-04-02T14:24:07.049 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:06.194+0000 7ffb1ac00700 1 -- 172.21.15.90:0/2551958438 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffb1407cd80 msgr2=0x7ffb1419e370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:24:07.049 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:06.194+0000 7ffb1ac00700 1 --2- 172.21.15.90:0/2551958438 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffb1407cd80 0x7ffb1419e370 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7ffb04004350 tx=0x7ffb0400bd50 comp rx=0 tx=0).stop 2024-04-02T14:24:07.049 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:06.194+0000 7ffb1ac00700 1 -- 172.21.15.90:0/2551958438 shutdown_connections 2024-04-02T14:24:07.049 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:06.194+0000 7ffb1ac00700 1 --2- 172.21.15.90:0/2551958438 >> [v2:172.21.15.90:6800/129436436,v1:172.21.15.90:6801/129436436] conn(0x7ffaf80395b0 0x7ffaf803ba70 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:24:07.049 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:06.194+0000 7ffb1ac00700 1 --2- 172.21.15.90:0/2551958438 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffb1407cd80 0x7ffb1419e370 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:24:07.049 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:06.194+0000 7ffb1ac00700 1 -- 172.21.15.90:0/2551958438 >> 172.21.15.90:0/2551958438 conn(0x7ffb14103870 msgr2=0x7ffb141043d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:24:07.049 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:06.194+0000 7ffb1ac00700 1 -- 172.21.15.90:0/2551958438 shutdown_connections 2024-04-02T14:24:07.049 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:06.194+0000 7ffb1ac00700 1 -- 172.21.15.90:0/2551958438 wait complete. 2024-04-02T14:24:07.318 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:24:07 smithi090 ceph-mon[31674]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T14:24:07.319 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:24:07 smithi090 ceph-mon[31674]: Generating ssh key... 2024-04-02T14:24:07.319 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:24:07 smithi090 ceph-mon[31674]: from='mgr.14120 172.21.15.90:0/2851514084' entity='mgr.x' 2024-04-02T14:24:07.319 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:24:07 smithi090 ceph-mon[31674]: from='mgr.14120 172.21.15.90:0/2851514084' entity='mgr.x' 2024-04-02T14:24:08.559 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDfeF+PzYQUGcT56oUfI9bNXcaihfx5OtfE5wlPtWw1AeDM86UoN1Td2/VJOKnCrDQWsuVknsL6tN1Tau5W2YeLp/dCgscer9myhNXaiAhn69CGXpoO4Azkk5Pdt/jNx8IrRWrGHl44ST+fQSNzFvVV8bojOzVNRQbz+WRqXgur3xPHsw10ZLOgkj61iBdgWzVYwiI1k6nPJwaT0CjZnVT+cT7z6DZuSmgS3XKig0ojAyuvjm4stw4vwXSt652fqfGJO5nsFyMaP/cQpMlx0f53Pknp7zioqMfzwd6+HYgBaYayAiVVzVJWTFknS+R1HFPTMRbAR4+Mv7PZLFY2XJY4Q8scpVljbHw0J88JT6oE85afTE6COmCLbq/gnQtle/bfdwSXxptOKUcHZemCWzvdLKRa1WTQeXGug2lfEZcyhDSKh8etKm1pdO39tiqQCNtYwarJx1rYBmvY7/DYnvoCuTMaOeDU26QkYJHbWstOV/L+L9icIIimHk+LsR9T6is= ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4 2024-04-02T14:24:08.560 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:07.600+0000 7fca24200700 1 Processor -- start 2024-04-02T14:24:08.560 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:07.600+0000 7fca24200700 1 -- start start 2024-04-02T14:24:08.560 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:07.601+0000 7fca24200700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fca1c108280 0x7fca1c1086a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:24:08.560 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:07.601+0000 7fca24200700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fca1c108be0 con 0x7fca1c108280 2024-04-02T14:24:08.560 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:07.601+0000 7fca21800700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fca1c108280 0x7fca1c1086a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:24:08.560 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:07.601+0000 7fca21800700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fca1c108280 0x7fca1c1086a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38946/0 (socket says 172.21.15.90:38946) 2024-04-02T14:24:08.561 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:07.601+0000 7fca21800700 1 -- 172.21.15.90:0/2308544976 learned_addr learned my addr 172.21.15.90:0/2308544976 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:24:08.561 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:07.602+0000 7fca21800700 1 -- 172.21.15.90:0/2308544976 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca1c108d60 con 0x7fca1c108280 2024-04-02T14:24:08.561 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:07.602+0000 7fca21800700 1 --2- 172.21.15.90:0/2308544976 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fca1c108280 0x7fca1c1086a0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fca100098a0 tx=0x7fca10009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=868865732f5981c8 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:24:08.561 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:07.602+0000 7fca1be00700 1 -- 172.21.15.90:0/2308544976 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fca10004160 con 0x7fca1c108280 2024-04-02T14:24:08.561 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:07.602+0000 7fca1be00700 1 -- 172.21.15.90:0/2308544976 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fca10004300 con 0x7fca1c108280 2024-04-02T14:24:08.561 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:07.602+0000 7fca1be00700 1 -- 172.21.15.90:0/2308544976 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fca10011b50 con 0x7fca1c108280 2024-04-02T14:24:08.561 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:07.603+0000 7fca24200700 1 -- 172.21.15.90:0/2308544976 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fca1c108280 msgr2=0x7fca1c1086a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:24:08.561 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:07.603+0000 7fca24200700 1 --2- 172.21.15.90:0/2308544976 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fca1c108280 0x7fca1c1086a0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fca100098a0 tx=0x7fca10009bb0 comp rx=0 tx=0).stop 2024-04-02T14:24:08.561 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:07.603+0000 7fca24200700 1 -- 172.21.15.90:0/2308544976 shutdown_connections 2024-04-02T14:24:08.562 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:07.603+0000 7fca24200700 1 --2- 172.21.15.90:0/2308544976 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fca1c108280 0x7fca1c1086a0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:24:08.562 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:07.603+0000 7fca24200700 1 -- 172.21.15.90:0/2308544976 >> 172.21.15.90:0/2308544976 conn(0x7fca1c103900 msgr2=0x7fca1c105d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:24:08.562 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:07.603+0000 7fca24200700 1 -- 172.21.15.90:0/2308544976 shutdown_connections 2024-04-02T14:24:08.562 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:07.603+0000 7fca24200700 1 -- 172.21.15.90:0/2308544976 wait complete. 2024-04-02T14:24:08.562 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:07.604+0000 7fca24200700 1 Processor -- start 2024-04-02T14:24:08.562 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:07.604+0000 7fca24200700 1 -- start start 2024-04-02T14:24:08.562 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:07.605+0000 7fca24200700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fca1c108280 0x7fca1c19c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:24:08.562 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:07.605+0000 7fca24200700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fca1c10c6f0 con 0x7fca1c108280 2024-04-02T14:24:08.562 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:07.605+0000 7fca21800700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fca1c108280 0x7fca1c19c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:24:08.562 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:07.605+0000 7fca21800700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fca1c108280 0x7fca1c19c4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38954/0 (socket says 172.21.15.90:38954) 2024-04-02T14:24:08.563 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:07.605+0000 7fca21800700 1 -- 172.21.15.90:0/2416755095 learned_addr learned my addr 172.21.15.90:0/2416755095 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:24:08.563 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:07.605+0000 7fca21800700 1 -- 172.21.15.90:0/2416755095 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca1c19ca00 con 0x7fca1c108280 2024-04-02T14:24:08.563 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:07.606+0000 7fca21800700 1 --2- 172.21.15.90:0/2416755095 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fca1c108280 0x7fca1c19c4c0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fca10011b60 tx=0x7fca10011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:24:08.563 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:07.606+0000 7fca1a000700 1 -- 172.21.15.90:0/2416755095 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fca10003b20 con 0x7fca1c108280 2024-04-02T14:24:08.563 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:07.606+0000 7fca1a000700 1 -- 172.21.15.90:0/2416755095 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fca10003cc0 con 0x7fca1c108280 2024-04-02T14:24:08.563 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:07.606+0000 7fca24200700 1 -- 172.21.15.90:0/2416755095 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fca1c19cc90 con 0x7fca1c108280 2024-04-02T14:24:08.563 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:07.606+0000 7fca1a000700 1 -- 172.21.15.90:0/2416755095 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fca10024840 con 0x7fca1c108280 2024-04-02T14:24:08.563 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:07.606+0000 7fca24200700 1 -- 172.21.15.90:0/2416755095 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fca1c19f980 con 0x7fca1c108280 2024-04-02T14:24:08.563 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:07.607+0000 7fca1a000700 1 -- 172.21.15.90:0/2416755095 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7fca10022020 con 0x7fca1c108280 2024-04-02T14:24:08.563 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:07.607+0000 7fca24200700 1 -- 172.21.15.90:0/2416755095 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fca1c063c20 con 0x7fca1c108280 2024-04-02T14:24:08.564 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:07.608+0000 7fca1a000700 1 --2- 172.21.15.90:0/2416755095 >> [v2:172.21.15.90:6800/129436436,v1:172.21.15.90:6801/129436436] conn(0x7fca04039660 0x7fca0403bb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:24:08.564 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:07.608+0000 7fca1a000700 1 -- 172.21.15.90:0/2416755095 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fca10011e00 con 0x7fca1c108280 2024-04-02T14:24:08.564 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:07.608+0000 7fca20e00700 1 --2- 172.21.15.90:0/2416755095 >> [v2:172.21.15.90:6800/129436436,v1:172.21.15.90:6801/129436436] conn(0x7fca04039660 0x7fca0403bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:24:08.564 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:07.609+0000 7fca20e00700 1 --2- 172.21.15.90:0/2416755095 >> [v2:172.21.15.90:6800/129436436,v1:172.21.15.90:6801/129436436] conn(0x7fca04039660 0x7fca0403bb20 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fca08006fd0 tx=0x7fca08006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:24:08.564 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:07.613+0000 7fca1a000700 1 -- 172.21.15.90:0/2416755095 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7fca1000b3e0 con 0x7fca1c108280 2024-04-02T14:24:08.564 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:07.838+0000 7fca24200700 1 -- 172.21.15.90:0/2416755095 --> [v2:172.21.15.90:6800/129436436,v1:172.21.15.90:6801/129436436] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) -- 0x7fca1c02dda0 con 0x7fca04039660 2024-04-02T14:24:08.564 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:07.840+0000 7fca1a000700 1 -- 172.21.15.90:0/2416755095 <== mgr.14120 v2:172.21.15.90:6800/129436436 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+595 (secure 0 0 0) 0x7fca1c02dda0 con 0x7fca04039660 2024-04-02T14:24:08.564 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:07.843+0000 7fca24200700 1 -- 172.21.15.90:0/2416755095 >> [v2:172.21.15.90:6800/129436436,v1:172.21.15.90:6801/129436436] conn(0x7fca04039660 msgr2=0x7fca0403bb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:24:08.564 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:07.843+0000 7fca24200700 1 --2- 172.21.15.90:0/2416755095 >> [v2:172.21.15.90:6800/129436436,v1:172.21.15.90:6801/129436436] conn(0x7fca04039660 0x7fca0403bb20 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fca08006fd0 tx=0x7fca08006e40 comp rx=0 tx=0).stop 2024-04-02T14:24:08.564 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:07.843+0000 7fca24200700 1 -- 172.21.15.90:0/2416755095 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fca1c108280 msgr2=0x7fca1c19c4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:24:08.564 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:07.843+0000 7fca24200700 1 --2- 172.21.15.90:0/2416755095 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fca1c108280 0x7fca1c19c4c0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fca10011b60 tx=0x7fca10011c40 comp rx=0 tx=0).stop 2024-04-02T14:24:08.564 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:07.843+0000 7fca24200700 1 -- 172.21.15.90:0/2416755095 shutdown_connections 2024-04-02T14:24:08.564 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:07.843+0000 7fca24200700 1 --2- 172.21.15.90:0/2416755095 >> [v2:172.21.15.90:6800/129436436,v1:172.21.15.90:6801/129436436] conn(0x7fca04039660 0x7fca0403bb20 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:24:08.565 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:07.843+0000 7fca24200700 1 --2- 172.21.15.90:0/2416755095 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fca1c108280 0x7fca1c19c4c0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:24:08.565 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:07.843+0000 7fca24200700 1 -- 172.21.15.90:0/2416755095 >> 172.21.15.90:0/2416755095 conn(0x7fca1c103900 msgr2=0x7fca1c1044e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:24:08.565 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:07.843+0000 7fca24200700 1 -- 172.21.15.90:0/2416755095 shutdown_connections 2024-04-02T14:24:08.565 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:07.844+0000 7fca24200700 1 -- 172.21.15.90:0/2416755095 wait complete. 2024-04-02T14:24:08.565 INFO:teuthology.orchestra.run.smithi090.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-04-02T14:24:08.565 INFO:teuthology.orchestra.run.smithi090.stdout:Adding key to root@localhost authorized_keys... 2024-04-02T14:24:08.565 INFO:teuthology.orchestra.run.smithi090.stdout:Adding host smithi090... 2024-04-02T14:24:08.833 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:24:08 smithi090 ceph-mon[31674]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T14:24:09.938 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:24:09 smithi090 ceph-mon[31674]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi090", "addr": "172.21.15.90", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T14:24:12.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:24:11 smithi090 ceph-mon[31674]: Deploying cephadm binary to smithi090 2024-04-02T14:24:16.348 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout Added host 'smithi090' with addr '172.21.15.90' 2024-04-02T14:24:16.348 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:09.128+0000 7f0eb2800700 1 Processor -- start 2024-04-02T14:24:16.348 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:09.128+0000 7f0eb2800700 1 -- start start 2024-04-02T14:24:16.348 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:09.128+0000 7f0eb2800700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0eac10ef40 0x7f0eac10f360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:24:16.348 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:09.128+0000 7f0eb2800700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f0eac10f930 con 0x7f0eac10ef40 2024-04-02T14:24:16.348 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:09.129+0000 7f0eab400700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0eac10ef40 0x7f0eac10f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:24:16.348 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:09.129+0000 7f0eab400700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0eac10ef40 0x7f0eac10f360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38960/0 (socket says 172.21.15.90:38960) 2024-04-02T14:24:16.348 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:09.129+0000 7f0eab400700 1 -- 172.21.15.90:0/3502255813 learned_addr learned my addr 172.21.15.90:0/3502255813 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:24:16.348 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:09.129+0000 7f0eab400700 1 -- 172.21.15.90:0/3502255813 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0eac10fab0 con 0x7f0eac10ef40 2024-04-02T14:24:16.348 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:09.130+0000 7f0eab400700 1 --2- 172.21.15.90:0/3502255813 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0eac10ef40 0x7f0eac10f360 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f0e9c0098a0 tx=0x7f0e9c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53a8e0cf19dc453f server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:24:16.349 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:09.130+0000 7f0eaa000700 1 -- 172.21.15.90:0/3502255813 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0e9c004160 con 0x7f0eac10ef40 2024-04-02T14:24:16.349 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:09.130+0000 7f0eaa000700 1 -- 172.21.15.90:0/3502255813 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f0e9c004300 con 0x7f0eac10ef40 2024-04-02T14:24:16.349 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:09.130+0000 7f0eaa000700 1 -- 172.21.15.90:0/3502255813 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0e9c011b50 con 0x7f0eac10ef40 2024-04-02T14:24:16.349 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:09.131+0000 7f0eb2800700 1 -- 172.21.15.90:0/3502255813 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0eac10ef40 msgr2=0x7f0eac10f360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:24:16.349 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:09.131+0000 7f0eb2800700 1 --2- 172.21.15.90:0/3502255813 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0eac10ef40 0x7f0eac10f360 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f0e9c0098a0 tx=0x7f0e9c009bb0 comp rx=0 tx=0).stop 2024-04-02T14:24:16.349 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:09.131+0000 7f0eb2800700 1 -- 172.21.15.90:0/3502255813 shutdown_connections 2024-04-02T14:24:16.349 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:09.131+0000 7f0eb2800700 1 --2- 172.21.15.90:0/3502255813 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0eac10ef40 0x7f0eac10f360 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:24:16.349 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:09.131+0000 7f0eb2800700 1 -- 172.21.15.90:0/3502255813 >> 172.21.15.90:0/3502255813 conn(0x7f0eac07d990 msgr2=0x7f0eac07fdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:24:16.349 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:09.131+0000 7f0eb2800700 1 -- 172.21.15.90:0/3502255813 shutdown_connections 2024-04-02T14:24:16.349 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:09.131+0000 7f0eb2800700 1 -- 172.21.15.90:0/3502255813 wait complete. 2024-04-02T14:24:16.349 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:09.132+0000 7f0eb2800700 1 Processor -- start 2024-04-02T14:24:16.349 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:09.132+0000 7f0eb2800700 1 -- start start 2024-04-02T14:24:16.349 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:09.132+0000 7f0eb2800700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0eac10ef40 0x7f0eac1a27d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:24:16.349 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:09.132+0000 7f0eb2800700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f0eac112bd0 con 0x7f0eac10ef40 2024-04-02T14:24:16.349 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:09.133+0000 7f0eab400700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0eac10ef40 0x7f0eac1a27d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:24:16.350 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:09.133+0000 7f0eab400700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0eac10ef40 0x7f0eac1a27d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38970/0 (socket says 172.21.15.90:38970) 2024-04-02T14:24:16.350 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:09.133+0000 7f0eab400700 1 -- 172.21.15.90:0/3558838271 learned_addr learned my addr 172.21.15.90:0/3558838271 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:24:16.350 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:09.133+0000 7f0eab400700 1 -- 172.21.15.90:0/3558838271 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0eac1a2d10 con 0x7f0eac10ef40 2024-04-02T14:24:16.350 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:09.133+0000 7f0eab400700 1 --2- 172.21.15.90:0/3558838271 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0eac10ef40 0x7f0eac1a27d0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f0e9c011b60 tx=0x7f0e9c011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:24:16.350 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:09.134+0000 7f0ea3e00700 1 -- 172.21.15.90:0/3558838271 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0e9c003970 con 0x7f0eac10ef40 2024-04-02T14:24:16.350 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:09.134+0000 7f0ea3e00700 1 -- 172.21.15.90:0/3558838271 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f0e9c003b10 con 0x7f0eac10ef40 2024-04-02T14:24:16.350 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:09.134+0000 7f0eb2800700 1 -- 172.21.15.90:0/3558838271 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0eac1a2fa0 con 0x7f0eac10ef40 2024-04-02T14:24:16.350 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:09.134+0000 7f0ea3e00700 1 -- 172.21.15.90:0/3558838271 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0e9c024610 con 0x7f0eac10ef40 2024-04-02T14:24:16.350 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:09.134+0000 7f0eb2800700 1 -- 172.21.15.90:0/3558838271 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0eac1a5c90 con 0x7f0eac10ef40 2024-04-02T14:24:16.350 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:09.135+0000 7f0eb2800700 1 -- 172.21.15.90:0/3558838271 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0eac063c20 con 0x7f0eac10ef40 2024-04-02T14:24:16.350 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:09.135+0000 7f0ea3e00700 1 -- 172.21.15.90:0/3558838271 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f0e9c01a9f0 con 0x7f0eac10ef40 2024-04-02T14:24:16.350 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:09.136+0000 7f0ea3e00700 1 --2- 172.21.15.90:0/3558838271 >> [v2:172.21.15.90:6800/129436436,v1:172.21.15.90:6801/129436436] conn(0x7f0e90039600 0x7f0e9003bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:24:16.350 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:09.136+0000 7f0ea3e00700 1 -- 172.21.15.90:0/3558838271 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f0e9c011e00 con 0x7f0eac10ef40 2024-04-02T14:24:16.351 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:09.136+0000 7f0eaaa00700 1 --2- 172.21.15.90:0/3558838271 >> [v2:172.21.15.90:6800/129436436,v1:172.21.15.90:6801/129436436] conn(0x7f0e90039600 0x7f0e9003bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:24:16.351 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:09.137+0000 7f0eaaa00700 1 --2- 172.21.15.90:0/3558838271 >> [v2:172.21.15.90:6800/129436436,v1:172.21.15.90:6801/129436436] conn(0x7f0e90039600 0x7f0e9003bac0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f0e98006fd0 tx=0x7f0e98006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:24:16.351 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:09.141+0000 7f0ea3e00700 1 -- 172.21.15.90:0/3558838271 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7f0e9c016260 con 0x7f0eac10ef40 2024-04-02T14:24:16.351 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:09.367+0000 7f0eb2800700 1 -- 172.21.15.90:0/3558838271 --> [v2:172.21.15.90:6800/129436436,v1:172.21.15.90:6801/129436436] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi090", "addr": "172.21.15.90", "target": ["mon-mgr", ""]}) -- 0x7f0eac07ec00 con 0x7f0e90039600 2024-04-02T14:24:16.351 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:15.690+0000 7f0ea3e00700 1 -- 172.21.15.90:0/3558838271 <== mgr.14120 v2:172.21.15.90:6800/129436436 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+48 (secure 0 0 0) 0x7f0eac07ec00 con 0x7f0e90039600 2024-04-02T14:24:16.351 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:15.695+0000 7f0eb2800700 1 -- 172.21.15.90:0/3558838271 >> [v2:172.21.15.90:6800/129436436,v1:172.21.15.90:6801/129436436] conn(0x7f0e90039600 msgr2=0x7f0e9003bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:24:16.351 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:15.695+0000 7f0eb2800700 1 --2- 172.21.15.90:0/3558838271 >> [v2:172.21.15.90:6800/129436436,v1:172.21.15.90:6801/129436436] conn(0x7f0e90039600 0x7f0e9003bac0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f0e98006fd0 tx=0x7f0e98006e40 comp rx=0 tx=0).stop 2024-04-02T14:24:16.351 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:15.695+0000 7f0eb2800700 1 -- 172.21.15.90:0/3558838271 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0eac10ef40 msgr2=0x7f0eac1a27d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:24:16.351 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:15.695+0000 7f0eb2800700 1 --2- 172.21.15.90:0/3558838271 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0eac10ef40 0x7f0eac1a27d0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f0e9c011b60 tx=0x7f0e9c011c40 comp rx=0 tx=0).stop 2024-04-02T14:24:16.351 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:15.695+0000 7f0eb2800700 1 -- 172.21.15.90:0/3558838271 shutdown_connections 2024-04-02T14:24:16.351 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:15.695+0000 7f0eb2800700 1 --2- 172.21.15.90:0/3558838271 >> [v2:172.21.15.90:6800/129436436,v1:172.21.15.90:6801/129436436] conn(0x7f0e90039600 0x7f0e9003bac0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:24:16.351 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:15.695+0000 7f0eb2800700 1 --2- 172.21.15.90:0/3558838271 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0eac10ef40 0x7f0eac1a27d0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:24:16.351 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:15.695+0000 7f0eb2800700 1 -- 172.21.15.90:0/3558838271 >> 172.21.15.90:0/3558838271 conn(0x7f0eac07d990 msgr2=0x7f0eac07e4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:24:16.351 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:15.696+0000 7f0eb2800700 1 -- 172.21.15.90:0/3558838271 shutdown_connections 2024-04-02T14:24:16.352 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:15.696+0000 7f0eb2800700 1 -- 172.21.15.90:0/3558838271 wait complete. 2024-04-02T14:24:16.352 INFO:teuthology.orchestra.run.smithi090.stdout:Deploying unmanaged mon service... 2024-04-02T14:24:16.939 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:24:16 smithi090 ceph-mon[31674]: from='mgr.14120 172.21.15.90:0/2851514084' entity='mgr.x' 2024-04-02T14:24:16.939 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:24:16 smithi090 ceph-mon[31674]: Added host smithi090 2024-04-02T14:24:16.939 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:24:16 smithi090 ceph-mon[31674]: from='mgr.14120 172.21.15.90:0/2851514084' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T14:24:18.191 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-04-02T14:24:18.191 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:17.032+0000 7fc782000700 1 Processor -- start 2024-04-02T14:24:18.191 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:17.032+0000 7fc782000700 1 -- start start 2024-04-02T14:24:18.191 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:17.033+0000 7fc782000700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc77c108050 0x7fc77c108470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:24:18.191 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:17.033+0000 7fc782000700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fc77c108a40 con 0x7fc77c108050 2024-04-02T14:24:18.191 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:17.033+0000 7fc780c00700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc77c108050 0x7fc77c108470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:24:18.191 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:17.033+0000 7fc780c00700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc77c108050 0x7fc77c108470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:51946/0 (socket says 172.21.15.90:51946) 2024-04-02T14:24:18.192 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:17.033+0000 7fc780c00700 1 -- 172.21.15.90:0/4265286351 learned_addr learned my addr 172.21.15.90:0/4265286351 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:24:18.192 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:17.034+0000 7fc780c00700 1 -- 172.21.15.90:0/4265286351 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc77c109250 con 0x7fc77c108050 2024-04-02T14:24:18.192 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:17.034+0000 7fc780c00700 1 --2- 172.21.15.90:0/4265286351 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc77c108050 0x7fc77c108470 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fc7700098a0 tx=0x7fc770009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4f6e12dc49aba2d server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:24:18.192 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:17.034+0000 7fc77b400700 1 -- 172.21.15.90:0/4265286351 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc770004160 con 0x7fc77c108050 2024-04-02T14:24:18.192 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:17.034+0000 7fc77b400700 1 -- 172.21.15.90:0/4265286351 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fc770004300 con 0x7fc77c108050 2024-04-02T14:24:18.192 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:17.034+0000 7fc77b400700 1 -- 172.21.15.90:0/4265286351 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc770011b50 con 0x7fc77c108050 2024-04-02T14:24:18.192 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:17.035+0000 7fc782000700 1 -- 172.21.15.90:0/4265286351 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc77c108050 msgr2=0x7fc77c108470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:24:18.192 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:17.035+0000 7fc782000700 1 --2- 172.21.15.90:0/4265286351 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc77c108050 0x7fc77c108470 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fc7700098a0 tx=0x7fc770009bb0 comp rx=0 tx=0).stop 2024-04-02T14:24:18.192 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:17.035+0000 7fc782000700 1 -- 172.21.15.90:0/4265286351 shutdown_connections 2024-04-02T14:24:18.192 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:17.035+0000 7fc782000700 1 --2- 172.21.15.90:0/4265286351 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc77c108050 0x7fc77c108470 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:24:18.192 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:17.035+0000 7fc782000700 1 -- 172.21.15.90:0/4265286351 >> 172.21.15.90:0/4265286351 conn(0x7fc77c1035f0 msgr2=0x7fc77c105a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:24:18.192 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:17.035+0000 7fc782000700 1 -- 172.21.15.90:0/4265286351 shutdown_connections 2024-04-02T14:24:18.192 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:17.035+0000 7fc782000700 1 -- 172.21.15.90:0/4265286351 wait complete. 2024-04-02T14:24:18.192 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:17.036+0000 7fc782000700 1 Processor -- start 2024-04-02T14:24:18.193 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:17.036+0000 7fc782000700 1 -- start start 2024-04-02T14:24:18.193 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:17.037+0000 7fc782000700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc77c108050 0x7fc77c19e2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:24:18.193 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:17.037+0000 7fc782000700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fc77c10c600 con 0x7fc77c108050 2024-04-02T14:24:18.193 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:17.037+0000 7fc780c00700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc77c108050 0x7fc77c19e2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:24:18.193 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:17.037+0000 7fc780c00700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc77c108050 0x7fc77c19e2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:51956/0 (socket says 172.21.15.90:51956) 2024-04-02T14:24:18.193 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:17.037+0000 7fc780c00700 1 -- 172.21.15.90:0/407536648 learned_addr learned my addr 172.21.15.90:0/407536648 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:24:18.193 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:17.037+0000 7fc780c00700 1 -- 172.21.15.90:0/407536648 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc77c19e7e0 con 0x7fc77c108050 2024-04-02T14:24:18.193 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:17.038+0000 7fc780c00700 1 --2- 172.21.15.90:0/407536648 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc77c108050 0x7fc77c19e2a0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fc770000c00 tx=0x7fc770011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:24:18.193 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:17.038+0000 7fc779600700 1 -- 172.21.15.90:0/407536648 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc770003b20 con 0x7fc77c108050 2024-04-02T14:24:18.193 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:17.038+0000 7fc779600700 1 -- 172.21.15.90:0/407536648 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fc770003cc0 con 0x7fc77c108050 2024-04-02T14:24:18.193 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:17.038+0000 7fc779600700 1 -- 172.21.15.90:0/407536648 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc770024910 con 0x7fc77c108050 2024-04-02T14:24:18.193 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:17.038+0000 7fc782000700 1 -- 172.21.15.90:0/407536648 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc77c19ea70 con 0x7fc77c108050 2024-04-02T14:24:18.193 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:17.038+0000 7fc782000700 1 -- 172.21.15.90:0/407536648 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc77c1a1760 con 0x7fc77c108050 2024-04-02T14:24:18.193 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:17.039+0000 7fc779600700 1 -- 172.21.15.90:0/407536648 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7fc770022020 con 0x7fc77c108050 2024-04-02T14:24:18.193 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:17.039+0000 7fc782000700 1 -- 172.21.15.90:0/407536648 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc77c063c20 con 0x7fc77c108050 2024-04-02T14:24:18.194 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:17.039+0000 7fc779600700 1 --2- 172.21.15.90:0/407536648 >> [v2:172.21.15.90:6800/129436436,v1:172.21.15.90:6801/129436436] conn(0x7fc764039600 0x7fc76403bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:24:18.194 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:17.040+0000 7fc779600700 1 -- 172.21.15.90:0/407536648 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fc770011e00 con 0x7fc77c108050 2024-04-02T14:24:18.194 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:17.040+0000 7fc77be00700 1 --2- 172.21.15.90:0/407536648 >> [v2:172.21.15.90:6800/129436436,v1:172.21.15.90:6801/129436436] conn(0x7fc764039600 0x7fc76403bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:24:18.194 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:17.040+0000 7fc77be00700 1 --2- 172.21.15.90:0/407536648 >> [v2:172.21.15.90:6800/129436436,v1:172.21.15.90:6801/129436436] conn(0x7fc764039600 0x7fc76403bac0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fc768006fd0 tx=0x7fc768006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:24:18.194 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:17.045+0000 7fc779600700 1 -- 172.21.15.90:0/407536648 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7fc77001ad80 con 0x7fc77c108050 2024-04-02T14:24:18.194 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:17.294+0000 7fc782000700 1 -- 172.21.15.90:0/407536648 --> [v2:172.21.15.90:6800/129436436,v1:172.21.15.90:6801/129436436] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7fc77c1047e0 con 0x7fc764039600 2024-04-02T14:24:18.194 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:17.301+0000 7fc779600700 1 -- 172.21.15.90:0/407536648 <== mgr.14120 v2:172.21.15.90:6800/129436436 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7fc77c1047e0 con 0x7fc764039600 2024-04-02T14:24:18.194 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:17.304+0000 7fc782000700 1 -- 172.21.15.90:0/407536648 >> [v2:172.21.15.90:6800/129436436,v1:172.21.15.90:6801/129436436] conn(0x7fc764039600 msgr2=0x7fc76403bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:24:18.194 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:17.304+0000 7fc782000700 1 --2- 172.21.15.90:0/407536648 >> [v2:172.21.15.90:6800/129436436,v1:172.21.15.90:6801/129436436] conn(0x7fc764039600 0x7fc76403bac0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fc768006fd0 tx=0x7fc768006e40 comp rx=0 tx=0).stop 2024-04-02T14:24:18.194 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:17.304+0000 7fc782000700 1 -- 172.21.15.90:0/407536648 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc77c108050 msgr2=0x7fc77c19e2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:24:18.194 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:17.304+0000 7fc782000700 1 --2- 172.21.15.90:0/407536648 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc77c108050 0x7fc77c19e2a0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fc770000c00 tx=0x7fc770011c10 comp rx=0 tx=0).stop 2024-04-02T14:24:18.194 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:17.304+0000 7fc782000700 1 -- 172.21.15.90:0/407536648 shutdown_connections 2024-04-02T14:24:18.194 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:17.304+0000 7fc782000700 1 --2- 172.21.15.90:0/407536648 >> [v2:172.21.15.90:6800/129436436,v1:172.21.15.90:6801/129436436] conn(0x7fc764039600 0x7fc76403bac0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:24:18.195 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:17.304+0000 7fc782000700 1 --2- 172.21.15.90:0/407536648 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc77c108050 0x7fc77c19e2a0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:24:18.195 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:17.304+0000 7fc782000700 1 -- 172.21.15.90:0/407536648 >> 172.21.15.90:0/407536648 conn(0x7fc77c1035f0 msgr2=0x7fc77c1040d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:24:18.195 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:17.305+0000 7fc782000700 1 -- 172.21.15.90:0/407536648 shutdown_connections 2024-04-02T14:24:18.195 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:17.305+0000 7fc782000700 1 -- 172.21.15.90:0/407536648 wait complete. 2024-04-02T14:24:18.195 INFO:teuthology.orchestra.run.smithi090.stdout:Deploying unmanaged mgr service... 2024-04-02T14:24:18.439 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:24:18 smithi090 ceph-mon[31674]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T14:24:18.439 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:24:18 smithi090 ceph-mon[31674]: Saving service mon spec with placement count:5 2024-04-02T14:24:18.439 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:24:18 smithi090 ceph-mon[31674]: from='mgr.14120 172.21.15.90:0/2851514084' entity='mgr.x' 2024-04-02T14:24:20.386 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:24:19 smithi090 ceph-mon[31674]: from='mgr.14120 172.21.15.90:0/2851514084' entity='mgr.x' 2024-04-02T14:24:20.386 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:24:19 smithi090 ceph-mon[31674]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T14:24:20.386 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:24:19 smithi090 ceph-mon[31674]: Saving service mgr spec with placement count:2 2024-04-02T14:24:20.386 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:24:19 smithi090 ceph-mon[31674]: from='mgr.14120 172.21.15.90:0/2851514084' entity='mgr.x' 2024-04-02T14:24:20.386 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:24:19 smithi090 ceph-mon[31674]: from='mgr.14120 172.21.15.90:0/2851514084' entity='mgr.x' 2024-04-02T14:24:20.395 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-04-02T14:24:20.396 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:19.438+0000 7f443be00700 1 Processor -- start 2024-04-02T14:24:20.396 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:19.438+0000 7f443be00700 1 -- start start 2024-04-02T14:24:20.396 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:19.439+0000 7f443be00700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f443407cd80 0x7f443407b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:24:20.396 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:19.439+0000 7f443be00700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f443407b7b0 con 0x7f443407cd80 2024-04-02T14:24:20.396 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:19.439+0000 7f4439400700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f443407cd80 0x7f443407b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:24:20.396 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:19.439+0000 7f4439400700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f443407cd80 0x7f443407b1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:51960/0 (socket says 172.21.15.90:51960) 2024-04-02T14:24:20.396 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:19.439+0000 7f4439400700 1 -- 172.21.15.90:0/1937530612 learned_addr learned my addr 172.21.15.90:0/1937530612 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:24:20.396 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:19.440+0000 7f4439400700 1 -- 172.21.15.90:0/1937530612 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f443407b930 con 0x7f443407cd80 2024-04-02T14:24:20.396 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:19.440+0000 7f4439400700 1 --2- 172.21.15.90:0/1937530612 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f443407cd80 0x7f443407b1e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f44280098a0 tx=0x7f4428009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=469fdebab9312073 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:24:20.396 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:19.440+0000 7f4433e00700 1 -- 172.21.15.90:0/1937530612 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4428004160 con 0x7f443407cd80 2024-04-02T14:24:20.396 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:19.440+0000 7f4433e00700 1 -- 172.21.15.90:0/1937530612 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4428004300 con 0x7f443407cd80 2024-04-02T14:24:20.396 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:19.440+0000 7f4433e00700 1 -- 172.21.15.90:0/1937530612 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4428011b20 con 0x7f443407cd80 2024-04-02T14:24:20.396 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:19.441+0000 7f443be00700 1 -- 172.21.15.90:0/1937530612 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f443407cd80 msgr2=0x7f443407b1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:24:20.397 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:19.441+0000 7f443be00700 1 --2- 172.21.15.90:0/1937530612 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f443407cd80 0x7f443407b1e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f44280098a0 tx=0x7f4428009bb0 comp rx=0 tx=0).stop 2024-04-02T14:24:20.397 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:19.441+0000 7f443be00700 1 -- 172.21.15.90:0/1937530612 shutdown_connections 2024-04-02T14:24:20.397 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:19.441+0000 7f443be00700 1 --2- 172.21.15.90:0/1937530612 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f443407cd80 0x7f443407b1e0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:24:20.397 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:19.441+0000 7f443be00700 1 -- 172.21.15.90:0/1937530612 >> 172.21.15.90:0/1937530612 conn(0x7f4434103860 msgr2=0x7f4434105c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:24:20.397 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:19.441+0000 7f443be00700 1 -- 172.21.15.90:0/1937530612 shutdown_connections 2024-04-02T14:24:20.397 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:19.441+0000 7f443be00700 1 -- 172.21.15.90:0/1937530612 wait complete. 2024-04-02T14:24:20.397 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:19.443+0000 7f443be00700 1 Processor -- start 2024-04-02T14:24:20.397 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:19.443+0000 7f443be00700 1 -- start start 2024-04-02T14:24:20.397 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:19.443+0000 7f443be00700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f443407cd80 0x7f443419e380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:24:20.397 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:19.443+0000 7f443be00700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f443410e8e0 con 0x7f443407cd80 2024-04-02T14:24:20.397 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:19.443+0000 7f4439400700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f443407cd80 0x7f443419e380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:24:20.397 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:19.443+0000 7f4439400700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f443407cd80 0x7f443419e380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:51964/0 (socket says 172.21.15.90:51964) 2024-04-02T14:24:20.397 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:19.444+0000 7f4439400700 1 -- 172.21.15.90:0/210471014 learned_addr learned my addr 172.21.15.90:0/210471014 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:24:20.397 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:19.444+0000 7f4439400700 1 -- 172.21.15.90:0/210471014 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f443419e8c0 con 0x7f443407cd80 2024-04-02T14:24:20.397 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:19.444+0000 7f4439400700 1 --2- 172.21.15.90:0/210471014 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f443407cd80 0x7f443419e380 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f44280117e0 tx=0x7f4428011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:24:20.398 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:19.444+0000 7f4432000700 1 -- 172.21.15.90:0/210471014 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4428003ba0 con 0x7f443407cd80 2024-04-02T14:24:20.398 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:19.445+0000 7f4432000700 1 -- 172.21.15.90:0/210471014 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f442801a910 con 0x7f443407cd80 2024-04-02T14:24:20.398 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:19.445+0000 7f4432000700 1 -- 172.21.15.90:0/210471014 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f44280248f0 con 0x7f443407cd80 2024-04-02T14:24:20.398 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:19.445+0000 7f443be00700 1 -- 172.21.15.90:0/210471014 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f443419eb50 con 0x7f443407cd80 2024-04-02T14:24:20.398 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:19.445+0000 7f443be00700 1 -- 172.21.15.90:0/210471014 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f443419efb0 con 0x7f443407cd80 2024-04-02T14:24:20.398 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:19.446+0000 7f443be00700 1 -- 172.21.15.90:0/210471014 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4434063c20 con 0x7f443407cd80 2024-04-02T14:24:20.398 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:19.446+0000 7f4432000700 1 -- 172.21.15.90:0/210471014 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f4428022020 con 0x7f443407cd80 2024-04-02T14:24:20.398 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:19.446+0000 7f4432000700 1 --2- 172.21.15.90:0/210471014 >> [v2:172.21.15.90:6800/129436436,v1:172.21.15.90:6801/129436436] conn(0x7f441c039600 0x7f441c03bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:24:20.398 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:19.446+0000 7f4432000700 1 -- 172.21.15.90:0/210471014 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f442804db20 con 0x7f443407cd80 2024-04-02T14:24:20.398 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:19.447+0000 7f4438a00700 1 --2- 172.21.15.90:0/210471014 >> [v2:172.21.15.90:6800/129436436,v1:172.21.15.90:6801/129436436] conn(0x7f441c039600 0x7f441c03bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:24:20.398 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:19.447+0000 7f4438a00700 1 --2- 172.21.15.90:0/210471014 >> [v2:172.21.15.90:6800/129436436,v1:172.21.15.90:6801/129436436] conn(0x7f441c039600 0x7f441c03bac0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f4420006fd0 tx=0x7f4420006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:24:20.398 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:19.456+0000 7f4432000700 1 -- 172.21.15.90:0/210471014 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7f442800f2e0 con 0x7f443407cd80 2024-04-02T14:24:20.398 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:19.700+0000 7f443be00700 1 -- 172.21.15.90:0/210471014 --> [v2:172.21.15.90:6800/129436436,v1:172.21.15.90:6801/129436436] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7f44341049e0 con 0x7f441c039600 2024-04-02T14:24:20.399 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:19.705+0000 7f4432000700 1 -- 172.21.15.90:0/210471014 <== mgr.14120 v2:172.21.15.90:6800/129436436 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f44341049e0 con 0x7f441c039600 2024-04-02T14:24:20.399 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:19.708+0000 7f443be00700 1 -- 172.21.15.90:0/210471014 >> [v2:172.21.15.90:6800/129436436,v1:172.21.15.90:6801/129436436] conn(0x7f441c039600 msgr2=0x7f441c03bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:24:20.399 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:19.708+0000 7f443be00700 1 --2- 172.21.15.90:0/210471014 >> [v2:172.21.15.90:6800/129436436,v1:172.21.15.90:6801/129436436] conn(0x7f441c039600 0x7f441c03bac0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f4420006fd0 tx=0x7f4420006e40 comp rx=0 tx=0).stop 2024-04-02T14:24:20.399 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:19.708+0000 7f443be00700 1 -- 172.21.15.90:0/210471014 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f443407cd80 msgr2=0x7f443419e380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:24:20.399 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:19.708+0000 7f443be00700 1 --2- 172.21.15.90:0/210471014 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f443407cd80 0x7f443419e380 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f44280117e0 tx=0x7f4428011ba0 comp rx=0 tx=0).stop 2024-04-02T14:24:20.399 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:19.708+0000 7f443be00700 1 -- 172.21.15.90:0/210471014 shutdown_connections 2024-04-02T14:24:20.399 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:19.708+0000 7f443be00700 1 --2- 172.21.15.90:0/210471014 >> [v2:172.21.15.90:6800/129436436,v1:172.21.15.90:6801/129436436] conn(0x7f441c039600 0x7f441c03bac0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:24:20.399 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:19.708+0000 7f443be00700 1 --2- 172.21.15.90:0/210471014 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f443407cd80 0x7f443419e380 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:24:20.399 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:19.708+0000 7f443be00700 1 -- 172.21.15.90:0/210471014 >> 172.21.15.90:0/210471014 conn(0x7f4434103860 msgr2=0x7f44341042d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:24:20.399 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:19.708+0000 7f443be00700 1 -- 172.21.15.90:0/210471014 shutdown_connections 2024-04-02T14:24:20.399 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:19.708+0000 7f443be00700 1 -- 172.21.15.90:0/210471014 wait complete. 2024-04-02T14:24:21.688 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:24:21 smithi090 ceph-mon[31674]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T14:24:21.689 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:24:21 smithi090 ceph-mon[31674]: from='client.? 172.21.15.90:0/4038969870' entity='client.admin' 2024-04-02T14:24:22.363 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:20.961+0000 7f83fa400700 1 Processor -- start 2024-04-02T14:24:22.363 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:20.962+0000 7f83fa400700 1 -- start start 2024-04-02T14:24:22.363 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:20.962+0000 7f83fa400700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f83f407cde0 0x7f83f407b240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:24:22.363 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:20.962+0000 7f83fa400700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f83f407b810 con 0x7f83f407cde0 2024-04-02T14:24:22.363 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:20.962+0000 7f83f9000700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f83f407cde0 0x7f83f407b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:24:22.364 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:20.962+0000 7f83f9000700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f83f407cde0 0x7f83f407b240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:51966/0 (socket says 172.21.15.90:51966) 2024-04-02T14:24:22.364 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:20.962+0000 7f83f9000700 1 -- 172.21.15.90:0/3721535702 learned_addr learned my addr 172.21.15.90:0/3721535702 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:24:22.364 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:20.963+0000 7f83f9000700 1 -- 172.21.15.90:0/3721535702 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83f407b990 con 0x7f83f407cde0 2024-04-02T14:24:22.364 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:20.963+0000 7f83f9000700 1 --2- 172.21.15.90:0/3721535702 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f83f407cde0 0x7f83f407b240 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f83e40098a0 tx=0x7f83e4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=326d3082f363c5e4 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:24:22.364 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:20.963+0000 7f83f3e00700 1 -- 172.21.15.90:0/3721535702 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f83e4004160 con 0x7f83f407cde0 2024-04-02T14:24:22.364 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:20.963+0000 7f83f3e00700 1 -- 172.21.15.90:0/3721535702 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f83e4004300 con 0x7f83f407cde0 2024-04-02T14:24:22.364 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:20.963+0000 7f83f3e00700 1 -- 172.21.15.90:0/3721535702 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f83e4011b20 con 0x7f83f407cde0 2024-04-02T14:24:22.364 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:20.964+0000 7f83fa400700 1 -- 172.21.15.90:0/3721535702 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f83f407cde0 msgr2=0x7f83f407b240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:24:22.364 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:20.964+0000 7f83fa400700 1 --2- 172.21.15.90:0/3721535702 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f83f407cde0 0x7f83f407b240 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f83e40098a0 tx=0x7f83e4009bb0 comp rx=0 tx=0).stop 2024-04-02T14:24:22.364 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:20.964+0000 7f83fa400700 1 -- 172.21.15.90:0/3721535702 shutdown_connections 2024-04-02T14:24:22.365 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:20.964+0000 7f83fa400700 1 --2- 172.21.15.90:0/3721535702 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f83f407cde0 0x7f83f407b240 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:24:22.365 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:20.964+0000 7f83fa400700 1 -- 172.21.15.90:0/3721535702 >> 172.21.15.90:0/3721535702 conn(0x7f83f4103640 msgr2=0x7f83f4105a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:24:22.365 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:20.964+0000 7f83fa400700 1 -- 172.21.15.90:0/3721535702 shutdown_connections 2024-04-02T14:24:22.365 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:20.964+0000 7f83fa400700 1 -- 172.21.15.90:0/3721535702 wait complete. 2024-04-02T14:24:22.365 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:20.965+0000 7f83fa400700 1 Processor -- start 2024-04-02T14:24:22.365 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:20.966+0000 7f83fa400700 1 -- start start 2024-04-02T14:24:22.365 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:20.966+0000 7f83fa400700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f83f407cde0 0x7f83f41a2660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:24:22.365 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:20.966+0000 7f83fa400700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f83f410a300 con 0x7f83f407cde0 2024-04-02T14:24:22.365 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:20.966+0000 7f83f9000700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f83f407cde0 0x7f83f41a2660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:24:22.365 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:20.966+0000 7f83f9000700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f83f407cde0 0x7f83f41a2660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:51968/0 (socket says 172.21.15.90:51968) 2024-04-02T14:24:22.365 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:20.966+0000 7f83f9000700 1 -- 172.21.15.90:0/4038969870 learned_addr learned my addr 172.21.15.90:0/4038969870 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:24:22.366 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:20.967+0000 7f83f9000700 1 -- 172.21.15.90:0/4038969870 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83f41a2ba0 con 0x7f83f407cde0 2024-04-02T14:24:22.366 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:20.967+0000 7f83f9000700 1 --2- 172.21.15.90:0/4038969870 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f83f407cde0 0x7f83f41a2660 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f83e40116a0 tx=0x7f83e4011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:24:22.366 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:20.967+0000 7f83f2000700 1 -- 172.21.15.90:0/4038969870 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f83e4003a40 con 0x7f83f407cde0 2024-04-02T14:24:22.366 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:20.967+0000 7f83f2000700 1 -- 172.21.15.90:0/4038969870 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f83e4003be0 con 0x7f83f407cde0 2024-04-02T14:24:22.366 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:20.967+0000 7f83f2000700 1 -- 172.21.15.90:0/4038969870 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f83e401a5e0 con 0x7f83f407cde0 2024-04-02T14:24:22.366 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:20.967+0000 7f83fa400700 1 -- 172.21.15.90:0/4038969870 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83f41a2e30 con 0x7f83f407cde0 2024-04-02T14:24:22.366 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:20.968+0000 7f83fa400700 1 -- 172.21.15.90:0/4038969870 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83f41a3290 con 0x7f83f407cde0 2024-04-02T14:24:22.366 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:20.969+0000 7f83f2000700 1 -- 172.21.15.90:0/4038969870 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f83e40249a0 con 0x7f83f407cde0 2024-04-02T14:24:22.366 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:20.969+0000 7f83fa400700 1 -- 172.21.15.90:0/4038969870 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f83f4063c20 con 0x7f83f407cde0 2024-04-02T14:24:22.367 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:20.969+0000 7f83f2000700 1 --2- 172.21.15.90:0/4038969870 >> [v2:172.21.15.90:6800/129436436,v1:172.21.15.90:6801/129436436] conn(0x7f83d80395b0 0x7f83d803ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:24:22.367 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:20.969+0000 7f83f2000700 1 -- 172.21.15.90:0/4038969870 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f83e404daa0 con 0x7f83f407cde0 2024-04-02T14:24:22.367 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:20.970+0000 7f83ebe00700 1 --2- 172.21.15.90:0/4038969870 >> [v2:172.21.15.90:6800/129436436,v1:172.21.15.90:6801/129436436] conn(0x7f83d80395b0 0x7f83d803ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:24:22.367 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:20.970+0000 7f83ebe00700 1 --2- 172.21.15.90:0/4038969870 >> [v2:172.21.15.90:6800/129436436,v1:172.21.15.90:6801/129436436] conn(0x7f83d80395b0 0x7f83d803ba70 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f83dc006fd0 tx=0x7f83dc006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:24:22.367 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:20.975+0000 7f83f2000700 1 -- 172.21.15.90:0/4038969870 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7f83e4011e20 con 0x7f83f407cde0 2024-04-02T14:24:22.367 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:21.207+0000 7f83fa400700 1 -- 172.21.15.90:0/4038969870 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) -- 0x7f83f419bbc0 con 0x7f83f407cde0 2024-04-02T14:24:22.367 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:21.210+0000 7f83f2000700 1 -- 172.21.15.90:0/4038969870 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) ==== 142+0+0 (secure 0 0 0) 0x7f83f419bbc0 con 0x7f83f407cde0 2024-04-02T14:24:22.367 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:21.213+0000 7f83fa400700 1 -- 172.21.15.90:0/4038969870 >> [v2:172.21.15.90:6800/129436436,v1:172.21.15.90:6801/129436436] conn(0x7f83d80395b0 msgr2=0x7f83d803ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:24:22.367 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:21.213+0000 7f83fa400700 1 --2- 172.21.15.90:0/4038969870 >> [v2:172.21.15.90:6800/129436436,v1:172.21.15.90:6801/129436436] conn(0x7f83d80395b0 0x7f83d803ba70 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f83dc006fd0 tx=0x7f83dc006e40 comp rx=0 tx=0).stop 2024-04-02T14:24:22.367 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:21.213+0000 7f83fa400700 1 -- 172.21.15.90:0/4038969870 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f83f407cde0 msgr2=0x7f83f41a2660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:24:22.367 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:21.213+0000 7f83fa400700 1 --2- 172.21.15.90:0/4038969870 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f83f407cde0 0x7f83f41a2660 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f83e40116a0 tx=0x7f83e4011ba0 comp rx=0 tx=0).stop 2024-04-02T14:24:22.368 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:21.213+0000 7f83fa400700 1 -- 172.21.15.90:0/4038969870 shutdown_connections 2024-04-02T14:24:22.368 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:21.213+0000 7f83fa400700 1 --2- 172.21.15.90:0/4038969870 >> [v2:172.21.15.90:6800/129436436,v1:172.21.15.90:6801/129436436] conn(0x7f83d80395b0 0x7f83d803ba70 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:24:22.368 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:21.213+0000 7f83fa400700 1 --2- 172.21.15.90:0/4038969870 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f83f407cde0 0x7f83f41a2660 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:24:22.368 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:21.213+0000 7f83fa400700 1 -- 172.21.15.90:0/4038969870 >> 172.21.15.90:0/4038969870 conn(0x7f83f4103640 msgr2=0x7f83f4105a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:24:22.368 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:21.213+0000 7f83fa400700 1 -- 172.21.15.90:0/4038969870 shutdown_connections 2024-04-02T14:24:22.368 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:21.213+0000 7f83fa400700 1 -- 172.21.15.90:0/4038969870 wait complete. 2024-04-02T14:24:23.588 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:24:23 smithi090 ceph-mon[31674]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T14:24:23.588 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:24:23 smithi090 ceph-mon[31674]: from='mgr.14120 172.21.15.90:0/2851514084' entity='mgr.x' 2024-04-02T14:24:23.588 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:24:23 smithi090 ceph-mon[31674]: from='client.? 172.21.15.90:0/1488833414' entity='client.admin' 2024-04-02T14:24:23.991 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:22.975+0000 7f0d33400700 1 Processor -- start 2024-04-02T14:24:23.996 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:22.975+0000 7f0d33400700 1 -- start start 2024-04-02T14:24:23.996 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:22.976+0000 7f0d33400700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0d2c108cd0 0x7f0d2c10b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:24:23.997 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:22.976+0000 7f0d33400700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f0d2c07bec0 con 0x7f0d2c108cd0 2024-04-02T14:24:23.997 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:22.976+0000 7f0d30a00700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0d2c108cd0 0x7f0d2c10b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:24:23.997 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:22.976+0000 7f0d30a00700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0d2c108cd0 0x7f0d2c10b0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:51978/0 (socket says 172.21.15.90:51978) 2024-04-02T14:24:23.997 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:22.976+0000 7f0d30a00700 1 -- 172.21.15.90:0/2437313059 learned_addr learned my addr 172.21.15.90:0/2437313059 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:24:23.997 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:22.977+0000 7f0d30a00700 1 -- 172.21.15.90:0/2437313059 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d2c10b600 con 0x7f0d2c108cd0 2024-04-02T14:24:23.997 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:22.977+0000 7f0d30a00700 1 --2- 172.21.15.90:0/2437313059 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0d2c108cd0 0x7f0d2c10b0c0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f0d200098a0 tx=0x7f0d20009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c210617a7ab8073 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:24:23.997 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:22.977+0000 7f0d2b400700 1 -- 172.21.15.90:0/2437313059 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0d20004160 con 0x7f0d2c108cd0 2024-04-02T14:24:23.997 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:22.977+0000 7f0d2b400700 1 -- 172.21.15.90:0/2437313059 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0d20004300 con 0x7f0d2c108cd0 2024-04-02T14:24:23.997 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:22.977+0000 7f0d2b400700 1 -- 172.21.15.90:0/2437313059 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0d20011b20 con 0x7f0d2c108cd0 2024-04-02T14:24:23.997 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:22.978+0000 7f0d33400700 1 -- 172.21.15.90:0/2437313059 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0d2c108cd0 msgr2=0x7f0d2c10b0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:24:23.997 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:22.978+0000 7f0d33400700 1 --2- 172.21.15.90:0/2437313059 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0d2c108cd0 0x7f0d2c10b0c0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f0d200098a0 tx=0x7f0d20009bb0 comp rx=0 tx=0).stop 2024-04-02T14:24:23.997 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:22.978+0000 7f0d33400700 1 -- 172.21.15.90:0/2437313059 shutdown_connections 2024-04-02T14:24:23.997 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:22.978+0000 7f0d33400700 1 --2- 172.21.15.90:0/2437313059 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0d2c108cd0 0x7f0d2c10b0c0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:24:23.998 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:22.978+0000 7f0d33400700 1 -- 172.21.15.90:0/2437313059 >> 172.21.15.90:0/2437313059 conn(0x7f0d2c102760 msgr2=0x7f0d2c104b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:24:23.998 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:22.978+0000 7f0d33400700 1 -- 172.21.15.90:0/2437313059 shutdown_connections 2024-04-02T14:24:23.998 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:22.978+0000 7f0d33400700 1 -- 172.21.15.90:0/2437313059 wait complete. 2024-04-02T14:24:23.998 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:22.979+0000 7f0d33400700 1 Processor -- start 2024-04-02T14:24:23.998 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:22.980+0000 7f0d33400700 1 -- start start 2024-04-02T14:24:23.998 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:22.980+0000 7f0d33400700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0d2c108cd0 0x7f0d2c19e440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:24:23.998 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:22.980+0000 7f0d33400700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f0d2c10c670 con 0x7f0d2c108cd0 2024-04-02T14:24:23.998 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:22.980+0000 7f0d30a00700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0d2c108cd0 0x7f0d2c19e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:24:23.998 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:22.980+0000 7f0d30a00700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0d2c108cd0 0x7f0d2c19e440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:51988/0 (socket says 172.21.15.90:51988) 2024-04-02T14:24:23.998 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:22.980+0000 7f0d30a00700 1 -- 172.21.15.90:0/1488833414 learned_addr learned my addr 172.21.15.90:0/1488833414 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:24:23.998 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:22.980+0000 7f0d30a00700 1 -- 172.21.15.90:0/1488833414 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d2c19e980 con 0x7f0d2c108cd0 2024-04-02T14:24:23.998 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:22.981+0000 7f0d30a00700 1 --2- 172.21.15.90:0/1488833414 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0d2c108cd0 0x7f0d2c19e440 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f0d200116a0 tx=0x7f0d20011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:24:23.998 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:22.981+0000 7f0d29600700 1 -- 172.21.15.90:0/1488833414 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0d20003a40 con 0x7f0d2c108cd0 2024-04-02T14:24:23.998 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:22.981+0000 7f0d29600700 1 -- 172.21.15.90:0/1488833414 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0d20003be0 con 0x7f0d2c108cd0 2024-04-02T14:24:23.999 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:22.981+0000 7f0d29600700 1 -- 172.21.15.90:0/1488833414 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0d2001a750 con 0x7f0d2c108cd0 2024-04-02T14:24:23.999 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:22.981+0000 7f0d33400700 1 -- 172.21.15.90:0/1488833414 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d2c19ec10 con 0x7f0d2c108cd0 2024-04-02T14:24:23.999 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:22.981+0000 7f0d33400700 1 -- 172.21.15.90:0/1488833414 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d2c19f070 con 0x7f0d2c108cd0 2024-04-02T14:24:23.999 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:22.983+0000 7f0d33400700 1 -- 172.21.15.90:0/1488833414 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0d2c063c20 con 0x7f0d2c108cd0 2024-04-02T14:24:23.999 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:22.983+0000 7f0d29600700 1 -- 172.21.15.90:0/1488833414 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f0d20024aa0 con 0x7f0d2c108cd0 2024-04-02T14:24:23.999 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:22.983+0000 7f0d29600700 1 --2- 172.21.15.90:0/1488833414 >> [v2:172.21.15.90:6800/129436436,v1:172.21.15.90:6801/129436436] conn(0x7f0d14039560 0x7f0d1403ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:24:24.001 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:22.983+0000 7f0d29600700 1 -- 172.21.15.90:0/1488833414 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f0d2004de70 con 0x7f0d2c108cd0 2024-04-02T14:24:24.001 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:22.983+0000 7f0d2be00700 1 --2- 172.21.15.90:0/1488833414 >> [v2:172.21.15.90:6800/129436436,v1:172.21.15.90:6801/129436436] conn(0x7f0d14039560 0x7f0d1403ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:24:24.001 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:22.984+0000 7f0d2be00700 1 --2- 172.21.15.90:0/1488833414 >> [v2:172.21.15.90:6800/129436436,v1:172.21.15.90:6801/129436436] conn(0x7f0d14039560 0x7f0d1403ba20 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f0d18006fd0 tx=0x7f0d18006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:24:24.002 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:22.988+0000 7f0d29600700 1 -- 172.21.15.90:0/1488833414 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7f0d20011e20 con 0x7f0d2c108cd0 2024-04-02T14:24:24.002 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:23.219+0000 7f0d33400700 1 -- 172.21.15.90:0/1488833414 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) -- 0x7f0d2c1a1af0 con 0x7f0d2c108cd0 2024-04-02T14:24:24.002 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:23.222+0000 7f0d29600700 1 -- 172.21.15.90:0/1488833414 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) ==== 130+0+0 (secure 0 0 0) 0x7f0d2004d740 con 0x7f0d2c108cd0 2024-04-02T14:24:24.002 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:23.225+0000 7f0d33400700 1 -- 172.21.15.90:0/1488833414 >> [v2:172.21.15.90:6800/129436436,v1:172.21.15.90:6801/129436436] conn(0x7f0d14039560 msgr2=0x7f0d1403ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:24:24.002 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:23.225+0000 7f0d33400700 1 --2- 172.21.15.90:0/1488833414 >> [v2:172.21.15.90:6800/129436436,v1:172.21.15.90:6801/129436436] conn(0x7f0d14039560 0x7f0d1403ba20 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f0d18006fd0 tx=0x7f0d18006e40 comp rx=0 tx=0).stop 2024-04-02T14:24:24.002 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:23.225+0000 7f0d33400700 1 -- 172.21.15.90:0/1488833414 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0d2c108cd0 msgr2=0x7f0d2c19e440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:24:24.002 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:23.225+0000 7f0d33400700 1 --2- 172.21.15.90:0/1488833414 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0d2c108cd0 0x7f0d2c19e440 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f0d200116a0 tx=0x7f0d20011ba0 comp rx=0 tx=0).stop 2024-04-02T14:24:24.002 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:23.225+0000 7f0d33400700 1 -- 172.21.15.90:0/1488833414 shutdown_connections 2024-04-02T14:24:24.002 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:23.225+0000 7f0d33400700 1 --2- 172.21.15.90:0/1488833414 >> [v2:172.21.15.90:6800/129436436,v1:172.21.15.90:6801/129436436] conn(0x7f0d14039560 0x7f0d1403ba20 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:24:24.003 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:23.225+0000 7f0d33400700 1 --2- 172.21.15.90:0/1488833414 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0d2c108cd0 0x7f0d2c19e440 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:24:24.003 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:23.225+0000 7f0d33400700 1 -- 172.21.15.90:0/1488833414 >> 172.21.15.90:0/1488833414 conn(0x7f0d2c102760 msgr2=0x7f0d2c1031d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:24:24.003 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:23.226+0000 7f0d33400700 1 -- 172.21.15.90:0/1488833414 shutdown_connections 2024-04-02T14:24:24.003 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:23.226+0000 7f0d33400700 1 -- 172.21.15.90:0/1488833414 wait complete. 2024-04-02T14:24:24.003 INFO:teuthology.orchestra.run.smithi090.stdout:Enabling the dashboard module... 2024-04-02T14:24:25.811 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:24:25 smithi090 ceph-mon[31674]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T14:24:25.811 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:24:25 smithi090 ceph-mon[31674]: from='mgr.14120 172.21.15.90:0/2851514084' entity='mgr.x' 2024-04-02T14:24:25.812 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:24:25 smithi090 ceph-mon[31674]: from='client.? 172.21.15.90:0/3208063140' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-04-02T14:24:26.115 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 02 14:24:25 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-x[31910]: ignoring --setuser ceph since I am not root 2024-04-02T14:24:26.115 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 02 14:24:25 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-x[31910]: ignoring --setgroup ceph since I am not root 2024-04-02T14:24:26.938 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:24:26 smithi090 ceph-mon[31674]: from='client.? 172.21.15.90:0/3208063140' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-04-02T14:24:26.939 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:24:26 smithi090 ceph-mon[31674]: mgrmap e9: x(active, since 25s) 2024-04-02T14:24:26.939 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 02 14:24:26 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-x[31910]: 2024-04-02T14:24:26.513+0000 7fd595e4b200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-02T14:24:26.939 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 02 14:24:26 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-x[31910]: 2024-04-02T14:24:26.635+0000 7fd595e4b200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-02T14:24:27.136 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:24.619+0000 7f077ee00700 1 Processor -- start 2024-04-02T14:24:27.136 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:24.619+0000 7f077ee00700 1 -- start start 2024-04-02T14:24:27.136 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:24.620+0000 7f077ee00700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0778106b60 0x7f0778106f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:24:27.136 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:24.620+0000 7f077ee00700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f0778075e50 con 0x7f0778106b60 2024-04-02T14:24:27.136 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:24.620+0000 7f0777e00700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0778106b60 0x7f0778106f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:24:27.136 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:24.620+0000 7f0777e00700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0778106b60 0x7f0778106f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38394/0 (socket says 172.21.15.90:38394) 2024-04-02T14:24:27.136 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:24.620+0000 7f0777e00700 1 -- 172.21.15.90:0/3806496945 learned_addr learned my addr 172.21.15.90:0/3806496945 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:24:27.136 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:24.621+0000 7f0777e00700 1 -- 172.21.15.90:0/3806496945 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07781074c0 con 0x7f0778106b60 2024-04-02T14:24:27.136 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:24.621+0000 7f0777e00700 1 --2- 172.21.15.90:0/3806496945 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0778106b60 0x7f0778106f80 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f07680098a0 tx=0x7f0768009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb2858883efb44b8 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:24:27.136 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:24.621+0000 7f0776a00700 1 -- 172.21.15.90:0/3806496945 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0768004160 con 0x7f0778106b60 2024-04-02T14:24:27.136 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:24.621+0000 7f0776a00700 1 -- 172.21.15.90:0/3806496945 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0768004300 con 0x7f0778106b60 2024-04-02T14:24:27.137 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:24.621+0000 7f0776a00700 1 -- 172.21.15.90:0/3806496945 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0768011b20 con 0x7f0778106b60 2024-04-02T14:24:27.137 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:24.622+0000 7f077ee00700 1 -- 172.21.15.90:0/3806496945 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0778106b60 msgr2=0x7f0778106f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:24:27.137 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:24.622+0000 7f077ee00700 1 --2- 172.21.15.90:0/3806496945 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0778106b60 0x7f0778106f80 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f07680098a0 tx=0x7f0768009bb0 comp rx=0 tx=0).stop 2024-04-02T14:24:27.137 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:24.623+0000 7f077ee00700 1 -- 172.21.15.90:0/3806496945 shutdown_connections 2024-04-02T14:24:27.137 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:24.623+0000 7f077ee00700 1 --2- 172.21.15.90:0/3806496945 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0778106b60 0x7f0778106f80 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:24:27.137 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:24.623+0000 7f077ee00700 1 -- 172.21.15.90:0/3806496945 >> 172.21.15.90:0/3806496945 conn(0x7f0778102720 msgr2=0x7f0778104b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:24:27.137 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:24.623+0000 7f077ee00700 1 -- 172.21.15.90:0/3806496945 shutdown_connections 2024-04-02T14:24:27.137 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:24.623+0000 7f077ee00700 1 -- 172.21.15.90:0/3806496945 wait complete. 2024-04-02T14:24:27.137 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:24.624+0000 7f077ee00700 1 Processor -- start 2024-04-02T14:24:27.137 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:24.624+0000 7f077ee00700 1 -- start start 2024-04-02T14:24:27.137 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:24.625+0000 7f077ee00700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0778106b60 0x7f077819e3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:24:27.137 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:24.625+0000 7f077ee00700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f0778108590 con 0x7f0778106b60 2024-04-02T14:24:27.137 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:24.625+0000 7f0777e00700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0778106b60 0x7f077819e3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:24:27.137 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:24.625+0000 7f0777e00700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0778106b60 0x7f077819e3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38408/0 (socket says 172.21.15.90:38408) 2024-04-02T14:24:27.138 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:24.625+0000 7f0777e00700 1 -- 172.21.15.90:0/3208063140 learned_addr learned my addr 172.21.15.90:0/3208063140 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:24:27.138 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:24.626+0000 7f0777e00700 1 -- 172.21.15.90:0/3208063140 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f077819e8f0 con 0x7f0778106b60 2024-04-02T14:24:27.138 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:24.626+0000 7f0777e00700 1 --2- 172.21.15.90:0/3208063140 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0778106b60 0x7f077819e3b0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f0768011b60 tx=0x7f0768011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:24:27.138 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:24.626+0000 7f0774c00700 1 -- 172.21.15.90:0/3208063140 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0768003b60 con 0x7f0778106b60 2024-04-02T14:24:27.138 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:24.626+0000 7f0774c00700 1 -- 172.21.15.90:0/3208063140 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f076801a910 con 0x7f0778106b60 2024-04-02T14:24:27.138 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:24.626+0000 7f077ee00700 1 -- 172.21.15.90:0/3208063140 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f077819eb80 con 0x7f0778106b60 2024-04-02T14:24:27.138 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:24.626+0000 7f0774c00700 1 -- 172.21.15.90:0/3208063140 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0768024930 con 0x7f0778106b60 2024-04-02T14:24:27.138 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:24.626+0000 7f077ee00700 1 -- 172.21.15.90:0/3208063140 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f077819efe0 con 0x7f0778106b60 2024-04-02T14:24:27.138 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:24.628+0000 7f0774c00700 1 -- 172.21.15.90:0/3208063140 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f0768022020 con 0x7f0778106b60 2024-04-02T14:24:27.138 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:24.628+0000 7f077ee00700 1 -- 172.21.15.90:0/3208063140 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0778063c20 con 0x7f0778106b60 2024-04-02T14:24:27.138 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:24.628+0000 7f0774c00700 1 --2- 172.21.15.90:0/3208063140 >> [v2:172.21.15.90:6800/129436436,v1:172.21.15.90:6801/129436436] conn(0x7f075c039600 0x7f075c03bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:24:27.138 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:24.628+0000 7f0774c00700 1 -- 172.21.15.90:0/3208063140 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f076804d8f0 con 0x7f0778106b60 2024-04-02T14:24:27.138 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:24.629+0000 7f0777400700 1 --2- 172.21.15.90:0/3208063140 >> [v2:172.21.15.90:6800/129436436,v1:172.21.15.90:6801/129436436] conn(0x7f075c039600 0x7f075c03bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:24:27.139 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:24.629+0000 7f0777400700 1 --2- 172.21.15.90:0/3208063140 >> [v2:172.21.15.90:6800/129436436,v1:172.21.15.90:6801/129436436] conn(0x7f075c039600 0x7f075c03bac0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f0764006fd0 tx=0x7f0764006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:24:27.139 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:24.635+0000 7f0774c00700 1 -- 172.21.15.90:0/3208063140 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7f076800fdd0 con 0x7f0778106b60 2024-04-02T14:24:27.139 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:24.929+0000 7f077ee00700 1 -- 172.21.15.90:0/3208063140 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) -- 0x7f0778063200 con 0x7f0778106b60 2024-04-02T14:24:27.139 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:25.688+0000 7f0774c00700 1 -- 172.21.15.90:0/3208063140 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) ==== 88+0+0 (secure 0 0 0) 0x7f076800bc70 con 0x7f0778106b60 2024-04-02T14:24:27.139 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:25.692+0000 7f0774c00700 1 -- 172.21.15.90:0/3208063140 <== mon.0 v2:172.21.15.90:3300/0 8 ==== mgrmap(e 9) ==== 46571+0+0 (secure 0 0 0) 0x7f076802d770 con 0x7f0778106b60 2024-04-02T14:24:27.139 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:25.694+0000 7f077ee00700 1 -- 172.21.15.90:0/3208063140 >> [v2:172.21.15.90:6800/129436436,v1:172.21.15.90:6801/129436436] conn(0x7f075c039600 msgr2=0x7f075c03bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:24:27.139 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:25.694+0000 7f077ee00700 1 --2- 172.21.15.90:0/3208063140 >> [v2:172.21.15.90:6800/129436436,v1:172.21.15.90:6801/129436436] conn(0x7f075c039600 0x7f075c03bac0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f0764006fd0 tx=0x7f0764006e40 comp rx=0 tx=0).stop 2024-04-02T14:24:27.139 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:25.694+0000 7f077ee00700 1 -- 172.21.15.90:0/3208063140 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0778106b60 msgr2=0x7f077819e3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:24:27.139 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:25.694+0000 7f077ee00700 1 --2- 172.21.15.90:0/3208063140 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0778106b60 0x7f077819e3b0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f0768011b60 tx=0x7f0768011c40 comp rx=0 tx=0).stop 2024-04-02T14:24:27.139 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:25.695+0000 7f077ee00700 1 -- 172.21.15.90:0/3208063140 shutdown_connections 2024-04-02T14:24:27.139 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:25.695+0000 7f077ee00700 1 --2- 172.21.15.90:0/3208063140 >> [v2:172.21.15.90:6800/129436436,v1:172.21.15.90:6801/129436436] conn(0x7f075c039600 0x7f075c03bac0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:24:27.139 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:25.695+0000 7f077ee00700 1 --2- 172.21.15.90:0/3208063140 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0778106b60 0x7f077819e3b0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:24:27.139 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:25.695+0000 7f077ee00700 1 -- 172.21.15.90:0/3208063140 >> 172.21.15.90:0/3208063140 conn(0x7f0778102720 msgr2=0x7f0778194fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:24:27.139 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:25.695+0000 7f077ee00700 1 -- 172.21.15.90:0/3208063140 shutdown_connections 2024-04-02T14:24:27.139 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:25.695+0000 7f077ee00700 1 -- 172.21.15.90:0/3208063140 wait complete. 2024-04-02T14:24:27.689 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 02 14:24:27 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-x[31910]: 2024-04-02T14:24:27.426+0000 7fd595e4b200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-02T14:24:27.689 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 02 14:24:27 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-x[31910]: 2024-04-02T14:24:27.550+0000 7fd595e4b200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-02T14:24:27.975 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 02 14:24:27 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-x[31910]: 2024-04-02T14:24:27.732+0000 7fd595e4b200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-02T14:24:27.975 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 02 14:24:27 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-x[31910]: 2024-04-02T14:24:27.856+0000 7fd595e4b200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-02T14:24:27.976 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 02 14:24:27 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-x[31910]: 2024-04-02T14:24:27.972+0000 7fd595e4b200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-02T14:24:28.596 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:24:28 smithi090 ceph-mon[31674]: from='client.? 172.21.15.90:0/2187023304' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-02T14:24:28.596 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 02 14:24:28 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-x[31910]: 2024-04-02T14:24:28.337+0000 7fd595e4b200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-02T14:24:28.596 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 02 14:24:28 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-x[31910]: 2024-04-02T14:24:28.482+0000 7fd595e4b200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-02T14:24:28.596 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 02 14:24:28 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-x[31910]: 2024-04-02T14:24:28.594+0000 7fd595e4b200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-02T14:24:28.938 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 02 14:24:28 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-x[31910]: 2024-04-02T14:24:28.721+0000 7fd595e4b200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-02T14:24:29.082 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout { 2024-04-02T14:24:29.082 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-04-02T14:24:29.083 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-02T14:24:29.083 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-04-02T14:24:29.083 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-04-02T14:24:29.083 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout } 2024-04-02T14:24:29.083 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:28.044+0000 7fcaf7c00700 1 Processor -- start 2024-04-02T14:24:29.083 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:28.044+0000 7fcaf7c00700 1 -- start start 2024-04-02T14:24:29.083 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:28.045+0000 7fcaf7c00700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcaf007cda0 0x7fcaf007b200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:24:29.083 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:28.045+0000 7fcaf7c00700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fcaf007b7d0 con 0x7fcaf007cda0 2024-04-02T14:24:29.083 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:28.045+0000 7fcaf5200700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcaf007cda0 0x7fcaf007b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:24:29.083 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:28.045+0000 7fcaf5200700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcaf007cda0 0x7fcaf007b200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38444/0 (socket says 172.21.15.90:38444) 2024-04-02T14:24:29.083 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:28.045+0000 7fcaf5200700 1 -- 172.21.15.90:0/3604546113 learned_addr learned my addr 172.21.15.90:0/3604546113 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:24:29.083 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:28.045+0000 7fcaf5200700 1 -- 172.21.15.90:0/3604546113 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcaf007b950 con 0x7fcaf007cda0 2024-04-02T14:24:29.083 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:28.046+0000 7fcaf5200700 1 --2- 172.21.15.90:0/3604546113 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcaf007cda0 0x7fcaf007b200 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fcae40098a0 tx=0x7fcae4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3782c97978e825dc server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:24:29.083 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:28.046+0000 7fcaef400700 1 -- 172.21.15.90:0/3604546113 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcae4004160 con 0x7fcaf007cda0 2024-04-02T14:24:29.083 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:28.046+0000 7fcaef400700 1 -- 172.21.15.90:0/3604546113 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcae4004300 con 0x7fcaf007cda0 2024-04-02T14:24:29.084 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:28.046+0000 7fcaef400700 1 -- 172.21.15.90:0/3604546113 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcae4011b20 con 0x7fcaf007cda0 2024-04-02T14:24:29.084 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:28.047+0000 7fcaf7c00700 1 -- 172.21.15.90:0/3604546113 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcaf007cda0 msgr2=0x7fcaf007b200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:24:29.084 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:28.047+0000 7fcaf7c00700 1 --2- 172.21.15.90:0/3604546113 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcaf007cda0 0x7fcaf007b200 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fcae40098a0 tx=0x7fcae4009bb0 comp rx=0 tx=0).stop 2024-04-02T14:24:29.084 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:28.047+0000 7fcaf7c00700 1 -- 172.21.15.90:0/3604546113 shutdown_connections 2024-04-02T14:24:29.084 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:28.047+0000 7fcaf7c00700 1 --2- 172.21.15.90:0/3604546113 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcaf007cda0 0x7fcaf007b200 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:24:29.084 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:28.047+0000 7fcaf7c00700 1 -- 172.21.15.90:0/3604546113 >> 172.21.15.90:0/3604546113 conn(0x7fcaf0103830 msgr2=0x7fcaf0105c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:24:29.084 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:28.047+0000 7fcaf7c00700 1 -- 172.21.15.90:0/3604546113 shutdown_connections 2024-04-02T14:24:29.084 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:28.047+0000 7fcaf7c00700 1 -- 172.21.15.90:0/3604546113 wait complete. 2024-04-02T14:24:29.084 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:28.048+0000 7fcaf7c00700 1 Processor -- start 2024-04-02T14:24:29.084 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:28.048+0000 7fcaf7c00700 1 -- start start 2024-04-02T14:24:29.084 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:28.049+0000 7fcaf7c00700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcaf007cda0 0x7fcaf019e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:24:29.084 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:28.049+0000 7fcaf7c00700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fcaf010e8d0 con 0x7fcaf007cda0 2024-04-02T14:24:29.084 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:28.049+0000 7fcaf5200700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcaf007cda0 0x7fcaf019e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:24:29.084 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:28.049+0000 7fcaf5200700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcaf007cda0 0x7fcaf019e3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38458/0 (socket says 172.21.15.90:38458) 2024-04-02T14:24:29.085 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:28.049+0000 7fcaf5200700 1 -- 172.21.15.90:0/2187023304 learned_addr learned my addr 172.21.15.90:0/2187023304 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:24:29.085 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:28.049+0000 7fcaf5200700 1 -- 172.21.15.90:0/2187023304 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcaf019e920 con 0x7fcaf007cda0 2024-04-02T14:24:29.085 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:28.050+0000 7fcaf5200700 1 --2- 172.21.15.90:0/2187023304 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcaf007cda0 0x7fcaf019e3e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fcae40117e0 tx=0x7fcae4011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:24:29.085 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:28.050+0000 7fcaed600700 1 -- 172.21.15.90:0/2187023304 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcae4003bf0 con 0x7fcaf007cda0 2024-04-02T14:24:29.085 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:28.050+0000 7fcaed600700 1 -- 172.21.15.90:0/2187023304 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcae401a910 con 0x7fcaf007cda0 2024-04-02T14:24:29.085 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:28.050+0000 7fcaf7c00700 1 -- 172.21.15.90:0/2187023304 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcaf019ebb0 con 0x7fcaf007cda0 2024-04-02T14:24:29.085 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:28.050+0000 7fcaed600700 1 -- 172.21.15.90:0/2187023304 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcae40249a0 con 0x7fcaf007cda0 2024-04-02T14:24:29.085 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:28.050+0000 7fcaf7c00700 1 -- 172.21.15.90:0/2187023304 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcaf019f010 con 0x7fcaf007cda0 2024-04-02T14:24:29.085 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:28.051+0000 7fcaed600700 1 -- 172.21.15.90:0/2187023304 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 9) ==== 46571+0+0 (secure 0 0 0) 0x7fcae4022020 con 0x7fcaf007cda0 2024-04-02T14:24:29.085 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:28.051+0000 7fcaf7c00700 1 -- 172.21.15.90:0/2187023304 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcaf0063c20 con 0x7fcaf007cda0 2024-04-02T14:24:29.085 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:28.051+0000 7fcaed600700 1 --2- 172.21.15.90:0/2187023304 >> [v2:172.21.15.90:6800/129436436,v1:172.21.15.90:6801/129436436] conn(0x7fcad80395b0 0x7fcad803ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:24:29.085 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:28.052+0000 7fcaed600700 1 -- 172.21.15.90:0/2187023304 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fcae404dfb0 con 0x7fcaf007cda0 2024-04-02T14:24:29.085 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:28.052+0000 7fcaefe00700 1 -- 172.21.15.90:0/2187023304 >> [v2:172.21.15.90:6800/129436436,v1:172.21.15.90:6801/129436436] conn(0x7fcad80395b0 msgr2=0x7fcad803ba70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.90:6800/129436436 2024-04-02T14:24:29.085 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:28.052+0000 7fcaefe00700 1 --2- 172.21.15.90:0/2187023304 >> [v2:172.21.15.90:6800/129436436,v1:172.21.15.90:6801/129436436] conn(0x7fcad80395b0 0x7fcad803ba70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T14:24:29.086 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:28.057+0000 7fcaed600700 1 -- 172.21.15.90:0/2187023304 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7fcae4011e20 con 0x7fcaf007cda0 2024-04-02T14:24:29.086 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:28.253+0000 7fcaefe00700 1 -- 172.21.15.90:0/2187023304 >> [v2:172.21.15.90:6800/129436436,v1:172.21.15.90:6801/129436436] conn(0x7fcad80395b0 msgr2=0x7fcad803ba70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.90:6800/129436436 2024-04-02T14:24:29.086 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:28.253+0000 7fcaefe00700 1 --2- 172.21.15.90:0/2187023304 >> [v2:172.21.15.90:6800/129436436,v1:172.21.15.90:6801/129436436] conn(0x7fcad80395b0 0x7fcad803ba70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T14:24:29.086 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:28.349+0000 7fcaf7c00700 1 -- 172.21.15.90:0/2187023304 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7fcaf01a19c0 con 0x7fcaf007cda0 2024-04-02T14:24:29.086 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:28.350+0000 7fcaed600700 1 -- 172.21.15.90:0/2187023304 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) ==== 56+0+88 (secure 0 0 0) 0x7fcae402c350 con 0x7fcaf007cda0 2024-04-02T14:24:29.086 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:28.353+0000 7fcaf7c00700 1 -- 172.21.15.90:0/2187023304 >> [v2:172.21.15.90:6800/129436436,v1:172.21.15.90:6801/129436436] conn(0x7fcad80395b0 msgr2=0x7fcad803ba70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T14:24:29.086 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:28.353+0000 7fcaf7c00700 1 --2- 172.21.15.90:0/2187023304 >> [v2:172.21.15.90:6800/129436436,v1:172.21.15.90:6801/129436436] conn(0x7fcad80395b0 0x7fcad803ba70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:24:29.086 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:28.353+0000 7fcaf7c00700 1 -- 172.21.15.90:0/2187023304 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcaf007cda0 msgr2=0x7fcaf019e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:24:29.086 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:28.353+0000 7fcaf7c00700 1 --2- 172.21.15.90:0/2187023304 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcaf007cda0 0x7fcaf019e3e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fcae40117e0 tx=0x7fcae4011c10 comp rx=0 tx=0).stop 2024-04-02T14:24:29.086 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:28.353+0000 7fcaf7c00700 1 -- 172.21.15.90:0/2187023304 shutdown_connections 2024-04-02T14:24:29.086 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:28.353+0000 7fcaf7c00700 1 --2- 172.21.15.90:0/2187023304 >> [v2:172.21.15.90:6800/129436436,v1:172.21.15.90:6801/129436436] conn(0x7fcad80395b0 0x7fcad803ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:24:29.086 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:28.353+0000 7fcaf7c00700 1 --2- 172.21.15.90:0/2187023304 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcaf007cda0 0x7fcaf019e3e0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:24:29.086 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:28.353+0000 7fcaf7c00700 1 -- 172.21.15.90:0/2187023304 >> 172.21.15.90:0/2187023304 conn(0x7fcaf0103830 msgr2=0x7fcaf0104300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:24:29.086 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:28.353+0000 7fcaf7c00700 1 -- 172.21.15.90:0/2187023304 shutdown_connections 2024-04-02T14:24:29.086 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:28.353+0000 7fcaf7c00700 1 -- 172.21.15.90:0/2187023304 wait complete. 2024-04-02T14:24:29.087 INFO:teuthology.orchestra.run.smithi090.stdout:Waiting for the mgr to restart... 2024-04-02T14:24:29.087 INFO:teuthology.orchestra.run.smithi090.stdout:Waiting for mgr epoch 9... 2024-04-02T14:24:29.219 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 02 14:24:28 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-x[31910]: 2024-04-02T14:24:28.942+0000 7fd595e4b200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-02T14:24:30.689 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 02 14:24:30 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-x[31910]: 2024-04-02T14:24:30.271+0000 7fd595e4b200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-02T14:24:30.689 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 02 14:24:30 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-x[31910]: 2024-04-02T14:24:30.442+0000 7fd595e4b200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-02T14:24:31.188 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 02 14:24:30 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-x[31910]: 2024-04-02T14:24:30.858+0000 7fd595e4b200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-02T14:24:31.188 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 02 14:24:30 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-x[31910]: 2024-04-02T14:24:30.974+0000 7fd595e4b200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-02T14:24:31.688 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 02 14:24:31 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-x[31910]: 2024-04-02T14:24:31.241+0000 7fd595e4b200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-02T14:24:31.688 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 02 14:24:31 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-x[31910]: 2024-04-02T14:24:31.482+0000 7fd595e4b200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-02T14:24:32.172 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 02 14:24:31 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-x[31910]: 2024-04-02T14:24:31.853+0000 7fd595e4b200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-02T14:24:32.438 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 02 14:24:32 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-x[31910]: 2024-04-02T14:24:32.170+0000 7fd595e4b200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-02T14:24:32.938 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 02 14:24:32 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-x[31910]: 2024-04-02T14:24:32.471+0000 7fd595e4b200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-02T14:24:33.938 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 02 14:24:33 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-x[31910]: 2024-04-02T14:24:33.482+0000 7fd595e4b200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-02T14:24:34.438 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 02 14:24:34 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-x[31910]: 2024-04-02T14:24:34.172+0000 7fd595e4b200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-02T14:24:34.439 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 02 14:24:34 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-x[31910]: 2024-04-02T14:24:34.299+0000 7fd595e4b200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-02T14:24:34.697 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 02 14:24:34 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-x[31910]: 2024-04-02T14:24:34.592+0000 7fd595e4b200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-02T14:24:34.697 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:24:34 smithi090 ceph-mon[31674]: Active manager daemon x restarted 2024-04-02T14:24:34.697 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:24:34 smithi090 ceph-mon[31674]: Activating manager daemon x 2024-04-02T14:24:34.697 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:24:34 smithi090 ceph-mon[31674]: osdmap e3: 0 total, 0 up, 0 in 2024-04-02T14:24:34.697 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:24:34 smithi090 ceph-mon[31674]: mgrmap e10: x(active, starting, since 0.00828464s) 2024-04-02T14:24:34.697 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:24:34 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-02T14:24:34.697 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:24:34 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-02T14:24:34.698 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:24:34 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-02T14:24:34.698 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:24:34 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-02T14:24:34.698 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:24:34 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-02T14:24:34.698 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:24:34 smithi090 ceph-mon[31674]: Manager daemon x is now available 2024-04-02T14:24:35.929 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:24:35 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T14:24:35.929 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:24:35 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-02T14:24:35.929 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:24:35 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-02T14:24:35.929 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:24:35 smithi090 ceph-mon[31674]: mgrmap e11: x(active, since 1.00962s) 2024-04-02T14:24:36.479 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout { 2024-04-02T14:24:36.479 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-04-02T14:24:36.479 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "initialized": true 2024-04-02T14:24:36.479 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout } 2024-04-02T14:24:36.480 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:29.623+0000 7f7945e00700 1 Processor -- start 2024-04-02T14:24:36.480 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:29.623+0000 7f7945e00700 1 -- start start 2024-04-02T14:24:36.480 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:29.624+0000 7f7945e00700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7940108270 0x7f7940108690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:24:36.480 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:29.624+0000 7f7945e00700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f7940108c60 con 0x7f7940108270 2024-04-02T14:24:36.480 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:29.624+0000 7f793ea00700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7940108270 0x7f7940108690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:24:36.480 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:29.624+0000 7f793ea00700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7940108270 0x7f7940108690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38464/0 (socket says 172.21.15.90:38464) 2024-04-02T14:24:36.480 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:29.624+0000 7f793ea00700 1 -- 172.21.15.90:0/2079869599 learned_addr learned my addr 172.21.15.90:0/2079869599 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:24:36.480 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:29.625+0000 7f793ea00700 1 -- 172.21.15.90:0/2079869599 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7940109470 con 0x7f7940108270 2024-04-02T14:24:36.480 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:29.625+0000 7f793ea00700 1 --2- 172.21.15.90:0/2079869599 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7940108270 0x7f7940108690 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f79300098a0 tx=0x7f7930009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7b489e93cc5c7bf server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:24:36.480 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:29.625+0000 7f793e000700 1 -- 172.21.15.90:0/2079869599 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7930004160 con 0x7f7940108270 2024-04-02T14:24:36.480 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:29.625+0000 7f793e000700 1 -- 172.21.15.90:0/2079869599 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7930004300 con 0x7f7940108270 2024-04-02T14:24:36.480 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:29.625+0000 7f793e000700 1 -- 172.21.15.90:0/2079869599 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7930011b20 con 0x7f7940108270 2024-04-02T14:24:36.480 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:29.626+0000 7f7945e00700 1 -- 172.21.15.90:0/2079869599 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7940108270 msgr2=0x7f7940108690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:24:36.481 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:29.626+0000 7f7945e00700 1 --2- 172.21.15.90:0/2079869599 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7940108270 0x7f7940108690 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f79300098a0 tx=0x7f7930009bb0 comp rx=0 tx=0).stop 2024-04-02T14:24:36.481 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:29.626+0000 7f7945e00700 1 -- 172.21.15.90:0/2079869599 shutdown_connections 2024-04-02T14:24:36.481 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:29.626+0000 7f7945e00700 1 --2- 172.21.15.90:0/2079869599 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7940108270 0x7f7940108690 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:24:36.481 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:29.626+0000 7f7945e00700 1 -- 172.21.15.90:0/2079869599 >> 172.21.15.90:0/2079869599 conn(0x7f7940103830 msgr2=0x7f7940105c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:24:36.481 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:29.626+0000 7f7945e00700 1 -- 172.21.15.90:0/2079869599 shutdown_connections 2024-04-02T14:24:36.481 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:29.626+0000 7f7945e00700 1 -- 172.21.15.90:0/2079869599 wait complete. 2024-04-02T14:24:36.481 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:29.627+0000 7f7945e00700 1 Processor -- start 2024-04-02T14:24:36.481 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:29.627+0000 7f7945e00700 1 -- start start 2024-04-02T14:24:36.481 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:29.628+0000 7f7945e00700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7940108270 0x7f794019e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:24:36.481 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:29.628+0000 7f7945e00700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f794010c820 con 0x7f7940108270 2024-04-02T14:24:36.481 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:29.628+0000 7f793ea00700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7940108270 0x7f794019e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:24:36.481 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:29.628+0000 7f793ea00700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7940108270 0x7f794019e3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38466/0 (socket says 172.21.15.90:38466) 2024-04-02T14:24:36.481 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:29.628+0000 7f793ea00700 1 -- 172.21.15.90:0/2071975795 learned_addr learned my addr 172.21.15.90:0/2071975795 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:24:36.481 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:29.628+0000 7f793ea00700 1 -- 172.21.15.90:0/2071975795 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f794019e910 con 0x7f7940108270 2024-04-02T14:24:36.482 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:29.629+0000 7f793ea00700 1 --2- 172.21.15.90:0/2071975795 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7940108270 0x7f794019e3d0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f79300117e0 tx=0x7f7930011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:24:36.482 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:29.629+0000 7f7944a00700 1 -- 172.21.15.90:0/2071975795 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7930003bf0 con 0x7f7940108270 2024-04-02T14:24:36.482 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:29.629+0000 7f7944a00700 1 -- 172.21.15.90:0/2071975795 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f793001a910 con 0x7f7940108270 2024-04-02T14:24:36.482 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:29.629+0000 7f7944a00700 1 -- 172.21.15.90:0/2071975795 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f79300249a0 con 0x7f7940108270 2024-04-02T14:24:36.482 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:29.629+0000 7f7945e00700 1 -- 172.21.15.90:0/2071975795 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f794019eba0 con 0x7f7940108270 2024-04-02T14:24:36.482 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:29.629+0000 7f7945e00700 1 -- 172.21.15.90:0/2071975795 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f794019f000 con 0x7f7940108270 2024-04-02T14:24:36.482 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:29.630+0000 7f7944a00700 1 -- 172.21.15.90:0/2071975795 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 9) ==== 46571+0+0 (secure 0 0 0) 0x7f7930022020 con 0x7f7940108270 2024-04-02T14:24:36.482 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:29.631+0000 7f7944a00700 1 --2- 172.21.15.90:0/2071975795 >> [v2:172.21.15.90:6800/129436436,v1:172.21.15.90:6801/129436436] conn(0x7f79240395b0 0x7f792403ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:24:36.482 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:29.631+0000 7f7944a00700 1 -- 172.21.15.90:0/2071975795 --> [v2:172.21.15.90:6800/129436436,v1:172.21.15.90:6801/129436436] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f792403c180 con 0x7f79240395b0 2024-04-02T14:24:36.482 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:29.631+0000 7f7944a00700 1 -- 172.21.15.90:0/2071975795 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f793004db00 con 0x7f7940108270 2024-04-02T14:24:36.482 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:29.631+0000 7f7937e00700 1 -- 172.21.15.90:0/2071975795 >> [v2:172.21.15.90:6800/129436436,v1:172.21.15.90:6801/129436436] conn(0x7f79240395b0 msgr2=0x7f792403ba70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.90:6800/129436436 2024-04-02T14:24:36.482 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:29.631+0000 7f7937e00700 1 --2- 172.21.15.90:0/2071975795 >> [v2:172.21.15.90:6800/129436436,v1:172.21.15.90:6801/129436436] conn(0x7f79240395b0 0x7f792403ba70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T14:24:36.482 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:29.832+0000 7f7937e00700 1 -- 172.21.15.90:0/2071975795 >> [v2:172.21.15.90:6800/129436436,v1:172.21.15.90:6801/129436436] conn(0x7f79240395b0 msgr2=0x7f792403ba70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.90:6800/129436436 2024-04-02T14:24:36.483 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:29.832+0000 7f7937e00700 1 --2- 172.21.15.90:0/2071975795 >> [v2:172.21.15.90:6800/129436436,v1:172.21.15.90:6801/129436436] conn(0x7f79240395b0 0x7f792403ba70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T14:24:36.483 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:30.233+0000 7f7937e00700 1 -- 172.21.15.90:0/2071975795 >> [v2:172.21.15.90:6800/129436436,v1:172.21.15.90:6801/129436436] conn(0x7f79240395b0 msgr2=0x7f792403ba70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.90:6800/129436436 2024-04-02T14:24:36.483 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:30.233+0000 7f7937e00700 1 --2- 172.21.15.90:0/2071975795 >> [v2:172.21.15.90:6800/129436436,v1:172.21.15.90:6801/129436436] conn(0x7f79240395b0 0x7f792403ba70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-02T14:24:36.483 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:31.034+0000 7f7937e00700 1 -- 172.21.15.90:0/2071975795 >> [v2:172.21.15.90:6800/129436436,v1:172.21.15.90:6801/129436436] conn(0x7f79240395b0 msgr2=0x7f792403ba70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.90:6800/129436436 2024-04-02T14:24:36.483 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:31.034+0000 7f7937e00700 1 --2- 172.21.15.90:0/2071975795 >> [v2:172.21.15.90:6800/129436436,v1:172.21.15.90:6801/129436436] conn(0x7f79240395b0 0x7f792403ba70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-02T14:24:36.483 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:32.636+0000 7f7937e00700 1 -- 172.21.15.90:0/2071975795 >> [v2:172.21.15.90:6800/129436436,v1:172.21.15.90:6801/129436436] conn(0x7f79240395b0 msgr2=0x7f792403ba70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.90:6800/129436436 2024-04-02T14:24:36.483 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:32.636+0000 7f7937e00700 1 --2- 172.21.15.90:0/2071975795 >> [v2:172.21.15.90:6800/129436436,v1:172.21.15.90:6801/129436436] conn(0x7f79240395b0 0x7f792403ba70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-02T14:24:36.483 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:34.599+0000 7f7944a00700 1 -- 172.21.15.90:0/2071975795 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mgrmap(e 10) ==== 46338+0+0 (secure 0 0 0) 0x7f7930019ca0 con 0x7f7940108270 2024-04-02T14:24:36.483 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:34.599+0000 7f7944a00700 1 -- 172.21.15.90:0/2071975795 >> [v2:172.21.15.90:6800/129436436,v1:172.21.15.90:6801/129436436] conn(0x7f79240395b0 msgr2=0x7f792403ba70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T14:24:36.483 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:34.600+0000 7f7944a00700 1 --2- 172.21.15.90:0/2071975795 >> [v2:172.21.15.90:6800/129436436,v1:172.21.15.90:6801/129436436] conn(0x7f79240395b0 0x7f792403ba70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:24:36.483 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:35.601+0000 7f7944a00700 1 -- 172.21.15.90:0/2071975795 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mgrmap(e 11) ==== 46465+0+0 (secure 0 0 0) 0x7f793004f670 con 0x7f7940108270 2024-04-02T14:24:36.483 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:35.601+0000 7f7944a00700 1 --2- 172.21.15.90:0/2071975795 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f792403cfe0 0x7f792403f3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:24:36.483 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:35.601+0000 7f7944a00700 1 -- 172.21.15.90:0/2071975795 --> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f793001cec0 con 0x7f792403cfe0 2024-04-02T14:24:36.484 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:35.603+0000 7f7937e00700 1 --2- 172.21.15.90:0/2071975795 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f792403cfe0 0x7f792403f3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:24:36.484 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:35.605+0000 7f7937e00700 1 --2- 172.21.15.90:0/2071975795 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f792403cfe0 0x7f792403f3d0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f7928003a10 tx=0x7f79280092b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:24:36.484 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:35.607+0000 7f7944a00700 1 -- 172.21.15.90:0/2071975795 <== mgr.14152 v2:172.21.15.90:6800/4266054817 1 ==== command_reply(tid 0: 0 ) ==== 8+0+7774 (secure 0 0 0) 0x7f793001cec0 con 0x7f792403cfe0 2024-04-02T14:24:36.484 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:35.614+0000 7f7945e00700 1 -- 172.21.15.90:0/2071975795 --> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7f7940063c20 con 0x7f792403cfe0 2024-04-02T14:24:36.484 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:35.615+0000 7f7944a00700 1 -- 172.21.15.90:0/2071975795 <== mgr.14152 v2:172.21.15.90:6800/4266054817 2 ==== command_reply(tid 1: 0 ) ==== 8+0+52 (secure 0 0 0) 0x7f7940063c20 con 0x7f792403cfe0 2024-04-02T14:24:36.484 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:35.615+0000 7f7945e00700 1 -- 172.21.15.90:0/2071975795 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f792403cfe0 msgr2=0x7f792403f3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:24:36.484 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:35.615+0000 7f7945e00700 1 --2- 172.21.15.90:0/2071975795 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f792403cfe0 0x7f792403f3d0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f7928003a10 tx=0x7f79280092b0 comp rx=0 tx=0).stop 2024-04-02T14:24:36.484 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:35.615+0000 7f7945e00700 1 -- 172.21.15.90:0/2071975795 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7940108270 msgr2=0x7f794019e3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:24:36.484 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:35.615+0000 7f7945e00700 1 --2- 172.21.15.90:0/2071975795 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7940108270 0x7f794019e3d0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f79300117e0 tx=0x7f7930011c10 comp rx=0 tx=0).stop 2024-04-02T14:24:36.484 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:35.615+0000 7f7945e00700 1 -- 172.21.15.90:0/2071975795 shutdown_connections 2024-04-02T14:24:36.484 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:35.615+0000 7f7945e00700 1 --2- 172.21.15.90:0/2071975795 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f792403cfe0 0x7f792403f3d0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:24:36.484 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:35.615+0000 7f7945e00700 1 --2- 172.21.15.90:0/2071975795 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7940108270 0x7f794019e3d0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:24:36.484 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:35.615+0000 7f7945e00700 1 --2- 172.21.15.90:0/2071975795 >> [v2:172.21.15.90:6800/129436436,v1:172.21.15.90:6801/129436436] conn(0x7f79240395b0 0x7f792403ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:24:36.484 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:35.615+0000 7f7945e00700 1 -- 172.21.15.90:0/2071975795 >> 172.21.15.90:0/2071975795 conn(0x7f7940103830 msgr2=0x7f7940104220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:24:36.485 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:35.616+0000 7f7945e00700 1 -- 172.21.15.90:0/2071975795 shutdown_connections 2024-04-02T14:24:36.485 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:35.616+0000 7f7945e00700 1 -- 172.21.15.90:0/2071975795 wait complete. 2024-04-02T14:24:36.485 INFO:teuthology.orchestra.run.smithi090.stdout:mgr epoch 9 is available 2024-04-02T14:24:36.485 INFO:teuthology.orchestra.run.smithi090.stdout:Generating a dashboard self-signed certificate... 2024-04-02T14:24:36.739 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:24:36 smithi090 ceph-mon[31674]: [02/Apr/2024:14:24:35] ENGINE Bus STARTING 2024-04-02T14:24:36.739 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:24:36 smithi090 ceph-mon[31674]: [02/Apr/2024:14:24:35] ENGINE Serving on http://172.21.15.90:8765 2024-04-02T14:24:36.739 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:24:36 smithi090 ceph-mon[31674]: [02/Apr/2024:14:24:35] ENGINE Serving on https://172.21.15.90:7150 2024-04-02T14:24:36.739 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:24:36 smithi090 ceph-mon[31674]: [02/Apr/2024:14:24:35] ENGINE Bus STARTED 2024-04-02T14:24:36.739 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:24:36 smithi090 ceph-mon[31674]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-02T14:24:36.740 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:24:36 smithi090 ceph-mon[31674]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-02T14:24:37.688 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:24:37 smithi090 ceph-mon[31674]: mgrmap e12: x(active, since 2s) 2024-04-02T14:24:37.689 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:24:37 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:24:38.556 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-04-02T14:24:38.557 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:37.580+0000 7fcaa6000700 1 Processor -- start 2024-04-02T14:24:38.557 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:37.580+0000 7fcaa6000700 1 -- start start 2024-04-02T14:24:38.557 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:37.581+0000 7fcaa6000700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcaa010a470 0x7fcaa010a890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:24:38.557 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:37.581+0000 7fcaa6000700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fcaa010ae60 con 0x7fcaa010a470 2024-04-02T14:24:38.557 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:37.581+0000 7fca9f400700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcaa010a470 0x7fcaa010a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:24:38.557 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:37.581+0000 7fca9f400700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcaa010a470 0x7fcaa010a890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:46762/0 (socket says 172.21.15.90:46762) 2024-04-02T14:24:38.557 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:37.582+0000 7fca9f400700 1 -- 172.21.15.90:0/1152673119 learned_addr learned my addr 172.21.15.90:0/1152673119 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:24:38.557 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:37.582+0000 7fca9f400700 1 -- 172.21.15.90:0/1152673119 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcaa010b620 con 0x7fcaa010a470 2024-04-02T14:24:38.557 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:37.582+0000 7fca9f400700 1 --2- 172.21.15.90:0/1152673119 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcaa010a470 0x7fcaa010a890 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fca900098a0 tx=0x7fca90009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ba5290c81c0a7b7 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:24:38.557 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:37.583+0000 7fca9ea00700 1 -- 172.21.15.90:0/1152673119 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fca90004160 con 0x7fcaa010a470 2024-04-02T14:24:38.558 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:37.583+0000 7fca9ea00700 1 -- 172.21.15.90:0/1152673119 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fca90004300 con 0x7fcaa010a470 2024-04-02T14:24:38.558 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:37.583+0000 7fca9ea00700 1 -- 172.21.15.90:0/1152673119 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fca90011b20 con 0x7fcaa010a470 2024-04-02T14:24:38.558 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:37.583+0000 7fcaa6000700 1 -- 172.21.15.90:0/1152673119 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcaa010a470 msgr2=0x7fcaa010a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:24:38.558 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:37.583+0000 7fcaa6000700 1 --2- 172.21.15.90:0/1152673119 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcaa010a470 0x7fcaa010a890 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fca900098a0 tx=0x7fca90009bb0 comp rx=0 tx=0).stop 2024-04-02T14:24:38.558 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:37.583+0000 7fcaa6000700 1 -- 172.21.15.90:0/1152673119 shutdown_connections 2024-04-02T14:24:38.558 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:37.583+0000 7fcaa6000700 1 --2- 172.21.15.90:0/1152673119 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcaa010a470 0x7fcaa010a890 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:24:38.558 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:37.583+0000 7fcaa6000700 1 -- 172.21.15.90:0/1152673119 >> 172.21.15.90:0/1152673119 conn(0x7fcaa007d980 msgr2=0x7fcaa01084e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:24:38.559 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:37.584+0000 7fcaa6000700 1 -- 172.21.15.90:0/1152673119 shutdown_connections 2024-04-02T14:24:38.559 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:37.584+0000 7fcaa6000700 1 -- 172.21.15.90:0/1152673119 wait complete. 2024-04-02T14:24:38.559 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:37.585+0000 7fcaa6000700 1 Processor -- start 2024-04-02T14:24:38.559 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:37.585+0000 7fcaa6000700 1 -- start start 2024-04-02T14:24:38.559 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:37.585+0000 7fcaa6000700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcaa010a470 0x7fcaa019e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:24:38.559 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:37.585+0000 7fcaa6000700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fcaa010e9d0 con 0x7fcaa010a470 2024-04-02T14:24:38.559 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:37.585+0000 7fca9f400700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcaa010a470 0x7fcaa019e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:24:38.560 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:37.586+0000 7fca9f400700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcaa010a470 0x7fcaa019e320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:46768/0 (socket says 172.21.15.90:46768) 2024-04-02T14:24:38.560 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:37.586+0000 7fca9f400700 1 -- 172.21.15.90:0/4056756320 learned_addr learned my addr 172.21.15.90:0/4056756320 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:24:38.560 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:37.586+0000 7fca9f400700 1 -- 172.21.15.90:0/4056756320 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcaa019e860 con 0x7fcaa010a470 2024-04-02T14:24:38.560 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:37.586+0000 7fca9f400700 1 --2- 172.21.15.90:0/4056756320 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcaa010a470 0x7fcaa019e320 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fca900117e0 tx=0x7fca90011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:24:38.560 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:37.586+0000 7fca9cc00700 1 -- 172.21.15.90:0/4056756320 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fca90003bf0 con 0x7fcaa010a470 2024-04-02T14:24:38.560 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:37.587+0000 7fca9cc00700 1 -- 172.21.15.90:0/4056756320 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fca9001a910 con 0x7fcaa010a470 2024-04-02T14:24:38.560 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:37.587+0000 7fcaa6000700 1 -- 172.21.15.90:0/4056756320 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcaa019eaf0 con 0x7fcaa010a470 2024-04-02T14:24:38.560 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:37.587+0000 7fca9cc00700 1 -- 172.21.15.90:0/4056756320 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fca90024970 con 0x7fcaa010a470 2024-04-02T14:24:38.561 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:37.587+0000 7fcaa6000700 1 -- 172.21.15.90:0/4056756320 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcaa019ef50 con 0x7fcaa010a470 2024-04-02T14:24:38.561 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:37.588+0000 7fcaa6000700 1 -- 172.21.15.90:0/4056756320 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcaa0063c20 con 0x7fcaa010a470 2024-04-02T14:24:38.561 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:37.588+0000 7fca9cc00700 1 -- 172.21.15.90:0/4056756320 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 12) ==== 46571+0+0 (secure 0 0 0) 0x7fca90022020 con 0x7fcaa010a470 2024-04-02T14:24:38.561 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:37.589+0000 7fca9cc00700 1 --2- 172.21.15.90:0/4056756320 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fca84039660 0x7fca8403bb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:24:38.561 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:37.589+0000 7fca9cc00700 1 -- 172.21.15.90:0/4056756320 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fca9004dce0 con 0x7fcaa010a470 2024-04-02T14:24:38.561 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:37.589+0000 7fca97e00700 1 --2- 172.21.15.90:0/4056756320 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fca84039660 0x7fca8403bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:24:38.561 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:37.590+0000 7fca97e00700 1 --2- 172.21.15.90:0/4056756320 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fca84039660 0x7fca8403bb20 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fca88006fd0 tx=0x7fca88006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:24:38.561 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:37.595+0000 7fca9cc00700 1 -- 172.21.15.90:0/4056756320 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7fca9002d910 con 0x7fcaa010a470 2024-04-02T14:24:38.562 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:37.837+0000 7fcaa6000700 1 -- 172.21.15.90:0/4056756320 --> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) -- 0x7fcaa0108220 con 0x7fca84039660 2024-04-02T14:24:38.562 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:37.924+0000 7fca9cc00700 1 -- 172.21.15.90:0/4056756320 <== mgr.14152 v2:172.21.15.90:6800/4266054817 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+32 (secure 0 0 0) 0x7fcaa0108220 con 0x7fca84039660 2024-04-02T14:24:38.562 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:37.927+0000 7fcaa6000700 1 -- 172.21.15.90:0/4056756320 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fca84039660 msgr2=0x7fca8403bb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:24:38.562 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:37.927+0000 7fcaa6000700 1 --2- 172.21.15.90:0/4056756320 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fca84039660 0x7fca8403bb20 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fca88006fd0 tx=0x7fca88006e40 comp rx=0 tx=0).stop 2024-04-02T14:24:38.562 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:37.927+0000 7fcaa6000700 1 -- 172.21.15.90:0/4056756320 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcaa010a470 msgr2=0x7fcaa019e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:24:38.562 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:37.927+0000 7fcaa6000700 1 --2- 172.21.15.90:0/4056756320 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcaa010a470 0x7fcaa019e320 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fca900117e0 tx=0x7fca90011ba0 comp rx=0 tx=0).stop 2024-04-02T14:24:38.562 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:37.927+0000 7fcaa6000700 1 -- 172.21.15.90:0/4056756320 shutdown_connections 2024-04-02T14:24:38.562 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:37.927+0000 7fcaa6000700 1 --2- 172.21.15.90:0/4056756320 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fca84039660 0x7fca8403bb20 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:24:38.563 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:37.927+0000 7fcaa6000700 1 --2- 172.21.15.90:0/4056756320 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcaa010a470 0x7fcaa019e320 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:24:38.563 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:37.927+0000 7fcaa6000700 1 -- 172.21.15.90:0/4056756320 >> 172.21.15.90:0/4056756320 conn(0x7fcaa007d980 msgr2=0x7fcaa0107b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:24:38.563 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:37.927+0000 7fcaa6000700 1 -- 172.21.15.90:0/4056756320 shutdown_connections 2024-04-02T14:24:38.563 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:37.927+0000 7fcaa6000700 1 -- 172.21.15.90:0/4056756320 wait complete. 2024-04-02T14:24:38.563 INFO:teuthology.orchestra.run.smithi090.stdout:Creating initial admin user... 2024-04-02T14:24:39.094 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:24:38 smithi090 ceph-mon[31674]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T14:24:39.094 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:24:38 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:24:39.094 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:24:38 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:24:40.351 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$oibc/BCXeBOqlbA4Oig0Du9vGE8BXKE7nN9eUPhdTafxphx/VAxj6", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1712067879, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-04-02T14:24:40.352 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:39.185+0000 7f529f200700 1 Processor -- start 2024-04-02T14:24:40.352 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:39.186+0000 7f529f200700 1 -- start start 2024-04-02T14:24:40.352 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:39.186+0000 7f529f200700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5298107170 0x7f5298107590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:24:40.352 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:39.186+0000 7f529f200700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f5298107b60 con 0x7f5298107170 2024-04-02T14:24:40.352 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:39.186+0000 7f5297e00700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5298107170 0x7f5298107590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:24:40.352 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:39.187+0000 7f5297e00700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5298107170 0x7f5298107590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:46770/0 (socket says 172.21.15.90:46770) 2024-04-02T14:24:40.354 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:39.187+0000 7f5297e00700 1 -- 172.21.15.90:0/3121453375 learned_addr learned my addr 172.21.15.90:0/3121453375 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:24:40.354 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:39.187+0000 7f5297e00700 1 -- 172.21.15.90:0/3121453375 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5298108370 con 0x7f5298107170 2024-04-02T14:24:40.354 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:39.187+0000 7f5297e00700 1 --2- 172.21.15.90:0/3121453375 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5298107170 0x7f5298107590 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f52880098a0 tx=0x7f5288009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76bc4ffe2f601d1b server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:24:40.354 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:39.188+0000 7f5296a00700 1 -- 172.21.15.90:0/3121453375 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5288004160 con 0x7f5298107170 2024-04-02T14:24:40.354 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:39.188+0000 7f5296a00700 1 -- 172.21.15.90:0/3121453375 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5288004300 con 0x7f5298107170 2024-04-02T14:24:40.354 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:39.188+0000 7f5296a00700 1 -- 172.21.15.90:0/3121453375 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5288011b20 con 0x7f5298107170 2024-04-02T14:24:40.354 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:39.188+0000 7f529f200700 1 -- 172.21.15.90:0/3121453375 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5298107170 msgr2=0x7f5298107590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:24:40.354 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:39.188+0000 7f529f200700 1 --2- 172.21.15.90:0/3121453375 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5298107170 0x7f5298107590 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f52880098a0 tx=0x7f5288009bb0 comp rx=0 tx=0).stop 2024-04-02T14:24:40.354 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:39.189+0000 7f529f200700 1 -- 172.21.15.90:0/3121453375 shutdown_connections 2024-04-02T14:24:40.354 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:39.189+0000 7f529f200700 1 --2- 172.21.15.90:0/3121453375 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5298107170 0x7f5298107590 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:24:40.354 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:39.189+0000 7f529f200700 1 -- 172.21.15.90:0/3121453375 >> 172.21.15.90:0/3121453375 conn(0x7f5298102730 msgr2=0x7f5298104b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:24:40.355 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:39.189+0000 7f529f200700 1 -- 172.21.15.90:0/3121453375 shutdown_connections 2024-04-02T14:24:40.357 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:39.189+0000 7f529f200700 1 -- 172.21.15.90:0/3121453375 wait complete. 2024-04-02T14:24:40.357 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:39.190+0000 7f529f200700 1 Processor -- start 2024-04-02T14:24:40.357 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:39.190+0000 7f529f200700 1 -- start start 2024-04-02T14:24:40.357 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:39.190+0000 7f529f200700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5298107170 0x7f529819e420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:24:40.357 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:39.190+0000 7f529f200700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f529810b720 con 0x7f5298107170 2024-04-02T14:24:40.358 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:39.190+0000 7f5297e00700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5298107170 0x7f529819e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:24:40.358 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:39.191+0000 7f5297e00700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5298107170 0x7f529819e420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:46780/0 (socket says 172.21.15.90:46780) 2024-04-02T14:24:40.358 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:39.191+0000 7f5297e00700 1 -- 172.21.15.90:0/1336675932 learned_addr learned my addr 172.21.15.90:0/1336675932 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:24:40.358 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:39.191+0000 7f5297e00700 1 -- 172.21.15.90:0/1336675932 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f529819e960 con 0x7f5298107170 2024-04-02T14:24:40.358 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:39.191+0000 7f5297e00700 1 --2- 172.21.15.90:0/1336675932 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5298107170 0x7f529819e420 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f5288000c00 tx=0x7f5288011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:24:40.358 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:39.191+0000 7f5294c00700 1 -- 172.21.15.90:0/1336675932 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5288003b60 con 0x7f5298107170 2024-04-02T14:24:40.358 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:39.192+0000 7f5294c00700 1 -- 172.21.15.90:0/1336675932 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f528801a910 con 0x7f5298107170 2024-04-02T14:24:40.358 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:39.192+0000 7f5294c00700 1 -- 172.21.15.90:0/1336675932 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5288024930 con 0x7f5298107170 2024-04-02T14:24:40.358 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:39.192+0000 7f529f200700 1 -- 172.21.15.90:0/1336675932 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f529819ebf0 con 0x7f5298107170 2024-04-02T14:24:40.358 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:39.192+0000 7f529f200700 1 -- 172.21.15.90:0/1336675932 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f529819f050 con 0x7f5298107170 2024-04-02T14:24:40.358 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:39.193+0000 7f5294c00700 1 -- 172.21.15.90:0/1336675932 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 12) ==== 46571+0+0 (secure 0 0 0) 0x7f5288022020 con 0x7f5298107170 2024-04-02T14:24:40.358 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:39.193+0000 7f529f200700 1 -- 172.21.15.90:0/1336675932 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5298063c20 con 0x7f5298107170 2024-04-02T14:24:40.358 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:39.193+0000 7f5294c00700 1 --2- 172.21.15.90:0/1336675932 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f527c039660 0x7f527c03bb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:24:40.359 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:39.193+0000 7f5294c00700 1 -- 172.21.15.90:0/1336675932 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f528804dc20 con 0x7f5298107170 2024-04-02T14:24:40.359 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:39.194+0000 7f5297400700 1 --2- 172.21.15.90:0/1336675932 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f527c039660 0x7f527c03bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:24:40.359 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:39.194+0000 7f5297400700 1 --2- 172.21.15.90:0/1336675932 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f527c039660 0x7f527c03bb20 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f5284006fd0 tx=0x7f5284006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:24:40.359 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:39.199+0000 7f5294c00700 1 -- 172.21.15.90:0/1336675932 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7f528800b5a0 con 0x7f5298107170 2024-04-02T14:24:40.359 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:39.446+0000 7f529f200700 1 -- 172.21.15.90:0/1336675932 --> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) -- 0x7f5298198650 con 0x7f527c039660 2024-04-02T14:24:40.359 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:39.685+0000 7f5294c00700 1 -- 172.21.15.90:0/1336675932 <== mgr.14152 v2:172.21.15.90:6800/4266054817 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+252 (secure 0 0 0) 0x7f5298198650 con 0x7f527c039660 2024-04-02T14:24:40.360 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:39.687+0000 7f529f200700 1 -- 172.21.15.90:0/1336675932 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f527c039660 msgr2=0x7f527c03bb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:24:40.360 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:39.687+0000 7f529f200700 1 --2- 172.21.15.90:0/1336675932 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f527c039660 0x7f527c03bb20 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f5284006fd0 tx=0x7f5284006e40 comp rx=0 tx=0).stop 2024-04-02T14:24:40.360 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:39.687+0000 7f529f200700 1 -- 172.21.15.90:0/1336675932 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5298107170 msgr2=0x7f529819e420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:24:40.360 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:39.687+0000 7f529f200700 1 --2- 172.21.15.90:0/1336675932 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5298107170 0x7f529819e420 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f5288000c00 tx=0x7f5288011c10 comp rx=0 tx=0).stop 2024-04-02T14:24:40.361 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:39.688+0000 7f529f200700 1 -- 172.21.15.90:0/1336675932 shutdown_connections 2024-04-02T14:24:40.361 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:39.688+0000 7f529f200700 1 --2- 172.21.15.90:0/1336675932 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f527c039660 0x7f527c03bb20 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:24:40.361 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:39.688+0000 7f529f200700 1 --2- 172.21.15.90:0/1336675932 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5298107170 0x7f529819e420 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:24:40.361 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:39.688+0000 7f529f200700 1 -- 172.21.15.90:0/1336675932 >> 172.21.15.90:0/1336675932 conn(0x7f5298102730 msgr2=0x7f52981035b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:24:40.361 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:39.689+0000 7f529f200700 1 -- 172.21.15.90:0/1336675932 shutdown_connections 2024-04-02T14:24:40.361 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:39.689+0000 7f529f200700 1 -- 172.21.15.90:0/1336675932 wait complete. 2024-04-02T14:24:40.361 INFO:teuthology.orchestra.run.smithi090.stdout:Fetching dashboard port number... 2024-04-02T14:24:40.660 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:24:40 smithi090 ceph-mon[31674]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T14:24:40.660 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:24:40 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:24:40.660 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:24:40 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:24:41.865 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:24:41 smithi090 ceph-mon[31674]: from='client.? 172.21.15.90:0/2629582378' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-04-02T14:24:41.881 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout 8443 2024-04-02T14:24:41.881 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:40.942+0000 7fb549400700 1 Processor -- start 2024-04-02T14:24:41.882 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:40.942+0000 7fb549400700 1 -- start start 2024-04-02T14:24:41.882 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:40.942+0000 7fb549400700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb544106b20 0x7fb544106f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:24:41.882 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:40.942+0000 7fb549400700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fb544075e90 con 0x7fb544106b20 2024-04-02T14:24:41.882 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:40.943+0000 7fb542800700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb544106b20 0x7fb544106f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:24:41.882 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:40.943+0000 7fb542800700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb544106b20 0x7fb544106f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:46790/0 (socket says 172.21.15.90:46790) 2024-04-02T14:24:41.882 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:40.943+0000 7fb542800700 1 -- 172.21.15.90:0/3415909037 learned_addr learned my addr 172.21.15.90:0/3415909037 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:24:41.882 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:40.943+0000 7fb542800700 1 -- 172.21.15.90:0/3415909037 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb544107480 con 0x7fb544106b20 2024-04-02T14:24:41.882 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:40.943+0000 7fb542800700 1 --2- 172.21.15.90:0/3415909037 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb544106b20 0x7fb544106f40 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fb5340098a0 tx=0x7fb534009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19378abcb60d2d27 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:24:41.882 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:40.944+0000 7fb541400700 1 -- 172.21.15.90:0/3415909037 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb534004160 con 0x7fb544106b20 2024-04-02T14:24:41.882 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:40.944+0000 7fb541400700 1 -- 172.21.15.90:0/3415909037 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb534004300 con 0x7fb544106b20 2024-04-02T14:24:41.882 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:40.944+0000 7fb541400700 1 -- 172.21.15.90:0/3415909037 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb534011b20 con 0x7fb544106b20 2024-04-02T14:24:41.882 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:40.944+0000 7fb549400700 1 -- 172.21.15.90:0/3415909037 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb544106b20 msgr2=0x7fb544106f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:24:41.882 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:40.944+0000 7fb549400700 1 --2- 172.21.15.90:0/3415909037 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb544106b20 0x7fb544106f40 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fb5340098a0 tx=0x7fb534009bb0 comp rx=0 tx=0).stop 2024-04-02T14:24:41.883 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:40.945+0000 7fb549400700 1 -- 172.21.15.90:0/3415909037 shutdown_connections 2024-04-02T14:24:41.883 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:40.945+0000 7fb549400700 1 --2- 172.21.15.90:0/3415909037 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb544106b20 0x7fb544106f40 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:24:41.883 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:40.945+0000 7fb549400700 1 -- 172.21.15.90:0/3415909037 >> 172.21.15.90:0/3415909037 conn(0x7fb544102720 msgr2=0x7fb544104b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:24:41.883 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:40.945+0000 7fb549400700 1 -- 172.21.15.90:0/3415909037 shutdown_connections 2024-04-02T14:24:41.883 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:40.945+0000 7fb549400700 1 -- 172.21.15.90:0/3415909037 wait complete. 2024-04-02T14:24:41.883 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:40.946+0000 7fb549400700 1 Processor -- start 2024-04-02T14:24:41.883 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:40.946+0000 7fb549400700 1 -- start start 2024-04-02T14:24:41.883 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:40.947+0000 7fb549400700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb544106b20 0x7fb54419e3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:24:41.883 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:40.947+0000 7fb549400700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fb544108550 con 0x7fb544106b20 2024-04-02T14:24:41.883 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:40.947+0000 7fb542800700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb544106b20 0x7fb54419e3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:24:41.883 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:40.947+0000 7fb542800700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb544106b20 0x7fb54419e3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:46796/0 (socket says 172.21.15.90:46796) 2024-04-02T14:24:41.883 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:40.947+0000 7fb542800700 1 -- 172.21.15.90:0/2629582378 learned_addr learned my addr 172.21.15.90:0/2629582378 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:24:41.883 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:40.947+0000 7fb542800700 1 -- 172.21.15.90:0/2629582378 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb54419e8f0 con 0x7fb544106b20 2024-04-02T14:24:41.883 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:40.948+0000 7fb542800700 1 --2- 172.21.15.90:0/2629582378 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb544106b20 0x7fb54419e3b0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fb534011b60 tx=0x7fb534011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:24:41.883 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:40.948+0000 7fb53b400700 1 -- 172.21.15.90:0/2629582378 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb534003b60 con 0x7fb544106b20 2024-04-02T14:24:41.884 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:40.948+0000 7fb53b400700 1 -- 172.21.15.90:0/2629582378 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb53401a910 con 0x7fb544106b20 2024-04-02T14:24:41.884 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:40.948+0000 7fb549400700 1 -- 172.21.15.90:0/2629582378 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb54419eb80 con 0x7fb544106b20 2024-04-02T14:24:41.884 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:40.948+0000 7fb53b400700 1 -- 172.21.15.90:0/2629582378 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb534024930 con 0x7fb544106b20 2024-04-02T14:24:41.884 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:40.948+0000 7fb549400700 1 -- 172.21.15.90:0/2629582378 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb54419efe0 con 0x7fb544106b20 2024-04-02T14:24:41.884 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:40.949+0000 7fb53b400700 1 -- 172.21.15.90:0/2629582378 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 12) ==== 46571+0+0 (secure 0 0 0) 0x7fb534022020 con 0x7fb544106b20 2024-04-02T14:24:41.884 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:40.949+0000 7fb549400700 1 -- 172.21.15.90:0/2629582378 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb544063c20 con 0x7fb544106b20 2024-04-02T14:24:41.884 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:40.950+0000 7fb53b400700 1 --2- 172.21.15.90:0/2629582378 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fb528039660 0x7fb52803bb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:24:41.884 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:40.950+0000 7fb53b400700 1 -- 172.21.15.90:0/2629582378 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fb53404dc10 con 0x7fb544106b20 2024-04-02T14:24:41.884 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:40.950+0000 7fb541e00700 1 --2- 172.21.15.90:0/2629582378 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fb528039660 0x7fb52803bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:24:41.884 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:40.951+0000 7fb541e00700 1 --2- 172.21.15.90:0/2629582378 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fb528039660 0x7fb52803bb20 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fb530006fd0 tx=0x7fb530006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:24:41.884 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:40.956+0000 7fb53b400700 1 -- 172.21.15.90:0/2629582378 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7fb53402d3f0 con 0x7fb544106b20 2024-04-02T14:24:41.884 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:41.195+0000 7fb549400700 1 -- 172.21.15.90:0/2629582378 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) -- 0x7fb544063200 con 0x7fb544106b20 2024-04-02T14:24:41.884 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:41.196+0000 7fb53b400700 1 -- 172.21.15.90:0/2629582378 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) ==== 112+0+5 (secure 0 0 0) 0x7fb534024ad0 con 0x7fb544106b20 2024-04-02T14:24:41.884 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:41.199+0000 7fb549400700 1 -- 172.21.15.90:0/2629582378 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fb528039660 msgr2=0x7fb52803bb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:24:41.885 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:41.199+0000 7fb549400700 1 --2- 172.21.15.90:0/2629582378 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fb528039660 0x7fb52803bb20 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fb530006fd0 tx=0x7fb530006e40 comp rx=0 tx=0).stop 2024-04-02T14:24:41.885 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:41.199+0000 7fb549400700 1 -- 172.21.15.90:0/2629582378 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb544106b20 msgr2=0x7fb54419e3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:24:41.885 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:41.199+0000 7fb549400700 1 --2- 172.21.15.90:0/2629582378 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb544106b20 0x7fb54419e3b0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fb534011b60 tx=0x7fb534011c40 comp rx=0 tx=0).stop 2024-04-02T14:24:41.885 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:41.199+0000 7fb549400700 1 -- 172.21.15.90:0/2629582378 shutdown_connections 2024-04-02T14:24:41.885 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:41.199+0000 7fb549400700 1 --2- 172.21.15.90:0/2629582378 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fb528039660 0x7fb52803bb20 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:24:41.885 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:41.199+0000 7fb549400700 1 --2- 172.21.15.90:0/2629582378 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb544106b20 0x7fb54419e3b0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:24:41.885 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:41.199+0000 7fb549400700 1 -- 172.21.15.90:0/2629582378 >> 172.21.15.90:0/2629582378 conn(0x7fb544102720 msgr2=0x7fb544194fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:24:41.885 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:41.199+0000 7fb549400700 1 -- 172.21.15.90:0/2629582378 shutdown_connections 2024-04-02T14:24:41.885 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:41.199+0000 7fb549400700 1 -- 172.21.15.90:0/2629582378 wait complete. 2024-04-02T14:24:41.905 INFO:teuthology.orchestra.run.smithi090.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-02T14:24:41.905 INFO:teuthology.orchestra.run.smithi090.stdout:systemctl: stdout disabled 2024-04-02T14:24:41.928 INFO:teuthology.orchestra.run.smithi090.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-02T14:24:41.929 INFO:teuthology.orchestra.run.smithi090.stdout:systemctl: stdout inactive 2024-04-02T14:24:41.929 INFO:teuthology.orchestra.run.smithi090.stdout:firewalld.service is not enabled 2024-04-02T14:24:41.929 INFO:teuthology.orchestra.run.smithi090.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-04-02T14:24:41.936 INFO:teuthology.orchestra.run.smithi090.stdout:Ceph Dashboard is now available at: 2024-04-02T14:24:41.937 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-02T14:24:41.937 INFO:teuthology.orchestra.run.smithi090.stdout: URL: https://smithi090.front.sepia.ceph.com:8443/ 2024-04-02T14:24:41.937 INFO:teuthology.orchestra.run.smithi090.stdout: User: admin 2024-04-02T14:24:41.937 INFO:teuthology.orchestra.run.smithi090.stdout: Password: 78qub9lle1 2024-04-02T14:24:41.937 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-02T14:24:41.938 INFO:teuthology.orchestra.run.smithi090.stdout:Saving cluster configuration to /var/lib/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/config directory 2024-04-02T14:24:41.940 INFO:teuthology.orchestra.run.smithi090.stdout:Enabling autotune for osd_memory_target 2024-04-02T14:24:42.690 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:24:42 smithi090 ceph-mon[31674]: mgrmap e13: x(active, since 6s) 2024-04-02T14:24:43.812 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:42.993+0000 7f1998400700 1 Processor -- start 2024-04-02T14:24:43.812 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:42.993+0000 7f1998400700 1 -- start start 2024-04-02T14:24:43.812 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:42.993+0000 7f1998400700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1990108240 0x7f1990108660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:24:43.813 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:42.993+0000 7f1998400700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f1990108c30 con 0x7f1990108240 2024-04-02T14:24:43.813 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:42.994+0000 7f1995a00700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1990108240 0x7f1990108660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:24:43.813 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:42.994+0000 7f1995a00700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1990108240 0x7f1990108660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:46806/0 (socket says 172.21.15.90:46806) 2024-04-02T14:24:43.813 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:42.994+0000 7f1995a00700 1 -- 172.21.15.90:0/714024845 learned_addr learned my addr 172.21.15.90:0/714024845 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:24:43.813 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:42.994+0000 7f1995a00700 1 -- 172.21.15.90:0/714024845 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19901093f0 con 0x7f1990108240 2024-04-02T14:24:43.815 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:42.995+0000 7f1995a00700 1 --2- 172.21.15.90:0/714024845 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1990108240 0x7f1990108660 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f19840098a0 tx=0x7f1984009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eae59650624c67da server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:24:43.815 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:42.995+0000 7f198fe00700 1 -- 172.21.15.90:0/714024845 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1984004160 con 0x7f1990108240 2024-04-02T14:24:43.815 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:42.995+0000 7f198fe00700 1 -- 172.21.15.90:0/714024845 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1984004300 con 0x7f1990108240 2024-04-02T14:24:43.815 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:42.995+0000 7f198fe00700 1 -- 172.21.15.90:0/714024845 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1984011b20 con 0x7f1990108240 2024-04-02T14:24:43.815 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:42.996+0000 7f1998400700 1 -- 172.21.15.90:0/714024845 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1990108240 msgr2=0x7f1990108660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:24:43.816 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:42.996+0000 7f1998400700 1 --2- 172.21.15.90:0/714024845 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1990108240 0x7f1990108660 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f19840098a0 tx=0x7f1984009bb0 comp rx=0 tx=0).stop 2024-04-02T14:24:43.816 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:42.996+0000 7f1998400700 1 -- 172.21.15.90:0/714024845 shutdown_connections 2024-04-02T14:24:43.816 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:42.996+0000 7f1998400700 1 --2- 172.21.15.90:0/714024845 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1990108240 0x7f1990108660 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:24:43.816 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:42.996+0000 7f1998400700 1 -- 172.21.15.90:0/714024845 >> 172.21.15.90:0/714024845 conn(0x7f1990103860 msgr2=0x7f1990105c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:24:43.816 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:42.996+0000 7f1998400700 1 -- 172.21.15.90:0/714024845 shutdown_connections 2024-04-02T14:24:43.816 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:42.996+0000 7f1998400700 1 -- 172.21.15.90:0/714024845 wait complete. 2024-04-02T14:24:43.816 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:42.997+0000 7f1998400700 1 Processor -- start 2024-04-02T14:24:43.816 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:42.997+0000 7f1998400700 1 -- start start 2024-04-02T14:24:43.816 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:42.997+0000 7f1998400700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1990108240 0x7f199019c430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:24:43.816 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:42.997+0000 7f1998400700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f199010c7a0 con 0x7f1990108240 2024-04-02T14:24:43.818 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:42.998+0000 7f1995a00700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1990108240 0x7f199019c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:24:43.819 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:42.998+0000 7f1995a00700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1990108240 0x7f199019c430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:46822/0 (socket says 172.21.15.90:46822) 2024-04-02T14:24:43.819 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:42.998+0000 7f1995a00700 1 -- 172.21.15.90:0/1562087834 learned_addr learned my addr 172.21.15.90:0/1562087834 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:24:43.819 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:42.998+0000 7f1995a00700 1 -- 172.21.15.90:0/1562087834 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f199019c970 con 0x7f1990108240 2024-04-02T14:24:43.819 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:42.998+0000 7f1995a00700 1 --2- 172.21.15.90:0/1562087834 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1990108240 0x7f199019c430 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f19840117e0 tx=0x7f1984011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:24:43.819 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:42.998+0000 7f198e000700 1 -- 172.21.15.90:0/1562087834 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1984003bf0 con 0x7f1990108240 2024-04-02T14:24:43.819 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:42.998+0000 7f198e000700 1 -- 172.21.15.90:0/1562087834 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f198401a910 con 0x7f1990108240 2024-04-02T14:24:43.819 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:42.999+0000 7f198e000700 1 -- 172.21.15.90:0/1562087834 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f19840249a0 con 0x7f1990108240 2024-04-02T14:24:43.819 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:42.999+0000 7f1998400700 1 -- 172.21.15.90:0/1562087834 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f199019cc00 con 0x7f1990108240 2024-04-02T14:24:43.819 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:42.999+0000 7f1998400700 1 -- 172.21.15.90:0/1562087834 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f199019d060 con 0x7f1990108240 2024-04-02T14:24:43.819 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:43.000+0000 7f198e000700 1 -- 172.21.15.90:0/1562087834 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f1984022020 con 0x7f1990108240 2024-04-02T14:24:43.819 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:43.000+0000 7f1998400700 1 -- 172.21.15.90:0/1562087834 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1990063c20 con 0x7f1990108240 2024-04-02T14:24:43.819 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:43.000+0000 7f198e000700 1 --2- 172.21.15.90:0/1562087834 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f19780396b0 0x7f197803bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:24:43.819 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:43.000+0000 7f198e000700 1 -- 172.21.15.90:0/1562087834 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f198404dc60 con 0x7f1990108240 2024-04-02T14:24:43.820 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:43.001+0000 7f1995000700 1 --2- 172.21.15.90:0/1562087834 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f19780396b0 0x7f197803bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:24:43.820 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:43.001+0000 7f1995000700 1 --2- 172.21.15.90:0/1562087834 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f19780396b0 0x7f197803bb70 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f197c006fd0 tx=0x7f197c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:24:43.820 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:43.005+0000 7f198e000700 1 -- 172.21.15.90:0/1562087834 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7f1984011e20 con 0x7f1990108240 2024-04-02T14:24:43.820 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:43.228+0000 7f1998400700 1 -- 172.21.15.90:0/1562087834 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) -- 0x7f199002de40 con 0x7f1990108240 2024-04-02T14:24:43.820 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:43.228+0000 7f198e000700 1 -- 172.21.15.90:0/1562087834 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) ==== 127+0+0 (secure 0 0 0) 0x7f1984004300 con 0x7f1990108240 2024-04-02T14:24:43.820 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:43.231+0000 7f1998400700 1 -- 172.21.15.90:0/1562087834 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f19780396b0 msgr2=0x7f197803bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:24:43.820 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:43.231+0000 7f1998400700 1 --2- 172.21.15.90:0/1562087834 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f19780396b0 0x7f197803bb70 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f197c006fd0 tx=0x7f197c006e40 comp rx=0 tx=0).stop 2024-04-02T14:24:43.821 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:43.231+0000 7f1998400700 1 -- 172.21.15.90:0/1562087834 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1990108240 msgr2=0x7f199019c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:24:43.821 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:43.231+0000 7f1998400700 1 --2- 172.21.15.90:0/1562087834 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1990108240 0x7f199019c430 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f19840117e0 tx=0x7f1984011c10 comp rx=0 tx=0).stop 2024-04-02T14:24:43.821 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:43.231+0000 7f1998400700 1 -- 172.21.15.90:0/1562087834 shutdown_connections 2024-04-02T14:24:43.821 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:43.231+0000 7f1998400700 1 --2- 172.21.15.90:0/1562087834 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f19780396b0 0x7f197803bb70 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:24:43.821 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:43.231+0000 7f1998400700 1 --2- 172.21.15.90:0/1562087834 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1990108240 0x7f199019c430 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:24:43.821 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:43.231+0000 7f1998400700 1 -- 172.21.15.90:0/1562087834 >> 172.21.15.90:0/1562087834 conn(0x7f1990103860 msgr2=0x7f1990104440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:24:43.821 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:43.232+0000 7f1998400700 1 -- 172.21.15.90:0/1562087834 shutdown_connections 2024-04-02T14:24:43.822 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:43.232+0000 7f1998400700 1 -- 172.21.15.90:0/1562087834 wait complete. 2024-04-02T14:24:45.365 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:44.359+0000 7efe13a00700 1 Processor -- start 2024-04-02T14:24:45.371 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:44.360+0000 7efe13a00700 1 -- start start 2024-04-02T14:24:45.371 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:44.360+0000 7efe13a00700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efe0c07cda0 0x7efe0c07b200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:24:45.371 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:44.360+0000 7efe13a00700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7efe0c07b7d0 con 0x7efe0c07cda0 2024-04-02T14:24:45.371 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:44.360+0000 7efe11000700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efe0c07cda0 0x7efe0c07b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:24:45.371 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:44.360+0000 7efe11000700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efe0c07cda0 0x7efe0c07b200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:39574/0 (socket says 172.21.15.90:39574) 2024-04-02T14:24:45.371 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:44.360+0000 7efe11000700 1 -- 172.21.15.90:0/3262090782 learned_addr learned my addr 172.21.15.90:0/3262090782 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:24:45.371 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:44.361+0000 7efe11000700 1 -- 172.21.15.90:0/3262090782 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe0c07b950 con 0x7efe0c07cda0 2024-04-02T14:24:45.371 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:44.361+0000 7efe11000700 1 --2- 172.21.15.90:0/3262090782 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efe0c07cda0 0x7efe0c07b200 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7efdfc0098a0 tx=0x7efdfc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15f2cb211620f2b4 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:24:45.371 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:44.361+0000 7efe0be00700 1 -- 172.21.15.90:0/3262090782 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efdfc004160 con 0x7efe0c07cda0 2024-04-02T14:24:45.371 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:44.361+0000 7efe0be00700 1 -- 172.21.15.90:0/3262090782 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efdfc004300 con 0x7efe0c07cda0 2024-04-02T14:24:45.371 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:44.361+0000 7efe0be00700 1 -- 172.21.15.90:0/3262090782 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efdfc011b20 con 0x7efe0c07cda0 2024-04-02T14:24:45.372 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:44.362+0000 7efe13a00700 1 -- 172.21.15.90:0/3262090782 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efe0c07cda0 msgr2=0x7efe0c07b200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:24:45.372 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:44.362+0000 7efe13a00700 1 --2- 172.21.15.90:0/3262090782 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efe0c07cda0 0x7efe0c07b200 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7efdfc0098a0 tx=0x7efdfc009bb0 comp rx=0 tx=0).stop 2024-04-02T14:24:45.372 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:44.362+0000 7efe13a00700 1 -- 172.21.15.90:0/3262090782 shutdown_connections 2024-04-02T14:24:45.372 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:44.362+0000 7efe13a00700 1 --2- 172.21.15.90:0/3262090782 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efe0c07cda0 0x7efe0c07b200 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:24:45.372 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:44.362+0000 7efe13a00700 1 -- 172.21.15.90:0/3262090782 >> 172.21.15.90:0/3262090782 conn(0x7efe0c103810 msgr2=0x7efe0c105c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:24:45.372 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:44.362+0000 7efe13a00700 1 -- 172.21.15.90:0/3262090782 shutdown_connections 2024-04-02T14:24:45.372 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:44.362+0000 7efe13a00700 1 -- 172.21.15.90:0/3262090782 wait complete. 2024-04-02T14:24:45.372 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:44.364+0000 7efe13a00700 1 Processor -- start 2024-04-02T14:24:45.372 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:44.364+0000 7efe13a00700 1 -- start start 2024-04-02T14:24:45.372 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:44.364+0000 7efe13a00700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efe0c07cda0 0x7efe0c19e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:24:45.372 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:44.364+0000 7efe13a00700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7efe0c10e890 con 0x7efe0c07cda0 2024-04-02T14:24:45.372 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:44.364+0000 7efe11000700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efe0c07cda0 0x7efe0c19e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:24:45.372 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:44.364+0000 7efe11000700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efe0c07cda0 0x7efe0c19e3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:39590/0 (socket says 172.21.15.90:39590) 2024-04-02T14:24:45.372 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:44.364+0000 7efe11000700 1 -- 172.21.15.90:0/634139116 learned_addr learned my addr 172.21.15.90:0/634139116 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:24:45.373 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:44.365+0000 7efe11000700 1 -- 172.21.15.90:0/634139116 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe0c19e8e0 con 0x7efe0c07cda0 2024-04-02T14:24:45.373 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:44.365+0000 7efe11000700 1 --2- 172.21.15.90:0/634139116 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efe0c07cda0 0x7efe0c19e3a0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7efdfc0117e0 tx=0x7efdfc011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:24:45.373 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:44.365+0000 7efe0a000700 1 -- 172.21.15.90:0/634139116 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efdfc003bf0 con 0x7efe0c07cda0 2024-04-02T14:24:45.373 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:44.365+0000 7efe0a000700 1 -- 172.21.15.90:0/634139116 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efdfc01a910 con 0x7efe0c07cda0 2024-04-02T14:24:45.373 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:44.365+0000 7efe13a00700 1 -- 172.21.15.90:0/634139116 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efe0c19eb70 con 0x7efe0c07cda0 2024-04-02T14:24:45.373 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:44.365+0000 7efe0a000700 1 -- 172.21.15.90:0/634139116 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efdfc0249a0 con 0x7efe0c07cda0 2024-04-02T14:24:45.373 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:44.366+0000 7efe13a00700 1 -- 172.21.15.90:0/634139116 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efe0c19efd0 con 0x7efe0c07cda0 2024-04-02T14:24:45.373 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:44.367+0000 7efe0a000700 1 -- 172.21.15.90:0/634139116 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7efdfc022020 con 0x7efe0c07cda0 2024-04-02T14:24:45.373 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:44.367+0000 7efe13a00700 1 -- 172.21.15.90:0/634139116 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efe0c063c20 con 0x7efe0c07cda0 2024-04-02T14:24:45.373 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:44.367+0000 7efe0a000700 1 --2- 172.21.15.90:0/634139116 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7efdf00396b0 0x7efdf003bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:24:45.373 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:44.367+0000 7efe0a000700 1 -- 172.21.15.90:0/634139116 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7efdfc04dbe0 con 0x7efe0c07cda0 2024-04-02T14:24:45.373 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:44.368+0000 7efe03e00700 1 --2- 172.21.15.90:0/634139116 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7efdf00396b0 0x7efdf003bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:24:45.373 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:44.368+0000 7efe03e00700 1 --2- 172.21.15.90:0/634139116 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7efdf00396b0 0x7efdf003bb70 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7efdf4006fd0 tx=0x7efdf4006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:24:45.373 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:44.373+0000 7efe0a000700 1 -- 172.21.15.90:0/634139116 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7efdfc011e20 con 0x7efe0c07cda0 2024-04-02T14:24:45.374 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:44.713+0000 7efe13a00700 1 -- 172.21.15.90:0/634139116 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) -- 0x7efe0c1a19a0 con 0x7efe0c07cda0 2024-04-02T14:24:45.374 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:44.718+0000 7efe0a000700 1 -- 172.21.15.90:0/634139116 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v28)=0 set mgr/dashboard/cluster/status v28) ==== 153+0+0 (secure 0 0 0) 0x7efdfc004300 con 0x7efe0c07cda0 2024-04-02T14:24:45.374 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:44.721+0000 7efe13a00700 1 -- 172.21.15.90:0/634139116 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7efdf00396b0 msgr2=0x7efdf003bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:24:45.374 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:44.721+0000 7efe13a00700 1 --2- 172.21.15.90:0/634139116 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7efdf00396b0 0x7efdf003bb70 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7efdf4006fd0 tx=0x7efdf4006e40 comp rx=0 tx=0).stop 2024-04-02T14:24:45.374 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:44.721+0000 7efe13a00700 1 -- 172.21.15.90:0/634139116 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efe0c07cda0 msgr2=0x7efe0c19e3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:24:45.374 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:44.721+0000 7efe13a00700 1 --2- 172.21.15.90:0/634139116 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efe0c07cda0 0x7efe0c19e3a0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7efdfc0117e0 tx=0x7efdfc011c10 comp rx=0 tx=0).stop 2024-04-02T14:24:45.374 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:44.721+0000 7efe13a00700 1 -- 172.21.15.90:0/634139116 shutdown_connections 2024-04-02T14:24:45.374 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:44.721+0000 7efe13a00700 1 --2- 172.21.15.90:0/634139116 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7efdf00396b0 0x7efdf003bb70 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:24:45.374 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:44.721+0000 7efe13a00700 1 --2- 172.21.15.90:0/634139116 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efe0c07cda0 0x7efe0c19e3a0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:24:45.374 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:44.721+0000 7efe13a00700 1 -- 172.21.15.90:0/634139116 >> 172.21.15.90:0/634139116 conn(0x7efe0c103810 msgr2=0x7efe0c1042e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:24:45.374 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:44.722+0000 7efe13a00700 1 -- 172.21.15.90:0/634139116 shutdown_connections 2024-04-02T14:24:45.374 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-04-02T14:24:44.722+0000 7efe13a00700 1 -- 172.21.15.90:0/634139116 wait complete. 2024-04-02T14:24:45.374 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-04-02T14:24:45.374 INFO:teuthology.orchestra.run.smithi090.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-04-02T14:24:45.375 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-02T14:24:45.375 INFO:teuthology.orchestra.run.smithi090.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 6449448a-f0fc-11ee-b647-cb9ed24678a4 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-04-02T14:24:45.375 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-02T14:24:45.375 INFO:teuthology.orchestra.run.smithi090.stdout:Or, if you are only running a single cluster on this host: 2024-04-02T14:24:45.375 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-02T14:24:45.375 INFO:teuthology.orchestra.run.smithi090.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-04-02T14:24:45.375 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-02T14:24:45.375 INFO:teuthology.orchestra.run.smithi090.stdout:Please consider enabling telemetry to help improve Ceph: 2024-04-02T14:24:45.375 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-02T14:24:45.375 INFO:teuthology.orchestra.run.smithi090.stdout: ceph telemetry on 2024-04-02T14:24:45.375 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-02T14:24:45.375 INFO:teuthology.orchestra.run.smithi090.stdout:For more information see: 2024-04-02T14:24:45.375 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-02T14:24:45.375 INFO:teuthology.orchestra.run.smithi090.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-04-02T14:24:45.375 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-02T14:24:45.376 INFO:teuthology.orchestra.run.smithi090.stdout:Bootstrap complete. 2024-04-02T14:24:45.461 INFO:tasks.cephadm:Fetching config... 2024-04-02T14:24:45.462 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-04-02T14:24:45.462 DEBUG:teuthology.orchestra.run.smithi090:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-04-02T14:24:45.511 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-04-02T14:24:45.511 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-04-02T14:24:45.511 DEBUG:teuthology.orchestra.run.smithi090:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-04-02T14:24:45.591 INFO:tasks.cephadm:Fetching mon keyring... 2024-04-02T14:24:45.591 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-04-02T14:24:45.591 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd if=/var/lib/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/mon.a/keyring of=/dev/stdout 2024-04-02T14:24:45.712 INFO:tasks.cephadm:Fetching pub ssh key... 2024-04-02T14:24:45.712 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-04-02T14:24:45.712 DEBUG:teuthology.orchestra.run.smithi090:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-04-02T14:24:45.805 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-04-02T14:24:45.806 DEBUG:teuthology.orchestra.run.smithi090:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDfeF+PzYQUGcT56oUfI9bNXcaihfx5OtfE5wlPtWw1AeDM86UoN1Td2/VJOKnCrDQWsuVknsL6tN1Tau5W2YeLp/dCgscer9myhNXaiAhn69CGXpoO4Azkk5Pdt/jNx8IrRWrGHl44ST+fQSNzFvVV8bojOzVNRQbz+WRqXgur3xPHsw10ZLOgkj61iBdgWzVYwiI1k6nPJwaT0CjZnVT+cT7z6DZuSmgS3XKig0ojAyuvjm4stw4vwXSt652fqfGJO5nsFyMaP/cQpMlx0f53Pknp7zioqMfzwd6+HYgBaYayAiVVzVJWTFknS+R1HFPTMRbAR4+Mv7PZLFY2XJY4Q8scpVljbHw0J88JT6oE85afTE6COmCLbq/gnQtle/bfdwSXxptOKUcHZemCWzvdLKRa1WTQeXGug2lfEZcyhDSKh8etKm1pdO39tiqQCNtYwarJx1rYBmvY7/DYnvoCuTMaOeDU26QkYJHbWstOV/L+L9icIIimHk+LsR9T6is= ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-02T14:24:45.916 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:24:45 smithi090 ceph-mon[31674]: from='client.? 172.21.15.90:0/634139116' entity='client.admin' 2024-04-02T14:24:45.925 INFO:teuthology.orchestra.run.smithi090.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDfeF+PzYQUGcT56oUfI9bNXcaihfx5OtfE5wlPtWw1AeDM86UoN1Td2/VJOKnCrDQWsuVknsL6tN1Tau5W2YeLp/dCgscer9myhNXaiAhn69CGXpoO4Azkk5Pdt/jNx8IrRWrGHl44ST+fQSNzFvVV8bojOzVNRQbz+WRqXgur3xPHsw10ZLOgkj61iBdgWzVYwiI1k6nPJwaT0CjZnVT+cT7z6DZuSmgS3XKig0ojAyuvjm4stw4vwXSt652fqfGJO5nsFyMaP/cQpMlx0f53Pknp7zioqMfzwd6+HYgBaYayAiVVzVJWTFknS+R1HFPTMRbAR4+Mv7PZLFY2XJY4Q8scpVljbHw0J88JT6oE85afTE6COmCLbq/gnQtle/bfdwSXxptOKUcHZemCWzvdLKRa1WTQeXGug2lfEZcyhDSKh8etKm1pdO39tiqQCNtYwarJx1rYBmvY7/DYnvoCuTMaOeDU26QkYJHbWstOV/L+L9icIIimHk+LsR9T6is= ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4 2024-04-02T14:24:45.989 DEBUG:teuthology.orchestra.run.smithi092:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDfeF+PzYQUGcT56oUfI9bNXcaihfx5OtfE5wlPtWw1AeDM86UoN1Td2/VJOKnCrDQWsuVknsL6tN1Tau5W2YeLp/dCgscer9myhNXaiAhn69CGXpoO4Azkk5Pdt/jNx8IrRWrGHl44ST+fQSNzFvVV8bojOzVNRQbz+WRqXgur3xPHsw10ZLOgkj61iBdgWzVYwiI1k6nPJwaT0CjZnVT+cT7z6DZuSmgS3XKig0ojAyuvjm4stw4vwXSt652fqfGJO5nsFyMaP/cQpMlx0f53Pknp7zioqMfzwd6+HYgBaYayAiVVzVJWTFknS+R1HFPTMRbAR4+Mv7PZLFY2XJY4Q8scpVljbHw0J88JT6oE85afTE6COmCLbq/gnQtle/bfdwSXxptOKUcHZemCWzvdLKRa1WTQeXGug2lfEZcyhDSKh8etKm1pdO39tiqQCNtYwarJx1rYBmvY7/DYnvoCuTMaOeDU26QkYJHbWstOV/L+L9icIIimHk+LsR9T6is= ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-02T14:24:46.034 INFO:teuthology.orchestra.run.smithi092.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDfeF+PzYQUGcT56oUfI9bNXcaihfx5OtfE5wlPtWw1AeDM86UoN1Td2/VJOKnCrDQWsuVknsL6tN1Tau5W2YeLp/dCgscer9myhNXaiAhn69CGXpoO4Azkk5Pdt/jNx8IrRWrGHl44ST+fQSNzFvVV8bojOzVNRQbz+WRqXgur3xPHsw10ZLOgkj61iBdgWzVYwiI1k6nPJwaT0CjZnVT+cT7z6DZuSmgS3XKig0ojAyuvjm4stw4vwXSt652fqfGJO5nsFyMaP/cQpMlx0f53Pknp7zioqMfzwd6+HYgBaYayAiVVzVJWTFknS+R1HFPTMRbAR4+Mv7PZLFY2XJY4Q8scpVljbHw0J88JT6oE85afTE6COmCLbq/gnQtle/bfdwSXxptOKUcHZemCWzvdLKRa1WTQeXGug2lfEZcyhDSKh8etKm1pdO39tiqQCNtYwarJx1rYBmvY7/DYnvoCuTMaOeDU26QkYJHbWstOV/L+L9icIIimHk+LsR9T6is= ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4 2024-04-02T14:24:46.048 DEBUG:teuthology.orchestra.run.smithi177:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDfeF+PzYQUGcT56oUfI9bNXcaihfx5OtfE5wlPtWw1AeDM86UoN1Td2/VJOKnCrDQWsuVknsL6tN1Tau5W2YeLp/dCgscer9myhNXaiAhn69CGXpoO4Azkk5Pdt/jNx8IrRWrGHl44ST+fQSNzFvVV8bojOzVNRQbz+WRqXgur3xPHsw10ZLOgkj61iBdgWzVYwiI1k6nPJwaT0CjZnVT+cT7z6DZuSmgS3XKig0ojAyuvjm4stw4vwXSt652fqfGJO5nsFyMaP/cQpMlx0f53Pknp7zioqMfzwd6+HYgBaYayAiVVzVJWTFknS+R1HFPTMRbAR4+Mv7PZLFY2XJY4Q8scpVljbHw0J88JT6oE85afTE6COmCLbq/gnQtle/bfdwSXxptOKUcHZemCWzvdLKRa1WTQeXGug2lfEZcyhDSKh8etKm1pdO39tiqQCNtYwarJx1rYBmvY7/DYnvoCuTMaOeDU26QkYJHbWstOV/L+L9icIIimHk+LsR9T6is= ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-02T14:24:46.093 INFO:teuthology.orchestra.run.smithi177.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDfeF+PzYQUGcT56oUfI9bNXcaihfx5OtfE5wlPtWw1AeDM86UoN1Td2/VJOKnCrDQWsuVknsL6tN1Tau5W2YeLp/dCgscer9myhNXaiAhn69CGXpoO4Azkk5Pdt/jNx8IrRWrGHl44ST+fQSNzFvVV8bojOzVNRQbz+WRqXgur3xPHsw10ZLOgkj61iBdgWzVYwiI1k6nPJwaT0CjZnVT+cT7z6DZuSmgS3XKig0ojAyuvjm4stw4vwXSt652fqfGJO5nsFyMaP/cQpMlx0f53Pknp7zioqMfzwd6+HYgBaYayAiVVzVJWTFknS+R1HFPTMRbAR4+Mv7PZLFY2XJY4Q8scpVljbHw0J88JT6oE85afTE6COmCLbq/gnQtle/bfdwSXxptOKUcHZemCWzvdLKRa1WTQeXGug2lfEZcyhDSKh8etKm1pdO39tiqQCNtYwarJx1rYBmvY7/DYnvoCuTMaOeDU26QkYJHbWstOV/L+L9icIIimHk+LsR9T6is= ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4 2024-04-02T14:24:46.109 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:191d963ac9de7322da8aff20b567b3fa458e2ef6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6449448a-f0fc-11ee-b647-cb9ed24678a4 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-04-02T14:24:46.520 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T14:24:49.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:24:49.190+0000 7f535b200700 1 -- 172.21.15.90:0/1238802920 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5354102020 msgr2=0x7f5354102440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:24:49.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:24:49.190+0000 7f535b200700 1 --2- 172.21.15.90:0/1238802920 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5354102020 0x7f5354102440 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f5344009910 tx=0x7f5344009c20 comp rx=0 tx=0).stop 2024-04-02T14:24:49.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:24:49.190+0000 7f535b200700 1 -- 172.21.15.90:0/1238802920 shutdown_connections 2024-04-02T14:24:49.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:24:49.190+0000 7f535b200700 1 --2- 172.21.15.90:0/1238802920 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5354102020 0x7f5354102440 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:24:49.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:24:49.190+0000 7f535b200700 1 -- 172.21.15.90:0/1238802920 >> 172.21.15.90:0/1238802920 conn(0x7f53540fd5c0 msgr2=0x7f53540ffa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:24:49.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:24:49.190+0000 7f535b200700 1 -- 172.21.15.90:0/1238802920 shutdown_connections 2024-04-02T14:24:49.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:24:49.190+0000 7f535b200700 1 -- 172.21.15.90:0/1238802920 wait complete. 2024-04-02T14:24:49.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:24:49.191+0000 7f535b200700 1 Processor -- start 2024-04-02T14:24:49.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:24:49.192+0000 7f535b200700 1 -- start start 2024-04-02T14:24:49.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:24:49.192+0000 7f535b200700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5354102020 0x7f5354197720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:24:49.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:24:49.192+0000 7f535b200700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f53541080b0 con 0x7f5354102020 2024-04-02T14:24:49.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:24:49.192+0000 7f5353e00700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5354102020 0x7f5354197720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:24:49.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:24:49.192+0000 7f5353e00700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5354102020 0x7f5354197720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:39608/0 (socket says 172.21.15.90:39608) 2024-04-02T14:24:49.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:24:49.192+0000 7f5353e00700 1 -- 172.21.15.90:0/2605994433 learned_addr learned my addr 172.21.15.90:0/2605994433 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:24:49.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:24:49.193+0000 7f5353e00700 1 -- 172.21.15.90:0/2605994433 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5354197c60 con 0x7f5354102020 2024-04-02T14:24:49.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:24:49.193+0000 7f5353e00700 1 --2- 172.21.15.90:0/2605994433 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5354102020 0x7f5354197720 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f5344005850 tx=0x7f5344005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:24:49.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:24:49.193+0000 7f5350c00700 1 -- 172.21.15.90:0/2605994433 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5344026070 con 0x7f5354102020 2024-04-02T14:24:49.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:24:49.193+0000 7f5350c00700 1 -- 172.21.15.90:0/2605994433 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f53440044a0 con 0x7f5354102020 2024-04-02T14:24:49.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:24:49.193+0000 7f535b200700 1 -- 172.21.15.90:0/2605994433 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5354197ef0 con 0x7f5354102020 2024-04-02T14:24:49.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:24:49.193+0000 7f5350c00700 1 -- 172.21.15.90:0/2605994433 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5344003ab0 con 0x7f5354102020 2024-04-02T14:24:49.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:24:49.193+0000 7f535b200700 1 -- 172.21.15.90:0/2605994433 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f535419abe0 con 0x7f5354102020 2024-04-02T14:24:49.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:24:49.195+0000 7f5350c00700 1 -- 172.21.15.90:0/2605994433 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f5344029070 con 0x7f5354102020 2024-04-02T14:24:49.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:24:49.195+0000 7f535b200700 1 -- 172.21.15.90:0/2605994433 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5354108830 con 0x7f5354102020 2024-04-02T14:24:49.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:24:49.195+0000 7f5350c00700 1 --2- 172.21.15.90:0/2605994433 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f5338039650 0x7f533803bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:24:49.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:24:49.195+0000 7f5350c00700 1 -- 172.21.15.90:0/2605994433 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f534401e070 con 0x7f5354102020 2024-04-02T14:24:49.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:24:49.195+0000 7f5353400700 1 --2- 172.21.15.90:0/2605994433 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f5338039650 0x7f533803bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:24:49.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:24:49.196+0000 7f5353400700 1 --2- 172.21.15.90:0/2605994433 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f5338039650 0x7f533803bb10 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f5340006fd0 tx=0x7f5340006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:24:49.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:24:49.201+0000 7f5350c00700 1 -- 172.21.15.90:0/2605994433 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7f5344013070 con 0x7f5354102020 2024-04-02T14:24:49.429 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:24:49.428+0000 7f535b200700 1 -- 172.21.15.90:0/2605994433 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) -- 0x7f535402de40 con 0x7f5354102020 2024-04-02T14:24:49.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:24:49.432+0000 7f5350c00700 1 -- 172.21.15.90:0/2605994433 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) ==== 125+0+0 (secure 0 0 0) 0x7f53440300c0 con 0x7f5354102020 2024-04-02T14:24:49.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:24:49.435+0000 7f535b200700 1 -- 172.21.15.90:0/2605994433 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f5338039650 msgr2=0x7f533803bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:24:49.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:24:49.435+0000 7f535b200700 1 --2- 172.21.15.90:0/2605994433 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f5338039650 0x7f533803bb10 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f5340006fd0 tx=0x7f5340006e40 comp rx=0 tx=0).stop 2024-04-02T14:24:49.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:24:49.435+0000 7f535b200700 1 -- 172.21.15.90:0/2605994433 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5354102020 msgr2=0x7f5354197720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:24:49.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:24:49.435+0000 7f535b200700 1 --2- 172.21.15.90:0/2605994433 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5354102020 0x7f5354197720 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f5344005850 tx=0x7f5344005030 comp rx=0 tx=0).stop 2024-04-02T14:24:49.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:24:49.435+0000 7f535b200700 1 -- 172.21.15.90:0/2605994433 shutdown_connections 2024-04-02T14:24:49.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:24:49.435+0000 7f535b200700 1 --2- 172.21.15.90:0/2605994433 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f5338039650 0x7f533803bb10 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:24:49.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:24:49.435+0000 7f535b200700 1 --2- 172.21.15.90:0/2605994433 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5354102020 0x7f5354197720 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:24:49.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:24:49.435+0000 7f535b200700 1 -- 172.21.15.90:0/2605994433 >> 172.21.15.90:0/2605994433 conn(0x7f53540fd5c0 msgr2=0x7f53540fe010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:24:49.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:24:49.436+0000 7f535b200700 1 -- 172.21.15.90:0/2605994433 shutdown_connections 2024-04-02T14:24:49.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:24:49.436+0000 7f535b200700 1 -- 172.21.15.90:0/2605994433 wait complete. 2024-04-02T14:24:50.251 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-04-02T14:24:50.252 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:191d963ac9de7322da8aff20b567b3fa458e2ef6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6449448a-f0fc-11ee-b647-cb9ed24678a4 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-04-02T14:24:50.438 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:24:50 smithi090 ceph-mon[31674]: from='client.? 172.21.15.90:0/2605994433' entity='client.admin' 2024-04-02T14:24:51.235 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T14:24:52.349 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:24:52 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:24:52.349 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:24:52 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:24:52.349 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:24:52 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi090", "name": "osd_memory_target"}]: dispatch 2024-04-02T14:24:52.349 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:24:52 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:24:52.349 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:24:52 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T14:24:52.349 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:24:52 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:24:53.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:24:53.191+0000 7faf66200700 1 -- 172.21.15.90:0/3953518936 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faf60102b10 msgr2=0x7faf60104f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:24:53.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:24:53.191+0000 7faf66200700 1 --2- 172.21.15.90:0/3953518936 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faf60102b10 0x7faf60104f00 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7faf50009910 tx=0x7faf50009c20 comp rx=0 tx=0).stop 2024-04-02T14:24:53.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:24:53.191+0000 7faf66200700 1 -- 172.21.15.90:0/3953518936 shutdown_connections 2024-04-02T14:24:53.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:24:53.191+0000 7faf66200700 1 --2- 172.21.15.90:0/3953518936 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faf60102b10 0x7faf60104f00 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:24:53.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:24:53.191+0000 7faf66200700 1 -- 172.21.15.90:0/3953518936 >> 172.21.15.90:0/3953518936 conn(0x7faf600fc4c0 msgr2=0x7faf600fe900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:24:53.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:24:53.191+0000 7faf66200700 1 -- 172.21.15.90:0/3953518936 shutdown_connections 2024-04-02T14:24:53.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:24:53.191+0000 7faf66200700 1 -- 172.21.15.90:0/3953518936 wait complete. 2024-04-02T14:24:53.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:24:53.193+0000 7faf66200700 1 Processor -- start 2024-04-02T14:24:53.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:24:53.193+0000 7faf66200700 1 -- start start 2024-04-02T14:24:53.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:24:53.193+0000 7faf66200700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faf60102b10 0x7faf60199920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:24:53.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:24:53.193+0000 7faf66200700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7faf601080c0 con 0x7faf60102b10 2024-04-02T14:24:53.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:24:53.193+0000 7faf5f400700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faf60102b10 0x7faf60199920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:24:53.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:24:53.193+0000 7faf5f400700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faf60102b10 0x7faf60199920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:39634/0 (socket says 172.21.15.90:39634) 2024-04-02T14:24:53.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:24:53.193+0000 7faf5f400700 1 -- 172.21.15.90:0/753250927 learned_addr learned my addr 172.21.15.90:0/753250927 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:24:53.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:24:53.194+0000 7faf5f400700 1 -- 172.21.15.90:0/753250927 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf60199e60 con 0x7faf60102b10 2024-04-02T14:24:53.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:24:53.194+0000 7faf5f400700 1 --2- 172.21.15.90:0/753250927 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faf60102b10 0x7faf60199920 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7faf50005950 tx=0x7faf50005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:24:53.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:24:53.194+0000 7faf5cc00700 1 -- 172.21.15.90:0/753250927 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7faf50026070 con 0x7faf60102b10 2024-04-02T14:24:53.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:24:53.194+0000 7faf66200700 1 -- 172.21.15.90:0/753250927 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faf6019a0f0 con 0x7faf60102b10 2024-04-02T14:24:53.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:24:53.194+0000 7faf5cc00700 1 -- 172.21.15.90:0/753250927 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faf5000bc30 con 0x7faf60102b10 2024-04-02T14:24:53.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:24:53.194+0000 7faf5cc00700 1 -- 172.21.15.90:0/753250927 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7faf50003b30 con 0x7faf60102b10 2024-04-02T14:24:53.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:24:53.194+0000 7faf66200700 1 -- 172.21.15.90:0/753250927 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faf6019cde0 con 0x7faf60102b10 2024-04-02T14:24:53.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:24:53.195+0000 7faf66200700 1 -- 172.21.15.90:0/753250927 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faf60108840 con 0x7faf60102b10 2024-04-02T14:24:53.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:24:53.196+0000 7faf5cc00700 1 -- 172.21.15.90:0/753250927 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7faf50029070 con 0x7faf60102b10 2024-04-02T14:24:53.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:24:53.196+0000 7faf5cc00700 1 --2- 172.21.15.90:0/753250927 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7faf44039650 0x7faf4403bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:24:53.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:24:53.196+0000 7faf5cc00700 1 -- 172.21.15.90:0/753250927 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7faf5001e070 con 0x7faf60102b10 2024-04-02T14:24:53.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:24:53.196+0000 7faf56a00700 1 --2- 172.21.15.90:0/753250927 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7faf44039650 0x7faf4403bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:24:53.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:24:53.197+0000 7faf56a00700 1 --2- 172.21.15.90:0/753250927 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7faf44039650 0x7faf4403bb10 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7faf4800ad30 tx=0x7faf480093f0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:24:53.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:24:53.202+0000 7faf5cc00700 1 -- 172.21.15.90:0/753250927 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7faf50013070 con 0x7faf60102b10 2024-04-02T14:24:53.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:24:53.433+0000 7faf66200700 1 -- 172.21.15.90:0/753250927 --> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) -- 0x7faf6002de40 con 0x7faf44039650 2024-04-02T14:24:53.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:24:53.439+0000 7faf5cc00700 1 -- 172.21.15.90:0/753250927 <== mgr.14152 v2:172.21.15.90:6800/4266054817 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7faf6002de40 con 0x7faf44039650 2024-04-02T14:24:53.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:24:53.442+0000 7faf66200700 1 -- 172.21.15.90:0/753250927 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7faf44039650 msgr2=0x7faf4403bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:24:53.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:24:53.442+0000 7faf66200700 1 --2- 172.21.15.90:0/753250927 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7faf44039650 0x7faf4403bb10 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7faf4800ad30 tx=0x7faf480093f0 comp rx=0 tx=0).stop 2024-04-02T14:24:53.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:24:53.442+0000 7faf66200700 1 -- 172.21.15.90:0/753250927 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faf60102b10 msgr2=0x7faf60199920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:24:53.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:24:53.442+0000 7faf66200700 1 --2- 172.21.15.90:0/753250927 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faf60102b10 0x7faf60199920 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7faf50005950 tx=0x7faf50005030 comp rx=0 tx=0).stop 2024-04-02T14:24:53.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:24:53.443+0000 7faf66200700 1 -- 172.21.15.90:0/753250927 shutdown_connections 2024-04-02T14:24:53.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:24:53.443+0000 7faf66200700 1 --2- 172.21.15.90:0/753250927 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7faf44039650 0x7faf4403bb10 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:24:53.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:24:53.443+0000 7faf66200700 1 --2- 172.21.15.90:0/753250927 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faf60102b10 0x7faf60199920 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:24:53.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:24:53.443+0000 7faf66200700 1 -- 172.21.15.90:0/753250927 >> 172.21.15.90:0/753250927 conn(0x7faf600fc4c0 msgr2=0x7faf600fe900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:24:53.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:24:53.443+0000 7faf66200700 1 -- 172.21.15.90:0/753250927 shutdown_connections 2024-04-02T14:24:53.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:24:53.443+0000 7faf66200700 1 -- 172.21.15.90:0/753250927 wait complete. 2024-04-02T14:24:54.250 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi092 2024-04-02T14:24:54.250 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2024-04-02T14:24:54.250 DEBUG:teuthology.orchestra.run.smithi092:> dd of=/etc/ceph/ceph.conf 2024-04-02T14:24:54.268 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2024-04-02T14:24:54.268 DEBUG:teuthology.orchestra.run.smithi092:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-02T14:24:54.325 INFO:tasks.cephadm:Adding host smithi092 to orchestrator... 2024-04-02T14:24:54.326 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:191d963ac9de7322da8aff20b567b3fa458e2ef6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6449448a-f0fc-11ee-b647-cb9ed24678a4 -- ceph orch host add smithi092 2024-04-02T14:24:54.688 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:24:54 smithi090 ceph-mon[31674]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T14:24:54.688 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:24:54 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:24:54.689 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:24:54 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T14:24:54.689 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:24:54 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:24:54.689 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:24:54 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T14:24:54.689 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:24:54 smithi090 ceph-mon[31674]: Updating smithi090:/etc/ceph/ceph.conf 2024-04-02T14:24:54.745 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T14:24:55.886 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:24:55 smithi090 ceph-mon[31674]: Updating smithi090:/var/lib/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-04-02T14:24:55.886 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:24:55 smithi090 ceph-mon[31674]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T14:24:55.887 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:24:55 smithi090 ceph-mon[31674]: Updating smithi090:/etc/ceph/ceph.client.admin.keyring 2024-04-02T14:24:56.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:24:56.591+0000 7f897ca00700 1 -- 172.21.15.90:0/844352683 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f897810d3c0 msgr2=0x7f897810d7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:24:56.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:24:56.591+0000 7f897ca00700 1 --2- 172.21.15.90:0/844352683 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f897810d3c0 0x7f897810d7a0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f8968009910 tx=0x7f8968009c20 comp rx=0 tx=0).stop 2024-04-02T14:24:56.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:24:56.592+0000 7f897ca00700 1 -- 172.21.15.90:0/844352683 shutdown_connections 2024-04-02T14:24:56.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:24:56.592+0000 7f897ca00700 1 --2- 172.21.15.90:0/844352683 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f897810d3c0 0x7f897810d7a0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:24:56.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:24:56.592+0000 7f897ca00700 1 -- 172.21.15.90:0/844352683 >> 172.21.15.90:0/844352683 conn(0x7f8978077bd0 msgr2=0x7f8978079ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:24:56.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:24:56.592+0000 7f897ca00700 1 -- 172.21.15.90:0/844352683 shutdown_connections 2024-04-02T14:24:56.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:24:56.592+0000 7f897ca00700 1 -- 172.21.15.90:0/844352683 wait complete. 2024-04-02T14:24:56.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:24:56.593+0000 7f897ca00700 1 Processor -- start 2024-04-02T14:24:56.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:24:56.593+0000 7f897ca00700 1 -- start start 2024-04-02T14:24:56.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:24:56.593+0000 7f897ca00700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f897810d3c0 0x7f897819df10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:24:56.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:24:56.593+0000 7f897ca00700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f8978112a80 con 0x7f897810d3c0 2024-04-02T14:24:56.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:24:56.594+0000 7f8976800700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f897810d3c0 0x7f897819df10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:24:56.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:24:56.594+0000 7f8976800700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f897810d3c0 0x7f897819df10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34314/0 (socket says 172.21.15.90:34314) 2024-04-02T14:24:56.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:24:56.594+0000 7f8976800700 1 -- 172.21.15.90:0/1231034507 learned_addr learned my addr 172.21.15.90:0/1231034507 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:24:56.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:24:56.594+0000 7f8976800700 1 -- 172.21.15.90:0/1231034507 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f897819e450 con 0x7f897810d3c0 2024-04-02T14:24:56.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:24:56.594+0000 7f8976800700 1 --2- 172.21.15.90:0/1231034507 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f897810d3c0 0x7f897819df10 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f8968006010 tx=0x7f8968005050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:24:56.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:24:56.595+0000 7f896f400700 1 -- 172.21.15.90:0/1231034507 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8968026070 con 0x7f897810d3c0 2024-04-02T14:24:56.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:24:56.595+0000 7f896f400700 1 -- 172.21.15.90:0/1231034507 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f896800bbd0 con 0x7f897810d3c0 2024-04-02T14:24:56.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:24:56.595+0000 7f896f400700 1 -- 172.21.15.90:0/1231034507 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8968003a30 con 0x7f897810d3c0 2024-04-02T14:24:56.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:24:56.595+0000 7f897ca00700 1 -- 172.21.15.90:0/1231034507 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f897819e740 con 0x7f897810d3c0 2024-04-02T14:24:56.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:24:56.595+0000 7f897ca00700 1 -- 172.21.15.90:0/1231034507 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89781a27e0 con 0x7f897810d3c0 2024-04-02T14:24:56.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:24:56.596+0000 7f897ca00700 1 -- 172.21.15.90:0/1231034507 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8978109b50 con 0x7f897810d3c0 2024-04-02T14:24:56.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:24:56.596+0000 7f896f400700 1 -- 172.21.15.90:0/1231034507 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f8968029070 con 0x7f897810d3c0 2024-04-02T14:24:56.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:24:56.596+0000 7f896f400700 1 --2- 172.21.15.90:0/1231034507 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f895c0396b0 0x7f895c03bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:24:56.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:24:56.597+0000 7f896f400700 1 -- 172.21.15.90:0/1231034507 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f896801e070 con 0x7f897810d3c0 2024-04-02T14:24:56.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:24:56.597+0000 7f8975e00700 1 --2- 172.21.15.90:0/1231034507 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f895c0396b0 0x7f895c03bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:24:56.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:24:56.597+0000 7f8975e00700 1 --2- 172.21.15.90:0/1231034507 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f895c0396b0 0x7f895c03bb70 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f8964006fd0 tx=0x7f8964006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:24:56.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:24:56.602+0000 7f896f400700 1 -- 172.21.15.90:0/1231034507 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7f8968013070 con 0x7f897810d3c0 2024-04-02T14:24:56.689 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:24:56 smithi090 ceph-mon[31674]: Updating smithi090:/var/lib/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/config/ceph.client.admin.keyring 2024-04-02T14:24:56.689 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:24:56 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:24:56.689 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:24:56 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:24:56.689 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:24:56 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:24:56.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:24:56.832+0000 7f897ca00700 1 -- 172.21.15.90:0/1231034507 --> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi092", "target": ["mon-mgr", ""]}) -- 0x7f897819f270 con 0x7f895c0396b0 2024-04-02T14:24:57.938 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:24:57 smithi090 ceph-mon[31674]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T14:24:58.938 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:24:58 smithi090 ceph-mon[31674]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi092", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T14:24:58.938 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:24:58 smithi090 ceph-mon[31674]: Deploying cephadm binary to smithi092 2024-04-02T14:24:59.938 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:24:59 smithi090 ceph-mon[31674]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T14:25:01.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:01.607+0000 7f896f400700 1 -- 172.21.15.90:0/1231034507 <== mgr.14152 v2:172.21.15.90:6800/4266054817 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+48 (secure 0 0 0) 0x7f897819f270 con 0x7f895c0396b0 2024-04-02T14:25:01.610 INFO:teuthology.orchestra.run.smithi090.stdout:Added host 'smithi092' with addr '172.21.15.92' 2024-04-02T14:25:01.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:01.611+0000 7f897ca00700 1 -- 172.21.15.90:0/1231034507 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f895c0396b0 msgr2=0x7f895c03bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:25:01.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:01.611+0000 7f897ca00700 1 --2- 172.21.15.90:0/1231034507 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f895c0396b0 0x7f895c03bb70 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f8964006fd0 tx=0x7f8964006e40 comp rx=0 tx=0).stop 2024-04-02T14:25:01.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:01.611+0000 7f897ca00700 1 -- 172.21.15.90:0/1231034507 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f897810d3c0 msgr2=0x7f897819df10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:25:01.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:01.611+0000 7f897ca00700 1 --2- 172.21.15.90:0/1231034507 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f897810d3c0 0x7f897819df10 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f8968006010 tx=0x7f8968005050 comp rx=0 tx=0).stop 2024-04-02T14:25:01.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:01.611+0000 7f897ca00700 1 -- 172.21.15.90:0/1231034507 shutdown_connections 2024-04-02T14:25:01.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:01.611+0000 7f897ca00700 1 --2- 172.21.15.90:0/1231034507 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f895c0396b0 0x7f895c03bb70 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:25:01.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:01.611+0000 7f897ca00700 1 --2- 172.21.15.90:0/1231034507 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f897810d3c0 0x7f897819df10 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:25:01.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:01.611+0000 7f897ca00700 1 -- 172.21.15.90:0/1231034507 >> 172.21.15.90:0/1231034507 conn(0x7f8978077bd0 msgr2=0x7f897810feb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:25:01.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:01.612+0000 7f897ca00700 1 -- 172.21.15.90:0/1231034507 shutdown_connections 2024-04-02T14:25:01.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:01.612+0000 7f897ca00700 1 -- 172.21.15.90:0/1231034507 wait complete. 2024-04-02T14:25:01.938 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:01 smithi090 ceph-mon[31674]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T14:25:01.938 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:01 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:25:01.939 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:01 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T14:25:02.374 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:191d963ac9de7322da8aff20b567b3fa458e2ef6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6449448a-f0fc-11ee-b647-cb9ed24678a4 -- ceph orch host ls --format=json 2024-04-02T14:25:02.795 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T14:25:02.931 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:02 smithi090 ceph-mon[31674]: Added host smithi092 2024-04-02T14:25:02.931 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:02 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:25:03.939 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:03 smithi090 ceph-mon[31674]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T14:25:03.939 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:03 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:25:04.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:04.698+0000 7f40b5c00700 1 -- 172.21.15.90:0/1556783668 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f40b0102070 msgr2=0x7f40b0102450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:25:04.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:04.698+0000 7f40b5c00700 1 --2- 172.21.15.90:0/1556783668 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f40b0102070 0x7f40b0102450 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f40a000b0a0 tx=0x7f40a000b3b0 comp rx=0 tx=0).stop 2024-04-02T14:25:04.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:04.698+0000 7f40b5c00700 1 -- 172.21.15.90:0/1556783668 shutdown_connections 2024-04-02T14:25:04.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:04.698+0000 7f40b5c00700 1 --2- 172.21.15.90:0/1556783668 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f40b0102070 0x7f40b0102450 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:25:04.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:04.698+0000 7f40b5c00700 1 -- 172.21.15.90:0/1556783668 >> 172.21.15.90:0/1556783668 conn(0x7f40b00fd920 msgr2=0x7f40b00ffd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:25:04.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:04.698+0000 7f40b5c00700 1 -- 172.21.15.90:0/1556783668 shutdown_connections 2024-04-02T14:25:04.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:04.698+0000 7f40b5c00700 1 -- 172.21.15.90:0/1556783668 wait complete. 2024-04-02T14:25:04.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:04.699+0000 7f40b5c00700 1 Processor -- start 2024-04-02T14:25:04.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:04.700+0000 7f40b5c00700 1 -- start start 2024-04-02T14:25:04.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:04.700+0000 7f40b5c00700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f40b0102070 0x7f40b019f710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:25:04.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:04.700+0000 7f40b5c00700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f40b010c440 con 0x7f40b0102070 2024-04-02T14:25:04.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:04.700+0000 7f40afe00700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f40b0102070 0x7f40b019f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:25:04.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:04.700+0000 7f40afe00700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f40b0102070 0x7f40b019f710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:39848/0 (socket says 172.21.15.90:39848) 2024-04-02T14:25:04.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:04.700+0000 7f40afe00700 1 -- 172.21.15.90:0/3774993618 learned_addr learned my addr 172.21.15.90:0/3774993618 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:25:04.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:04.701+0000 7f40afe00700 1 -- 172.21.15.90:0/3774993618 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40b019fc50 con 0x7f40b0102070 2024-04-02T14:25:04.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:04.701+0000 7f40afe00700 1 --2- 172.21.15.90:0/3774993618 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f40b0102070 0x7f40b019f710 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f40a0015040 tx=0x7f40a0003d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:25:04.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:04.701+0000 7f40ad600700 1 -- 172.21.15.90:0/3774993618 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f40a0017040 con 0x7f40b0102070 2024-04-02T14:25:04.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:04.701+0000 7f40ad600700 1 -- 172.21.15.90:0/3774993618 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f40a0022070 con 0x7f40b0102070 2024-04-02T14:25:04.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:04.701+0000 7f40b5c00700 1 -- 172.21.15.90:0/3774993618 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40b019fee0 con 0x7f40b0102070 2024-04-02T14:25:04.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:04.701+0000 7f40ad600700 1 -- 172.21.15.90:0/3774993618 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f40a001d5d0 con 0x7f40b0102070 2024-04-02T14:25:04.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:04.701+0000 7f40b5c00700 1 -- 172.21.15.90:0/3774993618 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40b01998d0 con 0x7f40b0102070 2024-04-02T14:25:04.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:04.703+0000 7f40ad600700 1 -- 172.21.15.90:0/3774993618 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f40a0004000 con 0x7f40b0102070 2024-04-02T14:25:04.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:04.703+0000 7f40b5c00700 1 -- 172.21.15.90:0/3774993618 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f40b0103550 con 0x7f40b0102070 2024-04-02T14:25:04.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:04.703+0000 7f40ad600700 1 --2- 172.21.15.90:0/3774993618 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f4094039650 0x7f409403bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:25:04.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:04.703+0000 7f40ad600700 1 -- 172.21.15.90:0/3774993618 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f40a000f050 con 0x7f40b0102070 2024-04-02T14:25:04.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:04.703+0000 7f40a7400700 1 --2- 172.21.15.90:0/3774993618 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f4094039650 0x7f409403bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:25:04.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:04.704+0000 7f40a7400700 1 --2- 172.21.15.90:0/3774993618 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f4094039650 0x7f409403bb10 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f4098006fd0 tx=0x7f4098006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:25:04.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:04.709+0000 7f40ad600700 1 -- 172.21.15.90:0/3774993618 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7f40a0007e20 con 0x7f40b0102070 2024-04-02T14:25:04.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:04.941+0000 7f40b5c00700 1 -- 172.21.15.90:0/3774993618 --> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f40b019ae00 con 0x7f4094039650 2024-04-02T14:25:04.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:04.943+0000 7f40ad600700 1 -- 172.21.15.90:0/3774993618 <== mgr.14152 v2:172.21.15.90:6800/4266054817 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+159 (secure 0 0 0) 0x7f40b019ae00 con 0x7f4094039650 2024-04-02T14:25:04.945 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-02T14:25:04.945 INFO:teuthology.orchestra.run.smithi090.stdout:[{"addr": "172.21.15.90", "hostname": "smithi090", "labels": [], "status": ""}, {"addr": "172.21.15.92", "hostname": "smithi092", "labels": [], "status": ""}] 2024-04-02T14:25:04.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:04.946+0000 7f40b5c00700 1 -- 172.21.15.90:0/3774993618 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f4094039650 msgr2=0x7f409403bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:25:04.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:04.946+0000 7f40b5c00700 1 --2- 172.21.15.90:0/3774993618 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f4094039650 0x7f409403bb10 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f4098006fd0 tx=0x7f4098006e40 comp rx=0 tx=0).stop 2024-04-02T14:25:04.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:04.946+0000 7f40b5c00700 1 -- 172.21.15.90:0/3774993618 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f40b0102070 msgr2=0x7f40b019f710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:25:04.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:04.946+0000 7f40b5c00700 1 --2- 172.21.15.90:0/3774993618 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f40b0102070 0x7f40b019f710 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f40a0015040 tx=0x7f40a0003d50 comp rx=0 tx=0).stop 2024-04-02T14:25:04.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:04.946+0000 7f40b5c00700 1 -- 172.21.15.90:0/3774993618 shutdown_connections 2024-04-02T14:25:04.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:04.946+0000 7f40b5c00700 1 --2- 172.21.15.90:0/3774993618 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f4094039650 0x7f409403bb10 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:25:04.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:04.946+0000 7f40b5c00700 1 --2- 172.21.15.90:0/3774993618 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f40b0102070 0x7f40b019f710 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:25:04.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:04.946+0000 7f40b5c00700 1 -- 172.21.15.90:0/3774993618 >> 172.21.15.90:0/3774993618 conn(0x7f40b00fd920 msgr2=0x7f40b0105a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:25:04.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:04.947+0000 7f40b5c00700 1 -- 172.21.15.90:0/3774993618 shutdown_connections 2024-04-02T14:25:04.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:04.947+0000 7f40b5c00700 1 -- 172.21.15.90:0/3774993618 wait complete. 2024-04-02T14:25:05.631 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:05 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:25:05.631 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:05 smithi090 ceph-mon[31674]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T14:25:05.708 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi177 2024-04-02T14:25:05.708 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-04-02T14:25:05.708 DEBUG:teuthology.orchestra.run.smithi177:> dd of=/etc/ceph/ceph.conf 2024-04-02T14:25:05.727 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-04-02T14:25:05.727 DEBUG:teuthology.orchestra.run.smithi177:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-02T14:25:05.784 INFO:tasks.cephadm:Adding host smithi177 to orchestrator... 2024-04-02T14:25:05.784 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:191d963ac9de7322da8aff20b567b3fa458e2ef6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6449448a-f0fc-11ee-b647-cb9ed24678a4 -- ceph orch host add smithi177 2024-04-02T14:25:06.207 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T14:25:06.606 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:06 smithi090 ceph-mon[31674]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-02T14:25:07.566 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:07 smithi090 ceph-mon[31674]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T14:25:07.566 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:07 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:25:07.567 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:07 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:25:07.567 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:07 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:25:07.567 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:07 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:25:07.567 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:07 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi092", "name": "osd_memory_target"}]: dispatch 2024-04-02T14:25:07.567 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:07 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:25:07.567 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:07 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T14:25:08.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:08.018+0000 7f32c6200700 1 -- 172.21.15.90:0/4234822728 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f32c0102070 msgr2=0x7f32c0102450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:25:08.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:08.018+0000 7f32c6200700 1 --2- 172.21.15.90:0/4234822728 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f32c0102070 0x7f32c0102450 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f32b4009910 tx=0x7f32b4009c20 comp rx=0 tx=0).stop 2024-04-02T14:25:08.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:08.018+0000 7f32c6200700 1 -- 172.21.15.90:0/4234822728 shutdown_connections 2024-04-02T14:25:08.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:08.018+0000 7f32c6200700 1 --2- 172.21.15.90:0/4234822728 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f32c0102070 0x7f32c0102450 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:25:08.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:08.018+0000 7f32c6200700 1 -- 172.21.15.90:0/4234822728 >> 172.21.15.90:0/4234822728 conn(0x7f32c00fd920 msgr2=0x7f32c00ffd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:25:08.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:08.018+0000 7f32c6200700 1 -- 172.21.15.90:0/4234822728 shutdown_connections 2024-04-02T14:25:08.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:08.018+0000 7f32c6200700 1 -- 172.21.15.90:0/4234822728 wait complete. 2024-04-02T14:25:08.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:08.019+0000 7f32c6200700 1 Processor -- start 2024-04-02T14:25:08.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:08.019+0000 7f32c6200700 1 -- start start 2024-04-02T14:25:08.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:08.020+0000 7f32c6200700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f32c0102070 0x7f32c019f7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:25:08.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:08.020+0000 7f32c6200700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f32c010c440 con 0x7f32c0102070 2024-04-02T14:25:08.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:08.020+0000 7f32c4e00700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f32c0102070 0x7f32c019f7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:25:08.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:08.020+0000 7f32c4e00700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f32c0102070 0x7f32c019f7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:39878/0 (socket says 172.21.15.90:39878) 2024-04-02T14:25:08.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:08.020+0000 7f32c4e00700 1 -- 172.21.15.90:0/3210938448 learned_addr learned my addr 172.21.15.90:0/3210938448 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:25:08.025 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:08.020+0000 7f32c4e00700 1 -- 172.21.15.90:0/3210938448 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32c019fd00 con 0x7f32c0102070 2024-04-02T14:25:08.025 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:08.021+0000 7f32c4e00700 1 --2- 172.21.15.90:0/3210938448 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f32c0102070 0x7f32c019f7c0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f32b4006010 tx=0x7f32b4005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:25:08.025 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:08.021+0000 7f32bd600700 1 -- 172.21.15.90:0/3210938448 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f32b4026070 con 0x7f32c0102070 2024-04-02T14:25:08.025 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:08.021+0000 7f32bd600700 1 -- 172.21.15.90:0/3210938448 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f32b400bbd0 con 0x7f32c0102070 2024-04-02T14:25:08.025 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:08.021+0000 7f32c6200700 1 -- 172.21.15.90:0/3210938448 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f32c019fff0 con 0x7f32c0102070 2024-04-02T14:25:08.025 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:08.021+0000 7f32bd600700 1 -- 172.21.15.90:0/3210938448 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f32b4003910 con 0x7f32c0102070 2024-04-02T14:25:08.025 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:08.021+0000 7f32c6200700 1 -- 172.21.15.90:0/3210938448 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32c0199980 con 0x7f32c0102070 2024-04-02T14:25:08.025 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:08.022+0000 7f32c6200700 1 -- 172.21.15.90:0/3210938448 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f32c0103550 con 0x7f32c0102070 2024-04-02T14:25:08.025 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:08.022+0000 7f32bd600700 1 -- 172.21.15.90:0/3210938448 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f32b4029070 con 0x7f32c0102070 2024-04-02T14:25:08.025 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:08.023+0000 7f32bd600700 1 --2- 172.21.15.90:0/3210938448 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f32a8039650 0x7f32a803bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:25:08.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:08.023+0000 7f32bd600700 1 -- 172.21.15.90:0/3210938448 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f32b401e070 con 0x7f32c0102070 2024-04-02T14:25:08.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:08.024+0000 7f32bfe00700 1 --2- 172.21.15.90:0/3210938448 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f32a8039650 0x7f32a803bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:25:08.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:08.024+0000 7f32bfe00700 1 --2- 172.21.15.90:0/3210938448 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f32a8039650 0x7f32a803bb10 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f32ac006fd0 tx=0x7f32ac006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:25:08.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:08.028+0000 7f32bd600700 1 -- 172.21.15.90:0/3210938448 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7f32b4013070 con 0x7f32c0102070 2024-04-02T14:25:08.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:08.248+0000 7f32c6200700 1 -- 172.21.15.90:0/3210938448 --> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi177", "target": ["mon-mgr", ""]}) -- 0x7f32c019aeb0 con 0x7f32a8039650 2024-04-02T14:25:08.688 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:08 smithi090 ceph-mon[31674]: Updating smithi092:/etc/ceph/ceph.conf 2024-04-02T14:25:08.688 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:08 smithi090 ceph-mon[31674]: Updating smithi092:/var/lib/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-04-02T14:25:09.938 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:09 smithi090 ceph-mon[31674]: Updating smithi092:/etc/ceph/ceph.client.admin.keyring 2024-04-02T14:25:09.939 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:09 smithi090 ceph-mon[31674]: Updating smithi092:/var/lib/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/config/ceph.client.admin.keyring 2024-04-02T14:25:09.939 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:09 smithi090 ceph-mon[31674]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi177", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T14:25:09.939 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:09 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:25:09.939 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:09 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:25:09.939 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:09 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:25:09.939 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:09 smithi090 ceph-mon[31674]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T14:25:10.938 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:10 smithi090 ceph-mon[31674]: Deploying cephadm binary to smithi177 2024-04-02T14:25:11.938 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:11 smithi090 ceph-mon[31674]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T14:25:12.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:12.987+0000 7f32bd600700 1 -- 172.21.15.90:0/3210938448 <== mgr.14152 v2:172.21.15.90:6800/4266054817 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7f32c019aeb0 con 0x7f32a8039650 2024-04-02T14:25:12.990 INFO:teuthology.orchestra.run.smithi090.stdout:Added host 'smithi177' with addr '172.21.15.177' 2024-04-02T14:25:12.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:12.991+0000 7f32c6200700 1 -- 172.21.15.90:0/3210938448 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f32a8039650 msgr2=0x7f32a803bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:25:12.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:12.991+0000 7f32c6200700 1 --2- 172.21.15.90:0/3210938448 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f32a8039650 0x7f32a803bb10 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f32ac006fd0 tx=0x7f32ac006e40 comp rx=0 tx=0).stop 2024-04-02T14:25:12.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:12.991+0000 7f32c6200700 1 -- 172.21.15.90:0/3210938448 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f32c0102070 msgr2=0x7f32c019f7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:25:12.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:12.991+0000 7f32c6200700 1 --2- 172.21.15.90:0/3210938448 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f32c0102070 0x7f32c019f7c0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f32b4006010 tx=0x7f32b4005030 comp rx=0 tx=0).stop 2024-04-02T14:25:12.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:12.991+0000 7f32c6200700 1 -- 172.21.15.90:0/3210938448 shutdown_connections 2024-04-02T14:25:12.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:12.991+0000 7f32c6200700 1 --2- 172.21.15.90:0/3210938448 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f32a8039650 0x7f32a803bb10 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:25:12.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:12.991+0000 7f32c6200700 1 --2- 172.21.15.90:0/3210938448 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f32c0102070 0x7f32c019f7c0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:25:12.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:12.991+0000 7f32c6200700 1 -- 172.21.15.90:0/3210938448 >> 172.21.15.90:0/3210938448 conn(0x7f32c00fd920 msgr2=0x7f32c0074820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:25:12.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:12.991+0000 7f32c6200700 1 -- 172.21.15.90:0/3210938448 shutdown_connections 2024-04-02T14:25:12.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:12.992+0000 7f32c6200700 1 -- 172.21.15.90:0/3210938448 wait complete. 2024-04-02T14:25:13.669 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:191d963ac9de7322da8aff20b567b3fa458e2ef6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6449448a-f0fc-11ee-b647-cb9ed24678a4 -- ceph orch host ls --format=json 2024-04-02T14:25:13.938 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:13 smithi090 ceph-mon[31674]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T14:25:13.939 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:13 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:25:13.939 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:13 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T14:25:13.939 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:13 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:25:14.102 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T14:25:14.939 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:14 smithi090 ceph-mon[31674]: Added host smithi177 2024-04-02T14:25:14.939 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:14 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:25:15.853 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:15 smithi090 ceph-mon[31674]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T14:25:16.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:16.018+0000 7fc874a00700 1 -- 172.21.15.90:0/3122761802 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc870103170 msgr2=0x7fc870103550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:25:16.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:16.018+0000 7fc874a00700 1 --2- 172.21.15.90:0/3122761802 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc870103170 0x7fc870103550 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fc860009910 tx=0x7fc860009c20 comp rx=0 tx=0).stop 2024-04-02T14:25:16.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:16.018+0000 7fc874a00700 1 -- 172.21.15.90:0/3122761802 shutdown_connections 2024-04-02T14:25:16.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:16.018+0000 7fc874a00700 1 --2- 172.21.15.90:0/3122761802 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc870103170 0x7fc870103550 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:25:16.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:16.018+0000 7fc874a00700 1 -- 172.21.15.90:0/3122761802 >> 172.21.15.90:0/3122761802 conn(0x7fc8700fc9a0 msgr2=0x7fc8700fedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:25:16.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:16.018+0000 7fc874a00700 1 -- 172.21.15.90:0/3122761802 shutdown_connections 2024-04-02T14:25:16.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:16.018+0000 7fc874a00700 1 -- 172.21.15.90:0/3122761802 wait complete. 2024-04-02T14:25:16.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:16.019+0000 7fc874a00700 1 Processor -- start 2024-04-02T14:25:16.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:16.019+0000 7fc874a00700 1 -- start start 2024-04-02T14:25:16.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:16.019+0000 7fc874a00700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc870103170 0x7fc870199d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:25:16.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:16.019+0000 7fc874a00700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fc87010c660 con 0x7fc870103170 2024-04-02T14:25:16.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:16.020+0000 7fc86de00700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc870103170 0x7fc870199d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:25:16.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:16.020+0000 7fc86de00700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc870103170 0x7fc870199d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34682/0 (socket says 172.21.15.90:34682) 2024-04-02T14:25:16.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:16.020+0000 7fc86de00700 1 -- 172.21.15.90:0/2586585483 learned_addr learned my addr 172.21.15.90:0/2586585483 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:25:16.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:16.020+0000 7fc86de00700 1 -- 172.21.15.90:0/2586585483 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc87019a270 con 0x7fc870103170 2024-04-02T14:25:16.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:16.020+0000 7fc86de00700 1 --2- 172.21.15.90:0/2586585483 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc870103170 0x7fc870199d30 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fc860006010 tx=0x7fc860005050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:25:16.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:16.020+0000 7fc866a00700 1 -- 172.21.15.90:0/2586585483 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc860026070 con 0x7fc870103170 2024-04-02T14:25:16.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:16.021+0000 7fc866a00700 1 -- 172.21.15.90:0/2586585483 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc86000bc30 con 0x7fc870103170 2024-04-02T14:25:16.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:16.021+0000 7fc866a00700 1 -- 172.21.15.90:0/2586585483 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc860003ab0 con 0x7fc870103170 2024-04-02T14:25:16.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:16.021+0000 7fc874a00700 1 -- 172.21.15.90:0/2586585483 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc87019a500 con 0x7fc870103170 2024-04-02T14:25:16.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:16.021+0000 7fc874a00700 1 -- 172.21.15.90:0/2586585483 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc87019a960 con 0x7fc870103170 2024-04-02T14:25:16.025 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:16.022+0000 7fc874a00700 1 -- 172.21.15.90:0/2586585483 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc87006ab50 con 0x7fc870103170 2024-04-02T14:25:16.025 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:16.022+0000 7fc866a00700 1 -- 172.21.15.90:0/2586585483 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7fc860029070 con 0x7fc870103170 2024-04-02T14:25:16.025 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:16.023+0000 7fc866a00700 1 --2- 172.21.15.90:0/2586585483 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fc8540396b0 0x7fc85403bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:25:16.025 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:16.023+0000 7fc866a00700 1 -- 172.21.15.90:0/2586585483 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fc86001e070 con 0x7fc870103170 2024-04-02T14:25:16.025 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:16.023+0000 7fc86d400700 1 --2- 172.21.15.90:0/2586585483 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fc8540396b0 0x7fc85403bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:25:16.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:16.024+0000 7fc86d400700 1 --2- 172.21.15.90:0/2586585483 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fc8540396b0 0x7fc85403bb70 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fc85c006fd0 tx=0x7fc85c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:25:16.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:16.028+0000 7fc866a00700 1 -- 172.21.15.90:0/2586585483 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7fc860013070 con 0x7fc870103170 2024-04-02T14:25:16.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:16.263+0000 7fc874a00700 1 -- 172.21.15.90:0/2586585483 --> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fc87019b090 con 0x7fc8540396b0 2024-04-02T14:25:16.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:16.265+0000 7fc866a00700 1 -- 172.21.15.90:0/2586585483 <== mgr.14152 v2:172.21.15.90:6800/4266054817 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+239 (secure 0 0 0) 0x7fc87019b090 con 0x7fc8540396b0 2024-04-02T14:25:16.268 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-02T14:25:16.268 INFO:teuthology.orchestra.run.smithi090.stdout:[{"addr": "172.21.15.90", "hostname": "smithi090", "labels": [], "status": ""}, {"addr": "172.21.15.92", "hostname": "smithi092", "labels": [], "status": ""}, {"addr": "172.21.15.177", "hostname": "smithi177", "labels": [], "status": ""}] 2024-04-02T14:25:16.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:16.268+0000 7fc874a00700 1 -- 172.21.15.90:0/2586585483 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fc8540396b0 msgr2=0x7fc85403bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:25:16.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:16.268+0000 7fc874a00700 1 --2- 172.21.15.90:0/2586585483 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fc8540396b0 0x7fc85403bb70 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fc85c006fd0 tx=0x7fc85c006e40 comp rx=0 tx=0).stop 2024-04-02T14:25:16.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:16.268+0000 7fc874a00700 1 -- 172.21.15.90:0/2586585483 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc870103170 msgr2=0x7fc870199d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:25:16.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:16.268+0000 7fc874a00700 1 --2- 172.21.15.90:0/2586585483 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc870103170 0x7fc870199d30 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fc860006010 tx=0x7fc860005050 comp rx=0 tx=0).stop 2024-04-02T14:25:16.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:16.269+0000 7fc874a00700 1 -- 172.21.15.90:0/2586585483 shutdown_connections 2024-04-02T14:25:16.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:16.269+0000 7fc874a00700 1 --2- 172.21.15.90:0/2586585483 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fc8540396b0 0x7fc85403bb70 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:25:16.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:16.269+0000 7fc874a00700 1 --2- 172.21.15.90:0/2586585483 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc870103170 0x7fc870199d30 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:25:16.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:16.269+0000 7fc874a00700 1 -- 172.21.15.90:0/2586585483 >> 172.21.15.90:0/2586585483 conn(0x7fc8700fc9a0 msgr2=0x7fc870108330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:25:16.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:16.269+0000 7fc874a00700 1 -- 172.21.15.90:0/2586585483 shutdown_connections 2024-04-02T14:25:16.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:16.269+0000 7fc874a00700 1 -- 172.21.15.90:0/2586585483 wait complete. 2024-04-02T14:25:16.915 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:16 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:25:16.986 INFO:tasks.cephadm:Setting crush tunables to default 2024-04-02T14:25:16.987 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:191d963ac9de7322da8aff20b567b3fa458e2ef6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6449448a-f0fc-11ee-b647-cb9ed24678a4 -- ceph osd crush tunables default 2024-04-02T14:25:17.421 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T14:25:17.870 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:17 smithi090 ceph-mon[31674]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-02T14:25:17.870 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:17 smithi090 ceph-mon[31674]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T14:25:19.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:19.378+0000 7fc18c200700 1 -- 172.21.15.90:0/2864093391 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc184104450 msgr2=0x7fc184104830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:25:19.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:19.378+0000 7fc18c200700 1 --2- 172.21.15.90:0/2864093391 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc184104450 0x7fc184104830 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fc178009910 tx=0x7fc178009c20 comp rx=0 tx=0).stop 2024-04-02T14:25:19.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:19.378+0000 7fc18c200700 1 -- 172.21.15.90:0/2864093391 shutdown_connections 2024-04-02T14:25:19.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:19.378+0000 7fc18c200700 1 --2- 172.21.15.90:0/2864093391 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc184104450 0x7fc184104830 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:25:19.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:19.378+0000 7fc18c200700 1 -- 172.21.15.90:0/2864093391 >> 172.21.15.90:0/2864093391 conn(0x7fc1840ffce0 msgr2=0x7fc184102100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:25:19.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:19.379+0000 7fc18c200700 1 -- 172.21.15.90:0/2864093391 shutdown_connections 2024-04-02T14:25:19.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:19.379+0000 7fc18c200700 1 -- 172.21.15.90:0/2864093391 wait complete. 2024-04-02T14:25:19.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:19.380+0000 7fc18c200700 1 Processor -- start 2024-04-02T14:25:19.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:19.380+0000 7fc18c200700 1 -- start start 2024-04-02T14:25:19.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:19.380+0000 7fc18c200700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc184104450 0x7fc18419fa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:25:19.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:19.380+0000 7fc18c200700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fc18410e820 con 0x7fc184104450 2024-04-02T14:25:19.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:19.380+0000 7fc189800700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc184104450 0x7fc18419fa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:25:19.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:19.380+0000 7fc189800700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc184104450 0x7fc18419fa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34700/0 (socket says 172.21.15.90:34700) 2024-04-02T14:25:19.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:19.380+0000 7fc189800700 1 -- 172.21.15.90:0/3042809071 learned_addr learned my addr 172.21.15.90:0/3042809071 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:25:19.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:19.381+0000 7fc189800700 1 -- 172.21.15.90:0/3042809071 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc18419ff80 con 0x7fc184104450 2024-04-02T14:25:19.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:19.381+0000 7fc189800700 1 --2- 172.21.15.90:0/3042809071 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc184104450 0x7fc18419fa40 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fc178006010 tx=0x7fc178005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:25:19.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:19.381+0000 7fc182000700 1 -- 172.21.15.90:0/3042809071 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc178026070 con 0x7fc184104450 2024-04-02T14:25:19.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:19.381+0000 7fc182000700 1 -- 172.21.15.90:0/3042809071 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc17800bbd0 con 0x7fc184104450 2024-04-02T14:25:19.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:19.381+0000 7fc182000700 1 -- 172.21.15.90:0/3042809071 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc178003910 con 0x7fc184104450 2024-04-02T14:25:19.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:19.382+0000 7fc18c200700 1 -- 172.21.15.90:0/3042809071 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc1841a0270 con 0x7fc184104450 2024-04-02T14:25:19.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:19.382+0000 7fc18c200700 1 -- 172.21.15.90:0/3042809071 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc184199ce0 con 0x7fc184104450 2024-04-02T14:25:19.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:19.383+0000 7fc182000700 1 -- 172.21.15.90:0/3042809071 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7fc178029070 con 0x7fc184104450 2024-04-02T14:25:19.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:19.383+0000 7fc18c200700 1 -- 172.21.15.90:0/3042809071 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc184105930 con 0x7fc184104450 2024-04-02T14:25:19.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:19.383+0000 7fc182000700 1 --2- 172.21.15.90:0/3042809071 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fc16c0396b0 0x7fc16c03bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:25:19.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:19.384+0000 7fc182000700 1 -- 172.21.15.90:0/3042809071 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fc17801e070 con 0x7fc184104450 2024-04-02T14:25:19.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:19.384+0000 7fc188e00700 1 --2- 172.21.15.90:0/3042809071 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fc16c0396b0 0x7fc16c03bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:25:19.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:19.385+0000 7fc188e00700 1 --2- 172.21.15.90:0/3042809071 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fc16c0396b0 0x7fc16c03bb70 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fc170006fd0 tx=0x7fc170006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:25:19.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:19.389+0000 7fc182000700 1 -- 172.21.15.90:0/3042809071 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7fc178013070 con 0x7fc184104450 2024-04-02T14:25:19.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:19.620+0000 7fc18c200700 1 -- 172.21.15.90:0/3042809071 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7fc18419afd0 con 0x7fc184104450 2024-04-02T14:25:19.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:19.919+0000 7fc182000700 1 -- 172.21.15.90:0/3042809071 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) ==== 124+0+0 (secure 0 0 0) 0x7fc17800fe20 con 0x7fc184104450 2024-04-02T14:25:19.922 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:19 smithi090 ceph-mon[31674]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T14:25:19.922 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:19 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:25:19.922 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:19 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:25:19.922 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:19 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:25:19.922 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:19 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:25:19.922 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:19 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi177", "name": "osd_memory_target"}]: dispatch 2024-04-02T14:25:19.922 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:19 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:25:19.922 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:19 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T14:25:19.922 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:19 smithi090 ceph-mon[31674]: from='client.? 172.21.15.90:0/3042809071' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-02T14:25:19.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:19.924+0000 7fc18c200700 1 -- 172.21.15.90:0/3042809071 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fc16c0396b0 msgr2=0x7fc16c03bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:25:19.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:19.924+0000 7fc18c200700 1 --2- 172.21.15.90:0/3042809071 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fc16c0396b0 0x7fc16c03bb70 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fc170006fd0 tx=0x7fc170006e40 comp rx=0 tx=0).stop 2024-04-02T14:25:19.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:19.924+0000 7fc18c200700 1 -- 172.21.15.90:0/3042809071 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc184104450 msgr2=0x7fc18419fa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:25:19.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:19.924+0000 7fc18c200700 1 --2- 172.21.15.90:0/3042809071 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc184104450 0x7fc18419fa40 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fc178006010 tx=0x7fc178005030 comp rx=0 tx=0).stop 2024-04-02T14:25:19.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:19.924+0000 7fc18c200700 1 -- 172.21.15.90:0/3042809071 shutdown_connections 2024-04-02T14:25:19.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:19.924+0000 7fc18c200700 1 --2- 172.21.15.90:0/3042809071 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fc16c0396b0 0x7fc16c03bb70 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:25:19.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:19.924+0000 7fc18c200700 1 --2- 172.21.15.90:0/3042809071 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc184104450 0x7fc18419fa40 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:25:19.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:19.924+0000 7fc18c200700 1 -- 172.21.15.90:0/3042809071 >> 172.21.15.90:0/3042809071 conn(0x7fc1840ffce0 msgr2=0x7fc184105e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:25:19.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:19.924+0000 7fc18c200700 1 -- 172.21.15.90:0/3042809071 shutdown_connections 2024-04-02T14:25:19.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:19.925+0000 7fc18c200700 1 -- 172.21.15.90:0/3042809071 wait complete. 2024-04-02T14:25:19.928 INFO:teuthology.orchestra.run.smithi090.stderr:adjusted tunables profile to default 2024-04-02T14:25:20.658 INFO:tasks.cephadm:Adding mon.a on smithi090 2024-04-02T14:25:20.658 INFO:tasks.cephadm:Adding mon.b on smithi092 2024-04-02T14:25:20.658 INFO:tasks.cephadm:Adding mon.c on smithi177 2024-04-02T14:25:20.658 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:191d963ac9de7322da8aff20b567b3fa458e2ef6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6449448a-f0fc-11ee-b647-cb9ed24678a4 -- ceph orch apply mon '3;smithi090:172.21.15.90=a;smithi092:172.21.15.92=b;smithi177:172.21.15.177=c' 2024-04-02T14:25:20.887 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-04-02T14:25:20.939 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:20 smithi090 ceph-mon[31674]: Updating smithi177:/etc/ceph/ceph.conf 2024-04-02T14:25:20.939 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:20 smithi090 ceph-mon[31674]: Updating smithi177:/var/lib/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-04-02T14:25:20.940 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:20 smithi090 ceph-mon[31674]: Updating smithi177:/etc/ceph/ceph.client.admin.keyring 2024-04-02T14:25:20.940 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:20 smithi090 ceph-mon[31674]: Updating smithi177:/var/lib/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/config/ceph.client.admin.keyring 2024-04-02T14:25:20.940 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:20 smithi090 ceph-mon[31674]: from='client.? 172.21.15.90:0/3042809071' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-02T14:25:20.940 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:20 smithi090 ceph-mon[31674]: osdmap e4: 0 total, 0 up, 0 in 2024-04-02T14:25:20.940 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:20 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:25:20.940 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:20 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:25:20.940 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:20 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:25:22.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:21 smithi090 ceph-mon[31674]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T14:25:22.203 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:22.201+0000 7ffbe81d3700 1 -- 172.21.15.177:0/961464171 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffbe0102230 msgr2=0x7ffbe0102610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:25:22.203 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:22.201+0000 7ffbe81d3700 1 --2- 172.21.15.177:0/961464171 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffbe0102230 0x7ffbe0102610 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7ffbdc009910 tx=0x7ffbdc009c20 comp rx=0 tx=0).stop 2024-04-02T14:25:22.204 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:22.201+0000 7ffbe81d3700 1 -- 172.21.15.177:0/961464171 shutdown_connections 2024-04-02T14:25:22.204 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:22.201+0000 7ffbe81d3700 1 --2- 172.21.15.177:0/961464171 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffbe0102230 0x7ffbe0102610 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:25:22.204 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:22.201+0000 7ffbe81d3700 1 -- 172.21.15.177:0/961464171 >> 172.21.15.177:0/961464171 conn(0x7ffbe00fdb00 msgr2=0x7ffbe00fff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:25:22.204 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:22.201+0000 7ffbe81d3700 1 -- 172.21.15.177:0/961464171 shutdown_connections 2024-04-02T14:25:22.204 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:22.201+0000 7ffbe81d3700 1 -- 172.21.15.177:0/961464171 wait complete. 2024-04-02T14:25:22.204 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:22.202+0000 7ffbe81d3700 1 Processor -- start 2024-04-02T14:25:22.204 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:22.202+0000 7ffbe81d3700 1 -- start start 2024-04-02T14:25:22.204 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:22.202+0000 7ffbe81d3700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffbe0102230 0x7ffbe019d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:25:22.204 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:22.202+0000 7ffbe81d3700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7ffbe010c600 con 0x7ffbe0102230 2024-04-02T14:25:22.205 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:22.202+0000 7ffbe5f6f700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffbe0102230 0x7ffbe019d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:25:22.205 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:22.202+0000 7ffbe5f6f700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffbe0102230 0x7ffbe019d810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.177:45064/0 (socket says 172.21.15.177:45064) 2024-04-02T14:25:22.205 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:22.202+0000 7ffbe5f6f700 1 -- 172.21.15.177:0/2997408242 learned_addr learned my addr 172.21.15.177:0/2997408242 (peer_addr_for_me v2:172.21.15.177:0/0) 2024-04-02T14:25:22.206 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:22.203+0000 7ffbe5f6f700 1 -- 172.21.15.177:0/2997408242 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffbe019dd50 con 0x7ffbe0102230 2024-04-02T14:25:22.206 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:22.203+0000 7ffbe5f6f700 1 --2- 172.21.15.177:0/2997408242 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffbe0102230 0x7ffbe019d810 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7ffbdc005850 tx=0x7ffbdc005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:25:22.206 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:22.203+0000 7ffbd6ffd700 1 -- 172.21.15.177:0/2997408242 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ffbdc026070 con 0x7ffbe0102230 2024-04-02T14:25:22.206 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:22.203+0000 7ffbe81d3700 1 -- 172.21.15.177:0/2997408242 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffbe019e040 con 0x7ffbe0102230 2024-04-02T14:25:22.206 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:22.204+0000 7ffbe81d3700 1 -- 172.21.15.177:0/2997408242 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffbe01979d0 con 0x7ffbe0102230 2024-04-02T14:25:22.206 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:22.204+0000 7ffbd6ffd700 1 -- 172.21.15.177:0/2997408242 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffbdc00b6f0 con 0x7ffbe0102230 2024-04-02T14:25:22.206 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:22.204+0000 7ffbd6ffd700 1 -- 172.21.15.177:0/2997408242 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ffbdc030060 con 0x7ffbe0102230 2024-04-02T14:25:22.207 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:22.204+0000 7ffbe81d3700 1 -- 172.21.15.177:0/2997408242 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffbc4005330 con 0x7ffbe0102230 2024-04-02T14:25:22.207 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:22.205+0000 7ffbd6ffd700 1 -- 172.21.15.177:0/2997408242 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7ffbdc029070 con 0x7ffbe0102230 2024-04-02T14:25:22.207 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:22.205+0000 7ffbd6ffd700 1 --2- 172.21.15.177:0/2997408242 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7ffbcc039600 0x7ffbcc03bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:25:22.207 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:22.205+0000 7ffbd6ffd700 1 -- 172.21.15.177:0/2997408242 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7ffbdc01e070 con 0x7ffbe0102230 2024-04-02T14:25:22.207 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:22.206+0000 7ffbe576e700 1 --2- 172.21.15.177:0/2997408242 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7ffbcc039600 0x7ffbcc03bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:25:22.208 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:22.207+0000 7ffbe576e700 1 --2- 172.21.15.177:0/2997408242 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7ffbcc039600 0x7ffbcc03bac0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7ffbd0006fd0 tx=0x7ffbd0006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:25:22.223 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:22.221+0000 7ffbd6ffd700 1 -- 172.21.15.177:0/2997408242 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7ffbdc013070 con 0x7ffbe0102230 2024-04-02T14:25:22.432 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:22.430+0000 7ffbe81d3700 1 -- 172.21.15.177:0/2997408242 --> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi090:172.21.15.90=a;smithi092:172.21.15.92=b;smithi177:172.21.15.177=c", "target": ["mon-mgr", ""]}) -- 0x7ffbc4000c80 con 0x7ffbcc039600 2024-04-02T14:25:22.447 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:22.445+0000 7ffbd6ffd700 1 -- 172.21.15.177:0/2997408242 <== mgr.14152 v2:172.21.15.90:6800/4266054817 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7ffbc4000c80 con 0x7ffbcc039600 2024-04-02T14:25:22.447 INFO:teuthology.orchestra.run.smithi177.stdout:Scheduled mon update... 2024-04-02T14:25:22.449 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:22.447+0000 7ffbe81d3700 1 -- 172.21.15.177:0/2997408242 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7ffbcc039600 msgr2=0x7ffbcc03bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:25:22.449 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:22.447+0000 7ffbe81d3700 1 --2- 172.21.15.177:0/2997408242 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7ffbcc039600 0x7ffbcc03bac0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7ffbd0006fd0 tx=0x7ffbd0006e40 comp rx=0 tx=0).stop 2024-04-02T14:25:22.449 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:22.447+0000 7ffbe81d3700 1 -- 172.21.15.177:0/2997408242 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffbe0102230 msgr2=0x7ffbe019d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:25:22.449 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:22.447+0000 7ffbe81d3700 1 --2- 172.21.15.177:0/2997408242 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffbe0102230 0x7ffbe019d810 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7ffbdc005850 tx=0x7ffbdc005030 comp rx=0 tx=0).stop 2024-04-02T14:25:22.450 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:22.447+0000 7ffbe81d3700 1 -- 172.21.15.177:0/2997408242 shutdown_connections 2024-04-02T14:25:22.450 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:22.448+0000 7ffbe81d3700 1 --2- 172.21.15.177:0/2997408242 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7ffbcc039600 0x7ffbcc03bac0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:25:22.450 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:22.448+0000 7ffbe81d3700 1 --2- 172.21.15.177:0/2997408242 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffbe0102230 0x7ffbe019d810 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:25:22.450 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:22.448+0000 7ffbe81d3700 1 -- 172.21.15.177:0/2997408242 >> 172.21.15.177:0/2997408242 conn(0x7ffbe00fdb00 msgr2=0x7ffbe0105060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:25:22.450 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:22.448+0000 7ffbe81d3700 1 -- 172.21.15.177:0/2997408242 shutdown_connections 2024-04-02T14:25:22.450 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:22.448+0000 7ffbe81d3700 1 -- 172.21.15.177:0/2997408242 wait complete. 2024-04-02T14:25:23.158 DEBUG:teuthology.orchestra.run.smithi092:mon.b> sudo journalctl -f -n 0 -u ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4@mon.b.service 2024-04-02T14:25:23.160 DEBUG:teuthology.orchestra.run.smithi177:mon.c> sudo journalctl -f -n 0 -u ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4@mon.c.service 2024-04-02T14:25:23.163 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-02T14:25:23.163 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:191d963ac9de7322da8aff20b567b3fa458e2ef6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6449448a-f0fc-11ee-b647-cb9ed24678a4 -- ceph mon dump -f json 2024-04-02T14:25:23.939 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:23 smithi090 ceph-mon[31674]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi090:172.21.15.90=a;smithi092:172.21.15.92=b;smithi177:172.21.15.177=c", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T14:25:23.939 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:23 smithi090 ceph-mon[31674]: Saving service mon spec with placement smithi090:172.21.15.90=a;smithi092:172.21.15.92=b;smithi177:172.21.15.177=c;count:3 2024-04-02T14:25:23.940 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:23 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:25:23.940 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:23 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T14:25:23.940 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:23 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:25:23.940 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:23 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T14:25:23.940 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:23 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:25:23.940 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:23 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-02T14:25:23.940 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:23 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:25:23.940 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:23 smithi090 ceph-mon[31674]: Deploying daemon mon.c on smithi177 2024-04-02T14:25:23.940 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:23 smithi090 ceph-mon[31674]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T14:25:25.173 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/mon.c/config 2024-04-02T14:25:25.939 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:25 smithi090 ceph-mon[31674]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T14:25:26.728 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:26.725+0000 7f2f76a38700 1 -- 172.21.15.177:0/3298906246 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2f70101210 msgr2=0x7f2f701015f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:25:26.728 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:26.725+0000 7f2f76a38700 1 --2- 172.21.15.177:0/3298906246 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2f70101210 0x7f2f701015f0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f2f60009910 tx=0x7f2f60009c20 comp rx=0 tx=0).stop 2024-04-02T14:25:26.728 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:26.725+0000 7f2f76a38700 1 -- 172.21.15.177:0/3298906246 shutdown_connections 2024-04-02T14:25:26.728 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:26.725+0000 7f2f76a38700 1 --2- 172.21.15.177:0/3298906246 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2f70101210 0x7f2f701015f0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:25:26.728 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:26.725+0000 7f2f76a38700 1 -- 172.21.15.177:0/3298906246 >> 172.21.15.177:0/3298906246 conn(0x7f2f700fcaa0 msgr2=0x7f2f700feec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:25:26.728 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:26.725+0000 7f2f76a38700 1 -- 172.21.15.177:0/3298906246 shutdown_connections 2024-04-02T14:25:26.729 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:26.725+0000 7f2f76a38700 1 -- 172.21.15.177:0/3298906246 wait complete. 2024-04-02T14:25:26.729 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:26.726+0000 7f2f76a38700 1 Processor -- start 2024-04-02T14:25:26.729 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:26.726+0000 7f2f76a38700 1 -- start start 2024-04-02T14:25:26.729 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:26.726+0000 7f2f76a38700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2f70101210 0x7f2f7019c760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:25:26.729 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:26.726+0000 7f2f76a38700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f2f7010b5e0 con 0x7f2f70101210 2024-04-02T14:25:26.730 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:26.727+0000 7f2f6ffff700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2f70101210 0x7f2f7019c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:25:26.731 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:26.727+0000 7f2f6ffff700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2f70101210 0x7f2f7019c760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.177:45090/0 (socket says 172.21.15.177:45090) 2024-04-02T14:25:26.731 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:26.727+0000 7f2f6ffff700 1 -- 172.21.15.177:0/1263534294 learned_addr learned my addr 172.21.15.177:0/1263534294 (peer_addr_for_me v2:172.21.15.177:0/0) 2024-04-02T14:25:26.731 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:26.727+0000 7f2f6ffff700 1 -- 172.21.15.177:0/1263534294 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f7019cca0 con 0x7f2f70101210 2024-04-02T14:25:26.731 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:26.728+0000 7f2f6ffff700 1 --2- 172.21.15.177:0/1263534294 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2f70101210 0x7f2f7019c760 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f2f60005850 tx=0x7f2f60005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:25:26.731 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:26.728+0000 7f2f6d7fa700 1 -- 172.21.15.177:0/1263534294 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2f60026070 con 0x7f2f70101210 2024-04-02T14:25:26.731 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:26.728+0000 7f2f76a38700 1 -- 172.21.15.177:0/1263534294 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2f7019cf90 con 0x7f2f70101210 2024-04-02T14:25:26.731 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:26.728+0000 7f2f76a38700 1 -- 172.21.15.177:0/1263534294 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2f70196920 con 0x7f2f70101210 2024-04-02T14:25:26.731 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:26.728+0000 7f2f6d7fa700 1 -- 172.21.15.177:0/1263534294 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2f6000bbd0 con 0x7f2f70101210 2024-04-02T14:25:26.731 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:26.728+0000 7f2f6d7fa700 1 -- 172.21.15.177:0/1263534294 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2f600039b0 con 0x7f2f70101210 2024-04-02T14:25:26.731 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:26.729+0000 7f2f76a38700 1 -- 172.21.15.177:0/1263534294 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2f701026f0 con 0x7f2f70101210 2024-04-02T14:25:26.732 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:26.729+0000 7f2f6d7fa700 1 -- 172.21.15.177:0/1263534294 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f2f60029070 con 0x7f2f70101210 2024-04-02T14:25:26.732 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:26.730+0000 7f2f6d7fa700 1 --2- 172.21.15.177:0/1263534294 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f2f58039650 0x7f2f5803bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:25:26.732 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:26.730+0000 7f2f6d7fa700 1 -- 172.21.15.177:0/1263534294 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f2f6001e070 con 0x7f2f70101210 2024-04-02T14:25:26.733 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:26.730+0000 7f2f6f7fe700 1 --2- 172.21.15.177:0/1263534294 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f2f58039650 0x7f2f5803bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:25:26.733 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:26.731+0000 7f2f6f7fe700 1 --2- 172.21.15.177:0/1263534294 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f2f58039650 0x7f2f5803bb10 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f2f64006fd0 tx=0x7f2f64006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:25:26.747 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:26.745+0000 7f2f6d7fa700 1 -- 172.21.15.177:0/1263534294 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7f2f60013070 con 0x7f2f70101210 2024-04-02T14:25:27.041 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:27.038+0000 7f2f76a38700 1 -- 172.21.15.177:0/1263534294 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f2f70197e50 con 0x7f2f70101210 2024-04-02T14:25:27.041 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:27.039+0000 7f2f6d7fa700 1 -- 172.21.15.177:0/1263534294 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) ==== 95+0+745 (secure 0 0 0) 0x7f2f60034090 con 0x7f2f70101210 2024-04-02T14:25:27.041 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-02T14:25:27.041 INFO:teuthology.orchestra.run.smithi177.stdout:{"epoch":1,"fsid":"6449448a-f0fc-11ee-b647-cb9ed24678a4","modified":"2024-04-02T14:23:12.545436Z","created":"2024-04-02T14:23:12.545436Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-02T14:25:27.042 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:27.041+0000 7f2f76a38700 1 -- 172.21.15.177:0/1263534294 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f2f58039650 msgr2=0x7f2f5803bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:25:27.043 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:27.041+0000 7f2f76a38700 1 --2- 172.21.15.177:0/1263534294 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f2f58039650 0x7f2f5803bb10 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f2f64006fd0 tx=0x7f2f64006e40 comp rx=0 tx=0).stop 2024-04-02T14:25:27.043 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:27.041+0000 7f2f76a38700 1 -- 172.21.15.177:0/1263534294 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2f70101210 msgr2=0x7f2f7019c760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:25:27.043 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:27.041+0000 7f2f76a38700 1 --2- 172.21.15.177:0/1263534294 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2f70101210 0x7f2f7019c760 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f2f60005850 tx=0x7f2f60005030 comp rx=0 tx=0).stop 2024-04-02T14:25:27.043 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:27.041+0000 7f2f76a38700 1 -- 172.21.15.177:0/1263534294 shutdown_connections 2024-04-02T14:25:27.043 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:27.041+0000 7f2f76a38700 1 --2- 172.21.15.177:0/1263534294 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f2f58039650 0x7f2f5803bb10 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:25:27.043 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:27.041+0000 7f2f76a38700 1 --2- 172.21.15.177:0/1263534294 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2f70101210 0x7f2f7019c760 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:25:27.043 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:27.041+0000 7f2f76a38700 1 -- 172.21.15.177:0/1263534294 >> 172.21.15.177:0/1263534294 conn(0x7f2f700fcaa0 msgr2=0x7f2f70102be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:25:27.043 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:27.041+0000 7f2f76a38700 1 -- 172.21.15.177:0/1263534294 shutdown_connections 2024-04-02T14:25:27.043 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:27.041+0000 7f2f76a38700 1 -- 172.21.15.177:0/1263534294 wait complete. 2024-04-02T14:25:27.044 INFO:teuthology.orchestra.run.smithi177.stderr:dumped monmap epoch 1 2024-04-02T14:25:27.304 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:27 smithi177 ceph-mon[36431]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-04-02T14:25:27.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:27 smithi177 bash[36403]: caa84bb1a070cfe74a5016e64d8314759ba36a3063035b9aec22e9a11b8d3669 2024-04-02T14:25:27.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:27 smithi177 systemd[1]: Started Ceph mon.c for 6449448a-f0fc-11ee-b647-cb9ed24678a4. 2024-04-02T14:25:28.758 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-02T14:25:28.758 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:191d963ac9de7322da8aff20b567b3fa458e2ef6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6449448a-f0fc-11ee-b647-cb9ed24678a4 -- ceph mon dump -f json 2024-04-02T14:25:29.002 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/mon.c/config 2024-04-02T14:25:32.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:32 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-02T14:25:32.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:32 smithi177 ceph-mon[36431]: mon.a calling monitor election 2024-04-02T14:25:32.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:32 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-02T14:25:32.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:32 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-02T14:25:32.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:32 smithi177 ceph-mon[36431]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T14:25:32.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:32 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-02T14:25:32.688 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:32 smithi177 ceph-mon[36431]: mon.c calling monitor election 2024-04-02T14:25:32.688 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:32 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-02T14:25:32.688 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:32 smithi177 ceph-mon[36431]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T14:25:32.688 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:32 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-02T14:25:32.688 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:32 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-02T14:25:32.688 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:32 smithi177 ceph-mon[36431]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-02T14:25:32.688 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:32 smithi177 ceph-mon[36431]: monmap epoch 2 2024-04-02T14:25:32.688 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:32 smithi177 ceph-mon[36431]: fsid 6449448a-f0fc-11ee-b647-cb9ed24678a4 2024-04-02T14:25:32.688 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:32 smithi177 ceph-mon[36431]: last_changed 2024-04-02T14:25:27.128759+0000 2024-04-02T14:25:32.688 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:32 smithi177 ceph-mon[36431]: created 2024-04-02T14:23:12.545436+0000 2024-04-02T14:25:32.688 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:32 smithi177 ceph-mon[36431]: min_mon_release 19 (squid) 2024-04-02T14:25:32.688 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:32 smithi177 ceph-mon[36431]: election_strategy: 1 2024-04-02T14:25:32.689 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:32 smithi177 ceph-mon[36431]: 0: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.a 2024-04-02T14:25:32.689 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:32 smithi177 ceph-mon[36431]: 1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.c 2024-04-02T14:25:32.689 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:32 smithi177 ceph-mon[36431]: fsmap 2024-04-02T14:25:32.689 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:32 smithi177 ceph-mon[36431]: osdmap e4: 0 total, 0 up, 0 in 2024-04-02T14:25:32.689 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:32 smithi177 ceph-mon[36431]: mgrmap e13: x(active, since 57s) 2024-04-02T14:25:32.689 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:32 smithi177 ceph-mon[36431]: overall HEALTH_OK 2024-04-02T14:25:32.689 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:32 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:25:32.689 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:32 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:25:32.689 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:32 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:25:32.690 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:32 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-02T14:25:32.690 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:32 smithi090 ceph-mon[31674]: mon.a calling monitor election 2024-04-02T14:25:32.690 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:32 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-02T14:25:32.690 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:32 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-02T14:25:32.690 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:32 smithi090 ceph-mon[31674]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T14:25:32.691 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:32 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-02T14:25:32.691 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:32 smithi090 ceph-mon[31674]: mon.c calling monitor election 2024-04-02T14:25:32.691 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:32 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-02T14:25:32.691 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:32 smithi090 ceph-mon[31674]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T14:25:32.691 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:32 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-02T14:25:32.691 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:32 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-02T14:25:32.691 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:32 smithi090 ceph-mon[31674]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-02T14:25:32.691 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:32 smithi090 ceph-mon[31674]: monmap epoch 2 2024-04-02T14:25:32.691 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:32 smithi090 ceph-mon[31674]: fsid 6449448a-f0fc-11ee-b647-cb9ed24678a4 2024-04-02T14:25:32.691 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:32 smithi090 ceph-mon[31674]: last_changed 2024-04-02T14:25:27.128759+0000 2024-04-02T14:25:32.691 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:32 smithi090 ceph-mon[31674]: created 2024-04-02T14:23:12.545436+0000 2024-04-02T14:25:32.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:32 smithi090 ceph-mon[31674]: min_mon_release 19 (squid) 2024-04-02T14:25:32.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:32 smithi090 ceph-mon[31674]: election_strategy: 1 2024-04-02T14:25:32.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:32 smithi090 ceph-mon[31674]: 0: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.a 2024-04-02T14:25:32.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:32 smithi090 ceph-mon[31674]: 1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.c 2024-04-02T14:25:32.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:32 smithi090 ceph-mon[31674]: fsmap 2024-04-02T14:25:32.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:32 smithi090 ceph-mon[31674]: osdmap e4: 0 total, 0 up, 0 in 2024-04-02T14:25:32.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:32 smithi090 ceph-mon[31674]: mgrmap e13: x(active, since 57s) 2024-04-02T14:25:32.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:32 smithi090 ceph-mon[31674]: overall HEALTH_OK 2024-04-02T14:25:32.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:32 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:25:32.692 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:32 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:25:32.693 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:32 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:25:33.520 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:33 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-02T14:25:33.520 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:33 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:25:33.520 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:33 smithi177 ceph-mon[36431]: Deploying daemon mon.b on smithi092 2024-04-02T14:25:33.521 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:33 smithi177 ceph-mon[36431]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T14:25:33.521 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:33 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-02T14:25:33.522 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:33.518+0000 7f9a05322700 1 -- 172.21.15.177:0/1976047450 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f99e80035c0 msgr2=0x7f99e8005a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:25:33.522 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:33.518+0000 7f9a05322700 1 --2- 172.21.15.177:0/1976047450 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f99e80035c0 0x7f99e8005a50 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f9a00100f30 tx=0x7f99f000feb0 comp rx=0 tx=0).stop 2024-04-02T14:25:33.522 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:33.518+0000 7f9a05322700 1 -- 172.21.15.177:0/1976047450 shutdown_connections 2024-04-02T14:25:33.522 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:33.518+0000 7f9a05322700 1 --2- 172.21.15.177:0/1976047450 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f99e80035c0 0x7f99e8005a50 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:25:33.522 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:33.518+0000 7f9a05322700 1 -- 172.21.15.177:0/1976047450 >> 172.21.15.177:0/1976047450 conn(0x7f9a000fcb10 msgr2=0x7f9a000fef30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:25:33.522 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:33.518+0000 7f9a05322700 1 -- 172.21.15.177:0/1976047450 shutdown_connections 2024-04-02T14:25:33.522 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:33.518+0000 7f9a05322700 1 -- 172.21.15.177:0/1976047450 wait complete. 2024-04-02T14:25:33.522 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:33.519+0000 7f9a05322700 1 Processor -- start 2024-04-02T14:25:33.523 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:33.519+0000 7f9a05322700 1 -- start start 2024-04-02T14:25:33.523 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:33.519+0000 7f9a05322700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f99e80035c0 0x7f9a00198b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:25:33.523 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:33.519+0000 7f9a05322700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9a00199040 0x7f9a001994c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:25:33.523 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:33.519+0000 7f9a05322700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f9a00109090 con 0x7f9a00199040 2024-04-02T14:25:33.523 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:33.519+0000 7f9a05322700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f9a00108f10 con 0x7f99e80035c0 2024-04-02T14:25:33.523 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:33.519+0000 7f99feffd700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f99e80035c0 0x7f9a00198b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:25:33.523 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:33.520+0000 7f99feffd700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f99e80035c0 0x7f9a00198b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.177:57676/0 (socket says 172.21.15.177:57676) 2024-04-02T14:25:33.523 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:33.520+0000 7f99feffd700 1 -- 172.21.15.177:0/716314386 learned_addr learned my addr 172.21.15.177:0/716314386 (peer_addr_for_me v2:172.21.15.177:0/0) 2024-04-02T14:25:33.523 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:33.520+0000 7f99feffd700 1 -- 172.21.15.177:0/716314386 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f99e80035c0 msgr2=0x7f9a00198b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-02T14:25:33.523 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:33.520+0000 7f99feffd700 1 -- 172.21.15.177:0/716314386 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f99e80035c0 msgr2=0x7f9a00198b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-02T14:25:33.523 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:33.520+0000 7f99feffd700 1 --2- 172.21.15.177:0/716314386 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f99e80035c0 0x7f9a00198b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-02T14:25:33.523 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:33.520+0000 7f99feffd700 1 --2- 172.21.15.177:0/716314386 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f99e80035c0 0x7f9a00198b00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T14:25:33.523 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:33.520+0000 7f99fe7fc700 1 --2- 172.21.15.177:0/716314386 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9a00199040 0x7f9a001994c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:25:33.523 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:33.520+0000 7f99fe7fc700 1 -- 172.21.15.177:0/716314386 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f99e80035c0 msgr2=0x7f9a00198b00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T14:25:33.523 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:33.520+0000 7f99fe7fc700 1 --2- 172.21.15.177:0/716314386 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f99e80035c0 0x7f9a00198b00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:25:33.524 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:33.520+0000 7f99fe7fc700 1 -- 172.21.15.177:0/716314386 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a001a1b70 con 0x7f9a00199040 2024-04-02T14:25:33.525 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:33.521+0000 7f99fe7fc700 1 --2- 172.21.15.177:0/716314386 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9a00199040 0x7f9a001994c0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f99f400b4d0 tx=0x7f99f400b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:25:33.525 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:33.521+0000 7f99e7fff700 1 -- 172.21.15.177:0/716314386 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f99f4011630 con 0x7f9a00199040 2024-04-02T14:25:33.525 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:33.521+0000 7f9a05322700 1 -- 172.21.15.177:0/716314386 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a001a1d40 con 0x7f9a00199040 2024-04-02T14:25:33.525 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:33.521+0000 7f9a05322700 1 -- 172.21.15.177:0/716314386 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a001a22d0 con 0x7f9a00199040 2024-04-02T14:25:33.525 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:33.521+0000 7f99e7fff700 1 -- 172.21.15.177:0/716314386 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f99f4011cb0 con 0x7f9a00199040 2024-04-02T14:25:33.525 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:33.521+0000 7f99e7fff700 1 -- 172.21.15.177:0/716314386 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f99f40193b0 con 0x7f9a00199040 2024-04-02T14:25:33.525 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:33.522+0000 7f9a05322700 1 -- 172.21.15.177:0/716314386 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9a00102740 con 0x7f9a00199040 2024-04-02T14:25:33.525 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:33.523+0000 7f99e7fff700 1 -- 172.21.15.177:0/716314386 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f99f4018020 con 0x7f9a00199040 2024-04-02T14:25:33.525 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:33.523+0000 7f99e7fff700 1 --2- 172.21.15.177:0/716314386 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f99ec0393f0 0x7f99ec03b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:25:33.525 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:33.523+0000 7f99e7fff700 1 -- 172.21.15.177:0/716314386 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f99f404c330 con 0x7f9a00199040 2024-04-02T14:25:33.525 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:33.523+0000 7f99feffd700 1 --2- 172.21.15.177:0/716314386 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f99ec0393f0 0x7f99ec03b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:25:33.526 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:33.524+0000 7f99feffd700 1 --2- 172.21.15.177:0/716314386 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f99ec0393f0 0x7f99ec03b8b0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f99f000fa60 tx=0x7f99f0005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:25:33.539 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:33.537+0000 7f99e7fff700 1 -- 172.21.15.177:0/716314386 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7f99f401f020 con 0x7f9a00199040 2024-04-02T14:25:33.689 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:33 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-02T14:25:33.689 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:33 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:25:33.689 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:33 smithi090 ceph-mon[31674]: Deploying daemon mon.b on smithi092 2024-04-02T14:25:33.689 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:33 smithi090 ceph-mon[31674]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T14:25:33.689 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:33 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-02T14:25:33.822 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:33.820+0000 7f9a05322700 1 -- 172.21.15.177:0/716314386 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f9a0002dd30 con 0x7f9a00199040 2024-04-02T14:25:33.823 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:33.821+0000 7f99e7fff700 1 -- 172.21.15.177:0/716314386 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) ==== 95+0+1015 (secure 0 0 0) 0x7f99f401f200 con 0x7f9a00199040 2024-04-02T14:25:33.823 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-02T14:25:33.823 INFO:teuthology.orchestra.run.smithi177.stdout:{"epoch":2,"fsid":"6449448a-f0fc-11ee-b647-cb9ed24678a4","modified":"2024-04-02T14:25:27.128759Z","created":"2024-04-02T14:23:12.545436Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:3300","nonce":0},{"type":"v1","addr":"172.21.15.177:6789","nonce":0}]},"addr":"172.21.15.177:6789/0","public_addr":"172.21.15.177:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-04-02T14:25:33.825 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:33.823+0000 7f9a05322700 1 -- 172.21.15.177:0/716314386 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f99ec0393f0 msgr2=0x7f99ec03b8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:25:33.825 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:33.823+0000 7f9a05322700 1 --2- 172.21.15.177:0/716314386 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f99ec0393f0 0x7f99ec03b8b0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f99f000fa60 tx=0x7f99f0005960 comp rx=0 tx=0).stop 2024-04-02T14:25:33.825 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:33.823+0000 7f9a05322700 1 -- 172.21.15.177:0/716314386 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9a00199040 msgr2=0x7f9a001994c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:25:33.826 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:33.823+0000 7f9a05322700 1 --2- 172.21.15.177:0/716314386 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9a00199040 0x7f9a001994c0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f99f400b4d0 tx=0x7f99f400b890 comp rx=0 tx=0).stop 2024-04-02T14:25:33.826 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:33.823+0000 7f9a05322700 1 -- 172.21.15.177:0/716314386 shutdown_connections 2024-04-02T14:25:33.826 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:33.823+0000 7f9a05322700 1 --2- 172.21.15.177:0/716314386 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f99ec0393f0 0x7f99ec03b8b0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:25:33.826 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:33.823+0000 7f9a05322700 1 --2- 172.21.15.177:0/716314386 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f99e80035c0 0x7f9a00198b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:25:33.826 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:33.823+0000 7f9a05322700 1 --2- 172.21.15.177:0/716314386 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9a00199040 0x7f9a001994c0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:25:33.826 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:33.823+0000 7f9a05322700 1 -- 172.21.15.177:0/716314386 >> 172.21.15.177:0/716314386 conn(0x7f9a000fcb10 msgr2=0x7f9a000fffe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:25:33.826 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:33.824+0000 7f9a05322700 1 -- 172.21.15.177:0/716314386 shutdown_connections 2024-04-02T14:25:33.826 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:33.824+0000 7f9a05322700 1 -- 172.21.15.177:0/716314386 wait complete. 2024-04-02T14:25:33.827 INFO:teuthology.orchestra.run.smithi177.stderr:dumped monmap epoch 2 2024-04-02T14:25:34.436 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:34 smithi177 ceph-mon[36431]: from='client.? 172.21.15.177:0/716314386' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-02T14:25:34.590 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:34 smithi090 ceph-mon[31674]: from='client.? 172.21.15.177:0/716314386' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-02T14:25:35.330 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-02T14:25:35.331 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:191d963ac9de7322da8aff20b567b3fa458e2ef6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6449448a-f0fc-11ee-b647-cb9ed24678a4 -- ceph mon dump -f json 2024-04-02T14:25:35.575 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/mon.c/config 2024-04-02T14:25:35.612 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:35 smithi090 ceph-mon[31674]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T14:25:35.686 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:35 smithi177 ceph-mon[36431]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T14:25:36.611 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:25:36 smithi092 systemd[1]: Started Ceph mon.b for 6449448a-f0fc-11ee-b647-cb9ed24678a4. 2024-04-02T14:25:41.418 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:41.415+0000 7f904cf6d700 1 -- 172.21.15.177:0/2050376477 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f90300057d0 msgr2=0x7f9030007c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:25:41.418 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:41.415+0000 7f904cf6d700 1 --2- 172.21.15.177:0/2050376477 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f90300057d0 0x7f9030007c60 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f9048100f20 tx=0x7f9040009e80 comp rx=0 tx=0).stop 2024-04-02T14:25:41.418 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:41.416+0000 7f904cf6d700 1 -- 172.21.15.177:0/2050376477 shutdown_connections 2024-04-02T14:25:41.418 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:41.416+0000 7f904cf6d700 1 --2- 172.21.15.177:0/2050376477 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f90300057d0 0x7f9030007c60 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:25:41.418 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:41.416+0000 7f904cf6d700 1 -- 172.21.15.177:0/2050376477 >> 172.21.15.177:0/2050376477 conn(0x7f90480fcb00 msgr2=0x7f90480fef20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:25:41.418 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:41.416+0000 7f904cf6d700 1 -- 172.21.15.177:0/2050376477 shutdown_connections 2024-04-02T14:25:41.418 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:41.416+0000 7f904cf6d700 1 -- 172.21.15.177:0/2050376477 wait complete. 2024-04-02T14:25:41.420 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:41.417+0000 7f904cf6d700 1 Processor -- start 2024-04-02T14:25:41.421 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:41.417+0000 7f904cf6d700 1 -- start start 2024-04-02T14:25:41.421 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:41.417+0000 7f904cf6d700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f90300057d0 0x7f904819b800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:25:41.421 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:41.417+0000 7f904cf6d700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f904819bd40 0x7f90481959c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:25:41.421 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:41.417+0000 7f904cf6d700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9048195f00 0x7f90481963b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:25:41.421 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:41.417+0000 7f904cf6d700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f9048108050 con 0x7f90300057d0 2024-04-02T14:25:41.421 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:41.417+0000 7f904cf6d700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f9048107ed0 con 0x7f9048195f00 2024-04-02T14:25:41.421 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:41.417+0000 7f904cf6d700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f90481081d0 con 0x7f904819bd40 2024-04-02T14:25:41.421 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:41.418+0000 7f9045d9b700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f904819bd40 0x7f90481959c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:25:41.421 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:41.418+0000 7f9045d9b700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f904819bd40 0x7f90481959c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.177:57680/0 (socket says 172.21.15.177:57680) 2024-04-02T14:25:41.421 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:41.418+0000 7f9045d9b700 1 -- 172.21.15.177:0/235327055 learned_addr learned my addr 172.21.15.177:0/235327055 (peer_addr_for_me v2:172.21.15.177:0/0) 2024-04-02T14:25:41.421 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:41.418+0000 7f9045d9b700 1 -- 172.21.15.177:0/235327055 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f904819bd40 msgr2=0x7f90481959c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-04-02T14:25:41.421 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:41.418+0000 7f9045d9b700 1 -- 172.21.15.177:0/235327055 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f904819bd40 msgr2=0x7f90481959c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-02T14:25:41.422 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:41.418+0000 7f9045d9b700 1 --2- 172.21.15.177:0/235327055 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f904819bd40 0x7f90481959c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-02T14:25:41.422 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:41.418+0000 7f9045d9b700 1 --2- 172.21.15.177:0/235327055 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f904819bd40 0x7f90481959c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T14:25:41.422 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:41.418+0000 7f9046d9d700 1 --2- 172.21.15.177:0/235327055 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9048195f00 0x7f90481963b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:25:41.422 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:41.418+0000 7f9046d9d700 1 -- 172.21.15.177:0/235327055 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9048195f00 msgr2=0x7f90481963b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-04-02T14:25:41.422 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:41.418+0000 7f9046d9d700 1 -- 172.21.15.177:0/235327055 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9048195f00 msgr2=0x7f90481963b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-02T14:25:41.422 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:41.418+0000 7f9046d9d700 1 --2- 172.21.15.177:0/235327055 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9048195f00 0x7f90481963b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-02T14:25:41.422 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:41.418+0000 7f9046d9d700 1 --2- 172.21.15.177:0/235327055 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9048195f00 0x7f90481963b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T14:25:41.422 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:41.418+0000 7f904659c700 1 --2- 172.21.15.177:0/235327055 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f90300057d0 0x7f904819b800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:25:41.422 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:41.419+0000 7f904659c700 1 -- 172.21.15.177:0/235327055 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f904819bd40 msgr2=0x7f90481959c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T14:25:41.422 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:41.419+0000 7f904659c700 1 --2- 172.21.15.177:0/235327055 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f904819bd40 0x7f90481959c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:25:41.423 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:41.419+0000 7f904659c700 1 -- 172.21.15.177:0/235327055 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9048195f00 msgr2=0x7f90481963b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T14:25:41.423 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:41.419+0000 7f904659c700 1 --2- 172.21.15.177:0/235327055 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9048195f00 0x7f90481963b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:25:41.423 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:41.419+0000 7f904659c700 1 -- 172.21.15.177:0/235327055 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90481a26d0 con 0x7f90300057d0 2024-04-02T14:25:41.424 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:41.419+0000 7f904659c700 1 --2- 172.21.15.177:0/235327055 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f90300057d0 0x7f904819b800 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f9040002230 tx=0x7f9040004940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:25:41.425 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:41.423+0000 7f90377fe700 1 -- 172.21.15.177:0/235327055 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f904001ab00 con 0x7f90300057d0 2024-04-02T14:25:41.425 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:41.423+0000 7f904cf6d700 1 -- 172.21.15.177:0/235327055 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f90481a28a0 con 0x7f90300057d0 2024-04-02T14:25:41.425 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:41.423+0000 7f904cf6d700 1 -- 172.21.15.177:0/235327055 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f90481a2e30 con 0x7f90300057d0 2024-04-02T14:25:41.425 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:41.423+0000 7f90377fe700 1 -- 172.21.15.177:0/235327055 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f904001aca0 con 0x7f90300057d0 2024-04-02T14:25:41.425 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:41.423+0000 7f90377fe700 1 -- 172.21.15.177:0/235327055 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9040003c60 con 0x7f90300057d0 2024-04-02T14:25:41.426 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:41.424+0000 7f904cf6d700 1 -- 172.21.15.177:0/235327055 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9030002140 con 0x7f90300057d0 2024-04-02T14:25:41.426 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:41.425+0000 7f90377fe700 1 -- 172.21.15.177:0/235327055 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f9040015040 con 0x7f90300057d0 2024-04-02T14:25:41.427 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:41.425+0000 7f90377fe700 1 --2- 172.21.15.177:0/235327055 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f9028039ae0 0x7f902803bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:25:41.427 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:41.425+0000 7f90377fe700 1 -- 172.21.15.177:0/235327055 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f904004ee70 con 0x7f90300057d0 2024-04-02T14:25:41.427 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:41.425+0000 7f9045d9b700 1 --2- 172.21.15.177:0/235327055 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f9028039ae0 0x7f902803bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:25:41.428 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:41.426+0000 7f9045d9b700 1 --2- 172.21.15.177:0/235327055 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f9028039ae0 0x7f902803bfa0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f903c009180 tx=0x7f903c008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:25:41.443 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:41.442+0000 7f90377fe700 1 -- 172.21.15.177:0/235327055 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7f9040033840 con 0x7f90300057d0 2024-04-02T14:25:41.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:41 smithi177 ceph-mon[36431]: mon.c calling monitor election 2024-04-02T14:25:41.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:41 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-02T14:25:41.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:41 smithi177 ceph-mon[36431]: mon.a calling monitor election 2024-04-02T14:25:41.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:41 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-02T14:25:41.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:41 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-02T14:25:41.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:41 smithi177 ceph-mon[36431]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T14:25:41.688 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:41 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-02T14:25:41.688 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:41 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-02T14:25:41.688 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:41 smithi177 ceph-mon[36431]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T14:25:41.688 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:41 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-02T14:25:41.688 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:41 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-02T14:25:41.688 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:41 smithi177 ceph-mon[36431]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T14:25:41.688 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:41 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-02T14:25:41.688 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:41 smithi177 ceph-mon[36431]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-02T14:25:41.688 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:41 smithi177 ceph-mon[36431]: monmap epoch 3 2024-04-02T14:25:41.688 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:41 smithi177 ceph-mon[36431]: fsid 6449448a-f0fc-11ee-b647-cb9ed24678a4 2024-04-02T14:25:41.689 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:41 smithi177 ceph-mon[36431]: last_changed 2024-04-02T14:25:36.384389+0000 2024-04-02T14:25:41.689 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:41 smithi177 ceph-mon[36431]: created 2024-04-02T14:23:12.545436+0000 2024-04-02T14:25:41.689 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:41 smithi177 ceph-mon[36431]: min_mon_release 19 (squid) 2024-04-02T14:25:41.689 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:41 smithi177 ceph-mon[36431]: election_strategy: 1 2024-04-02T14:25:41.689 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:41 smithi177 ceph-mon[36431]: 0: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.a 2024-04-02T14:25:41.689 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:41 smithi177 ceph-mon[36431]: 1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.c 2024-04-02T14:25:41.689 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:41 smithi177 ceph-mon[36431]: 2: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2024-04-02T14:25:41.689 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:41 smithi177 ceph-mon[36431]: fsmap 2024-04-02T14:25:41.689 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:41 smithi177 ceph-mon[36431]: osdmap e4: 0 total, 0 up, 0 in 2024-04-02T14:25:41.689 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:41 smithi177 ceph-mon[36431]: mgrmap e13: x(active, since 66s) 2024-04-02T14:25:41.689 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:41 smithi177 ceph-mon[36431]: overall HEALTH_OK 2024-04-02T14:25:41.689 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:41 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:25:41.689 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:41 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:25:41.737 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:41.735+0000 7f904cf6d700 1 -- 172.21.15.177:0/235327055 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f9030002c30 con 0x7f90300057d0 2024-04-02T14:25:41.738 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:41.737+0000 7f90377fe700 1 -- 172.21.15.177:0/235327055 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) ==== 95+0+1279 (secure 0 0 0) 0x7f9040039090 con 0x7f90300057d0 2024-04-02T14:25:41.738 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-02T14:25:41.739 INFO:teuthology.orchestra.run.smithi177.stdout:{"epoch":3,"fsid":"6449448a-f0fc-11ee-b647-cb9ed24678a4","modified":"2024-04-02T14:25:36.384389Z","created":"2024-04-02T14:23:12.545436Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:3300","nonce":0},{"type":"v1","addr":"172.21.15.177:6789","nonce":0}]},"addr":"172.21.15.177:6789/0","public_addr":"172.21.15.177:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:3300","nonce":0},{"type":"v1","addr":"172.21.15.92:6789","nonce":0}]},"addr":"172.21.15.92:6789/0","public_addr":"172.21.15.92:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-04-02T14:25:41.740 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:41.739+0000 7f904cf6d700 1 -- 172.21.15.177:0/235327055 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f9028039ae0 msgr2=0x7f902803bfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:25:41.741 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:41.739+0000 7f904cf6d700 1 --2- 172.21.15.177:0/235327055 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f9028039ae0 0x7f902803bfa0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f903c009180 tx=0x7f903c008040 comp rx=0 tx=0).stop 2024-04-02T14:25:41.741 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:41.739+0000 7f904cf6d700 1 -- 172.21.15.177:0/235327055 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f90300057d0 msgr2=0x7f904819b800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:25:41.741 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:41.739+0000 7f904cf6d700 1 --2- 172.21.15.177:0/235327055 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f90300057d0 0x7f904819b800 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f9040002230 tx=0x7f9040004940 comp rx=0 tx=0).stop 2024-04-02T14:25:41.741 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:41.739+0000 7f904cf6d700 1 -- 172.21.15.177:0/235327055 shutdown_connections 2024-04-02T14:25:41.741 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:41.739+0000 7f904cf6d700 1 --2- 172.21.15.177:0/235327055 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f9028039ae0 0x7f902803bfa0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:25:41.741 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:41.739+0000 7f904cf6d700 1 --2- 172.21.15.177:0/235327055 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f90300057d0 0x7f904819b800 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:25:41.741 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:41.739+0000 7f904cf6d700 1 --2- 172.21.15.177:0/235327055 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9048195f00 0x7f90481963b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:25:41.741 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:41.739+0000 7f904cf6d700 1 --2- 172.21.15.177:0/235327055 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f904819bd40 0x7f90481959c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:25:41.741 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:41.739+0000 7f904cf6d700 1 -- 172.21.15.177:0/235327055 >> 172.21.15.177:0/235327055 conn(0x7f90480fcb00 msgr2=0x7f90480fdcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:25:41.741 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:41.739+0000 7f904cf6d700 1 -- 172.21.15.177:0/235327055 shutdown_connections 2024-04-02T14:25:41.741 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:41.739+0000 7f904cf6d700 1 -- 172.21.15.177:0/235327055 wait complete. 2024-04-02T14:25:41.742 INFO:teuthology.orchestra.run.smithi177.stderr:dumped monmap epoch 3 2024-04-02T14:25:41.939 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:41 smithi090 ceph-mon[31674]: mon.c calling monitor election 2024-04-02T14:25:41.939 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:41 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-02T14:25:41.939 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:41 smithi090 ceph-mon[31674]: mon.a calling monitor election 2024-04-02T14:25:41.940 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:41 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-02T14:25:41.940 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:41 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-02T14:25:41.940 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:41 smithi090 ceph-mon[31674]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T14:25:41.940 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:41 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-02T14:25:41.940 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:41 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-02T14:25:41.940 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:41 smithi090 ceph-mon[31674]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T14:25:41.940 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:41 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-02T14:25:41.940 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:41 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-02T14:25:41.940 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:41 smithi090 ceph-mon[31674]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T14:25:41.941 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:41 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-02T14:25:41.941 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:41 smithi090 ceph-mon[31674]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-02T14:25:41.941 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:41 smithi090 ceph-mon[31674]: monmap epoch 3 2024-04-02T14:25:41.941 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:41 smithi090 ceph-mon[31674]: fsid 6449448a-f0fc-11ee-b647-cb9ed24678a4 2024-04-02T14:25:41.941 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:41 smithi090 ceph-mon[31674]: last_changed 2024-04-02T14:25:36.384389+0000 2024-04-02T14:25:41.941 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:41 smithi090 ceph-mon[31674]: created 2024-04-02T14:23:12.545436+0000 2024-04-02T14:25:41.941 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:41 smithi090 ceph-mon[31674]: min_mon_release 19 (squid) 2024-04-02T14:25:41.941 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:41 smithi090 ceph-mon[31674]: election_strategy: 1 2024-04-02T14:25:41.941 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:41 smithi090 ceph-mon[31674]: 0: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.a 2024-04-02T14:25:41.941 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:41 smithi090 ceph-mon[31674]: 1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.c 2024-04-02T14:25:41.941 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:41 smithi090 ceph-mon[31674]: 2: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2024-04-02T14:25:41.942 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:41 smithi090 ceph-mon[31674]: fsmap 2024-04-02T14:25:41.942 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:41 smithi090 ceph-mon[31674]: osdmap e4: 0 total, 0 up, 0 in 2024-04-02T14:25:41.942 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:41 smithi090 ceph-mon[31674]: mgrmap e13: x(active, since 66s) 2024-04-02T14:25:41.942 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:41 smithi090 ceph-mon[31674]: overall HEALTH_OK 2024-04-02T14:25:41.942 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:41 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:25:41.942 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:41 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:25:42.378 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-04-02T14:25:42.379 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:191d963ac9de7322da8aff20b567b3fa458e2ef6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6449448a-f0fc-11ee-b647-cb9ed24678a4 -- ceph config generate-minimal-conf 2024-04-02T14:25:42.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:42 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:25:42.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:42 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:25:42.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:42 smithi177 ceph-mon[36431]: from='client.? 172.21.15.177:0/235327055' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-02T14:25:42.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:42 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T14:25:42.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:42 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-02T14:25:42.796 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T14:25:42.939 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:42 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:25:42.939 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:42 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:25:42.939 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:42 smithi090 ceph-mon[31674]: from='client.? 172.21.15.177:0/235327055' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-02T14:25:42.939 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:42 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T14:25:42.939 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:42 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-02T14:25:43.861 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:25:43 smithi092 ceph-mon[36497]: mon.c calling monitor election 2024-04-02T14:25:43.861 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:25:43 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-02T14:25:43.861 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:25:43 smithi092 ceph-mon[36497]: mon.a calling monitor election 2024-04-02T14:25:43.861 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:25:43 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-02T14:25:43.861 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:25:43 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-02T14:25:43.861 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:25:43 smithi092 ceph-mon[36497]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T14:25:43.861 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:25:43 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-02T14:25:43.861 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:25:43 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-02T14:25:43.862 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:25:43 smithi092 ceph-mon[36497]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T14:25:43.862 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:25:43 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-02T14:25:43.862 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:25:43 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-02T14:25:43.862 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:25:43 smithi092 ceph-mon[36497]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T14:25:43.862 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:25:43 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-02T14:25:43.862 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:25:43 smithi092 ceph-mon[36497]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-02T14:25:43.862 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:25:43 smithi092 ceph-mon[36497]: monmap epoch 3 2024-04-02T14:25:43.862 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:25:43 smithi092 ceph-mon[36497]: fsid 6449448a-f0fc-11ee-b647-cb9ed24678a4 2024-04-02T14:25:43.862 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:25:43 smithi092 ceph-mon[36497]: last_changed 2024-04-02T14:25:36.384389+0000 2024-04-02T14:25:43.862 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:25:43 smithi092 ceph-mon[36497]: created 2024-04-02T14:23:12.545436+0000 2024-04-02T14:25:43.862 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:25:43 smithi092 ceph-mon[36497]: min_mon_release 19 (squid) 2024-04-02T14:25:43.862 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:25:43 smithi092 ceph-mon[36497]: election_strategy: 1 2024-04-02T14:25:43.862 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:25:43 smithi092 ceph-mon[36497]: 0: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.a 2024-04-02T14:25:43.862 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:25:43 smithi092 ceph-mon[36497]: 1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.c 2024-04-02T14:25:43.863 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:25:43 smithi092 ceph-mon[36497]: 2: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2024-04-02T14:25:43.863 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:25:43 smithi092 ceph-mon[36497]: fsmap 2024-04-02T14:25:43.863 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:25:43 smithi092 ceph-mon[36497]: osdmap e4: 0 total, 0 up, 0 in 2024-04-02T14:25:43.863 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:25:43 smithi092 ceph-mon[36497]: mgrmap e13: x(active, since 66s) 2024-04-02T14:25:43.863 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:25:43 smithi092 ceph-mon[36497]: overall HEALTH_OK 2024-04-02T14:25:43.863 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:25:43 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:25:43.863 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:25:43 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:25:43.863 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:25:43 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:25:43.863 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:25:43 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:25:43.863 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:25:43 smithi092 ceph-mon[36497]: from='client.? 172.21.15.177:0/235327055' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-02T14:25:43.863 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:25:43 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T14:25:43.863 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:25:43 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-02T14:25:44.248 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:44 smithi090 ceph-mon[31674]: mon.b calling monitor election 2024-04-02T14:25:44.248 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:44 smithi090 ceph-mon[31674]: Updating smithi090:/etc/ceph/ceph.conf 2024-04-02T14:25:44.249 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:44 smithi090 ceph-mon[31674]: Updating smithi092:/etc/ceph/ceph.conf 2024-04-02T14:25:44.249 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:44 smithi090 ceph-mon[31674]: Updating smithi177:/etc/ceph/ceph.conf 2024-04-02T14:25:44.249 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:44 smithi090 ceph-mon[31674]: mon.b calling monitor election 2024-04-02T14:25:44.249 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:44 smithi090 ceph-mon[31674]: mon.c calling monitor election 2024-04-02T14:25:44.249 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:44 smithi090 ceph-mon[31674]: mon.a calling monitor election 2024-04-02T14:25:44.249 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:44 smithi090 ceph-mon[31674]: Updating smithi092:/var/lib/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-04-02T14:25:44.249 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:44 smithi090 ceph-mon[31674]: Updating smithi177:/var/lib/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-04-02T14:25:44.249 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:44 smithi090 ceph-mon[31674]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-02T14:25:44.249 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:44 smithi090 ceph-mon[31674]: monmap epoch 3 2024-04-02T14:25:44.249 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:44 smithi090 ceph-mon[31674]: fsid 6449448a-f0fc-11ee-b647-cb9ed24678a4 2024-04-02T14:25:44.249 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:44 smithi090 ceph-mon[31674]: last_changed 2024-04-02T14:25:36.384389+0000 2024-04-02T14:25:44.249 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:44 smithi090 ceph-mon[31674]: created 2024-04-02T14:23:12.545436+0000 2024-04-02T14:25:44.249 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:44 smithi090 ceph-mon[31674]: min_mon_release 19 (squid) 2024-04-02T14:25:44.249 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:44 smithi090 ceph-mon[31674]: election_strategy: 1 2024-04-02T14:25:44.249 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:44 smithi090 ceph-mon[31674]: 0: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.a 2024-04-02T14:25:44.250 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:44 smithi090 ceph-mon[31674]: 1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.c 2024-04-02T14:25:44.250 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:44 smithi090 ceph-mon[31674]: 2: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2024-04-02T14:25:44.250 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:44 smithi090 ceph-mon[31674]: fsmap 2024-04-02T14:25:44.250 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:44 smithi090 ceph-mon[31674]: osdmap e4: 0 total, 0 up, 0 in 2024-04-02T14:25:44.250 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:44 smithi090 ceph-mon[31674]: mgrmap e13: x(active, since 68s) 2024-04-02T14:25:44.250 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:44 smithi090 ceph-mon[31674]: overall HEALTH_OK 2024-04-02T14:25:44.250 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:44 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:25:44.250 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:44 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:25:44.250 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:44 smithi090 ceph-mon[31674]: Updating smithi090:/var/lib/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-04-02T14:25:44.250 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:44 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:25:44.250 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:44 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:25:44.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:25:44 smithi092 ceph-mon[36497]: mon.b calling monitor election 2024-04-02T14:25:44.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:25:44 smithi092 ceph-mon[36497]: Updating smithi090:/etc/ceph/ceph.conf 2024-04-02T14:25:44.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:25:44 smithi092 ceph-mon[36497]: Updating smithi092:/etc/ceph/ceph.conf 2024-04-02T14:25:44.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:25:44 smithi092 ceph-mon[36497]: Updating smithi177:/etc/ceph/ceph.conf 2024-04-02T14:25:44.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:25:44 smithi092 ceph-mon[36497]: mon.b calling monitor election 2024-04-02T14:25:44.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:25:44 smithi092 ceph-mon[36497]: mon.c calling monitor election 2024-04-02T14:25:44.362 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:25:44 smithi092 ceph-mon[36497]: mon.a calling monitor election 2024-04-02T14:25:44.362 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:25:44 smithi092 ceph-mon[36497]: Updating smithi092:/var/lib/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-04-02T14:25:44.362 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:25:44 smithi092 ceph-mon[36497]: Updating smithi177:/var/lib/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-04-02T14:25:44.362 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:25:44 smithi092 ceph-mon[36497]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-02T14:25:44.362 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:25:44 smithi092 ceph-mon[36497]: monmap epoch 3 2024-04-02T14:25:44.362 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:25:44 smithi092 ceph-mon[36497]: fsid 6449448a-f0fc-11ee-b647-cb9ed24678a4 2024-04-02T14:25:44.362 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:25:44 smithi092 ceph-mon[36497]: last_changed 2024-04-02T14:25:36.384389+0000 2024-04-02T14:25:44.362 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:25:44 smithi092 ceph-mon[36497]: created 2024-04-02T14:23:12.545436+0000 2024-04-02T14:25:44.362 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:25:44 smithi092 ceph-mon[36497]: min_mon_release 19 (squid) 2024-04-02T14:25:44.362 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:25:44 smithi092 ceph-mon[36497]: election_strategy: 1 2024-04-02T14:25:44.362 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:25:44 smithi092 ceph-mon[36497]: 0: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.a 2024-04-02T14:25:44.362 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:25:44 smithi092 ceph-mon[36497]: 1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.c 2024-04-02T14:25:44.362 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:25:44 smithi092 ceph-mon[36497]: 2: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2024-04-02T14:25:44.362 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:25:44 smithi092 ceph-mon[36497]: fsmap 2024-04-02T14:25:44.363 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:25:44 smithi092 ceph-mon[36497]: osdmap e4: 0 total, 0 up, 0 in 2024-04-02T14:25:44.363 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:25:44 smithi092 ceph-mon[36497]: mgrmap e13: x(active, since 68s) 2024-04-02T14:25:44.363 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:25:44 smithi092 ceph-mon[36497]: overall HEALTH_OK 2024-04-02T14:25:44.363 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:25:44 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:25:44.363 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:25:44 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:25:44.363 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:25:44 smithi092 ceph-mon[36497]: Updating smithi090:/var/lib/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-04-02T14:25:44.363 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:25:44 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:25:44.363 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:25:44 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:25:44.437 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:44 smithi177 ceph-mon[36431]: mon.b calling monitor election 2024-04-02T14:25:44.437 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:44 smithi177 ceph-mon[36431]: Updating smithi090:/etc/ceph/ceph.conf 2024-04-02T14:25:44.437 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:44 smithi177 ceph-mon[36431]: Updating smithi092:/etc/ceph/ceph.conf 2024-04-02T14:25:44.437 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:44 smithi177 ceph-mon[36431]: Updating smithi177:/etc/ceph/ceph.conf 2024-04-02T14:25:44.437 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:44 smithi177 ceph-mon[36431]: mon.b calling monitor election 2024-04-02T14:25:44.437 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:44 smithi177 ceph-mon[36431]: mon.c calling monitor election 2024-04-02T14:25:44.438 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:44 smithi177 ceph-mon[36431]: mon.a calling monitor election 2024-04-02T14:25:44.438 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:44 smithi177 ceph-mon[36431]: Updating smithi092:/var/lib/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-04-02T14:25:44.438 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:44 smithi177 ceph-mon[36431]: Updating smithi177:/var/lib/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-04-02T14:25:44.438 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:44 smithi177 ceph-mon[36431]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-02T14:25:44.438 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:44 smithi177 ceph-mon[36431]: monmap epoch 3 2024-04-02T14:25:44.438 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:44 smithi177 ceph-mon[36431]: fsid 6449448a-f0fc-11ee-b647-cb9ed24678a4 2024-04-02T14:25:44.438 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:44 smithi177 ceph-mon[36431]: last_changed 2024-04-02T14:25:36.384389+0000 2024-04-02T14:25:44.438 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:44 smithi177 ceph-mon[36431]: created 2024-04-02T14:23:12.545436+0000 2024-04-02T14:25:44.438 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:44 smithi177 ceph-mon[36431]: min_mon_release 19 (squid) 2024-04-02T14:25:44.438 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:44 smithi177 ceph-mon[36431]: election_strategy: 1 2024-04-02T14:25:44.439 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:44 smithi177 ceph-mon[36431]: 0: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.a 2024-04-02T14:25:44.439 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:44 smithi177 ceph-mon[36431]: 1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.c 2024-04-02T14:25:44.439 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:44 smithi177 ceph-mon[36431]: 2: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2024-04-02T14:25:44.439 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:44 smithi177 ceph-mon[36431]: fsmap 2024-04-02T14:25:44.439 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:44 smithi177 ceph-mon[36431]: osdmap e4: 0 total, 0 up, 0 in 2024-04-02T14:25:44.439 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:44 smithi177 ceph-mon[36431]: mgrmap e13: x(active, since 68s) 2024-04-02T14:25:44.439 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:44 smithi177 ceph-mon[36431]: overall HEALTH_OK 2024-04-02T14:25:44.439 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:44 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:25:44.439 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:44 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:25:44.439 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:44 smithi177 ceph-mon[36431]: Updating smithi090:/var/lib/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-04-02T14:25:44.439 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:44 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:25:44.439 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:44 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:25:44.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:44.699+0000 7fa774800700 1 -- 172.21.15.90:0/1912132662 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa76c074910 msgr2=0x7fa76c074cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:25:44.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:44.699+0000 7fa774800700 1 --2- 172.21.15.90:0/1912132662 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa76c074910 0x7fa76c074cf0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fa760009910 tx=0x7fa760009c20 comp rx=0 tx=0).stop 2024-04-02T14:25:44.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:44.699+0000 7fa774800700 1 -- 172.21.15.90:0/1912132662 shutdown_connections 2024-04-02T14:25:44.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:44.699+0000 7fa774800700 1 --2- 172.21.15.90:0/1912132662 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa76c074910 0x7fa76c074cf0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:25:44.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:44.699+0000 7fa774800700 1 -- 172.21.15.90:0/1912132662 >> 172.21.15.90:0/1912132662 conn(0x7fa76c0fdad0 msgr2=0x7fa76c0ffef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:25:44.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:44.699+0000 7fa774800700 1 -- 172.21.15.90:0/1912132662 shutdown_connections 2024-04-02T14:25:44.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:44.699+0000 7fa774800700 1 -- 172.21.15.90:0/1912132662 wait complete. 2024-04-02T14:25:44.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:44.700+0000 7fa774800700 1 Processor -- start 2024-04-02T14:25:44.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:44.700+0000 7fa774800700 1 -- start start 2024-04-02T14:25:44.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:44.701+0000 7fa774800700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa76c074910 0x7fa76c199fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:25:44.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:44.701+0000 7fa774800700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa76c19a4e0 0x7fa76c19e870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:25:44.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:44.701+0000 7fa774800700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa76c19edb0 0x7fa76c19f260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:25:44.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:44.701+0000 7fa774800700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fa76c10e850 con 0x7fa76c19a4e0 2024-04-02T14:25:44.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:44.701+0000 7fa774800700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fa76c10e6d0 con 0x7fa76c074910 2024-04-02T14:25:44.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:44.701+0000 7fa774800700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fa76c10e9d0 con 0x7fa76c19edb0 2024-04-02T14:25:44.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:44.701+0000 7fa771400700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa76c19a4e0 0x7fa76c19e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:25:44.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:44.701+0000 7fa771400700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa76c19a4e0 0x7fa76c19e870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:37692/0 (socket says 172.21.15.90:37692) 2024-04-02T14:25:44.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:44.701+0000 7fa771400700 1 -- 172.21.15.90:0/703705145 learned_addr learned my addr 172.21.15.90:0/703705145 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:25:44.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:44.701+0000 7fa771e00700 1 --2- 172.21.15.90:0/703705145 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa76c074910 0x7fa76c199fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:25:44.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:44.701+0000 7fa772800700 1 --2- 172.21.15.90:0/703705145 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa76c19edb0 0x7fa76c19f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:25:44.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:44.701+0000 7fa771400700 1 -- 172.21.15.90:0/703705145 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa76c19edb0 msgr2=0x7fa76c19f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:25:44.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:44.701+0000 7fa771400700 1 --2- 172.21.15.90:0/703705145 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa76c19edb0 0x7fa76c19f260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:25:44.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:44.702+0000 7fa771400700 1 -- 172.21.15.90:0/703705145 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa76c074910 msgr2=0x7fa76c199fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:25:44.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:44.702+0000 7fa771400700 1 --2- 172.21.15.90:0/703705145 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa76c074910 0x7fa76c199fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:25:44.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:44.702+0000 7fa771400700 1 -- 172.21.15.90:0/703705145 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa76c1a6b20 con 0x7fa76c19a4e0 2024-04-02T14:25:44.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:44.702+0000 7fa771400700 1 --2- 172.21.15.90:0/703705145 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa76c19a4e0 0x7fa76c19e870 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fa75c00b780 tx=0x7fa75c00ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:25:44.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:44.702+0000 7fa766a00700 1 -- 172.21.15.90:0/703705145 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa75c004230 con 0x7fa76c19a4e0 2024-04-02T14:25:44.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:44.702+0000 7fa766a00700 1 -- 172.21.15.90:0/703705145 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa75c0043d0 con 0x7fa76c19a4e0 2024-04-02T14:25:44.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:44.702+0000 7fa766a00700 1 -- 172.21.15.90:0/703705145 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa75c011c90 con 0x7fa76c19a4e0 2024-04-02T14:25:44.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:44.702+0000 7fa774800700 1 -- 172.21.15.90:0/703705145 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa76c1a6d50 con 0x7fa76c19a4e0 2024-04-02T14:25:44.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:44.702+0000 7fa774800700 1 -- 172.21.15.90:0/703705145 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa76c1a7250 con 0x7fa76c19a4e0 2024-04-02T14:25:44.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:44.704+0000 7fa766a00700 1 -- 172.21.15.90:0/703705145 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7fa75c0036a0 con 0x7fa76c19a4e0 2024-04-02T14:25:44.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:44.704+0000 7fa774800700 1 -- 172.21.15.90:0/703705145 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa76c063c20 con 0x7fa76c19a4e0 2024-04-02T14:25:44.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:44.704+0000 7fa766a00700 1 --2- 172.21.15.90:0/703705145 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fa754039ae0 0x7fa75403bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:25:44.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:44.704+0000 7fa766a00700 1 -- 172.21.15.90:0/703705145 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fa75c04c330 con 0x7fa76c19a4e0 2024-04-02T14:25:44.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:44.705+0000 7fa771e00700 1 --2- 172.21.15.90:0/703705145 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fa754039ae0 0x7fa75403bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:25:44.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:44.705+0000 7fa771e00700 1 --2- 172.21.15.90:0/703705145 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fa754039ae0 0x7fa75403bfa0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fa760006010 tx=0x7fa7600058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:25:44.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:44.710+0000 7fa766a00700 1 -- 172.21.15.90:0/703705145 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7fa75c018e00 con 0x7fa76c19a4e0 2024-04-02T14:25:44.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:44.937+0000 7fa774800700 1 -- 172.21.15.90:0/703705145 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7fa76c1a7680 con 0x7fa76c19a4e0 2024-04-02T14:25:44.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:44.938+0000 7fa766a00700 1 -- 172.21.15.90:0/703705145 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) ==== 76+0+273 (secure 0 0 0) 0x7fa75c003d50 con 0x7fa76c19a4e0 2024-04-02T14:25:44.939 INFO:teuthology.orchestra.run.smithi090.stdout:# minimal ceph.conf for 6449448a-f0fc-11ee-b647-cb9ed24678a4 2024-04-02T14:25:44.939 INFO:teuthology.orchestra.run.smithi090.stdout:[global] 2024-04-02T14:25:44.939 INFO:teuthology.orchestra.run.smithi090.stdout: fsid = 6449448a-f0fc-11ee-b647-cb9ed24678a4 2024-04-02T14:25:44.939 INFO:teuthology.orchestra.run.smithi090.stdout: mon_host = [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-04-02T14:25:44.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:44.941+0000 7fa774800700 1 -- 172.21.15.90:0/703705145 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fa754039ae0 msgr2=0x7fa75403bfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:25:44.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:44.941+0000 7fa774800700 1 --2- 172.21.15.90:0/703705145 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fa754039ae0 0x7fa75403bfa0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fa760006010 tx=0x7fa7600058e0 comp rx=0 tx=0).stop 2024-04-02T14:25:44.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:44.941+0000 7fa774800700 1 -- 172.21.15.90:0/703705145 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa76c19a4e0 msgr2=0x7fa76c19e870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:25:44.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:44.941+0000 7fa774800700 1 --2- 172.21.15.90:0/703705145 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa76c19a4e0 0x7fa76c19e870 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fa75c00b780 tx=0x7fa75c00ba90 comp rx=0 tx=0).stop 2024-04-02T14:25:44.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:44.941+0000 7fa774800700 1 -- 172.21.15.90:0/703705145 shutdown_connections 2024-04-02T14:25:44.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:44.941+0000 7fa774800700 1 --2- 172.21.15.90:0/703705145 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fa754039ae0 0x7fa75403bfa0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:25:44.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:44.941+0000 7fa774800700 1 --2- 172.21.15.90:0/703705145 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa76c19edb0 0x7fa76c19f260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:25:44.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:44.941+0000 7fa774800700 1 --2- 172.21.15.90:0/703705145 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa76c074910 0x7fa76c199fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:25:44.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:44.941+0000 7fa774800700 1 --2- 172.21.15.90:0/703705145 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa76c19a4e0 0x7fa76c19e870 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:25:44.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:44.941+0000 7fa774800700 1 -- 172.21.15.90:0/703705145 >> 172.21.15.90:0/703705145 conn(0x7fa76c0fdad0 msgr2=0x7fa76c106420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:25:44.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:44.941+0000 7fa774800700 1 -- 172.21.15.90:0/703705145 shutdown_connections 2024-04-02T14:25:44.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:44.941+0000 7fa774800700 1 -- 172.21.15.90:0/703705145 wait complete. 2024-04-02T14:25:45.582 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:45 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:25:45.582 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:45 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:25:45.583 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:45 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:25:45.583 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:45 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-02T14:25:45.583 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:45 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:25:45.583 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:45 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:25:45.583 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:45 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:25:45.583 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:45 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:25:45.583 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:45 smithi090 ceph-mon[31674]: Reconfiguring mon.a (unknown last config time)... 2024-04-02T14:25:45.583 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:45 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-02T14:25:45.583 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:45 smithi090 ceph-mon[31674]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T14:25:45.583 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:45 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-02T14:25:45.583 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:45 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:25:45.583 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:45 smithi090 ceph-mon[31674]: Reconfiguring daemon mon.a on smithi090 2024-04-02T14:25:45.583 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:45 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:25:45.583 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:45 smithi090 ceph-mon[31674]: from='client.? 172.21.15.90:0/703705145' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:25:45.611 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:25:45 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:25:45.611 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:25:45 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:25:45.611 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:25:45 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:25:45.611 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:25:45 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-02T14:25:45.611 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:25:45 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:25:45.611 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:25:45 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:25:45.611 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:25:45 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:25:45.611 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:25:45 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:25:45.611 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:25:45 smithi092 ceph-mon[36497]: Reconfiguring mon.a (unknown last config time)... 2024-04-02T14:25:45.611 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:25:45 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-02T14:25:45.611 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:25:45 smithi092 ceph-mon[36497]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T14:25:45.611 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:25:45 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-02T14:25:45.612 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:25:45 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:25:45.612 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:25:45 smithi092 ceph-mon[36497]: Reconfiguring daemon mon.a on smithi090 2024-04-02T14:25:45.612 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:25:45 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:25:45.612 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:25:45 smithi092 ceph-mon[36497]: from='client.? 172.21.15.90:0/703705145' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:25:45.639 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-04-02T14:25:45.639 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-04-02T14:25:45.639 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd of=/etc/ceph/ceph.conf 2024-04-02T14:25:45.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:45 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:25:45.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:45 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:25:45.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:45 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:25:45.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:45 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-02T14:25:45.688 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:45 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:25:45.688 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:45 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:25:45.688 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:45 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:25:45.688 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:45 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:25:45.688 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:45 smithi177 ceph-mon[36431]: Reconfiguring mon.a (unknown last config time)... 2024-04-02T14:25:45.688 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:45 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-02T14:25:45.688 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:45 smithi177 ceph-mon[36431]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T14:25:45.688 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:45 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-02T14:25:45.688 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:45 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:25:45.688 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:45 smithi177 ceph-mon[36431]: Reconfiguring daemon mon.a on smithi090 2024-04-02T14:25:45.688 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:45 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:25:45.689 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:45 smithi177 ceph-mon[36431]: from='client.? 172.21.15.90:0/703705145' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:25:45.733 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-04-02T14:25:45.734 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-02T14:25:45.822 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2024-04-02T14:25:45.822 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd of=/etc/ceph/ceph.conf 2024-04-02T14:25:45.858 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2024-04-02T14:25:45.858 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-02T14:25:45.929 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-04-02T14:25:45.929 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd of=/etc/ceph/ceph.conf 2024-04-02T14:25:45.966 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-04-02T14:25:45.966 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-02T14:25:46.032 INFO:tasks.cephadm:Adding mgr.x on smithi090 2024-04-02T14:25:46.032 INFO:tasks.cephadm:Adding mgr.y on smithi092 2024-04-02T14:25:46.032 INFO:tasks.cephadm:Adding mgr.z on smithi177 2024-04-02T14:25:46.033 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:191d963ac9de7322da8aff20b567b3fa458e2ef6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6449448a-f0fc-11ee-b647-cb9ed24678a4 -- ceph orch apply mgr '3;smithi090=x;smithi092=y;smithi177=z' 2024-04-02T14:25:46.307 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/mon.c/config 2024-04-02T14:25:47.613 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:47.608+0000 7fb14bbdd700 1 -- 172.21.15.177:0/2386620694 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb144101010 msgr2=0x7fb144101430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:25:47.613 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:47.608+0000 7fb14bbdd700 1 --2- 172.21.15.177:0/2386620694 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb144101010 0x7fb144101430 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fb140009910 tx=0x7fb140009c20 comp rx=0 tx=0).stop 2024-04-02T14:25:47.613 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:47.608+0000 7fb14bbdd700 1 -- 172.21.15.177:0/2386620694 shutdown_connections 2024-04-02T14:25:47.613 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:47.608+0000 7fb14bbdd700 1 --2- 172.21.15.177:0/2386620694 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb144101010 0x7fb144101430 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:25:47.614 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:47.608+0000 7fb14bbdd700 1 -- 172.21.15.177:0/2386620694 >> 172.21.15.177:0/2386620694 conn(0x7fb1440fc5d0 msgr2=0x7fb1440fe9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:25:47.614 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:47.609+0000 7fb14bbdd700 1 -- 172.21.15.177:0/2386620694 shutdown_connections 2024-04-02T14:25:47.614 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:47.609+0000 7fb14bbdd700 1 -- 172.21.15.177:0/2386620694 wait complete. 2024-04-02T14:25:47.614 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:47.609+0000 7fb14bbdd700 1 Processor -- start 2024-04-02T14:25:47.614 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:47.609+0000 7fb14bbdd700 1 -- start start 2024-04-02T14:25:47.614 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:47.609+0000 7fb14bbdd700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb144101010 0x7fb14406aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:25:47.614 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:47.609+0000 7fb14bbdd700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb14419a6c0 0x7fb14419cab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:25:47.614 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:47.609+0000 7fb14bbdd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb14419d040 0x7fb14419f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:25:47.614 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:47.609+0000 7fb14bbdd700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fb144107050 con 0x7fb14419a6c0 2024-04-02T14:25:47.614 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:47.609+0000 7fb14bbdd700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fb144106ed0 con 0x7fb14419d040 2024-04-02T14:25:47.614 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:47.609+0000 7fb14bbdd700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fb1441071d0 con 0x7fb144101010 2024-04-02T14:25:47.614 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:47.609+0000 7fb149979700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb144101010 0x7fb14406aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:25:47.614 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:47.610+0000 7fb149979700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb144101010 0x7fb14406aaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.177:43442/0 (socket says 172.21.15.177:43442) 2024-04-02T14:25:47.615 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:47.610+0000 7fb149979700 1 -- 172.21.15.177:0/3080249077 learned_addr learned my addr 172.21.15.177:0/3080249077 (peer_addr_for_me v2:172.21.15.177:0/0) 2024-04-02T14:25:47.615 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:47.610+0000 7fb14a17a700 1 --2- 172.21.15.177:0/3080249077 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb14419d040 0x7fb14419f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:25:47.615 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:47.610+0000 7fb149979700 1 -- 172.21.15.177:0/3080249077 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb14419d040 msgr2=0x7fb14419f4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:25:47.615 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:47.610+0000 7fb149979700 1 --2- 172.21.15.177:0/3080249077 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb14419d040 0x7fb14419f4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:25:47.615 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:47.610+0000 7fb149979700 1 -- 172.21.15.177:0/3080249077 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb14419a6c0 msgr2=0x7fb14419cab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:25:47.615 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:47.610+0000 7fb149979700 1 --2- 172.21.15.177:0/3080249077 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb14419a6c0 0x7fb14419cab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:25:47.615 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:47.610+0000 7fb149979700 1 -- 172.21.15.177:0/3080249077 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb14419fda0 con 0x7fb144101010 2024-04-02T14:25:47.615 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:47.610+0000 7fb149979700 1 --2- 172.21.15.177:0/3080249077 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb144101010 0x7fb14406aaa0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fb140000c00 tx=0x7fb1400050b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:25:47.615 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:47.610+0000 7fb13affd700 1 -- 172.21.15.177:0/3080249077 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb140027070 con 0x7fb144101010 2024-04-02T14:25:47.615 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:47.610+0000 7fb14bbdd700 1 -- 172.21.15.177:0/3080249077 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1441a0030 con 0x7fb144101010 2024-04-02T14:25:47.615 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:47.610+0000 7fb13affd700 1 -- 172.21.15.177:0/3080249077 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb140022470 con 0x7fb144101010 2024-04-02T14:25:47.615 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:47.610+0000 7fb13affd700 1 -- 172.21.15.177:0/3080249077 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb14000f6c0 con 0x7fb144101010 2024-04-02T14:25:47.615 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:47.610+0000 7fb14bbdd700 1 -- 172.21.15.177:0/3080249077 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1441a05e0 con 0x7fb144101010 2024-04-02T14:25:47.616 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:47.611+0000 7fb14bbdd700 1 -- 172.21.15.177:0/3080249077 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb144063c20 con 0x7fb144101010 2024-04-02T14:25:47.616 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:47.611+0000 7fb13affd700 1 -- 172.21.15.177:0/3080249077 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7fb14002a070 con 0x7fb144101010 2024-04-02T14:25:47.616 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:47.611+0000 7fb13affd700 1 --2- 172.21.15.177:0/3080249077 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fb130039ae0 0x7fb13003bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:25:47.616 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:47.612+0000 7fb13affd700 1 -- 172.21.15.177:0/3080249077 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fb14004da50 con 0x7fb144101010 2024-04-02T14:25:47.617 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:47.612+0000 7fb149178700 1 --2- 172.21.15.177:0/3080249077 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fb130039ae0 0x7fb13003bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:25:47.617 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:47.613+0000 7fb149178700 1 --2- 172.21.15.177:0/3080249077 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fb130039ae0 0x7fb13003bfa0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fb14406b020 tx=0x7fb134006dd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:25:47.623 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:47.621+0000 7fb13affd700 1 -- 172.21.15.177:0/3080249077 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7fb140013070 con 0x7fb144101010 2024-04-02T14:25:47.836 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:47.833+0000 7fb14bbdd700 1 -- 172.21.15.177:0/3080249077 --> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi090=x;smithi092=y;smithi177=z", "target": ["mon-mgr", ""]}) -- 0x7fb144062ac0 con 0x7fb130039ae0 2024-04-02T14:25:47.849 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:47.847+0000 7fb13affd700 1 -- 172.21.15.177:0/3080249077 <== mgr.14152 v2:172.21.15.90:6800/4266054817 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7fb144062ac0 con 0x7fb130039ae0 2024-04-02T14:25:47.849 INFO:teuthology.orchestra.run.smithi177.stdout:Scheduled mgr update... 2024-04-02T14:25:47.851 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:47.849+0000 7fb14bbdd700 1 -- 172.21.15.177:0/3080249077 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fb130039ae0 msgr2=0x7fb13003bfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:25:47.852 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:47.849+0000 7fb14bbdd700 1 --2- 172.21.15.177:0/3080249077 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fb130039ae0 0x7fb13003bfa0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fb14406b020 tx=0x7fb134006dd0 comp rx=0 tx=0).stop 2024-04-02T14:25:47.852 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:47.849+0000 7fb14bbdd700 1 -- 172.21.15.177:0/3080249077 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb144101010 msgr2=0x7fb14406aaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:25:47.852 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:47.849+0000 7fb14bbdd700 1 --2- 172.21.15.177:0/3080249077 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb144101010 0x7fb14406aaa0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fb140000c00 tx=0x7fb1400050b0 comp rx=0 tx=0).stop 2024-04-02T14:25:47.852 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:47.849+0000 7fb14bbdd700 1 -- 172.21.15.177:0/3080249077 shutdown_connections 2024-04-02T14:25:47.852 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:47.849+0000 7fb14bbdd700 1 --2- 172.21.15.177:0/3080249077 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fb130039ae0 0x7fb13003bfa0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:25:47.852 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:47.849+0000 7fb14bbdd700 1 --2- 172.21.15.177:0/3080249077 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb14419d040 0x7fb14419f4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:25:47.852 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:47.849+0000 7fb14bbdd700 1 --2- 172.21.15.177:0/3080249077 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb144101010 0x7fb14406aaa0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:25:47.852 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:47.849+0000 7fb14bbdd700 1 --2- 172.21.15.177:0/3080249077 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb14419a6c0 0x7fb14419cab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:25:47.852 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:47.849+0000 7fb14bbdd700 1 -- 172.21.15.177:0/3080249077 >> 172.21.15.177:0/3080249077 conn(0x7fb1440fc5d0 msgr2=0x7fb1440fd240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:25:47.852 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:47.849+0000 7fb14bbdd700 1 -- 172.21.15.177:0/3080249077 shutdown_connections 2024-04-02T14:25:47.852 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:25:47.849+0000 7fb14bbdd700 1 -- 172.21.15.177:0/3080249077 wait complete. 2024-04-02T14:25:47.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:47 smithi177 ceph-mon[36431]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T14:25:47.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:47 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:25:47.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:47 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:25:47.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:47 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-02T14:25:47.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:47 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-02T14:25:47.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:47 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:25:48.110 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:25:47 smithi092 ceph-mon[36497]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T14:25:48.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:25:47 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:25:48.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:25:47 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:25:48.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:25:47 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-02T14:25:48.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:25:47 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-02T14:25:48.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:25:47 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:25:48.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:47 smithi090 ceph-mon[31674]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T14:25:48.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:47 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:25:48.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:47 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:25:48.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:47 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-02T14:25:48.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:47 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-02T14:25:48.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:47 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:25:48.471 DEBUG:teuthology.orchestra.run.smithi092:mgr.y> sudo journalctl -f -n 0 -u ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4@mgr.y.service 2024-04-02T14:25:48.473 DEBUG:teuthology.orchestra.run.smithi177:mgr.z> sudo journalctl -f -n 0 -u ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4@mgr.z.service 2024-04-02T14:25:48.475 INFO:tasks.cephadm:Deploying OSDs... 2024-04-02T14:25:48.475 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-04-02T14:25:48.476 DEBUG:teuthology.orchestra.run.smithi090:> dd if=/scratch_devs of=/dev/stdout 2024-04-02T14:25:48.532 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-02T14:25:48.532 DEBUG:teuthology.orchestra.run.smithi090:> stat /dev/vg_nvme/lv_1 2024-04-02T14:25:48.582 INFO:teuthology.orchestra.run.smithi090.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-02T14:25:48.583 INFO:teuthology.orchestra.run.smithi090.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-02T14:25:48.583 INFO:teuthology.orchestra.run.smithi090.stdout:Device: 5h/5d Inode: 634 Links: 1 2024-04-02T14:25:48.583 INFO:teuthology.orchestra.run.smithi090.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-02T14:25:48.583 INFO:teuthology.orchestra.run.smithi090.stdout:Context: system_u:object_r:device_t:s0 2024-04-02T14:25:48.583 INFO:teuthology.orchestra.run.smithi090.stdout:Access: 2024-04-02 14:24:50.102025836 +0000 2024-04-02T14:25:48.583 INFO:teuthology.orchestra.run.smithi090.stdout:Modify: 2024-04-02 14:19:25.769051685 +0000 2024-04-02T14:25:48.583 INFO:teuthology.orchestra.run.smithi090.stdout:Change: 2024-04-02 14:19:25.769051685 +0000 2024-04-02T14:25:48.583 INFO:teuthology.orchestra.run.smithi090.stdout: Birth: 2024-04-02 14:19:25.769051685 +0000 2024-04-02T14:25:48.583 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-02T14:25:48.666 INFO:teuthology.orchestra.run.smithi090.stderr:1+0 records in 2024-04-02T14:25:48.666 INFO:teuthology.orchestra.run.smithi090.stderr:1+0 records out 2024-04-02T14:25:48.667 INFO:teuthology.orchestra.run.smithi090.stderr:512 bytes copied, 0.000345088 s, 1.5 MB/s 2024-04-02T14:25:48.670 DEBUG:teuthology.orchestra.run.smithi090:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-02T14:25:48.724 DEBUG:teuthology.orchestra.run.smithi090:> stat /dev/vg_nvme/lv_2 2024-04-02T14:25:48.773 INFO:teuthology.orchestra.run.smithi090.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-02T14:25:48.773 INFO:teuthology.orchestra.run.smithi090.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-02T14:25:48.773 INFO:teuthology.orchestra.run.smithi090.stdout:Device: 5h/5d Inode: 650 Links: 1 2024-04-02T14:25:48.773 INFO:teuthology.orchestra.run.smithi090.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-02T14:25:48.773 INFO:teuthology.orchestra.run.smithi090.stdout:Context: system_u:object_r:device_t:s0 2024-04-02T14:25:48.773 INFO:teuthology.orchestra.run.smithi090.stdout:Access: 2024-04-02 14:24:50.182024581 +0000 2024-04-02T14:25:48.773 INFO:teuthology.orchestra.run.smithi090.stdout:Modify: 2024-04-02 14:19:26.344042525 +0000 2024-04-02T14:25:48.773 INFO:teuthology.orchestra.run.smithi090.stdout:Change: 2024-04-02 14:19:26.344042525 +0000 2024-04-02T14:25:48.773 INFO:teuthology.orchestra.run.smithi090.stdout: Birth: 2024-04-02 14:19:26.344042525 +0000 2024-04-02T14:25:48.773 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-02T14:25:48.850 INFO:teuthology.orchestra.run.smithi090.stderr:1+0 records in 2024-04-02T14:25:48.850 INFO:teuthology.orchestra.run.smithi090.stderr:1+0 records out 2024-04-02T14:25:48.850 INFO:teuthology.orchestra.run.smithi090.stderr:512 bytes copied, 0.000337402 s, 1.5 MB/s 2024-04-02T14:25:48.853 DEBUG:teuthology.orchestra.run.smithi090:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-02T14:25:48.905 DEBUG:teuthology.orchestra.run.smithi090:> stat /dev/vg_nvme/lv_3 2024-04-02T14:25:48.938 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:48 smithi090 ceph-mon[31674]: Reconfiguring mon.b (monmap changed)... 2024-04-02T14:25:48.938 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:48 smithi090 ceph-mon[31674]: Reconfiguring daemon mon.b on smithi092 2024-04-02T14:25:48.939 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:48 smithi090 ceph-mon[31674]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi090=x;smithi092=y;smithi177=z", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T14:25:48.939 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:48 smithi090 ceph-mon[31674]: Saving service mgr spec with placement smithi090=x;smithi092=y;smithi177=z;count:3 2024-04-02T14:25:48.939 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:48 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:25:48.939 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:48 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:25:48.939 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:48 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:25:48.939 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:48 smithi090 ceph-mon[31674]: Reconfiguring mon.c (monmap changed)... 2024-04-02T14:25:48.939 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:48 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-02T14:25:48.939 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:48 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-02T14:25:48.939 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:48 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:25:48.939 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:48 smithi090 ceph-mon[31674]: Reconfiguring daemon mon.c on smithi177 2024-04-02T14:25:48.939 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:48 smithi090 ceph-mon[31674]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T14:25:48.952 INFO:teuthology.orchestra.run.smithi090.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-02T14:25:48.952 INFO:teuthology.orchestra.run.smithi090.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-02T14:25:48.952 INFO:teuthology.orchestra.run.smithi090.stdout:Device: 5h/5d Inode: 672 Links: 1 2024-04-02T14:25:48.952 INFO:teuthology.orchestra.run.smithi090.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-02T14:25:48.952 INFO:teuthology.orchestra.run.smithi090.stdout:Context: system_u:object_r:device_t:s0 2024-04-02T14:25:48.952 INFO:teuthology.orchestra.run.smithi090.stdout:Access: 2024-04-02 14:24:50.259023373 +0000 2024-04-02T14:25:48.952 INFO:teuthology.orchestra.run.smithi090.stdout:Modify: 2024-04-02 14:19:26.896033732 +0000 2024-04-02T14:25:48.952 INFO:teuthology.orchestra.run.smithi090.stdout:Change: 2024-04-02 14:19:26.896033732 +0000 2024-04-02T14:25:48.952 INFO:teuthology.orchestra.run.smithi090.stdout: Birth: 2024-04-02 14:19:26.896033732 +0000 2024-04-02T14:25:48.953 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-02T14:25:49.085 INFO:teuthology.orchestra.run.smithi090.stderr:1+0 records in 2024-04-02T14:25:49.085 INFO:teuthology.orchestra.run.smithi090.stderr:1+0 records out 2024-04-02T14:25:49.085 INFO:teuthology.orchestra.run.smithi090.stderr:512 bytes copied, 0.000333483 s, 1.5 MB/s 2024-04-02T14:25:49.090 DEBUG:teuthology.orchestra.run.smithi090:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-02T14:25:49.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:25:48 smithi092 ceph-mon[36497]: Reconfiguring mon.b (monmap changed)... 2024-04-02T14:25:49.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:25:48 smithi092 ceph-mon[36497]: Reconfiguring daemon mon.b on smithi092 2024-04-02T14:25:49.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:25:48 smithi092 ceph-mon[36497]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi090=x;smithi092=y;smithi177=z", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T14:25:49.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:25:48 smithi092 ceph-mon[36497]: Saving service mgr spec with placement smithi090=x;smithi092=y;smithi177=z;count:3 2024-04-02T14:25:49.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:25:48 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:25:49.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:25:48 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:25:49.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:25:48 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:25:49.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:25:48 smithi092 ceph-mon[36497]: Reconfiguring mon.c (monmap changed)... 2024-04-02T14:25:49.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:25:48 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-02T14:25:49.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:25:48 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-02T14:25:49.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:25:48 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:25:49.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:25:48 smithi092 ceph-mon[36497]: Reconfiguring daemon mon.c on smithi177 2024-04-02T14:25:49.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:25:48 smithi092 ceph-mon[36497]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T14:25:49.126 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:48 smithi177 ceph-mon[36431]: Reconfiguring mon.b (monmap changed)... 2024-04-02T14:25:49.126 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:48 smithi177 ceph-mon[36431]: Reconfiguring daemon mon.b on smithi092 2024-04-02T14:25:49.126 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:48 smithi177 ceph-mon[36431]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi090=x;smithi092=y;smithi177=z", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T14:25:49.126 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:48 smithi177 ceph-mon[36431]: Saving service mgr spec with placement smithi090=x;smithi092=y;smithi177=z;count:3 2024-04-02T14:25:49.126 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:48 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:25:49.126 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:48 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:25:49.126 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:48 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:25:49.126 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:48 smithi177 ceph-mon[36431]: Reconfiguring mon.c (monmap changed)... 2024-04-02T14:25:49.127 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:48 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-02T14:25:49.127 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:48 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-02T14:25:49.127 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:48 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:25:49.127 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:48 smithi177 ceph-mon[36431]: Reconfiguring daemon mon.c on smithi177 2024-04-02T14:25:49.127 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:48 smithi177 ceph-mon[36431]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T14:25:49.145 DEBUG:teuthology.orchestra.run.smithi090:> stat /dev/vg_nvme/lv_4 2024-04-02T14:25:49.193 INFO:teuthology.orchestra.run.smithi090.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-02T14:25:49.194 INFO:teuthology.orchestra.run.smithi090.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-02T14:25:49.194 INFO:teuthology.orchestra.run.smithi090.stdout:Device: 5h/5d Inode: 691 Links: 1 2024-04-02T14:25:49.194 INFO:teuthology.orchestra.run.smithi090.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-02T14:25:49.194 INFO:teuthology.orchestra.run.smithi090.stdout:Context: system_u:object_r:device_t:s0 2024-04-02T14:25:49.194 INFO:teuthology.orchestra.run.smithi090.stdout:Access: 2024-04-02 14:24:50.307022620 +0000 2024-04-02T14:25:49.194 INFO:teuthology.orchestra.run.smithi090.stdout:Modify: 2024-04-02 14:19:27.470024587 +0000 2024-04-02T14:25:49.194 INFO:teuthology.orchestra.run.smithi090.stdout:Change: 2024-04-02 14:19:27.470024587 +0000 2024-04-02T14:25:49.194 INFO:teuthology.orchestra.run.smithi090.stdout: Birth: 2024-04-02 14:19:27.470024587 +0000 2024-04-02T14:25:49.194 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-02T14:25:49.277 INFO:teuthology.orchestra.run.smithi090.stderr:1+0 records in 2024-04-02T14:25:49.277 INFO:teuthology.orchestra.run.smithi090.stderr:1+0 records out 2024-04-02T14:25:49.277 INFO:teuthology.orchestra.run.smithi090.stderr:512 bytes copied, 0.000321186 s, 1.6 MB/s 2024-04-02T14:25:49.281 DEBUG:teuthology.orchestra.run.smithi090:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-02T14:25:49.334 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2024-04-02T14:25:49.334 DEBUG:teuthology.orchestra.run.smithi092:> dd if=/scratch_devs of=/dev/stdout 2024-04-02T14:25:49.352 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-02T14:25:49.352 DEBUG:teuthology.orchestra.run.smithi092:> stat /dev/vg_nvme/lv_1 2024-04-02T14:25:49.409 INFO:teuthology.orchestra.run.smithi092.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-02T14:25:49.409 INFO:teuthology.orchestra.run.smithi092.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-02T14:25:49.409 INFO:teuthology.orchestra.run.smithi092.stdout:Device: 5h/5d Inode: 662 Links: 1 2024-04-02T14:25:49.409 INFO:teuthology.orchestra.run.smithi092.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-02T14:25:49.409 INFO:teuthology.orchestra.run.smithi092.stdout:Context: system_u:object_r:device_t:s0 2024-04-02T14:25:49.409 INFO:teuthology.orchestra.run.smithi092.stdout:Access: 2024-04-02 14:25:35.881375170 +0000 2024-04-02T14:25:49.409 INFO:teuthology.orchestra.run.smithi092.stdout:Modify: 2024-04-02 14:18:24.101717203 +0000 2024-04-02T14:25:49.409 INFO:teuthology.orchestra.run.smithi092.stdout:Change: 2024-04-02 14:18:24.101717203 +0000 2024-04-02T14:25:49.409 INFO:teuthology.orchestra.run.smithi092.stdout: Birth: 2024-04-02 14:18:24.101717203 +0000 2024-04-02T14:25:49.410 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-02T14:25:49.478 INFO:teuthology.orchestra.run.smithi092.stderr:1+0 records in 2024-04-02T14:25:49.478 INFO:teuthology.orchestra.run.smithi092.stderr:1+0 records out 2024-04-02T14:25:49.478 INFO:teuthology.orchestra.run.smithi092.stderr:512 bytes copied, 0.000226573 s, 2.3 MB/s 2024-04-02T14:25:49.479 DEBUG:teuthology.orchestra.run.smithi092:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-02T14:25:49.536 DEBUG:teuthology.orchestra.run.smithi092:> stat /dev/vg_nvme/lv_2 2024-04-02T14:25:49.592 INFO:teuthology.orchestra.run.smithi092.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-02T14:25:49.592 INFO:teuthology.orchestra.run.smithi092.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-02T14:25:49.592 INFO:teuthology.orchestra.run.smithi092.stdout:Device: 5h/5d Inode: 681 Links: 1 2024-04-02T14:25:49.592 INFO:teuthology.orchestra.run.smithi092.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-02T14:25:49.592 INFO:teuthology.orchestra.run.smithi092.stdout:Context: system_u:object_r:device_t:s0 2024-04-02T14:25:49.593 INFO:teuthology.orchestra.run.smithi092.stdout:Access: 2024-04-02 14:25:35.882375140 +0000 2024-04-02T14:25:49.593 INFO:teuthology.orchestra.run.smithi092.stdout:Modify: 2024-04-02 14:18:24.468706226 +0000 2024-04-02T14:25:49.593 INFO:teuthology.orchestra.run.smithi092.stdout:Change: 2024-04-02 14:18:24.468706226 +0000 2024-04-02T14:25:49.593 INFO:teuthology.orchestra.run.smithi092.stdout: Birth: 2024-04-02 14:18:24.468706226 +0000 2024-04-02T14:25:49.593 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-02T14:25:49.659 INFO:teuthology.orchestra.run.smithi092.stderr:1+0 records in 2024-04-02T14:25:49.660 INFO:teuthology.orchestra.run.smithi092.stderr:1+0 records out 2024-04-02T14:25:49.660 INFO:teuthology.orchestra.run.smithi092.stderr:512 bytes copied, 0.000238697 s, 2.1 MB/s 2024-04-02T14:25:49.661 DEBUG:teuthology.orchestra.run.smithi092:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-02T14:25:49.718 DEBUG:teuthology.orchestra.run.smithi092:> stat /dev/vg_nvme/lv_3 2024-04-02T14:25:49.774 INFO:teuthology.orchestra.run.smithi092.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-02T14:25:49.774 INFO:teuthology.orchestra.run.smithi092.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-02T14:25:49.774 INFO:teuthology.orchestra.run.smithi092.stdout:Device: 5h/5d Inode: 702 Links: 1 2024-04-02T14:25:49.774 INFO:teuthology.orchestra.run.smithi092.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-02T14:25:49.774 INFO:teuthology.orchestra.run.smithi092.stdout:Context: system_u:object_r:device_t:s0 2024-04-02T14:25:49.775 INFO:teuthology.orchestra.run.smithi092.stdout:Access: 2024-04-02 14:25:35.882375140 +0000 2024-04-02T14:25:49.775 INFO:teuthology.orchestra.run.smithi092.stdout:Modify: 2024-04-02 14:18:24.800696295 +0000 2024-04-02T14:25:49.775 INFO:teuthology.orchestra.run.smithi092.stdout:Change: 2024-04-02 14:18:24.800696295 +0000 2024-04-02T14:25:49.775 INFO:teuthology.orchestra.run.smithi092.stdout: Birth: 2024-04-02 14:18:24.800696295 +0000 2024-04-02T14:25:49.775 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-02T14:25:49.841 INFO:teuthology.orchestra.run.smithi092.stderr:1+0 records in 2024-04-02T14:25:49.841 INFO:teuthology.orchestra.run.smithi092.stderr:1+0 records out 2024-04-02T14:25:49.842 INFO:teuthology.orchestra.run.smithi092.stderr:512 bytes copied, 0.000243506 s, 2.1 MB/s 2024-04-02T14:25:49.843 DEBUG:teuthology.orchestra.run.smithi092:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-02T14:25:49.900 DEBUG:teuthology.orchestra.run.smithi092:> stat /dev/vg_nvme/lv_4 2024-04-02T14:25:49.956 INFO:teuthology.orchestra.run.smithi092.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-02T14:25:49.956 INFO:teuthology.orchestra.run.smithi092.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-02T14:25:49.956 INFO:teuthology.orchestra.run.smithi092.stdout:Device: 5h/5d Inode: 720 Links: 1 2024-04-02T14:25:49.956 INFO:teuthology.orchestra.run.smithi092.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-02T14:25:49.956 INFO:teuthology.orchestra.run.smithi092.stdout:Context: system_u:object_r:device_t:s0 2024-04-02T14:25:49.956 INFO:teuthology.orchestra.run.smithi092.stdout:Access: 2024-04-02 14:25:35.883375111 +0000 2024-04-02T14:25:49.956 INFO:teuthology.orchestra.run.smithi092.stdout:Modify: 2024-04-02 14:18:25.153685737 +0000 2024-04-02T14:25:49.956 INFO:teuthology.orchestra.run.smithi092.stdout:Change: 2024-04-02 14:18:25.153685737 +0000 2024-04-02T14:25:49.956 INFO:teuthology.orchestra.run.smithi092.stdout: Birth: 2024-04-02 14:18:25.153685737 +0000 2024-04-02T14:25:49.956 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-02T14:25:50.023 INFO:teuthology.orchestra.run.smithi092.stderr:1+0 records in 2024-04-02T14:25:50.024 INFO:teuthology.orchestra.run.smithi092.stderr:1+0 records out 2024-04-02T14:25:50.024 INFO:teuthology.orchestra.run.smithi092.stderr:512 bytes copied, 0.000202465 s, 2.5 MB/s 2024-04-02T14:25:50.025 DEBUG:teuthology.orchestra.run.smithi092:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-02T14:25:50.081 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-04-02T14:25:50.081 DEBUG:teuthology.orchestra.run.smithi177:> dd if=/scratch_devs of=/dev/stdout 2024-04-02T14:25:50.096 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-02T14:25:50.096 DEBUG:teuthology.orchestra.run.smithi177:> stat /dev/vg_nvme/lv_1 2024-04-02T14:25:50.151 INFO:teuthology.orchestra.run.smithi177.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-02T14:25:50.151 INFO:teuthology.orchestra.run.smithi177.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-02T14:25:50.151 INFO:teuthology.orchestra.run.smithi177.stdout:Device: 5h/5d Inode: 663 Links: 1 2024-04-02T14:25:50.151 INFO:teuthology.orchestra.run.smithi177.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-02T14:25:50.151 INFO:teuthology.orchestra.run.smithi177.stdout:Context: system_u:object_r:device_t:s0 2024-04-02T14:25:50.151 INFO:teuthology.orchestra.run.smithi177.stdout:Access: 2024-04-02 14:25:26.055553133 +0000 2024-04-02T14:25:50.152 INFO:teuthology.orchestra.run.smithi177.stdout:Modify: 2024-04-02 14:18:24.109403714 +0000 2024-04-02T14:25:50.152 INFO:teuthology.orchestra.run.smithi177.stdout:Change: 2024-04-02 14:18:24.109403714 +0000 2024-04-02T14:25:50.152 INFO:teuthology.orchestra.run.smithi177.stdout: Birth: 2024-04-02 14:18:24.109403714 +0000 2024-04-02T14:25:50.152 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-02T14:25:50.218 INFO:teuthology.orchestra.run.smithi177.stderr:1+0 records in 2024-04-02T14:25:50.218 INFO:teuthology.orchestra.run.smithi177.stderr:1+0 records out 2024-04-02T14:25:50.218 INFO:teuthology.orchestra.run.smithi177.stderr:512 bytes copied, 0.000199216 s, 2.6 MB/s 2024-04-02T14:25:50.219 DEBUG:teuthology.orchestra.run.smithi177:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-02T14:25:50.277 DEBUG:teuthology.orchestra.run.smithi177:> stat /dev/vg_nvme/lv_2 2024-04-02T14:25:50.332 INFO:teuthology.orchestra.run.smithi177.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-02T14:25:50.332 INFO:teuthology.orchestra.run.smithi177.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-02T14:25:50.332 INFO:teuthology.orchestra.run.smithi177.stdout:Device: 5h/5d Inode: 682 Links: 1 2024-04-02T14:25:50.333 INFO:teuthology.orchestra.run.smithi177.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-02T14:25:50.333 INFO:teuthology.orchestra.run.smithi177.stdout:Context: system_u:object_r:device_t:s0 2024-04-02T14:25:50.333 INFO:teuthology.orchestra.run.smithi177.stdout:Access: 2024-04-02 14:25:26.056553107 +0000 2024-04-02T14:25:50.333 INFO:teuthology.orchestra.run.smithi177.stdout:Modify: 2024-04-02 14:18:24.442394383 +0000 2024-04-02T14:25:50.333 INFO:teuthology.orchestra.run.smithi177.stdout:Change: 2024-04-02 14:18:24.442394383 +0000 2024-04-02T14:25:50.333 INFO:teuthology.orchestra.run.smithi177.stdout: Birth: 2024-04-02 14:18:24.442394383 +0000 2024-04-02T14:25:50.333 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-02T14:25:50.398 INFO:teuthology.orchestra.run.smithi177.stderr:1+0 records in 2024-04-02T14:25:50.398 INFO:teuthology.orchestra.run.smithi177.stderr:1+0 records out 2024-04-02T14:25:50.398 INFO:teuthology.orchestra.run.smithi177.stderr:512 bytes copied, 0.000178146 s, 2.9 MB/s 2024-04-02T14:25:50.400 DEBUG:teuthology.orchestra.run.smithi177:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-02T14:25:50.455 DEBUG:teuthology.orchestra.run.smithi177:> stat /dev/vg_nvme/lv_3 2024-04-02T14:25:50.510 INFO:teuthology.orchestra.run.smithi177.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-02T14:25:50.510 INFO:teuthology.orchestra.run.smithi177.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-02T14:25:50.510 INFO:teuthology.orchestra.run.smithi177.stdout:Device: 5h/5d Inode: 699 Links: 1 2024-04-02T14:25:50.511 INFO:teuthology.orchestra.run.smithi177.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-02T14:25:50.511 INFO:teuthology.orchestra.run.smithi177.stdout:Context: system_u:object_r:device_t:s0 2024-04-02T14:25:50.511 INFO:teuthology.orchestra.run.smithi177.stdout:Access: 2024-04-02 14:25:26.056553107 +0000 2024-04-02T14:25:50.511 INFO:teuthology.orchestra.run.smithi177.stdout:Modify: 2024-04-02 14:18:24.770385192 +0000 2024-04-02T14:25:50.511 INFO:teuthology.orchestra.run.smithi177.stdout:Change: 2024-04-02 14:18:24.770385192 +0000 2024-04-02T14:25:50.511 INFO:teuthology.orchestra.run.smithi177.stdout: Birth: 2024-04-02 14:18:24.770385192 +0000 2024-04-02T14:25:50.511 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-02T14:25:50.577 INFO:teuthology.orchestra.run.smithi177.stderr:1+0 records in 2024-04-02T14:25:50.577 INFO:teuthology.orchestra.run.smithi177.stderr:1+0 records out 2024-04-02T14:25:50.577 INFO:teuthology.orchestra.run.smithi177.stderr:512 bytes copied, 0.000195329 s, 2.6 MB/s 2024-04-02T14:25:50.578 DEBUG:teuthology.orchestra.run.smithi177:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-02T14:25:50.634 DEBUG:teuthology.orchestra.run.smithi177:> stat /dev/vg_nvme/lv_4 2024-04-02T14:25:50.692 INFO:teuthology.orchestra.run.smithi177.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-02T14:25:50.693 INFO:teuthology.orchestra.run.smithi177.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-02T14:25:50.693 INFO:teuthology.orchestra.run.smithi177.stdout:Device: 5h/5d Inode: 720 Links: 1 2024-04-02T14:25:50.693 INFO:teuthology.orchestra.run.smithi177.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-02T14:25:50.693 INFO:teuthology.orchestra.run.smithi177.stdout:Context: system_u:object_r:device_t:s0 2024-04-02T14:25:50.693 INFO:teuthology.orchestra.run.smithi177.stdout:Access: 2024-04-02 14:25:26.056553107 +0000 2024-04-02T14:25:50.693 INFO:teuthology.orchestra.run.smithi177.stdout:Modify: 2024-04-02 14:18:25.099375973 +0000 2024-04-02T14:25:50.693 INFO:teuthology.orchestra.run.smithi177.stdout:Change: 2024-04-02 14:18:25.099375973 +0000 2024-04-02T14:25:50.693 INFO:teuthology.orchestra.run.smithi177.stdout: Birth: 2024-04-02 14:18:25.099375973 +0000 2024-04-02T14:25:50.693 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-02T14:25:50.763 INFO:teuthology.orchestra.run.smithi177.stderr:1+0 records in 2024-04-02T14:25:50.763 INFO:teuthology.orchestra.run.smithi177.stderr:1+0 records out 2024-04-02T14:25:50.763 INFO:teuthology.orchestra.run.smithi177.stderr:512 bytes copied, 0.000195894 s, 2.6 MB/s 2024-04-02T14:25:50.764 DEBUG:teuthology.orchestra.run.smithi177:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-02T14:25:50.823 INFO:tasks.cephadm:Deploying osd.0 on smithi090 with /dev/vg_nvme/lv_4... 2024-04-02T14:25:50.823 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:191d963ac9de7322da8aff20b567b3fa458e2ef6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6449448a-f0fc-11ee-b647-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-02T14:25:51.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:50 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:25:51.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:50 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:25:51.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:50 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T14:25:51.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:50 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:25:51.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:50 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T14:25:51.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:50 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:25:51.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:50 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-02T14:25:51.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:50 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-02T14:25:51.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:50 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-02T14:25:51.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:50 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:25:51.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:50 smithi177 ceph-mon[36431]: Deploying daemon mgr.z on smithi177 2024-04-02T14:25:51.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:50 smithi177 ceph-mon[36431]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T14:25:51.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:50 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:25:51.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:50 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:25:51.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:50 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T14:25:51.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:50 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:25:51.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:50 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T14:25:51.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:50 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:25:51.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:50 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-02T14:25:51.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:50 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-02T14:25:51.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:50 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-02T14:25:51.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:50 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:25:51.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:50 smithi090 ceph-mon[31674]: Deploying daemon mgr.z on smithi177 2024-04-02T14:25:51.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:50 smithi090 ceph-mon[31674]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T14:25:51.251 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T14:25:51.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:25:50 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:25:51.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:25:50 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:25:51.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:25:50 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T14:25:51.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:25:50 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:25:51.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:25:50 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T14:25:51.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:25:50 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:25:51.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:25:50 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-02T14:25:51.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:25:50 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-02T14:25:51.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:25:50 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-02T14:25:51.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:25:50 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:25:51.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:25:50 smithi092 ceph-mon[36497]: Deploying daemon mgr.z on smithi177 2024-04-02T14:25:51.362 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:25:50 smithi092 ceph-mon[36497]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T14:25:52.687 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 02 14:25:52 smithi177 systemd[1]: Started Ceph mgr.z for 6449448a-f0fc-11ee-b647-cb9ed24678a4. 2024-04-02T14:25:53.437 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 02 14:25:53 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-z[38299]: 2024-04-02T14:25:53.054+0000 7f72f9970200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-02T14:25:53.437 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 02 14:25:53 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-z[38299]: 2024-04-02T14:25:53.147+0000 7f72f9970200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-02T14:25:53.937 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 02 14:25:53 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-z[38299]: 2024-04-02T14:25:53.791+0000 7f72f9970200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-02T14:25:53.937 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 02 14:25:53 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-z[38299]: 2024-04-02T14:25:53.881+0000 7f72f9970200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-02T14:25:53.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:53 smithi177 ceph-mon[36431]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T14:25:53.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:53 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:25:53.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:53 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:25:53.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:53 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:25:53.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:53 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-02T14:25:53.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:53 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-02T14:25:53.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:53 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-02T14:25:53.938 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:53 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:25:53.938 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:53 smithi177 ceph-mon[36431]: Deploying daemon mgr.y on smithi092 2024-04-02T14:25:53.939 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:53 smithi090 ceph-mon[31674]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T14:25:53.939 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:53 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:25:53.939 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:53 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:25:53.939 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:53 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:25:53.939 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:53 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-02T14:25:53.939 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:53 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-02T14:25:53.939 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:53 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-02T14:25:53.939 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:53 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:25:53.939 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:53 smithi090 ceph-mon[31674]: Deploying daemon mgr.y on smithi092 2024-04-02T14:25:54.047 INFO:teuthology.orchestra.run.smithi090.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-02T14:25:54.047 INFO:teuthology.orchestra.run.smithi090.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-02T14:25:54.047 INFO:teuthology.orchestra.run.smithi090.stderr: stderr: 10+0 records in 2024-04-02T14:25:54.047 INFO:teuthology.orchestra.run.smithi090.stderr:10+0 records out 2024-04-02T14:25:54.048 INFO:teuthology.orchestra.run.smithi090.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0338044 s, 310 MB/s 2024-04-02T14:25:54.048 INFO:teuthology.orchestra.run.smithi090.stderr:--> Zapping successful for: 2024-04-02T14:25:54.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:25:53 smithi092 ceph-mon[36497]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T14:25:54.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:25:53 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:25:54.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:25:53 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:25:54.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:25:53 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:25:54.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:25:53 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-02T14:25:54.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:25:53 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-02T14:25:54.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:25:53 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-02T14:25:54.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:25:53 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:25:54.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:25:53 smithi092 ceph-mon[36497]: Deploying daemon mgr.y on smithi092 2024-04-02T14:25:54.437 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 02 14:25:54 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-z[38299]: 2024-04-02T14:25:54.023+0000 7f72f9970200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-02T14:25:54.437 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 02 14:25:54 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-z[38299]: 2024-04-02T14:25:54.118+0000 7f72f9970200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-02T14:25:54.437 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 02 14:25:54 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-z[38299]: 2024-04-02T14:25:54.200+0000 7f72f9970200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-02T14:25:54.748 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 02 14:25:54 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-z[38299]: 2024-04-02T14:25:54.461+0000 7f72f9970200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-02T14:25:54.749 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 02 14:25:54 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-z[38299]: 2024-04-02T14:25:54.568+0000 7f72f9970200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-02T14:25:54.749 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 02 14:25:54 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-z[38299]: 2024-04-02T14:25:54.652+0000 7f72f9970200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-02T14:25:54.749 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 02 14:25:54 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-z[38299]: 2024-04-02T14:25:54.746+0000 7f72f9970200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-02T14:25:54.802 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:191d963ac9de7322da8aff20b567b3fa458e2ef6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6449448a-f0fc-11ee-b647-cb9ed24678a4 -- ceph orch daemon add osd smithi090:vg_nvme/lv_4 2024-04-02T14:25:54.861 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 02 14:25:54 smithi092 systemd[1]: Starting Ceph mgr.y for 6449448a-f0fc-11ee-b647-cb9ed24678a4... 2024-04-02T14:25:55.150 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 02 14:25:54 smithi092 podman[37864]: 2024-04-02 14:25:54.793902628 +0000 UTC m=+0.017568902 image pull f7301e070545fc23d8e5b7b052a2c15e832a0e57102f1acad8e4f3583709d23f quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534 2024-04-02T14:25:55.150 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 02 14:25:54 smithi092 podman[37864]: 2024-04-02 14:25:54.902220285 +0000 UTC m=+0.125886558 container create f24422517204d21896612990dfe79732e99158d6e42b90160a73f59da125b809 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-y, org.label-schema.build-date=20240326, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240402.004512-191d963, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-04-02T14:25:55.150 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 02 14:25:55 smithi092 podman[37864]: 2024-04-02 14:25:55.121448249 +0000 UTC m=+0.345114522 container init f24422517204d21896612990dfe79732e99158d6e42b90160a73f59da125b809 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-y, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240326, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240402.004512-191d963) 2024-04-02T14:25:55.150 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 02 14:25:55 smithi092 podman[37864]: 2024-04-02 14:25:55.124221262 +0000 UTC m=+0.347887546 container start f24422517204d21896612990dfe79732e99158d6e42b90160a73f59da125b809 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-y, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240402.004512-191d963, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240326, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-04-02T14:25:55.150 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 02 14:25:55 smithi092 ceph-mgr[37884]: -- 172.21.15.92:0/1562486830 <== mon.2 v2:172.21.15.92:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55dc7616a5a0 con 0x55dc7613f000 2024-04-02T14:25:55.187 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 02 14:25:54 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-z[38299]: 2024-04-02T14:25:54.923+0000 7f72f9970200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-02T14:25:55.251 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T14:25:55.464 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 02 14:25:55 smithi092 bash[37864]: f24422517204d21896612990dfe79732e99158d6e42b90160a73f59da125b809 2024-04-02T14:25:55.465 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 02 14:25:55 smithi092 systemd[1]: Started Ceph mgr.y for 6449448a-f0fc-11ee-b647-cb9ed24678a4. 2024-04-02T14:25:55.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:25:55 smithi092 ceph-mon[36497]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T14:25:55.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:25:55 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:25:55.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:25:55 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:25:55.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:25:55 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:25:55.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:25:55 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:25:55.716 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:25:55 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T14:25:55.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:55 smithi177 ceph-mon[36431]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T14:25:55.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:55 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:25:55.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:55 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:25:55.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:55 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:25:55.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:55 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:25:55.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:55 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T14:25:55.939 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:55 smithi090 ceph-mon[31674]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T14:25:55.939 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:55 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:25:55.939 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:55 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:25:55.939 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:55 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:25:55.939 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:55 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:25:55.939 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:55 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T14:25:56.110 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 02 14:25:55 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-y[37879]: 2024-04-02T14:25:55.713+0000 7fc8568cb200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-02T14:25:56.111 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 02 14:25:55 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-y[37879]: 2024-04-02T14:25:55.820+0000 7fc8568cb200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-02T14:25:56.186 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 02 14:25:55 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-z[38299]: 2024-04-02T14:25:55.915+0000 7f72f9970200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-02T14:25:56.187 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 02 14:25:56 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-z[38299]: 2024-04-02T14:25:56.051+0000 7f72f9970200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-02T14:25:56.610 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 02 14:25:56 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-y[37879]: 2024-04-02T14:25:56.513+0000 7fc8568cb200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-02T14:25:56.610 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 02 14:25:56 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-y[37879]: 2024-04-02T14:25:56.608+0000 7fc8568cb200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-02T14:25:56.687 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 02 14:25:56 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-z[38299]: 2024-04-02T14:25:56.406+0000 7f72f9970200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-02T14:25:56.687 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 02 14:25:56 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-z[38299]: 2024-04-02T14:25:56.499+0000 7f72f9970200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-02T14:25:56.860 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 02 14:25:56 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-y[37879]: 2024-04-02T14:25:56.754+0000 7fc8568cb200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-02T14:25:56.861 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 02 14:25:56 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-y[37879]: 2024-04-02T14:25:56.851+0000 7fc8568cb200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-02T14:25:57.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:57.174+0000 7f0b1ca00700 1 -- 172.21.15.90:0/1233664049 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0b181059e0 msgr2=0x7f0b18107dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:25:57.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:57.174+0000 7f0b1ca00700 1 --2- 172.21.15.90:0/1233664049 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0b181059e0 0x7f0b18107dd0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f0b04009990 tx=0x7f0b04009ca0 comp rx=0 tx=0).stop 2024-04-02T14:25:57.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:57.174+0000 7f0b1ca00700 1 -- 172.21.15.90:0/1233664049 shutdown_connections 2024-04-02T14:25:57.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:57.174+0000 7f0b1ca00700 1 --2- 172.21.15.90:0/1233664049 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0b181030b0 0x7f0b181054a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:25:57.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:57.174+0000 7f0b1ca00700 1 --2- 172.21.15.90:0/1233664049 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0b18108310 0x7f0b1810a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:25:57.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:57.174+0000 7f0b1ca00700 1 --2- 172.21.15.90:0/1233664049 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0b181059e0 0x7f0b18107dd0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:25:57.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:57.174+0000 7f0b1ca00700 1 -- 172.21.15.90:0/1233664049 >> 172.21.15.90:0/1233664049 conn(0x7f0b180fcac0 msgr2=0x7f0b180fef00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:25:57.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:57.174+0000 7f0b1ca00700 1 -- 172.21.15.90:0/1233664049 shutdown_connections 2024-04-02T14:25:57.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:57.174+0000 7f0b1ca00700 1 -- 172.21.15.90:0/1233664049 wait complete. 2024-04-02T14:25:57.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:57.175+0000 7f0b1ca00700 1 Processor -- start 2024-04-02T14:25:57.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:57.175+0000 7f0b1ca00700 1 -- start start 2024-04-02T14:25:57.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:57.175+0000 7f0b1ca00700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0b181030b0 0x7f0b1819ab30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:25:57.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:57.175+0000 7f0b1ca00700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0b181059e0 0x7f0b1819b070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:25:57.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:57.175+0000 7f0b1ca00700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0b18108310 0x7f0b181a20f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:25:57.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:57.175+0000 7f0b1ca00700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f0b1810e260 con 0x7f0b18108310 2024-04-02T14:25:57.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:57.175+0000 7f0b1ca00700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f0b1810e0e0 con 0x7f0b181030b0 2024-04-02T14:25:57.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:57.175+0000 7f0b1ca00700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f0b1810e3e0 con 0x7f0b181059e0 2024-04-02T14:25:57.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:57.176+0000 7f0b15e00700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0b181030b0 0x7f0b1819ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:25:57.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:57.176+0000 7f0b15e00700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0b181030b0 0x7f0b1819ab30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:40982/0 (socket says 172.21.15.90:40982) 2024-04-02T14:25:57.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:57.176+0000 7f0b15e00700 1 -- 172.21.15.90:0/4109028961 learned_addr learned my addr 172.21.15.90:0/4109028961 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:25:57.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:57.176+0000 7f0b15400700 1 --2- 172.21.15.90:0/4109028961 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0b181059e0 0x7f0b1819b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:25:57.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:57.176+0000 7f0b16800700 1 --2- 172.21.15.90:0/4109028961 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0b18108310 0x7f0b181a20f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:25:57.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:57.176+0000 7f0b15e00700 1 -- 172.21.15.90:0/4109028961 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0b181059e0 msgr2=0x7f0b1819b070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:25:57.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:57.176+0000 7f0b15e00700 1 --2- 172.21.15.90:0/4109028961 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0b181059e0 0x7f0b1819b070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:25:57.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:57.176+0000 7f0b15e00700 1 -- 172.21.15.90:0/4109028961 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0b18108310 msgr2=0x7f0b181a20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:25:57.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:57.176+0000 7f0b15e00700 1 --2- 172.21.15.90:0/4109028961 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0b18108310 0x7f0b181a20f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:25:57.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:57.176+0000 7f0b15e00700 1 -- 172.21.15.90:0/4109028961 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b181a2790 con 0x7f0b181030b0 2024-04-02T14:25:57.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:57.176+0000 7f0b16800700 1 --2- 172.21.15.90:0/4109028961 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0b18108310 0x7f0b181a20f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:25:57.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:57.176+0000 7f0b15e00700 1 --2- 172.21.15.90:0/4109028961 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0b181030b0 0x7f0b1819ab30 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f0b0800eb30 tx=0x7f0b0800ee40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:25:57.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:57.177+0000 7f0b0ea00700 1 -- 172.21.15.90:0/4109028961 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b0800cc90 con 0x7f0b181030b0 2024-04-02T14:25:57.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:57.177+0000 7f0b0ea00700 1 -- 172.21.15.90:0/4109028961 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0b08004500 con 0x7f0b181030b0 2024-04-02T14:25:57.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:57.177+0000 7f0b0ea00700 1 -- 172.21.15.90:0/4109028961 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b08010430 con 0x7f0b181030b0 2024-04-02T14:25:57.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:57.177+0000 7f0b1ca00700 1 -- 172.21.15.90:0/4109028961 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0b181a2a80 con 0x7f0b181030b0 2024-04-02T14:25:57.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:57.177+0000 7f0b1ca00700 1 -- 172.21.15.90:0/4109028961 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0b181a2f70 con 0x7f0b181030b0 2024-04-02T14:25:57.181 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 02 14:25:56 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-z[38299]: 2024-04-02T14:25:56.699+0000 7f72f9970200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-02T14:25:57.181 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 02 14:25:56 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-z[38299]: 2024-04-02T14:25:56.896+0000 7f72f9970200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-02T14:25:57.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:57.178+0000 7f0b1ca00700 1 -- 172.21.15.90:0/4109028961 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0b180692b0 con 0x7f0b181030b0 2024-04-02T14:25:57.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:57.178+0000 7f0b0ea00700 1 -- 172.21.15.90:0/4109028961 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f0b08003fa0 con 0x7f0b181030b0 2024-04-02T14:25:57.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:57.178+0000 7f0b0ea00700 1 --2- 172.21.15.90:0/4109028961 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f0afc039890 0x7f0afc03bd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:25:57.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:57.178+0000 7f0b0ea00700 1 -- 172.21.15.90:0/4109028961 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f0b0804cd90 con 0x7f0b181030b0 2024-04-02T14:25:57.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:57.179+0000 7f0b15400700 1 --2- 172.21.15.90:0/4109028961 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f0afc039890 0x7f0afc03bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:25:57.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:57.179+0000 7f0b15400700 1 --2- 172.21.15.90:0/4109028961 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f0afc039890 0x7f0afc03bd50 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f0b0400b7b0 tx=0x7f0b04005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:25:57.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:57.184+0000 7f0b0ea00700 1 -- 172.21.15.90:0/4109028961 <== mon.2 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7f0b0801fba0 con 0x7f0b181030b0 2024-04-02T14:25:57.208 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 02 14:25:56 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-y[37879]: 2024-04-02T14:25:56.934+0000 7fc8568cb200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-02T14:25:57.208 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 02 14:25:57 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-y[37879]: 2024-04-02T14:25:57.206+0000 7fc8568cb200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-02T14:25:57.439 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 02 14:25:57 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-z[38299]: 2024-04-02T14:25:57.178+0000 7f72f9970200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-02T14:25:57.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:25:57.444+0000 7f0b1ca00700 1 -- 172.21.15.90:0/4109028961 --> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi090:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f0b18062a30 con 0x7f0afc039890 2024-04-02T14:25:57.491 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 02 14:25:57 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-y[37879]: 2024-04-02T14:25:57.314+0000 7fc8568cb200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-02T14:25:57.491 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 02 14:25:57 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-y[37879]: 2024-04-02T14:25:57.397+0000 7fc8568cb200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-02T14:25:57.491 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 02 14:25:57 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-y[37879]: 2024-04-02T14:25:57.489+0000 7fc8568cb200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-02T14:25:57.833 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 02 14:25:57 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-z[38299]: 2024-04-02T14:25:57.444+0000 7f72f9970200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-02T14:25:57.833 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 02 14:25:57 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-z[38299]: 2024-04-02T14:25:57.677+0000 7f72f9970200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-02T14:25:57.833 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 02 14:25:57 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-y[37879]: 2024-04-02T14:25:57.661+0000 7fc8568cb200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-02T14:25:58.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:25:57 smithi092 ceph-mon[36497]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T14:25:58.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:25:57 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:25:58.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:25:57 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:25:58.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:25:57 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:25:58.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:25:57 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:25:58.112 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:25:57 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:25:58.112 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:25:57 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T14:25:58.112 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:25:57 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:25:58.112 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:25:57 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-02T14:25:58.112 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:25:57 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-02T14:25:58.112 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:25:57 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:25:58.112 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:25:57 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-02T14:25:58.112 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:25:57 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-02T14:25:58.113 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:25:57 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:25:58.139 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:57 smithi090 ceph-mon[31674]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T14:25:58.140 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:57 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:25:58.140 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:57 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:25:58.140 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:57 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:25:58.140 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:57 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:25:58.140 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:57 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:25:58.140 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:57 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T14:25:58.140 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:57 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:25:58.140 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:57 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-02T14:25:58.141 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:57 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-02T14:25:58.141 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:57 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:25:58.141 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:57 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-02T14:25:58.141 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:57 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-02T14:25:58.141 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:57 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:25:58.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:57 smithi177 ceph-mon[36431]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T14:25:58.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:57 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:25:58.188 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:57 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:25:58.188 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:57 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:25:58.188 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:57 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:25:58.188 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:57 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:25:58.188 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:57 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T14:25:58.188 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:57 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:25:58.188 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:57 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-02T14:25:58.189 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:57 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-02T14:25:58.189 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:57 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:25:58.189 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:57 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-02T14:25:58.189 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:57 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-02T14:25:58.189 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:57 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:25:58.835 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 02 14:25:58 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-z[38299]: 2024-04-02T14:25:58.472+0000 7f72f9970200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-02T14:25:59.075 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:25:58 smithi092 ceph-mon[36497]: Reconfiguring mgr.x (unknown last config time)... 2024-04-02T14:25:59.076 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:25:58 smithi092 ceph-mon[36497]: Reconfiguring daemon mgr.x on smithi090 2024-04-02T14:25:59.076 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:25:58 smithi092 ceph-mon[36497]: from='client.24116 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi090:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T14:25:59.076 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 02 14:25:58 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-y[37879]: 2024-04-02T14:25:58.615+0000 7fc8568cb200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-02T14:25:59.076 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 02 14:25:58 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-y[37879]: 2024-04-02T14:25:58.744+0000 7fc8568cb200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-02T14:25:59.076 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 02 14:25:59 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-y[37879]: 2024-04-02T14:25:59.073+0000 7fc8568cb200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-02T14:25:59.140 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:58 smithi177 ceph-mon[36431]: Reconfiguring mgr.x (unknown last config time)... 2024-04-02T14:25:59.140 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:58 smithi177 ceph-mon[36431]: Reconfiguring daemon mgr.x on smithi090 2024-04-02T14:25:59.140 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:58 smithi177 ceph-mon[36431]: from='client.24116 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi090:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T14:25:59.141 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 02 14:25:59 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-z[38299]: 2024-04-02T14:25:59.042+0000 7f72f9970200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-02T14:25:59.141 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 02 14:25:59 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-z[38299]: 2024-04-02T14:25:59.138+0000 7f72f9970200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-02T14:25:59.153 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:58 smithi090 ceph-mon[31674]: Reconfiguring mgr.x (unknown last config time)... 2024-04-02T14:25:59.153 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:58 smithi090 ceph-mon[31674]: Reconfiguring daemon mgr.x on smithi090 2024-04-02T14:25:59.153 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:58 smithi090 ceph-mon[31674]: from='client.24116 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi090:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T14:25:59.358 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 02 14:25:59 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-y[37879]: 2024-04-02T14:25:59.162+0000 7fc8568cb200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-02T14:25:59.359 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 02 14:25:59 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-y[37879]: 2024-04-02T14:25:59.355+0000 7fc8568cb200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-02T14:25:59.437 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 02 14:25:59 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-z[38299]: 2024-04-02T14:25:59.352+0000 7f72f9970200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-02T14:25:59.611 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 02 14:25:59 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-y[37879]: 2024-04-02T14:25:59.540+0000 7fc8568cb200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-02T14:26:00.082 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 02 14:25:59 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-y[37879]: 2024-04-02T14:25:59.824+0000 7fc8568cb200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-02T14:26:00.082 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 02 14:26:00 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-y[37879]: 2024-04-02T14:26:00.079+0000 7fc8568cb200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-02T14:26:00.082 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:25:59 smithi092 ceph-mon[36497]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T14:26:00.082 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:25:59 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:26:00.083 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:25:59 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:26:00.083 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:25:59 smithi092 ceph-mon[36497]: from='mgr.? 172.21.15.177:0/1969830370' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-02T14:26:00.083 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:25:59 smithi092 ceph-mon[36497]: from='mgr.? 172.21.15.177:0/1969830370' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-02T14:26:00.083 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:25:59 smithi092 ceph-mon[36497]: Standby manager daemon z started 2024-04-02T14:26:00.083 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:25:59 smithi092 ceph-mon[36497]: from='mgr.? 172.21.15.177:0/1969830370' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-02T14:26:00.083 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:25:59 smithi092 ceph-mon[36497]: from='mgr.? 172.21.15.177:0/1969830370' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-02T14:26:00.083 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:25:59 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T14:26:00.083 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:25:59 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:26:00.083 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:25:59 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T14:26:00.083 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:25:59 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:26:00.083 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:25:59 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:26:00.127 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:59 smithi090 ceph-mon[31674]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T14:26:00.127 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:59 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:26:00.127 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:59 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:26:00.127 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:59 smithi090 ceph-mon[31674]: from='mgr.? 172.21.15.177:0/1969830370' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-02T14:26:00.127 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:59 smithi090 ceph-mon[31674]: from='mgr.? 172.21.15.177:0/1969830370' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-02T14:26:00.128 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:59 smithi090 ceph-mon[31674]: Standby manager daemon z started 2024-04-02T14:26:00.128 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:59 smithi090 ceph-mon[31674]: from='mgr.? 172.21.15.177:0/1969830370' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-02T14:26:00.128 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:59 smithi090 ceph-mon[31674]: from='mgr.? 172.21.15.177:0/1969830370' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-02T14:26:00.128 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:59 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T14:26:00.128 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:59 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:26:00.128 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:59 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T14:26:00.128 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:59 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:26:00.128 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:25:59 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:26:00.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:59 smithi177 ceph-mon[36431]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T14:26:00.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:59 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:26:00.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:59 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:26:00.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:59 smithi177 ceph-mon[36431]: from='mgr.? 172.21.15.177:0/1969830370' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-02T14:26:00.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:59 smithi177 ceph-mon[36431]: from='mgr.? 172.21.15.177:0/1969830370' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-02T14:26:00.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:59 smithi177 ceph-mon[36431]: Standby manager daemon z started 2024-04-02T14:26:00.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:59 smithi177 ceph-mon[36431]: from='mgr.? 172.21.15.177:0/1969830370' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-02T14:26:00.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:59 smithi177 ceph-mon[36431]: from='mgr.? 172.21.15.177:0/1969830370' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-02T14:26:00.188 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:59 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T14:26:00.188 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:59 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:26:00.188 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:59 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T14:26:00.188 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:59 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:26:00.188 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:25:59 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:26:00.361 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 02 14:26:00 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-y[37879]: 2024-04-02T14:26:00.316+0000 7fc8568cb200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-02T14:26:00.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:26:00.373+0000 7f0b0ea00700 1 -- 172.21.15.90:0/4109028961 <== mon.2 v2:172.21.15.92:3300/0 7 ==== mgrmap(e 14) ==== 92602+0+0 (secure 0 0 0) 0x7f0b080217b0 con 0x7f0b181030b0 2024-04-02T14:26:01.376 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 02 14:26:01 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-y[37879]: 2024-04-02T14:26:01.121+0000 7fc8568cb200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-02T14:26:01.686 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:26:01 smithi177 ceph-mon[36431]: mgrmap e14: x(active, since 85s), standbys: z 2024-04-02T14:26:01.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:26:01 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-02T14:26:01.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:26:01 smithi177 ceph-mon[36431]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T14:26:01.688 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:26:01 smithi092 ceph-mon[36497]: mgrmap e14: x(active, since 85s), standbys: z 2024-04-02T14:26:01.688 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:26:01 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-02T14:26:01.688 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:26:01 smithi092 ceph-mon[36497]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T14:26:01.688 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:26:01 smithi090 ceph-mon[31674]: mgrmap e14: x(active, since 85s), standbys: z 2024-04-02T14:26:01.688 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:26:01 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-02T14:26:01.689 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:26:01 smithi090 ceph-mon[31674]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T14:26:02.000 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 02 14:26:01 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-y[37879]: 2024-04-02T14:26:01.686+0000 7fc8568cb200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-02T14:26:02.000 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 02 14:26:01 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-y[37879]: 2024-04-02T14:26:01.783+0000 7fc8568cb200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-02T14:26:02.001 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 02 14:26:02 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-y[37879]: 2024-04-02T14:26:01.999+0000 7fc8568cb200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-02T14:26:02.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:26:02.399+0000 7f0b0ea00700 1 -- 172.21.15.90:0/4109028961 <== mon.2 v2:172.21.15.92:3300/0 8 ==== mgrmap(e 15) ==== 138590+0+0 (secure 0 0 0) 0x7f0b08020cb0 con 0x7f0b181030b0 2024-04-02T14:26:02.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:26:02 smithi177 ceph-mon[36431]: from='mgr.? 172.21.15.92:0/2734962758' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-02T14:26:02.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:26:02 smithi177 ceph-mon[36431]: Standby manager daemon y started 2024-04-02T14:26:02.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:26:02 smithi177 ceph-mon[36431]: from='mgr.? 172.21.15.92:0/2734962758' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-02T14:26:02.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:26:02 smithi177 ceph-mon[36431]: from='mgr.? 172.21.15.92:0/2734962758' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-02T14:26:02.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:26:02 smithi177 ceph-mon[36431]: from='mgr.? 172.21.15.92:0/2734962758' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-02T14:26:02.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:26:02 smithi177 ceph-mon[36431]: from='client.? 172.21.15.90:0/1662689013' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a1a6a856-ab44-44bd-bc5e-45c718a70cd2"}]: dispatch 2024-04-02T14:26:02.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:26:02 smithi177 ceph-mon[36431]: from='client.? 172.21.15.90:0/1662689013' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a1a6a856-ab44-44bd-bc5e-45c718a70cd2"}]': finished 2024-04-02T14:26:02.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:26:02 smithi177 ceph-mon[36431]: osdmap e5: 1 total, 0 up, 1 in 2024-04-02T14:26:02.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:26:02 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-02T14:26:02.688 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:26:02 smithi090 ceph-mon[31674]: from='mgr.? 172.21.15.92:0/2734962758' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-02T14:26:02.688 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:26:02 smithi090 ceph-mon[31674]: Standby manager daemon y started 2024-04-02T14:26:02.688 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:26:02 smithi090 ceph-mon[31674]: from='mgr.? 172.21.15.92:0/2734962758' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-02T14:26:02.688 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:26:02 smithi090 ceph-mon[31674]: from='mgr.? 172.21.15.92:0/2734962758' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-02T14:26:02.688 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:26:02 smithi090 ceph-mon[31674]: from='mgr.? 172.21.15.92:0/2734962758' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-02T14:26:02.688 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:26:02 smithi090 ceph-mon[31674]: from='client.? 172.21.15.90:0/1662689013' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a1a6a856-ab44-44bd-bc5e-45c718a70cd2"}]: dispatch 2024-04-02T14:26:02.688 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:26:02 smithi090 ceph-mon[31674]: from='client.? 172.21.15.90:0/1662689013' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a1a6a856-ab44-44bd-bc5e-45c718a70cd2"}]': finished 2024-04-02T14:26:02.689 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:26:02 smithi090 ceph-mon[31674]: osdmap e5: 1 total, 0 up, 1 in 2024-04-02T14:26:02.689 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:26:02 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-02T14:26:02.861 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:26:02 smithi092 ceph-mon[36497]: from='mgr.? 172.21.15.92:0/2734962758' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-02T14:26:02.861 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:26:02 smithi092 ceph-mon[36497]: Standby manager daemon y started 2024-04-02T14:26:02.861 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:26:02 smithi092 ceph-mon[36497]: from='mgr.? 172.21.15.92:0/2734962758' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-02T14:26:02.861 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:26:02 smithi092 ceph-mon[36497]: from='mgr.? 172.21.15.92:0/2734962758' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-02T14:26:02.861 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:26:02 smithi092 ceph-mon[36497]: from='mgr.? 172.21.15.92:0/2734962758' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-02T14:26:02.861 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:26:02 smithi092 ceph-mon[36497]: from='client.? 172.21.15.90:0/1662689013' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a1a6a856-ab44-44bd-bc5e-45c718a70cd2"}]: dispatch 2024-04-02T14:26:02.861 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:26:02 smithi092 ceph-mon[36497]: from='client.? 172.21.15.90:0/1662689013' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a1a6a856-ab44-44bd-bc5e-45c718a70cd2"}]': finished 2024-04-02T14:26:02.861 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:26:02 smithi092 ceph-mon[36497]: osdmap e5: 1 total, 0 up, 1 in 2024-04-02T14:26:02.862 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:26:02 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-02T14:26:03.861 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:26:03 smithi092 ceph-mon[36497]: mgrmap e15: x(active, since 87s), standbys: z, y 2024-04-02T14:26:03.861 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:26:03 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-02T14:26:03.861 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:26:03 smithi092 ceph-mon[36497]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T14:26:03.861 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:26:03 smithi092 ceph-mon[36497]: from='client.? 172.21.15.90:0/907623760' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-02T14:26:03.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:26:03 smithi177 ceph-mon[36431]: mgrmap e15: x(active, since 87s), standbys: z, y 2024-04-02T14:26:03.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:26:03 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-02T14:26:03.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:26:03 smithi177 ceph-mon[36431]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T14:26:03.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:26:03 smithi177 ceph-mon[36431]: from='client.? 172.21.15.90:0/907623760' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-02T14:26:03.938 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:26:03 smithi090 ceph-mon[31674]: mgrmap e15: x(active, since 87s), standbys: z, y 2024-04-02T14:26:03.938 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:26:03 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-02T14:26:03.939 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:26:03 smithi090 ceph-mon[31674]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T14:26:03.939 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:26:03 smithi090 ceph-mon[31674]: from='client.? 172.21.15.90:0/907623760' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-02T14:26:05.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:26:05 smithi177 ceph-mon[36431]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T14:26:05.938 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:26:05 smithi090 ceph-mon[31674]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T14:26:06.110 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:26:05 smithi092 ceph-mon[36497]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T14:26:07.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:26:07 smithi177 ceph-mon[36431]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T14:26:07.938 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:26:07 smithi090 ceph-mon[31674]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T14:26:08.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:26:07 smithi092 ceph-mon[36497]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T14:26:09.970 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:26:09 smithi090 ceph-mon[31674]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T14:26:10.110 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:26:09 smithi092 ceph-mon[36497]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T14:26:10.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:26:09 smithi177 ceph-mon[36431]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T14:26:12.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:26:11 smithi092 ceph-mon[36497]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T14:26:12.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:26:11 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-02T14:26:12.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:26:11 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:26:12.185 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:26:11 smithi090 ceph-mon[31674]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T14:26:12.186 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:26:11 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-02T14:26:12.186 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:26:11 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:26:12.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:26:11 smithi177 ceph-mon[36431]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T14:26:12.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:26:11 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-02T14:26:12.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:26:11 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:26:12.939 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:26:12 smithi090 ceph-mon[31674]: Deploying daemon osd.0 on smithi090 2024-04-02T14:26:13.110 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:26:12 smithi092 ceph-mon[36497]: Deploying daemon osd.0 on smithi090 2024-04-02T14:26:13.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:26:12 smithi177 ceph-mon[36431]: Deploying daemon osd.0 on smithi090 2024-04-02T14:26:14.008 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:26:13 smithi090 ceph-mon[31674]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T14:26:14.110 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:26:13 smithi092 ceph-mon[36497]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T14:26:14.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:26:13 smithi177 ceph-mon[36431]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T14:26:15.897 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:26:15 smithi090 ceph-mon[31674]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T14:26:16.110 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:26:15 smithi092 ceph-mon[36497]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T14:26:16.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:26:15 smithi177 ceph-mon[36431]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T14:26:17.989 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:26:17 smithi177 ceph-mon[36431]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T14:26:18.033 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:26:17 smithi090 ceph-mon[31674]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T14:26:18.110 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:26:17 smithi092 ceph-mon[36497]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T14:26:20.110 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:26:19 smithi092 ceph-mon[36497]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T14:26:20.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:26:19 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:26:20.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:26:19 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:26:20.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:26:19 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T14:26:20.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:26:19 smithi177 ceph-mon[36431]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T14:26:20.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:26:19 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:26:20.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:26:19 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:26:20.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:26:19 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T14:26:20.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:26:19 smithi090 ceph-mon[31674]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T14:26:20.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:26:19 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:26:20.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:26:19 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:26:20.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:26:19 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T14:26:21.938 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:26:21 smithi090 ceph-mon[31674]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T14:26:22.110 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:26:21 smithi092 ceph-mon[36497]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T14:26:22.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:26:21 smithi177 ceph-mon[36431]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T14:26:23.439 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:26:23 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:26:23.439 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:26:23 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:26:23.439 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:26:23 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:26:23.439 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:26:23 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T14:26:23.439 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:26:23 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:26:23.439 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:26:23 smithi090 ceph-mon[31674]: pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T14:26:23.610 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:26:23 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:26:23.611 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:26:23 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:26:23.611 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:26:23 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:26:23.611 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:26:23 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T14:26:23.611 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:26:23 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:26:23.611 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:26:23 smithi092 ceph-mon[36497]: pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T14:26:23.686 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:26:23 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:26:23.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:26:23 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:26:23.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:26:23 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:26:23.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:26:23 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T14:26:23.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:26:23 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:26:23.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:26:23 smithi177 ceph-mon[36431]: pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T14:26:24.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:26:24.388+0000 7f0b0ea00700 1 -- 172.21.15.90:0/4109028961 <== mgr.14152 v2:172.21.15.90:6800/4266054817 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f0b18062a30 con 0x7f0afc039890 2024-04-02T14:26:24.391 INFO:teuthology.orchestra.run.smithi090.stdout:Created osd(s) 0 on host 'smithi090' 2024-04-02T14:26:24.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:26:24.392+0000 7f0b1ca00700 1 -- 172.21.15.90:0/4109028961 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f0afc039890 msgr2=0x7f0afc03bd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:26:24.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:26:24.392+0000 7f0b1ca00700 1 --2- 172.21.15.90:0/4109028961 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f0afc039890 0x7f0afc03bd50 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f0b0400b7b0 tx=0x7f0b04005fd0 comp rx=0 tx=0).stop 2024-04-02T14:26:24.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:26:24.392+0000 7f0b1ca00700 1 -- 172.21.15.90:0/4109028961 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0b181030b0 msgr2=0x7f0b1819ab30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:26:24.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:26:24.392+0000 7f0b1ca00700 1 --2- 172.21.15.90:0/4109028961 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0b181030b0 0x7f0b1819ab30 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f0b0800eb30 tx=0x7f0b0800ee40 comp rx=0 tx=0).stop 2024-04-02T14:26:24.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:26:24.392+0000 7f0b1ca00700 1 -- 172.21.15.90:0/4109028961 shutdown_connections 2024-04-02T14:26:24.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:26:24.392+0000 7f0b1ca00700 1 --2- 172.21.15.90:0/4109028961 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f0afc039890 0x7f0afc03bd50 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:26:24.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:26:24.392+0000 7f0b1ca00700 1 --2- 172.21.15.90:0/4109028961 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0b18108310 0x7f0b181a20f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:26:24.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:26:24.392+0000 7f0b1ca00700 1 --2- 172.21.15.90:0/4109028961 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0b181030b0 0x7f0b1819ab30 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:26:24.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:26:24.392+0000 7f0b1ca00700 1 --2- 172.21.15.90:0/4109028961 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0b181059e0 0x7f0b1819b070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:26:24.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:26:24.392+0000 7f0b1ca00700 1 -- 172.21.15.90:0/4109028961 >> 172.21.15.90:0/4109028961 conn(0x7f0b180fcac0 msgr2=0x7f0b18102930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:26:24.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:26:24.392+0000 7f0b1ca00700 1 -- 172.21.15.90:0/4109028961 shutdown_connections 2024-04-02T14:26:24.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:26:24.392+0000 7f0b1ca00700 1 -- 172.21.15.90:0/4109028961 wait complete. 2024-04-02T14:26:24.610 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:26:24 smithi092 ceph-mon[36497]: from='osd.0 [v2:172.21.15.90:6802/3635087609,v1:172.21.15.90:6803/3635087609]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-02T14:26:24.611 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:26:24 smithi092 ceph-mon[36497]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-02T14:26:24.686 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:26:24 smithi177 ceph-mon[36431]: from='osd.0 [v2:172.21.15.90:6802/3635087609,v1:172.21.15.90:6803/3635087609]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-02T14:26:24.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:26:24 smithi177 ceph-mon[36431]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-02T14:26:24.688 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:26:24 smithi090 ceph-mon[31674]: from='osd.0 [v2:172.21.15.90:6802/3635087609,v1:172.21.15.90:6803/3635087609]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-02T14:26:24.688 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:26:24 smithi090 ceph-mon[31674]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-02T14:26:25.086 DEBUG:teuthology.orchestra.run.smithi090:osd.0> sudo journalctl -f -n 0 -u ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4@osd.0.service 2024-04-02T14:26:25.091 INFO:tasks.cephadm:Deploying osd.1 on smithi092 with /dev/vg_nvme/lv_4... 2024-04-02T14:26:25.091 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:191d963ac9de7322da8aff20b567b3fa458e2ef6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6449448a-f0fc-11ee-b647-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-02T14:26:25.307 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:26:25 smithi090 ceph-mon[31674]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-02T14:26:25.307 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:26:25 smithi090 ceph-mon[31674]: from='osd.0 [v2:172.21.15.90:6802/3635087609,v1:172.21.15.90:6803/3635087609]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]: dispatch 2024-04-02T14:26:25.307 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:26:25 smithi090 ceph-mon[31674]: osdmap e6: 1 total, 0 up, 1 in 2024-04-02T14:26:25.307 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:26:25 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-02T14:26:25.307 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:26:25 smithi090 ceph-mon[31674]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]: dispatch 2024-04-02T14:26:25.307 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:26:25 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:26:25.307 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:26:25 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:26:25.308 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:26:25 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T14:26:25.308 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:26:25 smithi090 ceph-mon[31674]: pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T14:26:25.334 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/mon.b/config 2024-04-02T14:26:25.498 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:26:25 smithi092 ceph-mon[36497]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-02T14:26:25.498 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:26:25 smithi092 ceph-mon[36497]: from='osd.0 [v2:172.21.15.90:6802/3635087609,v1:172.21.15.90:6803/3635087609]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]: dispatch 2024-04-02T14:26:25.498 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:26:25 smithi092 ceph-mon[36497]: osdmap e6: 1 total, 0 up, 1 in 2024-04-02T14:26:25.498 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:26:25 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-02T14:26:25.498 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:26:25 smithi092 ceph-mon[36497]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]: dispatch 2024-04-02T14:26:25.498 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:26:25 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:26:25.498 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:26:25 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:26:25.498 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:26:25 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T14:26:25.498 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:26:25 smithi092 ceph-mon[36497]: pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T14:26:25.686 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:26:25 smithi177 ceph-mon[36431]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-02T14:26:25.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:26:25 smithi177 ceph-mon[36431]: from='osd.0 [v2:172.21.15.90:6802/3635087609,v1:172.21.15.90:6803/3635087609]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]: dispatch 2024-04-02T14:26:25.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:26:25 smithi177 ceph-mon[36431]: osdmap e6: 1 total, 0 up, 1 in 2024-04-02T14:26:25.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:26:25 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-02T14:26:25.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:26:25 smithi177 ceph-mon[36431]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]: dispatch 2024-04-02T14:26:25.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:26:25 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:26:25.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:26:25 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:26:25.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:26:25 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T14:26:25.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:26:25 smithi177 ceph-mon[36431]: pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T14:26:26.268 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:26:26 smithi090 ceph-mon[31674]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]': finished 2024-04-02T14:26:26.268 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:26:26 smithi090 ceph-mon[31674]: osdmap e7: 1 total, 0 up, 1 in 2024-04-02T14:26:26.268 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:26:26 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-02T14:26:26.268 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:26:26 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-02T14:26:26.268 INFO:journalctl@ceph.osd.0.smithi090.stdout:Apr 02 14:26:26 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-0[46353]: 2024-04-02T14:26:26.232+0000 7f8228c00700 -1 osd.0 0 waiting for initial osdmap 2024-04-02T14:26:26.501 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:26:26 smithi092 ceph-mon[36497]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]': finished 2024-04-02T14:26:26.501 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:26:26 smithi092 ceph-mon[36497]: osdmap e7: 1 total, 0 up, 1 in 2024-04-02T14:26:26.501 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:26:26 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-02T14:26:26.501 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:26:26 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-02T14:26:26.570 INFO:journalctl@ceph.osd.0.smithi090.stdout:Apr 02 14:26:26 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-0[46353]: 2024-04-02T14:26:26.284+0000 7f821ea00700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-02T14:26:26.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:26:26 smithi177 ceph-mon[36431]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]': finished 2024-04-02T14:26:26.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:26:26 smithi177 ceph-mon[36431]: osdmap e7: 1 total, 0 up, 1 in 2024-04-02T14:26:26.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:26:26 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-02T14:26:26.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:26:26 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-02T14:26:27.285 INFO:teuthology.orchestra.run.smithi092.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-02T14:26:27.286 INFO:teuthology.orchestra.run.smithi092.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-02T14:26:27.286 INFO:teuthology.orchestra.run.smithi092.stderr: stderr: 10+0 records in 2024-04-02T14:26:27.286 INFO:teuthology.orchestra.run.smithi092.stderr:10+0 records out 2024-04-02T14:26:27.286 INFO:teuthology.orchestra.run.smithi092.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0111989 s, 936 MB/s 2024-04-02T14:26:27.286 INFO:teuthology.orchestra.run.smithi092.stderr:--> Zapping successful for: 2024-04-02T14:26:27.585 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:26:27 smithi092 ceph-mon[36497]: purged_snaps scrub starts 2024-04-02T14:26:27.586 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:26:27 smithi092 ceph-mon[36497]: purged_snaps scrub ok 2024-04-02T14:26:27.586 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:26:27 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-02T14:26:27.586 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:26:27 smithi092 ceph-mon[36497]: from='osd.0 ' entity='osd.0' 2024-04-02T14:26:27.586 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:26:27 smithi092 ceph-mon[36497]: pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T14:26:27.686 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:26:27 smithi177 ceph-mon[36431]: purged_snaps scrub starts 2024-04-02T14:26:27.686 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:26:27 smithi177 ceph-mon[36431]: purged_snaps scrub ok 2024-04-02T14:26:27.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:26:27 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-02T14:26:27.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:26:27 smithi177 ceph-mon[36431]: from='osd.0 ' entity='osd.0' 2024-04-02T14:26:27.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:26:27 smithi177 ceph-mon[36431]: pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T14:26:27.688 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:26:27 smithi090 ceph-mon[31674]: purged_snaps scrub starts 2024-04-02T14:26:27.688 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:26:27 smithi090 ceph-mon[31674]: purged_snaps scrub ok 2024-04-02T14:26:27.688 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:26:27 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-02T14:26:27.688 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:26:27 smithi090 ceph-mon[31674]: from='osd.0 ' entity='osd.0' 2024-04-02T14:26:27.688 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:26:27 smithi090 ceph-mon[31674]: pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-02T14:26:27.829 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:191d963ac9de7322da8aff20b567b3fa458e2ef6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6449448a-f0fc-11ee-b647-cb9ed24678a4 -- ceph orch daemon add osd smithi092:vg_nvme/lv_4 2024-04-02T14:26:28.063 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/mon.b/config 2024-04-02T14:26:28.471 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:26:28 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-02T14:26:28.471 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:26:28 smithi092 ceph-mon[36497]: osd.0 [v2:172.21.15.90:6802/3635087609,v1:172.21.15.90:6803/3635087609] boot 2024-04-02T14:26:28.471 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:26:28 smithi092 ceph-mon[36497]: osdmap e8: 1 total, 1 up, 1 in 2024-04-02T14:26:28.471 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:26:28 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-02T14:26:28.632 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:26:28 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-02T14:26:28.632 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:26:28 smithi090 ceph-mon[31674]: osd.0 [v2:172.21.15.90:6802/3635087609,v1:172.21.15.90:6803/3635087609] boot 2024-04-02T14:26:28.632 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:26:28 smithi090 ceph-mon[31674]: osdmap e8: 1 total, 1 up, 1 in 2024-04-02T14:26:28.632 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:26:28 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-02T14:26:28.686 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:26:28 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-02T14:26:28.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:26:28 smithi177 ceph-mon[36431]: osd.0 [v2:172.21.15.90:6802/3635087609,v1:172.21.15.90:6803/3635087609] boot 2024-04-02T14:26:28.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:26:28 smithi177 ceph-mon[36431]: osdmap e8: 1 total, 1 up, 1 in 2024-04-02T14:26:28.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:26:28 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-02T14:26:29.511 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:26:29 smithi092 ceph-mon[36497]: osdmap e9: 1 total, 1 up, 1 in 2024-04-02T14:26:29.511 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:26:29 smithi092 ceph-mon[36497]: pgmap v56: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-02T14:26:29.543 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:26:29.537+0000 7fb64c9c0700 1 -- 172.21.15.92:0/2803162676 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb648101530 msgr2=0x7fb648101950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:26:29.543 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:26:29.537+0000 7fb64c9c0700 1 --2- 172.21.15.92:0/2803162676 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb648101530 0x7fb648101950 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fb6380099c0 tx=0x7fb638009cd0 comp rx=0 tx=0).stop 2024-04-02T14:26:29.543 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:26:29.537+0000 7fb64c9c0700 1 -- 172.21.15.92:0/2803162676 shutdown_connections 2024-04-02T14:26:29.544 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:26:29.537+0000 7fb64c9c0700 1 --2- 172.21.15.92:0/2803162676 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb6481030d0 0x7fb648109980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:26:29.544 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:26:29.537+0000 7fb64c9c0700 1 --2- 172.21.15.92:0/2803162676 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb648101530 0x7fb648101950 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:26:29.544 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:26:29.537+0000 7fb64c9c0700 1 --2- 172.21.15.92:0/2803162676 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb648102730 0x7fb648102b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:26:29.544 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:26:29.537+0000 7fb64c9c0700 1 -- 172.21.15.92:0/2803162676 >> 172.21.15.92:0/2803162676 conn(0x7fb6480fcab0 msgr2=0x7fb6480fef10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:26:29.544 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:26:29.537+0000 7fb64c9c0700 1 -- 172.21.15.92:0/2803162676 shutdown_connections 2024-04-02T14:26:29.544 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:26:29.537+0000 7fb64c9c0700 1 -- 172.21.15.92:0/2803162676 wait complete. 2024-04-02T14:26:29.544 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:26:29.538+0000 7fb64c9c0700 1 Processor -- start 2024-04-02T14:26:29.544 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:26:29.538+0000 7fb64c9c0700 1 -- start start 2024-04-02T14:26:29.544 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:26:29.538+0000 7fb64c9c0700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb648101530 0x7fb6481989a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:26:29.544 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:26:29.538+0000 7fb64c9c0700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb648102730 0x7fb648198ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:26:29.545 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:26:29.538+0000 7fb64c9c0700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb6481030d0 0x7fb64819ff60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:26:29.545 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:26:29.538+0000 7fb64c9c0700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fb64810d1f0 con 0x7fb648101530 2024-04-02T14:26:29.545 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:26:29.538+0000 7fb64c9c0700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fb64810d070 con 0x7fb6481030d0 2024-04-02T14:26:29.545 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:26:29.538+0000 7fb64c9c0700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fb64810d370 con 0x7fb648102730 2024-04-02T14:26:29.545 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:26:29.538+0000 7fb646d9d700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb6481030d0 0x7fb64819ff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:26:29.545 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:26:29.538+0000 7fb646d9d700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb6481030d0 0x7fb64819ff60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:45208/0 (socket says 172.21.15.92:45208) 2024-04-02T14:26:29.545 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:26:29.538+0000 7fb646d9d700 1 -- 172.21.15.92:0/3480141774 learned_addr learned my addr 172.21.15.92:0/3480141774 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-04-02T14:26:29.545 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:26:29.538+0000 7fb645d9b700 1 --2- 172.21.15.92:0/3480141774 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb648102730 0x7fb648198ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:26:29.545 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:26:29.539+0000 7fb646d9d700 1 -- 172.21.15.92:0/3480141774 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb648102730 msgr2=0x7fb648198ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:26:29.545 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:26:29.539+0000 7fb646d9d700 1 --2- 172.21.15.92:0/3480141774 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb648102730 0x7fb648198ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:26:29.546 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:26:29.539+0000 7fb646d9d700 1 -- 172.21.15.92:0/3480141774 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb648101530 msgr2=0x7fb6481989a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:26:29.546 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:26:29.539+0000 7fb646d9d700 1 --2- 172.21.15.92:0/3480141774 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb648101530 0x7fb6481989a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:26:29.546 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:26:29.539+0000 7fb646d9d700 1 -- 172.21.15.92:0/3480141774 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6481a0600 con 0x7fb6481030d0 2024-04-02T14:26:29.546 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:26:29.539+0000 7fb646d9d700 1 --2- 172.21.15.92:0/3480141774 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb6481030d0 0x7fb64819ff60 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fb64000b970 tx=0x7fb64000bc80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:26:29.546 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:26:29.539+0000 7fb6377fe700 1 -- 172.21.15.92:0/3480141774 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb640019070 con 0x7fb6481030d0 2024-04-02T14:26:29.546 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:26:29.539+0000 7fb6377fe700 1 -- 172.21.15.92:0/3480141774 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb64000ce10 con 0x7fb6481030d0 2024-04-02T14:26:29.546 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:26:29.539+0000 7fb64c9c0700 1 -- 172.21.15.92:0/3480141774 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6481a08f0 con 0x7fb6481030d0 2024-04-02T14:26:29.546 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:26:29.539+0000 7fb6377fe700 1 -- 172.21.15.92:0/3480141774 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6400073d0 con 0x7fb6481030d0 2024-04-02T14:26:29.546 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:26:29.539+0000 7fb64c9c0700 1 -- 172.21.15.92:0/3480141774 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6481a0eb0 con 0x7fb6481030d0 2024-04-02T14:26:29.547 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:26:29.540+0000 7fb64c9c0700 1 -- 172.21.15.92:0/3480141774 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb6480692b0 con 0x7fb6481030d0 2024-04-02T14:26:29.547 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:26:29.541+0000 7fb6377fe700 1 -- 172.21.15.92:0/3480141774 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 15) ==== 138590+0+0 (secure 0 0 0) 0x7fb64001c070 con 0x7fb6481030d0 2024-04-02T14:26:29.547 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:26:29.541+0000 7fb6377fe700 1 --2- 172.21.15.92:0/3480141774 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fb6300a3bb0 0x7fb6300a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:26:29.547 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:26:29.542+0000 7fb6377fe700 1 -- 172.21.15.92:0/3480141774 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(9..9 src has 1..9) ==== 1767+0+0 (secure 0 0 0) 0x7fb6400cdc30 con 0x7fb6481030d0 2024-04-02T14:26:29.548 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:26:29.542+0000 7fb64659c700 1 --2- 172.21.15.92:0/3480141774 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fb6300a3bb0 0x7fb6300a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:26:29.548 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:26:29.543+0000 7fb64659c700 1 --2- 172.21.15.92:0/3480141774 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fb6300a3bb0 0x7fb6300a6070 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fb638003890 tx=0x7fb638000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:26:29.548 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:26:29.546+0000 7fb6377fe700 1 -- 172.21.15.92:0/3480141774 <== mon.2 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7fb64009c080 con 0x7fb6481030d0 2024-04-02T14:26:29.588 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:26:29 smithi090 ceph-mon[31674]: osdmap e9: 1 total, 1 up, 1 in 2024-04-02T14:26:29.588 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:26:29 smithi090 ceph-mon[31674]: pgmap v56: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-02T14:26:29.686 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:26:29 smithi177 ceph-mon[36431]: osdmap e9: 1 total, 1 up, 1 in 2024-04-02T14:26:29.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:26:29 smithi177 ceph-mon[36431]: pgmap v56: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-02T14:26:29.755 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:26:29.753+0000 7fb64c9c0700 1 -- 172.21.15.92:0/3480141774 --> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi092:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7fb64806b420 con 0x7fb6300a3bb0 2024-04-02T14:26:31.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:26:30 smithi092 ceph-mon[36497]: Detected new or changed devices on smithi090 2024-04-02T14:26:31.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:26:30 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:26:31.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:26:30 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:26:31.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:26:30 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-02T14:26:31.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:26:30 smithi092 ceph-mon[36497]: Adjusting osd_memory_target on smithi090 to 17091M 2024-04-02T14:26:31.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:26:30 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:26:31.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:26:30 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:26:31.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:26:30 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T14:26:31.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:26:30 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:26:31.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:26:30 smithi092 ceph-mon[36497]: from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi092:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T14:26:31.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:26:30 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-02T14:26:31.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:26:30 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-02T14:26:31.112 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:26:30 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:26:31.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:26:30 smithi177 ceph-mon[36431]: Detected new or changed devices on smithi090 2024-04-02T14:26:31.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:26:30 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:26:31.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:26:30 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:26:31.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:26:30 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-02T14:26:31.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:26:30 smithi177 ceph-mon[36431]: Adjusting osd_memory_target on smithi090 to 17091M 2024-04-02T14:26:31.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:26:30 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:26:31.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:26:30 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:26:31.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:26:30 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T14:26:31.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:26:30 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:26:31.188 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:26:30 smithi177 ceph-mon[36431]: from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi092:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T14:26:31.188 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:26:30 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-02T14:26:31.188 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:26:30 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-02T14:26:31.188 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:26:30 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:26:31.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:26:30 smithi090 ceph-mon[31674]: Detected new or changed devices on smithi090 2024-04-02T14:26:31.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:26:30 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:26:31.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:26:30 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:26:31.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:26:30 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-02T14:26:31.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:26:30 smithi090 ceph-mon[31674]: Adjusting osd_memory_target on smithi090 to 17091M 2024-04-02T14:26:31.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:26:30 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:26:31.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:26:30 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:26:31.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:26:30 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T14:26:31.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:26:30 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:26:31.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:26:30 smithi090 ceph-mon[31674]: from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi092:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T14:26:31.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:26:30 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-02T14:26:31.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:26:30 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-02T14:26:31.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:26:30 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:26:32.110 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:26:31 smithi092 ceph-mon[36497]: pgmap v57: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-02T14:26:32.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:26:31 smithi177 ceph-mon[36431]: pgmap v57: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-02T14:26:32.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:26:31 smithi090 ceph-mon[31674]: pgmap v57: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-02T14:26:32.861 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:26:32 smithi092 ceph-mon[36497]: from='client.? 172.21.15.92:0/961030963' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "124139cd-5793-463e-9607-9d9c10164183"}]: dispatch 2024-04-02T14:26:32.861 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:26:32 smithi092 ceph-mon[36497]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "124139cd-5793-463e-9607-9d9c10164183"}]: dispatch 2024-04-02T14:26:32.861 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:26:32 smithi092 ceph-mon[36497]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "124139cd-5793-463e-9607-9d9c10164183"}]': finished 2024-04-02T14:26:32.861 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:26:32 smithi092 ceph-mon[36497]: osdmap e10: 2 total, 1 up, 2 in 2024-04-02T14:26:32.861 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:26:32 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-02T14:26:33.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:26:32 smithi177 ceph-mon[36431]: from='client.? 172.21.15.92:0/961030963' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "124139cd-5793-463e-9607-9d9c10164183"}]: dispatch 2024-04-02T14:26:33.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:26:32 smithi177 ceph-mon[36431]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "124139cd-5793-463e-9607-9d9c10164183"}]: dispatch 2024-04-02T14:26:33.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:26:32 smithi177 ceph-mon[36431]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "124139cd-5793-463e-9607-9d9c10164183"}]': finished 2024-04-02T14:26:33.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:26:32 smithi177 ceph-mon[36431]: osdmap e10: 2 total, 1 up, 2 in 2024-04-02T14:26:33.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:26:32 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-02T14:26:33.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:26:32 smithi090 ceph-mon[31674]: from='client.? 172.21.15.92:0/961030963' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "124139cd-5793-463e-9607-9d9c10164183"}]: dispatch 2024-04-02T14:26:33.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:26:32 smithi090 ceph-mon[31674]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "124139cd-5793-463e-9607-9d9c10164183"}]: dispatch 2024-04-02T14:26:33.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:26:32 smithi090 ceph-mon[31674]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "124139cd-5793-463e-9607-9d9c10164183"}]': finished 2024-04-02T14:26:33.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:26:32 smithi090 ceph-mon[31674]: osdmap e10: 2 total, 1 up, 2 in 2024-04-02T14:26:33.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:26:32 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-02T14:26:34.110 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:26:33 smithi092 ceph-mon[36497]: pgmap v59: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-02T14:26:34.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:26:33 smithi092 ceph-mon[36497]: from='client.? 172.21.15.92:0/2977962627' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-02T14:26:34.124 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:26:33 smithi090 ceph-mon[31674]: pgmap v59: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-02T14:26:34.124 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:26:33 smithi090 ceph-mon[31674]: from='client.? 172.21.15.92:0/2977962627' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-02T14:26:34.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:26:33 smithi177 ceph-mon[36431]: pgmap v59: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-02T14:26:34.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:26:33 smithi177 ceph-mon[36431]: from='client.? 172.21.15.92:0/2977962627' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-02T14:26:36.110 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:26:35 smithi092 ceph-mon[36497]: pgmap v60: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-02T14:26:36.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:26:35 smithi177 ceph-mon[36431]: pgmap v60: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-02T14:26:36.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:26:35 smithi090 ceph-mon[31674]: pgmap v60: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-02T14:26:37.864 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:26:37 smithi090 ceph-mon[31674]: pgmap v61: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-02T14:26:37.881 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:26:37 smithi092 ceph-mon[36497]: pgmap v61: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-02T14:26:38.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:26:37 smithi177 ceph-mon[36431]: pgmap v61: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-02T14:26:40.054 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:26:39 smithi092 ceph-mon[36497]: pgmap v62: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-02T14:26:40.054 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:26:39 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-02T14:26:40.055 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:26:39 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:26:40.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:26:39 smithi177 ceph-mon[36431]: pgmap v62: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-02T14:26:40.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:26:39 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-02T14:26:40.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:26:39 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:26:40.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:26:39 smithi090 ceph-mon[31674]: pgmap v62: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-02T14:26:40.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:26:39 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-02T14:26:40.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:26:39 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:26:40.827 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:26:40 smithi092 ceph-mon[36497]: Deploying daemon osd.1 on smithi092 2024-04-02T14:26:41.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:26:40 smithi177 ceph-mon[36431]: Deploying daemon osd.1 on smithi092 2024-04-02T14:26:41.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:26:40 smithi090 ceph-mon[31674]: Deploying daemon osd.1 on smithi092 2024-04-02T14:26:42.030 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:26:41 smithi092 ceph-mon[36497]: pgmap v63: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-02T14:26:42.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:26:41 smithi177 ceph-mon[36431]: pgmap v63: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-02T14:26:42.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:26:41 smithi090 ceph-mon[31674]: pgmap v63: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-02T14:26:43.876 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:26:43 smithi092 ceph-mon[36497]: pgmap v64: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-02T14:26:44.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:26:43 smithi177 ceph-mon[36431]: pgmap v64: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-02T14:26:44.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:26:43 smithi090 ceph-mon[31674]: pgmap v64: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-02T14:26:45.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:26:45 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:26:45.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:26:45 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:26:45.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:26:45 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T14:26:45.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:26:45 smithi177 ceph-mon[36431]: pgmap v65: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-02T14:26:45.689 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:26:45 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:26:45.689 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:26:45 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:26:45.689 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:26:45 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T14:26:45.689 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:26:45 smithi090 ceph-mon[31674]: pgmap v65: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-02T14:26:45.819 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:26:45 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:26:45.819 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:26:45 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:26:45.819 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:26:45 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T14:26:45.819 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:26:45 smithi092 ceph-mon[36497]: pgmap v65: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-02T14:26:47.962 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:26:47 smithi092 ceph-mon[36497]: pgmap v66: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-02T14:26:47.962 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:26:47 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:26:47.962 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:26:47 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:26:47.962 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:26:47 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:26:47.962 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:26:47 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T14:26:47.962 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:26:47 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:26:48.041 INFO:teuthology.orchestra.run.smithi092.stdout:Created osd(s) 1 on host 'smithi092' 2024-04-02T14:26:48.041 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:26:48.038+0000 7fb6377fe700 1 -- 172.21.15.92:0/3480141774 <== mgr.14152 v2:172.21.15.90:6800/4266054817 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fb64806b420 con 0x7fb6300a3bb0 2024-04-02T14:26:48.042 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:26:48.040+0000 7fb64c9c0700 1 -- 172.21.15.92:0/3480141774 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fb6300a3bb0 msgr2=0x7fb6300a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:26:48.042 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:26:48.040+0000 7fb64c9c0700 1 --2- 172.21.15.92:0/3480141774 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fb6300a3bb0 0x7fb6300a6070 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fb638003890 tx=0x7fb638000bc0 comp rx=0 tx=0).stop 2024-04-02T14:26:48.042 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:26:48.040+0000 7fb64c9c0700 1 -- 172.21.15.92:0/3480141774 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb6481030d0 msgr2=0x7fb64819ff60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:26:48.043 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:26:48.040+0000 7fb64c9c0700 1 --2- 172.21.15.92:0/3480141774 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb6481030d0 0x7fb64819ff60 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fb64000b970 tx=0x7fb64000bc80 comp rx=0 tx=0).stop 2024-04-02T14:26:48.043 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:26:48.040+0000 7fb64c9c0700 1 -- 172.21.15.92:0/3480141774 shutdown_connections 2024-04-02T14:26:48.043 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:26:48.040+0000 7fb64c9c0700 1 --2- 172.21.15.92:0/3480141774 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fb6300a3bb0 0x7fb6300a6070 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:26:48.043 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:26:48.040+0000 7fb64c9c0700 1 --2- 172.21.15.92:0/3480141774 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb648101530 0x7fb6481989a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:26:48.043 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:26:48.041+0000 7fb64c9c0700 1 --2- 172.21.15.92:0/3480141774 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb6481030d0 0x7fb64819ff60 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:26:48.043 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:26:48.041+0000 7fb64c9c0700 1 --2- 172.21.15.92:0/3480141774 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb648102730 0x7fb648198ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:26:48.043 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:26:48.041+0000 7fb64c9c0700 1 -- 172.21.15.92:0/3480141774 >> 172.21.15.92:0/3480141774 conn(0x7fb6480fcab0 msgr2=0x7fb64806ad10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:26:48.043 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:26:48.041+0000 7fb64c9c0700 1 -- 172.21.15.92:0/3480141774 shutdown_connections 2024-04-02T14:26:48.043 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:26:48.041+0000 7fb64c9c0700 1 -- 172.21.15.92:0/3480141774 wait complete. 2024-04-02T14:26:48.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:26:47 smithi177 ceph-mon[36431]: pgmap v66: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-02T14:26:48.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:26:47 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:26:48.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:26:47 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:26:48.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:26:47 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:26:48.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:26:47 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T14:26:48.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:26:47 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:26:48.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:26:47 smithi090 ceph-mon[31674]: pgmap v66: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-02T14:26:48.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:26:47 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:26:48.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:26:47 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:26:48.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:26:47 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:26:48.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:26:47 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T14:26:48.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:26:47 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:26:48.664 DEBUG:teuthology.orchestra.run.smithi092:osd.1> sudo journalctl -f -n 0 -u ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4@osd.1.service 2024-04-02T14:26:48.666 INFO:tasks.cephadm:Deploying osd.2 on smithi177 with /dev/vg_nvme/lv_4... 2024-04-02T14:26:48.666 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:191d963ac9de7322da8aff20b567b3fa458e2ef6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6449448a-f0fc-11ee-b647-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-02T14:26:48.910 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/mon.c/config 2024-04-02T14:26:49.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:26:49 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:26:49.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:26:49 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:26:49.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:26:49 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T14:26:49.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:26:49 smithi090 ceph-mon[31674]: from='osd.1 [v2:172.21.15.92:6800/1994536059,v1:172.21.15.92:6801/1994536059]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-02T14:26:49.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:26:49 smithi090 ceph-mon[31674]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-02T14:26:49.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:26:49 smithi090 ceph-mon[31674]: pgmap v67: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-02T14:26:49.359 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:26:49 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:26:49.359 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:26:49 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:26:49.359 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:26:49 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T14:26:49.359 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:26:49 smithi177 ceph-mon[36431]: from='osd.1 [v2:172.21.15.92:6800/1994536059,v1:172.21.15.92:6801/1994536059]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-02T14:26:49.359 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:26:49 smithi177 ceph-mon[36431]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-02T14:26:49.359 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:26:49 smithi177 ceph-mon[36431]: pgmap v67: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-02T14:26:49.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:26:49 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:26:49.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:26:49 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:26:49.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:26:49 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T14:26:49.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:26:49 smithi092 ceph-mon[36497]: from='osd.1 [v2:172.21.15.92:6800/1994536059,v1:172.21.15.92:6801/1994536059]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-02T14:26:49.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:26:49 smithi092 ceph-mon[36497]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-02T14:26:49.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:26:49 smithi092 ceph-mon[36497]: pgmap v67: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-02T14:26:50.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:26:50 smithi092 ceph-mon[36497]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-02T14:26:50.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:26:50 smithi092 ceph-mon[36497]: from='osd.1 [v2:172.21.15.92:6800/1994536059,v1:172.21.15.92:6801/1994536059]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]: dispatch 2024-04-02T14:26:50.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:26:50 smithi092 ceph-mon[36497]: osdmap e11: 2 total, 1 up, 2 in 2024-04-02T14:26:50.112 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:26:50 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-02T14:26:50.112 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:26:50 smithi092 ceph-mon[36497]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]: dispatch 2024-04-02T14:26:50.282 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:26:50 smithi177 ceph-mon[36431]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-02T14:26:50.283 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:26:50 smithi177 ceph-mon[36431]: from='osd.1 [v2:172.21.15.92:6800/1994536059,v1:172.21.15.92:6801/1994536059]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]: dispatch 2024-04-02T14:26:50.283 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:26:50 smithi177 ceph-mon[36431]: osdmap e11: 2 total, 1 up, 2 in 2024-04-02T14:26:50.283 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:26:50 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-02T14:26:50.283 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:26:50 smithi177 ceph-mon[36431]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]: dispatch 2024-04-02T14:26:50.439 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:26:50 smithi090 ceph-mon[31674]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-02T14:26:50.439 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:26:50 smithi090 ceph-mon[31674]: from='osd.1 [v2:172.21.15.92:6800/1994536059,v1:172.21.15.92:6801/1994536059]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]: dispatch 2024-04-02T14:26:50.439 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:26:50 smithi090 ceph-mon[31674]: osdmap e11: 2 total, 1 up, 2 in 2024-04-02T14:26:50.439 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:26:50 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-02T14:26:50.439 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:26:50 smithi090 ceph-mon[31674]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]: dispatch 2024-04-02T14:26:50.855 INFO:teuthology.orchestra.run.smithi177.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-02T14:26:50.855 INFO:teuthology.orchestra.run.smithi177.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-02T14:26:50.855 INFO:teuthology.orchestra.run.smithi177.stderr: stderr: 10+0 records in 2024-04-02T14:26:50.855 INFO:teuthology.orchestra.run.smithi177.stderr:10+0 records out 2024-04-02T14:26:50.856 INFO:teuthology.orchestra.run.smithi177.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0121111 s, 866 MB/s 2024-04-02T14:26:50.856 INFO:teuthology.orchestra.run.smithi177.stderr:--> Zapping successful for: 2024-04-02T14:26:51.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:26:51 smithi092 ceph-mon[36497]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]': finished 2024-04-02T14:26:51.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:26:51 smithi092 ceph-mon[36497]: osdmap e12: 2 total, 1 up, 2 in 2024-04-02T14:26:51.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:26:51 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-02T14:26:51.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:26:51 smithi092 ceph-mon[36497]: pgmap v70: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-02T14:26:51.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:26:51 smithi092 ceph-mon[36497]: from='osd.1 ' entity='osd.1' 2024-04-02T14:26:51.111 INFO:journalctl@ceph.osd.1.smithi092.stdout:Apr 02 14:26:50 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-1[41091]: 2024-04-02T14:26:50.964+0000 7fa1ab02b700 -1 osd.1 0 waiting for initial osdmap 2024-04-02T14:26:51.401 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:26:51 smithi177 ceph-mon[36431]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]': finished 2024-04-02T14:26:51.401 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:26:51 smithi177 ceph-mon[36431]: osdmap e12: 2 total, 1 up, 2 in 2024-04-02T14:26:51.401 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:26:51 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-02T14:26:51.401 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:26:51 smithi177 ceph-mon[36431]: pgmap v70: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-02T14:26:51.401 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:26:51 smithi177 ceph-mon[36431]: from='osd.1 ' entity='osd.1' 2024-04-02T14:26:51.427 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:191d963ac9de7322da8aff20b567b3fa458e2ef6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6449448a-f0fc-11ee-b647-cb9ed24678a4 -- ceph orch daemon add osd smithi177:vg_nvme/lv_4 2024-04-02T14:26:51.438 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:26:51 smithi090 ceph-mon[31674]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]': finished 2024-04-02T14:26:51.438 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:26:51 smithi090 ceph-mon[31674]: osdmap e12: 2 total, 1 up, 2 in 2024-04-02T14:26:51.439 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:26:51 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-02T14:26:51.439 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:26:51 smithi090 ceph-mon[31674]: pgmap v70: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-02T14:26:51.439 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:26:51 smithi090 ceph-mon[31674]: from='osd.1 ' entity='osd.1' 2024-04-02T14:26:51.453 INFO:journalctl@ceph.osd.1.smithi092.stdout:Apr 02 14:26:51 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-1[41091]: 2024-04-02T14:26:51.141+0000 7fa1a2e16700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-02T14:26:51.674 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/mon.c/config 2024-04-02T14:26:52.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:26:52 smithi092 ceph-mon[36497]: purged_snaps scrub starts 2024-04-02T14:26:52.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:26:52 smithi092 ceph-mon[36497]: purged_snaps scrub ok 2024-04-02T14:26:52.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:26:52 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-02T14:26:52.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:26:52 smithi092 ceph-mon[36497]: Detected new or changed devices on smithi092 2024-04-02T14:26:52.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:26:52 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:26:52.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:26:52 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:26:52.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:26:52 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-02T14:26:52.362 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:26:52 smithi092 ceph-mon[36497]: Adjusting osd_memory_target on smithi092 to 17080M 2024-04-02T14:26:52.362 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:26:52 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:26:52.362 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:26:52 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:26:52.362 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:26:52 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T14:26:52.362 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:26:52 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:26:52.362 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:26:52 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-02T14:26:52.407 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:26:52 smithi177 ceph-mon[36431]: purged_snaps scrub starts 2024-04-02T14:26:52.407 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:26:52 smithi177 ceph-mon[36431]: purged_snaps scrub ok 2024-04-02T14:26:52.407 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:26:52 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-02T14:26:52.407 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:26:52 smithi177 ceph-mon[36431]: Detected new or changed devices on smithi092 2024-04-02T14:26:52.407 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:26:52 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:26:52.407 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:26:52 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:26:52.407 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:26:52 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-02T14:26:52.407 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:26:52 smithi177 ceph-mon[36431]: Adjusting osd_memory_target on smithi092 to 17080M 2024-04-02T14:26:52.408 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:26:52 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:26:52.408 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:26:52 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:26:52.408 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:26:52 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T14:26:52.408 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:26:52 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:26:52.408 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:26:52 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-02T14:26:52.439 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:26:52 smithi090 ceph-mon[31674]: purged_snaps scrub starts 2024-04-02T14:26:52.439 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:26:52 smithi090 ceph-mon[31674]: purged_snaps scrub ok 2024-04-02T14:26:52.439 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:26:52 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-02T14:26:52.439 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:26:52 smithi090 ceph-mon[31674]: Detected new or changed devices on smithi092 2024-04-02T14:26:52.439 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:26:52 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:26:52.439 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:26:52 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:26:52.439 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:26:52 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-02T14:26:52.439 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:26:52 smithi090 ceph-mon[31674]: Adjusting osd_memory_target on smithi092 to 17080M 2024-04-02T14:26:52.440 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:26:52 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:26:52.440 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:26:52 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:26:52.440 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:26:52 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T14:26:52.440 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:26:52 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:26:52.440 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:26:52 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-02T14:26:53.224 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:26:53.220+0000 7f756b8fa700 1 -- 172.21.15.177:0/1348134913 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f75641041c0 msgr2=0x7f756410aa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:26:53.224 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:26:53.220+0000 7f756b8fa700 1 --2- 172.21.15.177:0/1348134913 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f75641041c0 0x7f756410aa70 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f755c00b0a0 tx=0x7f755c00b3b0 comp rx=0 tx=0).stop 2024-04-02T14:26:53.224 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:26:53.221+0000 7f756b8fa700 1 -- 172.21.15.177:0/1348134913 shutdown_connections 2024-04-02T14:26:53.224 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:26:53.221+0000 7f756b8fa700 1 --2- 172.21.15.177:0/1348134913 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7564102620 0x7f7564102a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:26:53.224 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:26:53.221+0000 7f756b8fa700 1 --2- 172.21.15.177:0/1348134913 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f75641041c0 0x7f756410aa70 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:26:53.224 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:26:53.221+0000 7f756b8fa700 1 --2- 172.21.15.177:0/1348134913 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7564103820 0x7f7564103c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:26:53.225 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:26:53.221+0000 7f756b8fa700 1 -- 172.21.15.177:0/1348134913 >> 172.21.15.177:0/1348134913 conn(0x7f75640fdc00 msgr2=0x7f7564100020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:26:53.225 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:26:53.221+0000 7f756b8fa700 1 -- 172.21.15.177:0/1348134913 shutdown_connections 2024-04-02T14:26:53.225 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:26:53.221+0000 7f756b8fa700 1 -- 172.21.15.177:0/1348134913 wait complete. 2024-04-02T14:26:53.225 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:26:53.221+0000 7f756b8fa700 1 Processor -- start 2024-04-02T14:26:53.225 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:26:53.221+0000 7f756b8fa700 1 -- start start 2024-04-02T14:26:53.225 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:26:53.221+0000 7f756b8fa700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7564102620 0x7f75641987a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:26:53.225 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:26:53.221+0000 7f756b8fa700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7564103820 0x7f7564198ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:26:53.225 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:26:53.221+0000 7f756b8fa700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f75641041c0 0x7f756419fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:26:53.225 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:26:53.221+0000 7f756b8fa700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f756410e280 con 0x7f75641041c0 2024-04-02T14:26:53.225 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:26:53.221+0000 7f756b8fa700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f756410e100 con 0x7f7564103820 2024-04-02T14:26:53.226 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:26:53.221+0000 7f756b8fa700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f756410e400 con 0x7f7564102620 2024-04-02T14:26:53.226 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:26:53.221+0000 7f7569696700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7564102620 0x7f75641987a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:26:53.226 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:26:53.222+0000 7f7569696700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7564102620 0x7f75641987a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.177:38714/0 (socket says 172.21.15.177:38714) 2024-04-02T14:26:53.226 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:26:53.222+0000 7f7569696700 1 -- 172.21.15.177:0/3840833455 learned_addr learned my addr 172.21.15.177:0/3840833455 (peer_addr_for_me v2:172.21.15.177:0/0) 2024-04-02T14:26:53.226 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:26:53.222+0000 7f7568e95700 1 --2- 172.21.15.177:0/3840833455 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7564103820 0x7f7564198ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:26:53.226 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:26:53.222+0000 7f7569696700 1 -- 172.21.15.177:0/3840833455 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7564103820 msgr2=0x7f7564198ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:26:53.226 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:26:53.222+0000 7f7569696700 1 --2- 172.21.15.177:0/3840833455 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7564103820 0x7f7564198ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:26:53.226 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:26:53.222+0000 7f7569696700 1 -- 172.21.15.177:0/3840833455 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f75641041c0 msgr2=0x7f756419fd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:26:53.226 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:26:53.222+0000 7f7569696700 1 --2- 172.21.15.177:0/3840833455 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f75641041c0 0x7f756419fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:26:53.226 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:26:53.222+0000 7f7569696700 1 -- 172.21.15.177:0/3840833455 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75641a0400 con 0x7f7564102620 2024-04-02T14:26:53.226 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:26:53.222+0000 7f7568e95700 1 --2- 172.21.15.177:0/3840833455 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7564103820 0x7f7564198ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:26:53.227 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:26:53.222+0000 7f7569696700 1 --2- 172.21.15.177:0/3840833455 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7564102620 0x7f75641987a0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f756000b530 tx=0x7f756000b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:26:53.227 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:26:53.222+0000 7f755a7fc700 1 -- 172.21.15.177:0/3840833455 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7560015020 con 0x7f7564102620 2024-04-02T14:26:53.227 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:26:53.222+0000 7f755a7fc700 1 -- 172.21.15.177:0/3840833455 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7560011d10 con 0x7f7564102620 2024-04-02T14:26:53.227 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:26:53.222+0000 7f755a7fc700 1 -- 172.21.15.177:0/3840833455 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f756000f2e0 con 0x7f7564102620 2024-04-02T14:26:53.227 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:26:53.222+0000 7f756b8fa700 1 -- 172.21.15.177:0/3840833455 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75641a06f0 con 0x7f7564102620 2024-04-02T14:26:53.227 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:26:53.222+0000 7f756b8fa700 1 -- 172.21.15.177:0/3840833455 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75641a0d00 con 0x7f7564102620 2024-04-02T14:26:53.228 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:26:53.223+0000 7f756b8fa700 1 -- 172.21.15.177:0/3840833455 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f75640692b0 con 0x7f7564102620 2024-04-02T14:26:53.228 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:26:53.224+0000 7f755a7fc700 1 -- 172.21.15.177:0/3840833455 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 15) ==== 138590+0+0 (secure 0 0 0) 0x7f7560018020 con 0x7f7564102620 2024-04-02T14:26:53.228 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:26:53.225+0000 7f755a7fc700 1 --2- 172.21.15.177:0/3840833455 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f75500a3c30 0x7f75500a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:26:53.228 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:26:53.225+0000 7f755a7fc700 1 -- 172.21.15.177:0/3840833455 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(14..14 src has 1..14) ==== 2309+0+0 (secure 0 0 0) 0x7f75600ce850 con 0x7f7564102620 2024-04-02T14:26:53.228 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:26:53.226+0000 7f7568e95700 1 --2- 172.21.15.177:0/3840833455 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f75500a3c30 0x7f75500a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:26:53.229 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:26:53.226+0000 7f7568e95700 1 --2- 172.21.15.177:0/3840833455 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f75500a3c30 0x7f75500a60f0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f7554005fd0 tx=0x7f7554005e20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:26:53.230 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:26:53.229+0000 7f755a7fc700 1 -- 172.21.15.177:0/3840833455 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7f756001f020 con 0x7f7564102620 2024-04-02T14:26:53.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:26:53 smithi092 ceph-mon[36497]: osd.1 [v2:172.21.15.92:6800/1994536059,v1:172.21.15.92:6801/1994536059] boot 2024-04-02T14:26:53.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:26:53 smithi092 ceph-mon[36497]: osdmap e13: 2 total, 2 up, 2 in 2024-04-02T14:26:53.362 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:26:53 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-02T14:26:53.362 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:26:53 smithi092 ceph-mon[36497]: pgmap v72: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-02T14:26:53.434 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:26:53.433+0000 7f756b8fa700 1 -- 172.21.15.177:0/3840833455 --> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi177:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f7564062a30 con 0x7f75500a3c30 2024-04-02T14:26:53.435 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:26:53 smithi177 ceph-mon[36431]: osd.1 [v2:172.21.15.92:6800/1994536059,v1:172.21.15.92:6801/1994536059] boot 2024-04-02T14:26:53.435 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:26:53 smithi177 ceph-mon[36431]: osdmap e13: 2 total, 2 up, 2 in 2024-04-02T14:26:53.435 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:26:53 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-02T14:26:53.435 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:26:53 smithi177 ceph-mon[36431]: pgmap v72: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-02T14:26:53.439 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:26:53 smithi090 ceph-mon[31674]: osd.1 [v2:172.21.15.92:6800/1994536059,v1:172.21.15.92:6801/1994536059] boot 2024-04-02T14:26:53.439 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:26:53 smithi090 ceph-mon[31674]: osdmap e13: 2 total, 2 up, 2 in 2024-04-02T14:26:53.439 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:26:53 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-02T14:26:53.439 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:26:53 smithi090 ceph-mon[31674]: pgmap v72: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-02T14:26:54.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:26:54 smithi092 ceph-mon[36497]: osdmap e14: 2 total, 2 up, 2 in 2024-04-02T14:26:54.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:26:54 smithi092 ceph-mon[36497]: from='client.24166 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi177:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T14:26:54.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:26:54 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-02T14:26:54.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:26:54 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-02T14:26:54.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:26:54 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:26:54.371 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:26:54 smithi177 ceph-mon[36431]: osdmap e14: 2 total, 2 up, 2 in 2024-04-02T14:26:54.371 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:26:54 smithi177 ceph-mon[36431]: from='client.24166 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi177:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T14:26:54.371 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:26:54 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-02T14:26:54.371 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:26:54 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-02T14:26:54.371 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:26:54 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:26:54.438 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:26:54 smithi090 ceph-mon[31674]: osdmap e14: 2 total, 2 up, 2 in 2024-04-02T14:26:54.439 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:26:54 smithi090 ceph-mon[31674]: from='client.24166 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi177:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T14:26:54.439 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:26:54 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-02T14:26:54.439 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:26:54 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-02T14:26:54.439 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:26:54 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:26:55.265 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:26:55 smithi177 ceph-mon[36431]: pgmap v74: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-02T14:26:55.360 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:26:55 smithi092 ceph-mon[36497]: pgmap v74: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-02T14:26:55.438 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:26:55 smithi090 ceph-mon[31674]: pgmap v74: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-02T14:26:56.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:26:56 smithi092 ceph-mon[36497]: from='client.? 172.21.15.177:0/4240106203' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a61fbef2-9dca-4875-a814-020fff0db0e5"}]: dispatch 2024-04-02T14:26:56.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:26:56 smithi092 ceph-mon[36497]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a61fbef2-9dca-4875-a814-020fff0db0e5"}]: dispatch 2024-04-02T14:26:56.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:26:56 smithi092 ceph-mon[36497]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a61fbef2-9dca-4875-a814-020fff0db0e5"}]': finished 2024-04-02T14:26:56.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:26:56 smithi092 ceph-mon[36497]: osdmap e15: 3 total, 2 up, 3 in 2024-04-02T14:26:56.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:26:56 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-02T14:26:56.409 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:26:56 smithi177 ceph-mon[36431]: from='client.? 172.21.15.177:0/4240106203' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a61fbef2-9dca-4875-a814-020fff0db0e5"}]: dispatch 2024-04-02T14:26:56.409 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:26:56 smithi177 ceph-mon[36431]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a61fbef2-9dca-4875-a814-020fff0db0e5"}]: dispatch 2024-04-02T14:26:56.409 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:26:56 smithi177 ceph-mon[36431]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a61fbef2-9dca-4875-a814-020fff0db0e5"}]': finished 2024-04-02T14:26:56.409 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:26:56 smithi177 ceph-mon[36431]: osdmap e15: 3 total, 2 up, 3 in 2024-04-02T14:26:56.409 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:26:56 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-02T14:26:56.439 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:26:56 smithi090 ceph-mon[31674]: from='client.? 172.21.15.177:0/4240106203' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a61fbef2-9dca-4875-a814-020fff0db0e5"}]: dispatch 2024-04-02T14:26:56.439 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:26:56 smithi090 ceph-mon[31674]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a61fbef2-9dca-4875-a814-020fff0db0e5"}]: dispatch 2024-04-02T14:26:56.439 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:26:56 smithi090 ceph-mon[31674]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a61fbef2-9dca-4875-a814-020fff0db0e5"}]': finished 2024-04-02T14:26:56.439 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:26:56 smithi090 ceph-mon[31674]: osdmap e15: 3 total, 2 up, 3 in 2024-04-02T14:26:56.439 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:26:56 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-02T14:26:57.436 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:26:57 smithi177 ceph-mon[36431]: pgmap v76: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-02T14:26:57.436 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:26:57 smithi177 ceph-mon[36431]: from='client.? 172.21.15.177:0/1954665020' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-02T14:26:57.438 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:26:57 smithi090 ceph-mon[31674]: pgmap v76: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-02T14:26:57.438 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:26:57 smithi090 ceph-mon[31674]: from='client.? 172.21.15.177:0/1954665020' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-02T14:26:57.610 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:26:57 smithi092 ceph-mon[36497]: pgmap v76: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-02T14:26:57.611 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:26:57 smithi092 ceph-mon[36497]: from='client.? 172.21.15.177:0/1954665020' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-02T14:26:59.963 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:26:59 smithi177 ceph-mon[36431]: pgmap v77: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-02T14:27:00.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:26:59 smithi092 ceph-mon[36497]: pgmap v77: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-02T14:27:00.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:26:59 smithi090 ceph-mon[31674]: pgmap v77: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-02T14:27:01.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:27:01 smithi177 ceph-mon[36431]: pgmap v78: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-02T14:27:02.110 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:27:01 smithi092 ceph-mon[36497]: pgmap v78: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-02T14:27:02.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:27:01 smithi090 ceph-mon[31674]: pgmap v78: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-02T14:27:04.043 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:27:03 smithi177 ceph-mon[36431]: pgmap v79: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-02T14:27:04.043 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:27:03 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-02T14:27:04.043 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:27:03 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:27:04.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:27:03 smithi092 ceph-mon[36497]: pgmap v79: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-02T14:27:04.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:27:03 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-02T14:27:04.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:27:03 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:27:04.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:27:03 smithi090 ceph-mon[31674]: pgmap v79: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-02T14:27:04.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:27:03 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-02T14:27:04.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:27:03 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:27:04.964 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:27:04 smithi092 ceph-mon[36497]: Deploying daemon osd.2 on smithi177 2024-04-02T14:27:05.141 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:27:04 smithi177 ceph-mon[36431]: Deploying daemon osd.2 on smithi177 2024-04-02T14:27:05.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:27:04 smithi090 ceph-mon[31674]: Deploying daemon osd.2 on smithi177 2024-04-02T14:27:05.860 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:27:05 smithi092 ceph-mon[36497]: pgmap v80: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-02T14:27:05.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:27:05 smithi177 ceph-mon[36431]: pgmap v80: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-02T14:27:06.090 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:27:05 smithi090 ceph-mon[31674]: pgmap v80: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-02T14:27:07.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:27:07 smithi177 ceph-mon[36431]: pgmap v81: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-02T14:27:08.110 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:27:07 smithi092 ceph-mon[36497]: pgmap v81: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-02T14:27:08.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:27:07 smithi090 ceph-mon[31674]: pgmap v81: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-02T14:27:09.686 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:27:09 smithi177 ceph-mon[36431]: pgmap v82: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-02T14:27:09.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:27:09 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:27:09.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:27:09 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:27:09.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:27:09 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T14:27:09.938 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:27:09 smithi090 ceph-mon[31674]: pgmap v82: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-02T14:27:09.939 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:27:09 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:27:09.939 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:27:09 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:27:09.939 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:27:09 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T14:27:10.110 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:27:09 smithi092 ceph-mon[36497]: pgmap v82: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-02T14:27:10.113 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:27:09 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:27:10.113 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:27:09 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:27:10.113 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:27:09 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T14:27:11.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:27:11 smithi177 ceph-mon[36431]: pgmap v83: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-02T14:27:11.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:27:11 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:27:11.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:27:11 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:27:11.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:27:11 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:27:11.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:27:11 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T14:27:11.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:27:11 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:27:11.938 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:27:11 smithi090 ceph-mon[31674]: pgmap v83: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-02T14:27:11.938 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:27:11 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:27:11.938 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:27:11 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:27:11.939 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:27:11 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:27:11.939 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:27:11 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T14:27:11.939 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:27:11 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:27:12.110 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:27:11 smithi092 ceph-mon[36497]: pgmap v83: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-02T14:27:12.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:27:11 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:27:12.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:27:11 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:27:12.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:27:11 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:27:12.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:27:11 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T14:27:12.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:27:11 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:27:12.370 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:27:12.369+0000 7f755a7fc700 1 -- 172.21.15.177:0/3840833455 <== mgr.14152 v2:172.21.15.90:6800/4266054817 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f7564062a30 con 0x7f75500a3c30 2024-04-02T14:27:12.371 INFO:teuthology.orchestra.run.smithi177.stdout:Created osd(s) 2 on host 'smithi177' 2024-04-02T14:27:12.373 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:27:12.371+0000 7f756b8fa700 1 -- 172.21.15.177:0/3840833455 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f75500a3c30 msgr2=0x7f75500a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:27:12.373 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:27:12.371+0000 7f756b8fa700 1 --2- 172.21.15.177:0/3840833455 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f75500a3c30 0x7f75500a60f0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f7554005fd0 tx=0x7f7554005e20 comp rx=0 tx=0).stop 2024-04-02T14:27:12.373 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:27:12.371+0000 7f756b8fa700 1 -- 172.21.15.177:0/3840833455 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7564102620 msgr2=0x7f75641987a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:27:12.373 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:27:12.371+0000 7f756b8fa700 1 --2- 172.21.15.177:0/3840833455 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7564102620 0x7f75641987a0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f756000b530 tx=0x7f756000b8f0 comp rx=0 tx=0).stop 2024-04-02T14:27:12.373 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:27:12.371+0000 7f756b8fa700 1 -- 172.21.15.177:0/3840833455 shutdown_connections 2024-04-02T14:27:12.373 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:27:12.371+0000 7f756b8fa700 1 --2- 172.21.15.177:0/3840833455 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f75500a3c30 0x7f75500a60f0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:27:12.373 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:27:12.371+0000 7f756b8fa700 1 --2- 172.21.15.177:0/3840833455 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f75641041c0 0x7f756419fd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:27:12.373 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:27:12.371+0000 7f756b8fa700 1 --2- 172.21.15.177:0/3840833455 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7564102620 0x7f75641987a0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:27:12.373 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:27:12.371+0000 7f756b8fa700 1 --2- 172.21.15.177:0/3840833455 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7564103820 0x7f7564198ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:27:12.373 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:27:12.371+0000 7f756b8fa700 1 -- 172.21.15.177:0/3840833455 >> 172.21.15.177:0/3840833455 conn(0x7f75640fdc00 msgr2=0x7f7564106ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:27:12.373 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:27:12.371+0000 7f756b8fa700 1 -- 172.21.15.177:0/3840833455 shutdown_connections 2024-04-02T14:27:12.374 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:27:12.371+0000 7f756b8fa700 1 -- 172.21.15.177:0/3840833455 wait complete. 2024-04-02T14:27:12.943 DEBUG:teuthology.orchestra.run.smithi177:osd.2> sudo journalctl -f -n 0 -u ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4@osd.2.service 2024-04-02T14:27:12.945 INFO:tasks.cephadm:Deploying osd.3 on smithi090 with /dev/vg_nvme/lv_3... 2024-04-02T14:27:12.945 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:191d963ac9de7322da8aff20b567b3fa458e2ef6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6449448a-f0fc-11ee-b647-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-02T14:27:13.376 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T14:27:13.574 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:27:13 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:27:13.574 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:27:13 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T14:27:13.575 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:27:13 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:27:13.575 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:27:13 smithi090 ceph-mon[31674]: from='osd.2 [v2:172.21.15.177:6800/1358472800,v1:172.21.15.177:6801/1358472800]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-02T14:27:13.575 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:27:13 smithi090 ceph-mon[31674]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-02T14:27:13.575 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:27:13 smithi090 ceph-mon[31674]: pgmap v84: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-02T14:27:13.610 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:27:13 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:27:13.611 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:27:13 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T14:27:13.611 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:27:13 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:27:13.611 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:27:13 smithi092 ceph-mon[36497]: from='osd.2 [v2:172.21.15.177:6800/1358472800,v1:172.21.15.177:6801/1358472800]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-02T14:27:13.611 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:27:13 smithi092 ceph-mon[36497]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-02T14:27:13.611 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:27:13 smithi092 ceph-mon[36497]: pgmap v84: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-02T14:27:13.686 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:27:13 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:27:13.686 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:27:13 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T14:27:13.686 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:27:13 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:27:13.686 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:27:13 smithi177 ceph-mon[36431]: from='osd.2 [v2:172.21.15.177:6800/1358472800,v1:172.21.15.177:6801/1358472800]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-02T14:27:13.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:27:13 smithi177 ceph-mon[36431]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-02T14:27:13.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:27:13 smithi177 ceph-mon[36431]: pgmap v84: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-02T14:27:14.610 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:27:14 smithi092 ceph-mon[36497]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-02T14:27:14.611 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:27:14 smithi092 ceph-mon[36497]: from='osd.2 [v2:172.21.15.177:6800/1358472800,v1:172.21.15.177:6801/1358472800]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-04-02T14:27:14.611 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:27:14 smithi092 ceph-mon[36497]: osdmap e16: 3 total, 2 up, 3 in 2024-04-02T14:27:14.611 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:27:14 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-02T14:27:14.611 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:27:14 smithi092 ceph-mon[36497]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-04-02T14:27:14.624 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:27:14 smithi090 ceph-mon[31674]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-02T14:27:14.624 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:27:14 smithi090 ceph-mon[31674]: from='osd.2 [v2:172.21.15.177:6800/1358472800,v1:172.21.15.177:6801/1358472800]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-04-02T14:27:14.624 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:27:14 smithi090 ceph-mon[31674]: osdmap e16: 3 total, 2 up, 3 in 2024-04-02T14:27:14.624 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:27:14 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-02T14:27:14.624 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:27:14 smithi090 ceph-mon[31674]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-04-02T14:27:14.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:27:14 smithi177 ceph-mon[36431]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-02T14:27:14.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:27:14 smithi177 ceph-mon[36431]: from='osd.2 [v2:172.21.15.177:6800/1358472800,v1:172.21.15.177:6801/1358472800]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-04-02T14:27:14.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:27:14 smithi177 ceph-mon[36431]: osdmap e16: 3 total, 2 up, 3 in 2024-04-02T14:27:14.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:27:14 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-02T14:27:14.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:27:14 smithi177 ceph-mon[36431]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-04-02T14:27:15.611 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:27:15 smithi092 ceph-mon[36497]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]': finished 2024-04-02T14:27:15.611 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:27:15 smithi092 ceph-mon[36497]: osdmap e17: 3 total, 2 up, 3 in 2024-04-02T14:27:15.611 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:27:15 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-02T14:27:15.611 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:27:15 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-02T14:27:15.611 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:27:15 smithi092 ceph-mon[36497]: pgmap v87: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-02T14:27:15.611 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:27:15 smithi092 ceph-mon[36497]: from='osd.2 ' entity='osd.2' 2024-04-02T14:27:15.611 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:27:15 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-02T14:27:15.629 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:27:15 smithi177 ceph-mon[36431]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]': finished 2024-04-02T14:27:15.630 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:27:15 smithi177 ceph-mon[36431]: osdmap e17: 3 total, 2 up, 3 in 2024-04-02T14:27:15.630 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:27:15 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-02T14:27:15.630 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:27:15 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-02T14:27:15.630 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:27:15 smithi177 ceph-mon[36431]: pgmap v87: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-02T14:27:15.630 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:27:15 smithi177 ceph-mon[36431]: from='osd.2 ' entity='osd.2' 2024-04-02T14:27:15.630 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:27:15 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-02T14:27:15.630 INFO:journalctl@ceph.osd.2.smithi177.stdout:Apr 02 14:27:15 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-2[41565]: 2024-04-02T14:27:15.226+0000 7f0c9636e700 -1 osd.2 0 waiting for initial osdmap 2024-04-02T14:27:15.630 INFO:journalctl@ceph.osd.2.smithi177.stdout:Apr 02 14:27:15 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-2[41565]: 2024-04-02T14:27:15.304+0000 7f0c8e159700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-02T14:27:15.689 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:27:15 smithi090 ceph-mon[31674]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]': finished 2024-04-02T14:27:15.689 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:27:15 smithi090 ceph-mon[31674]: osdmap e17: 3 total, 2 up, 3 in 2024-04-02T14:27:15.689 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:27:15 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-02T14:27:15.689 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:27:15 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-02T14:27:15.689 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:27:15 smithi090 ceph-mon[31674]: pgmap v87: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-02T14:27:15.689 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:27:15 smithi090 ceph-mon[31674]: from='osd.2 ' entity='osd.2' 2024-04-02T14:27:15.689 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:27:15 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-02T14:27:16.066 INFO:teuthology.orchestra.run.smithi090.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-02T14:27:16.067 INFO:teuthology.orchestra.run.smithi090.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-02T14:27:16.067 INFO:teuthology.orchestra.run.smithi090.stderr: stderr: 10+0 records in 2024-04-02T14:27:16.067 INFO:teuthology.orchestra.run.smithi090.stderr:10+0 records out 2024-04-02T14:27:16.067 INFO:teuthology.orchestra.run.smithi090.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0335476 s, 313 MB/s 2024-04-02T14:27:16.067 INFO:teuthology.orchestra.run.smithi090.stderr:--> Zapping successful for: 2024-04-02T14:27:16.611 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:27:16 smithi092 ceph-mon[36497]: purged_snaps scrub starts 2024-04-02T14:27:16.611 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:27:16 smithi092 ceph-mon[36497]: purged_snaps scrub ok 2024-04-02T14:27:16.611 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:27:16 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:27:16.611 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:27:16 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:27:16.611 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:27:16 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-02T14:27:16.611 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:27:16 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:27:16.611 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:27:16 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:27:16.611 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:27:16 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T14:27:16.611 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:27:16 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:27:16.611 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:27:16 smithi092 ceph-mon[36497]: osd.2 [v2:172.21.15.177:6800/1358472800,v1:172.21.15.177:6801/1358472800] boot 2024-04-02T14:27:16.611 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:27:16 smithi092 ceph-mon[36497]: osdmap e18: 3 total, 3 up, 3 in 2024-04-02T14:27:16.611 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:27:16 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-02T14:27:16.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:27:16 smithi177 ceph-mon[36431]: purged_snaps scrub starts 2024-04-02T14:27:16.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:27:16 smithi177 ceph-mon[36431]: purged_snaps scrub ok 2024-04-02T14:27:16.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:27:16 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:27:16.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:27:16 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:27:16.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:27:16 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-02T14:27:16.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:27:16 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:27:16.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:27:16 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:27:16.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:27:16 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T14:27:16.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:27:16 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:27:16.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:27:16 smithi177 ceph-mon[36431]: osd.2 [v2:172.21.15.177:6800/1358472800,v1:172.21.15.177:6801/1358472800] boot 2024-04-02T14:27:16.688 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:27:16 smithi177 ceph-mon[36431]: osdmap e18: 3 total, 3 up, 3 in 2024-04-02T14:27:16.688 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:27:16 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-02T14:27:16.689 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:27:16 smithi090 ceph-mon[31674]: purged_snaps scrub starts 2024-04-02T14:27:16.689 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:27:16 smithi090 ceph-mon[31674]: purged_snaps scrub ok 2024-04-02T14:27:16.689 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:27:16 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:27:16.689 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:27:16 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:27:16.689 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:27:16 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-02T14:27:16.689 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:27:16 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:27:16.690 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:27:16 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:27:16.690 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:27:16 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T14:27:16.690 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:27:16 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:27:16.690 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:27:16 smithi090 ceph-mon[31674]: osd.2 [v2:172.21.15.177:6800/1358472800,v1:172.21.15.177:6801/1358472800] boot 2024-04-02T14:27:16.690 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:27:16 smithi090 ceph-mon[31674]: osdmap e18: 3 total, 3 up, 3 in 2024-04-02T14:27:16.690 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:27:16 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-02T14:27:16.806 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:191d963ac9de7322da8aff20b567b3fa458e2ef6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6449448a-f0fc-11ee-b647-cb9ed24678a4 -- ceph orch daemon add osd smithi090:vg_nvme/lv_3 2024-04-02T14:27:17.256 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T14:27:17.662 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:27:17 smithi090 ceph-mon[31674]: Detected new or changed devices on smithi177 2024-04-02T14:27:17.662 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:27:17 smithi090 ceph-mon[31674]: Adjusting osd_memory_target on smithi177 to 17080M 2024-04-02T14:27:17.662 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:27:17 smithi090 ceph-mon[31674]: pgmap v89: 0 pgs: ; 0 B data, 96 MiB used, 268 GiB / 268 GiB avail 2024-04-02T14:27:17.662 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:27:17 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-02T14:27:17.686 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:27:17 smithi177 ceph-mon[36431]: Detected new or changed devices on smithi177 2024-04-02T14:27:17.686 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:27:17 smithi177 ceph-mon[36431]: Adjusting osd_memory_target on smithi177 to 17080M 2024-04-02T14:27:17.686 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:27:17 smithi177 ceph-mon[36431]: pgmap v89: 0 pgs: ; 0 B data, 96 MiB used, 268 GiB / 268 GiB avail 2024-04-02T14:27:17.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:27:17 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-02T14:27:17.860 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:27:17 smithi092 ceph-mon[36497]: Detected new or changed devices on smithi177 2024-04-02T14:27:17.861 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:27:17 smithi092 ceph-mon[36497]: Adjusting osd_memory_target on smithi177 to 17080M 2024-04-02T14:27:17.861 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:27:17 smithi092 ceph-mon[36497]: pgmap v89: 0 pgs: ; 0 B data, 96 MiB used, 268 GiB / 268 GiB avail 2024-04-02T14:27:17.861 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:27:17 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-02T14:27:18.686 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:27:18 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-02T14:27:18.686 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:27:18 smithi177 ceph-mon[36431]: osdmap e19: 3 total, 3 up, 3 in 2024-04-02T14:27:18.686 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:27:18 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-02T14:27:18.688 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:27:18 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-02T14:27:18.688 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:27:18 smithi090 ceph-mon[31674]: osdmap e19: 3 total, 3 up, 3 in 2024-04-02T14:27:18.688 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:27:18 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-02T14:27:18.861 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:27:18 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-02T14:27:18.861 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:27:18 smithi092 ceph-mon[36497]: osdmap e19: 3 total, 3 up, 3 in 2024-04-02T14:27:18.861 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:27:18 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-02T14:27:19.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:27:19.235+0000 7fe99a000700 1 -- 172.21.15.90:0/4149509118 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe994104100 msgr2=0x7fe99410a9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:27:19.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:27:19.235+0000 7fe99a000700 1 --2- 172.21.15.90:0/4149509118 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe994104100 0x7fe99410a9b0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fe98c009fc0 tx=0x7fe98c00b330 comp rx=0 tx=0).stop 2024-04-02T14:27:19.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:27:19.235+0000 7fe99a000700 1 -- 172.21.15.90:0/4149509118 shutdown_connections 2024-04-02T14:27:19.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:27:19.235+0000 7fe99a000700 1 --2- 172.21.15.90:0/4149509118 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe994102560 0x7fe994102980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:27:19.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:27:19.235+0000 7fe99a000700 1 --2- 172.21.15.90:0/4149509118 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe994104100 0x7fe99410a9b0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:27:19.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:27:19.235+0000 7fe99a000700 1 --2- 172.21.15.90:0/4149509118 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe994103760 0x7fe994103bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:27:19.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:27:19.235+0000 7fe99a000700 1 -- 172.21.15.90:0/4149509118 >> 172.21.15.90:0/4149509118 conn(0x7fe9940fdb00 msgr2=0x7fe9940fff40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:27:19.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:27:19.235+0000 7fe99a000700 1 -- 172.21.15.90:0/4149509118 shutdown_connections 2024-04-02T14:27:19.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:27:19.235+0000 7fe99a000700 1 -- 172.21.15.90:0/4149509118 wait complete. 2024-04-02T14:27:19.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:27:19.236+0000 7fe99a000700 1 Processor -- start 2024-04-02T14:27:19.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:27:19.237+0000 7fe99a000700 1 -- start start 2024-04-02T14:27:19.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:27:19.237+0000 7fe99a000700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe994102560 0x7fe99419a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:27:19.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:27:19.237+0000 7fe99a000700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe994103760 0x7fe99419ae20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:27:19.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:27:19.237+0000 7fe99a000700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe994104100 0x7fe9941a1ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:27:19.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:27:19.237+0000 7fe99a000700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fe99410e180 con 0x7fe994102560 2024-04-02T14:27:19.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:27:19.237+0000 7fe99a000700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fe99410e000 con 0x7fe994104100 2024-04-02T14:27:19.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:27:19.237+0000 7fe99a000700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fe99410e300 con 0x7fe994103760 2024-04-02T14:27:19.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:27:19.237+0000 7fe993e00700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe994103760 0x7fe99419ae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:27:19.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:27:19.237+0000 7fe998c00700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe994102560 0x7fe99419a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:27:19.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:27:19.237+0000 7fe993e00700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe994103760 0x7fe99419ae20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:57844/0 (socket says 172.21.15.90:57844) 2024-04-02T14:27:19.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:27:19.237+0000 7fe993e00700 1 -- 172.21.15.90:0/3701270363 learned_addr learned my addr 172.21.15.90:0/3701270363 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:27:19.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:27:19.237+0000 7fe999600700 1 --2- 172.21.15.90:0/3701270363 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe994104100 0x7fe9941a1ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:27:19.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:27:19.238+0000 7fe993e00700 1 -- 172.21.15.90:0/3701270363 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe994104100 msgr2=0x7fe9941a1ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:27:19.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:27:19.238+0000 7fe993e00700 1 --2- 172.21.15.90:0/3701270363 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe994104100 0x7fe9941a1ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:27:19.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:27:19.238+0000 7fe993e00700 1 -- 172.21.15.90:0/3701270363 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe994102560 msgr2=0x7fe99419a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:27:19.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:27:19.238+0000 7fe993e00700 1 --2- 172.21.15.90:0/3701270363 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe994102560 0x7fe99419a8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:27:19.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:27:19.238+0000 7fe993e00700 1 -- 172.21.15.90:0/3701270363 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9941a2540 con 0x7fe994103760 2024-04-02T14:27:19.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:27:19.238+0000 7fe993e00700 1 --2- 172.21.15.90:0/3701270363 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe994103760 0x7fe99419ae20 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fe98000ec40 tx=0x7fe98000ef50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:27:19.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:27:19.238+0000 7fe991600700 1 -- 172.21.15.90:0/3701270363 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe98000ccf0 con 0x7fe994103760 2024-04-02T14:27:19.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:27:19.239+0000 7fe99a000700 1 -- 172.21.15.90:0/3701270363 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9941a2830 con 0x7fe994103760 2024-04-02T14:27:19.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:27:19.239+0000 7fe991600700 1 -- 172.21.15.90:0/3701270363 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe98000ce90 con 0x7fe994103760 2024-04-02T14:27:19.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:27:19.239+0000 7fe991600700 1 -- 172.21.15.90:0/3701270363 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe980010690 con 0x7fe994103760 2024-04-02T14:27:19.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:27:19.239+0000 7fe99a000700 1 -- 172.21.15.90:0/3701270363 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9941a2d70 con 0x7fe994103760 2024-04-02T14:27:19.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:27:19.240+0000 7fe99a000700 1 -- 172.21.15.90:0/3701270363 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe9940692b0 con 0x7fe994103760 2024-04-02T14:27:19.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:27:19.242+0000 7fe991600700 1 -- 172.21.15.90:0/3701270363 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 15) ==== 138590+0+0 (secure 0 0 0) 0x7fe980010830 con 0x7fe994103760 2024-04-02T14:27:19.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:27:19.243+0000 7fe991600700 1 --2- 172.21.15.90:0/3701270363 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fe97c0a3bb0 0x7fe97c0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:27:19.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:27:19.243+0000 7fe991600700 1 -- 172.21.15.90:0/3701270363 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 3237+0+0 (secure 0 0 0) 0x7fe9800cf4f0 con 0x7fe994103760 2024-04-02T14:27:19.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:27:19.244+0000 7fe998c00700 1 --2- 172.21.15.90:0/3701270363 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fe97c0a3bb0 0x7fe97c0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:27:19.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:27:19.244+0000 7fe998c00700 1 --2- 172.21.15.90:0/3701270363 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fe97c0a3bb0 0x7fe97c0a6070 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fe9941035c0 tx=0x7fe988009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:27:19.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:27:19.247+0000 7fe991600700 1 -- 172.21.15.90:0/3701270363 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7fe98009d2e0 con 0x7fe994103760 2024-04-02T14:27:19.514 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:27:19.511+0000 7fe99a000700 1 -- 172.21.15.90:0/3701270363 --> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi090:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7fe9941a3030 con 0x7fe97c0a3bb0 2024-04-02T14:27:19.636 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:27:19 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-02T14:27:19.636 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:27:19 smithi090 ceph-mon[31674]: osdmap e20: 3 total, 3 up, 3 in 2024-04-02T14:27:19.636 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:27:19 smithi090 ceph-mon[31674]: pgmap v92: 1 pgs: 1 creating+peering; 0 B data, 96 MiB used, 268 GiB / 268 GiB avail 2024-04-02T14:27:19.637 INFO:journalctl@ceph.osd.0.smithi090.stdout:Apr 02 14:27:19 smithi090 sudo[50878]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-02T14:27:19.637 INFO:journalctl@ceph.osd.0.smithi090.stdout:Apr 02 14:27:19 smithi090 sudo[50878]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-02T14:27:19.637 INFO:journalctl@ceph.osd.0.smithi090.stdout:Apr 02 14:27:19 smithi090 sudo[50878]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-02T14:27:19.637 INFO:journalctl@ceph.osd.0.smithi090.stdout:Apr 02 14:27:19 smithi090 sudo[50878]: pam_unix(sudo:session): session closed for user root 2024-04-02T14:27:19.637 INFO:journalctl@ceph.osd.0.smithi090.stdout:Apr 02 14:27:19 smithi090 sudo[50881]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-02T14:27:19.637 INFO:journalctl@ceph.osd.0.smithi090.stdout:Apr 02 14:27:19 smithi090 sudo[50881]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-02T14:27:19.637 INFO:journalctl@ceph.osd.0.smithi090.stdout:Apr 02 14:27:19 smithi090 sudo[50881]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-02T14:27:19.637 INFO:journalctl@ceph.osd.0.smithi090.stdout:Apr 02 14:27:19 smithi090 sudo[50881]: pam_unix(sudo:session): session closed for user root 2024-04-02T14:27:19.686 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:27:19 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-02T14:27:19.686 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:27:19 smithi177 ceph-mon[36431]: osdmap e20: 3 total, 3 up, 3 in 2024-04-02T14:27:19.686 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:27:19 smithi177 ceph-mon[36431]: pgmap v92: 1 pgs: 1 creating+peering; 0 B data, 96 MiB used, 268 GiB / 268 GiB avail 2024-04-02T14:27:19.687 INFO:journalctl@ceph.osd.2.smithi177.stdout:Apr 02 14:27:19 smithi177 sudo[45208]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-02T14:27:19.687 INFO:journalctl@ceph.osd.2.smithi177.stdout:Apr 02 14:27:19 smithi177 sudo[45208]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-02T14:27:19.687 INFO:journalctl@ceph.osd.2.smithi177.stdout:Apr 02 14:27:19 smithi177 sudo[45208]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-02T14:27:19.687 INFO:journalctl@ceph.osd.2.smithi177.stdout:Apr 02 14:27:19 smithi177 sudo[45208]: pam_unix(sudo:session): session closed for user root 2024-04-02T14:27:19.687 INFO:journalctl@ceph.osd.2.smithi177.stdout:Apr 02 14:27:19 smithi177 sudo[45212]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-02T14:27:19.687 INFO:journalctl@ceph.osd.2.smithi177.stdout:Apr 02 14:27:19 smithi177 sudo[45212]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-02T14:27:19.687 INFO:journalctl@ceph.osd.2.smithi177.stdout:Apr 02 14:27:19 smithi177 sudo[45212]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-02T14:27:19.687 INFO:journalctl@ceph.osd.2.smithi177.stdout:Apr 02 14:27:19 smithi177 sudo[45212]: pam_unix(sudo:session): session closed for user root 2024-04-02T14:27:19.740 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:27:19 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-02T14:27:19.741 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:27:19 smithi092 ceph-mon[36497]: osdmap e20: 3 total, 3 up, 3 in 2024-04-02T14:27:19.741 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:27:19 smithi092 ceph-mon[36497]: pgmap v92: 1 pgs: 1 creating+peering; 0 B data, 96 MiB used, 268 GiB / 268 GiB avail 2024-04-02T14:27:19.741 INFO:journalctl@ceph.osd.1.smithi092.stdout:Apr 02 14:27:19 smithi092 sudo[44803]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-02T14:27:19.741 INFO:journalctl@ceph.osd.1.smithi092.stdout:Apr 02 14:27:19 smithi092 sudo[44803]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-02T14:27:19.741 INFO:journalctl@ceph.osd.1.smithi092.stdout:Apr 02 14:27:19 smithi092 sudo[44803]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-02T14:27:19.741 INFO:journalctl@ceph.osd.1.smithi092.stdout:Apr 02 14:27:19 smithi092 sudo[44803]: pam_unix(sudo:session): session closed for user root 2024-04-02T14:27:19.741 INFO:journalctl@ceph.osd.1.smithi092.stdout:Apr 02 14:27:19 smithi092 sudo[44806]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-02T14:27:19.741 INFO:journalctl@ceph.osd.1.smithi092.stdout:Apr 02 14:27:19 smithi092 sudo[44806]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-02T14:27:19.741 INFO:journalctl@ceph.osd.1.smithi092.stdout:Apr 02 14:27:19 smithi092 sudo[44806]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-02T14:27:19.741 INFO:journalctl@ceph.osd.1.smithi092.stdout:Apr 02 14:27:19 smithi092 sudo[44806]: pam_unix(sudo:session): session closed for user root 2024-04-02T14:27:19.939 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:27:19 smithi090 sudo[50919]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-02T14:27:19.939 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:27:19 smithi090 sudo[50919]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-02T14:27:19.939 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:27:19 smithi090 sudo[50919]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-02T14:27:19.939 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:27:19 smithi090 sudo[50919]: pam_unix(sudo:session): session closed for user root 2024-04-02T14:27:19.939 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:27:19 smithi090 sudo[50939]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-02T14:27:19.939 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:27:19 smithi090 sudo[50939]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-02T14:27:19.939 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:27:19 smithi090 sudo[50939]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-02T14:27:19.939 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:27:19 smithi090 sudo[50939]: pam_unix(sudo:session): session closed for user root 2024-04-02T14:27:19.998 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:27:19 smithi177 sudo[45215]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-02T14:27:19.998 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:27:19 smithi177 sudo[45215]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-02T14:27:19.998 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:27:19 smithi177 sudo[45215]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-02T14:27:19.998 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:27:19 smithi177 sudo[45215]: pam_unix(sudo:session): session closed for user root 2024-04-02T14:27:19.999 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:27:19 smithi177 sudo[45218]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-02T14:27:19.999 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:27:19 smithi177 sudo[45218]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-02T14:27:19.999 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:27:19 smithi177 sudo[45218]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-02T14:27:19.999 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:27:19 smithi177 sudo[45218]: pam_unix(sudo:session): session closed for user root 2024-04-02T14:27:20.110 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:27:19 smithi092 sudo[44809]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-02T14:27:20.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:27:19 smithi092 sudo[44809]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-02T14:27:20.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:27:19 smithi092 sudo[44809]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-02T14:27:20.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:27:19 smithi092 sudo[44809]: pam_unix(sudo:session): session closed for user root 2024-04-02T14:27:20.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:27:19 smithi092 sudo[44812]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-02T14:27:20.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:27:19 smithi092 sudo[44812]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-02T14:27:20.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:27:19 smithi092 sudo[44812]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-02T14:27:20.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:27:19 smithi092 sudo[44812]: pam_unix(sudo:session): session closed for user root 2024-04-02T14:27:20.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:27:20 smithi177 ceph-mon[36431]: osdmap e21: 3 total, 3 up, 3 in 2024-04-02T14:27:20.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:27:20 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:27:20.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:27:20 smithi177 ceph-mon[36431]: from='client.24193 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi090:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T14:27:20.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:27:20 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-02T14:27:20.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:27:20 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-02T14:27:20.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:27:20 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:27:20.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:27:20 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:27:20.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:27:20 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:27:20.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:27:20 smithi177 ceph-mon[36431]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-02T14:27:20.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:27:20 smithi177 ceph-mon[36431]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-02T14:27:20.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:27:20 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-02T14:27:20.688 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:27:20 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-02T14:27:20.688 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:27:20 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-02T14:27:20.688 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:27:20 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-02T14:27:20.688 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:27:20 smithi177 ceph-mon[36431]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-02T14:27:20.688 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:27:20 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-02T14:27:20.688 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:27:20 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-02T14:27:20.688 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:27:20 smithi177 ceph-mon[36431]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-02T14:27:20.688 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:27:20 smithi177 ceph-mon[36431]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-02T14:27:20.688 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:27:20 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-02T14:27:20.688 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:27:20 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-02T14:27:20.688 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:27:20 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-02T14:27:20.688 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:27:20 smithi177 ceph-mon[36431]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-02T14:27:20.689 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:27:20 smithi090 ceph-mon[31674]: osdmap e21: 3 total, 3 up, 3 in 2024-04-02T14:27:20.689 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:27:20 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:27:20.689 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:27:20 smithi090 ceph-mon[31674]: from='client.24193 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi090:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T14:27:20.689 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:27:20 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-02T14:27:20.689 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:27:20 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-02T14:27:20.689 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:27:20 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:27:20.690 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:27:20 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:27:20.690 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:27:20 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:27:20.690 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:27:20 smithi090 ceph-mon[31674]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-02T14:27:20.690 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:27:20 smithi090 ceph-mon[31674]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-02T14:27:20.690 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:27:20 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-02T14:27:20.690 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:27:20 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-02T14:27:20.690 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:27:20 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-02T14:27:20.690 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:27:20 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-02T14:27:20.690 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:27:20 smithi090 ceph-mon[31674]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-02T14:27:20.690 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:27:20 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-02T14:27:20.690 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:27:20 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-02T14:27:20.690 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:27:20 smithi090 ceph-mon[31674]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-02T14:27:20.690 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:27:20 smithi090 ceph-mon[31674]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-02T14:27:20.690 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:27:20 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-02T14:27:20.690 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:27:20 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-02T14:27:20.691 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:27:20 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-02T14:27:20.691 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:27:20 smithi090 ceph-mon[31674]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-02T14:27:20.861 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:27:20 smithi092 ceph-mon[36497]: osdmap e21: 3 total, 3 up, 3 in 2024-04-02T14:27:20.861 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:27:20 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:27:20.861 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:27:20 smithi092 ceph-mon[36497]: from='client.24193 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi090:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T14:27:20.861 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:27:20 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-02T14:27:20.861 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:27:20 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-02T14:27:20.861 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:27:20 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:27:20.861 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:27:20 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:27:20.862 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:27:20 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:27:20.862 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:27:20 smithi092 ceph-mon[36497]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-02T14:27:20.862 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:27:20 smithi092 ceph-mon[36497]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-02T14:27:20.862 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:27:20 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-02T14:27:20.862 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:27:20 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-02T14:27:20.862 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:27:20 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-02T14:27:20.862 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:27:20 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-02T14:27:20.862 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:27:20 smithi092 ceph-mon[36497]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-02T14:27:20.862 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:27:20 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-02T14:27:20.862 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:27:20 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-02T14:27:20.862 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:27:20 smithi092 ceph-mon[36497]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-02T14:27:20.862 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:27:20 smithi092 ceph-mon[36497]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-02T14:27:20.862 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:27:20 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-02T14:27:20.863 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:27:20 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-02T14:27:20.863 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:27:20 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-02T14:27:20.863 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:27:20 smithi092 ceph-mon[36497]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-02T14:27:21.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:27:21.415+0000 7fe991600700 1 -- 172.21.15.90:0/3701270363 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fe980099250 con 0x7fe994103760 2024-04-02T14:27:21.604 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:27:21 smithi090 ceph-mon[31674]: pgmap v94: 1 pgs: 1 creating+peering; 0 B data, 96 MiB used, 268 GiB / 268 GiB avail 2024-04-02T14:27:21.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:27:21 smithi177 ceph-mon[36431]: pgmap v94: 1 pgs: 1 creating+peering; 0 B data, 96 MiB used, 268 GiB / 268 GiB avail 2024-04-02T14:27:21.860 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:27:21 smithi092 ceph-mon[36497]: pgmap v94: 1 pgs: 1 creating+peering; 0 B data, 96 MiB used, 268 GiB / 268 GiB avail 2024-04-02T14:27:22.686 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:27:22 smithi177 ceph-mon[36431]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-04-02T14:27:22.688 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:27:22 smithi090 ceph-mon[31674]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-04-02T14:27:22.860 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:27:22 smithi092 ceph-mon[36497]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-04-02T14:27:23.686 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:27:23 smithi177 ceph-mon[36431]: pgmap v95: 1 pgs: 1 creating+peering; 0 B data, 97 MiB used, 268 GiB / 268 GiB avail 2024-04-02T14:27:23.686 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:27:23 smithi177 ceph-mon[36431]: from='client.? 172.21.15.90:0/2570446540' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b175a2f4-7859-4eb0-bc1f-fba8c2b92a2a"}]: dispatch 2024-04-02T14:27:23.686 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:27:23 smithi177 ceph-mon[36431]: from='client.? 172.21.15.90:0/2570446540' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b175a2f4-7859-4eb0-bc1f-fba8c2b92a2a"}]': finished 2024-04-02T14:27:23.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:27:23 smithi177 ceph-mon[36431]: osdmap e22: 4 total, 3 up, 4 in 2024-04-02T14:27:23.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:27:23 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-02T14:27:23.689 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:27:23 smithi090 ceph-mon[31674]: pgmap v95: 1 pgs: 1 creating+peering; 0 B data, 97 MiB used, 268 GiB / 268 GiB avail 2024-04-02T14:27:23.689 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:27:23 smithi090 ceph-mon[31674]: from='client.? 172.21.15.90:0/2570446540' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b175a2f4-7859-4eb0-bc1f-fba8c2b92a2a"}]: dispatch 2024-04-02T14:27:23.689 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:27:23 smithi090 ceph-mon[31674]: from='client.? 172.21.15.90:0/2570446540' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b175a2f4-7859-4eb0-bc1f-fba8c2b92a2a"}]': finished 2024-04-02T14:27:23.689 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:27:23 smithi090 ceph-mon[31674]: osdmap e22: 4 total, 3 up, 4 in 2024-04-02T14:27:23.689 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:27:23 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-02T14:27:23.762 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:27:23 smithi092 ceph-mon[36497]: pgmap v95: 1 pgs: 1 creating+peering; 0 B data, 97 MiB used, 268 GiB / 268 GiB avail 2024-04-02T14:27:23.762 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:27:23 smithi092 ceph-mon[36497]: from='client.? 172.21.15.90:0/2570446540' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b175a2f4-7859-4eb0-bc1f-fba8c2b92a2a"}]: dispatch 2024-04-02T14:27:23.762 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:27:23 smithi092 ceph-mon[36497]: from='client.? 172.21.15.90:0/2570446540' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b175a2f4-7859-4eb0-bc1f-fba8c2b92a2a"}]': finished 2024-04-02T14:27:23.762 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:27:23 smithi092 ceph-mon[36497]: osdmap e22: 4 total, 3 up, 4 in 2024-04-02T14:27:23.762 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:27:23 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-02T14:27:24.775 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:27:24 smithi092 ceph-mon[36497]: from='client.? 172.21.15.90:0/4022771166' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-02T14:27:24.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:27:24 smithi177 ceph-mon[36431]: from='client.? 172.21.15.90:0/4022771166' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-02T14:27:24.938 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:27:24 smithi090 ceph-mon[31674]: from='client.? 172.21.15.90:0/4022771166' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-02T14:27:25.860 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:27:25 smithi092 ceph-mon[36497]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 99 MiB used, 268 GiB / 268 GiB avail 2024-04-02T14:27:25.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:27:25 smithi177 ceph-mon[36431]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 99 MiB used, 268 GiB / 268 GiB avail 2024-04-02T14:27:25.938 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:27:25 smithi090 ceph-mon[31674]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 99 MiB used, 268 GiB / 268 GiB avail 2024-04-02T14:27:28.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:27:27 smithi092 ceph-mon[36497]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 99 MiB used, 268 GiB / 268 GiB avail 2024-04-02T14:27:28.135 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:27:27 smithi090 ceph-mon[31674]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 99 MiB used, 268 GiB / 268 GiB avail 2024-04-02T14:27:28.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:27:27 smithi177 ceph-mon[36431]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 99 MiB used, 268 GiB / 268 GiB avail 2024-04-02T14:27:29.823 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:27:29 smithi090 ceph-mon[31674]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-04-02T14:27:30.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:27:29 smithi092 ceph-mon[36497]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-04-02T14:27:30.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:27:29 smithi177 ceph-mon[36431]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-04-02T14:27:31.939 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:27:31 smithi090 ceph-mon[31674]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-04-02T14:27:32.110 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:27:31 smithi092 ceph-mon[36497]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-04-02T14:27:32.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:27:31 smithi177 ceph-mon[36431]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-04-02T14:27:33.110 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:27:32 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-02T14:27:33.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:27:32 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:27:33.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:27:32 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-02T14:27:33.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:27:32 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:27:33.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:27:32 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-02T14:27:33.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:27:32 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:27:33.939 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:27:33 smithi090 ceph-mon[31674]: Deploying daemon osd.3 on smithi090 2024-04-02T14:27:33.939 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:27:33 smithi090 ceph-mon[31674]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-04-02T14:27:34.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:27:33 smithi092 ceph-mon[36497]: Deploying daemon osd.3 on smithi090 2024-04-02T14:27:34.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:27:33 smithi092 ceph-mon[36497]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-04-02T14:27:34.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:27:33 smithi177 ceph-mon[36431]: Deploying daemon osd.3 on smithi090 2024-04-02T14:27:34.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:27:33 smithi177 ceph-mon[36431]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-04-02T14:27:35.861 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:27:35 smithi092 ceph-mon[36497]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-04-02T14:27:35.938 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:27:35 smithi090 ceph-mon[31674]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-04-02T14:27:36.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:27:35 smithi177 ceph-mon[36431]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-04-02T14:27:37.811 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:27:37 smithi090 ceph-mon[31674]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-04-02T14:27:38.110 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:27:37 smithi092 ceph-mon[36497]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-04-02T14:27:38.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:27:37 smithi177 ceph-mon[36431]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-04-02T14:27:39.812 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:27:39 smithi090 ceph-mon[31674]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-04-02T14:27:40.110 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:27:39 smithi092 ceph-mon[36497]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-04-02T14:27:40.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:27:39 smithi177 ceph-mon[36431]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-04-02T14:27:41.889 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:27:41 smithi090 ceph-mon[31674]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-04-02T14:27:41.889 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:27:41 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:27:41.889 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:27:41 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:27:41.889 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:27:41 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T14:27:41.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:27:41 smithi177 ceph-mon[36431]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-04-02T14:27:41.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:27:41 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:27:41.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:27:41 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:27:41.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:27:41 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T14:27:42.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:27:41 smithi092 ceph-mon[36497]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-04-02T14:27:42.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:27:41 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:27:42.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:27:41 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:27:42.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:27:41 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T14:27:43.894 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:27:43 smithi090 ceph-mon[31674]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-04-02T14:27:43.894 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:27:43 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:27:43.894 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:27:43 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:27:43.894 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:27:43 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:27:43.894 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:27:43 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T14:27:43.894 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:27:43 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:27:43.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:27:43 smithi177 ceph-mon[36431]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-04-02T14:27:43.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:27:43 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:27:43.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:27:43 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:27:43.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:27:43 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:27:43.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:27:43 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T14:27:43.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:27:43 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:27:44.110 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:27:43 smithi092 ceph-mon[36497]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-04-02T14:27:44.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:27:43 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:27:44.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:27:43 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:27:44.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:27:43 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:27:44.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:27:43 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T14:27:44.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:27:43 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:27:44.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:27:44 smithi177 ceph-mon[36431]: from='osd.3 [v2:172.21.15.90:6810/3114811216,v1:172.21.15.90:6811/3114811216]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-02T14:27:44.938 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:27:44 smithi090 ceph-mon[31674]: from='osd.3 [v2:172.21.15.90:6810/3114811216,v1:172.21.15.90:6811/3114811216]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-02T14:27:45.110 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:27:44 smithi092 ceph-mon[36497]: from='osd.3 [v2:172.21.15.90:6810/3114811216,v1:172.21.15.90:6811/3114811216]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-02T14:27:45.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:27:45.654+0000 7fe991600700 1 -- 172.21.15.90:0/3701270363 <== mgr.14152 v2:172.21.15.90:6800/4266054817 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fe9941a3030 con 0x7fe97c0a3bb0 2024-04-02T14:27:45.657 INFO:teuthology.orchestra.run.smithi090.stdout:Created osd(s) 3 on host 'smithi090' 2024-04-02T14:27:45.660 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:27:45.659+0000 7fe99a000700 1 -- 172.21.15.90:0/3701270363 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fe97c0a3bb0 msgr2=0x7fe97c0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:27:45.660 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:27:45.659+0000 7fe99a000700 1 --2- 172.21.15.90:0/3701270363 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fe97c0a3bb0 0x7fe97c0a6070 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fe9941035c0 tx=0x7fe988009260 comp rx=0 tx=0).stop 2024-04-02T14:27:45.660 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:27:45.659+0000 7fe99a000700 1 -- 172.21.15.90:0/3701270363 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe994103760 msgr2=0x7fe99419ae20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:27:45.660 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:27:45.659+0000 7fe99a000700 1 --2- 172.21.15.90:0/3701270363 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe994103760 0x7fe99419ae20 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fe98000ec40 tx=0x7fe98000ef50 comp rx=0 tx=0).stop 2024-04-02T14:27:45.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:27:45.659+0000 7fe99a000700 1 -- 172.21.15.90:0/3701270363 shutdown_connections 2024-04-02T14:27:45.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:27:45.659+0000 7fe99a000700 1 --2- 172.21.15.90:0/3701270363 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fe97c0a3bb0 0x7fe97c0a6070 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:27:45.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:27:45.659+0000 7fe99a000700 1 --2- 172.21.15.90:0/3701270363 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe994102560 0x7fe99419a8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:27:45.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:27:45.659+0000 7fe99a000700 1 --2- 172.21.15.90:0/3701270363 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe994104100 0x7fe9941a1ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:27:45.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:27:45.660+0000 7fe99a000700 1 --2- 172.21.15.90:0/3701270363 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe994103760 0x7fe99419ae20 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:27:45.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:27:45.660+0000 7fe99a000700 1 -- 172.21.15.90:0/3701270363 >> 172.21.15.90:0/3701270363 conn(0x7fe9940fdb00 msgr2=0x7fe994074820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:27:45.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:27:45.660+0000 7fe99a000700 1 -- 172.21.15.90:0/3701270363 shutdown_connections 2024-04-02T14:27:45.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:27:45.660+0000 7fe99a000700 1 -- 172.21.15.90:0/3701270363 wait complete. 2024-04-02T14:27:45.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:27:45 smithi177 ceph-mon[36431]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-04-02T14:27:45.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:27:45 smithi177 ceph-mon[36431]: from='osd.3 [v2:172.21.15.90:6810/3114811216,v1:172.21.15.90:6811/3114811216]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-02T14:27:45.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:27:45 smithi177 ceph-mon[36431]: osdmap e23: 4 total, 3 up, 4 in 2024-04-02T14:27:45.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:27:45 smithi177 ceph-mon[36431]: from='osd.3 [v2:172.21.15.90:6810/3114811216,v1:172.21.15.90:6811/3114811216]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]: dispatch 2024-04-02T14:27:45.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:27:45 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-02T14:27:45.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:27:45 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:27:45.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:27:45 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:27:45.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:27:45 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T14:27:45.938 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:27:45 smithi090 ceph-mon[31674]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-04-02T14:27:45.938 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:27:45 smithi090 ceph-mon[31674]: from='osd.3 [v2:172.21.15.90:6810/3114811216,v1:172.21.15.90:6811/3114811216]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-02T14:27:45.938 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:27:45 smithi090 ceph-mon[31674]: osdmap e23: 4 total, 3 up, 4 in 2024-04-02T14:27:45.939 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:27:45 smithi090 ceph-mon[31674]: from='osd.3 [v2:172.21.15.90:6810/3114811216,v1:172.21.15.90:6811/3114811216]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]: dispatch 2024-04-02T14:27:45.939 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:27:45 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-02T14:27:45.939 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:27:45 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:27:45.939 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:27:45 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:27:45.939 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:27:45 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T14:27:46.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:27:45 smithi092 ceph-mon[36497]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-04-02T14:27:46.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:27:45 smithi092 ceph-mon[36497]: from='osd.3 [v2:172.21.15.90:6810/3114811216,v1:172.21.15.90:6811/3114811216]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-02T14:27:46.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:27:45 smithi092 ceph-mon[36497]: osdmap e23: 4 total, 3 up, 4 in 2024-04-02T14:27:46.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:27:45 smithi092 ceph-mon[36497]: from='osd.3 [v2:172.21.15.90:6810/3114811216,v1:172.21.15.90:6811/3114811216]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]: dispatch 2024-04-02T14:27:46.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:27:45 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-02T14:27:46.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:27:45 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:27:46.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:27:45 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:27:46.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:27:45 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T14:27:46.285 DEBUG:teuthology.orchestra.run.smithi090:osd.3> sudo journalctl -f -n 0 -u ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4@osd.3.service 2024-04-02T14:27:46.291 INFO:tasks.cephadm:Deploying osd.4 on smithi092 with /dev/vg_nvme/lv_3... 2024-04-02T14:27:46.291 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:191d963ac9de7322da8aff20b567b3fa458e2ef6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6449448a-f0fc-11ee-b647-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-02T14:27:46.534 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/mon.b/config 2024-04-02T14:27:46.800 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:27:46 smithi090 ceph-mon[31674]: from='osd.3 [v2:172.21.15.90:6810/3114811216,v1:172.21.15.90:6811/3114811216]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]': finished 2024-04-02T14:27:46.800 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:27:46 smithi090 ceph-mon[31674]: osdmap e24: 4 total, 3 up, 4 in 2024-04-02T14:27:46.800 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:27:46 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-02T14:27:46.800 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:27:46 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-02T14:27:46.957 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:27:46 smithi092 ceph-mon[36497]: from='osd.3 [v2:172.21.15.90:6810/3114811216,v1:172.21.15.90:6811/3114811216]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]': finished 2024-04-02T14:27:46.957 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:27:46 smithi092 ceph-mon[36497]: osdmap e24: 4 total, 3 up, 4 in 2024-04-02T14:27:46.958 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:27:46 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-02T14:27:46.958 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:27:46 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-02T14:27:47.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:27:46 smithi177 ceph-mon[36431]: from='osd.3 [v2:172.21.15.90:6810/3114811216,v1:172.21.15.90:6811/3114811216]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]': finished 2024-04-02T14:27:47.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:27:46 smithi177 ceph-mon[36431]: osdmap e24: 4 total, 3 up, 4 in 2024-04-02T14:27:47.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:27:46 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-02T14:27:47.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:27:46 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-02T14:27:47.818 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:27:47 smithi090 ceph-mon[31674]: purged_snaps scrub starts 2024-04-02T14:27:47.819 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:27:47 smithi090 ceph-mon[31674]: purged_snaps scrub ok 2024-04-02T14:27:47.819 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:27:47 smithi090 ceph-mon[31674]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-04-02T14:27:47.819 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:27:47 smithi090 ceph-mon[31674]: osd.3 [v2:172.21.15.90:6810/3114811216,v1:172.21.15.90:6811/3114811216] boot 2024-04-02T14:27:47.819 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:27:47 smithi090 ceph-mon[31674]: osdmap e25: 4 total, 4 up, 4 in 2024-04-02T14:27:47.819 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:27:47 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-02T14:27:47.819 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:27:47 smithi090 ceph-mon[31674]: osdmap e26: 4 total, 4 up, 4 in 2024-04-02T14:27:47.821 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:27:47 smithi092 ceph-mon[36497]: purged_snaps scrub starts 2024-04-02T14:27:47.821 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:27:47 smithi092 ceph-mon[36497]: purged_snaps scrub ok 2024-04-02T14:27:47.821 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:27:47 smithi092 ceph-mon[36497]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-04-02T14:27:47.821 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:27:47 smithi092 ceph-mon[36497]: osd.3 [v2:172.21.15.90:6810/3114811216,v1:172.21.15.90:6811/3114811216] boot 2024-04-02T14:27:47.822 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:27:47 smithi092 ceph-mon[36497]: osdmap e25: 4 total, 4 up, 4 in 2024-04-02T14:27:47.822 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:27:47 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-02T14:27:47.822 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:27:47 smithi092 ceph-mon[36497]: osdmap e26: 4 total, 4 up, 4 in 2024-04-02T14:27:48.088 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:27:47 smithi177 ceph-mon[36431]: purged_snaps scrub starts 2024-04-02T14:27:48.088 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:27:47 smithi177 ceph-mon[36431]: purged_snaps scrub ok 2024-04-02T14:27:48.088 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:27:47 smithi177 ceph-mon[36431]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-04-02T14:27:48.088 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:27:47 smithi177 ceph-mon[36431]: osd.3 [v2:172.21.15.90:6810/3114811216,v1:172.21.15.90:6811/3114811216] boot 2024-04-02T14:27:48.088 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:27:47 smithi177 ceph-mon[36431]: osdmap e25: 4 total, 4 up, 4 in 2024-04-02T14:27:48.088 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:27:47 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-02T14:27:48.088 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:27:47 smithi177 ceph-mon[36431]: osdmap e26: 4 total, 4 up, 4 in 2024-04-02T14:27:48.382 INFO:teuthology.orchestra.run.smithi092.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-02T14:27:48.382 INFO:teuthology.orchestra.run.smithi092.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-02T14:27:48.382 INFO:teuthology.orchestra.run.smithi092.stderr: stderr: 10+0 records in 2024-04-02T14:27:48.382 INFO:teuthology.orchestra.run.smithi092.stderr:10+0 records out 2024-04-02T14:27:48.382 INFO:teuthology.orchestra.run.smithi092.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0118257 s, 887 MB/s 2024-04-02T14:27:48.382 INFO:teuthology.orchestra.run.smithi092.stderr:--> Zapping successful for: 2024-04-02T14:27:48.994 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:191d963ac9de7322da8aff20b567b3fa458e2ef6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6449448a-f0fc-11ee-b647-cb9ed24678a4 -- ceph orch daemon add osd smithi092:vg_nvme/lv_3 2024-04-02T14:27:49.242 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/mon.b/config 2024-04-02T14:27:49.938 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:27:49 smithi090 ceph-mon[31674]: pgmap v113: 1 pgs: 1 remapped+peering; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-04-02T14:27:49.939 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:27:49 smithi090 ceph-mon[31674]: osdmap e27: 4 total, 4 up, 4 in 2024-04-02T14:27:49.998 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:27:49 smithi092 ceph-mon[36497]: pgmap v113: 1 pgs: 1 remapped+peering; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-04-02T14:27:49.998 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:27:49 smithi092 ceph-mon[36497]: osdmap e27: 4 total, 4 up, 4 in 2024-04-02T14:27:50.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:27:49 smithi177 ceph-mon[36431]: pgmap v113: 1 pgs: 1 remapped+peering; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-04-02T14:27:50.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:27:49 smithi177 ceph-mon[36431]: osdmap e27: 4 total, 4 up, 4 in 2024-04-02T14:27:50.723 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:27:50.719+0000 7f4990715700 1 -- 172.21.15.92:0/653679973 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4988103840 msgr2=0x7f4988103ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:27:50.723 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:27:50.719+0000 7f4990715700 1 --2- 172.21.15.92:0/653679973 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4988103840 0x7f4988103ca0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f49780099c0 tx=0x7f4978009cd0 comp rx=0 tx=0).stop 2024-04-02T14:27:50.723 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:27:50.719+0000 7f4990715700 1 -- 172.21.15.92:0/653679973 shutdown_connections 2024-04-02T14:27:50.723 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:27:50.719+0000 7f4990715700 1 --2- 172.21.15.92:0/653679973 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f49881041e0 0x7f498810aa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:27:50.723 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:27:50.719+0000 7f4990715700 1 --2- 172.21.15.92:0/653679973 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4988102640 0x7f4988102a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:27:50.723 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:27:50.719+0000 7f4990715700 1 --2- 172.21.15.92:0/653679973 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4988103840 0x7f4988103ca0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:27:50.724 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:27:50.719+0000 7f4990715700 1 -- 172.21.15.92:0/653679973 >> 172.21.15.92:0/653679973 conn(0x7f49880fdc00 msgr2=0x7f4988100020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:27:50.724 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:27:50.719+0000 7f4990715700 1 -- 172.21.15.92:0/653679973 shutdown_connections 2024-04-02T14:27:50.724 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:27:50.719+0000 7f4990715700 1 -- 172.21.15.92:0/653679973 wait complete. 2024-04-02T14:27:50.724 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:27:50.720+0000 7f4990715700 1 Processor -- start 2024-04-02T14:27:50.724 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:27:50.720+0000 7f4990715700 1 -- start start 2024-04-02T14:27:50.724 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:27:50.720+0000 7f4990715700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4988102640 0x7f4988198860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:27:50.724 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:27:50.720+0000 7f4990715700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4988103840 0x7f4988198da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:27:50.724 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:27:50.720+0000 7f4990715700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f49881041e0 0x7f498819fe20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:27:50.724 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:27:50.720+0000 7f4990715700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f498810e2a0 con 0x7f49881041e0 2024-04-02T14:27:50.724 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:27:50.720+0000 7f4990715700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f498810e120 con 0x7f4988103840 2024-04-02T14:27:50.724 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:27:50.720+0000 7f4990715700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f498810e420 con 0x7f4988102640 2024-04-02T14:27:50.724 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:27:50.720+0000 7f498dcb0700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4988103840 0x7f4988198da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:27:50.724 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:27:50.720+0000 7f498dcb0700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4988103840 0x7f4988198da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:47750/0 (socket says 172.21.15.92:47750) 2024-04-02T14:27:50.725 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:27:50.720+0000 7f498dcb0700 1 -- 172.21.15.92:0/214522837 learned_addr learned my addr 172.21.15.92:0/214522837 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-04-02T14:27:50.725 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:27:50.720+0000 7f498e4b1700 1 --2- 172.21.15.92:0/214522837 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4988102640 0x7f4988198860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:27:50.725 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:27:50.721+0000 7f498ecb2700 1 --2- 172.21.15.92:0/214522837 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f49881041e0 0x7f498819fe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:27:50.725 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:27:50.721+0000 7f498dcb0700 1 -- 172.21.15.92:0/214522837 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4988102640 msgr2=0x7f4988198860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:27:50.725 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:27:50.721+0000 7f498dcb0700 1 --2- 172.21.15.92:0/214522837 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4988102640 0x7f4988198860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:27:50.725 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:27:50.721+0000 7f498dcb0700 1 -- 172.21.15.92:0/214522837 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f49881041e0 msgr2=0x7f498819fe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:27:50.725 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:27:50.721+0000 7f498dcb0700 1 --2- 172.21.15.92:0/214522837 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f49881041e0 0x7f498819fe20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:27:50.725 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:27:50.721+0000 7f498dcb0700 1 -- 172.21.15.92:0/214522837 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49881a04c0 con 0x7f4988103840 2024-04-02T14:27:50.725 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:27:50.721+0000 7f498dcb0700 1 --2- 172.21.15.92:0/214522837 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4988103840 0x7f4988198da0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f4978005910 tx=0x7f497800fa00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:27:50.725 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:27:50.721+0000 7f497f7fe700 1 -- 172.21.15.92:0/214522837 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4978027070 con 0x7f4988103840 2024-04-02T14:27:50.725 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:27:50.721+0000 7f4990715700 1 -- 172.21.15.92:0/214522837 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49881a0750 con 0x7f4988103840 2024-04-02T14:27:50.725 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:27:50.721+0000 7f497f7fe700 1 -- 172.21.15.92:0/214522837 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4978022950 con 0x7f4988103840 2024-04-02T14:27:50.725 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:27:50.721+0000 7f497f7fe700 1 -- 172.21.15.92:0/214522837 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4978009e60 con 0x7f4988103840 2024-04-02T14:27:50.726 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:27:50.721+0000 7f4990715700 1 -- 172.21.15.92:0/214522837 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49881a0c30 con 0x7f4988103840 2024-04-02T14:27:50.726 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:27:50.722+0000 7f4990715700 1 -- 172.21.15.92:0/214522837 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f49880692b0 con 0x7f4988103840 2024-04-02T14:27:50.727 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:27:50.723+0000 7f497f7fe700 1 -- 172.21.15.92:0/214522837 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f497802a070 con 0x7f4988103840 2024-04-02T14:27:50.727 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:27:50.723+0000 7f497f7fe700 1 --2- 172.21.15.92:0/214522837 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f49740a3c40 0x7f49740a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:27:50.727 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:27:50.724+0000 7f497f7fe700 1 -- 172.21.15.92:0/214522837 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 3669+0+0 (secure 0 0 0) 0x7f49780cfe80 con 0x7f4988103840 2024-04-02T14:27:50.727 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:27:50.724+0000 7f498e4b1700 1 --2- 172.21.15.92:0/214522837 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f49740a3c40 0x7f49740a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:27:50.727 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:27:50.725+0000 7f498e4b1700 1 --2- 172.21.15.92:0/214522837 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f49740a3c40 0x7f49740a6100 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f49881036a0 tx=0x7f4984009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:27:50.730 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:27:50.728+0000 7f497f7fe700 1 -- 172.21.15.92:0/214522837 <== mon.2 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7f4978013070 con 0x7f4988103840 2024-04-02T14:27:50.938 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:27:50.936+0000 7f4990715700 1 -- 172.21.15.92:0/214522837 --> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi092:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f4988062a30 con 0x7f49740a3c40 2024-04-02T14:27:51.110 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:27:50 smithi092 ceph-mon[36497]: osdmap e28: 4 total, 4 up, 4 in 2024-04-02T14:27:51.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:27:50 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:27:51.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:27:50 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:27:51.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:27:50 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-02T14:27:51.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:27:50 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-02T14:27:51.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:27:50 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:27:51.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:27:50 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:27:51.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:27:50 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T14:27:51.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:27:50 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:27:51.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:27:50 smithi177 ceph-mon[36431]: osdmap e28: 4 total, 4 up, 4 in 2024-04-02T14:27:51.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:27:50 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:27:51.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:27:50 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:27:51.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:27:50 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-02T14:27:51.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:27:50 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-02T14:27:51.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:27:50 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:27:51.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:27:50 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:27:51.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:27:50 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T14:27:51.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:27:50 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:27:51.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:27:50 smithi090 ceph-mon[31674]: osdmap e28: 4 total, 4 up, 4 in 2024-04-02T14:27:51.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:27:50 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:27:51.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:27:50 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:27:51.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:27:50 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-02T14:27:51.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:27:50 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-02T14:27:51.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:27:50 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:27:51.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:27:50 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:27:51.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:27:50 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T14:27:51.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:27:50 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:27:51.838 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:27:51 smithi092 ceph-mon[36497]: Detected new or changed devices on smithi090 2024-04-02T14:27:51.838 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:27:51 smithi092 ceph-mon[36497]: Adjusting osd_memory_target on smithi090 to 8545M 2024-04-02T14:27:51.838 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:27:51 smithi092 ceph-mon[36497]: pgmap v116: 1 pgs: 1 remapped+peering; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-04-02T14:27:51.838 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:27:51 smithi092 ceph-mon[36497]: osdmap e29: 4 total, 4 up, 4 in 2024-04-02T14:27:51.838 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:27:51 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-02T14:27:51.838 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:27:51 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-02T14:27:51.838 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:27:51 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:27:52.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:27:51 smithi177 ceph-mon[36431]: Detected new or changed devices on smithi090 2024-04-02T14:27:52.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:27:51 smithi177 ceph-mon[36431]: Adjusting osd_memory_target on smithi090 to 8545M 2024-04-02T14:27:52.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:27:51 smithi177 ceph-mon[36431]: pgmap v116: 1 pgs: 1 remapped+peering; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-04-02T14:27:52.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:27:51 smithi177 ceph-mon[36431]: osdmap e29: 4 total, 4 up, 4 in 2024-04-02T14:27:52.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:27:51 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-02T14:27:52.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:27:51 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-02T14:27:52.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:27:51 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:27:52.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:27:51 smithi090 ceph-mon[31674]: Detected new or changed devices on smithi090 2024-04-02T14:27:52.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:27:51 smithi090 ceph-mon[31674]: Adjusting osd_memory_target on smithi090 to 8545M 2024-04-02T14:27:52.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:27:51 smithi090 ceph-mon[31674]: pgmap v116: 1 pgs: 1 remapped+peering; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-04-02T14:27:52.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:27:51 smithi090 ceph-mon[31674]: osdmap e29: 4 total, 4 up, 4 in 2024-04-02T14:27:52.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:27:51 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-02T14:27:52.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:27:51 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-02T14:27:52.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:27:51 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:27:53.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:27:52 smithi092 ceph-mon[36497]: from='client.24212 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi092:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T14:27:53.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:27:52 smithi177 ceph-mon[36431]: from='client.24212 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi092:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T14:27:53.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:27:52 smithi090 ceph-mon[31674]: from='client.24212 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi092:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T14:27:54.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:27:53 smithi092 ceph-mon[36497]: pgmap v118: 1 pgs: 1 remapped+peering; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-04-02T14:27:54.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:27:53 smithi092 ceph-mon[36497]: from='client.? 172.21.15.92:0/2582948796' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "38cc68ea-c109-4d8d-b77d-1c6f9b56c875"}]: dispatch 2024-04-02T14:27:54.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:27:53 smithi092 ceph-mon[36497]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "38cc68ea-c109-4d8d-b77d-1c6f9b56c875"}]: dispatch 2024-04-02T14:27:54.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:27:53 smithi092 ceph-mon[36497]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "38cc68ea-c109-4d8d-b77d-1c6f9b56c875"}]': finished 2024-04-02T14:27:54.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:27:53 smithi092 ceph-mon[36497]: osdmap e30: 5 total, 4 up, 5 in 2024-04-02T14:27:54.112 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:27:53 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-02T14:27:54.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:27:53 smithi177 ceph-mon[36431]: pgmap v118: 1 pgs: 1 remapped+peering; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-04-02T14:27:54.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:27:53 smithi177 ceph-mon[36431]: from='client.? 172.21.15.92:0/2582948796' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "38cc68ea-c109-4d8d-b77d-1c6f9b56c875"}]: dispatch 2024-04-02T14:27:54.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:27:53 smithi177 ceph-mon[36431]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "38cc68ea-c109-4d8d-b77d-1c6f9b56c875"}]: dispatch 2024-04-02T14:27:54.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:27:53 smithi177 ceph-mon[36431]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "38cc68ea-c109-4d8d-b77d-1c6f9b56c875"}]': finished 2024-04-02T14:27:54.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:27:53 smithi177 ceph-mon[36431]: osdmap e30: 5 total, 4 up, 5 in 2024-04-02T14:27:54.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:27:53 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-02T14:27:54.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:27:53 smithi090 ceph-mon[31674]: pgmap v118: 1 pgs: 1 remapped+peering; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-04-02T14:27:54.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:27:53 smithi090 ceph-mon[31674]: from='client.? 172.21.15.92:0/2582948796' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "38cc68ea-c109-4d8d-b77d-1c6f9b56c875"}]: dispatch 2024-04-02T14:27:54.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:27:53 smithi090 ceph-mon[31674]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "38cc68ea-c109-4d8d-b77d-1c6f9b56c875"}]: dispatch 2024-04-02T14:27:54.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:27:53 smithi090 ceph-mon[31674]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "38cc68ea-c109-4d8d-b77d-1c6f9b56c875"}]': finished 2024-04-02T14:27:54.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:27:53 smithi090 ceph-mon[31674]: osdmap e30: 5 total, 4 up, 5 in 2024-04-02T14:27:54.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:27:53 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-02T14:27:55.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:27:54 smithi092 ceph-mon[36497]: from='client.? 172.21.15.92:0/331902961' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-02T14:27:55.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:27:54 smithi177 ceph-mon[36431]: from='client.? 172.21.15.92:0/331902961' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-02T14:27:55.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:27:54 smithi090 ceph-mon[31674]: from='client.? 172.21.15.92:0/331902961' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-02T14:27:56.110 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:27:55 smithi092 ceph-mon[36497]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail; 97 KiB/s, 0 objects/s recovering 2024-04-02T14:27:56.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:27:55 smithi177 ceph-mon[36431]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail; 97 KiB/s, 0 objects/s recovering 2024-04-02T14:27:56.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:27:55 smithi090 ceph-mon[31674]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail; 97 KiB/s, 0 objects/s recovering 2024-04-02T14:27:58.110 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:27:57 smithi092 ceph-mon[36497]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail; 83 KiB/s, 0 objects/s recovering 2024-04-02T14:27:58.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:27:57 smithi177 ceph-mon[36431]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail; 83 KiB/s, 0 objects/s recovering 2024-04-02T14:27:58.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:27:57 smithi090 ceph-mon[31674]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail; 83 KiB/s, 0 objects/s recovering 2024-04-02T14:27:59.977 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:27:59 smithi092 ceph-mon[36497]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-02T14:28:00.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:27:59 smithi177 ceph-mon[36431]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-02T14:28:00.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:27:59 smithi090 ceph-mon[31674]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-02T14:28:01.860 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:28:01 smithi092 ceph-mon[36497]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-02T14:28:01.861 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:28:01 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-02T14:28:01.861 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:28:01 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:28:02.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:28:01 smithi177 ceph-mon[36431]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-02T14:28:02.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:28:01 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-02T14:28:02.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:28:01 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:28:02.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:28:01 smithi090 ceph-mon[31674]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-02T14:28:02.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:28:01 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-02T14:28:02.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:28:01 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:28:02.830 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:28:02 smithi092 ceph-mon[36497]: Deploying daemon osd.4 on smithi092 2024-04-02T14:28:03.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:28:02 smithi177 ceph-mon[36431]: Deploying daemon osd.4 on smithi092 2024-04-02T14:28:03.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:28:02 smithi090 ceph-mon[31674]: Deploying daemon osd.4 on smithi092 2024-04-02T14:28:04.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:28:03 smithi092 ceph-mon[36497]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-02T14:28:04.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:28:03 smithi177 ceph-mon[36431]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-02T14:28:04.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:28:03 smithi090 ceph-mon[31674]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-02T14:28:05.861 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:28:05 smithi092 ceph-mon[36497]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-04-02T14:28:06.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:28:05 smithi177 ceph-mon[36431]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-04-02T14:28:06.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:28:05 smithi090 ceph-mon[31674]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-04-02T14:28:06.831 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:28:06 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T14:28:06.831 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:28:06 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:28:06.831 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:28:06 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:28:07.090 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:28:06 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T14:28:07.090 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:28:06 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:28:07.090 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:28:06 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:28:07.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:28:06 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T14:28:07.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:28:06 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:28:07.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:28:06 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:28:08.011 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:28:07 smithi092 ceph-mon[36497]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-04-02T14:28:08.071 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:28:07 smithi090 ceph-mon[31674]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-04-02T14:28:08.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:28:07 smithi177 ceph-mon[36431]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-04-02T14:28:09.481 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:28:09.479+0000 7f497f7fe700 1 -- 172.21.15.92:0/214522837 <== mgr.14152 v2:172.21.15.90:6800/4266054817 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f4988062a30 con 0x7f49740a3c40 2024-04-02T14:28:09.481 INFO:teuthology.orchestra.run.smithi092.stdout:Created osd(s) 4 on host 'smithi092' 2024-04-02T14:28:09.483 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:28:09.481+0000 7f4990715700 1 -- 172.21.15.92:0/214522837 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f49740a3c40 msgr2=0x7f49740a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:28:09.483 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:28:09.481+0000 7f4990715700 1 --2- 172.21.15.92:0/214522837 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f49740a3c40 0x7f49740a6100 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f49881036a0 tx=0x7f4984009260 comp rx=0 tx=0).stop 2024-04-02T14:28:09.483 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:28:09.481+0000 7f4990715700 1 -- 172.21.15.92:0/214522837 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4988103840 msgr2=0x7f4988198da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:28:09.483 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:28:09.481+0000 7f4990715700 1 --2- 172.21.15.92:0/214522837 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4988103840 0x7f4988198da0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f4978005910 tx=0x7f497800fa00 comp rx=0 tx=0).stop 2024-04-02T14:28:09.483 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:28:09.481+0000 7f4990715700 1 -- 172.21.15.92:0/214522837 shutdown_connections 2024-04-02T14:28:09.483 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:28:09.481+0000 7f4990715700 1 --2- 172.21.15.92:0/214522837 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f49740a3c40 0x7f49740a6100 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:28:09.483 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:28:09.481+0000 7f4990715700 1 --2- 172.21.15.92:0/214522837 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f49881041e0 0x7f498819fe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:28:09.484 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:28:09.481+0000 7f4990715700 1 --2- 172.21.15.92:0/214522837 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4988102640 0x7f4988198860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:28:09.484 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:28:09.481+0000 7f4990715700 1 --2- 172.21.15.92:0/214522837 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4988103840 0x7f4988198da0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:28:09.484 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:28:09.481+0000 7f4990715700 1 -- 172.21.15.92:0/214522837 >> 172.21.15.92:0/214522837 conn(0x7f49880fdc00 msgr2=0x7f4988106ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:28:09.484 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:28:09.481+0000 7f4990715700 1 -- 172.21.15.92:0/214522837 shutdown_connections 2024-04-02T14:28:09.484 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:28:09.481+0000 7f4990715700 1 -- 172.21.15.92:0/214522837 wait complete. 2024-04-02T14:28:10.008 DEBUG:teuthology.orchestra.run.smithi092:osd.4> sudo journalctl -f -n 0 -u ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4@osd.4.service 2024-04-02T14:28:10.010 INFO:tasks.cephadm:Deploying osd.5 on smithi177 with /dev/vg_nvme/lv_3... 2024-04-02T14:28:10.010 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:191d963ac9de7322da8aff20b567b3fa458e2ef6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6449448a-f0fc-11ee-b647-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-02T14:28:10.039 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:28:09 smithi092 ceph-mon[36497]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-04-02T14:28:10.039 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:28:09 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:28:10.039 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:28:09 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:28:10.039 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:28:09 smithi177 ceph-mon[36431]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-04-02T14:28:10.039 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:28:09 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:28:10.040 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:28:09 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:28:10.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:28:09 smithi090 ceph-mon[31674]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-04-02T14:28:10.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:28:09 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:28:10.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:28:09 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:28:10.254 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/mon.c/config 2024-04-02T14:28:11.068 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:28:10 smithi092 ceph-mon[36497]: from='osd.4 [v2:172.21.15.92:6808/2987208879,v1:172.21.15.92:6809/2987208879]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-02T14:28:11.069 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:28:10 smithi092 ceph-mon[36497]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-02T14:28:11.069 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:28:10 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:28:11.069 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:28:10 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:28:11.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:28:10 smithi177 ceph-mon[36431]: from='osd.4 [v2:172.21.15.92:6808/2987208879,v1:172.21.15.92:6809/2987208879]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-02T14:28:11.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:28:10 smithi177 ceph-mon[36431]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-02T14:28:11.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:28:10 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:28:11.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:28:10 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:28:11.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:28:10 smithi090 ceph-mon[31674]: from='osd.4 [v2:172.21.15.92:6808/2987208879,v1:172.21.15.92:6809/2987208879]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-02T14:28:11.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:28:10 smithi090 ceph-mon[31674]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-02T14:28:11.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:28:10 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:28:11.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:28:10 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:28:12.083 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:28:11 smithi092 ceph-mon[36497]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-04-02T14:28:12.083 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:28:11 smithi092 ceph-mon[36497]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-02T14:28:12.083 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:28:11 smithi092 ceph-mon[36497]: from='osd.4 [v2:172.21.15.92:6808/2987208879,v1:172.21.15.92:6809/2987208879]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]: dispatch 2024-04-02T14:28:12.083 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:28:11 smithi092 ceph-mon[36497]: osdmap e31: 5 total, 4 up, 5 in 2024-04-02T14:28:12.083 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:28:11 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-02T14:28:12.083 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:28:11 smithi092 ceph-mon[36497]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]: dispatch 2024-04-02T14:28:12.083 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:28:11 smithi092 ceph-mon[36497]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]': finished 2024-04-02T14:28:12.083 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:28:11 smithi092 ceph-mon[36497]: osdmap e32: 5 total, 4 up, 5 in 2024-04-02T14:28:12.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:28:11 smithi177 ceph-mon[36431]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-04-02T14:28:12.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:28:11 smithi177 ceph-mon[36431]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-02T14:28:12.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:28:11 smithi177 ceph-mon[36431]: from='osd.4 [v2:172.21.15.92:6808/2987208879,v1:172.21.15.92:6809/2987208879]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]: dispatch 2024-04-02T14:28:12.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:28:11 smithi177 ceph-mon[36431]: osdmap e31: 5 total, 4 up, 5 in 2024-04-02T14:28:12.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:28:11 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-02T14:28:12.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:28:11 smithi177 ceph-mon[36431]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]: dispatch 2024-04-02T14:28:12.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:28:11 smithi177 ceph-mon[36431]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]': finished 2024-04-02T14:28:12.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:28:11 smithi177 ceph-mon[36431]: osdmap e32: 5 total, 4 up, 5 in 2024-04-02T14:28:12.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:28:11 smithi090 ceph-mon[31674]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-04-02T14:28:12.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:28:11 smithi090 ceph-mon[31674]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-02T14:28:12.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:28:11 smithi090 ceph-mon[31674]: from='osd.4 [v2:172.21.15.92:6808/2987208879,v1:172.21.15.92:6809/2987208879]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]: dispatch 2024-04-02T14:28:12.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:28:11 smithi090 ceph-mon[31674]: osdmap e31: 5 total, 4 up, 5 in 2024-04-02T14:28:12.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:28:11 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-02T14:28:12.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:28:11 smithi090 ceph-mon[31674]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]: dispatch 2024-04-02T14:28:12.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:28:11 smithi090 ceph-mon[31674]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]': finished 2024-04-02T14:28:12.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:28:11 smithi090 ceph-mon[31674]: osdmap e32: 5 total, 4 up, 5 in 2024-04-02T14:28:12.348 INFO:teuthology.orchestra.run.smithi177.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-02T14:28:12.348 INFO:teuthology.orchestra.run.smithi177.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-02T14:28:12.348 INFO:teuthology.orchestra.run.smithi177.stderr: stderr: 10+0 records in 2024-04-02T14:28:12.348 INFO:teuthology.orchestra.run.smithi177.stderr:10+0 records out 2024-04-02T14:28:12.348 INFO:teuthology.orchestra.run.smithi177.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0122275 s, 858 MB/s 2024-04-02T14:28:12.348 INFO:teuthology.orchestra.run.smithi177.stderr:--> Zapping successful for: 2024-04-02T14:28:12.939 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:191d963ac9de7322da8aff20b567b3fa458e2ef6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6449448a-f0fc-11ee-b647-cb9ed24678a4 -- ceph orch daemon add osd smithi177:vg_nvme/lv_3 2024-04-02T14:28:12.964 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:28:12 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-02T14:28:12.964 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:28:12 smithi092 ceph-mon[36497]: from='osd.4 ' entity='osd.4' 2024-04-02T14:28:12.964 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:28:12 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-02T14:28:12.964 INFO:journalctl@ceph.osd.4.smithi092.stdout:Apr 02 14:28:12 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-4[47625]: 2024-04-02T14:28:12.704+0000 7f30767b8700 -1 osd.4 0 waiting for initial osdmap 2024-04-02T14:28:12.964 INFO:journalctl@ceph.osd.4.smithi092.stdout:Apr 02 14:28:12 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-4[47625]: 2024-04-02T14:28:12.877+0000 7f306e5a3700 -1 osd.4 32 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-02T14:28:13.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:28:12 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-02T14:28:13.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:28:12 smithi177 ceph-mon[36431]: from='osd.4 ' entity='osd.4' 2024-04-02T14:28:13.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:28:12 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-02T14:28:13.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:28:12 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-02T14:28:13.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:28:12 smithi090 ceph-mon[31674]: from='osd.4 ' entity='osd.4' 2024-04-02T14:28:13.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:28:12 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-02T14:28:13.224 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/mon.c/config 2024-04-02T14:28:13.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:28:13 smithi177 ceph-mon[36431]: purged_snaps scrub starts 2024-04-02T14:28:13.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:28:13 smithi177 ceph-mon[36431]: purged_snaps scrub ok 2024-04-02T14:28:13.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:28:13 smithi177 ceph-mon[36431]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-04-02T14:28:13.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:28:13 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:28:13.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:28:13 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:28:13.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:28:13 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-02T14:28:13.938 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:28:13 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-02T14:28:13.938 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:28:13 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:28:13.938 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:28:13 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:28:13.938 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:28:13 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T14:28:13.938 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:28:13 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:28:13.938 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:28:13 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-02T14:28:14.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:28:13 smithi092 ceph-mon[36497]: purged_snaps scrub starts 2024-04-02T14:28:14.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:28:13 smithi092 ceph-mon[36497]: purged_snaps scrub ok 2024-04-02T14:28:14.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:28:13 smithi092 ceph-mon[36497]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-04-02T14:28:14.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:28:13 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:28:14.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:28:13 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:28:14.112 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:28:13 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-02T14:28:14.112 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:28:13 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-02T14:28:14.112 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:28:13 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:28:14.112 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:28:13 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:28:14.112 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:28:13 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T14:28:14.112 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:28:13 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:28:14.113 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:28:13 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-02T14:28:14.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:28:13 smithi090 ceph-mon[31674]: purged_snaps scrub starts 2024-04-02T14:28:14.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:28:13 smithi090 ceph-mon[31674]: purged_snaps scrub ok 2024-04-02T14:28:14.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:28:13 smithi090 ceph-mon[31674]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-04-02T14:28:14.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:28:13 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:28:14.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:28:13 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:28:14.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:28:13 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-02T14:28:14.190 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:28:13 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-02T14:28:14.190 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:28:13 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:28:14.190 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:28:13 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:28:14.190 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:28:13 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T14:28:14.190 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:28:13 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:28:14.190 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:28:13 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-02T14:28:14.845 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:28:14.841+0000 7f6d8a619700 1 -- 172.21.15.177:0/2703820204 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6d84102720 msgr2=0x7f6d84102b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:28:14.845 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:28:14.841+0000 7f6d8a619700 1 --2- 172.21.15.177:0/2703820204 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6d84102720 0x7f6d84102b80 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f6d78009990 tx=0x7f6d78009ca0 comp rx=0 tx=0).stop 2024-04-02T14:28:14.845 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:28:14.841+0000 7f6d8a619700 1 -- 172.21.15.177:0/2703820204 shutdown_connections 2024-04-02T14:28:14.845 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:28:14.841+0000 7f6d8a619700 1 --2- 172.21.15.177:0/2703820204 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6d841030c0 0x7f6d84109970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:28:14.846 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:28:14.841+0000 7f6d8a619700 1 --2- 172.21.15.177:0/2703820204 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6d84101520 0x7f6d84101940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:28:14.846 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:28:14.841+0000 7f6d8a619700 1 --2- 172.21.15.177:0/2703820204 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6d84102720 0x7f6d84102b80 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:28:14.846 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:28:14.841+0000 7f6d8a619700 1 -- 172.21.15.177:0/2703820204 >> 172.21.15.177:0/2703820204 conn(0x7f6d840fcac0 msgr2=0x7f6d840fef00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:28:14.846 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:28:14.841+0000 7f6d8a619700 1 -- 172.21.15.177:0/2703820204 shutdown_connections 2024-04-02T14:28:14.846 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:28:14.841+0000 7f6d8a619700 1 -- 172.21.15.177:0/2703820204 wait complete. 2024-04-02T14:28:14.846 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:28:14.841+0000 7f6d8a619700 1 Processor -- start 2024-04-02T14:28:14.846 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:28:14.841+0000 7f6d8a619700 1 -- start start 2024-04-02T14:28:14.846 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:28:14.842+0000 7f6d8a619700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6d84101520 0x7f6d84198910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:28:14.846 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:28:14.842+0000 7f6d8a619700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6d84102720 0x7f6d84198e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:28:14.848 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:28:14.842+0000 7f6d8a619700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6d841030c0 0x7f6d8419fed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:28:14.848 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:28:14.842+0000 7f6d8a619700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f6d8410d1b0 con 0x7f6d84101520 2024-04-02T14:28:14.848 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:28:14.842+0000 7f6d8a619700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f6d8410d030 con 0x7f6d841030c0 2024-04-02T14:28:14.848 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:28:14.842+0000 7f6d8a619700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f6d8410d330 con 0x7f6d84102720 2024-04-02T14:28:14.849 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:28:14.842+0000 7f6d837fe700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6d84102720 0x7f6d84198e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:28:14.849 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:28:14.842+0000 7f6d837fe700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6d84102720 0x7f6d84198e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.177:51944/0 (socket says 172.21.15.177:51944) 2024-04-02T14:28:14.849 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:28:14.842+0000 7f6d837fe700 1 -- 172.21.15.177:0/2612661773 learned_addr learned my addr 172.21.15.177:0/2612661773 (peer_addr_for_me v2:172.21.15.177:0/0) 2024-04-02T14:28:14.849 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:28:14.842+0000 7f6d88bb6700 1 --2- 172.21.15.177:0/2612661773 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6d841030c0 0x7f6d8419fed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:28:14.849 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:28:14.842+0000 7f6d837fe700 1 -- 172.21.15.177:0/2612661773 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6d841030c0 msgr2=0x7f6d8419fed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:28:14.849 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:28:14.842+0000 7f6d837fe700 1 --2- 172.21.15.177:0/2612661773 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6d841030c0 0x7f6d8419fed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:28:14.849 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:28:14.842+0000 7f6d837fe700 1 -- 172.21.15.177:0/2612661773 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6d84101520 msgr2=0x7f6d84198910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:28:14.849 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:28:14.842+0000 7f6d837fe700 1 --2- 172.21.15.177:0/2612661773 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6d84101520 0x7f6d84198910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:28:14.849 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:28:14.842+0000 7f6d837fe700 1 -- 172.21.15.177:0/2612661773 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d841a0570 con 0x7f6d84102720 2024-04-02T14:28:14.849 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:28:14.842+0000 7f6d837fe700 1 --2- 172.21.15.177:0/2612661773 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6d84102720 0x7f6d84198e50 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f6d78000c00 tx=0x7f6d78005e80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:28:14.849 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:28:14.843+0000 7f6d817fa700 1 -- 172.21.15.177:0/2612661773 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d78027070 con 0x7f6d84102720 2024-04-02T14:28:14.849 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:28:14.843+0000 7f6d8a619700 1 -- 172.21.15.177:0/2612661773 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d841a0800 con 0x7f6d84102720 2024-04-02T14:28:14.849 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:28:14.843+0000 7f6d817fa700 1 -- 172.21.15.177:0/2612661773 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6d78005690 con 0x7f6d84102720 2024-04-02T14:28:14.849 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:28:14.843+0000 7f6d817fa700 1 -- 172.21.15.177:0/2612661773 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d78022da0 con 0x7f6d84102720 2024-04-02T14:28:14.849 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:28:14.843+0000 7f6d8a619700 1 -- 172.21.15.177:0/2612661773 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d841a0c60 con 0x7f6d84102720 2024-04-02T14:28:14.850 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:28:14.844+0000 7f6d8a619700 1 -- 172.21.15.177:0/2612661773 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6d840692b0 con 0x7f6d84102720 2024-04-02T14:28:14.850 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:28:14.845+0000 7f6d817fa700 1 -- 172.21.15.177:0/2612661773 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f6d7802a070 con 0x7f6d84102720 2024-04-02T14:28:14.850 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:28:14.845+0000 7f6d817fa700 1 --2- 172.21.15.177:0/2612661773 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f6d6c0a3c40 0x7f6d6c0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:28:14.850 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:28:14.845+0000 7f6d817fa700 1 -- 172.21.15.177:0/2612661773 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 4101+0+0 (secure 0 0 0) 0x7f6d780d0480 con 0x7f6d84102720 2024-04-02T14:28:14.850 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:28:14.846+0000 7f6d83fff700 1 --2- 172.21.15.177:0/2612661773 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f6d6c0a3c40 0x7f6d6c0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:28:14.850 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:28:14.846+0000 7f6d83fff700 1 --2- 172.21.15.177:0/2612661773 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f6d6c0a3c40 0x7f6d6c0a6100 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f6d74005fd0 tx=0x7f6d74005d00 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:28:14.851 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:28:14.850+0000 7f6d817fa700 1 -- 172.21.15.177:0/2612661773 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7f6d78013070 con 0x7f6d84102720 2024-04-02T14:28:15.058 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:28:15.057+0000 7f6d8a619700 1 -- 172.21.15.177:0/2612661773 --> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi177:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f6d841a0f80 con 0x7f6d6c0a3c40 2024-04-02T14:28:15.059 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:28:14 smithi177 ceph-mon[36431]: Detected new or changed devices on smithi092 2024-04-02T14:28:15.059 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:28:14 smithi177 ceph-mon[36431]: Adjusting osd_memory_target on smithi092 to 8540M 2024-04-02T14:28:15.059 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:28:14 smithi177 ceph-mon[36431]: osd.4 [v2:172.21.15.92:6808/2987208879,v1:172.21.15.92:6809/2987208879] boot 2024-04-02T14:28:15.059 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:28:14 smithi177 ceph-mon[36431]: osdmap e33: 5 total, 5 up, 5 in 2024-04-02T14:28:15.059 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:28:14 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-02T14:28:15.059 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:28:14 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T14:28:15.059 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:28:14 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:28:15.059 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:28:14 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T14:28:15.059 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:28:14 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:28:15.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:28:14 smithi092 ceph-mon[36497]: Detected new or changed devices on smithi092 2024-04-02T14:28:15.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:28:14 smithi092 ceph-mon[36497]: Adjusting osd_memory_target on smithi092 to 8540M 2024-04-02T14:28:15.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:28:14 smithi092 ceph-mon[36497]: osd.4 [v2:172.21.15.92:6808/2987208879,v1:172.21.15.92:6809/2987208879] boot 2024-04-02T14:28:15.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:28:14 smithi092 ceph-mon[36497]: osdmap e33: 5 total, 5 up, 5 in 2024-04-02T14:28:15.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:28:14 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-02T14:28:15.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:28:14 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T14:28:15.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:28:14 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:28:15.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:28:14 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T14:28:15.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:28:14 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:28:15.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:28:14 smithi090 ceph-mon[31674]: Detected new or changed devices on smithi092 2024-04-02T14:28:15.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:28:14 smithi090 ceph-mon[31674]: Adjusting osd_memory_target on smithi092 to 8540M 2024-04-02T14:28:15.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:28:14 smithi090 ceph-mon[31674]: osd.4 [v2:172.21.15.92:6808/2987208879,v1:172.21.15.92:6809/2987208879] boot 2024-04-02T14:28:15.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:28:14 smithi090 ceph-mon[31674]: osdmap e33: 5 total, 5 up, 5 in 2024-04-02T14:28:15.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:28:14 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-02T14:28:15.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:28:14 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T14:28:15.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:28:14 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:28:15.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:28:14 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T14:28:15.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:28:14 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:28:15.981 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:28:15 smithi177 ceph-mon[36431]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail 2024-04-02T14:28:15.982 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:28:15 smithi177 ceph-mon[36431]: osdmap e34: 5 total, 5 up, 5 in 2024-04-02T14:28:15.982 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:28:15 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-02T14:28:15.982 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:28:15 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-02T14:28:15.982 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:28:15 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:28:16.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:28:15 smithi092 ceph-mon[36497]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail 2024-04-02T14:28:16.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:28:15 smithi092 ceph-mon[36497]: osdmap e34: 5 total, 5 up, 5 in 2024-04-02T14:28:16.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:28:15 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-02T14:28:16.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:28:15 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-02T14:28:16.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:28:15 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:28:16.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:28:15 smithi090 ceph-mon[31674]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail 2024-04-02T14:28:16.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:28:15 smithi090 ceph-mon[31674]: osdmap e34: 5 total, 5 up, 5 in 2024-04-02T14:28:16.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:28:15 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-02T14:28:16.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:28:15 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-02T14:28:16.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:28:15 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:28:16.874 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:28:16 smithi177 ceph-mon[36431]: from='client.24247 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi177:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T14:28:17.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:28:16 smithi092 ceph-mon[36497]: from='client.24247 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi177:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T14:28:17.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:28:16 smithi090 ceph-mon[31674]: from='client.24247 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi177:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T14:28:18.106 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:28:17 smithi177 ceph-mon[36431]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-04-02T14:28:18.106 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:28:17 smithi177 ceph-mon[36431]: from='client.? 172.21.15.177:0/1164797006' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2347232e-ac91-466f-8ac9-496f397679ba"}]: dispatch 2024-04-02T14:28:18.106 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:28:17 smithi177 ceph-mon[36431]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2347232e-ac91-466f-8ac9-496f397679ba"}]: dispatch 2024-04-02T14:28:18.106 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:28:17 smithi177 ceph-mon[36431]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2347232e-ac91-466f-8ac9-496f397679ba"}]': finished 2024-04-02T14:28:18.106 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:28:17 smithi177 ceph-mon[36431]: osdmap e35: 6 total, 5 up, 6 in 2024-04-02T14:28:18.106 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:28:17 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-02T14:28:18.110 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:28:17 smithi092 ceph-mon[36497]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-04-02T14:28:18.110 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:28:17 smithi092 ceph-mon[36497]: from='client.? 172.21.15.177:0/1164797006' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2347232e-ac91-466f-8ac9-496f397679ba"}]: dispatch 2024-04-02T14:28:18.110 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:28:17 smithi092 ceph-mon[36497]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2347232e-ac91-466f-8ac9-496f397679ba"}]: dispatch 2024-04-02T14:28:18.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:28:17 smithi092 ceph-mon[36497]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2347232e-ac91-466f-8ac9-496f397679ba"}]': finished 2024-04-02T14:28:18.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:28:17 smithi092 ceph-mon[36497]: osdmap e35: 6 total, 5 up, 6 in 2024-04-02T14:28:18.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:28:17 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-02T14:28:18.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:28:17 smithi090 ceph-mon[31674]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-04-02T14:28:18.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:28:17 smithi090 ceph-mon[31674]: from='client.? 172.21.15.177:0/1164797006' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2347232e-ac91-466f-8ac9-496f397679ba"}]: dispatch 2024-04-02T14:28:18.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:28:17 smithi090 ceph-mon[31674]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2347232e-ac91-466f-8ac9-496f397679ba"}]: dispatch 2024-04-02T14:28:18.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:28:17 smithi090 ceph-mon[31674]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2347232e-ac91-466f-8ac9-496f397679ba"}]': finished 2024-04-02T14:28:18.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:28:17 smithi090 ceph-mon[31674]: osdmap e35: 6 total, 5 up, 6 in 2024-04-02T14:28:18.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:28:17 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-02T14:28:19.110 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:28:18 smithi092 ceph-mon[36497]: from='client.? 172.21.15.177:0/1329151717' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-02T14:28:19.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:28:18 smithi177 ceph-mon[36431]: from='client.? 172.21.15.177:0/1329151717' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-02T14:28:19.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:28:18 smithi090 ceph-mon[31674]: from='client.? 172.21.15.177:0/1329151717' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-02T14:28:20.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:28:19 smithi177 ceph-mon[36431]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-04-02T14:28:20.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:28:19 smithi090 ceph-mon[31674]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-04-02T14:28:20.362 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:28:19 smithi092 ceph-mon[36497]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-04-02T14:28:22.142 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:28:21 smithi177 ceph-mon[36431]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-04-02T14:28:22.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:28:21 smithi090 ceph-mon[31674]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-04-02T14:28:22.360 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:28:21 smithi092 ceph-mon[36497]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-04-02T14:28:23.973 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:28:23 smithi177 ceph-mon[36431]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-04-02T14:28:24.163 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:28:23 smithi092 ceph-mon[36497]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-04-02T14:28:24.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:28:23 smithi090 ceph-mon[31674]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-04-02T14:28:26.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:28:25 smithi177 ceph-mon[36431]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-04-02T14:28:26.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:28:25 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-02T14:28:26.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:28:25 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:28:26.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:28:25 smithi090 ceph-mon[31674]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-04-02T14:28:26.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:28:25 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-02T14:28:26.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:28:25 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:28:26.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:28:25 smithi092 ceph-mon[36497]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-04-02T14:28:26.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:28:25 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-02T14:28:26.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:28:25 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:28:26.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:28:26 smithi177 ceph-mon[36431]: Deploying daemon osd.5 on smithi177 2024-04-02T14:28:27.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:28:26 smithi090 ceph-mon[31674]: Deploying daemon osd.5 on smithi177 2024-04-02T14:28:27.360 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:28:26 smithi092 ceph-mon[36497]: Deploying daemon osd.5 on smithi177 2024-04-02T14:28:28.129 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:28:27 smithi177 ceph-mon[36431]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-04-02T14:28:28.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:28:27 smithi090 ceph-mon[31674]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-04-02T14:28:28.362 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:28:27 smithi092 ceph-mon[36497]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-04-02T14:28:29.189 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:28:28 smithi177 ceph-mon[36431]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-04-02T14:28:29.190 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:28:28 smithi090 ceph-mon[31674]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-04-02T14:28:29.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:28:28 smithi092 ceph-mon[36497]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-04-02T14:28:30.436 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:28:30 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T14:28:30.436 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:28:30 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:28:30.436 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:28:30 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:28:30.611 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:28:30 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T14:28:30.611 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:28:30 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:28:30.611 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:28:30 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:28:30.688 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:28:30 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T14:28:30.689 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:28:30 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:28:30.689 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:28:30 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:28:31.436 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:28:31 smithi177 ceph-mon[36431]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-04-02T14:28:31.610 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:28:31 smithi092 ceph-mon[36497]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-04-02T14:28:31.688 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:28:31 smithi090 ceph-mon[31674]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-04-02T14:28:33.695 INFO:teuthology.orchestra.run.smithi177.stdout:Created osd(s) 5 on host 'smithi177' 2024-04-02T14:28:33.696 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:28:33.693+0000 7f6d817fa700 1 -- 172.21.15.177:0/2612661773 <== mgr.14152 v2:172.21.15.90:6800/4266054817 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f6d841a0f80 con 0x7f6d6c0a3c40 2024-04-02T14:28:33.697 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:28:33.696+0000 7f6d8a619700 1 -- 172.21.15.177:0/2612661773 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f6d6c0a3c40 msgr2=0x7f6d6c0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:28:33.697 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:28:33.696+0000 7f6d8a619700 1 --2- 172.21.15.177:0/2612661773 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f6d6c0a3c40 0x7f6d6c0a6100 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f6d74005fd0 tx=0x7f6d74005d00 comp rx=0 tx=0).stop 2024-04-02T14:28:33.697 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:28:33.696+0000 7f6d8a619700 1 -- 172.21.15.177:0/2612661773 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6d84102720 msgr2=0x7f6d84198e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:28:33.697 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:28:33.696+0000 7f6d8a619700 1 --2- 172.21.15.177:0/2612661773 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6d84102720 0x7f6d84198e50 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f6d78000c00 tx=0x7f6d78005e80 comp rx=0 tx=0).stop 2024-04-02T14:28:33.698 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:28:33.696+0000 7f6d8a619700 1 -- 172.21.15.177:0/2612661773 shutdown_connections 2024-04-02T14:28:33.698 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:28:33.696+0000 7f6d8a619700 1 --2- 172.21.15.177:0/2612661773 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f6d6c0a3c40 0x7f6d6c0a6100 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:28:33.698 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:28:33.696+0000 7f6d8a619700 1 --2- 172.21.15.177:0/2612661773 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6d84101520 0x7f6d84198910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:28:33.698 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:28:33.696+0000 7f6d8a619700 1 --2- 172.21.15.177:0/2612661773 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6d841030c0 0x7f6d8419fed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:28:33.698 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:28:33.696+0000 7f6d8a619700 1 --2- 172.21.15.177:0/2612661773 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6d84102720 0x7f6d84198e50 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:28:33.698 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:28:33.696+0000 7f6d8a619700 1 -- 172.21.15.177:0/2612661773 >> 172.21.15.177:0/2612661773 conn(0x7f6d840fcac0 msgr2=0x7f6d840fe580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:28:33.698 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:28:33.696+0000 7f6d8a619700 1 -- 172.21.15.177:0/2612661773 shutdown_connections 2024-04-02T14:28:33.698 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:28:33.696+0000 7f6d8a619700 1 -- 172.21.15.177:0/2612661773 wait complete. 2024-04-02T14:28:33.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:28:33 smithi177 ceph-mon[36431]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-04-02T14:28:33.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:28:33 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:28:33.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:28:33 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:28:33.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:28:33 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:28:34.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:28:33 smithi092 ceph-mon[36497]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-04-02T14:28:34.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:28:33 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:28:34.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:28:33 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:28:34.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:28:33 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:28:34.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:28:33 smithi090 ceph-mon[31674]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-04-02T14:28:34.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:28:33 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:28:34.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:28:33 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:28:34.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:28:33 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:28:34.312 DEBUG:teuthology.orchestra.run.smithi177:osd.5> sudo journalctl -f -n 0 -u ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4@osd.5.service 2024-04-02T14:28:34.314 INFO:tasks.cephadm:Deploying osd.6 on smithi090 with /dev/vg_nvme/lv_2... 2024-04-02T14:28:34.314 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:191d963ac9de7322da8aff20b567b3fa458e2ef6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6449448a-f0fc-11ee-b647-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-02T14:28:34.765 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T14:28:34.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:28:34 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:28:34.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:28:34 smithi177 ceph-mon[36431]: from='osd.5 [v2:172.21.15.177:6808/3486820237,v1:172.21.15.177:6809/3486820237]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-02T14:28:34.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:28:34 smithi177 ceph-mon[36431]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-02T14:28:35.024 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:28:34 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:28:35.024 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:28:34 smithi090 ceph-mon[31674]: from='osd.5 [v2:172.21.15.177:6808/3486820237,v1:172.21.15.177:6809/3486820237]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-02T14:28:35.024 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:28:34 smithi090 ceph-mon[31674]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-02T14:28:35.110 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:28:34 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:28:35.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:28:34 smithi092 ceph-mon[36497]: from='osd.5 [v2:172.21.15.177:6808/3486820237,v1:172.21.15.177:6809/3486820237]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-02T14:28:35.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:28:34 smithi092 ceph-mon[36497]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-02T14:28:35.827 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:28:35 smithi177 ceph-mon[36431]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-04-02T14:28:35.827 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:28:35 smithi177 ceph-mon[36431]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-02T14:28:35.827 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:28:35 smithi177 ceph-mon[36431]: from='osd.5 [v2:172.21.15.177:6808/3486820237,v1:172.21.15.177:6809/3486820237]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-04-02T14:28:35.827 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:28:35 smithi177 ceph-mon[36431]: osdmap e36: 6 total, 5 up, 6 in 2024-04-02T14:28:35.827 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:28:35 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-02T14:28:35.827 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:28:35 smithi177 ceph-mon[36431]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-04-02T14:28:35.827 INFO:journalctl@ceph.osd.5.smithi177.stdout:Apr 02 14:28:35 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-5[48118]: 2024-04-02T14:28:35.747+0000 7f34e56e7700 -1 osd.5 0 waiting for initial osdmap 2024-04-02T14:28:35.827 INFO:journalctl@ceph.osd.5.smithi177.stdout:Apr 02 14:28:35 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-5[48118]: 2024-04-02T14:28:35.754+0000 7f34e14da700 -1 osd.5 37 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-02T14:28:36.107 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:28:35 smithi090 ceph-mon[31674]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-04-02T14:28:36.107 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:28:35 smithi090 ceph-mon[31674]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-02T14:28:36.107 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:28:35 smithi090 ceph-mon[31674]: from='osd.5 [v2:172.21.15.177:6808/3486820237,v1:172.21.15.177:6809/3486820237]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-04-02T14:28:36.107 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:28:35 smithi090 ceph-mon[31674]: osdmap e36: 6 total, 5 up, 6 in 2024-04-02T14:28:36.107 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:28:35 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-02T14:28:36.108 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:28:35 smithi090 ceph-mon[31674]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-04-02T14:28:36.110 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:28:35 smithi092 ceph-mon[36497]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-04-02T14:28:36.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:28:35 smithi092 ceph-mon[36497]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-02T14:28:36.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:28:35 smithi092 ceph-mon[36497]: from='osd.5 [v2:172.21.15.177:6808/3486820237,v1:172.21.15.177:6809/3486820237]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-04-02T14:28:36.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:28:35 smithi092 ceph-mon[36497]: osdmap e36: 6 total, 5 up, 6 in 2024-04-02T14:28:36.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:28:35 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-02T14:28:36.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:28:35 smithi092 ceph-mon[36497]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-04-02T14:28:36.870 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:28:36 smithi177 ceph-mon[36431]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]': finished 2024-04-02T14:28:36.870 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:28:36 smithi177 ceph-mon[36431]: osdmap e37: 6 total, 5 up, 6 in 2024-04-02T14:28:36.870 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:28:36 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-02T14:28:37.110 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:28:36 smithi092 ceph-mon[36497]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]': finished 2024-04-02T14:28:37.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:28:36 smithi092 ceph-mon[36497]: osdmap e37: 6 total, 5 up, 6 in 2024-04-02T14:28:37.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:28:36 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-02T14:28:37.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:28:36 smithi090 ceph-mon[31674]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]': finished 2024-04-02T14:28:37.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:28:36 smithi090 ceph-mon[31674]: osdmap e37: 6 total, 5 up, 6 in 2024-04-02T14:28:37.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:28:36 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-02T14:28:37.594 INFO:teuthology.orchestra.run.smithi090.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-02T14:28:37.594 INFO:teuthology.orchestra.run.smithi090.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-02T14:28:37.594 INFO:teuthology.orchestra.run.smithi090.stderr: stderr: 10+0 records in 2024-04-02T14:28:37.594 INFO:teuthology.orchestra.run.smithi090.stderr:10+0 records out 2024-04-02T14:28:37.595 INFO:teuthology.orchestra.run.smithi090.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0606143 s, 173 MB/s 2024-04-02T14:28:37.595 INFO:teuthology.orchestra.run.smithi090.stderr:--> Zapping successful for: 2024-04-02T14:28:37.939 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:28:37 smithi090 ceph-mon[31674]: purged_snaps scrub starts 2024-04-02T14:28:37.939 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:28:37 smithi090 ceph-mon[31674]: purged_snaps scrub ok 2024-04-02T14:28:37.939 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:28:37 smithi090 ceph-mon[31674]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-04-02T14:28:37.940 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:28:37 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-02T14:28:37.940 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:28:37 smithi090 ceph-mon[31674]: osd.5 [v2:172.21.15.177:6808/3486820237,v1:172.21.15.177:6809/3486820237] boot 2024-04-02T14:28:37.940 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:28:37 smithi090 ceph-mon[31674]: osdmap e38: 6 total, 6 up, 6 in 2024-04-02T14:28:37.940 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:28:37 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-02T14:28:37.940 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:28:37 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:28:37.940 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:28:37 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:28:37.940 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:28:37 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-02T14:28:37.940 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:28:37 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-02T14:28:37.940 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:28:37 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:28:37.940 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:28:37 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:28:37.940 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:28:37 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T14:28:37.940 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:28:37 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:28:37.940 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:28:37 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T14:28:37.940 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:28:37 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:28:37.941 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:28:37 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T14:28:37.941 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:28:37 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:28:38.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:28:37 smithi092 ceph-mon[36497]: purged_snaps scrub starts 2024-04-02T14:28:38.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:28:37 smithi092 ceph-mon[36497]: purged_snaps scrub ok 2024-04-02T14:28:38.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:28:37 smithi092 ceph-mon[36497]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-04-02T14:28:38.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:28:37 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-02T14:28:38.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:28:37 smithi092 ceph-mon[36497]: osd.5 [v2:172.21.15.177:6808/3486820237,v1:172.21.15.177:6809/3486820237] boot 2024-04-02T14:28:38.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:28:37 smithi092 ceph-mon[36497]: osdmap e38: 6 total, 6 up, 6 in 2024-04-02T14:28:38.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:28:37 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-02T14:28:38.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:28:37 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:28:38.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:28:37 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:28:38.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:28:37 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-02T14:28:38.112 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:28:37 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-02T14:28:38.112 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:28:37 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:28:38.112 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:28:37 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:28:38.112 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:28:37 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T14:28:38.112 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:28:37 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:28:38.112 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:28:37 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T14:28:38.112 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:28:37 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:28:38.112 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:28:37 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T14:28:38.112 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:28:37 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:28:38.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:28:37 smithi177 ceph-mon[36431]: purged_snaps scrub starts 2024-04-02T14:28:38.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:28:37 smithi177 ceph-mon[36431]: purged_snaps scrub ok 2024-04-02T14:28:38.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:28:37 smithi177 ceph-mon[36431]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-04-02T14:28:38.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:28:37 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-02T14:28:38.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:28:37 smithi177 ceph-mon[36431]: osd.5 [v2:172.21.15.177:6808/3486820237,v1:172.21.15.177:6809/3486820237] boot 2024-04-02T14:28:38.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:28:37 smithi177 ceph-mon[36431]: osdmap e38: 6 total, 6 up, 6 in 2024-04-02T14:28:38.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:28:37 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-02T14:28:38.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:28:37 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:28:38.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:28:37 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:28:38.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:28:37 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-02T14:28:38.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:28:37 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-02T14:28:38.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:28:37 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:28:38.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:28:37 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:28:38.188 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:28:37 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T14:28:38.188 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:28:37 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:28:38.188 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:28:37 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T14:28:38.188 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:28:37 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:28:38.188 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:28:37 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T14:28:38.188 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:28:37 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:28:38.487 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:191d963ac9de7322da8aff20b567b3fa458e2ef6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6449448a-f0fc-11ee-b647-cb9ed24678a4 -- ceph orch daemon add osd smithi090:vg_nvme/lv_2 2024-04-02T14:28:38.903 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T14:28:39.046 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:28:38 smithi090 ceph-mon[31674]: Detected new or changed devices on smithi177 2024-04-02T14:28:39.046 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:28:38 smithi090 ceph-mon[31674]: Adjusting osd_memory_target on smithi177 to 8540M 2024-04-02T14:28:39.046 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:28:38 smithi090 ceph-mon[31674]: osdmap e39: 6 total, 6 up, 6 in 2024-04-02T14:28:39.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:28:38 smithi092 ceph-mon[36497]: Detected new or changed devices on smithi177 2024-04-02T14:28:39.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:28:38 smithi092 ceph-mon[36497]: Adjusting osd_memory_target on smithi177 to 8540M 2024-04-02T14:28:39.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:28:38 smithi092 ceph-mon[36497]: osdmap e39: 6 total, 6 up, 6 in 2024-04-02T14:28:39.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:28:38 smithi177 ceph-mon[36431]: Detected new or changed devices on smithi177 2024-04-02T14:28:39.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:28:38 smithi177 ceph-mon[36431]: Adjusting osd_memory_target on smithi177 to 8540M 2024-04-02T14:28:39.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:28:38 smithi177 ceph-mon[36431]: osdmap e39: 6 total, 6 up, 6 in 2024-04-02T14:28:40.063 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:28:39 smithi090 ceph-mon[31674]: pgmap v151: 1 pgs: 1 remapped+peering; 577 KiB data, 182 MiB used, 536 GiB / 536 GiB avail 2024-04-02T14:28:40.063 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:28:39 smithi090 ceph-mon[31674]: osdmap e40: 6 total, 6 up, 6 in 2024-04-02T14:28:40.063 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:28:39 smithi090 ceph-mon[31674]: osdmap e41: 6 total, 6 up, 6 in 2024-04-02T14:28:40.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:28:39 smithi092 ceph-mon[36497]: pgmap v151: 1 pgs: 1 remapped+peering; 577 KiB data, 182 MiB used, 536 GiB / 536 GiB avail 2024-04-02T14:28:40.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:28:39 smithi092 ceph-mon[36497]: osdmap e40: 6 total, 6 up, 6 in 2024-04-02T14:28:40.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:28:39 smithi092 ceph-mon[36497]: osdmap e41: 6 total, 6 up, 6 in 2024-04-02T14:28:40.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:28:39 smithi177 ceph-mon[36431]: pgmap v151: 1 pgs: 1 remapped+peering; 577 KiB data, 182 MiB used, 536 GiB / 536 GiB avail 2024-04-02T14:28:40.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:28:39 smithi177 ceph-mon[36431]: osdmap e40: 6 total, 6 up, 6 in 2024-04-02T14:28:40.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:28:39 smithi177 ceph-mon[36431]: osdmap e41: 6 total, 6 up, 6 in 2024-04-02T14:28:40.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:28:40.965+0000 7ff4f3c00700 1 -- 172.21.15.90:0/2026890714 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff4ec108310 msgr2=0x7ff4ec10a780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:28:40.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:28:40.965+0000 7ff4f3c00700 1 --2- 172.21.15.90:0/2026890714 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff4ec108310 0x7ff4ec10a780 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7ff4e400b0a0 tx=0x7ff4e400b3b0 comp rx=0 tx=0).stop 2024-04-02T14:28:40.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:28:40.965+0000 7ff4f3c00700 1 -- 172.21.15.90:0/2026890714 shutdown_connections 2024-04-02T14:28:40.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:28:40.965+0000 7ff4f3c00700 1 --2- 172.21.15.90:0/2026890714 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff4ec108310 0x7ff4ec10a780 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:28:40.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:28:40.965+0000 7ff4f3c00700 1 --2- 172.21.15.90:0/2026890714 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff4ec1030b0 0x7ff4ec1054a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:28:40.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:28:40.965+0000 7ff4f3c00700 1 --2- 172.21.15.90:0/2026890714 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff4ec1059e0 0x7ff4ec107dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:28:40.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:28:40.965+0000 7ff4f3c00700 1 -- 172.21.15.90:0/2026890714 >> 172.21.15.90:0/2026890714 conn(0x7ff4ec0fcac0 msgr2=0x7ff4ec0fef00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:28:40.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:28:40.965+0000 7ff4f3c00700 1 -- 172.21.15.90:0/2026890714 shutdown_connections 2024-04-02T14:28:40.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:28:40.965+0000 7ff4f3c00700 1 -- 172.21.15.90:0/2026890714 wait complete. 2024-04-02T14:28:40.969 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:28:40.967+0000 7ff4f3c00700 1 Processor -- start 2024-04-02T14:28:40.969 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:28:40.967+0000 7ff4f3c00700 1 -- start start 2024-04-02T14:28:40.969 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:28:40.967+0000 7ff4f3c00700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff4ec1030b0 0x7ff4ec102a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:28:40.969 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:28:40.967+0000 7ff4f3c00700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff4ec1059e0 0x7ff4ec101060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:28:40.969 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:28:40.967+0000 7ff4f3c00700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff4ec108310 0x7ff4ec1015a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:28:40.970 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:28:40.967+0000 7ff4f3c00700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7ff4ec10e230 con 0x7ff4ec1059e0 2024-04-02T14:28:40.970 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:28:40.967+0000 7ff4f3c00700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7ff4ec10e0b0 con 0x7ff4ec108310 2024-04-02T14:28:40.970 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:28:40.967+0000 7ff4f3c00700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7ff4ec10e3b0 con 0x7ff4ec1030b0 2024-04-02T14:28:40.970 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:28:40.968+0000 7ff4ebe00700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff4ec1059e0 0x7ff4ec101060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:28:40.970 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:28:40.968+0000 7ff4f1200700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff4ec1030b0 0x7ff4ec102a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:28:40.970 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:28:40.968+0000 7ff4ebe00700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff4ec1059e0 0x7ff4ec101060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:51900/0 (socket says 172.21.15.90:51900) 2024-04-02T14:28:40.970 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:28:40.968+0000 7ff4ebe00700 1 -- 172.21.15.90:0/2589098727 learned_addr learned my addr 172.21.15.90:0/2589098727 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:28:40.970 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:28:40.968+0000 7ff4f1200700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff4ec1030b0 0x7ff4ec102a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:36048/0 (socket says 172.21.15.90:36048) 2024-04-02T14:28:40.970 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:28:40.968+0000 7ff4f1c00700 1 --2- 172.21.15.90:0/2589098727 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff4ec108310 0x7ff4ec1015a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:28:40.970 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:28:40.968+0000 7ff4f1200700 1 -- 172.21.15.90:0/2589098727 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff4ec108310 msgr2=0x7ff4ec1015a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:28:40.970 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:28:40.968+0000 7ff4f1200700 1 --2- 172.21.15.90:0/2589098727 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff4ec108310 0x7ff4ec1015a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:28:40.970 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:28:40.968+0000 7ff4f1200700 1 -- 172.21.15.90:0/2589098727 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff4ec1059e0 msgr2=0x7ff4ec101060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:28:40.970 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:28:40.968+0000 7ff4f1200700 1 --2- 172.21.15.90:0/2589098727 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff4ec1059e0 0x7ff4ec101060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:28:40.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:28:40.968+0000 7ff4f1200700 1 -- 172.21.15.90:0/2589098727 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4ec101e50 con 0x7ff4ec1030b0 2024-04-02T14:28:40.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:28:40.968+0000 7ff4ebe00700 1 --2- 172.21.15.90:0/2589098727 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff4ec1059e0 0x7ff4ec101060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:28:40.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:28:40.968+0000 7ff4f1200700 1 --2- 172.21.15.90:0/2589098727 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff4ec1030b0 0x7ff4ec102a10 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7ff4e000e9f0 tx=0x7ff4e000ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:28:40.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:28:40.969+0000 7ff4e9600700 1 -- 172.21.15.90:0/2589098727 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4e000c9c0 con 0x7ff4ec1030b0 2024-04-02T14:28:40.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:28:40.969+0000 7ff4e9600700 1 -- 172.21.15.90:0/2589098727 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff4e000cb60 con 0x7ff4ec1030b0 2024-04-02T14:28:40.972 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:28:40.969+0000 7ff4f3c00700 1 -- 172.21.15.90:0/2589098727 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4ec1a2a30 con 0x7ff4ec1030b0 2024-04-02T14:28:40.972 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:28:40.969+0000 7ff4e9600700 1 -- 172.21.15.90:0/2589098727 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4e0018680 con 0x7ff4ec1030b0 2024-04-02T14:28:40.972 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:28:40.969+0000 7ff4f3c00700 1 -- 172.21.15.90:0/2589098727 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4ec1a2f70 con 0x7ff4ec1030b0 2024-04-02T14:28:40.972 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:28:40.970+0000 7ff4f3c00700 1 -- 172.21.15.90:0/2589098727 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff4ec0692b0 con 0x7ff4ec1030b0 2024-04-02T14:28:40.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:28:40.972+0000 7ff4e9600700 1 -- 172.21.15.90:0/2589098727 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7ff4e0010430 con 0x7ff4ec1030b0 2024-04-02T14:28:40.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:28:40.973+0000 7ff4e9600700 1 --2- 172.21.15.90:0/2589098727 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7ff4d40a3cc0 0x7ff4d40a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:28:40.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:28:40.973+0000 7ff4e9600700 1 -- 172.21.15.90:0/2589098727 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 4533+0+0 (secure 0 0 0) 0x7ff4e00cf620 con 0x7ff4ec1030b0 2024-04-02T14:28:40.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:28:40.974+0000 7ff4ebe00700 1 --2- 172.21.15.90:0/2589098727 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7ff4d40a3cc0 0x7ff4d40a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:28:40.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:28:40.974+0000 7ff4ebe00700 1 --2- 172.21.15.90:0/2589098727 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7ff4d40a3cc0 0x7ff4d40a6180 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7ff4d8011550 tx=0x7ff4d8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:28:40.977 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:28:40.976+0000 7ff4e9600700 1 -- 172.21.15.90:0/2589098727 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7ff4e009ce20 con 0x7ff4ec1030b0 2024-04-02T14:28:41.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:28:41.216+0000 7ff4f3c00700 1 -- 172.21.15.90:0/2589098727 --> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi090:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7ff4ec062a30 con 0x7ff4d40a3cc0 2024-04-02T14:28:42.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:28:41 smithi092 ceph-mon[36497]: pgmap v154: 1 pgs: 1 remapped+peering; 577 KiB data, 182 MiB used, 536 GiB / 536 GiB avail 2024-04-02T14:28:42.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:28:41 smithi092 ceph-mon[36497]: osdmap e42: 6 total, 6 up, 6 in 2024-04-02T14:28:42.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:28:41 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-02T14:28:42.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:28:41 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-02T14:28:42.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:28:41 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:28:42.163 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:28:41 smithi090 ceph-mon[31674]: pgmap v154: 1 pgs: 1 remapped+peering; 577 KiB data, 182 MiB used, 536 GiB / 536 GiB avail 2024-04-02T14:28:42.164 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:28:41 smithi090 ceph-mon[31674]: osdmap e42: 6 total, 6 up, 6 in 2024-04-02T14:28:42.164 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:28:41 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-02T14:28:42.164 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:28:41 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-02T14:28:42.164 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:28:41 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:28:42.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:28:41 smithi177 ceph-mon[36431]: pgmap v154: 1 pgs: 1 remapped+peering; 577 KiB data, 182 MiB used, 536 GiB / 536 GiB avail 2024-04-02T14:28:42.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:28:41 smithi177 ceph-mon[36431]: osdmap e42: 6 total, 6 up, 6 in 2024-04-02T14:28:42.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:28:41 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-02T14:28:42.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:28:41 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-02T14:28:42.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:28:41 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:28:42.939 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:28:42 smithi090 ceph-mon[31674]: from='client.24274 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi090:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T14:28:43.110 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:28:42 smithi092 ceph-mon[36497]: from='client.24274 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi090:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T14:28:43.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:28:42 smithi177 ceph-mon[36431]: from='client.24274 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi090:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T14:28:44.010 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:28:43 smithi090 ceph-mon[31674]: pgmap v156: 1 pgs: 1 remapped+peering; 577 KiB data, 182 MiB used, 536 GiB / 536 GiB avail 2024-04-02T14:28:44.110 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:28:43 smithi092 ceph-mon[36497]: pgmap v156: 1 pgs: 1 remapped+peering; 577 KiB data, 182 MiB used, 536 GiB / 536 GiB avail 2024-04-02T14:28:44.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:28:43 smithi177 ceph-mon[36431]: pgmap v156: 1 pgs: 1 remapped+peering; 577 KiB data, 182 MiB used, 536 GiB / 536 GiB avail 2024-04-02T14:28:46.110 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:28:45 smithi092 ceph-mon[36497]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-02T14:28:46.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:28:45 smithi092 ceph-mon[36497]: from='client.? 172.21.15.90:0/2132780761' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "aa72392f-0498-44db-a8d2-eccc89a64871"}]: dispatch 2024-04-02T14:28:46.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:28:45 smithi092 ceph-mon[36497]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "aa72392f-0498-44db-a8d2-eccc89a64871"}]: dispatch 2024-04-02T14:28:46.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:28:45 smithi092 ceph-mon[36497]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "aa72392f-0498-44db-a8d2-eccc89a64871"}]': finished 2024-04-02T14:28:46.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:28:45 smithi092 ceph-mon[36497]: osdmap e43: 7 total, 6 up, 7 in 2024-04-02T14:28:46.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:28:45 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-02T14:28:46.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:28:45 smithi177 ceph-mon[36431]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-02T14:28:46.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:28:45 smithi177 ceph-mon[36431]: from='client.? 172.21.15.90:0/2132780761' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "aa72392f-0498-44db-a8d2-eccc89a64871"}]: dispatch 2024-04-02T14:28:46.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:28:45 smithi177 ceph-mon[36431]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "aa72392f-0498-44db-a8d2-eccc89a64871"}]: dispatch 2024-04-02T14:28:46.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:28:45 smithi177 ceph-mon[36431]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "aa72392f-0498-44db-a8d2-eccc89a64871"}]': finished 2024-04-02T14:28:46.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:28:45 smithi177 ceph-mon[36431]: osdmap e43: 7 total, 6 up, 7 in 2024-04-02T14:28:46.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:28:45 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-02T14:28:46.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:28:45 smithi090 ceph-mon[31674]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-02T14:28:46.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:28:45 smithi090 ceph-mon[31674]: from='client.? 172.21.15.90:0/2132780761' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "aa72392f-0498-44db-a8d2-eccc89a64871"}]: dispatch 2024-04-02T14:28:46.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:28:45 smithi090 ceph-mon[31674]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "aa72392f-0498-44db-a8d2-eccc89a64871"}]: dispatch 2024-04-02T14:28:46.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:28:45 smithi090 ceph-mon[31674]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "aa72392f-0498-44db-a8d2-eccc89a64871"}]': finished 2024-04-02T14:28:46.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:28:45 smithi090 ceph-mon[31674]: osdmap e43: 7 total, 6 up, 7 in 2024-04-02T14:28:46.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:28:45 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-02T14:28:47.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:28:46 smithi092 ceph-mon[36497]: from='client.? 172.21.15.90:0/2927839522' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-02T14:28:47.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:28:46 smithi177 ceph-mon[36431]: from='client.? 172.21.15.90:0/2927839522' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-02T14:28:47.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:28:46 smithi090 ceph-mon[31674]: from='client.? 172.21.15.90:0/2927839522' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-02T14:28:48.088 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:28:47 smithi177 ceph-mon[36431]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 83 KiB/s, 0 objects/s recovering 2024-04-02T14:28:48.110 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:28:47 smithi092 ceph-mon[36497]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 83 KiB/s, 0 objects/s recovering 2024-04-02T14:28:48.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:28:47 smithi090 ceph-mon[31674]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 83 KiB/s, 0 objects/s recovering 2024-04-02T14:28:50.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:28:49 smithi092 ceph-mon[36497]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-02T14:28:50.171 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:28:49 smithi090 ceph-mon[31674]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-02T14:28:50.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:28:49 smithi177 ceph-mon[36431]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-02T14:28:51.924 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:28:51 smithi090 ceph-mon[31674]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-02T14:28:52.110 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:28:51 smithi092 ceph-mon[36497]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-02T14:28:52.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:28:51 smithi177 ceph-mon[36431]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-02T14:28:54.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:28:53 smithi092 ceph-mon[36497]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-02T14:28:54.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:28:53 smithi177 ceph-mon[36431]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-02T14:28:54.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:28:53 smithi090 ceph-mon[31674]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-02T14:28:54.938 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:28:54 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-02T14:28:54.938 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:28:54 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:28:55.110 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:28:54 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-02T14:28:55.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:28:54 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:28:55.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:28:54 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-02T14:28:55.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:28:54 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:28:55.967 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:28:55 smithi090 ceph-mon[31674]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-04-02T14:28:55.967 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:28:55 smithi090 ceph-mon[31674]: Deploying daemon osd.6 on smithi090 2024-04-02T14:28:56.110 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:28:55 smithi092 ceph-mon[36497]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-04-02T14:28:56.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:28:55 smithi092 ceph-mon[36497]: Deploying daemon osd.6 on smithi090 2024-04-02T14:28:56.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:28:55 smithi177 ceph-mon[36431]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-04-02T14:28:56.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:28:55 smithi177 ceph-mon[36431]: Deploying daemon osd.6 on smithi090 2024-04-02T14:28:57.888 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:28:57 smithi090 ceph-mon[31674]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-04-02T14:28:58.110 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:28:57 smithi092 ceph-mon[36497]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-04-02T14:28:58.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:28:57 smithi177 ceph-mon[36431]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-04-02T14:29:00.035 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:28:59 smithi090 ceph-mon[31674]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-04-02T14:29:00.110 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:28:59 smithi092 ceph-mon[36497]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-04-02T14:29:00.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:28:59 smithi177 ceph-mon[36431]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-04-02T14:29:02.110 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:29:01 smithi092 ceph-mon[36497]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-04-02T14:29:02.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:29:01 smithi177 ceph-mon[36431]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-04-02T14:29:02.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:29:01 smithi090 ceph-mon[31674]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-04-02T14:29:04.110 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:29:03 smithi092 ceph-mon[36497]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-04-02T14:29:04.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:29:03 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:29:04.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:29:03 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:29:04.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:29:03 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T14:29:04.137 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:29:03 smithi090 ceph-mon[31674]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-04-02T14:29:04.137 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:29:03 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:29:04.137 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:29:03 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:29:04.137 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:29:03 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T14:29:04.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:29:03 smithi177 ceph-mon[36431]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-04-02T14:29:04.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:29:03 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:29:04.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:29:03 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:29:04.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:29:03 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T14:29:06.110 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:29:05 smithi092 ceph-mon[36497]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-04-02T14:29:06.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:29:05 smithi177 ceph-mon[36431]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-04-02T14:29:06.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:29:05 smithi090 ceph-mon[31674]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-04-02T14:29:07.818 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:29:07.814+0000 7ff4e9600700 1 -- 172.21.15.90:0/2589098727 <== mgr.14152 v2:172.21.15.90:6800/4266054817 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7ff4ec062a30 con 0x7ff4d40a3cc0 2024-04-02T14:29:07.818 INFO:teuthology.orchestra.run.smithi090.stdout:Created osd(s) 6 on host 'smithi090' 2024-04-02T14:29:07.821 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:29:07.818+0000 7ff4f3c00700 1 -- 172.21.15.90:0/2589098727 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7ff4d40a3cc0 msgr2=0x7ff4d40a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:29:07.821 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:29:07.818+0000 7ff4f3c00700 1 --2- 172.21.15.90:0/2589098727 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7ff4d40a3cc0 0x7ff4d40a6180 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7ff4d8011550 tx=0x7ff4d8009260 comp rx=0 tx=0).stop 2024-04-02T14:29:07.821 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:29:07.818+0000 7ff4f3c00700 1 -- 172.21.15.90:0/2589098727 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff4ec1030b0 msgr2=0x7ff4ec102a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:29:07.821 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:29:07.819+0000 7ff4f3c00700 1 --2- 172.21.15.90:0/2589098727 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff4ec1030b0 0x7ff4ec102a10 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7ff4e000e9f0 tx=0x7ff4e000ed00 comp rx=0 tx=0).stop 2024-04-02T14:29:07.821 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:29:07.819+0000 7ff4f3c00700 1 -- 172.21.15.90:0/2589098727 shutdown_connections 2024-04-02T14:29:07.821 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:29:07.819+0000 7ff4f3c00700 1 --2- 172.21.15.90:0/2589098727 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7ff4d40a3cc0 0x7ff4d40a6180 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:29:07.821 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:29:07.819+0000 7ff4f3c00700 1 --2- 172.21.15.90:0/2589098727 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff4ec108310 0x7ff4ec1015a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:29:07.821 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:29:07.819+0000 7ff4f3c00700 1 --2- 172.21.15.90:0/2589098727 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff4ec1030b0 0x7ff4ec102a10 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:29:07.821 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:29:07.819+0000 7ff4f3c00700 1 --2- 172.21.15.90:0/2589098727 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff4ec1059e0 0x7ff4ec101060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:29:07.822 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:29:07.819+0000 7ff4f3c00700 1 -- 172.21.15.90:0/2589098727 >> 172.21.15.90:0/2589098727 conn(0x7ff4ec0fcac0 msgr2=0x7ff4ec103950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:29:07.822 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:29:07.819+0000 7ff4f3c00700 1 -- 172.21.15.90:0/2589098727 shutdown_connections 2024-04-02T14:29:07.822 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:29:07.819+0000 7ff4f3c00700 1 -- 172.21.15.90:0/2589098727 wait complete. 2024-04-02T14:29:08.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:29:07 smithi092 ceph-mon[36497]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-04-02T14:29:08.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:29:07 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:29:08.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:29:07 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:29:08.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:29:07 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:29:08.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:29:07 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T14:29:08.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:29:07 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:29:08.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:29:07 smithi092 ceph-mon[36497]: from='osd.6 [v2:172.21.15.90:6818/2579716443,v1:172.21.15.90:6819/2579716443]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-02T14:29:08.186 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:29:07 smithi090 ceph-mon[31674]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-04-02T14:29:08.186 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:29:07 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:29:08.186 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:29:07 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:29:08.186 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:29:07 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:29:08.186 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:29:07 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T14:29:08.186 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:29:07 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:29:08.186 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:29:07 smithi090 ceph-mon[31674]: from='osd.6 [v2:172.21.15.90:6818/2579716443,v1:172.21.15.90:6819/2579716443]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-02T14:29:08.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:29:07 smithi177 ceph-mon[36431]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-04-02T14:29:08.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:29:07 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:29:08.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:29:07 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:29:08.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:29:07 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:29:08.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:29:07 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T14:29:08.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:29:07 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:29:08.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:29:07 smithi177 ceph-mon[36431]: from='osd.6 [v2:172.21.15.90:6818/2579716443,v1:172.21.15.90:6819/2579716443]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-02T14:29:08.631 DEBUG:teuthology.orchestra.run.smithi090:osd.6> sudo journalctl -f -n 0 -u ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4@osd.6.service 2024-04-02T14:29:08.637 INFO:tasks.cephadm:Deploying osd.7 on smithi092 with /dev/vg_nvme/lv_2... 2024-04-02T14:29:08.638 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:191d963ac9de7322da8aff20b567b3fa458e2ef6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6449448a-f0fc-11ee-b647-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-02T14:29:08.886 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/mon.b/config 2024-04-02T14:29:08.938 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:29:08 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:29:08.939 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:29:08 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T14:29:08.939 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:29:08 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:29:08.939 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:29:08 smithi090 ceph-mon[31674]: from='osd.6 [v2:172.21.15.90:6818/2579716443,v1:172.21.15.90:6819/2579716443]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-02T14:29:08.939 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:29:08 smithi090 ceph-mon[31674]: osdmap e44: 7 total, 6 up, 7 in 2024-04-02T14:29:08.939 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:29:08 smithi090 ceph-mon[31674]: from='osd.6 [v2:172.21.15.90:6818/2579716443,v1:172.21.15.90:6819/2579716443]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]: dispatch 2024-04-02T14:29:08.939 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:29:08 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-02T14:29:09.005 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:29:08 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:29:09.005 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:29:08 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T14:29:09.005 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:29:08 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:29:09.005 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:29:08 smithi092 ceph-mon[36497]: from='osd.6 [v2:172.21.15.90:6818/2579716443,v1:172.21.15.90:6819/2579716443]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-02T14:29:09.006 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:29:08 smithi092 ceph-mon[36497]: osdmap e44: 7 total, 6 up, 7 in 2024-04-02T14:29:09.006 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:29:08 smithi092 ceph-mon[36497]: from='osd.6 [v2:172.21.15.90:6818/2579716443,v1:172.21.15.90:6819/2579716443]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]: dispatch 2024-04-02T14:29:09.006 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:29:08 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-02T14:29:09.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:29:08 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:29:09.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:29:08 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T14:29:09.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:29:08 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:29:09.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:29:08 smithi177 ceph-mon[36431]: from='osd.6 [v2:172.21.15.90:6818/2579716443,v1:172.21.15.90:6819/2579716443]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-02T14:29:09.188 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:29:08 smithi177 ceph-mon[36431]: osdmap e44: 7 total, 6 up, 7 in 2024-04-02T14:29:09.188 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:29:08 smithi177 ceph-mon[36431]: from='osd.6 [v2:172.21.15.90:6818/2579716443,v1:172.21.15.90:6819/2579716443]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]: dispatch 2024-04-02T14:29:09.188 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:29:08 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-02T14:29:10.073 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:29:09 smithi090 ceph-mon[31674]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-04-02T14:29:10.073 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:29:09 smithi090 ceph-mon[31674]: from='osd.6 [v2:172.21.15.90:6818/2579716443,v1:172.21.15.90:6819/2579716443]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]': finished 2024-04-02T14:29:10.073 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:29:09 smithi090 ceph-mon[31674]: osdmap e45: 7 total, 6 up, 7 in 2024-04-02T14:29:10.073 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:29:09 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-02T14:29:10.073 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:29:09 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-02T14:29:10.073 INFO:journalctl@ceph.osd.6.smithi090.stdout:Apr 02 14:29:10 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-6[59804]: 2024-04-02T14:29:10.000+0000 7fab5d800700 -1 osd.6 0 waiting for initial osdmap 2024-04-02T14:29:10.085 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:29:09 smithi092 ceph-mon[36497]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-04-02T14:29:10.086 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:29:09 smithi092 ceph-mon[36497]: from='osd.6 [v2:172.21.15.90:6818/2579716443,v1:172.21.15.90:6819/2579716443]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]': finished 2024-04-02T14:29:10.086 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:29:09 smithi092 ceph-mon[36497]: osdmap e45: 7 total, 6 up, 7 in 2024-04-02T14:29:10.086 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:29:09 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-02T14:29:10.086 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:29:09 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-02T14:29:10.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:29:09 smithi177 ceph-mon[36431]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-04-02T14:29:10.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:29:09 smithi177 ceph-mon[36431]: from='osd.6 [v2:172.21.15.90:6818/2579716443,v1:172.21.15.90:6819/2579716443]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]': finished 2024-04-02T14:29:10.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:29:09 smithi177 ceph-mon[36431]: osdmap e45: 7 total, 6 up, 7 in 2024-04-02T14:29:10.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:29:09 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-02T14:29:10.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:29:09 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-02T14:29:10.365 INFO:journalctl@ceph.osd.6.smithi090.stdout:Apr 02 14:29:10 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-6[59804]: 2024-04-02T14:29:10.075+0000 7fab53600700 -1 osd.6 45 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-02T14:29:10.711 INFO:teuthology.orchestra.run.smithi092.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-02T14:29:10.711 INFO:teuthology.orchestra.run.smithi092.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-02T14:29:10.711 INFO:teuthology.orchestra.run.smithi092.stderr: stderr: 10+0 records in 2024-04-02T14:29:10.711 INFO:teuthology.orchestra.run.smithi092.stderr:10+0 records out 2024-04-02T14:29:10.711 INFO:teuthology.orchestra.run.smithi092.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.011419 s, 918 MB/s 2024-04-02T14:29:10.711 INFO:teuthology.orchestra.run.smithi092.stderr:--> Zapping successful for: 2024-04-02T14:29:11.015 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:29:10 smithi092 ceph-mon[36497]: purged_snaps scrub starts 2024-04-02T14:29:11.016 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:29:10 smithi092 ceph-mon[36497]: purged_snaps scrub ok 2024-04-02T14:29:11.016 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:29:10 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-02T14:29:11.016 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:29:10 smithi092 ceph-mon[36497]: from='osd.6 [v2:172.21.15.90:6818/2579716443,v1:172.21.15.90:6819/2579716443]' entity='osd.6' 2024-04-02T14:29:11.063 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:29:10 smithi090 ceph-mon[31674]: purged_snaps scrub starts 2024-04-02T14:29:11.063 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:29:10 smithi090 ceph-mon[31674]: purged_snaps scrub ok 2024-04-02T14:29:11.063 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:29:10 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-02T14:29:11.063 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:29:10 smithi090 ceph-mon[31674]: from='osd.6 [v2:172.21.15.90:6818/2579716443,v1:172.21.15.90:6819/2579716443]' entity='osd.6' 2024-04-02T14:29:11.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:29:10 smithi177 ceph-mon[36431]: purged_snaps scrub starts 2024-04-02T14:29:11.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:29:10 smithi177 ceph-mon[36431]: purged_snaps scrub ok 2024-04-02T14:29:11.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:29:10 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-02T14:29:11.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:29:10 smithi177 ceph-mon[36431]: from='osd.6 [v2:172.21.15.90:6818/2579716443,v1:172.21.15.90:6819/2579716443]' entity='osd.6' 2024-04-02T14:29:11.302 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:191d963ac9de7322da8aff20b567b3fa458e2ef6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6449448a-f0fc-11ee-b647-cb9ed24678a4 -- ceph orch daemon add osd smithi092:vg_nvme/lv_2 2024-04-02T14:29:11.550 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/mon.b/config 2024-04-02T14:29:11.861 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:29:11 smithi092 ceph-mon[36497]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-04-02T14:29:11.861 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:29:11 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-02T14:29:11.861 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:29:11 smithi092 ceph-mon[36497]: osd.6 [v2:172.21.15.90:6818/2579716443,v1:172.21.15.90:6819/2579716443] boot 2024-04-02T14:29:11.861 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:29:11 smithi092 ceph-mon[36497]: osdmap e46: 7 total, 7 up, 7 in 2024-04-02T14:29:11.861 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:29:11 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-02T14:29:12.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:29:11 smithi177 ceph-mon[36431]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-04-02T14:29:12.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:29:11 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-02T14:29:12.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:29:11 smithi177 ceph-mon[36431]: osd.6 [v2:172.21.15.90:6818/2579716443,v1:172.21.15.90:6819/2579716443] boot 2024-04-02T14:29:12.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:29:11 smithi177 ceph-mon[36431]: osdmap e46: 7 total, 7 up, 7 in 2024-04-02T14:29:12.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:29:11 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-02T14:29:12.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:29:11 smithi090 ceph-mon[31674]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-04-02T14:29:12.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:29:11 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-02T14:29:12.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:29:11 smithi090 ceph-mon[31674]: osd.6 [v2:172.21.15.90:6818/2579716443,v1:172.21.15.90:6819/2579716443] boot 2024-04-02T14:29:12.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:29:11 smithi090 ceph-mon[31674]: osdmap e46: 7 total, 7 up, 7 in 2024-04-02T14:29:12.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:29:11 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-02T14:29:13.047 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:29:13.043+0000 7ff8f5595700 1 -- 172.21.15.92:0/145756812 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff8f0103820 msgr2=0x7ff8f0103c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:29:13.047 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:29:13.043+0000 7ff8f5595700 1 --2- 172.21.15.92:0/145756812 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff8f0103820 0x7ff8f0103c80 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7ff8e40099c0 tx=0x7ff8e4009cd0 comp rx=0 tx=0).stop 2024-04-02T14:29:13.047 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:29:13.044+0000 7ff8f5595700 1 -- 172.21.15.92:0/145756812 shutdown_connections 2024-04-02T14:29:13.047 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:29:13.044+0000 7ff8f5595700 1 --2- 172.21.15.92:0/145756812 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff8f01041c0 0x7ff8f010aa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:29:13.047 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:29:13.044+0000 7ff8f5595700 1 --2- 172.21.15.92:0/145756812 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff8f0102620 0x7ff8f0102a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:29:13.048 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:29:13.044+0000 7ff8f5595700 1 --2- 172.21.15.92:0/145756812 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff8f0103820 0x7ff8f0103c80 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:29:13.048 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:29:13.044+0000 7ff8f5595700 1 -- 172.21.15.92:0/145756812 >> 172.21.15.92:0/145756812 conn(0x7ff8f00fdc00 msgr2=0x7ff8f0100020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:29:13.048 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:29:13.044+0000 7ff8f5595700 1 -- 172.21.15.92:0/145756812 shutdown_connections 2024-04-02T14:29:13.048 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:29:13.044+0000 7ff8f5595700 1 -- 172.21.15.92:0/145756812 wait complete. 2024-04-02T14:29:13.048 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:29:13.044+0000 7ff8f5595700 1 Processor -- start 2024-04-02T14:29:13.048 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:29:13.044+0000 7ff8f5595700 1 -- start start 2024-04-02T14:29:13.048 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:29:13.044+0000 7ff8f5595700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff8f0102620 0x7ff8f01987b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:29:13.048 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:29:13.044+0000 7ff8f5595700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff8f0103820 0x7ff8f0198cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:29:13.048 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:29:13.044+0000 7ff8f5595700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff8f01041c0 0x7ff8f019fd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:29:13.048 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:29:13.044+0000 7ff8f5595700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7ff8f010e1f0 con 0x7ff8f0103820 2024-04-02T14:29:13.048 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:29:13.044+0000 7ff8f5595700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7ff8f010e070 con 0x7ff8f01041c0 2024-04-02T14:29:13.048 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:29:13.044+0000 7ff8f5595700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7ff8f010e370 con 0x7ff8f0102620 2024-04-02T14:29:13.049 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:29:13.045+0000 7ff8ef7fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff8f01041c0 0x7ff8f019fd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:29:13.049 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:29:13.045+0000 7ff8ef7fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff8f01041c0 0x7ff8f019fd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:58346/0 (socket says 172.21.15.92:58346) 2024-04-02T14:29:13.049 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:29:13.045+0000 7ff8eeffd700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff8f0102620 0x7ff8f01987b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:29:13.049 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:29:13.045+0000 7ff8ef7fe700 1 -- 172.21.15.92:0/4008582088 learned_addr learned my addr 172.21.15.92:0/4008582088 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-04-02T14:29:13.049 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:29:13.045+0000 7ff8ee7fc700 1 --2- 172.21.15.92:0/4008582088 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff8f0103820 0x7ff8f0198cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:29:13.049 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:29:13.045+0000 7ff8ef7fe700 1 -- 172.21.15.92:0/4008582088 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff8f0102620 msgr2=0x7ff8f01987b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:29:13.049 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:29:13.045+0000 7ff8ef7fe700 1 --2- 172.21.15.92:0/4008582088 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff8f0102620 0x7ff8f01987b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:29:13.049 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:29:13.045+0000 7ff8ef7fe700 1 -- 172.21.15.92:0/4008582088 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff8f0103820 msgr2=0x7ff8f0198cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:29:13.049 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:29:13.045+0000 7ff8ef7fe700 1 --2- 172.21.15.92:0/4008582088 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff8f0103820 0x7ff8f0198cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:29:13.049 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:29:13.045+0000 7ff8ef7fe700 1 -- 172.21.15.92:0/4008582088 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8f0074820 con 0x7ff8f01041c0 2024-04-02T14:29:13.049 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:29:13.045+0000 7ff8ef7fe700 1 --2- 172.21.15.92:0/4008582088 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff8f01041c0 0x7ff8f019fd70 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7ff8e800ada0 tx=0x7ff8e800c330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:29:13.049 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:29:13.045+0000 7ff8d7fff700 1 -- 172.21.15.92:0/4008582088 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8e8017020 con 0x7ff8f01041c0 2024-04-02T14:29:13.049 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:29:13.045+0000 7ff8d7fff700 1 -- 172.21.15.92:0/4008582088 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff8e800a0d0 con 0x7ff8f01041c0 2024-04-02T14:29:13.050 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:29:13.045+0000 7ff8f5595700 1 -- 172.21.15.92:0/4008582088 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff8f0074b10 con 0x7ff8f01041c0 2024-04-02T14:29:13.050 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:29:13.045+0000 7ff8d7fff700 1 -- 172.21.15.92:0/4008582088 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8e8005480 con 0x7ff8f01041c0 2024-04-02T14:29:13.050 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:29:13.045+0000 7ff8f5595700 1 -- 172.21.15.92:0/4008582088 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8f0075000 con 0x7ff8f01041c0 2024-04-02T14:29:13.051 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:29:13.046+0000 7ff8f5595700 1 -- 172.21.15.92:0/4008582088 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff8f00692b0 con 0x7ff8f01041c0 2024-04-02T14:29:13.051 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:29:13.047+0000 7ff8d7fff700 1 -- 172.21.15.92:0/4008582088 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7ff8e8005620 con 0x7ff8f01041c0 2024-04-02T14:29:13.051 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:29:13.048+0000 7ff8d7fff700 1 --2- 172.21.15.92:0/4008582088 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7ff8d80a3c40 0x7ff8d80a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:29:13.051 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:29:13.048+0000 7ff8d7fff700 1 -- 172.21.15.92:0/4008582088 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 4965+0+0 (secure 0 0 0) 0x7ff8e80cf540 con 0x7ff8f01041c0 2024-04-02T14:29:13.051 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:29:13.048+0000 7ff8eeffd700 1 --2- 172.21.15.92:0/4008582088 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7ff8d80a3c40 0x7ff8d80a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:29:13.051 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:29:13.049+0000 7ff8eeffd700 1 --2- 172.21.15.92:0/4008582088 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7ff8d80a3c40 0x7ff8d80a6100 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7ff8f0103680 tx=0x7ff8e000a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:29:13.053 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:29:13.052+0000 7ff8d7fff700 1 -- 172.21.15.92:0/4008582088 <== mon.2 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7ff8e8021060 con 0x7ff8f01041c0 2024-04-02T14:29:13.262 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:29:13.260+0000 7ff8f5595700 1 -- 172.21.15.92:0/4008582088 --> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi092:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7ff8f0109500 con 0x7ff8d80a3c40 2024-04-02T14:29:14.092 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:29:13 smithi092 ceph-mon[36497]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 218 MiB used, 626 GiB / 626 GiB avail 2024-04-02T14:29:14.092 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:29:13 smithi092 ceph-mon[36497]: osdmap e47: 7 total, 7 up, 7 in 2024-04-02T14:29:14.092 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:29:13 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-02T14:29:14.092 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:29:13 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-02T14:29:14.093 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:29:13 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:29:14.162 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:29:13 smithi090 ceph-mon[31674]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 218 MiB used, 626 GiB / 626 GiB avail 2024-04-02T14:29:14.162 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:29:13 smithi090 ceph-mon[31674]: osdmap e47: 7 total, 7 up, 7 in 2024-04-02T14:29:14.162 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:29:13 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-02T14:29:14.162 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:29:13 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-02T14:29:14.162 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:29:13 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:29:14.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:29:13 smithi177 ceph-mon[36431]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 218 MiB used, 626 GiB / 626 GiB avail 2024-04-02T14:29:14.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:29:13 smithi177 ceph-mon[36431]: osdmap e47: 7 total, 7 up, 7 in 2024-04-02T14:29:14.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:29:13 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-02T14:29:14.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:29:13 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-02T14:29:14.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:29:13 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:29:14.861 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:29:14 smithi092 ceph-mon[36497]: from='client.24290 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi092:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T14:29:14.861 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:29:14 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:29:14.861 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:29:14 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:29:14.861 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:29:14 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-02T14:29:14.861 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:29:14 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-02T14:29:14.861 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:29:14 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-02T14:29:14.861 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:29:14 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:29:14.861 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:29:14 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:29:14.861 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:29:14 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T14:29:14.861 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:29:14 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:29:15.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:29:14 smithi177 ceph-mon[36431]: from='client.24290 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi092:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T14:29:15.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:29:14 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:29:15.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:29:14 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:29:15.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:29:14 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-02T14:29:15.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:29:14 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-02T14:29:15.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:29:14 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-02T14:29:15.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:29:14 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:29:15.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:29:14 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:29:15.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:29:14 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T14:29:15.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:29:14 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:29:15.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:29:14 smithi090 ceph-mon[31674]: from='client.24290 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi092:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T14:29:15.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:29:14 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:29:15.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:29:14 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:29:15.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:29:14 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-02T14:29:15.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:29:14 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-02T14:29:15.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:29:14 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-02T14:29:15.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:29:14 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:29:15.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:29:14 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:29:15.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:29:14 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T14:29:15.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:29:14 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:29:16.112 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:29:15 smithi092 ceph-mon[36497]: Detected new or changed devices on smithi090 2024-04-02T14:29:16.112 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:29:15 smithi092 ceph-mon[36497]: Adjusting osd_memory_target on smithi090 to 5697M 2024-04-02T14:29:16.112 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:29:15 smithi092 ceph-mon[36497]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 218 MiB used, 626 GiB / 626 GiB avail 2024-04-02T14:29:16.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:29:15 smithi177 ceph-mon[36431]: Detected new or changed devices on smithi090 2024-04-02T14:29:16.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:29:15 smithi177 ceph-mon[36431]: Adjusting osd_memory_target on smithi090 to 5697M 2024-04-02T14:29:16.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:29:15 smithi177 ceph-mon[36431]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 218 MiB used, 626 GiB / 626 GiB avail 2024-04-02T14:29:16.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:29:15 smithi090 ceph-mon[31674]: Detected new or changed devices on smithi090 2024-04-02T14:29:16.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:29:15 smithi090 ceph-mon[31674]: Adjusting osd_memory_target on smithi090 to 5697M 2024-04-02T14:29:16.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:29:15 smithi090 ceph-mon[31674]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 218 MiB used, 626 GiB / 626 GiB avail 2024-04-02T14:29:17.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:29:16 smithi177 ceph-mon[36431]: from='client.? 172.21.15.92:0/1244228834' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "931bd105-bc14-4a1e-bfe4-c7ad58fa4b73"}]: dispatch 2024-04-02T14:29:17.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:29:16 smithi177 ceph-mon[36431]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "931bd105-bc14-4a1e-bfe4-c7ad58fa4b73"}]: dispatch 2024-04-02T14:29:17.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:29:16 smithi177 ceph-mon[36431]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "931bd105-bc14-4a1e-bfe4-c7ad58fa4b73"}]': finished 2024-04-02T14:29:17.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:29:16 smithi177 ceph-mon[36431]: osdmap e48: 8 total, 7 up, 8 in 2024-04-02T14:29:17.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:29:16 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-02T14:29:17.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:29:16 smithi177 ceph-mon[36431]: from='client.? 172.21.15.92:0/840544298' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-02T14:29:17.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:29:16 smithi090 ceph-mon[31674]: from='client.? 172.21.15.92:0/1244228834' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "931bd105-bc14-4a1e-bfe4-c7ad58fa4b73"}]: dispatch 2024-04-02T14:29:17.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:29:16 smithi090 ceph-mon[31674]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "931bd105-bc14-4a1e-bfe4-c7ad58fa4b73"}]: dispatch 2024-04-02T14:29:17.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:29:16 smithi090 ceph-mon[31674]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "931bd105-bc14-4a1e-bfe4-c7ad58fa4b73"}]': finished 2024-04-02T14:29:17.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:29:16 smithi090 ceph-mon[31674]: osdmap e48: 8 total, 7 up, 8 in 2024-04-02T14:29:17.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:29:16 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-02T14:29:17.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:29:16 smithi090 ceph-mon[31674]: from='client.? 172.21.15.92:0/840544298' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-02T14:29:17.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:29:16 smithi092 ceph-mon[36497]: from='client.? 172.21.15.92:0/1244228834' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "931bd105-bc14-4a1e-bfe4-c7ad58fa4b73"}]: dispatch 2024-04-02T14:29:17.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:29:16 smithi092 ceph-mon[36497]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "931bd105-bc14-4a1e-bfe4-c7ad58fa4b73"}]: dispatch 2024-04-02T14:29:17.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:29:16 smithi092 ceph-mon[36497]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "931bd105-bc14-4a1e-bfe4-c7ad58fa4b73"}]': finished 2024-04-02T14:29:17.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:29:16 smithi092 ceph-mon[36497]: osdmap e48: 8 total, 7 up, 8 in 2024-04-02T14:29:17.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:29:16 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-02T14:29:17.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:29:16 smithi092 ceph-mon[36497]: from='client.? 172.21.15.92:0/840544298' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-02T14:29:18.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:29:17 smithi177 ceph-mon[36431]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 218 MiB used, 626 GiB / 626 GiB avail 2024-04-02T14:29:18.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:29:17 smithi090 ceph-mon[31674]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 218 MiB used, 626 GiB / 626 GiB avail 2024-04-02T14:29:18.360 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:29:17 smithi092 ceph-mon[36497]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 218 MiB used, 626 GiB / 626 GiB avail 2024-04-02T14:29:20.112 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:29:19 smithi092 ceph-mon[36497]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 218 MiB used, 626 GiB / 626 GiB avail 2024-04-02T14:29:20.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:29:19 smithi177 ceph-mon[36431]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 218 MiB used, 626 GiB / 626 GiB avail 2024-04-02T14:29:20.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:29:19 smithi090 ceph-mon[31674]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 218 MiB used, 626 GiB / 626 GiB avail 2024-04-02T14:29:22.076 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:29:21 smithi092 ceph-mon[36497]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 218 MiB used, 626 GiB / 626 GiB avail 2024-04-02T14:29:22.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:29:21 smithi177 ceph-mon[36431]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 218 MiB used, 626 GiB / 626 GiB avail 2024-04-02T14:29:22.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:29:21 smithi090 ceph-mon[31674]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 218 MiB used, 626 GiB / 626 GiB avail 2024-04-02T14:29:24.046 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:29:23 smithi092 ceph-mon[36497]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 218 MiB used, 626 GiB / 626 GiB avail 2024-04-02T14:29:24.046 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:29:23 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-02T14:29:24.046 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:29:23 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:29:24.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:29:23 smithi177 ceph-mon[36431]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 218 MiB used, 626 GiB / 626 GiB avail 2024-04-02T14:29:24.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:29:23 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-02T14:29:24.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:29:23 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:29:24.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:29:23 smithi090 ceph-mon[31674]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 218 MiB used, 626 GiB / 626 GiB avail 2024-04-02T14:29:24.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:29:23 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-02T14:29:24.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:29:23 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:29:25.010 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:29:24 smithi092 ceph-mon[36497]: Deploying daemon osd.7 on smithi092 2024-04-02T14:29:25.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:29:24 smithi177 ceph-mon[36431]: Deploying daemon osd.7 on smithi092 2024-04-02T14:29:25.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:29:24 smithi090 ceph-mon[31674]: Deploying daemon osd.7 on smithi092 2024-04-02T14:29:26.096 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:29:25 smithi092 ceph-mon[36497]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 218 MiB used, 626 GiB / 626 GiB avail 2024-04-02T14:29:26.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:29:25 smithi177 ceph-mon[36431]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 218 MiB used, 626 GiB / 626 GiB avail 2024-04-02T14:29:26.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:29:25 smithi090 ceph-mon[31674]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 218 MiB used, 626 GiB / 626 GiB avail 2024-04-02T14:29:27.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:29:26 smithi092 ceph-mon[36497]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 218 MiB used, 626 GiB / 626 GiB avail 2024-04-02T14:29:27.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:29:26 smithi177 ceph-mon[36431]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 218 MiB used, 626 GiB / 626 GiB avail 2024-04-02T14:29:27.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:29:26 smithi090 ceph-mon[31674]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 218 MiB used, 626 GiB / 626 GiB avail 2024-04-02T14:29:29.686 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:29:29 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:29:29.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:29:29 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T14:29:29.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:29:29 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:29:29.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:29:29 smithi177 ceph-mon[36431]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 218 MiB used, 626 GiB / 626 GiB avail 2024-04-02T14:29:29.688 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:29:29 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:29:29.688 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:29:29 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T14:29:29.688 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:29:29 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:29:29.689 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:29:29 smithi090 ceph-mon[31674]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 218 MiB used, 626 GiB / 626 GiB avail 2024-04-02T14:29:29.861 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:29:29 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:29:29.861 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:29:29 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T14:29:29.861 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:29:29 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:29:29.861 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:29:29 smithi092 ceph-mon[36497]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 218 MiB used, 626 GiB / 626 GiB avail 2024-04-02T14:29:31.832 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:29:31 smithi092 ceph-mon[36497]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 218 MiB used, 626 GiB / 626 GiB avail 2024-04-02T14:29:31.832 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:29:31 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:29:31.832 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:29:31 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:29:31.832 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:29:31 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:29:31.833 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:29:31 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T14:29:31.833 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:29:31 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:29:32.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:29:31 smithi177 ceph-mon[36431]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 218 MiB used, 626 GiB / 626 GiB avail 2024-04-02T14:29:32.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:29:31 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:29:32.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:29:31 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:29:32.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:29:31 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:29:32.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:29:31 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T14:29:32.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:29:31 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:29:32.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:29:31 smithi090 ceph-mon[31674]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 218 MiB used, 626 GiB / 626 GiB avail 2024-04-02T14:29:32.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:29:31 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:29:32.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:29:31 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:29:32.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:29:31 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:29:32.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:29:31 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T14:29:32.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:29:31 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:29:32.190 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:29:32.187+0000 7ff8d7fff700 1 -- 172.21.15.92:0/4008582088 <== mgr.14152 v2:172.21.15.90:6800/4266054817 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7ff8f0109500 con 0x7ff8d80a3c40 2024-04-02T14:29:32.190 INFO:teuthology.orchestra.run.smithi092.stdout:Created osd(s) 7 on host 'smithi092' 2024-04-02T14:29:32.191 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:29:32.190+0000 7ff8f5595700 1 -- 172.21.15.92:0/4008582088 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7ff8d80a3c40 msgr2=0x7ff8d80a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:29:32.191 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:29:32.190+0000 7ff8f5595700 1 --2- 172.21.15.92:0/4008582088 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7ff8d80a3c40 0x7ff8d80a6100 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7ff8f0103680 tx=0x7ff8e000a380 comp rx=0 tx=0).stop 2024-04-02T14:29:32.191 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:29:32.190+0000 7ff8f5595700 1 -- 172.21.15.92:0/4008582088 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff8f01041c0 msgr2=0x7ff8f019fd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:29:32.192 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:29:32.190+0000 7ff8f5595700 1 --2- 172.21.15.92:0/4008582088 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff8f01041c0 0x7ff8f019fd70 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7ff8e800ada0 tx=0x7ff8e800c330 comp rx=0 tx=0).stop 2024-04-02T14:29:32.192 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:29:32.190+0000 7ff8f5595700 1 -- 172.21.15.92:0/4008582088 shutdown_connections 2024-04-02T14:29:32.192 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:29:32.190+0000 7ff8f5595700 1 --2- 172.21.15.92:0/4008582088 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7ff8d80a3c40 0x7ff8d80a6100 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:29:32.192 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:29:32.190+0000 7ff8f5595700 1 --2- 172.21.15.92:0/4008582088 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff8f01041c0 0x7ff8f019fd70 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:29:32.192 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:29:32.190+0000 7ff8f5595700 1 --2- 172.21.15.92:0/4008582088 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff8f0102620 0x7ff8f01987b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:29:32.192 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:29:32.190+0000 7ff8f5595700 1 --2- 172.21.15.92:0/4008582088 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff8f0103820 0x7ff8f0198cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:29:32.192 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:29:32.190+0000 7ff8f5595700 1 -- 172.21.15.92:0/4008582088 >> 172.21.15.92:0/4008582088 conn(0x7ff8f00fdc00 msgr2=0x7ff8f0106b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:29:32.192 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:29:32.190+0000 7ff8f5595700 1 -- 172.21.15.92:0/4008582088 shutdown_connections 2024-04-02T14:29:32.192 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:29:32.190+0000 7ff8f5595700 1 -- 172.21.15.92:0/4008582088 wait complete. 2024-04-02T14:29:32.745 DEBUG:teuthology.orchestra.run.smithi092:osd.7> sudo journalctl -f -n 0 -u ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4@osd.7.service 2024-04-02T14:29:32.747 INFO:tasks.cephadm:Deploying osd.8 on smithi177 with /dev/vg_nvme/lv_2... 2024-04-02T14:29:32.747 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:191d963ac9de7322da8aff20b567b3fa458e2ef6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6449448a-f0fc-11ee-b647-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-02T14:29:32.990 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/mon.c/config 2024-04-02T14:29:33.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:29:33 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:29:33.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:29:33 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T14:29:33.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:29:33 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:29:33.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:29:33 smithi092 ceph-mon[36497]: from='osd.7 [v2:172.21.15.92:6816/3371649139,v1:172.21.15.92:6817/3371649139]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-02T14:29:33.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:29:33 smithi092 ceph-mon[36497]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-02T14:29:33.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:29:33 smithi092 ceph-mon[36497]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 218 MiB used, 626 GiB / 626 GiB avail 2024-04-02T14:29:33.402 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:29:33 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:29:33.403 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:29:33 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T14:29:33.403 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:29:33 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:29:33.403 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:29:33 smithi177 ceph-mon[36431]: from='osd.7 [v2:172.21.15.92:6816/3371649139,v1:172.21.15.92:6817/3371649139]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-02T14:29:33.403 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:29:33 smithi177 ceph-mon[36431]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-02T14:29:33.403 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:29:33 smithi177 ceph-mon[36431]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 218 MiB used, 626 GiB / 626 GiB avail 2024-04-02T14:29:33.439 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:29:33 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:29:33.439 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:29:33 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T14:29:33.439 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:29:33 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:29:33.439 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:29:33 smithi090 ceph-mon[31674]: from='osd.7 [v2:172.21.15.92:6816/3371649139,v1:172.21.15.92:6817/3371649139]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-02T14:29:33.439 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:29:33 smithi090 ceph-mon[31674]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-02T14:29:33.439 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:29:33 smithi090 ceph-mon[31674]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 218 MiB used, 626 GiB / 626 GiB avail 2024-04-02T14:29:34.253 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:29:34 smithi092 ceph-mon[36497]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-02T14:29:34.254 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:29:34 smithi092 ceph-mon[36497]: from='osd.7 [v2:172.21.15.92:6816/3371649139,v1:172.21.15.92:6817/3371649139]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]: dispatch 2024-04-02T14:29:34.254 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:29:34 smithi092 ceph-mon[36497]: osdmap e49: 8 total, 7 up, 8 in 2024-04-02T14:29:34.254 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:29:34 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-02T14:29:34.254 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:29:34 smithi092 ceph-mon[36497]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]: dispatch 2024-04-02T14:29:34.437 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:29:34 smithi177 ceph-mon[36431]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-02T14:29:34.437 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:29:34 smithi177 ceph-mon[36431]: from='osd.7 [v2:172.21.15.92:6816/3371649139,v1:172.21.15.92:6817/3371649139]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]: dispatch 2024-04-02T14:29:34.437 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:29:34 smithi177 ceph-mon[36431]: osdmap e49: 8 total, 7 up, 8 in 2024-04-02T14:29:34.437 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:29:34 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-02T14:29:34.437 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:29:34 smithi177 ceph-mon[36431]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]: dispatch 2024-04-02T14:29:34.438 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:29:34 smithi090 ceph-mon[31674]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-02T14:29:34.439 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:29:34 smithi090 ceph-mon[31674]: from='osd.7 [v2:172.21.15.92:6816/3371649139,v1:172.21.15.92:6817/3371649139]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]: dispatch 2024-04-02T14:29:34.439 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:29:34 smithi090 ceph-mon[31674]: osdmap e49: 8 total, 7 up, 8 in 2024-04-02T14:29:34.439 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:29:34 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-02T14:29:34.439 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:29:34 smithi090 ceph-mon[31674]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]: dispatch 2024-04-02T14:29:34.840 INFO:teuthology.orchestra.run.smithi177.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-02T14:29:34.840 INFO:teuthology.orchestra.run.smithi177.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-02T14:29:34.840 INFO:teuthology.orchestra.run.smithi177.stderr: stderr: 10+0 records in 2024-04-02T14:29:34.840 INFO:teuthology.orchestra.run.smithi177.stderr:10+0 records out 2024-04-02T14:29:34.840 INFO:teuthology.orchestra.run.smithi177.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0134602 s, 779 MB/s 2024-04-02T14:29:34.841 INFO:teuthology.orchestra.run.smithi177.stderr:--> Zapping successful for: 2024-04-02T14:29:35.288 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 02 14:29:35 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-7[54644]: 2024-04-02T14:29:35.124+0000 7f0bbd655700 -1 osd.7 0 waiting for initial osdmap 2024-04-02T14:29:35.288 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 02 14:29:35 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-7[54644]: 2024-04-02T14:29:35.285+0000 7f0bb5440700 -1 osd.7 51 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-02T14:29:35.288 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:29:35 smithi092 ceph-mon[36497]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]': finished 2024-04-02T14:29:35.288 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:29:35 smithi092 ceph-mon[36497]: osdmap e50: 8 total, 7 up, 8 in 2024-04-02T14:29:35.288 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:29:35 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-02T14:29:35.288 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:29:35 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-02T14:29:35.288 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:29:35 smithi092 ceph-mon[36497]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 218 MiB used, 626 GiB / 626 GiB avail 2024-04-02T14:29:35.289 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:29:35 smithi092 ceph-mon[36497]: osdmap e51: 8 total, 7 up, 8 in 2024-04-02T14:29:35.289 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:29:35 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-02T14:29:35.289 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:29:35 smithi092 ceph-mon[36497]: from='osd.7 ' entity='osd.7' 2024-04-02T14:29:35.436 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:29:35 smithi177 ceph-mon[36431]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]': finished 2024-04-02T14:29:35.436 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:29:35 smithi177 ceph-mon[36431]: osdmap e50: 8 total, 7 up, 8 in 2024-04-02T14:29:35.436 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:29:35 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-02T14:29:35.437 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:29:35 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-02T14:29:35.437 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:29:35 smithi177 ceph-mon[36431]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 218 MiB used, 626 GiB / 626 GiB avail 2024-04-02T14:29:35.437 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:29:35 smithi177 ceph-mon[36431]: osdmap e51: 8 total, 7 up, 8 in 2024-04-02T14:29:35.437 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:29:35 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-02T14:29:35.437 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:29:35 smithi177 ceph-mon[36431]: from='osd.7 ' entity='osd.7' 2024-04-02T14:29:35.438 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:29:35 smithi090 ceph-mon[31674]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]': finished 2024-04-02T14:29:35.439 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:29:35 smithi090 ceph-mon[31674]: osdmap e50: 8 total, 7 up, 8 in 2024-04-02T14:29:35.439 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:29:35 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-02T14:29:35.439 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:29:35 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-02T14:29:35.439 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:29:35 smithi090 ceph-mon[31674]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 218 MiB used, 626 GiB / 626 GiB avail 2024-04-02T14:29:35.439 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:29:35 smithi090 ceph-mon[31674]: osdmap e51: 8 total, 7 up, 8 in 2024-04-02T14:29:35.439 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:29:35 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-02T14:29:35.439 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:29:35 smithi090 ceph-mon[31674]: from='osd.7 ' entity='osd.7' 2024-04-02T14:29:35.501 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:191d963ac9de7322da8aff20b567b3fa458e2ef6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6449448a-f0fc-11ee-b647-cb9ed24678a4 -- ceph orch daemon add osd smithi177:vg_nvme/lv_2 2024-04-02T14:29:35.748 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/mon.c/config 2024-04-02T14:29:36.402 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:29:36 smithi177 ceph-mon[36431]: purged_snaps scrub starts 2024-04-02T14:29:36.402 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:29:36 smithi177 ceph-mon[36431]: purged_snaps scrub ok 2024-04-02T14:29:36.402 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:29:36 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-02T14:29:36.402 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:29:36 smithi177 ceph-mon[36431]: Detected new or changed devices on smithi092 2024-04-02T14:29:36.403 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:29:36 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:29:36.403 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:29:36 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:29:36.403 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:29:36 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-02T14:29:36.403 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:29:36 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-02T14:29:36.403 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:29:36 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-02T14:29:36.403 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:29:36 smithi177 ceph-mon[36431]: Adjusting osd_memory_target on smithi092 to 5693M 2024-04-02T14:29:36.403 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:29:36 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:29:36.403 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:29:36 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:29:36.403 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:29:36 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T14:29:36.403 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:29:36 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:29:36.403 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:29:36 smithi177 ceph-mon[36431]: osd.7 [v2:172.21.15.92:6816/3371649139,v1:172.21.15.92:6817/3371649139] boot 2024-04-02T14:29:36.403 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:29:36 smithi177 ceph-mon[36431]: osdmap e52: 8 total, 8 up, 8 in 2024-04-02T14:29:36.403 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:29:36 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-02T14:29:36.439 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:29:36 smithi090 ceph-mon[31674]: purged_snaps scrub starts 2024-04-02T14:29:36.439 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:29:36 smithi090 ceph-mon[31674]: purged_snaps scrub ok 2024-04-02T14:29:36.439 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:29:36 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-02T14:29:36.440 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:29:36 smithi090 ceph-mon[31674]: Detected new or changed devices on smithi092 2024-04-02T14:29:36.440 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:29:36 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:29:36.440 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:29:36 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:29:36.440 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:29:36 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-02T14:29:36.440 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:29:36 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-02T14:29:36.440 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:29:36 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-02T14:29:36.440 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:29:36 smithi090 ceph-mon[31674]: Adjusting osd_memory_target on smithi092 to 5693M 2024-04-02T14:29:36.440 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:29:36 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:29:36.440 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:29:36 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:29:36.440 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:29:36 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T14:29:36.440 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:29:36 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:29:36.440 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:29:36 smithi090 ceph-mon[31674]: osd.7 [v2:172.21.15.92:6816/3371649139,v1:172.21.15.92:6817/3371649139] boot 2024-04-02T14:29:36.440 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:29:36 smithi090 ceph-mon[31674]: osdmap e52: 8 total, 8 up, 8 in 2024-04-02T14:29:36.440 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:29:36 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-02T14:29:36.611 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:29:36 smithi092 ceph-mon[36497]: purged_snaps scrub starts 2024-04-02T14:29:36.611 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:29:36 smithi092 ceph-mon[36497]: purged_snaps scrub ok 2024-04-02T14:29:36.611 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:29:36 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-02T14:29:36.611 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:29:36 smithi092 ceph-mon[36497]: Detected new or changed devices on smithi092 2024-04-02T14:29:36.611 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:29:36 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:29:36.611 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:29:36 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:29:36.611 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:29:36 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-02T14:29:36.611 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:29:36 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-02T14:29:36.611 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:29:36 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-02T14:29:36.611 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:29:36 smithi092 ceph-mon[36497]: Adjusting osd_memory_target on smithi092 to 5693M 2024-04-02T14:29:36.611 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:29:36 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:29:36.611 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:29:36 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:29:36.611 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:29:36 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T14:29:36.612 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:29:36 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:29:36.612 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:29:36 smithi092 ceph-mon[36497]: osd.7 [v2:172.21.15.92:6816/3371649139,v1:172.21.15.92:6817/3371649139] boot 2024-04-02T14:29:36.612 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:29:36 smithi092 ceph-mon[36497]: osdmap e52: 8 total, 8 up, 8 in 2024-04-02T14:29:36.612 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:29:36 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-02T14:29:37.171 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:29:37.167+0000 7f1b66f1c700 1 -- 172.21.15.177:0/3613308786 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1b60102720 msgr2=0x7f1b60102b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:29:37.171 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:29:37.167+0000 7f1b66f1c700 1 --2- 172.21.15.177:0/3613308786 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1b60102720 0x7f1b60102b80 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f1b4c009990 tx=0x7f1b4c009ca0 comp rx=0 tx=0).stop 2024-04-02T14:29:37.171 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:29:37.167+0000 7f1b66f1c700 1 -- 172.21.15.177:0/3613308786 shutdown_connections 2024-04-02T14:29:37.171 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:29:37.167+0000 7f1b66f1c700 1 --2- 172.21.15.177:0/3613308786 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1b601030c0 0x7f1b60109970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:29:37.171 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:29:37.167+0000 7f1b66f1c700 1 --2- 172.21.15.177:0/3613308786 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1b60101520 0x7f1b60101940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:29:37.171 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:29:37.167+0000 7f1b66f1c700 1 --2- 172.21.15.177:0/3613308786 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1b60102720 0x7f1b60102b80 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:29:37.172 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:29:37.167+0000 7f1b66f1c700 1 -- 172.21.15.177:0/3613308786 >> 172.21.15.177:0/3613308786 conn(0x7f1b600fcac0 msgr2=0x7f1b600fef00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:29:37.172 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:29:37.167+0000 7f1b66f1c700 1 -- 172.21.15.177:0/3613308786 shutdown_connections 2024-04-02T14:29:37.172 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:29:37.167+0000 7f1b66f1c700 1 -- 172.21.15.177:0/3613308786 wait complete. 2024-04-02T14:29:37.172 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:29:37.167+0000 7f1b66f1c700 1 Processor -- start 2024-04-02T14:29:37.172 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:29:37.167+0000 7f1b66f1c700 1 -- start start 2024-04-02T14:29:37.172 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:29:37.167+0000 7f1b66f1c700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1b60101520 0x7f1b6006b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:29:37.172 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:29:37.167+0000 7f1b66f1c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1b60102720 0x7f1b6019d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:29:37.172 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:29:37.167+0000 7f1b66f1c700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1b601030c0 0x7f1b6019fdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:29:37.172 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:29:37.167+0000 7f1b66f1c700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f1b6010d250 con 0x7f1b60101520 2024-04-02T14:29:37.173 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:29:37.167+0000 7f1b66f1c700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f1b6010d0d0 con 0x7f1b60102720 2024-04-02T14:29:37.173 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:29:37.167+0000 7f1b66f1c700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f1b6010d3d0 con 0x7f1b601030c0 2024-04-02T14:29:37.173 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:29:37.168+0000 7f1b654b9700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1b601030c0 0x7f1b6019fdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:29:37.173 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:29:37.168+0000 7f1b654b9700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1b601030c0 0x7f1b6019fdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.177:48894/0 (socket says 172.21.15.177:48894) 2024-04-02T14:29:37.173 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:29:37.168+0000 7f1b654b9700 1 -- 172.21.15.177:0/3806390003 learned_addr learned my addr 172.21.15.177:0/3806390003 (peer_addr_for_me v2:172.21.15.177:0/0) 2024-04-02T14:29:37.173 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:29:37.168+0000 7f1b5ffff700 1 --2- 172.21.15.177:0/3806390003 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1b60102720 0x7f1b6019d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:29:37.173 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:29:37.168+0000 7f1b654b9700 1 -- 172.21.15.177:0/3806390003 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1b60102720 msgr2=0x7f1b6019d780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:29:37.173 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:29:37.168+0000 7f1b654b9700 1 --2- 172.21.15.177:0/3806390003 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1b60102720 0x7f1b6019d780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:29:37.173 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:29:37.168+0000 7f1b654b9700 1 -- 172.21.15.177:0/3806390003 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1b60101520 msgr2=0x7f1b6006b7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:29:37.173 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:29:37.168+0000 7f1b654b9700 1 --2- 172.21.15.177:0/3806390003 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1b60101520 0x7f1b6006b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:29:37.174 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:29:37.168+0000 7f1b654b9700 1 -- 172.21.15.177:0/3806390003 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b601a0690 con 0x7f1b601030c0 2024-04-02T14:29:37.174 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:29:37.168+0000 7f1b654b9700 1 --2- 172.21.15.177:0/3806390003 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1b601030c0 0x7f1b6019fdc0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f1b5800b970 tx=0x7f1b5800bd30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:29:37.174 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:29:37.168+0000 7f1b5dffb700 1 -- 172.21.15.177:0/3806390003 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b58019070 con 0x7f1b601030c0 2024-04-02T14:29:37.174 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:29:37.168+0000 7f1b5dffb700 1 -- 172.21.15.177:0/3806390003 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1b58014440 con 0x7f1b601030c0 2024-04-02T14:29:37.174 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:29:37.168+0000 7f1b66f1c700 1 -- 172.21.15.177:0/3806390003 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b601a0980 con 0x7f1b601030c0 2024-04-02T14:29:37.174 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:29:37.168+0000 7f1b5dffb700 1 -- 172.21.15.177:0/3806390003 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b580075d0 con 0x7f1b601030c0 2024-04-02T14:29:37.174 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:29:37.168+0000 7f1b66f1c700 1 -- 172.21.15.177:0/3806390003 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b601a0f40 con 0x7f1b601030c0 2024-04-02T14:29:37.174 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:29:37.169+0000 7f1b66f1c700 1 -- 172.21.15.177:0/3806390003 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1b600692b0 con 0x7f1b601030c0 2024-04-02T14:29:37.175 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:29:37.170+0000 7f1b5dffb700 1 -- 172.21.15.177:0/3806390003 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f1b5801c070 con 0x7f1b601030c0 2024-04-02T14:29:37.176 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:29:37.171+0000 7f1b5dffb700 1 --2- 172.21.15.177:0/3806390003 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f1b500a3c40 0x7f1b500a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:29:37.176 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:29:37.171+0000 7f1b5dffb700 1 -- 172.21.15.177:0/3806390003 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 5426+0+0 (secure 0 0 0) 0x7f1b580cecc0 con 0x7f1b601030c0 2024-04-02T14:29:37.176 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:29:37.172+0000 7f1b64cb8700 1 --2- 172.21.15.177:0/3806390003 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f1b500a3c40 0x7f1b500a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:29:37.176 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:29:37.173+0000 7f1b64cb8700 1 --2- 172.21.15.177:0/3806390003 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f1b500a3c40 0x7f1b500a6100 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f1b60102580 tx=0x7f1b54006d20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:29:37.177 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:29:37.176+0000 7f1b5dffb700 1 -- 172.21.15.177:0/3806390003 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7f1b5809c1e0 con 0x7f1b601030c0 2024-04-02T14:29:37.385 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:29:37.383+0000 7f1b66f1c700 1 -- 172.21.15.177:0/3806390003 --> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi177:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f1b60062a30 con 0x7f1b500a3c40 2024-04-02T14:29:38.015 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:29:37 smithi177 ceph-mon[36431]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 649 MiB used, 715 GiB / 715 GiB avail 2024-04-02T14:29:38.015 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:29:37 smithi177 ceph-mon[36431]: osdmap e53: 8 total, 8 up, 8 in 2024-04-02T14:29:38.015 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:29:37 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-02T14:29:38.015 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:29:37 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-02T14:29:38.015 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:29:37 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:29:38.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:29:37 smithi092 ceph-mon[36497]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 649 MiB used, 715 GiB / 715 GiB avail 2024-04-02T14:29:38.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:29:37 smithi092 ceph-mon[36497]: osdmap e53: 8 total, 8 up, 8 in 2024-04-02T14:29:38.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:29:37 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-02T14:29:38.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:29:37 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-02T14:29:38.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:29:37 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:29:38.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:29:37 smithi090 ceph-mon[31674]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 649 MiB used, 715 GiB / 715 GiB avail 2024-04-02T14:29:38.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:29:37 smithi090 ceph-mon[31674]: osdmap e53: 8 total, 8 up, 8 in 2024-04-02T14:29:38.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:29:37 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-02T14:29:38.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:29:37 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-02T14:29:38.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:29:37 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:29:38.824 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:29:38 smithi177 ceph-mon[36431]: from='client.24316 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi177:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T14:29:38.824 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:29:38 smithi177 ceph-mon[36431]: osdmap e54: 8 total, 8 up, 8 in 2024-04-02T14:29:39.110 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:29:38 smithi092 ceph-mon[36497]: from='client.24316 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi177:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T14:29:39.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:29:38 smithi092 ceph-mon[36497]: osdmap e54: 8 total, 8 up, 8 in 2024-04-02T14:29:39.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:29:38 smithi090 ceph-mon[31674]: from='client.24316 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi177:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T14:29:39.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:29:38 smithi090 ceph-mon[31674]: osdmap e54: 8 total, 8 up, 8 in 2024-04-02T14:29:39.874 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:29:39 smithi090 ceph-mon[31674]: pgmap v196: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 649 MiB used, 715 GiB / 715 GiB avail; 2/6 objects degraded (33.333%) 2024-04-02T14:29:39.875 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:29:39 smithi090 ceph-mon[31674]: Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2024-04-02T14:29:39.875 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:29:39 smithi090 ceph-mon[31674]: Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2024-04-02T14:29:39.875 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:29:39 smithi090 ceph-mon[31674]: osdmap e55: 8 total, 8 up, 8 in 2024-04-02T14:29:40.110 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:29:39 smithi092 ceph-mon[36497]: pgmap v196: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 649 MiB used, 715 GiB / 715 GiB avail; 2/6 objects degraded (33.333%) 2024-04-02T14:29:40.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:29:39 smithi092 ceph-mon[36497]: Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2024-04-02T14:29:40.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:29:39 smithi092 ceph-mon[36497]: Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2024-04-02T14:29:40.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:29:39 smithi092 ceph-mon[36497]: osdmap e55: 8 total, 8 up, 8 in 2024-04-02T14:29:40.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:29:39 smithi177 ceph-mon[36431]: pgmap v196: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 649 MiB used, 715 GiB / 715 GiB avail; 2/6 objects degraded (33.333%) 2024-04-02T14:29:40.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:29:39 smithi177 ceph-mon[36431]: Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2024-04-02T14:29:40.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:29:39 smithi177 ceph-mon[36431]: Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2024-04-02T14:29:40.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:29:39 smithi177 ceph-mon[36431]: osdmap e55: 8 total, 8 up, 8 in 2024-04-02T14:29:41.110 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:29:40 smithi092 ceph-mon[36497]: osdmap e56: 8 total, 8 up, 8 in 2024-04-02T14:29:41.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:29:40 smithi092 ceph-mon[36497]: from='client.? 172.21.15.177:0/44889577' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "82624ec2-b382-4886-9f40-7f5f376de3ce"}]: dispatch 2024-04-02T14:29:41.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:29:40 smithi092 ceph-mon[36497]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "82624ec2-b382-4886-9f40-7f5f376de3ce"}]: dispatch 2024-04-02T14:29:41.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:29:40 smithi092 ceph-mon[36497]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "82624ec2-b382-4886-9f40-7f5f376de3ce"}]': finished 2024-04-02T14:29:41.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:29:40 smithi092 ceph-mon[36497]: osdmap e57: 9 total, 8 up, 9 in 2024-04-02T14:29:41.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:29:40 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-02T14:29:41.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:29:40 smithi092 ceph-mon[36497]: from='client.? 172.21.15.177:0/3063511725' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-02T14:29:41.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:29:40 smithi177 ceph-mon[36431]: osdmap e56: 8 total, 8 up, 8 in 2024-04-02T14:29:41.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:29:40 smithi177 ceph-mon[36431]: from='client.? 172.21.15.177:0/44889577' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "82624ec2-b382-4886-9f40-7f5f376de3ce"}]: dispatch 2024-04-02T14:29:41.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:29:40 smithi177 ceph-mon[36431]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "82624ec2-b382-4886-9f40-7f5f376de3ce"}]: dispatch 2024-04-02T14:29:41.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:29:40 smithi177 ceph-mon[36431]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "82624ec2-b382-4886-9f40-7f5f376de3ce"}]': finished 2024-04-02T14:29:41.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:29:40 smithi177 ceph-mon[36431]: osdmap e57: 9 total, 8 up, 9 in 2024-04-02T14:29:41.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:29:40 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-02T14:29:41.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:29:40 smithi177 ceph-mon[36431]: from='client.? 172.21.15.177:0/3063511725' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-02T14:29:41.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:29:40 smithi090 ceph-mon[31674]: osdmap e56: 8 total, 8 up, 8 in 2024-04-02T14:29:41.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:29:40 smithi090 ceph-mon[31674]: from='client.? 172.21.15.177:0/44889577' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "82624ec2-b382-4886-9f40-7f5f376de3ce"}]: dispatch 2024-04-02T14:29:41.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:29:40 smithi090 ceph-mon[31674]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "82624ec2-b382-4886-9f40-7f5f376de3ce"}]: dispatch 2024-04-02T14:29:41.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:29:40 smithi090 ceph-mon[31674]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "82624ec2-b382-4886-9f40-7f5f376de3ce"}]': finished 2024-04-02T14:29:41.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:29:40 smithi090 ceph-mon[31674]: osdmap e57: 9 total, 8 up, 9 in 2024-04-02T14:29:41.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:29:40 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-02T14:29:41.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:29:40 smithi090 ceph-mon[31674]: from='client.? 172.21.15.177:0/3063511725' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-02T14:29:42.110 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:29:41 smithi092 ceph-mon[36497]: pgmap v200: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 649 MiB used, 715 GiB / 715 GiB avail; 2/6 objects degraded (33.333%) 2024-04-02T14:29:42.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:29:41 smithi177 ceph-mon[36431]: pgmap v200: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 649 MiB used, 715 GiB / 715 GiB avail; 2/6 objects degraded (33.333%) 2024-04-02T14:29:42.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:29:41 smithi090 ceph-mon[31674]: pgmap v200: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 649 MiB used, 715 GiB / 715 GiB avail; 2/6 objects degraded (33.333%) 2024-04-02T14:29:43.110 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:29:42 smithi092 ceph-mon[36497]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive) 2024-04-02T14:29:43.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:29:42 smithi092 ceph-mon[36497]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2024-04-02T14:29:43.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:29:42 smithi092 ceph-mon[36497]: Cluster is now healthy 2024-04-02T14:29:43.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:29:42 smithi177 ceph-mon[36431]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive) 2024-04-02T14:29:43.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:29:42 smithi177 ceph-mon[36431]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2024-04-02T14:29:43.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:29:42 smithi177 ceph-mon[36431]: Cluster is now healthy 2024-04-02T14:29:43.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:29:42 smithi090 ceph-mon[31674]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive) 2024-04-02T14:29:43.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:29:42 smithi090 ceph-mon[31674]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2024-04-02T14:29:43.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:29:42 smithi090 ceph-mon[31674]: Cluster is now healthy 2024-04-02T14:29:44.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:29:43 smithi092 ceph-mon[36497]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 250 MiB used, 715 GiB / 715 GiB avail 2024-04-02T14:29:44.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:29:43 smithi177 ceph-mon[36431]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 250 MiB used, 715 GiB / 715 GiB avail 2024-04-02T14:29:44.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:29:43 smithi090 ceph-mon[31674]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 250 MiB used, 715 GiB / 715 GiB avail 2024-04-02T14:29:46.110 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:29:45 smithi092 ceph-mon[36497]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-04-02T14:29:46.119 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:29:45 smithi177 ceph-mon[36431]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-04-02T14:29:46.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:29:45 smithi090 ceph-mon[31674]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-04-02T14:29:48.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:29:47 smithi092 ceph-mon[36497]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-04-02T14:29:48.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:29:47 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-04-02T14:29:48.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:29:47 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:29:48.128 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:29:47 smithi177 ceph-mon[36431]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-04-02T14:29:48.128 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:29:47 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-04-02T14:29:48.128 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:29:47 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:29:48.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:29:47 smithi090 ceph-mon[31674]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-04-02T14:29:48.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:29:47 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-04-02T14:29:48.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:29:47 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:29:49.102 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:29:48 smithi177 ceph-mon[36431]: Deploying daemon osd.8 on smithi177 2024-04-02T14:29:49.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:29:48 smithi090 ceph-mon[31674]: Deploying daemon osd.8 on smithi177 2024-04-02T14:29:49.360 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:29:48 smithi092 ceph-mon[36497]: Deploying daemon osd.8 on smithi177 2024-04-02T14:29:50.006 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:29:49 smithi177 ceph-mon[36431]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-04-02T14:29:50.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:29:49 smithi090 ceph-mon[31674]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-04-02T14:29:50.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:29:49 smithi092 ceph-mon[36497]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-04-02T14:29:52.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:29:51 smithi177 ceph-mon[36431]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-04-02T14:29:52.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:29:51 smithi090 ceph-mon[31674]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-04-02T14:29:52.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:29:51 smithi092 ceph-mon[36497]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-04-02T14:29:53.861 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:29:53 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:29:53.861 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:29:53 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:29:53.861 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:29:53 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T14:29:53.861 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:29:53 smithi092 ceph-mon[36497]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-04-02T14:29:53.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:29:53 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:29:53.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:29:53 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:29:53.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:29:53 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T14:29:53.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:29:53 smithi177 ceph-mon[36431]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-04-02T14:29:53.938 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:29:53 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:29:53.938 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:29:53 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:29:53.938 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:29:53 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T14:29:53.938 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:29:53 smithi090 ceph-mon[31674]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-04-02T14:29:55.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:29:55 smithi177 ceph-mon[36431]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-04-02T14:29:55.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:29:55 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:29:55.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:29:55 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:29:55.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:29:55 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:29:55.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:29:55 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T14:29:55.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:29:55 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:29:56.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:29:55 smithi092 ceph-mon[36497]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-04-02T14:29:56.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:29:55 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:29:56.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:29:55 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:29:56.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:29:55 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:29:56.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:29:55 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T14:29:56.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:29:55 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:29:56.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:29:55 smithi090 ceph-mon[31674]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-04-02T14:29:56.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:29:55 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:29:56.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:29:55 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:29:56.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:29:55 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:29:56.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:29:55 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T14:29:56.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:29:55 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:29:56.364 INFO:teuthology.orchestra.run.smithi177.stdout:Created osd(s) 8 on host 'smithi177' 2024-04-02T14:29:56.364 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:29:56.362+0000 7f1b5dffb700 1 -- 172.21.15.177:0/3806390003 <== mgr.14152 v2:172.21.15.90:6800/4266054817 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f1b60062a30 con 0x7f1b500a3c40 2024-04-02T14:29:56.365 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:29:56.364+0000 7f1b66f1c700 1 -- 172.21.15.177:0/3806390003 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f1b500a3c40 msgr2=0x7f1b500a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:29:56.365 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:29:56.364+0000 7f1b66f1c700 1 --2- 172.21.15.177:0/3806390003 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f1b500a3c40 0x7f1b500a6100 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f1b60102580 tx=0x7f1b54006d20 comp rx=0 tx=0).stop 2024-04-02T14:29:56.365 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:29:56.364+0000 7f1b66f1c700 1 -- 172.21.15.177:0/3806390003 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1b601030c0 msgr2=0x7f1b6019fdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:29:56.366 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:29:56.364+0000 7f1b66f1c700 1 --2- 172.21.15.177:0/3806390003 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1b601030c0 0x7f1b6019fdc0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f1b5800b970 tx=0x7f1b5800bd30 comp rx=0 tx=0).stop 2024-04-02T14:29:56.366 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:29:56.364+0000 7f1b66f1c700 1 -- 172.21.15.177:0/3806390003 shutdown_connections 2024-04-02T14:29:56.366 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:29:56.364+0000 7f1b66f1c700 1 --2- 172.21.15.177:0/3806390003 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f1b500a3c40 0x7f1b500a6100 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:29:56.366 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:29:56.364+0000 7f1b66f1c700 1 --2- 172.21.15.177:0/3806390003 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1b60101520 0x7f1b6006b7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:29:56.366 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:29:56.364+0000 7f1b66f1c700 1 --2- 172.21.15.177:0/3806390003 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1b601030c0 0x7f1b6019fdc0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:29:56.366 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:29:56.364+0000 7f1b66f1c700 1 --2- 172.21.15.177:0/3806390003 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1b60102720 0x7f1b6019d780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:29:56.366 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:29:56.364+0000 7f1b66f1c700 1 -- 172.21.15.177:0/3806390003 >> 172.21.15.177:0/3806390003 conn(0x7f1b600fcac0 msgr2=0x7f1b60105af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:29:56.366 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:29:56.364+0000 7f1b66f1c700 1 -- 172.21.15.177:0/3806390003 shutdown_connections 2024-04-02T14:29:56.366 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:29:56.364+0000 7f1b66f1c700 1 -- 172.21.15.177:0/3806390003 wait complete. 2024-04-02T14:29:56.938 DEBUG:teuthology.orchestra.run.smithi177:osd.8> sudo journalctl -f -n 0 -u ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4@osd.8.service 2024-04-02T14:29:56.940 INFO:tasks.cephadm:Deploying osd.9 on smithi090 with /dev/vg_nvme/lv_1... 2024-04-02T14:29:56.940 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:191d963ac9de7322da8aff20b567b3fa458e2ef6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6449448a-f0fc-11ee-b647-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-02T14:29:57.391 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T14:29:57.611 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:29:57 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:29:57.611 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:29:57 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:29:57.611 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:29:57 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T14:29:57.611 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:29:57 smithi092 ceph-mon[36497]: from='osd.8 [v2:172.21.15.177:6816/1459053034,v1:172.21.15.177:6817/1459053034]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-02T14:29:57.611 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:29:57 smithi092 ceph-mon[36497]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-02T14:29:57.611 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:29:57 smithi092 ceph-mon[36497]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-04-02T14:29:57.644 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:29:57 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:29:57.644 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:29:57 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:29:57.644 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:29:57 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T14:29:57.644 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:29:57 smithi090 ceph-mon[31674]: from='osd.8 [v2:172.21.15.177:6816/1459053034,v1:172.21.15.177:6817/1459053034]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-02T14:29:57.644 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:29:57 smithi090 ceph-mon[31674]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-02T14:29:57.644 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:29:57 smithi090 ceph-mon[31674]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-04-02T14:29:57.686 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:29:57 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:29:57.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:29:57 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:29:57.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:29:57 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T14:29:57.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:29:57 smithi177 ceph-mon[36431]: from='osd.8 [v2:172.21.15.177:6816/1459053034,v1:172.21.15.177:6817/1459053034]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-02T14:29:57.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:29:57 smithi177 ceph-mon[36431]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-02T14:29:57.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:29:57 smithi177 ceph-mon[36431]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-04-02T14:29:58.542 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:29:58 smithi090 ceph-mon[31674]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-04-02T14:29:58.542 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:29:58 smithi090 ceph-mon[31674]: from='osd.8 [v2:172.21.15.177:6816/1459053034,v1:172.21.15.177:6817/1459053034]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-04-02T14:29:58.543 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:29:58 smithi090 ceph-mon[31674]: osdmap e58: 9 total, 8 up, 9 in 2024-04-02T14:29:58.543 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:29:58 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-02T14:29:58.543 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:29:58 smithi090 ceph-mon[31674]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-04-02T14:29:58.610 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:29:58 smithi092 ceph-mon[36497]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-04-02T14:29:58.611 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:29:58 smithi092 ceph-mon[36497]: from='osd.8 [v2:172.21.15.177:6816/1459053034,v1:172.21.15.177:6817/1459053034]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-04-02T14:29:58.611 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:29:58 smithi092 ceph-mon[36497]: osdmap e58: 9 total, 8 up, 9 in 2024-04-02T14:29:58.611 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:29:58 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-02T14:29:58.611 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:29:58 smithi092 ceph-mon[36497]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-04-02T14:29:58.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:29:58 smithi177 ceph-mon[36431]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-04-02T14:29:58.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:29:58 smithi177 ceph-mon[36431]: from='osd.8 [v2:172.21.15.177:6816/1459053034,v1:172.21.15.177:6817/1459053034]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-04-02T14:29:58.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:29:58 smithi177 ceph-mon[36431]: osdmap e58: 9 total, 8 up, 9 in 2024-04-02T14:29:58.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:29:58 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-02T14:29:58.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:29:58 smithi177 ceph-mon[36431]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-04-02T14:29:59.500 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:29:59 smithi090 ceph-mon[31674]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]': finished 2024-04-02T14:29:59.500 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:29:59 smithi090 ceph-mon[31674]: osdmap e59: 9 total, 8 up, 9 in 2024-04-02T14:29:59.500 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:29:59 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-02T14:29:59.500 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:29:59 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-02T14:29:59.500 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:29:59 smithi090 ceph-mon[31674]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-04-02T14:29:59.500 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:29:59 smithi090 ceph-mon[31674]: from='osd.8 ' entity='osd.8' 2024-04-02T14:29:59.500 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:29:59 smithi090 ceph-mon[31674]: osdmap e60: 9 total, 8 up, 9 in 2024-04-02T14:29:59.500 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:29:59 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-02T14:29:59.611 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:29:59 smithi092 ceph-mon[36497]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]': finished 2024-04-02T14:29:59.611 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:29:59 smithi092 ceph-mon[36497]: osdmap e59: 9 total, 8 up, 9 in 2024-04-02T14:29:59.611 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:29:59 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-02T14:29:59.611 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:29:59 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-02T14:29:59.611 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:29:59 smithi092 ceph-mon[36497]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-04-02T14:29:59.611 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:29:59 smithi092 ceph-mon[36497]: from='osd.8 ' entity='osd.8' 2024-04-02T14:29:59.611 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:29:59 smithi092 ceph-mon[36497]: osdmap e60: 9 total, 8 up, 9 in 2024-04-02T14:29:59.611 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:29:59 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-02T14:29:59.686 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:29:59 smithi177 ceph-mon[36431]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]': finished 2024-04-02T14:29:59.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:29:59 smithi177 ceph-mon[36431]: osdmap e59: 9 total, 8 up, 9 in 2024-04-02T14:29:59.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:29:59 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-02T14:29:59.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:29:59 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-02T14:29:59.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:29:59 smithi177 ceph-mon[36431]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-04-02T14:29:59.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:29:59 smithi177 ceph-mon[36431]: from='osd.8 ' entity='osd.8' 2024-04-02T14:29:59.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:29:59 smithi177 ceph-mon[36431]: osdmap e60: 9 total, 8 up, 9 in 2024-04-02T14:29:59.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:29:59 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-02T14:29:59.687 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 02 14:29:59 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-8[54680]: 2024-04-02T14:29:59.239+0000 7f64f95db700 -1 osd.8 0 waiting for initial osdmap 2024-04-02T14:29:59.687 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 02 14:29:59 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-8[54680]: 2024-04-02T14:29:59.424+0000 7f64f0bc5700 -1 osd.8 60 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-02T14:30:00.053 INFO:teuthology.orchestra.run.smithi090.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-02T14:30:00.053 INFO:teuthology.orchestra.run.smithi090.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-02T14:30:00.053 INFO:teuthology.orchestra.run.smithi090.stderr: stderr: 10+0 records in 2024-04-02T14:30:00.053 INFO:teuthology.orchestra.run.smithi090.stderr:10+0 records out 2024-04-02T14:30:00.053 INFO:teuthology.orchestra.run.smithi090.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0333204 s, 315 MB/s 2024-04-02T14:30:00.053 INFO:teuthology.orchestra.run.smithi090.stderr:--> Zapping successful for: 2024-04-02T14:30:00.611 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:30:00 smithi092 ceph-mon[36497]: purged_snaps scrub starts 2024-04-02T14:30:00.611 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:30:00 smithi092 ceph-mon[36497]: purged_snaps scrub ok 2024-04-02T14:30:00.611 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:30:00 smithi092 ceph-mon[36497]: overall HEALTH_OK 2024-04-02T14:30:00.611 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:30:00 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:30:00.611 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:30:00 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:30:00.611 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:30:00 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-02T14:30:00.611 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:30:00 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-02T14:30:00.611 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:30:00 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-02T14:30:00.611 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:30:00 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:30:00.611 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:30:00 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:30:00.612 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:30:00 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T14:30:00.612 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:30:00 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:30:00.612 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:30:00 smithi092 ceph-mon[36497]: osd.8 [v2:172.21.15.177:6816/1459053034,v1:172.21.15.177:6817/1459053034] boot 2024-04-02T14:30:00.612 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:30:00 smithi092 ceph-mon[36497]: osdmap e61: 9 total, 9 up, 9 in 2024-04-02T14:30:00.612 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:30:00 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-02T14:30:00.686 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:30:00 smithi177 ceph-mon[36431]: purged_snaps scrub starts 2024-04-02T14:30:00.686 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:30:00 smithi177 ceph-mon[36431]: purged_snaps scrub ok 2024-04-02T14:30:00.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:30:00 smithi177 ceph-mon[36431]: overall HEALTH_OK 2024-04-02T14:30:00.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:30:00 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:30:00.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:30:00 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:30:00.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:30:00 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-02T14:30:00.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:30:00 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-02T14:30:00.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:30:00 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-02T14:30:00.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:30:00 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:30:00.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:30:00 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:30:00.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:30:00 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T14:30:00.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:30:00 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:30:00.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:30:00 smithi177 ceph-mon[36431]: osd.8 [v2:172.21.15.177:6816/1459053034,v1:172.21.15.177:6817/1459053034] boot 2024-04-02T14:30:00.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:30:00 smithi177 ceph-mon[36431]: osdmap e61: 9 total, 9 up, 9 in 2024-04-02T14:30:00.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:30:00 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-02T14:30:00.688 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:30:00 smithi090 ceph-mon[31674]: purged_snaps scrub starts 2024-04-02T14:30:00.689 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:30:00 smithi090 ceph-mon[31674]: purged_snaps scrub ok 2024-04-02T14:30:00.689 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:30:00 smithi090 ceph-mon[31674]: overall HEALTH_OK 2024-04-02T14:30:00.689 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:30:00 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:30:00.689 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:30:00 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:30:00.689 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:30:00 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-02T14:30:00.689 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:30:00 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-02T14:30:00.689 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:30:00 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-02T14:30:00.689 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:30:00 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:30:00.689 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:30:00 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:30:00.689 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:30:00 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T14:30:00.689 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:30:00 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:30:00.689 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:30:00 smithi090 ceph-mon[31674]: osd.8 [v2:172.21.15.177:6816/1459053034,v1:172.21.15.177:6817/1459053034] boot 2024-04-02T14:30:00.689 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:30:00 smithi090 ceph-mon[31674]: osdmap e61: 9 total, 9 up, 9 in 2024-04-02T14:30:00.689 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:30:00 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-02T14:30:00.861 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:191d963ac9de7322da8aff20b567b3fa458e2ef6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6449448a-f0fc-11ee-b647-cb9ed24678a4 -- ceph orch daemon add osd smithi090:vg_nvme/lv_1 2024-04-02T14:30:01.321 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T14:30:01.644 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:30:01 smithi090 ceph-mon[31674]: Detected new or changed devices on smithi177 2024-04-02T14:30:01.644 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:30:01 smithi090 ceph-mon[31674]: Adjusting osd_memory_target on smithi177 to 5693M 2024-04-02T14:30:01.644 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:30:01 smithi090 ceph-mon[31674]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-04-02T14:30:01.644 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:30:01 smithi090 ceph-mon[31674]: osdmap e62: 9 total, 9 up, 9 in 2024-04-02T14:30:01.686 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:30:01 smithi177 ceph-mon[36431]: Detected new or changed devices on smithi177 2024-04-02T14:30:01.686 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:30:01 smithi177 ceph-mon[36431]: Adjusting osd_memory_target on smithi177 to 5693M 2024-04-02T14:30:01.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:30:01 smithi177 ceph-mon[36431]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-04-02T14:30:01.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:30:01 smithi177 ceph-mon[36431]: osdmap e62: 9 total, 9 up, 9 in 2024-04-02T14:30:01.860 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:30:01 smithi092 ceph-mon[36497]: Detected new or changed devices on smithi177 2024-04-02T14:30:01.860 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:30:01 smithi092 ceph-mon[36497]: Adjusting osd_memory_target on smithi177 to 5693M 2024-04-02T14:30:01.861 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:30:01 smithi092 ceph-mon[36497]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-04-02T14:30:01.861 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:30:01 smithi092 ceph-mon[36497]: osdmap e62: 9 total, 9 up, 9 in 2024-04-02T14:30:03.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:30:03.355+0000 7f8f8a000700 1 -- 172.21.15.90:0/1723657471 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8f84101840 msgr2=0x7f8f84101cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:30:03.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:30:03.355+0000 7f8f8a000700 1 --2- 172.21.15.90:0/1723657471 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8f84101840 0x7f8f84101cc0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f8f700098f0 tx=0x7f8f70009c00 comp rx=0 tx=0).stop 2024-04-02T14:30:03.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:30:03.356+0000 7f8f8a000700 1 -- 172.21.15.90:0/1723657471 shutdown_connections 2024-04-02T14:30:03.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:30:03.356+0000 7f8f8a000700 1 --2- 172.21.15.90:0/1723657471 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8f84102200 0x7f8f84102680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:30:03.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:30:03.356+0000 7f8f8a000700 1 --2- 172.21.15.90:0/1723657471 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8f84100ee0 0x7f8f84101300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:30:03.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:30:03.356+0000 7f8f8a000700 1 --2- 172.21.15.90:0/1723657471 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8f84101840 0x7f8f84101cc0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:30:03.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:30:03.356+0000 7f8f8a000700 1 -- 172.21.15.90:0/1723657471 >> 172.21.15.90:0/1723657471 conn(0x7f8f840fcac0 msgr2=0x7f8f840fef00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:30:03.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:30:03.356+0000 7f8f8a000700 1 -- 172.21.15.90:0/1723657471 shutdown_connections 2024-04-02T14:30:03.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:30:03.356+0000 7f8f8a000700 1 -- 172.21.15.90:0/1723657471 wait complete. 2024-04-02T14:30:03.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:30:03.357+0000 7f8f8a000700 1 Processor -- start 2024-04-02T14:30:03.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:30:03.357+0000 7f8f8a000700 1 -- start start 2024-04-02T14:30:03.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:30:03.357+0000 7f8f8a000700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8f84100ee0 0x7f8f8419aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:30:03.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:30:03.357+0000 7f8f8a000700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8f84101840 0x7f8f8419afe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:30:03.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:30:03.357+0000 7f8f8a000700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8f84102200 0x7f8f841a2060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:30:03.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:30:03.357+0000 7f8f8a000700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f8f841103e0 con 0x7f8f84100ee0 2024-04-02T14:30:03.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:30:03.357+0000 7f8f8a000700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f8f84110260 con 0x7f8f84102200 2024-04-02T14:30:03.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:30:03.357+0000 7f8f8a000700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f8f84110560 con 0x7f8f84101840 2024-04-02T14:30:03.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:30:03.358+0000 7f8f83400700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8f84100ee0 0x7f8f8419aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:30:03.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:30:03.358+0000 7f8f83400700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8f84100ee0 0x7f8f8419aaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:42530/0 (socket says 172.21.15.90:42530) 2024-04-02T14:30:03.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:30:03.358+0000 7f8f83400700 1 -- 172.21.15.90:0/2566670757 learned_addr learned my addr 172.21.15.90:0/2566670757 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:30:03.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:30:03.358+0000 7f8f82a00700 1 --2- 172.21.15.90:0/2566670757 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8f84101840 0x7f8f8419afe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:30:03.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:30:03.358+0000 7f8f83e00700 1 --2- 172.21.15.90:0/2566670757 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8f84102200 0x7f8f841a2060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:30:03.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:30:03.358+0000 7f8f82a00700 1 -- 172.21.15.90:0/2566670757 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8f84102200 msgr2=0x7f8f841a2060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:30:03.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:30:03.358+0000 7f8f82a00700 1 --2- 172.21.15.90:0/2566670757 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8f84102200 0x7f8f841a2060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:30:03.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:30:03.358+0000 7f8f82a00700 1 -- 172.21.15.90:0/2566670757 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8f84100ee0 msgr2=0x7f8f8419aaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:30:03.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:30:03.358+0000 7f8f82a00700 1 --2- 172.21.15.90:0/2566670757 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8f84100ee0 0x7f8f8419aaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:30:03.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:30:03.358+0000 7f8f82a00700 1 -- 172.21.15.90:0/2566670757 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f841a2700 con 0x7f8f84101840 2024-04-02T14:30:03.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:30:03.358+0000 7f8f83400700 1 --2- 172.21.15.90:0/2566670757 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8f84100ee0 0x7f8f8419aaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:30:03.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:30:03.359+0000 7f8f82a00700 1 --2- 172.21.15.90:0/2566670757 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8f84101840 0x7f8f8419afe0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f8f70009230 tx=0x7f8f70010f80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:30:03.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:30:03.359+0000 7f8f7be00700 1 -- 172.21.15.90:0/2566670757 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f70025070 con 0x7f8f84101840 2024-04-02T14:30:03.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:30:03.359+0000 7f8f8a000700 1 -- 172.21.15.90:0/2566670757 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8f841a2990 con 0x7f8f84101840 2024-04-02T14:30:03.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:30:03.359+0000 7f8f7be00700 1 -- 172.21.15.90:0/2566670757 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8f7000b560 con 0x7f8f84101840 2024-04-02T14:30:03.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:30:03.359+0000 7f8f7be00700 1 -- 172.21.15.90:0/2566670757 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f70003b70 con 0x7f8f84101840 2024-04-02T14:30:03.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:30:03.359+0000 7f8f8a000700 1 -- 172.21.15.90:0/2566670757 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8f841a2ef0 con 0x7f8f84101840 2024-04-02T14:30:03.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:30:03.360+0000 7f8f8a000700 1 -- 172.21.15.90:0/2566670757 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8f840692b0 con 0x7f8f84101840 2024-04-02T14:30:03.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:30:03.364+0000 7f8f7be00700 1 -- 172.21.15.90:0/2566670757 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f8f70028070 con 0x7f8f84101840 2024-04-02T14:30:03.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:30:03.365+0000 7f8f7be00700 1 --2- 172.21.15.90:0/2566670757 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f8f680a3cc0 0x7f8f680a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:30:03.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:30:03.365+0000 7f8f7be00700 1 -- 172.21.15.90:0/2566670757 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(64..64 src has 1..64) ==== 5829+0+0 (secure 0 0 0) 0x7f8f700d0290 con 0x7f8f84101840 2024-04-02T14:30:03.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:30:03.366+0000 7f8f83400700 1 --2- 172.21.15.90:0/2566670757 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f8f680a3cc0 0x7f8f680a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:30:03.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:30:03.366+0000 7f8f7be00700 1 -- 172.21.15.90:0/2566670757 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7f8f70013030 con 0x7f8f84101840 2024-04-02T14:30:03.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:30:03.366+0000 7f8f83400700 1 --2- 172.21.15.90:0/2566670757 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f8f680a3cc0 0x7f8f680a6180 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f8f74009750 tx=0x7f8f74009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:30:03.369 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:30:03 smithi090 ceph-mon[31674]: osdmap e63: 9 total, 9 up, 9 in 2024-04-02T14:30:03.369 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:30:03 smithi090 ceph-mon[31674]: pgmap v217: 1 pgs: 1 remapped+peering; 577 KiB data, 285 MiB used, 804 GiB / 805 GiB avail 2024-04-02T14:30:03.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:30:03.602+0000 7f8f8a000700 1 -- 172.21.15.90:0/2566670757 --> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi090:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f8f84062a30 con 0x7f8f680a3cc0 2024-04-02T14:30:03.610 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:30:03 smithi092 ceph-mon[36497]: osdmap e63: 9 total, 9 up, 9 in 2024-04-02T14:30:03.611 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:30:03 smithi092 ceph-mon[36497]: pgmap v217: 1 pgs: 1 remapped+peering; 577 KiB data, 285 MiB used, 804 GiB / 805 GiB avail 2024-04-02T14:30:03.686 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:30:03 smithi177 ceph-mon[36431]: osdmap e63: 9 total, 9 up, 9 in 2024-04-02T14:30:03.686 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:30:03 smithi177 ceph-mon[36431]: pgmap v217: 1 pgs: 1 remapped+peering; 577 KiB data, 285 MiB used, 804 GiB / 805 GiB avail 2024-04-02T14:30:04.611 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:30:04 smithi092 ceph-mon[36497]: osdmap e64: 9 total, 9 up, 9 in 2024-04-02T14:30:04.611 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:30:04 smithi092 ceph-mon[36497]: from='client.24343 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi090:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T14:30:04.611 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:30:04 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-02T14:30:04.611 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:30:04 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-02T14:30:04.611 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:30:04 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:30:04.686 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:30:04 smithi177 ceph-mon[36431]: osdmap e64: 9 total, 9 up, 9 in 2024-04-02T14:30:04.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:30:04 smithi177 ceph-mon[36431]: from='client.24343 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi090:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T14:30:04.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:30:04 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-02T14:30:04.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:30:04 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-02T14:30:04.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:30:04 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:30:04.688 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:30:04 smithi090 ceph-mon[31674]: osdmap e64: 9 total, 9 up, 9 in 2024-04-02T14:30:04.688 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:30:04 smithi090 ceph-mon[31674]: from='client.24343 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi090:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T14:30:04.688 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:30:04 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-02T14:30:04.689 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:30:04 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-02T14:30:04.689 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:30:04 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:30:05.611 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:30:05 smithi092 ceph-mon[36497]: osdmap e65: 9 total, 9 up, 9 in 2024-04-02T14:30:05.611 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:30:05 smithi092 ceph-mon[36497]: pgmap v220: 1 pgs: 1 remapped+peering; 577 KiB data, 285 MiB used, 804 GiB / 805 GiB avail 2024-04-02T14:30:05.686 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:30:05 smithi177 ceph-mon[36431]: osdmap e65: 9 total, 9 up, 9 in 2024-04-02T14:30:05.686 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:30:05 smithi177 ceph-mon[36431]: pgmap v220: 1 pgs: 1 remapped+peering; 577 KiB data, 285 MiB used, 804 GiB / 805 GiB avail 2024-04-02T14:30:05.689 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:30:05 smithi090 ceph-mon[31674]: osdmap e65: 9 total, 9 up, 9 in 2024-04-02T14:30:05.689 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:30:05 smithi090 ceph-mon[31674]: pgmap v220: 1 pgs: 1 remapped+peering; 577 KiB data, 285 MiB used, 804 GiB / 805 GiB avail 2024-04-02T14:30:08.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:30:07 smithi092 ceph-mon[36497]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 285 MiB used, 804 GiB / 805 GiB avail; 108 KiB/s, 0 objects/s recovering 2024-04-02T14:30:08.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:30:07 smithi092 ceph-mon[36497]: from='client.? 172.21.15.90:0/1152934732' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fbe36e48-3468-42b2-8094-e16552f3cb16"}]: dispatch 2024-04-02T14:30:08.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:30:07 smithi092 ceph-mon[36497]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fbe36e48-3468-42b2-8094-e16552f3cb16"}]: dispatch 2024-04-02T14:30:08.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:30:07 smithi092 ceph-mon[36497]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fbe36e48-3468-42b2-8094-e16552f3cb16"}]': finished 2024-04-02T14:30:08.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:30:07 smithi092 ceph-mon[36497]: osdmap e66: 10 total, 9 up, 10 in 2024-04-02T14:30:08.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:30:07 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-02T14:30:08.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:30:07 smithi177 ceph-mon[36431]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 285 MiB used, 804 GiB / 805 GiB avail; 108 KiB/s, 0 objects/s recovering 2024-04-02T14:30:08.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:30:07 smithi177 ceph-mon[36431]: from='client.? 172.21.15.90:0/1152934732' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fbe36e48-3468-42b2-8094-e16552f3cb16"}]: dispatch 2024-04-02T14:30:08.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:30:07 smithi177 ceph-mon[36431]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fbe36e48-3468-42b2-8094-e16552f3cb16"}]: dispatch 2024-04-02T14:30:08.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:30:07 smithi177 ceph-mon[36431]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fbe36e48-3468-42b2-8094-e16552f3cb16"}]': finished 2024-04-02T14:30:08.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:30:07 smithi177 ceph-mon[36431]: osdmap e66: 10 total, 9 up, 10 in 2024-04-02T14:30:08.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:30:07 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-02T14:30:08.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:30:07 smithi090 ceph-mon[31674]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 285 MiB used, 804 GiB / 805 GiB avail; 108 KiB/s, 0 objects/s recovering 2024-04-02T14:30:08.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:30:07 smithi090 ceph-mon[31674]: from='client.? 172.21.15.90:0/1152934732' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fbe36e48-3468-42b2-8094-e16552f3cb16"}]: dispatch 2024-04-02T14:30:08.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:30:07 smithi090 ceph-mon[31674]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fbe36e48-3468-42b2-8094-e16552f3cb16"}]: dispatch 2024-04-02T14:30:08.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:30:07 smithi090 ceph-mon[31674]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fbe36e48-3468-42b2-8094-e16552f3cb16"}]': finished 2024-04-02T14:30:08.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:30:07 smithi090 ceph-mon[31674]: osdmap e66: 10 total, 9 up, 10 in 2024-04-02T14:30:08.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:30:07 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-02T14:30:09.110 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:30:08 smithi092 ceph-mon[36497]: from='client.? 172.21.15.90:0/2564187001' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-02T14:30:09.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:30:08 smithi177 ceph-mon[36431]: from='client.? 172.21.15.90:0/2564187001' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-02T14:30:09.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:30:08 smithi090 ceph-mon[31674]: from='client.? 172.21.15.90:0/2564187001' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-02T14:30:10.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:30:09 smithi092 ceph-mon[36497]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-02T14:30:10.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:30:09 smithi177 ceph-mon[36431]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-02T14:30:10.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:30:09 smithi090 ceph-mon[31674]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-02T14:30:12.110 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:30:11 smithi092 ceph-mon[36497]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 804 GiB / 805 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-04-02T14:30:12.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:30:11 smithi177 ceph-mon[36431]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 804 GiB / 805 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-04-02T14:30:12.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:30:11 smithi090 ceph-mon[31674]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 804 GiB / 805 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-04-02T14:30:14.110 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:30:13 smithi092 ceph-mon[36497]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 804 GiB / 805 GiB avail; 69 KiB/s, 0 objects/s recovering 2024-04-02T14:30:14.178 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:30:13 smithi090 ceph-mon[31674]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 804 GiB / 805 GiB avail; 69 KiB/s, 0 objects/s recovering 2024-04-02T14:30:14.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:30:13 smithi177 ceph-mon[36431]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 804 GiB / 805 GiB avail; 69 KiB/s, 0 objects/s recovering 2024-04-02T14:30:15.926 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:30:15 smithi090 ceph-mon[31674]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-02T14:30:16.110 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:30:15 smithi092 ceph-mon[36497]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-02T14:30:16.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:30:15 smithi177 ceph-mon[36431]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-02T14:30:18.110 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:30:17 smithi092 ceph-mon[36497]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 804 GiB / 805 GiB avail 2024-04-02T14:30:18.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:30:17 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-04-02T14:30:18.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:30:17 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:30:18.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:30:17 smithi177 ceph-mon[36431]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 804 GiB / 805 GiB avail 2024-04-02T14:30:18.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:30:17 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-04-02T14:30:18.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:30:17 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:30:18.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:30:17 smithi090 ceph-mon[31674]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 804 GiB / 805 GiB avail 2024-04-02T14:30:18.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:30:17 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-04-02T14:30:18.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:30:17 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:30:19.102 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:30:18 smithi177 ceph-mon[36431]: Deploying daemon osd.9 on smithi090 2024-04-02T14:30:19.112 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:30:18 smithi092 ceph-mon[36497]: Deploying daemon osd.9 on smithi090 2024-04-02T14:30:19.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:30:18 smithi090 ceph-mon[31674]: Deploying daemon osd.9 on smithi090 2024-04-02T14:30:20.110 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:30:19 smithi092 ceph-mon[36497]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 804 GiB / 805 GiB avail 2024-04-02T14:30:20.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:30:19 smithi177 ceph-mon[36431]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 804 GiB / 805 GiB avail 2024-04-02T14:30:20.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:30:19 smithi090 ceph-mon[31674]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 804 GiB / 805 GiB avail 2024-04-02T14:30:22.110 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:30:21 smithi092 ceph-mon[36497]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 804 GiB / 805 GiB avail 2024-04-02T14:30:22.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:30:21 smithi177 ceph-mon[36431]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 804 GiB / 805 GiB avail 2024-04-02T14:30:22.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:30:21 smithi090 ceph-mon[31674]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 804 GiB / 805 GiB avail 2024-04-02T14:30:24.106 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:30:23 smithi090 ceph-mon[31674]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 804 GiB / 805 GiB avail 2024-04-02T14:30:24.110 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:30:23 smithi092 ceph-mon[36497]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 804 GiB / 805 GiB avail 2024-04-02T14:30:24.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:30:23 smithi177 ceph-mon[36431]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 804 GiB / 805 GiB avail 2024-04-02T14:30:25.938 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:30:25 smithi090 ceph-mon[31674]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 804 GiB / 805 GiB avail 2024-04-02T14:30:25.938 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:30:25 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:30:25.939 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:30:25 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T14:30:25.939 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:30:25 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:30:26.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:30:25 smithi092 ceph-mon[36497]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 804 GiB / 805 GiB avail 2024-04-02T14:30:26.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:30:25 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:30:26.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:30:25 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T14:30:26.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:30:25 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:30:26.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:30:25 smithi177 ceph-mon[36431]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 804 GiB / 805 GiB avail 2024-04-02T14:30:26.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:30:25 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:30:26.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:30:25 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T14:30:26.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:30:25 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:30:27.939 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:30:27 smithi090 ceph-mon[31674]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 804 GiB / 805 GiB avail 2024-04-02T14:30:28.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:30:27 smithi177 ceph-mon[36431]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 804 GiB / 805 GiB avail 2024-04-02T14:30:28.360 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:30:27 smithi092 ceph-mon[36497]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 804 GiB / 805 GiB avail 2024-04-02T14:30:29.938 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:30:29 smithi090 ceph-mon[31674]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 804 GiB / 805 GiB avail 2024-04-02T14:30:29.938 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:30:29 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:30:29.939 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:30:29 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:30:29.939 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:30:29 smithi090 ceph-mon[31674]: from='osd.9 [v2:172.21.15.90:6826/2592604332,v1:172.21.15.90:6827/2592604332]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-02T14:30:30.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:30:30.111+0000 7f8f7be00700 1 -- 172.21.15.90:0/2566670757 <== mgr.14152 v2:172.21.15.90:6800/4266054817 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f8f84062a30 con 0x7f8f680a3cc0 2024-04-02T14:30:30.114 INFO:teuthology.orchestra.run.smithi090.stdout:Created osd(s) 9 on host 'smithi090' 2024-04-02T14:30:30.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:30:30.115+0000 7f8f8a000700 1 -- 172.21.15.90:0/2566670757 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f8f680a3cc0 msgr2=0x7f8f680a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:30:30.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:30:30.115+0000 7f8f8a000700 1 --2- 172.21.15.90:0/2566670757 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f8f680a3cc0 0x7f8f680a6180 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f8f74009750 tx=0x7f8f74009310 comp rx=0 tx=0).stop 2024-04-02T14:30:30.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:30:30.115+0000 7f8f8a000700 1 -- 172.21.15.90:0/2566670757 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8f84101840 msgr2=0x7f8f8419afe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:30:30.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:30:30.115+0000 7f8f8a000700 1 --2- 172.21.15.90:0/2566670757 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8f84101840 0x7f8f8419afe0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f8f70009230 tx=0x7f8f70010f80 comp rx=0 tx=0).stop 2024-04-02T14:30:30.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:30:30.115+0000 7f8f8a000700 1 -- 172.21.15.90:0/2566670757 shutdown_connections 2024-04-02T14:30:30.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:30:30.115+0000 7f8f8a000700 1 --2- 172.21.15.90:0/2566670757 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f8f680a3cc0 0x7f8f680a6180 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:30:30.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:30:30.115+0000 7f8f8a000700 1 --2- 172.21.15.90:0/2566670757 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8f84100ee0 0x7f8f8419aaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:30:30.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:30:30.115+0000 7f8f8a000700 1 --2- 172.21.15.90:0/2566670757 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8f84102200 0x7f8f841a2060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:30:30.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:30:30.115+0000 7f8f8a000700 1 --2- 172.21.15.90:0/2566670757 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8f84101840 0x7f8f8419afe0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:30:30.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:30:30.115+0000 7f8f8a000700 1 -- 172.21.15.90:0/2566670757 >> 172.21.15.90:0/2566670757 conn(0x7f8f840fcac0 msgr2=0x7f8f840fe5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:30:30.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:30:30.115+0000 7f8f8a000700 1 -- 172.21.15.90:0/2566670757 shutdown_connections 2024-04-02T14:30:30.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:30:30.115+0000 7f8f8a000700 1 -- 172.21.15.90:0/2566670757 wait complete. 2024-04-02T14:30:30.121 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:30:29 smithi092 ceph-mon[36497]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 804 GiB / 805 GiB avail 2024-04-02T14:30:30.121 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:30:29 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:30:30.121 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:30:29 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:30:30.121 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:30:29 smithi092 ceph-mon[36497]: from='osd.9 [v2:172.21.15.90:6826/2592604332,v1:172.21.15.90:6827/2592604332]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-02T14:30:30.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:30:29 smithi177 ceph-mon[36431]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 804 GiB / 805 GiB avail 2024-04-02T14:30:30.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:30:29 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:30:30.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:30:29 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:30:30.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:30:29 smithi177 ceph-mon[36431]: from='osd.9 [v2:172.21.15.90:6826/2592604332,v1:172.21.15.90:6827/2592604332]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-02T14:30:30.953 DEBUG:teuthology.orchestra.run.smithi090:osd.9> sudo journalctl -f -n 0 -u ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4@osd.9.service 2024-04-02T14:30:30.957 INFO:tasks.cephadm:Deploying osd.10 on smithi092 with /dev/vg_nvme/lv_1... 2024-04-02T14:30:30.958 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:191d963ac9de7322da8aff20b567b3fa458e2ef6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6449448a-f0fc-11ee-b647-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-02T14:30:31.190 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:30:30 smithi090 ceph-mon[31674]: from='osd.9 [v2:172.21.15.90:6826/2592604332,v1:172.21.15.90:6827/2592604332]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-04-02T14:30:31.190 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:30:30 smithi090 ceph-mon[31674]: osdmap e67: 10 total, 9 up, 10 in 2024-04-02T14:30:31.190 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:30:30 smithi090 ceph-mon[31674]: from='osd.9 [v2:172.21.15.90:6826/2592604332,v1:172.21.15.90:6827/2592604332]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]: dispatch 2024-04-02T14:30:31.190 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:30:30 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-02T14:30:31.190 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:30:30 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:30:31.190 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:30:30 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:30:31.190 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:30:30 smithi090 ceph-mon[31674]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 804 GiB / 805 GiB avail 2024-04-02T14:30:31.204 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/mon.b/config 2024-04-02T14:30:31.317 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:30:30 smithi092 ceph-mon[36497]: from='osd.9 [v2:172.21.15.90:6826/2592604332,v1:172.21.15.90:6827/2592604332]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-04-02T14:30:31.317 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:30:30 smithi092 ceph-mon[36497]: osdmap e67: 10 total, 9 up, 10 in 2024-04-02T14:30:31.317 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:30:30 smithi092 ceph-mon[36497]: from='osd.9 [v2:172.21.15.90:6826/2592604332,v1:172.21.15.90:6827/2592604332]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]: dispatch 2024-04-02T14:30:31.317 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:30:30 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-02T14:30:31.317 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:30:30 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:30:31.317 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:30:30 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:30:31.317 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:30:30 smithi092 ceph-mon[36497]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 804 GiB / 805 GiB avail 2024-04-02T14:30:31.436 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:30:30 smithi177 ceph-mon[36431]: from='osd.9 [v2:172.21.15.90:6826/2592604332,v1:172.21.15.90:6827/2592604332]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-04-02T14:30:31.437 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:30:30 smithi177 ceph-mon[36431]: osdmap e67: 10 total, 9 up, 10 in 2024-04-02T14:30:31.437 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:30:30 smithi177 ceph-mon[36431]: from='osd.9 [v2:172.21.15.90:6826/2592604332,v1:172.21.15.90:6827/2592604332]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]: dispatch 2024-04-02T14:30:31.437 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:30:30 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-02T14:30:31.437 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:30:30 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:30:31.437 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:30:30 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:30:31.437 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:30:30 smithi177 ceph-mon[36431]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 804 GiB / 805 GiB avail 2024-04-02T14:30:32.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:30:31 smithi090 ceph-mon[31674]: from='osd.9 [v2:172.21.15.90:6826/2592604332,v1:172.21.15.90:6827/2592604332]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]': finished 2024-04-02T14:30:32.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:30:31 smithi090 ceph-mon[31674]: osdmap e68: 10 total, 9 up, 10 in 2024-04-02T14:30:32.190 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:30:31 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-02T14:30:32.190 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:30:31 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-02T14:30:32.190 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:30:31 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:30:31.955+0000 7fc1a9800700 -1 osd.9 0 waiting for initial osdmap 2024-04-02T14:30:32.190 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:30:32 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:30:32.011+0000 7fc19ec00700 -1 osd.9 68 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-02T14:30:32.320 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:30:31 smithi092 ceph-mon[36497]: from='osd.9 [v2:172.21.15.90:6826/2592604332,v1:172.21.15.90:6827/2592604332]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]': finished 2024-04-02T14:30:32.321 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:30:31 smithi092 ceph-mon[36497]: osdmap e68: 10 total, 9 up, 10 in 2024-04-02T14:30:32.321 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:30:31 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-02T14:30:32.321 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:30:31 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-02T14:30:32.436 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:30:31 smithi177 ceph-mon[36431]: from='osd.9 [v2:172.21.15.90:6826/2592604332,v1:172.21.15.90:6827/2592604332]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]': finished 2024-04-02T14:30:32.436 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:30:31 smithi177 ceph-mon[36431]: osdmap e68: 10 total, 9 up, 10 in 2024-04-02T14:30:32.436 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:30:31 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-02T14:30:32.437 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:30:31 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-02T14:30:33.042 INFO:teuthology.orchestra.run.smithi092.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-02T14:30:33.043 INFO:teuthology.orchestra.run.smithi092.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-02T14:30:33.043 INFO:teuthology.orchestra.run.smithi092.stderr: stderr: 10+0 records in 2024-04-02T14:30:33.043 INFO:teuthology.orchestra.run.smithi092.stderr:10+0 records out 2024-04-02T14:30:33.043 INFO:teuthology.orchestra.run.smithi092.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0114336 s, 917 MB/s 2024-04-02T14:30:33.043 INFO:teuthology.orchestra.run.smithi092.stderr:--> Zapping successful for: 2024-04-02T14:30:33.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:30:32 smithi090 ceph-mon[31674]: purged_snaps scrub starts 2024-04-02T14:30:33.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:30:32 smithi090 ceph-mon[31674]: purged_snaps scrub ok 2024-04-02T14:30:33.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:30:32 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-02T14:30:33.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:30:32 smithi090 ceph-mon[31674]: from='osd.9 [v2:172.21.15.90:6826/2592604332,v1:172.21.15.90:6827/2592604332]' entity='osd.9' 2024-04-02T14:30:33.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:30:32 smithi090 ceph-mon[31674]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 804 GiB / 805 GiB avail 2024-04-02T14:30:33.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:30:32 smithi092 ceph-mon[36497]: purged_snaps scrub starts 2024-04-02T14:30:33.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:30:32 smithi092 ceph-mon[36497]: purged_snaps scrub ok 2024-04-02T14:30:33.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:30:32 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-02T14:30:33.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:30:32 smithi092 ceph-mon[36497]: from='osd.9 [v2:172.21.15.90:6826/2592604332,v1:172.21.15.90:6827/2592604332]' entity='osd.9' 2024-04-02T14:30:33.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:30:32 smithi092 ceph-mon[36497]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 804 GiB / 805 GiB avail 2024-04-02T14:30:33.436 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:30:32 smithi177 ceph-mon[36431]: purged_snaps scrub starts 2024-04-02T14:30:33.436 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:30:32 smithi177 ceph-mon[36431]: purged_snaps scrub ok 2024-04-02T14:30:33.437 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:30:32 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-02T14:30:33.437 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:30:32 smithi177 ceph-mon[36431]: from='osd.9 [v2:172.21.15.90:6826/2592604332,v1:172.21.15.90:6827/2592604332]' entity='osd.9' 2024-04-02T14:30:33.437 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:30:32 smithi177 ceph-mon[36431]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 804 GiB / 805 GiB avail 2024-04-02T14:30:33.774 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:191d963ac9de7322da8aff20b567b3fa458e2ef6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6449448a-f0fc-11ee-b647-cb9ed24678a4 -- ceph orch daemon add osd smithi092:vg_nvme/lv_1 2024-04-02T14:30:34.011 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/mon.b/config 2024-04-02T14:30:34.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:30:33 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-02T14:30:34.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:30:33 smithi092 ceph-mon[36497]: osd.9 [v2:172.21.15.90:6826/2592604332,v1:172.21.15.90:6827/2592604332] boot 2024-04-02T14:30:34.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:30:33 smithi092 ceph-mon[36497]: osdmap e69: 10 total, 10 up, 10 in 2024-04-02T14:30:34.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:30:33 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-02T14:30:34.436 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:30:33 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-02T14:30:34.437 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:30:33 smithi177 ceph-mon[36431]: osd.9 [v2:172.21.15.90:6826/2592604332,v1:172.21.15.90:6827/2592604332] boot 2024-04-02T14:30:34.437 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:30:33 smithi177 ceph-mon[36431]: osdmap e69: 10 total, 10 up, 10 in 2024-04-02T14:30:34.437 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:30:33 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-02T14:30:34.438 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:30:33 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-02T14:30:34.438 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:30:33 smithi090 ceph-mon[31674]: osd.9 [v2:172.21.15.90:6826/2592604332,v1:172.21.15.90:6827/2592604332] boot 2024-04-02T14:30:34.438 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:30:33 smithi090 ceph-mon[31674]: osdmap e69: 10 total, 10 up, 10 in 2024-04-02T14:30:34.438 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:30:33 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-02T14:30:35.086 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:30:34 smithi092 ceph-mon[36497]: osdmap e70: 10 total, 10 up, 10 in 2024-04-02T14:30:35.086 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:30:34 smithi092 ceph-mon[36497]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 894 GiB / 894 GiB avail 2024-04-02T14:30:35.406 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:30:34 smithi090 ceph-mon[31674]: osdmap e70: 10 total, 10 up, 10 in 2024-04-02T14:30:35.406 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:30:34 smithi090 ceph-mon[31674]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 894 GiB / 894 GiB avail 2024-04-02T14:30:35.436 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:30:34 smithi177 ceph-mon[36431]: osdmap e70: 10 total, 10 up, 10 in 2024-04-02T14:30:35.437 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:30:34 smithi177 ceph-mon[36431]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 894 GiB / 894 GiB avail 2024-04-02T14:30:35.606 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:30:35.602+0000 7f833ae0a700 1 -- 172.21.15.92:0/3711409010 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8334102860 msgr2=0x7f8334109110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:30:35.606 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:30:35.602+0000 7f833ae0a700 1 --2- 172.21.15.92:0/3711409010 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8334102860 0x7f8334109110 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f832c00b0a0 tx=0x7f832c00b3b0 comp rx=0 tx=0).stop 2024-04-02T14:30:35.606 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:30:35.602+0000 7f833ae0a700 1 -- 172.21.15.92:0/3711409010 shutdown_connections 2024-04-02T14:30:35.606 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:30:35.602+0000 7f833ae0a700 1 --2- 172.21.15.92:0/3711409010 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8334102860 0x7f8334109110 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:30:35.607 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:30:35.602+0000 7f833ae0a700 1 --2- 172.21.15.92:0/3711409010 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8334100cc0 0x7f83341010e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:30:35.607 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:30:35.602+0000 7f833ae0a700 1 --2- 172.21.15.92:0/3711409010 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8334101ec0 0x7f8334102320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:30:35.607 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:30:35.602+0000 7f833ae0a700 1 -- 172.21.15.92:0/3711409010 >> 172.21.15.92:0/3711409010 conn(0x7f83340fc280 msgr2=0x7f83340fe6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:30:35.607 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:30:35.602+0000 7f833ae0a700 1 -- 172.21.15.92:0/3711409010 shutdown_connections 2024-04-02T14:30:35.607 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:30:35.602+0000 7f833ae0a700 1 -- 172.21.15.92:0/3711409010 wait complete. 2024-04-02T14:30:35.607 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:30:35.603+0000 7f833ae0a700 1 Processor -- start 2024-04-02T14:30:35.607 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:30:35.603+0000 7f833ae0a700 1 -- start start 2024-04-02T14:30:35.607 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:30:35.603+0000 7f833ae0a700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8334100cc0 0x7f8334198880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:30:35.607 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:30:35.603+0000 7f833ae0a700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8334101ec0 0x7f8334198dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:30:35.607 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:30:35.603+0000 7f833ae0a700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8334102860 0x7f833419fe40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:30:35.607 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:30:35.603+0000 7f833ae0a700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f833410e330 con 0x7f8334101ec0 2024-04-02T14:30:35.608 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:30:35.603+0000 7f833ae0a700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f833410e1b0 con 0x7f8334100cc0 2024-04-02T14:30:35.608 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:30:35.603+0000 7f833ae0a700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f833410e4b0 con 0x7f8334102860 2024-04-02T14:30:35.608 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:30:35.603+0000 7f8338ba6700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8334100cc0 0x7f8334198880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:30:35.608 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:30:35.603+0000 7f8338ba6700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8334100cc0 0x7f8334198880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:51294/0 (socket says 172.21.15.92:51294) 2024-04-02T14:30:35.608 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:30:35.603+0000 7f8338ba6700 1 -- 172.21.15.92:0/1093846663 learned_addr learned my addr 172.21.15.92:0/1093846663 (peer_addr_for_me v2:172.21.15.92:0/0) 2024-04-02T14:30:35.608 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:30:35.603+0000 7f8338ba6700 1 -- 172.21.15.92:0/1093846663 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8334102860 msgr2=0x7f833419fe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:30:35.608 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:30:35.603+0000 7f8338ba6700 1 --2- 172.21.15.92:0/1093846663 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8334102860 0x7f833419fe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:30:35.608 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:30:35.603+0000 7f8338ba6700 1 -- 172.21.15.92:0/1093846663 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8334101ec0 msgr2=0x7f8334198dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:30:35.608 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:30:35.603+0000 7f8338ba6700 1 --2- 172.21.15.92:0/1093846663 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8334101ec0 0x7f8334198dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:30:35.608 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:30:35.603+0000 7f8338ba6700 1 -- 172.21.15.92:0/1093846663 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83341a04e0 con 0x7f8334100cc0 2024-04-02T14:30:35.608 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:30:35.604+0000 7f8338ba6700 1 --2- 172.21.15.92:0/1093846663 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8334100cc0 0x7f8334198880 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f832800b500 tx=0x7f832800b8c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:30:35.608 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:30:35.604+0000 7f8331ffb700 1 -- 172.21.15.92:0/1093846663 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8328015020 con 0x7f8334100cc0 2024-04-02T14:30:35.609 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:30:35.604+0000 7f833ae0a700 1 -- 172.21.15.92:0/1093846663 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83341a07d0 con 0x7f8334100cc0 2024-04-02T14:30:35.609 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:30:35.604+0000 7f8331ffb700 1 -- 172.21.15.92:0/1093846663 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8328011d10 con 0x7f8334100cc0 2024-04-02T14:30:35.609 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:30:35.604+0000 7f8331ffb700 1 -- 172.21.15.92:0/1093846663 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f832800f2e0 con 0x7f8334100cc0 2024-04-02T14:30:35.609 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:30:35.604+0000 7f833ae0a700 1 -- 172.21.15.92:0/1093846663 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83341a0d40 con 0x7f8334100cc0 2024-04-02T14:30:35.610 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:30:35.605+0000 7f833ae0a700 1 -- 172.21.15.92:0/1093846663 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f83340692b0 con 0x7f8334100cc0 2024-04-02T14:30:35.610 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:30:35.606+0000 7f8331ffb700 1 -- 172.21.15.92:0/1093846663 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f8328018020 con 0x7f8334100cc0 2024-04-02T14:30:35.610 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:30:35.606+0000 7f8331ffb700 1 --2- 172.21.15.92:0/1093846663 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f83240a3c40 0x7f83240a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:30:35.610 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:30:35.607+0000 7f8331ffb700 1 -- 172.21.15.92:0/1093846663 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 6261+0+0 (secure 0 0 0) 0x7f83280cf400 con 0x7f8334100cc0 2024-04-02T14:30:35.610 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:30:35.607+0000 7f8333fff700 1 --2- 172.21.15.92:0/1093846663 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f83240a3c40 0x7f83240a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:30:35.610 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:30:35.608+0000 7f8333fff700 1 --2- 172.21.15.92:0/1093846663 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f83240a3c40 0x7f83240a6100 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f83200097b0 tx=0x7f8320006d20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:30:35.613 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:30:35.611+0000 7f8331ffb700 1 -- 172.21.15.92:0/1093846663 <== mon.2 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7f832801f020 con 0x7f8334100cc0 2024-04-02T14:30:35.825 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:30:35.823+0000 7f833ae0a700 1 -- 172.21.15.92:0/1093846663 --> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi092:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f8334062a30 con 0x7f83240a3c40 2024-04-02T14:30:36.793 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:30:36 smithi092 ceph-mon[36497]: Detected new or changed devices on smithi090 2024-04-02T14:30:36.793 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:30:36 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:30:36.793 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:30:36 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:30:36.793 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:30:36 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-02T14:30:36.793 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:30:36 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-02T14:30:36.793 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:30:36 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-02T14:30:36.793 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:30:36 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-02T14:30:36.793 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:30:36 smithi092 ceph-mon[36497]: Adjusting osd_memory_target on smithi090 to 4272M 2024-04-02T14:30:36.793 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:30:36 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:30:36.793 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:30:36 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:30:36.793 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:30:36 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T14:30:36.794 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:30:36 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:30:36.794 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:30:36 smithi092 ceph-mon[36497]: from='client.24365 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi092:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T14:30:36.794 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:30:36 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-02T14:30:36.794 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:30:36 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-02T14:30:36.794 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:30:36 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:30:36.794 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:30:36 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T14:30:36.794 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:30:36 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:30:36.794 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:30:36 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T14:30:36.794 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:30:36 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:30:36.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:30:36 smithi177 ceph-mon[36431]: Detected new or changed devices on smithi090 2024-04-02T14:30:36.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:30:36 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:30:36.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:30:36 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:30:36.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:30:36 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-02T14:30:36.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:30:36 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-02T14:30:36.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:30:36 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-02T14:30:36.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:30:36 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-02T14:30:36.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:30:36 smithi177 ceph-mon[36431]: Adjusting osd_memory_target on smithi090 to 4272M 2024-04-02T14:30:36.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:30:36 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:30:36.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:30:36 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:30:36.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:30:36 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T14:30:36.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:30:36 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:30:36.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:30:36 smithi177 ceph-mon[36431]: from='client.24365 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi092:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T14:30:36.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:30:36 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-02T14:30:36.938 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:30:36 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-02T14:30:36.938 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:30:36 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:30:36.938 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:30:36 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T14:30:36.938 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:30:36 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:30:36.938 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:30:36 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T14:30:36.938 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:30:36 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:30:36.939 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:30:36 smithi090 ceph-mon[31674]: Detected new or changed devices on smithi090 2024-04-02T14:30:36.939 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:30:36 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:30:36.939 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:30:36 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:30:36.939 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:30:36 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-02T14:30:36.939 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:30:36 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-02T14:30:36.939 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:30:36 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-02T14:30:36.939 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:30:36 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-02T14:30:36.940 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:30:36 smithi090 ceph-mon[31674]: Adjusting osd_memory_target on smithi090 to 4272M 2024-04-02T14:30:36.940 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:30:36 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:30:36.940 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:30:36 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:30:36.940 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:30:36 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T14:30:36.940 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:30:36 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:30:36.940 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:30:36 smithi090 ceph-mon[31674]: from='client.24365 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi092:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T14:30:36.940 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:30:36 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-02T14:30:36.940 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:30:36 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-02T14:30:36.940 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:30:36 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:30:36.940 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:30:36 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T14:30:36.940 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:30:36 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:30:36.940 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:30:36 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T14:30:36.940 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:30:36 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:30:37.610 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:30:37 smithi092 ceph-mon[36497]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 894 GiB / 894 GiB avail 2024-04-02T14:30:37.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:30:37 smithi177 ceph-mon[36431]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 894 GiB / 894 GiB avail 2024-04-02T14:30:37.939 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:30:37 smithi090 ceph-mon[31674]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 894 GiB / 894 GiB avail 2024-04-02T14:30:38.841 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:30:38 smithi090 ceph-mon[31674]: from='client.? 172.21.15.92:0/2745335389' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9240df75-e7e2-47f1-8aae-534b70b3c210"}]: dispatch 2024-04-02T14:30:38.841 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:30:38 smithi090 ceph-mon[31674]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9240df75-e7e2-47f1-8aae-534b70b3c210"}]: dispatch 2024-04-02T14:30:38.841 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:30:38 smithi090 ceph-mon[31674]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9240df75-e7e2-47f1-8aae-534b70b3c210"}]': finished 2024-04-02T14:30:38.841 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:30:38 smithi090 ceph-mon[31674]: osdmap e71: 11 total, 10 up, 11 in 2024-04-02T14:30:38.841 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:30:38 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-02T14:30:38.861 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:30:38 smithi092 ceph-mon[36497]: from='client.? 172.21.15.92:0/2745335389' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9240df75-e7e2-47f1-8aae-534b70b3c210"}]: dispatch 2024-04-02T14:30:38.861 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:30:38 smithi092 ceph-mon[36497]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9240df75-e7e2-47f1-8aae-534b70b3c210"}]: dispatch 2024-04-02T14:30:38.861 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:30:38 smithi092 ceph-mon[36497]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9240df75-e7e2-47f1-8aae-534b70b3c210"}]': finished 2024-04-02T14:30:38.861 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:30:38 smithi092 ceph-mon[36497]: osdmap e71: 11 total, 10 up, 11 in 2024-04-02T14:30:38.861 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:30:38 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-02T14:30:38.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:30:38 smithi177 ceph-mon[36431]: from='client.? 172.21.15.92:0/2745335389' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9240df75-e7e2-47f1-8aae-534b70b3c210"}]: dispatch 2024-04-02T14:30:38.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:30:38 smithi177 ceph-mon[36431]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9240df75-e7e2-47f1-8aae-534b70b3c210"}]: dispatch 2024-04-02T14:30:38.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:30:38 smithi177 ceph-mon[36431]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9240df75-e7e2-47f1-8aae-534b70b3c210"}]': finished 2024-04-02T14:30:38.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:30:38 smithi177 ceph-mon[36431]: osdmap e71: 11 total, 10 up, 11 in 2024-04-02T14:30:38.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:30:38 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-02T14:30:39.858 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:30:39 smithi090 ceph-mon[31674]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 894 GiB / 894 GiB avail 2024-04-02T14:30:39.858 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:30:39 smithi090 ceph-mon[31674]: from='client.? 172.21.15.92:0/2041469987' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-02T14:30:39.860 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:30:39 smithi092 ceph-mon[36497]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 894 GiB / 894 GiB avail 2024-04-02T14:30:39.860 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:30:39 smithi092 ceph-mon[36497]: from='client.? 172.21.15.92:0/2041469987' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-02T14:30:39.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:30:39 smithi177 ceph-mon[36431]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 894 GiB / 894 GiB avail 2024-04-02T14:30:39.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:30:39 smithi177 ceph-mon[36431]: from='client.? 172.21.15.92:0/2041469987' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-02T14:30:42.110 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:30:41 smithi092 ceph-mon[36497]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 894 GiB / 894 GiB avail 2024-04-02T14:30:42.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:30:41 smithi177 ceph-mon[36431]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 894 GiB / 894 GiB avail 2024-04-02T14:30:42.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:30:41 smithi090 ceph-mon[31674]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 894 GiB / 894 GiB avail 2024-04-02T14:30:43.998 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:30:43 smithi092 ceph-mon[36497]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 894 GiB / 894 GiB avail 2024-04-02T14:30:44.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:30:43 smithi177 ceph-mon[36431]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 894 GiB / 894 GiB avail 2024-04-02T14:30:44.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:30:43 smithi090 ceph-mon[31674]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 894 GiB / 894 GiB avail 2024-04-02T14:30:46.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:30:45 smithi092 ceph-mon[36497]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 894 GiB / 894 GiB avail 2024-04-02T14:30:46.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:30:45 smithi177 ceph-mon[36431]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 894 GiB / 894 GiB avail 2024-04-02T14:30:46.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:30:45 smithi090 ceph-mon[31674]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 894 GiB / 894 GiB avail 2024-04-02T14:30:46.919 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:30:46 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-04-02T14:30:46.919 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:30:46 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:30:47.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:30:46 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-04-02T14:30:47.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:30:46 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:30:47.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:30:46 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-04-02T14:30:47.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:30:46 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:30:48.113 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:30:47 smithi092 ceph-mon[36497]: Deploying daemon osd.10 on smithi092 2024-04-02T14:30:48.113 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:30:47 smithi092 ceph-mon[36497]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 894 GiB / 894 GiB avail 2024-04-02T14:30:48.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:30:47 smithi090 ceph-mon[31674]: Deploying daemon osd.10 on smithi092 2024-04-02T14:30:48.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:30:47 smithi090 ceph-mon[31674]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 894 GiB / 894 GiB avail 2024-04-02T14:30:48.193 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:30:47 smithi177 ceph-mon[36431]: Deploying daemon osd.10 on smithi092 2024-04-02T14:30:48.193 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:30:47 smithi177 ceph-mon[36431]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 894 GiB / 894 GiB avail 2024-04-02T14:30:49.861 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:30:49 smithi092 ceph-mon[36497]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 894 GiB / 894 GiB avail 2024-04-02T14:30:50.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:30:49 smithi177 ceph-mon[36431]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 894 GiB / 894 GiB avail 2024-04-02T14:30:50.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:30:49 smithi090 ceph-mon[31674]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 894 GiB / 894 GiB avail 2024-04-02T14:30:51.991 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:30:51 smithi092 ceph-mon[36497]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 894 GiB / 894 GiB avail 2024-04-02T14:30:51.991 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:30:51 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T14:30:51.991 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:30:51 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:30:51.991 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:30:51 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:30:52.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:30:51 smithi177 ceph-mon[36431]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 894 GiB / 894 GiB avail 2024-04-02T14:30:52.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:30:51 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T14:30:52.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:30:51 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:30:52.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:30:51 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:30:52.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:30:51 smithi090 ceph-mon[31674]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 894 GiB / 894 GiB avail 2024-04-02T14:30:52.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:30:51 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T14:30:52.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:30:51 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:30:52.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:30:51 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:30:54.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:30:53 smithi092 ceph-mon[36497]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 894 GiB / 894 GiB avail 2024-04-02T14:30:54.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:30:53 smithi177 ceph-mon[36431]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 894 GiB / 894 GiB avail 2024-04-02T14:30:54.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:30:53 smithi090 ceph-mon[31674]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 894 GiB / 894 GiB avail 2024-04-02T14:30:54.663 INFO:teuthology.orchestra.run.smithi092.stdout:Created osd(s) 10 on host 'smithi092' 2024-04-02T14:30:54.663 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:30:54.661+0000 7f8331ffb700 1 -- 172.21.15.92:0/1093846663 <== mgr.14152 v2:172.21.15.90:6800/4266054817 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7f8334062a30 con 0x7f83240a3c40 2024-04-02T14:30:54.666 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:30:54.664+0000 7f833ae0a700 1 -- 172.21.15.92:0/1093846663 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f83240a3c40 msgr2=0x7f83240a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:30:54.666 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:30:54.664+0000 7f833ae0a700 1 --2- 172.21.15.92:0/1093846663 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f83240a3c40 0x7f83240a6100 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f83200097b0 tx=0x7f8320006d20 comp rx=0 tx=0).stop 2024-04-02T14:30:54.666 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:30:54.664+0000 7f833ae0a700 1 -- 172.21.15.92:0/1093846663 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8334100cc0 msgr2=0x7f8334198880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:30:54.666 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:30:54.664+0000 7f833ae0a700 1 --2- 172.21.15.92:0/1093846663 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8334100cc0 0x7f8334198880 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f832800b500 tx=0x7f832800b8c0 comp rx=0 tx=0).stop 2024-04-02T14:30:54.666 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:30:54.664+0000 7f833ae0a700 1 -- 172.21.15.92:0/1093846663 shutdown_connections 2024-04-02T14:30:54.666 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:30:54.664+0000 7f833ae0a700 1 --2- 172.21.15.92:0/1093846663 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f83240a3c40 0x7f83240a6100 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:30:54.666 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:30:54.664+0000 7f833ae0a700 1 --2- 172.21.15.92:0/1093846663 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8334102860 0x7f833419fe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:30:54.666 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:30:54.664+0000 7f833ae0a700 1 --2- 172.21.15.92:0/1093846663 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8334100cc0 0x7f8334198880 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:30:54.666 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:30:54.664+0000 7f833ae0a700 1 --2- 172.21.15.92:0/1093846663 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8334101ec0 0x7f8334198dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:30:54.666 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:30:54.664+0000 7f833ae0a700 1 -- 172.21.15.92:0/1093846663 >> 172.21.15.92:0/1093846663 conn(0x7f83340fc280 msgr2=0x7f8334105160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:30:54.666 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:30:54.664+0000 7f833ae0a700 1 -- 172.21.15.92:0/1093846663 shutdown_connections 2024-04-02T14:30:54.666 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-02T14:30:54.664+0000 7f833ae0a700 1 -- 172.21.15.92:0/1093846663 wait complete. 2024-04-02T14:30:55.233 DEBUG:teuthology.orchestra.run.smithi092:osd.10> sudo journalctl -f -n 0 -u ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4@osd.10.service 2024-04-02T14:30:55.236 INFO:tasks.cephadm:Deploying osd.11 on smithi177 with /dev/vg_nvme/lv_1... 2024-04-02T14:30:55.236 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:191d963ac9de7322da8aff20b567b3fa458e2ef6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6449448a-f0fc-11ee-b647-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-02T14:30:55.483 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/mon.c/config 2024-04-02T14:30:55.855 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:30:55 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:30:55.855 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:30:55 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:30:55.855 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:30:55 smithi177 ceph-mon[36431]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 894 GiB / 894 GiB avail 2024-04-02T14:30:55.855 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:30:55 smithi177 ceph-mon[36431]: from='osd.10 [v2:172.21.15.92:6824/1594153142,v1:172.21.15.92:6825/1594153142]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-02T14:30:55.856 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:30:55 smithi177 ceph-mon[36431]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-02T14:30:55.856 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:30:55 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:30:55.856 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:30:55 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:30:55.939 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:30:55 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:30:55.939 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:30:55 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:30:55.939 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:30:55 smithi090 ceph-mon[31674]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 894 GiB / 894 GiB avail 2024-04-02T14:30:55.939 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:30:55 smithi090 ceph-mon[31674]: from='osd.10 [v2:172.21.15.92:6824/1594153142,v1:172.21.15.92:6825/1594153142]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-02T14:30:55.939 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:30:55 smithi090 ceph-mon[31674]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-02T14:30:55.939 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:30:55 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:30:55.939 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:30:55 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:30:56.068 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:30:55 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:30:56.068 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:30:55 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:30:56.068 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:30:55 smithi092 ceph-mon[36497]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 894 GiB / 894 GiB avail 2024-04-02T14:30:56.068 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:30:55 smithi092 ceph-mon[36497]: from='osd.10 [v2:172.21.15.92:6824/1594153142,v1:172.21.15.92:6825/1594153142]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-02T14:30:56.068 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:30:55 smithi092 ceph-mon[36497]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-02T14:30:56.068 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:30:55 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:30:56.068 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:30:55 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:30:56.866 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:30:56 smithi177 ceph-mon[36431]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-04-02T14:30:56.866 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:30:56 smithi177 ceph-mon[36431]: from='osd.10 [v2:172.21.15.92:6824/1594153142,v1:172.21.15.92:6825/1594153142]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]: dispatch 2024-04-02T14:30:56.866 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:30:56 smithi177 ceph-mon[36431]: osdmap e72: 11 total, 10 up, 11 in 2024-04-02T14:30:56.867 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:30:56 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-02T14:30:56.867 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:30:56 smithi177 ceph-mon[36431]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]: dispatch 2024-04-02T14:30:56.939 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:30:56 smithi090 ceph-mon[31674]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-04-02T14:30:56.939 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:30:56 smithi090 ceph-mon[31674]: from='osd.10 [v2:172.21.15.92:6824/1594153142,v1:172.21.15.92:6825/1594153142]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]: dispatch 2024-04-02T14:30:56.939 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:30:56 smithi090 ceph-mon[31674]: osdmap e72: 11 total, 10 up, 11 in 2024-04-02T14:30:56.939 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:30:56 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-02T14:30:56.939 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:30:56 smithi090 ceph-mon[31674]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]: dispatch 2024-04-02T14:30:57.024 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:30:56 smithi092 ceph-mon[36497]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-04-02T14:30:57.024 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:30:56 smithi092 ceph-mon[36497]: from='osd.10 [v2:172.21.15.92:6824/1594153142,v1:172.21.15.92:6825/1594153142]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]: dispatch 2024-04-02T14:30:57.024 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:30:56 smithi092 ceph-mon[36497]: osdmap e72: 11 total, 10 up, 11 in 2024-04-02T14:30:57.024 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:30:56 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-02T14:30:57.024 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:30:56 smithi092 ceph-mon[36497]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]: dispatch 2024-04-02T14:30:57.460 INFO:teuthology.orchestra.run.smithi177.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-02T14:30:57.461 INFO:teuthology.orchestra.run.smithi177.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-02T14:30:57.461 INFO:teuthology.orchestra.run.smithi177.stderr: stderr: 10+0 records in 2024-04-02T14:30:57.461 INFO:teuthology.orchestra.run.smithi177.stderr:10+0 records out 2024-04-02T14:30:57.461 INFO:teuthology.orchestra.run.smithi177.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0110849 s, 946 MB/s 2024-04-02T14:30:57.461 INFO:teuthology.orchestra.run.smithi177.stderr:--> Zapping successful for: 2024-04-02T14:30:57.611 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:30:57 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:30:57.539+0000 7f6ae155c700 -1 osd.10 0 waiting for initial osdmap 2024-04-02T14:30:57.776 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:30:57 smithi177 ceph-mon[36431]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]': finished 2024-04-02T14:30:57.777 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:30:57 smithi177 ceph-mon[36431]: osdmap e73: 11 total, 10 up, 11 in 2024-04-02T14:30:57.777 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:30:57 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-02T14:30:57.777 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:30:57 smithi177 ceph-mon[36431]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 894 GiB / 894 GiB avail 2024-04-02T14:30:57.777 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:30:57 smithi177 ceph-mon[36431]: from='osd.10 ' entity='osd.10' 2024-04-02T14:30:57.939 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:30:57 smithi090 ceph-mon[31674]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]': finished 2024-04-02T14:30:57.939 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:30:57 smithi090 ceph-mon[31674]: osdmap e73: 11 total, 10 up, 11 in 2024-04-02T14:30:57.939 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:30:57 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-02T14:30:57.939 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:30:57 smithi090 ceph-mon[31674]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 894 GiB / 894 GiB avail 2024-04-02T14:30:57.939 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:30:57 smithi090 ceph-mon[31674]: from='osd.10 ' entity='osd.10' 2024-04-02T14:30:58.070 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:191d963ac9de7322da8aff20b567b3fa458e2ef6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6449448a-f0fc-11ee-b647-cb9ed24678a4 -- ceph orch daemon add osd smithi177:vg_nvme/lv_1 2024-04-02T14:30:58.110 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:30:57 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:30:57.676+0000 7f6ad8b46700 -1 osd.10 74 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-02T14:30:58.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:30:57 smithi092 ceph-mon[36497]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]': finished 2024-04-02T14:30:58.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:30:57 smithi092 ceph-mon[36497]: osdmap e73: 11 total, 10 up, 11 in 2024-04-02T14:30:58.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:30:57 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-02T14:30:58.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:30:57 smithi092 ceph-mon[36497]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 894 GiB / 894 GiB avail 2024-04-02T14:30:58.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:30:57 smithi092 ceph-mon[36497]: from='osd.10 ' entity='osd.10' 2024-04-02T14:30:58.315 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/mon.c/config 2024-04-02T14:30:58.738 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:30:58 smithi177 ceph-mon[36431]: purged_snaps scrub starts 2024-04-02T14:30:58.739 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:30:58 smithi177 ceph-mon[36431]: purged_snaps scrub ok 2024-04-02T14:30:58.739 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:30:58 smithi177 ceph-mon[36431]: osdmap e74: 11 total, 10 up, 11 in 2024-04-02T14:30:58.739 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:30:58 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-02T14:30:58.739 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:30:58 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-02T14:30:58.848 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:30:58 smithi092 ceph-mon[36497]: purged_snaps scrub starts 2024-04-02T14:30:58.849 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:30:58 smithi092 ceph-mon[36497]: purged_snaps scrub ok 2024-04-02T14:30:58.849 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:30:58 smithi092 ceph-mon[36497]: osdmap e74: 11 total, 10 up, 11 in 2024-04-02T14:30:58.849 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:30:58 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-02T14:30:58.849 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:30:58 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-02T14:30:58.939 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:30:58 smithi090 ceph-mon[31674]: purged_snaps scrub starts 2024-04-02T14:30:58.939 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:30:58 smithi090 ceph-mon[31674]: purged_snaps scrub ok 2024-04-02T14:30:58.939 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:30:58 smithi090 ceph-mon[31674]: osdmap e74: 11 total, 10 up, 11 in 2024-04-02T14:30:58.939 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:30:58 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-02T14:30:58.939 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:30:58 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-02T14:30:59.861 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:30:59 smithi092 ceph-mon[36497]: osd.10 [v2:172.21.15.92:6824/1594153142,v1:172.21.15.92:6825/1594153142] boot 2024-04-02T14:30:59.861 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:30:59 smithi092 ceph-mon[36497]: osdmap e75: 11 total, 11 up, 11 in 2024-04-02T14:30:59.861 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:30:59 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-02T14:30:59.861 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:30:59 smithi092 ceph-mon[36497]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 894 GiB / 894 GiB avail 2024-04-02T14:30:59.861 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:30:59 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:30:59.861 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:30:59 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:30:59.861 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:30:59 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-02T14:30:59.861 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:30:59 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-02T14:30:59.862 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:30:59 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-02T14:30:59.862 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:30:59 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-02T14:30:59.862 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:30:59 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:30:59.862 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:30:59 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:30:59.862 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:30:59 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T14:30:59.862 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:30:59 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:30:59.862 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:30:59 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T14:30:59.862 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:30:59 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:30:59.862 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:30:59 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T14:30:59.862 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:30:59 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:30:59.862 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:30:59 smithi092 ceph-mon[36497]: osdmap e76: 11 total, 11 up, 11 in 2024-04-02T14:30:59.892 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:30:59.888+0000 7f4018e6e700 1 -- 172.21.15.177:0/4229669192 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4014101590 msgr2=0x7f40141019b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:30:59.892 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:30:59.888+0000 7f4018e6e700 1 --2- 172.21.15.177:0/4229669192 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4014101590 0x7f40141019b0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f4004009990 tx=0x7f4004009ca0 comp rx=0 tx=0).stop 2024-04-02T14:30:59.892 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:30:59.889+0000 7f4018e6e700 1 -- 172.21.15.177:0/4229669192 shutdown_connections 2024-04-02T14:30:59.892 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:30:59.889+0000 7f4018e6e700 1 --2- 172.21.15.177:0/4229669192 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4014103130 0x7f40141099e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:30:59.892 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:30:59.889+0000 7f4018e6e700 1 --2- 172.21.15.177:0/4229669192 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4014101590 0x7f40141019b0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:30:59.892 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:30:59.889+0000 7f4018e6e700 1 --2- 172.21.15.177:0/4229669192 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4014102790 0x7f4014102bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:30:59.892 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:30:59.889+0000 7f4018e6e700 1 -- 172.21.15.177:0/4229669192 >> 172.21.15.177:0/4229669192 conn(0x7f40140fcb50 msgr2=0x7f40140fef70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:30:59.892 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:30:59.889+0000 7f4018e6e700 1 -- 172.21.15.177:0/4229669192 shutdown_connections 2024-04-02T14:30:59.892 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:30:59.889+0000 7f4018e6e700 1 -- 172.21.15.177:0/4229669192 wait complete. 2024-04-02T14:30:59.892 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:30:59.889+0000 7f4018e6e700 1 Processor -- start 2024-04-02T14:30:59.893 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:30:59.889+0000 7f4018e6e700 1 -- start start 2024-04-02T14:30:59.893 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:30:59.890+0000 7f4018e6e700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4014101590 0x7f401406b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:30:59.893 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:30:59.890+0000 7f4018e6e700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4014102790 0x7f401419d7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:30:59.893 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:30:59.890+0000 7f4018e6e700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4014103130 0x7f401419fd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:30:59.893 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:30:59.890+0000 7f4018e6e700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f401410d1e0 con 0x7f4014102790 2024-04-02T14:30:59.893 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:30:59.890+0000 7f4018e6e700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f401410d060 con 0x7f4014103130 2024-04-02T14:30:59.893 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:30:59.890+0000 7f4018e6e700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f401410d360 con 0x7f4014101590 2024-04-02T14:30:59.893 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:30:59.890+0000 7f401259c700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4014101590 0x7f401406b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:30:59.893 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:30:59.890+0000 7f401259c700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4014101590 0x7f401406b7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.177:37392/0 (socket says 172.21.15.177:37392) 2024-04-02T14:30:59.893 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:30:59.890+0000 7f401259c700 1 -- 172.21.15.177:0/618276255 learned_addr learned my addr 172.21.15.177:0/618276255 (peer_addr_for_me v2:172.21.15.177:0/0) 2024-04-02T14:30:59.893 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:30:59.890+0000 7f4012d9d700 1 --2- 172.21.15.177:0/618276255 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4014103130 0x7f401419fd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:30:59.893 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:30:59.890+0000 7f401259c700 1 -- 172.21.15.177:0/618276255 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4014103130 msgr2=0x7f401419fd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:30:59.893 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:30:59.890+0000 7f401259c700 1 --2- 172.21.15.177:0/618276255 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4014103130 0x7f401419fd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:30:59.893 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:30:59.890+0000 7f401259c700 1 -- 172.21.15.177:0/618276255 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4014102790 msgr2=0x7f401419d7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:30:59.893 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:30:59.890+0000 7f401259c700 1 --2- 172.21.15.177:0/618276255 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4014102790 0x7f401419d7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:30:59.894 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:30:59.890+0000 7f401259c700 1 -- 172.21.15.177:0/618276255 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40141a0620 con 0x7f4014101590 2024-04-02T14:30:59.894 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:30:59.890+0000 7f401259c700 1 --2- 172.21.15.177:0/618276255 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4014101590 0x7f401406b7d0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f4004000c00 tx=0x7f4004009f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:30:59.894 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:30:59.890+0000 7f40037fe700 1 -- 172.21.15.177:0/618276255 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4004027070 con 0x7f4014101590 2024-04-02T14:30:59.894 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:30:59.890+0000 7f40037fe700 1 -- 172.21.15.177:0/618276255 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4004022470 con 0x7f4014101590 2024-04-02T14:30:59.894 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:30:59.890+0000 7f4018e6e700 1 -- 172.21.15.177:0/618276255 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40141a08b0 con 0x7f4014101590 2024-04-02T14:30:59.894 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:30:59.890+0000 7f40037fe700 1 -- 172.21.15.177:0/618276255 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f400400f6c0 con 0x7f4014101590 2024-04-02T14:30:59.894 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:30:59.890+0000 7f4018e6e700 1 -- 172.21.15.177:0/618276255 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40141a0de0 con 0x7f4014101590 2024-04-02T14:30:59.895 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:30:59.891+0000 7f4018e6e700 1 -- 172.21.15.177:0/618276255 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f40140692b0 con 0x7f4014101590 2024-04-02T14:30:59.895 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:30:59.892+0000 7f40037fe700 1 -- 172.21.15.177:0/618276255 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f400402a070 con 0x7f4014101590 2024-04-02T14:30:59.895 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:30:59.893+0000 7f40037fe700 1 --2- 172.21.15.177:0/618276255 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f3ffc0a3cc0 0x7f3ffc0a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:30:59.895 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:30:59.893+0000 7f40037fe700 1 -- 172.21.15.177:0/618276255 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 6723+0+0 (secure 0 0 0) 0x7f40040d1180 con 0x7f4014101590 2024-04-02T14:30:59.895 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:30:59.893+0000 7f4011d9b700 1 --2- 172.21.15.177:0/618276255 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f3ffc0a3cc0 0x7f3ffc0a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:30:59.895 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:30:59.894+0000 7f4011d9b700 1 --2- 172.21.15.177:0/618276255 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f3ffc0a3cc0 0x7f3ffc0a6180 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f40080097b0 tx=0x7f4008006d20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:30:59.897 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:30:59.896+0000 7f40037fe700 1 -- 172.21.15.177:0/618276255 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7f4004013070 con 0x7f4014101590 2024-04-02T14:30:59.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:30:59 smithi177 ceph-mon[36431]: osd.10 [v2:172.21.15.92:6824/1594153142,v1:172.21.15.92:6825/1594153142] boot 2024-04-02T14:30:59.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:30:59 smithi177 ceph-mon[36431]: osdmap e75: 11 total, 11 up, 11 in 2024-04-02T14:30:59.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:30:59 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-02T14:30:59.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:30:59 smithi177 ceph-mon[36431]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 894 GiB / 894 GiB avail 2024-04-02T14:30:59.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:30:59 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:30:59.938 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:30:59 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:30:59.938 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:30:59 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-02T14:30:59.938 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:30:59 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-02T14:30:59.938 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:30:59 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-02T14:30:59.938 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:30:59 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-02T14:30:59.939 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:30:59 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:30:59.939 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:30:59 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:30:59.939 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:30:59 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T14:30:59.939 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:30:59 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:30:59.939 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:30:59 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T14:30:59.939 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:30:59 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:30:59.940 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:30:59 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T14:30:59.940 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:30:59 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:30:59.940 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:30:59 smithi177 ceph-mon[36431]: osdmap e76: 11 total, 11 up, 11 in 2024-04-02T14:30:59.941 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:30:59 smithi090 ceph-mon[31674]: osd.10 [v2:172.21.15.92:6824/1594153142,v1:172.21.15.92:6825/1594153142] boot 2024-04-02T14:30:59.941 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:30:59 smithi090 ceph-mon[31674]: osdmap e75: 11 total, 11 up, 11 in 2024-04-02T14:30:59.941 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:30:59 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-02T14:30:59.941 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:30:59 smithi090 ceph-mon[31674]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 894 GiB / 894 GiB avail 2024-04-02T14:30:59.941 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:30:59 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:30:59.941 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:30:59 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:30:59.941 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:30:59 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-02T14:30:59.942 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:30:59 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-02T14:30:59.942 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:30:59 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-02T14:30:59.942 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:30:59 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-02T14:30:59.942 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:30:59 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:30:59.942 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:30:59 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:30:59.942 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:30:59 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T14:30:59.942 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:30:59 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:30:59.942 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:30:59 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T14:30:59.942 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:30:59 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:30:59.942 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:30:59 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T14:30:59.942 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:30:59 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:30:59.942 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:30:59 smithi090 ceph-mon[31674]: osdmap e76: 11 total, 11 up, 11 in 2024-04-02T14:31:00.105 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:31:00.104+0000 7f4018e6e700 1 -- 172.21.15.177:0/618276255 --> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi177:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f4014062a30 con 0x7f3ffc0a3cc0 2024-04-02T14:31:00.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:31:00 smithi177 ceph-mon[36431]: Detected new or changed devices on smithi092 2024-04-02T14:31:00.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:31:00 smithi177 ceph-mon[36431]: Adjusting osd_memory_target on smithi092 to 4270M 2024-04-02T14:31:00.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:31:00 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-02T14:31:00.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:31:00 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-02T14:31:00.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:31:00 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:31:00.938 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:31:00 smithi090 ceph-mon[31674]: Detected new or changed devices on smithi092 2024-04-02T14:31:00.938 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:31:00 smithi090 ceph-mon[31674]: Adjusting osd_memory_target on smithi092 to 4270M 2024-04-02T14:31:00.938 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:31:00 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-02T14:31:00.939 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:31:00 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-02T14:31:00.939 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:31:00 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:31:01.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:31:00 smithi092 ceph-mon[36497]: Detected new or changed devices on smithi092 2024-04-02T14:31:01.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:31:00 smithi092 ceph-mon[36497]: Adjusting osd_memory_target on smithi092 to 4270M 2024-04-02T14:31:01.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:31:00 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-02T14:31:01.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:31:00 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-02T14:31:01.112 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:31:00 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:31:01.727 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:31:01 smithi177 ceph-mon[36431]: from='client.24379 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi177:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T14:31:01.727 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:31:01 smithi177 ceph-mon[36431]: osdmap e77: 11 total, 11 up, 11 in 2024-04-02T14:31:01.728 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:31:01 smithi177 ceph-mon[36431]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 983 GiB / 983 GiB avail 2024-04-02T14:31:01.939 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:31:01 smithi090 ceph-mon[31674]: from='client.24379 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi177:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T14:31:01.939 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:31:01 smithi090 ceph-mon[31674]: osdmap e77: 11 total, 11 up, 11 in 2024-04-02T14:31:01.939 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:31:01 smithi090 ceph-mon[31674]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 983 GiB / 983 GiB avail 2024-04-02T14:31:02.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:31:01 smithi092 ceph-mon[36497]: from='client.24379 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi177:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T14:31:02.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:31:01 smithi092 ceph-mon[36497]: osdmap e77: 11 total, 11 up, 11 in 2024-04-02T14:31:02.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:31:01 smithi092 ceph-mon[36497]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 983 GiB / 983 GiB avail 2024-04-02T14:31:03.110 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:31:02 smithi092 ceph-mon[36497]: osdmap e78: 11 total, 11 up, 11 in 2024-04-02T14:31:03.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:31:02 smithi177 ceph-mon[36431]: osdmap e78: 11 total, 11 up, 11 in 2024-04-02T14:31:03.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:31:02 smithi090 ceph-mon[31674]: osdmap e78: 11 total, 11 up, 11 in 2024-04-02T14:31:04.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:31:03 smithi092 ceph-mon[36497]: osdmap e79: 11 total, 11 up, 11 in 2024-04-02T14:31:04.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:31:03 smithi092 ceph-mon[36497]: pgmap v263: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 349 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-02T14:31:04.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:31:03 smithi092 ceph-mon[36497]: from='client.? 172.21.15.177:0/543428656' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "83cba9fa-b745-4df1-adda-e86837d4b91a"}]: dispatch 2024-04-02T14:31:04.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:31:03 smithi092 ceph-mon[36497]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "83cba9fa-b745-4df1-adda-e86837d4b91a"}]: dispatch 2024-04-02T14:31:04.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:31:03 smithi092 ceph-mon[36497]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "83cba9fa-b745-4df1-adda-e86837d4b91a"}]': finished 2024-04-02T14:31:04.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:31:03 smithi092 ceph-mon[36497]: osdmap e80: 12 total, 11 up, 12 in 2024-04-02T14:31:04.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:31:03 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-02T14:31:04.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:31:03 smithi092 ceph-mon[36497]: from='client.? 172.21.15.177:0/2460854483' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-02T14:31:04.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:31:03 smithi177 ceph-mon[36431]: osdmap e79: 11 total, 11 up, 11 in 2024-04-02T14:31:04.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:31:03 smithi177 ceph-mon[36431]: pgmap v263: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 349 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-02T14:31:04.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:31:03 smithi177 ceph-mon[36431]: from='client.? 172.21.15.177:0/543428656' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "83cba9fa-b745-4df1-adda-e86837d4b91a"}]: dispatch 2024-04-02T14:31:04.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:31:03 smithi177 ceph-mon[36431]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "83cba9fa-b745-4df1-adda-e86837d4b91a"}]: dispatch 2024-04-02T14:31:04.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:31:03 smithi177 ceph-mon[36431]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "83cba9fa-b745-4df1-adda-e86837d4b91a"}]': finished 2024-04-02T14:31:04.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:31:03 smithi177 ceph-mon[36431]: osdmap e80: 12 total, 11 up, 12 in 2024-04-02T14:31:04.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:31:03 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-02T14:31:04.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:31:03 smithi177 ceph-mon[36431]: from='client.? 172.21.15.177:0/2460854483' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-02T14:31:04.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:31:03 smithi090 ceph-mon[31674]: osdmap e79: 11 total, 11 up, 11 in 2024-04-02T14:31:04.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:31:03 smithi090 ceph-mon[31674]: pgmap v263: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 349 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-02T14:31:04.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:31:03 smithi090 ceph-mon[31674]: from='client.? 172.21.15.177:0/543428656' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "83cba9fa-b745-4df1-adda-e86837d4b91a"}]: dispatch 2024-04-02T14:31:04.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:31:03 smithi090 ceph-mon[31674]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "83cba9fa-b745-4df1-adda-e86837d4b91a"}]: dispatch 2024-04-02T14:31:04.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:31:03 smithi090 ceph-mon[31674]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "83cba9fa-b745-4df1-adda-e86837d4b91a"}]': finished 2024-04-02T14:31:04.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:31:03 smithi090 ceph-mon[31674]: osdmap e80: 12 total, 11 up, 12 in 2024-04-02T14:31:04.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:31:03 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-02T14:31:04.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:31:03 smithi090 ceph-mon[31674]: from='client.? 172.21.15.177:0/2460854483' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-02T14:31:06.110 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:31:05 smithi092 ceph-mon[36497]: pgmap v265: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 350 MiB used, 983 GiB / 983 GiB avail; 143 KiB/s, 0 objects/s recovering 2024-04-02T14:31:06.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:31:05 smithi177 ceph-mon[36431]: pgmap v265: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 350 MiB used, 983 GiB / 983 GiB avail; 143 KiB/s, 0 objects/s recovering 2024-04-02T14:31:06.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:31:05 smithi090 ceph-mon[31674]: pgmap v265: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 350 MiB used, 983 GiB / 983 GiB avail; 143 KiB/s, 0 objects/s recovering 2024-04-02T14:31:08.110 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:31:07 smithi092 ceph-mon[36497]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-02T14:31:08.120 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:31:07 smithi177 ceph-mon[36431]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-02T14:31:08.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:31:07 smithi090 ceph-mon[31674]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-02T14:31:09.860 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:31:09 smithi092 ceph-mon[36497]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 983 GiB / 983 GiB avail 2024-04-02T14:31:10.074 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:31:09 smithi177 ceph-mon[36431]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 983 GiB / 983 GiB avail 2024-04-02T14:31:10.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:31:09 smithi090 ceph-mon[31674]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 983 GiB / 983 GiB avail 2024-04-02T14:31:11.104 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:31:10 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-04-02T14:31:11.104 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:31:10 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:31:11.110 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:31:10 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-04-02T14:31:11.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:31:10 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:31:11.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:31:10 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-04-02T14:31:11.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:31:10 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:31:12.048 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:31:11 smithi177 ceph-mon[36431]: Deploying daemon osd.11 on smithi177 2024-04-02T14:31:12.049 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:31:11 smithi177 ceph-mon[36431]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 983 GiB / 983 GiB avail 2024-04-02T14:31:12.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:31:11 smithi092 ceph-mon[36497]: Deploying daemon osd.11 on smithi177 2024-04-02T14:31:12.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:31:11 smithi092 ceph-mon[36497]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 983 GiB / 983 GiB avail 2024-04-02T14:31:12.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:31:11 smithi090 ceph-mon[31674]: Deploying daemon osd.11 on smithi177 2024-04-02T14:31:12.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:31:11 smithi090 ceph-mon[31674]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 983 GiB / 983 GiB avail 2024-04-02T14:31:14.057 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:31:13 smithi177 ceph-mon[36431]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 983 GiB / 983 GiB avail 2024-04-02T14:31:14.110 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:31:13 smithi092 ceph-mon[36497]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 983 GiB / 983 GiB avail 2024-04-02T14:31:14.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:31:13 smithi090 ceph-mon[31674]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 983 GiB / 983 GiB avail 2024-04-02T14:31:15.901 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:31:15 smithi177 ceph-mon[36431]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 983 GiB / 983 GiB avail 2024-04-02T14:31:16.110 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:31:15 smithi092 ceph-mon[36497]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 983 GiB / 983 GiB avail 2024-04-02T14:31:16.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:31:15 smithi090 ceph-mon[31674]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 983 GiB / 983 GiB avail 2024-04-02T14:31:17.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:31:16 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:31:17.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:31:16 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:31:17.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:31:16 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T14:31:17.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:31:16 smithi092 ceph-mon[36497]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 983 GiB / 983 GiB avail 2024-04-02T14:31:17.436 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:31:16 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:31:17.436 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:31:16 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:31:17.436 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:31:16 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T14:31:17.436 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:31:16 smithi177 ceph-mon[36431]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 983 GiB / 983 GiB avail 2024-04-02T14:31:17.438 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:31:16 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:31:17.438 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:31:16 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:31:17.438 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:31:16 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T14:31:17.438 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:31:16 smithi090 ceph-mon[31674]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 983 GiB / 983 GiB avail 2024-04-02T14:31:19.743 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:31:19.742+0000 7f40037fe700 1 -- 172.21.15.177:0/618276255 <== mgr.14152 v2:172.21.15.90:6800/4266054817 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7f4014062a30 con 0x7f3ffc0a3cc0 2024-04-02T14:31:19.743 INFO:teuthology.orchestra.run.smithi177.stdout:Created osd(s) 11 on host 'smithi177' 2024-04-02T14:31:19.745 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:31:19.744+0000 7f4018e6e700 1 -- 172.21.15.177:0/618276255 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f3ffc0a3cc0 msgr2=0x7f3ffc0a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:31:19.746 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:31:19.744+0000 7f4018e6e700 1 --2- 172.21.15.177:0/618276255 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f3ffc0a3cc0 0x7f3ffc0a6180 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f40080097b0 tx=0x7f4008006d20 comp rx=0 tx=0).stop 2024-04-02T14:31:19.746 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:31:19.744+0000 7f4018e6e700 1 -- 172.21.15.177:0/618276255 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4014101590 msgr2=0x7f401406b7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:31:19.746 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:31:19.744+0000 7f4018e6e700 1 --2- 172.21.15.177:0/618276255 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4014101590 0x7f401406b7d0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f4004000c00 tx=0x7f4004009f90 comp rx=0 tx=0).stop 2024-04-02T14:31:19.746 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:31:19.744+0000 7f4018e6e700 1 -- 172.21.15.177:0/618276255 shutdown_connections 2024-04-02T14:31:19.746 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:31:19.744+0000 7f4018e6e700 1 --2- 172.21.15.177:0/618276255 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f3ffc0a3cc0 0x7f3ffc0a6180 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:31:19.746 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:31:19.744+0000 7f4018e6e700 1 --2- 172.21.15.177:0/618276255 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4014103130 0x7f401419fd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:31:19.746 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:31:19.744+0000 7f4018e6e700 1 --2- 172.21.15.177:0/618276255 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4014101590 0x7f401406b7d0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:31:19.746 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:31:19.744+0000 7f4018e6e700 1 --2- 172.21.15.177:0/618276255 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4014102790 0x7f401419d7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:31:19.746 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:31:19.744+0000 7f4018e6e700 1 -- 172.21.15.177:0/618276255 >> 172.21.15.177:0/618276255 conn(0x7f40140fcb50 msgr2=0x7f4014105b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:31:19.746 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:31:19.744+0000 7f4018e6e700 1 -- 172.21.15.177:0/618276255 shutdown_connections 2024-04-02T14:31:19.746 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:31:19.744+0000 7f4018e6e700 1 -- 172.21.15.177:0/618276255 wait complete. 2024-04-02T14:31:19.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:31:19 smithi177 ceph-mon[36431]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 983 GiB / 983 GiB avail 2024-04-02T14:31:19.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:31:19 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:31:19.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:31:19 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:31:19.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:31:19 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:31:19.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:31:19 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T14:31:19.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:31:19 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:31:19.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:31:19 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:31:19.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:31:19 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:31:19.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:31:19 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T14:31:20.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:31:19 smithi092 ceph-mon[36497]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 983 GiB / 983 GiB avail 2024-04-02T14:31:20.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:31:19 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:31:20.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:31:19 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:31:20.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:31:19 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:31:20.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:31:19 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T14:31:20.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:31:19 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:31:20.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:31:19 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:31:20.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:31:19 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:31:20.112 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:31:19 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T14:31:20.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:31:19 smithi090 ceph-mon[31674]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 983 GiB / 983 GiB avail 2024-04-02T14:31:20.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:31:19 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:31:20.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:31:19 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:31:20.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:31:19 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:31:20.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:31:19 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T14:31:20.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:31:19 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:31:20.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:31:19 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:31:20.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:31:19 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:31:20.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:31:19 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T14:31:20.343 DEBUG:teuthology.orchestra.run.smithi177:osd.11> sudo journalctl -f -n 0 -u ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4@osd.11.service 2024-04-02T14:31:20.345 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2024-04-02T14:31:20.346 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:191d963ac9de7322da8aff20b567b3fa458e2ef6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6449448a-f0fc-11ee-b647-cb9ed24678a4 -- ceph osd stat -f json 2024-04-02T14:31:20.805 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T14:31:20.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:31:20 smithi177 ceph-mon[36431]: from='osd.11 [v2:172.21.15.177:6824/1622582126,v1:172.21.15.177:6825/1622582126]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-02T14:31:20.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:31:20 smithi177 ceph-mon[36431]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-02T14:31:21.110 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:31:20 smithi092 ceph-mon[36497]: from='osd.11 [v2:172.21.15.177:6824/1622582126,v1:172.21.15.177:6825/1622582126]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-02T14:31:21.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:31:20 smithi092 ceph-mon[36497]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-02T14:31:21.143 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:31:20 smithi090 ceph-mon[31674]: from='osd.11 [v2:172.21.15.177:6824/1622582126,v1:172.21.15.177:6825/1622582126]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-02T14:31:21.143 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:31:20 smithi090 ceph-mon[31674]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-02T14:31:22.093 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:31:21 smithi177 ceph-mon[36431]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 983 GiB / 983 GiB avail 2024-04-02T14:31:22.093 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:31:21 smithi177 ceph-mon[36431]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-04-02T14:31:22.093 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:31:21 smithi177 ceph-mon[36431]: from='osd.11 [v2:172.21.15.177:6824/1622582126,v1:172.21.15.177:6825/1622582126]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-04-02T14:31:22.093 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:31:21 smithi177 ceph-mon[36431]: osdmap e81: 12 total, 11 up, 12 in 2024-04-02T14:31:22.093 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:31:21 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-02T14:31:22.093 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:31:21 smithi177 ceph-mon[36431]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-04-02T14:31:22.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:31:21 smithi092 ceph-mon[36497]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 983 GiB / 983 GiB avail 2024-04-02T14:31:22.112 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:31:21 smithi092 ceph-mon[36497]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-04-02T14:31:22.112 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:31:21 smithi092 ceph-mon[36497]: from='osd.11 [v2:172.21.15.177:6824/1622582126,v1:172.21.15.177:6825/1622582126]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-04-02T14:31:22.112 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:31:21 smithi092 ceph-mon[36497]: osdmap e81: 12 total, 11 up, 12 in 2024-04-02T14:31:22.112 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:31:21 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-02T14:31:22.113 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:31:21 smithi092 ceph-mon[36497]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-04-02T14:31:22.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:31:21 smithi090 ceph-mon[31674]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 983 GiB / 983 GiB avail 2024-04-02T14:31:22.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:31:21 smithi090 ceph-mon[31674]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-04-02T14:31:22.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:31:21 smithi090 ceph-mon[31674]: from='osd.11 [v2:172.21.15.177:6824/1622582126,v1:172.21.15.177:6825/1622582126]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-04-02T14:31:22.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:31:21 smithi090 ceph-mon[31674]: osdmap e81: 12 total, 11 up, 12 in 2024-04-02T14:31:22.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:31:21 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-02T14:31:22.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:31:21 smithi090 ceph-mon[31674]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-04-02T14:31:22.875 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:31:22 smithi090 ceph-mon[31674]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]': finished 2024-04-02T14:31:22.875 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:31:22 smithi090 ceph-mon[31674]: osdmap e82: 12 total, 11 up, 12 in 2024-04-02T14:31:22.875 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:31:22 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-02T14:31:22.875 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:31:22 smithi090 ceph-mon[31674]: from='osd.11 ' entity='osd.11' 2024-04-02T14:31:22.875 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:31:22 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-02T14:31:23.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:23.039+0000 7f97f6c00700 1 -- 172.21.15.90:0/1368909536 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f97f0101520 msgr2=0x7f97f0101940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:31:23.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:23.039+0000 7f97f6c00700 1 --2- 172.21.15.90:0/1368909536 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f97f0101520 0x7f97f0101940 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f97e0009990 tx=0x7f97e0009ca0 comp rx=0 tx=0).stop 2024-04-02T14:31:23.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:23.039+0000 7f97f6c00700 1 -- 172.21.15.90:0/1368909536 shutdown_connections 2024-04-02T14:31:23.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:23.039+0000 7f97f6c00700 1 --2- 172.21.15.90:0/1368909536 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f97f01030c0 0x7f97f0109970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:31:23.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:23.039+0000 7f97f6c00700 1 --2- 172.21.15.90:0/1368909536 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f97f0101520 0x7f97f0101940 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:31:23.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:23.039+0000 7f97f6c00700 1 --2- 172.21.15.90:0/1368909536 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f97f0102720 0x7f97f0102b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:31:23.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:23.039+0000 7f97f6c00700 1 -- 172.21.15.90:0/1368909536 >> 172.21.15.90:0/1368909536 conn(0x7f97f00fcac0 msgr2=0x7f97f00fef00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:31:23.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:23.039+0000 7f97f6c00700 1 -- 172.21.15.90:0/1368909536 shutdown_connections 2024-04-02T14:31:23.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:23.039+0000 7f97f6c00700 1 -- 172.21.15.90:0/1368909536 wait complete. 2024-04-02T14:31:23.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:23.041+0000 7f97f6c00700 1 Processor -- start 2024-04-02T14:31:23.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:23.041+0000 7f97f6c00700 1 -- start start 2024-04-02T14:31:23.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:23.041+0000 7f97f6c00700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f97f0101520 0x7f97f006adb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:31:23.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:23.041+0000 7f97f6c00700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f97f0102720 0x7f97f019d670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:31:23.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:23.041+0000 7f97f6c00700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f97f01030c0 0x7f97f019fc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:31:23.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:23.041+0000 7f97f6c00700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f97f010d110 con 0x7f97f0101520 2024-04-02T14:31:23.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:23.041+0000 7f97f6c00700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f97f010cf90 con 0x7f97f0102720 2024-04-02T14:31:23.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:23.041+0000 7f97f6c00700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f97f010d290 con 0x7f97f01030c0 2024-04-02T14:31:23.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:23.042+0000 7f97efe00700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f97f0101520 0x7f97f006adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:31:23.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:23.042+0000 7f97f4c00700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f97f01030c0 0x7f97f019fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:31:23.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:23.042+0000 7f97ef400700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f97f0102720 0x7f97f019d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:31:23.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:23.042+0000 7f97f4c00700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f97f01030c0 0x7f97f019fc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:38736/0 (socket says 172.21.15.90:38736) 2024-04-02T14:31:23.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:23.042+0000 7f97f4c00700 1 -- 172.21.15.90:0/3208174781 learned_addr learned my addr 172.21.15.90:0/3208174781 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:31:23.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:23.042+0000 7f97f4c00700 1 -- 172.21.15.90:0/3208174781 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f97f0102720 msgr2=0x7f97f019d670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:31:23.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:23.042+0000 7f97f4c00700 1 --2- 172.21.15.90:0/3208174781 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f97f0102720 0x7f97f019d670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:31:23.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:23.042+0000 7f97f4c00700 1 -- 172.21.15.90:0/3208174781 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f97f0101520 msgr2=0x7f97f006adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:31:23.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:23.042+0000 7f97f4c00700 1 --2- 172.21.15.90:0/3208174781 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f97f0101520 0x7f97f006adb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:31:23.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:23.042+0000 7f97f4c00700 1 -- 172.21.15.90:0/3208174781 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97f01a0510 con 0x7f97f01030c0 2024-04-02T14:31:23.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:23.042+0000 7f97f4c00700 1 --2- 172.21.15.90:0/3208174781 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f97f01030c0 0x7f97f019fc40 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f97e800ef30 tx=0x7f97e800c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:31:23.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:23.042+0000 7f97ecc00700 1 -- 172.21.15.90:0/3208174781 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97e80092e0 con 0x7f97f01030c0 2024-04-02T14:31:23.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:23.043+0000 7f97f6c00700 1 -- 172.21.15.90:0/3208174781 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97f01a07a0 con 0x7f97f01030c0 2024-04-02T14:31:23.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:23.043+0000 7f97ecc00700 1 -- 172.21.15.90:0/3208174781 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f97e8004bb0 con 0x7f97f01030c0 2024-04-02T14:31:23.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:23.043+0000 7f97ecc00700 1 -- 172.21.15.90:0/3208174781 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97e8004280 con 0x7f97f01030c0 2024-04-02T14:31:23.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:23.043+0000 7f97f6c00700 1 -- 172.21.15.90:0/3208174781 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97f01a0ce0 con 0x7f97f01030c0 2024-04-02T14:31:23.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:23.044+0000 7f97f6c00700 1 -- 172.21.15.90:0/3208174781 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f97f00692b0 con 0x7f97f01030c0 2024-04-02T14:31:23.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:23.046+0000 7f97ecc00700 1 -- 172.21.15.90:0/3208174781 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f97e801b030 con 0x7f97f01030c0 2024-04-02T14:31:23.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:23.047+0000 7f97ecc00700 1 --2- 172.21.15.90:0/3208174781 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f97d40a3c40 0x7f97d40a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:31:23.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:23.047+0000 7f97ecc00700 1 -- 172.21.15.90:0/3208174781 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 6847+0+0 (secure 0 0 0) 0x7f97e80cf950 con 0x7f97f01030c0 2024-04-02T14:31:23.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:23.047+0000 7f97efe00700 1 --2- 172.21.15.90:0/3208174781 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f97d40a3c40 0x7f97d40a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:31:23.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:23.048+0000 7f97efe00700 1 --2- 172.21.15.90:0/3208174781 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f97d40a3c40 0x7f97d40a6100 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f97e000b2e0 tx=0x7f97e000b8f0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:31:23.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:23.055+0000 7f97ecc00700 1 -- 172.21.15.90:0/3208174781 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7f97e8022020 con 0x7f97f01030c0 2024-04-02T14:31:23.110 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:31:22 smithi092 ceph-mon[36497]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]': finished 2024-04-02T14:31:23.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:31:22 smithi092 ceph-mon[36497]: osdmap e82: 12 total, 11 up, 12 in 2024-04-02T14:31:23.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:31:22 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-02T14:31:23.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:31:22 smithi092 ceph-mon[36497]: from='osd.11 ' entity='osd.11' 2024-04-02T14:31:23.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:31:22 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-02T14:31:23.146 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:31:22 smithi177 ceph-mon[36431]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]': finished 2024-04-02T14:31:23.146 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:31:22 smithi177 ceph-mon[36431]: osdmap e82: 12 total, 11 up, 12 in 2024-04-02T14:31:23.146 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:31:22 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-02T14:31:23.146 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:31:22 smithi177 ceph-mon[36431]: from='osd.11 ' entity='osd.11' 2024-04-02T14:31:23.146 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:31:22 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-02T14:31:23.147 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:31:22 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:31:22.688+0000 7fba07b3b700 -1 osd.11 0 waiting for initial osdmap 2024-04-02T14:31:23.147 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:31:22 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:31:22.807+0000 7fb9ff926700 -1 osd.11 82 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-02T14:31:23.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:23.284+0000 7f97f6c00700 1 -- 172.21.15.90:0/3208174781 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7f97f0063200 con 0x7f97f01030c0 2024-04-02T14:31:23.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:23.284+0000 7f97ecc00700 1 -- 172.21.15.90:0/3208174781 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v82) ==== 74+0+133 (secure 0 0 0) 0x7f97e8010040 con 0x7f97f01030c0 2024-04-02T14:31:23.287 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-02T14:31:23.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:23.287+0000 7f97f6c00700 1 -- 172.21.15.90:0/3208174781 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f97d40a3c40 msgr2=0x7f97d40a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:31:23.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:23.287+0000 7f97f6c00700 1 --2- 172.21.15.90:0/3208174781 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f97d40a3c40 0x7f97d40a6100 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f97e000b2e0 tx=0x7f97e000b8f0 comp rx=0 tx=0).stop 2024-04-02T14:31:23.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:23.287+0000 7f97f6c00700 1 -- 172.21.15.90:0/3208174781 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f97f01030c0 msgr2=0x7f97f019fc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:31:23.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:23.287+0000 7f97f6c00700 1 --2- 172.21.15.90:0/3208174781 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f97f01030c0 0x7f97f019fc40 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f97e800ef30 tx=0x7f97e800c3a0 comp rx=0 tx=0).stop 2024-04-02T14:31:23.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:23.287+0000 7f97f6c00700 1 -- 172.21.15.90:0/3208174781 shutdown_connections 2024-04-02T14:31:23.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:23.287+0000 7f97f6c00700 1 --2- 172.21.15.90:0/3208174781 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f97d40a3c40 0x7f97d40a6100 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:31:23.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:23.287+0000 7f97f6c00700 1 --2- 172.21.15.90:0/3208174781 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f97f0101520 0x7f97f006adb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:31:23.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:23.287+0000 7f97f6c00700 1 --2- 172.21.15.90:0/3208174781 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f97f01030c0 0x7f97f019fc40 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:31:23.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:23.287+0000 7f97f6c00700 1 --2- 172.21.15.90:0/3208174781 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f97f0102720 0x7f97f019d670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:31:23.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:23.287+0000 7f97f6c00700 1 -- 172.21.15.90:0/3208174781 >> 172.21.15.90:0/3208174781 conn(0x7f97f00fcac0 msgr2=0x7f97f0105b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:31:23.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:23.288+0000 7f97f6c00700 1 -- 172.21.15.90:0/3208174781 shutdown_connections 2024-04-02T14:31:23.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:23.288+0000 7f97f6c00700 1 -- 172.21.15.90:0/3208174781 wait complete. 2024-04-02T14:31:23.940 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:31:23 smithi090 ceph-mon[31674]: purged_snaps scrub starts 2024-04-02T14:31:23.940 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:31:23 smithi090 ceph-mon[31674]: purged_snaps scrub ok 2024-04-02T14:31:23.940 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:31:23 smithi090 ceph-mon[31674]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 983 GiB / 983 GiB avail 2024-04-02T14:31:23.940 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:31:23 smithi090 ceph-mon[31674]: from='client.? 172.21.15.90:0/3208174781' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-02T14:31:23.940 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:31:23 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:31:23.940 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:31:23 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:31:23.940 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:31:23 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-02T14:31:23.940 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:31:23 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-02T14:31:23.941 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:31:23 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-02T14:31:23.941 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:31:23 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-02T14:31:23.941 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:31:23 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:31:23.941 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:31:23 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:31:23.941 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:31:23 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T14:31:23.941 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:31:23 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:31:23.941 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:31:23 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-02T14:31:24.095 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":82,"num_osds":12,"num_up_osds":11,"osd_up_since":1712068258,"num_in_osds":12,"osd_in_since":1712068262,"num_remapped_pgs":0} 2024-04-02T14:31:24.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:31:23 smithi092 ceph-mon[36497]: purged_snaps scrub starts 2024-04-02T14:31:24.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:31:23 smithi092 ceph-mon[36497]: purged_snaps scrub ok 2024-04-02T14:31:24.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:31:23 smithi092 ceph-mon[36497]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 983 GiB / 983 GiB avail 2024-04-02T14:31:24.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:31:23 smithi092 ceph-mon[36497]: from='client.? 172.21.15.90:0/3208174781' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-02T14:31:24.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:31:23 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:31:24.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:31:23 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:31:24.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:31:23 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-02T14:31:24.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:31:23 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-02T14:31:24.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:31:23 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-02T14:31:24.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:31:23 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-02T14:31:24.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:31:23 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:31:24.112 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:31:23 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:31:24.112 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:31:23 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T14:31:24.112 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:31:23 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:31:24.112 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:31:23 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-02T14:31:24.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:31:23 smithi177 ceph-mon[36431]: purged_snaps scrub starts 2024-04-02T14:31:24.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:31:23 smithi177 ceph-mon[36431]: purged_snaps scrub ok 2024-04-02T14:31:24.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:31:23 smithi177 ceph-mon[36431]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 983 GiB / 983 GiB avail 2024-04-02T14:31:24.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:31:23 smithi177 ceph-mon[36431]: from='client.? 172.21.15.90:0/3208174781' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-02T14:31:24.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:31:23 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:31:24.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:31:23 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:31:24.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:31:23 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-02T14:31:24.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:31:23 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-02T14:31:24.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:31:23 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-02T14:31:24.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:31:23 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-02T14:31:24.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:31:23 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:31:24.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:31:23 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:31:24.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:31:23 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T14:31:24.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:31:23 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:31:24.188 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:31:23 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-02T14:31:25.096 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:191d963ac9de7322da8aff20b567b3fa458e2ef6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6449448a-f0fc-11ee-b647-cb9ed24678a4 -- ceph osd stat -f json 2024-04-02T14:31:25.110 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:31:24 smithi092 ceph-mon[36497]: Detected new or changed devices on smithi177 2024-04-02T14:31:25.110 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:31:24 smithi092 ceph-mon[36497]: Adjusting osd_memory_target on smithi177 to 4270M 2024-04-02T14:31:25.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:31:24 smithi092 ceph-mon[36497]: osd.11 [v2:172.21.15.177:6824/1622582126,v1:172.21.15.177:6825/1622582126] boot 2024-04-02T14:31:25.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:31:24 smithi092 ceph-mon[36497]: osdmap e83: 12 total, 12 up, 12 in 2024-04-02T14:31:25.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:31:24 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-02T14:31:25.184 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:31:24 smithi090 ceph-mon[31674]: Detected new or changed devices on smithi177 2024-04-02T14:31:25.184 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:31:24 smithi090 ceph-mon[31674]: Adjusting osd_memory_target on smithi177 to 4270M 2024-04-02T14:31:25.184 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:31:24 smithi090 ceph-mon[31674]: osd.11 [v2:172.21.15.177:6824/1622582126,v1:172.21.15.177:6825/1622582126] boot 2024-04-02T14:31:25.184 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:31:24 smithi090 ceph-mon[31674]: osdmap e83: 12 total, 12 up, 12 in 2024-04-02T14:31:25.185 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:31:24 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-02T14:31:25.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:31:24 smithi177 ceph-mon[36431]: Detected new or changed devices on smithi177 2024-04-02T14:31:25.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:31:24 smithi177 ceph-mon[36431]: Adjusting osd_memory_target on smithi177 to 4270M 2024-04-02T14:31:25.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:31:24 smithi177 ceph-mon[36431]: osd.11 [v2:172.21.15.177:6824/1622582126,v1:172.21.15.177:6825/1622582126] boot 2024-04-02T14:31:25.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:31:24 smithi177 ceph-mon[36431]: osdmap e83: 12 total, 12 up, 12 in 2024-04-02T14:31:25.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:31:24 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-02T14:31:25.544 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T14:31:26.020 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:31:25 smithi090 ceph-mon[31674]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 983 GiB / 983 GiB avail 2024-04-02T14:31:26.021 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:31:25 smithi090 ceph-mon[31674]: osdmap e84: 12 total, 12 up, 12 in 2024-04-02T14:31:26.110 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:31:25 smithi092 ceph-mon[36497]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 983 GiB / 983 GiB avail 2024-04-02T14:31:26.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:31:25 smithi092 ceph-mon[36497]: osdmap e84: 12 total, 12 up, 12 in 2024-04-02T14:31:26.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:31:25 smithi177 ceph-mon[36431]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 983 GiB / 983 GiB avail 2024-04-02T14:31:26.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:31:25 smithi177 ceph-mon[36431]: osdmap e84: 12 total, 12 up, 12 in 2024-04-02T14:31:27.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:27.750+0000 7f8e37200700 1 -- 172.21.15.90:0/2362728743 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8e3810eb40 msgr2=0x7f8e38110f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:31:27.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:27.750+0000 7f8e37200700 1 --2- 172.21.15.90:0/2362728743 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8e3810eb40 0x7f8e38110f30 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f8e3000b0a0 tx=0x7f8e3000b3b0 comp rx=0 tx=0).stop 2024-04-02T14:31:27.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:27.750+0000 7f8e37200700 1 -- 172.21.15.90:0/2362728743 shutdown_connections 2024-04-02T14:31:27.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:27.750+0000 7f8e37200700 1 --2- 172.21.15.90:0/2362728743 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8e3810eb40 0x7f8e38110f30 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:31:27.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:27.750+0000 7f8e37200700 1 --2- 172.21.15.90:0/2362728743 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8e38076620 0x7f8e38074a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:31:27.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:27.750+0000 7f8e37200700 1 --2- 172.21.15.90:0/2362728743 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8e38075050 0x7f8e380754d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:31:27.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:27.750+0000 7f8e37200700 1 -- 172.21.15.90:0/2362728743 >> 172.21.15.90:0/2362728743 conn(0x7f8e380fdb00 msgr2=0x7f8e380fff40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:31:27.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:27.750+0000 7f8e37200700 1 -- 172.21.15.90:0/2362728743 shutdown_connections 2024-04-02T14:31:27.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:27.750+0000 7f8e37200700 1 -- 172.21.15.90:0/2362728743 wait complete. 2024-04-02T14:31:27.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:27.752+0000 7f8e37200700 1 Processor -- start 2024-04-02T14:31:27.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:27.752+0000 7f8e37200700 1 -- start start 2024-04-02T14:31:27.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:27.752+0000 7f8e37200700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8e38075050 0x7f8e3819ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:31:27.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:27.752+0000 7f8e37200700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8e38076620 0x7f8e3819f230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:31:27.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:27.752+0000 7f8e37200700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8e3810eb40 0x7f8e381a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:31:27.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:27.752+0000 7f8e37200700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f8e38114740 con 0x7f8e38076620 2024-04-02T14:31:27.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:27.752+0000 7f8e37200700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f8e381145c0 con 0x7f8e38075050 2024-04-02T14:31:27.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:27.752+0000 7f8e37200700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f8e381148c0 con 0x7f8e3810eb40 2024-04-02T14:31:27.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:27.753+0000 7f8e35400700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8e38076620 0x7f8e3819f230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:31:27.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:27.753+0000 7f8e35400700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8e38076620 0x7f8e3819f230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:39298/0 (socket says 172.21.15.90:39298) 2024-04-02T14:31:27.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:27.753+0000 7f8e35400700 1 -- 172.21.15.90:0/3345624132 learned_addr learned my addr 172.21.15.90:0/3345624132 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:31:27.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:27.753+0000 7f8e35e00700 1 --2- 172.21.15.90:0/3345624132 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8e38075050 0x7f8e3819ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:31:27.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:27.753+0000 7f8e36800700 1 --2- 172.21.15.90:0/3345624132 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8e3810eb40 0x7f8e381a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:31:27.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:27.753+0000 7f8e35400700 1 -- 172.21.15.90:0/3345624132 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8e3810eb40 msgr2=0x7f8e381a6270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:31:27.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:27.753+0000 7f8e35400700 1 --2- 172.21.15.90:0/3345624132 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8e3810eb40 0x7f8e381a6270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:31:27.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:27.753+0000 7f8e35400700 1 -- 172.21.15.90:0/3345624132 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8e38075050 msgr2=0x7f8e3819ecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:31:27.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:27.753+0000 7f8e35400700 1 --2- 172.21.15.90:0/3345624132 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8e38075050 0x7f8e3819ecf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:31:27.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:27.753+0000 7f8e35400700 1 -- 172.21.15.90:0/3345624132 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e381a6940 con 0x7f8e38076620 2024-04-02T14:31:27.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:27.753+0000 7f8e36800700 1 --2- 172.21.15.90:0/3345624132 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8e3810eb40 0x7f8e381a6270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:31:27.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:27.753+0000 7f8e35e00700 1 --2- 172.21.15.90:0/3345624132 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8e38075050 0x7f8e3819ecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:31:27.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:27.753+0000 7f8e35400700 1 --2- 172.21.15.90:0/3345624132 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8e38076620 0x7f8e3819f230 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f8e2400e970 tx=0x7f8e2400ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:31:27.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:27.754+0000 7f8e2ea00700 1 -- 172.21.15.90:0/3345624132 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e2400cbd0 con 0x7f8e38076620 2024-04-02T14:31:27.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:27.754+0000 7f8e37200700 1 -- 172.21.15.90:0/3345624132 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8e381a6c30 con 0x7f8e38076620 2024-04-02T14:31:27.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:27.754+0000 7f8e2ea00700 1 -- 172.21.15.90:0/3345624132 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8e24004500 con 0x7f8e38076620 2024-04-02T14:31:27.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:27.754+0000 7f8e2ea00700 1 -- 172.21.15.90:0/3345624132 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e24003e30 con 0x7f8e38076620 2024-04-02T14:31:27.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:27.754+0000 7f8e37200700 1 -- 172.21.15.90:0/3345624132 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8e381a71c0 con 0x7f8e38076620 2024-04-02T14:31:27.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:27.755+0000 7f8e37200700 1 -- 172.21.15.90:0/3345624132 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8e380692b0 con 0x7f8e38076620 2024-04-02T14:31:27.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:27.757+0000 7f8e2ea00700 1 -- 172.21.15.90:0/3345624132 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f8e24004000 con 0x7f8e38076620 2024-04-02T14:31:27.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:27.758+0000 7f8e2ea00700 1 --2- 172.21.15.90:0/3345624132 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f8e1c0a3c40 0x7f8e1c0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:31:27.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:27.758+0000 7f8e2ea00700 1 -- 172.21.15.90:0/3345624132 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 7127+0+0 (secure 0 0 0) 0x7f8e240d02e0 con 0x7f8e38076620 2024-04-02T14:31:27.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:27.758+0000 7f8e35e00700 1 --2- 172.21.15.90:0/3345624132 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f8e1c0a3c40 0x7f8e1c0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:31:27.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:27.759+0000 7f8e35e00700 1 --2- 172.21.15.90:0/3345624132 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f8e1c0a3c40 0x7f8e1c0a6100 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f8e381a01d0 tx=0x7f8e28009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:31:27.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:27.763+0000 7f8e2ea00700 1 -- 172.21.15.90:0/3345624132 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7f8e2409d0c0 con 0x7f8e38076620 2024-04-02T14:31:27.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:27.993+0000 7f8e37200700 1 -- 172.21.15.90:0/3345624132 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7f8e38063200 con 0x7f8e38076620 2024-04-02T14:31:27.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:27.994+0000 7f8e2ea00700 1 -- 172.21.15.90:0/3345624132 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v84) ==== 74+0+133 (secure 0 0 0) 0x7f8e2409c9c0 con 0x7f8e38076620 2024-04-02T14:31:27.996 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-02T14:31:27.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:27.996+0000 7f8e37200700 1 -- 172.21.15.90:0/3345624132 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f8e1c0a3c40 msgr2=0x7f8e1c0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:31:27.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:27.997+0000 7f8e37200700 1 --2- 172.21.15.90:0/3345624132 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f8e1c0a3c40 0x7f8e1c0a6100 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f8e381a01d0 tx=0x7f8e28009260 comp rx=0 tx=0).stop 2024-04-02T14:31:27.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:27.997+0000 7f8e37200700 1 -- 172.21.15.90:0/3345624132 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8e38076620 msgr2=0x7f8e3819f230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:31:27.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:27.997+0000 7f8e37200700 1 --2- 172.21.15.90:0/3345624132 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8e38076620 0x7f8e3819f230 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f8e2400e970 tx=0x7f8e2400ec80 comp rx=0 tx=0).stop 2024-04-02T14:31:27.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:27.997+0000 7f8e37200700 1 -- 172.21.15.90:0/3345624132 shutdown_connections 2024-04-02T14:31:28.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:27.997+0000 7f8e37200700 1 --2- 172.21.15.90:0/3345624132 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f8e1c0a3c40 0x7f8e1c0a6100 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:31:28.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:27.997+0000 7f8e37200700 1 --2- 172.21.15.90:0/3345624132 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8e3810eb40 0x7f8e381a6270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:31:28.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:27.997+0000 7f8e37200700 1 --2- 172.21.15.90:0/3345624132 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8e38075050 0x7f8e3819ecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:31:28.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:27.997+0000 7f8e37200700 1 --2- 172.21.15.90:0/3345624132 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8e38076620 0x7f8e3819f230 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:31:28.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:27.997+0000 7f8e37200700 1 -- 172.21.15.90:0/3345624132 >> 172.21.15.90:0/3345624132 conn(0x7f8e380fdb00 msgr2=0x7f8e38101ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:31:28.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:27.997+0000 7f8e37200700 1 -- 172.21.15.90:0/3345624132 shutdown_connections 2024-04-02T14:31:28.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:27.997+0000 7f8e37200700 1 -- 172.21.15.90:0/3345624132 wait complete. 2024-04-02T14:31:28.110 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:31:27 smithi092 ceph-mon[36497]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-02T14:31:28.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:31:27 smithi177 ceph-mon[36431]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-02T14:31:28.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:31:27 smithi090 ceph-mon[31674]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-02T14:31:28.816 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":84,"num_osds":12,"num_up_osds":12,"osd_up_since":1712068283,"num_in_osds":12,"osd_in_since":1712068262,"num_remapped_pgs":0} 2024-04-02T14:31:28.816 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:191d963ac9de7322da8aff20b567b3fa458e2ef6 shell --fsid 6449448a-f0fc-11ee-b647-cb9ed24678a4 -- ceph osd dump --format=json 2024-04-02T14:31:29.110 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:31:28 smithi092 ceph-mon[36497]: from='client.? 172.21.15.90:0/3345624132' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-02T14:31:29.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:31:28 smithi177 ceph-mon[36431]: from='client.? 172.21.15.90:0/3345624132' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-02T14:31:29.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:31:28 smithi090 ceph-mon[31674]: from='client.? 172.21.15.90:0/3345624132' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-02T14:31:29.259 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T14:31:30.110 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:31:29 smithi092 ceph-mon[36497]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-02T14:31:30.132 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:31:29 smithi090 ceph-mon[31674]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-02T14:31:30.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:31:29 smithi177 ceph-mon[36431]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-02T14:31:31.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:31.336+0000 7f879a800700 1 -- 172.21.15.90:0/1771438901 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f879406ac60 msgr2=0x7f879410f450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:31:31.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:31.336+0000 7f879a800700 1 --2- 172.21.15.90:0/1771438901 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f879406ac60 0x7f879410f450 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f878c00b0a0 tx=0x7f878c00b3b0 comp rx=0 tx=0).stop 2024-04-02T14:31:31.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:31.336+0000 7f879a800700 1 -- 172.21.15.90:0/1771438901 shutdown_connections 2024-04-02T14:31:31.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:31.336+0000 7f879a800700 1 --2- 172.21.15.90:0/1771438901 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f879406ac60 0x7f879410f450 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:31:31.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:31.336+0000 7f879a800700 1 --2- 172.21.15.90:0/1771438901 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8794108650 0x7f8794108a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:31:31.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:31.336+0000 7f879a800700 1 --2- 172.21.15.90:0/1771438901 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f879406a2a0 0x7f879406a720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:31:31.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:31.336+0000 7f879a800700 1 -- 172.21.15.90:0/1771438901 >> 172.21.15.90:0/1771438901 conn(0x7f8794078420 msgr2=0x7f8794078830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:31:31.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:31.336+0000 7f879a800700 1 -- 172.21.15.90:0/1771438901 shutdown_connections 2024-04-02T14:31:31.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:31.336+0000 7f879a800700 1 -- 172.21.15.90:0/1771438901 wait complete. 2024-04-02T14:31:31.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:31.337+0000 7f879a800700 1 Processor -- start 2024-04-02T14:31:31.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:31.338+0000 7f879a800700 1 -- start start 2024-04-02T14:31:31.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:31.338+0000 7f879a800700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f879406a2a0 0x7f87941015c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:31:31.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:31.338+0000 7f879a800700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f879406ac60 0x7f8794101b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:31:31.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:31.338+0000 7f879a800700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8794108650 0x7f8794102040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:31:31.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:31.338+0000 7f879a800700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f8794112940 con 0x7f879406ac60 2024-04-02T14:31:31.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:31.338+0000 7f879a800700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f87941127c0 con 0x7f8794108650 2024-04-02T14:31:31.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:31.338+0000 7f879a800700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f8794112ac0 con 0x7f879406a2a0 2024-04-02T14:31:31.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:31.338+0000 7f8793400700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f879406a2a0 0x7f87941015c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:31:31.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:31.339+0000 7f8793400700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f879406a2a0 0x7f87941015c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:60728/0 (socket says 172.21.15.90:60728) 2024-04-02T14:31:31.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:31.339+0000 7f8793400700 1 -- 172.21.15.90:0/584340183 learned_addr learned my addr 172.21.15.90:0/584340183 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:31:31.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:31.339+0000 7f8793e00700 1 --2- 172.21.15.90:0/584340183 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8794108650 0x7f8794102040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:31:31.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:31.339+0000 7f878be00700 1 --2- 172.21.15.90:0/584340183 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f879406ac60 0x7f8794101b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:31:31.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:31.339+0000 7f8793e00700 1 -- 172.21.15.90:0/584340183 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f879406a2a0 msgr2=0x7f87941015c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:31:31.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:31.339+0000 7f8793e00700 1 --2- 172.21.15.90:0/584340183 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f879406a2a0 0x7f87941015c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:31:31.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:31.339+0000 7f8793e00700 1 -- 172.21.15.90:0/584340183 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f879406ac60 msgr2=0x7f8794101b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:31:31.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:31.339+0000 7f8793e00700 1 --2- 172.21.15.90:0/584340183 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f879406ac60 0x7f8794101b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:31:31.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:31.339+0000 7f8793e00700 1 -- 172.21.15.90:0/584340183 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87941a7280 con 0x7f8794108650 2024-04-02T14:31:31.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:31.339+0000 7f8793400700 1 --2- 172.21.15.90:0/584340183 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f879406a2a0 0x7f87941015c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:31:31.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:31.339+0000 7f8793e00700 1 --2- 172.21.15.90:0/584340183 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8794108650 0x7f8794102040 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f878c009bc0 tx=0x7f878c004c80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:31:31.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:31.340+0000 7f8790c00700 1 -- 172.21.15.90:0/584340183 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f878c021070 con 0x7f8794108650 2024-04-02T14:31:31.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:31.340+0000 7f8790c00700 1 -- 172.21.15.90:0/584340183 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f878c007b80 con 0x7f8794108650 2024-04-02T14:31:31.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:31.340+0000 7f879a800700 1 -- 172.21.15.90:0/584340183 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87941a7510 con 0x7f8794108650 2024-04-02T14:31:31.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:31.340+0000 7f8790c00700 1 -- 172.21.15.90:0/584340183 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f878c005040 con 0x7f8794108650 2024-04-02T14:31:31.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:31.340+0000 7f879a800700 1 -- 172.21.15.90:0/584340183 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87941a79c0 con 0x7f8794108650 2024-04-02T14:31:31.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:31.341+0000 7f879a800700 1 -- 172.21.15.90:0/584340183 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f879410aa30 con 0x7f8794108650 2024-04-02T14:31:31.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:31.343+0000 7f8790c00700 1 -- 172.21.15.90:0/584340183 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f878c024070 con 0x7f8794108650 2024-04-02T14:31:31.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:31.344+0000 7f8790c00700 1 --2- 172.21.15.90:0/584340183 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f87780a3c40 0x7f87780a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:31:31.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:31.344+0000 7f8790c00700 1 -- 172.21.15.90:0/584340183 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 7127+0+0 (secure 0 0 0) 0x7f878c00f020 con 0x7f8794108650 2024-04-02T14:31:31.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:31.345+0000 7f8793400700 1 --2- 172.21.15.90:0/584340183 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f87780a3c40 0x7f87780a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:31:31.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:31.345+0000 7f8793400700 1 --2- 172.21.15.90:0/584340183 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f87780a3c40 0x7f87780a6100 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f8784011550 tx=0x7f8784009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:31:31.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:31.346+0000 7f8790c00700 1 -- 172.21.15.90:0/584340183 <== mon.2 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7f878c02b020 con 0x7f8794108650 2024-04-02T14:31:31.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:31.572+0000 7f879a800700 1 -- 172.21.15.90:0/584340183 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f8794102af0 con 0x7f8794108650 2024-04-02T14:31:31.576 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:31.574+0000 7f8790c00700 1 -- 172.21.15.90:0/584340183 <== mon.2 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v84) ==== 74+0+18441 (secure 0 0 0) 0x7f878c09e500 con 0x7f8794108650 2024-04-02T14:31:31.577 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-02T14:31:31.577 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":84,"fsid":"6449448a-f0fc-11ee-b647-cb9ed24678a4","created":"2024-04-02T14:23:16.530201+0000","modified":"2024-04-02T14:31:24.832762+0000","last_up_change":"2024-04-02T14:31:23.819732+0000","last_in_change":"2024-04-02T14:31:02.824284+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T14:27:16.758622+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a1a6a856-ab44-44bd-bc5e-45c718a70cd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":3635087609},{"type":"v1","addr":"172.21.15.90:6803","nonce":3635087609}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":3635087609},{"type":"v1","addr":"172.21.15.90:6805","nonce":3635087609}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":3635087609},{"type":"v1","addr":"172.21.15.90:6809","nonce":3635087609}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":3635087609},{"type":"v1","addr":"172.21.15.90:6807","nonce":3635087609}]},"public_addr":"172.21.15.90:6803/3635087609","cluster_addr":"172.21.15.90:6805/3635087609","heartbeat_back_addr":"172.21.15.90:6809/3635087609","heartbeat_front_addr":"172.21.15.90:6807/3635087609","state":["exists","up"]},{"osd":1,"uuid":"124139cd-5793-463e-9607-9d9c10164183","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":1994536059},{"type":"v1","addr":"172.21.15.92:6801","nonce":1994536059}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":1994536059},{"type":"v1","addr":"172.21.15.92:6803","nonce":1994536059}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":1994536059},{"type":"v1","addr":"172.21.15.92:6807","nonce":1994536059}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":1994536059},{"type":"v1","addr":"172.21.15.92:6805","nonce":1994536059}]},"public_addr":"172.21.15.92:6801/1994536059","cluster_addr":"172.21.15.92:6803/1994536059","heartbeat_back_addr":"172.21.15.92:6807/1994536059","heartbeat_front_addr":"172.21.15.92:6805/1994536059","state":["exists","up"]},{"osd":2,"uuid":"a61fbef2-9dca-4875-a814-020fff0db0e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":1358472800},{"type":"v1","addr":"172.21.15.177:6801","nonce":1358472800}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":1358472800},{"type":"v1","addr":"172.21.15.177:6803","nonce":1358472800}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":1358472800},{"type":"v1","addr":"172.21.15.177:6807","nonce":1358472800}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":1358472800},{"type":"v1","addr":"172.21.15.177:6805","nonce":1358472800}]},"public_addr":"172.21.15.177:6801/1358472800","cluster_addr":"172.21.15.177:6803/1358472800","heartbeat_back_addr":"172.21.15.177:6807/1358472800","heartbeat_front_addr":"172.21.15.177:6805/1358472800","state":["exists","up"]},{"osd":3,"uuid":"b175a2f4-7859-4eb0-bc1f-fba8c2b92a2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":3114811216},{"type":"v1","addr":"172.21.15.90:6811","nonce":3114811216}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":3114811216},{"type":"v1","addr":"172.21.15.90:6813","nonce":3114811216}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":3114811216},{"type":"v1","addr":"172.21.15.90:6817","nonce":3114811216}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":3114811216},{"type":"v1","addr":"172.21.15.90:6815","nonce":3114811216}]},"public_addr":"172.21.15.90:6811/3114811216","cluster_addr":"172.21.15.90:6813/3114811216","heartbeat_back_addr":"172.21.15.90:6817/3114811216","heartbeat_front_addr":"172.21.15.90:6815/3114811216","state":["exists","up"]},{"osd":4,"uuid":"38cc68ea-c109-4d8d-b77d-1c6f9b56c875","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":2987208879},{"type":"v1","addr":"172.21.15.92:6809","nonce":2987208879}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6810","nonce":2987208879},{"type":"v1","addr":"172.21.15.92:6811","nonce":2987208879}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6814","nonce":2987208879},{"type":"v1","addr":"172.21.15.92:6815","nonce":2987208879}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":2987208879},{"type":"v1","addr":"172.21.15.92:6813","nonce":2987208879}]},"public_addr":"172.21.15.92:6809/2987208879","cluster_addr":"172.21.15.92:6811/2987208879","heartbeat_back_addr":"172.21.15.92:6815/2987208879","heartbeat_front_addr":"172.21.15.92:6813/2987208879","state":["exists","up"]},{"osd":5,"uuid":"2347232e-ac91-466f-8ac9-496f397679ba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":3486820237},{"type":"v1","addr":"172.21.15.177:6809","nonce":3486820237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":3486820237},{"type":"v1","addr":"172.21.15.177:6811","nonce":3486820237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":3486820237},{"type":"v1","addr":"172.21.15.177:6815","nonce":3486820237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":3486820237},{"type":"v1","addr":"172.21.15.177:6813","nonce":3486820237}]},"public_addr":"172.21.15.177:6809/3486820237","cluster_addr":"172.21.15.177:6811/3486820237","heartbeat_back_addr":"172.21.15.177:6815/3486820237","heartbeat_front_addr":"172.21.15.177:6813/3486820237","state":["exists","up"]},{"osd":6,"uuid":"aa72392f-0498-44db-a8d2-eccc89a64871","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":2579716443},{"type":"v1","addr":"172.21.15.90:6819","nonce":2579716443}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6820","nonce":2579716443},{"type":"v1","addr":"172.21.15.90:6821","nonce":2579716443}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":2579716443},{"type":"v1","addr":"172.21.15.90:6825","nonce":2579716443}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2579716443},{"type":"v1","addr":"172.21.15.90:6823","nonce":2579716443}]},"public_addr":"172.21.15.90:6819/2579716443","cluster_addr":"172.21.15.90:6821/2579716443","heartbeat_back_addr":"172.21.15.90:6825/2579716443","heartbeat_front_addr":"172.21.15.90:6823/2579716443","state":["exists","up"]},{"osd":7,"uuid":"931bd105-bc14-4a1e-bfe4-c7ad58fa4b73","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":3371649139},{"type":"v1","addr":"172.21.15.92:6817","nonce":3371649139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":3371649139},{"type":"v1","addr":"172.21.15.92:6819","nonce":3371649139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":3371649139},{"type":"v1","addr":"172.21.15.92:6823","nonce":3371649139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6820","nonce":3371649139},{"type":"v1","addr":"172.21.15.92:6821","nonce":3371649139}]},"public_addr":"172.21.15.92:6817/3371649139","cluster_addr":"172.21.15.92:6819/3371649139","heartbeat_back_addr":"172.21.15.92:6823/3371649139","heartbeat_front_addr":"172.21.15.92:6821/3371649139","state":["exists","up"]},{"osd":8,"uuid":"82624ec2-b382-4886-9f40-7f5f376de3ce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":1459053034},{"type":"v1","addr":"172.21.15.177:6817","nonce":1459053034}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":1459053034},{"type":"v1","addr":"172.21.15.177:6819","nonce":1459053034}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":1459053034},{"type":"v1","addr":"172.21.15.177:6823","nonce":1459053034}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":1459053034},{"type":"v1","addr":"172.21.15.177:6821","nonce":1459053034}]},"public_addr":"172.21.15.177:6817/1459053034","cluster_addr":"172.21.15.177:6819/1459053034","heartbeat_back_addr":"172.21.15.177:6823/1459053034","heartbeat_front_addr":"172.21.15.177:6821/1459053034","state":["exists","up"]},{"osd":9,"uuid":"fbe36e48-3468-42b2-8094-e16552f3cb16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":2592604332},{"type":"v1","addr":"172.21.15.90:6827","nonce":2592604332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":2592604332},{"type":"v1","addr":"172.21.15.90:6829","nonce":2592604332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6832","nonce":2592604332},{"type":"v1","addr":"172.21.15.90:6833","nonce":2592604332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":2592604332},{"type":"v1","addr":"172.21.15.90:6831","nonce":2592604332}]},"public_addr":"172.21.15.90:6827/2592604332","cluster_addr":"172.21.15.90:6829/2592604332","heartbeat_back_addr":"172.21.15.90:6833/2592604332","heartbeat_front_addr":"172.21.15.90:6831/2592604332","state":["exists","up"]},{"osd":10,"uuid":"9240df75-e7e2-47f1-8aae-534b70b3c210","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":1594153142},{"type":"v1","addr":"172.21.15.92:6825","nonce":1594153142}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6826","nonce":1594153142},{"type":"v1","addr":"172.21.15.92:6827","nonce":1594153142}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":1594153142},{"type":"v1","addr":"172.21.15.92:6831","nonce":1594153142}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":1594153142},{"type":"v1","addr":"172.21.15.92:6829","nonce":1594153142}]},"public_addr":"172.21.15.92:6825/1594153142","cluster_addr":"172.21.15.92:6827/1594153142","heartbeat_back_addr":"172.21.15.92:6831/1594153142","heartbeat_front_addr":"172.21.15.92:6829/1594153142","state":["exists","up"]},{"osd":11,"uuid":"83cba9fa-b745-4df1-adda-e86837d4b91a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":83,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1622582126},{"type":"v1","addr":"172.21.15.177:6825","nonce":1622582126}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":1622582126},{"type":"v1","addr":"172.21.15.177:6827","nonce":1622582126}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":1622582126},{"type":"v1","addr":"172.21.15.177:6831","nonce":1622582126}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1622582126},{"type":"v1","addr":"172.21.15.177:6829","nonce":1622582126}]},"public_addr":"172.21.15.177:6825/1622582126","cluster_addr":"172.21.15.177:6827/1622582126","heartbeat_back_addr":"172.21.15.177:6831/1622582126","heartbeat_front_addr":"172.21.15.177:6829/1622582126","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:26:24.290365+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:26:49.405988+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:27:13.639196+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:27:45.603746+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:28:11.019126+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:28:35.084163+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:29:08.064626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:29:33.266133+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:29:57.381816+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:30:30.286847+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:30:56.226279+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:31:20.809611+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.90:0/314641601":"2024-04-03T14:24:34.595561+0000","172.21.15.90:0/1400895868":"2024-04-03T14:24:34.595561+0000","172.21.15.90:6801/129436436":"2024-04-03T14:24:34.595561+0000","172.21.15.90:6801/2372398598":"2024-04-03T14:24:00.362147+0000","172.21.15.90:0/278569338":"2024-04-03T14:24:34.595561+0000","172.21.15.90:0/2321464594":"2024-04-03T14:24:00.362147+0000","172.21.15.90:6800/2372398598":"2024-04-03T14:24:00.362147+0000","172.21.15.90:0/2962656672":"2024-04-03T14:24:00.362147+0000","172.21.15.90:0/126693610":"2024-04-03T14:24:00.362147+0000","172.21.15.90:6800/129436436":"2024-04-03T14:24:34.595561+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T14:31:31.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:31.577+0000 7f879a800700 1 -- 172.21.15.90:0/584340183 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f87780a3c40 msgr2=0x7f87780a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:31:31.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:31.577+0000 7f879a800700 1 --2- 172.21.15.90:0/584340183 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f87780a3c40 0x7f87780a6100 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f8784011550 tx=0x7f8784009260 comp rx=0 tx=0).stop 2024-04-02T14:31:31.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:31.577+0000 7f879a800700 1 -- 172.21.15.90:0/584340183 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8794108650 msgr2=0x7f8794102040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:31:31.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:31.577+0000 7f879a800700 1 --2- 172.21.15.90:0/584340183 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8794108650 0x7f8794102040 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f878c009bc0 tx=0x7f878c004c80 comp rx=0 tx=0).stop 2024-04-02T14:31:31.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:31.577+0000 7f879a800700 1 -- 172.21.15.90:0/584340183 shutdown_connections 2024-04-02T14:31:31.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:31.577+0000 7f879a800700 1 --2- 172.21.15.90:0/584340183 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f87780a3c40 0x7f87780a6100 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:31:31.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:31.577+0000 7f879a800700 1 --2- 172.21.15.90:0/584340183 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8794108650 0x7f8794102040 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:31:31.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:31.577+0000 7f879a800700 1 --2- 172.21.15.90:0/584340183 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f879406a2a0 0x7f87941015c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:31:31.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:31.577+0000 7f879a800700 1 --2- 172.21.15.90:0/584340183 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f879406ac60 0x7f8794101b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:31:31.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:31.577+0000 7f879a800700 1 -- 172.21.15.90:0/584340183 >> 172.21.15.90:0/584340183 conn(0x7f8794078420 msgr2=0x7f87941001d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:31:31.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:31.577+0000 7f879a800700 1 -- 172.21.15.90:0/584340183 shutdown_connections 2024-04-02T14:31:31.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:31.578+0000 7f879a800700 1 -- 172.21.15.90:0/584340183 wait complete. 2024-04-02T14:31:32.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:31:31 smithi177 ceph-mon[36431]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-02T14:31:32.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:31:31 smithi177 ceph-mon[36431]: from='client.? 172.21.15.90:0/584340183' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-02T14:31:32.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:31:31 smithi090 ceph-mon[31674]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-02T14:31:32.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:31:31 smithi090 ceph-mon[31674]: from='client.? 172.21.15.90:0/584340183' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-02T14:31:32.302 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-02T14:27:16.758622+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-02T14:31:32.303 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:191d963ac9de7322da8aff20b567b3fa458e2ef6 shell --fsid 6449448a-f0fc-11ee-b647-cb9ed24678a4 -- ceph osd pool get .mgr pg_num 2024-04-02T14:31:32.360 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:31:31 smithi092 ceph-mon[36497]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-02T14:31:32.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:31:31 smithi092 ceph-mon[36497]: from='client.? 172.21.15.90:0/584340183' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-02T14:31:32.752 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T14:31:33.924 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:31:33 smithi090 ceph-mon[31674]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-02T14:31:34.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:31:33 smithi177 ceph-mon[36431]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-02T14:31:34.360 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:31:33 smithi092 ceph-mon[36497]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-02T14:31:34.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:34.751+0000 7fe904c00700 1 -- 172.21.15.90:0/491779120 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe90006acf0 msgr2=0x7fe90006b170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:31:34.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:34.751+0000 7fe904c00700 1 --2- 172.21.15.90:0/491779120 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe90006acf0 0x7fe90006b170 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fe8ec0099c0 tx=0x7fe8ec009cd0 comp rx=0 tx=0).stop 2024-04-02T14:31:34.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:34.751+0000 7fe904c00700 1 -- 172.21.15.90:0/491779120 shutdown_connections 2024-04-02T14:31:34.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:34.751+0000 7fe904c00700 1 --2- 172.21.15.90:0/491779120 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe90006b6b0 0x7fe90010f4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:31:34.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:34.751+0000 7fe904c00700 1 --2- 172.21.15.90:0/491779120 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe900108650 0x7fe900108a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:31:34.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:34.751+0000 7fe904c00700 1 --2- 172.21.15.90:0/491779120 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe90006acf0 0x7fe90006b170 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:31:34.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:34.751+0000 7fe904c00700 1 -- 172.21.15.90:0/491779120 >> 172.21.15.90:0/491779120 conn(0x7fe900077510 msgr2=0x7fe900077920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:31:34.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:34.751+0000 7fe904c00700 1 -- 172.21.15.90:0/491779120 shutdown_connections 2024-04-02T14:31:34.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:34.751+0000 7fe904c00700 1 -- 172.21.15.90:0/491779120 wait complete. 2024-04-02T14:31:34.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:34.752+0000 7fe904c00700 1 Processor -- start 2024-04-02T14:31:34.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:34.752+0000 7fe904c00700 1 -- start start 2024-04-02T14:31:34.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:34.753+0000 7fe904c00700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe90006acf0 0x7fe90019e870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:31:34.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:34.753+0000 7fe904c00700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe90006b6b0 0x7fe90019edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:31:34.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:34.753+0000 7fe904c00700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe900108650 0x7fe900198a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:31:34.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:34.753+0000 7fe904c00700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fe9001128c0 con 0x7fe90006acf0 2024-04-02T14:31:34.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:34.753+0000 7fe904c00700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fe900112740 con 0x7fe90006b6b0 2024-04-02T14:31:34.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:34.753+0000 7fe904c00700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fe900112a40 con 0x7fe900108650 2024-04-02T14:31:34.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:34.753+0000 7fe8fde00700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe90006acf0 0x7fe90019e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:31:34.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:34.753+0000 7fe8fde00700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe90006acf0 0x7fe90019e870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:47788/0 (socket says 172.21.15.90:47788) 2024-04-02T14:31:34.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:34.753+0000 7fe8fde00700 1 -- 172.21.15.90:0/4091341327 learned_addr learned my addr 172.21.15.90:0/4091341327 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:31:34.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:34.753+0000 7fe8fe800700 1 --2- 172.21.15.90:0/4091341327 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe900108650 0x7fe900198a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:31:34.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:34.753+0000 7fe8fd400700 1 --2- 172.21.15.90:0/4091341327 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe90006b6b0 0x7fe90019edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:31:34.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:34.753+0000 7fe8fde00700 1 -- 172.21.15.90:0/4091341327 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe900108650 msgr2=0x7fe900198a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:31:34.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:34.753+0000 7fe8fde00700 1 --2- 172.21.15.90:0/4091341327 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe900108650 0x7fe900198a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:31:34.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:34.753+0000 7fe8fde00700 1 -- 172.21.15.90:0/4091341327 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe90006b6b0 msgr2=0x7fe90019edb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:31:34.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:34.753+0000 7fe8fde00700 1 --2- 172.21.15.90:0/4091341327 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe90006b6b0 0x7fe90019edb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:31:34.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:34.754+0000 7fe8fde00700 1 -- 172.21.15.90:0/4091341327 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9001991e0 con 0x7fe90006acf0 2024-04-02T14:31:34.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:34.754+0000 7fe8fde00700 1 --2- 172.21.15.90:0/4091341327 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe90006acf0 0x7fe90019e870 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fe8f000eb30 tx=0x7fe8f000ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:31:34.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:34.754+0000 7fe8f6a00700 1 -- 172.21.15.90:0/4091341327 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8f000cbd0 con 0x7fe90006acf0 2024-04-02T14:31:34.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:34.754+0000 7fe8f6a00700 1 -- 172.21.15.90:0/4091341327 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe8f0004500 con 0x7fe90006acf0 2024-04-02T14:31:34.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:34.754+0000 7fe8f6a00700 1 -- 172.21.15.90:0/4091341327 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8f0010430 con 0x7fe90006acf0 2024-04-02T14:31:34.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:34.754+0000 7fe904c00700 1 -- 172.21.15.90:0/4091341327 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9001994d0 con 0x7fe90006acf0 2024-04-02T14:31:34.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:34.754+0000 7fe904c00700 1 -- 172.21.15.90:0/4091341327 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9001a5740 con 0x7fe90006acf0 2024-04-02T14:31:34.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:34.755+0000 7fe904c00700 1 -- 172.21.15.90:0/4091341327 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe900103d20 con 0x7fe90006acf0 2024-04-02T14:31:34.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:34.757+0000 7fe8f6a00700 1 -- 172.21.15.90:0/4091341327 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fe8f0003f00 con 0x7fe90006acf0 2024-04-02T14:31:34.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:34.759+0000 7fe8f6a00700 1 --2- 172.21.15.90:0/4091341327 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fe8e40a3cc0 0x7fe8e40a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:31:34.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:34.759+0000 7fe8f6a00700 1 -- 172.21.15.90:0/4091341327 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 7127+0+0 (secure 0 0 0) 0x7fe8f00d0300 con 0x7fe90006acf0 2024-04-02T14:31:34.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:34.759+0000 7fe8fd400700 1 --2- 172.21.15.90:0/4091341327 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fe8e40a3cc0 0x7fe8e40a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:31:34.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:34.760+0000 7fe8fd400700 1 --2- 172.21.15.90:0/4091341327 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fe8e40a3cc0 0x7fe8e40a6180 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fe900077b10 tx=0x7fe8ec005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:31:34.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:34.764+0000 7fe8f6a00700 1 -- 172.21.15.90:0/4091341327 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7fe8f009d0e0 con 0x7fe90006acf0 2024-04-02T14:31:34.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:34.992+0000 7fe904c00700 1 -- 172.21.15.90:0/4091341327 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fe90002dd30 con 0x7fe90006acf0 2024-04-02T14:31:34.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:34.993+0000 7fe8f6a00700 1 -- 172.21.15.90:0/4091341327 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v84) ==== 93+0+10 (secure 0 0 0) 0x7fe8f009c9e0 con 0x7fe90006acf0 2024-04-02T14:31:34.996 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 1 2024-04-02T14:31:34.998 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:34.996+0000 7fe904c00700 1 -- 172.21.15.90:0/4091341327 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fe8e40a3cc0 msgr2=0x7fe8e40a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:31:34.998 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:34.996+0000 7fe904c00700 1 --2- 172.21.15.90:0/4091341327 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fe8e40a3cc0 0x7fe8e40a6180 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fe900077b10 tx=0x7fe8ec005fb0 comp rx=0 tx=0).stop 2024-04-02T14:31:34.998 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:34.996+0000 7fe904c00700 1 -- 172.21.15.90:0/4091341327 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe90006acf0 msgr2=0x7fe90019e870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:31:34.998 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:34.996+0000 7fe904c00700 1 --2- 172.21.15.90:0/4091341327 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe90006acf0 0x7fe90019e870 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fe8f000eb30 tx=0x7fe8f000ee40 comp rx=0 tx=0).stop 2024-04-02T14:31:34.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:34.996+0000 7fe904c00700 1 -- 172.21.15.90:0/4091341327 shutdown_connections 2024-04-02T14:31:34.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:34.996+0000 7fe904c00700 1 --2- 172.21.15.90:0/4091341327 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fe8e40a3cc0 0x7fe8e40a6180 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:31:34.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:34.996+0000 7fe904c00700 1 --2- 172.21.15.90:0/4091341327 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe90006acf0 0x7fe90019e870 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:31:34.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:34.996+0000 7fe904c00700 1 --2- 172.21.15.90:0/4091341327 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe900108650 0x7fe900198a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:31:34.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:34.996+0000 7fe904c00700 1 --2- 172.21.15.90:0/4091341327 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe90006b6b0 0x7fe90019edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:31:34.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:34.996+0000 7fe904c00700 1 -- 172.21.15.90:0/4091341327 >> 172.21.15.90:0/4091341327 conn(0x7fe900077510 msgr2=0x7fe900100150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:31:35.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:34.996+0000 7fe904c00700 1 -- 172.21.15.90:0/4091341327 shutdown_connections 2024-04-02T14:31:35.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:34.997+0000 7fe904c00700 1 -- 172.21.15.90:0/4091341327 wait complete. 2024-04-02T14:31:35.704 INFO:tasks.cephadm:Adding mds.a on smithi090 2024-04-02T14:31:35.704 INFO:tasks.cephadm:Adding mds.d on smithi090 2024-04-02T14:31:35.704 INFO:tasks.cephadm:Adding mds.g on smithi090 2024-04-02T14:31:35.704 INFO:tasks.cephadm:Adding mds.j on smithi090 2024-04-02T14:31:35.704 INFO:tasks.cephadm:Adding mds.b on smithi092 2024-04-02T14:31:35.704 INFO:tasks.cephadm:Adding mds.e on smithi092 2024-04-02T14:31:35.704 INFO:tasks.cephadm:Adding mds.h on smithi092 2024-04-02T14:31:35.705 INFO:tasks.cephadm:Adding mds.k on smithi092 2024-04-02T14:31:35.705 INFO:tasks.cephadm:Adding mds.c on smithi177 2024-04-02T14:31:35.705 INFO:tasks.cephadm:Adding mds.f on smithi177 2024-04-02T14:31:35.705 INFO:tasks.cephadm:Adding mds.i on smithi177 2024-04-02T14:31:35.705 INFO:tasks.cephadm:Adding mds.l on smithi177 2024-04-02T14:31:35.705 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:191d963ac9de7322da8aff20b567b3fa458e2ef6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6449448a-f0fc-11ee-b647-cb9ed24678a4 -- ceph orch apply mds all '12;smithi090=a;smithi090=d;smithi090=g;smithi090=j;smithi092=b;smithi092=e;smithi092=h;smithi092=k;smithi177=c;smithi177=f;smithi177=i;smithi177=l' 2024-04-02T14:31:35.939 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:31:35 smithi090 ceph-mon[31674]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-02T14:31:35.939 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:31:35 smithi090 ceph-mon[31674]: from='client.? 172.21.15.90:0/4091341327' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-02T14:31:35.957 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/mon.c/config 2024-04-02T14:31:36.079 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:31:35 smithi177 ceph-mon[36431]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-02T14:31:36.079 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:31:35 smithi177 ceph-mon[36431]: from='client.? 172.21.15.90:0/4091341327' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-02T14:31:36.360 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:31:35 smithi092 ceph-mon[36497]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-02T14:31:36.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:31:35 smithi092 ceph-mon[36497]: from='client.? 172.21.15.90:0/4091341327' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-02T14:31:37.588 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:31:37.584+0000 7f0c634c5700 1 -- 172.21.15.177:0/1813189549 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0c5c102640 msgr2=0x7f0c5c102a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:31:37.588 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:31:37.584+0000 7f0c634c5700 1 --2- 172.21.15.177:0/1813189549 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0c5c102640 0x7f0c5c102a60 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f0c4c009990 tx=0x7f0c4c009ca0 comp rx=0 tx=0).stop 2024-04-02T14:31:37.588 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:31:37.584+0000 7f0c634c5700 1 -- 172.21.15.177:0/1813189549 shutdown_connections 2024-04-02T14:31:37.589 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:31:37.584+0000 7f0c634c5700 1 --2- 172.21.15.177:0/1813189549 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0c5c1041e0 0x7f0c5c10aa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:31:37.589 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:31:37.584+0000 7f0c634c5700 1 --2- 172.21.15.177:0/1813189549 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0c5c102640 0x7f0c5c102a60 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:31:37.589 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:31:37.584+0000 7f0c634c5700 1 --2- 172.21.15.177:0/1813189549 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0c5c103840 0x7f0c5c103ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:31:37.589 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:31:37.584+0000 7f0c634c5700 1 -- 172.21.15.177:0/1813189549 >> 172.21.15.177:0/1813189549 conn(0x7f0c5c0fdc00 msgr2=0x7f0c5c100020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:31:37.589 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:31:37.584+0000 7f0c634c5700 1 -- 172.21.15.177:0/1813189549 shutdown_connections 2024-04-02T14:31:37.589 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:31:37.584+0000 7f0c634c5700 1 -- 172.21.15.177:0/1813189549 wait complete. 2024-04-02T14:31:37.589 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:31:37.585+0000 7f0c634c5700 1 Processor -- start 2024-04-02T14:31:37.589 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:31:37.585+0000 7f0c634c5700 1 -- start start 2024-04-02T14:31:37.590 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:31:37.585+0000 7f0c634c5700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0c5c102640 0x7f0c5c076390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:31:37.590 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:31:37.585+0000 7f0c634c5700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0c5c103840 0x7f0c5c0749e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:31:37.590 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:31:37.585+0000 7f0c634c5700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0c5c1041e0 0x7f0c5c074f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:31:37.590 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:31:37.585+0000 7f0c634c5700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f0c5c10e2a0 con 0x7f0c5c1041e0 2024-04-02T14:31:37.590 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:31:37.585+0000 7f0c634c5700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f0c5c10e120 con 0x7f0c5c102640 2024-04-02T14:31:37.590 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:31:37.585+0000 7f0c634c5700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f0c5c10e420 con 0x7f0c5c103840 2024-04-02T14:31:37.590 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:31:37.585+0000 7f0c60a60700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0c5c103840 0x7f0c5c0749e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:31:37.590 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:31:37.585+0000 7f0c60a60700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0c5c103840 0x7f0c5c0749e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.177:37498/0 (socket says 172.21.15.177:37498) 2024-04-02T14:31:37.590 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:31:37.585+0000 7f0c60a60700 1 -- 172.21.15.177:0/131326033 learned_addr learned my addr 172.21.15.177:0/131326033 (peer_addr_for_me v2:172.21.15.177:0/0) 2024-04-02T14:31:37.590 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:31:37.585+0000 7f0c61261700 1 --2- 172.21.15.177:0/131326033 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0c5c102640 0x7f0c5c076390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:31:37.590 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:31:37.586+0000 7f0c60a60700 1 -- 172.21.15.177:0/131326033 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0c5c102640 msgr2=0x7f0c5c076390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:31:37.591 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:31:37.586+0000 7f0c60a60700 1 --2- 172.21.15.177:0/131326033 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0c5c102640 0x7f0c5c076390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:31:37.591 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:31:37.586+0000 7f0c60a60700 1 -- 172.21.15.177:0/131326033 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0c5c1041e0 msgr2=0x7f0c5c074f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:31:37.591 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:31:37.586+0000 7f0c60a60700 1 --2- 172.21.15.177:0/131326033 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0c5c1041e0 0x7f0c5c074f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:31:37.591 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:31:37.586+0000 7f0c60a60700 1 -- 172.21.15.177:0/131326033 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c5c0757d0 con 0x7f0c5c103840 2024-04-02T14:31:37.591 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:31:37.586+0000 7f0c60a60700 1 --2- 172.21.15.177:0/131326033 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0c5c103840 0x7f0c5c0749e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f0c5800b530 tx=0x7f0c5800b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:31:37.591 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:31:37.586+0000 7f0c527fc700 1 -- 172.21.15.177:0/131326033 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c58015020 con 0x7f0c5c103840 2024-04-02T14:31:37.591 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:31:37.586+0000 7f0c527fc700 1 -- 172.21.15.177:0/131326033 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0c58011d10 con 0x7f0c5c103840 2024-04-02T14:31:37.591 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:31:37.586+0000 7f0c634c5700 1 -- 172.21.15.177:0/131326033 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0c5c1a2a50 con 0x7f0c5c103840 2024-04-02T14:31:37.591 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:31:37.586+0000 7f0c527fc700 1 -- 172.21.15.177:0/131326033 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c5800f2e0 con 0x7f0c5c103840 2024-04-02T14:31:37.591 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:31:37.586+0000 7f0c634c5700 1 -- 172.21.15.177:0/131326033 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0c5c1a2fe0 con 0x7f0c5c103840 2024-04-02T14:31:37.591 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:31:37.587+0000 7f0c634c5700 1 -- 172.21.15.177:0/131326033 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0c5c0692b0 con 0x7f0c5c103840 2024-04-02T14:31:37.592 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:31:37.588+0000 7f0c527fc700 1 -- 172.21.15.177:0/131326033 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f0c58018020 con 0x7f0c5c103840 2024-04-02T14:31:37.592 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:31:37.588+0000 7f0c527fc700 1 --2- 172.21.15.177:0/131326033 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f0c480a3c40 0x7f0c480a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:31:37.592 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:31:37.589+0000 7f0c527fc700 1 -- 172.21.15.177:0/131326033 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 7127+0+0 (secure 0 0 0) 0x7f0c580cf7e0 con 0x7f0c5c103840 2024-04-02T14:31:37.592 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:31:37.589+0000 7f0c61261700 1 --2- 172.21.15.177:0/131326033 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f0c480a3c40 0x7f0c480a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:31:37.592 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:31:37.590+0000 7f0c61261700 1 --2- 172.21.15.177:0/131326033 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f0c480a3c40 0x7f0c480a6100 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f0c4c003890 tx=0x7f0c4c005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:31:37.594 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:31:37.593+0000 7f0c527fc700 1 -- 172.21.15.177:0/131326033 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7f0c5801f020 con 0x7f0c5c103840 2024-04-02T14:31:37.803 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:31:37.802+0000 7f0c634c5700 1 -- 172.21.15.177:0/131326033 --> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] -- mgr_command(tid 0: {"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi090=a;smithi090=d;smithi090=g;smithi090=j;smithi092=b;smithi092=e;smithi092=h;smithi092=k;smithi177=c;smithi177=f;smithi177=i;smithi177=l", "target": ["mon-mgr", ""]}) -- 0x7f0c5c062c30 con 0x7f0c480a3c40 2024-04-02T14:31:37.819 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:31:37.818+0000 7f0c527fc700 1 -- 172.21.15.177:0/131326033 <== mgr.14152 v2:172.21.15.90:6800/4266054817 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+28 (secure 0 0 0) 0x7f0c5c062c30 con 0x7f0c480a3c40 2024-04-02T14:31:37.820 INFO:teuthology.orchestra.run.smithi177.stdout:Scheduled mds.all update... 2024-04-02T14:31:37.822 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:31:37.820+0000 7f0c634c5700 1 -- 172.21.15.177:0/131326033 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f0c480a3c40 msgr2=0x7f0c480a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:31:37.822 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:31:37.820+0000 7f0c634c5700 1 --2- 172.21.15.177:0/131326033 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f0c480a3c40 0x7f0c480a6100 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f0c4c003890 tx=0x7f0c4c005fd0 comp rx=0 tx=0).stop 2024-04-02T14:31:37.822 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:31:37.820+0000 7f0c634c5700 1 -- 172.21.15.177:0/131326033 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0c5c103840 msgr2=0x7f0c5c0749e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:31:37.822 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:31:37.820+0000 7f0c634c5700 1 --2- 172.21.15.177:0/131326033 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0c5c103840 0x7f0c5c0749e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f0c5800b530 tx=0x7f0c5800b840 comp rx=0 tx=0).stop 2024-04-02T14:31:37.822 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:31:37.820+0000 7f0c634c5700 1 -- 172.21.15.177:0/131326033 shutdown_connections 2024-04-02T14:31:37.822 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:31:37.820+0000 7f0c634c5700 1 --2- 172.21.15.177:0/131326033 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f0c480a3c40 0x7f0c480a6100 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:31:37.823 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:31:37.820+0000 7f0c634c5700 1 --2- 172.21.15.177:0/131326033 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0c5c1041e0 0x7f0c5c074f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:31:37.823 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:31:37.820+0000 7f0c634c5700 1 --2- 172.21.15.177:0/131326033 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0c5c102640 0x7f0c5c076390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:31:37.823 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:31:37.820+0000 7f0c634c5700 1 --2- 172.21.15.177:0/131326033 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0c5c103840 0x7f0c5c0749e0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:31:37.823 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:31:37.820+0000 7f0c634c5700 1 -- 172.21.15.177:0/131326033 >> 172.21.15.177:0/131326033 conn(0x7f0c5c0fdc00 msgr2=0x7f0c5c106b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:31:37.823 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:31:37.821+0000 7f0c634c5700 1 -- 172.21.15.177:0/131326033 shutdown_connections 2024-04-02T14:31:37.823 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-02T14:31:37.821+0000 7f0c634c5700 1 -- 172.21.15.177:0/131326033 wait complete. 2024-04-02T14:31:38.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:31:37 smithi177 ceph-mon[36431]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-02T14:31:38.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:31:37 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:31:38.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:31:37 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T14:31:38.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:31:37 smithi090 ceph-mon[31674]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-02T14:31:38.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:31:37 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:31:38.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:31:37 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T14:31:38.360 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:31:37 smithi092 ceph-mon[36497]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-02T14:31:38.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:31:37 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:31:38.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:31:37 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T14:31:38.521 DEBUG:teuthology.orchestra.run.smithi090:mds.a> sudo journalctl -f -n 0 -u ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4@mds.a.service 2024-04-02T14:31:38.526 DEBUG:teuthology.orchestra.run.smithi090:mds.d> sudo journalctl -f -n 0 -u ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4@mds.d.service 2024-04-02T14:31:38.572 DEBUG:teuthology.orchestra.run.smithi090:mds.g> sudo journalctl -f -n 0 -u ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4@mds.g.service 2024-04-02T14:31:38.618 DEBUG:teuthology.orchestra.run.smithi090:mds.j> sudo journalctl -f -n 0 -u ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4@mds.j.service 2024-04-02T14:31:38.664 DEBUG:teuthology.orchestra.run.smithi092:mds.b> sudo journalctl -f -n 0 -u ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4@mds.b.service 2024-04-02T14:31:38.667 DEBUG:teuthology.orchestra.run.smithi092:mds.e> sudo journalctl -f -n 0 -u ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4@mds.e.service 2024-04-02T14:31:38.710 DEBUG:teuthology.orchestra.run.smithi092:mds.h> sudo journalctl -f -n 0 -u ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4@mds.h.service 2024-04-02T14:31:38.753 DEBUG:teuthology.orchestra.run.smithi092:mds.k> sudo journalctl -f -n 0 -u ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4@mds.k.service 2024-04-02T14:31:38.796 DEBUG:teuthology.orchestra.run.smithi177:mds.c> sudo journalctl -f -n 0 -u ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4@mds.c.service 2024-04-02T14:31:38.798 DEBUG:teuthology.orchestra.run.smithi177:mds.f> sudo journalctl -f -n 0 -u ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4@mds.f.service 2024-04-02T14:31:38.842 DEBUG:teuthology.orchestra.run.smithi177:mds.i> sudo journalctl -f -n 0 -u ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4@mds.i.service 2024-04-02T14:31:38.885 DEBUG:teuthology.orchestra.run.smithi177:mds.l> sudo journalctl -f -n 0 -u ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4@mds.l.service 2024-04-02T14:31:38.928 INFO:tasks.cephadm:Setting up CephFS filesystem(s)... 2024-04-02T14:31:38.929 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T14:31:39.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:39.156+0000 7f8203000640 1 -- 172.21.15.90:0/2061173216 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f81fc1090e0 msgr2=0x7f81fc109560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:31:39.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:39.156+0000 7f8203000640 1 --2- 172.21.15.90:0/2061173216 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f81fc1090e0 0x7f81fc109560 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f81e8009be0 tx=0x7f81e802bef0 comp rx=0 tx=0).stop 2024-04-02T14:31:39.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:39.157+0000 7f8203000640 1 -- 172.21.15.90:0/2061173216 shutdown_connections 2024-04-02T14:31:39.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:39.157+0000 7f8203000640 1 --2- 172.21.15.90:0/2061173216 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f81fc109aa0 0x7f81fc1103c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:31:39.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:39.157+0000 7f8203000640 1 --2- 172.21.15.90:0/2061173216 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f81fc1090e0 0x7f81fc109560 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:31:39.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:39.157+0000 7f8203000640 1 --2- 172.21.15.90:0/2061173216 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f81fc107f10 0x7f81fc108350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:31:39.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:39.157+0000 7f8203000640 1 -- 172.21.15.90:0/2061173216 >> 172.21.15.90:0/2061173216 conn(0x7f81fc076f10 msgr2=0x7f81fc079330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:31:39.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:39.157+0000 7f8203000640 1 -- 172.21.15.90:0/2061173216 shutdown_connections 2024-04-02T14:31:39.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:39.157+0000 7f8203000640 1 -- 172.21.15.90:0/2061173216 wait complete. 2024-04-02T14:31:39.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:39.158+0000 7f8203000640 1 Processor -- start 2024-04-02T14:31:39.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:39.158+0000 7f8203000640 1 -- start start 2024-04-02T14:31:39.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:39.158+0000 7f8203000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f81fc107f10 0x7f81fc19e250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:31:39.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:39.158+0000 7f8203000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f81fc1090e0 0x7f81fc19e790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:31:39.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:39.158+0000 7f8203000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f81fc109aa0 0x7f81fc1a5800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:31:39.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:39.158+0000 7f8203000640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f81fc113c30 con 0x7f81fc1090e0 2024-04-02T14:31:39.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:39.158+0000 7f8203000640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f81fc113ab0 con 0x7f81fc107f10 2024-04-02T14:31:39.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:39.158+0000 7f8203000640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f81fc113db0 con 0x7f81fc109aa0 2024-04-02T14:31:39.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:39.159+0000 7f8201c00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f81fc107f10 0x7f81fc19e250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:31:39.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:39.159+0000 7f8201200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f81fc1090e0 0x7f81fc19e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:31:39.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:39.159+0000 7f8201c00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f81fc107f10 0x7f81fc19e250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:35496/0 (socket says 172.21.15.90:35496) 2024-04-02T14:31:39.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:39.159+0000 7f8201c00640 1 -- 172.21.15.90:0/108967349 learned_addr learned my addr 172.21.15.90:0/108967349 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:31:39.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:39.159+0000 7f8202600640 1 --2- 172.21.15.90:0/108967349 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f81fc109aa0 0x7f81fc1a5800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:31:39.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:39.159+0000 7f8201c00640 1 -- 172.21.15.90:0/108967349 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f81fc109aa0 msgr2=0x7f81fc1a5800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:31:39.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:39.159+0000 7f8201c00640 1 --2- 172.21.15.90:0/108967349 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f81fc109aa0 0x7f81fc1a5800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:31:39.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:39.159+0000 7f8201c00640 1 -- 172.21.15.90:0/108967349 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f81fc1090e0 msgr2=0x7f81fc19e790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:31:39.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:39.159+0000 7f8201c00640 1 --2- 172.21.15.90:0/108967349 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f81fc1090e0 0x7f81fc19e790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:31:39.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:39.159+0000 7f8201c00640 1 -- 172.21.15.90:0/108967349 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81fc1a5e70 con 0x7f81fc107f10 2024-04-02T14:31:39.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:39.159+0000 7f8201200640 1 --2- 172.21.15.90:0/108967349 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f81fc1090e0 0x7f81fc19e790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:31:39.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:39.159+0000 7f8201c00640 1 --2- 172.21.15.90:0/108967349 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f81fc107f10 0x7f81fc19e250 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f81ec00e9c0 tx=0x7f81ec00edc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:31:39.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:39.160+0000 7f81fa000640 1 -- 172.21.15.90:0/108967349 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81ec00ccc0 con 0x7f81fc107f10 2024-04-02T14:31:39.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:39.160+0000 7f81fa000640 1 -- 172.21.15.90:0/108967349 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f81ec0035f0 con 0x7f81fc107f10 2024-04-02T14:31:39.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:39.160+0000 7f8203000640 1 -- 172.21.15.90:0/108967349 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f81fc1a6160 con 0x7f81fc107f10 2024-04-02T14:31:39.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:39.160+0000 7f81fa000640 1 -- 172.21.15.90:0/108967349 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81ec00ce60 con 0x7f81fc107f10 2024-04-02T14:31:39.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:39.160+0000 7f8203000640 1 -- 172.21.15.90:0/108967349 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81fc1a6680 con 0x7f81fc107f10 2024-04-02T14:31:39.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:39.161+0000 7f8203000640 1 -- 172.21.15.90:0/108967349 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f81fc10f2a0 con 0x7f81fc107f10 2024-04-02T14:31:39.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:39.163+0000 7f81fa000640 1 -- 172.21.15.90:0/108967349 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f81ec0037a0 con 0x7f81fc107f10 2024-04-02T14:31:39.166 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:31:38 smithi092 ceph-mon[36497]: from='client.24430 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi090=a;smithi090=d;smithi090=g;smithi090=j;smithi092=b;smithi092=e;smithi092=h;smithi092=k;smithi177=c;smithi177=f;smithi177=i;smithi177=l", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T14:31:39.166 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:31:38 smithi092 ceph-mon[36497]: Saving service mds.all spec with placement smithi090=a;smithi090=d;smithi090=g;smithi090=j;smithi092=b;smithi092=e;smithi092=h;smithi092=k;smithi177=c;smithi177=f;smithi177=i;smithi177=l;count:12 2024-04-02T14:31:39.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:39.164+0000 7f81fa000640 1 --2- 172.21.15.90:0/108967349 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f81e00a3c30 0x7f81e00a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:31:39.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:39.164+0000 7f81fa000640 1 -- 172.21.15.90:0/108967349 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 7127+0+0 (secure 0 0 0) 0x7f81ec0d02d0 con 0x7f81fc107f10 2024-04-02T14:31:39.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:39.164+0000 7f8201200640 1 --2- 172.21.15.90:0/108967349 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f81e00a3c30 0x7f81e00a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:31:39.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:39.165+0000 7f8201200640 1 --2- 172.21.15.90:0/108967349 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f81e00a3c30 0x7f81e00a60f0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f81e8006010 tx=0x7f81e8000f40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:31:39.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:39.168+0000 7f81fa000640 1 -- 172.21.15.90:0/108967349 <== mon.2 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7f81ec09d0b0 con 0x7f81fc107f10 2024-04-02T14:31:39.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:31:38 smithi177 ceph-mon[36431]: from='client.24430 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi090=a;smithi090=d;smithi090=g;smithi090=j;smithi092=b;smithi092=e;smithi092=h;smithi092=k;smithi177=c;smithi177=f;smithi177=i;smithi177=l", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T14:31:39.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:31:38 smithi177 ceph-mon[36431]: Saving service mds.all spec with placement smithi090=a;smithi090=d;smithi090=g;smithi090=j;smithi092=b;smithi092=e;smithi092=h;smithi092=k;smithi177=c;smithi177=f;smithi177=i;smithi177=l;count:12 2024-04-02T14:31:39.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:31:38 smithi090 ceph-mon[31674]: from='client.24430 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi090=a;smithi090=d;smithi090=g;smithi090=j;smithi092=b;smithi092=e;smithi092=h;smithi092=k;smithi177=c;smithi177=f;smithi177=i;smithi177=l", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T14:31:39.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:31:38 smithi090 ceph-mon[31674]: Saving service mds.all spec with placement smithi090=a;smithi090=d;smithi090=g;smithi090=j;smithi092=b;smithi092=e;smithi092=h;smithi092=k;smithi177=c;smithi177=f;smithi177=i;smithi177=l;count:12 2024-04-02T14:31:39.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:39.377+0000 7f8203000640 1 -- 172.21.15.90:0/108967349 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f81fc02d370 con 0x7f81fc107f10 2024-04-02T14:31:39.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:39.378+0000 7f81fa000640 1 -- 172.21.15.90:0/108967349 <== mon.2 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v84) ==== 74+0+18441 (secure 0 0 0) 0x7f81ec09c9b0 con 0x7f81fc107f10 2024-04-02T14:31:39.381 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-02T14:31:39.381 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":84,"fsid":"6449448a-f0fc-11ee-b647-cb9ed24678a4","created":"2024-04-02T14:23:16.530201+0000","modified":"2024-04-02T14:31:24.832762+0000","last_up_change":"2024-04-02T14:31:23.819732+0000","last_in_change":"2024-04-02T14:31:02.824284+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T14:27:16.758622+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a1a6a856-ab44-44bd-bc5e-45c718a70cd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":3635087609},{"type":"v1","addr":"172.21.15.90:6803","nonce":3635087609}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":3635087609},{"type":"v1","addr":"172.21.15.90:6805","nonce":3635087609}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":3635087609},{"type":"v1","addr":"172.21.15.90:6809","nonce":3635087609}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":3635087609},{"type":"v1","addr":"172.21.15.90:6807","nonce":3635087609}]},"public_addr":"172.21.15.90:6803/3635087609","cluster_addr":"172.21.15.90:6805/3635087609","heartbeat_back_addr":"172.21.15.90:6809/3635087609","heartbeat_front_addr":"172.21.15.90:6807/3635087609","state":["exists","up"]},{"osd":1,"uuid":"124139cd-5793-463e-9607-9d9c10164183","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":1994536059},{"type":"v1","addr":"172.21.15.92:6801","nonce":1994536059}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":1994536059},{"type":"v1","addr":"172.21.15.92:6803","nonce":1994536059}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":1994536059},{"type":"v1","addr":"172.21.15.92:6807","nonce":1994536059}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":1994536059},{"type":"v1","addr":"172.21.15.92:6805","nonce":1994536059}]},"public_addr":"172.21.15.92:6801/1994536059","cluster_addr":"172.21.15.92:6803/1994536059","heartbeat_back_addr":"172.21.15.92:6807/1994536059","heartbeat_front_addr":"172.21.15.92:6805/1994536059","state":["exists","up"]},{"osd":2,"uuid":"a61fbef2-9dca-4875-a814-020fff0db0e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":1358472800},{"type":"v1","addr":"172.21.15.177:6801","nonce":1358472800}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":1358472800},{"type":"v1","addr":"172.21.15.177:6803","nonce":1358472800}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":1358472800},{"type":"v1","addr":"172.21.15.177:6807","nonce":1358472800}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":1358472800},{"type":"v1","addr":"172.21.15.177:6805","nonce":1358472800}]},"public_addr":"172.21.15.177:6801/1358472800","cluster_addr":"172.21.15.177:6803/1358472800","heartbeat_back_addr":"172.21.15.177:6807/1358472800","heartbeat_front_addr":"172.21.15.177:6805/1358472800","state":["exists","up"]},{"osd":3,"uuid":"b175a2f4-7859-4eb0-bc1f-fba8c2b92a2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":3114811216},{"type":"v1","addr":"172.21.15.90:6811","nonce":3114811216}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":3114811216},{"type":"v1","addr":"172.21.15.90:6813","nonce":3114811216}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":3114811216},{"type":"v1","addr":"172.21.15.90:6817","nonce":3114811216}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":3114811216},{"type":"v1","addr":"172.21.15.90:6815","nonce":3114811216}]},"public_addr":"172.21.15.90:6811/3114811216","cluster_addr":"172.21.15.90:6813/3114811216","heartbeat_back_addr":"172.21.15.90:6817/3114811216","heartbeat_front_addr":"172.21.15.90:6815/3114811216","state":["exists","up"]},{"osd":4,"uuid":"38cc68ea-c109-4d8d-b77d-1c6f9b56c875","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":2987208879},{"type":"v1","addr":"172.21.15.92:6809","nonce":2987208879}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6810","nonce":2987208879},{"type":"v1","addr":"172.21.15.92:6811","nonce":2987208879}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6814","nonce":2987208879},{"type":"v1","addr":"172.21.15.92:6815","nonce":2987208879}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":2987208879},{"type":"v1","addr":"172.21.15.92:6813","nonce":2987208879}]},"public_addr":"172.21.15.92:6809/2987208879","cluster_addr":"172.21.15.92:6811/2987208879","heartbeat_back_addr":"172.21.15.92:6815/2987208879","heartbeat_front_addr":"172.21.15.92:6813/2987208879","state":["exists","up"]},{"osd":5,"uuid":"2347232e-ac91-466f-8ac9-496f397679ba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":3486820237},{"type":"v1","addr":"172.21.15.177:6809","nonce":3486820237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":3486820237},{"type":"v1","addr":"172.21.15.177:6811","nonce":3486820237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":3486820237},{"type":"v1","addr":"172.21.15.177:6815","nonce":3486820237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":3486820237},{"type":"v1","addr":"172.21.15.177:6813","nonce":3486820237}]},"public_addr":"172.21.15.177:6809/3486820237","cluster_addr":"172.21.15.177:6811/3486820237","heartbeat_back_addr":"172.21.15.177:6815/3486820237","heartbeat_front_addr":"172.21.15.177:6813/3486820237","state":["exists","up"]},{"osd":6,"uuid":"aa72392f-0498-44db-a8d2-eccc89a64871","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":2579716443},{"type":"v1","addr":"172.21.15.90:6819","nonce":2579716443}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6820","nonce":2579716443},{"type":"v1","addr":"172.21.15.90:6821","nonce":2579716443}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":2579716443},{"type":"v1","addr":"172.21.15.90:6825","nonce":2579716443}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2579716443},{"type":"v1","addr":"172.21.15.90:6823","nonce":2579716443}]},"public_addr":"172.21.15.90:6819/2579716443","cluster_addr":"172.21.15.90:6821/2579716443","heartbeat_back_addr":"172.21.15.90:6825/2579716443","heartbeat_front_addr":"172.21.15.90:6823/2579716443","state":["exists","up"]},{"osd":7,"uuid":"931bd105-bc14-4a1e-bfe4-c7ad58fa4b73","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":3371649139},{"type":"v1","addr":"172.21.15.92:6817","nonce":3371649139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":3371649139},{"type":"v1","addr":"172.21.15.92:6819","nonce":3371649139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":3371649139},{"type":"v1","addr":"172.21.15.92:6823","nonce":3371649139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6820","nonce":3371649139},{"type":"v1","addr":"172.21.15.92:6821","nonce":3371649139}]},"public_addr":"172.21.15.92:6817/3371649139","cluster_addr":"172.21.15.92:6819/3371649139","heartbeat_back_addr":"172.21.15.92:6823/3371649139","heartbeat_front_addr":"172.21.15.92:6821/3371649139","state":["exists","up"]},{"osd":8,"uuid":"82624ec2-b382-4886-9f40-7f5f376de3ce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":1459053034},{"type":"v1","addr":"172.21.15.177:6817","nonce":1459053034}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":1459053034},{"type":"v1","addr":"172.21.15.177:6819","nonce":1459053034}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":1459053034},{"type":"v1","addr":"172.21.15.177:6823","nonce":1459053034}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":1459053034},{"type":"v1","addr":"172.21.15.177:6821","nonce":1459053034}]},"public_addr":"172.21.15.177:6817/1459053034","cluster_addr":"172.21.15.177:6819/1459053034","heartbeat_back_addr":"172.21.15.177:6823/1459053034","heartbeat_front_addr":"172.21.15.177:6821/1459053034","state":["exists","up"]},{"osd":9,"uuid":"fbe36e48-3468-42b2-8094-e16552f3cb16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":2592604332},{"type":"v1","addr":"172.21.15.90:6827","nonce":2592604332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":2592604332},{"type":"v1","addr":"172.21.15.90:6829","nonce":2592604332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6832","nonce":2592604332},{"type":"v1","addr":"172.21.15.90:6833","nonce":2592604332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":2592604332},{"type":"v1","addr":"172.21.15.90:6831","nonce":2592604332}]},"public_addr":"172.21.15.90:6827/2592604332","cluster_addr":"172.21.15.90:6829/2592604332","heartbeat_back_addr":"172.21.15.90:6833/2592604332","heartbeat_front_addr":"172.21.15.90:6831/2592604332","state":["exists","up"]},{"osd":10,"uuid":"9240df75-e7e2-47f1-8aae-534b70b3c210","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":1594153142},{"type":"v1","addr":"172.21.15.92:6825","nonce":1594153142}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6826","nonce":1594153142},{"type":"v1","addr":"172.21.15.92:6827","nonce":1594153142}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":1594153142},{"type":"v1","addr":"172.21.15.92:6831","nonce":1594153142}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":1594153142},{"type":"v1","addr":"172.21.15.92:6829","nonce":1594153142}]},"public_addr":"172.21.15.92:6825/1594153142","cluster_addr":"172.21.15.92:6827/1594153142","heartbeat_back_addr":"172.21.15.92:6831/1594153142","heartbeat_front_addr":"172.21.15.92:6829/1594153142","state":["exists","up"]},{"osd":11,"uuid":"83cba9fa-b745-4df1-adda-e86837d4b91a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":83,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1622582126},{"type":"v1","addr":"172.21.15.177:6825","nonce":1622582126}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":1622582126},{"type":"v1","addr":"172.21.15.177:6827","nonce":1622582126}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":1622582126},{"type":"v1","addr":"172.21.15.177:6831","nonce":1622582126}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1622582126},{"type":"v1","addr":"172.21.15.177:6829","nonce":1622582126}]},"public_addr":"172.21.15.177:6825/1622582126","cluster_addr":"172.21.15.177:6827/1622582126","heartbeat_back_addr":"172.21.15.177:6831/1622582126","heartbeat_front_addr":"172.21.15.177:6829/1622582126","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:26:24.290365+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:26:49.405988+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:27:13.639196+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:27:45.603746+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:28:11.019126+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:28:35.084163+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:29:08.064626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:29:33.266133+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:29:57.381816+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:30:30.286847+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:30:56.226279+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:31:20.809611+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.90:0/314641601":"2024-04-03T14:24:34.595561+0000","172.21.15.90:0/1400895868":"2024-04-03T14:24:34.595561+0000","172.21.15.90:6801/129436436":"2024-04-03T14:24:34.595561+0000","172.21.15.90:6801/2372398598":"2024-04-03T14:24:00.362147+0000","172.21.15.90:0/278569338":"2024-04-03T14:24:34.595561+0000","172.21.15.90:0/2321464594":"2024-04-03T14:24:00.362147+0000","172.21.15.90:6800/2372398598":"2024-04-03T14:24:00.362147+0000","172.21.15.90:0/2962656672":"2024-04-03T14:24:00.362147+0000","172.21.15.90:0/126693610":"2024-04-03T14:24:00.362147+0000","172.21.15.90:6800/129436436":"2024-04-03T14:24:34.595561+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T14:31:39.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:39.381+0000 7f8203000640 1 -- 172.21.15.90:0/108967349 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f81e00a3c30 msgr2=0x7f81e00a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:31:39.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:39.381+0000 7f8203000640 1 --2- 172.21.15.90:0/108967349 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f81e00a3c30 0x7f81e00a60f0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f81e8006010 tx=0x7f81e8000f40 comp rx=0 tx=0).stop 2024-04-02T14:31:39.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:39.381+0000 7f8203000640 1 -- 172.21.15.90:0/108967349 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f81fc107f10 msgr2=0x7f81fc19e250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:31:39.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:39.381+0000 7f8203000640 1 --2- 172.21.15.90:0/108967349 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f81fc107f10 0x7f81fc19e250 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f81ec00e9c0 tx=0x7f81ec00edc0 comp rx=0 tx=0).stop 2024-04-02T14:31:39.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:39.381+0000 7f8203000640 1 -- 172.21.15.90:0/108967349 shutdown_connections 2024-04-02T14:31:39.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:39.381+0000 7f8203000640 1 --2- 172.21.15.90:0/108967349 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f81e00a3c30 0x7f81e00a60f0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:31:39.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:39.381+0000 7f8203000640 1 --2- 172.21.15.90:0/108967349 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f81fc109aa0 0x7f81fc1a5800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:31:39.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:39.381+0000 7f8203000640 1 --2- 172.21.15.90:0/108967349 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f81fc1090e0 0x7f81fc19e790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:31:39.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:39.381+0000 7f8203000640 1 --2- 172.21.15.90:0/108967349 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f81fc107f10 0x7f81fc19e250 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:31:39.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:39.381+0000 7f8203000640 1 -- 172.21.15.90:0/108967349 >> 172.21.15.90:0/108967349 conn(0x7f81fc076f10 msgr2=0x7f81fc10c470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:31:39.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:39.381+0000 7f8203000640 1 -- 172.21.15.90:0/108967349 shutdown_connections 2024-04-02T14:31:39.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:39.382+0000 7f8203000640 1 -- 172.21.15.90:0/108967349 wait complete. 2024-04-02T14:31:39.404 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-02T14:27:16.758622+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-02T14:31:39.404 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-02T14:31:39.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:39.613+0000 7f9d15400640 1 -- 172.21.15.90:0/328563452 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9d10103d20 msgr2=0x7f9d10104140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:31:39.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:39.613+0000 7f9d15400640 1 --2- 172.21.15.90:0/328563452 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9d10103d20 0x7f9d10104140 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f9d00009900 tx=0x7f9d0002bc30 comp rx=0 tx=0).stop 2024-04-02T14:31:39.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:39.613+0000 7f9d15400640 1 -- 172.21.15.90:0/328563452 shutdown_connections 2024-04-02T14:31:39.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:39.613+0000 7f9d15400640 1 --2- 172.21.15.90:0/328563452 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9d101058d0 0x7f9d1010c160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:31:39.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:39.613+0000 7f9d15400640 1 --2- 172.21.15.90:0/328563452 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9d10104f10 0x7f9d10105390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:31:39.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:39.613+0000 7f9d15400640 1 --2- 172.21.15.90:0/328563452 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9d10103d20 0x7f9d10104140 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:31:39.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:39.613+0000 7f9d15400640 1 -- 172.21.15.90:0/328563452 >> 172.21.15.90:0/328563452 conn(0x7f9d100ff2b0 msgr2=0x7f9d101016d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:31:39.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:39.613+0000 7f9d15400640 1 -- 172.21.15.90:0/328563452 shutdown_connections 2024-04-02T14:31:39.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:39.613+0000 7f9d15400640 1 -- 172.21.15.90:0/328563452 wait complete. 2024-04-02T14:31:39.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:39.614+0000 7f9d15400640 1 Processor -- start 2024-04-02T14:31:39.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:39.614+0000 7f9d15400640 1 -- start start 2024-04-02T14:31:39.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:39.614+0000 7f9d15400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9d10103d20 0x7f9d1019a0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:31:39.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:39.614+0000 7f9d15400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9d10104f10 0x7f9d1019a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:31:39.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:39.614+0000 7f9d15400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9d101058d0 0x7f9d101a15b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:31:39.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:39.614+0000 7f9d15400640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f9d1010f970 con 0x7f9d101058d0 2024-04-02T14:31:39.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:39.614+0000 7f9d15400640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f9d1010f7f0 con 0x7f9d10103d20 2024-04-02T14:31:39.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:39.614+0000 7f9d15400640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f9d1010faf0 con 0x7f9d10104f10 2024-04-02T14:31:39.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:39.614+0000 7f9d0e800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9d10103d20 0x7f9d1019a0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:31:39.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:39.614+0000 7f9d0f200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9d101058d0 0x7f9d101a15b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:31:39.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:39.615+0000 7f9d0e800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9d10103d20 0x7f9d1019a0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:35512/0 (socket says 172.21.15.90:35512) 2024-04-02T14:31:39.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:39.615+0000 7f9d0e800640 1 -- 172.21.15.90:0/2645583297 learned_addr learned my addr 172.21.15.90:0/2645583297 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:31:39.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:39.615+0000 7f9d0de00640 1 --2- 172.21.15.90:0/2645583297 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9d10104f10 0x7f9d1019a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:31:39.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:39.615+0000 7f9d0e800640 1 -- 172.21.15.90:0/2645583297 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9d10104f10 msgr2=0x7f9d1019a5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:31:39.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:39.615+0000 7f9d0e800640 1 --2- 172.21.15.90:0/2645583297 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9d10104f10 0x7f9d1019a5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:31:39.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:39.615+0000 7f9d0e800640 1 -- 172.21.15.90:0/2645583297 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9d101058d0 msgr2=0x7f9d101a15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:31:39.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:39.615+0000 7f9d0e800640 1 --2- 172.21.15.90:0/2645583297 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9d101058d0 0x7f9d101a15b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:31:39.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:39.615+0000 7f9d0e800640 1 -- 172.21.15.90:0/2645583297 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d101a1c20 con 0x7f9d10103d20 2024-04-02T14:31:39.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:39.615+0000 7f9d0f200640 1 --2- 172.21.15.90:0/2645583297 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9d101058d0 0x7f9d101a15b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:31:39.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:39.615+0000 7f9d0e800640 1 --2- 172.21.15.90:0/2645583297 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9d10103d20 0x7f9d1019a0a0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f9d00000c00 tx=0x7f9d0002c640 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:31:39.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:39.615+0000 7f9d07400640 1 -- 172.21.15.90:0/2645583297 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d00041070 con 0x7f9d10103d20 2024-04-02T14:31:39.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:39.615+0000 7f9d07400640 1 -- 172.21.15.90:0/2645583297 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9d00004450 con 0x7f9d10103d20 2024-04-02T14:31:39.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:39.615+0000 7f9d15400640 1 -- 172.21.15.90:0/2645583297 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9d101a1eb0 con 0x7f9d10103d20 2024-04-02T14:31:39.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:39.615+0000 7f9d07400640 1 -- 172.21.15.90:0/2645583297 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d00034440 con 0x7f9d10103d20 2024-04-02T14:31:39.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:39.616+0000 7f9d15400640 1 -- 172.21.15.90:0/2645583297 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9d101a22c0 con 0x7f9d10103d20 2024-04-02T14:31:39.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:39.617+0000 7f9d15400640 1 -- 172.21.15.90:0/2645583297 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9d1010b040 con 0x7f9d10103d20 2024-04-02T14:31:39.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:39.618+0000 7f9d07400640 1 -- 172.21.15.90:0/2645583297 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f9d00044070 con 0x7f9d10103d20 2024-04-02T14:31:39.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:39.620+0000 7f9d07400640 1 --2- 172.21.15.90:0/2645583297 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f9cf40a3bb0 0x7f9cf40a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:31:39.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:39.620+0000 7f9d07400640 1 -- 172.21.15.90:0/2645583297 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 7127+0+0 (secure 0 0 0) 0x7f9d000f1ab0 con 0x7f9d10103d20 2024-04-02T14:31:39.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:39.620+0000 7f9d0de00640 1 --2- 172.21.15.90:0/2645583297 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f9cf40a3bb0 0x7f9cf40a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:31:39.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:39.620+0000 7f9d0de00640 1 --2- 172.21.15.90:0/2645583297 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f9cf40a3bb0 0x7f9cf40a6070 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f9d1019b510 tx=0x7f9cfc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:31:39.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:39.624+0000 7f9d07400640 1 -- 172.21.15.90:0/2645583297 <== mon.2 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7f9d000be2d0 con 0x7f9d10103d20 2024-04-02T14:31:39.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:39.826+0000 7f9d15400640 1 -- 172.21.15.90:0/2645583297 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f9d10062850 con 0x7f9d10103d20 2024-04-02T14:31:39.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:39.827+0000 7f9d07400640 1 -- 172.21.15.90:0/2645583297 <== mon.2 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v84) ==== 93+0+10 (secure 0 0 0) 0x7f9d000c25d0 con 0x7f9d10103d20 2024-04-02T14:31:39.829 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 1 2024-04-02T14:31:39.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:39.829+0000 7f9d15400640 1 -- 172.21.15.90:0/2645583297 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f9cf40a3bb0 msgr2=0x7f9cf40a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:31:39.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:39.829+0000 7f9d15400640 1 --2- 172.21.15.90:0/2645583297 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f9cf40a3bb0 0x7f9cf40a6070 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f9d1019b510 tx=0x7f9cfc005f90 comp rx=0 tx=0).stop 2024-04-02T14:31:39.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:39.829+0000 7f9d15400640 1 -- 172.21.15.90:0/2645583297 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9d10103d20 msgr2=0x7f9d1019a0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:31:39.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:39.829+0000 7f9d15400640 1 --2- 172.21.15.90:0/2645583297 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9d10103d20 0x7f9d1019a0a0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f9d00000c00 tx=0x7f9d0002c640 comp rx=0 tx=0).stop 2024-04-02T14:31:39.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:39.829+0000 7f9d15400640 1 -- 172.21.15.90:0/2645583297 shutdown_connections 2024-04-02T14:31:39.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:39.829+0000 7f9d15400640 1 --2- 172.21.15.90:0/2645583297 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f9cf40a3bb0 0x7f9cf40a6070 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:31:39.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:39.829+0000 7f9d15400640 1 --2- 172.21.15.90:0/2645583297 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9d101058d0 0x7f9d101a15b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:31:39.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:39.829+0000 7f9d15400640 1 --2- 172.21.15.90:0/2645583297 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9d10104f10 0x7f9d1019a5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:31:39.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:39.829+0000 7f9d15400640 1 --2- 172.21.15.90:0/2645583297 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9d10103d20 0x7f9d1019a0a0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:31:39.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:39.829+0000 7f9d15400640 1 -- 172.21.15.90:0/2645583297 >> 172.21.15.90:0/2645583297 conn(0x7f9d100ff2b0 msgr2=0x7f9d10108210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:31:39.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:39.830+0000 7f9d15400640 1 -- 172.21.15.90:0/2645583297 shutdown_connections 2024-04-02T14:31:39.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:39.830+0000 7f9d15400640 1 -- 172.21.15.90:0/2645583297 wait complete. 2024-04-02T14:31:39.851 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T14:31:40.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:40.039+0000 7f18a8200640 1 -- 172.21.15.90:0/1951478597 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f18a0104f30 msgr2=0x7f18a01053b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:31:40.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:40.039+0000 7f18a8200640 1 --2- 172.21.15.90:0/1951478597 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f18a0104f30 0x7f18a01053b0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f188c009900 tx=0x7f188c02bc30 comp rx=0 tx=0).stop 2024-04-02T14:31:40.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:40.039+0000 7f18a8200640 1 -- 172.21.15.90:0/1951478597 shutdown_connections 2024-04-02T14:31:40.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:40.039+0000 7f18a8200640 1 --2- 172.21.15.90:0/1951478597 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f18a01058f0 0x7f18a010c180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:31:40.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:40.039+0000 7f18a8200640 1 --2- 172.21.15.90:0/1951478597 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f18a0104f30 0x7f18a01053b0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:31:40.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:40.039+0000 7f18a8200640 1 --2- 172.21.15.90:0/1951478597 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f18a0103d40 0x7f18a0104160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:31:40.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:40.039+0000 7f18a8200640 1 -- 172.21.15.90:0/1951478597 >> 172.21.15.90:0/1951478597 conn(0x7f18a00ff2f0 msgr2=0x7f18a0101710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:31:40.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:40.039+0000 7f18a8200640 1 -- 172.21.15.90:0/1951478597 shutdown_connections 2024-04-02T14:31:40.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:40.039+0000 7f18a8200640 1 -- 172.21.15.90:0/1951478597 wait complete. 2024-04-02T14:31:40.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:40.040+0000 7f18a8200640 1 Processor -- start 2024-04-02T14:31:40.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:40.041+0000 7f18a8200640 1 -- start start 2024-04-02T14:31:40.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:40.041+0000 7f18a8200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f18a0103d40 0x7f18a019a000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:31:40.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:40.041+0000 7f18a8200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f18a0104f30 0x7f18a019a540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:31:40.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:40.041+0000 7f18a8200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f18a01058f0 0x7f18a01a15b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:31:40.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:40.041+0000 7f18a8200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f18a010f9f0 con 0x7f18a01058f0 2024-04-02T14:31:40.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:40.041+0000 7f18a8200640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f18a010f870 con 0x7f18a0104f30 2024-04-02T14:31:40.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:40.041+0000 7f18a8200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f18a010fb70 con 0x7f18a0103d40 2024-04-02T14:31:40.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:40.041+0000 7f18a5800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f18a0103d40 0x7f18a019a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:31:40.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:40.041+0000 7f18a5800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f18a0103d40 0x7f18a019a000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:51780/0 (socket says 172.21.15.90:51780) 2024-04-02T14:31:40.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:40.041+0000 7f18a5800640 1 -- 172.21.15.90:0/3814103704 learned_addr learned my addr 172.21.15.90:0/3814103704 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:31:40.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:40.041+0000 7f18a6200640 1 --2- 172.21.15.90:0/3814103704 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f18a01058f0 0x7f18a01a15b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:31:40.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:40.041+0000 7f18a4e00640 1 --2- 172.21.15.90:0/3814103704 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f18a0104f30 0x7f18a019a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:31:40.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:40.041+0000 7f18a5800640 1 -- 172.21.15.90:0/3814103704 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f18a0104f30 msgr2=0x7f18a019a540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:31:40.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:40.041+0000 7f18a5800640 1 --2- 172.21.15.90:0/3814103704 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f18a0104f30 0x7f18a019a540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:31:40.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:40.041+0000 7f18a5800640 1 -- 172.21.15.90:0/3814103704 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f18a01058f0 msgr2=0x7f18a01a15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:31:40.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:40.041+0000 7f18a5800640 1 --2- 172.21.15.90:0/3814103704 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f18a01058f0 0x7f18a01a15b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:31:40.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:40.042+0000 7f18a5800640 1 -- 172.21.15.90:0/3814103704 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18a01a1c20 con 0x7f18a0103d40 2024-04-02T14:31:40.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:40.042+0000 7f18a5800640 1 --2- 172.21.15.90:0/3814103704 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f18a0103d40 0x7f18a019a000 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f189400e9c0 tx=0x7f189400ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:31:40.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:40.042+0000 7f189e000640 1 -- 172.21.15.90:0/3814103704 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f189400cd80 con 0x7f18a0103d40 2024-04-02T14:31:40.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:40.042+0000 7f189e000640 1 -- 172.21.15.90:0/3814103704 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f18940035f0 con 0x7f18a0103d40 2024-04-02T14:31:40.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:40.042+0000 7f189e000640 1 -- 172.21.15.90:0/3814103704 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1894010690 con 0x7f18a0103d40 2024-04-02T14:31:40.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:40.042+0000 7f18a8200640 1 -- 172.21.15.90:0/3814103704 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f18a01a1f10 con 0x7f18a0103d40 2024-04-02T14:31:40.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:40.042+0000 7f18a8200640 1 -- 172.21.15.90:0/3814103704 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18a01a2380 con 0x7f18a0103d40 2024-04-02T14:31:40.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:40.043+0000 7f18a8200640 1 -- 172.21.15.90:0/3814103704 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f18a010b060 con 0x7f18a0103d40 2024-04-02T14:31:40.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:40.045+0000 7f189e000640 1 -- 172.21.15.90:0/3814103704 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f18940037a0 con 0x7f18a0103d40 2024-04-02T14:31:40.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:40.046+0000 7f189e000640 1 --2- 172.21.15.90:0/3814103704 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f18880a3c30 0x7f18880a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:31:40.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:40.046+0000 7f189e000640 1 -- 172.21.15.90:0/3814103704 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 7127+0+0 (secure 0 0 0) 0x7f18940d0340 con 0x7f18a0103d40 2024-04-02T14:31:40.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:40.046+0000 7f18a4e00640 1 --2- 172.21.15.90:0/3814103704 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f18880a3c30 0x7f18880a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:31:40.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:40.047+0000 7f18a4e00640 1 --2- 172.21.15.90:0/3814103704 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f18880a3c30 0x7f18880a60f0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f188c02c240 tx=0x7f188c02df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:31:40.052 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:40.050+0000 7f189e000640 1 -- 172.21.15.90:0/3814103704 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7f189409d120 con 0x7f18a0103d40 2024-04-02T14:31:40.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:31:39 smithi177 ceph-mon[36431]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-02T14:31:40.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:31:39 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:31:40.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:31:39 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T14:31:40.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:31:39 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:31:40.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:31:39 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-02T14:31:40.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:31:39 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-02T14:31:40.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:31:39 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:31:40.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:31:39 smithi177 ceph-mon[36431]: from='client.? 172.21.15.90:0/108967349' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-02T14:31:40.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:31:39 smithi177 ceph-mon[36431]: from='client.? 172.21.15.90:0/2645583297' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-02T14:31:40.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:31:39 smithi090 ceph-mon[31674]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-02T14:31:40.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:31:39 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:31:40.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:31:39 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T14:31:40.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:31:39 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:31:40.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:31:39 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-02T14:31:40.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:31:39 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-02T14:31:40.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:31:39 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:31:40.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:31:39 smithi090 ceph-mon[31674]: from='client.? 172.21.15.90:0/108967349' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-02T14:31:40.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:31:39 smithi090 ceph-mon[31674]: from='client.? 172.21.15.90:0/2645583297' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-02T14:31:40.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:40.290+0000 7f18a8200640 1 -- 172.21.15.90:0/3814103704 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f18a002d370 con 0x7f18a0103d40 2024-04-02T14:31:40.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:40.291+0000 7f189e000640 1 -- 172.21.15.90:0/3814103704 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 1 v1) ==== 93+0+569 (secure 0 0 0) 0x7f18a002d370 con 0x7f18a0103d40 2024-04-02T14:31:40.293 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-02T14:31:40.293 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":1,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[]} 2024-04-02T14:31:40.294 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 1 2024-04-02T14:31:40.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:40.293+0000 7f18a8200640 1 -- 172.21.15.90:0/3814103704 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f18880a3c30 msgr2=0x7f18880a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:31:40.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:40.293+0000 7f18a8200640 1 --2- 172.21.15.90:0/3814103704 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f18880a3c30 0x7f18880a60f0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f188c02c240 tx=0x7f188c02df90 comp rx=0 tx=0).stop 2024-04-02T14:31:40.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:40.293+0000 7f18a8200640 1 -- 172.21.15.90:0/3814103704 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f18a0103d40 msgr2=0x7f18a019a000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:31:40.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:40.294+0000 7f18a8200640 1 --2- 172.21.15.90:0/3814103704 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f18a0103d40 0x7f18a019a000 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f189400e9c0 tx=0x7f189400ee80 comp rx=0 tx=0).stop 2024-04-02T14:31:40.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:40.294+0000 7f18a8200640 1 -- 172.21.15.90:0/3814103704 shutdown_connections 2024-04-02T14:31:40.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:40.294+0000 7f18a8200640 1 --2- 172.21.15.90:0/3814103704 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f18880a3c30 0x7f18880a60f0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:31:40.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:40.294+0000 7f18a8200640 1 --2- 172.21.15.90:0/3814103704 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f18a01058f0 0x7f18a01a15b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:31:40.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:40.294+0000 7f18a8200640 1 --2- 172.21.15.90:0/3814103704 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f18a0104f30 0x7f18a019a540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:31:40.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:40.294+0000 7f18a8200640 1 --2- 172.21.15.90:0/3814103704 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f18a0103d40 0x7f18a019a000 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:31:40.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:40.294+0000 7f18a8200640 1 -- 172.21.15.90:0/3814103704 >> 172.21.15.90:0/3814103704 conn(0x7f18a00ff2f0 msgr2=0x7f18a0108230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:31:40.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:40.294+0000 7f18a8200640 1 -- 172.21.15.90:0/3814103704 shutdown_connections 2024-04-02T14:31:40.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:40.294+0000 7f18a8200640 1 -- 172.21.15.90:0/3814103704 wait complete. 2024-04-02T14:31:40.360 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:31:39 smithi092 ceph-mon[36497]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-02T14:31:40.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:31:39 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:31:40.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:31:39 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T14:31:40.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:31:39 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:31:40.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:31:39 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-02T14:31:40.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:31:39 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-02T14:31:40.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:31:39 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:31:40.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:31:39 smithi092 ceph-mon[36497]: from='client.? 172.21.15.90:0/108967349' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-02T14:31:40.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:31:39 smithi092 ceph-mon[36497]: from='client.? 172.21.15.90:0/2645583297' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-02T14:31:41.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:31:40 smithi177 ceph-mon[36431]: Deploying daemon mds.l on smithi177 2024-04-02T14:31:41.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:31:40 smithi177 ceph-mon[36431]: from='client.? 172.21.15.90:0/3814103704' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-02T14:31:41.187 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 02 14:31:41 smithi177 systemd[1]: Starting Ceph mds.l for 6449448a-f0fc-11ee-b647-cb9ed24678a4... 2024-04-02T14:31:41.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:31:40 smithi090 ceph-mon[31674]: Deploying daemon mds.l on smithi177 2024-04-02T14:31:41.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:31:40 smithi090 ceph-mon[31674]: from='client.? 172.21.15.90:0/3814103704' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-02T14:31:41.360 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:31:40 smithi092 ceph-mon[36497]: Deploying daemon mds.l on smithi177 2024-04-02T14:31:41.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:31:40 smithi092 ceph-mon[36497]: from='client.? 172.21.15.90:0/3814103704' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-02T14:31:41.590 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 02 14:31:41 smithi177 podman[65741]: 2024-04-02 14:31:41.204892414 +0000 UTC m=+0.017708386 image pull f7301e070545fc23d8e5b7b052a2c15e832a0e57102f1acad8e4f3583709d23f quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534 2024-04-02T14:31:41.590 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 02 14:31:41 smithi177 podman[65741]: 2024-04-02 14:31:41.361462949 +0000 UTC m=+0.174278893 container create 4bd8c1a305ce27556da2de18974dfa1b668caa4851466acf21193c539d78da79 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mds-l, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, org.label-schema.build-date=20240326, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240402.004512-191d963, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-04-02T14:31:41.590 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 02 14:31:41 smithi177 podman[65741]: 2024-04-02 14:31:41.58920253 +0000 UTC m=+0.402018482 container init 4bd8c1a305ce27556da2de18974dfa1b668caa4851466acf21193c539d78da79 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mds-l, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240402.004512-191d963, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240326, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-04-02T14:31:41.899 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 02 14:31:41 smithi177 podman[65741]: 2024-04-02 14:31:41.592006441 +0000 UTC m=+0.404822408 container start 4bd8c1a305ce27556da2de18974dfa1b668caa4851466acf21193c539d78da79 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mds-l, org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=wip-batrick-testing-20240402.004512-191d963, org.label-schema.build-date=20240326, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-04-02T14:31:41.899 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 02 14:31:41 smithi177 ceph-mds[65760]: -- 172.21.15.177:0/2888714143 <== mon.1 v2:172.21.15.177:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5622d9601a40 con 0x5622d95d5000 2024-04-02T14:31:41.900 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 02 14:31:41 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mds-l[65756]: starting mds.l at 2024-04-02T14:31:41.900 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 02 14:31:41 smithi177 bash[65741]: 4bd8c1a305ce27556da2de18974dfa1b668caa4851466acf21193c539d78da79 2024-04-02T14:31:41.900 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 02 14:31:41 smithi177 systemd[1]: Started Ceph mds.l for 6449448a-f0fc-11ee-b647-cb9ed24678a4. 2024-04-02T14:31:42.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:31:41 smithi177 ceph-mon[36431]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-02T14:31:42.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:31:41 smithi177 ceph-mon[36431]: mds.? [v2:172.21.15.177:6832/1564319431,v1:172.21.15.177:6833/1564319431] up:boot 2024-04-02T14:31:42.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:31:41 smithi177 ceph-mon[36431]: fsmap 1 up:standby 2024-04-02T14:31:42.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:31:41 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-02T14:31:42.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:31:41 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:31:42.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:31:41 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:31:42.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:31:41 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:31:42.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:31:41 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-02T14:31:42.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:31:41 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-02T14:31:42.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:31:41 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:31:42.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:31:41 smithi090 ceph-mon[31674]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-02T14:31:42.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:31:41 smithi090 ceph-mon[31674]: mds.? [v2:172.21.15.177:6832/1564319431,v1:172.21.15.177:6833/1564319431] up:boot 2024-04-02T14:31:42.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:31:41 smithi090 ceph-mon[31674]: fsmap 1 up:standby 2024-04-02T14:31:42.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:31:41 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-02T14:31:42.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:31:41 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:31:42.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:31:41 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:31:42.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:31:41 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:31:42.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:31:41 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-02T14:31:42.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:31:41 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-02T14:31:42.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:31:41 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:31:42.314 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T14:31:42.360 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:31:41 smithi092 ceph-mon[36497]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-02T14:31:42.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:31:41 smithi092 ceph-mon[36497]: mds.? [v2:172.21.15.177:6832/1564319431,v1:172.21.15.177:6833/1564319431] up:boot 2024-04-02T14:31:42.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:31:41 smithi092 ceph-mon[36497]: fsmap 1 up:standby 2024-04-02T14:31:42.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:31:41 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-02T14:31:42.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:31:41 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:31:42.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:31:41 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:31:42.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:31:41 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:31:42.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:31:41 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-02T14:31:42.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:31:41 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-02T14:31:42.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:31:41 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:31:42.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:42.531+0000 7f13f1200640 1 -- 172.21.15.90:0/2516937638 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f13ec077700 msgr2=0x7f13ec077b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:31:42.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:42.531+0000 7f13f1200640 1 --2- 172.21.15.90:0/2516937638 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f13ec077700 0x7f13ec077b60 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f13d8009860 tx=0x7f13d802bb90 comp rx=0 tx=0).stop 2024-04-02T14:31:42.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:42.531+0000 7f13f1200640 1 -- 172.21.15.90:0/2516937638 shutdown_connections 2024-04-02T14:31:42.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:42.531+0000 7f13f1200640 1 --2- 172.21.15.90:0/2516937638 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f13ec0780a0 0x7f13ec078940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:31:42.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:42.531+0000 7f13f1200640 1 --2- 172.21.15.90:0/2516937638 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f13ec077700 0x7f13ec077b60 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:31:42.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:42.531+0000 7f13f1200640 1 --2- 172.21.15.90:0/2516937638 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f13ec0764c0 0x7f13ec0768e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:31:42.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:42.531+0000 7f13f1200640 1 -- 172.21.15.90:0/2516937638 >> 172.21.15.90:0/2516937638 conn(0x7f13ec0ff2b0 msgr2=0x7f13ec1016f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:31:42.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:42.531+0000 7f13f1200640 1 -- 172.21.15.90:0/2516937638 shutdown_connections 2024-04-02T14:31:42.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:42.531+0000 7f13f1200640 1 -- 172.21.15.90:0/2516937638 wait complete. 2024-04-02T14:31:42.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:42.532+0000 7f13f1200640 1 Processor -- start 2024-04-02T14:31:42.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:42.532+0000 7f13f1200640 1 -- start start 2024-04-02T14:31:42.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:42.533+0000 7f13f1200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f13ec0764c0 0x7f13ec19e430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:31:42.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:42.533+0000 7f13f1200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f13ec077700 0x7f13ec19e970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:31:42.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:42.533+0000 7f13f1200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f13ec0780a0 0x7f13ec1a59e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:31:42.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:42.533+0000 7f13f1200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f13ec113dd0 con 0x7f13ec0764c0 2024-04-02T14:31:42.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:42.533+0000 7f13f1200640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f13ec113c50 con 0x7f13ec0780a0 2024-04-02T14:31:42.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:42.533+0000 7f13f1200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f13ec113f50 con 0x7f13ec077700 2024-04-02T14:31:42.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:42.533+0000 7f13e9e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f13ec0764c0 0x7f13ec19e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:31:42.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:42.533+0000 7f13e9e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f13ec0764c0 0x7f13ec19e430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:47860/0 (socket says 172.21.15.90:47860) 2024-04-02T14:31:42.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:42.533+0000 7f13e9e00640 1 -- 172.21.15.90:0/2850400993 learned_addr learned my addr 172.21.15.90:0/2850400993 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:31:42.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:42.533+0000 7f13e9400640 1 --2- 172.21.15.90:0/2850400993 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f13ec077700 0x7f13ec19e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:31:42.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:42.533+0000 7f13ea800640 1 --2- 172.21.15.90:0/2850400993 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f13ec0780a0 0x7f13ec1a59e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:31:42.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:42.533+0000 7f13e9e00640 1 -- 172.21.15.90:0/2850400993 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f13ec077700 msgr2=0x7f13ec19e970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:31:42.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:42.533+0000 7f13e9e00640 1 --2- 172.21.15.90:0/2850400993 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f13ec077700 0x7f13ec19e970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:31:42.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:42.533+0000 7f13e9e00640 1 -- 172.21.15.90:0/2850400993 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f13ec0780a0 msgr2=0x7f13ec1a59e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:31:42.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:42.533+0000 7f13e9e00640 1 --2- 172.21.15.90:0/2850400993 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f13ec0780a0 0x7f13ec1a59e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:31:42.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:42.533+0000 7f13e9e00640 1 -- 172.21.15.90:0/2850400993 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13ec1a6050 con 0x7f13ec0764c0 2024-04-02T14:31:42.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:42.533+0000 7f13ea800640 1 --2- 172.21.15.90:0/2850400993 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f13ec0780a0 0x7f13ec1a59e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:31:42.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:42.534+0000 7f13e9e00640 1 --2- 172.21.15.90:0/2850400993 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f13ec0764c0 0x7f13ec19e430 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f13dc00e9c0 tx=0x7f13dc00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:31:42.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:42.534+0000 7f13e2a00640 1 -- 172.21.15.90:0/2850400993 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13dc00cde0 con 0x7f13ec0764c0 2024-04-02T14:31:42.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:42.534+0000 7f13e2a00640 1 -- 172.21.15.90:0/2850400993 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f13dc0035f0 con 0x7f13ec0764c0 2024-04-02T14:31:42.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:42.534+0000 7f13e2a00640 1 -- 172.21.15.90:0/2850400993 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13dc010690 con 0x7f13ec0764c0 2024-04-02T14:31:42.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:42.534+0000 7f13f1200640 1 -- 172.21.15.90:0/2850400993 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13ec1a6340 con 0x7f13ec0764c0 2024-04-02T14:31:42.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:42.534+0000 7f13f1200640 1 -- 172.21.15.90:0/2850400993 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13ec1a6810 con 0x7f13ec0764c0 2024-04-02T14:31:42.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:42.535+0000 7f13f1200640 1 -- 172.21.15.90:0/2850400993 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f13ec110860 con 0x7f13ec0764c0 2024-04-02T14:31:42.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:42.537+0000 7f13e2a00640 1 -- 172.21.15.90:0/2850400993 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f13dc0037a0 con 0x7f13ec0764c0 2024-04-02T14:31:42.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:42.539+0000 7f13e2a00640 1 --2- 172.21.15.90:0/2850400993 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f13d00a3c30 0x7f13d00a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:31:42.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:42.539+0000 7f13e2a00640 1 -- 172.21.15.90:0/2850400993 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 7127+0+0 (secure 0 0 0) 0x7f13dc0d0320 con 0x7f13ec0764c0 2024-04-02T14:31:42.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:42.540+0000 7f13e9400640 1 --2- 172.21.15.90:0/2850400993 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f13d00a3c30 0x7f13d00a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:31:42.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:42.540+0000 7f13e9400640 1 --2- 172.21.15.90:0/2850400993 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f13d00a3c30 0x7f13d00a60f0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f13d802c1a0 tx=0x7f13d8034ad0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:31:42.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:42.544+0000 7f13e2a00640 1 -- 172.21.15.90:0/2850400993 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7f13dc09d100 con 0x7f13ec0764c0 2024-04-02T14:31:42.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:42.784+0000 7f13f1200640 1 -- 172.21.15.90:0/2850400993 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f13ec02d370 con 0x7f13ec0764c0 2024-04-02T14:31:42.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:42.785+0000 7f13e2a00640 1 -- 172.21.15.90:0/2850400993 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1396 (secure 0 0 0) 0x7f13dc09ca00 con 0x7f13ec0764c0 2024-04-02T14:31:42.787 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-02T14:31:42.787 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24454,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/1564319431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":1564319431},{"type":"v1","addr":"172.21.15.177:6833","nonce":1564319431}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-04-02T14:31:42.788 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 2 2024-04-02T14:31:42.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:42.788+0000 7f13f1200640 1 -- 172.21.15.90:0/2850400993 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f13d00a3c30 msgr2=0x7f13d00a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:31:42.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:42.788+0000 7f13f1200640 1 --2- 172.21.15.90:0/2850400993 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f13d00a3c30 0x7f13d00a60f0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f13d802c1a0 tx=0x7f13d8034ad0 comp rx=0 tx=0).stop 2024-04-02T14:31:42.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:42.788+0000 7f13f1200640 1 -- 172.21.15.90:0/2850400993 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f13ec0764c0 msgr2=0x7f13ec19e430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:31:42.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:42.788+0000 7f13f1200640 1 --2- 172.21.15.90:0/2850400993 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f13ec0764c0 0x7f13ec19e430 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f13dc00e9c0 tx=0x7f13dc00ee80 comp rx=0 tx=0).stop 2024-04-02T14:31:42.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:42.788+0000 7f13f1200640 1 -- 172.21.15.90:0/2850400993 shutdown_connections 2024-04-02T14:31:42.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:42.788+0000 7f13f1200640 1 --2- 172.21.15.90:0/2850400993 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f13d00a3c30 0x7f13d00a60f0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:31:42.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:42.788+0000 7f13f1200640 1 --2- 172.21.15.90:0/2850400993 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f13ec0780a0 0x7f13ec1a59e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:31:42.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:42.788+0000 7f13f1200640 1 --2- 172.21.15.90:0/2850400993 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f13ec077700 0x7f13ec19e970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:31:42.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:42.788+0000 7f13f1200640 1 --2- 172.21.15.90:0/2850400993 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f13ec0764c0 0x7f13ec19e430 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:31:42.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:42.788+0000 7f13f1200640 1 -- 172.21.15.90:0/2850400993 >> 172.21.15.90:0/2850400993 conn(0x7f13ec0ff2b0 msgr2=0x7f13ec1009b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:31:42.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:42.788+0000 7f13f1200640 1 -- 172.21.15.90:0/2850400993 shutdown_connections 2024-04-02T14:31:42.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:42.788+0000 7f13f1200640 1 -- 172.21.15.90:0/2850400993 wait complete. 2024-04-02T14:31:43.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:31:42 smithi092 ceph-mon[36497]: Deploying daemon mds.k on smithi092 2024-04-02T14:31:43.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:31:42 smithi092 ceph-mon[36497]: from='client.? 172.21.15.90:0/2850400993' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-02T14:31:43.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:31:42 smithi177 ceph-mon[36431]: Deploying daemon mds.k on smithi092 2024-04-02T14:31:43.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:31:42 smithi177 ceph-mon[36431]: from='client.? 172.21.15.90:0/2850400993' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-02T14:31:43.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:31:42 smithi090 ceph-mon[31674]: Deploying daemon mds.k on smithi092 2024-04-02T14:31:43.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:31:42 smithi090 ceph-mon[31674]: from='client.? 172.21.15.90:0/2850400993' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-02T14:31:44.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:31:43 smithi092 ceph-mon[36497]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-02T14:31:44.111 INFO:journalctl@ceph.mds.k.smithi092.stdout:Apr 02 14:31:43 smithi092 systemd[1]: Starting Ceph mds.k for 6449448a-f0fc-11ee-b647-cb9ed24678a4... 2024-04-02T14:31:44.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:31:43 smithi177 ceph-mon[36431]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-02T14:31:44.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:31:43 smithi090 ceph-mon[31674]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-02T14:31:44.490 INFO:journalctl@ceph.mds.k.smithi092.stdout:Apr 02 14:31:44 smithi092 podman[66451]: 2024-04-02 14:31:44.094316306 +0000 UTC m=+0.017321955 image pull f7301e070545fc23d8e5b7b052a2c15e832a0e57102f1acad8e4f3583709d23f quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534 2024-04-02T14:31:44.490 INFO:journalctl@ceph.mds.k.smithi092.stdout:Apr 02 14:31:44 smithi092 podman[66451]: 2024-04-02 14:31:44.194934935 +0000 UTC m=+0.117940559 container create e4972a161fe41e632a181beea3c83297e14ab752718bd75694072d3b571eb5b1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mds-k, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240402.004512-191d963, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240326, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5) 2024-04-02T14:31:44.491 INFO:journalctl@ceph.mds.k.smithi092.stdout:Apr 02 14:31:44 smithi092 podman[66451]: 2024-04-02 14:31:44.413694052 +0000 UTC m=+0.336699688 container init e4972a161fe41e632a181beea3c83297e14ab752718bd75694072d3b571eb5b1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mds-k, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240326, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240402.004512-191d963, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_CLEAN=True) 2024-04-02T14:31:44.491 INFO:journalctl@ceph.mds.k.smithi092.stdout:Apr 02 14:31:44 smithi092 podman[66451]: 2024-04-02 14:31:44.416238986 +0000 UTC m=+0.339244620 container start e4972a161fe41e632a181beea3c83297e14ab752718bd75694072d3b571eb5b1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mds-k, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240402.004512-191d963, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240326, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-02T14:31:44.491 INFO:journalctl@ceph.mds.k.smithi092.stdout:Apr 02 14:31:44 smithi092 ceph-mds[66470]: -- 172.21.15.92:0/833080570 <== mon.2 v2:172.21.15.92:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55fb741f1a40 con 0x55fb741c4c00 2024-04-02T14:31:44.491 INFO:journalctl@ceph.mds.k.smithi092.stdout:Apr 02 14:31:44 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mds-k[66466]: starting mds.k at 2024-04-02T14:31:44.491 INFO:journalctl@ceph.mds.k.smithi092.stdout:Apr 02 14:31:44 smithi092 bash[66451]: e4972a161fe41e632a181beea3c83297e14ab752718bd75694072d3b571eb5b1 2024-04-02T14:31:44.810 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T14:31:44.861 INFO:journalctl@ceph.mds.k.smithi092.stdout:Apr 02 14:31:44 smithi092 systemd[1]: Started Ceph mds.k for 6449448a-f0fc-11ee-b647-cb9ed24678a4. 2024-04-02T14:31:45.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:45.002+0000 7fdf43400640 1 -- 172.21.15.90:0/779862143 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdf3c0764c0 msgr2=0x7fdf3c0768e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:31:45.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:45.002+0000 7fdf43400640 1 --2- 172.21.15.90:0/779862143 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdf3c0764c0 0x7fdf3c0768e0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fdf30009900 tx=0x7fdf3002bc40 comp rx=0 tx=0).stop 2024-04-02T14:31:45.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:45.002+0000 7fdf43400640 1 -- 172.21.15.90:0/779862143 shutdown_connections 2024-04-02T14:31:45.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:45.002+0000 7fdf43400640 1 --2- 172.21.15.90:0/779862143 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdf3c0780a0 0x7fdf3c078940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:31:45.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:45.002+0000 7fdf43400640 1 --2- 172.21.15.90:0/779862143 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdf3c077700 0x7fdf3c077b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:31:45.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:45.002+0000 7fdf43400640 1 --2- 172.21.15.90:0/779862143 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdf3c0764c0 0x7fdf3c0768e0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:31:45.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:45.002+0000 7fdf43400640 1 -- 172.21.15.90:0/779862143 >> 172.21.15.90:0/779862143 conn(0x7fdf3c0ff2b0 msgr2=0x7fdf3c1016f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:31:45.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:45.003+0000 7fdf43400640 1 -- 172.21.15.90:0/779862143 shutdown_connections 2024-04-02T14:31:45.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:45.003+0000 7fdf43400640 1 -- 172.21.15.90:0/779862143 wait complete. 2024-04-02T14:31:45.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:45.003+0000 7fdf43400640 1 Processor -- start 2024-04-02T14:31:45.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:45.004+0000 7fdf43400640 1 -- start start 2024-04-02T14:31:45.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:45.004+0000 7fdf43400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdf3c0764c0 0x7fdf3c19e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:31:45.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:45.004+0000 7fdf43400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdf3c077700 0x7fdf3c19e950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:31:45.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:45.004+0000 7fdf43400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdf3c0780a0 0x7fdf3c1a59c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:31:45.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:45.004+0000 7fdf43400640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fdf3c113dd0 con 0x7fdf3c0780a0 2024-04-02T14:31:45.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:45.004+0000 7fdf43400640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fdf3c113c50 con 0x7fdf3c0764c0 2024-04-02T14:31:45.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:45.004+0000 7fdf43400640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fdf3c113f50 con 0x7fdf3c077700 2024-04-02T14:31:45.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:45.004+0000 7fdf40a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdf3c0764c0 0x7fdf3c19e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:31:45.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:45.004+0000 7fdf3be00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdf3c077700 0x7fdf3c19e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:31:45.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:45.004+0000 7fdf41400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdf3c0780a0 0x7fdf3c1a59c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:31:45.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:45.004+0000 7fdf3be00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdf3c077700 0x7fdf3c19e950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:38506/0 (socket says 172.21.15.90:38506) 2024-04-02T14:31:45.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:45.004+0000 7fdf40a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdf3c0764c0 0x7fdf3c19e410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:59868/0 (socket says 172.21.15.90:59868) 2024-04-02T14:31:45.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:45.004+0000 7fdf3be00640 1 -- 172.21.15.90:0/1063314176 learned_addr learned my addr 172.21.15.90:0/1063314176 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:31:45.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:45.005+0000 7fdf3be00640 1 -- 172.21.15.90:0/1063314176 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdf3c0764c0 msgr2=0x7fdf3c19e410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:31:45.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:45.005+0000 7fdf3be00640 1 --2- 172.21.15.90:0/1063314176 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdf3c0764c0 0x7fdf3c19e410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:31:45.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:45.005+0000 7fdf3be00640 1 -- 172.21.15.90:0/1063314176 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdf3c0780a0 msgr2=0x7fdf3c1a59c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:31:45.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:45.005+0000 7fdf3be00640 1 --2- 172.21.15.90:0/1063314176 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdf3c0780a0 0x7fdf3c1a59c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:31:45.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:45.005+0000 7fdf3be00640 1 -- 172.21.15.90:0/1063314176 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf3c1a6030 con 0x7fdf3c077700 2024-04-02T14:31:45.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:45.005+0000 7fdf40a00640 1 --2- 172.21.15.90:0/1063314176 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdf3c0764c0 0x7fdf3c19e410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:31:45.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:45.005+0000 7fdf3be00640 1 --2- 172.21.15.90:0/1063314176 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdf3c077700 0x7fdf3c19e950 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fdf2800e9f0 tx=0x7fdf2800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:31:45.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:45.005+0000 7fdf39600640 1 -- 172.21.15.90:0/1063314176 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf2800cd80 con 0x7fdf3c077700 2024-04-02T14:31:45.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:45.005+0000 7fdf43400640 1 -- 172.21.15.90:0/1063314176 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf3c1a6320 con 0x7fdf3c077700 2024-04-02T14:31:45.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:45.005+0000 7fdf39600640 1 -- 172.21.15.90:0/1063314176 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdf28010910 con 0x7fdf3c077700 2024-04-02T14:31:45.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:45.005+0000 7fdf39600640 1 -- 172.21.15.90:0/1063314176 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf280188f0 con 0x7fdf3c077700 2024-04-02T14:31:45.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:45.006+0000 7fdf43400640 1 -- 172.21.15.90:0/1063314176 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf3c1a6870 con 0x7fdf3c077700 2024-04-02T14:31:45.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:45.007+0000 7fdf43400640 1 -- 172.21.15.90:0/1063314176 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdf3c110840 con 0x7fdf3c077700 2024-04-02T14:31:45.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:45.009+0000 7fdf39600640 1 -- 172.21.15.90:0/1063314176 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fdf28010ac0 con 0x7fdf3c077700 2024-04-02T14:31:45.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:45.010+0000 7fdf39600640 1 --2- 172.21.15.90:0/1063314176 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fdf240a3bb0 0x7fdf240a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:31:45.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:45.010+0000 7fdf39600640 1 -- 172.21.15.90:0/1063314176 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 7127+0+0 (secure 0 0 0) 0x7fdf280d0560 con 0x7fdf3c077700 2024-04-02T14:31:45.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:45.011+0000 7fdf40a00640 1 --2- 172.21.15.90:0/1063314176 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fdf240a3bb0 0x7fdf240a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:31:45.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:45.011+0000 7fdf40a00640 1 --2- 172.21.15.90:0/1063314176 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fdf240a3bb0 0x7fdf240a6070 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fdf3002c250 tx=0x7fdf3002cf90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:31:45.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:45.012+0000 7fdf39600640 1 -- 172.21.15.90:0/1063314176 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7fdf2809ccf0 con 0x7fdf3c077700 2024-04-02T14:31:45.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:45.259+0000 7fdf43400640 1 -- 172.21.15.90:0/1063314176 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdf3c02d5c0 con 0x7fdf3c077700 2024-04-02T14:31:45.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:45.259+0000 7fdf39600640 1 -- 172.21.15.90:0/1063314176 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+2221 (secure 0 0 0) 0x7fdf280a0ff0 con 0x7fdf3c077700 2024-04-02T14:31:45.262 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-02T14:31:45.262 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24454,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/1564319431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":1564319431},{"type":"v1","addr":"172.21.15.177:6833","nonce":1564319431}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24461,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6833/1570337372","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":1570337372},{"type":"v1","addr":"172.21.15.92:6833","nonce":1570337372}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-04-02T14:31:45.262 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 3 2024-04-02T14:31:45.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:45.262+0000 7fdf43400640 1 -- 172.21.15.90:0/1063314176 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fdf240a3bb0 msgr2=0x7fdf240a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:31:45.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:45.262+0000 7fdf43400640 1 --2- 172.21.15.90:0/1063314176 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fdf240a3bb0 0x7fdf240a6070 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fdf3002c250 tx=0x7fdf3002cf90 comp rx=0 tx=0).stop 2024-04-02T14:31:45.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:45.262+0000 7fdf43400640 1 -- 172.21.15.90:0/1063314176 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdf3c077700 msgr2=0x7fdf3c19e950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:31:45.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:45.262+0000 7fdf43400640 1 --2- 172.21.15.90:0/1063314176 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdf3c077700 0x7fdf3c19e950 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fdf2800e9f0 tx=0x7fdf2800eeb0 comp rx=0 tx=0).stop 2024-04-02T14:31:45.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:45.262+0000 7fdf43400640 1 -- 172.21.15.90:0/1063314176 shutdown_connections 2024-04-02T14:31:45.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:45.262+0000 7fdf43400640 1 --2- 172.21.15.90:0/1063314176 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fdf240a3bb0 0x7fdf240a6070 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:31:45.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:45.262+0000 7fdf43400640 1 --2- 172.21.15.90:0/1063314176 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdf3c0780a0 0x7fdf3c1a59c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:31:45.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:45.262+0000 7fdf43400640 1 --2- 172.21.15.90:0/1063314176 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdf3c077700 0x7fdf3c19e950 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:31:45.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:45.262+0000 7fdf43400640 1 --2- 172.21.15.90:0/1063314176 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdf3c0764c0 0x7fdf3c19e410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:31:45.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:45.262+0000 7fdf43400640 1 -- 172.21.15.90:0/1063314176 >> 172.21.15.90:0/1063314176 conn(0x7fdf3c0ff2b0 msgr2=0x7fdf3c1009b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:31:45.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:45.263+0000 7fdf43400640 1 -- 172.21.15.90:0/1063314176 shutdown_connections 2024-04-02T14:31:45.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:45.263+0000 7fdf43400640 1 -- 172.21.15.90:0/1063314176 wait complete. 2024-04-02T14:31:45.810 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:31:45 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:31:45.810 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:31:45 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:31:45.811 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:31:45 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:31:45.811 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:31:45 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-02T14:31:45.811 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:31:45 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-02T14:31:45.811 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:31:45 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:31:45.811 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:31:45 smithi092 ceph-mon[36497]: Deploying daemon mds.j on smithi090 2024-04-02T14:31:45.811 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:31:45 smithi092 ceph-mon[36497]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-02T14:31:45.811 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:31:45 smithi092 ceph-mon[36497]: mds.? [v2:172.21.15.92:6832/1570337372,v1:172.21.15.92:6833/1570337372] up:boot 2024-04-02T14:31:45.811 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:31:45 smithi092 ceph-mon[36497]: fsmap 2 up:standby 2024-04-02T14:31:45.811 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:31:45 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-02T14:31:45.811 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:31:45 smithi092 ceph-mon[36497]: from='client.? 172.21.15.90:0/1063314176' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-02T14:31:45.931 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:31:45 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:31:45.931 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:31:45 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:31:45.931 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:31:45 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:31:45.931 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:31:45 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-02T14:31:45.931 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:31:45 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-02T14:31:45.931 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:31:45 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:31:45.931 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:31:45 smithi090 ceph-mon[31674]: Deploying daemon mds.j on smithi090 2024-04-02T14:31:45.931 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:31:45 smithi090 ceph-mon[31674]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-02T14:31:45.931 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:31:45 smithi090 ceph-mon[31674]: mds.? [v2:172.21.15.92:6832/1570337372,v1:172.21.15.92:6833/1570337372] up:boot 2024-04-02T14:31:45.931 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:31:45 smithi090 ceph-mon[31674]: fsmap 2 up:standby 2024-04-02T14:31:45.932 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:31:45 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-02T14:31:45.932 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:31:45 smithi090 ceph-mon[31674]: from='client.? 172.21.15.90:0/1063314176' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-02T14:31:45.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:31:45 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:31:45.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:31:45 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:31:45.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:31:45 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:31:45.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:31:45 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-02T14:31:45.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:31:45 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-02T14:31:45.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:31:45 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:31:45.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:31:45 smithi177 ceph-mon[36431]: Deploying daemon mds.j on smithi090 2024-04-02T14:31:45.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:31:45 smithi177 ceph-mon[36431]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-02T14:31:45.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:31:45 smithi177 ceph-mon[36431]: mds.? [v2:172.21.15.92:6832/1570337372,v1:172.21.15.92:6833/1570337372] up:boot 2024-04-02T14:31:45.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:31:45 smithi177 ceph-mon[36431]: fsmap 2 up:standby 2024-04-02T14:31:45.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:31:45 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-02T14:31:45.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:31:45 smithi177 ceph-mon[36431]: from='client.? 172.21.15.90:0/1063314176' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-02T14:31:47.284 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T14:31:47.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:47.669+0000 7f10d4400640 1 -- 172.21.15.90:0/365735902 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f10cc101b20 msgr2=0x7f10cc101f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:31:47.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:47.669+0000 7f10d4400640 1 --2- 172.21.15.90:0/365735902 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f10cc101b20 0x7f10cc101f40 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f10c0009900 tx=0x7f10c002bc30 comp rx=0 tx=0).stop 2024-04-02T14:31:47.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:47.670+0000 7f10d4400640 1 -- 172.21.15.90:0/365735902 shutdown_connections 2024-04-02T14:31:47.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:47.670+0000 7f10d4400640 1 --2- 172.21.15.90:0/365735902 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f10cc1036d0 0x7f10cc109f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:31:47.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:47.670+0000 7f10d4400640 1 --2- 172.21.15.90:0/365735902 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f10cc102d10 0x7f10cc103190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:31:47.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:47.670+0000 7f10d4400640 1 --2- 172.21.15.90:0/365735902 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f10cc101b20 0x7f10cc101f40 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:31:47.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:47.670+0000 7f10d4400640 1 -- 172.21.15.90:0/365735902 >> 172.21.15.90:0/365735902 conn(0x7f10cc0fd0b0 msgr2=0x7f10cc0ff4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:31:47.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:47.670+0000 7f10d4400640 1 -- 172.21.15.90:0/365735902 shutdown_connections 2024-04-02T14:31:47.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:47.670+0000 7f10d4400640 1 -- 172.21.15.90:0/365735902 wait complete. 2024-04-02T14:31:47.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:47.671+0000 7f10d4400640 1 Processor -- start 2024-04-02T14:31:47.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:47.671+0000 7f10d4400640 1 -- start start 2024-04-02T14:31:47.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:47.671+0000 7f10d4400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f10cc101b20 0x7f10cc197e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:31:47.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:47.671+0000 7f10d4400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f10cc102d10 0x7f10cc198350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:31:47.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:47.671+0000 7f10d4400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f10cc1036d0 0x7f10cc19f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:31:47.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:47.671+0000 7f10d4400640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f10cc10d7a0 con 0x7f10cc101b20 2024-04-02T14:31:47.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:47.671+0000 7f10d4400640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f10cc10d620 con 0x7f10cc1036d0 2024-04-02T14:31:47.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:47.671+0000 7f10d4400640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f10cc10d920 con 0x7f10cc102d10 2024-04-02T14:31:47.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:47.672+0000 7f10d1a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f10cc101b20 0x7f10cc197e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:31:47.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:47.672+0000 7f10d1a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f10cc101b20 0x7f10cc197e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:56846/0 (socket says 172.21.15.90:56846) 2024-04-02T14:31:47.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:47.672+0000 7f10d1a00640 1 -- 172.21.15.90:0/3150684232 learned_addr learned my addr 172.21.15.90:0/3150684232 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:31:47.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:47.672+0000 7f10d1000640 1 --2- 172.21.15.90:0/3150684232 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f10cc102d10 0x7f10cc198350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:31:47.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:47.672+0000 7f10d2400640 1 --2- 172.21.15.90:0/3150684232 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f10cc1036d0 0x7f10cc19f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:31:47.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:47.672+0000 7f10d1a00640 1 -- 172.21.15.90:0/3150684232 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f10cc102d10 msgr2=0x7f10cc198350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:31:47.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:47.672+0000 7f10d1a00640 1 --2- 172.21.15.90:0/3150684232 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f10cc102d10 0x7f10cc198350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:31:47.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:47.672+0000 7f10d1a00640 1 -- 172.21.15.90:0/3150684232 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f10cc1036d0 msgr2=0x7f10cc19f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:31:47.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:47.672+0000 7f10d1a00640 1 --2- 172.21.15.90:0/3150684232 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f10cc1036d0 0x7f10cc19f3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:31:47.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:47.672+0000 7f10d1a00640 1 -- 172.21.15.90:0/3150684232 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10cc19fa30 con 0x7f10cc101b20 2024-04-02T14:31:47.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:47.672+0000 7f10d1000640 1 --2- 172.21.15.90:0/3150684232 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f10cc102d10 0x7f10cc198350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:31:47.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:47.672+0000 7f10d1a00640 1 --2- 172.21.15.90:0/3150684232 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f10cc101b20 0x7f10cc197e10 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f10c0000c00 tx=0x7f10c002c640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:31:47.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:47.673+0000 7f10ca000640 1 -- 172.21.15.90:0/3150684232 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10c0041070 con 0x7f10cc101b20 2024-04-02T14:31:47.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:47.673+0000 7f10ca000640 1 -- 172.21.15.90:0/3150684232 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f10c0004450 con 0x7f10cc101b20 2024-04-02T14:31:47.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:47.673+0000 7f10d4400640 1 -- 172.21.15.90:0/3150684232 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10cc19fcc0 con 0x7f10cc101b20 2024-04-02T14:31:47.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:47.673+0000 7f10ca000640 1 -- 172.21.15.90:0/3150684232 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10c0034440 con 0x7f10cc101b20 2024-04-02T14:31:47.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:47.673+0000 7f10d4400640 1 -- 172.21.15.90:0/3150684232 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10cc1a01b0 con 0x7f10cc101b20 2024-04-02T14:31:47.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:47.674+0000 7f10d4400640 1 -- 172.21.15.90:0/3150684232 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f10cc108e40 con 0x7f10cc101b20 2024-04-02T14:31:47.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:47.679+0000 7f10ca000640 1 -- 172.21.15.90:0/3150684232 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f10c0044070 con 0x7f10cc101b20 2024-04-02T14:31:47.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:47.679+0000 7f10ca000640 1 --2- 172.21.15.90:0/3150684232 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f10b40a3bb0 0x7f10b40a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:31:47.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:47.679+0000 7f10ca000640 1 -- 172.21.15.90:0/3150684232 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 7127+0+0 (secure 0 0 0) 0x7f10c00f1c20 con 0x7f10cc101b20 2024-04-02T14:31:47.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:47.679+0000 7f10d1000640 1 --2- 172.21.15.90:0/3150684232 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f10b40a3bb0 0x7f10b40a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:31:47.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:47.680+0000 7f10d1000640 1 --2- 172.21.15.90:0/3150684232 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f10b40a3bb0 0x7f10b40a6070 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f10cc199320 tx=0x7f10b8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:31:47.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:47.682+0000 7f10ca000640 1 -- 172.21.15.90:0/3150684232 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7f10c00be4e0 con 0x7f10cc101b20 2024-04-02T14:31:47.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:47.933+0000 7f10d4400640 1 -- 172.21.15.90:0/3150684232 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f10cc02d370 con 0x7f10cc101b20 2024-04-02T14:31:47.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:47.934+0000 7f10ca000640 1 -- 172.21.15.90:0/3150684232 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+2221 (secure 0 0 0) 0x7f10c00c27e0 con 0x7f10cc101b20 2024-04-02T14:31:47.937 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-02T14:31:47.937 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24454,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/1564319431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":1564319431},{"type":"v1","addr":"172.21.15.177:6833","nonce":1564319431}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24461,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6833/1570337372","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":1570337372},{"type":"v1","addr":"172.21.15.92:6833","nonce":1570337372}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-04-02T14:31:47.937 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 3 2024-04-02T14:31:47.938 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:31:47 smithi090 ceph-mon[31674]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-02T14:31:47.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:47.937+0000 7f10d4400640 1 -- 172.21.15.90:0/3150684232 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f10b40a3bb0 msgr2=0x7f10b40a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:31:47.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:47.937+0000 7f10d4400640 1 --2- 172.21.15.90:0/3150684232 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f10b40a3bb0 0x7f10b40a6070 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f10cc199320 tx=0x7f10b8005f90 comp rx=0 tx=0).stop 2024-04-02T14:31:47.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:47.937+0000 7f10d4400640 1 -- 172.21.15.90:0/3150684232 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f10cc101b20 msgr2=0x7f10cc197e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:31:47.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:47.937+0000 7f10d4400640 1 --2- 172.21.15.90:0/3150684232 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f10cc101b20 0x7f10cc197e10 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f10c0000c00 tx=0x7f10c002c640 comp rx=0 tx=0).stop 2024-04-02T14:31:47.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:47.938+0000 7f10d4400640 1 -- 172.21.15.90:0/3150684232 shutdown_connections 2024-04-02T14:31:47.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:47.938+0000 7f10d4400640 1 --2- 172.21.15.90:0/3150684232 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f10b40a3bb0 0x7f10b40a6070 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:31:47.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:47.938+0000 7f10d4400640 1 --2- 172.21.15.90:0/3150684232 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f10cc1036d0 0x7f10cc19f3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:31:47.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:47.938+0000 7f10d4400640 1 --2- 172.21.15.90:0/3150684232 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f10cc102d10 0x7f10cc198350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:31:47.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:47.938+0000 7f10d4400640 1 --2- 172.21.15.90:0/3150684232 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f10cc101b20 0x7f10cc197e10 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:31:47.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:47.938+0000 7f10d4400640 1 -- 172.21.15.90:0/3150684232 >> 172.21.15.90:0/3150684232 conn(0x7f10cc0fd0b0 msgr2=0x7f10cc106010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:31:47.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:47.938+0000 7f10d4400640 1 -- 172.21.15.90:0/3150684232 shutdown_connections 2024-04-02T14:31:47.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:47.938+0000 7f10d4400640 1 -- 172.21.15.90:0/3150684232 wait complete. 2024-04-02T14:31:48.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:31:47 smithi092 ceph-mon[36497]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-02T14:31:48.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:31:47 smithi177 ceph-mon[36431]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-02T14:31:48.689 INFO:journalctl@ceph.mds.j.smithi090.stdout:Apr 02 14:31:48 smithi090 systemd[1]: Started Ceph mds.j for 6449448a-f0fc-11ee-b647-cb9ed24678a4. 2024-04-02T14:31:49.088 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:31:48 smithi177 ceph-mon[36431]: from='client.? 172.21.15.90:0/3150684232' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-02T14:31:49.088 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:31:48 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:31:49.088 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:31:48 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:31:49.088 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:31:48 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:31:49.088 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:31:48 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-02T14:31:49.089 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:31:48 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-02T14:31:49.089 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:31:48 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:31:49.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:31:48 smithi092 ceph-mon[36497]: from='client.? 172.21.15.90:0/3150684232' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-02T14:31:49.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:31:48 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:31:49.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:31:48 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:31:49.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:31:48 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:31:49.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:31:48 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-02T14:31:49.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:31:48 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-02T14:31:49.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:31:48 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:31:49.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:31:48 smithi090 ceph-mon[31674]: from='client.? 172.21.15.90:0/3150684232' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-02T14:31:49.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:31:48 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:31:49.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:31:48 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:31:49.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:31:48 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:31:49.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:31:48 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-02T14:31:49.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:31:48 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-02T14:31:49.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:31:48 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:31:49.962 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T14:31:50.047 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:31:49 smithi090 ceph-mon[31674]: Deploying daemon mds.d on smithi090 2024-04-02T14:31:50.047 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:31:49 smithi090 ceph-mon[31674]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-02T14:31:50.047 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:31:49 smithi090 ceph-mon[31674]: mds.? [v2:172.21.15.90:6834/1277124891,v1:172.21.15.90:6835/1277124891] up:boot 2024-04-02T14:31:50.047 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:31:49 smithi090 ceph-mon[31674]: fsmap 3 up:standby 2024-04-02T14:31:50.047 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:31:49 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-02T14:31:50.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:31:49 smithi092 ceph-mon[36497]: Deploying daemon mds.d on smithi090 2024-04-02T14:31:50.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:31:49 smithi092 ceph-mon[36497]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-02T14:31:50.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:31:49 smithi092 ceph-mon[36497]: mds.? [v2:172.21.15.90:6834/1277124891,v1:172.21.15.90:6835/1277124891] up:boot 2024-04-02T14:31:50.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:31:49 smithi092 ceph-mon[36497]: fsmap 3 up:standby 2024-04-02T14:31:50.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:31:49 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-02T14:31:50.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:50.185+0000 7f6070200640 1 -- 172.21.15.90:0/1625933732 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6068074560 msgr2=0x7f60680749e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:31:50.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:50.185+0000 7f6070200640 1 --2- 172.21.15.90:0/1625933732 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6068074560 0x7f60680749e0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f6054009930 tx=0x7f605402bc20 comp rx=0 tx=0).stop 2024-04-02T14:31:50.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:50.185+0000 7f6070200640 1 -- 172.21.15.90:0/1625933732 shutdown_connections 2024-04-02T14:31:50.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:50.185+0000 7f6070200640 1 --2- 172.21.15.90:0/1625933732 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f606810e100 0x7f60681104f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:31:50.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:50.185+0000 7f6070200640 1 --2- 172.21.15.90:0/1625933732 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6068074560 0x7f60680749e0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:31:50.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:50.185+0000 7f6070200640 1 --2- 172.21.15.90:0/1625933732 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6068075b40 0x7f6068073f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:31:50.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:50.185+0000 7f6070200640 1 -- 172.21.15.90:0/1625933732 >> 172.21.15.90:0/1625933732 conn(0x7f60680fd0a0 msgr2=0x7f60680ff4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:31:50.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:50.185+0000 7f6070200640 1 -- 172.21.15.90:0/1625933732 shutdown_connections 2024-04-02T14:31:50.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:50.185+0000 7f6070200640 1 -- 172.21.15.90:0/1625933732 wait complete. 2024-04-02T14:31:50.189 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:31:49 smithi177 ceph-mon[36431]: Deploying daemon mds.d on smithi090 2024-04-02T14:31:50.189 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:31:49 smithi177 ceph-mon[36431]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-02T14:31:50.189 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:31:49 smithi177 ceph-mon[36431]: mds.? [v2:172.21.15.90:6834/1277124891,v1:172.21.15.90:6835/1277124891] up:boot 2024-04-02T14:31:50.190 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:31:49 smithi177 ceph-mon[36431]: fsmap 3 up:standby 2024-04-02T14:31:50.190 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:31:49 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-02T14:31:50.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:50.187+0000 7f6070200640 1 Processor -- start 2024-04-02T14:31:50.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:50.187+0000 7f6070200640 1 -- start start 2024-04-02T14:31:50.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:50.187+0000 7f6070200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6068074560 0x7f606819e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:31:50.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:50.187+0000 7f6070200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6068075b40 0x7f606819ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:31:50.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:50.187+0000 7f6070200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f606810e100 0x7f60681a5ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:31:50.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:50.187+0000 7f6070200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f6068113e50 con 0x7f606810e100 2024-04-02T14:31:50.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:50.187+0000 7f6070200640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f6068113cd0 con 0x7f6068074560 2024-04-02T14:31:50.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:50.187+0000 7f6070200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f6068113fd0 con 0x7f6068075b40 2024-04-02T14:31:50.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:50.188+0000 7f606ce00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6068075b40 0x7f606819ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:31:50.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:50.188+0000 7f606ce00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6068075b40 0x7f606819ea40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:38568/0 (socket says 172.21.15.90:38568) 2024-04-02T14:31:50.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:50.188+0000 7f606ce00640 1 -- 172.21.15.90:0/3904110645 learned_addr learned my addr 172.21.15.90:0/3904110645 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:31:50.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:50.188+0000 7f606d800640 1 --2- 172.21.15.90:0/3904110645 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6068074560 0x7f606819e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:31:50.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:50.188+0000 7f606e200640 1 --2- 172.21.15.90:0/3904110645 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f606810e100 0x7f60681a5ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:31:50.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:50.188+0000 7f606ce00640 1 -- 172.21.15.90:0/3904110645 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6068074560 msgr2=0x7f606819e500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:31:50.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:50.188+0000 7f606ce00640 1 --2- 172.21.15.90:0/3904110645 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6068074560 0x7f606819e500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:31:50.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:50.188+0000 7f606ce00640 1 -- 172.21.15.90:0/3904110645 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f606810e100 msgr2=0x7f60681a5ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:31:50.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:50.188+0000 7f606ce00640 1 --2- 172.21.15.90:0/3904110645 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f606810e100 0x7f60681a5ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:31:50.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:50.188+0000 7f606ce00640 1 -- 172.21.15.90:0/3904110645 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60681a6120 con 0x7f6068075b40 2024-04-02T14:31:50.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:50.188+0000 7f606d800640 1 --2- 172.21.15.90:0/3904110645 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6068074560 0x7f606819e500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:31:50.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:50.189+0000 7f606ce00640 1 --2- 172.21.15.90:0/3904110645 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6068075b40 0x7f606819ea40 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f6054000c00 tx=0x7f605402c630 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:31:50.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:50.189+0000 7f6066000640 1 -- 172.21.15.90:0/3904110645 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6054043020 con 0x7f6068075b40 2024-04-02T14:31:50.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:50.189+0000 7f6070200640 1 -- 172.21.15.90:0/3904110645 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f60681a63b0 con 0x7f6068075b40 2024-04-02T14:31:50.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:50.189+0000 7f6066000640 1 -- 172.21.15.90:0/3904110645 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f60540054e0 con 0x7f6068075b40 2024-04-02T14:31:50.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:50.189+0000 7f6066000640 1 -- 172.21.15.90:0/3904110645 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6054003d30 con 0x7f6068075b40 2024-04-02T14:31:50.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:50.189+0000 7f6070200640 1 -- 172.21.15.90:0/3904110645 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60681a6950 con 0x7f6068075b40 2024-04-02T14:31:50.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:50.190+0000 7f6070200640 1 -- 172.21.15.90:0/3904110645 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f60681042d0 con 0x7f6068075b40 2024-04-02T14:31:50.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:50.192+0000 7f6066000640 1 -- 172.21.15.90:0/3904110645 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f6054046020 con 0x7f6068075b40 2024-04-02T14:31:50.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:50.193+0000 7f6066000640 1 --2- 172.21.15.90:0/3904110645 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f60500a3c80 0x7f60500a6140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:31:50.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:50.193+0000 7f6066000640 1 -- 172.21.15.90:0/3904110645 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 7127+0+0 (secure 0 0 0) 0x7f60540f2280 con 0x7f6068075b40 2024-04-02T14:31:50.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:50.194+0000 7f606d800640 1 --2- 172.21.15.90:0/3904110645 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f60500a3c80 0x7f60500a6140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:31:50.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:50.194+0000 7f606d800640 1 --2- 172.21.15.90:0/3904110645 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f60500a3c80 0x7f60500a6140 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f605c009fc0 tx=0x7f605c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:31:50.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:50.198+0000 7f6066000640 1 -- 172.21.15.90:0/3904110645 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7f605404d020 con 0x7f6068075b40 2024-04-02T14:31:50.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:50.446+0000 7f6070200640 1 -- 172.21.15.90:0/3904110645 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6068062700 con 0x7f6068075b40 2024-04-02T14:31:50.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:50.447+0000 7f6066000640 1 -- 172.21.15.90:0/3904110645 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) ==== 93+0+3046 (secure 0 0 0) 0x7f60540bf060 con 0x7f6068075b40 2024-04-02T14:31:50.449 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-02T14:31:50.449 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":4,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24454,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/1564319431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":1564319431},{"type":"v1","addr":"172.21.15.177:6833","nonce":1564319431}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24461,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6833/1570337372","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":1570337372},{"type":"v1","addr":"172.21.15.92:6833","nonce":1570337372}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24478,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6835/1277124891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6834","nonce":1277124891},{"type":"v1","addr":"172.21.15.90:6835","nonce":1277124891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4}],"filesystems":[]} 2024-04-02T14:31:50.449 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 4 2024-04-02T14:31:50.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:50.450+0000 7f6070200640 1 -- 172.21.15.90:0/3904110645 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f60500a3c80 msgr2=0x7f60500a6140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:31:50.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:50.450+0000 7f6070200640 1 --2- 172.21.15.90:0/3904110645 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f60500a3c80 0x7f60500a6140 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f605c009fc0 tx=0x7f605c005f90 comp rx=0 tx=0).stop 2024-04-02T14:31:50.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:50.450+0000 7f6070200640 1 -- 172.21.15.90:0/3904110645 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6068075b40 msgr2=0x7f606819ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:31:50.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:50.450+0000 7f6070200640 1 --2- 172.21.15.90:0/3904110645 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6068075b40 0x7f606819ea40 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f6054000c00 tx=0x7f605402c630 comp rx=0 tx=0).stop 2024-04-02T14:31:50.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:50.450+0000 7f6070200640 1 -- 172.21.15.90:0/3904110645 shutdown_connections 2024-04-02T14:31:50.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:50.450+0000 7f6070200640 1 --2- 172.21.15.90:0/3904110645 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f60500a3c80 0x7f60500a6140 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:31:50.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:50.450+0000 7f6070200640 1 --2- 172.21.15.90:0/3904110645 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f606810e100 0x7f60681a5ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:31:50.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:50.450+0000 7f6070200640 1 --2- 172.21.15.90:0/3904110645 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6068075b40 0x7f606819ea40 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:31:50.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:50.450+0000 7f6070200640 1 --2- 172.21.15.90:0/3904110645 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6068074560 0x7f606819e500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:31:50.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:50.450+0000 7f6070200640 1 -- 172.21.15.90:0/3904110645 >> 172.21.15.90:0/3904110645 conn(0x7f60680fd0a0 msgr2=0x7f60681014a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:31:50.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:50.450+0000 7f6070200640 1 -- 172.21.15.90:0/3904110645 shutdown_connections 2024-04-02T14:31:50.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:50.450+0000 7f6070200640 1 -- 172.21.15.90:0/3904110645 wait complete. 2024-04-02T14:31:50.938 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:31:50 smithi090 ceph-mon[31674]: from='client.? 172.21.15.90:0/3904110645' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-02T14:31:51.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:31:50 smithi092 ceph-mon[36497]: from='client.? 172.21.15.90:0/3904110645' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-02T14:31:51.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:31:50 smithi177 ceph-mon[36431]: from='client.? 172.21.15.90:0/3904110645' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-02T14:31:51.803 INFO:journalctl@ceph.mds.d.smithi090.stdout:Apr 02 14:31:51 smithi090 systemd[1]: Starting Ceph mds.d for 6449448a-f0fc-11ee-b647-cb9ed24678a4... 2024-04-02T14:31:52.091 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:31:51 smithi090 ceph-mon[31674]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-02T14:31:52.091 INFO:journalctl@ceph.mds.d.smithi090.stdout:Apr 02 14:31:51 smithi090 podman[72487]: 2024-04-02 14:31:51.708432761 +0000 UTC m=+0.030369894 image pull f7301e070545fc23d8e5b7b052a2c15e832a0e57102f1acad8e4f3583709d23f quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534 2024-04-02T14:31:52.091 INFO:journalctl@ceph.mds.d.smithi090.stdout:Apr 02 14:31:51 smithi090 podman[72487]: 2024-04-02 14:31:51.813633433 +0000 UTC m=+0.135570594 container create 675ab21b436c576e67e74eedf917205f2b15bff912fea39abee86a79c66cf35a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mds-d, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240326, RELEASE=wip-batrick-testing-20240402.004512-191d963, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-04-02T14:31:52.091 INFO:journalctl@ceph.mds.d.smithi090.stdout:Apr 02 14:31:52 smithi090 podman[72487]: 2024-04-02 14:31:52.035314101 +0000 UTC m=+0.357251305 container init 675ab21b436c576e67e74eedf917205f2b15bff912fea39abee86a79c66cf35a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mds-d, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240326, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240402.004512-191d963, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-04-02T14:31:52.091 INFO:journalctl@ceph.mds.d.smithi090.stdout:Apr 02 14:31:52 smithi090 podman[72487]: 2024-04-02 14:31:52.045349444 +0000 UTC m=+0.367286605 container start 675ab21b436c576e67e74eedf917205f2b15bff912fea39abee86a79c66cf35a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mds-d, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240402.004512-191d963, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240326, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-04-02T14:31:52.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:31:51 smithi092 ceph-mon[36497]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-02T14:31:52.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:31:51 smithi177 ceph-mon[36431]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-02T14:31:52.439 INFO:journalctl@ceph.mds.d.smithi090.stdout:Apr 02 14:31:52 smithi090 ceph-mds[72507]: -- 172.21.15.90:0/237848216 <== mon.0 v2:172.21.15.90:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5623b2a41a40 con 0x5623b2a14c00 2024-04-02T14:31:52.439 INFO:journalctl@ceph.mds.d.smithi090.stdout:Apr 02 14:31:52 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mds-d[72503]: starting mds.d at 2024-04-02T14:31:52.440 INFO:journalctl@ceph.mds.d.smithi090.stdout:Apr 02 14:31:52 smithi090 bash[72487]: 675ab21b436c576e67e74eedf917205f2b15bff912fea39abee86a79c66cf35a 2024-04-02T14:31:52.440 INFO:journalctl@ceph.mds.d.smithi090.stdout:Apr 02 14:31:52 smithi090 systemd[1]: Started Ceph mds.d for 6449448a-f0fc-11ee-b647-cb9ed24678a4. 2024-04-02T14:31:52.470 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T14:31:52.695 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:52.694+0000 7f84d4200640 1 -- 172.21.15.90:0/2911651056 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f84cc075b40 msgr2=0x7f84cc073f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:31:52.695 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:52.694+0000 7f84d4200640 1 --2- 172.21.15.90:0/2911651056 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f84cc075b40 0x7f84cc073f90 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f84bc009930 tx=0x7f84bc02bc20 comp rx=0 tx=0).stop 2024-04-02T14:31:52.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:52.694+0000 7f84d4200640 1 -- 172.21.15.90:0/2911651056 shutdown_connections 2024-04-02T14:31:52.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:52.694+0000 7f84d4200640 1 --2- 172.21.15.90:0/2911651056 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f84cc10e100 0x7f84cc1104f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:31:52.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:52.694+0000 7f84d4200640 1 --2- 172.21.15.90:0/2911651056 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f84cc074560 0x7f84cc0749e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:31:52.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:52.694+0000 7f84d4200640 1 --2- 172.21.15.90:0/2911651056 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f84cc075b40 0x7f84cc073f90 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:31:52.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:52.694+0000 7f84d4200640 1 -- 172.21.15.90:0/2911651056 >> 172.21.15.90:0/2911651056 conn(0x7f84cc0fd0a0 msgr2=0x7f84cc0ff4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:31:52.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:52.694+0000 7f84d4200640 1 -- 172.21.15.90:0/2911651056 shutdown_connections 2024-04-02T14:31:52.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:52.694+0000 7f84d4200640 1 -- 172.21.15.90:0/2911651056 wait complete. 2024-04-02T14:31:52.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:52.695+0000 7f84d4200640 1 Processor -- start 2024-04-02T14:31:52.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:52.695+0000 7f84d4200640 1 -- start start 2024-04-02T14:31:52.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:52.695+0000 7f84d4200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f84cc074560 0x7f84cc19e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:31:52.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:52.696+0000 7f84d4200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f84cc075b40 0x7f84cc19ea00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:31:52.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:52.696+0000 7f84d4200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f84cc10e100 0x7f84cc1a5a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:31:52.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:52.696+0000 7f84d4200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f84cc113de0 con 0x7f84cc075b40 2024-04-02T14:31:52.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:52.696+0000 7f84d4200640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f84cc113c60 con 0x7f84cc074560 2024-04-02T14:31:52.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:52.696+0000 7f84d4200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f84cc113f60 con 0x7f84cc10e100 2024-04-02T14:31:52.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:52.696+0000 7f84d1800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f84cc074560 0x7f84cc19e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:31:52.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:52.696+0000 7f84d1800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f84cc074560 0x7f84cc19e4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:59936/0 (socket says 172.21.15.90:59936) 2024-04-02T14:31:52.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:52.696+0000 7f84d1800640 1 -- 172.21.15.90:0/465449467 learned_addr learned my addr 172.21.15.90:0/465449467 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:31:52.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:52.696+0000 7f84d0e00640 1 --2- 172.21.15.90:0/465449467 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f84cc075b40 0x7f84cc19ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:31:52.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:52.696+0000 7f84d2200640 1 --2- 172.21.15.90:0/465449467 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f84cc10e100 0x7f84cc1a5a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:31:52.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:52.696+0000 7f84d1800640 1 -- 172.21.15.90:0/465449467 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f84cc10e100 msgr2=0x7f84cc1a5a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:31:52.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:52.696+0000 7f84d1800640 1 --2- 172.21.15.90:0/465449467 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f84cc10e100 0x7f84cc1a5a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:31:52.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:52.696+0000 7f84d1800640 1 -- 172.21.15.90:0/465449467 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f84cc075b40 msgr2=0x7f84cc19ea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:31:52.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:52.696+0000 7f84d1800640 1 --2- 172.21.15.90:0/465449467 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f84cc075b40 0x7f84cc19ea00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:31:52.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:52.696+0000 7f84d1800640 1 -- 172.21.15.90:0/465449467 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84cc1a6090 con 0x7f84cc074560 2024-04-02T14:31:52.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:52.696+0000 7f84d2200640 1 --2- 172.21.15.90:0/465449467 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f84cc10e100 0x7f84cc1a5a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:31:52.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:52.696+0000 7f84d1800640 1 --2- 172.21.15.90:0/465449467 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f84cc074560 0x7f84cc19e4c0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f84bc000c00 tx=0x7f84bc005d50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:31:52.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:52.697+0000 7f84ca000640 1 -- 172.21.15.90:0/465449467 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84bc043020 con 0x7f84cc074560 2024-04-02T14:31:52.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:52.697+0000 7f84ca000640 1 -- 172.21.15.90:0/465449467 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f84bc0054e0 con 0x7f84cc074560 2024-04-02T14:31:52.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:52.697+0000 7f84ca000640 1 -- 172.21.15.90:0/465449467 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84bc003d10 con 0x7f84cc074560 2024-04-02T14:31:52.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:52.697+0000 7f84d4200640 1 -- 172.21.15.90:0/465449467 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f84cc1a6320 con 0x7f84cc074560 2024-04-02T14:31:52.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:52.697+0000 7f84d4200640 1 -- 172.21.15.90:0/465449467 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84cc1a6820 con 0x7f84cc074560 2024-04-02T14:31:52.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:52.698+0000 7f84d4200640 1 -- 172.21.15.90:0/465449467 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f84cc1042d0 con 0x7f84cc074560 2024-04-02T14:31:52.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:52.700+0000 7f84ca000640 1 -- 172.21.15.90:0/465449467 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f84bc046020 con 0x7f84cc074560 2024-04-02T14:31:52.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:52.701+0000 7f84ca000640 1 --2- 172.21.15.90:0/465449467 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f84b00a3bb0 0x7f84b00a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:31:52.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:52.701+0000 7f84ca000640 1 -- 172.21.15.90:0/465449467 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 7127+0+0 (secure 0 0 0) 0x7f84bc0f2220 con 0x7f84cc074560 2024-04-02T14:31:52.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:52.701+0000 7f84d0e00640 1 --2- 172.21.15.90:0/465449467 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f84b00a3bb0 0x7f84b00a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:31:52.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:52.702+0000 7f84d0e00640 1 --2- 172.21.15.90:0/465449467 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f84b00a3bb0 0x7f84b00a6070 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f84cc19f980 tx=0x7f84b800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:31:52.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:52.705+0000 7f84ca000640 1 -- 172.21.15.90:0/465449467 <== mon.2 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7f84bc04d020 con 0x7f84cc074560 2024-04-02T14:31:52.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:52.952+0000 7f84d4200640 1 -- 172.21.15.90:0/465449467 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f84cc062700 con 0x7f84cc074560 2024-04-02T14:31:52.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:52.953+0000 7f84ca000640 1 -- 172.21.15.90:0/465449467 <== mon.2 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) ==== 93+0+3871 (secure 0 0 0) 0x7f84bc0bf000 con 0x7f84cc074560 2024-04-02T14:31:52.955 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-02T14:31:52.955 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":5,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14412,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6837/2852160309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6836","nonce":2852160309},{"type":"v1","addr":"172.21.15.90:6837","nonce":2852160309}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24454,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/1564319431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":1564319431},{"type":"v1","addr":"172.21.15.177:6833","nonce":1564319431}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24461,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6833/1570337372","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":1570337372},{"type":"v1","addr":"172.21.15.92:6833","nonce":1570337372}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24478,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6835/1277124891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6834","nonce":1277124891},{"type":"v1","addr":"172.21.15.90:6835","nonce":1277124891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4}],"filesystems":[]} 2024-04-02T14:31:52.955 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 5 2024-04-02T14:31:52.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:52.956+0000 7f84d4200640 1 -- 172.21.15.90:0/465449467 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f84b00a3bb0 msgr2=0x7f84b00a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:31:52.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:52.956+0000 7f84d4200640 1 --2- 172.21.15.90:0/465449467 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f84b00a3bb0 0x7f84b00a6070 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f84cc19f980 tx=0x7f84b800b410 comp rx=0 tx=0).stop 2024-04-02T14:31:52.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:52.956+0000 7f84d4200640 1 -- 172.21.15.90:0/465449467 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f84cc074560 msgr2=0x7f84cc19e4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:31:52.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:52.956+0000 7f84d4200640 1 --2- 172.21.15.90:0/465449467 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f84cc074560 0x7f84cc19e4c0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f84bc000c00 tx=0x7f84bc005d50 comp rx=0 tx=0).stop 2024-04-02T14:31:52.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:52.957+0000 7f84d4200640 1 -- 172.21.15.90:0/465449467 shutdown_connections 2024-04-02T14:31:52.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:52.957+0000 7f84d4200640 1 --2- 172.21.15.90:0/465449467 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f84b00a3bb0 0x7f84b00a6070 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:31:52.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:52.957+0000 7f84d4200640 1 --2- 172.21.15.90:0/465449467 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f84cc10e100 0x7f84cc1a5a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:31:52.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:52.957+0000 7f84d4200640 1 --2- 172.21.15.90:0/465449467 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f84cc075b40 0x7f84cc19ea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:31:52.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:52.957+0000 7f84d4200640 1 --2- 172.21.15.90:0/465449467 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f84cc074560 0x7f84cc19e4c0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:31:52.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:52.957+0000 7f84d4200640 1 -- 172.21.15.90:0/465449467 >> 172.21.15.90:0/465449467 conn(0x7f84cc0fd0a0 msgr2=0x7f84cc1014a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:31:52.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:52.957+0000 7f84d4200640 1 -- 172.21.15.90:0/465449467 shutdown_connections 2024-04-02T14:31:52.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:52.957+0000 7f84d4200640 1 -- 172.21.15.90:0/465449467 wait complete. 2024-04-02T14:31:53.611 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:31:53 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:31:53.611 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:31:53 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:31:53.611 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:31:53 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:31:53.611 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:31:53 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-02T14:31:53.611 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:31:53 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-02T14:31:53.611 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:31:53 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:31:53.611 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:31:53 smithi092 ceph-mon[36497]: Deploying daemon mds.e on smithi092 2024-04-02T14:31:53.611 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:31:53 smithi092 ceph-mon[36497]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-02T14:31:53.611 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:31:53 smithi092 ceph-mon[36497]: mds.? [v2:172.21.15.90:6836/2852160309,v1:172.21.15.90:6837/2852160309] up:boot 2024-04-02T14:31:53.611 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:31:53 smithi092 ceph-mon[36497]: fsmap 4 up:standby 2024-04-02T14:31:53.611 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:31:53 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-02T14:31:53.611 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:31:53 smithi092 ceph-mon[36497]: from='client.? 172.21.15.90:0/465449467' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-02T14:31:53.686 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:31:53 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:31:53.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:31:53 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:31:53.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:31:53 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:31:53.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:31:53 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-02T14:31:53.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:31:53 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-02T14:31:53.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:31:53 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:31:53.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:31:53 smithi177 ceph-mon[36431]: Deploying daemon mds.e on smithi092 2024-04-02T14:31:53.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:31:53 smithi177 ceph-mon[36431]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-02T14:31:53.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:31:53 smithi177 ceph-mon[36431]: mds.? [v2:172.21.15.90:6836/2852160309,v1:172.21.15.90:6837/2852160309] up:boot 2024-04-02T14:31:53.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:31:53 smithi177 ceph-mon[36431]: fsmap 4 up:standby 2024-04-02T14:31:53.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:31:53 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-02T14:31:53.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:31:53 smithi177 ceph-mon[36431]: from='client.? 172.21.15.90:0/465449467' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-02T14:31:53.689 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:31:53 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:31:53.689 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:31:53 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:31:53.689 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:31:53 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:31:53.689 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:31:53 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-02T14:31:53.689 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:31:53 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-02T14:31:53.689 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:31:53 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:31:53.689 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:31:53 smithi090 ceph-mon[31674]: Deploying daemon mds.e on smithi092 2024-04-02T14:31:53.689 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:31:53 smithi090 ceph-mon[31674]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-02T14:31:53.689 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:31:53 smithi090 ceph-mon[31674]: mds.? [v2:172.21.15.90:6836/2852160309,v1:172.21.15.90:6837/2852160309] up:boot 2024-04-02T14:31:53.690 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:31:53 smithi090 ceph-mon[31674]: fsmap 4 up:standby 2024-04-02T14:31:53.690 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:31:53 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-02T14:31:53.690 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:31:53 smithi090 ceph-mon[31674]: from='client.? 172.21.15.90:0/465449467' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-02T14:31:54.611 INFO:journalctl@ceph.mds.e.smithi092.stdout:Apr 02 14:31:54 smithi092 systemd[1]: Starting Ceph mds.e for 6449448a-f0fc-11ee-b647-cb9ed24678a4... 2024-04-02T14:31:54.977 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T14:31:55.008 INFO:journalctl@ceph.mds.e.smithi092.stdout:Apr 02 14:31:54 smithi092 podman[66776]: 2024-04-02 14:31:54.738493973 +0000 UTC m=+0.100673912 container create bf500f992b9b82e9bc3794a57e60d72326a20177c60a2a9b1ac24698e5759f1d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mds-e, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240402.004512-191d963, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240326, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-04-02T14:31:55.008 INFO:journalctl@ceph.mds.e.smithi092.stdout:Apr 02 14:31:54 smithi092 podman[66776]: 2024-04-02 14:31:54.655592821 +0000 UTC m=+0.017772778 image pull f7301e070545fc23d8e5b7b052a2c15e832a0e57102f1acad8e4f3583709d23f quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534 2024-04-02T14:31:55.008 INFO:journalctl@ceph.mds.e.smithi092.stdout:Apr 02 14:31:55 smithi092 podman[66776]: 2024-04-02 14:31:55.007419092 +0000 UTC m=+0.369599032 container init bf500f992b9b82e9bc3794a57e60d72326a20177c60a2a9b1ac24698e5759f1d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mds-e, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240402.004512-191d963, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240326, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, org.label-schema.license=GPLv2) 2024-04-02T14:31:55.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:55.198+0000 7fe4f9000640 1 -- 172.21.15.90:0/3746327411 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe4f4103d40 msgr2=0x7fe4f4104160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:31:55.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:55.198+0000 7fe4f9000640 1 --2- 172.21.15.90:0/3746327411 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe4f4103d40 0x7fe4f4104160 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fe4e4009900 tx=0x7fe4e402bc50 comp rx=0 tx=0).stop 2024-04-02T14:31:55.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:55.198+0000 7fe4f9000640 1 -- 172.21.15.90:0/3746327411 shutdown_connections 2024-04-02T14:31:55.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:55.198+0000 7fe4f9000640 1 --2- 172.21.15.90:0/3746327411 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe4f41058f0 0x7fe4f410c180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:31:55.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:55.198+0000 7fe4f9000640 1 --2- 172.21.15.90:0/3746327411 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe4f4104f30 0x7fe4f41053b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:31:55.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:55.198+0000 7fe4f9000640 1 --2- 172.21.15.90:0/3746327411 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe4f4103d40 0x7fe4f4104160 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:31:55.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:55.198+0000 7fe4f9000640 1 -- 172.21.15.90:0/3746327411 >> 172.21.15.90:0/3746327411 conn(0x7fe4f40ff2f0 msgr2=0x7fe4f4101710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:31:55.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:55.198+0000 7fe4f9000640 1 -- 172.21.15.90:0/3746327411 shutdown_connections 2024-04-02T14:31:55.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:55.198+0000 7fe4f9000640 1 -- 172.21.15.90:0/3746327411 wait complete. 2024-04-02T14:31:55.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:55.199+0000 7fe4f9000640 1 Processor -- start 2024-04-02T14:31:55.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:55.200+0000 7fe4f9000640 1 -- start start 2024-04-02T14:31:55.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:55.200+0000 7fe4f9000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe4f4103d40 0x7fe4f419a080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:31:55.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:55.200+0000 7fe4f9000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe4f4104f30 0x7fe4f419a5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:31:55.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:55.200+0000 7fe4f9000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe4f41058f0 0x7fe4f41a1630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:31:55.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:55.200+0000 7fe4f9000640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fe4f410f9e0 con 0x7fe4f4104f30 2024-04-02T14:31:55.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:55.200+0000 7fe4f9000640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fe4f410f860 con 0x7fe4f4103d40 2024-04-02T14:31:55.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:55.200+0000 7fe4f9000640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fe4f410fb60 con 0x7fe4f41058f0 2024-04-02T14:31:55.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:55.200+0000 7fe4f1e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe4f4103d40 0x7fe4f419a080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:31:55.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:55.200+0000 7fe4f1e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe4f4103d40 0x7fe4f419a080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:36948/0 (socket says 172.21.15.90:36948) 2024-04-02T14:31:55.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:55.200+0000 7fe4f1e00640 1 -- 172.21.15.90:0/4054336393 learned_addr learned my addr 172.21.15.90:0/4054336393 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:31:55.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:55.200+0000 7fe4ebe00640 1 --2- 172.21.15.90:0/4054336393 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe4f4104f30 0x7fe4f419a5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:31:55.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:55.200+0000 7fe4f2800640 1 --2- 172.21.15.90:0/4054336393 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe4f41058f0 0x7fe4f41a1630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:31:55.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:55.201+0000 7fe4f1e00640 1 -- 172.21.15.90:0/4054336393 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe4f41058f0 msgr2=0x7fe4f41a1630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:31:55.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:55.201+0000 7fe4f1e00640 1 --2- 172.21.15.90:0/4054336393 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe4f41058f0 0x7fe4f41a1630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:31:55.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:55.201+0000 7fe4f1e00640 1 -- 172.21.15.90:0/4054336393 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe4f4104f30 msgr2=0x7fe4f419a5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:31:55.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:55.201+0000 7fe4f1e00640 1 --2- 172.21.15.90:0/4054336393 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe4f4104f30 0x7fe4f419a5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:31:55.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:55.201+0000 7fe4f1e00640 1 -- 172.21.15.90:0/4054336393 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4f41a1ca0 con 0x7fe4f4103d40 2024-04-02T14:31:55.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:55.201+0000 7fe4f1e00640 1 --2- 172.21.15.90:0/4054336393 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe4f4103d40 0x7fe4f419a080 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fe4e4005ab0 tx=0x7fe4e402c6e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:31:55.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:55.201+0000 7fe4eaa00640 1 -- 172.21.15.90:0/4054336393 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4e4041070 con 0x7fe4f4103d40 2024-04-02T14:31:55.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:55.201+0000 7fe4eaa00640 1 -- 172.21.15.90:0/4054336393 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe4e4004450 con 0x7fe4f4103d40 2024-04-02T14:31:55.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:55.201+0000 7fe4f9000640 1 -- 172.21.15.90:0/4054336393 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4f41a1f30 con 0x7fe4f4103d40 2024-04-02T14:31:55.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:55.202+0000 7fe4eaa00640 1 -- 172.21.15.90:0/4054336393 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4e4034440 con 0x7fe4f4103d40 2024-04-02T14:31:55.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:55.202+0000 7fe4f9000640 1 -- 172.21.15.90:0/4054336393 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4f41a2390 con 0x7fe4f4103d40 2024-04-02T14:31:55.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:55.203+0000 7fe4f9000640 1 -- 172.21.15.90:0/4054336393 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe4f410b060 con 0x7fe4f4103d40 2024-04-02T14:31:55.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:55.204+0000 7fe4eaa00640 1 -- 172.21.15.90:0/4054336393 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fe4e4044070 con 0x7fe4f4103d40 2024-04-02T14:31:55.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:55.206+0000 7fe4eaa00640 1 --2- 172.21.15.90:0/4054336393 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fe4d80a3bb0 0x7fe4d80a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:31:55.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:55.206+0000 7fe4eaa00640 1 -- 172.21.15.90:0/4054336393 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 7127+0+0 (secure 0 0 0) 0x7fe4e40f1b70 con 0x7fe4f4103d40 2024-04-02T14:31:55.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:55.206+0000 7fe4ebe00640 1 --2- 172.21.15.90:0/4054336393 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fe4d80a3bb0 0x7fe4d80a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:31:55.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:55.206+0000 7fe4ebe00640 1 --2- 172.21.15.90:0/4054336393 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fe4d80a3bb0 0x7fe4d80a6070 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fe4dc0095f0 tx=0x7fe4dc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:31:55.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:55.210+0000 7fe4eaa00640 1 -- 172.21.15.90:0/4054336393 <== mon.2 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7fe4e40be300 con 0x7fe4f4103d40 2024-04-02T14:31:55.361 INFO:journalctl@ceph.mds.e.smithi092.stdout:Apr 02 14:31:55 smithi092 podman[66776]: 2024-04-02 14:31:55.010081106 +0000 UTC m=+0.372261058 container start bf500f992b9b82e9bc3794a57e60d72326a20177c60a2a9b1ac24698e5759f1d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mds-e, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240402.004512-191d963, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240326, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-02T14:31:55.361 INFO:journalctl@ceph.mds.e.smithi092.stdout:Apr 02 14:31:55 smithi092 ceph-mds[66795]: -- 172.21.15.92:0/70424704 <== mon.2 v2:172.21.15.92:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56302ee3ba40 con 0x56302ee0f000 2024-04-02T14:31:55.361 INFO:journalctl@ceph.mds.e.smithi092.stdout:Apr 02 14:31:55 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mds-e[66791]: starting mds.e at 2024-04-02T14:31:55.361 INFO:journalctl@ceph.mds.e.smithi092.stdout:Apr 02 14:31:55 smithi092 bash[66776]: bf500f992b9b82e9bc3794a57e60d72326a20177c60a2a9b1ac24698e5759f1d 2024-04-02T14:31:55.361 INFO:journalctl@ceph.mds.e.smithi092.stdout:Apr 02 14:31:55 smithi092 systemd[1]: Started Ceph mds.e for 6449448a-f0fc-11ee-b647-cb9ed24678a4. 2024-04-02T14:31:55.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:55.455+0000 7fe4f9000640 1 -- 172.21.15.90:0/4054336393 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe4f402d370 con 0x7fe4f4103d40 2024-04-02T14:31:55.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:55.456+0000 7fe4eaa00640 1 -- 172.21.15.90:0/4054336393 <== mon.2 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) ==== 93+0+3871 (secure 0 0 0) 0x7fe4e40c2600 con 0x7fe4f4103d40 2024-04-02T14:31:55.457 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-02T14:31:55.457 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":5,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14412,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6837/2852160309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6836","nonce":2852160309},{"type":"v1","addr":"172.21.15.90:6837","nonce":2852160309}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24454,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/1564319431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":1564319431},{"type":"v1","addr":"172.21.15.177:6833","nonce":1564319431}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24461,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6833/1570337372","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":1570337372},{"type":"v1","addr":"172.21.15.92:6833","nonce":1570337372}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24478,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6835/1277124891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6834","nonce":1277124891},{"type":"v1","addr":"172.21.15.90:6835","nonce":1277124891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4}],"filesystems":[]} 2024-04-02T14:31:55.457 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 5 2024-04-02T14:31:55.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:55.458+0000 7fe4f9000640 1 -- 172.21.15.90:0/4054336393 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fe4d80a3bb0 msgr2=0x7fe4d80a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:31:55.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:55.458+0000 7fe4f9000640 1 --2- 172.21.15.90:0/4054336393 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fe4d80a3bb0 0x7fe4d80a6070 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fe4dc0095f0 tx=0x7fe4dc005f90 comp rx=0 tx=0).stop 2024-04-02T14:31:55.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:55.458+0000 7fe4f9000640 1 -- 172.21.15.90:0/4054336393 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe4f4103d40 msgr2=0x7fe4f419a080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:31:55.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:55.458+0000 7fe4f9000640 1 --2- 172.21.15.90:0/4054336393 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe4f4103d40 0x7fe4f419a080 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fe4e4005ab0 tx=0x7fe4e402c6e0 comp rx=0 tx=0).stop 2024-04-02T14:31:55.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:55.459+0000 7fe4f9000640 1 -- 172.21.15.90:0/4054336393 shutdown_connections 2024-04-02T14:31:55.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:55.459+0000 7fe4f9000640 1 --2- 172.21.15.90:0/4054336393 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fe4d80a3bb0 0x7fe4d80a6070 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:31:55.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:55.459+0000 7fe4f9000640 1 --2- 172.21.15.90:0/4054336393 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe4f41058f0 0x7fe4f41a1630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:31:55.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:55.459+0000 7fe4f9000640 1 --2- 172.21.15.90:0/4054336393 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe4f4104f30 0x7fe4f419a5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:31:55.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:55.459+0000 7fe4f9000640 1 --2- 172.21.15.90:0/4054336393 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe4f4103d40 0x7fe4f419a080 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:31:55.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:55.459+0000 7fe4f9000640 1 -- 172.21.15.90:0/4054336393 >> 172.21.15.90:0/4054336393 conn(0x7fe4f40ff2f0 msgr2=0x7fe4f4108230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:31:55.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:55.459+0000 7fe4f9000640 1 -- 172.21.15.90:0/4054336393 shutdown_connections 2024-04-02T14:31:55.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:55.459+0000 7fe4f9000640 1 -- 172.21.15.90:0/4054336393 wait complete. 2024-04-02T14:31:56.041 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:31:55 smithi177 ceph-mon[36431]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-02T14:31:56.041 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:31:55 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:31:56.041 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:31:55 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:31:56.042 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:31:55 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:31:56.042 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:31:55 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-02T14:31:56.042 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:31:55 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-02T14:31:56.042 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:31:55 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:31:56.042 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:31:55 smithi177 ceph-mon[36431]: from='client.? 172.21.15.90:0/4054336393' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-02T14:31:56.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:31:55 smithi092 ceph-mon[36497]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-02T14:31:56.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:31:55 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:31:56.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:31:55 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:31:56.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:31:55 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:31:56.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:31:55 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-02T14:31:56.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:31:55 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-02T14:31:56.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:31:55 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:31:56.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:31:55 smithi092 ceph-mon[36497]: from='client.? 172.21.15.90:0/4054336393' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-02T14:31:56.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:31:55 smithi090 ceph-mon[31674]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-02T14:31:56.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:31:55 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:31:56.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:31:55 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:31:56.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:31:55 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:31:56.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:31:55 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-02T14:31:56.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:31:55 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-02T14:31:56.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:31:55 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:31:56.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:31:55 smithi090 ceph-mon[31674]: from='client.? 172.21.15.90:0/4054336393' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-02T14:31:57.110 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:31:56 smithi092 ceph-mon[36497]: Deploying daemon mds.f on smithi177 2024-04-02T14:31:57.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:31:56 smithi092 ceph-mon[36497]: mds.? [v2:172.21.15.92:6834/860721164,v1:172.21.15.92:6835/860721164] up:boot 2024-04-02T14:31:57.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:31:56 smithi092 ceph-mon[36497]: fsmap 5 up:standby 2024-04-02T14:31:57.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:31:56 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-02T14:31:57.174 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:31:56 smithi177 ceph-mon[36431]: Deploying daemon mds.f on smithi177 2024-04-02T14:31:57.174 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:31:56 smithi177 ceph-mon[36431]: mds.? [v2:172.21.15.92:6834/860721164,v1:172.21.15.92:6835/860721164] up:boot 2024-04-02T14:31:57.174 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:31:56 smithi177 ceph-mon[36431]: fsmap 5 up:standby 2024-04-02T14:31:57.175 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:31:56 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-02T14:31:57.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:31:56 smithi090 ceph-mon[31674]: Deploying daemon mds.f on smithi177 2024-04-02T14:31:57.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:31:56 smithi090 ceph-mon[31674]: mds.? [v2:172.21.15.92:6834/860721164,v1:172.21.15.92:6835/860721164] up:boot 2024-04-02T14:31:57.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:31:56 smithi090 ceph-mon[31674]: fsmap 5 up:standby 2024-04-02T14:31:57.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:31:56 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-02T14:31:57.437 INFO:journalctl@ceph.mds.f.smithi177.stdout:Apr 02 14:31:57 smithi177 systemd[1]: Starting Ceph mds.f for 6449448a-f0fc-11ee-b647-cb9ed24678a4... 2024-04-02T14:31:57.482 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T14:31:57.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:57.702+0000 7f3cf7200640 1 -- 172.21.15.90:0/2753678925 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3cf0103d40 msgr2=0x7f3cf0104160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:31:57.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:57.702+0000 7f3cf7200640 1 --2- 172.21.15.90:0/2753678925 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3cf0103d40 0x7f3cf0104160 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f3ce0009900 tx=0x7f3ce002bc30 comp rx=0 tx=0).stop 2024-04-02T14:31:57.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:57.702+0000 7f3cf7200640 1 -- 172.21.15.90:0/2753678925 shutdown_connections 2024-04-02T14:31:57.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:57.702+0000 7f3cf7200640 1 --2- 172.21.15.90:0/2753678925 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3cf01058f0 0x7f3cf010c180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:31:57.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:57.702+0000 7f3cf7200640 1 --2- 172.21.15.90:0/2753678925 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3cf0104f30 0x7f3cf01053b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:31:57.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:57.702+0000 7f3cf7200640 1 --2- 172.21.15.90:0/2753678925 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3cf0103d40 0x7f3cf0104160 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:31:57.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:57.702+0000 7f3cf7200640 1 -- 172.21.15.90:0/2753678925 >> 172.21.15.90:0/2753678925 conn(0x7f3cf00ff2f0 msgr2=0x7f3cf0101710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:31:57.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:57.702+0000 7f3cf7200640 1 -- 172.21.15.90:0/2753678925 shutdown_connections 2024-04-02T14:31:57.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:57.702+0000 7f3cf7200640 1 -- 172.21.15.90:0/2753678925 wait complete. 2024-04-02T14:31:57.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:57.703+0000 7f3cf7200640 1 Processor -- start 2024-04-02T14:31:57.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:57.703+0000 7f3cf7200640 1 -- start start 2024-04-02T14:31:57.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:57.704+0000 7f3cf7200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3cf0103d40 0x7f3cf019a040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:31:57.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:57.704+0000 7f3cf7200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3cf0104f30 0x7f3cf019a580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:31:57.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:57.704+0000 7f3cf7200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3cf01058f0 0x7f3cf01a15f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:31:57.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:57.704+0000 7f3cf7200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f3cf010f9c0 con 0x7f3cf0103d40 2024-04-02T14:31:57.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:57.704+0000 7f3cf7200640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f3cf010f840 con 0x7f3cf01058f0 2024-04-02T14:31:57.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:57.704+0000 7f3cf7200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f3cf010fb40 con 0x7f3cf0104f30 2024-04-02T14:31:57.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:57.704+0000 7f3cef400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3cf0104f30 0x7f3cf019a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:31:57.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:57.704+0000 7f3cef400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3cf0104f30 0x7f3cf019a580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:41786/0 (socket says 172.21.15.90:41786) 2024-04-02T14:31:57.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:57.704+0000 7f3cefe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3cf0103d40 0x7f3cf019a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:31:57.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:57.704+0000 7f3cef400640 1 -- 172.21.15.90:0/2843003040 learned_addr learned my addr 172.21.15.90:0/2843003040 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:31:57.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:57.704+0000 7f3cf5200640 1 --2- 172.21.15.90:0/2843003040 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3cf01058f0 0x7f3cf01a15f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:31:57.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:57.704+0000 7f3cef400640 1 -- 172.21.15.90:0/2843003040 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3cf01058f0 msgr2=0x7f3cf01a15f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:31:57.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:57.704+0000 7f3cef400640 1 --2- 172.21.15.90:0/2843003040 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3cf01058f0 0x7f3cf01a15f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:31:57.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:57.704+0000 7f3cef400640 1 -- 172.21.15.90:0/2843003040 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3cf0103d40 msgr2=0x7f3cf019a040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:31:57.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:57.704+0000 7f3cef400640 1 --2- 172.21.15.90:0/2843003040 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3cf0103d40 0x7f3cf019a040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:31:57.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:57.704+0000 7f3cef400640 1 -- 172.21.15.90:0/2843003040 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3cf01a1c60 con 0x7f3cf0104f30 2024-04-02T14:31:57.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:57.705+0000 7f3cef400640 1 --2- 172.21.15.90:0/2843003040 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3cf0104f30 0x7f3cf019a580 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f3cdc00e9f0 tx=0x7f3cdc00eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:31:57.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:57.705+0000 7f3cecc00640 1 -- 172.21.15.90:0/2843003040 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cdc00cd80 con 0x7f3cf0104f30 2024-04-02T14:31:57.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:57.705+0000 7f3cecc00640 1 -- 172.21.15.90:0/2843003040 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3cdc010910 con 0x7f3cf0104f30 2024-04-02T14:31:57.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:57.705+0000 7f3cf7200640 1 -- 172.21.15.90:0/2843003040 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3cf01a1f50 con 0x7f3cf0104f30 2024-04-02T14:31:57.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:57.705+0000 7f3cecc00640 1 -- 172.21.15.90:0/2843003040 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cdc0188f0 con 0x7f3cf0104f30 2024-04-02T14:31:57.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:57.705+0000 7f3cf7200640 1 -- 172.21.15.90:0/2843003040 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3cf01a2410 con 0x7f3cf0104f30 2024-04-02T14:31:57.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:57.706+0000 7f3cf7200640 1 -- 172.21.15.90:0/2843003040 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3cf010b080 con 0x7f3cf0104f30 2024-04-02T14:31:57.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:57.709+0000 7f3cecc00640 1 -- 172.21.15.90:0/2843003040 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f3cdc010ac0 con 0x7f3cf0104f30 2024-04-02T14:31:57.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:57.710+0000 7f3cecc00640 1 --2- 172.21.15.90:0/2843003040 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f3cd40a3bb0 0x7f3cd40a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:31:57.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:57.710+0000 7f3cecc00640 1 -- 172.21.15.90:0/2843003040 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 7127+0+0 (secure 0 0 0) 0x7f3cdc0d04e0 con 0x7f3cf0104f30 2024-04-02T14:31:57.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:57.710+0000 7f3cefe00640 1 --2- 172.21.15.90:0/2843003040 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f3cd40a3bb0 0x7f3cd40a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:31:57.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:57.711+0000 7f3cefe00640 1 --2- 172.21.15.90:0/2843003040 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f3cd40a3bb0 0x7f3cd40a6070 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f3ce0000c00 tx=0x7f3ce0005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:31:57.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:57.712+0000 7f3cecc00640 1 -- 172.21.15.90:0/2843003040 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7f3cdc09cc70 con 0x7f3cf0104f30 2024-04-02T14:31:57.842 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:31:57 smithi177 ceph-mon[36431]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-02T14:31:57.842 INFO:journalctl@ceph.mds.f.smithi177.stdout:Apr 02 14:31:57 smithi177 podman[66053]: 2024-04-02 14:31:57.471313612 +0000 UTC m=+0.017617728 image pull f7301e070545fc23d8e5b7b052a2c15e832a0e57102f1acad8e4f3583709d23f quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534 2024-04-02T14:31:57.842 INFO:journalctl@ceph.mds.f.smithi177.stdout:Apr 02 14:31:57 smithi177 podman[66053]: 2024-04-02 14:31:57.580185926 +0000 UTC m=+0.126490041 container create 20637d31e333ede5df8ec43de524ab8a118131f4c03a6423c8e9c29d4ee77265 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mds-f, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240326, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240402.004512-191d963) 2024-04-02T14:31:57.842 INFO:journalctl@ceph.mds.f.smithi177.stdout:Apr 02 14:31:57 smithi177 podman[66053]: 2024-04-02 14:31:57.816087333 +0000 UTC m=+0.362391445 container init 20637d31e333ede5df8ec43de524ab8a118131f4c03a6423c8e9c29d4ee77265 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mds-f, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240402.004512-191d963, org.label-schema.build-date=20240326, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-04-02T14:31:57.842 INFO:journalctl@ceph.mds.f.smithi177.stdout:Apr 02 14:31:57 smithi177 podman[66053]: 2024-04-02 14:31:57.818736466 +0000 UTC m=+0.365040582 container start 20637d31e333ede5df8ec43de524ab8a118131f4c03a6423c8e9c29d4ee77265 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mds-f, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240326, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240402.004512-191d963, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-02T14:31:57.938 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:31:57 smithi090 ceph-mon[31674]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-02T14:31:57.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:57.963+0000 7f3cf7200640 1 -- 172.21.15.90:0/2843003040 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3cf002d370 con 0x7f3cf0104f30 2024-04-02T14:31:57.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:57.966+0000 7f3cecc00640 1 -- 172.21.15.90:0/2843003040 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) ==== 93+0+4693 (secure 0 0 0) 0x7f3cdc0a0f70 con 0x7f3cf0104f30 2024-04-02T14:31:57.968 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-02T14:31:57.968 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":6,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14412,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6837/2852160309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6836","nonce":2852160309},{"type":"v1","addr":"172.21.15.90:6837","nonce":2852160309}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24454,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/1564319431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":1564319431},{"type":"v1","addr":"172.21.15.177:6833","nonce":1564319431}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24461,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6833/1570337372","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":1570337372},{"type":"v1","addr":"172.21.15.92:6833","nonce":1570337372}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24478,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6835/1277124891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6834","nonce":1277124891},{"type":"v1","addr":"172.21.15.90:6835","nonce":1277124891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24503,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6835/860721164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":860721164},{"type":"v1","addr":"172.21.15.92:6835","nonce":860721164}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[]} 2024-04-02T14:31:57.968 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 6 2024-04-02T14:31:57.970 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:57.969+0000 7f3cf7200640 1 -- 172.21.15.90:0/2843003040 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f3cd40a3bb0 msgr2=0x7f3cd40a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:31:57.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:57.969+0000 7f3cf7200640 1 --2- 172.21.15.90:0/2843003040 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f3cd40a3bb0 0x7f3cd40a6070 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f3ce0000c00 tx=0x7f3ce0005fd0 comp rx=0 tx=0).stop 2024-04-02T14:31:57.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:57.969+0000 7f3cf7200640 1 -- 172.21.15.90:0/2843003040 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3cf0104f30 msgr2=0x7f3cf019a580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:31:57.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:57.969+0000 7f3cf7200640 1 --2- 172.21.15.90:0/2843003040 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3cf0104f30 0x7f3cf019a580 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f3cdc00e9f0 tx=0x7f3cdc00eeb0 comp rx=0 tx=0).stop 2024-04-02T14:31:57.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:57.969+0000 7f3cf7200640 1 -- 172.21.15.90:0/2843003040 shutdown_connections 2024-04-02T14:31:57.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:57.969+0000 7f3cf7200640 1 --2- 172.21.15.90:0/2843003040 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f3cd40a3bb0 0x7f3cd40a6070 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:31:57.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:57.969+0000 7f3cf7200640 1 --2- 172.21.15.90:0/2843003040 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3cf01058f0 0x7f3cf01a15f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:31:57.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:57.969+0000 7f3cf7200640 1 --2- 172.21.15.90:0/2843003040 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3cf0104f30 0x7f3cf019a580 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:31:57.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:57.969+0000 7f3cf7200640 1 --2- 172.21.15.90:0/2843003040 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3cf0103d40 0x7f3cf019a040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:31:57.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:57.969+0000 7f3cf7200640 1 -- 172.21.15.90:0/2843003040 >> 172.21.15.90:0/2843003040 conn(0x7f3cf00ff2f0 msgr2=0x7f3cf0108250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:31:57.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:57.970+0000 7f3cf7200640 1 -- 172.21.15.90:0/2843003040 shutdown_connections 2024-04-02T14:31:57.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:31:57.970+0000 7f3cf7200640 1 -- 172.21.15.90:0/2843003040 wait complete. 2024-04-02T14:31:58.110 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:31:57 smithi092 ceph-mon[36497]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-02T14:31:58.186 INFO:journalctl@ceph.mds.f.smithi177.stdout:Apr 02 14:31:57 smithi177 ceph-mds[66072]: -- 172.21.15.177:0/3556864012 <== mon.1 v2:172.21.15.177:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55e686eb1a40 con 0x55e686e85000 2024-04-02T14:31:58.186 INFO:journalctl@ceph.mds.f.smithi177.stdout:Apr 02 14:31:57 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mds-f[66068]: starting mds.f at 2024-04-02T14:31:58.186 INFO:journalctl@ceph.mds.f.smithi177.stdout:Apr 02 14:31:57 smithi177 bash[66053]: 20637d31e333ede5df8ec43de524ab8a118131f4c03a6423c8e9c29d4ee77265 2024-04-02T14:31:58.187 INFO:journalctl@ceph.mds.f.smithi177.stdout:Apr 02 14:31:57 smithi177 systemd[1]: Started Ceph mds.f for 6449448a-f0fc-11ee-b647-cb9ed24678a4. 2024-04-02T14:31:59.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:31:58 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:31:59.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:31:58 smithi177 ceph-mon[36431]: from='client.? 172.21.15.90:0/2843003040' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-02T14:31:59.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:31:58 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:31:59.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:31:58 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:31:59.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:31:58 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-02T14:31:59.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:31:58 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-02T14:31:59.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:31:58 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:31:59.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:31:58 smithi177 ceph-mon[36431]: Deploying daemon mds.c on smithi177 2024-04-02T14:31:59.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:31:58 smithi177 ceph-mon[36431]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-02T14:31:59.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:31:58 smithi177 ceph-mon[36431]: mds.? [v2:172.21.15.177:6834/3177244988,v1:172.21.15.177:6835/3177244988] up:boot 2024-04-02T14:31:59.315 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:31:58 smithi177 ceph-mon[36431]: fsmap 6 up:standby 2024-04-02T14:31:59.316 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:31:58 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-02T14:31:59.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:31:58 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:31:59.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:31:58 smithi092 ceph-mon[36497]: from='client.? 172.21.15.90:0/2843003040' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-02T14:31:59.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:31:58 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:31:59.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:31:58 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:31:59.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:31:58 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-02T14:31:59.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:31:58 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-02T14:31:59.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:31:58 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:31:59.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:31:58 smithi092 ceph-mon[36497]: Deploying daemon mds.c on smithi177 2024-04-02T14:31:59.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:31:58 smithi092 ceph-mon[36497]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-02T14:31:59.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:31:58 smithi092 ceph-mon[36497]: mds.? [v2:172.21.15.177:6834/3177244988,v1:172.21.15.177:6835/3177244988] up:boot 2024-04-02T14:31:59.362 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:31:58 smithi092 ceph-mon[36497]: fsmap 6 up:standby 2024-04-02T14:31:59.362 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:31:58 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-02T14:31:59.439 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:31:58 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:31:59.439 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:31:58 smithi090 ceph-mon[31674]: from='client.? 172.21.15.90:0/2843003040' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-02T14:31:59.439 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:31:58 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:31:59.439 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:31:58 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:31:59.439 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:31:58 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-02T14:31:59.439 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:31:58 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-02T14:31:59.439 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:31:58 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:31:59.439 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:31:58 smithi090 ceph-mon[31674]: Deploying daemon mds.c on smithi177 2024-04-02T14:31:59.440 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:31:58 smithi090 ceph-mon[31674]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-02T14:31:59.440 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:31:58 smithi090 ceph-mon[31674]: mds.? [v2:172.21.15.177:6834/3177244988,v1:172.21.15.177:6835/3177244988] up:boot 2024-04-02T14:31:59.440 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:31:58 smithi090 ceph-mon[31674]: fsmap 6 up:standby 2024-04-02T14:31:59.440 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:31:58 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-02T14:31:59.990 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T14:32:00.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:00.210+0000 7fcbac400640 1 -- 172.21.15.90:0/2576749322 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcba41036e0 msgr2=0x7fcba4109f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:32:00.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:00.210+0000 7fcbac400640 1 --2- 172.21.15.90:0/2576749322 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcba41036e0 0x7fcba4109f70 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fcb9c009ca0 tx=0x7fcb9c02c100 comp rx=0 tx=0).stop 2024-04-02T14:32:00.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:00.210+0000 7fcbac400640 1 -- 172.21.15.90:0/2576749322 shutdown_connections 2024-04-02T14:32:00.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:00.210+0000 7fcbac400640 1 --2- 172.21.15.90:0/2576749322 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcba41036e0 0x7fcba4109f70 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:00.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:00.210+0000 7fcbac400640 1 --2- 172.21.15.90:0/2576749322 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcba4102d20 0x7fcba41031a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:00.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:00.210+0000 7fcbac400640 1 --2- 172.21.15.90:0/2576749322 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcba4101b30 0x7fcba4101f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:00.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:00.210+0000 7fcbac400640 1 -- 172.21.15.90:0/2576749322 >> 172.21.15.90:0/2576749322 conn(0x7fcba40fd0e0 msgr2=0x7fcba40ff500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:32:00.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:00.210+0000 7fcbac400640 1 -- 172.21.15.90:0/2576749322 shutdown_connections 2024-04-02T14:32:00.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:00.210+0000 7fcbac400640 1 -- 172.21.15.90:0/2576749322 wait complete. 2024-04-02T14:32:00.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:00.211+0000 7fcbac400640 1 Processor -- start 2024-04-02T14:32:00.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:00.211+0000 7fcbac400640 1 -- start start 2024-04-02T14:32:00.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:00.212+0000 7fcbac400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcba4101b30 0x7fcba419a060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:32:00.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:00.212+0000 7fcbac400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcba4102d20 0x7fcba419a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:32:00.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:00.212+0000 7fcbac400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcba41036e0 0x7fcba41a1610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:32:00.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:00.212+0000 7fcbac400640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fcba410d7b0 con 0x7fcba4102d20 2024-04-02T14:32:00.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:00.212+0000 7fcbac400640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fcba410d630 con 0x7fcba41036e0 2024-04-02T14:32:00.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:00.212+0000 7fcbac400640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fcba410d930 con 0x7fcba4101b30 2024-04-02T14:32:00.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:00.212+0000 7fcba9a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcba4101b30 0x7fcba419a060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:32:00.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:00.212+0000 7fcba9a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcba4101b30 0x7fcba419a060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:41814/0 (socket says 172.21.15.90:41814) 2024-04-02T14:32:00.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:00.212+0000 7fcba9a00640 1 -- 172.21.15.90:0/2907660394 learned_addr learned my addr 172.21.15.90:0/2907660394 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:32:00.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:00.212+0000 7fcba9000640 1 --2- 172.21.15.90:0/2907660394 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcba4102d20 0x7fcba419a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:32:00.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:00.212+0000 7fcbaa400640 1 --2- 172.21.15.90:0/2907660394 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcba41036e0 0x7fcba41a1610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:32:00.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:00.212+0000 7fcba9a00640 1 -- 172.21.15.90:0/2907660394 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcba41036e0 msgr2=0x7fcba41a1610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:32:00.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:00.212+0000 7fcba9a00640 1 --2- 172.21.15.90:0/2907660394 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcba41036e0 0x7fcba41a1610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:00.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:00.212+0000 7fcba9a00640 1 -- 172.21.15.90:0/2907660394 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcba4102d20 msgr2=0x7fcba419a5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:32:00.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:00.212+0000 7fcba9a00640 1 --2- 172.21.15.90:0/2907660394 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcba4102d20 0x7fcba419a5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:00.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:00.212+0000 7fcba9a00640 1 -- 172.21.15.90:0/2907660394 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcba41a1c80 con 0x7fcba4101b30 2024-04-02T14:32:00.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:00.212+0000 7fcba9000640 1 --2- 172.21.15.90:0/2907660394 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcba4102d20 0x7fcba419a5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:32:00.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:00.213+0000 7fcba9a00640 1 --2- 172.21.15.90:0/2907660394 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcba4101b30 0x7fcba419a060 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fcb9800e9c0 tx=0x7fcb9800ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:32:00.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:00.213+0000 7fcba2000640 1 -- 172.21.15.90:0/2907660394 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb9800cd80 con 0x7fcba4101b30 2024-04-02T14:32:00.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:00.213+0000 7fcba2000640 1 -- 172.21.15.90:0/2907660394 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcb980035f0 con 0x7fcba4101b30 2024-04-02T14:32:00.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:00.213+0000 7fcbac400640 1 -- 172.21.15.90:0/2907660394 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcba41a1f70 con 0x7fcba4101b30 2024-04-02T14:32:00.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:00.213+0000 7fcba2000640 1 -- 172.21.15.90:0/2907660394 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb98010690 con 0x7fcba4101b30 2024-04-02T14:32:00.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:00.213+0000 7fcbac400640 1 -- 172.21.15.90:0/2907660394 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcba41a23d0 con 0x7fcba4101b30 2024-04-02T14:32:00.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:00.214+0000 7fcbac400640 1 -- 172.21.15.90:0/2907660394 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcba41a2690 con 0x7fcba4101b30 2024-04-02T14:32:00.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:00.216+0000 7fcba2000640 1 -- 172.21.15.90:0/2907660394 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fcb980037a0 con 0x7fcba4101b30 2024-04-02T14:32:00.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:00.218+0000 7fcba2000640 1 --2- 172.21.15.90:0/2907660394 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fcb8c0a3c30 0x7fcb8c0a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:32:00.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:00.218+0000 7fcba2000640 1 -- 172.21.15.90:0/2907660394 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 7127+0+0 (secure 0 0 0) 0x7fcb980d0450 con 0x7fcba4101b30 2024-04-02T14:32:00.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:00.218+0000 7fcba9000640 1 --2- 172.21.15.90:0/2907660394 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fcb8c0a3c30 0x7fcb8c0a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:32:00.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:00.219+0000 7fcba9000640 1 --2- 172.21.15.90:0/2907660394 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fcb8c0a3c30 0x7fcb8c0a60f0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fcba419b570 tx=0x7fcb90005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:32:00.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:00.220+0000 7fcba2000640 1 -- 172.21.15.90:0/2907660394 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7fcb9809d230 con 0x7fcba4101b30 2024-04-02T14:32:00.390 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 02 14:32:00 smithi177 systemd[1]: Starting Ceph mds.c for 6449448a-f0fc-11ee-b647-cb9ed24678a4... 2024-04-02T14:32:00.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:00.464+0000 7fcbac400640 1 -- 172.21.15.90:0/2907660394 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcba41a2860 con 0x7fcba4101b30 2024-04-02T14:32:00.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:00.465+0000 7fcba2000640 1 -- 172.21.15.90:0/2907660394 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+5521 (secure 0 0 0) 0x7fcb9809cb30 con 0x7fcba4101b30 2024-04-02T14:32:00.467 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-02T14:32:00.467 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":7,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14412,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6837/2852160309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6836","nonce":2852160309},{"type":"v1","addr":"172.21.15.90:6837","nonce":2852160309}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24454,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/1564319431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":1564319431},{"type":"v1","addr":"172.21.15.177:6833","nonce":1564319431}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24461,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6833/1570337372","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":1570337372},{"type":"v1","addr":"172.21.15.92:6833","nonce":1570337372}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24478,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6835/1277124891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6834","nonce":1277124891},{"type":"v1","addr":"172.21.15.90:6835","nonce":1277124891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24503,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6835/860721164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":860721164},{"type":"v1","addr":"172.21.15.92:6835","nonce":860721164}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24511,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/3177244988","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":3177244988},{"type":"v1","addr":"172.21.15.177:6835","nonce":3177244988}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[]} 2024-04-02T14:32:00.467 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 7 2024-04-02T14:32:00.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:00.468+0000 7fcbac400640 1 -- 172.21.15.90:0/2907660394 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fcb8c0a3c30 msgr2=0x7fcb8c0a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:32:00.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:00.468+0000 7fcbac400640 1 --2- 172.21.15.90:0/2907660394 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fcb8c0a3c30 0x7fcb8c0a60f0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fcba419b570 tx=0x7fcb90005f90 comp rx=0 tx=0).stop 2024-04-02T14:32:00.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:00.468+0000 7fcbac400640 1 -- 172.21.15.90:0/2907660394 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcba4101b30 msgr2=0x7fcba419a060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:32:00.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:00.468+0000 7fcbac400640 1 --2- 172.21.15.90:0/2907660394 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcba4101b30 0x7fcba419a060 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fcb9800e9c0 tx=0x7fcb9800ee80 comp rx=0 tx=0).stop 2024-04-02T14:32:00.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:00.468+0000 7fcbac400640 1 -- 172.21.15.90:0/2907660394 shutdown_connections 2024-04-02T14:32:00.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:00.468+0000 7fcbac400640 1 --2- 172.21.15.90:0/2907660394 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fcb8c0a3c30 0x7fcb8c0a60f0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:00.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:00.468+0000 7fcbac400640 1 --2- 172.21.15.90:0/2907660394 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcba41036e0 0x7fcba41a1610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:00.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:00.468+0000 7fcbac400640 1 --2- 172.21.15.90:0/2907660394 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcba4102d20 0x7fcba419a5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:00.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:00.468+0000 7fcbac400640 1 --2- 172.21.15.90:0/2907660394 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcba4101b30 0x7fcba419a060 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:00.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:00.468+0000 7fcbac400640 1 -- 172.21.15.90:0/2907660394 >> 172.21.15.90:0/2907660394 conn(0x7fcba40fd0e0 msgr2=0x7fcba4073d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:32:00.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:00.468+0000 7fcbac400640 1 -- 172.21.15.90:0/2907660394 shutdown_connections 2024-04-02T14:32:00.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:00.468+0000 7fcbac400640 1 -- 172.21.15.90:0/2907660394 wait complete. 2024-04-02T14:32:00.645 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:00 smithi177 ceph-mon[36431]: from='client.? 172.21.15.90:0/2907660394' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-02T14:32:00.646 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 02 14:32:00 smithi177 podman[66356]: 2024-04-02 14:32:00.291966422 +0000 UTC m=+0.018034980 image pull f7301e070545fc23d8e5b7b052a2c15e832a0e57102f1acad8e4f3583709d23f quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534 2024-04-02T14:32:00.646 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 02 14:32:00 smithi177 podman[66356]: 2024-04-02 14:32:00.408945299 +0000 UTC m=+0.135013841 container create ab5bb25e316a74a8b503c0000f6bfce72e9a2438a34696c0ed85f1f14404585b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mds-c, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240326, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240402.004512-191d963, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-04-02T14:32:00.646 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 02 14:32:00 smithi177 podman[66356]: 2024-04-02 14:32:00.645172197 +0000 UTC m=+0.371240742 container init ab5bb25e316a74a8b503c0000f6bfce72e9a2438a34696c0ed85f1f14404585b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mds-c, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=wip-batrick-testing-20240402.004512-191d963, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240326) 2024-04-02T14:32:00.861 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:00 smithi092 ceph-mon[36497]: from='client.? 172.21.15.90:0/2907660394' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-02T14:32:00.937 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 02 14:32:00 smithi177 podman[66356]: 2024-04-02 14:32:00.648275448 +0000 UTC m=+0.374344000 container start ab5bb25e316a74a8b503c0000f6bfce72e9a2438a34696c0ed85f1f14404585b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mds-c, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240402.004512-191d963, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240326, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-02T14:32:00.937 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 02 14:32:00 smithi177 ceph-mds[66375]: -- 172.21.15.177:0/2250326094 <== mon.1 v2:172.21.15.177:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5566907e9a40 con 0x5566907bcc00 2024-04-02T14:32:00.937 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 02 14:32:00 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mds-c[66371]: starting mds.c at 2024-04-02T14:32:00.937 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 02 14:32:00 smithi177 bash[66356]: ab5bb25e316a74a8b503c0000f6bfce72e9a2438a34696c0ed85f1f14404585b 2024-04-02T14:32:00.937 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 02 14:32:00 smithi177 systemd[1]: Started Ceph mds.c for 6449448a-f0fc-11ee-b647-cb9ed24678a4. 2024-04-02T14:32:00.938 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:00 smithi090 ceph-mon[31674]: from='client.? 172.21.15.90:0/2907660394' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-02T14:32:02.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:01 smithi092 ceph-mon[36497]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-02T14:32:02.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:01 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:32:02.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:01 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:32:02.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:01 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:32:02.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:01 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-02T14:32:02.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:01 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-02T14:32:02.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:01 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:32:02.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:01 smithi092 ceph-mon[36497]: Deploying daemon mds.h on smithi092 2024-04-02T14:32:02.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:01 smithi092 ceph-mon[36497]: mds.? [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] up:boot 2024-04-02T14:32:02.112 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:01 smithi092 ceph-mon[36497]: fsmap 7 up:standby 2024-04-02T14:32:02.112 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:01 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-02T14:32:02.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:01 smithi177 ceph-mon[36431]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-02T14:32:02.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:01 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:32:02.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:01 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:32:02.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:01 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:32:02.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:01 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-02T14:32:02.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:01 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-02T14:32:02.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:01 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:32:02.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:01 smithi177 ceph-mon[36431]: Deploying daemon mds.h on smithi092 2024-04-02T14:32:02.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:01 smithi177 ceph-mon[36431]: mds.? [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] up:boot 2024-04-02T14:32:02.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:01 smithi177 ceph-mon[36431]: fsmap 7 up:standby 2024-04-02T14:32:02.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:01 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-02T14:32:02.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:01 smithi090 ceph-mon[31674]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-02T14:32:02.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:01 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:32:02.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:01 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:32:02.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:01 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:32:02.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:01 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-02T14:32:02.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:01 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-02T14:32:02.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:01 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:32:02.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:01 smithi090 ceph-mon[31674]: Deploying daemon mds.h on smithi092 2024-04-02T14:32:02.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:01 smithi090 ceph-mon[31674]: mds.? [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] up:boot 2024-04-02T14:32:02.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:01 smithi090 ceph-mon[31674]: fsmap 7 up:standby 2024-04-02T14:32:02.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:01 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-02T14:32:02.491 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T14:32:02.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:02.709+0000 7f0bd2c00640 1 -- 172.21.15.90:0/2501080519 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0bcc103b50 msgr2=0x7f0bcc103f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:32:02.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:02.709+0000 7f0bd2c00640 1 --2- 172.21.15.90:0/2501080519 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0bcc103b50 0x7f0bcc103f70 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f0bc0009900 tx=0x7f0bc002bc30 comp rx=0 tx=0).stop 2024-04-02T14:32:02.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:02.709+0000 7f0bd2c00640 1 -- 172.21.15.90:0/2501080519 shutdown_connections 2024-04-02T14:32:02.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:02.710+0000 7f0bd2c00640 1 --2- 172.21.15.90:0/2501080519 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0bcc105700 0x7f0bcc10bf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:02.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:02.710+0000 7f0bd2c00640 1 --2- 172.21.15.90:0/2501080519 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0bcc104d40 0x7f0bcc1051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:02.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:02.710+0000 7f0bd2c00640 1 --2- 172.21.15.90:0/2501080519 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0bcc103b50 0x7f0bcc103f70 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:02.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:02.710+0000 7f0bd2c00640 1 -- 172.21.15.90:0/2501080519 >> 172.21.15.90:0/2501080519 conn(0x7f0bcc0ff0e0 msgr2=0x7f0bcc101500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:32:02.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:02.710+0000 7f0bd2c00640 1 -- 172.21.15.90:0/2501080519 shutdown_connections 2024-04-02T14:32:02.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:02.710+0000 7f0bd2c00640 1 -- 172.21.15.90:0/2501080519 wait complete. 2024-04-02T14:32:02.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:02.711+0000 7f0bd2c00640 1 Processor -- start 2024-04-02T14:32:02.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:02.711+0000 7f0bd2c00640 1 -- start start 2024-04-02T14:32:02.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:02.711+0000 7f0bd2c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0bcc103b50 0x7f0bcc199e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:32:02.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:02.711+0000 7f0bd2c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0bcc104d40 0x7f0bcc19a390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:32:02.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:02.711+0000 7f0bd2c00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0bcc105700 0x7f0bcc1a1400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:32:02.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:02.711+0000 7f0bd2c00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f0bcc10f7a0 con 0x7f0bcc103b50 2024-04-02T14:32:02.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:02.711+0000 7f0bd2c00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f0bcc10f620 con 0x7f0bcc105700 2024-04-02T14:32:02.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:02.711+0000 7f0bd2c00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f0bcc10f920 con 0x7f0bcc104d40 2024-04-02T14:32:02.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:02.711+0000 7f0bd1800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0bcc103b50 0x7f0bcc199e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:32:02.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:02.711+0000 7f0bd1800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0bcc103b50 0x7f0bcc199e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:54468/0 (socket says 172.21.15.90:54468) 2024-04-02T14:32:02.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:02.711+0000 7f0bd1800640 1 -- 172.21.15.90:0/274663066 learned_addr learned my addr 172.21.15.90:0/274663066 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:32:02.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:02.711+0000 7f0bd2200640 1 --2- 172.21.15.90:0/274663066 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0bcc105700 0x7f0bcc1a1400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:32:02.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:02.711+0000 7f0bd0e00640 1 --2- 172.21.15.90:0/274663066 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0bcc104d40 0x7f0bcc19a390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:32:02.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:02.712+0000 7f0bd2200640 1 -- 172.21.15.90:0/274663066 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0bcc104d40 msgr2=0x7f0bcc19a390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:32:02.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:02.712+0000 7f0bd2200640 1 --2- 172.21.15.90:0/274663066 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0bcc104d40 0x7f0bcc19a390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:02.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:02.712+0000 7f0bd2200640 1 -- 172.21.15.90:0/274663066 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0bcc103b50 msgr2=0x7f0bcc199e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:32:02.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:02.712+0000 7f0bd2200640 1 --2- 172.21.15.90:0/274663066 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0bcc103b50 0x7f0bcc199e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:02.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:02.712+0000 7f0bd2200640 1 -- 172.21.15.90:0/274663066 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0bcc1a1a70 con 0x7f0bcc105700 2024-04-02T14:32:02.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:02.712+0000 7f0bd2200640 1 --2- 172.21.15.90:0/274663066 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0bcc105700 0x7f0bcc1a1400 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f0bc4009fd0 tx=0x7f0bc400ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:32:02.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:02.712+0000 7f0bca000640 1 -- 172.21.15.90:0/274663066 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bc400ce20 con 0x7f0bcc105700 2024-04-02T14:32:02.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:02.712+0000 7f0bca000640 1 -- 172.21.15.90:0/274663066 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0bc4005020 con 0x7f0bcc105700 2024-04-02T14:32:02.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:02.712+0000 7f0bd2c00640 1 -- 172.21.15.90:0/274663066 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0bcc1a1d60 con 0x7f0bcc105700 2024-04-02T14:32:02.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:02.712+0000 7f0bca000640 1 -- 172.21.15.90:0/274663066 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bc4010720 con 0x7f0bcc105700 2024-04-02T14:32:02.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:02.712+0000 7f0bd2c00640 1 -- 172.21.15.90:0/274663066 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0bcc1a2200 con 0x7f0bcc105700 2024-04-02T14:32:02.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:02.713+0000 7f0bd2c00640 1 -- 172.21.15.90:0/274663066 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0bcc10aef0 con 0x7f0bcc105700 2024-04-02T14:32:02.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:02.715+0000 7f0bca000640 1 -- 172.21.15.90:0/274663066 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f0bc4005540 con 0x7f0bcc105700 2024-04-02T14:32:02.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:02.716+0000 7f0bca000640 1 --2- 172.21.15.90:0/274663066 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f0bb40a3bb0 0x7f0bb40a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:32:02.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:02.717+0000 7f0bca000640 1 -- 172.21.15.90:0/274663066 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 7127+0+0 (secure 0 0 0) 0x7f0bc40d0af0 con 0x7f0bcc105700 2024-04-02T14:32:02.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:02.717+0000 7f0bd1800640 1 --2- 172.21.15.90:0/274663066 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f0bb40a3bb0 0x7f0bb40a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:32:02.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:02.717+0000 7f0bd1800640 1 --2- 172.21.15.90:0/274663066 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f0bb40a3bb0 0x7f0bb40a6070 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f0bc002c8e0 tx=0x7f0bc0005b50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:32:02.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:02.720+0000 7f0bca000640 1 -- 172.21.15.90:0/274663066 <== mon.2 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7f0bc409d8d0 con 0x7f0bcc105700 2024-04-02T14:32:02.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:02.961+0000 7f0bd2c00640 1 -- 172.21.15.90:0/274663066 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0bcc02d370 con 0x7f0bcc105700 2024-04-02T14:32:02.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:02.963+0000 7f0bca000640 1 -- 172.21.15.90:0/274663066 <== mon.2 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+6349 (secure 0 0 0) 0x7f0bc409d1d0 con 0x7f0bcc105700 2024-04-02T14:32:02.964 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-02T14:32:02.964 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":8,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14412,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6837/2852160309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6836","nonce":2852160309},{"type":"v1","addr":"172.21.15.90:6837","nonce":2852160309}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24454,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/1564319431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":1564319431},{"type":"v1","addr":"172.21.15.177:6833","nonce":1564319431}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24461,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6833/1570337372","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":1570337372},{"type":"v1","addr":"172.21.15.92:6833","nonce":1570337372}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24478,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6835/1277124891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6834","nonce":1277124891},{"type":"v1","addr":"172.21.15.90:6835","nonce":1277124891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24503,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6835/860721164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":860721164},{"type":"v1","addr":"172.21.15.92:6835","nonce":860721164}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24511,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/3177244988","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":3177244988},{"type":"v1","addr":"172.21.15.177:6835","nonce":3177244988}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24523,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6837/4223414858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":4223414858},{"type":"v1","addr":"172.21.15.177:6837","nonce":4223414858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[]} 2024-04-02T14:32:02.964 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 8 2024-04-02T14:32:02.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:02.965+0000 7f0bd2c00640 1 -- 172.21.15.90:0/274663066 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f0bb40a3bb0 msgr2=0x7f0bb40a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:32:02.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:02.965+0000 7f0bd2c00640 1 --2- 172.21.15.90:0/274663066 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f0bb40a3bb0 0x7f0bb40a6070 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f0bc002c8e0 tx=0x7f0bc0005b50 comp rx=0 tx=0).stop 2024-04-02T14:32:02.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:02.965+0000 7f0bd2c00640 1 -- 172.21.15.90:0/274663066 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0bcc105700 msgr2=0x7f0bcc1a1400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:32:02.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:02.965+0000 7f0bd2c00640 1 --2- 172.21.15.90:0/274663066 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0bcc105700 0x7f0bcc1a1400 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f0bc4009fd0 tx=0x7f0bc400ef10 comp rx=0 tx=0).stop 2024-04-02T14:32:02.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:02.965+0000 7f0bd2c00640 1 -- 172.21.15.90:0/274663066 shutdown_connections 2024-04-02T14:32:02.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:02.965+0000 7f0bd2c00640 1 --2- 172.21.15.90:0/274663066 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f0bb40a3bb0 0x7f0bb40a6070 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:02.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:02.965+0000 7f0bd2c00640 1 --2- 172.21.15.90:0/274663066 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0bcc105700 0x7f0bcc1a1400 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:02.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:02.965+0000 7f0bd2c00640 1 --2- 172.21.15.90:0/274663066 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0bcc104d40 0x7f0bcc19a390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:02.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:02.965+0000 7f0bd2c00640 1 --2- 172.21.15.90:0/274663066 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0bcc103b50 0x7f0bcc199e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:02.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:02.965+0000 7f0bd2c00640 1 -- 172.21.15.90:0/274663066 >> 172.21.15.90:0/274663066 conn(0x7f0bcc0ff0e0 msgr2=0x7f0bcc1080c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:32:02.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:02.966+0000 7f0bd2c00640 1 -- 172.21.15.90:0/274663066 shutdown_connections 2024-04-02T14:32:02.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:02.966+0000 7f0bd2c00640 1 -- 172.21.15.90:0/274663066 wait complete. 2024-04-02T14:32:03.111 INFO:journalctl@ceph.mds.h.smithi092.stdout:Apr 02 14:32:02 smithi092 systemd[1]: Starting Ceph mds.h for 6449448a-f0fc-11ee-b647-cb9ed24678a4... 2024-04-02T14:32:03.410 INFO:journalctl@ceph.mds.h.smithi092.stdout:Apr 02 14:32:03 smithi092 podman[67098]: 2024-04-02 14:32:03.032166483 +0000 UTC m=+0.017656983 image pull f7301e070545fc23d8e5b7b052a2c15e832a0e57102f1acad8e4f3583709d23f quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534 2024-04-02T14:32:03.411 INFO:journalctl@ceph.mds.h.smithi092.stdout:Apr 02 14:32:03 smithi092 podman[67098]: 2024-04-02 14:32:03.156868409 +0000 UTC m=+0.142358899 container create 1d4fecb3a6534db2a0288f7540ff8facc76d62efb44248ebfbac325a378023f2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mds-h, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240326, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=wip-batrick-testing-20240402.004512-191d963, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True) 2024-04-02T14:32:03.809 INFO:journalctl@ceph.mds.h.smithi092.stdout:Apr 02 14:32:03 smithi092 podman[67098]: 2024-04-02 14:32:03.40952077 +0000 UTC m=+0.395011268 container init 1d4fecb3a6534db2a0288f7540ff8facc76d62efb44248ebfbac325a378023f2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mds-h, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240326, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240402.004512-191d963, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-04-02T14:32:03.809 INFO:journalctl@ceph.mds.h.smithi092.stdout:Apr 02 14:32:03 smithi092 podman[67098]: 2024-04-02 14:32:03.412186126 +0000 UTC m=+0.397676636 container start 1d4fecb3a6534db2a0288f7540ff8facc76d62efb44248ebfbac325a378023f2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mds-h, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240402.004512-191d963, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240326, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-02T14:32:03.809 INFO:journalctl@ceph.mds.h.smithi092.stdout:Apr 02 14:32:03 smithi092 ceph-mds[67117]: -- 172.21.15.92:0/1328648 <== mon.2 v2:172.21.15.92:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5619e03d7a40 con 0x5619e03ab000 2024-04-02T14:32:03.809 INFO:journalctl@ceph.mds.h.smithi092.stdout:Apr 02 14:32:03 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mds-h[67113]: starting mds.h at 2024-04-02T14:32:03.809 INFO:journalctl@ceph.mds.h.smithi092.stdout:Apr 02 14:32:03 smithi092 bash[67098]: 1d4fecb3a6534db2a0288f7540ff8facc76d62efb44248ebfbac325a378023f2 2024-04-02T14:32:03.809 INFO:journalctl@ceph.mds.h.smithi092.stdout:Apr 02 14:32:03 smithi092 systemd[1]: Started Ceph mds.h for 6449448a-f0fc-11ee-b647-cb9ed24678a4. 2024-04-02T14:32:04.110 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:03 smithi092 ceph-mon[36497]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-02T14:32:04.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:03 smithi092 ceph-mon[36497]: from='client.? 172.21.15.90:0/274663066' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-02T14:32:04.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:03 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:32:04.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:03 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:32:04.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:03 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:32:04.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:03 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-02T14:32:04.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:03 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-02T14:32:04.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:03 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:32:04.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:03 smithi177 ceph-mon[36431]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-02T14:32:04.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:03 smithi177 ceph-mon[36431]: from='client.? 172.21.15.90:0/274663066' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-02T14:32:04.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:03 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:32:04.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:03 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:32:04.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:03 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:32:04.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:03 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-02T14:32:04.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:03 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-02T14:32:04.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:03 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:32:04.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:03 smithi090 ceph-mon[31674]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-02T14:32:04.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:03 smithi090 ceph-mon[31674]: from='client.? 172.21.15.90:0/274663066' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-02T14:32:04.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:03 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:32:04.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:03 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:32:04.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:03 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:32:04.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:03 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-02T14:32:04.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:03 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-02T14:32:04.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:03 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:32:04.935 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:04 smithi177 ceph-mon[36431]: Deploying daemon mds.i on smithi177 2024-04-02T14:32:04.935 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:04 smithi177 ceph-mon[36431]: mds.? [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] up:boot 2024-04-02T14:32:04.935 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:04 smithi177 ceph-mon[36431]: fsmap 8 up:standby 2024-04-02T14:32:04.935 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:04 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-02T14:32:04.984 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T14:32:05.071 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:04 smithi090 ceph-mon[31674]: Deploying daemon mds.i on smithi177 2024-04-02T14:32:05.072 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:04 smithi090 ceph-mon[31674]: mds.? [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] up:boot 2024-04-02T14:32:05.072 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:04 smithi090 ceph-mon[31674]: fsmap 8 up:standby 2024-04-02T14:32:05.072 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:04 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-02T14:32:05.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:04 smithi092 ceph-mon[36497]: Deploying daemon mds.i on smithi177 2024-04-02T14:32:05.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:04 smithi092 ceph-mon[36497]: mds.? [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] up:boot 2024-04-02T14:32:05.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:04 smithi092 ceph-mon[36497]: fsmap 8 up:standby 2024-04-02T14:32:05.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:04 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-02T14:32:05.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:05.206+0000 7f1922c00640 1 -- 172.21.15.90:0/2782779118 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f191c108140 msgr2=0x7f191c108560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:32:05.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:05.206+0000 7f1922c00640 1 --2- 172.21.15.90:0/2782779118 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f191c108140 0x7f191c108560 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f190c009900 tx=0x7f190c02bc30 comp rx=0 tx=0).stop 2024-04-02T14:32:05.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:05.206+0000 7f1922c00640 1 -- 172.21.15.90:0/2782779118 shutdown_connections 2024-04-02T14:32:05.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:05.206+0000 7f1922c00640 1 --2- 172.21.15.90:0/2782779118 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f191c109cf0 0x7f191c110580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:05.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:05.206+0000 7f1922c00640 1 --2- 172.21.15.90:0/2782779118 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f191c109330 0x7f191c1097b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:05.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:05.206+0000 7f1922c00640 1 --2- 172.21.15.90:0/2782779118 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f191c108140 0x7f191c108560 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:05.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:05.206+0000 7f1922c00640 1 -- 172.21.15.90:0/2782779118 >> 172.21.15.90:0/2782779118 conn(0x7f191c077110 msgr2=0x7f191c079530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:32:05.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:05.206+0000 7f1922c00640 1 -- 172.21.15.90:0/2782779118 shutdown_connections 2024-04-02T14:32:05.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:05.206+0000 7f1922c00640 1 -- 172.21.15.90:0/2782779118 wait complete. 2024-04-02T14:32:05.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:05.207+0000 7f1922c00640 1 Processor -- start 2024-04-02T14:32:05.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:05.207+0000 7f1922c00640 1 -- start start 2024-04-02T14:32:05.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:05.207+0000 7f1922c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f191c108140 0x7f191c19e420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:32:05.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:05.207+0000 7f1922c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f191c109330 0x7f191c19e960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:32:05.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:05.207+0000 7f1922c00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f191c109cf0 0x7f191c1a59d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:32:05.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:05.207+0000 7f1922c00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f191c113e20 con 0x7f191c109330 2024-04-02T14:32:05.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:05.207+0000 7f1922c00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f191c113ca0 con 0x7f191c109cf0 2024-04-02T14:32:05.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:05.207+0000 7f1922c00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f191c113fa0 con 0x7f191c108140 2024-04-02T14:32:05.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:05.208+0000 7f191be00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f191c108140 0x7f191c19e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:32:05.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:05.208+0000 7f191b400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f191c109330 0x7f191c19e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:32:05.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:05.208+0000 7f191be00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f191c108140 0x7f191c19e420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:47138/0 (socket says 172.21.15.90:47138) 2024-04-02T14:32:05.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:05.208+0000 7f191be00640 1 -- 172.21.15.90:0/2070851195 learned_addr learned my addr 172.21.15.90:0/2070851195 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:32:05.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:05.208+0000 7f1920c00640 1 --2- 172.21.15.90:0/2070851195 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f191c109cf0 0x7f191c1a59d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:32:05.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:05.208+0000 7f191be00640 1 -- 172.21.15.90:0/2070851195 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f191c109cf0 msgr2=0x7f191c1a59d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:32:05.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:05.208+0000 7f191be00640 1 --2- 172.21.15.90:0/2070851195 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f191c109cf0 0x7f191c1a59d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:05.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:05.208+0000 7f191be00640 1 -- 172.21.15.90:0/2070851195 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f191c109330 msgr2=0x7f191c19e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:32:05.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:05.208+0000 7f191be00640 1 --2- 172.21.15.90:0/2070851195 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f191c109330 0x7f191c19e960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:05.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:05.208+0000 7f191be00640 1 -- 172.21.15.90:0/2070851195 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f191c1a6040 con 0x7f191c108140 2024-04-02T14:32:05.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:05.208+0000 7f1920c00640 1 --2- 172.21.15.90:0/2070851195 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f191c109cf0 0x7f191c1a59d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:32:05.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:05.208+0000 7f191be00640 1 --2- 172.21.15.90:0/2070851195 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f191c108140 0x7f191c19e420 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f190c000c00 tx=0x7f190c02c650 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:32:05.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:05.209+0000 7f1918c00640 1 -- 172.21.15.90:0/2070851195 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f190c043020 con 0x7f191c108140 2024-04-02T14:32:05.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:05.209+0000 7f1918c00640 1 -- 172.21.15.90:0/2070851195 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f190c0054e0 con 0x7f191c108140 2024-04-02T14:32:05.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:05.209+0000 7f1922c00640 1 -- 172.21.15.90:0/2070851195 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f191c1a62d0 con 0x7f191c108140 2024-04-02T14:32:05.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:05.209+0000 7f1918c00640 1 -- 172.21.15.90:0/2070851195 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f190c003c40 con 0x7f191c108140 2024-04-02T14:32:05.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:05.209+0000 7f1922c00640 1 -- 172.21.15.90:0/2070851195 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f191c1a6870 con 0x7f191c108140 2024-04-02T14:32:05.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:05.210+0000 7f1922c00640 1 -- 172.21.15.90:0/2070851195 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f191c10f460 con 0x7f191c108140 2024-04-02T14:32:05.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:05.212+0000 7f1918c00640 1 -- 172.21.15.90:0/2070851195 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f190c046020 con 0x7f191c108140 2024-04-02T14:32:05.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:05.213+0000 7f1918c00640 1 --2- 172.21.15.90:0/2070851195 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f19000a3c30 0x7f19000a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:32:05.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:05.213+0000 7f1918c00640 1 -- 172.21.15.90:0/2070851195 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 7127+0+0 (secure 0 0 0) 0x7f190c0f2660 con 0x7f191c108140 2024-04-02T14:32:05.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:05.213+0000 7f191b400640 1 --2- 172.21.15.90:0/2070851195 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f19000a3c30 0x7f19000a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:32:05.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:05.214+0000 7f191b400640 1 --2- 172.21.15.90:0/2070851195 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f19000a3c30 0x7f19000a60f0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f191c19f930 tx=0x7f1908005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:32:05.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:05.216+0000 7f1918c00640 1 -- 172.21.15.90:0/2070851195 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7f190c04d020 con 0x7f191c108140 2024-04-02T14:32:05.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:05.456+0000 7f1922c00640 1 -- 172.21.15.90:0/2070851195 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f191c02d370 con 0x7f191c108140 2024-04-02T14:32:05.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:05.457+0000 7f1918c00640 1 -- 172.21.15.90:0/2070851195 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+7174 (secure 0 0 0) 0x7f190c0bf440 con 0x7f191c108140 2024-04-02T14:32:05.458 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-02T14:32:05.458 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":9,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14412,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6837/2852160309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6836","nonce":2852160309},{"type":"v1","addr":"172.21.15.90:6837","nonce":2852160309}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24454,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/1564319431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":1564319431},{"type":"v1","addr":"172.21.15.177:6833","nonce":1564319431}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24461,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6833/1570337372","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":1570337372},{"type":"v1","addr":"172.21.15.92:6833","nonce":1570337372}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24478,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6835/1277124891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6834","nonce":1277124891},{"type":"v1","addr":"172.21.15.90:6835","nonce":1277124891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24503,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6835/860721164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":860721164},{"type":"v1","addr":"172.21.15.92:6835","nonce":860721164}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24511,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/3177244988","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":3177244988},{"type":"v1","addr":"172.21.15.177:6835","nonce":3177244988}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24523,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6837/4223414858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":4223414858},{"type":"v1","addr":"172.21.15.177:6837","nonce":4223414858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24539,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6837/1270635939","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":1270635939},{"type":"v1","addr":"172.21.15.92:6837","nonce":1270635939}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[]} 2024-04-02T14:32:05.459 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 9 2024-04-02T14:32:05.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:05.460+0000 7f1922c00640 1 -- 172.21.15.90:0/2070851195 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f19000a3c30 msgr2=0x7f19000a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:32:05.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:05.460+0000 7f1922c00640 1 --2- 172.21.15.90:0/2070851195 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f19000a3c30 0x7f19000a60f0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f191c19f930 tx=0x7f1908005f90 comp rx=0 tx=0).stop 2024-04-02T14:32:05.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:05.460+0000 7f1922c00640 1 -- 172.21.15.90:0/2070851195 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f191c108140 msgr2=0x7f191c19e420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:32:05.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:05.460+0000 7f1922c00640 1 --2- 172.21.15.90:0/2070851195 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f191c108140 0x7f191c19e420 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f190c000c00 tx=0x7f190c02c650 comp rx=0 tx=0).stop 2024-04-02T14:32:05.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:05.460+0000 7f1922c00640 1 -- 172.21.15.90:0/2070851195 shutdown_connections 2024-04-02T14:32:05.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:05.460+0000 7f1922c00640 1 --2- 172.21.15.90:0/2070851195 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f19000a3c30 0x7f19000a60f0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:05.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:05.460+0000 7f1922c00640 1 --2- 172.21.15.90:0/2070851195 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f191c109cf0 0x7f191c1a59d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:05.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:05.460+0000 7f1922c00640 1 --2- 172.21.15.90:0/2070851195 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f191c109330 0x7f191c19e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:05.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:05.460+0000 7f1922c00640 1 --2- 172.21.15.90:0/2070851195 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f191c108140 0x7f191c19e420 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:05.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:05.460+0000 7f1922c00640 1 -- 172.21.15.90:0/2070851195 >> 172.21.15.90:0/2070851195 conn(0x7f191c077110 msgr2=0x7f191c10c630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:32:05.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:05.460+0000 7f1922c00640 1 -- 172.21.15.90:0/2070851195 shutdown_connections 2024-04-02T14:32:05.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:05.460+0000 7f1922c00640 1 -- 172.21.15.90:0/2070851195 wait complete. 2024-04-02T14:32:05.823 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 02 14:32:05 smithi177 systemd[1]: Starting Ceph mds.i for 6449448a-f0fc-11ee-b647-cb9ed24678a4... 2024-04-02T14:32:06.075 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:05 smithi177 ceph-mon[36431]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-02T14:32:06.076 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:05 smithi177 ceph-mon[36431]: from='client.? 172.21.15.90:0/2070851195' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-02T14:32:06.076 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 02 14:32:05 smithi177 podman[66666]: 2024-04-02 14:32:05.821734049 +0000 UTC m=+0.017865229 image pull f7301e070545fc23d8e5b7b052a2c15e832a0e57102f1acad8e4f3583709d23f quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534 2024-04-02T14:32:06.076 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 02 14:32:05 smithi177 podman[66666]: 2024-04-02 14:32:05.946527229 +0000 UTC m=+0.142658427 container create 689f6bf561b195e380da9b1fc9af69ee836d5c3417fdd6c4402e0a127d113595 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mds-i, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240326, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=wip-batrick-testing-20240402.004512-191d963, CEPH_POINT_RELEASE=) 2024-04-02T14:32:06.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:05 smithi092 ceph-mon[36497]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-02T14:32:06.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:05 smithi092 ceph-mon[36497]: from='client.? 172.21.15.90:0/2070851195' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-02T14:32:06.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:05 smithi090 ceph-mon[31674]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-02T14:32:06.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:05 smithi090 ceph-mon[31674]: from='client.? 172.21.15.90:0/2070851195' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-02T14:32:06.437 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 02 14:32:06 smithi177 podman[66666]: 2024-04-02 14:32:06.182409326 +0000 UTC m=+0.378540507 container init 689f6bf561b195e380da9b1fc9af69ee836d5c3417fdd6c4402e0a127d113595 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mds-i, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, RELEASE=wip-batrick-testing-20240402.004512-191d963, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240326, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-04-02T14:32:06.437 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 02 14:32:06 smithi177 podman[66666]: 2024-04-02 14:32:06.185044354 +0000 UTC m=+0.381175525 container start 689f6bf561b195e380da9b1fc9af69ee836d5c3417fdd6c4402e0a127d113595 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mds-i, org.label-schema.build-date=20240326, RELEASE=wip-batrick-testing-20240402.004512-191d963, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-04-02T14:32:06.437 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 02 14:32:06 smithi177 ceph-mds[66686]: -- 172.21.15.177:0/2968056968 <== mon.1 v2:172.21.15.177:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5647907d1a40 con 0x5647907a4800 2024-04-02T14:32:06.437 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 02 14:32:06 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mds-i[66682]: starting mds.i at 2024-04-02T14:32:06.437 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 02 14:32:06 smithi177 bash[66666]: 689f6bf561b195e380da9b1fc9af69ee836d5c3417fdd6c4402e0a127d113595 2024-04-02T14:32:06.437 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 02 14:32:06 smithi177 systemd[1]: Started Ceph mds.i for 6449448a-f0fc-11ee-b647-cb9ed24678a4. 2024-04-02T14:32:07.483 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T14:32:07.686 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:07 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:32:07.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:07 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:32:07.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:07 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:32:07.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:07 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-02T14:32:07.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:07 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-02T14:32:07.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:07 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:32:07.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:07 smithi177 ceph-mon[36431]: Deploying daemon mds.g on smithi090 2024-04-02T14:32:07.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:07 smithi177 ceph-mon[36431]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-02T14:32:07.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:07 smithi177 ceph-mon[36431]: mds.? [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] up:boot 2024-04-02T14:32:07.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:07 smithi177 ceph-mon[36431]: fsmap 9 up:standby 2024-04-02T14:32:07.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:07 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-02T14:32:07.689 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:07 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:32:07.689 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:07 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:32:07.689 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:07 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:32:07.689 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:07 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-02T14:32:07.689 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:07 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-02T14:32:07.689 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:07 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:32:07.689 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:07 smithi090 ceph-mon[31674]: Deploying daemon mds.g on smithi090 2024-04-02T14:32:07.689 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:07 smithi090 ceph-mon[31674]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-02T14:32:07.689 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:07 smithi090 ceph-mon[31674]: mds.? [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] up:boot 2024-04-02T14:32:07.689 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:07 smithi090 ceph-mon[31674]: fsmap 9 up:standby 2024-04-02T14:32:07.689 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:07 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-02T14:32:07.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:07.702+0000 7fbeaf000640 1 -- 172.21.15.90:0/2319873319 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbea8104f90 msgr2=0x7fbea8107380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:32:07.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:07.702+0000 7fbeaf000640 1 --2- 172.21.15.90:0/2319873319 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbea8104f90 0x7fbea8107380 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fbe90009900 tx=0x7fbe9002bc30 comp rx=0 tx=0).stop 2024-04-02T14:32:07.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:07.702+0000 7fbeaf000640 1 -- 172.21.15.90:0/2319873319 shutdown_connections 2024-04-02T14:32:07.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:07.702+0000 7fbeaf000640 1 --2- 172.21.15.90:0/2319873319 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbea81078c0 0x7fbea8109cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:07.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:07.702+0000 7fbeaf000640 1 --2- 172.21.15.90:0/2319873319 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbea8104f90 0x7fbea8107380 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:07.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:07.702+0000 7fbeaf000640 1 --2- 172.21.15.90:0/2319873319 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbea8102660 0x7fbea8104a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:07.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:07.702+0000 7fbeaf000640 1 -- 172.21.15.90:0/2319873319 >> 172.21.15.90:0/2319873319 conn(0x7fbea80fbfb0 msgr2=0x7fbea80fe3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:32:07.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:07.702+0000 7fbeaf000640 1 -- 172.21.15.90:0/2319873319 shutdown_connections 2024-04-02T14:32:07.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:07.702+0000 7fbeaf000640 1 -- 172.21.15.90:0/2319873319 wait complete. 2024-04-02T14:32:07.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:07.703+0000 7fbeaf000640 1 Processor -- start 2024-04-02T14:32:07.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:07.704+0000 7fbeaf000640 1 -- start start 2024-04-02T14:32:07.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:07.704+0000 7fbeaf000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbea8102660 0x7fbea8101fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:32:07.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:07.704+0000 7fbeaf000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbea8104f90 0x7fbea8100610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:32:07.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:07.704+0000 7fbeaf000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbea81078c0 0x7fbea8100b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:32:07.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:07.704+0000 7fbeaf000640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fbea810d7c0 con 0x7fbea8102660 2024-04-02T14:32:07.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:07.704+0000 7fbeaf000640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fbea810d640 con 0x7fbea8104f90 2024-04-02T14:32:07.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:07.704+0000 7fbeaf000640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fbea810d940 con 0x7fbea81078c0 2024-04-02T14:32:07.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:07.704+0000 7fbea7e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbea8102660 0x7fbea8101fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:32:07.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:07.704+0000 7fbea7e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbea8102660 0x7fbea8101fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:59130/0 (socket says 172.21.15.90:59130) 2024-04-02T14:32:07.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:07.704+0000 7fbea7e00640 1 -- 172.21.15.90:0/736581651 learned_addr learned my addr 172.21.15.90:0/736581651 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:32:07.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:07.704+0000 7fbead000640 1 --2- 172.21.15.90:0/736581651 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbea81078c0 0x7fbea8100b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:32:07.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:07.704+0000 7fbea7400640 1 --2- 172.21.15.90:0/736581651 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbea8104f90 0x7fbea8100610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:32:07.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:07.705+0000 7fbea7e00640 1 -- 172.21.15.90:0/736581651 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbea81078c0 msgr2=0x7fbea8100b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:32:07.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:07.705+0000 7fbea7e00640 1 --2- 172.21.15.90:0/736581651 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbea81078c0 0x7fbea8100b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:07.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:07.705+0000 7fbea7e00640 1 -- 172.21.15.90:0/736581651 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbea8104f90 msgr2=0x7fbea8100610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:32:07.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:07.705+0000 7fbea7e00640 1 --2- 172.21.15.90:0/736581651 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbea8104f90 0x7fbea8100610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:07.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:07.705+0000 7fbea7e00640 1 -- 172.21.15.90:0/736581651 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbea81013e0 con 0x7fbea8102660 2024-04-02T14:32:07.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:07.705+0000 7fbea7e00640 1 --2- 172.21.15.90:0/736581651 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbea8102660 0x7fbea8101fd0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fbe9800ebc0 tx=0x7fbe9800c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:32:07.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:07.705+0000 7fbea4c00640 1 -- 172.21.15.90:0/736581651 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe98010630 con 0x7fbea8102660 2024-04-02T14:32:07.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:07.705+0000 7fbea4c00640 1 -- 172.21.15.90:0/736581651 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbe98010cb0 con 0x7fbea8102660 2024-04-02T14:32:07.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:07.705+0000 7fbea4c00640 1 -- 172.21.15.90:0/736581651 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe98018bd0 con 0x7fbea8102660 2024-04-02T14:32:07.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:07.705+0000 7fbeaf000640 1 -- 172.21.15.90:0/736581651 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbea81a1f90 con 0x7fbea8102660 2024-04-02T14:32:07.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:07.706+0000 7fbeaf000640 1 -- 172.21.15.90:0/736581651 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbea81a2340 con 0x7fbea8102660 2024-04-02T14:32:07.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:07.707+0000 7fbeaf000640 1 -- 172.21.15.90:0/736581651 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbea80fdaa0 con 0x7fbea8102660 2024-04-02T14:32:07.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:07.709+0000 7fbea4c00640 1 -- 172.21.15.90:0/736581651 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fbe980107d0 con 0x7fbea8102660 2024-04-02T14:32:07.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:07.710+0000 7fbea4c00640 1 --2- 172.21.15.90:0/736581651 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fbe8c0a3bb0 0x7fbe8c0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:32:07.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:07.710+0000 7fbea4c00640 1 -- 172.21.15.90:0/736581651 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 7127+0+0 (secure 0 0 0) 0x7fbe980d04c0 con 0x7fbea8102660 2024-04-02T14:32:07.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:07.710+0000 7fbea7400640 1 --2- 172.21.15.90:0/736581651 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fbe8c0a3bb0 0x7fbe8c0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:32:07.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:07.711+0000 7fbea7400640 1 --2- 172.21.15.90:0/736581651 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fbe8c0a3bb0 0x7fbe8c0a6070 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fbe90000c00 tx=0x7fbe90005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:32:07.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:07.715+0000 7fbea4c00640 1 -- 172.21.15.90:0/736581651 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7fbe9809ccf0 con 0x7fbea8102660 2024-04-02T14:32:07.806 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:07 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:32:07.806 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:07 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:32:07.806 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:07 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:32:07.806 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:07 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-02T14:32:07.806 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:07 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-02T14:32:07.806 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:07 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:32:07.807 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:07 smithi092 ceph-mon[36497]: Deploying daemon mds.g on smithi090 2024-04-02T14:32:07.807 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:07 smithi092 ceph-mon[36497]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-02T14:32:07.807 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:07 smithi092 ceph-mon[36497]: mds.? [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] up:boot 2024-04-02T14:32:07.807 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:07 smithi092 ceph-mon[36497]: fsmap 9 up:standby 2024-04-02T14:32:07.807 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:07 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-02T14:32:07.969 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:07.968+0000 7fbeaf000640 1 -- 172.21.15.90:0/736581651 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbea802d370 con 0x7fbea8102660 2024-04-02T14:32:07.970 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:07.969+0000 7fbea4c00640 1 -- 172.21.15.90:0/736581651 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) ==== 94+0+8001 (secure 0 0 0) 0x7fbe980a0ff0 con 0x7fbea8102660 2024-04-02T14:32:07.970 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-02T14:32:07.970 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14412,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6837/2852160309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6836","nonce":2852160309},{"type":"v1","addr":"172.21.15.90:6837","nonce":2852160309}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24454,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/1564319431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":1564319431},{"type":"v1","addr":"172.21.15.177:6833","nonce":1564319431}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24461,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6833/1570337372","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":1570337372},{"type":"v1","addr":"172.21.15.92:6833","nonce":1570337372}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24478,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6835/1277124891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6834","nonce":1277124891},{"type":"v1","addr":"172.21.15.90:6835","nonce":1277124891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24503,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6835/860721164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":860721164},{"type":"v1","addr":"172.21.15.92:6835","nonce":860721164}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24511,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/3177244988","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":3177244988},{"type":"v1","addr":"172.21.15.177:6835","nonce":3177244988}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24523,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6837/4223414858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":4223414858},{"type":"v1","addr":"172.21.15.177:6837","nonce":4223414858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24539,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6837/1270635939","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":1270635939},{"type":"v1","addr":"172.21.15.92:6837","nonce":1270635939}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24547,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6839/664895663","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":664895663},{"type":"v1","addr":"172.21.15.177:6839","nonce":664895663}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-04-02T14:32:07.970 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 10 2024-04-02T14:32:07.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:07.971+0000 7fbeaf000640 1 -- 172.21.15.90:0/736581651 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fbe8c0a3bb0 msgr2=0x7fbe8c0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:32:07.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:07.971+0000 7fbeaf000640 1 --2- 172.21.15.90:0/736581651 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fbe8c0a3bb0 0x7fbe8c0a6070 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fbe90000c00 tx=0x7fbe90005fd0 comp rx=0 tx=0).stop 2024-04-02T14:32:07.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:07.971+0000 7fbeaf000640 1 -- 172.21.15.90:0/736581651 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbea8102660 msgr2=0x7fbea8101fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:32:07.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:07.971+0000 7fbeaf000640 1 --2- 172.21.15.90:0/736581651 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbea8102660 0x7fbea8101fd0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fbe9800ebc0 tx=0x7fbe9800c6a0 comp rx=0 tx=0).stop 2024-04-02T14:32:07.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:07.972+0000 7fbeaf000640 1 -- 172.21.15.90:0/736581651 shutdown_connections 2024-04-02T14:32:07.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:07.972+0000 7fbeaf000640 1 --2- 172.21.15.90:0/736581651 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fbe8c0a3bb0 0x7fbe8c0a6070 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:07.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:07.972+0000 7fbeaf000640 1 --2- 172.21.15.90:0/736581651 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbea81078c0 0x7fbea8100b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:07.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:07.972+0000 7fbeaf000640 1 --2- 172.21.15.90:0/736581651 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbea8104f90 0x7fbea8100610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:07.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:07.972+0000 7fbeaf000640 1 --2- 172.21.15.90:0/736581651 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbea8102660 0x7fbea8101fd0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:07.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:07.972+0000 7fbeaf000640 1 -- 172.21.15.90:0/736581651 >> 172.21.15.90:0/736581651 conn(0x7fbea80fbfb0 msgr2=0x7fbea8102f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:32:07.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:07.972+0000 7fbeaf000640 1 -- 172.21.15.90:0/736581651 shutdown_connections 2024-04-02T14:32:07.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:07.972+0000 7fbeaf000640 1 -- 172.21.15.90:0/736581651 wait complete. 2024-04-02T14:32:08.689 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:08 smithi090 ceph-mon[31674]: from='client.? 172.21.15.90:0/736581651' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-02T14:32:08.861 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:08 smithi092 ceph-mon[36497]: from='client.? 172.21.15.90:0/736581651' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-02T14:32:08.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:08 smithi177 ceph-mon[36431]: from='client.? 172.21.15.90:0/736581651' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-02T14:32:09.842 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:09 smithi090 ceph-mon[31674]: pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-02T14:32:09.860 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:09 smithi092 ceph-mon[36497]: pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-02T14:32:09.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:09 smithi177 ceph-mon[36431]: pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-02T14:32:09.995 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T14:32:10.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:10.198+0000 7fa02b200640 1 -- 172.21.15.90:0/3326683505 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa02c101970 msgr2=0x7fa02c101d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:32:10.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:10.198+0000 7fa02b200640 1 --2- 172.21.15.90:0/3326683505 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa02c101970 0x7fa02c101d90 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fa01c009900 tx=0x7fa01c02bc30 comp rx=0 tx=0).stop 2024-04-02T14:32:10.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:10.198+0000 7fa02b200640 1 -- 172.21.15.90:0/3326683505 shutdown_connections 2024-04-02T14:32:10.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:10.198+0000 7fa02b200640 1 --2- 172.21.15.90:0/3326683505 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa02c103520 0x7fa02c109db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:10.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:10.198+0000 7fa02b200640 1 --2- 172.21.15.90:0/3326683505 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa02c102b60 0x7fa02c102fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:10.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:10.198+0000 7fa02b200640 1 --2- 172.21.15.90:0/3326683505 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa02c101970 0x7fa02c101d90 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:10.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:10.198+0000 7fa02b200640 1 -- 172.21.15.90:0/3326683505 >> 172.21.15.90:0/3326683505 conn(0x7fa02c0fcee0 msgr2=0x7fa02c0ff320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:32:10.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:10.198+0000 7fa02b200640 1 -- 172.21.15.90:0/3326683505 shutdown_connections 2024-04-02T14:32:10.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:10.198+0000 7fa02b200640 1 -- 172.21.15.90:0/3326683505 wait complete. 2024-04-02T14:32:10.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:10.199+0000 7fa02b200640 1 Processor -- start 2024-04-02T14:32:10.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:10.199+0000 7fa02b200640 1 -- start start 2024-04-02T14:32:10.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:10.199+0000 7fa02b200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa02c101970 0x7fa02c0758b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:32:10.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:10.199+0000 7fa02b200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa02c102b60 0x7fa02c073ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:32:10.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:10.199+0000 7fa02b200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa02c103520 0x7fa02c074430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:32:10.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:10.199+0000 7fa02b200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fa02c10d5c0 con 0x7fa02c101970 2024-04-02T14:32:10.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:10.199+0000 7fa02b200640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fa02c10d440 con 0x7fa02c103520 2024-04-02T14:32:10.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:10.199+0000 7fa02b200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fa02c10d740 con 0x7fa02c102b60 2024-04-02T14:32:10.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:10.199+0000 7fa029e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa02c101970 0x7fa02c0758b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:32:10.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:10.199+0000 7fa029e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa02c101970 0x7fa02c0758b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:59162/0 (socket says 172.21.15.90:59162) 2024-04-02T14:32:10.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:10.199+0000 7fa029e00640 1 -- 172.21.15.90:0/649286526 learned_addr learned my addr 172.21.15.90:0/649286526 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:32:10.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:10.199+0000 7fa02a800640 1 --2- 172.21.15.90:0/649286526 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa02c103520 0x7fa02c074430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:32:10.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:10.199+0000 7fa029400640 1 --2- 172.21.15.90:0/649286526 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa02c102b60 0x7fa02c073ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:32:10.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:10.200+0000 7fa029e00640 1 -- 172.21.15.90:0/649286526 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa02c102b60 msgr2=0x7fa02c073ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:32:10.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:10.200+0000 7fa029e00640 1 --2- 172.21.15.90:0/649286526 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa02c102b60 0x7fa02c073ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:10.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:10.200+0000 7fa029e00640 1 -- 172.21.15.90:0/649286526 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa02c103520 msgr2=0x7fa02c074430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:32:10.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:10.200+0000 7fa029e00640 1 --2- 172.21.15.90:0/649286526 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa02c103520 0x7fa02c074430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:10.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:10.200+0000 7fa029e00640 1 -- 172.21.15.90:0/649286526 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa02c074cc0 con 0x7fa02c101970 2024-04-02T14:32:10.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:10.200+0000 7fa02a800640 1 --2- 172.21.15.90:0/649286526 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa02c103520 0x7fa02c074430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:32:10.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:10.200+0000 7fa029e00640 1 --2- 172.21.15.90:0/649286526 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa02c101970 0x7fa02c0758b0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fa01c000c00 tx=0x7fa01c005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:32:10.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:10.200+0000 7fa022a00640 1 -- 172.21.15.90:0/649286526 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa01c041070 con 0x7fa02c101970 2024-04-02T14:32:10.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:10.200+0000 7fa022a00640 1 -- 172.21.15.90:0/649286526 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa01c0044d0 con 0x7fa02c101970 2024-04-02T14:32:10.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:10.200+0000 7fa02b200640 1 -- 172.21.15.90:0/649286526 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa02c1a1d50 con 0x7fa02c101970 2024-04-02T14:32:10.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:10.200+0000 7fa022a00640 1 -- 172.21.15.90:0/649286526 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa01c034440 con 0x7fa02c101970 2024-04-02T14:32:10.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:10.201+0000 7fa02b200640 1 -- 172.21.15.90:0/649286526 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa02c1a2190 con 0x7fa02c101970 2024-04-02T14:32:10.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:10.202+0000 7fa02b200640 1 -- 172.21.15.90:0/649286526 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa02c108cb0 con 0x7fa02c101970 2024-04-02T14:32:10.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:10.204+0000 7fa022a00640 1 -- 172.21.15.90:0/649286526 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fa01c044070 con 0x7fa02c101970 2024-04-02T14:32:10.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:10.206+0000 7fa022a00640 1 --2- 172.21.15.90:0/649286526 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fa0100a3bb0 0x7fa0100a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:32:10.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:10.207+0000 7fa022a00640 1 -- 172.21.15.90:0/649286526 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 7127+0+0 (secure 0 0 0) 0x7fa01c0f1c70 con 0x7fa02c101970 2024-04-02T14:32:10.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:10.207+0000 7fa029400640 1 --2- 172.21.15.90:0/649286526 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fa0100a3bb0 0x7fa0100a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:32:10.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:10.207+0000 7fa029400640 1 --2- 172.21.15.90:0/649286526 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fa0100a3bb0 0x7fa0100a6070 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fa0180095f0 tx=0x7fa018005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:32:10.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:10.210+0000 7fa022a00640 1 -- 172.21.15.90:0/649286526 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7fa01c0be4a0 con 0x7fa02c101970 2024-04-02T14:32:10.438 INFO:journalctl@ceph.mds.g.smithi090.stdout:Apr 02 14:32:10 smithi090 systemd[1]: Started Ceph mds.g for 6449448a-f0fc-11ee-b647-cb9ed24678a4. 2024-04-02T14:32:10.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:10.468+0000 7fa02b200640 1 -- 172.21.15.90:0/649286526 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa02c02d530 con 0x7fa02c101970 2024-04-02T14:32:10.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:10.469+0000 7fa022a00640 1 -- 172.21.15.90:0/649286526 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) ==== 94+0+8001 (secure 0 0 0) 0x7fa01c0c27a0 con 0x7fa02c101970 2024-04-02T14:32:10.471 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-02T14:32:10.471 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14412,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6837/2852160309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6836","nonce":2852160309},{"type":"v1","addr":"172.21.15.90:6837","nonce":2852160309}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24454,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/1564319431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":1564319431},{"type":"v1","addr":"172.21.15.177:6833","nonce":1564319431}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24461,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6833/1570337372","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":1570337372},{"type":"v1","addr":"172.21.15.92:6833","nonce":1570337372}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24478,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6835/1277124891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6834","nonce":1277124891},{"type":"v1","addr":"172.21.15.90:6835","nonce":1277124891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24503,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6835/860721164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":860721164},{"type":"v1","addr":"172.21.15.92:6835","nonce":860721164}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24511,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/3177244988","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":3177244988},{"type":"v1","addr":"172.21.15.177:6835","nonce":3177244988}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24523,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6837/4223414858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":4223414858},{"type":"v1","addr":"172.21.15.177:6837","nonce":4223414858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24539,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6837/1270635939","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":1270635939},{"type":"v1","addr":"172.21.15.92:6837","nonce":1270635939}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24547,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6839/664895663","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":664895663},{"type":"v1","addr":"172.21.15.177:6839","nonce":664895663}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-04-02T14:32:10.471 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 10 2024-04-02T14:32:10.473 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:10.471+0000 7fa02b200640 1 -- 172.21.15.90:0/649286526 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fa0100a3bb0 msgr2=0x7fa0100a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:32:10.473 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:10.471+0000 7fa02b200640 1 --2- 172.21.15.90:0/649286526 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fa0100a3bb0 0x7fa0100a6070 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fa0180095f0 tx=0x7fa018005f90 comp rx=0 tx=0).stop 2024-04-02T14:32:10.473 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:10.472+0000 7fa02b200640 1 -- 172.21.15.90:0/649286526 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa02c101970 msgr2=0x7fa02c0758b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:32:10.473 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:10.472+0000 7fa02b200640 1 --2- 172.21.15.90:0/649286526 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa02c101970 0x7fa02c0758b0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fa01c000c00 tx=0x7fa01c005d50 comp rx=0 tx=0).stop 2024-04-02T14:32:10.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:10.472+0000 7fa02b200640 1 -- 172.21.15.90:0/649286526 shutdown_connections 2024-04-02T14:32:10.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:10.472+0000 7fa02b200640 1 --2- 172.21.15.90:0/649286526 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fa0100a3bb0 0x7fa0100a6070 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:10.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:10.472+0000 7fa02b200640 1 --2- 172.21.15.90:0/649286526 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa02c103520 0x7fa02c074430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:10.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:10.472+0000 7fa02b200640 1 --2- 172.21.15.90:0/649286526 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa02c102b60 0x7fa02c073ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:10.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:10.472+0000 7fa02b200640 1 --2- 172.21.15.90:0/649286526 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa02c101970 0x7fa02c0758b0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:10.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:10.472+0000 7fa02b200640 1 -- 172.21.15.90:0/649286526 >> 172.21.15.90:0/649286526 conn(0x7fa02c0fcee0 msgr2=0x7fa02c105e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:32:10.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:10.472+0000 7fa02b200640 1 -- 172.21.15.90:0/649286526 shutdown_connections 2024-04-02T14:32:10.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:10.472+0000 7fa02b200640 1 -- 172.21.15.90:0/649286526 wait complete. 2024-04-02T14:32:11.555 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:11 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:32:11.555 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:11 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:32:11.555 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:11 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:32:11.555 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:11 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-02T14:32:11.555 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:11 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-02T14:32:11.555 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:11 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:32:11.556 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:11 smithi090 ceph-mon[31674]: Deploying daemon mds.a on smithi090 2024-04-02T14:32:11.556 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:11 smithi090 ceph-mon[31674]: from='client.? 172.21.15.90:0/649286526' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-02T14:32:11.556 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:11 smithi090 ceph-mon[31674]: mds.? [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] up:boot 2024-04-02T14:32:11.556 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:11 smithi090 ceph-mon[31674]: fsmap 10 up:standby 2024-04-02T14:32:11.556 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:11 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-02T14:32:11.556 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:11 smithi090 ceph-mon[31674]: pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-02T14:32:11.611 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:11 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:32:11.611 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:11 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:32:11.611 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:11 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:32:11.611 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:11 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-02T14:32:11.611 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:11 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-02T14:32:11.611 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:11 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:32:11.611 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:11 smithi092 ceph-mon[36497]: Deploying daemon mds.a on smithi090 2024-04-02T14:32:11.612 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:11 smithi092 ceph-mon[36497]: from='client.? 172.21.15.90:0/649286526' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-02T14:32:11.612 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:11 smithi092 ceph-mon[36497]: mds.? [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] up:boot 2024-04-02T14:32:11.612 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:11 smithi092 ceph-mon[36497]: fsmap 10 up:standby 2024-04-02T14:32:11.612 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:11 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-02T14:32:11.612 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:11 smithi092 ceph-mon[36497]: pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-02T14:32:11.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:11 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:32:11.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:11 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:32:11.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:11 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:32:11.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:11 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-02T14:32:11.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:11 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-02T14:32:11.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:11 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:32:11.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:11 smithi177 ceph-mon[36431]: Deploying daemon mds.a on smithi090 2024-04-02T14:32:11.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:11 smithi177 ceph-mon[36431]: from='client.? 172.21.15.90:0/649286526' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-02T14:32:11.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:11 smithi177 ceph-mon[36431]: mds.? [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] up:boot 2024-04-02T14:32:11.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:11 smithi177 ceph-mon[36431]: fsmap 10 up:standby 2024-04-02T14:32:11.688 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:11 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-02T14:32:11.688 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:11 smithi177 ceph-mon[36431]: pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-02T14:32:12.494 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T14:32:12.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:12.962+0000 7f47bae00640 1 -- 172.21.15.90:0/1506438576 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f47b41058f0 msgr2=0x7f47b410c180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:32:12.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:12.962+0000 7f47bae00640 1 --2- 172.21.15.90:0/1506438576 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f47b41058f0 0x7f47b410c180 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f47ac009ca0 tx=0x7f47ac02c0f0 comp rx=0 tx=0).stop 2024-04-02T14:32:12.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:12.962+0000 7f47bae00640 1 -- 172.21.15.90:0/1506438576 shutdown_connections 2024-04-02T14:32:12.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:12.962+0000 7f47bae00640 1 --2- 172.21.15.90:0/1506438576 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f47b41058f0 0x7f47b410c180 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:12.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:12.962+0000 7f47bae00640 1 --2- 172.21.15.90:0/1506438576 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f47b4104f30 0x7f47b41053b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:12.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:12.962+0000 7f47bae00640 1 --2- 172.21.15.90:0/1506438576 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f47b4103d40 0x7f47b4104160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:12.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:12.962+0000 7f47bae00640 1 -- 172.21.15.90:0/1506438576 >> 172.21.15.90:0/1506438576 conn(0x7f47b40ff2f0 msgr2=0x7f47b4101710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:32:12.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:12.963+0000 7f47bae00640 1 -- 172.21.15.90:0/1506438576 shutdown_connections 2024-04-02T14:32:12.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:12.963+0000 7f47bae00640 1 -- 172.21.15.90:0/1506438576 wait complete. 2024-04-02T14:32:12.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:12.965+0000 7f47bae00640 1 Processor -- start 2024-04-02T14:32:12.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:12.965+0000 7f47bae00640 1 -- start start 2024-04-02T14:32:12.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:12.966+0000 7f47bae00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f47b4103d40 0x7f47b419a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:32:12.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:12.966+0000 7f47bae00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f47b4104f30 0x7f47b419a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:32:12.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:12.966+0000 7f47bae00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f47b41058f0 0x7f47b4077f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:32:12.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:12.966+0000 7f47bae00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f47b410f9a0 con 0x7f47b4104f30 2024-04-02T14:32:12.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:12.966+0000 7f47bae00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f47b410f820 con 0x7f47b41058f0 2024-04-02T14:32:12.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:12.966+0000 7f47b8e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f47b41058f0 0x7f47b4077f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:32:12.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:12.967+0000 7f47b8e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f47b41058f0 0x7f47b4077f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:58914/0 (socket says 172.21.15.90:58914) 2024-04-02T14:32:12.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:12.967+0000 7f47b8e00640 1 -- 172.21.15.90:0/2258884613 learned_addr learned my addr 172.21.15.90:0/2258884613 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:32:12.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:12.967+0000 7f47bae00640 1 -- 172.21.15.90:0/2258884613 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f47b410fb20 con 0x7f47b4103d40 2024-04-02T14:32:12.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:12.967+0000 7f47b3e00640 1 --2- 172.21.15.90:0/2258884613 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f47b4103d40 0x7f47b419a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:32:12.969 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:12.967+0000 7f47b3400640 1 --2- 172.21.15.90:0/2258884613 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f47b4104f30 0x7f47b419a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:32:12.969 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:12.968+0000 7f47b8e00640 1 -- 172.21.15.90:0/2258884613 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f47b4103d40 msgr2=0x7f47b419a010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:32:12.969 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:12.968+0000 7f47b8e00640 1 --2- 172.21.15.90:0/2258884613 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f47b4103d40 0x7f47b419a010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:12.969 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:12.968+0000 7f47b8e00640 1 -- 172.21.15.90:0/2258884613 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f47b4104f30 msgr2=0x7f47b419a550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:32:12.969 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:12.968+0000 7f47b8e00640 1 --2- 172.21.15.90:0/2258884613 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f47b4104f30 0x7f47b419a550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:12.969 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:12.968+0000 7f47b8e00640 1 -- 172.21.15.90:0/2258884613 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47b40785c0 con 0x7f47b41058f0 2024-04-02T14:32:12.969 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:12.968+0000 7f47b3e00640 1 --2- 172.21.15.90:0/2258884613 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f47b4103d40 0x7f47b419a010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:32:12.969 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:12.968+0000 7f47b3400640 1 --2- 172.21.15.90:0/2258884613 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f47b4104f30 0x7f47b419a550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:32:12.970 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:12.969+0000 7f47b8e00640 1 --2- 172.21.15.90:0/2258884613 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f47b41058f0 0x7f47b4077f50 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f47ac009bf0 tx=0x7f47ac032ea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:32:12.970 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:12.969+0000 7f47b0c00640 1 -- 172.21.15.90:0/2258884613 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47ac041020 con 0x7f47b41058f0 2024-04-02T14:32:12.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:12.970+0000 7f47b0c00640 1 -- 172.21.15.90:0/2258884613 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f47ac034cd0 con 0x7f47b41058f0 2024-04-02T14:32:12.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:12.970+0000 7f47b0c00640 1 -- 172.21.15.90:0/2258884613 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47ac039040 con 0x7f47b41058f0 2024-04-02T14:32:12.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:12.970+0000 7f47bae00640 1 -- 172.21.15.90:0/2258884613 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f47b4078850 con 0x7f47b41058f0 2024-04-02T14:32:12.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:12.970+0000 7f47bae00640 1 -- 172.21.15.90:0/2258884613 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47b4078cf0 con 0x7f47b41058f0 2024-04-02T14:32:12.976 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:12.975+0000 7f47bae00640 1 -- 172.21.15.90:0/2258884613 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f47940052c0 con 0x7f47b41058f0 2024-04-02T14:32:12.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:12.982+0000 7f47b0c00640 1 -- 172.21.15.90:0/2258884613 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f47ac044020 con 0x7f47b41058f0 2024-04-02T14:32:12.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:12.984+0000 7f47b0c00640 1 --2- 172.21.15.90:0/2258884613 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f47980a3d00 0x7f47980a61c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:32:12.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:12.985+0000 7f47b0c00640 1 -- 172.21.15.90:0/2258884613 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 7127+0+0 (secure 0 0 0) 0x7f47ac0f1d50 con 0x7f47b41058f0 2024-04-02T14:32:12.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:12.986+0000 7f47b3e00640 1 --2- 172.21.15.90:0/2258884613 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f47980a3d00 0x7f47980a61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:32:12.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:12.987+0000 7f47b3e00640 1 --2- 172.21.15.90:0/2258884613 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f47980a3d00 0x7f47980a61c0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f47b4104d90 tx=0x7f47a4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:32:12.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:12.988+0000 7f47b0c00640 1 -- 172.21.15.90:0/2258884613 <== mon.2 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7f47ac04b020 con 0x7f47b41058f0 2024-04-02T14:32:13.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:13.241+0000 7f47bae00640 1 -- 172.21.15.90:0/2258884613 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f47940055b0 con 0x7f47b41058f0 2024-04-02T14:32:13.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:13.241+0000 7f47b0c00640 1 -- 172.21.15.90:0/2258884613 <== mon.2 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) ==== 94+0+8827 (secure 0 0 0) 0x7f47ac0098b0 con 0x7f47b41058f0 2024-04-02T14:32:13.243 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-02T14:32:13.243 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14412,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6837/2852160309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6836","nonce":2852160309},{"type":"v1","addr":"172.21.15.90:6837","nonce":2852160309}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24454,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/1564319431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":1564319431},{"type":"v1","addr":"172.21.15.177:6833","nonce":1564319431}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24461,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6833/1570337372","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":1570337372},{"type":"v1","addr":"172.21.15.92:6833","nonce":1570337372}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24478,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6835/1277124891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6834","nonce":1277124891},{"type":"v1","addr":"172.21.15.90:6835","nonce":1277124891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24503,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6835/860721164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":860721164},{"type":"v1","addr":"172.21.15.92:6835","nonce":860721164}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24511,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/3177244988","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":3177244988},{"type":"v1","addr":"172.21.15.177:6835","nonce":3177244988}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24523,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6837/4223414858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":4223414858},{"type":"v1","addr":"172.21.15.177:6837","nonce":4223414858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24539,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6837/1270635939","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":1270635939},{"type":"v1","addr":"172.21.15.92:6837","nonce":1270635939}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24547,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6839/664895663","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":664895663},{"type":"v1","addr":"172.21.15.177:6839","nonce":664895663}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24560,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6839/1498915870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6838","nonce":1498915870},{"type":"v1","addr":"172.21.15.90:6839","nonce":1498915870}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[]} 2024-04-02T14:32:13.243 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 11 2024-04-02T14:32:13.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:13.244+0000 7f47bae00640 1 -- 172.21.15.90:0/2258884613 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f47980a3d00 msgr2=0x7f47980a61c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:32:13.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:13.244+0000 7f47bae00640 1 --2- 172.21.15.90:0/2258884613 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f47980a3d00 0x7f47980a61c0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f47b4104d90 tx=0x7f47a4005f90 comp rx=0 tx=0).stop 2024-04-02T14:32:13.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:13.244+0000 7f47bae00640 1 -- 172.21.15.90:0/2258884613 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f47b41058f0 msgr2=0x7f47b4077f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:32:13.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:13.244+0000 7f47bae00640 1 --2- 172.21.15.90:0/2258884613 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f47b41058f0 0x7f47b4077f50 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f47ac009bf0 tx=0x7f47ac032ea0 comp rx=0 tx=0).stop 2024-04-02T14:32:13.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:13.244+0000 7f47bae00640 1 -- 172.21.15.90:0/2258884613 shutdown_connections 2024-04-02T14:32:13.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:13.244+0000 7f47bae00640 1 --2- 172.21.15.90:0/2258884613 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f47980a3d00 0x7f47980a61c0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:13.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:13.244+0000 7f47bae00640 1 --2- 172.21.15.90:0/2258884613 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f47b41058f0 0x7f47b4077f50 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:13.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:13.244+0000 7f47bae00640 1 --2- 172.21.15.90:0/2258884613 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f47b4104f30 0x7f47b419a550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:13.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:13.244+0000 7f47bae00640 1 --2- 172.21.15.90:0/2258884613 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f47b4103d40 0x7f47b419a010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:13.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:13.244+0000 7f47bae00640 1 -- 172.21.15.90:0/2258884613 >> 172.21.15.90:0/2258884613 conn(0x7f47b40ff2f0 msgr2=0x7f47b4108230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:32:13.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:13.245+0000 7f47bae00640 1 -- 172.21.15.90:0/2258884613 shutdown_connections 2024-04-02T14:32:13.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:13.245+0000 7f47bae00640 1 -- 172.21.15.90:0/2258884613 wait complete. 2024-04-02T14:32:13.828 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:13 smithi090 ceph-mon[31674]: pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-02T14:32:13.829 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:13 smithi090 ceph-mon[31674]: from='client.? 172.21.15.90:0/2258884613' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-02T14:32:14.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:13 smithi092 ceph-mon[36497]: pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-02T14:32:14.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:13 smithi092 ceph-mon[36497]: from='client.? 172.21.15.90:0/2258884613' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-02T14:32:14.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:13 smithi177 ceph-mon[36431]: pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-02T14:32:14.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:13 smithi177 ceph-mon[36431]: from='client.? 172.21.15.90:0/2258884613' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-02T14:32:14.189 INFO:journalctl@ceph.mds.a.smithi090.stdout:Apr 02 14:32:13 smithi090 systemd[1]: Started Ceph mds.a for 6449448a-f0fc-11ee-b647-cb9ed24678a4. 2024-04-02T14:32:15.264 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T14:32:15.353 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:15 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:32:15.353 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:15 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:32:15.353 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:15 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:32:15.353 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:15 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-02T14:32:15.353 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:15 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-02T14:32:15.353 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:15 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:32:15.354 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:15 smithi090 ceph-mon[31674]: Deploying daemon mds.b on smithi092 2024-04-02T14:32:15.354 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:15 smithi090 ceph-mon[31674]: pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-02T14:32:15.354 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:15 smithi090 ceph-mon[31674]: mds.? [v2:172.21.15.90:6840/1919376858,v1:172.21.15.90:6841/1919376858] up:boot 2024-04-02T14:32:15.354 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:15 smithi090 ceph-mon[31674]: fsmap 11 up:standby 2024-04-02T14:32:15.354 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:15 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-02T14:32:15.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:15 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:32:15.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:15 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:32:15.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:15 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:32:15.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:15 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-02T14:32:15.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:15 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-02T14:32:15.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:15 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:32:15.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:15 smithi092 ceph-mon[36497]: Deploying daemon mds.b on smithi092 2024-04-02T14:32:15.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:15 smithi092 ceph-mon[36497]: pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-02T14:32:15.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:15 smithi092 ceph-mon[36497]: mds.? [v2:172.21.15.90:6840/1919376858,v1:172.21.15.90:6841/1919376858] up:boot 2024-04-02T14:32:15.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:15 smithi092 ceph-mon[36497]: fsmap 11 up:standby 2024-04-02T14:32:15.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:15 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-02T14:32:15.436 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:15 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:32:15.437 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:15 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:32:15.437 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:15 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:32:15.437 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:15 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-02T14:32:15.437 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:15 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-02T14:32:15.437 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:15 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:32:15.437 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:15 smithi177 ceph-mon[36431]: Deploying daemon mds.b on smithi092 2024-04-02T14:32:15.437 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:15 smithi177 ceph-mon[36431]: pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-02T14:32:15.437 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:15 smithi177 ceph-mon[36431]: mds.? [v2:172.21.15.90:6840/1919376858,v1:172.21.15.90:6841/1919376858] up:boot 2024-04-02T14:32:15.437 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:15 smithi177 ceph-mon[36431]: fsmap 11 up:standby 2024-04-02T14:32:15.437 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:15 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-02T14:32:15.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:15.487+0000 7f9513800640 1 -- 172.21.15.90:0/384286101 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f950c1058f0 msgr2=0x7f950c10c180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:32:15.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:15.487+0000 7f9513800640 1 --2- 172.21.15.90:0/384286101 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f950c1058f0 0x7f950c10c180 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f9504009be0 tx=0x7f950402bef0 comp rx=0 tx=0).stop 2024-04-02T14:32:15.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:15.487+0000 7f9513800640 1 -- 172.21.15.90:0/384286101 shutdown_connections 2024-04-02T14:32:15.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:15.487+0000 7f9513800640 1 --2- 172.21.15.90:0/384286101 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f950c1058f0 0x7f950c10c180 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:15.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:15.487+0000 7f9513800640 1 --2- 172.21.15.90:0/384286101 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f950c104f30 0x7f950c1053b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:15.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:15.487+0000 7f9513800640 1 --2- 172.21.15.90:0/384286101 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f950c103d40 0x7f950c104160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:15.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:15.487+0000 7f9513800640 1 -- 172.21.15.90:0/384286101 >> 172.21.15.90:0/384286101 conn(0x7f950c0ff2f0 msgr2=0x7f950c101710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:32:15.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:15.487+0000 7f9513800640 1 -- 172.21.15.90:0/384286101 shutdown_connections 2024-04-02T14:32:15.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:15.487+0000 7f9513800640 1 -- 172.21.15.90:0/384286101 wait complete. 2024-04-02T14:32:15.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:15.488+0000 7f9513800640 1 Processor -- start 2024-04-02T14:32:15.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:15.488+0000 7f9513800640 1 -- start start 2024-04-02T14:32:15.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:15.489+0000 7f9513800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f950c103d40 0x7f950c199fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:32:15.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:15.489+0000 7f9513800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f950c104f30 0x7f950c19a510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:32:15.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:15.489+0000 7f9513800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f950c1058f0 0x7f950c1a1580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:32:15.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:15.489+0000 7f9513800640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f950c10fa10 con 0x7f950c103d40 2024-04-02T14:32:15.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:15.489+0000 7f9513800640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f950c10f890 con 0x7f950c1058f0 2024-04-02T14:32:15.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:15.489+0000 7f9513800640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f950c10fb90 con 0x7f950c104f30 2024-04-02T14:32:15.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:15.489+0000 7f9510e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f950c103d40 0x7f950c199fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:32:15.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:15.489+0000 7f9510e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f950c103d40 0x7f950c199fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:54260/0 (socket says 172.21.15.90:54260) 2024-04-02T14:32:15.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:15.489+0000 7f9510e00640 1 -- 172.21.15.90:0/3884355417 learned_addr learned my addr 172.21.15.90:0/3884355417 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:32:15.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:15.489+0000 7f9511800640 1 --2- 172.21.15.90:0/3884355417 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f950c1058f0 0x7f950c1a1580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:32:15.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:15.489+0000 7f950be00640 1 --2- 172.21.15.90:0/3884355417 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f950c104f30 0x7f950c19a510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:32:15.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:15.489+0000 7f9510e00640 1 -- 172.21.15.90:0/3884355417 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f950c104f30 msgr2=0x7f950c19a510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:32:15.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:15.489+0000 7f9510e00640 1 --2- 172.21.15.90:0/3884355417 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f950c104f30 0x7f950c19a510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:15.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:15.489+0000 7f9510e00640 1 -- 172.21.15.90:0/3884355417 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f950c1058f0 msgr2=0x7f950c1a1580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:32:15.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:15.489+0000 7f9510e00640 1 --2- 172.21.15.90:0/3884355417 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f950c1058f0 0x7f950c1a1580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:15.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:15.489+0000 7f9510e00640 1 -- 172.21.15.90:0/3884355417 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f950c1a1bf0 con 0x7f950c103d40 2024-04-02T14:32:15.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:15.489+0000 7f9511800640 1 --2- 172.21.15.90:0/3884355417 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f950c1058f0 0x7f950c1a1580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:32:15.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:15.490+0000 7f9510e00640 1 --2- 172.21.15.90:0/3884355417 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f950c103d40 0x7f950c199fd0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f950000e9a0 tx=0x7f950000ee60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:32:15.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:15.490+0000 7f9509600640 1 -- 172.21.15.90:0/3884355417 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f950000cd80 con 0x7f950c103d40 2024-04-02T14:32:15.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:15.490+0000 7f9509600640 1 -- 172.21.15.90:0/3884355417 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9500010910 con 0x7f950c103d40 2024-04-02T14:32:15.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:15.490+0000 7f9513800640 1 -- 172.21.15.90:0/3884355417 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f950c1a1ee0 con 0x7f950c103d40 2024-04-02T14:32:15.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:15.490+0000 7f9509600640 1 -- 172.21.15.90:0/3884355417 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9500018a40 con 0x7f950c103d40 2024-04-02T14:32:15.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:15.490+0000 7f9513800640 1 -- 172.21.15.90:0/3884355417 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f950c1a2340 con 0x7f950c103d40 2024-04-02T14:32:15.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:15.491+0000 7f9513800640 1 -- 172.21.15.90:0/3884355417 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f950c068d40 con 0x7f950c103d40 2024-04-02T14:32:15.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:15.494+0000 7f9509600640 1 -- 172.21.15.90:0/3884355417 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f9500010ac0 con 0x7f950c103d40 2024-04-02T14:32:15.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:15.495+0000 7f9509600640 1 --2- 172.21.15.90:0/3884355417 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f94f40a3bb0 0x7f94f40a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:32:15.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:15.495+0000 7f9509600640 1 -- 172.21.15.90:0/3884355417 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 7127+0+0 (secure 0 0 0) 0x7f95000d0880 con 0x7f950c103d40 2024-04-02T14:32:15.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:15.496+0000 7f950be00640 1 --2- 172.21.15.90:0/3884355417 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f94f40a3bb0 0x7f94f40a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:32:15.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:15.496+0000 7f950be00640 1 --2- 172.21.15.90:0/3884355417 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f94f40a3bb0 0x7f94f40a6070 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f950c19b4e0 tx=0x7f94f8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:32:15.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:15.498+0000 7f9509600640 1 -- 172.21.15.90:0/3884355417 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7f9500026020 con 0x7f950c103d40 2024-04-02T14:32:15.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:15.741+0000 7f9513800640 1 -- 172.21.15.90:0/3884355417 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f950c10afe0 con 0x7f950c103d40 2024-04-02T14:32:15.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:15.743+0000 7f9509600640 1 -- 172.21.15.90:0/3884355417 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) ==== 94+0+9653 (secure 0 0 0) 0x7f950c10afe0 con 0x7f950c103d40 2024-04-02T14:32:15.744 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-02T14:32:15.744 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":12,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14412,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6837/2852160309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6836","nonce":2852160309},{"type":"v1","addr":"172.21.15.90:6837","nonce":2852160309}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14478,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6841/1919376858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6840","nonce":1919376858},{"type":"v1","addr":"172.21.15.90:6841","nonce":1919376858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24454,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/1564319431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":1564319431},{"type":"v1","addr":"172.21.15.177:6833","nonce":1564319431}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24461,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6833/1570337372","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":1570337372},{"type":"v1","addr":"172.21.15.92:6833","nonce":1570337372}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24478,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6835/1277124891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6834","nonce":1277124891},{"type":"v1","addr":"172.21.15.90:6835","nonce":1277124891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24503,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6835/860721164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":860721164},{"type":"v1","addr":"172.21.15.92:6835","nonce":860721164}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24511,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/3177244988","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":3177244988},{"type":"v1","addr":"172.21.15.177:6835","nonce":3177244988}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24523,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6837/4223414858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":4223414858},{"type":"v1","addr":"172.21.15.177:6837","nonce":4223414858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24539,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6837/1270635939","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":1270635939},{"type":"v1","addr":"172.21.15.92:6837","nonce":1270635939}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24547,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6839/664895663","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":664895663},{"type":"v1","addr":"172.21.15.177:6839","nonce":664895663}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24560,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6839/1498915870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6838","nonce":1498915870},{"type":"v1","addr":"172.21.15.90:6839","nonce":1498915870}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[]} 2024-04-02T14:32:15.744 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 12 2024-04-02T14:32:15.746 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:15.745+0000 7f9513800640 1 -- 172.21.15.90:0/3884355417 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f94f40a3bb0 msgr2=0x7f94f40a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:32:15.746 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:15.745+0000 7f9513800640 1 --2- 172.21.15.90:0/3884355417 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f94f40a3bb0 0x7f94f40a6070 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f950c19b4e0 tx=0x7f94f8005f90 comp rx=0 tx=0).stop 2024-04-02T14:32:15.746 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:15.745+0000 7f9513800640 1 -- 172.21.15.90:0/3884355417 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f950c103d40 msgr2=0x7f950c199fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:32:15.746 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:15.745+0000 7f9513800640 1 --2- 172.21.15.90:0/3884355417 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f950c103d40 0x7f950c199fd0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f950000e9a0 tx=0x7f950000ee60 comp rx=0 tx=0).stop 2024-04-02T14:32:15.746 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:15.745+0000 7f9513800640 1 -- 172.21.15.90:0/3884355417 shutdown_connections 2024-04-02T14:32:15.746 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:15.745+0000 7f9513800640 1 --2- 172.21.15.90:0/3884355417 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f94f40a3bb0 0x7f94f40a6070 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:15.746 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:15.745+0000 7f9513800640 1 --2- 172.21.15.90:0/3884355417 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f950c1058f0 0x7f950c1a1580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:15.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:15.745+0000 7f9513800640 1 --2- 172.21.15.90:0/3884355417 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f950c104f30 0x7f950c19a510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:15.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:15.745+0000 7f9513800640 1 --2- 172.21.15.90:0/3884355417 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f950c103d40 0x7f950c199fd0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:15.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:15.745+0000 7f9513800640 1 -- 172.21.15.90:0/3884355417 >> 172.21.15.90:0/3884355417 conn(0x7f950c0ff2f0 msgr2=0x7f950c108160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:32:15.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:15.745+0000 7f9513800640 1 -- 172.21.15.90:0/3884355417 shutdown_connections 2024-04-02T14:32:15.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:15.746+0000 7f9513800640 1 -- 172.21.15.90:0/3884355417 wait complete. 2024-04-02T14:32:16.363 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:16 smithi092 ceph-mon[36497]: from='client.? 172.21.15.90:0/3884355417' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-02T14:32:16.363 INFO:journalctl@ceph.mds.b.smithi092.stdout:Apr 02 14:32:16 smithi092 systemd[1]: Starting Ceph mds.b for 6449448a-f0fc-11ee-b647-cb9ed24678a4... 2024-04-02T14:32:16.436 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:16 smithi177 ceph-mon[36431]: from='client.? 172.21.15.90:0/3884355417' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-02T14:32:16.439 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:16 smithi090 ceph-mon[31674]: from='client.? 172.21.15.90:0/3884355417' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-02T14:32:16.662 INFO:journalctl@ceph.mds.b.smithi092.stdout:Apr 02 14:32:16 smithi092 podman[67407]: 2024-04-02 14:32:16.392239298 +0000 UTC m=+0.104939595 container create 1f09a480542acfd0be9c6d797a51dc98d110671d4ba1fec86d0c0fdc6ee6a18d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mds-b, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5, org.label-schema.build-date=20240326, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240402.004512-191d963) 2024-04-02T14:32:16.663 INFO:journalctl@ceph.mds.b.smithi092.stdout:Apr 02 14:32:16 smithi092 podman[67407]: 2024-04-02 14:32:16.305034786 +0000 UTC m=+0.017735104 image pull f7301e070545fc23d8e5b7b052a2c15e832a0e57102f1acad8e4f3583709d23f quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534 2024-04-02T14:32:16.663 INFO:journalctl@ceph.mds.b.smithi092.stdout:Apr 02 14:32:16 smithi092 podman[67407]: 2024-04-02 14:32:16.636580676 +0000 UTC m=+0.349280971 container init 1f09a480542acfd0be9c6d797a51dc98d110671d4ba1fec86d0c0fdc6ee6a18d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mds-b, org.label-schema.build-date=20240326, RELEASE=wip-batrick-testing-20240402.004512-191d963, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-02T14:32:16.663 INFO:journalctl@ceph.mds.b.smithi092.stdout:Apr 02 14:32:16 smithi092 podman[67407]: 2024-04-02 14:32:16.639254342 +0000 UTC m=+0.351954647 container start 1f09a480542acfd0be9c6d797a51dc98d110671d4ba1fec86d0c0fdc6ee6a18d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mds-b, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240402.004512-191d963, ceph=True, org.label-schema.build-date=20240326, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-04-02T14:32:17.110 INFO:journalctl@ceph.mds.b.smithi092.stdout:Apr 02 14:32:16 smithi092 ceph-mds[67426]: -- 172.21.15.92:0/3093184243 <== mon.2 v2:172.21.15.92:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x561816d83a40 con 0x561816d57000 2024-04-02T14:32:17.111 INFO:journalctl@ceph.mds.b.smithi092.stdout:Apr 02 14:32:16 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mds-b[67422]: starting mds.b at 2024-04-02T14:32:17.111 INFO:journalctl@ceph.mds.b.smithi092.stdout:Apr 02 14:32:16 smithi092 bash[67407]: 1f09a480542acfd0be9c6d797a51dc98d110671d4ba1fec86d0c0fdc6ee6a18d 2024-04-02T14:32:17.111 INFO:journalctl@ceph.mds.b.smithi092.stdout:Apr 02 14:32:16 smithi092 systemd[1]: Started Ceph mds.b for 6449448a-f0fc-11ee-b647-cb9ed24678a4. 2024-04-02T14:32:17.768 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T14:32:17.969 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:17.966+0000 7f2dbec00640 1 -- 172.21.15.90:0/2779260272 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2db81023d0 msgr2=0x7f2db81047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:32:17.969 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:17.966+0000 7f2dbec00640 1 --2- 172.21.15.90:0/2779260272 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2db81023d0 0x7f2db81047c0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f2dac009900 tx=0x7f2dac02bc50 comp rx=0 tx=0).stop 2024-04-02T14:32:17.969 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:17.966+0000 7f2dbec00640 1 -- 172.21.15.90:0/2779260272 shutdown_connections 2024-04-02T14:32:17.969 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:17.966+0000 7f2dbec00640 1 --2- 172.21.15.90:0/2779260272 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2db8107630 0x7f2db8109a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:17.969 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:17.966+0000 7f2dbec00640 1 --2- 172.21.15.90:0/2779260272 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2db8104d00 0x7f2db81070f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:17.969 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:17.966+0000 7f2dbec00640 1 --2- 172.21.15.90:0/2779260272 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2db81023d0 0x7f2db81047c0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:17.969 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:17.966+0000 7f2dbec00640 1 -- 172.21.15.90:0/2779260272 >> 172.21.15.90:0/2779260272 conn(0x7f2db80fbde0 msgr2=0x7f2db80fe200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:32:17.969 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:17.966+0000 7f2dbec00640 1 -- 172.21.15.90:0/2779260272 shutdown_connections 2024-04-02T14:32:17.969 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:17.966+0000 7f2dbec00640 1 -- 172.21.15.90:0/2779260272 wait complete. 2024-04-02T14:32:17.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:17.967+0000 7f2dbec00640 1 Processor -- start 2024-04-02T14:32:17.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:17.968+0000 7f2dbec00640 1 -- start start 2024-04-02T14:32:17.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:17.968+0000 7f2dbec00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2db81023d0 0x7f2db8199e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:32:17.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:17.968+0000 7f2dbec00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2db8104d00 0x7f2db819a3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:32:17.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:17.968+0000 7f2dbec00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2db8107630 0x7f2db81a1440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:32:17.972 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:17.968+0000 7f2dbec00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f2db810d620 con 0x7f2db81023d0 2024-04-02T14:32:17.972 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:17.968+0000 7f2dbec00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f2db810d4a0 con 0x7f2db8104d00 2024-04-02T14:32:17.972 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:17.968+0000 7f2dbec00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f2db810d7a0 con 0x7f2db8107630 2024-04-02T14:32:17.972 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:17.968+0000 7f2dbd800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2db81023d0 0x7f2db8199e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:32:17.972 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:17.968+0000 7f2dbce00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2db8104d00 0x7f2db819a3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:32:17.972 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:17.968+0000 7f2dbe200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2db8107630 0x7f2db81a1440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:32:17.972 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:17.968+0000 7f2dbce00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2db8104d00 0x7f2db819a3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:50822/0 (socket says 172.21.15.90:50822) 2024-04-02T14:32:17.972 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:17.968+0000 7f2dbce00640 1 -- 172.21.15.90:0/1572736417 learned_addr learned my addr 172.21.15.90:0/1572736417 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:32:17.972 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:17.968+0000 7f2dbce00640 1 -- 172.21.15.90:0/1572736417 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2db8107630 msgr2=0x7f2db81a1440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:32:17.972 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:17.968+0000 7f2dbce00640 1 --2- 172.21.15.90:0/1572736417 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2db8107630 0x7f2db81a1440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:17.972 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:17.969+0000 7f2dbce00640 1 -- 172.21.15.90:0/1572736417 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2db81023d0 msgr2=0x7f2db8199e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:32:17.972 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:17.969+0000 7f2dbce00640 1 --2- 172.21.15.90:0/1572736417 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2db81023d0 0x7f2db8199e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:17.972 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:17.969+0000 7f2dbce00640 1 -- 172.21.15.90:0/1572736417 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2db81a1ab0 con 0x7f2db8104d00 2024-04-02T14:32:17.972 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:17.969+0000 7f2dbce00640 1 --2- 172.21.15.90:0/1572736417 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2db8104d00 0x7f2db819a3d0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f2da400e9c0 tx=0x7f2da400ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:32:17.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:17.969+0000 7f2db6000640 1 -- 172.21.15.90:0/1572736417 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2da400cd80 con 0x7f2db8104d00 2024-04-02T14:32:17.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:17.969+0000 7f2db6000640 1 -- 172.21.15.90:0/1572736417 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2da4005500 con 0x7f2db8104d00 2024-04-02T14:32:17.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:17.969+0000 7f2dbec00640 1 -- 172.21.15.90:0/1572736417 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2db81a1da0 con 0x7f2db8104d00 2024-04-02T14:32:17.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:17.969+0000 7f2db6000640 1 -- 172.21.15.90:0/1572736417 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2da4010690 con 0x7f2db8104d00 2024-04-02T14:32:17.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:17.969+0000 7f2dbec00640 1 -- 172.21.15.90:0/1572736417 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2db81a2270 con 0x7f2db8104d00 2024-04-02T14:32:17.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:17.970+0000 7f2dbec00640 1 -- 172.21.15.90:0/1572736417 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2db8109180 con 0x7f2db8104d00 2024-04-02T14:32:17.976 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:17.974+0000 7f2db6000640 1 -- 172.21.15.90:0/1572736417 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f2da40046c0 con 0x7f2db8104d00 2024-04-02T14:32:17.977 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:17.975+0000 7f2db6000640 1 --2- 172.21.15.90:0/1572736417 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f2da00a3c30 0x7f2da00a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:32:17.977 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:17.975+0000 7f2db6000640 1 -- 172.21.15.90:0/1572736417 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 7127+0+0 (secure 0 0 0) 0x7f2da40d02a0 con 0x7f2db8104d00 2024-04-02T14:32:17.978 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:17.975+0000 7f2dbd800640 1 --2- 172.21.15.90:0/1572736417 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f2da00a3c30 0x7f2da00a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:32:17.978 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:17.976+0000 7f2dbd800640 1 --2- 172.21.15.90:0/1572736417 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f2da00a3c30 0x7f2da00a60f0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f2dac02c260 tx=0x7f2dac000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:32:17.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:17.979+0000 7f2db6000640 1 -- 172.21.15.90:0/1572736417 <== mon.2 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7f2da409d080 con 0x7f2db8104d00 2024-04-02T14:32:18.042 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:17 smithi092 ceph-mon[36497]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-02T14:32:18.042 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:17 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:32:18.042 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:17 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:32:18.042 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:17 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:32:18.042 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:17 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:32:18.042 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:17 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:32:18.042 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:17 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T14:32:18.042 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:17 smithi092 ceph-mon[36497]: mds.? [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] up:boot 2024-04-02T14:32:18.042 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:17 smithi092 ceph-mon[36497]: fsmap 12 up:standby 2024-04-02T14:32:18.042 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:17 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-02T14:32:18.056 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:17 smithi177 ceph-mon[36431]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-02T14:32:18.056 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:17 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:32:18.056 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:17 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:32:18.056 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:17 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:32:18.057 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:17 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:32:18.057 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:17 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:32:18.057 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:17 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T14:32:18.057 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:17 smithi177 ceph-mon[36431]: mds.? [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] up:boot 2024-04-02T14:32:18.057 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:17 smithi177 ceph-mon[36431]: fsmap 12 up:standby 2024-04-02T14:32:18.057 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:17 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-02T14:32:18.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:17 smithi090 ceph-mon[31674]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-02T14:32:18.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:17 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:32:18.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:17 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:32:18.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:17 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:32:18.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:17 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:32:18.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:17 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:32:18.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:17 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T14:32:18.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:17 smithi090 ceph-mon[31674]: mds.? [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] up:boot 2024-04-02T14:32:18.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:17 smithi090 ceph-mon[31674]: fsmap 12 up:standby 2024-04-02T14:32:18.190 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:17 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-02T14:32:18.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:18.241+0000 7f2dbec00640 1 -- 172.21.15.90:0/1572736417 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2db802d370 con 0x7f2db8104d00 2024-04-02T14:32:18.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:18.242+0000 7f2db6000640 1 -- 172.21.15.90:0/1572736417 <== mon.2 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) ==== 94+0+10479 (secure 0 0 0) 0x7f2da409c980 con 0x7f2db8104d00 2024-04-02T14:32:18.245 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-02T14:32:18.245 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":13,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14412,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6837/2852160309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6836","nonce":2852160309},{"type":"v1","addr":"172.21.15.90:6837","nonce":2852160309}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14478,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6841/1919376858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6840","nonce":1919376858},{"type":"v1","addr":"172.21.15.90:6841","nonce":1919376858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24454,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/1564319431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":1564319431},{"type":"v1","addr":"172.21.15.177:6833","nonce":1564319431}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24461,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6833/1570337372","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":1570337372},{"type":"v1","addr":"172.21.15.92:6833","nonce":1570337372}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24478,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6835/1277124891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6834","nonce":1277124891},{"type":"v1","addr":"172.21.15.90:6835","nonce":1277124891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24503,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6835/860721164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":860721164},{"type":"v1","addr":"172.21.15.92:6835","nonce":860721164}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24511,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/3177244988","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":3177244988},{"type":"v1","addr":"172.21.15.177:6835","nonce":3177244988}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24523,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6837/4223414858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":4223414858},{"type":"v1","addr":"172.21.15.177:6837","nonce":4223414858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24539,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6837/1270635939","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":1270635939},{"type":"v1","addr":"172.21.15.92:6837","nonce":1270635939}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24547,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6839/664895663","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":664895663},{"type":"v1","addr":"172.21.15.177:6839","nonce":664895663}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24560,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6839/1498915870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6838","nonce":1498915870},{"type":"v1","addr":"172.21.15.90:6839","nonce":1498915870}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24590,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6839/1054888219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":1054888219},{"type":"v1","addr":"172.21.15.92:6839","nonce":1054888219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13}],"filesystems":[]} 2024-04-02T14:32:18.245 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 13 2024-04-02T14:32:18.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:18.245+0000 7f2dbec00640 1 -- 172.21.15.90:0/1572736417 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f2da00a3c30 msgr2=0x7f2da00a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:32:18.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:18.245+0000 7f2dbec00640 1 --2- 172.21.15.90:0/1572736417 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f2da00a3c30 0x7f2da00a60f0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f2dac02c260 tx=0x7f2dac000bc0 comp rx=0 tx=0).stop 2024-04-02T14:32:18.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:18.245+0000 7f2dbec00640 1 -- 172.21.15.90:0/1572736417 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2db8104d00 msgr2=0x7f2db819a3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:32:18.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:18.245+0000 7f2dbec00640 1 --2- 172.21.15.90:0/1572736417 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2db8104d00 0x7f2db819a3d0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f2da400e9c0 tx=0x7f2da400ee80 comp rx=0 tx=0).stop 2024-04-02T14:32:18.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:18.245+0000 7f2dbec00640 1 -- 172.21.15.90:0/1572736417 shutdown_connections 2024-04-02T14:32:18.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:18.245+0000 7f2dbec00640 1 --2- 172.21.15.90:0/1572736417 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f2da00a3c30 0x7f2da00a60f0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:18.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:18.245+0000 7f2dbec00640 1 --2- 172.21.15.90:0/1572736417 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2db8107630 0x7f2db81a1440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:18.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:18.245+0000 7f2dbec00640 1 --2- 172.21.15.90:0/1572736417 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2db8104d00 0x7f2db819a3d0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:18.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:18.245+0000 7f2dbec00640 1 --2- 172.21.15.90:0/1572736417 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2db81023d0 0x7f2db8199e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:18.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:18.245+0000 7f2dbec00640 1 -- 172.21.15.90:0/1572736417 >> 172.21.15.90:0/1572736417 conn(0x7f2db80fbde0 msgr2=0x7f2db81019f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:32:18.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:18.245+0000 7f2dbec00640 1 -- 172.21.15.90:0/1572736417 shutdown_connections 2024-04-02T14:32:18.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:18.245+0000 7f2dbec00640 1 -- 172.21.15.90:0/1572736417 wait complete. 2024-04-02T14:32:18.265 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T14:32:18.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:18.458+0000 7f73ad600640 1 -- 172.21.15.90:0/3978058236 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f73a81025c0 msgr2=0x7f73a81049b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:32:18.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:18.458+0000 7f73ad600640 1 --2- 172.21.15.90:0/3978058236 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f73a81025c0 0x7f73a81049b0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f7398009860 tx=0x7f739802bb70 comp rx=0 tx=0).stop 2024-04-02T14:32:18.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:18.459+0000 7f73ad600640 1 -- 172.21.15.90:0/3978058236 shutdown_connections 2024-04-02T14:32:18.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:18.459+0000 7f73ad600640 1 --2- 172.21.15.90:0/3978058236 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f73a8107820 0x7f73a8109c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:18.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:18.459+0000 7f73ad600640 1 --2- 172.21.15.90:0/3978058236 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f73a8104ef0 0x7f73a81072e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:18.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:18.459+0000 7f73ad600640 1 --2- 172.21.15.90:0/3978058236 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f73a81025c0 0x7f73a81049b0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:18.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:18.459+0000 7f73ad600640 1 -- 172.21.15.90:0/3978058236 >> 172.21.15.90:0/3978058236 conn(0x7f73a80fbfb0 msgr2=0x7f73a80fe3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:32:18.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:18.459+0000 7f73ad600640 1 -- 172.21.15.90:0/3978058236 shutdown_connections 2024-04-02T14:32:18.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:18.459+0000 7f73ad600640 1 -- 172.21.15.90:0/3978058236 wait complete. 2024-04-02T14:32:18.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:18.460+0000 7f73ad600640 1 Processor -- start 2024-04-02T14:32:18.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:18.460+0000 7f73ad600640 1 -- start start 2024-04-02T14:32:18.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:18.460+0000 7f73ad600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f73a81025c0 0x7f73a819a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:32:18.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:18.460+0000 7f73ad600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f73a8104ef0 0x7f73a819a600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:32:18.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:18.460+0000 7f73ad600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f73a8107820 0x7f73a81a1670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:32:18.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:18.460+0000 7f73ad600640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f73a810d7f0 con 0x7f73a81025c0 2024-04-02T14:32:18.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:18.460+0000 7f73ad600640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f73a810d670 con 0x7f73a8104ef0 2024-04-02T14:32:18.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:18.460+0000 7f73ad600640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f73a810d970 con 0x7f73a8107820 2024-04-02T14:32:18.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:18.460+0000 7f73a6a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f73a81025c0 0x7f73a819a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:32:18.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:18.460+0000 7f73a6000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f73a8104ef0 0x7f73a819a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:32:18.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:18.461+0000 7f73a6a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f73a81025c0 0x7f73a819a0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:54308/0 (socket says 172.21.15.90:54308) 2024-04-02T14:32:18.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:18.461+0000 7f73a6a00640 1 -- 172.21.15.90:0/3842504723 learned_addr learned my addr 172.21.15.90:0/3842504723 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:32:18.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:18.461+0000 7f73a7400640 1 --2- 172.21.15.90:0/3842504723 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f73a8107820 0x7f73a81a1670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:32:18.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:18.461+0000 7f73a6000640 1 -- 172.21.15.90:0/3842504723 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f73a8107820 msgr2=0x7f73a81a1670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:32:18.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:18.461+0000 7f73a6000640 1 --2- 172.21.15.90:0/3842504723 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f73a8107820 0x7f73a81a1670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:18.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:18.461+0000 7f73a6000640 1 -- 172.21.15.90:0/3842504723 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f73a81025c0 msgr2=0x7f73a819a0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:32:18.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:18.461+0000 7f73a6000640 1 --2- 172.21.15.90:0/3842504723 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f73a81025c0 0x7f73a819a0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:18.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:18.461+0000 7f73a6000640 1 -- 172.21.15.90:0/3842504723 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73a81a1ce0 con 0x7f73a8104ef0 2024-04-02T14:32:18.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:18.461+0000 7f73a6a00640 1 --2- 172.21.15.90:0/3842504723 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f73a81025c0 0x7f73a819a0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:32:18.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:18.461+0000 7f73a6000640 1 --2- 172.21.15.90:0/3842504723 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f73a8104ef0 0x7f73a819a600 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f739400e9c0 tx=0x7f739400ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:32:18.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:18.461+0000 7f739f400640 1 -- 172.21.15.90:0/3842504723 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f739400ccc0 con 0x7f73a8104ef0 2024-04-02T14:32:18.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:18.461+0000 7f739f400640 1 -- 172.21.15.90:0/3842504723 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7394005020 con 0x7f73a8104ef0 2024-04-02T14:32:18.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:18.461+0000 7f739f400640 1 -- 172.21.15.90:0/3842504723 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7394004a00 con 0x7f73a8104ef0 2024-04-02T14:32:18.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:18.461+0000 7f73ad600640 1 -- 172.21.15.90:0/3842504723 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f73a81a1fd0 con 0x7f73a8104ef0 2024-04-02T14:32:18.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:18.462+0000 7f73ad600640 1 -- 172.21.15.90:0/3842504723 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f73a81a2440 con 0x7f73a8104ef0 2024-04-02T14:32:18.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:18.463+0000 7f73ad600640 1 -- 172.21.15.90:0/3842504723 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f73a8109370 con 0x7f73a8104ef0 2024-04-02T14:32:18.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:18.467+0000 7f739f400640 1 -- 172.21.15.90:0/3842504723 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f73940051d0 con 0x7f73a8104ef0 2024-04-02T14:32:18.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:18.469+0000 7f739f400640 1 --2- 172.21.15.90:0/3842504723 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f738c0a3bb0 0x7f738c0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:32:18.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:18.469+0000 7f739f400640 1 -- 172.21.15.90:0/3842504723 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 7127+0+0 (secure 0 0 0) 0x7f73940cff70 con 0x7f73a8104ef0 2024-04-02T14:32:18.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:18.469+0000 7f73a6a00640 1 --2- 172.21.15.90:0/3842504723 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f738c0a3bb0 0x7f738c0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:32:18.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:18.470+0000 7f73a6a00640 1 --2- 172.21.15.90:0/3842504723 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f738c0a3bb0 0x7f738c0a6070 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f73980095f0 tx=0x7f739802e040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:32:18.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:18.474+0000 7f739f400640 1 -- 172.21.15.90:0/3842504723 <== mon.2 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7f739409cd50 con 0x7f73a8104ef0 2024-04-02T14:32:18.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:18.679+0000 7f73ad600640 1 -- 172.21.15.90:0/3842504723 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f73a8062700 con 0x7f73a8104ef0 2024-04-02T14:32:18.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:18.681+0000 7f739f400640 1 -- 172.21.15.90:0/3842504723 <== mon.2 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v84) ==== 74+0+18441 (secure 0 0 0) 0x7f739409c650 con 0x7f73a8104ef0 2024-04-02T14:32:18.683 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-02T14:32:18.684 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":84,"fsid":"6449448a-f0fc-11ee-b647-cb9ed24678a4","created":"2024-04-02T14:23:16.530201+0000","modified":"2024-04-02T14:31:24.832762+0000","last_up_change":"2024-04-02T14:31:23.819732+0000","last_in_change":"2024-04-02T14:31:02.824284+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T14:27:16.758622+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a1a6a856-ab44-44bd-bc5e-45c718a70cd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":3635087609},{"type":"v1","addr":"172.21.15.90:6803","nonce":3635087609}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":3635087609},{"type":"v1","addr":"172.21.15.90:6805","nonce":3635087609}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":3635087609},{"type":"v1","addr":"172.21.15.90:6809","nonce":3635087609}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":3635087609},{"type":"v1","addr":"172.21.15.90:6807","nonce":3635087609}]},"public_addr":"172.21.15.90:6803/3635087609","cluster_addr":"172.21.15.90:6805/3635087609","heartbeat_back_addr":"172.21.15.90:6809/3635087609","heartbeat_front_addr":"172.21.15.90:6807/3635087609","state":["exists","up"]},{"osd":1,"uuid":"124139cd-5793-463e-9607-9d9c10164183","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":1994536059},{"type":"v1","addr":"172.21.15.92:6801","nonce":1994536059}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":1994536059},{"type":"v1","addr":"172.21.15.92:6803","nonce":1994536059}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":1994536059},{"type":"v1","addr":"172.21.15.92:6807","nonce":1994536059}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":1994536059},{"type":"v1","addr":"172.21.15.92:6805","nonce":1994536059}]},"public_addr":"172.21.15.92:6801/1994536059","cluster_addr":"172.21.15.92:6803/1994536059","heartbeat_back_addr":"172.21.15.92:6807/1994536059","heartbeat_front_addr":"172.21.15.92:6805/1994536059","state":["exists","up"]},{"osd":2,"uuid":"a61fbef2-9dca-4875-a814-020fff0db0e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":1358472800},{"type":"v1","addr":"172.21.15.177:6801","nonce":1358472800}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":1358472800},{"type":"v1","addr":"172.21.15.177:6803","nonce":1358472800}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":1358472800},{"type":"v1","addr":"172.21.15.177:6807","nonce":1358472800}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":1358472800},{"type":"v1","addr":"172.21.15.177:6805","nonce":1358472800}]},"public_addr":"172.21.15.177:6801/1358472800","cluster_addr":"172.21.15.177:6803/1358472800","heartbeat_back_addr":"172.21.15.177:6807/1358472800","heartbeat_front_addr":"172.21.15.177:6805/1358472800","state":["exists","up"]},{"osd":3,"uuid":"b175a2f4-7859-4eb0-bc1f-fba8c2b92a2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":3114811216},{"type":"v1","addr":"172.21.15.90:6811","nonce":3114811216}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":3114811216},{"type":"v1","addr":"172.21.15.90:6813","nonce":3114811216}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":3114811216},{"type":"v1","addr":"172.21.15.90:6817","nonce":3114811216}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":3114811216},{"type":"v1","addr":"172.21.15.90:6815","nonce":3114811216}]},"public_addr":"172.21.15.90:6811/3114811216","cluster_addr":"172.21.15.90:6813/3114811216","heartbeat_back_addr":"172.21.15.90:6817/3114811216","heartbeat_front_addr":"172.21.15.90:6815/3114811216","state":["exists","up"]},{"osd":4,"uuid":"38cc68ea-c109-4d8d-b77d-1c6f9b56c875","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":2987208879},{"type":"v1","addr":"172.21.15.92:6809","nonce":2987208879}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6810","nonce":2987208879},{"type":"v1","addr":"172.21.15.92:6811","nonce":2987208879}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6814","nonce":2987208879},{"type":"v1","addr":"172.21.15.92:6815","nonce":2987208879}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":2987208879},{"type":"v1","addr":"172.21.15.92:6813","nonce":2987208879}]},"public_addr":"172.21.15.92:6809/2987208879","cluster_addr":"172.21.15.92:6811/2987208879","heartbeat_back_addr":"172.21.15.92:6815/2987208879","heartbeat_front_addr":"172.21.15.92:6813/2987208879","state":["exists","up"]},{"osd":5,"uuid":"2347232e-ac91-466f-8ac9-496f397679ba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":3486820237},{"type":"v1","addr":"172.21.15.177:6809","nonce":3486820237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":3486820237},{"type":"v1","addr":"172.21.15.177:6811","nonce":3486820237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":3486820237},{"type":"v1","addr":"172.21.15.177:6815","nonce":3486820237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":3486820237},{"type":"v1","addr":"172.21.15.177:6813","nonce":3486820237}]},"public_addr":"172.21.15.177:6809/3486820237","cluster_addr":"172.21.15.177:6811/3486820237","heartbeat_back_addr":"172.21.15.177:6815/3486820237","heartbeat_front_addr":"172.21.15.177:6813/3486820237","state":["exists","up"]},{"osd":6,"uuid":"aa72392f-0498-44db-a8d2-eccc89a64871","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":2579716443},{"type":"v1","addr":"172.21.15.90:6819","nonce":2579716443}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6820","nonce":2579716443},{"type":"v1","addr":"172.21.15.90:6821","nonce":2579716443}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":2579716443},{"type":"v1","addr":"172.21.15.90:6825","nonce":2579716443}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2579716443},{"type":"v1","addr":"172.21.15.90:6823","nonce":2579716443}]},"public_addr":"172.21.15.90:6819/2579716443","cluster_addr":"172.21.15.90:6821/2579716443","heartbeat_back_addr":"172.21.15.90:6825/2579716443","heartbeat_front_addr":"172.21.15.90:6823/2579716443","state":["exists","up"]},{"osd":7,"uuid":"931bd105-bc14-4a1e-bfe4-c7ad58fa4b73","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":3371649139},{"type":"v1","addr":"172.21.15.92:6817","nonce":3371649139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":3371649139},{"type":"v1","addr":"172.21.15.92:6819","nonce":3371649139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":3371649139},{"type":"v1","addr":"172.21.15.92:6823","nonce":3371649139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6820","nonce":3371649139},{"type":"v1","addr":"172.21.15.92:6821","nonce":3371649139}]},"public_addr":"172.21.15.92:6817/3371649139","cluster_addr":"172.21.15.92:6819/3371649139","heartbeat_back_addr":"172.21.15.92:6823/3371649139","heartbeat_front_addr":"172.21.15.92:6821/3371649139","state":["exists","up"]},{"osd":8,"uuid":"82624ec2-b382-4886-9f40-7f5f376de3ce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":1459053034},{"type":"v1","addr":"172.21.15.177:6817","nonce":1459053034}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":1459053034},{"type":"v1","addr":"172.21.15.177:6819","nonce":1459053034}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":1459053034},{"type":"v1","addr":"172.21.15.177:6823","nonce":1459053034}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":1459053034},{"type":"v1","addr":"172.21.15.177:6821","nonce":1459053034}]},"public_addr":"172.21.15.177:6817/1459053034","cluster_addr":"172.21.15.177:6819/1459053034","heartbeat_back_addr":"172.21.15.177:6823/1459053034","heartbeat_front_addr":"172.21.15.177:6821/1459053034","state":["exists","up"]},{"osd":9,"uuid":"fbe36e48-3468-42b2-8094-e16552f3cb16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":2592604332},{"type":"v1","addr":"172.21.15.90:6827","nonce":2592604332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":2592604332},{"type":"v1","addr":"172.21.15.90:6829","nonce":2592604332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6832","nonce":2592604332},{"type":"v1","addr":"172.21.15.90:6833","nonce":2592604332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":2592604332},{"type":"v1","addr":"172.21.15.90:6831","nonce":2592604332}]},"public_addr":"172.21.15.90:6827/2592604332","cluster_addr":"172.21.15.90:6829/2592604332","heartbeat_back_addr":"172.21.15.90:6833/2592604332","heartbeat_front_addr":"172.21.15.90:6831/2592604332","state":["exists","up"]},{"osd":10,"uuid":"9240df75-e7e2-47f1-8aae-534b70b3c210","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":1594153142},{"type":"v1","addr":"172.21.15.92:6825","nonce":1594153142}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6826","nonce":1594153142},{"type":"v1","addr":"172.21.15.92:6827","nonce":1594153142}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":1594153142},{"type":"v1","addr":"172.21.15.92:6831","nonce":1594153142}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":1594153142},{"type":"v1","addr":"172.21.15.92:6829","nonce":1594153142}]},"public_addr":"172.21.15.92:6825/1594153142","cluster_addr":"172.21.15.92:6827/1594153142","heartbeat_back_addr":"172.21.15.92:6831/1594153142","heartbeat_front_addr":"172.21.15.92:6829/1594153142","state":["exists","up"]},{"osd":11,"uuid":"83cba9fa-b745-4df1-adda-e86837d4b91a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":83,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1622582126},{"type":"v1","addr":"172.21.15.177:6825","nonce":1622582126}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":1622582126},{"type":"v1","addr":"172.21.15.177:6827","nonce":1622582126}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":1622582126},{"type":"v1","addr":"172.21.15.177:6831","nonce":1622582126}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1622582126},{"type":"v1","addr":"172.21.15.177:6829","nonce":1622582126}]},"public_addr":"172.21.15.177:6825/1622582126","cluster_addr":"172.21.15.177:6827/1622582126","heartbeat_back_addr":"172.21.15.177:6831/1622582126","heartbeat_front_addr":"172.21.15.177:6829/1622582126","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:26:24.290365+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:26:49.405988+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:27:13.639196+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:27:45.603746+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:28:11.019126+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:28:35.084163+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:29:08.064626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:29:33.266133+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:29:57.381816+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:30:30.286847+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:30:56.226279+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:31:20.809611+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.90:0/314641601":"2024-04-03T14:24:34.595561+0000","172.21.15.90:0/1400895868":"2024-04-03T14:24:34.595561+0000","172.21.15.90:6801/129436436":"2024-04-03T14:24:34.595561+0000","172.21.15.90:6801/2372398598":"2024-04-03T14:24:00.362147+0000","172.21.15.90:0/278569338":"2024-04-03T14:24:34.595561+0000","172.21.15.90:0/2321464594":"2024-04-03T14:24:00.362147+0000","172.21.15.90:6800/2372398598":"2024-04-03T14:24:00.362147+0000","172.21.15.90:0/2962656672":"2024-04-03T14:24:00.362147+0000","172.21.15.90:0/126693610":"2024-04-03T14:24:00.362147+0000","172.21.15.90:6800/129436436":"2024-04-03T14:24:34.595561+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T14:32:18.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:18.683+0000 7f73ad600640 1 -- 172.21.15.90:0/3842504723 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f738c0a3bb0 msgr2=0x7f738c0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:32:18.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:18.683+0000 7f73ad600640 1 --2- 172.21.15.90:0/3842504723 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f738c0a3bb0 0x7f738c0a6070 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f73980095f0 tx=0x7f739802e040 comp rx=0 tx=0).stop 2024-04-02T14:32:18.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:18.683+0000 7f73ad600640 1 -- 172.21.15.90:0/3842504723 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f73a8104ef0 msgr2=0x7f73a819a600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:32:18.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:18.683+0000 7f73ad600640 1 --2- 172.21.15.90:0/3842504723 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f73a8104ef0 0x7f73a819a600 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f739400e9c0 tx=0x7f739400ee80 comp rx=0 tx=0).stop 2024-04-02T14:32:18.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:18.683+0000 7f73ad600640 1 -- 172.21.15.90:0/3842504723 shutdown_connections 2024-04-02T14:32:18.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:18.683+0000 7f73ad600640 1 --2- 172.21.15.90:0/3842504723 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f738c0a3bb0 0x7f738c0a6070 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:18.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:18.683+0000 7f73ad600640 1 --2- 172.21.15.90:0/3842504723 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f73a8107820 0x7f73a81a1670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:18.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:18.683+0000 7f73ad600640 1 --2- 172.21.15.90:0/3842504723 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f73a8104ef0 0x7f73a819a600 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:18.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:18.683+0000 7f73ad600640 1 --2- 172.21.15.90:0/3842504723 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f73a81025c0 0x7f73a819a0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:18.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:18.684+0000 7f73ad600640 1 -- 172.21.15.90:0/3842504723 >> 172.21.15.90:0/3842504723 conn(0x7f73a80fbfb0 msgr2=0x7f73a8101dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:32:18.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:18.684+0000 7f73ad600640 1 -- 172.21.15.90:0/3842504723 shutdown_connections 2024-04-02T14:32:18.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:18.684+0000 7f73ad600640 1 -- 172.21.15.90:0/3842504723 wait complete. 2024-04-02T14:32:18.706 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-02T14:27:16.758622+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-02T14:32:18.707 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-02T14:32:18.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:18.911+0000 7f1909600640 1 -- 172.21.15.90:0/127621503 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1904075ae0 msgr2=0x7f1904073f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:32:18.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:18.911+0000 7f1909600640 1 --2- 172.21.15.90:0/127621503 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1904075ae0 0x7f1904073f30 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f18f4009900 tx=0x7f18f402bcd0 comp rx=0 tx=0).stop 2024-04-02T14:32:18.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:18.911+0000 7f1909600640 1 -- 172.21.15.90:0/127621503 shutdown_connections 2024-04-02T14:32:18.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:18.911+0000 7f1909600640 1 --2- 172.21.15.90:0/127621503 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1904109990 0x7f190410be30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:18.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:18.911+0000 7f1909600640 1 --2- 172.21.15.90:0/127621503 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1904074500 0x7f1904074960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:18.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:18.911+0000 7f1909600640 1 --2- 172.21.15.90:0/127621503 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1904075ae0 0x7f1904073f30 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:18.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:18.911+0000 7f1909600640 1 -- 172.21.15.90:0/127621503 >> 172.21.15.90:0/127621503 conn(0x7f19040fd070 msgr2=0x7f19040ff4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:32:18.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:18.912+0000 7f1909600640 1 -- 172.21.15.90:0/127621503 shutdown_connections 2024-04-02T14:32:18.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:18.912+0000 7f1909600640 1 -- 172.21.15.90:0/127621503 wait complete. 2024-04-02T14:32:18.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:18.912+0000 7f1909600640 1 Processor -- start 2024-04-02T14:32:18.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:18.913+0000 7f1909600640 1 -- start start 2024-04-02T14:32:18.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:18.913+0000 7f1909600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1904074500 0x7f1904199ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:32:18.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:18.913+0000 7f1909600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1904075ae0 0x7f190419a530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:32:18.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:18.913+0000 7f1909600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1904109990 0x7f19041a15a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:32:18.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:18.913+0000 7f1909600640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f190410f9c0 con 0x7f1904074500 2024-04-02T14:32:18.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:18.913+0000 7f1909600640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f190410f840 con 0x7f1904109990 2024-04-02T14:32:18.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:18.913+0000 7f1909600640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f190410fb40 con 0x7f1904075ae0 2024-04-02T14:32:18.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:18.913+0000 7f1902a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1904074500 0x7f1904199ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:32:18.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:18.913+0000 7f1903400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1904109990 0x7f19041a15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:32:18.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:18.913+0000 7f1903400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1904109990 0x7f19041a15a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:50860/0 (socket says 172.21.15.90:50860) 2024-04-02T14:32:18.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:18.913+0000 7f1903400640 1 -- 172.21.15.90:0/151312156 learned_addr learned my addr 172.21.15.90:0/151312156 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:32:18.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:18.913+0000 7f1902000640 1 --2- 172.21.15.90:0/151312156 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1904075ae0 0x7f190419a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:32:18.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:18.913+0000 7f1903400640 1 -- 172.21.15.90:0/151312156 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1904075ae0 msgr2=0x7f190419a530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:32:18.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:18.913+0000 7f1903400640 1 --2- 172.21.15.90:0/151312156 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1904075ae0 0x7f190419a530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:18.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:18.913+0000 7f1903400640 1 -- 172.21.15.90:0/151312156 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1904074500 msgr2=0x7f1904199ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:32:18.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:18.913+0000 7f1903400640 1 --2- 172.21.15.90:0/151312156 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1904074500 0x7f1904199ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:18.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:18.913+0000 7f1903400640 1 -- 172.21.15.90:0/151312156 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19041a1c10 con 0x7f1904109990 2024-04-02T14:32:18.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:18.913+0000 7f1902000640 1 --2- 172.21.15.90:0/151312156 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1904075ae0 0x7f190419a530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:32:18.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:18.914+0000 7f1903400640 1 --2- 172.21.15.90:0/151312156 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1904109990 0x7f19041a15a0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f18fc00eb10 tx=0x7f18fc00c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:32:18.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:18.914+0000 7f18fb400640 1 -- 172.21.15.90:0/151312156 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18fc004b20 con 0x7f1904109990 2024-04-02T14:32:18.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:18.914+0000 7f18fb400640 1 -- 172.21.15.90:0/151312156 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f18fc004cc0 con 0x7f1904109990 2024-04-02T14:32:18.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:18.914+0000 7f1909600640 1 -- 172.21.15.90:0/151312156 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f19041a1f00 con 0x7f1904109990 2024-04-02T14:32:18.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:18.914+0000 7f18fb400640 1 -- 172.21.15.90:0/151312156 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18fc010760 con 0x7f1904109990 2024-04-02T14:32:18.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:18.914+0000 7f1909600640 1 -- 172.21.15.90:0/151312156 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19041a23c0 con 0x7f1904109990 2024-04-02T14:32:18.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:18.915+0000 7f1909600640 1 -- 172.21.15.90:0/151312156 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1904108750 con 0x7f1904109990 2024-04-02T14:32:18.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:18.918+0000 7f18fb400640 1 -- 172.21.15.90:0/151312156 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f18fc005020 con 0x7f1904109990 2024-04-02T14:32:18.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:18.919+0000 7f18fb400640 1 --2- 172.21.15.90:0/151312156 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f18e80a3bb0 0x7f18e80a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:32:18.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:18.919+0000 7f18fb400640 1 -- 172.21.15.90:0/151312156 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 7127+0+0 (secure 0 0 0) 0x7f18fc0d0700 con 0x7f1904109990 2024-04-02T14:32:18.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:18.919+0000 7f1902a00640 1 --2- 172.21.15.90:0/151312156 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f18e80a3bb0 0x7f18e80a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:32:18.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:18.920+0000 7f1902a00640 1 --2- 172.21.15.90:0/151312156 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f18e80a3bb0 0x7f18e80a6070 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f18f402c2e0 tx=0x7f18f402c920 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:32:18.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:18.923+0000 7f18fb400640 1 -- 172.21.15.90:0/151312156 <== mon.2 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7f18fc09d580 con 0x7f1904109990 2024-04-02T14:32:19.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:18 smithi092 ceph-mon[36497]: from='client.? 172.21.15.90:0/1572736417' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-02T14:32:19.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:18 smithi092 ceph-mon[36497]: from='client.? 172.21.15.90:0/3842504723' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-02T14:32:19.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:19.128+0000 7f1909600640 1 -- 172.21.15.90:0/151312156 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f190402d540 con 0x7f1904109990 2024-04-02T14:32:19.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:19.129+0000 7f18fb400640 1 -- 172.21.15.90:0/151312156 <== mon.2 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v84) ==== 93+0+10 (secure 0 0 0) 0x7f18fc09ce80 con 0x7f1904109990 2024-04-02T14:32:19.131 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 1 2024-04-02T14:32:19.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:19.131+0000 7f1909600640 1 -- 172.21.15.90:0/151312156 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f18e80a3bb0 msgr2=0x7f18e80a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:32:19.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:19.131+0000 7f1909600640 1 --2- 172.21.15.90:0/151312156 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f18e80a3bb0 0x7f18e80a6070 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f18f402c2e0 tx=0x7f18f402c920 comp rx=0 tx=0).stop 2024-04-02T14:32:19.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:19.131+0000 7f1909600640 1 -- 172.21.15.90:0/151312156 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1904109990 msgr2=0x7f19041a15a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:32:19.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:19.131+0000 7f1909600640 1 --2- 172.21.15.90:0/151312156 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1904109990 0x7f19041a15a0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f18fc00eb10 tx=0x7f18fc00c2d0 comp rx=0 tx=0).stop 2024-04-02T14:32:19.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:19.131+0000 7f1909600640 1 -- 172.21.15.90:0/151312156 shutdown_connections 2024-04-02T14:32:19.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:19.131+0000 7f1909600640 1 --2- 172.21.15.90:0/151312156 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f18e80a3bb0 0x7f18e80a6070 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:19.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:19.131+0000 7f1909600640 1 --2- 172.21.15.90:0/151312156 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1904109990 0x7f19041a15a0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:19.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:19.131+0000 7f1909600640 1 --2- 172.21.15.90:0/151312156 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1904075ae0 0x7f190419a530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:19.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:19.131+0000 7f1909600640 1 --2- 172.21.15.90:0/151312156 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1904074500 0x7f1904199ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:19.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:19.131+0000 7f1909600640 1 -- 172.21.15.90:0/151312156 >> 172.21.15.90:0/151312156 conn(0x7f19040fd070 msgr2=0x7f1904105920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:32:19.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:19.131+0000 7f1909600640 1 -- 172.21.15.90:0/151312156 shutdown_connections 2024-04-02T14:32:19.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:19.132+0000 7f1909600640 1 -- 172.21.15.90:0/151312156 wait complete. 2024-04-02T14:32:19.151 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:18 smithi090 ceph-mon[31674]: from='client.? 172.21.15.90:0/1572736417' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-02T14:32:19.151 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:18 smithi090 ceph-mon[31674]: from='client.? 172.21.15.90:0/3842504723' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-02T14:32:19.154 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-02T14:32:19.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:18 smithi177 ceph-mon[36431]: from='client.? 172.21.15.90:0/1572736417' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-02T14:32:19.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:18 smithi177 ceph-mon[36431]: from='client.? 172.21.15.90:0/3842504723' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-02T14:32:19.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:19.334+0000 7f2193800640 1 -- 172.21.15.90:0/1400682596 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f218c075ae0 msgr2=0x7f218c073f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:32:19.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:19.334+0000 7f2193800640 1 --2- 172.21.15.90:0/1400682596 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f218c075ae0 0x7f218c073f30 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f2180009900 tx=0x7f218002bcd0 comp rx=0 tx=0).stop 2024-04-02T14:32:19.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:19.335+0000 7f2193800640 1 -- 172.21.15.90:0/1400682596 shutdown_connections 2024-04-02T14:32:19.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:19.335+0000 7f2193800640 1 --2- 172.21.15.90:0/1400682596 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f218c1099b0 0x7f218c10be50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:19.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:19.335+0000 7f2193800640 1 --2- 172.21.15.90:0/1400682596 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f218c074500 0x7f218c074960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:19.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:19.335+0000 7f2193800640 1 --2- 172.21.15.90:0/1400682596 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f218c075ae0 0x7f218c073f30 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:19.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:19.335+0000 7f2193800640 1 -- 172.21.15.90:0/1400682596 >> 172.21.15.90:0/1400682596 conn(0x7f218c0fd0b0 msgr2=0x7f218c0ff4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:32:19.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:19.335+0000 7f2193800640 1 -- 172.21.15.90:0/1400682596 shutdown_connections 2024-04-02T14:32:19.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:19.335+0000 7f2193800640 1 -- 172.21.15.90:0/1400682596 wait complete. 2024-04-02T14:32:19.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:19.336+0000 7f2193800640 1 Processor -- start 2024-04-02T14:32:19.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:19.336+0000 7f2193800640 1 -- start start 2024-04-02T14:32:19.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:19.336+0000 7f2193800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f218c074500 0x7f218c19a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:32:19.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:19.336+0000 7f2193800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f218c075ae0 0x7f218c19a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:32:19.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:19.336+0000 7f2193800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f218c1099b0 0x7f218c1a15c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:32:19.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:19.336+0000 7f2193800640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f218c10f9e0 con 0x7f218c074500 2024-04-02T14:32:19.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:19.336+0000 7f2193800640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f218c10f860 con 0x7f218c1099b0 2024-04-02T14:32:19.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:19.336+0000 7f2193800640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f218c10fb60 con 0x7f218c075ae0 2024-04-02T14:32:19.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:19.336+0000 7f2190e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f218c074500 0x7f218c19a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:32:19.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:19.336+0000 7f2190e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f218c074500 0x7f218c19a010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:54338/0 (socket says 172.21.15.90:54338) 2024-04-02T14:32:19.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:19.336+0000 7f2190e00640 1 -- 172.21.15.90:0/1428250690 learned_addr learned my addr 172.21.15.90:0/1428250690 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:32:19.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:19.336+0000 7f218be00640 1 --2- 172.21.15.90:0/1428250690 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f218c075ae0 0x7f218c19a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:32:19.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:19.337+0000 7f2191800640 1 --2- 172.21.15.90:0/1428250690 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f218c1099b0 0x7f218c1a15c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:32:19.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:19.337+0000 7f2191800640 1 -- 172.21.15.90:0/1428250690 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f218c075ae0 msgr2=0x7f218c19a550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:32:19.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:19.337+0000 7f2191800640 1 --2- 172.21.15.90:0/1428250690 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f218c075ae0 0x7f218c19a550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:19.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:19.337+0000 7f2191800640 1 -- 172.21.15.90:0/1428250690 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f218c074500 msgr2=0x7f218c19a010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:32:19.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:19.337+0000 7f2191800640 1 --2- 172.21.15.90:0/1428250690 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f218c074500 0x7f218c19a010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:19.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:19.337+0000 7f2191800640 1 -- 172.21.15.90:0/1428250690 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f218c1a1c30 con 0x7f218c1099b0 2024-04-02T14:32:19.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:19.337+0000 7f218be00640 1 --2- 172.21.15.90:0/1428250690 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f218c075ae0 0x7f218c19a550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:32:19.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:19.337+0000 7f2190e00640 1 --2- 172.21.15.90:0/1428250690 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f218c074500 0x7f218c19a010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:32:19.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:19.337+0000 7f2191800640 1 --2- 172.21.15.90:0/1428250690 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f218c1099b0 0x7f218c1a15c0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f2184009fd0 tx=0x7f218400ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:32:19.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:19.337+0000 7f2189600640 1 -- 172.21.15.90:0/1428250690 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f218400ce20 con 0x7f218c1099b0 2024-04-02T14:32:19.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:19.337+0000 7f2189600640 1 -- 172.21.15.90:0/1428250690 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2184005020 con 0x7f218c1099b0 2024-04-02T14:32:19.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:19.338+0000 7f2193800640 1 -- 172.21.15.90:0/1428250690 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f218c1a1f20 con 0x7f218c1099b0 2024-04-02T14:32:19.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:19.338+0000 7f2189600640 1 -- 172.21.15.90:0/1428250690 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2184010720 con 0x7f218c1099b0 2024-04-02T14:32:19.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:19.338+0000 7f2193800640 1 -- 172.21.15.90:0/1428250690 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f218c1a2390 con 0x7f218c1099b0 2024-04-02T14:32:19.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:19.339+0000 7f2193800640 1 -- 172.21.15.90:0/1428250690 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f218c108770 con 0x7f218c1099b0 2024-04-02T14:32:19.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:19.340+0000 7f2189600640 1 -- 172.21.15.90:0/1428250690 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f2184005540 con 0x7f218c1099b0 2024-04-02T14:32:19.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:19.342+0000 7f2189600640 1 --2- 172.21.15.90:0/1428250690 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f21740a3bb0 0x7f21740a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:32:19.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:19.342+0000 7f2189600640 1 -- 172.21.15.90:0/1428250690 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 7127+0+0 (secure 0 0 0) 0x7f21840d06b0 con 0x7f218c1099b0 2024-04-02T14:32:19.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:19.342+0000 7f2190e00640 1 --2- 172.21.15.90:0/1428250690 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f21740a3bb0 0x7f21740a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:32:19.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:19.343+0000 7f2190e00640 1 --2- 172.21.15.90:0/1428250690 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f21740a3bb0 0x7f21740a6070 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f2180006010 tx=0x7f218002c920 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:32:19.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:19.346+0000 7f2189600640 1 -- 172.21.15.90:0/1428250690 <== mon.2 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7f218409d490 con 0x7f218c1099b0 2024-04-02T14:32:19.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:19.551+0000 7f2193800640 1 -- 172.21.15.90:0/1428250690 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f218c062700 con 0x7f218c1099b0 2024-04-02T14:32:19.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:19.552+0000 7f2189600640 1 -- 172.21.15.90:0/1428250690 <== mon.2 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v84) ==== 84+0+65 (secure 0 0 0) 0x7f218409cd90 con 0x7f218c1099b0 2024-04-02T14:32:19.554 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-02T14:32:19.554 INFO:teuthology.orchestra.run.smithi090.stdout:[ 2024-04-02T14:32:19.554 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-02T14:32:19.554 INFO:teuthology.orchestra.run.smithi090.stdout: "poolnum": 1, 2024-04-02T14:32:19.554 INFO:teuthology.orchestra.run.smithi090.stdout: "poolname": ".mgr" 2024-04-02T14:32:19.554 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:32:19.554 INFO:teuthology.orchestra.run.smithi090.stdout:] 2024-04-02T14:32:19.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:19.554+0000 7f2193800640 1 -- 172.21.15.90:0/1428250690 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f21740a3bb0 msgr2=0x7f21740a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:32:19.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:19.554+0000 7f2193800640 1 --2- 172.21.15.90:0/1428250690 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f21740a3bb0 0x7f21740a6070 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f2180006010 tx=0x7f218002c920 comp rx=0 tx=0).stop 2024-04-02T14:32:19.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:19.554+0000 7f2193800640 1 -- 172.21.15.90:0/1428250690 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f218c1099b0 msgr2=0x7f218c1a15c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:32:19.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:19.554+0000 7f2193800640 1 --2- 172.21.15.90:0/1428250690 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f218c1099b0 0x7f218c1a15c0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f2184009fd0 tx=0x7f218400ef10 comp rx=0 tx=0).stop 2024-04-02T14:32:19.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:19.554+0000 7f2193800640 1 -- 172.21.15.90:0/1428250690 shutdown_connections 2024-04-02T14:32:19.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:19.554+0000 7f2193800640 1 --2- 172.21.15.90:0/1428250690 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f21740a3bb0 0x7f21740a6070 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:19.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:19.554+0000 7f2193800640 1 --2- 172.21.15.90:0/1428250690 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f218c1099b0 0x7f218c1a15c0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:19.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:19.554+0000 7f2193800640 1 --2- 172.21.15.90:0/1428250690 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f218c075ae0 0x7f218c19a550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:19.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:19.554+0000 7f2193800640 1 --2- 172.21.15.90:0/1428250690 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f218c074500 0x7f218c19a010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:19.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:19.554+0000 7f2193800640 1 -- 172.21.15.90:0/1428250690 >> 172.21.15.90:0/1428250690 conn(0x7f218c0fd0b0 msgr2=0x7f218c105940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:32:19.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:19.554+0000 7f2193800640 1 -- 172.21.15.90:0/1428250690 shutdown_connections 2024-04-02T14:32:19.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:19.554+0000 7f2193800640 1 -- 172.21.15.90:0/1428250690 wait complete. 2024-04-02T14:32:19.576 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-02T14:32:19.576 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-02T14:32:19.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:19.772+0000 7fa228e00640 1 -- 172.21.15.90:0/1970167551 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa224103d40 msgr2=0x7fa224104160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:32:19.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:19.772+0000 7fa228e00640 1 --2- 172.21.15.90:0/1970167551 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa224103d40 0x7fa224104160 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fa214009900 tx=0x7fa21402bc30 comp rx=0 tx=0).stop 2024-04-02T14:32:19.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:19.772+0000 7fa228e00640 1 -- 172.21.15.90:0/1970167551 shutdown_connections 2024-04-02T14:32:19.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:19.772+0000 7fa228e00640 1 --2- 172.21.15.90:0/1970167551 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa2241058f0 0x7fa22410c180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:19.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:19.772+0000 7fa228e00640 1 --2- 172.21.15.90:0/1970167551 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa224104f30 0x7fa2241053b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:19.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:19.772+0000 7fa228e00640 1 --2- 172.21.15.90:0/1970167551 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa224103d40 0x7fa224104160 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:19.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:19.772+0000 7fa228e00640 1 -- 172.21.15.90:0/1970167551 >> 172.21.15.90:0/1970167551 conn(0x7fa2240ff2b0 msgr2=0x7fa2241016f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:32:19.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:19.772+0000 7fa228e00640 1 -- 172.21.15.90:0/1970167551 shutdown_connections 2024-04-02T14:32:19.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:19.772+0000 7fa228e00640 1 -- 172.21.15.90:0/1970167551 wait complete. 2024-04-02T14:32:19.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:19.773+0000 7fa228e00640 1 Processor -- start 2024-04-02T14:32:19.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:19.773+0000 7fa228e00640 1 -- start start 2024-04-02T14:32:19.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:19.773+0000 7fa228e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa224103d40 0x7fa22419a060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:32:19.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:19.773+0000 7fa228e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa224104f30 0x7fa22419a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:32:19.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:19.773+0000 7fa228e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa2241058f0 0x7fa2241a1610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:32:19.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:19.773+0000 7fa228e00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fa22410f990 con 0x7fa224103d40 2024-04-02T14:32:19.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:19.773+0000 7fa228e00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fa22410f810 con 0x7fa224104f30 2024-04-02T14:32:19.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:19.773+0000 7fa228e00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fa22410fb10 con 0x7fa2241058f0 2024-04-02T14:32:19.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:19.774+0000 7fa221e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa224103d40 0x7fa22419a060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:32:19.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:19.774+0000 7fa222800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa2241058f0 0x7fa2241a1610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:32:19.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:19.774+0000 7fa221400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa224104f30 0x7fa22419a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:32:19.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:19.774+0000 7fa222800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa2241058f0 0x7fa2241a1610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:37850/0 (socket says 172.21.15.90:37850) 2024-04-02T14:32:19.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:19.774+0000 7fa222800640 1 -- 172.21.15.90:0/3951146608 learned_addr learned my addr 172.21.15.90:0/3951146608 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:32:19.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:19.774+0000 7fa221400640 1 -- 172.21.15.90:0/3951146608 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa2241058f0 msgr2=0x7fa2241a1610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:32:19.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:19.774+0000 7fa221400640 1 --2- 172.21.15.90:0/3951146608 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa2241058f0 0x7fa2241a1610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:19.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:19.774+0000 7fa221400640 1 -- 172.21.15.90:0/3951146608 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa224103d40 msgr2=0x7fa22419a060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:32:19.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:19.774+0000 7fa221400640 1 --2- 172.21.15.90:0/3951146608 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa224103d40 0x7fa22419a060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:19.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:19.774+0000 7fa221400640 1 -- 172.21.15.90:0/3951146608 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2241a1c80 con 0x7fa224104f30 2024-04-02T14:32:19.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:19.774+0000 7fa221400640 1 --2- 172.21.15.90:0/3951146608 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa224104f30 0x7fa22419a5a0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fa21000e9f0 tx=0x7fa21000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:32:19.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:19.774+0000 7fa21aa00640 1 -- 172.21.15.90:0/3951146608 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa21000ccc0 con 0x7fa224104f30 2024-04-02T14:32:19.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:19.775+0000 7fa21aa00640 1 -- 172.21.15.90:0/3951146608 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa210010910 con 0x7fa224104f30 2024-04-02T14:32:19.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:19.775+0000 7fa228e00640 1 -- 172.21.15.90:0/3951146608 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa2241a1f70 con 0x7fa224104f30 2024-04-02T14:32:19.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:19.775+0000 7fa21aa00640 1 -- 172.21.15.90:0/3951146608 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa210018960 con 0x7fa224104f30 2024-04-02T14:32:19.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:19.775+0000 7fa228e00640 1 -- 172.21.15.90:0/3951146608 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2241a2480 con 0x7fa224104f30 2024-04-02T14:32:19.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:19.776+0000 7fa228e00640 1 -- 172.21.15.90:0/3951146608 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa22410b0e0 con 0x7fa224104f30 2024-04-02T14:32:19.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:19.777+0000 7fa21aa00640 1 -- 172.21.15.90:0/3951146608 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fa210010430 con 0x7fa224104f30 2024-04-02T14:32:19.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:19.779+0000 7fa21aa00640 1 --2- 172.21.15.90:0/3951146608 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fa2080a3c30 0x7fa2080a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:32:19.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:19.779+0000 7fa21aa00640 1 -- 172.21.15.90:0/3951146608 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 7127+0+0 (secure 0 0 0) 0x7fa2100d02f0 con 0x7fa224104f30 2024-04-02T14:32:19.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:19.779+0000 7fa221e00640 1 --2- 172.21.15.90:0/3951146608 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fa2080a3c30 0x7fa2080a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:32:19.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:19.780+0000 7fa221e00640 1 --2- 172.21.15.90:0/3951146608 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fa2080a3c30 0x7fa2080a60f0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fa214000c00 tx=0x7fa214005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:32:19.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:19.782+0000 7fa21aa00640 1 -- 172.21.15.90:0/3951146608 <== mon.2 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7fa21009ca80 con 0x7fa224104f30 2024-04-02T14:32:19.939 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:19 smithi090 ceph-mon[31674]: pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-02T14:32:19.939 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:19 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:32:19.939 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:19 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:32:19.939 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:19 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:32:19.939 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:19 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:32:19.939 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:19 smithi090 ceph-mon[31674]: from='client.? 172.21.15.90:0/151312156' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-02T14:32:19.939 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:19 smithi090 ceph-mon[31674]: from='client.? 172.21.15.90:0/1428250690' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-04-02T14:32:19.939 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:19 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:32:19.939 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:19 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:32:19.939 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:19 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:32:19.940 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:19 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T14:32:19.940 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:19 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:32:19.940 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:19 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:32:20.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:19.998+0000 7fa228e00640 1 -- 172.21.15.90:0/3951146608 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7fa224062700 con 0x7fa224104f30 2024-04-02T14:32:20.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:19 smithi092 ceph-mon[36497]: pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-02T14:32:20.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:19 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:32:20.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:19 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:32:20.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:19 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:32:20.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:19 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:32:20.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:19 smithi092 ceph-mon[36497]: from='client.? 172.21.15.90:0/151312156' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-02T14:32:20.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:19 smithi092 ceph-mon[36497]: from='client.? 172.21.15.90:0/1428250690' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-04-02T14:32:20.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:19 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:32:20.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:19 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:32:20.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:19 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:32:20.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:19 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T14:32:20.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:19 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:32:20.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:19 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:32:20.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:19 smithi177 ceph-mon[36431]: pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-02T14:32:20.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:19 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:32:20.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:19 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:32:20.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:19 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:32:20.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:19 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:32:20.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:19 smithi177 ceph-mon[36431]: from='client.? 172.21.15.90:0/151312156' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-02T14:32:20.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:19 smithi177 ceph-mon[36431]: from='client.? 172.21.15.90:0/1428250690' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-04-02T14:32:20.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:19 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:32:20.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:19 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:32:20.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:19 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:32:20.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:19 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T14:32:20.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:19 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:32:20.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:19 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:32:20.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:20.829+0000 7fa21aa00640 1 -- 172.21.15.90:0/3951146608 <== mon.2 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v85) ==== 138+0+0 (secure 0 0 0) 0x7fa2100a0d80 con 0x7fa224104f30 2024-04-02T14:32:20.834 INFO:teuthology.orchestra.run.smithi090.stderr:pool 'cephfs_metadata' created 2024-04-02T14:32:20.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:20.834+0000 7fa228e00640 1 -- 172.21.15.90:0/3951146608 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fa2080a3c30 msgr2=0x7fa2080a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:32:20.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:20.834+0000 7fa228e00640 1 --2- 172.21.15.90:0/3951146608 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fa2080a3c30 0x7fa2080a60f0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fa214000c00 tx=0x7fa214005fd0 comp rx=0 tx=0).stop 2024-04-02T14:32:20.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:20.834+0000 7fa228e00640 1 -- 172.21.15.90:0/3951146608 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa224104f30 msgr2=0x7fa22419a5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:32:20.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:20.834+0000 7fa228e00640 1 --2- 172.21.15.90:0/3951146608 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa224104f30 0x7fa22419a5a0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fa21000e9f0 tx=0x7fa21000eeb0 comp rx=0 tx=0).stop 2024-04-02T14:32:20.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:20.834+0000 7fa228e00640 1 -- 172.21.15.90:0/3951146608 shutdown_connections 2024-04-02T14:32:20.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:20.834+0000 7fa228e00640 1 --2- 172.21.15.90:0/3951146608 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fa2080a3c30 0x7fa2080a60f0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:20.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:20.834+0000 7fa228e00640 1 --2- 172.21.15.90:0/3951146608 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa2241058f0 0x7fa2241a1610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:20.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:20.834+0000 7fa228e00640 1 --2- 172.21.15.90:0/3951146608 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa224104f30 0x7fa22419a5a0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:20.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:20.834+0000 7fa228e00640 1 --2- 172.21.15.90:0/3951146608 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa224103d40 0x7fa22419a060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:20.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:20.834+0000 7fa228e00640 1 -- 172.21.15.90:0/3951146608 >> 172.21.15.90:0/3951146608 conn(0x7fa2240ff2b0 msgr2=0x7fa2241082b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:32:20.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:20.834+0000 7fa228e00640 1 -- 172.21.15.90:0/3951146608 shutdown_connections 2024-04-02T14:32:20.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:20.834+0000 7fa228e00640 1 -- 172.21.15.90:0/3951146608 wait complete. 2024-04-02T14:32:20.861 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-04-02T14:32:21.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:21.061+0000 7f3821e00640 1 -- 172.21.15.90:0/2554581672 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f381c103c00 msgr2=0x7f381c104020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:32:21.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:21.061+0000 7f3821e00640 1 --2- 172.21.15.90:0/2554581672 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f381c103c00 0x7f381c104020 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f3810009900 tx=0x7f381002bc30 comp rx=0 tx=0).stop 2024-04-02T14:32:21.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:21.061+0000 7f3821e00640 1 -- 172.21.15.90:0/2554581672 shutdown_connections 2024-04-02T14:32:21.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:21.061+0000 7f3821e00640 1 --2- 172.21.15.90:0/2554581672 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f381c1057b0 0x7f381c10c040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:21.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:21.061+0000 7f3821e00640 1 --2- 172.21.15.90:0/2554581672 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f381c104df0 0x7f381c105270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:21.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:21.061+0000 7f3821e00640 1 --2- 172.21.15.90:0/2554581672 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f381c103c00 0x7f381c104020 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:21.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:21.061+0000 7f3821e00640 1 -- 172.21.15.90:0/2554581672 >> 172.21.15.90:0/2554581672 conn(0x7f381c0ff170 msgr2=0x7f381c1015b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:32:21.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:21.061+0000 7f3821e00640 1 -- 172.21.15.90:0/2554581672 shutdown_connections 2024-04-02T14:32:21.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:21.061+0000 7f3821e00640 1 -- 172.21.15.90:0/2554581672 wait complete. 2024-04-02T14:32:21.129 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:20 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T14:32:21.129 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:20 smithi092 ceph-mon[36497]: from='client.? 172.21.15.90:0/3951146608' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-02T14:32:21.129 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:20 smithi092 ceph-mon[36497]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-02T14:32:21.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:21.062+0000 7f3821e00640 1 Processor -- start 2024-04-02T14:32:21.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:21.063+0000 7f3821e00640 1 -- start start 2024-04-02T14:32:21.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:21.063+0000 7f3821e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f381c103c00 0x7f381c199ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:32:21.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:21.063+0000 7f3821e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f381c104df0 0x7f381c19a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:32:21.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:21.063+0000 7f3821e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f381c1057b0 0x7f381c1a14a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:32:21.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:21.063+0000 7f3821e00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f381c10f8d0 con 0x7f381c1057b0 2024-04-02T14:32:21.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:21.063+0000 7f3821e00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f381c10f750 con 0x7f381c104df0 2024-04-02T14:32:21.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:21.063+0000 7f3821e00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f381c10fa50 con 0x7f381c103c00 2024-04-02T14:32:21.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:21.063+0000 7f3820a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f381c103c00 0x7f381c199ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:32:21.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:21.063+0000 7f3820a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f381c103c00 0x7f381c199ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:37870/0 (socket says 172.21.15.90:37870) 2024-04-02T14:32:21.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:21.063+0000 7f3820a00640 1 -- 172.21.15.90:0/3793770772 learned_addr learned my addr 172.21.15.90:0/3793770772 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:32:21.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:21.063+0000 7f3821400640 1 --2- 172.21.15.90:0/3793770772 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f381c1057b0 0x7f381c1a14a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:32:21.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:21.063+0000 7f381be00640 1 --2- 172.21.15.90:0/3793770772 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f381c104df0 0x7f381c19a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:32:21.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:21.063+0000 7f3820a00640 1 -- 172.21.15.90:0/3793770772 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f381c104df0 msgr2=0x7f381c19a430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:32:21.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:21.063+0000 7f3820a00640 1 --2- 172.21.15.90:0/3793770772 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f381c104df0 0x7f381c19a430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:21.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:21.063+0000 7f3820a00640 1 -- 172.21.15.90:0/3793770772 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f381c1057b0 msgr2=0x7f381c1a14a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:32:21.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:21.063+0000 7f3820a00640 1 --2- 172.21.15.90:0/3793770772 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f381c1057b0 0x7f381c1a14a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:21.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:21.063+0000 7f3820a00640 1 -- 172.21.15.90:0/3793770772 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f381c1a1b10 con 0x7f381c103c00 2024-04-02T14:32:21.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:21.064+0000 7f3820a00640 1 --2- 172.21.15.90:0/3793770772 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f381c103c00 0x7f381c199ef0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f3810000c00 tx=0x7f381002c650 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:32:21.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:21.064+0000 7f3819600640 1 -- 172.21.15.90:0/3793770772 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3810043020 con 0x7f381c103c00 2024-04-02T14:32:21.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:21.064+0000 7f3821e00640 1 -- 172.21.15.90:0/3793770772 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f381c1a1da0 con 0x7f381c103c00 2024-04-02T14:32:21.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:21.064+0000 7f3819600640 1 -- 172.21.15.90:0/3793770772 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f38100054e0 con 0x7f381c103c00 2024-04-02T14:32:21.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:21.064+0000 7f3819600640 1 -- 172.21.15.90:0/3793770772 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3810003c40 con 0x7f381c103c00 2024-04-02T14:32:21.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:21.064+0000 7f3821e00640 1 -- 172.21.15.90:0/3793770772 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f381c1a2210 con 0x7f381c103c00 2024-04-02T14:32:21.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:21.065+0000 7f3821e00640 1 -- 172.21.15.90:0/3793770772 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f381c10aec0 con 0x7f381c103c00 2024-04-02T14:32:21.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:21.067+0000 7f3819600640 1 -- 172.21.15.90:0/3793770772 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f3810046020 con 0x7f381c103c00 2024-04-02T14:32:21.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:21.068+0000 7f3819600640 1 --2- 172.21.15.90:0/3793770772 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f38040a3bb0 0x7f38040a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:32:21.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:21.068+0000 7f3819600640 1 -- 172.21.15.90:0/3793770772 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7497+0+0 (secure 0 0 0) 0x7f38100f2140 con 0x7f381c103c00 2024-04-02T14:32:21.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:21.069+0000 7f381be00640 1 --2- 172.21.15.90:0/3793770772 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f38040a3bb0 0x7f38040a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:32:21.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:21.069+0000 7f381be00640 1 --2- 172.21.15.90:0/3793770772 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f38040a3bb0 0x7f38040a6070 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f381c19b400 tx=0x7f3808005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:32:21.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:21.072+0000 7f3819600640 1 -- 172.21.15.90:0/3793770772 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7f381004d020 con 0x7f381c103c00 2024-04-02T14:32:21.177 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:20 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T14:32:21.177 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:20 smithi090 ceph-mon[31674]: from='client.? 172.21.15.90:0/3951146608' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-02T14:32:21.177 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:20 smithi090 ceph-mon[31674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-02T14:32:21.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:20 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T14:32:21.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:20 smithi177 ceph-mon[36431]: from='client.? 172.21.15.90:0/3951146608' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-02T14:32:21.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:20 smithi177 ceph-mon[36431]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-02T14:32:21.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:21.279+0000 7f3821e00640 1 -- 172.21.15.90:0/3793770772 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f381c0627a0 con 0x7f381c103c00 2024-04-02T14:32:21.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:21.833+0000 7f3819600640 1 -- 172.21.15.90:0/3793770772 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v86) ==== 170+0+0 (secure 0 0 0) 0x7f38100bedb0 con 0x7f381c103c00 2024-04-02T14:32:21.836 INFO:teuthology.orchestra.run.smithi090.stderr:pool 'cephfs_data' created 2024-04-02T14:32:21.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:21.842+0000 7f3821e00640 1 -- 172.21.15.90:0/3793770772 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f38040a3bb0 msgr2=0x7f38040a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:32:21.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:21.842+0000 7f3821e00640 1 --2- 172.21.15.90:0/3793770772 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f38040a3bb0 0x7f38040a6070 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f381c19b400 tx=0x7f3808005f90 comp rx=0 tx=0).stop 2024-04-02T14:32:21.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:21.842+0000 7f3821e00640 1 -- 172.21.15.90:0/3793770772 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f381c103c00 msgr2=0x7f381c199ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:32:21.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:21.842+0000 7f3821e00640 1 --2- 172.21.15.90:0/3793770772 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f381c103c00 0x7f381c199ef0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f3810000c00 tx=0x7f381002c650 comp rx=0 tx=0).stop 2024-04-02T14:32:21.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:21.843+0000 7f3821e00640 1 -- 172.21.15.90:0/3793770772 shutdown_connections 2024-04-02T14:32:21.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:21.843+0000 7f3821e00640 1 --2- 172.21.15.90:0/3793770772 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f38040a3bb0 0x7f38040a6070 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:21.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:21.843+0000 7f3821e00640 1 --2- 172.21.15.90:0/3793770772 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f381c1057b0 0x7f381c1a14a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:21.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:21.843+0000 7f3821e00640 1 --2- 172.21.15.90:0/3793770772 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f381c104df0 0x7f381c19a430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:21.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:21.843+0000 7f3821e00640 1 --2- 172.21.15.90:0/3793770772 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f381c103c00 0x7f381c199ef0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:21.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:21.843+0000 7f3821e00640 1 -- 172.21.15.90:0/3793770772 >> 172.21.15.90:0/3793770772 conn(0x7f381c0ff170 msgr2=0x7f381c108090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:32:21.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:21.845+0000 7f3821e00640 1 -- 172.21.15.90:0/3793770772 shutdown_connections 2024-04-02T14:32:21.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:21.845+0000 7f3821e00640 1 -- 172.21.15.90:0/3793770772 wait complete. 2024-04-02T14:32:21.876 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-02T14:32:22.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:22.056+0000 7fbfd9800640 1 -- 172.21.15.90:0/294738026 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbfd4075b40 msgr2=0x7fbfd4073f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:32:22.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:22.056+0000 7fbfd9800640 1 --2- 172.21.15.90:0/294738026 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbfd4075b40 0x7fbfd4073f90 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fbfc4009900 tx=0x7fbfc402bc30 comp rx=0 tx=0).stop 2024-04-02T14:32:22.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:22.056+0000 7fbfd9800640 1 -- 172.21.15.90:0/294738026 shutdown_connections 2024-04-02T14:32:22.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:22.056+0000 7fbfd9800640 1 --2- 172.21.15.90:0/294738026 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbfd4109a50 0x7fbfd410bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:22.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:22.056+0000 7fbfd9800640 1 --2- 172.21.15.90:0/294738026 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbfd4074560 0x7fbfd40749e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:22.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:22.056+0000 7fbfd9800640 1 --2- 172.21.15.90:0/294738026 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbfd4075b40 0x7fbfd4073f90 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:22.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:22.056+0000 7fbfd9800640 1 -- 172.21.15.90:0/294738026 >> 172.21.15.90:0/294738026 conn(0x7fbfd40fd120 msgr2=0x7fbfd40ff540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:32:22.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:22.056+0000 7fbfd9800640 1 -- 172.21.15.90:0/294738026 shutdown_connections 2024-04-02T14:32:22.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:22.056+0000 7fbfd9800640 1 -- 172.21.15.90:0/294738026 wait complete. 2024-04-02T14:32:22.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:22.057+0000 7fbfd9800640 1 Processor -- start 2024-04-02T14:32:22.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:22.057+0000 7fbfd9800640 1 -- start start 2024-04-02T14:32:22.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:22.057+0000 7fbfd9800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbfd4074560 0x7fbfd419a020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:32:22.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:22.057+0000 7fbfd9800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbfd4075b40 0x7fbfd419a560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:32:22.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:22.057+0000 7fbfd9800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbfd4109a50 0x7fbfd41a15d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:32:22.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:22.057+0000 7fbfd9800640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fbfd410f9b0 con 0x7fbfd4109a50 2024-04-02T14:32:22.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:22.057+0000 7fbfd9800640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fbfd410f830 con 0x7fbfd4074560 2024-04-02T14:32:22.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:22.057+0000 7fbfd9800640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fbfd410fb30 con 0x7fbfd4075b40 2024-04-02T14:32:22.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:22.058+0000 7fbfd2a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbfd4074560 0x7fbfd419a020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:32:22.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:22.058+0000 7fbfd3400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbfd4109a50 0x7fbfd41a15d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:32:22.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:22.058+0000 7fbfd2a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbfd4074560 0x7fbfd419a020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:50938/0 (socket says 172.21.15.90:50938) 2024-04-02T14:32:22.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:22.058+0000 7fbfd2a00640 1 -- 172.21.15.90:0/3409446908 learned_addr learned my addr 172.21.15.90:0/3409446908 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:32:22.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:22.058+0000 7fbfcbe00640 1 --2- 172.21.15.90:0/3409446908 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbfd4075b40 0x7fbfd419a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:32:22.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:22.058+0000 7fbfd2a00640 1 -- 172.21.15.90:0/3409446908 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbfd4075b40 msgr2=0x7fbfd419a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:32:22.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:22.058+0000 7fbfd2a00640 1 --2- 172.21.15.90:0/3409446908 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbfd4075b40 0x7fbfd419a560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:22.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:22.058+0000 7fbfd2a00640 1 -- 172.21.15.90:0/3409446908 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbfd4109a50 msgr2=0x7fbfd41a15d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:32:22.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:22.058+0000 7fbfd2a00640 1 --2- 172.21.15.90:0/3409446908 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbfd4109a50 0x7fbfd41a15d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:22.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:22.058+0000 7fbfd2a00640 1 -- 172.21.15.90:0/3409446908 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbfd41a1c40 con 0x7fbfd4074560 2024-04-02T14:32:22.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:22.058+0000 7fbfd2a00640 1 --2- 172.21.15.90:0/3409446908 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbfd4074560 0x7fbfd419a020 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fbfc4000c00 tx=0x7fbfc402c650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:32:22.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:22.058+0000 7fbfcb400640 1 -- 172.21.15.90:0/3409446908 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfc4043020 con 0x7fbfd4074560 2024-04-02T14:32:22.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:22.058+0000 7fbfcb400640 1 -- 172.21.15.90:0/3409446908 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbfc40054e0 con 0x7fbfd4074560 2024-04-02T14:32:22.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:22.059+0000 7fbfd9800640 1 -- 172.21.15.90:0/3409446908 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbfd41a1ed0 con 0x7fbfd4074560 2024-04-02T14:32:22.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:22.059+0000 7fbfcb400640 1 -- 172.21.15.90:0/3409446908 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfc4003bd0 con 0x7fbfd4074560 2024-04-02T14:32:22.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:22.059+0000 7fbfd9800640 1 -- 172.21.15.90:0/3409446908 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbfd41a2330 con 0x7fbfd4074560 2024-04-02T14:32:22.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:22.060+0000 7fbfd9800640 1 -- 172.21.15.90:0/3409446908 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbfd4108780 con 0x7fbfd4074560 2024-04-02T14:32:22.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:22.061+0000 7fbfcb400640 1 -- 172.21.15.90:0/3409446908 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fbfc4046020 con 0x7fbfd4074560 2024-04-02T14:32:22.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:22.062+0000 7fbfcb400640 1 --2- 172.21.15.90:0/3409446908 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fbfb80a3bb0 0x7fbfb80a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:32:22.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:22.063+0000 7fbfcb400640 1 -- 172.21.15.90:0/3409446908 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7879+0+0 (secure 0 0 0) 0x7fbfc40f23f0 con 0x7fbfd4074560 2024-04-02T14:32:22.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:22.063+0000 7fbfcbe00640 1 --2- 172.21.15.90:0/3409446908 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fbfb80a3bb0 0x7fbfb80a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:32:22.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:22.063+0000 7fbfcbe00640 1 --2- 172.21.15.90:0/3409446908 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fbfb80a3bb0 0x7fbfb80a6070 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fbfd419b530 tx=0x7fbfbc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:32:22.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:22.066+0000 7fbfcb400640 1 -- 172.21.15.90:0/3409446908 <== mon.2 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7fbfc404d020 con 0x7fbfd4074560 2024-04-02T14:32:22.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:21 smithi092 ceph-mon[36497]: pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-02T14:32:22.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:21 smithi092 ceph-mon[36497]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-04-02T14:32:22.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:21 smithi092 ceph-mon[36497]: osdmap e85: 12 total, 12 up, 12 in 2024-04-02T14:32:22.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:21 smithi092 ceph-mon[36497]: from='client.? 172.21.15.90:0/3793770772' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-02T14:32:22.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:21 smithi092 ceph-mon[36497]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-02T14:32:22.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:21 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:32:22.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:21 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:32:22.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:21 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:32:22.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:21 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:32:22.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:21 smithi177 ceph-mon[36431]: pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-02T14:32:22.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:21 smithi177 ceph-mon[36431]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-04-02T14:32:22.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:21 smithi177 ceph-mon[36431]: osdmap e85: 12 total, 12 up, 12 in 2024-04-02T14:32:22.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:21 smithi177 ceph-mon[36431]: from='client.? 172.21.15.90:0/3793770772' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-02T14:32:22.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:21 smithi177 ceph-mon[36431]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-02T14:32:22.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:21 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:32:22.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:21 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:32:22.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:21 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:32:22.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:21 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:32:22.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:21 smithi090 ceph-mon[31674]: pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-02T14:32:22.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:21 smithi090 ceph-mon[31674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-04-02T14:32:22.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:21 smithi090 ceph-mon[31674]: osdmap e85: 12 total, 12 up, 12 in 2024-04-02T14:32:22.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:21 smithi090 ceph-mon[31674]: from='client.? 172.21.15.90:0/3793770772' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-02T14:32:22.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:21 smithi090 ceph-mon[31674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-02T14:32:22.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:21 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:32:22.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:21 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:32:22.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:21 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:32:22.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:21 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:32:22.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:22.316+0000 7fbfd9800640 1 -- 172.21.15.90:0/3409446908 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7fbfd402d370 con 0x7fbfd4074560 2024-04-02T14:32:22.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:22.850+0000 7fbfcb400640 1 -- 172.21.15.90:0/3409446908 <== mon.2 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v14) ==== 172+0+0 (secure 0 0 0) 0x7fbfc40beee0 con 0x7fbfd4074560 2024-04-02T14:32:22.853 INFO:teuthology.orchestra.run.smithi090.stderr:new fs with metadata pool 2 and data pool 3 2024-04-02T14:32:22.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:22.855+0000 7fbfd9800640 1 -- 172.21.15.90:0/3409446908 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fbfb80a3bb0 msgr2=0x7fbfb80a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:32:22.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:22.855+0000 7fbfd9800640 1 --2- 172.21.15.90:0/3409446908 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fbfb80a3bb0 0x7fbfb80a6070 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fbfd419b530 tx=0x7fbfbc005f90 comp rx=0 tx=0).stop 2024-04-02T14:32:22.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:22.855+0000 7fbfd9800640 1 -- 172.21.15.90:0/3409446908 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbfd4074560 msgr2=0x7fbfd419a020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:32:22.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:22.855+0000 7fbfd9800640 1 --2- 172.21.15.90:0/3409446908 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbfd4074560 0x7fbfd419a020 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fbfc4000c00 tx=0x7fbfc402c650 comp rx=0 tx=0).stop 2024-04-02T14:32:22.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:22.858+0000 7fbfd9800640 1 -- 172.21.15.90:0/3409446908 shutdown_connections 2024-04-02T14:32:22.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:22.858+0000 7fbfd9800640 1 --2- 172.21.15.90:0/3409446908 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fbfb80a3bb0 0x7fbfb80a6070 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:22.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:22.858+0000 7fbfd9800640 1 --2- 172.21.15.90:0/3409446908 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbfd4109a50 0x7fbfd41a15d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:22.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:22.858+0000 7fbfd9800640 1 --2- 172.21.15.90:0/3409446908 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbfd4075b40 0x7fbfd419a560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:22.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:22.858+0000 7fbfd9800640 1 --2- 172.21.15.90:0/3409446908 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbfd4074560 0x7fbfd419a020 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:22.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:22.858+0000 7fbfd9800640 1 -- 172.21.15.90:0/3409446908 >> 172.21.15.90:0/3409446908 conn(0x7fbfd40fd120 msgr2=0x7fbfd4105950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:32:22.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:22.858+0000 7fbfd9800640 1 -- 172.21.15.90:0/3409446908 shutdown_connections 2024-04-02T14:32:22.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:22.858+0000 7fbfd9800640 1 -- 172.21.15.90:0/3409446908 wait complete. 2024-04-02T14:32:22.885 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T14:32:23.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:23.075+0000 7fa6d3c00640 1 -- 172.21.15.90:0/1724364857 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa6cc109330 msgr2=0x7fa6cc1097b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:32:23.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:23.075+0000 7fa6d3c00640 1 --2- 172.21.15.90:0/1724364857 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa6cc109330 0x7fa6cc1097b0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fa6b4009900 tx=0x7fa6b402bc30 comp rx=0 tx=0).stop 2024-04-02T14:32:23.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:23.075+0000 7fa6d3c00640 1 -- 172.21.15.90:0/1724364857 shutdown_connections 2024-04-02T14:32:23.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:23.075+0000 7fa6d3c00640 1 --2- 172.21.15.90:0/1724364857 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa6cc109cf0 0x7fa6cc110580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:23.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:23.075+0000 7fa6d3c00640 1 --2- 172.21.15.90:0/1724364857 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa6cc109330 0x7fa6cc1097b0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:23.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:23.075+0000 7fa6d3c00640 1 --2- 172.21.15.90:0/1724364857 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa6cc108140 0x7fa6cc108560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:23.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:23.075+0000 7fa6d3c00640 1 -- 172.21.15.90:0/1724364857 >> 172.21.15.90:0/1724364857 conn(0x7fa6cc077110 msgr2=0x7fa6cc079530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:32:23.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:23.076+0000 7fa6d3c00640 1 -- 172.21.15.90:0/1724364857 shutdown_connections 2024-04-02T14:32:23.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:23.076+0000 7fa6d3c00640 1 -- 172.21.15.90:0/1724364857 wait complete. 2024-04-02T14:32:23.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:23.077+0000 7fa6d3c00640 1 Processor -- start 2024-04-02T14:32:23.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:23.077+0000 7fa6d3c00640 1 -- start start 2024-04-02T14:32:23.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:23.077+0000 7fa6d3c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa6cc108140 0x7fa6cc19e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:32:23.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:23.077+0000 7fa6d3c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa6cc109330 0x7fa6cc19e930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:32:23.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:23.077+0000 7fa6d3c00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa6cc109cf0 0x7fa6cc1a59a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:32:23.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:23.077+0000 7fa6d3c00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fa6cc113dd0 con 0x7fa6cc108140 2024-04-02T14:32:23.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:23.077+0000 7fa6d3c00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fa6cc113c50 con 0x7fa6cc109cf0 2024-04-02T14:32:23.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:23.077+0000 7fa6d3c00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fa6cc113f50 con 0x7fa6cc109330 2024-04-02T14:32:23.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:23.077+0000 7fa6d1200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa6cc108140 0x7fa6cc19e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:32:23.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:23.077+0000 7fa6d1200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa6cc108140 0x7fa6cc19e3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:54432/0 (socket says 172.21.15.90:54432) 2024-04-02T14:32:23.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:23.077+0000 7fa6d1200640 1 -- 172.21.15.90:0/3911953847 learned_addr learned my addr 172.21.15.90:0/3911953847 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:32:23.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:23.077+0000 7fa6d1c00640 1 --2- 172.21.15.90:0/3911953847 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa6cc109cf0 0x7fa6cc1a59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:32:23.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:23.077+0000 7fa6c3e00640 1 --2- 172.21.15.90:0/3911953847 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa6cc109330 0x7fa6cc19e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:32:23.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:23.078+0000 7fa6d1200640 1 -- 172.21.15.90:0/3911953847 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa6cc109330 msgr2=0x7fa6cc19e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:32:23.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:23.078+0000 7fa6d1200640 1 --2- 172.21.15.90:0/3911953847 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa6cc109330 0x7fa6cc19e930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:23.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:23.078+0000 7fa6d1200640 1 -- 172.21.15.90:0/3911953847 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa6cc109cf0 msgr2=0x7fa6cc1a59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:32:23.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:23.078+0000 7fa6d1200640 1 --2- 172.21.15.90:0/3911953847 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa6cc109cf0 0x7fa6cc1a59a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:23.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:23.078+0000 7fa6d1200640 1 -- 172.21.15.90:0/3911953847 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6cc1a6010 con 0x7fa6cc108140 2024-04-02T14:32:23.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:23.078+0000 7fa6d1200640 1 --2- 172.21.15.90:0/3911953847 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa6cc108140 0x7fa6cc19e3f0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fa6bc00e9c0 tx=0x7fa6bc00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:32:23.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:23.078+0000 7fa6ca000640 1 -- 172.21.15.90:0/3911953847 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6bc00cd80 con 0x7fa6cc108140 2024-04-02T14:32:23.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:23.078+0000 7fa6ca000640 1 -- 172.21.15.90:0/3911953847 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa6bc0035f0 con 0x7fa6cc108140 2024-04-02T14:32:23.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:23.078+0000 7fa6d3c00640 1 -- 172.21.15.90:0/3911953847 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6cc1a6300 con 0x7fa6cc108140 2024-04-02T14:32:23.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:23.078+0000 7fa6ca000640 1 -- 172.21.15.90:0/3911953847 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6bc010690 con 0x7fa6cc108140 2024-04-02T14:32:23.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:23.078+0000 7fa6d3c00640 1 -- 172.21.15.90:0/3911953847 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6cc1a6820 con 0x7fa6cc108140 2024-04-02T14:32:23.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:23.080+0000 7fa6d3c00640 1 -- 172.21.15.90:0/3911953847 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa6cc10f460 con 0x7fa6cc108140 2024-04-02T14:32:23.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:23.082+0000 7fa6ca000640 1 -- 172.21.15.90:0/3911953847 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fa6bc0037a0 con 0x7fa6cc108140 2024-04-02T14:32:23.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:23.083+0000 7fa6ca000640 1 --2- 172.21.15.90:0/3911953847 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fa6ac0a3bb0 0x7fa6ac0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:32:23.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:23.083+0000 7fa6ca000640 1 -- 172.21.15.90:0/3911953847 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 7979+0+0 (secure 0 0 0) 0x7fa6bc0d07f0 con 0x7fa6cc108140 2024-04-02T14:32:23.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:23.083+0000 7fa6c3e00640 1 --2- 172.21.15.90:0/3911953847 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fa6ac0a3bb0 0x7fa6ac0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:32:23.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:23.084+0000 7fa6c3e00640 1 --2- 172.21.15.90:0/3911953847 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fa6ac0a3bb0 0x7fa6ac0a6070 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fa6b402c7b0 tx=0x7fa6b4005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:32:23.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:23.086+0000 7fa6ca000640 1 -- 172.21.15.90:0/3911953847 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7fa6bc09d310 con 0x7fa6cc108140 2024-04-02T14:32:23.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:22 smithi177 ceph-mon[36431]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-04-02T14:32:23.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:22 smithi177 ceph-mon[36431]: osdmap e86: 12 total, 12 up, 12 in 2024-04-02T14:32:23.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:22 smithi177 ceph-mon[36431]: from='client.? 172.21.15.90:0/3409446908' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-02T14:32:23.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:22 smithi177 ceph-mon[36431]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-02T14:32:23.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:22 smithi177 ceph-mon[36431]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-04-02T14:32:23.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:22 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:32:23.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:22 smithi177 ceph-mon[36431]: osdmap e87: 12 total, 12 up, 12 in 2024-04-02T14:32:23.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:22 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:32:23.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:22 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:32:23.188 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:22 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T14:32:23.188 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:22 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:32:23.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:22 smithi090 ceph-mon[31674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-04-02T14:32:23.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:22 smithi090 ceph-mon[31674]: osdmap e86: 12 total, 12 up, 12 in 2024-04-02T14:32:23.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:22 smithi090 ceph-mon[31674]: from='client.? 172.21.15.90:0/3409446908' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-02T14:32:23.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:22 smithi090 ceph-mon[31674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-02T14:32:23.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:22 smithi090 ceph-mon[31674]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-04-02T14:32:23.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:22 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:32:23.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:22 smithi090 ceph-mon[31674]: osdmap e87: 12 total, 12 up, 12 in 2024-04-02T14:32:23.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:22 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:32:23.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:22 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:32:23.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:22 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T14:32:23.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:22 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:32:23.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:23.288+0000 7fa6d3c00640 1 -- 172.21.15.90:0/3911953847 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa6cc062700 con 0x7fa6cc108140 2024-04-02T14:32:23.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:23.292+0000 7fa6ca000640 1 -- 172.21.15.90:0/3911953847 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) ==== 74+0+22217 (secure 0 0 0) 0x7fa6bc09cc10 con 0x7fa6cc108140 2024-04-02T14:32:23.298 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-02T14:32:23.298 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":87,"fsid":"6449448a-f0fc-11ee-b647-cb9ed24678a4","created":"2024-04-02T14:23:16.530201+0000","modified":"2024-04-02T14:32:22.318661+0000","last_up_change":"2024-04-02T14:31:23.819732+0000","last_in_change":"2024-04-02T14:31:02.824284+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T14:27:16.758622+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-02T14:32:20.016331+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-02T14:32:21.294367+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a1a6a856-ab44-44bd-bc5e-45c718a70cd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":3635087609},{"type":"v1","addr":"172.21.15.90:6803","nonce":3635087609}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":3635087609},{"type":"v1","addr":"172.21.15.90:6805","nonce":3635087609}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":3635087609},{"type":"v1","addr":"172.21.15.90:6809","nonce":3635087609}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":3635087609},{"type":"v1","addr":"172.21.15.90:6807","nonce":3635087609}]},"public_addr":"172.21.15.90:6803/3635087609","cluster_addr":"172.21.15.90:6805/3635087609","heartbeat_back_addr":"172.21.15.90:6809/3635087609","heartbeat_front_addr":"172.21.15.90:6807/3635087609","state":["exists","up"]},{"osd":1,"uuid":"124139cd-5793-463e-9607-9d9c10164183","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":1994536059},{"type":"v1","addr":"172.21.15.92:6801","nonce":1994536059}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":1994536059},{"type":"v1","addr":"172.21.15.92:6803","nonce":1994536059}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":1994536059},{"type":"v1","addr":"172.21.15.92:6807","nonce":1994536059}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":1994536059},{"type":"v1","addr":"172.21.15.92:6805","nonce":1994536059}]},"public_addr":"172.21.15.92:6801/1994536059","cluster_addr":"172.21.15.92:6803/1994536059","heartbeat_back_addr":"172.21.15.92:6807/1994536059","heartbeat_front_addr":"172.21.15.92:6805/1994536059","state":["exists","up"]},{"osd":2,"uuid":"a61fbef2-9dca-4875-a814-020fff0db0e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":1358472800},{"type":"v1","addr":"172.21.15.177:6801","nonce":1358472800}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":1358472800},{"type":"v1","addr":"172.21.15.177:6803","nonce":1358472800}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":1358472800},{"type":"v1","addr":"172.21.15.177:6807","nonce":1358472800}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":1358472800},{"type":"v1","addr":"172.21.15.177:6805","nonce":1358472800}]},"public_addr":"172.21.15.177:6801/1358472800","cluster_addr":"172.21.15.177:6803/1358472800","heartbeat_back_addr":"172.21.15.177:6807/1358472800","heartbeat_front_addr":"172.21.15.177:6805/1358472800","state":["exists","up"]},{"osd":3,"uuid":"b175a2f4-7859-4eb0-bc1f-fba8c2b92a2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":3114811216},{"type":"v1","addr":"172.21.15.90:6811","nonce":3114811216}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":3114811216},{"type":"v1","addr":"172.21.15.90:6813","nonce":3114811216}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":3114811216},{"type":"v1","addr":"172.21.15.90:6817","nonce":3114811216}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":3114811216},{"type":"v1","addr":"172.21.15.90:6815","nonce":3114811216}]},"public_addr":"172.21.15.90:6811/3114811216","cluster_addr":"172.21.15.90:6813/3114811216","heartbeat_back_addr":"172.21.15.90:6817/3114811216","heartbeat_front_addr":"172.21.15.90:6815/3114811216","state":["exists","up"]},{"osd":4,"uuid":"38cc68ea-c109-4d8d-b77d-1c6f9b56c875","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":2987208879},{"type":"v1","addr":"172.21.15.92:6809","nonce":2987208879}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6810","nonce":2987208879},{"type":"v1","addr":"172.21.15.92:6811","nonce":2987208879}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6814","nonce":2987208879},{"type":"v1","addr":"172.21.15.92:6815","nonce":2987208879}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":2987208879},{"type":"v1","addr":"172.21.15.92:6813","nonce":2987208879}]},"public_addr":"172.21.15.92:6809/2987208879","cluster_addr":"172.21.15.92:6811/2987208879","heartbeat_back_addr":"172.21.15.92:6815/2987208879","heartbeat_front_addr":"172.21.15.92:6813/2987208879","state":["exists","up"]},{"osd":5,"uuid":"2347232e-ac91-466f-8ac9-496f397679ba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":3486820237},{"type":"v1","addr":"172.21.15.177:6809","nonce":3486820237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":3486820237},{"type":"v1","addr":"172.21.15.177:6811","nonce":3486820237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":3486820237},{"type":"v1","addr":"172.21.15.177:6815","nonce":3486820237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":3486820237},{"type":"v1","addr":"172.21.15.177:6813","nonce":3486820237}]},"public_addr":"172.21.15.177:6809/3486820237","cluster_addr":"172.21.15.177:6811/3486820237","heartbeat_back_addr":"172.21.15.177:6815/3486820237","heartbeat_front_addr":"172.21.15.177:6813/3486820237","state":["exists","up"]},{"osd":6,"uuid":"aa72392f-0498-44db-a8d2-eccc89a64871","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":2579716443},{"type":"v1","addr":"172.21.15.90:6819","nonce":2579716443}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6820","nonce":2579716443},{"type":"v1","addr":"172.21.15.90:6821","nonce":2579716443}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":2579716443},{"type":"v1","addr":"172.21.15.90:6825","nonce":2579716443}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2579716443},{"type":"v1","addr":"172.21.15.90:6823","nonce":2579716443}]},"public_addr":"172.21.15.90:6819/2579716443","cluster_addr":"172.21.15.90:6821/2579716443","heartbeat_back_addr":"172.21.15.90:6825/2579716443","heartbeat_front_addr":"172.21.15.90:6823/2579716443","state":["exists","up"]},{"osd":7,"uuid":"931bd105-bc14-4a1e-bfe4-c7ad58fa4b73","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":3371649139},{"type":"v1","addr":"172.21.15.92:6817","nonce":3371649139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":3371649139},{"type":"v1","addr":"172.21.15.92:6819","nonce":3371649139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":3371649139},{"type":"v1","addr":"172.21.15.92:6823","nonce":3371649139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6820","nonce":3371649139},{"type":"v1","addr":"172.21.15.92:6821","nonce":3371649139}]},"public_addr":"172.21.15.92:6817/3371649139","cluster_addr":"172.21.15.92:6819/3371649139","heartbeat_back_addr":"172.21.15.92:6823/3371649139","heartbeat_front_addr":"172.21.15.92:6821/3371649139","state":["exists","up"]},{"osd":8,"uuid":"82624ec2-b382-4886-9f40-7f5f376de3ce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":1459053034},{"type":"v1","addr":"172.21.15.177:6817","nonce":1459053034}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":1459053034},{"type":"v1","addr":"172.21.15.177:6819","nonce":1459053034}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":1459053034},{"type":"v1","addr":"172.21.15.177:6823","nonce":1459053034}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":1459053034},{"type":"v1","addr":"172.21.15.177:6821","nonce":1459053034}]},"public_addr":"172.21.15.177:6817/1459053034","cluster_addr":"172.21.15.177:6819/1459053034","heartbeat_back_addr":"172.21.15.177:6823/1459053034","heartbeat_front_addr":"172.21.15.177:6821/1459053034","state":["exists","up"]},{"osd":9,"uuid":"fbe36e48-3468-42b2-8094-e16552f3cb16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":2592604332},{"type":"v1","addr":"172.21.15.90:6827","nonce":2592604332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":2592604332},{"type":"v1","addr":"172.21.15.90:6829","nonce":2592604332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6832","nonce":2592604332},{"type":"v1","addr":"172.21.15.90:6833","nonce":2592604332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":2592604332},{"type":"v1","addr":"172.21.15.90:6831","nonce":2592604332}]},"public_addr":"172.21.15.90:6827/2592604332","cluster_addr":"172.21.15.90:6829/2592604332","heartbeat_back_addr":"172.21.15.90:6833/2592604332","heartbeat_front_addr":"172.21.15.90:6831/2592604332","state":["exists","up"]},{"osd":10,"uuid":"9240df75-e7e2-47f1-8aae-534b70b3c210","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":1594153142},{"type":"v1","addr":"172.21.15.92:6825","nonce":1594153142}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6826","nonce":1594153142},{"type":"v1","addr":"172.21.15.92:6827","nonce":1594153142}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":1594153142},{"type":"v1","addr":"172.21.15.92:6831","nonce":1594153142}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":1594153142},{"type":"v1","addr":"172.21.15.92:6829","nonce":1594153142}]},"public_addr":"172.21.15.92:6825/1594153142","cluster_addr":"172.21.15.92:6827/1594153142","heartbeat_back_addr":"172.21.15.92:6831/1594153142","heartbeat_front_addr":"172.21.15.92:6829/1594153142","state":["exists","up"]},{"osd":11,"uuid":"83cba9fa-b745-4df1-adda-e86837d4b91a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":83,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1622582126},{"type":"v1","addr":"172.21.15.177:6825","nonce":1622582126}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":1622582126},{"type":"v1","addr":"172.21.15.177:6827","nonce":1622582126}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":1622582126},{"type":"v1","addr":"172.21.15.177:6831","nonce":1622582126}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1622582126},{"type":"v1","addr":"172.21.15.177:6829","nonce":1622582126}]},"public_addr":"172.21.15.177:6825/1622582126","cluster_addr":"172.21.15.177:6827/1622582126","heartbeat_back_addr":"172.21.15.177:6831/1622582126","heartbeat_front_addr":"172.21.15.177:6829/1622582126","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:26:24.290365+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:26:49.405988+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:27:13.639196+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:27:45.603746+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:28:11.019126+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:28:35.084163+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:29:08.064626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:29:33.266133+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:29:57.381816+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:30:30.286847+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:30:56.226279+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:31:20.809611+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.90:0/314641601":"2024-04-03T14:24:34.595561+0000","172.21.15.90:0/1400895868":"2024-04-03T14:24:34.595561+0000","172.21.15.90:6801/129436436":"2024-04-03T14:24:34.595561+0000","172.21.15.90:6801/2372398598":"2024-04-03T14:24:00.362147+0000","172.21.15.90:0/278569338":"2024-04-03T14:24:34.595561+0000","172.21.15.90:0/2321464594":"2024-04-03T14:24:00.362147+0000","172.21.15.90:6800/2372398598":"2024-04-03T14:24:00.362147+0000","172.21.15.90:0/2962656672":"2024-04-03T14:24:00.362147+0000","172.21.15.90:0/126693610":"2024-04-03T14:24:00.362147+0000","172.21.15.90:6800/129436436":"2024-04-03T14:24:34.595561+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T14:32:23.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:23.295+0000 7fa6d3c00640 1 -- 172.21.15.90:0/3911953847 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fa6ac0a3bb0 msgr2=0x7fa6ac0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:32:23.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:23.295+0000 7fa6d3c00640 1 --2- 172.21.15.90:0/3911953847 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fa6ac0a3bb0 0x7fa6ac0a6070 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fa6b402c7b0 tx=0x7fa6b4005fd0 comp rx=0 tx=0).stop 2024-04-02T14:32:23.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:23.295+0000 7fa6d3c00640 1 -- 172.21.15.90:0/3911953847 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa6cc108140 msgr2=0x7fa6cc19e3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:32:23.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:23.295+0000 7fa6d3c00640 1 --2- 172.21.15.90:0/3911953847 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa6cc108140 0x7fa6cc19e3f0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fa6bc00e9c0 tx=0x7fa6bc00ee80 comp rx=0 tx=0).stop 2024-04-02T14:32:23.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:23.295+0000 7fa6d3c00640 1 -- 172.21.15.90:0/3911953847 shutdown_connections 2024-04-02T14:32:23.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:23.295+0000 7fa6d3c00640 1 --2- 172.21.15.90:0/3911953847 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fa6ac0a3bb0 0x7fa6ac0a6070 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:23.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:23.295+0000 7fa6d3c00640 1 --2- 172.21.15.90:0/3911953847 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa6cc109cf0 0x7fa6cc1a59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:23.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:23.295+0000 7fa6d3c00640 1 --2- 172.21.15.90:0/3911953847 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa6cc109330 0x7fa6cc19e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:23.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:23.295+0000 7fa6d3c00640 1 --2- 172.21.15.90:0/3911953847 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa6cc108140 0x7fa6cc19e3f0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:23.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:23.295+0000 7fa6d3c00640 1 -- 172.21.15.90:0/3911953847 >> 172.21.15.90:0/3911953847 conn(0x7fa6cc077110 msgr2=0x7fa6cc10c630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:32:23.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:23.295+0000 7fa6d3c00640 1 -- 172.21.15.90:0/3911953847 shutdown_connections 2024-04-02T14:32:23.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:23.295+0000 7fa6d3c00640 1 -- 172.21.15.90:0/3911953847 wait complete. 2024-04-02T14:32:23.316 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T14:32:23.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:22 smithi092 ceph-mon[36497]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-04-02T14:32:23.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:22 smithi092 ceph-mon[36497]: osdmap e86: 12 total, 12 up, 12 in 2024-04-02T14:32:23.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:22 smithi092 ceph-mon[36497]: from='client.? 172.21.15.90:0/3409446908' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-02T14:32:23.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:22 smithi092 ceph-mon[36497]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-02T14:32:23.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:22 smithi092 ceph-mon[36497]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-04-02T14:32:23.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:22 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:32:23.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:22 smithi092 ceph-mon[36497]: osdmap e87: 12 total, 12 up, 12 in 2024-04-02T14:32:23.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:22 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:32:23.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:22 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:32:23.362 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:22 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T14:32:23.362 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:22 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:32:23.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:23.524+0000 7f5abca00640 1 -- 172.21.15.90:0/3313918344 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5ab8103520 msgr2=0x7f5ab8109db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:32:23.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:23.524+0000 7f5abca00640 1 --2- 172.21.15.90:0/3313918344 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5ab8103520 0x7f5ab8109db0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f5ab0009ca0 tx=0x7f5ab002c100 comp rx=0 tx=0).stop 2024-04-02T14:32:23.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:23.524+0000 7f5abca00640 1 -- 172.21.15.90:0/3313918344 shutdown_connections 2024-04-02T14:32:23.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:23.524+0000 7f5abca00640 1 --2- 172.21.15.90:0/3313918344 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5ab8103520 0x7f5ab8109db0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:23.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:23.524+0000 7f5abca00640 1 --2- 172.21.15.90:0/3313918344 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5ab8102b60 0x7f5ab8102fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:23.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:23.524+0000 7f5abca00640 1 --2- 172.21.15.90:0/3313918344 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5ab8101970 0x7f5ab8101d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:23.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:23.524+0000 7f5abca00640 1 -- 172.21.15.90:0/3313918344 >> 172.21.15.90:0/3313918344 conn(0x7f5ab80fcee0 msgr2=0x7f5ab80ff320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:32:23.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:23.524+0000 7f5abca00640 1 -- 172.21.15.90:0/3313918344 shutdown_connections 2024-04-02T14:32:23.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:23.524+0000 7f5abca00640 1 -- 172.21.15.90:0/3313918344 wait complete. 2024-04-02T14:32:23.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:23.525+0000 7f5abca00640 1 Processor -- start 2024-04-02T14:32:23.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:23.525+0000 7f5abca00640 1 -- start start 2024-04-02T14:32:23.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:23.525+0000 7f5abca00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5ab8101970 0x7f5ab8199d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:32:23.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:23.525+0000 7f5abca00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5ab8102b60 0x7f5ab819a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:32:23.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:23.525+0000 7f5abca00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5ab8103520 0x7f5ab81a1320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:32:23.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:23.525+0000 7f5abca00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f5ab810d5d0 con 0x7f5ab8103520 2024-04-02T14:32:23.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:23.525+0000 7f5abca00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f5ab810d450 con 0x7f5ab8101970 2024-04-02T14:32:23.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:23.525+0000 7f5abca00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f5ab810d750 con 0x7f5ab8102b60 2024-04-02T14:32:23.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:23.526+0000 7f5ab7200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5ab8103520 0x7f5ab81a1320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:32:23.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:23.526+0000 7f5ab7200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5ab8103520 0x7f5ab81a1320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:54456/0 (socket says 172.21.15.90:54456) 2024-04-02T14:32:23.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:23.526+0000 7f5ab7200640 1 -- 172.21.15.90:0/1004327531 learned_addr learned my addr 172.21.15.90:0/1004327531 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:32:23.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:23.526+0000 7f5ab6800640 1 --2- 172.21.15.90:0/1004327531 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5ab8101970 0x7f5ab8199d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:32:23.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:23.526+0000 7f5aafe00640 1 --2- 172.21.15.90:0/1004327531 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5ab8102b60 0x7f5ab819a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:32:23.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:23.526+0000 7f5ab6800640 1 -- 172.21.15.90:0/1004327531 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5ab8102b60 msgr2=0x7f5ab819a2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:32:23.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:23.526+0000 7f5ab6800640 1 --2- 172.21.15.90:0/1004327531 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5ab8102b60 0x7f5ab819a2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:23.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:23.526+0000 7f5ab6800640 1 -- 172.21.15.90:0/1004327531 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5ab8103520 msgr2=0x7f5ab81a1320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:32:23.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:23.526+0000 7f5ab6800640 1 --2- 172.21.15.90:0/1004327531 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5ab8103520 0x7f5ab81a1320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:23.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:23.526+0000 7f5ab6800640 1 -- 172.21.15.90:0/1004327531 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ab81a1990 con 0x7f5ab8101970 2024-04-02T14:32:23.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:23.527+0000 7f5ab6800640 1 --2- 172.21.15.90:0/1004327531 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5ab8101970 0x7f5ab8199d70 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f5aa800d6a0 tx=0x7f5aa800daa0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:32:23.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:23.527+0000 7f5aaf400640 1 -- 172.21.15.90:0/1004327531 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5aa8017070 con 0x7f5ab8101970 2024-04-02T14:32:23.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:23.527+0000 7f5abca00640 1 -- 172.21.15.90:0/1004327531 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ab81a1c80 con 0x7f5ab8101970 2024-04-02T14:32:23.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:23.527+0000 7f5aaf400640 1 -- 172.21.15.90:0/1004327531 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5aa8009c30 con 0x7f5ab8101970 2024-04-02T14:32:23.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:23.527+0000 7f5aaf400640 1 -- 172.21.15.90:0/1004327531 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5aa8003b60 con 0x7f5ab8101970 2024-04-02T14:32:23.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:23.527+0000 7f5abca00640 1 -- 172.21.15.90:0/1004327531 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ab81a21d0 con 0x7f5ab8101970 2024-04-02T14:32:23.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:23.528+0000 7f5abca00640 1 -- 172.21.15.90:0/1004327531 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5ab81a23a0 con 0x7f5ab8101970 2024-04-02T14:32:23.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:23.530+0000 7f5aaf400640 1 -- 172.21.15.90:0/1004327531 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f5aa80042b0 con 0x7f5ab8101970 2024-04-02T14:32:23.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:23.531+0000 7f5aaf400640 1 --2- 172.21.15.90:0/1004327531 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f5a980a3bb0 0x7f5a980a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:32:23.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:23.531+0000 7f5aaf400640 1 -- 172.21.15.90:0/1004327531 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 7979+0+0 (secure 0 0 0) 0x7f5aa80cfc90 con 0x7f5ab8101970 2024-04-02T14:32:23.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:23.531+0000 7f5aafe00640 1 --2- 172.21.15.90:0/1004327531 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f5a980a3bb0 0x7f5a980a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:32:23.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:23.532+0000 7f5aafe00640 1 --2- 172.21.15.90:0/1004327531 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f5a980a3bb0 0x7f5a980a6070 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f5ab819b280 tx=0x7f5aa000a2d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:32:23.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:23.536+0000 7f5aaf400640 1 -- 172.21.15.90:0/1004327531 <== mon.2 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7f5aa8009de0 con 0x7f5ab8101970 2024-04-02T14:32:23.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:23.738+0000 7f5abca00640 1 -- 172.21.15.90:0/1004327531 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f5ab81a2b70 con 0x7f5ab8101970 2024-04-02T14:32:23.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:23.740+0000 7f5aaf400640 1 -- 172.21.15.90:0/1004327531 <== mon.2 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+22204 (secure 0 0 0) 0x7f5aa809c720 con 0x7f5ab8101970 2024-04-02T14:32:23.743 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-02T14:32:23.743 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":88,"fsid":"6449448a-f0fc-11ee-b647-cb9ed24678a4","created":"2024-04-02T14:23:16.530201+0000","modified":"2024-04-02T14:32:23.517146+0000","last_up_change":"2024-04-02T14:31:23.819732+0000","last_in_change":"2024-04-02T14:31:02.824284+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T14:27:16.758622+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-02T14:32:20.016331+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-02T14:32:21.294367+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a1a6a856-ab44-44bd-bc5e-45c718a70cd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":3635087609},{"type":"v1","addr":"172.21.15.90:6803","nonce":3635087609}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":3635087609},{"type":"v1","addr":"172.21.15.90:6805","nonce":3635087609}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":3635087609},{"type":"v1","addr":"172.21.15.90:6809","nonce":3635087609}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":3635087609},{"type":"v1","addr":"172.21.15.90:6807","nonce":3635087609}]},"public_addr":"172.21.15.90:6803/3635087609","cluster_addr":"172.21.15.90:6805/3635087609","heartbeat_back_addr":"172.21.15.90:6809/3635087609","heartbeat_front_addr":"172.21.15.90:6807/3635087609","state":["exists","up"]},{"osd":1,"uuid":"124139cd-5793-463e-9607-9d9c10164183","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":1994536059},{"type":"v1","addr":"172.21.15.92:6801","nonce":1994536059}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":1994536059},{"type":"v1","addr":"172.21.15.92:6803","nonce":1994536059}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":1994536059},{"type":"v1","addr":"172.21.15.92:6807","nonce":1994536059}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":1994536059},{"type":"v1","addr":"172.21.15.92:6805","nonce":1994536059}]},"public_addr":"172.21.15.92:6801/1994536059","cluster_addr":"172.21.15.92:6803/1994536059","heartbeat_back_addr":"172.21.15.92:6807/1994536059","heartbeat_front_addr":"172.21.15.92:6805/1994536059","state":["exists","up"]},{"osd":2,"uuid":"a61fbef2-9dca-4875-a814-020fff0db0e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":1358472800},{"type":"v1","addr":"172.21.15.177:6801","nonce":1358472800}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":1358472800},{"type":"v1","addr":"172.21.15.177:6803","nonce":1358472800}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":1358472800},{"type":"v1","addr":"172.21.15.177:6807","nonce":1358472800}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":1358472800},{"type":"v1","addr":"172.21.15.177:6805","nonce":1358472800}]},"public_addr":"172.21.15.177:6801/1358472800","cluster_addr":"172.21.15.177:6803/1358472800","heartbeat_back_addr":"172.21.15.177:6807/1358472800","heartbeat_front_addr":"172.21.15.177:6805/1358472800","state":["exists","up"]},{"osd":3,"uuid":"b175a2f4-7859-4eb0-bc1f-fba8c2b92a2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":3114811216},{"type":"v1","addr":"172.21.15.90:6811","nonce":3114811216}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":3114811216},{"type":"v1","addr":"172.21.15.90:6813","nonce":3114811216}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":3114811216},{"type":"v1","addr":"172.21.15.90:6817","nonce":3114811216}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":3114811216},{"type":"v1","addr":"172.21.15.90:6815","nonce":3114811216}]},"public_addr":"172.21.15.90:6811/3114811216","cluster_addr":"172.21.15.90:6813/3114811216","heartbeat_back_addr":"172.21.15.90:6817/3114811216","heartbeat_front_addr":"172.21.15.90:6815/3114811216","state":["exists","up"]},{"osd":4,"uuid":"38cc68ea-c109-4d8d-b77d-1c6f9b56c875","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":2987208879},{"type":"v1","addr":"172.21.15.92:6809","nonce":2987208879}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6810","nonce":2987208879},{"type":"v1","addr":"172.21.15.92:6811","nonce":2987208879}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6814","nonce":2987208879},{"type":"v1","addr":"172.21.15.92:6815","nonce":2987208879}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":2987208879},{"type":"v1","addr":"172.21.15.92:6813","nonce":2987208879}]},"public_addr":"172.21.15.92:6809/2987208879","cluster_addr":"172.21.15.92:6811/2987208879","heartbeat_back_addr":"172.21.15.92:6815/2987208879","heartbeat_front_addr":"172.21.15.92:6813/2987208879","state":["exists","up"]},{"osd":5,"uuid":"2347232e-ac91-466f-8ac9-496f397679ba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":3486820237},{"type":"v1","addr":"172.21.15.177:6809","nonce":3486820237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":3486820237},{"type":"v1","addr":"172.21.15.177:6811","nonce":3486820237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":3486820237},{"type":"v1","addr":"172.21.15.177:6815","nonce":3486820237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":3486820237},{"type":"v1","addr":"172.21.15.177:6813","nonce":3486820237}]},"public_addr":"172.21.15.177:6809/3486820237","cluster_addr":"172.21.15.177:6811/3486820237","heartbeat_back_addr":"172.21.15.177:6815/3486820237","heartbeat_front_addr":"172.21.15.177:6813/3486820237","state":["exists","up"]},{"osd":6,"uuid":"aa72392f-0498-44db-a8d2-eccc89a64871","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":2579716443},{"type":"v1","addr":"172.21.15.90:6819","nonce":2579716443}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6820","nonce":2579716443},{"type":"v1","addr":"172.21.15.90:6821","nonce":2579716443}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":2579716443},{"type":"v1","addr":"172.21.15.90:6825","nonce":2579716443}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2579716443},{"type":"v1","addr":"172.21.15.90:6823","nonce":2579716443}]},"public_addr":"172.21.15.90:6819/2579716443","cluster_addr":"172.21.15.90:6821/2579716443","heartbeat_back_addr":"172.21.15.90:6825/2579716443","heartbeat_front_addr":"172.21.15.90:6823/2579716443","state":["exists","up"]},{"osd":7,"uuid":"931bd105-bc14-4a1e-bfe4-c7ad58fa4b73","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":3371649139},{"type":"v1","addr":"172.21.15.92:6817","nonce":3371649139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":3371649139},{"type":"v1","addr":"172.21.15.92:6819","nonce":3371649139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":3371649139},{"type":"v1","addr":"172.21.15.92:6823","nonce":3371649139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6820","nonce":3371649139},{"type":"v1","addr":"172.21.15.92:6821","nonce":3371649139}]},"public_addr":"172.21.15.92:6817/3371649139","cluster_addr":"172.21.15.92:6819/3371649139","heartbeat_back_addr":"172.21.15.92:6823/3371649139","heartbeat_front_addr":"172.21.15.92:6821/3371649139","state":["exists","up"]},{"osd":8,"uuid":"82624ec2-b382-4886-9f40-7f5f376de3ce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":1459053034},{"type":"v1","addr":"172.21.15.177:6817","nonce":1459053034}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":1459053034},{"type":"v1","addr":"172.21.15.177:6819","nonce":1459053034}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":1459053034},{"type":"v1","addr":"172.21.15.177:6823","nonce":1459053034}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":1459053034},{"type":"v1","addr":"172.21.15.177:6821","nonce":1459053034}]},"public_addr":"172.21.15.177:6817/1459053034","cluster_addr":"172.21.15.177:6819/1459053034","heartbeat_back_addr":"172.21.15.177:6823/1459053034","heartbeat_front_addr":"172.21.15.177:6821/1459053034","state":["exists","up"]},{"osd":9,"uuid":"fbe36e48-3468-42b2-8094-e16552f3cb16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":2592604332},{"type":"v1","addr":"172.21.15.90:6827","nonce":2592604332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":2592604332},{"type":"v1","addr":"172.21.15.90:6829","nonce":2592604332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6832","nonce":2592604332},{"type":"v1","addr":"172.21.15.90:6833","nonce":2592604332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":2592604332},{"type":"v1","addr":"172.21.15.90:6831","nonce":2592604332}]},"public_addr":"172.21.15.90:6827/2592604332","cluster_addr":"172.21.15.90:6829/2592604332","heartbeat_back_addr":"172.21.15.90:6833/2592604332","heartbeat_front_addr":"172.21.15.90:6831/2592604332","state":["exists","up"]},{"osd":10,"uuid":"9240df75-e7e2-47f1-8aae-534b70b3c210","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":1594153142},{"type":"v1","addr":"172.21.15.92:6825","nonce":1594153142}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6826","nonce":1594153142},{"type":"v1","addr":"172.21.15.92:6827","nonce":1594153142}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":1594153142},{"type":"v1","addr":"172.21.15.92:6831","nonce":1594153142}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":1594153142},{"type":"v1","addr":"172.21.15.92:6829","nonce":1594153142}]},"public_addr":"172.21.15.92:6825/1594153142","cluster_addr":"172.21.15.92:6827/1594153142","heartbeat_back_addr":"172.21.15.92:6831/1594153142","heartbeat_front_addr":"172.21.15.92:6829/1594153142","state":["exists","up"]},{"osd":11,"uuid":"83cba9fa-b745-4df1-adda-e86837d4b91a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":83,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1622582126},{"type":"v1","addr":"172.21.15.177:6825","nonce":1622582126}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":1622582126},{"type":"v1","addr":"172.21.15.177:6827","nonce":1622582126}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":1622582126},{"type":"v1","addr":"172.21.15.177:6831","nonce":1622582126}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1622582126},{"type":"v1","addr":"172.21.15.177:6829","nonce":1622582126}]},"public_addr":"172.21.15.177:6825/1622582126","cluster_addr":"172.21.15.177:6827/1622582126","heartbeat_back_addr":"172.21.15.177:6831/1622582126","heartbeat_front_addr":"172.21.15.177:6829/1622582126","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:26:24.290365+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:26:49.405988+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:27:13.639196+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:27:45.603746+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:28:11.019126+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:28:35.084163+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:29:08.064626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:29:33.266133+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:29:57.381816+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:30:30.286847+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:30:56.226279+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:31:20.809611+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.90:0/314641601":"2024-04-03T14:24:34.595561+0000","172.21.15.90:0/1400895868":"2024-04-03T14:24:34.595561+0000","172.21.15.90:6801/129436436":"2024-04-03T14:24:34.595561+0000","172.21.15.90:6801/2372398598":"2024-04-03T14:24:00.362147+0000","172.21.15.90:0/278569338":"2024-04-03T14:24:34.595561+0000","172.21.15.90:0/2321464594":"2024-04-03T14:24:00.362147+0000","172.21.15.90:6800/2372398598":"2024-04-03T14:24:00.362147+0000","172.21.15.90:0/2962656672":"2024-04-03T14:24:00.362147+0000","172.21.15.90:0/126693610":"2024-04-03T14:24:00.362147+0000","172.21.15.90:6800/129436436":"2024-04-03T14:24:34.595561+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T14:32:23.745 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:23.743+0000 7f5abca00640 1 -- 172.21.15.90:0/1004327531 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f5a980a3bb0 msgr2=0x7f5a980a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:32:23.745 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:23.743+0000 7f5abca00640 1 --2- 172.21.15.90:0/1004327531 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f5a980a3bb0 0x7f5a980a6070 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f5ab819b280 tx=0x7f5aa000a2d0 comp rx=0 tx=0).stop 2024-04-02T14:32:23.745 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:23.743+0000 7f5abca00640 1 -- 172.21.15.90:0/1004327531 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5ab8101970 msgr2=0x7f5ab8199d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:32:23.745 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:23.743+0000 7f5abca00640 1 --2- 172.21.15.90:0/1004327531 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5ab8101970 0x7f5ab8199d70 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f5aa800d6a0 tx=0x7f5aa800daa0 comp rx=0 tx=0).stop 2024-04-02T14:32:23.745 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:23.743+0000 7f5abca00640 1 -- 172.21.15.90:0/1004327531 shutdown_connections 2024-04-02T14:32:23.746 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:23.743+0000 7f5abca00640 1 --2- 172.21.15.90:0/1004327531 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f5a980a3bb0 0x7f5a980a6070 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:23.746 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:23.743+0000 7f5abca00640 1 --2- 172.21.15.90:0/1004327531 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5ab8103520 0x7f5ab81a1320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:23.746 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:23.743+0000 7f5abca00640 1 --2- 172.21.15.90:0/1004327531 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5ab8102b60 0x7f5ab819a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:23.746 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:23.743+0000 7f5abca00640 1 --2- 172.21.15.90:0/1004327531 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5ab8101970 0x7f5ab8199d70 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:23.746 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:23.743+0000 7f5abca00640 1 -- 172.21.15.90:0/1004327531 >> 172.21.15.90:0/1004327531 conn(0x7f5ab80fcee0 msgr2=0x7f5ab8073d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:32:23.746 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:23.743+0000 7f5abca00640 1 -- 172.21.15.90:0/1004327531 shutdown_connections 2024-04-02T14:32:23.746 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:23.743+0000 7f5abca00640 1 -- 172.21.15.90:0/1004327531 wait complete. 2024-04-02T14:32:23.763 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-04-02T14:32:23.980 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:23.977+0000 7ff070800640 1 -- 172.21.15.90:0/1049512024 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff068109310 msgr2=0x7ff068109790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:32:23.980 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:23.977+0000 7ff070800640 1 --2- 172.21.15.90:0/1049512024 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff068109310 0x7ff068109790 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7ff058009860 tx=0x7ff05802bb70 comp rx=0 tx=0).stop 2024-04-02T14:32:23.980 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:23.977+0000 7ff070800640 1 -- 172.21.15.90:0/1049512024 shutdown_connections 2024-04-02T14:32:23.980 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:23.977+0000 7ff070800640 1 --2- 172.21.15.90:0/1049512024 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff068109cd0 0x7ff068110560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:23.980 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:23.977+0000 7ff070800640 1 --2- 172.21.15.90:0/1049512024 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff068109310 0x7ff068109790 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:23.980 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:23.977+0000 7ff070800640 1 --2- 172.21.15.90:0/1049512024 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff068108120 0x7ff068108540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:23.980 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:23.977+0000 7ff070800640 1 -- 172.21.15.90:0/1049512024 >> 172.21.15.90:0/1049512024 conn(0x7ff0680770c0 msgr2=0x7ff068079500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:32:23.980 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:23.978+0000 7ff070800640 1 -- 172.21.15.90:0/1049512024 shutdown_connections 2024-04-02T14:32:23.980 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:23.978+0000 7ff070800640 1 -- 172.21.15.90:0/1049512024 wait complete. 2024-04-02T14:32:23.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:23.979+0000 7ff070800640 1 Processor -- start 2024-04-02T14:32:23.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:23.979+0000 7ff070800640 1 -- start start 2024-04-02T14:32:23.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:23.979+0000 7ff070800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff068108120 0x7ff06819e440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:32:23.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:23.979+0000 7ff070800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff068109310 0x7ff06819e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:32:23.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:23.979+0000 7ff070800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff068109cd0 0x7ff0681a59f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:32:23.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:23.979+0000 7ff070800640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7ff068113db0 con 0x7ff068109cd0 2024-04-02T14:32:23.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:23.979+0000 7ff070800640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7ff068113c30 con 0x7ff068109310 2024-04-02T14:32:23.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:23.979+0000 7ff070800640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7ff068113f30 con 0x7ff068108120 2024-04-02T14:32:23.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:23.979+0000 7ff06e800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff068109cd0 0x7ff0681a59f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:32:23.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:23.979+0000 7ff06e800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff068109cd0 0x7ff0681a59f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:49270/0 (socket says 172.21.15.90:49270) 2024-04-02T14:32:23.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:23.979+0000 7ff06e800640 1 -- 172.21.15.90:0/3764204241 learned_addr learned my addr 172.21.15.90:0/3764204241 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:32:23.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:23.980+0000 7ff06d400640 1 --2- 172.21.15.90:0/3764204241 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff068109310 0x7ff06819e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:32:23.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:23.980+0000 7ff06de00640 1 --2- 172.21.15.90:0/3764204241 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff068108120 0x7ff06819e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:32:23.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:23.980+0000 7ff06e800640 1 -- 172.21.15.90:0/3764204241 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff068108120 msgr2=0x7ff06819e440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:32:23.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:23.980+0000 7ff06e800640 1 --2- 172.21.15.90:0/3764204241 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff068108120 0x7ff06819e440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:23.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:23.980+0000 7ff06e800640 1 -- 172.21.15.90:0/3764204241 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff068109310 msgr2=0x7ff06819e980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:32:23.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:23.980+0000 7ff06e800640 1 --2- 172.21.15.90:0/3764204241 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff068109310 0x7ff06819e980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:23.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:23.980+0000 7ff06e800640 1 -- 172.21.15.90:0/3764204241 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0681a6060 con 0x7ff068109cd0 2024-04-02T14:32:23.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:23.980+0000 7ff06d400640 1 --2- 172.21.15.90:0/3764204241 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff068109310 0x7ff06819e980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:32:23.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:23.980+0000 7ff06e800640 1 --2- 172.21.15.90:0/3764204241 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff068109cd0 0x7ff0681a59f0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7ff06400efc0 tx=0x7ff06400c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:32:23.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:23.980+0000 7ff062a00640 1 -- 172.21.15.90:0/3764204241 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff064003970 con 0x7ff068109cd0 2024-04-02T14:32:23.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:23.980+0000 7ff062a00640 1 -- 172.21.15.90:0/3764204241 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff064004020 con 0x7ff068109cd0 2024-04-02T14:32:23.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:23.980+0000 7ff062a00640 1 -- 172.21.15.90:0/3764204241 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff06400b9f0 con 0x7ff068109cd0 2024-04-02T14:32:23.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:23.980+0000 7ff070800640 1 -- 172.21.15.90:0/3764204241 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0681a6350 con 0x7ff068109cd0 2024-04-02T14:32:23.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:23.981+0000 7ff070800640 1 -- 172.21.15.90:0/3764204241 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0681a6820 con 0x7ff068109cd0 2024-04-02T14:32:23.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:23.982+0000 7ff070800640 1 -- 172.21.15.90:0/3764204241 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff06810f440 con 0x7ff068109cd0 2024-04-02T14:32:23.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:23.984+0000 7ff062a00640 1 -- 172.21.15.90:0/3764204241 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7ff06401d020 con 0x7ff068109cd0 2024-04-02T14:32:23.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:23.985+0000 7ff062a00640 1 --2- 172.21.15.90:0/3764204241 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7ff0500a3bb0 0x7ff0500a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:32:23.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:23.985+0000 7ff062a00640 1 -- 172.21.15.90:0/3764204241 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 7979+0+0 (secure 0 0 0) 0x7ff0640d07a0 con 0x7ff068109cd0 2024-04-02T14:32:23.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:23.985+0000 7ff06de00640 1 --2- 172.21.15.90:0/3764204241 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7ff0500a3bb0 0x7ff0500a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:32:23.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:23.986+0000 7ff06de00640 1 --2- 172.21.15.90:0/3764204241 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7ff0500a3bb0 0x7ff0500a6070 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7ff068109170 tx=0x7ff05c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:32:23.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:23.988+0000 7ff062a00640 1 -- 172.21.15.90:0/3764204241 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7ff06409d230 con 0x7ff068109cd0 2024-04-02T14:32:24.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:23 smithi177 ceph-mon[36431]: pgmap v311: 97 pgs: 18 creating+peering, 65 unknown, 14 active+clean; 577 KiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-02T14:32:24.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:23 smithi177 ceph-mon[36431]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-04-02T14:32:24.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:23 smithi177 ceph-mon[36431]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-04-02T14:32:24.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:23 smithi177 ceph-mon[36431]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-04-02T14:32:24.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:23 smithi177 ceph-mon[36431]: from='client.? 172.21.15.90:0/3911953847' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-02T14:32:24.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:23 smithi177 ceph-mon[36431]: osdmap e88: 12 total, 12 up, 12 in 2024-04-02T14:32:24.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:23 smithi177 ceph-mon[36431]: from='client.? 172.21.15.90:0/1004327531' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-02T14:32:24.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:23 smithi090 ceph-mon[31674]: pgmap v311: 97 pgs: 18 creating+peering, 65 unknown, 14 active+clean; 577 KiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-02T14:32:24.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:23 smithi090 ceph-mon[31674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-04-02T14:32:24.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:23 smithi090 ceph-mon[31674]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-04-02T14:32:24.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:23 smithi090 ceph-mon[31674]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-04-02T14:32:24.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:23 smithi090 ceph-mon[31674]: from='client.? 172.21.15.90:0/3911953847' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-02T14:32:24.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:23 smithi090 ceph-mon[31674]: osdmap e88: 12 total, 12 up, 12 in 2024-04-02T14:32:24.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:23 smithi090 ceph-mon[31674]: from='client.? 172.21.15.90:0/1004327531' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-02T14:32:24.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:24.228+0000 7ff070800640 1 -- 172.21.15.90:0/3764204241 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7ff06802d370 con 0x7ff068109cd0 2024-04-02T14:32:24.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:23 smithi092 ceph-mon[36497]: pgmap v311: 97 pgs: 18 creating+peering, 65 unknown, 14 active+clean; 577 KiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-02T14:32:24.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:23 smithi092 ceph-mon[36497]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-04-02T14:32:24.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:23 smithi092 ceph-mon[36497]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-04-02T14:32:24.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:23 smithi092 ceph-mon[36497]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-04-02T14:32:24.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:23 smithi092 ceph-mon[36497]: from='client.? 172.21.15.90:0/3911953847' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-02T14:32:24.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:23 smithi092 ceph-mon[36497]: osdmap e88: 12 total, 12 up, 12 in 2024-04-02T14:32:24.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:23 smithi092 ceph-mon[36497]: from='client.? 172.21.15.90:0/1004327531' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-02T14:32:24.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:24.891+0000 7ff062a00640 1 -- 172.21.15.90:0/3764204241 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v17) ==== 118+0+0 (secure 0 0 0) 0x7ff06409cb30 con 0x7ff068109cd0 2024-04-02T14:32:24.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:24.894+0000 7ff070800640 1 -- 172.21.15.90:0/3764204241 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7ff0500a3bb0 msgr2=0x7ff0500a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:32:24.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:24.894+0000 7ff070800640 1 --2- 172.21.15.90:0/3764204241 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7ff0500a3bb0 0x7ff0500a6070 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7ff068109170 tx=0x7ff05c005f90 comp rx=0 tx=0).stop 2024-04-02T14:32:24.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:24.894+0000 7ff070800640 1 -- 172.21.15.90:0/3764204241 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff068109cd0 msgr2=0x7ff0681a59f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:32:24.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:24.894+0000 7ff070800640 1 --2- 172.21.15.90:0/3764204241 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff068109cd0 0x7ff0681a59f0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7ff06400efc0 tx=0x7ff06400c800 comp rx=0 tx=0).stop 2024-04-02T14:32:24.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:24.894+0000 7ff070800640 1 -- 172.21.15.90:0/3764204241 shutdown_connections 2024-04-02T14:32:24.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:24.894+0000 7ff070800640 1 --2- 172.21.15.90:0/3764204241 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7ff0500a3bb0 0x7ff0500a6070 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:24.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:24.894+0000 7ff070800640 1 --2- 172.21.15.90:0/3764204241 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff068109cd0 0x7ff0681a59f0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:24.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:24.894+0000 7ff070800640 1 --2- 172.21.15.90:0/3764204241 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff068109310 0x7ff06819e980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:24.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:24.894+0000 7ff070800640 1 --2- 172.21.15.90:0/3764204241 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff068108120 0x7ff06819e440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:24.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:24.894+0000 7ff070800640 1 -- 172.21.15.90:0/3764204241 >> 172.21.15.90:0/3764204241 conn(0x7ff0680770c0 msgr2=0x7ff06810c610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:32:24.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:24.895+0000 7ff070800640 1 -- 172.21.15.90:0/3764204241 shutdown_connections 2024-04-02T14:32:24.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:24.895+0000 7ff070800640 1 -- 172.21.15.90:0/3764204241 wait complete. 2024-04-02T14:32:24.918 DEBUG:tasks.cephfs.filesystem:fs_config: {'max_mds': 5, 'session_timeout': 300, 'standby_replay': True, 'subvols': {'create': 2, 'subvol_options': '--namespace-isolated'}} 2024-04-02T14:32:24.918 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 5 2024-04-02T14:32:25.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:25.130+0000 7fe71bc00640 1 -- 172.21.15.90:0/1717021674 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe7141058f0 msgr2=0x7fe71410c180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:32:25.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:25.130+0000 7fe71bc00640 1 --2- 172.21.15.90:0/1717021674 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe7141058f0 0x7fe71410c180 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fe70c009be0 tx=0x7fe70c02bef0 comp rx=0 tx=0).stop 2024-04-02T14:32:25.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:25.130+0000 7fe71bc00640 1 -- 172.21.15.90:0/1717021674 shutdown_connections 2024-04-02T14:32:25.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:25.130+0000 7fe71bc00640 1 --2- 172.21.15.90:0/1717021674 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe7141058f0 0x7fe71410c180 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:25.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:25.130+0000 7fe71bc00640 1 --2- 172.21.15.90:0/1717021674 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe714104f30 0x7fe7141053b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:25.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:25.130+0000 7fe71bc00640 1 --2- 172.21.15.90:0/1717021674 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe714103d40 0x7fe714104160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:25.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:25.130+0000 7fe71bc00640 1 -- 172.21.15.90:0/1717021674 >> 172.21.15.90:0/1717021674 conn(0x7fe7140ff2b0 msgr2=0x7fe7141016f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:32:25.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:25.131+0000 7fe71bc00640 1 -- 172.21.15.90:0/1717021674 shutdown_connections 2024-04-02T14:32:25.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:25.131+0000 7fe71bc00640 1 -- 172.21.15.90:0/1717021674 wait complete. 2024-04-02T14:32:25.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:25.131+0000 7fe71bc00640 1 Processor -- start 2024-04-02T14:32:25.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:25.132+0000 7fe71bc00640 1 -- start start 2024-04-02T14:32:25.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:25.132+0000 7fe71bc00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe714103d40 0x7fe71419a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:32:25.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:25.132+0000 7fe71bc00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe714104f30 0x7fe71419a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:32:25.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:25.132+0000 7fe71bc00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe7141058f0 0x7fe7141a15c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:32:25.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:25.132+0000 7fe71bc00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fe71410f9f0 con 0x7fe714104f30 2024-04-02T14:32:25.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:25.132+0000 7fe71bc00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fe71410f870 con 0x7fe7141058f0 2024-04-02T14:32:25.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:25.132+0000 7fe71bc00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fe71410fb70 con 0x7fe714103d40 2024-04-02T14:32:25.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:25.132+0000 7fe713e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe714104f30 0x7fe71419a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:32:25.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:25.132+0000 7fe713e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe714104f30 0x7fe71419a550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:49280/0 (socket says 172.21.15.90:49280) 2024-04-02T14:32:25.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:25.132+0000 7fe713e00640 1 -- 172.21.15.90:0/3479247345 learned_addr learned my addr 172.21.15.90:0/3479247345 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:32:25.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:25.132+0000 7fe719200640 1 --2- 172.21.15.90:0/3479247345 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe714103d40 0x7fe71419a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:32:25.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:25.132+0000 7fe719c00640 1 --2- 172.21.15.90:0/3479247345 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe7141058f0 0x7fe7141a15c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:32:25.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:25.133+0000 7fe713e00640 1 -- 172.21.15.90:0/3479247345 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe714103d40 msgr2=0x7fe71419a010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:32:25.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:25.133+0000 7fe713e00640 1 --2- 172.21.15.90:0/3479247345 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe714103d40 0x7fe71419a010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:25.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:25.133+0000 7fe713e00640 1 -- 172.21.15.90:0/3479247345 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe7141058f0 msgr2=0x7fe7141a15c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:32:25.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:25.133+0000 7fe713e00640 1 --2- 172.21.15.90:0/3479247345 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe7141058f0 0x7fe7141a15c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:25.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:25.133+0000 7fe713e00640 1 -- 172.21.15.90:0/3479247345 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7141a1c30 con 0x7fe714104f30 2024-04-02T14:32:25.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:25.133+0000 7fe719200640 1 --2- 172.21.15.90:0/3479247345 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe714103d40 0x7fe71419a010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:32:25.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:25.133+0000 7fe719c00640 1 --2- 172.21.15.90:0/3479247345 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe7141058f0 0x7fe7141a15c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:32:25.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:25.133+0000 7fe713e00640 1 --2- 172.21.15.90:0/3479247345 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe714104f30 0x7fe71419a550 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fe70000e9f0 tx=0x7fe70000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:32:25.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:25.133+0000 7fe711600640 1 -- 172.21.15.90:0/3479247345 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe70000ccc0 con 0x7fe714104f30 2024-04-02T14:32:25.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:25.133+0000 7fe711600640 1 -- 172.21.15.90:0/3479247345 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe700010910 con 0x7fe714104f30 2024-04-02T14:32:25.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:25.133+0000 7fe711600640 1 -- 172.21.15.90:0/3479247345 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7000188a0 con 0x7fe714104f30 2024-04-02T14:32:25.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:25.133+0000 7fe71bc00640 1 -- 172.21.15.90:0/3479247345 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe7141a1f20 con 0x7fe714104f30 2024-04-02T14:32:25.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:25.133+0000 7fe71bc00640 1 -- 172.21.15.90:0/3479247345 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7141a23e0 con 0x7fe714104f30 2024-04-02T14:32:25.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:25.134+0000 7fe71bc00640 1 -- 172.21.15.90:0/3479247345 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe71410b060 con 0x7fe714104f30 2024-04-02T14:32:25.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:25.138+0000 7fe711600640 1 -- 172.21.15.90:0/3479247345 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fe700010ac0 con 0x7fe714104f30 2024-04-02T14:32:25.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:25.139+0000 7fe711600640 1 --2- 172.21.15.90:0/3479247345 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fe6fc0a3bb0 0x7fe6fc0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:32:25.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:25.139+0000 7fe711600640 1 -- 172.21.15.90:0/3479247345 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 7979+0+0 (secure 0 0 0) 0x7fe7000d0c30 con 0x7fe714104f30 2024-04-02T14:32:25.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:25.139+0000 7fe719200640 1 --2- 172.21.15.90:0/3479247345 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fe6fc0a3bb0 0x7fe6fc0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:32:25.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:25.140+0000 7fe719200640 1 --2- 172.21.15.90:0/3479247345 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fe6fc0a3bb0 0x7fe6fc0a6070 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fe714104d90 tx=0x7fe708009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:32:25.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:25.142+0000 7fe711600640 1 -- 172.21.15.90:0/3479247345 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7fe70009d6c0 con 0x7fe714104f30 2024-04-02T14:32:25.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:24 smithi177 ceph-mon[36431]: mds.? [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] up:active 2024-04-02T14:32:25.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:24 smithi177 ceph-mon[36431]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-02T14:32:25.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:24 smithi177 ceph-mon[36431]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-02T14:32:25.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:24 smithi177 ceph-mon[36431]: from='client.? 172.21.15.90:0/3764204241' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-02T14:32:25.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:24 smithi090 ceph-mon[31674]: mds.? [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] up:active 2024-04-02T14:32:25.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:24 smithi090 ceph-mon[31674]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-02T14:32:25.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:24 smithi090 ceph-mon[31674]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-02T14:32:25.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:24 smithi090 ceph-mon[31674]: from='client.? 172.21.15.90:0/3764204241' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-02T14:32:25.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:24 smithi092 ceph-mon[36497]: mds.? [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] up:active 2024-04-02T14:32:25.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:24 smithi092 ceph-mon[36497]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-02T14:32:25.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:24 smithi092 ceph-mon[36497]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-02T14:32:25.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:24 smithi092 ceph-mon[36497]: from='client.? 172.21.15.90:0/3764204241' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-02T14:32:25.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:25.370+0000 7fe71bc00640 1 -- 172.21.15.90:0/3479247345 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"} v 0) -- 0x7fe71402d410 con 0x7fe714104f30 2024-04-02T14:32:25.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:25.902+0000 7fe711600640 1 -- 172.21.15.90:0/3479247345 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]=0 v18) ==== 105+0+0 (secure 0 0 0) 0x7fe70009cfc0 con 0x7fe714104f30 2024-04-02T14:32:25.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:25.906+0000 7fe71bc00640 1 -- 172.21.15.90:0/3479247345 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fe6fc0a3bb0 msgr2=0x7fe6fc0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:32:25.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:25.906+0000 7fe71bc00640 1 --2- 172.21.15.90:0/3479247345 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fe6fc0a3bb0 0x7fe6fc0a6070 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fe714104d90 tx=0x7fe708009260 comp rx=0 tx=0).stop 2024-04-02T14:32:25.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:25.906+0000 7fe71bc00640 1 -- 172.21.15.90:0/3479247345 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe714104f30 msgr2=0x7fe71419a550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:32:25.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:25.906+0000 7fe71bc00640 1 --2- 172.21.15.90:0/3479247345 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe714104f30 0x7fe71419a550 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fe70000e9f0 tx=0x7fe70000edf0 comp rx=0 tx=0).stop 2024-04-02T14:32:25.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:25.906+0000 7fe71bc00640 1 -- 172.21.15.90:0/3479247345 shutdown_connections 2024-04-02T14:32:25.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:25.906+0000 7fe71bc00640 1 --2- 172.21.15.90:0/3479247345 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fe6fc0a3bb0 0x7fe6fc0a6070 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:25.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:25.906+0000 7fe71bc00640 1 --2- 172.21.15.90:0/3479247345 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe7141058f0 0x7fe7141a15c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:25.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:25.906+0000 7fe71bc00640 1 --2- 172.21.15.90:0/3479247345 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe714104f30 0x7fe71419a550 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:25.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:25.906+0000 7fe71bc00640 1 --2- 172.21.15.90:0/3479247345 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe714103d40 0x7fe71419a010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:25.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:25.906+0000 7fe71bc00640 1 -- 172.21.15.90:0/3479247345 >> 172.21.15.90:0/3479247345 conn(0x7fe7140ff2b0 msgr2=0x7fe714108230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:32:25.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:25.906+0000 7fe71bc00640 1 -- 172.21.15.90:0/3479247345 shutdown_connections 2024-04-02T14:32:25.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:25.906+0000 7fe71bc00640 1 -- 172.21.15.90:0/3479247345 wait complete. 2024-04-02T14:32:25.934 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay true 2024-04-02T14:32:26.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:26.120+0000 7f078ee00640 1 -- 172.21.15.90:0/3983811368 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f07881058f0 msgr2=0x7f078810c180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:32:26.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:26.120+0000 7f078ee00640 1 --2- 172.21.15.90:0/3983811368 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f07881058f0 0x7f078810c180 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f0780009ca0 tx=0x7f078002c100 comp rx=0 tx=0).stop 2024-04-02T14:32:26.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:26.120+0000 7f078ee00640 1 -- 172.21.15.90:0/3983811368 shutdown_connections 2024-04-02T14:32:26.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:26.120+0000 7f078ee00640 1 --2- 172.21.15.90:0/3983811368 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f07881058f0 0x7f078810c180 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:26.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:26.120+0000 7f078ee00640 1 --2- 172.21.15.90:0/3983811368 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0788104f30 0x7f07881053b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:26.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:26.120+0000 7f078ee00640 1 --2- 172.21.15.90:0/3983811368 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0788103d40 0x7f0788104160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:26.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:26.120+0000 7f078ee00640 1 -- 172.21.15.90:0/3983811368 >> 172.21.15.90:0/3983811368 conn(0x7f07880ff2b0 msgr2=0x7f07881016f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:32:26.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:26.120+0000 7f078ee00640 1 -- 172.21.15.90:0/3983811368 shutdown_connections 2024-04-02T14:32:26.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:26.120+0000 7f078ee00640 1 -- 172.21.15.90:0/3983811368 wait complete. 2024-04-02T14:32:26.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:26.121+0000 7f078ee00640 1 Processor -- start 2024-04-02T14:32:26.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:26.121+0000 7f078ee00640 1 -- start start 2024-04-02T14:32:26.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:26.122+0000 7f078ee00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0788103d40 0x7f078819a000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:32:26.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:26.122+0000 7f078ee00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0788104f30 0x7f078819a540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:32:26.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:26.122+0000 7f078ee00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f07881058f0 0x7f07881a15b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:32:26.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:26.122+0000 7f078ee00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f078810f9a0 con 0x7f0788103d40 2024-04-02T14:32:26.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:26.122+0000 7f078ee00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f078810f820 con 0x7f07881058f0 2024-04-02T14:32:26.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:26.122+0000 7f078ee00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f078810fb20 con 0x7f0788104f30 2024-04-02T14:32:26.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:26.122+0000 7f0787e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0788103d40 0x7f078819a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:32:26.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:26.122+0000 7f0787e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0788103d40 0x7f078819a000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:49298/0 (socket says 172.21.15.90:49298) 2024-04-02T14:32:26.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:26.122+0000 7f0787e00640 1 -- 172.21.15.90:0/460876236 learned_addr learned my addr 172.21.15.90:0/460876236 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:32:26.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:26.122+0000 7f078ce00640 1 --2- 172.21.15.90:0/460876236 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f07881058f0 0x7f07881a15b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:32:26.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:26.122+0000 7f077fe00640 1 --2- 172.21.15.90:0/460876236 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0788104f30 0x7f078819a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:32:26.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:26.122+0000 7f0787e00640 1 -- 172.21.15.90:0/460876236 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0788104f30 msgr2=0x7f078819a540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:32:26.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:26.122+0000 7f0787e00640 1 --2- 172.21.15.90:0/460876236 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0788104f30 0x7f078819a540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:26.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:26.122+0000 7f0787e00640 1 -- 172.21.15.90:0/460876236 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f07881058f0 msgr2=0x7f07881a15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:32:26.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:26.122+0000 7f0787e00640 1 --2- 172.21.15.90:0/460876236 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f07881058f0 0x7f07881a15b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:26.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:26.122+0000 7f0787e00640 1 -- 172.21.15.90:0/460876236 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07881a1c20 con 0x7f0788103d40 2024-04-02T14:32:26.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:26.123+0000 7f0787e00640 1 --2- 172.21.15.90:0/460876236 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0788103d40 0x7f078819a000 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f077800e9c0 tx=0x7f077800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:32:26.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:26.123+0000 7f0785600640 1 -- 172.21.15.90:0/460876236 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f077800cd80 con 0x7f0788103d40 2024-04-02T14:32:26.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:26.123+0000 7f0785600640 1 -- 172.21.15.90:0/460876236 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f07780035f0 con 0x7f0788103d40 2024-04-02T14:32:26.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:26.123+0000 7f078ee00640 1 -- 172.21.15.90:0/460876236 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f07881a1f10 con 0x7f0788103d40 2024-04-02T14:32:26.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:26.123+0000 7f0785600640 1 -- 172.21.15.90:0/460876236 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0778010690 con 0x7f0788103d40 2024-04-02T14:32:26.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:26.123+0000 7f078ee00640 1 -- 172.21.15.90:0/460876236 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07881a23d0 con 0x7f0788103d40 2024-04-02T14:32:26.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:26.124+0000 7f078ee00640 1 -- 172.21.15.90:0/460876236 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f078810b000 con 0x7f0788103d40 2024-04-02T14:32:26.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:26.126+0000 7f0785600640 1 -- 172.21.15.90:0/460876236 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f07780037a0 con 0x7f0788103d40 2024-04-02T14:32:26.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:26.127+0000 7f0785600640 1 --2- 172.21.15.90:0/460876236 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f076c0a3c30 0x7f076c0a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:32:26.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:26.127+0000 7f0785600640 1 -- 172.21.15.90:0/460876236 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 7979+0+0 (secure 0 0 0) 0x7f07780d0640 con 0x7f0788103d40 2024-04-02T14:32:26.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:26.128+0000 7f077fe00640 1 --2- 172.21.15.90:0/460876236 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f076c0a3c30 0x7f076c0a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:32:26.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:26.128+0000 7f077fe00640 1 --2- 172.21.15.90:0/460876236 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f076c0a3c30 0x7f076c0a60f0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f078819b510 tx=0x7f077000b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:32:26.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:26.132+0000 7f0785600640 1 -- 172.21.15.90:0/460876236 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7f077809d0d0 con 0x7f0788103d40 2024-04-02T14:32:26.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:25 smithi177 ceph-mon[36431]: pgmap v313: 97 pgs: 18 creating+peering, 28 unknown, 51 active+clean; 579 KiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s wr, 2 op/s 2024-04-02T14:32:26.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:25 smithi177 ceph-mon[36431]: from='client.? 172.21.15.90:0/3764204241' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-04-02T14:32:26.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:25 smithi177 ceph-mon[36431]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-02T14:32:26.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:25 smithi177 ceph-mon[36431]: from='client.? 172.21.15.90:0/3479247345' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-04-02T14:32:26.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:25 smithi090 ceph-mon[31674]: pgmap v313: 97 pgs: 18 creating+peering, 28 unknown, 51 active+clean; 579 KiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s wr, 2 op/s 2024-04-02T14:32:26.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:25 smithi090 ceph-mon[31674]: from='client.? 172.21.15.90:0/3764204241' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-04-02T14:32:26.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:25 smithi090 ceph-mon[31674]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-02T14:32:26.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:25 smithi090 ceph-mon[31674]: from='client.? 172.21.15.90:0/3479247345' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-04-02T14:32:26.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:26.357+0000 7f078ee00640 1 -- 172.21.15.90:0/460876236 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) -- 0x7f078802d370 con 0x7f0788103d40 2024-04-02T14:32:26.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:25 smithi092 ceph-mon[36497]: pgmap v313: 97 pgs: 18 creating+peering, 28 unknown, 51 active+clean; 579 KiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s wr, 2 op/s 2024-04-02T14:32:26.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:25 smithi092 ceph-mon[36497]: from='client.? 172.21.15.90:0/3764204241' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-04-02T14:32:26.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:25 smithi092 ceph-mon[36497]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-02T14:32:26.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:25 smithi092 ceph-mon[36497]: from='client.? 172.21.15.90:0/3479247345' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-04-02T14:32:26.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:26.912+0000 7f0785600640 1 -- 172.21.15.90:0/460876236 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v20) ==== 121+0+0 (secure 0 0 0) 0x7f078802d370 con 0x7f0788103d40 2024-04-02T14:32:26.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:26.917+0000 7f078ee00640 1 -- 172.21.15.90:0/460876236 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f076c0a3c30 msgr2=0x7f076c0a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:32:26.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:26.917+0000 7f078ee00640 1 --2- 172.21.15.90:0/460876236 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f076c0a3c30 0x7f076c0a60f0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f078819b510 tx=0x7f077000b390 comp rx=0 tx=0).stop 2024-04-02T14:32:26.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:26.917+0000 7f078ee00640 1 -- 172.21.15.90:0/460876236 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0788103d40 msgr2=0x7f078819a000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:32:26.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:26.917+0000 7f078ee00640 1 --2- 172.21.15.90:0/460876236 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0788103d40 0x7f078819a000 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f077800e9c0 tx=0x7f077800ee80 comp rx=0 tx=0).stop 2024-04-02T14:32:26.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:26.917+0000 7f078ee00640 1 -- 172.21.15.90:0/460876236 shutdown_connections 2024-04-02T14:32:26.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:26.917+0000 7f078ee00640 1 --2- 172.21.15.90:0/460876236 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f076c0a3c30 0x7f076c0a60f0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:26.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:26.917+0000 7f078ee00640 1 --2- 172.21.15.90:0/460876236 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f07881058f0 0x7f07881a15b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:26.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:26.917+0000 7f078ee00640 1 --2- 172.21.15.90:0/460876236 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0788104f30 0x7f078819a540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:26.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:26.917+0000 7f078ee00640 1 --2- 172.21.15.90:0/460876236 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0788103d40 0x7f078819a000 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:26.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:26.917+0000 7f078ee00640 1 -- 172.21.15.90:0/460876236 >> 172.21.15.90:0/460876236 conn(0x7f07880ff2b0 msgr2=0x7f07881081d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:32:26.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:26.918+0000 7f078ee00640 1 -- 172.21.15.90:0/460876236 shutdown_connections 2024-04-02T14:32:26.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:26.918+0000 7f078ee00640 1 -- 172.21.15.90:0/460876236 wait complete. 2024-04-02T14:32:26.945 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2024-04-02T14:32:27.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:27.132+0000 7f2b8f200640 1 -- 172.21.15.90:0/3283431491 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2b88109310 msgr2=0x7f2b88109790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:32:27.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:27.132+0000 7f2b8f200640 1 --2- 172.21.15.90:0/3283431491 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2b88109310 0x7f2b88109790 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f2b70009900 tx=0x7f2b7002bc30 comp rx=0 tx=0).stop 2024-04-02T14:32:27.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:27.132+0000 7f2b8f200640 1 -- 172.21.15.90:0/3283431491 shutdown_connections 2024-04-02T14:32:27.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:27.132+0000 7f2b8f200640 1 --2- 172.21.15.90:0/3283431491 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2b88109cd0 0x7f2b88110560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:27.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:27.132+0000 7f2b8f200640 1 --2- 172.21.15.90:0/3283431491 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2b88109310 0x7f2b88109790 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:27.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:27.132+0000 7f2b8f200640 1 --2- 172.21.15.90:0/3283431491 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2b88108120 0x7f2b88108540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:27.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:27.132+0000 7f2b8f200640 1 -- 172.21.15.90:0/3283431491 >> 172.21.15.90:0/3283431491 conn(0x7f2b880770c0 msgr2=0x7f2b88079500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:32:27.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:27.132+0000 7f2b8f200640 1 -- 172.21.15.90:0/3283431491 shutdown_connections 2024-04-02T14:32:27.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:27.132+0000 7f2b8f200640 1 -- 172.21.15.90:0/3283431491 wait complete. 2024-04-02T14:32:27.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:27.133+0000 7f2b8f200640 1 Processor -- start 2024-04-02T14:32:27.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:27.133+0000 7f2b8f200640 1 -- start start 2024-04-02T14:32:27.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:27.134+0000 7f2b8f200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2b88108120 0x7f2b8819e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:32:27.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:27.134+0000 7f2b8f200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2b88109310 0x7f2b8819e950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:32:27.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:27.134+0000 7f2b8f200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2b88109cd0 0x7f2b881a59c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:32:27.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:27.134+0000 7f2b8f200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f2b88113db0 con 0x7f2b88109cd0 2024-04-02T14:32:27.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:27.134+0000 7f2b8f200640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f2b88113c30 con 0x7f2b88109310 2024-04-02T14:32:27.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:27.134+0000 7f2b8f200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f2b88113f30 con 0x7f2b88108120 2024-04-02T14:32:27.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:27.134+0000 7f2b87400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2b88109310 0x7f2b8819e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:32:27.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:27.134+0000 7f2b87e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2b88108120 0x7f2b8819e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:32:27.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:27.134+0000 7f2b87400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2b88109310 0x7f2b8819e950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:44922/0 (socket says 172.21.15.90:44922) 2024-04-02T14:32:27.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:27.134+0000 7f2b87400640 1 -- 172.21.15.90:0/2996936658 learned_addr learned my addr 172.21.15.90:0/2996936658 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:32:27.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:27.134+0000 7f2b8d200640 1 --2- 172.21.15.90:0/2996936658 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2b88109cd0 0x7f2b881a59c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:32:27.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:27.134+0000 7f2b87400640 1 -- 172.21.15.90:0/2996936658 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2b88108120 msgr2=0x7f2b8819e410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:32:27.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:27.134+0000 7f2b87400640 1 --2- 172.21.15.90:0/2996936658 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2b88108120 0x7f2b8819e410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:27.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:27.134+0000 7f2b87400640 1 -- 172.21.15.90:0/2996936658 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2b88109cd0 msgr2=0x7f2b881a59c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:32:27.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:27.134+0000 7f2b87400640 1 --2- 172.21.15.90:0/2996936658 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2b88109cd0 0x7f2b881a59c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:27.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:27.134+0000 7f2b87400640 1 -- 172.21.15.90:0/2996936658 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b881a6030 con 0x7f2b88109310 2024-04-02T14:32:27.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:27.135+0000 7f2b87400640 1 --2- 172.21.15.90:0/2996936658 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2b88109310 0x7f2b8819e950 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f2b70009230 tx=0x7f2b7002c650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:32:27.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:27.135+0000 7f2b84c00640 1 -- 172.21.15.90:0/2996936658 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b70041070 con 0x7f2b88109310 2024-04-02T14:32:27.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:27.135+0000 7f2b84c00640 1 -- 172.21.15.90:0/2996936658 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2b70004450 con 0x7f2b88109310 2024-04-02T14:32:27.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:27.135+0000 7f2b8f200640 1 -- 172.21.15.90:0/2996936658 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b881a62c0 con 0x7f2b88109310 2024-04-02T14:32:27.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:27.135+0000 7f2b84c00640 1 -- 172.21.15.90:0/2996936658 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b70003e20 con 0x7f2b88109310 2024-04-02T14:32:27.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:27.135+0000 7f2b8f200640 1 -- 172.21.15.90:0/2996936658 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b881a6800 con 0x7f2b88109310 2024-04-02T14:32:27.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:27.136+0000 7f2b8f200640 1 -- 172.21.15.90:0/2996936658 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2b8810f440 con 0x7f2b88109310 2024-04-02T14:32:27.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:27.138+0000 7f2b84c00640 1 -- 172.21.15.90:0/2996936658 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f2b70044070 con 0x7f2b88109310 2024-04-02T14:32:27.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:27.139+0000 7f2b84c00640 1 --2- 172.21.15.90:0/2996936658 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f2b6c0a3c30 0x7f2b6c0a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:32:27.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:27.140+0000 7f2b84c00640 1 -- 172.21.15.90:0/2996936658 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 7979+0+0 (secure 0 0 0) 0x7f2b700f1f90 con 0x7f2b88109310 2024-04-02T14:32:27.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:27.140+0000 7f2b87e00640 1 --2- 172.21.15.90:0/2996936658 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f2b6c0a3c30 0x7f2b6c0a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:32:27.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:27.140+0000 7f2b87e00640 1 --2- 172.21.15.90:0/2996936658 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f2b6c0a3c30 0x7f2b6c0a60f0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f2b88109170 tx=0x7f2b78005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:32:27.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:27.142+0000 7f2b84c00640 1 -- 172.21.15.90:0/2996936658 <== mon.2 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7f2b700be3d0 con 0x7f2b88109310 2024-04-02T14:32:27.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:26 smithi177 ceph-mon[36431]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-02T14:32:27.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:26 smithi177 ceph-mon[36431]: from='client.? 172.21.15.90:0/3479247345' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-04-02T14:32:27.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:26 smithi177 ceph-mon[36431]: daemon mds.g assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-04-02T14:32:27.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:26 smithi177 ceph-mon[36431]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-02T14:32:27.188 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:26 smithi177 ceph-mon[36431]: fsmap cephfs:2 {0=b=up:active,1=g=up:creating} 10 up:standby 2024-04-02T14:32:27.188 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:26 smithi177 ceph-mon[36431]: daemon mds.g is now active in filesystem cephfs as rank 1 2024-04-02T14:32:27.188 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:26 smithi177 ceph-mon[36431]: from='client.? 172.21.15.90:0/460876236' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-02T14:32:27.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:26 smithi090 ceph-mon[31674]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-02T14:32:27.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:26 smithi090 ceph-mon[31674]: from='client.? 172.21.15.90:0/3479247345' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-04-02T14:32:27.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:26 smithi090 ceph-mon[31674]: daemon mds.g assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-04-02T14:32:27.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:26 smithi090 ceph-mon[31674]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-02T14:32:27.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:26 smithi090 ceph-mon[31674]: fsmap cephfs:2 {0=b=up:active,1=g=up:creating} 10 up:standby 2024-04-02T14:32:27.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:26 smithi090 ceph-mon[31674]: daemon mds.g is now active in filesystem cephfs as rank 1 2024-04-02T14:32:27.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:26 smithi090 ceph-mon[31674]: from='client.? 172.21.15.90:0/460876236' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-02T14:32:27.262 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:26 smithi092 ceph-mon[36497]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-02T14:32:27.263 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:26 smithi092 ceph-mon[36497]: from='client.? 172.21.15.90:0/3479247345' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-04-02T14:32:27.263 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:26 smithi092 ceph-mon[36497]: daemon mds.g assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-04-02T14:32:27.263 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:26 smithi092 ceph-mon[36497]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-02T14:32:27.264 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:26 smithi092 ceph-mon[36497]: fsmap cephfs:2 {0=b=up:active,1=g=up:creating} 10 up:standby 2024-04-02T14:32:27.264 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:26 smithi092 ceph-mon[36497]: daemon mds.g is now active in filesystem cephfs as rank 1 2024-04-02T14:32:27.264 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:26 smithi092 ceph-mon[36497]: from='client.? 172.21.15.90:0/460876236' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-02T14:32:27.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:27.380+0000 7f2b8f200640 1 -- 172.21.15.90:0/2996936658 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) -- 0x7f2b8802d370 con 0x7f2b88109310 2024-04-02T14:32:27.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:27.928+0000 7f2b84c00640 1 -- 172.21.15.90:0/2996936658 <== mon.2 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v22) ==== 115+0+0 (secure 0 0 0) 0x7f2b700c26d0 con 0x7f2b88109310 2024-04-02T14:32:27.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:27.931+0000 7f2b8f200640 1 -- 172.21.15.90:0/2996936658 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f2b6c0a3c30 msgr2=0x7f2b6c0a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:32:27.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:27.931+0000 7f2b8f200640 1 --2- 172.21.15.90:0/2996936658 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f2b6c0a3c30 0x7f2b6c0a60f0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f2b88109170 tx=0x7f2b78005f90 comp rx=0 tx=0).stop 2024-04-02T14:32:27.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:27.931+0000 7f2b8f200640 1 -- 172.21.15.90:0/2996936658 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2b88109310 msgr2=0x7f2b8819e950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:32:27.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:27.931+0000 7f2b8f200640 1 --2- 172.21.15.90:0/2996936658 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2b88109310 0x7f2b8819e950 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f2b70009230 tx=0x7f2b7002c650 comp rx=0 tx=0).stop 2024-04-02T14:32:27.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:27.932+0000 7f2b8f200640 1 -- 172.21.15.90:0/2996936658 shutdown_connections 2024-04-02T14:32:27.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:27.932+0000 7f2b8f200640 1 --2- 172.21.15.90:0/2996936658 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f2b6c0a3c30 0x7f2b6c0a60f0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:27.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:27.932+0000 7f2b8f200640 1 --2- 172.21.15.90:0/2996936658 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2b88109cd0 0x7f2b881a59c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:27.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:27.932+0000 7f2b8f200640 1 --2- 172.21.15.90:0/2996936658 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2b88109310 0x7f2b8819e950 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:27.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:27.932+0000 7f2b8f200640 1 --2- 172.21.15.90:0/2996936658 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2b88108120 0x7f2b8819e410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:27.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:27.932+0000 7f2b8f200640 1 -- 172.21.15.90:0/2996936658 >> 172.21.15.90:0/2996936658 conn(0x7f2b880770c0 msgr2=0x7f2b8810c610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:32:27.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:27.932+0000 7f2b8f200640 1 -- 172.21.15.90:0/2996936658 shutdown_connections 2024-04-02T14:32:27.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:27.932+0000 7f2b8f200640 1 -- 172.21.15.90:0/2996936658 wait complete. 2024-04-02T14:32:27.961 DEBUG:tasks.cephfs.filesystem:Creating {'create': 2, 'subvol_options': '--namespace-isolated'} subvols for filesystem 'cephfs' 2024-04-02T14:32:27.961 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolumegroup create cephfs qa 2024-04-02T14:32:28.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:28.147+0000 7fe825000640 1 -- 172.21.15.90:0/358682301 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe820109210 msgr2=0x7fe820109690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:32:28.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:28.147+0000 7fe825000640 1 --2- 172.21.15.90:0/358682301 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe820109210 0x7fe820109690 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fe80c009900 tx=0x7fe80c02bc30 comp rx=0 tx=0).stop 2024-04-02T14:32:28.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:28.147+0000 7fe825000640 1 -- 172.21.15.90:0/358682301 shutdown_connections 2024-04-02T14:32:28.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:28.147+0000 7fe825000640 1 --2- 172.21.15.90:0/358682301 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe820109bd0 0x7fe820110460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:28.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:28.147+0000 7fe825000640 1 --2- 172.21.15.90:0/358682301 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe820109210 0x7fe820109690 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:28.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:28.147+0000 7fe825000640 1 --2- 172.21.15.90:0/358682301 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe820108020 0x7fe820108440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:28.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:28.147+0000 7fe825000640 1 -- 172.21.15.90:0/358682301 >> 172.21.15.90:0/358682301 conn(0x7fe820076fd0 msgr2=0x7fe8200793f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:32:28.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:28.147+0000 7fe825000640 1 -- 172.21.15.90:0/358682301 shutdown_connections 2024-04-02T14:32:28.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:28.147+0000 7fe825000640 1 -- 172.21.15.90:0/358682301 wait complete. 2024-04-02T14:32:28.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:28.148+0000 7fe825000640 1 Processor -- start 2024-04-02T14:32:28.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:28.148+0000 7fe825000640 1 -- start start 2024-04-02T14:32:28.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:28.149+0000 7fe825000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe820108020 0x7fe82019e2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:32:28.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:28.149+0000 7fe825000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe820109210 0x7fe82019e810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:32:28.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:28.149+0000 7fe825000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe820109bd0 0x7fe8201a5880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:32:28.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:28.149+0000 7fe825000640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fe820113d00 con 0x7fe820109bd0 2024-04-02T14:32:28.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:28.149+0000 7fe825000640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fe820113b80 con 0x7fe820109210 2024-04-02T14:32:28.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:28.149+0000 7fe825000640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fe820113e80 con 0x7fe820108020 2024-04-02T14:32:28.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:28.149+0000 7fe81ea00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe820109210 0x7fe82019e810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:32:28.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:28.149+0000 7fe81ea00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe820109210 0x7fe82019e810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:44940/0 (socket says 172.21.15.90:44940) 2024-04-02T14:32:28.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:28.149+0000 7fe81ea00640 1 -- 172.21.15.90:0/2365561146 learned_addr learned my addr 172.21.15.90:0/2365561146 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:32:28.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:28.149+0000 7fe81fe00640 1 --2- 172.21.15.90:0/2365561146 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe820109bd0 0x7fe8201a5880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:32:28.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:28.149+0000 7fe81f400640 1 --2- 172.21.15.90:0/2365561146 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe820108020 0x7fe82019e2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:32:28.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:28.149+0000 7fe81ea00640 1 -- 172.21.15.90:0/2365561146 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe820108020 msgr2=0x7fe82019e2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:32:28.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:28.149+0000 7fe81ea00640 1 --2- 172.21.15.90:0/2365561146 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe820108020 0x7fe82019e2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:28.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:28.149+0000 7fe81ea00640 1 -- 172.21.15.90:0/2365561146 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe820109bd0 msgr2=0x7fe8201a5880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:32:28.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:28.149+0000 7fe81ea00640 1 --2- 172.21.15.90:0/2365561146 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe820109bd0 0x7fe8201a5880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:28.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:28.149+0000 7fe81ea00640 1 -- 172.21.15.90:0/2365561146 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8201a5ef0 con 0x7fe820109210 2024-04-02T14:32:28.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:28.149+0000 7fe81f400640 1 --2- 172.21.15.90:0/2365561146 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe820108020 0x7fe82019e2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:32:28.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:28.150+0000 7fe81ea00640 1 --2- 172.21.15.90:0/2365561146 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe820109210 0x7fe82019e810 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fe80c000c00 tx=0x7fe80c02c650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:32:28.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:28.150+0000 7fe817e00640 1 -- 172.21.15.90:0/2365561146 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe80c043020 con 0x7fe820109210 2024-04-02T14:32:28.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:28.150+0000 7fe817e00640 1 -- 172.21.15.90:0/2365561146 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe80c0054e0 con 0x7fe820109210 2024-04-02T14:32:28.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:28.150+0000 7fe817e00640 1 -- 172.21.15.90:0/2365561146 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe80c003bd0 con 0x7fe820109210 2024-04-02T14:32:28.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:28.150+0000 7fe825000640 1 -- 172.21.15.90:0/2365561146 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe8201a6180 con 0x7fe820109210 2024-04-02T14:32:28.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:28.150+0000 7fe825000640 1 -- 172.21.15.90:0/2365561146 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8201a6680 con 0x7fe820109210 2024-04-02T14:32:28.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:28.151+0000 7fe825000640 1 -- 172.21.15.90:0/2365561146 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe82010f2e0 con 0x7fe820109210 2024-04-02T14:32:28.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:28.153+0000 7fe817e00640 1 -- 172.21.15.90:0/2365561146 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fe80c046020 con 0x7fe820109210 2024-04-02T14:32:28.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:28.154+0000 7fe817e00640 1 --2- 172.21.15.90:0/2365561146 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fe8040a3c30 0x7fe8040a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:32:28.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:28.155+0000 7fe817e00640 1 -- 172.21.15.90:0/2365561146 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 7979+0+0 (secure 0 0 0) 0x7fe80c0f2760 con 0x7fe820109210 2024-04-02T14:32:28.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:28.155+0000 7fe81f400640 1 --2- 172.21.15.90:0/2365561146 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fe8040a3c30 0x7fe8040a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:32:28.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:28.155+0000 7fe81f400640 1 --2- 172.21.15.90:0/2365561146 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fe8040a3c30 0x7fe8040a60f0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fe820109070 tx=0x7fe810005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:32:28.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:28.157+0000 7fe817e00640 1 -- 172.21.15.90:0/2365561146 <== mon.2 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7fe80c04d020 con 0x7fe820109210 2024-04-02T14:32:28.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:27 smithi177 ceph-mon[36431]: pgmap v314: 97 pgs: 6 creating+peering, 91 active+clean; 579 KiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s wr, 8 op/s 2024-04-02T14:32:28.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:27 smithi177 ceph-mon[36431]: from='client.? 172.21.15.90:0/460876236' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-04-02T14:32:28.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:27 smithi177 ceph-mon[36431]: mds.? [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] up:active 2024-04-02T14:32:28.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:27 smithi177 ceph-mon[36431]: daemon mds.i assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-04-02T14:32:28.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:27 smithi177 ceph-mon[36431]: fsmap cephfs:2 {0=b=up:active,1=g=up:active} 10 up:standby 2024-04-02T14:32:28.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:27 smithi177 ceph-mon[36431]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=i=up:creating} 9 up:standby 2024-04-02T14:32:28.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:27 smithi177 ceph-mon[36431]: daemon mds.i is now active in filesystem cephfs as rank 2 2024-04-02T14:32:28.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:27 smithi177 ceph-mon[36431]: from='client.? 172.21.15.90:0/2996936658' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-02T14:32:28.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:27 smithi177 ceph-mon[36431]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-02T14:32:28.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:27 smithi090 ceph-mon[31674]: pgmap v314: 97 pgs: 6 creating+peering, 91 active+clean; 579 KiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s wr, 8 op/s 2024-04-02T14:32:28.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:27 smithi090 ceph-mon[31674]: from='client.? 172.21.15.90:0/460876236' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-04-02T14:32:28.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:27 smithi090 ceph-mon[31674]: mds.? [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] up:active 2024-04-02T14:32:28.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:27 smithi090 ceph-mon[31674]: daemon mds.i assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-04-02T14:32:28.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:27 smithi090 ceph-mon[31674]: fsmap cephfs:2 {0=b=up:active,1=g=up:active} 10 up:standby 2024-04-02T14:32:28.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:27 smithi090 ceph-mon[31674]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=i=up:creating} 9 up:standby 2024-04-02T14:32:28.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:27 smithi090 ceph-mon[31674]: daemon mds.i is now active in filesystem cephfs as rank 2 2024-04-02T14:32:28.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:27 smithi090 ceph-mon[31674]: from='client.? 172.21.15.90:0/2996936658' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-02T14:32:28.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:27 smithi090 ceph-mon[31674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-02T14:32:28.360 INFO:journalctl@ceph.mds.b.smithi092.stdout:Apr 02 14:32:28 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mds-b[67422]: 2024-04-02T14:32:28.121+0000 7f208e2f9700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-02T14:32:28.361 INFO:journalctl@ceph.mds.b.smithi092.stdout:Apr 02 14:32:28 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mds-b[67422]: 2024-04-02T14:32:28.121+0000 7f208e2f9700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-04-02T14:32:28.361 INFO:journalctl@ceph.mds.b.smithi092.stdout:Apr 02 14:32:28 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mds-b[67422]: 2024-04-02T14:32:28.121+0000 7f208e2f9700 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-04-02T14:32:28.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:27 smithi092 ceph-mon[36497]: pgmap v314: 97 pgs: 6 creating+peering, 91 active+clean; 579 KiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s wr, 8 op/s 2024-04-02T14:32:28.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:27 smithi092 ceph-mon[36497]: from='client.? 172.21.15.90:0/460876236' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-04-02T14:32:28.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:27 smithi092 ceph-mon[36497]: mds.? [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] up:active 2024-04-02T14:32:28.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:27 smithi092 ceph-mon[36497]: daemon mds.i assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-04-02T14:32:28.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:27 smithi092 ceph-mon[36497]: fsmap cephfs:2 {0=b=up:active,1=g=up:active} 10 up:standby 2024-04-02T14:32:28.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:27 smithi092 ceph-mon[36497]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=i=up:creating} 9 up:standby 2024-04-02T14:32:28.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:27 smithi092 ceph-mon[36497]: daemon mds.i is now active in filesystem cephfs as rank 2 2024-04-02T14:32:28.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:27 smithi092 ceph-mon[36497]: from='client.? 172.21.15.90:0/2996936658' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-02T14:32:28.362 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:27 smithi092 ceph-mon[36497]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-02T14:32:28.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:28.370+0000 7fe825000640 1 -- 172.21.15.90:0/2365561146 --> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7fe820061f40 con 0x7fe8040a3c30 2024-04-02T14:32:28.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:28.405+0000 7fe817e00640 1 -- 172.21.15.90:0/2365561146 <== mgr.14152 v2:172.21.15.90:6800/4266054817 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fe820061f40 con 0x7fe8040a3c30 2024-04-02T14:32:28.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:28.407+0000 7fe825000640 1 -- 172.21.15.90:0/2365561146 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fe8040a3c30 msgr2=0x7fe8040a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:32:28.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:28.407+0000 7fe825000640 1 --2- 172.21.15.90:0/2365561146 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fe8040a3c30 0x7fe8040a60f0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fe820109070 tx=0x7fe810005f90 comp rx=0 tx=0).stop 2024-04-02T14:32:28.410 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:28.407+0000 7fe825000640 1 -- 172.21.15.90:0/2365561146 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe820109210 msgr2=0x7fe82019e810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:32:28.410 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:28.407+0000 7fe825000640 1 --2- 172.21.15.90:0/2365561146 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe820109210 0x7fe82019e810 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fe80c000c00 tx=0x7fe80c02c650 comp rx=0 tx=0).stop 2024-04-02T14:32:28.410 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:28.407+0000 7fe825000640 1 -- 172.21.15.90:0/2365561146 shutdown_connections 2024-04-02T14:32:28.410 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:28.407+0000 7fe825000640 1 --2- 172.21.15.90:0/2365561146 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fe8040a3c30 0x7fe8040a60f0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:28.410 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:28.407+0000 7fe825000640 1 --2- 172.21.15.90:0/2365561146 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe820109bd0 0x7fe8201a5880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:28.410 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:28.407+0000 7fe825000640 1 --2- 172.21.15.90:0/2365561146 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe820109210 0x7fe82019e810 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:28.410 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:28.407+0000 7fe825000640 1 --2- 172.21.15.90:0/2365561146 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe820108020 0x7fe82019e2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:28.410 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:28.407+0000 7fe825000640 1 -- 172.21.15.90:0/2365561146 >> 172.21.15.90:0/2365561146 conn(0x7fe820076fd0 msgr2=0x7fe82010c4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:32:28.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:28.407+0000 7fe825000640 1 -- 172.21.15.90:0/2365561146 shutdown_connections 2024-04-02T14:32:28.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:28.407+0000 7fe825000640 1 -- 172.21.15.90:0/2365561146 wait complete. 2024-04-02T14:32:28.426 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_0 --group_name qa 2024-04-02T14:32:28.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:28.619+0000 7fc2bbe00640 1 -- 172.21.15.90:0/1914204367 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc2b4104f30 msgr2=0x7fc2b41053b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:32:28.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:28.619+0000 7fc2bbe00640 1 --2- 172.21.15.90:0/1914204367 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc2b4104f30 0x7fc2b41053b0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fc2a8009900 tx=0x7fc2a802bc50 comp rx=0 tx=0).stop 2024-04-02T14:32:28.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:28.619+0000 7fc2bbe00640 1 -- 172.21.15.90:0/1914204367 shutdown_connections 2024-04-02T14:32:28.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:28.619+0000 7fc2bbe00640 1 --2- 172.21.15.90:0/1914204367 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc2b41058f0 0x7fc2b410c180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:28.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:28.619+0000 7fc2bbe00640 1 --2- 172.21.15.90:0/1914204367 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc2b4104f30 0x7fc2b41053b0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:28.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:28.619+0000 7fc2bbe00640 1 --2- 172.21.15.90:0/1914204367 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc2b4103d40 0x7fc2b4104160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:28.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:28.619+0000 7fc2bbe00640 1 -- 172.21.15.90:0/1914204367 >> 172.21.15.90:0/1914204367 conn(0x7fc2b40ff2b0 msgr2=0x7fc2b41016f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:32:28.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:28.619+0000 7fc2bbe00640 1 -- 172.21.15.90:0/1914204367 shutdown_connections 2024-04-02T14:32:28.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:28.619+0000 7fc2bbe00640 1 -- 172.21.15.90:0/1914204367 wait complete. 2024-04-02T14:32:28.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:28.620+0000 7fc2bbe00640 1 Processor -- start 2024-04-02T14:32:28.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:28.620+0000 7fc2bbe00640 1 -- start start 2024-04-02T14:32:28.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:28.620+0000 7fc2bbe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc2b4103d40 0x7fc2b419a060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:32:28.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:28.620+0000 7fc2bbe00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc2b4104f30 0x7fc2b419a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:32:28.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:28.620+0000 7fc2bbe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc2b41058f0 0x7fc2b41a1610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:32:28.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:28.620+0000 7fc2bbe00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fc2b410fa10 con 0x7fc2b41058f0 2024-04-02T14:32:28.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:28.620+0000 7fc2bbe00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fc2b410f890 con 0x7fc2b4104f30 2024-04-02T14:32:28.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:28.620+0000 7fc2bbe00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fc2b410fb90 con 0x7fc2b4103d40 2024-04-02T14:32:28.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:28.620+0000 7fc2b9e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc2b41058f0 0x7fc2b41a1610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:32:28.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:28.620+0000 7fc2b8a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc2b4104f30 0x7fc2b419a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:32:28.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:28.620+0000 7fc2b9400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc2b4103d40 0x7fc2b419a060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:32:28.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:28.620+0000 7fc2b8a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc2b4104f30 0x7fc2b419a5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:44982/0 (socket says 172.21.15.90:44982) 2024-04-02T14:32:28.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:28.620+0000 7fc2b9400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc2b4103d40 0x7fc2b419a060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:35436/0 (socket says 172.21.15.90:35436) 2024-04-02T14:32:28.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:28.620+0000 7fc2b8a00640 1 -- 172.21.15.90:0/300851293 learned_addr learned my addr 172.21.15.90:0/300851293 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:32:28.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:28.621+0000 7fc2b8a00640 1 -- 172.21.15.90:0/300851293 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc2b4103d40 msgr2=0x7fc2b419a060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:32:28.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:28.621+0000 7fc2b8a00640 1 --2- 172.21.15.90:0/300851293 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc2b4103d40 0x7fc2b419a060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:28.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:28.621+0000 7fc2b8a00640 1 -- 172.21.15.90:0/300851293 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc2b41058f0 msgr2=0x7fc2b41a1610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:32:28.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:28.621+0000 7fc2b8a00640 1 --2- 172.21.15.90:0/300851293 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc2b41058f0 0x7fc2b41a1610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:28.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:28.621+0000 7fc2b8a00640 1 -- 172.21.15.90:0/300851293 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2b41a1c80 con 0x7fc2b4104f30 2024-04-02T14:32:28.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:28.621+0000 7fc2b9e00640 1 --2- 172.21.15.90:0/300851293 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc2b41058f0 0x7fc2b41a1610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:32:28.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:28.621+0000 7fc2b8a00640 1 --2- 172.21.15.90:0/300851293 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc2b4104f30 0x7fc2b419a5a0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fc2a8005ab0 tx=0x7fc2a802c6e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:32:28.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:28.621+0000 7fc2b2000640 1 -- 172.21.15.90:0/300851293 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2a8041070 con 0x7fc2b4104f30 2024-04-02T14:32:28.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:28.621+0000 7fc2b2000640 1 -- 172.21.15.90:0/300851293 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc2a8004450 con 0x7fc2b4104f30 2024-04-02T14:32:28.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:28.621+0000 7fc2bbe00640 1 -- 172.21.15.90:0/300851293 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2b41a1f10 con 0x7fc2b4104f30 2024-04-02T14:32:28.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:28.621+0000 7fc2b2000640 1 -- 172.21.15.90:0/300851293 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2a8034440 con 0x7fc2b4104f30 2024-04-02T14:32:28.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:28.621+0000 7fc2bbe00640 1 -- 172.21.15.90:0/300851293 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2b41a23d0 con 0x7fc2b4104f30 2024-04-02T14:32:28.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:28.622+0000 7fc2bbe00640 1 -- 172.21.15.90:0/300851293 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc2b410b060 con 0x7fc2b4104f30 2024-04-02T14:32:28.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:28.625+0000 7fc2b2000640 1 -- 172.21.15.90:0/300851293 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fc2a8044070 con 0x7fc2b4104f30 2024-04-02T14:32:28.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:28.626+0000 7fc2b2000640 1 --2- 172.21.15.90:0/300851293 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fc29c0a3bb0 0x7fc29c0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:32:28.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:28.626+0000 7fc2b2000640 1 -- 172.21.15.90:0/300851293 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 7979+0+0 (secure 0 0 0) 0x7fc2a80f2210 con 0x7fc2b4104f30 2024-04-02T14:32:28.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:28.626+0000 7fc2b9400640 1 --2- 172.21.15.90:0/300851293 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fc29c0a3bb0 0x7fc29c0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:32:28.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:28.627+0000 7fc2b9400640 1 --2- 172.21.15.90:0/300851293 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fc29c0a3bb0 0x7fc29c0a6070 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fc2b4104d90 tx=0x7fc2a4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:32:28.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:28.628+0000 7fc2b2000640 1 -- 172.21.15.90:0/300851293 <== mon.2 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7fc2a80be650 con 0x7fc2b4104f30 2024-04-02T14:32:28.689 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 02 14:32:28 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-x[31910]: 2024-04-02T14:32:28.380+0000 7fd4eb800700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T14:32:28.689 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 02 14:32:28 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-x[31910]: 2024-04-02T14:32:28.380+0000 7fd4eb800700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T14:32:28.689 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 02 14:32:28 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-x[31910]: 2024-04-02T14:32:28.380+0000 7fd4eb800700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T14:32:28.690 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 02 14:32:28 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-x[31910]: 2024-04-02T14:32:28.380+0000 7fd4eb800700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T14:32:28.690 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 02 14:32:28 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-x[31910]: 2024-04-02T14:32:28.380+0000 7fd4eb800700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T14:32:28.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:28.855+0000 7fc2bbe00640 1 -- 172.21.15.90:0/300851293 --> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7fc2b4061f40 con 0x7fc29c0a3bb0 2024-04-02T14:32:28.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:28.896+0000 7fc2b2000640 1 -- 172.21.15.90:0/300851293 <== mgr.14152 v2:172.21.15.90:6800/4266054817 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fc2b4061f40 con 0x7fc29c0a3bb0 2024-04-02T14:32:28.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:28.899+0000 7fc2bbe00640 1 -- 172.21.15.90:0/300851293 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fc29c0a3bb0 msgr2=0x7fc29c0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:32:28.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:28.899+0000 7fc2bbe00640 1 --2- 172.21.15.90:0/300851293 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fc29c0a3bb0 0x7fc29c0a6070 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fc2b4104d90 tx=0x7fc2a4005f90 comp rx=0 tx=0).stop 2024-04-02T14:32:28.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:28.899+0000 7fc2bbe00640 1 -- 172.21.15.90:0/300851293 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc2b4104f30 msgr2=0x7fc2b419a5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:32:28.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:28.899+0000 7fc2bbe00640 1 --2- 172.21.15.90:0/300851293 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc2b4104f30 0x7fc2b419a5a0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fc2a8005ab0 tx=0x7fc2a802c6e0 comp rx=0 tx=0).stop 2024-04-02T14:32:28.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:28.899+0000 7fc2bbe00640 1 -- 172.21.15.90:0/300851293 shutdown_connections 2024-04-02T14:32:28.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:28.899+0000 7fc2bbe00640 1 --2- 172.21.15.90:0/300851293 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fc29c0a3bb0 0x7fc29c0a6070 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:28.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:28.899+0000 7fc2bbe00640 1 --2- 172.21.15.90:0/300851293 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc2b41058f0 0x7fc2b41a1610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:28.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:28.899+0000 7fc2bbe00640 1 --2- 172.21.15.90:0/300851293 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc2b4104f30 0x7fc2b419a5a0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:28.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:28.899+0000 7fc2bbe00640 1 --2- 172.21.15.90:0/300851293 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc2b4103d40 0x7fc2b419a060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:28.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:28.899+0000 7fc2bbe00640 1 -- 172.21.15.90:0/300851293 >> 172.21.15.90:0/300851293 conn(0x7fc2b40ff2b0 msgr2=0x7fc2b4108230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:32:28.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:28.899+0000 7fc2bbe00640 1 -- 172.21.15.90:0/300851293 shutdown_connections 2024-04-02T14:32:28.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:28.899+0000 7fc2bbe00640 1 -- 172.21.15.90:0/300851293 wait complete. 2024-04-02T14:32:28.920 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0 2024-04-02T14:32:29.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:29.121+0000 7f5316400640 1 -- 172.21.15.90:0/433602214 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f53101056d0 msgr2=0x7f531010bf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:32:29.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:29.121+0000 7f5316400640 1 --2- 172.21.15.90:0/433602214 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f53101056d0 0x7f531010bf60 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f5308009ca0 tx=0x7f530802c100 comp rx=0 tx=0).stop 2024-04-02T14:32:29.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:29.121+0000 7f5316400640 1 -- 172.21.15.90:0/433602214 shutdown_connections 2024-04-02T14:32:29.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:29.121+0000 7f5316400640 1 --2- 172.21.15.90:0/433602214 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f53101056d0 0x7f531010bf60 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:29.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:29.121+0000 7f5316400640 1 --2- 172.21.15.90:0/433602214 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5310104d10 0x7f5310105190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:29.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:29.121+0000 7f5316400640 1 --2- 172.21.15.90:0/433602214 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5310103b20 0x7f5310103f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:29.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:29.121+0000 7f5316400640 1 -- 172.21.15.90:0/433602214 >> 172.21.15.90:0/433602214 conn(0x7f53100ff0d0 msgr2=0x7f53101014f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:32:29.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:29.121+0000 7f5316400640 1 -- 172.21.15.90:0/433602214 shutdown_connections 2024-04-02T14:32:29.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:29.121+0000 7f5316400640 1 -- 172.21.15.90:0/433602214 wait complete. 2024-04-02T14:32:29.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:29.122+0000 7f5316400640 1 Processor -- start 2024-04-02T14:32:29.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:29.122+0000 7f5316400640 1 -- start start 2024-04-02T14:32:29.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:29.123+0000 7f5316400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5310103b20 0x7f5310199df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:32:29.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:29.123+0000 7f5316400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5310104d10 0x7f531019a330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:32:29.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:29.123+0000 7f5316400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f53101056d0 0x7f53101a13a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:32:29.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:29.123+0000 7f5316400640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f531010f810 con 0x7f53101056d0 2024-04-02T14:32:29.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:29.123+0000 7f5316400640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f531010f690 con 0x7f5310103b20 2024-04-02T14:32:29.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:29.123+0000 7f5316400640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f531010f990 con 0x7f5310104d10 2024-04-02T14:32:29.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:29.123+0000 7f5315000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5310103b20 0x7f5310199df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:32:29.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:29.123+0000 7f5315000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5310103b20 0x7f5310199df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:44992/0 (socket says 172.21.15.90:44992) 2024-04-02T14:32:29.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:29.123+0000 7f5315000640 1 -- 172.21.15.90:0/1337980526 learned_addr learned my addr 172.21.15.90:0/1337980526 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:32:29.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:29.123+0000 7f5315a00640 1 --2- 172.21.15.90:0/1337980526 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f53101056d0 0x7f53101a13a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:32:29.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:29.123+0000 7f530fe00640 1 --2- 172.21.15.90:0/1337980526 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5310104d10 0x7f531019a330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:32:29.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:29.123+0000 7f5315000640 1 -- 172.21.15.90:0/1337980526 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5310104d10 msgr2=0x7f531019a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:32:29.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:29.123+0000 7f5315000640 1 --2- 172.21.15.90:0/1337980526 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5310104d10 0x7f531019a330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:29.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:29.123+0000 7f5315000640 1 -- 172.21.15.90:0/1337980526 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f53101056d0 msgr2=0x7f53101a13a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:32:29.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:29.123+0000 7f5315000640 1 --2- 172.21.15.90:0/1337980526 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f53101056d0 0x7f53101a13a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:29.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:29.123+0000 7f5315000640 1 -- 172.21.15.90:0/1337980526 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53101a1a10 con 0x7f5310103b20 2024-04-02T14:32:29.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:29.124+0000 7f5315000640 1 --2- 172.21.15.90:0/1337980526 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5310103b20 0x7f5310199df0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f530000b450 tx=0x7f530000b910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:32:29.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:29.124+0000 7f530d600640 1 -- 172.21.15.90:0/1337980526 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5300015020 con 0x7f5310103b20 2024-04-02T14:32:29.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:29.124+0000 7f530d600640 1 -- 172.21.15.90:0/1337980526 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5300011d70 con 0x7f5310103b20 2024-04-02T14:32:29.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:29.124+0000 7f5316400640 1 -- 172.21.15.90:0/1337980526 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53101a1d00 con 0x7f5310103b20 2024-04-02T14:32:29.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:29.124+0000 7f530d600640 1 -- 172.21.15.90:0/1337980526 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f530000f340 con 0x7f5310103b20 2024-04-02T14:32:29.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:29.124+0000 7f5316400640 1 -- 172.21.15.90:0/1337980526 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53101a2220 con 0x7f5310103b20 2024-04-02T14:32:29.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:29.125+0000 7f5316400640 1 -- 172.21.15.90:0/1337980526 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f531010ad70 con 0x7f5310103b20 2024-04-02T14:32:29.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:29.127+0000 7f530d600640 1 -- 172.21.15.90:0/1337980526 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f5300018020 con 0x7f5310103b20 2024-04-02T14:32:29.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:29.128+0000 7f530d600640 1 --2- 172.21.15.90:0/1337980526 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f52f40a3cc0 0x7f52f40a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:32:29.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:29.128+0000 7f530d600640 1 -- 172.21.15.90:0/1337980526 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 7979+0+0 (secure 0 0 0) 0x7f53000cfc80 con 0x7f5310103b20 2024-04-02T14:32:29.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:29.128+0000 7f530fe00640 1 --2- 172.21.15.90:0/1337980526 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f52f40a3cc0 0x7f52f40a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:32:29.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:29.129+0000 7f530fe00640 1 --2- 172.21.15.90:0/1337980526 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f52f40a3cc0 0x7f52f40a6180 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f52fc0095f0 tx=0x7f52fc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:32:29.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:29.132+0000 7f530d600640 1 -- 172.21.15.90:0/1337980526 <== mon.2 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7f530001f020 con 0x7f5310103b20 2024-04-02T14:32:29.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:28 smithi177 ceph-mon[36431]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-04-02T14:32:29.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:28 smithi177 ceph-mon[36431]: mds.? [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] up:active 2024-04-02T14:32:29.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:28 smithi177 ceph-mon[36431]: daemon mds.h assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-04-02T14:32:29.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:28 smithi177 ceph-mon[36431]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=i=up:active} 9 up:standby 2024-04-02T14:32:29.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:28 smithi177 ceph-mon[36431]: fsmap cephfs:4 {0=b=up:active,1=g=up:active,2=i=up:active,3=h=up:creating} 8 up:standby 2024-04-02T14:32:29.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:28 smithi177 ceph-mon[36431]: daemon mds.h is now active in filesystem cephfs as rank 3 2024-04-02T14:32:29.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:28 smithi177 ceph-mon[36431]: from='client.24674 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T14:32:29.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:28 smithi177 ceph-mon[36431]: pgmap v315: 97 pgs: 97 active+clean; 580 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s wr, 10 op/s 2024-04-02T14:32:29.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:28 smithi177 ceph-mon[36431]: from='client.24686 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T14:32:29.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:28 smithi090 ceph-mon[31674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-04-02T14:32:29.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:28 smithi090 ceph-mon[31674]: mds.? [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] up:active 2024-04-02T14:32:29.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:28 smithi090 ceph-mon[31674]: daemon mds.h assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-04-02T14:32:29.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:28 smithi090 ceph-mon[31674]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=i=up:active} 9 up:standby 2024-04-02T14:32:29.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:28 smithi090 ceph-mon[31674]: fsmap cephfs:4 {0=b=up:active,1=g=up:active,2=i=up:active,3=h=up:creating} 8 up:standby 2024-04-02T14:32:29.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:28 smithi090 ceph-mon[31674]: daemon mds.h is now active in filesystem cephfs as rank 3 2024-04-02T14:32:29.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:28 smithi090 ceph-mon[31674]: from='client.24674 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T14:32:29.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:28 smithi090 ceph-mon[31674]: pgmap v315: 97 pgs: 97 active+clean; 580 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s wr, 10 op/s 2024-04-02T14:32:29.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:28 smithi090 ceph-mon[31674]: from='client.24686 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T14:32:29.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:28 smithi092 ceph-mon[36497]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-04-02T14:32:29.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:28 smithi092 ceph-mon[36497]: mds.? [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] up:active 2024-04-02T14:32:29.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:28 smithi092 ceph-mon[36497]: daemon mds.h assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-04-02T14:32:29.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:28 smithi092 ceph-mon[36497]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=i=up:active} 9 up:standby 2024-04-02T14:32:29.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:28 smithi092 ceph-mon[36497]: fsmap cephfs:4 {0=b=up:active,1=g=up:active,2=i=up:active,3=h=up:creating} 8 up:standby 2024-04-02T14:32:29.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:28 smithi092 ceph-mon[36497]: daemon mds.h is now active in filesystem cephfs as rank 3 2024-04-02T14:32:29.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:28 smithi092 ceph-mon[36497]: from='client.24674 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T14:32:29.362 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:28 smithi092 ceph-mon[36497]: pgmap v315: 97 pgs: 97 active+clean; 580 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s wr, 10 op/s 2024-04-02T14:32:29.362 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:28 smithi092 ceph-mon[36497]: from='client.24686 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T14:32:29.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:29.367+0000 7f5316400640 1 -- 172.21.15.90:0/1337980526 --> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}) -- 0x7f5310061f40 con 0x7f52f40a3cc0 2024-04-02T14:32:29.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:29.379+0000 7f530d600640 1 -- 172.21.15.90:0/1337980526 <== mgr.14152 v2:172.21.15.90:6800/4266054817 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7f5310061f40 con 0x7f52f40a3cc0 2024-04-02T14:32:29.381 INFO:teuthology.orchestra.run.smithi090.stdout:/volumes/qa/sv_0/224f5436-6fd0-4fc0-849f-cb83e6e23018 2024-04-02T14:32:29.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:29.382+0000 7f5316400640 1 -- 172.21.15.90:0/1337980526 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f52f40a3cc0 msgr2=0x7f52f40a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:32:29.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:29.382+0000 7f5316400640 1 --2- 172.21.15.90:0/1337980526 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f52f40a3cc0 0x7f52f40a6180 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f52fc0095f0 tx=0x7f52fc005f90 comp rx=0 tx=0).stop 2024-04-02T14:32:29.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:29.382+0000 7f5316400640 1 -- 172.21.15.90:0/1337980526 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5310103b20 msgr2=0x7f5310199df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:32:29.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:29.382+0000 7f5316400640 1 --2- 172.21.15.90:0/1337980526 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5310103b20 0x7f5310199df0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f530000b450 tx=0x7f530000b910 comp rx=0 tx=0).stop 2024-04-02T14:32:29.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:29.382+0000 7f5316400640 1 -- 172.21.15.90:0/1337980526 shutdown_connections 2024-04-02T14:32:29.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:29.382+0000 7f5316400640 1 --2- 172.21.15.90:0/1337980526 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f52f40a3cc0 0x7f52f40a6180 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:29.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:29.382+0000 7f5316400640 1 --2- 172.21.15.90:0/1337980526 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f53101056d0 0x7f53101a13a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:29.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:29.382+0000 7f5316400640 1 --2- 172.21.15.90:0/1337980526 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5310104d10 0x7f531019a330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:29.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:29.382+0000 7f5316400640 1 --2- 172.21.15.90:0/1337980526 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5310103b20 0x7f5310199df0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:29.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:29.382+0000 7f5316400640 1 -- 172.21.15.90:0/1337980526 >> 172.21.15.90:0/1337980526 conn(0x7f53100ff0d0 msgr2=0x7f5310107f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:32:29.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:29.382+0000 7f5316400640 1 -- 172.21.15.90:0/1337980526 shutdown_connections 2024-04-02T14:32:29.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:29.382+0000 7f5316400640 1 -- 172.21.15.90:0/1337980526 wait complete. 2024-04-02T14:32:29.403 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_1 --group_name qa 2024-04-02T14:32:29.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:29.610+0000 7f4deec00640 1 -- 172.21.15.90:0/3592658199 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4de81058f0 msgr2=0x7f4de810c180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:32:29.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:29.610+0000 7f4deec00640 1 --2- 172.21.15.90:0/3592658199 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4de81058f0 0x7f4de810c180 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f4de0009be0 tx=0x7f4de002bef0 comp rx=0 tx=0).stop 2024-04-02T14:32:29.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:29.610+0000 7f4deec00640 1 -- 172.21.15.90:0/3592658199 shutdown_connections 2024-04-02T14:32:29.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:29.610+0000 7f4deec00640 1 --2- 172.21.15.90:0/3592658199 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4de81058f0 0x7f4de810c180 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:29.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:29.610+0000 7f4deec00640 1 --2- 172.21.15.90:0/3592658199 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4de8104f30 0x7f4de81053b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:29.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:29.610+0000 7f4deec00640 1 --2- 172.21.15.90:0/3592658199 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4de8103d40 0x7f4de8104160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:29.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:29.610+0000 7f4deec00640 1 -- 172.21.15.90:0/3592658199 >> 172.21.15.90:0/3592658199 conn(0x7f4de80ff2b0 msgr2=0x7f4de81016f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:32:29.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:29.610+0000 7f4deec00640 1 -- 172.21.15.90:0/3592658199 shutdown_connections 2024-04-02T14:32:29.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:29.610+0000 7f4deec00640 1 -- 172.21.15.90:0/3592658199 wait complete. 2024-04-02T14:32:29.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:29.612+0000 7f4deec00640 1 Processor -- start 2024-04-02T14:32:29.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:29.612+0000 7f4deec00640 1 -- start start 2024-04-02T14:32:29.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:29.612+0000 7f4deec00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4de8103d40 0x7f4de8199fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:32:29.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:29.612+0000 7f4deec00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4de8104f30 0x7f4de819a500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:32:29.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:29.612+0000 7f4deec00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4de81058f0 0x7f4de81a1570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:32:29.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:29.612+0000 7f4deec00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f4de810f9a0 con 0x7f4de8104f30 2024-04-02T14:32:29.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:29.612+0000 7f4deec00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f4de810f820 con 0x7f4de81058f0 2024-04-02T14:32:29.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:29.612+0000 7f4deec00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f4de810fb20 con 0x7f4de8103d40 2024-04-02T14:32:29.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:29.613+0000 7f4de7400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4de8104f30 0x7f4de819a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:32:29.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:29.613+0000 7f4de7400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4de8104f30 0x7f4de819a500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:49412/0 (socket says 172.21.15.90:49412) 2024-04-02T14:32:29.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:29.613+0000 7f4decc00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4de81058f0 0x7f4de81a1570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:32:29.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:29.613+0000 7f4de7400640 1 -- 172.21.15.90:0/3083721192 learned_addr learned my addr 172.21.15.90:0/3083721192 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:32:29.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:29.613+0000 7f4de7e00640 1 --2- 172.21.15.90:0/3083721192 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4de8103d40 0x7f4de8199fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:32:29.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:29.613+0000 7f4decc00640 1 -- 172.21.15.90:0/3083721192 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4de8103d40 msgr2=0x7f4de8199fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:32:29.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:29.613+0000 7f4decc00640 1 --2- 172.21.15.90:0/3083721192 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4de8103d40 0x7f4de8199fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:29.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:29.613+0000 7f4decc00640 1 -- 172.21.15.90:0/3083721192 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4de8104f30 msgr2=0x7f4de819a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:32:29.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:29.613+0000 7f4decc00640 1 --2- 172.21.15.90:0/3083721192 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4de8104f30 0x7f4de819a500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:29.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:29.613+0000 7f4decc00640 1 -- 172.21.15.90:0/3083721192 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4de81a1be0 con 0x7f4de81058f0 2024-04-02T14:32:29.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:29.613+0000 7f4decc00640 1 --2- 172.21.15.90:0/3083721192 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4de81058f0 0x7f4de81a1570 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f4de002c430 tx=0x7f4de002cb90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:32:29.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:29.613+0000 7f4de4c00640 1 -- 172.21.15.90:0/3083721192 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4de0043020 con 0x7f4de81058f0 2024-04-02T14:32:29.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:29.614+0000 7f4de4c00640 1 -- 172.21.15.90:0/3083721192 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4de0003e50 con 0x7f4de81058f0 2024-04-02T14:32:29.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:29.614+0000 7f4de4c00640 1 -- 172.21.15.90:0/3083721192 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4de0034430 con 0x7f4de81058f0 2024-04-02T14:32:29.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:29.614+0000 7f4deec00640 1 -- 172.21.15.90:0/3083721192 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4de81a1e70 con 0x7f4de81058f0 2024-04-02T14:32:29.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:29.614+0000 7f4deec00640 1 -- 172.21.15.90:0/3083721192 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4de81a2380 con 0x7f4de81058f0 2024-04-02T14:32:29.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:29.615+0000 7f4deec00640 1 -- 172.21.15.90:0/3083721192 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4de810b060 con 0x7f4de81058f0 2024-04-02T14:32:29.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:29.616+0000 7f4de4c00640 1 -- 172.21.15.90:0/3083721192 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f4de0046020 con 0x7f4de81058f0 2024-04-02T14:32:29.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:29.618+0000 7f4de4c00640 1 --2- 172.21.15.90:0/3083721192 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f4dcc0a3cc0 0x7f4dcc0a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:32:29.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:29.618+0000 7f4de4c00640 1 -- 172.21.15.90:0/3083721192 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 7979+0+0 (secure 0 0 0) 0x7f4de00f2840 con 0x7f4de81058f0 2024-04-02T14:32:29.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:29.618+0000 7f4de7e00640 1 --2- 172.21.15.90:0/3083721192 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f4dcc0a3cc0 0x7f4dcc0a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:32:29.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:29.618+0000 7f4de7e00640 1 --2- 172.21.15.90:0/3083721192 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f4dcc0a3cc0 0x7f4dcc0a6180 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f4de8104d90 tx=0x7f4dd8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:32:29.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:29.622+0000 7f4de4c00640 1 -- 172.21.15.90:0/3083721192 <== mon.2 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7f4de004d020 con 0x7f4de81058f0 2024-04-02T14:32:29.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:29.859+0000 7f4deec00640 1 -- 172.21.15.90:0/3083721192 --> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f4de8061fd0 con 0x7f4dcc0a3cc0 2024-04-02T14:32:29.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:29.893+0000 7f4de4c00640 1 -- 172.21.15.90:0/3083721192 <== mgr.14152 v2:172.21.15.90:6800/4266054817 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f4de8061fd0 con 0x7f4dcc0a3cc0 2024-04-02T14:32:29.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:29.895+0000 7f4deec00640 1 -- 172.21.15.90:0/3083721192 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f4dcc0a3cc0 msgr2=0x7f4dcc0a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:32:29.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:29.895+0000 7f4deec00640 1 --2- 172.21.15.90:0/3083721192 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f4dcc0a3cc0 0x7f4dcc0a6180 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f4de8104d90 tx=0x7f4dd8009260 comp rx=0 tx=0).stop 2024-04-02T14:32:29.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:29.895+0000 7f4deec00640 1 -- 172.21.15.90:0/3083721192 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4de81058f0 msgr2=0x7f4de81a1570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:32:29.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:29.895+0000 7f4deec00640 1 --2- 172.21.15.90:0/3083721192 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4de81058f0 0x7f4de81a1570 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f4de002c430 tx=0x7f4de002cb90 comp rx=0 tx=0).stop 2024-04-02T14:32:29.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:29.896+0000 7f4deec00640 1 -- 172.21.15.90:0/3083721192 shutdown_connections 2024-04-02T14:32:29.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:29.896+0000 7f4deec00640 1 --2- 172.21.15.90:0/3083721192 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f4dcc0a3cc0 0x7f4dcc0a6180 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:29.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:29.896+0000 7f4deec00640 1 --2- 172.21.15.90:0/3083721192 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4de81058f0 0x7f4de81a1570 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:29.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:29.896+0000 7f4deec00640 1 --2- 172.21.15.90:0/3083721192 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4de8104f30 0x7f4de819a500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:29.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:29.896+0000 7f4deec00640 1 --2- 172.21.15.90:0/3083721192 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4de8103d40 0x7f4de8199fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:29.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:29.896+0000 7f4deec00640 1 -- 172.21.15.90:0/3083721192 >> 172.21.15.90:0/3083721192 conn(0x7f4de80ff2b0 msgr2=0x7f4de8108230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:32:29.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:29.896+0000 7f4deec00640 1 -- 172.21.15.90:0/3083721192 shutdown_connections 2024-04-02T14:32:29.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:29.896+0000 7f4deec00640 1 -- 172.21.15.90:0/3083721192 wait complete. 2024-04-02T14:32:29.917 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_1 2024-04-02T14:32:30.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:30.124+0000 7f8a3d600640 1 -- 172.21.15.90:0/3066949014 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8a38104f00 msgr2=0x7f8a38105380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:32:30.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:30.124+0000 7f8a3d600640 1 --2- 172.21.15.90:0/3066949014 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8a38104f00 0x7f8a38105380 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f8a24009900 tx=0x7f8a2402bc40 comp rx=0 tx=0).stop 2024-04-02T14:32:30.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:30.124+0000 7f8a3d600640 1 -- 172.21.15.90:0/3066949014 shutdown_connections 2024-04-02T14:32:30.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:30.124+0000 7f8a3d600640 1 --2- 172.21.15.90:0/3066949014 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8a381058c0 0x7f8a3810c150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:30.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:30.124+0000 7f8a3d600640 1 --2- 172.21.15.90:0/3066949014 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8a38104f00 0x7f8a38105380 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:30.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:30.124+0000 7f8a3d600640 1 --2- 172.21.15.90:0/3066949014 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8a38103d10 0x7f8a38104130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:30.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:30.124+0000 7f8a3d600640 1 -- 172.21.15.90:0/3066949014 >> 172.21.15.90:0/3066949014 conn(0x7f8a380ff2c0 msgr2=0x7f8a381016e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:32:30.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:30.125+0000 7f8a3d600640 1 -- 172.21.15.90:0/3066949014 shutdown_connections 2024-04-02T14:32:30.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:30.125+0000 7f8a3d600640 1 -- 172.21.15.90:0/3066949014 wait complete. 2024-04-02T14:32:30.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:30.125+0000 7f8a3d600640 1 Processor -- start 2024-04-02T14:32:30.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:30.126+0000 7f8a3d600640 1 -- start start 2024-04-02T14:32:30.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:30.126+0000 7f8a3d600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8a38103d10 0x7f8a38199ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:32:30.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:30.126+0000 7f8a3d600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8a38104f00 0x7f8a3819a530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:32:30.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:30.126+0000 7f8a3d600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8a381058c0 0x7f8a381a15a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:32:30.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:30.126+0000 7f8a3d600640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f8a3810f9e0 con 0x7f8a38103d10 2024-04-02T14:32:30.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:30.126+0000 7f8a3d600640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f8a3810f860 con 0x7f8a38104f00 2024-04-02T14:32:30.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:30.126+0000 7f8a3d600640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f8a3810fb60 con 0x7f8a381058c0 2024-04-02T14:32:30.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:30.126+0000 7f8a36a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8a38103d10 0x7f8a38199ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:32:30.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:30.126+0000 7f8a36a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8a38103d10 0x7f8a38199ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:49424/0 (socket says 172.21.15.90:49424) 2024-04-02T14:32:30.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:30.126+0000 7f8a36a00640 1 -- 172.21.15.90:0/3987422262 learned_addr learned my addr 172.21.15.90:0/3987422262 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:32:30.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:30.126+0000 7f8a37400640 1 --2- 172.21.15.90:0/3987422262 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8a381058c0 0x7f8a381a15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:32:30.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:30.126+0000 7f8a36000640 1 --2- 172.21.15.90:0/3987422262 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8a38104f00 0x7f8a3819a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:32:30.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:30.126+0000 7f8a36a00640 1 -- 172.21.15.90:0/3987422262 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8a381058c0 msgr2=0x7f8a381a15a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:32:30.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:30.126+0000 7f8a36a00640 1 --2- 172.21.15.90:0/3987422262 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8a381058c0 0x7f8a381a15a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:30.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:30.126+0000 7f8a36a00640 1 -- 172.21.15.90:0/3987422262 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8a38104f00 msgr2=0x7f8a3819a530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:32:30.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:30.126+0000 7f8a36a00640 1 --2- 172.21.15.90:0/3987422262 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8a38104f00 0x7f8a3819a530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:30.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:30.126+0000 7f8a36a00640 1 -- 172.21.15.90:0/3987422262 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a381a1c10 con 0x7f8a38103d10 2024-04-02T14:32:30.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:30.127+0000 7f8a36a00640 1 --2- 172.21.15.90:0/3987422262 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8a38103d10 0x7f8a38199ff0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f8a2800e9c0 tx=0x7f8a2800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:32:30.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:30.127+0000 7f8a2f400640 1 -- 172.21.15.90:0/3987422262 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a2800cd80 con 0x7f8a38103d10 2024-04-02T14:32:30.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:30.127+0000 7f8a2f400640 1 -- 172.21.15.90:0/3987422262 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8a280035f0 con 0x7f8a38103d10 2024-04-02T14:32:30.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:30.127+0000 7f8a3d600640 1 -- 172.21.15.90:0/3987422262 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a381a1f00 con 0x7f8a38103d10 2024-04-02T14:32:30.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:30.127+0000 7f8a2f400640 1 -- 172.21.15.90:0/3987422262 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a28010690 con 0x7f8a38103d10 2024-04-02T14:32:30.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:30.127+0000 7f8a3d600640 1 -- 172.21.15.90:0/3987422262 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a381a2370 con 0x7f8a38103d10 2024-04-02T14:32:30.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:30.128+0000 7f8a3d600640 1 -- 172.21.15.90:0/3987422262 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8a3810b0b0 con 0x7f8a38103d10 2024-04-02T14:32:30.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:30.130+0000 7f8a2f400640 1 -- 172.21.15.90:0/3987422262 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f8a280037a0 con 0x7f8a38103d10 2024-04-02T14:32:30.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:30.131+0000 7f8a2f400640 1 --2- 172.21.15.90:0/3987422262 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f8a1c0a3cc0 0x7f8a1c0a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:32:30.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:30.132+0000 7f8a2f400640 1 -- 172.21.15.90:0/3987422262 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 7979+0+0 (secure 0 0 0) 0x7f8a280d0820 con 0x7f8a38103d10 2024-04-02T14:32:30.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:30.132+0000 7f8a36000640 1 --2- 172.21.15.90:0/3987422262 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f8a1c0a3cc0 0x7f8a1c0a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:32:30.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:30.132+0000 7f8a36000640 1 --2- 172.21.15.90:0/3987422262 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f8a1c0a3cc0 0x7f8a1c0a6180 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f8a2402c250 tx=0x7f8a2402cf90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:32:30.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:30.136+0000 7f8a2f400640 1 -- 172.21.15.90:0/3987422262 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7f8a2809d150 con 0x7f8a38103d10 2024-04-02T14:32:30.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:29 smithi177 ceph-mon[36431]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-04-02T14:32:30.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:29 smithi177 ceph-mon[36431]: mds.? [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] up:active 2024-04-02T14:32:30.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:29 smithi177 ceph-mon[36431]: daemon mds.c assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-04-02T14:32:30.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:29 smithi177 ceph-mon[36431]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-02T14:32:30.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:29 smithi177 ceph-mon[36431]: Cluster is now healthy 2024-04-02T14:32:30.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:29 smithi177 ceph-mon[36431]: fsmap cephfs:4 {0=b=up:active,1=g=up:active,2=i=up:active,3=h=up:active} 8 up:standby 2024-04-02T14:32:30.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:29 smithi177 ceph-mon[36431]: fsmap cephfs:5 1 up:creating 7 up:standby 4 up:active 2024-04-02T14:32:30.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:29 smithi177 ceph-mon[36431]: daemon mds.c is now active in filesystem cephfs as rank 4 2024-04-02T14:32:30.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:29 smithi177 ceph-mon[36431]: from='client.24692 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T14:32:30.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:29 smithi177 ceph-mon[36431]: mds.? [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] up:active 2024-04-02T14:32:30.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:29 smithi177 ceph-mon[36431]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-02T14:32:30.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:29 smithi177 ceph-mon[36431]: from='client.24698 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T14:32:30.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:29 smithi090 ceph-mon[31674]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-04-02T14:32:30.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:29 smithi090 ceph-mon[31674]: mds.? [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] up:active 2024-04-02T14:32:30.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:29 smithi090 ceph-mon[31674]: daemon mds.c assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-04-02T14:32:30.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:29 smithi090 ceph-mon[31674]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-02T14:32:30.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:29 smithi090 ceph-mon[31674]: Cluster is now healthy 2024-04-02T14:32:30.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:29 smithi090 ceph-mon[31674]: fsmap cephfs:4 {0=b=up:active,1=g=up:active,2=i=up:active,3=h=up:active} 8 up:standby 2024-04-02T14:32:30.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:29 smithi090 ceph-mon[31674]: fsmap cephfs:5 1 up:creating 7 up:standby 4 up:active 2024-04-02T14:32:30.190 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:29 smithi090 ceph-mon[31674]: daemon mds.c is now active in filesystem cephfs as rank 4 2024-04-02T14:32:30.190 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:29 smithi090 ceph-mon[31674]: from='client.24692 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T14:32:30.190 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:29 smithi090 ceph-mon[31674]: mds.? [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] up:active 2024-04-02T14:32:30.190 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:29 smithi090 ceph-mon[31674]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-02T14:32:30.190 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:29 smithi090 ceph-mon[31674]: from='client.24698 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T14:32:30.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:29 smithi092 ceph-mon[36497]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-04-02T14:32:30.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:29 smithi092 ceph-mon[36497]: mds.? [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] up:active 2024-04-02T14:32:30.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:29 smithi092 ceph-mon[36497]: daemon mds.c assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-04-02T14:32:30.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:29 smithi092 ceph-mon[36497]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-02T14:32:30.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:29 smithi092 ceph-mon[36497]: Cluster is now healthy 2024-04-02T14:32:30.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:29 smithi092 ceph-mon[36497]: fsmap cephfs:4 {0=b=up:active,1=g=up:active,2=i=up:active,3=h=up:active} 8 up:standby 2024-04-02T14:32:30.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:29 smithi092 ceph-mon[36497]: fsmap cephfs:5 1 up:creating 7 up:standby 4 up:active 2024-04-02T14:32:30.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:29 smithi092 ceph-mon[36497]: daemon mds.c is now active in filesystem cephfs as rank 4 2024-04-02T14:32:30.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:29 smithi092 ceph-mon[36497]: from='client.24692 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T14:32:30.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:29 smithi092 ceph-mon[36497]: mds.? [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] up:active 2024-04-02T14:32:30.362 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:29 smithi092 ceph-mon[36497]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-02T14:32:30.362 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:29 smithi092 ceph-mon[36497]: from='client.24698 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T14:32:30.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:30.371+0000 7f8a3d600640 1 -- 172.21.15.90:0/3987422262 --> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}) -- 0x7f8a38061f40 con 0x7f8a1c0a3cc0 2024-04-02T14:32:30.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:30.379+0000 7f8a2f400640 1 -- 172.21.15.90:0/3987422262 <== mgr.14152 v2:172.21.15.90:6800/4266054817 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7f8a38061f40 con 0x7f8a1c0a3cc0 2024-04-02T14:32:30.381 INFO:teuthology.orchestra.run.smithi090.stdout:/volumes/qa/sv_1/136ad3dc-4348-42ec-932a-abe31a1ca7ec 2024-04-02T14:32:30.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:30.381+0000 7f8a3d600640 1 -- 172.21.15.90:0/3987422262 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f8a1c0a3cc0 msgr2=0x7f8a1c0a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:32:30.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:30.381+0000 7f8a3d600640 1 --2- 172.21.15.90:0/3987422262 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f8a1c0a3cc0 0x7f8a1c0a6180 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f8a2402c250 tx=0x7f8a2402cf90 comp rx=0 tx=0).stop 2024-04-02T14:32:30.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:30.381+0000 7f8a3d600640 1 -- 172.21.15.90:0/3987422262 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8a38103d10 msgr2=0x7f8a38199ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:32:30.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:30.381+0000 7f8a3d600640 1 --2- 172.21.15.90:0/3987422262 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8a38103d10 0x7f8a38199ff0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f8a2800e9c0 tx=0x7f8a2800ee80 comp rx=0 tx=0).stop 2024-04-02T14:32:30.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:30.382+0000 7f8a3d600640 1 -- 172.21.15.90:0/3987422262 shutdown_connections 2024-04-02T14:32:30.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:30.382+0000 7f8a3d600640 1 --2- 172.21.15.90:0/3987422262 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f8a1c0a3cc0 0x7f8a1c0a6180 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:30.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:30.382+0000 7f8a3d600640 1 --2- 172.21.15.90:0/3987422262 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8a381058c0 0x7f8a381a15a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:30.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:30.382+0000 7f8a3d600640 1 --2- 172.21.15.90:0/3987422262 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8a38104f00 0x7f8a3819a530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:30.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:30.382+0000 7f8a3d600640 1 --2- 172.21.15.90:0/3987422262 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8a38103d10 0x7f8a38199ff0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:30.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:30.382+0000 7f8a3d600640 1 -- 172.21.15.90:0/3987422262 >> 172.21.15.90:0/3987422262 conn(0x7f8a380ff2c0 msgr2=0x7f8a38108280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:32:30.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:30.382+0000 7f8a3d600640 1 -- 172.21.15.90:0/3987422262 shutdown_connections 2024-04-02T14:32:30.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:30.382+0000 7f8a3d600640 1 -- 172.21.15.90:0/3987422262 wait complete. 2024-04-02T14:32:30.404 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T14:32:30.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:30.618+0000 7fc19d600640 1 -- 172.21.15.90:0/266558870 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc1980780a0 msgr2=0x7fc198078940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:32:30.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:30.619+0000 7fc19d600640 1 --2- 172.21.15.90:0/266558870 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc1980780a0 0x7fc198078940 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fc190009ca0 tx=0x7fc19002c0f0 comp rx=0 tx=0).stop 2024-04-02T14:32:30.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:30.619+0000 7fc19d600640 1 -- 172.21.15.90:0/266558870 shutdown_connections 2024-04-02T14:32:30.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:30.619+0000 7fc19d600640 1 --2- 172.21.15.90:0/266558870 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc1980780a0 0x7fc198078940 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:30.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:30.619+0000 7fc19d600640 1 --2- 172.21.15.90:0/266558870 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc198077700 0x7fc198077b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:30.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:30.619+0000 7fc19d600640 1 --2- 172.21.15.90:0/266558870 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc1980764c0 0x7fc1980768e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:30.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:30.619+0000 7fc19d600640 1 -- 172.21.15.90:0/266558870 >> 172.21.15.90:0/266558870 conn(0x7fc1980ff2b0 msgr2=0x7fc1981016f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:32:30.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:30.619+0000 7fc19d600640 1 -- 172.21.15.90:0/266558870 shutdown_connections 2024-04-02T14:32:30.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:30.619+0000 7fc19d600640 1 -- 172.21.15.90:0/266558870 wait complete. 2024-04-02T14:32:30.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:30.620+0000 7fc19d600640 1 Processor -- start 2024-04-02T14:32:30.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:30.620+0000 7fc19d600640 1 -- start start 2024-04-02T14:32:30.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:30.620+0000 7fc19d600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc1980764c0 0x7fc19819e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:32:30.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:30.620+0000 7fc19d600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc198077700 0x7fc19819e910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:32:30.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:30.620+0000 7fc19d600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc1980780a0 0x7fc1981a5980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:32:30.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:30.620+0000 7fc19d600640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fc198113dd0 con 0x7fc1980780a0 2024-04-02T14:32:30.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:30.620+0000 7fc19d600640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fc198113c50 con 0x7fc1980764c0 2024-04-02T14:32:30.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:30.620+0000 7fc19d600640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fc198113f50 con 0x7fc198077700 2024-04-02T14:32:30.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:30.620+0000 7fc197400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc1980780a0 0x7fc1981a5980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:32:30.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:30.620+0000 7fc18e000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc198077700 0x7fc19819e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:32:30.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:30.620+0000 7fc196a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc1980764c0 0x7fc19819e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:32:30.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:30.620+0000 7fc197400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc1980780a0 0x7fc1981a5980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:49446/0 (socket says 172.21.15.90:49446) 2024-04-02T14:32:30.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:30.620+0000 7fc197400640 1 -- 172.21.15.90:0/2875659570 learned_addr learned my addr 172.21.15.90:0/2875659570 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:32:30.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:30.621+0000 7fc18e000640 1 -- 172.21.15.90:0/2875659570 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc1980764c0 msgr2=0x7fc19819e3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:32:30.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:30.621+0000 7fc18e000640 1 --2- 172.21.15.90:0/2875659570 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc1980764c0 0x7fc19819e3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:30.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:30.621+0000 7fc18e000640 1 -- 172.21.15.90:0/2875659570 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc1980780a0 msgr2=0x7fc1981a5980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:32:30.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:30.621+0000 7fc18e000640 1 --2- 172.21.15.90:0/2875659570 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc1980780a0 0x7fc1981a5980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:30.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:30.621+0000 7fc18e000640 1 -- 172.21.15.90:0/2875659570 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1981a5ff0 con 0x7fc198077700 2024-04-02T14:32:30.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:30.621+0000 7fc196a00640 1 --2- 172.21.15.90:0/2875659570 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc1980764c0 0x7fc19819e3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:32:30.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:30.621+0000 7fc197400640 1 --2- 172.21.15.90:0/2875659570 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc1980780a0 0x7fc1981a5980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:32:30.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:30.621+0000 7fc18e000640 1 --2- 172.21.15.90:0/2875659570 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc198077700 0x7fc19819e910 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fc18000e9f0 tx=0x7fc18000eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:32:30.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:30.621+0000 7fc18fe00640 1 -- 172.21.15.90:0/2875659570 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc18000ccc0 con 0x7fc198077700 2024-04-02T14:32:30.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:30.621+0000 7fc19d600640 1 -- 172.21.15.90:0/2875659570 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc1981a62e0 con 0x7fc198077700 2024-04-02T14:32:30.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:30.621+0000 7fc18fe00640 1 -- 172.21.15.90:0/2875659570 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc180010910 con 0x7fc198077700 2024-04-02T14:32:30.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:30.621+0000 7fc18fe00640 1 -- 172.21.15.90:0/2875659570 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc180018960 con 0x7fc198077700 2024-04-02T14:32:30.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:30.621+0000 7fc19d600640 1 -- 172.21.15.90:0/2875659570 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1981a6880 con 0x7fc198077700 2024-04-02T14:32:30.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:30.623+0000 7fc19d600640 1 -- 172.21.15.90:0/2875659570 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc198110800 con 0x7fc198077700 2024-04-02T14:32:30.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:30.624+0000 7fc18fe00640 1 -- 172.21.15.90:0/2875659570 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fc180010ac0 con 0x7fc198077700 2024-04-02T14:32:30.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:30.626+0000 7fc18fe00640 1 --2- 172.21.15.90:0/2875659570 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fc17c0a3cc0 0x7fc17c0a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:32:30.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:30.626+0000 7fc18fe00640 1 -- 172.21.15.90:0/2875659570 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 7979+0+0 (secure 0 0 0) 0x7fc1800d07d0 con 0x7fc198077700 2024-04-02T14:32:30.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:30.626+0000 7fc196a00640 1 --2- 172.21.15.90:0/2875659570 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fc17c0a3cc0 0x7fc17c0a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:32:30.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:30.627+0000 7fc196a00640 1 --2- 172.21.15.90:0/2875659570 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fc17c0a3cc0 0x7fc17c0a6180 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fc188009fc0 tx=0x7fc188005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:32:30.630 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:30.628+0000 7fc18fe00640 1 -- 172.21.15.90:0/2875659570 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7fc18009cab0 con 0x7fc198077700 2024-04-02T14:32:30.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:30.872+0000 7fc19d600640 1 -- 172.21.15.90:0/2875659570 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc19802d5c0 con 0x7fc198077700 2024-04-02T14:32:30.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:30.873+0000 7fc18fe00640 1 -- 172.21.15.90:0/2875659570 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11927 (secure 0 0 0) 0x7fc1800a0db0 con 0x7fc198077700 2024-04-02T14:32:30.876 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-02T14:32:30.876 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14412,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6837/2852160309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6836","nonce":2852160309},{"type":"v1","addr":"172.21.15.90:6837","nonce":2852160309}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14478,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6841/1919376858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6840","nonce":1919376858},{"type":"v1","addr":"172.21.15.90:6841","nonce":1919376858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":26,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-02T14:32:22.319083+0000","modified":"2024-04-02T14:32:29.812071+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24590,"mds_1":24560,"mds_2":24547,"mds_3":24539,"mds_4":24523},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24454":{"gid":24454,"name":"l","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6833/1564319431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":1564319431},{"type":"v1","addr":"172.21.15.177:6833","nonce":1564319431}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.92:6833/1570337372","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":1570337372},{"type":"v1","addr":"172.21.15.92:6833","nonce":1570337372}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"j","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.90:6835/1277124891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6834","nonce":1277124891},{"type":"v1","addr":"172.21.15.90:6835","nonce":1277124891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.92:6835/860721164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":860721164},{"type":"v1","addr":"172.21.15.92:6835","nonce":860721164}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24511":{"gid":24511,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6835/3177244988","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":3177244988},{"type":"v1","addr":"172.21.15.177:6835","nonce":3177244988}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"c","rank":4,"incarnation":25,"state":"up:active","state_seq":9,"addr":"172.21.15.177:6837/4223414858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":4223414858},{"type":"v1","addr":"172.21.15.177:6837","nonce":4223414858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24539":{"gid":24539,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":8,"addr":"172.21.15.92:6837/1270635939","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":1270635939},{"type":"v1","addr":"172.21.15.92:6837","nonce":1270635939}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24547":{"gid":24547,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":7,"addr":"172.21.15.177:6839/664895663","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":664895663},{"type":"v1","addr":"172.21.15.177:6839","nonce":664895663}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"g","rank":1,"incarnation":19,"state":"up:active","state_seq":5,"addr":"172.21.15.90:6839/1498915870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6838","nonce":1498915870},{"type":"v1","addr":"172.21.15.90:6839","nonce":1498915870}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24590":{"gid":24590,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.92:6839/1054888219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":1054888219},{"type":"v1","addr":"172.21.15.92:6839","nonce":1054888219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24590,"qdb_cluster":[24539,24560,24523,24590,24547]},"id":1}]} 2024-04-02T14:32:30.877 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 26 2024-04-02T14:32:30.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:30.876+0000 7fc19d600640 1 -- 172.21.15.90:0/2875659570 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fc17c0a3cc0 msgr2=0x7fc17c0a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:32:30.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:30.876+0000 7fc19d600640 1 --2- 172.21.15.90:0/2875659570 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fc17c0a3cc0 0x7fc17c0a6180 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fc188009fc0 tx=0x7fc188005f90 comp rx=0 tx=0).stop 2024-04-02T14:32:30.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:30.876+0000 7fc19d600640 1 -- 172.21.15.90:0/2875659570 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc198077700 msgr2=0x7fc19819e910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:32:30.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:30.876+0000 7fc19d600640 1 --2- 172.21.15.90:0/2875659570 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc198077700 0x7fc19819e910 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fc18000e9f0 tx=0x7fc18000eeb0 comp rx=0 tx=0).stop 2024-04-02T14:32:30.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:30.877+0000 7fc19d600640 1 -- 172.21.15.90:0/2875659570 shutdown_connections 2024-04-02T14:32:30.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:30.877+0000 7fc19d600640 1 --2- 172.21.15.90:0/2875659570 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fc17c0a3cc0 0x7fc17c0a6180 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:30.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:30.877+0000 7fc19d600640 1 --2- 172.21.15.90:0/2875659570 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc1980780a0 0x7fc1981a5980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:30.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:30.877+0000 7fc19d600640 1 --2- 172.21.15.90:0/2875659570 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc198077700 0x7fc19819e910 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:30.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:30.877+0000 7fc19d600640 1 --2- 172.21.15.90:0/2875659570 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc1980764c0 0x7fc19819e3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:30.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:30.877+0000 7fc19d600640 1 -- 172.21.15.90:0/2875659570 >> 172.21.15.90:0/2875659570 conn(0x7fc1980ff2b0 msgr2=0x7fc198100950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:32:30.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:30.877+0000 7fc19d600640 1 -- 172.21.15.90:0/2875659570 shutdown_connections 2024-04-02T14:32:30.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:30.877+0000 7fc19d600640 1 -- 172.21.15.90:0/2875659570 wait complete. 2024-04-02T14:32:30.899 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T14:32:31.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:31.108+0000 7f18a1a00640 1 -- 172.21.15.90:0/496561032 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f189c1058f0 msgr2=0x7f189c10c180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:32:31.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:31.108+0000 7f18a1a00640 1 --2- 172.21.15.90:0/496561032 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f189c1058f0 0x7f189c10c180 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f1894009ca0 tx=0x7f189402c100 comp rx=0 tx=0).stop 2024-04-02T14:32:31.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:31.108+0000 7f18a1a00640 1 -- 172.21.15.90:0/496561032 shutdown_connections 2024-04-02T14:32:31.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:31.108+0000 7f18a1a00640 1 --2- 172.21.15.90:0/496561032 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f189c1058f0 0x7f189c10c180 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:31.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:31.108+0000 7f18a1a00640 1 --2- 172.21.15.90:0/496561032 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f189c104f30 0x7f189c1053b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:31.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:31.108+0000 7f18a1a00640 1 --2- 172.21.15.90:0/496561032 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f189c103d40 0x7f189c104160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:31.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:31.108+0000 7f18a1a00640 1 -- 172.21.15.90:0/496561032 >> 172.21.15.90:0/496561032 conn(0x7f189c0ff2b0 msgr2=0x7f189c1016f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:32:31.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:31.108+0000 7f18a1a00640 1 -- 172.21.15.90:0/496561032 shutdown_connections 2024-04-02T14:32:31.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:31.108+0000 7f18a1a00640 1 -- 172.21.15.90:0/496561032 wait complete. 2024-04-02T14:32:31.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:31.109+0000 7f18a1a00640 1 Processor -- start 2024-04-02T14:32:31.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:31.109+0000 7f18a1a00640 1 -- start start 2024-04-02T14:32:31.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:31.109+0000 7f18a1a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f189c103d40 0x7f189c19a000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:32:31.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:31.109+0000 7f18a1a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f189c104f30 0x7f189c19a540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:32:31.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:31.109+0000 7f18a1a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f189c1058f0 0x7f189c1a15b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:32:31.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:31.109+0000 7f18a1a00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f189c10f9c0 con 0x7f189c103d40 2024-04-02T14:32:31.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:31.109+0000 7f18a1a00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f189c10f840 con 0x7f189c104f30 2024-04-02T14:32:31.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:31.109+0000 7f18a1a00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f189c10fb40 con 0x7f189c1058f0 2024-04-02T14:32:31.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:31.109+0000 7f189aa00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f189c103d40 0x7f189c19a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:32:31.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:31.109+0000 7f189aa00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f189c103d40 0x7f189c19a000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:49458/0 (socket says 172.21.15.90:49458) 2024-04-02T14:32:31.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:31.109+0000 7f189aa00640 1 -- 172.21.15.90:0/563069497 learned_addr learned my addr 172.21.15.90:0/563069497 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:32:31.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:31.109+0000 7f189a000640 1 --2- 172.21.15.90:0/563069497 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f189c104f30 0x7f189c19a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:32:31.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:31.110+0000 7f189b400640 1 --2- 172.21.15.90:0/563069497 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f189c1058f0 0x7f189c1a15b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:32:31.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:31.110+0000 7f189aa00640 1 -- 172.21.15.90:0/563069497 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f189c1058f0 msgr2=0x7f189c1a15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:32:31.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:31.110+0000 7f189aa00640 1 --2- 172.21.15.90:0/563069497 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f189c1058f0 0x7f189c1a15b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:31.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:31.110+0000 7f189aa00640 1 -- 172.21.15.90:0/563069497 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f189c104f30 msgr2=0x7f189c19a540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:32:31.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:31.110+0000 7f189aa00640 1 --2- 172.21.15.90:0/563069497 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f189c104f30 0x7f189c19a540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:31.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:31.110+0000 7f189aa00640 1 -- 172.21.15.90:0/563069497 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f189c1a1c20 con 0x7f189c103d40 2024-04-02T14:32:31.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:31.110+0000 7f189a000640 1 --2- 172.21.15.90:0/563069497 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f189c104f30 0x7f189c19a540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:32:31.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:31.110+0000 7f189aa00640 1 --2- 172.21.15.90:0/563069497 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f189c103d40 0x7f189c19a000 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f188c00e9c0 tx=0x7f188c00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:32:31.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:31.110+0000 7f1893400640 1 -- 172.21.15.90:0/563069497 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f188c00cde0 con 0x7f189c103d40 2024-04-02T14:32:31.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:31.110+0000 7f1893400640 1 -- 172.21.15.90:0/563069497 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f188c010910 con 0x7f189c103d40 2024-04-02T14:32:31.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:31.110+0000 7f18a1a00640 1 -- 172.21.15.90:0/563069497 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f189c1a1f10 con 0x7f189c103d40 2024-04-02T14:32:31.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:31.110+0000 7f1893400640 1 -- 172.21.15.90:0/563069497 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f188c018a00 con 0x7f189c103d40 2024-04-02T14:32:31.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:31.111+0000 7f18a1a00640 1 -- 172.21.15.90:0/563069497 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f189c1a2370 con 0x7f189c103d40 2024-04-02T14:32:31.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:31.111+0000 7f18a1a00640 1 -- 172.21.15.90:0/563069497 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f189c10b060 con 0x7f189c103d40 2024-04-02T14:32:31.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:31.113+0000 7f1893400640 1 -- 172.21.15.90:0/563069497 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f188c01f020 con 0x7f189c103d40 2024-04-02T14:32:31.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:31.114+0000 7f1893400640 1 --2- 172.21.15.90:0/563069497 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f18800a3c40 0x7f18800a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:32:31.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:31.115+0000 7f1893400640 1 -- 172.21.15.90:0/563069497 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 7979+0+0 (secure 0 0 0) 0x7f188c0d06b0 con 0x7f189c103d40 2024-04-02T14:32:31.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:31.115+0000 7f189a000640 1 --2- 172.21.15.90:0/563069497 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f18800a3c40 0x7f18800a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:32:31.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:31.115+0000 7f189a000640 1 --2- 172.21.15.90:0/563069497 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f18800a3c40 0x7f18800a6100 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f189c19b510 tx=0x7f1888005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:32:31.120 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:31.119+0000 7f1893400640 1 -- 172.21.15.90:0/563069497 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7f188c09cfe0 con 0x7f189c103d40 2024-04-02T14:32:31.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:30 smithi090 ceph-mon[31674]: from='client.14598 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T14:32:31.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:30 smithi090 ceph-mon[31674]: pgmap v316: 97 pgs: 97 active+clean; 600 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 872 B/s rd, 6.7 KiB/s wr, 17 op/s 2024-04-02T14:32:31.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:30 smithi090 ceph-mon[31674]: from='client.? 172.21.15.90:0/2875659570' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-02T14:32:31.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:31.322+0000 7f18a1a00640 1 -- 172.21.15.90:0/563069497 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f189c02d5d0 con 0x7f189c103d40 2024-04-02T14:32:31.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:31.324+0000 7f1893400640 1 -- 172.21.15.90:0/563069497 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+22204 (secure 0 0 0) 0x7f188c09c8e0 con 0x7f189c103d40 2024-04-02T14:32:31.327 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-02T14:32:31.327 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":88,"fsid":"6449448a-f0fc-11ee-b647-cb9ed24678a4","created":"2024-04-02T14:23:16.530201+0000","modified":"2024-04-02T14:32:23.517146+0000","last_up_change":"2024-04-02T14:31:23.819732+0000","last_in_change":"2024-04-02T14:31:02.824284+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T14:27:16.758622+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-02T14:32:20.016331+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-02T14:32:21.294367+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a1a6a856-ab44-44bd-bc5e-45c718a70cd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":3635087609},{"type":"v1","addr":"172.21.15.90:6803","nonce":3635087609}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":3635087609},{"type":"v1","addr":"172.21.15.90:6805","nonce":3635087609}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":3635087609},{"type":"v1","addr":"172.21.15.90:6809","nonce":3635087609}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":3635087609},{"type":"v1","addr":"172.21.15.90:6807","nonce":3635087609}]},"public_addr":"172.21.15.90:6803/3635087609","cluster_addr":"172.21.15.90:6805/3635087609","heartbeat_back_addr":"172.21.15.90:6809/3635087609","heartbeat_front_addr":"172.21.15.90:6807/3635087609","state":["exists","up"]},{"osd":1,"uuid":"124139cd-5793-463e-9607-9d9c10164183","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":1994536059},{"type":"v1","addr":"172.21.15.92:6801","nonce":1994536059}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":1994536059},{"type":"v1","addr":"172.21.15.92:6803","nonce":1994536059}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":1994536059},{"type":"v1","addr":"172.21.15.92:6807","nonce":1994536059}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":1994536059},{"type":"v1","addr":"172.21.15.92:6805","nonce":1994536059}]},"public_addr":"172.21.15.92:6801/1994536059","cluster_addr":"172.21.15.92:6803/1994536059","heartbeat_back_addr":"172.21.15.92:6807/1994536059","heartbeat_front_addr":"172.21.15.92:6805/1994536059","state":["exists","up"]},{"osd":2,"uuid":"a61fbef2-9dca-4875-a814-020fff0db0e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":1358472800},{"type":"v1","addr":"172.21.15.177:6801","nonce":1358472800}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":1358472800},{"type":"v1","addr":"172.21.15.177:6803","nonce":1358472800}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":1358472800},{"type":"v1","addr":"172.21.15.177:6807","nonce":1358472800}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":1358472800},{"type":"v1","addr":"172.21.15.177:6805","nonce":1358472800}]},"public_addr":"172.21.15.177:6801/1358472800","cluster_addr":"172.21.15.177:6803/1358472800","heartbeat_back_addr":"172.21.15.177:6807/1358472800","heartbeat_front_addr":"172.21.15.177:6805/1358472800","state":["exists","up"]},{"osd":3,"uuid":"b175a2f4-7859-4eb0-bc1f-fba8c2b92a2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":3114811216},{"type":"v1","addr":"172.21.15.90:6811","nonce":3114811216}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":3114811216},{"type":"v1","addr":"172.21.15.90:6813","nonce":3114811216}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":3114811216},{"type":"v1","addr":"172.21.15.90:6817","nonce":3114811216}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":3114811216},{"type":"v1","addr":"172.21.15.90:6815","nonce":3114811216}]},"public_addr":"172.21.15.90:6811/3114811216","cluster_addr":"172.21.15.90:6813/3114811216","heartbeat_back_addr":"172.21.15.90:6817/3114811216","heartbeat_front_addr":"172.21.15.90:6815/3114811216","state":["exists","up"]},{"osd":4,"uuid":"38cc68ea-c109-4d8d-b77d-1c6f9b56c875","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":2987208879},{"type":"v1","addr":"172.21.15.92:6809","nonce":2987208879}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6810","nonce":2987208879},{"type":"v1","addr":"172.21.15.92:6811","nonce":2987208879}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6814","nonce":2987208879},{"type":"v1","addr":"172.21.15.92:6815","nonce":2987208879}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":2987208879},{"type":"v1","addr":"172.21.15.92:6813","nonce":2987208879}]},"public_addr":"172.21.15.92:6809/2987208879","cluster_addr":"172.21.15.92:6811/2987208879","heartbeat_back_addr":"172.21.15.92:6815/2987208879","heartbeat_front_addr":"172.21.15.92:6813/2987208879","state":["exists","up"]},{"osd":5,"uuid":"2347232e-ac91-466f-8ac9-496f397679ba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":3486820237},{"type":"v1","addr":"172.21.15.177:6809","nonce":3486820237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":3486820237},{"type":"v1","addr":"172.21.15.177:6811","nonce":3486820237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":3486820237},{"type":"v1","addr":"172.21.15.177:6815","nonce":3486820237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":3486820237},{"type":"v1","addr":"172.21.15.177:6813","nonce":3486820237}]},"public_addr":"172.21.15.177:6809/3486820237","cluster_addr":"172.21.15.177:6811/3486820237","heartbeat_back_addr":"172.21.15.177:6815/3486820237","heartbeat_front_addr":"172.21.15.177:6813/3486820237","state":["exists","up"]},{"osd":6,"uuid":"aa72392f-0498-44db-a8d2-eccc89a64871","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":2579716443},{"type":"v1","addr":"172.21.15.90:6819","nonce":2579716443}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6820","nonce":2579716443},{"type":"v1","addr":"172.21.15.90:6821","nonce":2579716443}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":2579716443},{"type":"v1","addr":"172.21.15.90:6825","nonce":2579716443}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2579716443},{"type":"v1","addr":"172.21.15.90:6823","nonce":2579716443}]},"public_addr":"172.21.15.90:6819/2579716443","cluster_addr":"172.21.15.90:6821/2579716443","heartbeat_back_addr":"172.21.15.90:6825/2579716443","heartbeat_front_addr":"172.21.15.90:6823/2579716443","state":["exists","up"]},{"osd":7,"uuid":"931bd105-bc14-4a1e-bfe4-c7ad58fa4b73","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":3371649139},{"type":"v1","addr":"172.21.15.92:6817","nonce":3371649139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":3371649139},{"type":"v1","addr":"172.21.15.92:6819","nonce":3371649139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":3371649139},{"type":"v1","addr":"172.21.15.92:6823","nonce":3371649139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6820","nonce":3371649139},{"type":"v1","addr":"172.21.15.92:6821","nonce":3371649139}]},"public_addr":"172.21.15.92:6817/3371649139","cluster_addr":"172.21.15.92:6819/3371649139","heartbeat_back_addr":"172.21.15.92:6823/3371649139","heartbeat_front_addr":"172.21.15.92:6821/3371649139","state":["exists","up"]},{"osd":8,"uuid":"82624ec2-b382-4886-9f40-7f5f376de3ce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":1459053034},{"type":"v1","addr":"172.21.15.177:6817","nonce":1459053034}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":1459053034},{"type":"v1","addr":"172.21.15.177:6819","nonce":1459053034}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":1459053034},{"type":"v1","addr":"172.21.15.177:6823","nonce":1459053034}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":1459053034},{"type":"v1","addr":"172.21.15.177:6821","nonce":1459053034}]},"public_addr":"172.21.15.177:6817/1459053034","cluster_addr":"172.21.15.177:6819/1459053034","heartbeat_back_addr":"172.21.15.177:6823/1459053034","heartbeat_front_addr":"172.21.15.177:6821/1459053034","state":["exists","up"]},{"osd":9,"uuid":"fbe36e48-3468-42b2-8094-e16552f3cb16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":2592604332},{"type":"v1","addr":"172.21.15.90:6827","nonce":2592604332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":2592604332},{"type":"v1","addr":"172.21.15.90:6829","nonce":2592604332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6832","nonce":2592604332},{"type":"v1","addr":"172.21.15.90:6833","nonce":2592604332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":2592604332},{"type":"v1","addr":"172.21.15.90:6831","nonce":2592604332}]},"public_addr":"172.21.15.90:6827/2592604332","cluster_addr":"172.21.15.90:6829/2592604332","heartbeat_back_addr":"172.21.15.90:6833/2592604332","heartbeat_front_addr":"172.21.15.90:6831/2592604332","state":["exists","up"]},{"osd":10,"uuid":"9240df75-e7e2-47f1-8aae-534b70b3c210","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":1594153142},{"type":"v1","addr":"172.21.15.92:6825","nonce":1594153142}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6826","nonce":1594153142},{"type":"v1","addr":"172.21.15.92:6827","nonce":1594153142}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":1594153142},{"type":"v1","addr":"172.21.15.92:6831","nonce":1594153142}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":1594153142},{"type":"v1","addr":"172.21.15.92:6829","nonce":1594153142}]},"public_addr":"172.21.15.92:6825/1594153142","cluster_addr":"172.21.15.92:6827/1594153142","heartbeat_back_addr":"172.21.15.92:6831/1594153142","heartbeat_front_addr":"172.21.15.92:6829/1594153142","state":["exists","up"]},{"osd":11,"uuid":"83cba9fa-b745-4df1-adda-e86837d4b91a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":83,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1622582126},{"type":"v1","addr":"172.21.15.177:6825","nonce":1622582126}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":1622582126},{"type":"v1","addr":"172.21.15.177:6827","nonce":1622582126}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":1622582126},{"type":"v1","addr":"172.21.15.177:6831","nonce":1622582126}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1622582126},{"type":"v1","addr":"172.21.15.177:6829","nonce":1622582126}]},"public_addr":"172.21.15.177:6825/1622582126","cluster_addr":"172.21.15.177:6827/1622582126","heartbeat_back_addr":"172.21.15.177:6831/1622582126","heartbeat_front_addr":"172.21.15.177:6829/1622582126","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:26:24.290365+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:26:49.405988+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:27:13.639196+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:27:45.603746+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:28:11.019126+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:28:35.084163+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:29:08.064626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:29:33.266133+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:29:57.381816+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:30:30.286847+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:30:56.226279+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:31:20.809611+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.90:0/314641601":"2024-04-03T14:24:34.595561+0000","172.21.15.90:0/1400895868":"2024-04-03T14:24:34.595561+0000","172.21.15.90:6801/129436436":"2024-04-03T14:24:34.595561+0000","172.21.15.90:6801/2372398598":"2024-04-03T14:24:00.362147+0000","172.21.15.90:0/278569338":"2024-04-03T14:24:34.595561+0000","172.21.15.90:0/2321464594":"2024-04-03T14:24:00.362147+0000","172.21.15.90:6800/2372398598":"2024-04-03T14:24:00.362147+0000","172.21.15.90:0/2962656672":"2024-04-03T14:24:00.362147+0000","172.21.15.90:0/126693610":"2024-04-03T14:24:00.362147+0000","172.21.15.90:6800/129436436":"2024-04-03T14:24:34.595561+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T14:32:31.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:31.327+0000 7f18a1a00640 1 -- 172.21.15.90:0/563069497 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f18800a3c40 msgr2=0x7f18800a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:32:31.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:31.327+0000 7f18a1a00640 1 --2- 172.21.15.90:0/563069497 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f18800a3c40 0x7f18800a6100 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f189c19b510 tx=0x7f1888005f90 comp rx=0 tx=0).stop 2024-04-02T14:32:31.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:31.327+0000 7f18a1a00640 1 -- 172.21.15.90:0/563069497 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f189c103d40 msgr2=0x7f189c19a000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:32:31.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:31.327+0000 7f18a1a00640 1 --2- 172.21.15.90:0/563069497 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f189c103d40 0x7f189c19a000 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f188c00e9c0 tx=0x7f188c00ee80 comp rx=0 tx=0).stop 2024-04-02T14:32:31.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:31.327+0000 7f18a1a00640 1 -- 172.21.15.90:0/563069497 shutdown_connections 2024-04-02T14:32:31.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:31.327+0000 7f18a1a00640 1 --2- 172.21.15.90:0/563069497 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f18800a3c40 0x7f18800a6100 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:31.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:31.327+0000 7f18a1a00640 1 --2- 172.21.15.90:0/563069497 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f189c1058f0 0x7f189c1a15b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:31.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:31.327+0000 7f18a1a00640 1 --2- 172.21.15.90:0/563069497 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f189c104f30 0x7f189c19a540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:31.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:31.327+0000 7f18a1a00640 1 --2- 172.21.15.90:0/563069497 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f189c103d40 0x7f189c19a000 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:31.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:31.327+0000 7f18a1a00640 1 -- 172.21.15.90:0/563069497 >> 172.21.15.90:0/563069497 conn(0x7f189c0ff2b0 msgr2=0x7f189c108230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:32:31.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:31.328+0000 7f18a1a00640 1 -- 172.21.15.90:0/563069497 shutdown_connections 2024-04-02T14:32:31.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:31.328+0000 7f18a1a00640 1 -- 172.21.15.90:0/563069497 wait complete. 2024-04-02T14:32:31.350 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-02T14:32:31.351 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-02T14:32:31.360 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:30 smithi092 ceph-mon[36497]: from='client.14598 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T14:32:31.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:30 smithi092 ceph-mon[36497]: pgmap v316: 97 pgs: 97 active+clean; 600 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 872 B/s rd, 6.7 KiB/s wr, 17 op/s 2024-04-02T14:32:31.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:30 smithi092 ceph-mon[36497]: from='client.? 172.21.15.90:0/2875659570' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-02T14:32:31.436 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:30 smithi177 ceph-mon[36431]: from='client.14598 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T14:32:31.436 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:30 smithi177 ceph-mon[36431]: pgmap v316: 97 pgs: 97 active+clean; 600 KiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 872 B/s rd, 6.7 KiB/s wr, 17 op/s 2024-04-02T14:32:31.437 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:30 smithi177 ceph-mon[36431]: from='client.? 172.21.15.90:0/2875659570' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-02T14:32:31.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:31.555+0000 7faa05400640 1 -- 172.21.15.90:0/2766994704 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7faa00104f30 msgr2=0x7faa001053b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:32:31.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:31.555+0000 7faa05400640 1 --2- 172.21.15.90:0/2766994704 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7faa00104f30 0x7faa001053b0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fa9ec009860 tx=0x7fa9ec02bb90 comp rx=0 tx=0).stop 2024-04-02T14:32:31.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:31.555+0000 7faa05400640 1 -- 172.21.15.90:0/2766994704 shutdown_connections 2024-04-02T14:32:31.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:31.555+0000 7faa05400640 1 --2- 172.21.15.90:0/2766994704 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faa001058f0 0x7faa0010c180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:31.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:31.555+0000 7faa05400640 1 --2- 172.21.15.90:0/2766994704 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7faa00104f30 0x7faa001053b0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:31.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:31.555+0000 7faa05400640 1 --2- 172.21.15.90:0/2766994704 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faa00103d40 0x7faa00104160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:31.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:31.555+0000 7faa05400640 1 -- 172.21.15.90:0/2766994704 >> 172.21.15.90:0/2766994704 conn(0x7faa000ff2b0 msgr2=0x7faa001016f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:32:31.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:31.555+0000 7faa05400640 1 -- 172.21.15.90:0/2766994704 shutdown_connections 2024-04-02T14:32:31.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:31.555+0000 7faa05400640 1 -- 172.21.15.90:0/2766994704 wait complete. 2024-04-02T14:32:31.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:31.556+0000 7faa05400640 1 Processor -- start 2024-04-02T14:32:31.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:31.556+0000 7faa05400640 1 -- start start 2024-04-02T14:32:31.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:31.556+0000 7faa05400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faa00103d40 0x7faa0019a090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:32:31.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:31.556+0000 7faa05400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7faa00104f30 0x7faa0019a5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:32:31.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:31.556+0000 7faa05400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faa001058f0 0x7faa001a1640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:32:31.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:31.556+0000 7faa05400640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7faa0010f9c0 con 0x7faa001058f0 2024-04-02T14:32:31.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:31.556+0000 7faa05400640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7faa0010f840 con 0x7faa00104f30 2024-04-02T14:32:31.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:31.556+0000 7faa05400640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7faa0010fb40 con 0x7faa00103d40 2024-04-02T14:32:31.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:31.557+0000 7fa9ff200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faa001058f0 0x7faa001a1640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:32:31.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:31.557+0000 7fa9fde00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7faa00104f30 0x7faa0019a5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:32:31.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:31.557+0000 7fa9ff200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faa001058f0 0x7faa001a1640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:49466/0 (socket says 172.21.15.90:49466) 2024-04-02T14:32:31.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:31.557+0000 7fa9ff200640 1 -- 172.21.15.90:0/583723366 learned_addr learned my addr 172.21.15.90:0/583723366 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:32:31.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:31.557+0000 7fa9fde00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7faa00104f30 0x7faa0019a5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:45082/0 (socket says 172.21.15.90:45082) 2024-04-02T14:32:31.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:31.557+0000 7fa9fe800640 1 --2- 172.21.15.90:0/583723366 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faa00103d40 0x7faa0019a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:32:31.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:31.557+0000 7fa9fde00640 1 -- 172.21.15.90:0/583723366 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faa00103d40 msgr2=0x7faa0019a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:32:31.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:31.557+0000 7fa9fde00640 1 --2- 172.21.15.90:0/583723366 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faa00103d40 0x7faa0019a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:31.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:31.557+0000 7fa9fde00640 1 -- 172.21.15.90:0/583723366 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faa001058f0 msgr2=0x7faa001a1640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:32:31.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:31.557+0000 7fa9fde00640 1 --2- 172.21.15.90:0/583723366 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faa001058f0 0x7faa001a1640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:31.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:31.557+0000 7fa9fde00640 1 -- 172.21.15.90:0/583723366 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa001a1cb0 con 0x7faa00104f30 2024-04-02T14:32:31.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:31.557+0000 7fa9ff200640 1 --2- 172.21.15.90:0/583723366 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faa001058f0 0x7faa001a1640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:32:31.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:31.557+0000 7fa9fde00640 1 --2- 172.21.15.90:0/583723366 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7faa00104f30 0x7faa0019a5d0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fa9ec02c0d0 tx=0x7fa9ec032430 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:32:31.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:31.558+0000 7fa9f7400640 1 -- 172.21.15.90:0/583723366 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9ec041020 con 0x7faa00104f30 2024-04-02T14:32:31.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:31.558+0000 7fa9f7400640 1 -- 172.21.15.90:0/583723366 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa9ec02ca10 con 0x7faa00104f30 2024-04-02T14:32:31.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:31.558+0000 7faa05400640 1 -- 172.21.15.90:0/583723366 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa001a1f40 con 0x7faa00104f30 2024-04-02T14:32:31.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:31.558+0000 7fa9f7400640 1 -- 172.21.15.90:0/583723366 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9ec039040 con 0x7faa00104f30 2024-04-02T14:32:31.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:31.558+0000 7faa05400640 1 -- 172.21.15.90:0/583723366 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa001a23a0 con 0x7faa00104f30 2024-04-02T14:32:31.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:31.559+0000 7faa05400640 1 -- 172.21.15.90:0/583723366 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faa0010b0e0 con 0x7faa00104f30 2024-04-02T14:32:31.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:31.562+0000 7fa9f7400640 1 -- 172.21.15.90:0/583723366 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fa9ec044020 con 0x7faa00104f30 2024-04-02T14:32:31.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:31.563+0000 7fa9f7400640 1 --2- 172.21.15.90:0/583723366 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fa9e40a3c40 0x7fa9e40a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:32:31.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:31.563+0000 7fa9f7400640 1 -- 172.21.15.90:0/583723366 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 7979+0+0 (secure 0 0 0) 0x7fa9ec0f1fa0 con 0x7faa00104f30 2024-04-02T14:32:31.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:31.563+0000 7fa9fe800640 1 --2- 172.21.15.90:0/583723366 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fa9e40a3c40 0x7fa9e40a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:32:31.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:31.563+0000 7fa9fe800640 1 --2- 172.21.15.90:0/583723366 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fa9e40a3c40 0x7fa9e40a6100 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7faa00104d90 tx=0x7fa9f0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:32:31.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:31.565+0000 7fa9f7400640 1 -- 172.21.15.90:0/583723366 <== mon.2 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7fa9ec04b020 con 0x7faa00104f30 2024-04-02T14:32:31.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:31.765+0000 7faa05400640 1 -- 172.21.15.90:0/583723366 --> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7faa00061f40 con 0x7fa9e40a3c40 2024-04-02T14:32:31.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:31.791+0000 7fa9f7400640 1 -- 172.21.15.90:0/583723366 <== mgr.14152 v2:172.21.15.90:6800/4266054817 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258734 (secure 0 0 0) 0x7faa00061f40 con 0x7fa9e40a3c40 2024-04-02T14:32:31.799 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-02T14:32:31.801 INFO:teuthology.orchestra.run.smithi090.stderr:dumped all 2024-04-02T14:32:31.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:31.797+0000 7faa05400640 1 -- 172.21.15.90:0/583723366 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fa9e40a3c40 msgr2=0x7fa9e40a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:32:31.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:31.797+0000 7faa05400640 1 --2- 172.21.15.90:0/583723366 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fa9e40a3c40 0x7fa9e40a6100 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7faa00104d90 tx=0x7fa9f0005f90 comp rx=0 tx=0).stop 2024-04-02T14:32:31.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:31.797+0000 7faa05400640 1 -- 172.21.15.90:0/583723366 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7faa00104f30 msgr2=0x7faa0019a5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:32:31.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:31.797+0000 7faa05400640 1 --2- 172.21.15.90:0/583723366 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7faa00104f30 0x7faa0019a5d0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fa9ec02c0d0 tx=0x7fa9ec032430 comp rx=0 tx=0).stop 2024-04-02T14:32:31.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:31.798+0000 7faa05400640 1 -- 172.21.15.90:0/583723366 shutdown_connections 2024-04-02T14:32:31.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:31.798+0000 7faa05400640 1 --2- 172.21.15.90:0/583723366 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fa9e40a3c40 0x7fa9e40a6100 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:31.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:31.798+0000 7faa05400640 1 --2- 172.21.15.90:0/583723366 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faa001058f0 0x7faa001a1640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:31.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:31.798+0000 7faa05400640 1 --2- 172.21.15.90:0/583723366 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7faa00104f30 0x7faa0019a5d0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:31.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:31.798+0000 7faa05400640 1 --2- 172.21.15.90:0/583723366 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faa00103d40 0x7faa0019a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:31.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:31.798+0000 7faa05400640 1 -- 172.21.15.90:0/583723366 >> 172.21.15.90:0/583723366 conn(0x7faa000ff2b0 msgr2=0x7faa001082b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:32:31.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:31.798+0000 7faa05400640 1 -- 172.21.15.90:0/583723366 shutdown_connections 2024-04-02T14:32:31.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:31.798+0000 7faa05400640 1 -- 172.21.15.90:0/583723366 wait complete. 2024-04-02T14:32:31.825 INFO:teuthology.orchestra.run.smithi090.stdout:{"pg_ready":true,"pg_map":{"version":316,"stamp":"2024-04-02T14:32:30.732293+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":614743,"num_objects":62,"num_object_clones":0,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":143,"num_read_kb":126,"num_write":321,"num_write_kb":2993,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":256,"ondisk_log_size":256,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":283,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":403912,"kb_used_data":13132,"kb_used_omap":18,"kb_used_meta":390637,"kb_avail":1124537912,"statfs":{"total":1151940427776,"available":1151526821888,"internally_reserved":0,"allocated":13447168,"data_stored":7282712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":400012602},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":24375,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":140,"num_write_kb":55,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.213893"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:25.547264+0000","last_change":"2024-04-02T14:32:22.558913+0000","last_active":"2024-04-02T14:32:25.547264+0000","last_peered":"2024-04-02T14:32:25.547264+0000","last_clean":"2024-04-02T14:32:25.547264+0000","last_became_active":"2024-04-02T14:32:22.558606+0000","last_became_peered":"2024-04-02T14:32:22.558606+0000","last_unstale":"2024-04-02T14:32:25.547264+0000","last_undegraded":"2024-04-02T14:32:25.547264+0000","last_fullsized":"2024-04-02T14:32:25.547264+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:59:49.205499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:23.523775+0000","last_change":"2024-04-02T14:32:22.551898+0000","last_active":"2024-04-02T14:32:23.523775+0000","last_peered":"2024-04-02T14:32:23.523775+0000","last_clean":"2024-04-02T14:32:23.523775+0000","last_became_active":"2024-04-02T14:32:22.549016+0000","last_became_peered":"2024-04-02T14:32:22.549016+0000","last_unstale":"2024-04-02T14:32:23.523775+0000","last_undegraded":"2024-04-02T14:32:23.523775+0000","last_fullsized":"2024-04-02T14:32:23.523775+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:38:03.529505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:23.523891+0000","last_change":"2024-04-02T14:32:22.556214+0000","last_active":"2024-04-02T14:32:23.523891+0000","last_peered":"2024-04-02T14:32:23.523891+0000","last_clean":"2024-04-02T14:32:23.523891+0000","last_became_active":"2024-04-02T14:32:22.555686+0000","last_became_peered":"2024-04-02T14:32:22.555686+0000","last_unstale":"2024-04-02T14:32:23.523891+0000","last_undegraded":"2024-04-02T14:32:23.523891+0000","last_fullsized":"2024-04-02T14:32:23.523891+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:06:40.161203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:25.547236+0000","last_change":"2024-04-02T14:32:22.537830+0000","last_active":"2024-04-02T14:32:25.547236+0000","last_peered":"2024-04-02T14:32:25.547236+0000","last_clean":"2024-04-02T14:32:25.547236+0000","last_became_active":"2024-04-02T14:32:22.536860+0000","last_became_peered":"2024-04-02T14:32:22.536860+0000","last_unstale":"2024-04-02T14:32:25.547236+0000","last_undegraded":"2024-04-02T14:32:25.547236+0000","last_fullsized":"2024-04-02T14:32:25.547236+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T01:07:57.244177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:24.857848+0000","last_change":"2024-04-02T14:32:22.549550+0000","last_active":"2024-04-02T14:32:24.857848+0000","last_peered":"2024-04-02T14:32:24.857848+0000","last_clean":"2024-04-02T14:32:24.857848+0000","last_became_active":"2024-04-02T14:32:22.549177+0000","last_became_peered":"2024-04-02T14:32:22.549177+0000","last_unstale":"2024-04-02T14:32:24.857848+0000","last_undegraded":"2024-04-02T14:32:24.857848+0000","last_fullsized":"2024-04-02T14:32:24.857848+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:31:49.175583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:23.521965+0000","last_change":"2024-04-02T14:32:22.548786+0000","last_active":"2024-04-02T14:32:23.521965+0000","last_peered":"2024-04-02T14:32:23.521965+0000","last_clean":"2024-04-02T14:32:23.521965+0000","last_became_active":"2024-04-02T14:32:22.548426+0000","last_became_peered":"2024-04-02T14:32:22.548426+0000","last_unstale":"2024-04-02T14:32:23.521965+0000","last_undegraded":"2024-04-02T14:32:23.521965+0000","last_fullsized":"2024-04-02T14:32:23.521965+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:21:58.623016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:25.416920+0000","last_change":"2024-04-02T14:32:22.546861+0000","last_active":"2024-04-02T14:32:25.416920+0000","last_peered":"2024-04-02T14:32:25.416920+0000","last_clean":"2024-04-02T14:32:25.416920+0000","last_became_active":"2024-04-02T14:32:22.546477+0000","last_became_peered":"2024-04-02T14:32:22.546477+0000","last_unstale":"2024-04-02T14:32:25.416920+0000","last_undegraded":"2024-04-02T14:32:25.416920+0000","last_fullsized":"2024-04-02T14:32:25.416920+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:59:17.133714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:24.858949+0000","last_change":"2024-04-02T14:32:22.554287+0000","last_active":"2024-04-02T14:32:24.858949+0000","last_peered":"2024-04-02T14:32:24.858949+0000","last_clean":"2024-04-02T14:32:24.858949+0000","last_became_active":"2024-04-02T14:32:22.553491+0000","last_became_peered":"2024-04-02T14:32:22.553491+0000","last_unstale":"2024-04-02T14:32:24.858949+0000","last_undegraded":"2024-04-02T14:32:24.858949+0000","last_fullsized":"2024-04-02T14:32:24.858949+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:51:30.785380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:25.547093+0000","last_change":"2024-04-02T14:32:22.538158+0000","last_active":"2024-04-02T14:32:25.547093+0000","last_peered":"2024-04-02T14:32:25.547093+0000","last_clean":"2024-04-02T14:32:25.547093+0000","last_became_active":"2024-04-02T14:32:22.537796+0000","last_became_peered":"2024-04-02T14:32:22.537796+0000","last_unstale":"2024-04-02T14:32:25.547093+0000","last_undegraded":"2024-04-02T14:32:25.547093+0000","last_fullsized":"2024-04-02T14:32:25.547093+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:09:48.054965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:23.522159+0000","last_change":"2024-04-02T14:32:22.535225+0000","last_active":"2024-04-02T14:32:23.522159+0000","last_peered":"2024-04-02T14:32:23.522159+0000","last_clean":"2024-04-02T14:32:23.522159+0000","last_became_active":"2024-04-02T14:32:22.534856+0000","last_became_peered":"2024-04-02T14:32:22.534856+0000","last_unstale":"2024-04-02T14:32:23.522159+0000","last_undegraded":"2024-04-02T14:32:23.522159+0000","last_fullsized":"2024-04-02T14:32:23.522159+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:22:11.118918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:25.417206+0000","last_change":"2024-04-02T14:32:22.539138+0000","last_active":"2024-04-02T14:32:25.417206+0000","last_peered":"2024-04-02T14:32:25.417206+0000","last_clean":"2024-04-02T14:32:25.417206+0000","last_became_active":"2024-04-02T14:32:22.538762+0000","last_became_peered":"2024-04-02T14:32:22.538762+0000","last_unstale":"2024-04-02T14:32:25.417206+0000","last_undegraded":"2024-04-02T14:32:25.417206+0000","last_fullsized":"2024-04-02T14:32:25.417206+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:18:59.174339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:25.547217+0000","last_change":"2024-04-02T14:32:22.536383+0000","last_active":"2024-04-02T14:32:25.547217+0000","last_peered":"2024-04-02T14:32:25.547217+0000","last_clean":"2024-04-02T14:32:25.547217+0000","last_became_active":"2024-04-02T14:32:22.536043+0000","last_became_peered":"2024-04-02T14:32:22.536043+0000","last_unstale":"2024-04-02T14:32:25.547217+0000","last_undegraded":"2024-04-02T14:32:25.547217+0000","last_fullsized":"2024-04-02T14:32:25.547217+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:14:48.628684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:23.523530+0000","last_change":"2024-04-02T14:32:22.552512+0000","last_active":"2024-04-02T14:32:23.523530+0000","last_peered":"2024-04-02T14:32:23.523530+0000","last_clean":"2024-04-02T14:32:23.523530+0000","last_became_active":"2024-04-02T14:32:22.549502+0000","last_became_peered":"2024-04-02T14:32:22.549502+0000","last_unstale":"2024-04-02T14:32:23.523530+0000","last_undegraded":"2024-04-02T14:32:23.523530+0000","last_fullsized":"2024-04-02T14:32:23.523530+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:37:45.609280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:23.522812+0000","last_change":"2024-04-02T14:32:22.555074+0000","last_active":"2024-04-02T14:32:23.522812+0000","last_peered":"2024-04-02T14:32:23.522812+0000","last_clean":"2024-04-02T14:32:23.522812+0000","last_became_active":"2024-04-02T14:32:22.554422+0000","last_became_peered":"2024-04-02T14:32:22.554422+0000","last_unstale":"2024-04-02T14:32:23.522812+0000","last_undegraded":"2024-04-02T14:32:23.522812+0000","last_fullsized":"2024-04-02T14:32:23.522812+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:27:49.536941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:23.522401+0000","last_change":"2024-04-02T14:32:22.550322+0000","last_active":"2024-04-02T14:32:23.522401+0000","last_peered":"2024-04-02T14:32:23.522401+0000","last_clean":"2024-04-02T14:32:23.522401+0000","last_became_active":"2024-04-02T14:32:22.548710+0000","last_became_peered":"2024-04-02T14:32:22.548710+0000","last_unstale":"2024-04-02T14:32:23.522401+0000","last_undegraded":"2024-04-02T14:32:23.522401+0000","last_fullsized":"2024-04-02T14:32:23.522401+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:19:09.226556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:25.546173+0000","last_change":"2024-04-02T14:32:22.558705+0000","last_active":"2024-04-02T14:32:25.546173+0000","last_peered":"2024-04-02T14:32:25.546173+0000","last_clean":"2024-04-02T14:32:25.546173+0000","last_became_active":"2024-04-02T14:32:22.558309+0000","last_became_peered":"2024-04-02T14:32:22.558309+0000","last_unstale":"2024-04-02T14:32:25.546173+0000","last_undegraded":"2024-04-02T14:32:25.546173+0000","last_fullsized":"2024-04-02T14:32:25.546173+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:52:23.929627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:25.547427+0000","last_change":"2024-04-02T14:32:22.551211+0000","last_active":"2024-04-02T14:32:25.547427+0000","last_peered":"2024-04-02T14:32:25.547427+0000","last_clean":"2024-04-02T14:32:25.547427+0000","last_became_active":"2024-04-02T14:32:22.550882+0000","last_became_peered":"2024-04-02T14:32:22.550882+0000","last_unstale":"2024-04-02T14:32:25.547427+0000","last_undegraded":"2024-04-02T14:32:25.547427+0000","last_fullsized":"2024-04-02T14:32:25.547427+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:54:54.533063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:23.879656+0000","last_change":"2024-04-02T14:32:22.550692+0000","last_active":"2024-04-02T14:32:23.879656+0000","last_peered":"2024-04-02T14:32:23.879656+0000","last_clean":"2024-04-02T14:32:23.879656+0000","last_became_active":"2024-04-02T14:32:22.550292+0000","last_became_peered":"2024-04-02T14:32:22.550292+0000","last_unstale":"2024-04-02T14:32:23.879656+0000","last_undegraded":"2024-04-02T14:32:23.879656+0000","last_fullsized":"2024-04-02T14:32:23.879656+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:14:42.317649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:25.546874+0000","last_change":"2024-04-02T14:32:22.553042+0000","last_active":"2024-04-02T14:32:25.546874+0000","last_peered":"2024-04-02T14:32:25.546874+0000","last_clean":"2024-04-02T14:32:25.546874+0000","last_became_active":"2024-04-02T14:32:22.552010+0000","last_became_peered":"2024-04-02T14:32:22.552010+0000","last_unstale":"2024-04-02T14:32:25.546874+0000","last_undegraded":"2024-04-02T14:32:25.546874+0000","last_fullsized":"2024-04-02T14:32:25.546874+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:31:13.947662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"88'11","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:27.943801+0000","last_change":"2024-04-02T14:32:21.841348+0000","last_active":"2024-04-02T14:32:27.943801+0000","last_peered":"2024-04-02T14:32:27.943801+0000","last_clean":"2024-04-02T14:32:27.943801+0000","last_became_active":"2024-04-02T14:32:21.839947+0000","last_became_peered":"2024-04-02T14:32:21.839947+0000","last_unstale":"2024-04-02T14:32:27.943801+0000","last_undegraded":"2024-04-02T14:32:27.943801+0000","last_fullsized":"2024-04-02T14:32:27.943801+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:03:49.484343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:24.859947+0000","last_change":"2024-04-02T14:32:22.549526+0000","last_active":"2024-04-02T14:32:24.859947+0000","last_peered":"2024-04-02T14:32:24.859947+0000","last_clean":"2024-04-02T14:32:24.859947+0000","last_became_active":"2024-04-02T14:32:22.548250+0000","last_became_peered":"2024-04-02T14:32:22.548250+0000","last_unstale":"2024-04-02T14:32:24.859947+0000","last_undegraded":"2024-04-02T14:32:24.859947+0000","last_fullsized":"2024-04-02T14:32:24.859947+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:35:34.866803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"88'2","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:24.860144+0000","last_change":"2024-04-02T14:32:21.865084+0000","last_active":"2024-04-02T14:32:24.860144+0000","last_peered":"2024-04-02T14:32:24.860144+0000","last_clean":"2024-04-02T14:32:24.860144+0000","last_became_active":"2024-04-02T14:32:21.864718+0000","last_became_peered":"2024-04-02T14:32:21.864718+0000","last_unstale":"2024-04-02T14:32:24.860144+0000","last_undegraded":"2024-04-02T14:32:24.860144+0000","last_fullsized":"2024-04-02T14:32:24.860144+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:10:50.611362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:23.524474+0000","last_change":"2024-04-02T14:32:22.549443+0000","last_active":"2024-04-02T14:32:23.524474+0000","last_peered":"2024-04-02T14:32:23.524474+0000","last_clean":"2024-04-02T14:32:23.524474+0000","last_became_active":"2024-04-02T14:32:22.549059+0000","last_became_peered":"2024-04-02T14:32:22.549059+0000","last_unstale":"2024-04-02T14:32:23.524474+0000","last_undegraded":"2024-04-02T14:32:23.524474+0000","last_fullsized":"2024-04-02T14:32:23.524474+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:04:21.489864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"87'2","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:24.858192+0000","last_change":"2024-04-02T14:32:21.841368+0000","last_active":"2024-04-02T14:32:24.858192+0000","last_peered":"2024-04-02T14:32:24.858192+0000","last_clean":"2024-04-02T14:32:24.858192+0000","last_became_active":"2024-04-02T14:32:21.840963+0000","last_became_peered":"2024-04-02T14:32:21.840963+0000","last_unstale":"2024-04-02T14:32:24.858192+0000","last_undegraded":"2024-04-02T14:32:24.858192+0000","last_fullsized":"2024-04-02T14:32:24.858192+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:36:40.549720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:25.546449+0000","last_change":"2024-04-02T14:32:22.549753+0000","last_active":"2024-04-02T14:32:25.546449+0000","last_peered":"2024-04-02T14:32:25.546449+0000","last_clean":"2024-04-02T14:32:25.546449+0000","last_became_active":"2024-04-02T14:32:22.549354+0000","last_became_peered":"2024-04-02T14:32:22.549354+0000","last_unstale":"2024-04-02T14:32:25.546449+0000","last_undegraded":"2024-04-02T14:32:25.546449+0000","last_fullsized":"2024-04-02T14:32:25.546449+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T02:17:09.650946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"87'2","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:24.858050+0000","last_change":"2024-04-02T14:32:21.842766+0000","last_active":"2024-04-02T14:32:24.858050+0000","last_peered":"2024-04-02T14:32:24.858050+0000","last_clean":"2024-04-02T14:32:24.858050+0000","last_became_active":"2024-04-02T14:32:21.842410+0000","last_became_peered":"2024-04-02T14:32:21.842410+0000","last_unstale":"2024-04-02T14:32:24.858050+0000","last_undegraded":"2024-04-02T14:32:24.858050+0000","last_fullsized":"2024-04-02T14:32:24.858050+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:49:31.651127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:25.547188+0000","last_change":"2024-04-02T14:32:22.538506+0000","last_active":"2024-04-02T14:32:25.547188+0000","last_peered":"2024-04-02T14:32:25.547188+0000","last_clean":"2024-04-02T14:32:25.547188+0000","last_became_active":"2024-04-02T14:32:22.538168+0000","last_became_peered":"2024-04-02T14:32:22.538168+0000","last_unstale":"2024-04-02T14:32:25.547188+0000","last_undegraded":"2024-04-02T14:32:25.547188+0000","last_fullsized":"2024-04-02T14:32:25.547188+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:30:04.746382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"88'5","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:25.925554+0000","last_change":"2024-04-02T14:32:21.844180+0000","last_active":"2024-04-02T14:32:25.925554+0000","last_peered":"2024-04-02T14:32:25.925554+0000","last_clean":"2024-04-02T14:32:25.925554+0000","last_became_active":"2024-04-02T14:32:21.843840+0000","last_became_peered":"2024-04-02T14:32:21.843840+0000","last_unstale":"2024-04-02T14:32:25.925554+0000","last_undegraded":"2024-04-02T14:32:25.925554+0000","last_fullsized":"2024-04-02T14:32:25.925554+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T01:56:10.767946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:24.857911+0000","last_change":"2024-04-02T14:32:22.548453+0000","last_active":"2024-04-02T14:32:24.857911+0000","last_peered":"2024-04-02T14:32:24.857911+0000","last_clean":"2024-04-02T14:32:24.857911+0000","last_became_active":"2024-04-02T14:32:22.548110+0000","last_became_peered":"2024-04-02T14:32:22.548110+0000","last_unstale":"2024-04-02T14:32:24.857911+0000","last_undegraded":"2024-04-02T14:32:24.857911+0000","last_fullsized":"2024-04-02T14:32:24.857911+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:57:13.067425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"88'7","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:27.946738+0000","last_change":"2024-04-02T14:32:21.846528+0000","last_active":"2024-04-02T14:32:27.946738+0000","last_peered":"2024-04-02T14:32:27.946738+0000","last_clean":"2024-04-02T14:32:27.946738+0000","last_became_active":"2024-04-02T14:32:21.846152+0000","last_became_peered":"2024-04-02T14:32:21.846152+0000","last_unstale":"2024-04-02T14:32:27.946738+0000","last_undegraded":"2024-04-02T14:32:27.946738+0000","last_fullsized":"2024-04-02T14:32:27.946738+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:58:09.286256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:25.547865+0000","last_change":"2024-04-02T14:32:22.546730+0000","last_active":"2024-04-02T14:32:25.547865+0000","last_peered":"2024-04-02T14:32:25.547865+0000","last_clean":"2024-04-02T14:32:25.547865+0000","last_became_active":"2024-04-02T14:32:22.546342+0000","last_became_peered":"2024-04-02T14:32:22.546342+0000","last_unstale":"2024-04-02T14:32:25.547865+0000","last_undegraded":"2024-04-02T14:32:25.547865+0000","last_fullsized":"2024-04-02T14:32:25.547865+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:38:37.616382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:23.522699+0000","last_change":"2024-04-02T14:32:21.853316+0000","last_active":"2024-04-02T14:32:23.522699+0000","last_peered":"2024-04-02T14:32:23.522699+0000","last_clean":"2024-04-02T14:32:23.522699+0000","last_became_active":"2024-04-02T14:32:21.852912+0000","last_became_peered":"2024-04-02T14:32:21.852912+0000","last_unstale":"2024-04-02T14:32:23.522699+0000","last_undegraded":"2024-04-02T14:32:23.522699+0000","last_fullsized":"2024-04-02T14:32:23.522699+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T02:30:44.601900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:24.858700+0000","last_change":"2024-04-02T14:32:22.558913+0000","last_active":"2024-04-02T14:32:24.858700+0000","last_peered":"2024-04-02T14:32:24.858700+0000","last_clean":"2024-04-02T14:32:24.858700+0000","last_became_active":"2024-04-02T14:32:22.558511+0000","last_became_peered":"2024-04-02T14:32:22.558511+0000","last_unstale":"2024-04-02T14:32:24.858700+0000","last_undegraded":"2024-04-02T14:32:24.858700+0000","last_fullsized":"2024-04-02T14:32:24.858700+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:48:36.449086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"88'1","reported_seq":16,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:25.546542+0000","last_change":"2024-04-02T14:32:21.844081+0000","last_active":"2024-04-02T14:32:25.546542+0000","last_peered":"2024-04-02T14:32:25.546542+0000","last_clean":"2024-04-02T14:32:25.546542+0000","last_became_active":"2024-04-02T14:32:21.843788+0000","last_became_peered":"2024-04-02T14:32:21.843788+0000","last_unstale":"2024-04-02T14:32:25.546542+0000","last_undegraded":"2024-04-02T14:32:25.546542+0000","last_fullsized":"2024-04-02T14:32:25.546542+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:19:02.184229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:23.879234+0000","last_change":"2024-04-02T14:32:22.556193+0000","last_active":"2024-04-02T14:32:23.879234+0000","last_peered":"2024-04-02T14:32:23.879234+0000","last_clean":"2024-04-02T14:32:23.879234+0000","last_became_active":"2024-04-02T14:32:22.555787+0000","last_became_peered":"2024-04-02T14:32:22.555787+0000","last_unstale":"2024-04-02T14:32:23.879234+0000","last_undegraded":"2024-04-02T14:32:23.879234+0000","last_fullsized":"2024-04-02T14:32:23.879234+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T02:02:13.730941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"88'4","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:25.926039+0000","last_change":"2024-04-02T14:32:21.840651+0000","last_active":"2024-04-02T14:32:25.926039+0000","last_peered":"2024-04-02T14:32:25.926039+0000","last_clean":"2024-04-02T14:32:25.926039+0000","last_became_active":"2024-04-02T14:32:21.840344+0000","last_became_peered":"2024-04-02T14:32:21.840344+0000","last_unstale":"2024-04-02T14:32:25.926039+0000","last_undegraded":"2024-04-02T14:32:25.926039+0000","last_fullsized":"2024-04-02T14:32:25.926039+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:49:22.384224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:25.546658+0000","last_change":"2024-04-02T14:32:22.548245+0000","last_active":"2024-04-02T14:32:25.546658+0000","last_peered":"2024-04-02T14:32:25.546658+0000","last_clean":"2024-04-02T14:32:25.546658+0000","last_became_active":"2024-04-02T14:32:22.547853+0000","last_became_peered":"2024-04-02T14:32:22.547853+0000","last_unstale":"2024-04-02T14:32:25.546658+0000","last_undegraded":"2024-04-02T14:32:25.546658+0000","last_fullsized":"2024-04-02T14:32:25.546658+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:02:43.036944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"88'2","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:25.922832+0000","last_change":"2024-04-02T14:32:21.842096+0000","last_active":"2024-04-02T14:32:25.922832+0000","last_peered":"2024-04-02T14:32:25.922832+0000","last_clean":"2024-04-02T14:32:25.922832+0000","last_became_active":"2024-04-02T14:32:21.841646+0000","last_became_peered":"2024-04-02T14:32:21.841646+0000","last_unstale":"2024-04-02T14:32:25.922832+0000","last_undegraded":"2024-04-02T14:32:25.922832+0000","last_fullsized":"2024-04-02T14:32:25.922832+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:04:08.149785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:23.524303+0000","last_change":"2024-04-02T14:32:22.557694+0000","last_active":"2024-04-02T14:32:23.524303+0000","last_peered":"2024-04-02T14:32:23.524303+0000","last_clean":"2024-04-02T14:32:23.524303+0000","last_became_active":"2024-04-02T14:32:22.557202+0000","last_became_peered":"2024-04-02T14:32:22.557202+0000","last_unstale":"2024-04-02T14:32:23.524303+0000","last_undegraded":"2024-04-02T14:32:23.524303+0000","last_fullsized":"2024-04-02T14:32:23.524303+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:35:41.471557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"88'4","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:25.547372+0000","last_change":"2024-04-02T14:32:21.864366+0000","last_active":"2024-04-02T14:32:25.547372+0000","last_peered":"2024-04-02T14:32:25.547372+0000","last_clean":"2024-04-02T14:32:25.547372+0000","last_became_active":"2024-04-02T14:32:21.864006+0000","last_became_peered":"2024-04-02T14:32:21.864006+0000","last_unstale":"2024-04-02T14:32:25.547372+0000","last_undegraded":"2024-04-02T14:32:25.547372+0000","last_fullsized":"2024-04-02T14:32:25.547372+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:08:43.176061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:25.546568+0000","last_change":"2024-04-02T14:32:22.541562+0000","last_active":"2024-04-02T14:32:25.546568+0000","last_peered":"2024-04-02T14:32:25.546568+0000","last_clean":"2024-04-02T14:32:25.546568+0000","last_became_active":"2024-04-02T14:32:22.541209+0000","last_became_peered":"2024-04-02T14:32:22.541209+0000","last_unstale":"2024-04-02T14:32:25.546568+0000","last_undegraded":"2024-04-02T14:32:25.546568+0000","last_fullsized":"2024-04-02T14:32:25.546568+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:49:40.310528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"88'17","reported_seq":32,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:28.968702+0000","last_change":"2024-04-02T14:32:21.839406+0000","last_active":"2024-04-02T14:32:28.968702+0000","last_peered":"2024-04-02T14:32:28.968702+0000","last_clean":"2024-04-02T14:32:28.968702+0000","last_became_active":"2024-04-02T14:32:21.839080+0000","last_became_peered":"2024-04-02T14:32:21.839080+0000","last_unstale":"2024-04-02T14:32:28.968702+0000","last_undegraded":"2024-04-02T14:32:28.968702+0000","last_fullsized":"2024-04-02T14:32:28.968702+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:33:43.128536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20895,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:24.858143+0000","last_change":"2024-04-02T14:32:22.535716+0000","last_active":"2024-04-02T14:32:24.858143+0000","last_peered":"2024-04-02T14:32:24.858143+0000","last_clean":"2024-04-02T14:32:24.858143+0000","last_became_active":"2024-04-02T14:32:22.534738+0000","last_became_peered":"2024-04-02T14:32:22.534738+0000","last_unstale":"2024-04-02T14:32:24.858143+0000","last_undegraded":"2024-04-02T14:32:24.858143+0000","last_fullsized":"2024-04-02T14:32:24.858143+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T01:25:07.763915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"88'4","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:25.927483+0000","last_change":"2024-04-02T14:32:21.854582+0000","last_active":"2024-04-02T14:32:25.927483+0000","last_peered":"2024-04-02T14:32:25.927483+0000","last_clean":"2024-04-02T14:32:25.927483+0000","last_became_active":"2024-04-02T14:32:21.854226+0000","last_became_peered":"2024-04-02T14:32:21.854226+0000","last_unstale":"2024-04-02T14:32:25.927483+0000","last_undegraded":"2024-04-02T14:32:25.927483+0000","last_fullsized":"2024-04-02T14:32:25.927483+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:30:02.289163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:23.522911+0000","last_change":"2024-04-02T14:32:22.555901+0000","last_active":"2024-04-02T14:32:23.522911+0000","last_peered":"2024-04-02T14:32:23.522911+0000","last_clean":"2024-04-02T14:32:23.522911+0000","last_became_active":"2024-04-02T14:32:22.555158+0000","last_became_peered":"2024-04-02T14:32:22.555158+0000","last_unstale":"2024-04-02T14:32:23.522911+0000","last_undegraded":"2024-04-02T14:32:23.522911+0000","last_fullsized":"2024-04-02T14:32:23.522911+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:04:28.766185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"88'3","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:25.927454+0000","last_change":"2024-04-02T14:32:21.855982+0000","last_active":"2024-04-02T14:32:25.927454+0000","last_peered":"2024-04-02T14:32:25.927454+0000","last_clean":"2024-04-02T14:32:25.927454+0000","last_became_active":"2024-04-02T14:32:21.855591+0000","last_became_peered":"2024-04-02T14:32:21.855591+0000","last_unstale":"2024-04-02T14:32:25.927454+0000","last_undegraded":"2024-04-02T14:32:25.927454+0000","last_fullsized":"2024-04-02T14:32:25.927454+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:28:51.806779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:25.417261+0000","last_change":"2024-04-02T14:32:22.549889+0000","last_active":"2024-04-02T14:32:25.417261+0000","last_peered":"2024-04-02T14:32:25.417261+0000","last_clean":"2024-04-02T14:32:25.417261+0000","last_became_active":"2024-04-02T14:32:22.549479+0000","last_became_peered":"2024-04-02T14:32:22.549479+0000","last_unstale":"2024-04-02T14:32:25.417261+0000","last_undegraded":"2024-04-02T14:32:25.417261+0000","last_fullsized":"2024-04-02T14:32:25.417261+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:29:25.937576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"88'6","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:28.390137+0000","last_change":"2024-04-02T14:32:21.838165+0000","last_active":"2024-04-02T14:32:28.390137+0000","last_peered":"2024-04-02T14:32:28.390137+0000","last_clean":"2024-04-02T14:32:28.390137+0000","last_became_active":"2024-04-02T14:32:21.837808+0000","last_became_peered":"2024-04-02T14:32:21.837808+0000","last_unstale":"2024-04-02T14:32:28.390137+0000","last_undegraded":"2024-04-02T14:32:28.390137+0000","last_fullsized":"2024-04-02T14:32:28.390137+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:00:44.605314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:25.416960+0000","last_change":"2024-04-02T14:32:22.532937+0000","last_active":"2024-04-02T14:32:25.416960+0000","last_peered":"2024-04-02T14:32:25.416960+0000","last_clean":"2024-04-02T14:32:25.416960+0000","last_became_active":"2024-04-02T14:32:22.532586+0000","last_became_peered":"2024-04-02T14:32:22.532586+0000","last_unstale":"2024-04-02T14:32:25.416960+0000","last_undegraded":"2024-04-02T14:32:25.416960+0000","last_fullsized":"2024-04-02T14:32:25.416960+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:50:48.030889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"88'10","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:25.923569+0000","last_change":"2024-04-02T14:32:21.860032+0000","last_active":"2024-04-02T14:32:25.923569+0000","last_peered":"2024-04-02T14:32:25.923569+0000","last_clean":"2024-04-02T14:32:25.923569+0000","last_became_active":"2024-04-02T14:32:21.859653+0000","last_became_peered":"2024-04-02T14:32:21.859653+0000","last_unstale":"2024-04-02T14:32:25.923569+0000","last_undegraded":"2024-04-02T14:32:25.923569+0000","last_fullsized":"2024-04-02T14:32:25.923569+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:06:01.144530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:25.546734+0000","last_change":"2024-04-02T14:32:22.542481+0000","last_active":"2024-04-02T14:32:25.546734+0000","last_peered":"2024-04-02T14:32:25.546734+0000","last_clean":"2024-04-02T14:32:25.546734+0000","last_became_active":"2024-04-02T14:32:22.542163+0000","last_became_peered":"2024-04-02T14:32:22.542163+0000","last_unstale":"2024-04-02T14:32:25.546734+0000","last_undegraded":"2024-04-02T14:32:25.546734+0000","last_fullsized":"2024-04-02T14:32:25.546734+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:40:28.173821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"87'1","reported_seq":16,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:24.858049+0000","last_change":"2024-04-02T14:32:21.842384+0000","last_active":"2024-04-02T14:32:24.858049+0000","last_peered":"2024-04-02T14:32:24.858049+0000","last_clean":"2024-04-02T14:32:24.858049+0000","last_became_active":"2024-04-02T14:32:21.842040+0000","last_became_peered":"2024-04-02T14:32:21.842040+0000","last_unstale":"2024-04-02T14:32:24.858049+0000","last_undegraded":"2024-04-02T14:32:24.858049+0000","last_fullsized":"2024-04-02T14:32:24.858049+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:33:29.541972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:25.546712+0000","last_change":"2024-04-02T14:32:22.538571+0000","last_active":"2024-04-02T14:32:25.546712+0000","last_peered":"2024-04-02T14:32:25.546712+0000","last_clean":"2024-04-02T14:32:25.546712+0000","last_became_active":"2024-04-02T14:32:22.538239+0000","last_became_peered":"2024-04-02T14:32:22.538239+0000","last_unstale":"2024-04-02T14:32:25.546712+0000","last_undegraded":"2024-04-02T14:32:25.546712+0000","last_fullsized":"2024-04-02T14:32:25.546712+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T01:19:53.120419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"88'1","reported_seq":16,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:23.963703+0000","last_change":"2024-04-02T14:32:21.857666+0000","last_active":"2024-04-02T14:32:23.963703+0000","last_peered":"2024-04-02T14:32:23.963703+0000","last_clean":"2024-04-02T14:32:23.963703+0000","last_became_active":"2024-04-02T14:32:21.856618+0000","last_became_peered":"2024-04-02T14:32:21.856618+0000","last_unstale":"2024-04-02T14:32:23.963703+0000","last_undegraded":"2024-04-02T14:32:23.963703+0000","last_fullsized":"2024-04-02T14:32:23.963703+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T01:20:53.183052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"88'1","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:24.656202+0000","last_change":"2024-04-02T14:32:22.552277+0000","last_active":"2024-04-02T14:32:24.656202+0000","last_peered":"2024-04-02T14:32:24.656202+0000","last_clean":"2024-04-02T14:32:24.656202+0000","last_became_active":"2024-04-02T14:32:22.551882+0000","last_became_peered":"2024-04-02T14:32:22.551882+0000","last_unstale":"2024-04-02T14:32:24.656202+0000","last_undegraded":"2024-04-02T14:32:24.656202+0000","last_fullsized":"2024-04-02T14:32:24.656202+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:30:43.592910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:25.416856+0000","last_change":"2024-04-02T14:32:21.848258+0000","last_active":"2024-04-02T14:32:25.416856+0000","last_peered":"2024-04-02T14:32:25.416856+0000","last_clean":"2024-04-02T14:32:25.416856+0000","last_became_active":"2024-04-02T14:32:21.847805+0000","last_became_peered":"2024-04-02T14:32:21.847805+0000","last_unstale":"2024-04-02T14:32:25.416856+0000","last_undegraded":"2024-04-02T14:32:25.416856+0000","last_fullsized":"2024-04-02T14:32:25.416856+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:46:07.552786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:23.521863+0000","last_change":"2024-04-02T14:32:22.560835+0000","last_active":"2024-04-02T14:32:23.521863+0000","last_peered":"2024-04-02T14:32:23.521863+0000","last_clean":"2024-04-02T14:32:23.521863+0000","last_became_active":"2024-04-02T14:32:22.560469+0000","last_became_peered":"2024-04-02T14:32:22.560469+0000","last_unstale":"2024-04-02T14:32:23.521863+0000","last_undegraded":"2024-04-02T14:32:23.521863+0000","last_fullsized":"2024-04-02T14:32:23.521863+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T02:29:24.869650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:25.546900+0000","last_change":"2024-04-02T14:32:22.538325+0000","last_active":"2024-04-02T14:32:25.546900+0000","last_peered":"2024-04-02T14:32:25.546900+0000","last_clean":"2024-04-02T14:32:25.546900+0000","last_became_active":"2024-04-02T14:32:22.537903+0000","last_became_peered":"2024-04-02T14:32:22.537903+0000","last_unstale":"2024-04-02T14:32:25.546900+0000","last_undegraded":"2024-04-02T14:32:25.546900+0000","last_fullsized":"2024-04-02T14:32:25.546900+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:12:26.586290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"88'1","reported_seq":16,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:23.522541+0000","last_change":"2024-04-02T14:32:21.861333+0000","last_active":"2024-04-02T14:32:23.522541+0000","last_peered":"2024-04-02T14:32:23.522541+0000","last_clean":"2024-04-02T14:32:23.522541+0000","last_became_active":"2024-04-02T14:32:21.860909+0000","last_became_peered":"2024-04-02T14:32:21.860909+0000","last_unstale":"2024-04-02T14:32:23.522541+0000","last_undegraded":"2024-04-02T14:32:23.522541+0000","last_fullsized":"2024-04-02T14:32:23.522541+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:21:52.616036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:24.859549+0000","last_change":"2024-04-02T14:32:22.554044+0000","last_active":"2024-04-02T14:32:24.859549+0000","last_peered":"2024-04-02T14:32:24.859549+0000","last_clean":"2024-04-02T14:32:24.859549+0000","last_became_active":"2024-04-02T14:32:22.553629+0000","last_became_peered":"2024-04-02T14:32:22.553629+0000","last_unstale":"2024-04-02T14:32:24.859549+0000","last_undegraded":"2024-04-02T14:32:24.859549+0000","last_fullsized":"2024-04-02T14:32:24.859549+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:16:16.919503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"88'6","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:29.821651+0000","last_change":"2024-04-02T14:32:21.837336+0000","last_active":"2024-04-02T14:32:29.821651+0000","last_peered":"2024-04-02T14:32:29.821651+0000","last_clean":"2024-04-02T14:32:29.821651+0000","last_became_active":"2024-04-02T14:32:21.836921+0000","last_became_peered":"2024-04-02T14:32:21.836921+0000","last_unstale":"2024-04-02T14:32:29.821651+0000","last_undegraded":"2024-04-02T14:32:29.821651+0000","last_fullsized":"2024-04-02T14:32:29.821651+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:34:52.883173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":351,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:25.416883+0000","last_change":"2024-04-02T14:31:02.710356+0000","last_active":"2024-04-02T14:32:25.416883+0000","last_peered":"2024-04-02T14:32:25.416883+0000","last_clean":"2024-04-02T14:32:25.416883+0000","last_became_active":"2024-04-02T14:31:02.709669+0000","last_became_peered":"2024-04-02T14:31:02.709669+0000","last_unstale":"2024-04-02T14:32:25.416883+0000","last_undegraded":"2024-04-02T14:32:25.416883+0000","last_fullsized":"2024-04-02T14:32:25.416883+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:27:17.358883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:27:17.358883+0000","last_clean_scrub_stamp":"2024-04-02T14:27:17.358883+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:45:55.893284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:23.522006+0000","last_change":"2024-04-02T14:32:22.545502+0000","last_active":"2024-04-02T14:32:23.522006+0000","last_peered":"2024-04-02T14:32:23.522006+0000","last_clean":"2024-04-02T14:32:23.522006+0000","last_became_active":"2024-04-02T14:32:22.545111+0000","last_became_peered":"2024-04-02T14:32:22.545111+0000","last_unstale":"2024-04-02T14:32:23.522006+0000","last_undegraded":"2024-04-02T14:32:23.522006+0000","last_fullsized":"2024-04-02T14:32:23.522006+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:02:32.178334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"88'1","reported_seq":16,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:25.417261+0000","last_change":"2024-04-02T14:32:21.842665+0000","last_active":"2024-04-02T14:32:25.417261+0000","last_peered":"2024-04-02T14:32:25.417261+0000","last_clean":"2024-04-02T14:32:25.417261+0000","last_became_active":"2024-04-02T14:32:21.842284+0000","last_became_peered":"2024-04-02T14:32:21.842284+0000","last_unstale":"2024-04-02T14:32:25.417261+0000","last_undegraded":"2024-04-02T14:32:25.417261+0000","last_fullsized":"2024-04-02T14:32:25.417261+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:05:59.547063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"88'1","reported_seq":16,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:24.858099+0000","last_change":"2024-04-02T14:32:21.860226+0000","last_active":"2024-04-02T14:32:24.858099+0000","last_peered":"2024-04-02T14:32:24.858099+0000","last_clean":"2024-04-02T14:32:24.858099+0000","last_became_active":"2024-04-02T14:32:21.859854+0000","last_became_peered":"2024-04-02T14:32:21.859854+0000","last_unstale":"2024-04-02T14:32:24.858099+0000","last_undegraded":"2024-04-02T14:32:24.858099+0000","last_fullsized":"2024-04-02T14:32:24.858099+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:30:36.710677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:23.523607+0000","last_change":"2024-04-02T14:32:22.552287+0000","last_active":"2024-04-02T14:32:23.523607+0000","last_peered":"2024-04-02T14:32:23.523607+0000","last_clean":"2024-04-02T14:32:23.523607+0000","last_became_active":"2024-04-02T14:32:22.551719+0000","last_became_peered":"2024-04-02T14:32:22.551719+0000","last_unstale":"2024-04-02T14:32:23.523607+0000","last_undegraded":"2024-04-02T14:32:23.523607+0000","last_fullsized":"2024-04-02T14:32:23.523607+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:26:12.007600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"88'8","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:27.947247+0000","last_change":"2024-04-02T14:32:21.851671+0000","last_active":"2024-04-02T14:32:27.947247+0000","last_peered":"2024-04-02T14:32:27.947247+0000","last_clean":"2024-04-02T14:32:27.947247+0000","last_became_active":"2024-04-02T14:32:21.851297+0000","last_became_peered":"2024-04-02T14:32:21.851297+0000","last_unstale":"2024-04-02T14:32:27.947247+0000","last_undegraded":"2024-04-02T14:32:27.947247+0000","last_fullsized":"2024-04-02T14:32:27.947247+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:51:20.232086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:24.859825+0000","last_change":"2024-04-02T14:32:22.554554+0000","last_active":"2024-04-02T14:32:24.859825+0000","last_peered":"2024-04-02T14:32:24.859825+0000","last_clean":"2024-04-02T14:32:24.859825+0000","last_became_active":"2024-04-02T14:32:22.554194+0000","last_became_peered":"2024-04-02T14:32:22.554194+0000","last_unstale":"2024-04-02T14:32:24.859825+0000","last_undegraded":"2024-04-02T14:32:24.859825+0000","last_fullsized":"2024-04-02T14:32:24.859825+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:27:51.382441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"88'7","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:28.969404+0000","last_change":"2024-04-02T14:32:21.850346+0000","last_active":"2024-04-02T14:32:28.969404+0000","last_peered":"2024-04-02T14:32:28.969404+0000","last_clean":"2024-04-02T14:32:28.969404+0000","last_became_active":"2024-04-02T14:32:21.850084+0000","last_became_peered":"2024-04-02T14:32:21.850084+0000","last_unstale":"2024-04-02T14:32:28.969404+0000","last_undegraded":"2024-04-02T14:32:28.969404+0000","last_fullsized":"2024-04-02T14:32:28.969404+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:56:17.774702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:23.523952+0000","last_change":"2024-04-02T14:32:22.557054+0000","last_active":"2024-04-02T14:32:23.523952+0000","last_peered":"2024-04-02T14:32:23.523952+0000","last_clean":"2024-04-02T14:32:23.523952+0000","last_became_active":"2024-04-02T14:32:22.556335+0000","last_became_peered":"2024-04-02T14:32:22.556335+0000","last_unstale":"2024-04-02T14:32:23.523952+0000","last_undegraded":"2024-04-02T14:32:23.523952+0000","last_fullsized":"2024-04-02T14:32:23.523952+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:06:48.907197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"88'2","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:25.922754+0000","last_change":"2024-04-02T14:32:21.860260+0000","last_active":"2024-04-02T14:32:25.922754+0000","last_peered":"2024-04-02T14:32:25.922754+0000","last_clean":"2024-04-02T14:32:25.922754+0000","last_became_active":"2024-04-02T14:32:21.859922+0000","last_became_peered":"2024-04-02T14:32:21.859922+0000","last_unstale":"2024-04-02T14:32:25.922754+0000","last_undegraded":"2024-04-02T14:32:25.922754+0000","last_fullsized":"2024-04-02T14:32:25.922754+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:07:45.850148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:24.857752+0000","last_change":"2024-04-02T14:32:22.536063+0000","last_active":"2024-04-02T14:32:24.857752+0000","last_peered":"2024-04-02T14:32:24.857752+0000","last_clean":"2024-04-02T14:32:24.857752+0000","last_became_active":"2024-04-02T14:32:22.534678+0000","last_became_peered":"2024-04-02T14:32:22.534678+0000","last_unstale":"2024-04-02T14:32:24.857752+0000","last_undegraded":"2024-04-02T14:32:24.857752+0000","last_fullsized":"2024-04-02T14:32:24.857752+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T02:07:26.186216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"88'3","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:25.926096+0000","last_change":"2024-04-02T14:32:21.848554+0000","last_active":"2024-04-02T14:32:25.926096+0000","last_peered":"2024-04-02T14:32:25.926096+0000","last_clean":"2024-04-02T14:32:25.926096+0000","last_became_active":"2024-04-02T14:32:21.848177+0000","last_became_peered":"2024-04-02T14:32:21.848177+0000","last_unstale":"2024-04-02T14:32:25.926096+0000","last_undegraded":"2024-04-02T14:32:25.926096+0000","last_fullsized":"2024-04-02T14:32:25.926096+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:47:04.213286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:25.546560+0000","last_change":"2024-04-02T14:32:22.547246+0000","last_active":"2024-04-02T14:32:25.546560+0000","last_peered":"2024-04-02T14:32:25.546560+0000","last_clean":"2024-04-02T14:32:25.546560+0000","last_became_active":"2024-04-02T14:32:22.546721+0000","last_became_peered":"2024-04-02T14:32:22.546721+0000","last_unstale":"2024-04-02T14:32:25.546560+0000","last_undegraded":"2024-04-02T14:32:25.546560+0000","last_fullsized":"2024-04-02T14:32:25.546560+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:11:30.615723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:25.546862+0000","last_change":"2024-04-02T14:32:21.856013+0000","last_active":"2024-04-02T14:32:25.546862+0000","last_peered":"2024-04-02T14:32:25.546862+0000","last_clean":"2024-04-02T14:32:25.546862+0000","last_became_active":"2024-04-02T14:32:21.855631+0000","last_became_peered":"2024-04-02T14:32:21.855631+0000","last_unstale":"2024-04-02T14:32:25.546862+0000","last_undegraded":"2024-04-02T14:32:25.546862+0000","last_fullsized":"2024-04-02T14:32:25.546862+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:46:38.902988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:24.859562+0000","last_change":"2024-04-02T14:32:22.561787+0000","last_active":"2024-04-02T14:32:24.859562+0000","last_peered":"2024-04-02T14:32:24.859562+0000","last_clean":"2024-04-02T14:32:24.859562+0000","last_became_active":"2024-04-02T14:32:22.561284+0000","last_became_peered":"2024-04-02T14:32:22.561284+0000","last_unstale":"2024-04-02T14:32:24.859562+0000","last_undegraded":"2024-04-02T14:32:24.859562+0000","last_fullsized":"2024-04-02T14:32:24.859562+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:46:06.450097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"88'4","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:25.923327+0000","last_change":"2024-04-02T14:32:21.841340+0000","last_active":"2024-04-02T14:32:25.923327+0000","last_peered":"2024-04-02T14:32:25.923327+0000","last_clean":"2024-04-02T14:32:25.923327+0000","last_became_active":"2024-04-02T14:32:21.840937+0000","last_became_peered":"2024-04-02T14:32:21.840937+0000","last_unstale":"2024-04-02T14:32:25.923327+0000","last_undegraded":"2024-04-02T14:32:25.923327+0000","last_fullsized":"2024-04-02T14:32:25.923327+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T01:47:44.834440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:25.417350+0000","last_change":"2024-04-02T14:32:22.538890+0000","last_active":"2024-04-02T14:32:25.417350+0000","last_peered":"2024-04-02T14:32:25.417350+0000","last_clean":"2024-04-02T14:32:25.417350+0000","last_became_active":"2024-04-02T14:32:22.538512+0000","last_became_peered":"2024-04-02T14:32:22.538512+0000","last_unstale":"2024-04-02T14:32:25.417350+0000","last_undegraded":"2024-04-02T14:32:25.417350+0000","last_fullsized":"2024-04-02T14:32:25.417350+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:59:23.792851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:25.547478+0000","last_change":"2024-04-02T14:32:21.862727+0000","last_active":"2024-04-02T14:32:25.547478+0000","last_peered":"2024-04-02T14:32:25.547478+0000","last_clean":"2024-04-02T14:32:25.547478+0000","last_became_active":"2024-04-02T14:32:21.862405+0000","last_became_peered":"2024-04-02T14:32:21.862405+0000","last_unstale":"2024-04-02T14:32:25.547478+0000","last_undegraded":"2024-04-02T14:32:25.547478+0000","last_fullsized":"2024-04-02T14:32:25.547478+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:36:54.849964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:25.417353+0000","last_change":"2024-04-02T14:32:22.540077+0000","last_active":"2024-04-02T14:32:25.417353+0000","last_peered":"2024-04-02T14:32:25.417353+0000","last_clean":"2024-04-02T14:32:25.417353+0000","last_became_active":"2024-04-02T14:32:22.539513+0000","last_became_peered":"2024-04-02T14:32:22.539513+0000","last_unstale":"2024-04-02T14:32:25.417353+0000","last_undegraded":"2024-04-02T14:32:25.417353+0000","last_fullsized":"2024-04-02T14:32:25.417353+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T02:00:39.819963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"87'1","reported_seq":16,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:25.547004+0000","last_change":"2024-04-02T14:32:21.860310+0000","last_active":"2024-04-02T14:32:25.547004+0000","last_peered":"2024-04-02T14:32:25.547004+0000","last_clean":"2024-04-02T14:32:25.547004+0000","last_became_active":"2024-04-02T14:32:21.859955+0000","last_became_peered":"2024-04-02T14:32:21.859955+0000","last_unstale":"2024-04-02T14:32:25.547004+0000","last_undegraded":"2024-04-02T14:32:25.547004+0000","last_fullsized":"2024-04-02T14:32:25.547004+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:11:32.732603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:25.417069+0000","last_change":"2024-04-02T14:32:22.552134+0000","last_active":"2024-04-02T14:32:25.417069+0000","last_peered":"2024-04-02T14:32:25.417069+0000","last_clean":"2024-04-02T14:32:25.417069+0000","last_became_active":"2024-04-02T14:32:22.551695+0000","last_became_peered":"2024-04-02T14:32:22.551695+0000","last_unstale":"2024-04-02T14:32:25.417069+0000","last_undegraded":"2024-04-02T14:32:25.417069+0000","last_fullsized":"2024-04-02T14:32:25.417069+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:09:36.902243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"87'2","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:23.523415+0000","last_change":"2024-04-02T14:32:21.857571+0000","last_active":"2024-04-02T14:32:23.523415+0000","last_peered":"2024-04-02T14:32:23.523415+0000","last_clean":"2024-04-02T14:32:23.523415+0000","last_became_active":"2024-04-02T14:32:21.856935+0000","last_became_peered":"2024-04-02T14:32:21.856935+0000","last_unstale":"2024-04-02T14:32:23.523415+0000","last_undegraded":"2024-04-02T14:32:23.523415+0000","last_fullsized":"2024-04-02T14:32:23.523415+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:55:46.079097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:25.417364+0000","last_change":"2024-04-02T14:32:22.534708+0000","last_active":"2024-04-02T14:32:25.417364+0000","last_peered":"2024-04-02T14:32:25.417364+0000","last_clean":"2024-04-02T14:32:25.417364+0000","last_became_active":"2024-04-02T14:32:22.534365+0000","last_became_peered":"2024-04-02T14:32:22.534365+0000","last_unstale":"2024-04-02T14:32:25.417364+0000","last_undegraded":"2024-04-02T14:32:25.417364+0000","last_fullsized":"2024-04-02T14:32:25.417364+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:34:47.435729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:25.547029+0000","last_change":"2024-04-02T14:32:22.538317+0000","last_active":"2024-04-02T14:32:25.547029+0000","last_peered":"2024-04-02T14:32:25.547029+0000","last_clean":"2024-04-02T14:32:25.547029+0000","last_became_active":"2024-04-02T14:32:22.537939+0000","last_became_peered":"2024-04-02T14:32:22.537939+0000","last_unstale":"2024-04-02T14:32:25.547029+0000","last_undegraded":"2024-04-02T14:32:25.547029+0000","last_fullsized":"2024-04-02T14:32:25.547029+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:31:47.075326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:25.547676+0000","last_change":"2024-04-02T14:32:22.535110+0000","last_active":"2024-04-02T14:32:25.547676+0000","last_peered":"2024-04-02T14:32:25.547676+0000","last_clean":"2024-04-02T14:32:25.547676+0000","last_became_active":"2024-04-02T14:32:22.534712+0000","last_became_peered":"2024-04-02T14:32:22.534712+0000","last_unstale":"2024-04-02T14:32:25.547676+0000","last_undegraded":"2024-04-02T14:32:25.547676+0000","last_fullsized":"2024-04-02T14:32:25.547676+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:41:12.960443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:25.546825+0000","last_change":"2024-04-02T14:32:22.556567+0000","last_active":"2024-04-02T14:32:25.546825+0000","last_peered":"2024-04-02T14:32:25.546825+0000","last_clean":"2024-04-02T14:32:25.546825+0000","last_became_active":"2024-04-02T14:32:22.556267+0000","last_became_peered":"2024-04-02T14:32:22.556267+0000","last_unstale":"2024-04-02T14:32:25.546825+0000","last_undegraded":"2024-04-02T14:32:25.546825+0000","last_fullsized":"2024-04-02T14:32:25.546825+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:35:30.039013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:23.523150+0000","last_change":"2024-04-02T14:32:22.560440+0000","last_active":"2024-04-02T14:32:23.523150+0000","last_peered":"2024-04-02T14:32:23.523150+0000","last_clean":"2024-04-02T14:32:23.523150+0000","last_became_active":"2024-04-02T14:32:22.560096+0000","last_became_peered":"2024-04-02T14:32:22.560096+0000","last_unstale":"2024-04-02T14:32:23.523150+0000","last_undegraded":"2024-04-02T14:32:23.523150+0000","last_fullsized":"2024-04-02T14:32:23.523150+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:25:51.639900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:23.523603+0000","last_change":"2024-04-02T14:32:22.558584+0000","last_active":"2024-04-02T14:32:23.523603+0000","last_peered":"2024-04-02T14:32:23.523603+0000","last_clean":"2024-04-02T14:32:23.523603+0000","last_became_active":"2024-04-02T14:32:22.557382+0000","last_became_peered":"2024-04-02T14:32:22.557382+0000","last_unstale":"2024-04-02T14:32:23.523603+0000","last_undegraded":"2024-04-02T14:32:23.523603+0000","last_fullsized":"2024-04-02T14:32:23.523603+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:09:25.107411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:25.547501+0000","last_change":"2024-04-02T14:32:22.538318+0000","last_active":"2024-04-02T14:32:25.547501+0000","last_peered":"2024-04-02T14:32:25.547501+0000","last_clean":"2024-04-02T14:32:25.547501+0000","last_became_active":"2024-04-02T14:32:22.538011+0000","last_became_peered":"2024-04-02T14:32:22.538011+0000","last_unstale":"2024-04-02T14:32:25.547501+0000","last_undegraded":"2024-04-02T14:32:25.547501+0000","last_fullsized":"2024-04-02T14:32:25.547501+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:11:21.269106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:25.547730+0000","last_change":"2024-04-02T14:32:22.535645+0000","last_active":"2024-04-02T14:32:25.547730+0000","last_peered":"2024-04-02T14:32:25.547730+0000","last_clean":"2024-04-02T14:32:25.547730+0000","last_became_active":"2024-04-02T14:32:22.535314+0000","last_became_peered":"2024-04-02T14:32:22.535314+0000","last_unstale":"2024-04-02T14:32:25.547730+0000","last_undegraded":"2024-04-02T14:32:25.547730+0000","last_fullsized":"2024-04-02T14:32:25.547730+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:51:27.879015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:24.859866+0000","last_change":"2024-04-02T14:32:22.553959+0000","last_active":"2024-04-02T14:32:24.859866+0000","last_peered":"2024-04-02T14:32:24.859866+0000","last_clean":"2024-04-02T14:32:24.859866+0000","last_became_active":"2024-04-02T14:32:22.553389+0000","last_became_peered":"2024-04-02T14:32:22.553389+0000","last_unstale":"2024-04-02T14:32:24.859866+0000","last_undegraded":"2024-04-02T14:32:24.859866+0000","last_fullsized":"2024-04-02T14:32:24.859866+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:55:00.552600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:25.546914+0000","last_change":"2024-04-02T14:32:22.551769+0000","last_active":"2024-04-02T14:32:25.546914+0000","last_peered":"2024-04-02T14:32:25.546914+0000","last_clean":"2024-04-02T14:32:25.546914+0000","last_became_active":"2024-04-02T14:32:22.551422+0000","last_became_peered":"2024-04-02T14:32:22.551422+0000","last_unstale":"2024-04-02T14:32:25.546914+0000","last_undegraded":"2024-04-02T14:32:25.546914+0000","last_fullsized":"2024-04-02T14:32:25.546914+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T01:15:07.262927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:24.859021+0000","last_change":"2024-04-02T14:32:22.548264+0000","last_active":"2024-04-02T14:32:24.859021+0000","last_peered":"2024-04-02T14:32:24.859021+0000","last_clean":"2024-04-02T14:32:24.859021+0000","last_became_active":"2024-04-02T14:32:22.547293+0000","last_became_peered":"2024-04-02T14:32:22.547293+0000","last_unstale":"2024-04-02T14:32:24.859021+0000","last_undegraded":"2024-04-02T14:32:24.859021+0000","last_fullsized":"2024-04-02T14:32:24.859021+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:53:19.896243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:25.546651+0000","last_change":"2024-04-02T14:32:22.556131+0000","last_active":"2024-04-02T14:32:25.546651+0000","last_peered":"2024-04-02T14:32:25.546651+0000","last_clean":"2024-04-02T14:32:25.546651+0000","last_became_active":"2024-04-02T14:32:22.555763+0000","last_became_peered":"2024-04-02T14:32:22.555763+0000","last_unstale":"2024-04-02T14:32:25.546651+0000","last_undegraded":"2024-04-02T14:32:25.546651+0000","last_fullsized":"2024-04-02T14:32:25.546651+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:45:22.485403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:23.523329+0000","last_change":"2024-04-02T14:32:22.562093+0000","last_active":"2024-04-02T14:32:23.523329+0000","last_peered":"2024-04-02T14:32:23.523329+0000","last_clean":"2024-04-02T14:32:23.523329+0000","last_became_active":"2024-04-02T14:32:22.561663+0000","last_became_peered":"2024-04-02T14:32:22.561663+0000","last_unstale":"2024-04-02T14:32:23.523329+0000","last_undegraded":"2024-04-02T14:32:23.523329+0000","last_fullsized":"2024-04-02T14:32:23.523329+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:14:25.209276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:25.546909+0000","last_change":"2024-04-02T14:32:22.541964+0000","last_active":"2024-04-02T14:32:25.546909+0000","last_peered":"2024-04-02T14:32:25.546909+0000","last_clean":"2024-04-02T14:32:25.546909+0000","last_became_active":"2024-04-02T14:32:22.541549+0000","last_became_peered":"2024-04-02T14:32:22.541549+0000","last_unstale":"2024-04-02T14:32:25.546909+0000","last_undegraded":"2024-04-02T14:32:25.546909+0000","last_fullsized":"2024-04-02T14:32:25.546909+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:04:56.783158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":24258,"num_objects":59,"num_object_clones":0,"num_object_copies":177,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":59,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":139,"num_write_kb":54,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":274432,"data_stored":36331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":118,"ondisk_log_size":118,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":83,"seq":356482285583,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32116,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713036,"statfs":{"total":95995035648,"available":95962148864,"internally_reserved":0,"allocated":962560,"data_stored":457516,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":75,"seq":322122547221,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36916,"kb_used_data":1580,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708236,"statfs":{"total":95995035648,"available":95957233664,"internally_reserved":0,"allocated":1617920,"data_stored":1069797,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":36174273},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743449,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36288,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708864,"statfs":{"total":95995035648,"available":95957876736,"internally_reserved":0,"allocated":974848,"data_stored":457232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":61,"seq":261993005088,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32116,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713036,"statfs":{"total":95995035648,"available":95962148864,"internally_reserved":0,"allocated":958464,"data_stored":456928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299429,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32696,"kb_used_data":1520,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712456,"statfs":{"total":95995035648,"available":95961554944,"internally_reserved":0,"allocated":1556480,"data_stored":1047896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495657,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36276,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708876,"statfs":{"total":95995035648,"available":95957889024,"internally_reserved":0,"allocated":950272,"data_stored":456324,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574919,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36280,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708872,"statfs":{"total":95995035648,"available":95957884928,"internally_reserved":0,"allocated":962560,"data_stored":457936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738444,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36296,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708856,"statfs":{"total":95995035648,"available":95957868544,"internally_reserved":0,"allocated":983040,"data_stored":458390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411393,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36292,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708860,"statfs":{"total":95995035648,"available":95957872640,"internally_reserved":0,"allocated":974848,"data_stored":458592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182458,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27960,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717192,"statfs":{"total":95995035648,"available":95966404608,"internally_reserved":0,"allocated":962560,"data_stored":456518,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920823,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32124,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713028,"statfs":{"total":95995035648,"available":95962140672,"internally_reserved":0,"allocated":970752,"data_stored":457415,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757295,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28552,"kb_used_data":1536,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716600,"statfs":{"total":95995035648,"available":95965798400,"internally_reserved":0,"allocated":1572864,"data_stored":1048168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":2066,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":23022,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-02T14:32:31.836 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-02T14:32:32.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:32.085+0000 7f0252400640 1 -- 172.21.15.90:0/2128483678 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f024c104f30 msgr2=0x7f024c1053b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:32:32.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:32.085+0000 7f0252400640 1 --2- 172.21.15.90:0/2128483678 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f024c104f30 0x7f024c1053b0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f0238009860 tx=0x7f023802bb70 comp rx=0 tx=0).stop 2024-04-02T14:32:32.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:32.086+0000 7f0252400640 1 -- 172.21.15.90:0/2128483678 shutdown_connections 2024-04-02T14:32:32.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:32.086+0000 7f0252400640 1 --2- 172.21.15.90:0/2128483678 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f024c1058f0 0x7f024c10c180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:32.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:32.086+0000 7f0252400640 1 --2- 172.21.15.90:0/2128483678 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f024c104f30 0x7f024c1053b0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:32.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:32.086+0000 7f0252400640 1 --2- 172.21.15.90:0/2128483678 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f024c103d40 0x7f024c104160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:32.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:32.086+0000 7f0252400640 1 -- 172.21.15.90:0/2128483678 >> 172.21.15.90:0/2128483678 conn(0x7f024c0ff2f0 msgr2=0x7f024c101710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:32:32.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:32.086+0000 7f0252400640 1 -- 172.21.15.90:0/2128483678 shutdown_connections 2024-04-02T14:32:32.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:32.086+0000 7f0252400640 1 -- 172.21.15.90:0/2128483678 wait complete. 2024-04-02T14:32:32.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:32.087+0000 7f0252400640 1 Processor -- start 2024-04-02T14:32:32.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:32.087+0000 7f0252400640 1 -- start start 2024-04-02T14:32:32.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:32.087+0000 7f0252400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f024c103d40 0x7f024c19a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:32:32.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:32.087+0000 7f0252400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f024c104f30 0x7f024c19a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:32:32.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:32.087+0000 7f0252400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f024c1058f0 0x7f024c1a15c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:32:32.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:32.087+0000 7f0252400640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f024c10f9a0 con 0x7f024c104f30 2024-04-02T14:32:32.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:32.087+0000 7f0252400640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f024c10f820 con 0x7f024c103d40 2024-04-02T14:32:32.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:32.087+0000 7f0252400640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f024c10fb20 con 0x7f024c1058f0 2024-04-02T14:32:32.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:32.087+0000 7f024b400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f024c103d40 0x7f024c19a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:32:32.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:32.087+0000 7f024aa00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f024c104f30 0x7f024c19a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:32:32.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:32.087+0000 7f024b400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f024c103d40 0x7f024c19a010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:45090/0 (socket says 172.21.15.90:45090) 2024-04-02T14:32:32.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:32.087+0000 7f024b400640 1 -- 172.21.15.90:0/1698334688 learned_addr learned my addr 172.21.15.90:0/1698334688 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:32:32.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:32.087+0000 7f024aa00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f024c104f30 0x7f024c19a550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:49488/0 (socket says 172.21.15.90:49488) 2024-04-02T14:32:32.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:32.088+0000 7f024be00640 1 --2- 172.21.15.90:0/1698334688 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f024c1058f0 0x7f024c1a15c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:32:32.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:32.088+0000 7f024b400640 1 -- 172.21.15.90:0/1698334688 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f024c1058f0 msgr2=0x7f024c1a15c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:32:32.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:32.088+0000 7f024b400640 1 --2- 172.21.15.90:0/1698334688 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f024c1058f0 0x7f024c1a15c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:32.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:32.088+0000 7f024b400640 1 -- 172.21.15.90:0/1698334688 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f024c104f30 msgr2=0x7f024c19a550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:32:32.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:32.088+0000 7f024b400640 1 --2- 172.21.15.90:0/1698334688 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f024c104f30 0x7f024c19a550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:32.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:32.088+0000 7f024b400640 1 -- 172.21.15.90:0/1698334688 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f024c1a1c30 con 0x7f024c103d40 2024-04-02T14:32:32.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:32.088+0000 7f024be00640 1 --2- 172.21.15.90:0/1698334688 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f024c1058f0 0x7f024c1a15c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:32:32.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:32.088+0000 7f024b400640 1 --2- 172.21.15.90:0/1698334688 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f024c103d40 0x7f024c19a010 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f023c00c970 tx=0x7f023c00ce30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:32:32.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:32.088+0000 7f0243e00640 1 -- 172.21.15.90:0/1698334688 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f023c007c10 con 0x7f024c103d40 2024-04-02T14:32:32.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:32.088+0000 7f0243e00640 1 -- 172.21.15.90:0/1698334688 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f023c007db0 con 0x7f024c103d40 2024-04-02T14:32:32.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:32.088+0000 7f0243e00640 1 -- 172.21.15.90:0/1698334688 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f023c00f940 con 0x7f024c103d40 2024-04-02T14:32:32.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:32.088+0000 7f0252400640 1 -- 172.21.15.90:0/1698334688 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f024c1a1f20 con 0x7f024c103d40 2024-04-02T14:32:32.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:32.089+0000 7f0252400640 1 -- 172.21.15.90:0/1698334688 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f024c1a2330 con 0x7f024c103d40 2024-04-02T14:32:32.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:32.090+0000 7f0252400640 1 -- 172.21.15.90:0/1698334688 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f024c10af90 con 0x7f024c103d40 2024-04-02T14:32:32.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:32.091+0000 7f0243e00640 1 -- 172.21.15.90:0/1698334688 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f023c00fae0 con 0x7f024c103d40 2024-04-02T14:32:32.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:32.092+0000 7f0243e00640 1 --2- 172.21.15.90:0/1698334688 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f02300a3c40 0x7f02300a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:32:32.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:32.093+0000 7f0243e00640 1 -- 172.21.15.90:0/1698334688 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 7979+0+0 (secure 0 0 0) 0x7f023c0d0b10 con 0x7f024c103d40 2024-04-02T14:32:32.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:32.093+0000 7f024aa00640 1 --2- 172.21.15.90:0/1698334688 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f02300a3c40 0x7f02300a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:32:32.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:32.093+0000 7f024aa00640 1 --2- 172.21.15.90:0/1698334688 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f02300a3c40 0x7f02300a6100 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f0238009830 tx=0x7f0238034ab0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:32:32.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:32.095+0000 7f0243e00640 1 -- 172.21.15.90:0/1698334688 <== mon.2 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7f023c09d4d0 con 0x7f024c103d40 2024-04-02T14:32:32.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:31 smithi090 ceph-mon[31674]: from='client.? 172.21.15.90:0/563069497' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-02T14:32:32.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:31 smithi090 ceph-mon[31674]: from='client.24722 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-02T14:32:32.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:32.297+0000 7f0252400640 1 -- 172.21.15.90:0/1698334688 --> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f024c061f40 con 0x7f02300a3c40 2024-04-02T14:32:32.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:32.320+0000 7f0243e00640 1 -- 172.21.15.90:0/1698334688 <== mgr.14152 v2:172.21.15.90:6800/4266054817 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258734 (secure 0 0 0) 0x7f024c061f40 con 0x7f02300a3c40 2024-04-02T14:32:32.326 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-02T14:32:32.328 INFO:teuthology.orchestra.run.smithi090.stderr:dumped all 2024-04-02T14:32:32.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:32.325+0000 7f0252400640 1 -- 172.21.15.90:0/1698334688 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f02300a3c40 msgr2=0x7f02300a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:32:32.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:32.325+0000 7f0252400640 1 --2- 172.21.15.90:0/1698334688 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f02300a3c40 0x7f02300a6100 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f0238009830 tx=0x7f0238034ab0 comp rx=0 tx=0).stop 2024-04-02T14:32:32.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:32.325+0000 7f0252400640 1 -- 172.21.15.90:0/1698334688 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f024c103d40 msgr2=0x7f024c19a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:32:32.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:32.325+0000 7f0252400640 1 --2- 172.21.15.90:0/1698334688 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f024c103d40 0x7f024c19a010 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f023c00c970 tx=0x7f023c00ce30 comp rx=0 tx=0).stop 2024-04-02T14:32:32.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:32.325+0000 7f0252400640 1 -- 172.21.15.90:0/1698334688 shutdown_connections 2024-04-02T14:32:32.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:32.325+0000 7f0252400640 1 --2- 172.21.15.90:0/1698334688 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f02300a3c40 0x7f02300a6100 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:32.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:32.325+0000 7f0252400640 1 --2- 172.21.15.90:0/1698334688 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f024c1058f0 0x7f024c1a15c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:32.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:32.325+0000 7f0252400640 1 --2- 172.21.15.90:0/1698334688 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f024c104f30 0x7f024c19a550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:32.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:32.325+0000 7f0252400640 1 --2- 172.21.15.90:0/1698334688 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f024c103d40 0x7f024c19a010 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:32.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:32.325+0000 7f0252400640 1 -- 172.21.15.90:0/1698334688 >> 172.21.15.90:0/1698334688 conn(0x7f024c0ff2f0 msgr2=0x7f024c108160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:32:32.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:32.325+0000 7f0252400640 1 -- 172.21.15.90:0/1698334688 shutdown_connections 2024-04-02T14:32:32.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:32.326+0000 7f0252400640 1 -- 172.21.15.90:0/1698334688 wait complete. 2024-04-02T14:32:32.348 INFO:teuthology.orchestra.run.smithi090.stdout:{"pg_ready":true,"pg_map":{"version":316,"stamp":"2024-04-02T14:32:30.732293+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":614743,"num_objects":62,"num_object_clones":0,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":143,"num_read_kb":126,"num_write":321,"num_write_kb":2993,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":256,"ondisk_log_size":256,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":283,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":403912,"kb_used_data":13132,"kb_used_omap":18,"kb_used_meta":390637,"kb_avail":1124537912,"statfs":{"total":1151940427776,"available":1151526821888,"internally_reserved":0,"allocated":13447168,"data_stored":7282712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":400012602},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":24375,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":140,"num_write_kb":55,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.213893"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:25.547264+0000","last_change":"2024-04-02T14:32:22.558913+0000","last_active":"2024-04-02T14:32:25.547264+0000","last_peered":"2024-04-02T14:32:25.547264+0000","last_clean":"2024-04-02T14:32:25.547264+0000","last_became_active":"2024-04-02T14:32:22.558606+0000","last_became_peered":"2024-04-02T14:32:22.558606+0000","last_unstale":"2024-04-02T14:32:25.547264+0000","last_undegraded":"2024-04-02T14:32:25.547264+0000","last_fullsized":"2024-04-02T14:32:25.547264+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:59:49.205499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:23.523775+0000","last_change":"2024-04-02T14:32:22.551898+0000","last_active":"2024-04-02T14:32:23.523775+0000","last_peered":"2024-04-02T14:32:23.523775+0000","last_clean":"2024-04-02T14:32:23.523775+0000","last_became_active":"2024-04-02T14:32:22.549016+0000","last_became_peered":"2024-04-02T14:32:22.549016+0000","last_unstale":"2024-04-02T14:32:23.523775+0000","last_undegraded":"2024-04-02T14:32:23.523775+0000","last_fullsized":"2024-04-02T14:32:23.523775+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:38:03.529505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:23.523891+0000","last_change":"2024-04-02T14:32:22.556214+0000","last_active":"2024-04-02T14:32:23.523891+0000","last_peered":"2024-04-02T14:32:23.523891+0000","last_clean":"2024-04-02T14:32:23.523891+0000","last_became_active":"2024-04-02T14:32:22.555686+0000","last_became_peered":"2024-04-02T14:32:22.555686+0000","last_unstale":"2024-04-02T14:32:23.523891+0000","last_undegraded":"2024-04-02T14:32:23.523891+0000","last_fullsized":"2024-04-02T14:32:23.523891+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:06:40.161203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:25.547236+0000","last_change":"2024-04-02T14:32:22.537830+0000","last_active":"2024-04-02T14:32:25.547236+0000","last_peered":"2024-04-02T14:32:25.547236+0000","last_clean":"2024-04-02T14:32:25.547236+0000","last_became_active":"2024-04-02T14:32:22.536860+0000","last_became_peered":"2024-04-02T14:32:22.536860+0000","last_unstale":"2024-04-02T14:32:25.547236+0000","last_undegraded":"2024-04-02T14:32:25.547236+0000","last_fullsized":"2024-04-02T14:32:25.547236+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T01:07:57.244177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:24.857848+0000","last_change":"2024-04-02T14:32:22.549550+0000","last_active":"2024-04-02T14:32:24.857848+0000","last_peered":"2024-04-02T14:32:24.857848+0000","last_clean":"2024-04-02T14:32:24.857848+0000","last_became_active":"2024-04-02T14:32:22.549177+0000","last_became_peered":"2024-04-02T14:32:22.549177+0000","last_unstale":"2024-04-02T14:32:24.857848+0000","last_undegraded":"2024-04-02T14:32:24.857848+0000","last_fullsized":"2024-04-02T14:32:24.857848+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:31:49.175583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:23.521965+0000","last_change":"2024-04-02T14:32:22.548786+0000","last_active":"2024-04-02T14:32:23.521965+0000","last_peered":"2024-04-02T14:32:23.521965+0000","last_clean":"2024-04-02T14:32:23.521965+0000","last_became_active":"2024-04-02T14:32:22.548426+0000","last_became_peered":"2024-04-02T14:32:22.548426+0000","last_unstale":"2024-04-02T14:32:23.521965+0000","last_undegraded":"2024-04-02T14:32:23.521965+0000","last_fullsized":"2024-04-02T14:32:23.521965+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:21:58.623016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:25.416920+0000","last_change":"2024-04-02T14:32:22.546861+0000","last_active":"2024-04-02T14:32:25.416920+0000","last_peered":"2024-04-02T14:32:25.416920+0000","last_clean":"2024-04-02T14:32:25.416920+0000","last_became_active":"2024-04-02T14:32:22.546477+0000","last_became_peered":"2024-04-02T14:32:22.546477+0000","last_unstale":"2024-04-02T14:32:25.416920+0000","last_undegraded":"2024-04-02T14:32:25.416920+0000","last_fullsized":"2024-04-02T14:32:25.416920+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:59:17.133714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:24.858949+0000","last_change":"2024-04-02T14:32:22.554287+0000","last_active":"2024-04-02T14:32:24.858949+0000","last_peered":"2024-04-02T14:32:24.858949+0000","last_clean":"2024-04-02T14:32:24.858949+0000","last_became_active":"2024-04-02T14:32:22.553491+0000","last_became_peered":"2024-04-02T14:32:22.553491+0000","last_unstale":"2024-04-02T14:32:24.858949+0000","last_undegraded":"2024-04-02T14:32:24.858949+0000","last_fullsized":"2024-04-02T14:32:24.858949+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:51:30.785380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:25.547093+0000","last_change":"2024-04-02T14:32:22.538158+0000","last_active":"2024-04-02T14:32:25.547093+0000","last_peered":"2024-04-02T14:32:25.547093+0000","last_clean":"2024-04-02T14:32:25.547093+0000","last_became_active":"2024-04-02T14:32:22.537796+0000","last_became_peered":"2024-04-02T14:32:22.537796+0000","last_unstale":"2024-04-02T14:32:25.547093+0000","last_undegraded":"2024-04-02T14:32:25.547093+0000","last_fullsized":"2024-04-02T14:32:25.547093+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:09:48.054965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:23.522159+0000","last_change":"2024-04-02T14:32:22.535225+0000","last_active":"2024-04-02T14:32:23.522159+0000","last_peered":"2024-04-02T14:32:23.522159+0000","last_clean":"2024-04-02T14:32:23.522159+0000","last_became_active":"2024-04-02T14:32:22.534856+0000","last_became_peered":"2024-04-02T14:32:22.534856+0000","last_unstale":"2024-04-02T14:32:23.522159+0000","last_undegraded":"2024-04-02T14:32:23.522159+0000","last_fullsized":"2024-04-02T14:32:23.522159+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:22:11.118918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:25.417206+0000","last_change":"2024-04-02T14:32:22.539138+0000","last_active":"2024-04-02T14:32:25.417206+0000","last_peered":"2024-04-02T14:32:25.417206+0000","last_clean":"2024-04-02T14:32:25.417206+0000","last_became_active":"2024-04-02T14:32:22.538762+0000","last_became_peered":"2024-04-02T14:32:22.538762+0000","last_unstale":"2024-04-02T14:32:25.417206+0000","last_undegraded":"2024-04-02T14:32:25.417206+0000","last_fullsized":"2024-04-02T14:32:25.417206+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:18:59.174339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:25.547217+0000","last_change":"2024-04-02T14:32:22.536383+0000","last_active":"2024-04-02T14:32:25.547217+0000","last_peered":"2024-04-02T14:32:25.547217+0000","last_clean":"2024-04-02T14:32:25.547217+0000","last_became_active":"2024-04-02T14:32:22.536043+0000","last_became_peered":"2024-04-02T14:32:22.536043+0000","last_unstale":"2024-04-02T14:32:25.547217+0000","last_undegraded":"2024-04-02T14:32:25.547217+0000","last_fullsized":"2024-04-02T14:32:25.547217+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:14:48.628684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:23.523530+0000","last_change":"2024-04-02T14:32:22.552512+0000","last_active":"2024-04-02T14:32:23.523530+0000","last_peered":"2024-04-02T14:32:23.523530+0000","last_clean":"2024-04-02T14:32:23.523530+0000","last_became_active":"2024-04-02T14:32:22.549502+0000","last_became_peered":"2024-04-02T14:32:22.549502+0000","last_unstale":"2024-04-02T14:32:23.523530+0000","last_undegraded":"2024-04-02T14:32:23.523530+0000","last_fullsized":"2024-04-02T14:32:23.523530+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:37:45.609280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:23.522812+0000","last_change":"2024-04-02T14:32:22.555074+0000","last_active":"2024-04-02T14:32:23.522812+0000","last_peered":"2024-04-02T14:32:23.522812+0000","last_clean":"2024-04-02T14:32:23.522812+0000","last_became_active":"2024-04-02T14:32:22.554422+0000","last_became_peered":"2024-04-02T14:32:22.554422+0000","last_unstale":"2024-04-02T14:32:23.522812+0000","last_undegraded":"2024-04-02T14:32:23.522812+0000","last_fullsized":"2024-04-02T14:32:23.522812+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:27:49.536941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:23.522401+0000","last_change":"2024-04-02T14:32:22.550322+0000","last_active":"2024-04-02T14:32:23.522401+0000","last_peered":"2024-04-02T14:32:23.522401+0000","last_clean":"2024-04-02T14:32:23.522401+0000","last_became_active":"2024-04-02T14:32:22.548710+0000","last_became_peered":"2024-04-02T14:32:22.548710+0000","last_unstale":"2024-04-02T14:32:23.522401+0000","last_undegraded":"2024-04-02T14:32:23.522401+0000","last_fullsized":"2024-04-02T14:32:23.522401+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:19:09.226556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:25.546173+0000","last_change":"2024-04-02T14:32:22.558705+0000","last_active":"2024-04-02T14:32:25.546173+0000","last_peered":"2024-04-02T14:32:25.546173+0000","last_clean":"2024-04-02T14:32:25.546173+0000","last_became_active":"2024-04-02T14:32:22.558309+0000","last_became_peered":"2024-04-02T14:32:22.558309+0000","last_unstale":"2024-04-02T14:32:25.546173+0000","last_undegraded":"2024-04-02T14:32:25.546173+0000","last_fullsized":"2024-04-02T14:32:25.546173+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:52:23.929627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:25.547427+0000","last_change":"2024-04-02T14:32:22.551211+0000","last_active":"2024-04-02T14:32:25.547427+0000","last_peered":"2024-04-02T14:32:25.547427+0000","last_clean":"2024-04-02T14:32:25.547427+0000","last_became_active":"2024-04-02T14:32:22.550882+0000","last_became_peered":"2024-04-02T14:32:22.550882+0000","last_unstale":"2024-04-02T14:32:25.547427+0000","last_undegraded":"2024-04-02T14:32:25.547427+0000","last_fullsized":"2024-04-02T14:32:25.547427+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:54:54.533063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:23.879656+0000","last_change":"2024-04-02T14:32:22.550692+0000","last_active":"2024-04-02T14:32:23.879656+0000","last_peered":"2024-04-02T14:32:23.879656+0000","last_clean":"2024-04-02T14:32:23.879656+0000","last_became_active":"2024-04-02T14:32:22.550292+0000","last_became_peered":"2024-04-02T14:32:22.550292+0000","last_unstale":"2024-04-02T14:32:23.879656+0000","last_undegraded":"2024-04-02T14:32:23.879656+0000","last_fullsized":"2024-04-02T14:32:23.879656+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:14:42.317649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:25.546874+0000","last_change":"2024-04-02T14:32:22.553042+0000","last_active":"2024-04-02T14:32:25.546874+0000","last_peered":"2024-04-02T14:32:25.546874+0000","last_clean":"2024-04-02T14:32:25.546874+0000","last_became_active":"2024-04-02T14:32:22.552010+0000","last_became_peered":"2024-04-02T14:32:22.552010+0000","last_unstale":"2024-04-02T14:32:25.546874+0000","last_undegraded":"2024-04-02T14:32:25.546874+0000","last_fullsized":"2024-04-02T14:32:25.546874+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:31:13.947662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"88'11","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:27.943801+0000","last_change":"2024-04-02T14:32:21.841348+0000","last_active":"2024-04-02T14:32:27.943801+0000","last_peered":"2024-04-02T14:32:27.943801+0000","last_clean":"2024-04-02T14:32:27.943801+0000","last_became_active":"2024-04-02T14:32:21.839947+0000","last_became_peered":"2024-04-02T14:32:21.839947+0000","last_unstale":"2024-04-02T14:32:27.943801+0000","last_undegraded":"2024-04-02T14:32:27.943801+0000","last_fullsized":"2024-04-02T14:32:27.943801+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:03:49.484343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:24.859947+0000","last_change":"2024-04-02T14:32:22.549526+0000","last_active":"2024-04-02T14:32:24.859947+0000","last_peered":"2024-04-02T14:32:24.859947+0000","last_clean":"2024-04-02T14:32:24.859947+0000","last_became_active":"2024-04-02T14:32:22.548250+0000","last_became_peered":"2024-04-02T14:32:22.548250+0000","last_unstale":"2024-04-02T14:32:24.859947+0000","last_undegraded":"2024-04-02T14:32:24.859947+0000","last_fullsized":"2024-04-02T14:32:24.859947+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:35:34.866803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"88'2","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:24.860144+0000","last_change":"2024-04-02T14:32:21.865084+0000","last_active":"2024-04-02T14:32:24.860144+0000","last_peered":"2024-04-02T14:32:24.860144+0000","last_clean":"2024-04-02T14:32:24.860144+0000","last_became_active":"2024-04-02T14:32:21.864718+0000","last_became_peered":"2024-04-02T14:32:21.864718+0000","last_unstale":"2024-04-02T14:32:24.860144+0000","last_undegraded":"2024-04-02T14:32:24.860144+0000","last_fullsized":"2024-04-02T14:32:24.860144+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:10:50.611362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:23.524474+0000","last_change":"2024-04-02T14:32:22.549443+0000","last_active":"2024-04-02T14:32:23.524474+0000","last_peered":"2024-04-02T14:32:23.524474+0000","last_clean":"2024-04-02T14:32:23.524474+0000","last_became_active":"2024-04-02T14:32:22.549059+0000","last_became_peered":"2024-04-02T14:32:22.549059+0000","last_unstale":"2024-04-02T14:32:23.524474+0000","last_undegraded":"2024-04-02T14:32:23.524474+0000","last_fullsized":"2024-04-02T14:32:23.524474+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:04:21.489864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"87'2","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:24.858192+0000","last_change":"2024-04-02T14:32:21.841368+0000","last_active":"2024-04-02T14:32:24.858192+0000","last_peered":"2024-04-02T14:32:24.858192+0000","last_clean":"2024-04-02T14:32:24.858192+0000","last_became_active":"2024-04-02T14:32:21.840963+0000","last_became_peered":"2024-04-02T14:32:21.840963+0000","last_unstale":"2024-04-02T14:32:24.858192+0000","last_undegraded":"2024-04-02T14:32:24.858192+0000","last_fullsized":"2024-04-02T14:32:24.858192+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:36:40.549720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:25.546449+0000","last_change":"2024-04-02T14:32:22.549753+0000","last_active":"2024-04-02T14:32:25.546449+0000","last_peered":"2024-04-02T14:32:25.546449+0000","last_clean":"2024-04-02T14:32:25.546449+0000","last_became_active":"2024-04-02T14:32:22.549354+0000","last_became_peered":"2024-04-02T14:32:22.549354+0000","last_unstale":"2024-04-02T14:32:25.546449+0000","last_undegraded":"2024-04-02T14:32:25.546449+0000","last_fullsized":"2024-04-02T14:32:25.546449+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T02:17:09.650946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"87'2","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:24.858050+0000","last_change":"2024-04-02T14:32:21.842766+0000","last_active":"2024-04-02T14:32:24.858050+0000","last_peered":"2024-04-02T14:32:24.858050+0000","last_clean":"2024-04-02T14:32:24.858050+0000","last_became_active":"2024-04-02T14:32:21.842410+0000","last_became_peered":"2024-04-02T14:32:21.842410+0000","last_unstale":"2024-04-02T14:32:24.858050+0000","last_undegraded":"2024-04-02T14:32:24.858050+0000","last_fullsized":"2024-04-02T14:32:24.858050+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:49:31.651127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:25.547188+0000","last_change":"2024-04-02T14:32:22.538506+0000","last_active":"2024-04-02T14:32:25.547188+0000","last_peered":"2024-04-02T14:32:25.547188+0000","last_clean":"2024-04-02T14:32:25.547188+0000","last_became_active":"2024-04-02T14:32:22.538168+0000","last_became_peered":"2024-04-02T14:32:22.538168+0000","last_unstale":"2024-04-02T14:32:25.547188+0000","last_undegraded":"2024-04-02T14:32:25.547188+0000","last_fullsized":"2024-04-02T14:32:25.547188+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:30:04.746382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"88'5","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:25.925554+0000","last_change":"2024-04-02T14:32:21.844180+0000","last_active":"2024-04-02T14:32:25.925554+0000","last_peered":"2024-04-02T14:32:25.925554+0000","last_clean":"2024-04-02T14:32:25.925554+0000","last_became_active":"2024-04-02T14:32:21.843840+0000","last_became_peered":"2024-04-02T14:32:21.843840+0000","last_unstale":"2024-04-02T14:32:25.925554+0000","last_undegraded":"2024-04-02T14:32:25.925554+0000","last_fullsized":"2024-04-02T14:32:25.925554+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T01:56:10.767946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:24.857911+0000","last_change":"2024-04-02T14:32:22.548453+0000","last_active":"2024-04-02T14:32:24.857911+0000","last_peered":"2024-04-02T14:32:24.857911+0000","last_clean":"2024-04-02T14:32:24.857911+0000","last_became_active":"2024-04-02T14:32:22.548110+0000","last_became_peered":"2024-04-02T14:32:22.548110+0000","last_unstale":"2024-04-02T14:32:24.857911+0000","last_undegraded":"2024-04-02T14:32:24.857911+0000","last_fullsized":"2024-04-02T14:32:24.857911+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:57:13.067425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"88'7","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:27.946738+0000","last_change":"2024-04-02T14:32:21.846528+0000","last_active":"2024-04-02T14:32:27.946738+0000","last_peered":"2024-04-02T14:32:27.946738+0000","last_clean":"2024-04-02T14:32:27.946738+0000","last_became_active":"2024-04-02T14:32:21.846152+0000","last_became_peered":"2024-04-02T14:32:21.846152+0000","last_unstale":"2024-04-02T14:32:27.946738+0000","last_undegraded":"2024-04-02T14:32:27.946738+0000","last_fullsized":"2024-04-02T14:32:27.946738+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:58:09.286256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:25.547865+0000","last_change":"2024-04-02T14:32:22.546730+0000","last_active":"2024-04-02T14:32:25.547865+0000","last_peered":"2024-04-02T14:32:25.547865+0000","last_clean":"2024-04-02T14:32:25.547865+0000","last_became_active":"2024-04-02T14:32:22.546342+0000","last_became_peered":"2024-04-02T14:32:22.546342+0000","last_unstale":"2024-04-02T14:32:25.547865+0000","last_undegraded":"2024-04-02T14:32:25.547865+0000","last_fullsized":"2024-04-02T14:32:25.547865+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:38:37.616382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:23.522699+0000","last_change":"2024-04-02T14:32:21.853316+0000","last_active":"2024-04-02T14:32:23.522699+0000","last_peered":"2024-04-02T14:32:23.522699+0000","last_clean":"2024-04-02T14:32:23.522699+0000","last_became_active":"2024-04-02T14:32:21.852912+0000","last_became_peered":"2024-04-02T14:32:21.852912+0000","last_unstale":"2024-04-02T14:32:23.522699+0000","last_undegraded":"2024-04-02T14:32:23.522699+0000","last_fullsized":"2024-04-02T14:32:23.522699+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T02:30:44.601900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:24.858700+0000","last_change":"2024-04-02T14:32:22.558913+0000","last_active":"2024-04-02T14:32:24.858700+0000","last_peered":"2024-04-02T14:32:24.858700+0000","last_clean":"2024-04-02T14:32:24.858700+0000","last_became_active":"2024-04-02T14:32:22.558511+0000","last_became_peered":"2024-04-02T14:32:22.558511+0000","last_unstale":"2024-04-02T14:32:24.858700+0000","last_undegraded":"2024-04-02T14:32:24.858700+0000","last_fullsized":"2024-04-02T14:32:24.858700+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:48:36.449086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"88'1","reported_seq":16,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:25.546542+0000","last_change":"2024-04-02T14:32:21.844081+0000","last_active":"2024-04-02T14:32:25.546542+0000","last_peered":"2024-04-02T14:32:25.546542+0000","last_clean":"2024-04-02T14:32:25.546542+0000","last_became_active":"2024-04-02T14:32:21.843788+0000","last_became_peered":"2024-04-02T14:32:21.843788+0000","last_unstale":"2024-04-02T14:32:25.546542+0000","last_undegraded":"2024-04-02T14:32:25.546542+0000","last_fullsized":"2024-04-02T14:32:25.546542+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:19:02.184229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:23.879234+0000","last_change":"2024-04-02T14:32:22.556193+0000","last_active":"2024-04-02T14:32:23.879234+0000","last_peered":"2024-04-02T14:32:23.879234+0000","last_clean":"2024-04-02T14:32:23.879234+0000","last_became_active":"2024-04-02T14:32:22.555787+0000","last_became_peered":"2024-04-02T14:32:22.555787+0000","last_unstale":"2024-04-02T14:32:23.879234+0000","last_undegraded":"2024-04-02T14:32:23.879234+0000","last_fullsized":"2024-04-02T14:32:23.879234+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T02:02:13.730941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"88'4","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:25.926039+0000","last_change":"2024-04-02T14:32:21.840651+0000","last_active":"2024-04-02T14:32:25.926039+0000","last_peered":"2024-04-02T14:32:25.926039+0000","last_clean":"2024-04-02T14:32:25.926039+0000","last_became_active":"2024-04-02T14:32:21.840344+0000","last_became_peered":"2024-04-02T14:32:21.840344+0000","last_unstale":"2024-04-02T14:32:25.926039+0000","last_undegraded":"2024-04-02T14:32:25.926039+0000","last_fullsized":"2024-04-02T14:32:25.926039+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:49:22.384224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:25.546658+0000","last_change":"2024-04-02T14:32:22.548245+0000","last_active":"2024-04-02T14:32:25.546658+0000","last_peered":"2024-04-02T14:32:25.546658+0000","last_clean":"2024-04-02T14:32:25.546658+0000","last_became_active":"2024-04-02T14:32:22.547853+0000","last_became_peered":"2024-04-02T14:32:22.547853+0000","last_unstale":"2024-04-02T14:32:25.546658+0000","last_undegraded":"2024-04-02T14:32:25.546658+0000","last_fullsized":"2024-04-02T14:32:25.546658+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:02:43.036944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"88'2","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:25.922832+0000","last_change":"2024-04-02T14:32:21.842096+0000","last_active":"2024-04-02T14:32:25.922832+0000","last_peered":"2024-04-02T14:32:25.922832+0000","last_clean":"2024-04-02T14:32:25.922832+0000","last_became_active":"2024-04-02T14:32:21.841646+0000","last_became_peered":"2024-04-02T14:32:21.841646+0000","last_unstale":"2024-04-02T14:32:25.922832+0000","last_undegraded":"2024-04-02T14:32:25.922832+0000","last_fullsized":"2024-04-02T14:32:25.922832+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:04:08.149785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:23.524303+0000","last_change":"2024-04-02T14:32:22.557694+0000","last_active":"2024-04-02T14:32:23.524303+0000","last_peered":"2024-04-02T14:32:23.524303+0000","last_clean":"2024-04-02T14:32:23.524303+0000","last_became_active":"2024-04-02T14:32:22.557202+0000","last_became_peered":"2024-04-02T14:32:22.557202+0000","last_unstale":"2024-04-02T14:32:23.524303+0000","last_undegraded":"2024-04-02T14:32:23.524303+0000","last_fullsized":"2024-04-02T14:32:23.524303+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:35:41.471557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"88'4","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:25.547372+0000","last_change":"2024-04-02T14:32:21.864366+0000","last_active":"2024-04-02T14:32:25.547372+0000","last_peered":"2024-04-02T14:32:25.547372+0000","last_clean":"2024-04-02T14:32:25.547372+0000","last_became_active":"2024-04-02T14:32:21.864006+0000","last_became_peered":"2024-04-02T14:32:21.864006+0000","last_unstale":"2024-04-02T14:32:25.547372+0000","last_undegraded":"2024-04-02T14:32:25.547372+0000","last_fullsized":"2024-04-02T14:32:25.547372+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:08:43.176061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:25.546568+0000","last_change":"2024-04-02T14:32:22.541562+0000","last_active":"2024-04-02T14:32:25.546568+0000","last_peered":"2024-04-02T14:32:25.546568+0000","last_clean":"2024-04-02T14:32:25.546568+0000","last_became_active":"2024-04-02T14:32:22.541209+0000","last_became_peered":"2024-04-02T14:32:22.541209+0000","last_unstale":"2024-04-02T14:32:25.546568+0000","last_undegraded":"2024-04-02T14:32:25.546568+0000","last_fullsized":"2024-04-02T14:32:25.546568+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:49:40.310528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"88'17","reported_seq":32,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:28.968702+0000","last_change":"2024-04-02T14:32:21.839406+0000","last_active":"2024-04-02T14:32:28.968702+0000","last_peered":"2024-04-02T14:32:28.968702+0000","last_clean":"2024-04-02T14:32:28.968702+0000","last_became_active":"2024-04-02T14:32:21.839080+0000","last_became_peered":"2024-04-02T14:32:21.839080+0000","last_unstale":"2024-04-02T14:32:28.968702+0000","last_undegraded":"2024-04-02T14:32:28.968702+0000","last_fullsized":"2024-04-02T14:32:28.968702+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:33:43.128536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20895,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:24.858143+0000","last_change":"2024-04-02T14:32:22.535716+0000","last_active":"2024-04-02T14:32:24.858143+0000","last_peered":"2024-04-02T14:32:24.858143+0000","last_clean":"2024-04-02T14:32:24.858143+0000","last_became_active":"2024-04-02T14:32:22.534738+0000","last_became_peered":"2024-04-02T14:32:22.534738+0000","last_unstale":"2024-04-02T14:32:24.858143+0000","last_undegraded":"2024-04-02T14:32:24.858143+0000","last_fullsized":"2024-04-02T14:32:24.858143+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T01:25:07.763915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"88'4","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:25.927483+0000","last_change":"2024-04-02T14:32:21.854582+0000","last_active":"2024-04-02T14:32:25.927483+0000","last_peered":"2024-04-02T14:32:25.927483+0000","last_clean":"2024-04-02T14:32:25.927483+0000","last_became_active":"2024-04-02T14:32:21.854226+0000","last_became_peered":"2024-04-02T14:32:21.854226+0000","last_unstale":"2024-04-02T14:32:25.927483+0000","last_undegraded":"2024-04-02T14:32:25.927483+0000","last_fullsized":"2024-04-02T14:32:25.927483+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:30:02.289163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:23.522911+0000","last_change":"2024-04-02T14:32:22.555901+0000","last_active":"2024-04-02T14:32:23.522911+0000","last_peered":"2024-04-02T14:32:23.522911+0000","last_clean":"2024-04-02T14:32:23.522911+0000","last_became_active":"2024-04-02T14:32:22.555158+0000","last_became_peered":"2024-04-02T14:32:22.555158+0000","last_unstale":"2024-04-02T14:32:23.522911+0000","last_undegraded":"2024-04-02T14:32:23.522911+0000","last_fullsized":"2024-04-02T14:32:23.522911+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:04:28.766185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"88'3","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:25.927454+0000","last_change":"2024-04-02T14:32:21.855982+0000","last_active":"2024-04-02T14:32:25.927454+0000","last_peered":"2024-04-02T14:32:25.927454+0000","last_clean":"2024-04-02T14:32:25.927454+0000","last_became_active":"2024-04-02T14:32:21.855591+0000","last_became_peered":"2024-04-02T14:32:21.855591+0000","last_unstale":"2024-04-02T14:32:25.927454+0000","last_undegraded":"2024-04-02T14:32:25.927454+0000","last_fullsized":"2024-04-02T14:32:25.927454+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:28:51.806779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:25.417261+0000","last_change":"2024-04-02T14:32:22.549889+0000","last_active":"2024-04-02T14:32:25.417261+0000","last_peered":"2024-04-02T14:32:25.417261+0000","last_clean":"2024-04-02T14:32:25.417261+0000","last_became_active":"2024-04-02T14:32:22.549479+0000","last_became_peered":"2024-04-02T14:32:22.549479+0000","last_unstale":"2024-04-02T14:32:25.417261+0000","last_undegraded":"2024-04-02T14:32:25.417261+0000","last_fullsized":"2024-04-02T14:32:25.417261+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:29:25.937576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"88'6","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:28.390137+0000","last_change":"2024-04-02T14:32:21.838165+0000","last_active":"2024-04-02T14:32:28.390137+0000","last_peered":"2024-04-02T14:32:28.390137+0000","last_clean":"2024-04-02T14:32:28.390137+0000","last_became_active":"2024-04-02T14:32:21.837808+0000","last_became_peered":"2024-04-02T14:32:21.837808+0000","last_unstale":"2024-04-02T14:32:28.390137+0000","last_undegraded":"2024-04-02T14:32:28.390137+0000","last_fullsized":"2024-04-02T14:32:28.390137+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:00:44.605314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:25.416960+0000","last_change":"2024-04-02T14:32:22.532937+0000","last_active":"2024-04-02T14:32:25.416960+0000","last_peered":"2024-04-02T14:32:25.416960+0000","last_clean":"2024-04-02T14:32:25.416960+0000","last_became_active":"2024-04-02T14:32:22.532586+0000","last_became_peered":"2024-04-02T14:32:22.532586+0000","last_unstale":"2024-04-02T14:32:25.416960+0000","last_undegraded":"2024-04-02T14:32:25.416960+0000","last_fullsized":"2024-04-02T14:32:25.416960+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:50:48.030889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"88'10","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:25.923569+0000","last_change":"2024-04-02T14:32:21.860032+0000","last_active":"2024-04-02T14:32:25.923569+0000","last_peered":"2024-04-02T14:32:25.923569+0000","last_clean":"2024-04-02T14:32:25.923569+0000","last_became_active":"2024-04-02T14:32:21.859653+0000","last_became_peered":"2024-04-02T14:32:21.859653+0000","last_unstale":"2024-04-02T14:32:25.923569+0000","last_undegraded":"2024-04-02T14:32:25.923569+0000","last_fullsized":"2024-04-02T14:32:25.923569+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:06:01.144530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:25.546734+0000","last_change":"2024-04-02T14:32:22.542481+0000","last_active":"2024-04-02T14:32:25.546734+0000","last_peered":"2024-04-02T14:32:25.546734+0000","last_clean":"2024-04-02T14:32:25.546734+0000","last_became_active":"2024-04-02T14:32:22.542163+0000","last_became_peered":"2024-04-02T14:32:22.542163+0000","last_unstale":"2024-04-02T14:32:25.546734+0000","last_undegraded":"2024-04-02T14:32:25.546734+0000","last_fullsized":"2024-04-02T14:32:25.546734+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:40:28.173821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"87'1","reported_seq":16,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:24.858049+0000","last_change":"2024-04-02T14:32:21.842384+0000","last_active":"2024-04-02T14:32:24.858049+0000","last_peered":"2024-04-02T14:32:24.858049+0000","last_clean":"2024-04-02T14:32:24.858049+0000","last_became_active":"2024-04-02T14:32:21.842040+0000","last_became_peered":"2024-04-02T14:32:21.842040+0000","last_unstale":"2024-04-02T14:32:24.858049+0000","last_undegraded":"2024-04-02T14:32:24.858049+0000","last_fullsized":"2024-04-02T14:32:24.858049+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:33:29.541972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:25.546712+0000","last_change":"2024-04-02T14:32:22.538571+0000","last_active":"2024-04-02T14:32:25.546712+0000","last_peered":"2024-04-02T14:32:25.546712+0000","last_clean":"2024-04-02T14:32:25.546712+0000","last_became_active":"2024-04-02T14:32:22.538239+0000","last_became_peered":"2024-04-02T14:32:22.538239+0000","last_unstale":"2024-04-02T14:32:25.546712+0000","last_undegraded":"2024-04-02T14:32:25.546712+0000","last_fullsized":"2024-04-02T14:32:25.546712+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T01:19:53.120419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"88'1","reported_seq":16,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:23.963703+0000","last_change":"2024-04-02T14:32:21.857666+0000","last_active":"2024-04-02T14:32:23.963703+0000","last_peered":"2024-04-02T14:32:23.963703+0000","last_clean":"2024-04-02T14:32:23.963703+0000","last_became_active":"2024-04-02T14:32:21.856618+0000","last_became_peered":"2024-04-02T14:32:21.856618+0000","last_unstale":"2024-04-02T14:32:23.963703+0000","last_undegraded":"2024-04-02T14:32:23.963703+0000","last_fullsized":"2024-04-02T14:32:23.963703+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T01:20:53.183052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"88'1","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:24.656202+0000","last_change":"2024-04-02T14:32:22.552277+0000","last_active":"2024-04-02T14:32:24.656202+0000","last_peered":"2024-04-02T14:32:24.656202+0000","last_clean":"2024-04-02T14:32:24.656202+0000","last_became_active":"2024-04-02T14:32:22.551882+0000","last_became_peered":"2024-04-02T14:32:22.551882+0000","last_unstale":"2024-04-02T14:32:24.656202+0000","last_undegraded":"2024-04-02T14:32:24.656202+0000","last_fullsized":"2024-04-02T14:32:24.656202+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:30:43.592910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:25.416856+0000","last_change":"2024-04-02T14:32:21.848258+0000","last_active":"2024-04-02T14:32:25.416856+0000","last_peered":"2024-04-02T14:32:25.416856+0000","last_clean":"2024-04-02T14:32:25.416856+0000","last_became_active":"2024-04-02T14:32:21.847805+0000","last_became_peered":"2024-04-02T14:32:21.847805+0000","last_unstale":"2024-04-02T14:32:25.416856+0000","last_undegraded":"2024-04-02T14:32:25.416856+0000","last_fullsized":"2024-04-02T14:32:25.416856+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:46:07.552786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:23.521863+0000","last_change":"2024-04-02T14:32:22.560835+0000","last_active":"2024-04-02T14:32:23.521863+0000","last_peered":"2024-04-02T14:32:23.521863+0000","last_clean":"2024-04-02T14:32:23.521863+0000","last_became_active":"2024-04-02T14:32:22.560469+0000","last_became_peered":"2024-04-02T14:32:22.560469+0000","last_unstale":"2024-04-02T14:32:23.521863+0000","last_undegraded":"2024-04-02T14:32:23.521863+0000","last_fullsized":"2024-04-02T14:32:23.521863+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T02:29:24.869650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:25.546900+0000","last_change":"2024-04-02T14:32:22.538325+0000","last_active":"2024-04-02T14:32:25.546900+0000","last_peered":"2024-04-02T14:32:25.546900+0000","last_clean":"2024-04-02T14:32:25.546900+0000","last_became_active":"2024-04-02T14:32:22.537903+0000","last_became_peered":"2024-04-02T14:32:22.537903+0000","last_unstale":"2024-04-02T14:32:25.546900+0000","last_undegraded":"2024-04-02T14:32:25.546900+0000","last_fullsized":"2024-04-02T14:32:25.546900+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:12:26.586290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"88'1","reported_seq":16,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:23.522541+0000","last_change":"2024-04-02T14:32:21.861333+0000","last_active":"2024-04-02T14:32:23.522541+0000","last_peered":"2024-04-02T14:32:23.522541+0000","last_clean":"2024-04-02T14:32:23.522541+0000","last_became_active":"2024-04-02T14:32:21.860909+0000","last_became_peered":"2024-04-02T14:32:21.860909+0000","last_unstale":"2024-04-02T14:32:23.522541+0000","last_undegraded":"2024-04-02T14:32:23.522541+0000","last_fullsized":"2024-04-02T14:32:23.522541+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:21:52.616036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:24.859549+0000","last_change":"2024-04-02T14:32:22.554044+0000","last_active":"2024-04-02T14:32:24.859549+0000","last_peered":"2024-04-02T14:32:24.859549+0000","last_clean":"2024-04-02T14:32:24.859549+0000","last_became_active":"2024-04-02T14:32:22.553629+0000","last_became_peered":"2024-04-02T14:32:22.553629+0000","last_unstale":"2024-04-02T14:32:24.859549+0000","last_undegraded":"2024-04-02T14:32:24.859549+0000","last_fullsized":"2024-04-02T14:32:24.859549+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:16:16.919503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"88'6","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:29.821651+0000","last_change":"2024-04-02T14:32:21.837336+0000","last_active":"2024-04-02T14:32:29.821651+0000","last_peered":"2024-04-02T14:32:29.821651+0000","last_clean":"2024-04-02T14:32:29.821651+0000","last_became_active":"2024-04-02T14:32:21.836921+0000","last_became_peered":"2024-04-02T14:32:21.836921+0000","last_unstale":"2024-04-02T14:32:29.821651+0000","last_undegraded":"2024-04-02T14:32:29.821651+0000","last_fullsized":"2024-04-02T14:32:29.821651+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:34:52.883173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":351,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:25.416883+0000","last_change":"2024-04-02T14:31:02.710356+0000","last_active":"2024-04-02T14:32:25.416883+0000","last_peered":"2024-04-02T14:32:25.416883+0000","last_clean":"2024-04-02T14:32:25.416883+0000","last_became_active":"2024-04-02T14:31:02.709669+0000","last_became_peered":"2024-04-02T14:31:02.709669+0000","last_unstale":"2024-04-02T14:32:25.416883+0000","last_undegraded":"2024-04-02T14:32:25.416883+0000","last_fullsized":"2024-04-02T14:32:25.416883+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:27:17.358883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:27:17.358883+0000","last_clean_scrub_stamp":"2024-04-02T14:27:17.358883+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:45:55.893284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:23.522006+0000","last_change":"2024-04-02T14:32:22.545502+0000","last_active":"2024-04-02T14:32:23.522006+0000","last_peered":"2024-04-02T14:32:23.522006+0000","last_clean":"2024-04-02T14:32:23.522006+0000","last_became_active":"2024-04-02T14:32:22.545111+0000","last_became_peered":"2024-04-02T14:32:22.545111+0000","last_unstale":"2024-04-02T14:32:23.522006+0000","last_undegraded":"2024-04-02T14:32:23.522006+0000","last_fullsized":"2024-04-02T14:32:23.522006+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:02:32.178334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"88'1","reported_seq":16,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:25.417261+0000","last_change":"2024-04-02T14:32:21.842665+0000","last_active":"2024-04-02T14:32:25.417261+0000","last_peered":"2024-04-02T14:32:25.417261+0000","last_clean":"2024-04-02T14:32:25.417261+0000","last_became_active":"2024-04-02T14:32:21.842284+0000","last_became_peered":"2024-04-02T14:32:21.842284+0000","last_unstale":"2024-04-02T14:32:25.417261+0000","last_undegraded":"2024-04-02T14:32:25.417261+0000","last_fullsized":"2024-04-02T14:32:25.417261+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:05:59.547063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"88'1","reported_seq":16,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:24.858099+0000","last_change":"2024-04-02T14:32:21.860226+0000","last_active":"2024-04-02T14:32:24.858099+0000","last_peered":"2024-04-02T14:32:24.858099+0000","last_clean":"2024-04-02T14:32:24.858099+0000","last_became_active":"2024-04-02T14:32:21.859854+0000","last_became_peered":"2024-04-02T14:32:21.859854+0000","last_unstale":"2024-04-02T14:32:24.858099+0000","last_undegraded":"2024-04-02T14:32:24.858099+0000","last_fullsized":"2024-04-02T14:32:24.858099+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:30:36.710677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:23.523607+0000","last_change":"2024-04-02T14:32:22.552287+0000","last_active":"2024-04-02T14:32:23.523607+0000","last_peered":"2024-04-02T14:32:23.523607+0000","last_clean":"2024-04-02T14:32:23.523607+0000","last_became_active":"2024-04-02T14:32:22.551719+0000","last_became_peered":"2024-04-02T14:32:22.551719+0000","last_unstale":"2024-04-02T14:32:23.523607+0000","last_undegraded":"2024-04-02T14:32:23.523607+0000","last_fullsized":"2024-04-02T14:32:23.523607+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:26:12.007600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"88'8","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:27.947247+0000","last_change":"2024-04-02T14:32:21.851671+0000","last_active":"2024-04-02T14:32:27.947247+0000","last_peered":"2024-04-02T14:32:27.947247+0000","last_clean":"2024-04-02T14:32:27.947247+0000","last_became_active":"2024-04-02T14:32:21.851297+0000","last_became_peered":"2024-04-02T14:32:21.851297+0000","last_unstale":"2024-04-02T14:32:27.947247+0000","last_undegraded":"2024-04-02T14:32:27.947247+0000","last_fullsized":"2024-04-02T14:32:27.947247+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:51:20.232086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:24.859825+0000","last_change":"2024-04-02T14:32:22.554554+0000","last_active":"2024-04-02T14:32:24.859825+0000","last_peered":"2024-04-02T14:32:24.859825+0000","last_clean":"2024-04-02T14:32:24.859825+0000","last_became_active":"2024-04-02T14:32:22.554194+0000","last_became_peered":"2024-04-02T14:32:22.554194+0000","last_unstale":"2024-04-02T14:32:24.859825+0000","last_undegraded":"2024-04-02T14:32:24.859825+0000","last_fullsized":"2024-04-02T14:32:24.859825+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:27:51.382441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"88'7","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:28.969404+0000","last_change":"2024-04-02T14:32:21.850346+0000","last_active":"2024-04-02T14:32:28.969404+0000","last_peered":"2024-04-02T14:32:28.969404+0000","last_clean":"2024-04-02T14:32:28.969404+0000","last_became_active":"2024-04-02T14:32:21.850084+0000","last_became_peered":"2024-04-02T14:32:21.850084+0000","last_unstale":"2024-04-02T14:32:28.969404+0000","last_undegraded":"2024-04-02T14:32:28.969404+0000","last_fullsized":"2024-04-02T14:32:28.969404+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:56:17.774702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:23.523952+0000","last_change":"2024-04-02T14:32:22.557054+0000","last_active":"2024-04-02T14:32:23.523952+0000","last_peered":"2024-04-02T14:32:23.523952+0000","last_clean":"2024-04-02T14:32:23.523952+0000","last_became_active":"2024-04-02T14:32:22.556335+0000","last_became_peered":"2024-04-02T14:32:22.556335+0000","last_unstale":"2024-04-02T14:32:23.523952+0000","last_undegraded":"2024-04-02T14:32:23.523952+0000","last_fullsized":"2024-04-02T14:32:23.523952+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:06:48.907197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"88'2","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:25.922754+0000","last_change":"2024-04-02T14:32:21.860260+0000","last_active":"2024-04-02T14:32:25.922754+0000","last_peered":"2024-04-02T14:32:25.922754+0000","last_clean":"2024-04-02T14:32:25.922754+0000","last_became_active":"2024-04-02T14:32:21.859922+0000","last_became_peered":"2024-04-02T14:32:21.859922+0000","last_unstale":"2024-04-02T14:32:25.922754+0000","last_undegraded":"2024-04-02T14:32:25.922754+0000","last_fullsized":"2024-04-02T14:32:25.922754+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:07:45.850148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:24.857752+0000","last_change":"2024-04-02T14:32:22.536063+0000","last_active":"2024-04-02T14:32:24.857752+0000","last_peered":"2024-04-02T14:32:24.857752+0000","last_clean":"2024-04-02T14:32:24.857752+0000","last_became_active":"2024-04-02T14:32:22.534678+0000","last_became_peered":"2024-04-02T14:32:22.534678+0000","last_unstale":"2024-04-02T14:32:24.857752+0000","last_undegraded":"2024-04-02T14:32:24.857752+0000","last_fullsized":"2024-04-02T14:32:24.857752+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T02:07:26.186216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"88'3","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:25.926096+0000","last_change":"2024-04-02T14:32:21.848554+0000","last_active":"2024-04-02T14:32:25.926096+0000","last_peered":"2024-04-02T14:32:25.926096+0000","last_clean":"2024-04-02T14:32:25.926096+0000","last_became_active":"2024-04-02T14:32:21.848177+0000","last_became_peered":"2024-04-02T14:32:21.848177+0000","last_unstale":"2024-04-02T14:32:25.926096+0000","last_undegraded":"2024-04-02T14:32:25.926096+0000","last_fullsized":"2024-04-02T14:32:25.926096+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:47:04.213286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:25.546560+0000","last_change":"2024-04-02T14:32:22.547246+0000","last_active":"2024-04-02T14:32:25.546560+0000","last_peered":"2024-04-02T14:32:25.546560+0000","last_clean":"2024-04-02T14:32:25.546560+0000","last_became_active":"2024-04-02T14:32:22.546721+0000","last_became_peered":"2024-04-02T14:32:22.546721+0000","last_unstale":"2024-04-02T14:32:25.546560+0000","last_undegraded":"2024-04-02T14:32:25.546560+0000","last_fullsized":"2024-04-02T14:32:25.546560+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:11:30.615723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:25.546862+0000","last_change":"2024-04-02T14:32:21.856013+0000","last_active":"2024-04-02T14:32:25.546862+0000","last_peered":"2024-04-02T14:32:25.546862+0000","last_clean":"2024-04-02T14:32:25.546862+0000","last_became_active":"2024-04-02T14:32:21.855631+0000","last_became_peered":"2024-04-02T14:32:21.855631+0000","last_unstale":"2024-04-02T14:32:25.546862+0000","last_undegraded":"2024-04-02T14:32:25.546862+0000","last_fullsized":"2024-04-02T14:32:25.546862+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:46:38.902988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:24.859562+0000","last_change":"2024-04-02T14:32:22.561787+0000","last_active":"2024-04-02T14:32:24.859562+0000","last_peered":"2024-04-02T14:32:24.859562+0000","last_clean":"2024-04-02T14:32:24.859562+0000","last_became_active":"2024-04-02T14:32:22.561284+0000","last_became_peered":"2024-04-02T14:32:22.561284+0000","last_unstale":"2024-04-02T14:32:24.859562+0000","last_undegraded":"2024-04-02T14:32:24.859562+0000","last_fullsized":"2024-04-02T14:32:24.859562+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:46:06.450097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"88'4","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:25.923327+0000","last_change":"2024-04-02T14:32:21.841340+0000","last_active":"2024-04-02T14:32:25.923327+0000","last_peered":"2024-04-02T14:32:25.923327+0000","last_clean":"2024-04-02T14:32:25.923327+0000","last_became_active":"2024-04-02T14:32:21.840937+0000","last_became_peered":"2024-04-02T14:32:21.840937+0000","last_unstale":"2024-04-02T14:32:25.923327+0000","last_undegraded":"2024-04-02T14:32:25.923327+0000","last_fullsized":"2024-04-02T14:32:25.923327+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T01:47:44.834440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:25.417350+0000","last_change":"2024-04-02T14:32:22.538890+0000","last_active":"2024-04-02T14:32:25.417350+0000","last_peered":"2024-04-02T14:32:25.417350+0000","last_clean":"2024-04-02T14:32:25.417350+0000","last_became_active":"2024-04-02T14:32:22.538512+0000","last_became_peered":"2024-04-02T14:32:22.538512+0000","last_unstale":"2024-04-02T14:32:25.417350+0000","last_undegraded":"2024-04-02T14:32:25.417350+0000","last_fullsized":"2024-04-02T14:32:25.417350+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:59:23.792851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:25.547478+0000","last_change":"2024-04-02T14:32:21.862727+0000","last_active":"2024-04-02T14:32:25.547478+0000","last_peered":"2024-04-02T14:32:25.547478+0000","last_clean":"2024-04-02T14:32:25.547478+0000","last_became_active":"2024-04-02T14:32:21.862405+0000","last_became_peered":"2024-04-02T14:32:21.862405+0000","last_unstale":"2024-04-02T14:32:25.547478+0000","last_undegraded":"2024-04-02T14:32:25.547478+0000","last_fullsized":"2024-04-02T14:32:25.547478+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:36:54.849964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:25.417353+0000","last_change":"2024-04-02T14:32:22.540077+0000","last_active":"2024-04-02T14:32:25.417353+0000","last_peered":"2024-04-02T14:32:25.417353+0000","last_clean":"2024-04-02T14:32:25.417353+0000","last_became_active":"2024-04-02T14:32:22.539513+0000","last_became_peered":"2024-04-02T14:32:22.539513+0000","last_unstale":"2024-04-02T14:32:25.417353+0000","last_undegraded":"2024-04-02T14:32:25.417353+0000","last_fullsized":"2024-04-02T14:32:25.417353+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T02:00:39.819963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"87'1","reported_seq":16,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:25.547004+0000","last_change":"2024-04-02T14:32:21.860310+0000","last_active":"2024-04-02T14:32:25.547004+0000","last_peered":"2024-04-02T14:32:25.547004+0000","last_clean":"2024-04-02T14:32:25.547004+0000","last_became_active":"2024-04-02T14:32:21.859955+0000","last_became_peered":"2024-04-02T14:32:21.859955+0000","last_unstale":"2024-04-02T14:32:25.547004+0000","last_undegraded":"2024-04-02T14:32:25.547004+0000","last_fullsized":"2024-04-02T14:32:25.547004+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:11:32.732603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:25.417069+0000","last_change":"2024-04-02T14:32:22.552134+0000","last_active":"2024-04-02T14:32:25.417069+0000","last_peered":"2024-04-02T14:32:25.417069+0000","last_clean":"2024-04-02T14:32:25.417069+0000","last_became_active":"2024-04-02T14:32:22.551695+0000","last_became_peered":"2024-04-02T14:32:22.551695+0000","last_unstale":"2024-04-02T14:32:25.417069+0000","last_undegraded":"2024-04-02T14:32:25.417069+0000","last_fullsized":"2024-04-02T14:32:25.417069+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:09:36.902243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"87'2","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:23.523415+0000","last_change":"2024-04-02T14:32:21.857571+0000","last_active":"2024-04-02T14:32:23.523415+0000","last_peered":"2024-04-02T14:32:23.523415+0000","last_clean":"2024-04-02T14:32:23.523415+0000","last_became_active":"2024-04-02T14:32:21.856935+0000","last_became_peered":"2024-04-02T14:32:21.856935+0000","last_unstale":"2024-04-02T14:32:23.523415+0000","last_undegraded":"2024-04-02T14:32:23.523415+0000","last_fullsized":"2024-04-02T14:32:23.523415+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:55:46.079097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:25.417364+0000","last_change":"2024-04-02T14:32:22.534708+0000","last_active":"2024-04-02T14:32:25.417364+0000","last_peered":"2024-04-02T14:32:25.417364+0000","last_clean":"2024-04-02T14:32:25.417364+0000","last_became_active":"2024-04-02T14:32:22.534365+0000","last_became_peered":"2024-04-02T14:32:22.534365+0000","last_unstale":"2024-04-02T14:32:25.417364+0000","last_undegraded":"2024-04-02T14:32:25.417364+0000","last_fullsized":"2024-04-02T14:32:25.417364+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:34:47.435729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:25.547029+0000","last_change":"2024-04-02T14:32:22.538317+0000","last_active":"2024-04-02T14:32:25.547029+0000","last_peered":"2024-04-02T14:32:25.547029+0000","last_clean":"2024-04-02T14:32:25.547029+0000","last_became_active":"2024-04-02T14:32:22.537939+0000","last_became_peered":"2024-04-02T14:32:22.537939+0000","last_unstale":"2024-04-02T14:32:25.547029+0000","last_undegraded":"2024-04-02T14:32:25.547029+0000","last_fullsized":"2024-04-02T14:32:25.547029+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:31:47.075326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:25.547676+0000","last_change":"2024-04-02T14:32:22.535110+0000","last_active":"2024-04-02T14:32:25.547676+0000","last_peered":"2024-04-02T14:32:25.547676+0000","last_clean":"2024-04-02T14:32:25.547676+0000","last_became_active":"2024-04-02T14:32:22.534712+0000","last_became_peered":"2024-04-02T14:32:22.534712+0000","last_unstale":"2024-04-02T14:32:25.547676+0000","last_undegraded":"2024-04-02T14:32:25.547676+0000","last_fullsized":"2024-04-02T14:32:25.547676+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:41:12.960443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:25.546825+0000","last_change":"2024-04-02T14:32:22.556567+0000","last_active":"2024-04-02T14:32:25.546825+0000","last_peered":"2024-04-02T14:32:25.546825+0000","last_clean":"2024-04-02T14:32:25.546825+0000","last_became_active":"2024-04-02T14:32:22.556267+0000","last_became_peered":"2024-04-02T14:32:22.556267+0000","last_unstale":"2024-04-02T14:32:25.546825+0000","last_undegraded":"2024-04-02T14:32:25.546825+0000","last_fullsized":"2024-04-02T14:32:25.546825+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:35:30.039013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:23.523150+0000","last_change":"2024-04-02T14:32:22.560440+0000","last_active":"2024-04-02T14:32:23.523150+0000","last_peered":"2024-04-02T14:32:23.523150+0000","last_clean":"2024-04-02T14:32:23.523150+0000","last_became_active":"2024-04-02T14:32:22.560096+0000","last_became_peered":"2024-04-02T14:32:22.560096+0000","last_unstale":"2024-04-02T14:32:23.523150+0000","last_undegraded":"2024-04-02T14:32:23.523150+0000","last_fullsized":"2024-04-02T14:32:23.523150+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:25:51.639900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:23.523603+0000","last_change":"2024-04-02T14:32:22.558584+0000","last_active":"2024-04-02T14:32:23.523603+0000","last_peered":"2024-04-02T14:32:23.523603+0000","last_clean":"2024-04-02T14:32:23.523603+0000","last_became_active":"2024-04-02T14:32:22.557382+0000","last_became_peered":"2024-04-02T14:32:22.557382+0000","last_unstale":"2024-04-02T14:32:23.523603+0000","last_undegraded":"2024-04-02T14:32:23.523603+0000","last_fullsized":"2024-04-02T14:32:23.523603+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:09:25.107411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:25.547501+0000","last_change":"2024-04-02T14:32:22.538318+0000","last_active":"2024-04-02T14:32:25.547501+0000","last_peered":"2024-04-02T14:32:25.547501+0000","last_clean":"2024-04-02T14:32:25.547501+0000","last_became_active":"2024-04-02T14:32:22.538011+0000","last_became_peered":"2024-04-02T14:32:22.538011+0000","last_unstale":"2024-04-02T14:32:25.547501+0000","last_undegraded":"2024-04-02T14:32:25.547501+0000","last_fullsized":"2024-04-02T14:32:25.547501+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:11:21.269106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:25.547730+0000","last_change":"2024-04-02T14:32:22.535645+0000","last_active":"2024-04-02T14:32:25.547730+0000","last_peered":"2024-04-02T14:32:25.547730+0000","last_clean":"2024-04-02T14:32:25.547730+0000","last_became_active":"2024-04-02T14:32:22.535314+0000","last_became_peered":"2024-04-02T14:32:22.535314+0000","last_unstale":"2024-04-02T14:32:25.547730+0000","last_undegraded":"2024-04-02T14:32:25.547730+0000","last_fullsized":"2024-04-02T14:32:25.547730+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:51:27.879015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:24.859866+0000","last_change":"2024-04-02T14:32:22.553959+0000","last_active":"2024-04-02T14:32:24.859866+0000","last_peered":"2024-04-02T14:32:24.859866+0000","last_clean":"2024-04-02T14:32:24.859866+0000","last_became_active":"2024-04-02T14:32:22.553389+0000","last_became_peered":"2024-04-02T14:32:22.553389+0000","last_unstale":"2024-04-02T14:32:24.859866+0000","last_undegraded":"2024-04-02T14:32:24.859866+0000","last_fullsized":"2024-04-02T14:32:24.859866+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:55:00.552600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:25.546914+0000","last_change":"2024-04-02T14:32:22.551769+0000","last_active":"2024-04-02T14:32:25.546914+0000","last_peered":"2024-04-02T14:32:25.546914+0000","last_clean":"2024-04-02T14:32:25.546914+0000","last_became_active":"2024-04-02T14:32:22.551422+0000","last_became_peered":"2024-04-02T14:32:22.551422+0000","last_unstale":"2024-04-02T14:32:25.546914+0000","last_undegraded":"2024-04-02T14:32:25.546914+0000","last_fullsized":"2024-04-02T14:32:25.546914+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T01:15:07.262927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:24.859021+0000","last_change":"2024-04-02T14:32:22.548264+0000","last_active":"2024-04-02T14:32:24.859021+0000","last_peered":"2024-04-02T14:32:24.859021+0000","last_clean":"2024-04-02T14:32:24.859021+0000","last_became_active":"2024-04-02T14:32:22.547293+0000","last_became_peered":"2024-04-02T14:32:22.547293+0000","last_unstale":"2024-04-02T14:32:24.859021+0000","last_undegraded":"2024-04-02T14:32:24.859021+0000","last_fullsized":"2024-04-02T14:32:24.859021+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:53:19.896243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:25.546651+0000","last_change":"2024-04-02T14:32:22.556131+0000","last_active":"2024-04-02T14:32:25.546651+0000","last_peered":"2024-04-02T14:32:25.546651+0000","last_clean":"2024-04-02T14:32:25.546651+0000","last_became_active":"2024-04-02T14:32:22.555763+0000","last_became_peered":"2024-04-02T14:32:22.555763+0000","last_unstale":"2024-04-02T14:32:25.546651+0000","last_undegraded":"2024-04-02T14:32:25.546651+0000","last_fullsized":"2024-04-02T14:32:25.546651+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:45:22.485403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:23.523329+0000","last_change":"2024-04-02T14:32:22.562093+0000","last_active":"2024-04-02T14:32:23.523329+0000","last_peered":"2024-04-02T14:32:23.523329+0000","last_clean":"2024-04-02T14:32:23.523329+0000","last_became_active":"2024-04-02T14:32:22.561663+0000","last_became_peered":"2024-04-02T14:32:22.561663+0000","last_unstale":"2024-04-02T14:32:23.523329+0000","last_undegraded":"2024-04-02T14:32:23.523329+0000","last_fullsized":"2024-04-02T14:32:23.523329+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:14:25.209276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-02T14:32:25.546909+0000","last_change":"2024-04-02T14:32:22.541964+0000","last_active":"2024-04-02T14:32:25.546909+0000","last_peered":"2024-04-02T14:32:25.546909+0000","last_clean":"2024-04-02T14:32:25.546909+0000","last_became_active":"2024-04-02T14:32:22.541549+0000","last_became_peered":"2024-04-02T14:32:22.541549+0000","last_unstale":"2024-04-02T14:32:25.546909+0000","last_undegraded":"2024-04-02T14:32:25.546909+0000","last_fullsized":"2024-04-02T14:32:25.546909+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:04:56.783158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":24258,"num_objects":59,"num_object_clones":0,"num_object_copies":177,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":59,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":139,"num_write_kb":54,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":274432,"data_stored":36331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":118,"ondisk_log_size":118,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":83,"seq":356482285583,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32116,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713036,"statfs":{"total":95995035648,"available":95962148864,"internally_reserved":0,"allocated":962560,"data_stored":457516,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":75,"seq":322122547221,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36916,"kb_used_data":1580,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708236,"statfs":{"total":95995035648,"available":95957233664,"internally_reserved":0,"allocated":1617920,"data_stored":1069797,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":36174273},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743449,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36288,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708864,"statfs":{"total":95995035648,"available":95957876736,"internally_reserved":0,"allocated":974848,"data_stored":457232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":61,"seq":261993005088,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32116,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713036,"statfs":{"total":95995035648,"available":95962148864,"internally_reserved":0,"allocated":958464,"data_stored":456928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299429,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32696,"kb_used_data":1520,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712456,"statfs":{"total":95995035648,"available":95961554944,"internally_reserved":0,"allocated":1556480,"data_stored":1047896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495657,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36276,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708876,"statfs":{"total":95995035648,"available":95957889024,"internally_reserved":0,"allocated":950272,"data_stored":456324,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574919,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36280,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708872,"statfs":{"total":95995035648,"available":95957884928,"internally_reserved":0,"allocated":962560,"data_stored":457936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738444,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36296,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708856,"statfs":{"total":95995035648,"available":95957868544,"internally_reserved":0,"allocated":983040,"data_stored":458390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411393,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36292,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708860,"statfs":{"total":95995035648,"available":95957872640,"internally_reserved":0,"allocated":974848,"data_stored":458592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182458,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27960,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717192,"statfs":{"total":95995035648,"available":95966404608,"internally_reserved":0,"allocated":962560,"data_stored":456518,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920823,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32124,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713028,"statfs":{"total":95995035648,"available":95962140672,"internally_reserved":0,"allocated":970752,"data_stored":457415,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757295,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28552,"kb_used_data":1536,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716600,"statfs":{"total":95995035648,"available":95965798400,"internally_reserved":0,"allocated":1572864,"data_stored":1048168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":2066,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":23022,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-02T14:32:32.356 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-02T14:32:32.356 INFO:tasks.cephadm:Setting up client nodes... 2024-04-02T14:32:32.357 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:191d963ac9de7322da8aff20b567b3fa458e2ef6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6449448a-f0fc-11ee-b647-cb9ed24678a4 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-04-02T14:32:32.360 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:31 smithi092 ceph-mon[36497]: from='client.? 172.21.15.90:0/563069497' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-02T14:32:32.360 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:31 smithi092 ceph-mon[36497]: from='client.24722 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-02T14:32:32.436 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:31 smithi177 ceph-mon[36431]: from='client.? 172.21.15.90:0/563069497' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-02T14:32:32.437 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:31 smithi177 ceph-mon[36431]: from='client.24722 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-02T14:32:32.837 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T14:32:33.217 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:32 smithi090 ceph-mon[31674]: from='client.24728 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-02T14:32:33.217 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:32 smithi090 ceph-mon[31674]: pgmap v317: 97 pgs: 97 active+clean; 603 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 7.8 KiB/s wr, 23 op/s 2024-04-02T14:32:33.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:32 smithi092 ceph-mon[36497]: from='client.24728 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-02T14:32:33.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:32 smithi092 ceph-mon[36497]: pgmap v317: 97 pgs: 97 active+clean; 603 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 7.8 KiB/s wr, 23 op/s 2024-04-02T14:32:33.361 INFO:journalctl@ceph.mds.b.smithi092.stdout:Apr 02 14:32:33 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mds-b[67422]: 2024-04-02T14:32:33.122+0000 7f208e2f9700 -1 mds.pinger is_rank_lagging: rank=3 was never sent ping request. 2024-04-02T14:32:33.361 INFO:journalctl@ceph.mds.b.smithi092.stdout:Apr 02 14:32:33 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mds-b[67422]: 2024-04-02T14:32:33.122+0000 7f208e2f9700 -1 mds.pinger is_rank_lagging: rank=4 was never sent ping request. 2024-04-02T14:32:33.436 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:32 smithi177 ceph-mon[36431]: from='client.24728 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-02T14:32:33.437 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:32 smithi177 ceph-mon[36431]: pgmap v317: 97 pgs: 97 active+clean; 603 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 7.8 KiB/s wr, 23 op/s 2024-04-02T14:32:34.957 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:34 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [7, 11]}]: dispatch 2024-04-02T14:32:34.958 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:34 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]: dispatch 2024-04-02T14:32:34.958 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:34 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.14", "id": [2, 4]}]: dispatch 2024-04-02T14:32:34.958 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:34 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [8, 4]}]: dispatch 2024-04-02T14:32:34.958 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:34 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [3, 9]}]: dispatch 2024-04-02T14:32:34.958 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:34 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 8, 5, 9]}]: dispatch 2024-04-02T14:32:34.958 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:34 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [5, 9]}]: dispatch 2024-04-02T14:32:34.958 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:34 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.23", "id": [4, 7]}]: dispatch 2024-04-02T14:32:34.959 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:34 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3a", "id": [4, 6]}]: dispatch 2024-04-02T14:32:35.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:34 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [7, 11]}]: dispatch 2024-04-02T14:32:35.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:34 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]: dispatch 2024-04-02T14:32:35.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:34 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.14", "id": [2, 4]}]: dispatch 2024-04-02T14:32:35.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:34 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [8, 4]}]: dispatch 2024-04-02T14:32:35.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:34 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [3, 9]}]: dispatch 2024-04-02T14:32:35.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:34 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 8, 5, 9]}]: dispatch 2024-04-02T14:32:35.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:34 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [5, 9]}]: dispatch 2024-04-02T14:32:35.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:34 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.23", "id": [4, 7]}]: dispatch 2024-04-02T14:32:35.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:34 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3a", "id": [4, 6]}]: dispatch 2024-04-02T14:32:35.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:34 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [7, 11]}]: dispatch 2024-04-02T14:32:35.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:34 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]: dispatch 2024-04-02T14:32:35.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:34 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.14", "id": [2, 4]}]: dispatch 2024-04-02T14:32:35.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:34 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [8, 4]}]: dispatch 2024-04-02T14:32:35.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:34 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [3, 9]}]: dispatch 2024-04-02T14:32:35.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:34 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 8, 5, 9]}]: dispatch 2024-04-02T14:32:35.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:34 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [5, 9]}]: dispatch 2024-04-02T14:32:35.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:34 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.23", "id": [4, 7]}]: dispatch 2024-04-02T14:32:35.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:34 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3a", "id": [4, 6]}]: dispatch 2024-04-02T14:32:35.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:35.231+0000 7fe734400700 1 -- 172.21.15.90:0/1664220181 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe72c101eb0 msgr2=0x7fe72c102340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:32:35.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:35.231+0000 7fe734400700 1 --2- 172.21.15.90:0/1664220181 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe72c101eb0 0x7fe72c102340 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fe724009990 tx=0x7fe724009ca0 comp rx=0 tx=0).stop 2024-04-02T14:32:35.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:35.231+0000 7fe734400700 1 -- 172.21.15.90:0/1664220181 shutdown_connections 2024-04-02T14:32:35.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:35.231+0000 7fe734400700 1 --2- 172.21.15.90:0/1664220181 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe72c101eb0 0x7fe72c102340 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:35.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:35.231+0000 7fe734400700 1 --2- 172.21.15.90:0/1664220181 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe72c069a10 0x7fe72c100cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:35.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:35.231+0000 7fe734400700 1 --2- 172.21.15.90:0/1664220181 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe72c069ec0 0x7fe72c101230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:35.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:35.231+0000 7fe734400700 1 -- 172.21.15.90:0/1664220181 >> 172.21.15.90:0/1664220181 conn(0x7fe72c0fc290 msgr2=0x7fe72c0fe6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:32:35.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:35.231+0000 7fe734400700 1 -- 172.21.15.90:0/1664220181 shutdown_connections 2024-04-02T14:32:35.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:35.231+0000 7fe734400700 1 -- 172.21.15.90:0/1664220181 wait complete. 2024-04-02T14:32:35.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:35.233+0000 7fe734400700 1 Processor -- start 2024-04-02T14:32:35.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:35.233+0000 7fe734400700 1 -- start start 2024-04-02T14:32:35.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:35.234+0000 7fe734400700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe72c069a10 0x7fe72c1986d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:32:35.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:35.234+0000 7fe734400700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe72c069ec0 0x7fe72c198c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:32:35.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:35.234+0000 7fe734400700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe72c101eb0 0x7fe72c19fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:32:35.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:35.234+0000 7fe734400700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fe72c10d120 con 0x7fe72c069a10 2024-04-02T14:32:35.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:35.234+0000 7fe734400700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fe72c10cfa0 con 0x7fe72c069ec0 2024-04-02T14:32:35.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:35.234+0000 7fe734400700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fe72c10d2a0 con 0x7fe72c101eb0 2024-04-02T14:32:35.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:35.234+0000 7fe731a00700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe72c069a10 0x7fe72c1986d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:32:35.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:35.234+0000 7fe731000700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe72c069ec0 0x7fe72c198c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:32:35.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:35.234+0000 7fe731a00700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe72c069a10 0x7fe72c1986d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:47964/0 (socket says 172.21.15.90:47964) 2024-04-02T14:32:35.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:35.234+0000 7fe731a00700 1 -- 172.21.15.90:0/670822779 learned_addr learned my addr 172.21.15.90:0/670822779 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:32:35.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:35.234+0000 7fe732400700 1 --2- 172.21.15.90:0/670822779 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe72c101eb0 0x7fe72c19fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:32:35.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:35.234+0000 7fe731000700 1 -- 172.21.15.90:0/670822779 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe72c101eb0 msgr2=0x7fe72c19fc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:32:35.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:35.234+0000 7fe731000700 1 --2- 172.21.15.90:0/670822779 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe72c101eb0 0x7fe72c19fc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:35.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:35.234+0000 7fe731000700 1 -- 172.21.15.90:0/670822779 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe72c069a10 msgr2=0x7fe72c1986d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:32:35.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:35.234+0000 7fe731000700 1 --2- 172.21.15.90:0/670822779 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe72c069a10 0x7fe72c1986d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:35.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:35.234+0000 7fe731000700 1 -- 172.21.15.90:0/670822779 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe72c1a0330 con 0x7fe72c069ec0 2024-04-02T14:32:35.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:35.234+0000 7fe731a00700 1 --2- 172.21.15.90:0/670822779 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe72c069a10 0x7fe72c1986d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:32:35.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:35.234+0000 7fe732400700 1 --2- 172.21.15.90:0/670822779 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe72c101eb0 0x7fe72c19fc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:32:35.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:35.235+0000 7fe731000700 1 --2- 172.21.15.90:0/670822779 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe72c069ec0 0x7fe72c198c10 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fe71800d720 tx=0x7fe71800da30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:32:35.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:35.235+0000 7fe72a000700 1 -- 172.21.15.90:0/670822779 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe718014030 con 0x7fe72c069ec0 2024-04-02T14:32:35.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:35.235+0000 7fe734400700 1 -- 172.21.15.90:0/670822779 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe72c1a0620 con 0x7fe72c069ec0 2024-04-02T14:32:35.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:35.235+0000 7fe72a000700 1 -- 172.21.15.90:0/670822779 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe718004500 con 0x7fe72c069ec0 2024-04-02T14:32:35.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:35.235+0000 7fe72a000700 1 -- 172.21.15.90:0/670822779 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7180054c0 con 0x7fe72c069ec0 2024-04-02T14:32:35.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:35.235+0000 7fe734400700 1 -- 172.21.15.90:0/670822779 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe72c1a0b60 con 0x7fe72c069ec0 2024-04-02T14:32:35.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:35.236+0000 7fe734400700 1 -- 172.21.15.90:0/670822779 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe72c050b00 con 0x7fe72c069ec0 2024-04-02T14:32:35.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:35.238+0000 7fe72a000700 1 -- 172.21.15.90:0/670822779 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fe718005660 con 0x7fe72c069ec0 2024-04-02T14:32:35.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:35.240+0000 7fe72a000700 1 --2- 172.21.15.90:0/670822779 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fe7140a3d80 0x7fe7140a6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:32:35.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:35.240+0000 7fe72a000700 1 -- 172.21.15.90:0/670822779 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8248+0+0 (secure 0 0 0) 0x7fe7180d02e0 con 0x7fe72c069ec0 2024-04-02T14:32:35.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:35.240+0000 7fe731a00700 1 --2- 172.21.15.90:0/670822779 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fe7140a3d80 0x7fe7140a6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:32:35.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:35.241+0000 7fe731a00700 1 --2- 172.21.15.90:0/670822779 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fe7140a3d80 0x7fe7140a6240 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fe720003ef0 tx=0x7fe72000b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:32:35.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:35.242+0000 7fe72a000700 1 -- 172.21.15.90:0/670822779 <== mon.2 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7fe71809caf0 con 0x7fe72c069ec0 2024-04-02T14:32:35.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:35.563+0000 7fe734400700 1 -- 172.21.15.90:0/670822779 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) -- 0x7fe72c02ddb0 con 0x7fe72c069ec0 2024-04-02T14:32:35.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:35.571+0000 7fe72a000700 1 -- 172.21.15.90:0/670822779 <== mon.2 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v32) ==== 170+0+59 (secure 0 0 0) 0x7fe71809c3f0 con 0x7fe72c069ec0 2024-04-02T14:32:35.573 INFO:teuthology.orchestra.run.smithi090.stdout:[client.0] 2024-04-02T14:32:35.573 INFO:teuthology.orchestra.run.smithi090.stdout: key = AQADFwxm3j/AIRAAywWlYSvioxqGYqUqRNeyQA== 2024-04-02T14:32:35.576 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:35.574+0000 7fe734400700 1 -- 172.21.15.90:0/670822779 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fe7140a3d80 msgr2=0x7fe7140a6240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:32:35.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:35.574+0000 7fe734400700 1 --2- 172.21.15.90:0/670822779 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fe7140a3d80 0x7fe7140a6240 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fe720003ef0 tx=0x7fe72000b040 comp rx=0 tx=0).stop 2024-04-02T14:32:35.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:35.574+0000 7fe734400700 1 -- 172.21.15.90:0/670822779 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe72c069ec0 msgr2=0x7fe72c198c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:32:35.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:35.574+0000 7fe734400700 1 --2- 172.21.15.90:0/670822779 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe72c069ec0 0x7fe72c198c10 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fe71800d720 tx=0x7fe71800da30 comp rx=0 tx=0).stop 2024-04-02T14:32:35.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:35.574+0000 7fe734400700 1 -- 172.21.15.90:0/670822779 shutdown_connections 2024-04-02T14:32:35.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:35.574+0000 7fe734400700 1 --2- 172.21.15.90:0/670822779 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fe7140a3d80 0x7fe7140a6240 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:35.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:35.574+0000 7fe734400700 1 --2- 172.21.15.90:0/670822779 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe72c069a10 0x7fe72c1986d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:35.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:35.574+0000 7fe734400700 1 --2- 172.21.15.90:0/670822779 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe72c101eb0 0x7fe72c19fc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:35.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:35.574+0000 7fe734400700 1 --2- 172.21.15.90:0/670822779 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe72c069ec0 0x7fe72c198c10 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:35.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:35.574+0000 7fe734400700 1 -- 172.21.15.90:0/670822779 >> 172.21.15.90:0/670822779 conn(0x7fe72c0fc290 msgr2=0x7fe72c0fd650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:32:35.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:35.574+0000 7fe734400700 1 -- 172.21.15.90:0/670822779 shutdown_connections 2024-04-02T14:32:35.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:35.574+0000 7fe734400700 1 -- 172.21.15.90:0/670822779 wait complete. 2024-04-02T14:32:35.939 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:35 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [7, 11]}]': finished 2024-04-02T14:32:35.939 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:35 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]': finished 2024-04-02T14:32:35.939 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:35 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.14", "id": [2, 4]}]': finished 2024-04-02T14:32:35.939 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:35 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [8, 4]}]': finished 2024-04-02T14:32:35.940 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:35 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [3, 9]}]': finished 2024-04-02T14:32:35.940 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:35 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 8, 5, 9]}]': finished 2024-04-02T14:32:35.940 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:35 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [5, 9]}]': finished 2024-04-02T14:32:35.940 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:35 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.23", "id": [4, 7]}]': finished 2024-04-02T14:32:35.940 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:35 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3a", "id": [4, 6]}]': finished 2024-04-02T14:32:35.940 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:35 smithi090 ceph-mon[31674]: osdmap e89: 12 total, 12 up, 12 in 2024-04-02T14:32:35.940 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:35 smithi090 ceph-mon[31674]: pgmap v319: 97 pgs: 97 active+clean; 626 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 9.6 KiB/s rd, 11 KiB/s wr, 28 op/s 2024-04-02T14:32:35.940 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:35 smithi090 ceph-mon[31674]: from='client.? 172.21.15.90:0/670822779' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-02T14:32:35.940 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:35 smithi090 ceph-mon[31674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-02T14:32:35.940 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:35 smithi090 ceph-mon[31674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-02T14:32:35.940 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:35 smithi090 ceph-mon[31674]: osdmap e90: 12 total, 12 up, 12 in 2024-04-02T14:32:36.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:35 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [7, 11]}]': finished 2024-04-02T14:32:36.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:35 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]': finished 2024-04-02T14:32:36.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:35 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.14", "id": [2, 4]}]': finished 2024-04-02T14:32:36.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:35 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [8, 4]}]': finished 2024-04-02T14:32:36.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:35 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [3, 9]}]': finished 2024-04-02T14:32:36.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:35 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 8, 5, 9]}]': finished 2024-04-02T14:32:36.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:35 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [5, 9]}]': finished 2024-04-02T14:32:36.112 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:35 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.23", "id": [4, 7]}]': finished 2024-04-02T14:32:36.112 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:35 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3a", "id": [4, 6]}]': finished 2024-04-02T14:32:36.112 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:35 smithi092 ceph-mon[36497]: osdmap e89: 12 total, 12 up, 12 in 2024-04-02T14:32:36.112 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:35 smithi092 ceph-mon[36497]: pgmap v319: 97 pgs: 97 active+clean; 626 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 9.6 KiB/s rd, 11 KiB/s wr, 28 op/s 2024-04-02T14:32:36.112 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:35 smithi092 ceph-mon[36497]: from='client.? 172.21.15.90:0/670822779' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-02T14:32:36.112 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:35 smithi092 ceph-mon[36497]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-02T14:32:36.112 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:35 smithi092 ceph-mon[36497]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-02T14:32:36.112 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:35 smithi092 ceph-mon[36497]: osdmap e90: 12 total, 12 up, 12 in 2024-04-02T14:32:36.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:35 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [7, 11]}]': finished 2024-04-02T14:32:36.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:35 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]': finished 2024-04-02T14:32:36.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:35 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.14", "id": [2, 4]}]': finished 2024-04-02T14:32:36.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:35 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [8, 4]}]': finished 2024-04-02T14:32:36.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:35 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [3, 9]}]': finished 2024-04-02T14:32:36.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:35 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 8, 5, 9]}]': finished 2024-04-02T14:32:36.188 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:35 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [5, 9]}]': finished 2024-04-02T14:32:36.188 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:35 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.23", "id": [4, 7]}]': finished 2024-04-02T14:32:36.188 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:35 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3a", "id": [4, 6]}]': finished 2024-04-02T14:32:36.188 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:35 smithi177 ceph-mon[36431]: osdmap e89: 12 total, 12 up, 12 in 2024-04-02T14:32:36.188 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:35 smithi177 ceph-mon[36431]: pgmap v319: 97 pgs: 97 active+clean; 626 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 9.6 KiB/s rd, 11 KiB/s wr, 28 op/s 2024-04-02T14:32:36.188 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:35 smithi177 ceph-mon[36431]: from='client.? 172.21.15.90:0/670822779' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-02T14:32:36.188 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:35 smithi177 ceph-mon[36431]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-02T14:32:36.188 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:35 smithi177 ceph-mon[36431]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-02T14:32:36.188 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:35 smithi177 ceph-mon[36431]: osdmap e90: 12 total, 12 up, 12 in 2024-04-02T14:32:36.507 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-04-02T14:32:36.507 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-04-02T14:32:36.507 DEBUG:teuthology.orchestra.run.smithi090:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-04-02T14:32:36.659 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-04-02T14:32:37.044 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-04-02T14:32:37.045 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:191d963ac9de7322da8aff20b567b3fa458e2ef6 shell --fsid 6449448a-f0fc-11ee-b647-cb9ed24678a4 -- ceph mgr dump --format=json 2024-04-02T14:32:37.493 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T14:32:37.939 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:37 smithi090 ceph-mon[31674]: pgmap v321: 97 pgs: 2 peering, 95 active+clean; 626 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 12 KiB/s wr, 30 op/s 2024-04-02T14:32:38.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:37 smithi092 ceph-mon[36497]: pgmap v321: 97 pgs: 2 peering, 95 active+clean; 626 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 12 KiB/s wr, 30 op/s 2024-04-02T14:32:38.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:37 smithi177 ceph-mon[36431]: pgmap v321: 97 pgs: 2 peering, 95 active+clean; 626 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 12 KiB/s wr, 30 op/s 2024-04-02T14:32:39.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:39.590+0000 7f9576400700 1 -- 172.21.15.90:0/892803376 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9570102640 msgr2=0x7f9570102ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:32:39.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:39.590+0000 7f9576400700 1 --2- 172.21.15.90:0/892803376 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9570102640 0x7f9570102ac0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f955c009990 tx=0x7f955c009ca0 comp rx=0 tx=0).stop 2024-04-02T14:32:39.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:39.591+0000 7f9576400700 1 -- 172.21.15.90:0/892803376 shutdown_connections 2024-04-02T14:32:39.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:39.591+0000 7f9576400700 1 --2- 172.21.15.90:0/892803376 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9570103000 0x7f957010f500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:39.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:39.591+0000 7f9576400700 1 --2- 172.21.15.90:0/892803376 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9570108630 0x7f9570108a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:39.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:39.591+0000 7f9576400700 1 --2- 172.21.15.90:0/892803376 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9570102640 0x7f9570102ac0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:39.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:39.591+0000 7f9576400700 1 -- 172.21.15.90:0/892803376 >> 172.21.15.90:0/892803376 conn(0x7f9570079750 msgr2=0x7f9570079b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:32:39.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:39.591+0000 7f9576400700 1 -- 172.21.15.90:0/892803376 shutdown_connections 2024-04-02T14:32:39.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:39.591+0000 7f9576400700 1 -- 172.21.15.90:0/892803376 wait complete. 2024-04-02T14:32:39.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:39.592+0000 7f9576400700 1 Processor -- start 2024-04-02T14:32:39.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:39.592+0000 7f9576400700 1 -- start start 2024-04-02T14:32:39.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:39.592+0000 7f9576400700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9570102640 0x7f957019b100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:32:39.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:39.592+0000 7f9576400700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9570103000 0x7f957019b640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:32:39.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:39.593+0000 7f9576400700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9570108630 0x7f957019f9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:32:39.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:39.593+0000 7f9576400700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f9570112a10 con 0x7f9570102640 2024-04-02T14:32:39.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:39.593+0000 7f9576400700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f9570112890 con 0x7f9570108630 2024-04-02T14:32:39.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:39.593+0000 7f9576400700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f9570112b90 con 0x7f9570103000 2024-04-02T14:32:39.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:39.593+0000 7f956f400700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9570102640 0x7f957019b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:32:39.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:39.593+0000 7f956f400700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9570102640 0x7f957019b100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:47988/0 (socket says 172.21.15.90:47988) 2024-04-02T14:32:39.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:39.593+0000 7f956f400700 1 -- 172.21.15.90:0/49907906 learned_addr learned my addr 172.21.15.90:0/49907906 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:32:39.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:39.593+0000 7f956ea00700 1 --2- 172.21.15.90:0/49907906 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9570103000 0x7f957019b640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:32:39.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:39.593+0000 7f956fe00700 1 --2- 172.21.15.90:0/49907906 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9570108630 0x7f957019f9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:32:39.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:39.593+0000 7f956f400700 1 -- 172.21.15.90:0/49907906 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9570103000 msgr2=0x7f957019b640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:32:39.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:39.593+0000 7f956f400700 1 --2- 172.21.15.90:0/49907906 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9570103000 0x7f957019b640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:39.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:39.593+0000 7f956f400700 1 -- 172.21.15.90:0/49907906 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9570108630 msgr2=0x7f957019f9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:32:39.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:39.593+0000 7f956f400700 1 --2- 172.21.15.90:0/49907906 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9570108630 0x7f957019f9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:39.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:39.593+0000 7f956f400700 1 -- 172.21.15.90:0/49907906 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95701a0110 con 0x7f9570102640 2024-04-02T14:32:39.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:39.593+0000 7f956ea00700 1 --2- 172.21.15.90:0/49907906 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9570103000 0x7f957019b640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:32:39.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:39.593+0000 7f956fe00700 1 --2- 172.21.15.90:0/49907906 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9570108630 0x7f957019f9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:32:39.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:39.594+0000 7f956f400700 1 --2- 172.21.15.90:0/49907906 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9570102640 0x7f957019b100 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f956000e9a0 tx=0x7f956000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:32:39.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:39.594+0000 7f9567e00700 1 -- 172.21.15.90:0/49907906 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f956000ca10 con 0x7f9570102640 2024-04-02T14:32:39.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:39.594+0000 7f9567e00700 1 -- 172.21.15.90:0/49907906 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f956000cbb0 con 0x7f9570102640 2024-04-02T14:32:39.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:39.594+0000 7f9576400700 1 -- 172.21.15.90:0/49907906 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f95701a0400 con 0x7f9570102640 2024-04-02T14:32:39.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:39.594+0000 7f9567e00700 1 -- 172.21.15.90:0/49907906 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9560018680 con 0x7f9570102640 2024-04-02T14:32:39.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:39.594+0000 7f9576400700 1 -- 172.21.15.90:0/49907906 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95701a7c80 con 0x7f9570102640 2024-04-02T14:32:39.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:39.596+0000 7f9576400700 1 -- 172.21.15.90:0/49907906 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f95700692b0 con 0x7f9570102640 2024-04-02T14:32:39.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:39.597+0000 7f9567e00700 1 -- 172.21.15.90:0/49907906 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f9560010430 con 0x7f9570102640 2024-04-02T14:32:39.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:39.598+0000 7f9567e00700 1 --2- 172.21.15.90:0/49907906 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f95540a3cd0 0x7f95540a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:32:39.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:39.599+0000 7f9567e00700 1 -- 172.21.15.90:0/49907906 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8248+0+0 (secure 0 0 0) 0x7f95600d07d0 con 0x7f9570102640 2024-04-02T14:32:39.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:39.599+0000 7f956ea00700 1 --2- 172.21.15.90:0/49907906 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f95540a3cd0 0x7f95540a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:32:39.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:39.600+0000 7f956ea00700 1 --2- 172.21.15.90:0/49907906 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f95540a3cd0 0x7f95540a6190 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f957010f890 tx=0x7f955c005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:32:39.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:39.603+0000 7f9567e00700 1 -- 172.21.15.90:0/49907906 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7f956009cfe0 con 0x7f9570102640 2024-04-02T14:32:39.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:39.904+0000 7f9576400700 1 -- 172.21.15.90:0/49907906 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7f95701a7e50 con 0x7f9570102640 2024-04-02T14:32:39.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:39.911+0000 7f9567e00700 1 -- 172.21.15.90:0/49907906 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v17) ==== 74+0+264998 (secure 0 0 0) 0x7f956009c8e0 con 0x7f9570102640 2024-04-02T14:32:39.914 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-02T14:32:39.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:39.916+0000 7f9576400700 1 -- 172.21.15.90:0/49907906 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f95540a3cd0 msgr2=0x7f95540a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:32:39.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:39.916+0000 7f9576400700 1 --2- 172.21.15.90:0/49907906 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f95540a3cd0 0x7f95540a6190 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f957010f890 tx=0x7f955c005fd0 comp rx=0 tx=0).stop 2024-04-02T14:32:39.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:39.916+0000 7f9576400700 1 -- 172.21.15.90:0/49907906 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9570102640 msgr2=0x7f957019b100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:32:39.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:39.916+0000 7f9576400700 1 --2- 172.21.15.90:0/49907906 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9570102640 0x7f957019b100 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f956000e9a0 tx=0x7f956000ecb0 comp rx=0 tx=0).stop 2024-04-02T14:32:39.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:39.917+0000 7f9576400700 1 -- 172.21.15.90:0/49907906 shutdown_connections 2024-04-02T14:32:39.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:39.917+0000 7f9576400700 1 --2- 172.21.15.90:0/49907906 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f95540a3cd0 0x7f95540a6190 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:39.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:39.917+0000 7f9576400700 1 --2- 172.21.15.90:0/49907906 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9570102640 0x7f957019b100 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:39.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:39.917+0000 7f9576400700 1 --2- 172.21.15.90:0/49907906 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9570108630 0x7f957019f9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:39.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:39.917+0000 7f9576400700 1 --2- 172.21.15.90:0/49907906 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9570103000 0x7f957019b640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:39.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:39.917+0000 7f9576400700 1 -- 172.21.15.90:0/49907906 >> 172.21.15.90:0/49907906 conn(0x7f9570079750 msgr2=0x7f95701004b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:32:39.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:39.917+0000 7f9576400700 1 -- 172.21.15.90:0/49907906 shutdown_connections 2024-04-02T14:32:39.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:39.917+0000 7f9576400700 1 -- 172.21.15.90:0/49907906 wait complete. 2024-04-02T14:32:40.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:39 smithi092 ceph-mon[36497]: pgmap v322: 97 pgs: 2 peering, 95 active+clean; 626 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 7.4 KiB/s wr, 21 op/s 2024-04-02T14:32:40.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:39 smithi177 ceph-mon[36431]: pgmap v322: 97 pgs: 2 peering, 95 active+clean; 626 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 7.4 KiB/s wr, 21 op/s 2024-04-02T14:32:40.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:39 smithi090 ceph-mon[31674]: pgmap v322: 97 pgs: 2 peering, 95 active+clean; 626 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 7.4 KiB/s wr, 21 op/s 2024-04-02T14:32:40.819 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":17,"flags":0,"active_gid":14152,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":4266054817},{"type":"v1","addr":"172.21.15.90:6801","nonce":4266054817}]},"active_addr":"172.21.15.90:6801/4266054817","active_change":"2024-04-02T14:24:34.595672+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24109,"name":"z","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24110,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.90:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.90:0","nonce":1757385857}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.90:0","nonce":1431307002}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.90:0","nonce":2805994468}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.90:0","nonce":1335673697}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.90:0","nonce":1067118691}]}]} 2024-04-02T14:32:40.824 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-04-02T14:32:40.825 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-04-02T14:32:40.825 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:191d963ac9de7322da8aff20b567b3fa458e2ef6 shell --fsid 6449448a-f0fc-11ee-b647-cb9ed24678a4 -- ceph osd dump --format=json 2024-04-02T14:32:41.110 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:40 smithi092 ceph-mon[36497]: from='client.? 172.21.15.90:0/49907906' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-02T14:32:41.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:40 smithi177 ceph-mon[36431]: from='client.? 172.21.15.90:0/49907906' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-02T14:32:41.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:40 smithi090 ceph-mon[31674]: from='client.? 172.21.15.90:0/49907906' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-02T14:32:41.247 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T14:32:42.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:41 smithi092 ceph-mon[36497]: pgmap v323: 97 pgs: 97 active+clean; 638 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 6.4 KiB/s wr, 13 op/s; 5.4 KiB/s, 1 objects/s recovering 2024-04-02T14:32:42.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:41 smithi177 ceph-mon[36431]: pgmap v323: 97 pgs: 97 active+clean; 638 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 6.4 KiB/s wr, 13 op/s; 5.4 KiB/s, 1 objects/s recovering 2024-04-02T14:32:42.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:41 smithi090 ceph-mon[31674]: pgmap v323: 97 pgs: 97 active+clean; 638 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 6.4 KiB/s wr, 13 op/s; 5.4 KiB/s, 1 objects/s recovering 2024-04-02T14:32:43.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:43.382+0000 7fb5c8c00700 1 -- 172.21.15.90:0/3805995044 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb5c41047e0 msgr2=0x7fb5c4104c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:32:43.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:43.382+0000 7fb5c8c00700 1 --2- 172.21.15.90:0/3805995044 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb5c41047e0 0x7fb5c4104c60 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fb5ac009990 tx=0x7fb5ac009ca0 comp rx=0 tx=0).stop 2024-04-02T14:32:43.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:43.382+0000 7fb5c8c00700 1 -- 172.21.15.90:0/3805995044 shutdown_connections 2024-04-02T14:32:43.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:43.382+0000 7fb5c8c00700 1 --2- 172.21.15.90:0/3805995044 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb5c41051a0 0x7fb5c41116b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:43.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:43.382+0000 7fb5c8c00700 1 --2- 172.21.15.90:0/3805995044 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb5c410a7e0 0x7fb5c410abc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:43.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:43.382+0000 7fb5c8c00700 1 --2- 172.21.15.90:0/3805995044 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb5c41047e0 0x7fb5c4104c60 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:43.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:43.382+0000 7fb5c8c00700 1 -- 172.21.15.90:0/3805995044 >> 172.21.15.90:0/3805995044 conn(0x7fb5c41002d0 msgr2=0x7fb5c41026f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:32:43.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:43.382+0000 7fb5c8c00700 1 -- 172.21.15.90:0/3805995044 shutdown_connections 2024-04-02T14:32:43.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:43.382+0000 7fb5c8c00700 1 -- 172.21.15.90:0/3805995044 wait complete. 2024-04-02T14:32:43.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:43.383+0000 7fb5c8c00700 1 Processor -- start 2024-04-02T14:32:43.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:43.384+0000 7fb5c8c00700 1 -- start start 2024-04-02T14:32:43.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:43.384+0000 7fb5c8c00700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb5c41047e0 0x7fb5c419afd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:32:43.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:43.384+0000 7fb5c8c00700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb5c41051a0 0x7fb5c419b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:32:43.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:43.384+0000 7fb5c8c00700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb5c410a7e0 0x7fb5c419f8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:32:43.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:43.384+0000 7fb5c8c00700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fb5c4114b80 con 0x7fb5c410a7e0 2024-04-02T14:32:43.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:43.384+0000 7fb5c8c00700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fb5c4114a00 con 0x7fb5c41051a0 2024-04-02T14:32:43.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:43.384+0000 7fb5c8c00700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fb5c4114d00 con 0x7fb5c41047e0 2024-04-02T14:32:43.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:43.384+0000 7fb5c2800700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb5c410a7e0 0x7fb5c419f8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:32:43.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:43.384+0000 7fb5c2800700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb5c410a7e0 0x7fb5c419f8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:47998/0 (socket says 172.21.15.90:47998) 2024-04-02T14:32:43.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:43.384+0000 7fb5c2800700 1 -- 172.21.15.90:0/440411125 learned_addr learned my addr 172.21.15.90:0/440411125 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:32:43.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:43.384+0000 7fb5c1e00700 1 --2- 172.21.15.90:0/440411125 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb5c41047e0 0x7fb5c419afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:32:43.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:43.384+0000 7fb5c1400700 1 --2- 172.21.15.90:0/440411125 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb5c41051a0 0x7fb5c419b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:32:43.406 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:43.385+0000 7fb5c2800700 1 -- 172.21.15.90:0/440411125 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb5c41047e0 msgr2=0x7fb5c419afd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:32:43.406 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:43.385+0000 7fb5c2800700 1 --2- 172.21.15.90:0/440411125 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb5c41047e0 0x7fb5c419afd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:43.406 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:43.385+0000 7fb5c2800700 1 -- 172.21.15.90:0/440411125 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb5c41051a0 msgr2=0x7fb5c419b510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:32:43.406 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:43.385+0000 7fb5c2800700 1 --2- 172.21.15.90:0/440411125 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb5c41051a0 0x7fb5c419b510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:43.406 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:43.385+0000 7fb5c2800700 1 -- 172.21.15.90:0/440411125 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5c419ffe0 con 0x7fb5c410a7e0 2024-04-02T14:32:43.406 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:43.385+0000 7fb5c2800700 1 --2- 172.21.15.90:0/440411125 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb5c410a7e0 0x7fb5c419f8a0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fb5bc00ef30 tx=0x7fb5bc00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:32:43.406 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:43.385+0000 7fb5baa00700 1 -- 172.21.15.90:0/440411125 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5bc0092e0 con 0x7fb5c410a7e0 2024-04-02T14:32:43.406 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:43.385+0000 7fb5c8c00700 1 -- 172.21.15.90:0/440411125 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb5c41a0270 con 0x7fb5c410a7e0 2024-04-02T14:32:43.406 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:43.385+0000 7fb5baa00700 1 -- 172.21.15.90:0/440411125 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb5bc004bb0 con 0x7fb5c410a7e0 2024-04-02T14:32:43.406 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:43.385+0000 7fb5baa00700 1 -- 172.21.15.90:0/440411125 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5bc004280 con 0x7fb5c410a7e0 2024-04-02T14:32:43.406 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:43.385+0000 7fb5c8c00700 1 -- 172.21.15.90:0/440411125 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5c41a7b50 con 0x7fb5c410a7e0 2024-04-02T14:32:43.406 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:43.387+0000 7fb5c8c00700 1 -- 172.21.15.90:0/440411125 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb5c40692b0 con 0x7fb5c410a7e0 2024-04-02T14:32:43.406 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:43.389+0000 7fb5baa00700 1 -- 172.21.15.90:0/440411125 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fb5bc01b030 con 0x7fb5c410a7e0 2024-04-02T14:32:43.406 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:43.391+0000 7fb5baa00700 1 --2- 172.21.15.90:0/440411125 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fb5a80a3cd0 0x7fb5a80a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:32:43.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:43.391+0000 7fb5baa00700 1 -- 172.21.15.90:0/440411125 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8248+0+0 (secure 0 0 0) 0x7fb5bc0d0aa0 con 0x7fb5c410a7e0 2024-04-02T14:32:43.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:43.391+0000 7fb5c1e00700 1 --2- 172.21.15.90:0/440411125 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fb5a80a3cd0 0x7fb5a80a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:32:43.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:43.392+0000 7fb5c1e00700 1 --2- 172.21.15.90:0/440411125 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fb5a80a3cd0 0x7fb5a80a6190 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fb5b4009be0 tx=0x7fb5b4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:32:43.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:43.394+0000 7fb5baa00700 1 -- 172.21.15.90:0/440411125 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7fb5bc022020 con 0x7fb5c410a7e0 2024-04-02T14:32:43.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:43.620+0000 7fb5c8c00700 1 -- 172.21.15.90:0/440411125 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb5c419c330 con 0x7fb5c410a7e0 2024-04-02T14:32:43.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:43.622+0000 7fb5baa00700 1 -- 172.21.15.90:0/440411125 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+22646 (secure 0 0 0) 0x7fb5bc010040 con 0x7fb5c410a7e0 2024-04-02T14:32:43.624 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-02T14:32:43.625 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":90,"fsid":"6449448a-f0fc-11ee-b647-cb9ed24678a4","created":"2024-04-02T14:23:16.530201+0000","modified":"2024-04-02T14:32:35.716929+0000","last_up_change":"2024-04-02T14:31:23.819732+0000","last_in_change":"2024-04-02T14:31:02.824284+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T14:27:16.758622+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-02T14:32:20.016331+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-02T14:32:21.294367+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a1a6a856-ab44-44bd-bc5e-45c718a70cd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":3635087609},{"type":"v1","addr":"172.21.15.90:6803","nonce":3635087609}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":3635087609},{"type":"v1","addr":"172.21.15.90:6805","nonce":3635087609}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":3635087609},{"type":"v1","addr":"172.21.15.90:6809","nonce":3635087609}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":3635087609},{"type":"v1","addr":"172.21.15.90:6807","nonce":3635087609}]},"public_addr":"172.21.15.90:6803/3635087609","cluster_addr":"172.21.15.90:6805/3635087609","heartbeat_back_addr":"172.21.15.90:6809/3635087609","heartbeat_front_addr":"172.21.15.90:6807/3635087609","state":["exists","up"]},{"osd":1,"uuid":"124139cd-5793-463e-9607-9d9c10164183","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":1994536059},{"type":"v1","addr":"172.21.15.92:6801","nonce":1994536059}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":1994536059},{"type":"v1","addr":"172.21.15.92:6803","nonce":1994536059}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":1994536059},{"type":"v1","addr":"172.21.15.92:6807","nonce":1994536059}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":1994536059},{"type":"v1","addr":"172.21.15.92:6805","nonce":1994536059}]},"public_addr":"172.21.15.92:6801/1994536059","cluster_addr":"172.21.15.92:6803/1994536059","heartbeat_back_addr":"172.21.15.92:6807/1994536059","heartbeat_front_addr":"172.21.15.92:6805/1994536059","state":["exists","up"]},{"osd":2,"uuid":"a61fbef2-9dca-4875-a814-020fff0db0e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":1358472800},{"type":"v1","addr":"172.21.15.177:6801","nonce":1358472800}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":1358472800},{"type":"v1","addr":"172.21.15.177:6803","nonce":1358472800}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":1358472800},{"type":"v1","addr":"172.21.15.177:6807","nonce":1358472800}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":1358472800},{"type":"v1","addr":"172.21.15.177:6805","nonce":1358472800}]},"public_addr":"172.21.15.177:6801/1358472800","cluster_addr":"172.21.15.177:6803/1358472800","heartbeat_back_addr":"172.21.15.177:6807/1358472800","heartbeat_front_addr":"172.21.15.177:6805/1358472800","state":["exists","up"]},{"osd":3,"uuid":"b175a2f4-7859-4eb0-bc1f-fba8c2b92a2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":3114811216},{"type":"v1","addr":"172.21.15.90:6811","nonce":3114811216}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":3114811216},{"type":"v1","addr":"172.21.15.90:6813","nonce":3114811216}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":3114811216},{"type":"v1","addr":"172.21.15.90:6817","nonce":3114811216}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":3114811216},{"type":"v1","addr":"172.21.15.90:6815","nonce":3114811216}]},"public_addr":"172.21.15.90:6811/3114811216","cluster_addr":"172.21.15.90:6813/3114811216","heartbeat_back_addr":"172.21.15.90:6817/3114811216","heartbeat_front_addr":"172.21.15.90:6815/3114811216","state":["exists","up"]},{"osd":4,"uuid":"38cc68ea-c109-4d8d-b77d-1c6f9b56c875","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":2987208879},{"type":"v1","addr":"172.21.15.92:6809","nonce":2987208879}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6810","nonce":2987208879},{"type":"v1","addr":"172.21.15.92:6811","nonce":2987208879}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6814","nonce":2987208879},{"type":"v1","addr":"172.21.15.92:6815","nonce":2987208879}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":2987208879},{"type":"v1","addr":"172.21.15.92:6813","nonce":2987208879}]},"public_addr":"172.21.15.92:6809/2987208879","cluster_addr":"172.21.15.92:6811/2987208879","heartbeat_back_addr":"172.21.15.92:6815/2987208879","heartbeat_front_addr":"172.21.15.92:6813/2987208879","state":["exists","up"]},{"osd":5,"uuid":"2347232e-ac91-466f-8ac9-496f397679ba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":3486820237},{"type":"v1","addr":"172.21.15.177:6809","nonce":3486820237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":3486820237},{"type":"v1","addr":"172.21.15.177:6811","nonce":3486820237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":3486820237},{"type":"v1","addr":"172.21.15.177:6815","nonce":3486820237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":3486820237},{"type":"v1","addr":"172.21.15.177:6813","nonce":3486820237}]},"public_addr":"172.21.15.177:6809/3486820237","cluster_addr":"172.21.15.177:6811/3486820237","heartbeat_back_addr":"172.21.15.177:6815/3486820237","heartbeat_front_addr":"172.21.15.177:6813/3486820237","state":["exists","up"]},{"osd":6,"uuid":"aa72392f-0498-44db-a8d2-eccc89a64871","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":2579716443},{"type":"v1","addr":"172.21.15.90:6819","nonce":2579716443}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6820","nonce":2579716443},{"type":"v1","addr":"172.21.15.90:6821","nonce":2579716443}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":2579716443},{"type":"v1","addr":"172.21.15.90:6825","nonce":2579716443}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2579716443},{"type":"v1","addr":"172.21.15.90:6823","nonce":2579716443}]},"public_addr":"172.21.15.90:6819/2579716443","cluster_addr":"172.21.15.90:6821/2579716443","heartbeat_back_addr":"172.21.15.90:6825/2579716443","heartbeat_front_addr":"172.21.15.90:6823/2579716443","state":["exists","up"]},{"osd":7,"uuid":"931bd105-bc14-4a1e-bfe4-c7ad58fa4b73","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":3371649139},{"type":"v1","addr":"172.21.15.92:6817","nonce":3371649139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":3371649139},{"type":"v1","addr":"172.21.15.92:6819","nonce":3371649139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":3371649139},{"type":"v1","addr":"172.21.15.92:6823","nonce":3371649139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6820","nonce":3371649139},{"type":"v1","addr":"172.21.15.92:6821","nonce":3371649139}]},"public_addr":"172.21.15.92:6817/3371649139","cluster_addr":"172.21.15.92:6819/3371649139","heartbeat_back_addr":"172.21.15.92:6823/3371649139","heartbeat_front_addr":"172.21.15.92:6821/3371649139","state":["exists","up"]},{"osd":8,"uuid":"82624ec2-b382-4886-9f40-7f5f376de3ce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":1459053034},{"type":"v1","addr":"172.21.15.177:6817","nonce":1459053034}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":1459053034},{"type":"v1","addr":"172.21.15.177:6819","nonce":1459053034}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":1459053034},{"type":"v1","addr":"172.21.15.177:6823","nonce":1459053034}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":1459053034},{"type":"v1","addr":"172.21.15.177:6821","nonce":1459053034}]},"public_addr":"172.21.15.177:6817/1459053034","cluster_addr":"172.21.15.177:6819/1459053034","heartbeat_back_addr":"172.21.15.177:6823/1459053034","heartbeat_front_addr":"172.21.15.177:6821/1459053034","state":["exists","up"]},{"osd":9,"uuid":"fbe36e48-3468-42b2-8094-e16552f3cb16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":2592604332},{"type":"v1","addr":"172.21.15.90:6827","nonce":2592604332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":2592604332},{"type":"v1","addr":"172.21.15.90:6829","nonce":2592604332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6832","nonce":2592604332},{"type":"v1","addr":"172.21.15.90:6833","nonce":2592604332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":2592604332},{"type":"v1","addr":"172.21.15.90:6831","nonce":2592604332}]},"public_addr":"172.21.15.90:6827/2592604332","cluster_addr":"172.21.15.90:6829/2592604332","heartbeat_back_addr":"172.21.15.90:6833/2592604332","heartbeat_front_addr":"172.21.15.90:6831/2592604332","state":["exists","up"]},{"osd":10,"uuid":"9240df75-e7e2-47f1-8aae-534b70b3c210","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":1594153142},{"type":"v1","addr":"172.21.15.92:6825","nonce":1594153142}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6826","nonce":1594153142},{"type":"v1","addr":"172.21.15.92:6827","nonce":1594153142}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":1594153142},{"type":"v1","addr":"172.21.15.92:6831","nonce":1594153142}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":1594153142},{"type":"v1","addr":"172.21.15.92:6829","nonce":1594153142}]},"public_addr":"172.21.15.92:6825/1594153142","cluster_addr":"172.21.15.92:6827/1594153142","heartbeat_back_addr":"172.21.15.92:6831/1594153142","heartbeat_front_addr":"172.21.15.92:6829/1594153142","state":["exists","up"]},{"osd":11,"uuid":"83cba9fa-b745-4df1-adda-e86837d4b91a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":83,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1622582126},{"type":"v1","addr":"172.21.15.177:6825","nonce":1622582126}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":1622582126},{"type":"v1","addr":"172.21.15.177:6827","nonce":1622582126}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":1622582126},{"type":"v1","addr":"172.21.15.177:6831","nonce":1622582126}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1622582126},{"type":"v1","addr":"172.21.15.177:6829","nonce":1622582126}]},"public_addr":"172.21.15.177:6825/1622582126","cluster_addr":"172.21.15.177:6827/1622582126","heartbeat_back_addr":"172.21.15.177:6831/1622582126","heartbeat_front_addr":"172.21.15.177:6829/1622582126","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:26:24.290365+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:26:49.405988+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:27:13.639196+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:27:45.603746+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:28:11.019126+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:28:35.084163+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:29:08.064626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:29:33.266133+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:29:57.381816+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:30:30.286847+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:30:56.226279+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:31:20.809611+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":7,"to":11}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":8,"to":4}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.11","mappings":[{"from":4,"to":8},{"from":5,"to":9}]},{"pgid":"3.21","mappings":[{"from":5,"to":9}]},{"pgid":"3.23","mappings":[{"from":4,"to":7}]},{"pgid":"3.3a","mappings":[{"from":4,"to":6}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.90:0/314641601":"2024-04-03T14:24:34.595561+0000","172.21.15.90:0/1400895868":"2024-04-03T14:24:34.595561+0000","172.21.15.90:6801/129436436":"2024-04-03T14:24:34.595561+0000","172.21.15.90:6801/2372398598":"2024-04-03T14:24:00.362147+0000","172.21.15.90:0/278569338":"2024-04-03T14:24:34.595561+0000","172.21.15.90:0/2321464594":"2024-04-03T14:24:00.362147+0000","172.21.15.90:6800/2372398598":"2024-04-03T14:24:00.362147+0000","172.21.15.90:0/2962656672":"2024-04-03T14:24:00.362147+0000","172.21.15.90:0/126693610":"2024-04-03T14:24:00.362147+0000","172.21.15.90:6800/129436436":"2024-04-03T14:24:34.595561+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T14:32:43.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:43.624+0000 7fb5c8c00700 1 -- 172.21.15.90:0/440411125 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fb5a80a3cd0 msgr2=0x7fb5a80a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:32:43.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:43.624+0000 7fb5c8c00700 1 --2- 172.21.15.90:0/440411125 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fb5a80a3cd0 0x7fb5a80a6190 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fb5b4009be0 tx=0x7fb5b4009260 comp rx=0 tx=0).stop 2024-04-02T14:32:43.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:43.625+0000 7fb5c8c00700 1 -- 172.21.15.90:0/440411125 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb5c410a7e0 msgr2=0x7fb5c419f8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:32:43.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:43.625+0000 7fb5c8c00700 1 --2- 172.21.15.90:0/440411125 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb5c410a7e0 0x7fb5c419f8a0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fb5bc00ef30 tx=0x7fb5bc00c3a0 comp rx=0 tx=0).stop 2024-04-02T14:32:43.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:43.625+0000 7fb5c8c00700 1 -- 172.21.15.90:0/440411125 shutdown_connections 2024-04-02T14:32:43.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:43.625+0000 7fb5c8c00700 1 --2- 172.21.15.90:0/440411125 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fb5a80a3cd0 0x7fb5a80a6190 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:43.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:43.625+0000 7fb5c8c00700 1 --2- 172.21.15.90:0/440411125 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb5c410a7e0 0x7fb5c419f8a0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:43.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:43.625+0000 7fb5c8c00700 1 --2- 172.21.15.90:0/440411125 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb5c41047e0 0x7fb5c419afd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:43.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:43.625+0000 7fb5c8c00700 1 --2- 172.21.15.90:0/440411125 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb5c41051a0 0x7fb5c419b510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:43.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:43.625+0000 7fb5c8c00700 1 -- 172.21.15.90:0/440411125 >> 172.21.15.90:0/440411125 conn(0x7fb5c41002d0 msgr2=0x7fb5c41007f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:32:43.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:43.625+0000 7fb5c8c00700 1 -- 172.21.15.90:0/440411125 shutdown_connections 2024-04-02T14:32:43.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:43.625+0000 7fb5c8c00700 1 -- 172.21.15.90:0/440411125 wait complete. 2024-04-02T14:32:43.939 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:43 smithi090 ceph-mon[31674]: pgmap v324: 97 pgs: 97 active+clean; 638 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 1.9 KiB/s wr, 8 op/s; 5.4 KiB/s, 1 objects/s recovering 2024-04-02T14:32:43.939 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:43 smithi090 ceph-mon[31674]: from='client.? 172.21.15.90:0/440411125' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-02T14:32:44.110 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:43 smithi092 ceph-mon[36497]: pgmap v324: 97 pgs: 97 active+clean; 638 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 1.9 KiB/s wr, 8 op/s; 5.4 KiB/s, 1 objects/s recovering 2024-04-02T14:32:44.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:43 smithi092 ceph-mon[36497]: from='client.? 172.21.15.90:0/440411125' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-02T14:32:44.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:43 smithi177 ceph-mon[36431]: pgmap v324: 97 pgs: 97 active+clean; 638 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 1.9 KiB/s wr, 8 op/s; 5.4 KiB/s, 1 objects/s recovering 2024-04-02T14:32:44.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:43 smithi177 ceph-mon[36431]: from='client.? 172.21.15.90:0/440411125' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-02T14:32:44.565 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-04-02T14:32:44.565 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:191d963ac9de7322da8aff20b567b3fa458e2ef6 shell --fsid 6449448a-f0fc-11ee-b647-cb9ed24678a4 -- ceph osd dump --format=json 2024-04-02T14:32:45.011 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T14:32:46.040 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:45 smithi090 ceph-mon[31674]: pgmap v325: 97 pgs: 97 active+clean; 638 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 1.5 KiB/s wr, 9 op/s; 4.3 KiB/s, 1 objects/s recovering 2024-04-02T14:32:46.112 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:45 smithi092 ceph-mon[36497]: pgmap v325: 97 pgs: 97 active+clean; 638 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 1.5 KiB/s wr, 9 op/s; 4.3 KiB/s, 1 objects/s recovering 2024-04-02T14:32:46.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:45 smithi177 ceph-mon[36431]: pgmap v325: 97 pgs: 97 active+clean; 638 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 1.5 KiB/s wr, 9 op/s; 4.3 KiB/s, 1 objects/s recovering 2024-04-02T14:32:47.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:47.347+0000 7fdd36000700 1 -- 172.21.15.90:0/719095363 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdd301134e0 msgr2=0x7fdd301158d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:32:47.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:47.347+0000 7fdd36000700 1 --2- 172.21.15.90:0/719095363 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdd301134e0 0x7fdd301158d0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fdd28009990 tx=0x7fdd28009ca0 comp rx=0 tx=0).stop 2024-04-02T14:32:47.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:47.347+0000 7fdd36000700 1 -- 172.21.15.90:0/719095363 shutdown_connections 2024-04-02T14:32:47.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:47.347+0000 7fdd36000700 1 --2- 172.21.15.90:0/719095363 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdd30074990 0x7fdd30074d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:47.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:47.347+0000 7fdd36000700 1 --2- 172.21.15.90:0/719095363 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdd301134e0 0x7fdd301158d0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:47.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:47.347+0000 7fdd36000700 1 --2- 172.21.15.90:0/719095363 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdd300752b0 0x7fdd30112fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:47.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:47.347+0000 7fdd36000700 1 -- 172.21.15.90:0/719095363 >> 172.21.15.90:0/719095363 conn(0x7fdd300fdfc0 msgr2=0x7fdd301003e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:32:47.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:47.347+0000 7fdd36000700 1 -- 172.21.15.90:0/719095363 shutdown_connections 2024-04-02T14:32:47.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:47.347+0000 7fdd36000700 1 -- 172.21.15.90:0/719095363 wait complete. 2024-04-02T14:32:47.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:47.348+0000 7fdd36000700 1 Processor -- start 2024-04-02T14:32:47.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:47.349+0000 7fdd36000700 1 -- start start 2024-04-02T14:32:47.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:47.349+0000 7fdd36000700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdd30074990 0x7fdd3019f240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:32:47.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:47.349+0000 7fdd36000700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdd300752b0 0x7fdd3019f780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:32:47.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:47.349+0000 7fdd36000700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdd301134e0 0x7fdd301a3b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:32:47.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:47.349+0000 7fdd36000700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fdd30118e30 con 0x7fdd300752b0 2024-04-02T14:32:47.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:47.349+0000 7fdd36000700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fdd30118cb0 con 0x7fdd30074990 2024-04-02T14:32:47.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:47.349+0000 7fdd36000700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fdd30118fb0 con 0x7fdd301134e0 2024-04-02T14:32:47.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:47.349+0000 7fdd34c00700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdd30074990 0x7fdd3019f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:32:47.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:47.349+0000 7fdd27e00700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdd300752b0 0x7fdd3019f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:32:47.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:47.349+0000 7fdd34c00700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdd30074990 0x7fdd3019f240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:56508/0 (socket says 172.21.15.90:56508) 2024-04-02T14:32:47.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:47.349+0000 7fdd34c00700 1 -- 172.21.15.90:0/1916663854 learned_addr learned my addr 172.21.15.90:0/1916663854 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:32:47.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:47.349+0000 7fdd35600700 1 --2- 172.21.15.90:0/1916663854 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdd301134e0 0x7fdd301a3b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:32:47.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:47.350+0000 7fdd34c00700 1 -- 172.21.15.90:0/1916663854 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdd301134e0 msgr2=0x7fdd301a3b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:32:47.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:47.350+0000 7fdd34c00700 1 --2- 172.21.15.90:0/1916663854 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdd301134e0 0x7fdd301a3b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:47.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:47.350+0000 7fdd34c00700 1 -- 172.21.15.90:0/1916663854 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdd300752b0 msgr2=0x7fdd3019f780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:32:47.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:47.350+0000 7fdd34c00700 1 --2- 172.21.15.90:0/1916663854 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdd300752b0 0x7fdd3019f780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:47.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:47.350+0000 7fdd34c00700 1 -- 172.21.15.90:0/1916663854 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd301a4250 con 0x7fdd30074990 2024-04-02T14:32:47.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:47.350+0000 7fdd27e00700 1 --2- 172.21.15.90:0/1916663854 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdd300752b0 0x7fdd3019f780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:32:47.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:47.350+0000 7fdd34c00700 1 --2- 172.21.15.90:0/1916663854 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdd30074990 0x7fdd3019f240 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fdd2000c370 tx=0x7fdd2000c680 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:32:47.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:47.350+0000 7fdd2e000700 1 -- 172.21.15.90:0/1916663854 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd2000cea0 con 0x7fdd30074990 2024-04-02T14:32:47.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:47.350+0000 7fdd2e000700 1 -- 172.21.15.90:0/1916663854 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdd200090d0 con 0x7fdd30074990 2024-04-02T14:32:47.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:47.350+0000 7fdd36000700 1 -- 172.21.15.90:0/1916663854 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd301a44e0 con 0x7fdd30074990 2024-04-02T14:32:47.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:47.350+0000 7fdd2e000700 1 -- 172.21.15.90:0/1916663854 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd20004880 con 0x7fdd30074990 2024-04-02T14:32:47.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:47.351+0000 7fdd36000700 1 -- 172.21.15.90:0/1916663854 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd301abdc0 con 0x7fdd30074990 2024-04-02T14:32:47.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:47.352+0000 7fdd36000700 1 -- 172.21.15.90:0/1916663854 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdd300692b0 con 0x7fdd30074990 2024-04-02T14:32:47.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:47.354+0000 7fdd2e000700 1 -- 172.21.15.90:0/1916663854 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fdd2001a020 con 0x7fdd30074990 2024-04-02T14:32:47.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:47.355+0000 7fdd2e000700 1 --2- 172.21.15.90:0/1916663854 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fdd140a3d50 0x7fdd140a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:32:47.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:47.355+0000 7fdd2e000700 1 -- 172.21.15.90:0/1916663854 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8248+0+0 (secure 0 0 0) 0x7fdd200cffa0 con 0x7fdd30074990 2024-04-02T14:32:47.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:47.355+0000 7fdd27e00700 1 --2- 172.21.15.90:0/1916663854 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fdd140a3d50 0x7fdd140a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:32:47.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:47.356+0000 7fdd27e00700 1 --2- 172.21.15.90:0/1916663854 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fdd140a3d50 0x7fdd140a6210 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fdd30106290 tx=0x7fdd18009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:32:47.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:47.359+0000 7fdd2e000700 1 -- 172.21.15.90:0/1916663854 <== mon.2 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7fdd20010040 con 0x7fdd30074990 2024-04-02T14:32:47.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:47.582+0000 7fdd36000700 1 -- 172.21.15.90:0/1916663854 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fdd301a05a0 con 0x7fdd30074990 2024-04-02T14:32:47.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:47.584+0000 7fdd2e000700 1 -- 172.21.15.90:0/1916663854 <== mon.2 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+22646 (secure 0 0 0) 0x7fdd2009c7b0 con 0x7fdd30074990 2024-04-02T14:32:47.587 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-02T14:32:47.587 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":90,"fsid":"6449448a-f0fc-11ee-b647-cb9ed24678a4","created":"2024-04-02T14:23:16.530201+0000","modified":"2024-04-02T14:32:35.716929+0000","last_up_change":"2024-04-02T14:31:23.819732+0000","last_in_change":"2024-04-02T14:31:02.824284+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T14:27:16.758622+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-02T14:32:20.016331+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-02T14:32:21.294367+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a1a6a856-ab44-44bd-bc5e-45c718a70cd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":3635087609},{"type":"v1","addr":"172.21.15.90:6803","nonce":3635087609}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":3635087609},{"type":"v1","addr":"172.21.15.90:6805","nonce":3635087609}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":3635087609},{"type":"v1","addr":"172.21.15.90:6809","nonce":3635087609}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":3635087609},{"type":"v1","addr":"172.21.15.90:6807","nonce":3635087609}]},"public_addr":"172.21.15.90:6803/3635087609","cluster_addr":"172.21.15.90:6805/3635087609","heartbeat_back_addr":"172.21.15.90:6809/3635087609","heartbeat_front_addr":"172.21.15.90:6807/3635087609","state":["exists","up"]},{"osd":1,"uuid":"124139cd-5793-463e-9607-9d9c10164183","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":1994536059},{"type":"v1","addr":"172.21.15.92:6801","nonce":1994536059}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":1994536059},{"type":"v1","addr":"172.21.15.92:6803","nonce":1994536059}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":1994536059},{"type":"v1","addr":"172.21.15.92:6807","nonce":1994536059}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":1994536059},{"type":"v1","addr":"172.21.15.92:6805","nonce":1994536059}]},"public_addr":"172.21.15.92:6801/1994536059","cluster_addr":"172.21.15.92:6803/1994536059","heartbeat_back_addr":"172.21.15.92:6807/1994536059","heartbeat_front_addr":"172.21.15.92:6805/1994536059","state":["exists","up"]},{"osd":2,"uuid":"a61fbef2-9dca-4875-a814-020fff0db0e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":1358472800},{"type":"v1","addr":"172.21.15.177:6801","nonce":1358472800}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":1358472800},{"type":"v1","addr":"172.21.15.177:6803","nonce":1358472800}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":1358472800},{"type":"v1","addr":"172.21.15.177:6807","nonce":1358472800}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":1358472800},{"type":"v1","addr":"172.21.15.177:6805","nonce":1358472800}]},"public_addr":"172.21.15.177:6801/1358472800","cluster_addr":"172.21.15.177:6803/1358472800","heartbeat_back_addr":"172.21.15.177:6807/1358472800","heartbeat_front_addr":"172.21.15.177:6805/1358472800","state":["exists","up"]},{"osd":3,"uuid":"b175a2f4-7859-4eb0-bc1f-fba8c2b92a2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":3114811216},{"type":"v1","addr":"172.21.15.90:6811","nonce":3114811216}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":3114811216},{"type":"v1","addr":"172.21.15.90:6813","nonce":3114811216}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":3114811216},{"type":"v1","addr":"172.21.15.90:6817","nonce":3114811216}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":3114811216},{"type":"v1","addr":"172.21.15.90:6815","nonce":3114811216}]},"public_addr":"172.21.15.90:6811/3114811216","cluster_addr":"172.21.15.90:6813/3114811216","heartbeat_back_addr":"172.21.15.90:6817/3114811216","heartbeat_front_addr":"172.21.15.90:6815/3114811216","state":["exists","up"]},{"osd":4,"uuid":"38cc68ea-c109-4d8d-b77d-1c6f9b56c875","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":2987208879},{"type":"v1","addr":"172.21.15.92:6809","nonce":2987208879}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6810","nonce":2987208879},{"type":"v1","addr":"172.21.15.92:6811","nonce":2987208879}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6814","nonce":2987208879},{"type":"v1","addr":"172.21.15.92:6815","nonce":2987208879}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":2987208879},{"type":"v1","addr":"172.21.15.92:6813","nonce":2987208879}]},"public_addr":"172.21.15.92:6809/2987208879","cluster_addr":"172.21.15.92:6811/2987208879","heartbeat_back_addr":"172.21.15.92:6815/2987208879","heartbeat_front_addr":"172.21.15.92:6813/2987208879","state":["exists","up"]},{"osd":5,"uuid":"2347232e-ac91-466f-8ac9-496f397679ba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":3486820237},{"type":"v1","addr":"172.21.15.177:6809","nonce":3486820237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":3486820237},{"type":"v1","addr":"172.21.15.177:6811","nonce":3486820237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":3486820237},{"type":"v1","addr":"172.21.15.177:6815","nonce":3486820237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":3486820237},{"type":"v1","addr":"172.21.15.177:6813","nonce":3486820237}]},"public_addr":"172.21.15.177:6809/3486820237","cluster_addr":"172.21.15.177:6811/3486820237","heartbeat_back_addr":"172.21.15.177:6815/3486820237","heartbeat_front_addr":"172.21.15.177:6813/3486820237","state":["exists","up"]},{"osd":6,"uuid":"aa72392f-0498-44db-a8d2-eccc89a64871","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":2579716443},{"type":"v1","addr":"172.21.15.90:6819","nonce":2579716443}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6820","nonce":2579716443},{"type":"v1","addr":"172.21.15.90:6821","nonce":2579716443}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":2579716443},{"type":"v1","addr":"172.21.15.90:6825","nonce":2579716443}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2579716443},{"type":"v1","addr":"172.21.15.90:6823","nonce":2579716443}]},"public_addr":"172.21.15.90:6819/2579716443","cluster_addr":"172.21.15.90:6821/2579716443","heartbeat_back_addr":"172.21.15.90:6825/2579716443","heartbeat_front_addr":"172.21.15.90:6823/2579716443","state":["exists","up"]},{"osd":7,"uuid":"931bd105-bc14-4a1e-bfe4-c7ad58fa4b73","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":3371649139},{"type":"v1","addr":"172.21.15.92:6817","nonce":3371649139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":3371649139},{"type":"v1","addr":"172.21.15.92:6819","nonce":3371649139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":3371649139},{"type":"v1","addr":"172.21.15.92:6823","nonce":3371649139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6820","nonce":3371649139},{"type":"v1","addr":"172.21.15.92:6821","nonce":3371649139}]},"public_addr":"172.21.15.92:6817/3371649139","cluster_addr":"172.21.15.92:6819/3371649139","heartbeat_back_addr":"172.21.15.92:6823/3371649139","heartbeat_front_addr":"172.21.15.92:6821/3371649139","state":["exists","up"]},{"osd":8,"uuid":"82624ec2-b382-4886-9f40-7f5f376de3ce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":1459053034},{"type":"v1","addr":"172.21.15.177:6817","nonce":1459053034}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":1459053034},{"type":"v1","addr":"172.21.15.177:6819","nonce":1459053034}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":1459053034},{"type":"v1","addr":"172.21.15.177:6823","nonce":1459053034}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":1459053034},{"type":"v1","addr":"172.21.15.177:6821","nonce":1459053034}]},"public_addr":"172.21.15.177:6817/1459053034","cluster_addr":"172.21.15.177:6819/1459053034","heartbeat_back_addr":"172.21.15.177:6823/1459053034","heartbeat_front_addr":"172.21.15.177:6821/1459053034","state":["exists","up"]},{"osd":9,"uuid":"fbe36e48-3468-42b2-8094-e16552f3cb16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":2592604332},{"type":"v1","addr":"172.21.15.90:6827","nonce":2592604332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":2592604332},{"type":"v1","addr":"172.21.15.90:6829","nonce":2592604332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6832","nonce":2592604332},{"type":"v1","addr":"172.21.15.90:6833","nonce":2592604332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":2592604332},{"type":"v1","addr":"172.21.15.90:6831","nonce":2592604332}]},"public_addr":"172.21.15.90:6827/2592604332","cluster_addr":"172.21.15.90:6829/2592604332","heartbeat_back_addr":"172.21.15.90:6833/2592604332","heartbeat_front_addr":"172.21.15.90:6831/2592604332","state":["exists","up"]},{"osd":10,"uuid":"9240df75-e7e2-47f1-8aae-534b70b3c210","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":1594153142},{"type":"v1","addr":"172.21.15.92:6825","nonce":1594153142}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6826","nonce":1594153142},{"type":"v1","addr":"172.21.15.92:6827","nonce":1594153142}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":1594153142},{"type":"v1","addr":"172.21.15.92:6831","nonce":1594153142}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":1594153142},{"type":"v1","addr":"172.21.15.92:6829","nonce":1594153142}]},"public_addr":"172.21.15.92:6825/1594153142","cluster_addr":"172.21.15.92:6827/1594153142","heartbeat_back_addr":"172.21.15.92:6831/1594153142","heartbeat_front_addr":"172.21.15.92:6829/1594153142","state":["exists","up"]},{"osd":11,"uuid":"83cba9fa-b745-4df1-adda-e86837d4b91a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":83,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1622582126},{"type":"v1","addr":"172.21.15.177:6825","nonce":1622582126}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":1622582126},{"type":"v1","addr":"172.21.15.177:6827","nonce":1622582126}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":1622582126},{"type":"v1","addr":"172.21.15.177:6831","nonce":1622582126}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1622582126},{"type":"v1","addr":"172.21.15.177:6829","nonce":1622582126}]},"public_addr":"172.21.15.177:6825/1622582126","cluster_addr":"172.21.15.177:6827/1622582126","heartbeat_back_addr":"172.21.15.177:6831/1622582126","heartbeat_front_addr":"172.21.15.177:6829/1622582126","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:26:24.290365+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:26:49.405988+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:27:13.639196+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:27:45.603746+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:28:11.019126+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:28:35.084163+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:29:08.064626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:29:33.266133+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:29:57.381816+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:30:30.286847+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:30:56.226279+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:31:20.809611+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":7,"to":11}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":8,"to":4}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.11","mappings":[{"from":4,"to":8},{"from":5,"to":9}]},{"pgid":"3.21","mappings":[{"from":5,"to":9}]},{"pgid":"3.23","mappings":[{"from":4,"to":7}]},{"pgid":"3.3a","mappings":[{"from":4,"to":6}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.90:0/314641601":"2024-04-03T14:24:34.595561+0000","172.21.15.90:0/1400895868":"2024-04-03T14:24:34.595561+0000","172.21.15.90:6801/129436436":"2024-04-03T14:24:34.595561+0000","172.21.15.90:6801/2372398598":"2024-04-03T14:24:00.362147+0000","172.21.15.90:0/278569338":"2024-04-03T14:24:34.595561+0000","172.21.15.90:0/2321464594":"2024-04-03T14:24:00.362147+0000","172.21.15.90:6800/2372398598":"2024-04-03T14:24:00.362147+0000","172.21.15.90:0/2962656672":"2024-04-03T14:24:00.362147+0000","172.21.15.90:0/126693610":"2024-04-03T14:24:00.362147+0000","172.21.15.90:6800/129436436":"2024-04-03T14:24:34.595561+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T14:32:47.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:47.587+0000 7fdd36000700 1 -- 172.21.15.90:0/1916663854 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fdd140a3d50 msgr2=0x7fdd140a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:32:47.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:47.587+0000 7fdd36000700 1 --2- 172.21.15.90:0/1916663854 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fdd140a3d50 0x7fdd140a6210 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fdd30106290 tx=0x7fdd18009260 comp rx=0 tx=0).stop 2024-04-02T14:32:47.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:47.587+0000 7fdd36000700 1 -- 172.21.15.90:0/1916663854 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdd30074990 msgr2=0x7fdd3019f240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:32:47.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:47.587+0000 7fdd36000700 1 --2- 172.21.15.90:0/1916663854 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdd30074990 0x7fdd3019f240 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fdd2000c370 tx=0x7fdd2000c680 comp rx=0 tx=0).stop 2024-04-02T14:32:47.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:47.587+0000 7fdd36000700 1 -- 172.21.15.90:0/1916663854 shutdown_connections 2024-04-02T14:32:47.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:47.587+0000 7fdd36000700 1 --2- 172.21.15.90:0/1916663854 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fdd140a3d50 0x7fdd140a6210 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:47.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:47.587+0000 7fdd36000700 1 --2- 172.21.15.90:0/1916663854 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdd301134e0 0x7fdd301a3b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:47.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:47.587+0000 7fdd36000700 1 --2- 172.21.15.90:0/1916663854 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdd30074990 0x7fdd3019f240 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:47.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:47.587+0000 7fdd36000700 1 --2- 172.21.15.90:0/1916663854 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdd300752b0 0x7fdd3019f780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:47.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:47.587+0000 7fdd36000700 1 -- 172.21.15.90:0/1916663854 >> 172.21.15.90:0/1916663854 conn(0x7fdd300fdfc0 msgr2=0x7fdd300fec30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:32:47.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:47.587+0000 7fdd36000700 1 -- 172.21.15.90:0/1916663854 shutdown_connections 2024-04-02T14:32:47.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:47.587+0000 7fdd36000700 1 -- 172.21.15.90:0/1916663854 wait complete. 2024-04-02T14:32:47.938 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:47 smithi090 ceph-mon[31674]: pgmap v326: 97 pgs: 97 active+clean; 638 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 1.4 KiB/s wr, 9 op/s; 3.9 KiB/s, 1 objects/s recovering 2024-04-02T14:32:47.939 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:47 smithi090 ceph-mon[31674]: from='client.? 172.21.15.90:0/1916663854' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-02T14:32:48.110 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:47 smithi092 ceph-mon[36497]: pgmap v326: 97 pgs: 97 active+clean; 638 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 1.4 KiB/s wr, 9 op/s; 3.9 KiB/s, 1 objects/s recovering 2024-04-02T14:32:48.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:47 smithi092 ceph-mon[36497]: from='client.? 172.21.15.90:0/1916663854' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-02T14:32:48.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:47 smithi177 ceph-mon[36431]: pgmap v326: 97 pgs: 97 active+clean; 638 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 1.4 KiB/s wr, 9 op/s; 3.9 KiB/s, 1 objects/s recovering 2024-04-02T14:32:48.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:47 smithi177 ceph-mon[36431]: from='client.? 172.21.15.90:0/1916663854' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-02T14:32:48.383 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:191d963ac9de7322da8aff20b567b3fa458e2ef6 shell --fsid 6449448a-f0fc-11ee-b647-cb9ed24678a4 -- ceph tell osd.0 flush_pg_stats 2024-04-02T14:32:48.384 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:191d963ac9de7322da8aff20b567b3fa458e2ef6 shell --fsid 6449448a-f0fc-11ee-b647-cb9ed24678a4 -- ceph tell osd.1 flush_pg_stats 2024-04-02T14:32:48.384 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:191d963ac9de7322da8aff20b567b3fa458e2ef6 shell --fsid 6449448a-f0fc-11ee-b647-cb9ed24678a4 -- ceph tell osd.2 flush_pg_stats 2024-04-02T14:32:48.385 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:191d963ac9de7322da8aff20b567b3fa458e2ef6 shell --fsid 6449448a-f0fc-11ee-b647-cb9ed24678a4 -- ceph tell osd.3 flush_pg_stats 2024-04-02T14:32:48.385 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:191d963ac9de7322da8aff20b567b3fa458e2ef6 shell --fsid 6449448a-f0fc-11ee-b647-cb9ed24678a4 -- ceph tell osd.4 flush_pg_stats 2024-04-02T14:32:48.385 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:191d963ac9de7322da8aff20b567b3fa458e2ef6 shell --fsid 6449448a-f0fc-11ee-b647-cb9ed24678a4 -- ceph tell osd.5 flush_pg_stats 2024-04-02T14:32:48.386 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:191d963ac9de7322da8aff20b567b3fa458e2ef6 shell --fsid 6449448a-f0fc-11ee-b647-cb9ed24678a4 -- ceph tell osd.6 flush_pg_stats 2024-04-02T14:32:48.386 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:191d963ac9de7322da8aff20b567b3fa458e2ef6 shell --fsid 6449448a-f0fc-11ee-b647-cb9ed24678a4 -- ceph tell osd.7 flush_pg_stats 2024-04-02T14:32:48.387 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:191d963ac9de7322da8aff20b567b3fa458e2ef6 shell --fsid 6449448a-f0fc-11ee-b647-cb9ed24678a4 -- ceph tell osd.8 flush_pg_stats 2024-04-02T14:32:48.387 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:191d963ac9de7322da8aff20b567b3fa458e2ef6 shell --fsid 6449448a-f0fc-11ee-b647-cb9ed24678a4 -- ceph tell osd.9 flush_pg_stats 2024-04-02T14:32:48.387 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:191d963ac9de7322da8aff20b567b3fa458e2ef6 shell --fsid 6449448a-f0fc-11ee-b647-cb9ed24678a4 -- ceph tell osd.10 flush_pg_stats 2024-04-02T14:32:48.388 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:191d963ac9de7322da8aff20b567b3fa458e2ef6 shell --fsid 6449448a-f0fc-11ee-b647-cb9ed24678a4 -- ceph tell osd.11 flush_pg_stats 2024-04-02T14:32:49.312 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T14:32:49.455 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T14:32:49.455 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T14:32:49.673 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T14:32:49.677 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T14:32:50.048 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:49 smithi090 ceph-mon[31674]: pgmap v327: 97 pgs: 97 active+clean; 638 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 1.2 KiB/s wr, 8 op/s; 3.6 KiB/s, 1 objects/s recovering 2024-04-02T14:32:50.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:49 smithi092 ceph-mon[36497]: pgmap v327: 97 pgs: 97 active+clean; 638 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 1.2 KiB/s wr, 8 op/s; 3.6 KiB/s, 1 objects/s recovering 2024-04-02T14:32:50.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:49 smithi177 ceph-mon[36431]: pgmap v327: 97 pgs: 97 active+clean; 638 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 1.2 KiB/s wr, 8 op/s; 3.6 KiB/s, 1 objects/s recovering 2024-04-02T14:32:52.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:51 smithi092 ceph-mon[36497]: pgmap v328: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 1.7 KiB/s wr, 10 op/s; 3.6 KiB/s, 1 objects/s recovering 2024-04-02T14:32:52.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:51 smithi177 ceph-mon[36431]: pgmap v328: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 1.7 KiB/s wr, 10 op/s; 3.6 KiB/s, 1 objects/s recovering 2024-04-02T14:32:52.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:51 smithi090 ceph-mon[31674]: pgmap v328: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 1.7 KiB/s wr, 10 op/s; 3.6 KiB/s, 1 objects/s recovering 2024-04-02T14:32:53.934 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:53 smithi090 ceph-mon[31674]: pgmap v329: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 426 B/s wr, 9 op/s 2024-04-02T14:32:54.110 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:53 smithi092 ceph-mon[36497]: pgmap v329: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 426 B/s wr, 9 op/s 2024-04-02T14:32:54.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:53 smithi177 ceph-mon[36431]: pgmap v329: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 426 B/s wr, 9 op/s 2024-04-02T14:32:54.643 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T14:32:54.643 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T14:32:54.666 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T14:32:54.801 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T14:32:56.038 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:55 smithi090 ceph-mon[31674]: pgmap v330: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 426 B/s wr, 11 op/s 2024-04-02T14:32:56.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:56.036+0000 7fcca3800700 1 -- 172.21.15.90:0/458100468 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcc9c102640 msgr2=0x7fcc9c102aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:32:56.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:56.036+0000 7fcca3800700 1 --2- 172.21.15.90:0/458100468 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcc9c102640 0x7fcc9c102aa0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fcc880098f0 tx=0x7fcc88009c00 comp rx=0 tx=0).stop 2024-04-02T14:32:56.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:56.037+0000 7fcca3800700 1 -- 172.21.15.90:0/458100468 shutdown_connections 2024-04-02T14:32:56.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:56.037+0000 7fcca3800700 1 --2- 172.21.15.90:0/458100468 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcc9c107520 0x7fcc9c107900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:56.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:56.037+0000 7fcca3800700 1 --2- 172.21.15.90:0/458100468 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcc9c102fe0 0x7fcc9c10f490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:56.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:56.037+0000 7fcca3800700 1 --2- 172.21.15.90:0/458100468 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcc9c102640 0x7fcc9c102aa0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:56.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:56.037+0000 7fcca3800700 1 -- 172.21.15.90:0/458100468 >> 172.21.15.90:0/458100468 conn(0x7fcc9c079720 msgr2=0x7fcc9c079b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:32:56.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:56.037+0000 7fcca3800700 1 -- 172.21.15.90:0/458100468 shutdown_connections 2024-04-02T14:32:56.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:56.037+0000 7fcca3800700 1 -- 172.21.15.90:0/458100468 wait complete. 2024-04-02T14:32:56.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:56.038+0000 7fcca3800700 1 Processor -- start 2024-04-02T14:32:56.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:56.039+0000 7fcca3800700 1 -- start start 2024-04-02T14:32:56.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:56.039+0000 7fcca3800700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcc9c102640 0x7fcc9c19b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:32:56.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:56.039+0000 7fcca3800700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcc9c102fe0 0x7fcc9c19b600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:32:56.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:56.039+0000 7fcca3800700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcc9c107520 0x7fcc9c19f990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:32:56.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:56.039+0000 7fcca3800700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fcc9c1128f0 con 0x7fcc9c107520 2024-04-02T14:32:56.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:56.039+0000 7fcca3800700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fcc9c112770 con 0x7fcc9c102fe0 2024-04-02T14:32:56.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:56.039+0000 7fcca3800700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fcc9c112a70 con 0x7fcc9c102640 2024-04-02T14:32:56.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:56.039+0000 7fcc9be00700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcc9c102fe0 0x7fcc9c19b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:32:56.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:56.039+0000 7fcca0e00700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcc9c102640 0x7fcc9c19b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:32:56.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:56.040+0000 7fcca1800700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcc9c107520 0x7fcc9c19f990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:32:56.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:56.040+0000 7fcca0e00700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcc9c102640 0x7fcc9c19b0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:53924/0 (socket says 172.21.15.90:53924) 2024-04-02T14:32:56.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:56.040+0000 7fcca0e00700 1 -- 172.21.15.90:0/1928367991 learned_addr learned my addr 172.21.15.90:0/1928367991 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:32:56.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:56.040+0000 7fcc9be00700 1 -- 172.21.15.90:0/1928367991 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcc9c102640 msgr2=0x7fcc9c19b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:32:56.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:56.040+0000 7fcc9be00700 1 --2- 172.21.15.90:0/1928367991 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcc9c102640 0x7fcc9c19b0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:56.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:56.040+0000 7fcc9be00700 1 -- 172.21.15.90:0/1928367991 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcc9c107520 msgr2=0x7fcc9c19f990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:32:56.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:56.040+0000 7fcc9be00700 1 --2- 172.21.15.90:0/1928367991 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcc9c107520 0x7fcc9c19f990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:56.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:56.040+0000 7fcc9be00700 1 -- 172.21.15.90:0/1928367991 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc9c1a00d0 con 0x7fcc9c102fe0 2024-04-02T14:32:56.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:56.040+0000 7fcca0e00700 1 --2- 172.21.15.90:0/1928367991 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcc9c102640 0x7fcc9c19b0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:32:56.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:56.040+0000 7fcc9be00700 1 --2- 172.21.15.90:0/1928367991 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcc9c102fe0 0x7fcc9c19b600 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fcc88000c00 tx=0x7fcc8800fa00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:32:56.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:56.041+0000 7fcc99600700 1 -- 172.21.15.90:0/1928367991 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc88005040 con 0x7fcc9c102fe0 2024-04-02T14:32:56.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:56.041+0000 7fcc99600700 1 -- 172.21.15.90:0/1928367991 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcc880056c0 con 0x7fcc9c102fe0 2024-04-02T14:32:56.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:56.041+0000 7fcca3800700 1 -- 172.21.15.90:0/1928367991 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc9c1a0360 con 0x7fcc9c102fe0 2024-04-02T14:32:56.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:56.041+0000 7fcc99600700 1 -- 172.21.15.90:0/1928367991 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc88003a00 con 0x7fcc9c102fe0 2024-04-02T14:32:56.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:56.041+0000 7fcca3800700 1 -- 172.21.15.90:0/1928367991 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc9c1a7c40 con 0x7fcc9c102fe0 2024-04-02T14:32:56.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:56.042+0000 7fcca3800700 1 -- 172.21.15.90:0/1928367991 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fcc9c100e90 con 0x7fcc9c102fe0 2024-04-02T14:32:56.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:56.044+0000 7fcc99600700 1 -- 172.21.15.90:0/1928367991 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fcc8800b560 con 0x7fcc9c102fe0 2024-04-02T14:32:56.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:56.045+0000 7fcc99600700 1 --2- 172.21.15.90:0/1928367991 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fcc840a3cd0 0x7fcc840a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:32:56.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:56.046+0000 7fcc99600700 1 -- 172.21.15.90:0/1928367991 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8248+0+0 (secure 0 0 0) 0x7fcc880d18a0 con 0x7fcc9c102fe0 2024-04-02T14:32:56.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:56.046+0000 7fcca0e00700 1 --2- 172.21.15.90:0/1928367991 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fcc840a3cd0 0x7fcc840a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:32:56.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:56.046+0000 7fcc99600700 1 --2- 172.21.15.90:0/1928367991 >> [v2:172.21.15.92:6808/2987208879,v1:172.21.15.92:6809/2987208879] conn(0x7fcc840ad760 0x7fcc840afc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:32:56.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:56.046+0000 7fcc99600700 1 -- 172.21.15.90:0/1928367991 --> [v2:172.21.15.92:6808/2987208879,v1:172.21.15.92:6809/2987208879] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fcc8800b0b0 con 0x7fcc840ad760 2024-04-02T14:32:56.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:56.046+0000 7fcc99600700 1 -- 172.21.15.90:0/1928367991 <== mon.2 v2:172.21.15.92:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7fcc880d1c50 con 0x7fcc9c102fe0 2024-04-02T14:32:56.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:56.046+0000 7fcca1800700 1 --2- 172.21.15.90:0/1928367991 >> [v2:172.21.15.92:6808/2987208879,v1:172.21.15.92:6809/2987208879] conn(0x7fcc840ad760 0x7fcc840afc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:32:56.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:56.047+0000 7fcca0e00700 1 --2- 172.21.15.90:0/1928367991 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fcc840a3cd0 0x7fcc840a6190 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fcc90009750 tx=0x7fcc90009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:32:56.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:56.047+0000 7fcca1800700 1 --2- 172.21.15.90:0/1928367991 >> [v2:172.21.15.92:6808/2987208879,v1:172.21.15.92:6809/2987208879] conn(0x7fcc840ad760 0x7fcc840afc00 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:32:56.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:56.048+0000 7fcc99600700 1 -- 172.21.15.90:0/1928367991 <== osd.4 v2:172.21.15.92:6808/2987208879 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fcc8800b0b0 con 0x7fcc840ad760 2024-04-02T14:32:56.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:56.066+0000 7fcca3800700 1 -- 172.21.15.90:0/1928367991 --> [v2:172.21.15.92:6808/2987208879,v1:172.21.15.92:6809/2987208879] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fcc9c103cd0 con 0x7fcc840ad760 2024-04-02T14:32:56.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:56.066+0000 7fcc99600700 1 -- 172.21.15.90:0/1928367991 <== osd.4 v2:172.21.15.92:6808/2987208879 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fcc9c103cd0 con 0x7fcc840ad760 2024-04-02T14:32:56.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:56.067+0000 7fcca3800700 1 -- 172.21.15.90:0/1928367991 >> [v2:172.21.15.92:6808/2987208879,v1:172.21.15.92:6809/2987208879] conn(0x7fcc840ad760 msgr2=0x7fcc840afc00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:32:56.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:56.067+0000 7fcca3800700 1 --2- 172.21.15.90:0/1928367991 >> [v2:172.21.15.92:6808/2987208879,v1:172.21.15.92:6809/2987208879] conn(0x7fcc840ad760 0x7fcc840afc00 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:56.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:56.067+0000 7fcca3800700 1 -- 172.21.15.90:0/1928367991 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fcc840a3cd0 msgr2=0x7fcc840a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:32:56.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:56.067+0000 7fcca3800700 1 --2- 172.21.15.90:0/1928367991 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fcc840a3cd0 0x7fcc840a6190 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fcc90009750 tx=0x7fcc90009310 comp rx=0 tx=0).stop 2024-04-02T14:32:56.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:56.067+0000 7fcca3800700 1 -- 172.21.15.90:0/1928367991 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcc9c102fe0 msgr2=0x7fcc9c19b600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:32:56.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:56.067+0000 7fcca3800700 1 --2- 172.21.15.90:0/1928367991 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcc9c102fe0 0x7fcc9c19b600 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fcc88000c00 tx=0x7fcc8800fa00 comp rx=0 tx=0).stop 2024-04-02T14:32:56.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:56.067+0000 7fcca1800700 1 -- 172.21.15.90:0/1928367991 reap_dead start 2024-04-02T14:32:56.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:56.067+0000 7fcca3800700 1 -- 172.21.15.90:0/1928367991 shutdown_connections 2024-04-02T14:32:56.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:56.067+0000 7fcca3800700 1 -- 172.21.15.90:0/1928367991 >> 172.21.15.90:0/1928367991 conn(0x7fcc9c079720 msgr2=0x7fcc9c1004b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:32:56.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:56.068+0000 7fcca3800700 1 -- 172.21.15.90:0/1928367991 shutdown_connections 2024-04-02T14:32:56.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:56.068+0000 7fcca3800700 1 -- 172.21.15.90:0/1928367991 wait complete. 2024-04-02T14:32:56.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:55 smithi092 ceph-mon[36497]: pgmap v330: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 426 B/s wr, 11 op/s 2024-04-02T14:32:56.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:55 smithi177 ceph-mon[36431]: pgmap v330: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 426 B/s wr, 11 op/s 2024-04-02T14:32:56.330 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T14:32:57.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:57.157+0000 7fb60f600700 1 -- 172.21.15.90:0/3184335611 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb608109810 msgr2=0x7fb60810d860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:32:57.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:57.157+0000 7fb60f600700 1 --2- 172.21.15.90:0/3184335611 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb608109810 0x7fb60810d860 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fb5f4009990 tx=0x7fb5f4009ca0 comp rx=0 tx=0).stop 2024-04-02T14:32:57.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:57.157+0000 7fb60f600700 1 -- 172.21.15.90:0/3184335611 shutdown_connections 2024-04-02T14:32:57.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:57.157+0000 7fb60f600700 1 --2- 172.21.15.90:0/3184335611 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb60810df40 0x7fb608115b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:57.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:57.157+0000 7fb60f600700 1 --2- 172.21.15.90:0/3184335611 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb608108e60 0x7fb608109240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:57.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:57.157+0000 7fb60f600700 1 --2- 172.21.15.90:0/3184335611 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb608109810 0x7fb60810d860 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:57.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:57.157+0000 7fb60f600700 1 -- 172.21.15.90:0/3184335611 >> 172.21.15.90:0/3184335611 conn(0x7fb608075dc0 msgr2=0x7fb6080781e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:32:57.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:57.157+0000 7fb60f600700 1 -- 172.21.15.90:0/3184335611 shutdown_connections 2024-04-02T14:32:57.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:57.157+0000 7fb60f600700 1 -- 172.21.15.90:0/3184335611 wait complete. 2024-04-02T14:32:57.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:57.158+0000 7fb60f600700 1 Processor -- start 2024-04-02T14:32:57.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:57.158+0000 7fb60f600700 1 -- start start 2024-04-02T14:32:57.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:57.159+0000 7fb60f600700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb608108e60 0x7fb60819f4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:32:57.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:57.159+0000 7fb60f600700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb608109810 0x7fb60819fa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:32:57.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:57.159+0000 7fb60f600700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb60810df40 0x7fb6081a3d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:32:57.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:57.159+0000 7fb60f600700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fb608118f70 con 0x7fb608108e60 2024-04-02T14:32:57.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:57.159+0000 7fb60f600700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fb608118df0 con 0x7fb60810df40 2024-04-02T14:32:57.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:57.159+0000 7fb60f600700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fb6081190f0 con 0x7fb608109810 2024-04-02T14:32:57.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:57.159+0000 7fb60cc00700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb608108e60 0x7fb60819f4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:32:57.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:57.159+0000 7fb60cc00700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb608108e60 0x7fb60819f4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:39764/0 (socket says 172.21.15.90:39764) 2024-04-02T14:32:57.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:57.159+0000 7fb60cc00700 1 -- 172.21.15.90:0/509142264 learned_addr learned my addr 172.21.15.90:0/509142264 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:32:57.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:57.159+0000 7fb60d600700 1 --2- 172.21.15.90:0/509142264 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb60810df40 0x7fb6081a3d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:32:57.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:57.159+0000 7fb607e00700 1 --2- 172.21.15.90:0/509142264 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb608109810 0x7fb60819fa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:32:57.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:57.160+0000 7fb60cc00700 1 -- 172.21.15.90:0/509142264 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb608109810 msgr2=0x7fb60819fa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:32:57.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:57.160+0000 7fb60cc00700 1 --2- 172.21.15.90:0/509142264 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb608109810 0x7fb60819fa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:57.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:57.160+0000 7fb60cc00700 1 -- 172.21.15.90:0/509142264 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb60810df40 msgr2=0x7fb6081a3d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:32:57.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:57.160+0000 7fb60cc00700 1 --2- 172.21.15.90:0/509142264 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb60810df40 0x7fb6081a3d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:57.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:57.160+0000 7fb60cc00700 1 -- 172.21.15.90:0/509142264 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6081a4480 con 0x7fb608108e60 2024-04-02T14:32:57.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:57.160+0000 7fb60cc00700 1 --2- 172.21.15.90:0/509142264 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb608108e60 0x7fb60819f4c0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fb5fc00e9c0 tx=0x7fb5fc00ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:32:57.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:57.160+0000 7fb605600700 1 -- 172.21.15.90:0/509142264 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5fc00cbd0 con 0x7fb608108e60 2024-04-02T14:32:57.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:57.160+0000 7fb605600700 1 -- 172.21.15.90:0/509142264 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb5fc00ee50 con 0x7fb608108e60 2024-04-02T14:32:57.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:57.160+0000 7fb60f600700 1 -- 172.21.15.90:0/509142264 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6081a4770 con 0x7fb608108e60 2024-04-02T14:32:57.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:57.160+0000 7fb605600700 1 -- 172.21.15.90:0/509142264 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5fc005680 con 0x7fb608108e60 2024-04-02T14:32:57.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:57.160+0000 7fb60f600700 1 -- 172.21.15.90:0/509142264 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6081abff0 con 0x7fb608108e60 2024-04-02T14:32:57.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:57.161+0000 7fb60f600700 1 -- 172.21.15.90:0/509142264 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fb608062c80 con 0x7fb608108e60 2024-04-02T14:32:57.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:57.164+0000 7fb605600700 1 -- 172.21.15.90:0/509142264 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fb5fc004750 con 0x7fb608108e60 2024-04-02T14:32:57.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:57.165+0000 7fb605600700 1 --2- 172.21.15.90:0/509142264 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fb5f00a3cd0 0x7fb5f00a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:32:57.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:57.165+0000 7fb605600700 1 -- 172.21.15.90:0/509142264 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8248+0+0 (secure 0 0 0) 0x7fb5fc0d0c60 con 0x7fb608108e60 2024-04-02T14:32:57.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:57.165+0000 7fb607e00700 1 --2- 172.21.15.90:0/509142264 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fb5f00a3cd0 0x7fb5f00a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:32:57.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:57.165+0000 7fb605600700 1 --2- 172.21.15.90:0/509142264 >> [v2:172.21.15.177:6816/1459053034,v1:172.21.15.177:6817/1459053034] conn(0x7fb5f00ad890 0x7fb5f00afd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:32:57.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:57.165+0000 7fb605600700 1 -- 172.21.15.90:0/509142264 --> [v2:172.21.15.177:6816/1459053034,v1:172.21.15.177:6817/1459053034] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb5f00b0400 con 0x7fb5f00ad890 2024-04-02T14:32:57.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:57.165+0000 7fb605600700 1 -- 172.21.15.90:0/509142264 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7fb5fc0d1010 con 0x7fb608108e60 2024-04-02T14:32:57.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:57.166+0000 7fb60d600700 1 --2- 172.21.15.90:0/509142264 >> [v2:172.21.15.177:6816/1459053034,v1:172.21.15.177:6817/1459053034] conn(0x7fb5f00ad890 0x7fb5f00afd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:32:57.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:57.166+0000 7fb607e00700 1 --2- 172.21.15.90:0/509142264 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fb5f00a3cd0 0x7fb5f00a6190 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fb608107fa0 tx=0x7fb5f400b260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:32:57.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:57.166+0000 7fb60d600700 1 --2- 172.21.15.90:0/509142264 >> [v2:172.21.15.177:6816/1459053034,v1:172.21.15.177:6817/1459053034] conn(0x7fb5f00ad890 0x7fb5f00afd30 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:32:57.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:57.167+0000 7fb605600700 1 -- 172.21.15.90:0/509142264 <== osd.8 v2:172.21.15.177:6816/1459053034 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fb5f00b0400 con 0x7fb5f00ad890 2024-04-02T14:32:57.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:57.185+0000 7fb60f600700 1 -- 172.21.15.90:0/509142264 --> [v2:172.21.15.177:6816/1459053034,v1:172.21.15.177:6817/1459053034] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fb608063100 con 0x7fb5f00ad890 2024-04-02T14:32:57.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:57.185+0000 7fb605600700 1 -- 172.21.15.90:0/509142264 <== osd.8 v2:172.21.15.177:6816/1459053034 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fb608063100 con 0x7fb5f00ad890 2024-04-02T14:32:57.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:57.186+0000 7fb60f600700 1 -- 172.21.15.90:0/509142264 >> [v2:172.21.15.177:6816/1459053034,v1:172.21.15.177:6817/1459053034] conn(0x7fb5f00ad890 msgr2=0x7fb5f00afd30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:32:57.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:57.186+0000 7fb60f600700 1 --2- 172.21.15.90:0/509142264 >> [v2:172.21.15.177:6816/1459053034,v1:172.21.15.177:6817/1459053034] conn(0x7fb5f00ad890 0x7fb5f00afd30 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:32:57.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:57.186+0000 7fb60f600700 1 -- 172.21.15.90:0/509142264 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fb5f00a3cd0 msgr2=0x7fb5f00a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:32:57.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:57.186+0000 7fb60f600700 1 --2- 172.21.15.90:0/509142264 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fb5f00a3cd0 0x7fb5f00a6190 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fb608107fa0 tx=0x7fb5f400b260 comp rx=0 tx=0).stop 2024-04-02T14:32:57.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:57.186+0000 7fb60f600700 1 -- 172.21.15.90:0/509142264 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb608108e60 msgr2=0x7fb60819f4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:32:57.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:57.186+0000 7fb60f600700 1 --2- 172.21.15.90:0/509142264 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb608108e60 0x7fb60819f4c0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fb5fc00e9c0 tx=0x7fb5fc00ecd0 comp rx=0 tx=0).stop 2024-04-02T14:32:57.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:57.186+0000 7fb60d600700 1 -- 172.21.15.90:0/509142264 reap_dead start 2024-04-02T14:32:57.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:57.186+0000 7fb60f600700 1 -- 172.21.15.90:0/509142264 shutdown_connections 2024-04-02T14:32:57.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:57.186+0000 7fb60f600700 1 -- 172.21.15.90:0/509142264 >> 172.21.15.90:0/509142264 conn(0x7fb608075dc0 msgr2=0x7fb6080769c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:32:57.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:57.186+0000 7fb60f600700 1 -- 172.21.15.90:0/509142264 shutdown_connections 2024-04-02T14:32:57.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:57.186+0000 7fb60f600700 1 -- 172.21.15.90:0/509142264 wait complete. 2024-04-02T14:32:58.110 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:57 smithi092 ceph-mon[36497]: pgmap v331: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 426 B/s wr, 9 op/s 2024-04-02T14:32:58.159 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:57 smithi090 ceph-mon[31674]: pgmap v331: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 426 B/s wr, 9 op/s 2024-04-02T14:32:58.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:57 smithi177 ceph-mon[36431]: pgmap v331: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 426 B/s wr, 9 op/s 2024-04-02T14:33:00.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:32:59 smithi092 ceph-mon[36497]: pgmap v332: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 426 B/s wr, 9 op/s 2024-04-02T14:33:00.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:32:59 smithi177 ceph-mon[36431]: pgmap v332: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 426 B/s wr, 9 op/s 2024-04-02T14:33:00.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:32:59 smithi090 ceph-mon[31674]: pgmap v332: 97 pgs: 97 active+clean; 642 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 426 B/s wr, 9 op/s 2024-04-02T14:33:01.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:01.400+0000 7f27f5600700 1 -- 172.21.15.90:0/4215861834 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f27f01051c0 msgr2=0x7f27f01116d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:33:01.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:01.400+0000 7f27f5600700 1 --2- 172.21.15.90:0/4215861834 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f27f01051c0 0x7f27f01116d0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f27e8009990 tx=0x7f27e8009ca0 comp rx=0 tx=0).stop 2024-04-02T14:33:01.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:01.400+0000 7f27f5600700 1 -- 172.21.15.90:0/4215861834 shutdown_connections 2024-04-02T14:33:01.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:01.400+0000 7f27f5600700 1 --2- 172.21.15.90:0/4215861834 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f27f01051c0 0x7f27f01116d0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:01.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:01.400+0000 7f27f5600700 1 --2- 172.21.15.90:0/4215861834 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f27f010a800 0x7f27f010abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:01.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:01.400+0000 7f27f5600700 1 --2- 172.21.15.90:0/4215861834 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f27f0104800 0x7f27f0104c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:01.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:01.400+0000 7f27f5600700 1 -- 172.21.15.90:0/4215861834 >> 172.21.15.90:0/4215861834 conn(0x7f27f01002d0 msgr2=0x7f27f01026f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:33:01.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:01.400+0000 7f27f5600700 1 -- 172.21.15.90:0/4215861834 shutdown_connections 2024-04-02T14:33:01.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:01.400+0000 7f27f5600700 1 -- 172.21.15.90:0/4215861834 wait complete. 2024-04-02T14:33:01.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:01.401+0000 7f27f5600700 1 Processor -- start 2024-04-02T14:33:01.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:01.401+0000 7f27f5600700 1 -- start start 2024-04-02T14:33:01.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:01.401+0000 7f27f5600700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f27f0104800 0x7f27f019b030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:33:01.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:01.401+0000 7f27f5600700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f27f01051c0 0x7f27f019b570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:33:01.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:01.401+0000 7f27f5600700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f27f010a800 0x7f27f019f8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:33:01.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:01.401+0000 7f27f5600700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f27f0114ba0 con 0x7f27f010a800 2024-04-02T14:33:01.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:01.401+0000 7f27f5600700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f27f0114a20 con 0x7f27f0104800 2024-04-02T14:33:01.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:01.401+0000 7f27f5600700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f27f0114d20 con 0x7f27f01051c0 2024-04-02T14:33:01.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:01.402+0000 7f27ef400700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f27f010a800 0x7f27f019f8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:33:01.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:01.402+0000 7f27ef400700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f27f010a800 0x7f27f019f8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:39798/0 (socket says 172.21.15.90:39798) 2024-04-02T14:33:01.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:01.402+0000 7f27ef400700 1 -- 172.21.15.90:0/2453495181 learned_addr learned my addr 172.21.15.90:0/2453495181 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:33:01.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:01.402+0000 7f27eea00700 1 --2- 172.21.15.90:0/2453495181 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f27f0104800 0x7f27f019b030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:33:01.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:01.402+0000 7f27e6000700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f27f01051c0 0x7f27f019b570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:33:01.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:01.402+0000 7f27ef400700 1 -- 172.21.15.90:0/2453495181 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f27f01051c0 msgr2=0x7f27f019b570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:33:01.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:01.402+0000 7f27ef400700 1 --2- 172.21.15.90:0/2453495181 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f27f01051c0 0x7f27f019b570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:01.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:01.402+0000 7f27ef400700 1 -- 172.21.15.90:0/2453495181 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f27f0104800 msgr2=0x7f27f019b030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:33:01.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:01.402+0000 7f27ef400700 1 --2- 172.21.15.90:0/2453495181 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f27f0104800 0x7f27f019b030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:01.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:01.402+0000 7f27ef400700 1 -- 172.21.15.90:0/2453495181 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27f019fff0 con 0x7f27f010a800 2024-04-02T14:33:01.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:01.402+0000 7f27e6000700 1 --2- 172.21.15.90:0/2453495181 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f27f01051c0 0x7f27f019b570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:33:01.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:01.402+0000 7f27eea00700 1 --2- 172.21.15.90:0/2453495181 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f27f0104800 0x7f27f019b030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:33:01.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:01.403+0000 7f27ef400700 1 --2- 172.21.15.90:0/2453495181 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f27f010a800 0x7f27f019f8b0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f27e800b2e0 tx=0x7f27e800bab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:33:01.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:01.403+0000 7f27e7e00700 1 -- 172.21.15.90:0/2453495181 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27e8026070 con 0x7f27f010a800 2024-04-02T14:33:01.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:01.403+0000 7f27e7e00700 1 -- 172.21.15.90:0/2453495181 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f27e8003d70 con 0x7f27f010a800 2024-04-02T14:33:01.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:01.403+0000 7f27f5600700 1 -- 172.21.15.90:0/2453495181 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27f01a0280 con 0x7f27f010a800 2024-04-02T14:33:01.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:01.403+0000 7f27e7e00700 1 -- 172.21.15.90:0/2453495181 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27e8021410 con 0x7f27f010a800 2024-04-02T14:33:01.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:01.403+0000 7f27f5600700 1 -- 172.21.15.90:0/2453495181 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27f01a7b60 con 0x7f27f010a800 2024-04-02T14:33:01.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:01.404+0000 7f27f5600700 1 -- 172.21.15.90:0/2453495181 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f27f0062d00 con 0x7f27f010a800 2024-04-02T14:33:01.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:01.406+0000 7f27e7e00700 1 -- 172.21.15.90:0/2453495181 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f27e8029070 con 0x7f27f010a800 2024-04-02T14:33:01.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:01.407+0000 7f27e7e00700 1 --2- 172.21.15.90:0/2453495181 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f27d40a3d50 0x7f27d40a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:33:01.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:01.407+0000 7f27e7e00700 1 -- 172.21.15.90:0/2453495181 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8248+0+0 (secure 0 0 0) 0x7f27e80d1a30 con 0x7f27f010a800 2024-04-02T14:33:01.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:01.408+0000 7f27eea00700 1 --2- 172.21.15.90:0/2453495181 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f27d40a3d50 0x7f27d40a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:33:01.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:01.408+0000 7f27e7e00700 1 --2- 172.21.15.90:0/2453495181 >> [v2:172.21.15.90:6826/2592604332,v1:172.21.15.90:6827/2592604332] conn(0x7f27d40ad910 0x7f27d40afdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:33:01.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:01.408+0000 7f27e7e00700 1 -- 172.21.15.90:0/2453495181 --> [v2:172.21.15.90:6826/2592604332,v1:172.21.15.90:6827/2592604332] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f27d40b0480 con 0x7f27d40ad910 2024-04-02T14:33:01.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:01.408+0000 7f27e7e00700 1 -- 172.21.15.90:0/2453495181 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7f27e80d1de0 con 0x7f27f010a800 2024-04-02T14:33:01.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:01.408+0000 7f27e6000700 1 --2- 172.21.15.90:0/2453495181 >> [v2:172.21.15.90:6826/2592604332,v1:172.21.15.90:6827/2592604332] conn(0x7f27d40ad910 0x7f27d40afdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:33:01.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:01.408+0000 7f27eea00700 1 --2- 172.21.15.90:0/2453495181 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f27d40a3d50 0x7f27d40a6210 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f27e0009480 tx=0x7f27e000b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:33:01.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:01.409+0000 7f27e6000700 1 --2- 172.21.15.90:0/2453495181 >> [v2:172.21.15.90:6826/2592604332,v1:172.21.15.90:6827/2592604332] conn(0x7f27d40ad910 0x7f27d40afdb0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.9 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:33:01.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:01.410+0000 7f27e7e00700 1 -- 172.21.15.90:0/2453495181 <== osd.9 v2:172.21.15.90:6826/2592604332 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f27d40b0480 con 0x7f27d40ad910 2024-04-02T14:33:01.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:01.428+0000 7f27f5600700 1 -- 172.21.15.90:0/2453495181 --> [v2:172.21.15.90:6826/2592604332,v1:172.21.15.90:6827/2592604332] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f27f0063100 con 0x7f27d40ad910 2024-04-02T14:33:01.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:01.428+0000 7f27e7e00700 1 -- 172.21.15.90:0/2453495181 <== osd.9 v2:172.21.15.90:6826/2592604332 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f27f0063100 con 0x7f27d40ad910 2024-04-02T14:33:01.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:01.429+0000 7f27f5600700 1 -- 172.21.15.90:0/2453495181 >> [v2:172.21.15.90:6826/2592604332,v1:172.21.15.90:6827/2592604332] conn(0x7f27d40ad910 msgr2=0x7f27d40afdb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:33:01.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:01.429+0000 7f27f5600700 1 --2- 172.21.15.90:0/2453495181 >> [v2:172.21.15.90:6826/2592604332,v1:172.21.15.90:6827/2592604332] conn(0x7f27d40ad910 0x7f27d40afdb0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:01.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:01.429+0000 7f27f5600700 1 -- 172.21.15.90:0/2453495181 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f27d40a3d50 msgr2=0x7f27d40a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:33:01.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:01.429+0000 7f27f5600700 1 --2- 172.21.15.90:0/2453495181 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f27d40a3d50 0x7f27d40a6210 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f27e0009480 tx=0x7f27e000b040 comp rx=0 tx=0).stop 2024-04-02T14:33:01.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:01.429+0000 7f27f5600700 1 -- 172.21.15.90:0/2453495181 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f27f010a800 msgr2=0x7f27f019f8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:33:01.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:01.429+0000 7f27f5600700 1 --2- 172.21.15.90:0/2453495181 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f27f010a800 0x7f27f019f8b0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f27e800b2e0 tx=0x7f27e800bab0 comp rx=0 tx=0).stop 2024-04-02T14:33:01.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:01.429+0000 7f27ef400700 1 -- 172.21.15.90:0/2453495181 reap_dead start 2024-04-02T14:33:01.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:01.429+0000 7f27f5600700 1 -- 172.21.15.90:0/2453495181 shutdown_connections 2024-04-02T14:33:01.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:01.429+0000 7f27f5600700 1 -- 172.21.15.90:0/2453495181 >> 172.21.15.90:0/2453495181 conn(0x7f27f01002d0 msgr2=0x7f27f0100890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:33:01.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:01.429+0000 7f27f5600700 1 -- 172.21.15.90:0/2453495181 shutdown_connections 2024-04-02T14:33:01.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:01.429+0000 7f27f5600700 1 -- 172.21.15.90:0/2453495181 wait complete. 2024-04-02T14:33:01.938 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:33:01 smithi090 ceph-mon[31674]: pgmap v333: 97 pgs: 97 active+clean; 643 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 597 B/s wr, 11 op/s 2024-04-02T14:33:02.110 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:33:01 smithi092 ceph-mon[36497]: pgmap v333: 97 pgs: 97 active+clean; 643 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 597 B/s wr, 11 op/s 2024-04-02T14:33:02.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:58.526+0000 7fddc5a00700 1 -- 172.21.15.90:0/43413346 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fddc010ed10 msgr2=0x7fddc0111180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:33:02.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:58.526+0000 7fddc5a00700 1 --2- 172.21.15.90:0/43413346 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fddc010ed10 0x7fddc0111180 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fddb8009f90 tx=0x7fddb800b330 comp rx=0 tx=0).stop 2024-04-02T14:33:02.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:58.526+0000 7fddc5a00700 1 -- 172.21.15.90:0/43413346 shutdown_connections 2024-04-02T14:33:02.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:58.526+0000 7fddc5a00700 1 --2- 172.21.15.90:0/43413346 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fddc010ed10 0x7fddc0111180 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:02.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:58.526+0000 7fddc5a00700 1 --2- 172.21.15.90:0/43413346 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fddc0074a20 0x7fddc0074e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:02.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:58.526+0000 7fddc5a00700 1 --2- 172.21.15.90:0/43413346 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fddc0075340 0x7fddc010e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:02.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:58.526+0000 7fddc5a00700 1 -- 172.21.15.90:0/43413346 >> 172.21.15.90:0/43413346 conn(0x7fddc00fdfc0 msgr2=0x7fddc01003e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:33:02.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:58.526+0000 7fddc5a00700 1 -- 172.21.15.90:0/43413346 shutdown_connections 2024-04-02T14:33:02.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:58.526+0000 7fddc5a00700 1 -- 172.21.15.90:0/43413346 wait complete. 2024-04-02T14:33:02.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:58.528+0000 7fddc5a00700 1 Processor -- start 2024-04-02T14:33:02.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:58.528+0000 7fddc5a00700 1 -- start start 2024-04-02T14:33:02.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:58.528+0000 7fddc5a00700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fddc0074a20 0x7fddc019aea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:33:02.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:58.528+0000 7fddc5a00700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fddc0075340 0x7fddc019b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:33:02.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:58.528+0000 7fddc5a00700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fddc010ed10 0x7fddc019f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:33:02.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:58.528+0000 7fddc5a00700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fddc01149f0 con 0x7fddc010ed10 2024-04-02T14:33:02.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:58.528+0000 7fddc5a00700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fddc0114870 con 0x7fddc0075340 2024-04-02T14:33:02.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:58.528+0000 7fddc5a00700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fddc0114b70 con 0x7fddc0074a20 2024-04-02T14:33:02.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:58.529+0000 7fddb7e00700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fddc0075340 0x7fddc019b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:33:02.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:58.529+0000 7fddbfe00700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fddc0074a20 0x7fddc019aea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:33:02.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:58.529+0000 7fddc5000700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fddc010ed10 0x7fddc019f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:33:02.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:58.529+0000 7fddb7e00700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fddc0075340 0x7fddc019b3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:53608/0 (socket says 172.21.15.90:53608) 2024-04-02T14:33:02.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:58.529+0000 7fddb7e00700 1 -- 172.21.15.90:0/1580207021 learned_addr learned my addr 172.21.15.90:0/1580207021 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:33:02.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:58.529+0000 7fddb7e00700 1 -- 172.21.15.90:0/1580207021 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fddc0074a20 msgr2=0x7fddc019aea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:33:02.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:58.529+0000 7fddb7e00700 1 --2- 172.21.15.90:0/1580207021 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fddc0074a20 0x7fddc019aea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:02.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:58.529+0000 7fddb7e00700 1 -- 172.21.15.90:0/1580207021 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fddc010ed10 msgr2=0x7fddc019f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:33:02.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:58.529+0000 7fddb7e00700 1 --2- 172.21.15.90:0/1580207021 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fddc010ed10 0x7fddc019f770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:02.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:58.529+0000 7fddb7e00700 1 -- 172.21.15.90:0/1580207021 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fddc019feb0 con 0x7fddc0075340 2024-04-02T14:33:02.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:58.530+0000 7fddb7e00700 1 --2- 172.21.15.90:0/1580207021 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fddc0075340 0x7fddc019b3e0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fdda800ec10 tx=0x7fdda800ef20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:33:02.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:58.530+0000 7fddbd600700 1 -- 172.21.15.90:0/1580207021 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdda800cc90 con 0x7fddc0075340 2024-04-02T14:33:02.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:58.530+0000 7fddc5a00700 1 -- 172.21.15.90:0/1580207021 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fddc01a01a0 con 0x7fddc0075340 2024-04-02T14:33:02.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:58.530+0000 7fddbd600700 1 -- 172.21.15.90:0/1580207021 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdda800ce30 con 0x7fddc0075340 2024-04-02T14:33:02.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:58.530+0000 7fddbd600700 1 -- 172.21.15.90:0/1580207021 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdda8010690 con 0x7fddc0075340 2024-04-02T14:33:02.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:58.530+0000 7fddc5a00700 1 -- 172.21.15.90:0/1580207021 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fddc01a7a20 con 0x7fddc0075340 2024-04-02T14:33:02.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:58.531+0000 7fddc5a00700 1 -- 172.21.15.90:0/1580207021 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fddc00ffed0 con 0x7fddc0075340 2024-04-02T14:33:02.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:58.533+0000 7fddbd600700 1 -- 172.21.15.90:0/1580207021 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fdda8003680 con 0x7fddc0075340 2024-04-02T14:33:02.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:58.535+0000 7fddbd600700 1 --2- 172.21.15.90:0/1580207021 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fdda40a3cd0 0x7fdda40a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:33:02.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:58.535+0000 7fddbd600700 1 -- 172.21.15.90:0/1580207021 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8248+0+0 (secure 0 0 0) 0x7fdda80d0b90 con 0x7fddc0075340 2024-04-02T14:33:02.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:58.535+0000 7fddbfe00700 1 --2- 172.21.15.90:0/1580207021 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fdda40a3cd0 0x7fdda40a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:33:02.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:58.535+0000 7fddbd600700 1 --2- 172.21.15.90:0/1580207021 >> [v2:172.21.15.92:6816/3371649139,v1:172.21.15.92:6817/3371649139] conn(0x7fdda40ad890 0x7fdda40afd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:33:02.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:58.535+0000 7fddbd600700 1 -- 172.21.15.90:0/1580207021 --> [v2:172.21.15.92:6816/3371649139,v1:172.21.15.92:6817/3371649139] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fdda8011e50 con 0x7fdda40ad890 2024-04-02T14:33:02.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:58.535+0000 7fddbd600700 1 -- 172.21.15.90:0/1580207021 <== mon.2 v2:172.21.15.92:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7fdda80d0f40 con 0x7fddc0075340 2024-04-02T14:33:02.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:58.536+0000 7fddc5000700 1 --2- 172.21.15.90:0/1580207021 >> [v2:172.21.15.92:6816/3371649139,v1:172.21.15.92:6817/3371649139] conn(0x7fdda40ad890 0x7fdda40afd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:33:02.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:58.536+0000 7fddbfe00700 1 --2- 172.21.15.90:0/1580207021 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fdda40a3cd0 0x7fdda40a6190 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fddb0009c60 tx=0x7fddb0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:33:02.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:58.536+0000 7fddc5000700 1 --2- 172.21.15.90:0/1580207021 >> [v2:172.21.15.92:6816/3371649139,v1:172.21.15.92:6817/3371649139] conn(0x7fdda40ad890 0x7fdda40afd30 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:33:02.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:58.537+0000 7fddbd600700 1 -- 172.21.15.90:0/1580207021 <== osd.7 v2:172.21.15.92:6816/3371649139 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fdda8011e50 con 0x7fdda40ad890 2024-04-02T14:33:02.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:58.555+0000 7fddc5a00700 1 -- 172.21.15.90:0/1580207021 --> [v2:172.21.15.92:6816/3371649139,v1:172.21.15.92:6817/3371649139] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fddc00760d0 con 0x7fdda40ad890 2024-04-02T14:33:02.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:58.556+0000 7fddbd600700 1 -- 172.21.15.90:0/1580207021 <== osd.7 v2:172.21.15.92:6816/3371649139 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fddc00760d0 con 0x7fdda40ad890 2024-04-02T14:33:02.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:58.557+0000 7fddc5a00700 1 -- 172.21.15.90:0/1580207021 >> [v2:172.21.15.92:6816/3371649139,v1:172.21.15.92:6817/3371649139] conn(0x7fdda40ad890 msgr2=0x7fdda40afd30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:33:02.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:58.557+0000 7fddc5a00700 1 --2- 172.21.15.90:0/1580207021 >> [v2:172.21.15.92:6816/3371649139,v1:172.21.15.92:6817/3371649139] conn(0x7fdda40ad890 0x7fdda40afd30 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:02.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:58.557+0000 7fddc5a00700 1 -- 172.21.15.90:0/1580207021 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fdda40a3cd0 msgr2=0x7fdda40a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:33:02.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:58.557+0000 7fddc5a00700 1 --2- 172.21.15.90:0/1580207021 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fdda40a3cd0 0x7fdda40a6190 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fddb0009c60 tx=0x7fddb0009260 comp rx=0 tx=0).stop 2024-04-02T14:33:02.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:58.557+0000 7fddc5a00700 1 -- 172.21.15.90:0/1580207021 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fddc0075340 msgr2=0x7fddc019b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:33:02.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:58.557+0000 7fddc5a00700 1 --2- 172.21.15.90:0/1580207021 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fddc0075340 0x7fddc019b3e0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fdda800ec10 tx=0x7fdda800ef20 comp rx=0 tx=0).stop 2024-04-02T14:33:02.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:58.557+0000 7fddc5000700 1 -- 172.21.15.90:0/1580207021 reap_dead start 2024-04-02T14:33:02.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:58.557+0000 7fddc5a00700 1 -- 172.21.15.90:0/1580207021 shutdown_connections 2024-04-02T14:33:02.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:58.557+0000 7fddc5a00700 1 -- 172.21.15.90:0/1580207021 >> 172.21.15.90:0/1580207021 conn(0x7fddc00fdfc0 msgr2=0x7fddc0110f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:33:02.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:58.557+0000 7fddc5a00700 1 -- 172.21.15.90:0/1580207021 shutdown_connections 2024-04-02T14:33:02.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:32:58.557+0000 7fddc5a00700 1 -- 172.21.15.90:0/1580207021 wait complete. 2024-04-02T14:33:02.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:33:01 smithi177 ceph-mon[36431]: pgmap v333: 97 pgs: 97 active+clean; 643 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 597 B/s wr, 11 op/s 2024-04-02T14:33:04.082 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:33:03 smithi090 ceph-mon[31674]: pgmap v334: 97 pgs: 97 active+clean; 643 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 170 B/s wr, 9 op/s 2024-04-02T14:33:04.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:33:03 smithi092 ceph-mon[36497]: pgmap v334: 97 pgs: 97 active+clean; 643 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 170 B/s wr, 9 op/s 2024-04-02T14:33:04.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:33:03 smithi177 ceph-mon[36431]: pgmap v334: 97 pgs: 97 active+clean; 643 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 170 B/s wr, 9 op/s 2024-04-02T14:33:04.327 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T14:33:04.328 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T14:33:04.683 INFO:teuthology.orchestra.run.smithi090.stdout:141733920829 2024-04-02T14:33:04.684 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:191d963ac9de7322da8aff20b567b3fa458e2ef6 shell --fsid 6449448a-f0fc-11ee-b647-cb9ed24678a4 -- ceph osd last-stat-seq osd.4 2024-04-02T14:33:06.110 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:33:05 smithi092 ceph-mon[36497]: pgmap v335: 97 pgs: 97 active+clean; 643 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 170 B/s wr, 10 op/s 2024-04-02T14:33:06.133 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:33:05 smithi090 ceph-mon[31674]: pgmap v335: 97 pgs: 97 active+clean; 643 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 170 B/s wr, 10 op/s 2024-04-02T14:33:06.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:33:05 smithi177 ceph-mon[36431]: pgmap v335: 97 pgs: 97 active+clean; 643 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 170 B/s wr, 10 op/s 2024-04-02T14:33:07.243 INFO:teuthology.orchestra.run.smithi090.stdout:223338299436 2024-04-02T14:33:07.243 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:191d963ac9de7322da8aff20b567b3fa458e2ef6 shell --fsid 6449448a-f0fc-11ee-b647-cb9ed24678a4 -- ceph osd last-stat-seq osd.7 2024-04-02T14:33:08.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:33:07 smithi092 ceph-mon[36497]: pgmap v336: 97 pgs: 97 active+clean; 643 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 170 B/s wr, 9 op/s 2024-04-02T14:33:08.118 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:33:07 smithi090 ceph-mon[31674]: pgmap v336: 97 pgs: 97 active+clean; 643 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 170 B/s wr, 9 op/s 2024-04-02T14:33:08.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:33:07 smithi177 ceph-mon[36431]: pgmap v336: 97 pgs: 97 active+clean; 643 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 170 B/s wr, 9 op/s 2024-04-02T14:33:08.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:05.389+0000 7f0510c00700 1 -- 172.21.15.90:0/2166938656 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f050c108630 msgr2=0x7f050c108a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:33:08.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:05.389+0000 7f0510c00700 1 --2- 172.21.15.90:0/2166938656 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f050c108630 0x7f050c108a10 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f04fc0098f0 tx=0x7f04fc009c00 comp rx=0 tx=0).stop 2024-04-02T14:33:08.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:05.389+0000 7f0510c00700 1 -- 172.21.15.90:0/2166938656 shutdown_connections 2024-04-02T14:33:08.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:05.389+0000 7f0510c00700 1 --2- 172.21.15.90:0/2166938656 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f050c103000 0x7f050c10f500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:08.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:05.389+0000 7f0510c00700 1 --2- 172.21.15.90:0/2166938656 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f050c108630 0x7f050c108a10 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:08.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:05.389+0000 7f0510c00700 1 --2- 172.21.15.90:0/2166938656 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f050c102640 0x7f050c102ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:08.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:05.389+0000 7f0510c00700 1 -- 172.21.15.90:0/2166938656 >> 172.21.15.90:0/2166938656 conn(0x7f050c079750 msgr2=0x7f050c079b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:33:08.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:05.389+0000 7f0510c00700 1 -- 172.21.15.90:0/2166938656 shutdown_connections 2024-04-02T14:33:08.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:05.389+0000 7f0510c00700 1 -- 172.21.15.90:0/2166938656 wait complete. 2024-04-02T14:33:08.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:05.390+0000 7f0510c00700 1 Processor -- start 2024-04-02T14:33:08.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:05.391+0000 7f0510c00700 1 -- start start 2024-04-02T14:33:08.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:05.391+0000 7f0510c00700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f050c102640 0x7f050c0748f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:33:08.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:05.391+0000 7f0510c00700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f050c103000 0x7f050c074e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:33:08.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:05.391+0000 7f0510c00700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f050c108630 0x7f050c075370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:33:08.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:05.391+0000 7f0510c00700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f050c1129b0 con 0x7f050c103000 2024-04-02T14:33:08.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:05.391+0000 7f0510c00700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f050c112830 con 0x7f050c102640 2024-04-02T14:33:08.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:05.391+0000 7f0510c00700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f050c112b30 con 0x7f050c108630 2024-04-02T14:33:08.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:05.391+0000 7f0509e00700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f050c102640 0x7f050c0748f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:33:08.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:05.391+0000 7f0509e00700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f050c102640 0x7f050c0748f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:39700/0 (socket says 172.21.15.90:39700) 2024-04-02T14:33:08.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:05.391+0000 7f0509e00700 1 -- 172.21.15.90:0/3839054485 learned_addr learned my addr 172.21.15.90:0/3839054485 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:33:08.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:05.391+0000 7f0509400700 1 --2- 172.21.15.90:0/3839054485 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f050c103000 0x7f050c074e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:33:08.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:05.391+0000 7f050a800700 1 --2- 172.21.15.90:0/3839054485 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f050c108630 0x7f050c075370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:33:08.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:05.392+0000 7f0509400700 1 -- 172.21.15.90:0/3839054485 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f050c108630 msgr2=0x7f050c075370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:33:08.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:05.392+0000 7f0509400700 1 --2- 172.21.15.90:0/3839054485 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f050c108630 0x7f050c075370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:08.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:05.392+0000 7f0509400700 1 -- 172.21.15.90:0/3839054485 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f050c102640 msgr2=0x7f050c0748f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:33:08.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:05.392+0000 7f0509400700 1 --2- 172.21.15.90:0/3839054485 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f050c102640 0x7f050c0748f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:08.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:05.392+0000 7f0509400700 1 -- 172.21.15.90:0/3839054485 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f050c1a7450 con 0x7f050c103000 2024-04-02T14:33:08.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:05.392+0000 7f0509400700 1 --2- 172.21.15.90:0/3839054485 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f050c103000 0x7f050c074e30 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f04f800eb90 tx=0x7f04f800eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:33:08.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:05.392+0000 7f0502a00700 1 -- 172.21.15.90:0/3839054485 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04f800cc90 con 0x7f050c103000 2024-04-02T14:33:08.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:05.392+0000 7f0510c00700 1 -- 172.21.15.90:0/3839054485 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f050c1a7740 con 0x7f050c103000 2024-04-02T14:33:08.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:05.393+0000 7f0502a00700 1 -- 172.21.15.90:0/3839054485 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f04f800ce30 con 0x7f050c103000 2024-04-02T14:33:08.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:05.393+0000 7f0502a00700 1 -- 172.21.15.90:0/3839054485 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04f8010430 con 0x7f050c103000 2024-04-02T14:33:08.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:05.393+0000 7f0510c00700 1 -- 172.21.15.90:0/3839054485 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f050c1a7cd0 con 0x7f050c103000 2024-04-02T14:33:08.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:05.394+0000 7f0510c00700 1 -- 172.21.15.90:0/3839054485 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f050c062d00 con 0x7f050c103000 2024-04-02T14:33:08.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:05.396+0000 7f0502a00700 1 -- 172.21.15.90:0/3839054485 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f04f8003680 con 0x7f050c103000 2024-04-02T14:33:08.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:05.397+0000 7f0502a00700 1 --2- 172.21.15.90:0/3839054485 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f04f00a3cd0 0x7f04f00a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:33:08.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:05.397+0000 7f0502a00700 1 -- 172.21.15.90:0/3839054485 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8248+0+0 (secure 0 0 0) 0x7f04f80d0b90 con 0x7f050c103000 2024-04-02T14:33:08.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:05.398+0000 7f0509e00700 1 --2- 172.21.15.90:0/3839054485 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f04f00a3cd0 0x7f04f00a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:33:08.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:05.398+0000 7f0502a00700 1 --2- 172.21.15.90:0/3839054485 >> [v2:172.21.15.90:6810/3114811216,v1:172.21.15.90:6811/3114811216] conn(0x7f04f00ad890 0x7f04f00afd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:33:08.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:05.398+0000 7f0502a00700 1 -- 172.21.15.90:0/3839054485 --> [v2:172.21.15.90:6810/3114811216,v1:172.21.15.90:6811/3114811216] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f04f8023e50 con 0x7f04f00ad890 2024-04-02T14:33:08.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:05.398+0000 7f0502a00700 1 -- 172.21.15.90:0/3839054485 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7f04f80d0f40 con 0x7f050c103000 2024-04-02T14:33:08.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:05.398+0000 7f050a800700 1 --2- 172.21.15.90:0/3839054485 >> [v2:172.21.15.90:6810/3114811216,v1:172.21.15.90:6811/3114811216] conn(0x7f04f00ad890 0x7f04f00afd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:33:08.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:05.398+0000 7f0509e00700 1 --2- 172.21.15.90:0/3839054485 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f04f00a3cd0 0x7f04f00a6190 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f04fc00c010 tx=0x7f04fc019040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:33:08.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:05.399+0000 7f050a800700 1 --2- 172.21.15.90:0/3839054485 >> [v2:172.21.15.90:6810/3114811216,v1:172.21.15.90:6811/3114811216] conn(0x7f04f00ad890 0x7f04f00afd30 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:33:08.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:05.400+0000 7f0502a00700 1 -- 172.21.15.90:0/3839054485 <== osd.3 v2:172.21.15.90:6810/3114811216 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f04f8023e50 con 0x7f04f00ad890 2024-04-02T14:33:08.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:05.420+0000 7f0510c00700 1 -- 172.21.15.90:0/3839054485 --> [v2:172.21.15.90:6810/3114811216,v1:172.21.15.90:6811/3114811216] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f050c063100 con 0x7f04f00ad890 2024-04-02T14:33:08.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:05.421+0000 7f0502a00700 1 -- 172.21.15.90:0/3839054485 <== osd.3 v2:172.21.15.90:6810/3114811216 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f050c063100 con 0x7f04f00ad890 2024-04-02T14:33:08.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:05.421+0000 7f0510c00700 1 -- 172.21.15.90:0/3839054485 >> [v2:172.21.15.90:6810/3114811216,v1:172.21.15.90:6811/3114811216] conn(0x7f04f00ad890 msgr2=0x7f04f00afd30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:33:08.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:05.422+0000 7f0510c00700 1 --2- 172.21.15.90:0/3839054485 >> [v2:172.21.15.90:6810/3114811216,v1:172.21.15.90:6811/3114811216] conn(0x7f04f00ad890 0x7f04f00afd30 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:08.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:05.422+0000 7f0510c00700 1 -- 172.21.15.90:0/3839054485 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f04f00a3cd0 msgr2=0x7f04f00a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:33:08.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:05.422+0000 7f0510c00700 1 --2- 172.21.15.90:0/3839054485 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f04f00a3cd0 0x7f04f00a6190 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f04fc00c010 tx=0x7f04fc019040 comp rx=0 tx=0).stop 2024-04-02T14:33:08.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:05.422+0000 7f0510c00700 1 -- 172.21.15.90:0/3839054485 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f050c103000 msgr2=0x7f050c074e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:33:08.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:05.422+0000 7f0510c00700 1 --2- 172.21.15.90:0/3839054485 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f050c103000 0x7f050c074e30 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f04f800eb90 tx=0x7f04f800eea0 comp rx=0 tx=0).stop 2024-04-02T14:33:08.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:05.422+0000 7f050a800700 1 -- 172.21.15.90:0/3839054485 reap_dead start 2024-04-02T14:33:08.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:05.422+0000 7f0510c00700 1 -- 172.21.15.90:0/3839054485 shutdown_connections 2024-04-02T14:33:08.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:05.422+0000 7f0510c00700 1 -- 172.21.15.90:0/3839054485 >> 172.21.15.90:0/3839054485 conn(0x7f050c079750 msgr2=0x7f050c1007f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:33:08.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:05.422+0000 7f0510c00700 1 -- 172.21.15.90:0/3839054485 shutdown_connections 2024-04-02T14:33:08.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:05.422+0000 7f0510c00700 1 -- 172.21.15.90:0/3839054485 wait complete. 2024-04-02T14:33:09.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:09.532+0000 7f6730e00700 1 -- 172.21.15.90:0/2343382542 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f672c10f0d0 msgr2=0x7f672c1114c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:33:09.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:09.532+0000 7f6730e00700 1 --2- 172.21.15.90:0/2343382542 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f672c10f0d0 0x7f672c1114c0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f672400b0a0 tx=0x7f672400b3b0 comp rx=0 tx=0).stop 2024-04-02T14:33:09.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:09.532+0000 7f6730e00700 1 -- 172.21.15.90:0/2343382542 shutdown_connections 2024-04-02T14:33:09.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:09.532+0000 7f6730e00700 1 --2- 172.21.15.90:0/2343382542 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f672c10f0d0 0x7f672c1114c0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:09.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:09.532+0000 7f6730e00700 1 --2- 172.21.15.90:0/2343382542 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f672c06ab30 0x7f672c06af10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:09.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:09.532+0000 7f6730e00700 1 --2- 172.21.15.90:0/2343382542 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f672c06b450 0x7f672c10eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:09.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:09.532+0000 7f6730e00700 1 -- 172.21.15.90:0/2343382542 >> 172.21.15.90:0/2343382542 conn(0x7f672c077540 msgr2=0x7f672c077950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:33:09.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:09.533+0000 7f6730e00700 1 -- 172.21.15.90:0/2343382542 shutdown_connections 2024-04-02T14:33:09.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:09.533+0000 7f6730e00700 1 -- 172.21.15.90:0/2343382542 wait complete. 2024-04-02T14:33:09.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:09.534+0000 7f6730e00700 1 Processor -- start 2024-04-02T14:33:09.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:09.534+0000 7f6730e00700 1 -- start start 2024-04-02T14:33:09.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:09.534+0000 7f6730e00700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f672c06ab30 0x7f672c19af30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:33:09.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:09.534+0000 7f6730e00700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f672c06b450 0x7f672c19b470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:33:09.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:09.534+0000 7f6730e00700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f672c10f0d0 0x7f672c19f800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:33:09.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:09.534+0000 7f6730e00700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f672c114b00 con 0x7f672c06b450 2024-04-02T14:33:09.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:09.534+0000 7f6730e00700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f672c114980 con 0x7f672c06ab30 2024-04-02T14:33:09.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:09.534+0000 7f6730e00700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f672c114c80 con 0x7f672c10f0d0 2024-04-02T14:33:09.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:09.534+0000 7f6729400700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f672c06b450 0x7f672c19b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:33:09.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:09.534+0000 7f6729e00700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f672c06ab30 0x7f672c19af30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:33:09.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:09.534+0000 7f6729e00700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f672c06ab30 0x7f672c19af30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:39738/0 (socket says 172.21.15.90:39738) 2024-04-02T14:33:09.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:09.534+0000 7f6729e00700 1 -- 172.21.15.90:0/3022175262 learned_addr learned my addr 172.21.15.90:0/3022175262 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:33:09.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:09.535+0000 7f672a800700 1 --2- 172.21.15.90:0/3022175262 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f672c10f0d0 0x7f672c19f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:33:09.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:09.535+0000 7f6729e00700 1 -- 172.21.15.90:0/3022175262 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f672c10f0d0 msgr2=0x7f672c19f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:33:09.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:09.535+0000 7f6729e00700 1 --2- 172.21.15.90:0/3022175262 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f672c10f0d0 0x7f672c19f800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:09.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:09.535+0000 7f6729e00700 1 -- 172.21.15.90:0/3022175262 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f672c06b450 msgr2=0x7f672c19b470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:33:09.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:09.535+0000 7f6729e00700 1 --2- 172.21.15.90:0/3022175262 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f672c06b450 0x7f672c19b470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:09.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:09.535+0000 7f6729e00700 1 -- 172.21.15.90:0/3022175262 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f672c19ff40 con 0x7f672c06ab30 2024-04-02T14:33:09.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:09.535+0000 7f6729e00700 1 --2- 172.21.15.90:0/3022175262 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f672c06ab30 0x7f672c19af30 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f671c00e9c0 tx=0x7f671c00ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:33:09.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:09.535+0000 7f6722a00700 1 -- 172.21.15.90:0/3022175262 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f671c00cbd0 con 0x7f672c06ab30 2024-04-02T14:33:09.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:09.536+0000 7f6730e00700 1 -- 172.21.15.90:0/3022175262 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f672c1a0230 con 0x7f672c06ab30 2024-04-02T14:33:09.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:09.536+0000 7f6722a00700 1 -- 172.21.15.90:0/3022175262 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f671c00ee60 con 0x7f672c06ab30 2024-04-02T14:33:09.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:09.536+0000 7f6722a00700 1 -- 172.21.15.90:0/3022175262 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f671c005630 con 0x7f672c06ab30 2024-04-02T14:33:09.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:09.536+0000 7f6730e00700 1 -- 172.21.15.90:0/3022175262 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f672c1a7ab0 con 0x7f672c06ab30 2024-04-02T14:33:09.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:09.537+0000 7f6730e00700 1 -- 172.21.15.90:0/3022175262 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f672c062d80 con 0x7f672c06ab30 2024-04-02T14:33:09.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:09.539+0000 7f6722a00700 1 -- 172.21.15.90:0/3022175262 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f671c004750 con 0x7f672c06ab30 2024-04-02T14:33:09.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:09.540+0000 7f6722a00700 1 --2- 172.21.15.90:0/3022175262 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f67100a3cd0 0x7f67100a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:33:09.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:09.540+0000 7f6722a00700 1 -- 172.21.15.90:0/3022175262 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8248+0+0 (secure 0 0 0) 0x7f671c0d0c70 con 0x7f672c06ab30 2024-04-02T14:33:09.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:09.541+0000 7f6729400700 1 --2- 172.21.15.90:0/3022175262 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f67100a3cd0 0x7f67100a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:33:09.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:09.541+0000 7f6722a00700 1 --2- 172.21.15.90:0/3022175262 >> [v2:172.21.15.177:6800/1358472800,v1:172.21.15.177:6801/1358472800] conn(0x7f67100ad890 0x7f67100afd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:33:09.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:09.541+0000 7f6722a00700 1 -- 172.21.15.90:0/3022175262 --> [v2:172.21.15.177:6800/1358472800,v1:172.21.15.177:6801/1358472800] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f671c017e50 con 0x7f67100ad890 2024-04-02T14:33:09.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:09.541+0000 7f6722a00700 1 -- 172.21.15.90:0/3022175262 <== mon.2 v2:172.21.15.92:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7f671c0d1020 con 0x7f672c06ab30 2024-04-02T14:33:09.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:09.541+0000 7f672a800700 1 --2- 172.21.15.90:0/3022175262 >> [v2:172.21.15.177:6800/1358472800,v1:172.21.15.177:6801/1358472800] conn(0x7f67100ad890 0x7f67100afd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:33:09.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:09.541+0000 7f6729400700 1 --2- 172.21.15.90:0/3022175262 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f67100a3cd0 0x7f67100a6190 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f6718009680 tx=0x7f6718009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:33:09.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:09.542+0000 7f672a800700 1 --2- 172.21.15.90:0/3022175262 >> [v2:172.21.15.177:6800/1358472800,v1:172.21.15.177:6801/1358472800] conn(0x7f67100ad890 0x7f67100afd30 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:33:09.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:09.543+0000 7f6722a00700 1 -- 172.21.15.90:0/3022175262 <== osd.2 v2:172.21.15.177:6800/1358472800 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f671c017e50 con 0x7f67100ad890 2024-04-02T14:33:09.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:09.561+0000 7f6730e00700 1 -- 172.21.15.90:0/3022175262 --> [v2:172.21.15.177:6800/1358472800,v1:172.21.15.177:6801/1358472800] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f672c063200 con 0x7f67100ad890 2024-04-02T14:33:09.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:09.561+0000 7f6722a00700 1 -- 172.21.15.90:0/3022175262 <== osd.2 v2:172.21.15.177:6800/1358472800 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f672c063200 con 0x7f67100ad890 2024-04-02T14:33:09.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:09.562+0000 7f6730e00700 1 -- 172.21.15.90:0/3022175262 >> [v2:172.21.15.177:6800/1358472800,v1:172.21.15.177:6801/1358472800] conn(0x7f67100ad890 msgr2=0x7f67100afd30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:33:09.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:09.562+0000 7f6730e00700 1 --2- 172.21.15.90:0/3022175262 >> [v2:172.21.15.177:6800/1358472800,v1:172.21.15.177:6801/1358472800] conn(0x7f67100ad890 0x7f67100afd30 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:09.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:09.562+0000 7f6730e00700 1 -- 172.21.15.90:0/3022175262 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f67100a3cd0 msgr2=0x7f67100a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:33:09.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:09.562+0000 7f6730e00700 1 --2- 172.21.15.90:0/3022175262 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f67100a3cd0 0x7f67100a6190 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f6718009680 tx=0x7f6718009260 comp rx=0 tx=0).stop 2024-04-02T14:33:09.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:09.562+0000 7f6730e00700 1 -- 172.21.15.90:0/3022175262 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f672c06ab30 msgr2=0x7f672c19af30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:33:09.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:09.562+0000 7f6730e00700 1 --2- 172.21.15.90:0/3022175262 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f672c06ab30 0x7f672c19af30 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f671c00e9c0 tx=0x7f671c00ecd0 comp rx=0 tx=0).stop 2024-04-02T14:33:09.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:09.562+0000 7f672a800700 1 -- 172.21.15.90:0/3022175262 reap_dead start 2024-04-02T14:33:09.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:09.562+0000 7f6730e00700 1 -- 172.21.15.90:0/3022175262 shutdown_connections 2024-04-02T14:33:09.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:09.562+0000 7f6730e00700 1 -- 172.21.15.90:0/3022175262 >> 172.21.15.90:0/3022175262 conn(0x7f672c077540 msgr2=0x7f672c1104a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:33:09.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:09.563+0000 7f6730e00700 1 -- 172.21.15.90:0/3022175262 shutdown_connections 2024-04-02T14:33:09.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:09.563+0000 7f6730e00700 1 -- 172.21.15.90:0/3022175262 wait complete. 2024-04-02T14:33:09.630 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:08.363+0000 7f5c91400700 1 -- 172.21.15.90:0/2687027294 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5c8c102640 msgr2=0x7f5c8c102aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:33:09.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:08.363+0000 7f5c91400700 1 --2- 172.21.15.90:0/2687027294 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5c8c102640 0x7f5c8c102aa0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f5c78009990 tx=0x7f5c78009ca0 comp rx=0 tx=0).stop 2024-04-02T14:33:09.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:08.363+0000 7f5c91400700 1 -- 172.21.15.90:0/2687027294 shutdown_connections 2024-04-02T14:33:09.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:08.363+0000 7f5c91400700 1 --2- 172.21.15.90:0/2687027294 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5c8c102fe0 0x7f5c8c10f490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:09.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:08.363+0000 7f5c91400700 1 --2- 172.21.15.90:0/2687027294 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5c8c107520 0x7f5c8c107900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:09.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:08.363+0000 7f5c91400700 1 --2- 172.21.15.90:0/2687027294 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5c8c102640 0x7f5c8c102aa0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:09.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:08.363+0000 7f5c91400700 1 -- 172.21.15.90:0/2687027294 >> 172.21.15.90:0/2687027294 conn(0x7f5c8c079720 msgr2=0x7f5c8c079b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:33:09.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:08.363+0000 7f5c91400700 1 -- 172.21.15.90:0/2687027294 shutdown_connections 2024-04-02T14:33:09.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:08.363+0000 7f5c91400700 1 -- 172.21.15.90:0/2687027294 wait complete. 2024-04-02T14:33:09.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:08.364+0000 7f5c91400700 1 Processor -- start 2024-04-02T14:33:09.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:08.365+0000 7f5c91400700 1 -- start start 2024-04-02T14:33:09.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:08.365+0000 7f5c91400700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5c8c102640 0x7f5c8c0748f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:33:09.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:08.365+0000 7f5c91400700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5c8c102fe0 0x7f5c8c074e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:33:09.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:08.365+0000 7f5c91400700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5c8c107520 0x7f5c8c075370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:33:09.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:08.365+0000 7f5c91400700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f5c8c1128f0 con 0x7f5c8c102fe0 2024-04-02T14:33:09.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:08.365+0000 7f5c91400700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f5c8c112770 con 0x7f5c8c102640 2024-04-02T14:33:09.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:08.365+0000 7f5c91400700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f5c8c112a70 con 0x7f5c8c107520 2024-04-02T14:33:09.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:08.365+0000 7f5c8a800700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5c8c102640 0x7f5c8c0748f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:33:09.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:08.365+0000 7f5c8a800700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5c8c102640 0x7f5c8c0748f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:39718/0 (socket says 172.21.15.90:39718) 2024-04-02T14:33:09.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:08.365+0000 7f5c8a800700 1 -- 172.21.15.90:0/1446684693 learned_addr learned my addr 172.21.15.90:0/1446684693 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:33:09.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:08.365+0000 7f5c89e00700 1 --2- 172.21.15.90:0/1446684693 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5c8c102fe0 0x7f5c8c074e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:33:09.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:08.365+0000 7f5c8b200700 1 --2- 172.21.15.90:0/1446684693 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5c8c107520 0x7f5c8c075370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:33:09.634 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:08.366+0000 7f5c8a800700 1 -- 172.21.15.90:0/1446684693 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5c8c107520 msgr2=0x7f5c8c075370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:33:09.634 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:08.366+0000 7f5c8a800700 1 --2- 172.21.15.90:0/1446684693 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5c8c107520 0x7f5c8c075370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:09.634 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:08.366+0000 7f5c8a800700 1 -- 172.21.15.90:0/1446684693 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5c8c102fe0 msgr2=0x7f5c8c074e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:33:09.634 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:08.366+0000 7f5c8a800700 1 --2- 172.21.15.90:0/1446684693 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5c8c102fe0 0x7f5c8c074e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:09.634 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:08.366+0000 7f5c8a800700 1 -- 172.21.15.90:0/1446684693 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c8c1a7350 con 0x7f5c8c102640 2024-04-02T14:33:09.634 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:08.366+0000 7f5c8b200700 1 --2- 172.21.15.90:0/1446684693 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5c8c107520 0x7f5c8c075370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:33:09.634 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:08.367+0000 7f5c8a800700 1 --2- 172.21.15.90:0/1446684693 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5c8c102640 0x7f5c8c0748f0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f5c7c00e970 tx=0x7f5c7c00ec80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:33:09.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:08.367+0000 7f5c83400700 1 -- 172.21.15.90:0/1446684693 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c7c00cbd0 con 0x7f5c8c102640 2024-04-02T14:33:09.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:08.367+0000 7f5c83400700 1 -- 172.21.15.90:0/1446684693 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5c7c004500 con 0x7f5c8c102640 2024-04-02T14:33:09.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:08.367+0000 7f5c91400700 1 -- 172.21.15.90:0/1446684693 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5c8c1a7640 con 0x7f5c8c102640 2024-04-02T14:33:09.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:08.367+0000 7f5c83400700 1 -- 172.21.15.90:0/1446684693 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c7c003e40 con 0x7f5c8c102640 2024-04-02T14:33:09.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:08.367+0000 7f5c91400700 1 -- 172.21.15.90:0/1446684693 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5c8c1a7bd0 con 0x7f5c8c102640 2024-04-02T14:33:09.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:08.368+0000 7f5c91400700 1 -- 172.21.15.90:0/1446684693 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f5c8c062d00 con 0x7f5c8c102640 2024-04-02T14:33:09.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:08.371+0000 7f5c83400700 1 -- 172.21.15.90:0/1446684693 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f5c7c004000 con 0x7f5c8c102640 2024-04-02T14:33:09.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:08.372+0000 7f5c83400700 1 --2- 172.21.15.90:0/1446684693 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f5c700a3d50 0x7f5c700a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:33:09.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:08.372+0000 7f5c83400700 1 -- 172.21.15.90:0/1446684693 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8248+0+0 (secure 0 0 0) 0x7f5c7c0d0880 con 0x7f5c8c102640 2024-04-02T14:33:09.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:08.373+0000 7f5c89e00700 1 --2- 172.21.15.90:0/1446684693 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f5c700a3d50 0x7f5c700a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:33:09.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:08.373+0000 7f5c83400700 1 --2- 172.21.15.90:0/1446684693 >> [v2:172.21.15.92:6800/1994536059,v1:172.21.15.92:6801/1994536059] conn(0x7f5c700ad910 0x7f5c700afdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:33:09.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:08.373+0000 7f5c83400700 1 -- 172.21.15.90:0/1446684693 --> [v2:172.21.15.92:6800/1994536059,v1:172.21.15.92:6801/1994536059] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5c7c00be50 con 0x7f5c700ad910 2024-04-02T14:33:09.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:08.373+0000 7f5c83400700 1 -- 172.21.15.90:0/1446684693 <== mon.2 v2:172.21.15.92:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7f5c7c0d52a0 con 0x7f5c8c102640 2024-04-02T14:33:09.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:08.373+0000 7f5c8b200700 1 --2- 172.21.15.90:0/1446684693 >> [v2:172.21.15.92:6800/1994536059,v1:172.21.15.92:6801/1994536059] conn(0x7f5c700ad910 0x7f5c700afdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:33:09.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:08.373+0000 7f5c89e00700 1 --2- 172.21.15.90:0/1446684693 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f5c700a3d50 0x7f5c700a6210 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f5c8c079e60 tx=0x7f5c7800b260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:33:09.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:08.374+0000 7f5c8b200700 1 --2- 172.21.15.90:0/1446684693 >> [v2:172.21.15.92:6800/1994536059,v1:172.21.15.92:6801/1994536059] conn(0x7f5c700ad910 0x7f5c700afdb0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:33:09.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:08.375+0000 7f5c83400700 1 -- 172.21.15.90:0/1446684693 <== osd.1 v2:172.21.15.92:6800/1994536059 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f5c7c00be50 con 0x7f5c700ad910 2024-04-02T14:33:09.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:08.393+0000 7f5c91400700 1 -- 172.21.15.90:0/1446684693 --> [v2:172.21.15.92:6800/1994536059,v1:172.21.15.92:6801/1994536059] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f5c8c063100 con 0x7f5c700ad910 2024-04-02T14:33:09.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:08.394+0000 7f5c83400700 1 -- 172.21.15.90:0/1446684693 <== osd.1 v2:172.21.15.92:6800/1994536059 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f5c8c063100 con 0x7f5c700ad910 2024-04-02T14:33:09.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:08.394+0000 7f5c91400700 1 -- 172.21.15.90:0/1446684693 >> [v2:172.21.15.92:6800/1994536059,v1:172.21.15.92:6801/1994536059] conn(0x7f5c700ad910 msgr2=0x7f5c700afdb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:33:09.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:08.394+0000 7f5c91400700 1 --2- 172.21.15.90:0/1446684693 >> [v2:172.21.15.92:6800/1994536059,v1:172.21.15.92:6801/1994536059] conn(0x7f5c700ad910 0x7f5c700afdb0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:09.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:08.394+0000 7f5c91400700 1 -- 172.21.15.90:0/1446684693 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f5c700a3d50 msgr2=0x7f5c700a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:33:09.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:08.394+0000 7f5c91400700 1 --2- 172.21.15.90:0/1446684693 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f5c700a3d50 0x7f5c700a6210 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f5c8c079e60 tx=0x7f5c7800b260 comp rx=0 tx=0).stop 2024-04-02T14:33:09.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:08.394+0000 7f5c91400700 1 -- 172.21.15.90:0/1446684693 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5c8c102640 msgr2=0x7f5c8c0748f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:33:09.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:08.394+0000 7f5c91400700 1 --2- 172.21.15.90:0/1446684693 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5c8c102640 0x7f5c8c0748f0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f5c7c00e970 tx=0x7f5c7c00ec80 comp rx=0 tx=0).stop 2024-04-02T14:33:09.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:08.394+0000 7f5c8b200700 1 -- 172.21.15.90:0/1446684693 reap_dead start 2024-04-02T14:33:09.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:08.394+0000 7f5c91400700 1 -- 172.21.15.90:0/1446684693 shutdown_connections 2024-04-02T14:33:09.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:08.394+0000 7f5c91400700 1 -- 172.21.15.90:0/1446684693 >> 172.21.15.90:0/1446684693 conn(0x7f5c8c079720 msgr2=0x7f5c8c1007f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:33:09.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:08.394+0000 7f5c91400700 1 -- 172.21.15.90:0/1446684693 shutdown_connections 2024-04-02T14:33:09.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:08.394+0000 7f5c91400700 1 -- 172.21.15.90:0/1446684693 wait complete. 2024-04-02T14:33:09.939 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:33:09 smithi090 ceph-mon[31674]: pgmap v337: 97 pgs: 97 active+clean; 643 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 170 B/s wr, 9 op/s 2024-04-02T14:33:10.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:33:09 smithi092 ceph-mon[36497]: pgmap v337: 97 pgs: 97 active+clean; 643 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 170 B/s wr, 9 op/s 2024-04-02T14:33:10.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:33:09 smithi177 ceph-mon[36431]: pgmap v337: 97 pgs: 97 active+clean; 643 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 170 B/s wr, 9 op/s 2024-04-02T14:33:10.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:10.872+0000 7f3347200700 1 -- 172.21.15.90:0/2872942761 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3340102640 msgr2=0x7f3340102ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:33:10.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:10.872+0000 7f3347200700 1 --2- 172.21.15.90:0/2872942761 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3340102640 0x7f3340102ac0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f33280098f0 tx=0x7f3328009c00 comp rx=0 tx=0).stop 2024-04-02T14:33:10.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:10.872+0000 7f3347200700 1 -- 172.21.15.90:0/2872942761 shutdown_connections 2024-04-02T14:33:10.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:10.872+0000 7f3347200700 1 --2- 172.21.15.90:0/2872942761 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3340108630 0x7f3340108a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:10.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:10.872+0000 7f3347200700 1 --2- 172.21.15.90:0/2872942761 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3340103000 0x7f334010f500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:10.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:10.872+0000 7f3347200700 1 --2- 172.21.15.90:0/2872942761 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3340102640 0x7f3340102ac0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:10.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:10.872+0000 7f3347200700 1 -- 172.21.15.90:0/2872942761 >> 172.21.15.90:0/2872942761 conn(0x7f3340079750 msgr2=0x7f3340079b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:33:10.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:10.872+0000 7f3347200700 1 -- 172.21.15.90:0/2872942761 shutdown_connections 2024-04-02T14:33:10.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:10.872+0000 7f3347200700 1 -- 172.21.15.90:0/2872942761 wait complete. 2024-04-02T14:33:10.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:10.873+0000 7f3347200700 1 Processor -- start 2024-04-02T14:33:10.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:10.874+0000 7f3347200700 1 -- start start 2024-04-02T14:33:10.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:10.874+0000 7f3347200700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3340102640 0x7f334019b050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:33:10.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:10.874+0000 7f3347200700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3340103000 0x7f334019b590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:33:10.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:10.874+0000 7f3347200700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3340108630 0x7f334019f920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:33:10.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:10.874+0000 7f3347200700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f33401129d0 con 0x7f3340108630 2024-04-02T14:33:10.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:10.874+0000 7f3347200700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f3340112850 con 0x7f3340102640 2024-04-02T14:33:10.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:10.874+0000 7f3347200700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f3340112b50 con 0x7f3340103000 2024-04-02T14:33:10.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:10.874+0000 7f333fe00700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3340102640 0x7f334019b050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:33:10.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:10.875+0000 7f333fe00700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3340102640 0x7f334019b050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:39744/0 (socket says 172.21.15.90:39744) 2024-04-02T14:33:10.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:10.875+0000 7f333fe00700 1 -- 172.21.15.90:0/626827456 learned_addr learned my addr 172.21.15.90:0/626827456 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:33:10.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:10.875+0000 7f3345200700 1 --2- 172.21.15.90:0/626827456 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3340108630 0x7f334019f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:33:10.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:10.875+0000 7f3337400700 1 --2- 172.21.15.90:0/626827456 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3340103000 0x7f334019b590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:33:10.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:10.875+0000 7f3345200700 1 -- 172.21.15.90:0/626827456 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3340103000 msgr2=0x7f334019b590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:33:10.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:10.875+0000 7f3345200700 1 --2- 172.21.15.90:0/626827456 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3340103000 0x7f334019b590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:10.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:10.875+0000 7f3345200700 1 -- 172.21.15.90:0/626827456 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3340102640 msgr2=0x7f334019b050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:33:10.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:10.875+0000 7f3345200700 1 --2- 172.21.15.90:0/626827456 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3340102640 0x7f334019b050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:10.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:10.875+0000 7f3345200700 1 -- 172.21.15.90:0/626827456 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33401a0060 con 0x7f3340108630 2024-04-02T14:33:10.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:10.875+0000 7f3345200700 1 --2- 172.21.15.90:0/626827456 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3340108630 0x7f334019f920 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f333800d650 tx=0x7f333800d960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:33:10.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:10.876+0000 7f333d600700 1 -- 172.21.15.90:0/626827456 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3338016070 con 0x7f3340108630 2024-04-02T14:33:10.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:10.876+0000 7f3347200700 1 -- 172.21.15.90:0/626827456 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f33401a0350 con 0x7f3340108630 2024-04-02T14:33:10.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:10.876+0000 7f333d600700 1 -- 172.21.15.90:0/626827456 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f333800fb70 con 0x7f3340108630 2024-04-02T14:33:10.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:10.876+0000 7f333d600700 1 -- 172.21.15.90:0/626827456 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3338003b40 con 0x7f3340108630 2024-04-02T14:33:10.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:10.876+0000 7f3347200700 1 -- 172.21.15.90:0/626827456 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f33401a7bd0 con 0x7f3340108630 2024-04-02T14:33:10.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:10.877+0000 7f3347200700 1 -- 172.21.15.90:0/626827456 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f3340100e90 con 0x7f3340108630 2024-04-02T14:33:10.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:10.879+0000 7f333d600700 1 -- 172.21.15.90:0/626827456 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f3338003ce0 con 0x7f3340108630 2024-04-02T14:33:10.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:10.880+0000 7f333d600700 1 --2- 172.21.15.90:0/626827456 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f33240a3d50 0x7f33240a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:33:10.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:10.880+0000 7f333d600700 1 -- 172.21.15.90:0/626827456 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8248+0+0 (secure 0 0 0) 0x7f33380d0290 con 0x7f3340108630 2024-04-02T14:33:10.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:10.880+0000 7f333fe00700 1 --2- 172.21.15.90:0/626827456 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f33240a3d50 0x7f33240a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:33:10.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:10.880+0000 7f333d600700 1 --2- 172.21.15.90:0/626827456 >> [v2:172.21.15.90:6818/2579716443,v1:172.21.15.90:6819/2579716443] conn(0x7f33240ad910 0x7f33240afdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:33:10.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:10.880+0000 7f333d600700 1 -- 172.21.15.90:0/626827456 --> [v2:172.21.15.90:6818/2579716443,v1:172.21.15.90:6819/2579716443] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f333800f9c0 con 0x7f33240ad910 2024-04-02T14:33:10.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:10.881+0000 7f333d600700 1 -- 172.21.15.90:0/626827456 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7f33380d06e0 con 0x7f3340108630 2024-04-02T14:33:10.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:10.881+0000 7f3337400700 1 --2- 172.21.15.90:0/626827456 >> [v2:172.21.15.90:6818/2579716443,v1:172.21.15.90:6819/2579716443] conn(0x7f33240ad910 0x7f33240afdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:33:10.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:10.881+0000 7f333fe00700 1 --2- 172.21.15.90:0/626827456 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f33240a3d50 0x7f33240a6210 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f3330009e60 tx=0x7f333000b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:33:10.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:10.881+0000 7f3337400700 1 --2- 172.21.15.90:0/626827456 >> [v2:172.21.15.90:6818/2579716443,v1:172.21.15.90:6819/2579716443] conn(0x7f33240ad910 0x7f33240afdb0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:33:10.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:10.882+0000 7f333d600700 1 -- 172.21.15.90:0/626827456 <== osd.6 v2:172.21.15.90:6818/2579716443 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f333800f9c0 con 0x7f33240ad910 2024-04-02T14:33:10.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:10.900+0000 7f3347200700 1 -- 172.21.15.90:0/626827456 --> [v2:172.21.15.90:6818/2579716443,v1:172.21.15.90:6819/2579716443] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f3340103d70 con 0x7f33240ad910 2024-04-02T14:33:10.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:10.901+0000 7f333d600700 1 -- 172.21.15.90:0/626827456 <== osd.6 v2:172.21.15.90:6818/2579716443 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f3340103d70 con 0x7f33240ad910 2024-04-02T14:33:10.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:10.901+0000 7f3347200700 1 -- 172.21.15.90:0/626827456 >> [v2:172.21.15.90:6818/2579716443,v1:172.21.15.90:6819/2579716443] conn(0x7f33240ad910 msgr2=0x7f33240afdb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:33:10.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:10.901+0000 7f3347200700 1 --2- 172.21.15.90:0/626827456 >> [v2:172.21.15.90:6818/2579716443,v1:172.21.15.90:6819/2579716443] conn(0x7f33240ad910 0x7f33240afdb0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:10.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:10.901+0000 7f3347200700 1 -- 172.21.15.90:0/626827456 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f33240a3d50 msgr2=0x7f33240a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:33:10.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:10.901+0000 7f3347200700 1 --2- 172.21.15.90:0/626827456 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f33240a3d50 0x7f33240a6210 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f3330009e60 tx=0x7f333000b040 comp rx=0 tx=0).stop 2024-04-02T14:33:10.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:10.901+0000 7f3347200700 1 -- 172.21.15.90:0/626827456 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3340108630 msgr2=0x7f334019f920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:33:10.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:10.901+0000 7f3347200700 1 --2- 172.21.15.90:0/626827456 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3340108630 0x7f334019f920 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f333800d650 tx=0x7f333800d960 comp rx=0 tx=0).stop 2024-04-02T14:33:10.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:10.901+0000 7f3345200700 1 -- 172.21.15.90:0/626827456 reap_dead start 2024-04-02T14:33:10.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:10.901+0000 7f3347200700 1 -- 172.21.15.90:0/626827456 shutdown_connections 2024-04-02T14:33:10.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:10.902+0000 7f3347200700 1 -- 172.21.15.90:0/626827456 >> 172.21.15.90:0/626827456 conn(0x7f3340079750 msgr2=0x7f33401004b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:33:10.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:10.902+0000 7f3347200700 1 -- 172.21.15.90:0/626827456 shutdown_connections 2024-04-02T14:33:10.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:10.902+0000 7f3347200700 1 -- 172.21.15.90:0/626827456 wait complete. 2024-04-02T14:33:10.983 INFO:teuthology.orchestra.run.smithi090.stdout:261993005095 2024-04-02T14:33:10.984 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:191d963ac9de7322da8aff20b567b3fa458e2ef6 shell --fsid 6449448a-f0fc-11ee-b647-cb9ed24678a4 -- ceph osd last-stat-seq osd.8 2024-04-02T14:33:11.111 INFO:teuthology.orchestra.run.smithi090.stdout:296352743456 2024-04-02T14:33:11.112 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:191d963ac9de7322da8aff20b567b3fa458e2ef6 shell --fsid 6449448a-f0fc-11ee-b647-cb9ed24678a4 -- ceph osd last-stat-seq osd.9 2024-04-02T14:33:12.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:11.008+0000 7fb319600700 1 -- 172.21.15.90:0/2808463030 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb314103000 msgr2=0x7fb31410f500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:33:12.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:11.008+0000 7fb319600700 1 --2- 172.21.15.90:0/2808463030 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb314103000 0x7fb31410f500 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fb30c009f90 tx=0x7fb30c00b330 comp rx=0 tx=0).stop 2024-04-02T14:33:12.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:11.008+0000 7fb319600700 1 -- 172.21.15.90:0/2808463030 shutdown_connections 2024-04-02T14:33:12.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:11.008+0000 7fb319600700 1 --2- 172.21.15.90:0/2808463030 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb314103000 0x7fb31410f500 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:12.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:11.008+0000 7fb319600700 1 --2- 172.21.15.90:0/2808463030 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb314108630 0x7fb314108a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:12.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:11.008+0000 7fb319600700 1 --2- 172.21.15.90:0/2808463030 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb314102640 0x7fb314102ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:12.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:11.008+0000 7fb319600700 1 -- 172.21.15.90:0/2808463030 >> 172.21.15.90:0/2808463030 conn(0x7fb314079750 msgr2=0x7fb314079b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:33:12.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:11.008+0000 7fb319600700 1 -- 172.21.15.90:0/2808463030 shutdown_connections 2024-04-02T14:33:12.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:11.008+0000 7fb319600700 1 -- 172.21.15.90:0/2808463030 wait complete. 2024-04-02T14:33:12.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:11.009+0000 7fb319600700 1 Processor -- start 2024-04-02T14:33:12.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:11.010+0000 7fb319600700 1 -- start start 2024-04-02T14:33:12.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:11.010+0000 7fb319600700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb314102640 0x7fb314198d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:33:12.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:11.010+0000 7fb319600700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb314103000 0x7fb3141992b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:33:12.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:11.010+0000 7fb319600700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb314108630 0x7fb31419d640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:33:12.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:11.010+0000 7fb319600700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fb3141129c0 con 0x7fb314103000 2024-04-02T14:33:12.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:11.010+0000 7fb319600700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fb314112840 con 0x7fb314102640 2024-04-02T14:33:12.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:11.010+0000 7fb319600700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fb314112b40 con 0x7fb314108630 2024-04-02T14:33:12.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:11.010+0000 7fb312000700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb314103000 0x7fb3141992b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:33:12.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:11.010+0000 7fb312000700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb314103000 0x7fb3141992b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:52356/0 (socket says 172.21.15.90:52356) 2024-04-02T14:33:12.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:11.010+0000 7fb312000700 1 -- 172.21.15.90:0/691295254 learned_addr learned my addr 172.21.15.90:0/691295254 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:33:12.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:11.010+0000 7fb312a00700 1 --2- 172.21.15.90:0/691295254 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb314102640 0x7fb314198d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:33:12.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:11.011+0000 7fb313400700 1 --2- 172.21.15.90:0/691295254 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb314108630 0x7fb31419d640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:33:12.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:11.011+0000 7fb312000700 1 -- 172.21.15.90:0/691295254 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb314108630 msgr2=0x7fb31419d640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:33:12.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:11.011+0000 7fb312000700 1 --2- 172.21.15.90:0/691295254 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb314108630 0x7fb31419d640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:12.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:11.011+0000 7fb312000700 1 -- 172.21.15.90:0/691295254 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb314102640 msgr2=0x7fb314198d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:33:12.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:11.011+0000 7fb312000700 1 --2- 172.21.15.90:0/691295254 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb314102640 0x7fb314198d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:12.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:11.011+0000 7fb312000700 1 -- 172.21.15.90:0/691295254 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb31419dd80 con 0x7fb314103000 2024-04-02T14:33:12.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:11.011+0000 7fb313400700 1 --2- 172.21.15.90:0/691295254 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb314108630 0x7fb31419d640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:33:12.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:11.011+0000 7fb312a00700 1 --2- 172.21.15.90:0/691295254 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb314102640 0x7fb314198d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:33:12.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:11.011+0000 7fb312000700 1 --2- 172.21.15.90:0/691295254 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb314103000 0x7fb3141992b0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fb30000ec10 tx=0x7fb30000ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:33:12.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:11.011+0000 7fb30b400700 1 -- 172.21.15.90:0/691295254 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb30000cd40 con 0x7fb314103000 2024-04-02T14:33:12.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:11.012+0000 7fb30b400700 1 -- 172.21.15.90:0/691295254 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb300004500 con 0x7fb314103000 2024-04-02T14:33:12.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:11.012+0000 7fb319600700 1 -- 172.21.15.90:0/691295254 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb31419e070 con 0x7fb314103000 2024-04-02T14:33:12.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:11.012+0000 7fb30b400700 1 -- 172.21.15.90:0/691295254 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb300010690 con 0x7fb314103000 2024-04-02T14:33:12.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:11.012+0000 7fb319600700 1 -- 172.21.15.90:0/691295254 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3141a58f0 con 0x7fb314103000 2024-04-02T14:33:12.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:11.013+0000 7fb319600700 1 -- 172.21.15.90:0/691295254 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fb314062d00 con 0x7fb314103000 2024-04-02T14:33:12.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:11.015+0000 7fb30b400700 1 -- 172.21.15.90:0/691295254 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fb300003f10 con 0x7fb314103000 2024-04-02T14:33:12.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:11.017+0000 7fb30b400700 1 --2- 172.21.15.90:0/691295254 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fb2f80a3cd0 0x7fb2f80a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:33:12.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:11.017+0000 7fb30b400700 1 -- 172.21.15.90:0/691295254 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8248+0+0 (secure 0 0 0) 0x7fb3000d0c40 con 0x7fb314103000 2024-04-02T14:33:12.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:11.017+0000 7fb312a00700 1 --2- 172.21.15.90:0/691295254 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fb2f80a3cd0 0x7fb2f80a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:33:12.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:11.017+0000 7fb30b400700 1 --2- 172.21.15.90:0/691295254 >> [v2:172.21.15.90:6802/3635087609,v1:172.21.15.90:6803/3635087609] conn(0x7fb2f80ad890 0x7fb2f80afd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:33:12.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:11.018+0000 7fb30b400700 1 -- 172.21.15.90:0/691295254 --> [v2:172.21.15.90:6802/3635087609,v1:172.21.15.90:6803/3635087609] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb30000e1a0 con 0x7fb2f80ad890 2024-04-02T14:33:12.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:11.018+0000 7fb30b400700 1 -- 172.21.15.90:0/691295254 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7fb3000d0ff0 con 0x7fb314103000 2024-04-02T14:33:12.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:11.018+0000 7fb313400700 1 --2- 172.21.15.90:0/691295254 >> [v2:172.21.15.90:6802/3635087609,v1:172.21.15.90:6803/3635087609] conn(0x7fb2f80ad890 0x7fb2f80afd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:33:12.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:11.018+0000 7fb312a00700 1 --2- 172.21.15.90:0/691295254 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fb2f80a3cd0 0x7fb2f80a6190 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fb304009fd0 tx=0x7fb304009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:33:12.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:11.018+0000 7fb313400700 1 --2- 172.21.15.90:0/691295254 >> [v2:172.21.15.90:6802/3635087609,v1:172.21.15.90:6803/3635087609] conn(0x7fb2f80ad890 0x7fb2f80afd30 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:33:12.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:11.019+0000 7fb30b400700 1 -- 172.21.15.90:0/691295254 <== osd.0 v2:172.21.15.90:6802/3635087609 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fb30000e1a0 con 0x7fb2f80ad890 2024-04-02T14:33:12.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:11.038+0000 7fb319600700 1 -- 172.21.15.90:0/691295254 --> [v2:172.21.15.90:6802/3635087609,v1:172.21.15.90:6803/3635087609] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fb314063180 con 0x7fb2f80ad890 2024-04-02T14:33:12.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:11.039+0000 7fb30b400700 1 -- 172.21.15.90:0/691295254 <== osd.0 v2:172.21.15.90:6802/3635087609 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fb314063180 con 0x7fb2f80ad890 2024-04-02T14:33:12.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:11.039+0000 7fb319600700 1 -- 172.21.15.90:0/691295254 >> [v2:172.21.15.90:6802/3635087609,v1:172.21.15.90:6803/3635087609] conn(0x7fb2f80ad890 msgr2=0x7fb2f80afd30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:33:12.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:11.039+0000 7fb319600700 1 --2- 172.21.15.90:0/691295254 >> [v2:172.21.15.90:6802/3635087609,v1:172.21.15.90:6803/3635087609] conn(0x7fb2f80ad890 0x7fb2f80afd30 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:12.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:11.039+0000 7fb319600700 1 -- 172.21.15.90:0/691295254 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fb2f80a3cd0 msgr2=0x7fb2f80a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:33:12.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:11.039+0000 7fb319600700 1 --2- 172.21.15.90:0/691295254 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fb2f80a3cd0 0x7fb2f80a6190 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fb304009fd0 tx=0x7fb304009260 comp rx=0 tx=0).stop 2024-04-02T14:33:12.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:11.039+0000 7fb319600700 1 -- 172.21.15.90:0/691295254 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb314103000 msgr2=0x7fb3141992b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:33:12.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:11.039+0000 7fb319600700 1 --2- 172.21.15.90:0/691295254 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb314103000 0x7fb3141992b0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fb30000ec10 tx=0x7fb30000ef20 comp rx=0 tx=0).stop 2024-04-02T14:33:12.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:11.039+0000 7fb313400700 1 -- 172.21.15.90:0/691295254 reap_dead start 2024-04-02T14:33:12.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:11.039+0000 7fb319600700 1 -- 172.21.15.90:0/691295254 shutdown_connections 2024-04-02T14:33:12.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:11.039+0000 7fb319600700 1 -- 172.21.15.90:0/691295254 >> 172.21.15.90:0/691295254 conn(0x7fb314079750 msgr2=0x7fb314100650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:33:12.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:11.040+0000 7fb319600700 1 -- 172.21.15.90:0/691295254 shutdown_connections 2024-04-02T14:33:12.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:11.040+0000 7fb319600700 1 -- 172.21.15.90:0/691295254 wait complete. 2024-04-02T14:33:12.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:33:11 smithi092 ceph-mon[36497]: pgmap v338: 97 pgs: 97 active+clean; 643 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 170 B/s wr, 10 op/s 2024-04-02T14:33:12.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:33:11 smithi177 ceph-mon[36431]: pgmap v338: 97 pgs: 97 active+clean; 643 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 170 B/s wr, 10 op/s 2024-04-02T14:33:12.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:33:11 smithi090 ceph-mon[31674]: pgmap v338: 97 pgs: 97 active+clean; 643 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 170 B/s wr, 10 op/s 2024-04-02T14:33:12.796 INFO:teuthology.orchestra.run.smithi090.stdout:107374182466 2024-04-02T14:33:12.797 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:191d963ac9de7322da8aff20b567b3fa458e2ef6 shell --fsid 6449448a-f0fc-11ee-b647-cb9ed24678a4 -- ceph osd last-stat-seq osd.3 2024-04-02T14:33:13.275 INFO:teuthology.orchestra.run.smithi090.stdout:55834574928 2024-04-02T14:33:13.275 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:191d963ac9de7322da8aff20b567b3fa458e2ef6 shell --fsid 6449448a-f0fc-11ee-b647-cb9ed24678a4 -- ceph osd last-stat-seq osd.1 2024-04-02T14:33:13.995 INFO:teuthology.orchestra.run.smithi090.stdout:77309411402 2024-04-02T14:33:13.995 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:191d963ac9de7322da8aff20b567b3fa458e2ef6 shell --fsid 6449448a-f0fc-11ee-b647-cb9ed24678a4 -- ceph osd last-stat-seq osd.2 2024-04-02T14:33:14.110 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:33:13 smithi092 ceph-mon[36497]: pgmap v339: 97 pgs: 97 active+clean; 643 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-02T14:33:14.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:33:13 smithi177 ceph-mon[36431]: pgmap v339: 97 pgs: 97 active+clean; 643 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-02T14:33:14.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:33:13 smithi090 ceph-mon[31674]: pgmap v339: 97 pgs: 97 active+clean; 643 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-02T14:33:14.426 INFO:teuthology.orchestra.run.smithi090.stdout:197568495666 2024-04-02T14:33:14.427 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:191d963ac9de7322da8aff20b567b3fa458e2ef6 shell --fsid 6449448a-f0fc-11ee-b647-cb9ed24678a4 -- ceph osd last-stat-seq osd.6 2024-04-02T14:33:15.336 INFO:teuthology.orchestra.run.smithi090.stdout:34359738453 2024-04-02T14:33:15.337 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:191d963ac9de7322da8aff20b567b3fa458e2ef6 shell --fsid 6449448a-f0fc-11ee-b647-cb9ed24678a4 -- ceph osd last-stat-seq osd.0 2024-04-02T14:33:15.939 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:33:15 smithi090 ceph-mon[31674]: pgmap v340: 97 pgs: 97 active+clean; 643 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 op/s 2024-04-02T14:33:16.110 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:33:15 smithi092 ceph-mon[36497]: pgmap v340: 97 pgs: 97 active+clean; 643 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 op/s 2024-04-02T14:33:16.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:33:15 smithi177 ceph-mon[36431]: pgmap v340: 97 pgs: 97 active+clean; 643 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 op/s 2024-04-02T14:33:16.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:16.011+0000 7f59b2000700 1 -- 172.21.15.90:0/2291003256 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f59ac1011a0 msgr2=0x7f59ac10d6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:33:16.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:16.011+0000 7f59b2000700 1 --2- 172.21.15.90:0/2291003256 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f59ac1011a0 0x7f59ac10d6b0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f59a400b0a0 tx=0x7f59a400b3b0 comp rx=0 tx=0).stop 2024-04-02T14:33:16.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:16.011+0000 7f59b2000700 1 -- 172.21.15.90:0/2291003256 shutdown_connections 2024-04-02T14:33:16.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:16.011+0000 7f59b2000700 1 --2- 172.21.15.90:0/2291003256 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f59ac1011a0 0x7f59ac10d6b0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:16.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:16.011+0000 7f59b2000700 1 --2- 172.21.15.90:0/2291003256 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f59ac1067e0 0x7f59ac106bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:16.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:16.011+0000 7f59b2000700 1 --2- 172.21.15.90:0/2291003256 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f59ac1007e0 0x7f59ac100c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:16.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:16.011+0000 7f59b2000700 1 -- 172.21.15.90:0/2291003256 >> 172.21.15.90:0/2291003256 conn(0x7f59ac0776c0 msgr2=0x7f59ac0fea40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:33:16.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:16.012+0000 7f59b2000700 1 -- 172.21.15.90:0/2291003256 shutdown_connections 2024-04-02T14:33:16.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:16.012+0000 7f59b2000700 1 -- 172.21.15.90:0/2291003256 wait complete. 2024-04-02T14:33:16.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:16.013+0000 7f59b2000700 1 Processor -- start 2024-04-02T14:33:16.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:16.013+0000 7f59b2000700 1 -- start start 2024-04-02T14:33:16.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:16.013+0000 7f59b2000700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f59ac1007e0 0x7f59ac19af30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:33:16.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:16.013+0000 7f59b2000700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f59ac1011a0 0x7f59ac19b470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:33:16.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:16.013+0000 7f59b2000700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f59ac1067e0 0x7f59ac19f800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:33:16.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:16.013+0000 7f59b2000700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f59ac110770 con 0x7f59ac1067e0 2024-04-02T14:33:16.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:16.013+0000 7f59b2000700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f59ac1105f0 con 0x7f59ac1011a0 2024-04-02T14:33:16.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:16.013+0000 7f59b2000700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f59ac1108f0 con 0x7f59ac1007e0 2024-04-02T14:33:16.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:16.013+0000 7f59abe00700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f59ac1067e0 0x7f59ac19f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:33:16.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:16.013+0000 7f59abe00700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f59ac1067e0 0x7f59ac19f800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:53334/0 (socket says 172.21.15.90:53334) 2024-04-02T14:33:16.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:16.013+0000 7f59abe00700 1 -- 172.21.15.90:0/2336659031 learned_addr learned my addr 172.21.15.90:0/2336659031 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:33:16.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:16.013+0000 7f59aaa00700 1 --2- 172.21.15.90:0/2336659031 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f59ac1011a0 0x7f59ac19b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:33:16.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:16.013+0000 7f59ab400700 1 --2- 172.21.15.90:0/2336659031 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f59ac1007e0 0x7f59ac19af30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:33:16.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:16.014+0000 7f59ab400700 1 -- 172.21.15.90:0/2336659031 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f59ac1011a0 msgr2=0x7f59ac19b470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:33:16.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:16.014+0000 7f59ab400700 1 --2- 172.21.15.90:0/2336659031 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f59ac1011a0 0x7f59ac19b470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:16.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:16.014+0000 7f59ab400700 1 -- 172.21.15.90:0/2336659031 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f59ac1067e0 msgr2=0x7f59ac19f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:33:16.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:16.014+0000 7f59ab400700 1 --2- 172.21.15.90:0/2336659031 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f59ac1067e0 0x7f59ac19f800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:16.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:16.014+0000 7f59ab400700 1 -- 172.21.15.90:0/2336659031 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59ac19ff40 con 0x7f59ac1007e0 2024-04-02T14:33:16.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:16.014+0000 7f59abe00700 1 --2- 172.21.15.90:0/2336659031 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f59ac1067e0 0x7f59ac19f800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:33:16.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:16.014+0000 7f59ab400700 1 --2- 172.21.15.90:0/2336659031 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f59ac1007e0 0x7f59ac19af30 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f599c0097d0 tx=0x7f599c009ae0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:33:16.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:16.014+0000 7f59a3e00700 1 -- 172.21.15.90:0/2336659031 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f599c013790 con 0x7f59ac1007e0 2024-04-02T14:33:16.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:16.014+0000 7f59a3e00700 1 -- 172.21.15.90:0/2336659031 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f599c019070 con 0x7f59ac1007e0 2024-04-02T14:33:16.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:16.014+0000 7f59b2000700 1 -- 172.21.15.90:0/2336659031 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59ac1a0230 con 0x7f59ac1007e0 2024-04-02T14:33:16.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:16.014+0000 7f59a3e00700 1 -- 172.21.15.90:0/2336659031 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f599c0122e0 con 0x7f59ac1007e0 2024-04-02T14:33:16.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:16.015+0000 7f59b2000700 1 -- 172.21.15.90:0/2336659031 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59ac1a7ab0 con 0x7f59ac1007e0 2024-04-02T14:33:16.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:16.015+0000 7f59b2000700 1 -- 172.21.15.90:0/2336659031 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f59ac103d50 con 0x7f59ac1007e0 2024-04-02T14:33:16.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:16.018+0000 7f59a3e00700 1 -- 172.21.15.90:0/2336659031 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f599c0145b0 con 0x7f59ac1007e0 2024-04-02T14:33:16.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:16.019+0000 7f59a3e00700 1 --2- 172.21.15.90:0/2336659031 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f59900a3cd0 0x7f59900a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:33:16.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:16.019+0000 7f59a3e00700 1 -- 172.21.15.90:0/2336659031 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8248+0+0 (secure 0 0 0) 0x7f599c0d0610 con 0x7f59ac1007e0 2024-04-02T14:33:16.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:16.019+0000 7f59aaa00700 1 --2- 172.21.15.90:0/2336659031 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f59900a3cd0 0x7f59900a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:33:16.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:16.019+0000 7f59a3e00700 1 --2- 172.21.15.90:0/2336659031 >> [v2:172.21.15.92:6824/1594153142,v1:172.21.15.92:6825/1594153142] conn(0x7f59900ad890 0x7f59900afd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:33:16.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:16.019+0000 7f59a3e00700 1 -- 172.21.15.90:0/2336659031 --> [v2:172.21.15.92:6824/1594153142,v1:172.21.15.92:6825/1594153142] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f59900b0400 con 0x7f59900ad890 2024-04-02T14:33:16.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:16.019+0000 7f59a3e00700 1 -- 172.21.15.90:0/2336659031 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7f599c0d09c0 con 0x7f59ac1007e0 2024-04-02T14:33:16.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:16.020+0000 7f59abe00700 1 --2- 172.21.15.90:0/2336659031 >> [v2:172.21.15.92:6824/1594153142,v1:172.21.15.92:6825/1594153142] conn(0x7f59900ad890 0x7f59900afd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:33:16.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:16.020+0000 7f59aaa00700 1 --2- 172.21.15.90:0/2336659031 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f59900a3cd0 0x7f59900a6190 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f59ac105300 tx=0x7f5998009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:33:16.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:16.020+0000 7f59abe00700 1 --2- 172.21.15.90:0/2336659031 >> [v2:172.21.15.92:6824/1594153142,v1:172.21.15.92:6825/1594153142] conn(0x7f59900ad890 0x7f59900afd30 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.10 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:33:16.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:16.021+0000 7f59a3e00700 1 -- 172.21.15.90:0/2336659031 <== osd.10 v2:172.21.15.92:6824/1594153142 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f59900b0400 con 0x7f59900ad890 2024-04-02T14:33:16.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:16.039+0000 7f59b2000700 1 -- 172.21.15.90:0/2336659031 --> [v2:172.21.15.92:6824/1594153142,v1:172.21.15.92:6825/1594153142] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f59ac101f20 con 0x7f59900ad890 2024-04-02T14:33:16.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:16.040+0000 7f59a3e00700 1 -- 172.21.15.90:0/2336659031 <== osd.10 v2:172.21.15.92:6824/1594153142 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f59ac101f20 con 0x7f59900ad890 2024-04-02T14:33:16.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:16.040+0000 7f59b2000700 1 -- 172.21.15.90:0/2336659031 >> [v2:172.21.15.92:6824/1594153142,v1:172.21.15.92:6825/1594153142] conn(0x7f59900ad890 msgr2=0x7f59900afd30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:33:16.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:16.040+0000 7f59b2000700 1 --2- 172.21.15.90:0/2336659031 >> [v2:172.21.15.92:6824/1594153142,v1:172.21.15.92:6825/1594153142] conn(0x7f59900ad890 0x7f59900afd30 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:16.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:16.041+0000 7f59b2000700 1 -- 172.21.15.90:0/2336659031 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f59900a3cd0 msgr2=0x7f59900a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:33:16.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:16.041+0000 7f59b2000700 1 --2- 172.21.15.90:0/2336659031 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f59900a3cd0 0x7f59900a6190 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f59ac105300 tx=0x7f5998009260 comp rx=0 tx=0).stop 2024-04-02T14:33:16.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:16.041+0000 7f59b2000700 1 -- 172.21.15.90:0/2336659031 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f59ac1007e0 msgr2=0x7f59ac19af30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:33:16.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:16.041+0000 7f59b2000700 1 --2- 172.21.15.90:0/2336659031 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f59ac1007e0 0x7f59ac19af30 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f599c0097d0 tx=0x7f599c009ae0 comp rx=0 tx=0).stop 2024-04-02T14:33:16.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:16.041+0000 7f59abe00700 1 -- 172.21.15.90:0/2336659031 reap_dead start 2024-04-02T14:33:16.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:16.041+0000 7f59b2000700 1 -- 172.21.15.90:0/2336659031 shutdown_connections 2024-04-02T14:33:16.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:16.041+0000 7f59b2000700 1 -- 172.21.15.90:0/2336659031 >> 172.21.15.90:0/2336659031 conn(0x7f59ac0776c0 msgr2=0x7f59ac1032f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:33:16.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:16.041+0000 7f59b2000700 1 -- 172.21.15.90:0/2336659031 shutdown_connections 2024-04-02T14:33:16.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:16.041+0000 7f59b2000700 1 -- 172.21.15.90:0/2336659031 wait complete. 2024-04-02T14:33:17.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:17.265+0000 7f8ca1a00700 1 -- 172.21.15.90:0/348059270 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8c9c101520 msgr2=0x7f8c9c1019a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:33:17.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:17.265+0000 7f8ca1a00700 1 --2- 172.21.15.90:0/348059270 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8c9c101520 0x7f8c9c1019a0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f8c880099c0 tx=0x7f8c88009cd0 comp rx=0 tx=0).stop 2024-04-02T14:33:17.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:17.266+0000 7f8ca1a00700 1 -- 172.21.15.90:0/348059270 shutdown_connections 2024-04-02T14:33:17.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:17.266+0000 7f8ca1a00700 1 --2- 172.21.15.90:0/348059270 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8c9c101ee0 0x7f8c9c10e330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:17.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:17.266+0000 7f8ca1a00700 1 --2- 172.21.15.90:0/348059270 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8c9c107560 0x7f8c9c107940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:17.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:17.266+0000 7f8ca1a00700 1 --2- 172.21.15.90:0/348059270 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8c9c101520 0x7f8c9c1019a0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:17.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:17.266+0000 7f8ca1a00700 1 -- 172.21.15.90:0/348059270 >> 172.21.15.90:0/348059270 conn(0x7f8c9c077540 msgr2=0x7f8c9c077950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:33:17.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:17.266+0000 7f8ca1a00700 1 -- 172.21.15.90:0/348059270 shutdown_connections 2024-04-02T14:33:17.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:17.266+0000 7f8ca1a00700 1 -- 172.21.15.90:0/348059270 wait complete. 2024-04-02T14:33:17.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:17.267+0000 7f8ca1a00700 1 Processor -- start 2024-04-02T14:33:17.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:17.267+0000 7f8ca1a00700 1 -- start start 2024-04-02T14:33:17.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:17.267+0000 7f8ca1a00700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8c9c101520 0x7f8c9c199ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:33:17.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:17.267+0000 7f8ca1a00700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8c9c101ee0 0x7f8c9c19a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:33:17.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:17.267+0000 7f8ca1a00700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8c9c107560 0x7f8c9c1a4c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:33:17.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:17.267+0000 7f8ca1a00700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f8c9c1117e0 con 0x7f8c9c101ee0 2024-04-02T14:33:17.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:17.267+0000 7f8ca1a00700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f8c9c111660 con 0x7f8c9c101520 2024-04-02T14:33:17.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:17.267+0000 7f8ca1a00700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f8c9c111960 con 0x7f8c9c107560 2024-04-02T14:33:17.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:17.268+0000 7f8c9a000700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8c9c101ee0 0x7f8c9c19a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:33:17.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:17.268+0000 7f8c9aa00700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8c9c101520 0x7f8c9c199ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:33:17.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:17.268+0000 7f8c9aa00700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8c9c101520 0x7f8c9c199ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:59088/0 (socket says 172.21.15.90:59088) 2024-04-02T14:33:17.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:17.268+0000 7f8c9aa00700 1 -- 172.21.15.90:0/2297292368 learned_addr learned my addr 172.21.15.90:0/2297292368 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:33:17.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:17.268+0000 7f8c9b400700 1 --2- 172.21.15.90:0/2297292368 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8c9c107560 0x7f8c9c1a4c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:33:17.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:17.268+0000 7f8c9b400700 1 -- 172.21.15.90:0/2297292368 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8c9c101520 msgr2=0x7f8c9c199ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:33:17.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:17.268+0000 7f8c9b400700 1 --2- 172.21.15.90:0/2297292368 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8c9c101520 0x7f8c9c199ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:17.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:17.268+0000 7f8c9b400700 1 -- 172.21.15.90:0/2297292368 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8c9c101ee0 msgr2=0x7f8c9c19a0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:33:17.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:17.268+0000 7f8c9b400700 1 --2- 172.21.15.90:0/2297292368 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8c9c101ee0 0x7f8c9c19a0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:17.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:17.268+0000 7f8c9b400700 1 -- 172.21.15.90:0/2297292368 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c9c1a5160 con 0x7f8c9c107560 2024-04-02T14:33:17.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:17.269+0000 7f8c9b400700 1 --2- 172.21.15.90:0/2297292368 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8c9c107560 0x7f8c9c1a4c20 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f8c9400efc0 tx=0x7f8c9400c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:33:17.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:17.269+0000 7f8c93400700 1 -- 172.21.15.90:0/2297292368 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c940092e0 con 0x7f8c9c107560 2024-04-02T14:33:17.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:17.269+0000 7f8ca1a00700 1 -- 172.21.15.90:0/2297292368 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c9c1a5330 con 0x7f8c9c107560 2024-04-02T14:33:17.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:17.269+0000 7f8c93400700 1 -- 172.21.15.90:0/2297292368 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8c94004bb0 con 0x7f8c9c107560 2024-04-02T14:33:17.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:17.269+0000 7f8c93400700 1 -- 172.21.15.90:0/2297292368 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c94004280 con 0x7f8c9c107560 2024-04-02T14:33:17.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:17.269+0000 7f8ca1a00700 1 -- 172.21.15.90:0/2297292368 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c9c1a57f0 con 0x7f8c9c107560 2024-04-02T14:33:17.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:17.270+0000 7f8ca1a00700 1 -- 172.21.15.90:0/2297292368 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f8c9c062d80 con 0x7f8c9c107560 2024-04-02T14:33:17.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:17.272+0000 7f8c93400700 1 -- 172.21.15.90:0/2297292368 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f8c9401b030 con 0x7f8c9c107560 2024-04-02T14:33:17.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:17.273+0000 7f8c93400700 1 --2- 172.21.15.90:0/2297292368 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f8c800a3cd0 0x7f8c800a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:33:17.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:17.274+0000 7f8c93400700 1 -- 172.21.15.90:0/2297292368 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8248+0+0 (secure 0 0 0) 0x7f8c940d0a80 con 0x7f8c9c107560 2024-04-02T14:33:17.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:17.274+0000 7f8c9aa00700 1 --2- 172.21.15.90:0/2297292368 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f8c800a3cd0 0x7f8c800a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:33:17.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:17.274+0000 7f8c93400700 1 --2- 172.21.15.90:0/2297292368 >> [v2:172.21.15.177:6808/3486820237,v1:172.21.15.177:6809/3486820237] conn(0x7f8c800ad890 0x7f8c800afd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:33:17.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:17.274+0000 7f8c93400700 1 -- 172.21.15.90:0/2297292368 --> [v2:172.21.15.177:6808/3486820237,v1:172.21.15.177:6809/3486820237] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8c800b0400 con 0x7f8c800ad890 2024-04-02T14:33:17.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:17.274+0000 7f8c93400700 1 -- 172.21.15.90:0/2297292368 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7f8c940d0e30 con 0x7f8c9c107560 2024-04-02T14:33:17.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:17.274+0000 7f8c9a000700 1 --2- 172.21.15.90:0/2297292368 >> [v2:172.21.15.177:6808/3486820237,v1:172.21.15.177:6809/3486820237] conn(0x7f8c800ad890 0x7f8c800afd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:33:17.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:17.274+0000 7f8c9aa00700 1 --2- 172.21.15.90:0/2297292368 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f8c800a3cd0 0x7f8c800a6190 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f8c8c009ca0 tx=0x7f8c8c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:33:17.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:17.275+0000 7f8c9a000700 1 --2- 172.21.15.90:0/2297292368 >> [v2:172.21.15.177:6808/3486820237,v1:172.21.15.177:6809/3486820237] conn(0x7f8c800ad890 0x7f8c800afd30 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:33:17.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:17.276+0000 7f8c93400700 1 -- 172.21.15.90:0/2297292368 <== osd.5 v2:172.21.15.177:6808/3486820237 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f8c800b0400 con 0x7f8c800ad890 2024-04-02T14:33:17.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:17.294+0000 7f8ca1a00700 1 -- 172.21.15.90:0/2297292368 --> [v2:172.21.15.177:6808/3486820237,v1:172.21.15.177:6809/3486820237] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f8c9c063200 con 0x7f8c800ad890 2024-04-02T14:33:17.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:17.295+0000 7f8c93400700 1 -- 172.21.15.90:0/2297292368 <== osd.5 v2:172.21.15.177:6808/3486820237 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f8c9c063200 con 0x7f8c800ad890 2024-04-02T14:33:17.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:17.295+0000 7f8ca1a00700 1 -- 172.21.15.90:0/2297292368 >> [v2:172.21.15.177:6808/3486820237,v1:172.21.15.177:6809/3486820237] conn(0x7f8c800ad890 msgr2=0x7f8c800afd30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:33:17.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:17.295+0000 7f8ca1a00700 1 --2- 172.21.15.90:0/2297292368 >> [v2:172.21.15.177:6808/3486820237,v1:172.21.15.177:6809/3486820237] conn(0x7f8c800ad890 0x7f8c800afd30 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:17.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:17.295+0000 7f8ca1a00700 1 -- 172.21.15.90:0/2297292368 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f8c800a3cd0 msgr2=0x7f8c800a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:33:17.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:17.295+0000 7f8ca1a00700 1 --2- 172.21.15.90:0/2297292368 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f8c800a3cd0 0x7f8c800a6190 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f8c8c009ca0 tx=0x7f8c8c009260 comp rx=0 tx=0).stop 2024-04-02T14:33:17.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:17.295+0000 7f8ca1a00700 1 -- 172.21.15.90:0/2297292368 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8c9c107560 msgr2=0x7f8c9c1a4c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:33:17.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:17.295+0000 7f8ca1a00700 1 --2- 172.21.15.90:0/2297292368 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8c9c107560 0x7f8c9c1a4c20 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f8c9400efc0 tx=0x7f8c9400c3a0 comp rx=0 tx=0).stop 2024-04-02T14:33:17.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:17.295+0000 7f8c9b400700 1 -- 172.21.15.90:0/2297292368 reap_dead start 2024-04-02T14:33:17.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:17.295+0000 7f8ca1a00700 1 -- 172.21.15.90:0/2297292368 shutdown_connections 2024-04-02T14:33:17.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:17.295+0000 7f8ca1a00700 1 -- 172.21.15.90:0/2297292368 >> 172.21.15.90:0/2297292368 conn(0x7f8c9c077540 msgr2=0x7f8c9c100100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:33:17.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:17.296+0000 7f8ca1a00700 1 -- 172.21.15.90:0/2297292368 shutdown_connections 2024-04-02T14:33:17.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:17.296+0000 7f8ca1a00700 1 -- 172.21.15.90:0/2297292368 wait complete. 2024-04-02T14:33:17.666 INFO:teuthology.orchestra.run.smithi090.stdout:322122547231 2024-04-02T14:33:17.670 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:191d963ac9de7322da8aff20b567b3fa458e2ef6 shell --fsid 6449448a-f0fc-11ee-b647-cb9ed24678a4 -- ceph osd last-stat-seq osd.10 2024-04-02T14:33:18.002 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T14:33:18.003 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T14:33:18.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:33:17 smithi092 ceph-mon[36497]: pgmap v341: 97 pgs: 97 active+clean; 643 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 10 op/s 2024-04-02T14:33:18.162 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:33:17 smithi090 ceph-mon[31674]: pgmap v341: 97 pgs: 97 active+clean; 643 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 10 op/s 2024-04-02T14:33:18.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:33:17 smithi177 ceph-mon[36431]: pgmap v341: 97 pgs: 97 active+clean; 643 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 10 op/s 2024-04-02T14:33:18.357 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T14:33:18.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:18.613+0000 7f3824e00700 1 -- 172.21.15.90:0/2613973982 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f38201046e0 msgr2=0x7f3820104b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:33:18.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:18.613+0000 7f3824e00700 1 --2- 172.21.15.90:0/2613973982 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f38201046e0 0x7f3820104b60 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f380c009990 tx=0x7f380c009ca0 comp rx=0 tx=0).stop 2024-04-02T14:33:18.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:18.613+0000 7f3824e00700 1 -- 172.21.15.90:0/2613973982 shutdown_connections 2024-04-02T14:33:18.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:18.613+0000 7f3824e00700 1 --2- 172.21.15.90:0/2613973982 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f38201050a0 0x7f38201115b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:18.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:18.613+0000 7f3824e00700 1 --2- 172.21.15.90:0/2613973982 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f382010a6e0 0x7f382010aac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:18.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:18.613+0000 7f3824e00700 1 --2- 172.21.15.90:0/2613973982 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f38201046e0 0x7f3820104b60 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:18.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:18.614+0000 7f3824e00700 1 -- 172.21.15.90:0/2613973982 >> 172.21.15.90:0/2613973982 conn(0x7f3820100210 msgr2=0x7f3820102630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:33:18.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:18.614+0000 7f3824e00700 1 -- 172.21.15.90:0/2613973982 shutdown_connections 2024-04-02T14:33:18.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:18.614+0000 7f3824e00700 1 -- 172.21.15.90:0/2613973982 wait complete. 2024-04-02T14:33:18.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:18.615+0000 7f3824e00700 1 Processor -- start 2024-04-02T14:33:18.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:18.615+0000 7f3824e00700 1 -- start start 2024-04-02T14:33:18.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:18.615+0000 7f3824e00700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f38201046e0 0x7f382019ae80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:33:18.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:18.615+0000 7f3824e00700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f38201050a0 0x7f382019b3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:33:18.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:18.615+0000 7f3824e00700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f382010a6e0 0x7f382019f700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:33:18.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:18.615+0000 7f3824e00700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f3820114a70 con 0x7f38201050a0 2024-04-02T14:33:18.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:18.615+0000 7f3824e00700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f38201148f0 con 0x7f38201046e0 2024-04-02T14:33:18.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:18.615+0000 7f3824e00700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f3820114bf0 con 0x7f382010a6e0 2024-04-02T14:33:18.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:18.615+0000 7f381ea00700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f38201050a0 0x7f382019b3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:33:18.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:18.615+0000 7f381f400700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f38201046e0 0x7f382019ae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:33:18.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:18.615+0000 7f381ea00700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f38201050a0 0x7f382019b3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:53374/0 (socket says 172.21.15.90:53374) 2024-04-02T14:33:18.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:18.615+0000 7f381ea00700 1 -- 172.21.15.90:0/3889694444 learned_addr learned my addr 172.21.15.90:0/3889694444 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:33:18.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:18.616+0000 7f381fe00700 1 --2- 172.21.15.90:0/3889694444 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f382010a6e0 0x7f382019f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:33:18.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:18.616+0000 7f381f400700 1 -- 172.21.15.90:0/3889694444 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f382010a6e0 msgr2=0x7f382019f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:33:18.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:18.616+0000 7f381f400700 1 --2- 172.21.15.90:0/3889694444 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f382010a6e0 0x7f382019f700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:18.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:18.616+0000 7f381f400700 1 -- 172.21.15.90:0/3889694444 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f38201050a0 msgr2=0x7f382019b3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:33:18.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:18.616+0000 7f381f400700 1 --2- 172.21.15.90:0/3889694444 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f38201050a0 0x7f382019b3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:18.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:18.616+0000 7f381f400700 1 -- 172.21.15.90:0/3889694444 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f382019fe40 con 0x7f38201046e0 2024-04-02T14:33:18.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:18.616+0000 7f381f400700 1 --2- 172.21.15.90:0/3889694444 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f38201046e0 0x7f382019ae80 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f381000d950 tx=0x7f381000dc60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:33:18.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:18.616+0000 7f3817e00700 1 -- 172.21.15.90:0/3889694444 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3810016070 con 0x7f38201046e0 2024-04-02T14:33:18.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:18.616+0000 7f3824e00700 1 -- 172.21.15.90:0/3889694444 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38201a0130 con 0x7f38201046e0 2024-04-02T14:33:18.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:18.616+0000 7f3817e00700 1 -- 172.21.15.90:0/3889694444 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f381000fd70 con 0x7f38201046e0 2024-04-02T14:33:18.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:18.617+0000 7f3817e00700 1 -- 172.21.15.90:0/3889694444 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f381000e3b0 con 0x7f38201046e0 2024-04-02T14:33:18.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:18.617+0000 7f3824e00700 1 -- 172.21.15.90:0/3889694444 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38201a79b0 con 0x7f38201046e0 2024-04-02T14:33:18.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:18.618+0000 7f3824e00700 1 -- 172.21.15.90:0/3889694444 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f3820062d80 con 0x7f38201046e0 2024-04-02T14:33:18.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:18.620+0000 7f3817e00700 1 -- 172.21.15.90:0/3889694444 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f3810019070 con 0x7f38201046e0 2024-04-02T14:33:18.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:18.621+0000 7f3817e00700 1 --2- 172.21.15.90:0/3889694444 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f38040a3cd0 0x7f38040a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:33:18.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:18.621+0000 7f3817e00700 1 -- 172.21.15.90:0/3889694444 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8248+0+0 (secure 0 0 0) 0x7f38100d0460 con 0x7f38201046e0 2024-04-02T14:33:18.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:18.621+0000 7f381ea00700 1 --2- 172.21.15.90:0/3889694444 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f38040a3cd0 0x7f38040a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:33:18.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:18.622+0000 7f3817e00700 1 --2- 172.21.15.90:0/3889694444 >> [v2:172.21.15.177:6824/1622582126,v1:172.21.15.177:6825/1622582126] conn(0x7f38040ad890 0x7f38040afd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:33:18.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:18.622+0000 7f3817e00700 1 -- 172.21.15.90:0/3889694444 --> [v2:172.21.15.177:6824/1622582126,v1:172.21.15.177:6825/1622582126] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f38040b0400 con 0x7f38040ad890 2024-04-02T14:33:18.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:18.622+0000 7f3817e00700 1 -- 172.21.15.90:0/3889694444 <== mon.2 v2:172.21.15.92:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7f38100d0810 con 0x7f38201046e0 2024-04-02T14:33:18.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:18.622+0000 7f381fe00700 1 --2- 172.21.15.90:0/3889694444 >> [v2:172.21.15.177:6824/1622582126,v1:172.21.15.177:6825/1622582126] conn(0x7f38040ad890 0x7f38040afd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:33:18.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:18.622+0000 7f381ea00700 1 --2- 172.21.15.90:0/3889694444 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f38040a3cd0 0x7f38040a6190 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f3820109ce0 tx=0x7f380c005be0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:33:18.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:18.622+0000 7f381fe00700 1 --2- 172.21.15.90:0/3889694444 >> [v2:172.21.15.177:6824/1622582126,v1:172.21.15.177:6825/1622582126] conn(0x7f38040ad890 0x7f38040afd30 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.11 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:33:18.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:18.623+0000 7f3817e00700 1 -- 172.21.15.90:0/3889694444 <== osd.11 v2:172.21.15.177:6824/1622582126 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f38040b0400 con 0x7f38040ad890 2024-04-02T14:33:18.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:18.641+0000 7f3824e00700 1 -- 172.21.15.90:0/3889694444 --> [v2:172.21.15.177:6824/1622582126,v1:172.21.15.177:6825/1622582126] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f3820063180 con 0x7f38040ad890 2024-04-02T14:33:18.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:18.642+0000 7f3817e00700 1 -- 172.21.15.90:0/3889694444 <== osd.11 v2:172.21.15.177:6824/1622582126 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f3820063180 con 0x7f38040ad890 2024-04-02T14:33:18.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:18.642+0000 7f3824e00700 1 -- 172.21.15.90:0/3889694444 >> [v2:172.21.15.177:6824/1622582126,v1:172.21.15.177:6825/1622582126] conn(0x7f38040ad890 msgr2=0x7f38040afd30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:33:18.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:18.642+0000 7f3824e00700 1 --2- 172.21.15.90:0/3889694444 >> [v2:172.21.15.177:6824/1622582126,v1:172.21.15.177:6825/1622582126] conn(0x7f38040ad890 0x7f38040afd30 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:18.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:18.642+0000 7f3824e00700 1 -- 172.21.15.90:0/3889694444 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f38040a3cd0 msgr2=0x7f38040a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:33:18.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:18.642+0000 7f3824e00700 1 --2- 172.21.15.90:0/3889694444 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f38040a3cd0 0x7f38040a6190 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f3820109ce0 tx=0x7f380c005be0 comp rx=0 tx=0).stop 2024-04-02T14:33:18.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:18.642+0000 7f3824e00700 1 -- 172.21.15.90:0/3889694444 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f38201046e0 msgr2=0x7f382019ae80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:33:18.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:18.642+0000 7f3824e00700 1 --2- 172.21.15.90:0/3889694444 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f38201046e0 0x7f382019ae80 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f381000d950 tx=0x7f381000dc60 comp rx=0 tx=0).stop 2024-04-02T14:33:18.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:18.643+0000 7f381fe00700 1 -- 172.21.15.90:0/3889694444 reap_dead start 2024-04-02T14:33:18.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:18.643+0000 7f3824e00700 1 -- 172.21.15.90:0/3889694444 shutdown_connections 2024-04-02T14:33:18.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:18.643+0000 7f3824e00700 1 -- 172.21.15.90:0/3889694444 >> 172.21.15.90:0/3889694444 conn(0x7f3820100210 msgr2=0x7f3820100810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:33:18.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:18.643+0000 7f3824e00700 1 -- 172.21.15.90:0/3889694444 shutdown_connections 2024-04-02T14:33:18.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:18.643+0000 7f3824e00700 1 -- 172.21.15.90:0/3889694444 wait complete. 2024-04-02T14:33:19.939 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:33:19 smithi090 ceph-mon[31674]: pgmap v342: 97 pgs: 97 active+clean; 643 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 9 op/s 2024-04-02T14:33:20.127 INFO:teuthology.orchestra.run.smithi090.stdout:163208757305 2024-04-02T14:33:20.128 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:191d963ac9de7322da8aff20b567b3fa458e2ef6 shell --fsid 6449448a-f0fc-11ee-b647-cb9ed24678a4 -- ceph osd last-stat-seq osd.5 2024-04-02T14:33:20.170 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T14:33:20.170 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T14:33:20.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:33:19 smithi177 ceph-mon[36431]: pgmap v342: 97 pgs: 97 active+clean; 643 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 9 op/s 2024-04-02T14:33:20.187 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T14:33:20.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:33:19 smithi092 ceph-mon[36497]: pgmap v342: 97 pgs: 97 active+clean; 643 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 9 op/s 2024-04-02T14:33:21.116 INFO:teuthology.orchestra.run.smithi090.stdout:356482285593 2024-04-02T14:33:21.116 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:191d963ac9de7322da8aff20b567b3fa458e2ef6 shell --fsid 6449448a-f0fc-11ee-b647-cb9ed24678a4 -- ceph osd last-stat-seq osd.11 2024-04-02T14:33:22.063 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:33:21 smithi090 ceph-mon[31674]: pgmap v343: 97 pgs: 97 active+clean; 643 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-02T14:33:22.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:33:21 smithi177 ceph-mon[36431]: pgmap v343: 97 pgs: 97 active+clean; 643 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-02T14:33:22.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:33:21 smithi092 ceph-mon[36497]: pgmap v343: 97 pgs: 97 active+clean; 643 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-02T14:33:22.938 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:33:22 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T14:33:23.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:33:22 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T14:33:23.252 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T14:33:23.256 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T14:33:23.360 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:33:22 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T14:33:23.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:33:23 smithi177 ceph-mon[36431]: pgmap v344: 97 pgs: 97 active+clean; 643 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-02T14:33:24.110 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:33:23 smithi092 ceph-mon[36497]: pgmap v344: 97 pgs: 97 active+clean; 643 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-02T14:33:24.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:33:23 smithi090 ceph-mon[31674]: pgmap v344: 97 pgs: 97 active+clean; 643 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-02T14:33:25.689 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:33:25 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:33:25.689 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:33:25 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:33:25.689 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:33:25 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:33:25.689 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:33:25 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:33:25.689 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:33:25 smithi090 ceph-mon[31674]: pgmap v345: 97 pgs: 97 active+clean; 643 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-02T14:33:25.861 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:33:25 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:33:25.861 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:33:25 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:33:25.861 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:33:25 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:33:25.861 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:33:25 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:33:25.862 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:33:25 smithi092 ceph-mon[36497]: pgmap v345: 97 pgs: 97 active+clean; 643 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-02T14:33:25.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:33:25 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:33:25.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:33:25 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:33:25.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:33:25 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:33:25.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:33:25 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:33:25.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:33:25 smithi177 ceph-mon[36431]: pgmap v345: 97 pgs: 97 active+clean; 643 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-02T14:33:28.110 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:33:27 smithi092 ceph-mon[36497]: pgmap v346: 97 pgs: 97 active+clean; 643 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-02T14:33:28.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:33:27 smithi177 ceph-mon[36431]: pgmap v346: 97 pgs: 97 active+clean; 643 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-02T14:33:28.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:33:27 smithi090 ceph-mon[31674]: pgmap v346: 97 pgs: 97 active+clean; 643 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-02T14:33:30.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:33:29 smithi092 ceph-mon[36497]: pgmap v347: 97 pgs: 97 active+clean; 643 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-04-02T14:33:30.174 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:33:29 smithi090 ceph-mon[31674]: pgmap v347: 97 pgs: 97 active+clean; 643 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-04-02T14:33:30.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:33:29 smithi177 ceph-mon[36431]: pgmap v347: 97 pgs: 97 active+clean; 643 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-04-02T14:33:31.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:31.502+0000 7f3cf9e00700 1 -- 172.21.15.90:0/3751583510 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3cf4108e60 msgr2=0x7f3cf4109240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:33:31.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:31.502+0000 7f3cf9e00700 1 --2- 172.21.15.90:0/3751583510 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3cf4108e60 0x7f3cf4109240 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f3ce4009920 tx=0x7f3ce4009c30 comp rx=0 tx=0).stop 2024-04-02T14:33:31.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:31.502+0000 7f3cf9e00700 1 -- 172.21.15.90:0/3751583510 shutdown_connections 2024-04-02T14:33:31.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:31.502+0000 7f3cf9e00700 1 --2- 172.21.15.90:0/3751583510 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3cf410df40 0x7f3cf4115b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:31.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:31.502+0000 7f3cf9e00700 1 --2- 172.21.15.90:0/3751583510 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3cf4108e60 0x7f3cf4109240 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:31.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:31.502+0000 7f3cf9e00700 1 --2- 172.21.15.90:0/3751583510 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3cf4109810 0x7f3cf410d860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:31.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:31.502+0000 7f3cf9e00700 1 -- 172.21.15.90:0/3751583510 >> 172.21.15.90:0/3751583510 conn(0x7f3cf4075dc0 msgr2=0x7f3cf40781e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:33:31.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:31.502+0000 7f3cf9e00700 1 -- 172.21.15.90:0/3751583510 shutdown_connections 2024-04-02T14:33:31.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:31.503+0000 7f3cf9e00700 1 -- 172.21.15.90:0/3751583510 wait complete. 2024-04-02T14:33:31.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:31.504+0000 7f3cf9e00700 1 Processor -- start 2024-04-02T14:33:31.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:31.504+0000 7f3cf9e00700 1 -- start start 2024-04-02T14:33:31.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:31.505+0000 7f3cf9e00700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3cf4108e60 0x7f3cf419f4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:33:31.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:31.505+0000 7f3cf9e00700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3cf4109810 0x7f3cf419fa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:33:31.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:31.505+0000 7f3cf9e00700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3cf410df40 0x7f3cf41a3d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:33:31.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:31.505+0000 7f3cf9e00700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f3cf4118f90 con 0x7f3cf410df40 2024-04-02T14:33:31.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:31.505+0000 7f3cf9e00700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f3cf4118e10 con 0x7f3cf4109810 2024-04-02T14:33:31.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:31.505+0000 7f3cf9e00700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f3cf4119110 con 0x7f3cf4108e60 2024-04-02T14:33:31.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:31.505+0000 7f3cf2a00700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3cf4108e60 0x7f3cf419f4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:33:31.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:31.505+0000 7f3cf2a00700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3cf4108e60 0x7f3cf419f4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:39458/0 (socket says 172.21.15.90:39458) 2024-04-02T14:33:31.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:31.505+0000 7f3cf2a00700 1 -- 172.21.15.90:0/1606016685 learned_addr learned my addr 172.21.15.90:0/1606016685 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:33:31.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:31.505+0000 7f3cf3400700 1 --2- 172.21.15.90:0/1606016685 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3cf410df40 0x7f3cf41a3d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:33:31.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:31.505+0000 7f3cf2000700 1 --2- 172.21.15.90:0/1606016685 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3cf4109810 0x7f3cf419fa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:33:31.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:31.506+0000 7f3cf2a00700 1 -- 172.21.15.90:0/1606016685 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3cf4109810 msgr2=0x7f3cf419fa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:33:31.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:31.506+0000 7f3cf2a00700 1 --2- 172.21.15.90:0/1606016685 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3cf4109810 0x7f3cf419fa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:31.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:31.506+0000 7f3cf2a00700 1 -- 172.21.15.90:0/1606016685 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3cf410df40 msgr2=0x7f3cf41a3d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:33:31.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:31.506+0000 7f3cf2a00700 1 --2- 172.21.15.90:0/1606016685 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3cf410df40 0x7f3cf41a3d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:31.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:31.506+0000 7f3cf2a00700 1 -- 172.21.15.90:0/1606016685 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3cf41a4480 con 0x7f3cf4108e60 2024-04-02T14:33:31.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:31.506+0000 7f3cf2a00700 1 --2- 172.21.15.90:0/1606016685 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3cf4108e60 0x7f3cf419f4c0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f3ce4000c00 tx=0x7f3ce400fa70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:33:31.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:31.506+0000 7f3cf8a00700 1 -- 172.21.15.90:0/1606016685 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ce4027070 con 0x7f3cf4108e60 2024-04-02T14:33:31.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:31.506+0000 7f3cf8a00700 1 -- 172.21.15.90:0/1606016685 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3ce4005620 con 0x7f3cf4108e60 2024-04-02T14:33:31.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:31.506+0000 7f3cf8a00700 1 -- 172.21.15.90:0/1606016685 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ce4004930 con 0x7f3cf4108e60 2024-04-02T14:33:31.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:31.507+0000 7f3cf9e00700 1 -- 172.21.15.90:0/1606016685 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3cf41a4710 con 0x7f3cf4108e60 2024-04-02T14:33:31.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:31.507+0000 7f3cf9e00700 1 -- 172.21.15.90:0/1606016685 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3cf41abff0 con 0x7f3cf4108e60 2024-04-02T14:33:31.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:31.508+0000 7f3cf9e00700 1 -- 172.21.15.90:0/1606016685 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3cf410a340 con 0x7f3cf4108e60 2024-04-02T14:33:31.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:31.512+0000 7f3cf8a00700 1 -- 172.21.15.90:0/1606016685 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f3ce402a070 con 0x7f3cf4108e60 2024-04-02T14:33:31.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:31.513+0000 7f3cf8a00700 1 --2- 172.21.15.90:0/1606016685 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f3cd80a3cd0 0x7f3cd80a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:33:31.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:31.513+0000 7f3cf8a00700 1 -- 172.21.15.90:0/1606016685 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8248+0+0 (secure 0 0 0) 0x7f3ce40d13d0 con 0x7f3cf4108e60 2024-04-02T14:33:31.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:31.513+0000 7f3cf2000700 1 --2- 172.21.15.90:0/1606016685 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f3cd80a3cd0 0x7f3cd80a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:33:31.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:31.513+0000 7f3cf8a00700 1 -- 172.21.15.90:0/1606016685 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7f3ce4031020 con 0x7f3cf4108e60 2024-04-02T14:33:31.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:31.514+0000 7f3cf2000700 1 --2- 172.21.15.90:0/1606016685 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f3cd80a3cd0 0x7f3cd80a6190 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f3cf4107ed0 tx=0x7f3ce0009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:33:31.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:31.758+0000 7f3cf9e00700 1 -- 172.21.15.90:0/1606016685 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f3cf41a07d0 con 0x7f3cf4108e60 2024-04-02T14:33:31.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:31.759+0000 7f3cf8a00700 1 -- 172.21.15.90:0/1606016685 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f3ce4013030 con 0x7f3cf4108e60 2024-04-02T14:33:31.914 INFO:teuthology.orchestra.run.smithi090.stdout:107374182471 2024-04-02T14:33:31.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:31.762+0000 7f3cf9e00700 1 -- 172.21.15.90:0/1606016685 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f3cd80a3cd0 msgr2=0x7f3cd80a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:33:31.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:31.762+0000 7f3cf9e00700 1 --2- 172.21.15.90:0/1606016685 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f3cd80a3cd0 0x7f3cd80a6190 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f3cf4107ed0 tx=0x7f3ce0009310 comp rx=0 tx=0).stop 2024-04-02T14:33:31.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:31.762+0000 7f3cf9e00700 1 -- 172.21.15.90:0/1606016685 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3cf4108e60 msgr2=0x7f3cf419f4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:33:31.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:31.762+0000 7f3cf9e00700 1 --2- 172.21.15.90:0/1606016685 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3cf4108e60 0x7f3cf419f4c0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f3ce4000c00 tx=0x7f3ce400fa70 comp rx=0 tx=0).stop 2024-04-02T14:33:31.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:31.762+0000 7f3cf9e00700 1 -- 172.21.15.90:0/1606016685 shutdown_connections 2024-04-02T14:33:31.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:31.762+0000 7f3cf9e00700 1 --2- 172.21.15.90:0/1606016685 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f3cd80a3cd0 0x7f3cd80a6190 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:31.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:31.762+0000 7f3cf9e00700 1 --2- 172.21.15.90:0/1606016685 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3cf410df40 0x7f3cf41a3d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:31.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:31.762+0000 7f3cf9e00700 1 --2- 172.21.15.90:0/1606016685 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3cf4108e60 0x7f3cf419f4c0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:31.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:31.762+0000 7f3cf9e00700 1 --2- 172.21.15.90:0/1606016685 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3cf4109810 0x7f3cf419fa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:31.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:31.762+0000 7f3cf9e00700 1 -- 172.21.15.90:0/1606016685 >> 172.21.15.90:0/1606016685 conn(0x7f3cf4075dc0 msgr2=0x7f3cf4078120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:33:31.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:31.763+0000 7f3cf9e00700 1 -- 172.21.15.90:0/1606016685 shutdown_connections 2024-04-02T14:33:31.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:31.763+0000 7f3cf9e00700 1 -- 172.21.15.90:0/1606016685 wait complete. 2024-04-02T14:33:32.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:33:31 smithi092 ceph-mon[36497]: pgmap v348: 97 pgs: 97 active+clean; 643 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 10 op/s 2024-04-02T14:33:32.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:33:31 smithi092 ceph-mon[36497]: from='client.? 172.21.15.90:0/1152580521' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-02T14:33:32.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:33:31 smithi092 ceph-mon[36497]: from='client.? 172.21.15.90:0/1155250529' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-02T14:33:32.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:33:31 smithi092 ceph-mon[36497]: from='client.? 172.21.15.90:0/1606016685' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-02T14:33:32.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:33:31 smithi177 ceph-mon[36431]: pgmap v348: 97 pgs: 97 active+clean; 643 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 10 op/s 2024-04-02T14:33:32.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:33:31 smithi177 ceph-mon[36431]: from='client.? 172.21.15.90:0/1152580521' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-02T14:33:32.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:33:31 smithi177 ceph-mon[36431]: from='client.? 172.21.15.90:0/1155250529' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-02T14:33:32.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:33:31 smithi177 ceph-mon[36431]: from='client.? 172.21.15.90:0/1606016685' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-02T14:33:32.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:33:31 smithi090 ceph-mon[31674]: pgmap v348: 97 pgs: 97 active+clean; 643 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 10 op/s 2024-04-02T14:33:32.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:33:31 smithi090 ceph-mon[31674]: from='client.? 172.21.15.90:0/1152580521' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-02T14:33:32.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:33:31 smithi090 ceph-mon[31674]: from='client.? 172.21.15.90:0/1155250529' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-02T14:33:32.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:33:31 smithi090 ceph-mon[31674]: from='client.? 172.21.15.90:0/1606016685' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-02T14:33:32.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:30.777+0000 7ff291c00700 1 -- 172.21.15.90:0/1251610499 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff28c1024f0 msgr2=0x7ff28c102970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:33:32.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:30.777+0000 7ff291c00700 1 --2- 172.21.15.90:0/1251610499 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff28c1024f0 0x7ff28c102970 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7ff2780099c0 tx=0x7ff278009cd0 comp rx=0 tx=0).stop 2024-04-02T14:33:32.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:30.778+0000 7ff291c00700 1 -- 172.21.15.90:0/1251610499 shutdown_connections 2024-04-02T14:33:32.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:30.778+0000 7ff291c00700 1 --2- 172.21.15.90:0/1251610499 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff28c102eb0 0x7ff28c10f370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:32.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:30.778+0000 7ff291c00700 1 --2- 172.21.15.90:0/1251610499 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff28c1085a0 0x7ff28c108980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:32.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:30.778+0000 7ff291c00700 1 --2- 172.21.15.90:0/1251610499 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff28c1024f0 0x7ff28c102970 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:32.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:30.778+0000 7ff291c00700 1 -- 172.21.15.90:0/1251610499 >> 172.21.15.90:0/1251610499 conn(0x7ff28c0fdfc0 msgr2=0x7ff28c1003e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:33:32.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:30.778+0000 7ff291c00700 1 -- 172.21.15.90:0/1251610499 shutdown_connections 2024-04-02T14:33:32.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:30.778+0000 7ff291c00700 1 -- 172.21.15.90:0/1251610499 wait complete. 2024-04-02T14:33:32.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:30.779+0000 7ff291c00700 1 Processor -- start 2024-04-02T14:33:32.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:30.779+0000 7ff291c00700 1 -- start start 2024-04-02T14:33:32.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:30.779+0000 7ff291c00700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff28c1024f0 0x7ff28c19ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:33:32.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:30.779+0000 7ff291c00700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff28c102eb0 0x7ff28c19b350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:33:32.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:30.779+0000 7ff291c00700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff28c1085a0 0x7ff28c19f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:33:32.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:30.779+0000 7ff291c00700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7ff28c112830 con 0x7ff28c1085a0 2024-04-02T14:33:32.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:30.779+0000 7ff291c00700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7ff28c1126b0 con 0x7ff28c102eb0 2024-04-02T14:33:32.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:30.779+0000 7ff291c00700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7ff28c1129b0 con 0x7ff28c1024f0 2024-04-02T14:33:32.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:30.780+0000 7ff291200700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff28c1085a0 0x7ff28c19f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:33:32.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:30.780+0000 7ff28b400700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff28c102eb0 0x7ff28c19b350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:33:32.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:30.780+0000 7ff28b400700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff28c102eb0 0x7ff28c19b350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:48290/0 (socket says 172.21.15.90:48290) 2024-04-02T14:33:32.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:30.780+0000 7ff291200700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff28c1085a0 0x7ff28c19f6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43986/0 (socket says 172.21.15.90:43986) 2024-04-02T14:33:32.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:30.780+0000 7ff28b400700 1 -- 172.21.15.90:0/1152580521 learned_addr learned my addr 172.21.15.90:0/1152580521 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:33:32.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:30.780+0000 7ff28be00700 1 --2- 172.21.15.90:0/1152580521 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff28c1024f0 0x7ff28c19ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:33:32.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:30.780+0000 7ff28b400700 1 -- 172.21.15.90:0/1152580521 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff28c1024f0 msgr2=0x7ff28c19ae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:33:32.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:30.780+0000 7ff28b400700 1 --2- 172.21.15.90:0/1152580521 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff28c1024f0 0x7ff28c19ae10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:32.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:30.780+0000 7ff28b400700 1 -- 172.21.15.90:0/1152580521 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff28c1085a0 msgr2=0x7ff28c19f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:33:32.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:30.780+0000 7ff28b400700 1 --2- 172.21.15.90:0/1152580521 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff28c1085a0 0x7ff28c19f6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:32.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:30.780+0000 7ff28b400700 1 -- 172.21.15.90:0/1152580521 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff28c19fe20 con 0x7ff28c102eb0 2024-04-02T14:33:32.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:30.780+0000 7ff28b400700 1 --2- 172.21.15.90:0/1152580521 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff28c102eb0 0x7ff28c19b350 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7ff278005910 tx=0x7ff27800fa00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:33:32.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:30.781+0000 7ff288c00700 1 -- 172.21.15.90:0/1152580521 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff278027070 con 0x7ff28c102eb0 2024-04-02T14:33:32.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:30.781+0000 7ff291c00700 1 -- 172.21.15.90:0/1152580521 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff28c1a00b0 con 0x7ff28c102eb0 2024-04-02T14:33:32.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:30.781+0000 7ff288c00700 1 -- 172.21.15.90:0/1152580521 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff278003680 con 0x7ff28c102eb0 2024-04-02T14:33:32.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:30.781+0000 7ff288c00700 1 -- 172.21.15.90:0/1152580521 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2780173f0 con 0x7ff28c102eb0 2024-04-02T14:33:32.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:30.781+0000 7ff291c00700 1 -- 172.21.15.90:0/1152580521 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff28c1a7990 con 0x7ff28c102eb0 2024-04-02T14:33:32.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:30.782+0000 7ff291c00700 1 -- 172.21.15.90:0/1152580521 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff28c0692b0 con 0x7ff28c102eb0 2024-04-02T14:33:32.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:30.784+0000 7ff288c00700 1 -- 172.21.15.90:0/1152580521 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7ff278022470 con 0x7ff28c102eb0 2024-04-02T14:33:32.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:30.785+0000 7ff288c00700 1 --2- 172.21.15.90:0/1152580521 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7ff2700a3d50 0x7ff2700a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:33:32.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:30.785+0000 7ff288c00700 1 -- 172.21.15.90:0/1152580521 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8248+0+0 (secure 0 0 0) 0x7ff2780d0ed0 con 0x7ff28c102eb0 2024-04-02T14:33:32.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:30.785+0000 7ff28be00700 1 --2- 172.21.15.90:0/1152580521 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7ff2700a3d50 0x7ff2700a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:33:32.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:30.786+0000 7ff28be00700 1 --2- 172.21.15.90:0/1152580521 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7ff2700a3d50 0x7ff2700a6210 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7ff27c009680 tx=0x7ff27c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:33:32.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:30.789+0000 7ff288c00700 1 -- 172.21.15.90:0/1152580521 <== mon.2 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7ff278013070 con 0x7ff28c102eb0 2024-04-02T14:33:32.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:31.028+0000 7ff291c00700 1 -- 172.21.15.90:0/1152580521 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 8} v 0) -- 0x7ff28c1a7b60 con 0x7ff28c102eb0 2024-04-02T14:33:32.250 INFO:teuthology.orchestra.run.smithi090.stdout:261993005101 2024-04-02T14:33:32.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:31.029+0000 7ff288c00700 1 -- 172.21.15.90:0/1152580521 <== mon.2 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 8}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7ff27809d770 con 0x7ff28c102eb0 2024-04-02T14:33:32.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:31.032+0000 7ff291c00700 1 -- 172.21.15.90:0/1152580521 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7ff2700a3d50 msgr2=0x7ff2700a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:33:32.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:31.032+0000 7ff291c00700 1 --2- 172.21.15.90:0/1152580521 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7ff2700a3d50 0x7ff2700a6210 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7ff27c009680 tx=0x7ff27c009260 comp rx=0 tx=0).stop 2024-04-02T14:33:32.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:31.032+0000 7ff291c00700 1 -- 172.21.15.90:0/1152580521 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff28c102eb0 msgr2=0x7ff28c19b350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:33:32.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:31.032+0000 7ff291c00700 1 --2- 172.21.15.90:0/1152580521 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff28c102eb0 0x7ff28c19b350 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7ff278005910 tx=0x7ff27800fa00 comp rx=0 tx=0).stop 2024-04-02T14:33:32.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:31.032+0000 7ff291c00700 1 -- 172.21.15.90:0/1152580521 shutdown_connections 2024-04-02T14:33:32.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:31.032+0000 7ff291c00700 1 --2- 172.21.15.90:0/1152580521 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7ff2700a3d50 0x7ff2700a6210 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:32.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:31.032+0000 7ff291c00700 1 --2- 172.21.15.90:0/1152580521 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff28c1085a0 0x7ff28c19f6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:32.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:31.032+0000 7ff291c00700 1 --2- 172.21.15.90:0/1152580521 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff28c1024f0 0x7ff28c19ae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:32.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:31.032+0000 7ff291c00700 1 --2- 172.21.15.90:0/1152580521 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff28c102eb0 0x7ff28c19b350 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:32.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:31.032+0000 7ff291c00700 1 -- 172.21.15.90:0/1152580521 >> 172.21.15.90:0/1152580521 conn(0x7ff28c0fdfc0 msgr2=0x7ff28c0ff400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:33:32.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:31.032+0000 7ff291c00700 1 -- 172.21.15.90:0/1152580521 shutdown_connections 2024-04-02T14:33:32.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:31.033+0000 7ff291c00700 1 -- 172.21.15.90:0/1152580521 wait complete. 2024-04-02T14:33:32.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:31.638+0000 7fa120000700 1 -- 172.21.15.90:0/15172140 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa118108650 msgr2=0x7fa118108a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:33:32.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:31.638+0000 7fa120000700 1 --2- 172.21.15.90:0/15172140 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa118108650 0x7fa118108a30 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fa10c009990 tx=0x7fa10c009ca0 comp rx=0 tx=0).stop 2024-04-02T14:33:32.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:31.639+0000 7fa120000700 1 -- 172.21.15.90:0/15172140 shutdown_connections 2024-04-02T14:33:32.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:31.639+0000 7fa120000700 1 --2- 172.21.15.90:0/15172140 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa118108650 0x7fa118108a30 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:32.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:31.639+0000 7fa120000700 1 --2- 172.21.15.90:0/15172140 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa11806ac60 0x7fa11810f450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:32.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:31.639+0000 7fa120000700 1 --2- 172.21.15.90:0/15172140 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa11806a2a0 0x7fa11806a720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:32.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:31.639+0000 7fa120000700 1 -- 172.21.15.90:0/15172140 >> 172.21.15.90:0/15172140 conn(0x7fa118078420 msgr2=0x7fa118078830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:33:32.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:31.639+0000 7fa120000700 1 -- 172.21.15.90:0/15172140 shutdown_connections 2024-04-02T14:33:32.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:31.639+0000 7fa120000700 1 -- 172.21.15.90:0/15172140 wait complete. 2024-04-02T14:33:32.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:31.640+0000 7fa120000700 1 Processor -- start 2024-04-02T14:33:32.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:31.640+0000 7fa120000700 1 -- start start 2024-04-02T14:33:32.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:31.640+0000 7fa120000700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa11806a2a0 0x7fa118198de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:33:32.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:31.640+0000 7fa120000700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa11806ac60 0x7fa118199320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:33:32.815 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:31.640+0000 7fa120000700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa118108650 0x7fa11819d660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:33:32.815 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:31.640+0000 7fa120000700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fa1181128d0 con 0x7fa11806ac60 2024-04-02T14:33:32.815 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:31.640+0000 7fa120000700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fa118112750 con 0x7fa118108650 2024-04-02T14:33:32.815 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:31.640+0000 7fa120000700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fa118112a50 con 0x7fa11806a2a0 2024-04-02T14:33:32.815 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:31.641+0000 7fa11cc00700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa11806ac60 0x7fa118199320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:33:32.815 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:31.641+0000 7fa11cc00700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa11806ac60 0x7fa118199320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:44046/0 (socket says 172.21.15.90:44046) 2024-04-02T14:33:32.815 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:31.641+0000 7fa11d600700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa11806a2a0 0x7fa118198de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:33:32.815 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:31.641+0000 7fa11cc00700 1 -- 172.21.15.90:0/2831512031 learned_addr learned my addr 172.21.15.90:0/2831512031 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:33:32.815 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:31.641+0000 7fa11e000700 1 --2- 172.21.15.90:0/2831512031 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa118108650 0x7fa11819d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:33:32.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:31.641+0000 7fa11d600700 1 -- 172.21.15.90:0/2831512031 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa118108650 msgr2=0x7fa11819d660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:33:32.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:31.641+0000 7fa11d600700 1 --2- 172.21.15.90:0/2831512031 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa118108650 0x7fa11819d660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:32.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:31.641+0000 7fa11d600700 1 -- 172.21.15.90:0/2831512031 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa11806ac60 msgr2=0x7fa118199320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:33:32.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:31.641+0000 7fa11d600700 1 --2- 172.21.15.90:0/2831512031 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa11806ac60 0x7fa118199320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:32.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:31.641+0000 7fa11d600700 1 -- 172.21.15.90:0/2831512031 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa11819dda0 con 0x7fa11806a2a0 2024-04-02T14:33:32.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:31.641+0000 7fa11e000700 1 --2- 172.21.15.90:0/2831512031 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa118108650 0x7fa11819d660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:33:32.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:31.641+0000 7fa11cc00700 1 --2- 172.21.15.90:0/2831512031 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa11806ac60 0x7fa118199320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:33:32.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:31.642+0000 7fa11d600700 1 --2- 172.21.15.90:0/2831512031 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa11806a2a0 0x7fa118198de0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fa10c00b5c0 tx=0x7fa10c0047b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:33:32.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:31.642+0000 7fa116000700 1 -- 172.21.15.90:0/2831512031 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa10c004830 con 0x7fa11806a2a0 2024-04-02T14:33:32.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:31.642+0000 7fa120000700 1 -- 172.21.15.90:0/2831512031 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa11819e030 con 0x7fa11806a2a0 2024-04-02T14:33:32.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:31.642+0000 7fa116000700 1 -- 172.21.15.90:0/2831512031 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa10c005020 con 0x7fa11806a2a0 2024-04-02T14:33:32.817 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:31.642+0000 7fa116000700 1 -- 172.21.15.90:0/2831512031 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa10c003c30 con 0x7fa11806a2a0 2024-04-02T14:33:32.817 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:31.642+0000 7fa120000700 1 -- 172.21.15.90:0/2831512031 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1181a5910 con 0x7fa11806a2a0 2024-04-02T14:33:32.817 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:31.643+0000 7fa120000700 1 -- 172.21.15.90:0/2831512031 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa118103cf0 con 0x7fa11806a2a0 2024-04-02T14:33:32.817 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:31.645+0000 7fa116000700 1 -- 172.21.15.90:0/2831512031 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fa10c029060 con 0x7fa11806a2a0 2024-04-02T14:33:32.817 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:31.647+0000 7fa116000700 1 --2- 172.21.15.90:0/2831512031 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fa1000a3cd0 0x7fa1000a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:33:32.817 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:31.647+0000 7fa116000700 1 -- 172.21.15.90:0/2831512031 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8248+0+0 (secure 0 0 0) 0x7fa10c0d0fd0 con 0x7fa11806a2a0 2024-04-02T14:33:32.817 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:31.647+0000 7fa11cc00700 1 --2- 172.21.15.90:0/2831512031 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fa1000a3cd0 0x7fa1000a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:33:32.818 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:31.648+0000 7fa11cc00700 1 --2- 172.21.15.90:0/2831512031 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fa1000a3cd0 0x7fa1000a6190 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fa118078a90 tx=0x7fa10400b500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:33:32.818 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:31.651+0000 7fa116000700 1 -- 172.21.15.90:0/2831512031 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7fa10c013070 con 0x7fa11806a2a0 2024-04-02T14:33:32.818 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:31.901+0000 7fa120000700 1 -- 172.21.15.90:0/2831512031 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7fa11802dd30 con 0x7fa11806a2a0 2024-04-02T14:33:32.818 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:31.902+0000 7fa116000700 1 -- 172.21.15.90:0/2831512031 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fa10c09d870 con 0x7fa11806a2a0 2024-04-02T14:33:32.818 INFO:teuthology.orchestra.run.smithi090.stdout:197568495670 2024-04-02T14:33:32.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:31.905+0000 7fa120000700 1 -- 172.21.15.90:0/2831512031 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fa1000a3cd0 msgr2=0x7fa1000a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:33:32.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:31.905+0000 7fa120000700 1 --2- 172.21.15.90:0/2831512031 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fa1000a3cd0 0x7fa1000a6190 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fa118078a90 tx=0x7fa10400b500 comp rx=0 tx=0).stop 2024-04-02T14:33:32.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:31.905+0000 7fa120000700 1 -- 172.21.15.90:0/2831512031 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa11806a2a0 msgr2=0x7fa118198de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:33:32.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:31.905+0000 7fa120000700 1 --2- 172.21.15.90:0/2831512031 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa11806a2a0 0x7fa118198de0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fa10c00b5c0 tx=0x7fa10c0047b0 comp rx=0 tx=0).stop 2024-04-02T14:33:32.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:31.905+0000 7fa120000700 1 -- 172.21.15.90:0/2831512031 shutdown_connections 2024-04-02T14:33:32.820 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:31.905+0000 7fa120000700 1 --2- 172.21.15.90:0/2831512031 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fa1000a3cd0 0x7fa1000a6190 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:32.820 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:31.905+0000 7fa120000700 1 --2- 172.21.15.90:0/2831512031 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa118108650 0x7fa11819d660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:32.820 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:31.905+0000 7fa120000700 1 --2- 172.21.15.90:0/2831512031 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa11806a2a0 0x7fa118198de0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:32.820 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:31.905+0000 7fa120000700 1 --2- 172.21.15.90:0/2831512031 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa11806ac60 0x7fa118199320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:32.820 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:31.905+0000 7fa120000700 1 -- 172.21.15.90:0/2831512031 >> 172.21.15.90:0/2831512031 conn(0x7fa118078420 msgr2=0x7fa118100180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:33:32.820 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:31.905+0000 7fa120000700 1 -- 172.21.15.90:0/2831512031 shutdown_connections 2024-04-02T14:33:32.820 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:31.905+0000 7fa120000700 1 -- 172.21.15.90:0/2831512031 wait complete. 2024-04-02T14:33:32.939 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:33:32 smithi090 ceph-mon[31674]: from='client.? 172.21.15.90:0/2831512031' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-02T14:33:33.110 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:33:32 smithi092 ceph-mon[36497]: from='client.? 172.21.15.90:0/2831512031' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-02T14:33:33.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:33:32 smithi177 ceph-mon[36431]: from='client.? 172.21.15.90:0/2831512031' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-02T14:33:33.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:33.201+0000 7f42c0c00700 1 -- 172.21.15.90:0/2998559497 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f42bc10f0a0 msgr2=0x7f42bc111490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:33:33.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:33.201+0000 7f42c0c00700 1 --2- 172.21.15.90:0/2998559497 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f42bc10f0a0 0x7f42bc111490 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f42b400b0a0 tx=0x7f42b400b3b0 comp rx=0 tx=0).stop 2024-04-02T14:33:33.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:33.201+0000 7f42c0c00700 1 -- 172.21.15.90:0/2998559497 shutdown_connections 2024-04-02T14:33:33.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:33.201+0000 7f42c0c00700 1 --2- 172.21.15.90:0/2998559497 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f42bc10f0a0 0x7f42bc111490 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:33.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:33.201+0000 7f42c0c00700 1 --2- 172.21.15.90:0/2998559497 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f42bc06a2a0 0x7f42bc06a680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:33.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:33.201+0000 7f42c0c00700 1 --2- 172.21.15.90:0/2998559497 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f42bc06abc0 0x7f42bc10eb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:33.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:33.201+0000 7f42c0c00700 1 -- 172.21.15.90:0/2998559497 >> 172.21.15.90:0/2998559497 conn(0x7f42bc078420 msgr2=0x7f42bc078830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:33:33.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:33.202+0000 7f42c0c00700 1 -- 172.21.15.90:0/2998559497 shutdown_connections 2024-04-02T14:33:33.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:33.202+0000 7f42c0c00700 1 -- 172.21.15.90:0/2998559497 wait complete. 2024-04-02T14:33:33.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:33.203+0000 7f42c0c00700 1 Processor -- start 2024-04-02T14:33:33.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:33.203+0000 7f42c0c00700 1 -- start start 2024-04-02T14:33:33.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:33.203+0000 7f42c0c00700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f42bc06a2a0 0x7f42bc19b140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:33:33.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:33.204+0000 7f42c0c00700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f42bc06abc0 0x7f42bc19b680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:33:33.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:33.204+0000 7f42c0c00700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f42bc10f0a0 0x7f42bc19f9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:33:33.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:33.204+0000 7f42c0c00700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f42bc114b10 con 0x7f42bc06a2a0 2024-04-02T14:33:33.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:33.204+0000 7f42c0c00700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f42bc114990 con 0x7f42bc06abc0 2024-04-02T14:33:33.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:33.204+0000 7f42c0c00700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f42bc114c90 con 0x7f42bc10f0a0 2024-04-02T14:33:33.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:33.204+0000 7f42ba800700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f42bc10f0a0 0x7f42bc19f9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:33:33.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:33.204+0000 7f42ba800700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f42bc10f0a0 0x7f42bc19f9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:39500/0 (socket says 172.21.15.90:39500) 2024-04-02T14:33:33.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:33.204+0000 7f42ba800700 1 -- 172.21.15.90:0/2734148832 learned_addr learned my addr 172.21.15.90:0/2734148832 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:33:33.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:33.204+0000 7f42b9e00700 1 --2- 172.21.15.90:0/2734148832 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f42bc06a2a0 0x7f42bc19b140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:33:33.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:33.204+0000 7f42ba800700 1 -- 172.21.15.90:0/2734148832 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f42bc06abc0 msgr2=0x7f42bc19b680 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T14:33:33.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:33.204+0000 7f42ba800700 1 --2- 172.21.15.90:0/2734148832 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f42bc06abc0 0x7f42bc19b680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:33.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:33.205+0000 7f42ba800700 1 -- 172.21.15.90:0/2734148832 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f42bc06a2a0 msgr2=0x7f42bc19b140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:33:33.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:33.205+0000 7f42ba800700 1 --2- 172.21.15.90:0/2734148832 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f42bc06a2a0 0x7f42bc19b140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:33.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:33.205+0000 7f42ba800700 1 -- 172.21.15.90:0/2734148832 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42bc1a0100 con 0x7f42bc10f0a0 2024-04-02T14:33:33.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:33.205+0000 7f42ba800700 1 --2- 172.21.15.90:0/2734148832 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f42bc10f0a0 0x7f42bc19f9c0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f42b4006070 tx=0x7f42b4004d10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:33:33.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:33.205+0000 7f42b2a00700 1 -- 172.21.15.90:0/2734148832 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42b4021070 con 0x7f42bc10f0a0 2024-04-02T14:33:33.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:33.205+0000 7f42b2a00700 1 -- 172.21.15.90:0/2734148832 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f42b4003c00 con 0x7f42bc10f0a0 2024-04-02T14:33:33.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:33.205+0000 7f42c0c00700 1 -- 172.21.15.90:0/2734148832 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f42bc1a0390 con 0x7f42bc10f0a0 2024-04-02T14:33:33.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:33.206+0000 7f42c0c00700 1 -- 172.21.15.90:0/2734148832 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42bc1a7c70 con 0x7f42bc10f0a0 2024-04-02T14:33:33.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:33.206+0000 7f42b2a00700 1 -- 172.21.15.90:0/2734148832 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42b4004830 con 0x7f42bc10f0a0 2024-04-02T14:33:33.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:33.207+0000 7f42c0c00700 1 -- 172.21.15.90:0/2734148832 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f429c005330 con 0x7f42bc10f0a0 2024-04-02T14:33:33.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:33.209+0000 7f42b2a00700 1 -- 172.21.15.90:0/2734148832 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f42b4007980 con 0x7f42bc10f0a0 2024-04-02T14:33:33.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:33.212+0000 7f42b2a00700 1 --2- 172.21.15.90:0/2734148832 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f42a00a3d50 0x7f42a00a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:33:33.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:33.212+0000 7f42b2a00700 1 -- 172.21.15.90:0/2734148832 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8248+0+0 (secure 0 0 0) 0x7f42b400f020 con 0x7f42bc10f0a0 2024-04-02T14:33:33.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:33.212+0000 7f42b9e00700 1 --2- 172.21.15.90:0/2734148832 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f42a00a3d50 0x7f42a00a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:33:33.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:33.213+0000 7f42b9e00700 1 --2- 172.21.15.90:0/2734148832 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f42a00a3d50 0x7f42a00a6210 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f42ac009bf0 tx=0x7f42ac009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:33:33.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:33.214+0000 7f42b2a00700 1 -- 172.21.15.90:0/2734148832 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7f42b402a020 con 0x7f42bc10f0a0 2024-04-02T14:33:33.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:33.465+0000 7f42c0c00700 1 -- 172.21.15.90:0/2734148832 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f429c0058f0 con 0x7f42bc10f0a0 2024-04-02T14:33:33.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:33.466+0000 7f42b2a00700 1 -- 172.21.15.90:0/2734148832 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f42b409e570 con 0x7f42bc10f0a0 2024-04-02T14:33:33.468 INFO:teuthology.orchestra.run.smithi090.stdout:55834574932 2024-04-02T14:33:33.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:33.468+0000 7f42c0c00700 1 -- 172.21.15.90:0/2734148832 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f42a00a3d50 msgr2=0x7f42a00a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:33:33.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:33.469+0000 7f42c0c00700 1 --2- 172.21.15.90:0/2734148832 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f42a00a3d50 0x7f42a00a6210 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f42ac009bf0 tx=0x7f42ac009260 comp rx=0 tx=0).stop 2024-04-02T14:33:33.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:33.469+0000 7f42c0c00700 1 -- 172.21.15.90:0/2734148832 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f42bc10f0a0 msgr2=0x7f42bc19f9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:33:33.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:33.469+0000 7f42c0c00700 1 --2- 172.21.15.90:0/2734148832 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f42bc10f0a0 0x7f42bc19f9c0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f42b4006070 tx=0x7f42b4004d10 comp rx=0 tx=0).stop 2024-04-02T14:33:33.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:33.469+0000 7f42c0c00700 1 -- 172.21.15.90:0/2734148832 shutdown_connections 2024-04-02T14:33:33.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:33.469+0000 7f42c0c00700 1 --2- 172.21.15.90:0/2734148832 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f42a00a3d50 0x7f42a00a6210 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:33.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:33.469+0000 7f42c0c00700 1 --2- 172.21.15.90:0/2734148832 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f42bc06a2a0 0x7f42bc19b140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:33.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:33.469+0000 7f42c0c00700 1 --2- 172.21.15.90:0/2734148832 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f42bc10f0a0 0x7f42bc19f9c0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:33.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:33.469+0000 7f42c0c00700 1 --2- 172.21.15.90:0/2734148832 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f42bc06abc0 0x7f42bc19b680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:33.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:33.469+0000 7f42c0c00700 1 -- 172.21.15.90:0/2734148832 >> 172.21.15.90:0/2734148832 conn(0x7f42bc078420 msgr2=0x7f42bc110590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:33:33.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:33.469+0000 7f42c0c00700 1 -- 172.21.15.90:0/2734148832 shutdown_connections 2024-04-02T14:33:33.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:33.469+0000 7f42c0c00700 1 -- 172.21.15.90:0/2734148832 wait complete. 2024-04-02T14:33:33.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:31.379+0000 7f12aca00700 1 -- 172.21.15.90:0/1924377134 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f12a806ac60 msgr2=0x7f12a810f450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:33:33.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:31.379+0000 7f12aca00700 1 --2- 172.21.15.90:0/1924377134 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f12a806ac60 0x7f12a810f450 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f12a0009f90 tx=0x7f12a000b330 comp rx=0 tx=0).stop 2024-04-02T14:33:33.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:31.379+0000 7f12aca00700 1 -- 172.21.15.90:0/1924377134 shutdown_connections 2024-04-02T14:33:33.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:31.379+0000 7f12aca00700 1 --2- 172.21.15.90:0/1924377134 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f12a806ac60 0x7f12a810f450 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:33.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:31.379+0000 7f12aca00700 1 --2- 172.21.15.90:0/1924377134 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f12a8108650 0x7f12a8108a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:33.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:31.379+0000 7f12aca00700 1 --2- 172.21.15.90:0/1924377134 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f12a806a2a0 0x7f12a806a720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:33.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:31.379+0000 7f12aca00700 1 -- 172.21.15.90:0/1924377134 >> 172.21.15.90:0/1924377134 conn(0x7f12a8078420 msgr2=0x7f12a8078830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:33:33.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:31.379+0000 7f12aca00700 1 -- 172.21.15.90:0/1924377134 shutdown_connections 2024-04-02T14:33:33.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:31.380+0000 7f12aca00700 1 -- 172.21.15.90:0/1924377134 wait complete. 2024-04-02T14:33:33.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:31.382+0000 7f12aca00700 1 Processor -- start 2024-04-02T14:33:33.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:31.382+0000 7f12aca00700 1 -- start start 2024-04-02T14:33:33.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:31.382+0000 7f12aca00700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f12a806a2a0 0x7f12a819b000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:33:33.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:31.382+0000 7f12aca00700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f12a806ac60 0x7f12a819b540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:33:33.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:31.382+0000 7f12aca00700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f12a8108650 0x7f12a819f8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:33:33.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:31.382+0000 7f12aca00700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f12a81128f0 con 0x7f12a8108650 2024-04-02T14:33:33.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:31.382+0000 7f12aca00700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f12a8112770 con 0x7f12a806ac60 2024-04-02T14:33:33.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:31.382+0000 7f12aca00700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f12a8112a70 con 0x7f12a806a2a0 2024-04-02T14:33:33.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:31.383+0000 7f12a5400700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f12a806ac60 0x7f12a819b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:33:33.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:31.383+0000 7f12a6800700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f12a8108650 0x7f12a819f8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:33:33.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:31.383+0000 7f12a6800700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f12a8108650 0x7f12a819f8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:44006/0 (socket says 172.21.15.90:44006) 2024-04-02T14:33:33.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:31.383+0000 7f12a6800700 1 -- 172.21.15.90:0/1155250529 learned_addr learned my addr 172.21.15.90:0/1155250529 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:33:33.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:31.383+0000 7f12a5e00700 1 --2- 172.21.15.90:0/1155250529 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f12a806a2a0 0x7f12a819b000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:33:33.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:31.383+0000 7f12a6800700 1 -- 172.21.15.90:0/1155250529 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f12a806a2a0 msgr2=0x7f12a819b000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:33:33.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:31.383+0000 7f12a6800700 1 --2- 172.21.15.90:0/1155250529 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f12a806a2a0 0x7f12a819b000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:33.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:31.383+0000 7f12a6800700 1 -- 172.21.15.90:0/1155250529 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f12a806ac60 msgr2=0x7f12a819b540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:33:33.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:31.383+0000 7f12a6800700 1 --2- 172.21.15.90:0/1155250529 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f12a806ac60 0x7f12a819b540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:33.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:31.383+0000 7f12a6800700 1 -- 172.21.15.90:0/1155250529 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12a81a0010 con 0x7f12a8108650 2024-04-02T14:33:33.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:31.383+0000 7f12a6800700 1 --2- 172.21.15.90:0/1155250529 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f12a8108650 0x7f12a819f8d0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f12a000d010 tx=0x7f12a0009510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:33:33.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:31.384+0000 7f129ea00700 1 -- 172.21.15.90:0/1155250529 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12a0021070 con 0x7f12a8108650 2024-04-02T14:33:33.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:31.384+0000 7f129ea00700 1 -- 172.21.15.90:0/1155250529 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f12a000be20 con 0x7f12a8108650 2024-04-02T14:33:33.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:31.384+0000 7f12aca00700 1 -- 172.21.15.90:0/1155250529 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f12a81a02a0 con 0x7f12a8108650 2024-04-02T14:33:33.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:31.384+0000 7f129ea00700 1 -- 172.21.15.90:0/1155250529 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12a00052a0 con 0x7f12a8108650 2024-04-02T14:33:33.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:31.384+0000 7f12aca00700 1 -- 172.21.15.90:0/1155250529 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12a81a7b80 con 0x7f12a8108650 2024-04-02T14:33:33.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:31.385+0000 7f12aca00700 1 -- 172.21.15.90:0/1155250529 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1288005330 con 0x7f12a8108650 2024-04-02T14:33:33.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:31.387+0000 7f129ea00700 1 -- 172.21.15.90:0/1155250529 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f12a0024070 con 0x7f12a8108650 2024-04-02T14:33:33.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:31.389+0000 7f129ea00700 1 --2- 172.21.15.90:0/1155250529 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f128c0a3cd0 0x7f128c0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:33:33.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:31.390+0000 7f129ea00700 1 -- 172.21.15.90:0/1155250529 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8248+0+0 (secure 0 0 0) 0x7f12a00d13a0 con 0x7f12a8108650 2024-04-02T14:33:33.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:31.390+0000 7f12a5e00700 1 --2- 172.21.15.90:0/1155250529 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f128c0a3cd0 0x7f128c0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:33:33.712 INFO:teuthology.orchestra.run.smithi090.stdout:296352743462 2024-04-02T14:33:33.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:31.390+0000 7f12a5e00700 1 --2- 172.21.15.90:0/1155250529 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f128c0a3cd0 0x7f128c0a6190 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f1298009750 tx=0x7f1298009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:33:33.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:31.394+0000 7f129ea00700 1 -- 172.21.15.90:0/1155250529 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7f12a009dc40 con 0x7f12a8108650 2024-04-02T14:33:33.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:31.629+0000 7f12aca00700 1 -- 172.21.15.90:0/1155250529 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 9} v 0) -- 0x7f12880058f0 con 0x7f12a8108650 2024-04-02T14:33:33.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:31.629+0000 7f129ea00700 1 -- 172.21.15.90:0/1155250529 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 9}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f12a009d540 con 0x7f12a8108650 2024-04-02T14:33:33.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:31.633+0000 7f12aca00700 1 -- 172.21.15.90:0/1155250529 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f128c0a3cd0 msgr2=0x7f128c0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:33:33.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:31.633+0000 7f12aca00700 1 --2- 172.21.15.90:0/1155250529 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f128c0a3cd0 0x7f128c0a6190 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f1298009750 tx=0x7f1298009310 comp rx=0 tx=0).stop 2024-04-02T14:33:33.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:31.633+0000 7f12aca00700 1 -- 172.21.15.90:0/1155250529 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f12a8108650 msgr2=0x7f12a819f8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:33:33.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:31.633+0000 7f12aca00700 1 --2- 172.21.15.90:0/1155250529 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f12a8108650 0x7f12a819f8d0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f12a000d010 tx=0x7f12a0009510 comp rx=0 tx=0).stop 2024-04-02T14:33:33.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:31.634+0000 7f12aca00700 1 -- 172.21.15.90:0/1155250529 shutdown_connections 2024-04-02T14:33:33.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:31.634+0000 7f12aca00700 1 --2- 172.21.15.90:0/1155250529 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f128c0a3cd0 0x7f128c0a6190 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:33.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:31.634+0000 7f12aca00700 1 --2- 172.21.15.90:0/1155250529 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f12a8108650 0x7f12a819f8d0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:33.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:31.634+0000 7f12aca00700 1 --2- 172.21.15.90:0/1155250529 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f12a806a2a0 0x7f12a819b000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:33.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:31.634+0000 7f12aca00700 1 --2- 172.21.15.90:0/1155250529 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f12a806ac60 0x7f12a819b540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:33.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:31.634+0000 7f12aca00700 1 -- 172.21.15.90:0/1155250529 >> 172.21.15.90:0/1155250529 conn(0x7f12a8078420 msgr2=0x7f12a810d7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:33:33.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:31.634+0000 7f12aca00700 1 -- 172.21.15.90:0/1155250529 shutdown_connections 2024-04-02T14:33:33.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:31.634+0000 7f12aca00700 1 -- 172.21.15.90:0/1155250529 wait complete. 2024-04-02T14:33:34.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:33:33 smithi177 ceph-mon[36431]: pgmap v349: 97 pgs: 97 active+clean; 643 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-02T14:33:34.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:33:33 smithi177 ceph-mon[36431]: from='client.? 172.21.15.90:0/2734148832' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-02T14:33:34.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:33:33 smithi090 ceph-mon[31674]: pgmap v349: 97 pgs: 97 active+clean; 643 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-02T14:33:34.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:33:33 smithi090 ceph-mon[31674]: from='client.? 172.21.15.90:0/2734148832' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-02T14:33:34.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:33:33 smithi092 ceph-mon[36497]: pgmap v349: 97 pgs: 97 active+clean; 643 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-02T14:33:34.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:33:33 smithi092 ceph-mon[36497]: from='client.? 172.21.15.90:0/2734148832' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-02T14:33:34.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:34.452+0000 7f9bad400700 1 -- 172.21.15.90:0/766122737 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9ba81051a0 msgr2=0x7f9ba81116b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:33:34.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:34.452+0000 7f9bad400700 1 --2- 172.21.15.90:0/766122737 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9ba81051a0 0x7f9ba81116b0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f9ba000b0a0 tx=0x7f9ba000b3b0 comp rx=0 tx=0).stop 2024-04-02T14:33:34.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:34.452+0000 7f9bad400700 1 -- 172.21.15.90:0/766122737 shutdown_connections 2024-04-02T14:33:34.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:34.452+0000 7f9bad400700 1 --2- 172.21.15.90:0/766122737 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9ba81051a0 0x7f9ba81116b0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:34.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:34.452+0000 7f9bad400700 1 --2- 172.21.15.90:0/766122737 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9ba810a7e0 0x7f9ba810abc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:34.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:34.452+0000 7f9bad400700 1 --2- 172.21.15.90:0/766122737 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9ba81047e0 0x7f9ba8104c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:34.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:34.452+0000 7f9bad400700 1 -- 172.21.15.90:0/766122737 >> 172.21.15.90:0/766122737 conn(0x7f9ba81002d0 msgr2=0x7f9ba81026f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:33:34.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:34.452+0000 7f9bad400700 1 -- 172.21.15.90:0/766122737 shutdown_connections 2024-04-02T14:33:34.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:34.452+0000 7f9bad400700 1 -- 172.21.15.90:0/766122737 wait complete. 2024-04-02T14:33:34.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:34.453+0000 7f9bad400700 1 Processor -- start 2024-04-02T14:33:34.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:34.453+0000 7f9bad400700 1 -- start start 2024-04-02T14:33:34.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:34.454+0000 7f9bad400700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9ba81047e0 0x7f9ba819afb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:33:34.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:34.454+0000 7f9bad400700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9ba81051a0 0x7f9ba819b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:33:34.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:34.454+0000 7f9bad400700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9ba810a7e0 0x7f9ba819f880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:33:34.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:34.454+0000 7f9bad400700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f9ba8114bc0 con 0x7f9ba81047e0 2024-04-02T14:33:34.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:34.454+0000 7f9bad400700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f9ba8114a40 con 0x7f9ba810a7e0 2024-04-02T14:33:34.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:34.454+0000 7f9bad400700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f9ba8114d40 con 0x7f9ba81051a0 2024-04-02T14:33:34.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:34.454+0000 7f9ba6800700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9ba81047e0 0x7f9ba819afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:33:34.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:34.454+0000 7f9ba6800700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9ba81047e0 0x7f9ba819afb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34936/0 (socket says 172.21.15.90:34936) 2024-04-02T14:33:34.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:34.454+0000 7f9ba6800700 1 -- 172.21.15.90:0/2620289033 learned_addr learned my addr 172.21.15.90:0/2620289033 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:33:34.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:34.454+0000 7f9ba5e00700 1 --2- 172.21.15.90:0/2620289033 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9ba81051a0 0x7f9ba819b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:33:34.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:34.454+0000 7f9ba7200700 1 --2- 172.21.15.90:0/2620289033 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9ba810a7e0 0x7f9ba819f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:33:34.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:34.455+0000 7f9ba5e00700 1 -- 172.21.15.90:0/2620289033 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9ba810a7e0 msgr2=0x7f9ba819f880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:33:34.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:34.455+0000 7f9ba5e00700 1 --2- 172.21.15.90:0/2620289033 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9ba810a7e0 0x7f9ba819f880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:34.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:34.455+0000 7f9ba5e00700 1 -- 172.21.15.90:0/2620289033 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9ba81047e0 msgr2=0x7f9ba819afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:33:34.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:34.455+0000 7f9ba5e00700 1 --2- 172.21.15.90:0/2620289033 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9ba81047e0 0x7f9ba819afb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:34.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:34.455+0000 7f9ba5e00700 1 -- 172.21.15.90:0/2620289033 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ba819ffc0 con 0x7f9ba81051a0 2024-04-02T14:33:34.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:34.455+0000 7f9ba7200700 1 --2- 172.21.15.90:0/2620289033 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9ba810a7e0 0x7f9ba819f880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:33:34.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:34.455+0000 7f9ba6800700 1 --2- 172.21.15.90:0/2620289033 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9ba81047e0 0x7f9ba819afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:33:34.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:34.455+0000 7f9ba5e00700 1 --2- 172.21.15.90:0/2620289033 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9ba81051a0 0x7f9ba819b4f0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f9b9400d750 tx=0x7f9b9400da60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:33:34.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:34.455+0000 7f9b9f400700 1 -- 172.21.15.90:0/2620289033 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b94014030 con 0x7f9ba81051a0 2024-04-02T14:33:34.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:34.455+0000 7f9bad400700 1 -- 172.21.15.90:0/2620289033 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ba81a02b0 con 0x7f9ba81051a0 2024-04-02T14:33:34.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:34.455+0000 7f9b9f400700 1 -- 172.21.15.90:0/2620289033 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9b94004500 con 0x7f9ba81051a0 2024-04-02T14:33:34.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:34.455+0000 7f9b9f400700 1 -- 172.21.15.90:0/2620289033 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b94005400 con 0x7f9ba81051a0 2024-04-02T14:33:34.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:34.455+0000 7f9bad400700 1 -- 172.21.15.90:0/2620289033 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ba81a7b30 con 0x7f9ba81051a0 2024-04-02T14:33:34.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:34.456+0000 7f9bad400700 1 -- 172.21.15.90:0/2620289033 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9ba80692b0 con 0x7f9ba81051a0 2024-04-02T14:33:34.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:34.458+0000 7f9b9f400700 1 -- 172.21.15.90:0/2620289033 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f9b94004750 con 0x7f9ba81051a0 2024-04-02T14:33:34.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:34.460+0000 7f9b9f400700 1 --2- 172.21.15.90:0/2620289033 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f9b8c0a3cd0 0x7f9b8c0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:33:34.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:34.460+0000 7f9b9f400700 1 -- 172.21.15.90:0/2620289033 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8248+0+0 (secure 0 0 0) 0x7f9b940d0320 con 0x7f9ba81051a0 2024-04-02T14:33:34.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:34.460+0000 7f9ba6800700 1 --2- 172.21.15.90:0/2620289033 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f9b8c0a3cd0 0x7f9b8c0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:33:34.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:34.461+0000 7f9ba6800700 1 --2- 172.21.15.90:0/2620289033 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f9b8c0a3cd0 0x7f9b8c0a6190 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f9b98011550 tx=0x7f9b98009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:33:34.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:34.462+0000 7f9b9f400700 1 -- 172.21.15.90:0/2620289033 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7f9b9409cb30 con 0x7f9ba81051a0 2024-04-02T14:33:34.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:34.705+0000 7f9bad400700 1 -- 172.21.15.90:0/2620289033 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f9ba802dd30 con 0x7f9ba81051a0 2024-04-02T14:33:34.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:34.706+0000 7f9b9f400700 1 -- 172.21.15.90:0/2620289033 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f9b9409c430 con 0x7f9ba81051a0 2024-04-02T14:33:34.716 INFO:teuthology.orchestra.run.smithi090.stdout:141733920836 2024-04-02T14:33:34.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:34.715+0000 7f9bad400700 1 -- 172.21.15.90:0/2620289033 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f9b8c0a3cd0 msgr2=0x7f9b8c0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:33:34.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:34.715+0000 7f9bad400700 1 --2- 172.21.15.90:0/2620289033 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f9b8c0a3cd0 0x7f9b8c0a6190 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f9b98011550 tx=0x7f9b98009260 comp rx=0 tx=0).stop 2024-04-02T14:33:34.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:34.715+0000 7f9bad400700 1 -- 172.21.15.90:0/2620289033 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9ba81051a0 msgr2=0x7f9ba819b4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:33:34.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:34.715+0000 7f9bad400700 1 --2- 172.21.15.90:0/2620289033 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9ba81051a0 0x7f9ba819b4f0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f9b9400d750 tx=0x7f9b9400da60 comp rx=0 tx=0).stop 2024-04-02T14:33:34.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:34.719+0000 7f9bad400700 1 -- 172.21.15.90:0/2620289033 shutdown_connections 2024-04-02T14:33:34.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:34.719+0000 7f9bad400700 1 --2- 172.21.15.90:0/2620289033 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f9b8c0a3cd0 0x7f9b8c0a6190 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:34.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:34.719+0000 7f9bad400700 1 --2- 172.21.15.90:0/2620289033 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9ba81047e0 0x7f9ba819afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:34.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:34.719+0000 7f9bad400700 1 --2- 172.21.15.90:0/2620289033 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9ba810a7e0 0x7f9ba819f880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:34.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:34.719+0000 7f9bad400700 1 --2- 172.21.15.90:0/2620289033 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9ba81051a0 0x7f9ba819b4f0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:34.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:34.719+0000 7f9bad400700 1 -- 172.21.15.90:0/2620289033 >> 172.21.15.90:0/2620289033 conn(0x7f9ba81002d0 msgr2=0x7f9ba81007f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:33:34.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:34.721+0000 7f9bad400700 1 -- 172.21.15.90:0/2620289033 shutdown_connections 2024-04-02T14:33:34.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:34.722+0000 7f9bad400700 1 -- 172.21.15.90:0/2620289033 wait complete. 2024-04-02T14:33:34.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:34.773+0000 7fc033e00700 1 -- 172.21.15.90:0/2718843853 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc02c10a800 msgr2=0x7fc02c10abe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:33:34.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:34.773+0000 7fc033e00700 1 --2- 172.21.15.90:0/2718843853 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc02c10a800 0x7fc02c10abe0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fc020009990 tx=0x7fc020009ca0 comp rx=0 tx=0).stop 2024-04-02T14:33:34.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:34.773+0000 7fc033e00700 1 -- 172.21.15.90:0/2718843853 shutdown_connections 2024-04-02T14:33:34.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:34.773+0000 7fc033e00700 1 --2- 172.21.15.90:0/2718843853 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc02c1051c0 0x7fc02c1116d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:34.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:34.773+0000 7fc033e00700 1 --2- 172.21.15.90:0/2718843853 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc02c10a800 0x7fc02c10abe0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:34.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:34.773+0000 7fc033e00700 1 --2- 172.21.15.90:0/2718843853 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc02c104800 0x7fc02c104c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:34.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:34.773+0000 7fc033e00700 1 -- 172.21.15.90:0/2718843853 >> 172.21.15.90:0/2718843853 conn(0x7fc02c1002f0 msgr2=0x7fc02c102710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:33:34.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:34.774+0000 7fc033e00700 1 -- 172.21.15.90:0/2718843853 shutdown_connections 2024-04-02T14:33:34.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:34.774+0000 7fc033e00700 1 -- 172.21.15.90:0/2718843853 wait complete. 2024-04-02T14:33:34.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:34.775+0000 7fc033e00700 1 Processor -- start 2024-04-02T14:33:34.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:34.775+0000 7fc033e00700 1 -- start start 2024-04-02T14:33:34.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:34.776+0000 7fc033e00700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc02c104800 0x7fc02c19b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:33:34.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:34.776+0000 7fc033e00700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc02c1051c0 0x7fc02c19b550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:33:34.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:34.776+0000 7fc033e00700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc02c10a800 0x7fc02c19f890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:33:34.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:34.776+0000 7fc033e00700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fc02c114be0 con 0x7fc02c104800 2024-04-02T14:33:34.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:34.776+0000 7fc033e00700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fc02c114a60 con 0x7fc02c1051c0 2024-04-02T14:33:34.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:34.776+0000 7fc033e00700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fc02c114d60 con 0x7fc02c10a800 2024-04-02T14:33:34.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:34.776+0000 7fc030a00700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc02c1051c0 0x7fc02c19b550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:33:34.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:34.776+0000 7fc031e00700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc02c10a800 0x7fc02c19f890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:33:34.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:34.776+0000 7fc030a00700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc02c1051c0 0x7fc02c19b550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:35028/0 (socket says 172.21.15.90:35028) 2024-04-02T14:33:34.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:34.776+0000 7fc030a00700 1 -- 172.21.15.90:0/4221042428 learned_addr learned my addr 172.21.15.90:0/4221042428 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:33:34.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:34.776+0000 7fc030a00700 1 -- 172.21.15.90:0/4221042428 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc02c10a800 msgr2=0x7fc02c19f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:33:34.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:34.776+0000 7fc031400700 1 --2- 172.21.15.90:0/4221042428 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc02c104800 0x7fc02c19b010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:33:34.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:34.776+0000 7fc030a00700 1 --2- 172.21.15.90:0/4221042428 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc02c10a800 0x7fc02c19f890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:34.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:34.776+0000 7fc030a00700 1 -- 172.21.15.90:0/4221042428 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc02c104800 msgr2=0x7fc02c19b010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:33:34.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:34.776+0000 7fc030a00700 1 --2- 172.21.15.90:0/4221042428 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc02c104800 0x7fc02c19b010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:34.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:34.776+0000 7fc030a00700 1 -- 172.21.15.90:0/4221042428 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc02c19ffd0 con 0x7fc02c1051c0 2024-04-02T14:33:34.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:34.776+0000 7fc031e00700 1 --2- 172.21.15.90:0/4221042428 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc02c10a800 0x7fc02c19f890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:33:34.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:34.777+0000 7fc030a00700 1 --2- 172.21.15.90:0/4221042428 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc02c1051c0 0x7fc02c19b550 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fc01800d980 tx=0x7fc01800dc90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:33:34.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:34.777+0000 7fc02a000700 1 -- 172.21.15.90:0/4221042428 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc018016070 con 0x7fc02c1051c0 2024-04-02T14:33:34.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:34.777+0000 7fc02a000700 1 -- 172.21.15.90:0/4221042428 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc018004d10 con 0x7fc02c1051c0 2024-04-02T14:33:34.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:34.777+0000 7fc033e00700 1 -- 172.21.15.90:0/4221042428 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc02c1a02c0 con 0x7fc02c1051c0 2024-04-02T14:33:34.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:34.777+0000 7fc02a000700 1 -- 172.21.15.90:0/4221042428 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc01800b750 con 0x7fc02c1051c0 2024-04-02T14:33:34.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:34.777+0000 7fc033e00700 1 -- 172.21.15.90:0/4221042428 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc02c1a7b40 con 0x7fc02c1051c0 2024-04-02T14:33:34.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:34.779+0000 7fc033e00700 1 -- 172.21.15.90:0/4221042428 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc02c0692b0 con 0x7fc02c1051c0 2024-04-02T14:33:34.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:34.781+0000 7fc02a000700 1 -- 172.21.15.90:0/4221042428 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fc018004750 con 0x7fc02c1051c0 2024-04-02T14:33:34.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:34.782+0000 7fc02a000700 1 --2- 172.21.15.90:0/4221042428 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fc0140a3d50 0x7fc0140a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:33:34.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:34.782+0000 7fc02a000700 1 -- 172.21.15.90:0/4221042428 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8248+0+0 (secure 0 0 0) 0x7fc0180d0370 con 0x7fc02c1051c0 2024-04-02T14:33:34.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:34.782+0000 7fc031400700 1 --2- 172.21.15.90:0/4221042428 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fc0140a3d50 0x7fc0140a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:33:34.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:34.783+0000 7fc031400700 1 --2- 172.21.15.90:0/4221042428 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fc0140a3d50 0x7fc0140a6210 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fc020005f50 tx=0x7fc020005ea0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:33:34.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:34.786+0000 7fc02a000700 1 -- 172.21.15.90:0/4221042428 <== mon.2 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7fc01801f020 con 0x7fc02c1051c0 2024-04-02T14:33:35.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:35.017+0000 7fc033e00700 1 -- 172.21.15.90:0/4221042428 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7fc02c19c320 con 0x7fc02c1051c0 2024-04-02T14:33:35.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:35.017+0000 7fc02a000700 1 -- 172.21.15.90:0/4221042428 <== mon.2 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fc01809cb80 con 0x7fc02c1051c0 2024-04-02T14:33:35.020 INFO:teuthology.orchestra.run.smithi090.stdout:223338299443 2024-04-02T14:33:35.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:35.020+0000 7fc033e00700 1 -- 172.21.15.90:0/4221042428 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fc0140a3d50 msgr2=0x7fc0140a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:33:35.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:35.020+0000 7fc033e00700 1 --2- 172.21.15.90:0/4221042428 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fc0140a3d50 0x7fc0140a6210 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fc020005f50 tx=0x7fc020005ea0 comp rx=0 tx=0).stop 2024-04-02T14:33:35.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:35.020+0000 7fc033e00700 1 -- 172.21.15.90:0/4221042428 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc02c1051c0 msgr2=0x7fc02c19b550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:33:35.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:35.020+0000 7fc033e00700 1 --2- 172.21.15.90:0/4221042428 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc02c1051c0 0x7fc02c19b550 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fc01800d980 tx=0x7fc01800dc90 comp rx=0 tx=0).stop 2024-04-02T14:33:35.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:35.020+0000 7fc033e00700 1 -- 172.21.15.90:0/4221042428 shutdown_connections 2024-04-02T14:33:35.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:35.020+0000 7fc033e00700 1 --2- 172.21.15.90:0/4221042428 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fc0140a3d50 0x7fc0140a6210 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:35.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:35.020+0000 7fc033e00700 1 --2- 172.21.15.90:0/4221042428 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc02c104800 0x7fc02c19b010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:35.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:35.020+0000 7fc033e00700 1 --2- 172.21.15.90:0/4221042428 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc02c10a800 0x7fc02c19f890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:35.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:35.020+0000 7fc033e00700 1 --2- 172.21.15.90:0/4221042428 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc02c1051c0 0x7fc02c19b550 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:35.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:35.021+0000 7fc033e00700 1 -- 172.21.15.90:0/4221042428 >> 172.21.15.90:0/4221042428 conn(0x7fc02c1002f0 msgr2=0x7fc02c1008d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:33:35.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:35.021+0000 7fc033e00700 1 -- 172.21.15.90:0/4221042428 shutdown_connections 2024-04-02T14:33:35.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:35.021+0000 7fc033e00700 1 -- 172.21.15.90:0/4221042428 wait complete. 2024-04-02T14:33:35.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:33:34 smithi177 ceph-mon[36431]: from='client.? 172.21.15.90:0/2620289033' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-02T14:33:35.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:33:34 smithi090 ceph-mon[31674]: from='client.? 172.21.15.90:0/2620289033' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-02T14:33:35.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:33:34 smithi092 ceph-mon[36497]: from='client.? 172.21.15.90:0/2620289033' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-02T14:33:35.412 INFO:tasks.cephadm.ceph_manager.ceph:need seq 197568495666 got 197568495670 for osd.6 2024-04-02T14:33:35.412 DEBUG:teuthology.parallel:result is None 2024-04-02T14:33:36.075 INFO:tasks.cephadm.ceph_manager.ceph:need seq 296352743456 got 296352743462 for osd.9 2024-04-02T14:33:36.076 DEBUG:teuthology.parallel:result is None 2024-04-02T14:33:36.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:33:35 smithi177 ceph-mon[36431]: pgmap v350: 97 pgs: 97 active+clean; 643 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 op/s 2024-04-02T14:33:36.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:33:35 smithi177 ceph-mon[36431]: from='client.? 172.21.15.90:0/4221042428' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-02T14:33:36.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:33:35 smithi090 ceph-mon[31674]: pgmap v350: 97 pgs: 97 active+clean; 643 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 op/s 2024-04-02T14:33:36.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:33:35 smithi090 ceph-mon[31674]: from='client.? 172.21.15.90:0/4221042428' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-02T14:33:36.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:33:35 smithi092 ceph-mon[36497]: pgmap v350: 97 pgs: 97 active+clean; 643 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 op/s 2024-04-02T14:33:36.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:33:35 smithi092 ceph-mon[36497]: from='client.? 172.21.15.90:0/4221042428' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-02T14:33:36.907 INFO:tasks.cephadm.ceph_manager.ceph:need seq 141733920829 got 141733920836 for osd.4 2024-04-02T14:33:36.907 DEBUG:teuthology.parallel:result is None 2024-04-02T14:33:37.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:33:36 smithi177 ceph-mon[36431]: mgrmap e18: x(active, since 9m), standbys: z, y 2024-04-02T14:33:37.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:33:36 smithi090 ceph-mon[31674]: mgrmap e18: x(active, since 9m), standbys: z, y 2024-04-02T14:33:37.360 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:33:36 smithi092 ceph-mon[36497]: mgrmap e18: x(active, since 9m), standbys: z, y 2024-04-02T14:33:37.811 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T14:33:37.816 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T14:33:37.857 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574928 got 55834574932 for osd.1 2024-04-02T14:33:37.857 DEBUG:teuthology.parallel:result is None 2024-04-02T14:33:38.129 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:33:37 smithi090 ceph-mon[31674]: pgmap v351: 97 pgs: 97 active+clean; 644 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 85 B/s wr, 9 op/s 2024-04-02T14:33:38.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:33:37 smithi177 ceph-mon[36431]: pgmap v351: 97 pgs: 97 active+clean; 644 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 85 B/s wr, 9 op/s 2024-04-02T14:33:38.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:33:37 smithi092 ceph-mon[36497]: pgmap v351: 97 pgs: 97 active+clean; 644 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 85 B/s wr, 9 op/s 2024-04-02T14:33:39.065 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:33:38 smithi090 ceph-mon[31674]: from='client.? 172.21.15.90:0/2734752300' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-02T14:33:39.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:38.492+0000 7f24bd200700 1 -- 172.21.15.90:0/741341475 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f24b810a7f0 msgr2=0x7f24b810abd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:33:39.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:38.492+0000 7f24bd200700 1 --2- 172.21.15.90:0/741341475 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f24b810a7f0 0x7f24b810abd0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f24a8009990 tx=0x7f24a8009ca0 comp rx=0 tx=0).stop 2024-04-02T14:33:39.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:38.492+0000 7f24bd200700 1 -- 172.21.15.90:0/741341475 shutdown_connections 2024-04-02T14:33:39.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:38.492+0000 7f24bd200700 1 --2- 172.21.15.90:0/741341475 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f24b810a7f0 0x7f24b810abd0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:39.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:38.492+0000 7f24bd200700 1 --2- 172.21.15.90:0/741341475 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f24b81051b0 0x7f24b81116c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:39.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:38.492+0000 7f24bd200700 1 --2- 172.21.15.90:0/741341475 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f24b81047f0 0x7f24b8104c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:39.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:38.492+0000 7f24bd200700 1 -- 172.21.15.90:0/741341475 >> 172.21.15.90:0/741341475 conn(0x7f24b81002c0 msgr2=0x7f24b81026e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:33:39.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:38.492+0000 7f24bd200700 1 -- 172.21.15.90:0/741341475 shutdown_connections 2024-04-02T14:33:39.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:38.492+0000 7f24bd200700 1 -- 172.21.15.90:0/741341475 wait complete. 2024-04-02T14:33:39.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:38.494+0000 7f24bd200700 1 Processor -- start 2024-04-02T14:33:39.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:38.494+0000 7f24bd200700 1 -- start start 2024-04-02T14:33:39.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:38.494+0000 7f24bd200700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f24b81047f0 0x7f24b819af90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:33:39.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:38.494+0000 7f24bd200700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f24b81051b0 0x7f24b819b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:33:39.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:38.494+0000 7f24bd200700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f24b810a7f0 0x7f24b819f810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:33:39.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:38.494+0000 7f24bd200700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f24b8114b20 con 0x7f24b81051b0 2024-04-02T14:33:39.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:38.494+0000 7f24bd200700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f24b81149a0 con 0x7f24b810a7f0 2024-04-02T14:33:39.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:38.494+0000 7f24bd200700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f24b8114ca0 con 0x7f24b81047f0 2024-04-02T14:33:39.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:38.495+0000 7f24b5400700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f24b81051b0 0x7f24b819b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:33:39.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:38.495+0000 7f24b5400700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f24b81051b0 0x7f24b819b4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34952/0 (socket says 172.21.15.90:34952) 2024-04-02T14:33:39.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:38.495+0000 7f24b5e00700 1 --2- 172.21.15.90:0/2734752300 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f24b81047f0 0x7f24b819af90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:33:39.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:38.495+0000 7f24b5400700 1 -- 172.21.15.90:0/2734752300 learned_addr learned my addr 172.21.15.90:0/2734752300 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:33:39.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:38.495+0000 7f24b6800700 1 --2- 172.21.15.90:0/2734752300 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f24b810a7f0 0x7f24b819f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:33:39.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:38.495+0000 7f24b5e00700 1 -- 172.21.15.90:0/2734752300 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f24b810a7f0 msgr2=0x7f24b819f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:33:39.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:38.495+0000 7f24b5e00700 1 --2- 172.21.15.90:0/2734752300 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f24b810a7f0 0x7f24b819f810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:39.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:38.495+0000 7f24b5e00700 1 -- 172.21.15.90:0/2734752300 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f24b81051b0 msgr2=0x7f24b819b4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:33:39.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:38.495+0000 7f24b5e00700 1 --2- 172.21.15.90:0/2734752300 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f24b81051b0 0x7f24b819b4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:39.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:38.495+0000 7f24b5e00700 1 -- 172.21.15.90:0/2734752300 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24b819ff50 con 0x7f24b81047f0 2024-04-02T14:33:39.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:38.495+0000 7f24b5400700 1 --2- 172.21.15.90:0/2734752300 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f24b81051b0 0x7f24b819b4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:33:39.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:38.495+0000 7f24b5e00700 1 --2- 172.21.15.90:0/2734752300 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f24b81047f0 0x7f24b819af90 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f24a8006010 tx=0x7f24a800bb10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:33:39.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:38.496+0000 7f24aea00700 1 -- 172.21.15.90:0/2734752300 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24a8026070 con 0x7f24b81047f0 2024-04-02T14:33:39.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:38.496+0000 7f24aea00700 1 -- 172.21.15.90:0/2734752300 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f24a8003d70 con 0x7f24b81047f0 2024-04-02T14:33:39.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:38.496+0000 7f24bd200700 1 -- 172.21.15.90:0/2734752300 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f24b81a01e0 con 0x7f24b81047f0 2024-04-02T14:33:39.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:38.496+0000 7f24aea00700 1 -- 172.21.15.90:0/2734752300 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24a8021410 con 0x7f24b81047f0 2024-04-02T14:33:39.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:38.496+0000 7f24bd200700 1 -- 172.21.15.90:0/2734752300 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24b81a7ac0 con 0x7f24b81047f0 2024-04-02T14:33:39.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:38.497+0000 7f24bd200700 1 -- 172.21.15.90:0/2734752300 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f24b80692b0 con 0x7f24b81047f0 2024-04-02T14:33:39.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:38.499+0000 7f24aea00700 1 -- 172.21.15.90:0/2734752300 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f24a8029070 con 0x7f24b81047f0 2024-04-02T14:33:39.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:38.500+0000 7f24aea00700 1 --2- 172.21.15.90:0/2734752300 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f249c0a3cc0 0x7f249c0a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:33:39.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:38.500+0000 7f24aea00700 1 -- 172.21.15.90:0/2734752300 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8248+0+0 (secure 0 0 0) 0x7f24a80d1210 con 0x7f24b81047f0 2024-04-02T14:33:39.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:38.500+0000 7f24b5400700 1 --2- 172.21.15.90:0/2734752300 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f249c0a3cc0 0x7f249c0a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:33:39.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:38.501+0000 7f24b5400700 1 --2- 172.21.15.90:0/2734752300 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f249c0a3cc0 0x7f249c0a6180 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f24b8109df0 tx=0x7f24a4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:33:39.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:38.504+0000 7f24aea00700 1 -- 172.21.15.90:0/2734752300 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7f24a805df20 con 0x7f24b81047f0 2024-04-02T14:33:39.189 INFO:teuthology.orchestra.run.smithi090.stdout:77309411408 2024-04-02T14:33:39.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:38.733+0000 7f24bd200700 1 -- 172.21.15.90:0/2734752300 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f24b802dd30 con 0x7f24b81047f0 2024-04-02T14:33:39.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:38.734+0000 7f24aea00700 1 -- 172.21.15.90:0/2734752300 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f24a8013070 con 0x7f24b81047f0 2024-04-02T14:33:39.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:38.737+0000 7f24bd200700 1 -- 172.21.15.90:0/2734752300 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f249c0a3cc0 msgr2=0x7f249c0a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:33:39.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:38.737+0000 7f24bd200700 1 --2- 172.21.15.90:0/2734752300 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f249c0a3cc0 0x7f249c0a6180 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f24b8109df0 tx=0x7f24a4009260 comp rx=0 tx=0).stop 2024-04-02T14:33:39.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:38.737+0000 7f24bd200700 1 -- 172.21.15.90:0/2734752300 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f24b81047f0 msgr2=0x7f24b819af90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:33:39.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:38.737+0000 7f24bd200700 1 --2- 172.21.15.90:0/2734752300 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f24b81047f0 0x7f24b819af90 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f24a8006010 tx=0x7f24a800bb10 comp rx=0 tx=0).stop 2024-04-02T14:33:39.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:38.737+0000 7f24bd200700 1 -- 172.21.15.90:0/2734752300 shutdown_connections 2024-04-02T14:33:39.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:38.737+0000 7f24bd200700 1 --2- 172.21.15.90:0/2734752300 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f249c0a3cc0 0x7f249c0a6180 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:39.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:38.737+0000 7f24bd200700 1 --2- 172.21.15.90:0/2734752300 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f24b810a7f0 0x7f24b819f810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:39.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:38.737+0000 7f24bd200700 1 --2- 172.21.15.90:0/2734752300 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f24b81047f0 0x7f24b819af90 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:39.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:38.737+0000 7f24bd200700 1 --2- 172.21.15.90:0/2734752300 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f24b81051b0 0x7f24b819b4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:39.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:38.737+0000 7f24bd200700 1 -- 172.21.15.90:0/2734752300 >> 172.21.15.90:0/2734752300 conn(0x7f24b81002c0 msgr2=0x7f24b81008a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:33:39.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:38.737+0000 7f24bd200700 1 -- 172.21.15.90:0/2734752300 shutdown_connections 2024-04-02T14:33:39.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:38.737+0000 7f24bd200700 1 -- 172.21.15.90:0/2734752300 wait complete. 2024-04-02T14:33:39.190 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:33:38 smithi177 ceph-mon[36431]: from='client.? 172.21.15.90:0/2734752300' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-02T14:33:39.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:33:38 smithi092 ceph-mon[36497]: from='client.? 172.21.15.90:0/2734752300' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-02T14:33:39.755 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182466 got 107374182471 for osd.3 2024-04-02T14:33:39.755 DEBUG:teuthology.parallel:result is None 2024-04-02T14:33:40.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:33:39 smithi092 ceph-mon[36497]: pgmap v352: 97 pgs: 97 active+clean; 644 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 255 B/s wr, 9 op/s 2024-04-02T14:33:40.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:33:39 smithi177 ceph-mon[36431]: pgmap v352: 97 pgs: 97 active+clean; 644 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 255 B/s wr, 9 op/s 2024-04-02T14:33:40.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:33:39 smithi090 ceph-mon[31674]: pgmap v352: 97 pgs: 97 active+clean; 644 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 255 B/s wr, 9 op/s 2024-04-02T14:33:40.424 INFO:tasks.cephadm.ceph_manager.ceph:need seq 223338299436 got 223338299443 for osd.7 2024-04-02T14:33:40.424 DEBUG:teuthology.parallel:result is None 2024-04-02T14:33:40.805 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411402 got 77309411408 for osd.2 2024-04-02T14:33:40.805 DEBUG:teuthology.parallel:result is None 2024-04-02T14:33:41.452 INFO:tasks.cephadm.ceph_manager.ceph:need seq 261993005095 got 261993005101 for osd.8 2024-04-02T14:33:41.452 DEBUG:teuthology.parallel:result is None 2024-04-02T14:33:42.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:33:41 smithi092 ceph-mon[36497]: pgmap v353: 97 pgs: 97 active+clean; 644 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 597 B/s wr, 11 op/s 2024-04-02T14:33:42.436 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:33:41 smithi177 ceph-mon[36431]: pgmap v353: 97 pgs: 97 active+clean; 644 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 597 B/s wr, 11 op/s 2024-04-02T14:33:42.439 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:33:41 smithi090 ceph-mon[31674]: pgmap v353: 97 pgs: 97 active+clean; 644 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 597 B/s wr, 11 op/s 2024-04-02T14:33:43.012 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T14:33:43.012 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T14:33:43.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:33:42 smithi092 ceph-mon[36497]: pgmap v354: 97 pgs: 97 active+clean; 644 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 597 B/s wr, 10 op/s 2024-04-02T14:33:43.400 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:33:42 smithi090 ceph-mon[31674]: pgmap v354: 97 pgs: 97 active+clean; 644 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 597 B/s wr, 10 op/s 2024-04-02T14:33:43.436 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:33:42 smithi177 ceph-mon[36431]: pgmap v354: 97 pgs: 97 active+clean; 644 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 597 B/s wr, 10 op/s 2024-04-02T14:33:44.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:44.354+0000 7f3129200700 1 -- 172.21.15.90:0/3702020400 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f31241051b0 msgr2=0x7f31241116c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:33:44.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:44.354+0000 7f3129200700 1 --2- 172.21.15.90:0/3702020400 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f31241051b0 0x7f31241116c0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f311c00b0a0 tx=0x7f311c00b3b0 comp rx=0 tx=0).stop 2024-04-02T14:33:44.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:44.354+0000 7f3129200700 1 -- 172.21.15.90:0/3702020400 shutdown_connections 2024-04-02T14:33:44.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:44.354+0000 7f3129200700 1 --2- 172.21.15.90:0/3702020400 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f31241051b0 0x7f31241116c0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:44.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:44.354+0000 7f3129200700 1 --2- 172.21.15.90:0/3702020400 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f312410a7f0 0x7f312410abd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:44.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:44.354+0000 7f3129200700 1 --2- 172.21.15.90:0/3702020400 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f31241047f0 0x7f3124104c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:44.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:44.354+0000 7f3129200700 1 -- 172.21.15.90:0/3702020400 >> 172.21.15.90:0/3702020400 conn(0x7f31241002c0 msgr2=0x7f31241026e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:33:44.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:44.354+0000 7f3129200700 1 -- 172.21.15.90:0/3702020400 shutdown_connections 2024-04-02T14:33:44.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:44.354+0000 7f3129200700 1 -- 172.21.15.90:0/3702020400 wait complete. 2024-04-02T14:33:44.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:44.355+0000 7f3129200700 1 Processor -- start 2024-04-02T14:33:44.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:44.355+0000 7f3129200700 1 -- start start 2024-04-02T14:33:44.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:44.356+0000 7f3129200700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f31241047f0 0x7f312419af50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:33:44.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:44.356+0000 7f3129200700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f31241051b0 0x7f312419b490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:33:44.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:44.356+0000 7f3129200700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f312410a7f0 0x7f312419f820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:33:44.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:44.356+0000 7f3129200700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f3124114b80 con 0x7f31241047f0 2024-04-02T14:33:44.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:44.356+0000 7f3129200700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f3124114a00 con 0x7f31241051b0 2024-04-02T14:33:44.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:44.356+0000 7f3129200700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f3124114d00 con 0x7f312410a7f0 2024-04-02T14:33:44.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:44.356+0000 7f3121e00700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f31241047f0 0x7f312419af50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:33:44.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:44.356+0000 7f3121e00700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f31241047f0 0x7f312419af50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:51260/0 (socket says 172.21.15.90:51260) 2024-04-02T14:33:44.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:44.356+0000 7f3121400700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f31241051b0 0x7f312419b490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:33:44.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:44.356+0000 7f3121e00700 1 -- 172.21.15.90:0/1820571826 learned_addr learned my addr 172.21.15.90:0/1820571826 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:33:44.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:44.356+0000 7f3122800700 1 --2- 172.21.15.90:0/1820571826 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f312410a7f0 0x7f312419f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:33:44.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:44.357+0000 7f3121400700 1 -- 172.21.15.90:0/1820571826 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f312410a7f0 msgr2=0x7f312419f820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:33:44.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:44.357+0000 7f3121400700 1 --2- 172.21.15.90:0/1820571826 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f312410a7f0 0x7f312419f820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:44.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:44.357+0000 7f3121400700 1 -- 172.21.15.90:0/1820571826 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f31241047f0 msgr2=0x7f312419af50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:33:44.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:44.357+0000 7f3121400700 1 --2- 172.21.15.90:0/1820571826 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f31241047f0 0x7f312419af50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:44.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:44.357+0000 7f3121400700 1 -- 172.21.15.90:0/1820571826 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f312419ff60 con 0x7f31241051b0 2024-04-02T14:33:44.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:44.357+0000 7f3121400700 1 --2- 172.21.15.90:0/1820571826 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f31241051b0 0x7f312419b490 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f311000e9f0 tx=0x7f311000ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:33:44.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:44.357+0000 7f311aa00700 1 -- 172.21.15.90:0/1820571826 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f311000c9c0 con 0x7f31241051b0 2024-04-02T14:33:44.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:44.357+0000 7f3129200700 1 -- 172.21.15.90:0/1820571826 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f31241a0250 con 0x7f31241051b0 2024-04-02T14:33:44.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:44.357+0000 7f311aa00700 1 -- 172.21.15.90:0/1820571826 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f311000cb60 con 0x7f31241051b0 2024-04-02T14:33:44.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:44.357+0000 7f311aa00700 1 -- 172.21.15.90:0/1820571826 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3110018680 con 0x7f31241051b0 2024-04-02T14:33:44.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:44.357+0000 7f3129200700 1 -- 172.21.15.90:0/1820571826 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31241a7ad0 con 0x7f31241051b0 2024-04-02T14:33:44.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:44.358+0000 7f3129200700 1 -- 172.21.15.90:0/1820571826 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f31240692b0 con 0x7f31241051b0 2024-04-02T14:33:44.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:44.360+0000 7f311aa00700 1 -- 172.21.15.90:0/1820571826 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f3110010430 con 0x7f31241051b0 2024-04-02T14:33:44.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:44.362+0000 7f311aa00700 1 --2- 172.21.15.90:0/1820571826 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f31080a3c40 0x7f31080a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:33:44.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:44.362+0000 7f311aa00700 1 -- 172.21.15.90:0/1820571826 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8248+0+0 (secure 0 0 0) 0x7f31100d0580 con 0x7f31241051b0 2024-04-02T14:33:44.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:44.362+0000 7f3121e00700 1 --2- 172.21.15.90:0/1820571826 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f31080a3c40 0x7f31080a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:33:44.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:44.363+0000 7f3121e00700 1 --2- 172.21.15.90:0/1820571826 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f31080a3c40 0x7f31080a6100 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f3114007c10 tx=0x7f3114007480 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:33:44.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:44.366+0000 7f311aa00700 1 -- 172.21.15.90:0/1820571826 <== mon.2 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7f311009cef0 con 0x7f31241051b0 2024-04-02T14:33:44.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:44.603+0000 7f3129200700 1 -- 172.21.15.90:0/1820571826 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f31241a7d00 con 0x7f31241051b0 2024-04-02T14:33:44.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:44.604+0000 7f311aa00700 1 -- 172.21.15.90:0/1820571826 <== mon.2 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f311009c7f0 con 0x7f31241051b0 2024-04-02T14:33:44.606 INFO:teuthology.orchestra.run.smithi090.stdout:34359738459 2024-04-02T14:33:44.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:44.606+0000 7f3129200700 1 -- 172.21.15.90:0/1820571826 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f31080a3c40 msgr2=0x7f31080a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:33:44.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:44.606+0000 7f3129200700 1 --2- 172.21.15.90:0/1820571826 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f31080a3c40 0x7f31080a6100 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f3114007c10 tx=0x7f3114007480 comp rx=0 tx=0).stop 2024-04-02T14:33:44.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:44.607+0000 7f3129200700 1 -- 172.21.15.90:0/1820571826 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f31241051b0 msgr2=0x7f312419b490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:33:44.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:44.607+0000 7f3129200700 1 --2- 172.21.15.90:0/1820571826 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f31241051b0 0x7f312419b490 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f311000e9f0 tx=0x7f311000ed00 comp rx=0 tx=0).stop 2024-04-02T14:33:44.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:44.607+0000 7f3129200700 1 -- 172.21.15.90:0/1820571826 shutdown_connections 2024-04-02T14:33:44.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:44.607+0000 7f3129200700 1 --2- 172.21.15.90:0/1820571826 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f31080a3c40 0x7f31080a6100 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:44.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:44.607+0000 7f3129200700 1 --2- 172.21.15.90:0/1820571826 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f31241047f0 0x7f312419af50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:44.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:44.607+0000 7f3129200700 1 --2- 172.21.15.90:0/1820571826 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f312410a7f0 0x7f312419f820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:44.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:44.607+0000 7f3129200700 1 --2- 172.21.15.90:0/1820571826 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f31241051b0 0x7f312419b490 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:44.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:44.607+0000 7f3129200700 1 -- 172.21.15.90:0/1820571826 >> 172.21.15.90:0/1820571826 conn(0x7f31241002c0 msgr2=0x7f31241007a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:33:44.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:44.607+0000 7f3129200700 1 -- 172.21.15.90:0/1820571826 shutdown_connections 2024-04-02T14:33:44.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:44.607+0000 7f3129200700 1 -- 172.21.15.90:0/1820571826 wait complete. 2024-04-02T14:33:44.827 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:33:44 smithi090 ceph-mon[31674]: from='client.? 172.21.15.90:0/1820571826' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-02T14:33:44.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:33:44 smithi177 ceph-mon[36431]: from='client.? 172.21.15.90:0/1820571826' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-02T14:33:45.110 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:33:44 smithi092 ceph-mon[36497]: from='client.? 172.21.15.90:0/1820571826' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-02T14:33:45.871 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:33:45 smithi090 ceph-mon[31674]: pgmap v355: 97 pgs: 97 active+clean; 644 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 597 B/s wr, 11 op/s 2024-04-02T14:33:45.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:33:45 smithi177 ceph-mon[36431]: pgmap v355: 97 pgs: 97 active+clean; 644 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 597 B/s wr, 11 op/s 2024-04-02T14:33:46.110 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:33:45 smithi092 ceph-mon[36497]: pgmap v355: 97 pgs: 97 active+clean; 644 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 597 B/s wr, 11 op/s 2024-04-02T14:33:46.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:33:46 smithi177 ceph-mon[36431]: from='client.? 172.21.15.90:0/1883452770' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-02T14:33:46.938 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:33:46 smithi090 ceph-mon[31674]: from='client.? 172.21.15.90:0/1883452770' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-02T14:33:47.110 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:33:46 smithi092 ceph-mon[36497]: from='client.? 172.21.15.90:0/1883452770' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-02T14:33:47.534 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738453 got 34359738459 for osd.0 2024-04-02T14:33:47.534 DEBUG:teuthology.parallel:result is None 2024-04-02T14:33:47.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:46.373+0000 7feebf000700 1 -- 172.21.15.90:0/2804220105 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7feeb810a7e0 msgr2=0x7feeb810abc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:33:47.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:46.373+0000 7feebf000700 1 --2- 172.21.15.90:0/2804220105 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7feeb810a7e0 0x7feeb810abc0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7feea8009990 tx=0x7feea8009ca0 comp rx=0 tx=0).stop 2024-04-02T14:33:47.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:46.373+0000 7feebf000700 1 -- 172.21.15.90:0/2804220105 shutdown_connections 2024-04-02T14:33:47.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:46.373+0000 7feebf000700 1 --2- 172.21.15.90:0/2804220105 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7feeb810a7e0 0x7feeb810abc0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:47.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:46.373+0000 7feebf000700 1 --2- 172.21.15.90:0/2804220105 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7feeb81051a0 0x7feeb81116b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:47.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:46.373+0000 7feebf000700 1 --2- 172.21.15.90:0/2804220105 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7feeb81047e0 0x7feeb8104c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:47.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:46.373+0000 7feebf000700 1 -- 172.21.15.90:0/2804220105 >> 172.21.15.90:0/2804220105 conn(0x7feeb81002f0 msgr2=0x7feeb8102710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:33:47.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:46.373+0000 7feebf000700 1 -- 172.21.15.90:0/2804220105 shutdown_connections 2024-04-02T14:33:47.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:46.373+0000 7feebf000700 1 -- 172.21.15.90:0/2804220105 wait complete. 2024-04-02T14:33:47.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:46.375+0000 7feebf000700 1 Processor -- start 2024-04-02T14:33:47.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:46.375+0000 7feebf000700 1 -- start start 2024-04-02T14:33:47.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:46.375+0000 7feebf000700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7feeb81047e0 0x7feeb819afb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:33:47.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:46.375+0000 7feebf000700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7feeb81051a0 0x7feeb819b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:33:47.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:46.375+0000 7feebf000700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7feeb810a7e0 0x7feeb819f880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:33:47.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:46.375+0000 7feebf000700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7feeb8114b80 con 0x7feeb81051a0 2024-04-02T14:33:47.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:46.375+0000 7feebf000700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7feeb8114a00 con 0x7feeb81047e0 2024-04-02T14:33:47.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:46.375+0000 7feebf000700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7feeb8114d00 con 0x7feeb810a7e0 2024-04-02T14:33:47.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:46.376+0000 7feeb7400700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7feeb81051a0 0x7feeb819b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:33:47.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:46.376+0000 7feeb7400700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7feeb81051a0 0x7feeb819b4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:51284/0 (socket says 172.21.15.90:51284) 2024-04-02T14:33:47.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:46.376+0000 7feebd000700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7feeb810a7e0 0x7feeb819f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:33:47.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:46.376+0000 7feeb7400700 1 -- 172.21.15.90:0/1883452770 learned_addr learned my addr 172.21.15.90:0/1883452770 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:33:47.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:46.376+0000 7feeb7e00700 1 --2- 172.21.15.90:0/1883452770 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7feeb81047e0 0x7feeb819afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:33:47.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:46.376+0000 7feeb7400700 1 -- 172.21.15.90:0/1883452770 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7feeb810a7e0 msgr2=0x7feeb819f880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:33:47.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:46.376+0000 7feeb7400700 1 --2- 172.21.15.90:0/1883452770 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7feeb810a7e0 0x7feeb819f880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:47.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:46.376+0000 7feeb7400700 1 -- 172.21.15.90:0/1883452770 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7feeb81047e0 msgr2=0x7feeb819afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:33:47.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:46.376+0000 7feeb7400700 1 --2- 172.21.15.90:0/1883452770 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7feeb81047e0 0x7feeb819afb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:47.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:46.376+0000 7feeb7400700 1 -- 172.21.15.90:0/1883452770 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feeb819ffc0 con 0x7feeb81051a0 2024-04-02T14:33:47.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:46.376+0000 7feeb7400700 1 --2- 172.21.15.90:0/1883452770 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7feeb81051a0 0x7feeb819b4f0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7feea400e9a0 tx=0x7feea400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:33:47.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:46.377+0000 7feeb4c00700 1 -- 172.21.15.90:0/1883452770 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feea400ca10 con 0x7feeb81051a0 2024-04-02T14:33:47.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:46.377+0000 7feeb4c00700 1 -- 172.21.15.90:0/1883452770 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feea400cbb0 con 0x7feeb81051a0 2024-04-02T14:33:47.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:46.377+0000 7feeb4c00700 1 -- 172.21.15.90:0/1883452770 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feea4018680 con 0x7feeb81051a0 2024-04-02T14:33:47.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:46.377+0000 7feebf000700 1 -- 172.21.15.90:0/1883452770 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feeb81a02b0 con 0x7feeb81051a0 2024-04-02T14:33:47.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:46.377+0000 7feebf000700 1 -- 172.21.15.90:0/1883452770 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feeb81a7b30 con 0x7feeb81051a0 2024-04-02T14:33:47.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:46.378+0000 7feebf000700 1 -- 172.21.15.90:0/1883452770 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feeb80692b0 con 0x7feeb81051a0 2024-04-02T14:33:47.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:46.380+0000 7feeb4c00700 1 -- 172.21.15.90:0/1883452770 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7feea4010430 con 0x7feeb81051a0 2024-04-02T14:33:47.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:46.381+0000 7feeb4c00700 1 --2- 172.21.15.90:0/1883452770 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fee9c0a3cc0 0x7fee9c0a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:33:47.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:46.381+0000 7feeb4c00700 1 -- 172.21.15.90:0/1883452770 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8248+0+0 (secure 0 0 0) 0x7feea40d05f0 con 0x7feeb81051a0 2024-04-02T14:33:47.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:46.382+0000 7feeb7e00700 1 --2- 172.21.15.90:0/1883452770 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fee9c0a3cc0 0x7fee9c0a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:33:47.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:46.382+0000 7feeb7e00700 1 --2- 172.21.15.90:0/1883452770 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fee9c0a3cc0 0x7fee9c0a6180 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7feea800b2e0 tx=0x7feea8005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:33:47.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:46.386+0000 7feeb4c00700 1 -- 172.21.15.90:0/1883452770 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7feea409cf60 con 0x7feeb81051a0 2024-04-02T14:33:47.726 INFO:teuthology.orchestra.run.smithi090.stdout:322122547237 2024-04-02T14:33:47.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:46.617+0000 7feebf000700 1 -- 172.21.15.90:0/1883452770 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 10} v 0) -- 0x7feeb802dd30 con 0x7feeb81051a0 2024-04-02T14:33:47.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:46.618+0000 7feeb4c00700 1 -- 172.21.15.90:0/1883452770 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 10}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7feea409c860 con 0x7feeb81051a0 2024-04-02T14:33:47.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:46.621+0000 7feebf000700 1 -- 172.21.15.90:0/1883452770 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fee9c0a3cc0 msgr2=0x7fee9c0a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:33:47.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:46.621+0000 7feebf000700 1 --2- 172.21.15.90:0/1883452770 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fee9c0a3cc0 0x7fee9c0a6180 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7feea800b2e0 tx=0x7feea8005fd0 comp rx=0 tx=0).stop 2024-04-02T14:33:47.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:46.621+0000 7feebf000700 1 -- 172.21.15.90:0/1883452770 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7feeb81051a0 msgr2=0x7feeb819b4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:33:47.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:46.621+0000 7feebf000700 1 --2- 172.21.15.90:0/1883452770 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7feeb81051a0 0x7feeb819b4f0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7feea400e9a0 tx=0x7feea400ecb0 comp rx=0 tx=0).stop 2024-04-02T14:33:47.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:46.621+0000 7feebf000700 1 -- 172.21.15.90:0/1883452770 shutdown_connections 2024-04-02T14:33:47.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:46.621+0000 7feebf000700 1 --2- 172.21.15.90:0/1883452770 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fee9c0a3cc0 0x7fee9c0a6180 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:47.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:46.621+0000 7feebf000700 1 --2- 172.21.15.90:0/1883452770 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7feeb810a7e0 0x7feeb819f880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:47.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:46.621+0000 7feebf000700 1 --2- 172.21.15.90:0/1883452770 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7feeb81047e0 0x7feeb819afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:47.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:46.621+0000 7feebf000700 1 --2- 172.21.15.90:0/1883452770 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7feeb81051a0 0x7feeb819b4f0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:47.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:46.621+0000 7feebf000700 1 -- 172.21.15.90:0/1883452770 >> 172.21.15.90:0/1883452770 conn(0x7feeb81002f0 msgr2=0x7feeb8100960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:33:47.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:46.621+0000 7feebf000700 1 -- 172.21.15.90:0/1883452770 shutdown_connections 2024-04-02T14:33:47.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:46.622+0000 7feebf000700 1 -- 172.21.15.90:0/1883452770 wait complete. 2024-04-02T14:33:47.727 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:33:47 smithi090 ceph-mon[31674]: pgmap v356: 97 pgs: 97 active+clean; 644 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 597 B/s wr, 10 op/s 2024-04-02T14:33:47.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:33:47 smithi177 ceph-mon[36431]: pgmap v356: 97 pgs: 97 active+clean; 644 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 597 B/s wr, 10 op/s 2024-04-02T14:33:48.110 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:33:47 smithi092 ceph-mon[36497]: pgmap v356: 97 pgs: 97 active+clean; 644 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 597 B/s wr, 10 op/s 2024-04-02T14:33:48.526 INFO:tasks.cephadm.ceph_manager.ceph:need seq 322122547231 got 322122547237 for osd.10 2024-04-02T14:33:48.526 DEBUG:teuthology.parallel:result is None 2024-04-02T14:33:49.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:49.295+0000 7f84f3e00700 1 -- 172.21.15.90:0/795639486 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f84ec109810 msgr2=0x7f84ec10d860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:33:49.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:49.295+0000 7f84f3e00700 1 --2- 172.21.15.90:0/795639486 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f84ec109810 0x7f84ec10d860 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f84dc0099c0 tx=0x7f84dc009cd0 comp rx=0 tx=0).stop 2024-04-02T14:33:49.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:49.295+0000 7f84f3e00700 1 -- 172.21.15.90:0/795639486 shutdown_connections 2024-04-02T14:33:49.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:49.295+0000 7f84f3e00700 1 --2- 172.21.15.90:0/795639486 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f84ec10df40 0x7f84ec115b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:49.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:49.295+0000 7f84f3e00700 1 --2- 172.21.15.90:0/795639486 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f84ec108e60 0x7f84ec109240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:49.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:49.295+0000 7f84f3e00700 1 --2- 172.21.15.90:0/795639486 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f84ec109810 0x7f84ec10d860 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:49.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:49.295+0000 7f84f3e00700 1 -- 172.21.15.90:0/795639486 >> 172.21.15.90:0/795639486 conn(0x7f84ec075dc0 msgr2=0x7f84ec0781e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:33:49.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:49.295+0000 7f84f3e00700 1 -- 172.21.15.90:0/795639486 shutdown_connections 2024-04-02T14:33:49.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:49.295+0000 7f84f3e00700 1 -- 172.21.15.90:0/795639486 wait complete. 2024-04-02T14:33:49.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:49.297+0000 7f84f3e00700 1 Processor -- start 2024-04-02T14:33:49.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:49.297+0000 7f84f3e00700 1 -- start start 2024-04-02T14:33:49.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:49.297+0000 7f84f3e00700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f84ec108e60 0x7f84ec19f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:33:49.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:49.297+0000 7f84f3e00700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f84ec109810 0x7f84ec19f8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:33:49.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:49.297+0000 7f84f3e00700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f84ec10df40 0x7f84ec1a3c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:33:49.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:49.297+0000 7f84f3e00700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f84ec118f70 con 0x7f84ec109810 2024-04-02T14:33:49.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:49.297+0000 7f84f3e00700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f84ec118df0 con 0x7f84ec10df40 2024-04-02T14:33:49.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:49.297+0000 7f84f3e00700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f84ec1190f0 con 0x7f84ec108e60 2024-04-02T14:33:49.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:49.298+0000 7f84f0a00700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f84ec109810 0x7f84ec19f8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:33:49.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:49.298+0000 7f84f1400700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f84ec108e60 0x7f84ec19f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:33:49.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:49.298+0000 7f84f1e00700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f84ec10df40 0x7f84ec1a3c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:33:49.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:49.298+0000 7f84f1400700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f84ec108e60 0x7f84ec19f3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:41998/0 (socket says 172.21.15.90:41998) 2024-04-02T14:33:49.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:49.298+0000 7f84f1400700 1 -- 172.21.15.90:0/4008758868 learned_addr learned my addr 172.21.15.90:0/4008758868 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:33:49.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:49.298+0000 7f84f1e00700 1 -- 172.21.15.90:0/4008758868 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f84ec108e60 msgr2=0x7f84ec19f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:33:49.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:49.298+0000 7f84f1e00700 1 --2- 172.21.15.90:0/4008758868 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f84ec108e60 0x7f84ec19f3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:49.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:49.298+0000 7f84f1e00700 1 -- 172.21.15.90:0/4008758868 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f84ec109810 msgr2=0x7f84ec19f8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:33:49.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:49.298+0000 7f84f1e00700 1 --2- 172.21.15.90:0/4008758868 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f84ec109810 0x7f84ec19f8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:49.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:49.298+0000 7f84f1e00700 1 -- 172.21.15.90:0/4008758868 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84ec1a43b0 con 0x7f84ec10df40 2024-04-02T14:33:49.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:49.298+0000 7f84f1400700 1 --2- 172.21.15.90:0/4008758868 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f84ec108e60 0x7f84ec19f3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:33:49.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:49.299+0000 7f84f1e00700 1 --2- 172.21.15.90:0/4008758868 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f84ec10df40 0x7f84ec1a3c70 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f84e400efc0 tx=0x7f84e400c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:33:49.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:49.299+0000 7f84ea000700 1 -- 172.21.15.90:0/4008758868 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84e40092e0 con 0x7f84ec10df40 2024-04-02T14:33:49.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:49.299+0000 7f84ea000700 1 -- 172.21.15.90:0/4008758868 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f84e4004bb0 con 0x7f84ec10df40 2024-04-02T14:33:49.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:49.299+0000 7f84ea000700 1 -- 172.21.15.90:0/4008758868 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84e4004280 con 0x7f84ec10df40 2024-04-02T14:33:49.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:49.299+0000 7f84f3e00700 1 -- 172.21.15.90:0/4008758868 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f84ec1a4640 con 0x7f84ec10df40 2024-04-02T14:33:49.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:49.299+0000 7f84f3e00700 1 -- 172.21.15.90:0/4008758868 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84ec1abf20 con 0x7f84ec10df40 2024-04-02T14:33:49.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:49.300+0000 7f84f3e00700 1 -- 172.21.15.90:0/4008758868 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f84ec10a340 con 0x7f84ec10df40 2024-04-02T14:33:49.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:49.302+0000 7f84ea000700 1 -- 172.21.15.90:0/4008758868 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f84e401b030 con 0x7f84ec10df40 2024-04-02T14:33:49.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:49.303+0000 7f84ea000700 1 --2- 172.21.15.90:0/4008758868 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f84d40a3cc0 0x7f84d40a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:33:49.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:49.303+0000 7f84ea000700 1 -- 172.21.15.90:0/4008758868 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8248+0+0 (secure 0 0 0) 0x7f84e40d0080 con 0x7f84ec10df40 2024-04-02T14:33:49.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:49.303+0000 7f84f1400700 1 --2- 172.21.15.90:0/4008758868 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f84d40a3cc0 0x7f84d40a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:33:49.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:49.304+0000 7f84f1400700 1 --2- 172.21.15.90:0/4008758868 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f84d40a3cc0 0x7f84d40a6180 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f84e0011630 tx=0x7f84e0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:33:49.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:49.306+0000 7f84ea000700 1 -- 172.21.15.90:0/4008758868 <== mon.2 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7f84e4022020 con 0x7f84ec10df40 2024-04-02T14:33:49.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:49.553+0000 7f84f3e00700 1 -- 172.21.15.90:0/4008758868 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f84ec02dd30 con 0x7f84ec10df40 2024-04-02T14:33:49.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:49.554+0000 7f84ea000700 1 -- 172.21.15.90:0/4008758868 <== mon.2 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f84e4010040 con 0x7f84ec10df40 2024-04-02T14:33:49.556 INFO:teuthology.orchestra.run.smithi090.stdout:163208757312 2024-04-02T14:33:49.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:49.558+0000 7f84f3e00700 1 -- 172.21.15.90:0/4008758868 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f84d40a3cc0 msgr2=0x7f84d40a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:33:49.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:49.558+0000 7f84f3e00700 1 --2- 172.21.15.90:0/4008758868 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f84d40a3cc0 0x7f84d40a6180 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f84e0011630 tx=0x7f84e0009260 comp rx=0 tx=0).stop 2024-04-02T14:33:49.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:49.558+0000 7f84f3e00700 1 -- 172.21.15.90:0/4008758868 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f84ec10df40 msgr2=0x7f84ec1a3c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:33:49.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:49.558+0000 7f84f3e00700 1 --2- 172.21.15.90:0/4008758868 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f84ec10df40 0x7f84ec1a3c70 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f84e400efc0 tx=0x7f84e400c3a0 comp rx=0 tx=0).stop 2024-04-02T14:33:49.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:49.559+0000 7f84f3e00700 1 -- 172.21.15.90:0/4008758868 shutdown_connections 2024-04-02T14:33:49.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:49.559+0000 7f84f3e00700 1 --2- 172.21.15.90:0/4008758868 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f84d40a3cc0 0x7f84d40a6180 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:49.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:49.559+0000 7f84f3e00700 1 --2- 172.21.15.90:0/4008758868 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f84ec10df40 0x7f84ec1a3c70 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:49.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:49.559+0000 7f84f3e00700 1 --2- 172.21.15.90:0/4008758868 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f84ec108e60 0x7f84ec19f3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:49.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:49.559+0000 7f84f3e00700 1 --2- 172.21.15.90:0/4008758868 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f84ec109810 0x7f84ec19f8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:49.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:49.559+0000 7f84f3e00700 1 -- 172.21.15.90:0/4008758868 >> 172.21.15.90:0/4008758868 conn(0x7f84ec075dc0 msgr2=0x7f84ec0768e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:33:49.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:49.559+0000 7f84f3e00700 1 -- 172.21.15.90:0/4008758868 shutdown_connections 2024-04-02T14:33:49.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:49.559+0000 7f84f3e00700 1 -- 172.21.15.90:0/4008758868 wait complete. 2024-04-02T14:33:50.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:50.101+0000 7f30b0e00700 1 -- 172.21.15.90:0/3572212761 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f30ac06a2a0 msgr2=0x7f30ac06a720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:33:50.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:50.101+0000 7f30b0e00700 1 --2- 172.21.15.90:0/3572212761 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f30ac06a2a0 0x7f30ac06a720 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f3098009990 tx=0x7f3098009ca0 comp rx=0 tx=0).stop 2024-04-02T14:33:50.104 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:33:49 smithi090 ceph-mon[31674]: pgmap v357: 97 pgs: 97 active+clean; 644 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 511 B/s wr, 10 op/s 2024-04-02T14:33:50.104 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:33:49 smithi090 ceph-mon[31674]: from='client.? 172.21.15.90:0/4008758868' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-02T14:33:50.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:50.102+0000 7f30b0e00700 1 -- 172.21.15.90:0/3572212761 shutdown_connections 2024-04-02T14:33:50.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:50.102+0000 7f30b0e00700 1 --2- 172.21.15.90:0/3572212761 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f30ac06ac60 0x7f30ac10f450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:50.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:50.102+0000 7f30b0e00700 1 --2- 172.21.15.90:0/3572212761 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f30ac108650 0x7f30ac108a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:50.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:50.102+0000 7f30b0e00700 1 --2- 172.21.15.90:0/3572212761 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f30ac06a2a0 0x7f30ac06a720 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:50.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:50.102+0000 7f30b0e00700 1 -- 172.21.15.90:0/3572212761 >> 172.21.15.90:0/3572212761 conn(0x7f30ac078420 msgr2=0x7f30ac078830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:33:50.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:50.102+0000 7f30b0e00700 1 -- 172.21.15.90:0/3572212761 shutdown_connections 2024-04-02T14:33:50.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:50.102+0000 7f30b0e00700 1 -- 172.21.15.90:0/3572212761 wait complete. 2024-04-02T14:33:50.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:50.103+0000 7f30b0e00700 1 Processor -- start 2024-04-02T14:33:50.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:50.103+0000 7f30b0e00700 1 -- start start 2024-04-02T14:33:50.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:50.104+0000 7f30b0e00700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f30ac06a2a0 0x7f30ac19afb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:33:50.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:50.104+0000 7f30b0e00700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f30ac06ac60 0x7f30ac19b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:33:50.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:50.104+0000 7f30b0e00700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f30ac108650 0x7f30ac19f880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:33:50.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:50.104+0000 7f30b0e00700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f30ac1128f0 con 0x7f30ac06ac60 2024-04-02T14:33:50.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:50.104+0000 7f30b0e00700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f30ac112770 con 0x7f30ac06a2a0 2024-04-02T14:33:50.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:50.104+0000 7f30b0e00700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f30ac112a70 con 0x7f30ac108650 2024-04-02T14:33:50.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:50.104+0000 7f30a9400700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f30ac06ac60 0x7f30ac19b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:33:50.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:50.104+0000 7f30a9e00700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f30ac06a2a0 0x7f30ac19afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:33:50.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:50.104+0000 7f30a9e00700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f30ac06a2a0 0x7f30ac19afb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:46050/0 (socket says 172.21.15.90:46050) 2024-04-02T14:33:50.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:50.104+0000 7f30a9e00700 1 -- 172.21.15.90:0/4285085778 learned_addr learned my addr 172.21.15.90:0/4285085778 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:33:50.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:50.104+0000 7f30aa800700 1 --2- 172.21.15.90:0/4285085778 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f30ac108650 0x7f30ac19f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:33:50.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:50.104+0000 7f30a9e00700 1 -- 172.21.15.90:0/4285085778 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f30ac108650 msgr2=0x7f30ac19f880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:33:50.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:50.104+0000 7f30a9e00700 1 --2- 172.21.15.90:0/4285085778 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f30ac108650 0x7f30ac19f880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:50.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:50.104+0000 7f30a9e00700 1 -- 172.21.15.90:0/4285085778 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f30ac06ac60 msgr2=0x7f30ac19b4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:33:50.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:50.104+0000 7f30a9e00700 1 --2- 172.21.15.90:0/4285085778 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f30ac06ac60 0x7f30ac19b4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:50.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:50.104+0000 7f30a9e00700 1 -- 172.21.15.90:0/4285085778 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30ac19ffc0 con 0x7f30ac06a2a0 2024-04-02T14:33:50.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:50.105+0000 7f30a9e00700 1 --2- 172.21.15.90:0/4285085778 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f30ac06a2a0 0x7f30ac19afb0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f309c00e9a0 tx=0x7f309c00ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:33:50.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:50.105+0000 7f30a2a00700 1 -- 172.21.15.90:0/4285085778 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f309c00ca10 con 0x7f30ac06a2a0 2024-04-02T14:33:50.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:50.105+0000 7f30b0e00700 1 -- 172.21.15.90:0/4285085778 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f30ac1a02b0 con 0x7f30ac06a2a0 2024-04-02T14:33:50.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:50.105+0000 7f30b0e00700 1 -- 172.21.15.90:0/4285085778 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30ac1a7b30 con 0x7f30ac06a2a0 2024-04-02T14:33:50.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:50.105+0000 7f30a2a00700 1 -- 172.21.15.90:0/4285085778 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f309c00cbb0 con 0x7f30ac06a2a0 2024-04-02T14:33:50.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:50.105+0000 7f30a2a00700 1 -- 172.21.15.90:0/4285085778 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f309c018680 con 0x7f30ac06a2a0 2024-04-02T14:33:50.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:50.107+0000 7f30b0e00700 1 -- 172.21.15.90:0/4285085778 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f308c005330 con 0x7f30ac06a2a0 2024-04-02T14:33:50.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:50.108+0000 7f30a2a00700 1 -- 172.21.15.90:0/4285085778 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f309c018820 con 0x7f30ac06a2a0 2024-04-02T14:33:50.110 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:33:49 smithi092 ceph-mon[36497]: pgmap v357: 97 pgs: 97 active+clean; 644 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 511 B/s wr, 10 op/s 2024-04-02T14:33:50.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:33:49 smithi092 ceph-mon[36497]: from='client.? 172.21.15.90:0/4008758868' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-02T14:33:50.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:50.110+0000 7f30a2a00700 1 --2- 172.21.15.90:0/4285085778 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f30900a3cc0 0x7f30900a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:33:50.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:50.110+0000 7f30a2a00700 1 -- 172.21.15.90:0/4285085778 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8248+0+0 (secure 0 0 0) 0x7f309c0d04b0 con 0x7f30ac06a2a0 2024-04-02T14:33:50.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:50.110+0000 7f30a9400700 1 --2- 172.21.15.90:0/4285085778 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f30900a3cc0 0x7f30900a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:33:50.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:50.110+0000 7f30a9400700 1 --2- 172.21.15.90:0/4285085778 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f30900a3cc0 0x7f30900a6180 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f30ac103570 tx=0x7f3098005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:33:50.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:50.114+0000 7f30a2a00700 1 -- 172.21.15.90:0/4285085778 <== mon.2 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7f309c09ce20 con 0x7f30ac06a2a0 2024-04-02T14:33:50.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:33:49 smithi177 ceph-mon[36431]: pgmap v357: 97 pgs: 97 active+clean; 644 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 511 B/s wr, 10 op/s 2024-04-02T14:33:50.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:33:49 smithi177 ceph-mon[36431]: from='client.? 172.21.15.90:0/4008758868' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-02T14:33:50.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:50.369+0000 7f30b0e00700 1 -- 172.21.15.90:0/4285085778 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 11} v 0) -- 0x7f308c005620 con 0x7f30ac06a2a0 2024-04-02T14:33:50.371 INFO:teuthology.orchestra.run.smithi090.stdout:356482285599 2024-04-02T14:33:50.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:50.369+0000 7f30a2a00700 1 -- 172.21.15.90:0/4285085778 <== mon.2 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 11}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7f309c09c720 con 0x7f30ac06a2a0 2024-04-02T14:33:50.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:50.372+0000 7f30b0e00700 1 -- 172.21.15.90:0/4285085778 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f30900a3cc0 msgr2=0x7f30900a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:33:50.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:50.372+0000 7f30b0e00700 1 --2- 172.21.15.90:0/4285085778 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f30900a3cc0 0x7f30900a6180 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f30ac103570 tx=0x7f3098005fd0 comp rx=0 tx=0).stop 2024-04-02T14:33:50.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:50.372+0000 7f30b0e00700 1 -- 172.21.15.90:0/4285085778 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f30ac06a2a0 msgr2=0x7f30ac19afb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:33:50.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:50.372+0000 7f30b0e00700 1 --2- 172.21.15.90:0/4285085778 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f30ac06a2a0 0x7f30ac19afb0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f309c00e9a0 tx=0x7f309c00ecb0 comp rx=0 tx=0).stop 2024-04-02T14:33:50.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:50.372+0000 7f30b0e00700 1 -- 172.21.15.90:0/4285085778 shutdown_connections 2024-04-02T14:33:50.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:50.372+0000 7f30b0e00700 1 --2- 172.21.15.90:0/4285085778 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f30900a3cc0 0x7f30900a6180 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:50.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:50.372+0000 7f30b0e00700 1 --2- 172.21.15.90:0/4285085778 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f30ac108650 0x7f30ac19f880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:50.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:50.372+0000 7f30b0e00700 1 --2- 172.21.15.90:0/4285085778 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f30ac06a2a0 0x7f30ac19afb0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:50.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:50.372+0000 7f30b0e00700 1 --2- 172.21.15.90:0/4285085778 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f30ac06ac60 0x7f30ac19b4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:50.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:50.372+0000 7f30b0e00700 1 -- 172.21.15.90:0/4285085778 >> 172.21.15.90:0/4285085778 conn(0x7f30ac078420 msgr2=0x7f30ac10d7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:33:50.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:50.372+0000 7f30b0e00700 1 -- 172.21.15.90:0/4285085778 shutdown_connections 2024-04-02T14:33:50.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:50.372+0000 7f30b0e00700 1 -- 172.21.15.90:0/4285085778 wait complete. 2024-04-02T14:33:50.757 INFO:tasks.cephadm.ceph_manager.ceph:need seq 163208757305 got 163208757312 for osd.5 2024-04-02T14:33:50.757 DEBUG:teuthology.parallel:result is None 2024-04-02T14:33:51.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:33:50 smithi092 ceph-mon[36497]: from='client.? 172.21.15.90:0/4285085778' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-02T14:33:51.176 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:33:50 smithi090 ceph-mon[31674]: from='client.? 172.21.15.90:0/4285085778' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-02T14:33:51.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:33:50 smithi177 ceph-mon[36431]: from='client.? 172.21.15.90:0/4285085778' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-02T14:33:51.608 INFO:tasks.cephadm.ceph_manager.ceph:need seq 356482285593 got 356482285599 for osd.11 2024-04-02T14:33:51.609 DEBUG:teuthology.parallel:result is None 2024-04-02T14:33:51.609 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-04-02T14:33:51.609 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:191d963ac9de7322da8aff20b567b3fa458e2ef6 shell --fsid 6449448a-f0fc-11ee-b647-cb9ed24678a4 -- ceph pg dump --format=json 2024-04-02T14:33:52.044 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T14:33:52.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:33:51 smithi092 ceph-mon[36497]: pgmap v358: 97 pgs: 97 active+clean; 644 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 341 B/s wr, 11 op/s 2024-04-02T14:33:52.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:33:51 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:33:52.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:33:51 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:33:52.159 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:33:51 smithi090 ceph-mon[31674]: pgmap v358: 97 pgs: 97 active+clean; 644 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 341 B/s wr, 11 op/s 2024-04-02T14:33:52.159 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:33:51 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:33:52.160 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:33:51 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:33:52.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:33:51 smithi177 ceph-mon[36431]: pgmap v358: 97 pgs: 97 active+clean; 644 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 341 B/s wr, 11 op/s 2024-04-02T14:33:52.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:33:51 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:33:52.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:33:51 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:33:53.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:33:52 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:33:53.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:33:52 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T14:33:53.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:33:52 smithi092 ceph-mon[36497]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:33:53.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:33:52 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:33:53.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:33:52 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T14:33:53.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:33:52 smithi177 ceph-mon[36431]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:33:53.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:33:52 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:33:53.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:33:52 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T14:33:53.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:33:52 smithi090 ceph-mon[31674]: from='mgr.14152 172.21.15.90:0/1350519148' entity='mgr.x' 2024-04-02T14:33:54.006 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:33:53 smithi090 ceph-mon[31674]: pgmap v359: 97 pgs: 97 active+clean; 644 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-02T14:33:54.110 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:33:53 smithi092 ceph-mon[36497]: pgmap v359: 97 pgs: 97 active+clean; 644 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-02T14:33:54.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:54.169+0000 7fa363e00700 1 -- 172.21.15.90:0/1658300557 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa364102500 msgr2=0x7fa364102980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:33:54.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:54.169+0000 7fa363e00700 1 --2- 172.21.15.90:0/1658300557 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa364102500 0x7fa364102980 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fa350009920 tx=0x7fa350009c30 comp rx=0 tx=0).stop 2024-04-02T14:33:54.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:54.169+0000 7fa363e00700 1 -- 172.21.15.90:0/1658300557 shutdown_connections 2024-04-02T14:33:54.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:54.169+0000 7fa363e00700 1 --2- 172.21.15.90:0/1658300557 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa3641085b0 0x7fa364108990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:54.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:54.169+0000 7fa363e00700 1 --2- 172.21.15.90:0/1658300557 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa364102ec0 0x7fa36410f380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:54.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:54.169+0000 7fa363e00700 1 --2- 172.21.15.90:0/1658300557 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa364102500 0x7fa364102980 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:54.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:54.169+0000 7fa363e00700 1 -- 172.21.15.90:0/1658300557 >> 172.21.15.90:0/1658300557 conn(0x7fa3640fdff0 msgr2=0x7fa364100410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:33:54.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:54.169+0000 7fa363e00700 1 -- 172.21.15.90:0/1658300557 shutdown_connections 2024-04-02T14:33:54.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:54.169+0000 7fa363e00700 1 -- 172.21.15.90:0/1658300557 wait complete. 2024-04-02T14:33:54.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:54.170+0000 7fa363e00700 1 Processor -- start 2024-04-02T14:33:54.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:54.170+0000 7fa363e00700 1 -- start start 2024-04-02T14:33:54.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:54.171+0000 7fa363e00700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa364102500 0x7fa3640748f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:33:54.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:54.171+0000 7fa363e00700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa364102ec0 0x7fa364074e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:33:54.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:54.171+0000 7fa363e00700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa3641085b0 0x7fa364075370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:33:54.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:54.171+0000 7fa363e00700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fa3641127f0 con 0x7fa364102ec0 2024-04-02T14:33:54.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:54.171+0000 7fa363e00700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fa364112670 con 0x7fa3641085b0 2024-04-02T14:33:54.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:54.171+0000 7fa363e00700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fa364112970 con 0x7fa364102500 2024-04-02T14:33:54.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:54.171+0000 7fa363400700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa3641085b0 0x7fa364075370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:33:54.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:54.171+0000 7fa362000700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa364102ec0 0x7fa364074e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:33:54.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:54.171+0000 7fa363400700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa3641085b0 0x7fa364075370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:54736/0 (socket says 172.21.15.90:54736) 2024-04-02T14:33:54.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:54.171+0000 7fa363400700 1 -- 172.21.15.90:0/2436440742 learned_addr learned my addr 172.21.15.90:0/2436440742 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:33:54.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:54.171+0000 7fa362a00700 1 --2- 172.21.15.90:0/2436440742 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa364102500 0x7fa3640748f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:33:54.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:54.172+0000 7fa363400700 1 -- 172.21.15.90:0/2436440742 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa364102500 msgr2=0x7fa3640748f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:33:54.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:54.172+0000 7fa363400700 1 --2- 172.21.15.90:0/2436440742 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa364102500 0x7fa3640748f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:54.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:54.172+0000 7fa363400700 1 -- 172.21.15.90:0/2436440742 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa364102ec0 msgr2=0x7fa364074e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:33:54.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:54.172+0000 7fa363400700 1 --2- 172.21.15.90:0/2436440742 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa364102ec0 0x7fa364074e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:54.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:54.172+0000 7fa363400700 1 -- 172.21.15.90:0/2436440742 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3641a71e0 con 0x7fa3641085b0 2024-04-02T14:33:54.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:54.172+0000 7fa363400700 1 --2- 172.21.15.90:0/2436440742 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa3641085b0 0x7fa364075370 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fa35c00ef30 tx=0x7fa35c00c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:33:54.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:54.172+0000 7fa35b400700 1 -- 172.21.15.90:0/2436440742 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa35c0092e0 con 0x7fa3641085b0 2024-04-02T14:33:54.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:54.172+0000 7fa35b400700 1 -- 172.21.15.90:0/2436440742 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa35c004bb0 con 0x7fa3641085b0 2024-04-02T14:33:54.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:54.172+0000 7fa35b400700 1 -- 172.21.15.90:0/2436440742 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa35c004280 con 0x7fa3641085b0 2024-04-02T14:33:54.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:54.172+0000 7fa363e00700 1 -- 172.21.15.90:0/2436440742 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa3641a7470 con 0x7fa3641085b0 2024-04-02T14:33:54.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:54.172+0000 7fa363e00700 1 -- 172.21.15.90:0/2436440742 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3641a7930 con 0x7fa3641085b0 2024-04-02T14:33:54.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:54.173+0000 7fa363e00700 1 -- 172.21.15.90:0/2436440742 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa3640692b0 con 0x7fa3641085b0 2024-04-02T14:33:54.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:54.175+0000 7fa35b400700 1 -- 172.21.15.90:0/2436440742 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fa35c01b030 con 0x7fa3641085b0 2024-04-02T14:33:54.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:54.177+0000 7fa35b400700 1 --2- 172.21.15.90:0/2436440742 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fa3480a3c40 0x7fa3480a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:33:54.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:54.177+0000 7fa35b400700 1 -- 172.21.15.90:0/2436440742 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8248+0+0 (secure 0 0 0) 0x7fa35c0d08b0 con 0x7fa3641085b0 2024-04-02T14:33:54.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:54.177+0000 7fa362a00700 1 --2- 172.21.15.90:0/2436440742 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fa3480a3c40 0x7fa3480a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:33:54.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:54.178+0000 7fa362a00700 1 --2- 172.21.15.90:0/2436440742 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fa3480a3c40 0x7fa3480a6100 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fa354005fd0 tx=0x7fa354009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:33:54.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:54.179+0000 7fa35b400700 1 -- 172.21.15.90:0/2436440742 <== mon.2 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7fa35c022020 con 0x7fa3641085b0 2024-04-02T14:33:54.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:33:53 smithi177 ceph-mon[36431]: pgmap v359: 97 pgs: 97 active+clean; 644 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-02T14:33:54.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:54.398+0000 7fa363e00700 1 -- 172.21.15.90:0/2436440742 --> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fa364062a30 con 0x7fa3480a3c40 2024-04-02T14:33:54.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:54.411+0000 7fa35b400700 1 -- 172.21.15.90:0/2436440742 <== mgr.14152 v2:172.21.15.90:6800/4266054817 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258894 (secure 0 0 0) 0x7fa364062a30 con 0x7fa3480a3c40 2024-04-02T14:33:54.420 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-02T14:33:54.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:54.418+0000 7fa363e00700 1 -- 172.21.15.90:0/2436440742 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fa3480a3c40 msgr2=0x7fa3480a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:33:54.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:54.418+0000 7fa363e00700 1 --2- 172.21.15.90:0/2436440742 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fa3480a3c40 0x7fa3480a6100 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fa354005fd0 tx=0x7fa354009310 comp rx=0 tx=0).stop 2024-04-02T14:33:54.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:54.418+0000 7fa363e00700 1 -- 172.21.15.90:0/2436440742 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa3641085b0 msgr2=0x7fa364075370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:33:54.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:54.418+0000 7fa363e00700 1 --2- 172.21.15.90:0/2436440742 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa3641085b0 0x7fa364075370 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fa35c00ef30 tx=0x7fa35c00c3a0 comp rx=0 tx=0).stop 2024-04-02T14:33:54.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:54.418+0000 7fa363e00700 1 -- 172.21.15.90:0/2436440742 shutdown_connections 2024-04-02T14:33:54.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:54.418+0000 7fa363e00700 1 --2- 172.21.15.90:0/2436440742 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fa3480a3c40 0x7fa3480a6100 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:54.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:54.418+0000 7fa363e00700 1 --2- 172.21.15.90:0/2436440742 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa3641085b0 0x7fa364075370 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:54.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:54.418+0000 7fa363e00700 1 --2- 172.21.15.90:0/2436440742 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa364102500 0x7fa3640748f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:54.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:54.418+0000 7fa363e00700 1 --2- 172.21.15.90:0/2436440742 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa364102ec0 0x7fa364074e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:54.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:54.418+0000 7fa363e00700 1 -- 172.21.15.90:0/2436440742 >> 172.21.15.90:0/2436440742 conn(0x7fa3640fdff0 msgr2=0x7fa3640ff7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:33:54.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:54.418+0000 7fa363e00700 1 -- 172.21.15.90:0/2436440742 shutdown_connections 2024-04-02T14:33:54.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:54.418+0000 7fa363e00700 1 -- 172.21.15.90:0/2436440742 wait complete. 2024-04-02T14:33:54.423 INFO:teuthology.orchestra.run.smithi090.stderr:dumped all 2024-04-02T14:33:55.301 INFO:teuthology.orchestra.run.smithi090.stdout:{"pg_ready":true,"pg_map":{"version":359,"stamp":"2024-04-02T14:33:52.768249+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":659383,"num_objects":102,"num_object_clones":0,"num_object_copies":306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":964,"num_read_kb":601,"num_write":431,"num_write_kb":3081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":1224785,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":854,"ondisk_log_size":854,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":404588,"kb_used_data":13836,"kb_used_omap":18,"kb_used_meta":390637,"kb_avail":1124537236,"statfs":{"total":1151940427776,"available":1151526129664,"internally_reserved":0,"allocated":14168064,"data_stored":7660497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":400012602},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":115,"num_read_kb":55,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005280"},"pg_stats":[{"pgid":"3.3f","version":"90'8","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:33:34.770820+0000","last_change":"2024-04-02T14:32:22.558913+0000","last_active":"2024-04-02T14:33:34.770820+0000","last_peered":"2024-04-02T14:33:34.770820+0000","last_clean":"2024-04-02T14:33:34.770820+0000","last_became_active":"2024-04-02T14:32:22.558606+0000","last_became_peered":"2024-04-02T14:32:22.558606+0000","last_unstale":"2024-04-02T14:33:34.770820+0000","last_undegraded":"2024-04-02T14:33:34.770820+0000","last_fullsized":"2024-04-02T14:33:34.770820+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:59:49.205499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"90'4","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:33:34.739039+0000","last_change":"2024-04-02T14:32:22.551898+0000","last_active":"2024-04-02T14:33:34.739039+0000","last_peered":"2024-04-02T14:33:34.739039+0000","last_clean":"2024-04-02T14:33:34.739039+0000","last_became_active":"2024-04-02T14:32:22.549016+0000","last_became_peered":"2024-04-02T14:32:22.549016+0000","last_unstale":"2024-04-02T14:33:34.739039+0000","last_undegraded":"2024-04-02T14:33:34.739039+0000","last_fullsized":"2024-04-02T14:33:34.739039+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:38:03.529505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"90'7","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:33:34.760941+0000","last_change":"2024-04-02T14:32:22.556214+0000","last_active":"2024-04-02T14:33:34.760941+0000","last_peered":"2024-04-02T14:33:34.760941+0000","last_clean":"2024-04-02T14:33:34.760941+0000","last_became_active":"2024-04-02T14:32:22.555686+0000","last_became_peered":"2024-04-02T14:32:22.555686+0000","last_unstale":"2024-04-02T14:33:34.760941+0000","last_undegraded":"2024-04-02T14:33:34.760941+0000","last_fullsized":"2024-04-02T14:33:34.760941+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:06:40.161203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"90'8","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:33:34.743043+0000","last_change":"2024-04-02T14:32:22.537830+0000","last_active":"2024-04-02T14:33:34.743043+0000","last_peered":"2024-04-02T14:33:34.743043+0000","last_clean":"2024-04-02T14:33:34.743043+0000","last_became_active":"2024-04-02T14:32:22.536860+0000","last_became_peered":"2024-04-02T14:32:22.536860+0000","last_unstale":"2024-04-02T14:33:34.743043+0000","last_undegraded":"2024-04-02T14:33:34.743043+0000","last_fullsized":"2024-04-02T14:33:34.743043+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T01:07:57.244177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"90'4","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:33:34.759688+0000","last_change":"2024-04-02T14:32:22.549550+0000","last_active":"2024-04-02T14:33:34.759688+0000","last_peered":"2024-04-02T14:33:34.759688+0000","last_clean":"2024-04-02T14:33:34.759688+0000","last_became_active":"2024-04-02T14:32:22.549177+0000","last_became_peered":"2024-04-02T14:32:22.549177+0000","last_unstale":"2024-04-02T14:33:34.759688+0000","last_undegraded":"2024-04-02T14:33:34.759688+0000","last_fullsized":"2024-04-02T14:33:34.759688+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:31:49.175583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"90'6","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:33:34.744861+0000","last_change":"2024-04-02T14:32:35.849338+0000","last_active":"2024-04-02T14:33:34.744861+0000","last_peered":"2024-04-02T14:33:34.744861+0000","last_clean":"2024-04-02T14:33:34.744861+0000","last_became_active":"2024-04-02T14:32:35.848929+0000","last_became_peered":"2024-04-02T14:32:35.848929+0000","last_unstale":"2024-04-02T14:33:34.744861+0000","last_undegraded":"2024-04-02T14:33:34.744861+0000","last_fullsized":"2024-04-02T14:33:34.744861+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:55:58.614290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,6],"acting":[10,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"90'5","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:33:34.752353+0000","last_change":"2024-04-02T14:32:22.546861+0000","last_active":"2024-04-02T14:33:34.752353+0000","last_peered":"2024-04-02T14:33:34.752353+0000","last_clean":"2024-04-02T14:33:34.752353+0000","last_became_active":"2024-04-02T14:32:22.546477+0000","last_became_peered":"2024-04-02T14:32:22.546477+0000","last_unstale":"2024-04-02T14:33:34.752353+0000","last_undegraded":"2024-04-02T14:33:34.752353+0000","last_fullsized":"2024-04-02T14:33:34.752353+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:59:17.133714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"90'9","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:33:34.763067+0000","last_change":"2024-04-02T14:32:22.554287+0000","last_active":"2024-04-02T14:33:34.763067+0000","last_peered":"2024-04-02T14:33:34.763067+0000","last_clean":"2024-04-02T14:33:34.763067+0000","last_became_active":"2024-04-02T14:32:22.553491+0000","last_became_peered":"2024-04-02T14:32:22.553491+0000","last_unstale":"2024-04-02T14:33:34.763067+0000","last_undegraded":"2024-04-02T14:33:34.763067+0000","last_fullsized":"2024-04-02T14:33:34.763067+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:51:30.785380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"90'11","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:33:34.757821+0000","last_change":"2024-04-02T14:32:22.538158+0000","last_active":"2024-04-02T14:33:34.757821+0000","last_peered":"2024-04-02T14:33:34.757821+0000","last_clean":"2024-04-02T14:33:34.757821+0000","last_became_active":"2024-04-02T14:32:22.537796+0000","last_became_peered":"2024-04-02T14:32:22.537796+0000","last_unstale":"2024-04-02T14:33:34.757821+0000","last_undegraded":"2024-04-02T14:33:34.757821+0000","last_fullsized":"2024-04-02T14:33:34.757821+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:09:48.054965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"90'16","reported_seq":33,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:33:34.742793+0000","last_change":"2024-04-02T14:32:22.535225+0000","last_active":"2024-04-02T14:33:34.742793+0000","last_peered":"2024-04-02T14:33:34.742793+0000","last_clean":"2024-04-02T14:33:34.742793+0000","last_became_active":"2024-04-02T14:32:22.534856+0000","last_became_peered":"2024-04-02T14:32:22.534856+0000","last_unstale":"2024-04-02T14:33:34.742793+0000","last_undegraded":"2024-04-02T14:33:34.742793+0000","last_fullsized":"2024-04-02T14:33:34.742793+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:22:11.118918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"90'6","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:33:34.755859+0000","last_change":"2024-04-02T14:32:22.539138+0000","last_active":"2024-04-02T14:33:34.755859+0000","last_peered":"2024-04-02T14:33:34.755859+0000","last_clean":"2024-04-02T14:33:34.755859+0000","last_became_active":"2024-04-02T14:32:22.538762+0000","last_became_peered":"2024-04-02T14:32:22.538762+0000","last_unstale":"2024-04-02T14:33:34.755859+0000","last_undegraded":"2024-04-02T14:33:34.755859+0000","last_fullsized":"2024-04-02T14:33:34.755859+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:18:59.174339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"90'7","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:33:34.747727+0000","last_change":"2024-04-02T14:32:22.536383+0000","last_active":"2024-04-02T14:33:34.747727+0000","last_peered":"2024-04-02T14:33:34.747727+0000","last_clean":"2024-04-02T14:33:34.747727+0000","last_became_active":"2024-04-02T14:32:22.536043+0000","last_became_peered":"2024-04-02T14:32:22.536043+0000","last_unstale":"2024-04-02T14:33:34.747727+0000","last_undegraded":"2024-04-02T14:33:34.747727+0000","last_fullsized":"2024-04-02T14:33:34.747727+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:14:48.628684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"90'10","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:33:34.769406+0000","last_change":"2024-04-02T14:32:22.552512+0000","last_active":"2024-04-02T14:33:34.769406+0000","last_peered":"2024-04-02T14:33:34.769406+0000","last_clean":"2024-04-02T14:33:34.769406+0000","last_became_active":"2024-04-02T14:32:22.549502+0000","last_became_peered":"2024-04-02T14:32:22.549502+0000","last_unstale":"2024-04-02T14:33:34.769406+0000","last_undegraded":"2024-04-02T14:33:34.769406+0000","last_fullsized":"2024-04-02T14:33:34.769406+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:37:45.609280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"90'7","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:33:34.754701+0000","last_change":"2024-04-02T14:32:22.555074+0000","last_active":"2024-04-02T14:33:34.754701+0000","last_peered":"2024-04-02T14:33:34.754701+0000","last_clean":"2024-04-02T14:33:34.754701+0000","last_became_active":"2024-04-02T14:32:22.554422+0000","last_became_peered":"2024-04-02T14:32:22.554422+0000","last_unstale":"2024-04-02T14:33:34.754701+0000","last_undegraded":"2024-04-02T14:33:34.754701+0000","last_fullsized":"2024-04-02T14:33:34.754701+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:27:49.536941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"90'7","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:33:34.751759+0000","last_change":"2024-04-02T14:32:22.550322+0000","last_active":"2024-04-02T14:33:34.751759+0000","last_peered":"2024-04-02T14:33:34.751759+0000","last_clean":"2024-04-02T14:33:34.751759+0000","last_became_active":"2024-04-02T14:32:22.548710+0000","last_became_peered":"2024-04-02T14:32:22.548710+0000","last_unstale":"2024-04-02T14:33:34.751759+0000","last_undegraded":"2024-04-02T14:33:34.751759+0000","last_fullsized":"2024-04-02T14:33:34.751759+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:19:09.226556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"90'9","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:33:34.769494+0000","last_change":"2024-04-02T14:32:22.558705+0000","last_active":"2024-04-02T14:33:34.769494+0000","last_peered":"2024-04-02T14:33:34.769494+0000","last_clean":"2024-04-02T14:33:34.769494+0000","last_became_active":"2024-04-02T14:32:22.558309+0000","last_became_peered":"2024-04-02T14:32:22.558309+0000","last_unstale":"2024-04-02T14:33:34.769494+0000","last_undegraded":"2024-04-02T14:33:34.769494+0000","last_fullsized":"2024-04-02T14:33:34.769494+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:52:23.929627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"90'8","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:33:34.771040+0000","last_change":"2024-04-02T14:32:22.551211+0000","last_active":"2024-04-02T14:33:34.771040+0000","last_peered":"2024-04-02T14:33:34.771040+0000","last_clean":"2024-04-02T14:33:34.771040+0000","last_became_active":"2024-04-02T14:32:22.550882+0000","last_became_peered":"2024-04-02T14:32:22.550882+0000","last_unstale":"2024-04-02T14:33:34.771040+0000","last_undegraded":"2024-04-02T14:33:34.771040+0000","last_fullsized":"2024-04-02T14:33:34.771040+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:54:54.533063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"90'10","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:33:34.760206+0000","last_change":"2024-04-02T14:32:22.550692+0000","last_active":"2024-04-02T14:33:34.760206+0000","last_peered":"2024-04-02T14:33:34.760206+0000","last_clean":"2024-04-02T14:33:34.760206+0000","last_became_active":"2024-04-02T14:32:22.550292+0000","last_became_peered":"2024-04-02T14:32:22.550292+0000","last_unstale":"2024-04-02T14:33:34.760206+0000","last_undegraded":"2024-04-02T14:33:34.760206+0000","last_fullsized":"2024-04-02T14:33:34.760206+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:14:42.317649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"90'7","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:33:34.760461+0000","last_change":"2024-04-02T14:32:22.553042+0000","last_active":"2024-04-02T14:33:34.760461+0000","last_peered":"2024-04-02T14:33:34.760461+0000","last_clean":"2024-04-02T14:33:34.760461+0000","last_became_active":"2024-04-02T14:32:22.552010+0000","last_became_peered":"2024-04-02T14:32:22.552010+0000","last_unstale":"2024-04-02T14:33:34.760461+0000","last_undegraded":"2024-04-02T14:33:34.760461+0000","last_fullsized":"2024-04-02T14:33:34.760461+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:31:13.947662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"88'13","reported_seq":131,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:33:47.004057+0000","last_change":"2024-04-02T14:32:35.868597+0000","last_active":"2024-04-02T14:33:47.004057+0000","last_peered":"2024-04-02T14:33:47.004057+0000","last_clean":"2024-04-02T14:33:47.004057+0000","last_became_active":"2024-04-02T14:32:35.851962+0000","last_became_peered":"2024-04-02T14:32:35.851962+0000","last_unstale":"2024-04-02T14:33:47.004057+0000","last_undegraded":"2024-04-02T14:33:47.004057+0000","last_fullsized":"2024-04-02T14:33:47.004057+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:17:06.400744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":75,"num_read_kb":75,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":672,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,4],"acting":[9,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"90'5","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:33:34.760600+0000","last_change":"2024-04-02T14:32:22.549526+0000","last_active":"2024-04-02T14:33:34.760600+0000","last_peered":"2024-04-02T14:33:34.760600+0000","last_clean":"2024-04-02T14:33:34.760600+0000","last_became_active":"2024-04-02T14:32:22.548250+0000","last_became_peered":"2024-04-02T14:32:22.548250+0000","last_unstale":"2024-04-02T14:33:34.760600+0000","last_undegraded":"2024-04-02T14:33:34.760600+0000","last_fullsized":"2024-04-02T14:33:34.760600+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:35:34.866803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"88'2","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:32:35.737760+0000","last_change":"2024-04-02T14:32:21.865084+0000","last_active":"2024-04-02T14:32:35.737760+0000","last_peered":"2024-04-02T14:32:35.737760+0000","last_clean":"2024-04-02T14:32:35.737760+0000","last_became_active":"2024-04-02T14:32:21.864718+0000","last_became_peered":"2024-04-02T14:32:21.864718+0000","last_unstale":"2024-04-02T14:32:35.737760+0000","last_undegraded":"2024-04-02T14:32:35.737760+0000","last_fullsized":"2024-04-02T14:32:35.737760+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:10:50.611362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"90'6","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:33:34.772233+0000","last_change":"2024-04-02T14:32:22.549443+0000","last_active":"2024-04-02T14:33:34.772233+0000","last_peered":"2024-04-02T14:33:34.772233+0000","last_clean":"2024-04-02T14:33:34.772233+0000","last_became_active":"2024-04-02T14:32:22.549059+0000","last_became_peered":"2024-04-02T14:32:22.549059+0000","last_unstale":"2024-04-02T14:33:34.772233+0000","last_undegraded":"2024-04-02T14:33:34.772233+0000","last_fullsized":"2024-04-02T14:33:34.772233+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:04:21.489864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"88'12","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:32:35.731852+0000","last_change":"2024-04-02T14:32:21.841368+0000","last_active":"2024-04-02T14:32:35.731852+0000","last_peered":"2024-04-02T14:32:35.731852+0000","last_clean":"2024-04-02T14:32:35.731852+0000","last_became_active":"2024-04-02T14:32:21.840963+0000","last_became_peered":"2024-04-02T14:32:21.840963+0000","last_unstale":"2024-04-02T14:32:35.731852+0000","last_undegraded":"2024-04-02T14:32:35.731852+0000","last_fullsized":"2024-04-02T14:32:35.731852+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:36:40.549720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"90'9","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:33:34.769081+0000","last_change":"2024-04-02T14:32:22.549753+0000","last_active":"2024-04-02T14:33:34.769081+0000","last_peered":"2024-04-02T14:33:34.769081+0000","last_clean":"2024-04-02T14:33:34.769081+0000","last_became_active":"2024-04-02T14:32:22.549354+0000","last_became_peered":"2024-04-02T14:32:22.549354+0000","last_unstale":"2024-04-02T14:33:34.769081+0000","last_undegraded":"2024-04-02T14:33:34.769081+0000","last_fullsized":"2024-04-02T14:33:34.769081+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T02:17:09.650946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"88'5","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:32:35.731222+0000","last_change":"2024-04-02T14:32:21.842766+0000","last_active":"2024-04-02T14:32:35.731222+0000","last_peered":"2024-04-02T14:32:35.731222+0000","last_clean":"2024-04-02T14:32:35.731222+0000","last_became_active":"2024-04-02T14:32:21.842410+0000","last_became_peered":"2024-04-02T14:32:21.842410+0000","last_unstale":"2024-04-02T14:32:35.731222+0000","last_undegraded":"2024-04-02T14:32:35.731222+0000","last_fullsized":"2024-04-02T14:32:35.731222+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:49:31.651127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"90'15","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:33:34.748161+0000","last_change":"2024-04-02T14:32:22.538506+0000","last_active":"2024-04-02T14:33:34.748161+0000","last_peered":"2024-04-02T14:33:34.748161+0000","last_clean":"2024-04-02T14:33:34.748161+0000","last_became_active":"2024-04-02T14:32:22.538168+0000","last_became_peered":"2024-04-02T14:32:22.538168+0000","last_unstale":"2024-04-02T14:33:34.748161+0000","last_undegraded":"2024-04-02T14:33:34.748161+0000","last_fullsized":"2024-04-02T14:33:34.748161+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:30:04.746382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"90'7","reported_seq":107,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:33:45.937637+0000","last_change":"2024-04-02T14:32:21.844180+0000","last_active":"2024-04-02T14:33:45.937637+0000","last_peered":"2024-04-02T14:33:45.937637+0000","last_clean":"2024-04-02T14:33:45.937637+0000","last_became_active":"2024-04-02T14:32:21.843840+0000","last_became_peered":"2024-04-02T14:32:21.843840+0000","last_unstale":"2024-04-02T14:33:45.937637+0000","last_undegraded":"2024-04-02T14:33:45.937637+0000","last_fullsized":"2024-04-02T14:33:45.937637+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T01:56:10.767946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1154,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":3,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"90'7","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:33:34.737512+0000","last_change":"2024-04-02T14:32:22.548453+0000","last_active":"2024-04-02T14:33:34.737512+0000","last_peered":"2024-04-02T14:33:34.737512+0000","last_clean":"2024-04-02T14:33:34.737512+0000","last_became_active":"2024-04-02T14:32:22.548110+0000","last_became_peered":"2024-04-02T14:32:22.548110+0000","last_unstale":"2024-04-02T14:33:34.737512+0000","last_undegraded":"2024-04-02T14:33:34.737512+0000","last_fullsized":"2024-04-02T14:33:34.737512+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:57:13.067425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"88'7","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:32:35.724559+0000","last_change":"2024-04-02T14:32:21.846528+0000","last_active":"2024-04-02T14:32:35.724559+0000","last_peered":"2024-04-02T14:32:35.724559+0000","last_clean":"2024-04-02T14:32:35.724559+0000","last_became_active":"2024-04-02T14:32:21.846152+0000","last_became_peered":"2024-04-02T14:32:21.846152+0000","last_unstale":"2024-04-02T14:32:35.724559+0000","last_undegraded":"2024-04-02T14:32:35.724559+0000","last_fullsized":"2024-04-02T14:32:35.724559+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:58:09.286256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"90'8","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:33:34.758200+0000","last_change":"2024-04-02T14:32:22.546730+0000","last_active":"2024-04-02T14:33:34.758200+0000","last_peered":"2024-04-02T14:33:34.758200+0000","last_clean":"2024-04-02T14:33:34.758200+0000","last_became_active":"2024-04-02T14:32:22.546342+0000","last_became_peered":"2024-04-02T14:32:22.546342+0000","last_unstale":"2024-04-02T14:33:34.758200+0000","last_undegraded":"2024-04-02T14:33:34.758200+0000","last_fullsized":"2024-04-02T14:33:34.758200+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:38:37.616382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"88'3","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:32:35.845954+0000","last_change":"2024-04-02T14:32:21.853316+0000","last_active":"2024-04-02T14:32:35.845954+0000","last_peered":"2024-04-02T14:32:35.845954+0000","last_clean":"2024-04-02T14:32:35.845954+0000","last_became_active":"2024-04-02T14:32:21.852912+0000","last_became_peered":"2024-04-02T14:32:21.852912+0000","last_unstale":"2024-04-02T14:32:35.845954+0000","last_undegraded":"2024-04-02T14:32:35.845954+0000","last_fullsized":"2024-04-02T14:32:35.845954+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T02:30:44.601900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"90'8","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:33:34.762722+0000","last_change":"2024-04-02T14:32:22.558913+0000","last_active":"2024-04-02T14:33:34.762722+0000","last_peered":"2024-04-02T14:33:34.762722+0000","last_clean":"2024-04-02T14:33:34.762722+0000","last_became_active":"2024-04-02T14:32:22.558511+0000","last_became_peered":"2024-04-02T14:32:22.558511+0000","last_unstale":"2024-04-02T14:33:34.762722+0000","last_undegraded":"2024-04-02T14:33:34.762722+0000","last_fullsized":"2024-04-02T14:33:34.762722+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:48:36.449086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"88'3","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:32:35.730749+0000","last_change":"2024-04-02T14:32:21.844081+0000","last_active":"2024-04-02T14:32:35.730749+0000","last_peered":"2024-04-02T14:32:35.730749+0000","last_clean":"2024-04-02T14:32:35.730749+0000","last_became_active":"2024-04-02T14:32:21.843788+0000","last_became_peered":"2024-04-02T14:32:21.843788+0000","last_unstale":"2024-04-02T14:32:35.730749+0000","last_undegraded":"2024-04-02T14:32:35.730749+0000","last_fullsized":"2024-04-02T14:32:35.730749+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:19:02.184229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"90'11","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:33:34.768194+0000","last_change":"2024-04-02T14:32:22.556193+0000","last_active":"2024-04-02T14:33:34.768194+0000","last_peered":"2024-04-02T14:33:34.768194+0000","last_clean":"2024-04-02T14:33:34.768194+0000","last_became_active":"2024-04-02T14:32:22.555787+0000","last_became_peered":"2024-04-02T14:32:22.555787+0000","last_unstale":"2024-04-02T14:33:34.768194+0000","last_undegraded":"2024-04-02T14:33:34.768194+0000","last_fullsized":"2024-04-02T14:33:34.768194+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T02:02:13.730941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"90'5","reported_seq":184,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:33:48.941174+0000","last_change":"2024-04-02T14:32:21.840651+0000","last_active":"2024-04-02T14:33:48.941174+0000","last_peered":"2024-04-02T14:33:48.941174+0000","last_clean":"2024-04-02T14:33:48.941174+0000","last_became_active":"2024-04-02T14:32:21.840344+0000","last_became_peered":"2024-04-02T14:32:21.840344+0000","last_unstale":"2024-04-02T14:33:48.941174+0000","last_undegraded":"2024-04-02T14:33:48.941174+0000","last_fullsized":"2024-04-02T14:33:48.941174+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:49:22.384224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":160,"num_read_kb":160,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"90'10","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:33:34.763370+0000","last_change":"2024-04-02T14:32:22.548245+0000","last_active":"2024-04-02T14:33:34.763370+0000","last_peered":"2024-04-02T14:33:34.763370+0000","last_clean":"2024-04-02T14:33:34.763370+0000","last_became_active":"2024-04-02T14:32:22.547853+0000","last_became_peered":"2024-04-02T14:32:22.547853+0000","last_unstale":"2024-04-02T14:33:34.763370+0000","last_undegraded":"2024-04-02T14:33:34.763370+0000","last_fullsized":"2024-04-02T14:33:34.763370+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:02:43.036944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"88'4","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:32:35.730780+0000","last_change":"2024-04-02T14:32:21.842096+0000","last_active":"2024-04-02T14:32:35.730780+0000","last_peered":"2024-04-02T14:32:35.730780+0000","last_clean":"2024-04-02T14:32:35.730780+0000","last_became_active":"2024-04-02T14:32:21.841646+0000","last_became_peered":"2024-04-02T14:32:21.841646+0000","last_unstale":"2024-04-02T14:32:35.730780+0000","last_undegraded":"2024-04-02T14:32:35.730780+0000","last_fullsized":"2024-04-02T14:32:35.730780+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:04:08.149785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"90'9","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:33:34.772870+0000","last_change":"2024-04-02T14:32:22.557694+0000","last_active":"2024-04-02T14:33:34.772870+0000","last_peered":"2024-04-02T14:33:34.772870+0000","last_clean":"2024-04-02T14:33:34.772870+0000","last_became_active":"2024-04-02T14:32:22.557202+0000","last_became_peered":"2024-04-02T14:32:22.557202+0000","last_unstale":"2024-04-02T14:33:34.772870+0000","last_undegraded":"2024-04-02T14:33:34.772870+0000","last_fullsized":"2024-04-02T14:33:34.772870+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:35:41.471557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"88'5","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:32:35.731380+0000","last_change":"2024-04-02T14:32:21.864366+0000","last_active":"2024-04-02T14:32:35.731380+0000","last_peered":"2024-04-02T14:32:35.731380+0000","last_clean":"2024-04-02T14:32:35.731380+0000","last_became_active":"2024-04-02T14:32:21.864006+0000","last_became_peered":"2024-04-02T14:32:21.864006+0000","last_unstale":"2024-04-02T14:32:35.731380+0000","last_undegraded":"2024-04-02T14:32:35.731380+0000","last_fullsized":"2024-04-02T14:32:35.731380+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:08:43.176061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"90'11","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:33:34.751922+0000","last_change":"2024-04-02T14:32:22.541562+0000","last_active":"2024-04-02T14:33:34.751922+0000","last_peered":"2024-04-02T14:33:34.751922+0000","last_clean":"2024-04-02T14:33:34.751922+0000","last_became_active":"2024-04-02T14:32:22.541209+0000","last_became_peered":"2024-04-02T14:32:22.541209+0000","last_unstale":"2024-04-02T14:33:34.751922+0000","last_undegraded":"2024-04-02T14:33:34.751922+0000","last_fullsized":"2024-04-02T14:33:34.751922+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:49:40.310528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"90'33","reported_seq":238,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:33:48.965101+0000","last_change":"2024-04-02T14:32:35.875429+0000","last_active":"2024-04-02T14:33:48.965101+0000","last_peered":"2024-04-02T14:33:48.965101+0000","last_clean":"2024-04-02T14:33:48.965101+0000","last_became_active":"2024-04-02T14:32:35.852417+0000","last_became_peered":"2024-04-02T14:32:35.852417+0000","last_unstale":"2024-04-02T14:33:48.965101+0000","last_undegraded":"2024-04-02T14:33:48.965101+0000","last_fullsized":"2024-04-02T14:33:48.965101+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:33:53.903712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":61295,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":159,"num_read_kb":134,"num_write":35,"num_write_kb":72,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":43377,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,4],"acting":[10,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"90'6","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:33:34.740087+0000","last_change":"2024-04-02T14:32:22.535716+0000","last_active":"2024-04-02T14:33:34.740087+0000","last_peered":"2024-04-02T14:33:34.740087+0000","last_clean":"2024-04-02T14:33:34.740087+0000","last_became_active":"2024-04-02T14:32:22.534738+0000","last_became_peered":"2024-04-02T14:32:22.534738+0000","last_unstale":"2024-04-02T14:33:34.740087+0000","last_undegraded":"2024-04-02T14:33:34.740087+0000","last_fullsized":"2024-04-02T14:33:34.740087+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T01:25:07.763915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"88'9","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:32:35.726311+0000","last_change":"2024-04-02T14:32:21.854582+0000","last_active":"2024-04-02T14:32:35.726311+0000","last_peered":"2024-04-02T14:32:35.726311+0000","last_clean":"2024-04-02T14:32:35.726311+0000","last_became_active":"2024-04-02T14:32:21.854226+0000","last_became_peered":"2024-04-02T14:32:21.854226+0000","last_unstale":"2024-04-02T14:32:35.726311+0000","last_undegraded":"2024-04-02T14:32:35.726311+0000","last_fullsized":"2024-04-02T14:32:35.726311+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:30:02.289163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"90'9","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:33:34.762399+0000","last_change":"2024-04-02T14:32:22.555901+0000","last_active":"2024-04-02T14:33:34.762399+0000","last_peered":"2024-04-02T14:33:34.762399+0000","last_clean":"2024-04-02T14:33:34.762399+0000","last_became_active":"2024-04-02T14:32:22.555158+0000","last_became_peered":"2024-04-02T14:32:22.555158+0000","last_unstale":"2024-04-02T14:33:34.762399+0000","last_undegraded":"2024-04-02T14:33:34.762399+0000","last_fullsized":"2024-04-02T14:33:34.762399+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:04:28.766185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"88'5","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:32:35.730716+0000","last_change":"2024-04-02T14:32:21.855982+0000","last_active":"2024-04-02T14:32:35.730716+0000","last_peered":"2024-04-02T14:32:35.730716+0000","last_clean":"2024-04-02T14:32:35.730716+0000","last_became_active":"2024-04-02T14:32:21.855591+0000","last_became_peered":"2024-04-02T14:32:21.855591+0000","last_unstale":"2024-04-02T14:32:35.730716+0000","last_undegraded":"2024-04-02T14:32:35.730716+0000","last_fullsized":"2024-04-02T14:32:35.730716+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:28:51.806779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"90'15","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:33:34.767837+0000","last_change":"2024-04-02T14:32:22.549889+0000","last_active":"2024-04-02T14:33:34.767837+0000","last_peered":"2024-04-02T14:33:34.767837+0000","last_clean":"2024-04-02T14:33:34.767837+0000","last_became_active":"2024-04-02T14:32:22.549479+0000","last_became_peered":"2024-04-02T14:32:22.549479+0000","last_unstale":"2024-04-02T14:33:34.767837+0000","last_undegraded":"2024-04-02T14:33:34.767837+0000","last_fullsized":"2024-04-02T14:33:34.767837+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:29:25.937576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"90'7","reported_seq":109,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:33:48.014587+0000","last_change":"2024-04-02T14:32:21.838165+0000","last_active":"2024-04-02T14:33:48.014587+0000","last_peered":"2024-04-02T14:33:48.014587+0000","last_clean":"2024-04-02T14:33:48.014587+0000","last_became_active":"2024-04-02T14:32:21.837808+0000","last_became_peered":"2024-04-02T14:32:21.837808+0000","last_unstale":"2024-04-02T14:33:48.014587+0000","last_undegraded":"2024-04-02T14:33:48.014587+0000","last_fullsized":"2024-04-02T14:33:48.014587+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:00:44.605314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1120,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":84,"num_read_kb":4,"num_write":7,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"90'6","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:33:34.745892+0000","last_change":"2024-04-02T14:32:22.532937+0000","last_active":"2024-04-02T14:33:34.745892+0000","last_peered":"2024-04-02T14:33:34.745892+0000","last_clean":"2024-04-02T14:33:34.745892+0000","last_became_active":"2024-04-02T14:32:22.532586+0000","last_became_peered":"2024-04-02T14:32:22.532586+0000","last_unstale":"2024-04-02T14:33:34.745892+0000","last_undegraded":"2024-04-02T14:33:34.745892+0000","last_fullsized":"2024-04-02T14:33:34.745892+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:50:48.030889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"88'17","reported_seq":37,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:32:35.731828+0000","last_change":"2024-04-02T14:32:21.860032+0000","last_active":"2024-04-02T14:32:35.731828+0000","last_peered":"2024-04-02T14:32:35.731828+0000","last_clean":"2024-04-02T14:32:35.731828+0000","last_became_active":"2024-04-02T14:32:21.859653+0000","last_became_peered":"2024-04-02T14:32:21.859653+0000","last_unstale":"2024-04-02T14:32:35.731828+0000","last_undegraded":"2024-04-02T14:32:35.731828+0000","last_fullsized":"2024-04-02T14:32:35.731828+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:06:01.144530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"90'6","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:33:34.761794+0000","last_change":"2024-04-02T14:32:35.736133+0000","last_active":"2024-04-02T14:33:34.761794+0000","last_peered":"2024-04-02T14:33:34.761794+0000","last_clean":"2024-04-02T14:33:34.761794+0000","last_became_active":"2024-04-02T14:32:35.734335+0000","last_became_peered":"2024-04-02T14:32:35.734335+0000","last_unstale":"2024-04-02T14:33:34.761794+0000","last_undegraded":"2024-04-02T14:33:34.761794+0000","last_fullsized":"2024-04-02T14:33:34.761794+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:36:21.828318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,1],"acting":[9,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.f","version":"88'2","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:32:35.731999+0000","last_change":"2024-04-02T14:32:21.842384+0000","last_active":"2024-04-02T14:32:35.731999+0000","last_peered":"2024-04-02T14:32:35.731999+0000","last_clean":"2024-04-02T14:32:35.731999+0000","last_became_active":"2024-04-02T14:32:21.842040+0000","last_became_peered":"2024-04-02T14:32:21.842040+0000","last_unstale":"2024-04-02T14:32:35.731999+0000","last_undegraded":"2024-04-02T14:32:35.731999+0000","last_fullsized":"2024-04-02T14:32:35.731999+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:33:29.541972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"90'5","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:33:34.748788+0000","last_change":"2024-04-02T14:32:22.538571+0000","last_active":"2024-04-02T14:33:34.748788+0000","last_peered":"2024-04-02T14:33:34.748788+0000","last_clean":"2024-04-02T14:33:34.748788+0000","last_became_active":"2024-04-02T14:32:22.538239+0000","last_became_peered":"2024-04-02T14:32:22.538239+0000","last_unstale":"2024-04-02T14:33:34.748788+0000","last_undegraded":"2024-04-02T14:33:34.748788+0000","last_fullsized":"2024-04-02T14:33:34.748788+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T01:19:53.120419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"88'1","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:32:35.726314+0000","last_change":"2024-04-02T14:32:21.857666+0000","last_active":"2024-04-02T14:32:35.726314+0000","last_peered":"2024-04-02T14:32:35.726314+0000","last_clean":"2024-04-02T14:32:35.726314+0000","last_became_active":"2024-04-02T14:32:21.856618+0000","last_became_peered":"2024-04-02T14:32:21.856618+0000","last_unstale":"2024-04-02T14:32:35.726314+0000","last_undegraded":"2024-04-02T14:32:35.726314+0000","last_fullsized":"2024-04-02T14:32:35.726314+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T01:20:53.183052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"90'5","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:33:34.758638+0000","last_change":"2024-04-02T14:32:22.552277+0000","last_active":"2024-04-02T14:33:34.758638+0000","last_peered":"2024-04-02T14:33:34.758638+0000","last_clean":"2024-04-02T14:33:34.758638+0000","last_became_active":"2024-04-02T14:32:22.551882+0000","last_became_peered":"2024-04-02T14:32:22.551882+0000","last_unstale":"2024-04-02T14:33:34.758638+0000","last_undegraded":"2024-04-02T14:33:34.758638+0000","last_fullsized":"2024-04-02T14:33:34.758638+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:30:43.592910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"88'5","reported_seq":106,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:33:50.012602+0000","last_change":"2024-04-02T14:32:21.848258+0000","last_active":"2024-04-02T14:33:50.012602+0000","last_peered":"2024-04-02T14:33:50.012602+0000","last_clean":"2024-04-02T14:33:50.012602+0000","last_became_active":"2024-04-02T14:32:21.847805+0000","last_became_peered":"2024-04-02T14:32:21.847805+0000","last_unstale":"2024-04-02T14:33:50.012602+0000","last_undegraded":"2024-04-02T14:33:50.012602+0000","last_fullsized":"2024-04-02T14:33:50.012602+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:46:07.552786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":82,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"90'7","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:33:34.760673+0000","last_change":"2024-04-02T14:32:22.560835+0000","last_active":"2024-04-02T14:33:34.760673+0000","last_peered":"2024-04-02T14:33:34.760673+0000","last_clean":"2024-04-02T14:33:34.760673+0000","last_became_active":"2024-04-02T14:32:22.560469+0000","last_became_peered":"2024-04-02T14:32:22.560469+0000","last_unstale":"2024-04-02T14:33:34.760673+0000","last_undegraded":"2024-04-02T14:33:34.760673+0000","last_fullsized":"2024-04-02T14:33:34.760673+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T02:29:24.869650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"90'8","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:33:34.742608+0000","last_change":"2024-04-02T14:32:22.538325+0000","last_active":"2024-04-02T14:33:34.742608+0000","last_peered":"2024-04-02T14:33:34.742608+0000","last_clean":"2024-04-02T14:33:34.742608+0000","last_became_active":"2024-04-02T14:32:22.537903+0000","last_became_peered":"2024-04-02T14:32:22.537903+0000","last_unstale":"2024-04-02T14:33:34.742608+0000","last_undegraded":"2024-04-02T14:33:34.742608+0000","last_fullsized":"2024-04-02T14:33:34.742608+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:12:26.586290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"90'6","reported_seq":108,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:33:50.018302+0000","last_change":"2024-04-02T14:32:21.861333+0000","last_active":"2024-04-02T14:33:50.018302+0000","last_peered":"2024-04-02T14:33:50.018302+0000","last_clean":"2024-04-02T14:33:50.018302+0000","last_became_active":"2024-04-02T14:32:21.860909+0000","last_became_peered":"2024-04-02T14:32:21.860909+0000","last_unstale":"2024-04-02T14:33:50.018302+0000","last_undegraded":"2024-04-02T14:33:50.018302+0000","last_fullsized":"2024-04-02T14:33:50.018302+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:21:52.616036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":83,"num_read_kb":83,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"90'8","reported_seq":19,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:33:34.758306+0000","last_change":"2024-04-02T14:32:35.736379+0000","last_active":"2024-04-02T14:33:34.758306+0000","last_peered":"2024-04-02T14:33:34.758306+0000","last_clean":"2024-04-02T14:33:34.758306+0000","last_became_active":"2024-04-02T14:32:35.735156+0000","last_became_peered":"2024-04-02T14:32:35.735156+0000","last_unstale":"2024-04-02T14:33:34.758306+0000","last_undegraded":"2024-04-02T14:33:34.758306+0000","last_fullsized":"2024-04-02T14:33:34.758306+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:16:28.405699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,1],"acting":[9,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.3","version":"88'6","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:32:35.850702+0000","last_change":"2024-04-02T14:32:21.837336+0000","last_active":"2024-04-02T14:32:35.850702+0000","last_peered":"2024-04-02T14:32:35.850702+0000","last_clean":"2024-04-02T14:32:35.850702+0000","last_became_active":"2024-04-02T14:32:21.836921+0000","last_became_peered":"2024-04-02T14:32:21.836921+0000","last_unstale":"2024-04-02T14:32:35.850702+0000","last_undegraded":"2024-04-02T14:32:35.850702+0000","last_fullsized":"2024-04-02T14:32:35.850702+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:34:52.883173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":356,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:32:35.730675+0000","last_change":"2024-04-02T14:31:02.710356+0000","last_active":"2024-04-02T14:32:35.730675+0000","last_peered":"2024-04-02T14:32:35.730675+0000","last_clean":"2024-04-02T14:32:35.730675+0000","last_became_active":"2024-04-02T14:31:02.709669+0000","last_became_peered":"2024-04-02T14:31:02.709669+0000","last_unstale":"2024-04-02T14:32:35.730675+0000","last_undegraded":"2024-04-02T14:32:35.730675+0000","last_fullsized":"2024-04-02T14:32:35.730675+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:27:17.358883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:27:17.358883+0000","last_clean_scrub_stamp":"2024-04-02T14:27:17.358883+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:45:55.893284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"90'5","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:33:34.740733+0000","last_change":"2024-04-02T14:32:22.545502+0000","last_active":"2024-04-02T14:33:34.740733+0000","last_peered":"2024-04-02T14:33:34.740733+0000","last_clean":"2024-04-02T14:33:34.740733+0000","last_became_active":"2024-04-02T14:32:22.545111+0000","last_became_peered":"2024-04-02T14:32:22.545111+0000","last_unstale":"2024-04-02T14:33:34.740733+0000","last_undegraded":"2024-04-02T14:33:34.740733+0000","last_fullsized":"2024-04-02T14:33:34.740733+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:02:32.178334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"88'4","reported_seq":15,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:32:35.736582+0000","last_change":"2024-04-02T14:32:35.736582+0000","last_active":"2024-04-02T14:32:35.736582+0000","last_peered":"2024-04-02T14:32:35.736582+0000","last_clean":"2024-04-02T14:32:35.736582+0000","last_became_active":"2024-04-02T14:32:35.733368+0000","last_became_peered":"2024-04-02T14:32:35.733368+0000","last_unstale":"2024-04-02T14:32:35.736582+0000","last_undegraded":"2024-04-02T14:32:35.736582+0000","last_fullsized":"2024-04-02T14:32:35.736582+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T02:14:35.713351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,8],"acting":[11,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"88'4","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:32:35.731654+0000","last_change":"2024-04-02T14:32:21.860226+0000","last_active":"2024-04-02T14:32:35.731654+0000","last_peered":"2024-04-02T14:32:35.731654+0000","last_clean":"2024-04-02T14:32:35.731654+0000","last_became_active":"2024-04-02T14:32:21.859854+0000","last_became_peered":"2024-04-02T14:32:21.859854+0000","last_unstale":"2024-04-02T14:32:35.731654+0000","last_undegraded":"2024-04-02T14:32:35.731654+0000","last_fullsized":"2024-04-02T14:32:35.731654+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:30:36.710677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"90'9","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:33:34.746141+0000","last_change":"2024-04-02T14:32:22.552287+0000","last_active":"2024-04-02T14:33:34.746141+0000","last_peered":"2024-04-02T14:33:34.746141+0000","last_clean":"2024-04-02T14:33:34.746141+0000","last_became_active":"2024-04-02T14:32:22.551719+0000","last_became_peered":"2024-04-02T14:32:22.551719+0000","last_unstale":"2024-04-02T14:33:34.746141+0000","last_undegraded":"2024-04-02T14:33:34.746141+0000","last_fullsized":"2024-04-02T14:33:34.746141+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:26:12.007600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"90'14","reported_seq":33,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:32:35.856522+0000","last_change":"2024-04-02T14:32:21.851671+0000","last_active":"2024-04-02T14:32:35.856522+0000","last_peered":"2024-04-02T14:32:35.856522+0000","last_clean":"2024-04-02T14:32:35.856522+0000","last_became_active":"2024-04-02T14:32:21.851297+0000","last_became_peered":"2024-04-02T14:32:21.851297+0000","last_unstale":"2024-04-02T14:32:35.856522+0000","last_undegraded":"2024-04-02T14:32:35.856522+0000","last_fullsized":"2024-04-02T14:32:35.856522+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:51:20.232086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"90'5","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:33:34.762221+0000","last_change":"2024-04-02T14:32:22.554554+0000","last_active":"2024-04-02T14:33:34.762221+0000","last_peered":"2024-04-02T14:33:34.762221+0000","last_clean":"2024-04-02T14:33:34.762221+0000","last_became_active":"2024-04-02T14:32:22.554194+0000","last_became_peered":"2024-04-02T14:32:22.554194+0000","last_unstale":"2024-04-02T14:33:34.762221+0000","last_undegraded":"2024-04-02T14:33:34.762221+0000","last_fullsized":"2024-04-02T14:33:34.762221+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:27:51.382441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"88'7","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:32:35.850845+0000","last_change":"2024-04-02T14:32:21.850346+0000","last_active":"2024-04-02T14:32:35.850845+0000","last_peered":"2024-04-02T14:32:35.850845+0000","last_clean":"2024-04-02T14:32:35.850845+0000","last_became_active":"2024-04-02T14:32:21.850084+0000","last_became_peered":"2024-04-02T14:32:21.850084+0000","last_unstale":"2024-04-02T14:32:35.850845+0000","last_undegraded":"2024-04-02T14:32:35.850845+0000","last_fullsized":"2024-04-02T14:32:35.850845+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:56:17.774702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"90'7","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:33:34.768581+0000","last_change":"2024-04-02T14:32:22.557054+0000","last_active":"2024-04-02T14:33:34.768581+0000","last_peered":"2024-04-02T14:33:34.768581+0000","last_clean":"2024-04-02T14:33:34.768581+0000","last_became_active":"2024-04-02T14:32:22.556335+0000","last_became_peered":"2024-04-02T14:32:22.556335+0000","last_unstale":"2024-04-02T14:33:34.768581+0000","last_undegraded":"2024-04-02T14:33:34.768581+0000","last_fullsized":"2024-04-02T14:33:34.768581+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:06:48.907197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"88'2","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:32:35.722876+0000","last_change":"2024-04-02T14:32:21.860260+0000","last_active":"2024-04-02T14:32:35.722876+0000","last_peered":"2024-04-02T14:32:35.722876+0000","last_clean":"2024-04-02T14:32:35.722876+0000","last_became_active":"2024-04-02T14:32:21.859922+0000","last_became_peered":"2024-04-02T14:32:21.859922+0000","last_unstale":"2024-04-02T14:32:35.722876+0000","last_undegraded":"2024-04-02T14:32:35.722876+0000","last_fullsized":"2024-04-02T14:32:35.722876+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:07:45.850148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"90'2","reported_seq":19,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:33:34.707329+0000","last_change":"2024-04-02T14:32:22.536063+0000","last_active":"2024-04-02T14:33:34.707329+0000","last_peered":"2024-04-02T14:33:34.707329+0000","last_clean":"2024-04-02T14:33:34.707329+0000","last_became_active":"2024-04-02T14:32:22.534678+0000","last_became_peered":"2024-04-02T14:32:22.534678+0000","last_unstale":"2024-04-02T14:33:34.707329+0000","last_undegraded":"2024-04-02T14:33:34.707329+0000","last_fullsized":"2024-04-02T14:33:34.707329+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T02:07:26.186216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"88'7","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:32:35.736679+0000","last_change":"2024-04-02T14:32:21.848554+0000","last_active":"2024-04-02T14:32:35.736679+0000","last_peered":"2024-04-02T14:32:35.736679+0000","last_clean":"2024-04-02T14:32:35.736679+0000","last_became_active":"2024-04-02T14:32:21.848177+0000","last_became_peered":"2024-04-02T14:32:21.848177+0000","last_unstale":"2024-04-02T14:32:35.736679+0000","last_undegraded":"2024-04-02T14:32:35.736679+0000","last_fullsized":"2024-04-02T14:32:35.736679+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:47:04.213286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"90'9","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:33:34.763600+0000","last_change":"2024-04-02T14:32:22.547246+0000","last_active":"2024-04-02T14:33:34.763600+0000","last_peered":"2024-04-02T14:33:34.763600+0000","last_clean":"2024-04-02T14:33:34.763600+0000","last_became_active":"2024-04-02T14:32:22.546721+0000","last_became_peered":"2024-04-02T14:32:22.546721+0000","last_unstale":"2024-04-02T14:33:34.763600+0000","last_undegraded":"2024-04-02T14:33:34.763600+0000","last_fullsized":"2024-04-02T14:33:34.763600+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:11:30.615723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"88'3","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:32:35.743474+0000","last_change":"2024-04-02T14:32:35.743474+0000","last_active":"2024-04-02T14:32:35.743474+0000","last_peered":"2024-04-02T14:32:35.743474+0000","last_clean":"2024-04-02T14:32:35.743474+0000","last_became_active":"2024-04-02T14:32:35.738522+0000","last_became_peered":"2024-04-02T14:32:35.738522+0000","last_unstale":"2024-04-02T14:32:35.743474+0000","last_undegraded":"2024-04-02T14:32:35.743474+0000","last_fullsized":"2024-04-02T14:32:35.743474+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T01:14:43.989279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"90'8","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:33:34.766374+0000","last_change":"2024-04-02T14:32:22.561787+0000","last_active":"2024-04-02T14:33:34.766374+0000","last_peered":"2024-04-02T14:33:34.766374+0000","last_clean":"2024-04-02T14:33:34.766374+0000","last_became_active":"2024-04-02T14:32:22.561284+0000","last_became_peered":"2024-04-02T14:32:22.561284+0000","last_unstale":"2024-04-02T14:33:34.766374+0000","last_undegraded":"2024-04-02T14:33:34.766374+0000","last_fullsized":"2024-04-02T14:33:34.766374+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:46:06.450097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"88'4","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:32:35.731779+0000","last_change":"2024-04-02T14:32:21.841340+0000","last_active":"2024-04-02T14:32:35.731779+0000","last_peered":"2024-04-02T14:32:35.731779+0000","last_clean":"2024-04-02T14:32:35.731779+0000","last_became_active":"2024-04-02T14:32:21.840937+0000","last_became_peered":"2024-04-02T14:32:21.840937+0000","last_unstale":"2024-04-02T14:32:35.731779+0000","last_undegraded":"2024-04-02T14:32:35.731779+0000","last_fullsized":"2024-04-02T14:32:35.731779+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T01:47:44.834440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"90'5","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:33:34.767552+0000","last_change":"2024-04-02T14:32:22.538890+0000","last_active":"2024-04-02T14:33:34.767552+0000","last_peered":"2024-04-02T14:33:34.767552+0000","last_clean":"2024-04-02T14:33:34.767552+0000","last_became_active":"2024-04-02T14:32:22.538512+0000","last_became_peered":"2024-04-02T14:32:22.538512+0000","last_unstale":"2024-04-02T14:33:34.767552+0000","last_undegraded":"2024-04-02T14:33:34.767552+0000","last_fullsized":"2024-04-02T14:33:34.767552+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:59:23.792851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"88'5","reported_seq":108,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:33:49.943622+0000","last_change":"2024-04-02T14:32:21.862727+0000","last_active":"2024-04-02T14:33:49.943622+0000","last_peered":"2024-04-02T14:33:49.943622+0000","last_clean":"2024-04-02T14:33:49.943622+0000","last_became_active":"2024-04-02T14:32:21.862405+0000","last_became_peered":"2024-04-02T14:32:21.862405+0000","last_unstale":"2024-04-02T14:33:49.943622+0000","last_undegraded":"2024-04-02T14:33:49.943622+0000","last_fullsized":"2024-04-02T14:33:49.943622+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:36:54.849964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":84,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"90'7","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:33:34.756023+0000","last_change":"2024-04-02T14:32:22.540077+0000","last_active":"2024-04-02T14:33:34.756023+0000","last_peered":"2024-04-02T14:33:34.756023+0000","last_clean":"2024-04-02T14:33:34.756023+0000","last_became_active":"2024-04-02T14:32:22.539513+0000","last_became_peered":"2024-04-02T14:32:22.539513+0000","last_unstale":"2024-04-02T14:33:34.756023+0000","last_undegraded":"2024-04-02T14:33:34.756023+0000","last_fullsized":"2024-04-02T14:33:34.756023+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T02:00:39.819963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"88'3","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:32:35.731129+0000","last_change":"2024-04-02T14:32:21.860310+0000","last_active":"2024-04-02T14:32:35.731129+0000","last_peered":"2024-04-02T14:32:35.731129+0000","last_clean":"2024-04-02T14:32:35.731129+0000","last_became_active":"2024-04-02T14:32:21.859955+0000","last_became_peered":"2024-04-02T14:32:21.859955+0000","last_unstale":"2024-04-02T14:32:35.731129+0000","last_undegraded":"2024-04-02T14:32:35.731129+0000","last_fullsized":"2024-04-02T14:32:35.731129+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:11:32.732603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"90'7","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:33:34.760162+0000","last_change":"2024-04-02T14:32:22.552134+0000","last_active":"2024-04-02T14:33:34.760162+0000","last_peered":"2024-04-02T14:33:34.760162+0000","last_clean":"2024-04-02T14:33:34.760162+0000","last_became_active":"2024-04-02T14:32:22.551695+0000","last_became_peered":"2024-04-02T14:32:22.551695+0000","last_unstale":"2024-04-02T14:33:34.760162+0000","last_undegraded":"2024-04-02T14:33:34.760162+0000","last_fullsized":"2024-04-02T14:33:34.760162+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:09:36.902243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"88'5","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:32:35.845910+0000","last_change":"2024-04-02T14:32:21.857571+0000","last_active":"2024-04-02T14:32:35.845910+0000","last_peered":"2024-04-02T14:32:35.845910+0000","last_clean":"2024-04-02T14:32:35.845910+0000","last_became_active":"2024-04-02T14:32:21.856935+0000","last_became_peered":"2024-04-02T14:32:21.856935+0000","last_unstale":"2024-04-02T14:32:35.845910+0000","last_undegraded":"2024-04-02T14:32:35.845910+0000","last_fullsized":"2024-04-02T14:32:35.845910+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:55:46.079097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"90'9","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:33:34.741354+0000","last_change":"2024-04-02T14:32:22.534708+0000","last_active":"2024-04-02T14:33:34.741354+0000","last_peered":"2024-04-02T14:33:34.741354+0000","last_clean":"2024-04-02T14:33:34.741354+0000","last_became_active":"2024-04-02T14:32:22.534365+0000","last_became_peered":"2024-04-02T14:32:22.534365+0000","last_unstale":"2024-04-02T14:33:34.741354+0000","last_undegraded":"2024-04-02T14:33:34.741354+0000","last_fullsized":"2024-04-02T14:33:34.741354+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:34:47.435729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"90'9","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:33:34.745155+0000","last_change":"2024-04-02T14:32:22.538317+0000","last_active":"2024-04-02T14:33:34.745155+0000","last_peered":"2024-04-02T14:33:34.745155+0000","last_clean":"2024-04-02T14:33:34.745155+0000","last_became_active":"2024-04-02T14:32:22.537939+0000","last_became_peered":"2024-04-02T14:32:22.537939+0000","last_unstale":"2024-04-02T14:33:34.745155+0000","last_undegraded":"2024-04-02T14:33:34.745155+0000","last_fullsized":"2024-04-02T14:33:34.745155+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:31:47.075326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"90'12","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:33:34.747907+0000","last_change":"2024-04-02T14:32:35.852202+0000","last_active":"2024-04-02T14:33:34.747907+0000","last_peered":"2024-04-02T14:33:34.747907+0000","last_clean":"2024-04-02T14:33:34.747907+0000","last_became_active":"2024-04-02T14:32:35.851785+0000","last_became_peered":"2024-04-02T14:32:35.851785+0000","last_unstale":"2024-04-02T14:33:34.747907+0000","last_undegraded":"2024-04-02T14:33:34.747907+0000","last_fullsized":"2024-04-02T14:33:34.747907+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:09:17.410727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,4],"acting":[2,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"90'8","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:33:34.765856+0000","last_change":"2024-04-02T14:32:22.556567+0000","last_active":"2024-04-02T14:33:34.765856+0000","last_peered":"2024-04-02T14:33:34.765856+0000","last_clean":"2024-04-02T14:33:34.765856+0000","last_became_active":"2024-04-02T14:32:22.556267+0000","last_became_peered":"2024-04-02T14:32:22.556267+0000","last_unstale":"2024-04-02T14:33:34.765856+0000","last_undegraded":"2024-04-02T14:33:34.765856+0000","last_fullsized":"2024-04-02T14:33:34.765856+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:35:30.039013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"90'7","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:33:34.772435+0000","last_change":"2024-04-02T14:32:35.848125+0000","last_active":"2024-04-02T14:33:34.772435+0000","last_peered":"2024-04-02T14:33:34.772435+0000","last_clean":"2024-04-02T14:33:34.772435+0000","last_became_active":"2024-04-02T14:32:35.847681+0000","last_became_peered":"2024-04-02T14:32:35.847681+0000","last_unstale":"2024-04-02T14:33:34.772435+0000","last_undegraded":"2024-04-02T14:33:34.772435+0000","last_fullsized":"2024-04-02T14:33:34.772435+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:51:38.040261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"90'4","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:33:34.766623+0000","last_change":"2024-04-02T14:32:22.558584+0000","last_active":"2024-04-02T14:33:34.766623+0000","last_peered":"2024-04-02T14:33:34.766623+0000","last_clean":"2024-04-02T14:33:34.766623+0000","last_became_active":"2024-04-02T14:32:22.557382+0000","last_became_peered":"2024-04-02T14:32:22.557382+0000","last_unstale":"2024-04-02T14:33:34.766623+0000","last_undegraded":"2024-04-02T14:33:34.766623+0000","last_fullsized":"2024-04-02T14:33:34.766623+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:09:25.107411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"90'6","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:33:34.740344+0000","last_change":"2024-04-02T14:32:22.538318+0000","last_active":"2024-04-02T14:33:34.740344+0000","last_peered":"2024-04-02T14:33:34.740344+0000","last_clean":"2024-04-02T14:33:34.740344+0000","last_became_active":"2024-04-02T14:32:22.538011+0000","last_became_peered":"2024-04-02T14:32:22.538011+0000","last_unstale":"2024-04-02T14:33:34.740344+0000","last_undegraded":"2024-04-02T14:33:34.740344+0000","last_fullsized":"2024-04-02T14:33:34.740344+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:11:21.269106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"90'15","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:33:34.735078+0000","last_change":"2024-04-02T14:32:22.535645+0000","last_active":"2024-04-02T14:33:34.735078+0000","last_peered":"2024-04-02T14:33:34.735078+0000","last_clean":"2024-04-02T14:33:34.735078+0000","last_became_active":"2024-04-02T14:32:22.535314+0000","last_became_peered":"2024-04-02T14:32:22.535314+0000","last_unstale":"2024-04-02T14:33:34.735078+0000","last_undegraded":"2024-04-02T14:33:34.735078+0000","last_fullsized":"2024-04-02T14:33:34.735078+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:51:27.879015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"90'2","reported_seq":19,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:33:34.760189+0000","last_change":"2024-04-02T14:32:22.553959+0000","last_active":"2024-04-02T14:33:34.760189+0000","last_peered":"2024-04-02T14:33:34.760189+0000","last_clean":"2024-04-02T14:33:34.760189+0000","last_became_active":"2024-04-02T14:32:22.553389+0000","last_became_peered":"2024-04-02T14:32:22.553389+0000","last_unstale":"2024-04-02T14:33:34.760189+0000","last_undegraded":"2024-04-02T14:33:34.760189+0000","last_fullsized":"2024-04-02T14:33:34.760189+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:55:00.552600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"90'6","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:33:34.764651+0000","last_change":"2024-04-02T14:32:22.551769+0000","last_active":"2024-04-02T14:33:34.764651+0000","last_peered":"2024-04-02T14:33:34.764651+0000","last_clean":"2024-04-02T14:33:34.764651+0000","last_became_active":"2024-04-02T14:32:22.551422+0000","last_became_peered":"2024-04-02T14:32:22.551422+0000","last_unstale":"2024-04-02T14:33:34.764651+0000","last_undegraded":"2024-04-02T14:33:34.764651+0000","last_fullsized":"2024-04-02T14:33:34.764651+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T01:15:07.262927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"90'8","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:33:34.760669+0000","last_change":"2024-04-02T14:32:22.548264+0000","last_active":"2024-04-02T14:33:34.760669+0000","last_peered":"2024-04-02T14:33:34.760669+0000","last_clean":"2024-04-02T14:33:34.760669+0000","last_became_active":"2024-04-02T14:32:22.547293+0000","last_became_peered":"2024-04-02T14:32:22.547293+0000","last_unstale":"2024-04-02T14:33:34.760669+0000","last_undegraded":"2024-04-02T14:33:34.760669+0000","last_fullsized":"2024-04-02T14:33:34.760669+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:53:19.896243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"90'10","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:33:34.766530+0000","last_change":"2024-04-02T14:32:22.556131+0000","last_active":"2024-04-02T14:33:34.766530+0000","last_peered":"2024-04-02T14:33:34.766530+0000","last_clean":"2024-04-02T14:33:34.766530+0000","last_became_active":"2024-04-02T14:32:22.555763+0000","last_became_peered":"2024-04-02T14:32:22.555763+0000","last_unstale":"2024-04-02T14:33:34.766530+0000","last_undegraded":"2024-04-02T14:33:34.766530+0000","last_fullsized":"2024-04-02T14:33:34.766530+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:45:22.485403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"90'11","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:33:34.769677+0000","last_change":"2024-04-02T14:32:22.562093+0000","last_active":"2024-04-02T14:33:34.769677+0000","last_peered":"2024-04-02T14:33:34.769677+0000","last_clean":"2024-04-02T14:33:34.769677+0000","last_became_active":"2024-04-02T14:32:22.561663+0000","last_became_peered":"2024-04-02T14:32:22.561663+0000","last_unstale":"2024-04-02T14:33:34.769677+0000","last_undegraded":"2024-04-02T14:33:34.769677+0000","last_fullsized":"2024-04-02T14:33:34.769677+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:14:25.209276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"90'8","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:33:34.748961+0000","last_change":"2024-04-02T14:32:22.541964+0000","last_active":"2024-04-02T14:33:34.748961+0000","last_peered":"2024-04-02T14:33:34.748961+0000","last_clean":"2024-04-02T14:33:34.748961+0000","last_became_active":"2024-04-02T14:32:22.541549+0000","last_became_peered":"2024-04-02T14:32:22.541549+0000","last_unstale":"2024-04-02T14:33:34.748961+0000","last_undegraded":"2024-04-02T14:33:34.748961+0000","last_fullsized":"2024-04-02T14:33:34.748961+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:04:56.783158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":68781,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":828,"num_read_kb":482,"num_write":248,"num_write_kb":141,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":44049,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":565248,"data_stored":206343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":215,"ondisk_log_size":215,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":83,"seq":356482285600,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32148,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713004,"statfs":{"total":95995035648,"available":95962116096,"internally_reserved":0,"allocated":995328,"data_stored":474721,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":75,"seq":322122547238,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36992,"kb_used_data":1656,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708160,"statfs":{"total":95995035648,"available":95957155840,"internally_reserved":0,"allocated":1695744,"data_stored":1127577,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":36174273},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743466,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36332,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708820,"statfs":{"total":95995035648,"available":95957831680,"internally_reserved":0,"allocated":1019904,"data_stored":474651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":61,"seq":261993005106,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32160,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712992,"statfs":{"total":95995035648,"available":95962103808,"internally_reserved":0,"allocated":1007616,"data_stored":474357,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299447,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32752,"kb_used_data":1576,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712400,"statfs":{"total":95995035648,"available":95961497600,"internally_reserved":0,"allocated":1613824,"data_stored":1068220,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495675,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36316,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708836,"statfs":{"total":95995035648,"available":95957848064,"internally_reserved":0,"allocated":1003520,"data_stored":474368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574936,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36376,"kb_used_data":1040,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708776,"statfs":{"total":95995035648,"available":95957786624,"internally_reserved":0,"allocated":1064960,"data_stored":536576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738461,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36336,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708816,"statfs":{"total":95995035648,"available":95957827584,"internally_reserved":0,"allocated":1024000,"data_stored":475707,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411411,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36328,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708824,"statfs":{"total":95995035648,"available":95957835776,"internally_reserved":0,"allocated":1015808,"data_stored":476033,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182476,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28012,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717140,"statfs":{"total":95995035648,"available":95966351360,"internally_reserved":0,"allocated":1019904,"data_stored":474193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920840,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32232,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712920,"statfs":{"total":95995035648,"available":95962030080,"internally_reserved":0,"allocated":1081344,"data_stored":536728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757312,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28604,"kb_used_data":1588,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716548,"statfs":{"total":95995035648,"available":95965745152,"internally_reserved":0,"allocated":1626112,"data_stored":1067366,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":63047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":2504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":63199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":4206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":828,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":63563,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-02T14:33:55.307 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:191d963ac9de7322da8aff20b567b3fa458e2ef6 shell --fsid 6449448a-f0fc-11ee-b647-cb9ed24678a4 -- ceph pg dump --format=json 2024-04-02T14:33:55.790 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T14:33:56.110 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:33:55 smithi092 ceph-mon[36497]: from='client.24899 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-02T14:33:56.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:33:55 smithi092 ceph-mon[36497]: pgmap v360: 97 pgs: 97 active+clean; 644 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-02T14:33:56.113 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:33:55 smithi090 ceph-mon[31674]: from='client.24899 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-02T14:33:56.114 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:33:55 smithi090 ceph-mon[31674]: pgmap v360: 97 pgs: 97 active+clean; 644 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-02T14:33:56.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:33:55 smithi177 ceph-mon[36431]: from='client.24899 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-02T14:33:56.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:33:55 smithi177 ceph-mon[36431]: pgmap v360: 97 pgs: 97 active+clean; 644 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-02T14:33:57.938 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:33:57 smithi090 ceph-mon[31674]: pgmap v361: 97 pgs: 97 active+clean; 644 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-02T14:33:58.110 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:33:57 smithi092 ceph-mon[36497]: pgmap v361: 97 pgs: 97 active+clean; 644 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-02T14:33:58.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:58.139+0000 7f1187e00700 1 -- 172.21.15.90:0/251073367 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1188102ec0 msgr2=0x7f118810f380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:33:58.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:58.139+0000 7f1187e00700 1 --2- 172.21.15.90:0/251073367 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1188102ec0 0x7f118810f380 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f118000b0d0 tx=0x7f118000b3e0 comp rx=0 tx=0).stop 2024-04-02T14:33:58.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:58.139+0000 7f1187e00700 1 -- 172.21.15.90:0/251073367 shutdown_connections 2024-04-02T14:33:58.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:58.139+0000 7f1187e00700 1 --2- 172.21.15.90:0/251073367 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1188102ec0 0x7f118810f380 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:58.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:58.139+0000 7f1187e00700 1 --2- 172.21.15.90:0/251073367 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f11881085b0 0x7f1188108990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:58.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:58.139+0000 7f1187e00700 1 --2- 172.21.15.90:0/251073367 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1188102500 0x7f1188102980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:58.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:58.139+0000 7f1187e00700 1 -- 172.21.15.90:0/251073367 >> 172.21.15.90:0/251073367 conn(0x7f11880fdff0 msgr2=0x7f1188100410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:33:58.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:58.139+0000 7f1187e00700 1 -- 172.21.15.90:0/251073367 shutdown_connections 2024-04-02T14:33:58.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:58.139+0000 7f1187e00700 1 -- 172.21.15.90:0/251073367 wait complete. 2024-04-02T14:33:58.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:58.140+0000 7f1187e00700 1 Processor -- start 2024-04-02T14:33:58.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:58.141+0000 7f1187e00700 1 -- start start 2024-04-02T14:33:58.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:58.141+0000 7f1187e00700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1188102500 0x7f11880748f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:33:58.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:58.141+0000 7f1187e00700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1188102ec0 0x7f1188074e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:33:58.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:58.141+0000 7f1187e00700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f11881085b0 0x7f1188075370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:33:58.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:58.141+0000 7f1187e00700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f1188112860 con 0x7f1188102500 2024-04-02T14:33:58.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:58.141+0000 7f1187e00700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f11881126e0 con 0x7f11881085b0 2024-04-02T14:33:58.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:58.141+0000 7f1187e00700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f11881129e0 con 0x7f1188102ec0 2024-04-02T14:33:58.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:58.141+0000 7f1186000700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1188102ec0 0x7f1188074e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:33:58.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:58.141+0000 7f1186a00700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1188102500 0x7f11880748f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:33:58.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:58.141+0000 7f1186000700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1188102ec0 0x7f1188074e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:51304/0 (socket says 172.21.15.90:51304) 2024-04-02T14:33:58.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:58.141+0000 7f1186000700 1 -- 172.21.15.90:0/3459642344 learned_addr learned my addr 172.21.15.90:0/3459642344 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:33:58.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:58.141+0000 7f1187400700 1 --2- 172.21.15.90:0/3459642344 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f11881085b0 0x7f1188075370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:33:58.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:58.142+0000 7f1186000700 1 -- 172.21.15.90:0/3459642344 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f11881085b0 msgr2=0x7f1188075370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:33:58.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:58.142+0000 7f1186000700 1 --2- 172.21.15.90:0/3459642344 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f11881085b0 0x7f1188075370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:58.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:58.142+0000 7f1186000700 1 -- 172.21.15.90:0/3459642344 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1188102500 msgr2=0x7f11880748f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:33:58.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:58.142+0000 7f1186000700 1 --2- 172.21.15.90:0/3459642344 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1188102500 0x7f11880748f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:58.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:58.142+0000 7f1186000700 1 -- 172.21.15.90:0/3459642344 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11881a7100 con 0x7f1188102ec0 2024-04-02T14:33:58.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:58.142+0000 7f1186a00700 1 --2- 172.21.15.90:0/3459642344 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1188102500 0x7f11880748f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:33:58.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:58.142+0000 7f1186000700 1 --2- 172.21.15.90:0/3459642344 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1188102ec0 0x7f1188074e30 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f117400eb30 tx=0x7f117400ee40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:33:58.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:58.142+0000 7f117f400700 1 -- 172.21.15.90:0/3459642344 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f117400cbd0 con 0x7f1188102ec0 2024-04-02T14:33:58.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:58.142+0000 7f1187e00700 1 -- 172.21.15.90:0/3459642344 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11881a73f0 con 0x7f1188102ec0 2024-04-02T14:33:58.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:58.143+0000 7f117f400700 1 -- 172.21.15.90:0/3459642344 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1174004500 con 0x7f1188102ec0 2024-04-02T14:33:58.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:58.143+0000 7f117f400700 1 -- 172.21.15.90:0/3459642344 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1174010430 con 0x7f1188102ec0 2024-04-02T14:33:58.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:58.143+0000 7f1187e00700 1 -- 172.21.15.90:0/3459642344 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11881a7980 con 0x7f1188102ec0 2024-04-02T14:33:58.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:58.144+0000 7f1187e00700 1 -- 172.21.15.90:0/3459642344 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f11880692b0 con 0x7f1188102ec0 2024-04-02T14:33:58.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:58.146+0000 7f117f400700 1 -- 172.21.15.90:0/3459642344 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f1174003f00 con 0x7f1188102ec0 2024-04-02T14:33:58.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:58.147+0000 7f117f400700 1 --2- 172.21.15.90:0/3459642344 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f116c0a3c40 0x7f116c0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:33:58.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:58.147+0000 7f117f400700 1 -- 172.21.15.90:0/3459642344 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8248+0+0 (secure 0 0 0) 0x7f11740d0940 con 0x7f1188102ec0 2024-04-02T14:33:58.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:58.147+0000 7f1186a00700 1 --2- 172.21.15.90:0/3459642344 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f116c0a3c40 0x7f116c0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:33:58.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:58.148+0000 7f1186a00700 1 --2- 172.21.15.90:0/3459642344 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f116c0a3c40 0x7f116c0a6100 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f1178009fd0 tx=0x7f1178009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:33:58.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:58.150+0000 7f117f400700 1 -- 172.21.15.90:0/3459642344 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7f117409d2b0 con 0x7f1188102ec0 2024-04-02T14:33:58.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:33:57 smithi177 ceph-mon[36431]: pgmap v361: 97 pgs: 97 active+clean; 644 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-02T14:33:58.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:58.370+0000 7f1187e00700 1 -- 172.21.15.90:0/3459642344 --> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f11880ffce0 con 0x7f116c0a3c40 2024-04-02T14:33:58.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:58.378+0000 7f117f400700 1 -- 172.21.15.90:0/3459642344 <== mgr.14152 v2:172.21.15.90:6800/4266054817 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258893 (secure 0 0 0) 0x7f11880ffce0 con 0x7f116c0a3c40 2024-04-02T14:33:58.385 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-02T14:33:58.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:58.384+0000 7f1187e00700 1 -- 172.21.15.90:0/3459642344 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f116c0a3c40 msgr2=0x7f116c0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:33:58.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:58.384+0000 7f1187e00700 1 --2- 172.21.15.90:0/3459642344 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f116c0a3c40 0x7f116c0a6100 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f1178009fd0 tx=0x7f1178009260 comp rx=0 tx=0).stop 2024-04-02T14:33:58.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:58.384+0000 7f1187e00700 1 -- 172.21.15.90:0/3459642344 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1188102ec0 msgr2=0x7f1188074e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:33:58.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:58.384+0000 7f1187e00700 1 --2- 172.21.15.90:0/3459642344 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1188102ec0 0x7f1188074e30 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f117400eb30 tx=0x7f117400ee40 comp rx=0 tx=0).stop 2024-04-02T14:33:58.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:58.384+0000 7f1187e00700 1 -- 172.21.15.90:0/3459642344 shutdown_connections 2024-04-02T14:33:58.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:58.384+0000 7f1187e00700 1 --2- 172.21.15.90:0/3459642344 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f116c0a3c40 0x7f116c0a6100 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:58.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:58.384+0000 7f1187e00700 1 --2- 172.21.15.90:0/3459642344 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1188102500 0x7f11880748f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:58.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:58.384+0000 7f1187e00700 1 --2- 172.21.15.90:0/3459642344 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f11881085b0 0x7f1188075370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:58.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:58.384+0000 7f1187e00700 1 --2- 172.21.15.90:0/3459642344 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1188102ec0 0x7f1188074e30 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:33:58.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:58.384+0000 7f1187e00700 1 -- 172.21.15.90:0/3459642344 >> 172.21.15.90:0/3459642344 conn(0x7f11880fdff0 msgr2=0x7f11880ff5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:33:58.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:58.385+0000 7f1187e00700 1 -- 172.21.15.90:0/3459642344 shutdown_connections 2024-04-02T14:33:58.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:33:58.385+0000 7f1187e00700 1 -- 172.21.15.90:0/3459642344 wait complete. 2024-04-02T14:33:58.388 INFO:teuthology.orchestra.run.smithi090.stderr:dumped all 2024-04-02T14:33:59.599 INFO:teuthology.orchestra.run.smithi090.stdout:{"pg_ready":true,"pg_map":{"version":361,"stamp":"2024-04-02T14:33:56.770499+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":659383,"num_objects":102,"num_object_clones":0,"num_object_copies":306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":1009,"num_read_kb":621,"num_write":431,"num_write_kb":3081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":1224785,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":854,"ondisk_log_size":854,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":404588,"kb_used_data":13836,"kb_used_omap":18,"kb_used_meta":390637,"kb_avail":1124537236,"statfs":{"total":1151940427776,"available":1151526129664,"internally_reserved":0,"allocated":14168064,"data_stored":7660497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":400012602},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":115,"num_read_kb":50,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005815"},"pg_stats":[{"pgid":"3.3f","version":"90'8","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:33:34.770820+0000","last_change":"2024-04-02T14:32:22.558913+0000","last_active":"2024-04-02T14:33:34.770820+0000","last_peered":"2024-04-02T14:33:34.770820+0000","last_clean":"2024-04-02T14:33:34.770820+0000","last_became_active":"2024-04-02T14:32:22.558606+0000","last_became_peered":"2024-04-02T14:32:22.558606+0000","last_unstale":"2024-04-02T14:33:34.770820+0000","last_undegraded":"2024-04-02T14:33:34.770820+0000","last_fullsized":"2024-04-02T14:33:34.770820+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:59:49.205499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"90'4","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:33:34.739039+0000","last_change":"2024-04-02T14:32:22.551898+0000","last_active":"2024-04-02T14:33:34.739039+0000","last_peered":"2024-04-02T14:33:34.739039+0000","last_clean":"2024-04-02T14:33:34.739039+0000","last_became_active":"2024-04-02T14:32:22.549016+0000","last_became_peered":"2024-04-02T14:32:22.549016+0000","last_unstale":"2024-04-02T14:33:34.739039+0000","last_undegraded":"2024-04-02T14:33:34.739039+0000","last_fullsized":"2024-04-02T14:33:34.739039+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:38:03.529505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"90'7","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:33:34.760941+0000","last_change":"2024-04-02T14:32:22.556214+0000","last_active":"2024-04-02T14:33:34.760941+0000","last_peered":"2024-04-02T14:33:34.760941+0000","last_clean":"2024-04-02T14:33:34.760941+0000","last_became_active":"2024-04-02T14:32:22.555686+0000","last_became_peered":"2024-04-02T14:32:22.555686+0000","last_unstale":"2024-04-02T14:33:34.760941+0000","last_undegraded":"2024-04-02T14:33:34.760941+0000","last_fullsized":"2024-04-02T14:33:34.760941+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:06:40.161203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"90'8","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:33:34.743043+0000","last_change":"2024-04-02T14:32:22.537830+0000","last_active":"2024-04-02T14:33:34.743043+0000","last_peered":"2024-04-02T14:33:34.743043+0000","last_clean":"2024-04-02T14:33:34.743043+0000","last_became_active":"2024-04-02T14:32:22.536860+0000","last_became_peered":"2024-04-02T14:32:22.536860+0000","last_unstale":"2024-04-02T14:33:34.743043+0000","last_undegraded":"2024-04-02T14:33:34.743043+0000","last_fullsized":"2024-04-02T14:33:34.743043+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T01:07:57.244177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"90'4","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:33:34.759688+0000","last_change":"2024-04-02T14:32:22.549550+0000","last_active":"2024-04-02T14:33:34.759688+0000","last_peered":"2024-04-02T14:33:34.759688+0000","last_clean":"2024-04-02T14:33:34.759688+0000","last_became_active":"2024-04-02T14:32:22.549177+0000","last_became_peered":"2024-04-02T14:32:22.549177+0000","last_unstale":"2024-04-02T14:33:34.759688+0000","last_undegraded":"2024-04-02T14:33:34.759688+0000","last_fullsized":"2024-04-02T14:33:34.759688+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:31:49.175583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"90'6","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:33:34.744861+0000","last_change":"2024-04-02T14:32:35.849338+0000","last_active":"2024-04-02T14:33:34.744861+0000","last_peered":"2024-04-02T14:33:34.744861+0000","last_clean":"2024-04-02T14:33:34.744861+0000","last_became_active":"2024-04-02T14:32:35.848929+0000","last_became_peered":"2024-04-02T14:32:35.848929+0000","last_unstale":"2024-04-02T14:33:34.744861+0000","last_undegraded":"2024-04-02T14:33:34.744861+0000","last_fullsized":"2024-04-02T14:33:34.744861+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:55:58.614290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,6],"acting":[10,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"90'5","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:33:34.752353+0000","last_change":"2024-04-02T14:32:22.546861+0000","last_active":"2024-04-02T14:33:34.752353+0000","last_peered":"2024-04-02T14:33:34.752353+0000","last_clean":"2024-04-02T14:33:34.752353+0000","last_became_active":"2024-04-02T14:32:22.546477+0000","last_became_peered":"2024-04-02T14:32:22.546477+0000","last_unstale":"2024-04-02T14:33:34.752353+0000","last_undegraded":"2024-04-02T14:33:34.752353+0000","last_fullsized":"2024-04-02T14:33:34.752353+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:59:17.133714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"90'9","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:33:34.763067+0000","last_change":"2024-04-02T14:32:22.554287+0000","last_active":"2024-04-02T14:33:34.763067+0000","last_peered":"2024-04-02T14:33:34.763067+0000","last_clean":"2024-04-02T14:33:34.763067+0000","last_became_active":"2024-04-02T14:32:22.553491+0000","last_became_peered":"2024-04-02T14:32:22.553491+0000","last_unstale":"2024-04-02T14:33:34.763067+0000","last_undegraded":"2024-04-02T14:33:34.763067+0000","last_fullsized":"2024-04-02T14:33:34.763067+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:51:30.785380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"90'11","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:33:34.757821+0000","last_change":"2024-04-02T14:32:22.538158+0000","last_active":"2024-04-02T14:33:34.757821+0000","last_peered":"2024-04-02T14:33:34.757821+0000","last_clean":"2024-04-02T14:33:34.757821+0000","last_became_active":"2024-04-02T14:32:22.537796+0000","last_became_peered":"2024-04-02T14:32:22.537796+0000","last_unstale":"2024-04-02T14:33:34.757821+0000","last_undegraded":"2024-04-02T14:33:34.757821+0000","last_fullsized":"2024-04-02T14:33:34.757821+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:09:48.054965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"90'16","reported_seq":33,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:33:34.742793+0000","last_change":"2024-04-02T14:32:22.535225+0000","last_active":"2024-04-02T14:33:34.742793+0000","last_peered":"2024-04-02T14:33:34.742793+0000","last_clean":"2024-04-02T14:33:34.742793+0000","last_became_active":"2024-04-02T14:32:22.534856+0000","last_became_peered":"2024-04-02T14:32:22.534856+0000","last_unstale":"2024-04-02T14:33:34.742793+0000","last_undegraded":"2024-04-02T14:33:34.742793+0000","last_fullsized":"2024-04-02T14:33:34.742793+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:22:11.118918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"90'6","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:33:34.755859+0000","last_change":"2024-04-02T14:32:22.539138+0000","last_active":"2024-04-02T14:33:34.755859+0000","last_peered":"2024-04-02T14:33:34.755859+0000","last_clean":"2024-04-02T14:33:34.755859+0000","last_became_active":"2024-04-02T14:32:22.538762+0000","last_became_peered":"2024-04-02T14:32:22.538762+0000","last_unstale":"2024-04-02T14:33:34.755859+0000","last_undegraded":"2024-04-02T14:33:34.755859+0000","last_fullsized":"2024-04-02T14:33:34.755859+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:18:59.174339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"90'7","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:33:34.747727+0000","last_change":"2024-04-02T14:32:22.536383+0000","last_active":"2024-04-02T14:33:34.747727+0000","last_peered":"2024-04-02T14:33:34.747727+0000","last_clean":"2024-04-02T14:33:34.747727+0000","last_became_active":"2024-04-02T14:32:22.536043+0000","last_became_peered":"2024-04-02T14:32:22.536043+0000","last_unstale":"2024-04-02T14:33:34.747727+0000","last_undegraded":"2024-04-02T14:33:34.747727+0000","last_fullsized":"2024-04-02T14:33:34.747727+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:14:48.628684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"90'10","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:33:34.769406+0000","last_change":"2024-04-02T14:32:22.552512+0000","last_active":"2024-04-02T14:33:34.769406+0000","last_peered":"2024-04-02T14:33:34.769406+0000","last_clean":"2024-04-02T14:33:34.769406+0000","last_became_active":"2024-04-02T14:32:22.549502+0000","last_became_peered":"2024-04-02T14:32:22.549502+0000","last_unstale":"2024-04-02T14:33:34.769406+0000","last_undegraded":"2024-04-02T14:33:34.769406+0000","last_fullsized":"2024-04-02T14:33:34.769406+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:37:45.609280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"90'7","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:33:34.754701+0000","last_change":"2024-04-02T14:32:22.555074+0000","last_active":"2024-04-02T14:33:34.754701+0000","last_peered":"2024-04-02T14:33:34.754701+0000","last_clean":"2024-04-02T14:33:34.754701+0000","last_became_active":"2024-04-02T14:32:22.554422+0000","last_became_peered":"2024-04-02T14:32:22.554422+0000","last_unstale":"2024-04-02T14:33:34.754701+0000","last_undegraded":"2024-04-02T14:33:34.754701+0000","last_fullsized":"2024-04-02T14:33:34.754701+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:27:49.536941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"90'7","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:33:34.751759+0000","last_change":"2024-04-02T14:32:22.550322+0000","last_active":"2024-04-02T14:33:34.751759+0000","last_peered":"2024-04-02T14:33:34.751759+0000","last_clean":"2024-04-02T14:33:34.751759+0000","last_became_active":"2024-04-02T14:32:22.548710+0000","last_became_peered":"2024-04-02T14:32:22.548710+0000","last_unstale":"2024-04-02T14:33:34.751759+0000","last_undegraded":"2024-04-02T14:33:34.751759+0000","last_fullsized":"2024-04-02T14:33:34.751759+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:19:09.226556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"90'9","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:33:34.769494+0000","last_change":"2024-04-02T14:32:22.558705+0000","last_active":"2024-04-02T14:33:34.769494+0000","last_peered":"2024-04-02T14:33:34.769494+0000","last_clean":"2024-04-02T14:33:34.769494+0000","last_became_active":"2024-04-02T14:32:22.558309+0000","last_became_peered":"2024-04-02T14:32:22.558309+0000","last_unstale":"2024-04-02T14:33:34.769494+0000","last_undegraded":"2024-04-02T14:33:34.769494+0000","last_fullsized":"2024-04-02T14:33:34.769494+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:52:23.929627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"90'8","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:33:34.771040+0000","last_change":"2024-04-02T14:32:22.551211+0000","last_active":"2024-04-02T14:33:34.771040+0000","last_peered":"2024-04-02T14:33:34.771040+0000","last_clean":"2024-04-02T14:33:34.771040+0000","last_became_active":"2024-04-02T14:32:22.550882+0000","last_became_peered":"2024-04-02T14:32:22.550882+0000","last_unstale":"2024-04-02T14:33:34.771040+0000","last_undegraded":"2024-04-02T14:33:34.771040+0000","last_fullsized":"2024-04-02T14:33:34.771040+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:54:54.533063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"90'10","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:33:34.760206+0000","last_change":"2024-04-02T14:32:22.550692+0000","last_active":"2024-04-02T14:33:34.760206+0000","last_peered":"2024-04-02T14:33:34.760206+0000","last_clean":"2024-04-02T14:33:34.760206+0000","last_became_active":"2024-04-02T14:32:22.550292+0000","last_became_peered":"2024-04-02T14:32:22.550292+0000","last_unstale":"2024-04-02T14:33:34.760206+0000","last_undegraded":"2024-04-02T14:33:34.760206+0000","last_fullsized":"2024-04-02T14:33:34.760206+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:14:42.317649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"90'7","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:33:34.760461+0000","last_change":"2024-04-02T14:32:22.553042+0000","last_active":"2024-04-02T14:33:34.760461+0000","last_peered":"2024-04-02T14:33:34.760461+0000","last_clean":"2024-04-02T14:33:34.760461+0000","last_became_active":"2024-04-02T14:32:22.552010+0000","last_became_peered":"2024-04-02T14:32:22.552010+0000","last_unstale":"2024-04-02T14:33:34.760461+0000","last_undegraded":"2024-04-02T14:33:34.760461+0000","last_fullsized":"2024-04-02T14:33:34.760461+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:31:13.947662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"88'13","reported_seq":136,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:33:52.015912+0000","last_change":"2024-04-02T14:32:35.868597+0000","last_active":"2024-04-02T14:33:52.015912+0000","last_peered":"2024-04-02T14:33:52.015912+0000","last_clean":"2024-04-02T14:33:52.015912+0000","last_became_active":"2024-04-02T14:32:35.851962+0000","last_became_peered":"2024-04-02T14:32:35.851962+0000","last_unstale":"2024-04-02T14:33:52.015912+0000","last_undegraded":"2024-04-02T14:33:52.015912+0000","last_fullsized":"2024-04-02T14:33:52.015912+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:17:06.400744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":80,"num_read_kb":80,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":672,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,4],"acting":[9,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"90'5","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:33:34.760600+0000","last_change":"2024-04-02T14:32:22.549526+0000","last_active":"2024-04-02T14:33:34.760600+0000","last_peered":"2024-04-02T14:33:34.760600+0000","last_clean":"2024-04-02T14:33:34.760600+0000","last_became_active":"2024-04-02T14:32:22.548250+0000","last_became_peered":"2024-04-02T14:32:22.548250+0000","last_unstale":"2024-04-02T14:33:34.760600+0000","last_undegraded":"2024-04-02T14:33:34.760600+0000","last_fullsized":"2024-04-02T14:33:34.760600+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:35:34.866803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"88'2","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:32:35.737760+0000","last_change":"2024-04-02T14:32:21.865084+0000","last_active":"2024-04-02T14:32:35.737760+0000","last_peered":"2024-04-02T14:32:35.737760+0000","last_clean":"2024-04-02T14:32:35.737760+0000","last_became_active":"2024-04-02T14:32:21.864718+0000","last_became_peered":"2024-04-02T14:32:21.864718+0000","last_unstale":"2024-04-02T14:32:35.737760+0000","last_undegraded":"2024-04-02T14:32:35.737760+0000","last_fullsized":"2024-04-02T14:32:35.737760+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:10:50.611362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"90'6","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:33:34.772233+0000","last_change":"2024-04-02T14:32:22.549443+0000","last_active":"2024-04-02T14:33:34.772233+0000","last_peered":"2024-04-02T14:33:34.772233+0000","last_clean":"2024-04-02T14:33:34.772233+0000","last_became_active":"2024-04-02T14:32:22.549059+0000","last_became_peered":"2024-04-02T14:32:22.549059+0000","last_unstale":"2024-04-02T14:33:34.772233+0000","last_undegraded":"2024-04-02T14:33:34.772233+0000","last_fullsized":"2024-04-02T14:33:34.772233+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:04:21.489864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"88'12","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:32:35.731852+0000","last_change":"2024-04-02T14:32:21.841368+0000","last_active":"2024-04-02T14:32:35.731852+0000","last_peered":"2024-04-02T14:32:35.731852+0000","last_clean":"2024-04-02T14:32:35.731852+0000","last_became_active":"2024-04-02T14:32:21.840963+0000","last_became_peered":"2024-04-02T14:32:21.840963+0000","last_unstale":"2024-04-02T14:32:35.731852+0000","last_undegraded":"2024-04-02T14:32:35.731852+0000","last_fullsized":"2024-04-02T14:32:35.731852+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:36:40.549720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"90'9","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:33:34.769081+0000","last_change":"2024-04-02T14:32:22.549753+0000","last_active":"2024-04-02T14:33:34.769081+0000","last_peered":"2024-04-02T14:33:34.769081+0000","last_clean":"2024-04-02T14:33:34.769081+0000","last_became_active":"2024-04-02T14:32:22.549354+0000","last_became_peered":"2024-04-02T14:32:22.549354+0000","last_unstale":"2024-04-02T14:33:34.769081+0000","last_undegraded":"2024-04-02T14:33:34.769081+0000","last_fullsized":"2024-04-02T14:33:34.769081+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T02:17:09.650946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"88'5","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:32:35.731222+0000","last_change":"2024-04-02T14:32:21.842766+0000","last_active":"2024-04-02T14:32:35.731222+0000","last_peered":"2024-04-02T14:32:35.731222+0000","last_clean":"2024-04-02T14:32:35.731222+0000","last_became_active":"2024-04-02T14:32:21.842410+0000","last_became_peered":"2024-04-02T14:32:21.842410+0000","last_unstale":"2024-04-02T14:32:35.731222+0000","last_undegraded":"2024-04-02T14:32:35.731222+0000","last_fullsized":"2024-04-02T14:32:35.731222+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:49:31.651127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"90'15","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:33:34.748161+0000","last_change":"2024-04-02T14:32:22.538506+0000","last_active":"2024-04-02T14:33:34.748161+0000","last_peered":"2024-04-02T14:33:34.748161+0000","last_clean":"2024-04-02T14:33:34.748161+0000","last_became_active":"2024-04-02T14:32:22.538168+0000","last_became_peered":"2024-04-02T14:32:22.538168+0000","last_unstale":"2024-04-02T14:33:34.748161+0000","last_undegraded":"2024-04-02T14:33:34.748161+0000","last_fullsized":"2024-04-02T14:33:34.748161+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:30:04.746382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"90'7","reported_seq":112,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:33:50.945215+0000","last_change":"2024-04-02T14:32:21.844180+0000","last_active":"2024-04-02T14:33:50.945215+0000","last_peered":"2024-04-02T14:33:50.945215+0000","last_clean":"2024-04-02T14:33:50.945215+0000","last_became_active":"2024-04-02T14:32:21.843840+0000","last_became_peered":"2024-04-02T14:32:21.843840+0000","last_unstale":"2024-04-02T14:33:50.945215+0000","last_undegraded":"2024-04-02T14:33:50.945215+0000","last_fullsized":"2024-04-02T14:33:50.945215+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T01:56:10.767946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1154,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":3,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"90'7","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:33:34.737512+0000","last_change":"2024-04-02T14:32:22.548453+0000","last_active":"2024-04-02T14:33:34.737512+0000","last_peered":"2024-04-02T14:33:34.737512+0000","last_clean":"2024-04-02T14:33:34.737512+0000","last_became_active":"2024-04-02T14:32:22.548110+0000","last_became_peered":"2024-04-02T14:32:22.548110+0000","last_unstale":"2024-04-02T14:33:34.737512+0000","last_undegraded":"2024-04-02T14:33:34.737512+0000","last_fullsized":"2024-04-02T14:33:34.737512+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:57:13.067425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"88'7","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:32:35.724559+0000","last_change":"2024-04-02T14:32:21.846528+0000","last_active":"2024-04-02T14:32:35.724559+0000","last_peered":"2024-04-02T14:32:35.724559+0000","last_clean":"2024-04-02T14:32:35.724559+0000","last_became_active":"2024-04-02T14:32:21.846152+0000","last_became_peered":"2024-04-02T14:32:21.846152+0000","last_unstale":"2024-04-02T14:32:35.724559+0000","last_undegraded":"2024-04-02T14:32:35.724559+0000","last_fullsized":"2024-04-02T14:32:35.724559+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:58:09.286256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"90'8","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:33:34.758200+0000","last_change":"2024-04-02T14:32:22.546730+0000","last_active":"2024-04-02T14:33:34.758200+0000","last_peered":"2024-04-02T14:33:34.758200+0000","last_clean":"2024-04-02T14:33:34.758200+0000","last_became_active":"2024-04-02T14:32:22.546342+0000","last_became_peered":"2024-04-02T14:32:22.546342+0000","last_unstale":"2024-04-02T14:33:34.758200+0000","last_undegraded":"2024-04-02T14:33:34.758200+0000","last_fullsized":"2024-04-02T14:33:34.758200+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:38:37.616382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"88'3","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:32:35.845954+0000","last_change":"2024-04-02T14:32:21.853316+0000","last_active":"2024-04-02T14:32:35.845954+0000","last_peered":"2024-04-02T14:32:35.845954+0000","last_clean":"2024-04-02T14:32:35.845954+0000","last_became_active":"2024-04-02T14:32:21.852912+0000","last_became_peered":"2024-04-02T14:32:21.852912+0000","last_unstale":"2024-04-02T14:32:35.845954+0000","last_undegraded":"2024-04-02T14:32:35.845954+0000","last_fullsized":"2024-04-02T14:32:35.845954+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T02:30:44.601900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"90'8","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:33:34.762722+0000","last_change":"2024-04-02T14:32:22.558913+0000","last_active":"2024-04-02T14:33:34.762722+0000","last_peered":"2024-04-02T14:33:34.762722+0000","last_clean":"2024-04-02T14:33:34.762722+0000","last_became_active":"2024-04-02T14:32:22.558511+0000","last_became_peered":"2024-04-02T14:32:22.558511+0000","last_unstale":"2024-04-02T14:33:34.762722+0000","last_undegraded":"2024-04-02T14:33:34.762722+0000","last_fullsized":"2024-04-02T14:33:34.762722+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:48:36.449086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"88'3","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:32:35.730749+0000","last_change":"2024-04-02T14:32:21.844081+0000","last_active":"2024-04-02T14:32:35.730749+0000","last_peered":"2024-04-02T14:32:35.730749+0000","last_clean":"2024-04-02T14:32:35.730749+0000","last_became_active":"2024-04-02T14:32:21.843788+0000","last_became_peered":"2024-04-02T14:32:21.843788+0000","last_unstale":"2024-04-02T14:32:35.730749+0000","last_undegraded":"2024-04-02T14:32:35.730749+0000","last_fullsized":"2024-04-02T14:32:35.730749+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:19:02.184229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"90'11","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:33:34.768194+0000","last_change":"2024-04-02T14:32:22.556193+0000","last_active":"2024-04-02T14:33:34.768194+0000","last_peered":"2024-04-02T14:33:34.768194+0000","last_clean":"2024-04-02T14:33:34.768194+0000","last_became_active":"2024-04-02T14:32:22.555787+0000","last_became_peered":"2024-04-02T14:32:22.555787+0000","last_unstale":"2024-04-02T14:33:34.768194+0000","last_undegraded":"2024-04-02T14:33:34.768194+0000","last_fullsized":"2024-04-02T14:33:34.768194+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T02:02:13.730941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"90'5","reported_seq":194,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:33:53.948731+0000","last_change":"2024-04-02T14:32:21.840651+0000","last_active":"2024-04-02T14:33:53.948731+0000","last_peered":"2024-04-02T14:33:53.948731+0000","last_clean":"2024-04-02T14:33:53.948731+0000","last_became_active":"2024-04-02T14:32:21.840344+0000","last_became_peered":"2024-04-02T14:32:21.840344+0000","last_unstale":"2024-04-02T14:33:53.948731+0000","last_undegraded":"2024-04-02T14:33:53.948731+0000","last_fullsized":"2024-04-02T14:33:53.948731+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:49:22.384224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":170,"num_read_kb":170,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"90'10","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:33:34.763370+0000","last_change":"2024-04-02T14:32:22.548245+0000","last_active":"2024-04-02T14:33:34.763370+0000","last_peered":"2024-04-02T14:33:34.763370+0000","last_clean":"2024-04-02T14:33:34.763370+0000","last_became_active":"2024-04-02T14:32:22.547853+0000","last_became_peered":"2024-04-02T14:32:22.547853+0000","last_unstale":"2024-04-02T14:33:34.763370+0000","last_undegraded":"2024-04-02T14:33:34.763370+0000","last_fullsized":"2024-04-02T14:33:34.763370+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:02:43.036944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"88'4","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:32:35.730780+0000","last_change":"2024-04-02T14:32:21.842096+0000","last_active":"2024-04-02T14:32:35.730780+0000","last_peered":"2024-04-02T14:32:35.730780+0000","last_clean":"2024-04-02T14:32:35.730780+0000","last_became_active":"2024-04-02T14:32:21.841646+0000","last_became_peered":"2024-04-02T14:32:21.841646+0000","last_unstale":"2024-04-02T14:32:35.730780+0000","last_undegraded":"2024-04-02T14:32:35.730780+0000","last_fullsized":"2024-04-02T14:32:35.730780+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:04:08.149785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"90'9","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:33:34.772870+0000","last_change":"2024-04-02T14:32:22.557694+0000","last_active":"2024-04-02T14:33:34.772870+0000","last_peered":"2024-04-02T14:33:34.772870+0000","last_clean":"2024-04-02T14:33:34.772870+0000","last_became_active":"2024-04-02T14:32:22.557202+0000","last_became_peered":"2024-04-02T14:32:22.557202+0000","last_unstale":"2024-04-02T14:33:34.772870+0000","last_undegraded":"2024-04-02T14:33:34.772870+0000","last_fullsized":"2024-04-02T14:33:34.772870+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:35:41.471557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"88'5","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:32:35.731380+0000","last_change":"2024-04-02T14:32:21.864366+0000","last_active":"2024-04-02T14:32:35.731380+0000","last_peered":"2024-04-02T14:32:35.731380+0000","last_clean":"2024-04-02T14:32:35.731380+0000","last_became_active":"2024-04-02T14:32:21.864006+0000","last_became_peered":"2024-04-02T14:32:21.864006+0000","last_unstale":"2024-04-02T14:32:35.731380+0000","last_undegraded":"2024-04-02T14:32:35.731380+0000","last_fullsized":"2024-04-02T14:32:35.731380+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:08:43.176061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"90'11","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:33:34.751922+0000","last_change":"2024-04-02T14:32:22.541562+0000","last_active":"2024-04-02T14:33:34.751922+0000","last_peered":"2024-04-02T14:33:34.751922+0000","last_clean":"2024-04-02T14:33:34.751922+0000","last_became_active":"2024-04-02T14:32:22.541209+0000","last_became_peered":"2024-04-02T14:32:22.541209+0000","last_unstale":"2024-04-02T14:33:34.751922+0000","last_undegraded":"2024-04-02T14:33:34.751922+0000","last_fullsized":"2024-04-02T14:33:34.751922+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:49:40.310528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"90'33","reported_seq":248,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:33:53.973006+0000","last_change":"2024-04-02T14:32:35.875429+0000","last_active":"2024-04-02T14:33:53.973006+0000","last_peered":"2024-04-02T14:33:53.973006+0000","last_clean":"2024-04-02T14:33:53.973006+0000","last_became_active":"2024-04-02T14:32:35.852417+0000","last_became_peered":"2024-04-02T14:32:35.852417+0000","last_unstale":"2024-04-02T14:33:53.973006+0000","last_undegraded":"2024-04-02T14:33:53.973006+0000","last_fullsized":"2024-04-02T14:33:53.973006+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:33:53.903712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":61295,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":169,"num_read_kb":139,"num_write":35,"num_write_kb":72,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":43377,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,4],"acting":[10,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"90'6","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:33:34.740087+0000","last_change":"2024-04-02T14:32:22.535716+0000","last_active":"2024-04-02T14:33:34.740087+0000","last_peered":"2024-04-02T14:33:34.740087+0000","last_clean":"2024-04-02T14:33:34.740087+0000","last_became_active":"2024-04-02T14:32:22.534738+0000","last_became_peered":"2024-04-02T14:32:22.534738+0000","last_unstale":"2024-04-02T14:33:34.740087+0000","last_undegraded":"2024-04-02T14:33:34.740087+0000","last_fullsized":"2024-04-02T14:33:34.740087+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T01:25:07.763915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"88'9","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:32:35.726311+0000","last_change":"2024-04-02T14:32:21.854582+0000","last_active":"2024-04-02T14:32:35.726311+0000","last_peered":"2024-04-02T14:32:35.726311+0000","last_clean":"2024-04-02T14:32:35.726311+0000","last_became_active":"2024-04-02T14:32:21.854226+0000","last_became_peered":"2024-04-02T14:32:21.854226+0000","last_unstale":"2024-04-02T14:32:35.726311+0000","last_undegraded":"2024-04-02T14:32:35.726311+0000","last_fullsized":"2024-04-02T14:32:35.726311+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:30:02.289163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"90'9","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:33:34.762399+0000","last_change":"2024-04-02T14:32:22.555901+0000","last_active":"2024-04-02T14:33:34.762399+0000","last_peered":"2024-04-02T14:33:34.762399+0000","last_clean":"2024-04-02T14:33:34.762399+0000","last_became_active":"2024-04-02T14:32:22.555158+0000","last_became_peered":"2024-04-02T14:32:22.555158+0000","last_unstale":"2024-04-02T14:33:34.762399+0000","last_undegraded":"2024-04-02T14:33:34.762399+0000","last_fullsized":"2024-04-02T14:33:34.762399+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:04:28.766185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"88'5","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:32:35.730716+0000","last_change":"2024-04-02T14:32:21.855982+0000","last_active":"2024-04-02T14:32:35.730716+0000","last_peered":"2024-04-02T14:32:35.730716+0000","last_clean":"2024-04-02T14:32:35.730716+0000","last_became_active":"2024-04-02T14:32:21.855591+0000","last_became_peered":"2024-04-02T14:32:21.855591+0000","last_unstale":"2024-04-02T14:32:35.730716+0000","last_undegraded":"2024-04-02T14:32:35.730716+0000","last_fullsized":"2024-04-02T14:32:35.730716+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:28:51.806779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"90'15","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:33:34.767837+0000","last_change":"2024-04-02T14:32:22.549889+0000","last_active":"2024-04-02T14:33:34.767837+0000","last_peered":"2024-04-02T14:33:34.767837+0000","last_clean":"2024-04-02T14:33:34.767837+0000","last_became_active":"2024-04-02T14:32:22.549479+0000","last_became_peered":"2024-04-02T14:32:22.549479+0000","last_unstale":"2024-04-02T14:33:34.767837+0000","last_undegraded":"2024-04-02T14:33:34.767837+0000","last_fullsized":"2024-04-02T14:33:34.767837+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:29:25.937576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"90'7","reported_seq":114,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:33:53.025843+0000","last_change":"2024-04-02T14:32:21.838165+0000","last_active":"2024-04-02T14:33:53.025843+0000","last_peered":"2024-04-02T14:33:53.025843+0000","last_clean":"2024-04-02T14:33:53.025843+0000","last_became_active":"2024-04-02T14:32:21.837808+0000","last_became_peered":"2024-04-02T14:32:21.837808+0000","last_unstale":"2024-04-02T14:33:53.025843+0000","last_undegraded":"2024-04-02T14:33:53.025843+0000","last_fullsized":"2024-04-02T14:33:53.025843+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:00:44.605314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1120,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":89,"num_read_kb":4,"num_write":7,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"90'6","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:33:34.745892+0000","last_change":"2024-04-02T14:32:22.532937+0000","last_active":"2024-04-02T14:33:34.745892+0000","last_peered":"2024-04-02T14:33:34.745892+0000","last_clean":"2024-04-02T14:33:34.745892+0000","last_became_active":"2024-04-02T14:32:22.532586+0000","last_became_peered":"2024-04-02T14:32:22.532586+0000","last_unstale":"2024-04-02T14:33:34.745892+0000","last_undegraded":"2024-04-02T14:33:34.745892+0000","last_fullsized":"2024-04-02T14:33:34.745892+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:50:48.030889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"88'17","reported_seq":37,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:32:35.731828+0000","last_change":"2024-04-02T14:32:21.860032+0000","last_active":"2024-04-02T14:32:35.731828+0000","last_peered":"2024-04-02T14:32:35.731828+0000","last_clean":"2024-04-02T14:32:35.731828+0000","last_became_active":"2024-04-02T14:32:21.859653+0000","last_became_peered":"2024-04-02T14:32:21.859653+0000","last_unstale":"2024-04-02T14:32:35.731828+0000","last_undegraded":"2024-04-02T14:32:35.731828+0000","last_fullsized":"2024-04-02T14:32:35.731828+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:06:01.144530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"90'6","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:33:34.761794+0000","last_change":"2024-04-02T14:32:35.736133+0000","last_active":"2024-04-02T14:33:34.761794+0000","last_peered":"2024-04-02T14:33:34.761794+0000","last_clean":"2024-04-02T14:33:34.761794+0000","last_became_active":"2024-04-02T14:32:35.734335+0000","last_became_peered":"2024-04-02T14:32:35.734335+0000","last_unstale":"2024-04-02T14:33:34.761794+0000","last_undegraded":"2024-04-02T14:33:34.761794+0000","last_fullsized":"2024-04-02T14:33:34.761794+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:36:21.828318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,1],"acting":[9,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.f","version":"88'2","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:32:35.731999+0000","last_change":"2024-04-02T14:32:21.842384+0000","last_active":"2024-04-02T14:32:35.731999+0000","last_peered":"2024-04-02T14:32:35.731999+0000","last_clean":"2024-04-02T14:32:35.731999+0000","last_became_active":"2024-04-02T14:32:21.842040+0000","last_became_peered":"2024-04-02T14:32:21.842040+0000","last_unstale":"2024-04-02T14:32:35.731999+0000","last_undegraded":"2024-04-02T14:32:35.731999+0000","last_fullsized":"2024-04-02T14:32:35.731999+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:33:29.541972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"90'5","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:33:34.748788+0000","last_change":"2024-04-02T14:32:22.538571+0000","last_active":"2024-04-02T14:33:34.748788+0000","last_peered":"2024-04-02T14:33:34.748788+0000","last_clean":"2024-04-02T14:33:34.748788+0000","last_became_active":"2024-04-02T14:32:22.538239+0000","last_became_peered":"2024-04-02T14:32:22.538239+0000","last_unstale":"2024-04-02T14:33:34.748788+0000","last_undegraded":"2024-04-02T14:33:34.748788+0000","last_fullsized":"2024-04-02T14:33:34.748788+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T01:19:53.120419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"88'1","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:32:35.726314+0000","last_change":"2024-04-02T14:32:21.857666+0000","last_active":"2024-04-02T14:32:35.726314+0000","last_peered":"2024-04-02T14:32:35.726314+0000","last_clean":"2024-04-02T14:32:35.726314+0000","last_became_active":"2024-04-02T14:32:21.856618+0000","last_became_peered":"2024-04-02T14:32:21.856618+0000","last_unstale":"2024-04-02T14:32:35.726314+0000","last_undegraded":"2024-04-02T14:32:35.726314+0000","last_fullsized":"2024-04-02T14:32:35.726314+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T01:20:53.183052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"90'5","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:33:34.758638+0000","last_change":"2024-04-02T14:32:22.552277+0000","last_active":"2024-04-02T14:33:34.758638+0000","last_peered":"2024-04-02T14:33:34.758638+0000","last_clean":"2024-04-02T14:33:34.758638+0000","last_became_active":"2024-04-02T14:32:22.551882+0000","last_became_peered":"2024-04-02T14:32:22.551882+0000","last_unstale":"2024-04-02T14:33:34.758638+0000","last_undegraded":"2024-04-02T14:33:34.758638+0000","last_fullsized":"2024-04-02T14:33:34.758638+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:30:43.592910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"88'5","reported_seq":111,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:33:55.024217+0000","last_change":"2024-04-02T14:32:21.848258+0000","last_active":"2024-04-02T14:33:55.024217+0000","last_peered":"2024-04-02T14:33:55.024217+0000","last_clean":"2024-04-02T14:33:55.024217+0000","last_became_active":"2024-04-02T14:32:21.847805+0000","last_became_peered":"2024-04-02T14:32:21.847805+0000","last_unstale":"2024-04-02T14:33:55.024217+0000","last_undegraded":"2024-04-02T14:33:55.024217+0000","last_fullsized":"2024-04-02T14:33:55.024217+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:46:07.552786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":87,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"90'7","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:33:34.760673+0000","last_change":"2024-04-02T14:32:22.560835+0000","last_active":"2024-04-02T14:33:34.760673+0000","last_peered":"2024-04-02T14:33:34.760673+0000","last_clean":"2024-04-02T14:33:34.760673+0000","last_became_active":"2024-04-02T14:32:22.560469+0000","last_became_peered":"2024-04-02T14:32:22.560469+0000","last_unstale":"2024-04-02T14:33:34.760673+0000","last_undegraded":"2024-04-02T14:33:34.760673+0000","last_fullsized":"2024-04-02T14:33:34.760673+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T02:29:24.869650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"90'8","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:33:34.742608+0000","last_change":"2024-04-02T14:32:22.538325+0000","last_active":"2024-04-02T14:33:34.742608+0000","last_peered":"2024-04-02T14:33:34.742608+0000","last_clean":"2024-04-02T14:33:34.742608+0000","last_became_active":"2024-04-02T14:32:22.537903+0000","last_became_peered":"2024-04-02T14:32:22.537903+0000","last_unstale":"2024-04-02T14:33:34.742608+0000","last_undegraded":"2024-04-02T14:33:34.742608+0000","last_fullsized":"2024-04-02T14:33:34.742608+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:12:26.586290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"90'6","reported_seq":108,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:33:50.018302+0000","last_change":"2024-04-02T14:32:21.861333+0000","last_active":"2024-04-02T14:33:50.018302+0000","last_peered":"2024-04-02T14:33:50.018302+0000","last_clean":"2024-04-02T14:33:50.018302+0000","last_became_active":"2024-04-02T14:32:21.860909+0000","last_became_peered":"2024-04-02T14:32:21.860909+0000","last_unstale":"2024-04-02T14:33:50.018302+0000","last_undegraded":"2024-04-02T14:33:50.018302+0000","last_fullsized":"2024-04-02T14:33:50.018302+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:21:52.616036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":83,"num_read_kb":83,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"90'8","reported_seq":19,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:33:34.758306+0000","last_change":"2024-04-02T14:32:35.736379+0000","last_active":"2024-04-02T14:33:34.758306+0000","last_peered":"2024-04-02T14:33:34.758306+0000","last_clean":"2024-04-02T14:33:34.758306+0000","last_became_active":"2024-04-02T14:32:35.735156+0000","last_became_peered":"2024-04-02T14:32:35.735156+0000","last_unstale":"2024-04-02T14:33:34.758306+0000","last_undegraded":"2024-04-02T14:33:34.758306+0000","last_fullsized":"2024-04-02T14:33:34.758306+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:16:28.405699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,1],"acting":[9,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.3","version":"88'6","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:32:35.850702+0000","last_change":"2024-04-02T14:32:21.837336+0000","last_active":"2024-04-02T14:32:35.850702+0000","last_peered":"2024-04-02T14:32:35.850702+0000","last_clean":"2024-04-02T14:32:35.850702+0000","last_became_active":"2024-04-02T14:32:21.836921+0000","last_became_peered":"2024-04-02T14:32:21.836921+0000","last_unstale":"2024-04-02T14:32:35.850702+0000","last_undegraded":"2024-04-02T14:32:35.850702+0000","last_fullsized":"2024-04-02T14:32:35.850702+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:34:52.883173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":356,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:32:35.730675+0000","last_change":"2024-04-02T14:31:02.710356+0000","last_active":"2024-04-02T14:32:35.730675+0000","last_peered":"2024-04-02T14:32:35.730675+0000","last_clean":"2024-04-02T14:32:35.730675+0000","last_became_active":"2024-04-02T14:31:02.709669+0000","last_became_peered":"2024-04-02T14:31:02.709669+0000","last_unstale":"2024-04-02T14:32:35.730675+0000","last_undegraded":"2024-04-02T14:32:35.730675+0000","last_fullsized":"2024-04-02T14:32:35.730675+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:27:17.358883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:27:17.358883+0000","last_clean_scrub_stamp":"2024-04-02T14:27:17.358883+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:45:55.893284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"90'5","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:33:34.740733+0000","last_change":"2024-04-02T14:32:22.545502+0000","last_active":"2024-04-02T14:33:34.740733+0000","last_peered":"2024-04-02T14:33:34.740733+0000","last_clean":"2024-04-02T14:33:34.740733+0000","last_became_active":"2024-04-02T14:32:22.545111+0000","last_became_peered":"2024-04-02T14:32:22.545111+0000","last_unstale":"2024-04-02T14:33:34.740733+0000","last_undegraded":"2024-04-02T14:33:34.740733+0000","last_fullsized":"2024-04-02T14:33:34.740733+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:02:32.178334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"88'4","reported_seq":15,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:32:35.736582+0000","last_change":"2024-04-02T14:32:35.736582+0000","last_active":"2024-04-02T14:32:35.736582+0000","last_peered":"2024-04-02T14:32:35.736582+0000","last_clean":"2024-04-02T14:32:35.736582+0000","last_became_active":"2024-04-02T14:32:35.733368+0000","last_became_peered":"2024-04-02T14:32:35.733368+0000","last_unstale":"2024-04-02T14:32:35.736582+0000","last_undegraded":"2024-04-02T14:32:35.736582+0000","last_fullsized":"2024-04-02T14:32:35.736582+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T02:14:35.713351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,8],"acting":[11,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"88'4","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:32:35.731654+0000","last_change":"2024-04-02T14:32:21.860226+0000","last_active":"2024-04-02T14:32:35.731654+0000","last_peered":"2024-04-02T14:32:35.731654+0000","last_clean":"2024-04-02T14:32:35.731654+0000","last_became_active":"2024-04-02T14:32:21.859854+0000","last_became_peered":"2024-04-02T14:32:21.859854+0000","last_unstale":"2024-04-02T14:32:35.731654+0000","last_undegraded":"2024-04-02T14:32:35.731654+0000","last_fullsized":"2024-04-02T14:32:35.731654+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:30:36.710677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"90'9","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:33:34.746141+0000","last_change":"2024-04-02T14:32:22.552287+0000","last_active":"2024-04-02T14:33:34.746141+0000","last_peered":"2024-04-02T14:33:34.746141+0000","last_clean":"2024-04-02T14:33:34.746141+0000","last_became_active":"2024-04-02T14:32:22.551719+0000","last_became_peered":"2024-04-02T14:32:22.551719+0000","last_unstale":"2024-04-02T14:33:34.746141+0000","last_undegraded":"2024-04-02T14:33:34.746141+0000","last_fullsized":"2024-04-02T14:33:34.746141+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:26:12.007600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"90'14","reported_seq":33,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:32:35.856522+0000","last_change":"2024-04-02T14:32:21.851671+0000","last_active":"2024-04-02T14:32:35.856522+0000","last_peered":"2024-04-02T14:32:35.856522+0000","last_clean":"2024-04-02T14:32:35.856522+0000","last_became_active":"2024-04-02T14:32:21.851297+0000","last_became_peered":"2024-04-02T14:32:21.851297+0000","last_unstale":"2024-04-02T14:32:35.856522+0000","last_undegraded":"2024-04-02T14:32:35.856522+0000","last_fullsized":"2024-04-02T14:32:35.856522+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:51:20.232086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"90'5","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:33:34.762221+0000","last_change":"2024-04-02T14:32:22.554554+0000","last_active":"2024-04-02T14:33:34.762221+0000","last_peered":"2024-04-02T14:33:34.762221+0000","last_clean":"2024-04-02T14:33:34.762221+0000","last_became_active":"2024-04-02T14:32:22.554194+0000","last_became_peered":"2024-04-02T14:32:22.554194+0000","last_unstale":"2024-04-02T14:33:34.762221+0000","last_undegraded":"2024-04-02T14:33:34.762221+0000","last_fullsized":"2024-04-02T14:33:34.762221+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:27:51.382441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"88'7","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:32:35.850845+0000","last_change":"2024-04-02T14:32:21.850346+0000","last_active":"2024-04-02T14:32:35.850845+0000","last_peered":"2024-04-02T14:32:35.850845+0000","last_clean":"2024-04-02T14:32:35.850845+0000","last_became_active":"2024-04-02T14:32:21.850084+0000","last_became_peered":"2024-04-02T14:32:21.850084+0000","last_unstale":"2024-04-02T14:32:35.850845+0000","last_undegraded":"2024-04-02T14:32:35.850845+0000","last_fullsized":"2024-04-02T14:32:35.850845+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:56:17.774702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"90'7","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:33:34.768581+0000","last_change":"2024-04-02T14:32:22.557054+0000","last_active":"2024-04-02T14:33:34.768581+0000","last_peered":"2024-04-02T14:33:34.768581+0000","last_clean":"2024-04-02T14:33:34.768581+0000","last_became_active":"2024-04-02T14:32:22.556335+0000","last_became_peered":"2024-04-02T14:32:22.556335+0000","last_unstale":"2024-04-02T14:33:34.768581+0000","last_undegraded":"2024-04-02T14:33:34.768581+0000","last_fullsized":"2024-04-02T14:33:34.768581+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:06:48.907197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"88'2","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:32:35.722876+0000","last_change":"2024-04-02T14:32:21.860260+0000","last_active":"2024-04-02T14:32:35.722876+0000","last_peered":"2024-04-02T14:32:35.722876+0000","last_clean":"2024-04-02T14:32:35.722876+0000","last_became_active":"2024-04-02T14:32:21.859922+0000","last_became_peered":"2024-04-02T14:32:21.859922+0000","last_unstale":"2024-04-02T14:32:35.722876+0000","last_undegraded":"2024-04-02T14:32:35.722876+0000","last_fullsized":"2024-04-02T14:32:35.722876+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:07:45.850148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"90'2","reported_seq":19,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:33:34.707329+0000","last_change":"2024-04-02T14:32:22.536063+0000","last_active":"2024-04-02T14:33:34.707329+0000","last_peered":"2024-04-02T14:33:34.707329+0000","last_clean":"2024-04-02T14:33:34.707329+0000","last_became_active":"2024-04-02T14:32:22.534678+0000","last_became_peered":"2024-04-02T14:32:22.534678+0000","last_unstale":"2024-04-02T14:33:34.707329+0000","last_undegraded":"2024-04-02T14:33:34.707329+0000","last_fullsized":"2024-04-02T14:33:34.707329+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T02:07:26.186216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"88'7","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:32:35.736679+0000","last_change":"2024-04-02T14:32:21.848554+0000","last_active":"2024-04-02T14:32:35.736679+0000","last_peered":"2024-04-02T14:32:35.736679+0000","last_clean":"2024-04-02T14:32:35.736679+0000","last_became_active":"2024-04-02T14:32:21.848177+0000","last_became_peered":"2024-04-02T14:32:21.848177+0000","last_unstale":"2024-04-02T14:32:35.736679+0000","last_undegraded":"2024-04-02T14:32:35.736679+0000","last_fullsized":"2024-04-02T14:32:35.736679+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:47:04.213286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"90'9","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:33:34.763600+0000","last_change":"2024-04-02T14:32:22.547246+0000","last_active":"2024-04-02T14:33:34.763600+0000","last_peered":"2024-04-02T14:33:34.763600+0000","last_clean":"2024-04-02T14:33:34.763600+0000","last_became_active":"2024-04-02T14:32:22.546721+0000","last_became_peered":"2024-04-02T14:32:22.546721+0000","last_unstale":"2024-04-02T14:33:34.763600+0000","last_undegraded":"2024-04-02T14:33:34.763600+0000","last_fullsized":"2024-04-02T14:33:34.763600+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:11:30.615723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"88'3","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:32:35.743474+0000","last_change":"2024-04-02T14:32:35.743474+0000","last_active":"2024-04-02T14:32:35.743474+0000","last_peered":"2024-04-02T14:32:35.743474+0000","last_clean":"2024-04-02T14:32:35.743474+0000","last_became_active":"2024-04-02T14:32:35.738522+0000","last_became_peered":"2024-04-02T14:32:35.738522+0000","last_unstale":"2024-04-02T14:32:35.743474+0000","last_undegraded":"2024-04-02T14:32:35.743474+0000","last_fullsized":"2024-04-02T14:32:35.743474+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T01:14:43.989279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"90'8","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:33:34.766374+0000","last_change":"2024-04-02T14:32:22.561787+0000","last_active":"2024-04-02T14:33:34.766374+0000","last_peered":"2024-04-02T14:33:34.766374+0000","last_clean":"2024-04-02T14:33:34.766374+0000","last_became_active":"2024-04-02T14:32:22.561284+0000","last_became_peered":"2024-04-02T14:32:22.561284+0000","last_unstale":"2024-04-02T14:33:34.766374+0000","last_undegraded":"2024-04-02T14:33:34.766374+0000","last_fullsized":"2024-04-02T14:33:34.766374+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:46:06.450097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"88'4","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:32:35.731779+0000","last_change":"2024-04-02T14:32:21.841340+0000","last_active":"2024-04-02T14:32:35.731779+0000","last_peered":"2024-04-02T14:32:35.731779+0000","last_clean":"2024-04-02T14:32:35.731779+0000","last_became_active":"2024-04-02T14:32:21.840937+0000","last_became_peered":"2024-04-02T14:32:21.840937+0000","last_unstale":"2024-04-02T14:32:35.731779+0000","last_undegraded":"2024-04-02T14:32:35.731779+0000","last_fullsized":"2024-04-02T14:32:35.731779+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T01:47:44.834440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"90'5","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:33:34.767552+0000","last_change":"2024-04-02T14:32:22.538890+0000","last_active":"2024-04-02T14:33:34.767552+0000","last_peered":"2024-04-02T14:33:34.767552+0000","last_clean":"2024-04-02T14:33:34.767552+0000","last_became_active":"2024-04-02T14:32:22.538512+0000","last_became_peered":"2024-04-02T14:32:22.538512+0000","last_unstale":"2024-04-02T14:33:34.767552+0000","last_undegraded":"2024-04-02T14:33:34.767552+0000","last_fullsized":"2024-04-02T14:33:34.767552+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:59:23.792851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"88'5","reported_seq":113,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:33:54.951267+0000","last_change":"2024-04-02T14:32:21.862727+0000","last_active":"2024-04-02T14:33:54.951267+0000","last_peered":"2024-04-02T14:33:54.951267+0000","last_clean":"2024-04-02T14:33:54.951267+0000","last_became_active":"2024-04-02T14:32:21.862405+0000","last_became_peered":"2024-04-02T14:32:21.862405+0000","last_unstale":"2024-04-02T14:33:54.951267+0000","last_undegraded":"2024-04-02T14:33:54.951267+0000","last_fullsized":"2024-04-02T14:33:54.951267+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:36:54.849964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":89,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"90'7","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:33:34.756023+0000","last_change":"2024-04-02T14:32:22.540077+0000","last_active":"2024-04-02T14:33:34.756023+0000","last_peered":"2024-04-02T14:33:34.756023+0000","last_clean":"2024-04-02T14:33:34.756023+0000","last_became_active":"2024-04-02T14:32:22.539513+0000","last_became_peered":"2024-04-02T14:32:22.539513+0000","last_unstale":"2024-04-02T14:33:34.756023+0000","last_undegraded":"2024-04-02T14:33:34.756023+0000","last_fullsized":"2024-04-02T14:33:34.756023+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T02:00:39.819963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"88'3","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:32:35.731129+0000","last_change":"2024-04-02T14:32:21.860310+0000","last_active":"2024-04-02T14:32:35.731129+0000","last_peered":"2024-04-02T14:32:35.731129+0000","last_clean":"2024-04-02T14:32:35.731129+0000","last_became_active":"2024-04-02T14:32:21.859955+0000","last_became_peered":"2024-04-02T14:32:21.859955+0000","last_unstale":"2024-04-02T14:32:35.731129+0000","last_undegraded":"2024-04-02T14:32:35.731129+0000","last_fullsized":"2024-04-02T14:32:35.731129+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:11:32.732603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"90'7","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:33:34.760162+0000","last_change":"2024-04-02T14:32:22.552134+0000","last_active":"2024-04-02T14:33:34.760162+0000","last_peered":"2024-04-02T14:33:34.760162+0000","last_clean":"2024-04-02T14:33:34.760162+0000","last_became_active":"2024-04-02T14:32:22.551695+0000","last_became_peered":"2024-04-02T14:32:22.551695+0000","last_unstale":"2024-04-02T14:33:34.760162+0000","last_undegraded":"2024-04-02T14:33:34.760162+0000","last_fullsized":"2024-04-02T14:33:34.760162+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:09:36.902243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"88'5","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:32:35.845910+0000","last_change":"2024-04-02T14:32:21.857571+0000","last_active":"2024-04-02T14:32:35.845910+0000","last_peered":"2024-04-02T14:32:35.845910+0000","last_clean":"2024-04-02T14:32:35.845910+0000","last_became_active":"2024-04-02T14:32:21.856935+0000","last_became_peered":"2024-04-02T14:32:21.856935+0000","last_unstale":"2024-04-02T14:32:35.845910+0000","last_undegraded":"2024-04-02T14:32:35.845910+0000","last_fullsized":"2024-04-02T14:32:35.845910+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:55:46.079097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"90'9","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:33:34.741354+0000","last_change":"2024-04-02T14:32:22.534708+0000","last_active":"2024-04-02T14:33:34.741354+0000","last_peered":"2024-04-02T14:33:34.741354+0000","last_clean":"2024-04-02T14:33:34.741354+0000","last_became_active":"2024-04-02T14:32:22.534365+0000","last_became_peered":"2024-04-02T14:32:22.534365+0000","last_unstale":"2024-04-02T14:33:34.741354+0000","last_undegraded":"2024-04-02T14:33:34.741354+0000","last_fullsized":"2024-04-02T14:33:34.741354+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:34:47.435729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"90'9","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:33:34.745155+0000","last_change":"2024-04-02T14:32:22.538317+0000","last_active":"2024-04-02T14:33:34.745155+0000","last_peered":"2024-04-02T14:33:34.745155+0000","last_clean":"2024-04-02T14:33:34.745155+0000","last_became_active":"2024-04-02T14:32:22.537939+0000","last_became_peered":"2024-04-02T14:32:22.537939+0000","last_unstale":"2024-04-02T14:33:34.745155+0000","last_undegraded":"2024-04-02T14:33:34.745155+0000","last_fullsized":"2024-04-02T14:33:34.745155+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:31:47.075326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"90'12","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:33:34.747907+0000","last_change":"2024-04-02T14:32:35.852202+0000","last_active":"2024-04-02T14:33:34.747907+0000","last_peered":"2024-04-02T14:33:34.747907+0000","last_clean":"2024-04-02T14:33:34.747907+0000","last_became_active":"2024-04-02T14:32:35.851785+0000","last_became_peered":"2024-04-02T14:32:35.851785+0000","last_unstale":"2024-04-02T14:33:34.747907+0000","last_undegraded":"2024-04-02T14:33:34.747907+0000","last_fullsized":"2024-04-02T14:33:34.747907+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:09:17.410727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,4],"acting":[2,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"90'8","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:33:34.765856+0000","last_change":"2024-04-02T14:32:22.556567+0000","last_active":"2024-04-02T14:33:34.765856+0000","last_peered":"2024-04-02T14:33:34.765856+0000","last_clean":"2024-04-02T14:33:34.765856+0000","last_became_active":"2024-04-02T14:32:22.556267+0000","last_became_peered":"2024-04-02T14:32:22.556267+0000","last_unstale":"2024-04-02T14:33:34.765856+0000","last_undegraded":"2024-04-02T14:33:34.765856+0000","last_fullsized":"2024-04-02T14:33:34.765856+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:35:30.039013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"90'7","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:33:34.772435+0000","last_change":"2024-04-02T14:32:35.848125+0000","last_active":"2024-04-02T14:33:34.772435+0000","last_peered":"2024-04-02T14:33:34.772435+0000","last_clean":"2024-04-02T14:33:34.772435+0000","last_became_active":"2024-04-02T14:32:35.847681+0000","last_became_peered":"2024-04-02T14:32:35.847681+0000","last_unstale":"2024-04-02T14:33:34.772435+0000","last_undegraded":"2024-04-02T14:33:34.772435+0000","last_fullsized":"2024-04-02T14:33:34.772435+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:51:38.040261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"90'4","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:33:34.766623+0000","last_change":"2024-04-02T14:32:22.558584+0000","last_active":"2024-04-02T14:33:34.766623+0000","last_peered":"2024-04-02T14:33:34.766623+0000","last_clean":"2024-04-02T14:33:34.766623+0000","last_became_active":"2024-04-02T14:32:22.557382+0000","last_became_peered":"2024-04-02T14:32:22.557382+0000","last_unstale":"2024-04-02T14:33:34.766623+0000","last_undegraded":"2024-04-02T14:33:34.766623+0000","last_fullsized":"2024-04-02T14:33:34.766623+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:09:25.107411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"90'6","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:33:34.740344+0000","last_change":"2024-04-02T14:32:22.538318+0000","last_active":"2024-04-02T14:33:34.740344+0000","last_peered":"2024-04-02T14:33:34.740344+0000","last_clean":"2024-04-02T14:33:34.740344+0000","last_became_active":"2024-04-02T14:32:22.538011+0000","last_became_peered":"2024-04-02T14:32:22.538011+0000","last_unstale":"2024-04-02T14:33:34.740344+0000","last_undegraded":"2024-04-02T14:33:34.740344+0000","last_fullsized":"2024-04-02T14:33:34.740344+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:11:21.269106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"90'15","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:33:34.735078+0000","last_change":"2024-04-02T14:32:22.535645+0000","last_active":"2024-04-02T14:33:34.735078+0000","last_peered":"2024-04-02T14:33:34.735078+0000","last_clean":"2024-04-02T14:33:34.735078+0000","last_became_active":"2024-04-02T14:32:22.535314+0000","last_became_peered":"2024-04-02T14:32:22.535314+0000","last_unstale":"2024-04-02T14:33:34.735078+0000","last_undegraded":"2024-04-02T14:33:34.735078+0000","last_fullsized":"2024-04-02T14:33:34.735078+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:51:27.879015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"90'2","reported_seq":19,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:33:34.760189+0000","last_change":"2024-04-02T14:32:22.553959+0000","last_active":"2024-04-02T14:33:34.760189+0000","last_peered":"2024-04-02T14:33:34.760189+0000","last_clean":"2024-04-02T14:33:34.760189+0000","last_became_active":"2024-04-02T14:32:22.553389+0000","last_became_peered":"2024-04-02T14:32:22.553389+0000","last_unstale":"2024-04-02T14:33:34.760189+0000","last_undegraded":"2024-04-02T14:33:34.760189+0000","last_fullsized":"2024-04-02T14:33:34.760189+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:55:00.552600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"90'6","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:33:34.764651+0000","last_change":"2024-04-02T14:32:22.551769+0000","last_active":"2024-04-02T14:33:34.764651+0000","last_peered":"2024-04-02T14:33:34.764651+0000","last_clean":"2024-04-02T14:33:34.764651+0000","last_became_active":"2024-04-02T14:32:22.551422+0000","last_became_peered":"2024-04-02T14:32:22.551422+0000","last_unstale":"2024-04-02T14:33:34.764651+0000","last_undegraded":"2024-04-02T14:33:34.764651+0000","last_fullsized":"2024-04-02T14:33:34.764651+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T01:15:07.262927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"90'8","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:33:34.760669+0000","last_change":"2024-04-02T14:32:22.548264+0000","last_active":"2024-04-02T14:33:34.760669+0000","last_peered":"2024-04-02T14:33:34.760669+0000","last_clean":"2024-04-02T14:33:34.760669+0000","last_became_active":"2024-04-02T14:32:22.547293+0000","last_became_peered":"2024-04-02T14:32:22.547293+0000","last_unstale":"2024-04-02T14:33:34.760669+0000","last_undegraded":"2024-04-02T14:33:34.760669+0000","last_fullsized":"2024-04-02T14:33:34.760669+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:53:19.896243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"90'10","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:33:34.766530+0000","last_change":"2024-04-02T14:32:22.556131+0000","last_active":"2024-04-02T14:33:34.766530+0000","last_peered":"2024-04-02T14:33:34.766530+0000","last_clean":"2024-04-02T14:33:34.766530+0000","last_became_active":"2024-04-02T14:32:22.555763+0000","last_became_peered":"2024-04-02T14:32:22.555763+0000","last_unstale":"2024-04-02T14:33:34.766530+0000","last_undegraded":"2024-04-02T14:33:34.766530+0000","last_fullsized":"2024-04-02T14:33:34.766530+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:45:22.485403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"90'11","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:33:34.769677+0000","last_change":"2024-04-02T14:32:22.562093+0000","last_active":"2024-04-02T14:33:34.769677+0000","last_peered":"2024-04-02T14:33:34.769677+0000","last_clean":"2024-04-02T14:33:34.769677+0000","last_became_active":"2024-04-02T14:32:22.561663+0000","last_became_peered":"2024-04-02T14:32:22.561663+0000","last_unstale":"2024-04-02T14:33:34.769677+0000","last_undegraded":"2024-04-02T14:33:34.769677+0000","last_fullsized":"2024-04-02T14:33:34.769677+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:14:25.209276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"90'8","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-02T14:33:34.748961+0000","last_change":"2024-04-02T14:32:22.541964+0000","last_active":"2024-04-02T14:33:34.748961+0000","last_peered":"2024-04-02T14:33:34.748961+0000","last_clean":"2024-04-02T14:33:34.748961+0000","last_became_active":"2024-04-02T14:32:22.541549+0000","last_became_peered":"2024-04-02T14:32:22.541549+0000","last_unstale":"2024-04-02T14:33:34.748961+0000","last_undegraded":"2024-04-02T14:33:34.748961+0000","last_fullsized":"2024-04-02T14:33:34.748961+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:04:56.783158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":68781,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":873,"num_read_kb":502,"num_write":248,"num_write_kb":141,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":44049,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":565248,"data_stored":206343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":215,"ondisk_log_size":215,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":83,"seq":356482285601,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32148,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713004,"statfs":{"total":95995035648,"available":95962116096,"internally_reserved":0,"allocated":995328,"data_stored":474721,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":75,"seq":322122547239,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36992,"kb_used_data":1656,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708160,"statfs":{"total":95995035648,"available":95957155840,"internally_reserved":0,"allocated":1695744,"data_stored":1127577,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":36174273},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743467,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36332,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708820,"statfs":{"total":95995035648,"available":95957831680,"internally_reserved":0,"allocated":1019904,"data_stored":474651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":61,"seq":261993005107,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32160,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712992,"statfs":{"total":95995035648,"available":95962103808,"internally_reserved":0,"allocated":1007616,"data_stored":474357,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299448,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32752,"kb_used_data":1576,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712400,"statfs":{"total":95995035648,"available":95961497600,"internally_reserved":0,"allocated":1613824,"data_stored":1068220,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495675,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36316,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708836,"statfs":{"total":95995035648,"available":95957848064,"internally_reserved":0,"allocated":1003520,"data_stored":474368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574937,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36376,"kb_used_data":1040,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708776,"statfs":{"total":95995035648,"available":95957786624,"internally_reserved":0,"allocated":1064960,"data_stored":536576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738462,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36336,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708816,"statfs":{"total":95995035648,"available":95957827584,"internally_reserved":0,"allocated":1024000,"data_stored":475707,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411412,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36328,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708824,"statfs":{"total":95995035648,"available":95957835776,"internally_reserved":0,"allocated":1015808,"data_stored":476033,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182476,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28012,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717140,"statfs":{"total":95995035648,"available":95966351360,"internally_reserved":0,"allocated":1019904,"data_stored":474193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920841,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32232,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712920,"statfs":{"total":95995035648,"available":95962030080,"internally_reserved":0,"allocated":1081344,"data_stored":536728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757313,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28604,"kb_used_data":1588,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716548,"statfs":{"total":95995035648,"available":95965745152,"internally_reserved":0,"allocated":1626112,"data_stored":1067366,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":63047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":2504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":63199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":4206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":828,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":63563,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-02T14:33:59.607 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-04-02T14:33:59.607 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-04-02T14:33:59.608 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-04-02T14:33:59.609 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:191d963ac9de7322da8aff20b567b3fa458e2ef6 shell --fsid 6449448a-f0fc-11ee-b647-cb9ed24678a4 -- ceph health --format=json 2024-04-02T14:34:00.109 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T14:34:00.110 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:33:59 smithi092 ceph-mon[36497]: from='client.24892 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-02T14:34:00.110 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:33:59 smithi092 ceph-mon[36497]: pgmap v362: 97 pgs: 97 active+clean; 644 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-02T14:34:00.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:33:59 smithi177 ceph-mon[36431]: from='client.24892 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-02T14:34:00.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:33:59 smithi177 ceph-mon[36431]: pgmap v362: 97 pgs: 97 active+clean; 644 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-02T14:34:00.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:33:59 smithi090 ceph-mon[31674]: from='client.24892 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-02T14:34:00.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:33:59 smithi090 ceph-mon[31674]: pgmap v362: 97 pgs: 97 active+clean; 644 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-02T14:34:02.052 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:34:01 smithi090 ceph-mon[31674]: pgmap v363: 97 pgs: 97 active+clean; 644 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-02T14:34:02.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:34:01 smithi177 ceph-mon[36431]: pgmap v363: 97 pgs: 97 active+clean; 644 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-02T14:34:02.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:02.316+0000 7f51d4800700 1 -- 172.21.15.90:0/2257348551 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f51cc1051a0 msgr2=0x7f51cc1116b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:02.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:02.316+0000 7f51d4800700 1 --2- 172.21.15.90:0/2257348551 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f51cc1051a0 0x7f51cc1116b0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f51c800b0a0 tx=0x7f51c800b3b0 comp rx=0 tx=0).stop 2024-04-02T14:34:02.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:02.316+0000 7f51d4800700 1 -- 172.21.15.90:0/2257348551 shutdown_connections 2024-04-02T14:34:02.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:02.316+0000 7f51d4800700 1 --2- 172.21.15.90:0/2257348551 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f51cc1051a0 0x7f51cc1116b0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:02.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:02.316+0000 7f51d4800700 1 --2- 172.21.15.90:0/2257348551 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f51cc10a7e0 0x7f51cc10abc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:02.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:02.316+0000 7f51d4800700 1 --2- 172.21.15.90:0/2257348551 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f51cc1047e0 0x7f51cc104c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:02.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:02.316+0000 7f51d4800700 1 -- 172.21.15.90:0/2257348551 >> 172.21.15.90:0/2257348551 conn(0x7f51cc1002d0 msgr2=0x7f51cc1026f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:34:02.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:02.316+0000 7f51d4800700 1 -- 172.21.15.90:0/2257348551 shutdown_connections 2024-04-02T14:34:02.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:02.317+0000 7f51d4800700 1 -- 172.21.15.90:0/2257348551 wait complete. 2024-04-02T14:34:02.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:02.318+0000 7f51d4800700 1 Processor -- start 2024-04-02T14:34:02.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:02.318+0000 7f51d4800700 1 -- start start 2024-04-02T14:34:02.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:02.318+0000 7f51d4800700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f51cc1047e0 0x7f51cc19af30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:02.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:02.318+0000 7f51d4800700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f51cc1051a0 0x7f51cc19b470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:02.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:02.318+0000 7f51d4800700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f51cc10a7e0 0x7f51cc19f800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:02.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:02.318+0000 7f51d4800700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f51cc114bc0 con 0x7f51cc1051a0 2024-04-02T14:34:02.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:02.318+0000 7f51d4800700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f51cc114a40 con 0x7f51cc10a7e0 2024-04-02T14:34:02.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:02.318+0000 7f51d4800700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f51cc114d40 con 0x7f51cc1047e0 2024-04-02T14:34:02.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:02.318+0000 7f51d1400700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f51cc1051a0 0x7f51cc19b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:02.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:02.319+0000 7f51d1400700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f51cc1051a0 0x7f51cc19b470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:57342/0 (socket says 172.21.15.90:57342) 2024-04-02T14:34:02.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:02.319+0000 7f51d1400700 1 -- 172.21.15.90:0/4022640292 learned_addr learned my addr 172.21.15.90:0/4022640292 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:34:02.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:02.319+0000 7f51d1e00700 1 --2- 172.21.15.90:0/4022640292 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f51cc1047e0 0x7f51cc19af30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:02.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:02.319+0000 7f51d2800700 1 --2- 172.21.15.90:0/4022640292 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f51cc10a7e0 0x7f51cc19f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:02.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:02.319+0000 7f51d2800700 1 -- 172.21.15.90:0/4022640292 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f51cc1047e0 msgr2=0x7f51cc19af30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:02.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:02.319+0000 7f51d2800700 1 --2- 172.21.15.90:0/4022640292 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f51cc1047e0 0x7f51cc19af30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:02.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:02.319+0000 7f51d2800700 1 -- 172.21.15.90:0/4022640292 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f51cc1051a0 msgr2=0x7f51cc19b470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:02.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:02.319+0000 7f51d2800700 1 --2- 172.21.15.90:0/4022640292 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f51cc1051a0 0x7f51cc19b470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:02.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:02.319+0000 7f51d2800700 1 -- 172.21.15.90:0/4022640292 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51cc19ff40 con 0x7f51cc10a7e0 2024-04-02T14:34:02.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:02.319+0000 7f51d1400700 1 --2- 172.21.15.90:0/4022640292 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f51cc1051a0 0x7f51cc19b470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:34:02.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:02.319+0000 7f51d2800700 1 --2- 172.21.15.90:0/4022640292 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f51cc10a7e0 0x7f51cc19f800 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f51c800b070 tx=0x7f51c8004c80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:34:02.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:02.320+0000 7f51c6a00700 1 -- 172.21.15.90:0/4022640292 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51c8021070 con 0x7f51cc10a7e0 2024-04-02T14:34:02.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:02.320+0000 7f51c6a00700 1 -- 172.21.15.90:0/4022640292 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f51c8007b80 con 0x7f51cc10a7e0 2024-04-02T14:34:02.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:02.320+0000 7f51d4800700 1 -- 172.21.15.90:0/4022640292 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51cc1a01d0 con 0x7f51cc10a7e0 2024-04-02T14:34:02.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:02.320+0000 7f51c6a00700 1 -- 172.21.15.90:0/4022640292 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51c80052c0 con 0x7f51cc10a7e0 2024-04-02T14:34:02.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:02.320+0000 7f51d4800700 1 -- 172.21.15.90:0/4022640292 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51cc1a7ab0 con 0x7f51cc10a7e0 2024-04-02T14:34:02.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:02.321+0000 7f51d4800700 1 -- 172.21.15.90:0/4022640292 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f51cc105f20 con 0x7f51cc10a7e0 2024-04-02T14:34:02.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:02.323+0000 7f51c6a00700 1 -- 172.21.15.90:0/4022640292 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f51c8024070 con 0x7f51cc10a7e0 2024-04-02T14:34:02.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:02.324+0000 7f51c6a00700 1 --2- 172.21.15.90:0/4022640292 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f51b40a3c40 0x7f51b40a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:02.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:02.324+0000 7f51c6a00700 1 -- 172.21.15.90:0/4022640292 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8248+0+0 (secure 0 0 0) 0x7f51c80d1700 con 0x7f51cc10a7e0 2024-04-02T14:34:02.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:02.324+0000 7f51d1e00700 1 --2- 172.21.15.90:0/4022640292 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f51b40a3c40 0x7f51b40a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:02.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:02.325+0000 7f51d1e00700 1 --2- 172.21.15.90:0/4022640292 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f51b40a3c40 0x7f51b40a6100 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f51c0009be0 tx=0x7f51c0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:34:02.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:02.329+0000 7f51c6a00700 1 -- 172.21.15.90:0/4022640292 <== mon.2 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7f51c809e100 con 0x7f51cc10a7e0 2024-04-02T14:34:02.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:34:01 smithi092 ceph-mon[36497]: pgmap v363: 97 pgs: 97 active+clean; 644 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-02T14:34:02.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:02.601+0000 7f51d4800700 1 -- 172.21.15.90:0/4022640292 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f51cc1a7c80 con 0x7f51cc10a7e0 2024-04-02T14:34:02.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:02.602+0000 7f51c6a00700 1 -- 172.21.15.90:0/4022640292 <== mon.2 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7f51c809da00 con 0x7f51cc10a7e0 2024-04-02T14:34:02.603 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-02T14:34:02.603 INFO:teuthology.orchestra.run.smithi090.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-02T14:34:02.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:02.605+0000 7f51d4800700 1 -- 172.21.15.90:0/4022640292 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f51b40a3c40 msgr2=0x7f51b40a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:02.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:02.605+0000 7f51d4800700 1 --2- 172.21.15.90:0/4022640292 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f51b40a3c40 0x7f51b40a6100 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f51c0009be0 tx=0x7f51c0009260 comp rx=0 tx=0).stop 2024-04-02T14:34:02.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:02.605+0000 7f51d4800700 1 -- 172.21.15.90:0/4022640292 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f51cc10a7e0 msgr2=0x7f51cc19f800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:02.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:02.605+0000 7f51d4800700 1 --2- 172.21.15.90:0/4022640292 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f51cc10a7e0 0x7f51cc19f800 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f51c800b070 tx=0x7f51c8004c80 comp rx=0 tx=0).stop 2024-04-02T14:34:02.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:02.605+0000 7f51d4800700 1 -- 172.21.15.90:0/4022640292 shutdown_connections 2024-04-02T14:34:02.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:02.605+0000 7f51d4800700 1 --2- 172.21.15.90:0/4022640292 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f51b40a3c40 0x7f51b40a6100 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:02.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:02.605+0000 7f51d4800700 1 --2- 172.21.15.90:0/4022640292 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f51cc10a7e0 0x7f51cc19f800 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:02.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:02.605+0000 7f51d4800700 1 --2- 172.21.15.90:0/4022640292 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f51cc1047e0 0x7f51cc19af30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:02.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:02.605+0000 7f51d4800700 1 --2- 172.21.15.90:0/4022640292 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f51cc1051a0 0x7f51cc19b470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:02.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:02.605+0000 7f51d4800700 1 -- 172.21.15.90:0/4022640292 >> 172.21.15.90:0/4022640292 conn(0x7f51cc1002d0 msgr2=0x7f51cc1007b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:34:02.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:02.605+0000 7f51d4800700 1 -- 172.21.15.90:0/4022640292 shutdown_connections 2024-04-02T14:34:02.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:02.605+0000 7f51d4800700 1 -- 172.21.15.90:0/4022640292 wait complete. 2024-04-02T14:34:03.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:34:02 smithi177 ceph-mon[36431]: from='client.? 172.21.15.90:0/4022640292' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-02T14:34:03.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:34:02 smithi090 ceph-mon[31674]: from='client.? 172.21.15.90:0/4022640292' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-02T14:34:03.360 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:34:02 smithi092 ceph-mon[36497]: from='client.? 172.21.15.90:0/4022640292' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-02T14:34:03.432 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-04-02T14:34:03.433 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T14:34:03.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:03.651+0000 7f729be00640 1 -- 172.21.15.90:0/3999113511 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f729c109aa0 msgr2=0x7f729c1103c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:03.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:03.651+0000 7f729be00640 1 --2- 172.21.15.90:0/3999113511 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f729c109aa0 0x7f729c1103c0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f7294009ca0 tx=0x7f729402c100 comp rx=0 tx=0).stop 2024-04-02T14:34:03.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:03.651+0000 7f729be00640 1 -- 172.21.15.90:0/3999113511 shutdown_connections 2024-04-02T14:34:03.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:03.651+0000 7f729be00640 1 --2- 172.21.15.90:0/3999113511 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f729c109aa0 0x7f729c1103c0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:03.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:03.651+0000 7f729be00640 1 --2- 172.21.15.90:0/3999113511 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f729c1090e0 0x7f729c109560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:03.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:03.651+0000 7f729be00640 1 --2- 172.21.15.90:0/3999113511 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f729c107f10 0x7f729c108350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:03.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:03.651+0000 7f729be00640 1 -- 172.21.15.90:0/3999113511 >> 172.21.15.90:0/3999113511 conn(0x7f729c076f10 msgr2=0x7f729c079330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:34:03.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:03.651+0000 7f729be00640 1 -- 172.21.15.90:0/3999113511 shutdown_connections 2024-04-02T14:34:03.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:03.651+0000 7f729be00640 1 -- 172.21.15.90:0/3999113511 wait complete. 2024-04-02T14:34:03.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:03.652+0000 7f729be00640 1 Processor -- start 2024-04-02T14:34:03.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:03.652+0000 7f729be00640 1 -- start start 2024-04-02T14:34:03.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:03.653+0000 7f729be00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f729c107f10 0x7f729c19e1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:03.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:03.653+0000 7f729be00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f729c1090e0 0x7f729c19e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:03.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:03.653+0000 7f729be00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f729c109aa0 0x7f729c1a5760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:03.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:03.653+0000 7f729be00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f729c113bd0 con 0x7f729c109aa0 2024-04-02T14:34:03.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:03.653+0000 7f729be00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f729c113a50 con 0x7f729c107f10 2024-04-02T14:34:03.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:03.653+0000 7f729be00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f729c113d50 con 0x7f729c1090e0 2024-04-02T14:34:03.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:03.653+0000 7f729b400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f729c109aa0 0x7f729c1a5760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:03.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:03.653+0000 7f729aa00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f729c107f10 0x7f729c19e1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:03.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:03.653+0000 7f7293e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f729c1090e0 0x7f729c19e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:03.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:03.653+0000 7f729aa00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f729c107f10 0x7f729c19e1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:54780/0 (socket says 172.21.15.90:54780) 2024-04-02T14:34:03.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:03.653+0000 7f729aa00640 1 -- 172.21.15.90:0/3768509317 learned_addr learned my addr 172.21.15.90:0/3768509317 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:34:03.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:03.654+0000 7f729aa00640 1 -- 172.21.15.90:0/3768509317 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f729c1090e0 msgr2=0x7f729c19e6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:03.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:03.654+0000 7f729aa00640 1 --2- 172.21.15.90:0/3768509317 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f729c1090e0 0x7f729c19e6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:03.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:03.654+0000 7f729aa00640 1 -- 172.21.15.90:0/3768509317 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f729c109aa0 msgr2=0x7f729c1a5760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:03.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:03.654+0000 7f729aa00640 1 --2- 172.21.15.90:0/3768509317 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f729c109aa0 0x7f729c1a5760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:03.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:03.654+0000 7f729aa00640 1 -- 172.21.15.90:0/3768509317 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f729c1a5dd0 con 0x7f729c107f10 2024-04-02T14:34:03.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:03.654+0000 7f729b400640 1 --2- 172.21.15.90:0/3768509317 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f729c109aa0 0x7f729c1a5760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:34:03.657 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:03.654+0000 7f729aa00640 1 --2- 172.21.15.90:0/3768509317 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f729c107f10 0x7f729c19e1b0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f728c00e9c0 tx=0x7f728c00ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:34:03.657 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:03.654+0000 7f7293400640 1 -- 172.21.15.90:0/3768509317 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f728c00cd80 con 0x7f729c107f10 2024-04-02T14:34:03.657 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:03.654+0000 7f729be00640 1 -- 172.21.15.90:0/3768509317 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f729c1a60c0 con 0x7f729c107f10 2024-04-02T14:34:03.657 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:03.654+0000 7f7293400640 1 -- 172.21.15.90:0/3768509317 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f728c0035f0 con 0x7f729c107f10 2024-04-02T14:34:03.657 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:03.654+0000 7f7293400640 1 -- 172.21.15.90:0/3768509317 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f728c010690 con 0x7f729c107f10 2024-04-02T14:34:03.657 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:03.654+0000 7f729be00640 1 -- 172.21.15.90:0/3768509317 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f729c1a6610 con 0x7f729c107f10 2024-04-02T14:34:03.657 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:03.655+0000 7f729be00640 1 -- 172.21.15.90:0/3768509317 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f729c10f320 con 0x7f729c107f10 2024-04-02T14:34:03.658 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:03.658+0000 7f7293400640 1 -- 172.21.15.90:0/3768509317 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f728c0037a0 con 0x7f729c107f10 2024-04-02T14:34:03.660 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:03.659+0000 7f7293400640 1 --2- 172.21.15.90:0/3768509317 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f72800a3bb0 0x7f72800a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:03.660 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:03.659+0000 7f7293400640 1 -- 172.21.15.90:0/3768509317 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8248+0+0 (secure 0 0 0) 0x7f728c0d07d0 con 0x7f729c107f10 2024-04-02T14:34:03.660 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:03.659+0000 7f7293e00640 1 --2- 172.21.15.90:0/3768509317 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f72800a3bb0 0x7f72800a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:03.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:03.660+0000 7f7293e00640 1 --2- 172.21.15.90:0/3768509317 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f72800a3bb0 0x7f72800a6070 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f729c19f6c0 tx=0x7f7284005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:34:03.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:03.663+0000 7f7293400640 1 -- 172.21.15.90:0/3768509317 <== mon.2 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7f728c09d140 con 0x7f729c107f10 2024-04-02T14:34:03.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:03.868+0000 7f729be00640 1 -- 172.21.15.90:0/3768509317 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f729c062700 con 0x7f729c107f10 2024-04-02T14:34:03.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:03.870+0000 7f7293400640 1 -- 172.21.15.90:0/3768509317 <== mon.2 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+22646 (secure 0 0 0) 0x7f728c09ca40 con 0x7f729c107f10 2024-04-02T14:34:03.871 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-02T14:34:03.871 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":90,"fsid":"6449448a-f0fc-11ee-b647-cb9ed24678a4","created":"2024-04-02T14:23:16.530201+0000","modified":"2024-04-02T14:32:35.716929+0000","last_up_change":"2024-04-02T14:31:23.819732+0000","last_in_change":"2024-04-02T14:31:02.824284+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T14:27:16.758622+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-02T14:32:20.016331+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-02T14:32:21.294367+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a1a6a856-ab44-44bd-bc5e-45c718a70cd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":3635087609},{"type":"v1","addr":"172.21.15.90:6803","nonce":3635087609}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":3635087609},{"type":"v1","addr":"172.21.15.90:6805","nonce":3635087609}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":3635087609},{"type":"v1","addr":"172.21.15.90:6809","nonce":3635087609}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":3635087609},{"type":"v1","addr":"172.21.15.90:6807","nonce":3635087609}]},"public_addr":"172.21.15.90:6803/3635087609","cluster_addr":"172.21.15.90:6805/3635087609","heartbeat_back_addr":"172.21.15.90:6809/3635087609","heartbeat_front_addr":"172.21.15.90:6807/3635087609","state":["exists","up"]},{"osd":1,"uuid":"124139cd-5793-463e-9607-9d9c10164183","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":1994536059},{"type":"v1","addr":"172.21.15.92:6801","nonce":1994536059}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":1994536059},{"type":"v1","addr":"172.21.15.92:6803","nonce":1994536059}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":1994536059},{"type":"v1","addr":"172.21.15.92:6807","nonce":1994536059}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":1994536059},{"type":"v1","addr":"172.21.15.92:6805","nonce":1994536059}]},"public_addr":"172.21.15.92:6801/1994536059","cluster_addr":"172.21.15.92:6803/1994536059","heartbeat_back_addr":"172.21.15.92:6807/1994536059","heartbeat_front_addr":"172.21.15.92:6805/1994536059","state":["exists","up"]},{"osd":2,"uuid":"a61fbef2-9dca-4875-a814-020fff0db0e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":1358472800},{"type":"v1","addr":"172.21.15.177:6801","nonce":1358472800}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":1358472800},{"type":"v1","addr":"172.21.15.177:6803","nonce":1358472800}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":1358472800},{"type":"v1","addr":"172.21.15.177:6807","nonce":1358472800}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":1358472800},{"type":"v1","addr":"172.21.15.177:6805","nonce":1358472800}]},"public_addr":"172.21.15.177:6801/1358472800","cluster_addr":"172.21.15.177:6803/1358472800","heartbeat_back_addr":"172.21.15.177:6807/1358472800","heartbeat_front_addr":"172.21.15.177:6805/1358472800","state":["exists","up"]},{"osd":3,"uuid":"b175a2f4-7859-4eb0-bc1f-fba8c2b92a2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":3114811216},{"type":"v1","addr":"172.21.15.90:6811","nonce":3114811216}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":3114811216},{"type":"v1","addr":"172.21.15.90:6813","nonce":3114811216}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":3114811216},{"type":"v1","addr":"172.21.15.90:6817","nonce":3114811216}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":3114811216},{"type":"v1","addr":"172.21.15.90:6815","nonce":3114811216}]},"public_addr":"172.21.15.90:6811/3114811216","cluster_addr":"172.21.15.90:6813/3114811216","heartbeat_back_addr":"172.21.15.90:6817/3114811216","heartbeat_front_addr":"172.21.15.90:6815/3114811216","state":["exists","up"]},{"osd":4,"uuid":"38cc68ea-c109-4d8d-b77d-1c6f9b56c875","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":2987208879},{"type":"v1","addr":"172.21.15.92:6809","nonce":2987208879}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6810","nonce":2987208879},{"type":"v1","addr":"172.21.15.92:6811","nonce":2987208879}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6814","nonce":2987208879},{"type":"v1","addr":"172.21.15.92:6815","nonce":2987208879}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":2987208879},{"type":"v1","addr":"172.21.15.92:6813","nonce":2987208879}]},"public_addr":"172.21.15.92:6809/2987208879","cluster_addr":"172.21.15.92:6811/2987208879","heartbeat_back_addr":"172.21.15.92:6815/2987208879","heartbeat_front_addr":"172.21.15.92:6813/2987208879","state":["exists","up"]},{"osd":5,"uuid":"2347232e-ac91-466f-8ac9-496f397679ba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":3486820237},{"type":"v1","addr":"172.21.15.177:6809","nonce":3486820237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":3486820237},{"type":"v1","addr":"172.21.15.177:6811","nonce":3486820237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":3486820237},{"type":"v1","addr":"172.21.15.177:6815","nonce":3486820237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":3486820237},{"type":"v1","addr":"172.21.15.177:6813","nonce":3486820237}]},"public_addr":"172.21.15.177:6809/3486820237","cluster_addr":"172.21.15.177:6811/3486820237","heartbeat_back_addr":"172.21.15.177:6815/3486820237","heartbeat_front_addr":"172.21.15.177:6813/3486820237","state":["exists","up"]},{"osd":6,"uuid":"aa72392f-0498-44db-a8d2-eccc89a64871","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":2579716443},{"type":"v1","addr":"172.21.15.90:6819","nonce":2579716443}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6820","nonce":2579716443},{"type":"v1","addr":"172.21.15.90:6821","nonce":2579716443}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":2579716443},{"type":"v1","addr":"172.21.15.90:6825","nonce":2579716443}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2579716443},{"type":"v1","addr":"172.21.15.90:6823","nonce":2579716443}]},"public_addr":"172.21.15.90:6819/2579716443","cluster_addr":"172.21.15.90:6821/2579716443","heartbeat_back_addr":"172.21.15.90:6825/2579716443","heartbeat_front_addr":"172.21.15.90:6823/2579716443","state":["exists","up"]},{"osd":7,"uuid":"931bd105-bc14-4a1e-bfe4-c7ad58fa4b73","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":3371649139},{"type":"v1","addr":"172.21.15.92:6817","nonce":3371649139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":3371649139},{"type":"v1","addr":"172.21.15.92:6819","nonce":3371649139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":3371649139},{"type":"v1","addr":"172.21.15.92:6823","nonce":3371649139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6820","nonce":3371649139},{"type":"v1","addr":"172.21.15.92:6821","nonce":3371649139}]},"public_addr":"172.21.15.92:6817/3371649139","cluster_addr":"172.21.15.92:6819/3371649139","heartbeat_back_addr":"172.21.15.92:6823/3371649139","heartbeat_front_addr":"172.21.15.92:6821/3371649139","state":["exists","up"]},{"osd":8,"uuid":"82624ec2-b382-4886-9f40-7f5f376de3ce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":1459053034},{"type":"v1","addr":"172.21.15.177:6817","nonce":1459053034}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":1459053034},{"type":"v1","addr":"172.21.15.177:6819","nonce":1459053034}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":1459053034},{"type":"v1","addr":"172.21.15.177:6823","nonce":1459053034}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":1459053034},{"type":"v1","addr":"172.21.15.177:6821","nonce":1459053034}]},"public_addr":"172.21.15.177:6817/1459053034","cluster_addr":"172.21.15.177:6819/1459053034","heartbeat_back_addr":"172.21.15.177:6823/1459053034","heartbeat_front_addr":"172.21.15.177:6821/1459053034","state":["exists","up"]},{"osd":9,"uuid":"fbe36e48-3468-42b2-8094-e16552f3cb16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":2592604332},{"type":"v1","addr":"172.21.15.90:6827","nonce":2592604332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":2592604332},{"type":"v1","addr":"172.21.15.90:6829","nonce":2592604332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6832","nonce":2592604332},{"type":"v1","addr":"172.21.15.90:6833","nonce":2592604332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":2592604332},{"type":"v1","addr":"172.21.15.90:6831","nonce":2592604332}]},"public_addr":"172.21.15.90:6827/2592604332","cluster_addr":"172.21.15.90:6829/2592604332","heartbeat_back_addr":"172.21.15.90:6833/2592604332","heartbeat_front_addr":"172.21.15.90:6831/2592604332","state":["exists","up"]},{"osd":10,"uuid":"9240df75-e7e2-47f1-8aae-534b70b3c210","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":1594153142},{"type":"v1","addr":"172.21.15.92:6825","nonce":1594153142}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6826","nonce":1594153142},{"type":"v1","addr":"172.21.15.92:6827","nonce":1594153142}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":1594153142},{"type":"v1","addr":"172.21.15.92:6831","nonce":1594153142}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":1594153142},{"type":"v1","addr":"172.21.15.92:6829","nonce":1594153142}]},"public_addr":"172.21.15.92:6825/1594153142","cluster_addr":"172.21.15.92:6827/1594153142","heartbeat_back_addr":"172.21.15.92:6831/1594153142","heartbeat_front_addr":"172.21.15.92:6829/1594153142","state":["exists","up"]},{"osd":11,"uuid":"83cba9fa-b745-4df1-adda-e86837d4b91a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":83,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1622582126},{"type":"v1","addr":"172.21.15.177:6825","nonce":1622582126}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":1622582126},{"type":"v1","addr":"172.21.15.177:6827","nonce":1622582126}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":1622582126},{"type":"v1","addr":"172.21.15.177:6831","nonce":1622582126}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1622582126},{"type":"v1","addr":"172.21.15.177:6829","nonce":1622582126}]},"public_addr":"172.21.15.177:6825/1622582126","cluster_addr":"172.21.15.177:6827/1622582126","heartbeat_back_addr":"172.21.15.177:6831/1622582126","heartbeat_front_addr":"172.21.15.177:6829/1622582126","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:26:24.290365+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:26:49.405988+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:27:13.639196+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:27:45.603746+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:28:11.019126+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:28:35.084163+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:29:08.064626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:29:33.266133+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:29:57.381816+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:30:30.286847+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:30:56.226279+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:31:20.809611+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":7,"to":11}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":8,"to":4}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.11","mappings":[{"from":4,"to":8},{"from":5,"to":9}]},{"pgid":"3.21","mappings":[{"from":5,"to":9}]},{"pgid":"3.23","mappings":[{"from":4,"to":7}]},{"pgid":"3.3a","mappings":[{"from":4,"to":6}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.90:0/314641601":"2024-04-03T14:24:34.595561+0000","172.21.15.90:0/1400895868":"2024-04-03T14:24:34.595561+0000","172.21.15.90:6801/129436436":"2024-04-03T14:24:34.595561+0000","172.21.15.90:6801/2372398598":"2024-04-03T14:24:00.362147+0000","172.21.15.90:0/278569338":"2024-04-03T14:24:34.595561+0000","172.21.15.90:0/2321464594":"2024-04-03T14:24:00.362147+0000","172.21.15.90:6800/2372398598":"2024-04-03T14:24:00.362147+0000","172.21.15.90:0/2962656672":"2024-04-03T14:24:00.362147+0000","172.21.15.90:0/126693610":"2024-04-03T14:24:00.362147+0000","172.21.15.90:6800/129436436":"2024-04-03T14:24:34.595561+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T14:34:03.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:03.872+0000 7f729be00640 1 -- 172.21.15.90:0/3768509317 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f72800a3bb0 msgr2=0x7f72800a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:03.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:03.872+0000 7f729be00640 1 --2- 172.21.15.90:0/3768509317 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f72800a3bb0 0x7f72800a6070 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f729c19f6c0 tx=0x7f7284005f90 comp rx=0 tx=0).stop 2024-04-02T14:34:03.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:03.872+0000 7f729be00640 1 -- 172.21.15.90:0/3768509317 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f729c107f10 msgr2=0x7f729c19e1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:03.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:03.872+0000 7f729be00640 1 --2- 172.21.15.90:0/3768509317 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f729c107f10 0x7f729c19e1b0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f728c00e9c0 tx=0x7f728c00ee80 comp rx=0 tx=0).stop 2024-04-02T14:34:03.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:03.872+0000 7f729be00640 1 -- 172.21.15.90:0/3768509317 shutdown_connections 2024-04-02T14:34:03.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:03.872+0000 7f729be00640 1 --2- 172.21.15.90:0/3768509317 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f72800a3bb0 0x7f72800a6070 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:03.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:03.872+0000 7f729be00640 1 --2- 172.21.15.90:0/3768509317 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f729c109aa0 0x7f729c1a5760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:03.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:03.872+0000 7f729be00640 1 --2- 172.21.15.90:0/3768509317 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f729c1090e0 0x7f729c19e6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:03.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:03.872+0000 7f729be00640 1 --2- 172.21.15.90:0/3768509317 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f729c107f10 0x7f729c19e1b0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:03.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:03.873+0000 7f729be00640 1 -- 172.21.15.90:0/3768509317 >> 172.21.15.90:0/3768509317 conn(0x7f729c076f10 msgr2=0x7f729c10c4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:34:03.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:03.873+0000 7f729be00640 1 -- 172.21.15.90:0/3768509317 shutdown_connections 2024-04-02T14:34:03.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:03.873+0000 7f729be00640 1 -- 172.21.15.90:0/3768509317 wait complete. 2024-04-02T14:34:03.895 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-02T14:27:16.758622+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-02T14:32:20.016331+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-02T14:32:21.294367+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '88', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-02T14:34:03.895 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-02T14:34:04.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:04.108+0000 7f5227e00640 1 -- 172.21.15.90:0/473394829 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f522010a550 msgr2=0x7f5220110e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:04.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:04.108+0000 7f5227e00640 1 --2- 172.21.15.90:0/473394829 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f522010a550 0x7f5220110e00 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f5218009ca0 tx=0x7f521802c100 comp rx=0 tx=0).stop 2024-04-02T14:34:04.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:04.108+0000 7f5227e00640 1 -- 172.21.15.90:0/473394829 shutdown_connections 2024-04-02T14:34:04.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:04.108+0000 7f5227e00640 1 --2- 172.21.15.90:0/473394829 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f522010a550 0x7f5220110e00 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:04.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:04.108+0000 7f5227e00640 1 --2- 172.21.15.90:0/473394829 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5220109bb0 0x7f522010a010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:04.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:04.108+0000 7f5227e00640 1 --2- 172.21.15.90:0/473394829 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5220108ef0 0x7f5220109310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:04.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:04.108+0000 7f5227e00640 1 -- 172.21.15.90:0/473394829 >> 172.21.15.90:0/473394829 conn(0x7f522006ece0 msgr2=0x7f5220071120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:34:04.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:04.109+0000 7f5227e00640 1 -- 172.21.15.90:0/473394829 shutdown_connections 2024-04-02T14:34:04.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:04.109+0000 7f5227e00640 1 -- 172.21.15.90:0/473394829 wait complete. 2024-04-02T14:34:04.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:04.110+0000 7f5227e00640 1 Processor -- start 2024-04-02T14:34:04.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:04.110+0000 7f5227e00640 1 -- start start 2024-04-02T14:34:04.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:04.110+0000 7f5227e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5220108ef0 0x7f52201a27e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:04.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:04.110+0000 7f5227e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5220109bb0 0x7f52201a2d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:04.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:04.110+0000 7f5227e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f522010a550 0x7f52201a9d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:04.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:04.110+0000 7f5227e00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f5220115fc0 con 0x7f5220109bb0 2024-04-02T14:34:04.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:04.110+0000 7f5227e00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f5220115e40 con 0x7f522010a550 2024-04-02T14:34:04.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:04.110+0000 7f5227e00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f5220116140 con 0x7f5220108ef0 2024-04-02T14:34:04.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:04.110+0000 7f5224a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5220109bb0 0x7f52201a2d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:04.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:04.110+0000 7f5224a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5220109bb0 0x7f52201a2d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:52048/0 (socket says 172.21.15.90:52048) 2024-04-02T14:34:04.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:04.110+0000 7f5225400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5220108ef0 0x7f52201a27e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:04.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:04.110+0000 7f5224a00640 1 -- 172.21.15.90:0/1532300770 learned_addr learned my addr 172.21.15.90:0/1532300770 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:34:04.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:04.110+0000 7f5225e00640 1 --2- 172.21.15.90:0/1532300770 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f522010a550 0x7f52201a9d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:04.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:04.111+0000 7f5225e00640 1 -- 172.21.15.90:0/1532300770 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5220108ef0 msgr2=0x7f52201a27e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:04.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:04.111+0000 7f5225e00640 1 --2- 172.21.15.90:0/1532300770 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5220108ef0 0x7f52201a27e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:04.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:04.111+0000 7f5225e00640 1 -- 172.21.15.90:0/1532300770 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5220109bb0 msgr2=0x7f52201a2d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:04.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:04.111+0000 7f5225e00640 1 --2- 172.21.15.90:0/1532300770 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5220109bb0 0x7f52201a2d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:04.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:04.111+0000 7f5225e00640 1 -- 172.21.15.90:0/1532300770 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52201aa400 con 0x7f522010a550 2024-04-02T14:34:04.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:04.111+0000 7f5224a00640 1 --2- 172.21.15.90:0/1532300770 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5220109bb0 0x7f52201a2d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:34:04.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:04.111+0000 7f5225e00640 1 --2- 172.21.15.90:0/1532300770 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f522010a550 0x7f52201a9d90 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f5218037040 tx=0x7f5218035d10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:34:04.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:04.111+0000 7f521e000640 1 -- 172.21.15.90:0/1532300770 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5218039040 con 0x7f522010a550 2024-04-02T14:34:04.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:04.111+0000 7f521e000640 1 -- 172.21.15.90:0/1532300770 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5218044070 con 0x7f522010a550 2024-04-02T14:34:04.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:04.111+0000 7f5227e00640 1 -- 172.21.15.90:0/1532300770 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52201aa690 con 0x7f522010a550 2024-04-02T14:34:04.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:04.111+0000 7f521e000640 1 -- 172.21.15.90:0/1532300770 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5218004350 con 0x7f522010a550 2024-04-02T14:34:04.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:04.112+0000 7f5227e00640 1 -- 172.21.15.90:0/1532300770 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52201aab50 con 0x7f522010a550 2024-04-02T14:34:04.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:04.113+0000 7f5227e00640 1 -- 172.21.15.90:0/1532300770 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f52201aadb0 con 0x7f522010a550 2024-04-02T14:34:04.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:04.115+0000 7f521e000640 1 -- 172.21.15.90:0/1532300770 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f52180044f0 con 0x7f522010a550 2024-04-02T14:34:04.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:04.116+0000 7f521e000640 1 --2- 172.21.15.90:0/1532300770 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f52080a3bb0 0x7f52080a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:04.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:04.116+0000 7f521e000640 1 -- 172.21.15.90:0/1532300770 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8248+0+0 (secure 0 0 0) 0x7f52180f29c0 con 0x7f522010a550 2024-04-02T14:34:04.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:04.116+0000 7f5225400640 1 --2- 172.21.15.90:0/1532300770 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f52080a3bb0 0x7f52080a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:04.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:04.117+0000 7f5225400640 1 --2- 172.21.15.90:0/1532300770 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f52080a3bb0 0x7f52080a6070 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f52140095f0 tx=0x7f5214005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:34:04.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:04.120+0000 7f521e000640 1 -- 172.21.15.90:0/1532300770 <== mon.2 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7f52180bf330 con 0x7f522010a550 2024-04-02T14:34:04.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:34:03 smithi177 ceph-mon[36431]: pgmap v364: 97 pgs: 97 active+clean; 644 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-02T14:34:04.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:34:03 smithi177 ceph-mon[36431]: from='client.? 172.21.15.90:0/3768509317' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-02T14:34:04.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:34:03 smithi090 ceph-mon[31674]: pgmap v364: 97 pgs: 97 active+clean; 644 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-02T14:34:04.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:34:03 smithi090 ceph-mon[31674]: from='client.? 172.21.15.90:0/3768509317' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-02T14:34:04.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:04.322+0000 7f5227e00640 1 -- 172.21.15.90:0/1532300770 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f52201ab140 con 0x7f522010a550 2024-04-02T14:34:04.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:04.323+0000 7f521e000640 1 -- 172.21.15.90:0/1532300770 <== mon.2 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v90) ==== 93+0+10 (secure 0 0 0) 0x7f52180098b0 con 0x7f522010a550 2024-04-02T14:34:04.324 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 1 2024-04-02T14:34:04.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:04.326+0000 7f5227e00640 1 -- 172.21.15.90:0/1532300770 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f52080a3bb0 msgr2=0x7f52080a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:04.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:04.326+0000 7f5227e00640 1 --2- 172.21.15.90:0/1532300770 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f52080a3bb0 0x7f52080a6070 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f52140095f0 tx=0x7f5214005f90 comp rx=0 tx=0).stop 2024-04-02T14:34:04.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:04.326+0000 7f5227e00640 1 -- 172.21.15.90:0/1532300770 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f522010a550 msgr2=0x7f52201a9d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:04.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:04.326+0000 7f5227e00640 1 --2- 172.21.15.90:0/1532300770 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f522010a550 0x7f52201a9d90 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f5218037040 tx=0x7f5218035d10 comp rx=0 tx=0).stop 2024-04-02T14:34:04.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:04.326+0000 7f5227e00640 1 -- 172.21.15.90:0/1532300770 shutdown_connections 2024-04-02T14:34:04.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:04.326+0000 7f5227e00640 1 --2- 172.21.15.90:0/1532300770 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f52080a3bb0 0x7f52080a6070 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:04.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:04.326+0000 7f5227e00640 1 --2- 172.21.15.90:0/1532300770 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f522010a550 0x7f52201a9d90 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:04.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:04.326+0000 7f5227e00640 1 --2- 172.21.15.90:0/1532300770 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5220109bb0 0x7f52201a2d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:04.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:04.326+0000 7f5227e00640 1 --2- 172.21.15.90:0/1532300770 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5220108ef0 0x7f52201a27e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:04.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:04.326+0000 7f5227e00640 1 -- 172.21.15.90:0/1532300770 >> 172.21.15.90:0/1532300770 conn(0x7f522006ece0 msgr2=0x7f52201125d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:34:04.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:04.326+0000 7f5227e00640 1 -- 172.21.15.90:0/1532300770 shutdown_connections 2024-04-02T14:34:04.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:04.326+0000 7f5227e00640 1 -- 172.21.15.90:0/1532300770 wait complete. 2024-04-02T14:34:04.347 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-02T14:34:04.360 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:34:03 smithi092 ceph-mon[36497]: pgmap v364: 97 pgs: 97 active+clean; 644 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-02T14:34:04.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:34:03 smithi092 ceph-mon[36497]: from='client.? 172.21.15.90:0/3768509317' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-02T14:34:04.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:04.563+0000 7f50fe200640 1 -- 172.21.15.90:0/1003107707 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f50f81058f0 msgr2=0x7f50f810c180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:04.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:04.563+0000 7f50fe200640 1 --2- 172.21.15.90:0/1003107707 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f50f81058f0 0x7f50f810c180 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f50f0009ca0 tx=0x7f50f002c0f0 comp rx=0 tx=0).stop 2024-04-02T14:34:04.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:04.563+0000 7f50fe200640 1 -- 172.21.15.90:0/1003107707 shutdown_connections 2024-04-02T14:34:04.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:04.563+0000 7f50fe200640 1 --2- 172.21.15.90:0/1003107707 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f50f81058f0 0x7f50f810c180 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:04.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:04.563+0000 7f50fe200640 1 --2- 172.21.15.90:0/1003107707 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f50f8104f30 0x7f50f81053b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:04.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:04.563+0000 7f50fe200640 1 --2- 172.21.15.90:0/1003107707 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f50f8103d40 0x7f50f8104160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:04.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:04.563+0000 7f50fe200640 1 -- 172.21.15.90:0/1003107707 >> 172.21.15.90:0/1003107707 conn(0x7f50f80ff2b0 msgr2=0x7f50f81016f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:34:04.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:04.563+0000 7f50fe200640 1 -- 172.21.15.90:0/1003107707 shutdown_connections 2024-04-02T14:34:04.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:04.563+0000 7f50fe200640 1 -- 172.21.15.90:0/1003107707 wait complete. 2024-04-02T14:34:04.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:04.564+0000 7f50fe200640 1 Processor -- start 2024-04-02T14:34:04.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:04.564+0000 7f50fe200640 1 -- start start 2024-04-02T14:34:04.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:04.564+0000 7f50fe200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f50f8103d40 0x7f50f819a040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:04.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:04.564+0000 7f50fe200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f50f8104f30 0x7f50f819a580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:04.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:04.564+0000 7f50fe200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f50f81058f0 0x7f50f81a15f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:04.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:04.564+0000 7f50fe200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f50f810f9a0 con 0x7f50f8103d40 2024-04-02T14:34:04.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:04.564+0000 7f50fe200640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f50f810f820 con 0x7f50f81058f0 2024-04-02T14:34:04.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:04.564+0000 7f50fe200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f50f810fb20 con 0x7f50f8104f30 2024-04-02T14:34:04.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:04.565+0000 7f50f7400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f50f8103d40 0x7f50f819a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:04.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:04.565+0000 7f50f7400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f50f8103d40 0x7f50f819a040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:52058/0 (socket says 172.21.15.90:52058) 2024-04-02T14:34:04.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:04.565+0000 7f50f7400640 1 -- 172.21.15.90:0/689743412 learned_addr learned my addr 172.21.15.90:0/689743412 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:34:04.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:04.565+0000 7f50f7e00640 1 --2- 172.21.15.90:0/689743412 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f50f81058f0 0x7f50f81a15f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:04.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:04.565+0000 7f50eea00640 1 --2- 172.21.15.90:0/689743412 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f50f8104f30 0x7f50f819a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:04.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:04.565+0000 7f50f7400640 1 -- 172.21.15.90:0/689743412 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f50f8104f30 msgr2=0x7f50f819a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:04.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:04.565+0000 7f50f7400640 1 --2- 172.21.15.90:0/689743412 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f50f8104f30 0x7f50f819a580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:04.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:04.565+0000 7f50f7400640 1 -- 172.21.15.90:0/689743412 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f50f81058f0 msgr2=0x7f50f81a15f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:04.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:04.565+0000 7f50f7400640 1 --2- 172.21.15.90:0/689743412 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f50f81058f0 0x7f50f81a15f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:04.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:04.565+0000 7f50f7400640 1 -- 172.21.15.90:0/689743412 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50f81a1c60 con 0x7f50f8103d40 2024-04-02T14:34:04.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:04.565+0000 7f50eea00640 1 --2- 172.21.15.90:0/689743412 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f50f8104f30 0x7f50f819a580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:34:04.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:04.565+0000 7f50f7400640 1 --2- 172.21.15.90:0/689743412 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f50f8103d40 0x7f50f819a040 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f50e800e9c0 tx=0x7f50e800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:34:04.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:04.565+0000 7f50f4c00640 1 -- 172.21.15.90:0/689743412 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50e800cde0 con 0x7f50f8103d40 2024-04-02T14:34:04.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:04.566+0000 7f50f4c00640 1 -- 172.21.15.90:0/689743412 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f50e8010910 con 0x7f50f8103d40 2024-04-02T14:34:04.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:04.566+0000 7f50fe200640 1 -- 172.21.15.90:0/689743412 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f50f81a1f50 con 0x7f50f8103d40 2024-04-02T14:34:04.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:04.566+0000 7f50f4c00640 1 -- 172.21.15.90:0/689743412 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50e8018a00 con 0x7f50f8103d40 2024-04-02T14:34:04.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:04.566+0000 7f50fe200640 1 -- 172.21.15.90:0/689743412 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50f81a23b0 con 0x7f50f8103d40 2024-04-02T14:34:04.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:04.567+0000 7f50fe200640 1 -- 172.21.15.90:0/689743412 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f50f810b000 con 0x7f50f8103d40 2024-04-02T14:34:04.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:04.569+0000 7f50f4c00640 1 -- 172.21.15.90:0/689743412 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f50e801f020 con 0x7f50f8103d40 2024-04-02T14:34:04.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:04.570+0000 7f50f4c00640 1 --2- 172.21.15.90:0/689743412 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f50dc0a3bb0 0x7f50dc0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:04.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:04.570+0000 7f50f4c00640 1 -- 172.21.15.90:0/689743412 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8248+0+0 (secure 0 0 0) 0x7f50e80d06f0 con 0x7f50f8103d40 2024-04-02T14:34:04.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:04.570+0000 7f50eea00640 1 --2- 172.21.15.90:0/689743412 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f50dc0a3bb0 0x7f50dc0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:04.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:04.571+0000 7f50eea00640 1 --2- 172.21.15.90:0/689743412 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f50dc0a3bb0 0x7f50dc0a6070 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f50f819b550 tx=0x7f50e0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:34:04.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:04.575+0000 7f50f4c00640 1 -- 172.21.15.90:0/689743412 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7f50e809d060 con 0x7f50f8103d40 2024-04-02T14:34:04.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:04.779+0000 7f50fe200640 1 -- 172.21.15.90:0/689743412 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f50f8062800 con 0x7f50f8103d40 2024-04-02T14:34:04.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:04.780+0000 7f50f4c00640 1 -- 172.21.15.90:0/689743412 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v90) ==== 104+0+11 (secure 0 0 0) 0x7f50e809c960 con 0x7f50f8103d40 2024-04-02T14:34:04.781 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-04-02T14:34:04.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:04.782+0000 7f50fe200640 1 -- 172.21.15.90:0/689743412 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f50dc0a3bb0 msgr2=0x7f50dc0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:04.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:04.782+0000 7f50fe200640 1 --2- 172.21.15.90:0/689743412 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f50dc0a3bb0 0x7f50dc0a6070 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f50f819b550 tx=0x7f50e0005f90 comp rx=0 tx=0).stop 2024-04-02T14:34:04.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:04.782+0000 7f50fe200640 1 -- 172.21.15.90:0/689743412 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f50f8103d40 msgr2=0x7f50f819a040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:04.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:04.782+0000 7f50fe200640 1 --2- 172.21.15.90:0/689743412 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f50f8103d40 0x7f50f819a040 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f50e800e9c0 tx=0x7f50e800ee80 comp rx=0 tx=0).stop 2024-04-02T14:34:04.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:04.782+0000 7f50fe200640 1 -- 172.21.15.90:0/689743412 shutdown_connections 2024-04-02T14:34:04.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:04.782+0000 7f50fe200640 1 --2- 172.21.15.90:0/689743412 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f50dc0a3bb0 0x7f50dc0a6070 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:04.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:04.782+0000 7f50fe200640 1 --2- 172.21.15.90:0/689743412 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f50f81058f0 0x7f50f81a15f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:04.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:04.782+0000 7f50fe200640 1 --2- 172.21.15.90:0/689743412 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f50f8104f30 0x7f50f819a580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:04.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:04.782+0000 7f50fe200640 1 --2- 172.21.15.90:0/689743412 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f50f8103d40 0x7f50f819a040 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:04.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:04.782+0000 7f50fe200640 1 -- 172.21.15.90:0/689743412 >> 172.21.15.90:0/689743412 conn(0x7f50f80ff2b0 msgr2=0x7f50f81081d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:34:04.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:04.783+0000 7f50fe200640 1 -- 172.21.15.90:0/689743412 shutdown_connections 2024-04-02T14:34:04.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:04.783+0000 7f50fe200640 1 -- 172.21.15.90:0/689743412 wait complete. 2024-04-02T14:34:04.804 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-02T14:34:05.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:05.012+0000 7ff5c3e00640 1 -- 172.21.15.90:0/1822654063 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff5c4103590 msgr2=0x7ff5c4109e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:05.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:05.012+0000 7ff5c3e00640 1 --2- 172.21.15.90:0/1822654063 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff5c4103590 0x7ff5c4109e20 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7ff5bc009900 tx=0x7ff5bc02bc40 comp rx=0 tx=0).stop 2024-04-02T14:34:05.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:05.013+0000 7ff5c3e00640 1 -- 172.21.15.90:0/1822654063 shutdown_connections 2024-04-02T14:34:05.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:05.013+0000 7ff5c3e00640 1 --2- 172.21.15.90:0/1822654063 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff5c4103590 0x7ff5c4109e20 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:05.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:05.013+0000 7ff5c3e00640 1 --2- 172.21.15.90:0/1822654063 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff5c4102bd0 0x7ff5c4103050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:05.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:05.013+0000 7ff5c3e00640 1 --2- 172.21.15.90:0/1822654063 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff5c41019e0 0x7ff5c4101e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:05.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:05.013+0000 7ff5c3e00640 1 -- 172.21.15.90:0/1822654063 >> 172.21.15.90:0/1822654063 conn(0x7ff5c40fcf70 msgr2=0x7ff5c40ff390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:34:05.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:05.013+0000 7ff5c3e00640 1 -- 172.21.15.90:0/1822654063 shutdown_connections 2024-04-02T14:34:05.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:05.013+0000 7ff5c3e00640 1 -- 172.21.15.90:0/1822654063 wait complete. 2024-04-02T14:34:05.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:05.014+0000 7ff5c3e00640 1 Processor -- start 2024-04-02T14:34:05.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:05.014+0000 7ff5c3e00640 1 -- start start 2024-04-02T14:34:05.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:05.014+0000 7ff5c3e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff5c41019e0 0x7ff5c4197c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:05.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:05.014+0000 7ff5c3e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff5c4102bd0 0x7ff5c41981b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:05.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:05.014+0000 7ff5c3e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff5c4103590 0x7ff5c419f210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:05.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:05.014+0000 7ff5c3e00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7ff5c410d690 con 0x7ff5c4103590 2024-04-02T14:34:05.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:05.014+0000 7ff5c3e00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7ff5c410d510 con 0x7ff5c4102bd0 2024-04-02T14:34:05.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:05.014+0000 7ff5c3e00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7ff5c410d810 con 0x7ff5c41019e0 2024-04-02T14:34:05.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:05.015+0000 7ff5c3400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff5c4103590 0x7ff5c419f210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:05.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:05.015+0000 7ff5c3400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff5c4103590 0x7ff5c419f210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:52080/0 (socket says 172.21.15.90:52080) 2024-04-02T14:34:05.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:05.015+0000 7ff5c3400640 1 -- 172.21.15.90:0/4272567884 learned_addr learned my addr 172.21.15.90:0/4272567884 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:34:05.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:05.015+0000 7ff5c2000640 1 --2- 172.21.15.90:0/4272567884 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff5c4102bd0 0x7ff5c41981b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:05.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:05.015+0000 7ff5c2a00640 1 --2- 172.21.15.90:0/4272567884 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff5c41019e0 0x7ff5c4197c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:05.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:05.015+0000 7ff5c3400640 1 -- 172.21.15.90:0/4272567884 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff5c41019e0 msgr2=0x7ff5c4197c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:05.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:05.015+0000 7ff5c3400640 1 --2- 172.21.15.90:0/4272567884 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff5c41019e0 0x7ff5c4197c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:05.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:05.015+0000 7ff5c3400640 1 -- 172.21.15.90:0/4272567884 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff5c4102bd0 msgr2=0x7ff5c41981b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:05.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:05.015+0000 7ff5c3400640 1 --2- 172.21.15.90:0/4272567884 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff5c4102bd0 0x7ff5c41981b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:05.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:05.015+0000 7ff5c3400640 1 -- 172.21.15.90:0/4272567884 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5c419f8b0 con 0x7ff5c4103590 2024-04-02T14:34:05.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:05.015+0000 7ff5c3400640 1 --2- 172.21.15.90:0/4272567884 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff5c4103590 0x7ff5c419f210 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7ff5bc02c180 tx=0x7ff5bc02c660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:34:05.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:05.016+0000 7ff5bb400640 1 -- 172.21.15.90:0/4272567884 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5bc041070 con 0x7ff5c4103590 2024-04-02T14:34:05.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:05.016+0000 7ff5bb400640 1 -- 172.21.15.90:0/4272567884 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff5bc0054e0 con 0x7ff5c4103590 2024-04-02T14:34:05.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:05.016+0000 7ff5c3e00640 1 -- 172.21.15.90:0/4272567884 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5c419fb40 con 0x7ff5c4103590 2024-04-02T14:34:05.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:05.016+0000 7ff5bb400640 1 -- 172.21.15.90:0/4272567884 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5bc034440 con 0x7ff5c4103590 2024-04-02T14:34:05.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:05.016+0000 7ff5c3e00640 1 -- 172.21.15.90:0/4272567884 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5c41a0030 con 0x7ff5c4103590 2024-04-02T14:34:05.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:05.017+0000 7ff5c3e00640 1 -- 172.21.15.90:0/4272567884 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff5c4108ca0 con 0x7ff5c4103590 2024-04-02T14:34:05.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:05.019+0000 7ff5bb400640 1 -- 172.21.15.90:0/4272567884 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7ff5bc044070 con 0x7ff5c4103590 2024-04-02T14:34:05.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:05.020+0000 7ff5bb400640 1 --2- 172.21.15.90:0/4272567884 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7ff5a80a3bb0 0x7ff5a80a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:05.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:05.020+0000 7ff5bb400640 1 -- 172.21.15.90:0/4272567884 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8248+0+0 (secure 0 0 0) 0x7ff5bc0f1fd0 con 0x7ff5c4103590 2024-04-02T14:34:05.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:05.020+0000 7ff5c2a00640 1 --2- 172.21.15.90:0/4272567884 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7ff5a80a3bb0 0x7ff5a80a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:05.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:05.021+0000 7ff5c2a00640 1 --2- 172.21.15.90:0/4272567884 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7ff5a80a3bb0 0x7ff5a80a6070 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7ff5c4198b40 tx=0x7ff5b4009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:34:05.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:05.025+0000 7ff5bb400640 1 -- 172.21.15.90:0/4272567884 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7ff5bc0be2f0 con 0x7ff5c4103590 2024-04-02T14:34:05.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:34:04 smithi177 ceph-mon[36431]: from='client.? 172.21.15.90:0/1532300770' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-02T14:34:05.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:34:04 smithi177 ceph-mon[36431]: from='client.? 172.21.15.90:0/689743412' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-02T14:34:05.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:34:04 smithi090 ceph-mon[31674]: from='client.? 172.21.15.90:0/1532300770' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-02T14:34:05.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:34:04 smithi090 ceph-mon[31674]: from='client.? 172.21.15.90:0/689743412' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-02T14:34:05.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:05.227+0000 7ff5c3e00640 1 -- 172.21.15.90:0/4272567884 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7ff5c4062700 con 0x7ff5c4103590 2024-04-02T14:34:05.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:05.228+0000 7ff5bb400640 1 -- 172.21.15.90:0/4272567884 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v90) ==== 100+0+11 (secure 0 0 0) 0x7ff5bc0c25f0 con 0x7ff5c4103590 2024-04-02T14:34:05.229 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 64 2024-04-02T14:34:05.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:05.231+0000 7ff5c3e00640 1 -- 172.21.15.90:0/4272567884 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7ff5a80a3bb0 msgr2=0x7ff5a80a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:05.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:05.231+0000 7ff5c3e00640 1 --2- 172.21.15.90:0/4272567884 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7ff5a80a3bb0 0x7ff5a80a6070 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7ff5c4198b40 tx=0x7ff5b4009500 comp rx=0 tx=0).stop 2024-04-02T14:34:05.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:05.231+0000 7ff5c3e00640 1 -- 172.21.15.90:0/4272567884 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff5c4103590 msgr2=0x7ff5c419f210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:05.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:05.231+0000 7ff5c3e00640 1 --2- 172.21.15.90:0/4272567884 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff5c4103590 0x7ff5c419f210 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7ff5bc02c180 tx=0x7ff5bc02c660 comp rx=0 tx=0).stop 2024-04-02T14:34:05.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:05.231+0000 7ff5c3e00640 1 -- 172.21.15.90:0/4272567884 shutdown_connections 2024-04-02T14:34:05.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:05.231+0000 7ff5c3e00640 1 --2- 172.21.15.90:0/4272567884 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7ff5a80a3bb0 0x7ff5a80a6070 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:05.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:05.231+0000 7ff5c3e00640 1 --2- 172.21.15.90:0/4272567884 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff5c4103590 0x7ff5c419f210 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:05.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:05.231+0000 7ff5c3e00640 1 --2- 172.21.15.90:0/4272567884 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff5c4102bd0 0x7ff5c41981b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:05.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:05.231+0000 7ff5c3e00640 1 --2- 172.21.15.90:0/4272567884 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff5c41019e0 0x7ff5c4197c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:05.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:05.231+0000 7ff5c3e00640 1 -- 172.21.15.90:0/4272567884 >> 172.21.15.90:0/4272567884 conn(0x7ff5c40fcf70 msgr2=0x7ff5c4105e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:34:05.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:05.231+0000 7ff5c3e00640 1 -- 172.21.15.90:0/4272567884 shutdown_connections 2024-04-02T14:34:05.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:05.232+0000 7ff5c3e00640 1 -- 172.21.15.90:0/4272567884 wait complete. 2024-04-02T14:34:05.253 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T14:34:05.360 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:34:04 smithi092 ceph-mon[36497]: from='client.? 172.21.15.90:0/1532300770' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-02T14:34:05.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:34:04 smithi092 ceph-mon[36497]: from='client.? 172.21.15.90:0/689743412' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-02T14:34:05.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:05.463+0000 7f4982800640 1 -- 172.21.15.90:0/1527795134 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f497c109330 msgr2=0x7f497c1097b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:05.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:05.463+0000 7f4982800640 1 --2- 172.21.15.90:0/1527795134 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f497c109330 0x7f497c1097b0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f4964009900 tx=0x7f496402bc40 comp rx=0 tx=0).stop 2024-04-02T14:34:05.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:05.463+0000 7f4982800640 1 -- 172.21.15.90:0/1527795134 shutdown_connections 2024-04-02T14:34:05.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:05.463+0000 7f4982800640 1 --2- 172.21.15.90:0/1527795134 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f497c109cf0 0x7f497c110580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:05.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:05.463+0000 7f4982800640 1 --2- 172.21.15.90:0/1527795134 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f497c109330 0x7f497c1097b0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:05.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:05.463+0000 7f4982800640 1 --2- 172.21.15.90:0/1527795134 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f497c108140 0x7f497c108560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:05.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:05.463+0000 7f4982800640 1 -- 172.21.15.90:0/1527795134 >> 172.21.15.90:0/1527795134 conn(0x7f497c077110 msgr2=0x7f497c079530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:34:05.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:05.463+0000 7f4982800640 1 -- 172.21.15.90:0/1527795134 shutdown_connections 2024-04-02T14:34:05.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:05.463+0000 7f4982800640 1 -- 172.21.15.90:0/1527795134 wait complete. 2024-04-02T14:34:05.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:05.464+0000 7f4982800640 1 Processor -- start 2024-04-02T14:34:05.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:05.464+0000 7f4982800640 1 -- start start 2024-04-02T14:34:05.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:05.465+0000 7f4982800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f497c108140 0x7f497c19e400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:05.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:05.465+0000 7f4982800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f497c109330 0x7f497c19e940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:05.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:05.465+0000 7f4982800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f497c109cf0 0x7f497c1a59b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:05.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:05.465+0000 7f4982800640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f497c113dd0 con 0x7f497c109cf0 2024-04-02T14:34:05.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:05.465+0000 7f4982800640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f497c113c50 con 0x7f497c109330 2024-04-02T14:34:05.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:05.465+0000 7f4982800640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f497c113f50 con 0x7f497c108140 2024-04-02T14:34:05.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:05.465+0000 7f497be00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f497c109cf0 0x7f497c1a59b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:05.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:05.465+0000 7f497b400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f497c108140 0x7f497c19e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:05.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:05.465+0000 7f4972a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f497c109330 0x7f497c19e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:05.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:05.465+0000 7f497be00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f497c109cf0 0x7f497c1a59b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:52100/0 (socket says 172.21.15.90:52100) 2024-04-02T14:34:05.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:05.465+0000 7f497be00640 1 -- 172.21.15.90:0/3599894414 learned_addr learned my addr 172.21.15.90:0/3599894414 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:34:05.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:05.465+0000 7f497b400640 1 -- 172.21.15.90:0/3599894414 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f497c109330 msgr2=0x7f497c19e940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:05.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:05.465+0000 7f497b400640 1 --2- 172.21.15.90:0/3599894414 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f497c109330 0x7f497c19e940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:05.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:05.465+0000 7f497b400640 1 -- 172.21.15.90:0/3599894414 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f497c109cf0 msgr2=0x7f497c1a59b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:05.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:05.465+0000 7f497b400640 1 --2- 172.21.15.90:0/3599894414 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f497c109cf0 0x7f497c1a59b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:05.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:05.465+0000 7f497b400640 1 -- 172.21.15.90:0/3599894414 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f497c1a6020 con 0x7f497c108140 2024-04-02T14:34:05.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:05.466+0000 7f497b400640 1 --2- 172.21.15.90:0/3599894414 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f497c108140 0x7f497c19e400 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f496c005fd0 tx=0x7f496c00ef30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:34:05.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:05.466+0000 7f4978c00640 1 -- 172.21.15.90:0/3599894414 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f496c00cde0 con 0x7f497c108140 2024-04-02T14:34:05.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:05.466+0000 7f4982800640 1 -- 172.21.15.90:0/3599894414 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f497c1a6310 con 0x7f497c108140 2024-04-02T14:34:05.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:05.466+0000 7f4978c00640 1 -- 172.21.15.90:0/3599894414 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f496c011950 con 0x7f497c108140 2024-04-02T14:34:05.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:05.466+0000 7f4978c00640 1 -- 172.21.15.90:0/3599894414 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f496c01aa80 con 0x7f497c108140 2024-04-02T14:34:05.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:05.466+0000 7f4982800640 1 -- 172.21.15.90:0/3599894414 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f497c1a6880 con 0x7f497c108140 2024-04-02T14:34:05.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:05.467+0000 7f4982800640 1 -- 172.21.15.90:0/3599894414 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f497c10f460 con 0x7f497c108140 2024-04-02T14:34:05.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:05.470+0000 7f4978c00640 1 -- 172.21.15.90:0/3599894414 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f496c011b00 con 0x7f497c108140 2024-04-02T14:34:05.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:05.471+0000 7f4978c00640 1 --2- 172.21.15.90:0/3599894414 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f49600a3bb0 0x7f49600a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:05.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:05.471+0000 7f4978c00640 1 -- 172.21.15.90:0/3599894414 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8248+0+0 (secure 0 0 0) 0x7f496c0d0b70 con 0x7f497c108140 2024-04-02T14:34:05.473 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:05.471+0000 7f4972a00640 1 --2- 172.21.15.90:0/3599894414 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f49600a3bb0 0x7f49600a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:05.473 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:05.472+0000 7f4972a00640 1 --2- 172.21.15.90:0/3599894414 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f49600a3bb0 0x7f49600a6070 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f496402c7c0 tx=0x7f4964005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:34:05.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:05.473+0000 7f4978c00640 1 -- 172.21.15.90:0/3599894414 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7f496c016030 con 0x7f497c108140 2024-04-02T14:34:05.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:05.714+0000 7f4982800640 1 -- 172.21.15.90:0/3599894414 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f497c02d760 con 0x7f497c108140 2024-04-02T14:34:05.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:05.716+0000 7f4978c00640 1 -- 172.21.15.90:0/3599894414 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11927 (secure 0 0 0) 0x7f496c09d540 con 0x7f497c108140 2024-04-02T14:34:05.717 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-02T14:34:05.717 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14412,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6837/2852160309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6836","nonce":2852160309},{"type":"v1","addr":"172.21.15.90:6837","nonce":2852160309}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14478,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6841/1919376858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6840","nonce":1919376858},{"type":"v1","addr":"172.21.15.90:6841","nonce":1919376858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":26,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-02T14:32:22.319083+0000","modified":"2024-04-02T14:32:29.812071+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24590,"mds_1":24560,"mds_2":24547,"mds_3":24539,"mds_4":24523},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24454":{"gid":24454,"name":"l","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6833/1564319431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":1564319431},{"type":"v1","addr":"172.21.15.177:6833","nonce":1564319431}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.92:6833/1570337372","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":1570337372},{"type":"v1","addr":"172.21.15.92:6833","nonce":1570337372}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"j","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.90:6835/1277124891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6834","nonce":1277124891},{"type":"v1","addr":"172.21.15.90:6835","nonce":1277124891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.92:6835/860721164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":860721164},{"type":"v1","addr":"172.21.15.92:6835","nonce":860721164}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24511":{"gid":24511,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6835/3177244988","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":3177244988},{"type":"v1","addr":"172.21.15.177:6835","nonce":3177244988}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"c","rank":4,"incarnation":25,"state":"up:active","state_seq":9,"addr":"172.21.15.177:6837/4223414858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":4223414858},{"type":"v1","addr":"172.21.15.177:6837","nonce":4223414858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24539":{"gid":24539,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":8,"addr":"172.21.15.92:6837/1270635939","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":1270635939},{"type":"v1","addr":"172.21.15.92:6837","nonce":1270635939}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24547":{"gid":24547,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":7,"addr":"172.21.15.177:6839/664895663","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":664895663},{"type":"v1","addr":"172.21.15.177:6839","nonce":664895663}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"g","rank":1,"incarnation":19,"state":"up:active","state_seq":5,"addr":"172.21.15.90:6839/1498915870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6838","nonce":1498915870},{"type":"v1","addr":"172.21.15.90:6839","nonce":1498915870}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24590":{"gid":24590,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.92:6839/1054888219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":1054888219},{"type":"v1","addr":"172.21.15.92:6839","nonce":1054888219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24590,"qdb_cluster":[24539,24560,24523,24590,24547]},"id":1}]} 2024-04-02T14:34:05.717 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 26 2024-04-02T14:34:05.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:05.719+0000 7f4982800640 1 -- 172.21.15.90:0/3599894414 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f49600a3bb0 msgr2=0x7f49600a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:05.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:05.719+0000 7f4982800640 1 --2- 172.21.15.90:0/3599894414 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f49600a3bb0 0x7f49600a6070 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f496402c7c0 tx=0x7f4964005fd0 comp rx=0 tx=0).stop 2024-04-02T14:34:05.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:05.719+0000 7f4982800640 1 -- 172.21.15.90:0/3599894414 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f497c108140 msgr2=0x7f497c19e400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:05.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:05.719+0000 7f4982800640 1 --2- 172.21.15.90:0/3599894414 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f497c108140 0x7f497c19e400 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f496c005fd0 tx=0x7f496c00ef30 comp rx=0 tx=0).stop 2024-04-02T14:34:05.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:05.719+0000 7f4982800640 1 -- 172.21.15.90:0/3599894414 shutdown_connections 2024-04-02T14:34:05.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:05.719+0000 7f4982800640 1 --2- 172.21.15.90:0/3599894414 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f49600a3bb0 0x7f49600a6070 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:05.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:05.719+0000 7f4982800640 1 --2- 172.21.15.90:0/3599894414 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f497c109cf0 0x7f497c1a59b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:05.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:05.719+0000 7f4982800640 1 --2- 172.21.15.90:0/3599894414 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f497c109330 0x7f497c19e940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:05.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:05.719+0000 7f4982800640 1 --2- 172.21.15.90:0/3599894414 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f497c108140 0x7f497c19e400 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:05.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:05.719+0000 7f4982800640 1 -- 172.21.15.90:0/3599894414 >> 172.21.15.90:0/3599894414 conn(0x7f497c077110 msgr2=0x7f497c10c630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:34:05.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:05.719+0000 7f4982800640 1 -- 172.21.15.90:0/3599894414 shutdown_connections 2024-04-02T14:34:05.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:05.719+0000 7f4982800640 1 -- 172.21.15.90:0/3599894414 wait complete. 2024-04-02T14:34:05.738 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T14:34:05.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:05.938+0000 7f0834e00640 1 -- 172.21.15.90:0/2539651530 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f08301025a0 msgr2=0x7f0830104990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:05.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:05.938+0000 7f0834e00640 1 --2- 172.21.15.90:0/2539651530 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f08301025a0 0x7f0830104990 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f0820009900 tx=0x7f082002bc30 comp rx=0 tx=0).stop 2024-04-02T14:34:05.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:05.938+0000 7f0834e00640 1 -- 172.21.15.90:0/2539651530 shutdown_connections 2024-04-02T14:34:05.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:05.938+0000 7f0834e00640 1 --2- 172.21.15.90:0/2539651530 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0830107800 0x7f0830109c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:05.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:05.938+0000 7f0834e00640 1 --2- 172.21.15.90:0/2539651530 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0830104ed0 0x7f08301072c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:05.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:05.938+0000 7f0834e00640 1 --2- 172.21.15.90:0/2539651530 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f08301025a0 0x7f0830104990 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:05.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:05.938+0000 7f0834e00640 1 -- 172.21.15.90:0/2539651530 >> 172.21.15.90:0/2539651530 conn(0x7f08300fbfb0 msgr2=0x7f08300fe3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:34:05.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:05.938+0000 7f0834e00640 1 -- 172.21.15.90:0/2539651530 shutdown_connections 2024-04-02T14:34:05.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:05.938+0000 7f0834e00640 1 -- 172.21.15.90:0/2539651530 wait complete. 2024-04-02T14:34:05.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:05.939+0000 7f0834e00640 1 Processor -- start 2024-04-02T14:34:05.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:05.939+0000 7f0834e00640 1 -- start start 2024-04-02T14:34:05.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:05.940+0000 7f0834e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f08301025a0 0x7f083019a080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:05.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:05.940+0000 7f0834e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0830104ed0 0x7f083019a5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:05.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:05.940+0000 7f0834e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0830107800 0x7f08301a1630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:05.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:05.940+0000 7f0834e00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f083010d7d0 con 0x7f0830104ed0 2024-04-02T14:34:05.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:05.940+0000 7f0834e00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f083010d650 con 0x7f08301025a0 2024-04-02T14:34:05.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:05.940+0000 7f0834e00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f083010d950 con 0x7f0830107800 2024-04-02T14:34:05.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:05.940+0000 7f082d400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0830104ed0 0x7f083019a5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:05.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:05.940+0000 7f082d400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0830104ed0 0x7f083019a5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:52126/0 (socket says 172.21.15.90:52126) 2024-04-02T14:34:05.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:05.940+0000 7f082d400640 1 -- 172.21.15.90:0/70235335 learned_addr learned my addr 172.21.15.90:0/70235335 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:34:05.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:05.940+0000 7f082e800640 1 --2- 172.21.15.90:0/70235335 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0830107800 0x7f08301a1630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:05.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:05.940+0000 7f082de00640 1 --2- 172.21.15.90:0/70235335 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f08301025a0 0x7f083019a080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:05.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:05.940+0000 7f082d400640 1 -- 172.21.15.90:0/70235335 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0830107800 msgr2=0x7f08301a1630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:05.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:05.940+0000 7f082d400640 1 --2- 172.21.15.90:0/70235335 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0830107800 0x7f08301a1630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:05.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:05.940+0000 7f082d400640 1 -- 172.21.15.90:0/70235335 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f08301025a0 msgr2=0x7f083019a080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:05.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:05.940+0000 7f082d400640 1 --2- 172.21.15.90:0/70235335 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f08301025a0 0x7f083019a080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:05.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:05.940+0000 7f082d400640 1 -- 172.21.15.90:0/70235335 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08301a1ca0 con 0x7f0830104ed0 2024-04-02T14:34:05.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:05.940+0000 7f082de00640 1 --2- 172.21.15.90:0/70235335 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f08301025a0 0x7f083019a080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:34:05.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:05.941+0000 7f082d400640 1 --2- 172.21.15.90:0/70235335 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0830104ed0 0x7f083019a5c0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f081800ebc0 tx=0x7f081800c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:34:05.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:05.941+0000 7f0826a00640 1 -- 172.21.15.90:0/70235335 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0818004d20 con 0x7f0830104ed0 2024-04-02T14:34:05.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:05.941+0000 7f0826a00640 1 -- 172.21.15.90:0/70235335 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0818005500 con 0x7f0830104ed0 2024-04-02T14:34:05.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:05.941+0000 7f0834e00640 1 -- 172.21.15.90:0/70235335 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f08301a1f90 con 0x7f0830104ed0 2024-04-02T14:34:05.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:05.941+0000 7f0826a00640 1 -- 172.21.15.90:0/70235335 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0818010890 con 0x7f0830104ed0 2024-04-02T14:34:05.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:05.941+0000 7f0834e00640 1 -- 172.21.15.90:0/70235335 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f08301a2400 con 0x7f0830104ed0 2024-04-02T14:34:05.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:05.942+0000 7f0834e00640 1 -- 172.21.15.90:0/70235335 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0830109350 con 0x7f0830104ed0 2024-04-02T14:34:05.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:05.945+0000 7f0826a00640 1 -- 172.21.15.90:0/70235335 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f0818010a30 con 0x7f0830104ed0 2024-04-02T14:34:05.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:05.946+0000 7f0826a00640 1 --2- 172.21.15.90:0/70235335 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f08100a3bb0 0x7f08100a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:05.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:05.946+0000 7f0826a00640 1 -- 172.21.15.90:0/70235335 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8248+0+0 (secure 0 0 0) 0x7f08180d0c30 con 0x7f0830104ed0 2024-04-02T14:34:05.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:05.947+0000 7f082de00640 1 --2- 172.21.15.90:0/70235335 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f08100a3bb0 0x7f08100a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:05.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:05.947+0000 7f082de00640 1 --2- 172.21.15.90:0/70235335 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f08100a3bb0 0x7f08100a6070 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f0820006010 tx=0x7f082002c8a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:34:05.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:05.948+0000 7f0826a00640 1 -- 172.21.15.90:0/70235335 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7f081809d600 con 0x7f0830104ed0 2024-04-02T14:34:06.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:06.149+0000 7f0834e00640 1 -- 172.21.15.90:0/70235335 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f083002d5c0 con 0x7f0830104ed0 2024-04-02T14:34:06.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:06.153+0000 7f0826a00640 1 -- 172.21.15.90:0/70235335 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+22646 (secure 0 0 0) 0x7f081809cf00 con 0x7f0830104ed0 2024-04-02T14:34:06.157 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-02T14:34:06.157 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":90,"fsid":"6449448a-f0fc-11ee-b647-cb9ed24678a4","created":"2024-04-02T14:23:16.530201+0000","modified":"2024-04-02T14:32:35.716929+0000","last_up_change":"2024-04-02T14:31:23.819732+0000","last_in_change":"2024-04-02T14:31:02.824284+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T14:27:16.758622+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-02T14:32:20.016331+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-02T14:32:21.294367+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a1a6a856-ab44-44bd-bc5e-45c718a70cd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":3635087609},{"type":"v1","addr":"172.21.15.90:6803","nonce":3635087609}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":3635087609},{"type":"v1","addr":"172.21.15.90:6805","nonce":3635087609}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":3635087609},{"type":"v1","addr":"172.21.15.90:6809","nonce":3635087609}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":3635087609},{"type":"v1","addr":"172.21.15.90:6807","nonce":3635087609}]},"public_addr":"172.21.15.90:6803/3635087609","cluster_addr":"172.21.15.90:6805/3635087609","heartbeat_back_addr":"172.21.15.90:6809/3635087609","heartbeat_front_addr":"172.21.15.90:6807/3635087609","state":["exists","up"]},{"osd":1,"uuid":"124139cd-5793-463e-9607-9d9c10164183","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":1994536059},{"type":"v1","addr":"172.21.15.92:6801","nonce":1994536059}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":1994536059},{"type":"v1","addr":"172.21.15.92:6803","nonce":1994536059}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":1994536059},{"type":"v1","addr":"172.21.15.92:6807","nonce":1994536059}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":1994536059},{"type":"v1","addr":"172.21.15.92:6805","nonce":1994536059}]},"public_addr":"172.21.15.92:6801/1994536059","cluster_addr":"172.21.15.92:6803/1994536059","heartbeat_back_addr":"172.21.15.92:6807/1994536059","heartbeat_front_addr":"172.21.15.92:6805/1994536059","state":["exists","up"]},{"osd":2,"uuid":"a61fbef2-9dca-4875-a814-020fff0db0e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":1358472800},{"type":"v1","addr":"172.21.15.177:6801","nonce":1358472800}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":1358472800},{"type":"v1","addr":"172.21.15.177:6803","nonce":1358472800}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":1358472800},{"type":"v1","addr":"172.21.15.177:6807","nonce":1358472800}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":1358472800},{"type":"v1","addr":"172.21.15.177:6805","nonce":1358472800}]},"public_addr":"172.21.15.177:6801/1358472800","cluster_addr":"172.21.15.177:6803/1358472800","heartbeat_back_addr":"172.21.15.177:6807/1358472800","heartbeat_front_addr":"172.21.15.177:6805/1358472800","state":["exists","up"]},{"osd":3,"uuid":"b175a2f4-7859-4eb0-bc1f-fba8c2b92a2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":3114811216},{"type":"v1","addr":"172.21.15.90:6811","nonce":3114811216}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":3114811216},{"type":"v1","addr":"172.21.15.90:6813","nonce":3114811216}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":3114811216},{"type":"v1","addr":"172.21.15.90:6817","nonce":3114811216}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":3114811216},{"type":"v1","addr":"172.21.15.90:6815","nonce":3114811216}]},"public_addr":"172.21.15.90:6811/3114811216","cluster_addr":"172.21.15.90:6813/3114811216","heartbeat_back_addr":"172.21.15.90:6817/3114811216","heartbeat_front_addr":"172.21.15.90:6815/3114811216","state":["exists","up"]},{"osd":4,"uuid":"38cc68ea-c109-4d8d-b77d-1c6f9b56c875","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":2987208879},{"type":"v1","addr":"172.21.15.92:6809","nonce":2987208879}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6810","nonce":2987208879},{"type":"v1","addr":"172.21.15.92:6811","nonce":2987208879}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6814","nonce":2987208879},{"type":"v1","addr":"172.21.15.92:6815","nonce":2987208879}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":2987208879},{"type":"v1","addr":"172.21.15.92:6813","nonce":2987208879}]},"public_addr":"172.21.15.92:6809/2987208879","cluster_addr":"172.21.15.92:6811/2987208879","heartbeat_back_addr":"172.21.15.92:6815/2987208879","heartbeat_front_addr":"172.21.15.92:6813/2987208879","state":["exists","up"]},{"osd":5,"uuid":"2347232e-ac91-466f-8ac9-496f397679ba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":3486820237},{"type":"v1","addr":"172.21.15.177:6809","nonce":3486820237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":3486820237},{"type":"v1","addr":"172.21.15.177:6811","nonce":3486820237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":3486820237},{"type":"v1","addr":"172.21.15.177:6815","nonce":3486820237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":3486820237},{"type":"v1","addr":"172.21.15.177:6813","nonce":3486820237}]},"public_addr":"172.21.15.177:6809/3486820237","cluster_addr":"172.21.15.177:6811/3486820237","heartbeat_back_addr":"172.21.15.177:6815/3486820237","heartbeat_front_addr":"172.21.15.177:6813/3486820237","state":["exists","up"]},{"osd":6,"uuid":"aa72392f-0498-44db-a8d2-eccc89a64871","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":2579716443},{"type":"v1","addr":"172.21.15.90:6819","nonce":2579716443}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6820","nonce":2579716443},{"type":"v1","addr":"172.21.15.90:6821","nonce":2579716443}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":2579716443},{"type":"v1","addr":"172.21.15.90:6825","nonce":2579716443}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2579716443},{"type":"v1","addr":"172.21.15.90:6823","nonce":2579716443}]},"public_addr":"172.21.15.90:6819/2579716443","cluster_addr":"172.21.15.90:6821/2579716443","heartbeat_back_addr":"172.21.15.90:6825/2579716443","heartbeat_front_addr":"172.21.15.90:6823/2579716443","state":["exists","up"]},{"osd":7,"uuid":"931bd105-bc14-4a1e-bfe4-c7ad58fa4b73","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":3371649139},{"type":"v1","addr":"172.21.15.92:6817","nonce":3371649139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":3371649139},{"type":"v1","addr":"172.21.15.92:6819","nonce":3371649139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":3371649139},{"type":"v1","addr":"172.21.15.92:6823","nonce":3371649139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6820","nonce":3371649139},{"type":"v1","addr":"172.21.15.92:6821","nonce":3371649139}]},"public_addr":"172.21.15.92:6817/3371649139","cluster_addr":"172.21.15.92:6819/3371649139","heartbeat_back_addr":"172.21.15.92:6823/3371649139","heartbeat_front_addr":"172.21.15.92:6821/3371649139","state":["exists","up"]},{"osd":8,"uuid":"82624ec2-b382-4886-9f40-7f5f376de3ce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":1459053034},{"type":"v1","addr":"172.21.15.177:6817","nonce":1459053034}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":1459053034},{"type":"v1","addr":"172.21.15.177:6819","nonce":1459053034}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":1459053034},{"type":"v1","addr":"172.21.15.177:6823","nonce":1459053034}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":1459053034},{"type":"v1","addr":"172.21.15.177:6821","nonce":1459053034}]},"public_addr":"172.21.15.177:6817/1459053034","cluster_addr":"172.21.15.177:6819/1459053034","heartbeat_back_addr":"172.21.15.177:6823/1459053034","heartbeat_front_addr":"172.21.15.177:6821/1459053034","state":["exists","up"]},{"osd":9,"uuid":"fbe36e48-3468-42b2-8094-e16552f3cb16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":2592604332},{"type":"v1","addr":"172.21.15.90:6827","nonce":2592604332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":2592604332},{"type":"v1","addr":"172.21.15.90:6829","nonce":2592604332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6832","nonce":2592604332},{"type":"v1","addr":"172.21.15.90:6833","nonce":2592604332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":2592604332},{"type":"v1","addr":"172.21.15.90:6831","nonce":2592604332}]},"public_addr":"172.21.15.90:6827/2592604332","cluster_addr":"172.21.15.90:6829/2592604332","heartbeat_back_addr":"172.21.15.90:6833/2592604332","heartbeat_front_addr":"172.21.15.90:6831/2592604332","state":["exists","up"]},{"osd":10,"uuid":"9240df75-e7e2-47f1-8aae-534b70b3c210","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":1594153142},{"type":"v1","addr":"172.21.15.92:6825","nonce":1594153142}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6826","nonce":1594153142},{"type":"v1","addr":"172.21.15.92:6827","nonce":1594153142}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":1594153142},{"type":"v1","addr":"172.21.15.92:6831","nonce":1594153142}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":1594153142},{"type":"v1","addr":"172.21.15.92:6829","nonce":1594153142}]},"public_addr":"172.21.15.92:6825/1594153142","cluster_addr":"172.21.15.92:6827/1594153142","heartbeat_back_addr":"172.21.15.92:6831/1594153142","heartbeat_front_addr":"172.21.15.92:6829/1594153142","state":["exists","up"]},{"osd":11,"uuid":"83cba9fa-b745-4df1-adda-e86837d4b91a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":83,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1622582126},{"type":"v1","addr":"172.21.15.177:6825","nonce":1622582126}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":1622582126},{"type":"v1","addr":"172.21.15.177:6827","nonce":1622582126}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":1622582126},{"type":"v1","addr":"172.21.15.177:6831","nonce":1622582126}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1622582126},{"type":"v1","addr":"172.21.15.177:6829","nonce":1622582126}]},"public_addr":"172.21.15.177:6825/1622582126","cluster_addr":"172.21.15.177:6827/1622582126","heartbeat_back_addr":"172.21.15.177:6831/1622582126","heartbeat_front_addr":"172.21.15.177:6829/1622582126","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:26:24.290365+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:26:49.405988+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:27:13.639196+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:27:45.603746+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:28:11.019126+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:28:35.084163+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:29:08.064626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:29:33.266133+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:29:57.381816+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:30:30.286847+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:30:56.226279+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:31:20.809611+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":7,"to":11}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":8,"to":4}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.11","mappings":[{"from":4,"to":8},{"from":5,"to":9}]},{"pgid":"3.21","mappings":[{"from":5,"to":9}]},{"pgid":"3.23","mappings":[{"from":4,"to":7}]},{"pgid":"3.3a","mappings":[{"from":4,"to":6}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.90:0/314641601":"2024-04-03T14:24:34.595561+0000","172.21.15.90:0/1400895868":"2024-04-03T14:24:34.595561+0000","172.21.15.90:6801/129436436":"2024-04-03T14:24:34.595561+0000","172.21.15.90:6801/2372398598":"2024-04-03T14:24:00.362147+0000","172.21.15.90:0/278569338":"2024-04-03T14:24:34.595561+0000","172.21.15.90:0/2321464594":"2024-04-03T14:24:00.362147+0000","172.21.15.90:6800/2372398598":"2024-04-03T14:24:00.362147+0000","172.21.15.90:0/2962656672":"2024-04-03T14:24:00.362147+0000","172.21.15.90:0/126693610":"2024-04-03T14:24:00.362147+0000","172.21.15.90:6800/129436436":"2024-04-03T14:24:34.595561+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T14:34:06.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:06.155+0000 7f0834e00640 1 -- 172.21.15.90:0/70235335 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f08100a3bb0 msgr2=0x7f08100a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:06.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:06.155+0000 7f0834e00640 1 --2- 172.21.15.90:0/70235335 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f08100a3bb0 0x7f08100a6070 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f0820006010 tx=0x7f082002c8a0 comp rx=0 tx=0).stop 2024-04-02T14:34:06.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:06.155+0000 7f0834e00640 1 -- 172.21.15.90:0/70235335 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0830104ed0 msgr2=0x7f083019a5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:06.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:06.155+0000 7f0834e00640 1 --2- 172.21.15.90:0/70235335 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0830104ed0 0x7f083019a5c0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f081800ebc0 tx=0x7f081800c6a0 comp rx=0 tx=0).stop 2024-04-02T14:34:06.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:06.155+0000 7f0834e00640 1 -- 172.21.15.90:0/70235335 shutdown_connections 2024-04-02T14:34:06.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:06.155+0000 7f0834e00640 1 --2- 172.21.15.90:0/70235335 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f08100a3bb0 0x7f08100a6070 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:06.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:06.155+0000 7f0834e00640 1 --2- 172.21.15.90:0/70235335 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0830107800 0x7f08301a1630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:06.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:06.155+0000 7f0834e00640 1 --2- 172.21.15.90:0/70235335 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0830104ed0 0x7f083019a5c0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:06.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:06.155+0000 7f0834e00640 1 --2- 172.21.15.90:0/70235335 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f08301025a0 0x7f083019a080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:06.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:06.155+0000 7f0834e00640 1 -- 172.21.15.90:0/70235335 >> 172.21.15.90:0/70235335 conn(0x7f08300fbfb0 msgr2=0x7f0830101c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:34:06.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:06.156+0000 7f0834e00640 1 -- 172.21.15.90:0/70235335 shutdown_connections 2024-04-02T14:34:06.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:06.156+0000 7f0834e00640 1 -- 172.21.15.90:0/70235335 wait complete. 2024-04-02T14:34:06.172 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:34:05 smithi090 ceph-mon[31674]: pgmap v365: 97 pgs: 97 active+clean; 644 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-02T14:34:06.172 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:34:05 smithi090 ceph-mon[31674]: from='client.? 172.21.15.90:0/4272567884' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-02T14:34:06.172 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:34:05 smithi090 ceph-mon[31674]: from='client.? 172.21.15.90:0/3599894414' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-02T14:34:06.176 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-02T14:27:16.758622+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-02T14:32:20.016331+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-02T14:32:21.294367+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '88', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-02T14:34:06.176 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-02T14:34:06.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:34:05 smithi177 ceph-mon[36431]: pgmap v365: 97 pgs: 97 active+clean; 644 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-02T14:34:06.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:34:05 smithi177 ceph-mon[36431]: from='client.? 172.21.15.90:0/4272567884' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-02T14:34:06.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:34:05 smithi177 ceph-mon[36431]: from='client.? 172.21.15.90:0/3599894414' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-02T14:34:06.362 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:34:05 smithi092 ceph-mon[36497]: pgmap v365: 97 pgs: 97 active+clean; 644 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-02T14:34:06.362 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:34:05 smithi092 ceph-mon[36497]: from='client.? 172.21.15.90:0/4272567884' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-02T14:34:06.362 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:34:05 smithi092 ceph-mon[36497]: from='client.? 172.21.15.90:0/3599894414' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-02T14:34:06.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:06.386+0000 7fda07000640 1 -- 172.21.15.90:0/4016638010 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fda00109cf0 msgr2=0x7fda00110580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:06.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:06.386+0000 7fda07000640 1 --2- 172.21.15.90:0/4016638010 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fda00109cf0 0x7fda00110580 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fd9f8009900 tx=0x7fd9f802bc30 comp rx=0 tx=0).stop 2024-04-02T14:34:06.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:06.387+0000 7fda07000640 1 -- 172.21.15.90:0/4016638010 shutdown_connections 2024-04-02T14:34:06.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:06.387+0000 7fda07000640 1 --2- 172.21.15.90:0/4016638010 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fda00109cf0 0x7fda00110580 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:06.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:06.387+0000 7fda07000640 1 --2- 172.21.15.90:0/4016638010 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fda00109330 0x7fda001097b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:06.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:06.387+0000 7fda07000640 1 --2- 172.21.15.90:0/4016638010 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fda00108140 0x7fda00108560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:06.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:06.387+0000 7fda07000640 1 -- 172.21.15.90:0/4016638010 >> 172.21.15.90:0/4016638010 conn(0x7fda000770f0 msgr2=0x7fda00079510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:34:06.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:06.387+0000 7fda07000640 1 -- 172.21.15.90:0/4016638010 shutdown_connections 2024-04-02T14:34:06.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:06.387+0000 7fda07000640 1 -- 172.21.15.90:0/4016638010 wait complete. 2024-04-02T14:34:06.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:06.387+0000 7fda07000640 1 Processor -- start 2024-04-02T14:34:06.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:06.388+0000 7fda07000640 1 -- start start 2024-04-02T14:34:06.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:06.388+0000 7fda07000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fda00108140 0x7fda0019e3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:06.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:06.388+0000 7fda07000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fda00109330 0x7fda0019e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:06.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:06.388+0000 7fda07000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fda00109cf0 0x7fda001a5960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:06.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:06.388+0000 7fda07000640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fda00113e20 con 0x7fda00108140 2024-04-02T14:34:06.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:06.388+0000 7fda07000640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fda00113ca0 con 0x7fda00109cf0 2024-04-02T14:34:06.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:06.388+0000 7fda07000640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fda00113fa0 con 0x7fda00109330 2024-04-02T14:34:06.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:06.388+0000 7fd9ffe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fda00108140 0x7fda0019e3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:06.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:06.388+0000 7fda05000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fda00109cf0 0x7fda001a5960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:06.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:06.388+0000 7fd9ffe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fda00108140 0x7fda0019e3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:52138/0 (socket says 172.21.15.90:52138) 2024-04-02T14:34:06.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:06.388+0000 7fd9ffe00640 1 -- 172.21.15.90:0/757604878 learned_addr learned my addr 172.21.15.90:0/757604878 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:34:06.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:06.388+0000 7fd9f7400640 1 --2- 172.21.15.90:0/757604878 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fda00109330 0x7fda0019e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:06.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:06.389+0000 7fda05000640 1 -- 172.21.15.90:0/757604878 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fda00109330 msgr2=0x7fda0019e8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:06.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:06.389+0000 7fda05000640 1 --2- 172.21.15.90:0/757604878 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fda00109330 0x7fda0019e8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:06.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:06.389+0000 7fda05000640 1 -- 172.21.15.90:0/757604878 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fda00108140 msgr2=0x7fda0019e3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:06.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:06.389+0000 7fda05000640 1 --2- 172.21.15.90:0/757604878 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fda00108140 0x7fda0019e3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:06.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:06.389+0000 7fda05000640 1 -- 172.21.15.90:0/757604878 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda001a5fd0 con 0x7fda00109cf0 2024-04-02T14:34:06.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:06.389+0000 7fda05000640 1 --2- 172.21.15.90:0/757604878 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fda00109cf0 0x7fda001a5960 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fd9f802c170 tx=0x7fd9f802c640 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:34:06.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:06.389+0000 7fd9fd600640 1 -- 172.21.15.90:0/757604878 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9f8043020 con 0x7fda00109cf0 2024-04-02T14:34:06.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:06.389+0000 7fda07000640 1 -- 172.21.15.90:0/757604878 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda001a6260 con 0x7fda00109cf0 2024-04-02T14:34:06.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:06.389+0000 7fd9fd600640 1 -- 172.21.15.90:0/757604878 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd9f80054e0 con 0x7fda00109cf0 2024-04-02T14:34:06.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:06.389+0000 7fd9fd600640 1 -- 172.21.15.90:0/757604878 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9f8003b40 con 0x7fda00109cf0 2024-04-02T14:34:06.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:06.389+0000 7fda07000640 1 -- 172.21.15.90:0/757604878 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fda001a67d0 con 0x7fda00109cf0 2024-04-02T14:34:06.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:06.390+0000 7fda07000640 1 -- 172.21.15.90:0/757604878 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fda0010f400 con 0x7fda00109cf0 2024-04-02T14:34:06.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:06.392+0000 7fd9fd600640 1 -- 172.21.15.90:0/757604878 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fd9f8046020 con 0x7fda00109cf0 2024-04-02T14:34:06.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:06.394+0000 7fd9fd600640 1 --2- 172.21.15.90:0/757604878 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fd9e40a3bb0 0x7fd9e40a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:06.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:06.394+0000 7fd9fd600640 1 -- 172.21.15.90:0/757604878 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8248+0+0 (secure 0 0 0) 0x7fd9f80f27b0 con 0x7fda00109cf0 2024-04-02T14:34:06.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:06.394+0000 7fd9ffe00640 1 --2- 172.21.15.90:0/757604878 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fd9e40a3bb0 0x7fd9e40a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:06.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:06.395+0000 7fd9ffe00640 1 --2- 172.21.15.90:0/757604878 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fd9e40a3bb0 0x7fd9e40a6070 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fd9f0005f40 tx=0x7fd9f0009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:34:06.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:06.396+0000 7fd9fd600640 1 -- 172.21.15.90:0/757604878 <== mon.2 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7fd9f804d020 con 0x7fda00109cf0 2024-04-02T14:34:06.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:06.588+0000 7fda07000640 1 -- 172.21.15.90:0/757604878 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fda00062700 con 0x7fda00109cf0 2024-04-02T14:34:06.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:06.589+0000 7fd9fd600640 1 -- 172.21.15.90:0/757604878 <== mon.2 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v90) ==== 93+0+10 (secure 0 0 0) 0x7fd9f80bf1b0 con 0x7fda00109cf0 2024-04-02T14:34:06.590 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 1 2024-04-02T14:34:06.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:06.591+0000 7fda07000640 1 -- 172.21.15.90:0/757604878 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fd9e40a3bb0 msgr2=0x7fd9e40a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:06.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:06.591+0000 7fda07000640 1 --2- 172.21.15.90:0/757604878 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fd9e40a3bb0 0x7fd9e40a6070 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fd9f0005f40 tx=0x7fd9f0009500 comp rx=0 tx=0).stop 2024-04-02T14:34:06.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:06.591+0000 7fda07000640 1 -- 172.21.15.90:0/757604878 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fda00109cf0 msgr2=0x7fda001a5960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:06.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:06.591+0000 7fda07000640 1 --2- 172.21.15.90:0/757604878 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fda00109cf0 0x7fda001a5960 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fd9f802c170 tx=0x7fd9f802c640 comp rx=0 tx=0).stop 2024-04-02T14:34:06.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:06.591+0000 7fda07000640 1 -- 172.21.15.90:0/757604878 shutdown_connections 2024-04-02T14:34:06.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:06.591+0000 7fda07000640 1 --2- 172.21.15.90:0/757604878 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fd9e40a3bb0 0x7fd9e40a6070 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:06.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:06.591+0000 7fda07000640 1 --2- 172.21.15.90:0/757604878 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fda00109cf0 0x7fda001a5960 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:06.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:06.591+0000 7fda07000640 1 --2- 172.21.15.90:0/757604878 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fda00109330 0x7fda0019e8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:06.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:06.591+0000 7fda07000640 1 --2- 172.21.15.90:0/757604878 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fda00108140 0x7fda0019e3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:06.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:06.591+0000 7fda07000640 1 -- 172.21.15.90:0/757604878 >> 172.21.15.90:0/757604878 conn(0x7fda000770f0 msgr2=0x7fda0010c5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:34:06.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:06.592+0000 7fda07000640 1 -- 172.21.15.90:0/757604878 shutdown_connections 2024-04-02T14:34:06.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:06.592+0000 7fda07000640 1 -- 172.21.15.90:0/757604878 wait complete. 2024-04-02T14:34:06.611 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-02T14:34:06.822 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:06.820+0000 7fe237400640 1 -- 172.21.15.90:0/2122287734 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe230077700 msgr2=0x7fe230077b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:06.822 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:06.820+0000 7fe237400640 1 --2- 172.21.15.90:0/2122287734 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe230077700 0x7fe230077b60 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fe21c009900 tx=0x7fe21c02bc30 comp rx=0 tx=0).stop 2024-04-02T14:34:06.822 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:06.820+0000 7fe237400640 1 -- 172.21.15.90:0/2122287734 shutdown_connections 2024-04-02T14:34:06.822 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:06.820+0000 7fe237400640 1 --2- 172.21.15.90:0/2122287734 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe2300780a0 0x7fe230078940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:06.822 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:06.820+0000 7fe237400640 1 --2- 172.21.15.90:0/2122287734 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe230077700 0x7fe230077b60 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:06.822 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:06.820+0000 7fe237400640 1 --2- 172.21.15.90:0/2122287734 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe2300764c0 0x7fe2300768e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:06.822 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:06.820+0000 7fe237400640 1 -- 172.21.15.90:0/2122287734 >> 172.21.15.90:0/2122287734 conn(0x7fe2300ff2b0 msgr2=0x7fe2301016f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:34:06.822 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:06.820+0000 7fe237400640 1 -- 172.21.15.90:0/2122287734 shutdown_connections 2024-04-02T14:34:06.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:06.820+0000 7fe237400640 1 -- 172.21.15.90:0/2122287734 wait complete. 2024-04-02T14:34:06.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:06.821+0000 7fe237400640 1 Processor -- start 2024-04-02T14:34:06.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:06.822+0000 7fe237400640 1 -- start start 2024-04-02T14:34:06.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:06.822+0000 7fe237400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe2300764c0 0x7fe23019e430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:06.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:06.822+0000 7fe237400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe230077700 0x7fe23019e970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:06.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:06.822+0000 7fe237400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe2300780a0 0x7fe2301a59e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:06.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:06.822+0000 7fe237400640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fe230113e10 con 0x7fe2300764c0 2024-04-02T14:34:06.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:06.822+0000 7fe237400640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fe230113c90 con 0x7fe230077700 2024-04-02T14:34:06.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:06.822+0000 7fe237400640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fe230113f90 con 0x7fe2300780a0 2024-04-02T14:34:06.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:06.822+0000 7fe234a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe2300764c0 0x7fe23019e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:06.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:06.822+0000 7fe234a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe2300764c0 0x7fe23019e430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:52158/0 (socket says 172.21.15.90:52158) 2024-04-02T14:34:06.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:06.822+0000 7fe234a00640 1 -- 172.21.15.90:0/1241613446 learned_addr learned my addr 172.21.15.90:0/1241613446 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:34:06.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:06.822+0000 7fe235400640 1 --2- 172.21.15.90:0/1241613446 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe2300780a0 0x7fe2301a59e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:06.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:06.822+0000 7fe22fe00640 1 --2- 172.21.15.90:0/1241613446 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe230077700 0x7fe23019e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:06.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:06.823+0000 7fe234a00640 1 -- 172.21.15.90:0/1241613446 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe2300780a0 msgr2=0x7fe2301a59e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:06.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:06.823+0000 7fe234a00640 1 --2- 172.21.15.90:0/1241613446 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe2300780a0 0x7fe2301a59e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:06.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:06.823+0000 7fe234a00640 1 -- 172.21.15.90:0/1241613446 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe230077700 msgr2=0x7fe23019e970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:06.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:06.823+0000 7fe234a00640 1 --2- 172.21.15.90:0/1241613446 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe230077700 0x7fe23019e970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:06.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:06.823+0000 7fe234a00640 1 -- 172.21.15.90:0/1241613446 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2301a6050 con 0x7fe2300764c0 2024-04-02T14:34:06.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:06.823+0000 7fe234a00640 1 --2- 172.21.15.90:0/1241613446 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe2300764c0 0x7fe23019e430 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fe22400ea60 tx=0x7fe22400ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:34:06.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:06.823+0000 7fe22d600640 1 -- 172.21.15.90:0/1241613446 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe22400cde0 con 0x7fe2300764c0 2024-04-02T14:34:06.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:06.823+0000 7fe237400640 1 -- 172.21.15.90:0/1241613446 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe2301a6340 con 0x7fe2300764c0 2024-04-02T14:34:06.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:06.823+0000 7fe22d600640 1 -- 172.21.15.90:0/1241613446 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe2240035f0 con 0x7fe2300764c0 2024-04-02T14:34:06.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:06.823+0000 7fe22d600640 1 -- 172.21.15.90:0/1241613446 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe224010670 con 0x7fe2300764c0 2024-04-02T14:34:06.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:06.823+0000 7fe237400640 1 -- 172.21.15.90:0/1241613446 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2301a6860 con 0x7fe2300764c0 2024-04-02T14:34:06.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:06.825+0000 7fe237400640 1 -- 172.21.15.90:0/1241613446 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe230110840 con 0x7fe2300764c0 2024-04-02T14:34:06.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:06.826+0000 7fe22d600640 1 -- 172.21.15.90:0/1241613446 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fe2240037a0 con 0x7fe2300764c0 2024-04-02T14:34:06.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:06.829+0000 7fe22d600640 1 --2- 172.21.15.90:0/1241613446 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fe2180a3bb0 0x7fe2180a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:06.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:06.829+0000 7fe22fe00640 1 --2- 172.21.15.90:0/1241613446 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fe2180a3bb0 0x7fe2180a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:06.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:06.829+0000 7fe22d600640 1 -- 172.21.15.90:0/1241613446 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8248+0+0 (secure 0 0 0) 0x7fe2240d0720 con 0x7fe2300764c0 2024-04-02T14:34:06.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:06.830+0000 7fe22fe00640 1 --2- 172.21.15.90:0/1241613446 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fe2180a3bb0 0x7fe2180a6070 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fe21c02c240 tx=0x7fe21c02df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:34:06.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:06.833+0000 7fe22d600640 1 -- 172.21.15.90:0/1241613446 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7fe22409d090 con 0x7fe2300764c0 2024-04-02T14:34:07.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:07.036+0000 7fe237400640 1 -- 172.21.15.90:0/1241613446 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fe230062700 con 0x7fe2300764c0 2024-04-02T14:34:07.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:07.037+0000 7fe22d600640 1 -- 172.21.15.90:0/1241613446 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v90) ==== 104+0+11 (secure 0 0 0) 0x7fe22409c990 con 0x7fe2300764c0 2024-04-02T14:34:07.038 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-04-02T14:34:07.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:07.039+0000 7fe237400640 1 -- 172.21.15.90:0/1241613446 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fe2180a3bb0 msgr2=0x7fe2180a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:07.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:07.039+0000 7fe237400640 1 --2- 172.21.15.90:0/1241613446 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fe2180a3bb0 0x7fe2180a6070 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fe21c02c240 tx=0x7fe21c02df90 comp rx=0 tx=0).stop 2024-04-02T14:34:07.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:07.040+0000 7fe237400640 1 -- 172.21.15.90:0/1241613446 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe2300764c0 msgr2=0x7fe23019e430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:07.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:07.040+0000 7fe237400640 1 --2- 172.21.15.90:0/1241613446 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe2300764c0 0x7fe23019e430 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fe22400ea60 tx=0x7fe22400ef20 comp rx=0 tx=0).stop 2024-04-02T14:34:07.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:07.040+0000 7fe237400640 1 -- 172.21.15.90:0/1241613446 shutdown_connections 2024-04-02T14:34:07.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:07.040+0000 7fe237400640 1 --2- 172.21.15.90:0/1241613446 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fe2180a3bb0 0x7fe2180a6070 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:07.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:07.040+0000 7fe237400640 1 --2- 172.21.15.90:0/1241613446 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe2300780a0 0x7fe2301a59e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:07.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:07.040+0000 7fe237400640 1 --2- 172.21.15.90:0/1241613446 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe230077700 0x7fe23019e970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:07.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:07.040+0000 7fe237400640 1 --2- 172.21.15.90:0/1241613446 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe2300764c0 0x7fe23019e430 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:07.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:07.040+0000 7fe237400640 1 -- 172.21.15.90:0/1241613446 >> 172.21.15.90:0/1241613446 conn(0x7fe2300ff2b0 msgr2=0x7fe2301009a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:34:07.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:07.040+0000 7fe237400640 1 -- 172.21.15.90:0/1241613446 shutdown_connections 2024-04-02T14:34:07.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:07.040+0000 7fe237400640 1 -- 172.21.15.90:0/1241613446 wait complete. 2024-04-02T14:34:07.062 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-02T14:34:07.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:34:06 smithi177 ceph-mon[36431]: from='client.? 172.21.15.90:0/70235335' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-02T14:34:07.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:34:06 smithi177 ceph-mon[36431]: from='client.? 172.21.15.90:0/757604878' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-02T14:34:07.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:34:06 smithi090 ceph-mon[31674]: from='client.? 172.21.15.90:0/70235335' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-02T14:34:07.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:34:06 smithi090 ceph-mon[31674]: from='client.? 172.21.15.90:0/757604878' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-02T14:34:07.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:07.278+0000 7fab1b400640 1 -- 172.21.15.90:0/2014698413 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fab14101b10 msgr2=0x7fab14101f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:07.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:07.278+0000 7fab1b400640 1 --2- 172.21.15.90:0/2014698413 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fab14101b10 0x7fab14101f30 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fab04009900 tx=0x7fab0402bc50 comp rx=0 tx=0).stop 2024-04-02T14:34:07.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:07.278+0000 7fab1b400640 1 -- 172.21.15.90:0/2014698413 shutdown_connections 2024-04-02T14:34:07.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:07.278+0000 7fab1b400640 1 --2- 172.21.15.90:0/2014698413 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fab141036c0 0x7fab14109f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:07.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:07.278+0000 7fab1b400640 1 --2- 172.21.15.90:0/2014698413 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fab14102d00 0x7fab14103180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:07.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:07.278+0000 7fab1b400640 1 --2- 172.21.15.90:0/2014698413 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fab14101b10 0x7fab14101f30 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:07.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:07.278+0000 7fab1b400640 1 -- 172.21.15.90:0/2014698413 >> 172.21.15.90:0/2014698413 conn(0x7fab140fd0a0 msgr2=0x7fab140ff4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:34:07.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:07.278+0000 7fab1b400640 1 -- 172.21.15.90:0/2014698413 shutdown_connections 2024-04-02T14:34:07.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:07.278+0000 7fab1b400640 1 -- 172.21.15.90:0/2014698413 wait complete. 2024-04-02T14:34:07.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:07.279+0000 7fab1b400640 1 Processor -- start 2024-04-02T14:34:07.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:07.280+0000 7fab1b400640 1 -- start start 2024-04-02T14:34:07.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:07.280+0000 7fab1b400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fab14101b10 0x7fab1419a070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:07.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:07.280+0000 7fab1b400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fab14102d00 0x7fab1419a5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:07.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:07.280+0000 7fab1b400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fab141036c0 0x7fab141a1620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:07.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:07.280+0000 7fab1b400640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fab1410d7b0 con 0x7fab14102d00 2024-04-02T14:34:07.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:07.280+0000 7fab1b400640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fab1410d630 con 0x7fab141036c0 2024-04-02T14:34:07.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:07.280+0000 7fab1b400640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fab1410d930 con 0x7fab14101b10 2024-04-02T14:34:07.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:07.280+0000 7fab0be00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fab14102d00 0x7fab1419a5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:07.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:07.280+0000 7fab0be00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fab14102d00 0x7fab1419a5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:52170/0 (socket says 172.21.15.90:52170) 2024-04-02T14:34:07.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:07.280+0000 7fab0be00640 1 -- 172.21.15.90:0/362986111 learned_addr learned my addr 172.21.15.90:0/362986111 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:34:07.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:07.280+0000 7fab18a00640 1 --2- 172.21.15.90:0/362986111 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fab14101b10 0x7fab1419a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:07.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:07.280+0000 7fab19400640 1 --2- 172.21.15.90:0/362986111 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fab141036c0 0x7fab141a1620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:07.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:07.281+0000 7fab0be00640 1 -- 172.21.15.90:0/362986111 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fab14101b10 msgr2=0x7fab1419a070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:07.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:07.281+0000 7fab0be00640 1 --2- 172.21.15.90:0/362986111 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fab14101b10 0x7fab1419a070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:07.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:07.281+0000 7fab0be00640 1 -- 172.21.15.90:0/362986111 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fab141036c0 msgr2=0x7fab141a1620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:07.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:07.281+0000 7fab0be00640 1 --2- 172.21.15.90:0/362986111 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fab141036c0 0x7fab141a1620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:07.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:07.281+0000 7fab0be00640 1 -- 172.21.15.90:0/362986111 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab141a1c90 con 0x7fab14102d00 2024-04-02T14:34:07.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:07.281+0000 7fab0be00640 1 --2- 172.21.15.90:0/362986111 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fab14102d00 0x7fab1419a5b0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7faafc00d950 tx=0x7faafc00dd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:34:07.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:07.281+0000 7fab12000640 1 -- 172.21.15.90:0/362986111 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faafc017070 con 0x7fab14102d00 2024-04-02T14:34:07.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:07.281+0000 7fab12000640 1 -- 172.21.15.90:0/362986111 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faafc00de80 con 0x7fab14102d00 2024-04-02T14:34:07.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:07.281+0000 7fab12000640 1 -- 172.21.15.90:0/362986111 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faafc00f620 con 0x7fab14102d00 2024-04-02T14:34:07.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:07.281+0000 7fab1b400640 1 -- 172.21.15.90:0/362986111 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fab141a1f80 con 0x7fab14102d00 2024-04-02T14:34:07.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:07.281+0000 7fab1b400640 1 -- 172.21.15.90:0/362986111 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fab141a23f0 con 0x7fab14102d00 2024-04-02T14:34:07.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:07.282+0000 7fab1b400640 1 -- 172.21.15.90:0/362986111 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fab141a2620 con 0x7fab14102d00 2024-04-02T14:34:07.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:07.285+0000 7fab12000640 1 -- 172.21.15.90:0/362986111 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7faafc009c30 con 0x7fab14102d00 2024-04-02T14:34:07.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:07.286+0000 7fab12000640 1 --2- 172.21.15.90:0/362986111 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7faaf80a3c30 0x7faaf80a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:07.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:07.286+0000 7fab12000640 1 -- 172.21.15.90:0/362986111 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8248+0+0 (secure 0 0 0) 0x7faafc0cffe0 con 0x7fab14102d00 2024-04-02T14:34:07.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:07.287+0000 7fab18a00640 1 --2- 172.21.15.90:0/362986111 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7faaf80a3c30 0x7faaf80a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:07.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:07.287+0000 7fab18a00640 1 --2- 172.21.15.90:0/362986111 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7faaf80a3c30 0x7faaf80a60f0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fab0402c190 tx=0x7fab04000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:34:07.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:07.290+0000 7fab12000640 1 -- 172.21.15.90:0/362986111 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7faafc09c950 con 0x7fab14102d00 2024-04-02T14:34:07.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:34:06 smithi092 ceph-mon[36497]: from='client.? 172.21.15.90:0/70235335' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-02T14:34:07.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:34:06 smithi092 ceph-mon[36497]: from='client.? 172.21.15.90:0/757604878' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-02T14:34:07.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:07.491+0000 7fab1b400640 1 -- 172.21.15.90:0/362986111 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fab141a2890 con 0x7fab14102d00 2024-04-02T14:34:07.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:07.492+0000 7fab12000640 1 -- 172.21.15.90:0/362986111 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v90) ==== 100+0+11 (secure 0 0 0) 0x7faafc09c250 con 0x7fab14102d00 2024-04-02T14:34:07.494 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 64 2024-04-02T14:34:07.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:07.494+0000 7fab1b400640 1 -- 172.21.15.90:0/362986111 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7faaf80a3c30 msgr2=0x7faaf80a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:07.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:07.494+0000 7fab1b400640 1 --2- 172.21.15.90:0/362986111 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7faaf80a3c30 0x7faaf80a60f0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fab0402c190 tx=0x7fab04000bc0 comp rx=0 tx=0).stop 2024-04-02T14:34:07.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:07.494+0000 7fab1b400640 1 -- 172.21.15.90:0/362986111 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fab14102d00 msgr2=0x7fab1419a5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:07.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:07.494+0000 7fab1b400640 1 --2- 172.21.15.90:0/362986111 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fab14102d00 0x7fab1419a5b0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7faafc00d950 tx=0x7faafc00dd50 comp rx=0 tx=0).stop 2024-04-02T14:34:07.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:07.495+0000 7fab1b400640 1 -- 172.21.15.90:0/362986111 shutdown_connections 2024-04-02T14:34:07.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:07.495+0000 7fab1b400640 1 --2- 172.21.15.90:0/362986111 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7faaf80a3c30 0x7faaf80a60f0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:07.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:07.495+0000 7fab1b400640 1 --2- 172.21.15.90:0/362986111 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fab141036c0 0x7fab141a1620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:07.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:07.495+0000 7fab1b400640 1 --2- 172.21.15.90:0/362986111 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fab14102d00 0x7fab1419a5b0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:07.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:07.495+0000 7fab1b400640 1 --2- 172.21.15.90:0/362986111 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fab14101b10 0x7fab1419a070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:07.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:07.495+0000 7fab1b400640 1 -- 172.21.15.90:0/362986111 >> 172.21.15.90:0/362986111 conn(0x7fab140fd0a0 msgr2=0x7fab14073d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:34:07.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:07.495+0000 7fab1b400640 1 -- 172.21.15.90:0/362986111 shutdown_connections 2024-04-02T14:34:07.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:07.495+0000 7fab1b400640 1 -- 172.21.15.90:0/362986111 wait complete. 2024-04-02T14:34:07.515 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T14:34:07.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:07.723+0000 7f6fb3800640 1 -- 172.21.15.90:0/754641626 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6fac109cf0 msgr2=0x7f6fac110580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:07.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:07.723+0000 7f6fb3800640 1 --2- 172.21.15.90:0/754641626 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6fac109cf0 0x7f6fac110580 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f6fa4009be0 tx=0x7f6fa402bef0 comp rx=0 tx=0).stop 2024-04-02T14:34:07.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:07.723+0000 7f6fb3800640 1 -- 172.21.15.90:0/754641626 shutdown_connections 2024-04-02T14:34:07.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:07.723+0000 7f6fb3800640 1 --2- 172.21.15.90:0/754641626 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6fac109cf0 0x7f6fac110580 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:07.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:07.723+0000 7f6fb3800640 1 --2- 172.21.15.90:0/754641626 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6fac109330 0x7f6fac1097b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:07.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:07.723+0000 7f6fb3800640 1 --2- 172.21.15.90:0/754641626 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6fac108140 0x7f6fac108560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:07.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:07.723+0000 7f6fb3800640 1 -- 172.21.15.90:0/754641626 >> 172.21.15.90:0/754641626 conn(0x7f6fac077110 msgr2=0x7f6fac079530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:34:07.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:07.724+0000 7f6fb3800640 1 -- 172.21.15.90:0/754641626 shutdown_connections 2024-04-02T14:34:07.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:07.724+0000 7f6fb3800640 1 -- 172.21.15.90:0/754641626 wait complete. 2024-04-02T14:34:07.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:07.724+0000 7f6fb3800640 1 Processor -- start 2024-04-02T14:34:07.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:07.725+0000 7f6fb3800640 1 -- start start 2024-04-02T14:34:07.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:07.725+0000 7f6fb3800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6fac108140 0x7f6fac19e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:07.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:07.725+0000 7f6fb3800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6fac109330 0x7f6fac19e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:07.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:07.725+0000 7f6fb3800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6fac109cf0 0x7f6fac1a5990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:07.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:07.725+0000 7f6fb3800640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f6fac113e20 con 0x7f6fac109330 2024-04-02T14:34:07.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:07.725+0000 7f6fb3800640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f6fac113ca0 con 0x7f6fac108140 2024-04-02T14:34:07.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:07.725+0000 7f6fb3800640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f6fac113fa0 con 0x7f6fac109cf0 2024-04-02T14:34:07.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:07.725+0000 7f6fb0e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6fac108140 0x7f6fac19e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:07.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:07.725+0000 7f6fabe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6fac109330 0x7f6fac19e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:07.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:07.725+0000 7f6fb0e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6fac108140 0x7f6fac19e3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:57154/0 (socket says 172.21.15.90:57154) 2024-04-02T14:34:07.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:07.725+0000 7f6fb0e00640 1 -- 172.21.15.90:0/333001413 learned_addr learned my addr 172.21.15.90:0/333001413 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:34:07.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:07.725+0000 7f6fb1800640 1 --2- 172.21.15.90:0/333001413 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6fac109cf0 0x7f6fac1a5990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:07.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:07.726+0000 7f6fb0e00640 1 -- 172.21.15.90:0/333001413 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6fac109cf0 msgr2=0x7f6fac1a5990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:07.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:07.726+0000 7f6fb0e00640 1 --2- 172.21.15.90:0/333001413 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6fac109cf0 0x7f6fac1a5990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:07.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:07.726+0000 7f6fb0e00640 1 -- 172.21.15.90:0/333001413 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6fac109330 msgr2=0x7f6fac19e920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:07.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:07.726+0000 7f6fb0e00640 1 --2- 172.21.15.90:0/333001413 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6fac109330 0x7f6fac19e920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:07.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:07.726+0000 7f6fb0e00640 1 -- 172.21.15.90:0/333001413 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6fac1a6000 con 0x7f6fac108140 2024-04-02T14:34:07.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:07.726+0000 7f6fb1800640 1 --2- 172.21.15.90:0/333001413 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6fac109cf0 0x7f6fac1a5990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:34:07.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:07.726+0000 7f6fabe00640 1 --2- 172.21.15.90:0/333001413 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6fac109330 0x7f6fac19e920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:34:07.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:07.726+0000 7f6fb0e00640 1 --2- 172.21.15.90:0/333001413 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6fac108140 0x7f6fac19e3e0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f6f9c00e9c0 tx=0x7f6f9c00edc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:34:07.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:07.726+0000 7f6fa9600640 1 -- 172.21.15.90:0/333001413 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f9c00ccc0 con 0x7f6fac108140 2024-04-02T14:34:07.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:07.726+0000 7f6fa9600640 1 -- 172.21.15.90:0/333001413 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6f9c0035f0 con 0x7f6fac108140 2024-04-02T14:34:07.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:07.726+0000 7f6fb3800640 1 -- 172.21.15.90:0/333001413 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6fac1a62f0 con 0x7f6fac108140 2024-04-02T14:34:07.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:07.726+0000 7f6fa9600640 1 -- 172.21.15.90:0/333001413 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f9c00ce60 con 0x7f6fac108140 2024-04-02T14:34:07.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:07.726+0000 7f6fb3800640 1 -- 172.21.15.90:0/333001413 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6fac1a6840 con 0x7f6fac108140 2024-04-02T14:34:07.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:07.727+0000 7f6fb3800640 1 -- 172.21.15.90:0/333001413 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6fac10f390 con 0x7f6fac108140 2024-04-02T14:34:07.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:07.730+0000 7f6fa9600640 1 -- 172.21.15.90:0/333001413 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f6f9c0037a0 con 0x7f6fac108140 2024-04-02T14:34:07.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:07.731+0000 7f6fa9600640 1 --2- 172.21.15.90:0/333001413 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f6f8c0a3bb0 0x7f6f8c0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:07.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:07.731+0000 7f6fa9600640 1 -- 172.21.15.90:0/333001413 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8248+0+0 (secure 0 0 0) 0x7f6f9c0d0950 con 0x7f6fac108140 2024-04-02T14:34:07.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:07.731+0000 7f6fabe00640 1 --2- 172.21.15.90:0/333001413 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f6f8c0a3bb0 0x7f6f8c0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:07.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:07.732+0000 7f6fabe00640 1 --2- 172.21.15.90:0/333001413 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f6f8c0a3bb0 0x7f6f8c0a6070 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f6fac19f8f0 tx=0x7f6f94009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:34:07.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:07.733+0000 7f6fa9600640 1 -- 172.21.15.90:0/333001413 <== mon.2 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7f6f9c01f340 con 0x7f6fac108140 2024-04-02T14:34:07.976 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:07.975+0000 7f6fb3800640 1 -- 172.21.15.90:0/333001413 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6fac02d370 con 0x7f6fac108140 2024-04-02T14:34:07.977 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:07.976+0000 7f6fa9600640 1 -- 172.21.15.90:0/333001413 <== mon.2 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11927 (secure 0 0 0) 0x7f6f9c09d2c0 con 0x7f6fac108140 2024-04-02T14:34:07.978 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-02T14:34:07.978 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14412,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6837/2852160309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6836","nonce":2852160309},{"type":"v1","addr":"172.21.15.90:6837","nonce":2852160309}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14478,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6841/1919376858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6840","nonce":1919376858},{"type":"v1","addr":"172.21.15.90:6841","nonce":1919376858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":26,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-02T14:32:22.319083+0000","modified":"2024-04-02T14:32:29.812071+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24590,"mds_1":24560,"mds_2":24547,"mds_3":24539,"mds_4":24523},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24454":{"gid":24454,"name":"l","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6833/1564319431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":1564319431},{"type":"v1","addr":"172.21.15.177:6833","nonce":1564319431}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.92:6833/1570337372","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":1570337372},{"type":"v1","addr":"172.21.15.92:6833","nonce":1570337372}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"j","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.90:6835/1277124891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6834","nonce":1277124891},{"type":"v1","addr":"172.21.15.90:6835","nonce":1277124891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.92:6835/860721164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":860721164},{"type":"v1","addr":"172.21.15.92:6835","nonce":860721164}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24511":{"gid":24511,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6835/3177244988","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":3177244988},{"type":"v1","addr":"172.21.15.177:6835","nonce":3177244988}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"c","rank":4,"incarnation":25,"state":"up:active","state_seq":9,"addr":"172.21.15.177:6837/4223414858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":4223414858},{"type":"v1","addr":"172.21.15.177:6837","nonce":4223414858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24539":{"gid":24539,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":8,"addr":"172.21.15.92:6837/1270635939","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":1270635939},{"type":"v1","addr":"172.21.15.92:6837","nonce":1270635939}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24547":{"gid":24547,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":7,"addr":"172.21.15.177:6839/664895663","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":664895663},{"type":"v1","addr":"172.21.15.177:6839","nonce":664895663}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"g","rank":1,"incarnation":19,"state":"up:active","state_seq":5,"addr":"172.21.15.90:6839/1498915870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6838","nonce":1498915870},{"type":"v1","addr":"172.21.15.90:6839","nonce":1498915870}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24590":{"gid":24590,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.92:6839/1054888219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":1054888219},{"type":"v1","addr":"172.21.15.92:6839","nonce":1054888219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24590,"qdb_cluster":[24539,24560,24523,24590,24547]},"id":1}]} 2024-04-02T14:34:07.978 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 26 2024-04-02T14:34:07.980 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:07.979+0000 7f6fb3800640 1 -- 172.21.15.90:0/333001413 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f6f8c0a3bb0 msgr2=0x7f6f8c0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:07.980 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:07.979+0000 7f6fb3800640 1 --2- 172.21.15.90:0/333001413 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f6f8c0a3bb0 0x7f6f8c0a6070 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f6fac19f8f0 tx=0x7f6f94009260 comp rx=0 tx=0).stop 2024-04-02T14:34:07.980 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:07.979+0000 7f6fb3800640 1 -- 172.21.15.90:0/333001413 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6fac108140 msgr2=0x7f6fac19e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:07.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:07.979+0000 7f6fb3800640 1 --2- 172.21.15.90:0/333001413 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6fac108140 0x7f6fac19e3e0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f6f9c00e9c0 tx=0x7f6f9c00edc0 comp rx=0 tx=0).stop 2024-04-02T14:34:07.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:07.979+0000 7f6fb3800640 1 -- 172.21.15.90:0/333001413 shutdown_connections 2024-04-02T14:34:07.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:07.979+0000 7f6fb3800640 1 --2- 172.21.15.90:0/333001413 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f6f8c0a3bb0 0x7f6f8c0a6070 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:07.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:07.979+0000 7f6fb3800640 1 --2- 172.21.15.90:0/333001413 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6fac109cf0 0x7f6fac1a5990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:07.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:07.979+0000 7f6fb3800640 1 --2- 172.21.15.90:0/333001413 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6fac109330 0x7f6fac19e920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:07.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:07.979+0000 7f6fb3800640 1 --2- 172.21.15.90:0/333001413 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6fac108140 0x7f6fac19e3e0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:07.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:07.979+0000 7f6fb3800640 1 -- 172.21.15.90:0/333001413 >> 172.21.15.90:0/333001413 conn(0x7f6fac077110 msgr2=0x7f6fac10c560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:34:07.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:07.979+0000 7f6fb3800640 1 -- 172.21.15.90:0/333001413 shutdown_connections 2024-04-02T14:34:07.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:07.979+0000 7f6fb3800640 1 -- 172.21.15.90:0/333001413 wait complete. 2024-04-02T14:34:08.001 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T14:34:08.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:34:07 smithi177 ceph-mon[36431]: pgmap v366: 97 pgs: 97 active+clean; 644 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-02T14:34:08.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:34:07 smithi177 ceph-mon[36431]: from='client.? 172.21.15.90:0/1241613446' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-02T14:34:08.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:34:07 smithi177 ceph-mon[36431]: from='client.? 172.21.15.90:0/362986111' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-02T14:34:08.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:34:07 smithi090 ceph-mon[31674]: pgmap v366: 97 pgs: 97 active+clean; 644 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-02T14:34:08.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:34:07 smithi090 ceph-mon[31674]: from='client.? 172.21.15.90:0/1241613446' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-02T14:34:08.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:34:07 smithi090 ceph-mon[31674]: from='client.? 172.21.15.90:0/362986111' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-02T14:34:08.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:08.202+0000 7fbc19600640 1 -- 172.21.15.90:0/267511199 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbc14077700 msgr2=0x7fbc14077b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:08.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:08.202+0000 7fbc19600640 1 --2- 172.21.15.90:0/267511199 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbc14077700 0x7fbc14077b60 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fbc00009860 tx=0x7fbc0002bb70 comp rx=0 tx=0).stop 2024-04-02T14:34:08.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:08.202+0000 7fbc19600640 1 -- 172.21.15.90:0/267511199 shutdown_connections 2024-04-02T14:34:08.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:08.202+0000 7fbc19600640 1 --2- 172.21.15.90:0/267511199 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbc140780a0 0x7fbc14078940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:08.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:08.202+0000 7fbc19600640 1 --2- 172.21.15.90:0/267511199 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbc14077700 0x7fbc14077b60 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:08.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:08.202+0000 7fbc19600640 1 --2- 172.21.15.90:0/267511199 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbc140764c0 0x7fbc140768e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:08.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:08.202+0000 7fbc19600640 1 -- 172.21.15.90:0/267511199 >> 172.21.15.90:0/267511199 conn(0x7fbc140ff2f0 msgr2=0x7fbc14101710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:34:08.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:08.202+0000 7fbc19600640 1 -- 172.21.15.90:0/267511199 shutdown_connections 2024-04-02T14:34:08.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:08.202+0000 7fbc19600640 1 -- 172.21.15.90:0/267511199 wait complete. 2024-04-02T14:34:08.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:08.203+0000 7fbc19600640 1 Processor -- start 2024-04-02T14:34:08.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:08.203+0000 7fbc19600640 1 -- start start 2024-04-02T14:34:08.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:08.204+0000 7fbc19600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbc140764c0 0x7fbc1419e420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:08.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:08.204+0000 7fbc19600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbc14077700 0x7fbc1419e960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:08.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:08.204+0000 7fbc19600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbc140780a0 0x7fbc141a59d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:08.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:08.204+0000 7fbc19600640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fbc14113dd0 con 0x7fbc14077700 2024-04-02T14:34:08.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:08.204+0000 7fbc19600640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fbc14113c50 con 0x7fbc140764c0 2024-04-02T14:34:08.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:08.204+0000 7fbc19600640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fbc14113f50 con 0x7fbc140780a0 2024-04-02T14:34:08.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:08.204+0000 7fbc12000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbc14077700 0x7fbc1419e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:08.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:08.204+0000 7fbc13400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbc140780a0 0x7fbc141a59d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:08.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:08.204+0000 7fbc13400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbc140780a0 0x7fbc141a59d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:39384/0 (socket says 172.21.15.90:39384) 2024-04-02T14:34:08.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:08.204+0000 7fbc13400640 1 -- 172.21.15.90:0/2052069513 learned_addr learned my addr 172.21.15.90:0/2052069513 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:34:08.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:08.204+0000 7fbc12000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbc14077700 0x7fbc1419e960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:52218/0 (socket says 172.21.15.90:52218) 2024-04-02T14:34:08.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:08.204+0000 7fbc12a00640 1 --2- 172.21.15.90:0/2052069513 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbc140764c0 0x7fbc1419e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:08.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:08.205+0000 7fbc12000640 1 -- 172.21.15.90:0/2052069513 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbc140780a0 msgr2=0x7fbc141a59d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:08.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:08.205+0000 7fbc12000640 1 --2- 172.21.15.90:0/2052069513 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbc140780a0 0x7fbc141a59d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:08.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:08.205+0000 7fbc12000640 1 -- 172.21.15.90:0/2052069513 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbc140764c0 msgr2=0x7fbc1419e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:08.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:08.205+0000 7fbc12000640 1 --2- 172.21.15.90:0/2052069513 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbc140764c0 0x7fbc1419e420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:08.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:08.205+0000 7fbc12000640 1 -- 172.21.15.90:0/2052069513 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc141a6040 con 0x7fbc14077700 2024-04-02T14:34:08.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:08.205+0000 7fbc13400640 1 --2- 172.21.15.90:0/2052069513 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbc140780a0 0x7fbc141a59d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:34:08.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:08.205+0000 7fbc12a00640 1 --2- 172.21.15.90:0/2052069513 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbc140764c0 0x7fbc1419e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:34:08.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:08.205+0000 7fbc12000640 1 --2- 172.21.15.90:0/2052069513 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbc14077700 0x7fbc1419e960 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fbc0002c0b0 tx=0x7fbc00032430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:34:08.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:08.205+0000 7fbc0b400640 1 -- 172.21.15.90:0/2052069513 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc00041020 con 0x7fbc14077700 2024-04-02T14:34:08.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:08.205+0000 7fbc0b400640 1 -- 172.21.15.90:0/2052069513 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbc0002c9f0 con 0x7fbc14077700 2024-04-02T14:34:08.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:08.205+0000 7fbc19600640 1 -- 172.21.15.90:0/2052069513 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc141a62d0 con 0x7fbc14077700 2024-04-02T14:34:08.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:08.205+0000 7fbc0b400640 1 -- 172.21.15.90:0/2052069513 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc00039040 con 0x7fbc14077700 2024-04-02T14:34:08.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:08.206+0000 7fbc19600640 1 -- 172.21.15.90:0/2052069513 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc141a6790 con 0x7fbc14077700 2024-04-02T14:34:08.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:08.207+0000 7fbc19600640 1 -- 172.21.15.90:0/2052069513 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbc14110840 con 0x7fbc14077700 2024-04-02T14:34:08.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:08.209+0000 7fbc0b400640 1 -- 172.21.15.90:0/2052069513 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fbc00044020 con 0x7fbc14077700 2024-04-02T14:34:08.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:08.210+0000 7fbc0b400640 1 --2- 172.21.15.90:0/2052069513 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fbbf80a3bb0 0x7fbbf80a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:08.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:08.211+0000 7fbc0b400640 1 -- 172.21.15.90:0/2052069513 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8248+0+0 (secure 0 0 0) 0x7fbc000f1c00 con 0x7fbc14077700 2024-04-02T14:34:08.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:08.211+0000 7fbc12a00640 1 --2- 172.21.15.90:0/2052069513 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fbbf80a3bb0 0x7fbbf80a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:08.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:08.211+0000 7fbc12a00640 1 --2- 172.21.15.90:0/2052069513 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fbbf80a3bb0 0x7fbbf80a6070 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fbc04009fc0 tx=0x7fbc04005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:34:08.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:08.215+0000 7fbc0b400640 1 -- 172.21.15.90:0/2052069513 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7fbc000f8080 con 0x7fbc14077700 2024-04-02T14:34:08.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:34:07 smithi092 ceph-mon[36497]: pgmap v366: 97 pgs: 97 active+clean; 644 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-02T14:34:08.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:34:07 smithi092 ceph-mon[36497]: from='client.? 172.21.15.90:0/1241613446' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-02T14:34:08.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:34:07 smithi092 ceph-mon[36497]: from='client.? 172.21.15.90:0/362986111' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-02T14:34:08.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:08.416+0000 7fbc19600640 1 -- 172.21.15.90:0/2052069513 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fbc14062700 con 0x7fbc14077700 2024-04-02T14:34:08.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:08.419+0000 7fbc0b400640 1 -- 172.21.15.90:0/2052069513 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+22646 (secure 0 0 0) 0x7fbc0004b020 con 0x7fbc14077700 2024-04-02T14:34:08.420 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-02T14:34:08.421 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":90,"fsid":"6449448a-f0fc-11ee-b647-cb9ed24678a4","created":"2024-04-02T14:23:16.530201+0000","modified":"2024-04-02T14:32:35.716929+0000","last_up_change":"2024-04-02T14:31:23.819732+0000","last_in_change":"2024-04-02T14:31:02.824284+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T14:27:16.758622+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-02T14:32:20.016331+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-02T14:32:21.294367+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a1a6a856-ab44-44bd-bc5e-45c718a70cd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":3635087609},{"type":"v1","addr":"172.21.15.90:6803","nonce":3635087609}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":3635087609},{"type":"v1","addr":"172.21.15.90:6805","nonce":3635087609}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":3635087609},{"type":"v1","addr":"172.21.15.90:6809","nonce":3635087609}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":3635087609},{"type":"v1","addr":"172.21.15.90:6807","nonce":3635087609}]},"public_addr":"172.21.15.90:6803/3635087609","cluster_addr":"172.21.15.90:6805/3635087609","heartbeat_back_addr":"172.21.15.90:6809/3635087609","heartbeat_front_addr":"172.21.15.90:6807/3635087609","state":["exists","up"]},{"osd":1,"uuid":"124139cd-5793-463e-9607-9d9c10164183","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":1994536059},{"type":"v1","addr":"172.21.15.92:6801","nonce":1994536059}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":1994536059},{"type":"v1","addr":"172.21.15.92:6803","nonce":1994536059}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":1994536059},{"type":"v1","addr":"172.21.15.92:6807","nonce":1994536059}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":1994536059},{"type":"v1","addr":"172.21.15.92:6805","nonce":1994536059}]},"public_addr":"172.21.15.92:6801/1994536059","cluster_addr":"172.21.15.92:6803/1994536059","heartbeat_back_addr":"172.21.15.92:6807/1994536059","heartbeat_front_addr":"172.21.15.92:6805/1994536059","state":["exists","up"]},{"osd":2,"uuid":"a61fbef2-9dca-4875-a814-020fff0db0e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":1358472800},{"type":"v1","addr":"172.21.15.177:6801","nonce":1358472800}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":1358472800},{"type":"v1","addr":"172.21.15.177:6803","nonce":1358472800}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":1358472800},{"type":"v1","addr":"172.21.15.177:6807","nonce":1358472800}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":1358472800},{"type":"v1","addr":"172.21.15.177:6805","nonce":1358472800}]},"public_addr":"172.21.15.177:6801/1358472800","cluster_addr":"172.21.15.177:6803/1358472800","heartbeat_back_addr":"172.21.15.177:6807/1358472800","heartbeat_front_addr":"172.21.15.177:6805/1358472800","state":["exists","up"]},{"osd":3,"uuid":"b175a2f4-7859-4eb0-bc1f-fba8c2b92a2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":3114811216},{"type":"v1","addr":"172.21.15.90:6811","nonce":3114811216}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":3114811216},{"type":"v1","addr":"172.21.15.90:6813","nonce":3114811216}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":3114811216},{"type":"v1","addr":"172.21.15.90:6817","nonce":3114811216}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":3114811216},{"type":"v1","addr":"172.21.15.90:6815","nonce":3114811216}]},"public_addr":"172.21.15.90:6811/3114811216","cluster_addr":"172.21.15.90:6813/3114811216","heartbeat_back_addr":"172.21.15.90:6817/3114811216","heartbeat_front_addr":"172.21.15.90:6815/3114811216","state":["exists","up"]},{"osd":4,"uuid":"38cc68ea-c109-4d8d-b77d-1c6f9b56c875","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":2987208879},{"type":"v1","addr":"172.21.15.92:6809","nonce":2987208879}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6810","nonce":2987208879},{"type":"v1","addr":"172.21.15.92:6811","nonce":2987208879}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6814","nonce":2987208879},{"type":"v1","addr":"172.21.15.92:6815","nonce":2987208879}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":2987208879},{"type":"v1","addr":"172.21.15.92:6813","nonce":2987208879}]},"public_addr":"172.21.15.92:6809/2987208879","cluster_addr":"172.21.15.92:6811/2987208879","heartbeat_back_addr":"172.21.15.92:6815/2987208879","heartbeat_front_addr":"172.21.15.92:6813/2987208879","state":["exists","up"]},{"osd":5,"uuid":"2347232e-ac91-466f-8ac9-496f397679ba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":3486820237},{"type":"v1","addr":"172.21.15.177:6809","nonce":3486820237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":3486820237},{"type":"v1","addr":"172.21.15.177:6811","nonce":3486820237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":3486820237},{"type":"v1","addr":"172.21.15.177:6815","nonce":3486820237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":3486820237},{"type":"v1","addr":"172.21.15.177:6813","nonce":3486820237}]},"public_addr":"172.21.15.177:6809/3486820237","cluster_addr":"172.21.15.177:6811/3486820237","heartbeat_back_addr":"172.21.15.177:6815/3486820237","heartbeat_front_addr":"172.21.15.177:6813/3486820237","state":["exists","up"]},{"osd":6,"uuid":"aa72392f-0498-44db-a8d2-eccc89a64871","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":2579716443},{"type":"v1","addr":"172.21.15.90:6819","nonce":2579716443}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6820","nonce":2579716443},{"type":"v1","addr":"172.21.15.90:6821","nonce":2579716443}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":2579716443},{"type":"v1","addr":"172.21.15.90:6825","nonce":2579716443}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2579716443},{"type":"v1","addr":"172.21.15.90:6823","nonce":2579716443}]},"public_addr":"172.21.15.90:6819/2579716443","cluster_addr":"172.21.15.90:6821/2579716443","heartbeat_back_addr":"172.21.15.90:6825/2579716443","heartbeat_front_addr":"172.21.15.90:6823/2579716443","state":["exists","up"]},{"osd":7,"uuid":"931bd105-bc14-4a1e-bfe4-c7ad58fa4b73","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":3371649139},{"type":"v1","addr":"172.21.15.92:6817","nonce":3371649139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":3371649139},{"type":"v1","addr":"172.21.15.92:6819","nonce":3371649139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":3371649139},{"type":"v1","addr":"172.21.15.92:6823","nonce":3371649139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6820","nonce":3371649139},{"type":"v1","addr":"172.21.15.92:6821","nonce":3371649139}]},"public_addr":"172.21.15.92:6817/3371649139","cluster_addr":"172.21.15.92:6819/3371649139","heartbeat_back_addr":"172.21.15.92:6823/3371649139","heartbeat_front_addr":"172.21.15.92:6821/3371649139","state":["exists","up"]},{"osd":8,"uuid":"82624ec2-b382-4886-9f40-7f5f376de3ce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":1459053034},{"type":"v1","addr":"172.21.15.177:6817","nonce":1459053034}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":1459053034},{"type":"v1","addr":"172.21.15.177:6819","nonce":1459053034}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":1459053034},{"type":"v1","addr":"172.21.15.177:6823","nonce":1459053034}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":1459053034},{"type":"v1","addr":"172.21.15.177:6821","nonce":1459053034}]},"public_addr":"172.21.15.177:6817/1459053034","cluster_addr":"172.21.15.177:6819/1459053034","heartbeat_back_addr":"172.21.15.177:6823/1459053034","heartbeat_front_addr":"172.21.15.177:6821/1459053034","state":["exists","up"]},{"osd":9,"uuid":"fbe36e48-3468-42b2-8094-e16552f3cb16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":2592604332},{"type":"v1","addr":"172.21.15.90:6827","nonce":2592604332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":2592604332},{"type":"v1","addr":"172.21.15.90:6829","nonce":2592604332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6832","nonce":2592604332},{"type":"v1","addr":"172.21.15.90:6833","nonce":2592604332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":2592604332},{"type":"v1","addr":"172.21.15.90:6831","nonce":2592604332}]},"public_addr":"172.21.15.90:6827/2592604332","cluster_addr":"172.21.15.90:6829/2592604332","heartbeat_back_addr":"172.21.15.90:6833/2592604332","heartbeat_front_addr":"172.21.15.90:6831/2592604332","state":["exists","up"]},{"osd":10,"uuid":"9240df75-e7e2-47f1-8aae-534b70b3c210","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":1594153142},{"type":"v1","addr":"172.21.15.92:6825","nonce":1594153142}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6826","nonce":1594153142},{"type":"v1","addr":"172.21.15.92:6827","nonce":1594153142}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":1594153142},{"type":"v1","addr":"172.21.15.92:6831","nonce":1594153142}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":1594153142},{"type":"v1","addr":"172.21.15.92:6829","nonce":1594153142}]},"public_addr":"172.21.15.92:6825/1594153142","cluster_addr":"172.21.15.92:6827/1594153142","heartbeat_back_addr":"172.21.15.92:6831/1594153142","heartbeat_front_addr":"172.21.15.92:6829/1594153142","state":["exists","up"]},{"osd":11,"uuid":"83cba9fa-b745-4df1-adda-e86837d4b91a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":83,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1622582126},{"type":"v1","addr":"172.21.15.177:6825","nonce":1622582126}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":1622582126},{"type":"v1","addr":"172.21.15.177:6827","nonce":1622582126}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":1622582126},{"type":"v1","addr":"172.21.15.177:6831","nonce":1622582126}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1622582126},{"type":"v1","addr":"172.21.15.177:6829","nonce":1622582126}]},"public_addr":"172.21.15.177:6825/1622582126","cluster_addr":"172.21.15.177:6827/1622582126","heartbeat_back_addr":"172.21.15.177:6831/1622582126","heartbeat_front_addr":"172.21.15.177:6829/1622582126","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:26:24.290365+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:26:49.405988+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:27:13.639196+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:27:45.603746+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:28:11.019126+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:28:35.084163+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:29:08.064626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:29:33.266133+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:29:57.381816+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:30:30.286847+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:30:56.226279+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:31:20.809611+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":7,"to":11}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":8,"to":4}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.11","mappings":[{"from":4,"to":8},{"from":5,"to":9}]},{"pgid":"3.21","mappings":[{"from":5,"to":9}]},{"pgid":"3.23","mappings":[{"from":4,"to":7}]},{"pgid":"3.3a","mappings":[{"from":4,"to":6}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.90:0/314641601":"2024-04-03T14:24:34.595561+0000","172.21.15.90:0/1400895868":"2024-04-03T14:24:34.595561+0000","172.21.15.90:6801/129436436":"2024-04-03T14:24:34.595561+0000","172.21.15.90:6801/2372398598":"2024-04-03T14:24:00.362147+0000","172.21.15.90:0/278569338":"2024-04-03T14:24:34.595561+0000","172.21.15.90:0/2321464594":"2024-04-03T14:24:00.362147+0000","172.21.15.90:6800/2372398598":"2024-04-03T14:24:00.362147+0000","172.21.15.90:0/2962656672":"2024-04-03T14:24:00.362147+0000","172.21.15.90:0/126693610":"2024-04-03T14:24:00.362147+0000","172.21.15.90:6800/129436436":"2024-04-03T14:24:34.595561+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T14:34:08.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:08.422+0000 7fbc19600640 1 -- 172.21.15.90:0/2052069513 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fbbf80a3bb0 msgr2=0x7fbbf80a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:08.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:08.422+0000 7fbc19600640 1 --2- 172.21.15.90:0/2052069513 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fbbf80a3bb0 0x7fbbf80a6070 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fbc04009fc0 tx=0x7fbc04005f90 comp rx=0 tx=0).stop 2024-04-02T14:34:08.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:08.422+0000 7fbc19600640 1 -- 172.21.15.90:0/2052069513 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbc14077700 msgr2=0x7fbc1419e960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:08.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:08.422+0000 7fbc19600640 1 --2- 172.21.15.90:0/2052069513 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbc14077700 0x7fbc1419e960 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fbc0002c0b0 tx=0x7fbc00032430 comp rx=0 tx=0).stop 2024-04-02T14:34:08.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:08.422+0000 7fbc19600640 1 -- 172.21.15.90:0/2052069513 shutdown_connections 2024-04-02T14:34:08.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:08.422+0000 7fbc19600640 1 --2- 172.21.15.90:0/2052069513 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fbbf80a3bb0 0x7fbbf80a6070 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:08.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:08.422+0000 7fbc19600640 1 --2- 172.21.15.90:0/2052069513 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbc140780a0 0x7fbc141a59d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:08.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:08.422+0000 7fbc19600640 1 --2- 172.21.15.90:0/2052069513 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbc14077700 0x7fbc1419e960 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:08.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:08.422+0000 7fbc19600640 1 --2- 172.21.15.90:0/2052069513 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbc140764c0 0x7fbc1419e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:08.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:08.422+0000 7fbc19600640 1 -- 172.21.15.90:0/2052069513 >> 172.21.15.90:0/2052069513 conn(0x7fbc140ff2f0 msgr2=0x7fbc141009f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:34:08.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:08.422+0000 7fbc19600640 1 -- 172.21.15.90:0/2052069513 shutdown_connections 2024-04-02T14:34:08.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:08.422+0000 7fbc19600640 1 -- 172.21.15.90:0/2052069513 wait complete. 2024-04-02T14:34:08.442 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T14:34:08.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:08.647+0000 7f8d33a00640 1 -- 172.21.15.90:0/3397022083 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8d2c104f30 msgr2=0x7f8d2c1053b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:08.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:08.647+0000 7f8d33a00640 1 --2- 172.21.15.90:0/3397022083 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8d2c104f30 0x7f8d2c1053b0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f8d18009900 tx=0x7f8d1802bc30 comp rx=0 tx=0).stop 2024-04-02T14:34:08.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:08.647+0000 7f8d33a00640 1 -- 172.21.15.90:0/3397022083 shutdown_connections 2024-04-02T14:34:08.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:08.647+0000 7f8d33a00640 1 --2- 172.21.15.90:0/3397022083 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8d2c1058f0 0x7f8d2c10c180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:08.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:08.647+0000 7f8d33a00640 1 --2- 172.21.15.90:0/3397022083 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8d2c104f30 0x7f8d2c1053b0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:08.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:08.647+0000 7f8d33a00640 1 --2- 172.21.15.90:0/3397022083 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8d2c103d40 0x7f8d2c104160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:08.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:08.647+0000 7f8d33a00640 1 -- 172.21.15.90:0/3397022083 >> 172.21.15.90:0/3397022083 conn(0x7f8d2c0ff2b0 msgr2=0x7f8d2c1016f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:34:08.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:08.647+0000 7f8d33a00640 1 -- 172.21.15.90:0/3397022083 shutdown_connections 2024-04-02T14:34:08.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:08.647+0000 7f8d33a00640 1 -- 172.21.15.90:0/3397022083 wait complete. 2024-04-02T14:34:08.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:08.648+0000 7f8d33a00640 1 Processor -- start 2024-04-02T14:34:08.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:08.648+0000 7f8d33a00640 1 -- start start 2024-04-02T14:34:08.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:08.648+0000 7f8d33a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8d2c103d40 0x7f8d2c19a000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:08.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:08.648+0000 7f8d33a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8d2c104f30 0x7f8d2c19a540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:08.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:08.648+0000 7f8d33a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8d2c1058f0 0x7f8d2c1a15b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:08.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:08.648+0000 7f8d33a00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f8d2c10fa10 con 0x7f8d2c103d40 2024-04-02T14:34:08.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:08.648+0000 7f8d33a00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f8d2c10f890 con 0x7f8d2c1058f0 2024-04-02T14:34:08.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:08.648+0000 7f8d33a00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f8d2c10fb90 con 0x7f8d2c104f30 2024-04-02T14:34:08.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:08.649+0000 7f8d31000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8d2c103d40 0x7f8d2c19a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:08.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:08.649+0000 7f8d31000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8d2c103d40 0x7f8d2c19a000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:52236/0 (socket says 172.21.15.90:52236) 2024-04-02T14:34:08.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:08.649+0000 7f8d31000640 1 -- 172.21.15.90:0/559140841 learned_addr learned my addr 172.21.15.90:0/559140841 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:34:08.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:08.649+0000 7f8d2be00640 1 --2- 172.21.15.90:0/559140841 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8d2c104f30 0x7f8d2c19a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:08.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:08.649+0000 7f8d31a00640 1 --2- 172.21.15.90:0/559140841 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8d2c1058f0 0x7f8d2c1a15b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:08.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:08.649+0000 7f8d31000640 1 -- 172.21.15.90:0/559140841 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8d2c104f30 msgr2=0x7f8d2c19a540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:08.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:08.649+0000 7f8d31000640 1 --2- 172.21.15.90:0/559140841 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8d2c104f30 0x7f8d2c19a540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:08.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:08.649+0000 7f8d31000640 1 -- 172.21.15.90:0/559140841 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8d2c1058f0 msgr2=0x7f8d2c1a15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:08.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:08.649+0000 7f8d31000640 1 --2- 172.21.15.90:0/559140841 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8d2c1058f0 0x7f8d2c1a15b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:08.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:08.649+0000 7f8d31000640 1 -- 172.21.15.90:0/559140841 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d2c1a1c20 con 0x7f8d2c103d40 2024-04-02T14:34:08.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:08.649+0000 7f8d31000640 1 --2- 172.21.15.90:0/559140841 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8d2c103d40 0x7f8d2c19a000 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f8d2000e9c0 tx=0x7f8d2000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:34:08.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:08.650+0000 7f8d29600640 1 -- 172.21.15.90:0/559140841 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d2000cd80 con 0x7f8d2c103d40 2024-04-02T14:34:08.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:08.650+0000 7f8d29600640 1 -- 172.21.15.90:0/559140841 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8d200035f0 con 0x7f8d2c103d40 2024-04-02T14:34:08.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:08.650+0000 7f8d33a00640 1 -- 172.21.15.90:0/559140841 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d2c1a1f10 con 0x7f8d2c103d40 2024-04-02T14:34:08.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:08.650+0000 7f8d29600640 1 -- 172.21.15.90:0/559140841 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d20010690 con 0x7f8d2c103d40 2024-04-02T14:34:08.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:08.650+0000 7f8d33a00640 1 -- 172.21.15.90:0/559140841 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d2c1a2380 con 0x7f8d2c103d40 2024-04-02T14:34:08.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:08.651+0000 7f8d33a00640 1 -- 172.21.15.90:0/559140841 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8d2c10b000 con 0x7f8d2c103d40 2024-04-02T14:34:08.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:08.653+0000 7f8d29600640 1 -- 172.21.15.90:0/559140841 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f8d200037a0 con 0x7f8d2c103d40 2024-04-02T14:34:08.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:08.655+0000 7f8d29600640 1 --2- 172.21.15.90:0/559140841 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f8d140a3c30 0x7f8d140a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:08.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:08.655+0000 7f8d29600640 1 -- 172.21.15.90:0/559140841 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8248+0+0 (secure 0 0 0) 0x7f8d200d0760 con 0x7f8d2c103d40 2024-04-02T14:34:08.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:08.655+0000 7f8d2be00640 1 --2- 172.21.15.90:0/559140841 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f8d140a3c30 0x7f8d140a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:08.657 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:08.656+0000 7f8d2be00640 1 --2- 172.21.15.90:0/559140841 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f8d140a3c30 0x7f8d140a60f0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f8d1802c240 tx=0x7f8d1802df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:34:08.660 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:08.659+0000 7f8d29600640 1 -- 172.21.15.90:0/559140841 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7f8d2009d0d0 con 0x7f8d2c103d40 2024-04-02T14:34:08.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:08.899+0000 7f8d33a00640 1 -- 172.21.15.90:0/559140841 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8d2c02d370 con 0x7f8d2c103d40 2024-04-02T14:34:08.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:08.900+0000 7f8d29600640 1 -- 172.21.15.90:0/559140841 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11927 (secure 0 0 0) 0x7f8d2c02d370 con 0x7f8d2c103d40 2024-04-02T14:34:08.903 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-02T14:34:08.904 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14412,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6837/2852160309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6836","nonce":2852160309},{"type":"v1","addr":"172.21.15.90:6837","nonce":2852160309}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14478,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6841/1919376858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6840","nonce":1919376858},{"type":"v1","addr":"172.21.15.90:6841","nonce":1919376858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":26,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-02T14:32:22.319083+0000","modified":"2024-04-02T14:32:29.812071+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24590,"mds_1":24560,"mds_2":24547,"mds_3":24539,"mds_4":24523},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24454":{"gid":24454,"name":"l","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6833/1564319431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":1564319431},{"type":"v1","addr":"172.21.15.177:6833","nonce":1564319431}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.92:6833/1570337372","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":1570337372},{"type":"v1","addr":"172.21.15.92:6833","nonce":1570337372}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"j","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.90:6835/1277124891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6834","nonce":1277124891},{"type":"v1","addr":"172.21.15.90:6835","nonce":1277124891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.92:6835/860721164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":860721164},{"type":"v1","addr":"172.21.15.92:6835","nonce":860721164}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24511":{"gid":24511,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6835/3177244988","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":3177244988},{"type":"v1","addr":"172.21.15.177:6835","nonce":3177244988}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"c","rank":4,"incarnation":25,"state":"up:active","state_seq":9,"addr":"172.21.15.177:6837/4223414858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":4223414858},{"type":"v1","addr":"172.21.15.177:6837","nonce":4223414858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24539":{"gid":24539,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":8,"addr":"172.21.15.92:6837/1270635939","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":1270635939},{"type":"v1","addr":"172.21.15.92:6837","nonce":1270635939}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24547":{"gid":24547,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":7,"addr":"172.21.15.177:6839/664895663","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":664895663},{"type":"v1","addr":"172.21.15.177:6839","nonce":664895663}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"g","rank":1,"incarnation":19,"state":"up:active","state_seq":5,"addr":"172.21.15.90:6839/1498915870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6838","nonce":1498915870},{"type":"v1","addr":"172.21.15.90:6839","nonce":1498915870}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24590":{"gid":24590,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.92:6839/1054888219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":1054888219},{"type":"v1","addr":"172.21.15.92:6839","nonce":1054888219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24590,"qdb_cluster":[24539,24560,24523,24590,24547]},"id":1}]} 2024-04-02T14:34:08.904 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 26 2024-04-02T14:34:08.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:08.902+0000 7f8d33a00640 1 -- 172.21.15.90:0/559140841 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f8d140a3c30 msgr2=0x7f8d140a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:08.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:08.902+0000 7f8d33a00640 1 --2- 172.21.15.90:0/559140841 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f8d140a3c30 0x7f8d140a60f0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f8d1802c240 tx=0x7f8d1802df90 comp rx=0 tx=0).stop 2024-04-02T14:34:08.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:08.902+0000 7f8d33a00640 1 -- 172.21.15.90:0/559140841 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8d2c103d40 msgr2=0x7f8d2c19a000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:08.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:08.902+0000 7f8d33a00640 1 --2- 172.21.15.90:0/559140841 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8d2c103d40 0x7f8d2c19a000 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f8d2000e9c0 tx=0x7f8d2000ee80 comp rx=0 tx=0).stop 2024-04-02T14:34:08.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:08.902+0000 7f8d33a00640 1 -- 172.21.15.90:0/559140841 shutdown_connections 2024-04-02T14:34:08.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:08.902+0000 7f8d33a00640 1 --2- 172.21.15.90:0/559140841 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f8d140a3c30 0x7f8d140a60f0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:08.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:08.902+0000 7f8d33a00640 1 --2- 172.21.15.90:0/559140841 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8d2c1058f0 0x7f8d2c1a15b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:08.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:08.902+0000 7f8d33a00640 1 --2- 172.21.15.90:0/559140841 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8d2c104f30 0x7f8d2c19a540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:08.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:08.902+0000 7f8d33a00640 1 --2- 172.21.15.90:0/559140841 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8d2c103d40 0x7f8d2c19a000 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:08.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:08.902+0000 7f8d33a00640 1 -- 172.21.15.90:0/559140841 >> 172.21.15.90:0/559140841 conn(0x7f8d2c0ff2b0 msgr2=0x7f8d2c1081d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:34:08.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:08.902+0000 7f8d33a00640 1 -- 172.21.15.90:0/559140841 shutdown_connections 2024-04-02T14:34:08.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:08.902+0000 7f8d33a00640 1 -- 172.21.15.90:0/559140841 wait complete. 2024-04-02T14:34:08.926 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-02T14:32:22.319083+0000', 'modified': '2024-04-02T14:32:29.812071+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 24590, 'mds_1': 24560, 'mds_2': 24547, 'mds_3': 24539, 'mds_4': 24523}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24454': {'gid': 24454, 'name': 'l', 'rank': 4, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.177:6833/1564319431', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6832', 'nonce': 1564319431}, {'type': 'v1', 'addr': '172.21.15.177:6833', 'nonce': 1564319431}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24461': {'gid': 24461, 'name': 'k', 'rank': 3, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.92:6833/1570337372', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6832', 'nonce': 1570337372}, {'type': 'v1', 'addr': '172.21.15.92:6833', 'nonce': 1570337372}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24478': {'gid': 24478, 'name': 'j', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.90:6835/1277124891', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6834', 'nonce': 1277124891}, {'type': 'v1', 'addr': '172.21.15.90:6835', 'nonce': 1277124891}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24503': {'gid': 24503, 'name': 'e', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.92:6835/860721164', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 860721164}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 860721164}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24511': {'gid': 24511, 'name': 'f', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.177:6835/3177244988', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6834', 'nonce': 3177244988}, {'type': 'v1', 'addr': '172.21.15.177:6835', 'nonce': 3177244988}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24523': {'gid': 24523, 'name': 'c', 'rank': 4, 'incarnation': 25, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.177:6837/4223414858', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6836', 'nonce': 4223414858}, {'type': 'v1', 'addr': '172.21.15.177:6837', 'nonce': 4223414858}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24539': {'gid': 24539, 'name': 'h', 'rank': 3, 'incarnation': 23, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.92:6837/1270635939', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6836', 'nonce': 1270635939}, {'type': 'v1', 'addr': '172.21.15.92:6837', 'nonce': 1270635939}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24547': {'gid': 24547, 'name': 'i', 'rank': 2, 'incarnation': 21, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.177:6839/664895663', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6838', 'nonce': 664895663}, {'type': 'v1', 'addr': '172.21.15.177:6839', 'nonce': 664895663}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24560': {'gid': 24560, 'name': 'g', 'rank': 1, 'incarnation': 19, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.90:6839/1498915870', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6838', 'nonce': 1498915870}, {'type': 'v1', 'addr': '172.21.15.90:6839', 'nonce': 1498915870}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24590': {'gid': 24590, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.92:6839/1054888219', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 1054888219}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 1054888219}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24590, 'qdb_cluster': [24539, 24560, 24523, 24590, 24547]} 2024-04-02T14:34:08.926 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 5/5 2024-04-02T14:34:08.926 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-04-02T14:34:09.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.144+0000 7f4cab600640 1 -- 172.21.15.90:0/2781524028 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4ca4074560 msgr2=0x7f4ca40749e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:09.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.144+0000 7f4cab600640 1 --2- 172.21.15.90:0/2781524028 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4ca4074560 0x7f4ca40749e0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f4c8c009900 tx=0x7f4c8c02bc30 comp rx=0 tx=0).stop 2024-04-02T14:34:09.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.144+0000 7f4cab600640 1 -- 172.21.15.90:0/2781524028 shutdown_connections 2024-04-02T14:34:09.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.144+0000 7f4cab600640 1 --2- 172.21.15.90:0/2781524028 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4ca41099d0 0x7f4ca410be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:09.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.144+0000 7f4cab600640 1 --2- 172.21.15.90:0/2781524028 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4ca4074560 0x7f4ca40749e0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:09.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.144+0000 7f4cab600640 1 --2- 172.21.15.90:0/2781524028 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4ca4075b40 0x7f4ca4073f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:09.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.144+0000 7f4cab600640 1 -- 172.21.15.90:0/2781524028 >> 172.21.15.90:0/2781524028 conn(0x7f4ca40fd0a0 msgr2=0x7f4ca40ff4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:34:09.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.144+0000 7f4cab600640 1 -- 172.21.15.90:0/2781524028 shutdown_connections 2024-04-02T14:34:09.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.144+0000 7f4cab600640 1 -- 172.21.15.90:0/2781524028 wait complete. 2024-04-02T14:34:09.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.145+0000 7f4cab600640 1 Processor -- start 2024-04-02T14:34:09.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.145+0000 7f4cab600640 1 -- start start 2024-04-02T14:34:09.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.146+0000 7f4cab600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4ca4074560 0x7f4ca419a030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:09.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.146+0000 7f4cab600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4ca4075b40 0x7f4ca419a570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:09.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.146+0000 7f4cab600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4ca41099d0 0x7f4ca41a15e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:09.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.146+0000 7f4cab600640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f4ca410f950 con 0x7f4ca41099d0 2024-04-02T14:34:09.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.146+0000 7f4cab600640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f4ca410f7d0 con 0x7f4ca4075b40 2024-04-02T14:34:09.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.146+0000 7f4cab600640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f4ca410fad0 con 0x7f4ca4074560 2024-04-02T14:34:09.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.146+0000 7f4ca9600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4ca41099d0 0x7f4ca41a15e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:09.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.146+0000 7f4c9be00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4ca4075b40 0x7f4ca419a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:09.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.146+0000 7f4ca9600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4ca41099d0 0x7f4ca41a15e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:52246/0 (socket says 172.21.15.90:52246) 2024-04-02T14:34:09.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.146+0000 7f4ca9600640 1 -- 172.21.15.90:0/173794501 learned_addr learned my addr 172.21.15.90:0/173794501 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:34:09.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.146+0000 7f4ca8c00640 1 --2- 172.21.15.90:0/173794501 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4ca4074560 0x7f4ca419a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:09.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.146+0000 7f4ca9600640 1 -- 172.21.15.90:0/173794501 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4ca4074560 msgr2=0x7f4ca419a030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:09.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.146+0000 7f4ca9600640 1 --2- 172.21.15.90:0/173794501 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4ca4074560 0x7f4ca419a030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:09.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.146+0000 7f4ca9600640 1 -- 172.21.15.90:0/173794501 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4ca4075b40 msgr2=0x7f4ca419a570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:09.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.146+0000 7f4ca9600640 1 --2- 172.21.15.90:0/173794501 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4ca4075b40 0x7f4ca419a570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:09.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.146+0000 7f4ca9600640 1 -- 172.21.15.90:0/173794501 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ca41a1c50 con 0x7f4ca41099d0 2024-04-02T14:34:09.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.147+0000 7f4ca9600640 1 --2- 172.21.15.90:0/173794501 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4ca41099d0 0x7f4ca41a15e0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f4c9c00eb10 tx=0x7f4c9c00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:34:09.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.147+0000 7f4ca2000640 1 -- 172.21.15.90:0/173794501 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c9c004b20 con 0x7f4ca41099d0 2024-04-02T14:34:09.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.147+0000 7f4ca2000640 1 -- 172.21.15.90:0/173794501 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4c9c004cc0 con 0x7f4ca41099d0 2024-04-02T14:34:09.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.147+0000 7f4cab600640 1 -- 172.21.15.90:0/173794501 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ca41a1f40 con 0x7f4ca41099d0 2024-04-02T14:34:09.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.147+0000 7f4ca2000640 1 -- 172.21.15.90:0/173794501 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c9c010760 con 0x7f4ca41099d0 2024-04-02T14:34:09.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.147+0000 7f4cab600640 1 -- 172.21.15.90:0/173794501 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ca41a23b0 con 0x7f4ca41099d0 2024-04-02T14:34:09.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.150+0000 7f4ca2000640 1 -- 172.21.15.90:0/173794501 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f4c9c005020 con 0x7f4ca41099d0 2024-04-02T14:34:09.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.150+0000 7f4cae27b740 1 Processor -- start 2024-04-02T14:34:09.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.150+0000 7f4cae27b740 1 -- start start 2024-04-02T14:34:09.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.151+0000 7f4cae27b740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f0a228f8f0 0x55f0a228fd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:09.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.151+0000 7f4cae27b740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f0a22902e0 0x55f0a2294de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:09.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.151+0000 7f4cae27b740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f0a2295450 0x55f0a22978e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:09.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.151+0000 7f4cae27b740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55f0a21f5060 con 0x55f0a22902e0 2024-04-02T14:34:09.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.151+0000 7f4cae27b740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55f0a219d560 con 0x55f0a228f8f0 2024-04-02T14:34:09.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.151+0000 7f4cae27b740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55f0a21f1010 con 0x55f0a2295450 2024-04-02T14:34:09.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.151+0000 7f4ca8c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f0a22902e0 0x55f0a2294de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:09.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.151+0000 7f4c9be00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f0a2295450 0x55f0a22978e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:09.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.151+0000 7f4c9be00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f0a2295450 0x55f0a22978e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:39416/0 (socket says 172.21.15.90:39416) 2024-04-02T14:34:09.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.151+0000 7f4c9be00640 1 -- 172.21.15.90:0/3706357827 learned_addr learned my addr 172.21.15.90:0/3706357827 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:34:09.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.151+0000 7f4c9be00640 1 --2- 172.21.15.90:0/3706357827 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f0a228f8f0 0x55f0a228fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:09.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.152+0000 7f4c9be00640 1 -- 172.21.15.90:0/3706357827 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f0a228f8f0 msgr2=0x55f0a228fd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:34:09.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.152+0000 7f4c9be00640 1 --2- 172.21.15.90:0/3706357827 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f0a228f8f0 0x55f0a228fd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:09.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.152+0000 7f4c9be00640 1 -- 172.21.15.90:0/3706357827 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f0a22902e0 msgr2=0x55f0a2294de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:34:09.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.152+0000 7f4c9be00640 1 --2- 172.21.15.90:0/3706357827 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f0a22902e0 0x55f0a2294de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:09.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.152+0000 7f4c9be00640 1 -- 172.21.15.90:0/3706357827 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f0a21bffd0 con 0x55f0a2295450 2024-04-02T14:34:09.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.152+0000 7f4ca8c00640 1 --2- 172.21.15.90:0/3706357827 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f0a22902e0 0x55f0a2294de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:34:09.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.152+0000 7f4ca2000640 1 --2- 172.21.15.90:0/173794501 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f4c880a3bb0 0x7f4c880a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:09.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.152+0000 7f4c9be00640 1 --2- 172.21.15.90:0/3706357827 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f0a2295450 0x55f0a22978e0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x55f0a2290b40 tx=0x7f4c8c005b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7cd22d7ba2096220 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:34:09.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.152+0000 7f4ca2000640 1 -- 172.21.15.90:0/173794501 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8248+0+0 (secure 0 0 0) 0x7f4c9c0d09e0 con 0x7f4ca41099d0 2024-04-02T14:34:09.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.152+0000 7f4ca8c00640 1 --2- 172.21.15.90:0/173794501 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f4c880a3bb0 0x7f4c880a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:09.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.152+0000 7f4c99c00640 1 -- 172.21.15.90:0/3706357827 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c8c004030 con 0x55f0a2295450 2024-04-02T14:34:09.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.152+0000 7f4c99c00640 1 -- 172.21.15.90:0/3706357827 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4c8c0041d0 con 0x55f0a2295450 2024-04-02T14:34:09.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.152+0000 7f4c99c00640 1 -- 172.21.15.90:0/3706357827 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c8c0346a0 con 0x55f0a2295450 2024-04-02T14:34:09.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.153+0000 7f4cae27b740 1 -- 172.21.15.90:0/3706357827 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f0a2295450 msgr2=0x55f0a22978e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:09.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.153+0000 7f4ca8c00640 1 --2- 172.21.15.90:0/173794501 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f4c880a3bb0 0x7f4c880a6070 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x55f0a22906e0 tx=0x7f4c94005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:34:09.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.153+0000 7f4cae27b740 1 --2- 172.21.15.90:0/3706357827 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f0a2295450 0x55f0a22978e0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x55f0a2290b40 tx=0x7f4c8c005b50 comp rx=0 tx=0).stop 2024-04-02T14:34:09.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.153+0000 7f4cae27b740 1 -- 172.21.15.90:0/3706357827 shutdown_connections 2024-04-02T14:34:09.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.153+0000 7f4cae27b740 1 --2- 172.21.15.90:0/3706357827 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f0a2295450 0x55f0a22978e0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:09.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.153+0000 7f4cae27b740 1 --2- 172.21.15.90:0/3706357827 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f0a22902e0 0x55f0a2294de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:09.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.153+0000 7f4cae27b740 1 --2- 172.21.15.90:0/3706357827 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f0a228f8f0 0x55f0a228fd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:09.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.153+0000 7f4cae27b740 1 -- 172.21.15.90:0/3706357827 >> 172.21.15.90:0/3706357827 conn(0x55f0a228ae60 msgr2=0x55f0a228b260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:34:09.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.153+0000 7f4cae27b740 1 -- 172.21.15.90:0/3706357827 shutdown_connections 2024-04-02T14:34:09.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.153+0000 7f4cae27b740 1 -- 172.21.15.90:0/3706357827 wait complete. 2024-04-02T14:34:09.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.154+0000 7f4cae27b740 1 -- start start 2024-04-02T14:34:09.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.154+0000 7f4cae27b740 1 Processor -- start 2024-04-02T14:34:09.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.154+0000 7f4cae27b740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f0a228f8f0 0x55f0a2299e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:09.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.154+0000 7f4cae27b740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f0a22902e0 0x55f0a229a370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:09.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.154+0000 7f4cae27b740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f0a2295450 0x55f0a22a13b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:09.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.154+0000 7f4cae27b740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55f0a22983a0 con 0x55f0a228f8f0 2024-04-02T14:34:09.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.154+0000 7f4cae27b740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55f0a21b0920 con 0x55f0a2295450 2024-04-02T14:34:09.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.154+0000 7f4cae27b740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55f0a2298520 con 0x55f0a22902e0 2024-04-02T14:34:09.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.154+0000 7f4cae27b740 10 client.0 _ll_register_callbacks cb 0x55f0a21b0310 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:34:09.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.155+0000 7f4ca8c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f0a228f8f0 0x55f0a2299e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:09.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.155+0000 7f4ca8c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f0a228f8f0 0x55f0a2299e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:52264/0 (socket says 172.21.15.90:52264) 2024-04-02T14:34:09.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.155+0000 7f4c9be00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f0a22902e0 0x55f0a229a370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:09.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.155+0000 7f4ca8c00640 1 -- 172.21.15.90:0/2041683724 learned_addr learned my addr 172.21.15.90:0/2041683724 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:34:09.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.155+0000 7f4ca9600640 1 --2- 172.21.15.90:0/2041683724 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f0a2295450 0x55f0a22a13b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:09.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.155+0000 7f4ca8c00640 1 -- 172.21.15.90:0/2041683724 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f0a22902e0 msgr2=0x55f0a229a370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:34:09.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.155+0000 7f4ca8c00640 1 --2- 172.21.15.90:0/2041683724 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f0a22902e0 0x55f0a229a370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:09.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.155+0000 7f4ca8c00640 1 -- 172.21.15.90:0/2041683724 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f0a2295450 msgr2=0x55f0a22a13b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:34:09.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.155+0000 7f4ca8c00640 1 --2- 172.21.15.90:0/2041683724 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f0a2295450 0x55f0a22a13b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:09.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.155+0000 7f4ca8c00640 1 -- 172.21.15.90:0/2041683724 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f0a1f092d0 con 0x55f0a228f8f0 2024-04-02T14:34:09.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.155+0000 7f4c9be00640 1 --2- 172.21.15.90:0/2041683724 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f0a22902e0 0x55f0a229a370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:34:09.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.155+0000 7f4ca8c00640 1 --2- 172.21.15.90:0/2041683724 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f0a228f8f0 0x55f0a2299e30 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f4c94009fd0 tx=0x7f4c94004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6462ee27a4b3350 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:34:09.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.155+0000 7f4c92000640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-02T14:34:09.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.156+0000 7f4c92000640 1 -- 172.21.15.90:0/2041683724 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c9401f640 con 0x55f0a228f8f0 2024-04-02T14:34:09.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.156+0000 7f4c92000640 1 -- 172.21.15.90:0/2041683724 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8248+0+0 (secure 0 0 0) 0x7f4c9401f7e0 con 0x55f0a228f8f0 2024-04-02T14:34:09.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.156+0000 7f4cae27b740 1 -- 172.21.15.90:0/2041683724 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4c8c0041d0 con 0x55f0a228f8f0 2024-04-02T14:34:09.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.156+0000 7f4c92000640 1 -- 172.21.15.90:0/2041683724 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f4c9401f9e0 con 0x55f0a228f8f0 2024-04-02T14:34:09.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.156+0000 7f4cae27b740 10 client.14886 fetch_fsmap learned FSMap version 26 2024-04-02T14:34:09.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.156+0000 7f4cae27b740 1 -- 172.21.15.90:0/2041683724 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55f0a1fd36b0 con 0x55f0a228f8f0 2024-04-02T14:34:09.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.156+0000 7f4c92000640 1 -- 172.21.15.90:0/2041683724 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4c8c0041d0 con 0x55f0a228f8f0 2024-04-02T14:34:09.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.156+0000 7f4c92000640 1 -- 172.21.15.90:0/2041683724 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f4c94032050 con 0x55f0a228f8f0 2024-04-02T14:34:09.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.156+0000 7f4cae27b740 10 client.14886 fetch_fsmap finished waiting for FSMap version 26 2024-04-02T14:34:09.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.157+0000 7f4cae27b740 10 client.14886 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-02T14:34:09.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.157+0000 7f4cae27b740 20 client.14886 populate_metadata read hostname 'smithi090' 2024-04-02T14:34:09.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.157+0000 7f4cae27b740 1 --2- 172.21.15.90:0/2041683724 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x55f0a22a6cb0 0x55f0a22a8120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:09.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.157+0000 7f4cae27b740 4 client.14886 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:34:09.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.157+0000 7f4cae27b740 1 -- 172.21.15.90:0/2041683724 --> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f0a22025a0 con 0x55f0a22a6cb0 2024-04-02T14:34:09.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.157+0000 7f4c9be00640 1 --2- 172.21.15.90:0/2041683724 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x55f0a22a6cb0 0x55f0a22a8120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:09.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.157+0000 7f4c9be00640 1 --2- 172.21.15.90:0/2041683724 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x55f0a22a6cb0 0x55f0a22a8120 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=c13cecf488e0f2e1 server_cookie=a63c3ce360947a52 in_seq=0 out_seq=0 2024-04-02T14:34:09.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.157+0000 7f4c92000640 10 client.14886 ms_handle_connect on v2:172.21.15.177:6836/4223414858 2024-04-02T14:34:09.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.158+0000 7f4c92000640 1 -- 172.21.15.90:0/2041683724 <== mds.4 v2:172.21.15.177:6836/4223414858 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f0a22025a0 con 0x55f0a22a6cb0 2024-04-02T14:34:09.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.158+0000 7f4c92000640 10 client.14886 handle_command_reply: tid=0 2024-04-02T14:34:09.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.158+0000 7f4cae27b740 1 client.14886 shutdown 2024-04-02T14:34:09.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.158+0000 7f4cae27b740 1 -- 172.21.15.90:0/2041683724 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f0a228f8f0 msgr2=0x55f0a2299e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:09.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.158+0000 7f4cae27b740 1 --2- 172.21.15.90:0/2041683724 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f0a228f8f0 0x55f0a2299e30 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f4c94009fd0 tx=0x7f4c94004730 comp rx=0 tx=0).stop 2024-04-02T14:34:09.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.159+0000 7f4cae27b740 1 -- 172.21.15.90:0/2041683724 shutdown_connections 2024-04-02T14:34:09.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.159+0000 7f4cae27b740 1 --2- 172.21.15.90:0/2041683724 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f0a2295450 0x55f0a22a13b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:09.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.159+0000 7f4cae27b740 1 --2- 172.21.15.90:0/2041683724 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f0a22902e0 0x55f0a229a370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:09.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.159+0000 7f4cae27b740 1 --2- 172.21.15.90:0/2041683724 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x55f0a22a6cb0 0x55f0a22a8120 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:09.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.159+0000 7f4cae27b740 1 --2- 172.21.15.90:0/2041683724 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f0a228f8f0 0x55f0a2299e30 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:09.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.159+0000 7f4c92000640 0 client.14886 ms_handle_reset on v2:172.21.15.177:6836/4223414858 2024-04-02T14:34:09.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.159+0000 7f4cae27b740 1 -- 172.21.15.90:0/2041683724 >> 172.21.15.90:0/2041683724 conn(0x55f0a228ae60 msgr2=0x55f0a228a080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:34:09.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.159+0000 7f4cae27b740 1 -- 172.21.15.90:0/2041683724 shutdown_connections 2024-04-02T14:34:09.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.159+0000 7f4cae27b740 1 -- 172.21.15.90:0/2041683724 wait complete. 2024-04-02T14:34:09.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.159+0000 7f4cae27b740 20 client.14886 trim_cache size 0 max 16384 2024-04-02T14:34:09.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.178+0000 7f4cae27b740 1 Processor -- start 2024-04-02T14:34:09.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.178+0000 7f4cae27b740 1 -- start start 2024-04-02T14:34:09.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.179+0000 7f4cae27b740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f0a228f8f0 0x55f0a228b240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:09.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.179+0000 7f4cae27b740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f0a22902e0 0x55f0a22a1340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:09.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.179+0000 7f4cae27b740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f0a2295450 0x55f0a22a5a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:09.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.179+0000 7f4cae27b740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55f0a22a6460 con 0x55f0a228f8f0 2024-04-02T14:34:09.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.179+0000 7f4cae27b740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55f0a22a6b30 con 0x55f0a22902e0 2024-04-02T14:34:09.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.179+0000 7f4cae27b740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55f0a22a7200 con 0x55f0a2295450 2024-04-02T14:34:09.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.179+0000 7f4ca8c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f0a228f8f0 0x55f0a228b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:09.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.179+0000 7f4ca8c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f0a228f8f0 0x55f0a228b240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:52270/0 (socket says 172.21.15.90:52270) 2024-04-02T14:34:09.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.179+0000 7f4ca8c00640 1 -- 172.21.15.90:0/4072084398 learned_addr learned my addr 172.21.15.90:0/4072084398 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:34:09.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.179+0000 7f4c9be00640 1 --2- 172.21.15.90:0/4072084398 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f0a22902e0 0x55f0a22a1340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:09.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.179+0000 7f4ca9600640 1 --2- 172.21.15.90:0/4072084398 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f0a2295450 0x55f0a22a5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:09.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.179+0000 7f4ca8c00640 1 -- 172.21.15.90:0/4072084398 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f0a2295450 msgr2=0x55f0a22a5a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:34:09.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.179+0000 7f4ca8c00640 1 --2- 172.21.15.90:0/4072084398 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f0a2295450 0x55f0a22a5a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:09.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.179+0000 7f4ca8c00640 1 -- 172.21.15.90:0/4072084398 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f0a22902e0 msgr2=0x55f0a22a1340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:34:09.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.179+0000 7f4ca8c00640 1 --2- 172.21.15.90:0/4072084398 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f0a22902e0 0x55f0a22a1340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:09.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.179+0000 7f4ca8c00640 1 -- 172.21.15.90:0/4072084398 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f0a228bbd0 con 0x55f0a228f8f0 2024-04-02T14:34:09.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.179+0000 7f4ca8c00640 1 --2- 172.21.15.90:0/4072084398 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f0a228f8f0 0x55f0a228b240 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f4c940058c0 tx=0x7f4c9401f6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6a1e45de2ab89d2 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:34:09.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.180+0000 7f4c99200640 1 -- 172.21.15.90:0/4072084398 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c9401fb80 con 0x55f0a228f8f0 2024-04-02T14:34:09.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.180+0000 7f4c99200640 1 -- 172.21.15.90:0/4072084398 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4c9401fd20 con 0x55f0a228f8f0 2024-04-02T14:34:09.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.180+0000 7f4c99200640 1 -- 172.21.15.90:0/4072084398 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c940339c0 con 0x55f0a228f8f0 2024-04-02T14:34:09.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.180+0000 7f4cae27b740 1 -- 172.21.15.90:0/4072084398 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f0a228f8f0 msgr2=0x55f0a228b240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:09.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.180+0000 7f4cae27b740 1 --2- 172.21.15.90:0/4072084398 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f0a228f8f0 0x55f0a228b240 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f4c940058c0 tx=0x7f4c9401f6c0 comp rx=0 tx=0).stop 2024-04-02T14:34:09.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.180+0000 7f4cae27b740 1 -- 172.21.15.90:0/4072084398 shutdown_connections 2024-04-02T14:34:09.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.180+0000 7f4cae27b740 1 --2- 172.21.15.90:0/4072084398 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f0a2295450 0x55f0a22a5a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:09.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.180+0000 7f4cae27b740 1 --2- 172.21.15.90:0/4072084398 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f0a22902e0 0x55f0a22a1340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:09.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.180+0000 7f4cae27b740 1 --2- 172.21.15.90:0/4072084398 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f0a228f8f0 0x55f0a228b240 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:09.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.180+0000 7f4cae27b740 1 -- 172.21.15.90:0/4072084398 >> 172.21.15.90:0/4072084398 conn(0x55f0a228ae60 msgr2=0x55f0a2290cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:34:09.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.180+0000 7f4cae27b740 1 -- 172.21.15.90:0/4072084398 shutdown_connections 2024-04-02T14:34:09.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.181+0000 7f4cae27b740 1 -- 172.21.15.90:0/4072084398 wait complete. 2024-04-02T14:34:09.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.181+0000 7f4cae27b740 1 -- start start 2024-04-02T14:34:09.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.181+0000 7f4cae27b740 1 Processor -- start 2024-04-02T14:34:09.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.181+0000 7f4cae27b740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f0a228f8f0 0x55f0a22938f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:09.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.181+0000 7f4cae27b740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f0a22902e0 0x55f0a2293e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:09.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.181+0000 7f4cae27b740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f0a2295450 0x55f0a229f930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:09.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.181+0000 7f4cae27b740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55f0a22a7fa0 con 0x55f0a228f8f0 2024-04-02T14:34:09.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.181+0000 7f4cae27b740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55f0a22a78d0 con 0x55f0a22902e0 2024-04-02T14:34:09.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.181+0000 7f4cae27b740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55f0a22a8670 con 0x55f0a2295450 2024-04-02T14:34:09.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.182+0000 7f4cae27b740 10 client.0 _ll_register_callbacks cb 0x7f4c940044b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:34:09.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.182+0000 7f4ca8c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f0a228f8f0 0x55f0a22938f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:09.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.182+0000 7f4ca8c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f0a228f8f0 0x55f0a22938f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:52282/0 (socket says 172.21.15.90:52282) 2024-04-02T14:34:09.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.182+0000 7f4ca8c00640 1 -- 172.21.15.90:0/48374192 learned_addr learned my addr 172.21.15.90:0/48374192 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:34:09.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.182+0000 7f4ca9600640 1 --2- 172.21.15.90:0/48374192 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f0a2295450 0x55f0a229f930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:09.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.182+0000 7f4c9be00640 1 --2- 172.21.15.90:0/48374192 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f0a22902e0 0x55f0a2293e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:09.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.182+0000 7f4ca8c00640 1 -- 172.21.15.90:0/48374192 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f0a2295450 msgr2=0x55f0a229f930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:34:09.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.182+0000 7f4ca8c00640 1 --2- 172.21.15.90:0/48374192 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f0a2295450 0x55f0a229f930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:09.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.182+0000 7f4ca8c00640 1 -- 172.21.15.90:0/48374192 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f0a22902e0 msgr2=0x55f0a2293e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:34:09.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.182+0000 7f4ca8c00640 1 --2- 172.21.15.90:0/48374192 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f0a22902e0 0x55f0a2293e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:09.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.182+0000 7f4ca8c00640 1 -- 172.21.15.90:0/48374192 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f0a229ff20 con 0x55f0a228f8f0 2024-04-02T14:34:09.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.182+0000 7f4ca8c00640 1 --2- 172.21.15.90:0/48374192 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f0a228f8f0 0x55f0a22938f0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f4c94019040 tx=0x7f4c940056b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e847a3b8cd807d3a server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:34:09.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.182+0000 7f4c92000640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-02T14:34:09.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.183+0000 7f4c92000640 1 -- 172.21.15.90:0/48374192 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c94003e50 con 0x55f0a228f8f0 2024-04-02T14:34:09.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.183+0000 7f4c92000640 1 -- 172.21.15.90:0/48374192 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8248+0+0 (secure 0 0 0) 0x7f4c94033950 con 0x55f0a228f8f0 2024-04-02T14:34:09.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.183+0000 7f4cae27b740 1 -- 172.21.15.90:0/48374192 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4c9401fd20 con 0x55f0a228f8f0 2024-04-02T14:34:09.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.183+0000 7f4c92000640 1 -- 172.21.15.90:0/48374192 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f4c94033b90 con 0x55f0a228f8f0 2024-04-02T14:34:09.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.183+0000 7f4cae27b740 10 client.14892 fetch_fsmap learned FSMap version 26 2024-04-02T14:34:09.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.183+0000 7f4cae27b740 1 -- 172.21.15.90:0/48374192 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55f0a22a1220 con 0x55f0a228f8f0 2024-04-02T14:34:09.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.183+0000 7f4c92000640 1 -- 172.21.15.90:0/48374192 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4c9401fd20 con 0x55f0a228f8f0 2024-04-02T14:34:09.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.183+0000 7f4c92000640 1 -- 172.21.15.90:0/48374192 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f4c94032070 con 0x55f0a228f8f0 2024-04-02T14:34:09.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.183+0000 7f4cae27b740 10 client.14892 fetch_fsmap finished waiting for FSMap version 26 2024-04-02T14:34:09.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.183+0000 7f4cae27b740 10 client.14892 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-02T14:34:09.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.183+0000 7f4cae27b740 20 client.14892 populate_metadata read hostname 'smithi090' 2024-04-02T14:34:09.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.183+0000 7f4cae27b740 1 --2- 172.21.15.90:0/48374192 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x55f0a2310e40 0x55f0a230fb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:09.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.183+0000 7f4cae27b740 4 client.14892 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-02T14:34:09.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.183+0000 7f4cae27b740 1 -- 172.21.15.90:0/48374192 --> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] -- command(tid 0: {"prefix": "status"}) -- 0x55f0a21fc8f0 con 0x55f0a2310e40 2024-04-02T14:34:09.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.184+0000 7f4c9be00640 1 --2- 172.21.15.90:0/48374192 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x55f0a2310e40 0x55f0a230fb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:09.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.184+0000 7f4c9be00640 1 --2- 172.21.15.90:0/48374192 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x55f0a2310e40 0x55f0a230fb50 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=439fd45e55e0c93 server_cookie=48b78c1471be4c9d in_seq=0 out_seq=0 2024-04-02T14:34:09.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.184+0000 7f4c92000640 10 client.14892 ms_handle_connect on v2:172.21.15.177:6836/4223414858 2024-04-02T14:34:09.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.184+0000 7f4c92000640 1 -- 172.21.15.90:0/48374192 <== mds.4 v2:172.21.15.177:6836/4223414858 1 ==== command_reply(tid 0: 0 ) ==== 8+0+322 (crc 0 0 0) 0x55f0a21fc8f0 con 0x55f0a2310e40 2024-04-02T14:34:09.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.185+0000 7f4c92000640 10 client.14892 handle_command_reply: tid=0 2024-04-02T14:34:09.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.185+0000 7f4cae27b740 1 client.14892 shutdown 2024-04-02T14:34:09.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.185+0000 7f4cae27b740 1 -- 172.21.15.90:0/48374192 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f0a228f8f0 msgr2=0x55f0a22938f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:09.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.185+0000 7f4cae27b740 1 --2- 172.21.15.90:0/48374192 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f0a228f8f0 0x55f0a22938f0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f4c94019040 tx=0x7f4c940056b0 comp rx=0 tx=0).stop 2024-04-02T14:34:09.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.185+0000 7f4cae27b740 1 -- 172.21.15.90:0/48374192 shutdown_connections 2024-04-02T14:34:09.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.185+0000 7f4cae27b740 1 --2- 172.21.15.90:0/48374192 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f0a2295450 0x55f0a229f930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:09.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.185+0000 7f4cae27b740 1 --2- 172.21.15.90:0/48374192 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f0a22902e0 0x55f0a2293e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:09.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.185+0000 7f4cae27b740 1 --2- 172.21.15.90:0/48374192 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x55f0a2310e40 0x55f0a230fb50 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:09.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.185+0000 7f4cae27b740 1 --2- 172.21.15.90:0/48374192 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f0a228f8f0 0x55f0a22938f0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:09.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.185+0000 7f4cae27b740 1 -- 172.21.15.90:0/48374192 >> 172.21.15.90:0/48374192 conn(0x55f0a228ae60 msgr2=0x55f0a228b240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:34:09.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.185+0000 7f4c92000640 0 client.14892 ms_handle_reset on v2:172.21.15.177:6836/4223414858 2024-04-02T14:34:09.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.185+0000 7f4cae27b740 1 -- 172.21.15.90:0/48374192 shutdown_connections 2024-04-02T14:34:09.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.185+0000 7f4cae27b740 1 -- 172.21.15.90:0/48374192 wait complete. 2024-04-02T14:34:09.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.185+0000 7f4cae27b740 20 client.14892 trim_cache size 0 max 16384 2024-04-02T14:34:09.194 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-02T14:34:09.194 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_fsid": "6449448a-f0fc-11ee-b647-cb9ed24678a4", 2024-04-02T14:34:09.194 INFO:teuthology.orchestra.run.smithi090.stdout: "whoami": 4, 2024-04-02T14:34:09.194 INFO:teuthology.orchestra.run.smithi090.stdout: "id": 24523, 2024-04-02T14:34:09.194 INFO:teuthology.orchestra.run.smithi090.stdout: "want_state": "up:active", 2024-04-02T14:34:09.194 INFO:teuthology.orchestra.run.smithi090.stdout: "state": "up:active", 2024-04-02T14:34:09.194 INFO:teuthology.orchestra.run.smithi090.stdout: "fs_name": "cephfs", 2024-04-02T14:34:09.194 INFO:teuthology.orchestra.run.smithi090.stdout: "rank_uptime": 100.225130357, 2024-04-02T14:34:09.195 INFO:teuthology.orchestra.run.smithi090.stdout: "mdsmap_epoch": 26, 2024-04-02T14:34:09.195 INFO:teuthology.orchestra.run.smithi090.stdout: "osdmap_epoch": 90, 2024-04-02T14:34:09.195 INFO:teuthology.orchestra.run.smithi090.stdout: "osdmap_epoch_barrier": 88, 2024-04-02T14:34:09.195 INFO:teuthology.orchestra.run.smithi090.stdout: "uptime": 128.511460683 2024-04-02T14:34:09.195 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-02T14:34:09.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.186+0000 7f4cab600640 1 -- 172.21.15.90:0/173794501 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f4c880a3bb0 msgr2=0x7f4c880a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:09.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.186+0000 7f4cab600640 1 --2- 172.21.15.90:0/173794501 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f4c880a3bb0 0x7f4c880a6070 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x55f0a22906e0 tx=0x7f4c94005f90 comp rx=0 tx=0).stop 2024-04-02T14:34:09.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.186+0000 7f4cab600640 1 -- 172.21.15.90:0/173794501 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4ca41099d0 msgr2=0x7f4ca41a15e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:09.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.186+0000 7f4cab600640 1 --2- 172.21.15.90:0/173794501 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4ca41099d0 0x7f4ca41a15e0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f4c9c00eb10 tx=0x7f4c9c00c2d0 comp rx=0 tx=0).stop 2024-04-02T14:34:09.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.186+0000 7f4cab600640 1 -- 172.21.15.90:0/173794501 shutdown_connections 2024-04-02T14:34:09.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.186+0000 7f4cab600640 1 --2- 172.21.15.90:0/173794501 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f4c880a3bb0 0x7f4c880a6070 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:09.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.186+0000 7f4cab600640 1 --2- 172.21.15.90:0/173794501 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4ca41099d0 0x7f4ca41a15e0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:09.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.186+0000 7f4cab600640 1 --2- 172.21.15.90:0/173794501 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4ca4075b40 0x7f4ca419a570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:09.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.186+0000 7f4cab600640 1 --2- 172.21.15.90:0/173794501 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4ca4074560 0x7f4ca419a030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:09.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.186+0000 7f4cab600640 1 -- 172.21.15.90:0/173794501 >> 172.21.15.90:0/173794501 conn(0x7f4ca40fd0a0 msgr2=0x7f4ca41058d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:34:09.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.186+0000 7f4cab600640 1 -- 172.21.15.90:0/173794501 shutdown_connections 2024-04-02T14:34:09.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.186+0000 7f4cab600640 1 -- 172.21.15.90:0/173794501 wait complete. 2024-04-02T14:34:09.196 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:34:08 smithi090 ceph-mon[31674]: from='client.? 172.21.15.90:0/333001413' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-02T14:34:09.197 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:34:08 smithi090 ceph-mon[31674]: from='client.? 172.21.15.90:0/2052069513' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-02T14:34:09.197 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:34:08 smithi177 ceph-mon[36431]: from='client.? 172.21.15.90:0/333001413' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-02T14:34:09.197 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:34:08 smithi177 ceph-mon[36431]: from='client.? 172.21.15.90:0/2052069513' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-02T14:34:09.204 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.h status 2024-04-02T14:34:09.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:34:08 smithi092 ceph-mon[36497]: from='client.? 172.21.15.90:0/333001413' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-02T14:34:09.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:34:08 smithi092 ceph-mon[36497]: from='client.? 172.21.15.90:0/2052069513' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-02T14:34:09.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.397+0000 7f80b5c00640 1 -- 172.21.15.90:0/358796394 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f80b00764c0 msgr2=0x7f80b00768e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:09.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.397+0000 7f80b5c00640 1 --2- 172.21.15.90:0/358796394 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f80b00764c0 0x7f80b00768e0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f80a0009900 tx=0x7f80a002bc30 comp rx=0 tx=0).stop 2024-04-02T14:34:09.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.397+0000 7f80b5c00640 1 -- 172.21.15.90:0/358796394 shutdown_connections 2024-04-02T14:34:09.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.397+0000 7f80b5c00640 1 --2- 172.21.15.90:0/358796394 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f80b00780a0 0x7f80b0078940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:09.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.397+0000 7f80b5c00640 1 --2- 172.21.15.90:0/358796394 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f80b0077700 0x7f80b0077b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:09.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.397+0000 7f80b5c00640 1 --2- 172.21.15.90:0/358796394 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f80b00764c0 0x7f80b00768e0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:09.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.397+0000 7f80b5c00640 1 -- 172.21.15.90:0/358796394 >> 172.21.15.90:0/358796394 conn(0x7f80b00ff2b0 msgr2=0x7f80b01016f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:34:09.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.397+0000 7f80b5c00640 1 -- 172.21.15.90:0/358796394 shutdown_connections 2024-04-02T14:34:09.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.397+0000 7f80b5c00640 1 -- 172.21.15.90:0/358796394 wait complete. 2024-04-02T14:34:09.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.399+0000 7f80b5c00640 1 Processor -- start 2024-04-02T14:34:09.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.399+0000 7f80b5c00640 1 -- start start 2024-04-02T14:34:09.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.399+0000 7f80b5c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f80b00764c0 0x7f80b019e450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:09.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.399+0000 7f80b5c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f80b0077700 0x7f80b019e990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:09.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.400+0000 7f80b5c00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f80b00780a0 0x7f80b01a5a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:09.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.400+0000 7f80b5c00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f80b0113dd0 con 0x7f80b0077700 2024-04-02T14:34:09.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.400+0000 7f80b5c00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f80b0113c50 con 0x7f80b00780a0 2024-04-02T14:34:09.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.400+0000 7f80b5c00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f80b0113f50 con 0x7f80b00764c0 2024-04-02T14:34:09.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.400+0000 7f80ae000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f80b0077700 0x7f80b019e990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:09.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.400+0000 7f80aea00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f80b00764c0 0x7f80b019e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:09.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.400+0000 7f80ae000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f80b0077700 0x7f80b019e990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:52298/0 (socket says 172.21.15.90:52298) 2024-04-02T14:34:09.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.400+0000 7f80ae000640 1 -- 172.21.15.90:0/2063714664 learned_addr learned my addr 172.21.15.90:0/2063714664 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:34:09.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.400+0000 7f80af400640 1 --2- 172.21.15.90:0/2063714664 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f80b00780a0 0x7f80b01a5a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:09.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.400+0000 7f80ae000640 1 -- 172.21.15.90:0/2063714664 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f80b00764c0 msgr2=0x7f80b019e450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:09.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.400+0000 7f80ae000640 1 --2- 172.21.15.90:0/2063714664 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f80b00764c0 0x7f80b019e450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:09.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.400+0000 7f80ae000640 1 -- 172.21.15.90:0/2063714664 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f80b00780a0 msgr2=0x7f80b01a5a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:09.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.400+0000 7f80ae000640 1 --2- 172.21.15.90:0/2063714664 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f80b00780a0 0x7f80b01a5a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:09.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.400+0000 7f80ae000640 1 -- 172.21.15.90:0/2063714664 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80b01a6070 con 0x7f80b0077700 2024-04-02T14:34:09.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.400+0000 7f80af400640 1 --2- 172.21.15.90:0/2063714664 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f80b00780a0 0x7f80b01a5a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:34:09.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.401+0000 7f80ae000640 1 --2- 172.21.15.90:0/2063714664 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f80b0077700 0x7f80b019e990 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f809c00ea60 tx=0x7f809c00ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:34:09.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.401+0000 7f80a7400640 1 -- 172.21.15.90:0/2063714664 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f809c00cde0 con 0x7f80b0077700 2024-04-02T14:34:09.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.401+0000 7f80a7400640 1 -- 172.21.15.90:0/2063714664 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f809c0035f0 con 0x7f80b0077700 2024-04-02T14:34:09.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.401+0000 7f80a7400640 1 -- 172.21.15.90:0/2063714664 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f809c010670 con 0x7f80b0077700 2024-04-02T14:34:09.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.401+0000 7f80b5c00640 1 -- 172.21.15.90:0/2063714664 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80b01a6360 con 0x7f80b0077700 2024-04-02T14:34:09.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.401+0000 7f80b5c00640 1 -- 172.21.15.90:0/2063714664 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80b01a6900 con 0x7f80b0077700 2024-04-02T14:34:09.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.404+0000 7f80b89d6740 1 Processor -- start 2024-04-02T14:34:09.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.404+0000 7f80b89d6740 1 -- start start 2024-04-02T14:34:09.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.404+0000 7f80b89d6740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bfa7f598f0 0x55bfa7f59d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:09.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.404+0000 7f80b89d6740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55bfa7f5a2e0 0x55bfa7f5ede0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:09.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.404+0000 7f80b89d6740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bfa7f5f450 0x55bfa7f618e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:09.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.404+0000 7f80b89d6740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55bfa7ebf060 con 0x55bfa7f598f0 2024-04-02T14:34:09.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.404+0000 7f80b89d6740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55bfa7e67560 con 0x55bfa7f5a2e0 2024-04-02T14:34:09.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.405+0000 7f80b89d6740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55bfa7ebb010 con 0x55bfa7f5f450 2024-04-02T14:34:09.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.405+0000 7f80af400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bfa7f598f0 0x55bfa7f59d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:09.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.405+0000 7f80af400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bfa7f598f0 0x55bfa7f59d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:52302/0 (socket says 172.21.15.90:52302) 2024-04-02T14:34:09.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.405+0000 7f80af400640 1 -- 172.21.15.90:0/1676897698 learned_addr learned my addr 172.21.15.90:0/1676897698 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:34:09.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.405+0000 7f80a7400640 1 -- 172.21.15.90:0/2063714664 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f809c0037a0 con 0x7f80b0077700 2024-04-02T14:34:09.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.405+0000 7f80aea00640 1 --2- 172.21.15.90:0/1676897698 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55bfa7f5a2e0 0x55bfa7f5ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:09.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.405+0000 7f80ae000640 1 --2- 172.21.15.90:0/1676897698 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bfa7f5f450 0x55bfa7f618e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:09.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.405+0000 7f80af400640 1 -- 172.21.15.90:0/1676897698 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bfa7f5f450 msgr2=0x55bfa7f618e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:34:09.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.405+0000 7f80af400640 1 --2- 172.21.15.90:0/1676897698 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bfa7f5f450 0x55bfa7f618e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:09.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.405+0000 7f80af400640 1 -- 172.21.15.90:0/1676897698 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55bfa7f5a2e0 msgr2=0x55bfa7f5ede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:34:09.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.405+0000 7f80af400640 1 --2- 172.21.15.90:0/1676897698 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55bfa7f5a2e0 0x55bfa7f5ede0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:09.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.405+0000 7f80af400640 1 -- 172.21.15.90:0/1676897698 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bfa7e89fd0 con 0x55bfa7f598f0 2024-04-02T14:34:09.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.405+0000 7f80ae000640 1 --2- 172.21.15.90:0/1676897698 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bfa7f5f450 0x55bfa7f618e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:34:09.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.405+0000 7f80aea00640 1 --2- 172.21.15.90:0/1676897698 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55bfa7f5a2e0 0x55bfa7f5ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:34:09.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.406+0000 7f80af400640 1 --2- 172.21.15.90:0/1676897698 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bfa7f598f0 0x55bfa7f59d10 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f80a8009fd0 tx=0x7f80a80108e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a997142cd5015582 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:34:09.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.406+0000 7f809be00640 1 -- 172.21.15.90:0/1676897698 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80a8011760 con 0x55bfa7f598f0 2024-04-02T14:34:09.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.406+0000 7f809be00640 1 -- 172.21.15.90:0/1676897698 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f80a8011de0 con 0x55bfa7f598f0 2024-04-02T14:34:09.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.406+0000 7f809be00640 1 -- 172.21.15.90:0/1676897698 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80a800e3c0 con 0x55bfa7f598f0 2024-04-02T14:34:09.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.407+0000 7f80a7400640 1 --2- 172.21.15.90:0/2063714664 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f80940a3c30 0x7f80940a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:09.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.407+0000 7f80b89d6740 1 -- 172.21.15.90:0/1676897698 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bfa7f598f0 msgr2=0x55bfa7f59d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:09.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.407+0000 7f80b89d6740 1 --2- 172.21.15.90:0/1676897698 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bfa7f598f0 0x55bfa7f59d10 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f80a8009fd0 tx=0x7f80a80108e0 comp rx=0 tx=0).stop 2024-04-02T14:34:09.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.407+0000 7f80a7400640 1 -- 172.21.15.90:0/2063714664 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8248+0+0 (secure 0 0 0) 0x7f809c0d0760 con 0x7f80b0077700 2024-04-02T14:34:09.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.407+0000 7f80aea00640 1 --2- 172.21.15.90:0/2063714664 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f80940a3c30 0x7f80940a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:09.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.407+0000 7f80b89d6740 1 -- 172.21.15.90:0/1676897698 shutdown_connections 2024-04-02T14:34:09.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.407+0000 7f80b89d6740 1 --2- 172.21.15.90:0/1676897698 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bfa7f5f450 0x55bfa7f618e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:09.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.407+0000 7f80b89d6740 1 --2- 172.21.15.90:0/1676897698 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55bfa7f5a2e0 0x55bfa7f5ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:09.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.407+0000 7f80b89d6740 1 --2- 172.21.15.90:0/1676897698 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bfa7f598f0 0x55bfa7f59d10 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:09.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.407+0000 7f80b89d6740 1 -- 172.21.15.90:0/1676897698 >> 172.21.15.90:0/1676897698 conn(0x55bfa7f54e60 msgr2=0x55bfa7f55260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:34:09.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.407+0000 7f80b89d6740 1 -- 172.21.15.90:0/1676897698 shutdown_connections 2024-04-02T14:34:09.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.407+0000 7f80b89d6740 1 -- 172.21.15.90:0/1676897698 wait complete. 2024-04-02T14:34:09.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.408+0000 7f80aea00640 1 --2- 172.21.15.90:0/2063714664 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f80940a3c30 0x7f80940a60f0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f80a002c7b0 tx=0x7f80a0005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:34:09.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.408+0000 7f80b89d6740 1 -- start start 2024-04-02T14:34:09.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.408+0000 7f80b89d6740 1 Processor -- start 2024-04-02T14:34:09.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.408+0000 7f80b89d6740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bfa7f598f0 0x55bfa7f63e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:09.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.408+0000 7f80b89d6740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bfa7f5a2e0 0x55bfa7f64360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:09.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.408+0000 7f80b89d6740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55bfa7f5f450 0x55bfa7f6b330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:09.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.408+0000 7f80b89d6740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55bfa7f62400 con 0x55bfa7f598f0 2024-04-02T14:34:09.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.408+0000 7f80b89d6740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55bfa7e7a920 con 0x55bfa7f5f450 2024-04-02T14:34:09.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.408+0000 7f80b89d6740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55bfa7f62580 con 0x55bfa7f5a2e0 2024-04-02T14:34:09.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.408+0000 7f80b89d6740 10 client.0 _ll_register_callbacks cb 0x55bfa7e7a310 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:34:09.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.409+0000 7f80aea00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bfa7f598f0 0x55bfa7f63e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:09.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.409+0000 7f80aea00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bfa7f598f0 0x55bfa7f63e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:52312/0 (socket says 172.21.15.90:52312) 2024-04-02T14:34:09.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.409+0000 7f80aea00640 1 -- 172.21.15.90:0/2136268381 learned_addr learned my addr 172.21.15.90:0/2136268381 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:34:09.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.409+0000 7f80af400640 1 --2- 172.21.15.90:0/2136268381 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55bfa7f5f450 0x55bfa7f6b330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:09.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.409+0000 7f80ae000640 1 --2- 172.21.15.90:0/2136268381 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bfa7f5a2e0 0x55bfa7f64360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:09.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.409+0000 7f80aea00640 1 -- 172.21.15.90:0/2136268381 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bfa7f5a2e0 msgr2=0x55bfa7f64360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:34:09.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.409+0000 7f80aea00640 1 --2- 172.21.15.90:0/2136268381 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bfa7f5a2e0 0x55bfa7f64360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:09.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.409+0000 7f80aea00640 1 -- 172.21.15.90:0/2136268381 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55bfa7f5f450 msgr2=0x55bfa7f6b330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:34:09.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.409+0000 7f80aea00640 1 --2- 172.21.15.90:0/2136268381 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55bfa7f5f450 0x55bfa7f6b330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:09.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.409+0000 7f80aea00640 1 -- 172.21.15.90:0/2136268381 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bfa7bd32d0 con 0x55bfa7f598f0 2024-04-02T14:34:09.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.409+0000 7f80af400640 1 --2- 172.21.15.90:0/2136268381 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55bfa7f5f450 0x55bfa7f6b330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:34:09.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.409+0000 7f80aea00640 1 --2- 172.21.15.90:0/2136268381 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bfa7f598f0 0x55bfa7f63e20 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f80a0009900 tx=0x7f80a0009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ade488d2362d59f server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:34:09.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.409+0000 7f8098c00640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-02T14:34:09.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.410+0000 7f8098c00640 1 -- 172.21.15.90:0/2136268381 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80a0043070 con 0x55bfa7f598f0 2024-04-02T14:34:09.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.410+0000 7f8098c00640 1 -- 172.21.15.90:0/2136268381 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8248+0+0 (secure 0 0 0) 0x7f80a0034640 con 0x55bfa7f598f0 2024-04-02T14:34:09.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.410+0000 7f80b89d6740 1 -- 172.21.15.90:0/2136268381 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f80a8011de0 con 0x55bfa7f598f0 2024-04-02T14:34:09.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.410+0000 7f8098c00640 1 -- 172.21.15.90:0/2136268381 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f80a0003e10 con 0x55bfa7f598f0 2024-04-02T14:34:09.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.410+0000 7f80b89d6740 10 client.14904 fetch_fsmap learned FSMap version 26 2024-04-02T14:34:09.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.410+0000 7f80b89d6740 1 -- 172.21.15.90:0/2136268381 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55bfa7c9d6b0 con 0x55bfa7f598f0 2024-04-02T14:34:09.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.410+0000 7f8098c00640 1 -- 172.21.15.90:0/2136268381 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f80a8011de0 con 0x55bfa7f598f0 2024-04-02T14:34:09.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.410+0000 7f8098c00640 1 -- 172.21.15.90:0/2136268381 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f80a0052b10 con 0x55bfa7f598f0 2024-04-02T14:34:09.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.410+0000 7f80b89d6740 10 client.14904 fetch_fsmap finished waiting for FSMap version 26 2024-04-02T14:34:09.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.411+0000 7f80b89d6740 10 client.14904 resolve_mds: resolved name 'h' to daemon mds.h 2024-04-02T14:34:09.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.411+0000 7f80b89d6740 20 client.14904 populate_metadata read hostname 'smithi090' 2024-04-02T14:34:09.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.411+0000 7f80b89d6740 1 --2- 172.21.15.90:0/2136268381 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x55bfa7f70bf0 0x55bfa7f72060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:09.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.411+0000 7f80b89d6740 4 client.14904 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:34:09.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.411+0000 7f80b89d6740 1 -- 172.21.15.90:0/2136268381 --> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bfa7ecc5a0 con 0x55bfa7f70bf0 2024-04-02T14:34:09.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.411+0000 7f80ae000640 1 --2- 172.21.15.90:0/2136268381 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x55bfa7f70bf0 0x55bfa7f72060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:09.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.412+0000 7f80ae000640 1 --2- 172.21.15.90:0/2136268381 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x55bfa7f70bf0 0x55bfa7f72060 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=a301a673f6aa2ded server_cookie=faadb53f59f9b620 in_seq=0 out_seq=0 2024-04-02T14:34:09.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.412+0000 7f8098c00640 10 client.14904 ms_handle_connect on v2:172.21.15.92:6836/1270635939 2024-04-02T14:34:09.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.412+0000 7f8098c00640 1 -- 172.21.15.90:0/2136268381 <== mds.3 v2:172.21.15.92:6836/1270635939 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55bfa7ecc5a0 con 0x55bfa7f70bf0 2024-04-02T14:34:09.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.412+0000 7f8098c00640 10 client.14904 handle_command_reply: tid=0 2024-04-02T14:34:09.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.413+0000 7f80b89d6740 1 client.14904 shutdown 2024-04-02T14:34:09.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.413+0000 7f80b89d6740 1 -- 172.21.15.90:0/2136268381 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bfa7f598f0 msgr2=0x55bfa7f63e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:09.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.413+0000 7f80b89d6740 1 --2- 172.21.15.90:0/2136268381 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bfa7f598f0 0x55bfa7f63e20 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f80a0009900 tx=0x7f80a0009260 comp rx=0 tx=0).stop 2024-04-02T14:34:09.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.413+0000 7f80b89d6740 1 -- 172.21.15.90:0/2136268381 shutdown_connections 2024-04-02T14:34:09.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.413+0000 7f80b89d6740 1 --2- 172.21.15.90:0/2136268381 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x55bfa7f70bf0 0x55bfa7f72060 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:09.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.413+0000 7f80b89d6740 1 --2- 172.21.15.90:0/2136268381 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55bfa7f5f450 0x55bfa7f6b330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:09.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.413+0000 7f80b89d6740 1 --2- 172.21.15.90:0/2136268381 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bfa7f5a2e0 0x55bfa7f64360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:09.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.413+0000 7f80b89d6740 1 --2- 172.21.15.90:0/2136268381 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bfa7f598f0 0x55bfa7f63e20 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:09.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.413+0000 7f8098c00640 0 client.14904 ms_handle_reset on v2:172.21.15.92:6836/1270635939 2024-04-02T14:34:09.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.413+0000 7f80b89d6740 1 -- 172.21.15.90:0/2136268381 >> 172.21.15.90:0/2136268381 conn(0x55bfa7f54e60 msgr2=0x55bfa7f54080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:34:09.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.413+0000 7f80b89d6740 1 -- 172.21.15.90:0/2136268381 shutdown_connections 2024-04-02T14:34:09.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.413+0000 7f80b89d6740 1 -- 172.21.15.90:0/2136268381 wait complete. 2024-04-02T14:34:09.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.413+0000 7f80b89d6740 20 client.14904 trim_cache size 0 max 16384 2024-04-02T14:34:09.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.433+0000 7f80b89d6740 1 Processor -- start 2024-04-02T14:34:09.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.433+0000 7f80b89d6740 1 -- start start 2024-04-02T14:34:09.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.433+0000 7f80b89d6740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55bfa7f598f0 0x55bfa7f55240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:09.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.433+0000 7f80b89d6740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bfa7f5a2e0 0x55bfa7f6b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:09.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.433+0000 7f80b89d6740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bfa7f5f450 0x55bfa7f6f9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:09.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.433+0000 7f80b89d6740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55bfa7f703a0 con 0x55bfa7f5a2e0 2024-04-02T14:34:09.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.433+0000 7f80b89d6740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55bfa7f70a70 con 0x55bfa7f598f0 2024-04-02T14:34:09.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.433+0000 7f80b89d6740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55bfa7f71140 con 0x55bfa7f5f450 2024-04-02T14:34:09.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.434+0000 7f80aea00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55bfa7f598f0 0x55bfa7f55240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:09.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.434+0000 7f80ae000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bfa7f5a2e0 0x55bfa7f6b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:09.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.434+0000 7f80aea00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55bfa7f598f0 0x55bfa7f55240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:57264/0 (socket says 172.21.15.90:57264) 2024-04-02T14:34:09.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.434+0000 7f80aea00640 1 -- 172.21.15.90:0/1116120156 learned_addr learned my addr 172.21.15.90:0/1116120156 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:34:09.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.434+0000 7f80af400640 1 --2- 172.21.15.90:0/1116120156 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bfa7f5f450 0x55bfa7f6f9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:09.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.434+0000 7f80aea00640 1 -- 172.21.15.90:0/1116120156 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bfa7f5f450 msgr2=0x55bfa7f6f9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:34:09.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.434+0000 7f80aea00640 1 --2- 172.21.15.90:0/1116120156 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bfa7f5f450 0x55bfa7f6f9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:09.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.434+0000 7f80aea00640 1 -- 172.21.15.90:0/1116120156 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bfa7f5a2e0 msgr2=0x55bfa7f6b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:34:09.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.434+0000 7f80aea00640 1 --2- 172.21.15.90:0/1116120156 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bfa7f5a2e0 0x55bfa7f6b2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:09.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.434+0000 7f80aea00640 1 -- 172.21.15.90:0/1116120156 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bfa7f6e550 con 0x55bfa7f598f0 2024-04-02T14:34:09.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.434+0000 7f80aea00640 1 --2- 172.21.15.90:0/1116120156 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55bfa7f598f0 0x55bfa7f55240 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f80a0009230 tx=0x7f80a0005910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=84363c0bf869d2a1 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:34:09.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.435+0000 7f809b400640 1 -- 172.21.15.90:0/1116120156 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80a0052610 con 0x55bfa7f598f0 2024-04-02T14:34:09.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.435+0000 7f809b400640 1 -- 172.21.15.90:0/1116120156 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f80a0048530 con 0x55bfa7f598f0 2024-04-02T14:34:09.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.435+0000 7f809b400640 1 -- 172.21.15.90:0/1116120156 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80a0052c90 con 0x55bfa7f598f0 2024-04-02T14:34:09.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.435+0000 7f80b89d6740 1 -- 172.21.15.90:0/1116120156 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55bfa7f598f0 msgr2=0x55bfa7f55240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:09.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.435+0000 7f80b89d6740 1 --2- 172.21.15.90:0/1116120156 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55bfa7f598f0 0x55bfa7f55240 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f80a0009230 tx=0x7f80a0005910 comp rx=0 tx=0).stop 2024-04-02T14:34:09.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.436+0000 7f80b89d6740 1 -- 172.21.15.90:0/1116120156 shutdown_connections 2024-04-02T14:34:09.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.436+0000 7f80b89d6740 1 --2- 172.21.15.90:0/1116120156 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bfa7f5f450 0x55bfa7f6f9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:09.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.436+0000 7f80b89d6740 1 --2- 172.21.15.90:0/1116120156 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bfa7f5a2e0 0x55bfa7f6b2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:09.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.436+0000 7f80b89d6740 1 --2- 172.21.15.90:0/1116120156 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55bfa7f598f0 0x55bfa7f55240 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:09.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.436+0000 7f80b89d6740 1 -- 172.21.15.90:0/1116120156 >> 172.21.15.90:0/1116120156 conn(0x55bfa7f54e60 msgr2=0x55bfa7f62950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:34:09.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.436+0000 7f80b89d6740 1 -- 172.21.15.90:0/1116120156 shutdown_connections 2024-04-02T14:34:09.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.436+0000 7f80b89d6740 1 -- 172.21.15.90:0/1116120156 wait complete. 2024-04-02T14:34:09.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.436+0000 7f80b89d6740 1 -- start start 2024-04-02T14:34:09.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.436+0000 7f80b89d6740 1 Processor -- start 2024-04-02T14:34:09.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.437+0000 7f80b89d6740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55bfa7f598f0 0x55bfa7f692e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:09.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.437+0000 7f80b89d6740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bfa7f5a2e0 0x55bfa7fd2b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:09.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.437+0000 7f80b89d6740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bfa7f5f450 0x55bfa7fd50d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:09.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.437+0000 7f80b89d6740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55bfa7f71ee0 con 0x55bfa7f5a2e0 2024-04-02T14:34:09.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.437+0000 7f80b89d6740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55bfa7f71810 con 0x55bfa7f598f0 2024-04-02T14:34:09.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.437+0000 7f80b89d6740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55bfa7f725b0 con 0x55bfa7f5f450 2024-04-02T14:34:09.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.437+0000 7f80b89d6740 10 client.0 _ll_register_callbacks cb 0x7f80a00041a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:34:09.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.437+0000 7f80ae000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bfa7f5a2e0 0x55bfa7fd2b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:09.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.437+0000 7f80aea00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55bfa7f598f0 0x55bfa7f692e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:09.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.437+0000 7f80ae000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bfa7f5a2e0 0x55bfa7fd2b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:52334/0 (socket says 172.21.15.90:52334) 2024-04-02T14:34:09.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.437+0000 7f80ae000640 1 -- 172.21.15.90:0/3326707324 learned_addr learned my addr 172.21.15.90:0/3326707324 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:34:09.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.437+0000 7f80af400640 1 --2- 172.21.15.90:0/3326707324 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bfa7f5f450 0x55bfa7fd50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:09.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.437+0000 7f80aea00640 1 -- 172.21.15.90:0/3326707324 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bfa7f5f450 msgr2=0x55bfa7fd50d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:34:09.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.437+0000 7f80aea00640 1 --2- 172.21.15.90:0/3326707324 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bfa7f5f450 0x55bfa7fd50d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:09.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.437+0000 7f80aea00640 1 -- 172.21.15.90:0/3326707324 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bfa7f5a2e0 msgr2=0x55bfa7fd2b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:34:09.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.437+0000 7f80aea00640 1 --2- 172.21.15.90:0/3326707324 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bfa7f5a2e0 0x55bfa7fd2b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:09.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.437+0000 7f80aea00640 1 -- 172.21.15.90:0/3326707324 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bfa7fd5610 con 0x55bfa7f598f0 2024-04-02T14:34:09.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.437+0000 7f80af400640 1 --2- 172.21.15.90:0/3326707324 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bfa7f5f450 0x55bfa7fd50d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:34:09.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.437+0000 7f80ae000640 1 --2- 172.21.15.90:0/3326707324 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bfa7f5a2e0 0x55bfa7fd2b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:34:09.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.438+0000 7f80aea00640 1 --2- 172.21.15.90:0/3326707324 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55bfa7f598f0 0x55bfa7f692e0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f80a0009900 tx=0x7f80a0048cc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=de8aa68ec752c719 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:34:09.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.438+0000 7f8098c00640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:34:09.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.438+0000 7f8098c00640 1 -- 172.21.15.90:0/3326707324 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80a006c070 con 0x55bfa7f598f0 2024-04-02T14:34:09.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.438+0000 7f8098c00640 1 -- 172.21.15.90:0/3326707324 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8248+0+0 (secure 0 0 0) 0x7f80a0052820 con 0x55bfa7f598f0 2024-04-02T14:34:09.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.438+0000 7f80b89d6740 1 -- 172.21.15.90:0/3326707324 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f80a0048530 con 0x55bfa7f598f0 2024-04-02T14:34:09.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.438+0000 7f8098c00640 1 -- 172.21.15.90:0/3326707324 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f80a0077050 con 0x55bfa7f598f0 2024-04-02T14:34:09.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.438+0000 7f80b89d6740 10 client.25019 fetch_fsmap learned FSMap version 26 2024-04-02T14:34:09.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.438+0000 7f80b89d6740 1 -- 172.21.15.90:0/3326707324 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55bfa7fd6780 con 0x55bfa7f598f0 2024-04-02T14:34:09.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.439+0000 7f8098c00640 1 -- 172.21.15.90:0/3326707324 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f80a0048530 con 0x55bfa7f598f0 2024-04-02T14:34:09.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.439+0000 7f8098c00640 1 -- 172.21.15.90:0/3326707324 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f80a0066430 con 0x55bfa7f598f0 2024-04-02T14:34:09.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.439+0000 7f80b89d6740 10 client.25019 fetch_fsmap finished waiting for FSMap version 26 2024-04-02T14:34:09.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.439+0000 7f80b89d6740 10 client.25019 resolve_mds: resolved name 'h' to daemon mds.h 2024-04-02T14:34:09.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.439+0000 7f80b89d6740 20 client.25019 populate_metadata read hostname 'smithi090' 2024-04-02T14:34:09.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.439+0000 7f80b89d6740 1 --2- 172.21.15.90:0/3326707324 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x55bfa7fdb130 0x55bfa7fd9e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:09.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.439+0000 7f80b89d6740 4 client.25019 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-02T14:34:09.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.439+0000 7f80b89d6740 1 -- 172.21.15.90:0/3326707324 --> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] -- command(tid 0: {"prefix": "status"}) -- 0x55bfa7ec68f0 con 0x55bfa7fdb130 2024-04-02T14:34:09.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.439+0000 7f80ae000640 1 --2- 172.21.15.90:0/3326707324 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x55bfa7fdb130 0x55bfa7fd9e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:09.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.440+0000 7f80ae000640 1 --2- 172.21.15.90:0/3326707324 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x55bfa7fdb130 0x55bfa7fd9e40 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=f9a1668bc0251aaa server_cookie=64c6024a746a78a6 in_seq=0 out_seq=0 2024-04-02T14:34:09.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.440+0000 7f8098c00640 10 client.25019 ms_handle_connect on v2:172.21.15.92:6836/1270635939 2024-04-02T14:34:09.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.440+0000 7f8098c00640 1 -- 172.21.15.90:0/3326707324 <== mds.3 v2:172.21.15.92:6836/1270635939 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x55bfa7ec68f0 con 0x55bfa7fdb130 2024-04-02T14:34:09.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.440+0000 7f8098c00640 10 client.25019 handle_command_reply: tid=0 2024-04-02T14:34:09.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.440+0000 7f80b89d6740 1 client.25019 shutdown 2024-04-02T14:34:09.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.440+0000 7f80b89d6740 1 -- 172.21.15.90:0/3326707324 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55bfa7f598f0 msgr2=0x55bfa7f692e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:09.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.440+0000 7f80b89d6740 1 --2- 172.21.15.90:0/3326707324 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55bfa7f598f0 0x55bfa7f692e0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f80a0009900 tx=0x7f80a0048cc0 comp rx=0 tx=0).stop 2024-04-02T14:34:09.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.440+0000 7f80b89d6740 1 -- 172.21.15.90:0/3326707324 shutdown_connections 2024-04-02T14:34:09.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.440+0000 7f80b89d6740 1 --2- 172.21.15.90:0/3326707324 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x55bfa7fdb130 0x55bfa7fd9e40 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:09.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.440+0000 7f80b89d6740 1 --2- 172.21.15.90:0/3326707324 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bfa7f5f450 0x55bfa7fd50d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:09.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.440+0000 7f80b89d6740 1 --2- 172.21.15.90:0/3326707324 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bfa7f5a2e0 0x55bfa7fd2b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:09.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.440+0000 7f80b89d6740 1 --2- 172.21.15.90:0/3326707324 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55bfa7f598f0 0x55bfa7f692e0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:09.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.440+0000 7f8098c00640 0 client.25019 ms_handle_reset on v2:172.21.15.92:6836/1270635939 2024-04-02T14:34:09.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.440+0000 7f80b89d6740 1 -- 172.21.15.90:0/3326707324 >> 172.21.15.90:0/3326707324 conn(0x55bfa7f54e60 msgr2=0x55bfa7f55240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:34:09.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.440+0000 7f80b89d6740 1 -- 172.21.15.90:0/3326707324 shutdown_connections 2024-04-02T14:34:09.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.440+0000 7f80b89d6740 1 -- 172.21.15.90:0/3326707324 wait complete. 2024-04-02T14:34:09.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.440+0000 7f80b89d6740 20 client.25019 trim_cache size 0 max 16384 2024-04-02T14:34:09.447 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-02T14:34:09.447 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_fsid": "6449448a-f0fc-11ee-b647-cb9ed24678a4", 2024-04-02T14:34:09.447 INFO:teuthology.orchestra.run.smithi090.stdout: "whoami": 3, 2024-04-02T14:34:09.447 INFO:teuthology.orchestra.run.smithi090.stdout: "id": 24539, 2024-04-02T14:34:09.447 INFO:teuthology.orchestra.run.smithi090.stdout: "want_state": "up:active", 2024-04-02T14:34:09.447 INFO:teuthology.orchestra.run.smithi090.stdout: "state": "up:active", 2024-04-02T14:34:09.447 INFO:teuthology.orchestra.run.smithi090.stdout: "fs_name": "cephfs", 2024-04-02T14:34:09.447 INFO:teuthology.orchestra.run.smithi090.stdout: "rank_uptime": 101.50653891899999, 2024-04-02T14:34:09.447 INFO:teuthology.orchestra.run.smithi090.stdout: "mdsmap_epoch": 26, 2024-04-02T14:34:09.448 INFO:teuthology.orchestra.run.smithi090.stdout: "osdmap_epoch": 90, 2024-04-02T14:34:09.448 INFO:teuthology.orchestra.run.smithi090.stdout: "osdmap_epoch_barrier": 88, 2024-04-02T14:34:09.448 INFO:teuthology.orchestra.run.smithi090.stdout: "uptime": 126.003109573 2024-04-02T14:34:09.448 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-02T14:34:09.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.441+0000 7f80b5c00640 1 -- 172.21.15.90:0/2063714664 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f80940a3c30 msgr2=0x7f80940a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:09.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.441+0000 7f80b5c00640 1 --2- 172.21.15.90:0/2063714664 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f80940a3c30 0x7f80940a60f0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f80a002c7b0 tx=0x7f80a0005fd0 comp rx=0 tx=0).stop 2024-04-02T14:34:09.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.441+0000 7f80b5c00640 1 -- 172.21.15.90:0/2063714664 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f80b0077700 msgr2=0x7f80b019e990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:09.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.441+0000 7f80b5c00640 1 --2- 172.21.15.90:0/2063714664 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f80b0077700 0x7f80b019e990 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f809c00ea60 tx=0x7f809c00ef20 comp rx=0 tx=0).stop 2024-04-02T14:34:09.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.441+0000 7f80b5c00640 1 -- 172.21.15.90:0/2063714664 shutdown_connections 2024-04-02T14:34:09.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.441+0000 7f80b5c00640 1 --2- 172.21.15.90:0/2063714664 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f80940a3c30 0x7f80940a60f0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:09.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.441+0000 7f80b5c00640 1 --2- 172.21.15.90:0/2063714664 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f80b00780a0 0x7f80b01a5a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:09.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.441+0000 7f80b5c00640 1 --2- 172.21.15.90:0/2063714664 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f80b0077700 0x7f80b019e990 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:09.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.441+0000 7f80b5c00640 1 --2- 172.21.15.90:0/2063714664 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f80b00764c0 0x7f80b019e450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:09.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.441+0000 7f80b5c00640 1 -- 172.21.15.90:0/2063714664 >> 172.21.15.90:0/2063714664 conn(0x7f80b00ff2b0 msgr2=0x7f80b0102610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:34:09.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.441+0000 7f80b5c00640 1 -- 172.21.15.90:0/2063714664 shutdown_connections 2024-04-02T14:34:09.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.441+0000 7f80b5c00640 1 -- 172.21.15.90:0/2063714664 wait complete. 2024-04-02T14:34:09.459 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2024-04-02T14:34:09.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.650+0000 7f367ca00640 1 -- 172.21.15.90:0/270352737 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f36781076c0 msgr2=0x7f3678109b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:09.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.650+0000 7f367ca00640 1 --2- 172.21.15.90:0/270352737 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f36781076c0 0x7f3678109b10 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f3670009ca0 tx=0x7f367002c100 comp rx=0 tx=0).stop 2024-04-02T14:34:09.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.650+0000 7f367ca00640 1 -- 172.21.15.90:0/270352737 shutdown_connections 2024-04-02T14:34:09.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.650+0000 7f367ca00640 1 --2- 172.21.15.90:0/270352737 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f36781076c0 0x7f3678109b10 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:09.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.651+0000 7f367ca00640 1 --2- 172.21.15.90:0/270352737 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3678104d90 0x7f3678107180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:09.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.651+0000 7f367ca00640 1 --2- 172.21.15.90:0/270352737 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3678102460 0x7f3678104850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:09.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.651+0000 7f367ca00640 1 -- 172.21.15.90:0/270352737 >> 172.21.15.90:0/270352737 conn(0x7f36780fbe70 msgr2=0x7f36780fe290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:34:09.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.651+0000 7f367ca00640 1 -- 172.21.15.90:0/270352737 shutdown_connections 2024-04-02T14:34:09.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.651+0000 7f367ca00640 1 -- 172.21.15.90:0/270352737 wait complete. 2024-04-02T14:34:09.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.651+0000 7f367ca00640 1 Processor -- start 2024-04-02T14:34:09.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.652+0000 7f367ca00640 1 -- start start 2024-04-02T14:34:09.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.652+0000 7f367ca00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3678102460 0x7f3678101dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:09.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.652+0000 7f367ca00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3678104d90 0x7f3678100410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:09.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.652+0000 7f367ca00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f36781076c0 0x7f3678100950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:09.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.652+0000 7f367ca00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f367810d680 con 0x7f36781076c0 2024-04-02T14:34:09.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.652+0000 7f367ca00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f367810d500 con 0x7f3678102460 2024-04-02T14:34:09.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.652+0000 7f367ca00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f367810d800 con 0x7f3678104d90 2024-04-02T14:34:09.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.652+0000 7f3677200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f36781076c0 0x7f3678100950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:09.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.652+0000 7f3677200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f36781076c0 0x7f3678100950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:52348/0 (socket says 172.21.15.90:52348) 2024-04-02T14:34:09.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.652+0000 7f3677200640 1 -- 172.21.15.90:0/315135037 learned_addr learned my addr 172.21.15.90:0/315135037 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:34:09.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.652+0000 7f3676800640 1 --2- 172.21.15.90:0/315135037 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3678102460 0x7f3678101dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:09.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.652+0000 7f366fe00640 1 --2- 172.21.15.90:0/315135037 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3678104d90 0x7f3678100410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:09.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.653+0000 7f3677200640 1 -- 172.21.15.90:0/315135037 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3678104d90 msgr2=0x7f3678100410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:09.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.653+0000 7f3677200640 1 --2- 172.21.15.90:0/315135037 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3678104d90 0x7f3678100410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:09.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.653+0000 7f3677200640 1 -- 172.21.15.90:0/315135037 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3678102460 msgr2=0x7f3678101dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:09.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.653+0000 7f3677200640 1 --2- 172.21.15.90:0/315135037 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3678102460 0x7f3678101dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:09.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.653+0000 7f3677200640 1 -- 172.21.15.90:0/315135037 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36781011e0 con 0x7f36781076c0 2024-04-02T14:34:09.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.653+0000 7f3677200640 1 --2- 172.21.15.90:0/315135037 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f36781076c0 0x7f3678100950 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f3670037040 tx=0x7f3670035d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:34:09.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.653+0000 7f366f400640 1 -- 172.21.15.90:0/315135037 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3670039040 con 0x7f36781076c0 2024-04-02T14:34:09.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.653+0000 7f366f400640 1 -- 172.21.15.90:0/315135037 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3670044070 con 0x7f36781076c0 2024-04-02T14:34:09.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.653+0000 7f366f400640 1 -- 172.21.15.90:0/315135037 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3670004460 con 0x7f36781076c0 2024-04-02T14:34:09.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.653+0000 7f367ca00640 1 -- 172.21.15.90:0/315135037 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f36781a1d80 con 0x7f36781076c0 2024-04-02T14:34:09.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.654+0000 7f367ca00640 1 -- 172.21.15.90:0/315135037 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36781a21c0 con 0x7f36781076c0 2024-04-02T14:34:09.657 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.656+0000 7f366f400640 1 -- 172.21.15.90:0/315135037 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f3670004600 con 0x7f36781076c0 2024-04-02T14:34:09.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.658+0000 7f366f400640 1 --2- 172.21.15.90:0/315135037 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f365c0a3c30 0x7f365c0a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:09.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.658+0000 7f3680183740 1 Processor -- start 2024-04-02T14:34:09.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.658+0000 7f366f400640 1 -- 172.21.15.90:0/315135037 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8248+0+0 (secure 0 0 0) 0x7f36700f2a20 con 0x7f36781076c0 2024-04-02T14:34:09.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.658+0000 7f3676800640 1 --2- 172.21.15.90:0/315135037 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f365c0a3c30 0x7f365c0a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:09.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.658+0000 7f3680183740 1 -- start start 2024-04-02T14:34:09.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.658+0000 7f3680183740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564662e2a320 0x564662e2a740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:09.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.658+0000 7f3680183740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564662e2ad10 0x564662e2f810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:09.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.658+0000 7f3680183740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564662e2fe80 0x564662e32310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:09.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.658+0000 7f3680183740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x564662d87060 con 0x564662e2a320 2024-04-02T14:34:09.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.658+0000 7f3680183740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x564662d2f560 con 0x564662e2ad10 2024-04-02T14:34:09.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.658+0000 7f3680183740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x564662d83010 con 0x564662e2fe80 2024-04-02T14:34:09.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.659+0000 7f3676800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564662e2a320 0x564662e2a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:09.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.659+0000 7f366fe00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564662e2ad10 0x564662e2f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:09.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.659+0000 7f366fe00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564662e2ad10 0x564662e2f810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:57300/0 (socket says 172.21.15.90:57300) 2024-04-02T14:34:09.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.659+0000 7f366fe00640 1 -- 172.21.15.90:0/3649077017 learned_addr learned my addr 172.21.15.90:0/3649077017 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:34:09.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.659+0000 7f3676800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564662e2a320 0x564662e2a740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:52364/0 (socket says 172.21.15.90:52364) 2024-04-02T14:34:09.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.659+0000 7f3676800640 1 --2- 172.21.15.90:0/315135037 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f365c0a3c30 0x7f365c0a60f0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f3668009bd0 tx=0x7f3668005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:34:09.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.659+0000 7f3677200640 1 --2- 172.21.15.90:0/3649077017 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564662e2fe80 0x564662e32310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:09.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.659+0000 7f366fe00640 1 -- 172.21.15.90:0/3649077017 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564662e2fe80 msgr2=0x564662e32310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:34:09.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.659+0000 7f366fe00640 1 --2- 172.21.15.90:0/3649077017 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564662e2fe80 0x564662e32310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:09.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.659+0000 7f366fe00640 1 -- 172.21.15.90:0/3649077017 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564662e2a320 msgr2=0x564662e2a740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:34:09.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.659+0000 7f366fe00640 1 --2- 172.21.15.90:0/3649077017 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564662e2a320 0x564662e2a740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:09.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.659+0000 7f366fe00640 1 -- 172.21.15.90:0/3649077017 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564662d51fd0 con 0x564662e2ad10 2024-04-02T14:34:09.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.659+0000 7f3677200640 1 --2- 172.21.15.90:0/3649077017 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564662e2fe80 0x564662e32310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:34:09.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.659+0000 7f366fe00640 1 --2- 172.21.15.90:0/3649077017 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564662e2ad10 0x564662e2f810 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f3660009c60 tx=0x7f3660011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dde1917510a6a08d server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:34:09.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.660+0000 7f3667400640 1 -- 172.21.15.90:0/3649077017 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3660004230 con 0x564662e2ad10 2024-04-02T14:34:09.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.660+0000 7f3667400640 1 -- 172.21.15.90:0/3649077017 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f36600043d0 con 0x564662e2ad10 2024-04-02T14:34:09.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.660+0000 7f3667400640 1 -- 172.21.15.90:0/3649077017 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3660019840 con 0x564662e2ad10 2024-04-02T14:34:09.666 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.661+0000 7f3680183740 1 -- 172.21.15.90:0/3649077017 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564662e2ad10 msgr2=0x564662e2f810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:09.666 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.661+0000 7f3680183740 1 --2- 172.21.15.90:0/3649077017 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564662e2ad10 0x564662e2f810 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f3660009c60 tx=0x7f3660011810 comp rx=0 tx=0).stop 2024-04-02T14:34:09.666 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.661+0000 7f3680183740 1 -- 172.21.15.90:0/3649077017 shutdown_connections 2024-04-02T14:34:09.666 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.661+0000 7f3680183740 1 --2- 172.21.15.90:0/3649077017 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564662e2fe80 0x564662e32310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:09.666 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.661+0000 7f3680183740 1 --2- 172.21.15.90:0/3649077017 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564662e2ad10 0x564662e2f810 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:09.666 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.661+0000 7f3680183740 1 --2- 172.21.15.90:0/3649077017 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564662e2a320 0x564662e2a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:09.666 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.661+0000 7f3680183740 1 -- 172.21.15.90:0/3649077017 >> 172.21.15.90:0/3649077017 conn(0x564662e1d080 msgr2=0x564662e1d480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:34:09.666 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.661+0000 7f3680183740 1 -- 172.21.15.90:0/3649077017 shutdown_connections 2024-04-02T14:34:09.666 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.661+0000 7f3680183740 1 -- 172.21.15.90:0/3649077017 wait complete. 2024-04-02T14:34:09.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.661+0000 7f3680183740 1 -- start start 2024-04-02T14:34:09.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.662+0000 7f3680183740 1 Processor -- start 2024-04-02T14:34:09.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.662+0000 7f3680183740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564662e2a320 0x564662e34840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:09.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.662+0000 7f3680183740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564662e2ad10 0x564662e34d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:09.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.662+0000 7f3680183740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564662e2fe80 0x564662e3bdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:09.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.662+0000 7f3680183740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x564662e32e30 con 0x564662e2a320 2024-04-02T14:34:09.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.662+0000 7f3680183740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x564662d42920 con 0x564662e2fe80 2024-04-02T14:34:09.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.662+0000 7f3680183740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x564662e32fb0 con 0x564662e2ad10 2024-04-02T14:34:09.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.662+0000 7f3680183740 10 client.0 _ll_register_callbacks cb 0x564662d42310 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:34:09.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.663+0000 7f3676800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564662e2a320 0x564662e34840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:09.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.663+0000 7f3676800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564662e2a320 0x564662e34840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:52370/0 (socket says 172.21.15.90:52370) 2024-04-02T14:34:09.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.663+0000 7f3676800640 1 -- 172.21.15.90:0/3936589215 learned_addr learned my addr 172.21.15.90:0/3936589215 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:34:09.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.663+0000 7f3677200640 1 --2- 172.21.15.90:0/3936589215 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564662e2fe80 0x564662e3bdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:09.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.663+0000 7f366fe00640 1 --2- 172.21.15.90:0/3936589215 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564662e2ad10 0x564662e34d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:09.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.663+0000 7f3676800640 1 -- 172.21.15.90:0/3936589215 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564662e2ad10 msgr2=0x564662e34d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:34:09.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.663+0000 7f3676800640 1 --2- 172.21.15.90:0/3936589215 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564662e2ad10 0x564662e34d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:09.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.663+0000 7f3676800640 1 -- 172.21.15.90:0/3936589215 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564662e2fe80 msgr2=0x564662e3bdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:34:09.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.663+0000 7f3676800640 1 --2- 172.21.15.90:0/3936589215 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564662e2fe80 0x564662e3bdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:09.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.663+0000 7f3676800640 1 -- 172.21.15.90:0/3936589215 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564662a9b2d0 con 0x564662e2a320 2024-04-02T14:34:09.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.663+0000 7f366fe00640 1 --2- 172.21.15.90:0/3936589215 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564662e2ad10 0x564662e34d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:34:09.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.663+0000 7f3677200640 1 --2- 172.21.15.90:0/3936589215 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564662e2fe80 0x564662e3bdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:34:09.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.663+0000 7f3676800640 1 --2- 172.21.15.90:0/3936589215 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564662e2a320 0x564662e34840 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f366800ed70 tx=0x7f3668004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58178bb1034b551f server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:34:09.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.663+0000 7f365be00640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-02T14:34:09.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.663+0000 7f365be00640 1 -- 172.21.15.90:0/3936589215 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f366801db20 con 0x564662e2a320 2024-04-02T14:34:09.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.663+0000 7f365be00640 1 -- 172.21.15.90:0/3936589215 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8248+0+0 (secure 0 0 0) 0x7f366801dcc0 con 0x564662e2a320 2024-04-02T14:34:09.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.663+0000 7f3680183740 1 -- 172.21.15.90:0/3936589215 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f36600043d0 con 0x564662e2a320 2024-04-02T14:34:09.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.664+0000 7f365be00640 1 -- 172.21.15.90:0/3936589215 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f3668004030 con 0x564662e2a320 2024-04-02T14:34:09.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.664+0000 7f3680183740 10 client.14922 fetch_fsmap learned FSMap version 26 2024-04-02T14:34:09.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.664+0000 7f3680183740 1 -- 172.21.15.90:0/3936589215 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x564662b656b0 con 0x564662e2a320 2024-04-02T14:34:09.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.664+0000 7f365be00640 1 -- 172.21.15.90:0/3936589215 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f36600043d0 con 0x564662e2a320 2024-04-02T14:34:09.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.664+0000 7f365be00640 1 -- 172.21.15.90:0/3936589215 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f3668026400 con 0x564662e2a320 2024-04-02T14:34:09.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.664+0000 7f3680183740 10 client.14922 fetch_fsmap finished waiting for FSMap version 26 2024-04-02T14:34:09.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.664+0000 7f3680183740 10 client.14922 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-02T14:34:09.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.664+0000 7f3680183740 20 client.14922 populate_metadata read hostname 'smithi090' 2024-04-02T14:34:09.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.664+0000 7f3680183740 1 --2- 172.21.15.90:0/3936589215 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x564662e416f0 0x564662e42b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:09.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.664+0000 7f3680183740 4 client.14922 mds_command: new command op to 24547 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:34:09.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.664+0000 7f3680183740 1 -- 172.21.15.90:0/3936589215 --> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564662d945a0 con 0x564662e416f0 2024-04-02T14:34:09.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.665+0000 7f366fe00640 1 --2- 172.21.15.90:0/3936589215 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x564662e416f0 0x564662e42b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:09.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.665+0000 7f366fe00640 1 --2- 172.21.15.90:0/3936589215 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x564662e416f0 0x564662e42b60 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=b82bb00b97b98fed server_cookie=29b7829676df7118 in_seq=0 out_seq=0 2024-04-02T14:34:09.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.665+0000 7f365be00640 10 client.14922 ms_handle_connect on v2:172.21.15.177:6838/664895663 2024-04-02T14:34:09.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.666+0000 7f365be00640 1 -- 172.21.15.90:0/3936589215 <== mds.2 v2:172.21.15.177:6838/664895663 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x564662d945a0 con 0x564662e416f0 2024-04-02T14:34:09.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.666+0000 7f365be00640 10 client.14922 handle_command_reply: tid=0 2024-04-02T14:34:09.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.666+0000 7f3680183740 1 client.14922 shutdown 2024-04-02T14:34:09.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.666+0000 7f3680183740 1 -- 172.21.15.90:0/3936589215 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564662e2a320 msgr2=0x564662e34840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:09.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.666+0000 7f3680183740 1 --2- 172.21.15.90:0/3936589215 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564662e2a320 0x564662e34840 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f366800ed70 tx=0x7f3668004730 comp rx=0 tx=0).stop 2024-04-02T14:34:09.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.666+0000 7f3680183740 1 -- 172.21.15.90:0/3936589215 shutdown_connections 2024-04-02T14:34:09.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.666+0000 7f3680183740 1 --2- 172.21.15.90:0/3936589215 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x564662e416f0 0x564662e42b60 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:09.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.666+0000 7f3680183740 1 --2- 172.21.15.90:0/3936589215 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564662e2fe80 0x564662e3bdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:09.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.666+0000 7f3680183740 1 --2- 172.21.15.90:0/3936589215 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564662e2ad10 0x564662e34d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:09.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.666+0000 7f3680183740 1 --2- 172.21.15.90:0/3936589215 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564662e2a320 0x564662e34840 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:09.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.666+0000 7f3680183740 1 -- 172.21.15.90:0/3936589215 >> 172.21.15.90:0/3936589215 conn(0x564662e1d080 msgr2=0x564662e1c1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:34:09.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.667+0000 7f365be00640 0 client.14922 ms_handle_reset on v2:172.21.15.177:6838/664895663 2024-04-02T14:34:09.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.667+0000 7f3680183740 1 -- 172.21.15.90:0/3936589215 shutdown_connections 2024-04-02T14:34:09.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.667+0000 7f3680183740 1 -- 172.21.15.90:0/3936589215 wait complete. 2024-04-02T14:34:09.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.667+0000 7f3680183740 20 client.14922 trim_cache size 0 max 16384 2024-04-02T14:34:09.695 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.693+0000 7f3680183740 1 Processor -- start 2024-04-02T14:34:09.695 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.693+0000 7f3680183740 1 -- start start 2024-04-02T14:34:09.695 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.693+0000 7f3680183740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564662e2a320 0x564662e17de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:09.695 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.693+0000 7f3680183740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564662e2ad10 0x564662e3bd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:09.695 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.693+0000 7f3680183740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564662e2fe80 0x564662e2c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:09.695 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.693+0000 7f3680183740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x564662e40ea0 con 0x564662e2fe80 2024-04-02T14:34:09.695 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.693+0000 7f3680183740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x564662e41570 con 0x564662e2ad10 2024-04-02T14:34:09.695 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.693+0000 7f3680183740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x564662e41c40 con 0x564662e2a320 2024-04-02T14:34:09.695 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.693+0000 7f3677200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564662e2fe80 0x564662e2c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:09.695 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.693+0000 7f3677200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564662e2fe80 0x564662e2c2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:52374/0 (socket says 172.21.15.90:52374) 2024-04-02T14:34:09.695 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.693+0000 7f3677200640 1 -- 172.21.15.90:0/1092817962 learned_addr learned my addr 172.21.15.90:0/1092817962 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:34:09.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.693+0000 7f3676800640 1 --2- 172.21.15.90:0/1092817962 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564662e2a320 0x564662e17de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:09.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.693+0000 7f366fe00640 1 --2- 172.21.15.90:0/1092817962 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564662e2ad10 0x564662e3bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:09.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.694+0000 7f3677200640 1 -- 172.21.15.90:0/1092817962 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564662e2a320 msgr2=0x564662e17de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:34:09.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.694+0000 7f3677200640 1 --2- 172.21.15.90:0/1092817962 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564662e2a320 0x564662e17de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:09.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.694+0000 7f3677200640 1 -- 172.21.15.90:0/1092817962 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564662e2ad10 msgr2=0x564662e3bd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:34:09.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.694+0000 7f3677200640 1 --2- 172.21.15.90:0/1092817962 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564662e2ad10 0x564662e3bd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:09.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.694+0000 7f3677200640 1 -- 172.21.15.90:0/1092817962 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564662e2f4f0 con 0x564662e2fe80 2024-04-02T14:34:09.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.694+0000 7f366fe00640 1 --2- 172.21.15.90:0/1092817962 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564662e2ad10 0x564662e3bd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:34:09.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.694+0000 7f3677200640 1 --2- 172.21.15.90:0/1092817962 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564662e2fe80 0x564662e2c2a0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f36700bec20 tx=0x7f367002c100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e31939ce0de2daa1 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:34:09.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.694+0000 7f3666a00640 1 -- 172.21.15.90:0/1092817962 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3670032890 con 0x564662e2fe80 2024-04-02T14:34:09.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.694+0000 7f3666a00640 1 -- 172.21.15.90:0/1092817962 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f36700dde50 con 0x564662e2fe80 2024-04-02T14:34:09.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.694+0000 7f3666a00640 1 -- 172.21.15.90:0/1092817962 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36700c7900 con 0x564662e2fe80 2024-04-02T14:34:09.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.695+0000 7f3680183740 1 -- 172.21.15.90:0/1092817962 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564662e2fe80 msgr2=0x564662e2c2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:09.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.695+0000 7f3680183740 1 --2- 172.21.15.90:0/1092817962 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564662e2fe80 0x564662e2c2a0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f36700bec20 tx=0x7f367002c100 comp rx=0 tx=0).stop 2024-04-02T14:34:09.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.695+0000 7f3680183740 1 -- 172.21.15.90:0/1092817962 shutdown_connections 2024-04-02T14:34:09.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.695+0000 7f3680183740 1 --2- 172.21.15.90:0/1092817962 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564662e2fe80 0x564662e2c2a0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:09.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.695+0000 7f3680183740 1 --2- 172.21.15.90:0/1092817962 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564662e2ad10 0x564662e3bd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:09.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.695+0000 7f3680183740 1 --2- 172.21.15.90:0/1092817962 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564662e2a320 0x564662e17de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:09.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.695+0000 7f3680183740 1 -- 172.21.15.90:0/1092817962 >> 172.21.15.90:0/1092817962 conn(0x564662e1d080 msgr2=0x564662e2b7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:34:09.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.695+0000 7f3680183740 1 -- 172.21.15.90:0/1092817962 shutdown_connections 2024-04-02T14:34:09.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.695+0000 7f3680183740 1 -- 172.21.15.90:0/1092817962 wait complete. 2024-04-02T14:34:09.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.696+0000 7f3680183740 1 -- start start 2024-04-02T14:34:09.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.696+0000 7f3680183740 1 Processor -- start 2024-04-02T14:34:09.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.696+0000 7f3680183740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564662e2a320 0x564662e34c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:09.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.696+0000 7f3680183740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564662e2ad10 0x564662e3bce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:09.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.696+0000 7f3680183740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564662e2fe80 0x564662ea5690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:09.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.696+0000 7f3680183740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x564662e429e0 con 0x564662e2ad10 2024-04-02T14:34:09.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.696+0000 7f3680183740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x564662e42310 con 0x564662e2fe80 2024-04-02T14:34:09.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.696+0000 7f3680183740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x564662e430b0 con 0x564662e2a320 2024-04-02T14:34:09.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.696+0000 7f3680183740 10 client.0 _ll_register_callbacks cb 0x7f366800e9f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:34:09.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.696+0000 7f366fe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564662e2ad10 0x564662e3bce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:09.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.696+0000 7f3677200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564662e2fe80 0x564662ea5690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:09.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.696+0000 7f3677200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564662e2fe80 0x564662ea5690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:57328/0 (socket says 172.21.15.90:57328) 2024-04-02T14:34:09.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.696+0000 7f3677200640 1 -- 172.21.15.90:0/1549358678 learned_addr learned my addr 172.21.15.90:0/1549358678 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:34:09.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.697+0000 7f3677200640 1 -- 172.21.15.90:0/1549358678 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564662e2a320 msgr2=0x564662e34c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:34:09.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.697+0000 7f3677200640 1 --2- 172.21.15.90:0/1549358678 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564662e2a320 0x564662e34c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:09.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.697+0000 7f3677200640 1 -- 172.21.15.90:0/1549358678 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564662e2ad10 msgr2=0x564662e3bce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:34:09.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.697+0000 7f3677200640 1 --2- 172.21.15.90:0/1549358678 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564662e2ad10 0x564662e3bce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:09.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.697+0000 7f3677200640 1 -- 172.21.15.90:0/1549358678 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564662ea5dd0 con 0x564662e2fe80 2024-04-02T14:34:09.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.697+0000 7f3677200640 1 --2- 172.21.15.90:0/1549358678 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564662e2fe80 0x564662ea5690 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f3670006010 tx=0x7f3670032340 comp rx=0 tx=0).ready entity=mon.2 client_cookie=55a1f638a1d17444 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:34:09.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.697+0000 7f365be00640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:34:09.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.697+0000 7f365be00640 1 -- 172.21.15.90:0/1549358678 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3670032890 con 0x564662e2fe80 2024-04-02T14:34:09.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.697+0000 7f3680183740 1 -- 172.21.15.90:0/1549358678 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f36700dde50 con 0x564662e2fe80 2024-04-02T14:34:09.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.697+0000 7f365be00640 1 -- 172.21.15.90:0/1549358678 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8248+0+0 (secure 0 0 0) 0x7f36700bcf90 con 0x564662e2fe80 2024-04-02T14:34:09.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.698+0000 7f365be00640 1 -- 172.21.15.90:0/1549358678 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f3670033bf0 con 0x564662e2fe80 2024-04-02T14:34:09.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.698+0000 7f3680183740 10 client.25037 fetch_fsmap learned FSMap version 26 2024-04-02T14:34:09.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.698+0000 7f3680183740 1 -- 172.21.15.90:0/1549358678 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x564662ea70d0 con 0x564662e2fe80 2024-04-02T14:34:09.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.698+0000 7f365be00640 1 -- 172.21.15.90:0/1549358678 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f36700dde50 con 0x564662e2fe80 2024-04-02T14:34:09.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.698+0000 7f365be00640 1 -- 172.21.15.90:0/1549358678 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f367003c030 con 0x564662e2fe80 2024-04-02T14:34:09.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.698+0000 7f3680183740 10 client.25037 fetch_fsmap finished waiting for FSMap version 26 2024-04-02T14:34:09.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.698+0000 7f3680183740 10 client.25037 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-02T14:34:09.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.698+0000 7f3680183740 20 client.25037 populate_metadata read hostname 'smithi090' 2024-04-02T14:34:09.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.698+0000 7f3680183740 1 --2- 172.21.15.90:0/1549358678 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x564662eaba30 0x564662eaa740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:09.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.698+0000 7f3680183740 4 client.25037 mds_command: new command op to 24547 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-02T14:34:09.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.698+0000 7f3680183740 1 -- 172.21.15.90:0/1549358678 --> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] -- command(tid 0: {"prefix": "status"}) -- 0x564662d8e8f0 con 0x564662eaba30 2024-04-02T14:34:09.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.698+0000 7f3676800640 1 --2- 172.21.15.90:0/1549358678 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x564662eaba30 0x564662eaa740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:09.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.699+0000 7f3676800640 1 --2- 172.21.15.90:0/1549358678 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x564662eaba30 0x564662eaa740 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=cd77012f4a06d51 server_cookie=321d7b0e127e72c0 in_seq=0 out_seq=0 2024-04-02T14:34:09.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.699+0000 7f365be00640 10 client.25037 ms_handle_connect on v2:172.21.15.177:6838/664895663 2024-04-02T14:34:09.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.699+0000 7f365be00640 1 -- 172.21.15.90:0/1549358678 <== mds.2 v2:172.21.15.177:6838/664895663 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x564662d8e8f0 con 0x564662eaba30 2024-04-02T14:34:09.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.699+0000 7f365be00640 10 client.25037 handle_command_reply: tid=0 2024-04-02T14:34:09.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.699+0000 7f3680183740 1 client.25037 shutdown 2024-04-02T14:34:09.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.699+0000 7f3680183740 1 -- 172.21.15.90:0/1549358678 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564662e2fe80 msgr2=0x564662ea5690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:09.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.699+0000 7f3680183740 1 --2- 172.21.15.90:0/1549358678 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564662e2fe80 0x564662ea5690 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f3670006010 tx=0x7f3670032340 comp rx=0 tx=0).stop 2024-04-02T14:34:09.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.699+0000 7f3680183740 1 -- 172.21.15.90:0/1549358678 shutdown_connections 2024-04-02T14:34:09.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.699+0000 7f3680183740 1 --2- 172.21.15.90:0/1549358678 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x564662eaba30 0x564662eaa740 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:09.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.699+0000 7f3680183740 1 --2- 172.21.15.90:0/1549358678 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564662e2fe80 0x564662ea5690 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:09.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.699+0000 7f3680183740 1 --2- 172.21.15.90:0/1549358678 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564662e2ad10 0x564662e3bce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:09.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.699+0000 7f365be00640 0 client.25037 ms_handle_reset on v2:172.21.15.177:6838/664895663 2024-04-02T14:34:09.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.699+0000 7f3680183740 1 --2- 172.21.15.90:0/1549358678 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564662e2a320 0x564662e34c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:09.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.699+0000 7f3680183740 1 -- 172.21.15.90:0/1549358678 >> 172.21.15.90:0/1549358678 conn(0x564662e1d080 msgr2=0x564662e17de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:34:09.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.699+0000 7f3680183740 1 -- 172.21.15.90:0/1549358678 shutdown_connections 2024-04-02T14:34:09.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.699+0000 7f3680183740 1 -- 172.21.15.90:0/1549358678 wait complete. 2024-04-02T14:34:09.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.700+0000 7f3680183740 20 client.25037 trim_cache size 0 max 16384 2024-04-02T14:34:09.705 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-02T14:34:09.705 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_fsid": "6449448a-f0fc-11ee-b647-cb9ed24678a4", 2024-04-02T14:34:09.705 INFO:teuthology.orchestra.run.smithi090.stdout: "whoami": 2, 2024-04-02T14:34:09.706 INFO:teuthology.orchestra.run.smithi090.stdout: "id": 24547, 2024-04-02T14:34:09.706 INFO:teuthology.orchestra.run.smithi090.stdout: "want_state": "up:active", 2024-04-02T14:34:09.706 INFO:teuthology.orchestra.run.smithi090.stdout: "state": "up:active", 2024-04-02T14:34:09.706 INFO:teuthology.orchestra.run.smithi090.stdout: "fs_name": "cephfs", 2024-04-02T14:34:09.706 INFO:teuthology.orchestra.run.smithi090.stdout: "rank_uptime": 102.77639323299999, 2024-04-02T14:34:09.706 INFO:teuthology.orchestra.run.smithi090.stdout: "mdsmap_epoch": 26, 2024-04-02T14:34:09.706 INFO:teuthology.orchestra.run.smithi090.stdout: "osdmap_epoch": 90, 2024-04-02T14:34:09.706 INFO:teuthology.orchestra.run.smithi090.stdout: "osdmap_epoch_barrier": 88, 2024-04-02T14:34:09.706 INFO:teuthology.orchestra.run.smithi090.stdout: "uptime": 123.489308248 2024-04-02T14:34:09.706 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-02T14:34:09.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.700+0000 7f367ca00640 1 -- 172.21.15.90:0/315135037 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f365c0a3c30 msgr2=0x7f365c0a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:09.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.700+0000 7f367ca00640 1 --2- 172.21.15.90:0/315135037 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f365c0a3c30 0x7f365c0a60f0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f3668009bd0 tx=0x7f3668005f90 comp rx=0 tx=0).stop 2024-04-02T14:34:09.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.700+0000 7f367ca00640 1 -- 172.21.15.90:0/315135037 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f36781076c0 msgr2=0x7f3678100950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:09.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.700+0000 7f367ca00640 1 --2- 172.21.15.90:0/315135037 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f36781076c0 0x7f3678100950 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f3670037040 tx=0x7f3670035d90 comp rx=0 tx=0).stop 2024-04-02T14:34:09.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.700+0000 7f367ca00640 1 -- 172.21.15.90:0/315135037 shutdown_connections 2024-04-02T14:34:09.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.700+0000 7f367ca00640 1 --2- 172.21.15.90:0/315135037 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f365c0a3c30 0x7f365c0a60f0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:09.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.700+0000 7f367ca00640 1 --2- 172.21.15.90:0/315135037 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f36781076c0 0x7f3678100950 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:09.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.700+0000 7f367ca00640 1 --2- 172.21.15.90:0/315135037 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3678104d90 0x7f3678100410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:09.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.700+0000 7f367ca00640 1 --2- 172.21.15.90:0/315135037 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3678102460 0x7f3678101dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:09.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.700+0000 7f367ca00640 1 -- 172.21.15.90:0/315135037 >> 172.21.15.90:0/315135037 conn(0x7f36780fbe70 msgr2=0x7f3678102da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:34:09.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.701+0000 7f367ca00640 1 -- 172.21.15.90:0/315135037 shutdown_connections 2024-04-02T14:34:09.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.701+0000 7f367ca00640 1 -- 172.21.15.90:0/315135037 wait complete. 2024-04-02T14:34:09.719 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.g status 2024-04-02T14:34:09.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.902+0000 7f5964200640 1 -- 172.21.15.90:0/1852864709 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f595c1003d0 msgr2=0x7f595c1007f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:09.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.902+0000 7f5964200640 1 --2- 172.21.15.90:0/1852864709 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f595c1003d0 0x7f595c1007f0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f5950009900 tx=0x7f595002bc30 comp rx=0 tx=0).stop 2024-04-02T14:34:09.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.902+0000 7f5964200640 1 -- 172.21.15.90:0/1852864709 shutdown_connections 2024-04-02T14:34:09.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.902+0000 7f5964200640 1 --2- 172.21.15.90:0/1852864709 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f595c1016d0 0x7f595c101b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:09.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.902+0000 7f5964200640 1 --2- 172.21.15.90:0/1852864709 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f595c100d30 0x7f595c101190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:09.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.902+0000 7f5964200640 1 --2- 172.21.15.90:0/1852864709 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f595c1003d0 0x7f595c1007f0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:09.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.902+0000 7f5964200640 1 -- 172.21.15.90:0/1852864709 >> 172.21.15.90:0/1852864709 conn(0x7f595c0fbf90 msgr2=0x7f595c0fe3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:34:09.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.902+0000 7f5964200640 1 -- 172.21.15.90:0/1852864709 shutdown_connections 2024-04-02T14:34:09.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.902+0000 7f5964200640 1 -- 172.21.15.90:0/1852864709 wait complete. 2024-04-02T14:34:09.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.903+0000 7f5964200640 1 Processor -- start 2024-04-02T14:34:09.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.903+0000 7f5964200640 1 -- start start 2024-04-02T14:34:09.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.903+0000 7f5964200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f595c1003d0 0x7f595c19a030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:09.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.903+0000 7f5964200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f595c100d30 0x7f595c19a570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:09.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.903+0000 7f5964200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f595c1016d0 0x7f595c1a15e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:09.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.903+0000 7f5964200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f595c10f960 con 0x7f595c1003d0 2024-04-02T14:34:09.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.903+0000 7f5964200640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f595c10f7e0 con 0x7f595c1016d0 2024-04-02T14:34:09.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.903+0000 7f5964200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f595c10fae0 con 0x7f595c100d30 2024-04-02T14:34:09.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.903+0000 7f5961800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f595c1003d0 0x7f595c19a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:09.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.904+0000 7f5961800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f595c1003d0 0x7f595c19a030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:52394/0 (socket says 172.21.15.90:52394) 2024-04-02T14:34:09.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.904+0000 7f5961800640 1 -- 172.21.15.90:0/291529999 learned_addr learned my addr 172.21.15.90:0/291529999 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:34:09.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.904+0000 7f5962200640 1 --2- 172.21.15.90:0/291529999 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f595c1016d0 0x7f595c1a15e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:09.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.904+0000 7f5960e00640 1 --2- 172.21.15.90:0/291529999 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f595c100d30 0x7f595c19a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:09.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.904+0000 7f5961800640 1 -- 172.21.15.90:0/291529999 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f595c100d30 msgr2=0x7f595c19a570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:09.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.904+0000 7f5961800640 1 --2- 172.21.15.90:0/291529999 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f595c100d30 0x7f595c19a570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:09.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.904+0000 7f5961800640 1 -- 172.21.15.90:0/291529999 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f595c1016d0 msgr2=0x7f595c1a15e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:09.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.904+0000 7f5961800640 1 --2- 172.21.15.90:0/291529999 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f595c1016d0 0x7f595c1a15e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:09.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.904+0000 7f5961800640 1 -- 172.21.15.90:0/291529999 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f595c1a1c50 con 0x7f595c1003d0 2024-04-02T14:34:09.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.904+0000 7f5962200640 1 --2- 172.21.15.90:0/291529999 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f595c1016d0 0x7f595c1a15e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:34:09.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.904+0000 7f5961800640 1 --2- 172.21.15.90:0/291529999 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f595c1003d0 0x7f595c19a030 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f5950009230 tx=0x7f595002c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:34:09.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.905+0000 7f595a000640 1 -- 172.21.15.90:0/291529999 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5950041070 con 0x7f595c1003d0 2024-04-02T14:34:09.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.905+0000 7f595a000640 1 -- 172.21.15.90:0/291529999 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5950004450 con 0x7f595c1003d0 2024-04-02T14:34:09.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.905+0000 7f595a000640 1 -- 172.21.15.90:0/291529999 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5950003e20 con 0x7f595c1003d0 2024-04-02T14:34:09.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.905+0000 7f5964200640 1 -- 172.21.15.90:0/291529999 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f595c1a1ee0 con 0x7f595c1003d0 2024-04-02T14:34:09.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.905+0000 7f5964200640 1 -- 172.21.15.90:0/291529999 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f595c1a22e0 con 0x7f595c1003d0 2024-04-02T14:34:09.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.908+0000 7f5966e46740 1 Processor -- start 2024-04-02T14:34:09.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.908+0000 7f5966e46740 1 -- start start 2024-04-02T14:34:09.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.908+0000 7f595a000640 1 -- 172.21.15.90:0/291529999 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f5950044070 con 0x7f595c1003d0 2024-04-02T14:34:09.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.908+0000 7f5966e46740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559ea031e8f0 0x559ea031ed10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:09.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.908+0000 7f5966e46740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559ea031f2e0 0x559ea0323de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:09.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.908+0000 7f5966e46740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559ea0324450 0x559ea03268e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:09.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.908+0000 7f5966e46740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x559ea0284060 con 0x559ea031e8f0 2024-04-02T14:34:09.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.908+0000 7f5966e46740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x559ea022c560 con 0x559ea0324450 2024-04-02T14:34:09.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.908+0000 7f5966e46740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x559ea0280010 con 0x559ea031f2e0 2024-04-02T14:34:09.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.908+0000 7f5962200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559ea031e8f0 0x559ea031ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:09.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.908+0000 7f5961800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559ea031f2e0 0x559ea0323de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:09.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.909+0000 7f5960e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559ea0324450 0x559ea03268e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:09.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.909+0000 7f5960e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559ea0324450 0x559ea03268e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:57352/0 (socket says 172.21.15.90:57352) 2024-04-02T14:34:09.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.909+0000 7f5960e00640 1 -- 172.21.15.90:0/433625207 learned_addr learned my addr 172.21.15.90:0/433625207 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:34:09.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.909+0000 7f5961800640 1 --2- 172.21.15.90:0/433625207 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559ea031f2e0 0x559ea0323de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:39526/0 (socket says 172.21.15.90:39526) 2024-04-02T14:34:09.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.909+0000 7f5960e00640 1 -- 172.21.15.90:0/433625207 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559ea031f2e0 msgr2=0x559ea0323de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:34:09.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.909+0000 7f5960e00640 1 --2- 172.21.15.90:0/433625207 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559ea031f2e0 0x559ea0323de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:09.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.909+0000 7f5960e00640 1 -- 172.21.15.90:0/433625207 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559ea031e8f0 msgr2=0x559ea031ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:34:09.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.909+0000 7f5960e00640 1 --2- 172.21.15.90:0/433625207 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559ea031e8f0 0x559ea031ed10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:09.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.909+0000 7f5960e00640 1 -- 172.21.15.90:0/433625207 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559ea024efd0 con 0x559ea0324450 2024-04-02T14:34:09.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.909+0000 7f5960e00640 1 --2- 172.21.15.90:0/433625207 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559ea0324450 0x559ea03268e0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f5948009ca0 tx=0x7f5948011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1d10110a9f5deda5 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:34:09.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.910+0000 7f595a000640 1 --2- 172.21.15.90:0/291529999 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f59440a3bb0 0x7f59440a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:09.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.910+0000 7f595a000640 1 -- 172.21.15.90:0/291529999 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8248+0+0 (secure 0 0 0) 0x7f59500f1f30 con 0x7f595c1003d0 2024-04-02T14:34:09.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.910+0000 7f5962200640 1 --2- 172.21.15.90:0/291529999 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f59440a3bb0 0x7f59440a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:09.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.911+0000 7f5962200640 1 --2- 172.21.15.90:0/291529999 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f59440a3bb0 0x7f59440a6070 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x559ea031fb40 tx=0x7f5954009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:34:09.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.913+0000 7f594ea00640 1 -- 172.21.15.90:0/433625207 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f594801e070 con 0x559ea0324450 2024-04-02T14:34:09.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.913+0000 7f594ea00640 1 -- 172.21.15.90:0/433625207 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5948019b10 con 0x559ea0324450 2024-04-02T14:34:09.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.913+0000 7f594ea00640 1 -- 172.21.15.90:0/433625207 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f594800eae0 con 0x559ea0324450 2024-04-02T14:34:09.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.914+0000 7f5966e46740 1 -- 172.21.15.90:0/433625207 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559ea0324450 msgr2=0x559ea03268e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:09.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.914+0000 7f5966e46740 1 --2- 172.21.15.90:0/433625207 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559ea0324450 0x559ea03268e0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f5948009ca0 tx=0x7f5948011810 comp rx=0 tx=0).stop 2024-04-02T14:34:09.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.914+0000 7f5966e46740 1 -- 172.21.15.90:0/433625207 shutdown_connections 2024-04-02T14:34:09.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.914+0000 7f5966e46740 1 --2- 172.21.15.90:0/433625207 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559ea0324450 0x559ea03268e0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:09.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.914+0000 7f5966e46740 1 --2- 172.21.15.90:0/433625207 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559ea031f2e0 0x559ea0323de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:09.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.914+0000 7f5966e46740 1 --2- 172.21.15.90:0/433625207 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559ea031e8f0 0x559ea031ed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:09.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.914+0000 7f5966e46740 1 -- 172.21.15.90:0/433625207 >> 172.21.15.90:0/433625207 conn(0x559ea0319e60 msgr2=0x559ea031a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:34:09.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.914+0000 7f5966e46740 1 -- 172.21.15.90:0/433625207 shutdown_connections 2024-04-02T14:34:09.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.914+0000 7f5966e46740 1 -- 172.21.15.90:0/433625207 wait complete. 2024-04-02T14:34:09.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.914+0000 7f5966e46740 1 -- start start 2024-04-02T14:34:09.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.915+0000 7f5966e46740 1 Processor -- start 2024-04-02T14:34:09.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.915+0000 7f5966e46740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559ea031e8f0 0x559ea0328df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:09.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.915+0000 7f5966e46740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559ea031f2e0 0x559ea0329330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:09.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.915+0000 7f5966e46740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559ea0324450 0x559ea0330370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:09.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.915+0000 7f5966e46740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x559ea0327400 con 0x559ea031f2e0 2024-04-02T14:34:09.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.915+0000 7f5966e46740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x559ea023f920 con 0x559ea0324450 2024-04-02T14:34:09.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.915+0000 7f5966e46740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x559ea0327580 con 0x559ea031e8f0 2024-04-02T14:34:09.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.915+0000 7f5966e46740 10 client.0 _ll_register_callbacks cb 0x559ea023f310 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:34:09.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.915+0000 7f5960e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559ea031f2e0 0x559ea0329330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:09.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.915+0000 7f5960e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559ea031f2e0 0x559ea0329330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:52406/0 (socket says 172.21.15.90:52406) 2024-04-02T14:34:09.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.915+0000 7f5960e00640 1 -- 172.21.15.90:0/1484061001 learned_addr learned my addr 172.21.15.90:0/1484061001 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:34:09.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.915+0000 7f5962200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559ea0324450 0x559ea0330370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:09.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.915+0000 7f5961800640 1 --2- 172.21.15.90:0/1484061001 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559ea031e8f0 0x559ea0328df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:09.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.916+0000 7f5960e00640 1 -- 172.21.15.90:0/1484061001 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559ea031e8f0 msgr2=0x559ea0328df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:34:09.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.916+0000 7f5960e00640 1 --2- 172.21.15.90:0/1484061001 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559ea031e8f0 0x559ea0328df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:09.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.916+0000 7f5960e00640 1 -- 172.21.15.90:0/1484061001 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559ea0324450 msgr2=0x559ea0330370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:34:09.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.916+0000 7f5960e00640 1 --2- 172.21.15.90:0/1484061001 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559ea0324450 0x559ea0330370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:09.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.916+0000 7f5960e00640 1 -- 172.21.15.90:0/1484061001 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559e9ff982d0 con 0x559ea031f2e0 2024-04-02T14:34:09.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.916+0000 7f5960e00640 1 --2- 172.21.15.90:0/1484061001 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559ea031f2e0 0x559ea0329330 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f5948005890 tx=0x7f5948019640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd217bd70e10861c server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:34:09.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.916+0000 7f5943400640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-02T14:34:09.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.916+0000 7f5943400640 1 -- 172.21.15.90:0/1484061001 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5948028020 con 0x559ea031f2e0 2024-04-02T14:34:09.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.916+0000 7f5943400640 1 -- 172.21.15.90:0/1484061001 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8248+0+0 (secure 0 0 0) 0x7f5948005050 con 0x559ea031f2e0 2024-04-02T14:34:09.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.916+0000 7f5966e46740 1 -- 172.21.15.90:0/1484061001 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5948019b10 con 0x559ea031f2e0 2024-04-02T14:34:09.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.917+0000 7f5943400640 1 -- 172.21.15.90:0/1484061001 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f5948033050 con 0x559ea031f2e0 2024-04-02T14:34:09.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.917+0000 7f5966e46740 10 client.14940 fetch_fsmap learned FSMap version 26 2024-04-02T14:34:09.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.917+0000 7f5966e46740 1 -- 172.21.15.90:0/1484061001 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x559ea00626b0 con 0x559ea031f2e0 2024-04-02T14:34:09.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.917+0000 7f5943400640 1 -- 172.21.15.90:0/1484061001 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5948019b10 con 0x559ea031f2e0 2024-04-02T14:34:09.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.917+0000 7f5943400640 1 -- 172.21.15.90:0/1484061001 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f59480046c0 con 0x559ea031f2e0 2024-04-02T14:34:09.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.917+0000 7f5966e46740 10 client.14940 fetch_fsmap finished waiting for FSMap version 26 2024-04-02T14:34:09.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.917+0000 7f5966e46740 10 client.14940 resolve_mds: resolved name 'g' to daemon mds.g 2024-04-02T14:34:09.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.917+0000 7f5966e46740 20 client.14940 populate_metadata read hostname 'smithi090' 2024-04-02T14:34:09.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.917+0000 7f5966e46740 1 --2- 172.21.15.90:0/1484061001 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x559ea03358a0 0x559ea0336d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:09.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.917+0000 7f5966e46740 4 client.14940 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:34:09.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.917+0000 7f5966e46740 1 -- 172.21.15.90:0/1484061001 --> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559ea02915a0 con 0x559ea03358a0 2024-04-02T14:34:09.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.918+0000 7f5961800640 1 --2- 172.21.15.90:0/1484061001 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x559ea03358a0 0x559ea0336d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:09.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.918+0000 7f5961800640 1 --2- 172.21.15.90:0/1484061001 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x559ea03358a0 0x559ea0336d10 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=5f9ffd7b02f6e0e3 server_cookie=88bc1962475ed420 in_seq=0 out_seq=0 2024-04-02T14:34:09.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.918+0000 7f5943400640 10 client.14940 ms_handle_connect on v2:172.21.15.90:6838/1498915870 2024-04-02T14:34:09.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.919+0000 7f5943400640 1 -- 172.21.15.90:0/1484061001 <== mds.1 v2:172.21.15.90:6838/1498915870 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x559ea02915a0 con 0x559ea03358a0 2024-04-02T14:34:09.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.919+0000 7f5943400640 10 client.14940 handle_command_reply: tid=0 2024-04-02T14:34:09.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.919+0000 7f5966e46740 1 client.14940 shutdown 2024-04-02T14:34:09.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.919+0000 7f5966e46740 1 -- 172.21.15.90:0/1484061001 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559ea031f2e0 msgr2=0x559ea0329330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:09.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.919+0000 7f5966e46740 1 --2- 172.21.15.90:0/1484061001 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559ea031f2e0 0x559ea0329330 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f5948005890 tx=0x7f5948019640 comp rx=0 tx=0).stop 2024-04-02T14:34:09.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.919+0000 7f5966e46740 1 -- 172.21.15.90:0/1484061001 shutdown_connections 2024-04-02T14:34:09.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.919+0000 7f5966e46740 1 --2- 172.21.15.90:0/1484061001 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x559ea03358a0 0x559ea0336d10 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:09.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.919+0000 7f5966e46740 1 --2- 172.21.15.90:0/1484061001 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559ea0324450 0x559ea0330370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:09.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.919+0000 7f5966e46740 1 --2- 172.21.15.90:0/1484061001 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559ea031f2e0 0x559ea0329330 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:09.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.919+0000 7f5966e46740 1 --2- 172.21.15.90:0/1484061001 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559ea031e8f0 0x559ea0328df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:09.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.919+0000 7f5966e46740 1 -- 172.21.15.90:0/1484061001 >> 172.21.15.90:0/1484061001 conn(0x559ea0319e60 msgr2=0x559ea0319080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:34:09.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.919+0000 7f5943400640 0 client.14940 ms_handle_reset on v2:172.21.15.90:6838/1498915870 2024-04-02T14:34:09.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.920+0000 7f5966e46740 1 -- 172.21.15.90:0/1484061001 shutdown_connections 2024-04-02T14:34:09.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.920+0000 7f5966e46740 1 -- 172.21.15.90:0/1484061001 wait complete. 2024-04-02T14:34:09.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.920+0000 7f5966e46740 20 client.14940 trim_cache size 0 max 16384 2024-04-02T14:34:09.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.960+0000 7f5966e46740 1 Processor -- start 2024-04-02T14:34:09.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.960+0000 7f5966e46740 1 -- start start 2024-04-02T14:34:09.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.961+0000 7f5966e46740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559ea031e8f0 0x559ea03272e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:09.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.961+0000 7f5966e46740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559ea031f2e0 0x559ea0330330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:09.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.961+0000 7f5966e46740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559ea0324450 0x559ea03348c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:09.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.961+0000 7f5966e46740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x559ea0335050 con 0x559ea031e8f0 2024-04-02T14:34:09.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.961+0000 7f5966e46740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x559ea0335720 con 0x559ea0324450 2024-04-02T14:34:09.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.961+0000 7f5966e46740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x559ea0335df0 con 0x559ea031f2e0 2024-04-02T14:34:09.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.962+0000 7f5961800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559ea031e8f0 0x559ea03272e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:09.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.962+0000 7f5960e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559ea031f2e0 0x559ea0330330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:09.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.962+0000 7f5960e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559ea031f2e0 0x559ea0330330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:39548/0 (socket says 172.21.15.90:39548) 2024-04-02T14:34:09.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.962+0000 7f5960e00640 1 -- 172.21.15.90:0/658156246 learned_addr learned my addr 172.21.15.90:0/658156246 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:34:09.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.962+0000 7f5961800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559ea031e8f0 0x559ea03272e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:52414/0 (socket says 172.21.15.90:52414) 2024-04-02T14:34:09.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.962+0000 7f5962200640 1 --2- 172.21.15.90:0/658156246 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559ea0324450 0x559ea03348c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:09.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.962+0000 7f5961800640 1 -- 172.21.15.90:0/658156246 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559ea031f2e0 msgr2=0x559ea0330330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:34:09.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.962+0000 7f5961800640 1 --2- 172.21.15.90:0/658156246 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559ea031f2e0 0x559ea0330330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:09.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.962+0000 7f5961800640 1 -- 172.21.15.90:0/658156246 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559ea0324450 msgr2=0x559ea03348c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:34:09.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.962+0000 7f5961800640 1 --2- 172.21.15.90:0/658156246 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559ea0324450 0x559ea03348c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:09.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.962+0000 7f5961800640 1 -- 172.21.15.90:0/658156246 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559ea0329f20 con 0x559ea031e8f0 2024-04-02T14:34:09.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.962+0000 7f5961800640 1 --2- 172.21.15.90:0/658156246 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559ea031e8f0 0x559ea03272e0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f59500efa70 tx=0x7f595002bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6bd506052daa034c server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:34:09.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.962+0000 7f594e000640 1 -- 172.21.15.90:0/658156246 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f595004b040 con 0x559ea031e8f0 2024-04-02T14:34:09.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.962+0000 7f594e000640 1 -- 172.21.15.90:0/658156246 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5950004c60 con 0x559ea031e8f0 2024-04-02T14:34:09.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.962+0000 7f594e000640 1 -- 172.21.15.90:0/658156246 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59500bc5b0 con 0x559ea031e8f0 2024-04-02T14:34:09.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.963+0000 7f5966e46740 1 -- 172.21.15.90:0/658156246 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559ea031e8f0 msgr2=0x559ea03272e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:09.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.963+0000 7f5966e46740 1 --2- 172.21.15.90:0/658156246 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559ea031e8f0 0x559ea03272e0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f59500efa70 tx=0x7f595002bc30 comp rx=0 tx=0).stop 2024-04-02T14:34:09.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.963+0000 7f5966e46740 1 -- 172.21.15.90:0/658156246 shutdown_connections 2024-04-02T14:34:09.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.963+0000 7f5966e46740 1 --2- 172.21.15.90:0/658156246 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559ea0324450 0x559ea03348c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:09.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.963+0000 7f5966e46740 1 --2- 172.21.15.90:0/658156246 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559ea031f2e0 0x559ea0330330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:09.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.963+0000 7f5966e46740 1 --2- 172.21.15.90:0/658156246 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559ea031e8f0 0x559ea03272e0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:09.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.963+0000 7f5966e46740 1 -- 172.21.15.90:0/658156246 >> 172.21.15.90:0/658156246 conn(0x559ea0319e60 msgr2=0x559ea031d950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:34:09.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.963+0000 7f5966e46740 1 -- 172.21.15.90:0/658156246 shutdown_connections 2024-04-02T14:34:09.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.963+0000 7f5966e46740 1 -- 172.21.15.90:0/658156246 wait complete. 2024-04-02T14:34:09.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.964+0000 7f5966e46740 1 -- start start 2024-04-02T14:34:09.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.964+0000 7f5966e46740 1 Processor -- start 2024-04-02T14:34:09.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.964+0000 7f5966e46740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559ea031e8f0 0x559ea032e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:09.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.964+0000 7f5966e46740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559ea031f2e0 0x559ea0397830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:09.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.964+0000 7f5966e46740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559ea0324450 0x559ea0399d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:09.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.964+0000 7f5966e46740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x559ea0336b90 con 0x559ea031e8f0 2024-04-02T14:34:09.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.964+0000 7f5966e46740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x559ea03364c0 con 0x559ea0324450 2024-04-02T14:34:09.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.964+0000 7f5966e46740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x559ea0337260 con 0x559ea031f2e0 2024-04-02T14:34:09.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.964+0000 7f5966e46740 10 client.0 _ll_register_callbacks cb 0x7f59500f58e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:34:09.972 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.965+0000 7f5961800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559ea031e8f0 0x559ea032e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:09.972 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.965+0000 7f5961800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559ea031e8f0 0x559ea032e280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:52422/0 (socket says 172.21.15.90:52422) 2024-04-02T14:34:09.972 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.965+0000 7f5961800640 1 -- 172.21.15.90:0/3870394550 learned_addr learned my addr 172.21.15.90:0/3870394550 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:34:09.972 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.965+0000 7f5962200640 1 --2- 172.21.15.90:0/3870394550 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559ea0324450 0x559ea0399d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:09.972 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.965+0000 7f5960e00640 1 --2- 172.21.15.90:0/3870394550 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559ea031f2e0 0x559ea0397830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:09.972 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.965+0000 7f5961800640 1 -- 172.21.15.90:0/3870394550 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559ea031f2e0 msgr2=0x559ea0397830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:34:09.972 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.965+0000 7f5961800640 1 --2- 172.21.15.90:0/3870394550 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559ea031f2e0 0x559ea0397830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:09.972 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.965+0000 7f5961800640 1 -- 172.21.15.90:0/3870394550 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559ea0324450 msgr2=0x559ea0399d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:34:09.972 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.965+0000 7f5961800640 1 --2- 172.21.15.90:0/3870394550 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559ea0324450 0x559ea0399d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:09.972 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.965+0000 7f5961800640 1 -- 172.21.15.90:0/3870394550 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559ea039a2c0 con 0x559ea031e8f0 2024-04-02T14:34:09.972 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.965+0000 7f5961800640 1 --2- 172.21.15.90:0/3870394550 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559ea031e8f0 0x559ea032e280 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f5950006010 tx=0x7f59500bde60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f35878721d38f2b3 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:34:09.972 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.965+0000 7f5943400640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-02T14:34:09.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.965+0000 7f5943400640 1 -- 172.21.15.90:0/3870394550 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f595004b070 con 0x559ea031e8f0 2024-04-02T14:34:09.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.965+0000 7f5943400640 1 -- 172.21.15.90:0/3870394550 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8248+0+0 (secure 0 0 0) 0x7f59500035f0 con 0x559ea031e8f0 2024-04-02T14:34:09.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.965+0000 7f5966e46740 1 -- 172.21.15.90:0/3870394550 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5950004c60 con 0x559ea031e8f0 2024-04-02T14:34:09.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.966+0000 7f5943400640 1 -- 172.21.15.90:0/3870394550 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f59500dc050 con 0x559ea031e8f0 2024-04-02T14:34:09.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.966+0000 7f5966e46740 10 client.14946 fetch_fsmap learned FSMap version 26 2024-04-02T14:34:09.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.966+0000 7f5966e46740 1 -- 172.21.15.90:0/3870394550 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x559ea039b430 con 0x559ea031e8f0 2024-04-02T14:34:09.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.966+0000 7f5943400640 1 -- 172.21.15.90:0/3870394550 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5950004c60 con 0x559ea031e8f0 2024-04-02T14:34:09.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.966+0000 7f5943400640 1 -- 172.21.15.90:0/3870394550 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f59500c69e0 con 0x559ea031e8f0 2024-04-02T14:34:09.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.966+0000 7f5966e46740 10 client.14946 fetch_fsmap finished waiting for FSMap version 26 2024-04-02T14:34:09.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.966+0000 7f5966e46740 10 client.14946 resolve_mds: resolved name 'g' to daemon mds.g 2024-04-02T14:34:09.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.966+0000 7f5966e46740 20 client.14946 populate_metadata read hostname 'smithi090' 2024-04-02T14:34:09.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.966+0000 7f5966e46740 1 --2- 172.21.15.90:0/3870394550 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x559ea039fdb0 0x559ea039c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:09.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.966+0000 7f5966e46740 4 client.14946 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-02T14:34:09.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.966+0000 7f5966e46740 1 -- 172.21.15.90:0/3870394550 --> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] -- command(tid 0: {"prefix": "status"}) -- 0x559ea028b8f0 con 0x559ea039fdb0 2024-04-02T14:34:09.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.966+0000 7f5960e00640 1 --2- 172.21.15.90:0/3870394550 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x559ea039fdb0 0x559ea039c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:09.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.967+0000 7f5960e00640 1 --2- 172.21.15.90:0/3870394550 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x559ea039fdb0 0x559ea039c8c0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=d250114343fdf03b server_cookie=c0f2fd8c9dff8685 in_seq=0 out_seq=0 2024-04-02T14:34:09.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.967+0000 7f5943400640 10 client.14946 ms_handle_connect on v2:172.21.15.90:6838/1498915870 2024-04-02T14:34:09.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.967+0000 7f5943400640 1 -- 172.21.15.90:0/3870394550 <== mds.1 v2:172.21.15.90:6838/1498915870 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x559ea028b8f0 con 0x559ea039fdb0 2024-04-02T14:34:09.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.967+0000 7f5943400640 10 client.14946 handle_command_reply: tid=0 2024-04-02T14:34:09.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.967+0000 7f5966e46740 1 client.14946 shutdown 2024-04-02T14:34:09.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.967+0000 7f5966e46740 1 -- 172.21.15.90:0/3870394550 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559ea031e8f0 msgr2=0x559ea032e280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:09.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.967+0000 7f5966e46740 1 --2- 172.21.15.90:0/3870394550 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559ea031e8f0 0x559ea032e280 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f5950006010 tx=0x7f59500bde60 comp rx=0 tx=0).stop 2024-04-02T14:34:09.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.967+0000 7f5966e46740 1 -- 172.21.15.90:0/3870394550 shutdown_connections 2024-04-02T14:34:09.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.967+0000 7f5966e46740 1 --2- 172.21.15.90:0/3870394550 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x559ea039fdb0 0x559ea039c8c0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:09.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.967+0000 7f5966e46740 1 --2- 172.21.15.90:0/3870394550 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559ea0324450 0x559ea0399d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:09.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.967+0000 7f5966e46740 1 --2- 172.21.15.90:0/3870394550 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559ea031f2e0 0x559ea0397830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:09.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.967+0000 7f5966e46740 1 --2- 172.21.15.90:0/3870394550 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559ea031e8f0 0x559ea032e280 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:09.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.967+0000 7f5943400640 0 client.14946 ms_handle_reset on v2:172.21.15.90:6838/1498915870 2024-04-02T14:34:09.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.967+0000 7f5966e46740 1 -- 172.21.15.90:0/3870394550 >> 172.21.15.90:0/3870394550 conn(0x559ea0319e60 msgr2=0x559ea031d0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:34:09.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.968+0000 7f5966e46740 1 -- 172.21.15.90:0/3870394550 shutdown_connections 2024-04-02T14:34:09.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.968+0000 7f5966e46740 1 -- 172.21.15.90:0/3870394550 wait complete. 2024-04-02T14:34:09.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.968+0000 7f5966e46740 20 client.14946 trim_cache size 0 max 16384 2024-04-02T14:34:09.975 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-02T14:34:09.976 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_fsid": "6449448a-f0fc-11ee-b647-cb9ed24678a4", 2024-04-02T14:34:09.976 INFO:teuthology.orchestra.run.smithi090.stdout: "whoami": 1, 2024-04-02T14:34:09.976 INFO:teuthology.orchestra.run.smithi090.stdout: "id": 24560, 2024-04-02T14:34:09.976 INFO:teuthology.orchestra.run.smithi090.stdout: "want_state": "up:active", 2024-04-02T14:34:09.976 INFO:teuthology.orchestra.run.smithi090.stdout: "state": "up:active", 2024-04-02T14:34:09.976 INFO:teuthology.orchestra.run.smithi090.stdout: "fs_name": "cephfs", 2024-04-02T14:34:09.976 INFO:teuthology.orchestra.run.smithi090.stdout: "rank_uptime": 104.05528913800001, 2024-04-02T14:34:09.976 INFO:teuthology.orchestra.run.smithi090.stdout: "mdsmap_epoch": 26, 2024-04-02T14:34:09.976 INFO:teuthology.orchestra.run.smithi090.stdout: "osdmap_epoch": 90, 2024-04-02T14:34:09.976 INFO:teuthology.orchestra.run.smithi090.stdout: "osdmap_epoch_barrier": 88, 2024-04-02T14:34:09.976 INFO:teuthology.orchestra.run.smithi090.stdout: "uptime": 119.777501534 2024-04-02T14:34:09.976 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-02T14:34:09.977 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.968+0000 7f5964200640 1 -- 172.21.15.90:0/291529999 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f59440a3bb0 msgr2=0x7f59440a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:09.977 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.968+0000 7f5964200640 1 --2- 172.21.15.90:0/291529999 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f59440a3bb0 0x7f59440a6070 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x559ea031fb40 tx=0x7f5954009500 comp rx=0 tx=0).stop 2024-04-02T14:34:09.977 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.968+0000 7f5964200640 1 -- 172.21.15.90:0/291529999 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f595c1003d0 msgr2=0x7f595c19a030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:09.977 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.968+0000 7f5964200640 1 --2- 172.21.15.90:0/291529999 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f595c1003d0 0x7f595c19a030 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f5950009230 tx=0x7f595002c650 comp rx=0 tx=0).stop 2024-04-02T14:34:09.978 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.968+0000 7f5964200640 1 -- 172.21.15.90:0/291529999 shutdown_connections 2024-04-02T14:34:09.978 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.968+0000 7f5964200640 1 --2- 172.21.15.90:0/291529999 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f59440a3bb0 0x7f59440a6070 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:09.978 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.968+0000 7f5964200640 1 --2- 172.21.15.90:0/291529999 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f595c1016d0 0x7f595c1a15e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:09.978 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.969+0000 7f5964200640 1 --2- 172.21.15.90:0/291529999 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f595c100d30 0x7f595c19a570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:09.978 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.969+0000 7f5964200640 1 --2- 172.21.15.90:0/291529999 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f595c1003d0 0x7f595c19a030 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:09.978 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.969+0000 7f5964200640 1 -- 172.21.15.90:0/291529999 >> 172.21.15.90:0/291529999 conn(0x7f595c0fbf90 msgr2=0x7f595c0fd750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:34:09.978 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.969+0000 7f5964200640 1 -- 172.21.15.90:0/291529999 shutdown_connections 2024-04-02T14:34:09.978 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:09.969+0000 7f5964200640 1 -- 172.21.15.90:0/291529999 wait complete. 2024-04-02T14:34:09.984 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:34:09 smithi090 ceph-mon[31674]: pgmap v367: 97 pgs: 97 active+clean; 644 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-02T14:34:09.984 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:34:09 smithi090 ceph-mon[31674]: from='client.? 172.21.15.90:0/559140841' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-02T14:34:09.988 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-04-02T14:34:10.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.184+0000 7f23d1000640 1 -- 172.21.15.90:0/306516668 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f23cc104ed0 msgr2=0x7f23cc1072c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:10.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.184+0000 7f23d1000640 1 --2- 172.21.15.90:0/306516668 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f23cc104ed0 0x7f23cc1072c0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f23b4009900 tx=0x7f23b402bc30 comp rx=0 tx=0).stop 2024-04-02T14:34:10.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.184+0000 7f23d1000640 1 -- 172.21.15.90:0/306516668 shutdown_connections 2024-04-02T14:34:10.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.184+0000 7f23d1000640 1 --2- 172.21.15.90:0/306516668 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f23cc107800 0x7f23cc109c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:10.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.184+0000 7f23d1000640 1 --2- 172.21.15.90:0/306516668 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f23cc104ed0 0x7f23cc1072c0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:10.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.184+0000 7f23d1000640 1 --2- 172.21.15.90:0/306516668 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f23cc1025a0 0x7f23cc104990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:10.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.184+0000 7f23d1000640 1 -- 172.21.15.90:0/306516668 >> 172.21.15.90:0/306516668 conn(0x7f23cc0fbf90 msgr2=0x7f23cc0fe3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:34:10.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.184+0000 7f23d1000640 1 -- 172.21.15.90:0/306516668 shutdown_connections 2024-04-02T14:34:10.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.184+0000 7f23d1000640 1 -- 172.21.15.90:0/306516668 wait complete. 2024-04-02T14:34:10.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:34:09 smithi177 ceph-mon[36431]: pgmap v367: 97 pgs: 97 active+clean; 644 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-02T14:34:10.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:34:09 smithi177 ceph-mon[36431]: from='client.? 172.21.15.90:0/559140841' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-02T14:34:10.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.185+0000 7f23d1000640 1 Processor -- start 2024-04-02T14:34:10.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.185+0000 7f23d1000640 1 -- start start 2024-04-02T14:34:10.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.186+0000 7f23d1000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f23cc1025a0 0x7f23cc101ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:10.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.186+0000 7f23d1000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f23cc104ed0 0x7f23cc100550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:10.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.186+0000 7f23d1000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f23cc107800 0x7f23cc100a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:10.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.186+0000 7f23d1000640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f23cc10d7b0 con 0x7f23cc107800 2024-04-02T14:34:10.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.186+0000 7f23d1000640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f23cc10d630 con 0x7f23cc104ed0 2024-04-02T14:34:10.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.186+0000 7f23d1000640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f23cc10d930 con 0x7f23cc1025a0 2024-04-02T14:34:10.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.186+0000 7f23ca800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f23cc107800 0x7f23cc100a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:10.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.186+0000 7f23c9e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f23cc1025a0 0x7f23cc101ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:10.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.186+0000 7f23c9e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f23cc1025a0 0x7f23cc101ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:39580/0 (socket says 172.21.15.90:39580) 2024-04-02T14:34:10.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.186+0000 7f23ca800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f23cc107800 0x7f23cc100a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:52436/0 (socket says 172.21.15.90:52436) 2024-04-02T14:34:10.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.186+0000 7f23c9e00640 1 -- 172.21.15.90:0/1990035966 learned_addr learned my addr 172.21.15.90:0/1990035966 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:34:10.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.186+0000 7f23c3e00640 1 --2- 172.21.15.90:0/1990035966 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f23cc104ed0 0x7f23cc100550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:10.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.186+0000 7f23ca800640 1 -- 172.21.15.90:0/1990035966 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f23cc1025a0 msgr2=0x7f23cc101ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:10.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.186+0000 7f23ca800640 1 --2- 172.21.15.90:0/1990035966 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f23cc1025a0 0x7f23cc101ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:10.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.186+0000 7f23ca800640 1 -- 172.21.15.90:0/1990035966 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f23cc104ed0 msgr2=0x7f23cc100550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:10.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.186+0000 7f23ca800640 1 --2- 172.21.15.90:0/1990035966 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f23cc104ed0 0x7f23cc100550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:10.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.186+0000 7f23ca800640 1 -- 172.21.15.90:0/1990035966 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23cc101350 con 0x7f23cc107800 2024-04-02T14:34:10.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.186+0000 7f23c3e00640 1 --2- 172.21.15.90:0/1990035966 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f23cc104ed0 0x7f23cc100550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:34:10.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.187+0000 7f23ca800640 1 --2- 172.21.15.90:0/1990035966 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f23cc107800 0x7f23cc100a90 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f23c400da10 tx=0x7f23c400ded0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:34:10.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.187+0000 7f23c2a00640 1 -- 172.21.15.90:0/1990035966 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23c400bd70 con 0x7f23cc107800 2024-04-02T14:34:10.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.187+0000 7f23c2a00640 1 -- 172.21.15.90:0/1990035966 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f23c4003e50 con 0x7f23cc107800 2024-04-02T14:34:10.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.187+0000 7f23d1000640 1 -- 172.21.15.90:0/1990035966 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f23cc1a2070 con 0x7f23cc107800 2024-04-02T14:34:10.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.187+0000 7f23c2a00640 1 -- 172.21.15.90:0/1990035966 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23c40106a0 con 0x7f23cc107800 2024-04-02T14:34:10.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.187+0000 7f23d1000640 1 -- 172.21.15.90:0/1990035966 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f23cc1a2540 con 0x7f23cc107800 2024-04-02T14:34:10.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.191+0000 7f23d3d6c740 1 Processor -- start 2024-04-02T14:34:10.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.192+0000 7f23c2a00640 1 -- 172.21.15.90:0/1990035966 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f23c4003970 con 0x7f23cc107800 2024-04-02T14:34:10.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.192+0000 7f23d3d6c740 1 -- start start 2024-04-02T14:34:10.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.192+0000 7f23d3d6c740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a033c2d8f0 0x55a033c2dd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:10.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.192+0000 7f23d3d6c740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a033c2e2e0 0x55a033c32de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:10.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.192+0000 7f23d3d6c740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a033c33450 0x55a033c358e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:10.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.192+0000 7f23d3d6c740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55a033b93060 con 0x55a033c2d8f0 2024-04-02T14:34:10.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.192+0000 7f23d3d6c740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55a033b3b560 con 0x55a033c33450 2024-04-02T14:34:10.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.192+0000 7f23d3d6c740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55a033b8f010 con 0x55a033c2e2e0 2024-04-02T14:34:10.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.192+0000 7f23c3e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a033c2d8f0 0x55a033c2dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:10.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.192+0000 7f23c3e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a033c2d8f0 0x55a033c2dd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:52440/0 (socket says 172.21.15.90:52440) 2024-04-02T14:34:10.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.192+0000 7f23c3e00640 1 -- 172.21.15.90:0/2537600179 learned_addr learned my addr 172.21.15.90:0/2537600179 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:34:10.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.193+0000 7f23c9e00640 1 --2- 172.21.15.90:0/2537600179 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a033c2e2e0 0x55a033c32de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:10.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.193+0000 7f23c3e00640 1 --2- 172.21.15.90:0/2537600179 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a033c33450 0x55a033c358e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:10.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.193+0000 7f23c3e00640 1 -- 172.21.15.90:0/2537600179 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a033c2e2e0 msgr2=0x55a033c32de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:34:10.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.193+0000 7f23c3e00640 1 --2- 172.21.15.90:0/2537600179 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a033c2e2e0 0x55a033c32de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:10.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.193+0000 7f23c3e00640 1 -- 172.21.15.90:0/2537600179 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a033c33450 msgr2=0x55a033c358e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:34:10.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.193+0000 7f23c3e00640 1 --2- 172.21.15.90:0/2537600179 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a033c33450 0x55a033c358e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:10.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.193+0000 7f23c3e00640 1 -- 172.21.15.90:0/2537600179 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a033b5dfd0 con 0x55a033c2d8f0 2024-04-02T14:34:10.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.193+0000 7f23c3e00640 1 --2- 172.21.15.90:0/2537600179 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a033c2d8f0 0x55a033c2dd10 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f23b402c8e0 tx=0x7f23b4005b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c625a7b38bb6b020 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:34:10.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.193+0000 7f23c2a00640 1 --2- 172.21.15.90:0/1990035966 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f23b00a3bb0 0x7f23b00a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:10.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.193+0000 7f23c2a00640 1 -- 172.21.15.90:0/1990035966 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8248+0+0 (secure 0 0 0) 0x7f23c40d0830 con 0x7f23cc107800 2024-04-02T14:34:10.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.193+0000 7f23bb400640 1 -- 172.21.15.90:0/2537600179 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23b4004020 con 0x55a033c2d8f0 2024-04-02T14:34:10.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.193+0000 7f23bb400640 1 -- 172.21.15.90:0/2537600179 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f23b4005520 con 0x55a033c2d8f0 2024-04-02T14:34:10.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.193+0000 7f23bb400640 1 -- 172.21.15.90:0/2537600179 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23b4004ba0 con 0x55a033c2d8f0 2024-04-02T14:34:10.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.194+0000 7f23c9e00640 1 --2- 172.21.15.90:0/1990035966 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f23b00a3bb0 0x7f23b00a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:10.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.194+0000 7f23d3d6c740 1 -- 172.21.15.90:0/2537600179 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a033c2d8f0 msgr2=0x55a033c2dd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:10.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.194+0000 7f23d3d6c740 1 --2- 172.21.15.90:0/2537600179 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a033c2d8f0 0x55a033c2dd10 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f23b402c8e0 tx=0x7f23b4005b50 comp rx=0 tx=0).stop 2024-04-02T14:34:10.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.194+0000 7f23c9e00640 1 --2- 172.21.15.90:0/1990035966 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f23b00a3bb0 0x7f23b00a6070 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x55a033c2e6e0 tx=0x7f23bc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:34:10.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.194+0000 7f23d3d6c740 1 -- 172.21.15.90:0/2537600179 shutdown_connections 2024-04-02T14:34:10.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.194+0000 7f23d3d6c740 1 --2- 172.21.15.90:0/2537600179 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a033c33450 0x55a033c358e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:10.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.194+0000 7f23d3d6c740 1 --2- 172.21.15.90:0/2537600179 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a033c2e2e0 0x55a033c32de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:10.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.194+0000 7f23d3d6c740 1 --2- 172.21.15.90:0/2537600179 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a033c2d8f0 0x55a033c2dd10 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:10.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.194+0000 7f23d3d6c740 1 -- 172.21.15.90:0/2537600179 >> 172.21.15.90:0/2537600179 conn(0x55a033c28e60 msgr2=0x55a033c29260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:34:10.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.195+0000 7f23d3d6c740 1 -- 172.21.15.90:0/2537600179 shutdown_connections 2024-04-02T14:34:10.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.195+0000 7f23d3d6c740 1 -- 172.21.15.90:0/2537600179 wait complete. 2024-04-02T14:34:10.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.195+0000 7f23d3d6c740 1 -- start start 2024-04-02T14:34:10.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.195+0000 7f23d3d6c740 1 Processor -- start 2024-04-02T14:34:10.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.195+0000 7f23d3d6c740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a033c2d8f0 0x55a033c37e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:10.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.196+0000 7f23d3d6c740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a033c2e2e0 0x55a033c38370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:10.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.196+0000 7f23d3d6c740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a033c33450 0x55a033c3f340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:10.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.196+0000 7f23d3d6c740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55a033c36340 con 0x55a033c2d8f0 2024-04-02T14:34:10.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.196+0000 7f23d3d6c740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55a033b4e920 con 0x55a033c2e2e0 2024-04-02T14:34:10.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.196+0000 7f23d3d6c740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55a033c364c0 con 0x55a033c33450 2024-04-02T14:34:10.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.196+0000 7f23d3d6c740 10 client.0 _ll_register_callbacks cb 0x55a033b4e310 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:34:10.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.196+0000 7f23c9e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a033c2d8f0 0x55a033c37e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:10.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.196+0000 7f23c9e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a033c2d8f0 0x55a033c37e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:52444/0 (socket says 172.21.15.90:52444) 2024-04-02T14:34:10.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.196+0000 7f23c9e00640 1 -- 172.21.15.90:0/3315307690 learned_addr learned my addr 172.21.15.90:0/3315307690 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:34:10.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.196+0000 7f23c3e00640 1 --2- 172.21.15.90:0/3315307690 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a033c2e2e0 0x55a033c38370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:10.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.196+0000 7f23ca800640 1 --2- 172.21.15.90:0/3315307690 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a033c33450 0x55a033c3f340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:10.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.196+0000 7f23c9e00640 1 -- 172.21.15.90:0/3315307690 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a033c33450 msgr2=0x55a033c3f340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:34:10.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.196+0000 7f23c9e00640 1 --2- 172.21.15.90:0/3315307690 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a033c33450 0x55a033c3f340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:10.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.196+0000 7f23c9e00640 1 -- 172.21.15.90:0/3315307690 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a033c2e2e0 msgr2=0x55a033c38370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:34:10.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.196+0000 7f23c9e00640 1 --2- 172.21.15.90:0/3315307690 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a033c2e2e0 0x55a033c38370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:10.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.196+0000 7f23c9e00640 1 -- 172.21.15.90:0/3315307690 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a0338a72d0 con 0x55a033c2d8f0 2024-04-02T14:34:10.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.197+0000 7f23c9e00640 1 --2- 172.21.15.90:0/3315307690 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a033c2d8f0 0x55a033c37e30 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f23bc009610 tx=0x7f23bc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2be02b606502f13 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:34:10.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.197+0000 7f23afe00640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-02T14:34:10.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.197+0000 7f23afe00640 1 -- 172.21.15.90:0/3315307690 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23bc02e060 con 0x55a033c2d8f0 2024-04-02T14:34:10.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.197+0000 7f23afe00640 1 -- 172.21.15.90:0/3315307690 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8248+0+0 (secure 0 0 0) 0x7f23bc018030 con 0x55a033c2d8f0 2024-04-02T14:34:10.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.197+0000 7f23d3d6c740 1 -- 172.21.15.90:0/3315307690 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f23b4005520 con 0x55a033c2d8f0 2024-04-02T14:34:10.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.197+0000 7f23afe00640 1 -- 172.21.15.90:0/3315307690 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f23bc00f480 con 0x55a033c2d8f0 2024-04-02T14:34:10.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.197+0000 7f23d3d6c740 10 client.14958 fetch_fsmap learned FSMap version 26 2024-04-02T14:34:10.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.197+0000 7f23d3d6c740 1 -- 172.21.15.90:0/3315307690 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55a0339716b0 con 0x55a033c2d8f0 2024-04-02T14:34:10.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.197+0000 7f23afe00640 1 -- 172.21.15.90:0/3315307690 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f23b4005520 con 0x55a033c2d8f0 2024-04-02T14:34:10.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.198+0000 7f23afe00640 1 -- 172.21.15.90:0/3315307690 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f23bc0348b0 con 0x55a033c2d8f0 2024-04-02T14:34:10.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.198+0000 7f23d3d6c740 10 client.14958 fetch_fsmap finished waiting for FSMap version 26 2024-04-02T14:34:10.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.198+0000 7f23d3d6c740 10 client.14958 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-02T14:34:10.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.198+0000 7f23d3d6c740 20 client.14958 populate_metadata read hostname 'smithi090' 2024-04-02T14:34:10.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.198+0000 7f23d3d6c740 1 --2- 172.21.15.90:0/3315307690 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55a033c44c10 0x55a033c46080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:10.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.198+0000 7f23d3d6c740 4 client.14958 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:34:10.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.198+0000 7f23d3d6c740 1 -- 172.21.15.90:0/3315307690 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a033ba05a0 con 0x55a033c44c10 2024-04-02T14:34:10.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.198+0000 7f23c3e00640 1 --2- 172.21.15.90:0/3315307690 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55a033c44c10 0x55a033c46080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:10.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.199+0000 7f23c3e00640 1 --2- 172.21.15.90:0/3315307690 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55a033c44c10 0x55a033c46080 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=65f3fe4353a57930 server_cookie=b3183a252ac48297 in_seq=0 out_seq=0 2024-04-02T14:34:10.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.199+0000 7f23afe00640 10 client.14958 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:34:10.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.199+0000 7f23afe00640 1 -- 172.21.15.90:0/3315307690 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a033ba05a0 con 0x55a033c44c10 2024-04-02T14:34:10.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.199+0000 7f23afe00640 10 client.14958 handle_command_reply: tid=0 2024-04-02T14:34:10.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.200+0000 7f23d3d6c740 1 client.14958 shutdown 2024-04-02T14:34:10.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.200+0000 7f23d3d6c740 1 -- 172.21.15.90:0/3315307690 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a033c2d8f0 msgr2=0x55a033c37e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:10.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.200+0000 7f23d3d6c740 1 --2- 172.21.15.90:0/3315307690 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a033c2d8f0 0x55a033c37e30 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f23bc009610 tx=0x7f23bc004730 comp rx=0 tx=0).stop 2024-04-02T14:34:10.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.200+0000 7f23d3d6c740 1 -- 172.21.15.90:0/3315307690 shutdown_connections 2024-04-02T14:34:10.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.200+0000 7f23d3d6c740 1 --2- 172.21.15.90:0/3315307690 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55a033c44c10 0x55a033c46080 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:10.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.200+0000 7f23d3d6c740 1 --2- 172.21.15.90:0/3315307690 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a033c33450 0x55a033c3f340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:10.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.200+0000 7f23d3d6c740 1 --2- 172.21.15.90:0/3315307690 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a033c2e2e0 0x55a033c38370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:10.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.200+0000 7f23d3d6c740 1 --2- 172.21.15.90:0/3315307690 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a033c2d8f0 0x55a033c37e30 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:10.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.200+0000 7f23afe00640 0 client.14958 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:34:10.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.200+0000 7f23d3d6c740 1 -- 172.21.15.90:0/3315307690 >> 172.21.15.90:0/3315307690 conn(0x55a033c28e60 msgr2=0x55a033c28080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:34:10.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.200+0000 7f23d3d6c740 1 -- 172.21.15.90:0/3315307690 shutdown_connections 2024-04-02T14:34:10.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.200+0000 7f23d3d6c740 1 -- 172.21.15.90:0/3315307690 wait complete. 2024-04-02T14:34:10.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.200+0000 7f23d3d6c740 20 client.14958 trim_cache size 0 max 16384 2024-04-02T14:34:10.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.219+0000 7f23d3d6c740 1 Processor -- start 2024-04-02T14:34:10.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.219+0000 7f23d3d6c740 1 -- start start 2024-04-02T14:34:10.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.219+0000 7f23d3d6c740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a033c2d8f0 0x55a033c2ece0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:10.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.219+0000 7f23d3d6c740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a033c2e2e0 0x55a033c3f300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:10.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.220+0000 7f23d3d6c740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a033c33450 0x55a033c2f880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:10.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.220+0000 7f23d3d6c740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55a033c443c0 con 0x55a033c33450 2024-04-02T14:34:10.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.220+0000 7f23d3d6c740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55a033c44a90 con 0x55a033c2d8f0 2024-04-02T14:34:10.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.220+0000 7f23d3d6c740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55a033c45160 con 0x55a033c2e2e0 2024-04-02T14:34:10.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.220+0000 7f23ca800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a033c33450 0x55a033c2f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:10.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.220+0000 7f23c9e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a033c2d8f0 0x55a033c2ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:10.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.220+0000 7f23c9e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a033c2d8f0 0x55a033c2ece0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:57418/0 (socket says 172.21.15.90:57418) 2024-04-02T14:34:10.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.220+0000 7f23ca800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a033c33450 0x55a033c2f880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:52456/0 (socket says 172.21.15.90:52456) 2024-04-02T14:34:10.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.220+0000 7f23c9e00640 1 -- 172.21.15.90:0/2171761757 learned_addr learned my addr 172.21.15.90:0/2171761757 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:34:10.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.220+0000 7f23c3e00640 1 --2- 172.21.15.90:0/2171761757 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a033c2e2e0 0x55a033c3f300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:10.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.220+0000 7f23ca800640 1 -- 172.21.15.90:0/2171761757 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a033c2e2e0 msgr2=0x55a033c3f300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:34:10.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.220+0000 7f23ca800640 1 --2- 172.21.15.90:0/2171761757 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a033c2e2e0 0x55a033c3f300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:10.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.220+0000 7f23ca800640 1 -- 172.21.15.90:0/2171761757 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a033c2d8f0 msgr2=0x55a033c2ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:34:10.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.220+0000 7f23ca800640 1 --2- 172.21.15.90:0/2171761757 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a033c2d8f0 0x55a033c2ece0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:10.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.220+0000 7f23ca800640 1 -- 172.21.15.90:0/2171761757 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a033c32b70 con 0x55a033c33450 2024-04-02T14:34:10.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.220+0000 7f23c9e00640 1 --2- 172.21.15.90:0/2171761757 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a033c2d8f0 0x55a033c2ece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:34:10.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.221+0000 7f23ca800640 1 --2- 172.21.15.90:0/2171761757 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a033c33450 0x55a033c2f880 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f23c40ce690 tx=0x7f23c40ce1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ea119eacad62698 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:34:10.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.221+0000 7f23baa00640 1 -- 172.21.15.90:0/2171761757 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23c409cb00 con 0x55a033c33450 2024-04-02T14:34:10.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.221+0000 7f23baa00640 1 -- 172.21.15.90:0/2171761757 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f23c40a0eb0 con 0x55a033c33450 2024-04-02T14:34:10.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.221+0000 7f23baa00640 1 -- 172.21.15.90:0/2171761757 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23c4004120 con 0x55a033c33450 2024-04-02T14:34:10.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.222+0000 7f23d3d6c740 1 -- 172.21.15.90:0/2171761757 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a033c33450 msgr2=0x55a033c2f880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:10.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.222+0000 7f23d3d6c740 1 --2- 172.21.15.90:0/2171761757 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a033c33450 0x55a033c2f880 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f23c40ce690 tx=0x7f23c40ce1f0 comp rx=0 tx=0).stop 2024-04-02T14:34:10.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.222+0000 7f23d3d6c740 1 -- 172.21.15.90:0/2171761757 shutdown_connections 2024-04-02T14:34:10.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.222+0000 7f23d3d6c740 1 --2- 172.21.15.90:0/2171761757 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a033c33450 0x55a033c2f880 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:10.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.222+0000 7f23d3d6c740 1 --2- 172.21.15.90:0/2171761757 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a033c2e2e0 0x55a033c3f300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:10.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.222+0000 7f23d3d6c740 1 --2- 172.21.15.90:0/2171761757 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a033c2d8f0 0x55a033c2ece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:10.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.222+0000 7f23d3d6c740 1 -- 172.21.15.90:0/2171761757 >> 172.21.15.90:0/2171761757 conn(0x55a033c28e60 msgr2=0x55a033c36890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:34:10.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.222+0000 7f23d3d6c740 1 -- 172.21.15.90:0/2171761757 shutdown_connections 2024-04-02T14:34:10.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.222+0000 7f23d3d6c740 1 -- 172.21.15.90:0/2171761757 wait complete. 2024-04-02T14:34:10.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.222+0000 7f23d3d6c740 1 -- start start 2024-04-02T14:34:10.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.222+0000 7f23d3d6c740 1 Processor -- start 2024-04-02T14:34:10.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.223+0000 7f23d3d6c740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a033c2d8f0 0x55a033c3b140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:10.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.223+0000 7f23d3d6c740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a033c2e2e0 0x55a033c3b680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:10.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.223+0000 7f23d3d6c740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a033c33450 0x55a033c3e630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:10.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.223+0000 7f23d3d6c740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55a033c45f00 con 0x55a033c33450 2024-04-02T14:34:10.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.223+0000 7f23d3d6c740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55a033c45830 con 0x55a033c2e2e0 2024-04-02T14:34:10.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.223+0000 7f23d3d6c740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55a033c465d0 con 0x55a033c2d8f0 2024-04-02T14:34:10.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.223+0000 7f23d3d6c740 10 client.0 _ll_register_callbacks cb 0x7f23bc01e8b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:34:10.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.223+0000 7f23ca800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a033c33450 0x55a033c3e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:10.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.223+0000 7f23ca800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a033c33450 0x55a033c3e630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:52460/0 (socket says 172.21.15.90:52460) 2024-04-02T14:34:10.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.223+0000 7f23ca800640 1 -- 172.21.15.90:0/3377526286 learned_addr learned my addr 172.21.15.90:0/3377526286 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:34:10.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.223+0000 7f23c3e00640 1 --2- 172.21.15.90:0/3377526286 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a033c2e2e0 0x55a033c3b680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:10.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.223+0000 7f23c9e00640 1 --2- 172.21.15.90:0/3377526286 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a033c2d8f0 0x55a033c3b140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:10.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.223+0000 7f23ca800640 1 -- 172.21.15.90:0/3377526286 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a033c2d8f0 msgr2=0x55a033c3b140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:34:10.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.223+0000 7f23ca800640 1 --2- 172.21.15.90:0/3377526286 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a033c2d8f0 0x55a033c3b140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:10.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.223+0000 7f23ca800640 1 -- 172.21.15.90:0/3377526286 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a033c2e2e0 msgr2=0x55a033c3b680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:34:10.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.223+0000 7f23ca800640 1 --2- 172.21.15.90:0/3377526286 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a033c2e2e0 0x55a033c3b680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:10.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.223+0000 7f23ca800640 1 -- 172.21.15.90:0/3377526286 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a033c3ec20 con 0x55a033c33450 2024-04-02T14:34:10.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.223+0000 7f23c3e00640 1 --2- 172.21.15.90:0/3377526286 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a033c2e2e0 0x55a033c3b680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:34:10.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.224+0000 7f23ca800640 1 --2- 172.21.15.90:0/3377526286 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a033c33450 0x55a033c3e630 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f23c40bd010 tx=0x7f23c400b760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a36621ba7ee75b3 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:34:10.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.224+0000 7f23afe00640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-02T14:34:10.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.224+0000 7f23afe00640 1 -- 172.21.15.90:0/3377526286 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23c409ae00 con 0x55a033c33450 2024-04-02T14:34:10.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.224+0000 7f23afe00640 1 -- 172.21.15.90:0/3377526286 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8248+0+0 (secure 0 0 0) 0x7f23c4099170 con 0x55a033c33450 2024-04-02T14:34:10.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.224+0000 7f23d3d6c740 1 -- 172.21.15.90:0/3377526286 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f23c40a0eb0 con 0x55a033c33450 2024-04-02T14:34:10.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.224+0000 7f23afe00640 1 -- 172.21.15.90:0/3377526286 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f23c409cb00 con 0x55a033c33450 2024-04-02T14:34:10.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.224+0000 7f23d3d6c740 10 client.14964 fetch_fsmap learned FSMap version 26 2024-04-02T14:34:10.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.224+0000 7f23d3d6c740 1 -- 172.21.15.90:0/3377526286 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55a033cab2a0 con 0x55a033c33450 2024-04-02T14:34:10.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.224+0000 7f23afe00640 1 -- 172.21.15.90:0/3377526286 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f23c40a0eb0 con 0x55a033c33450 2024-04-02T14:34:10.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.224+0000 7f23afe00640 1 -- 172.21.15.90:0/3377526286 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f23c4024060 con 0x55a033c33450 2024-04-02T14:34:10.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.224+0000 7f23d3d6c740 10 client.14964 fetch_fsmap finished waiting for FSMap version 26 2024-04-02T14:34:10.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.225+0000 7f23d3d6c740 10 client.14964 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-02T14:34:10.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.225+0000 7f23d3d6c740 20 client.14964 populate_metadata read hostname 'smithi090' 2024-04-02T14:34:10.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.225+0000 7f23d3d6c740 1 --2- 172.21.15.90:0/3377526286 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55a033caf860 0x55a033cae570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:10.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.225+0000 7f23d3d6c740 4 client.14964 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-02T14:34:10.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.225+0000 7f23d3d6c740 1 -- 172.21.15.90:0/3377526286 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "status"}) -- 0x55a033b9a8f0 con 0x55a033caf860 2024-04-02T14:34:10.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.225+0000 7f23c9e00640 1 --2- 172.21.15.90:0/3377526286 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55a033caf860 0x55a033cae570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:10.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.225+0000 7f23c9e00640 1 --2- 172.21.15.90:0/3377526286 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55a033caf860 0x55a033cae570 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9daa314e2a6fd976 server_cookie=913f02d7baad7952 in_seq=0 out_seq=0 2024-04-02T14:34:10.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.225+0000 7f23afe00640 10 client.14964 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:34:10.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.225+0000 7f23afe00640 1 -- 172.21.15.90:0/3377526286 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x55a033b9a8f0 con 0x55a033caf860 2024-04-02T14:34:10.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.226+0000 7f23afe00640 10 client.14964 handle_command_reply: tid=0 2024-04-02T14:34:10.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.226+0000 7f23d3d6c740 1 client.14964 shutdown 2024-04-02T14:34:10.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.226+0000 7f23d3d6c740 1 -- 172.21.15.90:0/3377526286 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a033c33450 msgr2=0x55a033c3e630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:10.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.226+0000 7f23d3d6c740 1 --2- 172.21.15.90:0/3377526286 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a033c33450 0x55a033c3e630 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f23c40bd010 tx=0x7f23c400b760 comp rx=0 tx=0).stop 2024-04-02T14:34:10.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.226+0000 7f23d3d6c740 1 -- 172.21.15.90:0/3377526286 shutdown_connections 2024-04-02T14:34:10.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.226+0000 7f23d3d6c740 1 --2- 172.21.15.90:0/3377526286 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55a033caf860 0x55a033cae570 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:10.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.226+0000 7f23d3d6c740 1 --2- 172.21.15.90:0/3377526286 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a033c33450 0x55a033c3e630 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:10.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.226+0000 7f23d3d6c740 1 --2- 172.21.15.90:0/3377526286 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a033c2e2e0 0x55a033c3b680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:10.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.226+0000 7f23d3d6c740 1 --2- 172.21.15.90:0/3377526286 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a033c2d8f0 0x55a033c3b140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:10.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.226+0000 7f23d3d6c740 1 -- 172.21.15.90:0/3377526286 >> 172.21.15.90:0/3377526286 conn(0x55a033c28e60 msgr2=0x55a033c2ece0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:34:10.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.226+0000 7f23afe00640 0 client.14964 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:34:10.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.226+0000 7f23d3d6c740 1 -- 172.21.15.90:0/3377526286 shutdown_connections 2024-04-02T14:34:10.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.226+0000 7f23d3d6c740 1 -- 172.21.15.90:0/3377526286 wait complete. 2024-04-02T14:34:10.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.226+0000 7f23d3d6c740 20 client.14964 trim_cache size 0 max 16384 2024-04-02T14:34:10.232 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-02T14:34:10.232 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_fsid": "6449448a-f0fc-11ee-b647-cb9ed24678a4", 2024-04-02T14:34:10.232 INFO:teuthology.orchestra.run.smithi090.stdout: "whoami": 0, 2024-04-02T14:34:10.232 INFO:teuthology.orchestra.run.smithi090.stdout: "id": 24590, 2024-04-02T14:34:10.232 INFO:teuthology.orchestra.run.smithi090.stdout: "want_state": "up:active", 2024-04-02T14:34:10.232 INFO:teuthology.orchestra.run.smithi090.stdout: "state": "up:active", 2024-04-02T14:34:10.232 INFO:teuthology.orchestra.run.smithi090.stdout: "fs_name": "cephfs", 2024-04-02T14:34:10.232 INFO:teuthology.orchestra.run.smithi090.stdout: "rank_uptime": 107.37652672500001, 2024-04-02T14:34:10.232 INFO:teuthology.orchestra.run.smithi090.stdout: "mdsmap_epoch": 26, 2024-04-02T14:34:10.232 INFO:teuthology.orchestra.run.smithi090.stdout: "osdmap_epoch": 90, 2024-04-02T14:34:10.233 INFO:teuthology.orchestra.run.smithi090.stdout: "osdmap_epoch_barrier": 88, 2024-04-02T14:34:10.233 INFO:teuthology.orchestra.run.smithi090.stdout: "uptime": 113.561644388 2024-04-02T14:34:10.233 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-02T14:34:10.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.227+0000 7f23d1000640 1 -- 172.21.15.90:0/1990035966 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f23b00a3bb0 msgr2=0x7f23b00a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:10.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.227+0000 7f23d1000640 1 --2- 172.21.15.90:0/1990035966 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f23b00a3bb0 0x7f23b00a6070 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x55a033c2e6e0 tx=0x7f23bc005f90 comp rx=0 tx=0).stop 2024-04-02T14:34:10.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.227+0000 7f23d1000640 1 -- 172.21.15.90:0/1990035966 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f23cc107800 msgr2=0x7f23cc100a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:10.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.227+0000 7f23d1000640 1 --2- 172.21.15.90:0/1990035966 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f23cc107800 0x7f23cc100a90 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f23c400da10 tx=0x7f23c400ded0 comp rx=0 tx=0).stop 2024-04-02T14:34:10.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.227+0000 7f23d1000640 1 -- 172.21.15.90:0/1990035966 shutdown_connections 2024-04-02T14:34:10.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.227+0000 7f23d1000640 1 --2- 172.21.15.90:0/1990035966 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f23b00a3bb0 0x7f23b00a6070 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:10.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.227+0000 7f23d1000640 1 --2- 172.21.15.90:0/1990035966 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f23cc107800 0x7f23cc100a90 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:10.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.227+0000 7f23d1000640 1 --2- 172.21.15.90:0/1990035966 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f23cc104ed0 0x7f23cc100550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:10.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.227+0000 7f23d1000640 1 --2- 172.21.15.90:0/1990035966 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f23cc1025a0 0x7f23cc101ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:10.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.227+0000 7f23d1000640 1 -- 172.21.15.90:0/1990035966 >> 172.21.15.90:0/1990035966 conn(0x7f23cc0fbf90 msgr2=0x7f23cc102fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:34:10.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.227+0000 7f23d1000640 1 -- 172.21.15.90:0/1990035966 shutdown_connections 2024-04-02T14:34:10.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:10.227+0000 7f23d1000640 1 -- 172.21.15.90:0/1990035966 wait complete. 2024-04-02T14:34:10.246 INFO:tasks.cephadm:Setup complete, yielding 2024-04-02T14:34:10.246 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-02T14:34:10.256 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi090.front.sepia.ceph.com 2024-04-02T14:34:10.257 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:191d963ac9de7322da8aff20b567b3fa458e2ef6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6449448a-f0fc-11ee-b647-cb9ed24678a4 -- bash -c 'ceph orch status' 2024-04-02T14:34:10.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:34:09 smithi092 ceph-mon[36497]: pgmap v367: 97 pgs: 97 active+clean; 644 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-02T14:34:10.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:34:09 smithi092 ceph-mon[36497]: from='client.? 172.21.15.90:0/559140841' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-02T14:34:10.681 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T14:34:11.920 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:34:11 smithi090 ceph-mon[31674]: pgmap v368: 97 pgs: 97 active+clean; 644 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-02T14:34:12.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:34:11 smithi177 ceph-mon[36431]: pgmap v368: 97 pgs: 97 active+clean; 644 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-02T14:34:12.360 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:34:11 smithi092 ceph-mon[36497]: pgmap v368: 97 pgs: 97 active+clean; 644 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-02T14:34:12.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:12.903+0000 7f1f85600700 1 -- 172.21.15.90:0/2021093201 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1f80101780 msgr2=0x7f1f80101c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:12.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:12.903+0000 7f1f85600700 1 --2- 172.21.15.90:0/2021093201 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1f80101780 0x7f1f80101c00 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f1f6c0098f0 tx=0x7f1f6c009c00 comp rx=0 tx=0).stop 2024-04-02T14:34:12.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:12.903+0000 7f1f85600700 1 -- 172.21.15.90:0/2021093201 shutdown_connections 2024-04-02T14:34:12.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:12.903+0000 7f1f85600700 1 --2- 172.21.15.90:0/2021093201 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1f80100e20 0x7f1f80101240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:12.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:12.903+0000 7f1f85600700 1 --2- 172.21.15.90:0/2021093201 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1f80102140 0x7f1f801025c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:12.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:12.903+0000 7f1f85600700 1 --2- 172.21.15.90:0/2021093201 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1f80101780 0x7f1f80101c00 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:12.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:12.903+0000 7f1f85600700 1 -- 172.21.15.90:0/2021093201 >> 172.21.15.90:0/2021093201 conn(0x7f1f800fc9c0 msgr2=0x7f1f800fee20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:34:12.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:12.904+0000 7f1f85600700 1 -- 172.21.15.90:0/2021093201 shutdown_connections 2024-04-02T14:34:12.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:12.904+0000 7f1f85600700 1 -- 172.21.15.90:0/2021093201 wait complete. 2024-04-02T14:34:12.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:12.905+0000 7f1f85600700 1 Processor -- start 2024-04-02T14:34:12.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:12.905+0000 7f1f85600700 1 -- start start 2024-04-02T14:34:12.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:12.906+0000 7f1f85600700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1f80100e20 0x7f1f8019a960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:12.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:12.906+0000 7f1f85600700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1f80101780 0x7f1f8019aea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:12.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:12.906+0000 7f1f85600700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1f80102140 0x7f1f801a1f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:12.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:12.906+0000 7f1f85600700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f1f801102f0 con 0x7f1f80101780 2024-04-02T14:34:12.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:12.906+0000 7f1f85600700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f1f80110170 con 0x7f1f80102140 2024-04-02T14:34:12.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:12.906+0000 7f1f85600700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f1f80110470 con 0x7f1f80100e20 2024-04-02T14:34:12.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:12.906+0000 7f1f7f400700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1f80101780 0x7f1f8019aea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:12.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:12.906+0000 7f1f7fe00700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1f80100e20 0x7f1f8019a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:12.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:12.906+0000 7f1f7f400700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1f80101780 0x7f1f8019aea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:52480/0 (socket says 172.21.15.90:52480) 2024-04-02T14:34:12.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:12.906+0000 7f1f7f400700 1 -- 172.21.15.90:0/2182138683 learned_addr learned my addr 172.21.15.90:0/2182138683 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:34:12.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:12.906+0000 7f1f84c00700 1 --2- 172.21.15.90:0/2182138683 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1f80102140 0x7f1f801a1f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:12.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:12.907+0000 7f1f7fe00700 1 -- 172.21.15.90:0/2182138683 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1f80102140 msgr2=0x7f1f801a1f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:12.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:12.907+0000 7f1f7fe00700 1 --2- 172.21.15.90:0/2182138683 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1f80102140 0x7f1f801a1f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:12.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:12.907+0000 7f1f7fe00700 1 -- 172.21.15.90:0/2182138683 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1f80101780 msgr2=0x7f1f8019aea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:12.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:12.907+0000 7f1f7fe00700 1 --2- 172.21.15.90:0/2182138683 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1f80101780 0x7f1f8019aea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:12.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:12.907+0000 7f1f7fe00700 1 -- 172.21.15.90:0/2182138683 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f801a25c0 con 0x7f1f80100e20 2024-04-02T14:34:12.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:12.907+0000 7f1f7f400700 1 --2- 172.21.15.90:0/2182138683 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1f80101780 0x7f1f8019aea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:34:12.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:12.907+0000 7f1f7fe00700 1 --2- 172.21.15.90:0/2182138683 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1f80100e20 0x7f1f8019a960 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f1f7000e9f0 tx=0x7f1f7000ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:34:12.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:12.907+0000 7f1f7cc00700 1 -- 172.21.15.90:0/2182138683 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f7000c9c0 con 0x7f1f80100e20 2024-04-02T14:34:12.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:12.907+0000 7f1f7cc00700 1 -- 172.21.15.90:0/2182138683 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1f70010430 con 0x7f1f80100e20 2024-04-02T14:34:12.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:12.907+0000 7f1f7cc00700 1 -- 172.21.15.90:0/2182138683 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f70018680 con 0x7f1f80100e20 2024-04-02T14:34:12.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:12.907+0000 7f1f85600700 1 -- 172.21.15.90:0/2182138683 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1f801a28b0 con 0x7f1f80100e20 2024-04-02T14:34:12.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:12.908+0000 7f1f85600700 1 -- 172.21.15.90:0/2182138683 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1f801a2df0 con 0x7f1f80100e20 2024-04-02T14:34:12.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:12.909+0000 7f1f85600700 1 -- 172.21.15.90:0/2182138683 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1f800692b0 con 0x7f1f80100e20 2024-04-02T14:34:12.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:12.911+0000 7f1f7cc00700 1 -- 172.21.15.90:0/2182138683 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f1f700105e0 con 0x7f1f80100e20 2024-04-02T14:34:12.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:12.912+0000 7f1f7cc00700 1 --2- 172.21.15.90:0/2182138683 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f1f640a3c40 0x7f1f640a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:12.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:12.912+0000 7f1f7cc00700 1 -- 172.21.15.90:0/2182138683 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8248+0+0 (secure 0 0 0) 0x7f1f700d08e0 con 0x7f1f80100e20 2024-04-02T14:34:12.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:12.912+0000 7f1f7f400700 1 --2- 172.21.15.90:0/2182138683 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f1f640a3c40 0x7f1f640a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:12.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:12.913+0000 7f1f7f400700 1 --2- 172.21.15.90:0/2182138683 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f1f640a3c40 0x7f1f640a6100 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f1f6c009fd0 tx=0x7f1f6c019040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:34:12.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:12.914+0000 7f1f7cc00700 1 -- 172.21.15.90:0/2182138683 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7f1f7009d250 con 0x7f1f80100e20 2024-04-02T14:34:13.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:13.161+0000 7f1f85600700 1 -- 172.21.15.90:0/2182138683 --> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) -- 0x7f1f80062a30 con 0x7f1f640a3c40 2024-04-02T14:34:13.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:13.163+0000 7f1f7cc00700 1 -- 172.21.15.90:0/2182138683 <== mgr.14152 v2:172.21.15.90:6800/4266054817 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+43 (secure 0 0 0) 0x7f1f80062a30 con 0x7f1f640a3c40 2024-04-02T14:34:13.165 INFO:teuthology.orchestra.run.smithi090.stdout:Backend: cephadm 2024-04-02T14:34:13.165 INFO:teuthology.orchestra.run.smithi090.stdout:Available: Yes 2024-04-02T14:34:13.165 INFO:teuthology.orchestra.run.smithi090.stdout:Paused: No 2024-04-02T14:34:13.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:13.166+0000 7f1f85600700 1 -- 172.21.15.90:0/2182138683 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f1f640a3c40 msgr2=0x7f1f640a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:13.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:13.166+0000 7f1f85600700 1 --2- 172.21.15.90:0/2182138683 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f1f640a3c40 0x7f1f640a6100 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f1f6c009fd0 tx=0x7f1f6c019040 comp rx=0 tx=0).stop 2024-04-02T14:34:13.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:13.166+0000 7f1f85600700 1 -- 172.21.15.90:0/2182138683 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1f80100e20 msgr2=0x7f1f8019a960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:13.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:13.166+0000 7f1f85600700 1 --2- 172.21.15.90:0/2182138683 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1f80100e20 0x7f1f8019a960 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f1f7000e9f0 tx=0x7f1f7000ed00 comp rx=0 tx=0).stop 2024-04-02T14:34:13.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:13.167+0000 7f1f85600700 1 -- 172.21.15.90:0/2182138683 shutdown_connections 2024-04-02T14:34:13.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:13.167+0000 7f1f85600700 1 --2- 172.21.15.90:0/2182138683 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f1f640a3c40 0x7f1f640a6100 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:13.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:13.167+0000 7f1f85600700 1 --2- 172.21.15.90:0/2182138683 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1f80102140 0x7f1f801a1f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:13.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:13.167+0000 7f1f85600700 1 --2- 172.21.15.90:0/2182138683 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1f80100e20 0x7f1f8019a960 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:13.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:13.167+0000 7f1f85600700 1 --2- 172.21.15.90:0/2182138683 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1f80101780 0x7f1f8019aea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:13.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:13.167+0000 7f1f85600700 1 -- 172.21.15.90:0/2182138683 >> 172.21.15.90:0/2182138683 conn(0x7f1f800fc9c0 msgr2=0x7f1f800fe4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:34:13.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:13.167+0000 7f1f85600700 1 -- 172.21.15.90:0/2182138683 shutdown_connections 2024-04-02T14:34:13.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:13.167+0000 7f1f85600700 1 -- 172.21.15.90:0/2182138683 wait complete. 2024-04-02T14:34:14.033 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:191d963ac9de7322da8aff20b567b3fa458e2ef6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6449448a-f0fc-11ee-b647-cb9ed24678a4 -- bash -c 'ceph orch ps' 2024-04-02T14:34:14.191 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:34:13 smithi177 ceph-mon[36431]: pgmap v369: 97 pgs: 97 active+clean; 644 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-02T14:34:14.193 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:34:13 smithi090 ceph-mon[31674]: pgmap v369: 97 pgs: 97 active+clean; 644 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-02T14:34:14.363 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:34:13 smithi092 ceph-mon[36497]: pgmap v369: 97 pgs: 97 active+clean; 644 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-02T14:34:14.471 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T14:34:15.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:34:14 smithi177 ceph-mon[36431]: from='client.25060 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T14:34:15.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:34:14 smithi090 ceph-mon[31674]: from='client.25060 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T14:34:15.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:34:14 smithi092 ceph-mon[36497]: from='client.25060 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T14:34:16.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:34:15 smithi177 ceph-mon[36431]: pgmap v370: 97 pgs: 97 active+clean; 644 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-02T14:34:16.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:34:15 smithi090 ceph-mon[31674]: pgmap v370: 97 pgs: 97 active+clean; 644 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-02T14:34:16.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:34:15 smithi092 ceph-mon[36497]: pgmap v370: 97 pgs: 97 active+clean; 644 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-02T14:34:16.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:16.711+0000 7ff371a00700 1 -- 172.21.15.90:0/2131921984 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff36c1083e0 msgr2=0x7ff36c10a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:16.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:16.711+0000 7ff371a00700 1 --2- 172.21.15.90:0/2131921984 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff36c1083e0 0x7ff36c10a7d0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7ff36400b0a0 tx=0x7ff36400b3b0 comp rx=0 tx=0).stop 2024-04-02T14:34:16.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:16.712+0000 7ff371a00700 1 -- 172.21.15.90:0/2131921984 shutdown_connections 2024-04-02T14:34:16.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:16.712+0000 7ff371a00700 1 --2- 172.21.15.90:0/2131921984 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff36c1083e0 0x7ff36c10a7d0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:16.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:16.712+0000 7ff371a00700 1 --2- 172.21.15.90:0/2131921984 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff36c103180 0x7ff36c105570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:16.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:16.712+0000 7ff371a00700 1 --2- 172.21.15.90:0/2131921984 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff36c105ab0 0x7ff36c107ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:16.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:16.712+0000 7ff371a00700 1 -- 172.21.15.90:0/2131921984 >> 172.21.15.90:0/2131921984 conn(0x7ff36c0fcaf0 msgr2=0x7ff36c0fef10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:34:16.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:16.712+0000 7ff371a00700 1 -- 172.21.15.90:0/2131921984 shutdown_connections 2024-04-02T14:34:16.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:16.712+0000 7ff371a00700 1 -- 172.21.15.90:0/2131921984 wait complete. 2024-04-02T14:34:16.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:16.713+0000 7ff371a00700 1 Processor -- start 2024-04-02T14:34:16.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:16.713+0000 7ff371a00700 1 -- start start 2024-04-02T14:34:16.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:16.713+0000 7ff371a00700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff36c103180 0x7ff36c198870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:16.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:16.713+0000 7ff371a00700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff36c105ab0 0x7ff36c198db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:16.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:16.713+0000 7ff371a00700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff36c1083e0 0x7ff36c19fde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:16.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:16.713+0000 7ff371a00700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7ff36c10e200 con 0x7ff36c1083e0 2024-04-02T14:34:16.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:16.713+0000 7ff371a00700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7ff36c10e080 con 0x7ff36c103180 2024-04-02T14:34:16.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:16.713+0000 7ff371a00700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7ff36c10e380 con 0x7ff36c105ab0 2024-04-02T14:34:16.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:16.714+0000 7ff36b400700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff36c1083e0 0x7ff36c19fde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:16.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:16.714+0000 7ff36aa00700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff36c103180 0x7ff36c198870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:16.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:16.714+0000 7ff36b400700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff36c1083e0 0x7ff36c19fde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33172/0 (socket says 172.21.15.90:33172) 2024-04-02T14:34:16.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:16.714+0000 7ff36aa00700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff36c103180 0x7ff36c198870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:48998/0 (socket says 172.21.15.90:48998) 2024-04-02T14:34:16.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:16.714+0000 7ff36b400700 1 -- 172.21.15.90:0/2895191309 learned_addr learned my addr 172.21.15.90:0/2895191309 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:34:16.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:16.714+0000 7ff36a000700 1 --2- 172.21.15.90:0/2895191309 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff36c105ab0 0x7ff36c198db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:16.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:16.714+0000 7ff36aa00700 1 -- 172.21.15.90:0/2895191309 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff36c105ab0 msgr2=0x7ff36c198db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:16.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:16.714+0000 7ff36aa00700 1 --2- 172.21.15.90:0/2895191309 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff36c105ab0 0x7ff36c198db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:16.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:16.714+0000 7ff36aa00700 1 -- 172.21.15.90:0/2895191309 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff36c1083e0 msgr2=0x7ff36c19fde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:16.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:16.714+0000 7ff36aa00700 1 --2- 172.21.15.90:0/2895191309 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff36c1083e0 0x7ff36c19fde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:16.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:16.714+0000 7ff36aa00700 1 -- 172.21.15.90:0/2895191309 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff36c1a0480 con 0x7ff36c103180 2024-04-02T14:34:16.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:16.714+0000 7ff36b400700 1 --2- 172.21.15.90:0/2895191309 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff36c1083e0 0x7ff36c19fde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:34:16.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:16.714+0000 7ff36aa00700 1 --2- 172.21.15.90:0/2895191309 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff36c103180 0x7ff36c198870 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7ff35c00e9f0 tx=0x7ff35c00ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:34:16.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:16.715+0000 7ff363400700 1 -- 172.21.15.90:0/2895191309 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff35c004220 con 0x7ff36c103180 2024-04-02T14:34:16.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:16.715+0000 7ff371a00700 1 -- 172.21.15.90:0/2895191309 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff36c1a0770 con 0x7ff36c103180 2024-04-02T14:34:16.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:16.715+0000 7ff363400700 1 -- 172.21.15.90:0/2895191309 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff35c004d10 con 0x7ff36c103180 2024-04-02T14:34:16.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:16.715+0000 7ff363400700 1 -- 172.21.15.90:0/2895191309 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff35c005020 con 0x7ff36c103180 2024-04-02T14:34:16.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:16.715+0000 7ff371a00700 1 -- 172.21.15.90:0/2895191309 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff36c1a0c60 con 0x7ff36c103180 2024-04-02T14:34:16.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:16.716+0000 7ff371a00700 1 -- 172.21.15.90:0/2895191309 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff36c0692b0 con 0x7ff36c103180 2024-04-02T14:34:16.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:16.718+0000 7ff363400700 1 -- 172.21.15.90:0/2895191309 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7ff35c004750 con 0x7ff36c103180 2024-04-02T14:34:16.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:16.719+0000 7ff363400700 1 --2- 172.21.15.90:0/2895191309 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7ff3500a3c40 0x7ff3500a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:16.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:16.719+0000 7ff363400700 1 -- 172.21.15.90:0/2895191309 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8248+0+0 (secure 0 0 0) 0x7ff35c0d0500 con 0x7ff36c103180 2024-04-02T14:34:16.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:16.720+0000 7ff36a000700 1 --2- 172.21.15.90:0/2895191309 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7ff3500a3c40 0x7ff3500a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:16.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:16.720+0000 7ff36a000700 1 --2- 172.21.15.90:0/2895191309 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7ff3500a3c40 0x7ff3500a6100 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7ff358009c60 tx=0x7ff358009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:34:16.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:16.722+0000 7ff363400700 1 -- 172.21.15.90:0/2895191309 <== mon.2 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7ff35c09ce70 con 0x7ff36c103180 2024-04-02T14:34:16.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:16.959+0000 7ff371a00700 1 -- 172.21.15.90:0/2895191309 --> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) -- 0x7ff36c062a30 con 0x7ff3500a3c40 2024-04-02T14:34:17.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:16.999+0000 7ff363400700 1 -- 172.21.15.90:0/2895191309 <== mgr.14152 v2:172.21.15.90:6800/4266054817 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+4123 (secure 0 0 0) 0x7ff36c062a30 con 0x7ff3500a3c40 2024-04-02T14:34:17.001 INFO:teuthology.orchestra.run.smithi090.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-02T14:34:17.001 INFO:teuthology.orchestra.run.smithi090.stdout:mds.a smithi090 running (2m) 25s ago 2m 12.4M - 19.0.0-2609-g191d963a f7301e070545 a132ebe3f492 2024-04-02T14:34:17.001 INFO:teuthology.orchestra.run.smithi090.stdout:mds.b smithi092 running (2m) 52s ago 2m 26.1M - 19.0.0-2609-g191d963a f7301e070545 1f09a480542a 2024-04-02T14:34:17.001 INFO:teuthology.orchestra.run.smithi090.stdout:mds.c smithi177 running (2m) 52s ago 2m 19.9M - 19.0.0-2609-g191d963a f7301e070545 ab5bb25e316a 2024-04-02T14:34:17.001 INFO:teuthology.orchestra.run.smithi090.stdout:mds.d smithi090 running (2m) 25s ago 2m 12.6M - 19.0.0-2609-g191d963a f7301e070545 675ab21b436c 2024-04-02T14:34:17.001 INFO:teuthology.orchestra.run.smithi090.stdout:mds.e smithi092 running (2m) 52s ago 2m 16.8M - 19.0.0-2609-g191d963a f7301e070545 bf500f992b9b 2024-04-02T14:34:17.002 INFO:teuthology.orchestra.run.smithi090.stdout:mds.f smithi177 running (2m) 52s ago 2m 19.2M - 19.0.0-2609-g191d963a f7301e070545 20637d31e333 2024-04-02T14:34:17.002 INFO:teuthology.orchestra.run.smithi090.stdout:mds.g smithi090 running (2m) 25s ago 2m 17.4M - 19.0.0-2609-g191d963a f7301e070545 6aff04420394 2024-04-02T14:34:17.002 INFO:teuthology.orchestra.run.smithi090.stdout:mds.h smithi092 running (2m) 52s ago 2m 19.9M - 19.0.0-2609-g191d963a f7301e070545 1d4fecb3a653 2024-04-02T14:34:17.002 INFO:teuthology.orchestra.run.smithi090.stdout:mds.i smithi177 running (2m) 52s ago 2m 20.1M - 19.0.0-2609-g191d963a f7301e070545 689f6bf561b1 2024-04-02T14:34:17.002 INFO:teuthology.orchestra.run.smithi090.stdout:mds.j smithi090 running (2m) 25s ago 2m 16.2M - 19.0.0-2609-g191d963a f7301e070545 d0bfbc2f5a23 2024-04-02T14:34:17.002 INFO:teuthology.orchestra.run.smithi090.stdout:mds.k smithi092 running (2m) 52s ago 2m 19.2M - 19.0.0-2609-g191d963a f7301e070545 e4972a161fe4 2024-04-02T14:34:17.002 INFO:teuthology.orchestra.run.smithi090.stdout:mds.l smithi177 running (2m) 52s ago 2m 19.4M - 19.0.0-2609-g191d963a f7301e070545 4bd8c1a305ce 2024-04-02T14:34:17.002 INFO:teuthology.orchestra.run.smithi090.stdout:mgr.x smithi090 *:9283,8765 running (10m) 25s ago 10m 514M - 19.0.0-2609-g191d963a f7301e070545 3c48d7f60a2e 2024-04-02T14:34:17.002 INFO:teuthology.orchestra.run.smithi090.stdout:mgr.y smithi092 *:8443,8765 running (8m) 52s ago 8m 453M - 19.0.0-2609-g191d963a f7301e070545 f24422517204 2024-04-02T14:34:17.002 INFO:teuthology.orchestra.run.smithi090.stdout:mgr.z smithi177 *:8443,8765 running (8m) 52s ago 8m 454M - 19.0.0-2609-g191d963a f7301e070545 83b9170d642e 2024-04-02T14:34:17.002 INFO:teuthology.orchestra.run.smithi090.stdout:mon.a smithi090 running (10m) 25s ago 11m 50.5M 2048M 19.0.0-2609-g191d963a f7301e070545 c279381a76c0 2024-04-02T14:34:17.002 INFO:teuthology.orchestra.run.smithi090.stdout:mon.b smithi092 running (8m) 52s ago 8m 45.4M 2048M 19.0.0-2609-g191d963a f7301e070545 458efc42a6e4 2024-04-02T14:34:17.003 INFO:teuthology.orchestra.run.smithi090.stdout:mon.c smithi177 running (8m) 52s ago 8m 42.4M 2048M 19.0.0-2609-g191d963a f7301e070545 caa84bb1a070 2024-04-02T14:34:17.003 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0 smithi090 running (7m) 25s ago 7m 56.2M 4272M 19.0.0-2609-g191d963a f7301e070545 9c4567e6e8f0 2024-04-02T14:34:17.003 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1 smithi092 running (7m) 52s ago 7m 59.6M 4270M 19.0.0-2609-g191d963a f7301e070545 238e77d289fb 2024-04-02T14:34:17.003 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2 smithi177 running (7m) 52s ago 7m 59.6M 4270M 19.0.0-2609-g191d963a f7301e070545 e85618ec2a59 2024-04-02T14:34:17.003 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3 smithi090 running (6m) 25s ago 6m 47.2M 4272M 19.0.0-2609-g191d963a f7301e070545 33ad45052eb4 2024-04-02T14:34:17.003 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4 smithi092 running (6m) 52s ago 6m 54.4M 4270M 19.0.0-2609-g191d963a f7301e070545 3f4278c9110f 2024-04-02T14:34:17.003 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5 smithi177 running (5m) 52s ago 5m 46.4M 4270M 19.0.0-2609-g191d963a f7301e070545 31f658b60a05 2024-04-02T14:34:17.003 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6 smithi090 running (5m) 25s ago 5m 52.7M 4272M 19.0.0-2609-g191d963a f7301e070545 16eea31106b9 2024-04-02T14:34:17.003 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7 smithi092 running (4m) 52s ago 4m 55.4M 4270M 19.0.0-2609-g191d963a f7301e070545 b6b4bdf8b23e 2024-04-02T14:34:17.003 INFO:teuthology.orchestra.run.smithi090.stdout:osd.8 smithi177 running (4m) 52s ago 4m 56.3M 4270M 19.0.0-2609-g191d963a f7301e070545 8c3e437d3784 2024-04-02T14:34:17.003 INFO:teuthology.orchestra.run.smithi090.stdout:osd.9 smithi090 running (3m) 25s ago 3m 51.2M 4272M 19.0.0-2609-g191d963a f7301e070545 28fa8627dda5 2024-04-02T14:34:17.003 INFO:teuthology.orchestra.run.smithi090.stdout:osd.10 smithi092 running (3m) 52s ago 3m 53.5M 4270M 19.0.0-2609-g191d963a f7301e070545 04dd1c26ccbb 2024-04-02T14:34:17.003 INFO:teuthology.orchestra.run.smithi090.stdout:osd.11 smithi177 running (3m) 52s ago 3m 54.5M 4270M 19.0.0-2609-g191d963a f7301e070545 f9f4927883d6 2024-04-02T14:34:17.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:17.002+0000 7ff371a00700 1 -- 172.21.15.90:0/2895191309 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7ff3500a3c40 msgr2=0x7ff3500a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:17.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:17.002+0000 7ff371a00700 1 --2- 172.21.15.90:0/2895191309 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7ff3500a3c40 0x7ff3500a6100 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7ff358009c60 tx=0x7ff358009260 comp rx=0 tx=0).stop 2024-04-02T14:34:17.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:17.002+0000 7ff371a00700 1 -- 172.21.15.90:0/2895191309 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff36c103180 msgr2=0x7ff36c198870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:17.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:17.002+0000 7ff371a00700 1 --2- 172.21.15.90:0/2895191309 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff36c103180 0x7ff36c198870 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7ff35c00e9f0 tx=0x7ff35c00ed00 comp rx=0 tx=0).stop 2024-04-02T14:34:17.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:17.003+0000 7ff371a00700 1 -- 172.21.15.90:0/2895191309 shutdown_connections 2024-04-02T14:34:17.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:17.003+0000 7ff371a00700 1 --2- 172.21.15.90:0/2895191309 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7ff3500a3c40 0x7ff3500a6100 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:17.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:17.003+0000 7ff371a00700 1 --2- 172.21.15.90:0/2895191309 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff36c1083e0 0x7ff36c19fde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:17.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:17.003+0000 7ff371a00700 1 --2- 172.21.15.90:0/2895191309 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff36c103180 0x7ff36c198870 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:17.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:17.003+0000 7ff371a00700 1 --2- 172.21.15.90:0/2895191309 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff36c105ab0 0x7ff36c198db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:17.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:17.003+0000 7ff371a00700 1 -- 172.21.15.90:0/2895191309 >> 172.21.15.90:0/2895191309 conn(0x7ff36c0fcaf0 msgr2=0x7ff36c103ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:34:17.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:17.003+0000 7ff371a00700 1 -- 172.21.15.90:0/2895191309 shutdown_connections 2024-04-02T14:34:17.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:17.003+0000 7ff371a00700 1 -- 172.21.15.90:0/2895191309 wait complete. 2024-04-02T14:34:17.834 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:191d963ac9de7322da8aff20b567b3fa458e2ef6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6449448a-f0fc-11ee-b647-cb9ed24678a4 -- bash -c 'ceph orch ls' 2024-04-02T14:34:18.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:34:17 smithi090 ceph-mon[31674]: pgmap v371: 97 pgs: 97 active+clean; 644 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-02T14:34:18.278 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T14:34:18.360 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:34:17 smithi092 ceph-mon[36497]: pgmap v371: 97 pgs: 97 active+clean; 644 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-02T14:34:18.436 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:34:17 smithi177 ceph-mon[36431]: pgmap v371: 97 pgs: 97 active+clean; 644 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-02T14:34:19.270 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:34:18 smithi090 ceph-mon[31674]: from='client.25085 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T14:34:19.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:34:18 smithi092 ceph-mon[36497]: from='client.25085 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T14:34:19.436 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:34:18 smithi177 ceph-mon[36431]: from='client.25085 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T14:34:20.146 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:34:19 smithi090 ceph-mon[31674]: pgmap v372: 97 pgs: 97 active+clean; 644 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-02T14:34:20.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:20.309+0000 7f18f9600700 1 -- 172.21.15.90:0/1332704329 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f18f4076620 msgr2=0x7f18f4074a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:20.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:20.309+0000 7f18f9600700 1 --2- 172.21.15.90:0/1332704329 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f18f4076620 0x7f18f4074a80 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f18e4009990 tx=0x7f18e4009ca0 comp rx=0 tx=0).stop 2024-04-02T14:34:20.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:20.309+0000 7f18f9600700 1 -- 172.21.15.90:0/1332704329 shutdown_connections 2024-04-02T14:34:20.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:20.309+0000 7f18f9600700 1 --2- 172.21.15.90:0/1332704329 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f18f4076620 0x7f18f4074a80 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:20.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:20.309+0000 7f18f9600700 1 --2- 172.21.15.90:0/1332704329 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f18f410a500 0x7f18f410c990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:20.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:20.309+0000 7f18f9600700 1 --2- 172.21.15.90:0/1332704329 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f18f4075050 0x7f18f40754d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:20.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:20.309+0000 7f18f9600700 1 -- 172.21.15.90:0/1332704329 >> 172.21.15.90:0/1332704329 conn(0x7f18f40fdb90 msgr2=0x7f18f4100010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:34:20.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:20.310+0000 7f18f9600700 1 -- 172.21.15.90:0/1332704329 shutdown_connections 2024-04-02T14:34:20.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:20.310+0000 7f18f9600700 1 -- 172.21.15.90:0/1332704329 wait complete. 2024-04-02T14:34:20.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:20.311+0000 7f18f9600700 1 Processor -- start 2024-04-02T14:34:20.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:20.311+0000 7f18f9600700 1 -- start start 2024-04-02T14:34:20.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:20.311+0000 7f18f9600700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f18f4075050 0x7f18f419aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:20.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:20.311+0000 7f18f9600700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f18f4076620 0x7f18f419b030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:20.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:20.311+0000 7f18f9600700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f18f410a500 0x7f18f41a2060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:20.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:20.311+0000 7f18f9600700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f18f4110460 con 0x7f18f4076620 2024-04-02T14:34:20.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:20.311+0000 7f18f9600700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f18f41102e0 con 0x7f18f410a500 2024-04-02T14:34:20.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:20.311+0000 7f18f9600700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f18f41105e0 con 0x7f18f4075050 2024-04-02T14:34:20.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:20.311+0000 7f18f2000700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f18f4076620 0x7f18f419b030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:20.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:20.311+0000 7f18f2000700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f18f4076620 0x7f18f419b030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33200/0 (socket says 172.21.15.90:33200) 2024-04-02T14:34:20.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:20.311+0000 7f18f2000700 1 -- 172.21.15.90:0/320308456 learned_addr learned my addr 172.21.15.90:0/320308456 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:34:20.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:20.312+0000 7f18f2a00700 1 --2- 172.21.15.90:0/320308456 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f18f4075050 0x7f18f419aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:20.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:20.312+0000 7f18f3400700 1 --2- 172.21.15.90:0/320308456 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f18f410a500 0x7f18f41a2060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:20.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:20.312+0000 7f18f2000700 1 -- 172.21.15.90:0/320308456 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f18f4075050 msgr2=0x7f18f419aaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:20.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:20.312+0000 7f18f2000700 1 --2- 172.21.15.90:0/320308456 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f18f4075050 0x7f18f419aaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:20.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:20.312+0000 7f18f2000700 1 -- 172.21.15.90:0/320308456 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f18f410a500 msgr2=0x7f18f41a2060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:20.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:20.312+0000 7f18f2000700 1 --2- 172.21.15.90:0/320308456 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f18f410a500 0x7f18f41a2060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:20.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:20.312+0000 7f18f2000700 1 -- 172.21.15.90:0/320308456 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18f41a2700 con 0x7f18f4076620 2024-04-02T14:34:20.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:20.312+0000 7f18f2000700 1 --2- 172.21.15.90:0/320308456 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f18f4076620 0x7f18f419b030 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f18e000e9a0 tx=0x7f18e000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:34:20.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:20.312+0000 7f18eb400700 1 -- 172.21.15.90:0/320308456 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18e0004220 con 0x7f18f4076620 2024-04-02T14:34:20.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:20.313+0000 7f18eb400700 1 -- 172.21.15.90:0/320308456 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f18e00043c0 con 0x7f18f4076620 2024-04-02T14:34:20.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:20.313+0000 7f18eb400700 1 -- 172.21.15.90:0/320308456 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18e0003df0 con 0x7f18f4076620 2024-04-02T14:34:20.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:20.313+0000 7f18f9600700 1 -- 172.21.15.90:0/320308456 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f18f41a29f0 con 0x7f18f4076620 2024-04-02T14:34:20.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:20.313+0000 7f18f9600700 1 -- 172.21.15.90:0/320308456 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18f41a2ee0 con 0x7f18f4076620 2024-04-02T14:34:20.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:20.314+0000 7f18f9600700 1 -- 172.21.15.90:0/320308456 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f18f40692b0 con 0x7f18f4076620 2024-04-02T14:34:20.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:20.316+0000 7f18eb400700 1 -- 172.21.15.90:0/320308456 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f18e000c810 con 0x7f18f4076620 2024-04-02T14:34:20.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:20.317+0000 7f18eb400700 1 --2- 172.21.15.90:0/320308456 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f18d80a3c40 0x7f18d80a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:20.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:20.318+0000 7f18eb400700 1 -- 172.21.15.90:0/320308456 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8248+0+0 (secure 0 0 0) 0x7f18e00d0610 con 0x7f18f4076620 2024-04-02T14:34:20.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:20.318+0000 7f18f2a00700 1 --2- 172.21.15.90:0/320308456 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f18d80a3c40 0x7f18d80a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:20.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:20.318+0000 7f18f2a00700 1 --2- 172.21.15.90:0/320308456 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f18d80a3c40 0x7f18d80a6100 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f18e4000c00 tx=0x7f18e4005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:34:20.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:20.322+0000 7f18eb400700 1 -- 172.21.15.90:0/320308456 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7f18e009cf80 con 0x7f18f4076620 2024-04-02T14:34:20.338 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:34:19 smithi177 ceph-mon[36431]: pgmap v372: 97 pgs: 97 active+clean; 644 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-02T14:34:20.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:34:19 smithi092 ceph-mon[36497]: pgmap v372: 97 pgs: 97 active+clean; 644 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-02T14:34:20.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:20.560+0000 7f18f9600700 1 -- 172.21.15.90:0/320308456 --> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) -- 0x7f18f4062a30 con 0x7f18d80a3c40 2024-04-02T14:34:20.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:20.576+0000 7f18eb400700 1 -- 172.21.15.90:0/320308456 <== mgr.14152 v2:172.21.15.90:6800/4266054817 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+980 (secure 0 0 0) 0x7f18f4062a30 con 0x7f18d80a3c40 2024-04-02T14:34:20.577 INFO:teuthology.orchestra.run.smithi090.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-04-02T14:34:20.577 INFO:teuthology.orchestra.run.smithi090.stdout:mds.all 12/12 56s ago 2m smithi090=a;smithi090=d;smithi090=g;smithi090=j;smithi092=b;smithi092=e;smithi092=h;smithi092=k;smithi177=c;smithi177=f;smithi177=i;smithi177=l;count:12 2024-04-02T14:34:20.577 INFO:teuthology.orchestra.run.smithi090.stdout:mgr 3/3 56s ago 8m smithi090=x;smithi092=y;smithi177=z;count:3 2024-04-02T14:34:20.578 INFO:teuthology.orchestra.run.smithi090.stdout:mon 3/3 56s ago 8m smithi090:172.21.15.90=a;smithi092:172.21.15.92=b;smithi177:172.21.15.177=c;count:3 2024-04-02T14:34:20.578 INFO:teuthology.orchestra.run.smithi090.stdout:osd 12 56s ago - 2024-04-02T14:34:20.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:20.579+0000 7f18f9600700 1 -- 172.21.15.90:0/320308456 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f18d80a3c40 msgr2=0x7f18d80a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:20.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:20.579+0000 7f18f9600700 1 --2- 172.21.15.90:0/320308456 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f18d80a3c40 0x7f18d80a6100 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f18e4000c00 tx=0x7f18e4005fb0 comp rx=0 tx=0).stop 2024-04-02T14:34:20.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:20.579+0000 7f18f9600700 1 -- 172.21.15.90:0/320308456 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f18f4076620 msgr2=0x7f18f419b030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:20.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:20.579+0000 7f18f9600700 1 --2- 172.21.15.90:0/320308456 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f18f4076620 0x7f18f419b030 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f18e000e9a0 tx=0x7f18e000ecb0 comp rx=0 tx=0).stop 2024-04-02T14:34:20.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:20.579+0000 7f18f9600700 1 -- 172.21.15.90:0/320308456 shutdown_connections 2024-04-02T14:34:20.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:20.579+0000 7f18f9600700 1 --2- 172.21.15.90:0/320308456 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f18d80a3c40 0x7f18d80a6100 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:20.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:20.579+0000 7f18f9600700 1 --2- 172.21.15.90:0/320308456 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f18f410a500 0x7f18f41a2060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:20.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:20.579+0000 7f18f9600700 1 --2- 172.21.15.90:0/320308456 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f18f4075050 0x7f18f419aaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:20.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:20.579+0000 7f18f9600700 1 --2- 172.21.15.90:0/320308456 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f18f4076620 0x7f18f419b030 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:20.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:20.579+0000 7f18f9600700 1 -- 172.21.15.90:0/320308456 >> 172.21.15.90:0/320308456 conn(0x7f18f40fdb90 msgr2=0x7f18f4106400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:34:20.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:20.579+0000 7f18f9600700 1 -- 172.21.15.90:0/320308456 shutdown_connections 2024-04-02T14:34:20.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:20.580+0000 7f18f9600700 1 -- 172.21.15.90:0/320308456 wait complete. 2024-04-02T14:34:21.360 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:34:20 smithi092 ceph-mon[36497]: from='client.14982 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T14:34:21.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:34:20 smithi092 ceph-mon[36497]: pgmap v373: 97 pgs: 97 active+clean; 644 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-02T14:34:21.382 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:34:20 smithi090 ceph-mon[31674]: from='client.14982 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T14:34:21.382 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:34:20 smithi090 ceph-mon[31674]: pgmap v373: 97 pgs: 97 active+clean; 644 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-02T14:34:21.436 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:34:20 smithi177 ceph-mon[36431]: from='client.14982 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T14:34:21.436 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:34:20 smithi177 ceph-mon[36431]: pgmap v373: 97 pgs: 97 active+clean; 644 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-02T14:34:21.438 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:191d963ac9de7322da8aff20b567b3fa458e2ef6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6449448a-f0fc-11ee-b647-cb9ed24678a4 -- bash -c 'ceph orch host ls' 2024-04-02T14:34:21.885 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T14:34:24.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:24.078+0000 7f37c4800700 1 -- 172.21.15.90:0/57172532 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f37bc105a70 msgr2=0x7f37bc105ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:24.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:24.078+0000 7f37c4800700 1 --2- 172.21.15.90:0/57172532 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f37bc105a70 0x7f37bc105ed0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f37ac0099c0 tx=0x7f37ac009cd0 comp rx=0 tx=0).stop 2024-04-02T14:34:24.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:24.078+0000 7f37c4800700 1 -- 172.21.15.90:0/57172532 shutdown_connections 2024-04-02T14:34:24.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:24.078+0000 7f37c4800700 1 --2- 172.21.15.90:0/57172532 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f37bc106410 0x7f37bc10ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:24.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:24.078+0000 7f37c4800700 1 --2- 172.21.15.90:0/57172532 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f37bc104870 0x7f37bc104c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:24.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:24.078+0000 7f37c4800700 1 --2- 172.21.15.90:0/57172532 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f37bc105a70 0x7f37bc105ed0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:24.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:24.078+0000 7f37c4800700 1 -- 172.21.15.90:0/57172532 >> 172.21.15.90:0/57172532 conn(0x7f37bc0ffe10 msgr2=0x7f37bc102250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:34:24.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:24.078+0000 7f37c4800700 1 -- 172.21.15.90:0/57172532 shutdown_connections 2024-04-02T14:34:24.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:24.078+0000 7f37c4800700 1 -- 172.21.15.90:0/57172532 wait complete. 2024-04-02T14:34:24.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:24.079+0000 7f37c4800700 1 Processor -- start 2024-04-02T14:34:24.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:24.079+0000 7f37c4800700 1 -- start start 2024-04-02T14:34:24.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:24.079+0000 7f37c4800700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f37bc104870 0x7f37bc19aa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:24.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:24.079+0000 7f37c4800700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f37bc105a70 0x7f37bc19af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:24.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:24.079+0000 7f37c4800700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f37bc106410 0x7f37bc1a2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:24.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:24.079+0000 7f37c4800700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f37bc1104b0 con 0x7f37bc106410 2024-04-02T14:34:24.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:24.079+0000 7f37c4800700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f37bc110330 con 0x7f37bc105a70 2024-04-02T14:34:24.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:24.079+0000 7f37c4800700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f37bc110630 con 0x7f37bc104870 2024-04-02T14:34:24.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:24.080+0000 7f37c1e00700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f37bc104870 0x7f37bc19aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:24.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:24.080+0000 7f37c2800700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f37bc106410 0x7f37bc1a2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:24.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:24.080+0000 7f37c1e00700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f37bc104870 0x7f37bc19aa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:37496/0 (socket says 172.21.15.90:37496) 2024-04-02T14:34:24.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:24.080+0000 7f37c1e00700 1 -- 172.21.15.90:0/570506208 learned_addr learned my addr 172.21.15.90:0/570506208 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:34:24.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:24.080+0000 7f37c1400700 1 --2- 172.21.15.90:0/570506208 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f37bc105a70 0x7f37bc19af80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:24.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:24.080+0000 7f37c1e00700 1 -- 172.21.15.90:0/570506208 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f37bc105a70 msgr2=0x7f37bc19af80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:24.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:24.080+0000 7f37c1e00700 1 --2- 172.21.15.90:0/570506208 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f37bc105a70 0x7f37bc19af80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:24.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:24.080+0000 7f37c1e00700 1 -- 172.21.15.90:0/570506208 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f37bc106410 msgr2=0x7f37bc1a2000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:24.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:24.080+0000 7f37c1e00700 1 --2- 172.21.15.90:0/570506208 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f37bc106410 0x7f37bc1a2000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:24.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:24.080+0000 7f37c1e00700 1 -- 172.21.15.90:0/570506208 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37bc1a26a0 con 0x7f37bc104870 2024-04-02T14:34:24.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:24.081+0000 7f37c1e00700 1 --2- 172.21.15.90:0/570506208 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f37bc104870 0x7f37bc19aa40 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f37b000ea80 tx=0x7f37b000ed90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:34:24.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:24.081+0000 7f37b6a00700 1 -- 172.21.15.90:0/570506208 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37b000cbd0 con 0x7f37bc104870 2024-04-02T14:34:24.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:24.081+0000 7f37b6a00700 1 -- 172.21.15.90:0/570506208 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f37b000cd70 con 0x7f37bc104870 2024-04-02T14:34:24.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:24.081+0000 7f37b6a00700 1 -- 172.21.15.90:0/570506208 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37b0018780 con 0x7f37bc104870 2024-04-02T14:34:24.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:24.081+0000 7f37c4800700 1 -- 172.21.15.90:0/570506208 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f37bc1a2990 con 0x7f37bc104870 2024-04-02T14:34:24.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:24.081+0000 7f37c4800700 1 -- 172.21.15.90:0/570506208 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37bc1a2f20 con 0x7f37bc104870 2024-04-02T14:34:24.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:24.082+0000 7f37c4800700 1 -- 172.21.15.90:0/570506208 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f37bc0692b0 con 0x7f37bc104870 2024-04-02T14:34:24.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:24.084+0000 7f37b6a00700 1 -- 172.21.15.90:0/570506208 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f37b001f020 con 0x7f37bc104870 2024-04-02T14:34:24.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:24.085+0000 7f37b6a00700 1 --2- 172.21.15.90:0/570506208 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f37a40a3cc0 0x7f37a40a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:24.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:24.086+0000 7f37b6a00700 1 -- 172.21.15.90:0/570506208 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8248+0+0 (secure 0 0 0) 0x7f37b00d0b60 con 0x7f37bc104870 2024-04-02T14:34:24.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:24.086+0000 7f37c1400700 1 --2- 172.21.15.90:0/570506208 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f37a40a3cc0 0x7f37a40a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:24.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:24.086+0000 7f37c1400700 1 --2- 172.21.15.90:0/570506208 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f37a40a3cc0 0x7f37a40a6180 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f37ac000c00 tx=0x7f37ac005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:34:24.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:24.088+0000 7f37b6a00700 1 -- 172.21.15.90:0/570506208 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7f37b0026070 con 0x7f37bc104870 2024-04-02T14:34:24.110 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:34:23 smithi092 ceph-mon[36497]: pgmap v374: 97 pgs: 97 active+clean; 644 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-02T14:34:24.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:34:23 smithi177 ceph-mon[36431]: pgmap v374: 97 pgs: 97 active+clean; 644 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-02T14:34:24.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:34:23 smithi090 ceph-mon[31674]: pgmap v374: 97 pgs: 97 active+clean; 644 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-02T14:34:24.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:24.321+0000 7f37c4800700 1 -- 172.21.15.90:0/570506208 --> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) -- 0x7f37bc062a30 con 0x7f37a40a3cc0 2024-04-02T14:34:24.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:24.323+0000 7f37b6a00700 1 -- 172.21.15.90:0/570506208 <== mgr.14152 v2:172.21.15.90:6800/4266054817 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+191 (secure 0 0 0) 0x7f37bc062a30 con 0x7f37a40a3cc0 2024-04-02T14:34:24.326 INFO:teuthology.orchestra.run.smithi090.stdout:HOST ADDR LABELS STATUS 2024-04-02T14:34:24.326 INFO:teuthology.orchestra.run.smithi090.stdout:smithi090 172.21.15.90 2024-04-02T14:34:24.327 INFO:teuthology.orchestra.run.smithi090.stdout:smithi092 172.21.15.92 2024-04-02T14:34:24.327 INFO:teuthology.orchestra.run.smithi090.stdout:smithi177 172.21.15.177 2024-04-02T14:34:24.327 INFO:teuthology.orchestra.run.smithi090.stdout:3 hosts in cluster 2024-04-02T14:34:24.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:24.326+0000 7f37c4800700 1 -- 172.21.15.90:0/570506208 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f37a40a3cc0 msgr2=0x7f37a40a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:24.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:24.326+0000 7f37c4800700 1 --2- 172.21.15.90:0/570506208 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f37a40a3cc0 0x7f37a40a6180 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f37ac000c00 tx=0x7f37ac005fb0 comp rx=0 tx=0).stop 2024-04-02T14:34:24.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:24.326+0000 7f37c4800700 1 -- 172.21.15.90:0/570506208 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f37bc104870 msgr2=0x7f37bc19aa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:24.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:24.326+0000 7f37c4800700 1 --2- 172.21.15.90:0/570506208 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f37bc104870 0x7f37bc19aa40 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f37b000ea80 tx=0x7f37b000ed90 comp rx=0 tx=0).stop 2024-04-02T14:34:24.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:24.326+0000 7f37c4800700 1 -- 172.21.15.90:0/570506208 shutdown_connections 2024-04-02T14:34:24.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:24.326+0000 7f37c4800700 1 --2- 172.21.15.90:0/570506208 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f37a40a3cc0 0x7f37a40a6180 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:24.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:24.326+0000 7f37c4800700 1 --2- 172.21.15.90:0/570506208 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f37bc106410 0x7f37bc1a2000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:24.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:24.326+0000 7f37c4800700 1 --2- 172.21.15.90:0/570506208 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f37bc104870 0x7f37bc19aa40 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:24.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:24.326+0000 7f37c4800700 1 --2- 172.21.15.90:0/570506208 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f37bc105a70 0x7f37bc19af80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:24.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:24.326+0000 7f37c4800700 1 -- 172.21.15.90:0/570506208 >> 172.21.15.90:0/570506208 conn(0x7f37bc0ffe10 msgr2=0x7f37bc108d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:34:24.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:24.326+0000 7f37c4800700 1 -- 172.21.15.90:0/570506208 shutdown_connections 2024-04-02T14:34:24.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:24.326+0000 7f37c4800700 1 -- 172.21.15.90:0/570506208 wait complete. 2024-04-02T14:34:25.143 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:191d963ac9de7322da8aff20b567b3fa458e2ef6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6449448a-f0fc-11ee-b647-cb9ed24678a4 -- bash -c 'ceph orch device ls' 2024-04-02T14:34:25.589 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T14:34:26.045 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:34:25 smithi090 ceph-mon[31674]: from='client.25078 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T14:34:26.045 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:34:25 smithi090 ceph-mon[31674]: pgmap v375: 97 pgs: 97 active+clean; 644 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-02T14:34:26.110 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:34:25 smithi092 ceph-mon[36497]: from='client.25078 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T14:34:26.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:34:25 smithi092 ceph-mon[36497]: pgmap v375: 97 pgs: 97 active+clean; 644 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-02T14:34:26.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:34:25 smithi177 ceph-mon[36431]: from='client.25078 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T14:34:26.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:34:25 smithi177 ceph-mon[36431]: pgmap v375: 97 pgs: 97 active+clean; 644 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-02T14:34:27.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:27.788+0000 7fa6e3e00700 1 -- 172.21.15.90:0/3501681580 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa6dc1041c0 msgr2=0x7fa6dc10aa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:27.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:27.788+0000 7fa6e3e00700 1 --2- 172.21.15.90:0/3501681580 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa6dc1041c0 0x7fa6dc10aa70 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fa6d4009920 tx=0x7fa6d4009c30 comp rx=0 tx=0).stop 2024-04-02T14:34:27.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:27.788+0000 7fa6e3e00700 1 -- 172.21.15.90:0/3501681580 shutdown_connections 2024-04-02T14:34:27.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:27.788+0000 7fa6e3e00700 1 --2- 172.21.15.90:0/3501681580 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa6dc102620 0x7fa6dc102a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:27.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:27.788+0000 7fa6e3e00700 1 --2- 172.21.15.90:0/3501681580 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa6dc1041c0 0x7fa6dc10aa70 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:27.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:27.788+0000 7fa6e3e00700 1 --2- 172.21.15.90:0/3501681580 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa6dc103820 0x7fa6dc103c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:27.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:27.788+0000 7fa6e3e00700 1 -- 172.21.15.90:0/3501681580 >> 172.21.15.90:0/3501681580 conn(0x7fa6dc0fdc00 msgr2=0x7fa6dc100020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:34:27.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:27.788+0000 7fa6e3e00700 1 -- 172.21.15.90:0/3501681580 shutdown_connections 2024-04-02T14:34:27.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:27.788+0000 7fa6e3e00700 1 -- 172.21.15.90:0/3501681580 wait complete. 2024-04-02T14:34:27.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:27.789+0000 7fa6e3e00700 1 Processor -- start 2024-04-02T14:34:27.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:27.790+0000 7fa6e3e00700 1 -- start start 2024-04-02T14:34:27.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:27.790+0000 7fa6e3e00700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa6dc102620 0x7fa6dc1988c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:27.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:27.790+0000 7fa6e3e00700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa6dc103820 0x7fa6dc198e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:27.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:27.790+0000 7fa6e3e00700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa6dc1041c0 0x7fa6dc19fe80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:27.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:27.790+0000 7fa6e3e00700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fa6dc10e260 con 0x7fa6dc1041c0 2024-04-02T14:34:27.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:27.790+0000 7fa6e3e00700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fa6dc10e0e0 con 0x7fa6dc102620 2024-04-02T14:34:27.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:27.790+0000 7fa6e3e00700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fa6dc10e3e0 con 0x7fa6dc103820 2024-04-02T14:34:27.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:27.790+0000 7fa6e1400700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa6dc102620 0x7fa6dc1988c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:27.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:27.790+0000 7fa6e0a00700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa6dc103820 0x7fa6dc198e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:27.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:27.790+0000 7fa6e1400700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa6dc102620 0x7fa6dc1988c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:57384/0 (socket says 172.21.15.90:57384) 2024-04-02T14:34:27.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:27.790+0000 7fa6e1400700 1 -- 172.21.15.90:0/362182741 learned_addr learned my addr 172.21.15.90:0/362182741 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:34:27.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:27.790+0000 7fa6e1e00700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa6dc1041c0 0x7fa6dc19fe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:27.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:27.790+0000 7fa6e0a00700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa6dc103820 0x7fa6dc198e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:37520/0 (socket says 172.21.15.90:37520) 2024-04-02T14:34:27.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:27.791+0000 7fa6e1400700 1 -- 172.21.15.90:0/362182741 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa6dc103820 msgr2=0x7fa6dc198e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:27.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:27.791+0000 7fa6e1400700 1 --2- 172.21.15.90:0/362182741 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa6dc103820 0x7fa6dc198e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:27.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:27.791+0000 7fa6e1400700 1 -- 172.21.15.90:0/362182741 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa6dc1041c0 msgr2=0x7fa6dc19fe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:27.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:27.791+0000 7fa6e1400700 1 --2- 172.21.15.90:0/362182741 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa6dc1041c0 0x7fa6dc19fe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:27.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:27.791+0000 7fa6e1400700 1 -- 172.21.15.90:0/362182741 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6dc1a0520 con 0x7fa6dc102620 2024-04-02T14:34:27.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:27.791+0000 7fa6e0a00700 1 --2- 172.21.15.90:0/362182741 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa6dc103820 0x7fa6dc198e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:34:27.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:27.791+0000 7fa6e1e00700 1 --2- 172.21.15.90:0/362182741 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa6dc1041c0 0x7fa6dc19fe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:34:27.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:27.791+0000 7fa6e1400700 1 --2- 172.21.15.90:0/362182741 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa6dc102620 0x7fa6dc1988c0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fa6d000efb0 tx=0x7fa6d000c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:34:27.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:27.791+0000 7fa6da000700 1 -- 172.21.15.90:0/362182741 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6d00092e0 con 0x7fa6dc102620 2024-04-02T14:34:27.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:27.791+0000 7fa6da000700 1 -- 172.21.15.90:0/362182741 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa6d0003a00 con 0x7fa6dc102620 2024-04-02T14:34:27.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:27.791+0000 7fa6e3e00700 1 -- 172.21.15.90:0/362182741 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6dc1a07b0 con 0x7fa6dc102620 2024-04-02T14:34:27.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:27.791+0000 7fa6da000700 1 -- 172.21.15.90:0/362182741 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6d0004700 con 0x7fa6dc102620 2024-04-02T14:34:27.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:27.792+0000 7fa6e3e00700 1 -- 172.21.15.90:0/362182741 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6dc1a0d40 con 0x7fa6dc102620 2024-04-02T14:34:27.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:27.793+0000 7fa6e3e00700 1 -- 172.21.15.90:0/362182741 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa6dc0692b0 con 0x7fa6dc102620 2024-04-02T14:34:27.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:27.795+0000 7fa6da000700 1 -- 172.21.15.90:0/362182741 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fa6d001b030 con 0x7fa6dc102620 2024-04-02T14:34:27.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:27.796+0000 7fa6da000700 1 --2- 172.21.15.90:0/362182741 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fa6c40a3cc0 0x7fa6c40a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:27.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:27.796+0000 7fa6da000700 1 -- 172.21.15.90:0/362182741 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8248+0+0 (secure 0 0 0) 0x7fa6d00d04a0 con 0x7fa6dc102620 2024-04-02T14:34:27.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:27.796+0000 7fa6e0a00700 1 --2- 172.21.15.90:0/362182741 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fa6c40a3cc0 0x7fa6c40a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:27.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:27.797+0000 7fa6e0a00700 1 --2- 172.21.15.90:0/362182741 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fa6c40a3cc0 0x7fa6c40a6180 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fa6c8011490 tx=0x7fa6c8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:34:27.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:27.799+0000 7fa6da000700 1 -- 172.21.15.90:0/362182741 <== mon.2 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7fa6d0022020 con 0x7fa6dc102620 2024-04-02T14:34:27.938 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:34:27 smithi090 ceph-mon[31674]: pgmap v376: 97 pgs: 97 active+clean; 644 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-02T14:34:28.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:28.036+0000 7fa6e3e00700 1 -- 172.21.15.90:0/362182741 --> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) -- 0x7fa6dc062a30 con 0x7fa6c40a3cc0 2024-04-02T14:34:28.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:28.040+0000 7fa6da000700 1 -- 172.21.15.90:0/362182741 <== mgr.14152 v2:172.21.15.90:6800/4266054817 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+532 (secure 0 0 0) 0x7fa6dc062a30 con 0x7fa6c40a3cc0 2024-04-02T14:34:28.042 INFO:teuthology.orchestra.run.smithi090.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-04-02T14:34:28.042 INFO:teuthology.orchestra.run.smithi090.stdout:smithi090 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620400UK400BGN 372G No 3m ago Has a FileSystem, LVM detected 2024-04-02T14:34:28.042 INFO:teuthology.orchestra.run.smithi090.stdout:smithi092 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620400QM400BGN 372G No 3m ago Has a FileSystem, LVM detected 2024-04-02T14:34:28.042 INFO:teuthology.orchestra.run.smithi090.stdout:smithi177 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT6233007S400BGN 372G No 3m ago Has a FileSystem, LVM detected 2024-04-02T14:34:28.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:28.043+0000 7fa6e3e00700 1 -- 172.21.15.90:0/362182741 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fa6c40a3cc0 msgr2=0x7fa6c40a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:28.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:28.043+0000 7fa6e3e00700 1 --2- 172.21.15.90:0/362182741 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fa6c40a3cc0 0x7fa6c40a6180 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fa6c8011490 tx=0x7fa6c8009260 comp rx=0 tx=0).stop 2024-04-02T14:34:28.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:28.043+0000 7fa6e3e00700 1 -- 172.21.15.90:0/362182741 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa6dc102620 msgr2=0x7fa6dc1988c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:28.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:28.043+0000 7fa6e3e00700 1 --2- 172.21.15.90:0/362182741 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa6dc102620 0x7fa6dc1988c0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fa6d000efb0 tx=0x7fa6d000c3a0 comp rx=0 tx=0).stop 2024-04-02T14:34:28.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:28.043+0000 7fa6e3e00700 1 -- 172.21.15.90:0/362182741 shutdown_connections 2024-04-02T14:34:28.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:28.043+0000 7fa6e3e00700 1 --2- 172.21.15.90:0/362182741 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fa6c40a3cc0 0x7fa6c40a6180 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:28.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:28.043+0000 7fa6e3e00700 1 --2- 172.21.15.90:0/362182741 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa6dc1041c0 0x7fa6dc19fe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:28.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:28.043+0000 7fa6e3e00700 1 --2- 172.21.15.90:0/362182741 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa6dc102620 0x7fa6dc1988c0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:28.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:28.043+0000 7fa6e3e00700 1 --2- 172.21.15.90:0/362182741 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa6dc103820 0x7fa6dc198e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:28.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:28.043+0000 7fa6e3e00700 1 -- 172.21.15.90:0/362182741 >> 172.21.15.90:0/362182741 conn(0x7fa6dc0fdc00 msgr2=0x7fa6dc106bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:34:28.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:28.043+0000 7fa6e3e00700 1 -- 172.21.15.90:0/362182741 shutdown_connections 2024-04-02T14:34:28.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:28.043+0000 7fa6e3e00700 1 -- 172.21.15.90:0/362182741 wait complete. 2024-04-02T14:34:28.110 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:34:27 smithi092 ceph-mon[36497]: pgmap v376: 97 pgs: 97 active+clean; 644 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-02T14:34:28.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:34:27 smithi177 ceph-mon[36431]: pgmap v376: 97 pgs: 97 active+clean; 644 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-02T14:34:28.861 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-02T14:34:28.870 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi090.front.sepia.ceph.com 2024-04-02T14:34:28.871 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:191d963ac9de7322da8aff20b567b3fa458e2ef6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6449448a-f0fc-11ee-b647-cb9ed24678a4 -- bash -c 'ceph fs dump' 2024-04-02T14:34:29.319 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T14:34:30.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:34:29 smithi092 ceph-mon[36497]: from='client.25103 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T14:34:30.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:34:29 smithi092 ceph-mon[36497]: pgmap v377: 97 pgs: 97 active+clean; 644 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-02T14:34:30.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:34:29 smithi177 ceph-mon[36431]: from='client.25103 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T14:34:30.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:34:29 smithi177 ceph-mon[36431]: pgmap v377: 97 pgs: 97 active+clean; 644 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-02T14:34:30.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:34:29 smithi090 ceph-mon[31674]: from='client.25103 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T14:34:30.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:34:29 smithi090 ceph-mon[31674]: pgmap v377: 97 pgs: 97 active+clean; 644 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-02T14:34:31.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:31.362+0000 7fc62a600700 1 -- 172.21.15.90:0/2774184613 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc624075050 msgr2=0x7fc6240754d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:31.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:31.362+0000 7fc62a600700 1 --2- 172.21.15.90:0/2774184613 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc624075050 0x7fc6240754d0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fc610009990 tx=0x7fc610009ca0 comp rx=0 tx=0).stop 2024-04-02T14:34:31.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:31.362+0000 7fc62a600700 1 -- 172.21.15.90:0/2774184613 shutdown_connections 2024-04-02T14:34:31.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:31.362+0000 7fc62a600700 1 --2- 172.21.15.90:0/2774184613 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc62410ec10 0x7fc624111000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:31.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:31.362+0000 7fc62a600700 1 --2- 172.21.15.90:0/2774184613 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc624076620 0x7fc624074a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:31.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:31.362+0000 7fc62a600700 1 --2- 172.21.15.90:0/2774184613 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc624075050 0x7fc6240754d0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:31.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:31.362+0000 7fc62a600700 1 -- 172.21.15.90:0/2774184613 >> 172.21.15.90:0/2774184613 conn(0x7fc6240fdb90 msgr2=0x7fc624100010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:34:31.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:31.362+0000 7fc62a600700 1 -- 172.21.15.90:0/2774184613 shutdown_connections 2024-04-02T14:34:31.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:31.363+0000 7fc62a600700 1 -- 172.21.15.90:0/2774184613 wait complete. 2024-04-02T14:34:31.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:31.364+0000 7fc62a600700 1 Processor -- start 2024-04-02T14:34:31.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:31.364+0000 7fc62a600700 1 -- start start 2024-04-02T14:34:31.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:31.364+0000 7fc62a600700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc624075050 0x7fc62419eee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:31.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:31.364+0000 7fc62a600700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc624076620 0x7fc62419f420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:31.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:31.364+0000 7fc62a600700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc62410ec10 0x7fc6241a64a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:31.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:31.364+0000 7fc62a600700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fc624114860 con 0x7fc624075050 2024-04-02T14:34:31.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:31.364+0000 7fc62a600700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fc6241146e0 con 0x7fc624076620 2024-04-02T14:34:31.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:31.364+0000 7fc62a600700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fc6241149e0 con 0x7fc62410ec10 2024-04-02T14:34:31.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:31.365+0000 7fc623400700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc624075050 0x7fc62419eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:31.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:31.365+0000 7fc623400700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc624075050 0x7fc62419eee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34044/0 (socket says 172.21.15.90:34044) 2024-04-02T14:34:31.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:31.365+0000 7fc623400700 1 -- 172.21.15.90:0/620055 learned_addr learned my addr 172.21.15.90:0/620055 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:34:31.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:31.365+0000 7fc622a00700 1 --2- 172.21.15.90:0/620055 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc624076620 0x7fc62419f420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:31.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:31.365+0000 7fc623e00700 1 --2- 172.21.15.90:0/620055 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc62410ec10 0x7fc6241a64a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:31.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:31.365+0000 7fc623400700 1 -- 172.21.15.90:0/620055 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc62410ec10 msgr2=0x7fc6241a64a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:31.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:31.365+0000 7fc623400700 1 --2- 172.21.15.90:0/620055 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc62410ec10 0x7fc6241a64a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:31.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:31.365+0000 7fc623400700 1 -- 172.21.15.90:0/620055 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc624076620 msgr2=0x7fc62419f420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:31.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:31.365+0000 7fc623400700 1 --2- 172.21.15.90:0/620055 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc624076620 0x7fc62419f420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:31.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:31.365+0000 7fc623400700 1 -- 172.21.15.90:0/620055 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6241a6b40 con 0x7fc624075050 2024-04-02T14:34:31.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:31.365+0000 7fc622a00700 1 --2- 172.21.15.90:0/620055 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc624076620 0x7fc62419f420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:34:31.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:31.366+0000 7fc623400700 1 --2- 172.21.15.90:0/620055 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc624075050 0x7fc62419eee0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fc61400e9a0 tx=0x7fc61400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:34:31.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:31.366+0000 7fc61be00700 1 -- 172.21.15.90:0/620055 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc614004a30 con 0x7fc624075050 2024-04-02T14:34:31.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:31.366+0000 7fc61be00700 1 -- 172.21.15.90:0/620055 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc61400ccf0 con 0x7fc624075050 2024-04-02T14:34:31.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:31.366+0000 7fc62a600700 1 -- 172.21.15.90:0/620055 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc6241a6e30 con 0x7fc624075050 2024-04-02T14:34:31.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:31.366+0000 7fc61be00700 1 -- 172.21.15.90:0/620055 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc614005680 con 0x7fc624075050 2024-04-02T14:34:31.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:31.366+0000 7fc62a600700 1 -- 172.21.15.90:0/620055 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6241a7290 con 0x7fc624075050 2024-04-02T14:34:31.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:31.367+0000 7fc62a600700 1 -- 172.21.15.90:0/620055 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc6240692b0 con 0x7fc624075050 2024-04-02T14:34:31.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:31.369+0000 7fc61be00700 1 -- 172.21.15.90:0/620055 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fc61400c810 con 0x7fc624075050 2024-04-02T14:34:31.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:31.370+0000 7fc61be00700 1 --2- 172.21.15.90:0/620055 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fc6080a3d40 0x7fc6080a6200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:31.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:31.371+0000 7fc61be00700 1 -- 172.21.15.90:0/620055 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8248+0+0 (secure 0 0 0) 0x7fc6140d0530 con 0x7fc624075050 2024-04-02T14:34:31.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:31.371+0000 7fc622a00700 1 --2- 172.21.15.90:0/620055 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fc6080a3d40 0x7fc6080a6200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:31.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:31.371+0000 7fc622a00700 1 --2- 172.21.15.90:0/620055 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fc6080a3d40 0x7fc6080a6200 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fc61000b7b0 tx=0x7fc610005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:34:31.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:31.375+0000 7fc61be00700 1 -- 172.21.15.90:0/620055 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7fc61409cea0 con 0x7fc624075050 2024-04-02T14:34:31.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:31.652+0000 7fc62a600700 1 -- 172.21.15.90:0/620055 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs dump"} v 0) -- 0x7fc6240632a0 con 0x7fc624075050 2024-04-02T14:34:31.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:31.653+0000 7fc61be00700 1 -- 172.21.15.90:0/620055 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 26 v26) ==== 76+0+3086 (secure 0 0 0) 0x7fc61409c7a0 con 0x7fc624075050 2024-04-02T14:34:31.656 INFO:teuthology.orchestra.run.smithi090.stdout:e26 2024-04-02T14:34:31.656 INFO:teuthology.orchestra.run.smithi090.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-02T14:34:31.656 INFO:teuthology.orchestra.run.smithi090.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-02T14:34:31.656 INFO:teuthology.orchestra.run.smithi090.stdout:legacy client fscid: 1 2024-04-02T14:34:31.657 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-02T14:34:31.657 INFO:teuthology.orchestra.run.smithi090.stdout:Filesystem 'cephfs' (1) 2024-04-02T14:34:31.657 INFO:teuthology.orchestra.run.smithi090.stdout:fs_name cephfs 2024-04-02T14:34:31.657 INFO:teuthology.orchestra.run.smithi090.stdout:epoch 26 2024-04-02T14:34:31.657 INFO:teuthology.orchestra.run.smithi090.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-04-02T14:34:31.657 INFO:teuthology.orchestra.run.smithi090.stdout:created 2024-04-02T14:32:22.319083+0000 2024-04-02T14:34:31.657 INFO:teuthology.orchestra.run.smithi090.stdout:modified 2024-04-02T14:32:29.812071+0000 2024-04-02T14:34:31.657 INFO:teuthology.orchestra.run.smithi090.stdout:tableserver 0 2024-04-02T14:34:31.657 INFO:teuthology.orchestra.run.smithi090.stdout:root 0 2024-04-02T14:34:31.657 INFO:teuthology.orchestra.run.smithi090.stdout:session_timeout 300 2024-04-02T14:34:31.657 INFO:teuthology.orchestra.run.smithi090.stdout:session_autoclose 300 2024-04-02T14:34:31.657 INFO:teuthology.orchestra.run.smithi090.stdout:max_file_size 1099511627776 2024-04-02T14:34:31.657 INFO:teuthology.orchestra.run.smithi090.stdout:max_xattr_size 65536 2024-04-02T14:34:31.657 INFO:teuthology.orchestra.run.smithi090.stdout:required_client_features {} 2024-04-02T14:34:31.657 INFO:teuthology.orchestra.run.smithi090.stdout:last_failure 0 2024-04-02T14:34:31.658 INFO:teuthology.orchestra.run.smithi090.stdout:last_failure_osd_epoch 0 2024-04-02T14:34:31.658 INFO:teuthology.orchestra.run.smithi090.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-02T14:34:31.658 INFO:teuthology.orchestra.run.smithi090.stdout:max_mds 5 2024-04-02T14:34:31.658 INFO:teuthology.orchestra.run.smithi090.stdout:in 0,1,2,3,4 2024-04-02T14:34:31.658 INFO:teuthology.orchestra.run.smithi090.stdout:up {0=24590,1=24560,2=24547,3=24539,4=24523} 2024-04-02T14:34:31.658 INFO:teuthology.orchestra.run.smithi090.stdout:failed 2024-04-02T14:34:31.658 INFO:teuthology.orchestra.run.smithi090.stdout:damaged 2024-04-02T14:34:31.658 INFO:teuthology.orchestra.run.smithi090.stdout:stopped 2024-04-02T14:34:31.658 INFO:teuthology.orchestra.run.smithi090.stdout:data_pools [3] 2024-04-02T14:34:31.658 INFO:teuthology.orchestra.run.smithi090.stdout:metadata_pool 2 2024-04-02T14:34:31.658 INFO:teuthology.orchestra.run.smithi090.stdout:inline_data disabled 2024-04-02T14:34:31.658 INFO:teuthology.orchestra.run.smithi090.stdout:balancer 2024-04-02T14:34:31.658 INFO:teuthology.orchestra.run.smithi090.stdout:bal_rank_mask -1 2024-04-02T14:34:31.658 INFO:teuthology.orchestra.run.smithi090.stdout:standby_count_wanted 0 2024-04-02T14:34:31.659 INFO:teuthology.orchestra.run.smithi090.stdout:qdb_cluster leader: 24590 members: 24539,24560,24523,24590,24547 2024-04-02T14:34:31.659 INFO:teuthology.orchestra.run.smithi090.stdout:[mds.b{0:24590} state up:active seq 3 addr [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] compat {c=[1],r=[1],i=[1fff]}] 2024-04-02T14:34:31.659 INFO:teuthology.orchestra.run.smithi090.stdout:[mds.f{0:24511} state up:standby-replay seq 1 addr [v2:172.21.15.177:6834/3177244988,v1:172.21.15.177:6835/3177244988] compat {c=[1],r=[1],i=[1fff]}] 2024-04-02T14:34:31.659 INFO:teuthology.orchestra.run.smithi090.stdout:[mds.g{1:24560} state up:active seq 5 addr [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] compat {c=[1],r=[1],i=[1fff]}] 2024-04-02T14:34:31.659 INFO:teuthology.orchestra.run.smithi090.stdout:[mds.e{1:24503} state up:standby-replay seq 1 addr [v2:172.21.15.92:6834/860721164,v1:172.21.15.92:6835/860721164] compat {c=[1],r=[1],i=[1fff]}] 2024-04-02T14:34:31.659 INFO:teuthology.orchestra.run.smithi090.stdout:[mds.i{2:24547} state up:active seq 7 addr [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] compat {c=[1],r=[1],i=[1fff]}] 2024-04-02T14:34:31.659 INFO:teuthology.orchestra.run.smithi090.stdout:[mds.j{2:24478} state up:standby-replay seq 1 addr [v2:172.21.15.90:6834/1277124891,v1:172.21.15.90:6835/1277124891] compat {c=[1],r=[1],i=[1fff]}] 2024-04-02T14:34:31.659 INFO:teuthology.orchestra.run.smithi090.stdout:[mds.h{3:24539} state up:active seq 8 addr [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] compat {c=[1],r=[1],i=[1fff]}] 2024-04-02T14:34:31.659 INFO:teuthology.orchestra.run.smithi090.stdout:[mds.k{3:24461} state up:standby-replay seq 1 addr [v2:172.21.15.92:6832/1570337372,v1:172.21.15.92:6833/1570337372] compat {c=[1],r=[1],i=[1fff]}] 2024-04-02T14:34:31.659 INFO:teuthology.orchestra.run.smithi090.stdout:[mds.c{4:24523} state up:active seq 9 addr [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] compat {c=[1],r=[1],i=[1fff]}] 2024-04-02T14:34:31.659 INFO:teuthology.orchestra.run.smithi090.stdout:[mds.l{4:24454} state up:standby-replay seq 1 addr [v2:172.21.15.177:6832/1564319431,v1:172.21.15.177:6833/1564319431] compat {c=[1],r=[1],i=[1fff]}] 2024-04-02T14:34:31.659 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-02T14:34:31.659 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-02T14:34:31.659 INFO:teuthology.orchestra.run.smithi090.stdout:Standby daemons: 2024-04-02T14:34:31.659 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-02T14:34:31.660 INFO:teuthology.orchestra.run.smithi090.stdout:[mds.d{-1:14412} state up:standby seq 1 addr [v2:172.21.15.90:6836/2852160309,v1:172.21.15.90:6837/2852160309] compat {c=[1],r=[1],i=[1fff]}] 2024-04-02T14:34:31.660 INFO:teuthology.orchestra.run.smithi090.stdout:[mds.a{-1:14478} state up:standby seq 1 addr [v2:172.21.15.90:6840/1919376858,v1:172.21.15.90:6841/1919376858] compat {c=[1],r=[1],i=[1fff]}] 2024-04-02T14:34:31.660 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:31.656+0000 7fc62a600700 1 -- 172.21.15.90:0/620055 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fc6080a3d40 msgr2=0x7fc6080a6200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:31.660 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:31.656+0000 7fc62a600700 1 --2- 172.21.15.90:0/620055 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fc6080a3d40 0x7fc6080a6200 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fc61000b7b0 tx=0x7fc610005fd0 comp rx=0 tx=0).stop 2024-04-02T14:34:31.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:31.657+0000 7fc62a600700 1 -- 172.21.15.90:0/620055 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc624075050 msgr2=0x7fc62419eee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:31.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:31.657+0000 7fc62a600700 1 --2- 172.21.15.90:0/620055 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc624075050 0x7fc62419eee0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fc61400e9a0 tx=0x7fc61400ecb0 comp rx=0 tx=0).stop 2024-04-02T14:34:31.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:31.657+0000 7fc62a600700 1 -- 172.21.15.90:0/620055 shutdown_connections 2024-04-02T14:34:31.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:31.657+0000 7fc62a600700 1 --2- 172.21.15.90:0/620055 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fc6080a3d40 0x7fc6080a6200 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:31.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:31.657+0000 7fc62a600700 1 --2- 172.21.15.90:0/620055 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc624075050 0x7fc62419eee0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:31.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:31.657+0000 7fc62a600700 1 --2- 172.21.15.90:0/620055 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc62410ec10 0x7fc6241a64a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:31.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:31.657+0000 7fc62a600700 1 --2- 172.21.15.90:0/620055 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc624076620 0x7fc62419f420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:31.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:31.657+0000 7fc62a600700 1 -- 172.21.15.90:0/620055 >> 172.21.15.90:0/620055 conn(0x7fc6240fdb90 msgr2=0x7fc624101fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:34:31.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:31.657+0000 7fc62a600700 1 -- 172.21.15.90:0/620055 shutdown_connections 2024-04-02T14:34:31.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:31.657+0000 7fc62a600700 1 -- 172.21.15.90:0/620055 wait complete. 2024-04-02T14:34:31.661 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 26 2024-04-02T14:34:31.939 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:34:31 smithi090 ceph-mon[31674]: pgmap v378: 97 pgs: 97 active+clean; 644 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-02T14:34:31.939 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:34:31 smithi090 ceph-mon[31674]: from='client.? 172.21.15.90:0/620055' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-02T14:34:32.110 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:34:31 smithi092 ceph-mon[36497]: pgmap v378: 97 pgs: 97 active+clean; 644 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-02T14:34:32.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:34:31 smithi092 ceph-mon[36497]: from='client.? 172.21.15.90:0/620055' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-02T14:34:32.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:34:31 smithi177 ceph-mon[36431]: pgmap v378: 97 pgs: 97 active+clean; 644 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-02T14:34:32.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:34:31 smithi177 ceph-mon[36431]: from='client.? 172.21.15.90:0/620055' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-02T14:34:32.464 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:191d963ac9de7322da8aff20b567b3fa458e2ef6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6449448a-f0fc-11ee-b647-cb9ed24678a4 -- bash -c 'ceph osd dump' 2024-04-02T14:34:32.914 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T14:34:34.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:34:33 smithi177 ceph-mon[36431]: pgmap v379: 97 pgs: 97 active+clean; 644 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-02T14:34:34.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:34:33 smithi090 ceph-mon[31674]: pgmap v379: 97 pgs: 97 active+clean; 644 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-02T14:34:34.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:34:33 smithi092 ceph-mon[36497]: pgmap v379: 97 pgs: 97 active+clean; 644 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-02T14:34:35.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:35.040+0000 7f0f02400700 1 -- 172.21.15.90:0/2284952311 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0efc105970 msgr2=0x7f0efc105dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:35.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:35.040+0000 7f0f02400700 1 --2- 172.21.15.90:0/2284952311 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0efc105970 0x7f0efc105dd0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f0ee8009920 tx=0x7f0ee8009c30 comp rx=0 tx=0).stop 2024-04-02T14:34:35.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:35.040+0000 7f0f02400700 1 -- 172.21.15.90:0/2284952311 shutdown_connections 2024-04-02T14:34:35.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:35.040+0000 7f0f02400700 1 --2- 172.21.15.90:0/2284952311 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0efc104770 0x7f0efc104b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:35.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:35.040+0000 7f0f02400700 1 --2- 172.21.15.90:0/2284952311 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0efc106310 0x7f0efc10cbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:35.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:35.040+0000 7f0f02400700 1 --2- 172.21.15.90:0/2284952311 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0efc105970 0x7f0efc105dd0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:35.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:35.040+0000 7f0f02400700 1 -- 172.21.15.90:0/2284952311 >> 172.21.15.90:0/2284952311 conn(0x7f0efc0ffd10 msgr2=0x7f0efc102150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:34:35.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:35.040+0000 7f0f02400700 1 -- 172.21.15.90:0/2284952311 shutdown_connections 2024-04-02T14:34:35.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:35.040+0000 7f0f02400700 1 -- 172.21.15.90:0/2284952311 wait complete. 2024-04-02T14:34:35.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:35.041+0000 7f0f02400700 1 Processor -- start 2024-04-02T14:34:35.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:35.042+0000 7f0f02400700 1 -- start start 2024-04-02T14:34:35.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:35.042+0000 7f0f02400700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0efc104770 0x7f0efc19aa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:35.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:35.042+0000 7f0f02400700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0efc105970 0x7f0efc19af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:35.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:35.042+0000 7f0f02400700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0efc106310 0x7f0efc1a1f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:35.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:35.042+0000 7f0f02400700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f0efc110330 con 0x7f0efc106310 2024-04-02T14:34:35.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:35.042+0000 7f0f02400700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f0efc1101b0 con 0x7f0efc104770 2024-04-02T14:34:35.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:35.042+0000 7f0f02400700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f0efc1104b0 con 0x7f0efc105970 2024-04-02T14:34:35.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:35.042+0000 7f0efbe00700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0efc105970 0x7f0efc19af40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:35.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:35.042+0000 7f0f01a00700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0efc106310 0x7f0efc1a1f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:35.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:35.042+0000 7f0efbe00700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0efc105970 0x7f0efc19af40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:52618/0 (socket says 172.21.15.90:52618) 2024-04-02T14:34:35.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:35.043+0000 7f0f01a00700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0efc106310 0x7f0efc1a1f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:36292/0 (socket says 172.21.15.90:36292) 2024-04-02T14:34:35.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:35.042+0000 7f0f01000700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0efc104770 0x7f0efc19aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:35.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:35.043+0000 7f0efbe00700 1 -- 172.21.15.90:0/1379178688 learned_addr learned my addr 172.21.15.90:0/1379178688 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:34:35.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:35.043+0000 7f0efbe00700 1 -- 172.21.15.90:0/1379178688 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0efc104770 msgr2=0x7f0efc19aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:35.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:35.043+0000 7f0efbe00700 1 --2- 172.21.15.90:0/1379178688 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0efc104770 0x7f0efc19aa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:35.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:35.043+0000 7f0efbe00700 1 -- 172.21.15.90:0/1379178688 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0efc106310 msgr2=0x7f0efc1a1f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:35.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:35.043+0000 7f0efbe00700 1 --2- 172.21.15.90:0/1379178688 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0efc106310 0x7f0efc1a1f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:35.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:35.043+0000 7f0efbe00700 1 -- 172.21.15.90:0/1379178688 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0efc1a2610 con 0x7f0efc105970 2024-04-02T14:34:35.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:35.043+0000 7f0efbe00700 1 --2- 172.21.15.90:0/1379178688 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0efc105970 0x7f0efc19af40 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f0ee800fca0 tx=0x7f0ee8005250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:34:35.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:35.043+0000 7f0ef9600700 1 -- 172.21.15.90:0/1379178688 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ee800f930 con 0x7f0efc105970 2024-04-02T14:34:35.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:35.044+0000 7f0ef9600700 1 -- 172.21.15.90:0/1379178688 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0ee800ba40 con 0x7f0efc105970 2024-04-02T14:34:35.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:35.044+0000 7f0ef9600700 1 -- 172.21.15.90:0/1379178688 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ee8003c20 con 0x7f0efc105970 2024-04-02T14:34:35.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:35.044+0000 7f0f02400700 1 -- 172.21.15.90:0/1379178688 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0efc1a28a0 con 0x7f0efc105970 2024-04-02T14:34:35.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:35.044+0000 7f0f02400700 1 -- 172.21.15.90:0/1379178688 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0efc1a2ca0 con 0x7f0efc105970 2024-04-02T14:34:35.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:35.045+0000 7f0f02400700 1 -- 172.21.15.90:0/1379178688 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0efc0692b0 con 0x7f0efc105970 2024-04-02T14:34:35.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:35.047+0000 7f0ef9600700 1 -- 172.21.15.90:0/1379178688 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f0ee80054d0 con 0x7f0efc105970 2024-04-02T14:34:35.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:35.048+0000 7f0ef9600700 1 --2- 172.21.15.90:0/1379178688 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f0ee40a3cc0 0x7f0ee40a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:35.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:35.048+0000 7f0ef9600700 1 -- 172.21.15.90:0/1379178688 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8248+0+0 (secure 0 0 0) 0x7f0ee80d1710 con 0x7f0efc105970 2024-04-02T14:34:35.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:35.048+0000 7f0f01000700 1 --2- 172.21.15.90:0/1379178688 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f0ee40a3cc0 0x7f0ee40a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:35.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:35.049+0000 7f0f01000700 1 --2- 172.21.15.90:0/1379178688 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f0ee40a3cc0 0x7f0ee40a6180 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f0ef0009750 tx=0x7f0ef0009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:34:35.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:35.051+0000 7f0ef9600700 1 -- 172.21.15.90:0/1379178688 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7f0ee8013030 con 0x7f0efc105970 2024-04-02T14:34:35.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:35.282+0000 7f0f02400700 1 -- 172.21.15.90:0/1379178688 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd dump"} v 0) -- 0x7f0efc02dd60 con 0x7f0efc105970 2024-04-02T14:34:35.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:35.283+0000 7f0ef9600700 1 -- 172.21.15.90:0/1379178688 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v90) ==== 56+0+5306 (secure 0 0 0) 0x7f0ee809e110 con 0x7f0efc105970 2024-04-02T14:34:35.286 INFO:teuthology.orchestra.run.smithi090.stdout:epoch 90 2024-04-02T14:34:35.286 INFO:teuthology.orchestra.run.smithi090.stdout:fsid 6449448a-f0fc-11ee-b647-cb9ed24678a4 2024-04-02T14:34:35.286 INFO:teuthology.orchestra.run.smithi090.stdout:created 2024-04-02T14:23:16.530201+0000 2024-04-02T14:34:35.286 INFO:teuthology.orchestra.run.smithi090.stdout:modified 2024-04-02T14:32:35.716929+0000 2024-04-02T14:34:35.286 INFO:teuthology.orchestra.run.smithi090.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-04-02T14:34:35.286 INFO:teuthology.orchestra.run.smithi090.stdout:crush_version 26 2024-04-02T14:34:35.286 INFO:teuthology.orchestra.run.smithi090.stdout:full_ratio 0.95 2024-04-02T14:34:35.286 INFO:teuthology.orchestra.run.smithi090.stdout:backfillfull_ratio 0.9 2024-04-02T14:34:35.287 INFO:teuthology.orchestra.run.smithi090.stdout:nearfull_ratio 0.85 2024-04-02T14:34:35.287 INFO:teuthology.orchestra.run.smithi090.stdout:require_min_compat_client luminous 2024-04-02T14:34:35.287 INFO:teuthology.orchestra.run.smithi090.stdout:min_compat_client luminous 2024-04-02T14:34:35.287 INFO:teuthology.orchestra.run.smithi090.stdout:require_osd_release squid 2024-04-02T14:34:35.287 INFO:teuthology.orchestra.run.smithi090.stdout:stretch_mode_enabled false 2024-04-02T14:34:35.287 INFO:teuthology.orchestra.run.smithi090.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 21 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 12.00 2024-04-02T14:34:35.287 INFO:teuthology.orchestra.run.smithi090.stdout:pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 87 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 2.25 2024-04-02T14:34:35.287 INFO:teuthology.orchestra.run.smithi090.stdout:pool 3 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode off last_change 88 flags hashpspool stripe_width 0 pg_num_min 64 target_size_ratio 0.9 application cephfs read_balance_score 1.50 2024-04-02T14:34:35.287 INFO:teuthology.orchestra.run.smithi090.stdout:max_osd 12 2024-04-02T14:34:35.287 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0 up in weight 1 up_from 8 up_thru 89 down_at 0 last_clean_interval [0,0) [v2:172.21.15.90:6802/3635087609,v1:172.21.15.90:6803/3635087609] [v2:172.21.15.90:6804/3635087609,v1:172.21.15.90:6805/3635087609] exists,up a1a6a856-ab44-44bd-bc5e-45c718a70cd2 2024-04-02T14:34:35.287 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1 up in weight 1 up_from 13 up_thru 86 down_at 0 last_clean_interval [0,0) [v2:172.21.15.92:6800/1994536059,v1:172.21.15.92:6801/1994536059] [v2:172.21.15.92:6802/1994536059,v1:172.21.15.92:6803/1994536059] exists,up 124139cd-5793-463e-9607-9d9c10164183 2024-04-02T14:34:35.287 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2 up in weight 1 up_from 18 up_thru 89 down_at 0 last_clean_interval [0,0) [v2:172.21.15.177:6800/1358472800,v1:172.21.15.177:6801/1358472800] [v2:172.21.15.177:6802/1358472800,v1:172.21.15.177:6803/1358472800] exists,up a61fbef2-9dca-4875-a814-020fff0db0e5 2024-04-02T14:34:35.287 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3 up in weight 1 up_from 25 up_thru 86 down_at 0 last_clean_interval [0,0) [v2:172.21.15.90:6810/3114811216,v1:172.21.15.90:6811/3114811216] [v2:172.21.15.90:6812/3114811216,v1:172.21.15.90:6813/3114811216] exists,up b175a2f4-7859-4eb0-bc1f-fba8c2b92a2a 2024-04-02T14:34:35.287 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4 up in weight 1 up_from 33 up_thru 86 down_at 0 last_clean_interval [0,0) [v2:172.21.15.92:6808/2987208879,v1:172.21.15.92:6809/2987208879] [v2:172.21.15.92:6810/2987208879,v1:172.21.15.92:6811/2987208879] exists,up 38cc68ea-c109-4d8d-b77d-1c6f9b56c875 2024-04-02T14:34:35.288 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5 up in weight 1 up_from 38 up_thru 86 down_at 0 last_clean_interval [0,0) [v2:172.21.15.177:6808/3486820237,v1:172.21.15.177:6809/3486820237] [v2:172.21.15.177:6810/3486820237,v1:172.21.15.177:6811/3486820237] exists,up 2347232e-ac91-466f-8ac9-496f397679ba 2024-04-02T14:34:35.288 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6 up in weight 1 up_from 46 up_thru 86 down_at 0 last_clean_interval [0,0) [v2:172.21.15.90:6818/2579716443,v1:172.21.15.90:6819/2579716443] [v2:172.21.15.90:6820/2579716443,v1:172.21.15.90:6821/2579716443] exists,up aa72392f-0498-44db-a8d2-eccc89a64871 2024-04-02T14:34:35.288 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7 up in weight 1 up_from 52 up_thru 86 down_at 0 last_clean_interval [0,0) [v2:172.21.15.92:6816/3371649139,v1:172.21.15.92:6817/3371649139] [v2:172.21.15.92:6818/3371649139,v1:172.21.15.92:6819/3371649139] exists,up 931bd105-bc14-4a1e-bfe4-c7ad58fa4b73 2024-04-02T14:34:35.288 INFO:teuthology.orchestra.run.smithi090.stdout:osd.8 up in weight 1 up_from 61 up_thru 86 down_at 0 last_clean_interval [0,0) [v2:172.21.15.177:6816/1459053034,v1:172.21.15.177:6817/1459053034] [v2:172.21.15.177:6818/1459053034,v1:172.21.15.177:6819/1459053034] exists,up 82624ec2-b382-4886-9f40-7f5f376de3ce 2024-04-02T14:34:35.288 INFO:teuthology.orchestra.run.smithi090.stdout:osd.9 up in weight 1 up_from 69 up_thru 89 down_at 0 last_clean_interval [0,0) [v2:172.21.15.90:6826/2592604332,v1:172.21.15.90:6827/2592604332] [v2:172.21.15.90:6828/2592604332,v1:172.21.15.90:6829/2592604332] exists,up fbe36e48-3468-42b2-8094-e16552f3cb16 2024-04-02T14:34:35.288 INFO:teuthology.orchestra.run.smithi090.stdout:osd.10 up in weight 1 up_from 75 up_thru 89 down_at 0 last_clean_interval [0,0) [v2:172.21.15.92:6824/1594153142,v1:172.21.15.92:6825/1594153142] [v2:172.21.15.92:6826/1594153142,v1:172.21.15.92:6827/1594153142] exists,up 9240df75-e7e2-47f1-8aae-534b70b3c210 2024-04-02T14:34:35.288 INFO:teuthology.orchestra.run.smithi090.stdout:osd.11 up in weight 1 up_from 83 up_thru 89 down_at 0 last_clean_interval [0,0) [v2:172.21.15.177:6824/1622582126,v1:172.21.15.177:6825/1622582126] [v2:172.21.15.177:6826/1622582126,v1:172.21.15.177:6827/1622582126] exists,up 83cba9fa-b745-4df1-adda-e86837d4b91a 2024-04-02T14:34:35.288 INFO:teuthology.orchestra.run.smithi090.stdout:pg_upmap_items 2.0 [7,11] 2024-04-02T14:34:35.289 INFO:teuthology.orchestra.run.smithi090.stdout:pg_upmap_items 2.a [2,11] 2024-04-02T14:34:35.289 INFO:teuthology.orchestra.run.smithi090.stdout:pg_upmap_items 2.14 [2,4] 2024-04-02T14:34:35.289 INFO:teuthology.orchestra.run.smithi090.stdout:pg_upmap_items 2.1f [8,4] 2024-04-02T14:34:35.289 INFO:teuthology.orchestra.run.smithi090.stdout:pg_upmap_items 3.2 [3,9] 2024-04-02T14:34:35.289 INFO:teuthology.orchestra.run.smithi090.stdout:pg_upmap_items 3.11 [4,8,5,9] 2024-04-02T14:34:35.289 INFO:teuthology.orchestra.run.smithi090.stdout:pg_upmap_items 3.21 [5,9] 2024-04-02T14:34:35.289 INFO:teuthology.orchestra.run.smithi090.stdout:pg_upmap_items 3.23 [4,7] 2024-04-02T14:34:35.289 INFO:teuthology.orchestra.run.smithi090.stdout:pg_upmap_items 3.3a [4,6] 2024-04-02T14:34:35.289 INFO:teuthology.orchestra.run.smithi090.stdout:blocklist 172.21.15.90:0/314641601 expires 2024-04-03T14:24:34.595561+0000 2024-04-02T14:34:35.289 INFO:teuthology.orchestra.run.smithi090.stdout:blocklist 172.21.15.90:0/1400895868 expires 2024-04-03T14:24:34.595561+0000 2024-04-02T14:34:35.289 INFO:teuthology.orchestra.run.smithi090.stdout:blocklist 172.21.15.90:6801/129436436 expires 2024-04-03T14:24:34.595561+0000 2024-04-02T14:34:35.289 INFO:teuthology.orchestra.run.smithi090.stdout:blocklist 172.21.15.90:6801/2372398598 expires 2024-04-03T14:24:00.362147+0000 2024-04-02T14:34:35.289 INFO:teuthology.orchestra.run.smithi090.stdout:blocklist 172.21.15.90:0/278569338 expires 2024-04-03T14:24:34.595561+0000 2024-04-02T14:34:35.289 INFO:teuthology.orchestra.run.smithi090.stdout:blocklist 172.21.15.90:0/2321464594 expires 2024-04-03T14:24:00.362147+0000 2024-04-02T14:34:35.289 INFO:teuthology.orchestra.run.smithi090.stdout:blocklist 172.21.15.90:6800/2372398598 expires 2024-04-03T14:24:00.362147+0000 2024-04-02T14:34:35.289 INFO:teuthology.orchestra.run.smithi090.stdout:blocklist 172.21.15.90:0/2962656672 expires 2024-04-03T14:24:00.362147+0000 2024-04-02T14:34:35.290 INFO:teuthology.orchestra.run.smithi090.stdout:blocklist 172.21.15.90:0/126693610 expires 2024-04-03T14:24:00.362147+0000 2024-04-02T14:34:35.290 INFO:teuthology.orchestra.run.smithi090.stdout:blocklist 172.21.15.90:6800/129436436 expires 2024-04-03T14:24:34.595561+0000 2024-04-02T14:34:35.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:35.286+0000 7f0f02400700 1 -- 172.21.15.90:0/1379178688 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f0ee40a3cc0 msgr2=0x7f0ee40a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:35.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:35.286+0000 7f0f02400700 1 --2- 172.21.15.90:0/1379178688 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f0ee40a3cc0 0x7f0ee40a6180 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f0ef0009750 tx=0x7f0ef0009310 comp rx=0 tx=0).stop 2024-04-02T14:34:35.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:35.286+0000 7f0f02400700 1 -- 172.21.15.90:0/1379178688 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0efc105970 msgr2=0x7f0efc19af40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:35.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:35.286+0000 7f0f02400700 1 --2- 172.21.15.90:0/1379178688 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0efc105970 0x7f0efc19af40 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f0ee800fca0 tx=0x7f0ee8005250 comp rx=0 tx=0).stop 2024-04-02T14:34:35.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:35.286+0000 7f0f02400700 1 -- 172.21.15.90:0/1379178688 shutdown_connections 2024-04-02T14:34:35.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:35.286+0000 7f0f02400700 1 --2- 172.21.15.90:0/1379178688 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f0ee40a3cc0 0x7f0ee40a6180 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:35.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:35.286+0000 7f0f02400700 1 --2- 172.21.15.90:0/1379178688 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0efc106310 0x7f0efc1a1f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:35.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:35.286+0000 7f0f02400700 1 --2- 172.21.15.90:0/1379178688 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0efc104770 0x7f0efc19aa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:35.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:35.286+0000 7f0f02400700 1 --2- 172.21.15.90:0/1379178688 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0efc105970 0x7f0efc19af40 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:35.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:35.286+0000 7f0f02400700 1 -- 172.21.15.90:0/1379178688 >> 172.21.15.90:0/1379178688 conn(0x7f0efc0ffd10 msgr2=0x7f0efc108c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:34:35.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:35.287+0000 7f0f02400700 1 -- 172.21.15.90:0/1379178688 shutdown_connections 2024-04-02T14:34:35.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:35.287+0000 7f0f02400700 1 -- 172.21.15.90:0/1379178688 wait complete. 2024-04-02T14:34:35.939 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:34:35 smithi090 ceph-mon[31674]: pgmap v380: 97 pgs: 97 active+clean; 644 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-02T14:34:35.939 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:34:35 smithi090 ceph-mon[31674]: from='client.? 172.21.15.90:0/1379178688' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-02T14:34:36.076 INFO:teuthology.run_tasks:Running task fs.ready... 2024-04-02T14:34:36.086 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T14:34:36.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:34:35 smithi177 ceph-mon[36431]: pgmap v380: 97 pgs: 97 active+clean; 644 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-02T14:34:36.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:34:35 smithi177 ceph-mon[36431]: from='client.? 172.21.15.90:0/1379178688' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-02T14:34:36.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:36.307+0000 7f1b63000640 1 -- 172.21.15.90:0/3826402401 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1b5c109330 msgr2=0x7f1b5c1097b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:36.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:36.307+0000 7f1b63000640 1 --2- 172.21.15.90:0/3826402401 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1b5c109330 0x7f1b5c1097b0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f1b48009900 tx=0x7f1b4802bc30 comp rx=0 tx=0).stop 2024-04-02T14:34:36.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:36.307+0000 7f1b63000640 1 -- 172.21.15.90:0/3826402401 shutdown_connections 2024-04-02T14:34:36.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:36.307+0000 7f1b63000640 1 --2- 172.21.15.90:0/3826402401 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1b5c109cf0 0x7f1b5c110580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:36.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:36.307+0000 7f1b63000640 1 --2- 172.21.15.90:0/3826402401 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1b5c109330 0x7f1b5c1097b0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:36.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:36.307+0000 7f1b63000640 1 --2- 172.21.15.90:0/3826402401 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1b5c108140 0x7f1b5c108560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:36.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:36.307+0000 7f1b63000640 1 -- 172.21.15.90:0/3826402401 >> 172.21.15.90:0/3826402401 conn(0x7f1b5c077110 msgr2=0x7f1b5c079530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:34:36.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:36.307+0000 7f1b63000640 1 -- 172.21.15.90:0/3826402401 shutdown_connections 2024-04-02T14:34:36.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:36.307+0000 7f1b63000640 1 -- 172.21.15.90:0/3826402401 wait complete. 2024-04-02T14:34:36.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:36.308+0000 7f1b63000640 1 Processor -- start 2024-04-02T14:34:36.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:36.309+0000 7f1b63000640 1 -- start start 2024-04-02T14:34:36.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:36.309+0000 7f1b63000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1b5c108140 0x7f1b5c19e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:36.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:36.309+0000 7f1b63000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1b5c109330 0x7f1b5c19e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:36.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:36.309+0000 7f1b63000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1b5c109cf0 0x7f1b5c1a5990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:36.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:36.309+0000 7f1b63000640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f1b5c113e20 con 0x7f1b5c109330 2024-04-02T14:34:36.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:36.309+0000 7f1b63000640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f1b5c113ca0 con 0x7f1b5c109cf0 2024-04-02T14:34:36.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:36.309+0000 7f1b63000640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f1b5c113fa0 con 0x7f1b5c108140 2024-04-02T14:34:36.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:36.309+0000 7f1b5b400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1b5c109330 0x7f1b5c19e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:36.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:36.309+0000 7f1b5be00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1b5c108140 0x7f1b5c19e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:36.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:36.309+0000 7f1b5b400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1b5c109330 0x7f1b5c19e920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:36306/0 (socket says 172.21.15.90:36306) 2024-04-02T14:34:36.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:36.309+0000 7f1b5b400640 1 -- 172.21.15.90:0/2073972226 learned_addr learned my addr 172.21.15.90:0/2073972226 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:34:36.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:36.309+0000 7f1b61000640 1 --2- 172.21.15.90:0/2073972226 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1b5c109cf0 0x7f1b5c1a5990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:36.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:36.310+0000 7f1b5b400640 1 -- 172.21.15.90:0/2073972226 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1b5c108140 msgr2=0x7f1b5c19e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:36.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:36.310+0000 7f1b5b400640 1 --2- 172.21.15.90:0/2073972226 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1b5c108140 0x7f1b5c19e3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:36.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:36.310+0000 7f1b5b400640 1 -- 172.21.15.90:0/2073972226 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1b5c109cf0 msgr2=0x7f1b5c1a5990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:36.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:36.310+0000 7f1b5b400640 1 --2- 172.21.15.90:0/2073972226 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1b5c109cf0 0x7f1b5c1a5990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:36.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:36.310+0000 7f1b5b400640 1 -- 172.21.15.90:0/2073972226 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b5c1a6000 con 0x7f1b5c109330 2024-04-02T14:34:36.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:36.310+0000 7f1b5b400640 1 --2- 172.21.15.90:0/2073972226 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1b5c109330 0x7f1b5c19e920 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f1b48000c00 tx=0x7f1b4802c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:34:36.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:36.310+0000 7f1b58c00640 1 -- 172.21.15.90:0/2073972226 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b48043020 con 0x7f1b5c109330 2024-04-02T14:34:36.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:36.310+0000 7f1b58c00640 1 -- 172.21.15.90:0/2073972226 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1b480054e0 con 0x7f1b5c109330 2024-04-02T14:34:36.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:36.310+0000 7f1b58c00640 1 -- 172.21.15.90:0/2073972226 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b48003c40 con 0x7f1b5c109330 2024-04-02T14:34:36.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:36.310+0000 7f1b63000640 1 -- 172.21.15.90:0/2073972226 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b5c1a6290 con 0x7f1b5c109330 2024-04-02T14:34:36.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:36.310+0000 7f1b63000640 1 -- 172.21.15.90:0/2073972226 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b5c1a6830 con 0x7f1b5c109330 2024-04-02T14:34:36.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:36.311+0000 7f1b63000640 1 -- 172.21.15.90:0/2073972226 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1b5c10f460 con 0x7f1b5c109330 2024-04-02T14:34:36.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:36.313+0000 7f1b58c00640 1 -- 172.21.15.90:0/2073972226 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f1b48046020 con 0x7f1b5c109330 2024-04-02T14:34:36.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:36.314+0000 7f1b58c00640 1 --2- 172.21.15.90:0/2073972226 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f1b400a3c30 0x7f1b400a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:36.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:36.315+0000 7f1b58c00640 1 -- 172.21.15.90:0/2073972226 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8248+0+0 (secure 0 0 0) 0x7f1b480f2410 con 0x7f1b5c109330 2024-04-02T14:34:36.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:36.315+0000 7f1b5be00640 1 --2- 172.21.15.90:0/2073972226 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f1b400a3c30 0x7f1b400a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:36.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:36.315+0000 7f1b5be00640 1 --2- 172.21.15.90:0/2073972226 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f1b400a3c30 0x7f1b400a60f0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f1b5c109190 tx=0x7f1b4c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:34:36.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:36.319+0000 7f1b58c00640 1 -- 172.21.15.90:0/2073972226 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7f1b4804d020 con 0x7f1b5c109330 2024-04-02T14:34:36.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:34:35 smithi092 ceph-mon[36497]: pgmap v380: 97 pgs: 97 active+clean; 644 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-02T14:34:36.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:34:35 smithi092 ceph-mon[36497]: from='client.? 172.21.15.90:0/1379178688' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-02T14:34:36.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:36.523+0000 7f1b63000640 1 -- 172.21.15.90:0/2073972226 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f1b5c062700 con 0x7f1b5c109330 2024-04-02T14:34:36.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:36.526+0000 7f1b58c00640 1 -- 172.21.15.90:0/2073972226 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+22646 (secure 0 0 0) 0x7f1b480bed80 con 0x7f1b5c109330 2024-04-02T14:34:36.529 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-02T14:34:36.529 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":90,"fsid":"6449448a-f0fc-11ee-b647-cb9ed24678a4","created":"2024-04-02T14:23:16.530201+0000","modified":"2024-04-02T14:32:35.716929+0000","last_up_change":"2024-04-02T14:31:23.819732+0000","last_in_change":"2024-04-02T14:31:02.824284+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T14:27:16.758622+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-02T14:32:20.016331+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-02T14:32:21.294367+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a1a6a856-ab44-44bd-bc5e-45c718a70cd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":3635087609},{"type":"v1","addr":"172.21.15.90:6803","nonce":3635087609}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":3635087609},{"type":"v1","addr":"172.21.15.90:6805","nonce":3635087609}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":3635087609},{"type":"v1","addr":"172.21.15.90:6809","nonce":3635087609}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":3635087609},{"type":"v1","addr":"172.21.15.90:6807","nonce":3635087609}]},"public_addr":"172.21.15.90:6803/3635087609","cluster_addr":"172.21.15.90:6805/3635087609","heartbeat_back_addr":"172.21.15.90:6809/3635087609","heartbeat_front_addr":"172.21.15.90:6807/3635087609","state":["exists","up"]},{"osd":1,"uuid":"124139cd-5793-463e-9607-9d9c10164183","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":1994536059},{"type":"v1","addr":"172.21.15.92:6801","nonce":1994536059}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":1994536059},{"type":"v1","addr":"172.21.15.92:6803","nonce":1994536059}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":1994536059},{"type":"v1","addr":"172.21.15.92:6807","nonce":1994536059}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":1994536059},{"type":"v1","addr":"172.21.15.92:6805","nonce":1994536059}]},"public_addr":"172.21.15.92:6801/1994536059","cluster_addr":"172.21.15.92:6803/1994536059","heartbeat_back_addr":"172.21.15.92:6807/1994536059","heartbeat_front_addr":"172.21.15.92:6805/1994536059","state":["exists","up"]},{"osd":2,"uuid":"a61fbef2-9dca-4875-a814-020fff0db0e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":1358472800},{"type":"v1","addr":"172.21.15.177:6801","nonce":1358472800}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":1358472800},{"type":"v1","addr":"172.21.15.177:6803","nonce":1358472800}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":1358472800},{"type":"v1","addr":"172.21.15.177:6807","nonce":1358472800}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":1358472800},{"type":"v1","addr":"172.21.15.177:6805","nonce":1358472800}]},"public_addr":"172.21.15.177:6801/1358472800","cluster_addr":"172.21.15.177:6803/1358472800","heartbeat_back_addr":"172.21.15.177:6807/1358472800","heartbeat_front_addr":"172.21.15.177:6805/1358472800","state":["exists","up"]},{"osd":3,"uuid":"b175a2f4-7859-4eb0-bc1f-fba8c2b92a2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":3114811216},{"type":"v1","addr":"172.21.15.90:6811","nonce":3114811216}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":3114811216},{"type":"v1","addr":"172.21.15.90:6813","nonce":3114811216}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":3114811216},{"type":"v1","addr":"172.21.15.90:6817","nonce":3114811216}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":3114811216},{"type":"v1","addr":"172.21.15.90:6815","nonce":3114811216}]},"public_addr":"172.21.15.90:6811/3114811216","cluster_addr":"172.21.15.90:6813/3114811216","heartbeat_back_addr":"172.21.15.90:6817/3114811216","heartbeat_front_addr":"172.21.15.90:6815/3114811216","state":["exists","up"]},{"osd":4,"uuid":"38cc68ea-c109-4d8d-b77d-1c6f9b56c875","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":2987208879},{"type":"v1","addr":"172.21.15.92:6809","nonce":2987208879}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6810","nonce":2987208879},{"type":"v1","addr":"172.21.15.92:6811","nonce":2987208879}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6814","nonce":2987208879},{"type":"v1","addr":"172.21.15.92:6815","nonce":2987208879}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":2987208879},{"type":"v1","addr":"172.21.15.92:6813","nonce":2987208879}]},"public_addr":"172.21.15.92:6809/2987208879","cluster_addr":"172.21.15.92:6811/2987208879","heartbeat_back_addr":"172.21.15.92:6815/2987208879","heartbeat_front_addr":"172.21.15.92:6813/2987208879","state":["exists","up"]},{"osd":5,"uuid":"2347232e-ac91-466f-8ac9-496f397679ba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":3486820237},{"type":"v1","addr":"172.21.15.177:6809","nonce":3486820237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":3486820237},{"type":"v1","addr":"172.21.15.177:6811","nonce":3486820237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":3486820237},{"type":"v1","addr":"172.21.15.177:6815","nonce":3486820237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":3486820237},{"type":"v1","addr":"172.21.15.177:6813","nonce":3486820237}]},"public_addr":"172.21.15.177:6809/3486820237","cluster_addr":"172.21.15.177:6811/3486820237","heartbeat_back_addr":"172.21.15.177:6815/3486820237","heartbeat_front_addr":"172.21.15.177:6813/3486820237","state":["exists","up"]},{"osd":6,"uuid":"aa72392f-0498-44db-a8d2-eccc89a64871","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":2579716443},{"type":"v1","addr":"172.21.15.90:6819","nonce":2579716443}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6820","nonce":2579716443},{"type":"v1","addr":"172.21.15.90:6821","nonce":2579716443}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":2579716443},{"type":"v1","addr":"172.21.15.90:6825","nonce":2579716443}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2579716443},{"type":"v1","addr":"172.21.15.90:6823","nonce":2579716443}]},"public_addr":"172.21.15.90:6819/2579716443","cluster_addr":"172.21.15.90:6821/2579716443","heartbeat_back_addr":"172.21.15.90:6825/2579716443","heartbeat_front_addr":"172.21.15.90:6823/2579716443","state":["exists","up"]},{"osd":7,"uuid":"931bd105-bc14-4a1e-bfe4-c7ad58fa4b73","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":3371649139},{"type":"v1","addr":"172.21.15.92:6817","nonce":3371649139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":3371649139},{"type":"v1","addr":"172.21.15.92:6819","nonce":3371649139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":3371649139},{"type":"v1","addr":"172.21.15.92:6823","nonce":3371649139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6820","nonce":3371649139},{"type":"v1","addr":"172.21.15.92:6821","nonce":3371649139}]},"public_addr":"172.21.15.92:6817/3371649139","cluster_addr":"172.21.15.92:6819/3371649139","heartbeat_back_addr":"172.21.15.92:6823/3371649139","heartbeat_front_addr":"172.21.15.92:6821/3371649139","state":["exists","up"]},{"osd":8,"uuid":"82624ec2-b382-4886-9f40-7f5f376de3ce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":1459053034},{"type":"v1","addr":"172.21.15.177:6817","nonce":1459053034}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":1459053034},{"type":"v1","addr":"172.21.15.177:6819","nonce":1459053034}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":1459053034},{"type":"v1","addr":"172.21.15.177:6823","nonce":1459053034}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":1459053034},{"type":"v1","addr":"172.21.15.177:6821","nonce":1459053034}]},"public_addr":"172.21.15.177:6817/1459053034","cluster_addr":"172.21.15.177:6819/1459053034","heartbeat_back_addr":"172.21.15.177:6823/1459053034","heartbeat_front_addr":"172.21.15.177:6821/1459053034","state":["exists","up"]},{"osd":9,"uuid":"fbe36e48-3468-42b2-8094-e16552f3cb16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":2592604332},{"type":"v1","addr":"172.21.15.90:6827","nonce":2592604332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":2592604332},{"type":"v1","addr":"172.21.15.90:6829","nonce":2592604332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6832","nonce":2592604332},{"type":"v1","addr":"172.21.15.90:6833","nonce":2592604332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":2592604332},{"type":"v1","addr":"172.21.15.90:6831","nonce":2592604332}]},"public_addr":"172.21.15.90:6827/2592604332","cluster_addr":"172.21.15.90:6829/2592604332","heartbeat_back_addr":"172.21.15.90:6833/2592604332","heartbeat_front_addr":"172.21.15.90:6831/2592604332","state":["exists","up"]},{"osd":10,"uuid":"9240df75-e7e2-47f1-8aae-534b70b3c210","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":1594153142},{"type":"v1","addr":"172.21.15.92:6825","nonce":1594153142}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6826","nonce":1594153142},{"type":"v1","addr":"172.21.15.92:6827","nonce":1594153142}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":1594153142},{"type":"v1","addr":"172.21.15.92:6831","nonce":1594153142}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":1594153142},{"type":"v1","addr":"172.21.15.92:6829","nonce":1594153142}]},"public_addr":"172.21.15.92:6825/1594153142","cluster_addr":"172.21.15.92:6827/1594153142","heartbeat_back_addr":"172.21.15.92:6831/1594153142","heartbeat_front_addr":"172.21.15.92:6829/1594153142","state":["exists","up"]},{"osd":11,"uuid":"83cba9fa-b745-4df1-adda-e86837d4b91a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":83,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1622582126},{"type":"v1","addr":"172.21.15.177:6825","nonce":1622582126}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":1622582126},{"type":"v1","addr":"172.21.15.177:6827","nonce":1622582126}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":1622582126},{"type":"v1","addr":"172.21.15.177:6831","nonce":1622582126}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1622582126},{"type":"v1","addr":"172.21.15.177:6829","nonce":1622582126}]},"public_addr":"172.21.15.177:6825/1622582126","cluster_addr":"172.21.15.177:6827/1622582126","heartbeat_back_addr":"172.21.15.177:6831/1622582126","heartbeat_front_addr":"172.21.15.177:6829/1622582126","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:26:24.290365+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:26:49.405988+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:27:13.639196+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:27:45.603746+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:28:11.019126+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:28:35.084163+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:29:08.064626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:29:33.266133+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:29:57.381816+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:30:30.286847+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:30:56.226279+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:31:20.809611+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":7,"to":11}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":8,"to":4}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.11","mappings":[{"from":4,"to":8},{"from":5,"to":9}]},{"pgid":"3.21","mappings":[{"from":5,"to":9}]},{"pgid":"3.23","mappings":[{"from":4,"to":7}]},{"pgid":"3.3a","mappings":[{"from":4,"to":6}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.90:0/314641601":"2024-04-03T14:24:34.595561+0000","172.21.15.90:0/1400895868":"2024-04-03T14:24:34.595561+0000","172.21.15.90:6801/129436436":"2024-04-03T14:24:34.595561+0000","172.21.15.90:6801/2372398598":"2024-04-03T14:24:00.362147+0000","172.21.15.90:0/278569338":"2024-04-03T14:24:34.595561+0000","172.21.15.90:0/2321464594":"2024-04-03T14:24:00.362147+0000","172.21.15.90:6800/2372398598":"2024-04-03T14:24:00.362147+0000","172.21.15.90:0/2962656672":"2024-04-03T14:24:00.362147+0000","172.21.15.90:0/126693610":"2024-04-03T14:24:00.362147+0000","172.21.15.90:6800/129436436":"2024-04-03T14:24:34.595561+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T14:34:36.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:36.529+0000 7f1b63000640 1 -- 172.21.15.90:0/2073972226 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f1b400a3c30 msgr2=0x7f1b400a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:36.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:36.529+0000 7f1b63000640 1 --2- 172.21.15.90:0/2073972226 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f1b400a3c30 0x7f1b400a60f0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f1b5c109190 tx=0x7f1b4c005f90 comp rx=0 tx=0).stop 2024-04-02T14:34:36.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:36.529+0000 7f1b63000640 1 -- 172.21.15.90:0/2073972226 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1b5c109330 msgr2=0x7f1b5c19e920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:36.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:36.529+0000 7f1b63000640 1 --2- 172.21.15.90:0/2073972226 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1b5c109330 0x7f1b5c19e920 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f1b48000c00 tx=0x7f1b4802c650 comp rx=0 tx=0).stop 2024-04-02T14:34:36.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:36.529+0000 7f1b63000640 1 -- 172.21.15.90:0/2073972226 shutdown_connections 2024-04-02T14:34:36.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:36.529+0000 7f1b63000640 1 --2- 172.21.15.90:0/2073972226 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f1b400a3c30 0x7f1b400a60f0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:36.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:36.529+0000 7f1b63000640 1 --2- 172.21.15.90:0/2073972226 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1b5c109cf0 0x7f1b5c1a5990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:36.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:36.529+0000 7f1b63000640 1 --2- 172.21.15.90:0/2073972226 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1b5c109330 0x7f1b5c19e920 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:36.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:36.530+0000 7f1b63000640 1 --2- 172.21.15.90:0/2073972226 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1b5c108140 0x7f1b5c19e3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:36.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:36.530+0000 7f1b63000640 1 -- 172.21.15.90:0/2073972226 >> 172.21.15.90:0/2073972226 conn(0x7f1b5c077110 msgr2=0x7f1b5c10c630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:34:36.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:36.530+0000 7f1b63000640 1 -- 172.21.15.90:0/2073972226 shutdown_connections 2024-04-02T14:34:36.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:36.530+0000 7f1b63000640 1 -- 172.21.15.90:0/2073972226 wait complete. 2024-04-02T14:34:36.552 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-02T14:27:16.758622+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-02T14:32:20.016331+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-02T14:32:21.294367+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '88', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-02T14:34:36.553 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-02T14:34:36.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:36.768+0000 7f37b9800640 1 -- 172.21.15.90:0/1639695502 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f37b4104f20 msgr2=0x7f37b41053a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:36.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:36.768+0000 7f37b9800640 1 --2- 172.21.15.90:0/1639695502 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f37b4104f20 0x7f37b41053a0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f37a0009900 tx=0x7f37a002bc30 comp rx=0 tx=0).stop 2024-04-02T14:34:36.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:36.768+0000 7f37b9800640 1 -- 172.21.15.90:0/1639695502 shutdown_connections 2024-04-02T14:34:36.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:36.768+0000 7f37b9800640 1 --2- 172.21.15.90:0/1639695502 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f37b41058e0 0x7f37b410c170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:36.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:36.768+0000 7f37b9800640 1 --2- 172.21.15.90:0/1639695502 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f37b4104f20 0x7f37b41053a0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:36.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:36.768+0000 7f37b9800640 1 --2- 172.21.15.90:0/1639695502 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f37b4103d30 0x7f37b4104150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:36.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:36.768+0000 7f37b9800640 1 -- 172.21.15.90:0/1639695502 >> 172.21.15.90:0/1639695502 conn(0x7f37b40ff2c0 msgr2=0x7f37b41016e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:34:36.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:36.768+0000 7f37b9800640 1 -- 172.21.15.90:0/1639695502 shutdown_connections 2024-04-02T14:34:36.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:36.768+0000 7f37b9800640 1 -- 172.21.15.90:0/1639695502 wait complete. 2024-04-02T14:34:36.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:36.769+0000 7f37b9800640 1 Processor -- start 2024-04-02T14:34:36.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:36.769+0000 7f37b9800640 1 -- start start 2024-04-02T14:34:36.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:36.769+0000 7f37b9800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f37b4103d30 0x7f37b419a040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:36.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:36.769+0000 7f37b9800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f37b4104f20 0x7f37b419a580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:36.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:36.769+0000 7f37b9800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f37b41058e0 0x7f37b41a15f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:36.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:36.770+0000 7f37b9800640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f37b410fa00 con 0x7f37b41058e0 2024-04-02T14:34:36.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:36.770+0000 7f37b9800640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f37b410f880 con 0x7f37b4103d30 2024-04-02T14:34:36.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:36.770+0000 7f37b9800640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f37b410fb80 con 0x7f37b4104f20 2024-04-02T14:34:36.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:36.770+0000 7f37b3400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f37b41058e0 0x7f37b41a15f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:36.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:36.770+0000 7f37b2a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f37b4103d30 0x7f37b419a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:36.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:36.770+0000 7f37b2a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f37b4103d30 0x7f37b419a040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:60176/0 (socket says 172.21.15.90:60176) 2024-04-02T14:34:36.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:36.770+0000 7f37b2a00640 1 -- 172.21.15.90:0/3868089414 learned_addr learned my addr 172.21.15.90:0/3868089414 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:34:36.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:36.770+0000 7f37b2000640 1 --2- 172.21.15.90:0/3868089414 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f37b4104f20 0x7f37b419a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:36.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:36.770+0000 7f37b2a00640 1 -- 172.21.15.90:0/3868089414 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f37b4104f20 msgr2=0x7f37b419a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:36.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:36.770+0000 7f37b2a00640 1 --2- 172.21.15.90:0/3868089414 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f37b4104f20 0x7f37b419a580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:36.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:36.770+0000 7f37b2a00640 1 -- 172.21.15.90:0/3868089414 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f37b41058e0 msgr2=0x7f37b41a15f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:36.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:36.770+0000 7f37b2a00640 1 --2- 172.21.15.90:0/3868089414 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f37b41058e0 0x7f37b41a15f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:36.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:36.770+0000 7f37b2a00640 1 -- 172.21.15.90:0/3868089414 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37b41a1c60 con 0x7f37b4103d30 2024-04-02T14:34:36.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:36.770+0000 7f37b3400640 1 --2- 172.21.15.90:0/3868089414 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f37b41058e0 0x7f37b41a15f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:34:36.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:36.771+0000 7f37b2a00640 1 --2- 172.21.15.90:0/3868089414 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f37b4103d30 0x7f37b419a040 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f37a400d950 tx=0x7f37a400dd50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:34:36.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:36.771+0000 7f37ab400640 1 -- 172.21.15.90:0/3868089414 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37a4017070 con 0x7f37b4103d30 2024-04-02T14:34:36.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:36.771+0000 7f37ab400640 1 -- 172.21.15.90:0/3868089414 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f37a400be10 con 0x7f37b4103d30 2024-04-02T14:34:36.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:36.771+0000 7f37b9800640 1 -- 172.21.15.90:0/3868089414 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f37b41a1f50 con 0x7f37b4103d30 2024-04-02T14:34:36.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:36.771+0000 7f37ab400640 1 -- 172.21.15.90:0/3868089414 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37a400f620 con 0x7f37b4103d30 2024-04-02T14:34:36.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:36.771+0000 7f37b9800640 1 -- 172.21.15.90:0/3868089414 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37b41a2360 con 0x7f37b4103d30 2024-04-02T14:34:36.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:36.772+0000 7f37b9800640 1 -- 172.21.15.90:0/3868089414 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f37b410b050 con 0x7f37b4103d30 2024-04-02T14:34:36.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:36.774+0000 7f37ab400640 1 -- 172.21.15.90:0/3868089414 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f37a401a070 con 0x7f37b4103d30 2024-04-02T14:34:36.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:36.776+0000 7f37ab400640 1 --2- 172.21.15.90:0/3868089414 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f37980a3bb0 0x7f37980a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:36.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:36.776+0000 7f37b2000640 1 --2- 172.21.15.90:0/3868089414 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f37980a3bb0 0x7f37980a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:36.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:36.776+0000 7f37ab400640 1 -- 172.21.15.90:0/3868089414 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8248+0+0 (secure 0 0 0) 0x7f37a40cfe10 con 0x7f37b4103d30 2024-04-02T14:34:36.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:36.777+0000 7f37b2000640 1 --2- 172.21.15.90:0/3868089414 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f37980a3bb0 0x7f37980a6070 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f37a002c240 tx=0x7f37a002df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:34:36.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:36.778+0000 7f37ab400640 1 -- 172.21.15.90:0/3868089414 <== mon.2 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7f37a409c780 con 0x7f37b4103d30 2024-04-02T14:34:36.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:36.984+0000 7f37b9800640 1 -- 172.21.15.90:0/3868089414 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f37b402d500 con 0x7f37b4103d30 2024-04-02T14:34:36.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:36.985+0000 7f37ab400640 1 -- 172.21.15.90:0/3868089414 <== mon.2 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v90) ==== 93+0+10 (secure 0 0 0) 0x7f37a409c080 con 0x7f37b4103d30 2024-04-02T14:34:36.987 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 1 2024-04-02T14:34:36.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:36.987+0000 7f37b9800640 1 -- 172.21.15.90:0/3868089414 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f37980a3bb0 msgr2=0x7f37980a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:36.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:36.987+0000 7f37b9800640 1 --2- 172.21.15.90:0/3868089414 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f37980a3bb0 0x7f37980a6070 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f37a002c240 tx=0x7f37a002df90 comp rx=0 tx=0).stop 2024-04-02T14:34:36.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:36.987+0000 7f37b9800640 1 -- 172.21.15.90:0/3868089414 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f37b4103d30 msgr2=0x7f37b419a040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:36.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:36.987+0000 7f37b9800640 1 --2- 172.21.15.90:0/3868089414 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f37b4103d30 0x7f37b419a040 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f37a400d950 tx=0x7f37a400dd50 comp rx=0 tx=0).stop 2024-04-02T14:34:36.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:36.988+0000 7f37b9800640 1 -- 172.21.15.90:0/3868089414 shutdown_connections 2024-04-02T14:34:36.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:36.988+0000 7f37b9800640 1 --2- 172.21.15.90:0/3868089414 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f37980a3bb0 0x7f37980a6070 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:36.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:36.988+0000 7f37b9800640 1 --2- 172.21.15.90:0/3868089414 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f37b41058e0 0x7f37b41a15f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:36.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:36.988+0000 7f37b9800640 1 --2- 172.21.15.90:0/3868089414 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f37b4104f20 0x7f37b419a580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:36.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:36.988+0000 7f37b9800640 1 --2- 172.21.15.90:0/3868089414 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f37b4103d30 0x7f37b419a040 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:36.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:36.988+0000 7f37b9800640 1 -- 172.21.15.90:0/3868089414 >> 172.21.15.90:0/3868089414 conn(0x7f37b40ff2c0 msgr2=0x7f37b4108220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:34:36.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:36.988+0000 7f37b9800640 1 -- 172.21.15.90:0/3868089414 shutdown_connections 2024-04-02T14:34:36.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:36.988+0000 7f37b9800640 1 -- 172.21.15.90:0/3868089414 wait complete. 2024-04-02T14:34:37.008 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-02T14:34:37.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:34:36 smithi177 ceph-mon[36431]: from='client.? 172.21.15.90:0/2073972226' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-02T14:34:37.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:34:36 smithi090 ceph-mon[31674]: from='client.? 172.21.15.90:0/2073972226' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-02T14:34:37.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:37.218+0000 7f44a3e00640 1 -- 172.21.15.90:0/1495190225 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f44a41057b0 msgr2=0x7f44a410c040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:37.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:37.218+0000 7f44a3e00640 1 --2- 172.21.15.90:0/1495190225 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f44a41057b0 0x7f44a410c040 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f449c009be0 tx=0x7f449c02bef0 comp rx=0 tx=0).stop 2024-04-02T14:34:37.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:37.218+0000 7f44a3e00640 1 -- 172.21.15.90:0/1495190225 shutdown_connections 2024-04-02T14:34:37.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:37.218+0000 7f44a3e00640 1 --2- 172.21.15.90:0/1495190225 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f44a41057b0 0x7f44a410c040 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:37.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:37.218+0000 7f44a3e00640 1 --2- 172.21.15.90:0/1495190225 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f44a4104df0 0x7f44a4105270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:37.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:37.218+0000 7f44a3e00640 1 --2- 172.21.15.90:0/1495190225 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f44a4103c00 0x7f44a4104020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:37.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:37.218+0000 7f44a3e00640 1 -- 172.21.15.90:0/1495190225 >> 172.21.15.90:0/1495190225 conn(0x7f44a40ff170 msgr2=0x7f44a41015b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:34:37.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:37.218+0000 7f44a3e00640 1 -- 172.21.15.90:0/1495190225 shutdown_connections 2024-04-02T14:34:37.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:37.218+0000 7f44a3e00640 1 -- 172.21.15.90:0/1495190225 wait complete. 2024-04-02T14:34:37.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:37.219+0000 7f44a3e00640 1 Processor -- start 2024-04-02T14:34:37.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:37.219+0000 7f44a3e00640 1 -- start start 2024-04-02T14:34:37.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:37.219+0000 7f44a3e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f44a4103c00 0x7f44a4199ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:37.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:37.219+0000 7f44a3e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f44a4104df0 0x7f44a419a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:37.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:37.219+0000 7f44a3e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f44a41057b0 0x7f44a41a1470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:37.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:37.219+0000 7f44a3e00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f44a410f880 con 0x7f44a4104df0 2024-04-02T14:34:37.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:37.219+0000 7f44a3e00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f44a410f700 con 0x7f44a41057b0 2024-04-02T14:34:37.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:37.219+0000 7f44a3e00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f44a410fa00 con 0x7f44a4103c00 2024-04-02T14:34:37.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:37.219+0000 7f44a2000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f44a4104df0 0x7f44a419a400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:37.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:37.220+0000 7f44a2a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f44a4103c00 0x7f44a4199ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:37.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:37.220+0000 7f44a2000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f44a4104df0 0x7f44a419a400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:36332/0 (socket says 172.21.15.90:36332) 2024-04-02T14:34:37.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:37.220+0000 7f44a2000640 1 -- 172.21.15.90:0/2561472502 learned_addr learned my addr 172.21.15.90:0/2561472502 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:34:37.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:37.220+0000 7f44a3400640 1 --2- 172.21.15.90:0/2561472502 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f44a41057b0 0x7f44a41a1470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:37.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:37.220+0000 7f44a2a00640 1 -- 172.21.15.90:0/2561472502 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f44a41057b0 msgr2=0x7f44a41a1470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:37.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:37.220+0000 7f44a2a00640 1 --2- 172.21.15.90:0/2561472502 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f44a41057b0 0x7f44a41a1470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:37.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:37.220+0000 7f44a2a00640 1 -- 172.21.15.90:0/2561472502 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f44a4104df0 msgr2=0x7f44a419a400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:37.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:37.220+0000 7f44a2a00640 1 --2- 172.21.15.90:0/2561472502 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f44a4104df0 0x7f44a419a400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:37.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:37.220+0000 7f44a2a00640 1 -- 172.21.15.90:0/2561472502 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44a41a1ae0 con 0x7f44a4103c00 2024-04-02T14:34:37.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:37.220+0000 7f44a2000640 1 --2- 172.21.15.90:0/2561472502 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f44a4104df0 0x7f44a419a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:34:37.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:37.220+0000 7f44a3400640 1 --2- 172.21.15.90:0/2561472502 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f44a41057b0 0x7f44a41a1470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:34:37.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:37.220+0000 7f44a2a00640 1 --2- 172.21.15.90:0/2561472502 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f44a4103c00 0x7f44a4199ec0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f44a4104c50 tx=0x7f449400ee50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:34:37.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:37.220+0000 7f449b400640 1 -- 172.21.15.90:0/2561472502 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4494004440 con 0x7f44a4103c00 2024-04-02T14:34:37.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:37.221+0000 7f449b400640 1 -- 172.21.15.90:0/2561472502 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f449400cda0 con 0x7f44a4103c00 2024-04-02T14:34:37.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:37.221+0000 7f44a3e00640 1 -- 172.21.15.90:0/2561472502 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f44a41a1dd0 con 0x7f44a4103c00 2024-04-02T14:34:37.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:37.221+0000 7f449b400640 1 -- 172.21.15.90:0/2561472502 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44940196e0 con 0x7f44a4103c00 2024-04-02T14:34:37.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:37.221+0000 7f44a3e00640 1 -- 172.21.15.90:0/2561472502 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44a41a2230 con 0x7f44a4103c00 2024-04-02T14:34:37.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:37.222+0000 7f44a3e00640 1 -- 172.21.15.90:0/2561472502 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f44a4068c60 con 0x7f44a4103c00 2024-04-02T14:34:37.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:37.224+0000 7f449b400640 1 -- 172.21.15.90:0/2561472502 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f4494003780 con 0x7f44a4103c00 2024-04-02T14:34:37.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:37.225+0000 7f449b400640 1 --2- 172.21.15.90:0/2561472502 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f44880a3bb0 0x7f44880a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:37.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:37.225+0000 7f449b400640 1 -- 172.21.15.90:0/2561472502 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8248+0+0 (secure 0 0 0) 0x7f44940d0800 con 0x7f44a4103c00 2024-04-02T14:34:37.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:37.225+0000 7f44a2000640 1 --2- 172.21.15.90:0/2561472502 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f44880a3bb0 0x7f44880a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:37.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:37.226+0000 7f44a2000640 1 --2- 172.21.15.90:0/2561472502 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f44880a3bb0 0x7f44880a6070 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f44a419b3d0 tx=0x7f4490005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:34:37.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:37.230+0000 7f449b400640 1 -- 172.21.15.90:0/2561472502 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7f4494003e20 con 0x7f44a4103c00 2024-04-02T14:34:37.360 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:34:36 smithi092 ceph-mon[36497]: from='client.? 172.21.15.90:0/2073972226' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-02T14:34:37.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:37.431+0000 7f44a3e00640 1 -- 172.21.15.90:0/2561472502 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f44a4062700 con 0x7f44a4103c00 2024-04-02T14:34:37.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:37.432+0000 7f449b400640 1 -- 172.21.15.90:0/2561472502 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v90) ==== 104+0+11 (secure 0 0 0) 0x7f4494009510 con 0x7f44a4103c00 2024-04-02T14:34:37.434 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-04-02T14:34:37.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:37.434+0000 7f44a3e00640 1 -- 172.21.15.90:0/2561472502 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f44880a3bb0 msgr2=0x7f44880a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:37.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:37.434+0000 7f44a3e00640 1 --2- 172.21.15.90:0/2561472502 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f44880a3bb0 0x7f44880a6070 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f44a419b3d0 tx=0x7f4490005f90 comp rx=0 tx=0).stop 2024-04-02T14:34:37.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:37.434+0000 7f44a3e00640 1 -- 172.21.15.90:0/2561472502 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f44a4103c00 msgr2=0x7f44a4199ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:37.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:37.434+0000 7f44a3e00640 1 --2- 172.21.15.90:0/2561472502 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f44a4103c00 0x7f44a4199ec0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f44a4104c50 tx=0x7f449400ee50 comp rx=0 tx=0).stop 2024-04-02T14:34:37.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:37.435+0000 7f44a3e00640 1 -- 172.21.15.90:0/2561472502 shutdown_connections 2024-04-02T14:34:37.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:37.435+0000 7f44a3e00640 1 --2- 172.21.15.90:0/2561472502 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f44880a3bb0 0x7f44880a6070 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:37.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:37.435+0000 7f44a3e00640 1 --2- 172.21.15.90:0/2561472502 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f44a41057b0 0x7f44a41a1470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:37.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:37.435+0000 7f44a3e00640 1 --2- 172.21.15.90:0/2561472502 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f44a4104df0 0x7f44a419a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:37.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:37.435+0000 7f44a3e00640 1 --2- 172.21.15.90:0/2561472502 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f44a4103c00 0x7f44a4199ec0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:37.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:37.435+0000 7f44a3e00640 1 -- 172.21.15.90:0/2561472502 >> 172.21.15.90:0/2561472502 conn(0x7f44a40ff170 msgr2=0x7f44a41080f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:34:37.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:37.435+0000 7f44a3e00640 1 -- 172.21.15.90:0/2561472502 shutdown_connections 2024-04-02T14:34:37.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:37.435+0000 7f44a3e00640 1 -- 172.21.15.90:0/2561472502 wait complete. 2024-04-02T14:34:37.457 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-02T14:34:37.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:37.665+0000 7fe92f400640 1 -- 172.21.15.90:0/1208336969 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe928103d40 msgr2=0x7fe928104160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:37.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:37.665+0000 7fe92f400640 1 --2- 172.21.15.90:0/1208336969 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe928103d40 0x7fe928104160 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fe918009ca0 tx=0x7fe91802c100 comp rx=0 tx=0).stop 2024-04-02T14:34:37.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:37.665+0000 7fe92f400640 1 -- 172.21.15.90:0/1208336969 shutdown_connections 2024-04-02T14:34:37.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:37.665+0000 7fe92f400640 1 --2- 172.21.15.90:0/1208336969 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe9281058f0 0x7fe92810c180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:37.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:37.665+0000 7fe92f400640 1 --2- 172.21.15.90:0/1208336969 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe928104f30 0x7fe9281053b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:37.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:37.665+0000 7fe92f400640 1 --2- 172.21.15.90:0/1208336969 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe928103d40 0x7fe928104160 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:37.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:37.665+0000 7fe92f400640 1 -- 172.21.15.90:0/1208336969 >> 172.21.15.90:0/1208336969 conn(0x7fe9280ff2b0 msgr2=0x7fe9281016f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:34:37.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:37.666+0000 7fe92f400640 1 -- 172.21.15.90:0/1208336969 shutdown_connections 2024-04-02T14:34:37.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:37.666+0000 7fe92f400640 1 -- 172.21.15.90:0/1208336969 wait complete. 2024-04-02T14:34:37.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:37.667+0000 7fe92f400640 1 Processor -- start 2024-04-02T14:34:37.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:37.667+0000 7fe92f400640 1 -- start start 2024-04-02T14:34:37.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:37.667+0000 7fe92f400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe928103d40 0x7fe92819a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:37.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:37.667+0000 7fe92f400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe928104f30 0x7fe92819a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:37.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:37.667+0000 7fe92f400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe9281058f0 0x7fe9281a1660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:37.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:37.667+0000 7fe92f400640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fe92810f9b0 con 0x7fe9281058f0 2024-04-02T14:34:37.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:37.667+0000 7fe92f400640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fe92810f830 con 0x7fe928104f30 2024-04-02T14:34:37.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:37.667+0000 7fe92f400640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fe92810fb30 con 0x7fe928103d40 2024-04-02T14:34:37.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:37.668+0000 7fe92d400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe9281058f0 0x7fe9281a1660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:37.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:37.668+0000 7fe92ca00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe928103d40 0x7fe92819a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:37.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:37.668+0000 7fe927e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe928104f30 0x7fe92819a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:37.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:37.668+0000 7fe92d400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe9281058f0 0x7fe9281a1660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:36364/0 (socket says 172.21.15.90:36364) 2024-04-02T14:34:37.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:37.668+0000 7fe92d400640 1 -- 172.21.15.90:0/2631444130 learned_addr learned my addr 172.21.15.90:0/2631444130 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:34:37.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:37.668+0000 7fe92ca00640 1 --2- 172.21.15.90:0/2631444130 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe928103d40 0x7fe92819a0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:52690/0 (socket says 172.21.15.90:52690) 2024-04-02T14:34:37.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:37.668+0000 7fe927e00640 1 -- 172.21.15.90:0/2631444130 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe928103d40 msgr2=0x7fe92819a0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:37.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:37.668+0000 7fe927e00640 1 --2- 172.21.15.90:0/2631444130 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe928103d40 0x7fe92819a0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:37.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:37.668+0000 7fe927e00640 1 -- 172.21.15.90:0/2631444130 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe9281058f0 msgr2=0x7fe9281a1660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:37.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:37.668+0000 7fe927e00640 1 --2- 172.21.15.90:0/2631444130 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe9281058f0 0x7fe9281a1660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:37.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:37.668+0000 7fe927e00640 1 -- 172.21.15.90:0/2631444130 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9281a1cd0 con 0x7fe928104f30 2024-04-02T14:34:37.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:37.668+0000 7fe92d400640 1 --2- 172.21.15.90:0/2631444130 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe9281058f0 0x7fe9281a1660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:34:37.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:37.668+0000 7fe92ca00640 1 --2- 172.21.15.90:0/2631444130 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe928103d40 0x7fe92819a0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:34:37.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:37.668+0000 7fe927e00640 1 --2- 172.21.15.90:0/2631444130 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe928104f30 0x7fe92819a5f0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fe91000e9f0 tx=0x7fe91000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:34:37.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:37.668+0000 7fe925600640 1 -- 172.21.15.90:0/2631444130 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe91000ccc0 con 0x7fe928104f30 2024-04-02T14:34:37.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:37.669+0000 7fe925600640 1 -- 172.21.15.90:0/2631444130 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe910005020 con 0x7fe928104f30 2024-04-02T14:34:37.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:37.669+0000 7fe925600640 1 -- 172.21.15.90:0/2631444130 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9100049e0 con 0x7fe928104f30 2024-04-02T14:34:37.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:37.669+0000 7fe92f400640 1 -- 172.21.15.90:0/2631444130 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9281a1fc0 con 0x7fe928104f30 2024-04-02T14:34:37.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:37.669+0000 7fe92f400640 1 -- 172.21.15.90:0/2631444130 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9281a2430 con 0x7fe928104f30 2024-04-02T14:34:37.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:37.670+0000 7fe92f400640 1 -- 172.21.15.90:0/2631444130 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe92810b0d0 con 0x7fe928104f30 2024-04-02T14:34:37.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:37.672+0000 7fe925600640 1 -- 172.21.15.90:0/2631444130 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fe910005560 con 0x7fe928104f30 2024-04-02T14:34:37.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:37.673+0000 7fe925600640 1 --2- 172.21.15.90:0/2631444130 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fe90c0a3bb0 0x7fe90c0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:37.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:37.674+0000 7fe925600640 1 -- 172.21.15.90:0/2631444130 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8248+0+0 (secure 0 0 0) 0x7fe9100d03e0 con 0x7fe928104f30 2024-04-02T14:34:37.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:37.674+0000 7fe92ca00640 1 --2- 172.21.15.90:0/2631444130 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fe90c0a3bb0 0x7fe90c0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:37.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:37.674+0000 7fe92ca00640 1 --2- 172.21.15.90:0/2631444130 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fe90c0a3bb0 0x7fe90c0a6070 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fe91802c710 tx=0x7fe918032590 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:34:37.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:37.678+0000 7fe925600640 1 -- 172.21.15.90:0/2631444130 <== mon.2 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7fe9100051d0 con 0x7fe928104f30 2024-04-02T14:34:37.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:37.884+0000 7fe92f400640 1 -- 172.21.15.90:0/2631444130 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fe9280627a0 con 0x7fe928104f30 2024-04-02T14:34:37.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:37.887+0000 7fe925600640 1 -- 172.21.15.90:0/2631444130 <== mon.2 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v90) ==== 100+0+11 (secure 0 0 0) 0x7fe91009cd50 con 0x7fe928104f30 2024-04-02T14:34:37.889 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 64 2024-04-02T14:34:37.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:37.889+0000 7fe92f400640 1 -- 172.21.15.90:0/2631444130 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fe90c0a3bb0 msgr2=0x7fe90c0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:37.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:37.889+0000 7fe92f400640 1 --2- 172.21.15.90:0/2631444130 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fe90c0a3bb0 0x7fe90c0a6070 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fe91802c710 tx=0x7fe918032590 comp rx=0 tx=0).stop 2024-04-02T14:34:37.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:37.889+0000 7fe92f400640 1 -- 172.21.15.90:0/2631444130 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe928104f30 msgr2=0x7fe92819a5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:37.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:37.889+0000 7fe92f400640 1 --2- 172.21.15.90:0/2631444130 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe928104f30 0x7fe92819a5f0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fe91000e9f0 tx=0x7fe91000eeb0 comp rx=0 tx=0).stop 2024-04-02T14:34:37.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:37.890+0000 7fe92f400640 1 -- 172.21.15.90:0/2631444130 shutdown_connections 2024-04-02T14:34:37.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:37.890+0000 7fe92f400640 1 --2- 172.21.15.90:0/2631444130 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fe90c0a3bb0 0x7fe90c0a6070 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:37.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:37.890+0000 7fe92f400640 1 --2- 172.21.15.90:0/2631444130 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe9281058f0 0x7fe9281a1660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:37.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:37.890+0000 7fe92f400640 1 --2- 172.21.15.90:0/2631444130 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe928104f30 0x7fe92819a5f0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:37.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:37.890+0000 7fe92f400640 1 --2- 172.21.15.90:0/2631444130 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe928103d40 0x7fe92819a0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:37.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:37.890+0000 7fe92f400640 1 -- 172.21.15.90:0/2631444130 >> 172.21.15.90:0/2631444130 conn(0x7fe9280ff2b0 msgr2=0x7fe9281082a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:34:37.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:37.890+0000 7fe92f400640 1 -- 172.21.15.90:0/2631444130 shutdown_connections 2024-04-02T14:34:37.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:37.890+0000 7fe92f400640 1 -- 172.21.15.90:0/2631444130 wait complete. 2024-04-02T14:34:37.912 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T14:34:38.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:38.121+0000 7f8fc0e00640 1 -- 172.21.15.90:0/1045082082 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8fbc1090e0 msgr2=0x7f8fbc109560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:38.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:38.121+0000 7f8fc0e00640 1 --2- 172.21.15.90:0/1045082082 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8fbc1090e0 0x7f8fbc109560 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f8fa4009900 tx=0x7f8fa402bc30 comp rx=0 tx=0).stop 2024-04-02T14:34:38.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:38.121+0000 7f8fc0e00640 1 -- 172.21.15.90:0/1045082082 shutdown_connections 2024-04-02T14:34:38.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:38.121+0000 7f8fc0e00640 1 --2- 172.21.15.90:0/1045082082 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8fbc109aa0 0x7f8fbc1103c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:38.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:38.121+0000 7f8fc0e00640 1 --2- 172.21.15.90:0/1045082082 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8fbc1090e0 0x7f8fbc109560 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:38.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:38.121+0000 7f8fc0e00640 1 --2- 172.21.15.90:0/1045082082 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8fbc107f10 0x7f8fbc108350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:38.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:38.121+0000 7f8fc0e00640 1 -- 172.21.15.90:0/1045082082 >> 172.21.15.90:0/1045082082 conn(0x7f8fbc076f10 msgr2=0x7f8fbc079330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:34:38.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:38.122+0000 7f8fc0e00640 1 -- 172.21.15.90:0/1045082082 shutdown_connections 2024-04-02T14:34:38.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:38.122+0000 7f8fc0e00640 1 -- 172.21.15.90:0/1045082082 wait complete. 2024-04-02T14:34:38.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:38.123+0000 7f8fc0e00640 1 Processor -- start 2024-04-02T14:34:38.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:38.123+0000 7f8fc0e00640 1 -- start start 2024-04-02T14:34:38.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:38.123+0000 7f8fc0e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8fbc107f10 0x7f8fbc19e200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:38.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:38.123+0000 7f8fc0e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8fbc1090e0 0x7f8fbc19e740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:38.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:38.123+0000 7f8fc0e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8fbc109aa0 0x7f8fbc1a57b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:38.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:38.123+0000 7f8fc0e00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f8fbc113be0 con 0x7f8fbc107f10 2024-04-02T14:34:38.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:38.123+0000 7f8fc0e00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f8fbc113a60 con 0x7f8fbc109aa0 2024-04-02T14:34:38.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:38.123+0000 7f8fc0e00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f8fbc113d60 con 0x7f8fbc1090e0 2024-04-02T14:34:38.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:38.123+0000 7f8fbb400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8fbc107f10 0x7f8fbc19e200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:38.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:38.123+0000 7f8fbb400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8fbc107f10 0x7f8fbc19e200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:36388/0 (socket says 172.21.15.90:36388) 2024-04-02T14:34:38.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:38.123+0000 7f8fbb400640 1 -- 172.21.15.90:0/2214195592 learned_addr learned my addr 172.21.15.90:0/2214195592 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:34:38.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:38.123+0000 7f8fbbe00640 1 --2- 172.21.15.90:0/2214195592 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8fbc109aa0 0x7f8fbc1a57b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:38.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:38.124+0000 7f8fb2a00640 1 --2- 172.21.15.90:0/2214195592 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8fbc1090e0 0x7f8fbc19e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:38.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:38.124+0000 7f8fbb400640 1 -- 172.21.15.90:0/2214195592 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8fbc1090e0 msgr2=0x7f8fbc19e740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:38.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:38.124+0000 7f8fbb400640 1 --2- 172.21.15.90:0/2214195592 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8fbc1090e0 0x7f8fbc19e740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:38.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:38.124+0000 7f8fbb400640 1 -- 172.21.15.90:0/2214195592 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8fbc109aa0 msgr2=0x7f8fbc1a57b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:38.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:38.124+0000 7f8fbb400640 1 --2- 172.21.15.90:0/2214195592 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8fbc109aa0 0x7f8fbc1a57b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:38.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:38.124+0000 7f8fbb400640 1 -- 172.21.15.90:0/2214195592 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8fbc1a5e20 con 0x7f8fbc107f10 2024-04-02T14:34:38.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:38.124+0000 7f8fbbe00640 1 --2- 172.21.15.90:0/2214195592 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8fbc109aa0 0x7f8fbc1a57b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:34:38.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:38.124+0000 7f8fbb400640 1 --2- 172.21.15.90:0/2214195592 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8fbc107f10 0x7f8fbc19e200 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f8fac00e9c0 tx=0x7f8fac00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:34:38.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:38.124+0000 7f8fb8c00640 1 -- 172.21.15.90:0/2214195592 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fac00cd80 con 0x7f8fbc107f10 2024-04-02T14:34:38.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:38.124+0000 7f8fc0e00640 1 -- 172.21.15.90:0/2214195592 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8fbc1a6110 con 0x7f8fbc107f10 2024-04-02T14:34:38.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:38.124+0000 7f8fb8c00640 1 -- 172.21.15.90:0/2214195592 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8fac0035f0 con 0x7f8fbc107f10 2024-04-02T14:34:38.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:38.125+0000 7f8fb8c00640 1 -- 172.21.15.90:0/2214195592 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fac010690 con 0x7f8fbc107f10 2024-04-02T14:34:38.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:38.125+0000 7f8fc0e00640 1 -- 172.21.15.90:0/2214195592 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8fbc1a6630 con 0x7f8fbc107f10 2024-04-02T14:34:38.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:38.126+0000 7f8fc0e00640 1 -- 172.21.15.90:0/2214195592 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8fbc10f2a0 con 0x7f8fbc107f10 2024-04-02T14:34:38.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:38.127+0000 7f8fb8c00640 1 -- 172.21.15.90:0/2214195592 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f8fac0037a0 con 0x7f8fbc107f10 2024-04-02T14:34:38.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:38.129+0000 7f8fb8c00640 1 --2- 172.21.15.90:0/2214195592 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f8fa00a3bb0 0x7f8fa00a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:38.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:38.129+0000 7f8fb8c00640 1 -- 172.21.15.90:0/2214195592 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8248+0+0 (secure 0 0 0) 0x7f8fac0d0850 con 0x7f8fbc107f10 2024-04-02T14:34:38.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:38.129+0000 7f8fb2a00640 1 --2- 172.21.15.90:0/2214195592 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f8fa00a3bb0 0x7f8fa00a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:38.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:38.130+0000 7f8fb2a00640 1 --2- 172.21.15.90:0/2214195592 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f8fa00a3bb0 0x7f8fa00a6070 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f8fa402c7b0 tx=0x7f8fa4005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:34:38.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:38.133+0000 7f8fb8c00640 1 -- 172.21.15.90:0/2214195592 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7f8fac09d1c0 con 0x7f8fbc107f10 2024-04-02T14:34:38.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:34:37 smithi177 ceph-mon[36431]: pgmap v381: 97 pgs: 97 active+clean; 644 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-02T14:34:38.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:34:37 smithi177 ceph-mon[36431]: from='client.? 172.21.15.90:0/3868089414' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-02T14:34:38.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:34:37 smithi177 ceph-mon[36431]: from='client.? 172.21.15.90:0/2561472502' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-02T14:34:38.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:34:37 smithi090 ceph-mon[31674]: pgmap v381: 97 pgs: 97 active+clean; 644 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-02T14:34:38.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:34:37 smithi090 ceph-mon[31674]: from='client.? 172.21.15.90:0/3868089414' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-02T14:34:38.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:34:37 smithi090 ceph-mon[31674]: from='client.? 172.21.15.90:0/2561472502' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-02T14:34:38.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:34:37 smithi092 ceph-mon[36497]: pgmap v381: 97 pgs: 97 active+clean; 644 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-02T14:34:38.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:34:37 smithi092 ceph-mon[36497]: from='client.? 172.21.15.90:0/3868089414' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-02T14:34:38.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:34:37 smithi092 ceph-mon[36497]: from='client.? 172.21.15.90:0/2561472502' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-02T14:34:38.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:38.381+0000 7f8fc0e00640 1 -- 172.21.15.90:0/2214195592 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8fbc02d370 con 0x7f8fbc107f10 2024-04-02T14:34:38.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:38.382+0000 7f8fb8c00640 1 -- 172.21.15.90:0/2214195592 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11927 (secure 0 0 0) 0x7f8fac09cac0 con 0x7f8fbc107f10 2024-04-02T14:34:38.385 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-02T14:34:38.385 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14412,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6837/2852160309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6836","nonce":2852160309},{"type":"v1","addr":"172.21.15.90:6837","nonce":2852160309}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14478,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6841/1919376858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6840","nonce":1919376858},{"type":"v1","addr":"172.21.15.90:6841","nonce":1919376858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":26,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-02T14:32:22.319083+0000","modified":"2024-04-02T14:32:29.812071+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24590,"mds_1":24560,"mds_2":24547,"mds_3":24539,"mds_4":24523},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24454":{"gid":24454,"name":"l","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6833/1564319431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":1564319431},{"type":"v1","addr":"172.21.15.177:6833","nonce":1564319431}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.92:6833/1570337372","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":1570337372},{"type":"v1","addr":"172.21.15.92:6833","nonce":1570337372}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"j","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.90:6835/1277124891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6834","nonce":1277124891},{"type":"v1","addr":"172.21.15.90:6835","nonce":1277124891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.92:6835/860721164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":860721164},{"type":"v1","addr":"172.21.15.92:6835","nonce":860721164}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24511":{"gid":24511,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6835/3177244988","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":3177244988},{"type":"v1","addr":"172.21.15.177:6835","nonce":3177244988}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"c","rank":4,"incarnation":25,"state":"up:active","state_seq":9,"addr":"172.21.15.177:6837/4223414858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":4223414858},{"type":"v1","addr":"172.21.15.177:6837","nonce":4223414858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24539":{"gid":24539,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":8,"addr":"172.21.15.92:6837/1270635939","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":1270635939},{"type":"v1","addr":"172.21.15.92:6837","nonce":1270635939}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24547":{"gid":24547,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":7,"addr":"172.21.15.177:6839/664895663","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":664895663},{"type":"v1","addr":"172.21.15.177:6839","nonce":664895663}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"g","rank":1,"incarnation":19,"state":"up:active","state_seq":5,"addr":"172.21.15.90:6839/1498915870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6838","nonce":1498915870},{"type":"v1","addr":"172.21.15.90:6839","nonce":1498915870}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24590":{"gid":24590,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.92:6839/1054888219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":1054888219},{"type":"v1","addr":"172.21.15.92:6839","nonce":1054888219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24590,"qdb_cluster":[24539,24560,24523,24590,24547]},"id":1}]} 2024-04-02T14:34:38.385 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 26 2024-04-02T14:34:38.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:38.385+0000 7f8fc0e00640 1 -- 172.21.15.90:0/2214195592 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f8fa00a3bb0 msgr2=0x7f8fa00a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:38.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:38.385+0000 7f8fc0e00640 1 --2- 172.21.15.90:0/2214195592 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f8fa00a3bb0 0x7f8fa00a6070 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f8fa402c7b0 tx=0x7f8fa4005fd0 comp rx=0 tx=0).stop 2024-04-02T14:34:38.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:38.385+0000 7f8fc0e00640 1 -- 172.21.15.90:0/2214195592 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8fbc107f10 msgr2=0x7f8fbc19e200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:38.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:38.385+0000 7f8fc0e00640 1 --2- 172.21.15.90:0/2214195592 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8fbc107f10 0x7f8fbc19e200 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f8fac00e9c0 tx=0x7f8fac00ee80 comp rx=0 tx=0).stop 2024-04-02T14:34:38.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:38.385+0000 7f8fc0e00640 1 -- 172.21.15.90:0/2214195592 shutdown_connections 2024-04-02T14:34:38.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:38.385+0000 7f8fc0e00640 1 --2- 172.21.15.90:0/2214195592 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f8fa00a3bb0 0x7f8fa00a6070 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:38.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:38.385+0000 7f8fc0e00640 1 --2- 172.21.15.90:0/2214195592 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8fbc109aa0 0x7f8fbc1a57b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:38.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:38.385+0000 7f8fc0e00640 1 --2- 172.21.15.90:0/2214195592 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8fbc1090e0 0x7f8fbc19e740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:38.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:38.385+0000 7f8fc0e00640 1 --2- 172.21.15.90:0/2214195592 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8fbc107f10 0x7f8fbc19e200 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:38.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:38.385+0000 7f8fc0e00640 1 -- 172.21.15.90:0/2214195592 >> 172.21.15.90:0/2214195592 conn(0x7f8fbc076f10 msgr2=0x7f8fbc10c470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:34:38.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:38.386+0000 7f8fc0e00640 1 -- 172.21.15.90:0/2214195592 shutdown_connections 2024-04-02T14:34:38.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:38.386+0000 7f8fc0e00640 1 -- 172.21.15.90:0/2214195592 wait complete. 2024-04-02T14:34:38.406 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T14:34:38.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:38.621+0000 7f7cd4600640 1 -- 172.21.15.90:0/3384363145 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7ccc1058f0 msgr2=0x7f7ccc10c180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:38.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:38.621+0000 7f7cd4600640 1 --2- 172.21.15.90:0/3384363145 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7ccc1058f0 0x7f7ccc10c180 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f7cc4009be0 tx=0x7f7cc402bef0 comp rx=0 tx=0).stop 2024-04-02T14:34:38.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:38.621+0000 7f7cd4600640 1 -- 172.21.15.90:0/3384363145 shutdown_connections 2024-04-02T14:34:38.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:38.621+0000 7f7cd4600640 1 --2- 172.21.15.90:0/3384363145 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7ccc1058f0 0x7f7ccc10c180 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:38.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:38.621+0000 7f7cd4600640 1 --2- 172.21.15.90:0/3384363145 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7ccc104f30 0x7f7ccc1053b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:38.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:38.621+0000 7f7cd4600640 1 --2- 172.21.15.90:0/3384363145 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7ccc103d40 0x7f7ccc104160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:38.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:38.621+0000 7f7cd4600640 1 -- 172.21.15.90:0/3384363145 >> 172.21.15.90:0/3384363145 conn(0x7f7ccc0ff2b0 msgr2=0x7f7ccc1016f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:34:38.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:38.621+0000 7f7cd4600640 1 -- 172.21.15.90:0/3384363145 shutdown_connections 2024-04-02T14:34:38.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:38.621+0000 7f7cd4600640 1 -- 172.21.15.90:0/3384363145 wait complete. 2024-04-02T14:34:38.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:38.622+0000 7f7cd4600640 1 Processor -- start 2024-04-02T14:34:38.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:38.622+0000 7f7cd4600640 1 -- start start 2024-04-02T14:34:38.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:38.622+0000 7f7cd4600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7ccc103d40 0x7f7ccc199fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:38.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:38.622+0000 7f7cd4600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7ccc104f30 0x7f7ccc19a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:38.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:38.622+0000 7f7cd4600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7ccc1058f0 0x7f7ccc1a1560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:38.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:38.622+0000 7f7cd4600640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f7ccc10f9f0 con 0x7f7ccc1058f0 2024-04-02T14:34:38.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:38.622+0000 7f7cd4600640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f7ccc10f870 con 0x7f7ccc104f30 2024-04-02T14:34:38.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:38.622+0000 7f7cd4600640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f7ccc10fb70 con 0x7f7ccc103d40 2024-04-02T14:34:38.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:38.623+0000 7f7cd1c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7ccc103d40 0x7f7ccc199fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:38.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:38.623+0000 7f7cd2600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7ccc1058f0 0x7f7ccc1a1560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:38.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:38.623+0000 7f7cd1200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7ccc104f30 0x7f7ccc19a4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:38.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:38.623+0000 7f7cd1c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7ccc103d40 0x7f7ccc199fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:52734/0 (socket says 172.21.15.90:52734) 2024-04-02T14:34:38.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:38.623+0000 7f7cd1c00640 1 -- 172.21.15.90:0/1198034704 learned_addr learned my addr 172.21.15.90:0/1198034704 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:34:38.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:38.623+0000 7f7cd1c00640 1 -- 172.21.15.90:0/1198034704 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7ccc104f30 msgr2=0x7f7ccc19a4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:38.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:38.623+0000 7f7cd1c00640 1 --2- 172.21.15.90:0/1198034704 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7ccc104f30 0x7f7ccc19a4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:38.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:38.623+0000 7f7cd1c00640 1 -- 172.21.15.90:0/1198034704 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7ccc1058f0 msgr2=0x7f7ccc1a1560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:38.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:38.623+0000 7f7cd1c00640 1 --2- 172.21.15.90:0/1198034704 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7ccc1058f0 0x7f7ccc1a1560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:38.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:38.623+0000 7f7cd1c00640 1 -- 172.21.15.90:0/1198034704 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ccc1a1bd0 con 0x7f7ccc103d40 2024-04-02T14:34:38.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:38.623+0000 7f7cd1c00640 1 --2- 172.21.15.90:0/1198034704 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7ccc103d40 0x7f7ccc199fb0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f7cc000e9f0 tx=0x7f7cc000eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:34:38.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:38.624+0000 7f7cca000640 1 -- 172.21.15.90:0/1198034704 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cc000ccc0 con 0x7f7ccc103d40 2024-04-02T14:34:38.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:38.624+0000 7f7cd4600640 1 -- 172.21.15.90:0/1198034704 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ccc1a1ec0 con 0x7f7ccc103d40 2024-04-02T14:34:38.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:38.624+0000 7f7cca000640 1 -- 172.21.15.90:0/1198034704 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7cc00035f0 con 0x7f7ccc103d40 2024-04-02T14:34:38.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:38.624+0000 7f7cca000640 1 -- 172.21.15.90:0/1198034704 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cc000ce60 con 0x7f7ccc103d40 2024-04-02T14:34:38.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:38.624+0000 7f7cd4600640 1 -- 172.21.15.90:0/1198034704 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ccc1a2320 con 0x7f7ccc103d40 2024-04-02T14:34:38.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:38.625+0000 7f7cd4600640 1 -- 172.21.15.90:0/1198034704 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7ccc10b000 con 0x7f7ccc103d40 2024-04-02T14:34:38.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:38.627+0000 7f7cca000640 1 -- 172.21.15.90:0/1198034704 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f7cc00037a0 con 0x7f7ccc103d40 2024-04-02T14:34:38.630 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:38.628+0000 7f7cca000640 1 --2- 172.21.15.90:0/1198034704 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f7cb40a3bb0 0x7f7cb40a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:38.630 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:38.629+0000 7f7cca000640 1 -- 172.21.15.90:0/1198034704 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8248+0+0 (secure 0 0 0) 0x7f7cc00d0760 con 0x7f7ccc103d40 2024-04-02T14:34:38.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:38.629+0000 7f7cd1200640 1 --2- 172.21.15.90:0/1198034704 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f7cb40a3bb0 0x7f7cb40a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:38.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:38.629+0000 7f7cd1200640 1 --2- 172.21.15.90:0/1198034704 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f7cb40a3bb0 0x7f7cb40a6070 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f7ccc19b4c0 tx=0x7f7cb8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:34:38.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:38.631+0000 7f7cca000640 1 -- 172.21.15.90:0/1198034704 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7f7cc009d0d0 con 0x7f7ccc103d40 2024-04-02T14:34:38.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:38.835+0000 7f7cd4600640 1 -- 172.21.15.90:0/1198034704 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7ccc02d560 con 0x7f7ccc103d40 2024-04-02T14:34:38.840 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-02T14:34:38.841 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:38.838+0000 7f7cca000640 1 -- 172.21.15.90:0/1198034704 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+22646 (secure 0 0 0) 0x7f7cc009c9d0 con 0x7f7ccc103d40 2024-04-02T14:34:38.841 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":90,"fsid":"6449448a-f0fc-11ee-b647-cb9ed24678a4","created":"2024-04-02T14:23:16.530201+0000","modified":"2024-04-02T14:32:35.716929+0000","last_up_change":"2024-04-02T14:31:23.819732+0000","last_in_change":"2024-04-02T14:31:02.824284+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T14:27:16.758622+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-02T14:32:20.016331+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-02T14:32:21.294367+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a1a6a856-ab44-44bd-bc5e-45c718a70cd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":3635087609},{"type":"v1","addr":"172.21.15.90:6803","nonce":3635087609}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":3635087609},{"type":"v1","addr":"172.21.15.90:6805","nonce":3635087609}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":3635087609},{"type":"v1","addr":"172.21.15.90:6809","nonce":3635087609}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":3635087609},{"type":"v1","addr":"172.21.15.90:6807","nonce":3635087609}]},"public_addr":"172.21.15.90:6803/3635087609","cluster_addr":"172.21.15.90:6805/3635087609","heartbeat_back_addr":"172.21.15.90:6809/3635087609","heartbeat_front_addr":"172.21.15.90:6807/3635087609","state":["exists","up"]},{"osd":1,"uuid":"124139cd-5793-463e-9607-9d9c10164183","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":1994536059},{"type":"v1","addr":"172.21.15.92:6801","nonce":1994536059}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":1994536059},{"type":"v1","addr":"172.21.15.92:6803","nonce":1994536059}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":1994536059},{"type":"v1","addr":"172.21.15.92:6807","nonce":1994536059}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":1994536059},{"type":"v1","addr":"172.21.15.92:6805","nonce":1994536059}]},"public_addr":"172.21.15.92:6801/1994536059","cluster_addr":"172.21.15.92:6803/1994536059","heartbeat_back_addr":"172.21.15.92:6807/1994536059","heartbeat_front_addr":"172.21.15.92:6805/1994536059","state":["exists","up"]},{"osd":2,"uuid":"a61fbef2-9dca-4875-a814-020fff0db0e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":1358472800},{"type":"v1","addr":"172.21.15.177:6801","nonce":1358472800}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":1358472800},{"type":"v1","addr":"172.21.15.177:6803","nonce":1358472800}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":1358472800},{"type":"v1","addr":"172.21.15.177:6807","nonce":1358472800}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":1358472800},{"type":"v1","addr":"172.21.15.177:6805","nonce":1358472800}]},"public_addr":"172.21.15.177:6801/1358472800","cluster_addr":"172.21.15.177:6803/1358472800","heartbeat_back_addr":"172.21.15.177:6807/1358472800","heartbeat_front_addr":"172.21.15.177:6805/1358472800","state":["exists","up"]},{"osd":3,"uuid":"b175a2f4-7859-4eb0-bc1f-fba8c2b92a2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":3114811216},{"type":"v1","addr":"172.21.15.90:6811","nonce":3114811216}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":3114811216},{"type":"v1","addr":"172.21.15.90:6813","nonce":3114811216}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":3114811216},{"type":"v1","addr":"172.21.15.90:6817","nonce":3114811216}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":3114811216},{"type":"v1","addr":"172.21.15.90:6815","nonce":3114811216}]},"public_addr":"172.21.15.90:6811/3114811216","cluster_addr":"172.21.15.90:6813/3114811216","heartbeat_back_addr":"172.21.15.90:6817/3114811216","heartbeat_front_addr":"172.21.15.90:6815/3114811216","state":["exists","up"]},{"osd":4,"uuid":"38cc68ea-c109-4d8d-b77d-1c6f9b56c875","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":2987208879},{"type":"v1","addr":"172.21.15.92:6809","nonce":2987208879}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6810","nonce":2987208879},{"type":"v1","addr":"172.21.15.92:6811","nonce":2987208879}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6814","nonce":2987208879},{"type":"v1","addr":"172.21.15.92:6815","nonce":2987208879}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":2987208879},{"type":"v1","addr":"172.21.15.92:6813","nonce":2987208879}]},"public_addr":"172.21.15.92:6809/2987208879","cluster_addr":"172.21.15.92:6811/2987208879","heartbeat_back_addr":"172.21.15.92:6815/2987208879","heartbeat_front_addr":"172.21.15.92:6813/2987208879","state":["exists","up"]},{"osd":5,"uuid":"2347232e-ac91-466f-8ac9-496f397679ba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":3486820237},{"type":"v1","addr":"172.21.15.177:6809","nonce":3486820237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":3486820237},{"type":"v1","addr":"172.21.15.177:6811","nonce":3486820237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":3486820237},{"type":"v1","addr":"172.21.15.177:6815","nonce":3486820237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":3486820237},{"type":"v1","addr":"172.21.15.177:6813","nonce":3486820237}]},"public_addr":"172.21.15.177:6809/3486820237","cluster_addr":"172.21.15.177:6811/3486820237","heartbeat_back_addr":"172.21.15.177:6815/3486820237","heartbeat_front_addr":"172.21.15.177:6813/3486820237","state":["exists","up"]},{"osd":6,"uuid":"aa72392f-0498-44db-a8d2-eccc89a64871","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":2579716443},{"type":"v1","addr":"172.21.15.90:6819","nonce":2579716443}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6820","nonce":2579716443},{"type":"v1","addr":"172.21.15.90:6821","nonce":2579716443}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":2579716443},{"type":"v1","addr":"172.21.15.90:6825","nonce":2579716443}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2579716443},{"type":"v1","addr":"172.21.15.90:6823","nonce":2579716443}]},"public_addr":"172.21.15.90:6819/2579716443","cluster_addr":"172.21.15.90:6821/2579716443","heartbeat_back_addr":"172.21.15.90:6825/2579716443","heartbeat_front_addr":"172.21.15.90:6823/2579716443","state":["exists","up"]},{"osd":7,"uuid":"931bd105-bc14-4a1e-bfe4-c7ad58fa4b73","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":3371649139},{"type":"v1","addr":"172.21.15.92:6817","nonce":3371649139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":3371649139},{"type":"v1","addr":"172.21.15.92:6819","nonce":3371649139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":3371649139},{"type":"v1","addr":"172.21.15.92:6823","nonce":3371649139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6820","nonce":3371649139},{"type":"v1","addr":"172.21.15.92:6821","nonce":3371649139}]},"public_addr":"172.21.15.92:6817/3371649139","cluster_addr":"172.21.15.92:6819/3371649139","heartbeat_back_addr":"172.21.15.92:6823/3371649139","heartbeat_front_addr":"172.21.15.92:6821/3371649139","state":["exists","up"]},{"osd":8,"uuid":"82624ec2-b382-4886-9f40-7f5f376de3ce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":1459053034},{"type":"v1","addr":"172.21.15.177:6817","nonce":1459053034}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":1459053034},{"type":"v1","addr":"172.21.15.177:6819","nonce":1459053034}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":1459053034},{"type":"v1","addr":"172.21.15.177:6823","nonce":1459053034}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":1459053034},{"type":"v1","addr":"172.21.15.177:6821","nonce":1459053034}]},"public_addr":"172.21.15.177:6817/1459053034","cluster_addr":"172.21.15.177:6819/1459053034","heartbeat_back_addr":"172.21.15.177:6823/1459053034","heartbeat_front_addr":"172.21.15.177:6821/1459053034","state":["exists","up"]},{"osd":9,"uuid":"fbe36e48-3468-42b2-8094-e16552f3cb16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":2592604332},{"type":"v1","addr":"172.21.15.90:6827","nonce":2592604332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":2592604332},{"type":"v1","addr":"172.21.15.90:6829","nonce":2592604332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6832","nonce":2592604332},{"type":"v1","addr":"172.21.15.90:6833","nonce":2592604332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":2592604332},{"type":"v1","addr":"172.21.15.90:6831","nonce":2592604332}]},"public_addr":"172.21.15.90:6827/2592604332","cluster_addr":"172.21.15.90:6829/2592604332","heartbeat_back_addr":"172.21.15.90:6833/2592604332","heartbeat_front_addr":"172.21.15.90:6831/2592604332","state":["exists","up"]},{"osd":10,"uuid":"9240df75-e7e2-47f1-8aae-534b70b3c210","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":1594153142},{"type":"v1","addr":"172.21.15.92:6825","nonce":1594153142}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6826","nonce":1594153142},{"type":"v1","addr":"172.21.15.92:6827","nonce":1594153142}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":1594153142},{"type":"v1","addr":"172.21.15.92:6831","nonce":1594153142}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":1594153142},{"type":"v1","addr":"172.21.15.92:6829","nonce":1594153142}]},"public_addr":"172.21.15.92:6825/1594153142","cluster_addr":"172.21.15.92:6827/1594153142","heartbeat_back_addr":"172.21.15.92:6831/1594153142","heartbeat_front_addr":"172.21.15.92:6829/1594153142","state":["exists","up"]},{"osd":11,"uuid":"83cba9fa-b745-4df1-adda-e86837d4b91a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":83,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1622582126},{"type":"v1","addr":"172.21.15.177:6825","nonce":1622582126}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":1622582126},{"type":"v1","addr":"172.21.15.177:6827","nonce":1622582126}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":1622582126},{"type":"v1","addr":"172.21.15.177:6831","nonce":1622582126}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1622582126},{"type":"v1","addr":"172.21.15.177:6829","nonce":1622582126}]},"public_addr":"172.21.15.177:6825/1622582126","cluster_addr":"172.21.15.177:6827/1622582126","heartbeat_back_addr":"172.21.15.177:6831/1622582126","heartbeat_front_addr":"172.21.15.177:6829/1622582126","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:26:24.290365+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:26:49.405988+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:27:13.639196+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:27:45.603746+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:28:11.019126+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:28:35.084163+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:29:08.064626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:29:33.266133+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:29:57.381816+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:30:30.286847+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:30:56.226279+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:31:20.809611+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":7,"to":11}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":8,"to":4}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.11","mappings":[{"from":4,"to":8},{"from":5,"to":9}]},{"pgid":"3.21","mappings":[{"from":5,"to":9}]},{"pgid":"3.23","mappings":[{"from":4,"to":7}]},{"pgid":"3.3a","mappings":[{"from":4,"to":6}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.90:0/314641601":"2024-04-03T14:24:34.595561+0000","172.21.15.90:0/1400895868":"2024-04-03T14:24:34.595561+0000","172.21.15.90:6801/129436436":"2024-04-03T14:24:34.595561+0000","172.21.15.90:6801/2372398598":"2024-04-03T14:24:00.362147+0000","172.21.15.90:0/278569338":"2024-04-03T14:24:34.595561+0000","172.21.15.90:0/2321464594":"2024-04-03T14:24:00.362147+0000","172.21.15.90:6800/2372398598":"2024-04-03T14:24:00.362147+0000","172.21.15.90:0/2962656672":"2024-04-03T14:24:00.362147+0000","172.21.15.90:0/126693610":"2024-04-03T14:24:00.362147+0000","172.21.15.90:6800/129436436":"2024-04-03T14:24:34.595561+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T14:34:38.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:38.841+0000 7f7cd4600640 1 -- 172.21.15.90:0/1198034704 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f7cb40a3bb0 msgr2=0x7f7cb40a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:38.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:38.841+0000 7f7cd4600640 1 --2- 172.21.15.90:0/1198034704 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f7cb40a3bb0 0x7f7cb40a6070 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f7ccc19b4c0 tx=0x7f7cb8005f90 comp rx=0 tx=0).stop 2024-04-02T14:34:38.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:38.841+0000 7f7cd4600640 1 -- 172.21.15.90:0/1198034704 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7ccc103d40 msgr2=0x7f7ccc199fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:38.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:38.841+0000 7f7cd4600640 1 --2- 172.21.15.90:0/1198034704 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7ccc103d40 0x7f7ccc199fb0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f7cc000e9f0 tx=0x7f7cc000eeb0 comp rx=0 tx=0).stop 2024-04-02T14:34:38.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:38.841+0000 7f7cd4600640 1 -- 172.21.15.90:0/1198034704 shutdown_connections 2024-04-02T14:34:38.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:38.841+0000 7f7cd4600640 1 --2- 172.21.15.90:0/1198034704 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f7cb40a3bb0 0x7f7cb40a6070 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:38.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:38.841+0000 7f7cd4600640 1 --2- 172.21.15.90:0/1198034704 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7ccc1058f0 0x7f7ccc1a1560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:38.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:38.841+0000 7f7cd4600640 1 --2- 172.21.15.90:0/1198034704 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7ccc104f30 0x7f7ccc19a4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:38.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:38.841+0000 7f7cd4600640 1 --2- 172.21.15.90:0/1198034704 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7ccc103d40 0x7f7ccc199fb0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:38.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:38.841+0000 7f7cd4600640 1 -- 172.21.15.90:0/1198034704 >> 172.21.15.90:0/1198034704 conn(0x7f7ccc0ff2b0 msgr2=0x7f7ccc1081d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:34:38.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:38.841+0000 7f7cd4600640 1 -- 172.21.15.90:0/1198034704 shutdown_connections 2024-04-02T14:34:38.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:38.841+0000 7f7cd4600640 1 -- 172.21.15.90:0/1198034704 wait complete. 2024-04-02T14:34:38.862 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-02T14:27:16.758622+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-02T14:32:20.016331+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-02T14:32:21.294367+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '88', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-02T14:34:38.862 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-02T14:34:39.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:39.072+0000 7f9672000640 1 -- 172.21.15.90:0/1327946153 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f966c104f20 msgr2=0x7f966c1053a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:39.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:39.072+0000 7f9672000640 1 --2- 172.21.15.90:0/1327946153 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f966c104f20 0x7f966c1053a0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f9658009860 tx=0x7f965802bb70 comp rx=0 tx=0).stop 2024-04-02T14:34:39.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:39.072+0000 7f9672000640 1 -- 172.21.15.90:0/1327946153 shutdown_connections 2024-04-02T14:34:39.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:39.072+0000 7f9672000640 1 --2- 172.21.15.90:0/1327946153 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f966c1058e0 0x7f966c10c170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:39.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:39.072+0000 7f9672000640 1 --2- 172.21.15.90:0/1327946153 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f966c104f20 0x7f966c1053a0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:39.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:39.072+0000 7f9672000640 1 --2- 172.21.15.90:0/1327946153 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f966c103d30 0x7f966c104150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:39.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:39.072+0000 7f9672000640 1 -- 172.21.15.90:0/1327946153 >> 172.21.15.90:0/1327946153 conn(0x7f966c0ff2c0 msgr2=0x7f966c1016e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:34:39.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:39.072+0000 7f9672000640 1 -- 172.21.15.90:0/1327946153 shutdown_connections 2024-04-02T14:34:39.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:39.072+0000 7f9672000640 1 -- 172.21.15.90:0/1327946153 wait complete. 2024-04-02T14:34:39.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:39.073+0000 7f9672000640 1 Processor -- start 2024-04-02T14:34:39.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:39.073+0000 7f9672000640 1 -- start start 2024-04-02T14:34:39.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:39.073+0000 7f9672000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f966c103d30 0x7f966c19a060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:39.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:39.073+0000 7f9672000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f966c104f20 0x7f966c19a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:39.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:39.073+0000 7f9672000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f966c1058e0 0x7f966c1a1610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:39.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:39.073+0000 7f9672000640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f966c10f990 con 0x7f966c103d30 2024-04-02T14:34:39.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:39.073+0000 7f9672000640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f966c10f810 con 0x7f966c104f20 2024-04-02T14:34:39.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:39.073+0000 7f9672000640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f966c10fb10 con 0x7f966c1058e0 2024-04-02T14:34:39.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:39.074+0000 7f966b400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f966c103d30 0x7f966c19a060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:39.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:39.074+0000 7f966b400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f966c103d30 0x7f966c19a060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:36414/0 (socket says 172.21.15.90:36414) 2024-04-02T14:34:39.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:39.074+0000 7f966b400640 1 -- 172.21.15.90:0/3373745469 learned_addr learned my addr 172.21.15.90:0/3373745469 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:34:39.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:39.074+0000 7f966aa00640 1 --2- 172.21.15.90:0/3373745469 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f966c104f20 0x7f966c19a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:39.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:39.074+0000 7f966be00640 1 --2- 172.21.15.90:0/3373745469 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f966c1058e0 0x7f966c1a1610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:39.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:39.074+0000 7f966aa00640 1 -- 172.21.15.90:0/3373745469 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f966c1058e0 msgr2=0x7f966c1a1610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:39.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:39.074+0000 7f966aa00640 1 --2- 172.21.15.90:0/3373745469 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f966c1058e0 0x7f966c1a1610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:39.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:39.074+0000 7f966aa00640 1 -- 172.21.15.90:0/3373745469 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f966c103d30 msgr2=0x7f966c19a060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:39.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:39.074+0000 7f966aa00640 1 --2- 172.21.15.90:0/3373745469 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f966c103d30 0x7f966c19a060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:39.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:39.074+0000 7f966aa00640 1 -- 172.21.15.90:0/3373745469 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f966c1a1c80 con 0x7f966c104f20 2024-04-02T14:34:39.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:39.074+0000 7f966be00640 1 --2- 172.21.15.90:0/3373745469 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f966c1058e0 0x7f966c1a1610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:34:39.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:39.074+0000 7f966b400640 1 --2- 172.21.15.90:0/3373745469 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f966c103d30 0x7f966c19a060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:34:39.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:39.074+0000 7f966aa00640 1 --2- 172.21.15.90:0/3373745469 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f966c104f20 0x7f966c19a5a0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f965802c0b0 tx=0x7f9658032430 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:34:39.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:39.075+0000 7f9663e00640 1 -- 172.21.15.90:0/3373745469 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9658041020 con 0x7f966c104f20 2024-04-02T14:34:39.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:39.075+0000 7f9663e00640 1 -- 172.21.15.90:0/3373745469 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f965802c9f0 con 0x7f966c104f20 2024-04-02T14:34:39.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:39.075+0000 7f9672000640 1 -- 172.21.15.90:0/3373745469 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f966c1a1f10 con 0x7f966c104f20 2024-04-02T14:34:39.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:39.075+0000 7f9663e00640 1 -- 172.21.15.90:0/3373745469 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9658039040 con 0x7f966c104f20 2024-04-02T14:34:39.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:39.075+0000 7f9672000640 1 -- 172.21.15.90:0/3373745469 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f966c1a2380 con 0x7f966c104f20 2024-04-02T14:34:39.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:39.076+0000 7f9672000640 1 -- 172.21.15.90:0/3373745469 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f966c10b070 con 0x7f966c104f20 2024-04-02T14:34:39.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:39.079+0000 7f9663e00640 1 -- 172.21.15.90:0/3373745469 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f9658044020 con 0x7f966c104f20 2024-04-02T14:34:39.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:39.080+0000 7f9663e00640 1 --2- 172.21.15.90:0/3373745469 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f96500a3c30 0x7f96500a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:39.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:39.080+0000 7f9663e00640 1 -- 172.21.15.90:0/3373745469 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8248+0+0 (secure 0 0 0) 0x7f96580f1c80 con 0x7f966c104f20 2024-04-02T14:34:39.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:39.080+0000 7f966b400640 1 --2- 172.21.15.90:0/3373745469 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f96500a3c30 0x7f96500a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:39.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:39.081+0000 7f966b400640 1 --2- 172.21.15.90:0/3373745469 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f96500a3c30 0x7f96500a60f0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f965c0058c0 tx=0x7f965c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:34:39.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:39.084+0000 7f9663e00640 1 -- 172.21.15.90:0/3373745469 <== mon.2 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7f965804b020 con 0x7f966c104f20 2024-04-02T14:34:39.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:34:38 smithi177 ceph-mon[36431]: from='client.? 172.21.15.90:0/2631444130' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-02T14:34:39.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:34:38 smithi177 ceph-mon[36431]: from='client.? 172.21.15.90:0/2214195592' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-02T14:34:39.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:34:38 smithi177 ceph-mon[36431]: from='client.? 172.21.15.90:0/1198034704' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-02T14:34:39.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:34:38 smithi090 ceph-mon[31674]: from='client.? 172.21.15.90:0/2631444130' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-02T14:34:39.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:34:38 smithi090 ceph-mon[31674]: from='client.? 172.21.15.90:0/2214195592' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-02T14:34:39.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:34:38 smithi090 ceph-mon[31674]: from='client.? 172.21.15.90:0/1198034704' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-02T14:34:39.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:39.283+0000 7f9672000640 1 -- 172.21.15.90:0/3373745469 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f966c062700 con 0x7f966c104f20 2024-04-02T14:34:39.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:39.284+0000 7f9663e00640 1 -- 172.21.15.90:0/3373745469 <== mon.2 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v90) ==== 93+0+10 (secure 0 0 0) 0x7f96580be5f0 con 0x7f966c104f20 2024-04-02T14:34:39.286 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 1 2024-04-02T14:34:39.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:39.287+0000 7f9672000640 1 -- 172.21.15.90:0/3373745469 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f96500a3c30 msgr2=0x7f96500a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:39.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:39.287+0000 7f9672000640 1 --2- 172.21.15.90:0/3373745469 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f96500a3c30 0x7f96500a60f0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f965c0058c0 tx=0x7f965c005f90 comp rx=0 tx=0).stop 2024-04-02T14:34:39.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:39.287+0000 7f9672000640 1 -- 172.21.15.90:0/3373745469 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f966c104f20 msgr2=0x7f966c19a5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:39.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:39.287+0000 7f9672000640 1 --2- 172.21.15.90:0/3373745469 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f966c104f20 0x7f966c19a5a0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f965802c0b0 tx=0x7f9658032430 comp rx=0 tx=0).stop 2024-04-02T14:34:39.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:39.287+0000 7f9672000640 1 -- 172.21.15.90:0/3373745469 shutdown_connections 2024-04-02T14:34:39.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:39.287+0000 7f9672000640 1 --2- 172.21.15.90:0/3373745469 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f96500a3c30 0x7f96500a60f0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:39.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:39.287+0000 7f9672000640 1 --2- 172.21.15.90:0/3373745469 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f966c1058e0 0x7f966c1a1610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:39.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:39.287+0000 7f9672000640 1 --2- 172.21.15.90:0/3373745469 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f966c104f20 0x7f966c19a5a0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:39.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:39.287+0000 7f9672000640 1 --2- 172.21.15.90:0/3373745469 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f966c103d30 0x7f966c19a060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:39.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:39.287+0000 7f9672000640 1 -- 172.21.15.90:0/3373745469 >> 172.21.15.90:0/3373745469 conn(0x7f966c0ff2c0 msgr2=0x7f966c108240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:34:39.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:39.287+0000 7f9672000640 1 -- 172.21.15.90:0/3373745469 shutdown_connections 2024-04-02T14:34:39.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:39.287+0000 7f9672000640 1 -- 172.21.15.90:0/3373745469 wait complete. 2024-04-02T14:34:39.307 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-02T14:34:39.360 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:34:38 smithi092 ceph-mon[36497]: from='client.? 172.21.15.90:0/2631444130' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-02T14:34:39.360 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:34:38 smithi092 ceph-mon[36497]: from='client.? 172.21.15.90:0/2214195592' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-02T14:34:39.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:34:38 smithi092 ceph-mon[36497]: from='client.? 172.21.15.90:0/1198034704' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-02T14:34:39.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:39.515+0000 7f3189400640 1 -- 172.21.15.90:0/2053683999 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3184100380 msgr2=0x7f31841027b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:39.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:39.515+0000 7f3189400640 1 --2- 172.21.15.90:0/2053683999 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3184100380 0x7f31841027b0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f3174009860 tx=0x7f317402bb70 comp rx=0 tx=0).stop 2024-04-02T14:34:39.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:39.515+0000 7f3189400640 1 -- 172.21.15.90:0/2053683999 shutdown_connections 2024-04-02T14:34:39.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:39.515+0000 7f3189400640 1 --2- 172.21.15.90:0/2053683999 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f31841056a0 0x7f3184105b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:39.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:39.515+0000 7f3189400640 1 --2- 172.21.15.90:0/2053683999 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3184102cf0 0x7f3184105160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:39.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:39.515+0000 7f3189400640 1 --2- 172.21.15.90:0/2053683999 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3184100380 0x7f31841027b0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:39.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:39.515+0000 7f3189400640 1 -- 172.21.15.90:0/2053683999 >> 172.21.15.90:0/2053683999 conn(0x7f31840fbfa0 msgr2=0x7f31840fe3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:34:39.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:39.515+0000 7f3189400640 1 -- 172.21.15.90:0/2053683999 shutdown_connections 2024-04-02T14:34:39.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:39.515+0000 7f3189400640 1 -- 172.21.15.90:0/2053683999 wait complete. 2024-04-02T14:34:39.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:39.516+0000 7f3189400640 1 Processor -- start 2024-04-02T14:34:39.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:39.516+0000 7f3189400640 1 -- start start 2024-04-02T14:34:39.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:39.516+0000 7f3189400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3184100380 0x7f318419e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:39.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:39.516+0000 7f3189400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3184102cf0 0x7f318419e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:39.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:39.516+0000 7f3189400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f31841056a0 0x7f31841a5a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:39.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:39.516+0000 7f3189400640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f3184113db0 con 0x7f3184100380 2024-04-02T14:34:39.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:39.516+0000 7f3189400640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f3184113c30 con 0x7f3184102cf0 2024-04-02T14:34:39.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:39.516+0000 7f3189400640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f3184113f30 con 0x7f31841056a0 2024-04-02T14:34:39.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:39.517+0000 7f3182800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3184100380 0x7f318419e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:39.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:39.517+0000 7f3182800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3184100380 0x7f318419e470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:36422/0 (socket says 172.21.15.90:36422) 2024-04-02T14:34:39.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:39.517+0000 7f3182800640 1 -- 172.21.15.90:0/3043197937 learned_addr learned my addr 172.21.15.90:0/3043197937 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:34:39.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:39.517+0000 7f3179e00640 1 --2- 172.21.15.90:0/3043197937 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3184102cf0 0x7f318419e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:39.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:39.517+0000 7f3183200640 1 --2- 172.21.15.90:0/3043197937 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f31841056a0 0x7f31841a5a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:39.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:39.517+0000 7f3179e00640 1 -- 172.21.15.90:0/3043197937 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f31841056a0 msgr2=0x7f31841a5a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:39.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:39.517+0000 7f3179e00640 1 --2- 172.21.15.90:0/3043197937 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f31841056a0 0x7f31841a5a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:39.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:39.517+0000 7f3179e00640 1 -- 172.21.15.90:0/3043197937 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3184100380 msgr2=0x7f318419e470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:39.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:39.517+0000 7f3179e00640 1 --2- 172.21.15.90:0/3043197937 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3184100380 0x7f318419e470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:39.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:39.517+0000 7f3179e00640 1 -- 172.21.15.90:0/3043197937 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31841a6090 con 0x7f3184102cf0 2024-04-02T14:34:39.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:39.517+0000 7f3179e00640 1 --2- 172.21.15.90:0/3043197937 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3184102cf0 0x7f318419e9b0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f316c00e9f0 tx=0x7f316c00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:34:39.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:39.518+0000 7f317be00640 1 -- 172.21.15.90:0/3043197937 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f316c00ccc0 con 0x7f3184102cf0 2024-04-02T14:34:39.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:39.518+0000 7f317be00640 1 -- 172.21.15.90:0/3043197937 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f316c005020 con 0x7f3184102cf0 2024-04-02T14:34:39.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:39.518+0000 7f3189400640 1 -- 172.21.15.90:0/3043197937 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f31841a6380 con 0x7f3184102cf0 2024-04-02T14:34:39.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:39.518+0000 7f317be00640 1 -- 172.21.15.90:0/3043197937 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f316c0049e0 con 0x7f3184102cf0 2024-04-02T14:34:39.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:39.518+0000 7f3189400640 1 -- 172.21.15.90:0/3043197937 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31841a68d0 con 0x7f3184102cf0 2024-04-02T14:34:39.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:39.519+0000 7f3189400640 1 -- 172.21.15.90:0/3043197937 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f31840fda90 con 0x7f3184102cf0 2024-04-02T14:34:39.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:39.521+0000 7f317be00640 1 -- 172.21.15.90:0/3043197937 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f316c005560 con 0x7f3184102cf0 2024-04-02T14:34:39.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:39.522+0000 7f317be00640 1 --2- 172.21.15.90:0/3043197937 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f31680a3c30 0x7f31680a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:39.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:39.522+0000 7f317be00640 1 -- 172.21.15.90:0/3043197937 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8248+0+0 (secure 0 0 0) 0x7f316c0d0420 con 0x7f3184102cf0 2024-04-02T14:34:39.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:39.523+0000 7f3182800640 1 --2- 172.21.15.90:0/3043197937 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f31680a3c30 0x7f31680a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:39.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:39.523+0000 7f3182800640 1 --2- 172.21.15.90:0/3043197937 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f31680a3c30 0x7f31680a60f0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f31740095f0 tx=0x7f3174034ab0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:34:39.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:39.524+0000 7f317be00640 1 -- 172.21.15.90:0/3043197937 <== mon.2 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7f316c0051d0 con 0x7f3184102cf0 2024-04-02T14:34:39.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:39.719+0000 7f3189400640 1 -- 172.21.15.90:0/3043197937 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f318402d5c0 con 0x7f3184102cf0 2024-04-02T14:34:39.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:39.720+0000 7f317be00640 1 -- 172.21.15.90:0/3043197937 <== mon.2 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v90) ==== 104+0+11 (secure 0 0 0) 0x7f316c09ce20 con 0x7f3184102cf0 2024-04-02T14:34:39.722 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-04-02T14:34:39.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:39.722+0000 7f3189400640 1 -- 172.21.15.90:0/3043197937 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f31680a3c30 msgr2=0x7f31680a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:39.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:39.722+0000 7f3189400640 1 --2- 172.21.15.90:0/3043197937 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f31680a3c30 0x7f31680a60f0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f31740095f0 tx=0x7f3174034ab0 comp rx=0 tx=0).stop 2024-04-02T14:34:39.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:39.722+0000 7f3189400640 1 -- 172.21.15.90:0/3043197937 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3184102cf0 msgr2=0x7f318419e9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:39.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:39.722+0000 7f3189400640 1 --2- 172.21.15.90:0/3043197937 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3184102cf0 0x7f318419e9b0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f316c00e9f0 tx=0x7f316c00eeb0 comp rx=0 tx=0).stop 2024-04-02T14:34:39.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:39.722+0000 7f3189400640 1 -- 172.21.15.90:0/3043197937 shutdown_connections 2024-04-02T14:34:39.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:39.722+0000 7f3189400640 1 --2- 172.21.15.90:0/3043197937 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f31680a3c30 0x7f31680a60f0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:39.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:39.722+0000 7f3189400640 1 --2- 172.21.15.90:0/3043197937 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f31841056a0 0x7f31841a5a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:39.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:39.722+0000 7f3189400640 1 --2- 172.21.15.90:0/3043197937 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3184102cf0 0x7f318419e9b0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:39.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:39.722+0000 7f3189400640 1 --2- 172.21.15.90:0/3043197937 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3184100380 0x7f318419e470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:39.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:39.722+0000 7f3189400640 1 -- 172.21.15.90:0/3043197937 >> 172.21.15.90:0/3043197937 conn(0x7f31840fbfa0 msgr2=0x7f3184103610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:34:39.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:39.722+0000 7f3189400640 1 -- 172.21.15.90:0/3043197937 shutdown_connections 2024-04-02T14:34:39.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:39.722+0000 7f3189400640 1 -- 172.21.15.90:0/3043197937 wait complete. 2024-04-02T14:34:39.742 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-02T14:34:39.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:39.931+0000 7f6754400640 1 -- 172.21.15.90:0/3627443151 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f674c104ec0 msgr2=0x7f674c1072b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:39.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:39.931+0000 7f6754400640 1 --2- 172.21.15.90:0/3627443151 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f674c104ec0 0x7f674c1072b0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f6738009900 tx=0x7f673802bc30 comp rx=0 tx=0).stop 2024-04-02T14:34:39.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:39.931+0000 7f6754400640 1 -- 172.21.15.90:0/3627443151 shutdown_connections 2024-04-02T14:34:39.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:39.931+0000 7f6754400640 1 --2- 172.21.15.90:0/3627443151 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f674c1077f0 0x7f674c109c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:39.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:39.931+0000 7f6754400640 1 --2- 172.21.15.90:0/3627443151 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f674c104ec0 0x7f674c1072b0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:39.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:39.931+0000 7f6754400640 1 --2- 172.21.15.90:0/3627443151 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f674c102590 0x7f674c104980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:39.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:39.931+0000 7f6754400640 1 -- 172.21.15.90:0/3627443151 >> 172.21.15.90:0/3627443151 conn(0x7f674c0fbfa0 msgr2=0x7f674c0fe3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:34:39.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:39.931+0000 7f6754400640 1 -- 172.21.15.90:0/3627443151 shutdown_connections 2024-04-02T14:34:39.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:39.931+0000 7f6754400640 1 -- 172.21.15.90:0/3627443151 wait complete. 2024-04-02T14:34:39.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:39.932+0000 7f6754400640 1 Processor -- start 2024-04-02T14:34:39.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:39.932+0000 7f6754400640 1 -- start start 2024-04-02T14:34:39.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:39.932+0000 7f6754400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f674c102590 0x7f674c19a060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:39.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:39.932+0000 7f6754400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f674c104ec0 0x7f674c19a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:39.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:39.932+0000 7f6754400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f674c1077f0 0x7f674c1a1610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:39.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:39.932+0000 7f6754400640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f674c10d7f0 con 0x7f674c104ec0 2024-04-02T14:34:39.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:39.932+0000 7f6754400640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f674c10d670 con 0x7f674c102590 2024-04-02T14:34:39.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:39.932+0000 7f6754400640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f674c10d970 con 0x7f674c1077f0 2024-04-02T14:34:39.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:39.932+0000 7f6751000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f674c104ec0 0x7f674c19a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:39.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:39.932+0000 7f6751000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f674c104ec0 0x7f674c19a5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:36430/0 (socket says 172.21.15.90:36430) 2024-04-02T14:34:39.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:39.932+0000 7f6751000640 1 -- 172.21.15.90:0/3315420596 learned_addr learned my addr 172.21.15.90:0/3315420596 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:34:39.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:39.932+0000 7f6751a00640 1 --2- 172.21.15.90:0/3315420596 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f674c102590 0x7f674c19a060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:39.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:39.933+0000 7f6752400640 1 --2- 172.21.15.90:0/3315420596 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f674c1077f0 0x7f674c1a1610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:39.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:39.933+0000 7f6751000640 1 -- 172.21.15.90:0/3315420596 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f674c1077f0 msgr2=0x7f674c1a1610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:39.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:39.933+0000 7f6751000640 1 --2- 172.21.15.90:0/3315420596 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f674c1077f0 0x7f674c1a1610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:39.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:39.933+0000 7f6751000640 1 -- 172.21.15.90:0/3315420596 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f674c102590 msgr2=0x7f674c19a060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:39.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:39.933+0000 7f6751000640 1 --2- 172.21.15.90:0/3315420596 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f674c102590 0x7f674c19a060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:39.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:39.933+0000 7f6751000640 1 -- 172.21.15.90:0/3315420596 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f674c1a1c80 con 0x7f674c104ec0 2024-04-02T14:34:39.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:39.933+0000 7f6751a00640 1 --2- 172.21.15.90:0/3315420596 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f674c102590 0x7f674c19a060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:34:39.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:39.933+0000 7f6751000640 1 --2- 172.21.15.90:0/3315420596 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f674c104ec0 0x7f674c19a5a0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f6738000c00 tx=0x7f673802c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:34:39.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:39.933+0000 7f674a000640 1 -- 172.21.15.90:0/3315420596 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6738043020 con 0x7f674c104ec0 2024-04-02T14:34:39.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:39.933+0000 7f674a000640 1 -- 172.21.15.90:0/3315420596 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f67380054e0 con 0x7f674c104ec0 2024-04-02T14:34:39.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:39.933+0000 7f6754400640 1 -- 172.21.15.90:0/3315420596 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f674c1a1f10 con 0x7f674c104ec0 2024-04-02T14:34:39.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:39.933+0000 7f674a000640 1 -- 172.21.15.90:0/3315420596 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6738003c40 con 0x7f674c104ec0 2024-04-02T14:34:39.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:39.934+0000 7f6754400640 1 -- 172.21.15.90:0/3315420596 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f674c1a2420 con 0x7f674c104ec0 2024-04-02T14:34:39.938 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:34:39 smithi090 ceph-mon[31674]: pgmap v382: 97 pgs: 97 active+clean; 644 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-02T14:34:39.938 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:34:39 smithi090 ceph-mon[31674]: from='client.? 172.21.15.90:0/3373745469' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-02T14:34:39.938 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:34:39 smithi090 ceph-mon[31674]: from='client.? 172.21.15.90:0/3043197937' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-02T14:34:39.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:39.936+0000 7f6754400640 1 -- 172.21.15.90:0/3315420596 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f674c109340 con 0x7f674c104ec0 2024-04-02T14:34:39.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:39.937+0000 7f674a000640 1 -- 172.21.15.90:0/3315420596 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f6738046020 con 0x7f674c104ec0 2024-04-02T14:34:39.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:39.938+0000 7f674a000640 1 --2- 172.21.15.90:0/3315420596 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f67340a3c30 0x7f67340a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:39.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:39.939+0000 7f674a000640 1 -- 172.21.15.90:0/3315420596 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8248+0+0 (secure 0 0 0) 0x7f67380f2390 con 0x7f674c104ec0 2024-04-02T14:34:39.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:39.939+0000 7f6751a00640 1 --2- 172.21.15.90:0/3315420596 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f67340a3c30 0x7f67340a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:39.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:39.939+0000 7f6751a00640 1 --2- 172.21.15.90:0/3315420596 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f67340a3c30 0x7f67340a60f0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f6740009fc0 tx=0x7f6740005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:34:39.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:39.943+0000 7f674a000640 1 -- 172.21.15.90:0/3315420596 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7f673804d020 con 0x7f674c104ec0 2024-04-02T14:34:40.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:40.137+0000 7f6754400640 1 -- 172.21.15.90:0/3315420596 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f674c02d370 con 0x7f674c104ec0 2024-04-02T14:34:40.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:40.138+0000 7f674a000640 1 -- 172.21.15.90:0/3315420596 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v90) ==== 100+0+11 (secure 0 0 0) 0x7f67380bed00 con 0x7f674c104ec0 2024-04-02T14:34:40.140 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 64 2024-04-02T14:34:40.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:40.140+0000 7f6754400640 1 -- 172.21.15.90:0/3315420596 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f67340a3c30 msgr2=0x7f67340a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:40.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:40.140+0000 7f6754400640 1 --2- 172.21.15.90:0/3315420596 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f67340a3c30 0x7f67340a60f0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f6740009fc0 tx=0x7f6740005f90 comp rx=0 tx=0).stop 2024-04-02T14:34:40.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:40.140+0000 7f6754400640 1 -- 172.21.15.90:0/3315420596 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f674c104ec0 msgr2=0x7f674c19a5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:40.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:40.140+0000 7f6754400640 1 --2- 172.21.15.90:0/3315420596 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f674c104ec0 0x7f674c19a5a0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f6738000c00 tx=0x7f673802c650 comp rx=0 tx=0).stop 2024-04-02T14:34:40.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:40.140+0000 7f6754400640 1 -- 172.21.15.90:0/3315420596 shutdown_connections 2024-04-02T14:34:40.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:40.140+0000 7f6754400640 1 --2- 172.21.15.90:0/3315420596 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f67340a3c30 0x7f67340a60f0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:40.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:40.140+0000 7f6754400640 1 --2- 172.21.15.90:0/3315420596 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f674c1077f0 0x7f674c1a1610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:40.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:40.140+0000 7f6754400640 1 --2- 172.21.15.90:0/3315420596 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f674c104ec0 0x7f674c19a5a0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:40.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:40.140+0000 7f6754400640 1 --2- 172.21.15.90:0/3315420596 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f674c102590 0x7f674c19a060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:40.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:40.140+0000 7f6754400640 1 -- 172.21.15.90:0/3315420596 >> 172.21.15.90:0/3315420596 conn(0x7f674c0fbfa0 msgr2=0x7f674c101c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:34:40.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:40.140+0000 7f6754400640 1 -- 172.21.15.90:0/3315420596 shutdown_connections 2024-04-02T14:34:40.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:40.141+0000 7f6754400640 1 -- 172.21.15.90:0/3315420596 wait complete. 2024-04-02T14:34:40.160 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T14:34:40.188 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:34:39 smithi177 ceph-mon[36431]: pgmap v382: 97 pgs: 97 active+clean; 644 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-02T14:34:40.188 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:34:39 smithi177 ceph-mon[36431]: from='client.? 172.21.15.90:0/3373745469' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-02T14:34:40.188 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:34:39 smithi177 ceph-mon[36431]: from='client.? 172.21.15.90:0/3043197937' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-02T14:34:40.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:40.358+0000 7f0dcd000640 1 -- 172.21.15.90:0/1852155143 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0dc8101970 msgr2=0x7f0dc8101d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:40.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:40.358+0000 7f0dcd000640 1 --2- 172.21.15.90:0/1852155143 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0dc8101970 0x7f0dc8101d90 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f0db8009900 tx=0x7f0db802bc50 comp rx=0 tx=0).stop 2024-04-02T14:34:40.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:40.358+0000 7f0dcd000640 1 -- 172.21.15.90:0/1852155143 shutdown_connections 2024-04-02T14:34:40.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:40.358+0000 7f0dcd000640 1 --2- 172.21.15.90:0/1852155143 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0dc8103520 0x7f0dc8109db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:40.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:40.358+0000 7f0dcd000640 1 --2- 172.21.15.90:0/1852155143 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0dc8102b60 0x7f0dc8102fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:40.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:40.358+0000 7f0dcd000640 1 --2- 172.21.15.90:0/1852155143 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0dc8101970 0x7f0dc8101d90 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:40.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:40.358+0000 7f0dcd000640 1 -- 172.21.15.90:0/1852155143 >> 172.21.15.90:0/1852155143 conn(0x7f0dc80fcee0 msgr2=0x7f0dc80ff320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:34:40.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:40.358+0000 7f0dcd000640 1 -- 172.21.15.90:0/1852155143 shutdown_connections 2024-04-02T14:34:40.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:40.358+0000 7f0dcd000640 1 -- 172.21.15.90:0/1852155143 wait complete. 2024-04-02T14:34:40.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:40.359+0000 7f0dcd000640 1 Processor -- start 2024-04-02T14:34:40.362 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:34:39 smithi092 ceph-mon[36497]: pgmap v382: 97 pgs: 97 active+clean; 644 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-02T14:34:40.363 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:34:39 smithi092 ceph-mon[36497]: from='client.? 172.21.15.90:0/3373745469' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-02T14:34:40.363 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:34:39 smithi092 ceph-mon[36497]: from='client.? 172.21.15.90:0/3043197937' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-02T14:34:40.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:40.359+0000 7f0dcd000640 1 -- start start 2024-04-02T14:34:40.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:40.360+0000 7f0dcd000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0dc8101970 0x7f0dc80758b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:40.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:40.360+0000 7f0dcd000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0dc8102b60 0x7f0dc8073ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:40.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:40.360+0000 7f0dcd000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0dc8103520 0x7f0dc8074430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:40.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:40.360+0000 7f0dcd000640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f0dc810d5c0 con 0x7f0dc8103520 2024-04-02T14:34:40.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:40.360+0000 7f0dcd000640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f0dc810d440 con 0x7f0dc8102b60 2024-04-02T14:34:40.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:40.360+0000 7f0dcd000640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f0dc810d740 con 0x7f0dc8101970 2024-04-02T14:34:40.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:40.360+0000 7f0dc7e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0dc8103520 0x7f0dc8074430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:40.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:40.360+0000 7f0dc7e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0dc8103520 0x7f0dc8074430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:36442/0 (socket says 172.21.15.90:36442) 2024-04-02T14:34:40.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:40.360+0000 7f0dc7e00640 1 -- 172.21.15.90:0/151380853 learned_addr learned my addr 172.21.15.90:0/151380853 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:34:40.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:40.360+0000 7f0dc6a00640 1 --2- 172.21.15.90:0/151380853 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0dc8102b60 0x7f0dc8073ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:40.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:40.360+0000 7f0dc7400640 1 --2- 172.21.15.90:0/151380853 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0dc8101970 0x7f0dc80758b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:40.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:40.360+0000 7f0dc7e00640 1 -- 172.21.15.90:0/151380853 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0dc8101970 msgr2=0x7f0dc80758b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:40.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:40.360+0000 7f0dc7e00640 1 --2- 172.21.15.90:0/151380853 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0dc8101970 0x7f0dc80758b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:40.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:40.360+0000 7f0dc7e00640 1 -- 172.21.15.90:0/151380853 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0dc8102b60 msgr2=0x7f0dc8073ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:40.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:40.360+0000 7f0dc7e00640 1 --2- 172.21.15.90:0/151380853 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0dc8102b60 0x7f0dc8073ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:40.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:40.360+0000 7f0dc7e00640 1 -- 172.21.15.90:0/151380853 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0dc8074cc0 con 0x7f0dc8103520 2024-04-02T14:34:40.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:40.360+0000 7f0dc7400640 1 --2- 172.21.15.90:0/151380853 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0dc8101970 0x7f0dc80758b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:34:40.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:40.361+0000 7f0dc7e00640 1 --2- 172.21.15.90:0/151380853 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0dc8103520 0x7f0dc8074430 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f0dc000eb10 tx=0x7f0dc000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:34:40.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:40.361+0000 7f0dbfe00640 1 -- 172.21.15.90:0/151380853 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dc0004b20 con 0x7f0dc8103520 2024-04-02T14:34:40.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:40.361+0000 7f0dbfe00640 1 -- 172.21.15.90:0/151380853 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0dc0004cc0 con 0x7f0dc8103520 2024-04-02T14:34:40.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:40.361+0000 7f0dbfe00640 1 -- 172.21.15.90:0/151380853 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dc0010760 con 0x7f0dc8103520 2024-04-02T14:34:40.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:40.361+0000 7f0dcd000640 1 -- 172.21.15.90:0/151380853 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0dc81a1e20 con 0x7f0dc8103520 2024-04-02T14:34:40.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:40.361+0000 7f0dcd000640 1 -- 172.21.15.90:0/151380853 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0dc81a2230 con 0x7f0dc8103520 2024-04-02T14:34:40.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:40.362+0000 7f0dcd000640 1 -- 172.21.15.90:0/151380853 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0dc8108c90 con 0x7f0dc8103520 2024-04-02T14:34:40.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:40.365+0000 7f0dbfe00640 1 -- 172.21.15.90:0/151380853 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f0dc0005020 con 0x7f0dc8103520 2024-04-02T14:34:40.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:40.367+0000 7f0dbfe00640 1 --2- 172.21.15.90:0/151380853 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f0dac0a3bb0 0x7f0dac0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:40.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:40.367+0000 7f0dbfe00640 1 -- 172.21.15.90:0/151380853 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8248+0+0 (secure 0 0 0) 0x7f0dc00d0a10 con 0x7f0dc8103520 2024-04-02T14:34:40.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:40.367+0000 7f0dc7400640 1 --2- 172.21.15.90:0/151380853 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f0dac0a3bb0 0x7f0dac0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:40.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:40.368+0000 7f0dc7400640 1 --2- 172.21.15.90:0/151380853 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f0dac0a3bb0 0x7f0dac0a6070 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f0db802c7d0 tx=0x7f0db8005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:34:40.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:40.371+0000 7f0dbfe00640 1 -- 172.21.15.90:0/151380853 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7f0dc009d380 con 0x7f0dc8103520 2024-04-02T14:34:40.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:40.606+0000 7f0dcd000640 1 -- 172.21.15.90:0/151380853 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0dc802d370 con 0x7f0dc8103520 2024-04-02T14:34:40.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:40.607+0000 7f0dbfe00640 1 -- 172.21.15.90:0/151380853 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11927 (secure 0 0 0) 0x7f0dc009cc80 con 0x7f0dc8103520 2024-04-02T14:34:40.610 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-02T14:34:40.610 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14412,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6837/2852160309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6836","nonce":2852160309},{"type":"v1","addr":"172.21.15.90:6837","nonce":2852160309}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14478,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6841/1919376858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6840","nonce":1919376858},{"type":"v1","addr":"172.21.15.90:6841","nonce":1919376858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":26,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-02T14:32:22.319083+0000","modified":"2024-04-02T14:32:29.812071+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24590,"mds_1":24560,"mds_2":24547,"mds_3":24539,"mds_4":24523},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24454":{"gid":24454,"name":"l","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6833/1564319431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":1564319431},{"type":"v1","addr":"172.21.15.177:6833","nonce":1564319431}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.92:6833/1570337372","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":1570337372},{"type":"v1","addr":"172.21.15.92:6833","nonce":1570337372}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"j","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.90:6835/1277124891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6834","nonce":1277124891},{"type":"v1","addr":"172.21.15.90:6835","nonce":1277124891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.92:6835/860721164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":860721164},{"type":"v1","addr":"172.21.15.92:6835","nonce":860721164}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24511":{"gid":24511,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6835/3177244988","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":3177244988},{"type":"v1","addr":"172.21.15.177:6835","nonce":3177244988}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"c","rank":4,"incarnation":25,"state":"up:active","state_seq":9,"addr":"172.21.15.177:6837/4223414858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":4223414858},{"type":"v1","addr":"172.21.15.177:6837","nonce":4223414858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24539":{"gid":24539,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":8,"addr":"172.21.15.92:6837/1270635939","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":1270635939},{"type":"v1","addr":"172.21.15.92:6837","nonce":1270635939}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24547":{"gid":24547,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":7,"addr":"172.21.15.177:6839/664895663","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":664895663},{"type":"v1","addr":"172.21.15.177:6839","nonce":664895663}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"g","rank":1,"incarnation":19,"state":"up:active","state_seq":5,"addr":"172.21.15.90:6839/1498915870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6838","nonce":1498915870},{"type":"v1","addr":"172.21.15.90:6839","nonce":1498915870}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24590":{"gid":24590,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.92:6839/1054888219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":1054888219},{"type":"v1","addr":"172.21.15.92:6839","nonce":1054888219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24590,"qdb_cluster":[24539,24560,24523,24590,24547]},"id":1}]} 2024-04-02T14:34:40.610 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 26 2024-04-02T14:34:40.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:40.610+0000 7f0dcd000640 1 -- 172.21.15.90:0/151380853 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f0dac0a3bb0 msgr2=0x7f0dac0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:40.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:40.610+0000 7f0dcd000640 1 --2- 172.21.15.90:0/151380853 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f0dac0a3bb0 0x7f0dac0a6070 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f0db802c7d0 tx=0x7f0db8005fd0 comp rx=0 tx=0).stop 2024-04-02T14:34:40.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:40.610+0000 7f0dcd000640 1 -- 172.21.15.90:0/151380853 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0dc8103520 msgr2=0x7f0dc8074430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:40.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:40.610+0000 7f0dcd000640 1 --2- 172.21.15.90:0/151380853 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0dc8103520 0x7f0dc8074430 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f0dc000eb10 tx=0x7f0dc000c2d0 comp rx=0 tx=0).stop 2024-04-02T14:34:40.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:40.610+0000 7f0dcd000640 1 -- 172.21.15.90:0/151380853 shutdown_connections 2024-04-02T14:34:40.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:40.610+0000 7f0dcd000640 1 --2- 172.21.15.90:0/151380853 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f0dac0a3bb0 0x7f0dac0a6070 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:40.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:40.610+0000 7f0dcd000640 1 --2- 172.21.15.90:0/151380853 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0dc8103520 0x7f0dc8074430 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:40.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:40.610+0000 7f0dcd000640 1 --2- 172.21.15.90:0/151380853 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0dc8102b60 0x7f0dc8073ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:40.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:40.610+0000 7f0dcd000640 1 --2- 172.21.15.90:0/151380853 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0dc8101970 0x7f0dc80758b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:40.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:40.610+0000 7f0dcd000640 1 -- 172.21.15.90:0/151380853 >> 172.21.15.90:0/151380853 conn(0x7f0dc80fcee0 msgr2=0x7f0dc8105e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:34:40.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:40.610+0000 7f0dcd000640 1 -- 172.21.15.90:0/151380853 shutdown_connections 2024-04-02T14:34:40.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:40.610+0000 7f0dcd000640 1 -- 172.21.15.90:0/151380853 wait complete. 2024-04-02T14:34:40.629 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T14:34:40.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:40.821+0000 7f0850e00640 1 -- 172.21.15.90:0/85874766 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f084c100d10 msgr2=0x7f084c101170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:40.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:40.821+0000 7f0850e00640 1 --2- 172.21.15.90:0/85874766 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f084c100d10 0x7f084c101170 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f0838009860 tx=0x7f083802bb70 comp rx=0 tx=0).stop 2024-04-02T14:34:40.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:40.821+0000 7f0850e00640 1 -- 172.21.15.90:0/85874766 shutdown_connections 2024-04-02T14:34:40.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:40.821+0000 7f0850e00640 1 --2- 172.21.15.90:0/85874766 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f084c1016b0 0x7f084c101b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:40.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:40.821+0000 7f0850e00640 1 --2- 172.21.15.90:0/85874766 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f084c100d10 0x7f084c101170 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:40.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:40.821+0000 7f0850e00640 1 --2- 172.21.15.90:0/85874766 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f084c1003b0 0x7f084c1007d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:40.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:40.821+0000 7f0850e00640 1 -- 172.21.15.90:0/85874766 >> 172.21.15.90:0/85874766 conn(0x7f084c0fbfb0 msgr2=0x7f084c0fe3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:34:40.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:40.821+0000 7f0850e00640 1 -- 172.21.15.90:0/85874766 shutdown_connections 2024-04-02T14:34:40.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:40.821+0000 7f0850e00640 1 -- 172.21.15.90:0/85874766 wait complete. 2024-04-02T14:34:40.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:40.822+0000 7f0850e00640 1 Processor -- start 2024-04-02T14:34:40.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:40.822+0000 7f0850e00640 1 -- start start 2024-04-02T14:34:40.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:40.822+0000 7f0850e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f084c1003b0 0x7f084c19a020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:40.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:40.822+0000 7f0850e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f084c100d10 0x7f084c19a560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:40.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:40.822+0000 7f0850e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f084c1016b0 0x7f084c1a15d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:40.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:40.822+0000 7f0850e00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f084c10f9b0 con 0x7f084c1016b0 2024-04-02T14:34:40.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:40.822+0000 7f0850e00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f084c10f830 con 0x7f084c1003b0 2024-04-02T14:34:40.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:40.822+0000 7f0850e00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f084c10fb30 con 0x7f084c100d10 2024-04-02T14:34:40.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:40.822+0000 7f084a800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f084c1016b0 0x7f084c1a15d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:40.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:40.822+0000 7f084a800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f084c1016b0 0x7f084c1a15d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:36466/0 (socket says 172.21.15.90:36466) 2024-04-02T14:34:40.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:40.822+0000 7f084a800640 1 -- 172.21.15.90:0/3073919238 learned_addr learned my addr 172.21.15.90:0/3073919238 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:34:40.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:40.822+0000 7f0849e00640 1 --2- 172.21.15.90:0/3073919238 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f084c1003b0 0x7f084c19a020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:40.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:40.822+0000 7f0849400640 1 --2- 172.21.15.90:0/3073919238 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f084c100d10 0x7f084c19a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:40.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:40.823+0000 7f084a800640 1 -- 172.21.15.90:0/3073919238 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f084c100d10 msgr2=0x7f084c19a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:40.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:40.823+0000 7f084a800640 1 --2- 172.21.15.90:0/3073919238 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f084c100d10 0x7f084c19a560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:40.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:40.823+0000 7f084a800640 1 -- 172.21.15.90:0/3073919238 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f084c1003b0 msgr2=0x7f084c19a020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:40.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:40.823+0000 7f084a800640 1 --2- 172.21.15.90:0/3073919238 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f084c1003b0 0x7f084c19a020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:40.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:40.823+0000 7f084a800640 1 -- 172.21.15.90:0/3073919238 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f084c1a1c40 con 0x7f084c1016b0 2024-04-02T14:34:40.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:40.823+0000 7f084a800640 1 --2- 172.21.15.90:0/3073919238 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f084c1016b0 0x7f084c1a15d0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f084400eb10 tx=0x7f084400ef10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:34:40.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:40.823+0000 7f0842a00640 1 -- 172.21.15.90:0/3073919238 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f084400ce20 con 0x7f084c1016b0 2024-04-02T14:34:40.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:40.823+0000 7f0850e00640 1 -- 172.21.15.90:0/3073919238 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f084c1a1f30 con 0x7f084c1016b0 2024-04-02T14:34:40.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:40.823+0000 7f0842a00640 1 -- 172.21.15.90:0/3073919238 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0844005020 con 0x7f084c1016b0 2024-04-02T14:34:40.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:40.823+0000 7f0842a00640 1 -- 172.21.15.90:0/3073919238 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0844010690 con 0x7f084c1016b0 2024-04-02T14:34:40.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:40.823+0000 7f0850e00640 1 -- 172.21.15.90:0/3073919238 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f084c1a2440 con 0x7f084c1016b0 2024-04-02T14:34:40.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:40.825+0000 7f0850e00640 1 -- 172.21.15.90:0/3073919238 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f084c10b2a0 con 0x7f084c1016b0 2024-04-02T14:34:40.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:40.827+0000 7f0842a00640 1 -- 172.21.15.90:0/3073919238 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f0844005540 con 0x7f084c1016b0 2024-04-02T14:34:40.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:40.828+0000 7f0842a00640 1 --2- 172.21.15.90:0/3073919238 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f08300a3bb0 0x7f08300a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:40.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:40.828+0000 7f0842a00640 1 -- 172.21.15.90:0/3073919238 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8248+0+0 (secure 0 0 0) 0x7f08440d0ab0 con 0x7f084c1016b0 2024-04-02T14:34:40.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:40.828+0000 7f0849e00640 1 --2- 172.21.15.90:0/3073919238 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f08300a3bb0 0x7f08300a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:40.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:40.829+0000 7f0849e00640 1 --2- 172.21.15.90:0/3073919238 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f08300a3bb0 0x7f08300a6070 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f083c005fd0 tx=0x7f083c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:34:40.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:40.832+0000 7f0842a00640 1 -- 172.21.15.90:0/3073919238 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7f084409d480 con 0x7f084c1016b0 2024-04-02T14:34:41.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.037+0000 7f0850e00640 1 -- 172.21.15.90:0/3073919238 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f084c062700 con 0x7f084c1016b0 2024-04-02T14:34:41.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.039+0000 7f0842a00640 1 -- 172.21.15.90:0/3073919238 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+22646 (secure 0 0 0) 0x7f084409cd80 con 0x7f084c1016b0 2024-04-02T14:34:41.042 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-02T14:34:41.042 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":90,"fsid":"6449448a-f0fc-11ee-b647-cb9ed24678a4","created":"2024-04-02T14:23:16.530201+0000","modified":"2024-04-02T14:32:35.716929+0000","last_up_change":"2024-04-02T14:31:23.819732+0000","last_in_change":"2024-04-02T14:31:02.824284+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T14:27:16.758622+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-02T14:32:20.016331+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-02T14:32:21.294367+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a1a6a856-ab44-44bd-bc5e-45c718a70cd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":3635087609},{"type":"v1","addr":"172.21.15.90:6803","nonce":3635087609}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":3635087609},{"type":"v1","addr":"172.21.15.90:6805","nonce":3635087609}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":3635087609},{"type":"v1","addr":"172.21.15.90:6809","nonce":3635087609}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":3635087609},{"type":"v1","addr":"172.21.15.90:6807","nonce":3635087609}]},"public_addr":"172.21.15.90:6803/3635087609","cluster_addr":"172.21.15.90:6805/3635087609","heartbeat_back_addr":"172.21.15.90:6809/3635087609","heartbeat_front_addr":"172.21.15.90:6807/3635087609","state":["exists","up"]},{"osd":1,"uuid":"124139cd-5793-463e-9607-9d9c10164183","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":1994536059},{"type":"v1","addr":"172.21.15.92:6801","nonce":1994536059}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":1994536059},{"type":"v1","addr":"172.21.15.92:6803","nonce":1994536059}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":1994536059},{"type":"v1","addr":"172.21.15.92:6807","nonce":1994536059}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":1994536059},{"type":"v1","addr":"172.21.15.92:6805","nonce":1994536059}]},"public_addr":"172.21.15.92:6801/1994536059","cluster_addr":"172.21.15.92:6803/1994536059","heartbeat_back_addr":"172.21.15.92:6807/1994536059","heartbeat_front_addr":"172.21.15.92:6805/1994536059","state":["exists","up"]},{"osd":2,"uuid":"a61fbef2-9dca-4875-a814-020fff0db0e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":1358472800},{"type":"v1","addr":"172.21.15.177:6801","nonce":1358472800}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":1358472800},{"type":"v1","addr":"172.21.15.177:6803","nonce":1358472800}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":1358472800},{"type":"v1","addr":"172.21.15.177:6807","nonce":1358472800}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":1358472800},{"type":"v1","addr":"172.21.15.177:6805","nonce":1358472800}]},"public_addr":"172.21.15.177:6801/1358472800","cluster_addr":"172.21.15.177:6803/1358472800","heartbeat_back_addr":"172.21.15.177:6807/1358472800","heartbeat_front_addr":"172.21.15.177:6805/1358472800","state":["exists","up"]},{"osd":3,"uuid":"b175a2f4-7859-4eb0-bc1f-fba8c2b92a2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":3114811216},{"type":"v1","addr":"172.21.15.90:6811","nonce":3114811216}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":3114811216},{"type":"v1","addr":"172.21.15.90:6813","nonce":3114811216}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":3114811216},{"type":"v1","addr":"172.21.15.90:6817","nonce":3114811216}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":3114811216},{"type":"v1","addr":"172.21.15.90:6815","nonce":3114811216}]},"public_addr":"172.21.15.90:6811/3114811216","cluster_addr":"172.21.15.90:6813/3114811216","heartbeat_back_addr":"172.21.15.90:6817/3114811216","heartbeat_front_addr":"172.21.15.90:6815/3114811216","state":["exists","up"]},{"osd":4,"uuid":"38cc68ea-c109-4d8d-b77d-1c6f9b56c875","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":2987208879},{"type":"v1","addr":"172.21.15.92:6809","nonce":2987208879}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6810","nonce":2987208879},{"type":"v1","addr":"172.21.15.92:6811","nonce":2987208879}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6814","nonce":2987208879},{"type":"v1","addr":"172.21.15.92:6815","nonce":2987208879}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":2987208879},{"type":"v1","addr":"172.21.15.92:6813","nonce":2987208879}]},"public_addr":"172.21.15.92:6809/2987208879","cluster_addr":"172.21.15.92:6811/2987208879","heartbeat_back_addr":"172.21.15.92:6815/2987208879","heartbeat_front_addr":"172.21.15.92:6813/2987208879","state":["exists","up"]},{"osd":5,"uuid":"2347232e-ac91-466f-8ac9-496f397679ba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":3486820237},{"type":"v1","addr":"172.21.15.177:6809","nonce":3486820237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":3486820237},{"type":"v1","addr":"172.21.15.177:6811","nonce":3486820237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":3486820237},{"type":"v1","addr":"172.21.15.177:6815","nonce":3486820237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":3486820237},{"type":"v1","addr":"172.21.15.177:6813","nonce":3486820237}]},"public_addr":"172.21.15.177:6809/3486820237","cluster_addr":"172.21.15.177:6811/3486820237","heartbeat_back_addr":"172.21.15.177:6815/3486820237","heartbeat_front_addr":"172.21.15.177:6813/3486820237","state":["exists","up"]},{"osd":6,"uuid":"aa72392f-0498-44db-a8d2-eccc89a64871","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":2579716443},{"type":"v1","addr":"172.21.15.90:6819","nonce":2579716443}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6820","nonce":2579716443},{"type":"v1","addr":"172.21.15.90:6821","nonce":2579716443}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":2579716443},{"type":"v1","addr":"172.21.15.90:6825","nonce":2579716443}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2579716443},{"type":"v1","addr":"172.21.15.90:6823","nonce":2579716443}]},"public_addr":"172.21.15.90:6819/2579716443","cluster_addr":"172.21.15.90:6821/2579716443","heartbeat_back_addr":"172.21.15.90:6825/2579716443","heartbeat_front_addr":"172.21.15.90:6823/2579716443","state":["exists","up"]},{"osd":7,"uuid":"931bd105-bc14-4a1e-bfe4-c7ad58fa4b73","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":3371649139},{"type":"v1","addr":"172.21.15.92:6817","nonce":3371649139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":3371649139},{"type":"v1","addr":"172.21.15.92:6819","nonce":3371649139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":3371649139},{"type":"v1","addr":"172.21.15.92:6823","nonce":3371649139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6820","nonce":3371649139},{"type":"v1","addr":"172.21.15.92:6821","nonce":3371649139}]},"public_addr":"172.21.15.92:6817/3371649139","cluster_addr":"172.21.15.92:6819/3371649139","heartbeat_back_addr":"172.21.15.92:6823/3371649139","heartbeat_front_addr":"172.21.15.92:6821/3371649139","state":["exists","up"]},{"osd":8,"uuid":"82624ec2-b382-4886-9f40-7f5f376de3ce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":1459053034},{"type":"v1","addr":"172.21.15.177:6817","nonce":1459053034}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":1459053034},{"type":"v1","addr":"172.21.15.177:6819","nonce":1459053034}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":1459053034},{"type":"v1","addr":"172.21.15.177:6823","nonce":1459053034}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":1459053034},{"type":"v1","addr":"172.21.15.177:6821","nonce":1459053034}]},"public_addr":"172.21.15.177:6817/1459053034","cluster_addr":"172.21.15.177:6819/1459053034","heartbeat_back_addr":"172.21.15.177:6823/1459053034","heartbeat_front_addr":"172.21.15.177:6821/1459053034","state":["exists","up"]},{"osd":9,"uuid":"fbe36e48-3468-42b2-8094-e16552f3cb16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":2592604332},{"type":"v1","addr":"172.21.15.90:6827","nonce":2592604332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":2592604332},{"type":"v1","addr":"172.21.15.90:6829","nonce":2592604332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6832","nonce":2592604332},{"type":"v1","addr":"172.21.15.90:6833","nonce":2592604332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":2592604332},{"type":"v1","addr":"172.21.15.90:6831","nonce":2592604332}]},"public_addr":"172.21.15.90:6827/2592604332","cluster_addr":"172.21.15.90:6829/2592604332","heartbeat_back_addr":"172.21.15.90:6833/2592604332","heartbeat_front_addr":"172.21.15.90:6831/2592604332","state":["exists","up"]},{"osd":10,"uuid":"9240df75-e7e2-47f1-8aae-534b70b3c210","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":1594153142},{"type":"v1","addr":"172.21.15.92:6825","nonce":1594153142}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6826","nonce":1594153142},{"type":"v1","addr":"172.21.15.92:6827","nonce":1594153142}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":1594153142},{"type":"v1","addr":"172.21.15.92:6831","nonce":1594153142}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":1594153142},{"type":"v1","addr":"172.21.15.92:6829","nonce":1594153142}]},"public_addr":"172.21.15.92:6825/1594153142","cluster_addr":"172.21.15.92:6827/1594153142","heartbeat_back_addr":"172.21.15.92:6831/1594153142","heartbeat_front_addr":"172.21.15.92:6829/1594153142","state":["exists","up"]},{"osd":11,"uuid":"83cba9fa-b745-4df1-adda-e86837d4b91a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":83,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1622582126},{"type":"v1","addr":"172.21.15.177:6825","nonce":1622582126}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":1622582126},{"type":"v1","addr":"172.21.15.177:6827","nonce":1622582126}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":1622582126},{"type":"v1","addr":"172.21.15.177:6831","nonce":1622582126}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1622582126},{"type":"v1","addr":"172.21.15.177:6829","nonce":1622582126}]},"public_addr":"172.21.15.177:6825/1622582126","cluster_addr":"172.21.15.177:6827/1622582126","heartbeat_back_addr":"172.21.15.177:6831/1622582126","heartbeat_front_addr":"172.21.15.177:6829/1622582126","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:26:24.290365+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:26:49.405988+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:27:13.639196+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:27:45.603746+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:28:11.019126+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:28:35.084163+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:29:08.064626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:29:33.266133+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:29:57.381816+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:30:30.286847+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:30:56.226279+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:31:20.809611+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":7,"to":11}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":8,"to":4}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.11","mappings":[{"from":4,"to":8},{"from":5,"to":9}]},{"pgid":"3.21","mappings":[{"from":5,"to":9}]},{"pgid":"3.23","mappings":[{"from":4,"to":7}]},{"pgid":"3.3a","mappings":[{"from":4,"to":6}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.90:0/314641601":"2024-04-03T14:24:34.595561+0000","172.21.15.90:0/1400895868":"2024-04-03T14:24:34.595561+0000","172.21.15.90:6801/129436436":"2024-04-03T14:24:34.595561+0000","172.21.15.90:6801/2372398598":"2024-04-03T14:24:00.362147+0000","172.21.15.90:0/278569338":"2024-04-03T14:24:34.595561+0000","172.21.15.90:0/2321464594":"2024-04-03T14:24:00.362147+0000","172.21.15.90:6800/2372398598":"2024-04-03T14:24:00.362147+0000","172.21.15.90:0/2962656672":"2024-04-03T14:24:00.362147+0000","172.21.15.90:0/126693610":"2024-04-03T14:24:00.362147+0000","172.21.15.90:6800/129436436":"2024-04-03T14:24:34.595561+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T14:34:41.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.042+0000 7f0850e00640 1 -- 172.21.15.90:0/3073919238 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f08300a3bb0 msgr2=0x7f08300a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:41.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.042+0000 7f0850e00640 1 --2- 172.21.15.90:0/3073919238 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f08300a3bb0 0x7f08300a6070 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f083c005fd0 tx=0x7f083c009260 comp rx=0 tx=0).stop 2024-04-02T14:34:41.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.042+0000 7f0850e00640 1 -- 172.21.15.90:0/3073919238 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f084c1016b0 msgr2=0x7f084c1a15d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:41.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.042+0000 7f0850e00640 1 --2- 172.21.15.90:0/3073919238 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f084c1016b0 0x7f084c1a15d0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f084400eb10 tx=0x7f084400ef10 comp rx=0 tx=0).stop 2024-04-02T14:34:41.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.042+0000 7f0850e00640 1 -- 172.21.15.90:0/3073919238 shutdown_connections 2024-04-02T14:34:41.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.042+0000 7f0850e00640 1 --2- 172.21.15.90:0/3073919238 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f08300a3bb0 0x7f08300a6070 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:41.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.042+0000 7f0850e00640 1 --2- 172.21.15.90:0/3073919238 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f084c1016b0 0x7f084c1a15d0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:41.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.042+0000 7f0850e00640 1 --2- 172.21.15.90:0/3073919238 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f084c100d10 0x7f084c19a560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:41.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.042+0000 7f0850e00640 1 --2- 172.21.15.90:0/3073919238 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f084c1003b0 0x7f084c19a020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:41.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.042+0000 7f0850e00640 1 -- 172.21.15.90:0/3073919238 >> 172.21.15.90:0/3073919238 conn(0x7f084c0fbfb0 msgr2=0x7f084c0fd6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:34:41.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.042+0000 7f0850e00640 1 -- 172.21.15.90:0/3073919238 shutdown_connections 2024-04-02T14:34:41.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.042+0000 7f0850e00640 1 -- 172.21.15.90:0/3073919238 wait complete. 2024-04-02T14:34:41.062 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-02T14:32:22.319083+0000', 'modified': '2024-04-02T14:32:29.812071+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 24590, 'mds_1': 24560, 'mds_2': 24547, 'mds_3': 24539, 'mds_4': 24523}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24454': {'gid': 24454, 'name': 'l', 'rank': 4, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.177:6833/1564319431', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6832', 'nonce': 1564319431}, {'type': 'v1', 'addr': '172.21.15.177:6833', 'nonce': 1564319431}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24461': {'gid': 24461, 'name': 'k', 'rank': 3, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.92:6833/1570337372', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6832', 'nonce': 1570337372}, {'type': 'v1', 'addr': '172.21.15.92:6833', 'nonce': 1570337372}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24478': {'gid': 24478, 'name': 'j', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.90:6835/1277124891', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6834', 'nonce': 1277124891}, {'type': 'v1', 'addr': '172.21.15.90:6835', 'nonce': 1277124891}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24503': {'gid': 24503, 'name': 'e', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.92:6835/860721164', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6834', 'nonce': 860721164}, {'type': 'v1', 'addr': '172.21.15.92:6835', 'nonce': 860721164}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24511': {'gid': 24511, 'name': 'f', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.177:6835/3177244988', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6834', 'nonce': 3177244988}, {'type': 'v1', 'addr': '172.21.15.177:6835', 'nonce': 3177244988}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24523': {'gid': 24523, 'name': 'c', 'rank': 4, 'incarnation': 25, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.177:6837/4223414858', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6836', 'nonce': 4223414858}, {'type': 'v1', 'addr': '172.21.15.177:6837', 'nonce': 4223414858}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24539': {'gid': 24539, 'name': 'h', 'rank': 3, 'incarnation': 23, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.92:6837/1270635939', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6836', 'nonce': 1270635939}, {'type': 'v1', 'addr': '172.21.15.92:6837', 'nonce': 1270635939}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24547': {'gid': 24547, 'name': 'i', 'rank': 2, 'incarnation': 21, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.177:6839/664895663', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6838', 'nonce': 664895663}, {'type': 'v1', 'addr': '172.21.15.177:6839', 'nonce': 664895663}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24560': {'gid': 24560, 'name': 'g', 'rank': 1, 'incarnation': 19, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.90:6839/1498915870', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6838', 'nonce': 1498915870}, {'type': 'v1', 'addr': '172.21.15.90:6839', 'nonce': 1498915870}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24590': {'gid': 24590, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.92:6839/1054888219', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.92:6838', 'nonce': 1054888219}, {'type': 'v1', 'addr': '172.21.15.92:6839', 'nonce': 1054888219}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24590, 'qdb_cluster': [24539, 24560, 24523, 24590, 24547]} 2024-04-02T14:34:41.063 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 5/5 2024-04-02T14:34:41.063 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-04-02T14:34:41.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:34:40 smithi177 ceph-mon[36431]: from='client.? 172.21.15.90:0/3315420596' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-02T14:34:41.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:34:40 smithi177 ceph-mon[36431]: from='client.? 172.21.15.90:0/151380853' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-02T14:34:41.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:34:40 smithi090 ceph-mon[31674]: from='client.? 172.21.15.90:0/3315420596' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-02T14:34:41.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:34:40 smithi090 ceph-mon[31674]: from='client.? 172.21.15.90:0/151380853' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-02T14:34:41.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.267+0000 7fc968a00640 1 -- 172.21.15.90:0/3013894244 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc964108020 msgr2=0x7fc964108440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:41.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.267+0000 7fc968a00640 1 --2- 172.21.15.90:0/3013894244 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc964108020 0x7fc964108440 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fc954009900 tx=0x7fc95402bc50 comp rx=0 tx=0).stop 2024-04-02T14:34:41.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.268+0000 7fc968a00640 1 -- 172.21.15.90:0/3013894244 shutdown_connections 2024-04-02T14:34:41.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.268+0000 7fc968a00640 1 --2- 172.21.15.90:0/3013894244 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc964109bd0 0x7fc964110460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:41.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.268+0000 7fc968a00640 1 --2- 172.21.15.90:0/3013894244 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc964109210 0x7fc964109690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:41.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.268+0000 7fc968a00640 1 --2- 172.21.15.90:0/3013894244 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc964108020 0x7fc964108440 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:41.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.268+0000 7fc968a00640 1 -- 172.21.15.90:0/3013894244 >> 172.21.15.90:0/3013894244 conn(0x7fc964076fd0 msgr2=0x7fc9640793f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:34:41.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.268+0000 7fc968a00640 1 -- 172.21.15.90:0/3013894244 shutdown_connections 2024-04-02T14:34:41.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.268+0000 7fc968a00640 1 -- 172.21.15.90:0/3013894244 wait complete. 2024-04-02T14:34:41.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.269+0000 7fc968a00640 1 Processor -- start 2024-04-02T14:34:41.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.269+0000 7fc968a00640 1 -- start start 2024-04-02T14:34:41.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.269+0000 7fc968a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc964108020 0x7fc96419e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:41.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.269+0000 7fc968a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc964109210 0x7fc96419e910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:41.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.269+0000 7fc968a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc964109bd0 0x7fc9641a5980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:41.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.269+0000 7fc968a00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fc964113cb0 con 0x7fc964109210 2024-04-02T14:34:41.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.269+0000 7fc968a00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fc964113b30 con 0x7fc964109bd0 2024-04-02T14:34:41.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.269+0000 7fc968a00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fc964113e30 con 0x7fc964108020 2024-04-02T14:34:41.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.270+0000 7fc95aa00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc964109210 0x7fc96419e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:41.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.270+0000 7fc95aa00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc964109210 0x7fc96419e910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:36484/0 (socket says 172.21.15.90:36484) 2024-04-02T14:34:41.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.270+0000 7fc95aa00640 1 -- 172.21.15.90:0/2217405984 learned_addr learned my addr 172.21.15.90:0/2217405984 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:34:41.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.270+0000 7fc963e00640 1 --2- 172.21.15.90:0/2217405984 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc964109bd0 0x7fc9641a5980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:41.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.270+0000 7fc963400640 1 --2- 172.21.15.90:0/2217405984 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc964108020 0x7fc96419e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:41.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.270+0000 7fc963400640 1 -- 172.21.15.90:0/2217405984 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc964109bd0 msgr2=0x7fc9641a5980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:41.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.270+0000 7fc963400640 1 --2- 172.21.15.90:0/2217405984 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc964109bd0 0x7fc9641a5980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:41.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.270+0000 7fc963400640 1 -- 172.21.15.90:0/2217405984 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc964109210 msgr2=0x7fc96419e910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:41.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.270+0000 7fc963400640 1 --2- 172.21.15.90:0/2217405984 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc964109210 0x7fc96419e910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:41.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.270+0000 7fc963400640 1 -- 172.21.15.90:0/2217405984 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9641a5ff0 con 0x7fc964108020 2024-04-02T14:34:41.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.270+0000 7fc963e00640 1 --2- 172.21.15.90:0/2217405984 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc964109bd0 0x7fc9641a5980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:34:41.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.270+0000 7fc95aa00640 1 --2- 172.21.15.90:0/2217405984 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc964109210 0x7fc96419e910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:34:41.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.270+0000 7fc963400640 1 --2- 172.21.15.90:0/2217405984 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc964108020 0x7fc96419e3d0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fc95402c190 tx=0x7fc95402c6e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:34:41.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.271+0000 7fc960c00640 1 -- 172.21.15.90:0/2217405984 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc954042020 con 0x7fc964108020 2024-04-02T14:34:41.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.271+0000 7fc960c00640 1 -- 172.21.15.90:0/2217405984 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc954004450 con 0x7fc964108020 2024-04-02T14:34:41.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.271+0000 7fc968a00640 1 -- 172.21.15.90:0/2217405984 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9641a6280 con 0x7fc964108020 2024-04-02T14:34:41.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.271+0000 7fc960c00640 1 -- 172.21.15.90:0/2217405984 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc95403e460 con 0x7fc964108020 2024-04-02T14:34:41.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.271+0000 7fc968a00640 1 -- 172.21.15.90:0/2217405984 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9641a67a0 con 0x7fc964108020 2024-04-02T14:34:41.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.274+0000 7fc96cd6f740 1 Processor -- start 2024-04-02T14:34:41.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.274+0000 7fc96cd6f740 1 -- start start 2024-04-02T14:34:41.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.274+0000 7fc96cd6f740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ca3fa78b10 0x55ca3fa78f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:41.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.274+0000 7fc96cd6f740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ca3fa79500 0x55ca3fa7e000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:41.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.274+0000 7fc960c00640 1 -- 172.21.15.90:0/2217405984 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fc954045020 con 0x7fc964108020 2024-04-02T14:34:41.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.274+0000 7fc96cd6f740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ca3fa7e670 0x55ca3fa80b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:41.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.274+0000 7fc96cd6f740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55ca3f9de060 con 0x55ca3fa7e670 2024-04-02T14:34:41.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.274+0000 7fc96cd6f740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55ca3f986560 con 0x55ca3fa79500 2024-04-02T14:34:41.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.274+0000 7fc96cd6f740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55ca3f9da010 con 0x55ca3fa78b10 2024-04-02T14:34:41.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.274+0000 7fc963e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ca3fa78b10 0x55ca3fa78f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:41.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.274+0000 7fc95aa00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ca3fa7e670 0x55ca3fa80b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:41.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.274+0000 7fc963e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ca3fa78b10 0x55ca3fa78f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:52858/0 (socket says 172.21.15.90:52858) 2024-04-02T14:34:41.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.274+0000 7fc95aa00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ca3fa7e670 0x55ca3fa80b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:36488/0 (socket says 172.21.15.90:36488) 2024-04-02T14:34:41.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.274+0000 7fc963e00640 1 -- 172.21.15.90:0/620413778 learned_addr learned my addr 172.21.15.90:0/620413778 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:34:41.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.275+0000 7fc963400640 1 --2- 172.21.15.90:0/620413778 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ca3fa79500 0x55ca3fa7e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:41.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.275+0000 7fc963e00640 1 -- 172.21.15.90:0/620413778 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ca3fa79500 msgr2=0x55ca3fa7e000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:34:41.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.275+0000 7fc963e00640 1 --2- 172.21.15.90:0/620413778 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ca3fa79500 0x55ca3fa7e000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:41.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.275+0000 7fc963e00640 1 -- 172.21.15.90:0/620413778 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ca3fa7e670 msgr2=0x55ca3fa80b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:34:41.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.275+0000 7fc963e00640 1 --2- 172.21.15.90:0/620413778 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ca3fa7e670 0x55ca3fa80b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:41.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.275+0000 7fc963e00640 1 -- 172.21.15.90:0/620413778 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ca3f9a8fd0 con 0x55ca3fa78b10 2024-04-02T14:34:41.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.275+0000 7fc95aa00640 1 --2- 172.21.15.90:0/620413778 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ca3fa7e670 0x55ca3fa80b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:34:41.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.275+0000 7fc963400640 1 --2- 172.21.15.90:0/620413778 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ca3fa79500 0x55ca3fa7e000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:34:41.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.275+0000 7fc963e00640 1 --2- 172.21.15.90:0/620413778 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ca3fa78b10 0x55ca3fa78f30 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fc95c011810 tx=0x7fc95c011cd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=84b55a01297cdaf5 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:34:41.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.275+0000 7fc953e00640 1 -- 172.21.15.90:0/620413778 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc95c011310 con 0x55ca3fa78b10 2024-04-02T14:34:41.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.275+0000 7fc953e00640 1 -- 172.21.15.90:0/620413778 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc95c00f9a0 con 0x55ca3fa78b10 2024-04-02T14:34:41.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.275+0000 7fc953e00640 1 -- 172.21.15.90:0/620413778 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc95c005720 con 0x55ca3fa78b10 2024-04-02T14:34:41.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.276+0000 7fc960c00640 1 --2- 172.21.15.90:0/2217405984 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fc9480a3bb0 0x7fc9480a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:41.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.276+0000 7fc96cd6f740 1 -- 172.21.15.90:0/620413778 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ca3fa78b10 msgr2=0x55ca3fa78f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:41.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.276+0000 7fc96cd6f740 1 --2- 172.21.15.90:0/620413778 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ca3fa78b10 0x55ca3fa78f30 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fc95c011810 tx=0x7fc95c011cd0 comp rx=0 tx=0).stop 2024-04-02T14:34:41.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.276+0000 7fc963400640 1 --2- 172.21.15.90:0/2217405984 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fc9480a3bb0 0x7fc9480a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:41.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.276+0000 7fc960c00640 1 -- 172.21.15.90:0/2217405984 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8248+0+0 (secure 0 0 0) 0x7fc9540f2360 con 0x7fc964108020 2024-04-02T14:34:41.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.276+0000 7fc96cd6f740 1 -- 172.21.15.90:0/620413778 shutdown_connections 2024-04-02T14:34:41.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.276+0000 7fc96cd6f740 1 --2- 172.21.15.90:0/620413778 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ca3fa7e670 0x55ca3fa80b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:41.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.276+0000 7fc96cd6f740 1 --2- 172.21.15.90:0/620413778 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ca3fa79500 0x55ca3fa7e000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:41.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.276+0000 7fc96cd6f740 1 --2- 172.21.15.90:0/620413778 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ca3fa78b10 0x55ca3fa78f30 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:41.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.276+0000 7fc96cd6f740 1 -- 172.21.15.90:0/620413778 >> 172.21.15.90:0/620413778 conn(0x55ca3fa74080 msgr2=0x55ca3fa74480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:34:41.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.276+0000 7fc96cd6f740 1 -- 172.21.15.90:0/620413778 shutdown_connections 2024-04-02T14:34:41.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.276+0000 7fc96cd6f740 1 -- 172.21.15.90:0/620413778 wait complete. 2024-04-02T14:34:41.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.277+0000 7fc963400640 1 --2- 172.21.15.90:0/2217405984 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fc9480a3bb0 0x7fc9480a6070 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x55ca3fa79900 tx=0x7fc95402bc50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:34:41.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.277+0000 7fc96cd6f740 1 -- start start 2024-04-02T14:34:41.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.277+0000 7fc96cd6f740 1 Processor -- start 2024-04-02T14:34:41.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.277+0000 7fc96cd6f740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ca3fa78b10 0x55ca3fa83080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:41.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.277+0000 7fc96cd6f740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ca3fa79500 0x55ca3fa835c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:41.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.277+0000 7fc96cd6f740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ca3fa7e670 0x55ca3fa8a600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:41.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.277+0000 7fc96cd6f740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55ca3fa81620 con 0x55ca3fa78b10 2024-04-02T14:34:41.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.277+0000 7fc96cd6f740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55ca3f999920 con 0x55ca3fa79500 2024-04-02T14:34:41.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.277+0000 7fc96cd6f740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55ca3fa817a0 con 0x55ca3fa7e670 2024-04-02T14:34:41.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.277+0000 7fc96cd6f740 10 client.0 _ll_register_callbacks cb 0x55ca3f999310 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:34:41.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.277+0000 7fc963e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ca3fa78b10 0x55ca3fa83080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:41.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.277+0000 7fc963e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ca3fa78b10 0x55ca3fa83080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:36492/0 (socket says 172.21.15.90:36492) 2024-04-02T14:34:41.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.277+0000 7fc963e00640 1 -- 172.21.15.90:0/2922517865 learned_addr learned my addr 172.21.15.90:0/2922517865 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:34:41.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.277+0000 7fc95aa00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ca3fa79500 0x55ca3fa835c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:41.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.278+0000 7fc963e00640 1 --2- 172.21.15.90:0/2922517865 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ca3fa7e670 0x55ca3fa8a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:41.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.278+0000 7fc95aa00640 1 -- 172.21.15.90:0/2922517865 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ca3fa7e670 msgr2=0x55ca3fa8a600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:34:41.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.278+0000 7fc95aa00640 1 --2- 172.21.15.90:0/2922517865 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ca3fa7e670 0x55ca3fa8a600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:41.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.278+0000 7fc95aa00640 1 -- 172.21.15.90:0/2922517865 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ca3fa78b10 msgr2=0x55ca3fa83080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:34:41.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.278+0000 7fc95aa00640 1 --2- 172.21.15.90:0/2922517865 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ca3fa78b10 0x55ca3fa83080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:41.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.278+0000 7fc95aa00640 1 -- 172.21.15.90:0/2922517865 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ca3f6f22d0 con 0x55ca3fa79500 2024-04-02T14:34:41.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.278+0000 7fc963e00640 1 --2- 172.21.15.90:0/2922517865 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ca3fa78b10 0x55ca3fa83080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:34:41.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.278+0000 7fc95aa00640 1 --2- 172.21.15.90:0/2922517865 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ca3fa79500 0x55ca3fa835c0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fc96419f8e0 tx=0x7fc94c011c90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=22bf27275c4aed36 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:34:41.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.278+0000 7fc950c00640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:34:41.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.278+0000 7fc950c00640 1 -- 172.21.15.90:0/2922517865 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc94c010c60 con 0x55ca3fa79500 2024-04-02T14:34:41.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.278+0000 7fc950c00640 1 -- 172.21.15.90:0/2922517865 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8248+0+0 (secure 0 0 0) 0x7fc94c010e00 con 0x55ca3fa79500 2024-04-02T14:34:41.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.278+0000 7fc96cd6f740 1 -- 172.21.15.90:0/2922517865 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc95c00f9a0 con 0x55ca3fa79500 2024-04-02T14:34:41.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.279+0000 7fc950c00640 1 -- 172.21.15.90:0/2922517865 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fc94c0047a0 con 0x55ca3fa79500 2024-04-02T14:34:41.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.279+0000 7fc96cd6f740 10 client.25193 fetch_fsmap learned FSMap version 26 2024-04-02T14:34:41.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.279+0000 7fc96cd6f740 1 -- 172.21.15.90:0/2922517865 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55ca3f7bc6b0 con 0x55ca3fa79500 2024-04-02T14:34:41.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.279+0000 7fc950c00640 1 -- 172.21.15.90:0/2922517865 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc95c00f9a0 con 0x55ca3fa79500 2024-04-02T14:34:41.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.279+0000 7fc950c00640 1 -- 172.21.15.90:0/2922517865 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7fc94c00f640 con 0x55ca3fa79500 2024-04-02T14:34:41.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.279+0000 7fc96cd6f740 10 client.25193 fetch_fsmap finished waiting for FSMap version 26 2024-04-02T14:34:41.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.279+0000 7fc96cd6f740 10 client.25193 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-02T14:34:41.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.280+0000 7fc96cd6f740 20 client.25193 populate_metadata read hostname 'smithi090' 2024-04-02T14:34:41.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.280+0000 7fc96cd6f740 1 --2- 172.21.15.90:0/2922517865 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x55ca3fa8feb0 0x55ca3fa91320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:41.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.280+0000 7fc96cd6f740 4 client.25193 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:34:41.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.280+0000 7fc96cd6f740 1 -- 172.21.15.90:0/2922517865 --> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ca3f9eb5a0 con 0x55ca3fa8feb0 2024-04-02T14:34:41.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.280+0000 7fc963e00640 1 --2- 172.21.15.90:0/2922517865 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x55ca3fa8feb0 0x55ca3fa91320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:41.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.280+0000 7fc963e00640 1 --2- 172.21.15.90:0/2922517865 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x55ca3fa8feb0 0x55ca3fa91320 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=7e60cbd8904cc19b server_cookie=40bc5ebd9aefa447 in_seq=0 out_seq=0 2024-04-02T14:34:41.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.280+0000 7fc950c00640 10 client.25193 ms_handle_connect on v2:172.21.15.177:6836/4223414858 2024-04-02T14:34:41.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.281+0000 7fc950c00640 1 -- 172.21.15.90:0/2922517865 <== mds.4 v2:172.21.15.177:6836/4223414858 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ca3f9eb5a0 con 0x55ca3fa8feb0 2024-04-02T14:34:41.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.281+0000 7fc950c00640 10 client.25193 handle_command_reply: tid=0 2024-04-02T14:34:41.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.281+0000 7fc96cd6f740 1 client.25193 shutdown 2024-04-02T14:34:41.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.281+0000 7fc96cd6f740 1 -- 172.21.15.90:0/2922517865 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ca3fa79500 msgr2=0x55ca3fa835c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:41.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.281+0000 7fc96cd6f740 1 --2- 172.21.15.90:0/2922517865 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ca3fa79500 0x55ca3fa835c0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fc96419f8e0 tx=0x7fc94c011c90 comp rx=0 tx=0).stop 2024-04-02T14:34:41.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.281+0000 7fc96cd6f740 1 -- 172.21.15.90:0/2922517865 shutdown_connections 2024-04-02T14:34:41.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.281+0000 7fc96cd6f740 1 --2- 172.21.15.90:0/2922517865 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ca3fa7e670 0x55ca3fa8a600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:41.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.282+0000 7fc96cd6f740 1 --2- 172.21.15.90:0/2922517865 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ca3fa79500 0x55ca3fa835c0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:41.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.282+0000 7fc96cd6f740 1 --2- 172.21.15.90:0/2922517865 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x55ca3fa8feb0 0x55ca3fa91320 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:41.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.282+0000 7fc96cd6f740 1 --2- 172.21.15.90:0/2922517865 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ca3fa78b10 0x55ca3fa83080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:41.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.282+0000 7fc96cd6f740 1 -- 172.21.15.90:0/2922517865 >> 172.21.15.90:0/2922517865 conn(0x55ca3fa74080 msgr2=0x55ca3fa731c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:34:41.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.282+0000 7fc950c00640 0 client.25193 ms_handle_reset on v2:172.21.15.177:6836/4223414858 2024-04-02T14:34:41.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.282+0000 7fc96cd6f740 1 -- 172.21.15.90:0/2922517865 shutdown_connections 2024-04-02T14:34:41.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.282+0000 7fc96cd6f740 1 -- 172.21.15.90:0/2922517865 wait complete. 2024-04-02T14:34:41.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.282+0000 7fc96cd6f740 20 client.25193 trim_cache size 0 max 16384 2024-04-02T14:34:41.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.300+0000 7fc96cd6f740 1 Processor -- start 2024-04-02T14:34:41.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.301+0000 7fc96cd6f740 1 -- start start 2024-04-02T14:34:41.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.301+0000 7fc96cd6f740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ca3fa78b10 0x55ca3fa81500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:41.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.301+0000 7fc96cd6f740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ca3fa79500 0x55ca3fa8a5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:41.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.301+0000 7fc96cd6f740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ca3fa7e670 0x55ca3fa7aa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:41.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.301+0000 7fc96cd6f740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55ca3fa8f660 con 0x55ca3fa78b10 2024-04-02T14:34:41.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.301+0000 7fc96cd6f740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55ca3fa8fd30 con 0x55ca3fa7e670 2024-04-02T14:34:41.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.301+0000 7fc96cd6f740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55ca3fa90400 con 0x55ca3fa79500 2024-04-02T14:34:41.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.301+0000 7fc963e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ca3fa78b10 0x55ca3fa81500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:41.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.301+0000 7fc963e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ca3fa78b10 0x55ca3fa81500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:36504/0 (socket says 172.21.15.90:36504) 2024-04-02T14:34:41.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.301+0000 7fc963e00640 1 -- 172.21.15.90:0/1438783529 learned_addr learned my addr 172.21.15.90:0/1438783529 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:34:41.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.301+0000 7fc95aa00640 1 --2- 172.21.15.90:0/1438783529 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ca3fa79500 0x55ca3fa8a5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:41.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.302+0000 7fc963e00640 1 --2- 172.21.15.90:0/1438783529 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ca3fa7e670 0x55ca3fa7aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:41.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.302+0000 7fc963e00640 1 -- 172.21.15.90:0/1438783529 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ca3fa79500 msgr2=0x55ca3fa8a5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:34:41.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.302+0000 7fc963e00640 1 --2- 172.21.15.90:0/1438783529 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ca3fa79500 0x55ca3fa8a5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:41.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.302+0000 7fc963e00640 1 -- 172.21.15.90:0/1438783529 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ca3fa7e670 msgr2=0x55ca3fa7aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:34:41.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.302+0000 7fc963e00640 1 --2- 172.21.15.90:0/1438783529 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ca3fa7e670 0x55ca3fa7aa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:41.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.302+0000 7fc963e00640 1 -- 172.21.15.90:0/1438783529 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ca3fa91650 con 0x55ca3fa78b10 2024-04-02T14:34:41.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.302+0000 7fc963e00640 1 --2- 172.21.15.90:0/1438783529 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ca3fa78b10 0x55ca3fa81500 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fc95c005c40 tx=0x7fc95c009b10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd8aacca0084c420 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:34:41.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.302+0000 7fc953400640 1 -- 172.21.15.90:0/1438783529 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc95c00fa60 con 0x55ca3fa78b10 2024-04-02T14:34:41.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.302+0000 7fc953400640 1 -- 172.21.15.90:0/1438783529 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc95c01f040 con 0x55ca3fa78b10 2024-04-02T14:34:41.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.302+0000 7fc953400640 1 -- 172.21.15.90:0/1438783529 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc95c02e870 con 0x55ca3fa78b10 2024-04-02T14:34:41.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.303+0000 7fc96cd6f740 1 -- 172.21.15.90:0/1438783529 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ca3fa78b10 msgr2=0x55ca3fa81500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:41.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.303+0000 7fc96cd6f740 1 --2- 172.21.15.90:0/1438783529 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ca3fa78b10 0x55ca3fa81500 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fc95c005c40 tx=0x7fc95c009b10 comp rx=0 tx=0).stop 2024-04-02T14:34:41.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.303+0000 7fc96cd6f740 1 -- 172.21.15.90:0/1438783529 shutdown_connections 2024-04-02T14:34:41.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.303+0000 7fc96cd6f740 1 --2- 172.21.15.90:0/1438783529 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ca3fa7e670 0x55ca3fa7aa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:41.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.303+0000 7fc96cd6f740 1 --2- 172.21.15.90:0/1438783529 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ca3fa79500 0x55ca3fa8a5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:41.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.303+0000 7fc96cd6f740 1 --2- 172.21.15.90:0/1438783529 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ca3fa78b10 0x55ca3fa81500 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:41.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.303+0000 7fc96cd6f740 1 -- 172.21.15.90:0/1438783529 >> 172.21.15.90:0/1438783529 conn(0x55ca3fa74080 msgr2=0x55ca3fa74460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:34:41.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.303+0000 7fc96cd6f740 1 -- 172.21.15.90:0/1438783529 shutdown_connections 2024-04-02T14:34:41.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.303+0000 7fc96cd6f740 1 -- 172.21.15.90:0/1438783529 wait complete. 2024-04-02T14:34:41.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.303+0000 7fc96cd6f740 1 -- start start 2024-04-02T14:34:41.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.303+0000 7fc96cd6f740 1 Processor -- start 2024-04-02T14:34:41.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.304+0000 7fc96cd6f740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ca3fa78b10 0x55ca3fa86040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:41.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.304+0000 7fc96cd6f740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ca3fa79500 0x55ca3fa86580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:41.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.304+0000 7fc96cd6f740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ca3fa7e670 0x55ca3fa895a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:41.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.304+0000 7fc96cd6f740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55ca3fa911a0 con 0x55ca3fa7e670 2024-04-02T14:34:41.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.304+0000 7fc96cd6f740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55ca3fa90ad0 con 0x55ca3fa79500 2024-04-02T14:34:41.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.304+0000 7fc96cd6f740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55ca3fa91870 con 0x55ca3fa78b10 2024-04-02T14:34:41.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.304+0000 7fc96cd6f740 10 client.0 _ll_register_callbacks cb 0x7fc94c00be20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:34:41.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.304+0000 7fc963e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ca3fa7e670 0x55ca3fa895a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:41.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.304+0000 7fc963e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ca3fa7e670 0x55ca3fa895a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:36508/0 (socket says 172.21.15.90:36508) 2024-04-02T14:34:41.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.304+0000 7fc963e00640 1 -- 172.21.15.90:0/999267104 learned_addr learned my addr 172.21.15.90:0/999267104 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:34:41.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.304+0000 7fc95aa00640 1 --2- 172.21.15.90:0/999267104 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ca3fa79500 0x55ca3fa86580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:41.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.304+0000 7fc963e00640 1 --2- 172.21.15.90:0/999267104 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ca3fa78b10 0x55ca3fa86040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:41.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.304+0000 7fc95aa00640 1 -- 172.21.15.90:0/999267104 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ca3fa78b10 msgr2=0x55ca3fa86040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:34:41.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.304+0000 7fc95aa00640 1 --2- 172.21.15.90:0/999267104 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ca3fa78b10 0x55ca3fa86040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:41.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.304+0000 7fc95aa00640 1 -- 172.21.15.90:0/999267104 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ca3fa7e670 msgr2=0x55ca3fa895a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:34:41.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.304+0000 7fc95aa00640 1 --2- 172.21.15.90:0/999267104 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ca3fa7e670 0x55ca3fa895a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:41.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.304+0000 7fc95aa00640 1 -- 172.21.15.90:0/999267104 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ca3fa89b90 con 0x55ca3fa79500 2024-04-02T14:34:41.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.305+0000 7fc95aa00640 1 --2- 172.21.15.90:0/999267104 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ca3fa79500 0x55ca3fa86580 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fc94c00fb80 tx=0x7fc94c00ff80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4067dcaf43f4a03f server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:34:41.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.305+0000 7fc950c00640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:34:41.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.305+0000 7fc950c00640 1 -- 172.21.15.90:0/999267104 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc94c029030 con 0x55ca3fa79500 2024-04-02T14:34:41.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.305+0000 7fc950c00640 1 -- 172.21.15.90:0/999267104 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8248+0+0 (secure 0 0 0) 0x7fc94c017070 con 0x55ca3fa79500 2024-04-02T14:34:41.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.305+0000 7fc96cd6f740 1 -- 172.21.15.90:0/999267104 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc95c01f040 con 0x55ca3fa79500 2024-04-02T14:34:41.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.305+0000 7fc950c00640 1 -- 172.21.15.90:0/999267104 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fc94c00f710 con 0x55ca3fa79500 2024-04-02T14:34:41.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.305+0000 7fc96cd6f740 10 client.25196 fetch_fsmap learned FSMap version 26 2024-04-02T14:34:41.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.305+0000 7fc96cd6f740 1 -- 172.21.15.90:0/999267104 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55ca3faf6170 con 0x55ca3fa79500 2024-04-02T14:34:41.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.306+0000 7fc950c00640 1 -- 172.21.15.90:0/999267104 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc95c01f040 con 0x55ca3fa79500 2024-04-02T14:34:41.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.306+0000 7fc950c00640 1 -- 172.21.15.90:0/999267104 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7fc94c025420 con 0x55ca3fa79500 2024-04-02T14:34:41.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.306+0000 7fc96cd6f740 10 client.25196 fetch_fsmap finished waiting for FSMap version 26 2024-04-02T14:34:41.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.306+0000 7fc96cd6f740 10 client.25196 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-02T14:34:41.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.306+0000 7fc96cd6f740 20 client.25196 populate_metadata read hostname 'smithi090' 2024-04-02T14:34:41.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.306+0000 7fc96cd6f740 1 --2- 172.21.15.90:0/999267104 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x55ca3fafad30 0x55ca3faf9a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:41.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.306+0000 7fc96cd6f740 4 client.25196 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-02T14:34:41.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.306+0000 7fc96cd6f740 1 -- 172.21.15.90:0/999267104 --> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] -- command(tid 0: {"prefix": "status"}) -- 0x55ca3f9e58f0 con 0x55ca3fafad30 2024-04-02T14:34:41.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.306+0000 7fc963e00640 1 --2- 172.21.15.90:0/999267104 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x55ca3fafad30 0x55ca3faf9a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:41.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.307+0000 7fc963e00640 1 --2- 172.21.15.90:0/999267104 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x55ca3fafad30 0x55ca3faf9a40 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=a53b53dd9f134d34 server_cookie=6b23a61ab64f494d in_seq=0 out_seq=0 2024-04-02T14:34:41.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.307+0000 7fc950c00640 10 client.25196 ms_handle_connect on v2:172.21.15.177:6836/4223414858 2024-04-02T14:34:41.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.307+0000 7fc950c00640 1 -- 172.21.15.90:0/999267104 <== mds.4 v2:172.21.15.177:6836/4223414858 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x55ca3f9e58f0 con 0x55ca3fafad30 2024-04-02T14:34:41.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.307+0000 7fc950c00640 10 client.25196 handle_command_reply: tid=0 2024-04-02T14:34:41.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.307+0000 7fc96cd6f740 1 client.25196 shutdown 2024-04-02T14:34:41.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.307+0000 7fc96cd6f740 1 -- 172.21.15.90:0/999267104 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ca3fa79500 msgr2=0x55ca3fa86580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:41.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.307+0000 7fc96cd6f740 1 --2- 172.21.15.90:0/999267104 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ca3fa79500 0x55ca3fa86580 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fc94c00fb80 tx=0x7fc94c00ff80 comp rx=0 tx=0).stop 2024-04-02T14:34:41.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.307+0000 7fc96cd6f740 1 -- 172.21.15.90:0/999267104 shutdown_connections 2024-04-02T14:34:41.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.307+0000 7fc96cd6f740 1 --2- 172.21.15.90:0/999267104 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x55ca3fafad30 0x55ca3faf9a40 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:41.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.307+0000 7fc96cd6f740 1 --2- 172.21.15.90:0/999267104 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ca3fa7e670 0x55ca3fa895a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:41.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.307+0000 7fc96cd6f740 1 --2- 172.21.15.90:0/999267104 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ca3fa79500 0x55ca3fa86580 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:41.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.307+0000 7fc96cd6f740 1 --2- 172.21.15.90:0/999267104 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ca3fa78b10 0x55ca3fa86040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:41.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.307+0000 7fc950c00640 0 client.25196 ms_handle_reset on v2:172.21.15.177:6836/4223414858 2024-04-02T14:34:41.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.307+0000 7fc96cd6f740 1 -- 172.21.15.90:0/999267104 >> 172.21.15.90:0/999267104 conn(0x55ca3fa74080 msgr2=0x55ca3fa7aa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:34:41.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.307+0000 7fc96cd6f740 1 -- 172.21.15.90:0/999267104 shutdown_connections 2024-04-02T14:34:41.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.307+0000 7fc96cd6f740 1 -- 172.21.15.90:0/999267104 wait complete. 2024-04-02T14:34:41.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.307+0000 7fc96cd6f740 20 client.25196 trim_cache size 0 max 16384 2024-04-02T14:34:41.317 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-02T14:34:41.317 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_fsid": "6449448a-f0fc-11ee-b647-cb9ed24678a4", 2024-04-02T14:34:41.317 INFO:teuthology.orchestra.run.smithi090.stdout: "whoami": 4, 2024-04-02T14:34:41.317 INFO:teuthology.orchestra.run.smithi090.stdout: "id": 24523, 2024-04-02T14:34:41.317 INFO:teuthology.orchestra.run.smithi090.stdout: "want_state": "up:active", 2024-04-02T14:34:41.317 INFO:teuthology.orchestra.run.smithi090.stdout: "state": "up:active", 2024-04-02T14:34:41.317 INFO:teuthology.orchestra.run.smithi090.stdout: "fs_name": "cephfs", 2024-04-02T14:34:41.317 INFO:teuthology.orchestra.run.smithi090.stdout: "rank_uptime": 132.34842695699999, 2024-04-02T14:34:41.317 INFO:teuthology.orchestra.run.smithi090.stdout: "mdsmap_epoch": 26, 2024-04-02T14:34:41.317 INFO:teuthology.orchestra.run.smithi090.stdout: "osdmap_epoch": 90, 2024-04-02T14:34:41.318 INFO:teuthology.orchestra.run.smithi090.stdout: "osdmap_epoch_barrier": 88, 2024-04-02T14:34:41.318 INFO:teuthology.orchestra.run.smithi090.stdout: "uptime": 160.63475707800001 2024-04-02T14:34:41.318 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-02T14:34:41.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.308+0000 7fc968a00640 1 -- 172.21.15.90:0/2217405984 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fc9480a3bb0 msgr2=0x7fc9480a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:41.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.308+0000 7fc968a00640 1 --2- 172.21.15.90:0/2217405984 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fc9480a3bb0 0x7fc9480a6070 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x55ca3fa79900 tx=0x7fc95402bc50 comp rx=0 tx=0).stop 2024-04-02T14:34:41.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.308+0000 7fc968a00640 1 -- 172.21.15.90:0/2217405984 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc964108020 msgr2=0x7fc96419e3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:41.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.308+0000 7fc968a00640 1 --2- 172.21.15.90:0/2217405984 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc964108020 0x7fc96419e3d0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fc95402c190 tx=0x7fc95402c6e0 comp rx=0 tx=0).stop 2024-04-02T14:34:41.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.308+0000 7fc968a00640 1 -- 172.21.15.90:0/2217405984 shutdown_connections 2024-04-02T14:34:41.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.308+0000 7fc968a00640 1 --2- 172.21.15.90:0/2217405984 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fc9480a3bb0 0x7fc9480a6070 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:41.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.308+0000 7fc968a00640 1 --2- 172.21.15.90:0/2217405984 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc964109bd0 0x7fc9641a5980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:41.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.308+0000 7fc968a00640 1 --2- 172.21.15.90:0/2217405984 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc964109210 0x7fc96419e910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:41.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.308+0000 7fc968a00640 1 --2- 172.21.15.90:0/2217405984 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc964108020 0x7fc96419e3d0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:41.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.308+0000 7fc968a00640 1 -- 172.21.15.90:0/2217405984 >> 172.21.15.90:0/2217405984 conn(0x7fc964076fd0 msgr2=0x7fc96410c510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:34:41.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.308+0000 7fc968a00640 1 -- 172.21.15.90:0/2217405984 shutdown_connections 2024-04-02T14:34:41.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.308+0000 7fc968a00640 1 -- 172.21.15.90:0/2217405984 wait complete. 2024-04-02T14:34:41.325 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.h status 2024-04-02T14:34:41.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:34:40 smithi092 ceph-mon[36497]: from='client.? 172.21.15.90:0/3315420596' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-02T14:34:41.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:34:40 smithi092 ceph-mon[36497]: from='client.? 172.21.15.90:0/151380853' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-02T14:34:41.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.505+0000 7fa195a00640 1 -- 172.21.15.90:0/1595984409 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa190103d40 msgr2=0x7fa190104160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:41.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.505+0000 7fa195a00640 1 --2- 172.21.15.90:0/1595984409 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa190103d40 0x7fa190104160 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fa180009900 tx=0x7fa18002bc30 comp rx=0 tx=0).stop 2024-04-02T14:34:41.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.505+0000 7fa195a00640 1 -- 172.21.15.90:0/1595984409 shutdown_connections 2024-04-02T14:34:41.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.505+0000 7fa195a00640 1 --2- 172.21.15.90:0/1595984409 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa1901058f0 0x7fa19010c180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:41.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.505+0000 7fa195a00640 1 --2- 172.21.15.90:0/1595984409 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa190104f30 0x7fa1901053b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:41.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.505+0000 7fa195a00640 1 --2- 172.21.15.90:0/1595984409 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa190103d40 0x7fa190104160 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:41.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.505+0000 7fa195a00640 1 -- 172.21.15.90:0/1595984409 >> 172.21.15.90:0/1595984409 conn(0x7fa1900ff2b0 msgr2=0x7fa1901016f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:34:41.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.505+0000 7fa195a00640 1 -- 172.21.15.90:0/1595984409 shutdown_connections 2024-04-02T14:34:41.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.505+0000 7fa195a00640 1 -- 172.21.15.90:0/1595984409 wait complete. 2024-04-02T14:34:41.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.506+0000 7fa195a00640 1 Processor -- start 2024-04-02T14:34:41.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.506+0000 7fa195a00640 1 -- start start 2024-04-02T14:34:41.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.506+0000 7fa195a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa190103d40 0x7fa19019a080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:41.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.506+0000 7fa195a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa190104f30 0x7fa19019a5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:41.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.506+0000 7fa195a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa1901058f0 0x7fa1901a1630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:41.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.506+0000 7fa195a00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fa19010fa00 con 0x7fa190104f30 2024-04-02T14:34:41.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.506+0000 7fa195a00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fa19010f880 con 0x7fa1901058f0 2024-04-02T14:34:41.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.506+0000 7fa195a00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fa19010fb80 con 0x7fa190103d40 2024-04-02T14:34:41.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.507+0000 7fa18e000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa190104f30 0x7fa19019a5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:41.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.507+0000 7fa18e000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa190104f30 0x7fa19019a5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:36514/0 (socket says 172.21.15.90:36514) 2024-04-02T14:34:41.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.507+0000 7fa18e000640 1 -- 172.21.15.90:0/529601369 learned_addr learned my addr 172.21.15.90:0/529601369 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:34:41.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.507+0000 7fa18ea00640 1 --2- 172.21.15.90:0/529601369 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa190103d40 0x7fa19019a080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:41.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.507+0000 7fa18f400640 1 --2- 172.21.15.90:0/529601369 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa1901058f0 0x7fa1901a1630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:41.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.507+0000 7fa18e000640 1 -- 172.21.15.90:0/529601369 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa190103d40 msgr2=0x7fa19019a080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:41.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.507+0000 7fa18e000640 1 --2- 172.21.15.90:0/529601369 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa190103d40 0x7fa19019a080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:41.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.507+0000 7fa18e000640 1 -- 172.21.15.90:0/529601369 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa1901058f0 msgr2=0x7fa1901a1630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:41.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.507+0000 7fa18e000640 1 --2- 172.21.15.90:0/529601369 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa1901058f0 0x7fa1901a1630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:41.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.507+0000 7fa18e000640 1 -- 172.21.15.90:0/529601369 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1901a1ca0 con 0x7fa190104f30 2024-04-02T14:34:41.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.507+0000 7fa18e000640 1 --2- 172.21.15.90:0/529601369 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa190104f30 0x7fa19019a5c0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fa17c00e9f0 tx=0x7fa17c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:34:41.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.507+0000 7fa187400640 1 -- 172.21.15.90:0/529601369 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa17c00cd80 con 0x7fa190104f30 2024-04-02T14:34:41.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.507+0000 7fa187400640 1 -- 172.21.15.90:0/529601369 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa17c005500 con 0x7fa190104f30 2024-04-02T14:34:41.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.507+0000 7fa195a00640 1 -- 172.21.15.90:0/529601369 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa1901a1f90 con 0x7fa190104f30 2024-04-02T14:34:41.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.507+0000 7fa187400640 1 -- 172.21.15.90:0/529601369 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa17c010690 con 0x7fa190104f30 2024-04-02T14:34:41.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.508+0000 7fa195a00640 1 -- 172.21.15.90:0/529601369 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1901a2400 con 0x7fa190104f30 2024-04-02T14:34:41.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.510+0000 7fa198813740 1 Processor -- start 2024-04-02T14:34:41.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.511+0000 7fa198813740 1 -- start start 2024-04-02T14:34:41.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.511+0000 7fa198813740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558fb9f5f8f0 0x558fb9f5fd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:41.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.511+0000 7fa198813740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558fb9f602e0 0x558fb9f64de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:41.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.511+0000 7fa198813740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558fb9f65450 0x558fb9f678e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:41.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.511+0000 7fa198813740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x558fb9ec5060 con 0x558fb9f5f8f0 2024-04-02T14:34:41.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.511+0000 7fa198813740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x558fb9e6d560 con 0x558fb9f602e0 2024-04-02T14:34:41.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.511+0000 7fa198813740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x558fb9ec1010 con 0x558fb9f65450 2024-04-02T14:34:41.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.511+0000 7fa18f400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558fb9f5f8f0 0x558fb9f5fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:41.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.511+0000 7fa18f400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558fb9f5f8f0 0x558fb9f5fd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:36520/0 (socket says 172.21.15.90:36520) 2024-04-02T14:34:41.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.511+0000 7fa18f400640 1 -- 172.21.15.90:0/3286263513 learned_addr learned my addr 172.21.15.90:0/3286263513 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:34:41.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.511+0000 7fa18ea00640 1 --2- 172.21.15.90:0/3286263513 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558fb9f602e0 0x558fb9f64de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:41.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.511+0000 7fa187400640 1 -- 172.21.15.90:0/529601369 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fa17c0046c0 con 0x7fa190104f30 2024-04-02T14:34:41.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.511+0000 7fa18f400640 1 -- 172.21.15.90:0/3286263513 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558fb9f65450 msgr2=0x558fb9f678e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-02T14:34:41.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.512+0000 7fa18f400640 1 --2- 172.21.15.90:0/3286263513 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558fb9f65450 0x558fb9f678e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:41.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.512+0000 7fa18f400640 1 -- 172.21.15.90:0/3286263513 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558fb9f602e0 msgr2=0x558fb9f64de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:34:41.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.512+0000 7fa18f400640 1 --2- 172.21.15.90:0/3286263513 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558fb9f602e0 0x558fb9f64de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:41.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.512+0000 7fa18f400640 1 -- 172.21.15.90:0/3286263513 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558fb9e8ffd0 con 0x558fb9f5f8f0 2024-04-02T14:34:41.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.512+0000 7fa18ea00640 1 --2- 172.21.15.90:0/3286263513 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558fb9f602e0 0x558fb9f64de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:34:41.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.512+0000 7fa18f400640 1 --2- 172.21.15.90:0/3286263513 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558fb9f5f8f0 0x558fb9f5fd10 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fa188009fd0 tx=0x7fa1880118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1aac53506472a3c8 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:34:41.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.512+0000 7fa17be00640 1 -- 172.21.15.90:0/3286263513 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa188004d20 con 0x558fb9f5f8f0 2024-04-02T14:34:41.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.512+0000 7fa17be00640 1 -- 172.21.15.90:0/3286263513 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa188004050 con 0x558fb9f5f8f0 2024-04-02T14:34:41.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.512+0000 7fa17be00640 1 -- 172.21.15.90:0/3286263513 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa188005260 con 0x558fb9f5f8f0 2024-04-02T14:34:41.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.513+0000 7fa198813740 1 -- 172.21.15.90:0/3286263513 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558fb9f5f8f0 msgr2=0x558fb9f5fd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:41.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.513+0000 7fa198813740 1 --2- 172.21.15.90:0/3286263513 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558fb9f5f8f0 0x558fb9f5fd10 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fa188009fd0 tx=0x7fa1880118d0 comp rx=0 tx=0).stop 2024-04-02T14:34:41.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.513+0000 7fa198813740 1 -- 172.21.15.90:0/3286263513 shutdown_connections 2024-04-02T14:34:41.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.513+0000 7fa198813740 1 --2- 172.21.15.90:0/3286263513 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558fb9f65450 0x558fb9f678e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:41.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.513+0000 7fa198813740 1 --2- 172.21.15.90:0/3286263513 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558fb9f602e0 0x558fb9f64de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:41.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.513+0000 7fa198813740 1 --2- 172.21.15.90:0/3286263513 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558fb9f5f8f0 0x558fb9f5fd10 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:41.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.513+0000 7fa198813740 1 -- 172.21.15.90:0/3286263513 >> 172.21.15.90:0/3286263513 conn(0x558fb9f5ae60 msgr2=0x558fb9f5b260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:34:41.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.513+0000 7fa198813740 1 -- 172.21.15.90:0/3286263513 shutdown_connections 2024-04-02T14:34:41.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.513+0000 7fa187400640 1 --2- 172.21.15.90:0/529601369 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fa1740a3b60 0x7fa1740a6020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:41.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.513+0000 7fa198813740 1 -- 172.21.15.90:0/3286263513 wait complete. 2024-04-02T14:34:41.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.513+0000 7fa187400640 1 -- 172.21.15.90:0/529601369 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8248+0+0 (secure 0 0 0) 0x7fa17c0d07a0 con 0x7fa190104f30 2024-04-02T14:34:41.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.513+0000 7fa18ea00640 1 --2- 172.21.15.90:0/529601369 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fa1740a3b60 0x7fa1740a6020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:41.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.514+0000 7fa198813740 1 -- start start 2024-04-02T14:34:41.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.514+0000 7fa198813740 1 Processor -- start 2024-04-02T14:34:41.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.514+0000 7fa18ea00640 1 --2- 172.21.15.90:0/529601369 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fa1740a3b60 0x7fa1740a6020 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fa18002c170 tx=0x7fa18002df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:34:41.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.514+0000 7fa198813740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558fb9f5f8f0 0x558fb9f69e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:41.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.514+0000 7fa198813740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558fb9f602e0 0x558fb9f6a370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:41.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.514+0000 7fa198813740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558fb9f65450 0x558fb9f713b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:41.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.514+0000 7fa198813740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x558fb9f68400 con 0x558fb9f65450 2024-04-02T14:34:41.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.514+0000 7fa198813740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x558fb9e80920 con 0x558fb9f5f8f0 2024-04-02T14:34:41.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.514+0000 7fa198813740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x558fb9f68580 con 0x558fb9f602e0 2024-04-02T14:34:41.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.514+0000 7fa198813740 10 client.0 _ll_register_callbacks cb 0x558fb9e80310 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:34:41.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.515+0000 7fa18f400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558fb9f65450 0x558fb9f713b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:41.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.515+0000 7fa18f400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558fb9f65450 0x558fb9f713b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:36524/0 (socket says 172.21.15.90:36524) 2024-04-02T14:34:41.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.515+0000 7fa18ea00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558fb9f5f8f0 0x558fb9f69e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:41.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.515+0000 7fa18f400640 1 -- 172.21.15.90:0/3249796000 learned_addr learned my addr 172.21.15.90:0/3249796000 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:34:41.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.515+0000 7fa18e000640 1 --2- 172.21.15.90:0/3249796000 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558fb9f602e0 0x558fb9f6a370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:41.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.515+0000 7fa18f400640 1 -- 172.21.15.90:0/3249796000 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558fb9f602e0 msgr2=0x558fb9f6a370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:34:41.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.515+0000 7fa18f400640 1 --2- 172.21.15.90:0/3249796000 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558fb9f602e0 0x558fb9f6a370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:41.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.515+0000 7fa18f400640 1 -- 172.21.15.90:0/3249796000 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558fb9f5f8f0 msgr2=0x558fb9f69e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:34:41.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.515+0000 7fa18f400640 1 --2- 172.21.15.90:0/3249796000 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558fb9f5f8f0 0x558fb9f69e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:41.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.515+0000 7fa18f400640 1 -- 172.21.15.90:0/3249796000 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558fb9bd92d0 con 0x558fb9f65450 2024-04-02T14:34:41.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.515+0000 7fa18f400640 1 --2- 172.21.15.90:0/3249796000 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558fb9f65450 0x558fb9f713b0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fa188011dd0 tx=0x7fa188004d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=957905b622ea2d79 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:34:41.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.515+0000 7fa178c00640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-02T14:34:41.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.516+0000 7fa178c00640 1 -- 172.21.15.90:0/3249796000 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa188028020 con 0x558fb9f65450 2024-04-02T14:34:41.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.516+0000 7fa178c00640 1 -- 172.21.15.90:0/3249796000 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8248+0+0 (secure 0 0 0) 0x7fa188004410 con 0x558fb9f65450 2024-04-02T14:34:41.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.516+0000 7fa198813740 1 -- 172.21.15.90:0/3249796000 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa188004050 con 0x558fb9f65450 2024-04-02T14:34:41.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.516+0000 7fa178c00640 1 -- 172.21.15.90:0/3249796000 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fa188033050 con 0x558fb9f65450 2024-04-02T14:34:41.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.516+0000 7fa198813740 10 client.15102 fetch_fsmap learned FSMap version 26 2024-04-02T14:34:41.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.516+0000 7fa198813740 1 -- 172.21.15.90:0/3249796000 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x558fb9ca36b0 con 0x558fb9f65450 2024-04-02T14:34:41.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.516+0000 7fa178c00640 1 -- 172.21.15.90:0/3249796000 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa188004050 con 0x558fb9f65450 2024-04-02T14:34:41.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.516+0000 7fa178c00640 1 -- 172.21.15.90:0/3249796000 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7fa18800f4e0 con 0x558fb9f65450 2024-04-02T14:34:41.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.516+0000 7fa198813740 10 client.15102 fetch_fsmap finished waiting for FSMap version 26 2024-04-02T14:34:41.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.517+0000 7fa198813740 10 client.15102 resolve_mds: resolved name 'h' to daemon mds.h 2024-04-02T14:34:41.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.517+0000 7fa198813740 20 client.15102 populate_metadata read hostname 'smithi090' 2024-04-02T14:34:41.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.517+0000 7fa198813740 1 --2- 172.21.15.90:0/3249796000 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x558fb9f769d0 0x558fb9f77e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:41.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.517+0000 7fa198813740 4 client.15102 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:34:41.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.517+0000 7fa198813740 1 -- 172.21.15.90:0/3249796000 --> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558fb9ed25a0 con 0x558fb9f769d0 2024-04-02T14:34:41.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.517+0000 7fa18ea00640 1 --2- 172.21.15.90:0/3249796000 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x558fb9f769d0 0x558fb9f77e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:41.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.518+0000 7fa18ea00640 1 --2- 172.21.15.90:0/3249796000 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x558fb9f769d0 0x558fb9f77e40 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=1f5cb563819ebb15 server_cookie=693cc2b08334dd4 in_seq=0 out_seq=0 2024-04-02T14:34:41.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.518+0000 7fa178c00640 10 client.15102 ms_handle_connect on v2:172.21.15.92:6836/1270635939 2024-04-02T14:34:41.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.518+0000 7fa178c00640 1 -- 172.21.15.90:0/3249796000 <== mds.3 v2:172.21.15.92:6836/1270635939 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x558fb9ed25a0 con 0x558fb9f769d0 2024-04-02T14:34:41.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.518+0000 7fa178c00640 10 client.15102 handle_command_reply: tid=0 2024-04-02T14:34:41.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.518+0000 7fa198813740 1 client.15102 shutdown 2024-04-02T14:34:41.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.518+0000 7fa198813740 1 -- 172.21.15.90:0/3249796000 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558fb9f65450 msgr2=0x558fb9f713b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:41.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.518+0000 7fa198813740 1 --2- 172.21.15.90:0/3249796000 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558fb9f65450 0x558fb9f713b0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fa188011dd0 tx=0x7fa188004d60 comp rx=0 tx=0).stop 2024-04-02T14:34:41.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.519+0000 7fa198813740 1 -- 172.21.15.90:0/3249796000 shutdown_connections 2024-04-02T14:34:41.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.519+0000 7fa198813740 1 --2- 172.21.15.90:0/3249796000 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x558fb9f769d0 0x558fb9f77e40 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:41.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.519+0000 7fa198813740 1 --2- 172.21.15.90:0/3249796000 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558fb9f65450 0x558fb9f713b0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:41.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.519+0000 7fa198813740 1 --2- 172.21.15.90:0/3249796000 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558fb9f602e0 0x558fb9f6a370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:41.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.519+0000 7fa178c00640 0 client.15102 ms_handle_reset on v2:172.21.15.92:6836/1270635939 2024-04-02T14:34:41.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.519+0000 7fa198813740 1 --2- 172.21.15.90:0/3249796000 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558fb9f5f8f0 0x558fb9f69e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:41.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.519+0000 7fa198813740 1 -- 172.21.15.90:0/3249796000 >> 172.21.15.90:0/3249796000 conn(0x558fb9f5ae60 msgr2=0x558fb9f5a080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:34:41.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.519+0000 7fa198813740 1 -- 172.21.15.90:0/3249796000 shutdown_connections 2024-04-02T14:34:41.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.519+0000 7fa198813740 1 -- 172.21.15.90:0/3249796000 wait complete. 2024-04-02T14:34:41.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.519+0000 7fa198813740 20 client.15102 trim_cache size 0 max 16384 2024-04-02T14:34:41.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.537+0000 7fa198813740 1 Processor -- start 2024-04-02T14:34:41.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.537+0000 7fa198813740 1 -- start start 2024-04-02T14:34:41.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.538+0000 7fa198813740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558fb9f5f8f0 0x558fb9f55de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:41.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.538+0000 7fa198813740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558fb9f602e0 0x558fb9f71370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:41.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.538+0000 7fa198813740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558fb9f65450 0x558fb9f5b240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:41.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.538+0000 7fa198813740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x558fb9f76180 con 0x558fb9f65450 2024-04-02T14:34:41.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.538+0000 7fa198813740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x558fb9f76850 con 0x558fb9f602e0 2024-04-02T14:34:41.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.538+0000 7fa198813740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x558fb9f76f20 con 0x558fb9f5f8f0 2024-04-02T14:34:41.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.538+0000 7fa18f400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558fb9f65450 0x558fb9f5b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:41.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.538+0000 7fa18f400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558fb9f65450 0x558fb9f5b240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:36530/0 (socket says 172.21.15.90:36530) 2024-04-02T14:34:41.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.538+0000 7fa18f400640 1 -- 172.21.15.90:0/2393366816 learned_addr learned my addr 172.21.15.90:0/2393366816 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:34:41.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.538+0000 7fa18ea00640 1 --2- 172.21.15.90:0/2393366816 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558fb9f5f8f0 0x558fb9f55de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:41.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.538+0000 7fa18e000640 1 --2- 172.21.15.90:0/2393366816 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558fb9f602e0 0x558fb9f71370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:41.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.538+0000 7fa18f400640 1 -- 172.21.15.90:0/2393366816 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558fb9f5f8f0 msgr2=0x558fb9f55de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:34:41.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.538+0000 7fa18f400640 1 --2- 172.21.15.90:0/2393366816 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558fb9f5f8f0 0x558fb9f55de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:41.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.538+0000 7fa18f400640 1 -- 172.21.15.90:0/2393366816 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558fb9f602e0 msgr2=0x558fb9f71370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:34:41.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.538+0000 7fa18f400640 1 --2- 172.21.15.90:0/2393366816 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558fb9f602e0 0x558fb9f71370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:41.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.538+0000 7fa18f400640 1 -- 172.21.15.90:0/2393366816 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558fb9f6acd0 con 0x558fb9f65450 2024-04-02T14:34:41.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.538+0000 7fa18f400640 1 --2- 172.21.15.90:0/2393366816 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558fb9f65450 0x558fb9f5b240 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fa188004cc0 tx=0x7fa188004cf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=627fea51ede31d56 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:34:41.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.539+0000 7fa17b400640 1 -- 172.21.15.90:0/2393366816 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa188028040 con 0x558fb9f65450 2024-04-02T14:34:41.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.539+0000 7fa17b400640 1 -- 172.21.15.90:0/2393366816 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa188004000 con 0x558fb9f65450 2024-04-02T14:34:41.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.539+0000 7fa17b400640 1 -- 172.21.15.90:0/2393366816 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa188004410 con 0x558fb9f65450 2024-04-02T14:34:41.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.539+0000 7fa198813740 1 -- 172.21.15.90:0/2393366816 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558fb9f65450 msgr2=0x558fb9f5b240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:41.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.539+0000 7fa198813740 1 --2- 172.21.15.90:0/2393366816 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558fb9f65450 0x558fb9f5b240 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fa188004cc0 tx=0x7fa188004cf0 comp rx=0 tx=0).stop 2024-04-02T14:34:41.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.539+0000 7fa198813740 1 -- 172.21.15.90:0/2393366816 shutdown_connections 2024-04-02T14:34:41.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.539+0000 7fa198813740 1 --2- 172.21.15.90:0/2393366816 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558fb9f65450 0x558fb9f5b240 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:41.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.539+0000 7fa198813740 1 --2- 172.21.15.90:0/2393366816 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558fb9f602e0 0x558fb9f71370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:41.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.539+0000 7fa198813740 1 --2- 172.21.15.90:0/2393366816 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558fb9f5f8f0 0x558fb9f55de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:41.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.539+0000 7fa198813740 1 -- 172.21.15.90:0/2393366816 >> 172.21.15.90:0/2393366816 conn(0x558fb9f5ae60 msgr2=0x558fb9f60c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:34:41.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.539+0000 7fa198813740 1 -- 172.21.15.90:0/2393366816 shutdown_connections 2024-04-02T14:34:41.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.539+0000 7fa198813740 1 -- 172.21.15.90:0/2393366816 wait complete. 2024-04-02T14:34:41.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.540+0000 7fa198813740 1 -- start start 2024-04-02T14:34:41.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.540+0000 7fa198813740 1 Processor -- start 2024-04-02T14:34:41.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.540+0000 7fa198813740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558fb9f5f8f0 0x558fb9f6c780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:41.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.540+0000 7fa198813740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558fb9f602e0 0x558fb9f6ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:41.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.540+0000 7fa198813740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558fb9f65450 0x558fb9f6fce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:41.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.540+0000 7fa198813740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x558fb9f77cc0 con 0x558fb9f602e0 2024-04-02T14:34:41.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.540+0000 7fa198813740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x558fb9f775f0 con 0x558fb9f65450 2024-04-02T14:34:41.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.540+0000 7fa198813740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x558fb9f78390 con 0x558fb9f5f8f0 2024-04-02T14:34:41.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.540+0000 7fa198813740 10 client.0 _ll_register_callbacks cb 0x7fa188003df0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:34:41.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.541+0000 7fa18e000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558fb9f602e0 0x558fb9f6ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:41.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.541+0000 7fa18f400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558fb9f65450 0x558fb9f6fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:41.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.541+0000 7fa18e000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558fb9f602e0 0x558fb9f6ccc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:36532/0 (socket says 172.21.15.90:36532) 2024-04-02T14:34:41.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.541+0000 7fa18e000640 1 -- 172.21.15.90:0/1242378529 learned_addr learned my addr 172.21.15.90:0/1242378529 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:34:41.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.541+0000 7fa18ea00640 1 --2- 172.21.15.90:0/1242378529 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558fb9f5f8f0 0x558fb9f6c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:41.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.541+0000 7fa18f400640 1 -- 172.21.15.90:0/1242378529 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558fb9f5f8f0 msgr2=0x558fb9f6c780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:34:41.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.541+0000 7fa18f400640 1 --2- 172.21.15.90:0/1242378529 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558fb9f5f8f0 0x558fb9f6c780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:41.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.541+0000 7fa18f400640 1 -- 172.21.15.90:0/1242378529 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558fb9f602e0 msgr2=0x558fb9f6ccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:34:41.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.541+0000 7fa18f400640 1 --2- 172.21.15.90:0/1242378529 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558fb9f602e0 0x558fb9f6ccc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:41.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.541+0000 7fa18f400640 1 -- 172.21.15.90:0/1242378529 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558fb9f702d0 con 0x558fb9f65450 2024-04-02T14:34:41.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.541+0000 7fa18e000640 1 --2- 172.21.15.90:0/1242378529 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558fb9f602e0 0x558fb9f6ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:34:41.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.541+0000 7fa18f400640 1 --2- 172.21.15.90:0/1242378529 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558fb9f65450 0x558fb9f6fce0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x558fb9f693f0 tx=0x7fa18800f810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6503db3a2c4dbca0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:34:41.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.541+0000 7fa178c00640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:34:41.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.541+0000 7fa178c00640 1 -- 172.21.15.90:0/1242378529 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa188028070 con 0x558fb9f65450 2024-04-02T14:34:41.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.541+0000 7fa178c00640 1 -- 172.21.15.90:0/1242378529 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8248+0+0 (secure 0 0 0) 0x7fa188004410 con 0x558fb9f65450 2024-04-02T14:34:41.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.541+0000 7fa198813740 1 -- 172.21.15.90:0/1242378529 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa188004000 con 0x558fb9f65450 2024-04-02T14:34:41.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.542+0000 7fa178c00640 1 -- 172.21.15.90:0/1242378529 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fa188004670 con 0x558fb9f65450 2024-04-02T14:34:41.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.542+0000 7fa198813740 10 client.25214 fetch_fsmap learned FSMap version 26 2024-04-02T14:34:41.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.542+0000 7fa198813740 1 -- 172.21.15.90:0/1242378529 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x558fb9f715d0 con 0x558fb9f65450 2024-04-02T14:34:41.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.542+0000 7fa178c00640 1 -- 172.21.15.90:0/1242378529 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa188004000 con 0x558fb9f65450 2024-04-02T14:34:41.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.542+0000 7fa178c00640 1 -- 172.21.15.90:0/1242378529 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7fa18804a400 con 0x558fb9f65450 2024-04-02T14:34:41.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.542+0000 7fa198813740 10 client.25214 fetch_fsmap finished waiting for FSMap version 26 2024-04-02T14:34:41.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.542+0000 7fa198813740 10 client.25214 resolve_mds: resolved name 'h' to daemon mds.h 2024-04-02T14:34:41.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.542+0000 7fa198813740 20 client.25214 populate_metadata read hostname 'smithi090' 2024-04-02T14:34:41.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.542+0000 7fa198813740 1 --2- 172.21.15.90:0/1242378529 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x558fb9fe1010 0x558fb9fdfd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:41.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.542+0000 7fa198813740 4 client.25214 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-02T14:34:41.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.542+0000 7fa198813740 1 -- 172.21.15.90:0/1242378529 --> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] -- command(tid 0: {"prefix": "status"}) -- 0x558fb9ecc8f0 con 0x558fb9fe1010 2024-04-02T14:34:41.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.543+0000 7fa18ea00640 1 --2- 172.21.15.90:0/1242378529 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x558fb9fe1010 0x558fb9fdfd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:41.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.543+0000 7fa18ea00640 1 --2- 172.21.15.90:0/1242378529 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x558fb9fe1010 0x558fb9fdfd20 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=55cab49d87062ca0 server_cookie=6886142cd3164dd2 in_seq=0 out_seq=0 2024-04-02T14:34:41.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.543+0000 7fa178c00640 10 client.25214 ms_handle_connect on v2:172.21.15.92:6836/1270635939 2024-04-02T14:34:41.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.543+0000 7fa178c00640 1 -- 172.21.15.90:0/1242378529 <== mds.3 v2:172.21.15.92:6836/1270635939 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x558fb9ecc8f0 con 0x558fb9fe1010 2024-04-02T14:34:41.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.543+0000 7fa178c00640 10 client.25214 handle_command_reply: tid=0 2024-04-02T14:34:41.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.543+0000 7fa198813740 1 client.25214 shutdown 2024-04-02T14:34:41.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.543+0000 7fa198813740 1 -- 172.21.15.90:0/1242378529 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558fb9f65450 msgr2=0x558fb9f6fce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:41.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.543+0000 7fa198813740 1 --2- 172.21.15.90:0/1242378529 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558fb9f65450 0x558fb9f6fce0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x558fb9f693f0 tx=0x7fa18800f810 comp rx=0 tx=0).stop 2024-04-02T14:34:41.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.544+0000 7fa198813740 1 -- 172.21.15.90:0/1242378529 shutdown_connections 2024-04-02T14:34:41.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.544+0000 7fa198813740 1 --2- 172.21.15.90:0/1242378529 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x558fb9fe1010 0x558fb9fdfd20 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:41.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.544+0000 7fa198813740 1 --2- 172.21.15.90:0/1242378529 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558fb9f65450 0x558fb9f6fce0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:41.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.544+0000 7fa198813740 1 --2- 172.21.15.90:0/1242378529 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558fb9f602e0 0x558fb9f6ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:41.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.544+0000 7fa178c00640 0 client.25214 ms_handle_reset on v2:172.21.15.92:6836/1270635939 2024-04-02T14:34:41.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.544+0000 7fa198813740 1 --2- 172.21.15.90:0/1242378529 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558fb9f5f8f0 0x558fb9f6c780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:41.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.544+0000 7fa198813740 1 -- 172.21.15.90:0/1242378529 >> 172.21.15.90:0/1242378529 conn(0x558fb9f5ae60 msgr2=0x558fb9f5b240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:34:41.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.544+0000 7fa198813740 1 -- 172.21.15.90:0/1242378529 shutdown_connections 2024-04-02T14:34:41.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.544+0000 7fa198813740 1 -- 172.21.15.90:0/1242378529 wait complete. 2024-04-02T14:34:41.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.544+0000 7fa198813740 20 client.25214 trim_cache size 0 max 16384 2024-04-02T14:34:41.555 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-02T14:34:41.555 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_fsid": "6449448a-f0fc-11ee-b647-cb9ed24678a4", 2024-04-02T14:34:41.555 INFO:teuthology.orchestra.run.smithi090.stdout: "whoami": 3, 2024-04-02T14:34:41.555 INFO:teuthology.orchestra.run.smithi090.stdout: "id": 24539, 2024-04-02T14:34:41.555 INFO:teuthology.orchestra.run.smithi090.stdout: "want_state": "up:active", 2024-04-02T14:34:41.555 INFO:teuthology.orchestra.run.smithi090.stdout: "state": "up:active", 2024-04-02T14:34:41.555 INFO:teuthology.orchestra.run.smithi090.stdout: "fs_name": "cephfs", 2024-04-02T14:34:41.555 INFO:teuthology.orchestra.run.smithi090.stdout: "rank_uptime": 133.610882022, 2024-04-02T14:34:41.555 INFO:teuthology.orchestra.run.smithi090.stdout: "mdsmap_epoch": 26, 2024-04-02T14:34:41.555 INFO:teuthology.orchestra.run.smithi090.stdout: "osdmap_epoch": 90, 2024-04-02T14:34:41.555 INFO:teuthology.orchestra.run.smithi090.stdout: "osdmap_epoch_barrier": 88, 2024-04-02T14:34:41.555 INFO:teuthology.orchestra.run.smithi090.stdout: "uptime": 158.10745192499999 2024-04-02T14:34:41.556 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-02T14:34:41.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.544+0000 7fa195a00640 1 -- 172.21.15.90:0/529601369 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fa1740a3b60 msgr2=0x7fa1740a6020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:41.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.544+0000 7fa195a00640 1 --2- 172.21.15.90:0/529601369 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fa1740a3b60 0x7fa1740a6020 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fa18002c170 tx=0x7fa18002df90 comp rx=0 tx=0).stop 2024-04-02T14:34:41.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.544+0000 7fa195a00640 1 -- 172.21.15.90:0/529601369 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa190104f30 msgr2=0x7fa19019a5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:41.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.544+0000 7fa195a00640 1 --2- 172.21.15.90:0/529601369 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa190104f30 0x7fa19019a5c0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fa17c00e9f0 tx=0x7fa17c00edf0 comp rx=0 tx=0).stop 2024-04-02T14:34:41.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.544+0000 7fa195a00640 1 -- 172.21.15.90:0/529601369 shutdown_connections 2024-04-02T14:34:41.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.544+0000 7fa195a00640 1 --2- 172.21.15.90:0/529601369 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fa1740a3b60 0x7fa1740a6020 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:41.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.544+0000 7fa195a00640 1 --2- 172.21.15.90:0/529601369 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa1901058f0 0x7fa1901a1630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:41.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.544+0000 7fa195a00640 1 --2- 172.21.15.90:0/529601369 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa190104f30 0x7fa19019a5c0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:41.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.544+0000 7fa195a00640 1 --2- 172.21.15.90:0/529601369 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa190103d40 0x7fa19019a080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:41.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.544+0000 7fa195a00640 1 -- 172.21.15.90:0/529601369 >> 172.21.15.90:0/529601369 conn(0x7fa1900ff2b0 msgr2=0x7fa190108300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:34:41.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.545+0000 7fa195a00640 1 -- 172.21.15.90:0/529601369 shutdown_connections 2024-04-02T14:34:41.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.545+0000 7fa195a00640 1 -- 172.21.15.90:0/529601369 wait complete. 2024-04-02T14:34:41.564 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2024-04-02T14:34:41.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.768+0000 7fcf44200640 1 -- 172.21.15.90:0/1339560884 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcf3c103d40 msgr2=0x7fcf3c104160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:41.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.768+0000 7fcf44200640 1 --2- 172.21.15.90:0/1339560884 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcf3c103d40 0x7fcf3c104160 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fcf30009ca0 tx=0x7fcf3002c100 comp rx=0 tx=0).stop 2024-04-02T14:34:41.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.769+0000 7fcf44200640 1 -- 172.21.15.90:0/1339560884 shutdown_connections 2024-04-02T14:34:41.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.769+0000 7fcf44200640 1 --2- 172.21.15.90:0/1339560884 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcf3c1058f0 0x7fcf3c10c180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:41.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.769+0000 7fcf44200640 1 --2- 172.21.15.90:0/1339560884 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcf3c104f30 0x7fcf3c1053b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:41.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.769+0000 7fcf44200640 1 --2- 172.21.15.90:0/1339560884 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcf3c103d40 0x7fcf3c104160 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:41.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.769+0000 7fcf44200640 1 -- 172.21.15.90:0/1339560884 >> 172.21.15.90:0/1339560884 conn(0x7fcf3c0ff2b0 msgr2=0x7fcf3c1016f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:34:41.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.769+0000 7fcf44200640 1 -- 172.21.15.90:0/1339560884 shutdown_connections 2024-04-02T14:34:41.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.769+0000 7fcf44200640 1 -- 172.21.15.90:0/1339560884 wait complete. 2024-04-02T14:34:41.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.770+0000 7fcf44200640 1 Processor -- start 2024-04-02T14:34:41.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.770+0000 7fcf44200640 1 -- start start 2024-04-02T14:34:41.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.770+0000 7fcf44200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcf3c103d40 0x7fcf3c19a050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:41.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.770+0000 7fcf44200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcf3c104f30 0x7fcf3c19a590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:41.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.770+0000 7fcf44200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcf3c1058f0 0x7fcf3c1a1600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:41.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.770+0000 7fcf44200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fcf3c10f9c0 con 0x7fcf3c1058f0 2024-04-02T14:34:41.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.770+0000 7fcf44200640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fcf3c10f840 con 0x7fcf3c104f30 2024-04-02T14:34:41.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.770+0000 7fcf44200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fcf3c10fb40 con 0x7fcf3c103d40 2024-04-02T14:34:41.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.770+0000 7fcf41800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcf3c103d40 0x7fcf3c19a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:41.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.771+0000 7fcf42200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcf3c1058f0 0x7fcf3c1a1600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:41.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.771+0000 7fcf41800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcf3c103d40 0x7fcf3c19a050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:52964/0 (socket says 172.21.15.90:52964) 2024-04-02T14:34:41.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.771+0000 7fcf41800640 1 -- 172.21.15.90:0/1190211963 learned_addr learned my addr 172.21.15.90:0/1190211963 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:34:41.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.771+0000 7fcf40e00640 1 --2- 172.21.15.90:0/1190211963 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcf3c104f30 0x7fcf3c19a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:41.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.771+0000 7fcf41800640 1 -- 172.21.15.90:0/1190211963 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcf3c104f30 msgr2=0x7fcf3c19a590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:41.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.771+0000 7fcf41800640 1 --2- 172.21.15.90:0/1190211963 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcf3c104f30 0x7fcf3c19a590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:41.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.771+0000 7fcf41800640 1 -- 172.21.15.90:0/1190211963 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcf3c1058f0 msgr2=0x7fcf3c1a1600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:41.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.771+0000 7fcf41800640 1 --2- 172.21.15.90:0/1190211963 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcf3c1058f0 0x7fcf3c1a1600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:41.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.771+0000 7fcf41800640 1 -- 172.21.15.90:0/1190211963 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf3c1a1c70 con 0x7fcf3c103d40 2024-04-02T14:34:41.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.771+0000 7fcf41800640 1 --2- 172.21.15.90:0/1190211963 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcf3c103d40 0x7fcf3c19a050 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fcf300324c0 tx=0x7fcf30035d90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:34:41.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.771+0000 7fcf3a000640 1 -- 172.21.15.90:0/1190211963 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf30039040 con 0x7fcf3c103d40 2024-04-02T14:34:41.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.771+0000 7fcf3a000640 1 -- 172.21.15.90:0/1190211963 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcf30044070 con 0x7fcf3c103d40 2024-04-02T14:34:41.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.772+0000 7fcf44200640 1 -- 172.21.15.90:0/1190211963 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcf3c1a1f00 con 0x7fcf3c103d40 2024-04-02T14:34:41.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.772+0000 7fcf3a000640 1 -- 172.21.15.90:0/1190211963 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf300044a0 con 0x7fcf3c103d40 2024-04-02T14:34:41.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.772+0000 7fcf44200640 1 -- 172.21.15.90:0/1190211963 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcf3c1a2300 con 0x7fcf3c103d40 2024-04-02T14:34:41.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.775+0000 7fcf46ff9740 1 Processor -- start 2024-04-02T14:34:41.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.775+0000 7fcf3a000640 1 -- 172.21.15.90:0/1190211963 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fcf30004640 con 0x7fcf3c103d40 2024-04-02T14:34:41.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.775+0000 7fcf46ff9740 1 -- start start 2024-04-02T14:34:41.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.775+0000 7fcf46ff9740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556c828c88f0 0x556c828c8d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:41.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.775+0000 7fcf46ff9740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556c828c92e0 0x556c828cdde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:41.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.775+0000 7fcf46ff9740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556c828ce450 0x556c828d08e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:41.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.775+0000 7fcf46ff9740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x556c8282e060 con 0x556c828c92e0 2024-04-02T14:34:41.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.775+0000 7fcf46ff9740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x556c827d6560 con 0x556c828ce450 2024-04-02T14:34:41.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.775+0000 7fcf46ff9740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x556c8282a010 con 0x556c828c88f0 2024-04-02T14:34:41.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.775+0000 7fcf41800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556c828c92e0 0x556c828cdde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:41.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.775+0000 7fcf40e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556c828ce450 0x556c828d08e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:41.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.775+0000 7fcf41800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556c828c92e0 0x556c828cdde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:36568/0 (socket says 172.21.15.90:36568) 2024-04-02T14:34:41.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.776+0000 7fcf41800640 1 -- 172.21.15.90:0/828607981 learned_addr learned my addr 172.21.15.90:0/828607981 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:34:41.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.776+0000 7fcf42200640 1 --2- 172.21.15.90:0/828607981 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556c828c88f0 0x556c828c8d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:41.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.776+0000 7fcf40e00640 1 -- 172.21.15.90:0/828607981 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556c828c88f0 msgr2=0x556c828c8d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:34:41.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.776+0000 7fcf40e00640 1 --2- 172.21.15.90:0/828607981 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556c828c88f0 0x556c828c8d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:41.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.776+0000 7fcf40e00640 1 -- 172.21.15.90:0/828607981 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556c828c92e0 msgr2=0x556c828cdde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:34:41.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.776+0000 7fcf40e00640 1 --2- 172.21.15.90:0/828607981 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556c828c92e0 0x556c828cdde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:41.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.776+0000 7fcf40e00640 1 -- 172.21.15.90:0/828607981 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556c827f8fd0 con 0x556c828ce450 2024-04-02T14:34:41.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.776+0000 7fcf41800640 1 --2- 172.21.15.90:0/828607981 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556c828c92e0 0x556c828cdde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:34:41.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.776+0000 7fcf40e00640 1 --2- 172.21.15.90:0/828607981 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556c828ce450 0x556c828d08e0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fcf28009bf0 tx=0x7fcf28011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8a02a75340663019 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:34:41.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.776+0000 7fcf2ea00640 1 -- 172.21.15.90:0/828607981 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf2801e070 con 0x556c828ce450 2024-04-02T14:34:41.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.776+0000 7fcf2ea00640 1 -- 172.21.15.90:0/828607981 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcf28009da0 con 0x556c828ce450 2024-04-02T14:34:41.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.776+0000 7fcf2ea00640 1 -- 172.21.15.90:0/828607981 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf28018870 con 0x556c828ce450 2024-04-02T14:34:41.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.777+0000 7fcf3a000640 1 --2- 172.21.15.90:0/1190211963 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fcf240a3bb0 0x7fcf240a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:41.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.777+0000 7fcf3a000640 1 -- 172.21.15.90:0/1190211963 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8248+0+0 (secure 0 0 0) 0x7fcf300f2ab0 con 0x7fcf3c103d40 2024-04-02T14:34:41.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.777+0000 7fcf42200640 1 --2- 172.21.15.90:0/1190211963 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fcf240a3bb0 0x7fcf240a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:41.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.777+0000 7fcf46ff9740 1 -- 172.21.15.90:0/828607981 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556c828ce450 msgr2=0x556c828d08e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:41.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.777+0000 7fcf46ff9740 1 --2- 172.21.15.90:0/828607981 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556c828ce450 0x556c828d08e0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fcf28009bf0 tx=0x7fcf28011810 comp rx=0 tx=0).stop 2024-04-02T14:34:41.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.777+0000 7fcf46ff9740 1 -- 172.21.15.90:0/828607981 shutdown_connections 2024-04-02T14:34:41.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.777+0000 7fcf46ff9740 1 --2- 172.21.15.90:0/828607981 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556c828ce450 0x556c828d08e0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:41.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.777+0000 7fcf46ff9740 1 --2- 172.21.15.90:0/828607981 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556c828c92e0 0x556c828cdde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:41.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.777+0000 7fcf46ff9740 1 --2- 172.21.15.90:0/828607981 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556c828c88f0 0x556c828c8d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:41.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.777+0000 7fcf46ff9740 1 -- 172.21.15.90:0/828607981 >> 172.21.15.90:0/828607981 conn(0x556c828c3e60 msgr2=0x556c828c4260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:34:41.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.777+0000 7fcf46ff9740 1 -- 172.21.15.90:0/828607981 shutdown_connections 2024-04-02T14:34:41.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.777+0000 7fcf42200640 1 --2- 172.21.15.90:0/1190211963 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fcf240a3bb0 0x7fcf240a6070 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x556c828c9b40 tx=0x7fcf3400b2e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:34:41.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.777+0000 7fcf46ff9740 1 -- 172.21.15.90:0/828607981 wait complete. 2024-04-02T14:34:41.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.778+0000 7fcf46ff9740 1 -- start start 2024-04-02T14:34:41.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.778+0000 7fcf46ff9740 1 Processor -- start 2024-04-02T14:34:41.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.779+0000 7fcf46ff9740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556c828c88f0 0x556c828d2e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:41.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.779+0000 7fcf46ff9740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556c828c92e0 0x556c828d33b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:41.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.779+0000 7fcf46ff9740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556c828ce450 0x556c828da380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:41.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.779+0000 7fcf46ff9740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x556c828d1400 con 0x556c828c92e0 2024-04-02T14:34:41.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.779+0000 7fcf46ff9740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x556c827e9920 con 0x556c828ce450 2024-04-02T14:34:41.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.779+0000 7fcf46ff9740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x556c828d1580 con 0x556c828c88f0 2024-04-02T14:34:41.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.779+0000 7fcf46ff9740 10 client.0 _ll_register_callbacks cb 0x556c827e9310 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:34:41.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.779+0000 7fcf41800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556c828c88f0 0x556c828d2e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:41.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.779+0000 7fcf40e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556c828c92e0 0x556c828d33b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:41.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.779+0000 7fcf42200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556c828ce450 0x556c828da380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:41.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.779+0000 7fcf40e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556c828c92e0 0x556c828d33b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:36572/0 (socket says 172.21.15.90:36572) 2024-04-02T14:34:41.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.779+0000 7fcf40e00640 1 -- 172.21.15.90:0/4033847163 learned_addr learned my addr 172.21.15.90:0/4033847163 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:34:41.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.779+0000 7fcf40e00640 1 -- 172.21.15.90:0/4033847163 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556c828c88f0 msgr2=0x556c828d2e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:34:41.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.779+0000 7fcf40e00640 1 --2- 172.21.15.90:0/4033847163 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556c828c88f0 0x556c828d2e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:41.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.779+0000 7fcf40e00640 1 -- 172.21.15.90:0/4033847163 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556c828ce450 msgr2=0x556c828da380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:34:41.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.779+0000 7fcf40e00640 1 --2- 172.21.15.90:0/4033847163 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556c828ce450 0x556c828da380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:41.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.779+0000 7fcf40e00640 1 -- 172.21.15.90:0/4033847163 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556c825422d0 con 0x556c828c92e0 2024-04-02T14:34:41.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.780+0000 7fcf40e00640 1 --2- 172.21.15.90:0/4033847163 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556c828c92e0 0x556c828d33b0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fcf2801c040 tx=0x7fcf28019eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14d15d32a7361f1a server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:34:41.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.780+0000 7fcf23400640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-02T14:34:41.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.780+0000 7fcf23400640 1 -- 172.21.15.90:0/4033847163 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf2801e040 con 0x556c828c92e0 2024-04-02T14:34:41.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.780+0000 7fcf46ff9740 1 -- 172.21.15.90:0/4033847163 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcf28009da0 con 0x556c828c92e0 2024-04-02T14:34:41.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.780+0000 7fcf23400640 1 -- 172.21.15.90:0/4033847163 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8248+0+0 (secure 0 0 0) 0x7fcf280050c0 con 0x556c828c92e0 2024-04-02T14:34:41.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.780+0000 7fcf23400640 1 -- 172.21.15.90:0/4033847163 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fcf280052c0 con 0x556c828c92e0 2024-04-02T14:34:41.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.780+0000 7fcf46ff9740 10 client.15120 fetch_fsmap learned FSMap version 26 2024-04-02T14:34:41.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.780+0000 7fcf46ff9740 1 -- 172.21.15.90:0/4033847163 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x556c8260c6b0 con 0x556c828c92e0 2024-04-02T14:34:41.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.780+0000 7fcf23400640 1 -- 172.21.15.90:0/4033847163 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcf28009da0 con 0x556c828c92e0 2024-04-02T14:34:41.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.781+0000 7fcf23400640 1 -- 172.21.15.90:0/4033847163 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7fcf280195b0 con 0x556c828c92e0 2024-04-02T14:34:41.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.781+0000 7fcf46ff9740 10 client.15120 fetch_fsmap finished waiting for FSMap version 26 2024-04-02T14:34:41.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.781+0000 7fcf46ff9740 10 client.15120 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-02T14:34:41.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.781+0000 7fcf46ff9740 20 client.15120 populate_metadata read hostname 'smithi090' 2024-04-02T14:34:41.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.781+0000 7fcf46ff9740 1 --2- 172.21.15.90:0/4033847163 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x556c828df9a0 0x556c828e0e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:41.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.781+0000 7fcf46ff9740 4 client.15120 mds_command: new command op to 24547 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:34:41.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.781+0000 7fcf46ff9740 1 -- 172.21.15.90:0/4033847163 --> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556c8283b5a0 con 0x556c828df9a0 2024-04-02T14:34:41.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.781+0000 7fcf41800640 1 --2- 172.21.15.90:0/4033847163 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x556c828df9a0 0x556c828e0e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:41.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.782+0000 7fcf41800640 1 --2- 172.21.15.90:0/4033847163 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x556c828df9a0 0x556c828e0e10 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=a271986b95d7c564 server_cookie=c74269ce955a9e12 in_seq=0 out_seq=0 2024-04-02T14:34:41.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.782+0000 7fcf23400640 10 client.15120 ms_handle_connect on v2:172.21.15.177:6838/664895663 2024-04-02T14:34:41.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.782+0000 7fcf23400640 1 -- 172.21.15.90:0/4033847163 <== mds.2 v2:172.21.15.177:6838/664895663 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x556c8283b5a0 con 0x556c828df9a0 2024-04-02T14:34:41.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.782+0000 7fcf23400640 10 client.15120 handle_command_reply: tid=0 2024-04-02T14:34:41.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.782+0000 7fcf46ff9740 1 client.15120 shutdown 2024-04-02T14:34:41.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.783+0000 7fcf46ff9740 1 -- 172.21.15.90:0/4033847163 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556c828c92e0 msgr2=0x556c828d33b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:41.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.783+0000 7fcf46ff9740 1 --2- 172.21.15.90:0/4033847163 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556c828c92e0 0x556c828d33b0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fcf2801c040 tx=0x7fcf28019eb0 comp rx=0 tx=0).stop 2024-04-02T14:34:41.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.783+0000 7fcf46ff9740 1 -- 172.21.15.90:0/4033847163 shutdown_connections 2024-04-02T14:34:41.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.783+0000 7fcf46ff9740 1 --2- 172.21.15.90:0/4033847163 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x556c828df9a0 0x556c828e0e10 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:41.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.783+0000 7fcf46ff9740 1 --2- 172.21.15.90:0/4033847163 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556c828ce450 0x556c828da380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:41.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.783+0000 7fcf46ff9740 1 --2- 172.21.15.90:0/4033847163 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556c828c92e0 0x556c828d33b0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:41.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.783+0000 7fcf23400640 0 client.15120 ms_handle_reset on v2:172.21.15.177:6838/664895663 2024-04-02T14:34:41.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.783+0000 7fcf46ff9740 1 --2- 172.21.15.90:0/4033847163 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556c828c88f0 0x556c828d2e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:41.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.783+0000 7fcf46ff9740 1 -- 172.21.15.90:0/4033847163 >> 172.21.15.90:0/4033847163 conn(0x556c828c3e60 msgr2=0x556c828c3080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:34:41.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.783+0000 7fcf46ff9740 1 -- 172.21.15.90:0/4033847163 shutdown_connections 2024-04-02T14:34:41.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.783+0000 7fcf46ff9740 1 -- 172.21.15.90:0/4033847163 wait complete. 2024-04-02T14:34:41.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.783+0000 7fcf46ff9740 20 client.15120 trim_cache size 0 max 16384 2024-04-02T14:34:41.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.802+0000 7fcf46ff9740 1 Processor -- start 2024-04-02T14:34:41.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.803+0000 7fcf46ff9740 1 -- start start 2024-04-02T14:34:41.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.803+0000 7fcf46ff9740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556c828c88f0 0x556c828d0840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:41.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.803+0000 7fcf46ff9740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556c828c92e0 0x556c828da340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:41.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.803+0000 7fcf46ff9740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556c828ce450 0x556c828de970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:41.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.803+0000 7fcf46ff9740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x556c828df150 con 0x556c828c88f0 2024-04-02T14:34:41.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.803+0000 7fcf46ff9740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x556c828df820 con 0x556c828c92e0 2024-04-02T14:34:41.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.803+0000 7fcf46ff9740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x556c828dfef0 con 0x556c828ce450 2024-04-02T14:34:41.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.803+0000 7fcf41800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556c828c88f0 0x556c828d0840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:41.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.803+0000 7fcf41800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556c828c88f0 0x556c828d0840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:36574/0 (socket says 172.21.15.90:36574) 2024-04-02T14:34:41.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.803+0000 7fcf41800640 1 -- 172.21.15.90:0/2661361765 learned_addr learned my addr 172.21.15.90:0/2661361765 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:34:41.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.803+0000 7fcf40e00640 1 --2- 172.21.15.90:0/2661361765 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556c828c92e0 0x556c828da340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:41.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.803+0000 7fcf41800640 1 -- 172.21.15.90:0/2661361765 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556c828ce450 msgr2=0x556c828de970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:34:41.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.803+0000 7fcf42200640 1 --2- 172.21.15.90:0/2661361765 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556c828ce450 0x556c828de970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:41.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.803+0000 7fcf41800640 1 --2- 172.21.15.90:0/2661361765 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556c828ce450 0x556c828de970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:41.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.803+0000 7fcf41800640 1 -- 172.21.15.90:0/2661361765 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556c828c92e0 msgr2=0x556c828da340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:34:41.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.803+0000 7fcf41800640 1 --2- 172.21.15.90:0/2661361765 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556c828c92e0 0x556c828da340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:41.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.803+0000 7fcf41800640 1 -- 172.21.15.90:0/2661361765 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556c828c9b70 con 0x556c828c88f0 2024-04-02T14:34:41.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.803+0000 7fcf40e00640 1 --2- 172.21.15.90:0/2661361765 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556c828c92e0 0x556c828da340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:34:41.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.804+0000 7fcf42200640 1 --2- 172.21.15.90:0/2661361765 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556c828ce450 0x556c828de970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:34:41.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.804+0000 7fcf41800640 1 --2- 172.21.15.90:0/2661361765 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556c828c88f0 0x556c828d0840 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fcf300bb6f0 tx=0x7fcf300f64f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4cbc966caf7e562c server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:34:41.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.804+0000 7fcf2e000640 1 -- 172.21.15.90:0/2661361765 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf300beaf0 con 0x556c828c88f0 2024-04-02T14:34:41.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.804+0000 7fcf2e000640 1 -- 172.21.15.90:0/2661361765 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcf300bf420 con 0x556c828c88f0 2024-04-02T14:34:41.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.804+0000 7fcf2e000640 1 -- 172.21.15.90:0/2661361765 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf300be680 con 0x556c828c88f0 2024-04-02T14:34:41.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.804+0000 7fcf46ff9740 1 -- 172.21.15.90:0/2661361765 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556c828c88f0 msgr2=0x556c828d0840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:41.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.804+0000 7fcf46ff9740 1 --2- 172.21.15.90:0/2661361765 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556c828c88f0 0x556c828d0840 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fcf300bb6f0 tx=0x7fcf300f64f0 comp rx=0 tx=0).stop 2024-04-02T14:34:41.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.805+0000 7fcf46ff9740 1 -- 172.21.15.90:0/2661361765 shutdown_connections 2024-04-02T14:34:41.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.805+0000 7fcf46ff9740 1 --2- 172.21.15.90:0/2661361765 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556c828ce450 0x556c828de970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:41.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.805+0000 7fcf46ff9740 1 --2- 172.21.15.90:0/2661361765 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556c828c92e0 0x556c828da340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:41.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.805+0000 7fcf46ff9740 1 --2- 172.21.15.90:0/2661361765 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556c828c88f0 0x556c828d0840 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:41.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.805+0000 7fcf46ff9740 1 -- 172.21.15.90:0/2661361765 >> 172.21.15.90:0/2661361765 conn(0x556c828c3e60 msgr2=0x556c828c9d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:34:41.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.805+0000 7fcf46ff9740 1 -- 172.21.15.90:0/2661361765 shutdown_connections 2024-04-02T14:34:41.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.805+0000 7fcf46ff9740 1 -- 172.21.15.90:0/2661361765 wait complete. 2024-04-02T14:34:41.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.805+0000 7fcf46ff9740 1 -- start start 2024-04-02T14:34:41.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.805+0000 7fcf46ff9740 1 Processor -- start 2024-04-02T14:34:41.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.806+0000 7fcf46ff9740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556c828c88f0 0x556c828cc650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:41.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.806+0000 7fcf46ff9740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556c828c92e0 0x556c828ccb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:41.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.806+0000 7fcf46ff9740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556c828ce450 0x556c82943940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:41.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.806+0000 7fcf46ff9740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x556c828e0c90 con 0x556c828ce450 2024-04-02T14:34:41.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.806+0000 7fcf46ff9740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x556c828e05c0 con 0x556c828c88f0 2024-04-02T14:34:41.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.806+0000 7fcf46ff9740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x556c828e1360 con 0x556c828c92e0 2024-04-02T14:34:41.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.806+0000 7fcf46ff9740 10 client.0 _ll_register_callbacks cb 0x7fcf28009a70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:34:41.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.806+0000 7fcf42200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556c828ce450 0x556c82943940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:41.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.806+0000 7fcf42200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556c828ce450 0x556c82943940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:36588/0 (socket says 172.21.15.90:36588) 2024-04-02T14:34:41.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.806+0000 7fcf42200640 1 -- 172.21.15.90:0/3931359984 learned_addr learned my addr 172.21.15.90:0/3931359984 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:34:41.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.806+0000 7fcf41800640 1 --2- 172.21.15.90:0/3931359984 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556c828c88f0 0x556c828cc650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:41.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.806+0000 7fcf40e00640 1 --2- 172.21.15.90:0/3931359984 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556c828c92e0 0x556c828ccb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:41.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.806+0000 7fcf41800640 1 -- 172.21.15.90:0/3931359984 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556c828c92e0 msgr2=0x556c828ccb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:34:41.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.806+0000 7fcf41800640 1 --2- 172.21.15.90:0/3931359984 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556c828c92e0 0x556c828ccb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:41.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.806+0000 7fcf41800640 1 -- 172.21.15.90:0/3931359984 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556c828ce450 msgr2=0x556c82943940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:34:41.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.806+0000 7fcf41800640 1 --2- 172.21.15.90:0/3931359984 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556c828ce450 0x556c82943940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:41.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.806+0000 7fcf41800640 1 -- 172.21.15.90:0/3931359984 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556c82943e80 con 0x556c828c88f0 2024-04-02T14:34:41.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.806+0000 7fcf42200640 1 --2- 172.21.15.90:0/3931359984 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556c828ce450 0x556c82943940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:34:41.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.807+0000 7fcf41800640 1 --2- 172.21.15.90:0/3931359984 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556c828c88f0 0x556c828cc650 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x556c828dd670 tx=0x7fcf30009870 comp rx=0 tx=0).ready entity=mon.2 client_cookie=78d3fc38e759d7a9 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:34:41.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.807+0000 7fcf23400640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:34:41.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.807+0000 7fcf23400640 1 -- 172.21.15.90:0/3931359984 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf300d8060 con 0x556c828c88f0 2024-04-02T14:34:41.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.807+0000 7fcf46ff9740 1 -- 172.21.15.90:0/3931359984 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcf300bf420 con 0x556c828c88f0 2024-04-02T14:34:41.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.807+0000 7fcf23400640 1 -- 172.21.15.90:0/3931359984 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8248+0+0 (secure 0 0 0) 0x7fcf30032890 con 0x556c828c88f0 2024-04-02T14:34:41.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.807+0000 7fcf23400640 1 -- 172.21.15.90:0/3931359984 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fcf300be860 con 0x556c828c88f0 2024-04-02T14:34:41.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.807+0000 7fcf46ff9740 10 client.25232 fetch_fsmap learned FSMap version 26 2024-04-02T14:34:41.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.807+0000 7fcf46ff9740 1 -- 172.21.15.90:0/3931359984 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x556c82944fe0 con 0x556c828c88f0 2024-04-02T14:34:41.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.807+0000 7fcf23400640 1 -- 172.21.15.90:0/3931359984 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcf300bf420 con 0x556c828c88f0 2024-04-02T14:34:41.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.808+0000 7fcf23400640 1 -- 172.21.15.90:0/3931359984 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7fcf3003c030 con 0x556c828c88f0 2024-04-02T14:34:41.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.808+0000 7fcf46ff9740 10 client.25232 fetch_fsmap finished waiting for FSMap version 26 2024-04-02T14:34:41.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.808+0000 7fcf46ff9740 10 client.25232 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-02T14:34:41.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.808+0000 7fcf46ff9740 20 client.25232 populate_metadata read hostname 'smithi090' 2024-04-02T14:34:41.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.808+0000 7fcf46ff9740 1 --2- 172.21.15.90:0/3931359984 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x556c82949a00 0x556c82945e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:41.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.808+0000 7fcf46ff9740 4 client.25232 mds_command: new command op to 24547 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-02T14:34:41.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.808+0000 7fcf46ff9740 1 -- 172.21.15.90:0/3931359984 --> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] -- command(tid 0: {"prefix": "status"}) -- 0x556c828358f0 con 0x556c82949a00 2024-04-02T14:34:41.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.808+0000 7fcf40e00640 1 --2- 172.21.15.90:0/3931359984 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x556c82949a00 0x556c82945e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:41.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.809+0000 7fcf23400640 10 client.25232 ms_handle_connect on v2:172.21.15.177:6838/664895663 2024-04-02T14:34:41.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.809+0000 7fcf40e00640 1 --2- 172.21.15.90:0/3931359984 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x556c82949a00 0x556c82945e40 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=d4af279a1be8ab2a server_cookie=d3260ba700f93c59 in_seq=0 out_seq=0 2024-04-02T14:34:41.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.809+0000 7fcf23400640 1 -- 172.21.15.90:0/3931359984 <== mds.2 v2:172.21.15.177:6838/664895663 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x556c828358f0 con 0x556c82949a00 2024-04-02T14:34:41.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.809+0000 7fcf23400640 10 client.25232 handle_command_reply: tid=0 2024-04-02T14:34:41.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.809+0000 7fcf46ff9740 1 client.25232 shutdown 2024-04-02T14:34:41.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.809+0000 7fcf46ff9740 1 -- 172.21.15.90:0/3931359984 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556c828c88f0 msgr2=0x556c828cc650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:41.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.809+0000 7fcf46ff9740 1 --2- 172.21.15.90:0/3931359984 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556c828c88f0 0x556c828cc650 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x556c828dd670 tx=0x7fcf30009870 comp rx=0 tx=0).stop 2024-04-02T14:34:41.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.809+0000 7fcf46ff9740 1 -- 172.21.15.90:0/3931359984 shutdown_connections 2024-04-02T14:34:41.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.809+0000 7fcf46ff9740 1 --2- 172.21.15.90:0/3931359984 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x556c82949a00 0x556c82945e40 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:41.817 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.809+0000 7fcf46ff9740 1 --2- 172.21.15.90:0/3931359984 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556c828ce450 0x556c82943940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:41.817 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.809+0000 7fcf46ff9740 1 --2- 172.21.15.90:0/3931359984 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556c828c92e0 0x556c828ccb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:41.817 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.809+0000 7fcf23400640 0 client.25232 ms_handle_reset on v2:172.21.15.177:6838/664895663 2024-04-02T14:34:41.817 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.809+0000 7fcf46ff9740 1 --2- 172.21.15.90:0/3931359984 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556c828c88f0 0x556c828cc650 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:41.817 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.809+0000 7fcf46ff9740 1 -- 172.21.15.90:0/3931359984 >> 172.21.15.90:0/3931359984 conn(0x556c828c3e60 msgr2=0x556c828c4240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:34:41.817 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.810+0000 7fcf46ff9740 1 -- 172.21.15.90:0/3931359984 shutdown_connections 2024-04-02T14:34:41.817 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.810+0000 7fcf46ff9740 1 -- 172.21.15.90:0/3931359984 wait complete. 2024-04-02T14:34:41.817 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.810+0000 7fcf46ff9740 20 client.25232 trim_cache size 0 max 16384 2024-04-02T14:34:41.817 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-02T14:34:41.818 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_fsid": "6449448a-f0fc-11ee-b647-cb9ed24678a4", 2024-04-02T14:34:41.818 INFO:teuthology.orchestra.run.smithi090.stdout: "whoami": 2, 2024-04-02T14:34:41.818 INFO:teuthology.orchestra.run.smithi090.stdout: "id": 24547, 2024-04-02T14:34:41.818 INFO:teuthology.orchestra.run.smithi090.stdout: "want_state": "up:active", 2024-04-02T14:34:41.818 INFO:teuthology.orchestra.run.smithi090.stdout: "state": "up:active", 2024-04-02T14:34:41.818 INFO:teuthology.orchestra.run.smithi090.stdout: "fs_name": "cephfs", 2024-04-02T14:34:41.818 INFO:teuthology.orchestra.run.smithi090.stdout: "rank_uptime": 134.887204403, 2024-04-02T14:34:41.818 INFO:teuthology.orchestra.run.smithi090.stdout: "mdsmap_epoch": 26, 2024-04-02T14:34:41.818 INFO:teuthology.orchestra.run.smithi090.stdout: "osdmap_epoch": 90, 2024-04-02T14:34:41.818 INFO:teuthology.orchestra.run.smithi090.stdout: "osdmap_epoch_barrier": 88, 2024-04-02T14:34:41.818 INFO:teuthology.orchestra.run.smithi090.stdout: "uptime": 155.60011947199999 2024-04-02T14:34:41.819 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-02T14:34:41.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.810+0000 7fcf44200640 1 -- 172.21.15.90:0/1190211963 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fcf240a3bb0 msgr2=0x7fcf240a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:41.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.810+0000 7fcf44200640 1 --2- 172.21.15.90:0/1190211963 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fcf240a3bb0 0x7fcf240a6070 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x556c828c9b40 tx=0x7fcf3400b2e0 comp rx=0 tx=0).stop 2024-04-02T14:34:41.820 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.810+0000 7fcf44200640 1 -- 172.21.15.90:0/1190211963 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcf3c103d40 msgr2=0x7fcf3c19a050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:41.820 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.810+0000 7fcf44200640 1 --2- 172.21.15.90:0/1190211963 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcf3c103d40 0x7fcf3c19a050 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fcf300324c0 tx=0x7fcf30035d90 comp rx=0 tx=0).stop 2024-04-02T14:34:41.820 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.810+0000 7fcf44200640 1 -- 172.21.15.90:0/1190211963 shutdown_connections 2024-04-02T14:34:41.820 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.810+0000 7fcf44200640 1 --2- 172.21.15.90:0/1190211963 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fcf240a3bb0 0x7fcf240a6070 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:41.820 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.810+0000 7fcf44200640 1 --2- 172.21.15.90:0/1190211963 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcf3c1058f0 0x7fcf3c1a1600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:41.820 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.810+0000 7fcf44200640 1 --2- 172.21.15.90:0/1190211963 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcf3c104f30 0x7fcf3c19a590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:41.820 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.810+0000 7fcf44200640 1 --2- 172.21.15.90:0/1190211963 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcf3c103d40 0x7fcf3c19a050 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:41.820 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.810+0000 7fcf44200640 1 -- 172.21.15.90:0/1190211963 >> 172.21.15.90:0/1190211963 conn(0x7fcf3c0ff2b0 msgr2=0x7fcf3c108300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:34:41.820 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.811+0000 7fcf44200640 1 -- 172.21.15.90:0/1190211963 shutdown_connections 2024-04-02T14:34:41.820 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:41.811+0000 7fcf44200640 1 -- 172.21.15.90:0/1190211963 wait complete. 2024-04-02T14:34:41.830 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.g status 2024-04-02T14:34:42.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.037+0000 7f1478000640 1 -- 172.21.15.90:0/2828172839 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1470109310 msgr2=0x7f1470109790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:42.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.037+0000 7f1478000640 1 --2- 172.21.15.90:0/2828172839 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1470109310 0x7f1470109790 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f145c009860 tx=0x7f145c02bb70 comp rx=0 tx=0).stop 2024-04-02T14:34:42.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.037+0000 7f1478000640 1 -- 172.21.15.90:0/2828172839 shutdown_connections 2024-04-02T14:34:42.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.037+0000 7f1478000640 1 --2- 172.21.15.90:0/2828172839 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1470109cd0 0x7f1470110560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:42.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.037+0000 7f1478000640 1 --2- 172.21.15.90:0/2828172839 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1470109310 0x7f1470109790 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:42.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.037+0000 7f1478000640 1 --2- 172.21.15.90:0/2828172839 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1470108120 0x7f1470108540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:42.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.037+0000 7f1478000640 1 -- 172.21.15.90:0/2828172839 >> 172.21.15.90:0/2828172839 conn(0x7f14700770c0 msgr2=0x7f1470079500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:34:42.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.037+0000 7f1478000640 1 -- 172.21.15.90:0/2828172839 shutdown_connections 2024-04-02T14:34:42.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.037+0000 7f1478000640 1 -- 172.21.15.90:0/2828172839 wait complete. 2024-04-02T14:34:42.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.038+0000 7f1478000640 1 Processor -- start 2024-04-02T14:34:42.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.038+0000 7f1478000640 1 -- start start 2024-04-02T14:34:42.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.039+0000 7f1478000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1470108120 0x7f147019e450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:42.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.039+0000 7f1478000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1470109310 0x7f147019e990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:42.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.039+0000 7f1478000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1470109cd0 0x7f14701a5a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:42.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.039+0000 7f1478000640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f1470113db0 con 0x7f1470109310 2024-04-02T14:34:42.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.039+0000 7f1478000640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f1470113c30 con 0x7f1470109cd0 2024-04-02T14:34:42.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.039+0000 7f1478000640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f1470113f30 con 0x7f1470108120 2024-04-02T14:34:42.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.039+0000 7f1474c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1470109310 0x7f147019e990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:42.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.039+0000 7f1475600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1470108120 0x7f147019e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:42.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.039+0000 7f1474c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1470109310 0x7f147019e990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:36606/0 (socket says 172.21.15.90:36606) 2024-04-02T14:34:42.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.039+0000 7f1474c00640 1 -- 172.21.15.90:0/2605790048 learned_addr learned my addr 172.21.15.90:0/2605790048 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:34:42.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.039+0000 7f1476000640 1 --2- 172.21.15.90:0/2605790048 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1470109cd0 0x7f14701a5a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:42.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.039+0000 7f1474c00640 1 -- 172.21.15.90:0/2605790048 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1470108120 msgr2=0x7f147019e450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:42.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.039+0000 7f1474c00640 1 --2- 172.21.15.90:0/2605790048 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1470108120 0x7f147019e450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:42.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.039+0000 7f1474c00640 1 -- 172.21.15.90:0/2605790048 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1470109cd0 msgr2=0x7f14701a5a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:42.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.039+0000 7f1474c00640 1 --2- 172.21.15.90:0/2605790048 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1470109cd0 0x7f14701a5a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:42.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.039+0000 7f1474c00640 1 -- 172.21.15.90:0/2605790048 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14701a6070 con 0x7f1470109310 2024-04-02T14:34:42.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.040+0000 7f1474c00640 1 --2- 172.21.15.90:0/2605790048 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1470109310 0x7f147019e990 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f145c005940 tx=0x7f145c032470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:34:42.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.040+0000 7f146e000640 1 -- 172.21.15.90:0/2605790048 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f145c041020 con 0x7f1470109310 2024-04-02T14:34:42.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.040+0000 7f146e000640 1 -- 172.21.15.90:0/2605790048 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f145c02c9f0 con 0x7f1470109310 2024-04-02T14:34:42.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.040+0000 7f1478000640 1 -- 172.21.15.90:0/2605790048 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14701a6300 con 0x7f1470109310 2024-04-02T14:34:42.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.040+0000 7f146e000640 1 -- 172.21.15.90:0/2605790048 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f145c039040 con 0x7f1470109310 2024-04-02T14:34:42.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.040+0000 7f1478000640 1 -- 172.21.15.90:0/2605790048 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14701a67f0 con 0x7f1470109310 2024-04-02T14:34:42.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.043+0000 7f147adac740 1 Processor -- start 2024-04-02T14:34:42.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.043+0000 7f146e000640 1 -- 172.21.15.90:0/2605790048 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f145c044020 con 0x7f1470109310 2024-04-02T14:34:42.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.043+0000 7f147adac740 1 -- start start 2024-04-02T14:34:42.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.044+0000 7f147adac740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557ee3e298f0 0x557ee3e29d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:42.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.044+0000 7f147adac740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ee3e2a2e0 0x557ee3e2ede0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:42.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.044+0000 7f147adac740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557ee3e2f450 0x557ee3e318e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:42.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.044+0000 7f147adac740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x557ee3d8f060 con 0x557ee3e2f450 2024-04-02T14:34:42.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.044+0000 7f147adac740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x557ee3d37560 con 0x557ee3e298f0 2024-04-02T14:34:42.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.044+0000 7f147adac740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x557ee3d8b010 con 0x557ee3e2a2e0 2024-04-02T14:34:42.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.044+0000 7f1476000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557ee3e298f0 0x557ee3e29d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:42.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.044+0000 7f1474c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557ee3e2f450 0x557ee3e318e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:42.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.044+0000 7f1476000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557ee3e298f0 0x557ee3e29d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:60562/0 (socket says 172.21.15.90:60562) 2024-04-02T14:34:42.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.044+0000 7f1476000640 1 -- 172.21.15.90:0/594351175 learned_addr learned my addr 172.21.15.90:0/594351175 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:34:42.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.044+0000 7f1475600640 1 --2- 172.21.15.90:0/594351175 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ee3e2a2e0 0x557ee3e2ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:42.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.044+0000 7f1476000640 1 -- 172.21.15.90:0/594351175 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ee3e2a2e0 msgr2=0x557ee3e2ede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:34:42.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.044+0000 7f1476000640 1 --2- 172.21.15.90:0/594351175 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ee3e2a2e0 0x557ee3e2ede0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:42.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.044+0000 7f1476000640 1 -- 172.21.15.90:0/594351175 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557ee3e2f450 msgr2=0x557ee3e318e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:34:42.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.044+0000 7f1476000640 1 --2- 172.21.15.90:0/594351175 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557ee3e2f450 0x557ee3e318e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:42.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.044+0000 7f1476000640 1 -- 172.21.15.90:0/594351175 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557ee3d59fd0 con 0x557ee3e298f0 2024-04-02T14:34:42.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.044+0000 7f1475600640 1 --2- 172.21.15.90:0/594351175 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ee3e2a2e0 0x557ee3e2ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:34:42.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.045+0000 7f1476000640 1 --2- 172.21.15.90:0/594351175 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557ee3e298f0 0x557ee3e29d10 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f1468009f60 tx=0x7f1468009f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1e3b384e8aef1480 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:34:42.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.045+0000 7f146e000640 1 --2- 172.21.15.90:0/2605790048 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f14580a3bb0 0x7f14580a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:42.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.045+0000 7f1462600640 1 -- 172.21.15.90:0/594351175 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f146801d070 con 0x557ee3e298f0 2024-04-02T14:34:42.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.045+0000 7f1462600640 1 -- 172.21.15.90:0/594351175 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f146800dad0 con 0x557ee3e298f0 2024-04-02T14:34:42.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.045+0000 7f1462600640 1 -- 172.21.15.90:0/594351175 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1468018c30 con 0x557ee3e298f0 2024-04-02T14:34:42.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.045+0000 7f146e000640 1 -- 172.21.15.90:0/2605790048 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8248+0+0 (secure 0 0 0) 0x7f145c0f1e80 con 0x7f1470109310 2024-04-02T14:34:42.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.045+0000 7f1475600640 1 --2- 172.21.15.90:0/2605790048 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f14580a3bb0 0x7f14580a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:42.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.046+0000 7f147adac740 1 -- 172.21.15.90:0/594351175 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557ee3e298f0 msgr2=0x557ee3e29d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:42.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.046+0000 7f147adac740 1 --2- 172.21.15.90:0/594351175 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557ee3e298f0 0x557ee3e29d10 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f1468009f60 tx=0x7f1468009f90 comp rx=0 tx=0).stop 2024-04-02T14:34:42.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.046+0000 7f1475600640 1 --2- 172.21.15.90:0/2605790048 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f14580a3bb0 0x7f14580a6070 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x557ee3e2a6e0 tx=0x7f1464005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:34:42.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.046+0000 7f147adac740 1 -- 172.21.15.90:0/594351175 shutdown_connections 2024-04-02T14:34:42.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.046+0000 7f147adac740 1 --2- 172.21.15.90:0/594351175 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557ee3e2f450 0x557ee3e318e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:42.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.046+0000 7f147adac740 1 --2- 172.21.15.90:0/594351175 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ee3e2a2e0 0x557ee3e2ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:42.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.046+0000 7f147adac740 1 --2- 172.21.15.90:0/594351175 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557ee3e298f0 0x557ee3e29d10 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:42.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.046+0000 7f147adac740 1 -- 172.21.15.90:0/594351175 >> 172.21.15.90:0/594351175 conn(0x557ee3e24e60 msgr2=0x557ee3e25260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:34:42.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.046+0000 7f147adac740 1 -- 172.21.15.90:0/594351175 shutdown_connections 2024-04-02T14:34:42.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.046+0000 7f147adac740 1 -- 172.21.15.90:0/594351175 wait complete. 2024-04-02T14:34:42.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.047+0000 7f147adac740 1 -- start start 2024-04-02T14:34:42.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.047+0000 7f147adac740 1 Processor -- start 2024-04-02T14:34:42.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.047+0000 7f147adac740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557ee3e298f0 0x557ee3e33e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:42.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.047+0000 7f147adac740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ee3e2a2e0 0x557ee3e343b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:42.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.047+0000 7f147adac740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557ee3e2f450 0x557ee3e3b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:42.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.047+0000 7f147adac740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x557ee3e32400 con 0x557ee3e2f450 2024-04-02T14:34:42.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.047+0000 7f147adac740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x557ee3d4a920 con 0x557ee3e298f0 2024-04-02T14:34:42.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.047+0000 7f147adac740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x557ee3e32580 con 0x557ee3e2a2e0 2024-04-02T14:34:42.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.047+0000 7f147adac740 10 client.0 _ll_register_callbacks cb 0x557ee3d4a310 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:34:42.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.047+0000 7f1475600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557ee3e298f0 0x557ee3e33e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:42.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.047+0000 7f1476000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557ee3e2f450 0x557ee3e3b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:42.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.047+0000 7f1475600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557ee3e298f0 0x557ee3e33e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:60564/0 (socket says 172.21.15.90:60564) 2024-04-02T14:34:42.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.047+0000 7f1475600640 1 -- 172.21.15.90:0/702798937 learned_addr learned my addr 172.21.15.90:0/702798937 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:34:42.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.047+0000 7f1474c00640 1 --2- 172.21.15.90:0/702798937 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ee3e2a2e0 0x557ee3e343b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:42.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.048+0000 7f1475600640 1 -- 172.21.15.90:0/702798937 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ee3e2a2e0 msgr2=0x557ee3e343b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:34:42.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.048+0000 7f1475600640 1 --2- 172.21.15.90:0/702798937 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ee3e2a2e0 0x557ee3e343b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:42.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.048+0000 7f1475600640 1 -- 172.21.15.90:0/702798937 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557ee3e2f450 msgr2=0x557ee3e3b3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:34:42.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.048+0000 7f1475600640 1 --2- 172.21.15.90:0/702798937 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557ee3e2f450 0x557ee3e3b3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:42.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.048+0000 7f1475600640 1 -- 172.21.15.90:0/702798937 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557ee3aa32d0 con 0x557ee3e298f0 2024-04-02T14:34:42.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.048+0000 7f1476000640 1 --2- 172.21.15.90:0/702798937 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557ee3e2f450 0x557ee3e3b3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:34:42.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.048+0000 7f1475600640 1 --2- 172.21.15.90:0/702798937 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557ee3e298f0 0x557ee3e33e70 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f146401e9d0 tx=0x7f1464004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3460870a0901dfea server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:34:42.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.048+0000 7f1456a00640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:34:42.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.049+0000 7f1456a00640 1 -- 172.21.15.90:0/702798937 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f146402e060 con 0x557ee3e298f0 2024-04-02T14:34:42.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.049+0000 7f147adac740 1 -- 172.21.15.90:0/702798937 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f146800dad0 con 0x557ee3e298f0 2024-04-02T14:34:42.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.049+0000 7f1456a00640 1 -- 172.21.15.90:0/702798937 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8248+0+0 (secure 0 0 0) 0x7f1464018030 con 0x557ee3e298f0 2024-04-02T14:34:42.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.049+0000 7f1456a00640 1 -- 172.21.15.90:0/702798937 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f1464009850 con 0x557ee3e298f0 2024-04-02T14:34:42.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.049+0000 7f147adac740 10 client.25244 fetch_fsmap learned FSMap version 26 2024-04-02T14:34:42.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.049+0000 7f147adac740 1 -- 172.21.15.90:0/702798937 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x557ee3b6d6b0 con 0x557ee3e298f0 2024-04-02T14:34:42.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.050+0000 7f1456a00640 1 -- 172.21.15.90:0/702798937 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f146800dad0 con 0x557ee3e298f0 2024-04-02T14:34:42.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.050+0000 7f1456a00640 1 -- 172.21.15.90:0/702798937 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f146400f5f0 con 0x557ee3e298f0 2024-04-02T14:34:42.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.050+0000 7f147adac740 10 client.25244 fetch_fsmap finished waiting for FSMap version 26 2024-04-02T14:34:42.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.050+0000 7f147adac740 10 client.25244 resolve_mds: resolved name 'g' to daemon mds.g 2024-04-02T14:34:42.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.050+0000 7f147adac740 20 client.25244 populate_metadata read hostname 'smithi090' 2024-04-02T14:34:42.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.050+0000 7f147adac740 1 --2- 172.21.15.90:0/702798937 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x557ee3e40c40 0x557ee3e420b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:42.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.050+0000 7f147adac740 4 client.25244 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:34:42.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.050+0000 7f147adac740 1 -- 172.21.15.90:0/702798937 --> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557ee3d9c5a0 con 0x557ee3e40c40 2024-04-02T14:34:42.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.050+0000 7f1474c00640 1 --2- 172.21.15.90:0/702798937 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x557ee3e40c40 0x557ee3e420b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:42.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.051+0000 7f1474c00640 1 --2- 172.21.15.90:0/702798937 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x557ee3e40c40 0x557ee3e420b0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=626b7c49f3ad36f1 server_cookie=b524507271ac9f98 in_seq=0 out_seq=0 2024-04-02T14:34:42.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.051+0000 7f1456a00640 10 client.25244 ms_handle_connect on v2:172.21.15.90:6838/1498915870 2024-04-02T14:34:42.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.051+0000 7f1456a00640 1 -- 172.21.15.90:0/702798937 <== mds.1 v2:172.21.15.90:6838/1498915870 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x557ee3d9c5a0 con 0x557ee3e40c40 2024-04-02T14:34:42.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.051+0000 7f1456a00640 10 client.25244 handle_command_reply: tid=0 2024-04-02T14:34:42.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.052+0000 7f147adac740 1 client.25244 shutdown 2024-04-02T14:34:42.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.052+0000 7f147adac740 1 -- 172.21.15.90:0/702798937 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557ee3e298f0 msgr2=0x557ee3e33e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:42.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.052+0000 7f147adac740 1 --2- 172.21.15.90:0/702798937 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557ee3e298f0 0x557ee3e33e70 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f146401e9d0 tx=0x7f1464004730 comp rx=0 tx=0).stop 2024-04-02T14:34:42.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.052+0000 7f147adac740 1 -- 172.21.15.90:0/702798937 shutdown_connections 2024-04-02T14:34:42.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.052+0000 7f147adac740 1 --2- 172.21.15.90:0/702798937 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x557ee3e40c40 0x557ee3e420b0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:42.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.052+0000 7f147adac740 1 --2- 172.21.15.90:0/702798937 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557ee3e2f450 0x557ee3e3b3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:42.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.052+0000 7f1456a00640 0 client.25244 ms_handle_reset on v2:172.21.15.90:6838/1498915870 2024-04-02T14:34:42.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.052+0000 7f147adac740 1 --2- 172.21.15.90:0/702798937 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ee3e2a2e0 0x557ee3e343b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:42.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.052+0000 7f147adac740 1 --2- 172.21.15.90:0/702798937 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557ee3e298f0 0x557ee3e33e70 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:42.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.052+0000 7f147adac740 1 -- 172.21.15.90:0/702798937 >> 172.21.15.90:0/702798937 conn(0x557ee3e24e60 msgr2=0x557ee3e24080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:34:42.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.052+0000 7f147adac740 1 -- 172.21.15.90:0/702798937 shutdown_connections 2024-04-02T14:34:42.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.052+0000 7f147adac740 1 -- 172.21.15.90:0/702798937 wait complete. 2024-04-02T14:34:42.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.052+0000 7f147adac740 20 client.25244 trim_cache size 0 max 16384 2024-04-02T14:34:42.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.072+0000 7f147adac740 1 Processor -- start 2024-04-02T14:34:42.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.072+0000 7f147adac740 1 -- start start 2024-04-02T14:34:42.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.072+0000 7f147adac740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ee3e298f0 0x557ee3e2ee10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:42.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.072+0000 7f147adac740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557ee3e2a2e0 0x557ee3e3b3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:42.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.072+0000 7f147adac740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557ee3e2f450 0x557ee3e2b840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:42.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.072+0000 7f147adac740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x557ee3e403f0 con 0x557ee3e2a2e0 2024-04-02T14:34:42.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.072+0000 7f147adac740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x557ee3e40ac0 con 0x557ee3e2f450 2024-04-02T14:34:42.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.072+0000 7f147adac740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x557ee3e41190 con 0x557ee3e298f0 2024-04-02T14:34:42.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.072+0000 7f1474c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557ee3e2a2e0 0x557ee3e3b3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:42.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.072+0000 7f1474c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557ee3e2a2e0 0x557ee3e3b3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:36646/0 (socket says 172.21.15.90:36646) 2024-04-02T14:34:42.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.072+0000 7f1474c00640 1 -- 172.21.15.90:0/971060426 learned_addr learned my addr 172.21.15.90:0/971060426 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:34:42.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.072+0000 7f1475600640 1 --2- 172.21.15.90:0/971060426 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ee3e298f0 0x557ee3e2ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:42.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.072+0000 7f1476000640 1 --2- 172.21.15.90:0/971060426 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557ee3e2f450 0x557ee3e2b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:42.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.073+0000 7f1474c00640 1 -- 172.21.15.90:0/971060426 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ee3e298f0 msgr2=0x557ee3e2ee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:34:42.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.073+0000 7f1474c00640 1 --2- 172.21.15.90:0/971060426 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ee3e298f0 0x557ee3e2ee10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:42.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.073+0000 7f1474c00640 1 -- 172.21.15.90:0/971060426 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557ee3e2f450 msgr2=0x557ee3e2b840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:34:42.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.073+0000 7f1474c00640 1 --2- 172.21.15.90:0/971060426 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557ee3e2f450 0x557ee3e2b840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:42.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.073+0000 7f1474c00640 1 -- 172.21.15.90:0/971060426 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557ee3e247b0 con 0x557ee3e2a2e0 2024-04-02T14:34:42.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.073+0000 7f1474c00640 1 --2- 172.21.15.90:0/971060426 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557ee3e2a2e0 0x557ee3e3b3b0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f145c0be080 tx=0x7f145c0f1040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ec107d2aacfaa61 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:34:42.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.073+0000 7f1461c00640 1 -- 172.21.15.90:0/971060426 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f145c0bdec0 con 0x557ee3e2a2e0 2024-04-02T14:34:42.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.073+0000 7f1461c00640 1 -- 172.21.15.90:0/971060426 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f145c0be7f0 con 0x557ee3e2a2e0 2024-04-02T14:34:42.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.073+0000 7f1461c00640 1 -- 172.21.15.90:0/971060426 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f145c0bda50 con 0x557ee3e2a2e0 2024-04-02T14:34:42.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.074+0000 7f147adac740 1 -- 172.21.15.90:0/971060426 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557ee3e2a2e0 msgr2=0x557ee3e3b3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:42.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.074+0000 7f147adac740 1 --2- 172.21.15.90:0/971060426 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557ee3e2a2e0 0x557ee3e3b3b0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f145c0be080 tx=0x7f145c0f1040 comp rx=0 tx=0).stop 2024-04-02T14:34:42.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.074+0000 7f147adac740 1 -- 172.21.15.90:0/971060426 shutdown_connections 2024-04-02T14:34:42.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.074+0000 7f147adac740 1 --2- 172.21.15.90:0/971060426 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557ee3e2f450 0x557ee3e2b840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:42.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.074+0000 7f147adac740 1 --2- 172.21.15.90:0/971060426 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557ee3e2a2e0 0x557ee3e3b3b0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:42.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.074+0000 7f147adac740 1 --2- 172.21.15.90:0/971060426 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ee3e298f0 0x557ee3e2ee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:42.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.074+0000 7f147adac740 1 -- 172.21.15.90:0/971060426 >> 172.21.15.90:0/971060426 conn(0x557ee3e24e60 msgr2=0x557ee3e2ace0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:34:42.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.074+0000 7f147adac740 1 -- 172.21.15.90:0/971060426 shutdown_connections 2024-04-02T14:34:42.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.074+0000 7f147adac740 1 -- 172.21.15.90:0/971060426 wait complete. 2024-04-02T14:34:42.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.074+0000 7f147adac740 1 -- start start 2024-04-02T14:34:42.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.075+0000 7f147adac740 1 Processor -- start 2024-04-02T14:34:42.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.075+0000 7f147adac740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557ee3e298f0 0x557ee3e375e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:42.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.075+0000 7f147adac740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557ee3e2a2e0 0x557ee3e37b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:42.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.075+0000 7f147adac740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ee3e2f450 0x557ee3e3a620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:42.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.075+0000 7f147adac740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x557ee3e41f30 con 0x557ee3e298f0 2024-04-02T14:34:42.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.075+0000 7f147adac740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x557ee3e41860 con 0x557ee3e2a2e0 2024-04-02T14:34:42.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.075+0000 7f147adac740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x557ee3e42600 con 0x557ee3e2f450 2024-04-02T14:34:42.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.075+0000 7f147adac740 10 client.0 _ll_register_callbacks cb 0x7f146401e890 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:34:42.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.075+0000 7f1475600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557ee3e298f0 0x557ee3e375e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:42.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.075+0000 7f1475600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557ee3e298f0 0x557ee3e375e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:36650/0 (socket says 172.21.15.90:36650) 2024-04-02T14:34:42.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.075+0000 7f1475600640 1 -- 172.21.15.90:0/2798108700 learned_addr learned my addr 172.21.15.90:0/2798108700 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:34:42.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.075+0000 7f1476000640 1 --2- 172.21.15.90:0/2798108700 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ee3e2f450 0x557ee3e3a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:42.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.075+0000 7f1474c00640 1 --2- 172.21.15.90:0/2798108700 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557ee3e2a2e0 0x557ee3e37b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:42.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.076+0000 7f1475600640 1 -- 172.21.15.90:0/2798108700 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ee3e2f450 msgr2=0x557ee3e3a620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:34:42.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.076+0000 7f1475600640 1 --2- 172.21.15.90:0/2798108700 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ee3e2f450 0x557ee3e3a620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:42.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.076+0000 7f1475600640 1 -- 172.21.15.90:0/2798108700 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557ee3e2a2e0 msgr2=0x557ee3e37b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:34:42.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.076+0000 7f1475600640 1 --2- 172.21.15.90:0/2798108700 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557ee3e2a2e0 0x557ee3e37b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:42.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.076+0000 7f1475600640 1 -- 172.21.15.90:0/2798108700 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557ee3e35000 con 0x557ee3e298f0 2024-04-02T14:34:42.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.076+0000 7f1474c00640 1 --2- 172.21.15.90:0/2798108700 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557ee3e2a2e0 0x557ee3e37b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:34:42.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.076+0000 7f1475600640 1 --2- 172.21.15.90:0/2798108700 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557ee3e298f0 0x557ee3e375e0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f1464004700 tx=0x7f146400f5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7386e93eee5c67f5 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:34:42.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.076+0000 7f1456a00640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-02T14:34:42.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.076+0000 7f1456a00640 1 -- 172.21.15.90:0/2798108700 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1464041060 con 0x557ee3e298f0 2024-04-02T14:34:42.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.076+0000 7f147adac740 1 -- 172.21.15.90:0/2798108700 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f145c0be7f0 con 0x557ee3e298f0 2024-04-02T14:34:42.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.076+0000 7f1456a00640 1 -- 172.21.15.90:0/2798108700 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8248+0+0 (secure 0 0 0) 0x7f1464029420 con 0x557ee3e298f0 2024-04-02T14:34:42.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.077+0000 7f1456a00640 1 -- 172.21.15.90:0/2798108700 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f1464029660 con 0x557ee3e298f0 2024-04-02T14:34:42.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.077+0000 7f147adac740 10 client.15144 fetch_fsmap learned FSMap version 26 2024-04-02T14:34:42.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.077+0000 7f147adac740 1 -- 172.21.15.90:0/2798108700 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x557ee3ea7070 con 0x557ee3e298f0 2024-04-02T14:34:42.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.077+0000 7f1456a00640 1 -- 172.21.15.90:0/2798108700 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f145c0be7f0 con 0x557ee3e298f0 2024-04-02T14:34:42.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.077+0000 7f1456a00640 1 -- 172.21.15.90:0/2798108700 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f146403c8a0 con 0x557ee3e298f0 2024-04-02T14:34:42.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.077+0000 7f147adac740 10 client.15144 fetch_fsmap finished waiting for FSMap version 26 2024-04-02T14:34:42.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.077+0000 7f147adac740 10 client.15144 resolve_mds: resolved name 'g' to daemon mds.g 2024-04-02T14:34:42.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.077+0000 7f147adac740 20 client.15144 populate_metadata read hostname 'smithi090' 2024-04-02T14:34:42.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.077+0000 7f147adac740 1 --2- 172.21.15.90:0/2798108700 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x557ee3eabd00 0x557ee3ea8140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:42.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.077+0000 7f147adac740 4 client.15144 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-02T14:34:42.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.077+0000 7f147adac740 1 -- 172.21.15.90:0/2798108700 --> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] -- command(tid 0: {"prefix": "status"}) -- 0x557ee3d968f0 con 0x557ee3eabd00 2024-04-02T14:34:42.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.078+0000 7f1474c00640 1 --2- 172.21.15.90:0/2798108700 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x557ee3eabd00 0x557ee3ea8140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:42.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.078+0000 7f1456a00640 10 client.15144 ms_handle_connect on v2:172.21.15.90:6838/1498915870 2024-04-02T14:34:42.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.078+0000 7f1474c00640 1 --2- 172.21.15.90:0/2798108700 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x557ee3eabd00 0x557ee3ea8140 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=29430614b97ced0c server_cookie=2f6c7d52397de3a5 in_seq=0 out_seq=0 2024-04-02T14:34:42.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.078+0000 7f1456a00640 1 -- 172.21.15.90:0/2798108700 <== mds.1 v2:172.21.15.90:6838/1498915870 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x557ee3d968f0 con 0x557ee3eabd00 2024-04-02T14:34:42.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.078+0000 7f1456a00640 10 client.15144 handle_command_reply: tid=0 2024-04-02T14:34:42.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.078+0000 7f147adac740 1 client.15144 shutdown 2024-04-02T14:34:42.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.079+0000 7f147adac740 1 -- 172.21.15.90:0/2798108700 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557ee3e298f0 msgr2=0x557ee3e375e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:42.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.079+0000 7f147adac740 1 --2- 172.21.15.90:0/2798108700 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557ee3e298f0 0x557ee3e375e0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f1464004700 tx=0x7f146400f5f0 comp rx=0 tx=0).stop 2024-04-02T14:34:42.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.079+0000 7f147adac740 1 -- 172.21.15.90:0/2798108700 shutdown_connections 2024-04-02T14:34:42.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.079+0000 7f147adac740 1 --2- 172.21.15.90:0/2798108700 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x557ee3eabd00 0x557ee3ea8140 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:42.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.079+0000 7f147adac740 1 --2- 172.21.15.90:0/2798108700 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ee3e2f450 0x557ee3e3a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:42.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.079+0000 7f147adac740 1 --2- 172.21.15.90:0/2798108700 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557ee3e2a2e0 0x557ee3e37b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:42.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.079+0000 7f1456a00640 0 client.15144 ms_handle_reset on v2:172.21.15.90:6838/1498915870 2024-04-02T14:34:42.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.079+0000 7f147adac740 1 --2- 172.21.15.90:0/2798108700 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557ee3e298f0 0x557ee3e375e0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:42.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.079+0000 7f147adac740 1 -- 172.21.15.90:0/2798108700 >> 172.21.15.90:0/2798108700 conn(0x557ee3e24e60 msgr2=0x557ee3e2b840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:34:42.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.079+0000 7f147adac740 1 -- 172.21.15.90:0/2798108700 shutdown_connections 2024-04-02T14:34:42.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.079+0000 7f147adac740 1 -- 172.21.15.90:0/2798108700 wait complete. 2024-04-02T14:34:42.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.079+0000 7f147adac740 20 client.15144 trim_cache size 0 max 16384 2024-04-02T14:34:42.085 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-02T14:34:42.085 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_fsid": "6449448a-f0fc-11ee-b647-cb9ed24678a4", 2024-04-02T14:34:42.085 INFO:teuthology.orchestra.run.smithi090.stdout: "whoami": 1, 2024-04-02T14:34:42.085 INFO:teuthology.orchestra.run.smithi090.stdout: "id": 24560, 2024-04-02T14:34:42.085 INFO:teuthology.orchestra.run.smithi090.stdout: "want_state": "up:active", 2024-04-02T14:34:42.086 INFO:teuthology.orchestra.run.smithi090.stdout: "state": "up:active", 2024-04-02T14:34:42.086 INFO:teuthology.orchestra.run.smithi090.stdout: "fs_name": "cephfs", 2024-04-02T14:34:42.086 INFO:teuthology.orchestra.run.smithi090.stdout: "rank_uptime": 136.16749520499999, 2024-04-02T14:34:42.086 INFO:teuthology.orchestra.run.smithi090.stdout: "mdsmap_epoch": 26, 2024-04-02T14:34:42.086 INFO:teuthology.orchestra.run.smithi090.stdout: "osdmap_epoch": 90, 2024-04-02T14:34:42.086 INFO:teuthology.orchestra.run.smithi090.stdout: "osdmap_epoch_barrier": 88, 2024-04-02T14:34:42.086 INFO:teuthology.orchestra.run.smithi090.stdout: "uptime": 151.889708133 2024-04-02T14:34:42.086 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-02T14:34:42.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.079+0000 7f1478000640 1 -- 172.21.15.90:0/2605790048 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f14580a3bb0 msgr2=0x7f14580a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:42.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.079+0000 7f1478000640 1 --2- 172.21.15.90:0/2605790048 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f14580a3bb0 0x7f14580a6070 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x557ee3e2a6e0 tx=0x7f1464005f90 comp rx=0 tx=0).stop 2024-04-02T14:34:42.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.079+0000 7f1478000640 1 -- 172.21.15.90:0/2605790048 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1470109310 msgr2=0x7f147019e990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:42.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.079+0000 7f1478000640 1 --2- 172.21.15.90:0/2605790048 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1470109310 0x7f147019e990 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f145c005940 tx=0x7f145c032470 comp rx=0 tx=0).stop 2024-04-02T14:34:42.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.080+0000 7f1478000640 1 -- 172.21.15.90:0/2605790048 shutdown_connections 2024-04-02T14:34:42.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.080+0000 7f1478000640 1 --2- 172.21.15.90:0/2605790048 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f14580a3bb0 0x7f14580a6070 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:42.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.080+0000 7f1478000640 1 --2- 172.21.15.90:0/2605790048 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1470109cd0 0x7f14701a5a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:42.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.080+0000 7f1478000640 1 --2- 172.21.15.90:0/2605790048 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1470109310 0x7f147019e990 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:42.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.080+0000 7f1478000640 1 --2- 172.21.15.90:0/2605790048 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1470108120 0x7f147019e450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:42.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.080+0000 7f1478000640 1 -- 172.21.15.90:0/2605790048 >> 172.21.15.90:0/2605790048 conn(0x7f14700770c0 msgr2=0x7f147010c630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:34:42.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.080+0000 7f1478000640 1 -- 172.21.15.90:0/2605790048 shutdown_connections 2024-04-02T14:34:42.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.080+0000 7f1478000640 1 -- 172.21.15.90:0/2605790048 wait complete. 2024-04-02T14:34:42.098 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-04-02T14:34:42.168 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:34:41 smithi090 ceph-mon[31674]: pgmap v383: 97 pgs: 97 active+clean; 644 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-02T14:34:42.169 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:34:41 smithi090 ceph-mon[31674]: from='client.? 172.21.15.90:0/3073919238' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-02T14:34:42.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:34:41 smithi177 ceph-mon[36431]: pgmap v383: 97 pgs: 97 active+clean; 644 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-02T14:34:42.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:34:41 smithi177 ceph-mon[36431]: from='client.? 172.21.15.90:0/3073919238' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-02T14:34:42.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.303+0000 7f3e9f800640 1 -- 172.21.15.90:0/1061315399 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3e981099a0 msgr2=0x7f3e9810be40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:42.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.303+0000 7f3e9f800640 1 --2- 172.21.15.90:0/1061315399 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3e981099a0 0x7f3e9810be40 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f3e90009900 tx=0x7f3e9002bcd0 comp rx=0 tx=0).stop 2024-04-02T14:34:42.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.303+0000 7f3e9f800640 1 -- 172.21.15.90:0/1061315399 shutdown_connections 2024-04-02T14:34:42.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.303+0000 7f3e9f800640 1 --2- 172.21.15.90:0/1061315399 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3e981099a0 0x7f3e9810be40 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:42.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.303+0000 7f3e9f800640 1 --2- 172.21.15.90:0/1061315399 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3e98074500 0x7f3e98074960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:42.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.303+0000 7f3e9f800640 1 --2- 172.21.15.90:0/1061315399 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3e98075ae0 0x7f3e98073f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:42.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.303+0000 7f3e9f800640 1 -- 172.21.15.90:0/1061315399 >> 172.21.15.90:0/1061315399 conn(0x7f3e980fd0a0 msgr2=0x7f3e980ff4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:34:42.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.303+0000 7f3e9f800640 1 -- 172.21.15.90:0/1061315399 shutdown_connections 2024-04-02T14:34:42.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.303+0000 7f3e9f800640 1 -- 172.21.15.90:0/1061315399 wait complete. 2024-04-02T14:34:42.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.304+0000 7f3e9f800640 1 Processor -- start 2024-04-02T14:34:42.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.304+0000 7f3e9f800640 1 -- start start 2024-04-02T14:34:42.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.304+0000 7f3e9f800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3e98074500 0x7f3e98199fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:42.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.304+0000 7f3e9f800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3e98075ae0 0x7f3e9819a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:42.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.304+0000 7f3e9f800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3e981099a0 0x7f3e981a1560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:42.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.304+0000 7f3e9f800640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f3e9810f9c0 con 0x7f3e981099a0 2024-04-02T14:34:42.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.304+0000 7f3e9f800640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f3e9810f840 con 0x7f3e98075ae0 2024-04-02T14:34:42.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.304+0000 7f3e9f800640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f3e9810fb40 con 0x7f3e98074500 2024-04-02T14:34:42.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.304+0000 7f3e9d800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3e981099a0 0x7f3e981a1560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:42.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.304+0000 7f3e97e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3e98075ae0 0x7f3e9819a4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:42.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.304+0000 7f3e97e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3e98075ae0 0x7f3e9819a4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:60606/0 (socket says 172.21.15.90:60606) 2024-04-02T14:34:42.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.304+0000 7f3e97e00640 1 -- 172.21.15.90:0/930871328 learned_addr learned my addr 172.21.15.90:0/930871328 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:34:42.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.305+0000 7f3e9ce00640 1 --2- 172.21.15.90:0/930871328 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3e98074500 0x7f3e98199fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:42.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.305+0000 7f3e97e00640 1 -- 172.21.15.90:0/930871328 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3e98074500 msgr2=0x7f3e98199fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:42.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.305+0000 7f3e97e00640 1 --2- 172.21.15.90:0/930871328 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3e98074500 0x7f3e98199fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:42.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.305+0000 7f3e97e00640 1 -- 172.21.15.90:0/930871328 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3e981099a0 msgr2=0x7f3e981a1560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:42.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.305+0000 7f3e97e00640 1 --2- 172.21.15.90:0/930871328 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3e981099a0 0x7f3e981a1560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:42.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.305+0000 7f3e97e00640 1 -- 172.21.15.90:0/930871328 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e981a1bd0 con 0x7f3e98075ae0 2024-04-02T14:34:42.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.305+0000 7f3e9d800640 1 --2- 172.21.15.90:0/930871328 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3e981099a0 0x7f3e981a1560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:34:42.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.305+0000 7f3e97e00640 1 --2- 172.21.15.90:0/930871328 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3e98075ae0 0x7f3e9819a4f0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f3e8000e9f0 tx=0x7f3e8000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:34:42.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.305+0000 7f3e95600640 1 -- 172.21.15.90:0/930871328 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e8000cd80 con 0x7f3e98075ae0 2024-04-02T14:34:42.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.305+0000 7f3e95600640 1 -- 172.21.15.90:0/930871328 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3e80010910 con 0x7f3e98075ae0 2024-04-02T14:34:42.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.305+0000 7f3e9f800640 1 -- 172.21.15.90:0/930871328 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e981a1ec0 con 0x7f3e98075ae0 2024-04-02T14:34:42.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.305+0000 7f3e95600640 1 -- 172.21.15.90:0/930871328 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e80018990 con 0x7f3e98075ae0 2024-04-02T14:34:42.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.306+0000 7f3e9f800640 1 -- 172.21.15.90:0/930871328 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e981a23d0 con 0x7f3e98075ae0 2024-04-02T14:34:42.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.308+0000 7f3ea240f740 1 Processor -- start 2024-04-02T14:34:42.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.309+0000 7f3ea240f740 1 -- start start 2024-04-02T14:34:42.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.309+0000 7f3e95600640 1 -- 172.21.15.90:0/930871328 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f3e80010ac0 con 0x7f3e98075ae0 2024-04-02T14:34:42.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.309+0000 7f3ea240f740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ad7bba88f0 0x55ad7bba8d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:42.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.309+0000 7f3ea240f740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ad7bba92e0 0x55ad7bbadde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:42.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.309+0000 7f3ea240f740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ad7bbae450 0x55ad7bbb08e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:42.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.309+0000 7f3ea240f740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55ad7bb0e060 con 0x55ad7bbae450 2024-04-02T14:34:42.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.309+0000 7f3ea240f740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55ad7bab6560 con 0x55ad7bba92e0 2024-04-02T14:34:42.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.309+0000 7f3ea240f740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55ad7bb0a010 con 0x55ad7bba88f0 2024-04-02T14:34:42.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.309+0000 7f3e9d800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ad7bba88f0 0x55ad7bba8d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:42.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.309+0000 7f3e97e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ad7bbae450 0x55ad7bbb08e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:42.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.309+0000 7f3e9d800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ad7bba88f0 0x55ad7bba8d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:53082/0 (socket says 172.21.15.90:53082) 2024-04-02T14:34:42.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.309+0000 7f3e9d800640 1 -- 172.21.15.90:0/3186244160 learned_addr learned my addr 172.21.15.90:0/3186244160 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:34:42.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.309+0000 7f3e9ce00640 1 --2- 172.21.15.90:0/3186244160 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ad7bba92e0 0x55ad7bbadde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:42.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.310+0000 7f3e9d800640 1 -- 172.21.15.90:0/3186244160 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ad7bba92e0 msgr2=0x55ad7bbadde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:34:42.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.310+0000 7f3e9d800640 1 --2- 172.21.15.90:0/3186244160 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ad7bba92e0 0x55ad7bbadde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:42.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.310+0000 7f3e9d800640 1 -- 172.21.15.90:0/3186244160 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ad7bbae450 msgr2=0x55ad7bbb08e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:34:42.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.310+0000 7f3e9d800640 1 --2- 172.21.15.90:0/3186244160 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ad7bbae450 0x55ad7bbb08e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:42.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.310+0000 7f3e9d800640 1 -- 172.21.15.90:0/3186244160 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ad7bad8fd0 con 0x55ad7bba88f0 2024-04-02T14:34:42.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.310+0000 7f3e9d800640 1 --2- 172.21.15.90:0/3186244160 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ad7bba88f0 0x55ad7bba8d10 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f3e9002c2e0 tx=0x7f3e9002c720 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2ad849010fe14c89 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:34:42.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.310+0000 7f3e8e000640 1 -- 172.21.15.90:0/3186244160 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e90043070 con 0x55ad7bba88f0 2024-04-02T14:34:42.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.310+0000 7f3e8e000640 1 -- 172.21.15.90:0/3186244160 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3e90004a70 con 0x55ad7bba88f0 2024-04-02T14:34:42.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.310+0000 7f3e8e000640 1 -- 172.21.15.90:0/3186244160 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e900051c0 con 0x55ad7bba88f0 2024-04-02T14:34:42.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.310+0000 7f3e95600640 1 --2- 172.21.15.90:0/930871328 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f3e7c0a3c30 0x7f3e7c0a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:42.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.311+0000 7f3e95600640 1 -- 172.21.15.90:0/930871328 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8248+0+0 (secure 0 0 0) 0x7f3e800d06b0 con 0x7f3e98075ae0 2024-04-02T14:34:42.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.311+0000 7f3e9ce00640 1 --2- 172.21.15.90:0/930871328 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f3e7c0a3c30 0x7f3e7c0a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:42.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.311+0000 7f3ea240f740 1 -- 172.21.15.90:0/3186244160 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ad7bba88f0 msgr2=0x55ad7bba8d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:42.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.311+0000 7f3ea240f740 1 --2- 172.21.15.90:0/3186244160 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ad7bba88f0 0x55ad7bba8d10 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f3e9002c2e0 tx=0x7f3e9002c720 comp rx=0 tx=0).stop 2024-04-02T14:34:42.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.311+0000 7f3ea240f740 1 -- 172.21.15.90:0/3186244160 shutdown_connections 2024-04-02T14:34:42.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.311+0000 7f3ea240f740 1 --2- 172.21.15.90:0/3186244160 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ad7bbae450 0x55ad7bbb08e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:42.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.311+0000 7f3ea240f740 1 --2- 172.21.15.90:0/3186244160 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ad7bba92e0 0x55ad7bbadde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:42.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.311+0000 7f3ea240f740 1 --2- 172.21.15.90:0/3186244160 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ad7bba88f0 0x55ad7bba8d10 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:42.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.311+0000 7f3ea240f740 1 -- 172.21.15.90:0/3186244160 >> 172.21.15.90:0/3186244160 conn(0x55ad7bba3e60 msgr2=0x55ad7bba4260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:34:42.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.311+0000 7f3ea240f740 1 -- 172.21.15.90:0/3186244160 shutdown_connections 2024-04-02T14:34:42.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.311+0000 7f3ea240f740 1 -- 172.21.15.90:0/3186244160 wait complete. 2024-04-02T14:34:42.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.312+0000 7f3e9ce00640 1 --2- 172.21.15.90:0/930871328 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f3e7c0a3c30 0x7f3e7c0a60f0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x55ad7bba4dd0 tx=0x7f3e88009760 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:34:42.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.312+0000 7f3ea240f740 1 -- start start 2024-04-02T14:34:42.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.312+0000 7f3ea240f740 1 Processor -- start 2024-04-02T14:34:42.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.312+0000 7f3ea240f740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ad7bba88f0 0x55ad7bbb2db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:42.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.312+0000 7f3ea240f740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ad7bba92e0 0x55ad7bbb32f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:42.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.312+0000 7f3ea240f740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ad7bbae450 0x55ad7bbba330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:42.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.312+0000 7f3ea240f740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55ad7bbb13a0 con 0x55ad7bbae450 2024-04-02T14:34:42.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.312+0000 7f3ea240f740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55ad7bac9920 con 0x55ad7bba88f0 2024-04-02T14:34:42.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.312+0000 7f3ea240f740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55ad7bbb1520 con 0x55ad7bba92e0 2024-04-02T14:34:42.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.312+0000 7f3ea240f740 10 client.0 _ll_register_callbacks cb 0x55ad7bac9310 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:34:42.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.313+0000 7f3e9ce00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ad7bba88f0 0x55ad7bbb2db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:42.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.313+0000 7f3e9ce00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ad7bba88f0 0x55ad7bbb2db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:60626/0 (socket says 172.21.15.90:60626) 2024-04-02T14:34:42.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.313+0000 7f3e9ce00640 1 -- 172.21.15.90:0/3818920034 learned_addr learned my addr 172.21.15.90:0/3818920034 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:34:42.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.313+0000 7f3e9d800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ad7bbae450 0x55ad7bbba330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:42.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.313+0000 7f3e97e00640 1 --2- 172.21.15.90:0/3818920034 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ad7bba92e0 0x55ad7bbb32f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:42.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.313+0000 7f3e9d800640 1 -- 172.21.15.90:0/3818920034 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ad7bba92e0 msgr2=0x55ad7bbb32f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:34:42.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.313+0000 7f3e9d800640 1 --2- 172.21.15.90:0/3818920034 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ad7bba92e0 0x55ad7bbb32f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:42.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.313+0000 7f3e9d800640 1 -- 172.21.15.90:0/3818920034 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ad7bba88f0 msgr2=0x55ad7bbb2db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:34:42.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.313+0000 7f3e9d800640 1 --2- 172.21.15.90:0/3818920034 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ad7bba88f0 0x55ad7bbb2db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:42.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.313+0000 7f3e9d800640 1 -- 172.21.15.90:0/3818920034 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ad7b8222d0 con 0x55ad7bbae450 2024-04-02T14:34:42.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.313+0000 7f3e9ce00640 1 --2- 172.21.15.90:0/3818920034 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ad7bba88f0 0x55ad7bbb2db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:34:42.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.313+0000 7f3e9d800640 1 --2- 172.21.15.90:0/3818920034 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ad7bbae450 0x55ad7bbba330 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f3e9002c850 tx=0x7f3e9002cf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5097d4eafe06cba server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:34:42.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.313+0000 7f3e86a00640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-02T14:34:42.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.314+0000 7f3e86a00640 1 -- 172.21.15.90:0/3818920034 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e90043050 con 0x55ad7bbae450 2024-04-02T14:34:42.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.314+0000 7f3e86a00640 1 -- 172.21.15.90:0/3818920034 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8248+0+0 (secure 0 0 0) 0x7f3e90005370 con 0x55ad7bbae450 2024-04-02T14:34:42.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.314+0000 7f3ea240f740 1 -- 172.21.15.90:0/3818920034 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3e90004a70 con 0x55ad7bbae450 2024-04-02T14:34:42.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.314+0000 7f3e86a00640 1 -- 172.21.15.90:0/3818920034 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f3e90005570 con 0x55ad7bbae450 2024-04-02T14:34:42.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.314+0000 7f3ea240f740 10 client.15156 fetch_fsmap learned FSMap version 26 2024-04-02T14:34:42.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.314+0000 7f3ea240f740 1 -- 172.21.15.90:0/3818920034 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55ad7b8ec6b0 con 0x55ad7bbae450 2024-04-02T14:34:42.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.314+0000 7f3e86a00640 1 -- 172.21.15.90:0/3818920034 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3e90004a70 con 0x55ad7bbae450 2024-04-02T14:34:42.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.314+0000 7f3e86a00640 1 -- 172.21.15.90:0/3818920034 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f3e9003f600 con 0x55ad7bbae450 2024-04-02T14:34:42.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.315+0000 7f3ea240f740 10 client.15156 fetch_fsmap finished waiting for FSMap version 26 2024-04-02T14:34:42.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.315+0000 7f3ea240f740 10 client.15156 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-02T14:34:42.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.315+0000 7f3ea240f740 20 client.15156 populate_metadata read hostname 'smithi090' 2024-04-02T14:34:42.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.315+0000 7f3ea240f740 1 --2- 172.21.15.90:0/3818920034 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55ad7bbbf6e0 0x55ad7bbc0b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:42.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.315+0000 7f3ea240f740 4 client.15156 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:34:42.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.315+0000 7f3ea240f740 1 -- 172.21.15.90:0/3818920034 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ad7bb1b5a0 con 0x55ad7bbbf6e0 2024-04-02T14:34:42.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.315+0000 7f3e9ce00640 1 --2- 172.21.15.90:0/3818920034 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55ad7bbbf6e0 0x55ad7bbc0b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:42.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.316+0000 7f3e9ce00640 1 --2- 172.21.15.90:0/3818920034 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55ad7bbbf6e0 0x55ad7bbc0b50 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=257b3838e367291 server_cookie=cca13dfaa9567c4c in_seq=0 out_seq=0 2024-04-02T14:34:42.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.316+0000 7f3e86a00640 10 client.15156 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:34:42.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.316+0000 7f3e86a00640 1 -- 172.21.15.90:0/3818920034 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ad7bb1b5a0 con 0x55ad7bbbf6e0 2024-04-02T14:34:42.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.316+0000 7f3e86a00640 10 client.15156 handle_command_reply: tid=0 2024-04-02T14:34:42.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.316+0000 7f3ea240f740 1 client.15156 shutdown 2024-04-02T14:34:42.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.317+0000 7f3ea240f740 1 -- 172.21.15.90:0/3818920034 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ad7bbae450 msgr2=0x55ad7bbba330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:42.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.317+0000 7f3ea240f740 1 --2- 172.21.15.90:0/3818920034 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ad7bbae450 0x55ad7bbba330 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f3e9002c850 tx=0x7f3e9002cf10 comp rx=0 tx=0).stop 2024-04-02T14:34:42.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.317+0000 7f3ea240f740 1 -- 172.21.15.90:0/3818920034 shutdown_connections 2024-04-02T14:34:42.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.317+0000 7f3ea240f740 1 --2- 172.21.15.90:0/3818920034 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55ad7bbbf6e0 0x55ad7bbc0b50 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:42.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.317+0000 7f3ea240f740 1 --2- 172.21.15.90:0/3818920034 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ad7bbae450 0x55ad7bbba330 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:42.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.317+0000 7f3ea240f740 1 --2- 172.21.15.90:0/3818920034 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ad7bba92e0 0x55ad7bbb32f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:42.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.317+0000 7f3e86a00640 0 client.15156 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:34:42.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.317+0000 7f3ea240f740 1 --2- 172.21.15.90:0/3818920034 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ad7bba88f0 0x55ad7bbb2db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:42.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.317+0000 7f3ea240f740 1 -- 172.21.15.90:0/3818920034 >> 172.21.15.90:0/3818920034 conn(0x55ad7bba3e60 msgr2=0x55ad7bba3080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:34:42.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.317+0000 7f3ea240f740 1 -- 172.21.15.90:0/3818920034 shutdown_connections 2024-04-02T14:34:42.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.317+0000 7f3ea240f740 1 -- 172.21.15.90:0/3818920034 wait complete. 2024-04-02T14:34:42.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.317+0000 7f3ea240f740 20 client.15156 trim_cache size 0 max 16384 2024-04-02T14:34:42.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.336+0000 7f3ea240f740 1 Processor -- start 2024-04-02T14:34:42.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.337+0000 7f3ea240f740 1 -- start start 2024-04-02T14:34:42.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.337+0000 7f3ea240f740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ad7bba88f0 0x55ad7bba4240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:42.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.337+0000 7f3ea240f740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ad7bba92e0 0x55ad7bbba250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:42.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.337+0000 7f3ea240f740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ad7bbae450 0x55ad7bba27f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:42.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.337+0000 7f3ea240f740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55ad7bbbee90 con 0x55ad7bba88f0 2024-04-02T14:34:42.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.337+0000 7f3ea240f740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55ad7bbbf560 con 0x55ad7bbae450 2024-04-02T14:34:42.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.337+0000 7f3ea240f740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55ad7bbbfc30 con 0x55ad7bba92e0 2024-04-02T14:34:42.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.337+0000 7f3e97e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ad7bba92e0 0x55ad7bbba250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:42.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.337+0000 7f3e97e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ad7bba92e0 0x55ad7bbba250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:53098/0 (socket says 172.21.15.90:53098) 2024-04-02T14:34:42.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.337+0000 7f3e97e00640 1 -- 172.21.15.90:0/2564106888 learned_addr learned my addr 172.21.15.90:0/2564106888 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:34:42.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.337+0000 7f3e9d800640 1 --2- 172.21.15.90:0/2564106888 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ad7bbae450 0x55ad7bba27f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:42.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.337+0000 7f3e9ce00640 1 --2- 172.21.15.90:0/2564106888 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ad7bba88f0 0x55ad7bba4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:42.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.337+0000 7f3e97e00640 1 -- 172.21.15.90:0/2564106888 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ad7bbae450 msgr2=0x55ad7bba27f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:34:42.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.337+0000 7f3e97e00640 1 --2- 172.21.15.90:0/2564106888 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ad7bbae450 0x55ad7bba27f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:42.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.337+0000 7f3e97e00640 1 -- 172.21.15.90:0/2564106888 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ad7bba88f0 msgr2=0x55ad7bba4240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:34:42.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.337+0000 7f3e97e00640 1 --2- 172.21.15.90:0/2564106888 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ad7bba88f0 0x55ad7bba4240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:42.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.337+0000 7f3e97e00640 1 -- 172.21.15.90:0/2564106888 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ad7bbada30 con 0x55ad7bba92e0 2024-04-02T14:34:42.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.338+0000 7f3e97e00640 1 --2- 172.21.15.90:0/2564106888 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ad7bba92e0 0x55ad7bbba250 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f3e800d0950 tx=0x7f3e800d6530 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5d461141d117295a server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:34:42.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.338+0000 7f3e8d600640 1 -- 172.21.15.90:0/2564106888 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e8009c510 con 0x55ad7bba92e0 2024-04-02T14:34:42.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.338+0000 7f3e8d600640 1 -- 172.21.15.90:0/2564106888 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3e8009c9d0 con 0x55ad7bba92e0 2024-04-02T14:34:42.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.338+0000 7f3e8d600640 1 -- 172.21.15.90:0/2564106888 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e8009c330 con 0x55ad7bba92e0 2024-04-02T14:34:42.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.338+0000 7f3ea240f740 1 -- 172.21.15.90:0/2564106888 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ad7bba92e0 msgr2=0x55ad7bbba250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:42.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.338+0000 7f3ea240f740 1 --2- 172.21.15.90:0/2564106888 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ad7bba92e0 0x55ad7bbba250 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f3e800d0950 tx=0x7f3e800d6530 comp rx=0 tx=0).stop 2024-04-02T14:34:42.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.339+0000 7f3ea240f740 1 -- 172.21.15.90:0/2564106888 shutdown_connections 2024-04-02T14:34:42.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.339+0000 7f3ea240f740 1 --2- 172.21.15.90:0/2564106888 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ad7bbae450 0x55ad7bba27f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:42.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.339+0000 7f3ea240f740 1 --2- 172.21.15.90:0/2564106888 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ad7bba92e0 0x55ad7bbba250 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:42.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.339+0000 7f3ea240f740 1 --2- 172.21.15.90:0/2564106888 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ad7bba88f0 0x55ad7bba4240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:42.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.339+0000 7f3ea240f740 1 -- 172.21.15.90:0/2564106888 >> 172.21.15.90:0/2564106888 conn(0x55ad7bba3e60 msgr2=0x55ad7bb9ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:34:42.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.339+0000 7f3ea240f740 1 -- 172.21.15.90:0/2564106888 shutdown_connections 2024-04-02T14:34:42.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.339+0000 7f3ea240f740 1 -- 172.21.15.90:0/2564106888 wait complete. 2024-04-02T14:34:42.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.339+0000 7f3ea240f740 1 -- start start 2024-04-02T14:34:42.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.339+0000 7f3ea240f740 1 Processor -- start 2024-04-02T14:34:42.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.340+0000 7f3ea240f740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ad7bba88f0 0x55ad7bbb8490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:42.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.340+0000 7f3ea240f740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ad7bba92e0 0x55ad7bbb89d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:42.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.340+0000 7f3ea240f740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ad7bbae450 0x55ad7bbb8fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:42.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.340+0000 7f3ea240f740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55ad7bbc09d0 con 0x55ad7bba88f0 2024-04-02T14:34:42.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.340+0000 7f3ea240f740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55ad7bbc0300 con 0x55ad7bba92e0 2024-04-02T14:34:42.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.340+0000 7f3ea240f740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55ad7bbc10a0 con 0x55ad7bbae450 2024-04-02T14:34:42.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.340+0000 7f3ea240f740 10 client.0 _ll_register_callbacks cb 0x7f3e9002c670 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:34:42.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.340+0000 7f3e9ce00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ad7bba88f0 0x55ad7bbb8490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:42.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.340+0000 7f3e9ce00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ad7bba88f0 0x55ad7bbb8490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:36704/0 (socket says 172.21.15.90:36704) 2024-04-02T14:34:42.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.340+0000 7f3e9ce00640 1 -- 172.21.15.90:0/2110821836 learned_addr learned my addr 172.21.15.90:0/2110821836 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:34:42.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.340+0000 7f3e97e00640 1 --2- 172.21.15.90:0/2110821836 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ad7bba92e0 0x55ad7bbb89d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:42.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.340+0000 7f3e9d800640 1 --2- 172.21.15.90:0/2110821836 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ad7bbae450 0x55ad7bbb8fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:42.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.340+0000 7f3e9ce00640 1 -- 172.21.15.90:0/2110821836 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ad7bbae450 msgr2=0x55ad7bbb8fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:34:42.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.340+0000 7f3e9ce00640 1 --2- 172.21.15.90:0/2110821836 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ad7bbae450 0x55ad7bbb8fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:42.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.340+0000 7f3e9ce00640 1 -- 172.21.15.90:0/2110821836 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ad7bba92e0 msgr2=0x55ad7bbb89d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:34:42.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.340+0000 7f3e9ce00640 1 --2- 172.21.15.90:0/2110821836 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ad7bba92e0 0x55ad7bbb89d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:42.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.340+0000 7f3e9ce00640 1 -- 172.21.15.90:0/2110821836 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ad7bbb9810 con 0x55ad7bba88f0 2024-04-02T14:34:42.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.340+0000 7f3e97e00640 1 --2- 172.21.15.90:0/2110821836 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ad7bba92e0 0x55ad7bbb89d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:34:42.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.341+0000 7f3e9ce00640 1 --2- 172.21.15.90:0/2110821836 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ad7bba88f0 0x55ad7bbb8490 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x55ad7bbbfb50 tx=0x7f3e88004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68fdf67f4e8df1a6 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:34:42.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.341+0000 7f3e86a00640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-02T14:34:42.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.341+0000 7f3e86a00640 1 -- 172.21.15.90:0/2110821836 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e8801eca0 con 0x55ad7bba88f0 2024-04-02T14:34:42.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.341+0000 7f3e86a00640 1 -- 172.21.15.90:0/2110821836 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8248+0+0 (secure 0 0 0) 0x7f3e8802a3b0 con 0x55ad7bba88f0 2024-04-02T14:34:42.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.341+0000 7f3ea240f740 1 -- 172.21.15.90:0/2110821836 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3e8009c9d0 con 0x55ad7bba88f0 2024-04-02T14:34:42.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.341+0000 7f3e86a00640 1 -- 172.21.15.90:0/2110821836 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f3e8801ee40 con 0x55ad7bba88f0 2024-04-02T14:34:42.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.341+0000 7f3ea240f740 10 client.15159 fetch_fsmap learned FSMap version 26 2024-04-02T14:34:42.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.341+0000 7f3ea240f740 1 -- 172.21.15.90:0/2110821836 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55ad7bc25970 con 0x55ad7bba88f0 2024-04-02T14:34:42.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.341+0000 7f3e86a00640 1 -- 172.21.15.90:0/2110821836 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3e8009c9d0 con 0x55ad7bba88f0 2024-04-02T14:34:42.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.342+0000 7f3e86a00640 1 -- 172.21.15.90:0/2110821836 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f3e88033cb0 con 0x55ad7bba88f0 2024-04-02T14:34:42.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.342+0000 7f3ea240f740 10 client.15159 fetch_fsmap finished waiting for FSMap version 26 2024-04-02T14:34:42.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.342+0000 7f3ea240f740 10 client.15159 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-02T14:34:42.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.342+0000 7f3ea240f740 20 client.15159 populate_metadata read hostname 'smithi090' 2024-04-02T14:34:42.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.342+0000 7f3ea240f740 1 --2- 172.21.15.90:0/2110821836 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55ad7bc2a510 0x55ad7bc29220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:42.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.342+0000 7f3ea240f740 4 client.15159 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-02T14:34:42.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.342+0000 7f3ea240f740 1 -- 172.21.15.90:0/2110821836 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "status"}) -- 0x55ad7bb158f0 con 0x55ad7bc2a510 2024-04-02T14:34:42.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.342+0000 7f3e97e00640 1 --2- 172.21.15.90:0/2110821836 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55ad7bc2a510 0x55ad7bc29220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:42.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.343+0000 7f3e97e00640 1 --2- 172.21.15.90:0/2110821836 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55ad7bc2a510 0x55ad7bc29220 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3575b79b30fc5f96 server_cookie=469848ec5557273a in_seq=0 out_seq=0 2024-04-02T14:34:42.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.343+0000 7f3e86a00640 10 client.15159 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:34:42.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.343+0000 7f3e86a00640 1 -- 172.21.15.90:0/2110821836 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x55ad7bb158f0 con 0x55ad7bc2a510 2024-04-02T14:34:42.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.343+0000 7f3e86a00640 10 client.15159 handle_command_reply: tid=0 2024-04-02T14:34:42.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.343+0000 7f3ea240f740 1 client.15159 shutdown 2024-04-02T14:34:42.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.343+0000 7f3ea240f740 1 -- 172.21.15.90:0/2110821836 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ad7bba88f0 msgr2=0x55ad7bbb8490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:42.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.343+0000 7f3ea240f740 1 --2- 172.21.15.90:0/2110821836 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ad7bba88f0 0x55ad7bbb8490 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x55ad7bbbfb50 tx=0x7f3e88004a40 comp rx=0 tx=0).stop 2024-04-02T14:34:42.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.343+0000 7f3ea240f740 1 -- 172.21.15.90:0/2110821836 shutdown_connections 2024-04-02T14:34:42.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.343+0000 7f3ea240f740 1 --2- 172.21.15.90:0/2110821836 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55ad7bc2a510 0x55ad7bc29220 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:42.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.343+0000 7f3ea240f740 1 --2- 172.21.15.90:0/2110821836 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ad7bbae450 0x55ad7bbb8fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:42.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.343+0000 7f3ea240f740 1 --2- 172.21.15.90:0/2110821836 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ad7bba92e0 0x55ad7bbb89d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:42.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.343+0000 7f3e86a00640 0 client.15159 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:34:42.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.343+0000 7f3ea240f740 1 --2- 172.21.15.90:0/2110821836 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ad7bba88f0 0x55ad7bbb8490 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:42.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.343+0000 7f3ea240f740 1 -- 172.21.15.90:0/2110821836 >> 172.21.15.90:0/2110821836 conn(0x55ad7bba3e60 msgr2=0x55ad7bba4240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:34:42.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.343+0000 7f3ea240f740 1 -- 172.21.15.90:0/2110821836 shutdown_connections 2024-04-02T14:34:42.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.343+0000 7f3ea240f740 1 -- 172.21.15.90:0/2110821836 wait complete. 2024-04-02T14:34:42.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.344+0000 7f3ea240f740 20 client.15159 trim_cache size 0 max 16384 2024-04-02T14:34:42.350 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-02T14:34:42.350 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_fsid": "6449448a-f0fc-11ee-b647-cb9ed24678a4", 2024-04-02T14:34:42.350 INFO:teuthology.orchestra.run.smithi090.stdout: "whoami": 0, 2024-04-02T14:34:42.351 INFO:teuthology.orchestra.run.smithi090.stdout: "id": 24590, 2024-04-02T14:34:42.351 INFO:teuthology.orchestra.run.smithi090.stdout: "want_state": "up:active", 2024-04-02T14:34:42.351 INFO:teuthology.orchestra.run.smithi090.stdout: "state": "up:active", 2024-04-02T14:34:42.351 INFO:teuthology.orchestra.run.smithi090.stdout: "fs_name": "cephfs", 2024-04-02T14:34:42.351 INFO:teuthology.orchestra.run.smithi090.stdout: "rank_uptime": 139.49501167299999, 2024-04-02T14:34:42.351 INFO:teuthology.orchestra.run.smithi090.stdout: "mdsmap_epoch": 26, 2024-04-02T14:34:42.351 INFO:teuthology.orchestra.run.smithi090.stdout: "osdmap_epoch": 90, 2024-04-02T14:34:42.351 INFO:teuthology.orchestra.run.smithi090.stdout: "osdmap_epoch_barrier": 88, 2024-04-02T14:34:42.351 INFO:teuthology.orchestra.run.smithi090.stdout: "uptime": 145.68012982400001 2024-04-02T14:34:42.351 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-02T14:34:42.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.344+0000 7f3e9f800640 1 -- 172.21.15.90:0/930871328 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f3e7c0a3c30 msgr2=0x7f3e7c0a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:42.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.344+0000 7f3e9f800640 1 --2- 172.21.15.90:0/930871328 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f3e7c0a3c30 0x7f3e7c0a60f0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x55ad7bba4dd0 tx=0x7f3e88009760 comp rx=0 tx=0).stop 2024-04-02T14:34:42.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.344+0000 7f3e9f800640 1 -- 172.21.15.90:0/930871328 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3e98075ae0 msgr2=0x7f3e9819a4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:42.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.344+0000 7f3e9f800640 1 --2- 172.21.15.90:0/930871328 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3e98075ae0 0x7f3e9819a4f0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f3e8000e9f0 tx=0x7f3e8000eeb0 comp rx=0 tx=0).stop 2024-04-02T14:34:42.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.344+0000 7f3e9f800640 1 -- 172.21.15.90:0/930871328 shutdown_connections 2024-04-02T14:34:42.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.344+0000 7f3e9f800640 1 --2- 172.21.15.90:0/930871328 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f3e7c0a3c30 0x7f3e7c0a60f0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:42.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.344+0000 7f3e9f800640 1 --2- 172.21.15.90:0/930871328 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3e981099a0 0x7f3e981a1560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:42.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.344+0000 7f3e9f800640 1 --2- 172.21.15.90:0/930871328 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3e98075ae0 0x7f3e9819a4f0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:42.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.344+0000 7f3e9f800640 1 --2- 172.21.15.90:0/930871328 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3e98074500 0x7f3e98199fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:42.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.344+0000 7f3e9f800640 1 -- 172.21.15.90:0/930871328 >> 172.21.15.90:0/930871328 conn(0x7f3e980fd0a0 msgr2=0x7f3e98105930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:34:42.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.344+0000 7f3e9f800640 1 -- 172.21.15.90:0/930871328 shutdown_connections 2024-04-02T14:34:42.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.344+0000 7f3e9f800640 1 -- 172.21.15.90:0/930871328 wait complete. 2024-04-02T14:34:42.360 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:34:41 smithi092 ceph-mon[36497]: pgmap v383: 97 pgs: 97 active+clean; 644 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-02T14:34:42.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:34:41 smithi092 ceph-mon[36497]: from='client.? 172.21.15.90:0/3073919238' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-02T14:34:42.365 INFO:teuthology.run_tasks:Running task sequential... 2024-04-02T14:34:42.377 INFO:teuthology.task.sequential:In sequential, running task sequential... 2024-04-02T14:34:42.378 INFO:teuthology.task.sequential:In sequential, running task print... 2024-04-02T14:34:42.378 INFO:teuthology.task.print:Enabling mgr modules 2024-04-02T14:34:42.378 INFO:teuthology.task.sequential:In sequential, running task exec... 2024-04-02T14:34:42.379 INFO:teuthology.task.exec:Executing custom commands... 2024-04-02T14:34:42.379 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi090.front.sepia.ceph.com 2024-04-02T14:34:42.379 DEBUG:teuthology.orchestra.run.smithi090:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph mgr module enable snap_schedule' 2024-04-02T14:34:42.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.576+0000 7fa3e2200640 1 -- 172.21.15.90:0/3539061589 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa3dc0764c0 msgr2=0x7fa3dc0768e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:42.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.576+0000 7fa3e2200640 1 --2- 172.21.15.90:0/3539061589 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa3dc0764c0 0x7fa3dc0768e0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fa3cc009900 tx=0x7fa3cc02bc30 comp rx=0 tx=0).stop 2024-04-02T14:34:42.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.576+0000 7fa3e2200640 1 -- 172.21.15.90:0/3539061589 shutdown_connections 2024-04-02T14:34:42.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.576+0000 7fa3e2200640 1 --2- 172.21.15.90:0/3539061589 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa3dc0780a0 0x7fa3dc078940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:42.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.576+0000 7fa3e2200640 1 --2- 172.21.15.90:0/3539061589 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa3dc077700 0x7fa3dc077b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:42.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.576+0000 7fa3e2200640 1 --2- 172.21.15.90:0/3539061589 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa3dc0764c0 0x7fa3dc0768e0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:42.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.576+0000 7fa3e2200640 1 -- 172.21.15.90:0/3539061589 >> 172.21.15.90:0/3539061589 conn(0x7fa3dc0ff2c0 msgr2=0x7fa3dc101700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:34:42.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.576+0000 7fa3e2200640 1 -- 172.21.15.90:0/3539061589 shutdown_connections 2024-04-02T14:34:42.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.576+0000 7fa3e2200640 1 -- 172.21.15.90:0/3539061589 wait complete. 2024-04-02T14:34:42.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.577+0000 7fa3e2200640 1 Processor -- start 2024-04-02T14:34:42.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.577+0000 7fa3e2200640 1 -- start start 2024-04-02T14:34:42.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.577+0000 7fa3e2200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa3dc0764c0 0x7fa3dc19e420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:42.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.577+0000 7fa3e2200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa3dc077700 0x7fa3dc19e960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:42.582 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.577+0000 7fa3e2200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa3dc0780a0 0x7fa3dc1a5980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:42.582 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.577+0000 7fa3e2200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fa3dc113de0 con 0x7fa3dc0764c0 2024-04-02T14:34:42.582 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.577+0000 7fa3e2200640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fa3dc113c60 con 0x7fa3dc077700 2024-04-02T14:34:42.582 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.577+0000 7fa3e2200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fa3dc113f60 con 0x7fa3dc0780a0 2024-04-02T14:34:42.582 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.578+0000 7fa3db400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa3dc0764c0 0x7fa3dc19e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:42.582 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.578+0000 7fa3db400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa3dc0764c0 0x7fa3dc19e420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:36728/0 (socket says 172.21.15.90:36728) 2024-04-02T14:34:42.582 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.578+0000 7fa3db400640 1 -- 172.21.15.90:0/2056420245 learned_addr learned my addr 172.21.15.90:0/2056420245 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:34:42.582 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.578+0000 7fa3daa00640 1 --2- 172.21.15.90:0/2056420245 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa3dc077700 0x7fa3dc19e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:42.582 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.578+0000 7fa3dbe00640 1 --2- 172.21.15.90:0/2056420245 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa3dc0780a0 0x7fa3dc1a5980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:42.582 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.578+0000 7fa3db400640 1 -- 172.21.15.90:0/2056420245 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa3dc0780a0 msgr2=0x7fa3dc1a5980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:42.582 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.578+0000 7fa3db400640 1 --2- 172.21.15.90:0/2056420245 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa3dc0780a0 0x7fa3dc1a5980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:42.582 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.578+0000 7fa3db400640 1 -- 172.21.15.90:0/2056420245 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa3dc077700 msgr2=0x7fa3dc19e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:42.582 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.578+0000 7fa3db400640 1 --2- 172.21.15.90:0/2056420245 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa3dc077700 0x7fa3dc19e960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:42.582 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.578+0000 7fa3db400640 1 -- 172.21.15.90:0/2056420245 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3dc1a5ff0 con 0x7fa3dc0764c0 2024-04-02T14:34:42.583 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.578+0000 7fa3db400640 1 --2- 172.21.15.90:0/2056420245 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa3dc0764c0 0x7fa3dc19e420 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fa3cc009230 tx=0x7fa3cc02c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:34:42.583 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.579+0000 7fa3d3e00640 1 -- 172.21.15.90:0/2056420245 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3cc041070 con 0x7fa3dc0764c0 2024-04-02T14:34:42.583 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.579+0000 7fa3d3e00640 1 -- 172.21.15.90:0/2056420245 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa3cc004450 con 0x7fa3dc0764c0 2024-04-02T14:34:42.583 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.579+0000 7fa3d3e00640 1 -- 172.21.15.90:0/2056420245 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3cc003e20 con 0x7fa3dc0764c0 2024-04-02T14:34:42.583 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.579+0000 7fa3e2200640 1 -- 172.21.15.90:0/2056420245 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa3dc1a6280 con 0x7fa3dc0764c0 2024-04-02T14:34:42.583 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.579+0000 7fa3e2200640 1 -- 172.21.15.90:0/2056420245 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3dc1a6720 con 0x7fa3dc0764c0 2024-04-02T14:34:42.583 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.580+0000 7fa3e2200640 1 -- 172.21.15.90:0/2056420245 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa3dc1106c0 con 0x7fa3dc0764c0 2024-04-02T14:34:42.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.582+0000 7fa3d3e00640 1 -- 172.21.15.90:0/2056420245 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fa3cc044070 con 0x7fa3dc0764c0 2024-04-02T14:34:42.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.583+0000 7fa3d3e00640 1 --2- 172.21.15.90:0/2056420245 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fa3c00a3cd0 0x7fa3c00a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:42.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.584+0000 7fa3d3e00640 1 -- 172.21.15.90:0/2056420245 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8248+0+0 (secure 0 0 0) 0x7fa3cc0f1f60 con 0x7fa3dc0764c0 2024-04-02T14:34:42.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.584+0000 7fa3daa00640 1 --2- 172.21.15.90:0/2056420245 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fa3c00a3cd0 0x7fa3c00a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:42.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.584+0000 7fa3daa00640 1 --2- 172.21.15.90:0/2056420245 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fa3c00a3cd0 0x7fa3c00a6190 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fa3c80095f0 tx=0x7fa3c8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:34:42.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.587+0000 7fa3d3e00640 1 -- 172.21.15.90:0/2056420245 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7fa3cc0be280 con 0x7fa3dc0764c0 2024-04-02T14:34:42.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.843+0000 7fa3e2200640 1 -- 172.21.15.90:0/2056420245 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) -- 0x7fa3dc02d5d0 con 0x7fa3dc0764c0 2024-04-02T14:34:42.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.914+0000 7fa3d3e00640 1 -- 172.21.15.90:0/2056420245 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v19) ==== 92+0+0 (secure 0 0 0) 0x7fa3cc0c2580 con 0x7fa3dc0764c0 2024-04-02T14:34:42.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.917+0000 7fa3e2200640 1 -- 172.21.15.90:0/2056420245 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fa3c00a3cd0 msgr2=0x7fa3c00a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:42.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.917+0000 7fa3e2200640 1 --2- 172.21.15.90:0/2056420245 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fa3c00a3cd0 0x7fa3c00a6190 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fa3c80095f0 tx=0x7fa3c8005f90 comp rx=0 tx=0).stop 2024-04-02T14:34:42.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.917+0000 7fa3e2200640 1 -- 172.21.15.90:0/2056420245 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa3dc0764c0 msgr2=0x7fa3dc19e420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:42.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.917+0000 7fa3e2200640 1 --2- 172.21.15.90:0/2056420245 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa3dc0764c0 0x7fa3dc19e420 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fa3cc009230 tx=0x7fa3cc02c650 comp rx=0 tx=0).stop 2024-04-02T14:34:42.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.917+0000 7fa3e2200640 1 -- 172.21.15.90:0/2056420245 shutdown_connections 2024-04-02T14:34:42.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.917+0000 7fa3e2200640 1 --2- 172.21.15.90:0/2056420245 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fa3c00a3cd0 0x7fa3c00a6190 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:42.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.917+0000 7fa3e2200640 1 --2- 172.21.15.90:0/2056420245 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa3dc0780a0 0x7fa3dc1a5980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:42.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.917+0000 7fa3e2200640 1 --2- 172.21.15.90:0/2056420245 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa3dc077700 0x7fa3dc19e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:42.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.917+0000 7fa3e2200640 1 --2- 172.21.15.90:0/2056420245 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa3dc0764c0 0x7fa3dc19e420 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:42.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.917+0000 7fa3e2200640 1 -- 172.21.15.90:0/2056420245 >> 172.21.15.90:0/2056420245 conn(0x7fa3dc0ff2c0 msgr2=0x7fa3dc100dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:34:42.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.917+0000 7fa3e2200640 1 -- 172.21.15.90:0/2056420245 shutdown_connections 2024-04-02T14:34:42.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:42.917+0000 7fa3e2200640 1 -- 172.21.15.90:0/2056420245 wait complete. 2024-04-02T14:34:42.947 DEBUG:teuthology.orchestra.run.smithi090:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/allow_m_granularity true' 2024-04-02T14:34:43.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:43.128+0000 7ff022e00640 1 -- 172.21.15.90:0/3557858525 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff01c075b40 msgr2=0x7ff01c073f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:43.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:43.128+0000 7ff022e00640 1 --2- 172.21.15.90:0/3557858525 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff01c075b40 0x7ff01c073f90 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7ff010009900 tx=0x7ff01002bc40 comp rx=0 tx=0).stop 2024-04-02T14:34:43.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:43.128+0000 7ff022e00640 1 -- 172.21.15.90:0/3557858525 shutdown_connections 2024-04-02T14:34:43.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:43.128+0000 7ff022e00640 1 --2- 172.21.15.90:0/3557858525 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff01c109830 0x7ff01c10bcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:43.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:43.128+0000 7ff022e00640 1 --2- 172.21.15.90:0/3557858525 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff01c074560 0x7ff01c0749e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:43.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:43.128+0000 7ff022e00640 1 --2- 172.21.15.90:0/3557858525 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff01c075b40 0x7ff01c073f90 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:43.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:43.128+0000 7ff022e00640 1 -- 172.21.15.90:0/3557858525 >> 172.21.15.90:0/3557858525 conn(0x7ff01c0fcee0 msgr2=0x7ff01c0ff320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:34:43.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:43.129+0000 7ff022e00640 1 -- 172.21.15.90:0/3557858525 shutdown_connections 2024-04-02T14:34:43.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:43.129+0000 7ff022e00640 1 -- 172.21.15.90:0/3557858525 wait complete. 2024-04-02T14:34:43.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:43.129+0000 7ff022e00640 1 Processor -- start 2024-04-02T14:34:43.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:43.130+0000 7ff022e00640 1 -- start start 2024-04-02T14:34:43.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:43.130+0000 7ff022e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff01c074560 0x7ff01c199e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:43.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:43.130+0000 7ff022e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff01c075b40 0x7ff01c19a360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:43.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:43.130+0000 7ff022e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff01c109830 0x7ff01c1a13d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:43.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:43.130+0000 7ff022e00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7ff01c10f800 con 0x7ff01c074560 2024-04-02T14:34:43.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:43.130+0000 7ff022e00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7ff01c10f680 con 0x7ff01c109830 2024-04-02T14:34:43.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:43.130+0000 7ff022e00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7ff01c10f980 con 0x7ff01c075b40 2024-04-02T14:34:43.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:43.130+0000 7ff021a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff01c074560 0x7ff01c199e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:43.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:43.130+0000 7ff021000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff01c075b40 0x7ff01c19a360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:43.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:43.130+0000 7ff021a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff01c074560 0x7ff01c199e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:36782/0 (socket says 172.21.15.90:36782) 2024-04-02T14:34:43.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:43.130+0000 7ff021000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff01c075b40 0x7ff01c19a360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:53160/0 (socket says 172.21.15.90:53160) 2024-04-02T14:34:43.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:43.130+0000 7ff021a00640 1 -- 172.21.15.90:0/1756069163 learned_addr learned my addr 172.21.15.90:0/1756069163 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:34:43.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:43.130+0000 7ff022400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff01c109830 0x7ff01c1a13d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:43.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:43.131+0000 7ff021a00640 1 -- 172.21.15.90:0/1756069163 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff01c075b40 msgr2=0x7ff01c19a360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:43.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:43.131+0000 7ff021a00640 1 --2- 172.21.15.90:0/1756069163 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff01c075b40 0x7ff01c19a360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:43.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:43.131+0000 7ff021a00640 1 -- 172.21.15.90:0/1756069163 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff01c109830 msgr2=0x7ff01c1a13d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:43.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:43.131+0000 7ff021a00640 1 --2- 172.21.15.90:0/1756069163 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff01c109830 0x7ff01c1a13d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:43.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:43.131+0000 7ff021a00640 1 -- 172.21.15.90:0/1756069163 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff01c1a1a40 con 0x7ff01c074560 2024-04-02T14:34:43.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:43.131+0000 7ff021000640 1 --2- 172.21.15.90:0/1756069163 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff01c075b40 0x7ff01c19a360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:34:43.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:43.131+0000 7ff022400640 1 --2- 172.21.15.90:0/1756069163 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff01c109830 0x7ff01c1a13d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:34:43.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:43.131+0000 7ff021a00640 1 --2- 172.21.15.90:0/1756069163 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff01c074560 0x7ff01c199e20 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7ff01002c8f0 tx=0x7ff01002c660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:34:43.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:43.131+0000 7ff01a000640 1 -- 172.21.15.90:0/1756069163 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff010041070 con 0x7ff01c074560 2024-04-02T14:34:43.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:43.131+0000 7ff022e00640 1 -- 172.21.15.90:0/1756069163 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff01c1a1cd0 con 0x7ff01c074560 2024-04-02T14:34:43.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:43.131+0000 7ff022e00640 1 -- 172.21.15.90:0/1756069163 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff01c1a21c0 con 0x7ff01c074560 2024-04-02T14:34:43.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:43.132+0000 7ff01a000640 1 -- 172.21.15.90:0/1756069163 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff010004450 con 0x7ff01c074560 2024-04-02T14:34:43.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:43.132+0000 7ff01a000640 1 -- 172.21.15.90:0/1756069163 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff010034440 con 0x7ff01c074560 2024-04-02T14:34:43.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:43.133+0000 7ff022e00640 1 -- 172.21.15.90:0/1756069163 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff01c108560 con 0x7ff01c074560 2024-04-02T14:34:43.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:43.135+0000 7ff01a000640 1 -- 172.21.15.90:0/1756069163 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 19) ==== 138663+0+0 (secure 0 0 0) 0x7ff010044070 con 0x7ff01c074560 2024-04-02T14:34:43.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:43.137+0000 7ff01a000640 1 --2- 172.21.15.90:0/1756069163 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7ff0040a3c70 0x7ff0040a6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:43.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:43.137+0000 7ff021000640 1 -- 172.21.15.90:0/1756069163 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7ff0040a3c70 msgr2=0x7ff0040a6130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.90:6800/4266054817 2024-04-02T14:34:43.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:43.137+0000 7ff01a000640 1 -- 172.21.15.90:0/1756069163 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8248+0+0 (secure 0 0 0) 0x7ff0100f1f70 con 0x7ff01c074560 2024-04-02T14:34:43.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:43.137+0000 7ff021000640 1 --2- 172.21.15.90:0/1756069163 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7ff0040a3c70 0x7ff0040a6130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T14:34:43.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:43.140+0000 7ff01a000640 1 -- 172.21.15.90:0/1756069163 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7ff0100be290 con 0x7ff01c074560 2024-04-02T14:34:43.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:34:42 smithi177 ceph-mon[36431]: from='client.? 172.21.15.90:0/2056420245' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-04-02T14:34:43.187 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 02 14:34:42 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-z[38299]: ignoring --setuser ceph since I am not root 2024-04-02T14:34:43.187 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 02 14:34:42 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-z[38299]: ignoring --setgroup ceph since I am not root 2024-04-02T14:34:43.187 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 02 14:34:42 smithi177 ceph-mgr[38303]: -- 172.21.15.177:0/2856224997 <== mon.1 v2:172.21.15.177:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55d49d2205a0 con 0x55d49d1f4c00 2024-04-02T14:34:43.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:34:42 smithi090 ceph-mon[31674]: from='client.? 172.21.15.90:0/2056420245' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-04-02T14:34:43.189 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 02 14:34:43 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-x[31910]: ignoring --setuser ceph since I am not root 2024-04-02T14:34:43.189 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 02 14:34:43 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-x[31910]: ignoring --setgroup ceph since I am not root 2024-04-02T14:34:43.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:43.338+0000 7ff021000640 1 -- 172.21.15.90:0/1756069163 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7ff0040a3c70 msgr2=0x7ff0040a6130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.90:6800/4266054817 2024-04-02T14:34:43.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:43.338+0000 7ff021000640 1 --2- 172.21.15.90:0/1756069163 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7ff0040a3c70 0x7ff0040a6130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T14:34:43.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:43.341+0000 7ff022e00640 1 -- 172.21.15.90:0/1756069163 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) -- 0x7ff01c062700 con 0x7ff01c074560 2024-04-02T14:34:43.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:43.347+0000 7ff01a000640 1 -- 172.21.15.90:0/1756069163 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v33)=0 v33) ==== 138+0+0 (secure 0 0 0) 0x7ff0100c2590 con 0x7ff01c074560 2024-04-02T14:34:43.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:43.349+0000 7ff022e00640 1 -- 172.21.15.90:0/1756069163 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7ff0040a3c70 msgr2=0x7ff0040a6130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T14:34:43.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:43.349+0000 7ff022e00640 1 --2- 172.21.15.90:0/1756069163 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7ff0040a3c70 0x7ff0040a6130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:43.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:43.349+0000 7ff022e00640 1 -- 172.21.15.90:0/1756069163 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff01c074560 msgr2=0x7ff01c199e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:43.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:43.349+0000 7ff022e00640 1 --2- 172.21.15.90:0/1756069163 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff01c074560 0x7ff01c199e20 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7ff01002c8f0 tx=0x7ff01002c660 comp rx=0 tx=0).stop 2024-04-02T14:34:43.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:43.349+0000 7ff022e00640 1 -- 172.21.15.90:0/1756069163 shutdown_connections 2024-04-02T14:34:43.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:43.349+0000 7ff022e00640 1 --2- 172.21.15.90:0/1756069163 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7ff0040a3c70 0x7ff0040a6130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:43.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:43.349+0000 7ff022e00640 1 --2- 172.21.15.90:0/1756069163 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff01c109830 0x7ff01c1a13d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:43.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:43.349+0000 7ff022e00640 1 --2- 172.21.15.90:0/1756069163 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff01c075b40 0x7ff01c19a360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:43.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:43.349+0000 7ff022e00640 1 --2- 172.21.15.90:0/1756069163 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff01c074560 0x7ff01c199e20 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:43.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:43.349+0000 7ff022e00640 1 -- 172.21.15.90:0/1756069163 >> 172.21.15.90:0/1756069163 conn(0x7ff01c0fcee0 msgr2=0x7ff01c105730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:34:43.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:43.349+0000 7ff022e00640 1 -- 172.21.15.90:0/1756069163 shutdown_connections 2024-04-02T14:34:43.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:43.350+0000 7ff022e00640 1 -- 172.21.15.90:0/1756069163 wait complete. 2024-04-02T14:34:43.360 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:34:42 smithi092 ceph-mon[36497]: from='client.? 172.21.15.90:0/2056420245' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-04-02T14:34:43.361 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 02 14:34:42 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-y[37879]: ignoring --setuser ceph since I am not root 2024-04-02T14:34:43.361 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 02 14:34:42 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-y[37879]: ignoring --setgroup ceph since I am not root 2024-04-02T14:34:43.361 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 02 14:34:42 smithi092 ceph-mgr[37884]: -- 172.21.15.92:0/3187145146 <== mon.2 v2:172.21.15.92:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55ba361e85a0 con 0x55ba361bcc00 2024-04-02T14:34:43.368 DEBUG:teuthology.orchestra.run.smithi090:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/dump_on_update true' 2024-04-02T14:34:43.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:43.552+0000 7f9d76000640 1 -- 172.21.15.90:0/2478760759 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9d70103d60 msgr2=0x7f9d70104180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:43.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:43.552+0000 7f9d76000640 1 --2- 172.21.15.90:0/2478760759 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9d70103d60 0x7f9d70104180 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f9d60009900 tx=0x7f9d6002bc30 comp rx=0 tx=0).stop 2024-04-02T14:34:43.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:43.552+0000 7f9d76000640 1 -- 172.21.15.90:0/2478760759 shutdown_connections 2024-04-02T14:34:43.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:43.552+0000 7f9d76000640 1 --2- 172.21.15.90:0/2478760759 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9d70105910 0x7f9d7010c1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:43.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:43.552+0000 7f9d76000640 1 --2- 172.21.15.90:0/2478760759 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9d70104f50 0x7f9d701053d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:43.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:43.552+0000 7f9d76000640 1 --2- 172.21.15.90:0/2478760759 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9d70103d60 0x7f9d70104180 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:43.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:43.552+0000 7f9d76000640 1 -- 172.21.15.90:0/2478760759 >> 172.21.15.90:0/2478760759 conn(0x7f9d700ff2f0 msgr2=0x7f9d70101710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:34:43.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:43.552+0000 7f9d76000640 1 -- 172.21.15.90:0/2478760759 shutdown_connections 2024-04-02T14:34:43.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:43.552+0000 7f9d76000640 1 -- 172.21.15.90:0/2478760759 wait complete. 2024-04-02T14:34:43.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:43.553+0000 7f9d76000640 1 Processor -- start 2024-04-02T14:34:43.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:43.553+0000 7f9d76000640 1 -- start start 2024-04-02T14:34:43.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:43.554+0000 7f9d76000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9d70103d60 0x7f9d70199fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:43.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:43.554+0000 7f9d76000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9d70104f50 0x7f9d7019a520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:43.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:43.554+0000 7f9d76000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9d70105910 0x7f9d701a1590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:43.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:43.554+0000 7f9d76000640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f9d7010f9d0 con 0x7f9d70105910 2024-04-02T14:34:43.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:43.554+0000 7f9d76000640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f9d7010f850 con 0x7f9d70103d60 2024-04-02T14:34:43.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:43.554+0000 7f9d76000640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f9d7010fb50 con 0x7f9d70104f50 2024-04-02T14:34:43.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:43.554+0000 7f9d6ea00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9d70104f50 0x7f9d7019a520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:43.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:43.554+0000 7f9d6f400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9d70103d60 0x7f9d70199fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:43.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:43.554+0000 7f9d6fe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9d70105910 0x7f9d701a1590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:43.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:43.554+0000 7f9d6f400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9d70103d60 0x7f9d70199fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:60680/0 (socket says 172.21.15.90:60680) 2024-04-02T14:34:43.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:43.554+0000 7f9d6f400640 1 -- 172.21.15.90:0/939286726 learned_addr learned my addr 172.21.15.90:0/939286726 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:34:43.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:43.554+0000 7f9d6ea00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9d70104f50 0x7f9d7019a520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:53166/0 (socket says 172.21.15.90:53166) 2024-04-02T14:34:43.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:43.554+0000 7f9d6f400640 1 -- 172.21.15.90:0/939286726 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9d70104f50 msgr2=0x7f9d7019a520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:43.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:43.554+0000 7f9d6f400640 1 --2- 172.21.15.90:0/939286726 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9d70104f50 0x7f9d7019a520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:43.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:43.554+0000 7f9d6f400640 1 -- 172.21.15.90:0/939286726 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9d70105910 msgr2=0x7f9d701a1590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:43.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:43.554+0000 7f9d6f400640 1 --2- 172.21.15.90:0/939286726 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9d70105910 0x7f9d701a1590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:43.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:43.554+0000 7f9d6f400640 1 -- 172.21.15.90:0/939286726 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d701a1c00 con 0x7f9d70103d60 2024-04-02T14:34:43.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:43.555+0000 7f9d6f400640 1 --2- 172.21.15.90:0/939286726 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9d70103d60 0x7f9d70199fe0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f9d60000c00 tx=0x7f9d6002c640 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:34:43.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:43.555+0000 7f9d67e00640 1 -- 172.21.15.90:0/939286726 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d60041070 con 0x7f9d70103d60 2024-04-02T14:34:43.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:43.555+0000 7f9d67e00640 1 -- 172.21.15.90:0/939286726 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9d60004450 con 0x7f9d70103d60 2024-04-02T14:34:43.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:43.555+0000 7f9d76000640 1 -- 172.21.15.90:0/939286726 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9d701a1e90 con 0x7f9d70103d60 2024-04-02T14:34:43.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:43.555+0000 7f9d67e00640 1 -- 172.21.15.90:0/939286726 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d60034440 con 0x7f9d70103d60 2024-04-02T14:34:43.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:43.555+0000 7f9d76000640 1 -- 172.21.15.90:0/939286726 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9d701a22f0 con 0x7f9d70103d60 2024-04-02T14:34:43.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:43.556+0000 7f9d76000640 1 -- 172.21.15.90:0/939286726 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9d7010b020 con 0x7f9d70103d60 2024-04-02T14:34:43.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:43.558+0000 7f9d67e00640 1 -- 172.21.15.90:0/939286726 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 19) ==== 138663+0+0 (secure 0 0 0) 0x7f9d60044070 con 0x7f9d70103d60 2024-04-02T14:34:43.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:43.559+0000 7f9d67e00640 1 --2- 172.21.15.90:0/939286726 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f9d540a3d20 0x7f9d540a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:43.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:43.559+0000 7f9d67e00640 1 -- 172.21.15.90:0/939286726 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8248+0+0 (secure 0 0 0) 0x7f9d600f1ef0 con 0x7f9d70103d60 2024-04-02T14:34:43.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:43.559+0000 7f9d6ea00640 1 -- 172.21.15.90:0/939286726 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f9d540a3d20 msgr2=0x7f9d540a61e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.90:6800/4266054817 2024-04-02T14:34:43.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:43.559+0000 7f9d6ea00640 1 --2- 172.21.15.90:0/939286726 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f9d540a3d20 0x7f9d540a61e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T14:34:43.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:43.563+0000 7f9d67e00640 1 -- 172.21.15.90:0/939286726 <== mon.2 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7f9d600be2a0 con 0x7f9d70103d60 2024-04-02T14:34:43.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:43.756+0000 7f9d76000640 1 -- 172.21.15.90:0/939286726 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) -- 0x7f9d7002d370 con 0x7f9d70103d60 2024-04-02T14:34:43.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:43.760+0000 7f9d6ea00640 1 -- 172.21.15.90:0/939286726 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f9d540a3d20 msgr2=0x7f9d540a61e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.90:6800/4266054817 2024-04-02T14:34:43.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:43.760+0000 7f9d6ea00640 1 --2- 172.21.15.90:0/939286726 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f9d540a3d20 0x7f9d540a61e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T14:34:43.764 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:43.762+0000 7f9d67e00640 1 -- 172.21.15.90:0/939286726 <== mon.2 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v34)=0 v34) ==== 133+0+0 (secure 0 0 0) 0x7f9d600c25a0 con 0x7f9d70103d60 2024-04-02T14:34:43.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:43.765+0000 7f9d76000640 1 -- 172.21.15.90:0/939286726 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f9d540a3d20 msgr2=0x7f9d540a61e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T14:34:43.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:43.765+0000 7f9d76000640 1 --2- 172.21.15.90:0/939286726 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f9d540a3d20 0x7f9d540a61e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:43.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:43.765+0000 7f9d76000640 1 -- 172.21.15.90:0/939286726 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9d70103d60 msgr2=0x7f9d70199fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:43.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:43.765+0000 7f9d76000640 1 --2- 172.21.15.90:0/939286726 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9d70103d60 0x7f9d70199fe0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f9d60000c00 tx=0x7f9d6002c640 comp rx=0 tx=0).stop 2024-04-02T14:34:43.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:43.765+0000 7f9d76000640 1 -- 172.21.15.90:0/939286726 shutdown_connections 2024-04-02T14:34:43.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:43.765+0000 7f9d76000640 1 --2- 172.21.15.90:0/939286726 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7f9d540a3d20 0x7f9d540a61e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:43.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:43.765+0000 7f9d76000640 1 --2- 172.21.15.90:0/939286726 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9d70105910 0x7f9d701a1590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:43.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:43.765+0000 7f9d76000640 1 --2- 172.21.15.90:0/939286726 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9d70104f50 0x7f9d7019a520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:43.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:43.765+0000 7f9d76000640 1 --2- 172.21.15.90:0/939286726 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9d70103d60 0x7f9d70199fe0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:43.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:43.765+0000 7f9d76000640 1 -- 172.21.15.90:0/939286726 >> 172.21.15.90:0/939286726 conn(0x7f9d700ff2f0 msgr2=0x7f9d701081f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:34:43.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:43.765+0000 7f9d76000640 1 -- 172.21.15.90:0/939286726 shutdown_connections 2024-04-02T14:34:43.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:43.765+0000 7f9d76000640 1 -- 172.21.15.90:0/939286726 wait complete. 2024-04-02T14:34:43.785 INFO:teuthology.run_tasks:Running task kclient... 2024-04-02T14:34:43.813 INFO:tasks.kclient:Mounting kernel clients... 2024-04-02T14:34:43.813 INFO:tasks.kclient:config is {'client.0': {}} 2024-04-02T14:34:43.813 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T14:34:43.813 DEBUG:teuthology.orchestra.run.smithi090:> ip netns list 2024-04-02T14:34:43.860 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 02 14:34:43 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-y[37879]: 2024-04-02T14:34:43.481+0000 7ffb14a3f200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-02T14:34:43.861 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 02 14:34:43 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-y[37879]: 2024-04-02T14:34:43.574+0000 7ffb14a3f200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-02T14:34:43.865 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T14:34:43.865 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip link delete ceph-brx 2024-04-02T14:34:43.917 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 02 14:34:43 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-z[38299]: 2024-04-02T14:34:43.486+0000 7fca5fe87200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-02T14:34:43.917 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 02 14:34:43 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-z[38299]: 2024-04-02T14:34:43.579+0000 7fca5fe87200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-02T14:34:43.951 INFO:teuthology.orchestra.run.smithi090.stderr:Cannot find device "ceph-brx" 2024-04-02T14:34:43.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T14:34:43.954 INFO:tasks.kclient:client.0 config is {'mntopts': ['ms_mode=secure', 'nowsync', 'context=system_u:object_r:postgresql_db_t:s0'], 'syntax': 'v2', 'mount_subvol_num': 1, 'mountpoint': '/tmp/cephfs'} 2024-04-02T14:34:43.955 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-02T14:34:43.955 INFO:tasks.cephfs.mount:using client_config["hostfs_mntpt"] = /tmp/cephfs 2024-04-02T14:34:43.955 INFO:tasks.cephfs.mount:hostfs_mntpt = /tmp/cephfs 2024-04-02T14:34:43.955 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:191d963ac9de7322da8aff20b567b3fa458e2ef6 shell --fsid 6449448a-f0fc-11ee-b647-cb9ed24678a4 -- ceph fs ls 2024-04-02T14:34:44.021 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:34:43 smithi090 ceph-mon[31674]: from='client.? 172.21.15.90:0/2056420245' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-04-02T14:34:44.022 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:34:43 smithi090 ceph-mon[31674]: mgrmap e19: x(active, since 10m), standbys: z, y 2024-04-02T14:34:44.022 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:34:43 smithi090 ceph-mon[31674]: from='client.? 172.21.15.90:0/1756069163' entity='client.admin' 2024-04-02T14:34:44.022 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:34:43 smithi090 ceph-mon[31674]: from='client.? ' entity='client.admin' 2024-04-02T14:34:44.022 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 02 14:34:43 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-x[31910]: 2024-04-02T14:34:43.745+0000 7ff6a1213200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-02T14:34:44.022 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 02 14:34:43 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-x[31910]: 2024-04-02T14:34:43.872+0000 7ff6a1213200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-02T14:34:44.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:34:43 smithi177 ceph-mon[36431]: from='client.? 172.21.15.90:0/2056420245' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-04-02T14:34:44.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:34:43 smithi177 ceph-mon[36431]: mgrmap e19: x(active, since 10m), standbys: z, y 2024-04-02T14:34:44.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:34:43 smithi177 ceph-mon[36431]: from='client.? 172.21.15.90:0/1756069163' entity='client.admin' 2024-04-02T14:34:44.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:34:43 smithi177 ceph-mon[36431]: from='client.? ' entity='client.admin' 2024-04-02T14:34:44.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:34:43 smithi092 ceph-mon[36497]: from='client.? 172.21.15.90:0/2056420245' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-04-02T14:34:44.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:34:43 smithi092 ceph-mon[36497]: mgrmap e19: x(active, since 10m), standbys: z, y 2024-04-02T14:34:44.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:34:43 smithi092 ceph-mon[36497]: from='client.? 172.21.15.90:0/1756069163' entity='client.admin' 2024-04-02T14:34:44.216 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:34:43 smithi092 ceph-mon[36497]: from='client.? ' entity='client.admin' 2024-04-02T14:34:44.364 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T14:34:44.545 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 02 14:34:44 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-y[37879]: 2024-04-02T14:34:44.214+0000 7ffb14a3f200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-02T14:34:44.546 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 02 14:34:44 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-y[37879]: 2024-04-02T14:34:44.307+0000 7ffb14a3f200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-02T14:34:44.546 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 02 14:34:44 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-y[37879]: 2024-04-02T14:34:44.448+0000 7ffb14a3f200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-02T14:34:44.546 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 02 14:34:44 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-y[37879]: 2024-04-02T14:34:44.543+0000 7ffb14a3f200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-02T14:34:44.553 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 02 14:34:44 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-z[38299]: 2024-04-02T14:34:44.223+0000 7fca5fe87200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-02T14:34:44.553 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 02 14:34:44 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-z[38299]: 2024-04-02T14:34:44.316+0000 7fca5fe87200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-02T14:34:44.554 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 02 14:34:44 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-z[38299]: 2024-04-02T14:34:44.458+0000 7fca5fe87200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-02T14:34:44.554 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 02 14:34:44 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-z[38299]: 2024-04-02T14:34:44.552+0000 7fca5fe87200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-02T14:34:44.801 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 02 14:34:44 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-x[31910]: 2024-04-02T14:34:44.701+0000 7ff6a1213200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-02T14:34:44.860 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 02 14:34:44 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-y[37879]: 2024-04-02T14:34:44.627+0000 7ffb14a3f200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-02T14:34:44.898 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 02 14:34:44 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-z[38299]: 2024-04-02T14:34:44.633+0000 7fca5fe87200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-02T14:34:45.166 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 02 14:34:44 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-x[31910]: 2024-04-02T14:34:44.833+0000 7ff6a1213200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-02T14:34:45.166 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 02 14:34:45 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-x[31910]: 2024-04-02T14:34:45.027+0000 7ff6a1213200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-02T14:34:45.185 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 02 14:34:44 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-y[37879]: 2024-04-02T14:34:44.892+0000 7ffb14a3f200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-02T14:34:45.186 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 02 14:34:45 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-y[37879]: 2024-04-02T14:34:45.002+0000 7ffb14a3f200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-02T14:34:45.186 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 02 14:34:45 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-y[37879]: 2024-04-02T14:34:45.088+0000 7ffb14a3f200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-02T14:34:45.186 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 02 14:34:45 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-y[37879]: 2024-04-02T14:34:45.184+0000 7ffb14a3f200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-02T14:34:45.186 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 02 14:34:44 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-z[38299]: 2024-04-02T14:34:44.897+0000 7fca5fe87200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-02T14:34:45.186 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 02 14:34:45 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-z[38299]: 2024-04-02T14:34:45.006+0000 7fca5fe87200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-02T14:34:45.186 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 02 14:34:45 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-z[38299]: 2024-04-02T14:34:45.093+0000 7fca5fe87200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-02T14:34:45.438 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 02 14:34:45 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-x[31910]: 2024-04-02T14:34:45.163+0000 7ff6a1213200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-02T14:34:45.439 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 02 14:34:45 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-x[31910]: 2024-04-02T14:34:45.276+0000 7ff6a1213200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-02T14:34:45.610 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 02 14:34:45 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-y[37879]: 2024-04-02T14:34:45.359+0000 7ffb14a3f200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-02T14:34:45.687 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 02 14:34:45 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-z[38299]: 2024-04-02T14:34:45.187+0000 7fca5fe87200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-02T14:34:45.687 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 02 14:34:45 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-z[38299]: 2024-04-02T14:34:45.364+0000 7fca5fe87200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-02T14:34:45.756 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 02 14:34:45 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-x[31910]: 2024-04-02T14:34:45.613+0000 7ff6a1213200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-02T14:34:46.091 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 02 14:34:45 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-x[31910]: 2024-04-02T14:34:45.754+0000 7ff6a1213200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-02T14:34:46.091 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 02 14:34:45 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-x[31910]: 2024-04-02T14:34:45.872+0000 7ff6a1213200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-02T14:34:46.091 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 02 14:34:46 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-x[31910]: 2024-04-02T14:34:46.003+0000 7ff6a1213200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-02T14:34:46.401 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 02 14:34:46 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-x[31910]: 2024-04-02T14:34:46.228+0000 7ff6a1213200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-02T14:34:46.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:46.560+0000 7fd64fc00700 1 -- 172.21.15.90:0/2362057195 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd648102fd0 msgr2=0x7fd64810f4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:46.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:46.560+0000 7fd64fc00700 1 --2- 172.21.15.90:0/2362057195 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd648102fd0 0x7fd64810f4e0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fd640009f90 tx=0x7fd64000b330 comp rx=0 tx=0).stop 2024-04-02T14:34:46.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:46.560+0000 7fd64fc00700 1 -- 172.21.15.90:0/2362057195 shutdown_connections 2024-04-02T14:34:46.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:46.560+0000 7fd64fc00700 1 --2- 172.21.15.90:0/2362057195 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd648102fd0 0x7fd64810f4e0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:46.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:46.560+0000 7fd64fc00700 1 --2- 172.21.15.90:0/2362057195 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd6481086a0 0x7fd648108a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:46.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:46.560+0000 7fd64fc00700 1 --2- 172.21.15.90:0/2362057195 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd648102610 0x7fd648102a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:46.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:46.560+0000 7fd64fc00700 1 -- 172.21.15.90:0/2362057195 >> 172.21.15.90:0/2362057195 conn(0x7fd6480fe0c0 msgr2=0x7fd6481004e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:34:46.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:46.560+0000 7fd64fc00700 1 -- 172.21.15.90:0/2362057195 shutdown_connections 2024-04-02T14:34:46.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:46.561+0000 7fd64fc00700 1 -- 172.21.15.90:0/2362057195 wait complete. 2024-04-02T14:34:46.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:46.562+0000 7fd64fc00700 1 Processor -- start 2024-04-02T14:34:46.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:46.562+0000 7fd64fc00700 1 -- start start 2024-04-02T14:34:46.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:46.563+0000 7fd64fc00700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd648102610 0x7fd64819f300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:46.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:46.563+0000 7fd64fc00700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd648102fd0 0x7fd64819f840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:46.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:46.563+0000 7fd64fc00700 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd6481086a0 0x7fd6481a3b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:46.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:46.563+0000 7fd64fc00700 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fd648110820 con 0x7fd6481086a0 2024-04-02T14:34:46.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:46.563+0000 7fd64fc00700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fd6481106a0 con 0x7fd648102610 2024-04-02T14:34:46.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:46.563+0000 7fd64fc00700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fd6481109a0 con 0x7fd648102fd0 2024-04-02T14:34:46.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:46.563+0000 7fd64d200700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd648102610 0x7fd64819f300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:46.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:46.563+0000 7fd64d200700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd648102610 0x7fd64819f300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:39520/0 (socket says 172.21.15.90:39520) 2024-04-02T14:34:46.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:46.563+0000 7fd64d200700 1 -- 172.21.15.90:0/3846971666 learned_addr learned my addr 172.21.15.90:0/3846971666 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:34:46.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:46.563+0000 7fd64dc00700 1 --2- 172.21.15.90:0/3846971666 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd6481086a0 0x7fd6481a3b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:46.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:46.563+0000 7fd64d200700 1 -- 172.21.15.90:0/3846971666 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd648102fd0 msgr2=0x7fd64819f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:46.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:46.563+0000 7fd64d200700 1 --2- 172.21.15.90:0/3846971666 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd648102fd0 0x7fd64819f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:46.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:46.563+0000 7fd64d200700 1 -- 172.21.15.90:0/3846971666 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd6481086a0 msgr2=0x7fd6481a3b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:46.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:46.563+0000 7fd64d200700 1 --2- 172.21.15.90:0/3846971666 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd6481086a0 0x7fd6481a3b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:46.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:46.563+0000 7fd64d200700 1 -- 172.21.15.90:0/3846971666 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6481a42c0 con 0x7fd648102610 2024-04-02T14:34:46.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:46.564+0000 7fd64dc00700 1 --2- 172.21.15.90:0/3846971666 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd6481086a0 0x7fd6481a3b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:34:46.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:46.564+0000 7fd64d200700 1 --2- 172.21.15.90:0/3846971666 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd648102610 0x7fd64819f300 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fd63c009b30 tx=0x7fd63c009ef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:34:46.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:46.564+0000 7fd645600700 1 -- 172.21.15.90:0/3846971666 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd63c01c070 con 0x7fd648102610 2024-04-02T14:34:46.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:46.564+0000 7fd645600700 1 -- 172.21.15.90:0/3846971666 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd63c00cc00 con 0x7fd648102610 2024-04-02T14:34:46.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:46.564+0000 7fd645600700 1 -- 172.21.15.90:0/3846971666 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd63c0166e0 con 0x7fd648102610 2024-04-02T14:34:46.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:46.565+0000 7fd64fc00700 1 -- 172.21.15.90:0/3846971666 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6481a45b0 con 0x7fd648102610 2024-04-02T14:34:46.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:46.565+0000 7fd64fc00700 1 -- 172.21.15.90:0/3846971666 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6481abe30 con 0x7fd648102610 2024-04-02T14:34:46.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:46.566+0000 7fd64fc00700 1 -- 172.21.15.90:0/3846971666 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd6480692b0 con 0x7fd648102610 2024-04-02T14:34:46.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:46.568+0000 7fd645600700 1 -- 172.21.15.90:0/3846971666 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 19) ==== 138663+0+0 (secure 0 0 0) 0x7fd63c01f070 con 0x7fd648102610 2024-04-02T14:34:46.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:46.569+0000 7fd645600700 1 --2- 172.21.15.90:0/3846971666 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fd6300a3d10 0x7fd6300a61d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:46.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:46.569+0000 7fd647e00700 1 -- 172.21.15.90:0/3846971666 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fd6300a3d10 msgr2=0x7fd6300a61d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.90:6800/4266054817 2024-04-02T14:34:46.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:46.569+0000 7fd645600700 1 -- 172.21.15.90:0/3846971666 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8248+0+0 (secure 0 0 0) 0x7fd63c0cff60 con 0x7fd648102610 2024-04-02T14:34:46.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:46.569+0000 7fd647e00700 1 --2- 172.21.15.90:0/3846971666 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fd6300a3d10 0x7fd6300a61d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T14:34:46.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:46.577+0000 7fd645600700 1 -- 172.21.15.90:0/3846971666 <== mon.2 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7fd63c09c8d0 con 0x7fd648102610 2024-04-02T14:34:46.611 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 02 14:34:46 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-y[37879]: 2024-04-02T14:34:46.320+0000 7ffb14a3f200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-02T14:34:46.611 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 02 14:34:46 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-y[37879]: 2024-04-02T14:34:46.452+0000 7ffb14a3f200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-02T14:34:46.686 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 02 14:34:46 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-z[38299]: 2024-04-02T14:34:46.326+0000 7fca5fe87200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-02T14:34:46.687 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 02 14:34:46 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-z[38299]: 2024-04-02T14:34:46.456+0000 7fca5fe87200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-02T14:34:46.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:46.770+0000 7fd647e00700 1 -- 172.21.15.90:0/3846971666 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fd6300a3d10 msgr2=0x7fd6300a61d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.90:6800/4266054817 2024-04-02T14:34:46.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:46.770+0000 7fd647e00700 1 --2- 172.21.15.90:0/3846971666 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fd6300a3d10 0x7fd6300a61d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T14:34:46.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:46.885+0000 7fd64fc00700 1 -- 172.21.15.90:0/3846971666 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fd6481ac1a0 con 0x7fd648102610 2024-04-02T14:34:46.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:46.886+0000 7fd645600700 1 -- 172.21.15.90:0/3846971666 <== mon.2 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v26) ==== 53+0+73 (secure 0 0 0) 0x7fd63c09c1d0 con 0x7fd648102610 2024-04-02T14:34:46.889 INFO:teuthology.orchestra.run.smithi090.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-02T14:34:46.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:46.889+0000 7fd64fc00700 1 -- 172.21.15.90:0/3846971666 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fd6300a3d10 msgr2=0x7fd6300a61d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T14:34:46.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:46.889+0000 7fd64fc00700 1 --2- 172.21.15.90:0/3846971666 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fd6300a3d10 0x7fd6300a61d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:46.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:46.889+0000 7fd64fc00700 1 -- 172.21.15.90:0/3846971666 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd648102610 msgr2=0x7fd64819f300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:46.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:46.889+0000 7fd64fc00700 1 --2- 172.21.15.90:0/3846971666 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd648102610 0x7fd64819f300 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fd63c009b30 tx=0x7fd63c009ef0 comp rx=0 tx=0).stop 2024-04-02T14:34:46.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:46.889+0000 7fd64fc00700 1 -- 172.21.15.90:0/3846971666 shutdown_connections 2024-04-02T14:34:46.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:46.889+0000 7fd64fc00700 1 --2- 172.21.15.90:0/3846971666 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fd6300a3d10 0x7fd6300a61d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:46.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:46.889+0000 7fd64fc00700 1 --2- 172.21.15.90:0/3846971666 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd6481086a0 0x7fd6481a3b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:46.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:46.889+0000 7fd64fc00700 1 --2- 172.21.15.90:0/3846971666 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd648102610 0x7fd64819f300 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:46.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:46.889+0000 7fd64fc00700 1 --2- 172.21.15.90:0/3846971666 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd648102fd0 0x7fd64819f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:46.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:46.889+0000 7fd64fc00700 1 -- 172.21.15.90:0/3846971666 >> 172.21.15.90:0/3846971666 conn(0x7fd6480fe0c0 msgr2=0x7fd6480ff470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:34:46.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:46.889+0000 7fd64fc00700 1 -- 172.21.15.90:0/3846971666 shutdown_connections 2024-04-02T14:34:46.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:46.890+0000 7fd64fc00700 1 -- 172.21.15.90:0/3846971666 wait complete. 2024-04-02T14:34:47.082 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:34:46 smithi092 ceph-mon[36497]: from='client.? 172.21.15.90:0/3846971666' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-02T14:34:47.082 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 02 14:34:46 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-y[37879]: 2024-04-02T14:34:46.790+0000 7ffb14a3f200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-02T14:34:47.082 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 02 14:34:46 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-y[37879]: 2024-04-02T14:34:46.879+0000 7ffb14a3f200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-02T14:34:47.082 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:34:46 smithi177 ceph-mon[36431]: from='client.? 172.21.15.90:0/3846971666' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-02T14:34:47.083 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 02 14:34:46 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-z[38299]: 2024-04-02T14:34:46.795+0000 7fca5fe87200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-02T14:34:47.083 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 02 14:34:46 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-z[38299]: 2024-04-02T14:34:46.883+0000 7fca5fe87200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-02T14:34:47.083 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 02 14:34:47 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-z[38299]: 2024-04-02T14:34:47.080+0000 7fca5fe87200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-02T14:34:47.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:34:46 smithi090 ceph-mon[31674]: from='client.? 172.21.15.90:0/3846971666' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-02T14:34:47.360 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 02 14:34:47 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-y[37879]: 2024-04-02T14:34:47.079+0000 7ffb14a3f200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-02T14:34:47.361 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 02 14:34:47 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-y[37879]: 2024-04-02T14:34:47.261+0000 7ffb14a3f200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-02T14:34:47.436 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 02 14:34:47 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-z[38299]: 2024-04-02T14:34:47.264+0000 7fca5fe87200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-02T14:34:47.711 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-02T14:34:47.711 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-02T14:34:47.711 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi090.front.sepia.ceph.com 2024-04-02T14:34:47.711 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-02T14:34:47.711 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /tmp/cephfs 2024-04-02T14:34:47.711 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-02T14:34:47.711 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-02T14:34:47.711 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-02T14:34:47.711 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/tmp/cephfs' 2024-04-02T14:34:47.711 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T14:34:47.712 DEBUG:teuthology.orchestra.run.smithi090:> ip addr 2024-04-02T14:34:47.763 INFO:teuthology.orchestra.run.smithi090.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-02T14:34:47.763 INFO:teuthology.orchestra.run.smithi090.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-02T14:34:47.763 INFO:teuthology.orchestra.run.smithi090.stdout: inet 127.0.0.1/8 scope host lo 2024-04-02T14:34:47.763 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-04-02T14:34:47.763 INFO:teuthology.orchestra.run.smithi090.stdout: inet6 ::1/128 scope host 2024-04-02T14:34:47.763 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-04-02T14:34:47.763 INFO:teuthology.orchestra.run.smithi090.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-02T14:34:47.764 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether 0c:c4:7a:d8:a1:a8 brd ff:ff:ff:ff:ff:ff 2024-04-02T14:34:47.764 INFO:teuthology.orchestra.run.smithi090.stdout: altname enp7s0f0 2024-04-02T14:34:47.764 INFO:teuthology.orchestra.run.smithi090.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-02T14:34:47.764 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether 0c:c4:7a:d8:a1:a9 brd ff:ff:ff:ff:ff:ff 2024-04-02T14:34:47.764 INFO:teuthology.orchestra.run.smithi090.stdout: altname enp7s0f1 2024-04-02T14:34:47.764 INFO:teuthology.orchestra.run.smithi090.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-02T14:34:47.764 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether 0c:c4:7a:8f:d7:66 brd ff:ff:ff:ff:ff:ff 2024-04-02T14:34:47.764 INFO:teuthology.orchestra.run.smithi090.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-02T14:34:47.764 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether 0c:c4:7a:8f:d7:67 brd ff:ff:ff:ff:ff:ff 2024-04-02T14:34:47.764 INFO:teuthology.orchestra.run.smithi090.stdout: inet 172.21.15.90/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-04-02T14:34:47.764 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft 41900sec preferred_lft 41900sec 2024-04-02T14:34:47.764 INFO:teuthology.orchestra.run.smithi090.stdout: inet6 fe80::ec4:7aff:fe8f:d767/64 scope link 2024-04-02T14:34:47.764 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-04-02T14:34:47.765 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-04-02T14:34:47.765 DEBUG:teuthology.orchestra.run.smithi090:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-02T14:34:47.765 DEBUG:teuthology.orchestra.run.smithi090:> set -e 2024-04-02T14:34:47.765 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip link add name ceph-brx type bridge 2024-04-02T14:34:47.765 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip addr flush dev ceph-brx 2024-04-02T14:34:47.765 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip link set ceph-brx up 2024-04-02T14:34:47.765 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-04-02T14:34:47.765 DEBUG:teuthology.orchestra.run.smithi090:> ') 2024-04-02T14:34:47.802 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 02 14:34:47 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-y[37879]: 2024-04-02T14:34:47.546+0000 7ffb14a3f200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-02T14:34:47.808 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 02 14:34:47 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-z[38299]: 2024-04-02T14:34:47.547+0000 7fca5fe87200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-02T14:34:47.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:47 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-02T14:34:47.893 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 02 14:34:47 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-x[31910]: 2024-04-02T14:34:47.591+0000 7ff6a1213200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-02T14:34:47.893 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 02 14:34:47 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-x[31910]: 2024-04-02T14:34:47.781+0000 7ff6a1213200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-02T14:34:48.110 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 02 14:34:47 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-y[37879]: 2024-04-02T14:34:47.800+0000 7ffb14a3f200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-02T14:34:48.111 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 02 14:34:48 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-y[37879]: 2024-04-02T14:34:48.040+0000 7ffb14a3f200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-02T14:34:48.186 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 02 14:34:47 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-z[38299]: 2024-04-02T14:34:47.807+0000 7fca5fe87200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-02T14:34:48.186 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 02 14:34:48 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-z[38299]: 2024-04-02T14:34:48.043+0000 7fca5fe87200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-02T14:34:48.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:48 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-02T14:34:48.268 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T14:34:48.269 DEBUG:teuthology.orchestra.run.smithi090:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-02T14:34:48.340 INFO:teuthology.orchestra.run.smithi090.stdout:1 2024-04-02T14:34:48.343 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T14:34:48.344 DEBUG:teuthology.orchestra.run.smithi090:> route 2024-04-02T14:34:48.397 INFO:teuthology.orchestra.run.smithi090.stdout:Kernel IP routing table 2024-04-02T14:34:48.397 INFO:teuthology.orchestra.run.smithi090.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-02T14:34:48.397 INFO:teuthology.orchestra.run.smithi090.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-02T14:34:48.397 INFO:teuthology.orchestra.run.smithi090.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-02T14:34:48.397 INFO:teuthology.orchestra.run.smithi090.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-02T14:34:48.398 DEBUG:teuthology.orchestra.run.smithi090:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-02T14:34:48.398 DEBUG:teuthology.orchestra.run.smithi090:> set -e 2024-04-02T14:34:48.398 DEBUG:teuthology.orchestra.run.smithi090:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-02T14:34:48.398 DEBUG:teuthology.orchestra.run.smithi090:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-02T14:34:48.398 DEBUG:teuthology.orchestra.run.smithi090:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-02T14:34:48.398 DEBUG:teuthology.orchestra.run.smithi090:> ') 2024-04-02T14:34:48.438 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 02 14:34:48 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-x[31910]: 2024-04-02T14:34:48.216+0000 7ff6a1213200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-02T14:34:48.438 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 02 14:34:48 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-x[31910]: 2024-04-02T14:34:48.346+0000 7ff6a1213200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-02T14:34:48.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:48 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-02T14:34:48.778 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 02 14:34:48 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-x[31910]: 2024-04-02T14:34:48.619+0000 7ff6a1213200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-02T14:34:49.110 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 02 14:34:48 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-y[37879]: 2024-04-02T14:34:48.841+0000 7ffb14a3f200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-02T14:34:49.171 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 02 14:34:48 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-x[31910]: 2024-04-02T14:34:48.882+0000 7ff6a1213200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-02T14:34:49.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:49 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-02T14:34:49.183 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T14:34:49.183 DEBUG:teuthology.orchestra.run.smithi090:> ip netns list 2024-04-02T14:34:49.186 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 02 14:34:48 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-z[38299]: 2024-04-02T14:34:48.839+0000 7fca5fe87200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-02T14:34:49.269 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T14:34:49.269 DEBUG:teuthology.orchestra.run.smithi090:> ip netns list-id 2024-04-02T14:34:49.360 DEBUG:teuthology.orchestra.run.smithi090:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-02T14:34:49.361 DEBUG:teuthology.orchestra.run.smithi090:> set -e 2024-04-02T14:34:49.361 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip netns add ceph-ns--tmp-cephfs 2024-04-02T14:34:49.361 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip netns set ceph-ns--tmp-cephfs 0 2024-04-02T14:34:49.361 DEBUG:teuthology.orchestra.run.smithi090:> ') 2024-04-02T14:34:49.438 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 02 14:34:49 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-x[31910]: 2024-04-02T14:34:49.279+0000 7ff6a1213200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-02T14:34:49.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:49 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-02T14:34:49.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:49 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-02T14:34:49.555 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--tmp-cephfs' with 192.168.0.1/16 2024-04-02T14:34:49.555 DEBUG:teuthology.orchestra.run.smithi090:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-02T14:34:49.555 DEBUG:teuthology.orchestra.run.smithi090:> set -e 2024-04-02T14:34:49.555 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip link add veth0 netns ceph-ns--tmp-cephfs type veth peer name brx.0 2024-04-02T14:34:49.555 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip netns exec ceph-ns--tmp-cephfs ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-04-02T14:34:49.555 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip netns exec ceph-ns--tmp-cephfs ip link set veth0 up 2024-04-02T14:34:49.555 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip netns exec ceph-ns--tmp-cephfs ip link set lo up 2024-04-02T14:34:49.555 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip netns exec ceph-ns--tmp-cephfs ip route add default via 192.168.255.254 2024-04-02T14:34:49.556 DEBUG:teuthology.orchestra.run.smithi090:> ') 2024-04-02T14:34:49.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:49 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-02T14:34:49.686 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 02 14:34:49 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-z[38299]: 2024-04-02T14:34:49.412+0000 7fca5fe87200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-02T14:34:49.686 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 02 14:34:49 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-z[38299]: 2024-04-02T14:34:49.508+0000 7fca5fe87200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-02T14:34:49.722 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 02 14:34:49 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-y[37879]: 2024-04-02T14:34:49.408+0000 7ffb14a3f200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-02T14:34:49.722 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 02 14:34:49 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-y[37879]: 2024-04-02T14:34:49.505+0000 7ffb14a3f200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-02T14:34:49.767 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 02 14:34:49 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-x[31910]: 2024-04-02T14:34:49.617+0000 7ff6a1213200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-02T14:34:49.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:49 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-02T14:34:49.863 DEBUG:teuthology.orchestra.run.smithi090:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-02T14:34:49.863 DEBUG:teuthology.orchestra.run.smithi090:> set -e 2024-04-02T14:34:49.863 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip link set brx.0 up 2024-04-02T14:34:49.863 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip link set dev brx.0 master ceph-brx 2024-04-02T14:34:49.863 DEBUG:teuthology.orchestra.run.smithi090:> ') 2024-04-02T14:34:49.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:49 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-02T14:34:50.018 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 02 14:34:49 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-x[31910]: 2024-04-02T14:34:49.952+0000 7ff6a1213200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-02T14:34:50.019 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:34:49 smithi090 ceph-mon[31674]: Standby manager daemon y restarted 2024-04-02T14:34:50.019 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:34:49 smithi090 ceph-mon[31674]: Standby manager daemon y started 2024-04-02T14:34:50.019 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:34:49 smithi090 ceph-mon[31674]: from='mgr.? 172.21.15.92:0/1795255663' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-02T14:34:50.019 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:34:49 smithi090 ceph-mon[31674]: from='mgr.? 172.21.15.92:0/1795255663' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-02T14:34:50.019 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:34:49 smithi090 ceph-mon[31674]: Standby manager daemon z restarted 2024-04-02T14:34:50.019 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:34:49 smithi090 ceph-mon[31674]: Standby manager daemon z started 2024-04-02T14:34:50.019 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:34:49 smithi090 ceph-mon[31674]: from='mgr.? 172.21.15.92:0/1795255663' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-02T14:34:50.019 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:34:49 smithi090 ceph-mon[31674]: from='mgr.? 172.21.15.177:0/3357890459' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-02T14:34:50.019 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:34:49 smithi090 ceph-mon[31674]: from='mgr.? 172.21.15.92:0/1795255663' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-02T14:34:50.019 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:34:49 smithi090 ceph-mon[31674]: from='mgr.? 172.21.15.177:0/3357890459' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-02T14:34:50.019 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:34:49 smithi090 ceph-mon[31674]: from='mgr.? 172.21.15.177:0/3357890459' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-02T14:34:50.019 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:34:49 smithi090 ceph-mon[31674]: from='mgr.? 172.21.15.177:0/3357890459' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-02T14:34:50.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:50 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-02T14:34:50.070 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-02T14:34:50.070 DEBUG:teuthology.orchestra.run.smithi090:> mkdir -p -v /tmp/cephfs 2024-04-02T14:34:50.108 INFO:teuthology.orchestra.run.smithi090.stdout:mkdir: created directory '/tmp/cephfs' 2024-04-02T14:34:50.108 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-02T14:34:50.109 DEBUG:teuthology.orchestra.run.smithi090:> chmod 0000 /tmp/cephfs 2024-04-02T14:34:50.110 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 02 14:34:49 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-y[37879]: 2024-04-02T14:34:49.720+0000 7ffb14a3f200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-02T14:34:50.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:34:49 smithi092 ceph-mon[36497]: Standby manager daemon y restarted 2024-04-02T14:34:50.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:34:49 smithi092 ceph-mon[36497]: Standby manager daemon y started 2024-04-02T14:34:50.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:34:49 smithi092 ceph-mon[36497]: from='mgr.? 172.21.15.92:0/1795255663' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-02T14:34:50.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:34:49 smithi092 ceph-mon[36497]: from='mgr.? 172.21.15.92:0/1795255663' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-02T14:34:50.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:34:49 smithi092 ceph-mon[36497]: Standby manager daemon z restarted 2024-04-02T14:34:50.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:34:49 smithi092 ceph-mon[36497]: Standby manager daemon z started 2024-04-02T14:34:50.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:34:49 smithi092 ceph-mon[36497]: from='mgr.? 172.21.15.92:0/1795255663' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-02T14:34:50.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:34:49 smithi092 ceph-mon[36497]: from='mgr.? 172.21.15.177:0/3357890459' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-02T14:34:50.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:34:49 smithi092 ceph-mon[36497]: from='mgr.? 172.21.15.92:0/1795255663' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-02T14:34:50.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:34:49 smithi092 ceph-mon[36497]: from='mgr.? 172.21.15.177:0/3357890459' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-02T14:34:50.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:34:49 smithi092 ceph-mon[36497]: from='mgr.? 172.21.15.177:0/3357890459' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-02T14:34:50.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:34:49 smithi092 ceph-mon[36497]: from='mgr.? 172.21.15.177:0/3357890459' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-02T14:34:50.186 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 02 14:34:49 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-z[38299]: 2024-04-02T14:34:49.721+0000 7fca5fe87200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-02T14:34:50.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:34:49 smithi177 ceph-mon[36431]: Standby manager daemon y restarted 2024-04-02T14:34:50.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:34:49 smithi177 ceph-mon[36431]: Standby manager daemon y started 2024-04-02T14:34:50.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:34:49 smithi177 ceph-mon[36431]: from='mgr.? 172.21.15.92:0/1795255663' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-02T14:34:50.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:34:49 smithi177 ceph-mon[36431]: from='mgr.? 172.21.15.92:0/1795255663' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-02T14:34:50.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:34:49 smithi177 ceph-mon[36431]: Standby manager daemon z restarted 2024-04-02T14:34:50.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:34:49 smithi177 ceph-mon[36431]: Standby manager daemon z started 2024-04-02T14:34:50.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:34:49 smithi177 ceph-mon[36431]: from='mgr.? 172.21.15.92:0/1795255663' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-02T14:34:50.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:34:49 smithi177 ceph-mon[36431]: from='mgr.? 172.21.15.177:0/3357890459' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-02T14:34:50.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:34:49 smithi177 ceph-mon[36431]: from='mgr.? 172.21.15.92:0/1795255663' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-02T14:34:50.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:34:49 smithi177 ceph-mon[36431]: from='mgr.? 172.21.15.177:0/3357890459' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-02T14:34:50.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:34:49 smithi177 ceph-mon[36431]: from='mgr.? 172.21.15.177:0/3357890459' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-02T14:34:50.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:34:49 smithi177 ceph-mon[36431]: from='mgr.? 172.21.15.177:0/3357890459' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-02T14:34:50.192 WARNING:tasks.cephfs.mount:You cannot specify both: cephfs_mntpt and mount_subvol_num 2024-04-02T14:34:50.192 INFO:tasks.cephfs.mount:Mounting subvol 1 for now 2024-04-02T14:34:50.193 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/volumes/qa/sv_1/136ad3dc-4348-42ec-932a-abe31a1ca7ec 2024-04-02T14:34:50.193 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T14:34:50.193 DEBUG:teuthology.orchestra.run.smithi090:> sudo nsenter --net=/var/run/netns/ceph-ns--tmp-cephfs adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/volumes/qa/sv_1/136ad3dc-4348-42ec-932a-abe31a1ca7ec /tmp/cephfs -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=secure,nowsync,context=system_u:object_r:postgresql_db_t:s0,nofallback 2024-04-02T14:34:50.354 INFO:teuthology.orchestra.run.smithi090.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=secure,nowsync,context="system_u:object_r:postgresql_db_t:s0",nofallback 2024-04-02T14:34:50.354 INFO:teuthology.orchestra.run.smithi090.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=secure,nowsync,context="system_u:object_r:postgresql_db_t:s0"". 2024-04-02T14:34:51.060 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:34:50 smithi090 ceph-mon[31674]: mgrmap e20: x(active, since 10m), standbys: z, y 2024-04-02T14:34:51.110 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:34:50 smithi092 ceph-mon[36497]: mgrmap e20: x(active, since 10m), standbys: z, y 2024-04-02T14:34:51.139 INFO:teuthology.orchestra.run.smithi090.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=secure,nowsync,context="system_u:object_r:postgresql_db_t:s0",nofallback 2024-04-02T14:34:51.139 INFO:teuthology.orchestra.run.smithi090.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=secure,nowsync,context="system_u:object_r:postgresql_db_t:s0"". 2024-04-02T14:34:51.139 INFO:teuthology.orchestra.run.smithi090.stdout:mount.ceph: resolved to: "172.21.15.90:3300,172.21.15.92:3300,172.21.15.177:3300" 2024-04-02T14:34:51.139 INFO:teuthology.orchestra.run.smithi090.stdout:mount.ceph: trying mount with new device syntax: 0@6449448a-f0fc-11ee-b647-cb9ed24678a4.cephfs=/volumes/qa/sv_1/136ad3dc-4348-42ec-932a-abe31a1ca7ec 2024-04-02T14:34:51.140 INFO:teuthology.orchestra.run.smithi090.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=secure,nowsync,context="system_u:object_r:postgresql_db_t:s0",name=0,key=0,mon_addr=172.21.15.90:3300/172.21.15.92:3300/172.21.15.177:3300" will pass to kernel 2024-04-02T14:34:51.145 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-02T14:34:51.145 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T14:34:51.145 DEBUG:teuthology.orchestra.run.smithi090:> sudo chmod 1777 /tmp/cephfs 2024-04-02T14:34:51.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:34:50 smithi177 ceph-mon[36431]: mgrmap e20: x(active, since 10m), standbys: z, y 2024-04-02T14:34:51.224 DEBUG:teuthology.orchestra.run.smithi090:> (cd /tmp/cephfs && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-02T14:34:51.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:51 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-02T14:34:51.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:51 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-02T14:34:51.346 INFO:teuthology.orchestra.run.smithi090.stdout:client15198 2024-04-02T14:34:51.346 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-04-02T14:34:51.346 DEBUG:teuthology.orchestra.run.smithi090:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T14:34:51.393 DEBUG:teuthology.orchestra.run.smithi090:> (cd /tmp/cephfs && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-02T14:34:51.442 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 02 14:34:51 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-x[31910]: 2024-04-02T14:34:51.057+0000 7ff6a1213200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-02T14:34:51.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:51 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-02T14:34:51.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:51 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-02T14:34:51.545 INFO:teuthology.orchestra.run.smithi090.stdout:client15198 2024-04-02T14:34:51.546 DEBUG:teuthology.orchestra.run.smithi090:> (cd /tmp/cephfs && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-02T14:34:51.640 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:51 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-02T14:34:51.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:51 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-02T14:34:51.652 INFO:teuthology.orchestra.run.smithi090.stdout:client15198 2024-04-02T14:34:51.653 DEBUG:teuthology.orchestra.run.smithi090:> (cd /tmp/cephfs && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4.client15198/status') 2024-04-02T14:34:51.745 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:51 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-02T14:34:51.784 INFO:teuthology.orchestra.run.smithi090.stdout:instance: client.15198 (3)192.168.0.1:0/3249334281 2024-04-02T14:34:51.784 INFO:teuthology.orchestra.run.smithi090.stdout:blocklisted: false 2024-04-02T14:34:51.784 INFO:teuthology.orchestra.run.smithi090.stderr:0+1 records in 2024-04-02T14:34:51.784 INFO:teuthology.orchestra.run.smithi090.stderr:0+1 records out 2024-04-02T14:34:51.784 INFO:teuthology.orchestra.run.smithi090.stderr:70 bytes copied, 6.3581e-05 s, 1.1 MB/s 2024-04-02T14:34:51.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:51 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-02T14:34:51.796 INFO:teuthology.run_tasks:Running task exec... 2024-04-02T14:34:51.806 INFO:teuthology.task.exec:Executing custom commands... 2024-04-02T14:34:51.806 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi090.front.sepia.ceph.com 2024-04-02T14:34:51.806 DEBUG:teuthology.orchestra.run.smithi090:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs set cephfs balance_automate false' 2024-04-02T14:34:52.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:52.002+0000 7fb208e00640 1 -- 172.21.15.90:0/1283042508 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb204103520 msgr2=0x7fb204109db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:52.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:52.002+0000 7fb208e00640 1 --2- 172.21.15.90:0/1283042508 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb204103520 0x7fb204109db0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fb1fc009ca0 tx=0x7fb1fc02c0f0 comp rx=0 tx=0).stop 2024-04-02T14:34:52.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:52.002+0000 7fb208e00640 1 -- 172.21.15.90:0/1283042508 shutdown_connections 2024-04-02T14:34:52.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:52.002+0000 7fb208e00640 1 --2- 172.21.15.90:0/1283042508 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb204103520 0x7fb204109db0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:52.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:52.002+0000 7fb208e00640 1 --2- 172.21.15.90:0/1283042508 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb204102b60 0x7fb204102fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:52.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:52.002+0000 7fb208e00640 1 --2- 172.21.15.90:0/1283042508 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb204101970 0x7fb204101d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:52.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:52.002+0000 7fb208e00640 1 -- 172.21.15.90:0/1283042508 >> 172.21.15.90:0/1283042508 conn(0x7fb2040fcee0 msgr2=0x7fb2040ff320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:34:52.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:52.002+0000 7fb208e00640 1 -- 172.21.15.90:0/1283042508 shutdown_connections 2024-04-02T14:34:52.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:52.002+0000 7fb208e00640 1 -- 172.21.15.90:0/1283042508 wait complete. 2024-04-02T14:34:52.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:52.003+0000 7fb208e00640 1 Processor -- start 2024-04-02T14:34:52.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:52.003+0000 7fb208e00640 1 -- start start 2024-04-02T14:34:52.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:52.003+0000 7fb208e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb204101970 0x7fb204197c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:52.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:52.003+0000 7fb208e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb204102b60 0x7fb204198180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:52.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:52.003+0000 7fb208e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb204103520 0x7fb20419f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:52.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:52.003+0000 7fb208e00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fb20410d5d0 con 0x7fb204102b60 2024-04-02T14:34:52.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:52.003+0000 7fb208e00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fb20410d450 con 0x7fb204101970 2024-04-02T14:34:52.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:52.003+0000 7fb208e00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fb20410d750 con 0x7fb204103520 2024-04-02T14:34:52.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:52.004+0000 7fb203400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb204101970 0x7fb204197c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:52.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:52.004+0000 7fb203400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb204101970 0x7fb204197c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:39544/0 (socket says 172.21.15.90:39544) 2024-04-02T14:34:52.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:52.004+0000 7fb203400640 1 -- 172.21.15.90:0/1866672004 learned_addr learned my addr 172.21.15.90:0/1866672004 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:34:52.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:52.004+0000 7fb203e00640 1 --2- 172.21.15.90:0/1866672004 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb204103520 0x7fb20419f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:52.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:52.004+0000 7fb202a00640 1 --2- 172.21.15.90:0/1866672004 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb204102b60 0x7fb204198180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:52.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:52.004+0000 7fb203e00640 1 -- 172.21.15.90:0/1866672004 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb204101970 msgr2=0x7fb204197c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:52.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:52.004+0000 7fb203e00640 1 --2- 172.21.15.90:0/1866672004 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb204101970 0x7fb204197c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:52.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:52.004+0000 7fb203e00640 1 -- 172.21.15.90:0/1866672004 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb204102b60 msgr2=0x7fb204198180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:52.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:52.004+0000 7fb203e00640 1 --2- 172.21.15.90:0/1866672004 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb204102b60 0x7fb204198180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:52.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:52.004+0000 7fb203e00640 1 -- 172.21.15.90:0/1866672004 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb20419f850 con 0x7fb204103520 2024-04-02T14:34:52.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:52.004+0000 7fb203400640 1 --2- 172.21.15.90:0/1866672004 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb204101970 0x7fb204197c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:34:52.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:52.004+0000 7fb203e00640 1 --2- 172.21.15.90:0/1866672004 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb204103520 0x7fb20419f1b0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fb1fc02c630 tx=0x7fb1fc032f20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:34:52.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:52.005+0000 7fb1fbe00640 1 -- 172.21.15.90:0/1866672004 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1fc041020 con 0x7fb204103520 2024-04-02T14:34:52.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:52.005+0000 7fb208e00640 1 -- 172.21.15.90:0/1866672004 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb20419fae0 con 0x7fb204103520 2024-04-02T14:34:52.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:52.005+0000 7fb1fbe00640 1 -- 172.21.15.90:0/1866672004 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb1fc034cd0 con 0x7fb204103520 2024-04-02T14:34:52.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:52.005+0000 7fb1fbe00640 1 -- 172.21.15.90:0/1866672004 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1fc039040 con 0x7fb204103520 2024-04-02T14:34:52.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:52.005+0000 7fb208e00640 1 -- 172.21.15.90:0/1866672004 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb20419ffa0 con 0x7fb204103520 2024-04-02T14:34:52.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:52.006+0000 7fb208e00640 1 -- 172.21.15.90:0/1866672004 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb204108c30 con 0x7fb204103520 2024-04-02T14:34:52.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:52.008+0000 7fb1fbe00640 1 -- 172.21.15.90:0/1866672004 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 20) ==== 138663+0+0 (secure 0 0 0) 0x7fb1fc044020 con 0x7fb204103520 2024-04-02T14:34:52.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:52.009+0000 7fb1fbe00640 1 --2- 172.21.15.90:0/1866672004 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fb1e80a3bf0 0x7fb1e80a60b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:52.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:52.009+0000 7fb203400640 1 -- 172.21.15.90:0/1866672004 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fb1e80a3bf0 msgr2=0x7fb1e80a60b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.90:6800/4266054817 2024-04-02T14:34:52.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:52.009+0000 7fb1fbe00640 1 -- 172.21.15.90:0/1866672004 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8248+0+0 (secure 0 0 0) 0x7fb1fc0f2720 con 0x7fb204103520 2024-04-02T14:34:52.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:52.009+0000 7fb203400640 1 --2- 172.21.15.90:0/1866672004 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fb1e80a3bf0 0x7fb1e80a60b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-02T14:34:52.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:52.011+0000 7fb1fbe00640 1 -- 172.21.15.90:0/1866672004 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7fb1fc04b020 con 0x7fb204103520 2024-04-02T14:34:52.188 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 02 14:34:51 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-x[31910]: 2024-04-02T14:34:51.802+0000 7ff6a1213200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-02T14:34:52.189 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 02 14:34:51 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-x[31910]: 2024-04-02T14:34:51.939+0000 7ff6a1213200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-02T14:34:52.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:52.210+0000 7fb203400640 1 -- 172.21.15.90:0/1866672004 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fb1e80a3bf0 msgr2=0x7fb1e80a60b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.90:6800/4266054817 2024-04-02T14:34:52.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:52.210+0000 7fb203400640 1 --2- 172.21.15.90:0/1866672004 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fb1e80a3bf0 0x7fb1e80a60b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-02T14:34:52.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:52.257+0000 7fb1fbe00640 1 -- 172.21.15.90:0/1866672004 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mgrmap(e 21) ==== 138331+0+0 (secure 0 0 0) 0x7fb1fc003970 con 0x7fb204103520 2024-04-02T14:34:52.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:52.257+0000 7fb1fbe00640 1 -- 172.21.15.90:0/1866672004 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fb1e80a3bf0 msgr2=0x7fb1e80a60b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T14:34:52.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:52.257+0000 7fb1fbe00640 1 --2- 172.21.15.90:0/1866672004 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fb1e80a3bf0 0x7fb1e80a60b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:52.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:52.263+0000 7fb208e00640 1 -- 172.21.15.90:0/1866672004 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"} v 0) -- 0x7fb20402d410 con 0x7fb204103520 2024-04-02T14:34:52.538 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:34:52 smithi090 ceph-mon[31674]: Active manager daemon x restarted 2024-04-02T14:34:52.538 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:34:52 smithi090 ceph-mon[31674]: Activating manager daemon x 2024-04-02T14:34:52.538 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:34:52 smithi090 ceph-mon[31674]: osdmap e91: 12 total, 12 up, 12 in 2024-04-02T14:34:52.538 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:34:52 smithi090 ceph-mon[31674]: mgrmap e21: x(active, starting, since 0.0236484s), standbys: z, y 2024-04-02T14:34:52.538 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:34:52 smithi090 ceph-mon[31674]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-02T14:34:52.538 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:34:52 smithi090 ceph-mon[31674]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-02T14:34:52.538 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:34:52 smithi090 ceph-mon[31674]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-02T14:34:52.538 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:34:52 smithi090 ceph-mon[31674]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-02T14:34:52.538 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:34:52 smithi090 ceph-mon[31674]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-02T14:34:52.538 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:34:52 smithi090 ceph-mon[31674]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-02T14:34:52.539 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:34:52 smithi090 ceph-mon[31674]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-02T14:34:52.539 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:34:52 smithi090 ceph-mon[31674]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-02T14:34:52.539 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:34:52 smithi090 ceph-mon[31674]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-02T14:34:52.539 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:34:52 smithi090 ceph-mon[31674]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-02T14:34:52.539 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:34:52 smithi090 ceph-mon[31674]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-02T14:34:52.539 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:34:52 smithi090 ceph-mon[31674]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-02T14:34:52.539 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:34:52 smithi090 ceph-mon[31674]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-02T14:34:52.539 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:34:52 smithi090 ceph-mon[31674]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-02T14:34:52.539 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:34:52 smithi090 ceph-mon[31674]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-02T14:34:52.539 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:34:52 smithi090 ceph-mon[31674]: from='client.? 172.21.15.90:0/1866672004' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-04-02T14:34:52.539 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:34:52 smithi090 ceph-mon[31674]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-02T14:34:52.539 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:34:52 smithi090 ceph-mon[31674]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-02T14:34:52.540 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:34:52 smithi090 ceph-mon[31674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-04-02T14:34:52.540 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:34:52 smithi090 ceph-mon[31674]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-02T14:34:52.540 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:34:52 smithi090 ceph-mon[31674]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-02T14:34:52.540 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:34:52 smithi090 ceph-mon[31674]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-02T14:34:52.540 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:34:52 smithi090 ceph-mon[31674]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-02T14:34:52.540 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 02 14:34:52 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-x[31910]: 2024-04-02T14:34:52.229+0000 7ff6a1213200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-02T14:34:52.540 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:34:52 smithi090 ceph-mon[31674]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-02T14:34:52.541 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:34:52 smithi090 ceph-mon[31674]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-02T14:34:52.541 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:34:52 smithi090 ceph-mon[31674]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-02T14:34:52.541 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:34:52 smithi090 ceph-mon[31674]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-02T14:34:52.541 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:34:52 smithi090 ceph-mon[31674]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-02T14:34:52.541 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:34:52 smithi090 ceph-mon[31674]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-02T14:34:52.541 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:34:52 smithi090 ceph-mon[31674]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-02T14:34:52.541 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:34:52 smithi090 ceph-mon[31674]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-02T14:34:52.541 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:34:52 smithi090 ceph-mon[31674]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-02T14:34:52.541 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:34:52 smithi090 ceph-mon[31674]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-02T14:34:52.541 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:34:52 smithi090 ceph-mon[31674]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-02T14:34:52.541 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:34:52 smithi090 ceph-mon[31674]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-02T14:34:52.542 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:34:52 smithi090 ceph-mon[31674]: Manager daemon x is now available 2024-04-02T14:34:52.611 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:34:52 smithi092 ceph-mon[36497]: Active manager daemon x restarted 2024-04-02T14:34:52.611 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:34:52 smithi092 ceph-mon[36497]: Activating manager daemon x 2024-04-02T14:34:52.611 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:34:52 smithi092 ceph-mon[36497]: osdmap e91: 12 total, 12 up, 12 in 2024-04-02T14:34:52.611 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:34:52 smithi092 ceph-mon[36497]: mgrmap e21: x(active, starting, since 0.0236484s), standbys: z, y 2024-04-02T14:34:52.611 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:34:52 smithi092 ceph-mon[36497]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-02T14:34:52.611 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:34:52 smithi092 ceph-mon[36497]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-02T14:34:52.612 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:34:52 smithi092 ceph-mon[36497]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-02T14:34:52.612 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:34:52 smithi092 ceph-mon[36497]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-02T14:34:52.612 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:34:52 smithi092 ceph-mon[36497]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-02T14:34:52.612 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:34:52 smithi092 ceph-mon[36497]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-02T14:34:52.612 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:34:52 smithi092 ceph-mon[36497]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-02T14:34:52.612 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:34:52 smithi092 ceph-mon[36497]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-02T14:34:52.612 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:34:52 smithi092 ceph-mon[36497]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-02T14:34:52.612 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:34:52 smithi092 ceph-mon[36497]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-02T14:34:52.612 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:34:52 smithi092 ceph-mon[36497]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-02T14:34:52.612 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:34:52 smithi092 ceph-mon[36497]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-02T14:34:52.612 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:34:52 smithi092 ceph-mon[36497]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-02T14:34:52.612 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:34:52 smithi092 ceph-mon[36497]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-02T14:34:52.612 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:34:52 smithi092 ceph-mon[36497]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-02T14:34:52.612 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:34:52 smithi092 ceph-mon[36497]: from='client.? 172.21.15.90:0/1866672004' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-04-02T14:34:52.613 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:34:52 smithi092 ceph-mon[36497]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-02T14:34:52.613 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:34:52 smithi092 ceph-mon[36497]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-02T14:34:52.613 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:34:52 smithi092 ceph-mon[36497]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-04-02T14:34:52.613 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:34:52 smithi092 ceph-mon[36497]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-02T14:34:52.613 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:34:52 smithi092 ceph-mon[36497]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-02T14:34:52.613 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:34:52 smithi092 ceph-mon[36497]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-02T14:34:52.613 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:34:52 smithi092 ceph-mon[36497]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-02T14:34:52.613 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:34:52 smithi092 ceph-mon[36497]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-02T14:34:52.613 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:34:52 smithi092 ceph-mon[36497]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-02T14:34:52.613 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:34:52 smithi092 ceph-mon[36497]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-02T14:34:52.613 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:34:52 smithi092 ceph-mon[36497]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-02T14:34:52.613 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:34:52 smithi092 ceph-mon[36497]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-02T14:34:52.613 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:34:52 smithi092 ceph-mon[36497]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-02T14:34:52.613 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:34:52 smithi092 ceph-mon[36497]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-02T14:34:52.614 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:34:52 smithi092 ceph-mon[36497]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-02T14:34:52.614 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:34:52 smithi092 ceph-mon[36497]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-02T14:34:52.614 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:34:52 smithi092 ceph-mon[36497]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-02T14:34:52.614 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:34:52 smithi092 ceph-mon[36497]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-02T14:34:52.614 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:34:52 smithi092 ceph-mon[36497]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-02T14:34:52.614 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:34:52 smithi092 ceph-mon[36497]: Manager daemon x is now available 2024-04-02T14:34:52.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:34:52 smithi177 ceph-mon[36431]: Active manager daemon x restarted 2024-04-02T14:34:52.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:34:52 smithi177 ceph-mon[36431]: Activating manager daemon x 2024-04-02T14:34:52.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:34:52 smithi177 ceph-mon[36431]: osdmap e91: 12 total, 12 up, 12 in 2024-04-02T14:34:52.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:34:52 smithi177 ceph-mon[36431]: mgrmap e21: x(active, starting, since 0.0236484s), standbys: z, y 2024-04-02T14:34:52.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:34:52 smithi177 ceph-mon[36431]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-02T14:34:52.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:34:52 smithi177 ceph-mon[36431]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-02T14:34:52.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:34:52 smithi177 ceph-mon[36431]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-02T14:34:52.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:34:52 smithi177 ceph-mon[36431]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-02T14:34:52.688 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:34:52 smithi177 ceph-mon[36431]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-02T14:34:52.688 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:34:52 smithi177 ceph-mon[36431]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-02T14:34:52.688 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:34:52 smithi177 ceph-mon[36431]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-02T14:34:52.688 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:34:52 smithi177 ceph-mon[36431]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-02T14:34:52.688 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:34:52 smithi177 ceph-mon[36431]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-02T14:34:52.688 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:34:52 smithi177 ceph-mon[36431]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-02T14:34:52.688 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:34:52 smithi177 ceph-mon[36431]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-02T14:34:52.688 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:34:52 smithi177 ceph-mon[36431]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-02T14:34:52.688 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:34:52 smithi177 ceph-mon[36431]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-02T14:34:52.688 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:34:52 smithi177 ceph-mon[36431]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-02T14:34:52.688 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:34:52 smithi177 ceph-mon[36431]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-02T14:34:52.688 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:34:52 smithi177 ceph-mon[36431]: from='client.? 172.21.15.90:0/1866672004' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-04-02T14:34:52.688 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:34:52 smithi177 ceph-mon[36431]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-02T14:34:52.688 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:34:52 smithi177 ceph-mon[36431]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-02T14:34:52.689 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:34:52 smithi177 ceph-mon[36431]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-04-02T14:34:52.689 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:34:52 smithi177 ceph-mon[36431]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-02T14:34:52.689 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:34:52 smithi177 ceph-mon[36431]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-02T14:34:52.689 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:34:52 smithi177 ceph-mon[36431]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-02T14:34:52.689 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:34:52 smithi177 ceph-mon[36431]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-02T14:34:52.689 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:34:52 smithi177 ceph-mon[36431]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-02T14:34:52.689 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:34:52 smithi177 ceph-mon[36431]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-02T14:34:52.689 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:34:52 smithi177 ceph-mon[36431]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-02T14:34:52.689 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:34:52 smithi177 ceph-mon[36431]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-02T14:34:52.689 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:34:52 smithi177 ceph-mon[36431]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-02T14:34:52.689 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:34:52 smithi177 ceph-mon[36431]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-02T14:34:52.689 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:34:52 smithi177 ceph-mon[36431]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-02T14:34:52.689 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:34:52 smithi177 ceph-mon[36431]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-02T14:34:52.689 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:34:52 smithi177 ceph-mon[36431]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-02T14:34:52.689 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:34:52 smithi177 ceph-mon[36431]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-02T14:34:52.689 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:34:52 smithi177 ceph-mon[36431]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-02T14:34:52.690 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:34:52 smithi177 ceph-mon[36431]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-02T14:34:52.690 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:34:52 smithi177 ceph-mon[36431]: Manager daemon x is now available 2024-04-02T14:34:52.939 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 02 14:34:52 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-x[31910]: 2024-04-02T14:34:52.534+0000 7ff5f7600700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T14:34:52.939 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 02 14:34:52 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-x[31910]: 2024-04-02T14:34:52.534+0000 7ff5f7600700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T14:34:52.939 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 02 14:34:52 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-x[31910]: 2024-04-02T14:34:52.534+0000 7ff5f7600700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T14:34:52.939 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 02 14:34:52 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-x[31910]: 2024-04-02T14:34:52.534+0000 7ff5f7600700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T14:34:52.939 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 02 14:34:52 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-x[31910]: 2024-04-02T14:34:52.534+0000 7ff5f7600700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T14:34:52.939 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 02 14:34:52 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-x[31910]: 2024-04-02T14:34:52.535+0000 7ff5f3a00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T14:34:52.939 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 02 14:34:52 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-x[31910]: 2024-04-02T14:34:52.535+0000 7ff5f3a00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T14:34:52.939 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 02 14:34:52 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-x[31910]: 2024-04-02T14:34:52.535+0000 7ff5f3a00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T14:34:52.940 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 02 14:34:52 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-x[31910]: 2024-04-02T14:34:52.535+0000 7ff5f3a00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T14:34:52.940 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 02 14:34:52 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-x[31910]: 2024-04-02T14:34:52.535+0000 7ff5f3a00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T14:34:53.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:53.249+0000 7fb1fbe00640 1 -- 172.21.15.90:0/1866672004 <== mon.1 v2:172.21.15.177:3300/0 8 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]=0 v27) ==== 118+0+0 (secure 0 0 0) 0x7fb1fc07f6d0 con 0x7fb204103520 2024-04-02T14:34:53.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:53.253+0000 7fb208e00640 1 -- 172.21.15.90:0/1866672004 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb204103520 msgr2=0x7fb20419f1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:53.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:53.253+0000 7fb208e00640 1 --2- 172.21.15.90:0/1866672004 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb204103520 0x7fb20419f1b0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fb1fc02c630 tx=0x7fb1fc032f20 comp rx=0 tx=0).stop 2024-04-02T14:34:53.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:53.253+0000 7fb208e00640 1 -- 172.21.15.90:0/1866672004 shutdown_connections 2024-04-02T14:34:53.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:53.253+0000 7fb208e00640 1 --2- 172.21.15.90:0/1866672004 >> [v2:172.21.15.90:6800/4266054817,v1:172.21.15.90:6801/4266054817] conn(0x7fb1e80a3bf0 0x7fb1e80a60b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:53.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:53.253+0000 7fb208e00640 1 --2- 172.21.15.90:0/1866672004 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb204103520 0x7fb20419f1b0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:53.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:53.253+0000 7fb208e00640 1 --2- 172.21.15.90:0/1866672004 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb204102b60 0x7fb204198180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:53.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:53.253+0000 7fb208e00640 1 --2- 172.21.15.90:0/1866672004 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb204101970 0x7fb204197c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:53.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:53.253+0000 7fb208e00640 1 -- 172.21.15.90:0/1866672004 >> 172.21.15.90:0/1866672004 conn(0x7fb2040fcee0 msgr2=0x7fb204105e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:34:53.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:53.253+0000 7fb208e00640 1 -- 172.21.15.90:0/1866672004 shutdown_connections 2024-04-02T14:34:53.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:53.253+0000 7fb208e00640 1 -- 172.21.15.90:0/1866672004 wait complete. 2024-04-02T14:34:53.289 DEBUG:teuthology.orchestra.run.smithi090:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs subvolumegroup pin cephfs qa random 0.10' 2024-04-02T14:34:53.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:53.462+0000 7f4e7e200640 1 -- 172.21.15.90:0/786665150 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4e78074500 msgr2=0x7f4e78074960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:53.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:53.462+0000 7f4e7e200640 1 --2- 172.21.15.90:0/786665150 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4e78074500 0x7f4e78074960 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f4e60009900 tx=0x7f4e6002bcd0 comp rx=0 tx=0).stop 2024-04-02T14:34:53.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:53.462+0000 7f4e7e200640 1 -- 172.21.15.90:0/786665150 shutdown_connections 2024-04-02T14:34:53.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:53.462+0000 7f4e7e200640 1 --2- 172.21.15.90:0/786665150 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4e781099c0 0x7f4e7810be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:53.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:53.462+0000 7f4e7e200640 1 --2- 172.21.15.90:0/786665150 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4e78074500 0x7f4e78074960 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:53.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:53.462+0000 7f4e7e200640 1 --2- 172.21.15.90:0/786665150 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4e78075ae0 0x7f4e78073f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:53.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:53.462+0000 7f4e7e200640 1 -- 172.21.15.90:0/786665150 >> 172.21.15.90:0/786665150 conn(0x7f4e780fd0c0 msgr2=0x7f4e780ff4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:34:53.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:53.462+0000 7f4e7e200640 1 -- 172.21.15.90:0/786665150 shutdown_connections 2024-04-02T14:34:53.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:53.462+0000 7f4e7e200640 1 -- 172.21.15.90:0/786665150 wait complete. 2024-04-02T14:34:53.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:53.463+0000 7f4e7e200640 1 Processor -- start 2024-04-02T14:34:53.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:53.463+0000 7f4e7e200640 1 -- start start 2024-04-02T14:34:53.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:53.464+0000 7f4e7e200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4e78074500 0x7f4e78199f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:53.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:53.464+0000 7f4e7e200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4e78075ae0 0x7f4e7819a4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:53.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:53.464+0000 7f4e7e200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4e781099c0 0x7f4e781a1510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:53.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:53.464+0000 7f4e7e200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f4e7810f9b0 con 0x7f4e78075ae0 2024-04-02T14:34:53.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:53.464+0000 7f4e7e200640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f4e7810f830 con 0x7f4e781099c0 2024-04-02T14:34:53.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:53.464+0000 7f4e7e200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f4e7810fb30 con 0x7f4e78074500 2024-04-02T14:34:53.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:53.464+0000 7f4e77e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4e781099c0 0x7f4e781a1510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:53.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:53.464+0000 7f4e77e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4e781099c0 0x7f4e781a1510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:39696/0 (socket says 172.21.15.90:39696) 2024-04-02T14:34:53.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:53.464+0000 7f4e77e00640 1 -- 172.21.15.90:0/1174415069 learned_addr learned my addr 172.21.15.90:0/1174415069 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:34:53.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:53.464+0000 7f4e77400640 1 --2- 172.21.15.90:0/1174415069 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4e78074500 0x7f4e78199f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:53.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:53.464+0000 7f4e6fe00640 1 --2- 172.21.15.90:0/1174415069 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4e78075ae0 0x7f4e7819a4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:53.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:53.464+0000 7f4e77e00640 1 -- 172.21.15.90:0/1174415069 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4e78074500 msgr2=0x7f4e78199f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:53.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:53.464+0000 7f4e77e00640 1 --2- 172.21.15.90:0/1174415069 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4e78074500 0x7f4e78199f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:53.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:53.464+0000 7f4e77e00640 1 -- 172.21.15.90:0/1174415069 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4e78075ae0 msgr2=0x7f4e7819a4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:53.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:53.464+0000 7f4e77e00640 1 --2- 172.21.15.90:0/1174415069 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4e78075ae0 0x7f4e7819a4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:53.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:53.464+0000 7f4e77e00640 1 -- 172.21.15.90:0/1174415069 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e781a1b80 con 0x7f4e781099c0 2024-04-02T14:34:53.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:53.464+0000 7f4e77400640 1 --2- 172.21.15.90:0/1174415069 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4e78074500 0x7f4e78199f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:34:53.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:53.465+0000 7f4e77e00640 1 --2- 172.21.15.90:0/1174415069 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4e781099c0 0x7f4e781a1510 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f4e7000eb10 tx=0x7f4e7000c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:34:53.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:53.465+0000 7f4e74c00640 1 -- 172.21.15.90:0/1174415069 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e70004b20 con 0x7f4e781099c0 2024-04-02T14:34:53.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:53.465+0000 7f4e74c00640 1 -- 172.21.15.90:0/1174415069 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4e70004cc0 con 0x7f4e781099c0 2024-04-02T14:34:53.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:53.465+0000 7f4e74c00640 1 -- 172.21.15.90:0/1174415069 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e70010760 con 0x7f4e781099c0 2024-04-02T14:34:53.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:53.465+0000 7f4e7e200640 1 -- 172.21.15.90:0/1174415069 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e781a1e70 con 0x7f4e781099c0 2024-04-02T14:34:53.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:53.465+0000 7f4e7e200640 1 -- 172.21.15.90:0/1174415069 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e781a22e0 con 0x7f4e781099c0 2024-04-02T14:34:53.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:53.466+0000 7f4e7e200640 1 -- 172.21.15.90:0/1174415069 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4e78108780 con 0x7f4e781099c0 2024-04-02T14:34:53.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:53.468+0000 7f4e74c00640 1 -- 172.21.15.90:0/1174415069 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 22) ==== 138458+0+0 (secure 0 0 0) 0x7f4e70005020 con 0x7f4e781099c0 2024-04-02T14:34:53.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:53.469+0000 7f4e74c00640 1 --2- 172.21.15.90:0/1174415069 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f4e5c0a3ae0 0x7f4e5c0a5fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:53.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:53.469+0000 7f4e74c00640 1 -- 172.21.15.90:0/1174415069 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f4e700d0770 con 0x7f4e781099c0 2024-04-02T14:34:53.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:53.470+0000 7f4e77400640 1 --2- 172.21.15.90:0/1174415069 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f4e5c0a3ae0 0x7f4e5c0a5fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:53.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:53.470+0000 7f4e77400640 1 --2- 172.21.15.90:0/1174415069 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f4e5c0a3ae0 0x7f4e5c0a5fa0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f4e68009fc0 tx=0x7f4e68005f90 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:34:53.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:53.473+0000 7f4e74c00640 1 -- 172.21.15.90:0/1174415069 <== mon.2 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7f4e7009d2f0 con 0x7f4e781099c0 2024-04-02T14:34:53.583 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:34:53 smithi092 ceph-mon[36497]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T14:34:53.583 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:34:53 smithi092 ceph-mon[36497]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-02T14:34:53.583 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:34:53 smithi092 ceph-mon[36497]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-02T14:34:53.583 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:34:53 smithi092 ceph-mon[36497]: [02/Apr/2024:14:34:52] ENGINE Bus STARTING 2024-04-02T14:34:53.583 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:34:53 smithi092 ceph-mon[36497]: [02/Apr/2024:14:34:52] ENGINE Serving on https://172.21.15.90:7150 2024-04-02T14:34:53.584 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:34:53 smithi092 ceph-mon[36497]: [02/Apr/2024:14:34:53] ENGINE Serving on http://172.21.15.90:8765 2024-04-02T14:34:53.584 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:34:53 smithi092 ceph-mon[36497]: [02/Apr/2024:14:34:53] ENGINE Bus STARTED 2024-04-02T14:34:53.584 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:34:53 smithi092 ceph-mon[36497]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-04-02T14:34:53.584 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:34:53 smithi092 ceph-mon[36497]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-02T14:34:53.584 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:34:53 smithi092 ceph-mon[36497]: mgrmap e22: x(active, since 1.03865s), standbys: z, y 2024-04-02T14:34:53.664 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:34:53 smithi177 ceph-mon[36431]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T14:34:53.664 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:34:53 smithi177 ceph-mon[36431]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-02T14:34:53.664 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:34:53 smithi177 ceph-mon[36431]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-02T14:34:53.664 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:34:53 smithi177 ceph-mon[36431]: [02/Apr/2024:14:34:52] ENGINE Bus STARTING 2024-04-02T14:34:53.664 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:34:53 smithi177 ceph-mon[36431]: [02/Apr/2024:14:34:52] ENGINE Serving on https://172.21.15.90:7150 2024-04-02T14:34:53.664 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:34:53 smithi177 ceph-mon[36431]: [02/Apr/2024:14:34:53] ENGINE Serving on http://172.21.15.90:8765 2024-04-02T14:34:53.664 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:34:53 smithi177 ceph-mon[36431]: [02/Apr/2024:14:34:53] ENGINE Bus STARTED 2024-04-02T14:34:53.664 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:34:53 smithi177 ceph-mon[36431]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-04-02T14:34:53.664 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:34:53 smithi177 ceph-mon[36431]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-02T14:34:53.664 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:34:53 smithi177 ceph-mon[36431]: mgrmap e22: x(active, since 1.03865s), standbys: z, y 2024-04-02T14:34:53.688 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:34:53 smithi090 ceph-mon[31674]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T14:34:53.688 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:34:53 smithi090 ceph-mon[31674]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-02T14:34:53.689 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:34:53 smithi090 ceph-mon[31674]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-02T14:34:53.689 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:34:53 smithi090 ceph-mon[31674]: [02/Apr/2024:14:34:52] ENGINE Bus STARTING 2024-04-02T14:34:53.689 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:34:53 smithi090 ceph-mon[31674]: [02/Apr/2024:14:34:52] ENGINE Serving on https://172.21.15.90:7150 2024-04-02T14:34:53.689 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:34:53 smithi090 ceph-mon[31674]: [02/Apr/2024:14:34:53] ENGINE Serving on http://172.21.15.90:8765 2024-04-02T14:34:53.689 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:34:53 smithi090 ceph-mon[31674]: [02/Apr/2024:14:34:53] ENGINE Bus STARTED 2024-04-02T14:34:53.689 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:34:53 smithi090 ceph-mon[31674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-04-02T14:34:53.689 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:34:53 smithi090 ceph-mon[31674]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-02T14:34:53.689 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:34:53 smithi090 ceph-mon[31674]: mgrmap e22: x(active, since 1.03865s), standbys: z, y 2024-04-02T14:34:53.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:53.701+0000 7f4e7e200640 1 -- 172.21.15.90:0/1174415069 --> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}) -- 0x7f4e78061fe0 con 0x7f4e5c0a3ae0 2024-04-02T14:34:53.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:53.725+0000 7f4e74c00640 1 -- 172.21.15.90:0/1174415069 <== mgr.25289 v2:172.21.15.90:6800/3904993754 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+3 (secure 0 0 0) 0x7f4e78061fe0 con 0x7f4e5c0a3ae0 2024-04-02T14:34:53.727 INFO:teuthology.orchestra.run.smithi090.stdout:{} 2024-04-02T14:34:53.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:53.727+0000 7f4e7e200640 1 -- 172.21.15.90:0/1174415069 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f4e5c0a3ae0 msgr2=0x7f4e5c0a5fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:53.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:53.727+0000 7f4e7e200640 1 --2- 172.21.15.90:0/1174415069 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f4e5c0a3ae0 0x7f4e5c0a5fa0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f4e68009fc0 tx=0x7f4e68005f90 comp rx=0 tx=0).stop 2024-04-02T14:34:53.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:53.727+0000 7f4e7e200640 1 -- 172.21.15.90:0/1174415069 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4e781099c0 msgr2=0x7f4e781a1510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:53.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:53.727+0000 7f4e7e200640 1 --2- 172.21.15.90:0/1174415069 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4e781099c0 0x7f4e781a1510 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f4e7000eb10 tx=0x7f4e7000c2d0 comp rx=0 tx=0).stop 2024-04-02T14:34:53.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:53.727+0000 7f4e7e200640 1 -- 172.21.15.90:0/1174415069 shutdown_connections 2024-04-02T14:34:53.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:53.727+0000 7f4e7e200640 1 --2- 172.21.15.90:0/1174415069 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f4e5c0a3ae0 0x7f4e5c0a5fa0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:53.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:53.727+0000 7f4e7e200640 1 --2- 172.21.15.90:0/1174415069 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4e781099c0 0x7f4e781a1510 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:53.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:53.727+0000 7f4e7e200640 1 --2- 172.21.15.90:0/1174415069 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4e78075ae0 0x7f4e7819a4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:53.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:53.727+0000 7f4e7e200640 1 --2- 172.21.15.90:0/1174415069 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4e78074500 0x7f4e78199f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:53.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:53.727+0000 7f4e7e200640 1 -- 172.21.15.90:0/1174415069 >> 172.21.15.90:0/1174415069 conn(0x7f4e780fd0c0 msgr2=0x7f4e78105950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:34:53.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:53.728+0000 7f4e7e200640 1 -- 172.21.15.90:0/1174415069 shutdown_connections 2024-04-02T14:34:53.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:53.728+0000 7f4e7e200640 1 -- 172.21.15.90:0/1174415069 wait complete. 2024-04-02T14:34:53.745 INFO:teuthology.run_tasks:Running task check-counter... 2024-04-02T14:34:53.757 INFO:teuthology.run_tasks:Running task fwd_scrub... 2024-04-02T14:34:53.791 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T14:34:53.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:53.980+0000 7f7749c00640 1 -- 172.21.15.90:0/210398365 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7744101b20 msgr2=0x7f7744101f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:53.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:53.980+0000 7f7749c00640 1 --2- 172.21.15.90:0/210398365 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7744101b20 0x7f7744101f40 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f7734009900 tx=0x7f773402bc30 comp rx=0 tx=0).stop 2024-04-02T14:34:53.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:53.980+0000 7f7749c00640 1 -- 172.21.15.90:0/210398365 shutdown_connections 2024-04-02T14:34:53.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:53.980+0000 7f7749c00640 1 --2- 172.21.15.90:0/210398365 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f77441036d0 0x7f7744109f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:53.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:53.980+0000 7f7749c00640 1 --2- 172.21.15.90:0/210398365 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7744102d10 0x7f7744103190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:53.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:53.980+0000 7f7749c00640 1 --2- 172.21.15.90:0/210398365 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7744101b20 0x7f7744101f40 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:53.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:53.980+0000 7f7749c00640 1 -- 172.21.15.90:0/210398365 >> 172.21.15.90:0/210398365 conn(0x7f77440fd0b0 msgr2=0x7f77440ff4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:34:53.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:53.981+0000 7f7749c00640 1 -- 172.21.15.90:0/210398365 shutdown_connections 2024-04-02T14:34:53.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:53.981+0000 7f7749c00640 1 -- 172.21.15.90:0/210398365 wait complete. 2024-04-02T14:34:53.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:53.981+0000 7f7749c00640 1 Processor -- start 2024-04-02T14:34:53.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:53.982+0000 7f7749c00640 1 -- start start 2024-04-02T14:34:53.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:53.982+0000 7f7749c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7744101b20 0x7f774419a070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:53.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:53.982+0000 7f7749c00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7744102d10 0x7f774419a5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:53.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:53.982+0000 7f7749c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f77441036d0 0x7f77441a1620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:53.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:53.982+0000 7f7749c00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f774410d7f0 con 0x7f7744101b20 2024-04-02T14:34:53.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:53.982+0000 7f7749c00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f774410d670 con 0x7f7744102d10 2024-04-02T14:34:53.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:53.982+0000 7f7749c00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f774410d970 con 0x7f77441036d0 2024-04-02T14:34:53.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:53.982+0000 7f7742a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7744101b20 0x7f774419a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:53.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:53.982+0000 7f7743400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f77441036d0 0x7f77441a1620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:53.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:53.982+0000 7f7742000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7744102d10 0x7f774419a5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:53.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:53.982+0000 7f7743400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f77441036d0 0x7f77441a1620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:49264/0 (socket says 172.21.15.90:49264) 2024-04-02T14:34:53.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:53.982+0000 7f7743400640 1 -- 172.21.15.90:0/3906444513 learned_addr learned my addr 172.21.15.90:0/3906444513 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:34:53.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:53.983+0000 7f7743400640 1 -- 172.21.15.90:0/3906444513 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7744102d10 msgr2=0x7f774419a5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:53.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:53.983+0000 7f7743400640 1 --2- 172.21.15.90:0/3906444513 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7744102d10 0x7f774419a5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:53.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:53.983+0000 7f7743400640 1 -- 172.21.15.90:0/3906444513 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7744101b20 msgr2=0x7f774419a070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:53.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:53.983+0000 7f7743400640 1 --2- 172.21.15.90:0/3906444513 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7744101b20 0x7f774419a070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:53.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:53.983+0000 7f7743400640 1 -- 172.21.15.90:0/3906444513 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77441a1c90 con 0x7f77441036d0 2024-04-02T14:34:53.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:53.983+0000 7f7743400640 1 --2- 172.21.15.90:0/3906444513 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f77441036d0 0x7f77441a1620 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f773c00eb10 tx=0x7f773c00c2d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:34:53.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:53.983+0000 7f773b400640 1 -- 172.21.15.90:0/3906444513 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f773c004b20 con 0x7f77441036d0 2024-04-02T14:34:53.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:53.983+0000 7f773b400640 1 -- 172.21.15.90:0/3906444513 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f773c004cc0 con 0x7f77441036d0 2024-04-02T14:34:53.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:53.983+0000 7f773b400640 1 -- 172.21.15.90:0/3906444513 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f773c010760 con 0x7f77441036d0 2024-04-02T14:34:53.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:53.983+0000 7f7749c00640 1 -- 172.21.15.90:0/3906444513 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f77441a1f80 con 0x7f77441036d0 2024-04-02T14:34:53.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:53.983+0000 7f7749c00640 1 -- 172.21.15.90:0/3906444513 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f77441a23f0 con 0x7f77441036d0 2024-04-02T14:34:53.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:53.984+0000 7f7749c00640 1 -- 172.21.15.90:0/3906444513 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f77441a26d0 con 0x7f77441036d0 2024-04-02T14:34:53.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:53.986+0000 7f773b400640 1 -- 172.21.15.90:0/3906444513 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 22) ==== 138458+0+0 (secure 0 0 0) 0x7f773c005020 con 0x7f77441036d0 2024-04-02T14:34:53.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:53.987+0000 7f773b400640 1 --2- 172.21.15.90:0/3906444513 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f77280a3a30 0x7f77280a5ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:53.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:53.988+0000 7f773b400640 1 -- 172.21.15.90:0/3906444513 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f773c0d0770 con 0x7f77441036d0 2024-04-02T14:34:53.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:53.988+0000 7f7742a00640 1 --2- 172.21.15.90:0/3906444513 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f77280a3a30 0x7f77280a5ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:53.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:53.989+0000 7f7742a00640 1 --2- 172.21.15.90:0/3906444513 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f77280a3a30 0x7f77280a5ef0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f773402c170 tx=0x7f773402df90 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:34:53.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:53.992+0000 7f773b400640 1 -- 172.21.15.90:0/3906444513 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7f773c09d2f0 con 0x7f77441036d0 2024-04-02T14:34:54.188 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 02 14:34:53 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-x[31910]: 2024-04-02T14:34:53.709+0000 7ff5efe00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T14:34:54.189 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 02 14:34:53 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-x[31910]: 2024-04-02T14:34:53.709+0000 7ff5efe00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T14:34:54.189 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 02 14:34:53 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-x[31910]: 2024-04-02T14:34:53.709+0000 7ff5efe00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T14:34:54.189 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 02 14:34:53 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-x[31910]: 2024-04-02T14:34:53.709+0000 7ff5efe00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T14:34:54.189 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 02 14:34:53 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-x[31910]: 2024-04-02T14:34:53.709+0000 7ff5efe00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T14:34:54.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:54.192+0000 7f7749c00640 1 -- 172.21.15.90:0/3906444513 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f77441a2a30 con 0x7f77441036d0 2024-04-02T14:34:54.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:54.194+0000 7f773b400640 1 -- 172.21.15.90:0/3906444513 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+23024 (secure 0 0 0) 0x7f773c09cbf0 con 0x7f77441036d0 2024-04-02T14:34:54.196 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-02T14:34:54.196 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":91,"fsid":"6449448a-f0fc-11ee-b647-cb9ed24678a4","created":"2024-04-02T14:23:16.530201+0000","modified":"2024-04-02T14:34:52.235281+0000","last_up_change":"2024-04-02T14:31:23.819732+0000","last_in_change":"2024-04-02T14:31:02.824284+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T14:27:16.758622+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-02T14:32:20.016331+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-02T14:32:21.294367+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a1a6a856-ab44-44bd-bc5e-45c718a70cd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":3635087609},{"type":"v1","addr":"172.21.15.90:6803","nonce":3635087609}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":3635087609},{"type":"v1","addr":"172.21.15.90:6805","nonce":3635087609}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":3635087609},{"type":"v1","addr":"172.21.15.90:6809","nonce":3635087609}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":3635087609},{"type":"v1","addr":"172.21.15.90:6807","nonce":3635087609}]},"public_addr":"172.21.15.90:6803/3635087609","cluster_addr":"172.21.15.90:6805/3635087609","heartbeat_back_addr":"172.21.15.90:6809/3635087609","heartbeat_front_addr":"172.21.15.90:6807/3635087609","state":["exists","up"]},{"osd":1,"uuid":"124139cd-5793-463e-9607-9d9c10164183","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":1994536059},{"type":"v1","addr":"172.21.15.92:6801","nonce":1994536059}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":1994536059},{"type":"v1","addr":"172.21.15.92:6803","nonce":1994536059}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":1994536059},{"type":"v1","addr":"172.21.15.92:6807","nonce":1994536059}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":1994536059},{"type":"v1","addr":"172.21.15.92:6805","nonce":1994536059}]},"public_addr":"172.21.15.92:6801/1994536059","cluster_addr":"172.21.15.92:6803/1994536059","heartbeat_back_addr":"172.21.15.92:6807/1994536059","heartbeat_front_addr":"172.21.15.92:6805/1994536059","state":["exists","up"]},{"osd":2,"uuid":"a61fbef2-9dca-4875-a814-020fff0db0e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":1358472800},{"type":"v1","addr":"172.21.15.177:6801","nonce":1358472800}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":1358472800},{"type":"v1","addr":"172.21.15.177:6803","nonce":1358472800}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":1358472800},{"type":"v1","addr":"172.21.15.177:6807","nonce":1358472800}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":1358472800},{"type":"v1","addr":"172.21.15.177:6805","nonce":1358472800}]},"public_addr":"172.21.15.177:6801/1358472800","cluster_addr":"172.21.15.177:6803/1358472800","heartbeat_back_addr":"172.21.15.177:6807/1358472800","heartbeat_front_addr":"172.21.15.177:6805/1358472800","state":["exists","up"]},{"osd":3,"uuid":"b175a2f4-7859-4eb0-bc1f-fba8c2b92a2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":3114811216},{"type":"v1","addr":"172.21.15.90:6811","nonce":3114811216}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":3114811216},{"type":"v1","addr":"172.21.15.90:6813","nonce":3114811216}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":3114811216},{"type":"v1","addr":"172.21.15.90:6817","nonce":3114811216}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":3114811216},{"type":"v1","addr":"172.21.15.90:6815","nonce":3114811216}]},"public_addr":"172.21.15.90:6811/3114811216","cluster_addr":"172.21.15.90:6813/3114811216","heartbeat_back_addr":"172.21.15.90:6817/3114811216","heartbeat_front_addr":"172.21.15.90:6815/3114811216","state":["exists","up"]},{"osd":4,"uuid":"38cc68ea-c109-4d8d-b77d-1c6f9b56c875","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":2987208879},{"type":"v1","addr":"172.21.15.92:6809","nonce":2987208879}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6810","nonce":2987208879},{"type":"v1","addr":"172.21.15.92:6811","nonce":2987208879}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6814","nonce":2987208879},{"type":"v1","addr":"172.21.15.92:6815","nonce":2987208879}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":2987208879},{"type":"v1","addr":"172.21.15.92:6813","nonce":2987208879}]},"public_addr":"172.21.15.92:6809/2987208879","cluster_addr":"172.21.15.92:6811/2987208879","heartbeat_back_addr":"172.21.15.92:6815/2987208879","heartbeat_front_addr":"172.21.15.92:6813/2987208879","state":["exists","up"]},{"osd":5,"uuid":"2347232e-ac91-466f-8ac9-496f397679ba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":3486820237},{"type":"v1","addr":"172.21.15.177:6809","nonce":3486820237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":3486820237},{"type":"v1","addr":"172.21.15.177:6811","nonce":3486820237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":3486820237},{"type":"v1","addr":"172.21.15.177:6815","nonce":3486820237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":3486820237},{"type":"v1","addr":"172.21.15.177:6813","nonce":3486820237}]},"public_addr":"172.21.15.177:6809/3486820237","cluster_addr":"172.21.15.177:6811/3486820237","heartbeat_back_addr":"172.21.15.177:6815/3486820237","heartbeat_front_addr":"172.21.15.177:6813/3486820237","state":["exists","up"]},{"osd":6,"uuid":"aa72392f-0498-44db-a8d2-eccc89a64871","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":2579716443},{"type":"v1","addr":"172.21.15.90:6819","nonce":2579716443}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6820","nonce":2579716443},{"type":"v1","addr":"172.21.15.90:6821","nonce":2579716443}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":2579716443},{"type":"v1","addr":"172.21.15.90:6825","nonce":2579716443}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2579716443},{"type":"v1","addr":"172.21.15.90:6823","nonce":2579716443}]},"public_addr":"172.21.15.90:6819/2579716443","cluster_addr":"172.21.15.90:6821/2579716443","heartbeat_back_addr":"172.21.15.90:6825/2579716443","heartbeat_front_addr":"172.21.15.90:6823/2579716443","state":["exists","up"]},{"osd":7,"uuid":"931bd105-bc14-4a1e-bfe4-c7ad58fa4b73","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":3371649139},{"type":"v1","addr":"172.21.15.92:6817","nonce":3371649139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":3371649139},{"type":"v1","addr":"172.21.15.92:6819","nonce":3371649139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":3371649139},{"type":"v1","addr":"172.21.15.92:6823","nonce":3371649139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6820","nonce":3371649139},{"type":"v1","addr":"172.21.15.92:6821","nonce":3371649139}]},"public_addr":"172.21.15.92:6817/3371649139","cluster_addr":"172.21.15.92:6819/3371649139","heartbeat_back_addr":"172.21.15.92:6823/3371649139","heartbeat_front_addr":"172.21.15.92:6821/3371649139","state":["exists","up"]},{"osd":8,"uuid":"82624ec2-b382-4886-9f40-7f5f376de3ce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":1459053034},{"type":"v1","addr":"172.21.15.177:6817","nonce":1459053034}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":1459053034},{"type":"v1","addr":"172.21.15.177:6819","nonce":1459053034}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":1459053034},{"type":"v1","addr":"172.21.15.177:6823","nonce":1459053034}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":1459053034},{"type":"v1","addr":"172.21.15.177:6821","nonce":1459053034}]},"public_addr":"172.21.15.177:6817/1459053034","cluster_addr":"172.21.15.177:6819/1459053034","heartbeat_back_addr":"172.21.15.177:6823/1459053034","heartbeat_front_addr":"172.21.15.177:6821/1459053034","state":["exists","up"]},{"osd":9,"uuid":"fbe36e48-3468-42b2-8094-e16552f3cb16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":2592604332},{"type":"v1","addr":"172.21.15.90:6827","nonce":2592604332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":2592604332},{"type":"v1","addr":"172.21.15.90:6829","nonce":2592604332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6832","nonce":2592604332},{"type":"v1","addr":"172.21.15.90:6833","nonce":2592604332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":2592604332},{"type":"v1","addr":"172.21.15.90:6831","nonce":2592604332}]},"public_addr":"172.21.15.90:6827/2592604332","cluster_addr":"172.21.15.90:6829/2592604332","heartbeat_back_addr":"172.21.15.90:6833/2592604332","heartbeat_front_addr":"172.21.15.90:6831/2592604332","state":["exists","up"]},{"osd":10,"uuid":"9240df75-e7e2-47f1-8aae-534b70b3c210","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":1594153142},{"type":"v1","addr":"172.21.15.92:6825","nonce":1594153142}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6826","nonce":1594153142},{"type":"v1","addr":"172.21.15.92:6827","nonce":1594153142}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":1594153142},{"type":"v1","addr":"172.21.15.92:6831","nonce":1594153142}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":1594153142},{"type":"v1","addr":"172.21.15.92:6829","nonce":1594153142}]},"public_addr":"172.21.15.92:6825/1594153142","cluster_addr":"172.21.15.92:6827/1594153142","heartbeat_back_addr":"172.21.15.92:6831/1594153142","heartbeat_front_addr":"172.21.15.92:6829/1594153142","state":["exists","up"]},{"osd":11,"uuid":"83cba9fa-b745-4df1-adda-e86837d4b91a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":83,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1622582126},{"type":"v1","addr":"172.21.15.177:6825","nonce":1622582126}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":1622582126},{"type":"v1","addr":"172.21.15.177:6827","nonce":1622582126}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":1622582126},{"type":"v1","addr":"172.21.15.177:6831","nonce":1622582126}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1622582126},{"type":"v1","addr":"172.21.15.177:6829","nonce":1622582126}]},"public_addr":"172.21.15.177:6825/1622582126","cluster_addr":"172.21.15.177:6827/1622582126","heartbeat_back_addr":"172.21.15.177:6831/1622582126","heartbeat_front_addr":"172.21.15.177:6829/1622582126","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:26:24.290365+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:26:49.405988+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:27:13.639196+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:27:45.603746+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:28:11.019126+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:28:35.084163+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:29:08.064626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:29:33.266133+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:29:57.381816+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:30:30.286847+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:30:56.226279+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:31:20.809611+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":7,"to":11}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":8,"to":4}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.11","mappings":[{"from":4,"to":8},{"from":5,"to":9}]},{"pgid":"3.21","mappings":[{"from":5,"to":9}]},{"pgid":"3.23","mappings":[{"from":4,"to":7}]},{"pgid":"3.3a","mappings":[{"from":4,"to":6}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.90:0/2805994468":"2024-04-03T14:34:52.235215+0000","172.21.15.90:0/1431307002":"2024-04-03T14:34:52.235215+0000","172.21.15.90:6801/4266054817":"2024-04-03T14:34:52.235215+0000","172.21.15.90:0/314641601":"2024-04-03T14:24:34.595561+0000","172.21.15.90:0/1400895868":"2024-04-03T14:24:34.595561+0000","172.21.15.90:6801/129436436":"2024-04-03T14:24:34.595561+0000","172.21.15.90:0/1335673697":"2024-04-03T14:34:52.235215+0000","172.21.15.90:6801/2372398598":"2024-04-03T14:24:00.362147+0000","172.21.15.90:0/278569338":"2024-04-03T14:24:34.595561+0000","172.21.15.90:0/2321464594":"2024-04-03T14:24:00.362147+0000","172.21.15.90:6800/4266054817":"2024-04-03T14:34:52.235215+0000","172.21.15.90:6800/2372398598":"2024-04-03T14:24:00.362147+0000","172.21.15.90:0/2962656672":"2024-04-03T14:24:00.362147+0000","172.21.15.90:0/126693610":"2024-04-03T14:24:00.362147+0000","172.21.15.90:0/1757385857":"2024-04-03T14:34:52.235215+0000","172.21.15.90:6800/129436436":"2024-04-03T14:24:34.595561+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T14:34:54.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:54.197+0000 7f7749c00640 1 -- 172.21.15.90:0/3906444513 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f77280a3a30 msgr2=0x7f77280a5ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:54.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:54.197+0000 7f7749c00640 1 --2- 172.21.15.90:0/3906444513 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f77280a3a30 0x7f77280a5ef0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f773402c170 tx=0x7f773402df90 comp rx=0 tx=0).stop 2024-04-02T14:34:54.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:54.197+0000 7f7749c00640 1 -- 172.21.15.90:0/3906444513 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f77441036d0 msgr2=0x7f77441a1620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:54.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:54.197+0000 7f7749c00640 1 --2- 172.21.15.90:0/3906444513 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f77441036d0 0x7f77441a1620 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f773c00eb10 tx=0x7f773c00c2d0 comp rx=0 tx=0).stop 2024-04-02T14:34:54.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:54.197+0000 7f7749c00640 1 -- 172.21.15.90:0/3906444513 shutdown_connections 2024-04-02T14:34:54.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:54.197+0000 7f7749c00640 1 --2- 172.21.15.90:0/3906444513 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f77280a3a30 0x7f77280a5ef0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:54.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:54.197+0000 7f7749c00640 1 --2- 172.21.15.90:0/3906444513 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f77441036d0 0x7f77441a1620 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:54.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:54.197+0000 7f7749c00640 1 --2- 172.21.15.90:0/3906444513 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7744102d10 0x7f774419a5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:54.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:54.197+0000 7f7749c00640 1 --2- 172.21.15.90:0/3906444513 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7744101b20 0x7f774419a070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:54.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:54.197+0000 7f7749c00640 1 -- 172.21.15.90:0/3906444513 >> 172.21.15.90:0/3906444513 conn(0x7f77440fd0b0 msgr2=0x7f7744073d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:34:54.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:54.197+0000 7f7749c00640 1 -- 172.21.15.90:0/3906444513 shutdown_connections 2024-04-02T14:34:54.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:54.197+0000 7f7749c00640 1 -- 172.21.15.90:0/3906444513 wait complete. 2024-04-02T14:34:54.216 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-02T14:27:16.758622+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-02T14:32:20.016331+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-02T14:32:21.294367+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '88', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-02T14:34:54.217 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-02T14:34:54.410 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:54.408+0000 7faea9e00640 1 -- 172.21.15.90:0/2237621862 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7faea4102580 msgr2=0x7faea4104970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:54.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:54.408+0000 7faea9e00640 1 --2- 172.21.15.90:0/2237621862 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7faea4102580 0x7faea4104970 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fae94009860 tx=0x7fae9402bb70 comp rx=0 tx=0).stop 2024-04-02T14:34:54.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:54.408+0000 7faea9e00640 1 -- 172.21.15.90:0/2237621862 shutdown_connections 2024-04-02T14:34:54.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:54.408+0000 7faea9e00640 1 --2- 172.21.15.90:0/2237621862 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faea41077e0 0x7faea4109c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:54.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:54.408+0000 7faea9e00640 1 --2- 172.21.15.90:0/2237621862 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faea4104eb0 0x7faea41072a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:54.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:54.408+0000 7faea9e00640 1 --2- 172.21.15.90:0/2237621862 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7faea4102580 0x7faea4104970 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:54.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:54.408+0000 7faea9e00640 1 -- 172.21.15.90:0/2237621862 >> 172.21.15.90:0/2237621862 conn(0x7faea40fbf70 msgr2=0x7faea40fe3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:34:54.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:54.408+0000 7faea9e00640 1 -- 172.21.15.90:0/2237621862 shutdown_connections 2024-04-02T14:34:54.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:54.408+0000 7faea9e00640 1 -- 172.21.15.90:0/2237621862 wait complete. 2024-04-02T14:34:54.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:54.409+0000 7faea9e00640 1 Processor -- start 2024-04-02T14:34:54.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:54.410+0000 7faea9e00640 1 -- start start 2024-04-02T14:34:54.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:54.410+0000 7faea9e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faea4102580 0x7faea4101ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:54.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:54.410+0000 7faea9e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faea4104eb0 0x7faea4100530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:54.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:54.410+0000 7faea9e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7faea41077e0 0x7faea4100a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:54.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:54.410+0000 7faea9e00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7faea410d7f0 con 0x7faea4102580 2024-04-02T14:34:54.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:54.410+0000 7faea9e00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7faea410d670 con 0x7faea41077e0 2024-04-02T14:34:54.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:54.410+0000 7faea9e00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7faea410d970 con 0x7faea4104eb0 2024-04-02T14:34:54.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:54.411+0000 7faea3400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7faea41077e0 0x7faea4100a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:54.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:54.411+0000 7faea3400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7faea41077e0 0x7faea4100a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:58746/0 (socket says 172.21.15.90:58746) 2024-04-02T14:34:54.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:54.411+0000 7faea3400640 1 -- 172.21.15.90:0/3121428140 learned_addr learned my addr 172.21.15.90:0/3121428140 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:34:54.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:54.411+0000 7faea2a00640 1 --2- 172.21.15.90:0/3121428140 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faea4102580 0x7faea4101ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:54.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:54.411+0000 7faea3400640 1 -- 172.21.15.90:0/3121428140 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faea4104eb0 msgr2=0x7faea4100530 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T14:34:54.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:54.411+0000 7faea3400640 1 --2- 172.21.15.90:0/3121428140 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faea4104eb0 0x7faea4100530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:54.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:54.411+0000 7faea3400640 1 -- 172.21.15.90:0/3121428140 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faea4102580 msgr2=0x7faea4101ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:54.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:54.411+0000 7faea3400640 1 --2- 172.21.15.90:0/3121428140 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faea4102580 0x7faea4101ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:54.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:54.411+0000 7faea3400640 1 -- 172.21.15.90:0/3121428140 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faea419bdf0 con 0x7faea41077e0 2024-04-02T14:34:54.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:54.411+0000 7faea3400640 1 --2- 172.21.15.90:0/3121428140 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7faea41077e0 0x7faea4100a70 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fae9c00b970 tx=0x7fae9c00be30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:34:54.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:54.412+0000 7faea8a00640 1 -- 172.21.15.90:0/3121428140 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae9c019070 con 0x7faea41077e0 2024-04-02T14:34:54.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:54.412+0000 7faea8a00640 1 -- 172.21.15.90:0/3121428140 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fae9c00cb40 con 0x7faea41077e0 2024-04-02T14:34:54.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:54.412+0000 7faea8a00640 1 -- 172.21.15.90:0/3121428140 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae9c007650 con 0x7faea41077e0 2024-04-02T14:34:54.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:54.412+0000 7faea9e00640 1 -- 172.21.15.90:0/3121428140 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faea419c0e0 con 0x7faea41077e0 2024-04-02T14:34:54.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:54.412+0000 7faea9e00640 1 -- 172.21.15.90:0/3121428140 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faea419c660 con 0x7faea41077e0 2024-04-02T14:34:54.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:54.414+0000 7faea9e00640 1 -- 172.21.15.90:0/3121428140 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fae840052c0 con 0x7faea41077e0 2024-04-02T14:34:54.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:54.415+0000 7faea8a00640 1 -- 172.21.15.90:0/3121428140 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 22) ==== 138458+0+0 (secure 0 0 0) 0x7fae9c01c070 con 0x7faea41077e0 2024-04-02T14:34:54.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:54.416+0000 7faea8a00640 1 --2- 172.21.15.90:0/3121428140 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fae880a3a30 0x7fae880a5ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:54.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:54.416+0000 7faea8a00640 1 -- 172.21.15.90:0/3121428140 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7fae9c0cf570 con 0x7faea41077e0 2024-04-02T14:34:54.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:54.416+0000 7faea2a00640 1 --2- 172.21.15.90:0/3121428140 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fae880a3a30 0x7fae880a5ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:54.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:54.417+0000 7faea2a00640 1 --2- 172.21.15.90:0/3121428140 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fae880a3a30 0x7fae880a5ef0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fae94009830 tx=0x7fae94034ab0 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:34:54.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:54.420+0000 7faea8a00640 1 -- 172.21.15.90:0/3121428140 <== mon.2 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7fae9c09baa0 con 0x7faea41077e0 2024-04-02T14:34:54.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:54.554+0000 7faea8a00640 1 -- 172.21.15.90:0/3121428140 <== mon.2 v2:172.21.15.92:3300/0 7 ==== mgrmap(e 23) ==== 138817+0+0 (secure 0 0 0) 0x7fae9c00ccf0 con 0x7faea41077e0 2024-04-02T14:34:54.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:54.645+0000 7faea9e00640 1 -- 172.21.15.90:0/3121428140 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fae84005dc0 con 0x7faea41077e0 2024-04-02T14:34:54.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:54.646+0000 7faea8a00640 1 -- 172.21.15.90:0/3121428140 <== mon.2 v2:172.21.15.92:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v91) ==== 93+0+10 (secure 0 0 0) 0x7fae9c092040 con 0x7faea41077e0 2024-04-02T14:34:54.650 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 1 2024-04-02T14:34:54.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:54.649+0000 7faea9e00640 1 -- 172.21.15.90:0/3121428140 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fae880a3a30 msgr2=0x7fae880a5ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:54.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:54.649+0000 7faea9e00640 1 --2- 172.21.15.90:0/3121428140 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fae880a3a30 0x7fae880a5ef0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fae94009830 tx=0x7fae94034ab0 comp rx=0 tx=0).stop 2024-04-02T14:34:54.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:54.649+0000 7faea9e00640 1 -- 172.21.15.90:0/3121428140 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7faea41077e0 msgr2=0x7faea4100a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:54.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:54.649+0000 7faea9e00640 1 --2- 172.21.15.90:0/3121428140 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7faea41077e0 0x7faea4100a70 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fae9c00b970 tx=0x7fae9c00be30 comp rx=0 tx=0).stop 2024-04-02T14:34:54.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:54.649+0000 7faea9e00640 1 -- 172.21.15.90:0/3121428140 shutdown_connections 2024-04-02T14:34:54.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:54.649+0000 7faea9e00640 1 --2- 172.21.15.90:0/3121428140 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fae880a3a30 0x7fae880a5ef0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:54.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:54.649+0000 7faea9e00640 1 --2- 172.21.15.90:0/3121428140 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7faea41077e0 0x7faea4100a70 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:54.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:54.649+0000 7faea9e00640 1 --2- 172.21.15.90:0/3121428140 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faea4104eb0 0x7faea4100530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:54.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:54.649+0000 7faea9e00640 1 --2- 172.21.15.90:0/3121428140 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faea4102580 0x7faea4101ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:54.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:54.649+0000 7faea9e00640 1 -- 172.21.15.90:0/3121428140 >> 172.21.15.90:0/3121428140 conn(0x7faea40fbf70 msgr2=0x7faea4102ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:34:54.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:54.649+0000 7faea9e00640 1 -- 172.21.15.90:0/3121428140 shutdown_connections 2024-04-02T14:34:54.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:54.649+0000 7faea9e00640 1 -- 172.21.15.90:0/3121428140 wait complete. 2024-04-02T14:34:54.668 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-02T14:34:54.827 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:34:54 smithi090 ceph-mon[31674]: pgmap v3: 97 pgs: 97 active+clean; 1.1 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-02T14:34:54.827 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:34:54 smithi090 ceph-mon[31674]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' 2024-04-02T14:34:54.827 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:34:54 smithi090 ceph-mon[31674]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' 2024-04-02T14:34:54.827 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:34:54 smithi090 ceph-mon[31674]: from='client.25355 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T14:34:54.827 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:34:54 smithi090 ceph-mon[31674]: from='client.? 172.21.15.90:0/3906444513' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-02T14:34:54.828 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:34:54 smithi090 ceph-mon[31674]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' 2024-04-02T14:34:54.828 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:34:54 smithi090 ceph-mon[31674]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' 2024-04-02T14:34:54.828 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:34:54 smithi090 ceph-mon[31674]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' 2024-04-02T14:34:54.828 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:34:54 smithi090 ceph-mon[31674]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' 2024-04-02T14:34:54.861 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:34:54 smithi092 ceph-mon[36497]: pgmap v3: 97 pgs: 97 active+clean; 1.1 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-02T14:34:54.861 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:34:54 smithi092 ceph-mon[36497]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' 2024-04-02T14:34:54.861 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:34:54 smithi092 ceph-mon[36497]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' 2024-04-02T14:34:54.861 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:34:54 smithi092 ceph-mon[36497]: from='client.25355 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T14:34:54.861 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:34:54 smithi092 ceph-mon[36497]: from='client.? 172.21.15.90:0/3906444513' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-02T14:34:54.861 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:34:54 smithi092 ceph-mon[36497]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' 2024-04-02T14:34:54.861 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:34:54 smithi092 ceph-mon[36497]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' 2024-04-02T14:34:54.861 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:34:54 smithi092 ceph-mon[36497]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' 2024-04-02T14:34:54.862 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:34:54 smithi092 ceph-mon[36497]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' 2024-04-02T14:34:54.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:54.861+0000 7f832e000640 1 -- 172.21.15.90:0/1619425656 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f83281003d0 msgr2=0x7f83281007f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:54.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:54.861+0000 7f832e000640 1 --2- 172.21.15.90:0/1619425656 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f83281003d0 0x7f83281007f0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f8318009860 tx=0x7f831802bb70 comp rx=0 tx=0).stop 2024-04-02T14:34:54.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:54.861+0000 7f832e000640 1 -- 172.21.15.90:0/1619425656 shutdown_connections 2024-04-02T14:34:54.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:54.861+0000 7f832e000640 1 --2- 172.21.15.90:0/1619425656 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f83281016d0 0x7f8328101b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:54.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:54.861+0000 7f832e000640 1 --2- 172.21.15.90:0/1619425656 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8328100d30 0x7f8328101190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:54.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:54.861+0000 7f832e000640 1 --2- 172.21.15.90:0/1619425656 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f83281003d0 0x7f83281007f0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:54.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:54.861+0000 7f832e000640 1 -- 172.21.15.90:0/1619425656 >> 172.21.15.90:0/1619425656 conn(0x7f83280fbf90 msgr2=0x7f83280fe3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:34:54.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:54.862+0000 7f832e000640 1 -- 172.21.15.90:0/1619425656 shutdown_connections 2024-04-02T14:34:54.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:54.862+0000 7f832e000640 1 -- 172.21.15.90:0/1619425656 wait complete. 2024-04-02T14:34:54.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:54.863+0000 7f832e000640 1 Processor -- start 2024-04-02T14:34:54.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:54.863+0000 7f832e000640 1 -- start start 2024-04-02T14:34:54.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:54.863+0000 7f832e000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f83281003d0 0x7f832819e430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:54.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:54.863+0000 7f832e000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8328100d30 0x7f832819e970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:54.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:54.863+0000 7f832e000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f83281016d0 0x7f83281a59e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:54.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:54.863+0000 7f832e000640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f832810f940 con 0x7f83281003d0 2024-04-02T14:34:54.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:54.863+0000 7f832e000640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f832810f7c0 con 0x7f8328100d30 2024-04-02T14:34:54.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:54.863+0000 7f832e000640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f832810fac0 con 0x7f83281016d0 2024-04-02T14:34:54.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:54.863+0000 7f8326a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8328100d30 0x7f832819e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:54.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:54.863+0000 7f8326a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8328100d30 0x7f832819e970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:58770/0 (socket says 172.21.15.90:58770) 2024-04-02T14:34:54.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:54.863+0000 7f8327400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f83281003d0 0x7f832819e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:54.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:54.863+0000 7f8326a00640 1 -- 172.21.15.90:0/84076376 learned_addr learned my addr 172.21.15.90:0/84076376 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:34:54.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:54.863+0000 7f8327e00640 1 --2- 172.21.15.90:0/84076376 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f83281016d0 0x7f83281a59e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:54.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:54.864+0000 7f8326a00640 1 -- 172.21.15.90:0/84076376 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f83281016d0 msgr2=0x7f83281a59e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:54.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:54.864+0000 7f8326a00640 1 --2- 172.21.15.90:0/84076376 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f83281016d0 0x7f83281a59e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:54.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:54.864+0000 7f8326a00640 1 -- 172.21.15.90:0/84076376 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f83281003d0 msgr2=0x7f832819e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:54.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:54.864+0000 7f8326a00640 1 --2- 172.21.15.90:0/84076376 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f83281003d0 0x7f832819e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:54.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:54.864+0000 7f8326a00640 1 -- 172.21.15.90:0/84076376 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83281a6050 con 0x7f8328100d30 2024-04-02T14:34:54.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:54.864+0000 7f8326a00640 1 --2- 172.21.15.90:0/84076376 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8328100d30 0x7f832819e970 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f831400e9c0 tx=0x7f831400ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:34:54.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:54.864+0000 7f831fe00640 1 -- 172.21.15.90:0/84076376 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f831400cde0 con 0x7f8328100d30 2024-04-02T14:34:54.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:54.864+0000 7f832e000640 1 -- 172.21.15.90:0/84076376 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83281a6340 con 0x7f8328100d30 2024-04-02T14:34:54.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:54.864+0000 7f831fe00640 1 -- 172.21.15.90:0/84076376 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f83140035f0 con 0x7f8328100d30 2024-04-02T14:34:54.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:54.864+0000 7f831fe00640 1 -- 172.21.15.90:0/84076376 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8314010690 con 0x7f8328100d30 2024-04-02T14:34:54.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:54.864+0000 7f832e000640 1 -- 172.21.15.90:0/84076376 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83281a6890 con 0x7f8328100d30 2024-04-02T14:34:54.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:54.865+0000 7f832e000640 1 -- 172.21.15.90:0/84076376 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8328106fa0 con 0x7f8328100d30 2024-04-02T14:34:54.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:54.868+0000 7f831fe00640 1 -- 172.21.15.90:0/84076376 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 23) ==== 138817+0+0 (secure 0 0 0) 0x7f83140037a0 con 0x7f8328100d30 2024-04-02T14:34:54.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:54.869+0000 7f831fe00640 1 --2- 172.21.15.90:0/84076376 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f830c0a3db0 0x7f830c0a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:54.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:54.869+0000 7f831fe00640 1 -- 172.21.15.90:0/84076376 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f83140d0e70 con 0x7f8328100d30 2024-04-02T14:34:54.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:54.870+0000 7f8327400640 1 --2- 172.21.15.90:0/84076376 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f830c0a3db0 0x7f830c0a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:54.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:54.870+0000 7f8327400640 1 --2- 172.21.15.90:0/84076376 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f830c0a3db0 0x7f830c0a6270 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f8318009830 tx=0x7f8318034ab0 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:34:54.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:54.871+0000 7f831fe00640 1 -- 172.21.15.90:0/84076376 <== mon.2 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7f831409d350 con 0x7f8328100d30 2024-04-02T14:34:54.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:34:54 smithi177 ceph-mon[36431]: pgmap v3: 97 pgs: 97 active+clean; 1.1 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-02T14:34:54.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:34:54 smithi177 ceph-mon[36431]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' 2024-04-02T14:34:54.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:34:54 smithi177 ceph-mon[36431]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' 2024-04-02T14:34:54.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:34:54 smithi177 ceph-mon[36431]: from='client.25355 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T14:34:54.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:34:54 smithi177 ceph-mon[36431]: from='client.? 172.21.15.90:0/3906444513' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-02T14:34:54.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:34:54 smithi177 ceph-mon[36431]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' 2024-04-02T14:34:54.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:34:54 smithi177 ceph-mon[36431]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' 2024-04-02T14:34:54.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:34:54 smithi177 ceph-mon[36431]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' 2024-04-02T14:34:54.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:34:54 smithi177 ceph-mon[36431]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' 2024-04-02T14:34:55.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:55.103+0000 7f832e000640 1 -- 172.21.15.90:0/84076376 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f832802d5c0 con 0x7f8328100d30 2024-04-02T14:34:55.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:55.104+0000 7f831fe00640 1 -- 172.21.15.90:0/84076376 <== mon.2 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v91) ==== 104+0+11 (secure 0 0 0) 0x7f831409cc50 con 0x7f8328100d30 2024-04-02T14:34:55.106 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-04-02T14:34:55.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:55.107+0000 7f832e000640 1 -- 172.21.15.90:0/84076376 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f830c0a3db0 msgr2=0x7f830c0a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:55.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:55.107+0000 7f832e000640 1 --2- 172.21.15.90:0/84076376 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f830c0a3db0 0x7f830c0a6270 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f8318009830 tx=0x7f8318034ab0 comp rx=0 tx=0).stop 2024-04-02T14:34:55.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:55.107+0000 7f832e000640 1 -- 172.21.15.90:0/84076376 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8328100d30 msgr2=0x7f832819e970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:55.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:55.107+0000 7f832e000640 1 --2- 172.21.15.90:0/84076376 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8328100d30 0x7f832819e970 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f831400e9c0 tx=0x7f831400ee80 comp rx=0 tx=0).stop 2024-04-02T14:34:55.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:55.107+0000 7f832e000640 1 -- 172.21.15.90:0/84076376 shutdown_connections 2024-04-02T14:34:55.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:55.107+0000 7f832e000640 1 --2- 172.21.15.90:0/84076376 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f830c0a3db0 0x7f830c0a6270 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:55.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:55.107+0000 7f832e000640 1 --2- 172.21.15.90:0/84076376 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f83281016d0 0x7f83281a59e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:55.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:55.107+0000 7f832e000640 1 --2- 172.21.15.90:0/84076376 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8328100d30 0x7f832819e970 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:55.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:55.107+0000 7f832e000640 1 --2- 172.21.15.90:0/84076376 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f83281003d0 0x7f832819e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:55.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:55.107+0000 7f832e000640 1 -- 172.21.15.90:0/84076376 >> 172.21.15.90:0/84076376 conn(0x7f83280fbf90 msgr2=0x7f83280fd730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:34:55.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:55.108+0000 7f832e000640 1 -- 172.21.15.90:0/84076376 shutdown_connections 2024-04-02T14:34:55.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:55.108+0000 7f832e000640 1 -- 172.21.15.90:0/84076376 wait complete. 2024-04-02T14:34:55.128 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-02T14:34:55.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:55.319+0000 7f9356600640 1 -- 172.21.15.90:0/2576363038 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9350075ac0 msgr2=0x7f9350073f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:55.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:55.319+0000 7f9356600640 1 --2- 172.21.15.90:0/2576363038 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9350075ac0 0x7f9350073f10 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f9340009900 tx=0x7f934002bcd0 comp rx=0 tx=0).stop 2024-04-02T14:34:55.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:55.319+0000 7f9356600640 1 -- 172.21.15.90:0/2576363038 shutdown_connections 2024-04-02T14:34:55.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:55.319+0000 7f9356600640 1 --2- 172.21.15.90:0/2576363038 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f93501099c0 0x7f935010be50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:55.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:55.319+0000 7f9356600640 1 --2- 172.21.15.90:0/2576363038 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f93500744e0 0x7f9350074940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:55.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:55.319+0000 7f9356600640 1 --2- 172.21.15.90:0/2576363038 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9350075ac0 0x7f9350073f10 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:55.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:55.319+0000 7f9356600640 1 -- 172.21.15.90:0/2576363038 >> 172.21.15.90:0/2576363038 conn(0x7f93500fd0a0 msgr2=0x7f93500ff4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:34:55.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:55.320+0000 7f9356600640 1 -- 172.21.15.90:0/2576363038 shutdown_connections 2024-04-02T14:34:55.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:55.320+0000 7f9356600640 1 -- 172.21.15.90:0/2576363038 wait complete. 2024-04-02T14:34:55.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:55.320+0000 7f9356600640 1 Processor -- start 2024-04-02T14:34:55.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:55.321+0000 7f9356600640 1 -- start start 2024-04-02T14:34:55.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:55.321+0000 7f9356600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f93500744e0 0x7f935019a080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:55.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:55.321+0000 7f9356600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9350075ac0 0x7f935019a5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:55.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:55.321+0000 7f9356600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f93501099c0 0x7f93501a1630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:55.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:55.321+0000 7f9356600640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f935010f990 con 0x7f9350075ac0 2024-04-02T14:34:55.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:55.321+0000 7f9356600640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f935010f810 con 0x7f93500744e0 2024-04-02T14:34:55.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:55.321+0000 7f9356600640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f935010fb10 con 0x7f93501099c0 2024-04-02T14:34:55.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:55.321+0000 7f934f400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f93500744e0 0x7f935019a080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:55.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:55.321+0000 7f934f400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f93500744e0 0x7f935019a080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:58800/0 (socket says 172.21.15.90:58800) 2024-04-02T14:34:55.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:55.321+0000 7f934f400640 1 -- 172.21.15.90:0/183147717 learned_addr learned my addr 172.21.15.90:0/183147717 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:34:55.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:55.321+0000 7f934ea00640 1 --2- 172.21.15.90:0/183147717 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9350075ac0 0x7f935019a5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:55.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:55.321+0000 7f934fe00640 1 --2- 172.21.15.90:0/183147717 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f93501099c0 0x7f93501a1630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:55.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:55.322+0000 7f934f400640 1 -- 172.21.15.90:0/183147717 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f93501099c0 msgr2=0x7f93501a1630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:55.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:55.322+0000 7f934f400640 1 --2- 172.21.15.90:0/183147717 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f93501099c0 0x7f93501a1630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:55.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:55.322+0000 7f934f400640 1 -- 172.21.15.90:0/183147717 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9350075ac0 msgr2=0x7f935019a5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:55.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:55.322+0000 7f934f400640 1 --2- 172.21.15.90:0/183147717 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9350075ac0 0x7f935019a5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:55.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:55.322+0000 7f934f400640 1 -- 172.21.15.90:0/183147717 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93501a1ca0 con 0x7f93500744e0 2024-04-02T14:34:55.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:55.322+0000 7f934ea00640 1 --2- 172.21.15.90:0/183147717 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9350075ac0 0x7f935019a5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:34:55.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:55.322+0000 7f934f400640 1 --2- 172.21.15.90:0/183147717 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f93500744e0 0x7f935019a080 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f9340005ab0 tx=0x7f934002c6e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:34:55.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:55.322+0000 7f9347e00640 1 -- 172.21.15.90:0/183147717 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9340041070 con 0x7f93500744e0 2024-04-02T14:34:55.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:55.322+0000 7f9347e00640 1 -- 172.21.15.90:0/183147717 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9340004450 con 0x7f93500744e0 2024-04-02T14:34:55.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:55.322+0000 7f9356600640 1 -- 172.21.15.90:0/183147717 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f93501a1f30 con 0x7f93500744e0 2024-04-02T14:34:55.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:55.322+0000 7f9347e00640 1 -- 172.21.15.90:0/183147717 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9340034440 con 0x7f93500744e0 2024-04-02T14:34:55.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:55.322+0000 7f9356600640 1 -- 172.21.15.90:0/183147717 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f93501a2340 con 0x7f93500744e0 2024-04-02T14:34:55.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:55.323+0000 7f9356600640 1 -- 172.21.15.90:0/183147717 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9350108780 con 0x7f93500744e0 2024-04-02T14:34:55.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:55.325+0000 7f9347e00640 1 -- 172.21.15.90:0/183147717 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 23) ==== 138817+0+0 (secure 0 0 0) 0x7f9340044070 con 0x7f93500744e0 2024-04-02T14:34:55.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:55.327+0000 7f9347e00640 1 --2- 172.21.15.90:0/183147717 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f93340a3db0 0x7f93340a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:55.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:55.327+0000 7f9347e00640 1 -- 172.21.15.90:0/183147717 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f93400f23e0 con 0x7f93500744e0 2024-04-02T14:34:55.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:55.327+0000 7f934ea00640 1 --2- 172.21.15.90:0/183147717 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f93340a3db0 0x7f93340a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:55.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:55.328+0000 7f934ea00640 1 --2- 172.21.15.90:0/183147717 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f93340a3db0 0x7f93340a6270 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f935019b590 tx=0x7f933c005f90 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:34:55.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:55.330+0000 7f9347e00640 1 -- 172.21.15.90:0/183147717 <== mon.2 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7f93400bea60 con 0x7f93500744e0 2024-04-02T14:34:55.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:55.562+0000 7f9356600640 1 -- 172.21.15.90:0/183147717 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f935002d590 con 0x7f93500744e0 2024-04-02T14:34:55.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:55.564+0000 7f9347e00640 1 -- 172.21.15.90:0/183147717 <== mon.2 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v91) ==== 100+0+11 (secure 0 0 0) 0x7f93400be360 con 0x7f93500744e0 2024-04-02T14:34:55.566 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 64 2024-04-02T14:34:55.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:55.567+0000 7f9356600640 1 -- 172.21.15.90:0/183147717 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f93340a3db0 msgr2=0x7f93340a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:55.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:55.567+0000 7f9356600640 1 --2- 172.21.15.90:0/183147717 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f93340a3db0 0x7f93340a6270 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f935019b590 tx=0x7f933c005f90 comp rx=0 tx=0).stop 2024-04-02T14:34:55.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:55.567+0000 7f9356600640 1 -- 172.21.15.90:0/183147717 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f93500744e0 msgr2=0x7f935019a080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:55.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:55.567+0000 7f9356600640 1 --2- 172.21.15.90:0/183147717 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f93500744e0 0x7f935019a080 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f9340005ab0 tx=0x7f934002c6e0 comp rx=0 tx=0).stop 2024-04-02T14:34:55.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:55.567+0000 7f9356600640 1 -- 172.21.15.90:0/183147717 shutdown_connections 2024-04-02T14:34:55.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:55.567+0000 7f9356600640 1 --2- 172.21.15.90:0/183147717 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f93340a3db0 0x7f93340a6270 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:55.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:55.567+0000 7f9356600640 1 --2- 172.21.15.90:0/183147717 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f93501099c0 0x7f93501a1630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:55.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:55.567+0000 7f9356600640 1 --2- 172.21.15.90:0/183147717 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9350075ac0 0x7f935019a5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:55.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:55.567+0000 7f9356600640 1 --2- 172.21.15.90:0/183147717 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f93500744e0 0x7f935019a080 secure :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f9340005ab0 tx=0x7f934002c6e0 comp rx=0 tx=0).stop 2024-04-02T14:34:55.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:55.567+0000 7f9356600640 1 -- 172.21.15.90:0/183147717 >> 172.21.15.90:0/183147717 conn(0x7f93500fd0a0 msgr2=0x7f9350105950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:34:55.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:55.567+0000 7f9356600640 1 -- 172.21.15.90:0/183147717 shutdown_connections 2024-04-02T14:34:55.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:55.567+0000 7f9356600640 1 -- 172.21.15.90:0/183147717 wait complete. 2024-04-02T14:34:55.589 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T14:34:55.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:55.763+0000 7f11a1200640 1 -- 172.21.15.90:0/2622253476 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f119c101a40 msgr2=0x7f119c101ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:55.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:55.763+0000 7f11a1200640 1 --2- 172.21.15.90:0/2622253476 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f119c101a40 0x7f119c101ec0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f1188009860 tx=0x7f118802bb70 comp rx=0 tx=0).stop 2024-04-02T14:34:55.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:55.763+0000 7f11a1200640 1 -- 172.21.15.90:0/2622253476 shutdown_connections 2024-04-02T14:34:55.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:55.763+0000 7f11a1200640 1 --2- 172.21.15.90:0/2622253476 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f119c102400 0x7f119c108c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:55.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:55.763+0000 7f11a1200640 1 --2- 172.21.15.90:0/2622253476 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f119c101a40 0x7f119c101ec0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:55.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:55.763+0000 7f11a1200640 1 --2- 172.21.15.90:0/2622253476 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f119c100850 0x7f119c100c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:55.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:55.763+0000 7f11a1200640 1 -- 172.21.15.90:0/2622253476 >> 172.21.15.90:0/2622253476 conn(0x7f119c0fbde0 msgr2=0x7f119c0fe200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:34:55.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:55.763+0000 7f11a1200640 1 -- 172.21.15.90:0/2622253476 shutdown_connections 2024-04-02T14:34:55.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:55.763+0000 7f11a1200640 1 -- 172.21.15.90:0/2622253476 wait complete. 2024-04-02T14:34:55.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:55.764+0000 7f11a1200640 1 Processor -- start 2024-04-02T14:34:55.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:55.765+0000 7f11a1200640 1 -- start start 2024-04-02T14:34:55.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:55.765+0000 7f11a1200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f119c100850 0x7f119c197c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:55.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:55.765+0000 7f11a1200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f119c101a40 0x7f119c1981b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:55.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:55.765+0000 7f11a1200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f119c102400 0x7f119c19f1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:55.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:55.765+0000 7f11a1200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f119c10d5f0 con 0x7f119c100850 2024-04-02T14:34:55.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:55.765+0000 7f11a1200640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f119c10d470 con 0x7f119c102400 2024-04-02T14:34:55.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:55.765+0000 7f11a1200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f119c10d770 con 0x7f119c101a40 2024-04-02T14:34:55.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:55.765+0000 7f119b400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f119c100850 0x7f119c197c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:55.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:55.765+0000 7f119aa00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f119c101a40 0x7f119c1981b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:55.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:55.765+0000 7f119aa00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f119c101a40 0x7f119c1981b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:49338/0 (socket says 172.21.15.90:49338) 2024-04-02T14:34:55.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:55.765+0000 7f119b400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f119c100850 0x7f119c197c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:53246/0 (socket says 172.21.15.90:53246) 2024-04-02T14:34:55.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:55.765+0000 7f119aa00640 1 -- 172.21.15.90:0/2836371880 learned_addr learned my addr 172.21.15.90:0/2836371880 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:34:55.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:55.765+0000 7f119be00640 1 --2- 172.21.15.90:0/2836371880 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f119c102400 0x7f119c19f1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:55.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:55.766+0000 7f119aa00640 1 -- 172.21.15.90:0/2836371880 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f119c102400 msgr2=0x7f119c19f1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:55.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:55.766+0000 7f119aa00640 1 --2- 172.21.15.90:0/2836371880 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f119c102400 0x7f119c19f1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:55.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:55.766+0000 7f119aa00640 1 -- 172.21.15.90:0/2836371880 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f119c100850 msgr2=0x7f119c197c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:55.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:55.766+0000 7f119aa00640 1 --2- 172.21.15.90:0/2836371880 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f119c100850 0x7f119c197c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:55.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:55.766+0000 7f119aa00640 1 -- 172.21.15.90:0/2836371880 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f119c19f880 con 0x7f119c101a40 2024-04-02T14:34:55.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:55.766+0000 7f119be00640 1 --2- 172.21.15.90:0/2836371880 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f119c102400 0x7f119c19f1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:34:55.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:55.766+0000 7f119b400640 1 --2- 172.21.15.90:0/2836371880 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f119c100850 0x7f119c197c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:34:55.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:55.766+0000 7f119aa00640 1 --2- 172.21.15.90:0/2836371880 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f119c101a40 0x7f119c1981b0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f11880349e0 tx=0x7f1188032450 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:34:55.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:55.766+0000 7f1193e00640 1 -- 172.21.15.90:0/2836371880 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1188041030 con 0x7f119c101a40 2024-04-02T14:34:55.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:55.766+0000 7f1193e00640 1 -- 172.21.15.90:0/2836371880 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f118802c510 con 0x7f119c101a40 2024-04-02T14:34:55.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:55.766+0000 7f1193e00640 1 -- 172.21.15.90:0/2836371880 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1188039040 con 0x7f119c101a40 2024-04-02T14:34:55.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:55.766+0000 7f11a1200640 1 -- 172.21.15.90:0/2836371880 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f119c19fb10 con 0x7f119c101a40 2024-04-02T14:34:55.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:55.767+0000 7f11a1200640 1 -- 172.21.15.90:0/2836371880 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f119c1a0050 con 0x7f119c101a40 2024-04-02T14:34:55.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:55.768+0000 7f11a1200640 1 -- 172.21.15.90:0/2836371880 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f119c107b90 con 0x7f119c101a40 2024-04-02T14:34:55.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:55.771+0000 7f1193e00640 1 -- 172.21.15.90:0/2836371880 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 23) ==== 138817+0+0 (secure 0 0 0) 0x7f1188044070 con 0x7f119c101a40 2024-04-02T14:34:55.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:55.772+0000 7f1193e00640 1 --2- 172.21.15.90:0/2836371880 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f11800a3e30 0x7f11800a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:55.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:55.772+0000 7f1193e00640 1 -- 172.21.15.90:0/2836371880 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f11880f1da0 con 0x7f119c101a40 2024-04-02T14:34:55.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:55.772+0000 7f119b400640 1 --2- 172.21.15.90:0/2836371880 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f11800a3e30 0x7f11800a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:55.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:55.773+0000 7f119b400640 1 --2- 172.21.15.90:0/2836371880 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f11800a3e30 0x7f11800a62f0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f118c0095f0 tx=0x7f118c005f90 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:34:55.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:55.776+0000 7f1193e00640 1 -- 172.21.15.90:0/2836371880 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7f11880be3f0 con 0x7f119c101a40 2024-04-02T14:34:55.861 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:34:55 smithi092 ceph-mon[36497]: pgmap v4: 97 pgs: 97 active+clean; 1.1 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-02T14:34:55.861 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:34:55 smithi092 ceph-mon[36497]: mgrmap e23: x(active, since 2s), standbys: z, y 2024-04-02T14:34:55.861 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:34:55 smithi092 ceph-mon[36497]: from='client.? 172.21.15.90:0/3121428140' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-02T14:34:55.861 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:34:55 smithi092 ceph-mon[36497]: from='client.? 172.21.15.90:0/84076376' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-02T14:34:55.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:34:55 smithi177 ceph-mon[36431]: pgmap v4: 97 pgs: 97 active+clean; 1.1 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-02T14:34:55.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:34:55 smithi177 ceph-mon[36431]: mgrmap e23: x(active, since 2s), standbys: z, y 2024-04-02T14:34:55.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:34:55 smithi177 ceph-mon[36431]: from='client.? 172.21.15.90:0/3121428140' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-02T14:34:55.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:34:55 smithi177 ceph-mon[36431]: from='client.? 172.21.15.90:0/84076376' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-02T14:34:55.938 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:34:55 smithi090 ceph-mon[31674]: pgmap v4: 97 pgs: 97 active+clean; 1.1 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-02T14:34:55.938 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:34:55 smithi090 ceph-mon[31674]: mgrmap e23: x(active, since 2s), standbys: z, y 2024-04-02T14:34:55.938 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:34:55 smithi090 ceph-mon[31674]: from='client.? 172.21.15.90:0/3121428140' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-02T14:34:55.939 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:34:55 smithi090 ceph-mon[31674]: from='client.? 172.21.15.90:0/84076376' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-02T14:34:55.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:55.976+0000 7f11a1200640 1 -- 172.21.15.90:0/2836371880 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f119c02d5b0 con 0x7f119c101a40 2024-04-02T14:34:55.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:55.978+0000 7f1193e00640 1 -- 172.21.15.90:0/2836371880 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+23024 (secure 0 0 0) 0x7f11880bdcf0 con 0x7f119c101a40 2024-04-02T14:34:55.983 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-02T14:34:55.983 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":91,"fsid":"6449448a-f0fc-11ee-b647-cb9ed24678a4","created":"2024-04-02T14:23:16.530201+0000","modified":"2024-04-02T14:34:52.235281+0000","last_up_change":"2024-04-02T14:31:23.819732+0000","last_in_change":"2024-04-02T14:31:02.824284+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T14:27:16.758622+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-02T14:32:20.016331+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-02T14:32:21.294367+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a1a6a856-ab44-44bd-bc5e-45c718a70cd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":3635087609},{"type":"v1","addr":"172.21.15.90:6803","nonce":3635087609}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":3635087609},{"type":"v1","addr":"172.21.15.90:6805","nonce":3635087609}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":3635087609},{"type":"v1","addr":"172.21.15.90:6809","nonce":3635087609}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":3635087609},{"type":"v1","addr":"172.21.15.90:6807","nonce":3635087609}]},"public_addr":"172.21.15.90:6803/3635087609","cluster_addr":"172.21.15.90:6805/3635087609","heartbeat_back_addr":"172.21.15.90:6809/3635087609","heartbeat_front_addr":"172.21.15.90:6807/3635087609","state":["exists","up"]},{"osd":1,"uuid":"124139cd-5793-463e-9607-9d9c10164183","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":1994536059},{"type":"v1","addr":"172.21.15.92:6801","nonce":1994536059}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":1994536059},{"type":"v1","addr":"172.21.15.92:6803","nonce":1994536059}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":1994536059},{"type":"v1","addr":"172.21.15.92:6807","nonce":1994536059}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":1994536059},{"type":"v1","addr":"172.21.15.92:6805","nonce":1994536059}]},"public_addr":"172.21.15.92:6801/1994536059","cluster_addr":"172.21.15.92:6803/1994536059","heartbeat_back_addr":"172.21.15.92:6807/1994536059","heartbeat_front_addr":"172.21.15.92:6805/1994536059","state":["exists","up"]},{"osd":2,"uuid":"a61fbef2-9dca-4875-a814-020fff0db0e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":1358472800},{"type":"v1","addr":"172.21.15.177:6801","nonce":1358472800}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":1358472800},{"type":"v1","addr":"172.21.15.177:6803","nonce":1358472800}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":1358472800},{"type":"v1","addr":"172.21.15.177:6807","nonce":1358472800}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":1358472800},{"type":"v1","addr":"172.21.15.177:6805","nonce":1358472800}]},"public_addr":"172.21.15.177:6801/1358472800","cluster_addr":"172.21.15.177:6803/1358472800","heartbeat_back_addr":"172.21.15.177:6807/1358472800","heartbeat_front_addr":"172.21.15.177:6805/1358472800","state":["exists","up"]},{"osd":3,"uuid":"b175a2f4-7859-4eb0-bc1f-fba8c2b92a2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":3114811216},{"type":"v1","addr":"172.21.15.90:6811","nonce":3114811216}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":3114811216},{"type":"v1","addr":"172.21.15.90:6813","nonce":3114811216}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":3114811216},{"type":"v1","addr":"172.21.15.90:6817","nonce":3114811216}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":3114811216},{"type":"v1","addr":"172.21.15.90:6815","nonce":3114811216}]},"public_addr":"172.21.15.90:6811/3114811216","cluster_addr":"172.21.15.90:6813/3114811216","heartbeat_back_addr":"172.21.15.90:6817/3114811216","heartbeat_front_addr":"172.21.15.90:6815/3114811216","state":["exists","up"]},{"osd":4,"uuid":"38cc68ea-c109-4d8d-b77d-1c6f9b56c875","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":2987208879},{"type":"v1","addr":"172.21.15.92:6809","nonce":2987208879}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6810","nonce":2987208879},{"type":"v1","addr":"172.21.15.92:6811","nonce":2987208879}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6814","nonce":2987208879},{"type":"v1","addr":"172.21.15.92:6815","nonce":2987208879}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":2987208879},{"type":"v1","addr":"172.21.15.92:6813","nonce":2987208879}]},"public_addr":"172.21.15.92:6809/2987208879","cluster_addr":"172.21.15.92:6811/2987208879","heartbeat_back_addr":"172.21.15.92:6815/2987208879","heartbeat_front_addr":"172.21.15.92:6813/2987208879","state":["exists","up"]},{"osd":5,"uuid":"2347232e-ac91-466f-8ac9-496f397679ba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":3486820237},{"type":"v1","addr":"172.21.15.177:6809","nonce":3486820237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":3486820237},{"type":"v1","addr":"172.21.15.177:6811","nonce":3486820237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":3486820237},{"type":"v1","addr":"172.21.15.177:6815","nonce":3486820237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":3486820237},{"type":"v1","addr":"172.21.15.177:6813","nonce":3486820237}]},"public_addr":"172.21.15.177:6809/3486820237","cluster_addr":"172.21.15.177:6811/3486820237","heartbeat_back_addr":"172.21.15.177:6815/3486820237","heartbeat_front_addr":"172.21.15.177:6813/3486820237","state":["exists","up"]},{"osd":6,"uuid":"aa72392f-0498-44db-a8d2-eccc89a64871","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":2579716443},{"type":"v1","addr":"172.21.15.90:6819","nonce":2579716443}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6820","nonce":2579716443},{"type":"v1","addr":"172.21.15.90:6821","nonce":2579716443}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":2579716443},{"type":"v1","addr":"172.21.15.90:6825","nonce":2579716443}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2579716443},{"type":"v1","addr":"172.21.15.90:6823","nonce":2579716443}]},"public_addr":"172.21.15.90:6819/2579716443","cluster_addr":"172.21.15.90:6821/2579716443","heartbeat_back_addr":"172.21.15.90:6825/2579716443","heartbeat_front_addr":"172.21.15.90:6823/2579716443","state":["exists","up"]},{"osd":7,"uuid":"931bd105-bc14-4a1e-bfe4-c7ad58fa4b73","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":3371649139},{"type":"v1","addr":"172.21.15.92:6817","nonce":3371649139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":3371649139},{"type":"v1","addr":"172.21.15.92:6819","nonce":3371649139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":3371649139},{"type":"v1","addr":"172.21.15.92:6823","nonce":3371649139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6820","nonce":3371649139},{"type":"v1","addr":"172.21.15.92:6821","nonce":3371649139}]},"public_addr":"172.21.15.92:6817/3371649139","cluster_addr":"172.21.15.92:6819/3371649139","heartbeat_back_addr":"172.21.15.92:6823/3371649139","heartbeat_front_addr":"172.21.15.92:6821/3371649139","state":["exists","up"]},{"osd":8,"uuid":"82624ec2-b382-4886-9f40-7f5f376de3ce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":1459053034},{"type":"v1","addr":"172.21.15.177:6817","nonce":1459053034}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":1459053034},{"type":"v1","addr":"172.21.15.177:6819","nonce":1459053034}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":1459053034},{"type":"v1","addr":"172.21.15.177:6823","nonce":1459053034}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":1459053034},{"type":"v1","addr":"172.21.15.177:6821","nonce":1459053034}]},"public_addr":"172.21.15.177:6817/1459053034","cluster_addr":"172.21.15.177:6819/1459053034","heartbeat_back_addr":"172.21.15.177:6823/1459053034","heartbeat_front_addr":"172.21.15.177:6821/1459053034","state":["exists","up"]},{"osd":9,"uuid":"fbe36e48-3468-42b2-8094-e16552f3cb16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":2592604332},{"type":"v1","addr":"172.21.15.90:6827","nonce":2592604332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":2592604332},{"type":"v1","addr":"172.21.15.90:6829","nonce":2592604332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6832","nonce":2592604332},{"type":"v1","addr":"172.21.15.90:6833","nonce":2592604332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":2592604332},{"type":"v1","addr":"172.21.15.90:6831","nonce":2592604332}]},"public_addr":"172.21.15.90:6827/2592604332","cluster_addr":"172.21.15.90:6829/2592604332","heartbeat_back_addr":"172.21.15.90:6833/2592604332","heartbeat_front_addr":"172.21.15.90:6831/2592604332","state":["exists","up"]},{"osd":10,"uuid":"9240df75-e7e2-47f1-8aae-534b70b3c210","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":1594153142},{"type":"v1","addr":"172.21.15.92:6825","nonce":1594153142}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6826","nonce":1594153142},{"type":"v1","addr":"172.21.15.92:6827","nonce":1594153142}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":1594153142},{"type":"v1","addr":"172.21.15.92:6831","nonce":1594153142}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":1594153142},{"type":"v1","addr":"172.21.15.92:6829","nonce":1594153142}]},"public_addr":"172.21.15.92:6825/1594153142","cluster_addr":"172.21.15.92:6827/1594153142","heartbeat_back_addr":"172.21.15.92:6831/1594153142","heartbeat_front_addr":"172.21.15.92:6829/1594153142","state":["exists","up"]},{"osd":11,"uuid":"83cba9fa-b745-4df1-adda-e86837d4b91a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":83,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1622582126},{"type":"v1","addr":"172.21.15.177:6825","nonce":1622582126}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":1622582126},{"type":"v1","addr":"172.21.15.177:6827","nonce":1622582126}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":1622582126},{"type":"v1","addr":"172.21.15.177:6831","nonce":1622582126}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1622582126},{"type":"v1","addr":"172.21.15.177:6829","nonce":1622582126}]},"public_addr":"172.21.15.177:6825/1622582126","cluster_addr":"172.21.15.177:6827/1622582126","heartbeat_back_addr":"172.21.15.177:6831/1622582126","heartbeat_front_addr":"172.21.15.177:6829/1622582126","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:26:24.290365+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:26:49.405988+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:27:13.639196+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:27:45.603746+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:28:11.019126+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:28:35.084163+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:29:08.064626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:29:33.266133+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:29:57.381816+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:30:30.286847+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:30:56.226279+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:31:20.809611+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":7,"to":11}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":8,"to":4}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.11","mappings":[{"from":4,"to":8},{"from":5,"to":9}]},{"pgid":"3.21","mappings":[{"from":5,"to":9}]},{"pgid":"3.23","mappings":[{"from":4,"to":7}]},{"pgid":"3.3a","mappings":[{"from":4,"to":6}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.90:0/2805994468":"2024-04-03T14:34:52.235215+0000","172.21.15.90:0/1431307002":"2024-04-03T14:34:52.235215+0000","172.21.15.90:6801/4266054817":"2024-04-03T14:34:52.235215+0000","172.21.15.90:0/314641601":"2024-04-03T14:24:34.595561+0000","172.21.15.90:0/1400895868":"2024-04-03T14:24:34.595561+0000","172.21.15.90:6801/129436436":"2024-04-03T14:24:34.595561+0000","172.21.15.90:0/1335673697":"2024-04-03T14:34:52.235215+0000","172.21.15.90:6801/2372398598":"2024-04-03T14:24:00.362147+0000","172.21.15.90:0/278569338":"2024-04-03T14:24:34.595561+0000","172.21.15.90:0/2321464594":"2024-04-03T14:24:00.362147+0000","172.21.15.90:6800/4266054817":"2024-04-03T14:34:52.235215+0000","172.21.15.90:6800/2372398598":"2024-04-03T14:24:00.362147+0000","172.21.15.90:0/2962656672":"2024-04-03T14:24:00.362147+0000","172.21.15.90:0/126693610":"2024-04-03T14:24:00.362147+0000","172.21.15.90:0/1757385857":"2024-04-03T14:34:52.235215+0000","172.21.15.90:6800/129436436":"2024-04-03T14:24:34.595561+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T14:34:55.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:55.980+0000 7f11a1200640 1 -- 172.21.15.90:0/2836371880 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f11800a3e30 msgr2=0x7f11800a62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:55.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:55.980+0000 7f11a1200640 1 --2- 172.21.15.90:0/2836371880 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f11800a3e30 0x7f11800a62f0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f118c0095f0 tx=0x7f118c005f90 comp rx=0 tx=0).stop 2024-04-02T14:34:55.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:55.980+0000 7f11a1200640 1 -- 172.21.15.90:0/2836371880 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f119c101a40 msgr2=0x7f119c1981b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:55.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:55.980+0000 7f11a1200640 1 --2- 172.21.15.90:0/2836371880 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f119c101a40 0x7f119c1981b0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f11880349e0 tx=0x7f1188032450 comp rx=0 tx=0).stop 2024-04-02T14:34:55.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:55.980+0000 7f11a1200640 1 -- 172.21.15.90:0/2836371880 shutdown_connections 2024-04-02T14:34:55.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:55.980+0000 7f11a1200640 1 --2- 172.21.15.90:0/2836371880 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f11800a3e30 0x7f11800a62f0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:55.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:55.980+0000 7f11a1200640 1 --2- 172.21.15.90:0/2836371880 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f119c102400 0x7f119c19f1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:55.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:55.980+0000 7f11a1200640 1 --2- 172.21.15.90:0/2836371880 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f119c101a40 0x7f119c1981b0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:55.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:55.980+0000 7f11a1200640 1 --2- 172.21.15.90:0/2836371880 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f119c100850 0x7f119c197c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:55.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:55.980+0000 7f11a1200640 1 -- 172.21.15.90:0/2836371880 >> 172.21.15.90:0/2836371880 conn(0x7f119c0fbde0 msgr2=0x7f119c104d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:34:55.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:55.980+0000 7f11a1200640 1 -- 172.21.15.90:0/2836371880 shutdown_connections 2024-04-02T14:34:55.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:55.980+0000 7f11a1200640 1 -- 172.21.15.90:0/2836371880 wait complete. 2024-04-02T14:34:56.000 INFO:tasks.fwd_scrub.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-02T14:27:16.758622+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-02T14:32:20.016331+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-02T14:32:21.294367+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '88', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-02T14:34:56.000 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-02T14:34:56.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:56.205+0000 7f192f800640 1 -- 172.21.15.90:0/2670497077 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f19281058c0 msgr2=0x7f192810c150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:56.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:56.205+0000 7f192f800640 1 --2- 172.21.15.90:0/2670497077 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f19281058c0 0x7f192810c150 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f1920009be0 tx=0x7f192002bef0 comp rx=0 tx=0).stop 2024-04-02T14:34:56.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:56.205+0000 7f192f800640 1 -- 172.21.15.90:0/2670497077 shutdown_connections 2024-04-02T14:34:56.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:56.205+0000 7f192f800640 1 --2- 172.21.15.90:0/2670497077 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f19281058c0 0x7f192810c150 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:56.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:56.205+0000 7f192f800640 1 --2- 172.21.15.90:0/2670497077 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1928104f00 0x7f1928105380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:56.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:56.205+0000 7f192f800640 1 --2- 172.21.15.90:0/2670497077 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1928103d10 0x7f1928104130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:56.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:56.205+0000 7f192f800640 1 -- 172.21.15.90:0/2670497077 >> 172.21.15.90:0/2670497077 conn(0x7f19280ff2c0 msgr2=0x7f19281016e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:34:56.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:56.205+0000 7f192f800640 1 -- 172.21.15.90:0/2670497077 shutdown_connections 2024-04-02T14:34:56.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:56.205+0000 7f192f800640 1 -- 172.21.15.90:0/2670497077 wait complete. 2024-04-02T14:34:56.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:56.206+0000 7f192f800640 1 Processor -- start 2024-04-02T14:34:56.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:56.206+0000 7f192f800640 1 -- start start 2024-04-02T14:34:56.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:56.206+0000 7f192f800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1928103d10 0x7f1928199f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:56.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:56.206+0000 7f192f800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1928104f00 0x7f192819a490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:56.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:56.206+0000 7f192f800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f19281058c0 0x7f19281a1500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:56.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:56.207+0000 7f192f800640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f192810f970 con 0x7f1928103d10 2024-04-02T14:34:56.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:56.207+0000 7f192f800640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f192810f7f0 con 0x7f1928104f00 2024-04-02T14:34:56.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:56.207+0000 7f192f800640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f192810faf0 con 0x7f19281058c0 2024-04-02T14:34:56.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:56.207+0000 7f192ce00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1928103d10 0x7f1928199f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:56.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:56.207+0000 7f192ce00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1928103d10 0x7f1928199f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:53268/0 (socket says 172.21.15.90:53268) 2024-04-02T14:34:56.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:56.207+0000 7f192ce00640 1 -- 172.21.15.90:0/2716691736 learned_addr learned my addr 172.21.15.90:0/2716691736 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:34:56.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:56.207+0000 7f1927e00640 1 --2- 172.21.15.90:0/2716691736 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1928104f00 0x7f192819a490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:56.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:56.207+0000 7f192d800640 1 --2- 172.21.15.90:0/2716691736 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f19281058c0 0x7f19281a1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:56.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:56.207+0000 7f192ce00640 1 -- 172.21.15.90:0/2716691736 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f19281058c0 msgr2=0x7f19281a1500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:56.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:56.207+0000 7f192ce00640 1 --2- 172.21.15.90:0/2716691736 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f19281058c0 0x7f19281a1500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:56.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:56.207+0000 7f192ce00640 1 -- 172.21.15.90:0/2716691736 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1928104f00 msgr2=0x7f192819a490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:56.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:56.207+0000 7f192ce00640 1 --2- 172.21.15.90:0/2716691736 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1928104f00 0x7f192819a490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:56.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:56.207+0000 7f192ce00640 1 -- 172.21.15.90:0/2716691736 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19281a1b70 con 0x7f1928103d10 2024-04-02T14:34:56.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:56.207+0000 7f1927e00640 1 --2- 172.21.15.90:0/2716691736 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1928104f00 0x7f192819a490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:34:56.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:56.207+0000 7f192ce00640 1 --2- 172.21.15.90:0/2716691736 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1928103d10 0x7f1928199f50 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f191c00e9a0 tx=0x7f191c00ee60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:34:56.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:56.208+0000 7f1925600640 1 -- 172.21.15.90:0/2716691736 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f191c00cd80 con 0x7f1928103d10 2024-04-02T14:34:56.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:56.208+0000 7f1925600640 1 -- 172.21.15.90:0/2716691736 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f191c010910 con 0x7f1928103d10 2024-04-02T14:34:56.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:56.208+0000 7f1925600640 1 -- 172.21.15.90:0/2716691736 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f191c018a30 con 0x7f1928103d10 2024-04-02T14:34:56.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:56.208+0000 7f192f800640 1 -- 172.21.15.90:0/2716691736 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f19281a1e60 con 0x7f1928103d10 2024-04-02T14:34:56.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:56.208+0000 7f192f800640 1 -- 172.21.15.90:0/2716691736 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19281a22c0 con 0x7f1928103d10 2024-04-02T14:34:56.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:56.209+0000 7f192f800640 1 -- 172.21.15.90:0/2716691736 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1928068d40 con 0x7f1928103d10 2024-04-02T14:34:56.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:56.212+0000 7f1925600640 1 -- 172.21.15.90:0/2716691736 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 23) ==== 138817+0+0 (secure 0 0 0) 0x7f191c010ac0 con 0x7f1928103d10 2024-04-02T14:34:56.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:56.213+0000 7f1925600640 1 --2- 172.21.15.90:0/2716691736 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f19100a3db0 0x7f19100a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:56.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:56.213+0000 7f1925600640 1 -- 172.21.15.90:0/2716691736 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f191c0d13f0 con 0x7f1928103d10 2024-04-02T14:34:56.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:56.213+0000 7f1927e00640 1 --2- 172.21.15.90:0/2716691736 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f19100a3db0 0x7f19100a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:56.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:56.214+0000 7f1927e00640 1 --2- 172.21.15.90:0/2716691736 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f19100a3db0 0x7f19100a6270 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f192819b460 tx=0x7f1914005f90 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:34:56.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:56.217+0000 7f1925600640 1 -- 172.21.15.90:0/2716691736 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7f191c026070 con 0x7f1928103d10 2024-04-02T14:34:56.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:56.414+0000 7f192f800640 1 -- 172.21.15.90:0/2716691736 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f192802d430 con 0x7f1928103d10 2024-04-02T14:34:56.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:56.415+0000 7f1925600640 1 -- 172.21.15.90:0/2716691736 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v91) ==== 93+0+10 (secure 0 0 0) 0x7f191c09d8d0 con 0x7f1928103d10 2024-04-02T14:34:56.417 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 1 2024-04-02T14:34:56.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:56.417+0000 7f192f800640 1 -- 172.21.15.90:0/2716691736 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f19100a3db0 msgr2=0x7f19100a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:56.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:56.417+0000 7f192f800640 1 --2- 172.21.15.90:0/2716691736 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f19100a3db0 0x7f19100a6270 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f192819b460 tx=0x7f1914005f90 comp rx=0 tx=0).stop 2024-04-02T14:34:56.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:56.417+0000 7f192f800640 1 -- 172.21.15.90:0/2716691736 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1928103d10 msgr2=0x7f1928199f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:56.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:56.417+0000 7f192f800640 1 --2- 172.21.15.90:0/2716691736 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1928103d10 0x7f1928199f50 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f191c00e9a0 tx=0x7f191c00ee60 comp rx=0 tx=0).stop 2024-04-02T14:34:56.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:56.417+0000 7f192f800640 1 -- 172.21.15.90:0/2716691736 shutdown_connections 2024-04-02T14:34:56.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:56.417+0000 7f192f800640 1 --2- 172.21.15.90:0/2716691736 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f19100a3db0 0x7f19100a6270 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:56.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:56.417+0000 7f192f800640 1 --2- 172.21.15.90:0/2716691736 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f19281058c0 0x7f19281a1500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:56.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:56.417+0000 7f192f800640 1 --2- 172.21.15.90:0/2716691736 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1928104f00 0x7f192819a490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:56.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:56.417+0000 7f192f800640 1 --2- 172.21.15.90:0/2716691736 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1928103d10 0x7f1928199f50 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:56.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:56.417+0000 7f192f800640 1 -- 172.21.15.90:0/2716691736 >> 172.21.15.90:0/2716691736 conn(0x7f19280ff2c0 msgr2=0x7f1928108200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:34:56.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:56.417+0000 7f192f800640 1 -- 172.21.15.90:0/2716691736 shutdown_connections 2024-04-02T14:34:56.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:56.417+0000 7f192f800640 1 -- 172.21.15.90:0/2716691736 wait complete. 2024-04-02T14:34:56.436 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-02T14:34:56.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:56.629+0000 7f3604200640 1 -- 172.21.15.90:0/3834507213 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f35fc1025a0 msgr2=0x7f35fc104990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:56.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:56.629+0000 7f3604200640 1 --2- 172.21.15.90:0/3834507213 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f35fc1025a0 0x7f35fc104990 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f35f0009860 tx=0x7f35f002bb70 comp rx=0 tx=0).stop 2024-04-02T14:34:56.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:56.629+0000 7f3604200640 1 -- 172.21.15.90:0/3834507213 shutdown_connections 2024-04-02T14:34:56.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:56.629+0000 7f3604200640 1 --2- 172.21.15.90:0/3834507213 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f35fc107800 0x7f35fc109c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:56.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:56.629+0000 7f3604200640 1 --2- 172.21.15.90:0/3834507213 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f35fc104ed0 0x7f35fc1072c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:56.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:56.629+0000 7f3604200640 1 --2- 172.21.15.90:0/3834507213 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f35fc1025a0 0x7f35fc104990 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:56.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:56.629+0000 7f3604200640 1 -- 172.21.15.90:0/3834507213 >> 172.21.15.90:0/3834507213 conn(0x7f35fc0fbf90 msgr2=0x7f35fc0fe3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:34:56.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:56.629+0000 7f3604200640 1 -- 172.21.15.90:0/3834507213 shutdown_connections 2024-04-02T14:34:56.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:56.629+0000 7f3604200640 1 -- 172.21.15.90:0/3834507213 wait complete. 2024-04-02T14:34:56.634 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:56.630+0000 7f3604200640 1 Processor -- start 2024-04-02T14:34:56.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:56.631+0000 7f3604200640 1 -- start start 2024-04-02T14:34:56.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:56.631+0000 7f3604200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f35fc1025a0 0x7f35fc101f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:56.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:56.631+0000 7f3604200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f35fc104ed0 0x7f35fc100550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:56.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:56.631+0000 7f3604200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f35fc107800 0x7f35fc100a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:56.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:56.631+0000 7f3604200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f35fc10d7a0 con 0x7f35fc1025a0 2024-04-02T14:34:56.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:56.631+0000 7f3604200640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f35fc10d620 con 0x7f35fc107800 2024-04-02T14:34:56.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:56.631+0000 7f3604200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f35fc10d920 con 0x7f35fc104ed0 2024-04-02T14:34:56.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:56.631+0000 7f3601800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f35fc1025a0 0x7f35fc101f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:56.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:56.631+0000 7f3601800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f35fc1025a0 0x7f35fc101f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:53284/0 (socket says 172.21.15.90:53284) 2024-04-02T14:34:56.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:56.631+0000 7f3601800640 1 -- 172.21.15.90:0/2251333281 learned_addr learned my addr 172.21.15.90:0/2251333281 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:34:56.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:56.631+0000 7f3600e00640 1 --2- 172.21.15.90:0/2251333281 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f35fc104ed0 0x7f35fc100550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:56.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:56.631+0000 7f3602200640 1 --2- 172.21.15.90:0/2251333281 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f35fc107800 0x7f35fc100a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:56.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:56.631+0000 7f3601800640 1 -- 172.21.15.90:0/2251333281 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f35fc104ed0 msgr2=0x7f35fc100550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:56.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:56.631+0000 7f3601800640 1 --2- 172.21.15.90:0/2251333281 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f35fc104ed0 0x7f35fc100550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:56.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:56.631+0000 7f3601800640 1 -- 172.21.15.90:0/2251333281 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f35fc107800 msgr2=0x7f35fc100a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:56.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:56.631+0000 7f3601800640 1 --2- 172.21.15.90:0/2251333281 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f35fc107800 0x7f35fc100a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:56.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:56.632+0000 7f3601800640 1 -- 172.21.15.90:0/2251333281 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35fc101320 con 0x7f35fc1025a0 2024-04-02T14:34:56.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:56.632+0000 7f3601800640 1 --2- 172.21.15.90:0/2251333281 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f35fc1025a0 0x7f35fc101f10 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f35f0034b10 tx=0x7f35f0032500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:34:56.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:56.632+0000 7f35fa000640 1 -- 172.21.15.90:0/2251333281 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35f0041020 con 0x7f35fc1025a0 2024-04-02T14:34:56.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:56.632+0000 7f35fa000640 1 -- 172.21.15.90:0/2251333281 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f35f002c9f0 con 0x7f35fc1025a0 2024-04-02T14:34:56.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:56.632+0000 7f35fa000640 1 -- 172.21.15.90:0/2251333281 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35f0039040 con 0x7f35fc1025a0 2024-04-02T14:34:56.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:56.632+0000 7f3604200640 1 -- 172.21.15.90:0/2251333281 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35fc1a1f90 con 0x7f35fc1025a0 2024-04-02T14:34:56.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:56.632+0000 7f3604200640 1 -- 172.21.15.90:0/2251333281 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35fc1a2340 con 0x7f35fc1025a0 2024-04-02T14:34:56.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:56.633+0000 7f3604200640 1 -- 172.21.15.90:0/2251333281 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f35fc0fda80 con 0x7f35fc1025a0 2024-04-02T14:34:56.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:56.637+0000 7f35fa000640 1 -- 172.21.15.90:0/2251333281 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 23) ==== 138817+0+0 (secure 0 0 0) 0x7f35f0044020 con 0x7f35fc1025a0 2024-04-02T14:34:56.640 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:56.638+0000 7f35fa000640 1 --2- 172.21.15.90:0/2251333281 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f35e40a3db0 0x7f35e40a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:56.640 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:56.638+0000 7f35fa000640 1 -- 172.21.15.90:0/2251333281 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f35f00f2620 con 0x7f35fc1025a0 2024-04-02T14:34:56.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:56.639+0000 7f3600e00640 1 --2- 172.21.15.90:0/2251333281 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f35e40a3db0 0x7f35e40a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:56.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:56.640+0000 7f3600e00640 1 --2- 172.21.15.90:0/2251333281 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f35e40a3db0 0x7f35e40a6270 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f35fc101c80 tx=0x7f35e8009260 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:34:56.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:56.641+0000 7f35fa000640 1 -- 172.21.15.90:0/2251333281 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7f35f004b020 con 0x7f35fc1025a0 2024-04-02T14:34:56.688 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:34:56 smithi090 ceph-mon[31674]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' 2024-04-02T14:34:56.688 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:34:56 smithi090 ceph-mon[31674]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' 2024-04-02T14:34:56.689 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:34:56 smithi090 ceph-mon[31674]: from='client.? 172.21.15.90:0/183147717' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-02T14:34:56.689 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:34:56 smithi090 ceph-mon[31674]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-02T14:34:56.689 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:34:56 smithi090 ceph-mon[31674]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-02T14:34:56.689 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:34:56 smithi090 ceph-mon[31674]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-02T14:34:56.689 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:34:56 smithi090 ceph-mon[31674]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-02T14:34:56.689 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:34:56 smithi090 ceph-mon[31674]: Adjusting osd_memory_target on smithi092 to 174.1M 2024-04-02T14:34:56.689 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:34:56 smithi090 ceph-mon[31674]: Unable to set osd_memory_target on smithi092 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-02T14:34:56.689 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:34:56 smithi090 ceph-mon[31674]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' 2024-04-02T14:34:56.689 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:34:56 smithi090 ceph-mon[31674]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' 2024-04-02T14:34:56.689 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:34:56 smithi090 ceph-mon[31674]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-02T14:34:56.689 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:34:56 smithi090 ceph-mon[31674]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-02T14:34:56.689 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:34:56 smithi090 ceph-mon[31674]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-02T14:34:56.689 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:34:56 smithi090 ceph-mon[31674]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-02T14:34:56.689 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:34:56 smithi090 ceph-mon[31674]: Adjusting osd_memory_target on smithi177 to 174.1M 2024-04-02T14:34:56.689 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:34:56 smithi090 ceph-mon[31674]: Unable to set osd_memory_target on smithi177 to 182576128: error parsing value: Value '182576128' is below minimum 939524096 2024-04-02T14:34:56.690 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:34:56 smithi090 ceph-mon[31674]: from='client.? 172.21.15.90:0/2836371880' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-02T14:34:56.690 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:34:56 smithi090 ceph-mon[31674]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' 2024-04-02T14:34:56.690 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:34:56 smithi090 ceph-mon[31674]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' 2024-04-02T14:34:56.690 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:34:56 smithi090 ceph-mon[31674]: from='client.? 172.21.15.90:0/2716691736' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-02T14:34:56.840 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:56.838+0000 7f3604200640 1 -- 172.21.15.90:0/2251333281 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f35fc02d500 con 0x7f35fc1025a0 2024-04-02T14:34:56.841 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:56.839+0000 7f35fa000640 1 -- 172.21.15.90:0/2251333281 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v91) ==== 104+0+11 (secure 0 0 0) 0x7f35f00bec10 con 0x7f35fc1025a0 2024-04-02T14:34:56.841 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-04-02T14:34:56.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:56.842+0000 7f3604200640 1 -- 172.21.15.90:0/2251333281 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f35e40a3db0 msgr2=0x7f35e40a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:56.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:56.842+0000 7f3604200640 1 --2- 172.21.15.90:0/2251333281 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f35e40a3db0 0x7f35e40a6270 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f35fc101c80 tx=0x7f35e8009260 comp rx=0 tx=0).stop 2024-04-02T14:34:56.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:56.842+0000 7f3604200640 1 -- 172.21.15.90:0/2251333281 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f35fc1025a0 msgr2=0x7f35fc101f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:56.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:56.842+0000 7f3604200640 1 --2- 172.21.15.90:0/2251333281 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f35fc1025a0 0x7f35fc101f10 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f35f0034b10 tx=0x7f35f0032500 comp rx=0 tx=0).stop 2024-04-02T14:34:56.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:56.842+0000 7f3604200640 1 -- 172.21.15.90:0/2251333281 shutdown_connections 2024-04-02T14:34:56.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:56.842+0000 7f3604200640 1 --2- 172.21.15.90:0/2251333281 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f35e40a3db0 0x7f35e40a6270 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:56.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:56.842+0000 7f3604200640 1 --2- 172.21.15.90:0/2251333281 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f35fc107800 0x7f35fc100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:56.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:56.842+0000 7f3604200640 1 --2- 172.21.15.90:0/2251333281 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f35fc104ed0 0x7f35fc100550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:56.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:56.842+0000 7f3604200640 1 --2- 172.21.15.90:0/2251333281 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f35fc1025a0 0x7f35fc101f10 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:56.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:56.842+0000 7f3604200640 1 -- 172.21.15.90:0/2251333281 >> 172.21.15.90:0/2251333281 conn(0x7f35fc0fbf90 msgr2=0x7f35fc1030e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:34:56.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:56.842+0000 7f3604200640 1 -- 172.21.15.90:0/2251333281 shutdown_connections 2024-04-02T14:34:56.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:56.842+0000 7f3604200640 1 -- 172.21.15.90:0/2251333281 wait complete. 2024-04-02T14:34:56.861 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:34:56 smithi092 ceph-mon[36497]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' 2024-04-02T14:34:56.861 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:34:56 smithi092 ceph-mon[36497]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' 2024-04-02T14:34:56.861 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:34:56 smithi092 ceph-mon[36497]: from='client.? 172.21.15.90:0/183147717' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-02T14:34:56.861 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:34:56 smithi092 ceph-mon[36497]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-02T14:34:56.861 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:34:56 smithi092 ceph-mon[36497]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-02T14:34:56.861 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:34:56 smithi092 ceph-mon[36497]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-02T14:34:56.861 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:34:56 smithi092 ceph-mon[36497]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-02T14:34:56.861 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:34:56 smithi092 ceph-mon[36497]: Adjusting osd_memory_target on smithi092 to 174.1M 2024-04-02T14:34:56.861 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:34:56 smithi092 ceph-mon[36497]: Unable to set osd_memory_target on smithi092 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-02T14:34:56.862 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:34:56 smithi092 ceph-mon[36497]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' 2024-04-02T14:34:56.862 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:34:56 smithi092 ceph-mon[36497]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' 2024-04-02T14:34:56.862 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:34:56 smithi092 ceph-mon[36497]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-02T14:34:56.862 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:34:56 smithi092 ceph-mon[36497]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-02T14:34:56.862 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:34:56 smithi092 ceph-mon[36497]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-02T14:34:56.862 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:34:56 smithi092 ceph-mon[36497]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-02T14:34:56.862 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:34:56 smithi092 ceph-mon[36497]: Adjusting osd_memory_target on smithi177 to 174.1M 2024-04-02T14:34:56.862 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:34:56 smithi092 ceph-mon[36497]: Unable to set osd_memory_target on smithi177 to 182576128: error parsing value: Value '182576128' is below minimum 939524096 2024-04-02T14:34:56.862 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:34:56 smithi092 ceph-mon[36497]: from='client.? 172.21.15.90:0/2836371880' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-02T14:34:56.862 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:34:56 smithi092 ceph-mon[36497]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' 2024-04-02T14:34:56.862 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:34:56 smithi092 ceph-mon[36497]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' 2024-04-02T14:34:56.862 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:34:56 smithi092 ceph-mon[36497]: from='client.? 172.21.15.90:0/2716691736' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-02T14:34:56.863 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-02T14:34:56.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:34:56 smithi177 ceph-mon[36431]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' 2024-04-02T14:34:56.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:34:56 smithi177 ceph-mon[36431]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' 2024-04-02T14:34:56.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:34:56 smithi177 ceph-mon[36431]: from='client.? 172.21.15.90:0/183147717' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-02T14:34:56.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:34:56 smithi177 ceph-mon[36431]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-02T14:34:56.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:34:56 smithi177 ceph-mon[36431]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-02T14:34:56.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:34:56 smithi177 ceph-mon[36431]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-02T14:34:56.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:34:56 smithi177 ceph-mon[36431]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-02T14:34:56.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:34:56 smithi177 ceph-mon[36431]: Adjusting osd_memory_target on smithi092 to 174.1M 2024-04-02T14:34:56.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:34:56 smithi177 ceph-mon[36431]: Unable to set osd_memory_target on smithi092 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-02T14:34:56.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:34:56 smithi177 ceph-mon[36431]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' 2024-04-02T14:34:56.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:34:56 smithi177 ceph-mon[36431]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' 2024-04-02T14:34:56.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:34:56 smithi177 ceph-mon[36431]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-02T14:34:56.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:34:56 smithi177 ceph-mon[36431]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-02T14:34:56.938 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:34:56 smithi177 ceph-mon[36431]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-02T14:34:56.938 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:34:56 smithi177 ceph-mon[36431]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-02T14:34:56.938 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:34:56 smithi177 ceph-mon[36431]: Adjusting osd_memory_target on smithi177 to 174.1M 2024-04-02T14:34:56.938 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:34:56 smithi177 ceph-mon[36431]: Unable to set osd_memory_target on smithi177 to 182576128: error parsing value: Value '182576128' is below minimum 939524096 2024-04-02T14:34:56.938 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:34:56 smithi177 ceph-mon[36431]: from='client.? 172.21.15.90:0/2836371880' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-02T14:34:56.938 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:34:56 smithi177 ceph-mon[36431]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' 2024-04-02T14:34:56.938 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:34:56 smithi177 ceph-mon[36431]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' 2024-04-02T14:34:56.938 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:34:56 smithi177 ceph-mon[36431]: from='client.? 172.21.15.90:0/2716691736' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-02T14:34:57.052 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:57.050+0000 7fb4fb200640 1 -- 172.21.15.90:0/2999485845 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb4f41098c0 msgr2=0x7fb4f410bd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:57.052 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:57.050+0000 7fb4fb200640 1 --2- 172.21.15.90:0/2999485845 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb4f41098c0 0x7fb4f410bd50 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fb4f0009be0 tx=0x7fb4f002bef0 comp rx=0 tx=0).stop 2024-04-02T14:34:57.052 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:57.050+0000 7fb4fb200640 1 -- 172.21.15.90:0/2999485845 shutdown_connections 2024-04-02T14:34:57.052 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:57.050+0000 7fb4fb200640 1 --2- 172.21.15.90:0/2999485845 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb4f41098c0 0x7fb4f410bd50 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:57.052 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:57.050+0000 7fb4fb200640 1 --2- 172.21.15.90:0/2999485845 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb4f4074560 0x7fb4f40749e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:57.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:57.050+0000 7fb4fb200640 1 --2- 172.21.15.90:0/2999485845 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb4f4075b40 0x7fb4f4073f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:57.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:57.050+0000 7fb4fb200640 1 -- 172.21.15.90:0/2999485845 >> 172.21.15.90:0/2999485845 conn(0x7fb4f40fcf70 msgr2=0x7fb4f40ff3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:34:57.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:57.050+0000 7fb4fb200640 1 -- 172.21.15.90:0/2999485845 shutdown_connections 2024-04-02T14:34:57.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:57.050+0000 7fb4fb200640 1 -- 172.21.15.90:0/2999485845 wait complete. 2024-04-02T14:34:57.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:57.051+0000 7fb4fb200640 1 Processor -- start 2024-04-02T14:34:57.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:57.051+0000 7fb4fb200640 1 -- start start 2024-04-02T14:34:57.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:57.051+0000 7fb4fb200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb4f4074560 0x7fb4f4199ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:57.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:57.051+0000 7fb4fb200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb4f4075b40 0x7fb4f419a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:57.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:57.051+0000 7fb4fb200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb4f41098c0 0x7fb4f41a1450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:57.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:57.051+0000 7fb4fb200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fb4f410f890 con 0x7fb4f4075b40 2024-04-02T14:34:57.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:57.051+0000 7fb4fb200640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fb4f410f710 con 0x7fb4f41098c0 2024-04-02T14:34:57.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:57.051+0000 7fb4fb200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fb4f410fa10 con 0x7fb4f4074560 2024-04-02T14:34:57.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:57.052+0000 7fb4f9400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb4f4075b40 0x7fb4f419a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:57.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:57.052+0000 7fb4fa800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb4f41098c0 0x7fb4f41a1450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:57.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:57.052+0000 7fb4f9400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb4f4075b40 0x7fb4f419a3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:53300/0 (socket says 172.21.15.90:53300) 2024-04-02T14:34:57.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:57.052+0000 7fb4fa800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb4f41098c0 0x7fb4f41a1450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:58876/0 (socket says 172.21.15.90:58876) 2024-04-02T14:34:57.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:57.052+0000 7fb4fa800640 1 -- 172.21.15.90:0/3762803642 learned_addr learned my addr 172.21.15.90:0/3762803642 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:34:57.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:57.052+0000 7fb4f9e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb4f4074560 0x7fb4f4199ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:57.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:57.052+0000 7fb4fa800640 1 -- 172.21.15.90:0/3762803642 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb4f4074560 msgr2=0x7fb4f4199ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:57.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:57.052+0000 7fb4fa800640 1 --2- 172.21.15.90:0/3762803642 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb4f4074560 0x7fb4f4199ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:57.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:57.052+0000 7fb4fa800640 1 -- 172.21.15.90:0/3762803642 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb4f4075b40 msgr2=0x7fb4f419a3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:57.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:57.052+0000 7fb4fa800640 1 --2- 172.21.15.90:0/3762803642 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb4f4075b40 0x7fb4f419a3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:57.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:57.052+0000 7fb4fa800640 1 -- 172.21.15.90:0/3762803642 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4f41a1ac0 con 0x7fb4f41098c0 2024-04-02T14:34:57.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:57.052+0000 7fb4f9e00640 1 --2- 172.21.15.90:0/3762803642 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb4f4074560 0x7fb4f4199ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:34:57.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:57.052+0000 7fb4f9400640 1 --2- 172.21.15.90:0/3762803642 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb4f4075b40 0x7fb4f419a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:34:57.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:57.052+0000 7fb4fa800640 1 --2- 172.21.15.90:0/3762803642 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb4f41098c0 0x7fb4f41a1450 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fb4f0036040 tx=0x7fb4f00337b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:34:57.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:57.053+0000 7fb4eea00640 1 -- 172.21.15.90:0/3762803642 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4f0043020 con 0x7fb4f41098c0 2024-04-02T14:34:57.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:57.053+0000 7fb4fb200640 1 -- 172.21.15.90:0/3762803642 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb4f41a1d50 con 0x7fb4f41098c0 2024-04-02T14:34:57.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:57.053+0000 7fb4eea00640 1 -- 172.21.15.90:0/3762803642 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb4f002ca60 con 0x7fb4f41098c0 2024-04-02T14:34:57.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:57.053+0000 7fb4eea00640 1 -- 172.21.15.90:0/3762803642 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4f0034690 con 0x7fb4f41098c0 2024-04-02T14:34:57.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:57.053+0000 7fb4fb200640 1 -- 172.21.15.90:0/3762803642 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4f41a2210 con 0x7fb4f41098c0 2024-04-02T14:34:57.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:57.054+0000 7fb4fb200640 1 -- 172.21.15.90:0/3762803642 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb4f41085f0 con 0x7fb4f41098c0 2024-04-02T14:34:57.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:57.056+0000 7fb4eea00640 1 -- 172.21.15.90:0/3762803642 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 23) ==== 138817+0+0 (secure 0 0 0) 0x7fb4f0046020 con 0x7fb4f41098c0 2024-04-02T14:34:57.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:57.057+0000 7fb4eea00640 1 --2- 172.21.15.90:0/3762803642 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fb4dc0a3db0 0x7fb4dc0a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:57.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:57.057+0000 7fb4eea00640 1 -- 172.21.15.90:0/3762803642 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7fb4f00f2e70 con 0x7fb4f41098c0 2024-04-02T14:34:57.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:57.057+0000 7fb4f9e00640 1 --2- 172.21.15.90:0/3762803642 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fb4dc0a3db0 0x7fb4dc0a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:57.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:57.058+0000 7fb4f9e00640 1 --2- 172.21.15.90:0/3762803642 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fb4dc0a3db0 0x7fb4dc0a6270 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fb4e8009fc0 tx=0x7fb4e8005f90 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:34:57.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:57.061+0000 7fb4eea00640 1 -- 172.21.15.90:0/3762803642 <== mon.2 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7fb4f004d020 con 0x7fb4f41098c0 2024-04-02T14:34:57.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:57.089+0000 7fb4eea00640 1 -- 172.21.15.90:0/3762803642 <== mon.2 v2:172.21.15.92:3300/0 7 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7fb4f00bb430 con 0x7fb4f41098c0 2024-04-02T14:34:57.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:57.288+0000 7fb4fb200640 1 -- 172.21.15.90:0/3762803642 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fb4f4062700 con 0x7fb4f41098c0 2024-04-02T14:34:57.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:57.289+0000 7fb4eea00640 1 -- 172.21.15.90:0/3762803642 <== mon.2 v2:172.21.15.92:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v91) ==== 100+0+11 (secure 0 0 0) 0x7fb4f00bf750 con 0x7fb4f41098c0 2024-04-02T14:34:57.291 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 64 2024-04-02T14:34:57.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:57.292+0000 7fb4fb200640 1 -- 172.21.15.90:0/3762803642 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fb4dc0a3db0 msgr2=0x7fb4dc0a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:57.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:57.292+0000 7fb4fb200640 1 --2- 172.21.15.90:0/3762803642 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fb4dc0a3db0 0x7fb4dc0a6270 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fb4e8009fc0 tx=0x7fb4e8005f90 comp rx=0 tx=0).stop 2024-04-02T14:34:57.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:57.292+0000 7fb4fb200640 1 -- 172.21.15.90:0/3762803642 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb4f41098c0 msgr2=0x7fb4f41a1450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:57.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:57.292+0000 7fb4fb200640 1 --2- 172.21.15.90:0/3762803642 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb4f41098c0 0x7fb4f41a1450 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fb4f0036040 tx=0x7fb4f00337b0 comp rx=0 tx=0).stop 2024-04-02T14:34:57.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:57.292+0000 7fb4fb200640 1 -- 172.21.15.90:0/3762803642 shutdown_connections 2024-04-02T14:34:57.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:57.292+0000 7fb4fb200640 1 --2- 172.21.15.90:0/3762803642 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fb4dc0a3db0 0x7fb4dc0a6270 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:57.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:57.292+0000 7fb4fb200640 1 --2- 172.21.15.90:0/3762803642 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb4f41098c0 0x7fb4f41a1450 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:57.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:57.292+0000 7fb4fb200640 1 --2- 172.21.15.90:0/3762803642 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb4f4075b40 0x7fb4f419a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:57.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:57.292+0000 7fb4fb200640 1 --2- 172.21.15.90:0/3762803642 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb4f4074560 0x7fb4f4199ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:57.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:57.292+0000 7fb4fb200640 1 -- 172.21.15.90:0/3762803642 >> 172.21.15.90:0/3762803642 conn(0x7fb4f40fcf70 msgr2=0x7fb4f41057c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:34:57.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:57.292+0000 7fb4fb200640 1 -- 172.21.15.90:0/3762803642 shutdown_connections 2024-04-02T14:34:57.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:57.292+0000 7fb4fb200640 1 -- 172.21.15.90:0/3762803642 wait complete. 2024-04-02T14:34:57.315 INFO:tasks.fwd_scrub:Wait for all MDSs to reach steady state... 2024-04-02T14:34:57.315 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T14:34:57.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:57.492+0000 7fb8b6c00640 1 -- 172.21.15.90:0/483822799 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb8b0104d50 msgr2=0x7fb8b01051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:57.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:57.492+0000 7fb8b6c00640 1 --2- 172.21.15.90:0/483822799 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb8b0104d50 0x7fb8b01051d0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fb8a0009900 tx=0x7fb8a002bc30 comp rx=0 tx=0).stop 2024-04-02T14:34:57.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:57.493+0000 7fb8b6c00640 1 -- 172.21.15.90:0/483822799 shutdown_connections 2024-04-02T14:34:57.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:57.493+0000 7fb8b6c00640 1 --2- 172.21.15.90:0/483822799 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb8b0105710 0x7fb8b010bfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:57.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:57.493+0000 7fb8b6c00640 1 --2- 172.21.15.90:0/483822799 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb8b0104d50 0x7fb8b01051d0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:57.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:57.493+0000 7fb8b6c00640 1 --2- 172.21.15.90:0/483822799 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8b0103b60 0x7fb8b0103f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:57.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:57.493+0000 7fb8b6c00640 1 -- 172.21.15.90:0/483822799 >> 172.21.15.90:0/483822799 conn(0x7fb8b00ff0d0 msgr2=0x7fb8b0101510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:34:57.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:57.493+0000 7fb8b6c00640 1 -- 172.21.15.90:0/483822799 shutdown_connections 2024-04-02T14:34:57.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:57.493+0000 7fb8b6c00640 1 -- 172.21.15.90:0/483822799 wait complete. 2024-04-02T14:34:57.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:57.494+0000 7fb8b6c00640 1 Processor -- start 2024-04-02T14:34:57.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:57.494+0000 7fb8b6c00640 1 -- start start 2024-04-02T14:34:57.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:57.494+0000 7fb8b6c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8b0103b60 0x7fb8b0199e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:57.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:57.494+0000 7fb8b6c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb8b0104d50 0x7fb8b019a380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:57.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:57.494+0000 7fb8b6c00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb8b0105710 0x7fb8b01a13f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:57.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:57.494+0000 7fb8b6c00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fb8b010f830 con 0x7fb8b0104d50 2024-04-02T14:34:57.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:57.494+0000 7fb8b6c00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fb8b010f6b0 con 0x7fb8b0105710 2024-04-02T14:34:57.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:57.494+0000 7fb8b6c00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fb8b010f9b0 con 0x7fb8b0103b60 2024-04-02T14:34:57.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:57.494+0000 7fb8b4e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb8b0104d50 0x7fb8b019a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:57.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:57.494+0000 7fb8b4e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb8b0104d50 0x7fb8b019a380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:53322/0 (socket says 172.21.15.90:53322) 2024-04-02T14:34:57.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:57.494+0000 7fb8b4e00640 1 -- 172.21.15.90:0/2936994380 learned_addr learned my addr 172.21.15.90:0/2936994380 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:34:57.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:57.494+0000 7fb8b5800640 1 --2- 172.21.15.90:0/2936994380 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8b0103b60 0x7fb8b0199e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:57.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:57.494+0000 7fb8b6200640 1 --2- 172.21.15.90:0/2936994380 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb8b0105710 0x7fb8b01a13f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:57.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:57.495+0000 7fb8b4e00640 1 -- 172.21.15.90:0/2936994380 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8b0103b60 msgr2=0x7fb8b0199e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:57.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:57.495+0000 7fb8b4e00640 1 --2- 172.21.15.90:0/2936994380 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8b0103b60 0x7fb8b0199e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:57.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:57.495+0000 7fb8b4e00640 1 -- 172.21.15.90:0/2936994380 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb8b0105710 msgr2=0x7fb8b01a13f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:57.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:57.495+0000 7fb8b4e00640 1 --2- 172.21.15.90:0/2936994380 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb8b0105710 0x7fb8b01a13f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:57.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:57.495+0000 7fb8b4e00640 1 -- 172.21.15.90:0/2936994380 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8b01a1a60 con 0x7fb8b0104d50 2024-04-02T14:34:57.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:57.495+0000 7fb8b4e00640 1 --2- 172.21.15.90:0/2936994380 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb8b0104d50 0x7fb8b019a380 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fb8a0000c00 tx=0x7fb8a002c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:34:57.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:57.495+0000 7fb8ae000640 1 -- 172.21.15.90:0/2936994380 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8a0043020 con 0x7fb8b0104d50 2024-04-02T14:34:57.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:57.495+0000 7fb8b6c00640 1 -- 172.21.15.90:0/2936994380 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb8b01a1cf0 con 0x7fb8b0104d50 2024-04-02T14:34:57.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:57.495+0000 7fb8ae000640 1 -- 172.21.15.90:0/2936994380 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb8a00054e0 con 0x7fb8b0104d50 2024-04-02T14:34:57.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:57.495+0000 7fb8ae000640 1 -- 172.21.15.90:0/2936994380 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8a0003c40 con 0x7fb8b0104d50 2024-04-02T14:34:57.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:57.495+0000 7fb8b6c00640 1 -- 172.21.15.90:0/2936994380 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8b01a21e0 con 0x7fb8b0104d50 2024-04-02T14:34:57.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:57.496+0000 7fb8b6c00640 1 -- 172.21.15.90:0/2936994380 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb8b010ae80 con 0x7fb8b0104d50 2024-04-02T14:34:57.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:57.500+0000 7fb8ae000640 1 -- 172.21.15.90:0/2936994380 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7fb8a0046020 con 0x7fb8b0104d50 2024-04-02T14:34:57.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:57.501+0000 7fb8ae000640 1 --2- 172.21.15.90:0/2936994380 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fb8980a3e40 0x7fb8980a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:57.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:57.501+0000 7fb8ae000640 1 -- 172.21.15.90:0/2936994380 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7fb8a00f2a00 con 0x7fb8b0104d50 2024-04-02T14:34:57.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:57.502+0000 7fb8b5800640 1 --2- 172.21.15.90:0/2936994380 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fb8980a3e40 0x7fb8980a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:57.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:57.502+0000 7fb8b5800640 1 --2- 172.21.15.90:0/2936994380 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fb8980a3e40 0x7fb8980a6300 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fb8b0104bb0 tx=0x7fb8a4005f90 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:34:57.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:57.506+0000 7fb8ae000640 1 -- 172.21.15.90:0/2936994380 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7fb8a004d020 con 0x7fb8b0104d50 2024-04-02T14:34:57.688 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:34:57 smithi090 ceph-mon[31674]: pgmap v5: 97 pgs: 97 active+clean; 1.1 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-02T14:34:57.688 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:34:57 smithi090 ceph-mon[31674]: from='client.? 172.21.15.90:0/2251333281' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-02T14:34:57.689 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:34:57 smithi090 ceph-mon[31674]: mgrmap e24: x(active, since 4s), standbys: z, y 2024-04-02T14:34:57.689 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:34:57 smithi090 ceph-mon[31674]: from='client.? 172.21.15.90:0/3762803642' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-02T14:34:57.745 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:57.743+0000 7fb8b6c00640 1 -- 172.21.15.90:0/2936994380 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb8b002d370 con 0x7fb8b0104d50 2024-04-02T14:34:57.746 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:57.744+0000 7fb8ae000640 1 -- 172.21.15.90:0/2936994380 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11927 (secure 0 0 0) 0x7fb8a00bef10 con 0x7fb8b0104d50 2024-04-02T14:34:57.746 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-02T14:34:57.746 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14412,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6837/2852160309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6836","nonce":2852160309},{"type":"v1","addr":"172.21.15.90:6837","nonce":2852160309}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14478,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6841/1919376858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6840","nonce":1919376858},{"type":"v1","addr":"172.21.15.90:6841","nonce":1919376858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-02T14:32:22.319083+0000","modified":"2024-04-02T14:34:52.266152+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24590,"mds_1":24560,"mds_2":24547,"mds_3":24539,"mds_4":24523},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24454":{"gid":24454,"name":"l","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6833/1564319431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":1564319431},{"type":"v1","addr":"172.21.15.177:6833","nonce":1564319431}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.92:6833/1570337372","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":1570337372},{"type":"v1","addr":"172.21.15.92:6833","nonce":1570337372}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"j","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.90:6835/1277124891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6834","nonce":1277124891},{"type":"v1","addr":"172.21.15.90:6835","nonce":1277124891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.92:6835/860721164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":860721164},{"type":"v1","addr":"172.21.15.92:6835","nonce":860721164}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24511":{"gid":24511,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6835/3177244988","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":3177244988},{"type":"v1","addr":"172.21.15.177:6835","nonce":3177244988}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"c","rank":4,"incarnation":25,"state":"up:active","state_seq":9,"addr":"172.21.15.177:6837/4223414858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":4223414858},{"type":"v1","addr":"172.21.15.177:6837","nonce":4223414858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24539":{"gid":24539,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":8,"addr":"172.21.15.92:6837/1270635939","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":1270635939},{"type":"v1","addr":"172.21.15.92:6837","nonce":1270635939}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24547":{"gid":24547,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":7,"addr":"172.21.15.177:6839/664895663","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":664895663},{"type":"v1","addr":"172.21.15.177:6839","nonce":664895663}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"g","rank":1,"incarnation":19,"state":"up:active","state_seq":5,"addr":"172.21.15.90:6839/1498915870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6838","nonce":1498915870},{"type":"v1","addr":"172.21.15.90:6839","nonce":1498915870}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24590":{"gid":24590,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.92:6839/1054888219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":1054888219},{"type":"v1","addr":"172.21.15.92:6839","nonce":1054888219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24590,"qdb_cluster":[24547,24590,24523,24539,24560]},"id":1}]} 2024-04-02T14:34:57.746 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 27 2024-04-02T14:34:57.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:57.746+0000 7fb8b6c00640 1 -- 172.21.15.90:0/2936994380 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fb8980a3e40 msgr2=0x7fb8980a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:57.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:57.746+0000 7fb8b6c00640 1 --2- 172.21.15.90:0/2936994380 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fb8980a3e40 0x7fb8980a6300 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fb8b0104bb0 tx=0x7fb8a4005f90 comp rx=0 tx=0).stop 2024-04-02T14:34:57.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:57.746+0000 7fb8b6c00640 1 -- 172.21.15.90:0/2936994380 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb8b0104d50 msgr2=0x7fb8b019a380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:57.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:57.746+0000 7fb8b6c00640 1 --2- 172.21.15.90:0/2936994380 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb8b0104d50 0x7fb8b019a380 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fb8a0000c00 tx=0x7fb8a002c650 comp rx=0 tx=0).stop 2024-04-02T14:34:57.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:57.747+0000 7fb8b6c00640 1 -- 172.21.15.90:0/2936994380 shutdown_connections 2024-04-02T14:34:57.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:57.747+0000 7fb8b6c00640 1 --2- 172.21.15.90:0/2936994380 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fb8980a3e40 0x7fb8980a6300 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:57.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:57.747+0000 7fb8b6c00640 1 --2- 172.21.15.90:0/2936994380 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb8b0105710 0x7fb8b01a13f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:57.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:57.747+0000 7fb8b6c00640 1 --2- 172.21.15.90:0/2936994380 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb8b0104d50 0x7fb8b019a380 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:57.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:57.747+0000 7fb8b6c00640 1 --2- 172.21.15.90:0/2936994380 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8b0103b60 0x7fb8b0199e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:57.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:57.747+0000 7fb8b6c00640 1 -- 172.21.15.90:0/2936994380 >> 172.21.15.90:0/2936994380 conn(0x7fb8b00ff0d0 msgr2=0x7fb8b0108050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:34:57.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:57.747+0000 7fb8b6c00640 1 -- 172.21.15.90:0/2936994380 shutdown_connections 2024-04-02T14:34:57.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:57.747+0000 7fb8b6c00640 1 -- 172.21.15.90:0/2936994380 wait complete. 2024-04-02T14:34:57.768 INFO:tasks.fwd_scrub:Ready to start scrub thrashing 2024-04-02T14:34:57.768 INFO:tasks.fwd_scrub.ceph_manager:waiting for clean 2024-04-02T14:34:57.768 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-02T14:34:57.861 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:34:57 smithi092 ceph-mon[36497]: pgmap v5: 97 pgs: 97 active+clean; 1.1 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-02T14:34:57.861 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:34:57 smithi092 ceph-mon[36497]: from='client.? 172.21.15.90:0/2251333281' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-02T14:34:57.861 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:34:57 smithi092 ceph-mon[36497]: mgrmap e24: x(active, since 4s), standbys: z, y 2024-04-02T14:34:57.861 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:34:57 smithi092 ceph-mon[36497]: from='client.? 172.21.15.90:0/3762803642' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-02T14:34:57.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:34:57 smithi177 ceph-mon[36431]: pgmap v5: 97 pgs: 97 active+clean; 1.1 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-02T14:34:57.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:34:57 smithi177 ceph-mon[36431]: from='client.? 172.21.15.90:0/2251333281' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-02T14:34:57.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:34:57 smithi177 ceph-mon[36431]: mgrmap e24: x(active, since 4s), standbys: z, y 2024-04-02T14:34:57.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:34:57 smithi177 ceph-mon[36431]: from='client.? 172.21.15.90:0/3762803642' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-02T14:34:58.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:57.999+0000 7f40d8000640 1 -- 172.21.15.90:0/460996425 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f40d01025c0 msgr2=0x7f40d01049b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:58.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:57.999+0000 7f40d8000640 1 --2- 172.21.15.90:0/460996425 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f40d01025c0 0x7f40d01049b0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f40c4009ca0 tx=0x7f40c402c100 comp rx=0 tx=0).stop 2024-04-02T14:34:58.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:58.000+0000 7f40d8000640 1 -- 172.21.15.90:0/460996425 shutdown_connections 2024-04-02T14:34:58.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:58.000+0000 7f40d8000640 1 --2- 172.21.15.90:0/460996425 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f40d0107820 0x7f40d0109c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:58.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:58.000+0000 7f40d8000640 1 --2- 172.21.15.90:0/460996425 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f40d0104ef0 0x7f40d01072e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:58.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:58.000+0000 7f40d8000640 1 --2- 172.21.15.90:0/460996425 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f40d01025c0 0x7f40d01049b0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:58.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:58.000+0000 7f40d8000640 1 -- 172.21.15.90:0/460996425 >> 172.21.15.90:0/460996425 conn(0x7f40d00fbfb0 msgr2=0x7f40d00fe3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:34:58.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:58.000+0000 7f40d8000640 1 -- 172.21.15.90:0/460996425 shutdown_connections 2024-04-02T14:34:58.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:58.000+0000 7f40d8000640 1 -- 172.21.15.90:0/460996425 wait complete. 2024-04-02T14:34:58.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:58.001+0000 7f40d8000640 1 Processor -- start 2024-04-02T14:34:58.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:58.001+0000 7f40d8000640 1 -- start start 2024-04-02T14:34:58.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:58.001+0000 7f40d8000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f40d01025c0 0x7f40d019a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:58.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:58.001+0000 7f40d8000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f40d0104ef0 0x7f40d019a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:58.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:58.001+0000 7f40d8000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f40d0107820 0x7f40d01a1660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:58.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:58.001+0000 7f40d8000640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f40d010d7c0 con 0x7f40d01025c0 2024-04-02T14:34:58.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:58.001+0000 7f40d8000640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f40d010d640 con 0x7f40d0107820 2024-04-02T14:34:58.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:58.001+0000 7f40d8000640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f40d010d940 con 0x7f40d0104ef0 2024-04-02T14:34:58.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:58.001+0000 7f40d6000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f40d0107820 0x7f40d01a1660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:58.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:58.001+0000 7f40d5600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f40d01025c0 0x7f40d019a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:58.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:58.001+0000 7f40d6000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f40d0107820 0x7f40d01a1660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:58902/0 (socket says 172.21.15.90:58902) 2024-04-02T14:34:58.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:58.001+0000 7f40d6000640 1 -- 172.21.15.90:0/3145756154 learned_addr learned my addr 172.21.15.90:0/3145756154 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:34:58.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:58.002+0000 7f40d4c00640 1 --2- 172.21.15.90:0/3145756154 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f40d0104ef0 0x7f40d019a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:58.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:58.002+0000 7f40d5600640 1 -- 172.21.15.90:0/3145756154 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f40d0104ef0 msgr2=0x7f40d019a5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:58.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:58.002+0000 7f40d5600640 1 --2- 172.21.15.90:0/3145756154 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f40d0104ef0 0x7f40d019a5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:58.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:58.002+0000 7f40d5600640 1 -- 172.21.15.90:0/3145756154 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f40d0107820 msgr2=0x7f40d01a1660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:58.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:58.002+0000 7f40d5600640 1 --2- 172.21.15.90:0/3145756154 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f40d0107820 0x7f40d01a1660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:58.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:58.002+0000 7f40d5600640 1 -- 172.21.15.90:0/3145756154 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40d01a1cd0 con 0x7f40d01025c0 2024-04-02T14:34:58.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:58.002+0000 7f40d6000640 1 --2- 172.21.15.90:0/3145756154 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f40d0107820 0x7f40d01a1660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:34:58.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:58.002+0000 7f40d5600640 1 --2- 172.21.15.90:0/3145756154 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f40d01025c0 0x7f40d019a0b0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f40c402c640 tx=0x7f40c4035d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:34:58.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:58.003+0000 7f40ce000640 1 -- 172.21.15.90:0/3145756154 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40c4039040 con 0x7f40d01025c0 2024-04-02T14:34:58.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:58.003+0000 7f40ce000640 1 -- 172.21.15.90:0/3145756154 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f40c4003970 con 0x7f40d01025c0 2024-04-02T14:34:58.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:58.003+0000 7f40ce000640 1 -- 172.21.15.90:0/3145756154 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40c4004330 con 0x7f40d01025c0 2024-04-02T14:34:58.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:58.003+0000 7f40d8000640 1 -- 172.21.15.90:0/3145756154 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40d01a1f60 con 0x7f40d01025c0 2024-04-02T14:34:58.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:58.003+0000 7f40d8000640 1 -- 172.21.15.90:0/3145756154 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40d01a2370 con 0x7f40d01025c0 2024-04-02T14:34:58.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:58.004+0000 7f40d8000640 1 -- 172.21.15.90:0/3145756154 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f40d0109370 con 0x7f40d01025c0 2024-04-02T14:34:58.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:58.006+0000 7f40ce000640 1 -- 172.21.15.90:0/3145756154 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7f40c4046030 con 0x7f40d01025c0 2024-04-02T14:34:58.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:58.008+0000 7f40ce000640 1 --2- 172.21.15.90:0/3145756154 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f40b80a3e40 0x7f40b80a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:58.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:58.008+0000 7f40ce000640 1 -- 172.21.15.90:0/3145756154 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f40c40f25e0 con 0x7f40d01025c0 2024-04-02T14:34:58.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:58.009+0000 7f40d4c00640 1 --2- 172.21.15.90:0/3145756154 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f40b80a3e40 0x7f40b80a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:58.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:58.009+0000 7f40d4c00640 1 --2- 172.21.15.90:0/3145756154 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f40b80a3e40 0x7f40b80a6300 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f40bc0095f0 tx=0x7f40bc005f90 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:34:58.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:58.013+0000 7f40ce000640 1 -- 172.21.15.90:0/3145756154 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7f40c404d060 con 0x7f40d01025c0 2024-04-02T14:34:58.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:58.238+0000 7f40d8000640 1 -- 172.21.15.90:0/3145756154 --> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f40d0061f40 con 0x7f40b80a3e40 2024-04-02T14:34:58.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:58.248+0000 7f40ce000640 1 -- 172.21.15.90:0/3145756154 <== mgr.25289 v2:172.21.15.90:6800/3904993754 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+257837 (secure 0 0 0) 0x7f40d0061f40 con 0x7f40b80a3e40 2024-04-02T14:34:58.253 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-02T14:34:58.254 INFO:teuthology.orchestra.run.smithi090.stderr:dumped all 2024-04-02T14:34:58.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:58.252+0000 7f40d8000640 1 -- 172.21.15.90:0/3145756154 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f40b80a3e40 msgr2=0x7f40b80a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:58.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:58.252+0000 7f40d8000640 1 --2- 172.21.15.90:0/3145756154 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f40b80a3e40 0x7f40b80a6300 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f40bc0095f0 tx=0x7f40bc005f90 comp rx=0 tx=0).stop 2024-04-02T14:34:58.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:58.252+0000 7f40d8000640 1 -- 172.21.15.90:0/3145756154 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f40d01025c0 msgr2=0x7f40d019a0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:58.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:58.252+0000 7f40d8000640 1 --2- 172.21.15.90:0/3145756154 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f40d01025c0 0x7f40d019a0b0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f40c402c640 tx=0x7f40c4035d90 comp rx=0 tx=0).stop 2024-04-02T14:34:58.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:58.252+0000 7f40d8000640 1 -- 172.21.15.90:0/3145756154 shutdown_connections 2024-04-02T14:34:58.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:58.252+0000 7f40d8000640 1 --2- 172.21.15.90:0/3145756154 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f40b80a3e40 0x7f40b80a6300 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:58.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:58.252+0000 7f40d8000640 1 --2- 172.21.15.90:0/3145756154 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f40d0107820 0x7f40d01a1660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:58.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:58.252+0000 7f40d8000640 1 --2- 172.21.15.90:0/3145756154 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f40d0104ef0 0x7f40d019a5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:58.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:58.252+0000 7f40d8000640 1 --2- 172.21.15.90:0/3145756154 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f40d01025c0 0x7f40d019a0b0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:58.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:58.252+0000 7f40d8000640 1 -- 172.21.15.90:0/3145756154 >> 172.21.15.90:0/3145756154 conn(0x7f40d00fbfb0 msgr2=0x7f40d0101d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:34:58.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:58.252+0000 7f40d8000640 1 -- 172.21.15.90:0/3145756154 shutdown_connections 2024-04-02T14:34:58.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:58.252+0000 7f40d8000640 1 -- 172.21.15.90:0/3145756154 wait complete. 2024-04-02T14:34:58.272 INFO:teuthology.orchestra.run.smithi090.stdout:{"pg_ready":true,"pg_map":{"version":5,"stamp":"2024-04-02T14:34:56.270431+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1191833,"num_objects":104,"num_object_clones":0,"num_object_copies":312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":104,"num_whiteouts":0,"num_read":1697,"num_read_kb":1008,"num_write":555,"num_write_kb":4405,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":1224785,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":939,"ondisk_log_size":939,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":409868,"kb_used_data":14920,"kb_used_omap":18,"kb_used_meta":394797,"kb_avail":1124531956,"statfs":{"total":1151940427776,"available":1151520722944,"internally_reserved":0,"allocated":15278080,"data_stored":8687161,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":404272442},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.977835"},"pg_stats":[{"pgid":"3.3f","version":"90'8","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.549650+0000","last_change":"2024-04-02T14:32:22.558913+0000","last_active":"2024-04-02T14:34:52.549650+0000","last_peered":"2024-04-02T14:34:52.549650+0000","last_clean":"2024-04-02T14:34:52.549650+0000","last_became_active":"2024-04-02T14:32:22.558606+0000","last_became_peered":"2024-04-02T14:32:22.558606+0000","last_unstale":"2024-04-02T14:34:52.549650+0000","last_undegraded":"2024-04-02T14:34:52.549650+0000","last_fullsized":"2024-04-02T14:34:52.549650+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:59:49.205499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"90'4","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.547653+0000","last_change":"2024-04-02T14:32:22.551898+0000","last_active":"2024-04-02T14:34:52.547653+0000","last_peered":"2024-04-02T14:34:52.547653+0000","last_clean":"2024-04-02T14:34:52.547653+0000","last_became_active":"2024-04-02T14:32:22.549016+0000","last_became_peered":"2024-04-02T14:32:22.549016+0000","last_unstale":"2024-04-02T14:34:52.547653+0000","last_undegraded":"2024-04-02T14:34:52.547653+0000","last_fullsized":"2024-04-02T14:34:52.547653+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:38:03.529505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"90'7","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.548042+0000","last_change":"2024-04-02T14:32:22.556214+0000","last_active":"2024-04-02T14:34:52.548042+0000","last_peered":"2024-04-02T14:34:52.548042+0000","last_clean":"2024-04-02T14:34:52.548042+0000","last_became_active":"2024-04-02T14:32:22.555686+0000","last_became_peered":"2024-04-02T14:32:22.555686+0000","last_unstale":"2024-04-02T14:34:52.548042+0000","last_undegraded":"2024-04-02T14:34:52.548042+0000","last_fullsized":"2024-04-02T14:34:52.548042+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:06:40.161203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"90'8","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.247614+0000","last_change":"2024-04-02T14:32:22.537830+0000","last_active":"2024-04-02T14:34:52.247614+0000","last_peered":"2024-04-02T14:34:52.247614+0000","last_clean":"2024-04-02T14:34:52.247614+0000","last_became_active":"2024-04-02T14:32:22.536860+0000","last_became_peered":"2024-04-02T14:32:22.536860+0000","last_unstale":"2024-04-02T14:34:52.247614+0000","last_undegraded":"2024-04-02T14:34:52.247614+0000","last_fullsized":"2024-04-02T14:34:52.247614+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T01:07:57.244177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"90'4","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.247802+0000","last_change":"2024-04-02T14:32:22.549550+0000","last_active":"2024-04-02T14:34:52.247802+0000","last_peered":"2024-04-02T14:34:52.247802+0000","last_clean":"2024-04-02T14:34:52.247802+0000","last_became_active":"2024-04-02T14:32:22.549177+0000","last_became_peered":"2024-04-02T14:32:22.549177+0000","last_unstale":"2024-04-02T14:34:52.247802+0000","last_undegraded":"2024-04-02T14:34:52.247802+0000","last_fullsized":"2024-04-02T14:34:52.247802+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:31:49.175583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"90'6","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.380610+0000","last_change":"2024-04-02T14:32:35.849338+0000","last_active":"2024-04-02T14:34:52.380610+0000","last_peered":"2024-04-02T14:34:52.380610+0000","last_clean":"2024-04-02T14:34:52.380610+0000","last_became_active":"2024-04-02T14:32:35.848929+0000","last_became_peered":"2024-04-02T14:32:35.848929+0000","last_unstale":"2024-04-02T14:34:52.380610+0000","last_undegraded":"2024-04-02T14:34:52.380610+0000","last_fullsized":"2024-04-02T14:34:52.380610+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:55:58.614290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,6],"acting":[10,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"90'5","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.543923+0000","last_change":"2024-04-02T14:32:22.546861+0000","last_active":"2024-04-02T14:34:52.543923+0000","last_peered":"2024-04-02T14:34:52.543923+0000","last_clean":"2024-04-02T14:34:52.543923+0000","last_became_active":"2024-04-02T14:32:22.546477+0000","last_became_peered":"2024-04-02T14:32:22.546477+0000","last_unstale":"2024-04-02T14:34:52.543923+0000","last_undegraded":"2024-04-02T14:34:52.543923+0000","last_fullsized":"2024-04-02T14:34:52.543923+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:59:17.133714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"90'9","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.246514+0000","last_change":"2024-04-02T14:32:22.554287+0000","last_active":"2024-04-02T14:34:52.246514+0000","last_peered":"2024-04-02T14:34:52.246514+0000","last_clean":"2024-04-02T14:34:52.246514+0000","last_became_active":"2024-04-02T14:32:22.553491+0000","last_became_peered":"2024-04-02T14:32:22.553491+0000","last_unstale":"2024-04-02T14:34:52.246514+0000","last_undegraded":"2024-04-02T14:34:52.246514+0000","last_fullsized":"2024-04-02T14:34:52.246514+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:51:30.785380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"90'11","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.546344+0000","last_change":"2024-04-02T14:32:22.538158+0000","last_active":"2024-04-02T14:34:52.546344+0000","last_peered":"2024-04-02T14:34:52.546344+0000","last_clean":"2024-04-02T14:34:52.546344+0000","last_became_active":"2024-04-02T14:32:22.537796+0000","last_became_peered":"2024-04-02T14:32:22.537796+0000","last_unstale":"2024-04-02T14:34:52.546344+0000","last_undegraded":"2024-04-02T14:34:52.546344+0000","last_fullsized":"2024-04-02T14:34:52.546344+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:09:48.054965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"90'16","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.381066+0000","last_change":"2024-04-02T14:32:22.535225+0000","last_active":"2024-04-02T14:34:52.381066+0000","last_peered":"2024-04-02T14:34:52.381066+0000","last_clean":"2024-04-02T14:34:52.381066+0000","last_became_active":"2024-04-02T14:32:22.534856+0000","last_became_peered":"2024-04-02T14:32:22.534856+0000","last_unstale":"2024-04-02T14:34:52.381066+0000","last_undegraded":"2024-04-02T14:34:52.381066+0000","last_fullsized":"2024-04-02T14:34:52.381066+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:22:11.118918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"90'6","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.545055+0000","last_change":"2024-04-02T14:32:22.539138+0000","last_active":"2024-04-02T14:34:52.545055+0000","last_peered":"2024-04-02T14:34:52.545055+0000","last_clean":"2024-04-02T14:34:52.545055+0000","last_became_active":"2024-04-02T14:32:22.538762+0000","last_became_peered":"2024-04-02T14:32:22.538762+0000","last_unstale":"2024-04-02T14:34:52.545055+0000","last_undegraded":"2024-04-02T14:34:52.545055+0000","last_fullsized":"2024-04-02T14:34:52.545055+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:18:59.174339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"90'7","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.247639+0000","last_change":"2024-04-02T14:32:22.536383+0000","last_active":"2024-04-02T14:34:52.247639+0000","last_peered":"2024-04-02T14:34:52.247639+0000","last_clean":"2024-04-02T14:34:52.247639+0000","last_became_active":"2024-04-02T14:32:22.536043+0000","last_became_peered":"2024-04-02T14:32:22.536043+0000","last_unstale":"2024-04-02T14:34:52.247639+0000","last_undegraded":"2024-04-02T14:34:52.247639+0000","last_fullsized":"2024-04-02T14:34:52.247639+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:14:48.628684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"90'10","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.545677+0000","last_change":"2024-04-02T14:32:22.552512+0000","last_active":"2024-04-02T14:34:52.545677+0000","last_peered":"2024-04-02T14:34:52.545677+0000","last_clean":"2024-04-02T14:34:52.545677+0000","last_became_active":"2024-04-02T14:32:22.549502+0000","last_became_peered":"2024-04-02T14:32:22.549502+0000","last_unstale":"2024-04-02T14:34:52.545677+0000","last_undegraded":"2024-04-02T14:34:52.545677+0000","last_fullsized":"2024-04-02T14:34:52.545677+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:37:45.609280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"90'7","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.547684+0000","last_change":"2024-04-02T14:32:22.555074+0000","last_active":"2024-04-02T14:34:52.547684+0000","last_peered":"2024-04-02T14:34:52.547684+0000","last_clean":"2024-04-02T14:34:52.547684+0000","last_became_active":"2024-04-02T14:32:22.554422+0000","last_became_peered":"2024-04-02T14:32:22.554422+0000","last_unstale":"2024-04-02T14:34:52.547684+0000","last_undegraded":"2024-04-02T14:34:52.547684+0000","last_fullsized":"2024-04-02T14:34:52.547684+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:27:49.536941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"90'7","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.547699+0000","last_change":"2024-04-02T14:32:22.550322+0000","last_active":"2024-04-02T14:34:52.547699+0000","last_peered":"2024-04-02T14:34:52.547699+0000","last_clean":"2024-04-02T14:34:52.547699+0000","last_became_active":"2024-04-02T14:32:22.548710+0000","last_became_peered":"2024-04-02T14:32:22.548710+0000","last_unstale":"2024-04-02T14:34:52.547699+0000","last_undegraded":"2024-04-02T14:34:52.547699+0000","last_fullsized":"2024-04-02T14:34:52.547699+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:19:09.226556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"90'9","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.545820+0000","last_change":"2024-04-02T14:32:22.558705+0000","last_active":"2024-04-02T14:34:52.545820+0000","last_peered":"2024-04-02T14:34:52.545820+0000","last_clean":"2024-04-02T14:34:52.545820+0000","last_became_active":"2024-04-02T14:32:22.558309+0000","last_became_peered":"2024-04-02T14:32:22.558309+0000","last_unstale":"2024-04-02T14:34:52.545820+0000","last_undegraded":"2024-04-02T14:34:52.545820+0000","last_fullsized":"2024-04-02T14:34:52.545820+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:52:23.929627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"90'8","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.549815+0000","last_change":"2024-04-02T14:32:22.551211+0000","last_active":"2024-04-02T14:34:52.549815+0000","last_peered":"2024-04-02T14:34:52.549815+0000","last_clean":"2024-04-02T14:34:52.549815+0000","last_became_active":"2024-04-02T14:32:22.550882+0000","last_became_peered":"2024-04-02T14:32:22.550882+0000","last_unstale":"2024-04-02T14:34:52.549815+0000","last_undegraded":"2024-04-02T14:34:52.549815+0000","last_fullsized":"2024-04-02T14:34:52.549815+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:54:54.533063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"90'10","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.368148+0000","last_change":"2024-04-02T14:32:22.550692+0000","last_active":"2024-04-02T14:34:52.368148+0000","last_peered":"2024-04-02T14:34:52.368148+0000","last_clean":"2024-04-02T14:34:52.368148+0000","last_became_active":"2024-04-02T14:32:22.550292+0000","last_became_peered":"2024-04-02T14:32:22.550292+0000","last_unstale":"2024-04-02T14:34:52.368148+0000","last_undegraded":"2024-04-02T14:34:52.368148+0000","last_fullsized":"2024-04-02T14:34:52.368148+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:14:42.317649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"90'7","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.283769+0000","last_change":"2024-04-02T14:32:22.553042+0000","last_active":"2024-04-02T14:34:52.283769+0000","last_peered":"2024-04-02T14:34:52.283769+0000","last_clean":"2024-04-02T14:34:52.283769+0000","last_became_active":"2024-04-02T14:32:22.552010+0000","last_became_peered":"2024-04-02T14:32:22.552010+0000","last_unstale":"2024-04-02T14:34:52.283769+0000","last_undegraded":"2024-04-02T14:34:52.283769+0000","last_fullsized":"2024-04-02T14:34:52.283769+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:31:13.947662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"88'13","reported_seq":199,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.367999+0000","last_change":"2024-04-02T14:32:35.868597+0000","last_active":"2024-04-02T14:34:52.367999+0000","last_peered":"2024-04-02T14:34:52.367999+0000","last_clean":"2024-04-02T14:34:52.367999+0000","last_became_active":"2024-04-02T14:32:35.851962+0000","last_became_peered":"2024-04-02T14:32:35.851962+0000","last_unstale":"2024-04-02T14:34:52.367999+0000","last_undegraded":"2024-04-02T14:34:52.367999+0000","last_fullsized":"2024-04-02T14:34:52.367999+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:17:06.400744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":140,"num_read_kb":140,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":672,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,4],"acting":[9,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"90'5","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.247056+0000","last_change":"2024-04-02T14:32:22.549526+0000","last_active":"2024-04-02T14:34:52.247056+0000","last_peered":"2024-04-02T14:34:52.247056+0000","last_clean":"2024-04-02T14:34:52.247056+0000","last_became_active":"2024-04-02T14:32:22.548250+0000","last_became_peered":"2024-04-02T14:32:22.548250+0000","last_unstale":"2024-04-02T14:34:52.247056+0000","last_undegraded":"2024-04-02T14:34:52.247056+0000","last_fullsized":"2024-04-02T14:34:52.247056+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:35:34.866803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"88'2","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.247157+0000","last_change":"2024-04-02T14:32:21.865084+0000","last_active":"2024-04-02T14:34:52.247157+0000","last_peered":"2024-04-02T14:34:52.247157+0000","last_clean":"2024-04-02T14:34:52.247157+0000","last_became_active":"2024-04-02T14:32:21.864718+0000","last_became_peered":"2024-04-02T14:32:21.864718+0000","last_unstale":"2024-04-02T14:34:52.247157+0000","last_undegraded":"2024-04-02T14:34:52.247157+0000","last_fullsized":"2024-04-02T14:34:52.247157+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:10:50.611362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"90'6","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.549008+0000","last_change":"2024-04-02T14:32:22.549443+0000","last_active":"2024-04-02T14:34:52.549008+0000","last_peered":"2024-04-02T14:34:52.549008+0000","last_clean":"2024-04-02T14:34:52.549008+0000","last_became_active":"2024-04-02T14:32:22.549059+0000","last_became_peered":"2024-04-02T14:32:22.549059+0000","last_unstale":"2024-04-02T14:34:52.549008+0000","last_undegraded":"2024-04-02T14:34:52.549008+0000","last_fullsized":"2024-04-02T14:34:52.549008+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:04:21.489864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"88'12","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.248319+0000","last_change":"2024-04-02T14:32:21.841368+0000","last_active":"2024-04-02T14:34:52.248319+0000","last_peered":"2024-04-02T14:34:52.248319+0000","last_clean":"2024-04-02T14:34:52.248319+0000","last_became_active":"2024-04-02T14:32:21.840963+0000","last_became_peered":"2024-04-02T14:32:21.840963+0000","last_unstale":"2024-04-02T14:34:52.248319+0000","last_undegraded":"2024-04-02T14:34:52.248319+0000","last_fullsized":"2024-04-02T14:34:52.248319+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:36:40.549720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"90'9","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.546094+0000","last_change":"2024-04-02T14:32:22.549753+0000","last_active":"2024-04-02T14:34:52.546094+0000","last_peered":"2024-04-02T14:34:52.546094+0000","last_clean":"2024-04-02T14:34:52.546094+0000","last_became_active":"2024-04-02T14:32:22.549354+0000","last_became_peered":"2024-04-02T14:32:22.549354+0000","last_unstale":"2024-04-02T14:34:52.546094+0000","last_undegraded":"2024-04-02T14:34:52.546094+0000","last_fullsized":"2024-04-02T14:34:52.546094+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T02:17:09.650946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"88'5","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.248127+0000","last_change":"2024-04-02T14:32:21.842766+0000","last_active":"2024-04-02T14:34:52.248127+0000","last_peered":"2024-04-02T14:34:52.248127+0000","last_clean":"2024-04-02T14:34:52.248127+0000","last_became_active":"2024-04-02T14:32:21.842410+0000","last_became_peered":"2024-04-02T14:32:21.842410+0000","last_unstale":"2024-04-02T14:34:52.248127+0000","last_undegraded":"2024-04-02T14:34:52.248127+0000","last_fullsized":"2024-04-02T14:34:52.248127+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:49:31.651127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"90'15","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.377579+0000","last_change":"2024-04-02T14:32:22.538506+0000","last_active":"2024-04-02T14:34:52.377579+0000","last_peered":"2024-04-02T14:34:52.377579+0000","last_clean":"2024-04-02T14:34:52.377579+0000","last_became_active":"2024-04-02T14:32:22.538168+0000","last_became_peered":"2024-04-02T14:32:22.538168+0000","last_unstale":"2024-04-02T14:34:52.377579+0000","last_undegraded":"2024-04-02T14:34:52.377579+0000","last_fullsized":"2024-04-02T14:34:52.377579+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:30:04.746382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"90'7","reported_seq":176,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:53.040014+0000","last_change":"2024-04-02T14:32:21.844180+0000","last_active":"2024-04-02T14:34:53.040014+0000","last_peered":"2024-04-02T14:34:53.040014+0000","last_clean":"2024-04-02T14:34:53.040014+0000","last_became_active":"2024-04-02T14:32:21.843840+0000","last_became_peered":"2024-04-02T14:32:21.843840+0000","last_unstale":"2024-04-02T14:34:53.040014+0000","last_undegraded":"2024-04-02T14:34:53.040014+0000","last_fullsized":"2024-04-02T14:34:53.040014+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T01:56:10.767946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1154,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":147,"num_read_kb":3,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"90'7","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.247921+0000","last_change":"2024-04-02T14:32:22.548453+0000","last_active":"2024-04-02T14:34:52.247921+0000","last_peered":"2024-04-02T14:34:52.247921+0000","last_clean":"2024-04-02T14:34:52.247921+0000","last_became_active":"2024-04-02T14:32:22.548110+0000","last_became_peered":"2024-04-02T14:32:22.548110+0000","last_unstale":"2024-04-02T14:34:52.247921+0000","last_undegraded":"2024-04-02T14:34:52.247921+0000","last_fullsized":"2024-04-02T14:34:52.247921+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:57:13.067425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"88'7","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.380564+0000","last_change":"2024-04-02T14:32:21.846528+0000","last_active":"2024-04-02T14:34:52.380564+0000","last_peered":"2024-04-02T14:34:52.380564+0000","last_clean":"2024-04-02T14:34:52.380564+0000","last_became_active":"2024-04-02T14:32:21.846152+0000","last_became_peered":"2024-04-02T14:32:21.846152+0000","last_unstale":"2024-04-02T14:34:52.380564+0000","last_undegraded":"2024-04-02T14:34:52.380564+0000","last_fullsized":"2024-04-02T14:34:52.380564+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:58:09.286256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"90'8","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.377374+0000","last_change":"2024-04-02T14:32:22.546730+0000","last_active":"2024-04-02T14:34:52.377374+0000","last_peered":"2024-04-02T14:34:52.377374+0000","last_clean":"2024-04-02T14:34:52.377374+0000","last_became_active":"2024-04-02T14:32:22.546342+0000","last_became_peered":"2024-04-02T14:32:22.546342+0000","last_unstale":"2024-04-02T14:34:52.377374+0000","last_undegraded":"2024-04-02T14:34:52.377374+0000","last_fullsized":"2024-04-02T14:34:52.377374+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:38:37.616382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"88'3","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.545031+0000","last_change":"2024-04-02T14:32:21.853316+0000","last_active":"2024-04-02T14:34:52.545031+0000","last_peered":"2024-04-02T14:34:52.545031+0000","last_clean":"2024-04-02T14:34:52.545031+0000","last_became_active":"2024-04-02T14:32:21.852912+0000","last_became_peered":"2024-04-02T14:32:21.852912+0000","last_unstale":"2024-04-02T14:34:52.545031+0000","last_undegraded":"2024-04-02T14:34:52.545031+0000","last_fullsized":"2024-04-02T14:34:52.545031+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T02:30:44.601900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"90'8","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.246607+0000","last_change":"2024-04-02T14:32:22.558913+0000","last_active":"2024-04-02T14:34:52.246607+0000","last_peered":"2024-04-02T14:34:52.246607+0000","last_clean":"2024-04-02T14:34:52.246607+0000","last_became_active":"2024-04-02T14:32:22.558511+0000","last_became_peered":"2024-04-02T14:32:22.558511+0000","last_unstale":"2024-04-02T14:34:52.246607+0000","last_undegraded":"2024-04-02T14:34:52.246607+0000","last_fullsized":"2024-04-02T14:34:52.246607+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:48:36.449086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"88'3","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.377293+0000","last_change":"2024-04-02T14:32:21.844081+0000","last_active":"2024-04-02T14:34:52.377293+0000","last_peered":"2024-04-02T14:34:52.377293+0000","last_clean":"2024-04-02T14:34:52.377293+0000","last_became_active":"2024-04-02T14:32:21.843788+0000","last_became_peered":"2024-04-02T14:32:21.843788+0000","last_unstale":"2024-04-02T14:34:52.377293+0000","last_undegraded":"2024-04-02T14:34:52.377293+0000","last_fullsized":"2024-04-02T14:34:52.377293+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:19:02.184229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"90'11","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.367476+0000","last_change":"2024-04-02T14:32:22.556193+0000","last_active":"2024-04-02T14:34:52.367476+0000","last_peered":"2024-04-02T14:34:52.367476+0000","last_clean":"2024-04-02T14:34:52.367476+0000","last_became_active":"2024-04-02T14:32:22.555787+0000","last_became_peered":"2024-04-02T14:32:22.555787+0000","last_unstale":"2024-04-02T14:34:52.367476+0000","last_undegraded":"2024-04-02T14:34:52.367476+0000","last_fullsized":"2024-04-02T14:34:52.367476+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T02:02:13.730941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"91'6","reported_seq":316,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:53.039409+0000","last_change":"2024-04-02T14:32:21.840651+0000","last_active":"2024-04-02T14:34:53.039409+0000","last_peered":"2024-04-02T14:34:53.039409+0000","last_clean":"2024-04-02T14:34:53.039409+0000","last_became_active":"2024-04-02T14:32:21.840344+0000","last_became_peered":"2024-04-02T14:32:21.840344+0000","last_unstale":"2024-04-02T14:34:53.039409+0000","last_undegraded":"2024-04-02T14:34:53.039409+0000","last_fullsized":"2024-04-02T14:34:53.039409+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:49:22.384224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":288,"num_read_kb":288,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"90'10","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.546191+0000","last_change":"2024-04-02T14:32:22.548245+0000","last_active":"2024-04-02T14:34:52.546191+0000","last_peered":"2024-04-02T14:34:52.546191+0000","last_clean":"2024-04-02T14:34:52.546191+0000","last_became_active":"2024-04-02T14:32:22.547853+0000","last_became_peered":"2024-04-02T14:32:22.547853+0000","last_unstale":"2024-04-02T14:34:52.546191+0000","last_undegraded":"2024-04-02T14:34:52.546191+0000","last_fullsized":"2024-04-02T14:34:52.546191+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:02:43.036944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"88'4","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.544349+0000","last_change":"2024-04-02T14:32:21.842096+0000","last_active":"2024-04-02T14:34:52.544349+0000","last_peered":"2024-04-02T14:34:52.544349+0000","last_clean":"2024-04-02T14:34:52.544349+0000","last_became_active":"2024-04-02T14:32:21.841646+0000","last_became_peered":"2024-04-02T14:32:21.841646+0000","last_unstale":"2024-04-02T14:34:52.544349+0000","last_undegraded":"2024-04-02T14:34:52.544349+0000","last_fullsized":"2024-04-02T14:34:52.544349+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:04:08.149785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"90'9","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.548949+0000","last_change":"2024-04-02T14:32:22.557694+0000","last_active":"2024-04-02T14:34:52.548949+0000","last_peered":"2024-04-02T14:34:52.548949+0000","last_clean":"2024-04-02T14:34:52.548949+0000","last_became_active":"2024-04-02T14:32:22.557202+0000","last_became_peered":"2024-04-02T14:32:22.557202+0000","last_unstale":"2024-04-02T14:34:52.548949+0000","last_undegraded":"2024-04-02T14:34:52.548949+0000","last_fullsized":"2024-04-02T14:34:52.548949+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:35:41.471557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"88'5","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.377599+0000","last_change":"2024-04-02T14:32:21.864366+0000","last_active":"2024-04-02T14:34:52.377599+0000","last_peered":"2024-04-02T14:34:52.377599+0000","last_clean":"2024-04-02T14:34:52.377599+0000","last_became_active":"2024-04-02T14:32:21.864006+0000","last_became_peered":"2024-04-02T14:32:21.864006+0000","last_unstale":"2024-04-02T14:34:52.377599+0000","last_undegraded":"2024-04-02T14:34:52.377599+0000","last_fullsized":"2024-04-02T14:34:52.377599+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:08:43.176061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"90'11","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.546034+0000","last_change":"2024-04-02T14:32:22.541562+0000","last_active":"2024-04-02T14:34:52.546034+0000","last_peered":"2024-04-02T14:34:52.546034+0000","last_clean":"2024-04-02T14:34:52.546034+0000","last_became_active":"2024-04-02T14:32:22.541209+0000","last_became_peered":"2024-04-02T14:32:22.541209+0000","last_unstale":"2024-04-02T14:34:52.546034+0000","last_undegraded":"2024-04-02T14:34:52.546034+0000","last_fullsized":"2024-04-02T14:34:52.546034+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:49:40.310528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"91'38","reported_seq":376,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:53.062247+0000","last_change":"2024-04-02T14:32:35.875429+0000","last_active":"2024-04-02T14:34:53.062247+0000","last_peered":"2024-04-02T14:34:53.062247+0000","last_clean":"2024-04-02T14:34:53.062247+0000","last_became_active":"2024-04-02T14:32:35.852417+0000","last_became_peered":"2024-04-02T14:32:35.852417+0000","last_unstale":"2024-04-02T14:34:53.062247+0000","last_undegraded":"2024-04-02T14:34:53.062247+0000","last_fullsized":"2024-04-02T14:34:53.062247+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:33:53.903712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67756,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":289,"num_read_kb":205,"num_write":40,"num_write_kb":82,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":43377,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,4],"acting":[10,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"90'6","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.248005+0000","last_change":"2024-04-02T14:32:22.535716+0000","last_active":"2024-04-02T14:34:52.248005+0000","last_peered":"2024-04-02T14:34:52.248005+0000","last_clean":"2024-04-02T14:34:52.248005+0000","last_became_active":"2024-04-02T14:32:22.534738+0000","last_became_peered":"2024-04-02T14:32:22.534738+0000","last_unstale":"2024-04-02T14:34:52.248005+0000","last_undegraded":"2024-04-02T14:34:52.248005+0000","last_fullsized":"2024-04-02T14:34:52.248005+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T01:25:07.763915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"88'9","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.283698+0000","last_change":"2024-04-02T14:32:21.854582+0000","last_active":"2024-04-02T14:34:52.283698+0000","last_peered":"2024-04-02T14:34:52.283698+0000","last_clean":"2024-04-02T14:34:52.283698+0000","last_became_active":"2024-04-02T14:32:21.854226+0000","last_became_peered":"2024-04-02T14:32:21.854226+0000","last_unstale":"2024-04-02T14:34:52.283698+0000","last_undegraded":"2024-04-02T14:34:52.283698+0000","last_fullsized":"2024-04-02T14:34:52.283698+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:30:02.289163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"90'9","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.545367+0000","last_change":"2024-04-02T14:32:22.555901+0000","last_active":"2024-04-02T14:34:52.545367+0000","last_peered":"2024-04-02T14:34:52.545367+0000","last_clean":"2024-04-02T14:34:52.545367+0000","last_became_active":"2024-04-02T14:32:22.555158+0000","last_became_peered":"2024-04-02T14:32:22.555158+0000","last_unstale":"2024-04-02T14:34:52.545367+0000","last_undegraded":"2024-04-02T14:34:52.545367+0000","last_fullsized":"2024-04-02T14:34:52.545367+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:04:28.766185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"88'5","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.377201+0000","last_change":"2024-04-02T14:32:21.855982+0000","last_active":"2024-04-02T14:34:52.377201+0000","last_peered":"2024-04-02T14:34:52.377201+0000","last_clean":"2024-04-02T14:34:52.377201+0000","last_became_active":"2024-04-02T14:32:21.855591+0000","last_became_peered":"2024-04-02T14:32:21.855591+0000","last_unstale":"2024-04-02T14:34:52.377201+0000","last_undegraded":"2024-04-02T14:34:52.377201+0000","last_fullsized":"2024-04-02T14:34:52.377201+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:28:51.806779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"90'15","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.544517+0000","last_change":"2024-04-02T14:32:22.549889+0000","last_active":"2024-04-02T14:34:52.544517+0000","last_peered":"2024-04-02T14:34:52.544517+0000","last_clean":"2024-04-02T14:34:52.544517+0000","last_became_active":"2024-04-02T14:32:22.549479+0000","last_became_peered":"2024-04-02T14:32:22.549479+0000","last_unstale":"2024-04-02T14:34:52.544517+0000","last_undegraded":"2024-04-02T14:34:52.544517+0000","last_fullsized":"2024-04-02T14:34:52.544517+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:29:25.937576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"91'9","reported_seq":179,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:53.156649+0000","last_change":"2024-04-02T14:32:21.838165+0000","last_active":"2024-04-02T14:34:53.156649+0000","last_peered":"2024-04-02T14:34:53.156649+0000","last_clean":"2024-04-02T14:34:53.156649+0000","last_became_active":"2024-04-02T14:32:21.837808+0000","last_became_peered":"2024-04-02T14:32:21.837808+0000","last_unstale":"2024-04-02T14:34:53.156649+0000","last_undegraded":"2024-04-02T14:34:53.156649+0000","last_fullsized":"2024-04-02T14:34:53.156649+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:00:44.605314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1902,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":149,"num_read_kb":5,"num_write":9,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"90'6","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.543871+0000","last_change":"2024-04-02T14:32:22.532937+0000","last_active":"2024-04-02T14:34:52.543871+0000","last_peered":"2024-04-02T14:34:52.543871+0000","last_clean":"2024-04-02T14:34:52.543871+0000","last_became_active":"2024-04-02T14:32:22.532586+0000","last_became_peered":"2024-04-02T14:32:22.532586+0000","last_unstale":"2024-04-02T14:34:52.543871+0000","last_undegraded":"2024-04-02T14:34:52.543871+0000","last_fullsized":"2024-04-02T14:34:52.543871+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:50:48.030889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"88'17","reported_seq":40,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.247642+0000","last_change":"2024-04-02T14:32:21.860032+0000","last_active":"2024-04-02T14:34:52.247642+0000","last_peered":"2024-04-02T14:34:52.247642+0000","last_clean":"2024-04-02T14:34:52.247642+0000","last_became_active":"2024-04-02T14:32:21.859653+0000","last_became_peered":"2024-04-02T14:32:21.859653+0000","last_unstale":"2024-04-02T14:34:52.247642+0000","last_undegraded":"2024-04-02T14:34:52.247642+0000","last_fullsized":"2024-04-02T14:34:52.247642+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:06:01.144530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"90'6","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.367598+0000","last_change":"2024-04-02T14:32:35.736133+0000","last_active":"2024-04-02T14:34:52.367598+0000","last_peered":"2024-04-02T14:34:52.367598+0000","last_clean":"2024-04-02T14:34:52.367598+0000","last_became_active":"2024-04-02T14:32:35.734335+0000","last_became_peered":"2024-04-02T14:32:35.734335+0000","last_unstale":"2024-04-02T14:34:52.367598+0000","last_undegraded":"2024-04-02T14:34:52.367598+0000","last_fullsized":"2024-04-02T14:34:52.367598+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:36:21.828318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,1],"acting":[9,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.f","version":"91'3","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.248316+0000","last_change":"2024-04-02T14:32:21.842384+0000","last_active":"2024-04-02T14:34:52.248316+0000","last_peered":"2024-04-02T14:34:52.248316+0000","last_clean":"2024-04-02T14:34:52.248316+0000","last_became_active":"2024-04-02T14:32:21.842040+0000","last_became_peered":"2024-04-02T14:32:21.842040+0000","last_unstale":"2024-04-02T14:34:52.248316+0000","last_undegraded":"2024-04-02T14:34:52.248316+0000","last_fullsized":"2024-04-02T14:34:52.248316+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:33:29.541972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"90'5","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.549637+0000","last_change":"2024-04-02T14:32:22.538571+0000","last_active":"2024-04-02T14:34:52.549637+0000","last_peered":"2024-04-02T14:34:52.549637+0000","last_clean":"2024-04-02T14:34:52.549637+0000","last_became_active":"2024-04-02T14:32:22.538239+0000","last_became_peered":"2024-04-02T14:32:22.538239+0000","last_unstale":"2024-04-02T14:34:52.549637+0000","last_undegraded":"2024-04-02T14:34:52.549637+0000","last_fullsized":"2024-04-02T14:34:52.549637+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T01:19:53.120419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"88'1","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.367816+0000","last_change":"2024-04-02T14:32:21.857666+0000","last_active":"2024-04-02T14:34:52.367816+0000","last_peered":"2024-04-02T14:34:52.367816+0000","last_clean":"2024-04-02T14:34:52.367816+0000","last_became_active":"2024-04-02T14:32:21.856618+0000","last_became_peered":"2024-04-02T14:32:21.856618+0000","last_unstale":"2024-04-02T14:34:52.367816+0000","last_undegraded":"2024-04-02T14:34:52.367816+0000","last_fullsized":"2024-04-02T14:34:52.367816+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T01:20:53.183052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"90'5","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.380890+0000","last_change":"2024-04-02T14:32:22.552277+0000","last_active":"2024-04-02T14:34:52.380890+0000","last_peered":"2024-04-02T14:34:52.380890+0000","last_clean":"2024-04-02T14:34:52.380890+0000","last_became_active":"2024-04-02T14:32:22.551882+0000","last_became_peered":"2024-04-02T14:32:22.551882+0000","last_unstale":"2024-04-02T14:34:52.380890+0000","last_undegraded":"2024-04-02T14:34:52.380890+0000","last_fullsized":"2024-04-02T14:34:52.380890+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:30:43.592910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"88'5","reported_seq":171,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.544165+0000","last_change":"2024-04-02T14:32:21.848258+0000","last_active":"2024-04-02T14:34:52.544165+0000","last_peered":"2024-04-02T14:34:52.544165+0000","last_clean":"2024-04-02T14:34:52.544165+0000","last_became_active":"2024-04-02T14:32:21.847805+0000","last_became_peered":"2024-04-02T14:32:21.847805+0000","last_unstale":"2024-04-02T14:34:52.544165+0000","last_undegraded":"2024-04-02T14:34:52.544165+0000","last_fullsized":"2024-04-02T14:34:52.544165+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:46:07.552786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":144,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"90'7","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.380644+0000","last_change":"2024-04-02T14:32:22.560835+0000","last_active":"2024-04-02T14:34:52.380644+0000","last_peered":"2024-04-02T14:34:52.380644+0000","last_clean":"2024-04-02T14:34:52.380644+0000","last_became_active":"2024-04-02T14:32:22.560469+0000","last_became_peered":"2024-04-02T14:32:22.560469+0000","last_unstale":"2024-04-02T14:34:52.380644+0000","last_undegraded":"2024-04-02T14:34:52.380644+0000","last_fullsized":"2024-04-02T14:34:52.380644+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T02:29:24.869650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"90'8","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.377408+0000","last_change":"2024-04-02T14:32:22.538325+0000","last_active":"2024-04-02T14:34:52.377408+0000","last_peered":"2024-04-02T14:34:52.377408+0000","last_clean":"2024-04-02T14:34:52.377408+0000","last_became_active":"2024-04-02T14:32:22.537903+0000","last_became_peered":"2024-04-02T14:32:22.537903+0000","last_unstale":"2024-04-02T14:34:52.377408+0000","last_undegraded":"2024-04-02T14:34:52.377408+0000","last_fullsized":"2024-04-02T14:34:52.377408+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:12:26.586290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"91'7","reported_seq":174,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.549482+0000","last_change":"2024-04-02T14:32:21.861333+0000","last_active":"2024-04-02T14:34:52.549482+0000","last_peered":"2024-04-02T14:34:52.549482+0000","last_clean":"2024-04-02T14:34:52.549482+0000","last_became_active":"2024-04-02T14:32:21.860909+0000","last_became_peered":"2024-04-02T14:32:21.860909+0000","last_unstale":"2024-04-02T14:34:52.549482+0000","last_undegraded":"2024-04-02T14:34:52.549482+0000","last_fullsized":"2024-04-02T14:34:52.549482+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:21:52.616036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":145,"num_read_kb":145,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"90'8","reported_seq":22,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.367541+0000","last_change":"2024-04-02T14:32:35.736379+0000","last_active":"2024-04-02T14:34:52.367541+0000","last_peered":"2024-04-02T14:34:52.367541+0000","last_clean":"2024-04-02T14:34:52.367541+0000","last_became_active":"2024-04-02T14:32:35.735156+0000","last_became_peered":"2024-04-02T14:32:35.735156+0000","last_unstale":"2024-04-02T14:34:52.367541+0000","last_undegraded":"2024-04-02T14:34:52.367541+0000","last_fullsized":"2024-04-02T14:34:52.367541+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:16:28.405699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,1],"acting":[9,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.3","version":"88'6","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.247327+0000","last_change":"2024-04-02T14:32:21.837336+0000","last_active":"2024-04-02T14:34:52.247327+0000","last_peered":"2024-04-02T14:34:52.247327+0000","last_clean":"2024-04-02T14:34:52.247327+0000","last_became_active":"2024-04-02T14:32:21.836921+0000","last_became_peered":"2024-04-02T14:32:21.836921+0000","last_unstale":"2024-04-02T14:34:52.247327+0000","last_undegraded":"2024-04-02T14:34:52.247327+0000","last_fullsized":"2024-04-02T14:34:52.247327+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:34:52.883173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":358,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.543898+0000","last_change":"2024-04-02T14:31:02.710356+0000","last_active":"2024-04-02T14:34:52.543898+0000","last_peered":"2024-04-02T14:34:52.543898+0000","last_clean":"2024-04-02T14:34:52.543898+0000","last_became_active":"2024-04-02T14:31:02.709669+0000","last_became_peered":"2024-04-02T14:31:02.709669+0000","last_unstale":"2024-04-02T14:34:52.543898+0000","last_undegraded":"2024-04-02T14:34:52.543898+0000","last_fullsized":"2024-04-02T14:34:52.543898+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:27:17.358883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:27:17.358883+0000","last_clean_scrub_stamp":"2024-04-02T14:27:17.358883+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:45:55.893284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"90'5","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.544201+0000","last_change":"2024-04-02T14:32:22.545502+0000","last_active":"2024-04-02T14:34:52.544201+0000","last_peered":"2024-04-02T14:34:52.544201+0000","last_clean":"2024-04-02T14:34:52.544201+0000","last_became_active":"2024-04-02T14:32:22.545111+0000","last_became_peered":"2024-04-02T14:32:22.545111+0000","last_unstale":"2024-04-02T14:34:52.544201+0000","last_undegraded":"2024-04-02T14:34:52.544201+0000","last_fullsized":"2024-04-02T14:34:52.544201+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:02:32.178334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"88'4","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.549305+0000","last_change":"2024-04-02T14:32:35.736582+0000","last_active":"2024-04-02T14:34:52.549305+0000","last_peered":"2024-04-02T14:34:52.549305+0000","last_clean":"2024-04-02T14:34:52.549305+0000","last_became_active":"2024-04-02T14:32:35.733368+0000","last_became_peered":"2024-04-02T14:32:35.733368+0000","last_unstale":"2024-04-02T14:34:52.549305+0000","last_undegraded":"2024-04-02T14:34:52.549305+0000","last_fullsized":"2024-04-02T14:34:52.549305+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T02:14:35.713351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,8],"acting":[11,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"88'4","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.248107+0000","last_change":"2024-04-02T14:32:21.860226+0000","last_active":"2024-04-02T14:34:52.248107+0000","last_peered":"2024-04-02T14:34:52.248107+0000","last_clean":"2024-04-02T14:34:52.248107+0000","last_became_active":"2024-04-02T14:32:21.859854+0000","last_became_peered":"2024-04-02T14:32:21.859854+0000","last_unstale":"2024-04-02T14:34:52.248107+0000","last_undegraded":"2024-04-02T14:34:52.248107+0000","last_fullsized":"2024-04-02T14:34:52.248107+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:30:36.710677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"90'9","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.545807+0000","last_change":"2024-04-02T14:32:22.552287+0000","last_active":"2024-04-02T14:34:52.545807+0000","last_peered":"2024-04-02T14:34:52.545807+0000","last_clean":"2024-04-02T14:34:52.545807+0000","last_became_active":"2024-04-02T14:32:22.551719+0000","last_became_peered":"2024-04-02T14:32:22.551719+0000","last_unstale":"2024-04-02T14:34:52.545807+0000","last_undegraded":"2024-04-02T14:34:52.545807+0000","last_fullsized":"2024-04-02T14:34:52.545807+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:26:12.007600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"91'50","reported_seq":95,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.507926+0000","last_change":"2024-04-02T14:32:21.851671+0000","last_active":"2024-04-02T14:34:52.507926+0000","last_peered":"2024-04-02T14:34:52.507926+0000","last_clean":"2024-04-02T14:34:52.507926+0000","last_became_active":"2024-04-02T14:32:21.851297+0000","last_became_peered":"2024-04-02T14:32:21.851297+0000","last_unstale":"2024-04-02T14:34:52.507926+0000","last_undegraded":"2024-04-02T14:34:52.507926+0000","last_fullsized":"2024-04-02T14:34:52.507926+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:51:20.232086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":82,"num_read_kb":70,"num_write":88,"num_write_kb":838,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"90'5","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.247133+0000","last_change":"2024-04-02T14:32:22.554554+0000","last_active":"2024-04-02T14:34:52.247133+0000","last_peered":"2024-04-02T14:34:52.247133+0000","last_clean":"2024-04-02T14:34:52.247133+0000","last_became_active":"2024-04-02T14:32:22.554194+0000","last_became_peered":"2024-04-02T14:32:22.554194+0000","last_unstale":"2024-04-02T14:34:52.247133+0000","last_undegraded":"2024-04-02T14:34:52.247133+0000","last_fullsized":"2024-04-02T14:34:52.247133+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:27:51.382441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"88'7","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.247855+0000","last_change":"2024-04-02T14:32:21.850346+0000","last_active":"2024-04-02T14:34:52.247855+0000","last_peered":"2024-04-02T14:34:52.247855+0000","last_clean":"2024-04-02T14:34:52.247855+0000","last_became_active":"2024-04-02T14:32:21.850084+0000","last_became_peered":"2024-04-02T14:32:21.850084+0000","last_unstale":"2024-04-02T14:34:52.247855+0000","last_undegraded":"2024-04-02T14:34:52.247855+0000","last_fullsized":"2024-04-02T14:34:52.247855+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:56:17.774702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"90'7","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.548807+0000","last_change":"2024-04-02T14:32:22.557054+0000","last_active":"2024-04-02T14:34:52.548807+0000","last_peered":"2024-04-02T14:34:52.548807+0000","last_clean":"2024-04-02T14:34:52.548807+0000","last_became_active":"2024-04-02T14:32:22.556335+0000","last_became_peered":"2024-04-02T14:32:22.556335+0000","last_unstale":"2024-04-02T14:34:52.548807+0000","last_undegraded":"2024-04-02T14:34:52.548807+0000","last_fullsized":"2024-04-02T14:34:52.548807+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:06:48.907197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"88'2","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.545713+0000","last_change":"2024-04-02T14:32:21.860260+0000","last_active":"2024-04-02T14:34:52.545713+0000","last_peered":"2024-04-02T14:34:52.545713+0000","last_clean":"2024-04-02T14:34:52.545713+0000","last_became_active":"2024-04-02T14:32:21.859922+0000","last_became_peered":"2024-04-02T14:32:21.859922+0000","last_unstale":"2024-04-02T14:34:52.545713+0000","last_undegraded":"2024-04-02T14:34:52.545713+0000","last_fullsized":"2024-04-02T14:34:52.545713+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:07:45.850148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"90'2","reported_seq":22,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.247646+0000","last_change":"2024-04-02T14:32:22.536063+0000","last_active":"2024-04-02T14:34:52.247646+0000","last_peered":"2024-04-02T14:34:52.247646+0000","last_clean":"2024-04-02T14:34:52.247646+0000","last_became_active":"2024-04-02T14:32:22.534678+0000","last_became_peered":"2024-04-02T14:32:22.534678+0000","last_unstale":"2024-04-02T14:34:52.247646+0000","last_undegraded":"2024-04-02T14:34:52.247646+0000","last_fullsized":"2024-04-02T14:34:52.247646+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T02:07:26.186216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"88'7","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.246470+0000","last_change":"2024-04-02T14:32:21.848554+0000","last_active":"2024-04-02T14:34:52.246470+0000","last_peered":"2024-04-02T14:34:52.246470+0000","last_clean":"2024-04-02T14:34:52.246470+0000","last_became_active":"2024-04-02T14:32:21.848177+0000","last_became_peered":"2024-04-02T14:32:21.848177+0000","last_unstale":"2024-04-02T14:34:52.246470+0000","last_undegraded":"2024-04-02T14:34:52.246470+0000","last_fullsized":"2024-04-02T14:34:52.246470+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:47:04.213286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"90'9","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.549385+0000","last_change":"2024-04-02T14:32:22.547246+0000","last_active":"2024-04-02T14:34:52.549385+0000","last_peered":"2024-04-02T14:34:52.549385+0000","last_clean":"2024-04-02T14:34:52.549385+0000","last_became_active":"2024-04-02T14:32:22.546721+0000","last_became_peered":"2024-04-02T14:32:22.546721+0000","last_unstale":"2024-04-02T14:34:52.549385+0000","last_undegraded":"2024-04-02T14:34:52.549385+0000","last_fullsized":"2024-04-02T14:34:52.549385+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:11:30.615723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"88'3","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.549450+0000","last_change":"2024-04-02T14:32:35.743474+0000","last_active":"2024-04-02T14:34:52.549450+0000","last_peered":"2024-04-02T14:34:52.549450+0000","last_clean":"2024-04-02T14:34:52.549450+0000","last_became_active":"2024-04-02T14:32:35.738522+0000","last_became_peered":"2024-04-02T14:32:35.738522+0000","last_unstale":"2024-04-02T14:34:52.549450+0000","last_undegraded":"2024-04-02T14:34:52.549450+0000","last_fullsized":"2024-04-02T14:34:52.549450+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T01:14:43.989279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"90'8","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.246782+0000","last_change":"2024-04-02T14:32:22.561787+0000","last_active":"2024-04-02T14:34:52.246782+0000","last_peered":"2024-04-02T14:34:52.246782+0000","last_clean":"2024-04-02T14:34:52.246782+0000","last_became_active":"2024-04-02T14:32:22.561284+0000","last_became_peered":"2024-04-02T14:32:22.561284+0000","last_unstale":"2024-04-02T14:34:52.246782+0000","last_undegraded":"2024-04-02T14:34:52.246782+0000","last_fullsized":"2024-04-02T14:34:52.246782+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:46:06.450097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"88'4","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.247746+0000","last_change":"2024-04-02T14:32:21.841340+0000","last_active":"2024-04-02T14:34:52.247746+0000","last_peered":"2024-04-02T14:34:52.247746+0000","last_clean":"2024-04-02T14:34:52.247746+0000","last_became_active":"2024-04-02T14:32:21.840937+0000","last_became_peered":"2024-04-02T14:32:21.840937+0000","last_unstale":"2024-04-02T14:34:52.247746+0000","last_undegraded":"2024-04-02T14:34:52.247746+0000","last_fullsized":"2024-04-02T14:34:52.247746+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T01:47:44.834440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"90'5","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.544165+0000","last_change":"2024-04-02T14:32:22.538890+0000","last_active":"2024-04-02T14:34:52.544165+0000","last_peered":"2024-04-02T14:34:52.544165+0000","last_clean":"2024-04-02T14:34:52.544165+0000","last_became_active":"2024-04-02T14:32:22.538512+0000","last_became_peered":"2024-04-02T14:32:22.538512+0000","last_unstale":"2024-04-02T14:34:52.544165+0000","last_undegraded":"2024-04-02T14:34:52.544165+0000","last_fullsized":"2024-04-02T14:34:52.544165+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:59:23.792851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"91'6","reported_seq":175,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:53.040074+0000","last_change":"2024-04-02T14:32:21.862727+0000","last_active":"2024-04-02T14:34:53.040074+0000","last_peered":"2024-04-02T14:34:53.040074+0000","last_clean":"2024-04-02T14:34:53.040074+0000","last_became_active":"2024-04-02T14:32:21.862405+0000","last_became_peered":"2024-04-02T14:32:21.862405+0000","last_unstale":"2024-04-02T14:34:53.040074+0000","last_undegraded":"2024-04-02T14:34:53.040074+0000","last_fullsized":"2024-04-02T14:34:53.040074+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:36:54.849964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1561,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":147,"num_read_kb":3,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"90'7","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.544768+0000","last_change":"2024-04-02T14:32:22.540077+0000","last_active":"2024-04-02T14:34:52.544768+0000","last_peered":"2024-04-02T14:34:52.544768+0000","last_clean":"2024-04-02T14:34:52.544768+0000","last_became_active":"2024-04-02T14:32:22.539513+0000","last_became_peered":"2024-04-02T14:32:22.539513+0000","last_unstale":"2024-04-02T14:34:52.544768+0000","last_undegraded":"2024-04-02T14:34:52.544768+0000","last_fullsized":"2024-04-02T14:34:52.544768+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T02:00:39.819963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"91'41","reported_seq":71,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.501385+0000","last_change":"2024-04-02T14:32:21.860310+0000","last_active":"2024-04-02T14:34:52.501385+0000","last_peered":"2024-04-02T14:34:52.501385+0000","last_clean":"2024-04-02T14:34:52.501385+0000","last_became_active":"2024-04-02T14:32:21.859955+0000","last_became_peered":"2024-04-02T14:32:21.859955+0000","last_unstale":"2024-04-02T14:34:52.501385+0000","last_undegraded":"2024-04-02T14:34:52.501385+0000","last_fullsized":"2024-04-02T14:34:52.501385+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:11:32.732603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131670,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":49,"num_write_kb":475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"90'7","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.544288+0000","last_change":"2024-04-02T14:32:22.552134+0000","last_active":"2024-04-02T14:34:52.544288+0000","last_peered":"2024-04-02T14:34:52.544288+0000","last_clean":"2024-04-02T14:34:52.544288+0000","last_became_active":"2024-04-02T14:32:22.551695+0000","last_became_peered":"2024-04-02T14:32:22.551695+0000","last_unstale":"2024-04-02T14:34:52.544288+0000","last_undegraded":"2024-04-02T14:34:52.544288+0000","last_fullsized":"2024-04-02T14:34:52.544288+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:09:36.902243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"88'5","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.545160+0000","last_change":"2024-04-02T14:32:21.857571+0000","last_active":"2024-04-02T14:34:52.545160+0000","last_peered":"2024-04-02T14:34:52.545160+0000","last_clean":"2024-04-02T14:34:52.545160+0000","last_became_active":"2024-04-02T14:32:21.856935+0000","last_became_peered":"2024-04-02T14:32:21.856935+0000","last_unstale":"2024-04-02T14:34:52.545160+0000","last_undegraded":"2024-04-02T14:34:52.545160+0000","last_fullsized":"2024-04-02T14:34:52.545160+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:55:46.079097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"90'9","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.544456+0000","last_change":"2024-04-02T14:32:22.534708+0000","last_active":"2024-04-02T14:34:52.544456+0000","last_peered":"2024-04-02T14:34:52.544456+0000","last_clean":"2024-04-02T14:34:52.544456+0000","last_became_active":"2024-04-02T14:32:22.534365+0000","last_became_peered":"2024-04-02T14:32:22.534365+0000","last_unstale":"2024-04-02T14:34:52.544456+0000","last_undegraded":"2024-04-02T14:34:52.544456+0000","last_fullsized":"2024-04-02T14:34:52.544456+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:34:47.435729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"90'9","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.549434+0000","last_change":"2024-04-02T14:32:22.538317+0000","last_active":"2024-04-02T14:34:52.549434+0000","last_peered":"2024-04-02T14:34:52.549434+0000","last_clean":"2024-04-02T14:34:52.549434+0000","last_became_active":"2024-04-02T14:32:22.537939+0000","last_became_peered":"2024-04-02T14:32:22.537939+0000","last_unstale":"2024-04-02T14:34:52.549434+0000","last_undegraded":"2024-04-02T14:34:52.549434+0000","last_fullsized":"2024-04-02T14:34:52.549434+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:31:47.075326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"90'12","reported_seq":38,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.283556+0000","last_change":"2024-04-02T14:32:35.852202+0000","last_active":"2024-04-02T14:34:52.283556+0000","last_peered":"2024-04-02T14:34:52.283556+0000","last_clean":"2024-04-02T14:34:52.283556+0000","last_became_active":"2024-04-02T14:32:35.851785+0000","last_became_peered":"2024-04-02T14:32:35.851785+0000","last_unstale":"2024-04-02T14:34:52.283556+0000","last_undegraded":"2024-04-02T14:34:52.283556+0000","last_fullsized":"2024-04-02T14:34:52.283556+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:09:17.410727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,4],"acting":[2,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"90'8","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.283542+0000","last_change":"2024-04-02T14:32:22.556567+0000","last_active":"2024-04-02T14:34:52.283542+0000","last_peered":"2024-04-02T14:34:52.283542+0000","last_clean":"2024-04-02T14:34:52.283542+0000","last_became_active":"2024-04-02T14:32:22.556267+0000","last_became_peered":"2024-04-02T14:32:22.556267+0000","last_unstale":"2024-04-02T14:34:52.283542+0000","last_undegraded":"2024-04-02T14:34:52.283542+0000","last_fullsized":"2024-04-02T14:34:52.283542+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:35:30.039013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"90'7","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.547650+0000","last_change":"2024-04-02T14:32:35.848125+0000","last_active":"2024-04-02T14:34:52.547650+0000","last_peered":"2024-04-02T14:34:52.547650+0000","last_clean":"2024-04-02T14:34:52.547650+0000","last_became_active":"2024-04-02T14:32:35.847681+0000","last_became_peered":"2024-04-02T14:32:35.847681+0000","last_unstale":"2024-04-02T14:34:52.547650+0000","last_undegraded":"2024-04-02T14:34:52.547650+0000","last_fullsized":"2024-04-02T14:34:52.547650+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:51:38.040261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"90'4","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.545854+0000","last_change":"2024-04-02T14:32:22.558584+0000","last_active":"2024-04-02T14:34:52.545854+0000","last_peered":"2024-04-02T14:34:52.545854+0000","last_clean":"2024-04-02T14:34:52.545854+0000","last_became_active":"2024-04-02T14:32:22.557382+0000","last_became_peered":"2024-04-02T14:32:22.557382+0000","last_unstale":"2024-04-02T14:34:52.545854+0000","last_undegraded":"2024-04-02T14:34:52.545854+0000","last_fullsized":"2024-04-02T14:34:52.545854+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:09:25.107411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"90'6","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.549497+0000","last_change":"2024-04-02T14:32:22.538318+0000","last_active":"2024-04-02T14:34:52.549497+0000","last_peered":"2024-04-02T14:34:52.549497+0000","last_clean":"2024-04-02T14:34:52.549497+0000","last_became_active":"2024-04-02T14:32:22.538011+0000","last_became_peered":"2024-04-02T14:32:22.538011+0000","last_unstale":"2024-04-02T14:34:52.549497+0000","last_undegraded":"2024-04-02T14:34:52.549497+0000","last_fullsized":"2024-04-02T14:34:52.549497+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:11:21.269106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"90'15","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.283948+0000","last_change":"2024-04-02T14:32:22.535645+0000","last_active":"2024-04-02T14:34:52.283948+0000","last_peered":"2024-04-02T14:34:52.283948+0000","last_clean":"2024-04-02T14:34:52.283948+0000","last_became_active":"2024-04-02T14:32:22.535314+0000","last_became_peered":"2024-04-02T14:32:22.535314+0000","last_unstale":"2024-04-02T14:34:52.283948+0000","last_undegraded":"2024-04-02T14:34:52.283948+0000","last_fullsized":"2024-04-02T14:34:52.283948+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:51:27.879015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"90'2","reported_seq":22,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.247021+0000","last_change":"2024-04-02T14:32:22.553959+0000","last_active":"2024-04-02T14:34:52.247021+0000","last_peered":"2024-04-02T14:34:52.247021+0000","last_clean":"2024-04-02T14:34:52.247021+0000","last_became_active":"2024-04-02T14:32:22.553389+0000","last_became_peered":"2024-04-02T14:32:22.553389+0000","last_unstale":"2024-04-02T14:34:52.247021+0000","last_undegraded":"2024-04-02T14:34:52.247021+0000","last_fullsized":"2024-04-02T14:34:52.247021+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:55:00.552600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"90'6","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.377384+0000","last_change":"2024-04-02T14:32:22.551769+0000","last_active":"2024-04-02T14:34:52.377384+0000","last_peered":"2024-04-02T14:34:52.377384+0000","last_clean":"2024-04-02T14:34:52.377384+0000","last_became_active":"2024-04-02T14:32:22.551422+0000","last_became_peered":"2024-04-02T14:32:22.551422+0000","last_unstale":"2024-04-02T14:34:52.377384+0000","last_undegraded":"2024-04-02T14:34:52.377384+0000","last_fullsized":"2024-04-02T14:34:52.377384+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T01:15:07.262927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"90'8","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.246610+0000","last_change":"2024-04-02T14:32:22.548264+0000","last_active":"2024-04-02T14:34:52.246610+0000","last_peered":"2024-04-02T14:34:52.246610+0000","last_clean":"2024-04-02T14:34:52.246610+0000","last_became_active":"2024-04-02T14:32:22.547293+0000","last_became_peered":"2024-04-02T14:32:22.547293+0000","last_unstale":"2024-04-02T14:34:52.246610+0000","last_undegraded":"2024-04-02T14:34:52.246610+0000","last_fullsized":"2024-04-02T14:34:52.246610+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:53:19.896243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"90'10","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.545911+0000","last_change":"2024-04-02T14:32:22.556131+0000","last_active":"2024-04-02T14:34:52.545911+0000","last_peered":"2024-04-02T14:34:52.545911+0000","last_clean":"2024-04-02T14:34:52.545911+0000","last_became_active":"2024-04-02T14:32:22.555763+0000","last_became_peered":"2024-04-02T14:32:22.555763+0000","last_unstale":"2024-04-02T14:34:52.545911+0000","last_undegraded":"2024-04-02T14:34:52.545911+0000","last_fullsized":"2024-04-02T14:34:52.545911+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:45:22.485403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"90'11","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.545531+0000","last_change":"2024-04-02T14:32:22.562093+0000","last_active":"2024-04-02T14:34:52.545531+0000","last_peered":"2024-04-02T14:34:52.545531+0000","last_clean":"2024-04-02T14:34:52.545531+0000","last_became_active":"2024-04-02T14:32:22.561663+0000","last_became_peered":"2024-04-02T14:32:22.561663+0000","last_unstale":"2024-04-02T14:34:52.545531+0000","last_undegraded":"2024-04-02T14:34:52.545531+0000","last_fullsized":"2024-04-02T14:34:52.545531+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:14:25.209276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"90'8","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.549387+0000","last_change":"2024-04-02T14:32:22.541964+0000","last_active":"2024-04-02T14:34:52.549387+0000","last_peered":"2024-04-02T14:34:52.549387+0000","last_clean":"2024-04-02T14:34:52.549387+0000","last_became_active":"2024-04-02T14:32:22.541549+0000","last_became_peered":"2024-04-02T14:32:22.541549+0000","last_unstale":"2024-04-02T14:34:52.549387+0000","last_undegraded":"2024-04-02T14:34:52.549387+0000","last_fullsized":"2024-04-02T14:34:52.549387+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:04:56.783158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":601231,"num_objects":100,"num_object_clones":0,"num_object_copies":300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":0,"num_read":1561,"num_read_kb":889,"num_write":372,"num_write_kb":1465,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":44049,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2174976,"data_stored":1803693,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":300,"ondisk_log_size":300,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":83,"seq":356482285611,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32168,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712984,"statfs":{"total":95995035648,"available":95962095616,"internally_reserved":0,"allocated":1011712,"data_stored":484076,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":75,"seq":322122547249,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":37000,"kb_used_data":1664,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708152,"statfs":{"total":95995035648,"available":95957147648,"internally_reserved":0,"allocated":1703936,"data_stored":1133256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":36174273},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743477,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36736,"kb_used_data":1396,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708416,"statfs":{"total":95995035648,"available":95957417984,"internally_reserved":0,"allocated":1429504,"data_stored":876831,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":61,"seq":261993005117,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32564,"kb_used_data":1384,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712588,"statfs":{"total":95995035648,"available":95961690112,"internally_reserved":0,"allocated":1417216,"data_stored":876537,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299458,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32772,"kb_used_data":1592,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712380,"statfs":{"total":95995035648,"available":95961477120,"internally_reserved":0,"allocated":1630208,"data_stored":1077966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495686,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36336,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708816,"statfs":{"total":95995035648,"available":95957827584,"internally_reserved":0,"allocated":1019904,"data_stored":483332,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574947,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36404,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708748,"statfs":{"total":95995035648,"available":95957757952,"internally_reserved":0,"allocated":1089536,"data_stored":552001,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738472,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36336,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708816,"statfs":{"total":95995035648,"available":95957827584,"internally_reserved":0,"allocated":1024000,"data_stored":475707,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411422,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36348,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708804,"statfs":{"total":95995035648,"available":95957815296,"internally_reserved":0,"allocated":1032192,"data_stored":484997,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182487,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28032,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717120,"statfs":{"total":95995035648,"available":95966330880,"internally_reserved":0,"allocated":1036288,"data_stored":483157,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920851,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36420,"kb_used_data":1080,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708732,"statfs":{"total":95995035648,"available":95957741568,"internally_reserved":0,"allocated":1105920,"data_stored":551371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757324,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28752,"kb_used_data":1736,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716400,"statfs":{"total":95995035648,"available":95965593600,"internally_reserved":0,"allocated":1777664,"data_stored":1207930,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2569,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":69508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":2895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":557056,"data_stored":525246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":70442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":180224,"data_stored":134952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":4988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":394044,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":430080,"data_stored":394338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":258048,"data_stored":202406,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-02T14:34:58.276 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-02T14:34:58.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:58.498+0000 7f5dc2c00640 1 -- 172.21.15.90:0/677287867 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5dbc074560 msgr2=0x7f5dbc0749e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:58.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:58.498+0000 7f5dc2c00640 1 --2- 172.21.15.90:0/677287867 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5dbc074560 0x7f5dbc0749e0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f5da8009900 tx=0x7f5da802bc30 comp rx=0 tx=0).stop 2024-04-02T14:34:58.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:58.498+0000 7f5dc2c00640 1 -- 172.21.15.90:0/677287867 shutdown_connections 2024-04-02T14:34:58.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:58.498+0000 7f5dc2c00640 1 --2- 172.21.15.90:0/677287867 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5dbc10e100 0x7f5dbc1104f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:58.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:58.498+0000 7f5dc2c00640 1 --2- 172.21.15.90:0/677287867 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5dbc074560 0x7f5dbc0749e0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:58.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:58.498+0000 7f5dc2c00640 1 --2- 172.21.15.90:0/677287867 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5dbc075b40 0x7f5dbc073f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:58.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:58.498+0000 7f5dc2c00640 1 -- 172.21.15.90:0/677287867 >> 172.21.15.90:0/677287867 conn(0x7f5dbc0fd0a0 msgr2=0x7f5dbc0ff4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:34:58.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:58.498+0000 7f5dc2c00640 1 -- 172.21.15.90:0/677287867 shutdown_connections 2024-04-02T14:34:58.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:58.498+0000 7f5dc2c00640 1 -- 172.21.15.90:0/677287867 wait complete. 2024-04-02T14:34:58.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:58.499+0000 7f5dc2c00640 1 Processor -- start 2024-04-02T14:34:58.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:58.500+0000 7f5dc2c00640 1 -- start start 2024-04-02T14:34:58.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:58.500+0000 7f5dc2c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5dbc074560 0x7f5dbc19e420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:58.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:58.500+0000 7f5dc2c00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5dbc075b40 0x7f5dbc19e960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:58.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:58.500+0000 7f5dc2c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5dbc10e100 0x7f5dbc1a59d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:58.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:58.500+0000 7f5dc2c00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f5dbc113e10 con 0x7f5dbc074560 2024-04-02T14:34:58.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:58.500+0000 7f5dc2c00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f5dbc113c90 con 0x7f5dbc075b40 2024-04-02T14:34:58.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:58.500+0000 7f5dc2c00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f5dbc113f90 con 0x7f5dbc10e100 2024-04-02T14:34:58.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:58.500+0000 7f5dbbe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5dbc074560 0x7f5dbc19e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:58.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:58.500+0000 7f5dbb400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5dbc075b40 0x7f5dbc19e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:58.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:58.500+0000 7f5dbb400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5dbc075b40 0x7f5dbc19e960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:58930/0 (socket says 172.21.15.90:58930) 2024-04-02T14:34:58.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:58.500+0000 7f5dbb400640 1 -- 172.21.15.90:0/826453301 learned_addr learned my addr 172.21.15.90:0/826453301 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:34:58.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:58.500+0000 7f5dc0c00640 1 --2- 172.21.15.90:0/826453301 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5dbc10e100 0x7f5dbc1a59d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:58.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:58.501+0000 7f5dbb400640 1 -- 172.21.15.90:0/826453301 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5dbc10e100 msgr2=0x7f5dbc1a59d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:58.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:58.501+0000 7f5dbb400640 1 --2- 172.21.15.90:0/826453301 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5dbc10e100 0x7f5dbc1a59d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:58.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:58.501+0000 7f5dbb400640 1 -- 172.21.15.90:0/826453301 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5dbc074560 msgr2=0x7f5dbc19e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:58.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:58.501+0000 7f5dbb400640 1 --2- 172.21.15.90:0/826453301 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5dbc074560 0x7f5dbc19e420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:58.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:58.501+0000 7f5dbb400640 1 -- 172.21.15.90:0/826453301 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5dbc1a6040 con 0x7f5dbc075b40 2024-04-02T14:34:58.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:58.501+0000 7f5dbb400640 1 --2- 172.21.15.90:0/826453301 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5dbc075b40 0x7f5dbc19e960 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f5da8005ab0 tx=0x7f5da802c6c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:34:58.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:58.501+0000 7f5db8c00640 1 -- 172.21.15.90:0/826453301 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5da8041070 con 0x7f5dbc075b40 2024-04-02T14:34:58.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:58.501+0000 7f5dc2c00640 1 -- 172.21.15.90:0/826453301 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5dbc1a62d0 con 0x7f5dbc075b40 2024-04-02T14:34:58.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:58.501+0000 7f5db8c00640 1 -- 172.21.15.90:0/826453301 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5da8004450 con 0x7f5dbc075b40 2024-04-02T14:34:58.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:58.501+0000 7f5db8c00640 1 -- 172.21.15.90:0/826453301 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5da8003da0 con 0x7f5dbc075b40 2024-04-02T14:34:58.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:58.502+0000 7f5dc2c00640 1 -- 172.21.15.90:0/826453301 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5dbc1a6790 con 0x7f5dbc075b40 2024-04-02T14:34:58.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:58.503+0000 7f5dc2c00640 1 -- 172.21.15.90:0/826453301 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5dbc1042d0 con 0x7f5dbc075b40 2024-04-02T14:34:58.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:58.507+0000 7f5db8c00640 1 -- 172.21.15.90:0/826453301 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7f5da8044070 con 0x7f5dbc075b40 2024-04-02T14:34:58.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:58.508+0000 7f5db8c00640 1 --2- 172.21.15.90:0/826453301 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f5da00a3e40 0x7f5da00a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:58.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:58.508+0000 7f5db8c00640 1 -- 172.21.15.90:0/826453301 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f5da80f28d0 con 0x7f5dbc075b40 2024-04-02T14:34:58.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:58.508+0000 7f5dbbe00640 1 --2- 172.21.15.90:0/826453301 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f5da00a3e40 0x7f5da00a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:58.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:58.509+0000 7f5db8c00640 1 -- 172.21.15.90:0/826453301 <== mon.2 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7f5da80be790 con 0x7f5dbc075b40 2024-04-02T14:34:58.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:58.509+0000 7f5dbbe00640 1 --2- 172.21.15.90:0/826453301 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f5da00a3e40 0x7f5da00a6300 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f5dac0095f0 tx=0x7f5dac005f90 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:34:58.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:58.713+0000 7f5dc2c00640 1 -- 172.21.15.90:0/826453301 --> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f5dbc061f40 con 0x7f5da00a3e40 2024-04-02T14:34:58.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:58.722+0000 7f5db8c00640 1 -- 172.21.15.90:0/826453301 <== mgr.25289 v2:172.21.15.90:6800/3904993754 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+257837 (secure 0 0 0) 0x7f5dbc061f40 con 0x7f5da00a3e40 2024-04-02T14:34:58.726 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-02T14:34:58.727 INFO:teuthology.orchestra.run.smithi090.stderr:dumped all 2024-04-02T14:34:58.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:58.726+0000 7f5dc2c00640 1 -- 172.21.15.90:0/826453301 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f5da00a3e40 msgr2=0x7f5da00a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:58.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:58.726+0000 7f5dc2c00640 1 --2- 172.21.15.90:0/826453301 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f5da00a3e40 0x7f5da00a6300 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f5dac0095f0 tx=0x7f5dac005f90 comp rx=0 tx=0).stop 2024-04-02T14:34:58.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:58.726+0000 7f5dc2c00640 1 -- 172.21.15.90:0/826453301 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5dbc075b40 msgr2=0x7f5dbc19e960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:58.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:58.726+0000 7f5dc2c00640 1 --2- 172.21.15.90:0/826453301 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5dbc075b40 0x7f5dbc19e960 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f5da8005ab0 tx=0x7f5da802c6c0 comp rx=0 tx=0).stop 2024-04-02T14:34:58.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:58.726+0000 7f5dc2c00640 1 -- 172.21.15.90:0/826453301 shutdown_connections 2024-04-02T14:34:58.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:58.726+0000 7f5dc2c00640 1 --2- 172.21.15.90:0/826453301 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f5da00a3e40 0x7f5da00a6300 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:58.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:58.726+0000 7f5dc2c00640 1 --2- 172.21.15.90:0/826453301 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5dbc10e100 0x7f5dbc1a59d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:58.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:58.726+0000 7f5dc2c00640 1 --2- 172.21.15.90:0/826453301 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5dbc075b40 0x7f5dbc19e960 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:58.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:58.726+0000 7f5dc2c00640 1 --2- 172.21.15.90:0/826453301 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5dbc074560 0x7f5dbc19e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:58.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:58.726+0000 7f5dc2c00640 1 -- 172.21.15.90:0/826453301 >> 172.21.15.90:0/826453301 conn(0x7f5dbc0fd0a0 msgr2=0x7f5dbc1014a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:34:58.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:58.726+0000 7f5dc2c00640 1 -- 172.21.15.90:0/826453301 shutdown_connections 2024-04-02T14:34:58.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:58.726+0000 7f5dc2c00640 1 -- 172.21.15.90:0/826453301 wait complete. 2024-04-02T14:34:58.745 INFO:teuthology.orchestra.run.smithi090.stdout:{"pg_ready":true,"pg_map":{"version":6,"stamp":"2024-04-02T14:34:58.270956+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1191833,"num_objects":104,"num_object_clones":0,"num_object_copies":312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":104,"num_whiteouts":0,"num_read":1697,"num_read_kb":1008,"num_write":555,"num_write_kb":4405,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":1224785,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":939,"ondisk_log_size":939,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":409868,"kb_used_data":14920,"kb_used_omap":18,"kb_used_meta":394797,"kb_avail":1124531956,"statfs":{"total":1151940427776,"available":1151520722944,"internally_reserved":0,"allocated":15278080,"data_stored":8687161,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":404272442},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.978360"},"pg_stats":[{"pgid":"3.3f","version":"90'8","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.549650+0000","last_change":"2024-04-02T14:32:22.558913+0000","last_active":"2024-04-02T14:34:52.549650+0000","last_peered":"2024-04-02T14:34:52.549650+0000","last_clean":"2024-04-02T14:34:52.549650+0000","last_became_active":"2024-04-02T14:32:22.558606+0000","last_became_peered":"2024-04-02T14:32:22.558606+0000","last_unstale":"2024-04-02T14:34:52.549650+0000","last_undegraded":"2024-04-02T14:34:52.549650+0000","last_fullsized":"2024-04-02T14:34:52.549650+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:59:49.205499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"90'4","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.547653+0000","last_change":"2024-04-02T14:32:22.551898+0000","last_active":"2024-04-02T14:34:52.547653+0000","last_peered":"2024-04-02T14:34:52.547653+0000","last_clean":"2024-04-02T14:34:52.547653+0000","last_became_active":"2024-04-02T14:32:22.549016+0000","last_became_peered":"2024-04-02T14:32:22.549016+0000","last_unstale":"2024-04-02T14:34:52.547653+0000","last_undegraded":"2024-04-02T14:34:52.547653+0000","last_fullsized":"2024-04-02T14:34:52.547653+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:38:03.529505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"90'7","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.548042+0000","last_change":"2024-04-02T14:32:22.556214+0000","last_active":"2024-04-02T14:34:52.548042+0000","last_peered":"2024-04-02T14:34:52.548042+0000","last_clean":"2024-04-02T14:34:52.548042+0000","last_became_active":"2024-04-02T14:32:22.555686+0000","last_became_peered":"2024-04-02T14:32:22.555686+0000","last_unstale":"2024-04-02T14:34:52.548042+0000","last_undegraded":"2024-04-02T14:34:52.548042+0000","last_fullsized":"2024-04-02T14:34:52.548042+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:06:40.161203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"90'8","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.247614+0000","last_change":"2024-04-02T14:32:22.537830+0000","last_active":"2024-04-02T14:34:52.247614+0000","last_peered":"2024-04-02T14:34:52.247614+0000","last_clean":"2024-04-02T14:34:52.247614+0000","last_became_active":"2024-04-02T14:32:22.536860+0000","last_became_peered":"2024-04-02T14:32:22.536860+0000","last_unstale":"2024-04-02T14:34:52.247614+0000","last_undegraded":"2024-04-02T14:34:52.247614+0000","last_fullsized":"2024-04-02T14:34:52.247614+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T01:07:57.244177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"90'4","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.247802+0000","last_change":"2024-04-02T14:32:22.549550+0000","last_active":"2024-04-02T14:34:52.247802+0000","last_peered":"2024-04-02T14:34:52.247802+0000","last_clean":"2024-04-02T14:34:52.247802+0000","last_became_active":"2024-04-02T14:32:22.549177+0000","last_became_peered":"2024-04-02T14:32:22.549177+0000","last_unstale":"2024-04-02T14:34:52.247802+0000","last_undegraded":"2024-04-02T14:34:52.247802+0000","last_fullsized":"2024-04-02T14:34:52.247802+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:31:49.175583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"90'6","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.380610+0000","last_change":"2024-04-02T14:32:35.849338+0000","last_active":"2024-04-02T14:34:52.380610+0000","last_peered":"2024-04-02T14:34:52.380610+0000","last_clean":"2024-04-02T14:34:52.380610+0000","last_became_active":"2024-04-02T14:32:35.848929+0000","last_became_peered":"2024-04-02T14:32:35.848929+0000","last_unstale":"2024-04-02T14:34:52.380610+0000","last_undegraded":"2024-04-02T14:34:52.380610+0000","last_fullsized":"2024-04-02T14:34:52.380610+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:55:58.614290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,6],"acting":[10,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"90'5","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.543923+0000","last_change":"2024-04-02T14:32:22.546861+0000","last_active":"2024-04-02T14:34:52.543923+0000","last_peered":"2024-04-02T14:34:52.543923+0000","last_clean":"2024-04-02T14:34:52.543923+0000","last_became_active":"2024-04-02T14:32:22.546477+0000","last_became_peered":"2024-04-02T14:32:22.546477+0000","last_unstale":"2024-04-02T14:34:52.543923+0000","last_undegraded":"2024-04-02T14:34:52.543923+0000","last_fullsized":"2024-04-02T14:34:52.543923+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:59:17.133714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"90'9","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.246514+0000","last_change":"2024-04-02T14:32:22.554287+0000","last_active":"2024-04-02T14:34:52.246514+0000","last_peered":"2024-04-02T14:34:52.246514+0000","last_clean":"2024-04-02T14:34:52.246514+0000","last_became_active":"2024-04-02T14:32:22.553491+0000","last_became_peered":"2024-04-02T14:32:22.553491+0000","last_unstale":"2024-04-02T14:34:52.246514+0000","last_undegraded":"2024-04-02T14:34:52.246514+0000","last_fullsized":"2024-04-02T14:34:52.246514+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:51:30.785380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"90'11","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.546344+0000","last_change":"2024-04-02T14:32:22.538158+0000","last_active":"2024-04-02T14:34:52.546344+0000","last_peered":"2024-04-02T14:34:52.546344+0000","last_clean":"2024-04-02T14:34:52.546344+0000","last_became_active":"2024-04-02T14:32:22.537796+0000","last_became_peered":"2024-04-02T14:32:22.537796+0000","last_unstale":"2024-04-02T14:34:52.546344+0000","last_undegraded":"2024-04-02T14:34:52.546344+0000","last_fullsized":"2024-04-02T14:34:52.546344+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:09:48.054965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"90'16","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.381066+0000","last_change":"2024-04-02T14:32:22.535225+0000","last_active":"2024-04-02T14:34:52.381066+0000","last_peered":"2024-04-02T14:34:52.381066+0000","last_clean":"2024-04-02T14:34:52.381066+0000","last_became_active":"2024-04-02T14:32:22.534856+0000","last_became_peered":"2024-04-02T14:32:22.534856+0000","last_unstale":"2024-04-02T14:34:52.381066+0000","last_undegraded":"2024-04-02T14:34:52.381066+0000","last_fullsized":"2024-04-02T14:34:52.381066+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:22:11.118918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"90'6","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.545055+0000","last_change":"2024-04-02T14:32:22.539138+0000","last_active":"2024-04-02T14:34:52.545055+0000","last_peered":"2024-04-02T14:34:52.545055+0000","last_clean":"2024-04-02T14:34:52.545055+0000","last_became_active":"2024-04-02T14:32:22.538762+0000","last_became_peered":"2024-04-02T14:32:22.538762+0000","last_unstale":"2024-04-02T14:34:52.545055+0000","last_undegraded":"2024-04-02T14:34:52.545055+0000","last_fullsized":"2024-04-02T14:34:52.545055+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:18:59.174339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"90'7","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.247639+0000","last_change":"2024-04-02T14:32:22.536383+0000","last_active":"2024-04-02T14:34:52.247639+0000","last_peered":"2024-04-02T14:34:52.247639+0000","last_clean":"2024-04-02T14:34:52.247639+0000","last_became_active":"2024-04-02T14:32:22.536043+0000","last_became_peered":"2024-04-02T14:32:22.536043+0000","last_unstale":"2024-04-02T14:34:52.247639+0000","last_undegraded":"2024-04-02T14:34:52.247639+0000","last_fullsized":"2024-04-02T14:34:52.247639+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:14:48.628684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"90'10","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.545677+0000","last_change":"2024-04-02T14:32:22.552512+0000","last_active":"2024-04-02T14:34:52.545677+0000","last_peered":"2024-04-02T14:34:52.545677+0000","last_clean":"2024-04-02T14:34:52.545677+0000","last_became_active":"2024-04-02T14:32:22.549502+0000","last_became_peered":"2024-04-02T14:32:22.549502+0000","last_unstale":"2024-04-02T14:34:52.545677+0000","last_undegraded":"2024-04-02T14:34:52.545677+0000","last_fullsized":"2024-04-02T14:34:52.545677+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:37:45.609280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"90'7","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.547684+0000","last_change":"2024-04-02T14:32:22.555074+0000","last_active":"2024-04-02T14:34:52.547684+0000","last_peered":"2024-04-02T14:34:52.547684+0000","last_clean":"2024-04-02T14:34:52.547684+0000","last_became_active":"2024-04-02T14:32:22.554422+0000","last_became_peered":"2024-04-02T14:32:22.554422+0000","last_unstale":"2024-04-02T14:34:52.547684+0000","last_undegraded":"2024-04-02T14:34:52.547684+0000","last_fullsized":"2024-04-02T14:34:52.547684+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:27:49.536941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"90'7","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.547699+0000","last_change":"2024-04-02T14:32:22.550322+0000","last_active":"2024-04-02T14:34:52.547699+0000","last_peered":"2024-04-02T14:34:52.547699+0000","last_clean":"2024-04-02T14:34:52.547699+0000","last_became_active":"2024-04-02T14:32:22.548710+0000","last_became_peered":"2024-04-02T14:32:22.548710+0000","last_unstale":"2024-04-02T14:34:52.547699+0000","last_undegraded":"2024-04-02T14:34:52.547699+0000","last_fullsized":"2024-04-02T14:34:52.547699+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:19:09.226556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"90'9","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.545820+0000","last_change":"2024-04-02T14:32:22.558705+0000","last_active":"2024-04-02T14:34:52.545820+0000","last_peered":"2024-04-02T14:34:52.545820+0000","last_clean":"2024-04-02T14:34:52.545820+0000","last_became_active":"2024-04-02T14:32:22.558309+0000","last_became_peered":"2024-04-02T14:32:22.558309+0000","last_unstale":"2024-04-02T14:34:52.545820+0000","last_undegraded":"2024-04-02T14:34:52.545820+0000","last_fullsized":"2024-04-02T14:34:52.545820+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:52:23.929627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"90'8","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.549815+0000","last_change":"2024-04-02T14:32:22.551211+0000","last_active":"2024-04-02T14:34:52.549815+0000","last_peered":"2024-04-02T14:34:52.549815+0000","last_clean":"2024-04-02T14:34:52.549815+0000","last_became_active":"2024-04-02T14:32:22.550882+0000","last_became_peered":"2024-04-02T14:32:22.550882+0000","last_unstale":"2024-04-02T14:34:52.549815+0000","last_undegraded":"2024-04-02T14:34:52.549815+0000","last_fullsized":"2024-04-02T14:34:52.549815+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:54:54.533063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"90'10","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.368148+0000","last_change":"2024-04-02T14:32:22.550692+0000","last_active":"2024-04-02T14:34:52.368148+0000","last_peered":"2024-04-02T14:34:52.368148+0000","last_clean":"2024-04-02T14:34:52.368148+0000","last_became_active":"2024-04-02T14:32:22.550292+0000","last_became_peered":"2024-04-02T14:32:22.550292+0000","last_unstale":"2024-04-02T14:34:52.368148+0000","last_undegraded":"2024-04-02T14:34:52.368148+0000","last_fullsized":"2024-04-02T14:34:52.368148+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:14:42.317649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"90'7","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.283769+0000","last_change":"2024-04-02T14:32:22.553042+0000","last_active":"2024-04-02T14:34:52.283769+0000","last_peered":"2024-04-02T14:34:52.283769+0000","last_clean":"2024-04-02T14:34:52.283769+0000","last_became_active":"2024-04-02T14:32:22.552010+0000","last_became_peered":"2024-04-02T14:32:22.552010+0000","last_unstale":"2024-04-02T14:34:52.283769+0000","last_undegraded":"2024-04-02T14:34:52.283769+0000","last_fullsized":"2024-04-02T14:34:52.283769+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:31:13.947662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"88'13","reported_seq":199,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.367999+0000","last_change":"2024-04-02T14:32:35.868597+0000","last_active":"2024-04-02T14:34:52.367999+0000","last_peered":"2024-04-02T14:34:52.367999+0000","last_clean":"2024-04-02T14:34:52.367999+0000","last_became_active":"2024-04-02T14:32:35.851962+0000","last_became_peered":"2024-04-02T14:32:35.851962+0000","last_unstale":"2024-04-02T14:34:52.367999+0000","last_undegraded":"2024-04-02T14:34:52.367999+0000","last_fullsized":"2024-04-02T14:34:52.367999+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:17:06.400744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":140,"num_read_kb":140,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":672,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,4],"acting":[9,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"90'5","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.247056+0000","last_change":"2024-04-02T14:32:22.549526+0000","last_active":"2024-04-02T14:34:52.247056+0000","last_peered":"2024-04-02T14:34:52.247056+0000","last_clean":"2024-04-02T14:34:52.247056+0000","last_became_active":"2024-04-02T14:32:22.548250+0000","last_became_peered":"2024-04-02T14:32:22.548250+0000","last_unstale":"2024-04-02T14:34:52.247056+0000","last_undegraded":"2024-04-02T14:34:52.247056+0000","last_fullsized":"2024-04-02T14:34:52.247056+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:35:34.866803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"88'2","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.247157+0000","last_change":"2024-04-02T14:32:21.865084+0000","last_active":"2024-04-02T14:34:52.247157+0000","last_peered":"2024-04-02T14:34:52.247157+0000","last_clean":"2024-04-02T14:34:52.247157+0000","last_became_active":"2024-04-02T14:32:21.864718+0000","last_became_peered":"2024-04-02T14:32:21.864718+0000","last_unstale":"2024-04-02T14:34:52.247157+0000","last_undegraded":"2024-04-02T14:34:52.247157+0000","last_fullsized":"2024-04-02T14:34:52.247157+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:10:50.611362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"90'6","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.549008+0000","last_change":"2024-04-02T14:32:22.549443+0000","last_active":"2024-04-02T14:34:52.549008+0000","last_peered":"2024-04-02T14:34:52.549008+0000","last_clean":"2024-04-02T14:34:52.549008+0000","last_became_active":"2024-04-02T14:32:22.549059+0000","last_became_peered":"2024-04-02T14:32:22.549059+0000","last_unstale":"2024-04-02T14:34:52.549008+0000","last_undegraded":"2024-04-02T14:34:52.549008+0000","last_fullsized":"2024-04-02T14:34:52.549008+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:04:21.489864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"88'12","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.248319+0000","last_change":"2024-04-02T14:32:21.841368+0000","last_active":"2024-04-02T14:34:52.248319+0000","last_peered":"2024-04-02T14:34:52.248319+0000","last_clean":"2024-04-02T14:34:52.248319+0000","last_became_active":"2024-04-02T14:32:21.840963+0000","last_became_peered":"2024-04-02T14:32:21.840963+0000","last_unstale":"2024-04-02T14:34:52.248319+0000","last_undegraded":"2024-04-02T14:34:52.248319+0000","last_fullsized":"2024-04-02T14:34:52.248319+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:36:40.549720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"90'9","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.546094+0000","last_change":"2024-04-02T14:32:22.549753+0000","last_active":"2024-04-02T14:34:52.546094+0000","last_peered":"2024-04-02T14:34:52.546094+0000","last_clean":"2024-04-02T14:34:52.546094+0000","last_became_active":"2024-04-02T14:32:22.549354+0000","last_became_peered":"2024-04-02T14:32:22.549354+0000","last_unstale":"2024-04-02T14:34:52.546094+0000","last_undegraded":"2024-04-02T14:34:52.546094+0000","last_fullsized":"2024-04-02T14:34:52.546094+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T02:17:09.650946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"88'5","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.248127+0000","last_change":"2024-04-02T14:32:21.842766+0000","last_active":"2024-04-02T14:34:52.248127+0000","last_peered":"2024-04-02T14:34:52.248127+0000","last_clean":"2024-04-02T14:34:52.248127+0000","last_became_active":"2024-04-02T14:32:21.842410+0000","last_became_peered":"2024-04-02T14:32:21.842410+0000","last_unstale":"2024-04-02T14:34:52.248127+0000","last_undegraded":"2024-04-02T14:34:52.248127+0000","last_fullsized":"2024-04-02T14:34:52.248127+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:49:31.651127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"90'15","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.377579+0000","last_change":"2024-04-02T14:32:22.538506+0000","last_active":"2024-04-02T14:34:52.377579+0000","last_peered":"2024-04-02T14:34:52.377579+0000","last_clean":"2024-04-02T14:34:52.377579+0000","last_became_active":"2024-04-02T14:32:22.538168+0000","last_became_peered":"2024-04-02T14:32:22.538168+0000","last_unstale":"2024-04-02T14:34:52.377579+0000","last_undegraded":"2024-04-02T14:34:52.377579+0000","last_fullsized":"2024-04-02T14:34:52.377579+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:30:04.746382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"90'7","reported_seq":176,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:53.040014+0000","last_change":"2024-04-02T14:32:21.844180+0000","last_active":"2024-04-02T14:34:53.040014+0000","last_peered":"2024-04-02T14:34:53.040014+0000","last_clean":"2024-04-02T14:34:53.040014+0000","last_became_active":"2024-04-02T14:32:21.843840+0000","last_became_peered":"2024-04-02T14:32:21.843840+0000","last_unstale":"2024-04-02T14:34:53.040014+0000","last_undegraded":"2024-04-02T14:34:53.040014+0000","last_fullsized":"2024-04-02T14:34:53.040014+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T01:56:10.767946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1154,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":147,"num_read_kb":3,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"90'7","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.247921+0000","last_change":"2024-04-02T14:32:22.548453+0000","last_active":"2024-04-02T14:34:52.247921+0000","last_peered":"2024-04-02T14:34:52.247921+0000","last_clean":"2024-04-02T14:34:52.247921+0000","last_became_active":"2024-04-02T14:32:22.548110+0000","last_became_peered":"2024-04-02T14:32:22.548110+0000","last_unstale":"2024-04-02T14:34:52.247921+0000","last_undegraded":"2024-04-02T14:34:52.247921+0000","last_fullsized":"2024-04-02T14:34:52.247921+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:57:13.067425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"88'7","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.380564+0000","last_change":"2024-04-02T14:32:21.846528+0000","last_active":"2024-04-02T14:34:52.380564+0000","last_peered":"2024-04-02T14:34:52.380564+0000","last_clean":"2024-04-02T14:34:52.380564+0000","last_became_active":"2024-04-02T14:32:21.846152+0000","last_became_peered":"2024-04-02T14:32:21.846152+0000","last_unstale":"2024-04-02T14:34:52.380564+0000","last_undegraded":"2024-04-02T14:34:52.380564+0000","last_fullsized":"2024-04-02T14:34:52.380564+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:58:09.286256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"90'8","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.377374+0000","last_change":"2024-04-02T14:32:22.546730+0000","last_active":"2024-04-02T14:34:52.377374+0000","last_peered":"2024-04-02T14:34:52.377374+0000","last_clean":"2024-04-02T14:34:52.377374+0000","last_became_active":"2024-04-02T14:32:22.546342+0000","last_became_peered":"2024-04-02T14:32:22.546342+0000","last_unstale":"2024-04-02T14:34:52.377374+0000","last_undegraded":"2024-04-02T14:34:52.377374+0000","last_fullsized":"2024-04-02T14:34:52.377374+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:38:37.616382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"88'3","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.545031+0000","last_change":"2024-04-02T14:32:21.853316+0000","last_active":"2024-04-02T14:34:52.545031+0000","last_peered":"2024-04-02T14:34:52.545031+0000","last_clean":"2024-04-02T14:34:52.545031+0000","last_became_active":"2024-04-02T14:32:21.852912+0000","last_became_peered":"2024-04-02T14:32:21.852912+0000","last_unstale":"2024-04-02T14:34:52.545031+0000","last_undegraded":"2024-04-02T14:34:52.545031+0000","last_fullsized":"2024-04-02T14:34:52.545031+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T02:30:44.601900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"90'8","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.246607+0000","last_change":"2024-04-02T14:32:22.558913+0000","last_active":"2024-04-02T14:34:52.246607+0000","last_peered":"2024-04-02T14:34:52.246607+0000","last_clean":"2024-04-02T14:34:52.246607+0000","last_became_active":"2024-04-02T14:32:22.558511+0000","last_became_peered":"2024-04-02T14:32:22.558511+0000","last_unstale":"2024-04-02T14:34:52.246607+0000","last_undegraded":"2024-04-02T14:34:52.246607+0000","last_fullsized":"2024-04-02T14:34:52.246607+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:48:36.449086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"88'3","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.377293+0000","last_change":"2024-04-02T14:32:21.844081+0000","last_active":"2024-04-02T14:34:52.377293+0000","last_peered":"2024-04-02T14:34:52.377293+0000","last_clean":"2024-04-02T14:34:52.377293+0000","last_became_active":"2024-04-02T14:32:21.843788+0000","last_became_peered":"2024-04-02T14:32:21.843788+0000","last_unstale":"2024-04-02T14:34:52.377293+0000","last_undegraded":"2024-04-02T14:34:52.377293+0000","last_fullsized":"2024-04-02T14:34:52.377293+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:19:02.184229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"90'11","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.367476+0000","last_change":"2024-04-02T14:32:22.556193+0000","last_active":"2024-04-02T14:34:52.367476+0000","last_peered":"2024-04-02T14:34:52.367476+0000","last_clean":"2024-04-02T14:34:52.367476+0000","last_became_active":"2024-04-02T14:32:22.555787+0000","last_became_peered":"2024-04-02T14:32:22.555787+0000","last_unstale":"2024-04-02T14:34:52.367476+0000","last_undegraded":"2024-04-02T14:34:52.367476+0000","last_fullsized":"2024-04-02T14:34:52.367476+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T02:02:13.730941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"91'6","reported_seq":316,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:53.039409+0000","last_change":"2024-04-02T14:32:21.840651+0000","last_active":"2024-04-02T14:34:53.039409+0000","last_peered":"2024-04-02T14:34:53.039409+0000","last_clean":"2024-04-02T14:34:53.039409+0000","last_became_active":"2024-04-02T14:32:21.840344+0000","last_became_peered":"2024-04-02T14:32:21.840344+0000","last_unstale":"2024-04-02T14:34:53.039409+0000","last_undegraded":"2024-04-02T14:34:53.039409+0000","last_fullsized":"2024-04-02T14:34:53.039409+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:49:22.384224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":288,"num_read_kb":288,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"90'10","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.546191+0000","last_change":"2024-04-02T14:32:22.548245+0000","last_active":"2024-04-02T14:34:52.546191+0000","last_peered":"2024-04-02T14:34:52.546191+0000","last_clean":"2024-04-02T14:34:52.546191+0000","last_became_active":"2024-04-02T14:32:22.547853+0000","last_became_peered":"2024-04-02T14:32:22.547853+0000","last_unstale":"2024-04-02T14:34:52.546191+0000","last_undegraded":"2024-04-02T14:34:52.546191+0000","last_fullsized":"2024-04-02T14:34:52.546191+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:02:43.036944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"88'4","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.544349+0000","last_change":"2024-04-02T14:32:21.842096+0000","last_active":"2024-04-02T14:34:52.544349+0000","last_peered":"2024-04-02T14:34:52.544349+0000","last_clean":"2024-04-02T14:34:52.544349+0000","last_became_active":"2024-04-02T14:32:21.841646+0000","last_became_peered":"2024-04-02T14:32:21.841646+0000","last_unstale":"2024-04-02T14:34:52.544349+0000","last_undegraded":"2024-04-02T14:34:52.544349+0000","last_fullsized":"2024-04-02T14:34:52.544349+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:04:08.149785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"90'9","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.548949+0000","last_change":"2024-04-02T14:32:22.557694+0000","last_active":"2024-04-02T14:34:52.548949+0000","last_peered":"2024-04-02T14:34:52.548949+0000","last_clean":"2024-04-02T14:34:52.548949+0000","last_became_active":"2024-04-02T14:32:22.557202+0000","last_became_peered":"2024-04-02T14:32:22.557202+0000","last_unstale":"2024-04-02T14:34:52.548949+0000","last_undegraded":"2024-04-02T14:34:52.548949+0000","last_fullsized":"2024-04-02T14:34:52.548949+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:35:41.471557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"88'5","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.377599+0000","last_change":"2024-04-02T14:32:21.864366+0000","last_active":"2024-04-02T14:34:52.377599+0000","last_peered":"2024-04-02T14:34:52.377599+0000","last_clean":"2024-04-02T14:34:52.377599+0000","last_became_active":"2024-04-02T14:32:21.864006+0000","last_became_peered":"2024-04-02T14:32:21.864006+0000","last_unstale":"2024-04-02T14:34:52.377599+0000","last_undegraded":"2024-04-02T14:34:52.377599+0000","last_fullsized":"2024-04-02T14:34:52.377599+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:08:43.176061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"90'11","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.546034+0000","last_change":"2024-04-02T14:32:22.541562+0000","last_active":"2024-04-02T14:34:52.546034+0000","last_peered":"2024-04-02T14:34:52.546034+0000","last_clean":"2024-04-02T14:34:52.546034+0000","last_became_active":"2024-04-02T14:32:22.541209+0000","last_became_peered":"2024-04-02T14:32:22.541209+0000","last_unstale":"2024-04-02T14:34:52.546034+0000","last_undegraded":"2024-04-02T14:34:52.546034+0000","last_fullsized":"2024-04-02T14:34:52.546034+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:49:40.310528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"91'38","reported_seq":376,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:53.062247+0000","last_change":"2024-04-02T14:32:35.875429+0000","last_active":"2024-04-02T14:34:53.062247+0000","last_peered":"2024-04-02T14:34:53.062247+0000","last_clean":"2024-04-02T14:34:53.062247+0000","last_became_active":"2024-04-02T14:32:35.852417+0000","last_became_peered":"2024-04-02T14:32:35.852417+0000","last_unstale":"2024-04-02T14:34:53.062247+0000","last_undegraded":"2024-04-02T14:34:53.062247+0000","last_fullsized":"2024-04-02T14:34:53.062247+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:33:53.903712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67756,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":289,"num_read_kb":205,"num_write":40,"num_write_kb":82,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":43377,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,4],"acting":[10,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"90'6","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.248005+0000","last_change":"2024-04-02T14:32:22.535716+0000","last_active":"2024-04-02T14:34:52.248005+0000","last_peered":"2024-04-02T14:34:52.248005+0000","last_clean":"2024-04-02T14:34:52.248005+0000","last_became_active":"2024-04-02T14:32:22.534738+0000","last_became_peered":"2024-04-02T14:32:22.534738+0000","last_unstale":"2024-04-02T14:34:52.248005+0000","last_undegraded":"2024-04-02T14:34:52.248005+0000","last_fullsized":"2024-04-02T14:34:52.248005+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T01:25:07.763915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"88'9","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.283698+0000","last_change":"2024-04-02T14:32:21.854582+0000","last_active":"2024-04-02T14:34:52.283698+0000","last_peered":"2024-04-02T14:34:52.283698+0000","last_clean":"2024-04-02T14:34:52.283698+0000","last_became_active":"2024-04-02T14:32:21.854226+0000","last_became_peered":"2024-04-02T14:32:21.854226+0000","last_unstale":"2024-04-02T14:34:52.283698+0000","last_undegraded":"2024-04-02T14:34:52.283698+0000","last_fullsized":"2024-04-02T14:34:52.283698+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:30:02.289163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"90'9","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.545367+0000","last_change":"2024-04-02T14:32:22.555901+0000","last_active":"2024-04-02T14:34:52.545367+0000","last_peered":"2024-04-02T14:34:52.545367+0000","last_clean":"2024-04-02T14:34:52.545367+0000","last_became_active":"2024-04-02T14:32:22.555158+0000","last_became_peered":"2024-04-02T14:32:22.555158+0000","last_unstale":"2024-04-02T14:34:52.545367+0000","last_undegraded":"2024-04-02T14:34:52.545367+0000","last_fullsized":"2024-04-02T14:34:52.545367+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:04:28.766185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"88'5","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.377201+0000","last_change":"2024-04-02T14:32:21.855982+0000","last_active":"2024-04-02T14:34:52.377201+0000","last_peered":"2024-04-02T14:34:52.377201+0000","last_clean":"2024-04-02T14:34:52.377201+0000","last_became_active":"2024-04-02T14:32:21.855591+0000","last_became_peered":"2024-04-02T14:32:21.855591+0000","last_unstale":"2024-04-02T14:34:52.377201+0000","last_undegraded":"2024-04-02T14:34:52.377201+0000","last_fullsized":"2024-04-02T14:34:52.377201+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:28:51.806779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"90'15","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.544517+0000","last_change":"2024-04-02T14:32:22.549889+0000","last_active":"2024-04-02T14:34:52.544517+0000","last_peered":"2024-04-02T14:34:52.544517+0000","last_clean":"2024-04-02T14:34:52.544517+0000","last_became_active":"2024-04-02T14:32:22.549479+0000","last_became_peered":"2024-04-02T14:32:22.549479+0000","last_unstale":"2024-04-02T14:34:52.544517+0000","last_undegraded":"2024-04-02T14:34:52.544517+0000","last_fullsized":"2024-04-02T14:34:52.544517+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:29:25.937576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"91'9","reported_seq":179,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:53.156649+0000","last_change":"2024-04-02T14:32:21.838165+0000","last_active":"2024-04-02T14:34:53.156649+0000","last_peered":"2024-04-02T14:34:53.156649+0000","last_clean":"2024-04-02T14:34:53.156649+0000","last_became_active":"2024-04-02T14:32:21.837808+0000","last_became_peered":"2024-04-02T14:32:21.837808+0000","last_unstale":"2024-04-02T14:34:53.156649+0000","last_undegraded":"2024-04-02T14:34:53.156649+0000","last_fullsized":"2024-04-02T14:34:53.156649+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:00:44.605314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1902,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":149,"num_read_kb":5,"num_write":9,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"90'6","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.543871+0000","last_change":"2024-04-02T14:32:22.532937+0000","last_active":"2024-04-02T14:34:52.543871+0000","last_peered":"2024-04-02T14:34:52.543871+0000","last_clean":"2024-04-02T14:34:52.543871+0000","last_became_active":"2024-04-02T14:32:22.532586+0000","last_became_peered":"2024-04-02T14:32:22.532586+0000","last_unstale":"2024-04-02T14:34:52.543871+0000","last_undegraded":"2024-04-02T14:34:52.543871+0000","last_fullsized":"2024-04-02T14:34:52.543871+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:50:48.030889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"88'17","reported_seq":40,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.247642+0000","last_change":"2024-04-02T14:32:21.860032+0000","last_active":"2024-04-02T14:34:52.247642+0000","last_peered":"2024-04-02T14:34:52.247642+0000","last_clean":"2024-04-02T14:34:52.247642+0000","last_became_active":"2024-04-02T14:32:21.859653+0000","last_became_peered":"2024-04-02T14:32:21.859653+0000","last_unstale":"2024-04-02T14:34:52.247642+0000","last_undegraded":"2024-04-02T14:34:52.247642+0000","last_fullsized":"2024-04-02T14:34:52.247642+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:06:01.144530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"90'6","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.367598+0000","last_change":"2024-04-02T14:32:35.736133+0000","last_active":"2024-04-02T14:34:52.367598+0000","last_peered":"2024-04-02T14:34:52.367598+0000","last_clean":"2024-04-02T14:34:52.367598+0000","last_became_active":"2024-04-02T14:32:35.734335+0000","last_became_peered":"2024-04-02T14:32:35.734335+0000","last_unstale":"2024-04-02T14:34:52.367598+0000","last_undegraded":"2024-04-02T14:34:52.367598+0000","last_fullsized":"2024-04-02T14:34:52.367598+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:36:21.828318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,1],"acting":[9,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.f","version":"91'3","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.248316+0000","last_change":"2024-04-02T14:32:21.842384+0000","last_active":"2024-04-02T14:34:52.248316+0000","last_peered":"2024-04-02T14:34:52.248316+0000","last_clean":"2024-04-02T14:34:52.248316+0000","last_became_active":"2024-04-02T14:32:21.842040+0000","last_became_peered":"2024-04-02T14:32:21.842040+0000","last_unstale":"2024-04-02T14:34:52.248316+0000","last_undegraded":"2024-04-02T14:34:52.248316+0000","last_fullsized":"2024-04-02T14:34:52.248316+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:33:29.541972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"90'5","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.549637+0000","last_change":"2024-04-02T14:32:22.538571+0000","last_active":"2024-04-02T14:34:52.549637+0000","last_peered":"2024-04-02T14:34:52.549637+0000","last_clean":"2024-04-02T14:34:52.549637+0000","last_became_active":"2024-04-02T14:32:22.538239+0000","last_became_peered":"2024-04-02T14:32:22.538239+0000","last_unstale":"2024-04-02T14:34:52.549637+0000","last_undegraded":"2024-04-02T14:34:52.549637+0000","last_fullsized":"2024-04-02T14:34:52.549637+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T01:19:53.120419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"88'1","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.367816+0000","last_change":"2024-04-02T14:32:21.857666+0000","last_active":"2024-04-02T14:34:52.367816+0000","last_peered":"2024-04-02T14:34:52.367816+0000","last_clean":"2024-04-02T14:34:52.367816+0000","last_became_active":"2024-04-02T14:32:21.856618+0000","last_became_peered":"2024-04-02T14:32:21.856618+0000","last_unstale":"2024-04-02T14:34:52.367816+0000","last_undegraded":"2024-04-02T14:34:52.367816+0000","last_fullsized":"2024-04-02T14:34:52.367816+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T01:20:53.183052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"90'5","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.380890+0000","last_change":"2024-04-02T14:32:22.552277+0000","last_active":"2024-04-02T14:34:52.380890+0000","last_peered":"2024-04-02T14:34:52.380890+0000","last_clean":"2024-04-02T14:34:52.380890+0000","last_became_active":"2024-04-02T14:32:22.551882+0000","last_became_peered":"2024-04-02T14:32:22.551882+0000","last_unstale":"2024-04-02T14:34:52.380890+0000","last_undegraded":"2024-04-02T14:34:52.380890+0000","last_fullsized":"2024-04-02T14:34:52.380890+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:30:43.592910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"88'5","reported_seq":171,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.544165+0000","last_change":"2024-04-02T14:32:21.848258+0000","last_active":"2024-04-02T14:34:52.544165+0000","last_peered":"2024-04-02T14:34:52.544165+0000","last_clean":"2024-04-02T14:34:52.544165+0000","last_became_active":"2024-04-02T14:32:21.847805+0000","last_became_peered":"2024-04-02T14:32:21.847805+0000","last_unstale":"2024-04-02T14:34:52.544165+0000","last_undegraded":"2024-04-02T14:34:52.544165+0000","last_fullsized":"2024-04-02T14:34:52.544165+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:46:07.552786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":144,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"90'7","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.380644+0000","last_change":"2024-04-02T14:32:22.560835+0000","last_active":"2024-04-02T14:34:52.380644+0000","last_peered":"2024-04-02T14:34:52.380644+0000","last_clean":"2024-04-02T14:34:52.380644+0000","last_became_active":"2024-04-02T14:32:22.560469+0000","last_became_peered":"2024-04-02T14:32:22.560469+0000","last_unstale":"2024-04-02T14:34:52.380644+0000","last_undegraded":"2024-04-02T14:34:52.380644+0000","last_fullsized":"2024-04-02T14:34:52.380644+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T02:29:24.869650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"90'8","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.377408+0000","last_change":"2024-04-02T14:32:22.538325+0000","last_active":"2024-04-02T14:34:52.377408+0000","last_peered":"2024-04-02T14:34:52.377408+0000","last_clean":"2024-04-02T14:34:52.377408+0000","last_became_active":"2024-04-02T14:32:22.537903+0000","last_became_peered":"2024-04-02T14:32:22.537903+0000","last_unstale":"2024-04-02T14:34:52.377408+0000","last_undegraded":"2024-04-02T14:34:52.377408+0000","last_fullsized":"2024-04-02T14:34:52.377408+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:12:26.586290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"91'7","reported_seq":174,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.549482+0000","last_change":"2024-04-02T14:32:21.861333+0000","last_active":"2024-04-02T14:34:52.549482+0000","last_peered":"2024-04-02T14:34:52.549482+0000","last_clean":"2024-04-02T14:34:52.549482+0000","last_became_active":"2024-04-02T14:32:21.860909+0000","last_became_peered":"2024-04-02T14:32:21.860909+0000","last_unstale":"2024-04-02T14:34:52.549482+0000","last_undegraded":"2024-04-02T14:34:52.549482+0000","last_fullsized":"2024-04-02T14:34:52.549482+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:21:52.616036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":145,"num_read_kb":145,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"90'8","reported_seq":22,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.367541+0000","last_change":"2024-04-02T14:32:35.736379+0000","last_active":"2024-04-02T14:34:52.367541+0000","last_peered":"2024-04-02T14:34:52.367541+0000","last_clean":"2024-04-02T14:34:52.367541+0000","last_became_active":"2024-04-02T14:32:35.735156+0000","last_became_peered":"2024-04-02T14:32:35.735156+0000","last_unstale":"2024-04-02T14:34:52.367541+0000","last_undegraded":"2024-04-02T14:34:52.367541+0000","last_fullsized":"2024-04-02T14:34:52.367541+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:16:28.405699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,1],"acting":[9,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.3","version":"88'6","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.247327+0000","last_change":"2024-04-02T14:32:21.837336+0000","last_active":"2024-04-02T14:34:52.247327+0000","last_peered":"2024-04-02T14:34:52.247327+0000","last_clean":"2024-04-02T14:34:52.247327+0000","last_became_active":"2024-04-02T14:32:21.836921+0000","last_became_peered":"2024-04-02T14:32:21.836921+0000","last_unstale":"2024-04-02T14:34:52.247327+0000","last_undegraded":"2024-04-02T14:34:52.247327+0000","last_fullsized":"2024-04-02T14:34:52.247327+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:34:52.883173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":358,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.543898+0000","last_change":"2024-04-02T14:31:02.710356+0000","last_active":"2024-04-02T14:34:52.543898+0000","last_peered":"2024-04-02T14:34:52.543898+0000","last_clean":"2024-04-02T14:34:52.543898+0000","last_became_active":"2024-04-02T14:31:02.709669+0000","last_became_peered":"2024-04-02T14:31:02.709669+0000","last_unstale":"2024-04-02T14:34:52.543898+0000","last_undegraded":"2024-04-02T14:34:52.543898+0000","last_fullsized":"2024-04-02T14:34:52.543898+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:27:17.358883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:27:17.358883+0000","last_clean_scrub_stamp":"2024-04-02T14:27:17.358883+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:45:55.893284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"90'5","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.544201+0000","last_change":"2024-04-02T14:32:22.545502+0000","last_active":"2024-04-02T14:34:52.544201+0000","last_peered":"2024-04-02T14:34:52.544201+0000","last_clean":"2024-04-02T14:34:52.544201+0000","last_became_active":"2024-04-02T14:32:22.545111+0000","last_became_peered":"2024-04-02T14:32:22.545111+0000","last_unstale":"2024-04-02T14:34:52.544201+0000","last_undegraded":"2024-04-02T14:34:52.544201+0000","last_fullsized":"2024-04-02T14:34:52.544201+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:02:32.178334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"88'4","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.549305+0000","last_change":"2024-04-02T14:32:35.736582+0000","last_active":"2024-04-02T14:34:52.549305+0000","last_peered":"2024-04-02T14:34:52.549305+0000","last_clean":"2024-04-02T14:34:52.549305+0000","last_became_active":"2024-04-02T14:32:35.733368+0000","last_became_peered":"2024-04-02T14:32:35.733368+0000","last_unstale":"2024-04-02T14:34:52.549305+0000","last_undegraded":"2024-04-02T14:34:52.549305+0000","last_fullsized":"2024-04-02T14:34:52.549305+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T02:14:35.713351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,8],"acting":[11,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"88'4","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.248107+0000","last_change":"2024-04-02T14:32:21.860226+0000","last_active":"2024-04-02T14:34:52.248107+0000","last_peered":"2024-04-02T14:34:52.248107+0000","last_clean":"2024-04-02T14:34:52.248107+0000","last_became_active":"2024-04-02T14:32:21.859854+0000","last_became_peered":"2024-04-02T14:32:21.859854+0000","last_unstale":"2024-04-02T14:34:52.248107+0000","last_undegraded":"2024-04-02T14:34:52.248107+0000","last_fullsized":"2024-04-02T14:34:52.248107+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:30:36.710677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"90'9","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.545807+0000","last_change":"2024-04-02T14:32:22.552287+0000","last_active":"2024-04-02T14:34:52.545807+0000","last_peered":"2024-04-02T14:34:52.545807+0000","last_clean":"2024-04-02T14:34:52.545807+0000","last_became_active":"2024-04-02T14:32:22.551719+0000","last_became_peered":"2024-04-02T14:32:22.551719+0000","last_unstale":"2024-04-02T14:34:52.545807+0000","last_undegraded":"2024-04-02T14:34:52.545807+0000","last_fullsized":"2024-04-02T14:34:52.545807+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:26:12.007600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"91'50","reported_seq":95,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.507926+0000","last_change":"2024-04-02T14:32:21.851671+0000","last_active":"2024-04-02T14:34:52.507926+0000","last_peered":"2024-04-02T14:34:52.507926+0000","last_clean":"2024-04-02T14:34:52.507926+0000","last_became_active":"2024-04-02T14:32:21.851297+0000","last_became_peered":"2024-04-02T14:32:21.851297+0000","last_unstale":"2024-04-02T14:34:52.507926+0000","last_undegraded":"2024-04-02T14:34:52.507926+0000","last_fullsized":"2024-04-02T14:34:52.507926+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:51:20.232086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":82,"num_read_kb":70,"num_write":88,"num_write_kb":838,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"90'5","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.247133+0000","last_change":"2024-04-02T14:32:22.554554+0000","last_active":"2024-04-02T14:34:52.247133+0000","last_peered":"2024-04-02T14:34:52.247133+0000","last_clean":"2024-04-02T14:34:52.247133+0000","last_became_active":"2024-04-02T14:32:22.554194+0000","last_became_peered":"2024-04-02T14:32:22.554194+0000","last_unstale":"2024-04-02T14:34:52.247133+0000","last_undegraded":"2024-04-02T14:34:52.247133+0000","last_fullsized":"2024-04-02T14:34:52.247133+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:27:51.382441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"88'7","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.247855+0000","last_change":"2024-04-02T14:32:21.850346+0000","last_active":"2024-04-02T14:34:52.247855+0000","last_peered":"2024-04-02T14:34:52.247855+0000","last_clean":"2024-04-02T14:34:52.247855+0000","last_became_active":"2024-04-02T14:32:21.850084+0000","last_became_peered":"2024-04-02T14:32:21.850084+0000","last_unstale":"2024-04-02T14:34:52.247855+0000","last_undegraded":"2024-04-02T14:34:52.247855+0000","last_fullsized":"2024-04-02T14:34:52.247855+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:56:17.774702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"90'7","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.548807+0000","last_change":"2024-04-02T14:32:22.557054+0000","last_active":"2024-04-02T14:34:52.548807+0000","last_peered":"2024-04-02T14:34:52.548807+0000","last_clean":"2024-04-02T14:34:52.548807+0000","last_became_active":"2024-04-02T14:32:22.556335+0000","last_became_peered":"2024-04-02T14:32:22.556335+0000","last_unstale":"2024-04-02T14:34:52.548807+0000","last_undegraded":"2024-04-02T14:34:52.548807+0000","last_fullsized":"2024-04-02T14:34:52.548807+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:06:48.907197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"88'2","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.545713+0000","last_change":"2024-04-02T14:32:21.860260+0000","last_active":"2024-04-02T14:34:52.545713+0000","last_peered":"2024-04-02T14:34:52.545713+0000","last_clean":"2024-04-02T14:34:52.545713+0000","last_became_active":"2024-04-02T14:32:21.859922+0000","last_became_peered":"2024-04-02T14:32:21.859922+0000","last_unstale":"2024-04-02T14:34:52.545713+0000","last_undegraded":"2024-04-02T14:34:52.545713+0000","last_fullsized":"2024-04-02T14:34:52.545713+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:07:45.850148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"90'2","reported_seq":22,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.247646+0000","last_change":"2024-04-02T14:32:22.536063+0000","last_active":"2024-04-02T14:34:52.247646+0000","last_peered":"2024-04-02T14:34:52.247646+0000","last_clean":"2024-04-02T14:34:52.247646+0000","last_became_active":"2024-04-02T14:32:22.534678+0000","last_became_peered":"2024-04-02T14:32:22.534678+0000","last_unstale":"2024-04-02T14:34:52.247646+0000","last_undegraded":"2024-04-02T14:34:52.247646+0000","last_fullsized":"2024-04-02T14:34:52.247646+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T02:07:26.186216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"88'7","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.246470+0000","last_change":"2024-04-02T14:32:21.848554+0000","last_active":"2024-04-02T14:34:52.246470+0000","last_peered":"2024-04-02T14:34:52.246470+0000","last_clean":"2024-04-02T14:34:52.246470+0000","last_became_active":"2024-04-02T14:32:21.848177+0000","last_became_peered":"2024-04-02T14:32:21.848177+0000","last_unstale":"2024-04-02T14:34:52.246470+0000","last_undegraded":"2024-04-02T14:34:52.246470+0000","last_fullsized":"2024-04-02T14:34:52.246470+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:47:04.213286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"90'9","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.549385+0000","last_change":"2024-04-02T14:32:22.547246+0000","last_active":"2024-04-02T14:34:52.549385+0000","last_peered":"2024-04-02T14:34:52.549385+0000","last_clean":"2024-04-02T14:34:52.549385+0000","last_became_active":"2024-04-02T14:32:22.546721+0000","last_became_peered":"2024-04-02T14:32:22.546721+0000","last_unstale":"2024-04-02T14:34:52.549385+0000","last_undegraded":"2024-04-02T14:34:52.549385+0000","last_fullsized":"2024-04-02T14:34:52.549385+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:11:30.615723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"88'3","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.549450+0000","last_change":"2024-04-02T14:32:35.743474+0000","last_active":"2024-04-02T14:34:52.549450+0000","last_peered":"2024-04-02T14:34:52.549450+0000","last_clean":"2024-04-02T14:34:52.549450+0000","last_became_active":"2024-04-02T14:32:35.738522+0000","last_became_peered":"2024-04-02T14:32:35.738522+0000","last_unstale":"2024-04-02T14:34:52.549450+0000","last_undegraded":"2024-04-02T14:34:52.549450+0000","last_fullsized":"2024-04-02T14:34:52.549450+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T01:14:43.989279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"90'8","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.246782+0000","last_change":"2024-04-02T14:32:22.561787+0000","last_active":"2024-04-02T14:34:52.246782+0000","last_peered":"2024-04-02T14:34:52.246782+0000","last_clean":"2024-04-02T14:34:52.246782+0000","last_became_active":"2024-04-02T14:32:22.561284+0000","last_became_peered":"2024-04-02T14:32:22.561284+0000","last_unstale":"2024-04-02T14:34:52.246782+0000","last_undegraded":"2024-04-02T14:34:52.246782+0000","last_fullsized":"2024-04-02T14:34:52.246782+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:46:06.450097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"88'4","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.247746+0000","last_change":"2024-04-02T14:32:21.841340+0000","last_active":"2024-04-02T14:34:52.247746+0000","last_peered":"2024-04-02T14:34:52.247746+0000","last_clean":"2024-04-02T14:34:52.247746+0000","last_became_active":"2024-04-02T14:32:21.840937+0000","last_became_peered":"2024-04-02T14:32:21.840937+0000","last_unstale":"2024-04-02T14:34:52.247746+0000","last_undegraded":"2024-04-02T14:34:52.247746+0000","last_fullsized":"2024-04-02T14:34:52.247746+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T01:47:44.834440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"90'5","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.544165+0000","last_change":"2024-04-02T14:32:22.538890+0000","last_active":"2024-04-02T14:34:52.544165+0000","last_peered":"2024-04-02T14:34:52.544165+0000","last_clean":"2024-04-02T14:34:52.544165+0000","last_became_active":"2024-04-02T14:32:22.538512+0000","last_became_peered":"2024-04-02T14:32:22.538512+0000","last_unstale":"2024-04-02T14:34:52.544165+0000","last_undegraded":"2024-04-02T14:34:52.544165+0000","last_fullsized":"2024-04-02T14:34:52.544165+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:59:23.792851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"91'6","reported_seq":175,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:53.040074+0000","last_change":"2024-04-02T14:32:21.862727+0000","last_active":"2024-04-02T14:34:53.040074+0000","last_peered":"2024-04-02T14:34:53.040074+0000","last_clean":"2024-04-02T14:34:53.040074+0000","last_became_active":"2024-04-02T14:32:21.862405+0000","last_became_peered":"2024-04-02T14:32:21.862405+0000","last_unstale":"2024-04-02T14:34:53.040074+0000","last_undegraded":"2024-04-02T14:34:53.040074+0000","last_fullsized":"2024-04-02T14:34:53.040074+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:36:54.849964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1561,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":147,"num_read_kb":3,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"90'7","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.544768+0000","last_change":"2024-04-02T14:32:22.540077+0000","last_active":"2024-04-02T14:34:52.544768+0000","last_peered":"2024-04-02T14:34:52.544768+0000","last_clean":"2024-04-02T14:34:52.544768+0000","last_became_active":"2024-04-02T14:32:22.539513+0000","last_became_peered":"2024-04-02T14:32:22.539513+0000","last_unstale":"2024-04-02T14:34:52.544768+0000","last_undegraded":"2024-04-02T14:34:52.544768+0000","last_fullsized":"2024-04-02T14:34:52.544768+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T02:00:39.819963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"91'41","reported_seq":71,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.501385+0000","last_change":"2024-04-02T14:32:21.860310+0000","last_active":"2024-04-02T14:34:52.501385+0000","last_peered":"2024-04-02T14:34:52.501385+0000","last_clean":"2024-04-02T14:34:52.501385+0000","last_became_active":"2024-04-02T14:32:21.859955+0000","last_became_peered":"2024-04-02T14:32:21.859955+0000","last_unstale":"2024-04-02T14:34:52.501385+0000","last_undegraded":"2024-04-02T14:34:52.501385+0000","last_fullsized":"2024-04-02T14:34:52.501385+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:11:32.732603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131670,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":49,"num_write_kb":475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"90'7","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.544288+0000","last_change":"2024-04-02T14:32:22.552134+0000","last_active":"2024-04-02T14:34:52.544288+0000","last_peered":"2024-04-02T14:34:52.544288+0000","last_clean":"2024-04-02T14:34:52.544288+0000","last_became_active":"2024-04-02T14:32:22.551695+0000","last_became_peered":"2024-04-02T14:32:22.551695+0000","last_unstale":"2024-04-02T14:34:52.544288+0000","last_undegraded":"2024-04-02T14:34:52.544288+0000","last_fullsized":"2024-04-02T14:34:52.544288+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:09:36.902243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"88'5","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.545160+0000","last_change":"2024-04-02T14:32:21.857571+0000","last_active":"2024-04-02T14:34:52.545160+0000","last_peered":"2024-04-02T14:34:52.545160+0000","last_clean":"2024-04-02T14:34:52.545160+0000","last_became_active":"2024-04-02T14:32:21.856935+0000","last_became_peered":"2024-04-02T14:32:21.856935+0000","last_unstale":"2024-04-02T14:34:52.545160+0000","last_undegraded":"2024-04-02T14:34:52.545160+0000","last_fullsized":"2024-04-02T14:34:52.545160+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:55:46.079097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"90'9","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.544456+0000","last_change":"2024-04-02T14:32:22.534708+0000","last_active":"2024-04-02T14:34:52.544456+0000","last_peered":"2024-04-02T14:34:52.544456+0000","last_clean":"2024-04-02T14:34:52.544456+0000","last_became_active":"2024-04-02T14:32:22.534365+0000","last_became_peered":"2024-04-02T14:32:22.534365+0000","last_unstale":"2024-04-02T14:34:52.544456+0000","last_undegraded":"2024-04-02T14:34:52.544456+0000","last_fullsized":"2024-04-02T14:34:52.544456+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:34:47.435729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"90'9","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.549434+0000","last_change":"2024-04-02T14:32:22.538317+0000","last_active":"2024-04-02T14:34:52.549434+0000","last_peered":"2024-04-02T14:34:52.549434+0000","last_clean":"2024-04-02T14:34:52.549434+0000","last_became_active":"2024-04-02T14:32:22.537939+0000","last_became_peered":"2024-04-02T14:32:22.537939+0000","last_unstale":"2024-04-02T14:34:52.549434+0000","last_undegraded":"2024-04-02T14:34:52.549434+0000","last_fullsized":"2024-04-02T14:34:52.549434+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:31:47.075326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"90'12","reported_seq":38,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.283556+0000","last_change":"2024-04-02T14:32:35.852202+0000","last_active":"2024-04-02T14:34:52.283556+0000","last_peered":"2024-04-02T14:34:52.283556+0000","last_clean":"2024-04-02T14:34:52.283556+0000","last_became_active":"2024-04-02T14:32:35.851785+0000","last_became_peered":"2024-04-02T14:32:35.851785+0000","last_unstale":"2024-04-02T14:34:52.283556+0000","last_undegraded":"2024-04-02T14:34:52.283556+0000","last_fullsized":"2024-04-02T14:34:52.283556+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:09:17.410727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,4],"acting":[2,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"90'8","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.283542+0000","last_change":"2024-04-02T14:32:22.556567+0000","last_active":"2024-04-02T14:34:52.283542+0000","last_peered":"2024-04-02T14:34:52.283542+0000","last_clean":"2024-04-02T14:34:52.283542+0000","last_became_active":"2024-04-02T14:32:22.556267+0000","last_became_peered":"2024-04-02T14:32:22.556267+0000","last_unstale":"2024-04-02T14:34:52.283542+0000","last_undegraded":"2024-04-02T14:34:52.283542+0000","last_fullsized":"2024-04-02T14:34:52.283542+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:35:30.039013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"90'7","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.547650+0000","last_change":"2024-04-02T14:32:35.848125+0000","last_active":"2024-04-02T14:34:52.547650+0000","last_peered":"2024-04-02T14:34:52.547650+0000","last_clean":"2024-04-02T14:34:52.547650+0000","last_became_active":"2024-04-02T14:32:35.847681+0000","last_became_peered":"2024-04-02T14:32:35.847681+0000","last_unstale":"2024-04-02T14:34:52.547650+0000","last_undegraded":"2024-04-02T14:34:52.547650+0000","last_fullsized":"2024-04-02T14:34:52.547650+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:51:38.040261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"90'4","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.545854+0000","last_change":"2024-04-02T14:32:22.558584+0000","last_active":"2024-04-02T14:34:52.545854+0000","last_peered":"2024-04-02T14:34:52.545854+0000","last_clean":"2024-04-02T14:34:52.545854+0000","last_became_active":"2024-04-02T14:32:22.557382+0000","last_became_peered":"2024-04-02T14:32:22.557382+0000","last_unstale":"2024-04-02T14:34:52.545854+0000","last_undegraded":"2024-04-02T14:34:52.545854+0000","last_fullsized":"2024-04-02T14:34:52.545854+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:09:25.107411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"90'6","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.549497+0000","last_change":"2024-04-02T14:32:22.538318+0000","last_active":"2024-04-02T14:34:52.549497+0000","last_peered":"2024-04-02T14:34:52.549497+0000","last_clean":"2024-04-02T14:34:52.549497+0000","last_became_active":"2024-04-02T14:32:22.538011+0000","last_became_peered":"2024-04-02T14:32:22.538011+0000","last_unstale":"2024-04-02T14:34:52.549497+0000","last_undegraded":"2024-04-02T14:34:52.549497+0000","last_fullsized":"2024-04-02T14:34:52.549497+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:11:21.269106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"90'15","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.283948+0000","last_change":"2024-04-02T14:32:22.535645+0000","last_active":"2024-04-02T14:34:52.283948+0000","last_peered":"2024-04-02T14:34:52.283948+0000","last_clean":"2024-04-02T14:34:52.283948+0000","last_became_active":"2024-04-02T14:32:22.535314+0000","last_became_peered":"2024-04-02T14:32:22.535314+0000","last_unstale":"2024-04-02T14:34:52.283948+0000","last_undegraded":"2024-04-02T14:34:52.283948+0000","last_fullsized":"2024-04-02T14:34:52.283948+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:51:27.879015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"90'2","reported_seq":22,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.247021+0000","last_change":"2024-04-02T14:32:22.553959+0000","last_active":"2024-04-02T14:34:52.247021+0000","last_peered":"2024-04-02T14:34:52.247021+0000","last_clean":"2024-04-02T14:34:52.247021+0000","last_became_active":"2024-04-02T14:32:22.553389+0000","last_became_peered":"2024-04-02T14:32:22.553389+0000","last_unstale":"2024-04-02T14:34:52.247021+0000","last_undegraded":"2024-04-02T14:34:52.247021+0000","last_fullsized":"2024-04-02T14:34:52.247021+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:55:00.552600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"90'6","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.377384+0000","last_change":"2024-04-02T14:32:22.551769+0000","last_active":"2024-04-02T14:34:52.377384+0000","last_peered":"2024-04-02T14:34:52.377384+0000","last_clean":"2024-04-02T14:34:52.377384+0000","last_became_active":"2024-04-02T14:32:22.551422+0000","last_became_peered":"2024-04-02T14:32:22.551422+0000","last_unstale":"2024-04-02T14:34:52.377384+0000","last_undegraded":"2024-04-02T14:34:52.377384+0000","last_fullsized":"2024-04-02T14:34:52.377384+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T01:15:07.262927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"90'8","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.246610+0000","last_change":"2024-04-02T14:32:22.548264+0000","last_active":"2024-04-02T14:34:52.246610+0000","last_peered":"2024-04-02T14:34:52.246610+0000","last_clean":"2024-04-02T14:34:52.246610+0000","last_became_active":"2024-04-02T14:32:22.547293+0000","last_became_peered":"2024-04-02T14:32:22.547293+0000","last_unstale":"2024-04-02T14:34:52.246610+0000","last_undegraded":"2024-04-02T14:34:52.246610+0000","last_fullsized":"2024-04-02T14:34:52.246610+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:53:19.896243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"90'10","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.545911+0000","last_change":"2024-04-02T14:32:22.556131+0000","last_active":"2024-04-02T14:34:52.545911+0000","last_peered":"2024-04-02T14:34:52.545911+0000","last_clean":"2024-04-02T14:34:52.545911+0000","last_became_active":"2024-04-02T14:32:22.555763+0000","last_became_peered":"2024-04-02T14:32:22.555763+0000","last_unstale":"2024-04-02T14:34:52.545911+0000","last_undegraded":"2024-04-02T14:34:52.545911+0000","last_fullsized":"2024-04-02T14:34:52.545911+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:45:22.485403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"90'11","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.545531+0000","last_change":"2024-04-02T14:32:22.562093+0000","last_active":"2024-04-02T14:34:52.545531+0000","last_peered":"2024-04-02T14:34:52.545531+0000","last_clean":"2024-04-02T14:34:52.545531+0000","last_became_active":"2024-04-02T14:32:22.561663+0000","last_became_peered":"2024-04-02T14:32:22.561663+0000","last_unstale":"2024-04-02T14:34:52.545531+0000","last_undegraded":"2024-04-02T14:34:52.545531+0000","last_fullsized":"2024-04-02T14:34:52.545531+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:14:25.209276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"90'8","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.549387+0000","last_change":"2024-04-02T14:32:22.541964+0000","last_active":"2024-04-02T14:34:52.549387+0000","last_peered":"2024-04-02T14:34:52.549387+0000","last_clean":"2024-04-02T14:34:52.549387+0000","last_became_active":"2024-04-02T14:32:22.541549+0000","last_became_peered":"2024-04-02T14:32:22.541549+0000","last_unstale":"2024-04-02T14:34:52.549387+0000","last_undegraded":"2024-04-02T14:34:52.549387+0000","last_fullsized":"2024-04-02T14:34:52.549387+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:04:56.783158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":601231,"num_objects":100,"num_object_clones":0,"num_object_copies":300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":0,"num_read":1561,"num_read_kb":889,"num_write":372,"num_write_kb":1465,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":44049,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2174976,"data_stored":1803693,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":300,"ondisk_log_size":300,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":83,"seq":356482285611,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32168,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712984,"statfs":{"total":95995035648,"available":95962095616,"internally_reserved":0,"allocated":1011712,"data_stored":484076,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":75,"seq":322122547249,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":37000,"kb_used_data":1664,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708152,"statfs":{"total":95995035648,"available":95957147648,"internally_reserved":0,"allocated":1703936,"data_stored":1133256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":36174273},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743477,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36736,"kb_used_data":1396,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708416,"statfs":{"total":95995035648,"available":95957417984,"internally_reserved":0,"allocated":1429504,"data_stored":876831,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":61,"seq":261993005117,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32564,"kb_used_data":1384,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712588,"statfs":{"total":95995035648,"available":95961690112,"internally_reserved":0,"allocated":1417216,"data_stored":876537,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299458,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32772,"kb_used_data":1592,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712380,"statfs":{"total":95995035648,"available":95961477120,"internally_reserved":0,"allocated":1630208,"data_stored":1077966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495686,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36336,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708816,"statfs":{"total":95995035648,"available":95957827584,"internally_reserved":0,"allocated":1019904,"data_stored":483332,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574947,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36404,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708748,"statfs":{"total":95995035648,"available":95957757952,"internally_reserved":0,"allocated":1089536,"data_stored":552001,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738472,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36336,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708816,"statfs":{"total":95995035648,"available":95957827584,"internally_reserved":0,"allocated":1024000,"data_stored":475707,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411422,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36348,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708804,"statfs":{"total":95995035648,"available":95957815296,"internally_reserved":0,"allocated":1032192,"data_stored":484997,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182487,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28032,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717120,"statfs":{"total":95995035648,"available":95966330880,"internally_reserved":0,"allocated":1036288,"data_stored":483157,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920851,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36420,"kb_used_data":1080,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708732,"statfs":{"total":95995035648,"available":95957741568,"internally_reserved":0,"allocated":1105920,"data_stored":551371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757324,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28752,"kb_used_data":1736,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716400,"statfs":{"total":95995035648,"available":95965593600,"internally_reserved":0,"allocated":1777664,"data_stored":1207930,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2569,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":69508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":2895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":557056,"data_stored":525246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":70442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":180224,"data_stored":134952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":4988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":394044,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":430080,"data_stored":394338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":258048,"data_stored":202406,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-02T14:34:58.749 INFO:tasks.fwd_scrub.ceph_manager:clean! 2024-04-02T14:34:58.749 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-02T14:34:58.848 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:34:58 smithi090 ceph-mon[31674]: from='client.? 172.21.15.90:0/2936994380' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-02T14:34:58.848 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:34:58 smithi090 ceph-mon[31674]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' 2024-04-02T14:34:58.848 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:34:58 smithi090 ceph-mon[31674]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' 2024-04-02T14:34:58.848 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:34:58 smithi090 ceph-mon[31674]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-02T14:34:58.848 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:34:58 smithi090 ceph-mon[31674]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-02T14:34:58.848 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:34:58 smithi090 ceph-mon[31674]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-02T14:34:58.848 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:34:58 smithi090 ceph-mon[31674]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-02T14:34:58.848 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:34:58 smithi090 ceph-mon[31674]: Adjusting osd_memory_target on smithi090 to 176.7M 2024-04-02T14:34:58.848 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:34:58 smithi090 ceph-mon[31674]: Unable to set osd_memory_target on smithi090 to 185366630: error parsing value: Value '185366630' is below minimum 939524096 2024-04-02T14:34:58.849 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:34:58 smithi090 ceph-mon[31674]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:34:58.849 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:34:58 smithi090 ceph-mon[31674]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T14:34:58.849 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:34:58 smithi090 ceph-mon[31674]: Updating smithi090:/etc/ceph/ceph.conf 2024-04-02T14:34:58.849 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:34:58 smithi090 ceph-mon[31674]: Updating smithi092:/etc/ceph/ceph.conf 2024-04-02T14:34:58.849 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:34:58 smithi090 ceph-mon[31674]: Updating smithi177:/etc/ceph/ceph.conf 2024-04-02T14:34:58.860 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:34:58 smithi092 ceph-mon[36497]: from='client.? 172.21.15.90:0/2936994380' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-02T14:34:58.861 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:34:58 smithi092 ceph-mon[36497]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' 2024-04-02T14:34:58.861 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:34:58 smithi092 ceph-mon[36497]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' 2024-04-02T14:34:58.861 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:34:58 smithi092 ceph-mon[36497]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-02T14:34:58.861 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:34:58 smithi092 ceph-mon[36497]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-02T14:34:58.861 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:34:58 smithi092 ceph-mon[36497]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-02T14:34:58.861 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:34:58 smithi092 ceph-mon[36497]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-02T14:34:58.861 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:34:58 smithi092 ceph-mon[36497]: Adjusting osd_memory_target on smithi090 to 176.7M 2024-04-02T14:34:58.861 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:34:58 smithi092 ceph-mon[36497]: Unable to set osd_memory_target on smithi090 to 185366630: error parsing value: Value '185366630' is below minimum 939524096 2024-04-02T14:34:58.861 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:34:58 smithi092 ceph-mon[36497]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:34:58.861 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:34:58 smithi092 ceph-mon[36497]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T14:34:58.861 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:34:58 smithi092 ceph-mon[36497]: Updating smithi090:/etc/ceph/ceph.conf 2024-04-02T14:34:58.861 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:34:58 smithi092 ceph-mon[36497]: Updating smithi092:/etc/ceph/ceph.conf 2024-04-02T14:34:58.862 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:34:58 smithi092 ceph-mon[36497]: Updating smithi177:/etc/ceph/ceph.conf 2024-04-02T14:34:58.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:34:58 smithi177 ceph-mon[36431]: from='client.? 172.21.15.90:0/2936994380' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-02T14:34:58.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:34:58 smithi177 ceph-mon[36431]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' 2024-04-02T14:34:58.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:34:58 smithi177 ceph-mon[36431]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' 2024-04-02T14:34:58.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:34:58 smithi177 ceph-mon[36431]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-02T14:34:58.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:34:58 smithi177 ceph-mon[36431]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-02T14:34:58.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:34:58 smithi177 ceph-mon[36431]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-02T14:34:58.938 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:34:58 smithi177 ceph-mon[36431]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-02T14:34:58.938 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:34:58 smithi177 ceph-mon[36431]: Adjusting osd_memory_target on smithi090 to 176.7M 2024-04-02T14:34:58.938 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:34:58 smithi177 ceph-mon[36431]: Unable to set osd_memory_target on smithi090 to 185366630: error parsing value: Value '185366630' is below minimum 939524096 2024-04-02T14:34:58.938 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:34:58 smithi177 ceph-mon[36431]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:34:58.938 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:34:58 smithi177 ceph-mon[36431]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T14:34:58.938 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:34:58 smithi177 ceph-mon[36431]: Updating smithi090:/etc/ceph/ceph.conf 2024-04-02T14:34:58.939 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:34:58 smithi177 ceph-mon[36431]: Updating smithi092:/etc/ceph/ceph.conf 2024-04-02T14:34:58.939 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:34:58 smithi177 ceph-mon[36431]: Updating smithi177:/etc/ceph/ceph.conf 2024-04-02T14:34:58.979 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:58.977+0000 7f90d4200640 1 -- 172.21.15.90:0/4013458165 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f90cc100d70 msgr2=0x7f90cc1011d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:58.979 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:58.977+0000 7f90d4200640 1 --2- 172.21.15.90:0/4013458165 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f90cc100d70 0x7f90cc1011d0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f90b8009900 tx=0x7f90b802bcc0 comp rx=0 tx=0).stop 2024-04-02T14:34:58.979 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:58.977+0000 7f90d4200640 1 -- 172.21.15.90:0/4013458165 shutdown_connections 2024-04-02T14:34:58.979 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:58.977+0000 7f90d4200640 1 --2- 172.21.15.90:0/4013458165 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f90cc101710 0x7f90cc101b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:58.979 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:58.977+0000 7f90d4200640 1 --2- 172.21.15.90:0/4013458165 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f90cc100d70 0x7f90cc1011d0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:58.979 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:58.977+0000 7f90d4200640 1 --2- 172.21.15.90:0/4013458165 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f90cc100410 0x7f90cc100830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:58.979 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:58.977+0000 7f90d4200640 1 -- 172.21.15.90:0/4013458165 >> 172.21.15.90:0/4013458165 conn(0x7f90cc0fbfb0 msgr2=0x7f90cc0fe3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:34:58.979 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:58.977+0000 7f90d4200640 1 -- 172.21.15.90:0/4013458165 shutdown_connections 2024-04-02T14:34:58.979 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:58.977+0000 7f90d4200640 1 -- 172.21.15.90:0/4013458165 wait complete. 2024-04-02T14:34:58.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:58.978+0000 7f90d4200640 1 Processor -- start 2024-04-02T14:34:58.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:58.978+0000 7f90d4200640 1 -- start start 2024-04-02T14:34:58.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:58.979+0000 7f90d4200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f90cc100410 0x7f90cc19a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:58.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:58.979+0000 7f90d4200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f90cc100d70 0x7f90cc19a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:58.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:58.979+0000 7f90d4200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f90cc101710 0x7f90cc1a15c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:58.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:58.979+0000 7f90d4200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f90cc10f9d0 con 0x7f90cc100410 2024-04-02T14:34:58.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:58.979+0000 7f90d4200640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f90cc10f850 con 0x7f90cc100d70 2024-04-02T14:34:58.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:58.979+0000 7f90d4200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f90cc10fb50 con 0x7f90cc101710 2024-04-02T14:34:58.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:58.979+0000 7f90d0e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f90cc100d70 0x7f90cc19a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:58.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:58.979+0000 7f90d2200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f90cc101710 0x7f90cc1a15c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:58.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:58.979+0000 7f90d1800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f90cc100410 0x7f90cc19a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:58.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:58.979+0000 7f90d2200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f90cc101710 0x7f90cc1a15c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:49472/0 (socket says 172.21.15.90:49472) 2024-04-02T14:34:58.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:58.979+0000 7f90d0e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f90cc100d70 0x7f90cc19a550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:58956/0 (socket says 172.21.15.90:58956) 2024-04-02T14:34:58.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:58.979+0000 7f90d2200640 1 -- 172.21.15.90:0/3943358202 learned_addr learned my addr 172.21.15.90:0/3943358202 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:34:58.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:58.980+0000 7f90d2200640 1 -- 172.21.15.90:0/3943358202 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f90cc100d70 msgr2=0x7f90cc19a550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:58.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:58.980+0000 7f90d2200640 1 --2- 172.21.15.90:0/3943358202 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f90cc100d70 0x7f90cc19a550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:58.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:58.980+0000 7f90d2200640 1 -- 172.21.15.90:0/3943358202 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f90cc100410 msgr2=0x7f90cc19a010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:58.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:58.980+0000 7f90d2200640 1 --2- 172.21.15.90:0/3943358202 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f90cc100410 0x7f90cc19a010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:58.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:58.980+0000 7f90d2200640 1 -- 172.21.15.90:0/3943358202 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90cc1a1c30 con 0x7f90cc101710 2024-04-02T14:34:58.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:58.980+0000 7f90d0e00640 1 --2- 172.21.15.90:0/3943358202 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f90cc100d70 0x7f90cc19a550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:34:58.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:58.980+0000 7f90d2200640 1 --2- 172.21.15.90:0/3943358202 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f90cc101710 0x7f90cc1a15c0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f90c400eb10 tx=0x7f90c400c2d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:34:58.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:58.980+0000 7f90ca000640 1 -- 172.21.15.90:0/3943358202 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90c4004b20 con 0x7f90cc101710 2024-04-02T14:34:58.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:58.980+0000 7f90ca000640 1 -- 172.21.15.90:0/3943358202 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f90c4004cc0 con 0x7f90cc101710 2024-04-02T14:34:58.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:58.980+0000 7f90d4200640 1 -- 172.21.15.90:0/3943358202 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f90cc1a1f20 con 0x7f90cc101710 2024-04-02T14:34:58.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:58.980+0000 7f90ca000640 1 -- 172.21.15.90:0/3943358202 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90c4010760 con 0x7f90cc101710 2024-04-02T14:34:58.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:58.981+0000 7f90d4200640 1 -- 172.21.15.90:0/3943358202 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f90cc1a2380 con 0x7f90cc101710 2024-04-02T14:34:58.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:58.982+0000 7f90d4200640 1 -- 172.21.15.90:0/3943358202 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f90cc10b2c0 con 0x7f90cc101710 2024-04-02T14:34:58.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:58.983+0000 7f90ca000640 1 -- 172.21.15.90:0/3943358202 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7f90c4005020 con 0x7f90cc101710 2024-04-02T14:34:58.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:58.985+0000 7f90ca000640 1 --2- 172.21.15.90:0/3943358202 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f90b40a3e40 0x7f90b40a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:58.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:58.985+0000 7f90ca000640 1 -- 172.21.15.90:0/3943358202 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f90c40d1080 con 0x7f90cc101710 2024-04-02T14:34:58.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:58.985+0000 7f90d1800640 1 --2- 172.21.15.90:0/3943358202 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f90b40a3e40 0x7f90b40a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:58.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:58.986+0000 7f90d1800640 1 --2- 172.21.15.90:0/3943358202 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f90b40a3e40 0x7f90b40a6300 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f90c0009bd0 tx=0x7f90c0005f90 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:34:58.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:58.989+0000 7f90ca000640 1 -- 172.21.15.90:0/3943358202 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7f90c409d480 con 0x7f90cc101710 2024-04-02T14:34:59.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:59.236+0000 7f90d4200640 1 -- 172.21.15.90:0/3943358202 --> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f90cc061f40 con 0x7f90b40a3e40 2024-04-02T14:34:59.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:59.245+0000 7f90ca000640 1 -- 172.21.15.90:0/3943358202 <== mgr.25289 v2:172.21.15.90:6800/3904993754 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+257837 (secure 0 0 0) 0x7f90cc061f40 con 0x7f90b40a3e40 2024-04-02T14:34:59.249 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-02T14:34:59.250 INFO:teuthology.orchestra.run.smithi090.stderr:dumped all 2024-04-02T14:34:59.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:59.249+0000 7f90d4200640 1 -- 172.21.15.90:0/3943358202 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f90b40a3e40 msgr2=0x7f90b40a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:59.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:59.249+0000 7f90d4200640 1 --2- 172.21.15.90:0/3943358202 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f90b40a3e40 0x7f90b40a6300 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f90c0009bd0 tx=0x7f90c0005f90 comp rx=0 tx=0).stop 2024-04-02T14:34:59.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:59.249+0000 7f90d4200640 1 -- 172.21.15.90:0/3943358202 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f90cc101710 msgr2=0x7f90cc1a15c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:59.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:59.249+0000 7f90d4200640 1 --2- 172.21.15.90:0/3943358202 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f90cc101710 0x7f90cc1a15c0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f90c400eb10 tx=0x7f90c400c2d0 comp rx=0 tx=0).stop 2024-04-02T14:34:59.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:59.249+0000 7f90d4200640 1 -- 172.21.15.90:0/3943358202 shutdown_connections 2024-04-02T14:34:59.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:59.249+0000 7f90d4200640 1 --2- 172.21.15.90:0/3943358202 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f90b40a3e40 0x7f90b40a6300 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:59.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:59.249+0000 7f90d4200640 1 --2- 172.21.15.90:0/3943358202 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f90cc101710 0x7f90cc1a15c0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:59.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:59.249+0000 7f90d4200640 1 --2- 172.21.15.90:0/3943358202 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f90cc100d70 0x7f90cc19a550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:59.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:59.249+0000 7f90d4200640 1 --2- 172.21.15.90:0/3943358202 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f90cc100410 0x7f90cc19a010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:59.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:59.249+0000 7f90d4200640 1 -- 172.21.15.90:0/3943358202 >> 172.21.15.90:0/3943358202 conn(0x7f90cc0fbfb0 msgr2=0x7f90cc0fd640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:34:59.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:59.249+0000 7f90d4200640 1 -- 172.21.15.90:0/3943358202 shutdown_connections 2024-04-02T14:34:59.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:59.249+0000 7f90d4200640 1 -- 172.21.15.90:0/3943358202 wait complete. 2024-04-02T14:34:59.269 INFO:teuthology.orchestra.run.smithi090.stdout:{"pg_ready":true,"pg_map":{"version":6,"stamp":"2024-04-02T14:34:58.270956+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1191833,"num_objects":104,"num_object_clones":0,"num_object_copies":312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":104,"num_whiteouts":0,"num_read":1697,"num_read_kb":1008,"num_write":555,"num_write_kb":4405,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":1224785,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":939,"ondisk_log_size":939,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":409868,"kb_used_data":14920,"kb_used_omap":18,"kb_used_meta":394797,"kb_avail":1124531956,"statfs":{"total":1151940427776,"available":1151520722944,"internally_reserved":0,"allocated":15278080,"data_stored":8687161,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":404272442},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.978360"},"pg_stats":[{"pgid":"3.3f","version":"90'8","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.549650+0000","last_change":"2024-04-02T14:32:22.558913+0000","last_active":"2024-04-02T14:34:52.549650+0000","last_peered":"2024-04-02T14:34:52.549650+0000","last_clean":"2024-04-02T14:34:52.549650+0000","last_became_active":"2024-04-02T14:32:22.558606+0000","last_became_peered":"2024-04-02T14:32:22.558606+0000","last_unstale":"2024-04-02T14:34:52.549650+0000","last_undegraded":"2024-04-02T14:34:52.549650+0000","last_fullsized":"2024-04-02T14:34:52.549650+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:59:49.205499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"90'4","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.547653+0000","last_change":"2024-04-02T14:32:22.551898+0000","last_active":"2024-04-02T14:34:52.547653+0000","last_peered":"2024-04-02T14:34:52.547653+0000","last_clean":"2024-04-02T14:34:52.547653+0000","last_became_active":"2024-04-02T14:32:22.549016+0000","last_became_peered":"2024-04-02T14:32:22.549016+0000","last_unstale":"2024-04-02T14:34:52.547653+0000","last_undegraded":"2024-04-02T14:34:52.547653+0000","last_fullsized":"2024-04-02T14:34:52.547653+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:38:03.529505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"90'7","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.548042+0000","last_change":"2024-04-02T14:32:22.556214+0000","last_active":"2024-04-02T14:34:52.548042+0000","last_peered":"2024-04-02T14:34:52.548042+0000","last_clean":"2024-04-02T14:34:52.548042+0000","last_became_active":"2024-04-02T14:32:22.555686+0000","last_became_peered":"2024-04-02T14:32:22.555686+0000","last_unstale":"2024-04-02T14:34:52.548042+0000","last_undegraded":"2024-04-02T14:34:52.548042+0000","last_fullsized":"2024-04-02T14:34:52.548042+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:06:40.161203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"90'8","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.247614+0000","last_change":"2024-04-02T14:32:22.537830+0000","last_active":"2024-04-02T14:34:52.247614+0000","last_peered":"2024-04-02T14:34:52.247614+0000","last_clean":"2024-04-02T14:34:52.247614+0000","last_became_active":"2024-04-02T14:32:22.536860+0000","last_became_peered":"2024-04-02T14:32:22.536860+0000","last_unstale":"2024-04-02T14:34:52.247614+0000","last_undegraded":"2024-04-02T14:34:52.247614+0000","last_fullsized":"2024-04-02T14:34:52.247614+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T01:07:57.244177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"90'4","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.247802+0000","last_change":"2024-04-02T14:32:22.549550+0000","last_active":"2024-04-02T14:34:52.247802+0000","last_peered":"2024-04-02T14:34:52.247802+0000","last_clean":"2024-04-02T14:34:52.247802+0000","last_became_active":"2024-04-02T14:32:22.549177+0000","last_became_peered":"2024-04-02T14:32:22.549177+0000","last_unstale":"2024-04-02T14:34:52.247802+0000","last_undegraded":"2024-04-02T14:34:52.247802+0000","last_fullsized":"2024-04-02T14:34:52.247802+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:31:49.175583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"90'6","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.380610+0000","last_change":"2024-04-02T14:32:35.849338+0000","last_active":"2024-04-02T14:34:52.380610+0000","last_peered":"2024-04-02T14:34:52.380610+0000","last_clean":"2024-04-02T14:34:52.380610+0000","last_became_active":"2024-04-02T14:32:35.848929+0000","last_became_peered":"2024-04-02T14:32:35.848929+0000","last_unstale":"2024-04-02T14:34:52.380610+0000","last_undegraded":"2024-04-02T14:34:52.380610+0000","last_fullsized":"2024-04-02T14:34:52.380610+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:55:58.614290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,6],"acting":[10,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"90'5","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.543923+0000","last_change":"2024-04-02T14:32:22.546861+0000","last_active":"2024-04-02T14:34:52.543923+0000","last_peered":"2024-04-02T14:34:52.543923+0000","last_clean":"2024-04-02T14:34:52.543923+0000","last_became_active":"2024-04-02T14:32:22.546477+0000","last_became_peered":"2024-04-02T14:32:22.546477+0000","last_unstale":"2024-04-02T14:34:52.543923+0000","last_undegraded":"2024-04-02T14:34:52.543923+0000","last_fullsized":"2024-04-02T14:34:52.543923+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:59:17.133714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"90'9","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.246514+0000","last_change":"2024-04-02T14:32:22.554287+0000","last_active":"2024-04-02T14:34:52.246514+0000","last_peered":"2024-04-02T14:34:52.246514+0000","last_clean":"2024-04-02T14:34:52.246514+0000","last_became_active":"2024-04-02T14:32:22.553491+0000","last_became_peered":"2024-04-02T14:32:22.553491+0000","last_unstale":"2024-04-02T14:34:52.246514+0000","last_undegraded":"2024-04-02T14:34:52.246514+0000","last_fullsized":"2024-04-02T14:34:52.246514+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:51:30.785380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"90'11","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.546344+0000","last_change":"2024-04-02T14:32:22.538158+0000","last_active":"2024-04-02T14:34:52.546344+0000","last_peered":"2024-04-02T14:34:52.546344+0000","last_clean":"2024-04-02T14:34:52.546344+0000","last_became_active":"2024-04-02T14:32:22.537796+0000","last_became_peered":"2024-04-02T14:32:22.537796+0000","last_unstale":"2024-04-02T14:34:52.546344+0000","last_undegraded":"2024-04-02T14:34:52.546344+0000","last_fullsized":"2024-04-02T14:34:52.546344+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:09:48.054965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"90'16","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.381066+0000","last_change":"2024-04-02T14:32:22.535225+0000","last_active":"2024-04-02T14:34:52.381066+0000","last_peered":"2024-04-02T14:34:52.381066+0000","last_clean":"2024-04-02T14:34:52.381066+0000","last_became_active":"2024-04-02T14:32:22.534856+0000","last_became_peered":"2024-04-02T14:32:22.534856+0000","last_unstale":"2024-04-02T14:34:52.381066+0000","last_undegraded":"2024-04-02T14:34:52.381066+0000","last_fullsized":"2024-04-02T14:34:52.381066+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:22:11.118918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"90'6","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.545055+0000","last_change":"2024-04-02T14:32:22.539138+0000","last_active":"2024-04-02T14:34:52.545055+0000","last_peered":"2024-04-02T14:34:52.545055+0000","last_clean":"2024-04-02T14:34:52.545055+0000","last_became_active":"2024-04-02T14:32:22.538762+0000","last_became_peered":"2024-04-02T14:32:22.538762+0000","last_unstale":"2024-04-02T14:34:52.545055+0000","last_undegraded":"2024-04-02T14:34:52.545055+0000","last_fullsized":"2024-04-02T14:34:52.545055+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:18:59.174339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"90'7","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.247639+0000","last_change":"2024-04-02T14:32:22.536383+0000","last_active":"2024-04-02T14:34:52.247639+0000","last_peered":"2024-04-02T14:34:52.247639+0000","last_clean":"2024-04-02T14:34:52.247639+0000","last_became_active":"2024-04-02T14:32:22.536043+0000","last_became_peered":"2024-04-02T14:32:22.536043+0000","last_unstale":"2024-04-02T14:34:52.247639+0000","last_undegraded":"2024-04-02T14:34:52.247639+0000","last_fullsized":"2024-04-02T14:34:52.247639+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:14:48.628684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"90'10","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.545677+0000","last_change":"2024-04-02T14:32:22.552512+0000","last_active":"2024-04-02T14:34:52.545677+0000","last_peered":"2024-04-02T14:34:52.545677+0000","last_clean":"2024-04-02T14:34:52.545677+0000","last_became_active":"2024-04-02T14:32:22.549502+0000","last_became_peered":"2024-04-02T14:32:22.549502+0000","last_unstale":"2024-04-02T14:34:52.545677+0000","last_undegraded":"2024-04-02T14:34:52.545677+0000","last_fullsized":"2024-04-02T14:34:52.545677+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:37:45.609280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"90'7","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.547684+0000","last_change":"2024-04-02T14:32:22.555074+0000","last_active":"2024-04-02T14:34:52.547684+0000","last_peered":"2024-04-02T14:34:52.547684+0000","last_clean":"2024-04-02T14:34:52.547684+0000","last_became_active":"2024-04-02T14:32:22.554422+0000","last_became_peered":"2024-04-02T14:32:22.554422+0000","last_unstale":"2024-04-02T14:34:52.547684+0000","last_undegraded":"2024-04-02T14:34:52.547684+0000","last_fullsized":"2024-04-02T14:34:52.547684+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:27:49.536941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"90'7","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.547699+0000","last_change":"2024-04-02T14:32:22.550322+0000","last_active":"2024-04-02T14:34:52.547699+0000","last_peered":"2024-04-02T14:34:52.547699+0000","last_clean":"2024-04-02T14:34:52.547699+0000","last_became_active":"2024-04-02T14:32:22.548710+0000","last_became_peered":"2024-04-02T14:32:22.548710+0000","last_unstale":"2024-04-02T14:34:52.547699+0000","last_undegraded":"2024-04-02T14:34:52.547699+0000","last_fullsized":"2024-04-02T14:34:52.547699+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:19:09.226556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"90'9","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.545820+0000","last_change":"2024-04-02T14:32:22.558705+0000","last_active":"2024-04-02T14:34:52.545820+0000","last_peered":"2024-04-02T14:34:52.545820+0000","last_clean":"2024-04-02T14:34:52.545820+0000","last_became_active":"2024-04-02T14:32:22.558309+0000","last_became_peered":"2024-04-02T14:32:22.558309+0000","last_unstale":"2024-04-02T14:34:52.545820+0000","last_undegraded":"2024-04-02T14:34:52.545820+0000","last_fullsized":"2024-04-02T14:34:52.545820+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:52:23.929627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"90'8","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.549815+0000","last_change":"2024-04-02T14:32:22.551211+0000","last_active":"2024-04-02T14:34:52.549815+0000","last_peered":"2024-04-02T14:34:52.549815+0000","last_clean":"2024-04-02T14:34:52.549815+0000","last_became_active":"2024-04-02T14:32:22.550882+0000","last_became_peered":"2024-04-02T14:32:22.550882+0000","last_unstale":"2024-04-02T14:34:52.549815+0000","last_undegraded":"2024-04-02T14:34:52.549815+0000","last_fullsized":"2024-04-02T14:34:52.549815+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:54:54.533063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"90'10","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.368148+0000","last_change":"2024-04-02T14:32:22.550692+0000","last_active":"2024-04-02T14:34:52.368148+0000","last_peered":"2024-04-02T14:34:52.368148+0000","last_clean":"2024-04-02T14:34:52.368148+0000","last_became_active":"2024-04-02T14:32:22.550292+0000","last_became_peered":"2024-04-02T14:32:22.550292+0000","last_unstale":"2024-04-02T14:34:52.368148+0000","last_undegraded":"2024-04-02T14:34:52.368148+0000","last_fullsized":"2024-04-02T14:34:52.368148+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:14:42.317649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"90'7","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.283769+0000","last_change":"2024-04-02T14:32:22.553042+0000","last_active":"2024-04-02T14:34:52.283769+0000","last_peered":"2024-04-02T14:34:52.283769+0000","last_clean":"2024-04-02T14:34:52.283769+0000","last_became_active":"2024-04-02T14:32:22.552010+0000","last_became_peered":"2024-04-02T14:32:22.552010+0000","last_unstale":"2024-04-02T14:34:52.283769+0000","last_undegraded":"2024-04-02T14:34:52.283769+0000","last_fullsized":"2024-04-02T14:34:52.283769+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:31:13.947662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"88'13","reported_seq":199,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.367999+0000","last_change":"2024-04-02T14:32:35.868597+0000","last_active":"2024-04-02T14:34:52.367999+0000","last_peered":"2024-04-02T14:34:52.367999+0000","last_clean":"2024-04-02T14:34:52.367999+0000","last_became_active":"2024-04-02T14:32:35.851962+0000","last_became_peered":"2024-04-02T14:32:35.851962+0000","last_unstale":"2024-04-02T14:34:52.367999+0000","last_undegraded":"2024-04-02T14:34:52.367999+0000","last_fullsized":"2024-04-02T14:34:52.367999+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:17:06.400744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":140,"num_read_kb":140,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":672,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,4],"acting":[9,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"90'5","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.247056+0000","last_change":"2024-04-02T14:32:22.549526+0000","last_active":"2024-04-02T14:34:52.247056+0000","last_peered":"2024-04-02T14:34:52.247056+0000","last_clean":"2024-04-02T14:34:52.247056+0000","last_became_active":"2024-04-02T14:32:22.548250+0000","last_became_peered":"2024-04-02T14:32:22.548250+0000","last_unstale":"2024-04-02T14:34:52.247056+0000","last_undegraded":"2024-04-02T14:34:52.247056+0000","last_fullsized":"2024-04-02T14:34:52.247056+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:35:34.866803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"88'2","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.247157+0000","last_change":"2024-04-02T14:32:21.865084+0000","last_active":"2024-04-02T14:34:52.247157+0000","last_peered":"2024-04-02T14:34:52.247157+0000","last_clean":"2024-04-02T14:34:52.247157+0000","last_became_active":"2024-04-02T14:32:21.864718+0000","last_became_peered":"2024-04-02T14:32:21.864718+0000","last_unstale":"2024-04-02T14:34:52.247157+0000","last_undegraded":"2024-04-02T14:34:52.247157+0000","last_fullsized":"2024-04-02T14:34:52.247157+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:10:50.611362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"90'6","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.549008+0000","last_change":"2024-04-02T14:32:22.549443+0000","last_active":"2024-04-02T14:34:52.549008+0000","last_peered":"2024-04-02T14:34:52.549008+0000","last_clean":"2024-04-02T14:34:52.549008+0000","last_became_active":"2024-04-02T14:32:22.549059+0000","last_became_peered":"2024-04-02T14:32:22.549059+0000","last_unstale":"2024-04-02T14:34:52.549008+0000","last_undegraded":"2024-04-02T14:34:52.549008+0000","last_fullsized":"2024-04-02T14:34:52.549008+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:04:21.489864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"88'12","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.248319+0000","last_change":"2024-04-02T14:32:21.841368+0000","last_active":"2024-04-02T14:34:52.248319+0000","last_peered":"2024-04-02T14:34:52.248319+0000","last_clean":"2024-04-02T14:34:52.248319+0000","last_became_active":"2024-04-02T14:32:21.840963+0000","last_became_peered":"2024-04-02T14:32:21.840963+0000","last_unstale":"2024-04-02T14:34:52.248319+0000","last_undegraded":"2024-04-02T14:34:52.248319+0000","last_fullsized":"2024-04-02T14:34:52.248319+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:36:40.549720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"90'9","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.546094+0000","last_change":"2024-04-02T14:32:22.549753+0000","last_active":"2024-04-02T14:34:52.546094+0000","last_peered":"2024-04-02T14:34:52.546094+0000","last_clean":"2024-04-02T14:34:52.546094+0000","last_became_active":"2024-04-02T14:32:22.549354+0000","last_became_peered":"2024-04-02T14:32:22.549354+0000","last_unstale":"2024-04-02T14:34:52.546094+0000","last_undegraded":"2024-04-02T14:34:52.546094+0000","last_fullsized":"2024-04-02T14:34:52.546094+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T02:17:09.650946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"88'5","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.248127+0000","last_change":"2024-04-02T14:32:21.842766+0000","last_active":"2024-04-02T14:34:52.248127+0000","last_peered":"2024-04-02T14:34:52.248127+0000","last_clean":"2024-04-02T14:34:52.248127+0000","last_became_active":"2024-04-02T14:32:21.842410+0000","last_became_peered":"2024-04-02T14:32:21.842410+0000","last_unstale":"2024-04-02T14:34:52.248127+0000","last_undegraded":"2024-04-02T14:34:52.248127+0000","last_fullsized":"2024-04-02T14:34:52.248127+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:49:31.651127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"90'15","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.377579+0000","last_change":"2024-04-02T14:32:22.538506+0000","last_active":"2024-04-02T14:34:52.377579+0000","last_peered":"2024-04-02T14:34:52.377579+0000","last_clean":"2024-04-02T14:34:52.377579+0000","last_became_active":"2024-04-02T14:32:22.538168+0000","last_became_peered":"2024-04-02T14:32:22.538168+0000","last_unstale":"2024-04-02T14:34:52.377579+0000","last_undegraded":"2024-04-02T14:34:52.377579+0000","last_fullsized":"2024-04-02T14:34:52.377579+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:30:04.746382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"90'7","reported_seq":176,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:53.040014+0000","last_change":"2024-04-02T14:32:21.844180+0000","last_active":"2024-04-02T14:34:53.040014+0000","last_peered":"2024-04-02T14:34:53.040014+0000","last_clean":"2024-04-02T14:34:53.040014+0000","last_became_active":"2024-04-02T14:32:21.843840+0000","last_became_peered":"2024-04-02T14:32:21.843840+0000","last_unstale":"2024-04-02T14:34:53.040014+0000","last_undegraded":"2024-04-02T14:34:53.040014+0000","last_fullsized":"2024-04-02T14:34:53.040014+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T01:56:10.767946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1154,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":147,"num_read_kb":3,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"90'7","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.247921+0000","last_change":"2024-04-02T14:32:22.548453+0000","last_active":"2024-04-02T14:34:52.247921+0000","last_peered":"2024-04-02T14:34:52.247921+0000","last_clean":"2024-04-02T14:34:52.247921+0000","last_became_active":"2024-04-02T14:32:22.548110+0000","last_became_peered":"2024-04-02T14:32:22.548110+0000","last_unstale":"2024-04-02T14:34:52.247921+0000","last_undegraded":"2024-04-02T14:34:52.247921+0000","last_fullsized":"2024-04-02T14:34:52.247921+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:57:13.067425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"88'7","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.380564+0000","last_change":"2024-04-02T14:32:21.846528+0000","last_active":"2024-04-02T14:34:52.380564+0000","last_peered":"2024-04-02T14:34:52.380564+0000","last_clean":"2024-04-02T14:34:52.380564+0000","last_became_active":"2024-04-02T14:32:21.846152+0000","last_became_peered":"2024-04-02T14:32:21.846152+0000","last_unstale":"2024-04-02T14:34:52.380564+0000","last_undegraded":"2024-04-02T14:34:52.380564+0000","last_fullsized":"2024-04-02T14:34:52.380564+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:58:09.286256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"90'8","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.377374+0000","last_change":"2024-04-02T14:32:22.546730+0000","last_active":"2024-04-02T14:34:52.377374+0000","last_peered":"2024-04-02T14:34:52.377374+0000","last_clean":"2024-04-02T14:34:52.377374+0000","last_became_active":"2024-04-02T14:32:22.546342+0000","last_became_peered":"2024-04-02T14:32:22.546342+0000","last_unstale":"2024-04-02T14:34:52.377374+0000","last_undegraded":"2024-04-02T14:34:52.377374+0000","last_fullsized":"2024-04-02T14:34:52.377374+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:38:37.616382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"88'3","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.545031+0000","last_change":"2024-04-02T14:32:21.853316+0000","last_active":"2024-04-02T14:34:52.545031+0000","last_peered":"2024-04-02T14:34:52.545031+0000","last_clean":"2024-04-02T14:34:52.545031+0000","last_became_active":"2024-04-02T14:32:21.852912+0000","last_became_peered":"2024-04-02T14:32:21.852912+0000","last_unstale":"2024-04-02T14:34:52.545031+0000","last_undegraded":"2024-04-02T14:34:52.545031+0000","last_fullsized":"2024-04-02T14:34:52.545031+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T02:30:44.601900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"90'8","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.246607+0000","last_change":"2024-04-02T14:32:22.558913+0000","last_active":"2024-04-02T14:34:52.246607+0000","last_peered":"2024-04-02T14:34:52.246607+0000","last_clean":"2024-04-02T14:34:52.246607+0000","last_became_active":"2024-04-02T14:32:22.558511+0000","last_became_peered":"2024-04-02T14:32:22.558511+0000","last_unstale":"2024-04-02T14:34:52.246607+0000","last_undegraded":"2024-04-02T14:34:52.246607+0000","last_fullsized":"2024-04-02T14:34:52.246607+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:48:36.449086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"88'3","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.377293+0000","last_change":"2024-04-02T14:32:21.844081+0000","last_active":"2024-04-02T14:34:52.377293+0000","last_peered":"2024-04-02T14:34:52.377293+0000","last_clean":"2024-04-02T14:34:52.377293+0000","last_became_active":"2024-04-02T14:32:21.843788+0000","last_became_peered":"2024-04-02T14:32:21.843788+0000","last_unstale":"2024-04-02T14:34:52.377293+0000","last_undegraded":"2024-04-02T14:34:52.377293+0000","last_fullsized":"2024-04-02T14:34:52.377293+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:19:02.184229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"90'11","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.367476+0000","last_change":"2024-04-02T14:32:22.556193+0000","last_active":"2024-04-02T14:34:52.367476+0000","last_peered":"2024-04-02T14:34:52.367476+0000","last_clean":"2024-04-02T14:34:52.367476+0000","last_became_active":"2024-04-02T14:32:22.555787+0000","last_became_peered":"2024-04-02T14:32:22.555787+0000","last_unstale":"2024-04-02T14:34:52.367476+0000","last_undegraded":"2024-04-02T14:34:52.367476+0000","last_fullsized":"2024-04-02T14:34:52.367476+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T02:02:13.730941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"91'6","reported_seq":316,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:53.039409+0000","last_change":"2024-04-02T14:32:21.840651+0000","last_active":"2024-04-02T14:34:53.039409+0000","last_peered":"2024-04-02T14:34:53.039409+0000","last_clean":"2024-04-02T14:34:53.039409+0000","last_became_active":"2024-04-02T14:32:21.840344+0000","last_became_peered":"2024-04-02T14:32:21.840344+0000","last_unstale":"2024-04-02T14:34:53.039409+0000","last_undegraded":"2024-04-02T14:34:53.039409+0000","last_fullsized":"2024-04-02T14:34:53.039409+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:49:22.384224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":288,"num_read_kb":288,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"90'10","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.546191+0000","last_change":"2024-04-02T14:32:22.548245+0000","last_active":"2024-04-02T14:34:52.546191+0000","last_peered":"2024-04-02T14:34:52.546191+0000","last_clean":"2024-04-02T14:34:52.546191+0000","last_became_active":"2024-04-02T14:32:22.547853+0000","last_became_peered":"2024-04-02T14:32:22.547853+0000","last_unstale":"2024-04-02T14:34:52.546191+0000","last_undegraded":"2024-04-02T14:34:52.546191+0000","last_fullsized":"2024-04-02T14:34:52.546191+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:02:43.036944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"88'4","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.544349+0000","last_change":"2024-04-02T14:32:21.842096+0000","last_active":"2024-04-02T14:34:52.544349+0000","last_peered":"2024-04-02T14:34:52.544349+0000","last_clean":"2024-04-02T14:34:52.544349+0000","last_became_active":"2024-04-02T14:32:21.841646+0000","last_became_peered":"2024-04-02T14:32:21.841646+0000","last_unstale":"2024-04-02T14:34:52.544349+0000","last_undegraded":"2024-04-02T14:34:52.544349+0000","last_fullsized":"2024-04-02T14:34:52.544349+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:04:08.149785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"90'9","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.548949+0000","last_change":"2024-04-02T14:32:22.557694+0000","last_active":"2024-04-02T14:34:52.548949+0000","last_peered":"2024-04-02T14:34:52.548949+0000","last_clean":"2024-04-02T14:34:52.548949+0000","last_became_active":"2024-04-02T14:32:22.557202+0000","last_became_peered":"2024-04-02T14:32:22.557202+0000","last_unstale":"2024-04-02T14:34:52.548949+0000","last_undegraded":"2024-04-02T14:34:52.548949+0000","last_fullsized":"2024-04-02T14:34:52.548949+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:35:41.471557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"88'5","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.377599+0000","last_change":"2024-04-02T14:32:21.864366+0000","last_active":"2024-04-02T14:34:52.377599+0000","last_peered":"2024-04-02T14:34:52.377599+0000","last_clean":"2024-04-02T14:34:52.377599+0000","last_became_active":"2024-04-02T14:32:21.864006+0000","last_became_peered":"2024-04-02T14:32:21.864006+0000","last_unstale":"2024-04-02T14:34:52.377599+0000","last_undegraded":"2024-04-02T14:34:52.377599+0000","last_fullsized":"2024-04-02T14:34:52.377599+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:08:43.176061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"90'11","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.546034+0000","last_change":"2024-04-02T14:32:22.541562+0000","last_active":"2024-04-02T14:34:52.546034+0000","last_peered":"2024-04-02T14:34:52.546034+0000","last_clean":"2024-04-02T14:34:52.546034+0000","last_became_active":"2024-04-02T14:32:22.541209+0000","last_became_peered":"2024-04-02T14:32:22.541209+0000","last_unstale":"2024-04-02T14:34:52.546034+0000","last_undegraded":"2024-04-02T14:34:52.546034+0000","last_fullsized":"2024-04-02T14:34:52.546034+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:49:40.310528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"91'38","reported_seq":376,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:53.062247+0000","last_change":"2024-04-02T14:32:35.875429+0000","last_active":"2024-04-02T14:34:53.062247+0000","last_peered":"2024-04-02T14:34:53.062247+0000","last_clean":"2024-04-02T14:34:53.062247+0000","last_became_active":"2024-04-02T14:32:35.852417+0000","last_became_peered":"2024-04-02T14:32:35.852417+0000","last_unstale":"2024-04-02T14:34:53.062247+0000","last_undegraded":"2024-04-02T14:34:53.062247+0000","last_fullsized":"2024-04-02T14:34:53.062247+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:33:53.903712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67756,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":289,"num_read_kb":205,"num_write":40,"num_write_kb":82,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":43377,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,4],"acting":[10,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"90'6","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.248005+0000","last_change":"2024-04-02T14:32:22.535716+0000","last_active":"2024-04-02T14:34:52.248005+0000","last_peered":"2024-04-02T14:34:52.248005+0000","last_clean":"2024-04-02T14:34:52.248005+0000","last_became_active":"2024-04-02T14:32:22.534738+0000","last_became_peered":"2024-04-02T14:32:22.534738+0000","last_unstale":"2024-04-02T14:34:52.248005+0000","last_undegraded":"2024-04-02T14:34:52.248005+0000","last_fullsized":"2024-04-02T14:34:52.248005+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T01:25:07.763915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"88'9","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.283698+0000","last_change":"2024-04-02T14:32:21.854582+0000","last_active":"2024-04-02T14:34:52.283698+0000","last_peered":"2024-04-02T14:34:52.283698+0000","last_clean":"2024-04-02T14:34:52.283698+0000","last_became_active":"2024-04-02T14:32:21.854226+0000","last_became_peered":"2024-04-02T14:32:21.854226+0000","last_unstale":"2024-04-02T14:34:52.283698+0000","last_undegraded":"2024-04-02T14:34:52.283698+0000","last_fullsized":"2024-04-02T14:34:52.283698+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:30:02.289163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"90'9","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.545367+0000","last_change":"2024-04-02T14:32:22.555901+0000","last_active":"2024-04-02T14:34:52.545367+0000","last_peered":"2024-04-02T14:34:52.545367+0000","last_clean":"2024-04-02T14:34:52.545367+0000","last_became_active":"2024-04-02T14:32:22.555158+0000","last_became_peered":"2024-04-02T14:32:22.555158+0000","last_unstale":"2024-04-02T14:34:52.545367+0000","last_undegraded":"2024-04-02T14:34:52.545367+0000","last_fullsized":"2024-04-02T14:34:52.545367+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:04:28.766185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"88'5","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.377201+0000","last_change":"2024-04-02T14:32:21.855982+0000","last_active":"2024-04-02T14:34:52.377201+0000","last_peered":"2024-04-02T14:34:52.377201+0000","last_clean":"2024-04-02T14:34:52.377201+0000","last_became_active":"2024-04-02T14:32:21.855591+0000","last_became_peered":"2024-04-02T14:32:21.855591+0000","last_unstale":"2024-04-02T14:34:52.377201+0000","last_undegraded":"2024-04-02T14:34:52.377201+0000","last_fullsized":"2024-04-02T14:34:52.377201+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:28:51.806779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"90'15","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.544517+0000","last_change":"2024-04-02T14:32:22.549889+0000","last_active":"2024-04-02T14:34:52.544517+0000","last_peered":"2024-04-02T14:34:52.544517+0000","last_clean":"2024-04-02T14:34:52.544517+0000","last_became_active":"2024-04-02T14:32:22.549479+0000","last_became_peered":"2024-04-02T14:32:22.549479+0000","last_unstale":"2024-04-02T14:34:52.544517+0000","last_undegraded":"2024-04-02T14:34:52.544517+0000","last_fullsized":"2024-04-02T14:34:52.544517+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:29:25.937576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"91'9","reported_seq":179,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:53.156649+0000","last_change":"2024-04-02T14:32:21.838165+0000","last_active":"2024-04-02T14:34:53.156649+0000","last_peered":"2024-04-02T14:34:53.156649+0000","last_clean":"2024-04-02T14:34:53.156649+0000","last_became_active":"2024-04-02T14:32:21.837808+0000","last_became_peered":"2024-04-02T14:32:21.837808+0000","last_unstale":"2024-04-02T14:34:53.156649+0000","last_undegraded":"2024-04-02T14:34:53.156649+0000","last_fullsized":"2024-04-02T14:34:53.156649+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:00:44.605314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1902,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":149,"num_read_kb":5,"num_write":9,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"90'6","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.543871+0000","last_change":"2024-04-02T14:32:22.532937+0000","last_active":"2024-04-02T14:34:52.543871+0000","last_peered":"2024-04-02T14:34:52.543871+0000","last_clean":"2024-04-02T14:34:52.543871+0000","last_became_active":"2024-04-02T14:32:22.532586+0000","last_became_peered":"2024-04-02T14:32:22.532586+0000","last_unstale":"2024-04-02T14:34:52.543871+0000","last_undegraded":"2024-04-02T14:34:52.543871+0000","last_fullsized":"2024-04-02T14:34:52.543871+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:50:48.030889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"88'17","reported_seq":40,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.247642+0000","last_change":"2024-04-02T14:32:21.860032+0000","last_active":"2024-04-02T14:34:52.247642+0000","last_peered":"2024-04-02T14:34:52.247642+0000","last_clean":"2024-04-02T14:34:52.247642+0000","last_became_active":"2024-04-02T14:32:21.859653+0000","last_became_peered":"2024-04-02T14:32:21.859653+0000","last_unstale":"2024-04-02T14:34:52.247642+0000","last_undegraded":"2024-04-02T14:34:52.247642+0000","last_fullsized":"2024-04-02T14:34:52.247642+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:06:01.144530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"90'6","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.367598+0000","last_change":"2024-04-02T14:32:35.736133+0000","last_active":"2024-04-02T14:34:52.367598+0000","last_peered":"2024-04-02T14:34:52.367598+0000","last_clean":"2024-04-02T14:34:52.367598+0000","last_became_active":"2024-04-02T14:32:35.734335+0000","last_became_peered":"2024-04-02T14:32:35.734335+0000","last_unstale":"2024-04-02T14:34:52.367598+0000","last_undegraded":"2024-04-02T14:34:52.367598+0000","last_fullsized":"2024-04-02T14:34:52.367598+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:36:21.828318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,1],"acting":[9,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.f","version":"91'3","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.248316+0000","last_change":"2024-04-02T14:32:21.842384+0000","last_active":"2024-04-02T14:34:52.248316+0000","last_peered":"2024-04-02T14:34:52.248316+0000","last_clean":"2024-04-02T14:34:52.248316+0000","last_became_active":"2024-04-02T14:32:21.842040+0000","last_became_peered":"2024-04-02T14:32:21.842040+0000","last_unstale":"2024-04-02T14:34:52.248316+0000","last_undegraded":"2024-04-02T14:34:52.248316+0000","last_fullsized":"2024-04-02T14:34:52.248316+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:33:29.541972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"90'5","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.549637+0000","last_change":"2024-04-02T14:32:22.538571+0000","last_active":"2024-04-02T14:34:52.549637+0000","last_peered":"2024-04-02T14:34:52.549637+0000","last_clean":"2024-04-02T14:34:52.549637+0000","last_became_active":"2024-04-02T14:32:22.538239+0000","last_became_peered":"2024-04-02T14:32:22.538239+0000","last_unstale":"2024-04-02T14:34:52.549637+0000","last_undegraded":"2024-04-02T14:34:52.549637+0000","last_fullsized":"2024-04-02T14:34:52.549637+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T01:19:53.120419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"88'1","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.367816+0000","last_change":"2024-04-02T14:32:21.857666+0000","last_active":"2024-04-02T14:34:52.367816+0000","last_peered":"2024-04-02T14:34:52.367816+0000","last_clean":"2024-04-02T14:34:52.367816+0000","last_became_active":"2024-04-02T14:32:21.856618+0000","last_became_peered":"2024-04-02T14:32:21.856618+0000","last_unstale":"2024-04-02T14:34:52.367816+0000","last_undegraded":"2024-04-02T14:34:52.367816+0000","last_fullsized":"2024-04-02T14:34:52.367816+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T01:20:53.183052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"90'5","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.380890+0000","last_change":"2024-04-02T14:32:22.552277+0000","last_active":"2024-04-02T14:34:52.380890+0000","last_peered":"2024-04-02T14:34:52.380890+0000","last_clean":"2024-04-02T14:34:52.380890+0000","last_became_active":"2024-04-02T14:32:22.551882+0000","last_became_peered":"2024-04-02T14:32:22.551882+0000","last_unstale":"2024-04-02T14:34:52.380890+0000","last_undegraded":"2024-04-02T14:34:52.380890+0000","last_fullsized":"2024-04-02T14:34:52.380890+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:30:43.592910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"88'5","reported_seq":171,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.544165+0000","last_change":"2024-04-02T14:32:21.848258+0000","last_active":"2024-04-02T14:34:52.544165+0000","last_peered":"2024-04-02T14:34:52.544165+0000","last_clean":"2024-04-02T14:34:52.544165+0000","last_became_active":"2024-04-02T14:32:21.847805+0000","last_became_peered":"2024-04-02T14:32:21.847805+0000","last_unstale":"2024-04-02T14:34:52.544165+0000","last_undegraded":"2024-04-02T14:34:52.544165+0000","last_fullsized":"2024-04-02T14:34:52.544165+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:46:07.552786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":144,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"90'7","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.380644+0000","last_change":"2024-04-02T14:32:22.560835+0000","last_active":"2024-04-02T14:34:52.380644+0000","last_peered":"2024-04-02T14:34:52.380644+0000","last_clean":"2024-04-02T14:34:52.380644+0000","last_became_active":"2024-04-02T14:32:22.560469+0000","last_became_peered":"2024-04-02T14:32:22.560469+0000","last_unstale":"2024-04-02T14:34:52.380644+0000","last_undegraded":"2024-04-02T14:34:52.380644+0000","last_fullsized":"2024-04-02T14:34:52.380644+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T02:29:24.869650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"90'8","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.377408+0000","last_change":"2024-04-02T14:32:22.538325+0000","last_active":"2024-04-02T14:34:52.377408+0000","last_peered":"2024-04-02T14:34:52.377408+0000","last_clean":"2024-04-02T14:34:52.377408+0000","last_became_active":"2024-04-02T14:32:22.537903+0000","last_became_peered":"2024-04-02T14:32:22.537903+0000","last_unstale":"2024-04-02T14:34:52.377408+0000","last_undegraded":"2024-04-02T14:34:52.377408+0000","last_fullsized":"2024-04-02T14:34:52.377408+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:12:26.586290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"91'7","reported_seq":174,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.549482+0000","last_change":"2024-04-02T14:32:21.861333+0000","last_active":"2024-04-02T14:34:52.549482+0000","last_peered":"2024-04-02T14:34:52.549482+0000","last_clean":"2024-04-02T14:34:52.549482+0000","last_became_active":"2024-04-02T14:32:21.860909+0000","last_became_peered":"2024-04-02T14:32:21.860909+0000","last_unstale":"2024-04-02T14:34:52.549482+0000","last_undegraded":"2024-04-02T14:34:52.549482+0000","last_fullsized":"2024-04-02T14:34:52.549482+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:21:52.616036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":145,"num_read_kb":145,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"90'8","reported_seq":22,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.367541+0000","last_change":"2024-04-02T14:32:35.736379+0000","last_active":"2024-04-02T14:34:52.367541+0000","last_peered":"2024-04-02T14:34:52.367541+0000","last_clean":"2024-04-02T14:34:52.367541+0000","last_became_active":"2024-04-02T14:32:35.735156+0000","last_became_peered":"2024-04-02T14:32:35.735156+0000","last_unstale":"2024-04-02T14:34:52.367541+0000","last_undegraded":"2024-04-02T14:34:52.367541+0000","last_fullsized":"2024-04-02T14:34:52.367541+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:16:28.405699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,1],"acting":[9,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.3","version":"88'6","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.247327+0000","last_change":"2024-04-02T14:32:21.837336+0000","last_active":"2024-04-02T14:34:52.247327+0000","last_peered":"2024-04-02T14:34:52.247327+0000","last_clean":"2024-04-02T14:34:52.247327+0000","last_became_active":"2024-04-02T14:32:21.836921+0000","last_became_peered":"2024-04-02T14:32:21.836921+0000","last_unstale":"2024-04-02T14:34:52.247327+0000","last_undegraded":"2024-04-02T14:34:52.247327+0000","last_fullsized":"2024-04-02T14:34:52.247327+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T14:34:52.883173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":358,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.543898+0000","last_change":"2024-04-02T14:31:02.710356+0000","last_active":"2024-04-02T14:34:52.543898+0000","last_peered":"2024-04-02T14:34:52.543898+0000","last_clean":"2024-04-02T14:34:52.543898+0000","last_became_active":"2024-04-02T14:31:02.709669+0000","last_became_peered":"2024-04-02T14:31:02.709669+0000","last_unstale":"2024-04-02T14:34:52.543898+0000","last_undegraded":"2024-04-02T14:34:52.543898+0000","last_fullsized":"2024-04-02T14:34:52.543898+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:27:17.358883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:27:17.358883+0000","last_clean_scrub_stamp":"2024-04-02T14:27:17.358883+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:45:55.893284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"90'5","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.544201+0000","last_change":"2024-04-02T14:32:22.545502+0000","last_active":"2024-04-02T14:34:52.544201+0000","last_peered":"2024-04-02T14:34:52.544201+0000","last_clean":"2024-04-02T14:34:52.544201+0000","last_became_active":"2024-04-02T14:32:22.545111+0000","last_became_peered":"2024-04-02T14:32:22.545111+0000","last_unstale":"2024-04-02T14:34:52.544201+0000","last_undegraded":"2024-04-02T14:34:52.544201+0000","last_fullsized":"2024-04-02T14:34:52.544201+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:02:32.178334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"88'4","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.549305+0000","last_change":"2024-04-02T14:32:35.736582+0000","last_active":"2024-04-02T14:34:52.549305+0000","last_peered":"2024-04-02T14:34:52.549305+0000","last_clean":"2024-04-02T14:34:52.549305+0000","last_became_active":"2024-04-02T14:32:35.733368+0000","last_became_peered":"2024-04-02T14:32:35.733368+0000","last_unstale":"2024-04-02T14:34:52.549305+0000","last_undegraded":"2024-04-02T14:34:52.549305+0000","last_fullsized":"2024-04-02T14:34:52.549305+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T02:14:35.713351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,8],"acting":[11,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"88'4","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.248107+0000","last_change":"2024-04-02T14:32:21.860226+0000","last_active":"2024-04-02T14:34:52.248107+0000","last_peered":"2024-04-02T14:34:52.248107+0000","last_clean":"2024-04-02T14:34:52.248107+0000","last_became_active":"2024-04-02T14:32:21.859854+0000","last_became_peered":"2024-04-02T14:32:21.859854+0000","last_unstale":"2024-04-02T14:34:52.248107+0000","last_undegraded":"2024-04-02T14:34:52.248107+0000","last_fullsized":"2024-04-02T14:34:52.248107+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T00:30:36.710677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"90'9","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.545807+0000","last_change":"2024-04-02T14:32:22.552287+0000","last_active":"2024-04-02T14:34:52.545807+0000","last_peered":"2024-04-02T14:34:52.545807+0000","last_clean":"2024-04-02T14:34:52.545807+0000","last_became_active":"2024-04-02T14:32:22.551719+0000","last_became_peered":"2024-04-02T14:32:22.551719+0000","last_unstale":"2024-04-02T14:34:52.545807+0000","last_undegraded":"2024-04-02T14:34:52.545807+0000","last_fullsized":"2024-04-02T14:34:52.545807+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:26:12.007600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"91'50","reported_seq":95,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.507926+0000","last_change":"2024-04-02T14:32:21.851671+0000","last_active":"2024-04-02T14:34:52.507926+0000","last_peered":"2024-04-02T14:34:52.507926+0000","last_clean":"2024-04-02T14:34:52.507926+0000","last_became_active":"2024-04-02T14:32:21.851297+0000","last_became_peered":"2024-04-02T14:32:21.851297+0000","last_unstale":"2024-04-02T14:34:52.507926+0000","last_undegraded":"2024-04-02T14:34:52.507926+0000","last_fullsized":"2024-04-02T14:34:52.507926+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:51:20.232086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":82,"num_read_kb":70,"num_write":88,"num_write_kb":838,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"90'5","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.247133+0000","last_change":"2024-04-02T14:32:22.554554+0000","last_active":"2024-04-02T14:34:52.247133+0000","last_peered":"2024-04-02T14:34:52.247133+0000","last_clean":"2024-04-02T14:34:52.247133+0000","last_became_active":"2024-04-02T14:32:22.554194+0000","last_became_peered":"2024-04-02T14:32:22.554194+0000","last_unstale":"2024-04-02T14:34:52.247133+0000","last_undegraded":"2024-04-02T14:34:52.247133+0000","last_fullsized":"2024-04-02T14:34:52.247133+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:27:51.382441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"88'7","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.247855+0000","last_change":"2024-04-02T14:32:21.850346+0000","last_active":"2024-04-02T14:34:52.247855+0000","last_peered":"2024-04-02T14:34:52.247855+0000","last_clean":"2024-04-02T14:34:52.247855+0000","last_became_active":"2024-04-02T14:32:21.850084+0000","last_became_peered":"2024-04-02T14:32:21.850084+0000","last_unstale":"2024-04-02T14:34:52.247855+0000","last_undegraded":"2024-04-02T14:34:52.247855+0000","last_fullsized":"2024-04-02T14:34:52.247855+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:56:17.774702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"90'7","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.548807+0000","last_change":"2024-04-02T14:32:22.557054+0000","last_active":"2024-04-02T14:34:52.548807+0000","last_peered":"2024-04-02T14:34:52.548807+0000","last_clean":"2024-04-02T14:34:52.548807+0000","last_became_active":"2024-04-02T14:32:22.556335+0000","last_became_peered":"2024-04-02T14:32:22.556335+0000","last_unstale":"2024-04-02T14:34:52.548807+0000","last_undegraded":"2024-04-02T14:34:52.548807+0000","last_fullsized":"2024-04-02T14:34:52.548807+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:06:48.907197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"88'2","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.545713+0000","last_change":"2024-04-02T14:32:21.860260+0000","last_active":"2024-04-02T14:34:52.545713+0000","last_peered":"2024-04-02T14:34:52.545713+0000","last_clean":"2024-04-02T14:34:52.545713+0000","last_became_active":"2024-04-02T14:32:21.859922+0000","last_became_peered":"2024-04-02T14:32:21.859922+0000","last_unstale":"2024-04-02T14:34:52.545713+0000","last_undegraded":"2024-04-02T14:34:52.545713+0000","last_fullsized":"2024-04-02T14:34:52.545713+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:07:45.850148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"90'2","reported_seq":22,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.247646+0000","last_change":"2024-04-02T14:32:22.536063+0000","last_active":"2024-04-02T14:34:52.247646+0000","last_peered":"2024-04-02T14:34:52.247646+0000","last_clean":"2024-04-02T14:34:52.247646+0000","last_became_active":"2024-04-02T14:32:22.534678+0000","last_became_peered":"2024-04-02T14:32:22.534678+0000","last_unstale":"2024-04-02T14:34:52.247646+0000","last_undegraded":"2024-04-02T14:34:52.247646+0000","last_fullsized":"2024-04-02T14:34:52.247646+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T02:07:26.186216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"88'7","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.246470+0000","last_change":"2024-04-02T14:32:21.848554+0000","last_active":"2024-04-02T14:34:52.246470+0000","last_peered":"2024-04-02T14:34:52.246470+0000","last_clean":"2024-04-02T14:34:52.246470+0000","last_became_active":"2024-04-02T14:32:21.848177+0000","last_became_peered":"2024-04-02T14:32:21.848177+0000","last_unstale":"2024-04-02T14:34:52.246470+0000","last_undegraded":"2024-04-02T14:34:52.246470+0000","last_fullsized":"2024-04-02T14:34:52.246470+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:47:04.213286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"90'9","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.549385+0000","last_change":"2024-04-02T14:32:22.547246+0000","last_active":"2024-04-02T14:34:52.549385+0000","last_peered":"2024-04-02T14:34:52.549385+0000","last_clean":"2024-04-02T14:34:52.549385+0000","last_became_active":"2024-04-02T14:32:22.546721+0000","last_became_peered":"2024-04-02T14:32:22.546721+0000","last_unstale":"2024-04-02T14:34:52.549385+0000","last_undegraded":"2024-04-02T14:34:52.549385+0000","last_fullsized":"2024-04-02T14:34:52.549385+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:11:30.615723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"88'3","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.549450+0000","last_change":"2024-04-02T14:32:35.743474+0000","last_active":"2024-04-02T14:34:52.549450+0000","last_peered":"2024-04-02T14:34:52.549450+0000","last_clean":"2024-04-02T14:34:52.549450+0000","last_became_active":"2024-04-02T14:32:35.738522+0000","last_became_peered":"2024-04-02T14:32:35.738522+0000","last_unstale":"2024-04-02T14:34:52.549450+0000","last_undegraded":"2024-04-02T14:34:52.549450+0000","last_fullsized":"2024-04-02T14:34:52.549450+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T01:14:43.989279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"90'8","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.246782+0000","last_change":"2024-04-02T14:32:22.561787+0000","last_active":"2024-04-02T14:34:52.246782+0000","last_peered":"2024-04-02T14:34:52.246782+0000","last_clean":"2024-04-02T14:34:52.246782+0000","last_became_active":"2024-04-02T14:32:22.561284+0000","last_became_peered":"2024-04-02T14:32:22.561284+0000","last_unstale":"2024-04-02T14:34:52.246782+0000","last_undegraded":"2024-04-02T14:34:52.246782+0000","last_fullsized":"2024-04-02T14:34:52.246782+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:46:06.450097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"88'4","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.247746+0000","last_change":"2024-04-02T14:32:21.841340+0000","last_active":"2024-04-02T14:34:52.247746+0000","last_peered":"2024-04-02T14:34:52.247746+0000","last_clean":"2024-04-02T14:34:52.247746+0000","last_became_active":"2024-04-02T14:32:21.840937+0000","last_became_peered":"2024-04-02T14:32:21.840937+0000","last_unstale":"2024-04-02T14:34:52.247746+0000","last_undegraded":"2024-04-02T14:34:52.247746+0000","last_fullsized":"2024-04-02T14:34:52.247746+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T01:47:44.834440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"90'5","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.544165+0000","last_change":"2024-04-02T14:32:22.538890+0000","last_active":"2024-04-02T14:34:52.544165+0000","last_peered":"2024-04-02T14:34:52.544165+0000","last_clean":"2024-04-02T14:34:52.544165+0000","last_became_active":"2024-04-02T14:32:22.538512+0000","last_became_peered":"2024-04-02T14:32:22.538512+0000","last_unstale":"2024-04-02T14:34:52.544165+0000","last_undegraded":"2024-04-02T14:34:52.544165+0000","last_fullsized":"2024-04-02T14:34:52.544165+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T22:59:23.792851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"91'6","reported_seq":175,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:53.040074+0000","last_change":"2024-04-02T14:32:21.862727+0000","last_active":"2024-04-02T14:34:53.040074+0000","last_peered":"2024-04-02T14:34:53.040074+0000","last_clean":"2024-04-02T14:34:53.040074+0000","last_became_active":"2024-04-02T14:32:21.862405+0000","last_became_peered":"2024-04-02T14:32:21.862405+0000","last_unstale":"2024-04-02T14:34:53.040074+0000","last_undegraded":"2024-04-02T14:34:53.040074+0000","last_fullsized":"2024-04-02T14:34:53.040074+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T19:36:54.849964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1561,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":147,"num_read_kb":3,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"90'7","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.544768+0000","last_change":"2024-04-02T14:32:22.540077+0000","last_active":"2024-04-02T14:34:52.544768+0000","last_peered":"2024-04-02T14:34:52.544768+0000","last_clean":"2024-04-02T14:34:52.544768+0000","last_became_active":"2024-04-02T14:32:22.539513+0000","last_became_peered":"2024-04-02T14:32:22.539513+0000","last_unstale":"2024-04-02T14:34:52.544768+0000","last_undegraded":"2024-04-02T14:34:52.544768+0000","last_fullsized":"2024-04-02T14:34:52.544768+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T02:00:39.819963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"91'41","reported_seq":71,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.501385+0000","last_change":"2024-04-02T14:32:21.860310+0000","last_active":"2024-04-02T14:34:52.501385+0000","last_peered":"2024-04-02T14:34:52.501385+0000","last_clean":"2024-04-02T14:34:52.501385+0000","last_became_active":"2024-04-02T14:32:21.859955+0000","last_became_peered":"2024-04-02T14:32:21.859955+0000","last_unstale":"2024-04-02T14:34:52.501385+0000","last_undegraded":"2024-04-02T14:34:52.501385+0000","last_fullsized":"2024-04-02T14:34:52.501385+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:11:32.732603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131670,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":49,"num_write_kb":475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"90'7","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.544288+0000","last_change":"2024-04-02T14:32:22.552134+0000","last_active":"2024-04-02T14:34:52.544288+0000","last_peered":"2024-04-02T14:34:52.544288+0000","last_clean":"2024-04-02T14:34:52.544288+0000","last_became_active":"2024-04-02T14:32:22.551695+0000","last_became_peered":"2024-04-02T14:32:22.551695+0000","last_unstale":"2024-04-02T14:34:52.544288+0000","last_undegraded":"2024-04-02T14:34:52.544288+0000","last_fullsized":"2024-04-02T14:34:52.544288+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T21:09:36.902243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"88'5","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.545160+0000","last_change":"2024-04-02T14:32:21.857571+0000","last_active":"2024-04-02T14:34:52.545160+0000","last_peered":"2024-04-02T14:34:52.545160+0000","last_clean":"2024-04-02T14:34:52.545160+0000","last_became_active":"2024-04-02T14:32:21.856935+0000","last_became_peered":"2024-04-02T14:32:21.856935+0000","last_unstale":"2024-04-02T14:34:52.545160+0000","last_undegraded":"2024-04-02T14:34:52.545160+0000","last_fullsized":"2024-04-02T14:34:52.545160+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:20.811698+0000","last_clean_scrub_stamp":"2024-04-02T14:32:20.811698+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:55:46.079097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"90'9","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.544456+0000","last_change":"2024-04-02T14:32:22.534708+0000","last_active":"2024-04-02T14:34:52.544456+0000","last_peered":"2024-04-02T14:34:52.544456+0000","last_clean":"2024-04-02T14:34:52.544456+0000","last_became_active":"2024-04-02T14:32:22.534365+0000","last_became_peered":"2024-04-02T14:32:22.534365+0000","last_unstale":"2024-04-02T14:34:52.544456+0000","last_undegraded":"2024-04-02T14:34:52.544456+0000","last_fullsized":"2024-04-02T14:34:52.544456+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T23:34:47.435729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"90'9","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.549434+0000","last_change":"2024-04-02T14:32:22.538317+0000","last_active":"2024-04-02T14:34:52.549434+0000","last_peered":"2024-04-02T14:34:52.549434+0000","last_clean":"2024-04-02T14:34:52.549434+0000","last_became_active":"2024-04-02T14:32:22.537939+0000","last_became_peered":"2024-04-02T14:32:22.537939+0000","last_unstale":"2024-04-02T14:34:52.549434+0000","last_undegraded":"2024-04-02T14:34:52.549434+0000","last_fullsized":"2024-04-02T14:34:52.549434+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:31:47.075326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"90'12","reported_seq":38,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.283556+0000","last_change":"2024-04-02T14:32:35.852202+0000","last_active":"2024-04-02T14:34:52.283556+0000","last_peered":"2024-04-02T14:34:52.283556+0000","last_clean":"2024-04-02T14:34:52.283556+0000","last_became_active":"2024-04-02T14:32:35.851785+0000","last_became_peered":"2024-04-02T14:32:35.851785+0000","last_unstale":"2024-04-02T14:34:52.283556+0000","last_undegraded":"2024-04-02T14:34:52.283556+0000","last_fullsized":"2024-04-02T14:34:52.283556+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T15:09:17.410727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,4],"acting":[2,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"90'8","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.283542+0000","last_change":"2024-04-02T14:32:22.556567+0000","last_active":"2024-04-02T14:34:52.283542+0000","last_peered":"2024-04-02T14:34:52.283542+0000","last_clean":"2024-04-02T14:34:52.283542+0000","last_became_active":"2024-04-02T14:32:22.556267+0000","last_became_peered":"2024-04-02T14:32:22.556267+0000","last_unstale":"2024-04-02T14:34:52.283542+0000","last_undegraded":"2024-04-02T14:34:52.283542+0000","last_fullsized":"2024-04-02T14:34:52.283542+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:35:30.039013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"90'7","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.547650+0000","last_change":"2024-04-02T14:32:35.848125+0000","last_active":"2024-04-02T14:34:52.547650+0000","last_peered":"2024-04-02T14:34:52.547650+0000","last_clean":"2024-04-02T14:34:52.547650+0000","last_became_active":"2024-04-02T14:32:35.847681+0000","last_became_peered":"2024-04-02T14:32:35.847681+0000","last_unstale":"2024-04-02T14:34:52.547650+0000","last_undegraded":"2024-04-02T14:34:52.547650+0000","last_fullsized":"2024-04-02T14:34:52.547650+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:51:38.040261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"90'4","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.545854+0000","last_change":"2024-04-02T14:32:22.558584+0000","last_active":"2024-04-02T14:34:52.545854+0000","last_peered":"2024-04-02T14:34:52.545854+0000","last_clean":"2024-04-02T14:34:52.545854+0000","last_became_active":"2024-04-02T14:32:22.557382+0000","last_became_peered":"2024-04-02T14:32:22.557382+0000","last_unstale":"2024-04-02T14:34:52.545854+0000","last_undegraded":"2024-04-02T14:34:52.545854+0000","last_fullsized":"2024-04-02T14:34:52.545854+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:09:25.107411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"90'6","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.549497+0000","last_change":"2024-04-02T14:32:22.538318+0000","last_active":"2024-04-02T14:34:52.549497+0000","last_peered":"2024-04-02T14:34:52.549497+0000","last_clean":"2024-04-02T14:34:52.549497+0000","last_became_active":"2024-04-02T14:32:22.538011+0000","last_became_peered":"2024-04-02T14:32:22.538011+0000","last_unstale":"2024-04-02T14:34:52.549497+0000","last_undegraded":"2024-04-02T14:34:52.549497+0000","last_fullsized":"2024-04-02T14:34:52.549497+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:11:21.269106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"90'15","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.283948+0000","last_change":"2024-04-02T14:32:22.535645+0000","last_active":"2024-04-02T14:34:52.283948+0000","last_peered":"2024-04-02T14:34:52.283948+0000","last_clean":"2024-04-02T14:34:52.283948+0000","last_became_active":"2024-04-02T14:32:22.535314+0000","last_became_peered":"2024-04-02T14:32:22.535314+0000","last_unstale":"2024-04-02T14:34:52.283948+0000","last_undegraded":"2024-04-02T14:34:52.283948+0000","last_fullsized":"2024-04-02T14:34:52.283948+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:51:27.879015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"90'2","reported_seq":22,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.247021+0000","last_change":"2024-04-02T14:32:22.553959+0000","last_active":"2024-04-02T14:34:52.247021+0000","last_peered":"2024-04-02T14:34:52.247021+0000","last_clean":"2024-04-02T14:34:52.247021+0000","last_became_active":"2024-04-02T14:32:22.553389+0000","last_became_peered":"2024-04-02T14:32:22.553389+0000","last_unstale":"2024-04-02T14:34:52.247021+0000","last_undegraded":"2024-04-02T14:34:52.247021+0000","last_fullsized":"2024-04-02T14:34:52.247021+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T20:55:00.552600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"90'6","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.377384+0000","last_change":"2024-04-02T14:32:22.551769+0000","last_active":"2024-04-02T14:34:52.377384+0000","last_peered":"2024-04-02T14:34:52.377384+0000","last_clean":"2024-04-02T14:34:52.377384+0000","last_became_active":"2024-04-02T14:32:22.551422+0000","last_became_peered":"2024-04-02T14:32:22.551422+0000","last_unstale":"2024-04-02T14:34:52.377384+0000","last_undegraded":"2024-04-02T14:34:52.377384+0000","last_fullsized":"2024-04-02T14:34:52.377384+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-04T01:15:07.262927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"90'8","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.246610+0000","last_change":"2024-04-02T14:32:22.548264+0000","last_active":"2024-04-02T14:34:52.246610+0000","last_peered":"2024-04-02T14:34:52.246610+0000","last_clean":"2024-04-02T14:34:52.246610+0000","last_became_active":"2024-04-02T14:32:22.547293+0000","last_became_peered":"2024-04-02T14:32:22.547293+0000","last_unstale":"2024-04-02T14:34:52.246610+0000","last_undegraded":"2024-04-02T14:34:52.246610+0000","last_fullsized":"2024-04-02T14:34:52.246610+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T16:53:19.896243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"90'10","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.545911+0000","last_change":"2024-04-02T14:32:22.556131+0000","last_active":"2024-04-02T14:34:52.545911+0000","last_peered":"2024-04-02T14:34:52.545911+0000","last_clean":"2024-04-02T14:34:52.545911+0000","last_became_active":"2024-04-02T14:32:22.555763+0000","last_became_peered":"2024-04-02T14:32:22.555763+0000","last_unstale":"2024-04-02T14:34:52.545911+0000","last_undegraded":"2024-04-02T14:34:52.545911+0000","last_fullsized":"2024-04-02T14:34:52.545911+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:45:22.485403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"90'11","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.545531+0000","last_change":"2024-04-02T14:32:22.562093+0000","last_active":"2024-04-02T14:34:52.545531+0000","last_peered":"2024-04-02T14:34:52.545531+0000","last_clean":"2024-04-02T14:34:52.545531+0000","last_became_active":"2024-04-02T14:32:22.561663+0000","last_became_peered":"2024-04-02T14:32:22.561663+0000","last_unstale":"2024-04-02T14:34:52.545531+0000","last_undegraded":"2024-04-02T14:34:52.545531+0000","last_fullsized":"2024-04-02T14:34:52.545531+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T17:14:25.209276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"90'8","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-02T14:34:52.549387+0000","last_change":"2024-04-02T14:32:22.541964+0000","last_active":"2024-04-02T14:34:52.549387+0000","last_peered":"2024-04-02T14:34:52.549387+0000","last_clean":"2024-04-02T14:34:52.549387+0000","last_became_active":"2024-04-02T14:32:22.541549+0000","last_became_peered":"2024-04-02T14:32:22.541549+0000","last_unstale":"2024-04-02T14:34:52.549387+0000","last_undegraded":"2024-04-02T14:34:52.549387+0000","last_fullsized":"2024-04-02T14:34:52.549387+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T14:32:21.825339+0000","last_clean_scrub_stamp":"2024-04-02T14:32:21.825339+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-03T18:04:56.783158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":601231,"num_objects":100,"num_object_clones":0,"num_object_copies":300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":0,"num_read":1561,"num_read_kb":889,"num_write":372,"num_write_kb":1465,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":44049,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2174976,"data_stored":1803693,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":300,"ondisk_log_size":300,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":83,"seq":356482285611,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32168,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712984,"statfs":{"total":95995035648,"available":95962095616,"internally_reserved":0,"allocated":1011712,"data_stored":484076,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":75,"seq":322122547249,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":37000,"kb_used_data":1664,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708152,"statfs":{"total":95995035648,"available":95957147648,"internally_reserved":0,"allocated":1703936,"data_stored":1133256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":36174273},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743477,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36736,"kb_used_data":1396,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708416,"statfs":{"total":95995035648,"available":95957417984,"internally_reserved":0,"allocated":1429504,"data_stored":876831,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":61,"seq":261993005117,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32564,"kb_used_data":1384,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712588,"statfs":{"total":95995035648,"available":95961690112,"internally_reserved":0,"allocated":1417216,"data_stored":876537,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299458,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32772,"kb_used_data":1592,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712380,"statfs":{"total":95995035648,"available":95961477120,"internally_reserved":0,"allocated":1630208,"data_stored":1077966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495686,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36336,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708816,"statfs":{"total":95995035648,"available":95957827584,"internally_reserved":0,"allocated":1019904,"data_stored":483332,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574947,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36404,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708748,"statfs":{"total":95995035648,"available":95957757952,"internally_reserved":0,"allocated":1089536,"data_stored":552001,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738472,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36336,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708816,"statfs":{"total":95995035648,"available":95957827584,"internally_reserved":0,"allocated":1024000,"data_stored":475707,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411422,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36348,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708804,"statfs":{"total":95995035648,"available":95957815296,"internally_reserved":0,"allocated":1032192,"data_stored":484997,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182487,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28032,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717120,"statfs":{"total":95995035648,"available":95966330880,"internally_reserved":0,"allocated":1036288,"data_stored":483157,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920851,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36420,"kb_used_data":1080,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708732,"statfs":{"total":95995035648,"available":95957741568,"internally_reserved":0,"allocated":1105920,"data_stored":551371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757324,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28752,"kb_used_data":1736,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716400,"statfs":{"total":95995035648,"available":95965593600,"internally_reserved":0,"allocated":1777664,"data_stored":1207930,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2569,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":69508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":2895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":557056,"data_stored":525246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":70442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":180224,"data_stored":134952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":4988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":394044,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":430080,"data_stored":394338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":258048,"data_stored":202406,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-02T14:34:59.274 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T14:34:59.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:59.498+0000 7f49c5400640 1 -- 172.21.15.90:0/2803579826 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f49c0077700 msgr2=0x7f49c0077b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:59.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:59.498+0000 7f49c5400640 1 --2- 172.21.15.90:0/2803579826 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f49c0077700 0x7f49c0077b60 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f49ac009900 tx=0x7f49ac02bc30 comp rx=0 tx=0).stop 2024-04-02T14:34:59.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:59.498+0000 7f49c5400640 1 -- 172.21.15.90:0/2803579826 shutdown_connections 2024-04-02T14:34:59.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:59.498+0000 7f49c5400640 1 --2- 172.21.15.90:0/2803579826 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f49c00780a0 0x7f49c0078940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:59.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:59.498+0000 7f49c5400640 1 --2- 172.21.15.90:0/2803579826 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f49c0077700 0x7f49c0077b60 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:59.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:59.498+0000 7f49c5400640 1 --2- 172.21.15.90:0/2803579826 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f49c00764c0 0x7f49c00768e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:59.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:59.498+0000 7f49c5400640 1 -- 172.21.15.90:0/2803579826 >> 172.21.15.90:0/2803579826 conn(0x7f49c00ff2b0 msgr2=0x7f49c01016f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:34:59.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:59.498+0000 7f49c5400640 1 -- 172.21.15.90:0/2803579826 shutdown_connections 2024-04-02T14:34:59.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:59.499+0000 7f49c5400640 1 -- 172.21.15.90:0/2803579826 wait complete. 2024-04-02T14:34:59.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:59.499+0000 7f49c5400640 1 Processor -- start 2024-04-02T14:34:59.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:59.500+0000 7f49c5400640 1 -- start start 2024-04-02T14:34:59.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:59.500+0000 7f49c5400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f49c00764c0 0x7f49c019e450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:59.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:59.500+0000 7f49c5400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f49c0077700 0x7f49c019e990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:59.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:59.500+0000 7f49c5400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f49c00780a0 0x7f49c01a59b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:59.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:59.500+0000 7f49c5400640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f49c0113dc0 con 0x7f49c0077700 2024-04-02T14:34:59.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:59.500+0000 7f49c5400640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f49c0113c40 con 0x7f49c00780a0 2024-04-02T14:34:59.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:59.500+0000 7f49c5400640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f49c0113f40 con 0x7f49c00764c0 2024-04-02T14:34:59.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:59.500+0000 7f49bf200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f49c00780a0 0x7f49c01a59b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:59.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:59.500+0000 7f49be800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f49c00764c0 0x7f49c019e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:59.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:59.500+0000 7f49bf200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f49c00780a0 0x7f49c01a59b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:58986/0 (socket says 172.21.15.90:58986) 2024-04-02T14:34:59.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:59.500+0000 7f49bf200640 1 -- 172.21.15.90:0/312997221 learned_addr learned my addr 172.21.15.90:0/312997221 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:34:59.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:59.500+0000 7f49bde00640 1 --2- 172.21.15.90:0/312997221 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f49c0077700 0x7f49c019e990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:59.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:59.501+0000 7f49bf200640 1 -- 172.21.15.90:0/312997221 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f49c00764c0 msgr2=0x7f49c019e450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:59.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:59.501+0000 7f49bf200640 1 --2- 172.21.15.90:0/312997221 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f49c00764c0 0x7f49c019e450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:59.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:59.501+0000 7f49bf200640 1 -- 172.21.15.90:0/312997221 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f49c0077700 msgr2=0x7f49c019e990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:59.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:59.501+0000 7f49bf200640 1 --2- 172.21.15.90:0/312997221 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f49c0077700 0x7f49c019e990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:59.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:59.501+0000 7f49bf200640 1 -- 172.21.15.90:0/312997221 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49c01a6020 con 0x7f49c00780a0 2024-04-02T14:34:59.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:59.501+0000 7f49bf200640 1 --2- 172.21.15.90:0/312997221 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f49c00780a0 0x7f49c01a59b0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f49b800eb10 tx=0x7f49b800c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:34:59.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:59.501+0000 7f49b7400640 1 -- 172.21.15.90:0/312997221 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49b8004b20 con 0x7f49c00780a0 2024-04-02T14:34:59.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:59.502+0000 7f49c5400640 1 -- 172.21.15.90:0/312997221 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49c01a6310 con 0x7f49c00780a0 2024-04-02T14:34:59.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:59.502+0000 7f49b7400640 1 -- 172.21.15.90:0/312997221 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f49b8004cc0 con 0x7f49c00780a0 2024-04-02T14:34:59.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:59.502+0000 7f49b7400640 1 -- 172.21.15.90:0/312997221 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49b8010760 con 0x7f49c00780a0 2024-04-02T14:34:59.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:59.502+0000 7f49c5400640 1 -- 172.21.15.90:0/312997221 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49c01a6880 con 0x7f49c00780a0 2024-04-02T14:34:59.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:59.503+0000 7f49c5400640 1 -- 172.21.15.90:0/312997221 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f49c01106b0 con 0x7f49c00780a0 2024-04-02T14:34:59.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:59.504+0000 7f49b7400640 1 -- 172.21.15.90:0/312997221 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7f49b8005020 con 0x7f49c00780a0 2024-04-02T14:34:59.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:59.506+0000 7f49b7400640 1 --2- 172.21.15.90:0/312997221 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f49a40a3e40 0x7f49a40a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:59.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:59.506+0000 7f49b7400640 1 -- 172.21.15.90:0/312997221 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f49b80d11a0 con 0x7f49c00780a0 2024-04-02T14:34:59.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:59.506+0000 7f49be800640 1 --2- 172.21.15.90:0/312997221 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f49a40a3e40 0x7f49a40a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:59.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:59.507+0000 7f49be800640 1 --2- 172.21.15.90:0/312997221 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f49a40a3e40 0x7f49a40a6300 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f49b0009bd0 tx=0x7f49b0005f90 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:34:59.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:59.509+0000 7f49b7400640 1 -- 172.21.15.90:0/312997221 <== mon.2 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7f49b809d5a0 con 0x7f49c00780a0 2024-04-02T14:34:59.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:59.715+0000 7f49c5400640 1 -- 172.21.15.90:0/312997221 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f49c0062700 con 0x7f49c00780a0 2024-04-02T14:34:59.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:59.716+0000 7f49b7400640 1 -- 172.21.15.90:0/312997221 <== mon.2 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+23024 (secure 0 0 0) 0x7f49b809cea0 con 0x7f49c00780a0 2024-04-02T14:34:59.718 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-02T14:34:59.718 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":91,"fsid":"6449448a-f0fc-11ee-b647-cb9ed24678a4","created":"2024-04-02T14:23:16.530201+0000","modified":"2024-04-02T14:34:52.235281+0000","last_up_change":"2024-04-02T14:31:23.819732+0000","last_in_change":"2024-04-02T14:31:02.824284+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T14:27:16.758622+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-02T14:32:20.016331+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-02T14:32:21.294367+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a1a6a856-ab44-44bd-bc5e-45c718a70cd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":3635087609},{"type":"v1","addr":"172.21.15.90:6803","nonce":3635087609}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":3635087609},{"type":"v1","addr":"172.21.15.90:6805","nonce":3635087609}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":3635087609},{"type":"v1","addr":"172.21.15.90:6809","nonce":3635087609}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":3635087609},{"type":"v1","addr":"172.21.15.90:6807","nonce":3635087609}]},"public_addr":"172.21.15.90:6803/3635087609","cluster_addr":"172.21.15.90:6805/3635087609","heartbeat_back_addr":"172.21.15.90:6809/3635087609","heartbeat_front_addr":"172.21.15.90:6807/3635087609","state":["exists","up"]},{"osd":1,"uuid":"124139cd-5793-463e-9607-9d9c10164183","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":1994536059},{"type":"v1","addr":"172.21.15.92:6801","nonce":1994536059}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":1994536059},{"type":"v1","addr":"172.21.15.92:6803","nonce":1994536059}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":1994536059},{"type":"v1","addr":"172.21.15.92:6807","nonce":1994536059}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":1994536059},{"type":"v1","addr":"172.21.15.92:6805","nonce":1994536059}]},"public_addr":"172.21.15.92:6801/1994536059","cluster_addr":"172.21.15.92:6803/1994536059","heartbeat_back_addr":"172.21.15.92:6807/1994536059","heartbeat_front_addr":"172.21.15.92:6805/1994536059","state":["exists","up"]},{"osd":2,"uuid":"a61fbef2-9dca-4875-a814-020fff0db0e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":1358472800},{"type":"v1","addr":"172.21.15.177:6801","nonce":1358472800}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":1358472800},{"type":"v1","addr":"172.21.15.177:6803","nonce":1358472800}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":1358472800},{"type":"v1","addr":"172.21.15.177:6807","nonce":1358472800}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":1358472800},{"type":"v1","addr":"172.21.15.177:6805","nonce":1358472800}]},"public_addr":"172.21.15.177:6801/1358472800","cluster_addr":"172.21.15.177:6803/1358472800","heartbeat_back_addr":"172.21.15.177:6807/1358472800","heartbeat_front_addr":"172.21.15.177:6805/1358472800","state":["exists","up"]},{"osd":3,"uuid":"b175a2f4-7859-4eb0-bc1f-fba8c2b92a2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":3114811216},{"type":"v1","addr":"172.21.15.90:6811","nonce":3114811216}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":3114811216},{"type":"v1","addr":"172.21.15.90:6813","nonce":3114811216}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":3114811216},{"type":"v1","addr":"172.21.15.90:6817","nonce":3114811216}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":3114811216},{"type":"v1","addr":"172.21.15.90:6815","nonce":3114811216}]},"public_addr":"172.21.15.90:6811/3114811216","cluster_addr":"172.21.15.90:6813/3114811216","heartbeat_back_addr":"172.21.15.90:6817/3114811216","heartbeat_front_addr":"172.21.15.90:6815/3114811216","state":["exists","up"]},{"osd":4,"uuid":"38cc68ea-c109-4d8d-b77d-1c6f9b56c875","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":2987208879},{"type":"v1","addr":"172.21.15.92:6809","nonce":2987208879}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6810","nonce":2987208879},{"type":"v1","addr":"172.21.15.92:6811","nonce":2987208879}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6814","nonce":2987208879},{"type":"v1","addr":"172.21.15.92:6815","nonce":2987208879}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":2987208879},{"type":"v1","addr":"172.21.15.92:6813","nonce":2987208879}]},"public_addr":"172.21.15.92:6809/2987208879","cluster_addr":"172.21.15.92:6811/2987208879","heartbeat_back_addr":"172.21.15.92:6815/2987208879","heartbeat_front_addr":"172.21.15.92:6813/2987208879","state":["exists","up"]},{"osd":5,"uuid":"2347232e-ac91-466f-8ac9-496f397679ba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":3486820237},{"type":"v1","addr":"172.21.15.177:6809","nonce":3486820237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":3486820237},{"type":"v1","addr":"172.21.15.177:6811","nonce":3486820237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":3486820237},{"type":"v1","addr":"172.21.15.177:6815","nonce":3486820237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":3486820237},{"type":"v1","addr":"172.21.15.177:6813","nonce":3486820237}]},"public_addr":"172.21.15.177:6809/3486820237","cluster_addr":"172.21.15.177:6811/3486820237","heartbeat_back_addr":"172.21.15.177:6815/3486820237","heartbeat_front_addr":"172.21.15.177:6813/3486820237","state":["exists","up"]},{"osd":6,"uuid":"aa72392f-0498-44db-a8d2-eccc89a64871","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":2579716443},{"type":"v1","addr":"172.21.15.90:6819","nonce":2579716443}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6820","nonce":2579716443},{"type":"v1","addr":"172.21.15.90:6821","nonce":2579716443}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":2579716443},{"type":"v1","addr":"172.21.15.90:6825","nonce":2579716443}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2579716443},{"type":"v1","addr":"172.21.15.90:6823","nonce":2579716443}]},"public_addr":"172.21.15.90:6819/2579716443","cluster_addr":"172.21.15.90:6821/2579716443","heartbeat_back_addr":"172.21.15.90:6825/2579716443","heartbeat_front_addr":"172.21.15.90:6823/2579716443","state":["exists","up"]},{"osd":7,"uuid":"931bd105-bc14-4a1e-bfe4-c7ad58fa4b73","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":3371649139},{"type":"v1","addr":"172.21.15.92:6817","nonce":3371649139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":3371649139},{"type":"v1","addr":"172.21.15.92:6819","nonce":3371649139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":3371649139},{"type":"v1","addr":"172.21.15.92:6823","nonce":3371649139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6820","nonce":3371649139},{"type":"v1","addr":"172.21.15.92:6821","nonce":3371649139}]},"public_addr":"172.21.15.92:6817/3371649139","cluster_addr":"172.21.15.92:6819/3371649139","heartbeat_back_addr":"172.21.15.92:6823/3371649139","heartbeat_front_addr":"172.21.15.92:6821/3371649139","state":["exists","up"]},{"osd":8,"uuid":"82624ec2-b382-4886-9f40-7f5f376de3ce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":1459053034},{"type":"v1","addr":"172.21.15.177:6817","nonce":1459053034}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":1459053034},{"type":"v1","addr":"172.21.15.177:6819","nonce":1459053034}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":1459053034},{"type":"v1","addr":"172.21.15.177:6823","nonce":1459053034}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":1459053034},{"type":"v1","addr":"172.21.15.177:6821","nonce":1459053034}]},"public_addr":"172.21.15.177:6817/1459053034","cluster_addr":"172.21.15.177:6819/1459053034","heartbeat_back_addr":"172.21.15.177:6823/1459053034","heartbeat_front_addr":"172.21.15.177:6821/1459053034","state":["exists","up"]},{"osd":9,"uuid":"fbe36e48-3468-42b2-8094-e16552f3cb16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":2592604332},{"type":"v1","addr":"172.21.15.90:6827","nonce":2592604332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":2592604332},{"type":"v1","addr":"172.21.15.90:6829","nonce":2592604332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6832","nonce":2592604332},{"type":"v1","addr":"172.21.15.90:6833","nonce":2592604332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":2592604332},{"type":"v1","addr":"172.21.15.90:6831","nonce":2592604332}]},"public_addr":"172.21.15.90:6827/2592604332","cluster_addr":"172.21.15.90:6829/2592604332","heartbeat_back_addr":"172.21.15.90:6833/2592604332","heartbeat_front_addr":"172.21.15.90:6831/2592604332","state":["exists","up"]},{"osd":10,"uuid":"9240df75-e7e2-47f1-8aae-534b70b3c210","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":1594153142},{"type":"v1","addr":"172.21.15.92:6825","nonce":1594153142}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6826","nonce":1594153142},{"type":"v1","addr":"172.21.15.92:6827","nonce":1594153142}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":1594153142},{"type":"v1","addr":"172.21.15.92:6831","nonce":1594153142}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":1594153142},{"type":"v1","addr":"172.21.15.92:6829","nonce":1594153142}]},"public_addr":"172.21.15.92:6825/1594153142","cluster_addr":"172.21.15.92:6827/1594153142","heartbeat_back_addr":"172.21.15.92:6831/1594153142","heartbeat_front_addr":"172.21.15.92:6829/1594153142","state":["exists","up"]},{"osd":11,"uuid":"83cba9fa-b745-4df1-adda-e86837d4b91a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":83,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1622582126},{"type":"v1","addr":"172.21.15.177:6825","nonce":1622582126}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":1622582126},{"type":"v1","addr":"172.21.15.177:6827","nonce":1622582126}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":1622582126},{"type":"v1","addr":"172.21.15.177:6831","nonce":1622582126}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1622582126},{"type":"v1","addr":"172.21.15.177:6829","nonce":1622582126}]},"public_addr":"172.21.15.177:6825/1622582126","cluster_addr":"172.21.15.177:6827/1622582126","heartbeat_back_addr":"172.21.15.177:6831/1622582126","heartbeat_front_addr":"172.21.15.177:6829/1622582126","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:26:24.290365+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:26:49.405988+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:27:13.639196+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:27:45.603746+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:28:11.019126+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:28:35.084163+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:29:08.064626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:29:33.266133+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:29:57.381816+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:30:30.286847+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:30:56.226279+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:31:20.809611+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":7,"to":11}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":8,"to":4}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.11","mappings":[{"from":4,"to":8},{"from":5,"to":9}]},{"pgid":"3.21","mappings":[{"from":5,"to":9}]},{"pgid":"3.23","mappings":[{"from":4,"to":7}]},{"pgid":"3.3a","mappings":[{"from":4,"to":6}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.90:0/2805994468":"2024-04-03T14:34:52.235215+0000","172.21.15.90:0/1431307002":"2024-04-03T14:34:52.235215+0000","172.21.15.90:6801/4266054817":"2024-04-03T14:34:52.235215+0000","172.21.15.90:0/314641601":"2024-04-03T14:24:34.595561+0000","172.21.15.90:0/1400895868":"2024-04-03T14:24:34.595561+0000","172.21.15.90:6801/129436436":"2024-04-03T14:24:34.595561+0000","172.21.15.90:0/1335673697":"2024-04-03T14:34:52.235215+0000","172.21.15.90:6801/2372398598":"2024-04-03T14:24:00.362147+0000","172.21.15.90:0/278569338":"2024-04-03T14:24:34.595561+0000","172.21.15.90:0/2321464594":"2024-04-03T14:24:00.362147+0000","172.21.15.90:6800/4266054817":"2024-04-03T14:34:52.235215+0000","172.21.15.90:6800/2372398598":"2024-04-03T14:24:00.362147+0000","172.21.15.90:0/2962656672":"2024-04-03T14:24:00.362147+0000","172.21.15.90:0/126693610":"2024-04-03T14:24:00.362147+0000","172.21.15.90:0/1757385857":"2024-04-03T14:34:52.235215+0000","172.21.15.90:6800/129436436":"2024-04-03T14:24:34.595561+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T14:34:59.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:59.719+0000 7f49c5400640 1 -- 172.21.15.90:0/312997221 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f49a40a3e40 msgr2=0x7f49a40a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:59.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:59.719+0000 7f49c5400640 1 --2- 172.21.15.90:0/312997221 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f49a40a3e40 0x7f49a40a6300 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f49b0009bd0 tx=0x7f49b0005f90 comp rx=0 tx=0).stop 2024-04-02T14:34:59.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:59.719+0000 7f49c5400640 1 -- 172.21.15.90:0/312997221 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f49c00780a0 msgr2=0x7f49c01a59b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:59.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:59.719+0000 7f49c5400640 1 --2- 172.21.15.90:0/312997221 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f49c00780a0 0x7f49c01a59b0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f49b800eb10 tx=0x7f49b800c2d0 comp rx=0 tx=0).stop 2024-04-02T14:34:59.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:59.719+0000 7f49c5400640 1 -- 172.21.15.90:0/312997221 shutdown_connections 2024-04-02T14:34:59.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:59.719+0000 7f49c5400640 1 --2- 172.21.15.90:0/312997221 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f49a40a3e40 0x7f49a40a6300 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:59.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:59.719+0000 7f49c5400640 1 --2- 172.21.15.90:0/312997221 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f49c00780a0 0x7f49c01a59b0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:59.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:59.719+0000 7f49c5400640 1 --2- 172.21.15.90:0/312997221 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f49c0077700 0x7f49c019e990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:59.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:59.719+0000 7f49c5400640 1 --2- 172.21.15.90:0/312997221 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f49c00764c0 0x7f49c019e450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:59.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:59.719+0000 7f49c5400640 1 -- 172.21.15.90:0/312997221 >> 172.21.15.90:0/312997221 conn(0x7f49c00ff2b0 msgr2=0x7f49c0100db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:34:59.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:59.719+0000 7f49c5400640 1 -- 172.21.15.90:0/312997221 shutdown_connections 2024-04-02T14:34:59.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:59.719+0000 7f49c5400640 1 -- 172.21.15.90:0/312997221 wait complete. 2024-04-02T14:34:59.738 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-02T14:27:16.758622+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-02T14:32:20.016331+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-02T14:32:21.294367+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '88', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-02T14:34:59.739 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-02T14:34:59.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:59.929+0000 7f9db3200640 1 -- 172.21.15.90:0/705063071 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9dac1078a0 msgr2=0x7f9dac107c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:59.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:59.929+0000 7f9db3200640 1 --2- 172.21.15.90:0/705063071 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9dac1078a0 0x7f9dac107c80 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f9da0009930 tx=0x7f9da002bc20 comp rx=0 tx=0).stop 2024-04-02T14:34:59.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:59.929+0000 7f9db3200640 1 -- 172.21.15.90:0/705063071 shutdown_connections 2024-04-02T14:34:59.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:59.929+0000 7f9db3200640 1 --2- 172.21.15.90:0/705063071 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9dac102250 0x7f9dac10e760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:59.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:59.929+0000 7f9db3200640 1 --2- 172.21.15.90:0/705063071 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9dac101890 0x7f9dac101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:59.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:59.929+0000 7f9db3200640 1 --2- 172.21.15.90:0/705063071 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9dac1078a0 0x7f9dac107c80 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:59.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:59.929+0000 7f9db3200640 1 -- 172.21.15.90:0/705063071 >> 172.21.15.90:0/705063071 conn(0x7f9dac0fd3a0 msgr2=0x7f9dac0ff7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:34:59.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:59.930+0000 7f9db3200640 1 -- 172.21.15.90:0/705063071 shutdown_connections 2024-04-02T14:34:59.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:59.930+0000 7f9db3200640 1 -- 172.21.15.90:0/705063071 wait complete. 2024-04-02T14:34:59.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:59.931+0000 7f9db3200640 1 Processor -- start 2024-04-02T14:34:59.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:59.931+0000 7f9db3200640 1 -- start start 2024-04-02T14:34:59.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:59.931+0000 7f9db3200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9dac101890 0x7f9dac19a350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:59.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:59.931+0000 7f9db3200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9dac102250 0x7f9dac19a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:59.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:59.931+0000 7f9db3200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9dac1078a0 0x7f9dac19ec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:59.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:59.931+0000 7f9db3200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f9dac111cc0 con 0x7f9dac102250 2024-04-02T14:34:59.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:59.931+0000 7f9db3200640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f9dac111b40 con 0x7f9dac101890 2024-04-02T14:34:59.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:59.931+0000 7f9db3200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f9dac111e40 con 0x7f9dac1078a0 2024-04-02T14:34:59.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:59.932+0000 7f9db1e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9dac101890 0x7f9dac19a350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:59.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:59.932+0000 7f9db1e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9dac101890 0x7f9dac19a350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:59002/0 (socket says 172.21.15.90:59002) 2024-04-02T14:34:59.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:59.932+0000 7f9db1e00640 1 -- 172.21.15.90:0/2523656788 learned_addr learned my addr 172.21.15.90:0/2523656788 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:34:59.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:59.932+0000 7f9db2800640 1 --2- 172.21.15.90:0/2523656788 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9dac1078a0 0x7f9dac19ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:59.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:59.932+0000 7f9db1400640 1 --2- 172.21.15.90:0/2523656788 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9dac102250 0x7f9dac19a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:59.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:59.932+0000 7f9db1e00640 1 -- 172.21.15.90:0/2523656788 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9dac1078a0 msgr2=0x7f9dac19ec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:59.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:59.932+0000 7f9db1e00640 1 --2- 172.21.15.90:0/2523656788 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9dac1078a0 0x7f9dac19ec10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:59.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:59.932+0000 7f9db1e00640 1 -- 172.21.15.90:0/2523656788 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9dac102250 msgr2=0x7f9dac19a890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:34:59.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:59.932+0000 7f9db1e00640 1 --2- 172.21.15.90:0/2523656788 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9dac102250 0x7f9dac19a890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:34:59.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:59.932+0000 7f9db1e00640 1 -- 172.21.15.90:0/2523656788 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9dac19f340 con 0x7f9dac101890 2024-04-02T14:34:59.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:59.932+0000 7f9db1400640 1 --2- 172.21.15.90:0/2523656788 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9dac102250 0x7f9dac19a890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:34:59.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:59.932+0000 7f9db1e00640 1 --2- 172.21.15.90:0/2523656788 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9dac101890 0x7f9dac19a350 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f9da0005ab0 tx=0x7f9da0005e80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:34:59.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:59.932+0000 7f9da6a00640 1 -- 172.21.15.90:0/2523656788 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9da0043020 con 0x7f9dac101890 2024-04-02T14:34:59.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:59.933+0000 7f9da6a00640 1 -- 172.21.15.90:0/2523656788 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9da0005000 con 0x7f9dac101890 2024-04-02T14:34:59.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:59.933+0000 7f9da6a00640 1 -- 172.21.15.90:0/2523656788 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9da0003ab0 con 0x7f9dac101890 2024-04-02T14:34:59.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:59.933+0000 7f9db3200640 1 -- 172.21.15.90:0/2523656788 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9dac19f5d0 con 0x7f9dac101890 2024-04-02T14:34:59.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:59.933+0000 7f9db3200640 1 -- 172.21.15.90:0/2523656788 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9dac1a6ec0 con 0x7f9dac101890 2024-04-02T14:34:59.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:59.934+0000 7f9db3200640 1 -- 172.21.15.90:0/2523656788 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9dac068810 con 0x7f9dac101890 2024-04-02T14:34:59.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:59.936+0000 7f9da6a00640 1 -- 172.21.15.90:0/2523656788 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7f9da0046020 con 0x7f9dac101890 2024-04-02T14:34:59.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:59.937+0000 7f9da6a00640 1 --2- 172.21.15.90:0/2523656788 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f9d940a3ec0 0x7f9d940a6380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:34:59.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:59.937+0000 7f9da6a00640 1 -- 172.21.15.90:0/2523656788 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f9da00f32d0 con 0x7f9dac101890 2024-04-02T14:34:59.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:59.937+0000 7f9db1400640 1 --2- 172.21.15.90:0/2523656788 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f9d940a3ec0 0x7f9d940a6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:34:59.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:59.938+0000 7f9db1400640 1 --2- 172.21.15.90:0/2523656788 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f9d940a3ec0 0x7f9d940a6380 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f9dac10db30 tx=0x7f9d9c005f90 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:34:59.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:34:59.939+0000 7f9da6a00640 1 -- 172.21.15.90:0/2523656788 <== mon.2 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7f9da004d020 con 0x7f9dac101890 2024-04-02T14:35:00.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:00.151+0000 7f9db3200640 1 -- 172.21.15.90:0/2523656788 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f9dac1a7090 con 0x7f9dac101890 2024-04-02T14:35:00.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:00.152+0000 7f9da6a00640 1 -- 172.21.15.90:0/2523656788 <== mon.2 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v91) ==== 93+0+10 (secure 0 0 0) 0x7f9da00bf7e0 con 0x7f9dac101890 2024-04-02T14:35:00.153 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 1 2024-04-02T14:35:00.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:00.154+0000 7f9db3200640 1 -- 172.21.15.90:0/2523656788 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f9d940a3ec0 msgr2=0x7f9d940a6380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:35:00.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:00.154+0000 7f9db3200640 1 --2- 172.21.15.90:0/2523656788 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f9d940a3ec0 0x7f9d940a6380 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f9dac10db30 tx=0x7f9d9c005f90 comp rx=0 tx=0).stop 2024-04-02T14:35:00.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:00.154+0000 7f9db3200640 1 -- 172.21.15.90:0/2523656788 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9dac101890 msgr2=0x7f9dac19a350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:35:00.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:00.154+0000 7f9db3200640 1 --2- 172.21.15.90:0/2523656788 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9dac101890 0x7f9dac19a350 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f9da0005ab0 tx=0x7f9da0005e80 comp rx=0 tx=0).stop 2024-04-02T14:35:00.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:00.154+0000 7f9db3200640 1 -- 172.21.15.90:0/2523656788 shutdown_connections 2024-04-02T14:35:00.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:00.154+0000 7f9db3200640 1 --2- 172.21.15.90:0/2523656788 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f9d940a3ec0 0x7f9d940a6380 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:00.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:00.154+0000 7f9db3200640 1 --2- 172.21.15.90:0/2523656788 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9dac1078a0 0x7f9dac19ec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:00.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:00.154+0000 7f9db3200640 1 --2- 172.21.15.90:0/2523656788 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9dac102250 0x7f9dac19a890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:00.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:00.154+0000 7f9db3200640 1 --2- 172.21.15.90:0/2523656788 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9dac101890 0x7f9dac19a350 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:00.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:00.154+0000 7f9db3200640 1 -- 172.21.15.90:0/2523656788 >> 172.21.15.90:0/2523656788 conn(0x7f9dac0fd3a0 msgr2=0x7f9dac106b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:35:00.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:00.155+0000 7f9db3200640 1 -- 172.21.15.90:0/2523656788 shutdown_connections 2024-04-02T14:35:00.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:00.155+0000 7f9db3200640 1 -- 172.21.15.90:0/2523656788 wait complete. 2024-04-02T14:35:00.173 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-02T14:35:00.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:35:00 smithi092 ceph-mon[36497]: from='client.15312 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-02T14:35:00.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:35:00 smithi092 ceph-mon[36497]: pgmap v6: 97 pgs: 97 active+clean; 1.1 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-02T14:35:00.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:35:00 smithi092 ceph-mon[36497]: Updating smithi177:/var/lib/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-04-02T14:35:00.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:35:00 smithi092 ceph-mon[36497]: Updating smithi092:/var/lib/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-04-02T14:35:00.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:35:00 smithi092 ceph-mon[36497]: Updating smithi177:/etc/ceph/ceph.client.admin.keyring 2024-04-02T14:35:00.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:35:00 smithi092 ceph-mon[36497]: Updating smithi090:/var/lib/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-04-02T14:35:00.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:35:00 smithi092 ceph-mon[36497]: Updating smithi092:/etc/ceph/ceph.client.admin.keyring 2024-04-02T14:35:00.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:35:00 smithi092 ceph-mon[36497]: from='client.25433 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-02T14:35:00.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:35:00 smithi092 ceph-mon[36497]: Updating smithi177:/var/lib/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/config/ceph.client.admin.keyring 2024-04-02T14:35:00.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:35:00 smithi092 ceph-mon[36497]: Updating smithi092:/var/lib/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/config/ceph.client.admin.keyring 2024-04-02T14:35:00.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:35:00 smithi092 ceph-mon[36497]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' 2024-04-02T14:35:00.362 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:35:00 smithi092 ceph-mon[36497]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' 2024-04-02T14:35:00.362 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:35:00 smithi092 ceph-mon[36497]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' 2024-04-02T14:35:00.362 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:35:00 smithi092 ceph-mon[36497]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' 2024-04-02T14:35:00.362 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:35:00 smithi092 ceph-mon[36497]: from='client.? 172.21.15.90:0/312997221' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-02T14:35:00.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:00.361+0000 7fabe5200640 1 -- 172.21.15.90:0/1310503953 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fabe0073ed0 msgr2=0x7fabe00742b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:35:00.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:00.361+0000 7fabe5200640 1 --2- 172.21.15.90:0/1310503953 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fabe0073ed0 0x7fabe00742b0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fabd0009be0 tx=0x7fabd002bf80 comp rx=0 tx=0).stop 2024-04-02T14:35:00.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:00.361+0000 7fabe5200640 1 -- 172.21.15.90:0/1310503953 shutdown_connections 2024-04-02T14:35:00.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:00.361+0000 7fabe5200640 1 --2- 172.21.15.90:0/1310503953 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fabe010e210 0x7fabe0110620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:00.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:00.361+0000 7fabe5200640 1 --2- 172.21.15.90:0/1310503953 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fabe00747f0 0x7fabe010dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:00.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:00.361+0000 7fabe5200640 1 --2- 172.21.15.90:0/1310503953 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fabe0073ed0 0x7fabe00742b0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:00.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:00.361+0000 7fabe5200640 1 -- 172.21.15.90:0/1310503953 >> 172.21.15.90:0/1310503953 conn(0x7fabe00fd4a0 msgr2=0x7fabe00ff8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:35:00.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:00.361+0000 7fabe5200640 1 -- 172.21.15.90:0/1310503953 shutdown_connections 2024-04-02T14:35:00.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:00.361+0000 7fabe5200640 1 -- 172.21.15.90:0/1310503953 wait complete. 2024-04-02T14:35:00.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:00.362+0000 7fabe5200640 1 Processor -- start 2024-04-02T14:35:00.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:00.363+0000 7fabe5200640 1 -- start start 2024-04-02T14:35:00.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:00.363+0000 7fabe5200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fabe0073ed0 0x7fabe019a4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:00.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:00.363+0000 7fabe5200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fabe00747f0 0x7fabe019a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:00.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:00.363+0000 7fabe5200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fabe010e210 0x7fabe019ed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:00.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:00.363+0000 7fabe5200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fabe0113fc0 con 0x7fabe00747f0 2024-04-02T14:35:00.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:00.363+0000 7fabe5200640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fabe0113e40 con 0x7fabe0073ed0 2024-04-02T14:35:00.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:00.363+0000 7fabe5200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fabe0114140 con 0x7fabe010e210 2024-04-02T14:35:00.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:00.363+0000 7fabdf400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fabe0073ed0 0x7fabe019a4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:00.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:00.363+0000 7fabd6a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fabe00747f0 0x7fabe019a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:00.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:00.363+0000 7fabdf400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fabe0073ed0 0x7fabe019a4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:59020/0 (socket says 172.21.15.90:59020) 2024-04-02T14:35:00.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:00.363+0000 7fabdf400640 1 -- 172.21.15.90:0/4008070079 learned_addr learned my addr 172.21.15.90:0/4008070079 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:35:00.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:00.364+0000 7fabdfe00640 1 --2- 172.21.15.90:0/4008070079 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fabe010e210 0x7fabe019ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:00.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:00.364+0000 7fabdf400640 1 -- 172.21.15.90:0/4008070079 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fabe010e210 msgr2=0x7fabe019ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:35:00.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:00.364+0000 7fabdf400640 1 --2- 172.21.15.90:0/4008070079 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fabe010e210 0x7fabe019ed60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:00.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:00.364+0000 7fabdf400640 1 -- 172.21.15.90:0/4008070079 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fabe00747f0 msgr2=0x7fabe019a9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:35:00.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:00.364+0000 7fabdf400640 1 --2- 172.21.15.90:0/4008070079 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fabe00747f0 0x7fabe019a9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:00.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:00.364+0000 7fabdf400640 1 -- 172.21.15.90:0/4008070079 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fabe019f490 con 0x7fabe0073ed0 2024-04-02T14:35:00.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:00.364+0000 7fabdf400640 1 --2- 172.21.15.90:0/4008070079 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fabe0073ed0 0x7fabe019a4a0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fabd002c590 tx=0x7fabd0004270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:35:00.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:00.364+0000 7fabdcc00640 1 -- 172.21.15.90:0/4008070079 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabd0043020 con 0x7fabe0073ed0 2024-04-02T14:35:00.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:00.364+0000 7fabe5200640 1 -- 172.21.15.90:0/4008070079 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fabe019f720 con 0x7fabe0073ed0 2024-04-02T14:35:00.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:00.364+0000 7fabdcc00640 1 -- 172.21.15.90:0/4008070079 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fabd0004670 con 0x7fabe0073ed0 2024-04-02T14:35:00.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:00.364+0000 7fabdcc00640 1 -- 172.21.15.90:0/4008070079 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabd003e6b0 con 0x7fabe0073ed0 2024-04-02T14:35:00.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:00.365+0000 7fabe5200640 1 -- 172.21.15.90:0/4008070079 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fabe01a7010 con 0x7fabe0073ed0 2024-04-02T14:35:00.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:00.366+0000 7fabe5200640 1 -- 172.21.15.90:0/4008070079 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fabe0068810 con 0x7fabe0073ed0 2024-04-02T14:35:00.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:00.368+0000 7fabdcc00640 1 -- 172.21.15.90:0/4008070079 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7fabd0046020 con 0x7fabe0073ed0 2024-04-02T14:35:00.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:00.369+0000 7fabdcc00640 1 --2- 172.21.15.90:0/4008070079 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fabc40a3e40 0x7fabc40a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:00.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:00.369+0000 7fabdcc00640 1 -- 172.21.15.90:0/4008070079 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7fabd00f3170 con 0x7fabe0073ed0 2024-04-02T14:35:00.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:00.369+0000 7fabd6a00640 1 --2- 172.21.15.90:0/4008070079 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fabc40a3e40 0x7fabc40a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:00.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:00.370+0000 7fabd6a00640 1 --2- 172.21.15.90:0/4008070079 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fabc40a3e40 0x7fabc40a6300 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fabe010f4e0 tx=0x7fabc8009260 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:35:00.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:00.373+0000 7fabdcc00640 1 -- 172.21.15.90:0/4008070079 <== mon.2 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7fabd004d020 con 0x7fabe0073ed0 2024-04-02T14:35:00.437 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:35:00 smithi177 ceph-mon[36431]: from='client.15312 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-02T14:35:00.437 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:35:00 smithi177 ceph-mon[36431]: pgmap v6: 97 pgs: 97 active+clean; 1.1 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-02T14:35:00.437 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:35:00 smithi177 ceph-mon[36431]: Updating smithi177:/var/lib/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-04-02T14:35:00.437 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:35:00 smithi177 ceph-mon[36431]: Updating smithi092:/var/lib/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-04-02T14:35:00.437 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:35:00 smithi177 ceph-mon[36431]: Updating smithi177:/etc/ceph/ceph.client.admin.keyring 2024-04-02T14:35:00.437 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:35:00 smithi177 ceph-mon[36431]: Updating smithi090:/var/lib/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-04-02T14:35:00.437 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:35:00 smithi177 ceph-mon[36431]: Updating smithi092:/etc/ceph/ceph.client.admin.keyring 2024-04-02T14:35:00.437 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:35:00 smithi177 ceph-mon[36431]: from='client.25433 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-02T14:35:00.437 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:35:00 smithi177 ceph-mon[36431]: Updating smithi177:/var/lib/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/config/ceph.client.admin.keyring 2024-04-02T14:35:00.437 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:35:00 smithi177 ceph-mon[36431]: Updating smithi092:/var/lib/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/config/ceph.client.admin.keyring 2024-04-02T14:35:00.438 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:35:00 smithi177 ceph-mon[36431]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' 2024-04-02T14:35:00.438 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:35:00 smithi177 ceph-mon[36431]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' 2024-04-02T14:35:00.438 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:35:00 smithi177 ceph-mon[36431]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' 2024-04-02T14:35:00.438 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:35:00 smithi177 ceph-mon[36431]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' 2024-04-02T14:35:00.438 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:35:00 smithi177 ceph-mon[36431]: from='client.? 172.21.15.90:0/312997221' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-02T14:35:00.439 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:35:00 smithi090 ceph-mon[31674]: from='client.15312 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-02T14:35:00.439 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:35:00 smithi090 ceph-mon[31674]: pgmap v6: 97 pgs: 97 active+clean; 1.1 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-02T14:35:00.439 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:35:00 smithi090 ceph-mon[31674]: Updating smithi177:/var/lib/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-04-02T14:35:00.439 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:35:00 smithi090 ceph-mon[31674]: Updating smithi092:/var/lib/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-04-02T14:35:00.439 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:35:00 smithi090 ceph-mon[31674]: Updating smithi177:/etc/ceph/ceph.client.admin.keyring 2024-04-02T14:35:00.439 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:35:00 smithi090 ceph-mon[31674]: Updating smithi090:/var/lib/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-04-02T14:35:00.439 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:35:00 smithi090 ceph-mon[31674]: Updating smithi092:/etc/ceph/ceph.client.admin.keyring 2024-04-02T14:35:00.439 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:35:00 smithi090 ceph-mon[31674]: from='client.25433 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-02T14:35:00.439 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:35:00 smithi090 ceph-mon[31674]: Updating smithi177:/var/lib/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/config/ceph.client.admin.keyring 2024-04-02T14:35:00.439 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:35:00 smithi090 ceph-mon[31674]: Updating smithi092:/var/lib/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/config/ceph.client.admin.keyring 2024-04-02T14:35:00.440 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:35:00 smithi090 ceph-mon[31674]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' 2024-04-02T14:35:00.440 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:35:00 smithi090 ceph-mon[31674]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' 2024-04-02T14:35:00.440 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:35:00 smithi090 ceph-mon[31674]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' 2024-04-02T14:35:00.440 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:35:00 smithi090 ceph-mon[31674]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' 2024-04-02T14:35:00.440 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:35:00 smithi090 ceph-mon[31674]: from='client.? 172.21.15.90:0/312997221' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-02T14:35:00.582 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:00.581+0000 7fabe5200640 1 -- 172.21.15.90:0/4008070079 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fabe0109390 con 0x7fabe0073ed0 2024-04-02T14:35:00.583 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:00.581+0000 7fabdcc00640 1 -- 172.21.15.90:0/4008070079 <== mon.2 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v91) ==== 104+0+11 (secure 0 0 0) 0x7fabd0004820 con 0x7fabe0073ed0 2024-04-02T14:35:00.583 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-04-02T14:35:00.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:00.584+0000 7fabe5200640 1 -- 172.21.15.90:0/4008070079 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fabc40a3e40 msgr2=0x7fabc40a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:35:00.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:00.584+0000 7fabe5200640 1 --2- 172.21.15.90:0/4008070079 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fabc40a3e40 0x7fabc40a6300 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fabe010f4e0 tx=0x7fabc8009260 comp rx=0 tx=0).stop 2024-04-02T14:35:00.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:00.584+0000 7fabe5200640 1 -- 172.21.15.90:0/4008070079 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fabe0073ed0 msgr2=0x7fabe019a4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:35:00.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:00.584+0000 7fabe5200640 1 --2- 172.21.15.90:0/4008070079 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fabe0073ed0 0x7fabe019a4a0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fabd002c590 tx=0x7fabd0004270 comp rx=0 tx=0).stop 2024-04-02T14:35:00.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:00.584+0000 7fabe5200640 1 -- 172.21.15.90:0/4008070079 shutdown_connections 2024-04-02T14:35:00.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:00.584+0000 7fabe5200640 1 --2- 172.21.15.90:0/4008070079 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fabc40a3e40 0x7fabc40a6300 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:00.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:00.584+0000 7fabe5200640 1 --2- 172.21.15.90:0/4008070079 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fabe010e210 0x7fabe019ed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:00.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:00.584+0000 7fabe5200640 1 --2- 172.21.15.90:0/4008070079 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fabe00747f0 0x7fabe019a9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:00.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:00.584+0000 7fabe5200640 1 --2- 172.21.15.90:0/4008070079 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fabe0073ed0 0x7fabe019a4a0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:00.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:00.584+0000 7fabe5200640 1 -- 172.21.15.90:0/4008070079 >> 172.21.15.90:0/4008070079 conn(0x7fabe00fd4a0 msgr2=0x7fabe00fe170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:35:00.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:00.584+0000 7fabe5200640 1 -- 172.21.15.90:0/4008070079 shutdown_connections 2024-04-02T14:35:00.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:00.584+0000 7fabe5200640 1 -- 172.21.15.90:0/4008070079 wait complete. 2024-04-02T14:35:00.604 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-02T14:35:00.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:00.806+0000 7f6805e00640 1 -- 172.21.15.90:0/2714742292 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f680010e2d0 msgr2=0x7f6800110720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:35:00.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:00.806+0000 7f6805e00640 1 --2- 172.21.15.90:0/2714742292 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f680010e2d0 0x7f6800110720 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f67f8009900 tx=0x7f67f802bcd0 comp rx=0 tx=0).stop 2024-04-02T14:35:00.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:00.806+0000 7f6805e00640 1 -- 172.21.15.90:0/2714742292 shutdown_connections 2024-04-02T14:35:00.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:00.806+0000 7f6805e00640 1 --2- 172.21.15.90:0/2714742292 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f680010e2d0 0x7f6800110720 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:00.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:00.806+0000 7f6805e00640 1 --2- 172.21.15.90:0/2714742292 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6800074850 0x7f680010dd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:00.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:00.806+0000 7f6805e00640 1 --2- 172.21.15.90:0/2714742292 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6800073f30 0x7f6800074310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:00.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:00.806+0000 7f6805e00640 1 -- 172.21.15.90:0/2714742292 >> 172.21.15.90:0/2714742292 conn(0x7f6800078f00 msgr2=0x7f6800079310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:35:00.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:00.806+0000 7f6805e00640 1 -- 172.21.15.90:0/2714742292 shutdown_connections 2024-04-02T14:35:00.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:00.806+0000 7f6805e00640 1 -- 172.21.15.90:0/2714742292 wait complete. 2024-04-02T14:35:00.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:00.807+0000 7f6805e00640 1 Processor -- start 2024-04-02T14:35:00.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:00.807+0000 7f6805e00640 1 -- start start 2024-04-02T14:35:00.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:00.808+0000 7f6805e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6800073f30 0x7f680019a600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:00.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:00.808+0000 7f6805e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6800074850 0x7f680019ab40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:00.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:00.808+0000 7f6805e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f680010e2d0 0x7f680019eec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:00.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:00.808+0000 7f6805e00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f68001140a0 con 0x7f680010e2d0 2024-04-02T14:35:00.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:00.808+0000 7f6805e00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f6800113f20 con 0x7f6800074850 2024-04-02T14:35:00.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:00.808+0000 7f6805e00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f6800114220 con 0x7f6800073f30 2024-04-02T14:35:00.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:00.808+0000 7f67f6000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6800074850 0x7f680019ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:00.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:00.808+0000 7f67ff400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f680010e2d0 0x7f680019eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:00.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:00.808+0000 7f67f6000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6800074850 0x7f680019ab40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:59030/0 (socket says 172.21.15.90:59030) 2024-04-02T14:35:00.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:00.808+0000 7f67f6000640 1 -- 172.21.15.90:0/3531898634 learned_addr learned my addr 172.21.15.90:0/3531898634 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:35:00.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:00.808+0000 7f67fea00640 1 --2- 172.21.15.90:0/3531898634 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6800073f30 0x7f680019a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:00.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:00.808+0000 7f67f6000640 1 -- 172.21.15.90:0/3531898634 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6800073f30 msgr2=0x7f680019a600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:35:00.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:00.808+0000 7f67f6000640 1 --2- 172.21.15.90:0/3531898634 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6800073f30 0x7f680019a600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:00.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:00.808+0000 7f67f6000640 1 -- 172.21.15.90:0/3531898634 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f680010e2d0 msgr2=0x7f680019eec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:35:00.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:00.808+0000 7f67f6000640 1 --2- 172.21.15.90:0/3531898634 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f680010e2d0 0x7f680019eec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:00.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:00.808+0000 7f67f6000640 1 -- 172.21.15.90:0/3531898634 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f680019f5f0 con 0x7f6800074850 2024-04-02T14:35:00.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:00.809+0000 7f67f6000640 1 --2- 172.21.15.90:0/3531898634 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6800074850 0x7f680019ab40 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f67e800ea60 tx=0x7f67e800ef20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:35:00.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:00.809+0000 7f6804a00640 1 -- 172.21.15.90:0/3531898634 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67e800cde0 con 0x7f6800074850 2024-04-02T14:35:00.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:00.809+0000 7f6804a00640 1 -- 172.21.15.90:0/3531898634 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f67e80035f0 con 0x7f6800074850 2024-04-02T14:35:00.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:00.809+0000 7f6805e00640 1 -- 172.21.15.90:0/3531898634 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f680019f8e0 con 0x7f6800074850 2024-04-02T14:35:00.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:00.809+0000 7f6804a00640 1 -- 172.21.15.90:0/3531898634 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67e8010670 con 0x7f6800074850 2024-04-02T14:35:00.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:00.809+0000 7f6805e00640 1 -- 172.21.15.90:0/3531898634 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68001a7170 con 0x7f6800074850 2024-04-02T14:35:00.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:00.810+0000 7f6805e00640 1 -- 172.21.15.90:0/3531898634 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6800068d40 con 0x7f6800074850 2024-04-02T14:35:00.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:00.812+0000 7f6804a00640 1 -- 172.21.15.90:0/3531898634 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7f67e80037a0 con 0x7f6800074850 2024-04-02T14:35:00.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:00.813+0000 7f6804a00640 1 --2- 172.21.15.90:0/3531898634 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f67e40a3e40 0x7f67e40a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:00.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:00.813+0000 7f6804a00640 1 -- 172.21.15.90:0/3531898634 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f67e80d0d30 con 0x7f6800074850 2024-04-02T14:35:00.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:00.813+0000 7f67fea00640 1 --2- 172.21.15.90:0/3531898634 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f67e40a3e40 0x7f67e40a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:00.815 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:00.814+0000 7f67fea00640 1 --2- 172.21.15.90:0/3531898634 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f67e40a3e40 0x7f67e40a6300 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f67f0005fd0 tx=0x7f67f0009500 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:35:00.818 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:00.817+0000 7f6804a00640 1 -- 172.21.15.90:0/3531898634 <== mon.2 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7f67e809d190 con 0x7f6800074850 2024-04-02T14:35:01.025 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:01.024+0000 7f6805e00640 1 -- 172.21.15.90:0/3531898634 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f68001094b0 con 0x7f6800074850 2024-04-02T14:35:01.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:01.025+0000 7f6804a00640 1 -- 172.21.15.90:0/3531898634 <== mon.2 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v91) ==== 100+0+11 (secure 0 0 0) 0x7f67e809ca90 con 0x7f6800074850 2024-04-02T14:35:01.026 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 64 2024-04-02T14:35:01.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:01.027+0000 7f6805e00640 1 -- 172.21.15.90:0/3531898634 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f67e40a3e40 msgr2=0x7f67e40a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:35:01.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:01.027+0000 7f6805e00640 1 --2- 172.21.15.90:0/3531898634 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f67e40a3e40 0x7f67e40a6300 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f67f0005fd0 tx=0x7f67f0009500 comp rx=0 tx=0).stop 2024-04-02T14:35:01.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:01.027+0000 7f6805e00640 1 -- 172.21.15.90:0/3531898634 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6800074850 msgr2=0x7f680019ab40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:35:01.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:01.027+0000 7f6805e00640 1 --2- 172.21.15.90:0/3531898634 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6800074850 0x7f680019ab40 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f67e800ea60 tx=0x7f67e800ef20 comp rx=0 tx=0).stop 2024-04-02T14:35:01.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:01.028+0000 7f6805e00640 1 -- 172.21.15.90:0/3531898634 shutdown_connections 2024-04-02T14:35:01.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:01.028+0000 7f6805e00640 1 --2- 172.21.15.90:0/3531898634 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f67e40a3e40 0x7f67e40a6300 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:01.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:01.028+0000 7f6805e00640 1 --2- 172.21.15.90:0/3531898634 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f680010e2d0 0x7f680019eec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:01.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:01.028+0000 7f6805e00640 1 --2- 172.21.15.90:0/3531898634 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6800074850 0x7f680019ab40 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:01.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:01.028+0000 7f6805e00640 1 --2- 172.21.15.90:0/3531898634 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6800073f30 0x7f680019a600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:01.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:01.028+0000 7f6805e00640 1 -- 172.21.15.90:0/3531898634 >> 172.21.15.90:0/3531898634 conn(0x7f6800078f00 msgr2=0x7f68000ffd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:35:01.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:01.028+0000 7f6805e00640 1 -- 172.21.15.90:0/3531898634 shutdown_connections 2024-04-02T14:35:01.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:01.028+0000 7f6805e00640 1 -- 172.21.15.90:0/3531898634 wait complete. 2024-04-02T14:35:01.049 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T14:35:01.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:35:01 smithi177 ceph-mon[36431]: from='client.25405 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-02T14:35:01.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:35:01 smithi177 ceph-mon[36431]: Updating smithi090:/etc/ceph/ceph.client.admin.keyring 2024-04-02T14:35:01.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:35:01 smithi177 ceph-mon[36431]: Updating smithi090:/var/lib/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/config/ceph.client.admin.keyring 2024-04-02T14:35:01.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:35:01 smithi177 ceph-mon[36431]: from='client.? 172.21.15.90:0/2523656788' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-02T14:35:01.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:35:01 smithi177 ceph-mon[36431]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' 2024-04-02T14:35:01.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:35:01 smithi177 ceph-mon[36431]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' 2024-04-02T14:35:01.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:35:01 smithi177 ceph-mon[36431]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' 2024-04-02T14:35:01.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:35:01 smithi177 ceph-mon[36431]: from='client.? 172.21.15.90:0/4008070079' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-02T14:35:01.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:35:01 smithi177 ceph-mon[36431]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T14:35:01.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:35:01 smithi177 ceph-mon[36431]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:35:01.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:35:01 smithi177 ceph-mon[36431]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T14:35:01.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:35:01 smithi177 ceph-mon[36431]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' 2024-04-02T14:35:01.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:01.246+0000 7fc6bf800640 1 -- 172.21.15.90:0/2645459301 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc6b810d3f0 msgr2=0x7fc6b8114fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:35:01.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:01.246+0000 7fc6bf800640 1 --2- 172.21.15.90:0/2645459301 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc6b810d3f0 0x7fc6b8114fc0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fc6b0009ca0 tx=0x7fc6b002c100 comp rx=0 tx=0).stop 2024-04-02T14:35:01.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:01.246+0000 7fc6bf800640 1 -- 172.21.15.90:0/2645459301 shutdown_connections 2024-04-02T14:35:01.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:01.246+0000 7fc6bf800640 1 --2- 172.21.15.90:0/2645459301 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc6b810d3f0 0x7fc6b8114fc0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:01.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:01.246+0000 7fc6bf800640 1 --2- 172.21.15.90:0/2645459301 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc6b8108cd0 0x7fc6b810cd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:01.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:01.246+0000 7fc6bf800640 1 --2- 172.21.15.90:0/2645459301 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc6b8108320 0x7fc6b8108700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:01.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:01.246+0000 7fc6bf800640 1 -- 172.21.15.90:0/2645459301 >> 172.21.15.90:0/2645459301 conn(0x7fc6b8076050 msgr2=0x7fc6b8078470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:35:01.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:01.247+0000 7fc6bf800640 1 -- 172.21.15.90:0/2645459301 shutdown_connections 2024-04-02T14:35:01.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:01.247+0000 7fc6bf800640 1 -- 172.21.15.90:0/2645459301 wait complete. 2024-04-02T14:35:01.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:01.247+0000 7fc6bf800640 1 Processor -- start 2024-04-02T14:35:01.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:01.248+0000 7fc6bf800640 1 -- start start 2024-04-02T14:35:01.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:01.248+0000 7fc6bf800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc6b8108320 0x7fc6b81a4660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:01.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:01.248+0000 7fc6bf800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc6b810d3f0 0x7fc6b81a4ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:01.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:01.248+0000 7fc6bf800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc6b819e7d0 0x7fc6b819ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:01.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:01.248+0000 7fc6bf800640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fc6b81184b0 con 0x7fc6b819e7d0 2024-04-02T14:35:01.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:01.248+0000 7fc6bf800640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fc6b8118330 con 0x7fc6b8108320 2024-04-02T14:35:01.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:01.248+0000 7fc6bf800640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fc6b8118630 con 0x7fc6b810d3f0 2024-04-02T14:35:01.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:01.248+0000 7fc6bce00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc6b8108320 0x7fc6b81a4660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:01.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:01.248+0000 7fc6bce00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc6b8108320 0x7fc6b81a4660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:59044/0 (socket says 172.21.15.90:59044) 2024-04-02T14:35:01.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:01.248+0000 7fc6bce00640 1 -- 172.21.15.90:0/1948793901 learned_addr learned my addr 172.21.15.90:0/1948793901 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:35:01.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:01.248+0000 7fc6b7e00640 1 --2- 172.21.15.90:0/1948793901 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc6b810d3f0 0x7fc6b81a4ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:01.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:01.248+0000 7fc6bd800640 1 --2- 172.21.15.90:0/1948793901 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc6b819e7d0 0x7fc6b819ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:01.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:01.249+0000 7fc6b7e00640 1 -- 172.21.15.90:0/1948793901 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc6b8108320 msgr2=0x7fc6b81a4660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:35:01.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:01.249+0000 7fc6b7e00640 1 --2- 172.21.15.90:0/1948793901 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc6b8108320 0x7fc6b81a4660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:01.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:01.249+0000 7fc6b7e00640 1 -- 172.21.15.90:0/1948793901 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc6b819e7d0 msgr2=0x7fc6b819ec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:35:01.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:01.249+0000 7fc6b7e00640 1 --2- 172.21.15.90:0/1948793901 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc6b819e7d0 0x7fc6b819ec80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:01.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:01.249+0000 7fc6b7e00640 1 -- 172.21.15.90:0/1948793901 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6b819f360 con 0x7fc6b810d3f0 2024-04-02T14:35:01.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:01.249+0000 7fc6bce00640 1 --2- 172.21.15.90:0/1948793901 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc6b8108320 0x7fc6b81a4660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:35:01.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:01.249+0000 7fc6b7e00640 1 --2- 172.21.15.90:0/1948793901 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc6b810d3f0 0x7fc6b81a4ba0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fc6a800e9c0 tx=0x7fc6a800ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:35:01.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:01.249+0000 7fc6b5600640 1 -- 172.21.15.90:0/1948793901 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6a800ccc0 con 0x7fc6b810d3f0 2024-04-02T14:35:01.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:01.249+0000 7fc6b5600640 1 -- 172.21.15.90:0/1948793901 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc6a8005020 con 0x7fc6b810d3f0 2024-04-02T14:35:01.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:01.249+0000 7fc6bf800640 1 -- 172.21.15.90:0/1948793901 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc6b819f650 con 0x7fc6b810d3f0 2024-04-02T14:35:01.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:01.249+0000 7fc6b5600640 1 -- 172.21.15.90:0/1948793901 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6a8004a00 con 0x7fc6b810d3f0 2024-04-02T14:35:01.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:01.249+0000 7fc6bf800640 1 -- 172.21.15.90:0/1948793901 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6b81ab7e0 con 0x7fc6b810d3f0 2024-04-02T14:35:01.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:01.250+0000 7fc6bf800640 1 -- 172.21.15.90:0/1948793901 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc6b81097f0 con 0x7fc6b810d3f0 2024-04-02T14:35:01.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:01.252+0000 7fc6b5600640 1 -- 172.21.15.90:0/1948793901 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7fc6a80051d0 con 0x7fc6b810d3f0 2024-04-02T14:35:01.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:01.254+0000 7fc6b5600640 1 --2- 172.21.15.90:0/1948793901 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fc69c0a3e40 0x7fc69c0a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:01.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:01.254+0000 7fc6b5600640 1 -- 172.21.15.90:0/1948793901 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7fc6a80d09b0 con 0x7fc6b810d3f0 2024-04-02T14:35:01.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:01.254+0000 7fc6bce00640 1 --2- 172.21.15.90:0/1948793901 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fc69c0a3e40 0x7fc69c0a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:01.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:01.255+0000 7fc6bce00640 1 --2- 172.21.15.90:0/1948793901 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fc69c0a3e40 0x7fc69c0a6300 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fc6a4005fd0 tx=0x7fc6a400b4a0 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:35:01.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:01.256+0000 7fc6b5600640 1 -- 172.21.15.90:0/1948793901 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7fc6b819f650 con 0x7fc6b810d3f0 2024-04-02T14:35:01.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:35:01 smithi092 ceph-mon[36497]: from='client.25405 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-02T14:35:01.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:35:01 smithi092 ceph-mon[36497]: Updating smithi090:/etc/ceph/ceph.client.admin.keyring 2024-04-02T14:35:01.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:35:01 smithi092 ceph-mon[36497]: Updating smithi090:/var/lib/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/config/ceph.client.admin.keyring 2024-04-02T14:35:01.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:35:01 smithi092 ceph-mon[36497]: from='client.? 172.21.15.90:0/2523656788' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-02T14:35:01.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:35:01 smithi092 ceph-mon[36497]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' 2024-04-02T14:35:01.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:35:01 smithi092 ceph-mon[36497]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' 2024-04-02T14:35:01.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:35:01 smithi092 ceph-mon[36497]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' 2024-04-02T14:35:01.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:35:01 smithi092 ceph-mon[36497]: from='client.? 172.21.15.90:0/4008070079' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-02T14:35:01.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:35:01 smithi092 ceph-mon[36497]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T14:35:01.362 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:35:01 smithi092 ceph-mon[36497]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:35:01.362 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:35:01 smithi092 ceph-mon[36497]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T14:35:01.362 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:35:01 smithi092 ceph-mon[36497]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' 2024-04-02T14:35:01.439 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:35:01 smithi090 ceph-mon[31674]: from='client.25405 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-02T14:35:01.440 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:35:01 smithi090 ceph-mon[31674]: Updating smithi090:/etc/ceph/ceph.client.admin.keyring 2024-04-02T14:35:01.440 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:35:01 smithi090 ceph-mon[31674]: Updating smithi090:/var/lib/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/config/ceph.client.admin.keyring 2024-04-02T14:35:01.440 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:35:01 smithi090 ceph-mon[31674]: from='client.? 172.21.15.90:0/2523656788' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-02T14:35:01.440 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:35:01 smithi090 ceph-mon[31674]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' 2024-04-02T14:35:01.440 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:35:01 smithi090 ceph-mon[31674]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' 2024-04-02T14:35:01.440 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:35:01 smithi090 ceph-mon[31674]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' 2024-04-02T14:35:01.440 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:35:01 smithi090 ceph-mon[31674]: from='client.? 172.21.15.90:0/4008070079' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-02T14:35:01.440 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:35:01 smithi090 ceph-mon[31674]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T14:35:01.440 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:35:01 smithi090 ceph-mon[31674]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:35:01.440 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:35:01 smithi090 ceph-mon[31674]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T14:35:01.440 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:35:01 smithi090 ceph-mon[31674]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' 2024-04-02T14:35:01.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:01.501+0000 7fc6bf800640 1 -- 172.21.15.90:0/1948793901 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc6b8068d40 con 0x7fc6b810d3f0 2024-04-02T14:35:01.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:01.502+0000 7fc6b5600640 1 -- 172.21.15.90:0/1948793901 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11927 (secure 0 0 0) 0x7fc6b8068d40 con 0x7fc6b810d3f0 2024-04-02T14:35:01.504 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-02T14:35:01.504 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14412,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6837/2852160309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6836","nonce":2852160309},{"type":"v1","addr":"172.21.15.90:6837","nonce":2852160309}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14478,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6841/1919376858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6840","nonce":1919376858},{"type":"v1","addr":"172.21.15.90:6841","nonce":1919376858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-02T14:32:22.319083+0000","modified":"2024-04-02T14:34:52.266152+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24590,"mds_1":24560,"mds_2":24547,"mds_3":24539,"mds_4":24523},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24454":{"gid":24454,"name":"l","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6833/1564319431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":1564319431},{"type":"v1","addr":"172.21.15.177:6833","nonce":1564319431}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.92:6833/1570337372","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":1570337372},{"type":"v1","addr":"172.21.15.92:6833","nonce":1570337372}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"j","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.90:6835/1277124891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6834","nonce":1277124891},{"type":"v1","addr":"172.21.15.90:6835","nonce":1277124891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.92:6835/860721164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":860721164},{"type":"v1","addr":"172.21.15.92:6835","nonce":860721164}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24511":{"gid":24511,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6835/3177244988","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":3177244988},{"type":"v1","addr":"172.21.15.177:6835","nonce":3177244988}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"c","rank":4,"incarnation":25,"state":"up:active","state_seq":9,"addr":"172.21.15.177:6837/4223414858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":4223414858},{"type":"v1","addr":"172.21.15.177:6837","nonce":4223414858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24539":{"gid":24539,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":8,"addr":"172.21.15.92:6837/1270635939","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":1270635939},{"type":"v1","addr":"172.21.15.92:6837","nonce":1270635939}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24547":{"gid":24547,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":7,"addr":"172.21.15.177:6839/664895663","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":664895663},{"type":"v1","addr":"172.21.15.177:6839","nonce":664895663}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"g","rank":1,"incarnation":19,"state":"up:active","state_seq":5,"addr":"172.21.15.90:6839/1498915870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6838","nonce":1498915870},{"type":"v1","addr":"172.21.15.90:6839","nonce":1498915870}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24590":{"gid":24590,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.92:6839/1054888219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":1054888219},{"type":"v1","addr":"172.21.15.92:6839","nonce":1054888219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24590,"qdb_cluster":[24547,24590,24523,24539,24560]},"id":1}]} 2024-04-02T14:35:01.504 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 27 2024-04-02T14:35:01.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:01.505+0000 7fc6bf800640 1 -- 172.21.15.90:0/1948793901 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fc69c0a3e40 msgr2=0x7fc69c0a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:35:01.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:01.505+0000 7fc6bf800640 1 --2- 172.21.15.90:0/1948793901 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fc69c0a3e40 0x7fc69c0a6300 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fc6a4005fd0 tx=0x7fc6a400b4a0 comp rx=0 tx=0).stop 2024-04-02T14:35:01.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:01.505+0000 7fc6bf800640 1 -- 172.21.15.90:0/1948793901 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc6b810d3f0 msgr2=0x7fc6b81a4ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:35:01.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:01.505+0000 7fc6bf800640 1 --2- 172.21.15.90:0/1948793901 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc6b810d3f0 0x7fc6b81a4ba0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fc6a800e9c0 tx=0x7fc6a800ee80 comp rx=0 tx=0).stop 2024-04-02T14:35:01.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:01.505+0000 7fc6bf800640 1 -- 172.21.15.90:0/1948793901 shutdown_connections 2024-04-02T14:35:01.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:01.505+0000 7fc6bf800640 1 --2- 172.21.15.90:0/1948793901 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fc69c0a3e40 0x7fc69c0a6300 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:01.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:01.505+0000 7fc6bf800640 1 --2- 172.21.15.90:0/1948793901 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc6b819e7d0 0x7fc6b819ec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:01.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:01.505+0000 7fc6bf800640 1 --2- 172.21.15.90:0/1948793901 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc6b810d3f0 0x7fc6b81a4ba0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:01.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:01.505+0000 7fc6bf800640 1 --2- 172.21.15.90:0/1948793901 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc6b8108320 0x7fc6b81a4660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:01.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:01.505+0000 7fc6bf800640 1 -- 172.21.15.90:0/1948793901 >> 172.21.15.90:0/1948793901 conn(0x7fc6b8076050 msgr2=0x7fc6b80783b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:35:01.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:01.505+0000 7fc6bf800640 1 -- 172.21.15.90:0/1948793901 shutdown_connections 2024-04-02T14:35:01.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:01.505+0000 7fc6bf800640 1 -- 172.21.15.90:0/1948793901 wait complete. 2024-04-02T14:35:01.526 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T14:35:01.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:01.735+0000 7ffadf600640 1 -- 172.21.15.90:0/2098356855 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffad8076030 msgr2=0x7ffad8076410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:35:01.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:01.735+0000 7ffadf600640 1 --2- 172.21.15.90:0/2098356855 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffad8076030 0x7ffad8076410 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7ffacc009900 tx=0x7ffacc02bc40 comp rx=0 tx=0).stop 2024-04-02T14:35:01.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:01.735+0000 7ffadf600640 1 -- 172.21.15.90:0/2098356855 shutdown_connections 2024-04-02T14:35:01.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:01.735+0000 7ffadf600640 1 --2- 172.21.15.90:0/2098356855 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffad8112930 0x7ffad8114d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:01.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:01.735+0000 7ffadf600640 1 --2- 172.21.15.90:0/2098356855 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ffad8076950 0x7ffad81123f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:01.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:01.735+0000 7ffadf600640 1 --2- 172.21.15.90:0/2098356855 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffad8076030 0x7ffad8076410 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:01.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:01.735+0000 7ffadf600640 1 -- 172.21.15.90:0/2098356855 >> 172.21.15.90:0/2098356855 conn(0x7ffad80ff760 msgr2=0x7ffad8101b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:35:01.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:01.735+0000 7ffadf600640 1 -- 172.21.15.90:0/2098356855 shutdown_connections 2024-04-02T14:35:01.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:01.735+0000 7ffadf600640 1 -- 172.21.15.90:0/2098356855 wait complete. 2024-04-02T14:35:01.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:01.736+0000 7ffadf600640 1 Processor -- start 2024-04-02T14:35:01.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:01.736+0000 7ffadf600640 1 -- start start 2024-04-02T14:35:01.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:01.736+0000 7ffadf600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffad8076030 0x7ffad81a4610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:01.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:01.736+0000 7ffadf600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ffad8076950 0x7ffad81a4b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:01.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:01.736+0000 7ffadf600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffad8112930 0x7ffad819e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:01.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:01.736+0000 7ffadf600640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7ffad81184d0 con 0x7ffad8076030 2024-04-02T14:35:01.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:01.736+0000 7ffadf600640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7ffad8118350 con 0x7ffad8076950 2024-04-02T14:35:01.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:01.736+0000 7ffadf600640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7ffad8118650 con 0x7ffad8112930 2024-04-02T14:35:01.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:01.737+0000 7ffadcc00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffad8076030 0x7ffad81a4610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:01.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:01.737+0000 7ffadcc00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffad8076030 0x7ffad81a4610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:53492/0 (socket says 172.21.15.90:53492) 2024-04-02T14:35:01.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:01.737+0000 7ffadcc00640 1 -- 172.21.15.90:0/2509016358 learned_addr learned my addr 172.21.15.90:0/2509016358 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:35:01.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:01.737+0000 7ffad7e00640 1 --2- 172.21.15.90:0/2509016358 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ffad8076950 0x7ffad81a4b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:01.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:01.737+0000 7ffadd600640 1 --2- 172.21.15.90:0/2509016358 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffad8112930 0x7ffad819e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:01.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:01.737+0000 7ffadcc00640 1 -- 172.21.15.90:0/2509016358 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffad8112930 msgr2=0x7ffad819e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:35:01.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:01.737+0000 7ffadcc00640 1 --2- 172.21.15.90:0/2509016358 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffad8112930 0x7ffad819e7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:01.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:01.737+0000 7ffadcc00640 1 -- 172.21.15.90:0/2509016358 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ffad8076950 msgr2=0x7ffad81a4b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:35:01.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:01.737+0000 7ffadcc00640 1 --2- 172.21.15.90:0/2509016358 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ffad8076950 0x7ffad81a4b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:01.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:01.737+0000 7ffadcc00640 1 -- 172.21.15.90:0/2509016358 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffad819efa0 con 0x7ffad8076030 2024-04-02T14:35:01.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:01.737+0000 7ffad7e00640 1 --2- 172.21.15.90:0/2509016358 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ffad8076950 0x7ffad81a4b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:35:01.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:01.737+0000 7ffadd600640 1 --2- 172.21.15.90:0/2509016358 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffad8112930 0x7ffad819e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:35:01.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:01.738+0000 7ffadcc00640 1 --2- 172.21.15.90:0/2509016358 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffad8076030 0x7ffad81a4610 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7ffacc005ab0 tx=0x7ffacc004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:35:01.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:01.738+0000 7ffad5600640 1 -- 172.21.15.90:0/2509016358 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffacc041070 con 0x7ffad8076030 2024-04-02T14:35:01.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:01.738+0000 7ffad5600640 1 -- 172.21.15.90:0/2509016358 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffacc0054e0 con 0x7ffad8076030 2024-04-02T14:35:01.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:01.738+0000 7ffadf600640 1 -- 172.21.15.90:0/2509016358 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffad819f230 con 0x7ffad8076030 2024-04-02T14:35:01.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:01.738+0000 7ffad5600640 1 -- 172.21.15.90:0/2509016358 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffacc003cc0 con 0x7ffad8076030 2024-04-02T14:35:01.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:01.738+0000 7ffadf600640 1 -- 172.21.15.90:0/2509016358 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffad81ab460 con 0x7ffad8076030 2024-04-02T14:35:01.741 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:01.739+0000 7ffadf600640 1 -- 172.21.15.90:0/2509016358 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffad8077770 con 0x7ffad8076030 2024-04-02T14:35:01.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:01.741+0000 7ffad5600640 1 -- 172.21.15.90:0/2509016358 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7ffacc044070 con 0x7ffad8076030 2024-04-02T14:35:01.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:01.742+0000 7ffad5600640 1 --2- 172.21.15.90:0/2509016358 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7ffac00a3e40 0x7ffac00a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:01.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:01.742+0000 7ffad5600640 1 -- 172.21.15.90:0/2509016358 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7ffacc0f26d0 con 0x7ffad8076030 2024-04-02T14:35:01.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:01.743+0000 7ffad7e00640 1 --2- 172.21.15.90:0/2509016358 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7ffac00a3e40 0x7ffac00a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:01.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:01.743+0000 7ffad7e00640 1 --2- 172.21.15.90:0/2509016358 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7ffac00a3e40 0x7ffac00a6300 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7ffac40095f0 tx=0x7ffac4005f90 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:35:01.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:01.747+0000 7ffad5600640 1 -- 172.21.15.90:0/2509016358 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7ffacc0bebe0 con 0x7ffad8076030 2024-04-02T14:35:01.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:01.955+0000 7ffadf600640 1 -- 172.21.15.90:0/2509016358 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ffad810dac0 con 0x7ffad8076030 2024-04-02T14:35:01.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:01.957+0000 7ffad5600640 1 -- 172.21.15.90:0/2509016358 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+23024 (secure 0 0 0) 0x7ffacc0be4e0 con 0x7ffad8076030 2024-04-02T14:35:01.959 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-02T14:35:01.959 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":91,"fsid":"6449448a-f0fc-11ee-b647-cb9ed24678a4","created":"2024-04-02T14:23:16.530201+0000","modified":"2024-04-02T14:34:52.235281+0000","last_up_change":"2024-04-02T14:31:23.819732+0000","last_in_change":"2024-04-02T14:31:02.824284+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T14:27:16.758622+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-02T14:32:20.016331+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-02T14:32:21.294367+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a1a6a856-ab44-44bd-bc5e-45c718a70cd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":3635087609},{"type":"v1","addr":"172.21.15.90:6803","nonce":3635087609}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":3635087609},{"type":"v1","addr":"172.21.15.90:6805","nonce":3635087609}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":3635087609},{"type":"v1","addr":"172.21.15.90:6809","nonce":3635087609}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":3635087609},{"type":"v1","addr":"172.21.15.90:6807","nonce":3635087609}]},"public_addr":"172.21.15.90:6803/3635087609","cluster_addr":"172.21.15.90:6805/3635087609","heartbeat_back_addr":"172.21.15.90:6809/3635087609","heartbeat_front_addr":"172.21.15.90:6807/3635087609","state":["exists","up"]},{"osd":1,"uuid":"124139cd-5793-463e-9607-9d9c10164183","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":1994536059},{"type":"v1","addr":"172.21.15.92:6801","nonce":1994536059}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":1994536059},{"type":"v1","addr":"172.21.15.92:6803","nonce":1994536059}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":1994536059},{"type":"v1","addr":"172.21.15.92:6807","nonce":1994536059}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":1994536059},{"type":"v1","addr":"172.21.15.92:6805","nonce":1994536059}]},"public_addr":"172.21.15.92:6801/1994536059","cluster_addr":"172.21.15.92:6803/1994536059","heartbeat_back_addr":"172.21.15.92:6807/1994536059","heartbeat_front_addr":"172.21.15.92:6805/1994536059","state":["exists","up"]},{"osd":2,"uuid":"a61fbef2-9dca-4875-a814-020fff0db0e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":1358472800},{"type":"v1","addr":"172.21.15.177:6801","nonce":1358472800}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":1358472800},{"type":"v1","addr":"172.21.15.177:6803","nonce":1358472800}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":1358472800},{"type":"v1","addr":"172.21.15.177:6807","nonce":1358472800}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":1358472800},{"type":"v1","addr":"172.21.15.177:6805","nonce":1358472800}]},"public_addr":"172.21.15.177:6801/1358472800","cluster_addr":"172.21.15.177:6803/1358472800","heartbeat_back_addr":"172.21.15.177:6807/1358472800","heartbeat_front_addr":"172.21.15.177:6805/1358472800","state":["exists","up"]},{"osd":3,"uuid":"b175a2f4-7859-4eb0-bc1f-fba8c2b92a2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":3114811216},{"type":"v1","addr":"172.21.15.90:6811","nonce":3114811216}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":3114811216},{"type":"v1","addr":"172.21.15.90:6813","nonce":3114811216}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":3114811216},{"type":"v1","addr":"172.21.15.90:6817","nonce":3114811216}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":3114811216},{"type":"v1","addr":"172.21.15.90:6815","nonce":3114811216}]},"public_addr":"172.21.15.90:6811/3114811216","cluster_addr":"172.21.15.90:6813/3114811216","heartbeat_back_addr":"172.21.15.90:6817/3114811216","heartbeat_front_addr":"172.21.15.90:6815/3114811216","state":["exists","up"]},{"osd":4,"uuid":"38cc68ea-c109-4d8d-b77d-1c6f9b56c875","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":2987208879},{"type":"v1","addr":"172.21.15.92:6809","nonce":2987208879}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6810","nonce":2987208879},{"type":"v1","addr":"172.21.15.92:6811","nonce":2987208879}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6814","nonce":2987208879},{"type":"v1","addr":"172.21.15.92:6815","nonce":2987208879}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":2987208879},{"type":"v1","addr":"172.21.15.92:6813","nonce":2987208879}]},"public_addr":"172.21.15.92:6809/2987208879","cluster_addr":"172.21.15.92:6811/2987208879","heartbeat_back_addr":"172.21.15.92:6815/2987208879","heartbeat_front_addr":"172.21.15.92:6813/2987208879","state":["exists","up"]},{"osd":5,"uuid":"2347232e-ac91-466f-8ac9-496f397679ba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":3486820237},{"type":"v1","addr":"172.21.15.177:6809","nonce":3486820237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":3486820237},{"type":"v1","addr":"172.21.15.177:6811","nonce":3486820237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":3486820237},{"type":"v1","addr":"172.21.15.177:6815","nonce":3486820237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":3486820237},{"type":"v1","addr":"172.21.15.177:6813","nonce":3486820237}]},"public_addr":"172.21.15.177:6809/3486820237","cluster_addr":"172.21.15.177:6811/3486820237","heartbeat_back_addr":"172.21.15.177:6815/3486820237","heartbeat_front_addr":"172.21.15.177:6813/3486820237","state":["exists","up"]},{"osd":6,"uuid":"aa72392f-0498-44db-a8d2-eccc89a64871","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":2579716443},{"type":"v1","addr":"172.21.15.90:6819","nonce":2579716443}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6820","nonce":2579716443},{"type":"v1","addr":"172.21.15.90:6821","nonce":2579716443}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":2579716443},{"type":"v1","addr":"172.21.15.90:6825","nonce":2579716443}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2579716443},{"type":"v1","addr":"172.21.15.90:6823","nonce":2579716443}]},"public_addr":"172.21.15.90:6819/2579716443","cluster_addr":"172.21.15.90:6821/2579716443","heartbeat_back_addr":"172.21.15.90:6825/2579716443","heartbeat_front_addr":"172.21.15.90:6823/2579716443","state":["exists","up"]},{"osd":7,"uuid":"931bd105-bc14-4a1e-bfe4-c7ad58fa4b73","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":3371649139},{"type":"v1","addr":"172.21.15.92:6817","nonce":3371649139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":3371649139},{"type":"v1","addr":"172.21.15.92:6819","nonce":3371649139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":3371649139},{"type":"v1","addr":"172.21.15.92:6823","nonce":3371649139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6820","nonce":3371649139},{"type":"v1","addr":"172.21.15.92:6821","nonce":3371649139}]},"public_addr":"172.21.15.92:6817/3371649139","cluster_addr":"172.21.15.92:6819/3371649139","heartbeat_back_addr":"172.21.15.92:6823/3371649139","heartbeat_front_addr":"172.21.15.92:6821/3371649139","state":["exists","up"]},{"osd":8,"uuid":"82624ec2-b382-4886-9f40-7f5f376de3ce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":1459053034},{"type":"v1","addr":"172.21.15.177:6817","nonce":1459053034}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":1459053034},{"type":"v1","addr":"172.21.15.177:6819","nonce":1459053034}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":1459053034},{"type":"v1","addr":"172.21.15.177:6823","nonce":1459053034}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":1459053034},{"type":"v1","addr":"172.21.15.177:6821","nonce":1459053034}]},"public_addr":"172.21.15.177:6817/1459053034","cluster_addr":"172.21.15.177:6819/1459053034","heartbeat_back_addr":"172.21.15.177:6823/1459053034","heartbeat_front_addr":"172.21.15.177:6821/1459053034","state":["exists","up"]},{"osd":9,"uuid":"fbe36e48-3468-42b2-8094-e16552f3cb16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":2592604332},{"type":"v1","addr":"172.21.15.90:6827","nonce":2592604332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":2592604332},{"type":"v1","addr":"172.21.15.90:6829","nonce":2592604332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6832","nonce":2592604332},{"type":"v1","addr":"172.21.15.90:6833","nonce":2592604332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":2592604332},{"type":"v1","addr":"172.21.15.90:6831","nonce":2592604332}]},"public_addr":"172.21.15.90:6827/2592604332","cluster_addr":"172.21.15.90:6829/2592604332","heartbeat_back_addr":"172.21.15.90:6833/2592604332","heartbeat_front_addr":"172.21.15.90:6831/2592604332","state":["exists","up"]},{"osd":10,"uuid":"9240df75-e7e2-47f1-8aae-534b70b3c210","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":1594153142},{"type":"v1","addr":"172.21.15.92:6825","nonce":1594153142}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6826","nonce":1594153142},{"type":"v1","addr":"172.21.15.92:6827","nonce":1594153142}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":1594153142},{"type":"v1","addr":"172.21.15.92:6831","nonce":1594153142}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":1594153142},{"type":"v1","addr":"172.21.15.92:6829","nonce":1594153142}]},"public_addr":"172.21.15.92:6825/1594153142","cluster_addr":"172.21.15.92:6827/1594153142","heartbeat_back_addr":"172.21.15.92:6831/1594153142","heartbeat_front_addr":"172.21.15.92:6829/1594153142","state":["exists","up"]},{"osd":11,"uuid":"83cba9fa-b745-4df1-adda-e86837d4b91a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":83,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1622582126},{"type":"v1","addr":"172.21.15.177:6825","nonce":1622582126}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":1622582126},{"type":"v1","addr":"172.21.15.177:6827","nonce":1622582126}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":1622582126},{"type":"v1","addr":"172.21.15.177:6831","nonce":1622582126}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1622582126},{"type":"v1","addr":"172.21.15.177:6829","nonce":1622582126}]},"public_addr":"172.21.15.177:6825/1622582126","cluster_addr":"172.21.15.177:6827/1622582126","heartbeat_back_addr":"172.21.15.177:6831/1622582126","heartbeat_front_addr":"172.21.15.177:6829/1622582126","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:26:24.290365+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:26:49.405988+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:27:13.639196+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:27:45.603746+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:28:11.019126+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:28:35.084163+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:29:08.064626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:29:33.266133+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:29:57.381816+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:30:30.286847+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:30:56.226279+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:31:20.809611+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":7,"to":11}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":8,"to":4}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.11","mappings":[{"from":4,"to":8},{"from":5,"to":9}]},{"pgid":"3.21","mappings":[{"from":5,"to":9}]},{"pgid":"3.23","mappings":[{"from":4,"to":7}]},{"pgid":"3.3a","mappings":[{"from":4,"to":6}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.90:0/2805994468":"2024-04-03T14:34:52.235215+0000","172.21.15.90:0/1431307002":"2024-04-03T14:34:52.235215+0000","172.21.15.90:6801/4266054817":"2024-04-03T14:34:52.235215+0000","172.21.15.90:0/314641601":"2024-04-03T14:24:34.595561+0000","172.21.15.90:0/1400895868":"2024-04-03T14:24:34.595561+0000","172.21.15.90:6801/129436436":"2024-04-03T14:24:34.595561+0000","172.21.15.90:0/1335673697":"2024-04-03T14:34:52.235215+0000","172.21.15.90:6801/2372398598":"2024-04-03T14:24:00.362147+0000","172.21.15.90:0/278569338":"2024-04-03T14:24:34.595561+0000","172.21.15.90:0/2321464594":"2024-04-03T14:24:00.362147+0000","172.21.15.90:6800/4266054817":"2024-04-03T14:34:52.235215+0000","172.21.15.90:6800/2372398598":"2024-04-03T14:24:00.362147+0000","172.21.15.90:0/2962656672":"2024-04-03T14:24:00.362147+0000","172.21.15.90:0/126693610":"2024-04-03T14:24:00.362147+0000","172.21.15.90:0/1757385857":"2024-04-03T14:34:52.235215+0000","172.21.15.90:6800/129436436":"2024-04-03T14:24:34.595561+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T14:35:01.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:01.960+0000 7ffadf600640 1 -- 172.21.15.90:0/2509016358 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7ffac00a3e40 msgr2=0x7ffac00a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:35:01.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:01.960+0000 7ffadf600640 1 --2- 172.21.15.90:0/2509016358 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7ffac00a3e40 0x7ffac00a6300 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7ffac40095f0 tx=0x7ffac4005f90 comp rx=0 tx=0).stop 2024-04-02T14:35:01.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:01.960+0000 7ffadf600640 1 -- 172.21.15.90:0/2509016358 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffad8076030 msgr2=0x7ffad81a4610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:35:01.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:01.960+0000 7ffadf600640 1 --2- 172.21.15.90:0/2509016358 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffad8076030 0x7ffad81a4610 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7ffacc005ab0 tx=0x7ffacc004270 comp rx=0 tx=0).stop 2024-04-02T14:35:01.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:01.960+0000 7ffadf600640 1 -- 172.21.15.90:0/2509016358 shutdown_connections 2024-04-02T14:35:01.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:01.960+0000 7ffadf600640 1 --2- 172.21.15.90:0/2509016358 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7ffac00a3e40 0x7ffac00a6300 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:01.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:01.960+0000 7ffadf600640 1 --2- 172.21.15.90:0/2509016358 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffad8112930 0x7ffad819e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:01.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:01.960+0000 7ffadf600640 1 --2- 172.21.15.90:0/2509016358 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ffad8076950 0x7ffad81a4b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:01.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:01.960+0000 7ffadf600640 1 --2- 172.21.15.90:0/2509016358 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffad8076030 0x7ffad81a4610 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:01.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:01.960+0000 7ffadf600640 1 -- 172.21.15.90:0/2509016358 >> 172.21.15.90:0/2509016358 conn(0x7ffad80ff760 msgr2=0x7ffad8113f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:35:01.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:01.961+0000 7ffadf600640 1 -- 172.21.15.90:0/2509016358 shutdown_connections 2024-04-02T14:35:01.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:01.961+0000 7ffadf600640 1 -- 172.21.15.90:0/2509016358 wait complete. 2024-04-02T14:35:01.982 DEBUG:tasks.fwd_scrub:Yielding 2024-04-02T14:35:01.982 INFO:teuthology.run_tasks:Running task exec... 2024-04-02T14:35:01.993 INFO:teuthology.task.exec:Executing custom commands... 2024-04-02T14:35:01.993 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi090.front.sepia.ceph.com 2024-04-02T14:35:01.993 DEBUG:teuthology.orchestra.run.smithi090:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m' 2024-04-02T14:35:01.993 INFO:tasks.fwd_scrub.fs.[cephfs]:start scrubbing fs: cephfs 2024-04-02T14:35:01.993 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-02T14:35:01.994 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T14:35:01.994 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-02T14:35:02.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.187+0000 7f957e000640 1 -- 172.21.15.90:0/286517489 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f95781018e0 msgr2=0x7f9578101d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:35:02.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.187+0000 7f957e000640 1 --2- 172.21.15.90:0/286517489 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f95781018e0 0x7f9578101d60 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f9564009900 tx=0x7f956402bc50 comp rx=0 tx=0).stop 2024-04-02T14:35:02.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.187+0000 7f957e000640 1 -- 172.21.15.90:0/286517489 shutdown_connections 2024-04-02T14:35:02.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.187+0000 7f957e000640 1 --2- 172.21.15.90:0/286517489 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f95781022a0 0x7f957810e7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:02.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.187+0000 7f957e000640 1 --2- 172.21.15.90:0/286517489 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f95781018e0 0x7f9578101d60 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:02.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.187+0000 7f957e000640 1 --2- 172.21.15.90:0/286517489 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f95781078f0 0x7f9578107cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:02.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.187+0000 7f957e000640 1 -- 172.21.15.90:0/286517489 >> 172.21.15.90:0/286517489 conn(0x7f95780fd450 msgr2=0x7f95780ff870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:35:02.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.187+0000 7f957e000640 1 -- 172.21.15.90:0/286517489 shutdown_connections 2024-04-02T14:35:02.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.187+0000 7f957e000640 1 -- 172.21.15.90:0/286517489 wait complete. 2024-04-02T14:35:02.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.188+0000 7f957e000640 1 Processor -- start 2024-04-02T14:35:02.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.188+0000 7f957e000640 1 -- start start 2024-04-02T14:35:02.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.188+0000 7f957e000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f95781018e0 0x7f957819a3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:02.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.189+0000 7fdbe7e00640 1 -- 172.21.15.90:0/3457937803 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdbe0102430 msgr2=0x7fdbe010e940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:35:02.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.189+0000 7f957e000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f95781022a0 0x7f957819a910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:02.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.189+0000 7fdbe7e00640 1 --2- 172.21.15.90:0/3457937803 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdbe0102430 0x7fdbe010e940 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fdbd8009cd0 tx=0x7fdbd802c0e0 comp rx=0 tx=0).stop 2024-04-02T14:35:02.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.189+0000 7f957e000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f95781078f0 0x7f957819ec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:02.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.189+0000 7f957e000640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f9578111d50 con 0x7f95781022a0 2024-04-02T14:35:02.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.189+0000 7f957e000640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f9578111bd0 con 0x7f95781078f0 2024-04-02T14:35:02.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.189+0000 7f957e000640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f9578111ed0 con 0x7f95781018e0 2024-04-02T14:35:02.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.189+0000 7fdbe7e00640 1 -- 172.21.15.90:0/3457937803 shutdown_connections 2024-04-02T14:35:02.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.189+0000 7fdbe7e00640 1 --2- 172.21.15.90:0/3457937803 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdbe0102430 0x7fdbe010e940 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:02.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.189+0000 7fdbe7e00640 1 --2- 172.21.15.90:0/3457937803 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdbe0101a70 0x7fdbe0101ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:02.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.189+0000 7fdbe7e00640 1 --2- 172.21.15.90:0/3457937803 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdbe0107a80 0x7fdbe0107e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:02.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.189+0000 7fdbe7e00640 1 -- 172.21.15.90:0/3457937803 >> 172.21.15.90:0/3457937803 conn(0x7fdbe00fd560 msgr2=0x7fdbe00ff980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:35:02.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.189+0000 7f957cc00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f95781018e0 0x7f957819a3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:02.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.189+0000 7f9577e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f95781022a0 0x7f957819a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:02.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.189+0000 7f957cc00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f95781018e0 0x7f957819a3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:49616/0 (socket says 172.21.15.90:49616) 2024-04-02T14:35:02.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.189+0000 7f957cc00640 1 -- 172.21.15.90:0/341079461 learned_addr learned my addr 172.21.15.90:0/341079461 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:35:02.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.189+0000 7f957d600640 1 --2- 172.21.15.90:0/341079461 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f95781078f0 0x7f957819ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:02.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.189+0000 7fdbe7e00640 1 -- 172.21.15.90:0/3457937803 shutdown_connections 2024-04-02T14:35:02.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.189+0000 7fdbe7e00640 1 -- 172.21.15.90:0/3457937803 wait complete. 2024-04-02T14:35:02.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.189+0000 7f957cc00640 1 -- 172.21.15.90:0/341079461 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f95781078f0 msgr2=0x7f957819ec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:35:02.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.189+0000 7f957cc00640 1 --2- 172.21.15.90:0/341079461 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f95781078f0 0x7f957819ec90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:02.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.189+0000 7f957cc00640 1 -- 172.21.15.90:0/341079461 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f95781022a0 msgr2=0x7f957819a910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:35:02.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.189+0000 7f957cc00640 1 --2- 172.21.15.90:0/341079461 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f95781022a0 0x7f957819a910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:02.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.189+0000 7f957cc00640 1 -- 172.21.15.90:0/341079461 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f957819f3c0 con 0x7f95781018e0 2024-04-02T14:35:02.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.189+0000 7f957d600640 1 --2- 172.21.15.90:0/341079461 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f95781078f0 0x7f957819ec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:35:02.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.190+0000 7f957cc00640 1 --2- 172.21.15.90:0/341079461 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f95781018e0 0x7f957819a3d0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f956c00e9f0 tx=0x7f956c00edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:35:02.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.190+0000 7f9575600640 1 -- 172.21.15.90:0/341079461 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f956c00cd80 con 0x7f95781018e0 2024-04-02T14:35:02.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.190+0000 7f9575600640 1 -- 172.21.15.90:0/341079461 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f956c005500 con 0x7f95781018e0 2024-04-02T14:35:02.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.190+0000 7fdbe7e00640 1 Processor -- start 2024-04-02T14:35:02.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.190+0000 7f957e000640 1 -- 172.21.15.90:0/341079461 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f957819f6b0 con 0x7f95781018e0 2024-04-02T14:35:02.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.190+0000 7f9575600640 1 -- 172.21.15.90:0/341079461 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f956c010690 con 0x7f95781018e0 2024-04-02T14:35:02.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.190+0000 7f957e000640 1 -- 172.21.15.90:0/341079461 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95781a6f40 con 0x7f95781018e0 2024-04-02T14:35:02.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.190+0000 7fdbe7e00640 1 -- start start 2024-04-02T14:35:02.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.191+0000 7fdbe7e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdbe0101a70 0x7fdbe0073e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:02.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.191+0000 7fdbe7e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdbe0102430 0x7fdbe0074340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:02.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.191+0000 7fdbe7e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdbe0107a80 0x7fdbe0074880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:02.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.191+0000 7fdbe7e00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fdbe0111e60 con 0x7fdbe0107a80 2024-04-02T14:35:02.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.191+0000 7fdbe7e00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fdbe0111ce0 con 0x7fdbe0102430 2024-04-02T14:35:02.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.191+0000 7fdbe7e00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fdbe0111fe0 con 0x7fdbe0101a70 2024-04-02T14:35:02.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.191+0000 7fdbe4a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdbe0102430 0x7fdbe0074340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:02.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.191+0000 7fdbe5400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdbe0101a70 0x7fdbe0073e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:02.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.191+0000 7fdbe5e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdbe0107a80 0x7fdbe0074880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:02.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.191+0000 7fdbe4a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdbe0102430 0x7fdbe0074340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:59102/0 (socket says 172.21.15.90:59102) 2024-04-02T14:35:02.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.191+0000 7fdbe5400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdbe0101a70 0x7fdbe0073e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:49628/0 (socket says 172.21.15.90:49628) 2024-04-02T14:35:02.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.191+0000 7fdbe4a00640 1 -- 172.21.15.90:0/2263896571 learned_addr learned my addr 172.21.15.90:0/2263896571 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:35:02.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.191+0000 7fdbe4a00640 1 -- 172.21.15.90:0/2263896571 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdbe0101a70 msgr2=0x7fdbe0073e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:35:02.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.191+0000 7fdbe4a00640 1 --2- 172.21.15.90:0/2263896571 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdbe0101a70 0x7fdbe0073e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:02.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.191+0000 7fdbe4a00640 1 -- 172.21.15.90:0/2263896571 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdbe0107a80 msgr2=0x7fdbe0074880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:35:02.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.191+0000 7fdbe4a00640 1 --2- 172.21.15.90:0/2263896571 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdbe0107a80 0x7fdbe0074880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:02.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.191+0000 7fdbe4a00640 1 -- 172.21.15.90:0/2263896571 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdbe01a6880 con 0x7fdbe0102430 2024-04-02T14:35:02.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.192+0000 7fdbe4a00640 1 --2- 172.21.15.90:0/2263896571 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdbe0102430 0x7fdbe0074340 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fdbcc00ea60 tx=0x7fdbcc00ef20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:35:02.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.192+0000 7f957e000640 1 -- 172.21.15.90:0/341079461 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9578103020 con 0x7f95781018e0 2024-04-02T14:35:02.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.192+0000 7fdbde000640 1 -- 172.21.15.90:0/2263896571 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbcc00cde0 con 0x7fdbe0102430 2024-04-02T14:35:02.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.192+0000 7fdbe7e00640 1 -- 172.21.15.90:0/2263896571 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdbe01a6b70 con 0x7fdbe0102430 2024-04-02T14:35:02.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.192+0000 7fdbde000640 1 -- 172.21.15.90:0/2263896571 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdbcc0035f0 con 0x7fdbe0102430 2024-04-02T14:35:02.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.192+0000 7fdbde000640 1 -- 172.21.15.90:0/2263896571 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbcc005220 con 0x7fdbe0102430 2024-04-02T14:35:02.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.192+0000 7fdbe7e00640 1 -- 172.21.15.90:0/2263896571 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdbe01a7110 con 0x7fdbe0102430 2024-04-02T14:35:02.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.194+0000 7f9575600640 1 -- 172.21.15.90:0/341079461 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7f956c0046c0 con 0x7f95781018e0 2024-04-02T14:35:02.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.195+0000 7fdbea7fd740 1 Processor -- start 2024-04-02T14:35:02.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.195+0000 7fdbea7fd740 1 -- start start 2024-04-02T14:35:02.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.196+0000 7fdbea7fd740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559b2cf5f030 0x559b2cf5dd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:02.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.196+0000 7fdbea7fd740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559b2cf5e350 0x559b2cf5e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:02.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.196+0000 7fdbea7fd740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559b2cf676c0 0x559b2cf69ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:02.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.196+0000 7fdbea7fd740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x559b2cec2310 con 0x559b2cf5f030 2024-04-02T14:35:02.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.196+0000 7fdbea7fd740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x559b2ce6a690 con 0x559b2cf676c0 2024-04-02T14:35:02.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.196+0000 7fdbea7fd740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x559b2cebe2c0 con 0x559b2cf5e350 2024-04-02T14:35:02.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.196+0000 7f9575600640 1 --2- 172.21.15.90:0/341079461 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f95600a3eb0 0x7f95600a6370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:02.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.196+0000 7fdbe5e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559b2cf5f030 0x559b2cf5dd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:02.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.196+0000 7fdbe5e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559b2cf5f030 0x559b2cf5dd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:53544/0 (socket says 172.21.15.90:53544) 2024-04-02T14:35:02.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.196+0000 7fdbe5e00640 1 -- 172.21.15.90:0/1665329339 learned_addr learned my addr 172.21.15.90:0/1665329339 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:35:02.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.196+0000 7f9575600640 1 -- 172.21.15.90:0/341079461 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f956c0d0df0 con 0x7f95781018e0 2024-04-02T14:35:02.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.196+0000 7fdbe5400640 1 --2- 172.21.15.90:0/1665329339 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559b2cf5e350 0x559b2cf5e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:02.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.196+0000 7f9577e00640 1 --2- 172.21.15.90:0/341079461 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f95600a3eb0 0x7f95600a6370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:02.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.196+0000 7fdbde000640 1 -- 172.21.15.90:0/2263896571 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7fdbcc0037a0 con 0x7fdbe0102430 2024-04-02T14:35:02.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.197+0000 7fdbe5e00640 1 -- 172.21.15.90:0/1665329339 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559b2cf5e350 msgr2=0x559b2cf5e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:35:02.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.197+0000 7fdbe5e00640 1 --2- 172.21.15.90:0/1665329339 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559b2cf5e350 0x559b2cf5e7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:02.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.197+0000 7fdbe5e00640 1 -- 172.21.15.90:0/1665329339 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559b2cf676c0 msgr2=0x559b2cf69ab0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-02T14:35:02.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.197+0000 7fdbe5e00640 1 --2- 172.21.15.90:0/1665329339 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559b2cf676c0 0x559b2cf69ab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:02.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.197+0000 7fdbe5e00640 1 -- 172.21.15.90:0/1665329339 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559b2cc9e7e0 con 0x559b2cf5f030 2024-04-02T14:35:02.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.197+0000 7fdbe5e00640 1 --2- 172.21.15.90:0/1665329339 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559b2cf5f030 0x559b2cf5dd80 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fdbd802c6f0 tx=0x7fdbd8009900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84d674af23b59b63 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:35:02.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.197+0000 7f9577e00640 1 --2- 172.21.15.90:0/341079461 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f95600a3eb0 0x7f95600a6370 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f9564005ab0 tx=0x7f9564005a20 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:35:02.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.197+0000 7fdbd2600640 1 -- 172.21.15.90:0/1665329339 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbd8043070 con 0x559b2cf5f030 2024-04-02T14:35:02.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.197+0000 7fdbd2600640 1 -- 172.21.15.90:0/1665329339 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdbd8032e70 con 0x559b2cf5f030 2024-04-02T14:35:02.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.197+0000 7fdbd2600640 1 -- 172.21.15.90:0/1665329339 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbd8039040 con 0x559b2cf5f030 2024-04-02T14:35:02.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.198+0000 7fdbde000640 1 --2- 172.21.15.90:0/2263896571 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fdbc80a3e40 0x7fdbc80a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:02.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.198+0000 7fdbea7fd740 1 -- 172.21.15.90:0/1665329339 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559b2cf5f030 msgr2=0x559b2cf5dd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:35:02.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.198+0000 7fdbea7fd740 1 --2- 172.21.15.90:0/1665329339 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559b2cf5f030 0x559b2cf5dd80 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fdbd802c6f0 tx=0x7fdbd8009900 comp rx=0 tx=0).stop 2024-04-02T14:35:02.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.198+0000 7fdbea7fd740 1 -- 172.21.15.90:0/1665329339 shutdown_connections 2024-04-02T14:35:02.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.198+0000 7fdbea7fd740 1 --2- 172.21.15.90:0/1665329339 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559b2cf676c0 0x559b2cf69ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:02.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.198+0000 7fdbea7fd740 1 --2- 172.21.15.90:0/1665329339 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559b2cf5e350 0x559b2cf5e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:02.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.198+0000 7fdbea7fd740 1 --2- 172.21.15.90:0/1665329339 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559b2cf5f030 0x559b2cf5dd80 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:02.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.198+0000 7fdbde000640 1 -- 172.21.15.90:0/2263896571 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7fdbcc0d0c90 con 0x7fdbe0102430 2024-04-02T14:35:02.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.198+0000 7fdbea7fd740 1 -- 172.21.15.90:0/1665329339 >> 172.21.15.90:0/1665329339 conn(0x559b2cef2270 msgr2=0x559b2cf58da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:35:02.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.198+0000 7fdbe5400640 1 --2- 172.21.15.90:0/2263896571 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fdbc80a3e40 0x7fdbc80a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:02.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.198+0000 7fdbea7fd740 1 -- 172.21.15.90:0/1665329339 shutdown_connections 2024-04-02T14:35:02.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.199+0000 7fdbea7fd740 1 -- 172.21.15.90:0/1665329339 wait complete. 2024-04-02T14:35:02.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.199+0000 7fdbea7fd740 1 -- start start 2024-04-02T14:35:02.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.199+0000 7fdbe5400640 1 --2- 172.21.15.90:0/2263896571 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fdbc80a3e40 0x7fdbc80a6300 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x559b2cf5f580 tx=0x7fdbd4005f90 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:35:02.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.199+0000 7fdbea7fd740 1 Processor -- start 2024-04-02T14:35:02.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.199+0000 7f9575600640 1 -- 172.21.15.90:0/341079461 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7f956c09d1f0 con 0x7f95781018e0 2024-04-02T14:35:02.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.200+0000 7fdbea7fd740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559b2cf5e350 0x559b2cf646b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:02.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.200+0000 7fdbea7fd740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559b2cf5f030 0x559b2cf64bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:02.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.200+0000 7fdbea7fd740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559b2cf676c0 0x559b2cf65130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:02.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.200+0000 7fdbea7fd740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x559b2cf6a4d0 con 0x559b2cf5f030 2024-04-02T14:35:02.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.200+0000 7fdbea7fd740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x559b2ce7da60 con 0x559b2cf5e350 2024-04-02T14:35:02.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.200+0000 7fdbea7fd740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x559b2cf6a650 con 0x559b2cf676c0 2024-04-02T14:35:02.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.200+0000 7fdbea7fd740 10 client.0 _ll_register_callbacks cb 0x559b2ce83030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:35:02.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.200+0000 7fdbe4a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559b2cf5f030 0x559b2cf64bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:02.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.200+0000 7fdbe4a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559b2cf5f030 0x559b2cf64bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:53546/0 (socket says 172.21.15.90:53546) 2024-04-02T14:35:02.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.200+0000 7fdbe4a00640 1 -- 172.21.15.90:0/3285412966 learned_addr learned my addr 172.21.15.90:0/3285412966 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:35:02.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.200+0000 7fdbe5400640 1 --2- 172.21.15.90:0/3285412966 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559b2cf5e350 0x559b2cf646b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:02.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.200+0000 7fdbe5e00640 1 --2- 172.21.15.90:0/3285412966 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559b2cf676c0 0x559b2cf65130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:02.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.200+0000 7fdbe4a00640 1 -- 172.21.15.90:0/3285412966 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559b2cf676c0 msgr2=0x559b2cf65130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:35:02.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.200+0000 7fdbe4a00640 1 --2- 172.21.15.90:0/3285412966 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559b2cf676c0 0x559b2cf65130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:02.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.200+0000 7fdbe4a00640 1 -- 172.21.15.90:0/3285412966 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559b2cf5e350 msgr2=0x559b2cf646b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:35:02.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.200+0000 7fdbe4a00640 1 --2- 172.21.15.90:0/3285412966 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559b2cf5e350 0x559b2cf646b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:02.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.200+0000 7fdbe4a00640 1 -- 172.21.15.90:0/3285412966 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559b2ccfef70 con 0x559b2cf5f030 2024-04-02T14:35:02.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.201+0000 7fdbe4a00640 1 --2- 172.21.15.90:0/3285412966 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559b2cf5f030 0x559b2cf64bf0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fdbcc00d010 tx=0x7fdbcc09d0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=852536e307da5abc server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:35:02.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.201+0000 7fdbc6a00640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-02T14:35:02.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.201+0000 7fdbc6a00640 1 -- 172.21.15.90:0/3285412966 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbcc0a0da0 con 0x559b2cf5f030 2024-04-02T14:35:02.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.201+0000 7fdbc6a00640 1 -- 172.21.15.90:0/3285412966 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7fdbcc0bbbb0 con 0x559b2cf5f030 2024-04-02T14:35:02.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.201+0000 7fdbea7fd740 1 -- 172.21.15.90:0/3285412966 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x559b2cecf850 con 0x559b2cf5f030 2024-04-02T14:35:02.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.201+0000 7fdbc6a00640 1 -- 172.21.15.90:0/3285412966 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fdbcc0a5660 con 0x559b2cf5f030 2024-04-02T14:35:02.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.201+0000 7fdbea7fd740 10 client.15378 fetch_fsmap learned FSMap version 27 2024-04-02T14:35:02.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.201+0000 7fdbea7fd740 1 -- 172.21.15.90:0/3285412966 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x559b2cb96170 con 0x559b2cf5f030 2024-04-02T14:35:02.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.202+0000 7fdbc6a00640 1 -- 172.21.15.90:0/3285412966 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x559b2cecf850 con 0x559b2cf5f030 2024-04-02T14:35:02.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.202+0000 7fdbc6a00640 1 -- 172.21.15.90:0/3285412966 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fdbcc005450 con 0x559b2cf5f030 2024-04-02T14:35:02.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.202+0000 7fdbea7fd740 10 client.15378 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:35:02.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.202+0000 7fdbea7fd740 10 client.15378 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:35:02.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.202+0000 7fdbea7fd740 20 client.15378 populate_metadata read hostname 'smithi090' 2024-04-02T14:35:02.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.202+0000 7fdbea7fd740 1 --2- 172.21.15.90:0/3285412966 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x559b2cf77b80 0x559b2cf79fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:02.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.202+0000 7fdbea7fd740 4 client.15378 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:35:02.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.202+0000 7fdbea7fd740 1 -- 172.21.15.90:0/3285412966 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559b2cec9ba0 con 0x559b2cf77b80 2024-04-02T14:35:02.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.202+0000 7fdbe5400640 1 --2- 172.21.15.90:0/3285412966 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x559b2cf77b80 0x559b2cf79fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:02.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.203+0000 7fdbe5400640 1 --2- 172.21.15.90:0/3285412966 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x559b2cf77b80 0x559b2cf79fa0 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=aa1befa477b72ebb server_cookie=e3ba6a4fb84d2535 in_seq=0 out_seq=0 2024-04-02T14:35:02.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.203+0000 7fdbc6a00640 10 client.15378 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:35:02.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.204+0000 7fdbc6a00640 1 -- 172.21.15.90:0/3285412966 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x559b2cec9ba0 con 0x559b2cf77b80 2024-04-02T14:35:02.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.204+0000 7fdbc6a00640 10 client.15378 handle_command_reply: tid=0 2024-04-02T14:35:02.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.204+0000 7fdbea7fd740 1 client.15378 shutdown 2024-04-02T14:35:02.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.204+0000 7fdbea7fd740 1 -- 172.21.15.90:0/3285412966 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559b2cf5f030 msgr2=0x559b2cf64bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:35:02.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.204+0000 7fdbea7fd740 1 --2- 172.21.15.90:0/3285412966 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559b2cf5f030 0x559b2cf64bf0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fdbcc00d010 tx=0x7fdbcc09d0f0 comp rx=0 tx=0).stop 2024-04-02T14:35:02.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.204+0000 7fdbea7fd740 1 -- 172.21.15.90:0/3285412966 shutdown_connections 2024-04-02T14:35:02.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.204+0000 7fdbea7fd740 1 --2- 172.21.15.90:0/3285412966 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x559b2cf77b80 0x559b2cf79fa0 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:02.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.204+0000 7fdbea7fd740 1 --2- 172.21.15.90:0/3285412966 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559b2cf676c0 0x559b2cf65130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:02.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.204+0000 7fdbea7fd740 1 --2- 172.21.15.90:0/3285412966 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559b2cf5f030 0x559b2cf64bf0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:02.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.204+0000 7fdbea7fd740 1 --2- 172.21.15.90:0/3285412966 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559b2cf5e350 0x559b2cf646b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:02.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.204+0000 7fdbea7fd740 1 -- 172.21.15.90:0/3285412966 >> 172.21.15.90:0/3285412966 conn(0x559b2cef2270 msgr2=0x559b2cf5aaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:35:02.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.204+0000 7fdbc6a00640 0 client.15378 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:35:02.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.204+0000 7fdbea7fd740 1 -- 172.21.15.90:0/3285412966 shutdown_connections 2024-04-02T14:35:02.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.204+0000 7fdbea7fd740 1 -- 172.21.15.90:0/3285412966 wait complete. 2024-04-02T14:35:02.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.204+0000 7fdbea7fd740 20 client.15378 trim_cache size 0 max 16384 2024-04-02T14:35:02.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.223+0000 7fdbea7fd740 1 Processor -- start 2024-04-02T14:35:02.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.223+0000 7fdbea7fd740 1 -- start start 2024-04-02T14:35:02.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.224+0000 7fdbea7fd740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559b2cf5e350 0x559b2cf66de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:02.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.224+0000 7fdbea7fd740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559b2cf5f030 0x559b2cf58f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:02.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.224+0000 7fdbea7fd740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559b2cf676c0 0x559b2cf73460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:02.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.224+0000 7fdbea7fd740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fdbe01af970 con 0x559b2cf676c0 2024-04-02T14:35:02.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.224+0000 7fdbea7fd740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fdbe01b14b0 con 0x559b2cf5e350 2024-04-02T14:35:02.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.224+0000 7fdbea7fd740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fdbe01b0040 con 0x559b2cf5f030 2024-04-02T14:35:02.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.224+0000 7fdbe5e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559b2cf676c0 0x559b2cf73460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:02.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.224+0000 7fdbe5400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559b2cf5e350 0x559b2cf66de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:02.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.224+0000 7fdbe5400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559b2cf5e350 0x559b2cf66de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:59134/0 (socket says 172.21.15.90:59134) 2024-04-02T14:35:02.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.224+0000 7fdbe5400640 1 -- 172.21.15.90:0/3641592091 learned_addr learned my addr 172.21.15.90:0/3641592091 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:35:02.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.224+0000 7fdbe4a00640 1 --2- 172.21.15.90:0/3641592091 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559b2cf5f030 0x559b2cf58f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:02.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.225+0000 7fdbe4a00640 1 -- 172.21.15.90:0/3641592091 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559b2cf5e350 msgr2=0x559b2cf66de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:35:02.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.225+0000 7fdbe4a00640 1 --2- 172.21.15.90:0/3641592091 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559b2cf5e350 0x559b2cf66de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:02.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.225+0000 7fdbe4a00640 1 -- 172.21.15.90:0/3641592091 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559b2cf676c0 msgr2=0x559b2cf73460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:35:02.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.225+0000 7fdbe4a00640 1 --2- 172.21.15.90:0/3641592091 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559b2cf676c0 0x559b2cf73460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:02.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.225+0000 7fdbe4a00640 1 -- 172.21.15.90:0/3641592091 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559b2cf63660 con 0x559b2cf5f030 2024-04-02T14:35:02.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.225+0000 7fdbe5400640 1 --2- 172.21.15.90:0/3641592091 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559b2cf5e350 0x559b2cf66de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:35:02.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.225+0000 7fdbe4a00640 1 --2- 172.21.15.90:0/3641592091 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559b2cf5f030 0x559b2cf58f40 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fdbcc09c980 tx=0x7fdbcc00ef90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6eaa528425771fea server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:35:02.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.225+0000 7fdbd1c00640 1 -- 172.21.15.90:0/3641592091 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbcc0b7020 con 0x559b2cf5f030 2024-04-02T14:35:02.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.225+0000 7fdbd1c00640 1 -- 172.21.15.90:0/3641592091 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdbcc0ced30 con 0x559b2cf5f030 2024-04-02T14:35:02.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.225+0000 7fdbd1c00640 1 -- 172.21.15.90:0/3641592091 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbcc0bbbb0 con 0x559b2cf5f030 2024-04-02T14:35:02.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.226+0000 7fdbea7fd740 1 -- 172.21.15.90:0/3641592091 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559b2cf5f030 msgr2=0x559b2cf58f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:35:02.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.226+0000 7fdbea7fd740 1 --2- 172.21.15.90:0/3641592091 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559b2cf5f030 0x559b2cf58f40 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fdbcc09c980 tx=0x7fdbcc00ef90 comp rx=0 tx=0).stop 2024-04-02T14:35:02.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.226+0000 7fdbea7fd740 1 -- 172.21.15.90:0/3641592091 shutdown_connections 2024-04-02T14:35:02.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.226+0000 7fdbea7fd740 1 --2- 172.21.15.90:0/3641592091 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559b2cf676c0 0x559b2cf73460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:02.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.226+0000 7fdbea7fd740 1 --2- 172.21.15.90:0/3641592091 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559b2cf5f030 0x559b2cf58f40 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:02.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.226+0000 7fdbea7fd740 1 --2- 172.21.15.90:0/3641592091 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559b2cf5e350 0x559b2cf66de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:02.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.226+0000 7fdbea7fd740 1 -- 172.21.15.90:0/3641592091 >> 172.21.15.90:0/3641592091 conn(0x559b2cef2270 msgr2=0x559b2cf57e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:35:02.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.226+0000 7fdbea7fd740 1 -- 172.21.15.90:0/3641592091 shutdown_connections 2024-04-02T14:35:02.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.227+0000 7fdbea7fd740 1 -- 172.21.15.90:0/3641592091 wait complete. 2024-04-02T14:35:02.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.227+0000 7fdbea7fd740 1 -- start start 2024-04-02T14:35:02.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.227+0000 7fdbea7fd740 1 Processor -- start 2024-04-02T14:35:02.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.228+0000 7fdbea7fd740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559b2cf5e350 0x559b2cf6f970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:02.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.228+0000 7fdbea7fd740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559b2cf5f030 0x559b2cf6feb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:02.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.228+0000 7fdbea7fd740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559b2cf676c0 0x559b2cf60980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:02.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.228+0000 7fdbea7fd740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fdbe01b0de0 con 0x559b2cf676c0 2024-04-02T14:35:02.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.228+0000 7fdbea7fd740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fdbe01b0710 con 0x559b2cf5e350 2024-04-02T14:35:02.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.228+0000 7fdbea7fd740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fdbe01ad950 con 0x559b2cf5f030 2024-04-02T14:35:02.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.228+0000 7fdbea7fd740 10 client.0 _ll_register_callbacks cb 0x7fdbcc003e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:35:02.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.228+0000 7fdbe5400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559b2cf5e350 0x559b2cf6f970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:02.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.228+0000 7fdbe5e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559b2cf676c0 0x559b2cf60980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:02.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.228+0000 7fdbe5400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559b2cf5e350 0x559b2cf6f970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:59148/0 (socket says 172.21.15.90:59148) 2024-04-02T14:35:02.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.228+0000 7fdbe5400640 1 -- 172.21.15.90:0/4104445170 learned_addr learned my addr 172.21.15.90:0/4104445170 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:35:02.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.228+0000 7fdbe4a00640 1 --2- 172.21.15.90:0/4104445170 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559b2cf5f030 0x559b2cf6feb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:02.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.229+0000 7fdbe5400640 1 -- 172.21.15.90:0/4104445170 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559b2cf5f030 msgr2=0x559b2cf6feb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:35:02.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.229+0000 7fdbe5400640 1 --2- 172.21.15.90:0/4104445170 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559b2cf5f030 0x559b2cf6feb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:02.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.229+0000 7fdbe5400640 1 -- 172.21.15.90:0/4104445170 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559b2cf676c0 msgr2=0x559b2cf60980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:35:02.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.229+0000 7fdbe5400640 1 --2- 172.21.15.90:0/4104445170 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559b2cf676c0 0x559b2cf60980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:02.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.229+0000 7fdbe5400640 1 -- 172.21.15.90:0/4104445170 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559b2cf611b0 con 0x559b2cf5e350 2024-04-02T14:35:02.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.229+0000 7fdbe5400640 1 --2- 172.21.15.90:0/4104445170 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559b2cf5e350 0x559b2cf6f970 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fdbd4009fc0 tx=0x7fdbd40046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=711a1160086ee202 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:35:02.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.229+0000 7fdbc6a00640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:35:02.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.229+0000 7fdbc6a00640 1 -- 172.21.15.90:0/4104445170 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbd402f030 con 0x559b2cf5e350 2024-04-02T14:35:02.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.229+0000 7fdbea7fd740 1 -- 172.21.15.90:0/4104445170 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdbcc0ced30 con 0x559b2cf5e350 2024-04-02T14:35:02.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.229+0000 7fdbc6a00640 1 -- 172.21.15.90:0/4104445170 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7fdbd4018030 con 0x559b2cf5e350 2024-04-02T14:35:02.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.230+0000 7fdbc6a00640 1 -- 172.21.15.90:0/4104445170 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fdbd402b410 con 0x559b2cf5e350 2024-04-02T14:35:02.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.230+0000 7fdbea7fd740 10 client.25496 fetch_fsmap learned FSMap version 27 2024-04-02T14:35:02.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.230+0000 7fdbea7fd740 1 -- 172.21.15.90:0/4104445170 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x559b2cf62550 con 0x559b2cf5e350 2024-04-02T14:35:02.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.230+0000 7fdbc6a00640 1 -- 172.21.15.90:0/4104445170 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdbcc0ced30 con 0x559b2cf5e350 2024-04-02T14:35:02.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.230+0000 7fdbc6a00640 1 -- 172.21.15.90:0/4104445170 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fdbd403d590 con 0x559b2cf5e350 2024-04-02T14:35:02.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.230+0000 7fdbea7fd740 10 client.25496 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:35:02.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.230+0000 7fdbea7fd740 10 client.25496 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:35:02.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.230+0000 7fdbea7fd740 20 client.25496 populate_metadata read hostname 'smithi090' 2024-04-02T14:35:02.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.230+0000 7fdbea7fd740 1 --2- 172.21.15.90:0/4104445170 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x559b2cfdf490 0x559b2cfe18b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:02.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.230+0000 7fdbea7fd740 4 client.25496 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-02T14:35:02.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.230+0000 7fdbea7fd740 1 -- 172.21.15.90:0/4104445170 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x559b2ce47aa0 con 0x559b2cfdf490 2024-04-02T14:35:02.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.230+0000 7fdbe4a00640 1 --2- 172.21.15.90:0/4104445170 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x559b2cfdf490 0x559b2cfe18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:02.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.231+0000 7fdbe4a00640 1 --2- 172.21.15.90:0/4104445170 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x559b2cfdf490 0x559b2cfe18b0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=965590bc3dceb1bd server_cookie=d74ad5d0070eab73 in_seq=0 out_seq=0 2024-04-02T14:35:02.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.231+0000 7fdbc6a00640 10 client.25496 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:35:02.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.232+0000 7fdbc6a00640 1 -- 172.21.15.90:0/4104445170 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x559b2ce47aa0 con 0x559b2cfdf490 2024-04-02T14:35:02.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.232+0000 7fdbc6a00640 10 client.25496 handle_command_reply: tid=0 2024-04-02T14:35:02.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.232+0000 7fdbea7fd740 1 client.25496 shutdown 2024-04-02T14:35:02.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.232+0000 7fdbea7fd740 1 -- 172.21.15.90:0/4104445170 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559b2cf5e350 msgr2=0x559b2cf6f970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:35:02.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.232+0000 7fdbea7fd740 1 --2- 172.21.15.90:0/4104445170 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559b2cf5e350 0x559b2cf6f970 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fdbd4009fc0 tx=0x7fdbd40046c0 comp rx=0 tx=0).stop 2024-04-02T14:35:02.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.232+0000 7fdbea7fd740 1 -- 172.21.15.90:0/4104445170 shutdown_connections 2024-04-02T14:35:02.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.232+0000 7fdbea7fd740 1 --2- 172.21.15.90:0/4104445170 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x559b2cfdf490 0x559b2cfe18b0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:02.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.232+0000 7fdbea7fd740 1 --2- 172.21.15.90:0/4104445170 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559b2cf676c0 0x559b2cf60980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:02.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.232+0000 7fdbea7fd740 1 --2- 172.21.15.90:0/4104445170 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559b2cf5f030 0x559b2cf6feb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:02.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.232+0000 7fdbea7fd740 1 --2- 172.21.15.90:0/4104445170 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559b2cf5e350 0x559b2cf6f970 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:02.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.232+0000 7fdbc6a00640 0 client.25496 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:35:02.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.232+0000 7fdbea7fd740 1 -- 172.21.15.90:0/4104445170 >> 172.21.15.90:0/4104445170 conn(0x559b2cef2270 msgr2=0x559b2cf645d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:35:02.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.232+0000 7fdbea7fd740 1 -- 172.21.15.90:0/4104445170 shutdown_connections 2024-04-02T14:35:02.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.232+0000 7fdbea7fd740 1 -- 172.21.15.90:0/4104445170 wait complete. 2024-04-02T14:35:02.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.233+0000 7fdbea7fd740 20 client.25496 trim_cache size 0 max 16384 2024-04-02T14:35:02.236 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-02T14:35:02.236 INFO:teuthology.orchestra.run.smithi090.stdout: "return_code": 0, 2024-04-02T14:35:02.236 INFO:teuthology.orchestra.run.smithi090.stdout: "scrub_tag": "2dc2f89c-fc11-4753-8afc-05ebdfa775f1", 2024-04-02T14:35:02.236 INFO:teuthology.orchestra.run.smithi090.stdout: "mode": "asynchronous" 2024-04-02T14:35:02.236 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-02T14:35:02.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.233+0000 7fdbe7e00640 1 -- 172.21.15.90:0/2263896571 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fdbc80a3e40 msgr2=0x7fdbc80a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:35:02.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.233+0000 7fdbe7e00640 1 --2- 172.21.15.90:0/2263896571 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fdbc80a3e40 0x7fdbc80a6300 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x559b2cf5f580 tx=0x7fdbd4005f90 comp rx=0 tx=0).stop 2024-04-02T14:35:02.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.233+0000 7fdbe7e00640 1 -- 172.21.15.90:0/2263896571 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdbe0102430 msgr2=0x7fdbe0074340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:35:02.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.233+0000 7fdbe7e00640 1 --2- 172.21.15.90:0/2263896571 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdbe0102430 0x7fdbe0074340 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fdbcc00ea60 tx=0x7fdbcc00ef20 comp rx=0 tx=0).stop 2024-04-02T14:35:02.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.233+0000 7fdbe7e00640 1 -- 172.21.15.90:0/2263896571 shutdown_connections 2024-04-02T14:35:02.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.233+0000 7fdbe7e00640 1 --2- 172.21.15.90:0/2263896571 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fdbc80a3e40 0x7fdbc80a6300 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:02.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.233+0000 7fdbe7e00640 1 --2- 172.21.15.90:0/2263896571 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdbe0107a80 0x7fdbe0074880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:02.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.233+0000 7fdbe7e00640 1 --2- 172.21.15.90:0/2263896571 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdbe0102430 0x7fdbe0074340 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:02.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.233+0000 7fdbe7e00640 1 --2- 172.21.15.90:0/2263896571 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdbe0101a70 0x7fdbe0073e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:02.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.233+0000 7fdbe7e00640 1 -- 172.21.15.90:0/2263896571 >> 172.21.15.90:0/2263896571 conn(0x7fdbe00fd560 msgr2=0x7fdbe00fdd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:35:02.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.234+0000 7fdbe7e00640 1 -- 172.21.15.90:0/2263896571 shutdown_connections 2024-04-02T14:35:02.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.234+0000 7fdbe7e00640 1 -- 172.21.15.90:0/2263896571 wait complete. 2024-04-02T14:35:02.250 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:35:02 smithi090 ceph-mon[31674]: pgmap v7: 97 pgs: 97 active+clean; 1.1 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 1.4 KiB/s wr, 26 op/s 2024-04-02T14:35:02.250 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:35:02 smithi090 ceph-mon[31674]: from='client.? 172.21.15.90:0/3531898634' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-02T14:35:02.250 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:35:02 smithi090 ceph-mon[31674]: from='client.? 172.21.15.90:0/1948793901' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-02T14:35:02.250 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:35:02 smithi090 ceph-mon[31674]: from='client.? 172.21.15.90:0/2509016358' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-02T14:35:02.252 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-02T14:35:02.253 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-02T14:35:02.360 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:35:02 smithi092 ceph-mon[36497]: pgmap v7: 97 pgs: 97 active+clean; 1.1 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 1.4 KiB/s wr, 26 op/s 2024-04-02T14:35:02.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:35:02 smithi092 ceph-mon[36497]: from='client.? 172.21.15.90:0/3531898634' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-02T14:35:02.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:35:02 smithi092 ceph-mon[36497]: from='client.? 172.21.15.90:0/1948793901' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-02T14:35:02.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:35:02 smithi092 ceph-mon[36497]: from='client.? 172.21.15.90:0/2509016358' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-02T14:35:02.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.433+0000 7f957e000640 1 -- 172.21.15.90:0/341079461 --> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) -- 0x7f95780fe990 con 0x7f95600a3eb0 2024-04-02T14:35:02.436 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:35:02 smithi177 ceph-mon[36431]: pgmap v7: 97 pgs: 97 active+clean; 1.1 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 1.4 KiB/s wr, 26 op/s 2024-04-02T14:35:02.436 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:35:02 smithi177 ceph-mon[36431]: from='client.? 172.21.15.90:0/3531898634' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-02T14:35:02.436 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:35:02 smithi177 ceph-mon[36431]: from='client.? 172.21.15.90:0/1948793901' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-02T14:35:02.436 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:35:02 smithi177 ceph-mon[36431]: from='client.? 172.21.15.90:0/2509016358' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-02T14:35:02.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.451+0000 7ff17ae00640 1 -- 172.21.15.90:0/1358686985 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff17410e210 msgr2=0x7ff174110620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:35:02.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.451+0000 7ff17ae00640 1 --2- 172.21.15.90:0/1358686985 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff17410e210 0x7ff174110620 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7ff16c009ca0 tx=0x7ff16c02c180 comp rx=0 tx=0).stop 2024-04-02T14:35:02.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.452+0000 7ff17ae00640 1 -- 172.21.15.90:0/1358686985 shutdown_connections 2024-04-02T14:35:02.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.452+0000 7ff17ae00640 1 --2- 172.21.15.90:0/1358686985 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff17410e210 0x7ff174110620 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:02.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.452+0000 7ff17ae00640 1 --2- 172.21.15.90:0/1358686985 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff1740747f0 0x7ff17410dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:02.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.452+0000 7ff17ae00640 1 --2- 172.21.15.90:0/1358686985 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff174073ed0 0x7ff1740742b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:02.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.452+0000 7ff17ae00640 1 -- 172.21.15.90:0/1358686985 >> 172.21.15.90:0/1358686985 conn(0x7ff1740fd4a0 msgr2=0x7ff1740ff8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:35:02.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.452+0000 7ff17ae00640 1 -- 172.21.15.90:0/1358686985 shutdown_connections 2024-04-02T14:35:02.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.452+0000 7ff17ae00640 1 -- 172.21.15.90:0/1358686985 wait complete. 2024-04-02T14:35:02.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.453+0000 7ff17ae00640 1 Processor -- start 2024-04-02T14:35:02.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.453+0000 7ff17ae00640 1 -- start start 2024-04-02T14:35:02.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.454+0000 7ff17ae00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff174073ed0 0x7ff17410db90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:02.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.454+0000 7ff17ae00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff1740747f0 0x7ff17410ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:02.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.454+0000 7ff17ae00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff17410e210 0x7ff17410b190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:02.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.454+0000 7ff17ae00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7ff174113f70 con 0x7ff17410e210 2024-04-02T14:35:02.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.454+0000 7ff17ae00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7ff174113df0 con 0x7ff1740747f0 2024-04-02T14:35:02.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.454+0000 7ff17ae00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7ff1741140f0 con 0x7ff174073ed0 2024-04-02T14:35:02.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.454+0000 7ff179000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff1740747f0 0x7ff17410ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:02.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.454+0000 7ff179a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff174073ed0 0x7ff17410db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:02.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.454+0000 7ff179000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff1740747f0 0x7ff17410ac30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:59166/0 (socket says 172.21.15.90:59166) 2024-04-02T14:35:02.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.454+0000 7ff179000640 1 -- 172.21.15.90:0/1088924657 learned_addr learned my addr 172.21.15.90:0/1088924657 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:35:02.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.454+0000 7ff17a400640 1 --2- 172.21.15.90:0/1088924657 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff17410e210 0x7ff17410b190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:02.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.455+0000 7ff179000640 1 -- 172.21.15.90:0/1088924657 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff174073ed0 msgr2=0x7ff17410db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:35:02.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.455+0000 7ff179000640 1 --2- 172.21.15.90:0/1088924657 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff174073ed0 0x7ff17410db90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:02.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.455+0000 7ff179000640 1 -- 172.21.15.90:0/1088924657 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff17410e210 msgr2=0x7ff17410b190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:35:02.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.455+0000 7ff179000640 1 --2- 172.21.15.90:0/1088924657 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff17410e210 0x7ff17410b190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:02.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.455+0000 7ff179000640 1 -- 172.21.15.90:0/1088924657 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff17410ba50 con 0x7ff1740747f0 2024-04-02T14:35:02.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.455+0000 7ff179000640 1 --2- 172.21.15.90:0/1088924657 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff1740747f0 0x7ff17410ac30 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7ff16000d950 tx=0x7ff16000de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:35:02.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.455+0000 7ff172000640 1 -- 172.21.15.90:0/1088924657 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff160017030 con 0x7ff1740747f0 2024-04-02T14:35:02.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.455+0000 7ff172000640 1 -- 172.21.15.90:0/1088924657 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff160004020 con 0x7ff1740747f0 2024-04-02T14:35:02.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.455+0000 7ff172000640 1 -- 172.21.15.90:0/1088924657 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1600047a0 con 0x7ff1740747f0 2024-04-02T14:35:02.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.455+0000 7ff17ae00640 1 -- 172.21.15.90:0/1088924657 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1741aad90 con 0x7ff1740747f0 2024-04-02T14:35:02.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.455+0000 7ff17ae00640 1 -- 172.21.15.90:0/1088924657 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1741ab260 con 0x7ff1740747f0 2024-04-02T14:35:02.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.459+0000 7ff172000640 1 -- 172.21.15.90:0/1088924657 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7ff1600041d0 con 0x7ff1740747f0 2024-04-02T14:35:02.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.460+0000 7ff17f215740 1 Processor -- start 2024-04-02T14:35:02.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.460+0000 7ff17f215740 1 -- start start 2024-04-02T14:35:02.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.461+0000 7ff17f215740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561b0b7fe960 0x561b0b7fed40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:02.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.461+0000 7ff17f215740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561b0b7ff280 0x561b0b807920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:02.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.461+0000 7ff17f215740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561b0b807e60 0x561b0b80a250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:02.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.461+0000 7ff17f215740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x561b0b762770 con 0x561b0b807e60 2024-04-02T14:35:02.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.461+0000 7ff17f215740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x561b0b70a700 con 0x561b0b7fe960 2024-04-02T14:35:02.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.461+0000 7ff17f215740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x561b0b75e720 con 0x561b0b7ff280 2024-04-02T14:35:02.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.461+0000 7ff179a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561b0b7ff280 0x561b0b807920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:02.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.461+0000 7ff179a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561b0b7ff280 0x561b0b807920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:49718/0 (socket says 172.21.15.90:49718) 2024-04-02T14:35:02.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.461+0000 7ff179a00640 1 -- 172.21.15.90:0/3288244988 learned_addr learned my addr 172.21.15.90:0/3288244988 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:35:02.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.461+0000 7ff17a400640 1 --2- 172.21.15.90:0/3288244988 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561b0b7fe960 0x561b0b7fed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:02.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.461+0000 7ff179000640 1 --2- 172.21.15.90:0/3288244988 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561b0b807e60 0x561b0b80a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:02.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.462+0000 7ff172000640 1 --2- 172.21.15.90:0/1088924657 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7ff15c0a3ec0 0x7ff15c0a6380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:02.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.462+0000 7ff179a00640 1 -- 172.21.15.90:0/3288244988 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561b0b7fe960 msgr2=0x561b0b7fed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:35:02.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.462+0000 7ff179a00640 1 --2- 172.21.15.90:0/3288244988 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561b0b7fe960 0x561b0b7fed40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:02.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.462+0000 7ff179a00640 1 -- 172.21.15.90:0/3288244988 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561b0b807e60 msgr2=0x561b0b80a250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:35:02.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.462+0000 7ff179a00640 1 --2- 172.21.15.90:0/3288244988 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561b0b807e60 0x561b0b80a250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:02.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.462+0000 7ff179a00640 1 -- 172.21.15.90:0/3288244988 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561b0b5407d0 con 0x561b0b7ff280 2024-04-02T14:35:02.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.462+0000 7ff179000640 1 --2- 172.21.15.90:0/3288244988 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561b0b807e60 0x561b0b80a250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:35:02.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.462+0000 7ff17a400640 1 --2- 172.21.15.90:0/3288244988 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561b0b7fe960 0x561b0b7fed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:35:02.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.462+0000 7ff172000640 1 -- 172.21.15.90:0/1088924657 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7ff1600d0400 con 0x7ff1740747f0 2024-04-02T14:35:02.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.462+0000 7ff17a400640 1 --2- 172.21.15.90:0/1088924657 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7ff15c0a3ec0 0x7ff15c0a6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:02.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.462+0000 7ff179a00640 1 --2- 172.21.15.90:0/3288244988 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561b0b7ff280 0x561b0b807920 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7ff16800bcd0 tx=0x7ff1680107a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c031c9278856b18d server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:35:02.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.462+0000 7ff166a00640 1 -- 172.21.15.90:0/3288244988 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff168005220 con 0x561b0b7ff280 2024-04-02T14:35:02.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.462+0000 7ff166a00640 1 -- 172.21.15.90:0/3288244988 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff168004c80 con 0x561b0b7ff280 2024-04-02T14:35:02.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.462+0000 7ff166a00640 1 -- 172.21.15.90:0/3288244988 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1680117c0 con 0x561b0b7ff280 2024-04-02T14:35:02.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.464+0000 7ff17f215740 1 -- 172.21.15.90:0/3288244988 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561b0b7ff280 msgr2=0x561b0b807920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:35:02.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.464+0000 7ff17f215740 1 --2- 172.21.15.90:0/3288244988 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561b0b7ff280 0x561b0b807920 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7ff16800bcd0 tx=0x7ff1680107a0 comp rx=0 tx=0).stop 2024-04-02T14:35:02.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.464+0000 7ff17a400640 1 --2- 172.21.15.90:0/1088924657 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7ff15c0a3ec0 0x7ff15c0a6380 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7ff16c009bf0 tx=0x7ff16c02e040 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:35:02.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.464+0000 7ff17f215740 1 -- 172.21.15.90:0/3288244988 shutdown_connections 2024-04-02T14:35:02.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.464+0000 7ff17f215740 1 --2- 172.21.15.90:0/3288244988 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561b0b807e60 0x561b0b80a250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:02.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.464+0000 7ff17f215740 1 --2- 172.21.15.90:0/3288244988 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561b0b7ff280 0x561b0b807920 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:02.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.464+0000 7ff17f215740 1 --2- 172.21.15.90:0/3288244988 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561b0b7fe960 0x561b0b7fed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:02.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.464+0000 7ff17f215740 1 -- 172.21.15.90:0/3288244988 >> 172.21.15.90:0/3288244988 conn(0x561b0b7f9090 msgr2=0x561b0b7f9470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:35:02.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.464+0000 7ff17f215740 1 -- 172.21.15.90:0/3288244988 shutdown_connections 2024-04-02T14:35:02.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.464+0000 7ff17f215740 1 -- 172.21.15.90:0/3288244988 wait complete. 2024-04-02T14:35:02.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.464+0000 7ff17f215740 1 -- start start 2024-04-02T14:35:02.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.464+0000 7ff17f215740 1 Processor -- start 2024-04-02T14:35:02.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.465+0000 7ff17f215740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561b0b7fe960 0x561b0b804ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:02.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.465+0000 7ff17f215740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561b0b7ff280 0x561b0b805400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:02.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.465+0000 7ff17f215740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561b0b807e60 0x561b0b805940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:02.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.465+0000 7ff17f215740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x561b0b80ac50 con 0x561b0b7ff280 2024-04-02T14:35:02.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.465+0000 7ff17f215740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x561b0b71dac0 con 0x561b0b7fe960 2024-04-02T14:35:02.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.465+0000 7ff17f215740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x561b0b80add0 con 0x561b0b807e60 2024-04-02T14:35:02.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.465+0000 7ff17a400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561b0b807e60 0x561b0b805940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:02.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.466+0000 7ff17a400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561b0b807e60 0x561b0b805940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:49732/0 (socket says 172.21.15.90:49732) 2024-04-02T14:35:02.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.466+0000 7ff17f215740 10 client.0 _ll_register_callbacks cb 0x561b0b720180 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:35:02.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.466+0000 7ff17a400640 1 -- 172.21.15.90:0/3321468514 learned_addr learned my addr 172.21.15.90:0/3321468514 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:35:02.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.466+0000 7ff179000640 1 --2- 172.21.15.90:0/3321468514 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561b0b7ff280 0x561b0b805400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:02.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.466+0000 7ff179a00640 1 --2- 172.21.15.90:0/3321468514 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561b0b7fe960 0x561b0b804ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:02.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.466+0000 7ff17a400640 1 -- 172.21.15.90:0/3321468514 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561b0b7fe960 msgr2=0x561b0b804ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:35:02.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.466+0000 7ff17a400640 1 --2- 172.21.15.90:0/3321468514 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561b0b7fe960 0x561b0b804ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:02.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.466+0000 7ff17a400640 1 -- 172.21.15.90:0/3321468514 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561b0b7ff280 msgr2=0x561b0b805400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:35:02.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.466+0000 7ff17a400640 1 --2- 172.21.15.90:0/3321468514 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561b0b7ff280 0x561b0b805400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:02.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.466+0000 7ff17a400640 1 -- 172.21.15.90:0/3321468514 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561b0b4763f0 con 0x561b0b807e60 2024-04-02T14:35:02.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.466+0000 7ff17a400640 1 --2- 172.21.15.90:0/3321468514 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561b0b807e60 0x561b0b805940 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7ff16c009ca0 tx=0x7ff16c004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=20c3fabb41f4cff6 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:35:02.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.467+0000 7ff15b400640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:35:02.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.467+0000 7ff15b400640 1 -- 172.21.15.90:0/3321468514 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff16c051060 con 0x561b0b807e60 2024-04-02T14:35:02.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.467+0000 7ff15b400640 1 -- 172.21.15.90:0/3321468514 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7ff16c034910 con 0x561b0b807e60 2024-04-02T14:35:02.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.467+0000 7ff17f215740 1 -- 172.21.15.90:0/3321468514 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff168004c80 con 0x561b0b807e60 2024-04-02T14:35:02.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.467+0000 7ff15b400640 1 -- 172.21.15.90:0/3321468514 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7ff16c045070 con 0x561b0b807e60 2024-04-02T14:35:02.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.467+0000 7ff17f215740 10 client.25477 fetch_fsmap learned FSMap version 27 2024-04-02T14:35:02.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.467+0000 7ff17f215740 1 -- 172.21.15.90:0/3321468514 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x561b0b4360f0 con 0x561b0b807e60 2024-04-02T14:35:02.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.468+0000 7ff15b400640 1 -- 172.21.15.90:0/3321468514 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff168004c80 con 0x561b0b807e60 2024-04-02T14:35:02.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.468+0000 7ff15b400640 1 -- 172.21.15.90:0/3321468514 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7ff16c03f040 con 0x561b0b807e60 2024-04-02T14:35:02.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.468+0000 7ff17f215740 10 client.25477 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:35:02.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.468+0000 7ff17f215740 10 client.25477 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:35:02.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.468+0000 7ff17f215740 20 client.25477 populate_metadata read hostname 'smithi090' 2024-04-02T14:35:02.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.468+0000 7ff17f215740 1 --2- 172.21.15.90:0/3321468514 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x561b0b817e80 0x561b0b81a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:02.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.468+0000 7ff17f215740 4 client.25477 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:35:02.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.468+0000 7ff17f215740 1 -- 172.21.15.90:0/3321468514 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561b0b703630 con 0x561b0b817e80 2024-04-02T14:35:02.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.468+0000 7ff179a00640 1 --2- 172.21.15.90:0/3321468514 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x561b0b817e80 0x561b0b81a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:02.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.469+0000 7ff179a00640 1 --2- 172.21.15.90:0/3321468514 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x561b0b817e80 0x561b0b81a2d0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cb553e95e5ae3a6b server_cookie=640a6da2411f45b0 in_seq=0 out_seq=0 2024-04-02T14:35:02.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.469+0000 7ff15b400640 10 client.25477 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:35:02.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.470+0000 7ff15b400640 1 -- 172.21.15.90:0/3321468514 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x561b0b703630 con 0x561b0b817e80 2024-04-02T14:35:02.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.470+0000 7ff15b400640 10 client.25477 handle_command_reply: tid=0 2024-04-02T14:35:02.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.470+0000 7ff17f215740 1 client.25477 shutdown 2024-04-02T14:35:02.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.470+0000 7ff17f215740 1 -- 172.21.15.90:0/3321468514 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561b0b807e60 msgr2=0x561b0b805940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:35:02.473 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.470+0000 7ff17f215740 1 --2- 172.21.15.90:0/3321468514 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561b0b807e60 0x561b0b805940 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7ff16c009ca0 tx=0x7ff16c004a40 comp rx=0 tx=0).stop 2024-04-02T14:35:02.473 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.470+0000 7ff17f215740 1 -- 172.21.15.90:0/3321468514 shutdown_connections 2024-04-02T14:35:02.473 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.470+0000 7ff17f215740 1 --2- 172.21.15.90:0/3321468514 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x561b0b817e80 0x561b0b81a2d0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:02.473 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.470+0000 7ff17f215740 1 --2- 172.21.15.90:0/3321468514 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561b0b807e60 0x561b0b805940 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:02.473 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.470+0000 7ff17f215740 1 --2- 172.21.15.90:0/3321468514 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561b0b7ff280 0x561b0b805400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:02.473 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.470+0000 7ff15b400640 0 client.25477 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:35:02.473 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.470+0000 7ff17f215740 1 --2- 172.21.15.90:0/3321468514 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561b0b7fe960 0x561b0b804ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:02.473 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.470+0000 7ff17f215740 1 -- 172.21.15.90:0/3321468514 >> 172.21.15.90:0/3321468514 conn(0x561b0b7f9090 msgr2=0x561b0b7fade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:35:02.473 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.470+0000 7ff17f215740 1 -- 172.21.15.90:0/3321468514 shutdown_connections 2024-04-02T14:35:02.473 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.470+0000 7ff17f215740 1 -- 172.21.15.90:0/3321468514 wait complete. 2024-04-02T14:35:02.473 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.470+0000 7ff17f215740 20 client.25477 trim_cache size 0 max 16384 2024-04-02T14:35:02.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.493+0000 7ff17f215740 1 Processor -- start 2024-04-02T14:35:02.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.493+0000 7ff17f215740 1 -- start start 2024-04-02T14:35:02.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.493+0000 7ff17f215740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561b0b7fe960 0x561b0b8075f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:02.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.493+0000 7ff17f215740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561b0b7ff280 0x561b0b813b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:02.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.493+0000 7ff17f215740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561b0b807e60 0x561b0b81fba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:02.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.493+0000 7ff17f215740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7ff1741b3cb0 con 0x561b0b807e60 2024-04-02T14:35:02.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.493+0000 7ff17f215740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7ff1741b57f0 con 0x561b0b7ff280 2024-04-02T14:35:02.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.493+0000 7ff17f215740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7ff1741b4380 con 0x561b0b7fe960 2024-04-02T14:35:02.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.493+0000 7ff179a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561b0b7fe960 0x561b0b8075f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:02.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.493+0000 7ff179a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561b0b7fe960 0x561b0b8075f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:49734/0 (socket says 172.21.15.90:49734) 2024-04-02T14:35:02.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.493+0000 7ff179a00640 1 -- 172.21.15.90:0/2990136863 learned_addr learned my addr 172.21.15.90:0/2990136863 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:35:02.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.493+0000 7ff179000640 1 --2- 172.21.15.90:0/2990136863 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561b0b7ff280 0x561b0b813b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:02.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.494+0000 7ff17a400640 1 --2- 172.21.15.90:0/2990136863 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561b0b807e60 0x561b0b81fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:02.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.494+0000 7ff179000640 1 -- 172.21.15.90:0/2990136863 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561b0b7fe960 msgr2=0x561b0b8075f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:35:02.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.494+0000 7ff179000640 1 --2- 172.21.15.90:0/2990136863 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561b0b7fe960 0x561b0b8075f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:02.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.494+0000 7ff179000640 1 -- 172.21.15.90:0/2990136863 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561b0b807e60 msgr2=0x561b0b81fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:35:02.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.494+0000 7ff179000640 1 --2- 172.21.15.90:0/2990136863 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561b0b807e60 0x561b0b81fba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:02.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.494+0000 7ff179000640 1 -- 172.21.15.90:0/2990136863 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561b0b80a250 con 0x561b0b7ff280 2024-04-02T14:35:02.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.494+0000 7ff17a400640 1 --2- 172.21.15.90:0/2990136863 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561b0b807e60 0x561b0b81fba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:35:02.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.494+0000 7ff179a00640 1 --2- 172.21.15.90:0/2990136863 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561b0b7fe960 0x561b0b8075f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:35:02.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.494+0000 7ff179000640 1 --2- 172.21.15.90:0/2990136863 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561b0b7ff280 0x561b0b813b10 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7ff1600cd980 tx=0x7ff160098460 comp rx=0 tx=0).ready entity=mon.2 client_cookie=219a9113ceed5b8 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:35:02.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.494+0000 7ff166000640 1 -- 172.21.15.90:0/2990136863 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff16009c800 con 0x561b0b7ff280 2024-04-02T14:35:02.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.494+0000 7ff166000640 1 -- 172.21.15.90:0/2990136863 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff16009c100 con 0x561b0b7ff280 2024-04-02T14:35:02.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.494+0000 7ff166000640 1 -- 172.21.15.90:0/2990136863 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1600d3de0 con 0x561b0b7ff280 2024-04-02T14:35:02.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.495+0000 7ff17f215740 1 -- 172.21.15.90:0/2990136863 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561b0b7ff280 msgr2=0x561b0b813b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:35:02.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.495+0000 7ff17f215740 1 --2- 172.21.15.90:0/2990136863 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561b0b7ff280 0x561b0b813b10 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7ff1600cd980 tx=0x7ff160098460 comp rx=0 tx=0).stop 2024-04-02T14:35:02.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.495+0000 7ff17f215740 1 -- 172.21.15.90:0/2990136863 shutdown_connections 2024-04-02T14:35:02.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.495+0000 7ff17f215740 1 --2- 172.21.15.90:0/2990136863 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561b0b807e60 0x561b0b81fba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:02.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.495+0000 7ff17f215740 1 --2- 172.21.15.90:0/2990136863 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561b0b7ff280 0x561b0b813b10 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:02.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.495+0000 7ff17f215740 1 --2- 172.21.15.90:0/2990136863 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561b0b7fe960 0x561b0b8075f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:02.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.495+0000 7ff17f215740 1 -- 172.21.15.90:0/2990136863 >> 172.21.15.90:0/2990136863 conn(0x561b0b7f9090 msgr2=0x561b0b7f8310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:35:02.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.495+0000 7ff17f215740 1 -- 172.21.15.90:0/2990136863 shutdown_connections 2024-04-02T14:35:02.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.495+0000 7ff17f215740 1 -- 172.21.15.90:0/2990136863 wait complete. 2024-04-02T14:35:02.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.496+0000 7ff17f215740 1 -- start start 2024-04-02T14:35:02.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.496+0000 7ff17f215740 1 Processor -- start 2024-04-02T14:35:02.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.496+0000 7ff17f215740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561b0b7fe960 0x561b0b80fc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:02.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.496+0000 7ff17f215740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561b0b7ff280 0x561b0b810190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:02.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.496+0000 7ff17f215740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561b0b807e60 0x561b0b80c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:02.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.496+0000 7ff17f215740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7ff1741b5120 con 0x561b0b7ff280 2024-04-02T14:35:02.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.496+0000 7ff17f215740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7ff1741b4a50 con 0x561b0b807e60 2024-04-02T14:35:02.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.496+0000 7ff17f215740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7ff1741b1c90 con 0x561b0b7fe960 2024-04-02T14:35:02.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.496+0000 7ff17f215740 10 client.0 _ll_register_callbacks cb 0x7ff16c009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:35:02.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.497+0000 7ff179a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561b0b7fe960 0x561b0b80fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:02.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.497+0000 7ff179000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561b0b7ff280 0x561b0b810190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:02.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.497+0000 7ff17a400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561b0b807e60 0x561b0b80c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:02.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.497+0000 7ff179a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561b0b7fe960 0x561b0b80fc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:49742/0 (socket says 172.21.15.90:49742) 2024-04-02T14:35:02.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.497+0000 7ff179a00640 1 -- 172.21.15.90:0/3665963035 learned_addr learned my addr 172.21.15.90:0/3665963035 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:35:02.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.497+0000 7ff179a00640 1 -- 172.21.15.90:0/3665963035 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561b0b807e60 msgr2=0x561b0b80c1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:35:02.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.497+0000 7ff179a00640 1 --2- 172.21.15.90:0/3665963035 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561b0b807e60 0x561b0b80c1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:02.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.497+0000 7ff179a00640 1 -- 172.21.15.90:0/3665963035 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561b0b7ff280 msgr2=0x561b0b810190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:35:02.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.497+0000 7ff179a00640 1 --2- 172.21.15.90:0/3665963035 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561b0b7ff280 0x561b0b810190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:02.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.497+0000 7ff179a00640 1 -- 172.21.15.90:0/3665963035 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561b0b80c920 con 0x561b0b7fe960 2024-04-02T14:35:02.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.497+0000 7ff17a400640 1 --2- 172.21.15.90:0/3665963035 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561b0b807e60 0x561b0b80c1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:35:02.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.498+0000 7ff179a00640 1 --2- 172.21.15.90:0/3665963035 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561b0b7fe960 0x561b0b80fc50 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7ff16801a000 tx=0x7ff168005d00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a70e6c85abb667a3 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:35:02.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.498+0000 7ff15b400640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:35:02.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.498+0000 7ff15b400640 1 -- 172.21.15.90:0/3665963035 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff168004220 con 0x561b0b7fe960 2024-04-02T14:35:02.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.498+0000 7ff15b400640 1 -- 172.21.15.90:0/3665963035 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7ff1680043c0 con 0x561b0b7fe960 2024-04-02T14:35:02.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.498+0000 7ff17f215740 1 -- 172.21.15.90:0/3665963035 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff16009c100 con 0x561b0b7fe960 2024-04-02T14:35:02.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.498+0000 7ff15b400640 1 -- 172.21.15.90:0/3665963035 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7ff16803b050 con 0x561b0b7fe960 2024-04-02T14:35:02.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.498+0000 7ff17f215740 10 client.25483 fetch_fsmap learned FSMap version 27 2024-04-02T14:35:02.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.498+0000 7ff17f215740 1 -- 172.21.15.90:0/3665963035 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x561b0b87ee30 con 0x561b0b7fe960 2024-04-02T14:35:02.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.499+0000 7ff15b400640 1 -- 172.21.15.90:0/3665963035 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff16009c100 con 0x561b0b7fe960 2024-04-02T14:35:02.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.499+0000 7ff15b400640 1 -- 172.21.15.90:0/3665963035 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7ff168004c20 con 0x561b0b7fe960 2024-04-02T14:35:02.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.499+0000 7ff17f215740 10 client.25483 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:35:02.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.499+0000 7ff17f215740 10 client.25483 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:35:02.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.499+0000 7ff17f215740 20 client.25483 populate_metadata read hostname 'smithi090' 2024-04-02T14:35:02.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.499+0000 7ff17f215740 1 --2- 172.21.15.90:0/3665963035 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x561b0b87f280 0x561b0b8816a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:02.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.499+0000 7ff17f215740 4 client.25483 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-02T14:35:02.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.499+0000 7ff17f215740 1 -- 172.21.15.90:0/3665963035 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "scrub status"}) -- 0x561b0b76fcb0 con 0x561b0b87f280 2024-04-02T14:35:02.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.499+0000 7ff179000640 1 --2- 172.21.15.90:0/3665963035 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x561b0b87f280 0x561b0b8816a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:02.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.500+0000 7ff179000640 1 --2- 172.21.15.90:0/3665963035 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x561b0b87f280 0x561b0b8816a0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bc08ac5ecab93e87 server_cookie=2ab056ab93537f29 in_seq=0 out_seq=0 2024-04-02T14:35:02.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.500+0000 7ff15b400640 10 client.25483 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:35:02.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.500+0000 7ff15b400640 1 -- 172.21.15.90:0/3665963035 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x561b0b76fcb0 con 0x561b0b87f280 2024-04-02T14:35:02.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.500+0000 7ff15b400640 10 client.25483 handle_command_reply: tid=0 2024-04-02T14:35:02.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.500+0000 7ff17f215740 1 client.25483 shutdown 2024-04-02T14:35:02.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.500+0000 7ff17f215740 1 -- 172.21.15.90:0/3665963035 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561b0b7fe960 msgr2=0x561b0b80fc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:35:02.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.500+0000 7ff17f215740 1 --2- 172.21.15.90:0/3665963035 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561b0b7fe960 0x561b0b80fc50 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7ff16801a000 tx=0x7ff168005d00 comp rx=0 tx=0).stop 2024-04-02T14:35:02.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.500+0000 7ff17f215740 1 -- 172.21.15.90:0/3665963035 shutdown_connections 2024-04-02T14:35:02.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.500+0000 7ff17f215740 1 --2- 172.21.15.90:0/3665963035 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x561b0b87f280 0x561b0b8816a0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:02.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.500+0000 7ff17f215740 1 --2- 172.21.15.90:0/3665963035 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561b0b807e60 0x561b0b80c1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:02.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.500+0000 7ff17f215740 1 --2- 172.21.15.90:0/3665963035 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561b0b7ff280 0x561b0b810190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:02.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.500+0000 7ff15b400640 0 client.25483 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:35:02.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.500+0000 7ff17f215740 1 --2- 172.21.15.90:0/3665963035 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561b0b7fe960 0x561b0b80fc50 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:02.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.500+0000 7ff17f215740 1 -- 172.21.15.90:0/3665963035 >> 172.21.15.90:0/3665963035 conn(0x561b0b7f9090 msgr2=0x561b0b804cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:35:02.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.500+0000 7ff17f215740 1 -- 172.21.15.90:0/3665963035 shutdown_connections 2024-04-02T14:35:02.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.500+0000 7ff17f215740 1 -- 172.21.15.90:0/3665963035 wait complete. 2024-04-02T14:35:02.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.501+0000 7ff17f215740 20 client.25483 trim_cache size 0 max 16384 2024-04-02T14:35:02.505 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-02T14:35:02.505 INFO:teuthology.orchestra.run.smithi090.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-02T14:35:02.505 INFO:teuthology.orchestra.run.smithi090.stdout: "scrubs": { 2024-04-02T14:35:02.505 INFO:teuthology.orchestra.run.smithi090.stdout: "2dc2f89c-fc11-4753-8afc-05ebdfa775f1": { 2024-04-02T14:35:02.505 INFO:teuthology.orchestra.run.smithi090.stdout: "path": "/", 2024-04-02T14:35:02.505 INFO:teuthology.orchestra.run.smithi090.stdout: "tag": "2dc2f89c-fc11-4753-8afc-05ebdfa775f1", 2024-04-02T14:35:02.505 INFO:teuthology.orchestra.run.smithi090.stdout: "options": "recursive,force" 2024-04-02T14:35:02.506 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:35:02.506 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:35:02.506 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-02T14:35:02.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.501+0000 7ff17ae00640 1 -- 172.21.15.90:0/1088924657 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7ff15c0a3ec0 msgr2=0x7ff15c0a6380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:35:02.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.501+0000 7ff17ae00640 1 --2- 172.21.15.90:0/1088924657 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7ff15c0a3ec0 0x7ff15c0a6380 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7ff16c009bf0 tx=0x7ff16c02e040 comp rx=0 tx=0).stop 2024-04-02T14:35:02.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.501+0000 7ff17ae00640 1 -- 172.21.15.90:0/1088924657 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff1740747f0 msgr2=0x7ff17410ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:35:02.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.501+0000 7ff17ae00640 1 --2- 172.21.15.90:0/1088924657 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff1740747f0 0x7ff17410ac30 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7ff16000d950 tx=0x7ff16000de10 comp rx=0 tx=0).stop 2024-04-02T14:35:02.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.501+0000 7ff17ae00640 1 -- 172.21.15.90:0/1088924657 shutdown_connections 2024-04-02T14:35:02.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.501+0000 7ff17ae00640 1 --2- 172.21.15.90:0/1088924657 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7ff15c0a3ec0 0x7ff15c0a6380 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:02.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.501+0000 7ff17ae00640 1 --2- 172.21.15.90:0/1088924657 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff17410e210 0x7ff17410b190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:02.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.501+0000 7ff17ae00640 1 --2- 172.21.15.90:0/1088924657 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff1740747f0 0x7ff17410ac30 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:02.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.501+0000 7ff17ae00640 1 --2- 172.21.15.90:0/1088924657 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff174073ed0 0x7ff17410db90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:02.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.501+0000 7ff17ae00640 1 -- 172.21.15.90:0/1088924657 >> 172.21.15.90:0/1088924657 conn(0x7ff1740fd4a0 msgr2=0x7ff1740fe1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:35:02.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.502+0000 7ff17ae00640 1 -- 172.21.15.90:0/1088924657 shutdown_connections 2024-04-02T14:35:02.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.502+0000 7ff17ae00640 1 -- 172.21.15.90:0/1088924657 wait complete. 2024-04-02T14:35:02.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.516+0000 7f9575600640 1 -- 172.21.15.90:0/341079461 <== mgr.25289 v2:172.21.15.90:6800/3904993754 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f95780fe990 con 0x7f95600a3eb0 2024-04-02T14:35:02.517 INFO:teuthology.orchestra.run.smithi090.stdout:Schedule set for path / 2024-04-02T14:35:02.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.518+0000 7f957e000640 1 -- 172.21.15.90:0/341079461 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f95600a3eb0 msgr2=0x7f95600a6370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:35:02.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.518+0000 7f957e000640 1 --2- 172.21.15.90:0/341079461 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f95600a3eb0 0x7f95600a6370 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f9564005ab0 tx=0x7f9564005a20 comp rx=0 tx=0).stop 2024-04-02T14:35:02.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.518+0000 7f957e000640 1 -- 172.21.15.90:0/341079461 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f95781018e0 msgr2=0x7f957819a3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:35:02.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.518+0000 7f957e000640 1 --2- 172.21.15.90:0/341079461 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f95781018e0 0x7f957819a3d0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f956c00e9f0 tx=0x7f956c00edf0 comp rx=0 tx=0).stop 2024-04-02T14:35:02.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.519+0000 7f957e000640 1 -- 172.21.15.90:0/341079461 shutdown_connections 2024-04-02T14:35:02.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.519+0000 7f957e000640 1 --2- 172.21.15.90:0/341079461 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f95600a3eb0 0x7f95600a6370 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:02.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.519+0000 7f957e000640 1 --2- 172.21.15.90:0/341079461 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f95781078f0 0x7f957819ec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:02.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.519+0000 7f957e000640 1 --2- 172.21.15.90:0/341079461 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f95781022a0 0x7f957819a910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:02.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.519+0000 7f957e000640 1 --2- 172.21.15.90:0/341079461 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f95781018e0 0x7f957819a3d0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:02.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.519+0000 7f957e000640 1 -- 172.21.15.90:0/341079461 >> 172.21.15.90:0/341079461 conn(0x7f95780fd450 msgr2=0x7f957810cf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:35:02.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.519+0000 7f957e000640 1 -- 172.21.15.90:0/341079461 shutdown_connections 2024-04-02T14:35:02.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.519+0000 7f957e000640 1 -- 172.21.15.90:0/341079461 wait complete. 2024-04-02T14:35:02.523 INFO:tasks.cephfs.filesystem:scrub status for tag:2dc2f89c-fc11-4753-8afc-05ebdfa775f1 - {'path': '/', 'tag': '2dc2f89c-fc11-4753-8afc-05ebdfa775f1', 'options': 'recursive,force'} 2024-04-02T14:35:02.538 DEBUG:teuthology.orchestra.run.smithi090:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h' 2024-04-02T14:35:02.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.771+0000 7f3200000640 1 -- 172.21.15.90:0/3004764982 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f31f810d410 msgr2=0x7f31f8114fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:35:02.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.771+0000 7f3200000640 1 --2- 172.21.15.90:0/3004764982 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f31f810d410 0x7f31f8114fe0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f31f0009ca0 tx=0x7f31f002c100 comp rx=0 tx=0).stop 2024-04-02T14:35:02.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.771+0000 7f3200000640 1 -- 172.21.15.90:0/3004764982 shutdown_connections 2024-04-02T14:35:02.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.771+0000 7f3200000640 1 --2- 172.21.15.90:0/3004764982 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f31f810d410 0x7f31f8114fe0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:02.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.771+0000 7f3200000640 1 --2- 172.21.15.90:0/3004764982 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f31f8108cf0 0x7f31f810cd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:02.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.771+0000 7f3200000640 1 --2- 172.21.15.90:0/3004764982 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f31f8108340 0x7f31f8108720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:02.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.771+0000 7f3200000640 1 -- 172.21.15.90:0/3004764982 >> 172.21.15.90:0/3004764982 conn(0x7f31f8076070 msgr2=0x7f31f8078490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:35:02.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.771+0000 7f3200000640 1 -- 172.21.15.90:0/3004764982 shutdown_connections 2024-04-02T14:35:02.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.771+0000 7f3200000640 1 -- 172.21.15.90:0/3004764982 wait complete. 2024-04-02T14:35:02.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.772+0000 7f3200000640 1 Processor -- start 2024-04-02T14:35:02.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.772+0000 7f3200000640 1 -- start start 2024-04-02T14:35:02.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.773+0000 7f3200000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f31f8108340 0x7f31f81a4650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:02.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.773+0000 7f3200000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f31f810d410 0x7f31f81a4b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:02.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.773+0000 7f3200000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f31f819e710 0x7f31f819eb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:02.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.773+0000 7f3200000640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f31f81184d0 con 0x7f31f810d410 2024-04-02T14:35:02.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.773+0000 7f3200000640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f31f8118350 con 0x7f31f8108340 2024-04-02T14:35:02.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.773+0000 7f3200000640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f31f8118650 con 0x7f31f819e710 2024-04-02T14:35:02.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.773+0000 7f31fcc00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f31f810d410 0x7f31f81a4b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:02.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.773+0000 7f31fcc00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f31f810d410 0x7f31f81a4b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:53638/0 (socket says 172.21.15.90:53638) 2024-04-02T14:35:02.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.773+0000 7f31fcc00640 1 -- 172.21.15.90:0/2169781805 learned_addr learned my addr 172.21.15.90:0/2169781805 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:35:02.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.773+0000 7f31fd600640 1 --2- 172.21.15.90:0/2169781805 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f31f8108340 0x7f31f81a4650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:02.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.773+0000 7f31fe000640 1 --2- 172.21.15.90:0/2169781805 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f31f819e710 0x7f31f819eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:02.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.773+0000 7f31fcc00640 1 -- 172.21.15.90:0/2169781805 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f31f819e710 msgr2=0x7f31f819eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:35:02.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.773+0000 7f31fcc00640 1 --2- 172.21.15.90:0/2169781805 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f31f819e710 0x7f31f819eb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:02.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.773+0000 7f31fcc00640 1 -- 172.21.15.90:0/2169781805 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f31f8108340 msgr2=0x7f31f81a4650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:35:02.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.773+0000 7f31fcc00640 1 --2- 172.21.15.90:0/2169781805 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f31f8108340 0x7f31f81a4650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:02.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.773+0000 7f31fcc00640 1 -- 172.21.15.90:0/2169781805 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31f819f310 con 0x7f31f810d410 2024-04-02T14:35:02.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.773+0000 7f31fd600640 1 --2- 172.21.15.90:0/2169781805 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f31f8108340 0x7f31f81a4650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:35:02.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.774+0000 7f31fcc00640 1 --2- 172.21.15.90:0/2169781805 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f31f810d410 0x7f31f81a4b90 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f31e400e9c0 tx=0x7f31e400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:35:02.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.774+0000 7f31f6000640 1 -- 172.21.15.90:0/2169781805 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31e400cd80 con 0x7f31f810d410 2024-04-02T14:35:02.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.774+0000 7f31f6000640 1 -- 172.21.15.90:0/2169781805 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f31e4005500 con 0x7f31f810d410 2024-04-02T14:35:02.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.774+0000 7f3200000640 1 -- 172.21.15.90:0/2169781805 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f31f81ab3f0 con 0x7f31f810d410 2024-04-02T14:35:02.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.774+0000 7f31f6000640 1 -- 172.21.15.90:0/2169781805 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31e4010690 con 0x7f31f810d410 2024-04-02T14:35:02.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.774+0000 7f3200000640 1 -- 172.21.15.90:0/2169781805 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31f81ab7f0 con 0x7f31f810d410 2024-04-02T14:35:02.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.775+0000 7f3200000640 1 -- 172.21.15.90:0/2169781805 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f31f8109810 con 0x7f31f810d410 2024-04-02T14:35:02.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.778+0000 7f31f6000640 1 -- 172.21.15.90:0/2169781805 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7f31e40046c0 con 0x7f31f810d410 2024-04-02T14:35:02.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.779+0000 7f31f6000640 1 --2- 172.21.15.90:0/2169781805 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f31e00a3ee0 0x7f31e00a63a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:02.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.779+0000 7f31f6000640 1 -- 172.21.15.90:0/2169781805 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f31e40d0dd0 con 0x7f31f810d410 2024-04-02T14:35:02.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.779+0000 7f31fd600640 1 --2- 172.21.15.90:0/2169781805 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f31e00a3ee0 0x7f31e00a63a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:02.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.780+0000 7f31fd600640 1 --2- 172.21.15.90:0/2169781805 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f31e00a3ee0 0x7f31e00a63a0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f31ec009fc0 tx=0x7f31ec005f90 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:35:02.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.784+0000 7f31f6000640 1 -- 172.21.15.90:0/2169781805 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7f31e409d1d0 con 0x7f31f810d410 2024-04-02T14:35:02.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:02.992+0000 7f3200000640 1 -- 172.21.15.90:0/2169781805 --> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}) -- 0x7f31f8110710 con 0x7f31e00a3ee0 2024-04-02T14:35:03.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:03.139+0000 7f31f6000640 1 -- 172.21.15.90:0/2169781805 <== mgr.25289 v2:172.21.15.90:6800/3904993754 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+26 (secure 0 0 0) 0x7f31ec004030 con 0x7f31e00a3ee0 2024-04-02T14:35:03.141 INFO:teuthology.orchestra.run.smithi090.stdout:Retention added to path / 2024-04-02T14:35:03.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:03.143+0000 7f3200000640 1 -- 172.21.15.90:0/2169781805 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f31e00a3ee0 msgr2=0x7f31e00a63a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:35:03.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:03.143+0000 7f3200000640 1 --2- 172.21.15.90:0/2169781805 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f31e00a3ee0 0x7f31e00a63a0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f31ec009fc0 tx=0x7f31ec005f90 comp rx=0 tx=0).stop 2024-04-02T14:35:03.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:03.143+0000 7f3200000640 1 -- 172.21.15.90:0/2169781805 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f31f810d410 msgr2=0x7f31f81a4b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:35:03.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:03.143+0000 7f3200000640 1 --2- 172.21.15.90:0/2169781805 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f31f810d410 0x7f31f81a4b90 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f31e400e9c0 tx=0x7f31e400ee80 comp rx=0 tx=0).stop 2024-04-02T14:35:03.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:03.143+0000 7f3200000640 1 -- 172.21.15.90:0/2169781805 shutdown_connections 2024-04-02T14:35:03.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:03.143+0000 7f3200000640 1 --2- 172.21.15.90:0/2169781805 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f31e00a3ee0 0x7f31e00a63a0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:03.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:03.143+0000 7f3200000640 1 --2- 172.21.15.90:0/2169781805 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f31f819e710 0x7f31f819eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:03.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:03.143+0000 7f3200000640 1 --2- 172.21.15.90:0/2169781805 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f31f810d410 0x7f31f81a4b90 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:03.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:03.143+0000 7f3200000640 1 --2- 172.21.15.90:0/2169781805 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f31f8108340 0x7f31f81a4650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:03.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:03.143+0000 7f3200000640 1 -- 172.21.15.90:0/2169781805 >> 172.21.15.90:0/2169781805 conn(0x7f31f8076070 msgr2=0x7f31f8076bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:35:03.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:03.143+0000 7f3200000640 1 -- 172.21.15.90:0/2169781805 shutdown_connections 2024-04-02T14:35:03.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:03.143+0000 7f3200000640 1 -- 172.21.15.90:0/2169781805 wait complete. 2024-04-02T14:35:03.164 DEBUG:teuthology.orchestra.run.smithi090:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-04-02T14:35:03.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:35:03 smithi092 ceph-mon[36497]: scrub queued for path: / 2024-04-02T14:35:03.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:35:03 smithi092 ceph-mon[36497]: scrub summary: idle+waiting paths [/] 2024-04-02T14:35:03.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:35:03 smithi092 ceph-mon[36497]: scrub summary: active paths [/] 2024-04-02T14:35:03.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:35:03 smithi092 ceph-mon[36497]: scrub summary: idle+waiting paths [/] 2024-04-02T14:35:03.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:03.363+0000 7fab32800640 1 -- 172.21.15.90:0/3303673793 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fab2c10a0d0 msgr2=0x7fab2c110aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:35:03.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:03.363+0000 7fab32800640 1 --2- 172.21.15.90:0/3303673793 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fab2c10a0d0 0x7fab2c110aa0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fab24009c10 tx=0x7fab2402bf80 comp rx=0 tx=0).stop 2024-04-02T14:35:03.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:03.363+0000 7fab32800640 1 -- 172.21.15.90:0/3303673793 shutdown_connections 2024-04-02T14:35:03.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:03.363+0000 7fab32800640 1 --2- 172.21.15.90:0/3303673793 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fab2c10a0d0 0x7fab2c110aa0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:03.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:03.363+0000 7fab32800640 1 --2- 172.21.15.90:0/3303673793 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fab2c100530 0x7fab2c1099f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:03.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:03.363+0000 7fab32800640 1 --2- 172.21.15.90:0/3303673793 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fab2c0ffc10 0x7fab2c0ffff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:03.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:03.363+0000 7fab32800640 1 -- 172.21.15.90:0/3303673793 >> 172.21.15.90:0/3303673793 conn(0x7fab2c077970 msgr2=0x7fab2c0fdcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:35:03.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:03.363+0000 7fab32800640 1 -- 172.21.15.90:0/3303673793 shutdown_connections 2024-04-02T14:35:03.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:03.363+0000 7fab32800640 1 -- 172.21.15.90:0/3303673793 wait complete. 2024-04-02T14:35:03.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:03.364+0000 7fab32800640 1 Processor -- start 2024-04-02T14:35:03.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:03.364+0000 7fab32800640 1 -- start start 2024-04-02T14:35:03.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:03.365+0000 7fab32800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fab2c0ffc10 0x7fab2c19a340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:03.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:03.365+0000 7fab32800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fab2c100530 0x7fab2c19a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:03.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:03.365+0000 7fab32800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fab2c10a0d0 0x7fab2c19ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:03.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:03.365+0000 7fab32800640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fab2c113f00 con 0x7fab2c100530 2024-04-02T14:35:03.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:03.365+0000 7fab32800640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fab2c113d80 con 0x7fab2c0ffc10 2024-04-02T14:35:03.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:03.365+0000 7fab32800640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fab2c114080 con 0x7fab2c10a0d0 2024-04-02T14:35:03.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:03.365+0000 7fab30a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fab2c100530 0x7fab2c19a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:03.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:03.365+0000 7fab31e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fab2c10a0d0 0x7fab2c19ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:03.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:03.365+0000 7fab30a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fab2c100530 0x7fab2c19a880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:53650/0 (socket says 172.21.15.90:53650) 2024-04-02T14:35:03.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:03.365+0000 7fab30a00640 1 -- 172.21.15.90:0/2182023845 learned_addr learned my addr 172.21.15.90:0/2182023845 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:35:03.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:03.365+0000 7fab31400640 1 --2- 172.21.15.90:0/2182023845 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fab2c0ffc10 0x7fab2c19a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:03.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:03.366+0000 7fab31e00640 1 -- 172.21.15.90:0/2182023845 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fab2c0ffc10 msgr2=0x7fab2c19a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:35:03.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:03.366+0000 7fab31e00640 1 --2- 172.21.15.90:0/2182023845 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fab2c0ffc10 0x7fab2c19a340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:03.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:03.366+0000 7fab31e00640 1 -- 172.21.15.90:0/2182023845 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fab2c100530 msgr2=0x7fab2c19a880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:35:03.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:03.366+0000 7fab31e00640 1 --2- 172.21.15.90:0/2182023845 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fab2c100530 0x7fab2c19a880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:03.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:03.366+0000 7fab31e00640 1 -- 172.21.15.90:0/2182023845 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab2c19f330 con 0x7fab2c10a0d0 2024-04-02T14:35:03.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:03.366+0000 7fab30a00640 1 --2- 172.21.15.90:0/2182023845 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fab2c100530 0x7fab2c19a880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:35:03.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:03.366+0000 7fab31e00640 1 --2- 172.21.15.90:0/2182023845 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fab2c10a0d0 0x7fab2c19ec00 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fab2402c4c0 tx=0x7fab24004270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:35:03.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:03.366+0000 7fab2a000640 1 -- 172.21.15.90:0/2182023845 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab24043020 con 0x7fab2c10a0d0 2024-04-02T14:35:03.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:03.366+0000 7fab32800640 1 -- 172.21.15.90:0/2182023845 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fab2c19f5c0 con 0x7fab2c10a0d0 2024-04-02T14:35:03.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:03.366+0000 7fab2a000640 1 -- 172.21.15.90:0/2182023845 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fab24004670 con 0x7fab2c10a0d0 2024-04-02T14:35:03.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:03.366+0000 7fab2a000640 1 -- 172.21.15.90:0/2182023845 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab24034690 con 0x7fab2c10a0d0 2024-04-02T14:35:03.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:03.366+0000 7fab32800640 1 -- 172.21.15.90:0/2182023845 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fab2c1a6eb0 con 0x7fab2c10a0d0 2024-04-02T14:35:03.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:03.367+0000 7fab32800640 1 -- 172.21.15.90:0/2182023845 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fab2c0500b0 con 0x7fab2c10a0d0 2024-04-02T14:35:03.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:03.369+0000 7fab2a000640 1 -- 172.21.15.90:0/2182023845 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7fab24046020 con 0x7fab2c10a0d0 2024-04-02T14:35:03.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:03.371+0000 7fab2a000640 1 --2- 172.21.15.90:0/2182023845 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fab140a3e30 0x7fab140a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:03.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:03.371+0000 7fab2a000640 1 -- 172.21.15.90:0/2182023845 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7fab240f36d0 con 0x7fab2c10a0d0 2024-04-02T14:35:03.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:03.371+0000 7fab31400640 1 --2- 172.21.15.90:0/2182023845 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fab140a3e30 0x7fab140a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:03.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:03.372+0000 7fab31400640 1 --2- 172.21.15.90:0/2182023845 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fab140a3e30 0x7fab140a62f0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fab2c068740 tx=0x7fab20005f90 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:35:03.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:03.373+0000 7fab2a000640 1 -- 172.21.15.90:0/2182023845 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7fab2404d020 con 0x7fab2c10a0d0 2024-04-02T14:35:03.436 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:35:03 smithi177 ceph-mon[36431]: scrub queued for path: / 2024-04-02T14:35:03.436 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:35:03 smithi177 ceph-mon[36431]: scrub summary: idle+waiting paths [/] 2024-04-02T14:35:03.437 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:35:03 smithi177 ceph-mon[36431]: scrub summary: active paths [/] 2024-04-02T14:35:03.437 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:35:03 smithi177 ceph-mon[36431]: scrub summary: idle+waiting paths [/] 2024-04-02T14:35:03.438 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:35:03 smithi090 ceph-mon[31674]: scrub queued for path: / 2024-04-02T14:35:03.438 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:35:03 smithi090 ceph-mon[31674]: scrub summary: idle+waiting paths [/] 2024-04-02T14:35:03.438 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:35:03 smithi090 ceph-mon[31674]: scrub summary: active paths [/] 2024-04-02T14:35:03.439 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:35:03 smithi090 ceph-mon[31674]: scrub summary: idle+waiting paths [/] 2024-04-02T14:35:03.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:03.597+0000 7fab32800640 1 -- 172.21.15.90:0/2182023845 --> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) -- 0x7fab2c061f40 con 0x7fab140a3e30 2024-04-02T14:35:03.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:03.612+0000 7fab2a000640 1 -- 172.21.15.90:0/2182023845 <== mgr.25289 v2:172.21.15.90:6800/3904993754 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+298 (secure 0 0 0) 0x7fab2c061f40 con 0x7fab140a3e30 2024-04-02T14:35:03.614 INFO:teuthology.orchestra.run.smithi090.stdout:{"fs": "cephfs", "subvol": null, "group": null, "path": "/", "rel_path": "/", "schedule": "1m", "retention": {"m": 6, "h": 3}, "start": "2024-04-02T00:00:00", "created": "2024-04-02T14:35:02", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": true} 2024-04-02T14:35:03.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:03.615+0000 7fab32800640 1 -- 172.21.15.90:0/2182023845 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fab140a3e30 msgr2=0x7fab140a62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:35:03.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:03.615+0000 7fab32800640 1 --2- 172.21.15.90:0/2182023845 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fab140a3e30 0x7fab140a62f0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fab2c068740 tx=0x7fab20005f90 comp rx=0 tx=0).stop 2024-04-02T14:35:03.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:03.615+0000 7fab32800640 1 -- 172.21.15.90:0/2182023845 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fab2c10a0d0 msgr2=0x7fab2c19ec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:35:03.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:03.615+0000 7fab32800640 1 --2- 172.21.15.90:0/2182023845 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fab2c10a0d0 0x7fab2c19ec00 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fab2402c4c0 tx=0x7fab24004270 comp rx=0 tx=0).stop 2024-04-02T14:35:03.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:03.616+0000 7fab32800640 1 -- 172.21.15.90:0/2182023845 shutdown_connections 2024-04-02T14:35:03.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:03.616+0000 7fab32800640 1 --2- 172.21.15.90:0/2182023845 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fab140a3e30 0x7fab140a62f0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:03.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:03.616+0000 7fab32800640 1 --2- 172.21.15.90:0/2182023845 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fab2c10a0d0 0x7fab2c19ec00 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:03.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:03.616+0000 7fab32800640 1 --2- 172.21.15.90:0/2182023845 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fab2c100530 0x7fab2c19a880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:03.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:03.616+0000 7fab32800640 1 --2- 172.21.15.90:0/2182023845 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fab2c0ffc10 0x7fab2c19a340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:03.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:03.616+0000 7fab32800640 1 -- 172.21.15.90:0/2182023845 >> 172.21.15.90:0/2182023845 conn(0x7fab2c077970 msgr2=0x7fab2c0fdcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:35:03.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:03.616+0000 7fab32800640 1 -- 172.21.15.90:0/2182023845 shutdown_connections 2024-04-02T14:35:03.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:03.616+0000 7fab32800640 1 -- 172.21.15.90:0/2182023845 wait complete. 2024-04-02T14:35:03.636 DEBUG:teuthology.orchestra.run.smithi090:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2024-04-02T14:35:03.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:03.846+0000 7f7536200640 1 -- 172.21.15.90:0/302603672 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7530108340 msgr2=0x7f7530108720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:35:03.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:03.846+0000 7f7536200640 1 --2- 172.21.15.90:0/302603672 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7530108340 0x7f7530108720 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f7520009900 tx=0x7f752002bc30 comp rx=0 tx=0).stop 2024-04-02T14:35:03.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:03.846+0000 7f7536200640 1 -- 172.21.15.90:0/302603672 shutdown_connections 2024-04-02T14:35:03.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:03.846+0000 7f7536200640 1 --2- 172.21.15.90:0/302603672 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f753010d410 0x7f7530114fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:03.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:03.846+0000 7f7536200640 1 --2- 172.21.15.90:0/302603672 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7530108cf0 0x7f753010cd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:03.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:03.846+0000 7f7536200640 1 --2- 172.21.15.90:0/302603672 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7530108340 0x7f7530108720 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:03.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:03.846+0000 7f7536200640 1 -- 172.21.15.90:0/302603672 >> 172.21.15.90:0/302603672 conn(0x7f7530076070 msgr2=0x7f7530078490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:35:03.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:03.847+0000 7f7536200640 1 -- 172.21.15.90:0/302603672 shutdown_connections 2024-04-02T14:35:03.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:03.847+0000 7f7536200640 1 -- 172.21.15.90:0/302603672 wait complete. 2024-04-02T14:35:03.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:03.847+0000 7f7536200640 1 Processor -- start 2024-04-02T14:35:03.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:03.848+0000 7f7536200640 1 -- start start 2024-04-02T14:35:03.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:03.848+0000 7f7536200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7530108340 0x7f75301a4620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:03.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:03.848+0000 7f7536200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f753010d410 0x7f75301a4b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:03.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:03.848+0000 7f7536200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f753019e790 0x7f753019ec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:03.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:03.848+0000 7f7536200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f7530118510 con 0x7f753010d410 2024-04-02T14:35:03.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:03.848+0000 7f7536200640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f7530118390 con 0x7f753019e790 2024-04-02T14:35:03.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:03.848+0000 7f7536200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f7530118690 con 0x7f7530108340 2024-04-02T14:35:03.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:03.848+0000 7f752f400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7530108340 0x7f75301a4620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:03.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:03.848+0000 7f7526a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f753010d410 0x7f75301a4b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:03.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:03.848+0000 7f752f400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7530108340 0x7f75301a4620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:43088/0 (socket says 172.21.15.90:43088) 2024-04-02T14:35:03.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:03.848+0000 7f752f400640 1 -- 172.21.15.90:0/3843400671 learned_addr learned my addr 172.21.15.90:0/3843400671 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:35:03.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:03.848+0000 7f752fe00640 1 --2- 172.21.15.90:0/3843400671 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f753019e790 0x7f753019ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:03.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:03.849+0000 7f752f400640 1 -- 172.21.15.90:0/3843400671 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f753019e790 msgr2=0x7f753019ec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:35:03.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:03.849+0000 7f752f400640 1 --2- 172.21.15.90:0/3843400671 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f753019e790 0x7f753019ec40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:03.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:03.849+0000 7f752f400640 1 -- 172.21.15.90:0/3843400671 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f753010d410 msgr2=0x7f75301a4b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:35:03.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:03.849+0000 7f752f400640 1 --2- 172.21.15.90:0/3843400671 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f753010d410 0x7f75301a4b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:03.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:03.849+0000 7f752f400640 1 -- 172.21.15.90:0/3843400671 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f753019f320 con 0x7f7530108340 2024-04-02T14:35:03.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:03.849+0000 7f7526a00640 1 --2- 172.21.15.90:0/3843400671 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f753010d410 0x7f75301a4b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:35:03.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:03.849+0000 7f752fe00640 1 --2- 172.21.15.90:0/3843400671 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f753019e790 0x7f753019ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:35:03.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:03.849+0000 7f752f400640 1 --2- 172.21.15.90:0/3843400671 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7530108340 0x7f75301a4620 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f752002c240 tx=0x7f75200042f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:35:03.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:03.849+0000 7f752cc00640 1 -- 172.21.15.90:0/3843400671 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7520043020 con 0x7f7530108340 2024-04-02T14:35:03.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:03.849+0000 7f7536200640 1 -- 172.21.15.90:0/3843400671 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f753019f5b0 con 0x7f7530108340 2024-04-02T14:35:03.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:03.849+0000 7f752cc00640 1 -- 172.21.15.90:0/3843400671 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f75200054e0 con 0x7f7530108340 2024-04-02T14:35:03.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:03.849+0000 7f752cc00640 1 -- 172.21.15.90:0/3843400671 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7520003b50 con 0x7f7530108340 2024-04-02T14:35:03.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:03.850+0000 7f7536200640 1 -- 172.21.15.90:0/3843400671 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75301ab790 con 0x7f7530108340 2024-04-02T14:35:03.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:03.851+0000 7f7536200640 1 -- 172.21.15.90:0/3843400671 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7530109810 con 0x7f7530108340 2024-04-02T14:35:03.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:03.853+0000 7f752cc00640 1 -- 172.21.15.90:0/3843400671 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7f7520046020 con 0x7f7530108340 2024-04-02T14:35:03.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:03.854+0000 7f752cc00640 1 --2- 172.21.15.90:0/3843400671 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f75140a3ee0 0x7f75140a63a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:03.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:03.854+0000 7f752cc00640 1 -- 172.21.15.90:0/3843400671 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f75200f3300 con 0x7f7530108340 2024-04-02T14:35:03.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:03.854+0000 7f7526a00640 1 --2- 172.21.15.90:0/3843400671 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f75140a3ee0 0x7f75140a63a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:03.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:03.855+0000 7f7526a00640 1 --2- 172.21.15.90:0/3843400671 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f75140a3ee0 0x7f75140a63a0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f7530107400 tx=0x7f7518005f90 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:35:03.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:03.856+0000 7f752cc00640 1 -- 172.21.15.90:0/3843400671 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7f752004d020 con 0x7f7530108340 2024-04-02T14:35:04.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:04.082+0000 7f7536200640 1 -- 172.21.15.90:0/3843400671 --> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) -- 0x7f7530108cf0 con 0x7f75140a3ee0 2024-04-02T14:35:04.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:04.096+0000 7f752cc00640 1 -- 172.21.15.90:0/3843400671 <== mgr.25289 v2:172.21.15.90:6800/3904993754 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+10 (secure 0 0 0) 0x7f7530108cf0 con 0x7f75140a3ee0 2024-04-02T14:35:04.097 INFO:teuthology.orchestra.run.smithi090.stdout:/ 1m 6m3h 2024-04-02T14:35:04.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:04.098+0000 7f7536200640 1 -- 172.21.15.90:0/3843400671 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f75140a3ee0 msgr2=0x7f75140a63a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:35:04.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:04.098+0000 7f7536200640 1 --2- 172.21.15.90:0/3843400671 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f75140a3ee0 0x7f75140a63a0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f7530107400 tx=0x7f7518005f90 comp rx=0 tx=0).stop 2024-04-02T14:35:04.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:04.098+0000 7f7536200640 1 -- 172.21.15.90:0/3843400671 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7530108340 msgr2=0x7f75301a4620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:35:04.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:04.098+0000 7f7536200640 1 --2- 172.21.15.90:0/3843400671 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7530108340 0x7f75301a4620 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f752002c240 tx=0x7f75200042f0 comp rx=0 tx=0).stop 2024-04-02T14:35:04.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:04.098+0000 7f7536200640 1 -- 172.21.15.90:0/3843400671 shutdown_connections 2024-04-02T14:35:04.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:04.098+0000 7f7536200640 1 --2- 172.21.15.90:0/3843400671 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f75140a3ee0 0x7f75140a63a0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:04.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:04.098+0000 7f7536200640 1 --2- 172.21.15.90:0/3843400671 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f753019e790 0x7f753019ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:04.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:04.098+0000 7f7536200640 1 --2- 172.21.15.90:0/3843400671 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f753010d410 0x7f75301a4b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:04.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:04.098+0000 7f7536200640 1 --2- 172.21.15.90:0/3843400671 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7530108340 0x7f75301a4620 secure :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f752002c240 tx=0x7f75200042f0 comp rx=0 tx=0).stop 2024-04-02T14:35:04.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:04.098+0000 7f7536200640 1 -- 172.21.15.90:0/3843400671 >> 172.21.15.90:0/3843400671 conn(0x7f7530076070 msgr2=0x7f7530078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:35:04.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:04.099+0000 7f7536200640 1 -- 172.21.15.90:0/3843400671 shutdown_connections 2024-04-02T14:35:04.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:04.099+0000 7f7536200640 1 -- 172.21.15.90:0/3843400671 wait complete. 2024-04-02T14:35:04.119 DEBUG:teuthology.orchestra.run.smithi090:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > START_TIME' 2024-04-02T14:35:04.210 INFO:teuthology.run_tasks:Running task full_sequential_finally... 2024-04-02T14:35:04.221 INFO:teuthology.run_tasks:Running task background_exec... 2024-04-02T14:35:04.324 INFO:teuthology.task.background_exec:Running background command on role mon.a host ubuntu@smithi090.front.sepia.ceph.com 2024-04-02T14:35:04.325 DEBUG:teuthology.orchestra.run.smithi090:> sudo TESTDIR=/home/ubuntu/cephtest daemon-helper kill --kill-group bash -c 'while sleep 13; do ceph tell mds.cephfs:0 flush journal; done' 2024-04-02T14:35:04.333 INFO:teuthology.run_tasks:Running task exec... 2024-04-02T14:35:04.356 INFO:teuthology.task.exec:Executing custom commands... 2024-04-02T14:35:04.356 INFO:teuthology.task.exec:Running commands on role client.0 host ubuntu@smithi090.front.sepia.ceph.com 2024-04-02T14:35:04.356 DEBUG:teuthology.orchestra.run.smithi090:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ls -l /tmp/cephfs/ && sudo df -h /tmp/cephfs/' 2024-04-02T14:35:04.360 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:35:04 smithi092 ceph-mon[36497]: pgmap v8: 97 pgs: 97 active+clean; 1.1 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 1.1 KiB/s wr, 20 op/s 2024-04-02T14:35:04.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:35:04 smithi092 ceph-mon[36497]: from='client.25450 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T14:35:04.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:35:04 smithi092 ceph-mon[36497]: from='client.15408 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T14:35:04.402 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:35:04 smithi090 ceph-mon[31674]: pgmap v8: 97 pgs: 97 active+clean; 1.1 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 1.1 KiB/s wr, 20 op/s 2024-04-02T14:35:04.402 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:35:04 smithi090 ceph-mon[31674]: from='client.25450 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T14:35:04.402 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:35:04 smithi090 ceph-mon[31674]: from='client.15408 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T14:35:04.437 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:35:04 smithi177 ceph-mon[36431]: pgmap v8: 97 pgs: 97 active+clean; 1.1 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 1.1 KiB/s wr, 20 op/s 2024-04-02T14:35:04.437 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:35:04 smithi177 ceph-mon[36431]: from='client.25450 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T14:35:04.438 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:35:04 smithi177 ceph-mon[36431]: from='client.15408 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T14:35:04.465 INFO:teuthology.orchestra.run.smithi090.stdout:total 0 2024-04-02T14:35:04.514 INFO:teuthology.orchestra.run.smithi090.stdout:Filesystem Size Used Avail Use% Mounted on 2024-04-02T14:35:04.514 INFO:teuthology.orchestra.run.smithi090.stdout:0@6449448a-f0fc-11ee-b647-cb9ed24678a4.cephfs=/volumes/qa/sv_1/136ad3dc-4348-42ec-932a-abe31a1ca7ec 340G 0 340G 0% /tmp/cephfs 2024-04-02T14:35:04.522 DEBUG:teuthology.orchestra.run.smithi090:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo mkdir -m 755 --context=system_u:system_r:postgresql_t:s0 /tmp/cephfs/postgres && sudo chown postgres:postgres /tmp/cephfs/postgres' 2024-04-02T14:35:04.753 DEBUG:teuthology.orchestra.run.smithi090:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo -u postgres -- mkdir -m 700 --context=system_u:system_r:postgresql_t:s0 /tmp/cephfs/postgres/data' 2024-04-02T14:35:04.878 DEBUG:teuthology.orchestra.run.smithi090:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo semanage fcontext -a -t postgresql_db_t "/tmp/cephfs/postgres(/.*)?"' 2024-04-02T14:35:05.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:35:05 smithi092 ceph-mon[36497]: from='client.25495 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T14:35:05.436 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:35:05 smithi177 ceph-mon[36431]: from='client.25495 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T14:35:05.438 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:35:05 smithi090 ceph-mon[31674]: from='client.25495 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T14:35:06.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:35:06 smithi092 ceph-mon[36497]: from='client.25501 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T14:35:06.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:35:06 smithi092 ceph-mon[36497]: pgmap v9: 97 pgs: 97 active+clean; 1.3 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 84 KiB/s wr, 53 op/s 2024-04-02T14:35:06.436 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:35:06 smithi177 ceph-mon[36431]: from='client.25501 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T14:35:06.437 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:35:06 smithi177 ceph-mon[36431]: pgmap v9: 97 pgs: 97 active+clean; 1.3 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 84 KiB/s wr, 53 op/s 2024-04-02T14:35:06.438 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:35:06 smithi090 ceph-mon[31674]: from='client.25501 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T14:35:06.438 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:35:06 smithi090 ceph-mon[31674]: pgmap v9: 97 pgs: 97 active+clean; 1.3 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 84 KiB/s wr, 53 op/s 2024-04-02T14:35:07.345 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:35:07 smithi090 ceph-mon[31674]: pgmap v10: 97 pgs: 97 active+clean; 1.3 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 77 KiB/s wr, 48 op/s 2024-04-02T14:35:07.345 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:35:07 smithi090 ceph-mon[31674]: scrub summary: idle 2024-04-02T14:35:07.350 DEBUG:teuthology.orchestra.run.smithi090:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo grep -i postgresql /etc/selinux/targeted/contexts/files/file_contexts.local' 2024-04-02T14:35:07.360 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:35:07 smithi092 ceph-mon[36497]: pgmap v10: 97 pgs: 97 active+clean; 1.3 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 77 KiB/s wr, 48 op/s 2024-04-02T14:35:07.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:35:07 smithi092 ceph-mon[36497]: scrub summary: idle 2024-04-02T14:35:07.436 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:35:07 smithi177 ceph-mon[36431]: pgmap v10: 97 pgs: 97 active+clean; 1.3 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 77 KiB/s wr, 48 op/s 2024-04-02T14:35:07.436 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:35:07 smithi177 ceph-mon[36431]: scrub summary: idle 2024-04-02T14:35:07.485 INFO:teuthology.orchestra.run.smithi090.stdout:/tmp/cephfs/postgres(/.*)? system_u:object_r:postgresql_db_t:s0 2024-04-02T14:35:07.493 DEBUG:teuthology.orchestra.run.smithi090:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo restorecon -R -v /tmp/cephfs/postgres' 2024-04-02T14:35:07.631 DEBUG:teuthology.orchestra.run.smithi090:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ls -lZaR /tmp/cephfs/postgres/' 2024-04-02T14:35:07.758 INFO:teuthology.orchestra.run.smithi090.stdout:/tmp/cephfs/postgres/: 2024-04-02T14:35:07.758 INFO:teuthology.orchestra.run.smithi090.stdout:total 0 2024-04-02T14:35:07.758 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x. 3 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Apr 2 14:35 . 2024-04-02T14:35:07.758 INFO:teuthology.orchestra.run.smithi090.stdout:drwxrwxrwt. 3 root root system_u:object_r:postgresql_db_t:s0 1 Apr 2 14:35 .. 2024-04-02T14:35:07.758 INFO:teuthology.orchestra.run.smithi090.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 data 2024-04-02T14:35:07.758 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-02T14:35:07.758 INFO:teuthology.orchestra.run.smithi090.stdout:/tmp/cephfs/postgres/data: 2024-04-02T14:35:07.758 INFO:teuthology.orchestra.run.smithi090.stdout:total 0 2024-04-02T14:35:07.758 INFO:teuthology.orchestra.run.smithi090.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 . 2024-04-02T14:35:07.759 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x. 3 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Apr 2 14:35 .. 2024-04-02T14:35:07.766 DEBUG:teuthology.orchestra.run.smithi090:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo mkdir -p /etc/systemd/system/postgresql.service.d/ && printf '"'"'[Service]\nEnvironment=PGDATA=/tmp/cephfs/postgres/data\nEnvironment=PGLOG=/tmp/cephfs/postgres/pgstartup.log\n'"'"' | sudo tee /etc/systemd/system/postgresql.service.d/env.conf' 2024-04-02T14:35:07.935 INFO:teuthology.orchestra.run.smithi090.stdout:[Service] 2024-04-02T14:35:07.936 INFO:teuthology.orchestra.run.smithi090.stdout:Environment=PGDATA=/tmp/cephfs/postgres/data 2024-04-02T14:35:07.936 INFO:teuthology.orchestra.run.smithi090.stdout:Environment=PGLOG=/tmp/cephfs/postgres/pgstartup.log 2024-04-02T14:35:07.944 DEBUG:teuthology.orchestra.run.smithi090:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo -u postgres -- postgresql-setup --initdb' 2024-04-02T14:35:08.229 INFO:teuthology.orchestra.run.smithi090.stderr: * Initializing database in '/tmp/cephfs/postgres/data' 2024-04-02T14:35:09.610 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:35:09 smithi092 ceph-mon[36497]: pgmap v11: 97 pgs: 97 active+clean; 1.3 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 77 KiB/s wr, 48 op/s 2024-04-02T14:35:09.686 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:35:09 smithi177 ceph-mon[36431]: pgmap v11: 97 pgs: 97 active+clean; 1.3 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 77 KiB/s wr, 48 op/s 2024-04-02T14:35:09.688 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:35:09 smithi090 ceph-mon[31674]: pgmap v11: 97 pgs: 97 active+clean; 1.3 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 77 KiB/s wr, 48 op/s 2024-04-02T14:35:11.610 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:35:11 smithi092 ceph-mon[36497]: pgmap v12: 97 pgs: 97 active+clean; 1.3 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 78 KiB/s wr, 56 op/s 2024-04-02T14:35:11.686 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:35:11 smithi177 ceph-mon[36431]: pgmap v12: 97 pgs: 97 active+clean; 1.3 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 78 KiB/s wr, 56 op/s 2024-04-02T14:35:11.688 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:35:11 smithi090 ceph-mon[31674]: pgmap v12: 97 pgs: 97 active+clean; 1.3 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 78 KiB/s wr, 56 op/s 2024-04-02T14:35:13.611 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:35:13 smithi092 ceph-mon[36497]: pgmap v13: 97 pgs: 97 active+clean; 1.3 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 78 KiB/s wr, 41 op/s 2024-04-02T14:35:13.686 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:35:13 smithi177 ceph-mon[36431]: pgmap v13: 97 pgs: 97 active+clean; 1.3 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 78 KiB/s wr, 41 op/s 2024-04-02T14:35:13.688 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:35:13 smithi090 ceph-mon[31674]: pgmap v13: 97 pgs: 97 active+clean; 1.3 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 78 KiB/s wr, 41 op/s 2024-04-02T14:35:15.610 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:35:15 smithi092 ceph-mon[36497]: pgmap v14: 97 pgs: 97 active+clean; 21 MiB data, 469 MiB used, 1.0 TiB / 1.0 TiB avail; 159 KiB/s rd, 1.7 MiB/s wr, 55 op/s 2024-04-02T14:35:15.686 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:35:15 smithi177 ceph-mon[36431]: pgmap v14: 97 pgs: 97 active+clean; 21 MiB data, 469 MiB used, 1.0 TiB / 1.0 TiB avail; 159 KiB/s rd, 1.7 MiB/s wr, 55 op/s 2024-04-02T14:35:15.688 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:35:15 smithi090 ceph-mon[31674]: pgmap v14: 97 pgs: 97 active+clean; 21 MiB data, 469 MiB used, 1.0 TiB / 1.0 TiB avail; 159 KiB/s rd, 1.7 MiB/s wr, 55 op/s 2024-04-02T14:35:17.571 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.569+0000 7ffacae00640 1 -- 172.21.15.90:0/4228777181 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffac41048d0 msgr2=0x7ffac4108910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:35:17.571 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.569+0000 7ffacae00640 1 --2- 172.21.15.90:0/4228777181 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffac41048d0 0x7ffac4108910 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7ffaac009900 tx=0x7ffaac02bc50 comp rx=0 tx=0).stop 2024-04-02T14:35:17.571 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.569+0000 7ffacae00640 1 -- 172.21.15.90:0/4228777181 shutdown_connections 2024-04-02T14:35:17.571 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.569+0000 7ffacae00640 1 --2- 172.21.15.90:0/4228777181 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ffac4108ff0 0x7ffac4110bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:17.571 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.569+0000 7ffacae00640 1 --2- 172.21.15.90:0/4228777181 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffac41048d0 0x7ffac4108910 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:17.571 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.569+0000 7ffacae00640 1 --2- 172.21.15.90:0/4228777181 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffac4103f20 0x7ffac4104300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:17.571 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.569+0000 7ffacae00640 1 -- 172.21.15.90:0/4228777181 >> 172.21.15.90:0/4228777181 conn(0x7ffac40ff7c0 msgr2=0x7ffac4101be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:35:17.571 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.569+0000 7ffacae00640 1 -- 172.21.15.90:0/4228777181 shutdown_connections 2024-04-02T14:35:17.572 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.569+0000 7ffacae00640 1 -- 172.21.15.90:0/4228777181 wait complete. 2024-04-02T14:35:17.574 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.570+0000 7ffacae00640 1 Processor -- start 2024-04-02T14:35:17.574 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.570+0000 7ffacae00640 1 -- start start 2024-04-02T14:35:17.574 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.571+0000 7ffacae00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffac4103f20 0x7ffac419a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:17.574 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.571+0000 7ffacae00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ffac4108ff0 0x7ffac419ab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:17.574 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.571+0000 7ffacae00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffac419eea0 0x7ffac419f350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:17.574 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.571+0000 7ffacae00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7ffac4114060 con 0x7ffac419eea0 2024-04-02T14:35:17.574 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.571+0000 7ffacae00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7ffac4113ee0 con 0x7ffac4108ff0 2024-04-02T14:35:17.574 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.571+0000 7ffacae00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7ffac41141e0 con 0x7ffac4103f20 2024-04-02T14:35:17.574 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.571+0000 7ffac3400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ffac4108ff0 0x7ffac419ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:17.574 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.571+0000 7ffac3e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffac4103f20 0x7ffac419a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:17.575 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.571+0000 7ffac8e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffac419eea0 0x7ffac419f350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:17.575 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.571+0000 7ffac3400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ffac4108ff0 0x7ffac419ab20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:32808/0 (socket says 172.21.15.90:32808) 2024-04-02T14:35:17.575 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.571+0000 7ffac3e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffac4103f20 0x7ffac419a5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:36964/0 (socket says 172.21.15.90:36964) 2024-04-02T14:35:17.575 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.571+0000 7ffac3400640 1 -- 172.21.15.90:0/2161290921 learned_addr learned my addr 172.21.15.90:0/2161290921 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:35:17.575 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.571+0000 7ffac3e00640 1 -- 172.21.15.90:0/2161290921 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ffac4108ff0 msgr2=0x7ffac419ab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:35:17.575 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.571+0000 7ffac3e00640 1 --2- 172.21.15.90:0/2161290921 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ffac4108ff0 0x7ffac419ab20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:17.575 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.571+0000 7ffac3e00640 1 -- 172.21.15.90:0/2161290921 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffac419eea0 msgr2=0x7ffac419f350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:35:17.575 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.571+0000 7ffac3e00640 1 --2- 172.21.15.90:0/2161290921 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffac419eea0 0x7ffac419f350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:17.575 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.571+0000 7ffac3e00640 1 -- 172.21.15.90:0/2161290921 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffac419f990 con 0x7ffac4103f20 2024-04-02T14:35:17.575 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.572+0000 7ffac3e00640 1 --2- 172.21.15.90:0/2161290921 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffac4103f20 0x7ffac419a5e0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7ffab400eb00 tx=0x7ffab400c2d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:35:17.575 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.572+0000 7ffac0c00640 1 -- 172.21.15.90:0/2161290921 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffab4010430 con 0x7ffac4103f20 2024-04-02T14:35:17.575 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.572+0000 7ffac0c00640 1 -- 172.21.15.90:0/2161290921 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffab4010ab0 con 0x7ffac4103f20 2024-04-02T14:35:17.575 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.572+0000 7ffac0c00640 1 -- 172.21.15.90:0/2161290921 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffab4017bb0 con 0x7ffac4103f20 2024-04-02T14:35:17.576 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.572+0000 7ffacae00640 1 -- 172.21.15.90:0/2161290921 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffac419fc80 con 0x7ffac4103f20 2024-04-02T14:35:17.576 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.572+0000 7ffacae00640 1 -- 172.21.15.90:0/2161290921 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffac41a74c0 con 0x7ffac4103f20 2024-04-02T14:35:17.576 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.575+0000 7ffac0c00640 1 -- 172.21.15.90:0/2161290921 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7ffab4010c60 con 0x7ffac4103f20 2024-04-02T14:35:17.576 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.575+0000 7ffacdbf9740 1 Processor -- start 2024-04-02T14:35:17.576 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.575+0000 7ffacdbf9740 1 -- start start 2024-04-02T14:35:17.578 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.576+0000 7ffacdbf9740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b41977aeb0 0x55b41977b290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:17.578 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.576+0000 7ffacdbf9740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b41977b7d0 0x55b419783e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:17.578 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.576+0000 7ffacdbf9740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b4197843b0 0x55b4197867a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:17.578 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.576+0000 7ffacdbf9740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55b4196dfd70 con 0x55b4197843b0 2024-04-02T14:35:17.578 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.576+0000 7ffacdbf9740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55b4196882a0 con 0x55b41977b7d0 2024-04-02T14:35:17.578 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.576+0000 7ffacdbf9740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55b4196dbd20 con 0x55b41977aeb0 2024-04-02T14:35:17.579 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.576+0000 7ffac3400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b4197843b0 0x55b4197867a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:17.579 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.576+0000 7ffac3e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b41977b7d0 0x55b419783e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:17.579 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.576+0000 7ffac8e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b41977aeb0 0x55b41977b290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:17.579 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.576+0000 7ffac3400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b4197843b0 0x55b4197867a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34218/0 (socket says 172.21.15.90:34218) 2024-04-02T14:35:17.579 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.576+0000 7ffac3400640 1 -- 172.21.15.90:0/656823512 learned_addr learned my addr 172.21.15.90:0/656823512 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:35:17.579 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.577+0000 7ffac3400640 1 -- 172.21.15.90:0/656823512 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b41977aeb0 msgr2=0x55b41977b290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:35:17.579 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.577+0000 7ffac3400640 1 --2- 172.21.15.90:0/656823512 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b41977aeb0 0x55b41977b290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:17.579 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.577+0000 7ffac3400640 1 -- 172.21.15.90:0/656823512 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b41977b7d0 msgr2=0x55b419783e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:35:17.579 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.577+0000 7ffac3400640 1 --2- 172.21.15.90:0/656823512 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b41977b7d0 0x55b419783e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:17.579 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.577+0000 7ffac3400640 1 -- 172.21.15.90:0/656823512 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b41969c840 con 0x55b4197843b0 2024-04-02T14:35:17.579 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.577+0000 7ffac8e00640 1 --2- 172.21.15.90:0/656823512 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b41977aeb0 0x55b41977b290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:35:17.579 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.577+0000 7ffac3e00640 1 --2- 172.21.15.90:0/656823512 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b41977b7d0 0x55b419783e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:35:17.579 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.577+0000 7ffac0c00640 1 --2- 172.21.15.90:0/2161290921 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7ffaa80a3f60 0x7ffaa80a6420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:17.580 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.577+0000 7ffac3400640 1 --2- 172.21.15.90:0/656823512 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b4197843b0 0x55b4197867a0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7ffaac036040 tx=0x7ffaac005dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38ced6d743bb1e5d server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:35:17.580 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.577+0000 7ffac0c00640 1 -- 172.21.15.90:0/2161290921 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7ffab40d13e0 con 0x7ffac4103f20 2024-04-02T14:35:17.584 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.577+0000 7ffab9600640 1 -- 172.21.15.90:0/656823512 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffaac02caa0 con 0x55b4197843b0 2024-04-02T14:35:17.584 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.577+0000 7ffac8e00640 1 --2- 172.21.15.90:0/2161290921 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7ffaa80a3f60 0x7ffaa80a6420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:17.584 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.578+0000 7ffab9600640 1 -- 172.21.15.90:0/656823512 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffaac02cc40 con 0x55b4197843b0 2024-04-02T14:35:17.584 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.578+0000 7ffab9600640 1 -- 172.21.15.90:0/656823512 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffaac005140 con 0x55b4197843b0 2024-04-02T14:35:17.584 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.578+0000 7ffacdbf9740 1 -- 172.21.15.90:0/656823512 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b4197843b0 msgr2=0x55b4197867a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:35:17.584 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.578+0000 7ffacdbf9740 1 --2- 172.21.15.90:0/656823512 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b4197843b0 0x55b4197867a0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7ffaac036040 tx=0x7ffaac005dd0 comp rx=0 tx=0).stop 2024-04-02T14:35:17.584 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.578+0000 7ffac8e00640 1 --2- 172.21.15.90:0/2161290921 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7ffaa80a3f60 0x7ffaa80a6420 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7ffabc00b890 tx=0x7ffabc00b2e0 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:35:17.585 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.578+0000 7ffacdbf9740 1 -- 172.21.15.90:0/656823512 shutdown_connections 2024-04-02T14:35:17.585 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.578+0000 7ffacdbf9740 1 --2- 172.21.15.90:0/656823512 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b4197843b0 0x55b4197867a0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:17.585 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.578+0000 7ffacdbf9740 1 --2- 172.21.15.90:0/656823512 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b41977b7d0 0x55b419783e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:17.585 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.578+0000 7ffacdbf9740 1 --2- 172.21.15.90:0/656823512 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b41977aeb0 0x55b41977b290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:17.585 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.579+0000 7ffacdbf9740 1 -- 172.21.15.90:0/656823512 >> 172.21.15.90:0/656823512 conn(0x55b4197755e0 msgr2=0x55b4197759c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:35:17.585 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.579+0000 7ffacdbf9740 1 -- 172.21.15.90:0/656823512 shutdown_connections 2024-04-02T14:35:17.585 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.579+0000 7ffacdbf9740 1 -- 172.21.15.90:0/656823512 wait complete. 2024-04-02T14:35:17.585 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.579+0000 7ffacdbf9740 1 -- start start 2024-04-02T14:35:17.585 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.579+0000 7ffacdbf9740 1 Processor -- start 2024-04-02T14:35:17.585 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.580+0000 7ffacdbf9740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b41977aeb0 0x55b419781430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:17.585 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.580+0000 7ffacdbf9740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b41977b7d0 0x55b419781970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:17.585 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.580+0000 7ffacdbf9740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b4197843b0 0x55b419781eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:17.586 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.580+0000 7ffacdbf9740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55b419787140 con 0x55b41977aeb0 2024-04-02T14:35:17.586 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.580+0000 7ffacdbf9740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55b41969b660 con 0x55b4197843b0 2024-04-02T14:35:17.586 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.580+0000 7ffacdbf9740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55b4197872c0 con 0x55b41977b7d0 2024-04-02T14:35:17.586 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.580+0000 7ffacdbf9740 10 client.0 _ll_register_callbacks cb 0x55b41969b050 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:35:17.586 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.580+0000 7ffac3e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b41977aeb0 0x55b419781430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:17.586 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.580+0000 7ffac8e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b4197843b0 0x55b419781eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:17.586 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.580+0000 7ffac3e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b41977aeb0 0x55b419781430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34230/0 (socket says 172.21.15.90:34230) 2024-04-02T14:35:17.586 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.580+0000 7ffac8e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b4197843b0 0x55b419781eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:32826/0 (socket says 172.21.15.90:32826) 2024-04-02T14:35:17.586 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.580+0000 7ffac3e00640 1 -- 172.21.15.90:0/1405373324 learned_addr learned my addr 172.21.15.90:0/1405373324 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:35:17.586 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.580+0000 7ffac3400640 1 --2- 172.21.15.90:0/1405373324 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b41977b7d0 0x55b419781970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:17.587 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.580+0000 7ffac3e00640 1 -- 172.21.15.90:0/1405373324 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b41977b7d0 msgr2=0x55b419781970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:35:17.587 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.580+0000 7ffac3e00640 1 --2- 172.21.15.90:0/1405373324 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b41977b7d0 0x55b419781970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:17.587 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.580+0000 7ffac3e00640 1 -- 172.21.15.90:0/1405373324 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b4197843b0 msgr2=0x55b419781eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:35:17.587 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.580+0000 7ffac3e00640 1 --2- 172.21.15.90:0/1405373324 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b4197843b0 0x55b419781eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:17.587 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.580+0000 7ffac3e00640 1 -- 172.21.15.90:0/1405373324 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b4196aae80 con 0x55b41977aeb0 2024-04-02T14:35:17.587 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.581+0000 7ffac3e00640 1 --2- 172.21.15.90:0/1405373324 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b41977aeb0 0x55b419781430 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7ffab400c7d0 tx=0x7ffab4098d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d999da43a12d535a server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:35:17.587 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.581+0000 7ffab2000640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-02T14:35:17.587 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.581+0000 7ffab2000640 1 -- 172.21.15.90:0/1405373324 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffab409ca40 con 0x55b41977aeb0 2024-04-02T14:35:17.587 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.581+0000 7ffab2000640 1 -- 172.21.15.90:0/1405373324 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7ffab4099750 con 0x55b41977aeb0 2024-04-02T14:35:17.587 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.581+0000 7ffacdbf9740 1 -- 172.21.15.90:0/1405373324 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffaac02cc40 con 0x55b41977aeb0 2024-04-02T14:35:17.587 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.581+0000 7ffab2000640 1 -- 172.21.15.90:0/1405373324 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7ffab409d7e0 con 0x55b41977aeb0 2024-04-02T14:35:17.588 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.581+0000 7ffacdbf9740 10 client.15432 fetch_fsmap learned FSMap version 27 2024-04-02T14:35:17.588 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.581+0000 7ffacdbf9740 1 -- 172.21.15.90:0/1405373324 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55b4193f4210 con 0x55b41977aeb0 2024-04-02T14:35:17.588 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.581+0000 7ffab2000640 1 -- 172.21.15.90:0/1405373324 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffaac02cc40 con 0x55b41977aeb0 2024-04-02T14:35:17.588 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.582+0000 7ffab2000640 1 -- 172.21.15.90:0/1405373324 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7ffab40cecd0 con 0x55b41977aeb0 2024-04-02T14:35:17.588 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.582+0000 7ffacdbf9740 10 client.15432 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:35:17.588 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.582+0000 7ffacdbf9740 10 client.15432 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-02T14:35:17.588 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.582+0000 7ffacdbf9740 20 client.15432 populate_metadata read hostname 'smithi090' 2024-04-02T14:35:17.588 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.582+0000 7ffacdbf9740 1 --2- 172.21.15.90:0/1405373324 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55b4197946d0 0x55b419796ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:17.588 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.582+0000 7ffacdbf9740 4 client.15432 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:35:17.588 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.582+0000 7ffacdbf9740 1 -- 172.21.15.90:0/1405373324 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b41968fcd0 con 0x55b4197946d0 2024-04-02T14:35:17.588 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.582+0000 7ffac3400640 1 --2- 172.21.15.90:0/1405373324 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55b4197946d0 0x55b419796ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:17.588 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.583+0000 7ffac3400640 1 --2- 172.21.15.90:0/1405373324 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55b4197946d0 0x55b419796ac0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f401e61d015127cf server_cookie=af4660f62df8aa16 in_seq=0 out_seq=0 2024-04-02T14:35:17.589 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.583+0000 7ffab2000640 10 client.15432 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:35:17.590 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.586+0000 7ffab2000640 1 -- 172.21.15.90:0/1405373324 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b41968fcd0 con 0x55b4197946d0 2024-04-02T14:35:17.590 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.586+0000 7ffab2000640 10 client.15432 handle_command_reply: tid=0 2024-04-02T14:35:17.590 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.586+0000 7ffacdbf9740 1 client.15432 shutdown 2024-04-02T14:35:17.590 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.586+0000 7ffacdbf9740 1 -- 172.21.15.90:0/1405373324 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b41977aeb0 msgr2=0x55b419781430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:35:17.590 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.586+0000 7ffacdbf9740 1 --2- 172.21.15.90:0/1405373324 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b41977aeb0 0x55b419781430 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7ffab400c7d0 tx=0x7ffab4098d20 comp rx=0 tx=0).stop 2024-04-02T14:35:17.590 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.587+0000 7ffacdbf9740 1 -- 172.21.15.90:0/1405373324 shutdown_connections 2024-04-02T14:35:17.590 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.587+0000 7ffacdbf9740 1 --2- 172.21.15.90:0/1405373324 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55b4197946d0 0x55b419796ac0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:17.590 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.587+0000 7ffacdbf9740 1 --2- 172.21.15.90:0/1405373324 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b4197843b0 0x55b419781eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:17.591 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.587+0000 7ffacdbf9740 1 --2- 172.21.15.90:0/1405373324 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b41977b7d0 0x55b419781970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:17.591 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.587+0000 7ffacdbf9740 1 --2- 172.21.15.90:0/1405373324 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b41977aeb0 0x55b419781430 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:17.591 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.587+0000 7ffab2000640 0 client.15432 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:35:17.591 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.587+0000 7ffacdbf9740 1 -- 172.21.15.90:0/1405373324 >> 172.21.15.90:0/1405373324 conn(0x55b4197755e0 msgr2=0x55b419777330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:35:17.591 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.587+0000 7ffacdbf9740 1 -- 172.21.15.90:0/1405373324 shutdown_connections 2024-04-02T14:35:17.591 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.587+0000 7ffacdbf9740 1 -- 172.21.15.90:0/1405373324 wait complete. 2024-04-02T14:35:17.591 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.587+0000 7ffacdbf9740 20 client.15432 trim_cache size 0 max 16384 2024-04-02T14:35:17.607 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.605+0000 7ffacdbf9740 1 Processor -- start 2024-04-02T14:35:17.607 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.605+0000 7ffacdbf9740 1 -- start start 2024-04-02T14:35:17.608 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.605+0000 7ffacdbf9740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b41977aeb0 0x55b419783af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:17.608 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.605+0000 7ffacdbf9740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b41977b7d0 0x55b419790000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:17.608 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.605+0000 7ffacdbf9740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b4197843b0 0x55b419781b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:17.608 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.605+0000 7ffacdbf9740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7ffac41af9d0 con 0x55b41977aeb0 2024-04-02T14:35:17.608 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.605+0000 7ffacdbf9740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7ffac41b1510 con 0x55b4197843b0 2024-04-02T14:35:17.608 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.605+0000 7ffacdbf9740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7ffac41b00a0 con 0x55b41977b7d0 2024-04-02T14:35:17.608 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.605+0000 7ffac3e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b41977aeb0 0x55b419783af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:17.608 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.605+0000 7ffac3e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b41977aeb0 0x55b419783af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34246/0 (socket says 172.21.15.90:34246) 2024-04-02T14:35:17.608 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.605+0000 7ffac3e00640 1 -- 172.21.15.90:0/1137563979 learned_addr learned my addr 172.21.15.90:0/1137563979 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:35:17.608 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.605+0000 7ffac8e00640 1 --2- 172.21.15.90:0/1137563979 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b4197843b0 0x55b419781b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:17.608 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.606+0000 7ffac3400640 1 --2- 172.21.15.90:0/1137563979 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b41977b7d0 0x55b419790000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:17.608 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.606+0000 7ffac3e00640 1 -- 172.21.15.90:0/1137563979 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b41977b7d0 msgr2=0x55b419790000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:35:17.608 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.606+0000 7ffac3e00640 1 --2- 172.21.15.90:0/1137563979 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b41977b7d0 0x55b419790000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:17.609 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.606+0000 7ffac3e00640 1 -- 172.21.15.90:0/1137563979 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b4197843b0 msgr2=0x55b419781b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:35:17.609 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.606+0000 7ffac3e00640 1 --2- 172.21.15.90:0/1137563979 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b4197843b0 0x55b419781b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:17.609 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.606+0000 7ffac3e00640 1 -- 172.21.15.90:0/1137563979 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b4194c0620 con 0x55b41977aeb0 2024-04-02T14:35:17.609 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.606+0000 7ffac8e00640 1 --2- 172.21.15.90:0/1137563979 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b4197843b0 0x55b419781b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:35:17.609 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.606+0000 7ffac3e00640 1 --2- 172.21.15.90:0/1137563979 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b41977aeb0 0x55b419783af0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7ffab4017470 tx=0x7ffab4017f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d7001e8335ca1af server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:35:17.609 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.606+0000 7ffab8c00640 1 -- 172.21.15.90:0/1137563979 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffab409ca40 con 0x55b41977aeb0 2024-04-02T14:35:17.610 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.606+0000 7ffab8c00640 1 -- 172.21.15.90:0/1137563979 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffab40ce8a0 con 0x55b41977aeb0 2024-04-02T14:35:17.610 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.606+0000 7ffab8c00640 1 -- 172.21.15.90:0/1137563979 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffab409ceb0 con 0x55b41977aeb0 2024-04-02T14:35:17.614 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.607+0000 7ffacdbf9740 1 -- 172.21.15.90:0/1137563979 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b41977aeb0 msgr2=0x55b419783af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:35:17.614 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.607+0000 7ffacdbf9740 1 --2- 172.21.15.90:0/1137563979 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b41977aeb0 0x55b419783af0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7ffab4017470 tx=0x7ffab4017f60 comp rx=0 tx=0).stop 2024-04-02T14:35:17.614 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.607+0000 7ffacdbf9740 1 -- 172.21.15.90:0/1137563979 shutdown_connections 2024-04-02T14:35:17.614 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.607+0000 7ffacdbf9740 1 --2- 172.21.15.90:0/1137563979 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b4197843b0 0x55b419781b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:17.614 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.607+0000 7ffacdbf9740 1 --2- 172.21.15.90:0/1137563979 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b41977b7d0 0x55b419790000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:17.614 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.607+0000 7ffacdbf9740 1 --2- 172.21.15.90:0/1137563979 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b41977aeb0 0x55b419783af0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:17.614 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.607+0000 7ffacdbf9740 1 -- 172.21.15.90:0/1137563979 >> 172.21.15.90:0/1137563979 conn(0x55b4197755e0 msgr2=0x55b4197813f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:35:17.614 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.607+0000 7ffacdbf9740 1 -- 172.21.15.90:0/1137563979 shutdown_connections 2024-04-02T14:35:17.614 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.607+0000 7ffacdbf9740 1 -- 172.21.15.90:0/1137563979 wait complete. 2024-04-02T14:35:17.615 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.608+0000 7ffacdbf9740 1 -- start start 2024-04-02T14:35:17.615 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.608+0000 7ffacdbf9740 1 Processor -- start 2024-04-02T14:35:17.615 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.608+0000 7ffacdbf9740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b41977aeb0 0x55b41978c9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:17.615 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.608+0000 7ffacdbf9740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b41977b7d0 0x55b41977def0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:17.615 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.608+0000 7ffacdbf9740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b4197843b0 0x55b41977e470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:17.615 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.608+0000 7ffacdbf9740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7ffac41b0e40 con 0x55b4197843b0 2024-04-02T14:35:17.615 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.608+0000 7ffacdbf9740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7ffac41b0770 con 0x55b41977b7d0 2024-04-02T14:35:17.615 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.608+0000 7ffacdbf9740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7ffac41ad9d0 con 0x55b41977aeb0 2024-04-02T14:35:17.615 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.608+0000 7ffacdbf9740 10 client.0 _ll_register_callbacks cb 0x7ffab40174d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:35:17.615 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.608+0000 7ffac3e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b41977aeb0 0x55b41978c9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:17.615 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.608+0000 7ffac8e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b4197843b0 0x55b41977e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:17.616 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.608+0000 7ffac3400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b41977b7d0 0x55b41977def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:17.616 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.608+0000 7ffac3e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b41977aeb0 0x55b41978c9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:37006/0 (socket says 172.21.15.90:37006) 2024-04-02T14:35:17.616 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.608+0000 7ffac3e00640 1 -- 172.21.15.90:0/2174721391 learned_addr learned my addr 172.21.15.90:0/2174721391 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:35:17.616 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.609+0000 7ffac3e00640 1 -- 172.21.15.90:0/2174721391 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b41977b7d0 msgr2=0x55b41977def0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:35:17.616 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.609+0000 7ffac3e00640 1 --2- 172.21.15.90:0/2174721391 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b41977b7d0 0x55b41977def0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:17.616 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.609+0000 7ffac3e00640 1 -- 172.21.15.90:0/2174721391 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b4197843b0 msgr2=0x55b41977e470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:35:17.616 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.609+0000 7ffac3e00640 1 --2- 172.21.15.90:0/2174721391 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b4197843b0 0x55b41977e470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:17.616 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.609+0000 7ffac3e00640 1 -- 172.21.15.90:0/2174721391 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b41977eca0 con 0x55b41977aeb0 2024-04-02T14:35:17.616 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.609+0000 7ffac3e00640 1 --2- 172.21.15.90:0/2174721391 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b41977aeb0 0x55b41978c9f0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7ffab4017580 tx=0x7ffab40d4e70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8da11e62e321ce70 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:35:17.616 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.609+0000 7ffab2000640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:35:17.616 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.609+0000 7ffab2000640 1 -- 172.21.15.90:0/2174721391 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffab40bb060 con 0x55b41977aeb0 2024-04-02T14:35:17.616 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.609+0000 7ffab2000640 1 -- 172.21.15.90:0/2174721391 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7ffab40cee40 con 0x55b41977aeb0 2024-04-02T14:35:17.617 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.609+0000 7ffacdbf9740 1 -- 172.21.15.90:0/2174721391 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffab40ce8a0 con 0x55b41977aeb0 2024-04-02T14:35:17.617 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.609+0000 7ffab2000640 1 -- 172.21.15.90:0/2174721391 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7ffab40a1490 con 0x55b41977aeb0 2024-04-02T14:35:17.617 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.610+0000 7ffacdbf9740 10 client.25519 fetch_fsmap learned FSMap version 27 2024-04-02T14:35:17.617 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.610+0000 7ffacdbf9740 1 -- 172.21.15.90:0/2174721391 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55b4197facb0 con 0x55b41977aeb0 2024-04-02T14:35:17.617 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.610+0000 7ffab2000640 1 -- 172.21.15.90:0/2174721391 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffab40ce8a0 con 0x55b41977aeb0 2024-04-02T14:35:17.617 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.610+0000 7ffab2000640 1 -- 172.21.15.90:0/2174721391 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7ffab4086210 con 0x55b41977aeb0 2024-04-02T14:35:17.617 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.610+0000 7ffacdbf9740 10 client.25519 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:35:17.617 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.610+0000 7ffacdbf9740 10 client.25519 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-02T14:35:17.617 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.610+0000 7ffacdbf9740 20 client.25519 populate_metadata read hostname 'smithi090' 2024-04-02T14:35:17.617 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.610+0000 7ffacdbf9740 1 --2- 172.21.15.90:0/2174721391 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55b4197fb130 0x55b4197fd520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:17.617 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.610+0000 7ffacdbf9740 4 client.25519 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-02T14:35:17.617 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.610+0000 7ffacdbf9740 1 -- 172.21.15.90:0/2174721391 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55b4196ed2b0 con 0x55b4197fb130 2024-04-02T14:35:17.618 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.610+0000 7ffac3400640 1 --2- 172.21.15.90:0/2174721391 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55b4197fb130 0x55b4197fd520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:17.618 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.611+0000 7ffac3400640 1 --2- 172.21.15.90:0/2174721391 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55b4197fb130 0x55b4197fd520 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cfedcd22cae6be server_cookie=664b6cb02a6d2c0c in_seq=0 out_seq=0 2024-04-02T14:35:17.618 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.611+0000 7ffab2000640 10 client.25519 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:35:17.686 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:35:17 smithi177 ceph-mon[36431]: pgmap v15: 97 pgs: 97 active+clean; 21 MiB data, 469 MiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 1.7 MiB/s wr, 22 op/s 2024-04-02T14:35:17.689 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:35:17 smithi090 ceph-mon[31674]: pgmap v15: 97 pgs: 97 active+clean; 21 MiB data, 469 MiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 1.7 MiB/s wr, 22 op/s 2024-04-02T14:35:17.861 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:35:17 smithi092 ceph-mon[36497]: pgmap v15: 97 pgs: 97 active+clean; 21 MiB data, 469 MiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 1.7 MiB/s wr, 22 op/s 2024-04-02T14:35:17.984 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.983+0000 7ffab2000640 1 -- 172.21.15.90:0/2174721391 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55b4196ed2b0 con 0x55b4197fb130 2024-04-02T14:35:17.984 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.983+0000 7ffab2000640 10 client.25519 handle_command_reply: tid=0 2024-04-02T14:35:17.984 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.983+0000 7ffacdbf9740 1 client.25519 shutdown 2024-04-02T14:35:17.985 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.983+0000 7ffacdbf9740 1 -- 172.21.15.90:0/2174721391 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b41977aeb0 msgr2=0x55b41978c9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:35:17.985 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.983+0000 7ffacdbf9740 1 --2- 172.21.15.90:0/2174721391 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b41977aeb0 0x55b41978c9f0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7ffab4017580 tx=0x7ffab40d4e70 comp rx=0 tx=0).stop 2024-04-02T14:35:17.985 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.983+0000 7ffacdbf9740 1 -- 172.21.15.90:0/2174721391 shutdown_connections 2024-04-02T14:35:17.985 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.983+0000 7ffacdbf9740 1 --2- 172.21.15.90:0/2174721391 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55b4197fb130 0x55b4197fd520 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:17.985 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.983+0000 7ffacdbf9740 1 --2- 172.21.15.90:0/2174721391 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b4197843b0 0x55b41977e470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:17.985 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.983+0000 7ffacdbf9740 1 --2- 172.21.15.90:0/2174721391 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b41977b7d0 0x55b41977def0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:17.985 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.983+0000 7ffacdbf9740 1 --2- 172.21.15.90:0/2174721391 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b41977aeb0 0x55b41978c9f0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:17.986 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.983+0000 7ffacdbf9740 1 -- 172.21.15.90:0/2174721391 >> 172.21.15.90:0/2174721391 conn(0x55b4197755e0 msgr2=0x55b419781c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:35:17.986 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.983+0000 7ffab2000640 0 client.25519 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:35:17.986 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.984+0000 7ffacdbf9740 1 -- 172.21.15.90:0/2174721391 shutdown_connections 2024-04-02T14:35:17.986 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.984+0000 7ffacdbf9740 1 -- 172.21.15.90:0/2174721391 wait complete. 2024-04-02T14:35:17.986 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.984+0000 7ffacdbf9740 20 client.25519 trim_cache size 0 max 16384 2024-04-02T14:35:17.986 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:{ 2024-04-02T14:35:17.986 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "message": "", 2024-04-02T14:35:17.987 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "return_code": 0 2024-04-02T14:35:17.987 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:} 2024-04-02T14:35:17.987 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.985+0000 7ffacae00640 1 -- 172.21.15.90:0/2161290921 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7ffaa80a3f60 msgr2=0x7ffaa80a6420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:35:17.987 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.985+0000 7ffacae00640 1 --2- 172.21.15.90:0/2161290921 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7ffaa80a3f60 0x7ffaa80a6420 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7ffabc00b890 tx=0x7ffabc00b2e0 comp rx=0 tx=0).stop 2024-04-02T14:35:17.988 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.986+0000 7ffacae00640 1 -- 172.21.15.90:0/2161290921 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffac4103f20 msgr2=0x7ffac419a5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:35:17.988 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.986+0000 7ffacae00640 1 --2- 172.21.15.90:0/2161290921 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffac4103f20 0x7ffac419a5e0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7ffab400eb00 tx=0x7ffab400c2d0 comp rx=0 tx=0).stop 2024-04-02T14:35:17.988 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.986+0000 7ffacae00640 1 -- 172.21.15.90:0/2161290921 shutdown_connections 2024-04-02T14:35:17.988 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.986+0000 7ffacae00640 1 --2- 172.21.15.90:0/2161290921 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7ffaa80a3f60 0x7ffaa80a6420 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:17.988 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.986+0000 7ffacae00640 1 --2- 172.21.15.90:0/2161290921 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffac419eea0 0x7ffac419f350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:17.988 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.986+0000 7ffacae00640 1 --2- 172.21.15.90:0/2161290921 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ffac4108ff0 0x7ffac419ab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:17.988 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.986+0000 7ffacae00640 1 --2- 172.21.15.90:0/2161290921 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffac4103f20 0x7ffac419a5e0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:17.988 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.986+0000 7ffacae00640 1 -- 172.21.15.90:0/2161290921 >> 172.21.15.90:0/2161290921 conn(0x7ffac40ff7c0 msgr2=0x7ffac4101a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:35:17.989 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.986+0000 7ffacae00640 1 -- 172.21.15.90:0/2161290921 shutdown_connections 2024-04-02T14:35:17.989 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:17.986+0000 7ffacae00640 1 -- 172.21.15.90:0/2161290921 wait complete. 2024-04-02T14:35:18.118 INFO:teuthology.orchestra.run.smithi090.stderr: * Initialized, logs are in /var/lib/pgsql/initdb_postgresql.log 2024-04-02T14:35:18.127 DEBUG:teuthology.orchestra.run.smithi090:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ls -lZaR /tmp/cephfs/postgres/' 2024-04-02T14:35:18.280 INFO:teuthology.orchestra.run.smithi090.stdout:/tmp/cephfs/postgres/: 2024-04-02T14:35:18.280 INFO:teuthology.orchestra.run.smithi090.stdout:total 0 2024-04-02T14:35:18.280 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x. 3 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Apr 2 14:35 . 2024-04-02T14:35:18.280 INFO:teuthology.orchestra.run.smithi090.stdout:drwxrwxrwt. 3 root root system_u:object_r:postgresql_db_t:s0 1 Apr 2 14:35 .. 2024-04-02T14:35:18.281 INFO:teuthology.orchestra.run.smithi090.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Apr 2 14:35 data 2024-04-02T14:35:18.281 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-02T14:35:18.281 INFO:teuthology.orchestra.run.smithi090.stdout:/tmp/cephfs/postgres/data: 2024-04-02T14:35:18.281 INFO:teuthology.orchestra.run.smithi090.stdout:total 35 2024-04-02T14:35:18.281 INFO:teuthology.orchestra.run.smithi090.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Apr 2 14:35 . 2024-04-02T14:35:18.281 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x. 3 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Apr 2 14:35 .. 2024-04-02T14:35:18.281 INFO:teuthology.orchestra.run.smithi090.stdout:drwx------. 5 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Apr 2 14:35 base 2024-04-02T14:35:18.281 INFO:teuthology.orchestra.run.smithi090.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 57 Apr 2 14:35 global 2024-04-02T14:35:18.281 INFO:teuthology.orchestra.run.smithi090.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 log 2024-04-02T14:35:18.281 INFO:teuthology.orchestra.run.smithi090.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 pg_commit_ts 2024-04-02T14:35:18.281 INFO:teuthology.orchestra.run.smithi090.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 pg_dynshmem 2024-04-02T14:35:18.281 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 4516 Apr 2 14:35 pg_hba.conf 2024-04-02T14:35:18.281 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 1636 Apr 2 14:35 pg_ident.conf 2024-04-02T14:35:18.281 INFO:teuthology.orchestra.run.smithi090.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Apr 2 14:35 pg_logical 2024-04-02T14:35:18.282 INFO:teuthology.orchestra.run.smithi090.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 2 Apr 2 14:35 pg_multixact 2024-04-02T14:35:18.282 INFO:teuthology.orchestra.run.smithi090.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 pg_notify 2024-04-02T14:35:18.282 INFO:teuthology.orchestra.run.smithi090.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 pg_replslot 2024-04-02T14:35:18.282 INFO:teuthology.orchestra.run.smithi090.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 pg_serial 2024-04-02T14:35:18.282 INFO:teuthology.orchestra.run.smithi090.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 pg_snapshots 2024-04-02T14:35:18.282 INFO:teuthology.orchestra.run.smithi090.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 pg_stat 2024-04-02T14:35:18.282 INFO:teuthology.orchestra.run.smithi090.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 pg_stat_tmp 2024-04-02T14:35:18.282 INFO:teuthology.orchestra.run.smithi090.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Apr 2 14:35 pg_subtrans 2024-04-02T14:35:18.282 INFO:teuthology.orchestra.run.smithi090.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 pg_tblspc 2024-04-02T14:35:18.282 INFO:teuthology.orchestra.run.smithi090.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 pg_twophase 2024-04-02T14:35:18.282 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Apr 2 14:35 PG_VERSION 2024-04-02T14:35:18.282 INFO:teuthology.orchestra.run.smithi090.stdout:drwx------. 3 postgres postgres system_u:object_r:postgresql_db_t:s0 2 Apr 2 14:35 pg_wal 2024-04-02T14:35:18.282 INFO:teuthology.orchestra.run.smithi090.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Apr 2 14:35 pg_xact 2024-04-02T14:35:18.282 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 88 Apr 2 14:35 postgresql.auto.conf 2024-04-02T14:35:18.283 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 28082 Apr 2 14:35 postgresql.conf 2024-04-02T14:35:18.283 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-02T14:35:18.283 INFO:teuthology.orchestra.run.smithi090.stdout:/tmp/cephfs/postgres/data/base: 2024-04-02T14:35:18.283 INFO:teuthology.orchestra.run.smithi090.stdout:total 0 2024-04-02T14:35:18.283 INFO:teuthology.orchestra.run.smithi090.stdout:drwx------. 5 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Apr 2 14:35 . 2024-04-02T14:35:18.283 INFO:teuthology.orchestra.run.smithi090.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Apr 2 14:35 .. 2024-04-02T14:35:18.283 INFO:teuthology.orchestra.run.smithi090.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 289 Apr 2 14:35 1 2024-04-02T14:35:18.283 INFO:teuthology.orchestra.run.smithi090.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 289 Apr 2 14:35 13493 2024-04-02T14:35:18.283 INFO:teuthology.orchestra.run.smithi090.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 289 Apr 2 14:35 13494 2024-04-02T14:35:18.283 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-02T14:35:18.283 INFO:teuthology.orchestra.run.smithi090.stdout:/tmp/cephfs/postgres/data/base/1: 2024-04-02T14:35:18.283 INFO:teuthology.orchestra.run.smithi090.stdout:total 7777 2024-04-02T14:35:18.283 INFO:teuthology.orchestra.run.smithi090.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 289 Apr 2 14:35 . 2024-04-02T14:35:18.283 INFO:teuthology.orchestra.run.smithi090.stdout:drwx------. 5 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Apr 2 14:35 .. 2024-04-02T14:35:18.284 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 112 2024-04-02T14:35:18.284 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 113 2024-04-02T14:35:18.284 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 81920 Apr 2 14:35 1247 2024-04-02T14:35:18.284 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 1247_fsm 2024-04-02T14:35:18.284 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 1247_vm 2024-04-02T14:35:18.284 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 434176 Apr 2 14:35 1249 2024-04-02T14:35:18.284 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 1249_fsm 2024-04-02T14:35:18.284 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 1249_vm 2024-04-02T14:35:18.284 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 663552 Apr 2 14:35 1255 2024-04-02T14:35:18.284 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 1255_fsm 2024-04-02T14:35:18.284 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 1255_vm 2024-04-02T14:35:18.284 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 106496 Apr 2 14:35 1259 2024-04-02T14:35:18.284 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 1259_fsm 2024-04-02T14:35:18.284 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 1259_vm 2024-04-02T14:35:18.284 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 65536 Apr 2 14:35 13344 2024-04-02T14:35:18.284 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 13344_fsm 2024-04-02T14:35:18.285 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 13344_vm 2024-04-02T14:35:18.285 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 13346 2024-04-02T14:35:18.285 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 13348 2024-04-02T14:35:18.285 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 13349 2024-04-02T14:35:18.285 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 13349_fsm 2024-04-02T14:35:18.285 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 13349_vm 2024-04-02T14:35:18.285 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 13351 2024-04-02T14:35:18.285 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 13353 2024-04-02T14:35:18.285 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 13354 2024-04-02T14:35:18.285 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 13354_fsm 2024-04-02T14:35:18.285 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 13354_vm 2024-04-02T14:35:18.285 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 13356 2024-04-02T14:35:18.285 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 13358 2024-04-02T14:35:18.285 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 13359 2024-04-02T14:35:18.285 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 13359_fsm 2024-04-02T14:35:18.285 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 13359_vm 2024-04-02T14:35:18.286 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 13361 2024-04-02T14:35:18.286 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 13363 2024-04-02T14:35:18.286 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 1417 2024-04-02T14:35:18.286 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 1418 2024-04-02T14:35:18.286 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 174 2024-04-02T14:35:18.286 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 175 2024-04-02T14:35:18.286 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2187 2024-04-02T14:35:18.286 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 2224 2024-04-02T14:35:18.286 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 2328 2024-04-02T14:35:18.286 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 2336 2024-04-02T14:35:18.286 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2337 2024-04-02T14:35:18.286 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2579 2024-04-02T14:35:18.286 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2600 2024-04-02T14:35:18.286 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 2600_fsm 2024-04-02T14:35:18.286 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2600_vm 2024-04-02T14:35:18.286 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2601 2024-04-02T14:35:18.287 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 2601_fsm 2024-04-02T14:35:18.287 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2601_vm 2024-04-02T14:35:18.287 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Apr 2 14:35 2602 2024-04-02T14:35:18.287 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 2602_fsm 2024-04-02T14:35:18.287 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2602_vm 2024-04-02T14:35:18.287 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 2 14:35 2603 2024-04-02T14:35:18.287 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 2603_fsm 2024-04-02T14:35:18.287 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2603_vm 2024-04-02T14:35:18.287 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 2604 2024-04-02T14:35:18.287 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2605 2024-04-02T14:35:18.287 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 2605_fsm 2024-04-02T14:35:18.287 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2605_vm 2024-04-02T14:35:18.287 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2606 2024-04-02T14:35:18.288 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 2606_fsm 2024-04-02T14:35:18.288 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2606_vm 2024-04-02T14:35:18.288 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2607 2024-04-02T14:35:18.288 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 2607_fsm 2024-04-02T14:35:18.288 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2607_vm 2024-04-02T14:35:18.288 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 466944 Apr 2 14:35 2608 2024-04-02T14:35:18.288 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 2608_fsm 2024-04-02T14:35:18.288 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2608_vm 2024-04-02T14:35:18.288 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 344064 Apr 2 14:35 2609 2024-04-02T14:35:18.288 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 2609_fsm 2024-04-02T14:35:18.288 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2609_vm 2024-04-02T14:35:18.288 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 2 14:35 2610 2024-04-02T14:35:18.288 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 2610_fsm 2024-04-02T14:35:18.288 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2610_vm 2024-04-02T14:35:18.288 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 2611 2024-04-02T14:35:18.289 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2612 2024-04-02T14:35:18.289 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 2612_fsm 2024-04-02T14:35:18.289 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2612_vm 2024-04-02T14:35:18.289 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 2613 2024-04-02T14:35:18.289 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2615 2024-04-02T14:35:18.289 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 2615_fsm 2024-04-02T14:35:18.289 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2615_vm 2024-04-02T14:35:18.289 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2616 2024-04-02T14:35:18.289 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 2616_fsm 2024-04-02T14:35:18.289 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2616_vm 2024-04-02T14:35:18.289 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 114688 Apr 2 14:35 2617 2024-04-02T14:35:18.289 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 2617_fsm 2024-04-02T14:35:18.289 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2617_vm 2024-04-02T14:35:18.289 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 98304 Apr 2 14:35 2618 2024-04-02T14:35:18.289 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 2618_fsm 2024-04-02T14:35:18.290 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2618_vm 2024-04-02T14:35:18.290 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 147456 Apr 2 14:35 2619 2024-04-02T14:35:18.290 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 2619_fsm 2024-04-02T14:35:18.290 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2619_vm 2024-04-02T14:35:18.290 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 2620 2024-04-02T14:35:18.290 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2650 2024-04-02T14:35:18.290 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2651 2024-04-02T14:35:18.290 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2652 2024-04-02T14:35:18.290 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 2 14:35 2653 2024-04-02T14:35:18.290 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 2 14:35 2654 2024-04-02T14:35:18.290 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 2 14:35 2655 2024-04-02T14:35:18.290 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2656 2024-04-02T14:35:18.290 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2657 2024-04-02T14:35:18.290 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 122880 Apr 2 14:35 2658 2024-04-02T14:35:18.290 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 81920 Apr 2 14:35 2659 2024-04-02T14:35:18.291 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2660 2024-04-02T14:35:18.291 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2661 2024-04-02T14:35:18.291 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 2 14:35 2662 2024-04-02T14:35:18.291 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Apr 2 14:35 2663 2024-04-02T14:35:18.291 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2664 2024-04-02T14:35:18.291 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2665 2024-04-02T14:35:18.291 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2666 2024-04-02T14:35:18.291 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2667 2024-04-02T14:35:18.291 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2668 2024-04-02T14:35:18.291 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2669 2024-04-02T14:35:18.291 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2670 2024-04-02T14:35:18.291 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 278528 Apr 2 14:35 2673 2024-04-02T14:35:18.291 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 352256 Apr 2 14:35 2674 2024-04-02T14:35:18.291 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 196608 Apr 2 14:35 2675 2024-04-02T14:35:18.291 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2678 2024-04-02T14:35:18.291 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2679 2024-04-02T14:35:18.292 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2680 2024-04-02T14:35:18.292 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2681 2024-04-02T14:35:18.292 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2682 2024-04-02T14:35:18.292 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2683 2024-04-02T14:35:18.292 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2684 2024-04-02T14:35:18.292 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2685 2024-04-02T14:35:18.292 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2686 2024-04-02T14:35:18.292 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2687 2024-04-02T14:35:18.292 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 2 14:35 2688 2024-04-02T14:35:18.292 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Apr 2 14:35 2689 2024-04-02T14:35:18.292 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 90112 Apr 2 14:35 2690 2024-04-02T14:35:18.292 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 253952 Apr 2 14:35 2691 2024-04-02T14:35:18.292 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2692 2024-04-02T14:35:18.292 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2693 2024-04-02T14:35:18.292 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2696 2024-04-02T14:35:18.293 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2699 2024-04-02T14:35:18.293 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2701 2024-04-02T14:35:18.293 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2702 2024-04-02T14:35:18.293 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 2 14:35 2703 2024-04-02T14:35:18.293 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 2 14:35 2704 2024-04-02T14:35:18.293 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2753 2024-04-02T14:35:18.293 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 2753_fsm 2024-04-02T14:35:18.293 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2753_vm 2024-04-02T14:35:18.293 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2754 2024-04-02T14:35:18.293 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2755 2024-04-02T14:35:18.293 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 2 14:35 2756 2024-04-02T14:35:18.293 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 2 14:35 2757 2024-04-02T14:35:18.293 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 2830 2024-04-02T14:35:18.293 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2831 2024-04-02T14:35:18.293 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 2832 2024-04-02T14:35:18.294 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2833 2024-04-02T14:35:18.294 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 2834 2024-04-02T14:35:18.294 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2835 2024-04-02T14:35:18.294 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 2836 2024-04-02T14:35:18.294 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2837 2024-04-02T14:35:18.294 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 491520 Apr 2 14:35 2838 2024-04-02T14:35:18.294 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 2838_fsm 2024-04-02T14:35:18.294 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2838_vm 2024-04-02T14:35:18.294 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2839 2024-04-02T14:35:18.294 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 2840 2024-04-02T14:35:18.294 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 2840_fsm 2024-04-02T14:35:18.294 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2840_vm 2024-04-02T14:35:18.294 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2841 2024-04-02T14:35:18.295 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 2995 2024-04-02T14:35:18.295 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2996 2024-04-02T14:35:18.295 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3079 2024-04-02T14:35:18.295 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 3079_fsm 2024-04-02T14:35:18.295 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3079_vm 2024-04-02T14:35:18.295 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 3080 2024-04-02T14:35:18.295 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 3081 2024-04-02T14:35:18.295 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 2 14:35 3085 2024-04-02T14:35:18.296 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 3118 2024-04-02T14:35:18.296 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3119 2024-04-02T14:35:18.296 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 57344 Apr 2 14:35 3164 2024-04-02T14:35:18.296 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 3256 2024-04-02T14:35:18.296 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3257 2024-04-02T14:35:18.296 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3258 2024-04-02T14:35:18.296 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 3350 2024-04-02T14:35:18.296 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3351 2024-04-02T14:35:18.296 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3379 2024-04-02T14:35:18.296 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3380 2024-04-02T14:35:18.296 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 3381 2024-04-02T14:35:18.296 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 3394 2024-04-02T14:35:18.296 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 3394_fsm 2024-04-02T14:35:18.296 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3394_vm 2024-04-02T14:35:18.296 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 3395 2024-04-02T14:35:18.296 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 3429 2024-04-02T14:35:18.296 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 3430 2024-04-02T14:35:18.297 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3431 2024-04-02T14:35:18.297 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3433 2024-04-02T14:35:18.297 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 3439 2024-04-02T14:35:18.297 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3440 2024-04-02T14:35:18.297 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 2 14:35 3455 2024-04-02T14:35:18.297 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 221184 Apr 2 14:35 3456 2024-04-02T14:35:18.297 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 3456_fsm 2024-04-02T14:35:18.297 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3456_vm 2024-04-02T14:35:18.297 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 3466 2024-04-02T14:35:18.297 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3467 2024-04-02T14:35:18.297 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3468 2024-04-02T14:35:18.297 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 3501 2024-04-02T14:35:18.297 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3502 2024-04-02T14:35:18.297 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3503 2024-04-02T14:35:18.297 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3534 2024-04-02T14:35:18.297 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3541 2024-04-02T14:35:18.298 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 3541_fsm 2024-04-02T14:35:18.298 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3541_vm 2024-04-02T14:35:18.298 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 3542 2024-04-02T14:35:18.298 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3574 2024-04-02T14:35:18.298 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3575 2024-04-02T14:35:18.298 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 3576 2024-04-02T14:35:18.298 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 3596 2024-04-02T14:35:18.298 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3597 2024-04-02T14:35:18.298 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 3598 2024-04-02T14:35:18.298 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3599 2024-04-02T14:35:18.298 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3600 2024-04-02T14:35:18.298 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 3600_fsm 2024-04-02T14:35:18.298 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3600_vm 2024-04-02T14:35:18.298 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3601 2024-04-02T14:35:18.299 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 3601_fsm 2024-04-02T14:35:18.299 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3601_vm 2024-04-02T14:35:18.299 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3602 2024-04-02T14:35:18.299 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 3602_fsm 2024-04-02T14:35:18.299 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3602_vm 2024-04-02T14:35:18.299 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 3603 2024-04-02T14:35:18.299 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 3603_fsm 2024-04-02T14:35:18.299 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3603_vm 2024-04-02T14:35:18.299 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 3604 2024-04-02T14:35:18.299 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 3605 2024-04-02T14:35:18.299 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 3606 2024-04-02T14:35:18.299 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 3607 2024-04-02T14:35:18.299 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 3608 2024-04-02T14:35:18.299 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 2 14:35 3609 2024-04-02T14:35:18.299 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 3712 2024-04-02T14:35:18.299 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3764 2024-04-02T14:35:18.300 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 3764_fsm 2024-04-02T14:35:18.300 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3764_vm 2024-04-02T14:35:18.300 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 3766 2024-04-02T14:35:18.300 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 3767 2024-04-02T14:35:18.300 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3997 2024-04-02T14:35:18.300 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 4143 2024-04-02T14:35:18.300 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 4144 2024-04-02T14:35:18.300 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 4145 2024-04-02T14:35:18.300 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 4146 2024-04-02T14:35:18.300 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 4147 2024-04-02T14:35:18.301 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 4148 2024-04-02T14:35:18.301 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 4149 2024-04-02T14:35:18.301 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 4150 2024-04-02T14:35:18.301 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 4151 2024-04-02T14:35:18.301 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 4152 2024-04-02T14:35:18.301 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 4153 2024-04-02T14:35:18.301 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 4154 2024-04-02T14:35:18.301 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 4155 2024-04-02T14:35:18.301 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 4156 2024-04-02T14:35:18.301 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 4157 2024-04-02T14:35:18.301 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 4158 2024-04-02T14:35:18.302 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 4159 2024-04-02T14:35:18.302 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 4160 2024-04-02T14:35:18.302 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 4161 2024-04-02T14:35:18.302 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 4162 2024-04-02T14:35:18.302 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 4163 2024-04-02T14:35:18.302 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 4164 2024-04-02T14:35:18.302 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 4165 2024-04-02T14:35:18.303 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 4166 2024-04-02T14:35:18.303 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 4167 2024-04-02T14:35:18.303 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 4168 2024-04-02T14:35:18.303 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 4169 2024-04-02T14:35:18.303 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 4170 2024-04-02T14:35:18.303 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 4171 2024-04-02T14:35:18.303 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 4172 2024-04-02T14:35:18.303 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 4173 2024-04-02T14:35:18.303 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 4174 2024-04-02T14:35:18.303 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 5002 2024-04-02T14:35:18.303 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 548 2024-04-02T14:35:18.303 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 549 2024-04-02T14:35:18.304 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 6102 2024-04-02T14:35:18.304 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 6104 2024-04-02T14:35:18.304 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 6106 2024-04-02T14:35:18.304 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 6110 2024-04-02T14:35:18.304 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 6111 2024-04-02T14:35:18.304 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 6112 2024-04-02T14:35:18.304 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 6113 2024-04-02T14:35:18.304 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 6117 2024-04-02T14:35:18.304 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 826 2024-04-02T14:35:18.304 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 827 2024-04-02T14:35:18.304 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 828 2024-04-02T14:35:18.304 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 512 Apr 2 14:35 pg_filenode.map 2024-04-02T14:35:18.304 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Apr 2 14:35 PG_VERSION 2024-04-02T14:35:18.304 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-02T14:35:18.304 INFO:teuthology.orchestra.run.smithi090.stdout:/tmp/cephfs/postgres/data/base/13493: 2024-04-02T14:35:18.305 INFO:teuthology.orchestra.run.smithi090.stdout:total 7777 2024-04-02T14:35:18.305 INFO:teuthology.orchestra.run.smithi090.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 289 Apr 2 14:35 . 2024-04-02T14:35:18.305 INFO:teuthology.orchestra.run.smithi090.stdout:drwx------. 5 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Apr 2 14:35 .. 2024-04-02T14:35:18.305 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 112 2024-04-02T14:35:18.305 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 113 2024-04-02T14:35:18.305 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 81920 Apr 2 14:35 1247 2024-04-02T14:35:18.305 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 1247_fsm 2024-04-02T14:35:18.305 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 1247_vm 2024-04-02T14:35:18.305 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 434176 Apr 2 14:35 1249 2024-04-02T14:35:18.305 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 1249_fsm 2024-04-02T14:35:18.305 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 1249_vm 2024-04-02T14:35:18.305 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 663552 Apr 2 14:35 1255 2024-04-02T14:35:18.305 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 1255_fsm 2024-04-02T14:35:18.305 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 1255_vm 2024-04-02T14:35:18.306 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 106496 Apr 2 14:35 1259 2024-04-02T14:35:18.306 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 1259_fsm 2024-04-02T14:35:18.306 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 1259_vm 2024-04-02T14:35:18.306 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 65536 Apr 2 14:35 13344 2024-04-02T14:35:18.306 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 13344_fsm 2024-04-02T14:35:18.306 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 13344_vm 2024-04-02T14:35:18.306 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 13346 2024-04-02T14:35:18.306 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 13348 2024-04-02T14:35:18.306 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 13349 2024-04-02T14:35:18.306 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 13349_fsm 2024-04-02T14:35:18.306 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 13349_vm 2024-04-02T14:35:18.306 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 13351 2024-04-02T14:35:18.306 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 13353 2024-04-02T14:35:18.307 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 13354 2024-04-02T14:35:18.307 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 13354_fsm 2024-04-02T14:35:18.307 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 13354_vm 2024-04-02T14:35:18.307 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 13356 2024-04-02T14:35:18.307 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 13358 2024-04-02T14:35:18.307 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 13359 2024-04-02T14:35:18.307 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 13359_fsm 2024-04-02T14:35:18.307 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 13359_vm 2024-04-02T14:35:18.307 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 13361 2024-04-02T14:35:18.307 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 13363 2024-04-02T14:35:18.307 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 1417 2024-04-02T14:35:18.307 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 1418 2024-04-02T14:35:18.307 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 174 2024-04-02T14:35:18.307 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 175 2024-04-02T14:35:18.308 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2187 2024-04-02T14:35:18.308 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 2224 2024-04-02T14:35:18.308 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 2328 2024-04-02T14:35:18.308 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 2336 2024-04-02T14:35:18.308 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2337 2024-04-02T14:35:18.308 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2579 2024-04-02T14:35:18.308 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2600 2024-04-02T14:35:18.308 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 2600_fsm 2024-04-02T14:35:18.308 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2600_vm 2024-04-02T14:35:18.308 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2601 2024-04-02T14:35:18.308 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 2601_fsm 2024-04-02T14:35:18.308 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2601_vm 2024-04-02T14:35:18.308 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Apr 2 14:35 2602 2024-04-02T14:35:18.308 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 2602_fsm 2024-04-02T14:35:18.308 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2602_vm 2024-04-02T14:35:18.308 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 2 14:35 2603 2024-04-02T14:35:18.309 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 2603_fsm 2024-04-02T14:35:18.309 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2603_vm 2024-04-02T14:35:18.309 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 2604 2024-04-02T14:35:18.309 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2605 2024-04-02T14:35:18.309 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 2605_fsm 2024-04-02T14:35:18.309 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2605_vm 2024-04-02T14:35:18.309 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2606 2024-04-02T14:35:18.309 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 2606_fsm 2024-04-02T14:35:18.309 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2606_vm 2024-04-02T14:35:18.309 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2607 2024-04-02T14:35:18.309 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 2607_fsm 2024-04-02T14:35:18.309 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2607_vm 2024-04-02T14:35:18.309 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 466944 Apr 2 14:35 2608 2024-04-02T14:35:18.309 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 2608_fsm 2024-04-02T14:35:18.310 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2608_vm 2024-04-02T14:35:18.310 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 344064 Apr 2 14:35 2609 2024-04-02T14:35:18.310 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 2609_fsm 2024-04-02T14:35:18.310 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2609_vm 2024-04-02T14:35:18.310 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 2 14:35 2610 2024-04-02T14:35:18.310 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 2610_fsm 2024-04-02T14:35:18.310 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2610_vm 2024-04-02T14:35:18.310 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 2611 2024-04-02T14:35:18.310 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2612 2024-04-02T14:35:18.310 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 2612_fsm 2024-04-02T14:35:18.310 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2612_vm 2024-04-02T14:35:18.310 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 2613 2024-04-02T14:35:18.310 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2615 2024-04-02T14:35:18.310 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 2615_fsm 2024-04-02T14:35:18.311 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2615_vm 2024-04-02T14:35:18.311 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2616 2024-04-02T14:35:18.311 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 2616_fsm 2024-04-02T14:35:18.311 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2616_vm 2024-04-02T14:35:18.311 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 114688 Apr 2 14:35 2617 2024-04-02T14:35:18.311 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 2617_fsm 2024-04-02T14:35:18.311 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2617_vm 2024-04-02T14:35:18.311 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 98304 Apr 2 14:35 2618 2024-04-02T14:35:18.311 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 2618_fsm 2024-04-02T14:35:18.311 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2618_vm 2024-04-02T14:35:18.311 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 147456 Apr 2 14:35 2619 2024-04-02T14:35:18.311 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 2619_fsm 2024-04-02T14:35:18.311 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2619_vm 2024-04-02T14:35:18.311 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 2620 2024-04-02T14:35:18.312 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2650 2024-04-02T14:35:18.312 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2651 2024-04-02T14:35:18.312 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2652 2024-04-02T14:35:18.312 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 2 14:35 2653 2024-04-02T14:35:18.312 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 2 14:35 2654 2024-04-02T14:35:18.312 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 2 14:35 2655 2024-04-02T14:35:18.312 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2656 2024-04-02T14:35:18.312 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2657 2024-04-02T14:35:18.312 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 122880 Apr 2 14:35 2658 2024-04-02T14:35:18.312 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 81920 Apr 2 14:35 2659 2024-04-02T14:35:18.312 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2660 2024-04-02T14:35:18.312 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2661 2024-04-02T14:35:18.312 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 2 14:35 2662 2024-04-02T14:35:18.312 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Apr 2 14:35 2663 2024-04-02T14:35:18.313 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2664 2024-04-02T14:35:18.313 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2665 2024-04-02T14:35:18.313 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2666 2024-04-02T14:35:18.313 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2667 2024-04-02T14:35:18.313 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2668 2024-04-02T14:35:18.313 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2669 2024-04-02T14:35:18.313 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2670 2024-04-02T14:35:18.313 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 278528 Apr 2 14:35 2673 2024-04-02T14:35:18.313 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 352256 Apr 2 14:35 2674 2024-04-02T14:35:18.313 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 196608 Apr 2 14:35 2675 2024-04-02T14:35:18.313 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2678 2024-04-02T14:35:18.313 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2679 2024-04-02T14:35:18.313 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2680 2024-04-02T14:35:18.314 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2681 2024-04-02T14:35:18.314 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2682 2024-04-02T14:35:18.314 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2683 2024-04-02T14:35:18.314 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2684 2024-04-02T14:35:18.314 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2685 2024-04-02T14:35:18.314 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2686 2024-04-02T14:35:18.314 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2687 2024-04-02T14:35:18.314 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 2 14:35 2688 2024-04-02T14:35:18.314 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Apr 2 14:35 2689 2024-04-02T14:35:18.314 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 90112 Apr 2 14:35 2690 2024-04-02T14:35:18.314 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 253952 Apr 2 14:35 2691 2024-04-02T14:35:18.314 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2692 2024-04-02T14:35:18.314 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2693 2024-04-02T14:35:18.314 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2696 2024-04-02T14:35:18.314 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2699 2024-04-02T14:35:18.315 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2701 2024-04-02T14:35:18.315 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2702 2024-04-02T14:35:18.315 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 2 14:35 2703 2024-04-02T14:35:18.315 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 2 14:35 2704 2024-04-02T14:35:18.315 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2753 2024-04-02T14:35:18.315 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 2753_fsm 2024-04-02T14:35:18.315 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2753_vm 2024-04-02T14:35:18.315 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2754 2024-04-02T14:35:18.315 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2755 2024-04-02T14:35:18.315 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 2 14:35 2756 2024-04-02T14:35:18.315 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 2 14:35 2757 2024-04-02T14:35:18.315 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 2830 2024-04-02T14:35:18.315 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2831 2024-04-02T14:35:18.315 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 2832 2024-04-02T14:35:18.316 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2833 2024-04-02T14:35:18.316 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 2834 2024-04-02T14:35:18.316 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2835 2024-04-02T14:35:18.316 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 2836 2024-04-02T14:35:18.316 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2837 2024-04-02T14:35:18.316 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 491520 Apr 2 14:35 2838 2024-04-02T14:35:18.316 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 2838_fsm 2024-04-02T14:35:18.316 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2838_vm 2024-04-02T14:35:18.316 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2839 2024-04-02T14:35:18.316 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 2840 2024-04-02T14:35:18.316 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 2840_fsm 2024-04-02T14:35:18.316 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2840_vm 2024-04-02T14:35:18.316 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2841 2024-04-02T14:35:18.316 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 2995 2024-04-02T14:35:18.316 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2996 2024-04-02T14:35:18.317 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3079 2024-04-02T14:35:18.317 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 3079_fsm 2024-04-02T14:35:18.317 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3079_vm 2024-04-02T14:35:18.317 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 3080 2024-04-02T14:35:18.317 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 3081 2024-04-02T14:35:18.317 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 2 14:35 3085 2024-04-02T14:35:18.317 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 3118 2024-04-02T14:35:18.317 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3119 2024-04-02T14:35:18.317 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 57344 Apr 2 14:35 3164 2024-04-02T14:35:18.317 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 3256 2024-04-02T14:35:18.317 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3257 2024-04-02T14:35:18.317 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3258 2024-04-02T14:35:18.317 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 3350 2024-04-02T14:35:18.317 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3351 2024-04-02T14:35:18.318 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3379 2024-04-02T14:35:18.318 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3380 2024-04-02T14:35:18.318 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 3381 2024-04-02T14:35:18.318 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 3394 2024-04-02T14:35:18.318 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 3394_fsm 2024-04-02T14:35:18.318 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3394_vm 2024-04-02T14:35:18.318 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 3395 2024-04-02T14:35:18.318 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 3429 2024-04-02T14:35:18.318 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 3430 2024-04-02T14:35:18.318 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3431 2024-04-02T14:35:18.318 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3433 2024-04-02T14:35:18.318 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 3439 2024-04-02T14:35:18.318 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3440 2024-04-02T14:35:18.318 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 2 14:35 3455 2024-04-02T14:35:18.319 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 221184 Apr 2 14:35 3456 2024-04-02T14:35:18.319 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 3456_fsm 2024-04-02T14:35:18.319 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3456_vm 2024-04-02T14:35:18.319 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 3466 2024-04-02T14:35:18.319 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3467 2024-04-02T14:35:18.319 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3468 2024-04-02T14:35:18.319 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 3501 2024-04-02T14:35:18.319 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3502 2024-04-02T14:35:18.319 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3503 2024-04-02T14:35:18.319 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3534 2024-04-02T14:35:18.319 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3541 2024-04-02T14:35:18.319 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 3541_fsm 2024-04-02T14:35:18.319 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3541_vm 2024-04-02T14:35:18.319 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 3542 2024-04-02T14:35:18.319 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3574 2024-04-02T14:35:18.319 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3575 2024-04-02T14:35:18.320 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 3576 2024-04-02T14:35:18.320 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 3596 2024-04-02T14:35:18.320 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3597 2024-04-02T14:35:18.320 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 3598 2024-04-02T14:35:18.320 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3599 2024-04-02T14:35:18.320 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3600 2024-04-02T14:35:18.320 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 3600_fsm 2024-04-02T14:35:18.320 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3600_vm 2024-04-02T14:35:18.320 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3601 2024-04-02T14:35:18.320 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 3601_fsm 2024-04-02T14:35:18.320 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3601_vm 2024-04-02T14:35:18.320 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3602 2024-04-02T14:35:18.320 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 3602_fsm 2024-04-02T14:35:18.321 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3602_vm 2024-04-02T14:35:18.321 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 3603 2024-04-02T14:35:18.321 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 3603_fsm 2024-04-02T14:35:18.321 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3603_vm 2024-04-02T14:35:18.321 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 3604 2024-04-02T14:35:18.321 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 3605 2024-04-02T14:35:18.321 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 3606 2024-04-02T14:35:18.321 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 3607 2024-04-02T14:35:18.321 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 3608 2024-04-02T14:35:18.321 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 2 14:35 3609 2024-04-02T14:35:18.321 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 3712 2024-04-02T14:35:18.321 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3764 2024-04-02T14:35:18.321 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 3764_fsm 2024-04-02T14:35:18.321 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3764_vm 2024-04-02T14:35:18.321 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 3766 2024-04-02T14:35:18.322 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 3767 2024-04-02T14:35:18.322 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3997 2024-04-02T14:35:18.322 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 4143 2024-04-02T14:35:18.322 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 4144 2024-04-02T14:35:18.322 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 4145 2024-04-02T14:35:18.322 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 4146 2024-04-02T14:35:18.322 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 4147 2024-04-02T14:35:18.322 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 4148 2024-04-02T14:35:18.322 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 4149 2024-04-02T14:35:18.322 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 4150 2024-04-02T14:35:18.322 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 4151 2024-04-02T14:35:18.322 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 4152 2024-04-02T14:35:18.322 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 4153 2024-04-02T14:35:18.322 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 4154 2024-04-02T14:35:18.323 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 4155 2024-04-02T14:35:18.323 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 4156 2024-04-02T14:35:18.323 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 4157 2024-04-02T14:35:18.323 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 4158 2024-04-02T14:35:18.323 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 4159 2024-04-02T14:35:18.323 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 4160 2024-04-02T14:35:18.323 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 4161 2024-04-02T14:35:18.323 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 4162 2024-04-02T14:35:18.323 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 4163 2024-04-02T14:35:18.323 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 4164 2024-04-02T14:35:18.323 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 4165 2024-04-02T14:35:18.323 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 4166 2024-04-02T14:35:18.323 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 4167 2024-04-02T14:35:18.323 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 4168 2024-04-02T14:35:18.324 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 4169 2024-04-02T14:35:18.324 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 4170 2024-04-02T14:35:18.324 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 4171 2024-04-02T14:35:18.324 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 4172 2024-04-02T14:35:18.324 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 4173 2024-04-02T14:35:18.324 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 4174 2024-04-02T14:35:18.324 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 5002 2024-04-02T14:35:18.324 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 548 2024-04-02T14:35:18.325 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 549 2024-04-02T14:35:18.326 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 6102 2024-04-02T14:35:18.326 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 6104 2024-04-02T14:35:18.326 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 6106 2024-04-02T14:35:18.326 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 6110 2024-04-02T14:35:18.326 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 6111 2024-04-02T14:35:18.326 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 6112 2024-04-02T14:35:18.326 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 6113 2024-04-02T14:35:18.326 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 6117 2024-04-02T14:35:18.326 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 826 2024-04-02T14:35:18.326 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 827 2024-04-02T14:35:18.326 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 828 2024-04-02T14:35:18.326 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 512 Apr 2 14:35 pg_filenode.map 2024-04-02T14:35:18.327 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Apr 2 14:35 PG_VERSION 2024-04-02T14:35:18.327 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-02T14:35:18.327 INFO:teuthology.orchestra.run.smithi090.stdout:/tmp/cephfs/postgres/data/base/13494: 2024-04-02T14:35:18.327 INFO:teuthology.orchestra.run.smithi090.stdout:total 7777 2024-04-02T14:35:18.327 INFO:teuthology.orchestra.run.smithi090.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 289 Apr 2 14:35 . 2024-04-02T14:35:18.327 INFO:teuthology.orchestra.run.smithi090.stdout:drwx------. 5 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Apr 2 14:35 .. 2024-04-02T14:35:18.327 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 112 2024-04-02T14:35:18.327 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 113 2024-04-02T14:35:18.327 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 81920 Apr 2 14:35 1247 2024-04-02T14:35:18.327 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 1247_fsm 2024-04-02T14:35:18.327 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 1247_vm 2024-04-02T14:35:18.327 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 434176 Apr 2 14:35 1249 2024-04-02T14:35:18.327 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 1249_fsm 2024-04-02T14:35:18.327 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 1249_vm 2024-04-02T14:35:18.327 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 663552 Apr 2 14:35 1255 2024-04-02T14:35:18.328 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 1255_fsm 2024-04-02T14:35:18.328 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 1255_vm 2024-04-02T14:35:18.328 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 106496 Apr 2 14:35 1259 2024-04-02T14:35:18.328 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 1259_fsm 2024-04-02T14:35:18.328 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 1259_vm 2024-04-02T14:35:18.328 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 65536 Apr 2 14:35 13344 2024-04-02T14:35:18.328 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 13344_fsm 2024-04-02T14:35:18.328 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 13344_vm 2024-04-02T14:35:18.328 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 13346 2024-04-02T14:35:18.328 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 13348 2024-04-02T14:35:18.328 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 13349 2024-04-02T14:35:18.328 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 13349_fsm 2024-04-02T14:35:18.328 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 13349_vm 2024-04-02T14:35:18.328 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 13351 2024-04-02T14:35:18.328 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 13353 2024-04-02T14:35:18.329 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 13354 2024-04-02T14:35:18.329 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 13354_fsm 2024-04-02T14:35:18.329 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 13354_vm 2024-04-02T14:35:18.329 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 13356 2024-04-02T14:35:18.329 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 13358 2024-04-02T14:35:18.329 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 13359 2024-04-02T14:35:18.329 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 13359_fsm 2024-04-02T14:35:18.329 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 13359_vm 2024-04-02T14:35:18.329 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 13361 2024-04-02T14:35:18.329 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 13363 2024-04-02T14:35:18.329 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 1417 2024-04-02T14:35:18.329 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 1418 2024-04-02T14:35:18.329 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 174 2024-04-02T14:35:18.329 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 175 2024-04-02T14:35:18.329 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2187 2024-04-02T14:35:18.329 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 2224 2024-04-02T14:35:18.330 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 2328 2024-04-02T14:35:18.330 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 2336 2024-04-02T14:35:18.330 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2337 2024-04-02T14:35:18.330 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2579 2024-04-02T14:35:18.330 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2600 2024-04-02T14:35:18.330 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 2600_fsm 2024-04-02T14:35:18.330 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2600_vm 2024-04-02T14:35:18.330 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2601 2024-04-02T14:35:18.330 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 2601_fsm 2024-04-02T14:35:18.330 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2601_vm 2024-04-02T14:35:18.330 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Apr 2 14:35 2602 2024-04-02T14:35:18.330 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 2602_fsm 2024-04-02T14:35:18.330 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2602_vm 2024-04-02T14:35:18.330 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 2 14:35 2603 2024-04-02T14:35:18.331 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 2603_fsm 2024-04-02T14:35:18.331 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2603_vm 2024-04-02T14:35:18.331 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 2604 2024-04-02T14:35:18.331 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2605 2024-04-02T14:35:18.331 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 2605_fsm 2024-04-02T14:35:18.331 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2605_vm 2024-04-02T14:35:18.331 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2606 2024-04-02T14:35:18.331 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 2606_fsm 2024-04-02T14:35:18.331 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2606_vm 2024-04-02T14:35:18.331 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2607 2024-04-02T14:35:18.331 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 2607_fsm 2024-04-02T14:35:18.331 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2607_vm 2024-04-02T14:35:18.331 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 466944 Apr 2 14:35 2608 2024-04-02T14:35:18.331 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 2608_fsm 2024-04-02T14:35:18.331 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2608_vm 2024-04-02T14:35:18.332 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 344064 Apr 2 14:35 2609 2024-04-02T14:35:18.332 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 2609_fsm 2024-04-02T14:35:18.332 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2609_vm 2024-04-02T14:35:18.332 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 2 14:35 2610 2024-04-02T14:35:18.332 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 2610_fsm 2024-04-02T14:35:18.332 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2610_vm 2024-04-02T14:35:18.332 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 2611 2024-04-02T14:35:18.332 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2612 2024-04-02T14:35:18.332 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 2612_fsm 2024-04-02T14:35:18.332 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2612_vm 2024-04-02T14:35:18.332 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 2613 2024-04-02T14:35:18.332 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2615 2024-04-02T14:35:18.332 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 2615_fsm 2024-04-02T14:35:18.332 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2615_vm 2024-04-02T14:35:18.332 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2616 2024-04-02T14:35:18.333 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 2616_fsm 2024-04-02T14:35:18.333 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2616_vm 2024-04-02T14:35:18.333 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 114688 Apr 2 14:35 2617 2024-04-02T14:35:18.333 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 2617_fsm 2024-04-02T14:35:18.333 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2617_vm 2024-04-02T14:35:18.333 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 98304 Apr 2 14:35 2618 2024-04-02T14:35:18.333 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 2618_fsm 2024-04-02T14:35:18.333 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2618_vm 2024-04-02T14:35:18.333 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 147456 Apr 2 14:35 2619 2024-04-02T14:35:18.333 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 2619_fsm 2024-04-02T14:35:18.333 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2619_vm 2024-04-02T14:35:18.333 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 2620 2024-04-02T14:35:18.333 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2650 2024-04-02T14:35:18.333 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2651 2024-04-02T14:35:18.334 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2652 2024-04-02T14:35:18.334 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 2 14:35 2653 2024-04-02T14:35:18.334 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 2 14:35 2654 2024-04-02T14:35:18.334 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 2 14:35 2655 2024-04-02T14:35:18.334 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2656 2024-04-02T14:35:18.334 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2657 2024-04-02T14:35:18.334 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 122880 Apr 2 14:35 2658 2024-04-02T14:35:18.334 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 81920 Apr 2 14:35 2659 2024-04-02T14:35:18.334 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2660 2024-04-02T14:35:18.334 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2661 2024-04-02T14:35:18.334 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 2 14:35 2662 2024-04-02T14:35:18.334 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Apr 2 14:35 2663 2024-04-02T14:35:18.334 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2664 2024-04-02T14:35:18.335 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2665 2024-04-02T14:35:18.335 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2666 2024-04-02T14:35:18.335 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2667 2024-04-02T14:35:18.335 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2668 2024-04-02T14:35:18.335 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2669 2024-04-02T14:35:18.335 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2670 2024-04-02T14:35:18.335 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 278528 Apr 2 14:35 2673 2024-04-02T14:35:18.335 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 352256 Apr 2 14:35 2674 2024-04-02T14:35:18.335 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 196608 Apr 2 14:35 2675 2024-04-02T14:35:18.335 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2678 2024-04-02T14:35:18.335 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2679 2024-04-02T14:35:18.335 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2680 2024-04-02T14:35:18.335 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2681 2024-04-02T14:35:18.336 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2682 2024-04-02T14:35:18.336 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2683 2024-04-02T14:35:18.336 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2684 2024-04-02T14:35:18.336 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2685 2024-04-02T14:35:18.336 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2686 2024-04-02T14:35:18.336 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2687 2024-04-02T14:35:18.336 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 2 14:35 2688 2024-04-02T14:35:18.336 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Apr 2 14:35 2689 2024-04-02T14:35:18.336 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 90112 Apr 2 14:35 2690 2024-04-02T14:35:18.336 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 253952 Apr 2 14:35 2691 2024-04-02T14:35:18.336 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2692 2024-04-02T14:35:18.336 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2693 2024-04-02T14:35:18.336 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2696 2024-04-02T14:35:18.336 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2699 2024-04-02T14:35:18.336 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2701 2024-04-02T14:35:18.337 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2702 2024-04-02T14:35:18.337 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 2 14:35 2703 2024-04-02T14:35:18.337 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 2 14:35 2704 2024-04-02T14:35:18.337 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2753 2024-04-02T14:35:18.337 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 2753_fsm 2024-04-02T14:35:18.337 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2753_vm 2024-04-02T14:35:18.337 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2754 2024-04-02T14:35:18.337 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2755 2024-04-02T14:35:18.337 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 2 14:35 2756 2024-04-02T14:35:18.337 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 2 14:35 2757 2024-04-02T14:35:18.337 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 2830 2024-04-02T14:35:18.337 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2831 2024-04-02T14:35:18.337 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 2832 2024-04-02T14:35:18.337 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2833 2024-04-02T14:35:18.337 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 2834 2024-04-02T14:35:18.337 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2835 2024-04-02T14:35:18.338 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 2836 2024-04-02T14:35:18.338 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2837 2024-04-02T14:35:18.338 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 491520 Apr 2 14:35 2838 2024-04-02T14:35:18.338 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 2838_fsm 2024-04-02T14:35:18.338 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2838_vm 2024-04-02T14:35:18.338 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2839 2024-04-02T14:35:18.338 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 2840 2024-04-02T14:35:18.338 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 2840_fsm 2024-04-02T14:35:18.338 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2840_vm 2024-04-02T14:35:18.338 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2841 2024-04-02T14:35:18.338 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 2995 2024-04-02T14:35:18.338 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2996 2024-04-02T14:35:18.338 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3079 2024-04-02T14:35:18.338 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 3079_fsm 2024-04-02T14:35:18.339 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3079_vm 2024-04-02T14:35:18.339 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 3080 2024-04-02T14:35:18.339 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 3081 2024-04-02T14:35:18.339 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 2 14:35 3085 2024-04-02T14:35:18.339 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 3118 2024-04-02T14:35:18.339 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3119 2024-04-02T14:35:18.339 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 57344 Apr 2 14:35 3164 2024-04-02T14:35:18.339 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 3256 2024-04-02T14:35:18.339 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3257 2024-04-02T14:35:18.339 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3258 2024-04-02T14:35:18.339 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 3350 2024-04-02T14:35:18.339 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3351 2024-04-02T14:35:18.339 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3379 2024-04-02T14:35:18.339 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3380 2024-04-02T14:35:18.339 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 3381 2024-04-02T14:35:18.339 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 3394 2024-04-02T14:35:18.340 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 3394_fsm 2024-04-02T14:35:18.340 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3394_vm 2024-04-02T14:35:18.340 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 3395 2024-04-02T14:35:18.340 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 3429 2024-04-02T14:35:18.340 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 3430 2024-04-02T14:35:18.340 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3431 2024-04-02T14:35:18.340 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3433 2024-04-02T14:35:18.340 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 3439 2024-04-02T14:35:18.340 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3440 2024-04-02T14:35:18.340 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 2 14:35 3455 2024-04-02T14:35:18.340 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 221184 Apr 2 14:35 3456 2024-04-02T14:35:18.340 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 3456_fsm 2024-04-02T14:35:18.340 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3456_vm 2024-04-02T14:35:18.341 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 3466 2024-04-02T14:35:18.341 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3467 2024-04-02T14:35:18.341 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3468 2024-04-02T14:35:18.341 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 3501 2024-04-02T14:35:18.341 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3502 2024-04-02T14:35:18.341 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3503 2024-04-02T14:35:18.341 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3534 2024-04-02T14:35:18.341 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3541 2024-04-02T14:35:18.341 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 3541_fsm 2024-04-02T14:35:18.341 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3541_vm 2024-04-02T14:35:18.341 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 3542 2024-04-02T14:35:18.341 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3574 2024-04-02T14:35:18.341 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3575 2024-04-02T14:35:18.342 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 3576 2024-04-02T14:35:18.342 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 3596 2024-04-02T14:35:18.342 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3597 2024-04-02T14:35:18.342 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 3598 2024-04-02T14:35:18.342 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3599 2024-04-02T14:35:18.342 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3600 2024-04-02T14:35:18.342 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 3600_fsm 2024-04-02T14:35:18.342 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3600_vm 2024-04-02T14:35:18.342 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3601 2024-04-02T14:35:18.342 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 3601_fsm 2024-04-02T14:35:18.342 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3601_vm 2024-04-02T14:35:18.342 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3602 2024-04-02T14:35:18.342 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 3602_fsm 2024-04-02T14:35:18.342 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3602_vm 2024-04-02T14:35:18.343 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 3603 2024-04-02T14:35:18.343 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 3603_fsm 2024-04-02T14:35:18.343 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3603_vm 2024-04-02T14:35:18.343 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 3604 2024-04-02T14:35:18.343 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 3605 2024-04-02T14:35:18.343 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 3606 2024-04-02T14:35:18.343 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 3607 2024-04-02T14:35:18.343 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 3608 2024-04-02T14:35:18.343 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 2 14:35 3609 2024-04-02T14:35:18.343 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 3712 2024-04-02T14:35:18.343 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3764 2024-04-02T14:35:18.343 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 3764_fsm 2024-04-02T14:35:18.343 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3764_vm 2024-04-02T14:35:18.343 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 3766 2024-04-02T14:35:18.343 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 3767 2024-04-02T14:35:18.344 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3997 2024-04-02T14:35:18.344 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 4143 2024-04-02T14:35:18.344 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 4144 2024-04-02T14:35:18.344 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 4145 2024-04-02T14:35:18.344 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 4146 2024-04-02T14:35:18.344 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 4147 2024-04-02T14:35:18.344 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 4148 2024-04-02T14:35:18.344 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 4149 2024-04-02T14:35:18.344 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 4150 2024-04-02T14:35:18.344 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 4151 2024-04-02T14:35:18.344 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 4152 2024-04-02T14:35:18.344 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 4153 2024-04-02T14:35:18.344 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 4154 2024-04-02T14:35:18.344 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 4155 2024-04-02T14:35:18.345 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 4156 2024-04-02T14:35:18.345 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 4157 2024-04-02T14:35:18.345 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 4158 2024-04-02T14:35:18.345 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 4159 2024-04-02T14:35:18.345 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 4160 2024-04-02T14:35:18.345 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 4161 2024-04-02T14:35:18.345 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 4162 2024-04-02T14:35:18.345 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 4163 2024-04-02T14:35:18.345 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 4164 2024-04-02T14:35:18.345 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 4165 2024-04-02T14:35:18.345 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 4166 2024-04-02T14:35:18.345 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 4167 2024-04-02T14:35:18.345 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 4168 2024-04-02T14:35:18.345 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 4169 2024-04-02T14:35:18.345 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 4170 2024-04-02T14:35:18.346 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 4171 2024-04-02T14:35:18.346 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 4172 2024-04-02T14:35:18.346 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 4173 2024-04-02T14:35:18.346 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 4174 2024-04-02T14:35:18.346 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 5002 2024-04-02T14:35:18.346 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 548 2024-04-02T14:35:18.346 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 549 2024-04-02T14:35:18.346 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 6102 2024-04-02T14:35:18.346 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 6104 2024-04-02T14:35:18.346 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 6106 2024-04-02T14:35:18.346 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 6110 2024-04-02T14:35:18.346 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 6111 2024-04-02T14:35:18.346 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 6112 2024-04-02T14:35:18.346 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 6113 2024-04-02T14:35:18.347 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 6117 2024-04-02T14:35:18.347 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 826 2024-04-02T14:35:18.347 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 827 2024-04-02T14:35:18.347 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 828 2024-04-02T14:35:18.347 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 512 Apr 2 14:35 pg_filenode.map 2024-04-02T14:35:18.347 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Apr 2 14:35 PG_VERSION 2024-04-02T14:35:18.347 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-02T14:35:18.347 INFO:teuthology.orchestra.run.smithi090.stdout:/tmp/cephfs/postgres/data/global: 2024-04-02T14:35:18.347 INFO:teuthology.orchestra.run.smithi090.stdout:total 537 2024-04-02T14:35:18.347 INFO:teuthology.orchestra.run.smithi090.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 57 Apr 2 14:35 . 2024-04-02T14:35:18.347 INFO:teuthology.orchestra.run.smithi090.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Apr 2 14:35 .. 2024-04-02T14:35:18.347 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 1213 2024-04-02T14:35:18.347 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 1213_fsm 2024-04-02T14:35:18.348 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 1213_vm 2024-04-02T14:35:18.348 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 1214 2024-04-02T14:35:18.348 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 1214_fsm 2024-04-02T14:35:18.348 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 1214_vm 2024-04-02T14:35:18.348 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 1232 2024-04-02T14:35:18.348 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 1233 2024-04-02T14:35:18.348 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 1260 2024-04-02T14:35:18.348 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 1260_fsm 2024-04-02T14:35:18.348 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 1260_vm 2024-04-02T14:35:18.348 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 1261 2024-04-02T14:35:18.348 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 1261_fsm 2024-04-02T14:35:18.348 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 1261_vm 2024-04-02T14:35:18.348 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 1262 2024-04-02T14:35:18.348 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 1262_fsm 2024-04-02T14:35:18.348 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 1262_vm 2024-04-02T14:35:18.349 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2396 2024-04-02T14:35:18.349 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 2396_fsm 2024-04-02T14:35:18.349 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2396_vm 2024-04-02T14:35:18.349 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2397 2024-04-02T14:35:18.349 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2671 2024-04-02T14:35:18.349 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2672 2024-04-02T14:35:18.349 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2676 2024-04-02T14:35:18.349 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2677 2024-04-02T14:35:18.349 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2694 2024-04-02T14:35:18.349 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2695 2024-04-02T14:35:18.349 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2697 2024-04-02T14:35:18.349 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2698 2024-04-02T14:35:18.349 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 2846 2024-04-02T14:35:18.349 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2847 2024-04-02T14:35:18.349 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 2964 2024-04-02T14:35:18.350 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2965 2024-04-02T14:35:18.350 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 2966 2024-04-02T14:35:18.350 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2967 2024-04-02T14:35:18.350 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 3592 2024-04-02T14:35:18.350 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3593 2024-04-02T14:35:18.350 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 4060 2024-04-02T14:35:18.350 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 4061 2024-04-02T14:35:18.350 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 4175 2024-04-02T14:35:18.350 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 4176 2024-04-02T14:35:18.350 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 4177 2024-04-02T14:35:18.350 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 4178 2024-04-02T14:35:18.350 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 4181 2024-04-02T14:35:18.350 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 4182 2024-04-02T14:35:18.350 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 4183 2024-04-02T14:35:18.351 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 4184 2024-04-02T14:35:18.351 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 4185 2024-04-02T14:35:18.351 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 4186 2024-04-02T14:35:18.351 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 6000 2024-04-02T14:35:18.351 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 6001 2024-04-02T14:35:18.351 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 6002 2024-04-02T14:35:18.351 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 6100 2024-04-02T14:35:18.351 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 6114 2024-04-02T14:35:18.351 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 6115 2024-04-02T14:35:18.351 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 pg_control 2024-04-02T14:35:18.351 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 512 Apr 2 14:35 pg_filenode.map 2024-04-02T14:35:18.351 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-02T14:35:18.351 INFO:teuthology.orchestra.run.smithi090.stdout:/tmp/cephfs/postgres/data/log: 2024-04-02T14:35:18.351 INFO:teuthology.orchestra.run.smithi090.stdout:total 0 2024-04-02T14:35:18.351 INFO:teuthology.orchestra.run.smithi090.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 . 2024-04-02T14:35:18.351 INFO:teuthology.orchestra.run.smithi090.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Apr 2 14:35 .. 2024-04-02T14:35:18.352 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-02T14:35:18.352 INFO:teuthology.orchestra.run.smithi090.stdout:/tmp/cephfs/postgres/data/pg_commit_ts: 2024-04-02T14:35:18.352 INFO:teuthology.orchestra.run.smithi090.stdout:total 0 2024-04-02T14:35:18.352 INFO:teuthology.orchestra.run.smithi090.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 . 2024-04-02T14:35:18.352 INFO:teuthology.orchestra.run.smithi090.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Apr 2 14:35 .. 2024-04-02T14:35:18.352 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-02T14:35:18.352 INFO:teuthology.orchestra.run.smithi090.stdout:/tmp/cephfs/postgres/data/pg_dynshmem: 2024-04-02T14:35:18.352 INFO:teuthology.orchestra.run.smithi090.stdout:total 0 2024-04-02T14:35:18.352 INFO:teuthology.orchestra.run.smithi090.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 . 2024-04-02T14:35:18.352 INFO:teuthology.orchestra.run.smithi090.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Apr 2 14:35 .. 2024-04-02T14:35:18.352 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-02T14:35:18.352 INFO:teuthology.orchestra.run.smithi090.stdout:/tmp/cephfs/postgres/data/pg_logical: 2024-04-02T14:35:18.352 INFO:teuthology.orchestra.run.smithi090.stdout:total 1 2024-04-02T14:35:18.352 INFO:teuthology.orchestra.run.smithi090.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Apr 2 14:35 . 2024-04-02T14:35:18.352 INFO:teuthology.orchestra.run.smithi090.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Apr 2 14:35 .. 2024-04-02T14:35:18.352 INFO:teuthology.orchestra.run.smithi090.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 mappings 2024-04-02T14:35:18.353 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8 Apr 2 14:35 replorigin_checkpoint 2024-04-02T14:35:18.353 INFO:teuthology.orchestra.run.smithi090.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 snapshots 2024-04-02T14:35:18.353 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-02T14:35:18.353 INFO:teuthology.orchestra.run.smithi090.stdout:/tmp/cephfs/postgres/data/pg_logical/mappings: 2024-04-02T14:35:18.353 INFO:teuthology.orchestra.run.smithi090.stdout:total 0 2024-04-02T14:35:18.353 INFO:teuthology.orchestra.run.smithi090.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 . 2024-04-02T14:35:18.353 INFO:teuthology.orchestra.run.smithi090.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Apr 2 14:35 .. 2024-04-02T14:35:18.353 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-02T14:35:18.353 INFO:teuthology.orchestra.run.smithi090.stdout:/tmp/cephfs/postgres/data/pg_logical/snapshots: 2024-04-02T14:35:18.353 INFO:teuthology.orchestra.run.smithi090.stdout:total 0 2024-04-02T14:35:18.353 INFO:teuthology.orchestra.run.smithi090.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 . 2024-04-02T14:35:18.353 INFO:teuthology.orchestra.run.smithi090.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Apr 2 14:35 .. 2024-04-02T14:35:18.353 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-02T14:35:18.353 INFO:teuthology.orchestra.run.smithi090.stdout:/tmp/cephfs/postgres/data/pg_multixact: 2024-04-02T14:35:18.353 INFO:teuthology.orchestra.run.smithi090.stdout:total 0 2024-04-02T14:35:18.354 INFO:teuthology.orchestra.run.smithi090.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 2 Apr 2 14:35 . 2024-04-02T14:35:18.354 INFO:teuthology.orchestra.run.smithi090.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Apr 2 14:35 .. 2024-04-02T14:35:18.354 INFO:teuthology.orchestra.run.smithi090.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Apr 2 14:35 members 2024-04-02T14:35:18.354 INFO:teuthology.orchestra.run.smithi090.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Apr 2 14:35 offsets 2024-04-02T14:35:18.354 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-02T14:35:18.354 INFO:teuthology.orchestra.run.smithi090.stdout:/tmp/cephfs/postgres/data/pg_multixact/members: 2024-04-02T14:35:18.354 INFO:teuthology.orchestra.run.smithi090.stdout:total 8 2024-04-02T14:35:18.354 INFO:teuthology.orchestra.run.smithi090.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Apr 2 14:35 . 2024-04-02T14:35:18.354 INFO:teuthology.orchestra.run.smithi090.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 2 Apr 2 14:35 .. 2024-04-02T14:35:18.354 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 0000 2024-04-02T14:35:18.354 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-02T14:35:18.354 INFO:teuthology.orchestra.run.smithi090.stdout:/tmp/cephfs/postgres/data/pg_multixact/offsets: 2024-04-02T14:35:18.354 INFO:teuthology.orchestra.run.smithi090.stdout:total 8 2024-04-02T14:35:18.354 INFO:teuthology.orchestra.run.smithi090.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Apr 2 14:35 . 2024-04-02T14:35:18.354 INFO:teuthology.orchestra.run.smithi090.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 2 Apr 2 14:35 .. 2024-04-02T14:35:18.355 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 0000 2024-04-02T14:35:18.355 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-02T14:35:18.355 INFO:teuthology.orchestra.run.smithi090.stdout:/tmp/cephfs/postgres/data/pg_notify: 2024-04-02T14:35:18.355 INFO:teuthology.orchestra.run.smithi090.stdout:total 0 2024-04-02T14:35:18.355 INFO:teuthology.orchestra.run.smithi090.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 . 2024-04-02T14:35:18.355 INFO:teuthology.orchestra.run.smithi090.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Apr 2 14:35 .. 2024-04-02T14:35:18.355 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-02T14:35:18.355 INFO:teuthology.orchestra.run.smithi090.stdout:/tmp/cephfs/postgres/data/pg_replslot: 2024-04-02T14:35:18.355 INFO:teuthology.orchestra.run.smithi090.stdout:total 0 2024-04-02T14:35:18.355 INFO:teuthology.orchestra.run.smithi090.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 . 2024-04-02T14:35:18.355 INFO:teuthology.orchestra.run.smithi090.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Apr 2 14:35 .. 2024-04-02T14:35:18.355 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-02T14:35:18.355 INFO:teuthology.orchestra.run.smithi090.stdout:/tmp/cephfs/postgres/data/pg_serial: 2024-04-02T14:35:18.355 INFO:teuthology.orchestra.run.smithi090.stdout:total 0 2024-04-02T14:35:18.355 INFO:teuthology.orchestra.run.smithi090.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 . 2024-04-02T14:35:18.355 INFO:teuthology.orchestra.run.smithi090.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Apr 2 14:35 .. 2024-04-02T14:35:18.356 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-02T14:35:18.356 INFO:teuthology.orchestra.run.smithi090.stdout:/tmp/cephfs/postgres/data/pg_snapshots: 2024-04-02T14:35:18.356 INFO:teuthology.orchestra.run.smithi090.stdout:total 0 2024-04-02T14:35:18.356 INFO:teuthology.orchestra.run.smithi090.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 . 2024-04-02T14:35:18.356 INFO:teuthology.orchestra.run.smithi090.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Apr 2 14:35 .. 2024-04-02T14:35:18.356 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-02T14:35:18.356 INFO:teuthology.orchestra.run.smithi090.stdout:/tmp/cephfs/postgres/data/pg_stat: 2024-04-02T14:35:18.356 INFO:teuthology.orchestra.run.smithi090.stdout:total 0 2024-04-02T14:35:18.356 INFO:teuthology.orchestra.run.smithi090.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 . 2024-04-02T14:35:18.356 INFO:teuthology.orchestra.run.smithi090.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Apr 2 14:35 .. 2024-04-02T14:35:18.356 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-02T14:35:18.356 INFO:teuthology.orchestra.run.smithi090.stdout:/tmp/cephfs/postgres/data/pg_stat_tmp: 2024-04-02T14:35:18.356 INFO:teuthology.orchestra.run.smithi090.stdout:total 0 2024-04-02T14:35:18.356 INFO:teuthology.orchestra.run.smithi090.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 . 2024-04-02T14:35:18.356 INFO:teuthology.orchestra.run.smithi090.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Apr 2 14:35 .. 2024-04-02T14:35:18.356 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-02T14:35:18.357 INFO:teuthology.orchestra.run.smithi090.stdout:/tmp/cephfs/postgres/data/pg_subtrans: 2024-04-02T14:35:18.357 INFO:teuthology.orchestra.run.smithi090.stdout:total 8 2024-04-02T14:35:18.357 INFO:teuthology.orchestra.run.smithi090.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Apr 2 14:35 . 2024-04-02T14:35:18.357 INFO:teuthology.orchestra.run.smithi090.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Apr 2 14:35 .. 2024-04-02T14:35:18.357 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 0000 2024-04-02T14:35:18.357 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-02T14:35:18.357 INFO:teuthology.orchestra.run.smithi090.stdout:/tmp/cephfs/postgres/data/pg_tblspc: 2024-04-02T14:35:18.357 INFO:teuthology.orchestra.run.smithi090.stdout:total 0 2024-04-02T14:35:18.357 INFO:teuthology.orchestra.run.smithi090.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 . 2024-04-02T14:35:18.357 INFO:teuthology.orchestra.run.smithi090.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Apr 2 14:35 .. 2024-04-02T14:35:18.357 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-02T14:35:18.357 INFO:teuthology.orchestra.run.smithi090.stdout:/tmp/cephfs/postgres/data/pg_twophase: 2024-04-02T14:35:18.357 INFO:teuthology.orchestra.run.smithi090.stdout:total 0 2024-04-02T14:35:18.357 INFO:teuthology.orchestra.run.smithi090.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 . 2024-04-02T14:35:18.357 INFO:teuthology.orchestra.run.smithi090.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Apr 2 14:35 .. 2024-04-02T14:35:18.357 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-02T14:35:18.357 INFO:teuthology.orchestra.run.smithi090.stdout:/tmp/cephfs/postgres/data/pg_wal: 2024-04-02T14:35:18.358 INFO:teuthology.orchestra.run.smithi090.stdout:total 16384 2024-04-02T14:35:18.358 INFO:teuthology.orchestra.run.smithi090.stdout:drwx------. 3 postgres postgres system_u:object_r:postgresql_db_t:s0 2 Apr 2 14:35 . 2024-04-02T14:35:18.358 INFO:teuthology.orchestra.run.smithi090.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Apr 2 14:35 .. 2024-04-02T14:35:18.358 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 2 14:35 000000010000000000000001 2024-04-02T14:35:18.358 INFO:teuthology.orchestra.run.smithi090.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 archive_status 2024-04-02T14:35:18.358 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-02T14:35:18.358 INFO:teuthology.orchestra.run.smithi090.stdout:/tmp/cephfs/postgres/data/pg_wal/archive_status: 2024-04-02T14:35:18.358 INFO:teuthology.orchestra.run.smithi090.stdout:total 0 2024-04-02T14:35:18.358 INFO:teuthology.orchestra.run.smithi090.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 . 2024-04-02T14:35:18.358 INFO:teuthology.orchestra.run.smithi090.stdout:drwx------. 3 postgres postgres system_u:object_r:postgresql_db_t:s0 2 Apr 2 14:35 .. 2024-04-02T14:35:18.358 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-02T14:35:18.358 INFO:teuthology.orchestra.run.smithi090.stdout:/tmp/cephfs/postgres/data/pg_xact: 2024-04-02T14:35:18.358 INFO:teuthology.orchestra.run.smithi090.stdout:total 8 2024-04-02T14:35:18.358 INFO:teuthology.orchestra.run.smithi090.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Apr 2 14:35 . 2024-04-02T14:35:18.358 INFO:teuthology.orchestra.run.smithi090.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Apr 2 14:35 .. 2024-04-02T14:35:18.359 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 0000 2024-04-02T14:35:18.359 DEBUG:teuthology.orchestra.run.smithi090:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo systemctl start postgresql' 2024-04-02T14:35:18.677 DEBUG:teuthology.orchestra.run.smithi090:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo -u postgres -- pgbench -s 32 -i' 2024-04-02T14:35:18.801 INFO:teuthology.orchestra.run.smithi090.stderr:dropping old tables... 2024-04-02T14:35:18.801 INFO:teuthology.orchestra.run.smithi090.stderr:NOTICE: table "pgbench_accounts" does not exist, skipping 2024-04-02T14:35:18.801 INFO:teuthology.orchestra.run.smithi090.stderr:NOTICE: table "pgbench_branches" does not exist, skipping 2024-04-02T14:35:18.801 INFO:teuthology.orchestra.run.smithi090.stderr:NOTICE: table "pgbench_history" does not exist, skipping 2024-04-02T14:35:18.801 INFO:teuthology.orchestra.run.smithi090.stderr:NOTICE: table "pgbench_tellers" does not exist, skipping 2024-04-02T14:35:18.801 INFO:teuthology.orchestra.run.smithi090.stderr:creating tables... 2024-04-02T14:35:18.824 INFO:teuthology.orchestra.run.smithi090.stderr:generating data (client-side)... 2024-04-02T14:35:18.978 INFO:teuthology.orchestra.run.smithi090.stderr:100000 of 3200000 tuples (3%) done (elapsed 0.11 s, remaining 3.39 s) 2024-04-02T14:35:19.359 INFO:teuthology.orchestra.run.smithi090.stderr:200000 of 3200000 tuples (6%) done (elapsed 0.49 s, remaining 7.38 s) 2024-04-02T14:35:19.530 INFO:teuthology.orchestra.run.smithi090.stderr:300000 of 3200000 tuples (9%) done (elapsed 0.66 s, remaining 6.41 s) 2024-04-02T14:35:19.686 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:35:19 smithi177 ceph-mon[36431]: pgmap v16: 97 pgs: 97 active+clean; 21 MiB data, 469 MiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 1.7 MiB/s wr, 22 op/s 2024-04-02T14:35:19.688 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:35:19 smithi090 ceph-mon[31674]: pgmap v16: 97 pgs: 97 active+clean; 21 MiB data, 469 MiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 1.7 MiB/s wr, 22 op/s 2024-04-02T14:35:19.861 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:35:19 smithi092 ceph-mon[36497]: pgmap v16: 97 pgs: 97 active+clean; 21 MiB data, 469 MiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 1.7 MiB/s wr, 22 op/s 2024-04-02T14:35:19.867 INFO:teuthology.orchestra.run.smithi090.stderr:400000 of 3200000 tuples (12%) done (elapsed 1.00 s, remaining 7.01 s) 2024-04-02T14:35:20.174 INFO:teuthology.orchestra.run.smithi090.stderr:500000 of 3200000 tuples (15%) done (elapsed 1.31 s, remaining 7.06 s) 2024-04-02T14:35:20.343 INFO:teuthology.orchestra.run.smithi090.stderr:600000 of 3200000 tuples (18%) done (elapsed 1.48 s, remaining 6.40 s) 2024-04-02T14:35:20.749 INFO:teuthology.orchestra.run.smithi090.stderr:700000 of 3200000 tuples (21%) done (elapsed 1.88 s, remaining 6.72 s) 2024-04-02T14:35:21.128 INFO:teuthology.orchestra.run.smithi090.stderr:800000 of 3200000 tuples (25%) done (elapsed 2.26 s, remaining 6.78 s) 2024-04-02T14:35:21.333 INFO:teuthology.orchestra.run.smithi090.stderr:900000 of 3200000 tuples (28%) done (elapsed 2.47 s, remaining 6.30 s) 2024-04-02T14:35:21.694 INFO:teuthology.orchestra.run.smithi090.stderr:1000000 of 3200000 tuples (31%) done (elapsed 2.83 s, remaining 6.22 s) 2024-04-02T14:35:22.085 INFO:teuthology.orchestra.run.smithi090.stderr:1100000 of 3200000 tuples (34%) done (elapsed 3.22 s, remaining 6.14 s) 2024-04-02T14:35:22.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:35:21 smithi177 ceph-mon[36431]: pgmap v17: 97 pgs: 97 active+clean; 44 MiB data, 545 MiB used, 1.0 TiB / 1.0 TiB avail; 330 KiB/s rd, 4.1 MiB/s wr, 321 op/s 2024-04-02T14:35:22.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:35:21 smithi090 ceph-mon[31674]: pgmap v17: 97 pgs: 97 active+clean; 44 MiB data, 545 MiB used, 1.0 TiB / 1.0 TiB avail; 330 KiB/s rd, 4.1 MiB/s wr, 321 op/s 2024-04-02T14:35:22.279 INFO:teuthology.orchestra.run.smithi090.stderr:1200000 of 3200000 tuples (37%) done (elapsed 3.41 s, remaining 5.69 s) 2024-04-02T14:35:22.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:35:21 smithi092 ceph-mon[36497]: pgmap v17: 97 pgs: 97 active+clean; 44 MiB data, 545 MiB used, 1.0 TiB / 1.0 TiB avail; 330 KiB/s rd, 4.1 MiB/s wr, 321 op/s 2024-04-02T14:35:22.664 INFO:teuthology.orchestra.run.smithi090.stderr:1300000 of 3200000 tuples (40%) done (elapsed 3.80 s, remaining 5.55 s) 2024-04-02T14:35:23.015 INFO:teuthology.orchestra.run.smithi090.stderr:1400000 of 3200000 tuples (43%) done (elapsed 4.15 s, remaining 5.33 s) 2024-04-02T14:35:23.203 INFO:teuthology.orchestra.run.smithi090.stderr:1500000 of 3200000 tuples (46%) done (elapsed 4.34 s, remaining 4.91 s) 2024-04-02T14:35:23.631 INFO:teuthology.orchestra.run.smithi090.stderr:1600000 of 3200000 tuples (50%) done (elapsed 4.76 s, remaining 4.76 s) 2024-04-02T14:35:24.066 INFO:teuthology.orchestra.run.smithi090.stderr:1700000 of 3200000 tuples (53%) done (elapsed 5.20 s, remaining 4.59 s) 2024-04-02T14:35:24.166 INFO:teuthology.orchestra.run.smithi090.stderr:1800000 of 3200000 tuples (56%) done (elapsed 5.30 s, remaining 4.12 s) 2024-04-02T14:35:24.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:35:23 smithi177 ceph-mon[36431]: pgmap v18: 97 pgs: 97 active+clean; 44 MiB data, 545 MiB used, 1.0 TiB / 1.0 TiB avail; 325 KiB/s rd, 4.1 MiB/s wr, 313 op/s 2024-04-02T14:35:24.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:35:23 smithi090 ceph-mon[31674]: pgmap v18: 97 pgs: 97 active+clean; 44 MiB data, 545 MiB used, 1.0 TiB / 1.0 TiB avail; 325 KiB/s rd, 4.1 MiB/s wr, 313 op/s 2024-04-02T14:35:24.360 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:35:23 smithi092 ceph-mon[36497]: pgmap v18: 97 pgs: 97 active+clean; 44 MiB data, 545 MiB used, 1.0 TiB / 1.0 TiB avail; 325 KiB/s rd, 4.1 MiB/s wr, 313 op/s 2024-04-02T14:35:24.620 INFO:teuthology.orchestra.run.smithi090.stderr:1900000 of 3200000 tuples (59%) done (elapsed 5.75 s, remaining 3.94 s) 2024-04-02T14:35:24.936 INFO:teuthology.orchestra.run.smithi090.stderr:2000000 of 3200000 tuples (62%) done (elapsed 6.07 s, remaining 3.64 s) 2024-04-02T14:35:25.028 INFO:teuthology.orchestra.run.smithi090.stderr:2100000 of 3200000 tuples (65%) done (elapsed 6.16 s, remaining 3.23 s) 2024-04-02T14:35:25.337 INFO:teuthology.orchestra.run.smithi090.stderr:2200000 of 3200000 tuples (68%) done (elapsed 6.47 s, remaining 2.94 s) 2024-04-02T14:35:25.525 INFO:teuthology.orchestra.run.smithi090.stderr:2300000 of 3200000 tuples (71%) done (elapsed 6.66 s, remaining 2.61 s) 2024-04-02T14:35:25.888 INFO:teuthology.orchestra.run.smithi090.stderr:2400000 of 3200000 tuples (75%) done (elapsed 7.02 s, remaining 2.34 s) 2024-04-02T14:35:26.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:35:25 smithi177 ceph-mon[36431]: pgmap v19: 97 pgs: 97 active+clean; 189 MiB data, 956 MiB used, 1.0 TiB / 1.0 TiB avail; 350 KiB/s rd, 29 MiB/s wr, 334 op/s 2024-04-02T14:35:26.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:35:25 smithi090 ceph-mon[31674]: pgmap v19: 97 pgs: 97 active+clean; 189 MiB data, 956 MiB used, 1.0 TiB / 1.0 TiB avail; 350 KiB/s rd, 29 MiB/s wr, 334 op/s 2024-04-02T14:35:26.270 INFO:teuthology.orchestra.run.smithi090.stderr:2500000 of 3200000 tuples (78%) done (elapsed 7.40 s, remaining 2.07 s) 2024-04-02T14:35:26.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:35:25 smithi092 ceph-mon[36497]: pgmap v19: 97 pgs: 97 active+clean; 189 MiB data, 956 MiB used, 1.0 TiB / 1.0 TiB avail; 350 KiB/s rd, 29 MiB/s wr, 334 op/s 2024-04-02T14:35:26.373 INFO:teuthology.orchestra.run.smithi090.stderr:2600000 of 3200000 tuples (81%) done (elapsed 7.51 s, remaining 1.73 s) 2024-04-02T14:35:26.729 INFO:teuthology.orchestra.run.smithi090.stderr:2700000 of 3200000 tuples (84%) done (elapsed 7.86 s, remaining 1.46 s) 2024-04-02T14:35:26.896 INFO:teuthology.orchestra.run.smithi090.stderr:2800000 of 3200000 tuples (87%) done (elapsed 8.03 s, remaining 1.15 s) 2024-04-02T14:35:27.229 INFO:teuthology.orchestra.run.smithi090.stderr:2900000 of 3200000 tuples (90%) done (elapsed 8.36 s, remaining 0.87 s) 2024-04-02T14:35:27.558 INFO:teuthology.orchestra.run.smithi090.stderr:3000000 of 3200000 tuples (93%) done (elapsed 8.69 s, remaining 0.58 s) 2024-04-02T14:35:27.745 INFO:teuthology.orchestra.run.smithi090.stderr:3100000 of 3200000 tuples (96%) done (elapsed 8.88 s, remaining 0.29 s) 2024-04-02T14:35:28.081 INFO:teuthology.orchestra.run.smithi090.stderr:3200000 of 3200000 tuples (100%) done (elapsed 9.21 s, remaining 0.00 s) 2024-04-02T14:35:28.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:35:27 smithi177 ceph-mon[36431]: pgmap v20: 97 pgs: 97 active+clean; 189 MiB data, 956 MiB used, 1.0 TiB / 1.0 TiB avail; 215 KiB/s rd, 27 MiB/s wr, 320 op/s 2024-04-02T14:35:28.187 INFO:teuthology.orchestra.run.smithi090.stderr:vacuuming... 2024-04-02T14:35:28.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:35:27 smithi090 ceph-mon[31674]: pgmap v20: 97 pgs: 97 active+clean; 189 MiB data, 956 MiB used, 1.0 TiB / 1.0 TiB avail; 215 KiB/s rd, 27 MiB/s wr, 320 op/s 2024-04-02T14:35:28.360 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:35:27 smithi092 ceph-mon[36497]: pgmap v20: 97 pgs: 97 active+clean; 189 MiB data, 956 MiB used, 1.0 TiB / 1.0 TiB avail; 215 KiB/s rd, 27 MiB/s wr, 320 op/s 2024-04-02T14:35:29.172 INFO:teuthology.orchestra.run.smithi090.stderr:creating primary keys... 2024-04-02T14:35:30.163 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:35:29 smithi092 ceph-mon[36497]: pgmap v21: 97 pgs: 97 active+clean; 229 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 240 KiB/s rd, 34 MiB/s wr, 332 op/s 2024-04-02T14:35:30.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:35:29 smithi177 ceph-mon[36431]: pgmap v21: 97 pgs: 97 active+clean; 229 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 240 KiB/s rd, 34 MiB/s wr, 332 op/s 2024-04-02T14:35:30.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:35:29 smithi090 ceph-mon[31674]: pgmap v21: 97 pgs: 97 active+clean; 229 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 240 KiB/s rd, 34 MiB/s wr, 332 op/s 2024-04-02T14:35:31.129 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.127+0000 7f6561e00640 1 -- 172.21.15.90:0/379311097 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f655c101ab0 msgr2=0x7f655c101f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:35:31.130 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.127+0000 7f6561e00640 1 --2- 172.21.15.90:0/379311097 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f655c101ab0 0x7f655c101f30 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f6548009900 tx=0x7f654802bc30 comp rx=0 tx=0).stop 2024-04-02T14:35:31.130 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.127+0000 7f6561e00640 1 -- 172.21.15.90:0/379311097 shutdown_connections 2024-04-02T14:35:31.130 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.127+0000 7f6561e00640 1 --2- 172.21.15.90:0/379311097 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f655c102470 0x7f655c10e980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:31.130 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.127+0000 7f6561e00640 1 --2- 172.21.15.90:0/379311097 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f655c101ab0 0x7f655c101f30 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:31.130 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.127+0000 7f6561e00640 1 --2- 172.21.15.90:0/379311097 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f655c107ac0 0x7f655c107ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:31.130 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.127+0000 7f6561e00640 1 -- 172.21.15.90:0/379311097 >> 172.21.15.90:0/379311097 conn(0x7f655c0fd5c0 msgr2=0x7f655c0ff9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:35:31.130 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.127+0000 7f6561e00640 1 -- 172.21.15.90:0/379311097 shutdown_connections 2024-04-02T14:35:31.130 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.128+0000 7f6561e00640 1 -- 172.21.15.90:0/379311097 wait complete. 2024-04-02T14:35:31.132 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.128+0000 7f6561e00640 1 Processor -- start 2024-04-02T14:35:31.132 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.129+0000 7f6561e00640 1 -- start start 2024-04-02T14:35:31.132 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.129+0000 7f6561e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f655c101ab0 0x7f655c19a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:31.133 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.129+0000 7f6561e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f655c102470 0x7f655c19aa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:31.133 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.129+0000 7f6561e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f655c107ac0 0x7f655c19edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:31.133 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.129+0000 7f6561e00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f655c111f20 con 0x7f655c107ac0 2024-04-02T14:35:31.133 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.129+0000 7f6561e00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f655c111da0 con 0x7f655c101ab0 2024-04-02T14:35:31.133 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.129+0000 7f6561e00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f655c1120a0 con 0x7f655c102470 2024-04-02T14:35:31.133 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.129+0000 7f655aa00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f655c101ab0 0x7f655c19a4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:31.133 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.129+0000 7f655aa00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f655c101ab0 0x7f655c19a4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:35090/0 (socket says 172.21.15.90:35090) 2024-04-02T14:35:31.133 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.129+0000 7f655aa00640 1 -- 172.21.15.90:0/321319091 learned_addr learned my addr 172.21.15.90:0/321319091 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:35:31.133 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.129+0000 7f655b400640 1 --2- 172.21.15.90:0/321319091 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f655c107ac0 0x7f655c19edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:31.133 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.130+0000 7f655aa00640 1 -- 172.21.15.90:0/321319091 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f655c102470 msgr2=0x7f655c19aa30 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-02T14:35:31.133 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.130+0000 7f655aa00640 1 --2- 172.21.15.90:0/321319091 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f655c102470 0x7f655c19aa30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:31.133 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.130+0000 7f655aa00640 1 -- 172.21.15.90:0/321319091 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f655c107ac0 msgr2=0x7f655c19edb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:35:31.133 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.130+0000 7f655aa00640 1 --2- 172.21.15.90:0/321319091 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f655c107ac0 0x7f655c19edb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:31.133 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.130+0000 7f655aa00640 1 -- 172.21.15.90:0/321319091 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f655c19f4e0 con 0x7f655c101ab0 2024-04-02T14:35:31.134 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.130+0000 7f655aa00640 1 --2- 172.21.15.90:0/321319091 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f655c101ab0 0x7f655c19a4f0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f654c00e9f0 tx=0x7f654c00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:35:31.134 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.130+0000 7f6560a00640 1 -- 172.21.15.90:0/321319091 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f654c00cd80 con 0x7f655c101ab0 2024-04-02T14:35:31.134 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.130+0000 7f6560a00640 1 -- 172.21.15.90:0/321319091 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f654c010910 con 0x7f655c101ab0 2024-04-02T14:35:31.134 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.130+0000 7f6560a00640 1 -- 172.21.15.90:0/321319091 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f654c018990 con 0x7f655c101ab0 2024-04-02T14:35:31.134 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.130+0000 7f6561e00640 1 -- 172.21.15.90:0/321319091 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f655c19f7d0 con 0x7f655c101ab0 2024-04-02T14:35:31.134 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.130+0000 7f6561e00640 1 -- 172.21.15.90:0/321319091 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f655c1a7060 con 0x7f655c101ab0 2024-04-02T14:35:31.136 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.133+0000 7f6560a00640 1 -- 172.21.15.90:0/321319091 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7f654c010ac0 con 0x7f655c101ab0 2024-04-02T14:35:31.136 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.134+0000 7f6564a80740 1 Processor -- start 2024-04-02T14:35:31.138 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.134+0000 7f6564a80740 1 -- start start 2024-04-02T14:35:31.138 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.134+0000 7f6564a80740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555b42eb6eb0 0x555b42eb7290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:31.138 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.134+0000 7f6564a80740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555b42eb77d0 0x555b42ebfe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:31.138 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.134+0000 7f6564a80740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555b42ec03b0 0x555b42ec27a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:31.138 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.134+0000 7f6564a80740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x555b42e1bd70 con 0x555b42eb77d0 2024-04-02T14:35:31.138 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.134+0000 7f6564a80740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x555b42dc42a0 con 0x555b42eb6eb0 2024-04-02T14:35:31.138 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.134+0000 7f6564a80740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x555b42e17d20 con 0x555b42ec03b0 2024-04-02T14:35:31.138 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.134+0000 7f655b400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555b42eb6eb0 0x555b42eb7290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:31.138 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.134+0000 7f655b400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555b42eb6eb0 0x555b42eb7290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:35102/0 (socket says 172.21.15.90:35102) 2024-04-02T14:35:31.138 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.134+0000 7f655b400640 1 -- 172.21.15.90:0/2784647091 learned_addr learned my addr 172.21.15.90:0/2784647091 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:35:31.138 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.135+0000 7f655a000640 1 --2- 172.21.15.90:0/2784647091 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555b42ec03b0 0x555b42ec27a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:31.138 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.135+0000 7f655b400640 1 -- 172.21.15.90:0/2784647091 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555b42ec03b0 msgr2=0x555b42ec27a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:35:31.139 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.135+0000 7f655b400640 1 --2- 172.21.15.90:0/2784647091 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555b42ec03b0 0x555b42ec27a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:31.139 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.135+0000 7f655b400640 1 -- 172.21.15.90:0/2784647091 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555b42eb77d0 msgr2=0x555b42ebfe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:35:31.139 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.135+0000 7f655b400640 1 --2- 172.21.15.90:0/2784647091 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555b42eb77d0 0x555b42ebfe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:31.139 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.135+0000 7f655b400640 1 -- 172.21.15.90:0/2784647091 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555b42dd8840 con 0x555b42eb6eb0 2024-04-02T14:35:31.139 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.135+0000 7f6560a00640 1 --2- 172.21.15.90:0/321319091 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f65400a3f60 0x7f65400a6420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:31.139 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.135+0000 7f655b400640 1 --2- 172.21.15.90:0/2784647091 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555b42eb6eb0 0x555b42eb7290 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f6554009fc0 tx=0x7f6554010860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dcd43a956f56e2b8 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:35:31.139 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.135+0000 7f6560a00640 1 -- 172.21.15.90:0/321319091 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f654c0d0e40 con 0x7f655c101ab0 2024-04-02T14:35:31.139 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.135+0000 7f655aa00640 1 --2- 172.21.15.90:0/321319091 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f65400a3f60 0x7f65400a6420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:31.139 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.135+0000 7f6547e00640 1 -- 172.21.15.90:0/2784647091 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6554003b70 con 0x555b42eb6eb0 2024-04-02T14:35:31.139 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.136+0000 7f6547e00640 1 -- 172.21.15.90:0/2784647091 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6554003d10 con 0x555b42eb6eb0 2024-04-02T14:35:31.139 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.136+0000 7f6547e00640 1 -- 172.21.15.90:0/2784647091 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6554011a10 con 0x555b42eb6eb0 2024-04-02T14:35:31.143 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.136+0000 7f6564a80740 1 -- 172.21.15.90:0/2784647091 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555b42eb6eb0 msgr2=0x555b42eb7290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:35:31.143 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.136+0000 7f6564a80740 1 --2- 172.21.15.90:0/2784647091 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555b42eb6eb0 0x555b42eb7290 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f6554009fc0 tx=0x7f6554010860 comp rx=0 tx=0).stop 2024-04-02T14:35:31.143 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.136+0000 7f655aa00640 1 --2- 172.21.15.90:0/321319091 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f65400a3f60 0x7f65400a6420 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f654c0ce420 tx=0x7f654c098750 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:35:31.143 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.137+0000 7f6564a80740 1 -- 172.21.15.90:0/2784647091 shutdown_connections 2024-04-02T14:35:31.143 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.137+0000 7f6564a80740 1 --2- 172.21.15.90:0/2784647091 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555b42ec03b0 0x555b42ec27a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:31.143 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.137+0000 7f6564a80740 1 --2- 172.21.15.90:0/2784647091 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555b42eb77d0 0x555b42ebfe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:31.143 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.137+0000 7f6564a80740 1 --2- 172.21.15.90:0/2784647091 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555b42eb6eb0 0x555b42eb7290 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:31.143 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.137+0000 7f6564a80740 1 -- 172.21.15.90:0/2784647091 >> 172.21.15.90:0/2784647091 conn(0x555b42eb15e0 msgr2=0x555b42eb19c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:35:31.144 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.137+0000 7f6564a80740 1 -- 172.21.15.90:0/2784647091 shutdown_connections 2024-04-02T14:35:31.144 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.137+0000 7f6564a80740 1 -- 172.21.15.90:0/2784647091 wait complete. 2024-04-02T14:35:31.144 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.137+0000 7f6564a80740 1 -- start start 2024-04-02T14:35:31.144 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.137+0000 7f6564a80740 1 Processor -- start 2024-04-02T14:35:31.144 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.138+0000 7f6564a80740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555b42eb6eb0 0x555b42ebd3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:31.144 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.138+0000 7f6564a80740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555b42eb77d0 0x555b42ebd900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:31.144 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.138+0000 7f6564a80740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555b42ec03b0 0x555b42ebde40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:31.144 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.138+0000 7f6564a80740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x555b42ec31a0 con 0x555b42eb77d0 2024-04-02T14:35:31.144 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.138+0000 7f6564a80740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x555b42dd7660 con 0x555b42ec03b0 2024-04-02T14:35:31.144 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.138+0000 7f6564a80740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x555b42ec3320 con 0x555b42eb6eb0 2024-04-02T14:35:31.144 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.138+0000 7f6564a80740 10 client.0 _ll_register_callbacks cb 0x555b42dd7050 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:35:31.144 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.138+0000 7f655a000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555b42eb77d0 0x555b42ebd900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:31.144 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.138+0000 7f655b400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555b42eb6eb0 0x555b42ebd3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:31.144 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.138+0000 7f655b400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555b42eb6eb0 0x555b42ebd3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:56946/0 (socket says 172.21.15.90:56946) 2024-04-02T14:35:31.145 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.138+0000 7f655b400640 1 -- 172.21.15.90:0/1848736636 learned_addr learned my addr 172.21.15.90:0/1848736636 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:35:31.145 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.138+0000 7f655b400640 1 --2- 172.21.15.90:0/1848736636 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555b42ec03b0 0x555b42ebde40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:31.145 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.138+0000 7f655b400640 1 -- 172.21.15.90:0/1848736636 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555b42ec03b0 msgr2=0x555b42ebde40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:35:31.145 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.138+0000 7f655b400640 1 --2- 172.21.15.90:0/1848736636 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555b42ec03b0 0x555b42ebde40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:31.145 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.138+0000 7f655b400640 1 -- 172.21.15.90:0/1848736636 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555b42eb77d0 msgr2=0x555b42ebd900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:35:31.145 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.138+0000 7f655b400640 1 --2- 172.21.15.90:0/1848736636 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555b42eb77d0 0x555b42ebd900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:31.145 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.138+0000 7f655b400640 1 -- 172.21.15.90:0/1848736636 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555b42de6e80 con 0x555b42eb6eb0 2024-04-02T14:35:31.145 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.139+0000 7f655b400640 1 --2- 172.21.15.90:0/1848736636 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555b42eb6eb0 0x555b42ebd3c0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f6554003cc0 tx=0x7f6554000f40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d145c1e1b063d6d5 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:35:31.145 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.139+0000 7f6544c00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:35:31.145 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.139+0000 7f6544c00640 1 -- 172.21.15.90:0/1848736636 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6554024040 con 0x555b42eb6eb0 2024-04-02T14:35:31.145 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.140+0000 7f6544c00640 1 -- 172.21.15.90:0/1848736636 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f65540117b0 con 0x555b42eb6eb0 2024-04-02T14:35:31.145 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.140+0000 7f6564a80740 1 -- 172.21.15.90:0/1848736636 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6554003d10 con 0x555b42eb6eb0 2024-04-02T14:35:31.145 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.140+0000 7f6544c00640 1 -- 172.21.15.90:0/1848736636 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f6554032050 con 0x555b42eb6eb0 2024-04-02T14:35:31.145 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.140+0000 7f6564a80740 10 client.25528 fetch_fsmap learned FSMap version 27 2024-04-02T14:35:31.145 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.140+0000 7f6564a80740 1 -- 172.21.15.90:0/1848736636 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x555b42b30210 con 0x555b42eb6eb0 2024-04-02T14:35:31.146 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.140+0000 7f6544c00640 1 -- 172.21.15.90:0/1848736636 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6554003d10 con 0x555b42eb6eb0 2024-04-02T14:35:31.146 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.140+0000 7f6544c00640 1 -- 172.21.15.90:0/1848736636 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f655401b8c0 con 0x555b42eb6eb0 2024-04-02T14:35:31.146 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.141+0000 7f6564a80740 10 client.25528 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:35:31.146 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.141+0000 7f6564a80740 10 client.25528 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-02T14:35:31.146 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.141+0000 7f6564a80740 20 client.25528 populate_metadata read hostname 'smithi090' 2024-04-02T14:35:31.146 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.141+0000 7f6564a80740 1 --2- 172.21.15.90:0/1848736636 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x555b42ed0710 0x555b42ed2b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:31.146 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.141+0000 7f6564a80740 4 client.25528 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:35:31.146 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.141+0000 7f6564a80740 1 -- 172.21.15.90:0/1848736636 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555b42dcbcd0 con 0x555b42ed0710 2024-04-02T14:35:31.148 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.141+0000 7f655a000640 1 --2- 172.21.15.90:0/1848736636 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x555b42ed0710 0x555b42ed2b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:31.148 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.141+0000 7f655a000640 1 --2- 172.21.15.90:0/1848736636 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x555b42ed0710 0x555b42ed2b00 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4e2204edfe5c8d8a server_cookie=bab196aed77678d in_seq=0 out_seq=0 2024-04-02T14:35:31.148 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.141+0000 7f6544c00640 10 client.25528 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:35:31.148 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.142+0000 7f6544c00640 1 -- 172.21.15.90:0/1848736636 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x555b42dcbcd0 con 0x555b42ed0710 2024-04-02T14:35:31.148 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.142+0000 7f6544c00640 10 client.25528 handle_command_reply: tid=0 2024-04-02T14:35:31.148 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.142+0000 7f6564a80740 1 client.25528 shutdown 2024-04-02T14:35:31.148 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.142+0000 7f6564a80740 1 -- 172.21.15.90:0/1848736636 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555b42eb6eb0 msgr2=0x555b42ebd3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:35:31.148 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.142+0000 7f6564a80740 1 --2- 172.21.15.90:0/1848736636 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555b42eb6eb0 0x555b42ebd3c0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f6554003cc0 tx=0x7f6554000f40 comp rx=0 tx=0).stop 2024-04-02T14:35:31.148 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.143+0000 7f6564a80740 1 -- 172.21.15.90:0/1848736636 shutdown_connections 2024-04-02T14:35:31.149 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.143+0000 7f6564a80740 1 --2- 172.21.15.90:0/1848736636 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x555b42ed0710 0x555b42ed2b00 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:31.149 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.143+0000 7f6564a80740 1 --2- 172.21.15.90:0/1848736636 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555b42ec03b0 0x555b42ebde40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:31.149 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.143+0000 7f6564a80740 1 --2- 172.21.15.90:0/1848736636 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555b42eb77d0 0x555b42ebd900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:31.149 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.143+0000 7f6564a80740 1 --2- 172.21.15.90:0/1848736636 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555b42eb6eb0 0x555b42ebd3c0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:31.149 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.143+0000 7f6544c00640 0 client.25528 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:35:31.149 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.143+0000 7f6564a80740 1 -- 172.21.15.90:0/1848736636 >> 172.21.15.90:0/1848736636 conn(0x555b42eb15e0 msgr2=0x555b42eb3330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:35:31.149 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.143+0000 7f6564a80740 1 -- 172.21.15.90:0/1848736636 shutdown_connections 2024-04-02T14:35:31.149 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.143+0000 7f6564a80740 1 -- 172.21.15.90:0/1848736636 wait complete. 2024-04-02T14:35:31.149 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.143+0000 7f6564a80740 20 client.25528 trim_cache size 0 max 16384 2024-04-02T14:35:31.162 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.160+0000 7f6564a80740 1 Processor -- start 2024-04-02T14:35:31.163 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.161+0000 7f6564a80740 1 -- start start 2024-04-02T14:35:31.163 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.161+0000 7f6564a80740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555b42eb6eb0 0x555b42ebfaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:31.163 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.161+0000 7f6564a80740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555b42eb77d0 0x555b42ecc060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:31.164 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.161+0000 7f6564a80740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555b42ec03b0 0x555b42ebda90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:31.164 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.161+0000 7f6564a80740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f655c1afb40 con 0x555b42ec03b0 2024-04-02T14:35:31.164 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.161+0000 7f6564a80740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f655c1b1680 con 0x555b42eb77d0 2024-04-02T14:35:31.164 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.161+0000 7f6564a80740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f655c1b0210 con 0x555b42eb6eb0 2024-04-02T14:35:31.166 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.161+0000 7f655a000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555b42eb77d0 0x555b42ecc060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:31.166 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.161+0000 7f655a000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555b42eb77d0 0x555b42ecc060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:35126/0 (socket says 172.21.15.90:35126) 2024-04-02T14:35:31.167 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.161+0000 7f655a000640 1 -- 172.21.15.90:0/2023978958 learned_addr learned my addr 172.21.15.90:0/2023978958 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:35:31.167 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.162+0000 7f655a000640 1 -- 172.21.15.90:0/2023978958 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555b42eb6eb0 msgr2=0x555b42ebfaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:35:31.167 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.162+0000 7f655b400640 1 --2- 172.21.15.90:0/2023978958 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555b42eb6eb0 0x555b42ebfaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:31.167 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.162+0000 7f655a000640 1 --2- 172.21.15.90:0/2023978958 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555b42eb6eb0 0x555b42ebfaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:31.167 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.162+0000 7f655a000640 1 -- 172.21.15.90:0/2023978958 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555b42ec03b0 msgr2=0x555b42ebda90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-02T14:35:31.167 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.162+0000 7f655a000640 1 --2- 172.21.15.90:0/2023978958 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555b42ec03b0 0x555b42ebda90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:31.167 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.162+0000 7f655a000640 1 -- 172.21.15.90:0/2023978958 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555b42bfc620 con 0x555b42eb77d0 2024-04-02T14:35:31.167 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.162+0000 7f655b400640 1 --2- 172.21.15.90:0/2023978958 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555b42eb6eb0 0x555b42ebfaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:35:31.167 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.163+0000 7f655a000640 1 --2- 172.21.15.90:0/2023978958 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555b42eb77d0 0x555b42ecc060 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f6548006010 tx=0x7f654802bc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b183dbb3498d1faa server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:35:31.167 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.163+0000 7f6547400640 1 -- 172.21.15.90:0/2023978958 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6548047bc0 con 0x555b42eb77d0 2024-04-02T14:35:31.167 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.163+0000 7f6547400640 1 -- 172.21.15.90:0/2023978958 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6548047d60 con 0x555b42eb77d0 2024-04-02T14:35:31.167 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.163+0000 7f6547400640 1 -- 172.21.15.90:0/2023978958 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65480475d0 con 0x555b42eb77d0 2024-04-02T14:35:31.168 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.164+0000 7f6564a80740 1 -- 172.21.15.90:0/2023978958 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555b42eb77d0 msgr2=0x555b42ecc060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:35:31.168 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.164+0000 7f6564a80740 1 --2- 172.21.15.90:0/2023978958 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555b42eb77d0 0x555b42ecc060 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f6548006010 tx=0x7f654802bc30 comp rx=0 tx=0).stop 2024-04-02T14:35:31.168 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.164+0000 7f6564a80740 1 -- 172.21.15.90:0/2023978958 shutdown_connections 2024-04-02T14:35:31.168 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.164+0000 7f6564a80740 1 --2- 172.21.15.90:0/2023978958 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555b42ec03b0 0x555b42ebda90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:31.168 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.164+0000 7f6564a80740 1 --2- 172.21.15.90:0/2023978958 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555b42eb77d0 0x555b42ecc060 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:31.168 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.164+0000 7f6564a80740 1 --2- 172.21.15.90:0/2023978958 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555b42eb6eb0 0x555b42ebfaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:31.168 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.164+0000 7f6564a80740 1 -- 172.21.15.90:0/2023978958 >> 172.21.15.90:0/2023978958 conn(0x555b42eb15e0 msgr2=0x555b42ebd380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:35:31.168 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.164+0000 7f6564a80740 1 -- 172.21.15.90:0/2023978958 shutdown_connections 2024-04-02T14:35:31.168 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.164+0000 7f6564a80740 1 -- 172.21.15.90:0/2023978958 wait complete. 2024-04-02T14:35:31.168 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.164+0000 7f6564a80740 1 -- start start 2024-04-02T14:35:31.171 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.165+0000 7f6564a80740 1 Processor -- start 2024-04-02T14:35:31.172 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.165+0000 7f6564a80740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555b42eb6eb0 0x555b42ec8a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:31.172 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.165+0000 7f6564a80740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555b42eb77d0 0x555b42eb9e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:31.172 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.165+0000 7f6564a80740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555b42ec03b0 0x555b42eba5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:31.172 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.165+0000 7f6564a80740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f655c1b0fb0 con 0x555b42eb6eb0 2024-04-02T14:35:31.172 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.165+0000 7f6564a80740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f655c1b08e0 con 0x555b42ec03b0 2024-04-02T14:35:31.172 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.165+0000 7f6564a80740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f655c1adb20 con 0x555b42eb77d0 2024-04-02T14:35:31.172 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.165+0000 7f6564a80740 10 client.0 _ll_register_callbacks cb 0x7f6554004140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:35:31.172 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.165+0000 7f655a000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555b42eb77d0 0x555b42eb9e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:31.172 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.165+0000 7f655a000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555b42eb77d0 0x555b42eb9e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:56952/0 (socket says 172.21.15.90:56952) 2024-04-02T14:35:31.173 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.166+0000 7f655a000640 1 -- 172.21.15.90:0/781467337 learned_addr learned my addr 172.21.15.90:0/781467337 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:35:31.173 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.166+0000 7f655b400640 1 --2- 172.21.15.90:0/781467337 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555b42ec03b0 0x555b42eba5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:31.173 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.166+0000 7f655a000640 1 -- 172.21.15.90:0/781467337 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555b42ec03b0 msgr2=0x555b42eba5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:35:31.173 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.166+0000 7f655a000640 1 --2- 172.21.15.90:0/781467337 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555b42ec03b0 0x555b42eba5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:31.173 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.166+0000 7f655a000640 1 -- 172.21.15.90:0/781467337 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555b42eb6eb0 msgr2=0x555b42ec8a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:35:31.173 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.166+0000 7f655a000640 1 --2- 172.21.15.90:0/781467337 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555b42eb6eb0 0x555b42ec8a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:31.173 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.166+0000 7f655a000640 1 -- 172.21.15.90:0/781467337 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555b42f3e1b0 con 0x555b42eb77d0 2024-04-02T14:35:31.173 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.166+0000 7f655b400640 1 --2- 172.21.15.90:0/781467337 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555b42ec03b0 0x555b42eba5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:35:31.173 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.166+0000 7f655a000640 1 --2- 172.21.15.90:0/781467337 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555b42eb77d0 0x555b42eb9e90 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f654802c9a0 tx=0x7f654802c810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2c352ef27e665a3e server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:35:31.173 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.166+0000 7f6544c00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:35:31.174 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.166+0000 7f6544c00640 1 -- 172.21.15.90:0/781467337 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6548055060 con 0x555b42eb77d0 2024-04-02T14:35:31.174 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.167+0000 7f6564a80740 1 -- 172.21.15.90:0/781467337 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6548047d60 con 0x555b42eb77d0 2024-04-02T14:35:31.174 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.167+0000 7f6544c00640 1 -- 172.21.15.90:0/781467337 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f6548004250 con 0x555b42eb77d0 2024-04-02T14:35:31.174 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.167+0000 7f6544c00640 1 -- 172.21.15.90:0/781467337 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f654804c070 con 0x555b42eb77d0 2024-04-02T14:35:31.174 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.167+0000 7f6564a80740 10 client.25531 fetch_fsmap learned FSMap version 27 2024-04-02T14:35:31.174 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.167+0000 7f6564a80740 1 -- 172.21.15.90:0/781467337 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x555b42f3f210 con 0x555b42eb77d0 2024-04-02T14:35:31.174 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.167+0000 7f6544c00640 1 -- 172.21.15.90:0/781467337 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6548047d60 con 0x555b42eb77d0 2024-04-02T14:35:31.174 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.167+0000 7f6544c00640 1 -- 172.21.15.90:0/781467337 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f6548003680 con 0x555b42eb77d0 2024-04-02T14:35:31.174 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.167+0000 7f6564a80740 10 client.25531 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:35:31.174 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.167+0000 7f6564a80740 10 client.25531 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-02T14:35:31.174 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.167+0000 7f6564a80740 20 client.25531 populate_metadata read hostname 'smithi090' 2024-04-02T14:35:31.175 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.167+0000 7f6564a80740 1 --2- 172.21.15.90:0/781467337 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x555b42f3f8e0 0x555b42f41d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:31.175 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.167+0000 7f6564a80740 4 client.25531 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-02T14:35:31.175 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.167+0000 7f6564a80740 1 -- 172.21.15.90:0/781467337 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "flush journal"}) -- 0x555b42e292b0 con 0x555b42f3f8e0 2024-04-02T14:35:31.175 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.168+0000 7f655b400640 1 --2- 172.21.15.90:0/781467337 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x555b42f3f8e0 0x555b42f41d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:31.175 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.168+0000 7f655b400640 1 --2- 172.21.15.90:0/781467337 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x555b42f3f8e0 0x555b42f41d00 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ec4e581ce5c013e9 server_cookie=2dd9aed125622dbe in_seq=0 out_seq=0 2024-04-02T14:35:31.175 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.168+0000 7f6544c00640 10 client.25531 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:35:31.223 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.220+0000 7f6544c00640 1 -- 172.21.15.90:0/781467337 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x555b42e292b0 con 0x555b42f3f8e0 2024-04-02T14:35:31.223 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.220+0000 7f6544c00640 10 client.25531 handle_command_reply: tid=0 2024-04-02T14:35:31.223 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.220+0000 7f6564a80740 1 client.25531 shutdown 2024-04-02T14:35:31.223 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.221+0000 7f6564a80740 1 -- 172.21.15.90:0/781467337 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555b42eb77d0 msgr2=0x555b42eb9e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:35:31.224 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.221+0000 7f6564a80740 1 --2- 172.21.15.90:0/781467337 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555b42eb77d0 0x555b42eb9e90 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f654802c9a0 tx=0x7f654802c810 comp rx=0 tx=0).stop 2024-04-02T14:35:31.224 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.221+0000 7f6564a80740 1 -- 172.21.15.90:0/781467337 shutdown_connections 2024-04-02T14:35:31.224 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.221+0000 7f6564a80740 1 --2- 172.21.15.90:0/781467337 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x555b42f3f8e0 0x555b42f41d00 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:31.224 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.221+0000 7f6564a80740 1 --2- 172.21.15.90:0/781467337 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555b42ec03b0 0x555b42eba5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:31.224 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.221+0000 7f6564a80740 1 --2- 172.21.15.90:0/781467337 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555b42eb77d0 0x555b42eb9e90 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:31.224 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.221+0000 7f6564a80740 1 --2- 172.21.15.90:0/781467337 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555b42eb6eb0 0x555b42ec8a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:31.224 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.221+0000 7f6544c00640 0 client.25531 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:35:31.224 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.221+0000 7f6564a80740 1 -- 172.21.15.90:0/781467337 >> 172.21.15.90:0/781467337 conn(0x555b42eb15e0 msgr2=0x555b42eb3150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:35:31.224 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.221+0000 7f6564a80740 1 -- 172.21.15.90:0/781467337 shutdown_connections 2024-04-02T14:35:31.224 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.221+0000 7f6564a80740 1 -- 172.21.15.90:0/781467337 wait complete. 2024-04-02T14:35:31.224 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.221+0000 7f6564a80740 20 client.25531 trim_cache size 0 max 16384 2024-04-02T14:35:31.225 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:{ 2024-04-02T14:35:31.225 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "message": "", 2024-04-02T14:35:31.225 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "return_code": 0 2024-04-02T14:35:31.225 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:} 2024-04-02T14:35:31.226 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.222+0000 7f6561e00640 1 -- 172.21.15.90:0/321319091 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f65400a3f60 msgr2=0x7f65400a6420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:35:31.226 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.222+0000 7f6561e00640 1 --2- 172.21.15.90:0/321319091 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f65400a3f60 0x7f65400a6420 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f654c0ce420 tx=0x7f654c098750 comp rx=0 tx=0).stop 2024-04-02T14:35:31.226 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.222+0000 7f6561e00640 1 -- 172.21.15.90:0/321319091 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f655c101ab0 msgr2=0x7f655c19a4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:35:31.226 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.222+0000 7f6561e00640 1 --2- 172.21.15.90:0/321319091 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f655c101ab0 0x7f655c19a4f0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f654c00e9f0 tx=0x7f654c00eeb0 comp rx=0 tx=0).stop 2024-04-02T14:35:31.226 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.222+0000 7f6561e00640 1 -- 172.21.15.90:0/321319091 shutdown_connections 2024-04-02T14:35:31.226 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.222+0000 7f6561e00640 1 --2- 172.21.15.90:0/321319091 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f65400a3f60 0x7f65400a6420 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:31.226 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.222+0000 7f6561e00640 1 --2- 172.21.15.90:0/321319091 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f655c107ac0 0x7f655c19edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:31.226 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.222+0000 7f6561e00640 1 --2- 172.21.15.90:0/321319091 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f655c102470 0x7f655c19aa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:31.226 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.222+0000 7f6561e00640 1 --2- 172.21.15.90:0/321319091 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f655c101ab0 0x7f655c19a4f0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:31.226 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.222+0000 7f6561e00640 1 -- 172.21.15.90:0/321319091 >> 172.21.15.90:0/321319091 conn(0x7f655c0fd5c0 msgr2=0x7f655c10d220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:35:31.226 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.222+0000 7f6561e00640 1 -- 172.21.15.90:0/321319091 shutdown_connections 2024-04-02T14:35:31.226 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:31.222+0000 7f6561e00640 1 -- 172.21.15.90:0/321319091 wait complete. 2024-04-02T14:35:31.652 INFO:teuthology.orchestra.run.smithi090.stderr:done in 12.85 s (drop tables 0.00 s, create tables 0.02 s, client-side generate 9.36 s, vacuum 0.99 s, primary keys 2.48 s). 2024-04-02T14:35:31.661 DEBUG:teuthology.orchestra.run.smithi090:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo -u postgres -- pgbench -c 100 -j 4 --progress=5 --time=900' 2024-04-02T14:35:31.831 INFO:teuthology.orchestra.run.smithi090.stderr:starting vacuum...end. 2024-04-02T14:35:32.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:35:31 smithi177 ceph-mon[36431]: pgmap v22: 97 pgs: 97 active+clean; 365 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 241 KiB/s rd, 56 MiB/s wr, 342 op/s 2024-04-02T14:35:32.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:35:31 smithi090 ceph-mon[31674]: pgmap v22: 97 pgs: 97 active+clean; 365 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 241 KiB/s rd, 56 MiB/s wr, 342 op/s 2024-04-02T14:35:32.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:35:31 smithi092 ceph-mon[36497]: pgmap v22: 97 pgs: 97 active+clean; 365 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 241 KiB/s rd, 56 MiB/s wr, 342 op/s 2024-04-02T14:35:32.524 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-02T14:35:32.524 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-02T14:35:32.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.791+0000 7f197e000640 1 -- 172.21.15.90:0/2669009695 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f197810e4a0 msgr2=0x7f1978110890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:35:32.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.791+0000 7f197e000640 1 --2- 172.21.15.90:0/2669009695 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f197810e4a0 0x7f1978110890 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f1970009900 tx=0x7f197002bc40 comp rx=0 tx=0).stop 2024-04-02T14:35:32.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.791+0000 7f197e000640 1 -- 172.21.15.90:0/2669009695 shutdown_connections 2024-04-02T14:35:32.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.791+0000 7f197e000640 1 --2- 172.21.15.90:0/2669009695 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f197810e4a0 0x7f1978110890 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:32.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.791+0000 7f197e000640 1 --2- 172.21.15.90:0/2669009695 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1978069600 0x7f197810df60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:32.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.791+0000 7f197e000640 1 --2- 172.21.15.90:0/2669009695 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1978068ce0 0x7f19780690c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:32.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.791+0000 7f197e000640 1 -- 172.21.15.90:0/2669009695 >> 172.21.15.90:0/2669009695 conn(0x7f1978077b00 msgr2=0x7f1978077f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:35:32.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.791+0000 7f197e000640 1 -- 172.21.15.90:0/2669009695 shutdown_connections 2024-04-02T14:35:32.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.791+0000 7f197e000640 1 -- 172.21.15.90:0/2669009695 wait complete. 2024-04-02T14:35:32.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.792+0000 7f197e000640 1 Processor -- start 2024-04-02T14:35:32.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.792+0000 7f197e000640 1 -- start start 2024-04-02T14:35:32.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.793+0000 7f197e000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1978068ce0 0x7f197819a4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:32.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.793+0000 7f197e000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1978069600 0x7f197819a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:32.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.793+0000 7f197e000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f197810e4a0 0x7f197819ed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:32.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.793+0000 7f197e000640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f1978114060 con 0x7f1978069600 2024-04-02T14:35:32.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.793+0000 7f197e000640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f1978113ee0 con 0x7f197810e4a0 2024-04-02T14:35:32.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.793+0000 7f197e000640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f19781141e0 con 0x7f1978068ce0 2024-04-02T14:35:32.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.793+0000 7f1977e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f197810e4a0 0x7f197819ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:32.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.793+0000 7f1977e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f197810e4a0 0x7f197819ed60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:35164/0 (socket says 172.21.15.90:35164) 2024-04-02T14:35:32.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.793+0000 7f1977e00640 1 -- 172.21.15.90:0/2144284824 learned_addr learned my addr 172.21.15.90:0/2144284824 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:35:32.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.793+0000 7f1977400640 1 --2- 172.21.15.90:0/2144284824 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1978068ce0 0x7f197819a4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:32.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.793+0000 7f1977e00640 1 -- 172.21.15.90:0/2144284824 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1978068ce0 msgr2=0x7f197819a4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:35:32.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.794+0000 7f1977e00640 1 --2- 172.21.15.90:0/2144284824 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1978068ce0 0x7f197819a4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:32.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.794+0000 7f1977e00640 1 -- 172.21.15.90:0/2144284824 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1978069600 msgr2=0x7f197819a9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:35:32.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.794+0000 7f1977e00640 1 --2- 172.21.15.90:0/2144284824 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1978069600 0x7f197819a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:32.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.794+0000 7f1977e00640 1 -- 172.21.15.90:0/2144284824 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f197819f460 con 0x7f197810e4a0 2024-04-02T14:35:32.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.794+0000 7f1977e00640 1 --2- 172.21.15.90:0/2144284824 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f197810e4a0 0x7f197819ed60 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f1970036040 tx=0x7f19700045e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:35:32.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.794+0000 7f1974c00640 1 -- 172.21.15.90:0/2144284824 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1970041070 con 0x7f197810e4a0 2024-04-02T14:35:32.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.794+0000 7f1974c00640 1 -- 172.21.15.90:0/2144284824 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f197002cca0 con 0x7f197810e4a0 2024-04-02T14:35:32.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.794+0000 7f1974c00640 1 -- 172.21.15.90:0/2144284824 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1970034440 con 0x7f197810e4a0 2024-04-02T14:35:32.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.794+0000 7f197e000640 1 -- 172.21.15.90:0/2144284824 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f197819f6f0 con 0x7f197810e4a0 2024-04-02T14:35:32.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.795+0000 7f197e000640 1 -- 172.21.15.90:0/2144284824 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f197819fb90 con 0x7f197810e4a0 2024-04-02T14:35:32.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.798+0000 7f1974c00640 1 -- 172.21.15.90:0/2144284824 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7f1970044070 con 0x7f197810e4a0 2024-04-02T14:35:32.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.799+0000 7f1980da8740 1 Processor -- start 2024-04-02T14:35:32.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.799+0000 7f1980da8740 1 -- start start 2024-04-02T14:35:32.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.800+0000 7f1980da8740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d8f8510fc0 0x55d8f85113a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:32.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.800+0000 7f1980da8740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d8f85118e0 0x55d8f8519f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:32.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.800+0000 7f1980da8740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d8f851a4c0 0x55d8f851c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:32.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.800+0000 7f1980da8740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55d8f846c770 con 0x55d8f851a4c0 2024-04-02T14:35:32.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.800+0000 7f1980da8740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55d8f8414700 con 0x55d8f8510fc0 2024-04-02T14:35:32.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.800+0000 7f1980da8740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55d8f8468720 con 0x55d8f85118e0 2024-04-02T14:35:32.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.800+0000 7f1974c00640 1 --2- 172.21.15.90:0/2144284824 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f195c0a3f10 0x7f195c0a63d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:32.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.801+0000 7f1977400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d8f85118e0 0x55d8f8519f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:32.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.801+0000 7f1974c00640 1 -- 172.21.15.90:0/2144284824 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f19700f2640 con 0x7f197810e4a0 2024-04-02T14:35:32.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.801+0000 7f1977e00640 1 --2- 172.21.15.90:0/2144284824 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f195c0a3f10 0x7f195c0a63d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:32.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.801+0000 7f1977400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d8f85118e0 0x55d8f8519f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:56974/0 (socket says 172.21.15.90:56974) 2024-04-02T14:35:32.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.801+0000 7f1977400640 1 -- 172.21.15.90:0/3078664971 learned_addr learned my addr 172.21.15.90:0/3078664971 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:35:32.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.801+0000 7f196ea00640 1 --2- 172.21.15.90:0/3078664971 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d8f8510fc0 0x55d8f85113a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:32.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.801+0000 7f196ea00640 1 --2- 172.21.15.90:0/3078664971 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d8f851a4c0 0x55d8f851c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:32.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.801+0000 7f1977400640 1 -- 172.21.15.90:0/3078664971 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d8f8510fc0 msgr2=0x55d8f85113a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:35:32.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.801+0000 7f1977400640 1 --2- 172.21.15.90:0/3078664971 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d8f8510fc0 0x55d8f85113a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:32.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.801+0000 7f1977400640 1 -- 172.21.15.90:0/3078664971 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d8f851a4c0 msgr2=0x55d8f851c8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:35:32.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.801+0000 7f1977e00640 1 --2- 172.21.15.90:0/2144284824 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f195c0a3f10 0x7f195c0a63d0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f19700efd90 tx=0x7f1970005fd0 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:35:32.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.802+0000 7f1977400640 1 --2- 172.21.15.90:0/3078664971 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d8f851a4c0 0x55d8f851c8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:32.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.802+0000 7f1977400640 1 -- 172.21.15.90:0/3078664971 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d8f824a7d0 con 0x55d8f85118e0 2024-04-02T14:35:32.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.802+0000 7f196ea00640 1 --2- 172.21.15.90:0/3078664971 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d8f851a4c0 0x55d8f851c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:35:32.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.802+0000 7f1977400640 1 --2- 172.21.15.90:0/3078664971 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d8f85118e0 0x55d8f8519f80 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f196800bb10 tx=0x7f196800d340 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f8888634b427bb64 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:35:32.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.803+0000 7f1967e00640 1 -- 172.21.15.90:0/3078664971 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1968017420 con 0x55d8f85118e0 2024-04-02T14:35:32.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.803+0000 7f1967e00640 1 -- 172.21.15.90:0/3078664971 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1968017aa0 con 0x55d8f85118e0 2024-04-02T14:35:32.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.803+0000 7f1967e00640 1 -- 172.21.15.90:0/3078664971 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f196800fb20 con 0x55d8f85118e0 2024-04-02T14:35:32.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.804+0000 7f1980da8740 1 -- 172.21.15.90:0/3078664971 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d8f85118e0 msgr2=0x55d8f8519f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:35:32.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.804+0000 7f1980da8740 1 --2- 172.21.15.90:0/3078664971 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d8f85118e0 0x55d8f8519f80 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f196800bb10 tx=0x7f196800d340 comp rx=0 tx=0).stop 2024-04-02T14:35:32.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.804+0000 7f1980da8740 1 -- 172.21.15.90:0/3078664971 shutdown_connections 2024-04-02T14:35:32.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.804+0000 7f1980da8740 1 --2- 172.21.15.90:0/3078664971 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d8f851a4c0 0x55d8f851c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:32.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.804+0000 7f1980da8740 1 --2- 172.21.15.90:0/3078664971 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d8f85118e0 0x55d8f8519f80 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:32.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.804+0000 7f1980da8740 1 --2- 172.21.15.90:0/3078664971 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d8f8510fc0 0x55d8f85113a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:32.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.804+0000 7f1980da8740 1 -- 172.21.15.90:0/3078664971 >> 172.21.15.90:0/3078664971 conn(0x55d8f850b6f0 msgr2=0x55d8f850bad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:35:32.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.804+0000 7f1980da8740 1 -- 172.21.15.90:0/3078664971 shutdown_connections 2024-04-02T14:35:32.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.804+0000 7f1980da8740 1 -- 172.21.15.90:0/3078664971 wait complete. 2024-04-02T14:35:32.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.805+0000 7f1980da8740 1 -- start start 2024-04-02T14:35:32.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.805+0000 7f1980da8740 1 Processor -- start 2024-04-02T14:35:32.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.806+0000 7f1980da8740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d8f8510fc0 0x55d8f8517520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:32.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.806+0000 7f1980da8740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d8f85118e0 0x55d8f8517a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:32.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.806+0000 7f1980da8740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d8f851a4c0 0x55d8f8517fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:32.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.806+0000 7f1980da8740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55d8f851d2b0 con 0x55d8f8510fc0 2024-04-02T14:35:32.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.806+0000 7f1980da8740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55d8f8427ac0 con 0x55d8f85118e0 2024-04-02T14:35:32.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.806+0000 7f1980da8740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55d8f851d430 con 0x55d8f851a4c0 2024-04-02T14:35:32.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.806+0000 7f1980da8740 10 client.0 _ll_register_callbacks cb 0x55d8f842a180 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:35:32.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.807+0000 7f196ea00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d8f8510fc0 0x55d8f8517520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:32.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.807+0000 7f196ea00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d8f8510fc0 0x55d8f8517520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:57604/0 (socket says 172.21.15.90:57604) 2024-04-02T14:35:32.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.807+0000 7f196ea00640 1 -- 172.21.15.90:0/163641828 learned_addr learned my addr 172.21.15.90:0/163641828 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:35:32.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.807+0000 7f1977400640 1 --2- 172.21.15.90:0/163641828 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d8f85118e0 0x55d8f8517a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:32.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.807+0000 7f196ea00640 1 --2- 172.21.15.90:0/163641828 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d8f851a4c0 0x55d8f8517fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:32.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.808+0000 7f196ea00640 1 -- 172.21.15.90:0/163641828 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d8f85118e0 msgr2=0x55d8f8517a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:35:32.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.808+0000 7f196ea00640 1 --2- 172.21.15.90:0/163641828 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d8f85118e0 0x55d8f8517a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:32.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.808+0000 7f196ea00640 1 -- 172.21.15.90:0/163641828 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d8f8510fc0 msgr2=0x55d8f8517520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:35:32.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.808+0000 7f196ea00640 1 --2- 172.21.15.90:0/163641828 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d8f8510fc0 0x55d8f8517520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:32.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.808+0000 7f196ea00640 1 -- 172.21.15.90:0/163641828 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d8f81803f0 con 0x55d8f851a4c0 2024-04-02T14:35:32.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.808+0000 7f196ea00640 1 --2- 172.21.15.90:0/163641828 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d8f851a4c0 0x55d8f8517fa0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f1960014dc0 tx=0x7f1960004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9acc43ad45afc2fb server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:35:32.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.808+0000 7f1964c00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:35:32.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.808+0000 7f1964c00640 1 -- 172.21.15.90:0/163641828 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1960014110 con 0x55d8f851a4c0 2024-04-02T14:35:32.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.809+0000 7f1964c00640 1 -- 172.21.15.90:0/163641828 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f1960003970 con 0x55d8f851a4c0 2024-04-02T14:35:32.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.809+0000 7f1980da8740 1 -- 172.21.15.90:0/163641828 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1968017aa0 con 0x55d8f851a4c0 2024-04-02T14:35:32.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.809+0000 7f1964c00640 1 -- 172.21.15.90:0/163641828 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f1960014640 con 0x55d8f851a4c0 2024-04-02T14:35:32.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.809+0000 7f1980da8740 10 client.25537 fetch_fsmap learned FSMap version 27 2024-04-02T14:35:32.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.809+0000 7f1980da8740 1 -- 172.21.15.90:0/163641828 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55d8f81400f0 con 0x55d8f851a4c0 2024-04-02T14:35:32.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.810+0000 7f1964c00640 1 -- 172.21.15.90:0/163641828 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1960003b70 con 0x55d8f851a4c0 2024-04-02T14:35:32.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.810+0000 7f1964c00640 1 -- 172.21.15.90:0/163641828 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f1960027020 con 0x55d8f851a4c0 2024-04-02T14:35:32.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.811+0000 7f1980da8740 10 client.25537 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:35:32.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.811+0000 7f1980da8740 10 client.25537 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:35:32.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.811+0000 7f1980da8740 20 client.25537 populate_metadata read hostname 'smithi090' 2024-04-02T14:35:32.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.811+0000 7f1980da8740 1 --2- 172.21.15.90:0/163641828 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55d8f852a4d0 0x55d8f852c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:32.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.811+0000 7f1980da8740 4 client.25537 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:35:32.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.811+0000 7f1980da8740 1 -- 172.21.15.90:0/163641828 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f1968017aa0 con 0x55d8f852a4d0 2024-04-02T14:35:32.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.811+0000 7f1977400640 1 --2- 172.21.15.90:0/163641828 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55d8f852a4d0 0x55d8f852c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:32.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.812+0000 7f1977400640 1 --2- 172.21.15.90:0/163641828 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55d8f852a4d0 0x55d8f852c8f0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6ec0db4133f35da3 server_cookie=39bd564093deaeab in_seq=0 out_seq=0 2024-04-02T14:35:32.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.812+0000 7f1964c00640 10 client.25537 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:35:32.815 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.813+0000 7f1964c00640 1 -- 172.21.15.90:0/163641828 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x7f1968017aa0 con 0x55d8f852a4d0 2024-04-02T14:35:32.815 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.813+0000 7f1964c00640 10 client.25537 handle_command_reply: tid=0 2024-04-02T14:35:32.815 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.813+0000 7f1980da8740 1 client.25537 shutdown 2024-04-02T14:35:32.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.814+0000 7f1980da8740 1 -- 172.21.15.90:0/163641828 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d8f851a4c0 msgr2=0x55d8f8517fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:35:32.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.814+0000 7f1980da8740 1 --2- 172.21.15.90:0/163641828 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d8f851a4c0 0x55d8f8517fa0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f1960014dc0 tx=0x7f1960004a40 comp rx=0 tx=0).stop 2024-04-02T14:35:32.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.814+0000 7f1980da8740 1 -- 172.21.15.90:0/163641828 shutdown_connections 2024-04-02T14:35:32.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.814+0000 7f1980da8740 1 --2- 172.21.15.90:0/163641828 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55d8f852a4d0 0x55d8f852c8f0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:32.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.814+0000 7f1980da8740 1 --2- 172.21.15.90:0/163641828 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d8f851a4c0 0x55d8f8517fa0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:32.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.814+0000 7f1980da8740 1 --2- 172.21.15.90:0/163641828 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d8f85118e0 0x55d8f8517a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:32.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.814+0000 7f1980da8740 1 --2- 172.21.15.90:0/163641828 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d8f8510fc0 0x55d8f8517520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:32.817 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.814+0000 7f1980da8740 1 -- 172.21.15.90:0/163641828 >> 172.21.15.90:0/163641828 conn(0x55d8f850b6f0 msgr2=0x55d8f850d440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:35:32.817 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.814+0000 7f1964c00640 0 client.25537 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:35:32.817 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.815+0000 7f1980da8740 1 -- 172.21.15.90:0/163641828 shutdown_connections 2024-04-02T14:35:32.817 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.815+0000 7f1980da8740 1 -- 172.21.15.90:0/163641828 wait complete. 2024-04-02T14:35:32.817 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.815+0000 7f1980da8740 20 client.25537 trim_cache size 0 max 16384 2024-04-02T14:35:32.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.845+0000 7f1980da8740 1 Processor -- start 2024-04-02T14:35:32.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.845+0000 7f1980da8740 1 -- start start 2024-04-02T14:35:32.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.846+0000 7f1980da8740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d8f850b6f0 0x55d8f85261a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:32.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.846+0000 7f1980da8740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d8f8510fc0 0x55d8f8532180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:32.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.846+0000 7f1980da8740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d8f85118e0 0x55d8f85326c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:32.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.846+0000 7f1980da8740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f197806fa40 con 0x55d8f850b6f0 2024-04-02T14:35:32.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.846+0000 7f1980da8740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f1978071580 con 0x55d8f85118e0 2024-04-02T14:35:32.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.846+0000 7f1980da8740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f1978070110 con 0x55d8f8510fc0 2024-04-02T14:35:32.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.846+0000 7f196ea00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d8f850b6f0 0x55d8f85261a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:32.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.846+0000 7f1977400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d8f8510fc0 0x55d8f8532180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:32.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.847+0000 7f196ea00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d8f850b6f0 0x55d8f85261a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:57606/0 (socket says 172.21.15.90:57606) 2024-04-02T14:35:32.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.847+0000 7f196ea00640 1 -- 172.21.15.90:0/2942216483 learned_addr learned my addr 172.21.15.90:0/2942216483 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:35:32.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.847+0000 7f196ea00640 1 --2- 172.21.15.90:0/2942216483 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d8f85118e0 0x55d8f85326c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:32.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.848+0000 7f1977400640 1 -- 172.21.15.90:0/2942216483 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d8f85118e0 msgr2=0x55d8f85326c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:35:32.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.848+0000 7f1977400640 1 --2- 172.21.15.90:0/2942216483 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d8f85118e0 0x55d8f85326c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:32.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.848+0000 7f1977400640 1 -- 172.21.15.90:0/2942216483 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d8f850b6f0 msgr2=0x55d8f85261a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:35:32.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.848+0000 7f1977400640 1 --2- 172.21.15.90:0/2942216483 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d8f850b6f0 0x55d8f85261a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:32.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.848+0000 7f1977400640 1 -- 172.21.15.90:0/2942216483 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d8f852cf50 con 0x55d8f8510fc0 2024-04-02T14:35:32.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.848+0000 7f1977400640 1 --2- 172.21.15.90:0/2942216483 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d8f8510fc0 0x55d8f8532180 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f1968000c00 tx=0x7f196800d340 comp rx=0 tx=0).ready entity=mon.1 client_cookie=46a02d766191d526 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:35:32.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.849+0000 7f1967400640 1 -- 172.21.15.90:0/2942216483 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f196802c070 con 0x55d8f8510fc0 2024-04-02T14:35:32.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.849+0000 7f1967400640 1 -- 172.21.15.90:0/2942216483 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f196800f420 con 0x55d8f8510fc0 2024-04-02T14:35:32.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.849+0000 7f1967400640 1 -- 172.21.15.90:0/2942216483 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19680282d0 con 0x55d8f8510fc0 2024-04-02T14:35:32.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.850+0000 7f1980da8740 1 -- 172.21.15.90:0/2942216483 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d8f8510fc0 msgr2=0x55d8f8532180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:35:32.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.850+0000 7f1980da8740 1 --2- 172.21.15.90:0/2942216483 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d8f8510fc0 0x55d8f8532180 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f1968000c00 tx=0x7f196800d340 comp rx=0 tx=0).stop 2024-04-02T14:35:32.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.850+0000 7f1980da8740 1 -- 172.21.15.90:0/2942216483 shutdown_connections 2024-04-02T14:35:32.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.850+0000 7f1980da8740 1 --2- 172.21.15.90:0/2942216483 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d8f85118e0 0x55d8f85326c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:32.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.850+0000 7f1980da8740 1 --2- 172.21.15.90:0/2942216483 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d8f8510fc0 0x55d8f8532180 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:32.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.850+0000 7f1980da8740 1 --2- 172.21.15.90:0/2942216483 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d8f850b6f0 0x55d8f85261a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:32.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.850+0000 7f1980da8740 1 -- 172.21.15.90:0/2942216483 >> 172.21.15.90:0/2942216483 conn(0x55d8f8516160 msgr2=0x55d8f8501f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:35:32.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.851+0000 7f1980da8740 1 -- 172.21.15.90:0/2942216483 shutdown_connections 2024-04-02T14:35:32.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.851+0000 7f1980da8740 1 -- 172.21.15.90:0/2942216483 wait complete. 2024-04-02T14:35:32.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.851+0000 7f1980da8740 1 -- start start 2024-04-02T14:35:32.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.852+0000 7f1980da8740 1 Processor -- start 2024-04-02T14:35:32.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.853+0000 7f1980da8740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d8f850b6f0 0x55d8f851e4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:32.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.853+0000 7f1980da8740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d8f8510fc0 0x55d8f851e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:32.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.853+0000 7f1980da8740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d8f85118e0 0x55d8f858f9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:32.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.853+0000 7f1980da8740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f1978070eb0 con 0x55d8f850b6f0 2024-04-02T14:35:32.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.853+0000 7f1980da8740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f19780707e0 con 0x55d8f85118e0 2024-04-02T14:35:32.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.853+0000 7f1980da8740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f197806da40 con 0x55d8f8510fc0 2024-04-02T14:35:32.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.853+0000 7f1980da8740 10 client.0 _ll_register_callbacks cb 0x7f1960014510 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:35:32.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.853+0000 7f196ea00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d8f850b6f0 0x55d8f851e4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:32.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.854+0000 7f196ea00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d8f850b6f0 0x55d8f851e4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:57620/0 (socket says 172.21.15.90:57620) 2024-04-02T14:35:32.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.854+0000 7f196ea00640 1 -- 172.21.15.90:0/1154248061 learned_addr learned my addr 172.21.15.90:0/1154248061 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:35:32.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.854+0000 7f1977400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d8f8510fc0 0x55d8f851e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:32.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.855+0000 7f1977400640 1 -- 172.21.15.90:0/1154248061 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d8f85118e0 msgr2=0x55d8f858f9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:35:32.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.855+0000 7f196ea00640 1 --2- 172.21.15.90:0/1154248061 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d8f85118e0 0x55d8f858f9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:32.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.855+0000 7f1977400640 1 --2- 172.21.15.90:0/1154248061 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d8f85118e0 0x55d8f858f9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:32.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.855+0000 7f1977400640 1 -- 172.21.15.90:0/1154248061 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d8f850b6f0 msgr2=0x55d8f851e4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:35:32.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.855+0000 7f1977400640 1 --2- 172.21.15.90:0/1154248061 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d8f850b6f0 0x55d8f851e4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:32.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.855+0000 7f1977400640 1 -- 172.21.15.90:0/1154248061 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d8f851c8b0 con 0x55d8f8510fc0 2024-04-02T14:35:32.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.855+0000 7f196ea00640 1 --2- 172.21.15.90:0/1154248061 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d8f85118e0 0x55d8f858f9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:35:32.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.856+0000 7f1964c00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:35:32.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.857+0000 7f1977400640 1 --2- 172.21.15.90:0/1154248061 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d8f8510fc0 0x55d8f851e9f0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x55d8f8511840 tx=0x7f196800f660 comp rx=0 tx=0).ready entity=mon.1 client_cookie=542287e8fd88e140 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:35:32.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.857+0000 7f1964c00640 1 -- 172.21.15.90:0/1154248061 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f196802c040 con 0x55d8f8510fc0 2024-04-02T14:35:32.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.858+0000 7f1964c00640 1 -- 172.21.15.90:0/1154248061 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f1968005020 con 0x55d8f8510fc0 2024-04-02T14:35:32.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.858+0000 7f1980da8740 1 -- 172.21.15.90:0/1154248061 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f196800f420 con 0x55d8f8510fc0 2024-04-02T14:35:32.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.859+0000 7f1964c00640 1 -- 172.21.15.90:0/1154248061 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f1968005260 con 0x55d8f8510fc0 2024-04-02T14:35:32.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.859+0000 7f1980da8740 10 client.25543 fetch_fsmap learned FSMap version 27 2024-04-02T14:35:32.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.859+0000 7f1980da8740 1 -- 172.21.15.90:0/1154248061 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55d8f8590ea0 con 0x55d8f8510fc0 2024-04-02T14:35:32.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.860+0000 7f1964c00640 1 -- 172.21.15.90:0/1154248061 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f196800f420 con 0x55d8f8510fc0 2024-04-02T14:35:32.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.860+0000 7f1964c00640 1 -- 172.21.15.90:0/1154248061 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f19680047a0 con 0x55d8f8510fc0 2024-04-02T14:35:32.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.860+0000 7f1980da8740 10 client.25543 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:35:32.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.860+0000 7f1980da8740 10 client.25543 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:35:32.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.860+0000 7f1980da8740 20 client.25543 populate_metadata read hostname 'smithi090' 2024-04-02T14:35:32.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.860+0000 7f1980da8740 1 --2- 172.21.15.90:0/1154248061 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55d8f851a4c0 0x55d8f85933e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:32.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.860+0000 7f1980da8740 4 client.25543 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-02T14:35:32.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.860+0000 7f1980da8740 1 -- 172.21.15.90:0/1154248061 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55d8f840d630 con 0x55d8f851a4c0 2024-04-02T14:35:32.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.862+0000 7f196ea00640 1 --2- 172.21.15.90:0/1154248061 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55d8f851a4c0 0x55d8f85933e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:32.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.865+0000 7f1964c00640 10 client.25543 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:35:32.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.865+0000 7f196ea00640 1 --2- 172.21.15.90:0/1154248061 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55d8f851a4c0 0x55d8f85933e0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e8b8fa196d7718b9 server_cookie=321b22d6deaccdd8 in_seq=0 out_seq=0 2024-04-02T14:35:32.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.865+0000 7f1964c00640 1 -- 172.21.15.90:0/1154248061 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55d8f840d630 con 0x55d8f851a4c0 2024-04-02T14:35:32.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.865+0000 7f1964c00640 10 client.25543 handle_command_reply: tid=0 2024-04-02T14:35:32.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.865+0000 7f1980da8740 1 client.25543 shutdown 2024-04-02T14:35:32.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.866+0000 7f1980da8740 1 -- 172.21.15.90:0/1154248061 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d8f8510fc0 msgr2=0x55d8f851e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:35:32.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.866+0000 7f1980da8740 1 --2- 172.21.15.90:0/1154248061 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d8f8510fc0 0x55d8f851e9f0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x55d8f8511840 tx=0x7f196800f660 comp rx=0 tx=0).stop 2024-04-02T14:35:32.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.866+0000 7f1980da8740 1 -- 172.21.15.90:0/1154248061 shutdown_connections 2024-04-02T14:35:32.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.866+0000 7f1980da8740 1 --2- 172.21.15.90:0/1154248061 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55d8f851a4c0 0x55d8f85933e0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:32.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.866+0000 7f1980da8740 1 --2- 172.21.15.90:0/1154248061 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d8f85118e0 0x55d8f858f9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:32.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.867+0000 7f1980da8740 1 --2- 172.21.15.90:0/1154248061 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d8f8510fc0 0x55d8f851e9f0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:32.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.867+0000 7f1980da8740 1 --2- 172.21.15.90:0/1154248061 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d8f850b6f0 0x55d8f851e4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:32.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.867+0000 7f1980da8740 1 -- 172.21.15.90:0/1154248061 >> 172.21.15.90:0/1154248061 conn(0x55d8f8516160 msgr2=0x55d8f850c8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:35:32.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.867+0000 7f1964c00640 0 client.25543 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:35:32.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.867+0000 7f1980da8740 1 -- 172.21.15.90:0/1154248061 shutdown_connections 2024-04-02T14:35:32.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.867+0000 7f1980da8740 1 -- 172.21.15.90:0/1154248061 wait complete. 2024-04-02T14:35:32.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.868+0000 7f1980da8740 20 client.25543 trim_cache size 0 max 16384 2024-04-02T14:35:32.870 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-02T14:35:32.870 INFO:teuthology.orchestra.run.smithi090.stdout: "status": "no active scrubs running", 2024-04-02T14:35:32.870 INFO:teuthology.orchestra.run.smithi090.stdout: "scrubs": {} 2024-04-02T14:35:32.870 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-02T14:35:32.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.869+0000 7f197e000640 1 -- 172.21.15.90:0/2144284824 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f195c0a3f10 msgr2=0x7f195c0a63d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:35:32.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.869+0000 7f197e000640 1 --2- 172.21.15.90:0/2144284824 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f195c0a3f10 0x7f195c0a63d0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f19700efd90 tx=0x7f1970005fd0 comp rx=0 tx=0).stop 2024-04-02T14:35:32.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.869+0000 7f197e000640 1 -- 172.21.15.90:0/2144284824 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f197810e4a0 msgr2=0x7f197819ed60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:35:32.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.869+0000 7f197e000640 1 --2- 172.21.15.90:0/2144284824 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f197810e4a0 0x7f197819ed60 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f1970036040 tx=0x7f19700045e0 comp rx=0 tx=0).stop 2024-04-02T14:35:32.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.869+0000 7f197e000640 1 -- 172.21.15.90:0/2144284824 shutdown_connections 2024-04-02T14:35:32.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.869+0000 7f197e000640 1 --2- 172.21.15.90:0/2144284824 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f195c0a3f10 0x7f195c0a63d0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:32.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.869+0000 7f197e000640 1 --2- 172.21.15.90:0/2144284824 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f197810e4a0 0x7f197819ed60 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:32.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.869+0000 7f197e000640 1 --2- 172.21.15.90:0/2144284824 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1978069600 0x7f197819a9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:32.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.869+0000 7f197e000640 1 --2- 172.21.15.90:0/2144284824 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1978068ce0 0x7f197819a4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:32.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.869+0000 7f197e000640 1 -- 172.21.15.90:0/2144284824 >> 172.21.15.90:0/2144284824 conn(0x7f1978077b00 msgr2=0x7f197810f830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:35:32.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.869+0000 7f197e000640 1 -- 172.21.15.90:0/2144284824 shutdown_connections 2024-04-02T14:35:32.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:32.869+0000 7f197e000640 1 -- 172.21.15.90:0/2144284824 wait complete. 2024-04-02T14:35:32.902 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-02T14:35:32.903 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T14:35:33.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:33.188+0000 7f7d86c00640 1 -- 172.21.15.90:0/1845554064 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7d80076030 msgr2=0x7f7d80076410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:35:33.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:33.188+0000 7f7d86c00640 1 --2- 172.21.15.90:0/1845554064 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7d80076030 0x7f7d80076410 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f7d70009930 tx=0x7f7d7002bc30 comp rx=0 tx=0).stop 2024-04-02T14:35:33.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:33.188+0000 7f7d86c00640 1 -- 172.21.15.90:0/1845554064 shutdown_connections 2024-04-02T14:35:33.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:33.188+0000 7f7d86c00640 1 --2- 172.21.15.90:0/1845554064 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7d80112930 0x7f7d80114d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:33.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:33.188+0000 7f7d86c00640 1 --2- 172.21.15.90:0/1845554064 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7d80076950 0x7f7d801123f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:33.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:33.188+0000 7f7d86c00640 1 --2- 172.21.15.90:0/1845554064 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7d80076030 0x7f7d80076410 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:33.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:33.188+0000 7f7d86c00640 1 -- 172.21.15.90:0/1845554064 >> 172.21.15.90:0/1845554064 conn(0x7f7d800ff760 msgr2=0x7f7d80101b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:35:33.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:33.189+0000 7f7d86c00640 1 -- 172.21.15.90:0/1845554064 shutdown_connections 2024-04-02T14:35:33.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:33.189+0000 7f7d86c00640 1 -- 172.21.15.90:0/1845554064 wait complete. 2024-04-02T14:35:33.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:33.190+0000 7f7d86c00640 1 Processor -- start 2024-04-02T14:35:33.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:33.191+0000 7f7d86c00640 1 -- start start 2024-04-02T14:35:33.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:33.191+0000 7f7d86c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7d80076030 0x7f7d80111f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:33.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:33.191+0000 7f7d86c00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7d80076950 0x7f7d8010b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:33.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:33.191+0000 7f7d86c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7d80112930 0x7f7d8010b550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:33.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:33.191+0000 7f7d7f400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7d80076950 0x7f7d8010b010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:33.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:33.191+0000 7f7d7f400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7d80076950 0x7f7d8010b010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:35230/0 (socket says 172.21.15.90:35230) 2024-04-02T14:35:33.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:33.192+0000 7f7d7fe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7d80076030 0x7f7d80111f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:33.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:33.192+0000 7f7d7fe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7d80076030 0x7f7d80111f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:57034/0 (socket says 172.21.15.90:57034) 2024-04-02T14:35:33.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:33.192+0000 7f7d86c00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f7d801184d0 con 0x7f7d80112930 2024-04-02T14:35:33.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:33.192+0000 7f7d86c00640 1 -- 172.21.15.90:0/3486313092 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f7d80118350 con 0x7f7d80076950 2024-04-02T14:35:33.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:33.192+0000 7f7d7f400640 1 -- 172.21.15.90:0/3486313092 learned_addr learned my addr 172.21.15.90:0/3486313092 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:35:33.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:33.192+0000 7f7d86c00640 1 -- 172.21.15.90:0/3486313092 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f7d80118650 con 0x7f7d80076030 2024-04-02T14:35:33.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:33.192+0000 7f7d84c00640 1 --2- 172.21.15.90:0/3486313092 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7d80112930 0x7f7d8010b550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:33.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:33.192+0000 7f7d7f400640 1 -- 172.21.15.90:0/3486313092 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7d80076030 msgr2=0x7f7d80111f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:35:33.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:33.192+0000 7f7d7f400640 1 --2- 172.21.15.90:0/3486313092 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7d80076030 0x7f7d80111f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:33.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:33.192+0000 7f7d7f400640 1 -- 172.21.15.90:0/3486313092 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7d80112930 msgr2=0x7f7d8010b550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:35:33.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:33.192+0000 7f7d7f400640 1 --2- 172.21.15.90:0/3486313092 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7d80112930 0x7f7d8010b550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:33.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:33.192+0000 7f7d7f400640 1 -- 172.21.15.90:0/3486313092 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d8010bb40 con 0x7f7d80076950 2024-04-02T14:35:33.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:33.193+0000 7f7d7f400640 1 --2- 172.21.15.90:0/3486313092 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7d80076950 0x7f7d8010b010 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f7d6c00d950 tx=0x7f7d6c00de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:35:33.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:33.194+0000 7f7d7cc00640 1 -- 172.21.15.90:0/3486313092 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d6c017070 con 0x7f7d80076950 2024-04-02T14:35:33.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:33.194+0000 7f7d7cc00640 1 -- 172.21.15.90:0/3486313092 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7d6c009c30 con 0x7f7d80076950 2024-04-02T14:35:33.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:33.195+0000 7f7d86c00640 1 -- 172.21.15.90:0/3486313092 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d8010be30 con 0x7f7d80076950 2024-04-02T14:35:33.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:33.195+0000 7f7d86c00640 1 -- 172.21.15.90:0/3486313092 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d8010fde0 con 0x7f7d80076950 2024-04-02T14:35:33.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:33.195+0000 7f7d7cc00640 1 -- 172.21.15.90:0/3486313092 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d6c00f6b0 con 0x7f7d80076950 2024-04-02T14:35:33.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:33.198+0000 7f7d86c00640 1 -- 172.21.15.90:0/3486313092 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7d80077770 con 0x7f7d80076950 2024-04-02T14:35:33.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:33.202+0000 7f7d7cc00640 1 -- 172.21.15.90:0/3486313092 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7f7d6c00f850 con 0x7f7d80076950 2024-04-02T14:35:33.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:33.204+0000 7f7d7cc00640 1 --2- 172.21.15.90:0/3486313092 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f7d640a3e70 0x7f7d640a6330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:33.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:33.204+0000 7f7d7cc00640 1 -- 172.21.15.90:0/3486313092 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f7d6c0cfe00 con 0x7f7d80076950 2024-04-02T14:35:33.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:33.205+0000 7f7d7fe00640 1 --2- 172.21.15.90:0/3486313092 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f7d640a3e70 0x7f7d640a6330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:33.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:33.207+0000 7f7d7fe00640 1 --2- 172.21.15.90:0/3486313092 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f7d640a3e70 0x7f7d640a6330 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f7d70009900 tx=0x7f7d7002df90 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:35:33.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:33.209+0000 7f7d7cc00640 1 -- 172.21.15.90:0/3486313092 <== mon.2 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7f7d6c009de0 con 0x7f7d80076950 2024-04-02T14:35:33.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:33.675+0000 7f7d86c00640 1 -- 172.21.15.90:0/3486313092 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7d8010c4f0 con 0x7f7d80076950 2024-04-02T14:35:33.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:33.676+0000 7f7d7cc00640 1 -- 172.21.15.90:0/3486313092 <== mon.2 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11927 (secure 0 0 0) 0x7f7d6c09bc90 con 0x7f7d80076950 2024-04-02T14:35:33.679 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-02T14:35:33.679 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14412,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6837/2852160309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6836","nonce":2852160309},{"type":"v1","addr":"172.21.15.90:6837","nonce":2852160309}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14478,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6841/1919376858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6840","nonce":1919376858},{"type":"v1","addr":"172.21.15.90:6841","nonce":1919376858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-02T14:32:22.319083+0000","modified":"2024-04-02T14:34:52.266152+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24590,"mds_1":24560,"mds_2":24547,"mds_3":24539,"mds_4":24523},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24454":{"gid":24454,"name":"l","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6833/1564319431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":1564319431},{"type":"v1","addr":"172.21.15.177:6833","nonce":1564319431}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.92:6833/1570337372","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":1570337372},{"type":"v1","addr":"172.21.15.92:6833","nonce":1570337372}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"j","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.90:6835/1277124891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6834","nonce":1277124891},{"type":"v1","addr":"172.21.15.90:6835","nonce":1277124891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.92:6835/860721164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":860721164},{"type":"v1","addr":"172.21.15.92:6835","nonce":860721164}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24511":{"gid":24511,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6835/3177244988","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":3177244988},{"type":"v1","addr":"172.21.15.177:6835","nonce":3177244988}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"c","rank":4,"incarnation":25,"state":"up:active","state_seq":9,"addr":"172.21.15.177:6837/4223414858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":4223414858},{"type":"v1","addr":"172.21.15.177:6837","nonce":4223414858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24539":{"gid":24539,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":8,"addr":"172.21.15.92:6837/1270635939","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":1270635939},{"type":"v1","addr":"172.21.15.92:6837","nonce":1270635939}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24547":{"gid":24547,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":7,"addr":"172.21.15.177:6839/664895663","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":664895663},{"type":"v1","addr":"172.21.15.177:6839","nonce":664895663}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"g","rank":1,"incarnation":19,"state":"up:active","state_seq":5,"addr":"172.21.15.90:6839/1498915870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6838","nonce":1498915870},{"type":"v1","addr":"172.21.15.90:6839","nonce":1498915870}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24590":{"gid":24590,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.92:6839/1054888219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":1054888219},{"type":"v1","addr":"172.21.15.92:6839","nonce":1054888219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24590,"qdb_cluster":[24547,24590,24523,24539,24560]},"id":1}]} 2024-04-02T14:35:33.679 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 27 2024-04-02T14:35:33.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:33.680+0000 7f7d86c00640 1 -- 172.21.15.90:0/3486313092 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f7d640a3e70 msgr2=0x7f7d640a6330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:35:33.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:33.680+0000 7f7d86c00640 1 --2- 172.21.15.90:0/3486313092 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f7d640a3e70 0x7f7d640a6330 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f7d70009900 tx=0x7f7d7002df90 comp rx=0 tx=0).stop 2024-04-02T14:35:33.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:33.680+0000 7f7d86c00640 1 -- 172.21.15.90:0/3486313092 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7d80076950 msgr2=0x7f7d8010b010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:35:33.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:33.680+0000 7f7d86c00640 1 --2- 172.21.15.90:0/3486313092 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7d80076950 0x7f7d8010b010 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f7d6c00d950 tx=0x7f7d6c00de10 comp rx=0 tx=0).stop 2024-04-02T14:35:33.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:33.681+0000 7f7d86c00640 1 -- 172.21.15.90:0/3486313092 shutdown_connections 2024-04-02T14:35:33.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:33.681+0000 7f7d86c00640 1 --2- 172.21.15.90:0/3486313092 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f7d640a3e70 0x7f7d640a6330 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:33.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:33.681+0000 7f7d86c00640 1 --2- 172.21.15.90:0/3486313092 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7d80112930 0x7f7d8010b550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:33.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:33.681+0000 7f7d86c00640 1 --2- 172.21.15.90:0/3486313092 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7d80076950 0x7f7d8010b010 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:33.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:33.681+0000 7f7d86c00640 1 --2- 172.21.15.90:0/3486313092 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7d80076030 0x7f7d80111f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:33.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:33.681+0000 7f7d86c00640 1 -- 172.21.15.90:0/3486313092 >> 172.21.15.90:0/3486313092 conn(0x7f7d800ff760 msgr2=0x7f7d80113ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:35:33.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:33.681+0000 7f7d86c00640 1 -- 172.21.15.90:0/3486313092 shutdown_connections 2024-04-02T14:35:33.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:33.681+0000 7f7d86c00640 1 -- 172.21.15.90:0/3486313092 wait complete. 2024-04-02T14:35:33.713 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-04-02T14:35:33.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:33.991+0000 7f8ae5400640 1 -- 172.21.15.90:0/2213434596 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8ae00747f0 msgr2=0x7f8ae010dde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:35:33.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:33.991+0000 7f8ae5400640 1 --2- 172.21.15.90:0/2213434596 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8ae00747f0 0x7f8ae010dde0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f8ad0009ca0 tx=0x7f8ad002c180 comp rx=0 tx=0).stop 2024-04-02T14:35:33.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:33.991+0000 7f8ae5400640 1 -- 172.21.15.90:0/2213434596 shutdown_connections 2024-04-02T14:35:33.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:33.991+0000 7f8ae5400640 1 --2- 172.21.15.90:0/2213434596 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8ae010e320 0x7f8ae0110730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:33.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:33.991+0000 7f8ae5400640 1 --2- 172.21.15.90:0/2213434596 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8ae00747f0 0x7f8ae010dde0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:33.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:33.991+0000 7f8ae5400640 1 --2- 172.21.15.90:0/2213434596 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8ae0073ed0 0x7f8ae00742b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:33.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:33.991+0000 7f8ae5400640 1 -- 172.21.15.90:0/2213434596 >> 172.21.15.90:0/2213434596 conn(0x7f8ae00fd570 msgr2=0x7f8ae00ff990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:35:33.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:33.992+0000 7f8ae5400640 1 -- 172.21.15.90:0/2213434596 shutdown_connections 2024-04-02T14:35:33.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:33.992+0000 7f8ae5400640 1 -- 172.21.15.90:0/2213434596 wait complete. 2024-04-02T14:35:33.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:33.993+0000 7f8ae5400640 1 Processor -- start 2024-04-02T14:35:33.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:33.994+0000 7f8ae5400640 1 -- start start 2024-04-02T14:35:33.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:33.994+0000 7f8ae5400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8ae0073ed0 0x7f8ae01a4760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:33.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:33.994+0000 7f8ae5400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8ae00747f0 0x7f8ae01a4ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:33.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:33.994+0000 7f8ae5400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8ae010e320 0x7f8ae010c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:33.997 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:33.994+0000 7f8ae5400640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f8ae0102c70 con 0x7f8ae0073ed0 2024-04-02T14:35:33.997 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:33.994+0000 7f8ae5400640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f8ae0102af0 con 0x7f8ae00747f0 2024-04-02T14:35:33.997 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:33.994+0000 7f8ae5400640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f8ae0102df0 con 0x7f8ae010e320 2024-04-02T14:35:33.998 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:33.995+0000 7f8ade800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8ae0073ed0 0x7f8ae01a4760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:33.998 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:33.995+0000 7f8ade800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8ae0073ed0 0x7f8ae01a4760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:57386/0 (socket says 172.21.15.90:57386) 2024-04-02T14:35:33.998 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:33.995+0000 7f8ade800640 1 -- 172.21.15.90:0/1099855190 learned_addr learned my addr 172.21.15.90:0/1099855190 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:35:33.998 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:33.995+0000 7f8adf200640 1 --2- 172.21.15.90:0/1099855190 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8ae010e320 0x7f8ae010c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:33.998 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:33.995+0000 7f8adde00640 1 --2- 172.21.15.90:0/1099855190 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8ae00747f0 0x7f8ae01a4ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:33.998 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:33.996+0000 7f8ade800640 1 -- 172.21.15.90:0/1099855190 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8ae010e320 msgr2=0x7f8ae010c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:35:33.998 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:33.996+0000 7f8ade800640 1 --2- 172.21.15.90:0/1099855190 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8ae010e320 0x7f8ae010c8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:33.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:33.996+0000 7f8ade800640 1 -- 172.21.15.90:0/1099855190 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8ae00747f0 msgr2=0x7f8ae01a4ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:35:33.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:33.996+0000 7f8ade800640 1 --2- 172.21.15.90:0/1099855190 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8ae00747f0 0x7f8ae01a4ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:33.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:33.996+0000 7f8ade800640 1 -- 172.21.15.90:0/1099855190 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ae010d0c0 con 0x7f8ae0073ed0 2024-04-02T14:35:34.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:33.996+0000 7f8ade800640 1 --2- 172.21.15.90:0/1099855190 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8ae0073ed0 0x7f8ae01a4760 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f8ad800ea60 tx=0x7f8ad800ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:35:34.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:33.997+0000 7f8ad7400640 1 -- 172.21.15.90:0/1099855190 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ad800cde0 con 0x7f8ae0073ed0 2024-04-02T14:35:34.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:33.997+0000 7f8ad7400640 1 -- 172.21.15.90:0/1099855190 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8ad80035f0 con 0x7f8ae0073ed0 2024-04-02T14:35:34.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:33.997+0000 7f8ad7400640 1 -- 172.21.15.90:0/1099855190 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ad8010670 con 0x7f8ae0073ed0 2024-04-02T14:35:34.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:33.997+0000 7f8ae5400640 1 -- 172.21.15.90:0/1099855190 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ae010d3b0 con 0x7f8ae0073ed0 2024-04-02T14:35:34.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:33.997+0000 7f8ae5400640 1 -- 172.21.15.90:0/1099855190 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ae01ab540 con 0x7f8ae0073ed0 2024-04-02T14:35:34.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.003+0000 7f8ae800e740 1 Processor -- start 2024-04-02T14:35:34.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.003+0000 7f8ae800e740 1 -- start start 2024-04-02T14:35:34.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.003+0000 7f8ae800e740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a2befd09f0 0x55a2befcf740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:34.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.003+0000 7f8ae800e740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a2befcfd10 0x55a2befd0190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:34.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.003+0000 7f8ae800e740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a2befd9080 0x55a2befdb470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:34.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.003+0000 7f8ae800e740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55a2bef10400 con 0x55a2befd9080 2024-04-02T14:35:34.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.003+0000 7f8ae800e740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55a2beef4f10 con 0x55a2befd09f0 2024-04-02T14:35:34.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.003+0000 7f8ae800e740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55a2bef346f0 con 0x55a2befcfd10 2024-04-02T14:35:34.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.004+0000 7f8adf200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a2befd09f0 0x55a2befcf740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:34.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.004+0000 7f8adf200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a2befd09f0 0x55a2befcf740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:57772/0 (socket says 172.21.15.90:57772) 2024-04-02T14:35:34.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.004+0000 7f8adf200640 1 -- 172.21.15.90:0/811144917 learned_addr learned my addr 172.21.15.90:0/811144917 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:35:34.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.004+0000 7f8adf200640 1 -- 172.21.15.90:0/811144917 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a2befcfd10 msgr2=0x55a2befd0190 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-02T14:35:34.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.004+0000 7f8adf200640 1 --2- 172.21.15.90:0/811144917 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a2befcfd10 0x55a2befd0190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:34.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.004+0000 7f8adf200640 1 -- 172.21.15.90:0/811144917 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a2befd9080 msgr2=0x55a2befdb470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:35:34.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.004+0000 7f8adf200640 1 --2- 172.21.15.90:0/811144917 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a2befd9080 0x55a2befdb470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:34.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.004+0000 7f8adf200640 1 -- 172.21.15.90:0/811144917 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a2bec483f0 con 0x55a2befd09f0 2024-04-02T14:35:34.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.004+0000 7f8adf200640 1 --2- 172.21.15.90:0/811144917 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a2befd09f0 0x55a2befcf740 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f8ac8009b80 tx=0x7f8ac8011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6437af5d76d738a server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:35:34.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.005+0000 7f8acf400640 1 -- 172.21.15.90:0/811144917 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ac8004030 con 0x55a2befd09f0 2024-04-02T14:35:34.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.005+0000 7f8acf400640 1 -- 172.21.15.90:0/811144917 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8ac80041d0 con 0x55a2befd09f0 2024-04-02T14:35:34.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.005+0000 7f8acf400640 1 -- 172.21.15.90:0/811144917 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ac80196d0 con 0x55a2befd09f0 2024-04-02T14:35:34.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.006+0000 7f8ae800e740 1 -- 172.21.15.90:0/811144917 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a2befd09f0 msgr2=0x55a2befcf740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:35:34.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.006+0000 7f8ae800e740 1 --2- 172.21.15.90:0/811144917 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a2befd09f0 0x55a2befcf740 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f8ac8009b80 tx=0x7f8ac8011810 comp rx=0 tx=0).stop 2024-04-02T14:35:34.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.006+0000 7f8ae800e740 1 -- 172.21.15.90:0/811144917 shutdown_connections 2024-04-02T14:35:34.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.006+0000 7f8ae800e740 1 --2- 172.21.15.90:0/811144917 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a2befd9080 0x55a2befdb470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:34.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.006+0000 7f8ae800e740 1 --2- 172.21.15.90:0/811144917 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a2befcfd10 0x55a2befd0190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:34.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.006+0000 7f8ae800e740 1 --2- 172.21.15.90:0/811144917 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a2befd09f0 0x55a2befcf740 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:34.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.006+0000 7f8ae800e740 1 -- 172.21.15.90:0/811144917 >> 172.21.15.90:0/811144917 conn(0x55a2bef96a50 msgr2=0x55a2befca760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:35:34.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.006+0000 7f8ad7400640 1 -- 172.21.15.90:0/1099855190 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7f8ad8004020 con 0x7f8ae0073ed0 2024-04-02T14:35:34.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.008+0000 7f8ad7400640 1 --2- 172.21.15.90:0/1099855190 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f8ac40a3f90 0x7f8ac40a6450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:34.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.010+0000 7f8adde00640 1 --2- 172.21.15.90:0/1099855190 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f8ac40a3f90 0x7f8ac40a6450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:34.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.012+0000 7f8adde00640 1 --2- 172.21.15.90:0/1099855190 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f8ac40a3f90 0x7f8ac40a6450 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f8ad0000f80 tx=0x7f8ad0009360 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:35:34.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.012+0000 7f8ad7400640 1 -- 172.21.15.90:0/1099855190 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f8ad809aaa0 con 0x7f8ae0073ed0 2024-04-02T14:35:34.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.012+0000 7f8ae800e740 1 -- 172.21.15.90:0/811144917 shutdown_connections 2024-04-02T14:35:34.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.012+0000 7f8ae800e740 1 -- 172.21.15.90:0/811144917 wait complete. 2024-04-02T14:35:34.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.013+0000 7f8ae800e740 1 -- start start 2024-04-02T14:35:34.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.014+0000 7f8ae800e740 1 Processor -- start 2024-04-02T14:35:34.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.015+0000 7f8ae800e740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a2befd09f0 0x55a2befd6050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:34.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.015+0000 7f8ae800e740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a2befd9080 0x55a2befd6590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:34.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.016+0000 7f8ade800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a2befd09f0 0x55a2befd6050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:34.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.016+0000 7f8ade800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a2befd09f0 0x55a2befd6050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:57408/0 (socket says 172.21.15.90:57408) 2024-04-02T14:35:34.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.016+0000 7f8ade800640 1 -- 172.21.15.90:0/1666679590 learned_addr learned my addr 172.21.15.90:0/1666679590 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:35:34.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.016+0000 7f8ae800e740 1 --2- 172.21.15.90:0/1666679590 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a2befd6ad0 0x55a2befe50f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:34.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.016+0000 7f8ae800e740 1 -- 172.21.15.90:0/1666679590 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55a2bef306a0 con 0x55a2befd09f0 2024-04-02T14:35:34.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.016+0000 7f8ae800e740 1 -- 172.21.15.90:0/1666679590 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55a2beedc980 con 0x55a2befd6ad0 2024-04-02T14:35:34.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.016+0000 7f8ae800e740 1 -- 172.21.15.90:0/1666679590 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55a2beeefcb0 con 0x55a2befd9080 2024-04-02T14:35:34.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.016+0000 7f8adde00640 1 --2- 172.21.15.90:0/1666679590 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a2befd9080 0x55a2befd6590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:34.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.016+0000 7f8adf200640 1 --2- 172.21.15.90:0/1666679590 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a2befd6ad0 0x55a2befe50f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:34.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.017+0000 7f8ae800e740 10 client.0 _ll_register_callbacks cb 0x55a2beeef6a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:35:34.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.017+0000 7f8adde00640 1 -- 172.21.15.90:0/1666679590 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a2befd6ad0 msgr2=0x55a2befe50f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:35:34.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.017+0000 7f8adde00640 1 --2- 172.21.15.90:0/1666679590 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a2befd6ad0 0x55a2befe50f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:34.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.017+0000 7f8adde00640 1 -- 172.21.15.90:0/1666679590 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a2befd09f0 msgr2=0x55a2befd6050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:35:34.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.017+0000 7f8adde00640 1 --2- 172.21.15.90:0/1666679590 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a2befd09f0 0x55a2befd6050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:34.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.017+0000 7f8adde00640 1 -- 172.21.15.90:0/1666679590 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a2bed127d0 con 0x55a2befd9080 2024-04-02T14:35:34.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.017+0000 7f8adf200640 1 --2- 172.21.15.90:0/1666679590 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a2befd6ad0 0x55a2befe50f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:35:34.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.017+0000 7f8ae800e740 1 -- 172.21.15.90:0/1666679590 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8ac80041d0 con 0x55a2befd9080 2024-04-02T14:35:34.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.017+0000 7f8adde00640 1 --2- 172.21.15.90:0/1666679590 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a2befd9080 0x55a2befd6590 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f8ad002c6c0 tx=0x7f8ad00095c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5757056d46effa57 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:35:34.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.018+0000 7f8ac3e00640 10 client.25558 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:35:34.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.018+0000 7f8ac3e00640 1 -- 172.21.15.90:0/1666679590 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ad004f020 con 0x55a2befd9080 2024-04-02T14:35:34.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.018+0000 7f8ac3e00640 1 -- 172.21.15.90:0/1666679590 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f8ad0004790 con 0x55a2befd9080 2024-04-02T14:35:34.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.018+0000 7f8ac3e00640 1 -- 172.21.15.90:0/1666679590 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f8ad0039040 con 0x55a2befd9080 2024-04-02T14:35:34.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.019+0000 7f8ae800e740 10 client.25558 fetch_fsmap learned FSMap version 27 2024-04-02T14:35:34.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.019+0000 7f8ae800e740 1 -- 172.21.15.90:0/1666679590 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({fsmap=27}) -- 0x55a2befe8300 con 0x55a2befd9080 2024-04-02T14:35:34.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.019+0000 7f8ac3e00640 1 -- 172.21.15.90:0/1666679590 <== mon.1 v2:172.21.15.177:3300/0 4 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f8ad0032670 con 0x55a2befd9080 2024-04-02T14:35:34.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.019+0000 7f8ae800e740 10 client.25558 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:35:34.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.019+0000 7f8ae800e740 10 client.25558 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.c 2024-04-02T14:35:34.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.019+0000 7f8ae800e740 20 client.25558 populate_metadata read hostname 'smithi090' 2024-04-02T14:35:34.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.019+0000 7f8ae800e740 1 --2- 172.21.15.90:0/1666679590 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x55a2befe96f0 0x55a2befebae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:34.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.019+0000 7f8ae800e740 4 client.25558 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:35:34.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.019+0000 7f8ae800e740 1 -- 172.21.15.90:0/1666679590 --> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a2beef4040 con 0x55a2befe96f0 2024-04-02T14:35:34.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.020+0000 7f8ade800640 1 --2- 172.21.15.90:0/1666679590 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x55a2befe96f0 0x55a2befebae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:34.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.021+0000 7f8ade800640 1 --2- 172.21.15.90:0/1666679590 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x55a2befe96f0 0x55a2befebae0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=8309509ba4f33167 server_cookie=2e281fbddfd81991 in_seq=0 out_seq=0 2024-04-02T14:35:34.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.021+0000 7f8ac3e00640 10 client.25558 ms_handle_connect on v2:172.21.15.177:6836/4223414858 2024-04-02T14:35:34.025 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.022+0000 7f8ac3e00640 1 -- 172.21.15.90:0/1666679590 <== mds.4 v2:172.21.15.177:6836/4223414858 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a2beef4040 con 0x55a2befe96f0 2024-04-02T14:35:34.025 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.022+0000 7f8ac3e00640 10 client.25558 handle_command_reply: tid=0 2024-04-02T14:35:34.025 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.023+0000 7f8ae800e740 1 client.25558 shutdown 2024-04-02T14:35:34.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.023+0000 7f8ae800e740 1 -- 172.21.15.90:0/1666679590 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a2befd9080 msgr2=0x55a2befd6590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:35:34.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.023+0000 7f8ae800e740 1 --2- 172.21.15.90:0/1666679590 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a2befd9080 0x55a2befd6590 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f8ad002c6c0 tx=0x7f8ad00095c0 comp rx=0 tx=0).stop 2024-04-02T14:35:34.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.024+0000 7f8ae800e740 1 -- 172.21.15.90:0/1666679590 shutdown_connections 2024-04-02T14:35:34.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.024+0000 7f8ae800e740 1 --2- 172.21.15.90:0/1666679590 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a2befd6ad0 0x55a2befe50f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:34.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.024+0000 7f8ae800e740 1 --2- 172.21.15.90:0/1666679590 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a2befd9080 0x55a2befd6590 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:34.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.024+0000 7f8ae800e740 1 --2- 172.21.15.90:0/1666679590 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x55a2befe96f0 0x55a2befebae0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:34.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.024+0000 7f8ae800e740 1 --2- 172.21.15.90:0/1666679590 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a2befd09f0 0x55a2befd6050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:34.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.024+0000 7f8ac3e00640 0 client.25558 ms_handle_reset on v2:172.21.15.177:6836/4223414858 2024-04-02T14:35:34.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.024+0000 7f8ae800e740 1 -- 172.21.15.90:0/1666679590 >> 172.21.15.90:0/1666679590 conn(0x55a2bef96a50 msgr2=0x55a2befcc4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:35:34.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.025+0000 7f8ae800e740 1 -- 172.21.15.90:0/1666679590 shutdown_connections 2024-04-02T14:35:34.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.025+0000 7f8ae800e740 1 -- 172.21.15.90:0/1666679590 wait complete. 2024-04-02T14:35:34.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.025+0000 7f8ae800e740 20 client.25558 trim_cache size 0 max 16384 2024-04-02T14:35:34.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.054+0000 7f8ae800e740 1 Processor -- start 2024-04-02T14:35:34.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.054+0000 7f8ae800e740 1 -- start start 2024-04-02T14:35:34.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.054+0000 7f8ae800e740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a2befd09f0 0x55a2befd5030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:34.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.054+0000 7f8ae800e740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a2befd6ad0 0x55a2beff13b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:34.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.054+0000 7f8ae800e740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a2befd9080 0x55a2beff18f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:34.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.055+0000 7f8ae800e740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f8ae01b3db0 con 0x55a2befd09f0 2024-04-02T14:35:34.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.055+0000 7f8ae800e740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f8ae01b58f0 con 0x55a2befd9080 2024-04-02T14:35:34.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.055+0000 7f8ae800e740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f8ae01b4480 con 0x55a2befd6ad0 2024-04-02T14:35:34.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.055+0000 7f8ade800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a2befd09f0 0x55a2befd5030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:34.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.055+0000 7f8ade800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a2befd09f0 0x55a2befd5030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:57420/0 (socket says 172.21.15.90:57420) 2024-04-02T14:35:34.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.055+0000 7f8ade800640 1 -- 172.21.15.90:0/662710499 learned_addr learned my addr 172.21.15.90:0/662710499 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:35:34.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.055+0000 7f8adde00640 1 --2- 172.21.15.90:0/662710499 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a2befd6ad0 0x55a2beff13b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:34.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.055+0000 7f8adf200640 1 --2- 172.21.15.90:0/662710499 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a2befd9080 0x55a2beff18f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:34.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.056+0000 7f8ade800640 1 -- 172.21.15.90:0/662710499 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a2befd6ad0 msgr2=0x55a2beff13b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:35:34.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.056+0000 7f8ade800640 1 --2- 172.21.15.90:0/662710499 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a2befd6ad0 0x55a2beff13b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:34.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.056+0000 7f8ade800640 1 -- 172.21.15.90:0/662710499 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a2befd9080 msgr2=0x55a2beff18f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:35:34.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.056+0000 7f8ade800640 1 --2- 172.21.15.90:0/662710499 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a2befd9080 0x55a2beff18f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:34.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.056+0000 7f8ade800640 1 -- 172.21.15.90:0/662710499 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a2befcb8e0 con 0x55a2befd09f0 2024-04-02T14:35:34.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.056+0000 7f8ade800640 1 --2- 172.21.15.90:0/662710499 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a2befd09f0 0x55a2befd5030 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f8ad8099910 tx=0x7f8ad80a1ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b259703dd24cf00e server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:35:34.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.057+0000 7f8acea00640 1 -- 172.21.15.90:0/662710499 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ad8099430 con 0x55a2befd09f0 2024-04-02T14:35:34.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.057+0000 7f8acea00640 1 -- 172.21.15.90:0/662710499 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8ad80b7020 con 0x55a2befd09f0 2024-04-02T14:35:34.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.057+0000 7f8acea00640 1 -- 172.21.15.90:0/662710499 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ad80995d0 con 0x55a2befd09f0 2024-04-02T14:35:34.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.058+0000 7f8ae800e740 1 -- 172.21.15.90:0/662710499 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a2befd09f0 msgr2=0x55a2befd5030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:35:34.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.058+0000 7f8ae800e740 1 --2- 172.21.15.90:0/662710499 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a2befd09f0 0x55a2befd5030 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f8ad8099910 tx=0x7f8ad80a1ed0 comp rx=0 tx=0).stop 2024-04-02T14:35:34.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.059+0000 7f8ae800e740 1 -- 172.21.15.90:0/662710499 shutdown_connections 2024-04-02T14:35:34.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.059+0000 7f8ae800e740 1 --2- 172.21.15.90:0/662710499 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a2befd9080 0x55a2beff18f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:34.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.059+0000 7f8ae800e740 1 --2- 172.21.15.90:0/662710499 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a2befd6ad0 0x55a2beff13b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:34.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.059+0000 7f8ae800e740 1 --2- 172.21.15.90:0/662710499 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a2befd09f0 0x55a2befd5030 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:34.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.059+0000 7f8ae800e740 1 -- 172.21.15.90:0/662710499 >> 172.21.15.90:0/662710499 conn(0x55a2bef96a50 msgr2=0x55a2befd4920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:35:34.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.060+0000 7f8ae800e740 1 -- 172.21.15.90:0/662710499 shutdown_connections 2024-04-02T14:35:34.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.060+0000 7f8ae800e740 1 -- 172.21.15.90:0/662710499 wait complete. 2024-04-02T14:35:34.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.061+0000 7f8ae800e740 1 -- start start 2024-04-02T14:35:34.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.061+0000 7f8ae800e740 1 Processor -- start 2024-04-02T14:35:34.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.062+0000 7f8ae800e740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a2befd09f0 0x55a2befda380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:34.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.062+0000 7f8ae800e740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a2befd6ad0 0x55a2befda8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:34.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.062+0000 7f8ae800e740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a2befd9080 0x55a2befdf6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:34.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.062+0000 7f8ade800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a2befd09f0 0x55a2befda380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:34.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.063+0000 7f8ade800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a2befd09f0 0x55a2befda380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:57806/0 (socket says 172.21.15.90:57806) 2024-04-02T14:35:34.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.063+0000 7f8adf200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a2befd9080 0x55a2befdf6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:34.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.063+0000 7f8adf200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a2befd9080 0x55a2befdf6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:57430/0 (socket says 172.21.15.90:57430) 2024-04-02T14:35:34.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.063+0000 7f8adf200640 1 -- 172.21.15.90:0/2927709395 learned_addr learned my addr 172.21.15.90:0/2927709395 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:35:34.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.063+0000 7f8ae800e740 1 -- 172.21.15.90:0/2927709395 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f8ae01b5220 con 0x55a2befd9080 2024-04-02T14:35:34.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.063+0000 7f8ae800e740 1 -- 172.21.15.90:0/2927709395 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f8ae01b4b50 con 0x55a2befd09f0 2024-04-02T14:35:34.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.063+0000 7f8ae800e740 1 -- 172.21.15.90:0/2927709395 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f8ae01b1d90 con 0x55a2befd6ad0 2024-04-02T14:35:34.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.064+0000 7f8adde00640 1 --2- 172.21.15.90:0/2927709395 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a2befd6ad0 0x55a2befda8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:34.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.064+0000 7f8ae800e740 10 client.0 _ll_register_callbacks cb 0x7f8ad0009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:35:34.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.064+0000 7f8adf200640 1 -- 172.21.15.90:0/2927709395 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a2befd6ad0 msgr2=0x55a2befda8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:35:34.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.064+0000 7f8adf200640 1 --2- 172.21.15.90:0/2927709395 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a2befd6ad0 0x55a2befda8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:34.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.065+0000 7f8adf200640 1 -- 172.21.15.90:0/2927709395 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a2befd09f0 msgr2=0x55a2befda380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:35:34.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.065+0000 7f8adf200640 1 --2- 172.21.15.90:0/2927709395 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a2befd09f0 0x55a2befda380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:34.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.065+0000 7f8adf200640 1 -- 172.21.15.90:0/2927709395 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a2befdfc20 con 0x55a2befd9080 2024-04-02T14:35:34.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.065+0000 7f8ade800640 1 --2- 172.21.15.90:0/2927709395 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a2befd09f0 0x55a2befda380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:35:34.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.065+0000 7f8adde00640 1 --2- 172.21.15.90:0/2927709395 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a2befd6ad0 0x55a2befda8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:35:34.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.065+0000 7f8adf200640 1 --2- 172.21.15.90:0/2927709395 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a2befd9080 0x55a2befdf6e0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f8ac8009710 tx=0x7f8ac8004120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6690a4f12a38b21d server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:35:34.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.065+0000 7f8ac3e00640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-02T14:35:34.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.066+0000 7f8ac3e00640 1 -- 172.21.15.90:0/2927709395 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ac8029020 con 0x55a2befd9080 2024-04-02T14:35:34.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.066+0000 7f8ae800e740 1 -- 172.21.15.90:0/2927709395 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8ad80b7020 con 0x55a2befd9080 2024-04-02T14:35:34.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.066+0000 7f8ac3e00640 1 -- 172.21.15.90:0/2927709395 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f8ac80054f0 con 0x55a2befd9080 2024-04-02T14:35:34.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.066+0000 7f8ac3e00640 1 -- 172.21.15.90:0/2927709395 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f8ac8029020 con 0x55a2befd9080 2024-04-02T14:35:34.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.066+0000 7f8ae800e740 10 client.15477 fetch_fsmap learned FSMap version 27 2024-04-02T14:35:34.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.066+0000 7f8ae800e740 1 -- 172.21.15.90:0/2927709395 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55a2bf0503c0 con 0x55a2befd9080 2024-04-02T14:35:34.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.067+0000 7f8ac3e00640 1 -- 172.21.15.90:0/2927709395 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8ad80b7020 con 0x55a2befd9080 2024-04-02T14:35:34.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.067+0000 7f8ac3e00640 1 -- 172.21.15.90:0/2927709395 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f8ac8018070 con 0x55a2befd9080 2024-04-02T14:35:34.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.067+0000 7f8ae800e740 10 client.15477 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:35:34.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.067+0000 7f8ae800e740 10 client.15477 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.c 2024-04-02T14:35:34.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.067+0000 7f8ae800e740 20 client.15477 populate_metadata read hostname 'smithi090' 2024-04-02T14:35:34.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.067+0000 7f8ae800e740 1 --2- 172.21.15.90:0/2927709395 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x55a2bf050840 0x55a2bf052c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:34.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.067+0000 7f8ae800e740 4 client.15477 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-02T14:35:34.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.067+0000 7f8ae800e740 1 -- 172.21.15.90:0/2927709395 --> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55a2befd6920 con 0x55a2bf050840 2024-04-02T14:35:34.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.068+0000 7f8ade800640 1 --2- 172.21.15.90:0/2927709395 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x55a2bf050840 0x55a2bf052c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:34.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.069+0000 7f8ac3e00640 10 client.15477 ms_handle_connect on v2:172.21.15.177:6836/4223414858 2024-04-02T14:35:34.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.069+0000 7f8ade800640 1 --2- 172.21.15.90:0/2927709395 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x55a2bf050840 0x55a2bf052c30 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=a9ccecd2f26f295b server_cookie=9c090675b8f0013 in_seq=0 out_seq=0 2024-04-02T14:35:34.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.069+0000 7f8ac3e00640 1 -- 172.21.15.90:0/2927709395 <== mds.4 v2:172.21.15.177:6836/4223414858 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55a2befd6920 con 0x55a2bf050840 2024-04-02T14:35:34.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.069+0000 7f8ac3e00640 10 client.15477 handle_command_reply: tid=0 2024-04-02T14:35:34.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.069+0000 7f8ae800e740 1 client.15477 shutdown 2024-04-02T14:35:34.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.069+0000 7f8ae800e740 1 -- 172.21.15.90:0/2927709395 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a2befd9080 msgr2=0x55a2befdf6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:35:34.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.069+0000 7f8ae800e740 1 --2- 172.21.15.90:0/2927709395 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a2befd9080 0x55a2befdf6e0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f8ac8009710 tx=0x7f8ac8004120 comp rx=0 tx=0).stop 2024-04-02T14:35:34.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.070+0000 7f8ae800e740 1 -- 172.21.15.90:0/2927709395 shutdown_connections 2024-04-02T14:35:34.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.070+0000 7f8ae800e740 1 --2- 172.21.15.90:0/2927709395 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x55a2bf050840 0x55a2bf052c30 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:34.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.070+0000 7f8ac3e00640 0 client.15477 ms_handle_reset on v2:172.21.15.177:6836/4223414858 2024-04-02T14:35:34.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.070+0000 7f8ae800e740 1 --2- 172.21.15.90:0/2927709395 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a2befd9080 0x55a2befdf6e0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:34.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.070+0000 7f8ae800e740 1 --2- 172.21.15.90:0/2927709395 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a2befd6ad0 0x55a2befda8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:34.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.070+0000 7f8ae800e740 1 --2- 172.21.15.90:0/2927709395 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a2befd09f0 0x55a2befda380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:34.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.070+0000 7f8ae800e740 1 -- 172.21.15.90:0/2927709395 >> 172.21.15.90:0/2927709395 conn(0x55a2bef96a50 msgr2=0x55a2befcbb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:35:34.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.070+0000 7f8ae800e740 1 -- 172.21.15.90:0/2927709395 shutdown_connections 2024-04-02T14:35:34.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.071+0000 7f8ae800e740 1 -- 172.21.15.90:0/2927709395 wait complete. 2024-04-02T14:35:34.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.071+0000 7f8ae800e740 20 client.15477 trim_cache size 0 max 16384 2024-04-02T14:35:34.074 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-02T14:35:34.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.072+0000 7f8ae5400640 1 -- 172.21.15.90:0/1099855190 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f8ac40a3f90 msgr2=0x7f8ac40a6450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:35:34.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.072+0000 7f8ae5400640 1 --2- 172.21.15.90:0/1099855190 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f8ac40a3f90 0x7f8ac40a6450 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f8ad0000f80 tx=0x7f8ad0009360 comp rx=0 tx=0).stop 2024-04-02T14:35:34.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.072+0000 7f8ae5400640 1 -- 172.21.15.90:0/1099855190 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8ae0073ed0 msgr2=0x7f8ae01a4760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:35:34.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.072+0000 7f8ae5400640 1 --2- 172.21.15.90:0/1099855190 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8ae0073ed0 0x7f8ae01a4760 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f8ad800ea60 tx=0x7f8ad800ef20 comp rx=0 tx=0).stop 2024-04-02T14:35:34.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.072+0000 7f8ae5400640 1 -- 172.21.15.90:0/1099855190 shutdown_connections 2024-04-02T14:35:34.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.072+0000 7f8ae5400640 1 --2- 172.21.15.90:0/1099855190 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f8ac40a3f90 0x7f8ac40a6450 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:34.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.072+0000 7f8ae5400640 1 --2- 172.21.15.90:0/1099855190 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8ae010e320 0x7f8ae010c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:34.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.072+0000 7f8ae5400640 1 --2- 172.21.15.90:0/1099855190 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8ae00747f0 0x7f8ae01a4ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:34.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.072+0000 7f8ae5400640 1 --2- 172.21.15.90:0/1099855190 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8ae0073ed0 0x7f8ae01a4760 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:34.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.072+0000 7f8ae5400640 1 -- 172.21.15.90:0/1099855190 >> 172.21.15.90:0/1099855190 conn(0x7f8ae00fd570 msgr2=0x7f8ae00fee70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:35:34.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.073+0000 7f8ae5400640 1 -- 172.21.15.90:0/1099855190 shutdown_connections 2024-04-02T14:35:34.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.073+0000 7f8ae5400640 1 -- 172.21.15.90:0/1099855190 wait complete. 2024-04-02T14:35:34.099 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:35:33 smithi090 ceph-mon[31674]: pgmap v23: 97 pgs: 97 active+clean; 365 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 54 MiB/s wr, 43 op/s 2024-04-02T14:35:34.099 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:35:33 smithi090 ceph-mon[31674]: from='client.? 172.21.15.90:0/3486313092' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-02T14:35:34.100 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-04-02T14:35:34.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:35:33 smithi177 ceph-mon[36431]: pgmap v23: 97 pgs: 97 active+clean; 365 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 54 MiB/s wr, 43 op/s 2024-04-02T14:35:34.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:35:33 smithi177 ceph-mon[36431]: from='client.? 172.21.15.90:0/3486313092' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-02T14:35:34.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:35:33 smithi092 ceph-mon[36497]: pgmap v23: 97 pgs: 97 active+clean; 365 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 54 MiB/s wr, 43 op/s 2024-04-02T14:35:34.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:35:33 smithi092 ceph-mon[36497]: from='client.? 172.21.15.90:0/3486313092' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-02T14:35:34.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.387+0000 7f77b9a00640 1 -- 172.21.15.90:0/2352593208 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f77b4117040 msgr2=0x7f77b4119430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:35:34.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.387+0000 7f77b9a00640 1 --2- 172.21.15.90:0/2352593208 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f77b4117040 0x7f77b4119430 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f77ac009ca0 tx=0x7f77ac02c120 comp rx=0 tx=0).stop 2024-04-02T14:35:34.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.388+0000 7f77b9a00640 1 -- 172.21.15.90:0/2352593208 shutdown_connections 2024-04-02T14:35:34.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.388+0000 7f77b9a00640 1 --2- 172.21.15.90:0/2352593208 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f77b4117040 0x7f77b4119430 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:34.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.388+0000 7f77b9a00640 1 --2- 172.21.15.90:0/2352593208 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f77b4109600 0x7f77b4116b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:34.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.388+0000 7f77b9a00640 1 --2- 172.21.15.90:0/2352593208 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f77b4108ce0 0x7f77b41090c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:34.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.388+0000 7f77b9a00640 1 -- 172.21.15.90:0/2352593208 >> 172.21.15.90:0/2352593208 conn(0x7f77b406cbe0 msgr2=0x7f77b406cff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:35:34.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.388+0000 7f77b9a00640 1 -- 172.21.15.90:0/2352593208 shutdown_connections 2024-04-02T14:35:34.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.388+0000 7f77b9a00640 1 -- 172.21.15.90:0/2352593208 wait complete. 2024-04-02T14:35:34.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.389+0000 7f77b9a00640 1 Processor -- start 2024-04-02T14:35:34.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.389+0000 7f77b9a00640 1 -- start start 2024-04-02T14:35:34.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.390+0000 7f77b9a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f77b4108ce0 0x7f77b41a8790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:34.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.390+0000 7f77b9a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f77b4109600 0x7f77b41a8cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:34.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.390+0000 7f77b9a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f77b4117040 0x7f77b41a2940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:34.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.390+0000 7f77b2a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f77b4108ce0 0x7f77b41a8790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:34.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.390+0000 7f77b9a00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f77b411c820 con 0x7f77b4117040 2024-04-02T14:35:34.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.390+0000 7f77b9a00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f77b411c6a0 con 0x7f77b4108ce0 2024-04-02T14:35:34.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.390+0000 7f77b2000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f77b4109600 0x7f77b41a8cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:34.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.390+0000 7f77b2a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f77b4108ce0 0x7f77b41a8790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:57836/0 (socket says 172.21.15.90:57836) 2024-04-02T14:35:34.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.391+0000 7f77b2a00640 1 -- 172.21.15.90:0/3035416980 learned_addr learned my addr 172.21.15.90:0/3035416980 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:35:34.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.390+0000 7f77b3400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f77b4117040 0x7f77b41a2940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:34.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.391+0000 7f77b9a00640 1 -- 172.21.15.90:0/3035416980 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f77b411c9a0 con 0x7f77b4109600 2024-04-02T14:35:34.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.391+0000 7f77b3400640 1 -- 172.21.15.90:0/3035416980 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f77b4109600 msgr2=0x7f77b41a8cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:35:34.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.391+0000 7f77b3400640 1 --2- 172.21.15.90:0/3035416980 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f77b4109600 0x7f77b41a8cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:34.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.391+0000 7f77b3400640 1 -- 172.21.15.90:0/3035416980 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f77b4108ce0 msgr2=0x7f77b41a8790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:35:34.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.391+0000 7f77b3400640 1 --2- 172.21.15.90:0/3035416980 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f77b4108ce0 0x7f77b41a8790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:34.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.391+0000 7f77b3400640 1 -- 172.21.15.90:0/3035416980 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77b41a30c0 con 0x7f77b4117040 2024-04-02T14:35:34.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.392+0000 7f77b3400640 1 --2- 172.21.15.90:0/3035416980 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f77b4117040 0x7f77b41a2940 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f77ac037040 tx=0x7f77ac034d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:35:34.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.392+0000 7f77ab400640 1 -- 172.21.15.90:0/3035416980 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77ac041020 con 0x7f77b4117040 2024-04-02T14:35:34.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.392+0000 7f77ab400640 1 -- 172.21.15.90:0/3035416980 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f77ac032340 con 0x7f77b4117040 2024-04-02T14:35:34.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.392+0000 7f77ab400640 1 -- 172.21.15.90:0/3035416980 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77ac039040 con 0x7f77b4117040 2024-04-02T14:35:34.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.392+0000 7f77b9a00640 1 -- 172.21.15.90:0/3035416980 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f77b41a3350 con 0x7f77b4117040 2024-04-02T14:35:34.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.392+0000 7f77b9a00640 1 -- 172.21.15.90:0/3035416980 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f77b410c6b0 con 0x7f77b4117040 2024-04-02T14:35:34.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.398+0000 7f77bc7ae740 1 Processor -- start 2024-04-02T14:35:34.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.398+0000 7f77ab400640 1 -- 172.21.15.90:0/3035416980 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7f77ac044020 con 0x7f77b4117040 2024-04-02T14:35:34.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.399+0000 7f77bc7ae740 1 -- start start 2024-04-02T14:35:34.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.399+0000 7f77bc7ae740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f49deeb9f0 0x55f49deea740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:34.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.399+0000 7f77bc7ae740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f49deead10 0x55f49deeb190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:34.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.399+0000 7f77bc7ae740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f49def4080 0x55f49def6470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:34.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.399+0000 7f77bc7ae740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55f49de2b400 con 0x55f49def4080 2024-04-02T14:35:34.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.399+0000 7f77bc7ae740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55f49de0ff10 con 0x55f49deeb9f0 2024-04-02T14:35:34.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.399+0000 7f77bc7ae740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55f49de4f6f0 con 0x55f49deead10 2024-04-02T14:35:34.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.400+0000 7f77b2a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f49deead10 0x55f49deeb190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:34.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.400+0000 7f77b2a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f49deead10 0x55f49deeb190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:41022/0 (socket says 172.21.15.90:41022) 2024-04-02T14:35:34.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.400+0000 7f77b2a00640 1 -- 172.21.15.90:0/4235640758 learned_addr learned my addr 172.21.15.90:0/4235640758 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:35:34.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.400+0000 7f77b2000640 1 --2- 172.21.15.90:0/4235640758 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f49def4080 0x55f49def6470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:34.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.400+0000 7f77b2000640 1 --2- 172.21.15.90:0/4235640758 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f49deeb9f0 0x55f49deea740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:34.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.401+0000 7f77b2000640 1 -- 172.21.15.90:0/4235640758 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f49deead10 msgr2=0x55f49deeb190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:35:34.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.401+0000 7f77b2000640 1 --2- 172.21.15.90:0/4235640758 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f49deead10 0x55f49deeb190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:34.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.401+0000 7f77b2000640 1 -- 172.21.15.90:0/4235640758 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f49def4080 msgr2=0x55f49def6470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:35:34.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.401+0000 7f77b2000640 1 --2- 172.21.15.90:0/4235640758 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f49def4080 0x55f49def6470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:34.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.401+0000 7f77b2000640 1 -- 172.21.15.90:0/4235640758 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f49db633f0 con 0x55f49deeb9f0 2024-04-02T14:35:34.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.402+0000 7f77ab400640 1 --2- 172.21.15.90:0/3035416980 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f77980a3ec0 0x7f77980a6380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:34.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.402+0000 7f77ab400640 1 -- 172.21.15.90:0/3035416980 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f77ac0f2a40 con 0x7f77b4117040 2024-04-02T14:35:34.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.403+0000 7f77b2000640 1 --2- 172.21.15.90:0/4235640758 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f49deeb9f0 0x55f49deea740 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f77b4119290 tx=0x7f77a0004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=505e21c5e67f6e45 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:35:34.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.403+0000 7f77b2a00640 1 --2- 172.21.15.90:0/3035416980 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f77980a3ec0 0x7f77980a6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:34.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.403+0000 7f779fe00640 1 -- 172.21.15.90:0/4235640758 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77a00130f0 con 0x55f49deeb9f0 2024-04-02T14:35:34.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.403+0000 7f779fe00640 1 -- 172.21.15.90:0/4235640758 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f77a0017030 con 0x55f49deeb9f0 2024-04-02T14:35:34.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.403+0000 7f779fe00640 1 -- 172.21.15.90:0/4235640758 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77a0004ce0 con 0x55f49deeb9f0 2024-04-02T14:35:34.406 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.404+0000 7f77bc7ae740 1 -- 172.21.15.90:0/4235640758 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f49deeb9f0 msgr2=0x55f49deea740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:35:34.406 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.404+0000 7f77bc7ae740 1 --2- 172.21.15.90:0/4235640758 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f49deeb9f0 0x55f49deea740 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f77b4119290 tx=0x7f77a0004730 comp rx=0 tx=0).stop 2024-04-02T14:35:34.406 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.404+0000 7f77b2a00640 1 --2- 172.21.15.90:0/3035416980 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f77980a3ec0 0x7f77980a6380 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x55f49deebf40 tx=0x7f77a4019040 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:35:34.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.406+0000 7f77bc7ae740 1 -- 172.21.15.90:0/4235640758 shutdown_connections 2024-04-02T14:35:34.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.406+0000 7f77bc7ae740 1 --2- 172.21.15.90:0/4235640758 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f49def4080 0x55f49def6470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:34.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.406+0000 7f77bc7ae740 1 --2- 172.21.15.90:0/4235640758 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f49deead10 0x55f49deeb190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:34.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.406+0000 7f77bc7ae740 1 --2- 172.21.15.90:0/4235640758 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f49deeb9f0 0x55f49deea740 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:34.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.406+0000 7f77bc7ae740 1 -- 172.21.15.90:0/4235640758 >> 172.21.15.90:0/4235640758 conn(0x55f49deb1a50 msgr2=0x55f49dee5760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:35:34.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.407+0000 7f77bc7ae740 1 -- 172.21.15.90:0/4235640758 shutdown_connections 2024-04-02T14:35:34.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.407+0000 7f77bc7ae740 1 -- 172.21.15.90:0/4235640758 wait complete. 2024-04-02T14:35:34.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.408+0000 7f77bc7ae740 1 -- start start 2024-04-02T14:35:34.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.408+0000 7f77bc7ae740 1 Processor -- start 2024-04-02T14:35:34.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.411+0000 7f77bc7ae740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f49deead10 0x55f49def10c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:34.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.411+0000 7f77bc7ae740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f49deeb9f0 0x55f49def1600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:34.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.412+0000 7f77bc7ae740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f49def4080 0x55f49def1b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:34.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.412+0000 7f77b2a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f49deead10 0x55f49def10c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:34.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.412+0000 7f77b2000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f49deeb9f0 0x55f49def1600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:34.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.412+0000 7f77b2a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f49deead10 0x55f49def10c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:57842/0 (socket says 172.21.15.90:57842) 2024-04-02T14:35:34.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.412+0000 7f77b2000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f49deeb9f0 0x55f49def1600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:41030/0 (socket says 172.21.15.90:41030) 2024-04-02T14:35:34.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.413+0000 7f77b3400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f49def4080 0x55f49def1b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:34.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.413+0000 7f77b2a00640 1 -- 172.21.15.90:0/2447689244 learned_addr learned my addr 172.21.15.90:0/2447689244 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:35:34.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.413+0000 7f77bc7ae740 1 -- 172.21.15.90:0/2447689244 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55f49de4b6a0 con 0x55f49def4080 2024-04-02T14:35:34.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.414+0000 7f77bc7ae740 1 -- 172.21.15.90:0/2447689244 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55f49ddf7980 con 0x55f49deead10 2024-04-02T14:35:34.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.414+0000 7f77bc7ae740 1 -- 172.21.15.90:0/2447689244 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55f49de0acb0 con 0x55f49deeb9f0 2024-04-02T14:35:34.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.415+0000 7f77bc7ae740 10 client.0 _ll_register_callbacks cb 0x55f49de0a6a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:35:34.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.415+0000 7f77b2a00640 1 -- 172.21.15.90:0/2447689244 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f49deeb9f0 msgr2=0x55f49def1600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:35:34.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.415+0000 7f77b2a00640 1 --2- 172.21.15.90:0/2447689244 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f49deeb9f0 0x55f49def1600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:34.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.415+0000 7f77b2a00640 1 -- 172.21.15.90:0/2447689244 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f49def4080 msgr2=0x55f49def1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:35:34.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.415+0000 7f77b2a00640 1 --2- 172.21.15.90:0/2447689244 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f49def4080 0x55f49def1b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:34.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.416+0000 7f77b2a00640 1 -- 172.21.15.90:0/2447689244 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f49dc2d7d0 con 0x55f49deead10 2024-04-02T14:35:34.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.416+0000 7f77b2a00640 1 --2- 172.21.15.90:0/2447689244 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f49deead10 0x55f49def10c0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f77a40096c0 tx=0x7f77a4004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=196e84c1e7e201d1 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:35:34.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.416+0000 7f779cc00640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:35:34.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.417+0000 7f779cc00640 1 -- 172.21.15.90:0/2447689244 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77a4031030 con 0x55f49deead10 2024-04-02T14:35:34.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.417+0000 7f779cc00640 1 -- 172.21.15.90:0/2447689244 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f77a4005020 con 0x55f49deead10 2024-04-02T14:35:34.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.417+0000 7f77bc7ae740 1 -- 172.21.15.90:0/2447689244 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f77a0017030 con 0x55f49deead10 2024-04-02T14:35:34.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.418+0000 7f779cc00640 1 -- 172.21.15.90:0/2447689244 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f77a403b050 con 0x55f49deead10 2024-04-02T14:35:34.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.419+0000 7f77bc7ae740 10 client.25619 fetch_fsmap learned FSMap version 27 2024-04-02T14:35:34.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.419+0000 7f77bc7ae740 1 -- 172.21.15.90:0/2447689244 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55f49df02f80 con 0x55f49deead10 2024-04-02T14:35:34.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.419+0000 7f779cc00640 1 -- 172.21.15.90:0/2447689244 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f77a0017030 con 0x55f49deead10 2024-04-02T14:35:34.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.419+0000 7f779cc00640 1 -- 172.21.15.90:0/2447689244 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f77a402c030 con 0x55f49deead10 2024-04-02T14:35:34.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.420+0000 7f77bc7ae740 10 client.25619 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:35:34.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.420+0000 7f77bc7ae740 10 client.25619 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-04-02T14:35:34.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.420+0000 7f77bc7ae740 20 client.25619 populate_metadata read hostname 'smithi090' 2024-04-02T14:35:34.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.420+0000 7f77bc7ae740 1 --2- 172.21.15.90:0/2447689244 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x55f49df044e0 0x55f49df06930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:34.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.420+0000 7f77bc7ae740 4 client.25619 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:35:34.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.420+0000 7f77bc7ae740 1 -- 172.21.15.90:0/2447689244 --> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f49de0f040 con 0x55f49df044e0 2024-04-02T14:35:34.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.421+0000 7f77b2000640 1 --2- 172.21.15.90:0/2447689244 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x55f49df044e0 0x55f49df06930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:34.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.422+0000 7f77b2000640 1 --2- 172.21.15.90:0/2447689244 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x55f49df044e0 0x55f49df06930 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=ae9799660b5449c3 server_cookie=270dd916e017ef5c in_seq=0 out_seq=0 2024-04-02T14:35:34.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.422+0000 7f779cc00640 10 client.25619 ms_handle_connect on v2:172.21.15.92:6836/1270635939 2024-04-02T14:35:34.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.423+0000 7f779cc00640 1 -- 172.21.15.90:0/2447689244 <== mds.3 v2:172.21.15.92:6836/1270635939 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f49de0f040 con 0x55f49df044e0 2024-04-02T14:35:34.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.423+0000 7f779cc00640 10 client.25619 handle_command_reply: tid=0 2024-04-02T14:35:34.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.424+0000 7f77bc7ae740 1 client.25619 shutdown 2024-04-02T14:35:34.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.424+0000 7f77bc7ae740 1 -- 172.21.15.90:0/2447689244 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f49deead10 msgr2=0x55f49def10c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:35:34.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.424+0000 7f77bc7ae740 1 --2- 172.21.15.90:0/2447689244 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f49deead10 0x55f49def10c0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f77a40096c0 tx=0x7f77a4004730 comp rx=0 tx=0).stop 2024-04-02T14:35:34.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.425+0000 7f77bc7ae740 1 -- 172.21.15.90:0/2447689244 shutdown_connections 2024-04-02T14:35:34.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.425+0000 7f77bc7ae740 1 --2- 172.21.15.90:0/2447689244 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x55f49df044e0 0x55f49df06930 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:34.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.425+0000 7f779cc00640 0 client.25619 ms_handle_reset on v2:172.21.15.92:6836/1270635939 2024-04-02T14:35:34.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.425+0000 7f77bc7ae740 1 --2- 172.21.15.90:0/2447689244 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f49def4080 0x55f49def1b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:34.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.425+0000 7f77bc7ae740 1 --2- 172.21.15.90:0/2447689244 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f49deeb9f0 0x55f49def1600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:34.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.425+0000 7f77bc7ae740 1 --2- 172.21.15.90:0/2447689244 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f49deead10 0x55f49def10c0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:34.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.425+0000 7f77bc7ae740 1 -- 172.21.15.90:0/2447689244 >> 172.21.15.90:0/2447689244 conn(0x55f49deb1a50 msgr2=0x55f49dee4010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:35:34.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.425+0000 7f77bc7ae740 1 -- 172.21.15.90:0/2447689244 shutdown_connections 2024-04-02T14:35:34.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.425+0000 7f77bc7ae740 1 -- 172.21.15.90:0/2447689244 wait complete. 2024-04-02T14:35:34.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.426+0000 7f77bc7ae740 20 client.25619 trim_cache size 0 max 16384 2024-04-02T14:35:34.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.460+0000 7f77bc7ae740 1 Processor -- start 2024-04-02T14:35:34.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.460+0000 7f77bc7ae740 1 -- start start 2024-04-02T14:35:34.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.461+0000 7f77bc7ae740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f49deead10 0x55f49deffaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:34.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.461+0000 7f77bc7ae740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f49deeb9f0 0x55f49def1960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:34.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.461+0000 7f77bc7ae740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f49def4080 0x55f49def00a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:34.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.461+0000 7f77b2000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f49deeb9f0 0x55f49def1960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:34.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.461+0000 7f77b2a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f49deead10 0x55f49deffaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:34.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.462+0000 7f77bc7ae740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f77b41b3c30 con 0x55f49deeb9f0 2024-04-02T14:35:34.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.462+0000 7f77b3400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f49def4080 0x55f49def00a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:34.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.462+0000 7f77bc7ae740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f77b41b5770 con 0x55f49def4080 2024-04-02T14:35:34.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.462+0000 7f77bc7ae740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f77b41b4300 con 0x55f49deead10 2024-04-02T14:35:34.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.462+0000 7f77b3400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f49def4080 0x55f49def00a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:57858/0 (socket says 172.21.15.90:57858) 2024-04-02T14:35:34.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.462+0000 7f77b3400640 1 -- 172.21.15.90:0/1099205556 learned_addr learned my addr 172.21.15.90:0/1099205556 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:35:34.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.463+0000 7f77b2000640 1 -- 172.21.15.90:0/1099205556 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f49deead10 msgr2=0x55f49deffaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:35:34.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.463+0000 7f77b2000640 1 --2- 172.21.15.90:0/1099205556 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f49deead10 0x55f49deffaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:34.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.463+0000 7f77b2000640 1 -- 172.21.15.90:0/1099205556 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f49def4080 msgr2=0x55f49def00a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:35:34.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.463+0000 7f77b2000640 1 --2- 172.21.15.90:0/1099205556 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f49def4080 0x55f49def00a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:34.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.463+0000 7f77b2000640 1 -- 172.21.15.90:0/1099205556 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f49dee9d40 con 0x55f49deeb9f0 2024-04-02T14:35:34.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.463+0000 7f77b3400640 1 --2- 172.21.15.90:0/1099205556 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f49def4080 0x55f49def00a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:35:34.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.464+0000 7f77b2000640 1 --2- 172.21.15.90:0/1099205556 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f49deeb9f0 0x55f49def1960 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f77a00049c0 tx=0x7f77a0005d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5caa1defd6b533f server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:35:34.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.465+0000 7f779f400640 1 -- 172.21.15.90:0/1099205556 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77a00047e0 con 0x55f49deeb9f0 2024-04-02T14:35:34.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.465+0000 7f779f400640 1 -- 172.21.15.90:0/1099205556 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f77a0017040 con 0x55f49deeb9f0 2024-04-02T14:35:34.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.465+0000 7f779f400640 1 -- 172.21.15.90:0/1099205556 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77a0011760 con 0x55f49deeb9f0 2024-04-02T14:35:34.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.466+0000 7f77bc7ae740 1 -- 172.21.15.90:0/1099205556 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f49deeb9f0 msgr2=0x55f49def1960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:35:34.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.466+0000 7f77bc7ae740 1 --2- 172.21.15.90:0/1099205556 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f49deeb9f0 0x55f49def1960 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f77a00049c0 tx=0x7f77a0005d30 comp rx=0 tx=0).stop 2024-04-02T14:35:34.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.466+0000 7f77bc7ae740 1 -- 172.21.15.90:0/1099205556 shutdown_connections 2024-04-02T14:35:34.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.467+0000 7f77bc7ae740 1 --2- 172.21.15.90:0/1099205556 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f49def4080 0x55f49def00a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:34.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.467+0000 7f77bc7ae740 1 --2- 172.21.15.90:0/1099205556 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f49deeb9f0 0x55f49def1960 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:34.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.467+0000 7f77bc7ae740 1 --2- 172.21.15.90:0/1099205556 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f49deead10 0x55f49deffaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:34.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.467+0000 7f77bc7ae740 1 -- 172.21.15.90:0/1099205556 >> 172.21.15.90:0/1099205556 conn(0x55f49deb1a50 msgr2=0x55f49deef990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:35:34.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.468+0000 7f77bc7ae740 1 -- 172.21.15.90:0/1099205556 shutdown_connections 2024-04-02T14:35:34.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.468+0000 7f77bc7ae740 1 -- 172.21.15.90:0/1099205556 wait complete. 2024-04-02T14:35:34.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.468+0000 7f77bc7ae740 1 -- start start 2024-04-02T14:35:34.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.469+0000 7f77bc7ae740 1 Processor -- start 2024-04-02T14:35:34.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.470+0000 7f77bc7ae740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f49deead10 0x55f49def0d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:34.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.470+0000 7f77bc7ae740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f49deeb9f0 0x55f49def1270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:34.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.470+0000 7f77b2a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f49deead10 0x55f49def0d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:34.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.470+0000 7f77b2a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f49deead10 0x55f49def0d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:57868/0 (socket says 172.21.15.90:57868) 2024-04-02T14:35:34.473 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.471+0000 7f77bc7ae740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f49def4080 0x55f49def17b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:34.473 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.471+0000 7f77b2000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f49deeb9f0 0x55f49def1270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:34.473 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.471+0000 7f77b2000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f49deeb9f0 0x55f49def1270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:57488/0 (socket says 172.21.15.90:57488) 2024-04-02T14:35:34.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.471+0000 7f77bc7ae740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f77b41b50a0 con 0x55f49deeb9f0 2024-04-02T14:35:34.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.471+0000 7f77bc7ae740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f77b41b49d0 con 0x55f49deead10 2024-04-02T14:35:34.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.472+0000 7f77b2a00640 1 -- 172.21.15.90:0/916371879 learned_addr learned my addr 172.21.15.90:0/916371879 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:35:34.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.472+0000 7f77bc7ae740 1 -- 172.21.15.90:0/916371879 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f77b41b1c50 con 0x55f49def4080 2024-04-02T14:35:34.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.472+0000 7f77bc7ae740 10 client.0 _ll_register_callbacks cb 0x7f77a4023860 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:35:34.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.472+0000 7f77b3400640 1 --2- 172.21.15.90:0/916371879 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f49def4080 0x55f49def17b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:34.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.472+0000 7f77b2000640 1 -- 172.21.15.90:0/916371879 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f49def4080 msgr2=0x55f49def17b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:35:34.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.472+0000 7f77b2000640 1 --2- 172.21.15.90:0/916371879 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f49def4080 0x55f49def17b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:34.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.472+0000 7f77b2000640 1 -- 172.21.15.90:0/916371879 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f49deead10 msgr2=0x55f49def0d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:35:34.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.473+0000 7f77b2000640 1 --2- 172.21.15.90:0/916371879 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f49deead10 0x55f49def0d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:34.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.473+0000 7f77b2000640 1 -- 172.21.15.90:0/916371879 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f49df00030 con 0x55f49deeb9f0 2024-04-02T14:35:34.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.473+0000 7f77b3400640 1 --2- 172.21.15.90:0/916371879 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f49def4080 0x55f49def17b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:35:34.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.473+0000 7f77b2a00640 1 --2- 172.21.15.90:0/916371879 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f49deead10 0x55f49def0d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:35:34.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.474+0000 7f77b2000640 1 --2- 172.21.15.90:0/916371879 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f49deeb9f0 0x55f49def1270 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x55f49dee7c00 tx=0x7f77a00098d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe52686d2f460f29 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:35:34.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.474+0000 7f779cc00640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-02T14:35:34.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.476+0000 7f779cc00640 1 -- 172.21.15.90:0/916371879 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77a002b070 con 0x55f49deeb9f0 2024-04-02T14:35:34.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.476+0000 7f779cc00640 1 -- 172.21.15.90:0/916371879 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f77a002b210 con 0x55f49deeb9f0 2024-04-02T14:35:34.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.477+0000 7f77bc7ae740 1 -- 172.21.15.90:0/916371879 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f77a0017040 con 0x55f49deeb9f0 2024-04-02T14:35:34.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.477+0000 7f779cc00640 1 -- 172.21.15.90:0/916371879 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f77a002b410 con 0x55f49deeb9f0 2024-04-02T14:35:34.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.477+0000 7f77bc7ae740 10 client.15495 fetch_fsmap learned FSMap version 27 2024-04-02T14:35:34.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.477+0000 7f77bc7ae740 1 -- 172.21.15.90:0/916371879 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55f49df6abf0 con 0x55f49deeb9f0 2024-04-02T14:35:34.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.479+0000 7f779cc00640 1 -- 172.21.15.90:0/916371879 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f77a0017040 con 0x55f49deeb9f0 2024-04-02T14:35:34.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.479+0000 7f779cc00640 1 -- 172.21.15.90:0/916371879 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f77a0034060 con 0x55f49deeb9f0 2024-04-02T14:35:34.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.479+0000 7f77bc7ae740 10 client.15495 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:35:34.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.479+0000 7f77bc7ae740 10 client.15495 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-04-02T14:35:34.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.479+0000 7f77bc7ae740 20 client.15495 populate_metadata read hostname 'smithi090' 2024-04-02T14:35:34.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.479+0000 7f77bc7ae740 1 --2- 172.21.15.90:0/916371879 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x55f49df6b070 0x55f49df6d460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:34.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.479+0000 7f77bc7ae740 4 client.15495 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-02T14:35:34.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.479+0000 7f77bc7ae740 1 -- 172.21.15.90:0/916371879 --> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55f49de5cc30 con 0x55f49df6b070 2024-04-02T14:35:34.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.480+0000 7f77b2a00640 1 --2- 172.21.15.90:0/916371879 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x55f49df6b070 0x55f49df6d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:34.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.480+0000 7f779cc00640 10 client.15495 ms_handle_connect on v2:172.21.15.92:6836/1270635939 2024-04-02T14:35:34.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.480+0000 7f77b2a00640 1 --2- 172.21.15.90:0/916371879 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x55f49df6b070 0x55f49df6d460 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=2385e2a3c7add57f server_cookie=85e7bc424a5485f9 in_seq=0 out_seq=0 2024-04-02T14:35:34.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.481+0000 7f779cc00640 1 -- 172.21.15.90:0/916371879 <== mds.3 v2:172.21.15.92:6836/1270635939 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55f49de5cc30 con 0x55f49df6b070 2024-04-02T14:35:34.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.481+0000 7f779cc00640 10 client.15495 handle_command_reply: tid=0 2024-04-02T14:35:34.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.481+0000 7f77bc7ae740 1 client.15495 shutdown 2024-04-02T14:35:34.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.481+0000 7f77bc7ae740 1 -- 172.21.15.90:0/916371879 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f49deeb9f0 msgr2=0x55f49def1270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:35:34.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.481+0000 7f77bc7ae740 1 --2- 172.21.15.90:0/916371879 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f49deeb9f0 0x55f49def1270 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x55f49dee7c00 tx=0x7f77a00098d0 comp rx=0 tx=0).stop 2024-04-02T14:35:34.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.482+0000 7f77bc7ae740 1 -- 172.21.15.90:0/916371879 shutdown_connections 2024-04-02T14:35:34.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.482+0000 7f77bc7ae740 1 --2- 172.21.15.90:0/916371879 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x55f49df6b070 0x55f49df6d460 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:34.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.482+0000 7f77bc7ae740 1 --2- 172.21.15.90:0/916371879 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f49def4080 0x55f49def17b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:34.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.482+0000 7f77bc7ae740 1 --2- 172.21.15.90:0/916371879 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f49deeb9f0 0x55f49def1270 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:34.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.482+0000 7f77bc7ae740 1 --2- 172.21.15.90:0/916371879 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f49deead10 0x55f49def0d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:34.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.482+0000 7f77bc7ae740 1 -- 172.21.15.90:0/916371879 >> 172.21.15.90:0/916371879 conn(0x55f49deb1a50 msgr2=0x55f49dee73d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:35:34.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.482+0000 7f779cc00640 0 client.15495 ms_handle_reset on v2:172.21.15.92:6836/1270635939 2024-04-02T14:35:34.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.483+0000 7f77bc7ae740 1 -- 172.21.15.90:0/916371879 shutdown_connections 2024-04-02T14:35:34.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.483+0000 7f77bc7ae740 1 -- 172.21.15.90:0/916371879 wait complete. 2024-04-02T14:35:34.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.483+0000 7f77bc7ae740 20 client.15495 trim_cache size 0 max 16384 2024-04-02T14:35:34.486 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-02T14:35:34.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.485+0000 7f77b9a00640 1 -- 172.21.15.90:0/3035416980 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f77980a3ec0 msgr2=0x7f77980a6380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:35:34.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.485+0000 7f77b9a00640 1 --2- 172.21.15.90:0/3035416980 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f77980a3ec0 0x7f77980a6380 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x55f49deebf40 tx=0x7f77a4019040 comp rx=0 tx=0).stop 2024-04-02T14:35:34.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.485+0000 7f77b9a00640 1 -- 172.21.15.90:0/3035416980 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f77b4117040 msgr2=0x7f77b41a2940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:35:34.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.485+0000 7f77b9a00640 1 --2- 172.21.15.90:0/3035416980 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f77b4117040 0x7f77b41a2940 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f77ac037040 tx=0x7f77ac034d00 comp rx=0 tx=0).stop 2024-04-02T14:35:34.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.486+0000 7f77b9a00640 1 -- 172.21.15.90:0/3035416980 shutdown_connections 2024-04-02T14:35:34.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.486+0000 7f77b9a00640 1 --2- 172.21.15.90:0/3035416980 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f77980a3ec0 0x7f77980a6380 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:34.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.486+0000 7f77b9a00640 1 --2- 172.21.15.90:0/3035416980 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f77b4117040 0x7f77b41a2940 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:34.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.486+0000 7f77b9a00640 1 --2- 172.21.15.90:0/3035416980 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f77b4109600 0x7f77b41a8cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:34.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.486+0000 7f77b9a00640 1 --2- 172.21.15.90:0/3035416980 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f77b4108ce0 0x7f77b41a8790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:34.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.486+0000 7f77b9a00640 1 -- 172.21.15.90:0/3035416980 >> 172.21.15.90:0/3035416980 conn(0x7f77b406cbe0 msgr2=0x7f77b4117480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:35:34.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.486+0000 7f77b9a00640 1 -- 172.21.15.90:0/3035416980 shutdown_connections 2024-04-02T14:35:34.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.486+0000 7f77b9a00640 1 -- 172.21.15.90:0/3035416980 wait complete. 2024-04-02T14:35:34.531 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-02T14:35:34.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.831+0000 7f7392800640 1 -- 172.21.15.90:0/1694616534 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f738c102450 msgr2=0x7f738c10e960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:35:34.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.831+0000 7f7392800640 1 --2- 172.21.15.90:0/1694616534 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f738c102450 0x7f738c10e960 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f7374009930 tx=0x7f737402bc20 comp rx=0 tx=0).stop 2024-04-02T14:35:34.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.832+0000 7f7392800640 1 -- 172.21.15.90:0/1694616534 shutdown_connections 2024-04-02T14:35:34.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.832+0000 7f7392800640 1 --2- 172.21.15.90:0/1694616534 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f738c102450 0x7f738c10e960 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:34.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.832+0000 7f7392800640 1 --2- 172.21.15.90:0/1694616534 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f738c101a90 0x7f738c101f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:34.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.832+0000 7f7392800640 1 --2- 172.21.15.90:0/1694616534 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f738c107aa0 0x7f738c107e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:34.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.832+0000 7f7392800640 1 -- 172.21.15.90:0/1694616534 >> 172.21.15.90:0/1694616534 conn(0x7f738c078f00 msgr2=0x7f738c079310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:35:34.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.832+0000 7f7392800640 1 -- 172.21.15.90:0/1694616534 shutdown_connections 2024-04-02T14:35:34.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.832+0000 7f7392800640 1 -- 172.21.15.90:0/1694616534 wait complete. 2024-04-02T14:35:34.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.834+0000 7f7392800640 1 Processor -- start 2024-04-02T14:35:34.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.834+0000 7f7392800640 1 -- start start 2024-04-02T14:35:34.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.834+0000 7f7392800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f738c101a90 0x7f738c1a4550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:34.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.834+0000 7f7392800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f738c102450 0x7f738c1a4a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:34.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.834+0000 7f7392800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f738c107aa0 0x7f738c10d890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:34.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.834+0000 7f7392800640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f738c111f00 con 0x7f738c101a90 2024-04-02T14:35:34.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.834+0000 7f7392800640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f738c111d80 con 0x7f738c107aa0 2024-04-02T14:35:34.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.834+0000 7f7392800640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f738c112080 con 0x7f738c102450 2024-04-02T14:35:34.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.835+0000 7f738be00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f738c107aa0 0x7f738c10d890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:34.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.835+0000 7f738be00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f738c107aa0 0x7f738c10d890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:57878/0 (socket says 172.21.15.90:57878) 2024-04-02T14:35:34.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.835+0000 7f738be00640 1 -- 172.21.15.90:0/1490060787 learned_addr learned my addr 172.21.15.90:0/1490060787 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:35:34.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.835+0000 7f738aa00640 1 --2- 172.21.15.90:0/1490060787 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f738c102450 0x7f738c1a4a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:34.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.835+0000 7f738b400640 1 --2- 172.21.15.90:0/1490060787 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f738c101a90 0x7f738c1a4550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:34.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.836+0000 7f738be00640 1 -- 172.21.15.90:0/1490060787 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f738c102450 msgr2=0x7f738c1a4a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:35:34.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.836+0000 7f738be00640 1 --2- 172.21.15.90:0/1490060787 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f738c102450 0x7f738c1a4a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:34.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.836+0000 7f738be00640 1 -- 172.21.15.90:0/1490060787 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f738c101a90 msgr2=0x7f738c1a4550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:35:34.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.836+0000 7f738be00640 1 --2- 172.21.15.90:0/1490060787 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f738c101a90 0x7f738c1a4550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:34.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.836+0000 7f738be00640 1 -- 172.21.15.90:0/1490060787 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f738c10e070 con 0x7f738c107aa0 2024-04-02T14:35:34.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.836+0000 7f738b400640 1 --2- 172.21.15.90:0/1490060787 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f738c101a90 0x7f738c1a4550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:35:34.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.836+0000 7f738be00640 1 --2- 172.21.15.90:0/1490060787 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f738c107aa0 0x7f738c10d890 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f737402c230 tx=0x7f7374005e60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:35:34.840 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.837+0000 7f7383e00640 1 -- 172.21.15.90:0/1490060787 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7374043020 con 0x7f738c107aa0 2024-04-02T14:35:34.840 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.837+0000 7f7383e00640 1 -- 172.21.15.90:0/1490060787 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f73740054e0 con 0x7f738c107aa0 2024-04-02T14:35:34.840 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.837+0000 7f7383e00640 1 -- 172.21.15.90:0/1490060787 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7374003c50 con 0x7f738c107aa0 2024-04-02T14:35:34.840 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.837+0000 7f7392800640 1 -- 172.21.15.90:0/1490060787 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f738c10e300 con 0x7f738c107aa0 2024-04-02T14:35:34.840 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.838+0000 7f7392800640 1 -- 172.21.15.90:0/1490060787 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f738c06b520 con 0x7f738c107aa0 2024-04-02T14:35:34.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.841+0000 7f739562c740 1 Processor -- start 2024-04-02T14:35:34.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.842+0000 7f7383e00640 1 -- 172.21.15.90:0/1490060787 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7f7374046020 con 0x7f738c107aa0 2024-04-02T14:35:34.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.842+0000 7f739562c740 1 -- start start 2024-04-02T14:35:34.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.843+0000 7f739562c740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a0f0c789f0 0x55a0f0c77740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:34.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.843+0000 7f739562c740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a0f0c77d10 0x55a0f0c78190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:34.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.843+0000 7f739562c740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a0f0c81080 0x55a0f0c83470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:34.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.843+0000 7f739562c740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55a0f0bb8400 con 0x55a0f0c81080 2024-04-02T14:35:34.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.843+0000 7f739562c740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55a0f0b9cf10 con 0x55a0f0c789f0 2024-04-02T14:35:34.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.843+0000 7f739562c740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55a0f0bdc6f0 con 0x55a0f0c77d10 2024-04-02T14:35:34.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.843+0000 7f738b400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a0f0c77d10 0x55a0f0c78190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:34.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.843+0000 7f738aa00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a0f0c789f0 0x55a0f0c77740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:34.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.843+0000 7f738aa00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a0f0c789f0 0x55a0f0c77740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:57894/0 (socket says 172.21.15.90:57894) 2024-04-02T14:35:34.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.843+0000 7f738aa00640 1 -- 172.21.15.90:0/3853900340 learned_addr learned my addr 172.21.15.90:0/3853900340 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:35:34.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.844+0000 7f738aa00640 1 --2- 172.21.15.90:0/3853900340 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a0f0c81080 0x55a0f0c83470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:34.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.844+0000 7f738aa00640 1 -- 172.21.15.90:0/3853900340 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a0f0c77d10 msgr2=0x55a0f0c78190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:35:34.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.844+0000 7f738aa00640 1 --2- 172.21.15.90:0/3853900340 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a0f0c77d10 0x55a0f0c78190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:34.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.844+0000 7f738aa00640 1 -- 172.21.15.90:0/3853900340 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a0f0c81080 msgr2=0x55a0f0c83470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:35:34.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.844+0000 7f738aa00640 1 --2- 172.21.15.90:0/3853900340 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a0f0c81080 0x55a0f0c83470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:34.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.844+0000 7f738aa00640 1 -- 172.21.15.90:0/3853900340 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a0f08f03f0 con 0x55a0f0c789f0 2024-04-02T14:35:34.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.844+0000 7f738b400640 1 --2- 172.21.15.90:0/3853900340 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a0f0c77d10 0x55a0f0c78190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:35:34.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.844+0000 7f738aa00640 1 --2- 172.21.15.90:0/3853900340 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a0f0c789f0 0x55a0f0c77740 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f737c0117a0 tx=0x7f737c005bd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e92c5fe29be86049 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:35:34.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.844+0000 7f7383e00640 1 --2- 172.21.15.90:0/1490060787 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f73700a3e40 0x7f73700a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:34.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.844+0000 7f7380c00640 1 -- 172.21.15.90:0/3853900340 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f737c011030 con 0x55a0f0c789f0 2024-04-02T14:35:34.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.844+0000 7f7380c00640 1 -- 172.21.15.90:0/3853900340 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f737c003e50 con 0x55a0f0c789f0 2024-04-02T14:35:34.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.844+0000 7f7380c00640 1 -- 172.21.15.90:0/3853900340 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f737c004250 con 0x55a0f0c789f0 2024-04-02T14:35:34.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.844+0000 7f7383e00640 1 -- 172.21.15.90:0/1490060787 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f73740f29c0 con 0x7f738c107aa0 2024-04-02T14:35:34.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.845+0000 7f739562c740 1 -- 172.21.15.90:0/3853900340 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a0f0c789f0 msgr2=0x55a0f0c77740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:35:34.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.845+0000 7f739562c740 1 --2- 172.21.15.90:0/3853900340 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a0f0c789f0 0x55a0f0c77740 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f737c0117a0 tx=0x7f737c005bd0 comp rx=0 tx=0).stop 2024-04-02T14:35:34.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.846+0000 7f738b400640 1 --2- 172.21.15.90:0/1490060787 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f73700a3e40 0x7f73700a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:34.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.846+0000 7f739562c740 1 -- 172.21.15.90:0/3853900340 shutdown_connections 2024-04-02T14:35:34.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.846+0000 7f738b400640 1 --2- 172.21.15.90:0/1490060787 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f73700a3e40 0x7f73700a6300 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x55a0f0c78f40 tx=0x7f738400b550 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:35:34.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.849+0000 7f739562c740 1 --2- 172.21.15.90:0/3853900340 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a0f0c81080 0x55a0f0c83470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:34.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.849+0000 7f739562c740 1 --2- 172.21.15.90:0/3853900340 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a0f0c77d10 0x55a0f0c78190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:34.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.849+0000 7f739562c740 1 --2- 172.21.15.90:0/3853900340 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a0f0c789f0 0x55a0f0c77740 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:34.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.849+0000 7f739562c740 1 -- 172.21.15.90:0/3853900340 >> 172.21.15.90:0/3853900340 conn(0x55a0f0c3ea50 msgr2=0x55a0f0c72760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:35:34.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.850+0000 7f739562c740 1 -- 172.21.15.90:0/3853900340 shutdown_connections 2024-04-02T14:35:34.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.850+0000 7f739562c740 1 -- 172.21.15.90:0/3853900340 wait complete. 2024-04-02T14:35:34.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.851+0000 7f739562c740 1 -- start start 2024-04-02T14:35:34.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.852+0000 7f739562c740 1 Processor -- start 2024-04-02T14:35:34.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.853+0000 7f739562c740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a0f0c77d10 0x55a0f0c7e050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:34.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.853+0000 7f739562c740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a0f0c789f0 0x55a0f0c7e590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:34.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.853+0000 7f739562c740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a0f0c81080 0x55a0f0c7ead0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:34.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.853+0000 7f739562c740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55a0f0bd86a0 con 0x55a0f0c789f0 2024-04-02T14:35:34.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.853+0000 7f739562c740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55a0f0b84980 con 0x55a0f0c77d10 2024-04-02T14:35:34.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.853+0000 7f739562c740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55a0f0b97cb0 con 0x55a0f0c81080 2024-04-02T14:35:34.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.853+0000 7f739562c740 10 client.0 _ll_register_callbacks cb 0x55a0f0b976a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:35:34.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.854+0000 7f738be00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a0f0c81080 0x55a0f0c7ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:34.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.854+0000 7f738be00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a0f0c81080 0x55a0f0c7ead0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:41070/0 (socket says 172.21.15.90:41070) 2024-04-02T14:35:34.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.854+0000 7f738be00640 1 -- 172.21.15.90:0/3313953971 learned_addr learned my addr 172.21.15.90:0/3313953971 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:35:34.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.854+0000 7f738be00640 1 -- 172.21.15.90:0/3313953971 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a0f0c77d10 msgr2=0x55a0f0c7e050 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-02T14:35:34.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.854+0000 7f738be00640 1 --2- 172.21.15.90:0/3313953971 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a0f0c77d10 0x55a0f0c7e050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:34.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.854+0000 7f738be00640 1 -- 172.21.15.90:0/3313953971 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a0f0c789f0 msgr2=0x55a0f0c7e590 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-02T14:35:34.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.855+0000 7f738aa00640 1 --2- 172.21.15.90:0/3313953971 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a0f0c789f0 0x55a0f0c7e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:34.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.855+0000 7f738be00640 1 --2- 172.21.15.90:0/3313953971 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a0f0c789f0 0x55a0f0c7e590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:34.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.855+0000 7f738be00640 1 -- 172.21.15.90:0/3313953971 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a0f09ba7d0 con 0x55a0f0c81080 2024-04-02T14:35:34.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.856+0000 7f738be00640 1 --2- 172.21.15.90:0/3313953971 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a0f0c81080 0x55a0f0c7ead0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f73740c6f70 tx=0x7f73740bac00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=387537364d239677 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:35:34.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.856+0000 7f7379600640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:35:34.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.856+0000 7f7379600640 1 -- 172.21.15.90:0/3313953971 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73740bef30 con 0x55a0f0c81080 2024-04-02T14:35:34.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.856+0000 7f7379600640 1 -- 172.21.15.90:0/3313953971 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f73740a90c0 con 0x55a0f0c81080 2024-04-02T14:35:34.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.856+0000 7f739562c740 1 -- 172.21.15.90:0/3313953971 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f737c003e50 con 0x55a0f0c81080 2024-04-02T14:35:34.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.857+0000 7f7379600640 1 -- 172.21.15.90:0/3313953971 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f73740be830 con 0x55a0f0c81080 2024-04-02T14:35:34.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.857+0000 7f739562c740 10 client.25588 fetch_fsmap learned FSMap version 27 2024-04-02T14:35:34.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.857+0000 7f739562c740 1 -- 172.21.15.90:0/3313953971 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55a0f0ca0f40 con 0x55a0f0c81080 2024-04-02T14:35:34.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.857+0000 7f7379600640 1 -- 172.21.15.90:0/3313953971 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f737c003e50 con 0x55a0f0c81080 2024-04-02T14:35:34.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.857+0000 7f7379600640 1 -- 172.21.15.90:0/3313953971 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f73740c7010 con 0x55a0f0c81080 2024-04-02T14:35:34.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.858+0000 7f739562c740 10 client.25588 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:35:34.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.858+0000 7f739562c740 10 client.25588 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-04-02T14:35:34.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.858+0000 7f739562c740 20 client.25588 populate_metadata read hostname 'smithi090' 2024-04-02T14:35:34.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.858+0000 7f739562c740 1 --2- 172.21.15.90:0/3313953971 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x55a0f0ca2350 0x55a0f0ca47a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:34.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.858+0000 7f739562c740 4 client.25588 mds_command: new command op to 24547 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:35:34.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.858+0000 7f739562c740 1 -- 172.21.15.90:0/3313953971 --> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a0f0b9c040 con 0x55a0f0ca2350 2024-04-02T14:35:34.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.858+0000 7f738b400640 1 --2- 172.21.15.90:0/3313953971 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x55a0f0ca2350 0x55a0f0ca47a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:34.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.859+0000 7f7379600640 10 client.25588 ms_handle_connect on v2:172.21.15.177:6838/664895663 2024-04-02T14:35:34.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.859+0000 7f738b400640 1 --2- 172.21.15.90:0/3313953971 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x55a0f0ca2350 0x55a0f0ca47a0 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=da4a9aa2c6b58c85 server_cookie=5f92535d00f61883 in_seq=0 out_seq=0 2024-04-02T14:35:34.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.860+0000 7f7379600640 1 -- 172.21.15.90:0/3313953971 <== mds.2 v2:172.21.15.177:6838/664895663 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a0f0b9c040 con 0x55a0f0ca2350 2024-04-02T14:35:34.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.860+0000 7f7379600640 10 client.25588 handle_command_reply: tid=0 2024-04-02T14:35:34.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.860+0000 7f739562c740 1 client.25588 shutdown 2024-04-02T14:35:34.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.860+0000 7f739562c740 1 -- 172.21.15.90:0/3313953971 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a0f0c81080 msgr2=0x55a0f0c7ead0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:35:34.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.860+0000 7f739562c740 1 --2- 172.21.15.90:0/3313953971 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a0f0c81080 0x55a0f0c7ead0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f73740c6f70 tx=0x7f73740bac00 comp rx=0 tx=0).stop 2024-04-02T14:35:34.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.861+0000 7f739562c740 1 -- 172.21.15.90:0/3313953971 shutdown_connections 2024-04-02T14:35:34.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.861+0000 7f739562c740 1 --2- 172.21.15.90:0/3313953971 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x55a0f0ca2350 0x55a0f0ca47a0 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:34.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.861+0000 7f739562c740 1 --2- 172.21.15.90:0/3313953971 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a0f0c81080 0x55a0f0c7ead0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:34.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.861+0000 7f7379600640 0 client.25588 ms_handle_reset on v2:172.21.15.177:6838/664895663 2024-04-02T14:35:34.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.861+0000 7f739562c740 1 --2- 172.21.15.90:0/3313953971 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a0f0c789f0 0x55a0f0c7e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:34.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.861+0000 7f739562c740 1 --2- 172.21.15.90:0/3313953971 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a0f0c77d10 0x55a0f0c7e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:34.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.861+0000 7f739562c740 1 -- 172.21.15.90:0/3313953971 >> 172.21.15.90:0/3313953971 conn(0x55a0f0c3ea50 msgr2=0x55a0f0c71010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:35:34.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.862+0000 7f739562c740 1 -- 172.21.15.90:0/3313953971 shutdown_connections 2024-04-02T14:35:34.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.862+0000 7f739562c740 1 -- 172.21.15.90:0/3313953971 wait complete. 2024-04-02T14:35:34.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.862+0000 7f739562c740 20 client.25588 trim_cache size 0 max 16384 2024-04-02T14:35:34.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.886+0000 7f739562c740 1 Processor -- start 2024-04-02T14:35:34.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.887+0000 7f739562c740 1 -- start start 2024-04-02T14:35:34.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.887+0000 7f739562c740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a0f0c77d10 0x55a0f0c80780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:34.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.887+0000 7f739562c740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a0f0c789f0 0x55a0f0c9db30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:34.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.887+0000 7f739562c740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a0f0c81080 0x55a0f0c71720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:34.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.887+0000 7f739562c740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f738c197750 con 0x55a0f0c77d10 2024-04-02T14:35:34.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.887+0000 7f739562c740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f738c199290 con 0x55a0f0c789f0 2024-04-02T14:35:34.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.887+0000 7f739562c740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f738c197e20 con 0x55a0f0c81080 2024-04-02T14:35:34.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.888+0000 7f738be00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a0f0c81080 0x55a0f0c71720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:34.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.888+0000 7f738be00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a0f0c81080 0x55a0f0c71720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:41076/0 (socket says 172.21.15.90:41076) 2024-04-02T14:35:34.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.888+0000 7f738be00640 1 -- 172.21.15.90:0/3619167636 learned_addr learned my addr 172.21.15.90:0/3619167636 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:35:34.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.888+0000 7f738aa00640 1 --2- 172.21.15.90:0/3619167636 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a0f0c789f0 0x55a0f0c9db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:34.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.889+0000 7f738b400640 1 --2- 172.21.15.90:0/3619167636 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a0f0c77d10 0x55a0f0c80780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:34.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.889+0000 7f738aa00640 1 -- 172.21.15.90:0/3619167636 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a0f0c81080 msgr2=0x55a0f0c71720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:35:34.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.889+0000 7f738aa00640 1 --2- 172.21.15.90:0/3619167636 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a0f0c81080 0x55a0f0c71720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:34.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.889+0000 7f738aa00640 1 -- 172.21.15.90:0/3619167636 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a0f0c77d10 msgr2=0x55a0f0c80780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:35:34.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.890+0000 7f738aa00640 1 --2- 172.21.15.90:0/3619167636 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a0f0c77d10 0x55a0f0c80780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:34.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.890+0000 7f738aa00640 1 -- 172.21.15.90:0/3619167636 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a0f0c79580 con 0x55a0f0c789f0 2024-04-02T14:35:34.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.890+0000 7f738be00640 1 --2- 172.21.15.90:0/3619167636 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a0f0c81080 0x55a0f0c71720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:35:34.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.890+0000 7f738b400640 1 --2- 172.21.15.90:0/3619167636 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a0f0c77d10 0x55a0f0c80780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:35:34.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.891+0000 7f738aa00640 1 --2- 172.21.15.90:0/3619167636 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a0f0c789f0 0x55a0f0c9db30 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f737c005c40 tx=0x7f737c0092a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=46d4aed94ef4e4fa server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:35:34.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.891+0000 7f737be00640 1 -- 172.21.15.90:0/3619167636 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f737c003c30 con 0x55a0f0c789f0 2024-04-02T14:35:34.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.891+0000 7f737be00640 1 -- 172.21.15.90:0/3619167636 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f737c003dd0 con 0x55a0f0c789f0 2024-04-02T14:35:34.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.891+0000 7f737be00640 1 -- 172.21.15.90:0/3619167636 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f737c005840 con 0x55a0f0c789f0 2024-04-02T14:35:34.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.892+0000 7f739562c740 1 -- 172.21.15.90:0/3619167636 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a0f0c789f0 msgr2=0x55a0f0c9db30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:35:34.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.892+0000 7f739562c740 1 --2- 172.21.15.90:0/3619167636 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a0f0c789f0 0x55a0f0c9db30 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f737c005c40 tx=0x7f737c0092a0 comp rx=0 tx=0).stop 2024-04-02T14:35:34.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.892+0000 7f739562c740 1 -- 172.21.15.90:0/3619167636 shutdown_connections 2024-04-02T14:35:34.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.892+0000 7f739562c740 1 --2- 172.21.15.90:0/3619167636 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a0f0c81080 0x55a0f0c71720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:34.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.893+0000 7f739562c740 1 --2- 172.21.15.90:0/3619167636 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a0f0c789f0 0x55a0f0c9db30 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:34.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.893+0000 7f739562c740 1 --2- 172.21.15.90:0/3619167636 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a0f0c77d10 0x55a0f0c80780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:34.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.893+0000 7f739562c740 1 -- 172.21.15.90:0/3619167636 >> 172.21.15.90:0/3619167636 conn(0x55a0f0c3ea50 msgr2=0x55a0f0c71010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:35:34.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.893+0000 7f739562c740 1 -- 172.21.15.90:0/3619167636 shutdown_connections 2024-04-02T14:35:34.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.893+0000 7f739562c740 1 -- 172.21.15.90:0/3619167636 wait complete. 2024-04-02T14:35:34.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.894+0000 7f739562c740 1 -- start start 2024-04-02T14:35:34.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.894+0000 7f739562c740 1 Processor -- start 2024-04-02T14:35:34.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.895+0000 7f739562c740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a0f0c77d10 0x55a0f0c9a040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:34.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.895+0000 7f739562c740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a0f0c789f0 0x55a0f0c9a580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:34.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.895+0000 7f739562c740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a0f0c81080 0x55a0f0c97170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:34.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.895+0000 7f738b400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a0f0c77d10 0x55a0f0c9a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:34.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.896+0000 7f738b400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a0f0c77d10 0x55a0f0c9a040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:57914/0 (socket says 172.21.15.90:57914) 2024-04-02T14:35:34.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.896+0000 7f738aa00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a0f0c789f0 0x55a0f0c9a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:34.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.896+0000 7f738aa00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a0f0c789f0 0x55a0f0c9a580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:57548/0 (socket says 172.21.15.90:57548) 2024-04-02T14:35:34.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.896+0000 7f739562c740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f738c198bc0 con 0x55a0f0c789f0 2024-04-02T14:35:34.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.896+0000 7f739562c740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f738c1984f0 con 0x55a0f0c77d10 2024-04-02T14:35:34.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.896+0000 7f739562c740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f738c1031d0 con 0x55a0f0c81080 2024-04-02T14:35:34.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.896+0000 7f739562c740 10 client.0 _ll_register_callbacks cb 0x7f7374009560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:35:34.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.896+0000 7f738be00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a0f0c81080 0x55a0f0c97170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:34.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.896+0000 7f738be00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a0f0c81080 0x55a0f0c97170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:41080/0 (socket says 172.21.15.90:41080) 2024-04-02T14:35:34.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.896+0000 7f738be00640 1 -- 172.21.15.90:0/1310722812 learned_addr learned my addr 172.21.15.90:0/1310722812 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:35:34.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.897+0000 7f738be00640 1 -- 172.21.15.90:0/1310722812 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a0f0c77d10 msgr2=0x55a0f0c9a040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:35:34.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.897+0000 7f738be00640 1 --2- 172.21.15.90:0/1310722812 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a0f0c77d10 0x55a0f0c9a040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:34.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.897+0000 7f738be00640 1 -- 172.21.15.90:0/1310722812 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a0f0c789f0 msgr2=0x55a0f0c9a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:35:34.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.897+0000 7f738be00640 1 --2- 172.21.15.90:0/1310722812 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a0f0c789f0 0x55a0f0c9a580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:34.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.897+0000 7f738be00640 1 -- 172.21.15.90:0/1310722812 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a0f0c979a0 con 0x55a0f0c81080 2024-04-02T14:35:34.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.898+0000 7f738be00640 1 --2- 172.21.15.90:0/1310722812 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a0f0c81080 0x55a0f0c97170 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f73740bb200 tx=0x7f7374004310 comp rx=0 tx=0).ready entity=mon.1 client_cookie=40d9287d5e7ef43f server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:35:34.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.898+0000 7f7379600640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:35:34.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.898+0000 7f7379600640 1 -- 172.21.15.90:0/1310722812 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73740bec90 con 0x55a0f0c81080 2024-04-02T14:35:34.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.898+0000 7f7379600640 1 -- 172.21.15.90:0/1310722812 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f73740bfb60 con 0x55a0f0c81080 2024-04-02T14:35:34.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.899+0000 7f739562c740 1 -- 172.21.15.90:0/1310722812 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f737c003dd0 con 0x55a0f0c81080 2024-04-02T14:35:34.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.899+0000 7f7379600640 1 -- 172.21.15.90:0/1310722812 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f73740efff0 con 0x55a0f0c81080 2024-04-02T14:35:34.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.899+0000 7f739562c740 10 client.25594 fetch_fsmap learned FSMap version 27 2024-04-02T14:35:34.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.900+0000 7f739562c740 1 -- 172.21.15.90:0/1310722812 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55a0f0d09570 con 0x55a0f0c81080 2024-04-02T14:35:34.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.900+0000 7f7379600640 1 -- 172.21.15.90:0/1310722812 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f737c003dd0 con 0x55a0f0c81080 2024-04-02T14:35:34.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.900+0000 7f7379600640 1 -- 172.21.15.90:0/1310722812 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f73740bdc80 con 0x55a0f0c81080 2024-04-02T14:35:34.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.900+0000 7f739562c740 10 client.25594 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:35:34.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.900+0000 7f739562c740 10 client.25594 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-04-02T14:35:34.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.900+0000 7f739562c740 20 client.25594 populate_metadata read hostname 'smithi090' 2024-04-02T14:35:34.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.900+0000 7f739562c740 1 --2- 172.21.15.90:0/1310722812 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x55a0f0ca2350 0x55a0f0d0bb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:34.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.900+0000 7f739562c740 4 client.25594 mds_command: new command op to 24547 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-02T14:35:34.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.900+0000 7f739562c740 1 -- 172.21.15.90:0/1310722812 --> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55a0f0be9c30 con 0x55a0f0ca2350 2024-04-02T14:35:34.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.901+0000 7f738b400640 1 --2- 172.21.15.90:0/1310722812 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x55a0f0ca2350 0x55a0f0d0bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:34.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.901+0000 7f7379600640 10 client.25594 ms_handle_connect on v2:172.21.15.177:6838/664895663 2024-04-02T14:35:34.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.902+0000 7f738b400640 1 --2- 172.21.15.90:0/1310722812 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x55a0f0ca2350 0x55a0f0d0bb20 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=81b7e5ac19b1b852 server_cookie=542c624e0b23b17c in_seq=0 out_seq=0 2024-04-02T14:35:34.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.902+0000 7f7379600640 1 -- 172.21.15.90:0/1310722812 <== mds.2 v2:172.21.15.177:6838/664895663 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55a0f0be9c30 con 0x55a0f0ca2350 2024-04-02T14:35:34.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.903+0000 7f7379600640 10 client.25594 handle_command_reply: tid=0 2024-04-02T14:35:34.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.903+0000 7f739562c740 1 client.25594 shutdown 2024-04-02T14:35:34.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.903+0000 7f739562c740 1 -- 172.21.15.90:0/1310722812 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a0f0c81080 msgr2=0x55a0f0c97170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:35:34.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.903+0000 7f739562c740 1 --2- 172.21.15.90:0/1310722812 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a0f0c81080 0x55a0f0c97170 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f73740bb200 tx=0x7f7374004310 comp rx=0 tx=0).stop 2024-04-02T14:35:34.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.903+0000 7f739562c740 1 -- 172.21.15.90:0/1310722812 shutdown_connections 2024-04-02T14:35:34.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.903+0000 7f739562c740 1 --2- 172.21.15.90:0/1310722812 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x55a0f0ca2350 0x55a0f0d0bb20 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:34.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.903+0000 7f739562c740 1 --2- 172.21.15.90:0/1310722812 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a0f0c81080 0x55a0f0c97170 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:34.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.903+0000 7f739562c740 1 --2- 172.21.15.90:0/1310722812 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a0f0c789f0 0x55a0f0c9a580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:34.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.903+0000 7f739562c740 1 --2- 172.21.15.90:0/1310722812 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a0f0c77d10 0x55a0f0c9a040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:34.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.903+0000 7f739562c740 1 -- 172.21.15.90:0/1310722812 >> 172.21.15.90:0/1310722812 conn(0x55a0f0c3ea50 msgr2=0x55a0f0c73b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:35:34.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.903+0000 7f7379600640 0 client.25594 ms_handle_reset on v2:172.21.15.177:6838/664895663 2024-04-02T14:35:34.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.903+0000 7f739562c740 1 -- 172.21.15.90:0/1310722812 shutdown_connections 2024-04-02T14:35:34.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.904+0000 7f739562c740 1 -- 172.21.15.90:0/1310722812 wait complete. 2024-04-02T14:35:34.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.904+0000 7f739562c740 20 client.25594 trim_cache size 0 max 16384 2024-04-02T14:35:34.907 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-02T14:35:34.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.905+0000 7f7392800640 1 -- 172.21.15.90:0/1490060787 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f73700a3e40 msgr2=0x7f73700a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:35:34.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.905+0000 7f7392800640 1 --2- 172.21.15.90:0/1490060787 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f73700a3e40 0x7f73700a6300 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x55a0f0c78f40 tx=0x7f738400b550 comp rx=0 tx=0).stop 2024-04-02T14:35:34.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.905+0000 7f7392800640 1 -- 172.21.15.90:0/1490060787 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f738c107aa0 msgr2=0x7f738c10d890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:35:34.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.905+0000 7f7392800640 1 --2- 172.21.15.90:0/1490060787 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f738c107aa0 0x7f738c10d890 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f737402c230 tx=0x7f7374005e60 comp rx=0 tx=0).stop 2024-04-02T14:35:34.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.905+0000 7f7392800640 1 -- 172.21.15.90:0/1490060787 shutdown_connections 2024-04-02T14:35:34.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.905+0000 7f7392800640 1 --2- 172.21.15.90:0/1490060787 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f73700a3e40 0x7f73700a6300 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:34.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.905+0000 7f7392800640 1 --2- 172.21.15.90:0/1490060787 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f738c107aa0 0x7f738c10d890 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:34.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.905+0000 7f7392800640 1 --2- 172.21.15.90:0/1490060787 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f738c102450 0x7f738c1a4a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:34.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.905+0000 7f7392800640 1 --2- 172.21.15.90:0/1490060787 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f738c101a90 0x7f738c1a4550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:34.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.905+0000 7f7392800640 1 -- 172.21.15.90:0/1490060787 >> 172.21.15.90:0/1490060787 conn(0x7f738c078f00 msgr2=0x7f738c103f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:35:34.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.906+0000 7f7392800640 1 -- 172.21.15.90:0/1490060787 shutdown_connections 2024-04-02T14:35:34.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:34.906+0000 7f7392800640 1 -- 172.21.15.90:0/1490060787 wait complete. 2024-04-02T14:35:34.939 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-02T14:35:35.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.213+0000 7f8ec9e00640 1 -- 172.21.15.90:0/4029508275 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8ec41069c0 msgr2=0x7f8ec4106da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:35:35.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.213+0000 7f8ec9e00640 1 --2- 172.21.15.90:0/4029508275 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8ec41069c0 0x7f8ec4106da0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f8eb4009900 tx=0x7f8eb402de50 comp rx=0 tx=0).stop 2024-04-02T14:35:35.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.213+0000 7f8ec9e00640 1 -- 172.21.15.90:0/4029508275 shutdown_connections 2024-04-02T14:35:35.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.213+0000 7f8ec9e00640 1 --2- 172.21.15.90:0/4029508275 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8ec4101370 0x7f8ec410d880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:35.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.213+0000 7f8ec9e00640 1 --2- 172.21.15.90:0/4029508275 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8ec41009b0 0x7f8ec4100e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:35.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.213+0000 7f8ec9e00640 1 --2- 172.21.15.90:0/4029508275 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8ec41069c0 0x7f8ec4106da0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:35.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.213+0000 7f8ec9e00640 1 -- 172.21.15.90:0/4029508275 >> 172.21.15.90:0/4029508275 conn(0x7f8ec4076d80 msgr2=0x7f8ec4077190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:35:35.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.214+0000 7f8ec9e00640 1 -- 172.21.15.90:0/4029508275 shutdown_connections 2024-04-02T14:35:35.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.214+0000 7f8ec9e00640 1 -- 172.21.15.90:0/4029508275 wait complete. 2024-04-02T14:35:35.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.215+0000 7f8ec9e00640 1 Processor -- start 2024-04-02T14:35:35.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.215+0000 7f8ec9e00640 1 -- start start 2024-04-02T14:35:35.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.215+0000 7f8ec9e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8ec41009b0 0x7f8ec4196170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:35.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.216+0000 7f8ec9e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8ec4101370 0x7f8ec41966b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:35.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.216+0000 7f8ec9e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8ec41069c0 0x7f8ec419aa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:35.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.216+0000 7f8ec9e00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f8ec410fd10 con 0x7f8ec41069c0 2024-04-02T14:35:35.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.216+0000 7f8ec9e00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f8ec410fb90 con 0x7f8ec4101370 2024-04-02T14:35:35.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.216+0000 7f8ec9e00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f8ec410fe90 con 0x7f8ec41009b0 2024-04-02T14:35:35.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.216+0000 7f8ec2a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8ec41009b0 0x7f8ec4196170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:35.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.216+0000 7f8ec2a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8ec41009b0 0x7f8ec4196170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:41096/0 (socket says 172.21.15.90:41096) 2024-04-02T14:35:35.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.216+0000 7f8ec2a00640 1 -- 172.21.15.90:0/2474140794 learned_addr learned my addr 172.21.15.90:0/2474140794 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:35:35.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.216+0000 7f8ebbe00640 1 --2- 172.21.15.90:0/2474140794 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8ec4101370 0x7f8ec41966b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:35.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.217+0000 7f8ec2a00640 1 -- 172.21.15.90:0/2474140794 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8ec4101370 msgr2=0x7f8ec41966b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:35:35.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.217+0000 7f8ec2a00640 1 --2- 172.21.15.90:0/2474140794 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8ec4101370 0x7f8ec41966b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:35.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.217+0000 7f8ec3400640 1 --2- 172.21.15.90:0/2474140794 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8ec41069c0 0x7f8ec419aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:35.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.217+0000 7f8ec2a00640 1 -- 172.21.15.90:0/2474140794 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8ec41069c0 msgr2=0x7f8ec419aa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:35:35.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.217+0000 7f8ec2a00640 1 --2- 172.21.15.90:0/2474140794 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8ec41069c0 0x7f8ec419aa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:35.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.217+0000 7f8ec2a00640 1 -- 172.21.15.90:0/2474140794 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ec419b160 con 0x7f8ec41009b0 2024-04-02T14:35:35.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.217+0000 7f8ebbe00640 1 --2- 172.21.15.90:0/2474140794 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8ec4101370 0x7f8ec41966b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:35:35.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.218+0000 7f8ec2a00640 1 --2- 172.21.15.90:0/2474140794 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8ec41009b0 0x7f8ec4196170 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f8eb4000c00 tx=0x7f8eb40042d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:35:35.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.218+0000 7f8ec8a00640 1 -- 172.21.15.90:0/2474140794 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8eb4036b50 con 0x7f8ec41009b0 2024-04-02T14:35:35.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.218+0000 7f8ec9e00640 1 -- 172.21.15.90:0/2474140794 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ec419b3f0 con 0x7f8ec41009b0 2024-04-02T14:35:35.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.218+0000 7f8ec8a00640 1 -- 172.21.15.90:0/2474140794 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8eb4045070 con 0x7f8ec41009b0 2024-04-02T14:35:35.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.218+0000 7f8ec8a00640 1 -- 172.21.15.90:0/2474140794 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8eb4003e60 con 0x7f8ec41009b0 2024-04-02T14:35:35.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.218+0000 7f8ec9e00640 1 -- 172.21.15.90:0/2474140794 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ec41a70f0 con 0x7f8ec41009b0 2024-04-02T14:35:35.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.222+0000 7f8eccbbb740 1 Processor -- start 2024-04-02T14:35:35.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.222+0000 7f8eccbbb740 1 -- start start 2024-04-02T14:35:35.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.222+0000 7f8eccbbb740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ddb16129f0 0x55ddb1611740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:35.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.222+0000 7f8eccbbb740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ddb1611d10 0x55ddb1612190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:35.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.222+0000 7f8eccbbb740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ddb161b080 0x55ddb161d470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:35.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.222+0000 7f8eccbbb740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55ddb1552400 con 0x55ddb161b080 2024-04-02T14:35:35.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.222+0000 7f8eccbbb740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55ddb1536f10 con 0x55ddb1611d10 2024-04-02T14:35:35.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.222+0000 7f8eccbbb740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55ddb15766f0 con 0x55ddb16129f0 2024-04-02T14:35:35.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.223+0000 7f8ec3400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ddb16129f0 0x55ddb1611740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:35.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.223+0000 7f8ec3400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ddb16129f0 0x55ddb1611740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:41100/0 (socket says 172.21.15.90:41100) 2024-04-02T14:35:35.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.223+0000 7f8ec3400640 1 -- 172.21.15.90:0/1852745559 learned_addr learned my addr 172.21.15.90:0/1852745559 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:35:35.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.223+0000 7f8ec3400640 1 -- 172.21.15.90:0/1852745559 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ddb1611d10 msgr2=0x55ddb1612190 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-02T14:35:35.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.223+0000 7f8ec3400640 1 --2- 172.21.15.90:0/1852745559 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ddb1611d10 0x55ddb1612190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:35.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.223+0000 7f8ec3400640 1 -- 172.21.15.90:0/1852745559 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ddb161b080 msgr2=0x55ddb161d470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:35:35.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.223+0000 7f8ec3400640 1 --2- 172.21.15.90:0/1852745559 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ddb161b080 0x55ddb161d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:35.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.223+0000 7f8ec3400640 1 -- 172.21.15.90:0/1852745559 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ddb128a3f0 con 0x55ddb16129f0 2024-04-02T14:35:35.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.223+0000 7f8ec3400640 1 --2- 172.21.15.90:0/1852745559 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ddb16129f0 0x55ddb1611740 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f8ebc009d70 tx=0x7f8ebc00f730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=836b8ef607418775 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:35:35.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.224+0000 7f8eb3e00640 1 -- 172.21.15.90:0/1852745559 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ebc003970 con 0x55ddb16129f0 2024-04-02T14:35:35.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.224+0000 7f8eb3e00640 1 -- 172.21.15.90:0/1852745559 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8ebc004030 con 0x55ddb16129f0 2024-04-02T14:35:35.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.224+0000 7f8eb3e00640 1 -- 172.21.15.90:0/1852745559 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ebc011800 con 0x55ddb16129f0 2024-04-02T14:35:35.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.225+0000 7f8eccbbb740 1 -- 172.21.15.90:0/1852745559 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ddb16129f0 msgr2=0x55ddb1611740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:35:35.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.225+0000 7f8eccbbb740 1 --2- 172.21.15.90:0/1852745559 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ddb16129f0 0x55ddb1611740 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f8ebc009d70 tx=0x7f8ebc00f730 comp rx=0 tx=0).stop 2024-04-02T14:35:35.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.225+0000 7f8eccbbb740 1 -- 172.21.15.90:0/1852745559 shutdown_connections 2024-04-02T14:35:35.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.225+0000 7f8eccbbb740 1 --2- 172.21.15.90:0/1852745559 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ddb161b080 0x55ddb161d470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:35.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.225+0000 7f8eccbbb740 1 --2- 172.21.15.90:0/1852745559 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ddb1611d10 0x55ddb1612190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:35.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.225+0000 7f8eccbbb740 1 --2- 172.21.15.90:0/1852745559 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ddb16129f0 0x55ddb1611740 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:35.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.225+0000 7f8eccbbb740 1 -- 172.21.15.90:0/1852745559 >> 172.21.15.90:0/1852745559 conn(0x55ddb15d8a50 msgr2=0x55ddb160c760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:35:35.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.229+0000 7f8ec8a00640 1 -- 172.21.15.90:0/2474140794 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7f8eb40035f0 con 0x7f8ec41009b0 2024-04-02T14:35:35.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.230+0000 7f8eccbbb740 1 -- 172.21.15.90:0/1852745559 shutdown_connections 2024-04-02T14:35:35.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.232+0000 7f8ec8a00640 1 --2- 172.21.15.90:0/2474140794 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f8ea80a3dd0 0x7f8ea80a6290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:35.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.232+0000 7f8ec8a00640 1 -- 172.21.15.90:0/2474140794 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f8eb40f2f00 con 0x7f8ec41009b0 2024-04-02T14:35:35.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.233+0000 7f8eccbbb740 1 -- 172.21.15.90:0/1852745559 wait complete. 2024-04-02T14:35:35.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.233+0000 7f8ebbe00640 1 --2- 172.21.15.90:0/2474140794 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f8ea80a3dd0 0x7f8ea80a6290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:35.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.234+0000 7f8eccbbb740 1 -- start start 2024-04-02T14:35:35.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.234+0000 7f8eccbbb740 1 Processor -- start 2024-04-02T14:35:35.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.235+0000 7f8ebbe00640 1 --2- 172.21.15.90:0/2474140794 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f8ea80a3dd0 0x7f8ea80a6290 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f8eac004700 tx=0x7f8eac005850 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:35:35.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.235+0000 7f8eccbbb740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ddb16129f0 0x55ddb16180a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:35.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.235+0000 7f8eccbbb740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ddb161b080 0x55ddb16185e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:35.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.235+0000 7f8eccbbb740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ddb1618b20 0x55ddb16408f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:35.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.235+0000 7f8eccbbb740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55ddb15726a0 con 0x55ddb1618b20 2024-04-02T14:35:35.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.235+0000 7f8eccbbb740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55ddb151e980 con 0x55ddb161b080 2024-04-02T14:35:35.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.235+0000 7f8eccbbb740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55ddb1531cb0 con 0x55ddb16129f0 2024-04-02T14:35:35.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.236+0000 7f8eccbbb740 10 client.0 _ll_register_callbacks cb 0x55ddb15316a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:35:35.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.236+0000 7f8ec2a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ddb16129f0 0x55ddb16180a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:35.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.236+0000 7f8ec2a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ddb16129f0 0x55ddb16180a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:41108/0 (socket says 172.21.15.90:41108) 2024-04-02T14:35:35.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.236+0000 7f8ec2a00640 1 -- 172.21.15.90:0/3837115152 learned_addr learned my addr 172.21.15.90:0/3837115152 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:35:35.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.236+0000 7f8ec3400640 1 --2- 172.21.15.90:0/3837115152 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ddb1618b20 0x55ddb16408f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:35.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.236+0000 7f8ebbe00640 1 --2- 172.21.15.90:0/3837115152 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ddb161b080 0x55ddb16185e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:35.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.237+0000 7f8ebbe00640 1 -- 172.21.15.90:0/3837115152 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ddb16129f0 msgr2=0x55ddb16180a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:35:35.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.237+0000 7f8ebbe00640 1 --2- 172.21.15.90:0/3837115152 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ddb16129f0 0x55ddb16180a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:35.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.237+0000 7f8ebbe00640 1 -- 172.21.15.90:0/3837115152 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ddb1618b20 msgr2=0x55ddb16408f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:35:35.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.237+0000 7f8ebbe00640 1 --2- 172.21.15.90:0/3837115152 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ddb1618b20 0x55ddb16408f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:35.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.237+0000 7f8ebbe00640 1 -- 172.21.15.90:0/3837115152 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ddb13547d0 con 0x55ddb161b080 2024-04-02T14:35:35.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.237+0000 7f8ec3400640 1 --2- 172.21.15.90:0/3837115152 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ddb1618b20 0x55ddb16408f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:35:35.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.237+0000 7f8ec2a00640 1 --2- 172.21.15.90:0/3837115152 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ddb16129f0 0x55ddb16180a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:35:35.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.237+0000 7f8ebbe00640 1 --2- 172.21.15.90:0/3837115152 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ddb161b080 0x55ddb16185e0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f8ec4100140 tx=0x7f8eac004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e165d3c1c2cef3c5 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:35:35.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.237+0000 7f8eb0c00640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:35:35.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.238+0000 7f8eb0c00640 1 -- 172.21.15.90:0/3837115152 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8eac023070 con 0x55ddb161b080 2024-04-02T14:35:35.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.238+0000 7f8eb0c00640 1 -- 172.21.15.90:0/3837115152 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f8eac01f650 con 0x55ddb161b080 2024-04-02T14:35:35.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.238+0000 7f8eccbbb740 1 -- 172.21.15.90:0/3837115152 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8ebc004030 con 0x55ddb161b080 2024-04-02T14:35:35.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.239+0000 7f8eb0c00640 1 -- 172.21.15.90:0/3837115152 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f8eac02f050 con 0x55ddb161b080 2024-04-02T14:35:35.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.239+0000 7f8eccbbb740 10 client.25646 fetch_fsmap learned FSMap version 27 2024-04-02T14:35:35.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.239+0000 7f8eccbbb740 1 -- 172.21.15.90:0/3837115152 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55ddb1643c90 con 0x55ddb161b080 2024-04-02T14:35:35.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.239+0000 7f8eb0c00640 1 -- 172.21.15.90:0/3837115152 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8ebc004030 con 0x55ddb161b080 2024-04-02T14:35:35.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.240+0000 7f8eb0c00640 1 -- 172.21.15.90:0/3837115152 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f8eac00f040 con 0x55ddb161b080 2024-04-02T14:35:35.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.240+0000 7f8eccbbb740 10 client.25646 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:35:35.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.240+0000 7f8eccbbb740 10 client.25646 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2024-04-02T14:35:35.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.240+0000 7f8eccbbb740 20 client.25646 populate_metadata read hostname 'smithi090' 2024-04-02T14:35:35.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.240+0000 7f8eccbbb740 1 --2- 172.21.15.90:0/3837115152 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x55ddb1645160 0x55ddb1647570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:35.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.240+0000 7f8eccbbb740 4 client.25646 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:35:35.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.240+0000 7f8eccbbb740 1 -- 172.21.15.90:0/3837115152 --> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ddb1536040 con 0x55ddb1645160 2024-04-02T14:35:35.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.241+0000 7f8ec2a00640 1 --2- 172.21.15.90:0/3837115152 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x55ddb1645160 0x55ddb1647570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:35.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.243+0000 7f8eb0c00640 10 client.25646 ms_handle_connect on v2:172.21.15.90:6838/1498915870 2024-04-02T14:35:35.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.243+0000 7f8ec2a00640 1 --2- 172.21.15.90:0/3837115152 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x55ddb1645160 0x55ddb1647570 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=38da68d032c910e server_cookie=f1546a7774e5b6b9 in_seq=0 out_seq=0 2024-04-02T14:35:35.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.244+0000 7f8eb0c00640 1 -- 172.21.15.90:0/3837115152 <== mds.1 v2:172.21.15.90:6838/1498915870 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ddb1536040 con 0x55ddb1645160 2024-04-02T14:35:35.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.244+0000 7f8eb0c00640 10 client.25646 handle_command_reply: tid=0 2024-04-02T14:35:35.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.244+0000 7f8eccbbb740 1 client.25646 shutdown 2024-04-02T14:35:35.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.245+0000 7f8eccbbb740 1 -- 172.21.15.90:0/3837115152 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ddb161b080 msgr2=0x55ddb16185e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:35:35.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.245+0000 7f8eccbbb740 1 --2- 172.21.15.90:0/3837115152 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ddb161b080 0x55ddb16185e0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f8ec4100140 tx=0x7f8eac004730 comp rx=0 tx=0).stop 2024-04-02T14:35:35.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.245+0000 7f8eccbbb740 1 -- 172.21.15.90:0/3837115152 shutdown_connections 2024-04-02T14:35:35.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.245+0000 7f8eccbbb740 1 --2- 172.21.15.90:0/3837115152 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x55ddb1645160 0x55ddb1647570 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:35.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.245+0000 7f8eccbbb740 1 --2- 172.21.15.90:0/3837115152 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ddb1618b20 0x55ddb16408f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:35.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.245+0000 7f8eccbbb740 1 --2- 172.21.15.90:0/3837115152 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ddb161b080 0x55ddb16185e0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:35.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.245+0000 7f8eccbbb740 1 --2- 172.21.15.90:0/3837115152 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ddb16129f0 0x55ddb16180a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:35.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.245+0000 7f8eccbbb740 1 -- 172.21.15.90:0/3837115152 >> 172.21.15.90:0/3837115152 conn(0x55ddb15d8a50 msgr2=0x55ddb160e4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:35:35.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.245+0000 7f8eb0c00640 0 client.25646 ms_handle_reset on v2:172.21.15.90:6838/1498915870 2024-04-02T14:35:35.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.246+0000 7f8eccbbb740 1 -- 172.21.15.90:0/3837115152 shutdown_connections 2024-04-02T14:35:35.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.246+0000 7f8eccbbb740 1 -- 172.21.15.90:0/3837115152 wait complete. 2024-04-02T14:35:35.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.247+0000 7f8eccbbb740 20 client.25646 trim_cache size 0 max 16384 2024-04-02T14:35:35.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.277+0000 7f8eccbbb740 1 Processor -- start 2024-04-02T14:35:35.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.278+0000 7f8eccbbb740 1 -- start start 2024-04-02T14:35:35.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.278+0000 7f8eccbbb740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ddb15d8a50 0x55ddb160e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:35.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.278+0000 7f8eccbbb740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ddb16129f0 0x55ddb164ce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:35.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.278+0000 7f8eccbbb740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ddb1618b20 0x55ddb164d380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:35.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.278+0000 7f8eccbbb740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f8ec406f780 con 0x55ddb15d8a50 2024-04-02T14:35:35.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.278+0000 7f8eccbbb740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f8ec40712c0 con 0x55ddb1618b20 2024-04-02T14:35:35.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.278+0000 7f8eccbbb740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f8ec406fe50 con 0x55ddb16129f0 2024-04-02T14:35:35.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.279+0000 7f8ebbe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ddb16129f0 0x55ddb164ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:35.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.279+0000 7f8ec2a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ddb15d8a50 0x55ddb160e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:35.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.279+0000 7f8ec3400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ddb1618b20 0x55ddb164d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:35.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.279+0000 7f8ec2a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ddb15d8a50 0x55ddb160e2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:57590/0 (socket says 172.21.15.90:57590) 2024-04-02T14:35:35.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.279+0000 7f8ec2a00640 1 -- 172.21.15.90:0/2798120042 learned_addr learned my addr 172.21.15.90:0/2798120042 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:35:35.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.279+0000 7f8ebbe00640 1 -- 172.21.15.90:0/2798120042 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ddb1618b20 msgr2=0x55ddb164d380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:35:35.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.279+0000 7f8ebbe00640 1 --2- 172.21.15.90:0/2798120042 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ddb1618b20 0x55ddb164d380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:35.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.279+0000 7f8ebbe00640 1 -- 172.21.15.90:0/2798120042 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ddb15d8a50 msgr2=0x55ddb160e2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:35:35.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.279+0000 7f8ebbe00640 1 --2- 172.21.15.90:0/2798120042 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ddb15d8a50 0x55ddb160e2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:35.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.279+0000 7f8ebbe00640 1 -- 172.21.15.90:0/2798120042 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ddb1610d40 con 0x55ddb16129f0 2024-04-02T14:35:35.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.280+0000 7f8ebbe00640 1 --2- 172.21.15.90:0/2798120042 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ddb16129f0 0x55ddb164ce40 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x55ddb1619130 tx=0x7f8eac009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b9e941cd9a1c3b70 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:35:35.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.281+0000 7f8eb3400640 1 -- 172.21.15.90:0/2798120042 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8eac037070 con 0x55ddb16129f0 2024-04-02T14:35:35.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.281+0000 7f8eb3400640 1 -- 172.21.15.90:0/2798120042 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8eac0114f0 con 0x55ddb16129f0 2024-04-02T14:35:35.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.281+0000 7f8eb3400640 1 -- 172.21.15.90:0/2798120042 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8eac023040 con 0x55ddb16129f0 2024-04-02T14:35:35.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.282+0000 7f8eccbbb740 1 -- 172.21.15.90:0/2798120042 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ddb16129f0 msgr2=0x55ddb164ce40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:35:35.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.282+0000 7f8eccbbb740 1 --2- 172.21.15.90:0/2798120042 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ddb16129f0 0x55ddb164ce40 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x55ddb1619130 tx=0x7f8eac009840 comp rx=0 tx=0).stop 2024-04-02T14:35:35.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.283+0000 7f8eccbbb740 1 -- 172.21.15.90:0/2798120042 shutdown_connections 2024-04-02T14:35:35.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.283+0000 7f8eccbbb740 1 --2- 172.21.15.90:0/2798120042 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ddb1618b20 0x55ddb164d380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:35.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.283+0000 7f8eccbbb740 1 --2- 172.21.15.90:0/2798120042 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ddb16129f0 0x55ddb164ce40 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:35.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.283+0000 7f8eccbbb740 1 --2- 172.21.15.90:0/2798120042 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ddb15d8a50 0x55ddb160e2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:35.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.283+0000 7f8eccbbb740 1 -- 172.21.15.90:0/2798120042 >> 172.21.15.90:0/2798120042 conn(0x55ddb1618740 msgr2=0x55ddb160dbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:35:35.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.284+0000 7f8eccbbb740 1 -- 172.21.15.90:0/2798120042 shutdown_connections 2024-04-02T14:35:35.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.284+0000 7f8eccbbb740 1 -- 172.21.15.90:0/2798120042 wait complete. 2024-04-02T14:35:35.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.285+0000 7f8eccbbb740 1 -- start start 2024-04-02T14:35:35.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.285+0000 7f8eccbbb740 1 Processor -- start 2024-04-02T14:35:35.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.286+0000 7f8eccbbb740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ddb15d8a50 0x55ddb163cd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:35.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.286+0000 7f8eccbbb740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ddb16129f0 0x55ddb1615610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:35.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.286+0000 7f8eccbbb740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ddb1618b20 0x55ddb1615bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:35.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.286+0000 7f8ec2a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ddb15d8a50 0x55ddb163cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:35.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.286+0000 7f8ec3400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ddb1618b20 0x55ddb1615bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:35.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.286+0000 7f8ebbe00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ddb16129f0 0x55ddb1615610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:35.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.286+0000 7f8eccbbb740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f8ec4070bf0 con 0x55ddb1618b20 2024-04-02T14:35:35.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.286+0000 7f8eccbbb740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f8ec4070520 con 0x55ddb16129f0 2024-04-02T14:35:35.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.286+0000 7f8eccbbb740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f8ec406d780 con 0x55ddb15d8a50 2024-04-02T14:35:35.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.286+0000 7f8ebbe00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ddb16129f0 0x55ddb1615610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:57982/0 (socket says 172.21.15.90:57982) 2024-04-02T14:35:35.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.286+0000 7f8ebbe00640 1 -- 172.21.15.90:0/1688147107 learned_addr learned my addr 172.21.15.90:0/1688147107 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:35:35.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.286+0000 7f8eccbbb740 10 client.0 _ll_register_callbacks cb 0x7f8eac010b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:35:35.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.287+0000 7f8ec2a00640 1 --2- 172.21.15.90:0/1688147107 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ddb15d8a50 0x55ddb163cd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:41118/0 (socket says 172.21.15.90:41118) 2024-04-02T14:35:35.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.287+0000 7f8ec3400640 1 -- 172.21.15.90:0/1688147107 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ddb15d8a50 msgr2=0x55ddb163cd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:35:35.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.287+0000 7f8ec3400640 1 --2- 172.21.15.90:0/1688147107 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ddb15d8a50 0x55ddb163cd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:35.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.287+0000 7f8ec3400640 1 -- 172.21.15.90:0/1688147107 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ddb16129f0 msgr2=0x55ddb1615610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:35:35.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.287+0000 7f8ec3400640 1 --2- 172.21.15.90:0/1688147107 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ddb16129f0 0x55ddb1615610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:35.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.287+0000 7f8ec3400640 1 -- 172.21.15.90:0/1688147107 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ddb161be70 con 0x55ddb1618b20 2024-04-02T14:35:35.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.287+0000 7f8ec2a00640 1 --2- 172.21.15.90:0/1688147107 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ddb15d8a50 0x55ddb163cd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:35:35.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.288+0000 7f8ec3400640 1 --2- 172.21.15.90:0/1688147107 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ddb1618b20 0x55ddb1615bb0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f8ebc006010 tx=0x7f8ebc004ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8410c9b944164d7c server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:35:35.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.288+0000 7f8eb0c00640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-02T14:35:35.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.288+0000 7f8eb0c00640 1 -- 172.21.15.90:0/1688147107 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ebc005250 con 0x55ddb1618b20 2024-04-02T14:35:35.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.288+0000 7f8eccbbb740 1 -- 172.21.15.90:0/1688147107 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8eac0114f0 con 0x55ddb1618b20 2024-04-02T14:35:35.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.288+0000 7f8eb0c00640 1 -- 172.21.15.90:0/1688147107 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f8ebc0053f0 con 0x55ddb1618b20 2024-04-02T14:35:35.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.289+0000 7f8eb0c00640 1 -- 172.21.15.90:0/1688147107 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f8ebc0055f0 con 0x55ddb1618b20 2024-04-02T14:35:35.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.289+0000 7f8eccbbb740 10 client.15522 fetch_fsmap learned FSMap version 27 2024-04-02T14:35:35.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.289+0000 7f8eccbbb740 1 -- 172.21.15.90:0/1688147107 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55ddb16aba50 con 0x55ddb1618b20 2024-04-02T14:35:35.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.289+0000 7f8eb0c00640 1 -- 172.21.15.90:0/1688147107 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8eac0114f0 con 0x55ddb1618b20 2024-04-02T14:35:35.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.289+0000 7f8eb0c00640 1 -- 172.21.15.90:0/1688147107 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f8ebc027460 con 0x55ddb1618b20 2024-04-02T14:35:35.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.289+0000 7f8eccbbb740 10 client.15522 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:35:35.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.289+0000 7f8eccbbb740 10 client.15522 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2024-04-02T14:35:35.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.289+0000 7f8eccbbb740 20 client.15522 populate_metadata read hostname 'smithi090' 2024-04-02T14:35:35.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.290+0000 7f8eccbbb740 1 --2- 172.21.15.90:0/1688147107 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x55ddb16abf80 0x55ddb16ae3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:35.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.290+0000 7f8eccbbb740 4 client.15522 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-02T14:35:35.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.290+0000 7f8eccbbb740 1 -- 172.21.15.90:0/1688147107 --> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55ddb1583c30 con 0x55ddb16abf80 2024-04-02T14:35:35.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.290+0000 7f8ec2a00640 1 --2- 172.21.15.90:0/1688147107 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x55ddb16abf80 0x55ddb16ae3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:35.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.291+0000 7f8ec2a00640 1 --2- 172.21.15.90:0/1688147107 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x55ddb16abf80 0x55ddb16ae3a0 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=896cab1254f74a0 server_cookie=3dcedc653894688a in_seq=0 out_seq=0 2024-04-02T14:35:35.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.291+0000 7f8eb0c00640 10 client.15522 ms_handle_connect on v2:172.21.15.90:6838/1498915870 2024-04-02T14:35:35.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.291+0000 7f8eb0c00640 1 -- 172.21.15.90:0/1688147107 <== mds.1 v2:172.21.15.90:6838/1498915870 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55ddb1583c30 con 0x55ddb16abf80 2024-04-02T14:35:35.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.291+0000 7f8eb0c00640 10 client.15522 handle_command_reply: tid=0 2024-04-02T14:35:35.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.291+0000 7f8eccbbb740 1 client.15522 shutdown 2024-04-02T14:35:35.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.292+0000 7f8eccbbb740 1 -- 172.21.15.90:0/1688147107 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ddb1618b20 msgr2=0x55ddb1615bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:35:35.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.292+0000 7f8eccbbb740 1 --2- 172.21.15.90:0/1688147107 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ddb1618b20 0x55ddb1615bb0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f8ebc006010 tx=0x7f8ebc004ca0 comp rx=0 tx=0).stop 2024-04-02T14:35:35.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.292+0000 7f8eccbbb740 1 -- 172.21.15.90:0/1688147107 shutdown_connections 2024-04-02T14:35:35.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.292+0000 7f8eccbbb740 1 --2- 172.21.15.90:0/1688147107 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x55ddb16abf80 0x55ddb16ae3a0 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:35.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.292+0000 7f8eccbbb740 1 --2- 172.21.15.90:0/1688147107 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ddb1618b20 0x55ddb1615bb0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:35.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.292+0000 7f8eccbbb740 1 --2- 172.21.15.90:0/1688147107 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ddb16129f0 0x55ddb1615610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:35.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.292+0000 7f8eccbbb740 1 --2- 172.21.15.90:0/1688147107 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ddb15d8a50 0x55ddb163cd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:35.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.292+0000 7f8eccbbb740 1 -- 172.21.15.90:0/1688147107 >> 172.21.15.90:0/1688147107 conn(0x55ddb1618740 msgr2=0x55ddb160e3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:35:35.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.292+0000 7f8eb0c00640 0 client.15522 ms_handle_reset on v2:172.21.15.90:6838/1498915870 2024-04-02T14:35:35.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.292+0000 7f8eccbbb740 1 -- 172.21.15.90:0/1688147107 shutdown_connections 2024-04-02T14:35:35.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.293+0000 7f8eccbbb740 1 -- 172.21.15.90:0/1688147107 wait complete. 2024-04-02T14:35:35.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.293+0000 7f8eccbbb740 20 client.15522 trim_cache size 0 max 16384 2024-04-02T14:35:35.296 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-02T14:35:35.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.294+0000 7f8ec9e00640 1 -- 172.21.15.90:0/2474140794 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f8ea80a3dd0 msgr2=0x7f8ea80a6290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:35:35.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.294+0000 7f8ec9e00640 1 --2- 172.21.15.90:0/2474140794 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f8ea80a3dd0 0x7f8ea80a6290 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f8eac004700 tx=0x7f8eac005850 comp rx=0 tx=0).stop 2024-04-02T14:35:35.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.294+0000 7f8ec9e00640 1 -- 172.21.15.90:0/2474140794 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8ec41009b0 msgr2=0x7f8ec4196170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:35:35.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.294+0000 7f8ec9e00640 1 --2- 172.21.15.90:0/2474140794 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8ec41009b0 0x7f8ec4196170 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f8eb4000c00 tx=0x7f8eb40042d0 comp rx=0 tx=0).stop 2024-04-02T14:35:35.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.294+0000 7f8ec9e00640 1 -- 172.21.15.90:0/2474140794 shutdown_connections 2024-04-02T14:35:35.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.294+0000 7f8ec9e00640 1 --2- 172.21.15.90:0/2474140794 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f8ea80a3dd0 0x7f8ea80a6290 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:35.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.294+0000 7f8ec9e00640 1 --2- 172.21.15.90:0/2474140794 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8ec41069c0 0x7f8ec419aa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:35.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.294+0000 7f8ec9e00640 1 --2- 172.21.15.90:0/2474140794 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8ec4101370 0x7f8ec41966b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:35.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.294+0000 7f8ec9e00640 1 --2- 172.21.15.90:0/2474140794 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8ec41009b0 0x7f8ec4196170 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:35.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.294+0000 7f8ec9e00640 1 -- 172.21.15.90:0/2474140794 >> 172.21.15.90:0/2474140794 conn(0x7f8ec4076d80 msgr2=0x7f8ec40fe2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:35:35.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.294+0000 7f8ec9e00640 1 -- 172.21.15.90:0/2474140794 shutdown_connections 2024-04-02T14:35:35.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.295+0000 7f8ec9e00640 1 -- 172.21.15.90:0/2474140794 wait complete. 2024-04-02T14:35:35.327 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-02T14:35:35.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.614+0000 7fc043000640 1 -- 172.21.15.90:0/1650504938 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc03c1006f0 msgr2=0x7fc03c100ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:35:35.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.614+0000 7fc043000640 1 --2- 172.21.15.90:0/1650504938 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc03c1006f0 0x7fc03c100ad0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fc02c009900 tx=0x7fc02c02bc40 comp rx=0 tx=0).stop 2024-04-02T14:35:35.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.615+0000 7fc043000640 1 -- 172.21.15.90:0/1650504938 shutdown_connections 2024-04-02T14:35:35.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.615+0000 7fc043000640 1 --2- 172.21.15.90:0/1650504938 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc03c10e2a0 0x7fc03c110690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:35.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.615+0000 7fc043000640 1 --2- 172.21.15.90:0/1650504938 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc03c1010a0 0x7fc03c10dd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:35.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.615+0000 7fc043000640 1 --2- 172.21.15.90:0/1650504938 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc03c1006f0 0x7fc03c100ad0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:35.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.615+0000 7fc043000640 1 -- 172.21.15.90:0/1650504938 >> 172.21.15.90:0/1650504938 conn(0x7fc03c0fc350 msgr2=0x7fc03c0fe770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:35:35.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.615+0000 7fc043000640 1 -- 172.21.15.90:0/1650504938 shutdown_connections 2024-04-02T14:35:35.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.615+0000 7fc043000640 1 -- 172.21.15.90:0/1650504938 wait complete. 2024-04-02T14:35:35.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.617+0000 7fc043000640 1 Processor -- start 2024-04-02T14:35:35.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.618+0000 7fc043000640 1 -- start start 2024-04-02T14:35:35.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.618+0000 7fc043000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc03c1006f0 0x7fc03c073220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:35.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.619+0000 7fc043000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc03c1010a0 0x7fc03c073760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:35.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.619+0000 7fc041c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc03c1006f0 0x7fc03c073220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:35.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.619+0000 7fc041c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc03c1006f0 0x7fc03c073220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:41144/0 (socket says 172.21.15.90:41144) 2024-04-02T14:35:35.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.619+0000 7fc043000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc03c10e2a0 0x7fc03c06c4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:35.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.619+0000 7fc041c00640 1 -- 172.21.15.90:0/274670704 learned_addr learned my addr 172.21.15.90:0/274670704 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:35:35.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.620+0000 7fc043000640 1 -- 172.21.15.90:0/274670704 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fc03c113f70 con 0x7fc03c1010a0 2024-04-02T14:35:35.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.620+0000 7fc041200640 1 --2- 172.21.15.90:0/274670704 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc03c1010a0 0x7fc03c073760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:35.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.620+0000 7fc043000640 1 -- 172.21.15.90:0/274670704 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fc03c113df0 con 0x7fc03c10e2a0 2024-04-02T14:35:35.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.620+0000 7fc043000640 1 -- 172.21.15.90:0/274670704 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fc03c1140f0 con 0x7fc03c1006f0 2024-04-02T14:35:35.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.620+0000 7fc042600640 1 --2- 172.21.15.90:0/274670704 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc03c10e2a0 0x7fc03c06c4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:35.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.620+0000 7fc041c00640 1 -- 172.21.15.90:0/274670704 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc03c10e2a0 msgr2=0x7fc03c06c4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:35:35.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.621+0000 7fc041c00640 1 --2- 172.21.15.90:0/274670704 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc03c10e2a0 0x7fc03c06c4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:35.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.621+0000 7fc041c00640 1 -- 172.21.15.90:0/274670704 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc03c1010a0 msgr2=0x7fc03c073760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:35:35.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.621+0000 7fc041c00640 1 --2- 172.21.15.90:0/274670704 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc03c1010a0 0x7fc03c073760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:35.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.621+0000 7fc041c00640 1 -- 172.21.15.90:0/274670704 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc03c06cd00 con 0x7fc03c1006f0 2024-04-02T14:35:35.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.621+0000 7fc041200640 1 --2- 172.21.15.90:0/274670704 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc03c1010a0 0x7fc03c073760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:35:35.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.621+0000 7fc041c00640 1 --2- 172.21.15.90:0/274670704 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc03c1006f0 0x7fc03c073220 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fc02c02efd0 tx=0x7fc02c0045e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:35:35.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.622+0000 7fc03a000640 1 -- 172.21.15.90:0/274670704 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc02c041070 con 0x7fc03c1006f0 2024-04-02T14:35:35.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.622+0000 7fc043000640 1 -- 172.21.15.90:0/274670704 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc03c06cf90 con 0x7fc03c1006f0 2024-04-02T14:35:35.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.622+0000 7fc043000640 1 -- 172.21.15.90:0/274670704 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc03c070f20 con 0x7fc03c1006f0 2024-04-02T14:35:35.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.622+0000 7fc03a000640 1 -- 172.21.15.90:0/274670704 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc02c0054e0 con 0x7fc03c1006f0 2024-04-02T14:35:35.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.622+0000 7fc03a000640 1 -- 172.21.15.90:0/274670704 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc02c034440 con 0x7fc03c1006f0 2024-04-02T14:35:35.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.626+0000 7fc047215740 1 Processor -- start 2024-04-02T14:35:35.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.627+0000 7fc03a000640 1 -- 172.21.15.90:0/274670704 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7fc02c044070 con 0x7fc03c1006f0 2024-04-02T14:35:35.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.627+0000 7fc047215740 1 -- start start 2024-04-02T14:35:35.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.628+0000 7fc047215740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fe217803b0 0x55fe2177f100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:35.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.629+0000 7fc047215740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fe2177f6d0 0x55fe2177fb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:35.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.629+0000 7fc042600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fe217803b0 0x55fe2177f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:35.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.629+0000 7fc042600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fe217803b0 0x55fe2177f100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:57628/0 (socket says 172.21.15.90:57628) 2024-04-02T14:35:35.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.629+0000 7fc03a000640 1 --2- 172.21.15.90:0/274670704 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fc0240a3fd0 0x7fc0240a6490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:35.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.629+0000 7fc047215740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fe21788a40 0x55fe2178ae30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:35.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.629+0000 7fc047215740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55fe216b7400 con 0x55fe217803b0 2024-04-02T14:35:35.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.629+0000 7fc047215740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55fe2169bf10 con 0x55fe21788a40 2024-04-02T14:35:35.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.629+0000 7fc047215740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55fe216db6f0 con 0x55fe2177f6d0 2024-04-02T14:35:35.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.629+0000 7fc042600640 1 -- 172.21.15.90:0/963116633 learned_addr learned my addr 172.21.15.90:0/963116633 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:35:35.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.629+0000 7fc03a000640 1 -- 172.21.15.90:0/274670704 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7fc02c0f2430 con 0x7fc03c1006f0 2024-04-02T14:35:35.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.630+0000 7fc042600640 1 -- 172.21.15.90:0/963116633 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fe2177f6d0 msgr2=0x55fe2177fb50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-02T14:35:35.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.630+0000 7fc042600640 1 --2- 172.21.15.90:0/963116633 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fe2177f6d0 0x55fe2177fb50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:35.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.630+0000 7fc042600640 1 -- 172.21.15.90:0/963116633 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fe21788a40 msgr2=0x55fe2178ae30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:35:35.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.630+0000 7fc042600640 1 --2- 172.21.15.90:0/963116633 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fe21788a40 0x55fe2178ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:35.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.630+0000 7fc042600640 1 -- 172.21.15.90:0/963116633 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fe213ef3f0 con 0x55fe217803b0 2024-04-02T14:35:35.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.630+0000 7fc041200640 1 --2- 172.21.15.90:0/274670704 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fc0240a3fd0 0x7fc0240a6490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:35.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.630+0000 7fc042600640 1 --2- 172.21.15.90:0/963116633 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fe217803b0 0x55fe2177f100 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fc034009f30 tx=0x7fc034010800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a737e2eaef4891a5 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:35:35.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.630+0000 7fc02aa00640 1 -- 172.21.15.90:0/963116633 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc034016070 con 0x55fe217803b0 2024-04-02T14:35:35.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.631+0000 7fc02aa00640 1 -- 172.21.15.90:0/963116633 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc034004d20 con 0x55fe217803b0 2024-04-02T14:35:35.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.631+0000 7fc02aa00640 1 -- 172.21.15.90:0/963116633 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0340127b0 con 0x55fe217803b0 2024-04-02T14:35:35.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.631+0000 7fc041200640 1 --2- 172.21.15.90:0/274670704 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fc0240a3fd0 0x7fc0240a6490 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fc03c06da60 tx=0x7fc03000a5c0 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:35:35.634 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.632+0000 7fc047215740 1 -- 172.21.15.90:0/963116633 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fe217803b0 msgr2=0x55fe2177f100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:35:35.634 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.632+0000 7fc047215740 1 --2- 172.21.15.90:0/963116633 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fe217803b0 0x55fe2177f100 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fc034009f30 tx=0x7fc034010800 comp rx=0 tx=0).stop 2024-04-02T14:35:35.634 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.632+0000 7fc047215740 1 -- 172.21.15.90:0/963116633 shutdown_connections 2024-04-02T14:35:35.634 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.632+0000 7fc047215740 1 --2- 172.21.15.90:0/963116633 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fe21788a40 0x55fe2178ae30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:35.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.632+0000 7fc047215740 1 --2- 172.21.15.90:0/963116633 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fe2177f6d0 0x55fe2177fb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:35.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.632+0000 7fc047215740 1 --2- 172.21.15.90:0/963116633 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fe217803b0 0x55fe2177f100 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:35.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.632+0000 7fc047215740 1 -- 172.21.15.90:0/963116633 >> 172.21.15.90:0/963116633 conn(0x55fe2173da50 msgr2=0x55fe2177a120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:35:35.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.633+0000 7fc047215740 1 -- 172.21.15.90:0/963116633 shutdown_connections 2024-04-02T14:35:35.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.633+0000 7fc047215740 1 -- 172.21.15.90:0/963116633 wait complete. 2024-04-02T14:35:35.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.633+0000 7fc047215740 1 -- start start 2024-04-02T14:35:35.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.633+0000 7fc047215740 1 Processor -- start 2024-04-02T14:35:35.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.634+0000 7fc047215740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fe2177f6d0 0x55fe21785af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:35.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.634+0000 7fc047215740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fe217803b0 0x55fe21786030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:35.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.634+0000 7fc047215740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fe21788a40 0x55fe21786570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:35.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.634+0000 7fc047215740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55fe216d76a0 con 0x55fe2177f6d0 2024-04-02T14:35:35.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.634+0000 7fc047215740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55fe21683980 con 0x55fe21788a40 2024-04-02T14:35:35.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.634+0000 7fc047215740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55fe21696cb0 con 0x55fe217803b0 2024-04-02T14:35:35.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.635+0000 7fc047215740 10 client.0 _ll_register_callbacks cb 0x55fe216966a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:35:35.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.635+0000 7fc042600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fe21788a40 0x55fe21786570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:35.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.635+0000 7fc042600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fe21788a40 0x55fe21786570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:58018/0 (socket says 172.21.15.90:58018) 2024-04-02T14:35:35.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.635+0000 7fc042600640 1 -- 172.21.15.90:0/2696656721 learned_addr learned my addr 172.21.15.90:0/2696656721 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:35:35.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.635+0000 7fc041200640 1 --2- 172.21.15.90:0/2696656721 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fe217803b0 0x55fe21786030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:35.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.636+0000 7fc041200640 1 -- 172.21.15.90:0/2696656721 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fe21788a40 msgr2=0x55fe21786570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:35:35.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.636+0000 7fc041200640 1 --2- 172.21.15.90:0/2696656721 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fe21788a40 0x55fe21786570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:35.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.636+0000 7fc041200640 1 -- 172.21.15.90:0/2696656721 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fe2177f6d0 msgr2=0x55fe21785af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:35:35.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.636+0000 7fc041c00640 1 --2- 172.21.15.90:0/2696656721 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fe2177f6d0 0x55fe21785af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:35.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.636+0000 7fc041200640 1 --2- 172.21.15.90:0/2696656721 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fe2177f6d0 0x55fe21785af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:35.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.636+0000 7fc041200640 1 -- 172.21.15.90:0/2696656721 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fe214b97d0 con 0x55fe217803b0 2024-04-02T14:35:35.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.637+0000 7fc041c00640 1 --2- 172.21.15.90:0/2696656721 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fe2177f6d0 0x55fe21785af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:35:35.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.637+0000 7fc041200640 1 --2- 172.21.15.90:0/2696656721 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fe217803b0 0x55fe21786030 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fc03c06daf0 tx=0x7fc0300046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bb7fbfdb921dd71d server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:35:35.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.637+0000 7fc01f400640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:35:35.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.638+0000 7fc01f400640 1 -- 172.21.15.90:0/2696656721 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc030021070 con 0x55fe217803b0 2024-04-02T14:35:35.640 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.638+0000 7fc01f400640 1 -- 172.21.15.90:0/2696656721 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7fc03000fcd0 con 0x55fe217803b0 2024-04-02T14:35:35.640 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.638+0000 7fc047215740 1 -- 172.21.15.90:0/2696656721 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc034004d20 con 0x55fe217803b0 2024-04-02T14:35:35.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.639+0000 7fc01f400640 1 -- 172.21.15.90:0/2696656721 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fc03002b050 con 0x55fe217803b0 2024-04-02T14:35:35.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.639+0000 7fc047215740 10 client.25618 fetch_fsmap learned FSMap version 27 2024-04-02T14:35:35.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.639+0000 7fc047215740 1 -- 172.21.15.90:0/2696656721 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55fe21797a90 con 0x55fe217803b0 2024-04-02T14:35:35.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.639+0000 7fc01f400640 1 -- 172.21.15.90:0/2696656721 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc034004d20 con 0x55fe217803b0 2024-04-02T14:35:35.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.640+0000 7fc01f400640 1 -- 172.21.15.90:0/2696656721 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fc030026440 con 0x55fe217803b0 2024-04-02T14:35:35.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.640+0000 7fc047215740 10 client.25618 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:35:35.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.640+0000 7fc047215740 10 client.25618 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:35:35.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.640+0000 7fc047215740 20 client.25618 populate_metadata read hostname 'smithi090' 2024-04-02T14:35:35.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.640+0000 7fc047215740 1 --2- 172.21.15.90:0/2696656721 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55fe21798f50 0x55fe2179b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:35.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.640+0000 7fc047215740 4 client.25618 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:35:35.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.640+0000 7fc047215740 1 -- 172.21.15.90:0/2696656721 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fe2169b040 con 0x55fe21798f50 2024-04-02T14:35:35.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.640+0000 7fc041c00640 1 --2- 172.21.15.90:0/2696656721 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55fe21798f50 0x55fe2179b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:35.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.641+0000 7fc041c00640 1 --2- 172.21.15.90:0/2696656721 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55fe21798f50 0x55fe2179b3a0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=924c7847eba9fe6c server_cookie=fdb9fa6482e7a4f2 in_seq=0 out_seq=0 2024-04-02T14:35:35.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.641+0000 7fc01f400640 10 client.25618 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:35:35.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.642+0000 7fc01f400640 1 -- 172.21.15.90:0/2696656721 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55fe2169b040 con 0x55fe21798f50 2024-04-02T14:35:35.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.642+0000 7fc01f400640 10 client.25618 handle_command_reply: tid=0 2024-04-02T14:35:35.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.642+0000 7fc047215740 1 client.25618 shutdown 2024-04-02T14:35:35.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.642+0000 7fc047215740 1 -- 172.21.15.90:0/2696656721 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fe217803b0 msgr2=0x55fe21786030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:35:35.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.642+0000 7fc047215740 1 --2- 172.21.15.90:0/2696656721 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fe217803b0 0x55fe21786030 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fc03c06daf0 tx=0x7fc0300046c0 comp rx=0 tx=0).stop 2024-04-02T14:35:35.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.642+0000 7fc047215740 1 -- 172.21.15.90:0/2696656721 shutdown_connections 2024-04-02T14:35:35.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.642+0000 7fc047215740 1 --2- 172.21.15.90:0/2696656721 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55fe21798f50 0x55fe2179b3a0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:35.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.642+0000 7fc047215740 1 --2- 172.21.15.90:0/2696656721 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fe21788a40 0x55fe21786570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:35.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.643+0000 7fc047215740 1 --2- 172.21.15.90:0/2696656721 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fe217803b0 0x55fe21786030 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:35.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.643+0000 7fc047215740 1 --2- 172.21.15.90:0/2696656721 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fe2177f6d0 0x55fe21785af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:35.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.643+0000 7fc047215740 1 -- 172.21.15.90:0/2696656721 >> 172.21.15.90:0/2696656721 conn(0x55fe2173da50 msgr2=0x55fe2177b5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:35:35.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.643+0000 7fc01f400640 0 client.25618 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:35:35.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.643+0000 7fc047215740 1 -- 172.21.15.90:0/2696656721 shutdown_connections 2024-04-02T14:35:35.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.643+0000 7fc047215740 1 -- 172.21.15.90:0/2696656721 wait complete. 2024-04-02T14:35:35.646 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.643+0000 7fc047215740 20 client.25618 trim_cache size 0 max 16384 2024-04-02T14:35:35.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.671+0000 7fc047215740 1 Processor -- start 2024-04-02T14:35:35.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.671+0000 7fc047215740 1 -- start start 2024-04-02T14:35:35.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.671+0000 7fc047215740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fe2177f6d0 0x55fe21794510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:35.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.671+0000 7fc047215740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fe217803b0 0x55fe2179af60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:35.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.672+0000 7fc047215740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fe21788a40 0x55fe217a0d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:35.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.672+0000 7fc047215740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fc02000e7a0 con 0x55fe2177f6d0 2024-04-02T14:35:35.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.672+0000 7fc047215740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fc0200102e0 con 0x55fe21788a40 2024-04-02T14:35:35.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.672+0000 7fc047215740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fc02000ee70 con 0x55fe217803b0 2024-04-02T14:35:35.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.672+0000 7fc041c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fe2177f6d0 0x55fe21794510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:35.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.672+0000 7fc042600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fe21788a40 0x55fe217a0d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:35.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.672+0000 7fc042600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fe21788a40 0x55fe217a0d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:58032/0 (socket says 172.21.15.90:58032) 2024-04-02T14:35:35.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.672+0000 7fc042600640 1 -- 172.21.15.90:0/2119512451 learned_addr learned my addr 172.21.15.90:0/2119512451 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:35:35.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.672+0000 7fc041200640 1 --2- 172.21.15.90:0/2119512451 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fe217803b0 0x55fe2179af60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:35.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.672+0000 7fc042600640 1 -- 172.21.15.90:0/2119512451 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fe217803b0 msgr2=0x55fe2179af60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:35:35.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.672+0000 7fc042600640 1 --2- 172.21.15.90:0/2119512451 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fe217803b0 0x55fe2179af60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:35.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.672+0000 7fc042600640 1 -- 172.21.15.90:0/2119512451 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fe2177f6d0 msgr2=0x55fe21794510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:35:35.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.672+0000 7fc042600640 1 --2- 172.21.15.90:0/2119512451 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fe2177f6d0 0x55fe21794510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:35.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.672+0000 7fc042600640 1 -- 172.21.15.90:0/2119512451 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fe217808e0 con 0x55fe21788a40 2024-04-02T14:35:35.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.672+0000 7fc041c00640 1 --2- 172.21.15.90:0/2119512451 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fe2177f6d0 0x55fe21794510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:35:35.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.673+0000 7fc042600640 1 --2- 172.21.15.90:0/2119512451 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fe21788a40 0x55fe217a0d00 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fc034009fd0 tx=0x7fc034012450 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aabe83d7d88c60c5 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:35:35.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.673+0000 7fc02a000640 1 -- 172.21.15.90:0/2119512451 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc034028070 con 0x55fe21788a40 2024-04-02T14:35:35.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.673+0000 7fc02a000640 1 -- 172.21.15.90:0/2119512451 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc034004c60 con 0x55fe21788a40 2024-04-02T14:35:35.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.673+0000 7fc02a000640 1 -- 172.21.15.90:0/2119512451 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc034016040 con 0x55fe21788a40 2024-04-02T14:35:35.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.674+0000 7fc047215740 1 -- 172.21.15.90:0/2119512451 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fe21788a40 msgr2=0x55fe217a0d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:35:35.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.674+0000 7fc047215740 1 --2- 172.21.15.90:0/2119512451 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fe21788a40 0x55fe217a0d00 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fc034009fd0 tx=0x7fc034012450 comp rx=0 tx=0).stop 2024-04-02T14:35:35.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.675+0000 7fc047215740 1 -- 172.21.15.90:0/2119512451 shutdown_connections 2024-04-02T14:35:35.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.675+0000 7fc047215740 1 --2- 172.21.15.90:0/2119512451 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fe21788a40 0x55fe217a0d00 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:35.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.675+0000 7fc047215740 1 --2- 172.21.15.90:0/2119512451 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fe217803b0 0x55fe2179af60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:35.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.675+0000 7fc047215740 1 --2- 172.21.15.90:0/2119512451 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fe2177f6d0 0x55fe21794510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:35.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.675+0000 7fc047215740 1 -- 172.21.15.90:0/2119512451 >> 172.21.15.90:0/2119512451 conn(0x55fe2173da50 msgr2=0x55fe21779c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:35:35.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.675+0000 7fc047215740 1 -- 172.21.15.90:0/2119512451 shutdown_connections 2024-04-02T14:35:35.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.675+0000 7fc047215740 1 -- 172.21.15.90:0/2119512451 wait complete. 2024-04-02T14:35:35.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.675+0000 7fc047215740 1 -- start start 2024-04-02T14:35:35.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.676+0000 7fc047215740 1 Processor -- start 2024-04-02T14:35:35.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.676+0000 7fc047215740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fe2177f6d0 0x55fe21790dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:35.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.677+0000 7fc047215740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fe217803b0 0x55fe21782440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:35.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.677+0000 7fc047215740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fe21788a40 0x55fe21782980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:35.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.677+0000 7fc047215740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fc02000fc10 con 0x55fe2177f6d0 2024-04-02T14:35:35.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.677+0000 7fc047215740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fc02000f540 con 0x55fe21788a40 2024-04-02T14:35:35.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.677+0000 7fc047215740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fc02000c740 con 0x55fe217803b0 2024-04-02T14:35:35.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.677+0000 7fc047215740 10 client.0 _ll_register_callbacks cb 0x7fc030010660 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:35:35.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.677+0000 7fc042600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fe21788a40 0x55fe21782980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:35.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.677+0000 7fc042600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fe21788a40 0x55fe21782980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:58034/0 (socket says 172.21.15.90:58034) 2024-04-02T14:35:35.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.677+0000 7fc042600640 1 -- 172.21.15.90:0/263240769 learned_addr learned my addr 172.21.15.90:0/263240769 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:35:35.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.677+0000 7fc041200640 1 --2- 172.21.15.90:0/263240769 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fe217803b0 0x55fe21782440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:35.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.677+0000 7fc042600640 1 -- 172.21.15.90:0/263240769 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fe217803b0 msgr2=0x55fe21782440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:35:35.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.677+0000 7fc042600640 1 --2- 172.21.15.90:0/263240769 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fe217803b0 0x55fe21782440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:35.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.677+0000 7fc042600640 1 -- 172.21.15.90:0/263240769 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fe2177f6d0 msgr2=0x55fe21790dc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-02T14:35:35.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.677+0000 7fc041c00640 1 --2- 172.21.15.90:0/263240769 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fe2177f6d0 0x55fe21790dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:35.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.677+0000 7fc042600640 1 --2- 172.21.15.90:0/263240769 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fe2177f6d0 0x55fe21790dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:35.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.677+0000 7fc042600640 1 -- 172.21.15.90:0/263240769 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fe21783160 con 0x55fe21788a40 2024-04-02T14:35:35.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.678+0000 7fc042600640 1 --2- 172.21.15.90:0/263240769 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fe21788a40 0x55fe21782980 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fc034010d00 tx=0x7fc034005820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2b5f170bb2483b9c server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:35:35.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.678+0000 7fc01f400640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:35:35.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.678+0000 7fc01f400640 1 -- 172.21.15.90:0/263240769 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc034028070 con 0x55fe21788a40 2024-04-02T14:35:35.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.678+0000 7fc01f400640 1 -- 172.21.15.90:0/263240769 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7fc0340053c0 con 0x55fe21788a40 2024-04-02T14:35:35.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.678+0000 7fc047215740 1 -- 172.21.15.90:0/263240769 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc034004c60 con 0x55fe21788a40 2024-04-02T14:35:35.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.679+0000 7fc01f400640 1 -- 172.21.15.90:0/263240769 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fc034034050 con 0x55fe21788a40 2024-04-02T14:35:35.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.679+0000 7fc047215740 10 client.25667 fetch_fsmap learned FSMap version 27 2024-04-02T14:35:35.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.679+0000 7fc047215740 1 -- 172.21.15.90:0/263240769 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55fe217ff4c0 con 0x55fe21788a40 2024-04-02T14:35:35.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.679+0000 7fc01f400640 1 -- 172.21.15.90:0/263240769 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc034004c60 con 0x55fe21788a40 2024-04-02T14:35:35.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.681+0000 7fc01f400640 1 -- 172.21.15.90:0/263240769 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fc034022030 con 0x55fe21788a40 2024-04-02T14:35:35.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.681+0000 7fc047215740 10 client.25667 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:35:35.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.681+0000 7fc047215740 10 client.25667 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:35:35.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.681+0000 7fc047215740 20 client.25667 populate_metadata read hostname 'smithi090' 2024-04-02T14:35:35.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.681+0000 7fc047215740 1 --2- 172.21.15.90:0/263240769 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55fe217ff8e0 0x55fe21801cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:35.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.681+0000 7fc047215740 4 client.25667 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-02T14:35:35.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.681+0000 7fc047215740 1 -- 172.21.15.90:0/263240769 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55fe2179b730 con 0x55fe217ff8e0 2024-04-02T14:35:35.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.681+0000 7fc041c00640 1 --2- 172.21.15.90:0/263240769 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55fe217ff8e0 0x55fe21801cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:35.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.682+0000 7fc01f400640 10 client.25667 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:35:35.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.682+0000 7fc041c00640 1 --2- 172.21.15.90:0/263240769 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55fe217ff8e0 0x55fe21801cd0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e0287b1482b2c97e server_cookie=48620e2c999a2f35 in_seq=0 out_seq=0 2024-04-02T14:35:35.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.682+0000 7fc01f400640 1 -- 172.21.15.90:0/263240769 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55fe2179b730 con 0x55fe217ff8e0 2024-04-02T14:35:35.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.682+0000 7fc01f400640 10 client.25667 handle_command_reply: tid=0 2024-04-02T14:35:35.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.682+0000 7fc047215740 1 client.25667 shutdown 2024-04-02T14:35:35.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.683+0000 7fc047215740 1 -- 172.21.15.90:0/263240769 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fe21788a40 msgr2=0x55fe21782980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:35:35.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.683+0000 7fc047215740 1 --2- 172.21.15.90:0/263240769 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fe21788a40 0x55fe21782980 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fc034010d00 tx=0x7fc034005820 comp rx=0 tx=0).stop 2024-04-02T14:35:35.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.683+0000 7fc047215740 1 -- 172.21.15.90:0/263240769 shutdown_connections 2024-04-02T14:35:35.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.683+0000 7fc047215740 1 --2- 172.21.15.90:0/263240769 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55fe217ff8e0 0x55fe21801cd0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:35.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.683+0000 7fc047215740 1 --2- 172.21.15.90:0/263240769 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fe21788a40 0x55fe21782980 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:35.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.683+0000 7fc047215740 1 --2- 172.21.15.90:0/263240769 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fe217803b0 0x55fe21782440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:35.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.683+0000 7fc047215740 1 --2- 172.21.15.90:0/263240769 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fe2177f6d0 0x55fe21790dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:35.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.683+0000 7fc047215740 1 -- 172.21.15.90:0/263240769 >> 172.21.15.90:0/263240769 conn(0x55fe2173da50 msgr2=0x55fe21770260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:35:35.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.683+0000 7fc01f400640 0 client.25667 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:35:35.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.683+0000 7fc047215740 1 -- 172.21.15.90:0/263240769 shutdown_connections 2024-04-02T14:35:35.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.683+0000 7fc047215740 1 -- 172.21.15.90:0/263240769 wait complete. 2024-04-02T14:35:35.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.684+0000 7fc047215740 20 client.25667 trim_cache size 0 max 16384 2024-04-02T14:35:35.687 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-02T14:35:35.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.685+0000 7fc043000640 1 -- 172.21.15.90:0/274670704 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fc0240a3fd0 msgr2=0x7fc0240a6490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:35:35.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.685+0000 7fc043000640 1 --2- 172.21.15.90:0/274670704 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fc0240a3fd0 0x7fc0240a6490 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fc03c06da60 tx=0x7fc03000a5c0 comp rx=0 tx=0).stop 2024-04-02T14:35:35.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.685+0000 7fc043000640 1 -- 172.21.15.90:0/274670704 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc03c1006f0 msgr2=0x7fc03c073220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:35:35.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.685+0000 7fc043000640 1 --2- 172.21.15.90:0/274670704 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc03c1006f0 0x7fc03c073220 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fc02c02efd0 tx=0x7fc02c0045e0 comp rx=0 tx=0).stop 2024-04-02T14:35:35.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.685+0000 7fc043000640 1 -- 172.21.15.90:0/274670704 shutdown_connections 2024-04-02T14:35:35.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.685+0000 7fc043000640 1 --2- 172.21.15.90:0/274670704 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fc0240a3fd0 0x7fc0240a6490 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:35.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.685+0000 7fc043000640 1 --2- 172.21.15.90:0/274670704 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc03c10e2a0 0x7fc03c06c4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:35.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.685+0000 7fc043000640 1 --2- 172.21.15.90:0/274670704 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc03c1010a0 0x7fc03c073760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:35.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.685+0000 7fc043000640 1 --2- 172.21.15.90:0/274670704 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc03c1006f0 0x7fc03c073220 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:35.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.685+0000 7fc043000640 1 -- 172.21.15.90:0/274670704 >> 172.21.15.90:0/274670704 conn(0x7fc03c0fc350 msgr2=0x7fc03c0ff880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:35:35.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.685+0000 7fc043000640 1 -- 172.21.15.90:0/274670704 shutdown_connections 2024-04-02T14:35:35.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:35.685+0000 7fc043000640 1 -- 172.21.15.90:0/274670704 wait complete. 2024-04-02T14:35:36.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:35:35 smithi177 ceph-mon[36431]: pgmap v24: 97 pgs: 97 active+clean; 510 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 72 MiB/s wr, 113 op/s 2024-04-02T14:35:36.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:35:35 smithi090 ceph-mon[31674]: pgmap v24: 97 pgs: 97 active+clean; 510 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 72 MiB/s wr, 113 op/s 2024-04-02T14:35:36.360 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:35:35 smithi092 ceph-mon[36497]: pgmap v24: 97 pgs: 97 active+clean; 510 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 72 MiB/s wr, 113 op/s 2024-04-02T14:35:36.720 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-02T14:35:36.720 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T14:35:36.720 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-02T14:35:36.832 INFO:teuthology.orchestra.run.smithi090.stderr:progress: 5.0 s, 3335.6 tps, lat 28.826 ms stddev 30.146 2024-04-02T14:35:37.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.015+0000 7f85b2400640 1 -- 172.21.15.90:0/3710170866 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f85ac109b70 msgr2=0x7f85ac109f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:35:37.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.015+0000 7f85b2400640 1 --2- 172.21.15.90:0/3710170866 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f85ac109b70 0x7f85ac109f50 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f85a4009ca0 tx=0x7f85a402c100 comp rx=0 tx=0).stop 2024-04-02T14:35:37.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.015+0000 7f85b2400640 1 -- 172.21.15.90:0/3710170866 shutdown_connections 2024-04-02T14:35:37.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.015+0000 7f85b2400640 1 --2- 172.21.15.90:0/3710170866 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f85ac104490 0x7f85ac1109a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:37.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.015+0000 7f85b2400640 1 --2- 172.21.15.90:0/3710170866 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f85ac103ad0 0x7f85ac103f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:37.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.015+0000 7f85b2400640 1 --2- 172.21.15.90:0/3710170866 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f85ac109b70 0x7f85ac109f50 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:37.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.015+0000 7f85b2400640 1 -- 172.21.15.90:0/3710170866 >> 172.21.15.90:0/3710170866 conn(0x7f85ac0ff560 msgr2=0x7f85ac101980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:35:37.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.015+0000 7f85b2400640 1 -- 172.21.15.90:0/3710170866 shutdown_connections 2024-04-02T14:35:37.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.015+0000 7f85b2400640 1 -- 172.21.15.90:0/3710170866 wait complete. 2024-04-02T14:35:37.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.016+0000 7f85b2400640 1 Processor -- start 2024-04-02T14:35:37.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.016+0000 7f85b2400640 1 -- start start 2024-04-02T14:35:37.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.017+0000 7f85b2400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f85ac103ad0 0x7f85ac110780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:37.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.017+0000 7f85b2400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f85ac104490 0x7f85ac10d820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:37.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.017+0000 7f85b2400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f85ac109b70 0x7f85ac10dd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:37.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.017+0000 7f85b2400640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f85ac113ec0 con 0x7f85ac104490 2024-04-02T14:35:37.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.017+0000 7f85b2400640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f85ac113d40 con 0x7f85ac109b70 2024-04-02T14:35:37.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.017+0000 7f85b2400640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f85ac114040 con 0x7f85ac103ad0 2024-04-02T14:35:37.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.017+0000 7f85b1000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f85ac103ad0 0x7f85ac110780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:37.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.017+0000 7f85b1000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f85ac103ad0 0x7f85ac110780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:41200/0 (socket says 172.21.15.90:41200) 2024-04-02T14:35:37.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.017+0000 7f85b1000640 1 -- 172.21.15.90:0/2473526050 learned_addr learned my addr 172.21.15.90:0/2473526050 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:35:37.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.017+0000 7f85b1a00640 1 --2- 172.21.15.90:0/2473526050 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f85ac109b70 0x7f85ac10dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:37.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.018+0000 7f85b1000640 1 -- 172.21.15.90:0/2473526050 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f85ac109b70 msgr2=0x7f85ac10dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:35:37.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.018+0000 7f85b1000640 1 --2- 172.21.15.90:0/2473526050 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f85ac109b70 0x7f85ac10dd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:37.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.018+0000 7f85b1000640 1 -- 172.21.15.90:0/2473526050 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f85ac104490 msgr2=0x7f85ac10d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:35:37.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.018+0000 7f85abe00640 1 --2- 172.21.15.90:0/2473526050 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f85ac104490 0x7f85ac10d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:37.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.018+0000 7f85b1000640 1 --2- 172.21.15.90:0/2473526050 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f85ac104490 0x7f85ac10d820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:37.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.019+0000 7f85b1000640 1 -- 172.21.15.90:0/2473526050 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85ac10e5d0 con 0x7f85ac103ad0 2024-04-02T14:35:37.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.019+0000 7f85abe00640 1 --2- 172.21.15.90:0/2473526050 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f85ac104490 0x7f85ac10d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:35:37.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.019+0000 7f85b1000640 1 --2- 172.21.15.90:0/2473526050 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f85ac103ad0 0x7f85ac110780 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f85a402c710 tx=0x7f85a40346f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:35:37.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.020+0000 7f85a9600640 1 -- 172.21.15.90:0/2473526050 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85a4039050 con 0x7f85ac103ad0 2024-04-02T14:35:37.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.020+0000 7f85a9600640 1 -- 172.21.15.90:0/2473526050 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f85a4045070 con 0x7f85ac103ad0 2024-04-02T14:35:37.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.020+0000 7f85a9600640 1 -- 172.21.15.90:0/2473526050 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85a4040450 con 0x7f85ac103ad0 2024-04-02T14:35:37.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.020+0000 7f85b2400640 1 -- 172.21.15.90:0/2473526050 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f85ac1aabc0 con 0x7f85ac103ad0 2024-04-02T14:35:37.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.020+0000 7f85b2400640 1 -- 172.21.15.90:0/2473526050 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f85ac1ab090 con 0x7f85ac103ad0 2024-04-02T14:35:37.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.024+0000 7f85a9600640 1 -- 172.21.15.90:0/2473526050 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7f85a4004040 con 0x7f85ac103ad0 2024-04-02T14:35:37.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.025+0000 7f85b67cc740 1 Processor -- start 2024-04-02T14:35:37.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.026+0000 7f85b67cc740 1 -- start start 2024-04-02T14:35:37.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.026+0000 7f85a9600640 1 --2- 172.21.15.90:0/2473526050 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f85900a3f90 0x7f85900a6450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:37.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.027+0000 7f85a9600640 1 -- 172.21.15.90:0/2473526050 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f85a40f30b0 con 0x7f85ac103ad0 2024-04-02T14:35:37.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.028+0000 7f85b67cc740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a5a85b0220 0x55a5a85aef70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:37.032 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.028+0000 7f85b1a00640 1 --2- 172.21.15.90:0/2473526050 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f85900a3f90 0x7f85900a6450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:37.032 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.029+0000 7f85b67cc740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a5a85af540 0x55a5a85af9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:37.032 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.029+0000 7f85b67cc740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a5a85b88b0 0x55a5a85baca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:37.032 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.029+0000 7f85b67cc740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55a5a8513310 con 0x55a5a85af540 2024-04-02T14:35:37.032 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.029+0000 7f85b67cc740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55a5a84bb690 con 0x55a5a85b88b0 2024-04-02T14:35:37.032 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.029+0000 7f85b67cc740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55a5a850f2c0 con 0x55a5a85b0220 2024-04-02T14:35:37.032 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.029+0000 7f85b1000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a5a85b0220 0x55a5a85aef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:37.032 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.029+0000 7f85b1000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a5a85b0220 0x55a5a85aef70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:41204/0 (socket says 172.21.15.90:41204) 2024-04-02T14:35:37.032 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.029+0000 7f85b1000640 1 -- 172.21.15.90:0/759341853 learned_addr learned my addr 172.21.15.90:0/759341853 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:35:37.032 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.029+0000 7f85b1000640 1 -- 172.21.15.90:0/759341853 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a5a85b88b0 msgr2=0x55a5a85baca0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-02T14:35:37.032 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.029+0000 7f85b1000640 1 --2- 172.21.15.90:0/759341853 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a5a85b88b0 0x55a5a85baca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:37.032 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.029+0000 7f85b1000640 1 -- 172.21.15.90:0/759341853 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a5a85af540 msgr2=0x55a5a85af9c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-02T14:35:37.033 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.030+0000 7f85abe00640 1 --2- 172.21.15.90:0/759341853 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a5a85af540 0x55a5a85af9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:37.034 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.030+0000 7f85b1000640 1 --2- 172.21.15.90:0/759341853 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a5a85af540 0x55a5a85af9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:37.034 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.030+0000 7f85b1000640 1 -- 172.21.15.90:0/759341853 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a5a82ef7e0 con 0x55a5a85b0220 2024-04-02T14:35:37.034 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.030+0000 7f85b1000640 1 --2- 172.21.15.90:0/759341853 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a5a85b0220 0x55a5a85aef70 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f85a40becf0 tx=0x7f85a4032b40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ba6155192eb1723e server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:35:37.034 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.031+0000 7f85a2000640 1 -- 172.21.15.90:0/759341853 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85a40bf6f0 con 0x55a5a85b0220 2024-04-02T14:35:37.034 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.031+0000 7f85a2000640 1 -- 172.21.15.90:0/759341853 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f85a403a040 con 0x55a5a85b0220 2024-04-02T14:35:37.034 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.031+0000 7f85a2000640 1 -- 172.21.15.90:0/759341853 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85a40c77d0 con 0x55a5a85b0220 2024-04-02T14:35:37.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.032+0000 7f85b67cc740 1 -- 172.21.15.90:0/759341853 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a5a85b0220 msgr2=0x55a5a85aef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:35:37.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.032+0000 7f85b67cc740 1 --2- 172.21.15.90:0/759341853 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a5a85b0220 0x55a5a85aef70 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f85a40becf0 tx=0x7f85a4032b40 comp rx=0 tx=0).stop 2024-04-02T14:35:37.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.032+0000 7f85b1a00640 1 --2- 172.21.15.90:0/2473526050 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f85900a3f90 0x7f85900a6450 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f85ac110eb0 tx=0x7f859800a250 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:35:37.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.033+0000 7f85b67cc740 1 -- 172.21.15.90:0/759341853 shutdown_connections 2024-04-02T14:35:37.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.033+0000 7f85b67cc740 1 --2- 172.21.15.90:0/759341853 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a5a85b88b0 0x55a5a85baca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:37.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.033+0000 7f85b67cc740 1 --2- 172.21.15.90:0/759341853 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a5a85af540 0x55a5a85af9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:37.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.033+0000 7f85b67cc740 1 --2- 172.21.15.90:0/759341853 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a5a85b0220 0x55a5a85aef70 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:37.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.033+0000 7f85b67cc740 1 -- 172.21.15.90:0/759341853 >> 172.21.15.90:0/759341853 conn(0x55a5a8543270 msgr2=0x55a5a85a9f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:35:37.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.033+0000 7f85b67cc740 1 -- 172.21.15.90:0/759341853 shutdown_connections 2024-04-02T14:35:37.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.033+0000 7f85b67cc740 1 -- 172.21.15.90:0/759341853 wait complete. 2024-04-02T14:35:37.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.034+0000 7f85b67cc740 1 -- start start 2024-04-02T14:35:37.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.034+0000 7f85b67cc740 1 Processor -- start 2024-04-02T14:35:37.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.036+0000 7f85b67cc740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a5a85af540 0x55a5a85b58c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:37.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.037+0000 7f85b67cc740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a5a85b0220 0x55a5a85b5e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:37.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.038+0000 7f85b67cc740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a5a85b88b0 0x55a5a85b6340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:37.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.038+0000 7f85b67cc740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55a5a85bb4e0 con 0x55a5a85b0220 2024-04-02T14:35:37.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.038+0000 7f85b67cc740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55a5a84cea60 con 0x55a5a85af540 2024-04-02T14:35:37.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.038+0000 7f85b67cc740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55a5a85bb660 con 0x55a5a85b88b0 2024-04-02T14:35:37.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.038+0000 7f85b67cc740 10 client.0 _ll_register_callbacks cb 0x55a5a84d4030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:35:37.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.038+0000 7f85b1a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a5a85b88b0 0x55a5a85b6340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:37.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.039+0000 7f85b1a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a5a85b88b0 0x55a5a85b6340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:41220/0 (socket says 172.21.15.90:41220) 2024-04-02T14:35:37.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.039+0000 7f85b1a00640 1 -- 172.21.15.90:0/2487126929 learned_addr learned my addr 172.21.15.90:0/2487126929 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:35:37.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.037+0000 7f85b1000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a5a85af540 0x55a5a85b58c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:37.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.039+0000 7f85abe00640 1 --2- 172.21.15.90:0/2487126929 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a5a85b0220 0x55a5a85b5e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:37.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.040+0000 7f85b1a00640 1 -- 172.21.15.90:0/2487126929 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a5a85af540 msgr2=0x55a5a85b58c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:35:37.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.040+0000 7f85b1a00640 1 --2- 172.21.15.90:0/2487126929 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a5a85af540 0x55a5a85b58c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:37.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.040+0000 7f85b1a00640 1 -- 172.21.15.90:0/2487126929 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a5a85b0220 msgr2=0x55a5a85b5e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:35:37.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.040+0000 7f85b1a00640 1 --2- 172.21.15.90:0/2487126929 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a5a85b0220 0x55a5a85b5e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:37.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.040+0000 7f85b1a00640 1 -- 172.21.15.90:0/2487126929 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a5a834ff70 con 0x55a5a85b88b0 2024-04-02T14:35:37.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.041+0000 7f85b1a00640 1 --2- 172.21.15.90:0/2487126929 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a5a85b88b0 0x55a5a85b6340 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f8598010a10 tx=0x7f8598004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=369b692ccfbf2952 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:35:37.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.041+0000 7f8596a00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:35:37.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.042+0000 7f8596a00640 1 -- 172.21.15.90:0/2487126929 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8598016070 con 0x55a5a85b88b0 2024-04-02T14:35:37.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.042+0000 7f8596a00640 1 -- 172.21.15.90:0/2487126929 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f859801bd40 con 0x55a5a85b88b0 2024-04-02T14:35:37.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.042+0000 7f85b67cc740 1 -- 172.21.15.90:0/2487126929 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f85a403a040 con 0x55a5a85b88b0 2024-04-02T14:35:37.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.043+0000 7f8596a00640 1 -- 172.21.15.90:0/2487126929 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f8598027480 con 0x55a5a85b88b0 2024-04-02T14:35:37.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.043+0000 7f85b67cc740 10 client.25630 fetch_fsmap learned FSMap version 27 2024-04-02T14:35:37.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.043+0000 7f85b67cc740 1 -- 172.21.15.90:0/2487126929 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55a5a81e7170 con 0x55a5a85b88b0 2024-04-02T14:35:37.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.044+0000 7f8596a00640 1 -- 172.21.15.90:0/2487126929 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f85a403a040 con 0x55a5a85b88b0 2024-04-02T14:35:37.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.045+0000 7f8596a00640 1 -- 172.21.15.90:0/2487126929 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f8598030820 con 0x55a5a85b88b0 2024-04-02T14:35:37.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.045+0000 7f85b67cc740 10 client.25630 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:35:37.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.045+0000 7f85b67cc740 10 client.25630 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:35:37.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.045+0000 7f85b67cc740 20 client.25630 populate_metadata read hostname 'smithi090' 2024-04-02T14:35:37.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.045+0000 7f85b67cc740 1 --2- 172.21.15.90:0/2487126929 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55a5a85c8af0 0x55a5a85caee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:37.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.045+0000 7f85b67cc740 4 client.25630 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:35:37.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.045+0000 7f85b67cc740 1 -- 172.21.15.90:0/2487126929 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a5a8520850 con 0x55a5a85c8af0 2024-04-02T14:35:37.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.047+0000 7f85b1000640 1 --2- 172.21.15.90:0/2487126929 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55a5a85c8af0 0x55a5a85caee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:37.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.049+0000 7f85b1000640 1 --2- 172.21.15.90:0/2487126929 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55a5a85c8af0 0x55a5a85caee0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a37ae29fa37fd57c server_cookie=c6942bf63956f34a in_seq=0 out_seq=0 2024-04-02T14:35:37.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.049+0000 7f8596a00640 10 client.25630 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:35:37.052 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.050+0000 7f8596a00640 1 -- 172.21.15.90:0/2487126929 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a5a8520850 con 0x55a5a85c8af0 2024-04-02T14:35:37.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.050+0000 7f8596a00640 10 client.25630 handle_command_reply: tid=0 2024-04-02T14:35:37.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.051+0000 7f85b67cc740 1 client.25630 shutdown 2024-04-02T14:35:37.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.052+0000 7f85b67cc740 1 -- 172.21.15.90:0/2487126929 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a5a85b88b0 msgr2=0x55a5a85b6340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:35:37.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.052+0000 7f85b67cc740 1 --2- 172.21.15.90:0/2487126929 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a5a85b88b0 0x55a5a85b6340 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f8598010a10 tx=0x7f8598004730 comp rx=0 tx=0).stop 2024-04-02T14:35:37.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.052+0000 7f85b67cc740 1 -- 172.21.15.90:0/2487126929 shutdown_connections 2024-04-02T14:35:37.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.052+0000 7f85b67cc740 1 --2- 172.21.15.90:0/2487126929 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55a5a85c8af0 0x55a5a85caee0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:37.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.052+0000 7f85b67cc740 1 --2- 172.21.15.90:0/2487126929 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a5a85b88b0 0x55a5a85b6340 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:37.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.052+0000 7f85b67cc740 1 --2- 172.21.15.90:0/2487126929 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a5a85b0220 0x55a5a85b5e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:37.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.052+0000 7f85b67cc740 1 --2- 172.21.15.90:0/2487126929 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a5a85af540 0x55a5a85b58c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:37.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.052+0000 7f85b67cc740 1 -- 172.21.15.90:0/2487126929 >> 172.21.15.90:0/2487126929 conn(0x55a5a8543270 msgr2=0x55a5a85abce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:35:37.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.052+0000 7f8596a00640 0 client.25630 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:35:37.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.053+0000 7f85b67cc740 1 -- 172.21.15.90:0/2487126929 shutdown_connections 2024-04-02T14:35:37.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.053+0000 7f85b67cc740 1 -- 172.21.15.90:0/2487126929 wait complete. 2024-04-02T14:35:37.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.053+0000 7f85b67cc740 20 client.25630 trim_cache size 0 max 16384 2024-04-02T14:35:37.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.093+0000 7f85b67cc740 1 Processor -- start 2024-04-02T14:35:37.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.093+0000 7f85b67cc740 1 -- start start 2024-04-02T14:35:37.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.093+0000 7f85b67cc740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a5a85af540 0x55a5a85c43a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:37.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.093+0000 7f85b67cc740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a5a85b0220 0x55a5a85b6160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:37.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.093+0000 7f85b67cc740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a5a85b88b0 0x55a5a85d07d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:37.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.093+0000 7f85b67cc740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f85ac1b3b10 con 0x55a5a85b0220 2024-04-02T14:35:37.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.093+0000 7f85b67cc740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f85ac1b5650 con 0x55a5a85af540 2024-04-02T14:35:37.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.093+0000 7f85b67cc740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f85ac1b41e0 con 0x55a5a85b88b0 2024-04-02T14:35:37.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.094+0000 7f85b1a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a5a85b88b0 0x55a5a85d07d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:37.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.094+0000 7f85b1a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a5a85b88b0 0x55a5a85d07d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:41222/0 (socket says 172.21.15.90:41222) 2024-04-02T14:35:37.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.094+0000 7f85b1a00640 1 -- 172.21.15.90:0/2133217056 learned_addr learned my addr 172.21.15.90:0/2133217056 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:35:37.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.094+0000 7f85abe00640 1 --2- 172.21.15.90:0/2133217056 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a5a85b0220 0x55a5a85b6160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:37.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.094+0000 7f85b1a00640 1 -- 172.21.15.90:0/2133217056 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a5a85af540 msgr2=0x55a5a85c43a0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-02T14:35:37.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.094+0000 7f85b1000640 1 --2- 172.21.15.90:0/2133217056 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a5a85af540 0x55a5a85c43a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:37.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.094+0000 7f85b1a00640 1 --2- 172.21.15.90:0/2133217056 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a5a85af540 0x55a5a85c43a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:37.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.094+0000 7f85b1a00640 1 -- 172.21.15.90:0/2133217056 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a5a85b0220 msgr2=0x55a5a85b6160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:35:37.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.094+0000 7f85b1a00640 1 --2- 172.21.15.90:0/2133217056 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a5a85b0220 0x55a5a85b6160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:37.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.094+0000 7f85b1a00640 1 -- 172.21.15.90:0/2133217056 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a5a85c76a0 con 0x55a5a85b88b0 2024-04-02T14:35:37.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.095+0000 7f85abe00640 1 --2- 172.21.15.90:0/2133217056 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a5a85b0220 0x55a5a85b6160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:35:37.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.095+0000 7f85b1a00640 1 --2- 172.21.15.90:0/2133217056 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a5a85b88b0 0x55a5a85d07d0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f859800a900 tx=0x7f85980106f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c044390a7cec73f server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:35:37.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.095+0000 7f85a1600640 1 -- 172.21.15.90:0/2133217056 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f859803e070 con 0x55a5a85b88b0 2024-04-02T14:35:37.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.095+0000 7f85a1600640 1 -- 172.21.15.90:0/2133217056 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f859801b780 con 0x55a5a85b88b0 2024-04-02T14:35:37.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.095+0000 7f85a1600640 1 -- 172.21.15.90:0/2133217056 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8598027bf0 con 0x55a5a85b88b0 2024-04-02T14:35:37.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.096+0000 7f85b67cc740 1 -- 172.21.15.90:0/2133217056 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a5a85b88b0 msgr2=0x55a5a85d07d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:35:37.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.096+0000 7f85b67cc740 1 --2- 172.21.15.90:0/2133217056 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a5a85b88b0 0x55a5a85d07d0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f859800a900 tx=0x7f85980106f0 comp rx=0 tx=0).stop 2024-04-02T14:35:37.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.097+0000 7f85b67cc740 1 -- 172.21.15.90:0/2133217056 shutdown_connections 2024-04-02T14:35:37.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.097+0000 7f85b67cc740 1 --2- 172.21.15.90:0/2133217056 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a5a85b88b0 0x55a5a85d07d0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:37.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.097+0000 7f85b67cc740 1 --2- 172.21.15.90:0/2133217056 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a5a85b0220 0x55a5a85b6160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:37.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.097+0000 7f85b67cc740 1 --2- 172.21.15.90:0/2133217056 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a5a85af540 0x55a5a85c43a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:37.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.097+0000 7f85b67cc740 1 -- 172.21.15.90:0/2133217056 >> 172.21.15.90:0/2133217056 conn(0x55a5a8543270 msgr2=0x55a5a85abc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:35:37.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.097+0000 7f85b67cc740 1 -- 172.21.15.90:0/2133217056 shutdown_connections 2024-04-02T14:35:37.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.097+0000 7f85b67cc740 1 -- 172.21.15.90:0/2133217056 wait complete. 2024-04-02T14:35:37.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.098+0000 7f85b67cc740 1 -- start start 2024-04-02T14:35:37.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.098+0000 7f85b67cc740 1 Processor -- start 2024-04-02T14:35:37.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.099+0000 7f85b67cc740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a5a85af540 0x55a5a85c0aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:37.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.099+0000 7f85b67cc740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a5a85b0220 0x55a5a85b1b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:37.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.100+0000 7f85b67cc740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a5a85b88b0 0x55a5a85b2090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:37.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.100+0000 7f85b67cc740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f85ac1b4f80 con 0x55a5a85b0220 2024-04-02T14:35:37.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.100+0000 7f85b67cc740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f85ac1b48b0 con 0x55a5a85b88b0 2024-04-02T14:35:37.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.100+0000 7f85b67cc740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f85ac1b1af0 con 0x55a5a85af540 2024-04-02T14:35:37.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.100+0000 7f85b1000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a5a85af540 0x55a5a85c0aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:37.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.100+0000 7f85b1a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a5a85b88b0 0x55a5a85b2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:37.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.100+0000 7f85b1000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a5a85af540 0x55a5a85c0aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:41234/0 (socket says 172.21.15.90:41234) 2024-04-02T14:35:37.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.100+0000 7f85b1a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a5a85b88b0 0x55a5a85b2090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:58098/0 (socket says 172.21.15.90:58098) 2024-04-02T14:35:37.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.100+0000 7f85b1000640 1 -- 172.21.15.90:0/2909340016 learned_addr learned my addr 172.21.15.90:0/2909340016 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:35:37.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.100+0000 7f85b67cc740 10 client.0 _ll_register_callbacks cb 0x7f859800a7d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:35:37.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.100+0000 7f85b1a00640 1 -- 172.21.15.90:0/2909340016 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a5a85af540 msgr2=0x55a5a85c0aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:35:37.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.100+0000 7f85b1a00640 1 --2- 172.21.15.90:0/2909340016 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a5a85af540 0x55a5a85c0aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:37.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.100+0000 7f85b1a00640 1 -- 172.21.15.90:0/2909340016 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a5a85b0220 msgr2=0x55a5a85b1b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:35:37.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.101+0000 7f85abe00640 1 --2- 172.21.15.90:0/2909340016 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a5a85b0220 0x55a5a85b1b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:37.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.101+0000 7f85b1a00640 1 --2- 172.21.15.90:0/2909340016 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a5a85b0220 0x55a5a85b1b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:37.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.101+0000 7f85b1a00640 1 -- 172.21.15.90:0/2909340016 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a5a85a5170 con 0x55a5a85b88b0 2024-04-02T14:35:37.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.101+0000 7f85abe00640 1 --2- 172.21.15.90:0/2909340016 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a5a85b0220 0x55a5a85b1b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:35:37.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.101+0000 7f85b1a00640 1 --2- 172.21.15.90:0/2909340016 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a5a85b88b0 0x55a5a85b2090 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f8598030c40 tx=0x7f8598030700 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b8e5d6c78f5af704 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:35:37.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.101+0000 7f8596a00640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:35:37.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.102+0000 7f8596a00640 1 -- 172.21.15.90:0/2909340016 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f859803e030 con 0x55a5a85b88b0 2024-04-02T14:35:37.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.102+0000 7f8596a00640 1 -- 172.21.15.90:0/2909340016 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f8598005040 con 0x55a5a85b88b0 2024-04-02T14:35:37.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.102+0000 7f85b67cc740 1 -- 172.21.15.90:0/2909340016 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f859801b780 con 0x55a5a85b88b0 2024-04-02T14:35:37.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.102+0000 7f8596a00640 1 -- 172.21.15.90:0/2909340016 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f8598049050 con 0x55a5a85b88b0 2024-04-02T14:35:37.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.102+0000 7f85b67cc740 10 client.25679 fetch_fsmap learned FSMap version 27 2024-04-02T14:35:37.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.102+0000 7f85b67cc740 1 -- 172.21.15.90:0/2909340016 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55a5a85ae570 con 0x55a5a85b88b0 2024-04-02T14:35:37.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.103+0000 7f8596a00640 1 -- 172.21.15.90:0/2909340016 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f859801b780 con 0x55a5a85b88b0 2024-04-02T14:35:37.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.103+0000 7f8596a00640 1 -- 172.21.15.90:0/2909340016 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f8598004660 con 0x55a5a85b88b0 2024-04-02T14:35:37.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.103+0000 7f85b67cc740 10 client.25679 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:35:37.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.103+0000 7f85b67cc740 10 client.25679 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:35:37.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.103+0000 7f85b67cc740 20 client.25679 populate_metadata read hostname 'smithi090' 2024-04-02T14:35:37.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.103+0000 7f85b67cc740 1 --2- 172.21.15.90:0/2909340016 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55a5a8630050 0x55a5a8632440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:37.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.103+0000 7f85b67cc740 4 client.25679 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-02T14:35:37.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.103+0000 7f85b67cc740 1 -- 172.21.15.90:0/2909340016 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55a5a851aba0 con 0x55a5a8630050 2024-04-02T14:35:37.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.104+0000 7f85b1000640 1 --2- 172.21.15.90:0/2909340016 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55a5a8630050 0x55a5a8632440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:37.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.105+0000 7f85b1000640 1 --2- 172.21.15.90:0/2909340016 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55a5a8630050 0x55a5a8632440 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=31d73704095da994 server_cookie=479f00bc9157fc0d in_seq=0 out_seq=0 2024-04-02T14:35:37.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.105+0000 7f8596a00640 10 client.25679 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:35:37.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.106+0000 7f8596a00640 1 -- 172.21.15.90:0/2909340016 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55a5a851aba0 con 0x55a5a8630050 2024-04-02T14:35:37.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.106+0000 7f8596a00640 10 client.25679 handle_command_reply: tid=0 2024-04-02T14:35:37.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.106+0000 7f85b67cc740 1 client.25679 shutdown 2024-04-02T14:35:37.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.106+0000 7f85b67cc740 1 -- 172.21.15.90:0/2909340016 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a5a85b88b0 msgr2=0x55a5a85b2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:35:37.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.106+0000 7f85b67cc740 1 --2- 172.21.15.90:0/2909340016 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a5a85b88b0 0x55a5a85b2090 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f8598030c40 tx=0x7f8598030700 comp rx=0 tx=0).stop 2024-04-02T14:35:37.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.107+0000 7f85b67cc740 1 -- 172.21.15.90:0/2909340016 shutdown_connections 2024-04-02T14:35:37.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.107+0000 7f85b67cc740 1 --2- 172.21.15.90:0/2909340016 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55a5a8630050 0x55a5a8632440 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:37.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.107+0000 7f85b67cc740 1 --2- 172.21.15.90:0/2909340016 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a5a85b88b0 0x55a5a85b2090 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:37.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.107+0000 7f85b67cc740 1 --2- 172.21.15.90:0/2909340016 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a5a85b0220 0x55a5a85b1b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:37.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.107+0000 7f85b67cc740 1 --2- 172.21.15.90:0/2909340016 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a5a85af540 0x55a5a85c0aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:37.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.107+0000 7f85b67cc740 1 -- 172.21.15.90:0/2909340016 >> 172.21.15.90:0/2909340016 conn(0x55a5a8543270 msgr2=0x55a5a85a8760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:35:37.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.107+0000 7f8596a00640 0 client.25679 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:35:37.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.107+0000 7f85b67cc740 1 -- 172.21.15.90:0/2909340016 shutdown_connections 2024-04-02T14:35:37.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.107+0000 7f85b67cc740 1 -- 172.21.15.90:0/2909340016 wait complete. 2024-04-02T14:35:37.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.107+0000 7f85b67cc740 20 client.25679 trim_cache size 0 max 16384 2024-04-02T14:35:37.110 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-02T14:35:37.110 INFO:teuthology.orchestra.run.smithi090.stdout: "return_code": 0, 2024-04-02T14:35:37.110 INFO:teuthology.orchestra.run.smithi090.stdout: "scrub_tag": "2886f6e0-04af-43ca-9668-74cc1589d4fd", 2024-04-02T14:35:37.110 INFO:teuthology.orchestra.run.smithi090.stdout: "mode": "asynchronous" 2024-04-02T14:35:37.110 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-02T14:35:37.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.108+0000 7f85b2400640 1 -- 172.21.15.90:0/2473526050 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f85900a3f90 msgr2=0x7f85900a6450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:35:37.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.108+0000 7f85b2400640 1 --2- 172.21.15.90:0/2473526050 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f85900a3f90 0x7f85900a6450 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f85ac110eb0 tx=0x7f859800a250 comp rx=0 tx=0).stop 2024-04-02T14:35:37.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.108+0000 7f85b2400640 1 -- 172.21.15.90:0/2473526050 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f85ac103ad0 msgr2=0x7f85ac110780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:35:37.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.108+0000 7f85b2400640 1 --2- 172.21.15.90:0/2473526050 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f85ac103ad0 0x7f85ac110780 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f85a402c710 tx=0x7f85a40346f0 comp rx=0 tx=0).stop 2024-04-02T14:35:37.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.109+0000 7f85b2400640 1 -- 172.21.15.90:0/2473526050 shutdown_connections 2024-04-02T14:35:37.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.109+0000 7f85b2400640 1 --2- 172.21.15.90:0/2473526050 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f85900a3f90 0x7f85900a6450 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:37.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.109+0000 7f85b2400640 1 --2- 172.21.15.90:0/2473526050 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f85ac109b70 0x7f85ac10dd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:37.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.109+0000 7f85b2400640 1 --2- 172.21.15.90:0/2473526050 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f85ac104490 0x7f85ac10d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:37.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.109+0000 7f85b2400640 1 --2- 172.21.15.90:0/2473526050 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f85ac103ad0 0x7f85ac110780 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:37.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.109+0000 7f85b2400640 1 -- 172.21.15.90:0/2473526050 >> 172.21.15.90:0/2473526050 conn(0x7f85ac0ff560 msgr2=0x7f85ac0ffa40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:35:37.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.109+0000 7f85b2400640 1 -- 172.21.15.90:0/2473526050 shutdown_connections 2024-04-02T14:35:37.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.109+0000 7f85b2400640 1 -- 172.21.15.90:0/2473526050 wait complete. 2024-04-02T14:35:37.141 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-02T14:35:37.141 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-02T14:35:37.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.486+0000 7f35b8c00640 1 -- 172.21.15.90:0/3703952898 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f35b4101960 msgr2=0x7f35b4101de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:35:37.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.486+0000 7f35b8c00640 1 --2- 172.21.15.90:0/3703952898 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f35b4101960 0x7f35b4101de0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f35a0009890 tx=0x7f35a002bb70 comp rx=0 tx=0).stop 2024-04-02T14:35:37.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.487+0000 7f35b8c00640 1 -- 172.21.15.90:0/3703952898 shutdown_connections 2024-04-02T14:35:37.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.487+0000 7f35b8c00640 1 --2- 172.21.15.90:0/3703952898 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f35b4102320 0x7f35b410e830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:37.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.487+0000 7f35b8c00640 1 --2- 172.21.15.90:0/3703952898 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f35b4101960 0x7f35b4101de0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:37.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.487+0000 7f35b8c00640 1 --2- 172.21.15.90:0/3703952898 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f35b4107970 0x7f35b4107d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:37.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.487+0000 7f35b8c00640 1 -- 172.21.15.90:0/3703952898 >> 172.21.15.90:0/3703952898 conn(0x7f35b40fd470 msgr2=0x7f35b40ff890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:35:37.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.489+0000 7f35b8c00640 1 -- 172.21.15.90:0/3703952898 shutdown_connections 2024-04-02T14:35:37.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.489+0000 7f35b8c00640 1 -- 172.21.15.90:0/3703952898 wait complete. 2024-04-02T14:35:37.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.491+0000 7f35b8c00640 1 Processor -- start 2024-04-02T14:35:37.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.492+0000 7f35b8c00640 1 -- start start 2024-04-02T14:35:37.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.493+0000 7f35b8c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f35b4101960 0x7f35b410f4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:37.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.493+0000 7f35b8c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f35b4102320 0x7f35b410fa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:37.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.494+0000 7f35b8c00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f35b4107970 0x7f35b410ff70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:37.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.494+0000 7f35b8c00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f35b4113f50 con 0x7f35b4101960 2024-04-02T14:35:37.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.494+0000 7f35b8c00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f35b4113dd0 con 0x7f35b4107970 2024-04-02T14:35:37.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.494+0000 7f35b8c00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f35b41140d0 con 0x7f35b4102320 2024-04-02T14:35:37.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.494+0000 7f35b2a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f35b4102320 0x7f35b410fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:37.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.495+0000 7f35b2a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f35b4102320 0x7f35b410fa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:41254/0 (socket says 172.21.15.90:41254) 2024-04-02T14:35:37.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.495+0000 7f35b2a00640 1 -- 172.21.15.90:0/1335092904 learned_addr learned my addr 172.21.15.90:0/1335092904 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:35:37.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.495+0000 7f35b3e00640 1 --2- 172.21.15.90:0/1335092904 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f35b4107970 0x7f35b410ff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:37.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.496+0000 7f35b2a00640 1 -- 172.21.15.90:0/1335092904 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f35b4107970 msgr2=0x7f35b410ff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:35:37.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.496+0000 7f35b3400640 1 --2- 172.21.15.90:0/1335092904 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f35b4101960 0x7f35b410f4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:37.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.497+0000 7f35b2a00640 1 --2- 172.21.15.90:0/1335092904 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f35b4107970 0x7f35b410ff70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:37.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.497+0000 7f35b2a00640 1 -- 172.21.15.90:0/1335092904 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f35b4101960 msgr2=0x7f35b410f4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:35:37.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.497+0000 7f35b2a00640 1 --2- 172.21.15.90:0/1335092904 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f35b4101960 0x7f35b410f4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:37.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.497+0000 7f35b2a00640 1 -- 172.21.15.90:0/1335092904 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35b41aab80 con 0x7f35b4102320 2024-04-02T14:35:37.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.497+0000 7f35b3400640 1 --2- 172.21.15.90:0/1335092904 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f35b4101960 0x7f35b410f4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:35:37.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.497+0000 7f35b3e00640 1 --2- 172.21.15.90:0/1335092904 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f35b4107970 0x7f35b410ff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:35:37.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.498+0000 7f35b2a00640 1 --2- 172.21.15.90:0/1335092904 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f35b4102320 0x7f35b410fa30 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f35a0000c00 tx=0x7f35a0004530 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:35:37.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.499+0000 7f35abe00640 1 -- 172.21.15.90:0/1335092904 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35a0041020 con 0x7f35b4102320 2024-04-02T14:35:37.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.500+0000 7f35abe00640 1 -- 172.21.15.90:0/1335092904 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f35a002c9f0 con 0x7f35b4102320 2024-04-02T14:35:37.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.500+0000 7f35abe00640 1 -- 172.21.15.90:0/1335092904 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35a0039040 con 0x7f35b4102320 2024-04-02T14:35:37.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.501+0000 7f35b8c00640 1 -- 172.21.15.90:0/1335092904 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35b41aae10 con 0x7f35b4102320 2024-04-02T14:35:37.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.502+0000 7f35b8c00640 1 -- 172.21.15.90:0/1335092904 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35b41ab220 con 0x7f35b4102320 2024-04-02T14:35:37.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.509+0000 7f35bce6f740 1 Processor -- start 2024-04-02T14:35:37.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.509+0000 7f35abe00640 1 -- 172.21.15.90:0/1335092904 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7f35a0044020 con 0x7f35b4102320 2024-04-02T14:35:37.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.509+0000 7f35bce6f740 1 -- start start 2024-04-02T14:35:37.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.512+0000 7f35bce6f740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b297dda960 0x55b297ddad40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:37.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.513+0000 7f35bce6f740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b297ddb280 0x55b297de3920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:37.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.513+0000 7f35bce6f740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b297de3e60 0x55b297de6250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:37.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.513+0000 7f35bce6f740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55b297d3e770 con 0x55b297de3e60 2024-04-02T14:35:37.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.513+0000 7f35bce6f740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55b297ce6700 con 0x55b297dda960 2024-04-02T14:35:37.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.513+0000 7f35bce6f740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55b297d3a720 con 0x55b297ddb280 2024-04-02T14:35:37.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.513+0000 7f35b3e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b297dda960 0x55b297ddad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:37.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.513+0000 7f35b3e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b297dda960 0x55b297ddad40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:58126/0 (socket says 172.21.15.90:58126) 2024-04-02T14:35:37.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.513+0000 7f35b3e00640 1 -- 172.21.15.90:0/2324276435 learned_addr learned my addr 172.21.15.90:0/2324276435 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:35:37.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.514+0000 7f35abe00640 1 --2- 172.21.15.90:0/1335092904 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f35980a3f90 0x7f35980a6450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:37.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.514+0000 7f35b3400640 1 --2- 172.21.15.90:0/2324276435 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b297ddb280 0x55b297de3920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:37.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.514+0000 7f35b3e00640 1 -- 172.21.15.90:0/2324276435 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b297ddb280 msgr2=0x55b297de3920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:35:37.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.514+0000 7f35b2a00640 1 --2- 172.21.15.90:0/2324276435 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b297de3e60 0x55b297de6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:37.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.514+0000 7f35b3e00640 1 --2- 172.21.15.90:0/2324276435 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b297ddb280 0x55b297de3920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:37.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.514+0000 7f35b3e00640 1 -- 172.21.15.90:0/2324276435 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b297de3e60 msgr2=0x55b297de6250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:35:37.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.514+0000 7f35b3e00640 1 --2- 172.21.15.90:0/2324276435 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b297de3e60 0x55b297de6250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:37.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.514+0000 7f35b3e00640 1 -- 172.21.15.90:0/2324276435 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b297b1c7d0 con 0x55b297dda960 2024-04-02T14:35:37.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.515+0000 7f35abe00640 1 -- 172.21.15.90:0/1335092904 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f35a00f21d0 con 0x7f35b4102320 2024-04-02T14:35:37.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.515+0000 7f35b3e00640 1 --2- 172.21.15.90:0/1335092904 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f35980a3f90 0x7f35980a6450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:37.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.515+0000 7f35b3e00640 1 --2- 172.21.15.90:0/2324276435 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b297dda960 0x55b297ddad40 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f35ac011540 tx=0x7f35ac011a00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b772871ea242b553 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:35:37.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.516+0000 7f35b2a00640 1 --2- 172.21.15.90:0/2324276435 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b297de3e60 0x55b297de6250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:35:37.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.516+0000 7f359fe00640 1 -- 172.21.15.90:0/2324276435 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35ac00f5c0 con 0x55b297dda960 2024-04-02T14:35:37.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.516+0000 7f359fe00640 1 -- 172.21.15.90:0/2324276435 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f35ac01e070 con 0x55b297dda960 2024-04-02T14:35:37.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.516+0000 7f359fe00640 1 -- 172.21.15.90:0/2324276435 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35ac019d20 con 0x55b297dda960 2024-04-02T14:35:37.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.518+0000 7f35bce6f740 1 -- 172.21.15.90:0/2324276435 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b297dda960 msgr2=0x55b297ddad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:35:37.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.518+0000 7f35bce6f740 1 --2- 172.21.15.90:0/2324276435 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b297dda960 0x55b297ddad40 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f35ac011540 tx=0x7f35ac011a00 comp rx=0 tx=0).stop 2024-04-02T14:35:37.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.519+0000 7f35b3e00640 1 --2- 172.21.15.90:0/1335092904 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f35980a3f90 0x7f35980a6450 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f35ac010010 tx=0x7f35ac00ed40 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:35:37.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.519+0000 7f35bce6f740 1 -- 172.21.15.90:0/2324276435 shutdown_connections 2024-04-02T14:35:37.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.519+0000 7f35bce6f740 1 --2- 172.21.15.90:0/2324276435 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b297de3e60 0x55b297de6250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:37.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.519+0000 7f35bce6f740 1 --2- 172.21.15.90:0/2324276435 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b297ddb280 0x55b297de3920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:37.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.519+0000 7f35bce6f740 1 --2- 172.21.15.90:0/2324276435 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b297dda960 0x55b297ddad40 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:37.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.519+0000 7f35bce6f740 1 -- 172.21.15.90:0/2324276435 >> 172.21.15.90:0/2324276435 conn(0x55b297dd5090 msgr2=0x55b297dd5470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:35:37.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.520+0000 7f35bce6f740 1 -- 172.21.15.90:0/2324276435 shutdown_connections 2024-04-02T14:35:37.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.521+0000 7f35bce6f740 1 -- 172.21.15.90:0/2324276435 wait complete. 2024-04-02T14:35:37.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.522+0000 7f35bce6f740 1 -- start start 2024-04-02T14:35:37.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.522+0000 7f35bce6f740 1 Processor -- start 2024-04-02T14:35:37.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.523+0000 7f35bce6f740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b297dda960 0x55b297de0e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:37.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.523+0000 7f35bce6f740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b297ddb280 0x55b297de1380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:37.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.523+0000 7f35bce6f740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b297de3e60 0x55b297de18c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:37.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.523+0000 7f35bce6f740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55b297de6c50 con 0x55b297dda960 2024-04-02T14:35:37.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.523+0000 7f35bce6f740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55b297cf9ac0 con 0x55b297de3e60 2024-04-02T14:35:37.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.523+0000 7f35bce6f740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55b297de6dd0 con 0x55b297ddb280 2024-04-02T14:35:37.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.523+0000 7f35bce6f740 10 client.0 _ll_register_callbacks cb 0x55b297cfc180 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:35:37.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.524+0000 7f35b3400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b297dda960 0x55b297de0e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:37.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.524+0000 7f35b3e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b297de3e60 0x55b297de18c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:37.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.524+0000 7f35b3e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b297de3e60 0x55b297de18c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:58134/0 (socket says 172.21.15.90:58134) 2024-04-02T14:35:37.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.524+0000 7f35b3e00640 1 -- 172.21.15.90:0/58599597 learned_addr learned my addr 172.21.15.90:0/58599597 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:35:37.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.524+0000 7f35b2a00640 1 --2- 172.21.15.90:0/58599597 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b297ddb280 0x55b297de1380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:37.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.525+0000 7f35b3e00640 1 -- 172.21.15.90:0/58599597 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b297ddb280 msgr2=0x55b297de1380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:35:37.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.525+0000 7f35b3e00640 1 --2- 172.21.15.90:0/58599597 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b297ddb280 0x55b297de1380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:37.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.525+0000 7f35b3e00640 1 -- 172.21.15.90:0/58599597 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b297dda960 msgr2=0x55b297de0e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:35:37.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.525+0000 7f35b3e00640 1 --2- 172.21.15.90:0/58599597 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b297dda960 0x55b297de0e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:37.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.525+0000 7f35b3e00640 1 -- 172.21.15.90:0/58599597 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b297a523f0 con 0x55b297de3e60 2024-04-02T14:35:37.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.525+0000 7f35b3400640 1 --2- 172.21.15.90:0/58599597 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b297dda960 0x55b297de0e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:35:37.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.526+0000 7f35b3e00640 1 --2- 172.21.15.90:0/58599597 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b297de3e60 0x55b297de18c0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f35ac022db0 tx=0x7f35ac014000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7f8a41d01da682d7 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:35:37.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.526+0000 7f359cc00640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:35:37.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.526+0000 7f359cc00640 1 -- 172.21.15.90:0/58599597 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35ac029070 con 0x55b297de3e60 2024-04-02T14:35:37.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.526+0000 7f359cc00640 1 -- 172.21.15.90:0/58599597 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f35ac00f7d0 con 0x55b297de3e60 2024-04-02T14:35:37.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.526+0000 7f35bce6f740 1 -- 172.21.15.90:0/58599597 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f35ac01e070 con 0x55b297de3e60 2024-04-02T14:35:37.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.527+0000 7f359cc00640 1 -- 172.21.15.90:0/58599597 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f35ac03d050 con 0x55b297de3e60 2024-04-02T14:35:37.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.527+0000 7f35bce6f740 10 client.25691 fetch_fsmap learned FSMap version 27 2024-04-02T14:35:37.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.527+0000 7f35bce6f740 1 -- 172.21.15.90:0/58599597 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55b297a120f0 con 0x55b297de3e60 2024-04-02T14:35:37.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.527+0000 7f359cc00640 1 -- 172.21.15.90:0/58599597 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f35ac01e070 con 0x55b297de3e60 2024-04-02T14:35:37.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.529+0000 7f359cc00640 1 -- 172.21.15.90:0/58599597 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f35ac038440 con 0x55b297de3e60 2024-04-02T14:35:37.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.529+0000 7f35bce6f740 10 client.25691 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:35:37.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.529+0000 7f35bce6f740 10 client.25691 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:35:37.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.529+0000 7f35bce6f740 20 client.25691 populate_metadata read hostname 'smithi090' 2024-04-02T14:35:37.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.529+0000 7f35bce6f740 1 --2- 172.21.15.90:0/58599597 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55b297df3b60 0x55b297df5f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:37.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.529+0000 7f35bce6f740 4 client.25691 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:35:37.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.529+0000 7f35bce6f740 1 -- 172.21.15.90:0/58599597 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b297cdf630 con 0x55b297df3b60 2024-04-02T14:35:37.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.530+0000 7f35b3400640 1 --2- 172.21.15.90:0/58599597 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55b297df3b60 0x55b297df5f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:37.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.533+0000 7f35b3400640 1 --2- 172.21.15.90:0/58599597 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55b297df3b60 0x55b297df5f50 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=22e039b2d259fb1 server_cookie=389326122705362e in_seq=0 out_seq=0 2024-04-02T14:35:37.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.533+0000 7f359cc00640 10 client.25691 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:35:37.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.534+0000 7f359cc00640 1 -- 172.21.15.90:0/58599597 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b297cdf630 con 0x55b297df3b60 2024-04-02T14:35:37.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.534+0000 7f359cc00640 10 client.25691 handle_command_reply: tid=0 2024-04-02T14:35:37.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.534+0000 7f35bce6f740 1 client.25691 shutdown 2024-04-02T14:35:37.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.535+0000 7f35bce6f740 1 -- 172.21.15.90:0/58599597 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b297de3e60 msgr2=0x55b297de18c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:35:37.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.535+0000 7f35bce6f740 1 --2- 172.21.15.90:0/58599597 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b297de3e60 0x55b297de18c0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f35ac022db0 tx=0x7f35ac014000 comp rx=0 tx=0).stop 2024-04-02T14:35:37.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.535+0000 7f35bce6f740 1 -- 172.21.15.90:0/58599597 shutdown_connections 2024-04-02T14:35:37.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.535+0000 7f35bce6f740 1 --2- 172.21.15.90:0/58599597 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55b297df3b60 0x55b297df5f50 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:37.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.536+0000 7f35bce6f740 1 --2- 172.21.15.90:0/58599597 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b297de3e60 0x55b297de18c0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:37.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.536+0000 7f35bce6f740 1 --2- 172.21.15.90:0/58599597 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b297ddb280 0x55b297de1380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:37.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.536+0000 7f35bce6f740 1 --2- 172.21.15.90:0/58599597 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b297dda960 0x55b297de0e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:37.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.536+0000 7f35bce6f740 1 -- 172.21.15.90:0/58599597 >> 172.21.15.90:0/58599597 conn(0x55b297dd5090 msgr2=0x55b297dd6de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:35:37.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.536+0000 7f359cc00640 0 client.25691 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:35:37.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.537+0000 7f35bce6f740 1 -- 172.21.15.90:0/58599597 shutdown_connections 2024-04-02T14:35:37.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.537+0000 7f35bce6f740 1 -- 172.21.15.90:0/58599597 wait complete. 2024-04-02T14:35:37.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.537+0000 7f35bce6f740 20 client.25691 trim_cache size 0 max 16384 2024-04-02T14:35:37.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.572+0000 7f35bce6f740 1 Processor -- start 2024-04-02T14:35:37.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.573+0000 7f35bce6f740 1 -- start start 2024-04-02T14:35:37.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.573+0000 7f35bce6f740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b297dda960 0x55b297de3570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:37.576 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.573+0000 7f35bce6f740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b297ddb280 0x55b297defb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:37.576 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.573+0000 7f35bce6f740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b297de3e60 0x55b297ddfd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:37.576 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.573+0000 7f35bce6f740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f35b41b3dc0 con 0x55b297ddb280 2024-04-02T14:35:37.576 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.573+0000 7f35bce6f740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f35b41b5900 con 0x55b297dda960 2024-04-02T14:35:37.576 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.573+0000 7f35bce6f740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f35b41b4490 con 0x55b297de3e60 2024-04-02T14:35:37.576 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.574+0000 7f35b3e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b297de3e60 0x55b297ddfd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:37.576 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.574+0000 7f35b3e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b297de3e60 0x55b297ddfd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:41290/0 (socket says 172.21.15.90:41290) 2024-04-02T14:35:37.576 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.574+0000 7f35b3e00640 1 -- 172.21.15.90:0/3374204062 learned_addr learned my addr 172.21.15.90:0/3374204062 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:35:37.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.574+0000 7f35b2a00640 1 --2- 172.21.15.90:0/3374204062 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b297ddb280 0x55b297defb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:37.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.574+0000 7f35b3400640 1 --2- 172.21.15.90:0/3374204062 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b297dda960 0x55b297de3570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:37.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.575+0000 7f35b3400640 1 -- 172.21.15.90:0/3374204062 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b297de3e60 msgr2=0x55b297ddfd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:35:37.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.575+0000 7f35b3400640 1 --2- 172.21.15.90:0/3374204062 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b297de3e60 0x55b297ddfd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:37.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.575+0000 7f35b3400640 1 -- 172.21.15.90:0/3374204062 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b297ddb280 msgr2=0x55b297defb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:35:37.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.575+0000 7f35b3400640 1 --2- 172.21.15.90:0/3374204062 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b297ddb280 0x55b297defb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:37.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.575+0000 7f35b3400640 1 -- 172.21.15.90:0/3374204062 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b297dd0d10 con 0x55b297dda960 2024-04-02T14:35:37.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.575+0000 7f35b3e00640 1 --2- 172.21.15.90:0/3374204062 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b297de3e60 0x55b297ddfd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:35:37.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.575+0000 7f35b3400640 1 --2- 172.21.15.90:0/3374204062 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b297dda960 0x55b297de3570 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x55b297df65d0 tx=0x7f35a40095c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=405c5cb15e22a6c7 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:35:37.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.576+0000 7f359f400640 1 -- 172.21.15.90:0/3374204062 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35a4004d20 con 0x55b297dda960 2024-04-02T14:35:37.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.576+0000 7f359f400640 1 -- 172.21.15.90:0/3374204062 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f35a4021070 con 0x55b297dda960 2024-04-02T14:35:37.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.576+0000 7f359f400640 1 -- 172.21.15.90:0/3374204062 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35a400db30 con 0x55b297dda960 2024-04-02T14:35:37.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.577+0000 7f35bce6f740 1 -- 172.21.15.90:0/3374204062 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b297dda960 msgr2=0x55b297de3570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:35:37.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.577+0000 7f35bce6f740 1 --2- 172.21.15.90:0/3374204062 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b297dda960 0x55b297de3570 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x55b297df65d0 tx=0x7f35a40095c0 comp rx=0 tx=0).stop 2024-04-02T14:35:37.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.578+0000 7f35bce6f740 1 -- 172.21.15.90:0/3374204062 shutdown_connections 2024-04-02T14:35:37.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.578+0000 7f35bce6f740 1 --2- 172.21.15.90:0/3374204062 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b297de3e60 0x55b297ddfd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:37.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.578+0000 7f35bce6f740 1 --2- 172.21.15.90:0/3374204062 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b297ddb280 0x55b297defb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:37.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.578+0000 7f35bce6f740 1 --2- 172.21.15.90:0/3374204062 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b297dda960 0x55b297de3570 secure :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0x55b297df65d0 tx=0x7f35a40095c0 comp rx=0 tx=0).stop 2024-04-02T14:35:37.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.578+0000 7f35bce6f740 1 -- 172.21.15.90:0/3374204062 >> 172.21.15.90:0/3374204062 conn(0x55b297dd5090 msgr2=0x55b297dd4310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:35:37.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.578+0000 7f35bce6f740 1 -- 172.21.15.90:0/3374204062 shutdown_connections 2024-04-02T14:35:37.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.579+0000 7f35bce6f740 1 -- 172.21.15.90:0/3374204062 wait complete. 2024-04-02T14:35:37.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.579+0000 7f35bce6f740 1 -- start start 2024-04-02T14:35:37.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.579+0000 7f35bce6f740 1 Processor -- start 2024-04-02T14:35:37.583 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.581+0000 7f35bce6f740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b297ddb280 0x55b297dec020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:37.583 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.581+0000 7f35bce6f740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b297de3e60 0x55b297dec560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:37.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.581+0000 7f35bce6f740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b297dddf70 0x55b297dde3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:37.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.582+0000 7f35b3e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b297dddf70 0x55b297dde3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:37.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.582+0000 7f35b3e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b297dddf70 0x55b297dde3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:58156/0 (socket says 172.21.15.90:58156) 2024-04-02T14:35:37.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.582+0000 7f35b2a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b297de3e60 0x55b297dec560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:37.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.582+0000 7f35b3400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b297ddb280 0x55b297dec020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:37.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.582+0000 7f35b3400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b297ddb280 0x55b297dec020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:57746/0 (socket says 172.21.15.90:57746) 2024-04-02T14:35:37.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.582+0000 7f35b3e00640 1 -- 172.21.15.90:0/930289290 learned_addr learned my addr 172.21.15.90:0/930289290 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:35:37.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.582+0000 7f35bce6f740 1 -- 172.21.15.90:0/930289290 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f35b41b5230 con 0x55b297ddb280 2024-04-02T14:35:37.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.582+0000 7f35bce6f740 1 -- 172.21.15.90:0/930289290 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f35b41b4b60 con 0x55b297dddf70 2024-04-02T14:35:37.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.582+0000 7f35bce6f740 1 -- 172.21.15.90:0/930289290 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f35b41b1da0 con 0x55b297de3e60 2024-04-02T14:35:37.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.583+0000 7f35b3e00640 1 -- 172.21.15.90:0/930289290 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b297de3e60 msgr2=0x55b297dec560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:35:37.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.583+0000 7f35b3e00640 1 --2- 172.21.15.90:0/930289290 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b297de3e60 0x55b297dec560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:37.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.583+0000 7f35b3e00640 1 -- 172.21.15.90:0/930289290 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b297ddb280 msgr2=0x55b297dec020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:35:37.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.583+0000 7f35bce6f740 10 client.0 _ll_register_callbacks cb 0x7f35ac011440 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:35:37.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.583+0000 7f35b3e00640 1 --2- 172.21.15.90:0/930289290 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b297ddb280 0x55b297dec020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:37.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.583+0000 7f35b3e00640 1 -- 172.21.15.90:0/930289290 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b297dde930 con 0x55b297dddf70 2024-04-02T14:35:37.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.583+0000 7f35b3400640 1 --2- 172.21.15.90:0/930289290 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b297ddb280 0x55b297dec020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:35:37.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.583+0000 7f35b2a00640 1 --2- 172.21.15.90:0/930289290 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b297de3e60 0x55b297dec560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:35:37.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.583+0000 7f35bce6f740 1 -- 172.21.15.90:0/930289290 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f35a4021070 con 0x55b297dddf70 2024-04-02T14:35:37.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.584+0000 7f35b3e00640 1 --2- 172.21.15.90:0/930289290 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b297dddf70 0x55b297dde3f0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f35ac004600 tx=0x7f35ac00e620 comp rx=0 tx=0).ready entity=mon.2 client_cookie=37312b029d397222 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:35:37.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.584+0000 7f359cc00640 10 client.25697 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:35:37.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.585+0000 7f359cc00640 1 -- 172.21.15.90:0/930289290 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35ac005850 con 0x55b297dddf70 2024-04-02T14:35:37.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.585+0000 7f359cc00640 1 -- 172.21.15.90:0/930289290 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f35ac022d90 con 0x55b297dddf70 2024-04-02T14:35:37.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.585+0000 7f359cc00640 1 -- 172.21.15.90:0/930289290 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f35ac038440 con 0x55b297dddf70 2024-04-02T14:35:37.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.585+0000 7f35bce6f740 10 client.25697 fetch_fsmap learned FSMap version 27 2024-04-02T14:35:37.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.585+0000 7f35bce6f740 1 -- 172.21.15.90:0/930289290 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({fsmap=27}) -- 0x55b297e6ba20 con 0x55b297dddf70 2024-04-02T14:35:37.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.586+0000 7f359cc00640 1 -- 172.21.15.90:0/930289290 <== mon.2 v2:172.21.15.92:3300/0 4 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f35ac0194d0 con 0x55b297dddf70 2024-04-02T14:35:37.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.586+0000 7f35bce6f740 10 client.25697 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:35:37.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.586+0000 7f35bce6f740 10 client.25697 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:35:37.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.586+0000 7f35bce6f740 20 client.25697 populate_metadata read hostname 'smithi090' 2024-04-02T14:35:37.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.586+0000 7f35bce6f740 1 --2- 172.21.15.90:0/930289290 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55b297e6bfb0 0x55b297e6e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:37.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.586+0000 7f35bce6f740 4 client.25697 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-02T14:35:37.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.586+0000 7f35bce6f740 1 -- 172.21.15.90:0/930289290 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55b297d4bcb0 con 0x55b297e6bfb0 2024-04-02T14:35:37.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.587+0000 7f35b3400640 1 --2- 172.21.15.90:0/930289290 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55b297e6bfb0 0x55b297e6e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:37.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.589+0000 7f359cc00640 10 client.25697 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:35:37.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.590+0000 7f35b3400640 1 --2- 172.21.15.90:0/930289290 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55b297e6bfb0 0x55b297e6e3a0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=54b67d23decaf5e6 server_cookie=c5006bd62951086c in_seq=0 out_seq=0 2024-04-02T14:35:37.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.590+0000 7f359cc00640 1 -- 172.21.15.90:0/930289290 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55b297d4bcb0 con 0x55b297e6bfb0 2024-04-02T14:35:37.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.590+0000 7f359cc00640 10 client.25697 handle_command_reply: tid=0 2024-04-02T14:35:37.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.590+0000 7f35bce6f740 1 client.25697 shutdown 2024-04-02T14:35:37.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.591+0000 7f35bce6f740 1 -- 172.21.15.90:0/930289290 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b297dddf70 msgr2=0x55b297dde3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:35:37.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.591+0000 7f35bce6f740 1 --2- 172.21.15.90:0/930289290 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b297dddf70 0x55b297dde3f0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f35ac004600 tx=0x7f35ac00e620 comp rx=0 tx=0).stop 2024-04-02T14:35:37.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.591+0000 7f35bce6f740 1 -- 172.21.15.90:0/930289290 shutdown_connections 2024-04-02T14:35:37.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.591+0000 7f35bce6f740 1 --2- 172.21.15.90:0/930289290 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55b297e6bfb0 0x55b297e6e3a0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:37.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.591+0000 7f35bce6f740 1 --2- 172.21.15.90:0/930289290 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b297dddf70 0x55b297dde3f0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:37.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.591+0000 7f35bce6f740 1 --2- 172.21.15.90:0/930289290 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b297de3e60 0x55b297dec560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:37.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.591+0000 7f35bce6f740 1 --2- 172.21.15.90:0/930289290 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b297ddb280 0x55b297dec020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:37.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.591+0000 7f35bce6f740 1 -- 172.21.15.90:0/930289290 >> 172.21.15.90:0/930289290 conn(0x55b297dd5090 msgr2=0x55b297dd6d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:35:37.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.591+0000 7f359cc00640 0 client.25697 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:35:37.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.591+0000 7f35bce6f740 1 -- 172.21.15.90:0/930289290 shutdown_connections 2024-04-02T14:35:37.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.591+0000 7f35bce6f740 1 -- 172.21.15.90:0/930289290 wait complete. 2024-04-02T14:35:37.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.592+0000 7f35bce6f740 20 client.25697 trim_cache size 0 max 16384 2024-04-02T14:35:37.595 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-02T14:35:37.595 INFO:teuthology.orchestra.run.smithi090.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-02T14:35:37.595 INFO:teuthology.orchestra.run.smithi090.stdout: "scrubs": { 2024-04-02T14:35:37.595 INFO:teuthology.orchestra.run.smithi090.stdout: "2886f6e0-04af-43ca-9668-74cc1589d4fd": { 2024-04-02T14:35:37.595 INFO:teuthology.orchestra.run.smithi090.stdout: "path": "/", 2024-04-02T14:35:37.595 INFO:teuthology.orchestra.run.smithi090.stdout: "tag": "2886f6e0-04af-43ca-9668-74cc1589d4fd", 2024-04-02T14:35:37.595 INFO:teuthology.orchestra.run.smithi090.stdout: "options": "recursive,force" 2024-04-02T14:35:37.595 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:35:37.595 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:35:37.596 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-02T14:35:37.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.593+0000 7f35b8c00640 1 -- 172.21.15.90:0/1335092904 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f35980a3f90 msgr2=0x7f35980a6450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:35:37.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.593+0000 7f35b8c00640 1 --2- 172.21.15.90:0/1335092904 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f35980a3f90 0x7f35980a6450 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f35ac010010 tx=0x7f35ac00ed40 comp rx=0 tx=0).stop 2024-04-02T14:35:37.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.593+0000 7f35b8c00640 1 -- 172.21.15.90:0/1335092904 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f35b4102320 msgr2=0x7f35b410fa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:35:37.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.593+0000 7f35b8c00640 1 --2- 172.21.15.90:0/1335092904 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f35b4102320 0x7f35b410fa30 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f35a0000c00 tx=0x7f35a0004530 comp rx=0 tx=0).stop 2024-04-02T14:35:37.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.593+0000 7f35b8c00640 1 -- 172.21.15.90:0/1335092904 shutdown_connections 2024-04-02T14:35:37.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.593+0000 7f35b8c00640 1 --2- 172.21.15.90:0/1335092904 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f35980a3f90 0x7f35980a6450 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:37.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.593+0000 7f35b8c00640 1 --2- 172.21.15.90:0/1335092904 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f35b4107970 0x7f35b410ff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:37.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.593+0000 7f35b8c00640 1 --2- 172.21.15.90:0/1335092904 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f35b4102320 0x7f35b410fa30 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:37.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.593+0000 7f35b8c00640 1 --2- 172.21.15.90:0/1335092904 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f35b4101960 0x7f35b410f4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:37.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.593+0000 7f35b8c00640 1 -- 172.21.15.90:0/1335092904 >> 172.21.15.90:0/1335092904 conn(0x7f35b40fd470 msgr2=0x7f35b40fdd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:35:37.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.594+0000 7f35b8c00640 1 -- 172.21.15.90:0/1335092904 shutdown_connections 2024-04-02T14:35:37.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:35:37.594+0000 7f35b8c00640 1 -- 172.21.15.90:0/1335092904 wait complete. 2024-04-02T14:35:37.621 INFO:tasks.cephfs.filesystem:scrub status for tag:2886f6e0-04af-43ca-9668-74cc1589d4fd - {'path': '/', 'tag': '2886f6e0-04af-43ca-9668-74cc1589d4fd', 'options': 'recursive,force'} 2024-04-02T14:35:38.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:35:37 smithi177 ceph-mon[36431]: pgmap v25: 97 pgs: 97 active+clean; 510 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 47 MiB/s wr, 92 op/s 2024-04-02T14:35:38.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:35:37 smithi177 ceph-mon[36431]: scrub queued for path: / 2024-04-02T14:35:38.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:35:37 smithi177 ceph-mon[36431]: scrub summary: idle+waiting paths [/] 2024-04-02T14:35:38.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:35:37 smithi177 ceph-mon[36431]: scrub summary: active paths [/] 2024-04-02T14:35:38.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:35:37 smithi177 ceph-mon[36431]: scrub summary: idle+waiting paths [/] 2024-04-02T14:35:38.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:35:37 smithi090 ceph-mon[31674]: pgmap v25: 97 pgs: 97 active+clean; 510 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 47 MiB/s wr, 92 op/s 2024-04-02T14:35:38.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:35:37 smithi090 ceph-mon[31674]: scrub queued for path: / 2024-04-02T14:35:38.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:35:37 smithi090 ceph-mon[31674]: scrub summary: idle+waiting paths [/] 2024-04-02T14:35:38.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:35:37 smithi090 ceph-mon[31674]: scrub summary: active paths [/] 2024-04-02T14:35:38.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:35:37 smithi090 ceph-mon[31674]: scrub summary: idle+waiting paths [/] 2024-04-02T14:35:38.360 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:35:37 smithi092 ceph-mon[36497]: pgmap v25: 97 pgs: 97 active+clean; 510 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 47 MiB/s wr, 92 op/s 2024-04-02T14:35:38.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:35:37 smithi092 ceph-mon[36497]: scrub queued for path: / 2024-04-02T14:35:38.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:35:37 smithi092 ceph-mon[36497]: scrub summary: idle+waiting paths [/] 2024-04-02T14:35:38.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:35:37 smithi092 ceph-mon[36497]: scrub summary: active paths [/] 2024-04-02T14:35:38.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:35:37 smithi092 ceph-mon[36497]: scrub summary: idle+waiting paths [/] 2024-04-02T14:35:40.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:35:39 smithi177 ceph-mon[36431]: pgmap v26: 97 pgs: 97 active+clean; 632 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 48 MiB/s wr, 197 op/s 2024-04-02T14:35:40.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:35:39 smithi090 ceph-mon[31674]: pgmap v26: 97 pgs: 97 active+clean; 632 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 48 MiB/s wr, 197 op/s 2024-04-02T14:35:40.360 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:35:39 smithi092 ceph-mon[36497]: pgmap v26: 97 pgs: 97 active+clean; 632 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 48 MiB/s wr, 197 op/s 2024-04-02T14:35:41.831 INFO:teuthology.orchestra.run.smithi090.stderr:progress: 10.0 s, 3685.0 tps, lat 27.133 ms stddev 29.296 2024-04-02T14:35:42.360 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:35:41 smithi092 ceph-mon[36497]: pgmap v27: 97 pgs: 97 active+clean; 887 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 44 MiB/s wr, 439 op/s 2024-04-02T14:35:42.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:35:41 smithi092 ceph-mon[36497]: scrub summary: idle 2024-04-02T14:35:42.436 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:35:41 smithi177 ceph-mon[36431]: pgmap v27: 97 pgs: 97 active+clean; 887 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 44 MiB/s wr, 439 op/s 2024-04-02T14:35:42.436 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:35:41 smithi177 ceph-mon[36431]: scrub summary: idle 2024-04-02T14:35:42.438 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:35:41 smithi090 ceph-mon[31674]: pgmap v27: 97 pgs: 97 active+clean; 887 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 44 MiB/s wr, 439 op/s 2024-04-02T14:35:42.438 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:35:41 smithi090 ceph-mon[31674]: scrub summary: idle 2024-04-02T14:35:44.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:35:43 smithi092 ceph-mon[36497]: pgmap v28: 97 pgs: 97 active+clean; 887 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 21 MiB/s wr, 429 op/s 2024-04-02T14:35:44.402 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.400+0000 7f9a81e00640 1 -- 172.21.15.90:0/340500386 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9a7c109270 msgr2=0x7f9a7c1096d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:35:44.402 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.400+0000 7f9a81e00640 1 --2- 172.21.15.90:0/340500386 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9a7c109270 0x7f9a7c1096d0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f9a68009900 tx=0x7f9a6802bc40 comp rx=0 tx=0).stop 2024-04-02T14:35:44.402 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.400+0000 7f9a81e00640 1 -- 172.21.15.90:0/340500386 shutdown_connections 2024-04-02T14:35:44.403 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.401+0000 7f9a81e00640 1 --2- 172.21.15.90:0/340500386 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9a7c116b90 0x7f9a7c118f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:44.403 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.401+0000 7f9a81e00640 1 --2- 172.21.15.90:0/340500386 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9a7c109270 0x7f9a7c1096d0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:44.403 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.401+0000 7f9a81e00640 1 --2- 172.21.15.90:0/340500386 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9a7c10c200 0x7f9a7c108d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:44.403 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.401+0000 7f9a81e00640 1 -- 172.21.15.90:0/340500386 >> 172.21.15.90:0/340500386 conn(0x7f9a7c06cba0 msgr2=0x7f9a7c06cfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:35:44.404 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.401+0000 7f9a81e00640 1 -- 172.21.15.90:0/340500386 shutdown_connections 2024-04-02T14:35:44.404 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.401+0000 7f9a81e00640 1 -- 172.21.15.90:0/340500386 wait complete. 2024-04-02T14:35:44.406 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.405+0000 7f9a81e00640 1 Processor -- start 2024-04-02T14:35:44.407 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.405+0000 7f9a81e00640 1 -- start start 2024-04-02T14:35:44.408 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.406+0000 7f9a81e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9a7c109270 0x7f9a7c115180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:44.409 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.406+0000 7f9a81e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9a7c10c200 0x7f9a7c1156c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:44.409 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.406+0000 7f9a81e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9a7c116b90 0x7f9a7c1acc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:44.410 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.407+0000 7f9a7aa00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9a7c109270 0x7f9a7c115180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:44.410 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.407+0000 7f9a7aa00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9a7c109270 0x7f9a7c115180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:60514/0 (socket says 172.21.15.90:60514) 2024-04-02T14:35:44.410 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.407+0000 7f9a7b400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9a7c116b90 0x7f9a7c1acc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:44.410 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.407+0000 7f9a7b400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9a7c116b90 0x7f9a7c1acc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:49686/0 (socket says 172.21.15.90:49686) 2024-04-02T14:35:44.410 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.408+0000 7f9a81e00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f9a7c10d990 con 0x7f9a7c10c200 2024-04-02T14:35:44.410 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.408+0000 7f9a7a000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9a7c10c200 0x7f9a7c1156c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:44.411 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.408+0000 7f9a81e00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f9a7c10d810 con 0x7f9a7c109270 2024-04-02T14:35:44.411 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.408+0000 7f9a81e00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f9a7c10db10 con 0x7f9a7c116b90 2024-04-02T14:35:44.412 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.408+0000 7f9a7a000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9a7c10c200 0x7f9a7c1156c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:44856/0 (socket says 172.21.15.90:44856) 2024-04-02T14:35:44.413 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.408+0000 7f9a7a000640 1 -- 172.21.15.90:0/151645172 learned_addr learned my addr 172.21.15.90:0/151645172 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:35:44.413 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.409+0000 7f9a7b400640 1 -- 172.21.15.90:0/151645172 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9a7c109270 msgr2=0x7f9a7c115180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:35:44.413 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.409+0000 7f9a7b400640 1 --2- 172.21.15.90:0/151645172 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9a7c109270 0x7f9a7c115180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:44.413 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.409+0000 7f9a7b400640 1 -- 172.21.15.90:0/151645172 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9a7c10c200 msgr2=0x7f9a7c1156c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:35:44.413 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.409+0000 7f9a7b400640 1 --2- 172.21.15.90:0/151645172 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9a7c10c200 0x7f9a7c1156c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:44.413 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.409+0000 7f9a7b400640 1 -- 172.21.15.90:0/151645172 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a7c1ad320 con 0x7f9a7c116b90 2024-04-02T14:35:44.413 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.409+0000 7f9a7a000640 1 --2- 172.21.15.90:0/151645172 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9a7c10c200 0x7f9a7c1156c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:35:44.413 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.410+0000 7f9a7b400640 1 --2- 172.21.15.90:0/151645172 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9a7c116b90 0x7f9a7c1acc20 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f9a74005fd0 tx=0x7f9a7400dea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:35:44.414 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.410+0000 7f9a80a00640 1 -- 172.21.15.90:0/151645172 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a7400bd40 con 0x7f9a7c116b90 2024-04-02T14:35:44.416 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.411+0000 7f9a81e00640 1 -- 172.21.15.90:0/151645172 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a7c1ad610 con 0x7f9a7c116b90 2024-04-02T14:35:44.416 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.411+0000 7f9a81e00640 1 -- 172.21.15.90:0/151645172 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a7c1b3cd0 con 0x7f9a7c116b90 2024-04-02T14:35:44.416 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.411+0000 7f9a80a00640 1 -- 172.21.15.90:0/151645172 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9a74003e50 con 0x7f9a7c116b90 2024-04-02T14:35:44.416 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.411+0000 7f9a80a00640 1 -- 172.21.15.90:0/151645172 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a7400bd40 con 0x7f9a7c116b90 2024-04-02T14:35:44.419 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.418+0000 7f9a80a00640 1 -- 172.21.15.90:0/151645172 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7f9a74010460 con 0x7f9a7c116b90 2024-04-02T14:35:44.422 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.420+0000 7f9a80a00640 1 --2- 172.21.15.90:0/151645172 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f9a600a41f0 0x7f9a600a66b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:44.422 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.420+0000 7f9a80a00640 1 -- 172.21.15.90:0/151645172 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f9a740d19f0 con 0x7f9a7c116b90 2024-04-02T14:35:44.424 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.421+0000 7f9a84bbf740 1 Processor -- start 2024-04-02T14:35:44.424 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.422+0000 7f9a84bbf740 1 -- start start 2024-04-02T14:35:44.424 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.422+0000 7f9a7aa00640 1 --2- 172.21.15.90:0/151645172 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f9a600a41f0 0x7f9a600a66b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:44.424 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.422+0000 7f9a84bbf740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5646e9ca3eb0 0x5646e9ca4290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:44.425 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.422+0000 7f9a84bbf740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5646e9ca47d0 0x5646e9cace70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:44.425 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.422+0000 7f9a84bbf740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5646e9cad3b0 0x5646e9caf7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:44.425 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.422+0000 7f9a84bbf740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5646e9c08d70 con 0x5646e9ca3eb0 2024-04-02T14:35:44.425 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.422+0000 7f9a84bbf740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x5646e9bb12a0 con 0x5646e9cad3b0 2024-04-02T14:35:44.425 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.422+0000 7f9a84bbf740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5646e9c04d20 con 0x5646e9ca47d0 2024-04-02T14:35:44.426 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.423+0000 7f9a7b400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5646e9cad3b0 0x5646e9caf7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:44.426 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.423+0000 7f9a7aa00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5646e9ca3eb0 0x5646e9ca4290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:44.426 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.423+0000 7f9a7aa00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5646e9ca3eb0 0x5646e9ca4290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:44864/0 (socket says 172.21.15.90:44864) 2024-04-02T14:35:44.426 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.423+0000 7f9a7aa00640 1 -- 172.21.15.90:0/4050909635 learned_addr learned my addr 172.21.15.90:0/4050909635 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:35:44.426 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.423+0000 7f9a7a000640 1 --2- 172.21.15.90:0/4050909635 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5646e9ca47d0 0x5646e9cace70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:44.426 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.424+0000 7f9a7aa00640 1 -- 172.21.15.90:0/4050909635 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5646e9ca47d0 msgr2=0x5646e9cace70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:35:44.427 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.425+0000 7f9a7aa00640 1 --2- 172.21.15.90:0/4050909635 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5646e9ca47d0 0x5646e9cace70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:44.427 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.425+0000 7f9a7aa00640 1 -- 172.21.15.90:0/4050909635 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5646e9cad3b0 msgr2=0x5646e9caf7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:35:44.427 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.425+0000 7f9a7aa00640 1 --2- 172.21.15.90:0/4050909635 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5646e9cad3b0 0x5646e9caf7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:44.427 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.425+0000 7f9a7aa00640 1 -- 172.21.15.90:0/4050909635 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5646e9bc5840 con 0x5646e9ca3eb0 2024-04-02T14:35:44.428 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.425+0000 7f9a7b400640 1 --2- 172.21.15.90:0/4050909635 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5646e9cad3b0 0x5646e9caf7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:35:44.428 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.425+0000 7f9a7a000640 1 --2- 172.21.15.90:0/4050909635 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5646e9ca47d0 0x5646e9cace70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:35:44.428 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.426+0000 7f9a7aa00640 1 --2- 172.21.15.90:0/151645172 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f9a600a41f0 0x7f9a600a66b0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f9a6c0095f0 tx=0x7f9a6c011810 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:35:44.429 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.426+0000 7f9a7aa00640 1 --2- 172.21.15.90:0/4050909635 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5646e9ca3eb0 0x5646e9ca4290 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f9a6c009f90 tx=0x7f9a6c01c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5342d88b4655dba server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:35:44.429 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.427+0000 7f9a67e00640 1 -- 172.21.15.90:0/4050909635 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a6c017040 con 0x5646e9ca3eb0 2024-04-02T14:35:44.429 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.427+0000 7f9a67e00640 1 -- 172.21.15.90:0/4050909635 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9a6c019050 con 0x5646e9ca3eb0 2024-04-02T14:35:44.429 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.427+0000 7f9a67e00640 1 -- 172.21.15.90:0/4050909635 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a6c00e5f0 con 0x5646e9ca3eb0 2024-04-02T14:35:44.430 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.428+0000 7f9a84bbf740 1 -- 172.21.15.90:0/4050909635 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5646e9ca3eb0 msgr2=0x5646e9ca4290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:35:44.430 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.428+0000 7f9a84bbf740 1 --2- 172.21.15.90:0/4050909635 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5646e9ca3eb0 0x5646e9ca4290 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f9a6c009f90 tx=0x7f9a6c01c000 comp rx=0 tx=0).stop 2024-04-02T14:35:44.431 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.428+0000 7f9a84bbf740 1 -- 172.21.15.90:0/4050909635 shutdown_connections 2024-04-02T14:35:44.431 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.429+0000 7f9a84bbf740 1 --2- 172.21.15.90:0/4050909635 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5646e9cad3b0 0x5646e9caf7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:44.431 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.429+0000 7f9a84bbf740 1 --2- 172.21.15.90:0/4050909635 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5646e9ca47d0 0x5646e9cace70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:44.431 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.429+0000 7f9a84bbf740 1 --2- 172.21.15.90:0/4050909635 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5646e9ca3eb0 0x5646e9ca4290 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:44.431 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.429+0000 7f9a84bbf740 1 -- 172.21.15.90:0/4050909635 >> 172.21.15.90:0/4050909635 conn(0x5646e9c9e5e0 msgr2=0x5646e9c9e9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:35:44.432 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.429+0000 7f9a84bbf740 1 -- 172.21.15.90:0/4050909635 shutdown_connections 2024-04-02T14:35:44.432 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.429+0000 7f9a84bbf740 1 -- 172.21.15.90:0/4050909635 wait complete. 2024-04-02T14:35:44.432 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.430+0000 7f9a84bbf740 1 -- start start 2024-04-02T14:35:44.433 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.431+0000 7f9a84bbf740 1 Processor -- start 2024-04-02T14:35:44.434 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.432+0000 7f9a84bbf740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5646e9ca3eb0 0x5646e9caa2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:44.434 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.432+0000 7f9a84bbf740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5646e9ca47d0 0x5646e9caa800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:44.435 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.432+0000 7f9a84bbf740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5646e9cad3b0 0x5646e9caad40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:44.435 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.432+0000 7f9a84bbf740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5646e9cb01a0 con 0x5646e9cad3b0 2024-04-02T14:35:44.435 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.432+0000 7f9a84bbf740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x5646e9bc4660 con 0x5646e9ca47d0 2024-04-02T14:35:44.435 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.432+0000 7f9a84bbf740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5646e9cb0320 con 0x5646e9ca3eb0 2024-04-02T14:35:44.436 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.433+0000 7f9a7aa00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5646e9ca3eb0 0x5646e9caa2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:44.436 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.433+0000 7f9a7a000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5646e9ca47d0 0x5646e9caa800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:44.436 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.433+0000 7f9a7aa00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5646e9ca3eb0 0x5646e9caa2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:49714/0 (socket says 172.21.15.90:49714) 2024-04-02T14:35:44.436 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.433+0000 7f9a7aa00640 1 -- 172.21.15.90:0/3490693677 learned_addr learned my addr 172.21.15.90:0/3490693677 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:35:44.436 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.434+0000 7f9a7b400640 1 --2- 172.21.15.90:0/3490693677 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5646e9cad3b0 0x5646e9caad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:44.436 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.433+0000 7f9a84bbf740 10 client.0 _ll_register_callbacks cb 0x5646e9bc4050 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:35:44.437 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.434+0000 7f9a7aa00640 1 -- 172.21.15.90:0/3490693677 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5646e9ca47d0 msgr2=0x5646e9caa800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:35:44.437 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.434+0000 7f9a7aa00640 1 --2- 172.21.15.90:0/3490693677 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5646e9ca47d0 0x5646e9caa800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:44.437 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:35:43 smithi177 ceph-mon[36431]: pgmap v28: 97 pgs: 97 active+clean; 887 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 21 MiB/s wr, 429 op/s 2024-04-02T14:35:44.438 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.434+0000 7f9a7aa00640 1 -- 172.21.15.90:0/3490693677 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5646e9cad3b0 msgr2=0x5646e9caad40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:35:44.438 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.434+0000 7f9a7aa00640 1 --2- 172.21.15.90:0/3490693677 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5646e9cad3b0 0x5646e9caad40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:44.438 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.434+0000 7f9a7aa00640 1 -- 172.21.15.90:0/3490693677 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5646e9bd3e80 con 0x5646e9ca3eb0 2024-04-02T14:35:44.438 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.434+0000 7f9a7a000640 1 --2- 172.21.15.90:0/3490693677 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5646e9ca47d0 0x5646e9caa800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:35:44.439 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.434+0000 7f9a7b400640 1 --2- 172.21.15.90:0/3490693677 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5646e9cad3b0 0x5646e9caad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:35:44.439 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.435+0000 7f9a7aa00640 1 --2- 172.21.15.90:0/3490693677 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5646e9ca3eb0 0x5646e9caa2c0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f9a6c0093b0 tx=0x7f9a6c009d10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4dd2993d434bd90e server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:35:44.439 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.435+0000 7f9a64c00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:35:44.439 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.436+0000 7f9a64c00640 1 -- 172.21.15.90:0/3490693677 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a6c033060 con 0x5646e9ca3eb0 2024-04-02T14:35:44.439 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.436+0000 7f9a64c00640 1 -- 172.21.15.90:0/3490693677 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f9a6c00f690 con 0x5646e9ca3eb0 2024-04-02T14:35:44.439 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.436+0000 7f9a84bbf740 1 -- 172.21.15.90:0/3490693677 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9a6c019050 con 0x5646e9ca3eb0 2024-04-02T14:35:44.440 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:35:43 smithi090 ceph-mon[31674]: pgmap v28: 97 pgs: 97 active+clean; 887 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 21 MiB/s wr, 429 op/s 2024-04-02T14:35:44.440 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.437+0000 7f9a64c00640 1 -- 172.21.15.90:0/3490693677 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f9a6c017040 con 0x5646e9ca3eb0 2024-04-02T14:35:44.440 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.437+0000 7f9a84bbf740 10 client.25657 fetch_fsmap learned FSMap version 27 2024-04-02T14:35:44.440 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.437+0000 7f9a84bbf740 1 -- 172.21.15.90:0/3490693677 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5646e991d210 con 0x5646e9ca3eb0 2024-04-02T14:35:44.441 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.437+0000 7f9a64c00640 1 -- 172.21.15.90:0/3490693677 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9a6c019050 con 0x5646e9ca3eb0 2024-04-02T14:35:44.441 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.438+0000 7f9a64c00640 1 -- 172.21.15.90:0/3490693677 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f9a6c00f890 con 0x5646e9ca3eb0 2024-04-02T14:35:44.441 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.439+0000 7f9a84bbf740 10 client.25657 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:35:44.441 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.439+0000 7f9a84bbf740 10 client.25657 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-02T14:35:44.442 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.439+0000 7f9a84bbf740 20 client.25657 populate_metadata read hostname 'smithi090' 2024-04-02T14:35:44.442 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.439+0000 7f9a84bbf740 1 --2- 172.21.15.90:0/3490693677 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5646e9cbd4b0 0x5646e9cbf8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:44.442 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.439+0000 7f9a84bbf740 4 client.25657 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:35:44.442 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.439+0000 7f9a84bbf740 1 -- 172.21.15.90:0/3490693677 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5646e9bb8cd0 con 0x5646e9cbd4b0 2024-04-02T14:35:44.442 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.439+0000 7f9a7a000640 1 --2- 172.21.15.90:0/3490693677 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5646e9cbd4b0 0x5646e9cbf8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:44.443 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.441+0000 7f9a7a000640 1 --2- 172.21.15.90:0/3490693677 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5646e9cbd4b0 0x5646e9cbf8a0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9bc8b854be5b32e9 server_cookie=143a922851b94f9c in_seq=0 out_seq=0 2024-04-02T14:35:44.443 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.441+0000 7f9a64c00640 10 client.25657 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:35:44.444 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.442+0000 7f9a64c00640 1 -- 172.21.15.90:0/3490693677 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5646e9bb8cd0 con 0x5646e9cbd4b0 2024-04-02T14:35:44.444 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.442+0000 7f9a64c00640 10 client.25657 handle_command_reply: tid=0 2024-04-02T14:35:44.444 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.442+0000 7f9a84bbf740 1 client.25657 shutdown 2024-04-02T14:35:44.445 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.443+0000 7f9a84bbf740 1 -- 172.21.15.90:0/3490693677 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5646e9ca3eb0 msgr2=0x5646e9caa2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:35:44.445 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.443+0000 7f9a84bbf740 1 --2- 172.21.15.90:0/3490693677 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5646e9ca3eb0 0x5646e9caa2c0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f9a6c0093b0 tx=0x7f9a6c009d10 comp rx=0 tx=0).stop 2024-04-02T14:35:44.445 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.443+0000 7f9a84bbf740 1 -- 172.21.15.90:0/3490693677 shutdown_connections 2024-04-02T14:35:44.445 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.443+0000 7f9a84bbf740 1 --2- 172.21.15.90:0/3490693677 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5646e9cbd4b0 0x5646e9cbf8a0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:44.445 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.443+0000 7f9a84bbf740 1 --2- 172.21.15.90:0/3490693677 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5646e9cad3b0 0x5646e9caad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:44.445 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.443+0000 7f9a84bbf740 1 --2- 172.21.15.90:0/3490693677 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5646e9ca47d0 0x5646e9caa800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:44.445 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.443+0000 7f9a84bbf740 1 --2- 172.21.15.90:0/3490693677 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5646e9ca3eb0 0x5646e9caa2c0 secure :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f9a6c0093b0 tx=0x7f9a6c009d10 comp rx=0 tx=0).stop 2024-04-02T14:35:44.445 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.443+0000 7f9a84bbf740 1 -- 172.21.15.90:0/3490693677 >> 172.21.15.90:0/3490693677 conn(0x5646e9c9e5e0 msgr2=0x5646e9ca0330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:35:44.446 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.443+0000 7f9a64c00640 0 client.25657 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:35:44.446 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.443+0000 7f9a84bbf740 1 -- 172.21.15.90:0/3490693677 shutdown_connections 2024-04-02T14:35:44.446 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.443+0000 7f9a84bbf740 1 -- 172.21.15.90:0/3490693677 wait complete. 2024-04-02T14:35:44.446 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.444+0000 7f9a84bbf740 20 client.25657 trim_cache size 0 max 16384 2024-04-02T14:35:44.474 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.472+0000 7f9a84bbf740 1 Processor -- start 2024-04-02T14:35:44.475 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.473+0000 7f9a84bbf740 1 -- start start 2024-04-02T14:35:44.475 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.473+0000 7f9a84bbf740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5646e9ca47d0 0x5646e9cac9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:44.476 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.473+0000 7f9a84bbf740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5646e9cad3b0 0x5646e9cb9110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:44.476 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.473+0000 7f9a84bbf740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5646e9cc5100 0x5646e9cc5520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:44.476 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.473+0000 7f9a84bbf740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f9a5c00e7c0 con 0x5646e9cc5100 2024-04-02T14:35:44.476 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.473+0000 7f9a84bbf740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f9a5c010300 con 0x5646e9ca47d0 2024-04-02T14:35:44.476 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.473+0000 7f9a84bbf740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f9a5c00ee90 con 0x5646e9cad3b0 2024-04-02T14:35:44.476 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.474+0000 7f9a7a000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5646e9cad3b0 0x5646e9cb9110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:44.477 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.474+0000 7f9a7a000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5646e9cad3b0 0x5646e9cb9110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:49730/0 (socket says 172.21.15.90:49730) 2024-04-02T14:35:44.477 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.474+0000 7f9a7b400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5646e9cc5100 0x5646e9cc5520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:44.477 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.474+0000 7f9a7a000640 1 -- 172.21.15.90:0/342889617 learned_addr learned my addr 172.21.15.90:0/342889617 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:35:44.477 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.474+0000 7f9a7aa00640 1 --2- 172.21.15.90:0/342889617 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5646e9ca47d0 0x5646e9cac9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:44.477 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.474+0000 7f9a7aa00640 1 -- 172.21.15.90:0/342889617 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5646e9cad3b0 msgr2=0x5646e9cb9110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:35:44.477 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.475+0000 7f9a7aa00640 1 --2- 172.21.15.90:0/342889617 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5646e9cad3b0 0x5646e9cb9110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:44.477 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.475+0000 7f9a7aa00640 1 -- 172.21.15.90:0/342889617 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5646e9cc5100 msgr2=0x5646e9cc5520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:35:44.477 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.475+0000 7f9a7aa00640 1 --2- 172.21.15.90:0/342889617 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5646e9cc5100 0x5646e9cc5520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:44.477 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.475+0000 7f9a7aa00640 1 -- 172.21.15.90:0/342889617 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5646e99e9620 con 0x5646e9ca47d0 2024-04-02T14:35:44.477 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.475+0000 7f9a7aa00640 1 --2- 172.21.15.90:0/342889617 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5646e9ca47d0 0x5646e9cac9f0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f9a6c009a10 tx=0x7f9a6c01df00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2f8b9ba02efe9051 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:35:44.478 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.475+0000 7f9a67400640 1 -- 172.21.15.90:0/342889617 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a6c033040 con 0x5646e9ca47d0 2024-04-02T14:35:44.478 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.475+0000 7f9a67400640 1 -- 172.21.15.90:0/342889617 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9a6c026b10 con 0x5646e9ca47d0 2024-04-02T14:35:44.480 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.476+0000 7f9a67400640 1 -- 172.21.15.90:0/342889617 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a6c03d080 con 0x5646e9ca47d0 2024-04-02T14:35:44.480 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.477+0000 7f9a84bbf740 1 -- 172.21.15.90:0/342889617 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5646e9ca47d0 msgr2=0x5646e9cac9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:35:44.480 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.477+0000 7f9a84bbf740 1 --2- 172.21.15.90:0/342889617 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5646e9ca47d0 0x5646e9cac9f0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f9a6c009a10 tx=0x7f9a6c01df00 comp rx=0 tx=0).stop 2024-04-02T14:35:44.480 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.477+0000 7f9a84bbf740 1 -- 172.21.15.90:0/342889617 shutdown_connections 2024-04-02T14:35:44.480 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.477+0000 7f9a84bbf740 1 --2- 172.21.15.90:0/342889617 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5646e9cc5100 0x5646e9cc5520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:44.480 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.477+0000 7f9a84bbf740 1 --2- 172.21.15.90:0/342889617 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5646e9cad3b0 0x5646e9cb9110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:44.480 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.477+0000 7f9a84bbf740 1 --2- 172.21.15.90:0/342889617 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5646e9ca47d0 0x5646e9cac9f0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:44.480 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.477+0000 7f9a84bbf740 1 -- 172.21.15.90:0/342889617 >> 172.21.15.90:0/342889617 conn(0x5646e9c9e5e0 msgr2=0x5646e9ca02a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:35:44.480 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.478+0000 7f9a84bbf740 1 -- 172.21.15.90:0/342889617 shutdown_connections 2024-04-02T14:35:44.481 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.478+0000 7f9a84bbf740 1 -- 172.21.15.90:0/342889617 wait complete. 2024-04-02T14:35:44.481 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.478+0000 7f9a84bbf740 1 -- start start 2024-04-02T14:35:44.481 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.479+0000 7f9a84bbf740 1 Processor -- start 2024-04-02T14:35:44.482 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.480+0000 7f9a84bbf740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5646e9ca47d0 0x5646e9cb5620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:44.482 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.481+0000 7f9a84bbf740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5646e9cad3b0 0x5646e9cb5b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:44.483 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.481+0000 7f9a7aa00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5646e9ca47d0 0x5646e9cb5620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:44.483 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.481+0000 7f9a7aa00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5646e9ca47d0 0x5646e9cb5620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:60552/0 (socket says 172.21.15.90:60552) 2024-04-02T14:35:44.483 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.481+0000 7f9a7aa00640 1 -- 172.21.15.90:0/2044709450 learned_addr learned my addr 172.21.15.90:0/2044709450 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:35:44.483 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.481+0000 7f9a84bbf740 1 --2- 172.21.15.90:0/2044709450 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5646e9cc5100 0x5646e9d1ee80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:44.485 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.482+0000 7f9a7a000640 1 --2- 172.21.15.90:0/2044709450 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5646e9cad3b0 0x5646e9cb5b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:44.485 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.482+0000 7f9a84bbf740 1 -- 172.21.15.90:0/2044709450 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f9a5c00fc30 con 0x5646e9cc5100 2024-04-02T14:35:44.485 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.482+0000 7f9a84bbf740 1 -- 172.21.15.90:0/2044709450 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f9a5c00f560 con 0x5646e9ca47d0 2024-04-02T14:35:44.485 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.482+0000 7f9a84bbf740 1 -- 172.21.15.90:0/2044709450 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f9a5c00c7a0 con 0x5646e9cad3b0 2024-04-02T14:35:44.485 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.482+0000 7f9a84bbf740 10 client.0 _ll_register_callbacks cb 0x7f9a6c005670 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:35:44.485 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.482+0000 7f9a7b400640 1 --2- 172.21.15.90:0/2044709450 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5646e9cc5100 0x5646e9d1ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:44.485 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.483+0000 7f9a7aa00640 1 -- 172.21.15.90:0/2044709450 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5646e9cad3b0 msgr2=0x5646e9cb5b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:35:44.485 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.483+0000 7f9a7aa00640 1 --2- 172.21.15.90:0/2044709450 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5646e9cad3b0 0x5646e9cb5b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:44.485 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.483+0000 7f9a7aa00640 1 -- 172.21.15.90:0/2044709450 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5646e9cc5100 msgr2=0x5646e9d1ee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:35:44.485 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.483+0000 7f9a7aa00640 1 --2- 172.21.15.90:0/2044709450 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5646e9cc5100 0x5646e9d1ee80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:44.486 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.483+0000 7f9a7aa00640 1 -- 172.21.15.90:0/2044709450 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5646e9d1f6b0 con 0x5646e9ca47d0 2024-04-02T14:35:44.486 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.483+0000 7f9a7aa00640 1 --2- 172.21.15.90:0/2044709450 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5646e9ca47d0 0x5646e9cb5620 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f9a6c009350 tx=0x7f9a6c009c50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=53c7c59ff1e15cc9 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:35:44.486 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.483+0000 7f9a64c00640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:35:44.486 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.484+0000 7f9a7b400640 1 --2- 172.21.15.90:0/2044709450 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5646e9cc5100 0x5646e9d1ee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:35:44.486 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.484+0000 7f9a7a000640 1 --2- 172.21.15.90:0/2044709450 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5646e9cad3b0 0x5646e9cb5b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:35:44.487 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.484+0000 7f9a64c00640 1 -- 172.21.15.90:0/2044709450 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a6c033070 con 0x5646e9ca47d0 2024-04-02T14:35:44.487 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.484+0000 7f9a64c00640 1 -- 172.21.15.90:0/2044709450 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f9a6c005110 con 0x5646e9ca47d0 2024-04-02T14:35:44.487 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.484+0000 7f9a84bbf740 1 -- 172.21.15.90:0/2044709450 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9a6c026b10 con 0x5646e9ca47d0 2024-04-02T14:35:44.487 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.485+0000 7f9a64c00640 1 -- 172.21.15.90:0/2044709450 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f9a6c005330 con 0x5646e9ca47d0 2024-04-02T14:35:44.487 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.485+0000 7f9a84bbf740 10 client.25709 fetch_fsmap learned FSMap version 27 2024-04-02T14:35:44.487 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.485+0000 7f9a84bbf740 1 -- 172.21.15.90:0/2044709450 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5646e9d34980 con 0x5646e9ca47d0 2024-04-02T14:35:44.488 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.485+0000 7f9a64c00640 1 -- 172.21.15.90:0/2044709450 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9a6c026b10 con 0x5646e9ca47d0 2024-04-02T14:35:44.488 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.486+0000 7f9a64c00640 1 -- 172.21.15.90:0/2044709450 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f9a6c003b20 con 0x5646e9ca47d0 2024-04-02T14:35:44.488 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.486+0000 7f9a84bbf740 10 client.25709 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:35:44.488 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.486+0000 7f9a84bbf740 10 client.25709 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-02T14:35:44.488 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.486+0000 7f9a84bbf740 20 client.25709 populate_metadata read hostname 'smithi090' 2024-04-02T14:35:44.488 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.486+0000 7f9a84bbf740 1 --2- 172.21.15.90:0/2044709450 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5646e9d34f10 0x5646e9d37330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:44.488 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.486+0000 7f9a84bbf740 4 client.25709 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-02T14:35:44.489 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.486+0000 7f9a84bbf740 1 -- 172.21.15.90:0/2044709450 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5646e9c162b0 con 0x5646e9d34f10 2024-04-02T14:35:44.489 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.486+0000 7f9a7a000640 1 --2- 172.21.15.90:0/2044709450 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5646e9d34f10 0x5646e9d37330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:44.489 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.487+0000 7f9a7a000640 1 --2- 172.21.15.90:0/2044709450 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5646e9d34f10 0x5646e9d37330 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ed5508719dea4ae3 server_cookie=6a71656acbae85b6 in_seq=0 out_seq=0 2024-04-02T14:35:44.489 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.487+0000 7f9a64c00640 10 client.25709 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:35:44.512 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.509+0000 7f9a64c00640 1 -- 172.21.15.90:0/2044709450 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5646e9c162b0 con 0x5646e9d34f10 2024-04-02T14:35:44.512 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.509+0000 7f9a64c00640 10 client.25709 handle_command_reply: tid=0 2024-04-02T14:35:44.512 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.509+0000 7f9a84bbf740 1 client.25709 shutdown 2024-04-02T14:35:44.512 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.510+0000 7f9a84bbf740 1 -- 172.21.15.90:0/2044709450 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5646e9ca47d0 msgr2=0x5646e9cb5620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:35:44.512 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.510+0000 7f9a84bbf740 1 --2- 172.21.15.90:0/2044709450 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5646e9ca47d0 0x5646e9cb5620 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f9a6c009350 tx=0x7f9a6c009c50 comp rx=0 tx=0).stop 2024-04-02T14:35:44.513 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.510+0000 7f9a84bbf740 1 -- 172.21.15.90:0/2044709450 shutdown_connections 2024-04-02T14:35:44.513 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.510+0000 7f9a84bbf740 1 --2- 172.21.15.90:0/2044709450 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5646e9d34f10 0x5646e9d37330 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:44.513 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.510+0000 7f9a84bbf740 1 --2- 172.21.15.90:0/2044709450 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5646e9cc5100 0x5646e9d1ee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:44.513 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.510+0000 7f9a84bbf740 1 --2- 172.21.15.90:0/2044709450 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5646e9cad3b0 0x5646e9cb5b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:44.513 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.510+0000 7f9a84bbf740 1 --2- 172.21.15.90:0/2044709450 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5646e9ca47d0 0x5646e9cb5620 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:44.513 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.510+0000 7f9a84bbf740 1 -- 172.21.15.90:0/2044709450 >> 172.21.15.90:0/2044709450 conn(0x5646e9c9e5e0 msgr2=0x5646e9ca8a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:35:44.513 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.510+0000 7f9a64c00640 0 client.25709 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:35:44.513 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.510+0000 7f9a84bbf740 1 -- 172.21.15.90:0/2044709450 shutdown_connections 2024-04-02T14:35:44.513 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.510+0000 7f9a84bbf740 1 -- 172.21.15.90:0/2044709450 wait complete. 2024-04-02T14:35:44.516 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.514+0000 7f9a84bbf740 20 client.25709 trim_cache size 0 max 16384 2024-04-02T14:35:44.517 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:{ 2024-04-02T14:35:44.517 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "message": "", 2024-04-02T14:35:44.517 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "return_code": 0 2024-04-02T14:35:44.517 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:} 2024-04-02T14:35:44.518 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.516+0000 7f9a81e00640 1 -- 172.21.15.90:0/151645172 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f9a600a41f0 msgr2=0x7f9a600a66b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:35:44.518 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.516+0000 7f9a81e00640 1 --2- 172.21.15.90:0/151645172 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f9a600a41f0 0x7f9a600a66b0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f9a6c0095f0 tx=0x7f9a6c011810 comp rx=0 tx=0).stop 2024-04-02T14:35:44.518 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.516+0000 7f9a81e00640 1 -- 172.21.15.90:0/151645172 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9a7c116b90 msgr2=0x7f9a7c1acc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:35:44.518 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.516+0000 7f9a81e00640 1 --2- 172.21.15.90:0/151645172 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9a7c116b90 0x7f9a7c1acc20 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f9a74005fd0 tx=0x7f9a7400dea0 comp rx=0 tx=0).stop 2024-04-02T14:35:44.519 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.516+0000 7f9a81e00640 1 -- 172.21.15.90:0/151645172 shutdown_connections 2024-04-02T14:35:44.519 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.516+0000 7f9a81e00640 1 --2- 172.21.15.90:0/151645172 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f9a600a41f0 0x7f9a600a66b0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:44.519 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.516+0000 7f9a81e00640 1 --2- 172.21.15.90:0/151645172 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9a7c116b90 0x7f9a7c1acc20 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:44.519 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.516+0000 7f9a81e00640 1 --2- 172.21.15.90:0/151645172 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9a7c10c200 0x7f9a7c1156c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:44.519 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.516+0000 7f9a81e00640 1 --2- 172.21.15.90:0/151645172 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9a7c109270 0x7f9a7c115180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:44.519 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.516+0000 7f9a81e00640 1 -- 172.21.15.90:0/151645172 >> 172.21.15.90:0/151645172 conn(0x7f9a7c06cba0 msgr2=0x7f9a7c116f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:35:44.520 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.517+0000 7f9a81e00640 1 -- 172.21.15.90:0/151645172 shutdown_connections 2024-04-02T14:35:44.520 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:44.518+0000 7f9a81e00640 1 -- 172.21.15.90:0/151645172 wait complete. 2024-04-02T14:35:46.360 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:35:45 smithi092 ceph-mon[36497]: pgmap v29: 97 pgs: 97 active+clean; 925 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 26 MiB/s wr, 759 op/s 2024-04-02T14:35:46.436 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:35:45 smithi177 ceph-mon[36431]: pgmap v29: 97 pgs: 97 active+clean; 925 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 26 MiB/s wr, 759 op/s 2024-04-02T14:35:46.438 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:35:45 smithi090 ceph-mon[31674]: pgmap v29: 97 pgs: 97 active+clean; 925 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 26 MiB/s wr, 759 op/s 2024-04-02T14:35:46.831 INFO:teuthology.orchestra.run.smithi090.stderr:progress: 15.0 s, 4046.2 tps, lat 24.662 ms stddev 25.527 2024-04-02T14:35:48.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:35:47 smithi092 ceph-mon[36497]: pgmap v30: 97 pgs: 97 active+clean; 925 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 7.8 MiB/s wr, 689 op/s 2024-04-02T14:35:48.437 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:35:47 smithi177 ceph-mon[36431]: pgmap v30: 97 pgs: 97 active+clean; 925 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 7.8 MiB/s wr, 689 op/s 2024-04-02T14:35:48.438 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:35:47 smithi090 ceph-mon[31674]: pgmap v30: 97 pgs: 97 active+clean; 925 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 7.8 MiB/s wr, 689 op/s 2024-04-02T14:35:50.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:35:49 smithi092 ceph-mon[36497]: pgmap v31: 97 pgs: 97 active+clean; 930 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 10 MiB/s wr, 922 op/s 2024-04-02T14:35:50.436 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:35:49 smithi177 ceph-mon[36431]: pgmap v31: 97 pgs: 97 active+clean; 930 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 10 MiB/s wr, 922 op/s 2024-04-02T14:35:50.438 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:35:49 smithi090 ceph-mon[31674]: pgmap v31: 97 pgs: 97 active+clean; 930 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 10 MiB/s wr, 922 op/s 2024-04-02T14:35:51.832 INFO:teuthology.orchestra.run.smithi090.stderr:progress: 20.0 s, 3584.7 tps, lat 27.973 ms stddev 31.415 2024-04-02T14:35:52.238 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:35:51 smithi177 ceph-mon[36431]: pgmap v32: 97 pgs: 97 active+clean; 934 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 11 MiB/s wr, 940 op/s 2024-04-02T14:35:52.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:35:51 smithi092 ceph-mon[36497]: pgmap v32: 97 pgs: 97 active+clean; 934 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 11 MiB/s wr, 940 op/s 2024-04-02T14:35:52.438 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:35:51 smithi090 ceph-mon[31674]: pgmap v32: 97 pgs: 97 active+clean; 934 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 11 MiB/s wr, 940 op/s 2024-04-02T14:35:54.360 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:35:53 smithi092 ceph-mon[36497]: pgmap v33: 97 pgs: 97 active+clean; 934 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 8.2 MiB/s wr, 686 op/s 2024-04-02T14:35:54.436 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:35:53 smithi177 ceph-mon[36431]: pgmap v33: 97 pgs: 97 active+clean; 934 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 8.2 MiB/s wr, 686 op/s 2024-04-02T14:35:54.438 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:35:53 smithi090 ceph-mon[31674]: pgmap v33: 97 pgs: 97 active+clean; 934 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 8.2 MiB/s wr, 686 op/s 2024-04-02T14:35:56.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:35:55 smithi092 ceph-mon[36497]: pgmap v34: 97 pgs: 97 active+clean; 956 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 14 MiB/s wr, 1.04k op/s 2024-04-02T14:35:56.436 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:35:55 smithi177 ceph-mon[36431]: pgmap v34: 97 pgs: 97 active+clean; 956 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 14 MiB/s wr, 1.04k op/s 2024-04-02T14:35:56.438 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:35:55 smithi090 ceph-mon[31674]: pgmap v34: 97 pgs: 97 active+clean; 956 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 14 MiB/s wr, 1.04k op/s 2024-04-02T14:35:56.831 INFO:teuthology.orchestra.run.smithi090.stderr:progress: 25.0 s, 3842.5 tps, lat 25.846 ms stddev 27.165 2024-04-02T14:35:57.691 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.687+0000 7f75c8c00640 1 -- 172.21.15.90:0/1588561978 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f75c40696c0 msgr2=0x7f75c4069b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:35:57.691 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.687+0000 7f75c8c00640 1 --2- 172.21.15.90:0/1588561978 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f75c40696c0 0x7f75c4069b00 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f75b0009900 tx=0x7f75b002bcc0 comp rx=0 tx=0).stop 2024-04-02T14:35:57.691 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.687+0000 7f75c8c00640 1 -- 172.21.15.90:0/1588561978 shutdown_connections 2024-04-02T14:35:57.691 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.688+0000 7f75c8c00640 1 --2- 172.21.15.90:0/1588561978 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f75c410e350 0x7f75c4110760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:57.691 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.688+0000 7f75c8c00640 1 --2- 172.21.15.90:0/1588561978 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f75c40696c0 0x7f75c4069b00 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:57.691 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.688+0000 7f75c8c00640 1 --2- 172.21.15.90:0/1588561978 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f75c41029e0 0x7f75c4069180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:57.691 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.688+0000 7f75c8c00640 1 -- 172.21.15.90:0/1588561978 >> 172.21.15.90:0/1588561978 conn(0x7f75c4077850 msgr2=0x7f75c4077c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:35:57.691 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.688+0000 7f75c8c00640 1 -- 172.21.15.90:0/1588561978 shutdown_connections 2024-04-02T14:35:57.691 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.688+0000 7f75c8c00640 1 -- 172.21.15.90:0/1588561978 wait complete. 2024-04-02T14:35:57.693 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.689+0000 7f75c8c00640 1 Processor -- start 2024-04-02T14:35:57.693 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.689+0000 7f75c8c00640 1 -- start start 2024-04-02T14:35:57.693 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.690+0000 7f75c8c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f75c40696c0 0x7f75c410bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:57.693 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.690+0000 7f75c8c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f75c41029e0 0x7f75c410c4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:57.693 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.690+0000 7f75c8c00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f75c410e350 0x7f75c410ca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:57.694 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.690+0000 7f75c8c00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f75c41038b0 con 0x7f75c40696c0 2024-04-02T14:35:57.694 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.690+0000 7f75c8c00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f75c4103730 con 0x7f75c410e350 2024-04-02T14:35:57.694 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.690+0000 7f75c8c00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f75c4103a30 con 0x7f75c41029e0 2024-04-02T14:35:57.694 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.690+0000 7f75c1400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f75c41029e0 0x7f75c410c4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:57.694 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.690+0000 7f75c2800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f75c410e350 0x7f75c410ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:57.694 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.690+0000 7f75c1400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f75c41029e0 0x7f75c410c4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:40364/0 (socket says 172.21.15.90:40364) 2024-04-02T14:35:57.694 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.690+0000 7f75c2800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f75c410e350 0x7f75c410ca20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:44128/0 (socket says 172.21.15.90:44128) 2024-04-02T14:35:57.694 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.690+0000 7f75c1400640 1 -- 172.21.15.90:0/4180406180 learned_addr learned my addr 172.21.15.90:0/4180406180 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:35:57.694 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.690+0000 7f75c1e00640 1 --2- 172.21.15.90:0/4180406180 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f75c40696c0 0x7f75c410bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:57.694 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.691+0000 7f75c1400640 1 -- 172.21.15.90:0/4180406180 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f75c410e350 msgr2=0x7f75c410ca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:35:57.694 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.691+0000 7f75c1400640 1 --2- 172.21.15.90:0/4180406180 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f75c410e350 0x7f75c410ca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:57.695 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.691+0000 7f75c1400640 1 -- 172.21.15.90:0/4180406180 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f75c40696c0 msgr2=0x7f75c410bfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:35:57.695 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.691+0000 7f75c1400640 1 --2- 172.21.15.90:0/4180406180 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f75c40696c0 0x7f75c410bfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:57.695 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.691+0000 7f75c1400640 1 -- 172.21.15.90:0/4180406180 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75c41aad30 con 0x7f75c41029e0 2024-04-02T14:35:57.695 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.691+0000 7f75c1e00640 1 --2- 172.21.15.90:0/4180406180 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f75c40696c0 0x7f75c410bfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:35:57.695 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.691+0000 7f75c1400640 1 --2- 172.21.15.90:0/4180406180 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f75c41029e0 0x7f75c410c4e0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f75b002c2d0 tx=0x7f75b00043c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:35:57.695 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.691+0000 7f75baa00640 1 -- 172.21.15.90:0/4180406180 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75b0041020 con 0x7f75c41029e0 2024-04-02T14:35:57.696 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.691+0000 7f75baa00640 1 -- 172.21.15.90:0/4180406180 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f75b0005000 con 0x7f75c41029e0 2024-04-02T14:35:57.696 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.691+0000 7f75baa00640 1 -- 172.21.15.90:0/4180406180 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75b0003b60 con 0x7f75c41029e0 2024-04-02T14:35:57.696 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.692+0000 7f75c8c00640 1 -- 172.21.15.90:0/4180406180 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75c41aafc0 con 0x7f75c41029e0 2024-04-02T14:35:57.696 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.692+0000 7f75c8c00640 1 -- 172.21.15.90:0/4180406180 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75c41ab370 con 0x7f75c41029e0 2024-04-02T14:35:57.697 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.695+0000 7f75baa00640 1 -- 172.21.15.90:0/4180406180 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7f75b0044020 con 0x7f75c41029e0 2024-04-02T14:35:57.698 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.696+0000 7f75cb9b2740 1 Processor -- start 2024-04-02T14:35:57.698 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.696+0000 7f75cb9b2740 1 -- start start 2024-04-02T14:35:57.698 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.696+0000 7f75cb9b2740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fca4316eb0 0x55fca4317290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:57.699 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.696+0000 7f75cb9b2740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fca43177d0 0x55fca431fe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:57.699 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.696+0000 7f75cb9b2740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fca43203b0 0x55fca43227a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:57.699 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.696+0000 7f75cb9b2740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55fca427bd70 con 0x55fca4316eb0 2024-04-02T14:35:57.699 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.696+0000 7f75cb9b2740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55fca42242a0 con 0x55fca43177d0 2024-04-02T14:35:57.699 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.696+0000 7f75cb9b2740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55fca4277d20 con 0x55fca43203b0 2024-04-02T14:35:57.700 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.697+0000 7f75c1400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fca43203b0 0x55fca43227a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:57.700 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.697+0000 7f75c1e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fca43177d0 0x55fca431fe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:57.700 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.697+0000 7f75c1400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fca43203b0 0x55fca43227a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:40376/0 (socket says 172.21.15.90:40376) 2024-04-02T14:35:57.700 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.697+0000 7f75c1400640 1 -- 172.21.15.90:0/2064061980 learned_addr learned my addr 172.21.15.90:0/2064061980 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:35:57.700 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.697+0000 7f75c2800640 1 --2- 172.21.15.90:0/2064061980 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fca4316eb0 0x55fca4317290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:57.700 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.697+0000 7f75baa00640 1 --2- 172.21.15.90:0/4180406180 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f75a80a3f60 0x7f75a80a6420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:57.700 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.697+0000 7f75c1e00640 1 -- 172.21.15.90:0/2064061980 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fca43203b0 msgr2=0x55fca43227a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:35:57.700 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.697+0000 7f75c1e00640 1 --2- 172.21.15.90:0/2064061980 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fca43203b0 0x55fca43227a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:57.700 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.697+0000 7f75c1e00640 1 -- 172.21.15.90:0/2064061980 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fca4316eb0 msgr2=0x55fca4317290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:35:57.700 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.697+0000 7f75c1e00640 1 --2- 172.21.15.90:0/2064061980 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fca4316eb0 0x55fca4317290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:57.700 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.697+0000 7f75c1e00640 1 -- 172.21.15.90:0/2064061980 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fca4238840 con 0x55fca43177d0 2024-04-02T14:35:57.700 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.697+0000 7f75c1400640 1 --2- 172.21.15.90:0/2064061980 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fca43203b0 0x55fca43227a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:35:57.701 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.697+0000 7f75c2800640 1 --2- 172.21.15.90:0/2064061980 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fca4316eb0 0x55fca4317290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:35:57.701 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.697+0000 7f75baa00640 1 -- 172.21.15.90:0/4180406180 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f75b00f22b0 con 0x7f75c41029e0 2024-04-02T14:35:57.701 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.697+0000 7f75c1e00640 1 --2- 172.21.15.90:0/2064061980 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fca43177d0 0x55fca431fe70 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f75b400bab0 tx=0x7f75b400bf70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a458e5377f005e50 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:35:57.701 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.697+0000 7f75c2800640 1 --2- 172.21.15.90:0/4180406180 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f75a80a3f60 0x7f75a80a6420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:57.706 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.698+0000 7f75af000640 1 -- 172.21.15.90:0/2064061980 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75b4017420 con 0x55fca43177d0 2024-04-02T14:35:57.706 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.698+0000 7f75af000640 1 -- 172.21.15.90:0/2064061980 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f75b4017aa0 con 0x55fca43177d0 2024-04-02T14:35:57.706 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.698+0000 7f75af000640 1 -- 172.21.15.90:0/2064061980 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75b400fc00 con 0x55fca43177d0 2024-04-02T14:35:57.706 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.698+0000 7f75cb9b2740 1 -- 172.21.15.90:0/2064061980 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fca43177d0 msgr2=0x55fca431fe70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:35:57.706 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.698+0000 7f75cb9b2740 1 --2- 172.21.15.90:0/2064061980 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fca43177d0 0x55fca431fe70 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f75b400bab0 tx=0x7f75b400bf70 comp rx=0 tx=0).stop 2024-04-02T14:35:57.706 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.698+0000 7f75c2800640 1 --2- 172.21.15.90:0/4180406180 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f75a80a3f60 0x7f75a80a6420 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f75bc005c40 tx=0x7f75bc004a40 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:35:57.706 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.698+0000 7f75cb9b2740 1 -- 172.21.15.90:0/2064061980 shutdown_connections 2024-04-02T14:35:57.706 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.698+0000 7f75cb9b2740 1 --2- 172.21.15.90:0/2064061980 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fca43203b0 0x55fca43227a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:57.707 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.698+0000 7f75cb9b2740 1 --2- 172.21.15.90:0/2064061980 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fca43177d0 0x55fca431fe70 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:57.707 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.698+0000 7f75cb9b2740 1 --2- 172.21.15.90:0/2064061980 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fca4316eb0 0x55fca4317290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:57.707 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.698+0000 7f75cb9b2740 1 -- 172.21.15.90:0/2064061980 >> 172.21.15.90:0/2064061980 conn(0x55fca43115e0 msgr2=0x55fca43119c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:35:57.707 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.699+0000 7f75cb9b2740 1 -- 172.21.15.90:0/2064061980 shutdown_connections 2024-04-02T14:35:57.707 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.699+0000 7f75cb9b2740 1 -- 172.21.15.90:0/2064061980 wait complete. 2024-04-02T14:35:57.707 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.699+0000 7f75cb9b2740 1 -- start start 2024-04-02T14:35:57.707 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.699+0000 7f75cb9b2740 1 Processor -- start 2024-04-02T14:35:57.707 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.700+0000 7f75cb9b2740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fca4316eb0 0x55fca431d570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:57.707 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.700+0000 7f75cb9b2740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fca43203b0 0x55fca431dab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:57.707 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.700+0000 7f75cb9b2740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fca431dff0 0x55fca432c610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:57.707 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.700+0000 7f75cb9b2740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55fca43231a0 con 0x55fca4316eb0 2024-04-02T14:35:57.708 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.700+0000 7f75cb9b2740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55fca4237660 con 0x55fca431dff0 2024-04-02T14:35:57.708 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.700+0000 7f75cb9b2740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55fca4323320 con 0x55fca43203b0 2024-04-02T14:35:57.708 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.700+0000 7f75cb9b2740 10 client.0 _ll_register_callbacks cb 0x55fca4237050 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:35:57.708 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.700+0000 7f75c1400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fca43203b0 0x55fca431dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:57.708 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.700+0000 7f75c1e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fca4316eb0 0x55fca431d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:57.708 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.700+0000 7f75c1400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fca43203b0 0x55fca431dab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:40388/0 (socket says 172.21.15.90:40388) 2024-04-02T14:35:57.708 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.700+0000 7f75c1400640 1 -- 172.21.15.90:0/2623293389 learned_addr learned my addr 172.21.15.90:0/2623293389 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:35:57.708 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.700+0000 7f75c2800640 1 --2- 172.21.15.90:0/2623293389 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fca431dff0 0x55fca432c610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:57.708 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.700+0000 7f75c1400640 1 -- 172.21.15.90:0/2623293389 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fca431dff0 msgr2=0x55fca432c610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:35:57.708 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.700+0000 7f75c1400640 1 --2- 172.21.15.90:0/2623293389 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fca431dff0 0x55fca432c610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:57.708 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.700+0000 7f75c1400640 1 -- 172.21.15.90:0/2623293389 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fca4316eb0 msgr2=0x55fca431d570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:35:57.709 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.700+0000 7f75c1400640 1 --2- 172.21.15.90:0/2623293389 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fca4316eb0 0x55fca431d570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:57.709 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.700+0000 7f75c1400640 1 -- 172.21.15.90:0/2623293389 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fca4246e80 con 0x55fca43203b0 2024-04-02T14:35:57.709 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.700+0000 7f75c1400640 1 --2- 172.21.15.90:0/2623293389 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fca43203b0 0x55fca431dab0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f75b00c68d0 tx=0x7f75b00bdf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=132200f7ee5384f7 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:35:57.709 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.700+0000 7f75a3400640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:35:57.709 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.701+0000 7f75a3400640 1 -- 172.21.15.90:0/2623293389 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75b0004c80 con 0x55fca43203b0 2024-04-02T14:35:57.709 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.701+0000 7f75a3400640 1 -- 172.21.15.90:0/2623293389 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f75b00ba800 con 0x55fca43203b0 2024-04-02T14:35:57.709 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.701+0000 7f75cb9b2740 1 -- 172.21.15.90:0/2623293389 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f75b4017aa0 con 0x55fca43203b0 2024-04-02T14:35:57.709 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.701+0000 7f75a3400640 1 -- 172.21.15.90:0/2623293389 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f75b00be890 con 0x55fca43203b0 2024-04-02T14:35:57.710 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.701+0000 7f75cb9b2740 10 client.25675 fetch_fsmap learned FSMap version 27 2024-04-02T14:35:57.710 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.701+0000 7f75cb9b2740 1 -- 172.21.15.90:0/2623293389 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55fca3f90210 con 0x55fca43203b0 2024-04-02T14:35:57.710 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.701+0000 7f75a3400640 1 -- 172.21.15.90:0/2623293389 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f75b4017aa0 con 0x55fca43203b0 2024-04-02T14:35:57.710 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.701+0000 7f75a3400640 1 -- 172.21.15.90:0/2623293389 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f75b00051b0 con 0x55fca43203b0 2024-04-02T14:35:57.710 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.701+0000 7f75cb9b2740 10 client.25675 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:35:57.710 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.701+0000 7f75cb9b2740 10 client.25675 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-02T14:35:57.710 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.701+0000 7f75cb9b2740 20 client.25675 populate_metadata read hostname 'smithi090' 2024-04-02T14:35:57.710 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.701+0000 7f75cb9b2740 1 --2- 172.21.15.90:0/2623293389 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55fca4330c80 0x55fca43330c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:57.710 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.701+0000 7f75cb9b2740 4 client.25675 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:35:57.710 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.701+0000 7f75cb9b2740 1 -- 172.21.15.90:0/2623293389 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fca422bcd0 con 0x55fca4330c80 2024-04-02T14:35:57.710 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.702+0000 7f75c1e00640 1 --2- 172.21.15.90:0/2623293389 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55fca4330c80 0x55fca43330c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:57.710 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.702+0000 7f75a3400640 10 client.25675 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:35:57.711 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.702+0000 7f75c1e00640 1 --2- 172.21.15.90:0/2623293389 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55fca4330c80 0x55fca43330c0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1e0900b349f66ed9 server_cookie=5b9cff5052096357 in_seq=0 out_seq=0 2024-04-02T14:35:57.711 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.703+0000 7f75a3400640 1 -- 172.21.15.90:0/2623293389 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55fca422bcd0 con 0x55fca4330c80 2024-04-02T14:35:57.711 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.703+0000 7f75a3400640 10 client.25675 handle_command_reply: tid=0 2024-04-02T14:35:57.711 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.703+0000 7f75cb9b2740 1 client.25675 shutdown 2024-04-02T14:35:57.711 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.703+0000 7f75cb9b2740 1 -- 172.21.15.90:0/2623293389 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fca43203b0 msgr2=0x55fca431dab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:35:57.711 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.703+0000 7f75cb9b2740 1 --2- 172.21.15.90:0/2623293389 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fca43203b0 0x55fca431dab0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f75b00c68d0 tx=0x7f75b00bdf60 comp rx=0 tx=0).stop 2024-04-02T14:35:57.711 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.703+0000 7f75cb9b2740 1 -- 172.21.15.90:0/2623293389 shutdown_connections 2024-04-02T14:35:57.711 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.703+0000 7f75cb9b2740 1 --2- 172.21.15.90:0/2623293389 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55fca4330c80 0x55fca43330c0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:57.711 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.703+0000 7f75cb9b2740 1 --2- 172.21.15.90:0/2623293389 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fca431dff0 0x55fca432c610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:57.711 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.703+0000 7f75cb9b2740 1 --2- 172.21.15.90:0/2623293389 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fca43203b0 0x55fca431dab0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:57.711 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.703+0000 7f75cb9b2740 1 --2- 172.21.15.90:0/2623293389 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fca4316eb0 0x55fca431d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:57.711 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.703+0000 7f75cb9b2740 1 -- 172.21.15.90:0/2623293389 >> 172.21.15.90:0/2623293389 conn(0x55fca43115e0 msgr2=0x55fca4313330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:35:57.711 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.703+0000 7f75a3400640 0 client.25675 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:35:57.711 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.703+0000 7f75cb9b2740 1 -- 172.21.15.90:0/2623293389 shutdown_connections 2024-04-02T14:35:57.712 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.704+0000 7f75cb9b2740 1 -- 172.21.15.90:0/2623293389 wait complete. 2024-04-02T14:35:57.712 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.704+0000 7f75cb9b2740 20 client.25675 trim_cache size 0 max 16384 2024-04-02T14:35:57.731 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.729+0000 7f75cb9b2740 1 Processor -- start 2024-04-02T14:35:57.732 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.729+0000 7f75cb9b2740 1 -- start start 2024-04-02T14:35:57.732 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.730+0000 7f75cb9b2740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fca4316eb0 0x55fca4313120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:57.732 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.730+0000 7f75cb9b2740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fca431dff0 0x55fca4338990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:57.733 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.730+0000 7f75cb9b2740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fca43203b0 0x55fca4338ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:57.733 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.730+0000 7f75cb9b2740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f75c41b3d40 con 0x55fca43203b0 2024-04-02T14:35:57.733 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.730+0000 7f75cb9b2740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f75c41b5880 con 0x55fca4316eb0 2024-04-02T14:35:57.733 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.730+0000 7f75cb9b2740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f75c41b4410 con 0x55fca431dff0 2024-04-02T14:35:57.733 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.730+0000 7f75c1400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fca431dff0 0x55fca4338990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:57.733 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.730+0000 7f75c2800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fca43203b0 0x55fca4338ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:57.733 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.730+0000 7f75c1400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fca431dff0 0x55fca4338990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:40394/0 (socket says 172.21.15.90:40394) 2024-04-02T14:35:57.733 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.731+0000 7f75c1400640 1 -- 172.21.15.90:0/301999923 learned_addr learned my addr 172.21.15.90:0/301999923 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:35:57.733 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.731+0000 7f75c1400640 1 -- 172.21.15.90:0/301999923 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fca4316eb0 msgr2=0x55fca4313120 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-02T14:35:57.734 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.731+0000 7f75c1400640 1 --2- 172.21.15.90:0/301999923 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fca4316eb0 0x55fca4313120 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:57.734 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.731+0000 7f75c1400640 1 -- 172.21.15.90:0/301999923 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fca43203b0 msgr2=0x55fca4338ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:35:57.734 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.731+0000 7f75c1400640 1 --2- 172.21.15.90:0/301999923 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fca43203b0 0x55fca4338ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:57.734 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.731+0000 7f75c1400640 1 -- 172.21.15.90:0/301999923 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fca4322b10 con 0x55fca431dff0 2024-04-02T14:35:57.734 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.731+0000 7f75c2800640 1 --2- 172.21.15.90:0/301999923 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fca43203b0 0x55fca4338ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:35:57.734 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.732+0000 7f75c1400640 1 --2- 172.21.15.90:0/301999923 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fca431dff0 0x55fca4338990 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f75b00bab60 tx=0x7f75b00bc6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1f67d5a53657d435 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:35:57.734 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.732+0000 7f75ae600640 1 -- 172.21.15.90:0/301999923 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75b0004c80 con 0x55fca431dff0 2024-04-02T14:35:57.734 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.732+0000 7f75ae600640 1 -- 172.21.15.90:0/301999923 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f75b00f5a30 con 0x55fca431dff0 2024-04-02T14:35:57.735 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.732+0000 7f75ae600640 1 -- 172.21.15.90:0/301999923 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75b00051b0 con 0x55fca431dff0 2024-04-02T14:35:57.736 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.733+0000 7f75cb9b2740 1 -- 172.21.15.90:0/301999923 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fca431dff0 msgr2=0x55fca4338990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:35:57.736 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.733+0000 7f75cb9b2740 1 --2- 172.21.15.90:0/301999923 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fca431dff0 0x55fca4338990 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f75b00bab60 tx=0x7f75b00bc6a0 comp rx=0 tx=0).stop 2024-04-02T14:35:57.736 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.733+0000 7f75cb9b2740 1 -- 172.21.15.90:0/301999923 shutdown_connections 2024-04-02T14:35:57.736 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.733+0000 7f75cb9b2740 1 --2- 172.21.15.90:0/301999923 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fca43203b0 0x55fca4338ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:57.736 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.733+0000 7f75cb9b2740 1 --2- 172.21.15.90:0/301999923 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fca431dff0 0x55fca4338990 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:57.736 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.733+0000 7f75cb9b2740 1 --2- 172.21.15.90:0/301999923 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fca4316eb0 0x55fca4313120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:57.736 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.734+0000 7f75cb9b2740 1 -- 172.21.15.90:0/301999923 >> 172.21.15.90:0/301999923 conn(0x55fca43115e0 msgr2=0x55fca4312a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:35:57.736 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.734+0000 7f75cb9b2740 1 -- 172.21.15.90:0/301999923 shutdown_connections 2024-04-02T14:35:57.737 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.734+0000 7f75cb9b2740 1 -- 172.21.15.90:0/301999923 wait complete. 2024-04-02T14:35:57.738 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.735+0000 7f75cb9b2740 1 -- start start 2024-04-02T14:35:57.738 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.735+0000 7f75cb9b2740 1 Processor -- start 2024-04-02T14:35:57.738 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.736+0000 7f75cb9b2740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fca4316eb0 0x55fca4328570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:57.738 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.736+0000 7f75cb9b2740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fca431dff0 0x55fca4328ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:57.739 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.736+0000 7f75cb9b2740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fca43203b0 0x55fca4325540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:57.739 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.736+0000 7f75cb9b2740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f75c41b51b0 con 0x55fca431dff0 2024-04-02T14:35:57.739 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.736+0000 7f75cb9b2740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f75c41b4ae0 con 0x55fca43203b0 2024-04-02T14:35:57.739 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.736+0000 7f75cb9b2740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f75c41b1d20 con 0x55fca4316eb0 2024-04-02T14:35:57.739 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.736+0000 7f75cb9b2740 10 client.0 _ll_register_callbacks cb 0x7f75b0005b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:35:57.739 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.736+0000 7f75c2800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fca43203b0 0x55fca4325540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:57.739 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.736+0000 7f75c2800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fca43203b0 0x55fca4325540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:44160/0 (socket says 172.21.15.90:44160) 2024-04-02T14:35:57.739 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.736+0000 7f75c2800640 1 -- 172.21.15.90:0/640854306 learned_addr learned my addr 172.21.15.90:0/640854306 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:35:57.739 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.736+0000 7f75c1e00640 1 --2- 172.21.15.90:0/640854306 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fca4316eb0 0x55fca4328570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:57.739 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.736+0000 7f75c1400640 1 --2- 172.21.15.90:0/640854306 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fca431dff0 0x55fca4328ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:57.739 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.737+0000 7f75c2800640 1 -- 172.21.15.90:0/640854306 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fca4316eb0 msgr2=0x55fca4328570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:35:57.739 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.737+0000 7f75c2800640 1 --2- 172.21.15.90:0/640854306 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fca4316eb0 0x55fca4328570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:57.739 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.737+0000 7f75c2800640 1 -- 172.21.15.90:0/640854306 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fca431dff0 msgr2=0x55fca4328ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:35:57.741 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.737+0000 7f75c2800640 1 --2- 172.21.15.90:0/640854306 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fca431dff0 0x55fca4328ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:57.741 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.737+0000 7f75c2800640 1 -- 172.21.15.90:0/640854306 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fca405c620 con 0x55fca43203b0 2024-04-02T14:35:57.741 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.737+0000 7f75c1400640 1 --2- 172.21.15.90:0/640854306 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fca431dff0 0x55fca4328ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:35:57.741 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.737+0000 7f75c2800640 1 --2- 172.21.15.90:0/640854306 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fca43203b0 0x55fca4325540 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f75bc010bf0 tx=0x7f75bc004ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3d7ae36127e1610c server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:35:57.741 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.737+0000 7f75a3400640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:35:57.741 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.738+0000 7f75a3400640 1 -- 172.21.15.90:0/640854306 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75bc02d060 con 0x55fca43203b0 2024-04-02T14:35:57.741 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.738+0000 7f75a3400640 1 -- 172.21.15.90:0/640854306 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f75bc01eb30 con 0x55fca43203b0 2024-04-02T14:35:57.741 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.738+0000 7f75cb9b2740 1 -- 172.21.15.90:0/640854306 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f75b00f5a30 con 0x55fca43203b0 2024-04-02T14:35:57.741 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.738+0000 7f75a3400640 1 -- 172.21.15.90:0/640854306 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f75bc01ed30 con 0x55fca43203b0 2024-04-02T14:35:57.741 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.738+0000 7f75cb9b2740 10 client.25724 fetch_fsmap learned FSMap version 27 2024-04-02T14:35:57.741 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.738+0000 7f75cb9b2740 1 -- 172.21.15.90:0/640854306 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55fca43979e0 con 0x55fca43203b0 2024-04-02T14:35:57.741 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.739+0000 7f75a3400640 1 -- 172.21.15.90:0/640854306 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f75b00f5a30 con 0x55fca43203b0 2024-04-02T14:35:57.742 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.739+0000 7f75a3400640 1 -- 172.21.15.90:0/640854306 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f75bc027030 con 0x55fca43203b0 2024-04-02T14:35:57.742 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.740+0000 7f75cb9b2740 10 client.25724 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:35:57.742 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.740+0000 7f75cb9b2740 10 client.25724 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-02T14:35:57.742 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.740+0000 7f75cb9b2740 20 client.25724 populate_metadata read hostname 'smithi090' 2024-04-02T14:35:57.742 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.740+0000 7f75cb9b2740 1 --2- 172.21.15.90:0/640854306 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55fca4398110 0x55fca439a530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:35:57.742 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.740+0000 7f75cb9b2740 4 client.25724 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-02T14:35:57.742 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.740+0000 7f75cb9b2740 1 -- 172.21.15.90:0/640854306 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55fca42892b0 con 0x55fca4398110 2024-04-02T14:35:57.743 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.741+0000 7f75c1e00640 1 --2- 172.21.15.90:0/640854306 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55fca4398110 0x55fca439a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:35:57.745 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.743+0000 7f75c1e00640 1 --2- 172.21.15.90:0/640854306 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55fca4398110 0x55fca439a530 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d97862827738b0c8 server_cookie=39d71e78ccc7789b in_seq=0 out_seq=0 2024-04-02T14:35:57.746 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.743+0000 7f75a3400640 10 client.25724 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:35:57.764 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.762+0000 7f75a3400640 1 -- 172.21.15.90:0/640854306 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55fca42892b0 con 0x55fca4398110 2024-04-02T14:35:57.764 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.762+0000 7f75a3400640 10 client.25724 handle_command_reply: tid=0 2024-04-02T14:35:57.764 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.762+0000 7f75cb9b2740 1 client.25724 shutdown 2024-04-02T14:35:57.765 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.762+0000 7f75cb9b2740 1 -- 172.21.15.90:0/640854306 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fca43203b0 msgr2=0x55fca4325540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:35:57.765 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.762+0000 7f75cb9b2740 1 --2- 172.21.15.90:0/640854306 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fca43203b0 0x55fca4325540 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f75bc010bf0 tx=0x7f75bc004ab0 comp rx=0 tx=0).stop 2024-04-02T14:35:57.765 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.763+0000 7f75cb9b2740 1 -- 172.21.15.90:0/640854306 shutdown_connections 2024-04-02T14:35:57.765 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.763+0000 7f75cb9b2740 1 --2- 172.21.15.90:0/640854306 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55fca4398110 0x55fca439a530 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:57.765 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.763+0000 7f75cb9b2740 1 --2- 172.21.15.90:0/640854306 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fca43203b0 0x55fca4325540 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:57.766 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.763+0000 7f75a3400640 0 client.25724 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:35:57.766 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.763+0000 7f75cb9b2740 1 --2- 172.21.15.90:0/640854306 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fca431dff0 0x55fca4328ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:57.766 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.763+0000 7f75cb9b2740 1 --2- 172.21.15.90:0/640854306 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fca4316eb0 0x55fca4328570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:57.766 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.763+0000 7f75cb9b2740 1 -- 172.21.15.90:0/640854306 >> 172.21.15.90:0/640854306 conn(0x55fca43115e0 msgr2=0x55fca43132a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:35:57.766 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.763+0000 7f75cb9b2740 1 -- 172.21.15.90:0/640854306 shutdown_connections 2024-04-02T14:35:57.766 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.763+0000 7f75cb9b2740 1 -- 172.21.15.90:0/640854306 wait complete. 2024-04-02T14:35:57.766 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.764+0000 7f75cb9b2740 20 client.25724 trim_cache size 0 max 16384 2024-04-02T14:35:57.767 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:{ 2024-04-02T14:35:57.767 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "message": "", 2024-04-02T14:35:57.768 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "return_code": 0 2024-04-02T14:35:57.768 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:} 2024-04-02T14:35:57.768 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.766+0000 7f75c8c00640 1 -- 172.21.15.90:0/4180406180 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f75a80a3f60 msgr2=0x7f75a80a6420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:35:57.769 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.766+0000 7f75c8c00640 1 --2- 172.21.15.90:0/4180406180 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f75a80a3f60 0x7f75a80a6420 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f75bc005c40 tx=0x7f75bc004a40 comp rx=0 tx=0).stop 2024-04-02T14:35:57.769 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.766+0000 7f75c8c00640 1 -- 172.21.15.90:0/4180406180 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f75c41029e0 msgr2=0x7f75c410c4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:35:57.769 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.766+0000 7f75c8c00640 1 --2- 172.21.15.90:0/4180406180 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f75c41029e0 0x7f75c410c4e0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f75b002c2d0 tx=0x7f75b00043c0 comp rx=0 tx=0).stop 2024-04-02T14:35:57.769 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.766+0000 7f75c8c00640 1 -- 172.21.15.90:0/4180406180 shutdown_connections 2024-04-02T14:35:57.769 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.766+0000 7f75c8c00640 1 --2- 172.21.15.90:0/4180406180 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f75a80a3f60 0x7f75a80a6420 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:57.769 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.766+0000 7f75c8c00640 1 --2- 172.21.15.90:0/4180406180 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f75c410e350 0x7f75c410ca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:57.769 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.766+0000 7f75c8c00640 1 --2- 172.21.15.90:0/4180406180 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f75c41029e0 0x7f75c410c4e0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:57.769 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.766+0000 7f75c8c00640 1 --2- 172.21.15.90:0/4180406180 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f75c40696c0 0x7f75c410bfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:35:57.769 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.766+0000 7f75c8c00640 1 -- 172.21.15.90:0/4180406180 >> 172.21.15.90:0/4180406180 conn(0x7f75c4077850 msgr2=0x7f75c40ffc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:35:57.770 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.767+0000 7f75c8c00640 1 -- 172.21.15.90:0/4180406180 shutdown_connections 2024-04-02T14:35:57.770 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:35:57.767+0000 7f75c8c00640 1 -- 172.21.15.90:0/4180406180 wait complete. 2024-04-02T14:35:58.360 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:35:57 smithi092 ceph-mon[36497]: pgmap v35: 97 pgs: 97 active+clean; 956 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 9.1 MiB/s wr, 709 op/s 2024-04-02T14:35:58.436 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:35:57 smithi177 ceph-mon[36431]: pgmap v35: 97 pgs: 97 active+clean; 956 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 9.1 MiB/s wr, 709 op/s 2024-04-02T14:35:58.438 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:35:57 smithi090 ceph-mon[31674]: pgmap v35: 97 pgs: 97 active+clean; 956 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 9.1 MiB/s wr, 709 op/s 2024-04-02T14:36:00.360 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:35:59 smithi092 ceph-mon[36497]: pgmap v36: 97 pgs: 97 active+clean; 965 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 12 MiB/s wr, 909 op/s 2024-04-02T14:36:00.436 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:35:59 smithi177 ceph-mon[36431]: pgmap v36: 97 pgs: 97 active+clean; 965 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 12 MiB/s wr, 909 op/s 2024-04-02T14:36:00.438 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:35:59 smithi090 ceph-mon[31674]: pgmap v36: 97 pgs: 97 active+clean; 965 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 12 MiB/s wr, 909 op/s 2024-04-02T14:36:00.439 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 02 14:36:00 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-x[31910]: 2024-04-02T14:36:00.169+0000 7ff5c2400700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T14:36:00.439 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 02 14:36:00 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-x[31910]: 2024-04-02T14:36:00.169+0000 7ff5c2400700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T14:36:00.439 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 02 14:36:00 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-x[31910]: 2024-04-02T14:36:00.169+0000 7ff5c2400700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T14:36:00.439 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 02 14:36:00 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-x[31910]: 2024-04-02T14:36:00.169+0000 7ff5c2400700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T14:36:00.439 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 02 14:36:00 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-x[31910]: 2024-04-02T14:36:00.169+0000 7ff5c2400700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-02T14:36:01.360 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:36:00 smithi092 ceph-mon[36497]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T14:36:01.436 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:36:00 smithi177 ceph-mon[36431]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T14:36:01.438 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:36:00 smithi090 ceph-mon[31674]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T14:36:01.832 INFO:teuthology.orchestra.run.smithi090.stderr:progress: 30.0 s, 2598.5 tps, lat 30.854 ms stddev 50.253 2024-04-02T14:36:02.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:36:01 smithi092 ceph-mon[36497]: pgmap v37: 97 pgs: 97 active+clean; 974 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 11 MiB/s wr, 801 op/s 2024-04-02T14:36:02.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:36:01 smithi092 ceph-mon[36497]: mgrmap e25: x(active, since 68s), standbys: z, y 2024-04-02T14:36:02.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:36:01 smithi092 ceph-mon[36497]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' 2024-04-02T14:36:02.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:36:01 smithi092 ceph-mon[36497]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' 2024-04-02T14:36:02.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:36:01 smithi092 ceph-mon[36497]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' 2024-04-02T14:36:02.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:36:01 smithi092 ceph-mon[36497]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' 2024-04-02T14:36:02.436 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:36:02 smithi177 ceph-mon[36431]: pgmap v37: 97 pgs: 97 active+clean; 974 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 11 MiB/s wr, 801 op/s 2024-04-02T14:36:02.436 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:36:02 smithi177 ceph-mon[36431]: mgrmap e25: x(active, since 68s), standbys: z, y 2024-04-02T14:36:02.437 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:36:02 smithi177 ceph-mon[36431]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' 2024-04-02T14:36:02.437 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:36:02 smithi177 ceph-mon[36431]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' 2024-04-02T14:36:02.437 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:36:02 smithi177 ceph-mon[36431]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' 2024-04-02T14:36:02.437 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:36:02 smithi177 ceph-mon[36431]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' 2024-04-02T14:36:02.439 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:36:01 smithi090 ceph-mon[31674]: pgmap v37: 97 pgs: 97 active+clean; 974 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 11 MiB/s wr, 801 op/s 2024-04-02T14:36:02.439 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:36:01 smithi090 ceph-mon[31674]: mgrmap e25: x(active, since 68s), standbys: z, y 2024-04-02T14:36:02.439 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:36:01 smithi090 ceph-mon[31674]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' 2024-04-02T14:36:02.439 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:36:01 smithi090 ceph-mon[31674]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' 2024-04-02T14:36:02.439 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:36:01 smithi090 ceph-mon[31674]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' 2024-04-02T14:36:02.439 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:36:01 smithi090 ceph-mon[31674]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' 2024-04-02T14:36:03.360 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:36:03 smithi092 ceph-mon[36497]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:36:03.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:36:03 smithi092 ceph-mon[36497]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T14:36:03.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:36:03 smithi092 ceph-mon[36497]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' 2024-04-02T14:36:03.437 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:36:03 smithi177 ceph-mon[36431]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:36:03.437 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:36:03 smithi177 ceph-mon[36431]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T14:36:03.437 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:36:03 smithi177 ceph-mon[36431]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' 2024-04-02T14:36:03.438 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:36:03 smithi090 ceph-mon[31674]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:36:03.438 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:36:03 smithi090 ceph-mon[31674]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T14:36:03.438 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:36:03 smithi090 ceph-mon[31674]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' 2024-04-02T14:36:04.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:36:04 smithi092 ceph-mon[36497]: pgmap v38: 97 pgs: 97 active+clean; 974 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 10 MiB/s wr, 678 op/s 2024-04-02T14:36:04.436 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:36:04 smithi177 ceph-mon[36431]: pgmap v38: 97 pgs: 97 active+clean; 974 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 10 MiB/s wr, 678 op/s 2024-04-02T14:36:04.438 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:36:04 smithi090 ceph-mon[31674]: pgmap v38: 97 pgs: 97 active+clean; 974 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 10 MiB/s wr, 678 op/s 2024-04-02T14:36:06.360 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:36:06 smithi092 ceph-mon[36497]: pgmap v39: 97 pgs: 97 active+clean; 1.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 53 MiB/s wr, 1.46k op/s 2024-04-02T14:36:06.436 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:36:06 smithi177 ceph-mon[36431]: pgmap v39: 97 pgs: 97 active+clean; 1.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 53 MiB/s wr, 1.46k op/s 2024-04-02T14:36:06.438 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:36:06 smithi090 ceph-mon[31674]: pgmap v39: 97 pgs: 97 active+clean; 1.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 53 MiB/s wr, 1.46k op/s 2024-04-02T14:36:06.831 INFO:teuthology.orchestra.run.smithi090.stderr:progress: 35.0 s, 2851.0 tps, lat 42.096 ms stddev 188.242 2024-04-02T14:36:07.622 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-02T14:36:07.623 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-02T14:36:07.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:07.916+0000 7f0911e00640 1 -- 172.21.15.90:0/3652096413 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f090c108310 msgr2=0x7f090c1086f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:07.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:07.916+0000 7f0911e00640 1 --2- 172.21.15.90:0/3652096413 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f090c108310 0x7f090c1086f0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f08fc009900 tx=0x7f08fc02bc30 comp rx=0 tx=0).stop 2024-04-02T14:36:07.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:07.916+0000 7f0911e00640 1 -- 172.21.15.90:0/3652096413 shutdown_connections 2024-04-02T14:36:07.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:07.916+0000 7f0911e00640 1 --2- 172.21.15.90:0/3652096413 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f090c10d3e0 0x7f090c114fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:07.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:07.916+0000 7f0911e00640 1 --2- 172.21.15.90:0/3652096413 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f090c108cc0 0x7f090c10cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:07.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:07.916+0000 7f0911e00640 1 --2- 172.21.15.90:0/3652096413 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f090c108310 0x7f090c1086f0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:07.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:07.916+0000 7f0911e00640 1 -- 172.21.15.90:0/3652096413 >> 172.21.15.90:0/3652096413 conn(0x7f090c076040 msgr2=0x7f090c078460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:36:07.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:07.917+0000 7f0911e00640 1 -- 172.21.15.90:0/3652096413 shutdown_connections 2024-04-02T14:36:07.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:07.918+0000 7f0911e00640 1 -- 172.21.15.90:0/3652096413 wait complete. 2024-04-02T14:36:07.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:07.919+0000 7f0911e00640 1 Processor -- start 2024-04-02T14:36:07.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:07.919+0000 7f0911e00640 1 -- start start 2024-04-02T14:36:07.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:07.919+0000 7f0911e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f090c108310 0x7f090c1a4690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:07.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:07.919+0000 7f0911e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f090c10d3e0 0x7f090c1a4bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:07.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:07.919+0000 7f0911e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f090c19e800 0x7f090c19ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:07.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:07.920+0000 7f0911e00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f090c1184c0 con 0x7f090c108310 2024-04-02T14:36:07.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:07.920+0000 7f0911e00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f090c118340 con 0x7f090c10d3e0 2024-04-02T14:36:07.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:07.920+0000 7f0911e00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f090c118640 con 0x7f090c19e800 2024-04-02T14:36:07.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:07.920+0000 7f090a000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f090c10d3e0 0x7f090c1a4bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:07.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:07.920+0000 7f090a000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f090c10d3e0 0x7f090c1a4bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:32976/0 (socket says 172.21.15.90:32976) 2024-04-02T14:36:07.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:07.920+0000 7f090a000640 1 -- 172.21.15.90:0/2692333805 learned_addr learned my addr 172.21.15.90:0/2692333805 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:36:07.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:07.920+0000 7f090b400640 1 --2- 172.21.15.90:0/2692333805 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f090c19e800 0x7f090c19ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:07.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:07.920+0000 7f090a000640 1 -- 172.21.15.90:0/2692333805 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f090c19e800 msgr2=0x7f090c19ecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:07.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:07.920+0000 7f090a000640 1 --2- 172.21.15.90:0/2692333805 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f090c19e800 0x7f090c19ecb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:07.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:07.920+0000 7f090a000640 1 -- 172.21.15.90:0/2692333805 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f090c108310 msgr2=0x7f090c1a4690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:07.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:07.920+0000 7f090a000640 1 --2- 172.21.15.90:0/2692333805 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f090c108310 0x7f090c1a4690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:07.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:07.920+0000 7f090a000640 1 -- 172.21.15.90:0/2692333805 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f090c19f390 con 0x7f090c10d3e0 2024-04-02T14:36:07.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:07.921+0000 7f090a000640 1 --2- 172.21.15.90:0/2692333805 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f090c10d3e0 0x7f090c1a4bd0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f08f800e9c0 tx=0x7f08f800ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:36:07.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:07.921+0000 7f0910a00640 1 -- 172.21.15.90:0/2692333805 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08f800cd80 con 0x7f090c10d3e0 2024-04-02T14:36:07.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:07.921+0000 7f0910a00640 1 -- 172.21.15.90:0/2692333805 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f08f8005500 con 0x7f090c10d3e0 2024-04-02T14:36:07.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:07.921+0000 7f0910a00640 1 -- 172.21.15.90:0/2692333805 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08f8010690 con 0x7f090c10d3e0 2024-04-02T14:36:07.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:07.921+0000 7f0911e00640 1 -- 172.21.15.90:0/2692333805 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f090c19f680 con 0x7f090c10d3e0 2024-04-02T14:36:07.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:07.922+0000 7f0911e00640 1 -- 172.21.15.90:0/2692333805 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f090c1ab810 con 0x7f090c10d3e0 2024-04-02T14:36:07.929 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:07.927+0000 7f0910a00640 1 -- 172.21.15.90:0/2692333805 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 25) ==== 138930+0+0 (secure 0 0 0) 0x7f08f8005020 con 0x7f090c10d3e0 2024-04-02T14:36:07.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:07.931+0000 7f0914a57740 1 Processor -- start 2024-04-02T14:36:07.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:07.931+0000 7f0910a00640 1 --2- 172.21.15.90:0/2692333805 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f08f00a4020 0x7f08f00a64e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:07.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:07.931+0000 7f0910a00640 1 -- 172.21.15.90:0/2692333805 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f08f80d0ef0 con 0x7f090c10d3e0 2024-04-02T14:36:07.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:07.932+0000 7f090aa00640 1 --2- 172.21.15.90:0/2692333805 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f08f00a4020 0x7f08f00a64e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:07.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:07.932+0000 7f0914a57740 1 -- start start 2024-04-02T14:36:07.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:07.933+0000 7f0914a57740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563599b137b0 0x563599b13b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:07.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:07.933+0000 7f0914a57740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563599b140d0 0x563599b1c770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:07.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:07.933+0000 7f0914a57740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563599b1ccb0 0x563599b1f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:07.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:07.933+0000 7f0914a57740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x563599a77770 con 0x563599b137b0 2024-04-02T14:36:07.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:07.933+0000 7f0914a57740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x563599a1f700 con 0x563599b1ccb0 2024-04-02T14:36:07.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:07.933+0000 7f0914a57740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x563599a73720 con 0x563599b140d0 2024-04-02T14:36:07.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:07.933+0000 7f090a000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563599b140d0 0x563599b1c770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:07.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:07.933+0000 7f090a000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563599b140d0 0x563599b1c770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:53690/0 (socket says 172.21.15.90:53690) 2024-04-02T14:36:07.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:07.933+0000 7f090a000640 1 -- 172.21.15.90:0/709071739 learned_addr learned my addr 172.21.15.90:0/709071739 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:36:07.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:07.934+0000 7f090a000640 1 -- 172.21.15.90:0/709071739 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563599b1ccb0 msgr2=0x563599b1f0a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-02T14:36:07.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:07.934+0000 7f090a000640 1 --2- 172.21.15.90:0/709071739 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563599b1ccb0 0x563599b1f0a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:07.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:07.934+0000 7f090a000640 1 -- 172.21.15.90:0/709071739 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563599b137b0 msgr2=0x563599b13b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:36:07.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:07.934+0000 7f090a000640 1 --2- 172.21.15.90:0/709071739 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563599b137b0 0x563599b13b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:07.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:07.934+0000 7f090a000640 1 -- 172.21.15.90:0/709071739 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5635998557d0 con 0x563599b140d0 2024-04-02T14:36:07.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:07.935+0000 7f090aa00640 1 --2- 172.21.15.90:0/2692333805 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f08f00a4020 0x7f08f00a64e0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f08fc036040 tx=0x7f08fc009560 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:36:07.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:07.935+0000 7f090a000640 1 --2- 172.21.15.90:0/709071739 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563599b140d0 0x563599b1c770 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f08f80a53a0 tx=0x7f08f809c7a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8847bbdd99e0fddc server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:36:07.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:07.935+0000 7f08f7e00640 1 -- 172.21.15.90:0/709071739 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08f809cc60 con 0x563599b140d0 2024-04-02T14:36:07.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:07.936+0000 7f08f7e00640 1 -- 172.21.15.90:0/709071739 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f08f80a1010 con 0x563599b140d0 2024-04-02T14:36:07.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:07.936+0000 7f08f7e00640 1 -- 172.21.15.90:0/709071739 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08f80bd070 con 0x563599b140d0 2024-04-02T14:36:07.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:07.937+0000 7f0914a57740 1 -- 172.21.15.90:0/709071739 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563599b140d0 msgr2=0x563599b1c770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:07.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:07.937+0000 7f0914a57740 1 --2- 172.21.15.90:0/709071739 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563599b140d0 0x563599b1c770 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f08f80a53a0 tx=0x7f08f809c7a0 comp rx=0 tx=0).stop 2024-04-02T14:36:07.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:07.937+0000 7f0914a57740 1 -- 172.21.15.90:0/709071739 shutdown_connections 2024-04-02T14:36:07.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:07.937+0000 7f0914a57740 1 --2- 172.21.15.90:0/709071739 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563599b1ccb0 0x563599b1f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:07.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:07.937+0000 7f0914a57740 1 --2- 172.21.15.90:0/709071739 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563599b140d0 0x563599b1c770 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:07.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:07.937+0000 7f0914a57740 1 --2- 172.21.15.90:0/709071739 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563599b137b0 0x563599b13b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:07.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:07.937+0000 7f0914a57740 1 -- 172.21.15.90:0/709071739 >> 172.21.15.90:0/709071739 conn(0x563599b0dee0 msgr2=0x563599b0e2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:36:07.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:07.938+0000 7f0914a57740 1 -- 172.21.15.90:0/709071739 shutdown_connections 2024-04-02T14:36:07.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:07.938+0000 7f0914a57740 1 -- 172.21.15.90:0/709071739 wait complete. 2024-04-02T14:36:07.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:07.939+0000 7f0914a57740 1 -- start start 2024-04-02T14:36:07.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:07.940+0000 7f0914a57740 1 Processor -- start 2024-04-02T14:36:07.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:07.941+0000 7f0914a57740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563599b137b0 0x563599b19d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:07.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:07.941+0000 7f0914a57740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563599b140d0 0x563599b1a270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:07.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:07.941+0000 7f0914a57740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563599b1ccb0 0x563599b1a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:07.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:07.941+0000 7f0914a57740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x563599b30a50 con 0x563599b137b0 2024-04-02T14:36:07.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:07.941+0000 7f0914a57740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x563599a32ac0 con 0x563599b140d0 2024-04-02T14:36:07.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:07.941+0000 7f0914a57740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x563599b30bd0 con 0x563599b1ccb0 2024-04-02T14:36:07.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:07.941+0000 7f0914a57740 10 client.0 _ll_register_callbacks cb 0x563599a35180 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:36:07.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:07.942+0000 7f090a000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563599b140d0 0x563599b1a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:07.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:07.942+0000 7f090b400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563599b1ccb0 0x563599b1a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:07.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:07.942+0000 7f090b400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563599b1ccb0 0x563599b1a7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:53692/0 (socket says 172.21.15.90:53692) 2024-04-02T14:36:07.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:07.942+0000 7f090b400640 1 -- 172.21.15.90:0/3210120758 learned_addr learned my addr 172.21.15.90:0/3210120758 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:36:07.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:07.942+0000 7f090aa00640 1 --2- 172.21.15.90:0/3210120758 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563599b137b0 0x563599b19d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:07.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:07.942+0000 7f090b400640 1 -- 172.21.15.90:0/3210120758 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563599b140d0 msgr2=0x563599b1a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:36:07.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:07.943+0000 7f090b400640 1 --2- 172.21.15.90:0/3210120758 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563599b140d0 0x563599b1a270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:07.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:07.943+0000 7f090b400640 1 -- 172.21.15.90:0/3210120758 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563599b137b0 msgr2=0x563599b19d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:36:07.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:07.943+0000 7f090b400640 1 --2- 172.21.15.90:0/3210120758 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563599b137b0 0x563599b19d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:07.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:07.943+0000 7f090b400640 1 -- 172.21.15.90:0/3210120758 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56359978b3f0 con 0x563599b1ccb0 2024-04-02T14:36:07.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:07.943+0000 7f090b400640 1 --2- 172.21.15.90:0/3210120758 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563599b1ccb0 0x563599b1a7b0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f09040107d0 tx=0x7f0904010c90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d501ecd80ec4696b server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:36:07.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:07.943+0000 7f08f4c00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:36:07.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:07.944+0000 7f08f4c00640 1 -- 172.21.15.90:0/3210120758 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f090400ec60 con 0x563599b1ccb0 2024-04-02T14:36:07.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:07.944+0000 7f08f4c00640 1 -- 172.21.15.90:0/3210120758 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f090400ee00 con 0x563599b1ccb0 2024-04-02T14:36:07.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:07.944+0000 7f0914a57740 1 -- 172.21.15.90:0/3210120758 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f08f80a1010 con 0x563599b1ccb0 2024-04-02T14:36:07.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:07.944+0000 7f08f4c00640 1 -- 172.21.15.90:0/3210120758 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f0904012450 con 0x563599b1ccb0 2024-04-02T14:36:07.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:07.944+0000 7f0914a57740 10 client.25693 fetch_fsmap learned FSMap version 27 2024-04-02T14:36:07.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:07.944+0000 7f0914a57740 1 -- 172.21.15.90:0/3210120758 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x56359974b0f0 con 0x563599b1ccb0 2024-04-02T14:36:07.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:07.945+0000 7f08f4c00640 1 -- 172.21.15.90:0/3210120758 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f08f80a1010 con 0x563599b1ccb0 2024-04-02T14:36:07.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:07.945+0000 7f08f4c00640 1 -- 172.21.15.90:0/3210120758 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f090400f040 con 0x563599b1ccb0 2024-04-02T14:36:07.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:07.945+0000 7f0914a57740 10 client.25693 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:36:07.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:07.945+0000 7f0914a57740 10 client.25693 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:36:07.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:07.945+0000 7f0914a57740 20 client.25693 populate_metadata read hostname 'smithi090' 2024-04-02T14:36:07.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:07.945+0000 7f0914a57740 1 --2- 172.21.15.90:0/3210120758 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x563599b3dca0 0x563599b400d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:07.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:07.945+0000 7f0914a57740 4 client.25693 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:36:07.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:07.945+0000 7f0914a57740 1 -- 172.21.15.90:0/3210120758 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563599a18630 con 0x563599b3dca0 2024-04-02T14:36:07.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:07.946+0000 7f090aa00640 1 --2- 172.21.15.90:0/3210120758 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x563599b3dca0 0x563599b400d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:07.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:07.947+0000 7f08f4c00640 10 client.25693 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:36:07.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:07.950+0000 7f090aa00640 1 --2- 172.21.15.90:0/3210120758 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x563599b3dca0 0x563599b400d0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f6edff5faa7dc12d server_cookie=410bd4f4db65f57e in_seq=0 out_seq=0 2024-04-02T14:36:07.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:07.951+0000 7f08f4c00640 1 -- 172.21.15.90:0/3210120758 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x563599a18630 con 0x563599b3dca0 2024-04-02T14:36:07.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:07.951+0000 7f08f4c00640 10 client.25693 handle_command_reply: tid=0 2024-04-02T14:36:07.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:07.951+0000 7f0914a57740 1 client.25693 shutdown 2024-04-02T14:36:07.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:07.951+0000 7f0914a57740 1 -- 172.21.15.90:0/3210120758 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563599b1ccb0 msgr2=0x563599b1a7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:07.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:07.951+0000 7f0914a57740 1 --2- 172.21.15.90:0/3210120758 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563599b1ccb0 0x563599b1a7b0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f09040107d0 tx=0x7f0904010c90 comp rx=0 tx=0).stop 2024-04-02T14:36:07.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:07.951+0000 7f0914a57740 1 -- 172.21.15.90:0/3210120758 shutdown_connections 2024-04-02T14:36:07.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:07.951+0000 7f0914a57740 1 --2- 172.21.15.90:0/3210120758 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x563599b3dca0 0x563599b400d0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:07.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:07.951+0000 7f0914a57740 1 --2- 172.21.15.90:0/3210120758 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563599b1ccb0 0x563599b1a7b0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:07.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:07.951+0000 7f0914a57740 1 --2- 172.21.15.90:0/3210120758 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563599b140d0 0x563599b1a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:07.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:07.951+0000 7f0914a57740 1 --2- 172.21.15.90:0/3210120758 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563599b137b0 0x563599b19d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:07.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:07.951+0000 7f0914a57740 1 -- 172.21.15.90:0/3210120758 >> 172.21.15.90:0/3210120758 conn(0x563599b0dee0 msgr2=0x563599b0fc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:36:07.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:07.952+0000 7f08f4c00640 0 client.25693 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:36:07.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:07.952+0000 7f0914a57740 1 -- 172.21.15.90:0/3210120758 shutdown_connections 2024-04-02T14:36:07.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:07.952+0000 7f0914a57740 1 -- 172.21.15.90:0/3210120758 wait complete. 2024-04-02T14:36:07.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:07.952+0000 7f0914a57740 20 client.25693 trim_cache size 0 max 16384 2024-04-02T14:36:08.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:08.005+0000 7f0914a57740 1 Processor -- start 2024-04-02T14:36:08.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:08.005+0000 7f0914a57740 1 -- start start 2024-04-02T14:36:08.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:08.005+0000 7f0914a57740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563599b137b0 0x563599b1c460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:08.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:08.005+0000 7f0914a57740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563599b140d0 0x563599b39960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:08.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:08.005+0000 7f0914a57740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563599b1ccb0 0x563599b18600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:08.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:08.005+0000 7f0914a57740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f090c082780 con 0x563599b137b0 2024-04-02T14:36:08.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:08.005+0000 7f0914a57740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f090c1ad2a0 con 0x563599b1ccb0 2024-04-02T14:36:08.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:08.005+0000 7f0914a57740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f090c1abe30 con 0x563599b140d0 2024-04-02T14:36:08.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:08.006+0000 7f090b400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563599b1ccb0 0x563599b18600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:08.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:08.006+0000 7f090b400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563599b1ccb0 0x563599b18600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:32998/0 (socket says 172.21.15.90:32998) 2024-04-02T14:36:08.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:08.006+0000 7f090b400640 1 -- 172.21.15.90:0/1864122475 learned_addr learned my addr 172.21.15.90:0/1864122475 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:36:08.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:08.007+0000 7f090aa00640 1 --2- 172.21.15.90:0/1864122475 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563599b137b0 0x563599b1c460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:08.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:08.007+0000 7f090a000640 1 --2- 172.21.15.90:0/1864122475 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563599b140d0 0x563599b39960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:08.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:08.008+0000 7f090a000640 1 -- 172.21.15.90:0/1864122475 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563599b1ccb0 msgr2=0x563599b18600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:36:08.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:08.008+0000 7f090a000640 1 --2- 172.21.15.90:0/1864122475 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563599b1ccb0 0x563599b18600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:08.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:08.008+0000 7f090a000640 1 -- 172.21.15.90:0/1864122475 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563599b137b0 msgr2=0x563599b1c460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:36:08.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:08.008+0000 7f090a000640 1 --2- 172.21.15.90:0/1864122475 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563599b137b0 0x563599b1c460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:08.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:08.008+0000 7f090a000640 1 -- 172.21.15.90:0/1864122475 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563599b0da10 con 0x563599b140d0 2024-04-02T14:36:08.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:08.008+0000 7f090aa00640 1 --2- 172.21.15.90:0/1864122475 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563599b137b0 0x563599b1c460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:36:08.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:08.009+0000 7f090a000640 1 --2- 172.21.15.90:0/1864122475 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563599b140d0 0x563599b39960 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x563599b30780 tx=0x7f08f800efb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8f0857d19d5bf551 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:36:08.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:08.010+0000 7f08f7400640 1 -- 172.21.15.90:0/1864122475 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08f809cc60 con 0x563599b140d0 2024-04-02T14:36:08.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:08.010+0000 7f08f7400640 1 -- 172.21.15.90:0/1864122475 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f08f8005310 con 0x563599b140d0 2024-04-02T14:36:08.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:08.010+0000 7f08f7400640 1 -- 172.21.15.90:0/1864122475 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08f80bd0c0 con 0x563599b140d0 2024-04-02T14:36:08.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:08.011+0000 7f0914a57740 1 -- 172.21.15.90:0/1864122475 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563599b140d0 msgr2=0x563599b39960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:08.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:08.011+0000 7f0914a57740 1 --2- 172.21.15.90:0/1864122475 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563599b140d0 0x563599b39960 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x563599b30780 tx=0x7f08f800efb0 comp rx=0 tx=0).stop 2024-04-02T14:36:08.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:08.011+0000 7f0914a57740 1 -- 172.21.15.90:0/1864122475 shutdown_connections 2024-04-02T14:36:08.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:08.011+0000 7f0914a57740 1 --2- 172.21.15.90:0/1864122475 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563599b1ccb0 0x563599b18600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:08.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:08.011+0000 7f0914a57740 1 --2- 172.21.15.90:0/1864122475 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563599b140d0 0x563599b39960 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:08.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:08.011+0000 7f0914a57740 1 --2- 172.21.15.90:0/1864122475 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563599b137b0 0x563599b1c460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:08.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:08.011+0000 7f0914a57740 1 -- 172.21.15.90:0/1864122475 >> 172.21.15.90:0/1864122475 conn(0x563599b0dee0 msgr2=0x563599b0c6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:36:08.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:08.011+0000 7f0914a57740 1 -- 172.21.15.90:0/1864122475 shutdown_connections 2024-04-02T14:36:08.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:08.011+0000 7f0914a57740 1 -- 172.21.15.90:0/1864122475 wait complete. 2024-04-02T14:36:08.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:08.011+0000 7f0914a57740 1 -- start start 2024-04-02T14:36:08.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:08.012+0000 7f0914a57740 1 Processor -- start 2024-04-02T14:36:08.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:08.014+0000 7f0914a57740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563599b137b0 0x563599b16b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:08.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:08.014+0000 7f0914a57740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563599b140d0 0x563599b17090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:08.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:08.015+0000 7f0914a57740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563599b1ccb0 0x563599b175d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:08.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:08.015+0000 7f0914a57740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f090c1acbd0 con 0x563599b1ccb0 2024-04-02T14:36:08.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:08.015+0000 7f0914a57740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f090c1ac500 con 0x563599b137b0 2024-04-02T14:36:08.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:08.015+0000 7f0914a57740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f090c080780 con 0x563599b140d0 2024-04-02T14:36:08.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:08.015+0000 7f0914a57740 10 client.0 _ll_register_callbacks cb 0x7f0904009a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:36:08.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:08.016+0000 7f090aa00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563599b137b0 0x563599b16b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:08.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:08.016+0000 7f090b400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563599b1ccb0 0x563599b175d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:08.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:08.016+0000 7f090aa00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563599b137b0 0x563599b16b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:33000/0 (socket says 172.21.15.90:33000) 2024-04-02T14:36:08.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:08.016+0000 7f090aa00640 1 -- 172.21.15.90:0/4028707483 learned_addr learned my addr 172.21.15.90:0/4028707483 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:36:08.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:08.016+0000 7f090a000640 1 --2- 172.21.15.90:0/4028707483 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563599b140d0 0x563599b17090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:08.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:08.016+0000 7f090aa00640 1 -- 172.21.15.90:0/4028707483 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563599b140d0 msgr2=0x563599b17090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:36:08.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:08.016+0000 7f090aa00640 1 --2- 172.21.15.90:0/4028707483 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563599b140d0 0x563599b17090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:08.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:08.016+0000 7f090aa00640 1 -- 172.21.15.90:0/4028707483 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563599b1ccb0 msgr2=0x563599b175d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:36:08.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:08.016+0000 7f090aa00640 1 --2- 172.21.15.90:0/4028707483 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563599b1ccb0 0x563599b175d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:08.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:08.016+0000 7f090aa00640 1 -- 172.21.15.90:0/4028707483 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563599ba3370 con 0x563599b137b0 2024-04-02T14:36:08.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:08.016+0000 7f090aa00640 1 --2- 172.21.15.90:0/4028707483 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563599b137b0 0x563599b16b50 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f08fc005c40 tx=0x7f08fc005fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=404304a75ec215c9 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:36:08.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:08.016+0000 7f08f4c00640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:36:08.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:08.017+0000 7f08f4c00640 1 -- 172.21.15.90:0/4028707483 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08fc0043e0 con 0x563599b137b0 2024-04-02T14:36:08.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:08.017+0000 7f08f4c00640 1 -- 172.21.15.90:0/4028707483 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f08fc005050 con 0x563599b137b0 2024-04-02T14:36:08.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:08.017+0000 7f0914a57740 1 -- 172.21.15.90:0/4028707483 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f08f8005310 con 0x563599b137b0 2024-04-02T14:36:08.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:08.018+0000 7f08f4c00640 1 -- 172.21.15.90:0/4028707483 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f08fc005250 con 0x563599b137b0 2024-04-02T14:36:08.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:08.018+0000 7f0914a57740 10 client.25745 fetch_fsmap learned FSMap version 27 2024-04-02T14:36:08.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:08.018+0000 7f0914a57740 1 -- 172.21.15.90:0/4028707483 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x563599ba4690 con 0x563599b137b0 2024-04-02T14:36:08.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:08.018+0000 7f08f4c00640 1 -- 172.21.15.90:0/4028707483 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f08f8005310 con 0x563599b137b0 2024-04-02T14:36:08.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:08.018+0000 7f08f4c00640 1 -- 172.21.15.90:0/4028707483 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f08fc0046c0 con 0x563599b137b0 2024-04-02T14:36:08.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:08.019+0000 7f0914a57740 10 client.25745 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:36:08.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:08.019+0000 7f0914a57740 10 client.25745 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:36:08.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:08.019+0000 7f0914a57740 20 client.25745 populate_metadata read hostname 'smithi090' 2024-04-02T14:36:08.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:08.019+0000 7f0914a57740 1 --2- 172.21.15.90:0/4028707483 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x563599ba4e40 0x563599ba7230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:08.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:08.019+0000 7f0914a57740 4 client.25745 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-02T14:36:08.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:08.019+0000 7f0914a57740 1 -- 172.21.15.90:0/4028707483 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "scrub status"}) -- 0x563599a84cb0 con 0x563599ba4e40 2024-04-02T14:36:08.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:08.019+0000 7f090a000640 1 --2- 172.21.15.90:0/4028707483 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x563599ba4e40 0x563599ba7230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:08.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:08.019+0000 7f090a000640 1 --2- 172.21.15.90:0/4028707483 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x563599ba4e40 0x563599ba7230 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ae6c107bb7aeb64d server_cookie=b378d966b831520f in_seq=0 out_seq=0 2024-04-02T14:36:08.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:08.020+0000 7f08f4c00640 10 client.25745 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:36:08.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:08.020+0000 7f08f4c00640 1 -- 172.21.15.90:0/4028707483 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x563599a84cb0 con 0x563599ba4e40 2024-04-02T14:36:08.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:08.020+0000 7f08f4c00640 10 client.25745 handle_command_reply: tid=0 2024-04-02T14:36:08.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:08.020+0000 7f0914a57740 1 client.25745 shutdown 2024-04-02T14:36:08.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:08.020+0000 7f0914a57740 1 -- 172.21.15.90:0/4028707483 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563599b137b0 msgr2=0x563599b16b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:08.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:08.020+0000 7f0914a57740 1 --2- 172.21.15.90:0/4028707483 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563599b137b0 0x563599b16b50 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f08fc005c40 tx=0x7f08fc005fd0 comp rx=0 tx=0).stop 2024-04-02T14:36:08.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:08.020+0000 7f0914a57740 1 -- 172.21.15.90:0/4028707483 shutdown_connections 2024-04-02T14:36:08.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:08.020+0000 7f0914a57740 1 --2- 172.21.15.90:0/4028707483 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x563599ba4e40 0x563599ba7230 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:08.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:08.020+0000 7f08f4c00640 0 client.25745 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:36:08.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:08.020+0000 7f0914a57740 1 --2- 172.21.15.90:0/4028707483 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563599b1ccb0 0x563599b175d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:08.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:08.020+0000 7f0914a57740 1 --2- 172.21.15.90:0/4028707483 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563599b140d0 0x563599b17090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:08.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:08.020+0000 7f0914a57740 1 --2- 172.21.15.90:0/4028707483 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563599b137b0 0x563599b16b50 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:08.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:08.020+0000 7f0914a57740 1 -- 172.21.15.90:0/4028707483 >> 172.21.15.90:0/4028707483 conn(0x563599b0dee0 msgr2=0x563599b0cf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:36:08.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:08.021+0000 7f0914a57740 1 -- 172.21.15.90:0/4028707483 shutdown_connections 2024-04-02T14:36:08.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:08.021+0000 7f0914a57740 1 -- 172.21.15.90:0/4028707483 wait complete. 2024-04-02T14:36:08.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:08.021+0000 7f0914a57740 20 client.25745 trim_cache size 0 max 16384 2024-04-02T14:36:08.024 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-02T14:36:08.024 INFO:teuthology.orchestra.run.smithi090.stdout: "status": "no active scrubs running", 2024-04-02T14:36:08.024 INFO:teuthology.orchestra.run.smithi090.stdout: "scrubs": {} 2024-04-02T14:36:08.024 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-02T14:36:08.025 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:08.022+0000 7f0911e00640 1 -- 172.21.15.90:0/2692333805 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f08f00a4020 msgr2=0x7f08f00a64e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:08.025 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:08.023+0000 7f0911e00640 1 --2- 172.21.15.90:0/2692333805 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f08f00a4020 0x7f08f00a64e0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f08fc036040 tx=0x7f08fc009560 comp rx=0 tx=0).stop 2024-04-02T14:36:08.025 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:08.023+0000 7f0911e00640 1 -- 172.21.15.90:0/2692333805 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f090c10d3e0 msgr2=0x7f090c1a4bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:08.025 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:08.023+0000 7f0911e00640 1 --2- 172.21.15.90:0/2692333805 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f090c10d3e0 0x7f090c1a4bd0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f08f800e9c0 tx=0x7f08f800ee80 comp rx=0 tx=0).stop 2024-04-02T14:36:08.025 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:08.023+0000 7f0911e00640 1 -- 172.21.15.90:0/2692333805 shutdown_connections 2024-04-02T14:36:08.025 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:08.023+0000 7f0911e00640 1 --2- 172.21.15.90:0/2692333805 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f08f00a4020 0x7f08f00a64e0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:08.025 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:08.023+0000 7f0911e00640 1 --2- 172.21.15.90:0/2692333805 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f090c19e800 0x7f090c19ecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:08.025 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:08.023+0000 7f0911e00640 1 --2- 172.21.15.90:0/2692333805 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f090c10d3e0 0x7f090c1a4bd0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:08.025 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:08.023+0000 7f0911e00640 1 --2- 172.21.15.90:0/2692333805 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f090c108310 0x7f090c1a4690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:08.025 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:08.023+0000 7f0911e00640 1 -- 172.21.15.90:0/2692333805 >> 172.21.15.90:0/2692333805 conn(0x7f090c076040 msgr2=0x7f090c078350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:36:08.025 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:08.023+0000 7f0911e00640 1 -- 172.21.15.90:0/2692333805 shutdown_connections 2024-04-02T14:36:08.025 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:08.023+0000 7f0911e00640 1 -- 172.21.15.90:0/2692333805 wait complete. 2024-04-02T14:36:08.052 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-02T14:36:08.052 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T14:36:08.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:08.329+0000 7fbee6c00640 1 -- 172.21.15.90:0/1803186978 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbee0076030 msgr2=0x7fbee0076410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:08.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:08.329+0000 7fbee6c00640 1 --2- 172.21.15.90:0/1803186978 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbee0076030 0x7fbee0076410 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fbed0009930 tx=0x7fbed002bc30 comp rx=0 tx=0).stop 2024-04-02T14:36:08.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:08.330+0000 7fbee6c00640 1 -- 172.21.15.90:0/1803186978 shutdown_connections 2024-04-02T14:36:08.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:08.330+0000 7fbee6c00640 1 --2- 172.21.15.90:0/1803186978 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbee0112930 0x7fbee0114d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:08.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:08.330+0000 7fbee6c00640 1 --2- 172.21.15.90:0/1803186978 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbee0076950 0x7fbee01123f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:08.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:08.330+0000 7fbee6c00640 1 --2- 172.21.15.90:0/1803186978 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbee0076030 0x7fbee0076410 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:08.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:08.330+0000 7fbee6c00640 1 -- 172.21.15.90:0/1803186978 >> 172.21.15.90:0/1803186978 conn(0x7fbee00ff760 msgr2=0x7fbee0101b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:36:08.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:08.330+0000 7fbee6c00640 1 -- 172.21.15.90:0/1803186978 shutdown_connections 2024-04-02T14:36:08.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:08.330+0000 7fbee6c00640 1 -- 172.21.15.90:0/1803186978 wait complete. 2024-04-02T14:36:08.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:08.331+0000 7fbee6c00640 1 Processor -- start 2024-04-02T14:36:08.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:08.332+0000 7fbee6c00640 1 -- start start 2024-04-02T14:36:08.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:08.333+0000 7fbee6c00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbee0076030 0x7fbee01a4640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:08.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:08.333+0000 7fbee6c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbee0076950 0x7fbee01a4b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:08.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:08.333+0000 7fbee6c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbee0112930 0x7fbee019e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:08.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:08.333+0000 7fbee6c00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fbee01184d0 con 0x7fbee0076950 2024-04-02T14:36:08.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:08.333+0000 7fbee6c00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fbee0118350 con 0x7fbee0076030 2024-04-02T14:36:08.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:08.333+0000 7fbee6c00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fbee0118650 con 0x7fbee0112930 2024-04-02T14:36:08.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:08.333+0000 7fbedfe00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbee0076030 0x7fbee01a4640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:08.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:08.333+0000 7fbedfe00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbee0076030 0x7fbee01a4640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:33016/0 (socket says 172.21.15.90:33016) 2024-04-02T14:36:08.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:08.333+0000 7fbedfe00640 1 -- 172.21.15.90:0/382079757 learned_addr learned my addr 172.21.15.90:0/382079757 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:36:08.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:08.334+0000 7fbedf400640 1 --2- 172.21.15.90:0/382079757 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbee0076950 0x7fbee01a4b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:08.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:08.334+0000 7fbedfe00640 1 -- 172.21.15.90:0/382079757 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbee0112930 msgr2=0x7fbee019e7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:08.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:08.334+0000 7fbedfe00640 1 --2- 172.21.15.90:0/382079757 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbee0112930 0x7fbee019e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:08.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:08.334+0000 7fbedfe00640 1 -- 172.21.15.90:0/382079757 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbee0076950 msgr2=0x7fbee01a4b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:08.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:08.334+0000 7fbedfe00640 1 --2- 172.21.15.90:0/382079757 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbee0076950 0x7fbee01a4b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:08.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:08.334+0000 7fbedfe00640 1 -- 172.21.15.90:0/382079757 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbee019ef80 con 0x7fbee0076030 2024-04-02T14:36:08.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:08.334+0000 7fbedf400640 1 --2- 172.21.15.90:0/382079757 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbee0076950 0x7fbee01a4b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:36:08.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:08.334+0000 7fbedfe00640 1 --2- 172.21.15.90:0/382079757 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbee0076030 0x7fbee01a4640 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fbed0009900 tx=0x7fbed0004430 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:36:08.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:08.334+0000 7fbedcc00640 1 -- 172.21.15.90:0/382079757 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbed0043020 con 0x7fbee0076030 2024-04-02T14:36:08.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:08.334+0000 7fbedcc00640 1 -- 172.21.15.90:0/382079757 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbed00054e0 con 0x7fbee0076030 2024-04-02T14:36:08.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:08.334+0000 7fbee6c00640 1 -- 172.21.15.90:0/382079757 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbee019f210 con 0x7fbee0076030 2024-04-02T14:36:08.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:08.335+0000 7fbedcc00640 1 -- 172.21.15.90:0/382079757 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbed0003d00 con 0x7fbee0076030 2024-04-02T14:36:08.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:08.336+0000 7fbee6c00640 1 -- 172.21.15.90:0/382079757 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbee01ab490 con 0x7fbee0076030 2024-04-02T14:36:08.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:08.340+0000 7fbedcc00640 1 -- 172.21.15.90:0/382079757 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 25) ==== 138930+0+0 (secure 0 0 0) 0x7fbed0050060 con 0x7fbee0076030 2024-04-02T14:36:08.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:08.341+0000 7fbee6c00640 1 -- 172.21.15.90:0/382079757 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbee0077770 con 0x7fbee0076030 2024-04-02T14:36:08.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:08.342+0000 7fbedcc00640 1 --2- 172.21.15.90:0/382079757 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fbec40a3fd0 0x7fbec40a6490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:08.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:08.342+0000 7fbedcc00640 1 -- 172.21.15.90:0/382079757 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7fbed004e070 con 0x7fbee0076030 2024-04-02T14:36:08.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:08.343+0000 7fbedf400640 1 --2- 172.21.15.90:0/382079757 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fbec40a3fd0 0x7fbec40a6490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:08.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:08.344+0000 7fbedf400640 1 --2- 172.21.15.90:0/382079757 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fbec40a3fd0 0x7fbec40a6490 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fbecc005d00 tx=0x7fbecc009550 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:36:08.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:08.346+0000 7fbedcc00640 1 -- 172.21.15.90:0/382079757 <== mon.2 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7fbed00beb90 con 0x7fbee0076030 2024-04-02T14:36:08.360 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:36:08 smithi092 ceph-mon[36497]: pgmap v40: 97 pgs: 97 active+clean; 1.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 47 MiB/s wr, 1.11k op/s 2024-04-02T14:36:08.436 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:36:08 smithi177 ceph-mon[36431]: pgmap v40: 97 pgs: 97 active+clean; 1.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 47 MiB/s wr, 1.11k op/s 2024-04-02T14:36:08.438 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:36:08 smithi090 ceph-mon[31674]: pgmap v40: 97 pgs: 97 active+clean; 1.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 47 MiB/s wr, 1.11k op/s 2024-04-02T14:36:08.815 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:08.814+0000 7fbee6c00640 1 -- 172.21.15.90:0/382079757 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbee010dac0 con 0x7fbee0076030 2024-04-02T14:36:08.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:08.815+0000 7fbedcc00640 1 -- 172.21.15.90:0/382079757 <== mon.2 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11927 (secure 0 0 0) 0x7fbed00be490 con 0x7fbee0076030 2024-04-02T14:36:08.817 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-02T14:36:08.817 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14412,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6837/2852160309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6836","nonce":2852160309},{"type":"v1","addr":"172.21.15.90:6837","nonce":2852160309}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14478,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6841/1919376858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6840","nonce":1919376858},{"type":"v1","addr":"172.21.15.90:6841","nonce":1919376858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-02T14:32:22.319083+0000","modified":"2024-04-02T14:34:52.266152+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24590,"mds_1":24560,"mds_2":24547,"mds_3":24539,"mds_4":24523},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24454":{"gid":24454,"name":"l","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6833/1564319431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":1564319431},{"type":"v1","addr":"172.21.15.177:6833","nonce":1564319431}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.92:6833/1570337372","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":1570337372},{"type":"v1","addr":"172.21.15.92:6833","nonce":1570337372}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"j","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.90:6835/1277124891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6834","nonce":1277124891},{"type":"v1","addr":"172.21.15.90:6835","nonce":1277124891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.92:6835/860721164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":860721164},{"type":"v1","addr":"172.21.15.92:6835","nonce":860721164}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24511":{"gid":24511,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6835/3177244988","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":3177244988},{"type":"v1","addr":"172.21.15.177:6835","nonce":3177244988}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"c","rank":4,"incarnation":25,"state":"up:active","state_seq":9,"addr":"172.21.15.177:6837/4223414858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":4223414858},{"type":"v1","addr":"172.21.15.177:6837","nonce":4223414858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24539":{"gid":24539,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":8,"addr":"172.21.15.92:6837/1270635939","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":1270635939},{"type":"v1","addr":"172.21.15.92:6837","nonce":1270635939}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24547":{"gid":24547,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":7,"addr":"172.21.15.177:6839/664895663","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":664895663},{"type":"v1","addr":"172.21.15.177:6839","nonce":664895663}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"g","rank":1,"incarnation":19,"state":"up:active","state_seq":5,"addr":"172.21.15.90:6839/1498915870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6838","nonce":1498915870},{"type":"v1","addr":"172.21.15.90:6839","nonce":1498915870}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24590":{"gid":24590,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.92:6839/1054888219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":1054888219},{"type":"v1","addr":"172.21.15.92:6839","nonce":1054888219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24590,"qdb_cluster":[24547,24590,24523,24539,24560]},"id":1}]} 2024-04-02T14:36:08.817 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 27 2024-04-02T14:36:08.820 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:08.819+0000 7fbee6c00640 1 -- 172.21.15.90:0/382079757 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fbec40a3fd0 msgr2=0x7fbec40a6490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:08.820 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:08.819+0000 7fbee6c00640 1 --2- 172.21.15.90:0/382079757 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fbec40a3fd0 0x7fbec40a6490 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fbecc005d00 tx=0x7fbecc009550 comp rx=0 tx=0).stop 2024-04-02T14:36:08.820 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:08.819+0000 7fbee6c00640 1 -- 172.21.15.90:0/382079757 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbee0076030 msgr2=0x7fbee01a4640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:08.820 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:08.819+0000 7fbee6c00640 1 --2- 172.21.15.90:0/382079757 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbee0076030 0x7fbee01a4640 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fbed0009900 tx=0x7fbed0004430 comp rx=0 tx=0).stop 2024-04-02T14:36:08.820 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:08.819+0000 7fbee6c00640 1 -- 172.21.15.90:0/382079757 shutdown_connections 2024-04-02T14:36:08.821 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:08.819+0000 7fbee6c00640 1 --2- 172.21.15.90:0/382079757 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fbec40a3fd0 0x7fbec40a6490 secure :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fbecc005d00 tx=0x7fbecc009550 comp rx=0 tx=0).stop 2024-04-02T14:36:08.821 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:08.819+0000 7fbee6c00640 1 --2- 172.21.15.90:0/382079757 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbee0112930 0x7fbee019e7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:08.821 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:08.819+0000 7fbee6c00640 1 --2- 172.21.15.90:0/382079757 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbee0076950 0x7fbee01a4b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:08.821 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:08.819+0000 7fbee6c00640 1 --2- 172.21.15.90:0/382079757 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbee0076030 0x7fbee01a4640 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:08.821 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:08.819+0000 7fbee6c00640 1 -- 172.21.15.90:0/382079757 >> 172.21.15.90:0/382079757 conn(0x7fbee00ff760 msgr2=0x7fbee0113f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:36:08.821 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:08.820+0000 7fbee6c00640 1 -- 172.21.15.90:0/382079757 shutdown_connections 2024-04-02T14:36:08.821 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:08.820+0000 7fbee6c00640 1 -- 172.21.15.90:0/382079757 wait complete. 2024-04-02T14:36:08.849 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-04-02T14:36:09.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.133+0000 7f97bf600640 1 -- 172.21.15.90:0/320903103 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f97b80747f0 msgr2=0x7f97b810ddf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:09.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.133+0000 7f97bf600640 1 --2- 172.21.15.90:0/320903103 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f97b80747f0 0x7f97b810ddf0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f97a4009900 tx=0x7f97a402bcd0 comp rx=0 tx=0).stop 2024-04-02T14:36:09.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.133+0000 7f97bf600640 1 -- 172.21.15.90:0/320903103 shutdown_connections 2024-04-02T14:36:09.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.133+0000 7f97bf600640 1 --2- 172.21.15.90:0/320903103 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f97b810e330 0x7f97b8110740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:09.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.133+0000 7f97bf600640 1 --2- 172.21.15.90:0/320903103 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f97b80747f0 0x7f97b810ddf0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:09.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.133+0000 7f97bf600640 1 --2- 172.21.15.90:0/320903103 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f97b8073ed0 0x7f97b80742b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:09.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.133+0000 7f97bf600640 1 -- 172.21.15.90:0/320903103 >> 172.21.15.90:0/320903103 conn(0x7f97b8078f00 msgr2=0x7f97b8079310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:36:09.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.133+0000 7f97bf600640 1 -- 172.21.15.90:0/320903103 shutdown_connections 2024-04-02T14:36:09.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.133+0000 7f97bf600640 1 -- 172.21.15.90:0/320903103 wait complete. 2024-04-02T14:36:09.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.135+0000 7f97bf600640 1 Processor -- start 2024-04-02T14:36:09.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.136+0000 7f97bf600640 1 -- start start 2024-04-02T14:36:09.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.136+0000 7f97bf600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f97b8073ed0 0x7f97b810dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:09.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.136+0000 7f97bf600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f97b80747f0 0x7f97b810ad50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:09.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.136+0000 7f97bf600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f97b810e330 0x7f97b810b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:09.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.136+0000 7f97bf600640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f97b81140e0 con 0x7f97b8073ed0 2024-04-02T14:36:09.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.136+0000 7f97bf600640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f97b8113f60 con 0x7f97b80747f0 2024-04-02T14:36:09.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.136+0000 7f97bf600640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f97b8114260 con 0x7f97b810e330 2024-04-02T14:36:09.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.136+0000 7f97b7e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f97b80747f0 0x7f97b810ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:09.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.136+0000 7f97b7e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f97b80747f0 0x7f97b810ad50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:33038/0 (socket says 172.21.15.90:33038) 2024-04-02T14:36:09.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.136+0000 7f97b7e00640 1 -- 172.21.15.90:0/3774576867 learned_addr learned my addr 172.21.15.90:0/3774576867 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:36:09.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.137+0000 7f97b7e00640 1 -- 172.21.15.90:0/3774576867 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f97b810e330 msgr2=0x7f97b810b2b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-02T14:36:09.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.137+0000 7f97b7e00640 1 --2- 172.21.15.90:0/3774576867 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f97b810e330 0x7f97b810b2b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:09.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.137+0000 7f97b7e00640 1 -- 172.21.15.90:0/3774576867 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f97b8073ed0 msgr2=0x7f97b810dcb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T14:36:09.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.137+0000 7f97bcc00640 1 --2- 172.21.15.90:0/3774576867 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f97b8073ed0 0x7f97b810dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:09.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.137+0000 7f97b7e00640 1 --2- 172.21.15.90:0/3774576867 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f97b8073ed0 0x7f97b810dcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:09.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.137+0000 7f97b7e00640 1 -- 172.21.15.90:0/3774576867 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97b810bb70 con 0x7f97b80747f0 2024-04-02T14:36:09.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.137+0000 7f97b7e00640 1 --2- 172.21.15.90:0/3774576867 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f97b80747f0 0x7f97b810ad50 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f97a4000c00 tx=0x7f97a40043c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:36:09.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.138+0000 7f97b5600640 1 -- 172.21.15.90:0/3774576867 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97a4042020 con 0x7f97b80747f0 2024-04-02T14:36:09.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.138+0000 7f97b5600640 1 -- 172.21.15.90:0/3774576867 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f97a40054e0 con 0x7f97b80747f0 2024-04-02T14:36:09.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.138+0000 7f97b5600640 1 -- 172.21.15.90:0/3774576867 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97a4003af0 con 0x7f97b80747f0 2024-04-02T14:36:09.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.138+0000 7f97bf600640 1 -- 172.21.15.90:0/3774576867 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97b81aaee0 con 0x7f97b80747f0 2024-04-02T14:36:09.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.138+0000 7f97bf600640 1 -- 172.21.15.90:0/3774576867 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97b81ab350 con 0x7f97b80747f0 2024-04-02T14:36:09.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.145+0000 7f97c2412740 1 Processor -- start 2024-04-02T14:36:09.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.145+0000 7f97c2412740 1 -- start start 2024-04-02T14:36:09.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.146+0000 7f97b5600640 1 -- 172.21.15.90:0/3774576867 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 25) ==== 138930+0+0 (secure 0 0 0) 0x7f97a4045020 con 0x7f97b80747f0 2024-04-02T14:36:09.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.146+0000 7f97c2412740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5604320449f0 0x560432043740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:09.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.146+0000 7f97c2412740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560432043d10 0x560432044190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:09.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.146+0000 7f97c2412740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56043204d080 0x56043204f470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:09.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.146+0000 7f97c2412740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x560431f84400 con 0x56043204d080 2024-04-02T14:36:09.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.146+0000 7f97c2412740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x560431f68f10 con 0x5604320449f0 2024-04-02T14:36:09.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.146+0000 7f97c2412740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x560431fa86f0 con 0x560432043d10 2024-04-02T14:36:09.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.146+0000 7f97bd600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5604320449f0 0x560432043740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:09.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.147+0000 7f97bcc00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560432043d10 0x560432044190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:09.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.147+0000 7f97bcc00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560432043d10 0x560432044190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:53742/0 (socket says 172.21.15.90:53742) 2024-04-02T14:36:09.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.147+0000 7f97bcc00640 1 -- 172.21.15.90:0/2297925457 learned_addr learned my addr 172.21.15.90:0/2297925457 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:36:09.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.147+0000 7f97b7e00640 1 --2- 172.21.15.90:0/2297925457 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56043204d080 0x56043204f470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:09.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.147+0000 7f97bd600640 1 -- 172.21.15.90:0/2297925457 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560432043d10 msgr2=0x560432044190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:36:09.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.147+0000 7f97bd600640 1 --2- 172.21.15.90:0/2297925457 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560432043d10 0x560432044190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:09.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.147+0000 7f97bd600640 1 -- 172.21.15.90:0/2297925457 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56043204d080 msgr2=0x56043204f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:36:09.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.147+0000 7f97bd600640 1 --2- 172.21.15.90:0/2297925457 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56043204d080 0x56043204f470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:09.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.148+0000 7f97bd600640 1 -- 172.21.15.90:0/2297925457 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560431cbc3f0 con 0x5604320449f0 2024-04-02T14:36:09.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.149+0000 7f97bd600640 1 --2- 172.21.15.90:0/2297925457 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5604320449f0 0x560432043740 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f97b806b3a0 tx=0x7f97b000e720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=70a961893de14a0e server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:36:09.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.149+0000 7f97a9c00640 1 -- 172.21.15.90:0/2297925457 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97b0004b20 con 0x5604320449f0 2024-04-02T14:36:09.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.149+0000 7f97a9c00640 1 -- 172.21.15.90:0/2297925457 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f97b0003e50 con 0x5604320449f0 2024-04-02T14:36:09.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.149+0000 7f97a9c00640 1 -- 172.21.15.90:0/2297925457 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97b0004710 con 0x5604320449f0 2024-04-02T14:36:09.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.149+0000 7f97b5600640 1 --2- 172.21.15.90:0/3774576867 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f97a00a3fd0 0x7f97a00a6490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:09.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.150+0000 7f97bcc00640 1 --2- 172.21.15.90:0/3774576867 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f97a00a3fd0 0x7f97a00a6490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:09.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.150+0000 7f97c2412740 1 -- 172.21.15.90:0/2297925457 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5604320449f0 msgr2=0x560432043740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:09.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.150+0000 7f97c2412740 1 --2- 172.21.15.90:0/2297925457 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5604320449f0 0x560432043740 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f97b806b3a0 tx=0x7f97b000e720 comp rx=0 tx=0).stop 2024-04-02T14:36:09.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.150+0000 7f97b5600640 1 -- 172.21.15.90:0/3774576867 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f97a40f2740 con 0x7f97b80747f0 2024-04-02T14:36:09.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.151+0000 7f97c2412740 1 -- 172.21.15.90:0/2297925457 shutdown_connections 2024-04-02T14:36:09.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.151+0000 7f97c2412740 1 --2- 172.21.15.90:0/2297925457 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56043204d080 0x56043204f470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:09.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.151+0000 7f97c2412740 1 --2- 172.21.15.90:0/2297925457 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560432043d10 0x560432044190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:09.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.151+0000 7f97c2412740 1 --2- 172.21.15.90:0/2297925457 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5604320449f0 0x560432043740 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:09.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.151+0000 7f97c2412740 1 -- 172.21.15.90:0/2297925457 >> 172.21.15.90:0/2297925457 conn(0x56043200aa50 msgr2=0x56043203e760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:36:09.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.152+0000 7f97bcc00640 1 --2- 172.21.15.90:0/3774576867 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f97a00a3fd0 0x7f97a00a6490 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f97ac005fd0 tx=0x7f97ac0094e0 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:36:09.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.152+0000 7f97c2412740 1 -- 172.21.15.90:0/2297925457 shutdown_connections 2024-04-02T14:36:09.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.152+0000 7f97c2412740 1 -- 172.21.15.90:0/2297925457 wait complete. 2024-04-02T14:36:09.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.153+0000 7f97c2412740 1 -- start start 2024-04-02T14:36:09.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.153+0000 7f97c2412740 1 Processor -- start 2024-04-02T14:36:09.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.154+0000 7f97c2412740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560432043d10 0x560432049f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:09.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.154+0000 7f97c2412740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5604320449f0 0x56043204a440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:09.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.155+0000 7f97c2412740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56043204d080 0x56043204a980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:09.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.155+0000 7f97b7e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5604320449f0 0x56043204a440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:09.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.155+0000 7f97b7e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5604320449f0 0x56043204a440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:53752/0 (socket says 172.21.15.90:53752) 2024-04-02T14:36:09.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.155+0000 7f97b7e00640 1 -- 172.21.15.90:0/2254509951 learned_addr learned my addr 172.21.15.90:0/2254509951 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:36:09.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.155+0000 7f97c2412740 1 -- 172.21.15.90:0/2254509951 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x560431fa46a0 con 0x560432043d10 2024-04-02T14:36:09.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.155+0000 7f97c2412740 1 -- 172.21.15.90:0/2254509951 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x560431f50980 con 0x56043204d080 2024-04-02T14:36:09.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.155+0000 7f97c2412740 1 -- 172.21.15.90:0/2254509951 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x560431f63cb0 con 0x5604320449f0 2024-04-02T14:36:09.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.155+0000 7f97bcc00640 1 --2- 172.21.15.90:0/2254509951 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560432043d10 0x560432049f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:09.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.155+0000 7f97bd600640 1 --2- 172.21.15.90:0/2254509951 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56043204d080 0x56043204a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:09.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.156+0000 7f97c2412740 10 client.0 _ll_register_callbacks cb 0x560431f636a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:36:09.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.156+0000 7f97b7e00640 1 -- 172.21.15.90:0/2254509951 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56043204d080 msgr2=0x56043204a980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:36:09.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.156+0000 7f97b7e00640 1 --2- 172.21.15.90:0/2254509951 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56043204d080 0x56043204a980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:09.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.156+0000 7f97b7e00640 1 -- 172.21.15.90:0/2254509951 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560432043d10 msgr2=0x560432049f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:36:09.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.156+0000 7f97b7e00640 1 --2- 172.21.15.90:0/2254509951 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560432043d10 0x560432049f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:09.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.156+0000 7f97b7e00640 1 -- 172.21.15.90:0/2254509951 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560431d867d0 con 0x5604320449f0 2024-04-02T14:36:09.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.156+0000 7f97bd600640 1 --2- 172.21.15.90:0/2254509951 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56043204d080 0x56043204a980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:36:09.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.156+0000 7f979e000640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:36:09.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.157+0000 7f97b7e00640 1 --2- 172.21.15.90:0/2254509951 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5604320449f0 0x56043204a440 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f97a40baa20 tx=0x7f97a4009260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fae5da1f2512ff1a server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:36:09.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.157+0000 7f97c2412740 1 -- 172.21.15.90:0/2254509951 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f97b0003e50 con 0x5604320449f0 2024-04-02T14:36:09.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.157+0000 7f979e000640 1 -- 172.21.15.90:0/2254509951 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97a40bdf20 con 0x5604320449f0 2024-04-02T14:36:09.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.157+0000 7f979e000640 1 -- 172.21.15.90:0/2254509951 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f97a40be820 con 0x5604320449f0 2024-04-02T14:36:09.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.157+0000 7f979e000640 1 -- 172.21.15.90:0/2254509951 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f97a40bea60 con 0x5604320449f0 2024-04-02T14:36:09.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.157+0000 7f97c2412740 10 client.25708 fetch_fsmap learned FSMap version 27 2024-04-02T14:36:09.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.157+0000 7f97c2412740 1 -- 172.21.15.90:0/2254509951 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({fsmap=27}) -- 0x56043205bd80 con 0x5604320449f0 2024-04-02T14:36:09.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.158+0000 7f979e000640 1 -- 172.21.15.90:0/2254509951 <== mon.1 v2:172.21.15.177:3300/0 4 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f97a40c6da0 con 0x5604320449f0 2024-04-02T14:36:09.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.158+0000 7f97c2412740 10 client.25708 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:36:09.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.158+0000 7f97c2412740 10 client.25708 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.c 2024-04-02T14:36:09.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.158+0000 7f97c2412740 20 client.25708 populate_metadata read hostname 'smithi090' 2024-04-02T14:36:09.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.158+0000 7f97c2412740 1 --2- 172.21.15.90:0/2254509951 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x56043205d0d0 0x56043205f500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:09.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.158+0000 7f97c2412740 4 client.25708 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:36:09.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.158+0000 7f97c2412740 1 -- 172.21.15.90:0/2254509951 --> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560431f68040 con 0x56043205d0d0 2024-04-02T14:36:09.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.159+0000 7f97bcc00640 1 --2- 172.21.15.90:0/2254509951 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x56043205d0d0 0x56043205f500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:09.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.160+0000 7f97bcc00640 1 --2- 172.21.15.90:0/2254509951 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x56043205d0d0 0x56043205f500 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=178c54d49df99cbb server_cookie=26f6b2ee80e06fb5 in_seq=0 out_seq=0 2024-04-02T14:36:09.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.160+0000 7f979e000640 10 client.25708 ms_handle_connect on v2:172.21.15.177:6836/4223414858 2024-04-02T14:36:09.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.160+0000 7f979e000640 1 -- 172.21.15.90:0/2254509951 <== mds.4 v2:172.21.15.177:6836/4223414858 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x560431f68040 con 0x56043205d0d0 2024-04-02T14:36:09.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.160+0000 7f979e000640 10 client.25708 handle_command_reply: tid=0 2024-04-02T14:36:09.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.161+0000 7f97c2412740 1 client.25708 shutdown 2024-04-02T14:36:09.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.161+0000 7f97c2412740 1 -- 172.21.15.90:0/2254509951 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5604320449f0 msgr2=0x56043204a440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:09.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.161+0000 7f97c2412740 1 --2- 172.21.15.90:0/2254509951 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5604320449f0 0x56043204a440 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f97a40baa20 tx=0x7f97a4009260 comp rx=0 tx=0).stop 2024-04-02T14:36:09.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.162+0000 7f97c2412740 1 -- 172.21.15.90:0/2254509951 shutdown_connections 2024-04-02T14:36:09.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.162+0000 7f97c2412740 1 --2- 172.21.15.90:0/2254509951 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56043204d080 0x56043204a980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:09.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.162+0000 7f97c2412740 1 --2- 172.21.15.90:0/2254509951 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5604320449f0 0x56043204a440 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:09.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.162+0000 7f97c2412740 1 --2- 172.21.15.90:0/2254509951 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x56043205d0d0 0x56043205f500 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:09.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.162+0000 7f97c2412740 1 --2- 172.21.15.90:0/2254509951 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560432043d10 0x560432049f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:09.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.162+0000 7f979e000640 0 client.25708 ms_handle_reset on v2:172.21.15.177:6836/4223414858 2024-04-02T14:36:09.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.162+0000 7f97c2412740 1 -- 172.21.15.90:0/2254509951 >> 172.21.15.90:0/2254509951 conn(0x56043200aa50 msgr2=0x5604320404b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:36:09.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.162+0000 7f97c2412740 1 -- 172.21.15.90:0/2254509951 shutdown_connections 2024-04-02T14:36:09.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.163+0000 7f97c2412740 1 -- 172.21.15.90:0/2254509951 wait complete. 2024-04-02T14:36:09.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.163+0000 7f97c2412740 20 client.25708 trim_cache size 0 max 16384 2024-04-02T14:36:09.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:36:09 smithi090 ceph-mon[31674]: from='client.? 172.21.15.90:0/382079757' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-02T14:36:09.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.204+0000 7f97c2412740 1 Processor -- start 2024-04-02T14:36:09.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.204+0000 7f97c2412740 1 -- start start 2024-04-02T14:36:09.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.204+0000 7f97c2412740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560432043d10 0x56043204c630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:09.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.204+0000 7f97c2412740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5604320449f0 0x560432058bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:09.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.205+0000 7f97c2412740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56043204d080 0x560432048ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:09.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.205+0000 7f97c2412740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f97b81b3f40 con 0x560432043d10 2024-04-02T14:36:09.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.205+0000 7f97c2412740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f97b81b5a80 con 0x56043204d080 2024-04-02T14:36:09.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.205+0000 7f97c2412740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f97b81b4610 con 0x5604320449f0 2024-04-02T14:36:09.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.205+0000 7f97bd600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56043204d080 0x560432048ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:09.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.205+0000 7f97bd600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56043204d080 0x560432048ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:33072/0 (socket says 172.21.15.90:33072) 2024-04-02T14:36:09.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.205+0000 7f97bd600640 1 -- 172.21.15.90:0/548469640 learned_addr learned my addr 172.21.15.90:0/548469640 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:36:09.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.205+0000 7f97bcc00640 1 --2- 172.21.15.90:0/548469640 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560432043d10 0x56043204c630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:09.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.205+0000 7f97b7e00640 1 --2- 172.21.15.90:0/548469640 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5604320449f0 0x560432058bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:09.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.206+0000 7f97bd600640 1 -- 172.21.15.90:0/548469640 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5604320449f0 msgr2=0x560432058bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:36:09.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.206+0000 7f97bd600640 1 --2- 172.21.15.90:0/548469640 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5604320449f0 0x560432058bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:09.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.206+0000 7f97bd600640 1 -- 172.21.15.90:0/548469640 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560432043d10 msgr2=0x56043204c630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:36:09.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.206+0000 7f97bd600640 1 --2- 172.21.15.90:0/548469640 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560432043d10 0x56043204c630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:09.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.206+0000 7f97bd600640 1 -- 172.21.15.90:0/548469640 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56043205fb80 con 0x56043204d080 2024-04-02T14:36:09.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.206+0000 7f97bcc00640 1 --2- 172.21.15.90:0/548469640 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560432043d10 0x56043204c630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:36:09.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.206+0000 7f97b7e00640 1 --2- 172.21.15.90:0/548469640 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5604320449f0 0x560432058bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:36:09.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.207+0000 7f97bd600640 1 --2- 172.21.15.90:0/548469640 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56043204d080 0x560432048ee0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f97b0000f80 tx=0x7f97b0009ee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=41a91454804867e0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:36:09.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.207+0000 7f97a9200640 1 -- 172.21.15.90:0/548469640 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97b0005440 con 0x56043204d080 2024-04-02T14:36:09.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.207+0000 7f97a9200640 1 -- 172.21.15.90:0/548469640 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f97b0018470 con 0x56043204d080 2024-04-02T14:36:09.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.207+0000 7f97a9200640 1 -- 172.21.15.90:0/548469640 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97b0010670 con 0x56043204d080 2024-04-02T14:36:09.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.208+0000 7f97c2412740 1 -- 172.21.15.90:0/548469640 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56043204d080 msgr2=0x560432048ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:09.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.208+0000 7f97c2412740 1 --2- 172.21.15.90:0/548469640 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56043204d080 0x560432048ee0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f97b0000f80 tx=0x7f97b0009ee0 comp rx=0 tx=0).stop 2024-04-02T14:36:09.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.209+0000 7f97c2412740 1 -- 172.21.15.90:0/548469640 shutdown_connections 2024-04-02T14:36:09.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.209+0000 7f97c2412740 1 --2- 172.21.15.90:0/548469640 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56043204d080 0x560432048ee0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:09.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.209+0000 7f97c2412740 1 --2- 172.21.15.90:0/548469640 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5604320449f0 0x560432058bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:09.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.209+0000 7f97c2412740 1 --2- 172.21.15.90:0/548469640 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560432043d10 0x56043204c630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:09.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.209+0000 7f97c2412740 1 -- 172.21.15.90:0/548469640 >> 172.21.15.90:0/548469640 conn(0x56043200aa50 msgr2=0x5604320487d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:36:09.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.209+0000 7f97c2412740 1 -- 172.21.15.90:0/548469640 shutdown_connections 2024-04-02T14:36:09.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.210+0000 7f97c2412740 1 -- 172.21.15.90:0/548469640 wait complete. 2024-04-02T14:36:09.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.211+0000 7f97c2412740 1 -- start start 2024-04-02T14:36:09.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.211+0000 7f97c2412740 1 Processor -- start 2024-04-02T14:36:09.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.212+0000 7f97c2412740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560432043d10 0x560432055430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:09.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.212+0000 7f97c2412740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5604320449f0 0x560432047180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:09.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.212+0000 7f97c2412740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56043204d080 0x5604320476c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:09.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.212+0000 7f97c2412740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f97b81b53b0 con 0x5604320449f0 2024-04-02T14:36:09.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.212+0000 7f97c2412740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f97b81b4ce0 con 0x56043204d080 2024-04-02T14:36:09.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.212+0000 7f97c2412740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f97b81b1f20 con 0x560432043d10 2024-04-02T14:36:09.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.212+0000 7f97c2412740 10 client.0 _ll_register_callbacks cb 0x560431f636a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:36:09.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.213+0000 7f97bcc00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560432043d10 0x560432055430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:09.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.213+0000 7f97b7e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5604320449f0 0x560432047180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:09.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.213+0000 7f97bd600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56043204d080 0x5604320476c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:09.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.213+0000 7f97bcc00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560432043d10 0x560432055430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:53770/0 (socket says 172.21.15.90:53770) 2024-04-02T14:36:09.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.213+0000 7f97bcc00640 1 -- 172.21.15.90:0/2627499289 learned_addr learned my addr 172.21.15.90:0/2627499289 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:36:09.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.213+0000 7f97bd600640 1 -- 172.21.15.90:0/2627499289 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560432043d10 msgr2=0x560432055430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:36:09.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.213+0000 7f97bd600640 1 --2- 172.21.15.90:0/2627499289 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560432043d10 0x560432055430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:09.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.213+0000 7f97bd600640 1 -- 172.21.15.90:0/2627499289 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5604320449f0 msgr2=0x560432047180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:36:09.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.213+0000 7f97bd600640 1 --2- 172.21.15.90:0/2627499289 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5604320449f0 0x560432047180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:09.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.213+0000 7f97bd600640 1 -- 172.21.15.90:0/2627499289 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560432047ef0 con 0x56043204d080 2024-04-02T14:36:09.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.213+0000 7f97b7e00640 1 --2- 172.21.15.90:0/2627499289 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5604320449f0 0x560432047180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:36:09.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.214+0000 7f97bd600640 1 --2- 172.21.15.90:0/2627499289 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56043204d080 0x5604320476c0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f97b0005fd0 tx=0x7f97b0005670 comp rx=0 tx=0).ready entity=mon.2 client_cookie=846ccf9775b0dbac server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:36:09.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.214+0000 7f979e000640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:36:09.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.214+0000 7f979e000640 1 -- 172.21.15.90:0/2627499289 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97b0003c80 con 0x56043204d080 2024-04-02T14:36:09.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.214+0000 7f979e000640 1 -- 172.21.15.90:0/2627499289 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f97b00059f0 con 0x56043204d080 2024-04-02T14:36:09.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.214+0000 7f97c2412740 1 -- 172.21.15.90:0/2627499289 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f97b0018470 con 0x56043204d080 2024-04-02T14:36:09.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.214+0000 7f979e000640 1 -- 172.21.15.90:0/2627499289 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f97b0010430 con 0x56043204d080 2024-04-02T14:36:09.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.214+0000 7f97c2412740 10 client.25769 fetch_fsmap learned FSMap version 27 2024-04-02T14:36:09.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.214+0000 7f97c2412740 1 -- 172.21.15.90:0/2627499289 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5604320c3680 con 0x56043204d080 2024-04-02T14:36:09.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.215+0000 7f979e000640 1 -- 172.21.15.90:0/2627499289 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f97b0018470 con 0x56043204d080 2024-04-02T14:36:09.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.215+0000 7f979e000640 1 -- 172.21.15.90:0/2627499289 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f97b0036570 con 0x56043204d080 2024-04-02T14:36:09.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.215+0000 7f97c2412740 10 client.25769 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:36:09.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.215+0000 7f97c2412740 10 client.25769 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.c 2024-04-02T14:36:09.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.215+0000 7f97c2412740 20 client.25769 populate_metadata read hostname 'smithi090' 2024-04-02T14:36:09.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.215+0000 7f97c2412740 1 --2- 172.21.15.90:0/2627499289 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x5604320c3a00 0x5604320c5df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:09.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.216+0000 7f97c2412740 4 client.25769 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-02T14:36:09.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.216+0000 7f97c2412740 1 -- 172.21.15.90:0/2627499289 --> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] -- command(tid 0: {"prefix": "damage ls"}) -- 0x560431fb5c30 con 0x5604320c3a00 2024-04-02T14:36:09.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.216+0000 7f97bcc00640 1 --2- 172.21.15.90:0/2627499289 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x5604320c3a00 0x5604320c5df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:09.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.217+0000 7f979e000640 10 client.25769 ms_handle_connect on v2:172.21.15.177:6836/4223414858 2024-04-02T14:36:09.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.217+0000 7f97bcc00640 1 --2- 172.21.15.90:0/2627499289 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x5604320c3a00 0x5604320c5df0 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=82e14d7e1fca3237 server_cookie=9dfc0683a231dbc1 in_seq=0 out_seq=0 2024-04-02T14:36:09.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.217+0000 7f979e000640 1 -- 172.21.15.90:0/2627499289 <== mds.4 v2:172.21.15.177:6836/4223414858 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x560431fb5c30 con 0x5604320c3a00 2024-04-02T14:36:09.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.217+0000 7f979e000640 10 client.25769 handle_command_reply: tid=0 2024-04-02T14:36:09.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.217+0000 7f97c2412740 1 client.25769 shutdown 2024-04-02T14:36:09.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.218+0000 7f97c2412740 1 -- 172.21.15.90:0/2627499289 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56043204d080 msgr2=0x5604320476c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:09.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.218+0000 7f97c2412740 1 --2- 172.21.15.90:0/2627499289 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56043204d080 0x5604320476c0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f97b0005fd0 tx=0x7f97b0005670 comp rx=0 tx=0).stop 2024-04-02T14:36:09.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.218+0000 7f97c2412740 1 -- 172.21.15.90:0/2627499289 shutdown_connections 2024-04-02T14:36:09.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.218+0000 7f97c2412740 1 --2- 172.21.15.90:0/2627499289 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56043204d080 0x5604320476c0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:09.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.218+0000 7f97c2412740 1 --2- 172.21.15.90:0/2627499289 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x5604320c3a00 0x5604320c5df0 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:09.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.218+0000 7f97c2412740 1 --2- 172.21.15.90:0/2627499289 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5604320449f0 0x560432047180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:09.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.218+0000 7f97c2412740 1 --2- 172.21.15.90:0/2627499289 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560432043d10 0x560432055430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:09.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.218+0000 7f97c2412740 1 -- 172.21.15.90:0/2627499289 >> 172.21.15.90:0/2627499289 conn(0x56043200aa50 msgr2=0x560432049ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:36:09.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.219+0000 7f979e000640 0 client.25769 ms_handle_reset on v2:172.21.15.177:6836/4223414858 2024-04-02T14:36:09.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.219+0000 7f97c2412740 1 -- 172.21.15.90:0/2627499289 shutdown_connections 2024-04-02T14:36:09.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.219+0000 7f97c2412740 1 -- 172.21.15.90:0/2627499289 wait complete. 2024-04-02T14:36:09.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.220+0000 7f97c2412740 20 client.25769 trim_cache size 0 max 16384 2024-04-02T14:36:09.221 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-02T14:36:09.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.220+0000 7f97bf600640 1 -- 172.21.15.90:0/3774576867 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f97a00a3fd0 msgr2=0x7f97a00a6490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:09.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.220+0000 7f97bf600640 1 --2- 172.21.15.90:0/3774576867 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f97a00a3fd0 0x7f97a00a6490 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f97ac005fd0 tx=0x7f97ac0094e0 comp rx=0 tx=0).stop 2024-04-02T14:36:09.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.220+0000 7f97bf600640 1 -- 172.21.15.90:0/3774576867 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f97b80747f0 msgr2=0x7f97b810ad50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:09.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.220+0000 7f97bf600640 1 --2- 172.21.15.90:0/3774576867 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f97b80747f0 0x7f97b810ad50 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f97a4000c00 tx=0x7f97a40043c0 comp rx=0 tx=0).stop 2024-04-02T14:36:09.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.221+0000 7f97bf600640 1 -- 172.21.15.90:0/3774576867 shutdown_connections 2024-04-02T14:36:09.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.221+0000 7f97bf600640 1 --2- 172.21.15.90:0/3774576867 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f97a00a3fd0 0x7f97a00a6490 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:09.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.221+0000 7f97bf600640 1 --2- 172.21.15.90:0/3774576867 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f97b810e330 0x7f97b810b2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:09.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.221+0000 7f97bf600640 1 --2- 172.21.15.90:0/3774576867 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f97b80747f0 0x7f97b810ad50 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:09.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.221+0000 7f97bf600640 1 --2- 172.21.15.90:0/3774576867 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f97b8073ed0 0x7f97b810dcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:09.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.221+0000 7f97bf600640 1 -- 172.21.15.90:0/3774576867 >> 172.21.15.90:0/3774576867 conn(0x7f97b8078f00 msgr2=0x7f97b80ffbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:36:09.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.221+0000 7f97bf600640 1 -- 172.21.15.90:0/3774576867 shutdown_connections 2024-04-02T14:36:09.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.221+0000 7f97bf600640 1 -- 172.21.15.90:0/3774576867 wait complete. 2024-04-02T14:36:09.248 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-04-02T14:36:09.360 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:36:09 smithi092 ceph-mon[36497]: from='client.? 172.21.15.90:0/382079757' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-02T14:36:09.436 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:36:09 smithi177 ceph-mon[36431]: from='client.? 172.21.15.90:0/382079757' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-02T14:36:09.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.524+0000 7f9cadc00640 1 -- 172.21.15.90:0/124094323 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9ca8068ce0 msgr2=0x7f9ca80690c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:09.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.524+0000 7f9cadc00640 1 --2- 172.21.15.90:0/124094323 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9ca8068ce0 0x7f9ca80690c0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f9c98009860 tx=0x7f9c9802bb70 comp rx=0 tx=0).stop 2024-04-02T14:36:09.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.524+0000 7f9cadc00640 1 -- 172.21.15.90:0/124094323 shutdown_connections 2024-04-02T14:36:09.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.524+0000 7f9cadc00640 1 --2- 172.21.15.90:0/124094323 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9ca810e4a0 0x7f9ca8110890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:09.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.524+0000 7f9cadc00640 1 --2- 172.21.15.90:0/124094323 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9ca8069600 0x7f9ca810df60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:09.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.524+0000 7f9cadc00640 1 --2- 172.21.15.90:0/124094323 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9ca8068ce0 0x7f9ca80690c0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:09.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.524+0000 7f9cadc00640 1 -- 172.21.15.90:0/124094323 >> 172.21.15.90:0/124094323 conn(0x7f9ca8077b00 msgr2=0x7f9ca8077f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:36:09.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.525+0000 7f9cadc00640 1 -- 172.21.15.90:0/124094323 shutdown_connections 2024-04-02T14:36:09.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.525+0000 7f9cadc00640 1 -- 172.21.15.90:0/124094323 wait complete. 2024-04-02T14:36:09.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.526+0000 7f9cadc00640 1 Processor -- start 2024-04-02T14:36:09.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.526+0000 7f9cadc00640 1 -- start start 2024-04-02T14:36:09.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.527+0000 7f9cadc00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9ca8068ce0 0x7f9ca8073350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:09.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.527+0000 7f9cadc00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9ca8069600 0x7f9ca806c3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:09.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.527+0000 7f9cadc00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9ca810e4a0 0x7f9ca806c910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:09.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.527+0000 7f9ca6a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9ca8068ce0 0x7f9ca8073350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:09.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.527+0000 7f9ca6a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9ca8068ce0 0x7f9ca8073350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:53788/0 (socket says 172.21.15.90:53788) 2024-04-02T14:36:09.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.527+0000 7f9c9fe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9ca8069600 0x7f9ca806c3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:09.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.528+0000 7f9c9fe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9ca8069600 0x7f9ca806c3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:46028/0 (socket says 172.21.15.90:46028) 2024-04-02T14:36:09.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.528+0000 7f9ca6a00640 1 -- 172.21.15.90:0/2485463657 learned_addr learned my addr 172.21.15.90:0/2485463657 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:36:09.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.528+0000 7f9cadc00640 1 -- 172.21.15.90:0/2485463657 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f9ca8114060 con 0x7f9ca8069600 2024-04-02T14:36:09.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.528+0000 7f9cadc00640 1 -- 172.21.15.90:0/2485463657 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f9ca8113ee0 con 0x7f9ca810e4a0 2024-04-02T14:36:09.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.528+0000 7f9cadc00640 1 -- 172.21.15.90:0/2485463657 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f9ca81141e0 con 0x7f9ca8068ce0 2024-04-02T14:36:09.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.528+0000 7f9ca7400640 1 --2- 172.21.15.90:0/2485463657 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9ca810e4a0 0x7f9ca806c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:09.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.528+0000 7f9ca7400640 1 -- 172.21.15.90:0/2485463657 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9ca8068ce0 msgr2=0x7f9ca8073350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:09.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.528+0000 7f9ca7400640 1 --2- 172.21.15.90:0/2485463657 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9ca8068ce0 0x7f9ca8073350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:09.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.528+0000 7f9ca7400640 1 -- 172.21.15.90:0/2485463657 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9ca8069600 msgr2=0x7f9ca806c3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:09.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.528+0000 7f9ca7400640 1 --2- 172.21.15.90:0/2485463657 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9ca8069600 0x7f9ca806c3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:09.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.528+0000 7f9ca7400640 1 -- 172.21.15.90:0/2485463657 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ca806cf50 con 0x7f9ca810e4a0 2024-04-02T14:36:09.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.528+0000 7f9ca6a00640 1 --2- 172.21.15.90:0/2485463657 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9ca8068ce0 0x7f9ca8073350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:36:09.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.529+0000 7f9ca7400640 1 --2- 172.21.15.90:0/2485463657 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9ca810e4a0 0x7f9ca806c910 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f9ca0005fd0 tx=0x7f9ca000c800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:36:09.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.529+0000 7f9c9f400640 1 -- 172.21.15.90:0/2485463657 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ca0003970 con 0x7f9ca810e4a0 2024-04-02T14:36:09.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.529+0000 7f9cadc00640 1 -- 172.21.15.90:0/2485463657 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ca8070e70 con 0x7f9ca810e4a0 2024-04-02T14:36:09.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.529+0000 7f9cadc00640 1 -- 172.21.15.90:0/2485463657 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ca80713c0 con 0x7f9ca810e4a0 2024-04-02T14:36:09.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.529+0000 7f9c9f400640 1 -- 172.21.15.90:0/2485463657 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9ca0004020 con 0x7f9ca810e4a0 2024-04-02T14:36:09.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.529+0000 7f9c9f400640 1 -- 172.21.15.90:0/2485463657 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ca000b9f0 con 0x7f9ca810e4a0 2024-04-02T14:36:09.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.533+0000 7f9c9f400640 1 -- 172.21.15.90:0/2485463657 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 25) ==== 138930+0+0 (secure 0 0 0) 0x7f9ca001d020 con 0x7f9ca810e4a0 2024-04-02T14:36:09.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.534+0000 7f9cb0995740 1 Processor -- start 2024-04-02T14:36:09.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.534+0000 7f9cb0995740 1 -- start start 2024-04-02T14:36:09.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.534+0000 7f9cb0995740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561ea1f1d9f0 0x561ea1f1c740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:09.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.534+0000 7f9cb0995740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561ea1f1cd10 0x561ea1f1d190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:09.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.535+0000 7f9cb0995740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561ea1f2e890 0x561ea1f30c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:09.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.535+0000 7f9cb0995740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x561ea1e5d400 con 0x561ea1f1d9f0 2024-04-02T14:36:09.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.535+0000 7f9cb0995740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x561ea1e41f10 con 0x561ea1f1cd10 2024-04-02T14:36:09.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.535+0000 7f9cb0995740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x561ea1e816f0 con 0x561ea1f2e890 2024-04-02T14:36:09.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.535+0000 7f9c9fe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561ea1f1d9f0 0x561ea1f1c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:09.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.535+0000 7f9c9fe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561ea1f1d9f0 0x561ea1f1c740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:46042/0 (socket says 172.21.15.90:46042) 2024-04-02T14:36:09.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.535+0000 7f9c9fe00640 1 -- 172.21.15.90:0/1097772230 learned_addr learned my addr 172.21.15.90:0/1097772230 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:36:09.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.535+0000 7f9ca6a00640 1 --2- 172.21.15.90:0/1097772230 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561ea1f1cd10 0x561ea1f1d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:09.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.536+0000 7f9c9f400640 1 --2- 172.21.15.90:0/2485463657 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f9c8c0a3f50 0x7f9c8c0a6410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:09.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.536+0000 7f9c9f400640 1 -- 172.21.15.90:0/2485463657 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f9ca00d0830 con 0x7f9ca810e4a0 2024-04-02T14:36:09.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.536+0000 7f9ca6a00640 1 -- 172.21.15.90:0/1097772230 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561ea1f2e890 msgr2=0x561ea1f30c80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-02T14:36:09.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.536+0000 7f9ca6a00640 1 --2- 172.21.15.90:0/1097772230 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561ea1f2e890 0x561ea1f30c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:09.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.536+0000 7f9ca6a00640 1 -- 172.21.15.90:0/1097772230 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561ea1f1d9f0 msgr2=0x561ea1f1c740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:36:09.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.536+0000 7f9ca6a00640 1 --2- 172.21.15.90:0/1097772230 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561ea1f1d9f0 0x561ea1f1c740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:09.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.536+0000 7f9ca6a00640 1 -- 172.21.15.90:0/1097772230 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561ea1b953f0 con 0x561ea1f1cd10 2024-04-02T14:36:09.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.536+0000 7f9c9fe00640 1 --2- 172.21.15.90:0/1097772230 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561ea1f1d9f0 0x561ea1f1c740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:36:09.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.537+0000 7f9ca7400640 1 --2- 172.21.15.90:0/2485463657 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f9c8c0a3f50 0x7f9c8c0a6410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:09.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.537+0000 7f9ca6a00640 1 --2- 172.21.15.90:0/1097772230 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561ea1f1cd10 0x561ea1f1d190 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f9c98009830 tx=0x7f9c980056d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=319e3057ca77b7a2 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:36:09.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.537+0000 7f9c97e00640 1 -- 172.21.15.90:0/1097772230 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c98039040 con 0x561ea1f1cd10 2024-04-02T14:36:09.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.537+0000 7f9c97e00640 1 -- 172.21.15.90:0/1097772230 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9c98004030 con 0x561ea1f1cd10 2024-04-02T14:36:09.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.537+0000 7f9c97e00640 1 -- 172.21.15.90:0/1097772230 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c9802ca10 con 0x561ea1f1cd10 2024-04-02T14:36:09.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.538+0000 7f9cb0995740 1 -- 172.21.15.90:0/1097772230 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561ea1f1cd10 msgr2=0x561ea1f1d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:09.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.538+0000 7f9cb0995740 1 --2- 172.21.15.90:0/1097772230 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561ea1f1cd10 0x561ea1f1d190 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f9c98009830 tx=0x7f9c980056d0 comp rx=0 tx=0).stop 2024-04-02T14:36:09.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.538+0000 7f9ca7400640 1 --2- 172.21.15.90:0/2485463657 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f9c8c0a3f50 0x7f9c8c0a6410 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f9ca00cddb0 tx=0x7f9ca000ceb0 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:36:09.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.539+0000 7f9cb0995740 1 -- 172.21.15.90:0/1097772230 shutdown_connections 2024-04-02T14:36:09.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.539+0000 7f9cb0995740 1 --2- 172.21.15.90:0/1097772230 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561ea1f2e890 0x561ea1f30c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:09.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.539+0000 7f9cb0995740 1 --2- 172.21.15.90:0/1097772230 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561ea1f1cd10 0x561ea1f1d190 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:09.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.539+0000 7f9cb0995740 1 --2- 172.21.15.90:0/1097772230 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561ea1f1d9f0 0x561ea1f1c740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:09.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.539+0000 7f9cb0995740 1 -- 172.21.15.90:0/1097772230 >> 172.21.15.90:0/1097772230 conn(0x561ea1ee3a50 msgr2=0x561ea1f17760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:36:09.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.539+0000 7f9cb0995740 1 -- 172.21.15.90:0/1097772230 shutdown_connections 2024-04-02T14:36:09.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.539+0000 7f9cb0995740 1 -- 172.21.15.90:0/1097772230 wait complete. 2024-04-02T14:36:09.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.539+0000 7f9cb0995740 1 -- start start 2024-04-02T14:36:09.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.540+0000 7f9cb0995740 1 Processor -- start 2024-04-02T14:36:09.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.540+0000 7f9cb0995740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561ea1f1cd10 0x561ea1f23070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:09.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.541+0000 7f9c9fe00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561ea1f1cd10 0x561ea1f23070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:09.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.541+0000 7f9c9fe00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561ea1f1cd10 0x561ea1f23070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:33110/0 (socket says 172.21.15.90:33110) 2024-04-02T14:36:09.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.540+0000 7f9cb0995740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561ea1f1d9f0 0x561ea1f235b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:09.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.541+0000 7f9c9fe00640 1 -- 172.21.15.90:0/3796500930 learned_addr learned my addr 172.21.15.90:0/3796500930 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:36:09.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.541+0000 7f9cb0995740 1 --2- 172.21.15.90:0/3796500930 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561ea1f2e890 0x561ea1f23af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:09.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.541+0000 7f9cb0995740 1 -- 172.21.15.90:0/3796500930 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x561ea1e7d6a0 con 0x561ea1f2e890 2024-04-02T14:36:09.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.541+0000 7f9cb0995740 1 -- 172.21.15.90:0/3796500930 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x561ea1e29980 con 0x561ea1f1cd10 2024-04-02T14:36:09.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.541+0000 7f9cb0995740 1 -- 172.21.15.90:0/3796500930 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x561ea1e3ccb0 con 0x561ea1f1d9f0 2024-04-02T14:36:09.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.541+0000 7f9cb0995740 10 client.0 _ll_register_callbacks cb 0x561ea1e3c6a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:36:09.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.541+0000 7f9ca6a00640 1 --2- 172.21.15.90:0/3796500930 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561ea1f1d9f0 0x561ea1f235b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:09.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.541+0000 7f9c9fe00640 1 -- 172.21.15.90:0/3796500930 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561ea1f1d9f0 msgr2=0x561ea1f235b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:36:09.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.541+0000 7f9c9fe00640 1 --2- 172.21.15.90:0/3796500930 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561ea1f1d9f0 0x561ea1f235b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:09.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.541+0000 7f9c9fe00640 1 -- 172.21.15.90:0/3796500930 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561ea1f2e890 msgr2=0x561ea1f23af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:36:09.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.541+0000 7f9c9fe00640 1 --2- 172.21.15.90:0/3796500930 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561ea1f2e890 0x561ea1f23af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:09.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.542+0000 7f9c9fe00640 1 -- 172.21.15.90:0/3796500930 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561ea1c5f7d0 con 0x561ea1f1cd10 2024-04-02T14:36:09.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.542+0000 7f9ca6a00640 1 --2- 172.21.15.90:0/3796500930 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561ea1f1d9f0 0x561ea1f235b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:36:09.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.542+0000 7f9c9fe00640 1 --2- 172.21.15.90:0/3796500930 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561ea1f1cd10 0x561ea1f23070 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f9c90012820 tx=0x7f9c90012ce0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=78b463663303f330 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:36:09.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.542+0000 7f9c94c00640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:36:09.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.542+0000 7f9c94c00640 1 -- 172.21.15.90:0/3796500930 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c90011db0 con 0x561ea1f1cd10 2024-04-02T14:36:09.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.542+0000 7f9c94c00640 1 -- 172.21.15.90:0/3796500930 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f9c9000f480 con 0x561ea1f1cd10 2024-04-02T14:36:09.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.542+0000 7f9cb0995740 1 -- 172.21.15.90:0/3796500930 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9c98004030 con 0x561ea1f1cd10 2024-04-02T14:36:09.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.543+0000 7f9c94c00640 1 -- 172.21.15.90:0/3796500930 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f9c9000f6c0 con 0x561ea1f1cd10 2024-04-02T14:36:09.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.543+0000 7f9cb0995740 10 client.25781 fetch_fsmap learned FSMap version 27 2024-04-02T14:36:09.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.543+0000 7f9cb0995740 1 -- 172.21.15.90:0/3796500930 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x561ea1f3d750 con 0x561ea1f1cd10 2024-04-02T14:36:09.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.543+0000 7f9c94c00640 1 -- 172.21.15.90:0/3796500930 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9c98004030 con 0x561ea1f1cd10 2024-04-02T14:36:09.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.543+0000 7f9c94c00640 1 -- 172.21.15.90:0/3796500930 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f9c90028b10 con 0x561ea1f1cd10 2024-04-02T14:36:09.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.544+0000 7f9cb0995740 10 client.25781 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:36:09.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.544+0000 7f9cb0995740 10 client.25781 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-04-02T14:36:09.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.544+0000 7f9cb0995740 20 client.25781 populate_metadata read hostname 'smithi090' 2024-04-02T14:36:09.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.544+0000 7f9cb0995740 1 --2- 172.21.15.90:0/3796500930 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x561ea1f3ecb0 0x561ea1f410c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:09.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.544+0000 7f9cb0995740 4 client.25781 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:36:09.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.544+0000 7f9cb0995740 1 -- 172.21.15.90:0/3796500930 --> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561ea1e41040 con 0x561ea1f3ecb0 2024-04-02T14:36:09.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.544+0000 7f9ca6a00640 1 --2- 172.21.15.90:0/3796500930 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x561ea1f3ecb0 0x561ea1f410c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:09.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.545+0000 7f9ca6a00640 1 --2- 172.21.15.90:0/3796500930 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x561ea1f3ecb0 0x561ea1f410c0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=6caf9edac90f7263 server_cookie=3c05596f9650eb3e in_seq=0 out_seq=0 2024-04-02T14:36:09.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.545+0000 7f9c94c00640 10 client.25781 ms_handle_connect on v2:172.21.15.92:6836/1270635939 2024-04-02T14:36:09.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.546+0000 7f9c94c00640 1 -- 172.21.15.90:0/3796500930 <== mds.3 v2:172.21.15.92:6836/1270635939 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x561ea1e41040 con 0x561ea1f3ecb0 2024-04-02T14:36:09.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.546+0000 7f9c94c00640 10 client.25781 handle_command_reply: tid=0 2024-04-02T14:36:09.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.546+0000 7f9cb0995740 1 client.25781 shutdown 2024-04-02T14:36:09.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.547+0000 7f9cb0995740 1 -- 172.21.15.90:0/3796500930 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561ea1f1cd10 msgr2=0x561ea1f23070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:09.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.547+0000 7f9cb0995740 1 --2- 172.21.15.90:0/3796500930 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561ea1f1cd10 0x561ea1f23070 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f9c90012820 tx=0x7f9c90012ce0 comp rx=0 tx=0).stop 2024-04-02T14:36:09.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.547+0000 7f9cb0995740 1 -- 172.21.15.90:0/3796500930 shutdown_connections 2024-04-02T14:36:09.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.547+0000 7f9cb0995740 1 --2- 172.21.15.90:0/3796500930 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x561ea1f3ecb0 0x561ea1f410c0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:09.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.547+0000 7f9cb0995740 1 --2- 172.21.15.90:0/3796500930 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561ea1f2e890 0x561ea1f23af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:09.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.547+0000 7f9cb0995740 1 --2- 172.21.15.90:0/3796500930 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561ea1f1d9f0 0x561ea1f235b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:09.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.547+0000 7f9cb0995740 1 --2- 172.21.15.90:0/3796500930 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561ea1f1cd10 0x561ea1f23070 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:09.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.547+0000 7f9cb0995740 1 -- 172.21.15.90:0/3796500930 >> 172.21.15.90:0/3796500930 conn(0x561ea1ee3a50 msgr2=0x561ea1f194b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:36:09.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.547+0000 7f9c94c00640 0 client.25781 ms_handle_reset on v2:172.21.15.92:6836/1270635939 2024-04-02T14:36:09.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.547+0000 7f9cb0995740 1 -- 172.21.15.90:0/3796500930 shutdown_connections 2024-04-02T14:36:09.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.548+0000 7f9cb0995740 1 -- 172.21.15.90:0/3796500930 wait complete. 2024-04-02T14:36:09.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.548+0000 7f9cb0995740 20 client.25781 trim_cache size 0 max 16384 2024-04-02T14:36:09.583 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.583+0000 7f9cb0995740 1 Processor -- start 2024-04-02T14:36:09.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.583+0000 7f9cb0995740 1 -- start start 2024-04-02T14:36:09.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.584+0000 7f9cb0995740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561ea1f1cd10 0x561ea1f3a300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:09.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.584+0000 7f9cb0995740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561ea1f1d9f0 0x561ea1f192c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:09.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.584+0000 7f9cb0995740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561ea1f2e890 0x561ea1f469d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:09.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.584+0000 7f9cb0995740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f9c8400e7a0 con 0x561ea1f2e890 2024-04-02T14:36:09.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.584+0000 7f9cb0995740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f9c840102e0 con 0x561ea1f1d9f0 2024-04-02T14:36:09.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.584+0000 7f9cb0995740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f9c8400ee70 con 0x561ea1f1cd10 2024-04-02T14:36:09.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.584+0000 7f9c9fe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561ea1f1cd10 0x561ea1f3a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:09.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.584+0000 7f9c9fe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561ea1f1cd10 0x561ea1f3a300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:53798/0 (socket says 172.21.15.90:53798) 2024-04-02T14:36:09.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.584+0000 7f9c9fe00640 1 -- 172.21.15.90:0/2490173342 learned_addr learned my addr 172.21.15.90:0/2490173342 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:36:09.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.585+0000 7f9ca6a00640 1 --2- 172.21.15.90:0/2490173342 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561ea1f1d9f0 0x561ea1f192c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:09.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.585+0000 7f9c9fe00640 1 --2- 172.21.15.90:0/2490173342 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561ea1f2e890 0x561ea1f469d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:09.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.585+0000 7f9c9fe00640 1 -- 172.21.15.90:0/2490173342 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561ea1f1d9f0 msgr2=0x561ea1f192c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:36:09.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.585+0000 7f9c9fe00640 1 --2- 172.21.15.90:0/2490173342 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561ea1f1d9f0 0x561ea1f192c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:09.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.585+0000 7f9c9fe00640 1 -- 172.21.15.90:0/2490173342 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561ea1f2e890 msgr2=0x561ea1f469d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:36:09.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.585+0000 7f9c9fe00640 1 --2- 172.21.15.90:0/2490173342 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561ea1f2e890 0x561ea1f469d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:09.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.585+0000 7f9c9fe00640 1 -- 172.21.15.90:0/2490173342 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561ea1f1bd40 con 0x561ea1f1cd10 2024-04-02T14:36:09.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.586+0000 7f9c9fe00640 1 --2- 172.21.15.90:0/2490173342 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561ea1f1cd10 0x561ea1f3a300 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f9c90028fd0 tx=0x7f9c90012500 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5f57821c56e19a22 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:36:09.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.586+0000 7f9ca6a00640 1 --2- 172.21.15.90:0/2490173342 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561ea1f1d9f0 0x561ea1f192c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:36:09.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.586+0000 7f9c97400640 1 -- 172.21.15.90:0/2490173342 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c90011db0 con 0x561ea1f1cd10 2024-04-02T14:36:09.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.586+0000 7f9c97400640 1 -- 172.21.15.90:0/2490173342 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9c90019030 con 0x561ea1f1cd10 2024-04-02T14:36:09.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.586+0000 7f9c97400640 1 -- 172.21.15.90:0/2490173342 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c9001ebd0 con 0x561ea1f1cd10 2024-04-02T14:36:09.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.587+0000 7f9cb0995740 1 -- 172.21.15.90:0/2490173342 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561ea1f1cd10 msgr2=0x561ea1f3a300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:09.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.587+0000 7f9cb0995740 1 --2- 172.21.15.90:0/2490173342 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561ea1f1cd10 0x561ea1f3a300 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f9c90028fd0 tx=0x7f9c90012500 comp rx=0 tx=0).stop 2024-04-02T14:36:09.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.588+0000 7f9cb0995740 1 -- 172.21.15.90:0/2490173342 shutdown_connections 2024-04-02T14:36:09.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.588+0000 7f9cb0995740 1 --2- 172.21.15.90:0/2490173342 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561ea1f2e890 0x561ea1f469d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:09.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.588+0000 7f9cb0995740 1 --2- 172.21.15.90:0/2490173342 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561ea1f1d9f0 0x561ea1f192c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:09.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.588+0000 7f9cb0995740 1 --2- 172.21.15.90:0/2490173342 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561ea1f1cd10 0x561ea1f3a300 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:09.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.588+0000 7f9cb0995740 1 -- 172.21.15.90:0/2490173342 >> 172.21.15.90:0/2490173342 conn(0x561ea1ee3a50 msgr2=0x561ea1f18bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:36:09.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.588+0000 7f9cb0995740 1 -- 172.21.15.90:0/2490173342 shutdown_connections 2024-04-02T14:36:09.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.589+0000 7f9cb0995740 1 -- 172.21.15.90:0/2490173342 wait complete. 2024-04-02T14:36:09.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.589+0000 7f9cb0995740 1 -- start start 2024-04-02T14:36:09.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.589+0000 7f9cb0995740 1 Processor -- start 2024-04-02T14:36:09.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.591+0000 7f9cb0995740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561ea1f1cd10 0x561ea1f339a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:09.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.591+0000 7f9cb0995740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561ea1f1d9f0 0x561ea1f33ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:09.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.591+0000 7f9cb0995740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561ea1f2e890 0x561ea1fa3240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:09.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.591+0000 7f9cb0995740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f9c8400fc10 con 0x561ea1f2e890 2024-04-02T14:36:09.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.591+0000 7f9cb0995740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f9c8400f540 con 0x561ea1f1cd10 2024-04-02T14:36:09.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.591+0000 7f9cb0995740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f9c8400c780 con 0x561ea1f1d9f0 2024-04-02T14:36:09.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.591+0000 7f9cb0995740 10 client.0 _ll_register_callbacks cb 0x7f9c90012450 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:36:09.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.591+0000 7f9c9fe00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561ea1f1cd10 0x561ea1f339a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:09.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.591+0000 7f9c9fe00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561ea1f1cd10 0x561ea1f339a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:33120/0 (socket says 172.21.15.90:33120) 2024-04-02T14:36:09.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.591+0000 7f9c9fe00640 1 -- 172.21.15.90:0/1867796964 learned_addr learned my addr 172.21.15.90:0/1867796964 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:36:09.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.591+0000 7f9c9fe00640 1 --2- 172.21.15.90:0/1867796964 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561ea1f2e890 0x561ea1fa3240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:09.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.592+0000 7f9ca6a00640 1 --2- 172.21.15.90:0/1867796964 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561ea1f1d9f0 0x561ea1f33ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:09.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.592+0000 7f9c9fe00640 1 -- 172.21.15.90:0/1867796964 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561ea1f1d9f0 msgr2=0x561ea1f33ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:36:09.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.592+0000 7f9c9fe00640 1 --2- 172.21.15.90:0/1867796964 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561ea1f1d9f0 0x561ea1f33ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:09.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.592+0000 7f9c9fe00640 1 -- 172.21.15.90:0/1867796964 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561ea1f2e890 msgr2=0x561ea1fa3240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:36:09.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.592+0000 7f9c9fe00640 1 --2- 172.21.15.90:0/1867796964 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561ea1f2e890 0x561ea1fa3240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:09.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.592+0000 7f9c9fe00640 1 -- 172.21.15.90:0/1867796964 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561ea1fa3780 con 0x561ea1f1cd10 2024-04-02T14:36:09.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.593+0000 7f9c9fe00640 1 --2- 172.21.15.90:0/1867796964 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561ea1f1cd10 0x561ea1f339a0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f9c90009940 tx=0x7f9c9001e7b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4782163b3cfbee00 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:36:09.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.593+0000 7f9c94c00640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:36:09.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.593+0000 7f9c94c00640 1 -- 172.21.15.90:0/1867796964 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c90005110 con 0x561ea1f1cd10 2024-04-02T14:36:09.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.593+0000 7f9c94c00640 1 -- 172.21.15.90:0/1867796964 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f9c900052b0 con 0x561ea1f1cd10 2024-04-02T14:36:09.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.593+0000 7f9cb0995740 1 -- 172.21.15.90:0/1867796964 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9c90019030 con 0x561ea1f1cd10 2024-04-02T14:36:09.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.593+0000 7f9c94c00640 1 -- 172.21.15.90:0/1867796964 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f9c9001e950 con 0x561ea1f1cd10 2024-04-02T14:36:09.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.593+0000 7f9cb0995740 10 client.25784 fetch_fsmap learned FSMap version 27 2024-04-02T14:36:09.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.593+0000 7f9cb0995740 1 -- 172.21.15.90:0/1867796964 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x561ea1fa49c0 con 0x561ea1f1cd10 2024-04-02T14:36:09.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.594+0000 7f9c94c00640 1 -- 172.21.15.90:0/1867796964 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9c90019030 con 0x561ea1f1cd10 2024-04-02T14:36:09.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.594+0000 7f9c94c00640 1 -- 172.21.15.90:0/1867796964 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f9c90035030 con 0x561ea1f1cd10 2024-04-02T14:36:09.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.595+0000 7f9cb0995740 10 client.25784 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:36:09.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.595+0000 7f9cb0995740 10 client.25784 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-04-02T14:36:09.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.595+0000 7f9cb0995740 20 client.25784 populate_metadata read hostname 'smithi090' 2024-04-02T14:36:09.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.595+0000 7f9cb0995740 1 --2- 172.21.15.90:0/1867796964 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x561ea1fa5050 0x561ea1fa7440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:09.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.595+0000 7f9cb0995740 4 client.25784 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-02T14:36:09.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.595+0000 7f9cb0995740 1 -- 172.21.15.90:0/1867796964 --> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] -- command(tid 0: {"prefix": "damage ls"}) -- 0x561ea1e8ec30 con 0x561ea1fa5050 2024-04-02T14:36:09.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.595+0000 7f9ca6a00640 1 --2- 172.21.15.90:0/1867796964 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x561ea1fa5050 0x561ea1fa7440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:09.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.596+0000 7f9ca6a00640 1 --2- 172.21.15.90:0/1867796964 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x561ea1fa5050 0x561ea1fa7440 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=73768b65899afa85 server_cookie=75c1e08d3ff51270 in_seq=0 out_seq=0 2024-04-02T14:36:09.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.596+0000 7f9c94c00640 10 client.25784 ms_handle_connect on v2:172.21.15.92:6836/1270635939 2024-04-02T14:36:09.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.596+0000 7f9c94c00640 1 -- 172.21.15.90:0/1867796964 <== mds.3 v2:172.21.15.92:6836/1270635939 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x561ea1e8ec30 con 0x561ea1fa5050 2024-04-02T14:36:09.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.596+0000 7f9c94c00640 10 client.25784 handle_command_reply: tid=0 2024-04-02T14:36:09.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.597+0000 7f9cb0995740 1 client.25784 shutdown 2024-04-02T14:36:09.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.597+0000 7f9cb0995740 1 -- 172.21.15.90:0/1867796964 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561ea1f1cd10 msgr2=0x561ea1f339a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:09.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.597+0000 7f9cb0995740 1 --2- 172.21.15.90:0/1867796964 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561ea1f1cd10 0x561ea1f339a0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f9c90009940 tx=0x7f9c9001e7b0 comp rx=0 tx=0).stop 2024-04-02T14:36:09.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.597+0000 7f9cb0995740 1 -- 172.21.15.90:0/1867796964 shutdown_connections 2024-04-02T14:36:09.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.597+0000 7f9cb0995740 1 --2- 172.21.15.90:0/1867796964 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x561ea1fa5050 0x561ea1fa7440 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:09.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.597+0000 7f9cb0995740 1 --2- 172.21.15.90:0/1867796964 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561ea1f2e890 0x561ea1fa3240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:09.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.597+0000 7f9cb0995740 1 --2- 172.21.15.90:0/1867796964 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561ea1f1d9f0 0x561ea1f33ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:09.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.597+0000 7f9cb0995740 1 --2- 172.21.15.90:0/1867796964 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561ea1f1cd10 0x561ea1f339a0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:09.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.597+0000 7f9cb0995740 1 -- 172.21.15.90:0/1867796964 >> 172.21.15.90:0/1867796964 conn(0x561ea1ee3a50 msgr2=0x561ea1f168a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:36:09.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.597+0000 7f9c94c00640 0 client.25784 ms_handle_reset on v2:172.21.15.92:6836/1270635939 2024-04-02T14:36:09.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.598+0000 7f9cb0995740 1 -- 172.21.15.90:0/1867796964 shutdown_connections 2024-04-02T14:36:09.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.598+0000 7f9cb0995740 1 -- 172.21.15.90:0/1867796964 wait complete. 2024-04-02T14:36:09.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.598+0000 7f9cb0995740 20 client.25784 trim_cache size 0 max 16384 2024-04-02T14:36:09.600 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-02T14:36:09.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.601+0000 7f9cadc00640 1 -- 172.21.15.90:0/2485463657 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f9c8c0a3f50 msgr2=0x7f9c8c0a6410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:09.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.601+0000 7f9cadc00640 1 --2- 172.21.15.90:0/2485463657 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f9c8c0a3f50 0x7f9c8c0a6410 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f9ca00cddb0 tx=0x7f9ca000ceb0 comp rx=0 tx=0).stop 2024-04-02T14:36:09.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.601+0000 7f9cadc00640 1 -- 172.21.15.90:0/2485463657 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9ca810e4a0 msgr2=0x7f9ca806c910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:09.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.601+0000 7f9cadc00640 1 --2- 172.21.15.90:0/2485463657 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9ca810e4a0 0x7f9ca806c910 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f9ca0005fd0 tx=0x7f9ca000c800 comp rx=0 tx=0).stop 2024-04-02T14:36:09.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.601+0000 7f9cadc00640 1 -- 172.21.15.90:0/2485463657 shutdown_connections 2024-04-02T14:36:09.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.601+0000 7f9cadc00640 1 --2- 172.21.15.90:0/2485463657 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f9c8c0a3f50 0x7f9c8c0a6410 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:09.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.601+0000 7f9cadc00640 1 --2- 172.21.15.90:0/2485463657 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9ca810e4a0 0x7f9ca806c910 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:09.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.601+0000 7f9cadc00640 1 --2- 172.21.15.90:0/2485463657 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9ca8069600 0x7f9ca806c3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:09.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.601+0000 7f9cadc00640 1 --2- 172.21.15.90:0/2485463657 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9ca8068ce0 0x7f9ca8073350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:09.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.601+0000 7f9cadc00640 1 -- 172.21.15.90:0/2485463657 >> 172.21.15.90:0/2485463657 conn(0x7f9ca8077b00 msgr2=0x7f9ca810fb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:36:09.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.601+0000 7f9cadc00640 1 -- 172.21.15.90:0/2485463657 shutdown_connections 2024-04-02T14:36:09.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.601+0000 7f9cadc00640 1 -- 172.21.15.90:0/2485463657 wait complete. 2024-04-02T14:36:09.637 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-02T14:36:09.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.907+0000 7f88c9400640 1 -- 172.21.15.90:0/3209635505 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f88c4108fe0 msgr2=0x7f88c4110bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:09.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.907+0000 7f88c9400640 1 --2- 172.21.15.90:0/3209635505 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f88c4108fe0 0x7f88c4110bb0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f88b40098b0 tx=0x7f88b402bbc0 comp rx=0 tx=0).stop 2024-04-02T14:36:09.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.908+0000 7f88c9400640 1 -- 172.21.15.90:0/3209635505 shutdown_connections 2024-04-02T14:36:09.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.908+0000 7f88c9400640 1 --2- 172.21.15.90:0/3209635505 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f88c4108fe0 0x7f88c4110bb0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:09.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.908+0000 7f88c9400640 1 --2- 172.21.15.90:0/3209635505 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f88c41048c0 0x7f88c4108900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:09.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.908+0000 7f88c9400640 1 --2- 172.21.15.90:0/3209635505 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f88c4103f10 0x7f88c41042f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:09.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.908+0000 7f88c9400640 1 -- 172.21.15.90:0/3209635505 >> 172.21.15.90:0/3209635505 conn(0x7f88c40ff770 msgr2=0x7f88c4101b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:36:09.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.908+0000 7f88c9400640 1 -- 172.21.15.90:0/3209635505 shutdown_connections 2024-04-02T14:36:09.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.908+0000 7f88c9400640 1 -- 172.21.15.90:0/3209635505 wait complete. 2024-04-02T14:36:09.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.911+0000 7f88c9400640 1 Processor -- start 2024-04-02T14:36:09.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.911+0000 7f88c9400640 1 -- start start 2024-04-02T14:36:09.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.912+0000 7f88c9400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f88c4103f10 0x7f88c41a45a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:09.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.912+0000 7f88c9400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f88c4108fe0 0x7f88c41a4ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:09.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.912+0000 7f88c9400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f88c410ff90 0x7f88c4110440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:09.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.912+0000 7f88c9400640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f88c4077c60 con 0x7f88c410ff90 2024-04-02T14:36:09.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.912+0000 7f88c9400640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f88c4077ae0 con 0x7f88c4108fe0 2024-04-02T14:36:09.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.912+0000 7f88c9400640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f88c4077de0 con 0x7f88c4103f10 2024-04-02T14:36:09.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.912+0000 7f88c1e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f88c4108fe0 0x7f88c41a4ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:09.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.912+0000 7f88c1e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f88c4108fe0 0x7f88c41a4ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:33126/0 (socket says 172.21.15.90:33126) 2024-04-02T14:36:09.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.912+0000 7f88c1e00640 1 -- 172.21.15.90:0/1277550755 learned_addr learned my addr 172.21.15.90:0/1277550755 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:36:09.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.912+0000 7f88c3200640 1 --2- 172.21.15.90:0/1277550755 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f88c410ff90 0x7f88c4110440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:09.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.913+0000 7f88c2800640 1 --2- 172.21.15.90:0/1277550755 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f88c4103f10 0x7f88c41a45a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:09.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.913+0000 7f88c3200640 1 -- 172.21.15.90:0/1277550755 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f88c4103f10 msgr2=0x7f88c41a45a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:09.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.913+0000 7f88c3200640 1 --2- 172.21.15.90:0/1277550755 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f88c4103f10 0x7f88c41a45a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:09.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.913+0000 7f88c3200640 1 -- 172.21.15.90:0/1277550755 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f88c4108fe0 msgr2=0x7f88c41a4ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:09.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.913+0000 7f88c3200640 1 --2- 172.21.15.90:0/1277550755 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f88c4108fe0 0x7f88c41a4ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:09.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.913+0000 7f88c3200640 1 -- 172.21.15.90:0/1277550755 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88c4110b90 con 0x7f88c410ff90 2024-04-02T14:36:09.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.914+0000 7f88c3200640 1 --2- 172.21.15.90:0/1277550755 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f88c410ff90 0x7f88c4110440 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f88b40097d0 tx=0x7f88b40045e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:36:09.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.914+0000 7f88bb400640 1 -- 172.21.15.90:0/1277550755 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88b4039050 con 0x7f88c410ff90 2024-04-02T14:36:09.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.915+0000 7f88bb400640 1 -- 172.21.15.90:0/1277550755 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f88b4045070 con 0x7f88c410ff90 2024-04-02T14:36:09.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.915+0000 7f88bb400640 1 -- 172.21.15.90:0/1277550755 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88b4039050 con 0x7f88c410ff90 2024-04-02T14:36:09.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.915+0000 7f88c9400640 1 -- 172.21.15.90:0/1277550755 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88c4110e20 con 0x7f88c410ff90 2024-04-02T14:36:09.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.915+0000 7f88c9400640 1 -- 172.21.15.90:0/1277550755 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88c406b7c0 con 0x7f88c410ff90 2024-04-02T14:36:09.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.920+0000 7f88cc1e1740 1 Processor -- start 2024-04-02T14:36:09.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.921+0000 7f88bb400640 1 -- 172.21.15.90:0/1277550755 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 25) ==== 138930+0+0 (secure 0 0 0) 0x7f88b4032b10 con 0x7f88c410ff90 2024-04-02T14:36:09.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.921+0000 7f88cc1e1740 1 -- start start 2024-04-02T14:36:09.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.922+0000 7f88cc1e1740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5618778f99f0 0x5618778f8740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:09.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.922+0000 7f88cc1e1740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5618778f8d10 0x5618778f9190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:09.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.922+0000 7f88cc1e1740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561877902080 0x561877904470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:09.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.922+0000 7f88cc1e1740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x561877839400 con 0x5618778f99f0 2024-04-02T14:36:09.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.922+0000 7f88cc1e1740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x56187781df10 con 0x5618778f8d10 2024-04-02T14:36:09.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.922+0000 7f88cc1e1740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x56187785d6f0 con 0x561877902080 2024-04-02T14:36:09.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.923+0000 7f88bb400640 1 --2- 172.21.15.90:0/1277550755 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f88a80a4190 0x7f88a80a6650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:09.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.923+0000 7f88bb400640 1 -- 172.21.15.90:0/1277550755 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f88b40f24c0 con 0x7f88c410ff90 2024-04-02T14:36:09.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.923+0000 7f88c1e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561877902080 0x561877904470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:09.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.923+0000 7f88c1e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561877902080 0x561877904470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:53816/0 (socket says 172.21.15.90:53816) 2024-04-02T14:36:09.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.924+0000 7f88c1e00640 1 -- 172.21.15.90:0/868238499 learned_addr learned my addr 172.21.15.90:0/868238499 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:36:09.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.924+0000 7f88c2800640 1 --2- 172.21.15.90:0/868238499 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5618778f8d10 0x5618778f9190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:09.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.924+0000 7f88c1e00640 1 --2- 172.21.15.90:0/868238499 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5618778f99f0 0x5618778f8740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:09.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.924+0000 7f88c3200640 1 --2- 172.21.15.90:0/1277550755 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f88a80a4190 0x7f88a80a6650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:09.929 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.924+0000 7f88c1e00640 1 -- 172.21.15.90:0/868238499 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5618778f8d10 msgr2=0x5618778f9190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:36:09.929 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.925+0000 7f88c1e00640 1 --2- 172.21.15.90:0/868238499 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5618778f8d10 0x5618778f9190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:09.929 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.925+0000 7f88c1e00640 1 -- 172.21.15.90:0/868238499 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5618778f99f0 msgr2=0x5618778f8740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:36:09.929 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.925+0000 7f88c1e00640 1 --2- 172.21.15.90:0/868238499 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5618778f99f0 0x5618778f8740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:09.929 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.925+0000 7f88c1e00640 1 -- 172.21.15.90:0/868238499 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5618775713f0 con 0x561877902080 2024-04-02T14:36:09.929 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.925+0000 7f88c2800640 1 --2- 172.21.15.90:0/868238499 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5618778f8d10 0x5618778f9190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:36:09.929 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.925+0000 7f88c3200640 1 --2- 172.21.15.90:0/1277550755 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f88a80a4190 0x7f88a80a6650 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f88b40efb50 tx=0x7f88b4034810 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:36:09.929 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.926+0000 7f88c1e00640 1 --2- 172.21.15.90:0/868238499 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561877902080 0x561877904470 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f88b00115d0 tx=0x7f88b000a740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f9880dc8c0666e3 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:36:09.929 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.926+0000 7f88af400640 1 -- 172.21.15.90:0/868238499 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88b0004220 con 0x561877902080 2024-04-02T14:36:09.929 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.926+0000 7f88af400640 1 -- 172.21.15.90:0/868238499 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f88b00043c0 con 0x561877902080 2024-04-02T14:36:09.929 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.926+0000 7f88af400640 1 -- 172.21.15.90:0/868238499 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88b000f710 con 0x561877902080 2024-04-02T14:36:09.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.928+0000 7f88cc1e1740 1 -- 172.21.15.90:0/868238499 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561877902080 msgr2=0x561877904470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:09.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.928+0000 7f88cc1e1740 1 --2- 172.21.15.90:0/868238499 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561877902080 0x561877904470 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f88b00115d0 tx=0x7f88b000a740 comp rx=0 tx=0).stop 2024-04-02T14:36:09.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.928+0000 7f88cc1e1740 1 -- 172.21.15.90:0/868238499 shutdown_connections 2024-04-02T14:36:09.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.928+0000 7f88cc1e1740 1 --2- 172.21.15.90:0/868238499 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561877902080 0x561877904470 secure :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f88b00115d0 tx=0x7f88b000a740 comp rx=0 tx=0).stop 2024-04-02T14:36:09.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.928+0000 7f88cc1e1740 1 --2- 172.21.15.90:0/868238499 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5618778f8d10 0x5618778f9190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:09.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.928+0000 7f88cc1e1740 1 --2- 172.21.15.90:0/868238499 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5618778f99f0 0x5618778f8740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:09.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.928+0000 7f88cc1e1740 1 -- 172.21.15.90:0/868238499 >> 172.21.15.90:0/868238499 conn(0x5618778bfa50 msgr2=0x5618778f3760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:36:09.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.928+0000 7f88cc1e1740 1 -- 172.21.15.90:0/868238499 shutdown_connections 2024-04-02T14:36:09.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.928+0000 7f88cc1e1740 1 -- 172.21.15.90:0/868238499 wait complete. 2024-04-02T14:36:09.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.929+0000 7f88cc1e1740 1 -- start start 2024-04-02T14:36:09.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.929+0000 7f88cc1e1740 1 Processor -- start 2024-04-02T14:36:09.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.930+0000 7f88cc1e1740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5618778f8d10 0x5618778ff1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:09.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.930+0000 7f88cc1e1740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5618778f99f0 0x5618778ff730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:09.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.931+0000 7f88cc1e1740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5618778ffcc0 0x56187790e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:09.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.931+0000 7f88cc1e1740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5618778596a0 con 0x5618778f8d10 2024-04-02T14:36:09.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.931+0000 7f88cc1e1740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x561877805980 con 0x5618778f99f0 2024-04-02T14:36:09.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.931+0000 7f88cc1e1740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x561877818cb0 con 0x5618778ffcc0 2024-04-02T14:36:09.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.931+0000 7f88cc1e1740 10 client.0 _ll_register_callbacks cb 0x5618778186a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:36:09.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.931+0000 7f88c2800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5618778f99f0 0x5618778ff730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:09.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.931+0000 7f88c2800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5618778f99f0 0x5618778ff730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:33146/0 (socket says 172.21.15.90:33146) 2024-04-02T14:36:09.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.931+0000 7f88c2800640 1 -- 172.21.15.90:0/4248966340 learned_addr learned my addr 172.21.15.90:0/4248966340 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:36:09.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.931+0000 7f88c1e00640 1 --2- 172.21.15.90:0/4248966340 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5618778ffcc0 0x56187790e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:09.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.932+0000 7f88c2800640 1 -- 172.21.15.90:0/4248966340 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5618778ffcc0 msgr2=0x56187790e0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:36:09.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.932+0000 7f88c2800640 1 --2- 172.21.15.90:0/4248966340 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5618778ffcc0 0x56187790e0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:09.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.932+0000 7f88c2800640 1 -- 172.21.15.90:0/4248966340 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5618778f8d10 msgr2=0x5618778ff1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:36:09.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.932+0000 7f88c2800640 1 --2- 172.21.15.90:0/4248966340 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5618778f8d10 0x5618778ff1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:09.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.932+0000 7f88c2800640 1 -- 172.21.15.90:0/4248966340 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56187763b7d0 con 0x5618778f99f0 2024-04-02T14:36:09.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.932+0000 7f88c1e00640 1 --2- 172.21.15.90:0/4248966340 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5618778ffcc0 0x56187790e0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:36:09.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.932+0000 7f88c2800640 1 --2- 172.21.15.90:0/4248966340 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5618778f99f0 0x5618778ff730 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f88bc011b20 tx=0x7f88bc00f3c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=caeb529a934e50cb server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:36:09.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.932+0000 7f88a3e00640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:36:09.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.933+0000 7f88a3e00640 1 -- 172.21.15.90:0/4248966340 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88bc004b20 con 0x5618778f99f0 2024-04-02T14:36:09.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.933+0000 7f88a3e00640 1 -- 172.21.15.90:0/4248966340 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f88bc004cc0 con 0x5618778f99f0 2024-04-02T14:36:09.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.933+0000 7f88cc1e1740 1 -- 172.21.15.90:0/4248966340 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f88b00043c0 con 0x5618778f99f0 2024-04-02T14:36:09.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.933+0000 7f88a3e00640 1 -- 172.21.15.90:0/4248966340 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f88bc005020 con 0x5618778f99f0 2024-04-02T14:36:09.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.933+0000 7f88cc1e1740 10 client.25793 fetch_fsmap learned FSMap version 27 2024-04-02T14:36:09.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.933+0000 7f88cc1e1740 1 -- 172.21.15.90:0/4248966340 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x561877911590 con 0x5618778f99f0 2024-04-02T14:36:09.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.934+0000 7f88a3e00640 1 -- 172.21.15.90:0/4248966340 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f88b00043c0 con 0x5618778f99f0 2024-04-02T14:36:09.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.935+0000 7f88a3e00640 1 -- 172.21.15.90:0/4248966340 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f88bc0051c0 con 0x5618778f99f0 2024-04-02T14:36:09.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.935+0000 7f88cc1e1740 10 client.25793 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:36:09.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.935+0000 7f88cc1e1740 10 client.25793 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-04-02T14:36:09.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.935+0000 7f88cc1e1740 20 client.25793 populate_metadata read hostname 'smithi090' 2024-04-02T14:36:09.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.935+0000 7f88cc1e1740 1 --2- 172.21.15.90:0/4248966340 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x5618779129f0 0x561877914e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:09.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.935+0000 7f88cc1e1740 4 client.25793 mds_command: new command op to 24547 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:36:09.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.935+0000 7f88cc1e1740 1 -- 172.21.15.90:0/4248966340 --> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56187781d040 con 0x5618779129f0 2024-04-02T14:36:09.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.936+0000 7f88c1e00640 1 --2- 172.21.15.90:0/4248966340 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x5618779129f0 0x561877914e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:09.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.937+0000 7f88c1e00640 1 --2- 172.21.15.90:0/4248966340 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x5618779129f0 0x561877914e50 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=389d7910f2796f49 server_cookie=176f35062e480112 in_seq=0 out_seq=0 2024-04-02T14:36:09.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.937+0000 7f88a3e00640 10 client.25793 ms_handle_connect on v2:172.21.15.177:6838/664895663 2024-04-02T14:36:09.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.938+0000 7f88a3e00640 1 -- 172.21.15.90:0/4248966340 <== mds.2 v2:172.21.15.177:6838/664895663 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56187781d040 con 0x5618779129f0 2024-04-02T14:36:09.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.938+0000 7f88a3e00640 10 client.25793 handle_command_reply: tid=0 2024-04-02T14:36:09.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.938+0000 7f88cc1e1740 1 client.25793 shutdown 2024-04-02T14:36:09.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.938+0000 7f88cc1e1740 1 -- 172.21.15.90:0/4248966340 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5618778f99f0 msgr2=0x5618778ff730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:09.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.938+0000 7f88cc1e1740 1 --2- 172.21.15.90:0/4248966340 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5618778f99f0 0x5618778ff730 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f88bc011b20 tx=0x7f88bc00f3c0 comp rx=0 tx=0).stop 2024-04-02T14:36:09.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.939+0000 7f88cc1e1740 1 -- 172.21.15.90:0/4248966340 shutdown_connections 2024-04-02T14:36:09.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.939+0000 7f88cc1e1740 1 --2- 172.21.15.90:0/4248966340 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x5618779129f0 0x561877914e50 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:09.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.939+0000 7f88cc1e1740 1 --2- 172.21.15.90:0/4248966340 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5618778ffcc0 0x56187790e0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:09.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.939+0000 7f88cc1e1740 1 --2- 172.21.15.90:0/4248966340 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5618778f99f0 0x5618778ff730 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:09.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.939+0000 7f88cc1e1740 1 --2- 172.21.15.90:0/4248966340 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5618778f8d10 0x5618778ff1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:09.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.939+0000 7f88cc1e1740 1 -- 172.21.15.90:0/4248966340 >> 172.21.15.90:0/4248966340 conn(0x5618778bfa50 msgr2=0x5618778f54b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:36:09.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.939+0000 7f88a3e00640 0 client.25793 ms_handle_reset on v2:172.21.15.177:6838/664895663 2024-04-02T14:36:09.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.939+0000 7f88cc1e1740 1 -- 172.21.15.90:0/4248966340 shutdown_connections 2024-04-02T14:36:09.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.939+0000 7f88cc1e1740 1 -- 172.21.15.90:0/4248966340 wait complete. 2024-04-02T14:36:09.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.939+0000 7f88cc1e1740 20 client.25793 trim_cache size 0 max 16384 2024-04-02T14:36:09.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.954+0000 7f88cc1e1740 1 Processor -- start 2024-04-02T14:36:09.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.954+0000 7f88cc1e1740 1 -- start start 2024-04-02T14:36:09.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.954+0000 7f88cc1e1740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5618778f8d10 0x5618778f2720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:09.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.954+0000 7f88cc1e1740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5618778f99f0 0x56187791a760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:09.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.954+0000 7f88cc1e1740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5618778ffcc0 0x56187791aca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:09.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.954+0000 7f88cc1e1740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f88c41bc5f0 con 0x5618778ffcc0 2024-04-02T14:36:09.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.955+0000 7f88cc1e1740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f88c41be130 con 0x5618778f8d10 2024-04-02T14:36:09.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.955+0000 7f88cc1e1740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f88c41bccc0 con 0x5618778f99f0 2024-04-02T14:36:09.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.955+0000 7f88c1e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5618778f8d10 0x5618778f2720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:09.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.955+0000 7f88c1e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5618778f8d10 0x5618778f2720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:33158/0 (socket says 172.21.15.90:33158) 2024-04-02T14:36:09.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.955+0000 7f88c1e00640 1 -- 172.21.15.90:0/663851219 learned_addr learned my addr 172.21.15.90:0/663851219 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:36:09.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.955+0000 7f88c2800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5618778f99f0 0x56187791a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:09.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.955+0000 7f88c1e00640 1 --2- 172.21.15.90:0/663851219 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5618778ffcc0 0x56187791aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:09.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.955+0000 7f88c1e00640 1 -- 172.21.15.90:0/663851219 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5618778f99f0 msgr2=0x56187791a760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:36:09.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.955+0000 7f88c1e00640 1 --2- 172.21.15.90:0/663851219 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5618778f99f0 0x56187791a760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:09.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.955+0000 7f88c1e00640 1 -- 172.21.15.90:0/663851219 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5618778ffcc0 msgr2=0x56187791aca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:36:09.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.955+0000 7f88c1e00640 1 --2- 172.21.15.90:0/663851219 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5618778ffcc0 0x56187791aca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:09.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.955+0000 7f88c1e00640 1 -- 172.21.15.90:0/663851219 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5618778eedb0 con 0x5618778f8d10 2024-04-02T14:36:09.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.955+0000 7f88c2800640 1 --2- 172.21.15.90:0/663851219 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5618778f99f0 0x56187791a760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:36:09.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.955+0000 7f88c1e00640 1 --2- 172.21.15.90:0/663851219 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5618778f8d10 0x5618778f2720 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f88b0011fd0 tx=0x7f88b0013040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aef164ad184e61a5 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:36:09.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.956+0000 7f88aea00640 1 -- 172.21.15.90:0/663851219 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88b000f710 con 0x5618778f8d10 2024-04-02T14:36:09.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.956+0000 7f88aea00640 1 -- 172.21.15.90:0/663851219 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f88b0020040 con 0x5618778f8d10 2024-04-02T14:36:09.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.956+0000 7f88aea00640 1 -- 172.21.15.90:0/663851219 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88b0003c20 con 0x5618778f8d10 2024-04-02T14:36:09.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.956+0000 7f88cc1e1740 1 -- 172.21.15.90:0/663851219 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5618778f8d10 msgr2=0x5618778f2720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:09.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.956+0000 7f88cc1e1740 1 --2- 172.21.15.90:0/663851219 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5618778f8d10 0x5618778f2720 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f88b0011fd0 tx=0x7f88b0013040 comp rx=0 tx=0).stop 2024-04-02T14:36:09.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.957+0000 7f88cc1e1740 1 -- 172.21.15.90:0/663851219 shutdown_connections 2024-04-02T14:36:09.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.957+0000 7f88cc1e1740 1 --2- 172.21.15.90:0/663851219 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5618778ffcc0 0x56187791aca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:09.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.957+0000 7f88cc1e1740 1 --2- 172.21.15.90:0/663851219 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5618778f99f0 0x56187791a760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:09.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.957+0000 7f88cc1e1740 1 --2- 172.21.15.90:0/663851219 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5618778f8d10 0x5618778f2720 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:09.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.957+0000 7f88cc1e1740 1 -- 172.21.15.90:0/663851219 >> 172.21.15.90:0/663851219 conn(0x5618778bfa50 msgr2=0x5618778f2010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:36:09.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.957+0000 7f88cc1e1740 1 -- 172.21.15.90:0/663851219 shutdown_connections 2024-04-02T14:36:09.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.957+0000 7f88cc1e1740 1 -- 172.21.15.90:0/663851219 wait complete. 2024-04-02T14:36:09.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.957+0000 7f88cc1e1740 1 -- start start 2024-04-02T14:36:09.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.957+0000 7f88cc1e1740 1 Processor -- start 2024-04-02T14:36:09.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.958+0000 7f88cc1e1740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5618778f8d10 0x56187790a300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:09.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.958+0000 7f88cc1e1740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5618778f99f0 0x561877902b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:09.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.958+0000 7f88cc1e1740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5618778ffcc0 0x561877903040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:09.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.958+0000 7f88cc1e1740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f88c41bda60 con 0x5618778ffcc0 2024-04-02T14:36:09.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.958+0000 7f88cc1e1740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f88c41bd390 con 0x5618778f99f0 2024-04-02T14:36:09.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.958+0000 7f88cc1e1740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f88c4071f10 con 0x5618778f8d10 2024-04-02T14:36:09.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.958+0000 7f88cc1e1740 10 client.0 _ll_register_callbacks cb 0x7f88bc0116e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:36:09.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.958+0000 7f88c2800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5618778f99f0 0x561877902b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:09.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.958+0000 7f88c1e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5618778f8d10 0x56187790a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:09.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.958+0000 7f88c2800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5618778f99f0 0x561877902b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:33168/0 (socket says 172.21.15.90:33168) 2024-04-02T14:36:09.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.958+0000 7f88c2800640 1 -- 172.21.15.90:0/3577723533 learned_addr learned my addr 172.21.15.90:0/3577723533 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:36:09.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.958+0000 7f88c1e00640 1 --2- 172.21.15.90:0/3577723533 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5618778ffcc0 0x561877903040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:09.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.958+0000 7f88c2800640 1 -- 172.21.15.90:0/3577723533 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5618778f8d10 msgr2=0x56187790a300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:36:09.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.958+0000 7f88c2800640 1 --2- 172.21.15.90:0/3577723533 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5618778f8d10 0x56187790a300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:09.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.958+0000 7f88c2800640 1 -- 172.21.15.90:0/3577723533 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5618778ffcc0 msgr2=0x561877903040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:36:09.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.958+0000 7f88c2800640 1 --2- 172.21.15.90:0/3577723533 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5618778ffcc0 0x561877903040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:09.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.958+0000 7f88c2800640 1 -- 172.21.15.90:0/3577723533 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5618779037b0 con 0x5618778f99f0 2024-04-02T14:36:09.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.959+0000 7f88c2800640 1 --2- 172.21.15.90:0/3577723533 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5618778f99f0 0x561877902b00 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f88bc005590 tx=0x7f88bc005ac0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=95281b2923803964 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:36:09.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.959+0000 7f88a3e00640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:36:09.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.959+0000 7f88a3e00640 1 -- 172.21.15.90:0/3577723533 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88bc004c20 con 0x5618778f99f0 2024-04-02T14:36:09.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.959+0000 7f88a3e00640 1 -- 172.21.15.90:0/3577723533 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f88bc004dc0 con 0x5618778f99f0 2024-04-02T14:36:09.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.959+0000 7f88cc1e1740 1 -- 172.21.15.90:0/3577723533 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f88b0020040 con 0x5618778f99f0 2024-04-02T14:36:09.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.959+0000 7f88a3e00640 1 -- 172.21.15.90:0/3577723533 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f88bc033050 con 0x5618778f99f0 2024-04-02T14:36:09.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.959+0000 7f88cc1e1740 10 client.25799 fetch_fsmap learned FSMap version 27 2024-04-02T14:36:09.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.959+0000 7f88cc1e1740 1 -- 172.21.15.90:0/3577723533 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x561877978fd0 con 0x5618778f99f0 2024-04-02T14:36:09.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.959+0000 7f88a3e00640 1 -- 172.21.15.90:0/3577723533 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f88b0020040 con 0x5618778f99f0 2024-04-02T14:36:09.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.959+0000 7f88a3e00640 1 -- 172.21.15.90:0/3577723533 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f88bc02c060 con 0x5618778f99f0 2024-04-02T14:36:09.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.959+0000 7f88cc1e1740 10 client.25799 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:36:09.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.959+0000 7f88cc1e1740 10 client.25799 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-04-02T14:36:09.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.960+0000 7f88cc1e1740 20 client.25799 populate_metadata read hostname 'smithi090' 2024-04-02T14:36:09.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.960+0000 7f88cc1e1740 1 --2- 172.21.15.90:0/3577723533 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x561877979690 0x56187797bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:09.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.960+0000 7f88cc1e1740 4 client.25799 mds_command: new command op to 24547 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-02T14:36:09.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.960+0000 7f88cc1e1740 1 -- 172.21.15.90:0/3577723533 --> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56187786ac30 con 0x561877979690 2024-04-02T14:36:09.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.960+0000 7f88c1e00640 1 --2- 172.21.15.90:0/3577723533 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x561877979690 0x56187797bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:09.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.960+0000 7f88c1e00640 1 --2- 172.21.15.90:0/3577723533 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x561877979690 0x56187797bab0 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=8ca4d99382ffbaf2 server_cookie=2e9aaf56207063f6 in_seq=0 out_seq=0 2024-04-02T14:36:09.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.960+0000 7f88a3e00640 10 client.25799 ms_handle_connect on v2:172.21.15.177:6838/664895663 2024-04-02T14:36:09.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.960+0000 7f88a3e00640 1 -- 172.21.15.90:0/3577723533 <== mds.2 v2:172.21.15.177:6838/664895663 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56187786ac30 con 0x561877979690 2024-04-02T14:36:09.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.960+0000 7f88a3e00640 10 client.25799 handle_command_reply: tid=0 2024-04-02T14:36:09.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.960+0000 7f88cc1e1740 1 client.25799 shutdown 2024-04-02T14:36:09.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.961+0000 7f88cc1e1740 1 -- 172.21.15.90:0/3577723533 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5618778f99f0 msgr2=0x561877902b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:09.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.961+0000 7f88cc1e1740 1 --2- 172.21.15.90:0/3577723533 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5618778f99f0 0x561877902b00 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f88bc005590 tx=0x7f88bc005ac0 comp rx=0 tx=0).stop 2024-04-02T14:36:09.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.961+0000 7f88cc1e1740 1 -- 172.21.15.90:0/3577723533 shutdown_connections 2024-04-02T14:36:09.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.961+0000 7f88cc1e1740 1 --2- 172.21.15.90:0/3577723533 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x561877979690 0x56187797bab0 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:09.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.961+0000 7f88cc1e1740 1 --2- 172.21.15.90:0/3577723533 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5618778ffcc0 0x561877903040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:09.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.961+0000 7f88cc1e1740 1 --2- 172.21.15.90:0/3577723533 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5618778f99f0 0x561877902b00 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:09.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.961+0000 7f88cc1e1740 1 --2- 172.21.15.90:0/3577723533 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5618778f8d10 0x56187790a300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:09.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.961+0000 7f88cc1e1740 1 -- 172.21.15.90:0/3577723533 >> 172.21.15.90:0/3577723533 conn(0x5618778bfa50 msgr2=0x5618778f28a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:36:09.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.961+0000 7f88a3e00640 0 client.25799 ms_handle_reset on v2:172.21.15.177:6838/664895663 2024-04-02T14:36:09.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.961+0000 7f88cc1e1740 1 -- 172.21.15.90:0/3577723533 shutdown_connections 2024-04-02T14:36:09.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.961+0000 7f88cc1e1740 1 -- 172.21.15.90:0/3577723533 wait complete. 2024-04-02T14:36:09.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.961+0000 7f88cc1e1740 20 client.25799 trim_cache size 0 max 16384 2024-04-02T14:36:09.968 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-02T14:36:09.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.962+0000 7f88c9400640 1 -- 172.21.15.90:0/1277550755 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f88a80a4190 msgr2=0x7f88a80a6650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:09.969 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.962+0000 7f88c9400640 1 --2- 172.21.15.90:0/1277550755 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f88a80a4190 0x7f88a80a6650 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f88b40efb50 tx=0x7f88b4034810 comp rx=0 tx=0).stop 2024-04-02T14:36:09.969 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.962+0000 7f88c9400640 1 -- 172.21.15.90:0/1277550755 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f88c410ff90 msgr2=0x7f88c4110440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:09.969 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.962+0000 7f88c9400640 1 --2- 172.21.15.90:0/1277550755 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f88c410ff90 0x7f88c4110440 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f88b40097d0 tx=0x7f88b40045e0 comp rx=0 tx=0).stop 2024-04-02T14:36:09.969 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.962+0000 7f88c9400640 1 -- 172.21.15.90:0/1277550755 shutdown_connections 2024-04-02T14:36:09.969 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.962+0000 7f88c9400640 1 --2- 172.21.15.90:0/1277550755 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f88a80a4190 0x7f88a80a6650 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:09.969 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.962+0000 7f88c9400640 1 --2- 172.21.15.90:0/1277550755 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f88c410ff90 0x7f88c4110440 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:09.969 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.962+0000 7f88c9400640 1 --2- 172.21.15.90:0/1277550755 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f88c4108fe0 0x7f88c41a4ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:09.969 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.962+0000 7f88c9400640 1 --2- 172.21.15.90:0/1277550755 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f88c4103f10 0x7f88c41a45a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:09.969 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.962+0000 7f88c9400640 1 -- 172.21.15.90:0/1277550755 >> 172.21.15.90:0/1277550755 conn(0x7f88c40ff770 msgr2=0x7f88c4101b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:36:09.969 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.962+0000 7f88c9400640 1 -- 172.21.15.90:0/1277550755 shutdown_connections 2024-04-02T14:36:09.969 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:09.962+0000 7f88c9400640 1 -- 172.21.15.90:0/1277550755 wait complete. 2024-04-02T14:36:09.978 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-02T14:36:10.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.247+0000 7f4ecde00640 1 -- 172.21.15.90:0/3112588119 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4ec8106a50 msgr2=0x7f4ec8068d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:10.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.247+0000 7f4ecde00640 1 --2- 172.21.15.90:0/3112588119 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4ec8106a50 0x7f4ec8068d00 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f4eb8009900 tx=0x7f4eb802bc50 comp rx=0 tx=0).stop 2024-04-02T14:36:10.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.248+0000 7f4ecde00640 1 -- 172.21.15.90:0/3112588119 shutdown_connections 2024-04-02T14:36:10.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.248+0000 7f4ecde00640 1 --2- 172.21.15.90:0/3112588119 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4ec81045c0 0x7f4ec8104a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:10.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.248+0000 7f4ecde00640 1 --2- 172.21.15.90:0/3112588119 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4ec8069240 0x7f4ec80696a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:10.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.248+0000 7f4ecde00640 1 --2- 172.21.15.90:0/3112588119 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4ec8106a50 0x7f4ec8068d00 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:10.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.248+0000 7f4ecde00640 1 -- 172.21.15.90:0/3112588119 >> 172.21.15.90:0/3112588119 conn(0x7f4ec8077b00 msgr2=0x7f4ec8077f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:36:10.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.248+0000 7f4ecde00640 1 -- 172.21.15.90:0/3112588119 shutdown_connections 2024-04-02T14:36:10.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.248+0000 7f4ecde00640 1 -- 172.21.15.90:0/3112588119 wait complete. 2024-04-02T14:36:10.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.249+0000 7f4ecde00640 1 Processor -- start 2024-04-02T14:36:10.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.250+0000 7f4ecde00640 1 -- start start 2024-04-02T14:36:10.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.250+0000 7f4ecde00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4ec8069240 0x7f4ec81a4670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:10.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.250+0000 7f4ecde00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4ec81045c0 0x7f4ec81a4bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:10.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.250+0000 7f4ecde00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4ec8106a50 0x7f4ec819e7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:10.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.250+0000 7f4ecde00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f4ec8118520 con 0x7f4ec81045c0 2024-04-02T14:36:10.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.250+0000 7f4ecde00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f4ec81183a0 con 0x7f4ec8106a50 2024-04-02T14:36:10.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.250+0000 7f4ecde00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f4ec81186a0 con 0x7f4ec8069240 2024-04-02T14:36:10.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.250+0000 7f4ec6a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4ec8069240 0x7f4ec81a4670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:10.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.250+0000 7f4ec6a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4ec8069240 0x7f4ec81a4670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:53868/0 (socket says 172.21.15.90:53868) 2024-04-02T14:36:10.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.250+0000 7f4ec6a00640 1 -- 172.21.15.90:0/441885305 learned_addr learned my addr 172.21.15.90:0/441885305 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:36:10.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.250+0000 7f4ec7400640 1 --2- 172.21.15.90:0/441885305 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4ec8106a50 0x7f4ec819e7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:10.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.251+0000 7f4ec6a00640 1 -- 172.21.15.90:0/441885305 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4ec8106a50 msgr2=0x7f4ec819e7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:10.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.251+0000 7f4ec6a00640 1 --2- 172.21.15.90:0/441885305 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4ec8106a50 0x7f4ec819e7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:10.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.251+0000 7f4ec6a00640 1 -- 172.21.15.90:0/441885305 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4ec81045c0 msgr2=0x7f4ec81a4bb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T14:36:10.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.251+0000 7f4ec6000640 1 --2- 172.21.15.90:0/441885305 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4ec81045c0 0x7f4ec81a4bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:10.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.252+0000 7f4ec6a00640 1 --2- 172.21.15.90:0/441885305 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4ec81045c0 0x7f4ec81a4bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:10.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.253+0000 7f4ec6a00640 1 -- 172.21.15.90:0/441885305 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ec819efb0 con 0x7f4ec8069240 2024-04-02T14:36:10.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.254+0000 7f4ec6000640 1 --2- 172.21.15.90:0/441885305 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4ec81045c0 0x7f4ec81a4bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:36:10.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.254+0000 7f4ec6a00640 1 --2- 172.21.15.90:0/441885305 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4ec8069240 0x7f4ec81a4670 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f4eb8005ab0 tx=0x7f4eb8004360 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:36:10.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.255+0000 7f4ecca00640 1 -- 172.21.15.90:0/441885305 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4eb8041070 con 0x7f4ec8069240 2024-04-02T14:36:10.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.255+0000 7f4ecca00640 1 -- 172.21.15.90:0/441885305 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4eb802ccc0 con 0x7f4ec8069240 2024-04-02T14:36:10.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.255+0000 7f4ecca00640 1 -- 172.21.15.90:0/441885305 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4eb8003de0 con 0x7f4ec8069240 2024-04-02T14:36:10.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.255+0000 7f4ecde00640 1 -- 172.21.15.90:0/441885305 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ec819f240 con 0x7f4ec8069240 2024-04-02T14:36:10.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.255+0000 7f4ecde00640 1 -- 172.21.15.90:0/441885305 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ec81ab4c0 con 0x7f4ec8069240 2024-04-02T14:36:10.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.263+0000 7f4ed0a23740 1 Processor -- start 2024-04-02T14:36:10.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.264+0000 7f4ed0a23740 1 -- start start 2024-04-02T14:36:10.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.264+0000 7f4ed0a23740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dc779249f0 0x55dc77923740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:10.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.265+0000 7f4ed0a23740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55dc77923d10 0x55dc77924190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:10.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.265+0000 7f4ec7400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dc779249f0 0x55dc77923740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:10.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.266+0000 7f4ec7400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dc779249f0 0x55dc77923740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:53884/0 (socket says 172.21.15.90:53884) 2024-04-02T14:36:10.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.266+0000 7f4ed0a23740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dc7792d080 0x55dc7792f470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:10.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.266+0000 7f4ec7400640 1 -- 172.21.15.90:0/4217648399 learned_addr learned my addr 172.21.15.90:0/4217648399 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:36:10.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.266+0000 7f4ecca00640 1 -- 172.21.15.90:0/441885305 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 25) ==== 138930+0+0 (secure 0 0 0) 0x7f4eb8044070 con 0x7f4ec8069240 2024-04-02T14:36:10.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.266+0000 7f4ed0a23740 1 -- 172.21.15.90:0/4217648399 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55dc77864400 con 0x55dc77923d10 2024-04-02T14:36:10.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.266+0000 7f4ed0a23740 1 -- 172.21.15.90:0/4217648399 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55dc77848f10 con 0x55dc7792d080 2024-04-02T14:36:10.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.266+0000 7f4ed0a23740 1 -- 172.21.15.90:0/4217648399 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55dc778886f0 con 0x55dc779249f0 2024-04-02T14:36:10.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.266+0000 7f4ec7400640 1 -- 172.21.15.90:0/4217648399 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dc7792d080 msgr2=0x55dc7792f470 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-02T14:36:10.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.266+0000 7f4ec7400640 1 --2- 172.21.15.90:0/4217648399 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dc7792d080 0x55dc7792f470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:10.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.266+0000 7f4ec7400640 1 -- 172.21.15.90:0/4217648399 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55dc77923d10 msgr2=0x55dc77924190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:36:10.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.266+0000 7f4ec7400640 1 --2- 172.21.15.90:0/4217648399 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55dc77923d10 0x55dc77924190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:10.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.266+0000 7f4ec7400640 1 -- 172.21.15.90:0/4217648399 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dc7759c3f0 con 0x55dc779249f0 2024-04-02T14:36:10.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.267+0000 7f4ec7400640 1 --2- 172.21.15.90:0/4217648399 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dc779249f0 0x55dc77923740 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f4ec0009fd0 tx=0x7f4ec00118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b50e90cc4a514786 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:36:10.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.267+0000 7f4eb3e00640 1 -- 172.21.15.90:0/4217648399 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ec0004d20 con 0x55dc779249f0 2024-04-02T14:36:10.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.267+0000 7f4eb3e00640 1 -- 172.21.15.90:0/4217648399 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4ec0004050 con 0x55dc779249f0 2024-04-02T14:36:10.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.268+0000 7f4ed0a23740 1 -- 172.21.15.90:0/4217648399 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dc779249f0 msgr2=0x55dc77923740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:10.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.268+0000 7f4ed0a23740 1 --2- 172.21.15.90:0/4217648399 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dc779249f0 0x55dc77923740 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f4ec0009fd0 tx=0x7f4ec00118d0 comp rx=0 tx=0).stop 2024-04-02T14:36:10.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.268+0000 7f4ed0a23740 1 -- 172.21.15.90:0/4217648399 shutdown_connections 2024-04-02T14:36:10.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.268+0000 7f4ed0a23740 1 --2- 172.21.15.90:0/4217648399 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dc7792d080 0x55dc7792f470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:10.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.268+0000 7f4ed0a23740 1 --2- 172.21.15.90:0/4217648399 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55dc77923d10 0x55dc77924190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:10.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.268+0000 7f4ed0a23740 1 --2- 172.21.15.90:0/4217648399 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dc779249f0 0x55dc77923740 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:10.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.268+0000 7f4ed0a23740 1 -- 172.21.15.90:0/4217648399 >> 172.21.15.90:0/4217648399 conn(0x55dc778eaa50 msgr2=0x55dc7791e760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:36:10.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.268+0000 7f4ecca00640 1 --2- 172.21.15.90:0/441885305 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f4eac0a3f50 0x7f4eac0a6410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:10.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.268+0000 7f4ecca00640 1 -- 172.21.15.90:0/441885305 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f4eb80f24a0 con 0x7f4ec8069240 2024-04-02T14:36:10.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.269+0000 7f4ed0a23740 1 -- 172.21.15.90:0/4217648399 shutdown_connections 2024-04-02T14:36:10.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.270+0000 7f4ec6000640 1 --2- 172.21.15.90:0/441885305 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f4eac0a3f50 0x7f4eac0a6410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:10.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.271+0000 7f4ed0a23740 1 -- 172.21.15.90:0/4217648399 wait complete. 2024-04-02T14:36:10.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.272+0000 7f4ed0a23740 1 -- start start 2024-04-02T14:36:10.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.272+0000 7f4ed0a23740 1 Processor -- start 2024-04-02T14:36:10.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.273+0000 7f4ed0a23740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dc77923d10 0x55dc77929ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:10.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.273+0000 7f4ed0a23740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55dc779249f0 0x55dc7792a430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:10.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.273+0000 7f4ed0a23740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dc7792d080 0x55dc7792a970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:10.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.273+0000 7f4ed0a23740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55dc778846a0 con 0x55dc779249f0 2024-04-02T14:36:10.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.273+0000 7f4ed0a23740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55dc77830980 con 0x55dc77923d10 2024-04-02T14:36:10.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.273+0000 7f4ed0a23740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55dc77843cb0 con 0x55dc7792d080 2024-04-02T14:36:10.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.273+0000 7f4ed0a23740 10 client.0 _ll_register_callbacks cb 0x55dc778436a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:36:10.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.274+0000 7f4ec7400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dc7792d080 0x55dc7792a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:10.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.274+0000 7f4ec6a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dc77923d10 0x55dc77929ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:10.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.274+0000 7f4ec6a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dc77923d10 0x55dc77929ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:33194/0 (socket says 172.21.15.90:33194) 2024-04-02T14:36:10.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.274+0000 7f4ec6a00640 1 -- 172.21.15.90:0/2405401830 learned_addr learned my addr 172.21.15.90:0/2405401830 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:36:10.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.275+0000 7f4ec6000640 1 --2- 172.21.15.90:0/2405401830 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55dc779249f0 0x55dc7792a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:10.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.275+0000 7f4ec6a00640 1 -- 172.21.15.90:0/2405401830 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dc7792d080 msgr2=0x55dc7792a970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:36:10.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.275+0000 7f4ec6a00640 1 --2- 172.21.15.90:0/2405401830 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dc7792d080 0x55dc7792a970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:10.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.275+0000 7f4ec6a00640 1 -- 172.21.15.90:0/2405401830 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55dc779249f0 msgr2=0x55dc7792a430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:36:10.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.275+0000 7f4ec6a00640 1 --2- 172.21.15.90:0/2405401830 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55dc779249f0 0x55dc7792a430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:10.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.275+0000 7f4ec6a00640 1 -- 172.21.15.90:0/2405401830 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dc776667d0 con 0x55dc77923d10 2024-04-02T14:36:10.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.276+0000 7f4ec7400640 1 --2- 172.21.15.90:0/2405401830 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dc7792d080 0x55dc7792a970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:36:10.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.276+0000 7f4ec6000640 1 --2- 172.21.15.90:0/2405401830 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55dc779249f0 0x55dc7792a430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:36:10.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.276+0000 7f4ec6000640 1 --2- 172.21.15.90:0/441885305 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f4eac0a3f50 0x7f4eac0a6410 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f4ec8078250 tx=0x7f4eb4008040 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:36:10.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.277+0000 7f4ec6a00640 1 --2- 172.21.15.90:0/2405401830 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dc77923d10 0x55dc77929ef0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f4eb8009430 tx=0x7f4eb80ba5c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7370c82415060510 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:36:10.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.277+0000 7f4eb0c00640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:36:10.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.277+0000 7f4eb0c00640 1 -- 172.21.15.90:0/2405401830 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4eb804b040 con 0x55dc77923d10 2024-04-02T14:36:10.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.278+0000 7f4eb0c00640 1 -- 172.21.15.90:0/2405401830 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f4eb80c6d00 con 0x55dc77923d10 2024-04-02T14:36:10.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.278+0000 7f4ed0a23740 1 -- 172.21.15.90:0/2405401830 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4ec0004050 con 0x55dc77923d10 2024-04-02T14:36:10.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.278+0000 7f4eb0c00640 1 -- 172.21.15.90:0/2405401830 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f4eb80bdc80 con 0x55dc77923d10 2024-04-02T14:36:10.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.279+0000 7f4ed0a23740 10 client.25808 fetch_fsmap learned FSMap version 27 2024-04-02T14:36:10.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.279+0000 7f4ed0a23740 1 -- 172.21.15.90:0/2405401830 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55dc7794cee0 con 0x55dc77923d10 2024-04-02T14:36:10.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.279+0000 7f4eb0c00640 1 -- 172.21.15.90:0/2405401830 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4ec0004050 con 0x55dc77923d10 2024-04-02T14:36:10.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.280+0000 7f4eb0c00640 1 -- 172.21.15.90:0/2405401830 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f4eb80bd770 con 0x55dc77923d10 2024-04-02T14:36:10.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.280+0000 7f4ed0a23740 10 client.25808 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:36:10.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.280+0000 7f4ed0a23740 10 client.25808 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2024-04-02T14:36:10.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.280+0000 7f4ed0a23740 20 client.25808 populate_metadata read hostname 'smithi090' 2024-04-02T14:36:10.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.280+0000 7f4ed0a23740 1 --2- 172.21.15.90:0/2405401830 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x55dc7794e520 0x55dc77950960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:10.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.280+0000 7f4ed0a23740 4 client.25808 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:36:10.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.280+0000 7f4ed0a23740 1 -- 172.21.15.90:0/2405401830 --> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55dc77848040 con 0x55dc7794e520 2024-04-02T14:36:10.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.281+0000 7f4ec6000640 1 --2- 172.21.15.90:0/2405401830 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x55dc7794e520 0x55dc77950960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:10.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.282+0000 7f4ec6000640 1 --2- 172.21.15.90:0/2405401830 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x55dc7794e520 0x55dc77950960 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=3db75e52ac467f99 server_cookie=d25987009dfaf83 in_seq=0 out_seq=0 2024-04-02T14:36:10.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.282+0000 7f4eb0c00640 10 client.25808 ms_handle_connect on v2:172.21.15.90:6838/1498915870 2024-04-02T14:36:10.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.285+0000 7f4eb0c00640 1 -- 172.21.15.90:0/2405401830 <== mds.1 v2:172.21.15.90:6838/1498915870 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55dc77848040 con 0x55dc7794e520 2024-04-02T14:36:10.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.285+0000 7f4eb0c00640 10 client.25808 handle_command_reply: tid=0 2024-04-02T14:36:10.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.285+0000 7f4ed0a23740 1 client.25808 shutdown 2024-04-02T14:36:10.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.286+0000 7f4ed0a23740 1 -- 172.21.15.90:0/2405401830 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dc77923d10 msgr2=0x55dc77929ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:10.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.286+0000 7f4ed0a23740 1 --2- 172.21.15.90:0/2405401830 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dc77923d10 0x55dc77929ef0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f4eb8009430 tx=0x7f4eb80ba5c0 comp rx=0 tx=0).stop 2024-04-02T14:36:10.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.286+0000 7f4ed0a23740 1 -- 172.21.15.90:0/2405401830 shutdown_connections 2024-04-02T14:36:10.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.286+0000 7f4ed0a23740 1 --2- 172.21.15.90:0/2405401830 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x55dc7794e520 0x55dc77950960 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:10.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.286+0000 7f4eb0c00640 0 client.25808 ms_handle_reset on v2:172.21.15.90:6838/1498915870 2024-04-02T14:36:10.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.286+0000 7f4ed0a23740 1 --2- 172.21.15.90:0/2405401830 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dc7792d080 0x55dc7792a970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:10.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.286+0000 7f4ed0a23740 1 --2- 172.21.15.90:0/2405401830 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55dc779249f0 0x55dc7792a430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:10.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.286+0000 7f4ed0a23740 1 --2- 172.21.15.90:0/2405401830 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dc77923d10 0x55dc77929ef0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:10.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.286+0000 7f4ed0a23740 1 -- 172.21.15.90:0/2405401830 >> 172.21.15.90:0/2405401830 conn(0x55dc778eaa50 msgr2=0x55dc779204b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:36:10.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.287+0000 7f4ed0a23740 1 -- 172.21.15.90:0/2405401830 shutdown_connections 2024-04-02T14:36:10.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.287+0000 7f4ed0a23740 1 -- 172.21.15.90:0/2405401830 wait complete. 2024-04-02T14:36:10.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.287+0000 7f4ed0a23740 20 client.25808 trim_cache size 0 max 16384 2024-04-02T14:36:10.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.332+0000 7f4ed0a23740 1 Processor -- start 2024-04-02T14:36:10.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.333+0000 7f4ed0a23740 1 -- start start 2024-04-02T14:36:10.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.333+0000 7f4ed0a23740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dc77923d10 0x55dc77938c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:10.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.334+0000 7f4ed0a23740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55dc779249f0 0x55dc77929050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:10.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.334+0000 7f4ed0a23740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dc7792d080 0x55dc7792a790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:10.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.334+0000 7f4ec7400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dc7792d080 0x55dc7792a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:10.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.334+0000 7f4ec7400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dc7792d080 0x55dc7792a790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:33206/0 (socket says 172.21.15.90:33206) 2024-04-02T14:36:10.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.334+0000 7f4ec7400640 1 -- 172.21.15.90:0/546452772 learned_addr learned my addr 172.21.15.90:0/546452772 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:36:10.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.334+0000 7f4ec6000640 1 --2- 172.21.15.90:0/546452772 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55dc779249f0 0x55dc77929050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:10.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.334+0000 7f4ed0a23740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f4ec81ad820 con 0x55dc779249f0 2024-04-02T14:36:10.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.334+0000 7f4ec6a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dc77923d10 0x55dc77938c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:10.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.334+0000 7f4ed0a23740 1 -- 172.21.15.90:0/546452772 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f4ec81ab690 con 0x55dc7792d080 2024-04-02T14:36:10.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.334+0000 7f4ed0a23740 1 -- 172.21.15.90:0/546452772 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f4ec81ad150 con 0x55dc77923d10 2024-04-02T14:36:10.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.335+0000 7f4ec6000640 1 -- 172.21.15.90:0/546452772 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dc77923d10 msgr2=0x55dc77938c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:36:10.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.335+0000 7f4ec6000640 1 --2- 172.21.15.90:0/546452772 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dc77923d10 0x55dc77938c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:10.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.335+0000 7f4ec6000640 1 -- 172.21.15.90:0/546452772 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dc7792d080 msgr2=0x55dc7792a790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:36:10.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.335+0000 7f4ec6000640 1 --2- 172.21.15.90:0/546452772 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dc7792d080 0x55dc7792a790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:10.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.335+0000 7f4ec6000640 1 -- 172.21.15.90:0/546452772 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dc77922d40 con 0x55dc779249f0 2024-04-02T14:36:10.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.336+0000 7f4ec6000640 1 --2- 172.21.15.90:0/546452772 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55dc779249f0 0x55dc77929050 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x55dc77951210 tx=0x7f4eb4014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d110a450f2e8595a server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:36:10.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.336+0000 7f4eb3400640 1 -- 172.21.15.90:0/546452772 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4eb4028030 con 0x55dc779249f0 2024-04-02T14:36:10.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.336+0000 7f4eb3400640 1 -- 172.21.15.90:0/546452772 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4eb4017040 con 0x55dc779249f0 2024-04-02T14:36:10.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.336+0000 7f4eb3400640 1 -- 172.21.15.90:0/546452772 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4eb401e6d0 con 0x55dc779249f0 2024-04-02T14:36:10.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.337+0000 7f4ed0a23740 1 -- 172.21.15.90:0/546452772 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55dc779249f0 msgr2=0x55dc77929050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:10.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.337+0000 7f4ed0a23740 1 --2- 172.21.15.90:0/546452772 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55dc779249f0 0x55dc77929050 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x55dc77951210 tx=0x7f4eb4014000 comp rx=0 tx=0).stop 2024-04-02T14:36:10.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.337+0000 7f4ed0a23740 1 -- 172.21.15.90:0/546452772 shutdown_connections 2024-04-02T14:36:10.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.338+0000 7f4ed0a23740 1 --2- 172.21.15.90:0/546452772 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dc7792d080 0x55dc7792a790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:10.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.338+0000 7f4ed0a23740 1 --2- 172.21.15.90:0/546452772 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55dc779249f0 0x55dc77929050 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:10.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.338+0000 7f4ed0a23740 1 --2- 172.21.15.90:0/546452772 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dc77923d10 0x55dc77938c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:10.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.338+0000 7f4ed0a23740 1 -- 172.21.15.90:0/546452772 >> 172.21.15.90:0/546452772 conn(0x55dc778eaa50 msgr2=0x55dc7791fbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:36:10.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.338+0000 7f4ed0a23740 1 -- 172.21.15.90:0/546452772 shutdown_connections 2024-04-02T14:36:10.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.339+0000 7f4ed0a23740 1 -- 172.21.15.90:0/546452772 wait complete. 2024-04-02T14:36:10.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.339+0000 7f4ed0a23740 1 -- start start 2024-04-02T14:36:10.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.339+0000 7f4ed0a23740 1 Processor -- start 2024-04-02T14:36:10.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.340+0000 7f4ed0a23740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55dc77923d10 0x55dc779354f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:10.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.341+0000 7f4ed0a23740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dc779249f0 0x55dc77930860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:10.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.341+0000 7f4ed0a23740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dc7792d080 0x55dc77930da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:10.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.341+0000 7f4ed0a23740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f4ec81ac3b0 con 0x55dc77923d10 2024-04-02T14:36:10.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.341+0000 7f4ed0a23740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f4ec81abce0 con 0x55dc7792d080 2024-04-02T14:36:10.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.341+0000 7f4ed0a23740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f4ec81aca80 con 0x55dc779249f0 2024-04-02T14:36:10.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.341+0000 7f4ed0a23740 10 client.0 _ll_register_callbacks cb 0x7f4eb8009560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:36:10.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.341+0000 7f4ec6000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dc779249f0 0x55dc77930860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:10.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.341+0000 7f4ec6000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dc779249f0 0x55dc77930860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:53906/0 (socket says 172.21.15.90:53906) 2024-04-02T14:36:10.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.341+0000 7f4ec6000640 1 -- 172.21.15.90:0/2256964808 learned_addr learned my addr 172.21.15.90:0/2256964808 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:36:10.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.342+0000 7f4ec6000640 1 -- 172.21.15.90:0/2256964808 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dc7792d080 msgr2=0x55dc77930da0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-02T14:36:10.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.342+0000 7f4ec6000640 1 --2- 172.21.15.90:0/2256964808 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dc7792d080 0x55dc77930da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:10.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.342+0000 7f4ec6a00640 1 --2- 172.21.15.90:0/2256964808 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55dc77923d10 0x55dc779354f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:10.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.342+0000 7f4ec6000640 1 -- 172.21.15.90:0/2256964808 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55dc77923d10 msgr2=0x55dc779354f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:36:10.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.342+0000 7f4ec6000640 1 --2- 172.21.15.90:0/2256964808 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55dc77923d10 0x55dc779354f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:10.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.342+0000 7f4ec6000640 1 -- 172.21.15.90:0/2256964808 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dc77931580 con 0x55dc779249f0 2024-04-02T14:36:10.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.343+0000 7f4ec6000640 1 --2- 172.21.15.90:0/2256964808 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dc779249f0 0x55dc77930860 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f4eb4008010 tx=0x7f4eb400a620 comp rx=0 tx=0).ready entity=mon.1 client_cookie=30bc7540867a9b6f server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:36:10.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.343+0000 7f4eb0c00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:36:10.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.344+0000 7f4eb0c00640 1 -- 172.21.15.90:0/2256964808 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4eb4005dd0 con 0x55dc779249f0 2024-04-02T14:36:10.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.344+0000 7f4eb0c00640 1 -- 172.21.15.90:0/2256964808 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f4eb4023150 con 0x55dc779249f0 2024-04-02T14:36:10.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.344+0000 7f4ed0a23740 1 -- 172.21.15.90:0/2256964808 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4eb4017040 con 0x55dc779249f0 2024-04-02T14:36:10.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.345+0000 7f4eb0c00640 1 -- 172.21.15.90:0/2256964808 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f4eb4023350 con 0x55dc779249f0 2024-04-02T14:36:10.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.345+0000 7f4ed0a23740 10 client.25756 fetch_fsmap learned FSMap version 27 2024-04-02T14:36:10.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.345+0000 7f4ed0a23740 1 -- 172.21.15.90:0/2256964808 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55dc779b4af0 con 0x55dc779249f0 2024-04-02T14:36:10.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.345+0000 7f4eb0c00640 1 -- 172.21.15.90:0/2256964808 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4eb4017040 con 0x55dc779249f0 2024-04-02T14:36:10.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.346+0000 7f4eb0c00640 1 -- 172.21.15.90:0/2256964808 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f4eb401c040 con 0x55dc779249f0 2024-04-02T14:36:10.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.346+0000 7f4ed0a23740 10 client.25756 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:36:10.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.346+0000 7f4ed0a23740 10 client.25756 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2024-04-02T14:36:10.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.346+0000 7f4ed0a23740 20 client.25756 populate_metadata read hostname 'smithi090' 2024-04-02T14:36:10.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.346+0000 7f4ed0a23740 1 --2- 172.21.15.90:0/2256964808 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x55dc7794e520 0x55dc779b7110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:10.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.346+0000 7f4ed0a23740 4 client.25756 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-02T14:36:10.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.346+0000 7f4ed0a23740 1 -- 172.21.15.90:0/2256964808 --> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55dc77895c30 con 0x55dc7794e520 2024-04-02T14:36:10.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.347+0000 7f4ec6a00640 1 --2- 172.21.15.90:0/2256964808 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x55dc7794e520 0x55dc779b7110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:10.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.348+0000 7f4ec6a00640 1 --2- 172.21.15.90:0/2256964808 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x55dc7794e520 0x55dc779b7110 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=3abb0174abce0a5d server_cookie=99a45d2d05335f26 in_seq=0 out_seq=0 2024-04-02T14:36:10.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.348+0000 7f4eb0c00640 10 client.25756 ms_handle_connect on v2:172.21.15.90:6838/1498915870 2024-04-02T14:36:10.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.348+0000 7f4eb0c00640 1 -- 172.21.15.90:0/2256964808 <== mds.1 v2:172.21.15.90:6838/1498915870 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55dc77895c30 con 0x55dc7794e520 2024-04-02T14:36:10.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.348+0000 7f4eb0c00640 10 client.25756 handle_command_reply: tid=0 2024-04-02T14:36:10.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.348+0000 7f4ed0a23740 1 client.25756 shutdown 2024-04-02T14:36:10.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.349+0000 7f4ed0a23740 1 -- 172.21.15.90:0/2256964808 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dc779249f0 msgr2=0x55dc77930860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:10.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.349+0000 7f4ed0a23740 1 --2- 172.21.15.90:0/2256964808 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dc779249f0 0x55dc77930860 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f4eb4008010 tx=0x7f4eb400a620 comp rx=0 tx=0).stop 2024-04-02T14:36:10.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.349+0000 7f4ed0a23740 1 -- 172.21.15.90:0/2256964808 shutdown_connections 2024-04-02T14:36:10.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.349+0000 7f4ed0a23740 1 --2- 172.21.15.90:0/2256964808 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x55dc7794e520 0x55dc779b7110 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:10.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.349+0000 7f4ed0a23740 1 --2- 172.21.15.90:0/2256964808 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dc7792d080 0x55dc77930da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:10.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.349+0000 7f4ed0a23740 1 --2- 172.21.15.90:0/2256964808 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dc779249f0 0x55dc77930860 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:10.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.349+0000 7f4ed0a23740 1 --2- 172.21.15.90:0/2256964808 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55dc77923d10 0x55dc779354f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:10.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.349+0000 7f4eb0c00640 0 client.25756 ms_handle_reset on v2:172.21.15.90:6838/1498915870 2024-04-02T14:36:10.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.349+0000 7f4ed0a23740 1 -- 172.21.15.90:0/2256964808 >> 172.21.15.90:0/2256964808 conn(0x55dc778eaa50 msgr2=0x55dc7792a7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:36:10.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.350+0000 7f4ed0a23740 1 -- 172.21.15.90:0/2256964808 shutdown_connections 2024-04-02T14:36:10.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.350+0000 7f4ed0a23740 1 -- 172.21.15.90:0/2256964808 wait complete. 2024-04-02T14:36:10.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.350+0000 7f4ed0a23740 20 client.25756 trim_cache size 0 max 16384 2024-04-02T14:36:10.359 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-02T14:36:10.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.351+0000 7f4ecde00640 1 -- 172.21.15.90:0/441885305 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f4eac0a3f50 msgr2=0x7f4eac0a6410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:10.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.351+0000 7f4ecde00640 1 --2- 172.21.15.90:0/441885305 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f4eac0a3f50 0x7f4eac0a6410 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f4ec8078250 tx=0x7f4eb4008040 comp rx=0 tx=0).stop 2024-04-02T14:36:10.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.352+0000 7f4ecde00640 1 -- 172.21.15.90:0/441885305 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4ec8069240 msgr2=0x7f4ec81a4670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:10.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.352+0000 7f4ecde00640 1 --2- 172.21.15.90:0/441885305 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4ec8069240 0x7f4ec81a4670 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f4eb8005ab0 tx=0x7f4eb8004360 comp rx=0 tx=0).stop 2024-04-02T14:36:10.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.352+0000 7f4ecde00640 1 -- 172.21.15.90:0/441885305 shutdown_connections 2024-04-02T14:36:10.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.352+0000 7f4ecde00640 1 --2- 172.21.15.90:0/441885305 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f4eac0a3f50 0x7f4eac0a6410 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:10.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.352+0000 7f4ecde00640 1 --2- 172.21.15.90:0/441885305 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4ec8106a50 0x7f4ec819e7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:10.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.352+0000 7f4ecde00640 1 --2- 172.21.15.90:0/441885305 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4ec81045c0 0x7f4ec81a4bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:10.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.352+0000 7f4ecde00640 1 --2- 172.21.15.90:0/441885305 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4ec8069240 0x7f4ec81a4670 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:10.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.352+0000 7f4ecde00640 1 -- 172.21.15.90:0/441885305 >> 172.21.15.90:0/441885305 conn(0x7f4ec8077b00 msgr2=0x7f4ec81018e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:36:10.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.352+0000 7f4ecde00640 1 -- 172.21.15.90:0/441885305 shutdown_connections 2024-04-02T14:36:10.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.352+0000 7f4ecde00640 1 -- 172.21.15.90:0/441885305 wait complete. 2024-04-02T14:36:10.363 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:36:10 smithi092 ceph-mon[36497]: pgmap v41: 97 pgs: 97 active+clean; 1.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 49 MiB/s wr, 1.28k op/s 2024-04-02T14:36:10.375 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:36:10 smithi090 ceph-mon[31674]: pgmap v41: 97 pgs: 97 active+clean; 1.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 49 MiB/s wr, 1.28k op/s 2024-04-02T14:36:10.380 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-02T14:36:10.436 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:36:10 smithi177 ceph-mon[36431]: pgmap v41: 97 pgs: 97 active+clean; 1.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 49 MiB/s wr, 1.28k op/s 2024-04-02T14:36:10.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.654+0000 7f960a600640 1 -- 172.21.15.90:0/4080754074 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9604108fe0 msgr2=0x7f9604110bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:10.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.654+0000 7f960a600640 1 --2- 172.21.15.90:0/4080754074 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9604108fe0 0x7f9604110bb0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f95fc009900 tx=0x7f95fc02bc40 comp rx=0 tx=0).stop 2024-04-02T14:36:10.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.655+0000 7f960a600640 1 -- 172.21.15.90:0/4080754074 shutdown_connections 2024-04-02T14:36:10.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.655+0000 7f960a600640 1 --2- 172.21.15.90:0/4080754074 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9604108fe0 0x7f9604110bb0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:10.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.655+0000 7f960a600640 1 --2- 172.21.15.90:0/4080754074 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f96041048c0 0x7f9604108900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:10.657 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.655+0000 7f960a600640 1 --2- 172.21.15.90:0/4080754074 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9604103f10 0x7f96041042f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:10.657 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.655+0000 7f960a600640 1 -- 172.21.15.90:0/4080754074 >> 172.21.15.90:0/4080754074 conn(0x7f96040ff7b0 msgr2=0x7f9604101bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:36:10.657 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.655+0000 7f960a600640 1 -- 172.21.15.90:0/4080754074 shutdown_connections 2024-04-02T14:36:10.657 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.655+0000 7f960a600640 1 -- 172.21.15.90:0/4080754074 wait complete. 2024-04-02T14:36:10.658 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.657+0000 7f960a600640 1 Processor -- start 2024-04-02T14:36:10.658 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.657+0000 7f960a600640 1 -- start start 2024-04-02T14:36:10.658 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.658+0000 7f960a600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9604103f10 0x7f960410f090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:10.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.658+0000 7f960a600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9604108fe0 0x7f960410f5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:10.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.658+0000 7f960a600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f960410fb10 0x7f96041aa870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:10.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.658+0000 7f960a600640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f9604076d20 con 0x7f960410fb10 2024-04-02T14:36:10.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.658+0000 7f960a600640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f9604076ba0 con 0x7f9604103f10 2024-04-02T14:36:10.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.658+0000 7f960a600640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f9604076ea0 con 0x7f9604108fe0 2024-04-02T14:36:10.660 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.658+0000 7f95faa00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9604108fe0 0x7f960410f5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:10.660 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.658+0000 7f95faa00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9604108fe0 0x7f960410f5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:53924/0 (socket says 172.21.15.90:53924) 2024-04-02T14:36:10.660 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.658+0000 7f95faa00640 1 -- 172.21.15.90:0/3900261094 learned_addr learned my addr 172.21.15.90:0/3900261094 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:36:10.660 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.658+0000 7f9603400640 1 --2- 172.21.15.90:0/3900261094 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9604103f10 0x7f960410f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:10.660 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.658+0000 7f95faa00640 1 -- 172.21.15.90:0/3900261094 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9604103f10 msgr2=0x7f960410f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:10.660 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.658+0000 7f95faa00640 1 --2- 172.21.15.90:0/3900261094 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9604103f10 0x7f960410f090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:10.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.658+0000 7f95faa00640 1 -- 172.21.15.90:0/3900261094 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f960410fb10 msgr2=0x7f96041aa870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:10.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.658+0000 7f95faa00640 1 --2- 172.21.15.90:0/3900261094 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f960410fb10 0x7f96041aa870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:10.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.658+0000 7f95faa00640 1 -- 172.21.15.90:0/3900261094 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96041ab040 con 0x7f9604108fe0 2024-04-02T14:36:10.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.659+0000 7f95faa00640 1 --2- 172.21.15.90:0/3900261094 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9604108fe0 0x7f960410f5d0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f95ec00e9c0 tx=0x7f95ec00ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:36:10.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.659+0000 7f9600c00640 1 -- 172.21.15.90:0/3900261094 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95ec00cd80 con 0x7f9604108fe0 2024-04-02T14:36:10.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.659+0000 7f9600c00640 1 -- 172.21.15.90:0/3900261094 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f95ec005500 con 0x7f9604108fe0 2024-04-02T14:36:10.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.659+0000 7f9600c00640 1 -- 172.21.15.90:0/3900261094 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95ec010690 con 0x7f9604108fe0 2024-04-02T14:36:10.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.659+0000 7f960a600640 1 -- 172.21.15.90:0/3900261094 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f96041ab330 con 0x7f9604108fe0 2024-04-02T14:36:10.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.659+0000 7f960a600640 1 -- 172.21.15.90:0/3900261094 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96041ab830 con 0x7f9604108fe0 2024-04-02T14:36:10.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.663+0000 7f9600c00640 1 -- 172.21.15.90:0/3900261094 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 25) ==== 138930+0+0 (secure 0 0 0) 0x7f95ec005020 con 0x7f9604108fe0 2024-04-02T14:36:10.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.664+0000 7f9600c00640 1 --2- 172.21.15.90:0/3900261094 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f95e80a3fd0 0x7f95e80a6490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:10.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.664+0000 7f9600c00640 1 -- 172.21.15.90:0/3900261094 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f95ec0d0f60 con 0x7f9604108fe0 2024-04-02T14:36:10.666 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.665+0000 7f9603400640 1 --2- 172.21.15.90:0/3900261094 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f95e80a3fd0 0x7f95e80a6490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:10.666 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.665+0000 7f960d390740 1 Processor -- start 2024-04-02T14:36:10.666 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.666+0000 7f960d390740 1 -- start start 2024-04-02T14:36:10.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.666+0000 7f960d390740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560f6da3d9f0 0x560f6da3c740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:10.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.666+0000 7f960d390740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560f6da3cd10 0x560f6da3d190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:10.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.666+0000 7f960d390740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560f6da46080 0x560f6da48470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:10.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.666+0000 7f960d390740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x560f6d97d400 con 0x560f6da46080 2024-04-02T14:36:10.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.666+0000 7f960d390740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x560f6d961f10 con 0x560f6da3cd10 2024-04-02T14:36:10.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.666+0000 7f960d390740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x560f6d9a16f0 con 0x560f6da3d9f0 2024-04-02T14:36:10.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.667+0000 7f95faa00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560f6da3cd10 0x560f6da3d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:10.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.667+0000 7f95faa00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560f6da3cd10 0x560f6da3d190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:33246/0 (socket says 172.21.15.90:33246) 2024-04-02T14:36:10.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.667+0000 7f95faa00640 1 -- 172.21.15.90:0/2614458344 learned_addr learned my addr 172.21.15.90:0/2614458344 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:36:10.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.667+0000 7f9603e00640 1 --2- 172.21.15.90:0/2614458344 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560f6da46080 0x560f6da48470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:10.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.667+0000 7f95faa00640 1 -- 172.21.15.90:0/2614458344 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560f6da3d9f0 msgr2=0x560f6da3c740 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-02T14:36:10.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.667+0000 7f95faa00640 1 --2- 172.21.15.90:0/2614458344 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560f6da3d9f0 0x560f6da3c740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:10.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.667+0000 7f95faa00640 1 -- 172.21.15.90:0/2614458344 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560f6da46080 msgr2=0x560f6da48470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:36:10.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.667+0000 7f95faa00640 1 --2- 172.21.15.90:0/2614458344 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560f6da46080 0x560f6da48470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:10.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.667+0000 7f95faa00640 1 -- 172.21.15.90:0/2614458344 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560f6d6b53f0 con 0x560f6da3cd10 2024-04-02T14:36:10.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.667+0000 7f95faa00640 1 --2- 172.21.15.90:0/2614458344 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560f6da3cd10 0x560f6da3d190 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f95ec0996a0 tx=0x7f95ec09c810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=58e113b35e25ee27 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:36:10.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.668+0000 7f95f8c00640 1 -- 172.21.15.90:0/2614458344 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95ec09ccd0 con 0x560f6da3cd10 2024-04-02T14:36:10.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.668+0000 7f95f8c00640 1 -- 172.21.15.90:0/2614458344 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f95ec0a1080 con 0x560f6da3cd10 2024-04-02T14:36:10.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.668+0000 7f95f8c00640 1 -- 172.21.15.90:0/2614458344 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95ec0bd0f0 con 0x560f6da3cd10 2024-04-02T14:36:10.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.666+0000 7f9603400640 1 --2- 172.21.15.90:0/3900261094 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f95e80a3fd0 0x7f95e80a6490 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f95f4005fd0 tx=0x7f95f4009500 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:36:10.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.669+0000 7f960d390740 1 -- 172.21.15.90:0/2614458344 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560f6da3cd10 msgr2=0x560f6da3d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:10.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.669+0000 7f960d390740 1 --2- 172.21.15.90:0/2614458344 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560f6da3cd10 0x560f6da3d190 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f95ec0996a0 tx=0x7f95ec09c810 comp rx=0 tx=0).stop 2024-04-02T14:36:10.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.670+0000 7f960d390740 1 -- 172.21.15.90:0/2614458344 shutdown_connections 2024-04-02T14:36:10.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.670+0000 7f960d390740 1 --2- 172.21.15.90:0/2614458344 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560f6da46080 0x560f6da48470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:10.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.670+0000 7f960d390740 1 --2- 172.21.15.90:0/2614458344 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560f6da3cd10 0x560f6da3d190 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:10.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.670+0000 7f960d390740 1 --2- 172.21.15.90:0/2614458344 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560f6da3d9f0 0x560f6da3c740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:10.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.670+0000 7f960d390740 1 -- 172.21.15.90:0/2614458344 >> 172.21.15.90:0/2614458344 conn(0x560f6da03a50 msgr2=0x560f6da37760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:36:10.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.670+0000 7f960d390740 1 -- 172.21.15.90:0/2614458344 shutdown_connections 2024-04-02T14:36:10.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.670+0000 7f960d390740 1 -- 172.21.15.90:0/2614458344 wait complete. 2024-04-02T14:36:10.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.672+0000 7f960d390740 1 -- start start 2024-04-02T14:36:10.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.674+0000 7f960d390740 1 Processor -- start 2024-04-02T14:36:10.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.675+0000 7f960d390740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560f6da3cd10 0x560f6da430f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:10.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.675+0000 7f960d390740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560f6da3d9f0 0x560f6da43630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:10.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.675+0000 7f960d390740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560f6da46080 0x560f6da43b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:10.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.675+0000 7f960d390740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x560f6d99d6a0 con 0x560f6da3d9f0 2024-04-02T14:36:10.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.675+0000 7f960d390740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x560f6d949980 con 0x560f6da3cd10 2024-04-02T14:36:10.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.675+0000 7f960d390740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x560f6d95ccb0 con 0x560f6da46080 2024-04-02T14:36:10.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.675+0000 7f960d390740 10 client.0 _ll_register_callbacks cb 0x560f6d95c6a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:36:10.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.675+0000 7f95faa00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560f6da3d9f0 0x560f6da43630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:10.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.675+0000 7f95faa00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560f6da3d9f0 0x560f6da43630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:46214/0 (socket says 172.21.15.90:46214) 2024-04-02T14:36:10.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.675+0000 7f95faa00640 1 -- 172.21.15.90:0/1873469830 learned_addr learned my addr 172.21.15.90:0/1873469830 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:36:10.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.676+0000 7f95faa00640 1 -- 172.21.15.90:0/1873469830 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560f6da46080 msgr2=0x560f6da43b70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-02T14:36:10.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.676+0000 7f95faa00640 1 --2- 172.21.15.90:0/1873469830 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560f6da46080 0x560f6da43b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:10.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.676+0000 7f95faa00640 1 -- 172.21.15.90:0/1873469830 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560f6da3cd10 msgr2=0x560f6da430f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-02T14:36:10.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.676+0000 7f95faa00640 1 --2- 172.21.15.90:0/1873469830 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560f6da3cd10 0x560f6da430f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:10.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.676+0000 7f95faa00640 1 -- 172.21.15.90:0/1873469830 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560f6d77f7d0 con 0x560f6da3d9f0 2024-04-02T14:36:10.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.677+0000 7f95faa00640 1 --2- 172.21.15.90:0/1873469830 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560f6da3d9f0 0x560f6da43630 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f95ec0a5410 tx=0x7f95ec004960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe56f38b83c69fd9 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:36:10.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.677+0000 7f95f1600640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-02T14:36:10.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.678+0000 7f95f1600640 1 -- 172.21.15.90:0/1873469830 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95ec0bd250 con 0x560f6da3d9f0 2024-04-02T14:36:10.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.678+0000 7f95f1600640 1 -- 172.21.15.90:0/1873469830 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f95ec09c120 con 0x560f6da3d9f0 2024-04-02T14:36:10.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.678+0000 7f960d390740 1 -- 172.21.15.90:0/1873469830 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f95ec0a1080 con 0x560f6da3d9f0 2024-04-02T14:36:10.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.679+0000 7f95f1600640 1 -- 172.21.15.90:0/1873469830 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f95ec081050 con 0x560f6da3d9f0 2024-04-02T14:36:10.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.679+0000 7f960d390740 10 client.15642 fetch_fsmap learned FSMap version 27 2024-04-02T14:36:10.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.679+0000 7f960d390740 1 -- 172.21.15.90:0/1873469830 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x560f6da54f50 con 0x560f6da3d9f0 2024-04-02T14:36:10.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.679+0000 7f95f1600640 1 -- 172.21.15.90:0/1873469830 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f95ec0a1080 con 0x560f6da3d9f0 2024-04-02T14:36:10.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.679+0000 7f95f1600640 1 -- 172.21.15.90:0/1873469830 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f95ec0a5b90 con 0x560f6da3d9f0 2024-04-02T14:36:10.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.680+0000 7f960d390740 10 client.15642 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:36:10.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.680+0000 7f960d390740 10 client.15642 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:36:10.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.680+0000 7f960d390740 20 client.15642 populate_metadata read hostname 'smithi090' 2024-04-02T14:36:10.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.680+0000 7f960d390740 1 --2- 172.21.15.90:0/1873469830 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x560f6da562a0 0x560f6da58690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:10.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.680+0000 7f960d390740 4 client.15642 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:36:10.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.680+0000 7f960d390740 1 -- 172.21.15.90:0/1873469830 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560f6d961040 con 0x560f6da562a0 2024-04-02T14:36:10.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.680+0000 7f9603400640 1 --2- 172.21.15.90:0/1873469830 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x560f6da562a0 0x560f6da58690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:10.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.681+0000 7f9603400640 1 --2- 172.21.15.90:0/1873469830 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x560f6da562a0 0x560f6da58690 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=50fc5e8b8d56107e server_cookie=a69d5be0af83287d in_seq=0 out_seq=0 2024-04-02T14:36:10.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.681+0000 7f95f1600640 10 client.15642 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:36:10.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.682+0000 7f95f1600640 1 -- 172.21.15.90:0/1873469830 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x560f6d961040 con 0x560f6da562a0 2024-04-02T14:36:10.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.682+0000 7f95f1600640 10 client.15642 handle_command_reply: tid=0 2024-04-02T14:36:10.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.683+0000 7f960d390740 1 client.15642 shutdown 2024-04-02T14:36:10.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.683+0000 7f960d390740 1 -- 172.21.15.90:0/1873469830 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560f6da3d9f0 msgr2=0x560f6da43630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:10.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.683+0000 7f960d390740 1 --2- 172.21.15.90:0/1873469830 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560f6da3d9f0 0x560f6da43630 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f95ec0a5410 tx=0x7f95ec004960 comp rx=0 tx=0).stop 2024-04-02T14:36:10.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.683+0000 7f960d390740 1 -- 172.21.15.90:0/1873469830 shutdown_connections 2024-04-02T14:36:10.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.683+0000 7f960d390740 1 --2- 172.21.15.90:0/1873469830 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x560f6da562a0 0x560f6da58690 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:10.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.684+0000 7f960d390740 1 --2- 172.21.15.90:0/1873469830 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560f6da46080 0x560f6da43b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:10.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.684+0000 7f960d390740 1 --2- 172.21.15.90:0/1873469830 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560f6da3d9f0 0x560f6da43630 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:10.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.684+0000 7f95f1600640 0 client.15642 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:36:10.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.684+0000 7f960d390740 1 --2- 172.21.15.90:0/1873469830 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560f6da3cd10 0x560f6da430f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:10.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.684+0000 7f960d390740 1 -- 172.21.15.90:0/1873469830 >> 172.21.15.90:0/1873469830 conn(0x560f6da03a50 msgr2=0x560f6da394b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:36:10.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.684+0000 7f960d390740 1 -- 172.21.15.90:0/1873469830 shutdown_connections 2024-04-02T14:36:10.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.684+0000 7f960d390740 1 -- 172.21.15.90:0/1873469830 wait complete. 2024-04-02T14:36:10.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.684+0000 7f960d390740 20 client.15642 trim_cache size 0 max 16384 2024-04-02T14:36:10.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.711+0000 7f960d390740 1 Processor -- start 2024-04-02T14:36:10.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.711+0000 7f960d390740 1 -- start start 2024-04-02T14:36:10.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.712+0000 7f960d390740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560f6da3cd10 0x560f6da45820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:10.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.712+0000 7f960d390740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560f6da3d9f0 0x560f6da51a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:10.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.712+0000 7f960d390740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560f6da46080 0x560f6da392c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:10.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.712+0000 7f960d390740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f9604073890 con 0x560f6da3d9f0 2024-04-02T14:36:10.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.712+0000 7f960d390740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f96041ad190 con 0x560f6da3cd10 2024-04-02T14:36:10.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.712+0000 7f960d390740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f96041abd20 con 0x560f6da46080 2024-04-02T14:36:10.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.713+0000 7f9603400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560f6da3cd10 0x560f6da45820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:10.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.713+0000 7f9603400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560f6da3cd10 0x560f6da45820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:33256/0 (socket says 172.21.15.90:33256) 2024-04-02T14:36:10.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.713+0000 7f9603400640 1 -- 172.21.15.90:0/543330896 learned_addr learned my addr 172.21.15.90:0/543330896 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:36:10.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.713+0000 7f95faa00640 1 --2- 172.21.15.90:0/543330896 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560f6da3d9f0 0x560f6da51a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:10.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.713+0000 7f9603e00640 1 --2- 172.21.15.90:0/543330896 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560f6da46080 0x560f6da392c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:10.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.714+0000 7f9603400640 1 -- 172.21.15.90:0/543330896 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560f6da46080 msgr2=0x560f6da392c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:36:10.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.714+0000 7f9603400640 1 --2- 172.21.15.90:0/543330896 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560f6da46080 0x560f6da392c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:10.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.714+0000 7f9603400640 1 -- 172.21.15.90:0/543330896 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560f6da3d9f0 msgr2=0x560f6da51a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:36:10.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.714+0000 7f9603400640 1 --2- 172.21.15.90:0/543330896 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560f6da3d9f0 0x560f6da51a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:10.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.715+0000 7f9603400640 1 -- 172.21.15.90:0/543330896 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560f6da3bd40 con 0x560f6da3cd10 2024-04-02T14:36:10.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.715+0000 7f9603400640 1 --2- 172.21.15.90:0/543330896 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560f6da3cd10 0x560f6da45820 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x560f6da58c00 tx=0x7f95f4013000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b0bae876bafe9839 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:36:10.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.716+0000 7f95f3e00640 1 -- 172.21.15.90:0/543330896 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95f402b070 con 0x560f6da3cd10 2024-04-02T14:36:10.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.716+0000 7f95f3e00640 1 -- 172.21.15.90:0/543330896 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f95f4026d50 con 0x560f6da3cd10 2024-04-02T14:36:10.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.716+0000 7f95f3e00640 1 -- 172.21.15.90:0/543330896 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95f40042b0 con 0x560f6da3cd10 2024-04-02T14:36:10.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.717+0000 7f960d390740 1 -- 172.21.15.90:0/543330896 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560f6da3cd10 msgr2=0x560f6da45820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:10.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.717+0000 7f960d390740 1 --2- 172.21.15.90:0/543330896 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560f6da3cd10 0x560f6da45820 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x560f6da58c00 tx=0x7f95f4013000 comp rx=0 tx=0).stop 2024-04-02T14:36:10.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.717+0000 7f960d390740 1 -- 172.21.15.90:0/543330896 shutdown_connections 2024-04-02T14:36:10.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.717+0000 7f960d390740 1 --2- 172.21.15.90:0/543330896 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560f6da46080 0x560f6da392c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:10.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.717+0000 7f960d390740 1 --2- 172.21.15.90:0/543330896 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560f6da3d9f0 0x560f6da51a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:10.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.717+0000 7f960d390740 1 --2- 172.21.15.90:0/543330896 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560f6da3cd10 0x560f6da45820 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:10.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.717+0000 7f960d390740 1 -- 172.21.15.90:0/543330896 >> 172.21.15.90:0/543330896 conn(0x560f6da03a50 msgr2=0x560f6da38bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:36:10.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.718+0000 7f960d390740 1 -- 172.21.15.90:0/543330896 shutdown_connections 2024-04-02T14:36:10.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.718+0000 7f960d390740 1 -- 172.21.15.90:0/543330896 wait complete. 2024-04-02T14:36:10.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.719+0000 7f960d390740 1 -- start start 2024-04-02T14:36:10.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.719+0000 7f960d390740 1 Processor -- start 2024-04-02T14:36:10.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.720+0000 7f960d390740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560f6da3cd10 0x560f6da47030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:10.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.720+0000 7f960d390740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560f6da3d9f0 0x560f6da47570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:10.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.720+0000 7f960d390740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560f6da46080 0x560f6da4c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:10.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.720+0000 7f960d390740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f96041acac0 con 0x560f6da3d9f0 2024-04-02T14:36:10.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.720+0000 7f960d390740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f96041ac3f0 con 0x560f6da46080 2024-04-02T14:36:10.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.720+0000 7f960d390740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f96041105c0 con 0x560f6da3cd10 2024-04-02T14:36:10.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.720+0000 7f960d390740 10 client.0 _ll_register_callbacks cb 0x7f95ec0ce3f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:36:10.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.720+0000 7f9603e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560f6da46080 0x560f6da4c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:10.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.720+0000 7f9603e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560f6da46080 0x560f6da4c4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:33262/0 (socket says 172.21.15.90:33262) 2024-04-02T14:36:10.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.720+0000 7f9603e00640 1 -- 172.21.15.90:0/1997478030 learned_addr learned my addr 172.21.15.90:0/1997478030 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:36:10.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.721+0000 7f9603e00640 1 -- 172.21.15.90:0/1997478030 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560f6da3cd10 msgr2=0x560f6da47030 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-02T14:36:10.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.721+0000 7f9603e00640 1 --2- 172.21.15.90:0/1997478030 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560f6da3cd10 0x560f6da47030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:10.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.721+0000 7f9603e00640 1 -- 172.21.15.90:0/1997478030 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560f6da3d9f0 msgr2=0x560f6da47570 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-02T14:36:10.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.721+0000 7f9603e00640 1 --2- 172.21.15.90:0/1997478030 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560f6da3d9f0 0x560f6da47570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:10.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.721+0000 7f9603e00640 1 -- 172.21.15.90:0/1997478030 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560f6da4ca00 con 0x560f6da46080 2024-04-02T14:36:10.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.722+0000 7f9603e00640 1 --2- 172.21.15.90:0/1997478030 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560f6da46080 0x560f6da4c4c0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f95fc009230 tx=0x7f95fc0095a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1398c3b4d64e8713 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:36:10.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.722+0000 7f95f1600640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:36:10.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.722+0000 7f95f1600640 1 -- 172.21.15.90:0/1997478030 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95fc02caa0 con 0x560f6da46080 2024-04-02T14:36:10.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.723+0000 7f95f1600640 1 -- 172.21.15.90:0/1997478030 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f95fc04a040 con 0x560f6da46080 2024-04-02T14:36:10.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.723+0000 7f960d390740 1 -- 172.21.15.90:0/1997478030 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f95f4026d50 con 0x560f6da46080 2024-04-02T14:36:10.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.723+0000 7f95f1600640 1 -- 172.21.15.90:0/1997478030 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f95fc02caa0 con 0x560f6da46080 2024-04-02T14:36:10.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.724+0000 7f960d390740 10 client.25826 fetch_fsmap learned FSMap version 27 2024-04-02T14:36:10.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.724+0000 7f960d390740 1 -- 172.21.15.90:0/1997478030 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x560f6dabcbb0 con 0x560f6da46080 2024-04-02T14:36:10.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.724+0000 7f95f1600640 1 -- 172.21.15.90:0/1997478030 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f95f4026d50 con 0x560f6da46080 2024-04-02T14:36:10.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.724+0000 7f95f1600640 1 -- 172.21.15.90:0/1997478030 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f95fc004a90 con 0x560f6da46080 2024-04-02T14:36:10.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.724+0000 7f960d390740 10 client.25826 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:36:10.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.724+0000 7f960d390740 10 client.25826 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:36:10.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.724+0000 7f960d390740 20 client.25826 populate_metadata read hostname 'smithi090' 2024-04-02T14:36:10.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.724+0000 7f960d390740 1 --2- 172.21.15.90:0/1997478030 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x560f6dabd380 0x560f6dabf7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:10.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.724+0000 7f960d390740 4 client.25826 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-02T14:36:10.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.724+0000 7f960d390740 1 -- 172.21.15.90:0/1997478030 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "damage ls"}) -- 0x560f6d9aec30 con 0x560f6dabd380 2024-04-02T14:36:10.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.725+0000 7f9603400640 1 --2- 172.21.15.90:0/1997478030 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x560f6dabd380 0x560f6dabf7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:10.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.726+0000 7f9603400640 1 --2- 172.21.15.90:0/1997478030 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x560f6dabd380 0x560f6dabf7c0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=47bca610458380ce server_cookie=d35d058d32be6cb3 in_seq=0 out_seq=0 2024-04-02T14:36:10.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.727+0000 7f95f1600640 10 client.25826 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:36:10.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.727+0000 7f95f1600640 1 -- 172.21.15.90:0/1997478030 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x560f6d9aec30 con 0x560f6dabd380 2024-04-02T14:36:10.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.727+0000 7f95f1600640 10 client.25826 handle_command_reply: tid=0 2024-04-02T14:36:10.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.727+0000 7f960d390740 1 client.25826 shutdown 2024-04-02T14:36:10.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.727+0000 7f960d390740 1 -- 172.21.15.90:0/1997478030 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560f6da46080 msgr2=0x560f6da4c4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:10.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.727+0000 7f960d390740 1 --2- 172.21.15.90:0/1997478030 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560f6da46080 0x560f6da4c4c0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f95fc009230 tx=0x7f95fc0095a0 comp rx=0 tx=0).stop 2024-04-02T14:36:10.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.728+0000 7f960d390740 1 -- 172.21.15.90:0/1997478030 shutdown_connections 2024-04-02T14:36:10.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.728+0000 7f960d390740 1 --2- 172.21.15.90:0/1997478030 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x560f6dabd380 0x560f6dabf7c0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:10.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.728+0000 7f960d390740 1 --2- 172.21.15.90:0/1997478030 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560f6da46080 0x560f6da4c4c0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:10.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.728+0000 7f960d390740 1 --2- 172.21.15.90:0/1997478030 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560f6da3d9f0 0x560f6da47570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:10.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.728+0000 7f960d390740 1 --2- 172.21.15.90:0/1997478030 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560f6da3cd10 0x560f6da47030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:10.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.728+0000 7f960d390740 1 -- 172.21.15.90:0/1997478030 >> 172.21.15.90:0/1997478030 conn(0x560f6da03a50 msgr2=0x560f6da43920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:36:10.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.728+0000 7f95f1600640 0 client.25826 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:36:10.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.728+0000 7f960d390740 1 -- 172.21.15.90:0/1997478030 shutdown_connections 2024-04-02T14:36:10.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.728+0000 7f960d390740 1 -- 172.21.15.90:0/1997478030 wait complete. 2024-04-02T14:36:10.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.729+0000 7f960d390740 20 client.25826 trim_cache size 0 max 16384 2024-04-02T14:36:10.730 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-02T14:36:10.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.730+0000 7f960a600640 1 -- 172.21.15.90:0/3900261094 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f95e80a3fd0 msgr2=0x7f95e80a6490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:10.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.730+0000 7f960a600640 1 --2- 172.21.15.90:0/3900261094 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f95e80a3fd0 0x7f95e80a6490 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f95f4005fd0 tx=0x7f95f4009500 comp rx=0 tx=0).stop 2024-04-02T14:36:10.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.730+0000 7f960a600640 1 -- 172.21.15.90:0/3900261094 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9604108fe0 msgr2=0x7f960410f5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:10.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.730+0000 7f960a600640 1 --2- 172.21.15.90:0/3900261094 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9604108fe0 0x7f960410f5d0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f95ec00e9c0 tx=0x7f95ec00ee80 comp rx=0 tx=0).stop 2024-04-02T14:36:10.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.730+0000 7f960a600640 1 -- 172.21.15.90:0/3900261094 shutdown_connections 2024-04-02T14:36:10.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.730+0000 7f960a600640 1 --2- 172.21.15.90:0/3900261094 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f95e80a3fd0 0x7f95e80a6490 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:10.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.730+0000 7f960a600640 1 --2- 172.21.15.90:0/3900261094 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f960410fb10 0x7f96041aa870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:10.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.730+0000 7f960a600640 1 --2- 172.21.15.90:0/3900261094 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9604108fe0 0x7f960410f5d0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:10.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.730+0000 7f960a600640 1 --2- 172.21.15.90:0/3900261094 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9604103f10 0x7f960410f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:10.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.730+0000 7f960a600640 1 -- 172.21.15.90:0/3900261094 >> 172.21.15.90:0/3900261094 conn(0x7f96040ff7b0 msgr2=0x7f9604101af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:36:10.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.731+0000 7f960a600640 1 -- 172.21.15.90:0/3900261094 shutdown_connections 2024-04-02T14:36:10.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:10.731+0000 7f960a600640 1 -- 172.21.15.90:0/3900261094 wait complete. 2024-04-02T14:36:10.966 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:10.965+0000 7f087ac00640 1 -- 172.21.15.90:0/1930597262 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0874112880 msgr2=0x7f0874114c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:10.966 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:10.965+0000 7f087ac00640 1 --2- 172.21.15.90:0/1930597262 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0874112880 0x7f0874114c70 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f086c009860 tx=0x7f086c02bc10 comp rx=0 tx=0).stop 2024-04-02T14:36:10.966 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:10.965+0000 7f087ac00640 1 -- 172.21.15.90:0/1930597262 shutdown_connections 2024-04-02T14:36:10.966 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:10.965+0000 7f087ac00640 1 --2- 172.21.15.90:0/1930597262 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0874112880 0x7f0874114c70 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:10.966 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:10.965+0000 7f087ac00640 1 --2- 172.21.15.90:0/1930597262 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0874100480 0x7f0874112340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:10.967 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:10.965+0000 7f087ac00640 1 --2- 172.21.15.90:0/1930597262 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f08740ffb60 0x7f08740fff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:10.967 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:10.965+0000 7f087ac00640 1 -- 172.21.15.90:0/1930597262 >> 172.21.15.90:0/1930597262 conn(0x7f0874076ae0 msgr2=0x7f08740fde70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:36:10.967 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:10.966+0000 7f087ac00640 1 -- 172.21.15.90:0/1930597262 shutdown_connections 2024-04-02T14:36:10.967 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:10.966+0000 7f087ac00640 1 -- 172.21.15.90:0/1930597262 wait complete. 2024-04-02T14:36:10.968 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:10.967+0000 7f087ac00640 1 Processor -- start 2024-04-02T14:36:10.969 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:10.967+0000 7f087ac00640 1 -- start start 2024-04-02T14:36:10.969 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:10.967+0000 7f087ac00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f08740ffb60 0x7f08741a45a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:10.969 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:10.967+0000 7f087ac00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0874100480 0x7f08741a4ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:10.969 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:10.967+0000 7f087ac00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0874112880 0x7f087419e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:10.969 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:10.967+0000 7f087ac00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f0874118470 con 0x7f0874112880 2024-04-02T14:36:10.969 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:10.967+0000 7f087ac00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f08741182f0 con 0x7f08740ffb60 2024-04-02T14:36:10.969 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:10.967+0000 7f087ac00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f08741185f0 con 0x7f0874100480 2024-04-02T14:36:10.969 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:10.968+0000 7f0873400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0874100480 0x7f08741a4ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:10.969 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:10.968+0000 7f0873400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0874100480 0x7f08741a4ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:53944/0 (socket says 172.21.15.90:53944) 2024-04-02T14:36:10.969 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:10.968+0000 7f0873400640 1 -- 172.21.15.90:0/3969149251 learned_addr learned my addr 172.21.15.90:0/3969149251 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:36:10.970 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:10.968+0000 7f0873e00640 1 --2- 172.21.15.90:0/3969149251 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f08740ffb60 0x7f08741a45a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:10.970 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:10.968+0000 7f0878c00640 1 --2- 172.21.15.90:0/3969149251 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0874112880 0x7f087419e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:10.970 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:10.968+0000 7f0873400640 1 -- 172.21.15.90:0/3969149251 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f08740ffb60 msgr2=0x7f08741a45a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:10.970 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:10.969+0000 7f0873400640 1 --2- 172.21.15.90:0/3969149251 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f08740ffb60 0x7f08741a45a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:10.970 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:10.969+0000 7f0873400640 1 -- 172.21.15.90:0/3969149251 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0874112880 msgr2=0x7f087419e6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:10.970 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:10.969+0000 7f0873400640 1 --2- 172.21.15.90:0/3969149251 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0874112880 0x7f087419e6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:10.970 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:10.969+0000 7f0873400640 1 -- 172.21.15.90:0/3969149251 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f087419eec0 con 0x7f0874100480 2024-04-02T14:36:10.970 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:10.969+0000 7f0878c00640 1 --2- 172.21.15.90:0/3969149251 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0874112880 0x7f087419e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:36:10.971 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:10.969+0000 7f0873e00640 1 --2- 172.21.15.90:0/3969149251 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f08740ffb60 0x7f08741a45a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:36:10.971 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:10.969+0000 7f0873400640 1 --2- 172.21.15.90:0/3969149251 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0874100480 0x7f08741a4ae0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f086000ea20 tx=0x7f086000eee0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:36:10.971 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:10.969+0000 7f0870c00640 1 -- 172.21.15.90:0/3969149251 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f086000cd80 con 0x7f0874100480 2024-04-02T14:36:10.971 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:10.970+0000 7f087ac00640 1 -- 172.21.15.90:0/3969149251 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f087419f1b0 con 0x7f0874100480 2024-04-02T14:36:10.971 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:10.970+0000 7f0870c00640 1 -- 172.21.15.90:0/3969149251 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0860010910 con 0x7f0874100480 2024-04-02T14:36:10.971 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:10.970+0000 7f0870c00640 1 -- 172.21.15.90:0/3969149251 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0860018920 con 0x7f0874100480 2024-04-02T14:36:10.971 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:10.970+0000 7f087ac00640 1 -- 172.21.15.90:0/3969149251 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0874107d00 con 0x7f0874100480 2024-04-02T14:36:10.976 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:10.975+0000 7f0870c00640 1 -- 172.21.15.90:0/3969149251 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 25) ==== 138930+0+0 (secure 0 0 0) 0x7f0860010ac0 con 0x7f0874100480 2024-04-02T14:36:10.978 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:10.977+0000 7f0870c00640 1 --2- 172.21.15.90:0/3969149251 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f08580a3f70 0x7f08580a6430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:10.983 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:10.978+0000 7f0870c00640 1 -- 172.21.15.90:0/3969149251 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f08600d10a0 con 0x7f0874100480 2024-04-02T14:36:10.983 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:10.979+0000 7f087d980740 1 Processor -- start 2024-04-02T14:36:10.983 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:10.979+0000 7f087d980740 1 -- start start 2024-04-02T14:36:10.983 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:10.979+0000 7f0873e00640 1 --2- 172.21.15.90:0/3969149251 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f08580a3f70 0x7f08580a6430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:10.983 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:10.979+0000 7f087d980740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eeb0523eb0 0x55eeb0524290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:10.983 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:10.980+0000 7f087d980740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eeb05247d0 0x55eeb052ce70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:10.984 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:10.980+0000 7f087d980740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55eeb052d3b0 0x55eeb052f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:10.984 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:10.980+0000 7f087d980740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55eeb0488d70 con 0x55eeb05247d0 2024-04-02T14:36:10.984 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:10.980+0000 7f087d980740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55eeb04312a0 con 0x55eeb052d3b0 2024-04-02T14:36:10.984 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:10.980+0000 7f087d980740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55eeb0484d20 con 0x55eeb0523eb0 2024-04-02T14:36:10.984 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:10.980+0000 7f0878c00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55eeb052d3b0 0x55eeb052f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:10.984 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:10.980+0000 7f0878c00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55eeb052d3b0 0x55eeb052f7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:33284/0 (socket says 172.21.15.90:33284) 2024-04-02T14:36:10.984 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:10.980+0000 7f0878c00640 1 -- 172.21.15.90:0/2951716815 learned_addr learned my addr 172.21.15.90:0/2951716815 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:36:10.984 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:10.980+0000 7f0873400640 1 --2- 172.21.15.90:0/2951716815 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eeb05247d0 0x55eeb052ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:10.984 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:10.980+0000 7f0878c00640 1 -- 172.21.15.90:0/2951716815 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eeb0523eb0 msgr2=0x55eeb0524290 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-02T14:36:10.984 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:10.980+0000 7f0878c00640 1 --2- 172.21.15.90:0/2951716815 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eeb0523eb0 0x55eeb0524290 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:10.984 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:10.980+0000 7f0878c00640 1 -- 172.21.15.90:0/2951716815 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eeb05247d0 msgr2=0x55eeb052ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:36:10.984 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:10.980+0000 7f0878c00640 1 --2- 172.21.15.90:0/2951716815 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eeb05247d0 0x55eeb052ce70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:10.984 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:10.980+0000 7f0878c00640 1 -- 172.21.15.90:0/2951716815 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55eeb0445840 con 0x55eeb052d3b0 2024-04-02T14:36:10.984 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:10.980+0000 7f0873400640 1 --2- 172.21.15.90:0/2951716815 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eeb05247d0 0x55eeb052ce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:36:10.984 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:10.981+0000 7f0878c00640 1 --2- 172.21.15.90:0/2951716815 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55eeb052d3b0 0x55eeb052f7a0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f086c036000 tx=0x7f086c035b20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d8f78c34e451155d server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:36:10.984 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:10.981+0000 7f0869200640 1 -- 172.21.15.90:0/2951716815 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f086c039050 con 0x55eeb052d3b0 2024-04-02T14:36:10.985 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:10.981+0000 7f0869200640 1 -- 172.21.15.90:0/2951716815 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f086c043070 con 0x55eeb052d3b0 2024-04-02T14:36:10.985 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:10.981+0000 7f0869200640 1 -- 172.21.15.90:0/2951716815 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f086c039050 con 0x55eeb052d3b0 2024-04-02T14:36:10.985 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:10.983+0000 7f087d980740 1 -- 172.21.15.90:0/2951716815 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55eeb052d3b0 msgr2=0x55eeb052f7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:10.986 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:10.983+0000 7f087d980740 1 --2- 172.21.15.90:0/2951716815 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55eeb052d3b0 0x55eeb052f7a0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f086c036000 tx=0x7f086c035b20 comp rx=0 tx=0).stop 2024-04-02T14:36:10.986 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:10.983+0000 7f0873e00640 1 --2- 172.21.15.90:0/3969149251 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f08580a3f70 0x7f08580a6430 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f0864005fd0 tx=0x7f0864010040 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:36:10.986 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:10.984+0000 7f087d980740 1 -- 172.21.15.90:0/2951716815 shutdown_connections 2024-04-02T14:36:10.986 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:10.984+0000 7f087d980740 1 --2- 172.21.15.90:0/2951716815 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55eeb052d3b0 0x55eeb052f7a0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:10.986 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:10.984+0000 7f087d980740 1 --2- 172.21.15.90:0/2951716815 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eeb05247d0 0x55eeb052ce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:10.986 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:10.984+0000 7f087d980740 1 --2- 172.21.15.90:0/2951716815 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eeb0523eb0 0x55eeb0524290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:10.986 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:10.984+0000 7f087d980740 1 -- 172.21.15.90:0/2951716815 >> 172.21.15.90:0/2951716815 conn(0x55eeb051e5e0 msgr2=0x55eeb051e9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:36:10.986 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:10.984+0000 7f087d980740 1 -- 172.21.15.90:0/2951716815 shutdown_connections 2024-04-02T14:36:10.986 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:10.984+0000 7f087d980740 1 -- 172.21.15.90:0/2951716815 wait complete. 2024-04-02T14:36:10.986 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:10.985+0000 7f087d980740 1 -- start start 2024-04-02T14:36:10.986 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:10.985+0000 7f087d980740 1 Processor -- start 2024-04-02T14:36:10.987 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:10.985+0000 7f087d980740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eeb0523eb0 0x55eeb052a480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:10.987 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:10.985+0000 7f087d980740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55eeb05247d0 0x55eeb052a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:10.987 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:10.985+0000 7f087d980740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eeb052d3b0 0x55eeb052af00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:10.987 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:10.985+0000 7f087d980740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55eeb0530140 con 0x55eeb052d3b0 2024-04-02T14:36:10.987 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:10.985+0000 7f087d980740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55eeb0444660 con 0x55eeb05247d0 2024-04-02T14:36:10.987 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:10.985+0000 7f087d980740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55eeb05302c0 con 0x55eeb0523eb0 2024-04-02T14:36:10.987 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:10.985+0000 7f087d980740 10 client.0 _ll_register_callbacks cb 0x55eeb0444050 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:36:10.987 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:10.986+0000 7f0873e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eeb0523eb0 0x55eeb052a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:10.987 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:10.986+0000 7f0873e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eeb0523eb0 0x55eeb052a480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:53968/0 (socket says 172.21.15.90:53968) 2024-04-02T14:36:10.987 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:10.986+0000 7f0873400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55eeb05247d0 0x55eeb052a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:10.987 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:10.986+0000 7f0873e00640 1 -- 172.21.15.90:0/962282647 learned_addr learned my addr 172.21.15.90:0/962282647 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:36:10.988 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:10.986+0000 7f0878c00640 1 --2- 172.21.15.90:0/962282647 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eeb052d3b0 0x55eeb052af00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:10.988 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:10.986+0000 7f0878c00640 1 -- 172.21.15.90:0/962282647 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eeb0523eb0 msgr2=0x55eeb052a480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:36:10.989 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:10.986+0000 7f0878c00640 1 --2- 172.21.15.90:0/962282647 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eeb0523eb0 0x55eeb052a480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:10.989 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:10.986+0000 7f0878c00640 1 -- 172.21.15.90:0/962282647 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55eeb05247d0 msgr2=0x55eeb052a9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:36:10.989 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:10.986+0000 7f0878c00640 1 --2- 172.21.15.90:0/962282647 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55eeb05247d0 0x55eeb052a9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:10.989 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:10.986+0000 7f0878c00640 1 -- 172.21.15.90:0/962282647 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55eeb0453e80 con 0x55eeb052d3b0 2024-04-02T14:36:10.989 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:10.986+0000 7f0873400640 1 --2- 172.21.15.90:0/962282647 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55eeb05247d0 0x55eeb052a9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:36:10.989 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:10.987+0000 7f0878c00640 1 --2- 172.21.15.90:0/962282647 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eeb052d3b0 0x55eeb052af00 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f086c0095f0 tx=0x7f086c035f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5facb66d09482ce4 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:36:10.990 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:10.987+0000 7f085d600640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-02T14:36:10.990 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:10.988+0000 7f085d600640 1 -- 172.21.15.90:0/962282647 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f086c04d070 con 0x55eeb052d3b0 2024-04-02T14:36:10.990 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:10.988+0000 7f085d600640 1 -- 172.21.15.90:0/962282647 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f086c0046c0 con 0x55eeb052d3b0 2024-04-02T14:36:10.990 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:10.988+0000 7f087d980740 1 -- 172.21.15.90:0/962282647 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f086c043070 con 0x55eeb052d3b0 2024-04-02T14:36:10.991 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:10.989+0000 7f085d600640 1 -- 172.21.15.90:0/962282647 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f086c057050 con 0x55eeb052d3b0 2024-04-02T14:36:10.991 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:10.989+0000 7f087d980740 10 client.15651 fetch_fsmap learned FSMap version 27 2024-04-02T14:36:10.991 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:10.989+0000 7f087d980740 1 -- 172.21.15.90:0/962282647 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55eeb019d210 con 0x55eeb052d3b0 2024-04-02T14:36:10.991 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:10.989+0000 7f085d600640 1 -- 172.21.15.90:0/962282647 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f086c043070 con 0x55eeb052d3b0 2024-04-02T14:36:10.991 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:10.990+0000 7f085d600640 1 -- 172.21.15.90:0/962282647 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f086c02cb20 con 0x55eeb052d3b0 2024-04-02T14:36:10.991 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:10.990+0000 7f087d980740 10 client.15651 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:36:10.991 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:10.990+0000 7f087d980740 10 client.15651 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-02T14:36:10.991 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:10.990+0000 7f087d980740 20 client.15651 populate_metadata read hostname 'smithi090' 2024-04-02T14:36:10.992 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:10.990+0000 7f087d980740 1 --2- 172.21.15.90:0/962282647 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55eeb053d560 0x55eeb053f950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:10.992 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:10.990+0000 7f087d980740 4 client.15651 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:36:10.992 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:10.990+0000 7f087d980740 1 -- 172.21.15.90:0/962282647 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55eeb0438cd0 con 0x55eeb053d560 2024-04-02T14:36:10.992 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:10.990+0000 7f0873e00640 1 --2- 172.21.15.90:0/962282647 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55eeb053d560 0x55eeb053f950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:10.992 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:10.992+0000 7f085d600640 10 client.15651 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:36:10.993 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:10.992+0000 7f0873e00640 1 --2- 172.21.15.90:0/962282647 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55eeb053d560 0x55eeb053f950 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2cd6f0def3646ed server_cookie=3fde917bdac946a8 in_seq=0 out_seq=0 2024-04-02T14:36:10.994 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:10.993+0000 7f085d600640 1 -- 172.21.15.90:0/962282647 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55eeb0438cd0 con 0x55eeb053d560 2024-04-02T14:36:10.994 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:10.993+0000 7f085d600640 10 client.15651 handle_command_reply: tid=0 2024-04-02T14:36:10.994 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:10.993+0000 7f087d980740 1 client.15651 shutdown 2024-04-02T14:36:10.994 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:10.993+0000 7f087d980740 1 -- 172.21.15.90:0/962282647 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eeb052d3b0 msgr2=0x55eeb052af00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:10.995 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:10.993+0000 7f087d980740 1 --2- 172.21.15.90:0/962282647 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eeb052d3b0 0x55eeb052af00 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f086c0095f0 tx=0x7f086c035f70 comp rx=0 tx=0).stop 2024-04-02T14:36:10.995 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:10.993+0000 7f087d980740 1 -- 172.21.15.90:0/962282647 shutdown_connections 2024-04-02T14:36:10.995 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:10.993+0000 7f087d980740 1 --2- 172.21.15.90:0/962282647 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55eeb053d560 0x55eeb053f950 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:10.995 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:10.993+0000 7f087d980740 1 --2- 172.21.15.90:0/962282647 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eeb052d3b0 0x55eeb052af00 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:10.995 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:10.993+0000 7f087d980740 1 --2- 172.21.15.90:0/962282647 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55eeb05247d0 0x55eeb052a9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:10.995 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:10.993+0000 7f085d600640 0 client.15651 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:36:10.995 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:10.993+0000 7f087d980740 1 --2- 172.21.15.90:0/962282647 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eeb0523eb0 0x55eeb052a480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:10.995 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:10.993+0000 7f087d980740 1 -- 172.21.15.90:0/962282647 >> 172.21.15.90:0/962282647 conn(0x55eeb051e5e0 msgr2=0x55eeb0520330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:36:10.995 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:10.993+0000 7f087d980740 1 -- 172.21.15.90:0/962282647 shutdown_connections 2024-04-02T14:36:10.995 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:10.993+0000 7f087d980740 1 -- 172.21.15.90:0/962282647 wait complete. 2024-04-02T14:36:10.995 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:10.993+0000 7f087d980740 20 client.15651 trim_cache size 0 max 16384 2024-04-02T14:36:11.031 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:11.030+0000 7f087d980740 1 Processor -- start 2024-04-02T14:36:11.032 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:11.030+0000 7f087d980740 1 -- start start 2024-04-02T14:36:11.032 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:11.030+0000 7f087d980740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eeb0523eb0 0x55eeb052cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:11.032 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:11.030+0000 7f087d980740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eeb05247d0 0x55eeb0539000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:11.032 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:11.030+0000 7f087d980740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55eeb052d3b0 0x55eeb0544f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:11.032 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:11.030+0000 7f087d980740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f08741bc4a0 con 0x55eeb0523eb0 2024-04-02T14:36:11.032 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:11.030+0000 7f087d980740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f08741bdfe0 con 0x55eeb052d3b0 2024-04-02T14:36:11.032 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:11.030+0000 7f087d980740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f08741bcb70 con 0x55eeb05247d0 2024-04-02T14:36:11.032 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:11.031+0000 7f0878c00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55eeb052d3b0 0x55eeb0544f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:11.033 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:11.031+0000 7f0878c00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55eeb052d3b0 0x55eeb0544f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:33306/0 (socket says 172.21.15.90:33306) 2024-04-02T14:36:11.033 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:11.031+0000 7f0878c00640 1 -- 172.21.15.90:0/2185953198 learned_addr learned my addr 172.21.15.90:0/2185953198 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:36:11.033 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:11.031+0000 7f0873e00640 1 --2- 172.21.15.90:0/2185953198 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eeb0523eb0 0x55eeb052cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:11.033 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:11.031+0000 7f0873400640 1 --2- 172.21.15.90:0/2185953198 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eeb05247d0 0x55eeb0539000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:11.033 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:11.032+0000 7f0878c00640 1 -- 172.21.15.90:0/2185953198 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eeb05247d0 msgr2=0x55eeb0539000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:36:11.033 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:11.032+0000 7f0878c00640 1 --2- 172.21.15.90:0/2185953198 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eeb05247d0 0x55eeb0539000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:11.034 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:11.032+0000 7f0878c00640 1 -- 172.21.15.90:0/2185953198 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eeb0523eb0 msgr2=0x55eeb052cbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:36:11.034 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:11.032+0000 7f0878c00640 1 --2- 172.21.15.90:0/2185953198 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eeb0523eb0 0x55eeb052cbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:11.034 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:11.032+0000 7f0878c00640 1 -- 172.21.15.90:0/2185953198 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55eeb0269620 con 0x55eeb052d3b0 2024-04-02T14:36:11.034 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:11.032+0000 7f0873400640 1 --2- 172.21.15.90:0/2185953198 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eeb05247d0 0x55eeb0539000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:36:11.034 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:11.033+0000 7f0878c00640 1 --2- 172.21.15.90:0/2185953198 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55eeb052d3b0 0x55eeb0544f20 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f086c035b90 tx=0x7f086c049490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8727c7a1eca9f102 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:36:11.034 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:11.033+0000 7f085fe00640 1 -- 172.21.15.90:0/2185953198 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f086c061070 con 0x55eeb052d3b0 2024-04-02T14:36:11.035 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:11.033+0000 7f085fe00640 1 -- 172.21.15.90:0/2185953198 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f086c043040 con 0x55eeb052d3b0 2024-04-02T14:36:11.035 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:11.033+0000 7f085fe00640 1 -- 172.21.15.90:0/2185953198 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f086c039070 con 0x55eeb052d3b0 2024-04-02T14:36:11.035 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:11.034+0000 7f087d980740 1 -- 172.21.15.90:0/2185953198 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55eeb052d3b0 msgr2=0x55eeb0544f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:11.035 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:11.034+0000 7f087d980740 1 --2- 172.21.15.90:0/2185953198 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55eeb052d3b0 0x55eeb0544f20 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f086c035b90 tx=0x7f086c049490 comp rx=0 tx=0).stop 2024-04-02T14:36:11.036 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:11.035+0000 7f087d980740 1 -- 172.21.15.90:0/2185953198 shutdown_connections 2024-04-02T14:36:11.036 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:11.035+0000 7f087d980740 1 --2- 172.21.15.90:0/2185953198 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55eeb052d3b0 0x55eeb0544f20 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:11.036 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:11.035+0000 7f087d980740 1 --2- 172.21.15.90:0/2185953198 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eeb05247d0 0x55eeb0539000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:11.036 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:11.035+0000 7f087d980740 1 --2- 172.21.15.90:0/2185953198 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eeb0523eb0 0x55eeb052cbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:11.036 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:11.035+0000 7f087d980740 1 -- 172.21.15.90:0/2185953198 >> 172.21.15.90:0/2185953198 conn(0x55eeb051e5e0 msgr2=0x55eeb051d640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:36:11.036 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:11.035+0000 7f087d980740 1 -- 172.21.15.90:0/2185953198 shutdown_connections 2024-04-02T14:36:11.036 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:11.035+0000 7f087d980740 1 -- 172.21.15.90:0/2185953198 wait complete. 2024-04-02T14:36:11.037 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:11.036+0000 7f087d980740 1 -- start start 2024-04-02T14:36:11.037 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:11.036+0000 7f087d980740 1 Processor -- start 2024-04-02T14:36:11.038 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:11.037+0000 7f087d980740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eeb0523eb0 0x55eeb05312f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:11.038 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:11.037+0000 7f087d980740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eeb05247d0 0x55eeb0531830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:11.038 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:11.037+0000 7f087d980740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55eeb052d3b0 0x55eeb05a2790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:11.039 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:11.038+0000 7f087d980740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f08741bd910 con 0x55eeb05247d0 2024-04-02T14:36:11.039 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:11.038+0000 7f087d980740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f08741bd240 con 0x55eeb052d3b0 2024-04-02T14:36:11.039 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:11.038+0000 7f087d980740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f087410e480 con 0x55eeb0523eb0 2024-04-02T14:36:11.039 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:11.038+0000 7f087d980740 10 client.0 _ll_register_callbacks cb 0x7f086c0096b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:36:11.039 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:11.038+0000 7f0873e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eeb0523eb0 0x55eeb05312f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:11.039 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:11.038+0000 7f0873e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eeb0523eb0 0x55eeb05312f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:53990/0 (socket says 172.21.15.90:53990) 2024-04-02T14:36:11.039 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:11.038+0000 7f0873e00640 1 -- 172.21.15.90:0/4223762367 learned_addr learned my addr 172.21.15.90:0/4223762367 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:36:11.039 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:11.038+0000 7f0878c00640 1 --2- 172.21.15.90:0/4223762367 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55eeb052d3b0 0x55eeb05a2790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:11.040 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:11.038+0000 7f0873400640 1 --2- 172.21.15.90:0/4223762367 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eeb05247d0 0x55eeb0531830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:11.040 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:11.038+0000 7f0878c00640 1 -- 172.21.15.90:0/4223762367 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eeb0523eb0 msgr2=0x55eeb05312f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:36:11.040 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:11.038+0000 7f0878c00640 1 --2- 172.21.15.90:0/4223762367 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eeb0523eb0 0x55eeb05312f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:11.040 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:11.038+0000 7f0878c00640 1 -- 172.21.15.90:0/4223762367 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eeb05247d0 msgr2=0x55eeb0531830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:36:11.040 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:11.038+0000 7f0878c00640 1 --2- 172.21.15.90:0/4223762367 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eeb05247d0 0x55eeb0531830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:11.040 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:11.038+0000 7f0878c00640 1 -- 172.21.15.90:0/4223762367 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55eeb05a2cd0 con 0x55eeb052d3b0 2024-04-02T14:36:11.040 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:11.039+0000 7f0873400640 1 --2- 172.21.15.90:0/4223762367 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eeb05247d0 0x55eeb0531830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:36:11.040 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:11.039+0000 7f0878c00640 1 --2- 172.21.15.90:0/4223762367 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55eeb052d3b0 0x55eeb05a2790 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x55eeb0539580 tx=0x7f086c0056c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1c8fd8d7dd8b9563 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:36:11.040 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:11.039+0000 7f085d600640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:36:11.041 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:11.040+0000 7f085d600640 1 -- 172.21.15.90:0/4223762367 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f086c061070 con 0x55eeb052d3b0 2024-04-02T14:36:11.041 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:11.040+0000 7f085d600640 1 -- 172.21.15.90:0/4223762367 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f086c004230 con 0x55eeb052d3b0 2024-04-02T14:36:11.041 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:11.041+0000 7f087d980740 1 -- 172.21.15.90:0/4223762367 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f086c043040 con 0x55eeb052d3b0 2024-04-02T14:36:11.045 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:11.044+0000 7f085d600640 1 -- 172.21.15.90:0/4223762367 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f086c039070 con 0x55eeb052d3b0 2024-04-02T14:36:11.045 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:11.044+0000 7f087d980740 10 client.25844 fetch_fsmap learned FSMap version 27 2024-04-02T14:36:11.045 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:11.044+0000 7f087d980740 1 -- 172.21.15.90:0/4223762367 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55eeb05a3e60 con 0x55eeb052d3b0 2024-04-02T14:36:11.046 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:11.045+0000 7f085d600640 1 -- 172.21.15.90:0/4223762367 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f086c043040 con 0x55eeb052d3b0 2024-04-02T14:36:11.046 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:11.045+0000 7f085d600640 1 -- 172.21.15.90:0/4223762367 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f086c064030 con 0x55eeb052d3b0 2024-04-02T14:36:11.047 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:11.046+0000 7f087d980740 10 client.25844 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:36:11.047 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:11.046+0000 7f087d980740 10 client.25844 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-02T14:36:11.047 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:11.046+0000 7f087d980740 20 client.25844 populate_metadata read hostname 'smithi090' 2024-04-02T14:36:11.047 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:11.046+0000 7f087d980740 1 --2- 172.21.15.90:0/4223762367 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55eeb05a43c0 0x55eeb05a67b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:11.048 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:11.046+0000 7f087d980740 4 client.25844 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-02T14:36:11.048 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:11.046+0000 7f087d980740 1 -- 172.21.15.90:0/4223762367 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55eeb04962b0 con 0x55eeb05a43c0 2024-04-02T14:36:11.048 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:11.046+0000 7f0873e00640 1 --2- 172.21.15.90:0/4223762367 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55eeb05a43c0 0x55eeb05a67b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:11.048 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:11.047+0000 7f0873e00640 1 --2- 172.21.15.90:0/4223762367 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55eeb05a43c0 0x55eeb05a67b0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=401c5a610a759bf server_cookie=3e06e284f3e84c5e in_seq=0 out_seq=0 2024-04-02T14:36:11.048 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:11.047+0000 7f085d600640 10 client.25844 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:36:11.138 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:11.137+0000 7f085d600640 1 -- 172.21.15.90:0/4223762367 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55eeb04962b0 con 0x55eeb05a43c0 2024-04-02T14:36:11.138 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:11.137+0000 7f085d600640 10 client.25844 handle_command_reply: tid=0 2024-04-02T14:36:11.138 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:11.137+0000 7f087d980740 1 client.25844 shutdown 2024-04-02T14:36:11.139 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:11.138+0000 7f087d980740 1 -- 172.21.15.90:0/4223762367 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55eeb052d3b0 msgr2=0x55eeb05a2790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:11.139 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:11.138+0000 7f087d980740 1 --2- 172.21.15.90:0/4223762367 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55eeb052d3b0 0x55eeb05a2790 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x55eeb0539580 tx=0x7f086c0056c0 comp rx=0 tx=0).stop 2024-04-02T14:36:11.139 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:11.138+0000 7f087d980740 1 -- 172.21.15.90:0/4223762367 shutdown_connections 2024-04-02T14:36:11.139 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:11.138+0000 7f087d980740 1 --2- 172.21.15.90:0/4223762367 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55eeb05a43c0 0x55eeb05a67b0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:11.139 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:11.138+0000 7f087d980740 1 --2- 172.21.15.90:0/4223762367 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55eeb052d3b0 0x55eeb05a2790 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:11.140 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:11.138+0000 7f087d980740 1 --2- 172.21.15.90:0/4223762367 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eeb05247d0 0x55eeb0531830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:11.140 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:11.138+0000 7f087d980740 1 --2- 172.21.15.90:0/4223762367 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eeb0523eb0 0x55eeb05312f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:11.140 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:11.138+0000 7f085d600640 0 client.25844 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:36:11.140 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:11.138+0000 7f087d980740 1 -- 172.21.15.90:0/4223762367 >> 172.21.15.90:0/4223762367 conn(0x55eeb051e5e0 msgr2=0x55eeb052a420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:36:11.140 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:11.139+0000 7f087d980740 1 -- 172.21.15.90:0/4223762367 shutdown_connections 2024-04-02T14:36:11.141 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:11.139+0000 7f087d980740 1 -- 172.21.15.90:0/4223762367 wait complete. 2024-04-02T14:36:11.141 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:11.139+0000 7f087d980740 20 client.25844 trim_cache size 0 max 16384 2024-04-02T14:36:11.141 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:{ 2024-04-02T14:36:11.141 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "message": "", 2024-04-02T14:36:11.141 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "return_code": 0 2024-04-02T14:36:11.141 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:} 2024-04-02T14:36:11.142 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:11.141+0000 7f087ac00640 1 -- 172.21.15.90:0/3969149251 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f08580a3f70 msgr2=0x7f08580a6430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:11.142 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:11.141+0000 7f087ac00640 1 --2- 172.21.15.90:0/3969149251 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f08580a3f70 0x7f08580a6430 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f0864005fd0 tx=0x7f0864010040 comp rx=0 tx=0).stop 2024-04-02T14:36:11.142 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:11.141+0000 7f087ac00640 1 -- 172.21.15.90:0/3969149251 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0874100480 msgr2=0x7f08741a4ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:11.142 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:11.141+0000 7f087ac00640 1 --2- 172.21.15.90:0/3969149251 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0874100480 0x7f08741a4ae0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f086000ea20 tx=0x7f086000eee0 comp rx=0 tx=0).stop 2024-04-02T14:36:11.142 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:11.141+0000 7f087ac00640 1 -- 172.21.15.90:0/3969149251 shutdown_connections 2024-04-02T14:36:11.142 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:11.141+0000 7f087ac00640 1 --2- 172.21.15.90:0/3969149251 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f08580a3f70 0x7f08580a6430 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:11.142 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:11.141+0000 7f087ac00640 1 --2- 172.21.15.90:0/3969149251 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0874112880 0x7f087419e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:11.142 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:11.141+0000 7f087ac00640 1 --2- 172.21.15.90:0/3969149251 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0874100480 0x7f08741a4ae0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:11.142 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:11.141+0000 7f087ac00640 1 --2- 172.21.15.90:0/3969149251 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f08740ffb60 0x7f08741a45a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:11.143 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:11.141+0000 7f087ac00640 1 -- 172.21.15.90:0/3969149251 >> 172.21.15.90:0/3969149251 conn(0x7f0874076ae0 msgr2=0x7f08741111c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:36:11.143 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:11.141+0000 7f087ac00640 1 -- 172.21.15.90:0/3969149251 shutdown_connections 2024-04-02T14:36:11.143 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:11.141+0000 7f087ac00640 1 -- 172.21.15.90:0/3969149251 wait complete. 2024-04-02T14:36:11.756 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-02T14:36:11.756 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T14:36:11.756 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-02T14:36:11.831 INFO:teuthology.orchestra.run.smithi090.stderr:progress: 40.0 s, 2998.6 tps, lat 33.281 ms stddev 36.419 2024-04-02T14:36:12.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.019+0000 7f94a9600640 1 -- 172.21.15.90:0/3742685399 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f94a4106980 msgr2=0x7f94a4106d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:12.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.019+0000 7f94a9600640 1 --2- 172.21.15.90:0/3742685399 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f94a4106980 0x7f94a4106d60 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f9490009930 tx=0x7f949002bc30 comp rx=0 tx=0).stop 2024-04-02T14:36:12.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.019+0000 7f94a9600640 1 -- 172.21.15.90:0/3742685399 shutdown_connections 2024-04-02T14:36:12.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.020+0000 7f94a9600640 1 --2- 172.21.15.90:0/3742685399 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f94a4101330 0x7f94a410d840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:12.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.020+0000 7f94a9600640 1 --2- 172.21.15.90:0/3742685399 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f94a4100970 0x7f94a4100df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:12.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.020+0000 7f94a9600640 1 --2- 172.21.15.90:0/3742685399 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f94a4106980 0x7f94a4106d60 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:12.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.020+0000 7f94a9600640 1 -- 172.21.15.90:0/3742685399 >> 172.21.15.90:0/3742685399 conn(0x7f94a4077b00 msgr2=0x7f94a4077f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:36:12.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.020+0000 7f94a9600640 1 -- 172.21.15.90:0/3742685399 shutdown_connections 2024-04-02T14:36:12.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.020+0000 7f94a9600640 1 -- 172.21.15.90:0/3742685399 wait complete. 2024-04-02T14:36:12.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.021+0000 7f94a9600640 1 Processor -- start 2024-04-02T14:36:12.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.021+0000 7f94a9600640 1 -- start start 2024-04-02T14:36:12.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.022+0000 7f94a9600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f94a4100970 0x7f94a41078c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:12.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.022+0000 7f94a2a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f94a4100970 0x7f94a41078c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:12.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.022+0000 7f94a2a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f94a4100970 0x7f94a41078c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:33340/0 (socket says 172.21.15.90:33340) 2024-04-02T14:36:12.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.022+0000 7f94a9600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f94a4101330 0x7f94a4107e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:12.025 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.023+0000 7f94a2a00640 1 -- 172.21.15.90:0/3642897490 learned_addr learned my addr 172.21.15.90:0/3642897490 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:36:12.025 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.023+0000 7f94a9600640 1 --2- 172.21.15.90:0/3642897490 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f94a4106980 0x7f94a4108340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:12.025 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.023+0000 7f94a9600640 1 -- 172.21.15.90:0/3642897490 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f94a4110de0 con 0x7f94a4101330 2024-04-02T14:36:12.025 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.023+0000 7f94a9600640 1 -- 172.21.15.90:0/3642897490 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f94a4110c60 con 0x7f94a4100970 2024-04-02T14:36:12.025 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.023+0000 7f94a9600640 1 -- 172.21.15.90:0/3642897490 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f94a4110f60 con 0x7f94a4106980 2024-04-02T14:36:12.025 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.024+0000 7f94a3400640 1 --2- 172.21.15.90:0/3642897490 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f94a4106980 0x7f94a4108340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:12.025 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.024+0000 7f94a2000640 1 --2- 172.21.15.90:0/3642897490 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f94a4101330 0x7f94a4107e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:12.025 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.024+0000 7f94a2a00640 1 -- 172.21.15.90:0/3642897490 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f94a4106980 msgr2=0x7f94a4108340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:12.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.024+0000 7f94a2a00640 1 --2- 172.21.15.90:0/3642897490 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f94a4106980 0x7f94a4108340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:12.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.024+0000 7f94a2a00640 1 -- 172.21.15.90:0/3642897490 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f94a4101330 msgr2=0x7f94a4107e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:12.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.024+0000 7f94a2a00640 1 --2- 172.21.15.90:0/3642897490 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f94a4101330 0x7f94a4107e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:12.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.024+0000 7f94a2a00640 1 -- 172.21.15.90:0/3642897490 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94a41aac50 con 0x7f94a4100970 2024-04-02T14:36:12.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.024+0000 7f94a3400640 1 --2- 172.21.15.90:0/3642897490 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f94a4106980 0x7f94a4108340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:36:12.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.025+0000 7f94a2000640 1 --2- 172.21.15.90:0/3642897490 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f94a4101330 0x7f94a4107e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:36:12.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.025+0000 7f94a2a00640 1 --2- 172.21.15.90:0/3642897490 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f94a4100970 0x7f94a41078c0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f949002c170 tx=0x7f9490004360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:36:12.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.025+0000 7f949b400640 1 -- 172.21.15.90:0/3642897490 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9490043020 con 0x7f94a4100970 2024-04-02T14:36:12.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.025+0000 7f949b400640 1 -- 172.21.15.90:0/3642897490 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f94900054e0 con 0x7f94a4100970 2024-04-02T14:36:12.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.025+0000 7f949b400640 1 -- 172.21.15.90:0/3642897490 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9490003bc0 con 0x7f94a4100970 2024-04-02T14:36:12.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.026+0000 7f94a9600640 1 -- 172.21.15.90:0/3642897490 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94a41aaee0 con 0x7f94a4100970 2024-04-02T14:36:12.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.026+0000 7f94a9600640 1 -- 172.21.15.90:0/3642897490 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94a41ab350 con 0x7f94a4100970 2024-04-02T14:36:12.034 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.033+0000 7f949b400640 1 -- 172.21.15.90:0/3642897490 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 25) ==== 138930+0+0 (secure 0 0 0) 0x7f9490046020 con 0x7f94a4100970 2024-04-02T14:36:12.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.034+0000 7f94ac387740 1 Processor -- start 2024-04-02T14:36:12.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.035+0000 7f94ac387740 1 -- start start 2024-04-02T14:36:12.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.036+0000 7f94ac387740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556e34447030 0x556e34445d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:12.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.036+0000 7f94ac387740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556e34446350 0x556e344467d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:12.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.036+0000 7f94ac387740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556e3444f6c0 0x556e34451ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:12.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.036+0000 7f94ac387740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x556e343aa310 con 0x556e34447030 2024-04-02T14:36:12.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.036+0000 7f94ac387740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x556e34352690 con 0x556e3444f6c0 2024-04-02T14:36:12.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.036+0000 7f94ac387740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x556e343a62c0 con 0x556e34446350 2024-04-02T14:36:12.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.036+0000 7f94a3400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556e34447030 0x556e34445d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:12.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.037+0000 7f94a2000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556e3444f6c0 0x556e34451ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:12.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.037+0000 7f94a3400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556e34447030 0x556e34445d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:46298/0 (socket says 172.21.15.90:46298) 2024-04-02T14:36:12.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.037+0000 7f94a2a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556e34446350 0x556e344467d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:12.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.037+0000 7f94a3400640 1 -- 172.21.15.90:0/3905744953 learned_addr learned my addr 172.21.15.90:0/3905744953 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:36:12.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.037+0000 7f949b400640 1 --2- 172.21.15.90:0/3642897490 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f94880a4020 0x7f94880a64e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:12.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.037+0000 7f949b400640 1 -- 172.21.15.90:0/3642897490 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f94900f2430 con 0x7f94a4100970 2024-04-02T14:36:12.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.038+0000 7f94a2a00640 1 -- 172.21.15.90:0/3905744953 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556e3444f6c0 msgr2=0x556e34451ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:36:12.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.038+0000 7f94a2a00640 1 --2- 172.21.15.90:0/3905744953 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556e3444f6c0 0x556e34451ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:12.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.038+0000 7f94a2a00640 1 -- 172.21.15.90:0/3905744953 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556e34447030 msgr2=0x556e34445d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:36:12.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.038+0000 7f94a2a00640 1 --2- 172.21.15.90:0/3905744953 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556e34447030 0x556e34445d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:12.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.038+0000 7f94a2a00640 1 -- 172.21.15.90:0/3905744953 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556e341867e0 con 0x556e34446350 2024-04-02T14:36:12.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.038+0000 7f94a3400640 1 --2- 172.21.15.90:0/3905744953 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556e34447030 0x556e34445d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:36:12.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.039+0000 7f94a3400640 1 --2- 172.21.15.90:0/3642897490 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f94880a4020 0x7f94880a64e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:12.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.040+0000 7f94a2a00640 1 --2- 172.21.15.90:0/3905744953 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556e34446350 0x556e344467d0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f94900efa60 tx=0x7f9490004610 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5b1ce2439e2397b7 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:36:12.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.040+0000 7f948f400640 1 -- 172.21.15.90:0/3905744953 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94900bdc10 con 0x556e34446350 2024-04-02T14:36:12.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.040+0000 7f948f400640 1 -- 172.21.15.90:0/3905744953 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9490005690 con 0x556e34446350 2024-04-02T14:36:12.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.040+0000 7f948f400640 1 -- 172.21.15.90:0/3905744953 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94900f5d30 con 0x556e34446350 2024-04-02T14:36:12.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.042+0000 7f94ac387740 1 -- 172.21.15.90:0/3905744953 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556e34446350 msgr2=0x556e344467d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:12.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.042+0000 7f94a3400640 1 --2- 172.21.15.90:0/3642897490 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f94880a4020 0x7f94880a64e0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f949c010ad0 tx=0x7f949c004a40 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:36:12.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.042+0000 7f94ac387740 1 --2- 172.21.15.90:0/3905744953 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556e34446350 0x556e344467d0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f94900efa60 tx=0x7f9490004610 comp rx=0 tx=0).stop 2024-04-02T14:36:12.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.042+0000 7f94ac387740 1 -- 172.21.15.90:0/3905744953 shutdown_connections 2024-04-02T14:36:12.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.042+0000 7f94ac387740 1 --2- 172.21.15.90:0/3905744953 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556e3444f6c0 0x556e34451ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:12.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.042+0000 7f94ac387740 1 --2- 172.21.15.90:0/3905744953 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556e34446350 0x556e344467d0 secure :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f94900efa60 tx=0x7f9490004610 comp rx=0 tx=0).stop 2024-04-02T14:36:12.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.042+0000 7f94ac387740 1 --2- 172.21.15.90:0/3905744953 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556e34447030 0x556e34445d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:12.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.042+0000 7f94ac387740 1 -- 172.21.15.90:0/3905744953 >> 172.21.15.90:0/3905744953 conn(0x556e343da270 msgr2=0x556e34440da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:36:12.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.043+0000 7f94ac387740 1 -- 172.21.15.90:0/3905744953 shutdown_connections 2024-04-02T14:36:12.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.043+0000 7f94ac387740 1 -- 172.21.15.90:0/3905744953 wait complete. 2024-04-02T14:36:12.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.043+0000 7f94ac387740 1 -- start start 2024-04-02T14:36:12.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.044+0000 7f94ac387740 1 Processor -- start 2024-04-02T14:36:12.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.045+0000 7f94ac387740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556e34447030 0x556e3444c760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:12.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.045+0000 7f94ac387740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556e3444f6c0 0x556e3444cca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:12.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.045+0000 7f94ac387740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556e3444d1e0 0x556e3445b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:12.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.045+0000 7f94ac387740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x556e343aa310 con 0x556e3444f6c0 2024-04-02T14:36:12.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.045+0000 7f94ac387740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x556e34365a60 con 0x556e3444d1e0 2024-04-02T14:36:12.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.045+0000 7f94ac387740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x556e343a62c0 con 0x556e34447030 2024-04-02T14:36:12.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.045+0000 7f94ac387740 10 client.0 _ll_register_callbacks cb 0x556e3436b030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:36:12.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.045+0000 7f94a3400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556e3444d1e0 0x556e3445b4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:12.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.045+0000 7f94a2000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556e3444f6c0 0x556e3444cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:12.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.045+0000 7f94a3400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556e3444d1e0 0x556e3445b4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:33360/0 (socket says 172.21.15.90:33360) 2024-04-02T14:36:12.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.045+0000 7f94a3400640 1 -- 172.21.15.90:0/2387957300 learned_addr learned my addr 172.21.15.90:0/2387957300 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:36:12.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.045+0000 7f94a2a00640 1 --2- 172.21.15.90:0/2387957300 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556e34447030 0x556e3444c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:12.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.046+0000 7f94a3400640 1 -- 172.21.15.90:0/2387957300 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556e34447030 msgr2=0x556e3444c760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:36:12.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.046+0000 7f94a3400640 1 --2- 172.21.15.90:0/2387957300 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556e34447030 0x556e3444c760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:12.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.046+0000 7f94a3400640 1 -- 172.21.15.90:0/2387957300 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556e3444f6c0 msgr2=0x556e3444cca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:36:12.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.046+0000 7f94a3400640 1 --2- 172.21.15.90:0/2387957300 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556e3444f6c0 0x556e3444cca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:12.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.046+0000 7f94a3400640 1 -- 172.21.15.90:0/2387957300 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556e341e6f70 con 0x556e3444d1e0 2024-04-02T14:36:12.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.046+0000 7f94a3400640 1 --2- 172.21.15.90:0/2387957300 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556e3444d1e0 0x556e3445b4a0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f949c013bd0 tx=0x7f949c004ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dc6ac8e25badad4 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:36:12.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.047+0000 7f9483e00640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:36:12.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.047+0000 7f9483e00640 1 -- 172.21.15.90:0/2387957300 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f949c028420 con 0x556e3444d1e0 2024-04-02T14:36:12.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.047+0000 7f9483e00640 1 -- 172.21.15.90:0/2387957300 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f949c0285c0 con 0x556e3444d1e0 2024-04-02T14:36:12.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.047+0000 7f94ac387740 1 -- 172.21.15.90:0/2387957300 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x556e343b7850 con 0x556e3444d1e0 2024-04-02T14:36:12.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.050+0000 7f9483e00640 1 -- 172.21.15.90:0/2387957300 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f949c0287c0 con 0x556e3444d1e0 2024-04-02T14:36:12.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.050+0000 7f94ac387740 10 client.25856 fetch_fsmap learned FSMap version 27 2024-04-02T14:36:12.052 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.050+0000 7f94ac387740 1 -- 172.21.15.90:0/2387957300 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x556e3407e170 con 0x556e3444d1e0 2024-04-02T14:36:12.052 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.050+0000 7f9483e00640 1 -- 172.21.15.90:0/2387957300 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x556e343b7850 con 0x556e3444d1e0 2024-04-02T14:36:12.052 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.051+0000 7f9483e00640 1 -- 172.21.15.90:0/2387957300 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f949c032bb0 con 0x556e3444d1e0 2024-04-02T14:36:12.052 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.051+0000 7f94ac387740 10 client.25856 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:36:12.052 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.051+0000 7f94ac387740 10 client.25856 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:36:12.052 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.051+0000 7f94ac387740 20 client.25856 populate_metadata read hostname 'smithi090' 2024-04-02T14:36:12.052 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.051+0000 7f94ac387740 1 --2- 172.21.15.90:0/2387957300 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x556e3445fc50 0x556e34462040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:12.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.051+0000 7f94ac387740 4 client.25856 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:36:12.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.051+0000 7f94ac387740 1 -- 172.21.15.90:0/2387957300 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556e343b1ba0 con 0x556e3445fc50 2024-04-02T14:36:12.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.052+0000 7f94a2a00640 1 --2- 172.21.15.90:0/2387957300 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x556e3445fc50 0x556e34462040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:12.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.052+0000 7f9483e00640 10 client.25856 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:36:12.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.053+0000 7f94a2a00640 1 --2- 172.21.15.90:0/2387957300 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x556e3445fc50 0x556e34462040 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a8653d5a7babc595 server_cookie=801e91eead858967 in_seq=0 out_seq=0 2024-04-02T14:36:12.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.054+0000 7f9483e00640 1 -- 172.21.15.90:0/2387957300 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x556e343b1ba0 con 0x556e3445fc50 2024-04-02T14:36:12.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.054+0000 7f9483e00640 10 client.25856 handle_command_reply: tid=0 2024-04-02T14:36:12.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.054+0000 7f94ac387740 1 client.25856 shutdown 2024-04-02T14:36:12.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.055+0000 7f94ac387740 1 -- 172.21.15.90:0/2387957300 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556e3444d1e0 msgr2=0x556e3445b4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:12.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.055+0000 7f94ac387740 1 --2- 172.21.15.90:0/2387957300 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556e3444d1e0 0x556e3445b4a0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f949c013bd0 tx=0x7f949c004ab0 comp rx=0 tx=0).stop 2024-04-02T14:36:12.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.055+0000 7f94ac387740 1 -- 172.21.15.90:0/2387957300 shutdown_connections 2024-04-02T14:36:12.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.055+0000 7f94ac387740 1 --2- 172.21.15.90:0/2387957300 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x556e3445fc50 0x556e34462040 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:12.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.056+0000 7f94ac387740 1 --2- 172.21.15.90:0/2387957300 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556e3444d1e0 0x556e3445b4a0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:12.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.056+0000 7f94ac387740 1 --2- 172.21.15.90:0/2387957300 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556e3444f6c0 0x556e3444cca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:12.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.056+0000 7f94ac387740 1 --2- 172.21.15.90:0/2387957300 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556e34447030 0x556e3444c760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:12.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.056+0000 7f94ac387740 1 -- 172.21.15.90:0/2387957300 >> 172.21.15.90:0/2387957300 conn(0x556e343da270 msgr2=0x556e34442af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:36:12.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.056+0000 7f9483e00640 0 client.25856 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:36:12.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.056+0000 7f94ac387740 1 -- 172.21.15.90:0/2387957300 shutdown_connections 2024-04-02T14:36:12.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.056+0000 7f94ac387740 1 -- 172.21.15.90:0/2387957300 wait complete. 2024-04-02T14:36:12.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.056+0000 7f94ac387740 20 client.25856 trim_cache size 0 max 16384 2024-04-02T14:36:12.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.094+0000 7f94ac387740 1 Processor -- start 2024-04-02T14:36:12.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.094+0000 7f94ac387740 1 -- start start 2024-04-02T14:36:12.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.095+0000 7f94ac387740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556e34447030 0x556e34442900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:12.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.095+0000 7f94ac387740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556e3444d1e0 0x556e34467950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:12.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.095+0000 7f94ac387740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556e3444f6c0 0x556e34467e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:12.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.095+0000 7f94ac387740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f94a41b3e50 con 0x556e3444d1e0 2024-04-02T14:36:12.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.095+0000 7f94ac387740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f94a41b5990 con 0x556e34447030 2024-04-02T14:36:12.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.095+0000 7f94ac387740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f94a41b4520 con 0x556e3444f6c0 2024-04-02T14:36:12.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.095+0000 7f94a2000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556e3444d1e0 0x556e34467950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:12.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.095+0000 7f94a2a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556e34447030 0x556e34442900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:12.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.095+0000 7f94a2a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556e34447030 0x556e34442900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:33372/0 (socket says 172.21.15.90:33372) 2024-04-02T14:36:12.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.095+0000 7f94a2000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556e3444d1e0 0x556e34467950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:46314/0 (socket says 172.21.15.90:46314) 2024-04-02T14:36:12.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.095+0000 7f94a2a00640 1 -- 172.21.15.90:0/2268721748 learned_addr learned my addr 172.21.15.90:0/2268721748 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:36:12.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.095+0000 7f94a2a00640 1 -- 172.21.15.90:0/2268721748 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556e3444f6c0 msgr2=0x556e34467e90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-02T14:36:12.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.096+0000 7f94a3400640 1 --2- 172.21.15.90:0/2268721748 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556e3444f6c0 0x556e34467e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:12.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.096+0000 7f94a2a00640 1 --2- 172.21.15.90:0/2268721748 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556e3444f6c0 0x556e34467e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:12.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.096+0000 7f94a2a00640 1 -- 172.21.15.90:0/2268721748 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556e3444d1e0 msgr2=0x556e34467950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:36:12.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.096+0000 7f94a2a00640 1 --2- 172.21.15.90:0/2268721748 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556e3444d1e0 0x556e34467950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:12.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.096+0000 7f94a2a00640 1 -- 172.21.15.90:0/2268721748 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556e34441970 con 0x556e34447030 2024-04-02T14:36:12.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.096+0000 7f94a2000640 1 --2- 172.21.15.90:0/2268721748 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556e3444d1e0 0x556e34467950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:36:12.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.096+0000 7f94a3400640 1 --2- 172.21.15.90:0/2268721748 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556e3444f6c0 0x556e34467e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:36:12.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.096+0000 7f94a2a00640 1 --2- 172.21.15.90:0/2268721748 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556e34447030 0x556e34442900 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f94900bdfb0 tx=0x7f9490004ee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=707094d01cb32fb4 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:36:12.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.097+0000 7f948ea00640 1 -- 172.21.15.90:0/2268721748 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94900cdae0 con 0x556e34447030 2024-04-02T14:36:12.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.097+0000 7f948ea00640 1 -- 172.21.15.90:0/2268721748 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f94900be9b0 con 0x556e34447030 2024-04-02T14:36:12.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.097+0000 7f948ea00640 1 -- 172.21.15.90:0/2268721748 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94900ba920 con 0x556e34447030 2024-04-02T14:36:12.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.098+0000 7f94ac387740 1 -- 172.21.15.90:0/2268721748 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556e34447030 msgr2=0x556e34442900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:12.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.098+0000 7f94ac387740 1 --2- 172.21.15.90:0/2268721748 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556e34447030 0x556e34442900 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f94900bdfb0 tx=0x7f9490004ee0 comp rx=0 tx=0).stop 2024-04-02T14:36:12.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.098+0000 7f94ac387740 1 -- 172.21.15.90:0/2268721748 shutdown_connections 2024-04-02T14:36:12.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.098+0000 7f94ac387740 1 --2- 172.21.15.90:0/2268721748 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556e3444f6c0 0x556e34467e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:12.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.098+0000 7f94ac387740 1 --2- 172.21.15.90:0/2268721748 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556e3444d1e0 0x556e34467950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:12.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.098+0000 7f94ac387740 1 --2- 172.21.15.90:0/2268721748 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556e34447030 0x556e34442900 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:12.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.098+0000 7f94ac387740 1 -- 172.21.15.90:0/2268721748 >> 172.21.15.90:0/2268721748 conn(0x556e343da270 msgr2=0x556e344421f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:36:12.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.098+0000 7f94ac387740 1 -- 172.21.15.90:0/2268721748 shutdown_connections 2024-04-02T14:36:12.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.098+0000 7f94ac387740 1 -- 172.21.15.90:0/2268721748 wait complete. 2024-04-02T14:36:12.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.099+0000 7f94ac387740 1 -- start start 2024-04-02T14:36:12.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.099+0000 7f94ac387740 1 Processor -- start 2024-04-02T14:36:12.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.099+0000 7f94ac387740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556e34447030 0x556e34457400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:12.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.099+0000 7f94ac387740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556e3444d1e0 0x556e34457940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:12.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.099+0000 7f94ac387740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556e3444f6c0 0x556e344489d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:12.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.099+0000 7f94ac387740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f94a41b52c0 con 0x556e34447030 2024-04-02T14:36:12.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.099+0000 7f94ac387740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f94a41b4bf0 con 0x556e3444f6c0 2024-04-02T14:36:12.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.099+0000 7f94ac387740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f94a41b1e30 con 0x556e3444d1e0 2024-04-02T14:36:12.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.099+0000 7f94ac387740 10 client.0 _ll_register_callbacks cb 0x7f949c0103d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:36:12.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.100+0000 7f94a2000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556e3444d1e0 0x556e34457940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:12.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.100+0000 7f94a2000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556e3444d1e0 0x556e34457940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:54052/0 (socket says 172.21.15.90:54052) 2024-04-02T14:36:12.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.100+0000 7f94a2000640 1 -- 172.21.15.90:0/2221226407 learned_addr learned my addr 172.21.15.90:0/2221226407 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:36:12.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.100+0000 7f94a3400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556e3444f6c0 0x556e344489d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:12.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.100+0000 7f94a2a00640 1 --2- 172.21.15.90:0/2221226407 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556e34447030 0x556e34457400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:12.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.101+0000 7f94a2000640 1 -- 172.21.15.90:0/2221226407 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556e3444f6c0 msgr2=0x556e344489d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:36:12.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.101+0000 7f94a2000640 1 --2- 172.21.15.90:0/2221226407 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556e3444f6c0 0x556e344489d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:12.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.101+0000 7f94a2000640 1 -- 172.21.15.90:0/2221226407 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556e34447030 msgr2=0x556e34457400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:36:12.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.101+0000 7f94a2000640 1 --2- 172.21.15.90:0/2221226407 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556e34447030 0x556e34457400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:12.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.101+0000 7f94a2000640 1 -- 172.21.15.90:0/2221226407 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556e34449040 con 0x556e3444d1e0 2024-04-02T14:36:12.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.101+0000 7f94a2a00640 1 --2- 172.21.15.90:0/2221226407 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556e34447030 0x556e34457400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:36:12.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.102+0000 7f94a3400640 1 --2- 172.21.15.90:0/2221226407 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556e3444f6c0 0x556e344489d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:36:12.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.102+0000 7f94a2000640 1 --2- 172.21.15.90:0/2221226407 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556e3444d1e0 0x556e34457940 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f94a4109080 tx=0x7f9494009f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=81654084b74ac412 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:36:12.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.102+0000 7f9483e00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:36:12.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.102+0000 7f9483e00640 1 -- 172.21.15.90:0/2221226407 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9494011ad0 con 0x556e3444d1e0 2024-04-02T14:36:12.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.103+0000 7f9483e00640 1 -- 172.21.15.90:0/2221226407 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f9494011c70 con 0x556e3444d1e0 2024-04-02T14:36:12.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.103+0000 7f94ac387740 1 -- 172.21.15.90:0/2221226407 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f94900be9b0 con 0x556e3444d1e0 2024-04-02T14:36:12.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.103+0000 7f9483e00640 1 -- 172.21.15.90:0/2221226407 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f9494011e70 con 0x556e3444d1e0 2024-04-02T14:36:12.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.104+0000 7f94ac387740 10 client.25795 fetch_fsmap learned FSMap version 27 2024-04-02T14:36:12.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.104+0000 7f94ac387740 1 -- 172.21.15.90:0/2221226407 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x556e3444a360 con 0x556e3444d1e0 2024-04-02T14:36:12.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.104+0000 7f9483e00640 1 -- 172.21.15.90:0/2221226407 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f94900be9b0 con 0x556e3444d1e0 2024-04-02T14:36:12.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.104+0000 7f9483e00640 1 -- 172.21.15.90:0/2221226407 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f949401c070 con 0x556e3444d1e0 2024-04-02T14:36:12.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.104+0000 7f94ac387740 10 client.25795 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:36:12.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.104+0000 7f94ac387740 10 client.25795 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:36:12.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.104+0000 7f94ac387740 20 client.25795 populate_metadata read hostname 'smithi090' 2024-04-02T14:36:12.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.105+0000 7f94ac387740 1 --2- 172.21.15.90:0/2221226407 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x556e344d83c0 0x556e344da7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:12.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.105+0000 7f94ac387740 4 client.25795 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-02T14:36:12.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.105+0000 7f94ac387740 1 -- 172.21.15.90:0/2221226407 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x556e3444d030 con 0x556e344d83c0 2024-04-02T14:36:12.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.105+0000 7f94a2a00640 1 --2- 172.21.15.90:0/2221226407 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x556e344d83c0 0x556e344da7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:12.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.105+0000 7f94a2a00640 1 --2- 172.21.15.90:0/2221226407 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x556e344d83c0 0x556e344da7e0 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e89065831db3218e server_cookie=e56e1bc7aa0edba4 in_seq=0 out_seq=0 2024-04-02T14:36:12.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.105+0000 7f9483e00640 10 client.25795 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:36:12.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.106+0000 7f9483e00640 1 -- 172.21.15.90:0/2221226407 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x556e3444d030 con 0x556e344d83c0 2024-04-02T14:36:12.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.107+0000 7f9483e00640 10 client.25795 handle_command_reply: tid=0 2024-04-02T14:36:12.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.107+0000 7f94ac387740 1 client.25795 shutdown 2024-04-02T14:36:12.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.108+0000 7f94ac387740 1 -- 172.21.15.90:0/2221226407 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556e3444d1e0 msgr2=0x556e34457940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:12.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.108+0000 7f94ac387740 1 --2- 172.21.15.90:0/2221226407 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556e3444d1e0 0x556e34457940 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f94a4109080 tx=0x7f9494009f90 comp rx=0 tx=0).stop 2024-04-02T14:36:12.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.108+0000 7f94ac387740 1 -- 172.21.15.90:0/2221226407 shutdown_connections 2024-04-02T14:36:12.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.108+0000 7f94ac387740 1 --2- 172.21.15.90:0/2221226407 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x556e344d83c0 0x556e344da7e0 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:12.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.108+0000 7f9483e00640 0 client.25795 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:36:12.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.108+0000 7f94ac387740 1 --2- 172.21.15.90:0/2221226407 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556e3444f6c0 0x556e344489d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:12.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.108+0000 7f94ac387740 1 --2- 172.21.15.90:0/2221226407 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556e3444d1e0 0x556e34457940 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:12.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.108+0000 7f94ac387740 1 --2- 172.21.15.90:0/2221226407 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556e34447030 0x556e34457400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:12.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.108+0000 7f94ac387740 1 -- 172.21.15.90:0/2221226407 >> 172.21.15.90:0/2221226407 conn(0x556e343da270 msgr2=0x556e344429f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:36:12.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.109+0000 7f94ac387740 1 -- 172.21.15.90:0/2221226407 shutdown_connections 2024-04-02T14:36:12.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.109+0000 7f94ac387740 1 -- 172.21.15.90:0/2221226407 wait complete. 2024-04-02T14:36:12.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.109+0000 7f94ac387740 20 client.25795 trim_cache size 0 max 16384 2024-04-02T14:36:12.111 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-02T14:36:12.111 INFO:teuthology.orchestra.run.smithi090.stdout: "return_code": 0, 2024-04-02T14:36:12.111 INFO:teuthology.orchestra.run.smithi090.stdout: "scrub_tag": "22207f7f-a4b8-46d0-ba1f-01416d715cad", 2024-04-02T14:36:12.111 INFO:teuthology.orchestra.run.smithi090.stdout: "mode": "asynchronous" 2024-04-02T14:36:12.111 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-02T14:36:12.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.110+0000 7f94a9600640 1 -- 172.21.15.90:0/3642897490 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f94880a4020 msgr2=0x7f94880a64e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:12.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.110+0000 7f94a9600640 1 --2- 172.21.15.90:0/3642897490 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f94880a4020 0x7f94880a64e0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f949c010ad0 tx=0x7f949c004a40 comp rx=0 tx=0).stop 2024-04-02T14:36:12.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.110+0000 7f94a9600640 1 -- 172.21.15.90:0/3642897490 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f94a4100970 msgr2=0x7f94a41078c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:12.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.110+0000 7f94a9600640 1 --2- 172.21.15.90:0/3642897490 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f94a4100970 0x7f94a41078c0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f949002c170 tx=0x7f9490004360 comp rx=0 tx=0).stop 2024-04-02T14:36:12.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.110+0000 7f94a9600640 1 -- 172.21.15.90:0/3642897490 shutdown_connections 2024-04-02T14:36:12.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.110+0000 7f94a9600640 1 --2- 172.21.15.90:0/3642897490 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f94880a4020 0x7f94880a64e0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:12.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.110+0000 7f94a9600640 1 --2- 172.21.15.90:0/3642897490 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f94a4106980 0x7f94a4108340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:12.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.110+0000 7f94a9600640 1 --2- 172.21.15.90:0/3642897490 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f94a4101330 0x7f94a4107e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:12.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.110+0000 7f94a9600640 1 --2- 172.21.15.90:0/3642897490 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f94a4100970 0x7f94a41078c0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:12.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.110+0000 7f94a9600640 1 -- 172.21.15.90:0/3642897490 >> 172.21.15.90:0/3642897490 conn(0x7f94a4077b00 msgr2=0x7f94a4102f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:36:12.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.111+0000 7f94a9600640 1 -- 172.21.15.90:0/3642897490 shutdown_connections 2024-04-02T14:36:12.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.111+0000 7f94a9600640 1 -- 172.21.15.90:0/3642897490 wait complete. 2024-04-02T14:36:12.135 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:36:12 smithi090 ceph-mon[31674]: pgmap v42: 97 pgs: 97 active+clean; 1010 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 48 MiB/s wr, 1.24k op/s 2024-04-02T14:36:12.139 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-02T14:36:12.139 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-02T14:36:12.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:36:12 smithi092 ceph-mon[36497]: pgmap v42: 97 pgs: 97 active+clean; 1010 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 48 MiB/s wr, 1.24k op/s 2024-04-02T14:36:12.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.420+0000 7fa7f7600640 1 -- 172.21.15.90:0/2553781474 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa7f0114c50 msgr2=0x7fa7f0117040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:12.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.421+0000 7fa7f7600640 1 --2- 172.21.15.90:0/2553781474 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa7f0114c50 0x7fa7f0117040 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fa7e8009be0 tx=0x7fa7e802bef0 comp rx=0 tx=0).stop 2024-04-02T14:36:12.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.421+0000 7fa7f7600640 1 -- 172.21.15.90:0/2553781474 shutdown_connections 2024-04-02T14:36:12.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.421+0000 7fa7f7600640 1 --2- 172.21.15.90:0/2553781474 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa7f0114c50 0x7fa7f0117040 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:12.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.421+0000 7fa7f7600640 1 --2- 172.21.15.90:0/2553781474 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa7f0109600 0x7fa7f0114710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:12.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.421+0000 7fa7f7600640 1 --2- 172.21.15.90:0/2553781474 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa7f0108ce0 0x7fa7f01090c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:12.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.421+0000 7fa7f7600640 1 -- 172.21.15.90:0/2553781474 >> 172.21.15.90:0/2553781474 conn(0x7fa7f006cbe0 msgr2=0x7fa7f006cff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:36:12.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.421+0000 7fa7f7600640 1 -- 172.21.15.90:0/2553781474 shutdown_connections 2024-04-02T14:36:12.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.422+0000 7fa7f7600640 1 -- 172.21.15.90:0/2553781474 wait complete. 2024-04-02T14:36:12.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.423+0000 7fa7f7600640 1 Processor -- start 2024-04-02T14:36:12.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.424+0000 7fa7f7600640 1 -- start start 2024-04-02T14:36:12.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.425+0000 7fa7f7600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa7f0108ce0 0x7fa7f01a6810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:12.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.425+0000 7fa7f7600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa7f0109600 0x7fa7f01a6d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:12.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.425+0000 7fa7f7600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa7f0114c50 0x7fa7f01a09e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:12.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.425+0000 7fa7f7600640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fa7f011a660 con 0x7fa7f0109600 2024-04-02T14:36:12.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.425+0000 7fa7f7600640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fa7f011a4e0 con 0x7fa7f0114c50 2024-04-02T14:36:12.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.425+0000 7fa7f7600640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fa7f011a7e0 con 0x7fa7f0108ce0 2024-04-02T14:36:12.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.425+0000 7fa7f5600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa7f0114c50 0x7fa7f01a09e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:12.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.425+0000 7fa7f5600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa7f0114c50 0x7fa7f01a09e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:33400/0 (socket says 172.21.15.90:33400) 2024-04-02T14:36:12.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.425+0000 7fa7f5600640 1 -- 172.21.15.90:0/356092153 learned_addr learned my addr 172.21.15.90:0/356092153 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:36:12.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.425+0000 7fa7f4c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa7f0108ce0 0x7fa7f01a6810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:12.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.425+0000 7fa7efe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa7f0109600 0x7fa7f01a6d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:12.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.426+0000 7fa7f5600640 1 -- 172.21.15.90:0/356092153 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa7f0108ce0 msgr2=0x7fa7f01a6810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:12.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.426+0000 7fa7f5600640 1 --2- 172.21.15.90:0/356092153 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa7f0108ce0 0x7fa7f01a6810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:12.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.426+0000 7fa7f5600640 1 -- 172.21.15.90:0/356092153 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa7f0109600 msgr2=0x7fa7f01a6d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:12.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.426+0000 7fa7f5600640 1 --2- 172.21.15.90:0/356092153 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa7f0109600 0x7fa7f01a6d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:12.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.426+0000 7fa7f5600640 1 -- 172.21.15.90:0/356092153 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7f01a0f20 con 0x7fa7f0114c50 2024-04-02T14:36:12.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.426+0000 7fa7f4c00640 1 --2- 172.21.15.90:0/356092153 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa7f0108ce0 0x7fa7f01a6810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:36:12.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.426+0000 7fa7f5600640 1 --2- 172.21.15.90:0/356092153 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa7f0114c50 0x7fa7f01a09e0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fa7e8004af0 tx=0x7fa7e802cbc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:36:12.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.427+0000 7fa7ed600640 1 -- 172.21.15.90:0/356092153 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7e8043020 con 0x7fa7f0114c50 2024-04-02T14:36:12.429 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.427+0000 7fa7ed600640 1 -- 172.21.15.90:0/356092153 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa7e8003e50 con 0x7fa7f0114c50 2024-04-02T14:36:12.429 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.427+0000 7fa7ed600640 1 -- 172.21.15.90:0/356092153 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7e803e6b0 con 0x7fa7f0114c50 2024-04-02T14:36:12.429 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.427+0000 7fa7f7600640 1 -- 172.21.15.90:0/356092153 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7f01a11b0 con 0x7fa7f0114c50 2024-04-02T14:36:12.429 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.427+0000 7fa7f7600640 1 -- 172.21.15.90:0/356092153 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7f01a16b0 con 0x7fa7f0114c50 2024-04-02T14:36:12.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.432+0000 7fa7ed600640 1 -- 172.21.15.90:0/356092153 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 25) ==== 138930+0+0 (secure 0 0 0) 0x7fa7e8046020 con 0x7fa7f0114c50 2024-04-02T14:36:12.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.434+0000 7fa7fa37a740 1 Processor -- start 2024-04-02T14:36:12.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.435+0000 7fa7fa37a740 1 -- start start 2024-04-02T14:36:12.436 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:36:12 smithi177 ceph-mon[36431]: pgmap v42: 97 pgs: 97 active+clean; 1010 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 48 MiB/s wr, 1.24k op/s 2024-04-02T14:36:12.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.436+0000 7fa7ed600640 1 --2- 172.21.15.90:0/356092153 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fa7d80a4020 0x7fa7d80a64e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:12.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.436+0000 7fa7ed600640 1 -- 172.21.15.90:0/356092153 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7fa7e80f3000 con 0x7fa7f0114c50 2024-04-02T14:36:12.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.437+0000 7fa7fa37a740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55af24c9f7b0 0x55af24c9fb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:12.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.437+0000 7fa7fa37a740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55af24ca00d0 0x55af24ca8770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:12.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.437+0000 7fa7fa37a740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55af24ca8cb0 0x55af24cab0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:12.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.437+0000 7fa7fa37a740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55af24c03770 con 0x55af24ca8cb0 2024-04-02T14:36:12.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.437+0000 7fa7fa37a740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55af24bab700 con 0x55af24ca00d0 2024-04-02T14:36:12.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.437+0000 7fa7fa37a740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55af24bff720 con 0x55af24c9f7b0 2024-04-02T14:36:12.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.437+0000 7fa7efe00640 1 --2- 172.21.15.90:0/356092153 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fa7d80a4020 0x7fa7d80a64e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:12.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.437+0000 7fa7f4c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55af24c9f7b0 0x55af24c9fb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:12.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.437+0000 7fa7f4c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55af24c9f7b0 0x55af24c9fb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:54082/0 (socket says 172.21.15.90:54082) 2024-04-02T14:36:12.440 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.437+0000 7fa7f4c00640 1 -- 172.21.15.90:0/2353903978 learned_addr learned my addr 172.21.15.90:0/2353903978 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:36:12.440 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.438+0000 7fa7f5600640 1 --2- 172.21.15.90:0/2353903978 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55af24ca8cb0 0x55af24cab0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:12.440 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.438+0000 7fa7efe00640 1 --2- 172.21.15.90:0/2353903978 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55af24ca00d0 0x55af24ca8770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:12.440 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.438+0000 7fa7f4c00640 1 -- 172.21.15.90:0/2353903978 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55af24ca00d0 msgr2=0x55af24ca8770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:36:12.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.438+0000 7fa7f4c00640 1 --2- 172.21.15.90:0/2353903978 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55af24ca00d0 0x55af24ca8770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:12.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.438+0000 7fa7f4c00640 1 -- 172.21.15.90:0/2353903978 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55af24ca8cb0 msgr2=0x55af24cab0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:36:12.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.438+0000 7fa7f4c00640 1 --2- 172.21.15.90:0/2353903978 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55af24ca8cb0 0x55af24cab0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:12.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.438+0000 7fa7f4c00640 1 -- 172.21.15.90:0/2353903978 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55af249e17d0 con 0x55af24c9f7b0 2024-04-02T14:36:12.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.439+0000 7fa7f4c00640 1 --2- 172.21.15.90:0/2353903978 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55af24c9f7b0 0x55af24c9fb90 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fa7e000bd40 tx=0x7fa7e000f730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9953ebc7c3ebe8a2 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:36:12.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.439+0000 7fa7efe00640 1 --2- 172.21.15.90:0/356092153 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fa7d80a4020 0x7fa7d80a64e0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fa7f00731e0 tx=0x7fa7e4011860 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:36:12.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.439+0000 7fa7ddc00640 1 -- 172.21.15.90:0/2353903978 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7e0015070 con 0x55af24c9f7b0 2024-04-02T14:36:12.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.439+0000 7fa7ddc00640 1 -- 172.21.15.90:0/2353903978 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa7e000be30 con 0x55af24c9f7b0 2024-04-02T14:36:12.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.439+0000 7fa7ddc00640 1 -- 172.21.15.90:0/2353903978 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7e0015070 con 0x55af24c9f7b0 2024-04-02T14:36:12.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.440+0000 7fa7fa37a740 1 -- 172.21.15.90:0/2353903978 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55af24c9f7b0 msgr2=0x55af24c9fb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:12.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.440+0000 7fa7fa37a740 1 --2- 172.21.15.90:0/2353903978 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55af24c9f7b0 0x55af24c9fb90 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fa7e000bd40 tx=0x7fa7e000f730 comp rx=0 tx=0).stop 2024-04-02T14:36:12.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.440+0000 7fa7fa37a740 1 -- 172.21.15.90:0/2353903978 shutdown_connections 2024-04-02T14:36:12.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.440+0000 7fa7fa37a740 1 --2- 172.21.15.90:0/2353903978 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55af24ca8cb0 0x55af24cab0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:12.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.440+0000 7fa7fa37a740 1 --2- 172.21.15.90:0/2353903978 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55af24ca00d0 0x55af24ca8770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:12.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.440+0000 7fa7fa37a740 1 --2- 172.21.15.90:0/2353903978 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55af24c9f7b0 0x55af24c9fb90 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:12.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.440+0000 7fa7fa37a740 1 -- 172.21.15.90:0/2353903978 >> 172.21.15.90:0/2353903978 conn(0x55af24c99ee0 msgr2=0x55af24c9a2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:36:12.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.441+0000 7fa7fa37a740 1 -- 172.21.15.90:0/2353903978 shutdown_connections 2024-04-02T14:36:12.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.441+0000 7fa7fa37a740 1 -- 172.21.15.90:0/2353903978 wait complete. 2024-04-02T14:36:12.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.442+0000 7fa7fa37a740 1 -- start start 2024-04-02T14:36:12.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.443+0000 7fa7fa37a740 1 Processor -- start 2024-04-02T14:36:12.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.443+0000 7fa7fa37a740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55af24c9f7b0 0x55af24ca5c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:12.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.443+0000 7fa7fa37a740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55af24ca00d0 0x55af24ca61d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:12.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.443+0000 7fa7fa37a740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55af24ca8cb0 0x55af24ca6710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:12.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.443+0000 7fa7fa37a740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55af24cabaa0 con 0x55af24c9f7b0 2024-04-02T14:36:12.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.443+0000 7fa7fa37a740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55af24bbeac0 con 0x55af24ca8cb0 2024-04-02T14:36:12.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.443+0000 7fa7fa37a740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55af24cabc20 con 0x55af24ca00d0 2024-04-02T14:36:12.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.443+0000 7fa7fa37a740 10 client.0 _ll_register_callbacks cb 0x55af24bc1180 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:36:12.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.444+0000 7fa7efe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55af24ca00d0 0x55af24ca61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:12.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.444+0000 7fa7efe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55af24ca00d0 0x55af24ca61d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:54090/0 (socket says 172.21.15.90:54090) 2024-04-02T14:36:12.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.444+0000 7fa7efe00640 1 -- 172.21.15.90:0/813425091 learned_addr learned my addr 172.21.15.90:0/813425091 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:36:12.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.444+0000 7fa7f5600640 1 --2- 172.21.15.90:0/813425091 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55af24ca8cb0 0x55af24ca6710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:12.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.444+0000 7fa7f4c00640 1 --2- 172.21.15.90:0/813425091 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55af24c9f7b0 0x55af24ca5c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:12.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.445+0000 7fa7f5600640 1 -- 172.21.15.90:0/813425091 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55af24ca00d0 msgr2=0x55af24ca61d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:36:12.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.445+0000 7fa7f5600640 1 --2- 172.21.15.90:0/813425091 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55af24ca00d0 0x55af24ca61d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:12.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.445+0000 7fa7f5600640 1 -- 172.21.15.90:0/813425091 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55af24c9f7b0 msgr2=0x55af24ca5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:36:12.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.445+0000 7fa7f5600640 1 --2- 172.21.15.90:0/813425091 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55af24c9f7b0 0x55af24ca5c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:12.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.445+0000 7fa7f5600640 1 -- 172.21.15.90:0/813425091 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55af249173f0 con 0x55af24ca8cb0 2024-04-02T14:36:12.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.445+0000 7fa7efe00640 1 --2- 172.21.15.90:0/813425091 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55af24ca00d0 0x55af24ca61d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:36:12.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.445+0000 7fa7f5600640 1 --2- 172.21.15.90:0/813425091 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55af24ca8cb0 0x55af24ca6710 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fa7e80f0630 tx=0x7fa7e80bea80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b2aab5358015866c server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:36:12.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.445+0000 7fa7d2000640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:36:12.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.446+0000 7fa7d2000640 1 -- 172.21.15.90:0/813425091 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7e80dd060 con 0x55af24ca8cb0 2024-04-02T14:36:12.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.446+0000 7fa7d2000640 1 -- 172.21.15.90:0/813425091 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7fa7e80041b0 con 0x55af24ca8cb0 2024-04-02T14:36:12.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.446+0000 7fa7fa37a740 1 -- 172.21.15.90:0/813425091 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa7e000be30 con 0x55af24ca8cb0 2024-04-02T14:36:12.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.446+0000 7fa7d2000640 1 -- 172.21.15.90:0/813425091 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fa7e80cc050 con 0x55af24ca8cb0 2024-04-02T14:36:12.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.446+0000 7fa7fa37a740 10 client.25874 fetch_fsmap learned FSMap version 27 2024-04-02T14:36:12.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.446+0000 7fa7fa37a740 1 -- 172.21.15.90:0/813425091 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55af248d70f0 con 0x55af24ca8cb0 2024-04-02T14:36:12.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.447+0000 7fa7d2000640 1 -- 172.21.15.90:0/813425091 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa7e000be30 con 0x55af24ca8cb0 2024-04-02T14:36:12.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.447+0000 7fa7d2000640 1 -- 172.21.15.90:0/813425091 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fa7e8050060 con 0x55af24ca8cb0 2024-04-02T14:36:12.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.447+0000 7fa7fa37a740 10 client.25874 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:36:12.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.447+0000 7fa7fa37a740 10 client.25874 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:36:12.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.447+0000 7fa7fa37a740 20 client.25874 populate_metadata read hostname 'smithi090' 2024-04-02T14:36:12.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.447+0000 7fa7fa37a740 1 --2- 172.21.15.90:0/813425091 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55af24cc9b30 0x55af24ccbf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:12.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.447+0000 7fa7fa37a740 4 client.25874 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:36:12.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.447+0000 7fa7fa37a740 1 -- 172.21.15.90:0/813425091 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55af24ba4630 con 0x55af24cc9b30 2024-04-02T14:36:12.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.447+0000 7fa7f4c00640 1 --2- 172.21.15.90:0/813425091 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55af24cc9b30 0x55af24ccbf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:12.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.448+0000 7fa7d2000640 10 client.25874 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:36:12.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.448+0000 7fa7f4c00640 1 --2- 172.21.15.90:0/813425091 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55af24cc9b30 0x55af24ccbf70 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=aab3138cfac78a0b server_cookie=a1ea1ad7996abc11 in_seq=0 out_seq=0 2024-04-02T14:36:12.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.449+0000 7fa7d2000640 1 -- 172.21.15.90:0/813425091 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55af24ba4630 con 0x55af24cc9b30 2024-04-02T14:36:12.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.450+0000 7fa7d2000640 10 client.25874 handle_command_reply: tid=0 2024-04-02T14:36:12.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.450+0000 7fa7fa37a740 1 client.25874 shutdown 2024-04-02T14:36:12.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.450+0000 7fa7fa37a740 1 -- 172.21.15.90:0/813425091 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55af24ca8cb0 msgr2=0x55af24ca6710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:12.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.450+0000 7fa7fa37a740 1 --2- 172.21.15.90:0/813425091 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55af24ca8cb0 0x55af24ca6710 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fa7e80f0630 tx=0x7fa7e80bea80 comp rx=0 tx=0).stop 2024-04-02T14:36:12.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.451+0000 7fa7fa37a740 1 -- 172.21.15.90:0/813425091 shutdown_connections 2024-04-02T14:36:12.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.451+0000 7fa7fa37a740 1 --2- 172.21.15.90:0/813425091 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55af24cc9b30 0x55af24ccbf70 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:12.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.451+0000 7fa7fa37a740 1 --2- 172.21.15.90:0/813425091 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55af24ca8cb0 0x55af24ca6710 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:12.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.451+0000 7fa7fa37a740 1 --2- 172.21.15.90:0/813425091 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55af24ca00d0 0x55af24ca61d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:12.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.451+0000 7fa7d2000640 0 client.25874 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:36:12.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.451+0000 7fa7fa37a740 1 --2- 172.21.15.90:0/813425091 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55af24c9f7b0 0x55af24ca5c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:12.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.451+0000 7fa7fa37a740 1 -- 172.21.15.90:0/813425091 >> 172.21.15.90:0/813425091 conn(0x55af24c99ee0 msgr2=0x55af24c9bc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:36:12.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.451+0000 7fa7fa37a740 1 -- 172.21.15.90:0/813425091 shutdown_connections 2024-04-02T14:36:12.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.451+0000 7fa7fa37a740 1 -- 172.21.15.90:0/813425091 wait complete. 2024-04-02T14:36:12.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.451+0000 7fa7fa37a740 20 client.25874 trim_cache size 0 max 16384 2024-04-02T14:36:12.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.484+0000 7fa7fa37a740 1 Processor -- start 2024-04-02T14:36:12.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.484+0000 7fa7fa37a740 1 -- start start 2024-04-02T14:36:12.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.485+0000 7fa7fa37a740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55af24c9f7b0 0x55af24ca83c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:12.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.485+0000 7fa7fa37a740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55af24ca00d0 0x55af24c98f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:12.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.485+0000 7fa7fa37a740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55af24ca8cb0 0x55af24ca6530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:12.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.485+0000 7fa7fa37a740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fa7f01b3fc0 con 0x55af24ca00d0 2024-04-02T14:36:12.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.485+0000 7fa7fa37a740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fa7f01b5b00 con 0x55af24ca8cb0 2024-04-02T14:36:12.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.485+0000 7fa7fa37a740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fa7f01b4690 con 0x55af24c9f7b0 2024-04-02T14:36:12.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.485+0000 7fa7f4c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55af24c9f7b0 0x55af24ca83c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:12.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.485+0000 7fa7efe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55af24ca00d0 0x55af24c98f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:12.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.486+0000 7fa7f4c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55af24c9f7b0 0x55af24ca83c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:54094/0 (socket says 172.21.15.90:54094) 2024-04-02T14:36:12.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.486+0000 7fa7efe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55af24ca00d0 0x55af24c98f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:46356/0 (socket says 172.21.15.90:46356) 2024-04-02T14:36:12.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.486+0000 7fa7f4c00640 1 -- 172.21.15.90:0/929747234 learned_addr learned my addr 172.21.15.90:0/929747234 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:36:12.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.486+0000 7fa7f5600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55af24ca8cb0 0x55af24ca6530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:12.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.487+0000 7fa7f4c00640 1 -- 172.21.15.90:0/929747234 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55af24ca8cb0 msgr2=0x55af24ca6530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:36:12.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.487+0000 7fa7f4c00640 1 --2- 172.21.15.90:0/929747234 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55af24ca8cb0 0x55af24ca6530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:12.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.487+0000 7fa7f4c00640 1 -- 172.21.15.90:0/929747234 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55af24ca00d0 msgr2=0x55af24c98f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:36:12.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.487+0000 7fa7f4c00640 1 --2- 172.21.15.90:0/929747234 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55af24ca00d0 0x55af24c98f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:12.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.487+0000 7fa7f4c00640 1 -- 172.21.15.90:0/929747234 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55af24ca0a80 con 0x55af24c9f7b0 2024-04-02T14:36:12.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.488+0000 7fa7f4c00640 1 --2- 172.21.15.90:0/929747234 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55af24c9f7b0 0x55af24ca83c0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fa7e000bd40 tx=0x7fa7e0011f20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=515713b612432ea4 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:36:12.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.488+0000 7fa7dd200640 1 -- 172.21.15.90:0/929747234 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7e0015040 con 0x55af24c9f7b0 2024-04-02T14:36:12.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.488+0000 7fa7dd200640 1 -- 172.21.15.90:0/929747234 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55af24ba4630 con 0x55af24c9f7b0 2024-04-02T14:36:12.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.488+0000 7fa7dd200640 1 -- 172.21.15.90:0/929747234 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7e0011440 con 0x55af24c9f7b0 2024-04-02T14:36:12.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.489+0000 7fa7fa37a740 1 -- 172.21.15.90:0/929747234 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55af24c9f7b0 msgr2=0x55af24ca83c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:12.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.489+0000 7fa7fa37a740 1 --2- 172.21.15.90:0/929747234 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55af24c9f7b0 0x55af24ca83c0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fa7e000bd40 tx=0x7fa7e0011f20 comp rx=0 tx=0).stop 2024-04-02T14:36:12.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.489+0000 7fa7fa37a740 1 -- 172.21.15.90:0/929747234 shutdown_connections 2024-04-02T14:36:12.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.489+0000 7fa7fa37a740 1 --2- 172.21.15.90:0/929747234 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55af24ca8cb0 0x55af24ca6530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:12.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.489+0000 7fa7fa37a740 1 --2- 172.21.15.90:0/929747234 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55af24ca00d0 0x55af24c98f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:12.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.489+0000 7fa7fa37a740 1 --2- 172.21.15.90:0/929747234 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55af24c9f7b0 0x55af24ca83c0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:12.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.489+0000 7fa7fa37a740 1 -- 172.21.15.90:0/929747234 >> 172.21.15.90:0/929747234 conn(0x55af24c99ee0 msgr2=0x55af24c9b110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:36:12.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.490+0000 7fa7fa37a740 1 -- 172.21.15.90:0/929747234 shutdown_connections 2024-04-02T14:36:12.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.490+0000 7fa7fa37a740 1 -- 172.21.15.90:0/929747234 wait complete. 2024-04-02T14:36:12.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.490+0000 7fa7fa37a740 1 -- start start 2024-04-02T14:36:12.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.491+0000 7fa7fa37a740 1 Processor -- start 2024-04-02T14:36:12.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.493+0000 7fa7fa37a740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55af24c9f7b0 0x55af24ca3000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:12.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.493+0000 7fa7fa37a740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55af24ca00d0 0x55af24ca3540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:12.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.493+0000 7fa7fa37a740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55af24ca8cb0 0x55af24d2f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:12.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.493+0000 7fa7fa37a740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fa7f01b5430 con 0x55af24c9f7b0 2024-04-02T14:36:12.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.493+0000 7fa7fa37a740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fa7f01b4d60 con 0x55af24ca8cb0 2024-04-02T14:36:12.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.493+0000 7fa7fa37a740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fa7f01b1f60 con 0x55af24ca00d0 2024-04-02T14:36:12.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.493+0000 7fa7fa37a740 10 client.0 _ll_register_callbacks cb 0x7fa7e80098a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:36:12.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.494+0000 7fa7efe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55af24ca00d0 0x55af24ca3540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:12.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.494+0000 7fa7efe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55af24ca00d0 0x55af24ca3540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:54098/0 (socket says 172.21.15.90:54098) 2024-04-02T14:36:12.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.494+0000 7fa7efe00640 1 -- 172.21.15.90:0/3759079352 learned_addr learned my addr 172.21.15.90:0/3759079352 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:36:12.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.494+0000 7fa7f5600640 1 --2- 172.21.15.90:0/3759079352 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55af24ca8cb0 0x55af24d2f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:12.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.495+0000 7fa7f4c00640 1 --2- 172.21.15.90:0/3759079352 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55af24c9f7b0 0x55af24ca3000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:12.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.495+0000 7fa7efe00640 1 -- 172.21.15.90:0/3759079352 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55af24ca8cb0 msgr2=0x55af24d2f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:36:12.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.495+0000 7fa7efe00640 1 --2- 172.21.15.90:0/3759079352 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55af24ca8cb0 0x55af24d2f0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:12.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.495+0000 7fa7efe00640 1 -- 172.21.15.90:0/3759079352 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55af24c9f7b0 msgr2=0x55af24ca3000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:36:12.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.495+0000 7fa7efe00640 1 --2- 172.21.15.90:0/3759079352 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55af24c9f7b0 0x55af24ca3000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:12.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.495+0000 7fa7efe00640 1 -- 172.21.15.90:0/3759079352 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55af24d2f630 con 0x55af24ca00d0 2024-04-02T14:36:12.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.495+0000 7fa7f4c00640 1 --2- 172.21.15.90:0/3759079352 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55af24c9f7b0 0x55af24ca3000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:36:12.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.495+0000 7fa7efe00640 1 --2- 172.21.15.90:0/3759079352 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55af24ca00d0 0x55af24ca3540 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fa7e4003b10 tx=0x7fa7e4004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1736c0d5ba819953 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:36:12.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.495+0000 7fa7d2000640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:36:12.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.496+0000 7fa7d2000640 1 -- 172.21.15.90:0/3759079352 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7e401a040 con 0x55af24ca00d0 2024-04-02T14:36:12.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.496+0000 7fa7d2000640 1 -- 172.21.15.90:0/3759079352 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7fa7e400f960 con 0x55af24ca00d0 2024-04-02T14:36:12.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.496+0000 7fa7fa37a740 1 -- 172.21.15.90:0/3759079352 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55af24ba4630 con 0x55af24ca00d0 2024-04-02T14:36:12.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.497+0000 7fa7d2000640 1 -- 172.21.15.90:0/3759079352 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fa7e40044c0 con 0x55af24ca00d0 2024-04-02T14:36:12.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.497+0000 7fa7fa37a740 10 client.25813 fetch_fsmap learned FSMap version 27 2024-04-02T14:36:12.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.497+0000 7fa7fa37a740 1 -- 172.21.15.90:0/3759079352 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55af24d308b0 con 0x55af24ca00d0 2024-04-02T14:36:12.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.498+0000 7fa7d2000640 1 -- 172.21.15.90:0/3759079352 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55af24ba4630 con 0x55af24ca00d0 2024-04-02T14:36:12.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.499+0000 7fa7d2000640 1 -- 172.21.15.90:0/3759079352 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fa7e4020420 con 0x55af24ca00d0 2024-04-02T14:36:12.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.499+0000 7fa7fa37a740 10 client.25813 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:36:12.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.499+0000 7fa7fa37a740 10 client.25813 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:36:12.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.499+0000 7fa7fa37a740 20 client.25813 populate_metadata read hostname 'smithi090' 2024-04-02T14:36:12.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.499+0000 7fa7fa37a740 1 --2- 172.21.15.90:0/3759079352 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55af24d30fd0 0x55af24d333f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:12.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.499+0000 7fa7fa37a740 4 client.25813 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-02T14:36:12.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.499+0000 7fa7fa37a740 1 -- 172.21.15.90:0/3759079352 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55af24c10cb0 con 0x55af24d30fd0 2024-04-02T14:36:12.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.500+0000 7fa7f4c00640 1 --2- 172.21.15.90:0/3759079352 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55af24d30fd0 0x55af24d333f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:12.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.501+0000 7fa7f4c00640 1 --2- 172.21.15.90:0/3759079352 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55af24d30fd0 0x55af24d333f0 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4a0d5e153f0343a0 server_cookie=1a5b4dfad74c583e in_seq=0 out_seq=0 2024-04-02T14:36:12.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.501+0000 7fa7d2000640 10 client.25813 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:36:12.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.501+0000 7fa7d2000640 1 -- 172.21.15.90:0/3759079352 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55af24c10cb0 con 0x55af24d30fd0 2024-04-02T14:36:12.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.501+0000 7fa7d2000640 10 client.25813 handle_command_reply: tid=0 2024-04-02T14:36:12.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.502+0000 7fa7fa37a740 1 client.25813 shutdown 2024-04-02T14:36:12.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.502+0000 7fa7fa37a740 1 -- 172.21.15.90:0/3759079352 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55af24ca00d0 msgr2=0x55af24ca3540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:12.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.502+0000 7fa7fa37a740 1 --2- 172.21.15.90:0/3759079352 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55af24ca00d0 0x55af24ca3540 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fa7e4003b10 tx=0x7fa7e4004730 comp rx=0 tx=0).stop 2024-04-02T14:36:12.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.502+0000 7fa7fa37a740 1 -- 172.21.15.90:0/3759079352 shutdown_connections 2024-04-02T14:36:12.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.502+0000 7fa7fa37a740 1 --2- 172.21.15.90:0/3759079352 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55af24d30fd0 0x55af24d333f0 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:12.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.502+0000 7fa7fa37a740 1 --2- 172.21.15.90:0/3759079352 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55af24ca8cb0 0x55af24d2f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:12.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.502+0000 7fa7fa37a740 1 --2- 172.21.15.90:0/3759079352 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55af24ca00d0 0x55af24ca3540 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:12.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.502+0000 7fa7fa37a740 1 --2- 172.21.15.90:0/3759079352 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55af24c9f7b0 0x55af24ca3000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:12.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.502+0000 7fa7fa37a740 1 -- 172.21.15.90:0/3759079352 >> 172.21.15.90:0/3759079352 conn(0x55af24c99ee0 msgr2=0x55af24c98de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:36:12.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.502+0000 7fa7d2000640 0 client.25813 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:36:12.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.503+0000 7fa7fa37a740 1 -- 172.21.15.90:0/3759079352 shutdown_connections 2024-04-02T14:36:12.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.504+0000 7fa7fa37a740 1 -- 172.21.15.90:0/3759079352 wait complete. 2024-04-02T14:36:12.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.504+0000 7fa7fa37a740 20 client.25813 trim_cache size 0 max 16384 2024-04-02T14:36:12.505 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-02T14:36:12.505 INFO:teuthology.orchestra.run.smithi090.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-02T14:36:12.505 INFO:teuthology.orchestra.run.smithi090.stdout: "scrubs": { 2024-04-02T14:36:12.505 INFO:teuthology.orchestra.run.smithi090.stdout: "22207f7f-a4b8-46d0-ba1f-01416d715cad": { 2024-04-02T14:36:12.506 INFO:teuthology.orchestra.run.smithi090.stdout: "path": "/", 2024-04-02T14:36:12.506 INFO:teuthology.orchestra.run.smithi090.stdout: "tag": "22207f7f-a4b8-46d0-ba1f-01416d715cad", 2024-04-02T14:36:12.506 INFO:teuthology.orchestra.run.smithi090.stdout: "options": "recursive,force" 2024-04-02T14:36:12.506 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:36:12.506 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:36:12.506 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-02T14:36:12.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.505+0000 7fa7f7600640 1 -- 172.21.15.90:0/356092153 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fa7d80a4020 msgr2=0x7fa7d80a64e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:12.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.505+0000 7fa7f7600640 1 --2- 172.21.15.90:0/356092153 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fa7d80a4020 0x7fa7d80a64e0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fa7f00731e0 tx=0x7fa7e4011860 comp rx=0 tx=0).stop 2024-04-02T14:36:12.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.505+0000 7fa7f7600640 1 -- 172.21.15.90:0/356092153 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa7f0114c50 msgr2=0x7fa7f01a09e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:12.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.505+0000 7fa7f7600640 1 --2- 172.21.15.90:0/356092153 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa7f0114c50 0x7fa7f01a09e0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fa7e8004af0 tx=0x7fa7e802cbc0 comp rx=0 tx=0).stop 2024-04-02T14:36:12.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.505+0000 7fa7f7600640 1 -- 172.21.15.90:0/356092153 shutdown_connections 2024-04-02T14:36:12.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.505+0000 7fa7f7600640 1 --2- 172.21.15.90:0/356092153 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fa7d80a4020 0x7fa7d80a64e0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:12.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.505+0000 7fa7f7600640 1 --2- 172.21.15.90:0/356092153 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa7f0114c50 0x7fa7f01a09e0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:12.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.505+0000 7fa7f7600640 1 --2- 172.21.15.90:0/356092153 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa7f0109600 0x7fa7f01a6d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:12.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.505+0000 7fa7f7600640 1 --2- 172.21.15.90:0/356092153 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa7f0108ce0 0x7fa7f01a6810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:12.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.505+0000 7fa7f7600640 1 -- 172.21.15.90:0/356092153 >> 172.21.15.90:0/356092153 conn(0x7fa7f006cbe0 msgr2=0x7fa7f0071b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:36:12.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.506+0000 7fa7f7600640 1 -- 172.21.15.90:0/356092153 shutdown_connections 2024-04-02T14:36:12.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:12.506+0000 7fa7f7600640 1 -- 172.21.15.90:0/356092153 wait complete. 2024-04-02T14:36:12.531 INFO:tasks.cephfs.filesystem:scrub status for tag:22207f7f-a4b8-46d0-ba1f-01416d715cad - {'path': '/', 'tag': '22207f7f-a4b8-46d0-ba1f-01416d715cad', 'options': 'recursive,force'} 2024-04-02T14:36:13.362 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:36:13 smithi092 ceph-mon[36497]: scrub queued for path: / 2024-04-02T14:36:13.362 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:36:13 smithi092 ceph-mon[36497]: scrub summary: idle+waiting paths [/] 2024-04-02T14:36:13.363 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:36:13 smithi092 ceph-mon[36497]: scrub summary: active paths [/] 2024-04-02T14:36:13.363 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:36:13 smithi092 ceph-mon[36497]: scrub summary: idle+waiting paths [/] 2024-04-02T14:36:13.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:36:13 smithi090 ceph-mon[31674]: scrub queued for path: / 2024-04-02T14:36:13.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:36:13 smithi090 ceph-mon[31674]: scrub summary: idle+waiting paths [/] 2024-04-02T14:36:13.443 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:36:13 smithi090 ceph-mon[31674]: scrub summary: active paths [/] 2024-04-02T14:36:13.444 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:36:13 smithi090 ceph-mon[31674]: scrub summary: idle+waiting paths [/] 2024-04-02T14:36:13.445 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:36:13 smithi177 ceph-mon[36431]: scrub queued for path: / 2024-04-02T14:36:13.445 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:36:13 smithi177 ceph-mon[36431]: scrub summary: idle+waiting paths [/] 2024-04-02T14:36:13.445 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:36:13 smithi177 ceph-mon[36431]: scrub summary: active paths [/] 2024-04-02T14:36:13.445 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:36:13 smithi177 ceph-mon[36431]: scrub summary: idle+waiting paths [/] 2024-04-02T14:36:14.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:36:14 smithi092 ceph-mon[36497]: pgmap v43: 97 pgs: 97 active+clean; 1010 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 46 MiB/s wr, 1.11k op/s 2024-04-02T14:36:14.437 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:36:14 smithi177 ceph-mon[36431]: pgmap v43: 97 pgs: 97 active+clean; 1010 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 46 MiB/s wr, 1.11k op/s 2024-04-02T14:36:14.438 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:36:14 smithi090 ceph-mon[31674]: pgmap v43: 97 pgs: 97 active+clean; 1010 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 46 MiB/s wr, 1.11k op/s 2024-04-02T14:36:16.360 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:36:16 smithi092 ceph-mon[36497]: pgmap v44: 97 pgs: 97 active+clean; 1.0 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 50 MiB/s wr, 1.51k op/s 2024-04-02T14:36:16.436 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:36:16 smithi177 ceph-mon[36431]: pgmap v44: 97 pgs: 97 active+clean; 1.0 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 50 MiB/s wr, 1.51k op/s 2024-04-02T14:36:16.438 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:36:16 smithi090 ceph-mon[31674]: pgmap v44: 97 pgs: 97 active+clean; 1.0 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 50 MiB/s wr, 1.51k op/s 2024-04-02T14:36:16.831 INFO:teuthology.orchestra.run.smithi090.stderr:progress: 45.0 s, 3872.7 tps, lat 25.829 ms stddev 29.026 2024-04-02T14:36:17.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:36:17 smithi092 ceph-mon[36497]: scrub summary: idle 2024-04-02T14:36:17.436 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:36:17 smithi177 ceph-mon[36431]: scrub summary: idle 2024-04-02T14:36:17.438 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:36:17 smithi090 ceph-mon[31674]: scrub summary: idle 2024-04-02T14:36:18.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:36:18 smithi092 ceph-mon[36497]: pgmap v45: 97 pgs: 97 active+clean; 1.0 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 7.6 MiB/s wr, 728 op/s 2024-04-02T14:36:18.436 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:36:18 smithi177 ceph-mon[36431]: pgmap v45: 97 pgs: 97 active+clean; 1.0 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 7.6 MiB/s wr, 728 op/s 2024-04-02T14:36:18.438 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:36:18 smithi090 ceph-mon[31674]: pgmap v45: 97 pgs: 97 active+clean; 1.0 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 7.6 MiB/s wr, 728 op/s 2024-04-02T14:36:20.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:36:20 smithi092 ceph-mon[36497]: pgmap v46: 97 pgs: 97 active+clean; 1.0 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 9.3 MiB/s wr, 909 op/s 2024-04-02T14:36:20.436 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:36:20 smithi177 ceph-mon[36431]: pgmap v46: 97 pgs: 97 active+clean; 1.0 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 9.3 MiB/s wr, 909 op/s 2024-04-02T14:36:20.438 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:36:20 smithi090 ceph-mon[31674]: pgmap v46: 97 pgs: 97 active+clean; 1.0 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 9.3 MiB/s wr, 909 op/s 2024-04-02T14:36:21.436 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:36:21 smithi177 ceph-mon[36431]: pgmap v47: 97 pgs: 97 active+clean; 1.0 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 9.3 MiB/s wr, 899 op/s 2024-04-02T14:36:21.438 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:36:21 smithi090 ceph-mon[31674]: pgmap v47: 97 pgs: 97 active+clean; 1.0 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 9.3 MiB/s wr, 899 op/s 2024-04-02T14:36:21.610 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:36:21 smithi092 ceph-mon[36497]: pgmap v47: 97 pgs: 97 active+clean; 1.0 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 9.3 MiB/s wr, 899 op/s 2024-04-02T14:36:21.831 INFO:teuthology.orchestra.run.smithi090.stderr:progress: 50.0 s, 3208.4 tps, lat 31.234 ms stddev 35.895 2024-04-02T14:36:23.686 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:36:23 smithi177 ceph-mon[36431]: pgmap v48: 97 pgs: 97 active+clean; 1.0 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 7.6 MiB/s wr, 741 op/s 2024-04-02T14:36:23.688 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:36:23 smithi090 ceph-mon[31674]: pgmap v48: 97 pgs: 97 active+clean; 1.0 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 7.6 MiB/s wr, 741 op/s 2024-04-02T14:36:23.860 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:36:23 smithi092 ceph-mon[36497]: pgmap v48: 97 pgs: 97 active+clean; 1.0 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 7.6 MiB/s wr, 741 op/s 2024-04-02T14:36:24.345 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.343+0000 7f70b0e00640 1 -- 172.21.15.90:0/1196447047 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f70ac108340 msgr2=0x7f70ac108720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:24.345 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.343+0000 7f70b0e00640 1 --2- 172.21.15.90:0/1196447047 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f70ac108340 0x7f70ac108720 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f709c009860 tx=0x7f709c02bb90 comp rx=0 tx=0).stop 2024-04-02T14:36:24.345 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.343+0000 7f70b0e00640 1 -- 172.21.15.90:0/1196447047 shutdown_connections 2024-04-02T14:36:24.345 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.343+0000 7f70b0e00640 1 --2- 172.21.15.90:0/1196447047 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f70ac10d410 0x7f70ac114fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:24.345 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.343+0000 7f70b0e00640 1 --2- 172.21.15.90:0/1196447047 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f70ac108cf0 0x7f70ac10cd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:24.345 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.343+0000 7f70b0e00640 1 --2- 172.21.15.90:0/1196447047 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f70ac108340 0x7f70ac108720 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:24.346 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.343+0000 7f70b0e00640 1 -- 172.21.15.90:0/1196447047 >> 172.21.15.90:0/1196447047 conn(0x7f70ac076070 msgr2=0x7f70ac078490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:36:24.346 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.344+0000 7f70b0e00640 1 -- 172.21.15.90:0/1196447047 shutdown_connections 2024-04-02T14:36:24.346 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.344+0000 7f70b0e00640 1 -- 172.21.15.90:0/1196447047 wait complete. 2024-04-02T14:36:24.346 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.345+0000 7f70b0e00640 1 Processor -- start 2024-04-02T14:36:24.346 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.345+0000 7f70b0e00640 1 -- start start 2024-04-02T14:36:24.347 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.346+0000 7f70b0e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f70ac108340 0x7f70ac06d0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:24.347 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.346+0000 7f70a9e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f70ac108340 0x7f70ac06d0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:24.347 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.346+0000 7f70a9e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f70ac108340 0x7f70ac06d0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:35500/0 (socket says 172.21.15.90:35500) 2024-04-02T14:36:24.347 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.346+0000 7f70b0e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f70ac10d410 0x7f70ac06d620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:24.348 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.346+0000 7f70a9e00640 1 -- 172.21.15.90:0/318888037 learned_addr learned my addr 172.21.15.90:0/318888037 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:36:24.348 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.347+0000 7f70b0e00640 1 --2- 172.21.15.90:0/318888037 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f70ac071950 0x7f70ac071de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:24.348 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.347+0000 7f70a9400640 1 --2- 172.21.15.90:0/318888037 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f70ac10d410 0x7f70ac06d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:24.348 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.347+0000 7f70b0e00640 1 -- 172.21.15.90:0/318888037 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f70ac06b8b0 con 0x7f70ac071950 2024-04-02T14:36:24.349 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.347+0000 7f70b0e00640 1 -- 172.21.15.90:0/318888037 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f70ac06b730 con 0x7f70ac108340 2024-04-02T14:36:24.349 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.347+0000 7f70b0e00640 1 -- 172.21.15.90:0/318888037 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f70ac06ba30 con 0x7f70ac10d410 2024-04-02T14:36:24.349 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.347+0000 7f70aa800640 1 --2- 172.21.15.90:0/318888037 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f70ac071950 0x7f70ac071de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:24.349 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.348+0000 7f70a9e00640 1 -- 172.21.15.90:0/318888037 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f70ac10d410 msgr2=0x7f70ac06d620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:24.350 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.348+0000 7f70a9e00640 1 --2- 172.21.15.90:0/318888037 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f70ac10d410 0x7f70ac06d620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:24.350 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.348+0000 7f70a9e00640 1 -- 172.21.15.90:0/318888037 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f70ac071950 msgr2=0x7f70ac071de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:24.350 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.348+0000 7f70a9e00640 1 --2- 172.21.15.90:0/318888037 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f70ac071950 0x7f70ac071de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:24.350 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.348+0000 7f70a9e00640 1 -- 172.21.15.90:0/318888037 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70ac072350 con 0x7f70ac108340 2024-04-02T14:36:24.350 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.348+0000 7f70aa800640 1 --2- 172.21.15.90:0/318888037 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f70ac071950 0x7f70ac071de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:36:24.350 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.348+0000 7f70a9e00640 1 --2- 172.21.15.90:0/318888037 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f70ac108340 0x7f70ac06d0e0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f709c0057c0 tx=0x7f709c004360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:36:24.350 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.348+0000 7f70a2a00640 1 -- 172.21.15.90:0/318888037 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f709c039040 con 0x7f70ac108340 2024-04-02T14:36:24.351 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.348+0000 7f70a2a00640 1 -- 172.21.15.90:0/318888037 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f709c02c530 con 0x7f70ac108340 2024-04-02T14:36:24.351 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.348+0000 7f70a2a00640 1 -- 172.21.15.90:0/318888037 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f709c0054f0 con 0x7f70ac108340 2024-04-02T14:36:24.351 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.349+0000 7f70b0e00640 1 -- 172.21.15.90:0/318888037 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70ac0725e0 con 0x7f70ac108340 2024-04-02T14:36:24.351 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.349+0000 7f70b0e00640 1 -- 172.21.15.90:0/318888037 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70ac080560 con 0x7f70ac108340 2024-04-02T14:36:24.356 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.355+0000 7f70a2a00640 1 -- 172.21.15.90:0/318888037 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 25) ==== 138930+0+0 (secure 0 0 0) 0x7f709c046020 con 0x7f70ac108340 2024-04-02T14:36:24.358 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.357+0000 7f70a2a00640 1 --2- 172.21.15.90:0/318888037 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f70900a4140 0x7f70900a6600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:24.359 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.358+0000 7f70a2a00640 1 -- 172.21.15.90:0/318888037 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f709c0f2070 con 0x7f70ac108340 2024-04-02T14:36:24.359 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.358+0000 7f70a9400640 1 --2- 172.21.15.90:0/318888037 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f70900a4140 0x7f70900a6600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:24.360 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.359+0000 7f70b3a71740 1 Processor -- start 2024-04-02T14:36:24.361 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.359+0000 7f70b3a71740 1 -- start start 2024-04-02T14:36:24.361 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.360+0000 7f70a9400640 1 --2- 172.21.15.90:0/318888037 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f70900a4140 0x7f70900a6600 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f70ac06e670 tx=0x7f7098009260 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:36:24.361 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.360+0000 7f70b3a71740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558c82a74eb0 0x558c82a75290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:24.362 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.360+0000 7f70b3a71740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558c82a757d0 0x558c82a7de70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:24.362 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.360+0000 7f70b3a71740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558c82a7e3b0 0x558c82a807a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:24.362 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.360+0000 7f70b3a71740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x558c829d9d70 con 0x558c82a7e3b0 2024-04-02T14:36:24.362 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.360+0000 7f70b3a71740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x558c829822a0 con 0x558c82a757d0 2024-04-02T14:36:24.362 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.360+0000 7f70b3a71740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x558c829d5d20 con 0x558c82a74eb0 2024-04-02T14:36:24.362 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.360+0000 7f70a9e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558c82a74eb0 0x558c82a75290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:24.362 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.360+0000 7f70a9e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558c82a74eb0 0x558c82a75290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:56144/0 (socket says 172.21.15.90:56144) 2024-04-02T14:36:24.363 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.360+0000 7f70a9e00640 1 -- 172.21.15.90:0/2056792650 learned_addr learned my addr 172.21.15.90:0/2056792650 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:36:24.363 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.360+0000 7f70a9400640 1 --2- 172.21.15.90:0/2056792650 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558c82a757d0 0x558c82a7de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:24.363 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.361+0000 7f70aa800640 1 --2- 172.21.15.90:0/2056792650 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558c82a7e3b0 0x558c82a807a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:24.363 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.361+0000 7f70a9e00640 1 -- 172.21.15.90:0/2056792650 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558c82a757d0 msgr2=0x558c82a7de70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:36:24.363 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.361+0000 7f70a9e00640 1 --2- 172.21.15.90:0/2056792650 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558c82a757d0 0x558c82a7de70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:24.363 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.361+0000 7f70a9e00640 1 -- 172.21.15.90:0/2056792650 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558c82a7e3b0 msgr2=0x558c82a807a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:36:24.363 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.361+0000 7f70a9e00640 1 --2- 172.21.15.90:0/2056792650 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558c82a7e3b0 0x558c82a807a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:24.363 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.361+0000 7f70a9e00640 1 -- 172.21.15.90:0/2056792650 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558c82996840 con 0x558c82a74eb0 2024-04-02T14:36:24.363 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.361+0000 7f70a9400640 1 --2- 172.21.15.90:0/2056792650 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558c82a757d0 0x558c82a7de70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:36:24.363 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.361+0000 7f70aa800640 1 --2- 172.21.15.90:0/2056792650 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558c82a7e3b0 0x558c82a807a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:36:24.365 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.362+0000 7f70a9e00640 1 --2- 172.21.15.90:0/2056792650 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558c82a74eb0 0x558c82a75290 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f709c000c00 tx=0x7f709c0bd850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f749b28231aa3104 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:36:24.365 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.362+0000 7f7097400640 1 -- 172.21.15.90:0/2056792650 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f709c0d7070 con 0x558c82a74eb0 2024-04-02T14:36:24.365 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.362+0000 7f7097400640 1 -- 172.21.15.90:0/2056792650 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f709c0bdef0 con 0x558c82a74eb0 2024-04-02T14:36:24.365 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.362+0000 7f7097400640 1 -- 172.21.15.90:0/2056792650 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f709c02c6e0 con 0x558c82a74eb0 2024-04-02T14:36:24.365 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.363+0000 7f70b3a71740 1 -- 172.21.15.90:0/2056792650 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558c82a74eb0 msgr2=0x558c82a75290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:24.365 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.363+0000 7f70b3a71740 1 --2- 172.21.15.90:0/2056792650 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558c82a74eb0 0x558c82a75290 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f709c000c00 tx=0x7f709c0bd850 comp rx=0 tx=0).stop 2024-04-02T14:36:24.365 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.363+0000 7f70b3a71740 1 -- 172.21.15.90:0/2056792650 shutdown_connections 2024-04-02T14:36:24.365 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.363+0000 7f70b3a71740 1 --2- 172.21.15.90:0/2056792650 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558c82a7e3b0 0x558c82a807a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:24.365 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.363+0000 7f70b3a71740 1 --2- 172.21.15.90:0/2056792650 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558c82a757d0 0x558c82a7de70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:24.365 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.363+0000 7f70b3a71740 1 --2- 172.21.15.90:0/2056792650 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558c82a74eb0 0x558c82a75290 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:24.365 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.363+0000 7f70b3a71740 1 -- 172.21.15.90:0/2056792650 >> 172.21.15.90:0/2056792650 conn(0x558c82a6f5e0 msgr2=0x558c82a6f9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:36:24.366 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.364+0000 7f70b3a71740 1 -- 172.21.15.90:0/2056792650 shutdown_connections 2024-04-02T14:36:24.366 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.364+0000 7f70b3a71740 1 -- 172.21.15.90:0/2056792650 wait complete. 2024-04-02T14:36:24.366 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.364+0000 7f70b3a71740 1 -- start start 2024-04-02T14:36:24.366 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.365+0000 7f70b3a71740 1 Processor -- start 2024-04-02T14:36:24.367 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.366+0000 7f70b3a71740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558c82a74eb0 0x558c82a7b4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:24.367 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.366+0000 7f70b3a71740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558c82a757d0 0x558c82a7b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:24.367 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.366+0000 7f70b3a71740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558c82a7e3b0 0x558c82a7bf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:24.367 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.366+0000 7f70b3a71740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x558c82a80f60 con 0x558c82a757d0 2024-04-02T14:36:24.368 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.366+0000 7f70b3a71740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x558c82995660 con 0x558c82a7e3b0 2024-04-02T14:36:24.368 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.366+0000 7f70b3a71740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x558c82a810e0 con 0x558c82a74eb0 2024-04-02T14:36:24.368 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.366+0000 7f70b3a71740 10 client.0 _ll_register_callbacks cb 0x558c82995050 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:36:24.368 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.367+0000 7f70a9e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558c82a74eb0 0x558c82a7b4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:24.368 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.367+0000 7f70aa800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558c82a7e3b0 0x558c82a7bf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:24.370 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.367+0000 7f70aa800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558c82a7e3b0 0x558c82a7bf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:35512/0 (socket says 172.21.15.90:35512) 2024-04-02T14:36:24.370 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.367+0000 7f70aa800640 1 -- 172.21.15.90:0/935385870 learned_addr learned my addr 172.21.15.90:0/935385870 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:36:24.370 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.367+0000 7f70aa800640 1 -- 172.21.15.90:0/935385870 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558c82a74eb0 msgr2=0x558c82a7b4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:36:24.370 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.367+0000 7f70aa800640 1 --2- 172.21.15.90:0/935385870 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558c82a74eb0 0x558c82a7b4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:24.370 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.367+0000 7f70aa800640 1 -- 172.21.15.90:0/935385870 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558c82a757d0 msgr2=0x558c82a7b9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:36:24.370 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.367+0000 7f70a9400640 1 --2- 172.21.15.90:0/935385870 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558c82a757d0 0x558c82a7b9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:24.370 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.367+0000 7f70aa800640 1 --2- 172.21.15.90:0/935385870 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558c82a757d0 0x558c82a7b9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:24.370 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.367+0000 7f70aa800640 1 -- 172.21.15.90:0/935385870 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558c829a4e80 con 0x558c82a7e3b0 2024-04-02T14:36:24.370 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.368+0000 7f70a9400640 1 --2- 172.21.15.90:0/935385870 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558c82a757d0 0x558c82a7b9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:36:24.370 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.368+0000 7f70a9e00640 1 --2- 172.21.15.90:0/935385870 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558c82a74eb0 0x558c82a7b4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:36:24.371 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.369+0000 7f708be00640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:36:24.371 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.369+0000 7f70aa800640 1 --2- 172.21.15.90:0/935385870 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558c82a7e3b0 0x558c82a7bf30 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f70a4003c20 tx=0x7f70a4004410 comp rx=0 tx=0).ready entity=mon.2 client_cookie=44fc56a091467941 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:36:24.371 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.370+0000 7f708be00640 1 -- 172.21.15.90:0/935385870 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70a4012860 con 0x558c82a7e3b0 2024-04-02T14:36:24.371 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.370+0000 7f708be00640 1 -- 172.21.15.90:0/935385870 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f70a4012a00 con 0x558c82a7e3b0 2024-04-02T14:36:24.372 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.370+0000 7f70b3a71740 1 -- 172.21.15.90:0/935385870 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f709c0bdef0 con 0x558c82a7e3b0 2024-04-02T14:36:24.383 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.382+0000 7f70a2a00640 1 -- 172.21.15.90:0/318888037 <== mon.2 v2:172.21.15.92:3300/0 6 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f709c0c1030 con 0x7f70ac108340 2024-04-02T14:36:24.388 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.387+0000 7f708be00640 1 -- 172.21.15.90:0/935385870 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f70a4012c20 con 0x558c82a7e3b0 2024-04-02T14:36:24.388 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.387+0000 7f70b3a71740 10 client.25892 fetch_fsmap learned FSMap version 27 2024-04-02T14:36:24.388 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.387+0000 7f70b3a71740 1 -- 172.21.15.90:0/935385870 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x558c826ee210 con 0x558c82a7e3b0 2024-04-02T14:36:24.390 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.389+0000 7f708be00640 1 -- 172.21.15.90:0/935385870 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f709c0bdef0 con 0x558c82a7e3b0 2024-04-02T14:36:24.390 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.389+0000 7f708be00640 1 -- 172.21.15.90:0/935385870 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f70a4011450 con 0x558c82a7e3b0 2024-04-02T14:36:24.391 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.389+0000 7f70b3a71740 10 client.25892 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:36:24.391 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.389+0000 7f70b3a71740 10 client.25892 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-02T14:36:24.391 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.389+0000 7f70b3a71740 20 client.25892 populate_metadata read hostname 'smithi090' 2024-04-02T14:36:24.391 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.389+0000 7f70b3a71740 1 --2- 172.21.15.90:0/935385870 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x558c82a8e3d0 0x558c82a907c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:24.391 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.389+0000 7f70b3a71740 4 client.25892 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:36:24.391 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.389+0000 7f70b3a71740 1 -- 172.21.15.90:0/935385870 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558c82989cd0 con 0x558c82a8e3d0 2024-04-02T14:36:24.391 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.390+0000 7f70a9e00640 1 --2- 172.21.15.90:0/935385870 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x558c82a8e3d0 0x558c82a907c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:24.392 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.390+0000 7f708be00640 10 client.25892 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:36:24.392 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.391+0000 7f70a9e00640 1 --2- 172.21.15.90:0/935385870 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x558c82a8e3d0 0x558c82a907c0 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bb7dd475161a57fa server_cookie=3b89daeb40e08af2 in_seq=0 out_seq=0 2024-04-02T14:36:24.392 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.391+0000 7f708be00640 1 -- 172.21.15.90:0/935385870 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x558c82989cd0 con 0x558c82a8e3d0 2024-04-02T14:36:24.392 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.391+0000 7f708be00640 10 client.25892 handle_command_reply: tid=0 2024-04-02T14:36:24.393 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.392+0000 7f70b3a71740 1 client.25892 shutdown 2024-04-02T14:36:24.398 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.397+0000 7f70b3a71740 1 -- 172.21.15.90:0/935385870 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558c82a7e3b0 msgr2=0x558c82a7bf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:24.398 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.397+0000 7f70b3a71740 1 --2- 172.21.15.90:0/935385870 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558c82a7e3b0 0x558c82a7bf30 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f70a4003c20 tx=0x7f70a4004410 comp rx=0 tx=0).stop 2024-04-02T14:36:24.399 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.398+0000 7f70b3a71740 1 -- 172.21.15.90:0/935385870 shutdown_connections 2024-04-02T14:36:24.399 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.398+0000 7f70b3a71740 1 --2- 172.21.15.90:0/935385870 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x558c82a8e3d0 0x558c82a907c0 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:24.400 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.399+0000 7f70b3a71740 1 --2- 172.21.15.90:0/935385870 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558c82a7e3b0 0x558c82a7bf30 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:24.400 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.399+0000 7f70b3a71740 1 --2- 172.21.15.90:0/935385870 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558c82a757d0 0x558c82a7b9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:24.400 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.399+0000 7f708be00640 0 client.25892 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:36:24.400 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.399+0000 7f70b3a71740 1 --2- 172.21.15.90:0/935385870 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558c82a74eb0 0x558c82a7b4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:24.400 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.399+0000 7f70b3a71740 1 -- 172.21.15.90:0/935385870 >> 172.21.15.90:0/935385870 conn(0x558c82a6f5e0 msgr2=0x558c82a71330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:36:24.401 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.400+0000 7f70b3a71740 1 -- 172.21.15.90:0/935385870 shutdown_connections 2024-04-02T14:36:24.402 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.401+0000 7f70b3a71740 1 -- 172.21.15.90:0/935385870 wait complete. 2024-04-02T14:36:24.402 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.401+0000 7f70b3a71740 20 client.25892 trim_cache size 0 max 16384 2024-04-02T14:36:24.445 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.444+0000 7f70b3a71740 1 Processor -- start 2024-04-02T14:36:24.446 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.444+0000 7f70b3a71740 1 -- start start 2024-04-02T14:36:24.447 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.445+0000 7f70b3a71740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558c82a74eb0 0x558c82a89c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:24.447 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.445+0000 7f70b3a71740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558c82a757d0 0x558c82a7bd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:24.447 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.445+0000 7f70b3a71740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558c82a7e3b0 0x558c82a6e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:24.447 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.445+0000 7f70b3a71740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f70ac1c4de0 con 0x558c82a74eb0 2024-04-02T14:36:24.447 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.445+0000 7f70b3a71740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f70ac1c6920 con 0x558c82a757d0 2024-04-02T14:36:24.448 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.445+0000 7f70b3a71740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f70ac1c54b0 con 0x558c82a7e3b0 2024-04-02T14:36:24.448 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.445+0000 7f70a9e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558c82a74eb0 0x558c82a89c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:24.448 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.445+0000 7f70a9400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558c82a757d0 0x558c82a7bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:24.448 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.445+0000 7f70a9e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558c82a74eb0 0x558c82a89c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41822/0 (socket says 172.21.15.90:41822) 2024-04-02T14:36:24.448 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.445+0000 7f70a9e00640 1 -- 172.21.15.90:0/1932745717 learned_addr learned my addr 172.21.15.90:0/1932745717 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:36:24.448 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.446+0000 7f70a9e00640 1 -- 172.21.15.90:0/1932745717 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558c82a7e3b0 msgr2=0x558c82a6e4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:36:24.448 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.446+0000 7f70aa800640 1 --2- 172.21.15.90:0/1932745717 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558c82a7e3b0 0x558c82a6e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:24.448 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.446+0000 7f70a9e00640 1 --2- 172.21.15.90:0/1932745717 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558c82a7e3b0 0x558c82a6e4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:24.448 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.446+0000 7f70a9e00640 1 -- 172.21.15.90:0/1932745717 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558c82a757d0 msgr2=0x558c82a7bd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:36:24.448 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.446+0000 7f70a9e00640 1 --2- 172.21.15.90:0/1932745717 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558c82a757d0 0x558c82a7bd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:24.448 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.446+0000 7f70a9e00640 1 -- 172.21.15.90:0/1932745717 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558c827ba620 con 0x558c82a74eb0 2024-04-02T14:36:24.448 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.446+0000 7f70a9e00640 1 --2- 172.21.15.90:0/1932745717 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558c82a74eb0 0x558c82a89c50 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f709c000c00 tx=0x7f709c08a040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14489e900621dc15 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:36:24.449 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.446+0000 7f7096a00640 1 -- 172.21.15.90:0/1932745717 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f709c0d7040 con 0x558c82a74eb0 2024-04-02T14:36:24.450 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.447+0000 7f7096a00640 1 -- 172.21.15.90:0/1932745717 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f709c0ecb40 con 0x558c82a74eb0 2024-04-02T14:36:24.450 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.447+0000 7f70b3a71740 1 -- 172.21.15.90:0/1932745717 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558c82a74eb0 msgr2=0x558c82a89c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:24.450 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.447+0000 7f70b3a71740 1 --2- 172.21.15.90:0/1932745717 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558c82a74eb0 0x558c82a89c50 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f709c000c00 tx=0x7f709c08a040 comp rx=0 tx=0).stop 2024-04-02T14:36:24.450 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.448+0000 7f70b3a71740 1 -- 172.21.15.90:0/1932745717 shutdown_connections 2024-04-02T14:36:24.450 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.448+0000 7f70b3a71740 1 --2- 172.21.15.90:0/1932745717 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558c82a7e3b0 0x558c82a6e4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:24.450 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.448+0000 7f70b3a71740 1 --2- 172.21.15.90:0/1932745717 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558c82a757d0 0x558c82a7bd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:24.450 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.448+0000 7f70b3a71740 1 --2- 172.21.15.90:0/1932745717 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558c82a74eb0 0x558c82a89c50 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:24.450 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.448+0000 7f70b3a71740 1 -- 172.21.15.90:0/1932745717 >> 172.21.15.90:0/1932745717 conn(0x558c82a6f5e0 msgr2=0x558c82a6ddb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:36:24.450 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.449+0000 7f70b3a71740 1 -- 172.21.15.90:0/1932745717 shutdown_connections 2024-04-02T14:36:24.450 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.449+0000 7f70b3a71740 1 -- 172.21.15.90:0/1932745717 wait complete. 2024-04-02T14:36:24.451 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.449+0000 7f70b3a71740 1 -- start start 2024-04-02T14:36:24.451 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.449+0000 7f70b3a71740 1 Processor -- start 2024-04-02T14:36:24.452 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.450+0000 7f70b3a71740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558c82a74eb0 0x558c82a86640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:24.452 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.450+0000 7f70b3a71740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558c82a757d0 0x558c82a77f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:24.452 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.450+0000 7f70b3a71740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558c82a7e3b0 0x558c82a784b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:24.452 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.450+0000 7f70b3a71740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f70ac1c6250 con 0x558c82a74eb0 2024-04-02T14:36:24.452 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.450+0000 7f70b3a71740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f70ac1c5b80 con 0x558c82a757d0 2024-04-02T14:36:24.452 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.450+0000 7f70b3a71740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f70ac1c2de0 con 0x558c82a7e3b0 2024-04-02T14:36:24.452 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.450+0000 7f70b3a71740 10 client.0 _ll_register_callbacks cb 0x7f70a4009d90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:36:24.452 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.451+0000 7f70a9400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558c82a757d0 0x558c82a77f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:24.453 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.451+0000 7f70a9400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558c82a757d0 0x558c82a77f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:35532/0 (socket says 172.21.15.90:35532) 2024-04-02T14:36:24.453 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.451+0000 7f70a9400640 1 -- 172.21.15.90:0/871561864 learned_addr learned my addr 172.21.15.90:0/871561864 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:36:24.453 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.451+0000 7f70a9400640 1 -- 172.21.15.90:0/871561864 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558c82a7e3b0 msgr2=0x558c82a784b0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-02T14:36:24.453 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.451+0000 7f70a9400640 1 --2- 172.21.15.90:0/871561864 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558c82a7e3b0 0x558c82a784b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:24.454 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.451+0000 7f70a9400640 1 -- 172.21.15.90:0/871561864 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558c82a74eb0 msgr2=0x558c82a86640 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-02T14:36:24.454 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.452+0000 7f70a9e00640 1 --2- 172.21.15.90:0/871561864 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558c82a74eb0 0x558c82a86640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:24.454 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.452+0000 7f70a9400640 1 --2- 172.21.15.90:0/871561864 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558c82a74eb0 0x558c82a86640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:24.454 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.452+0000 7f70a9400640 1 -- 172.21.15.90:0/871561864 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558c82a78ce0 con 0x558c82a757d0 2024-04-02T14:36:24.454 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.452+0000 7f70a9e00640 1 --2- 172.21.15.90:0/871561864 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558c82a74eb0 0x558c82a86640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:36:24.454 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.452+0000 7f70a9400640 1 --2- 172.21.15.90:0/871561864 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558c82a757d0 0x558c82a77f10 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f70ac06b1e0 tx=0x7f7098004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3dce6e6180288f61 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:36:24.454 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.452+0000 7f708be00640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:36:24.455 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.453+0000 7f708be00640 1 -- 172.21.15.90:0/871561864 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f709802f060 con 0x558c82a757d0 2024-04-02T14:36:24.455 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.453+0000 7f708be00640 1 -- 172.21.15.90:0/871561864 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f7098021ca0 con 0x558c82a757d0 2024-04-02T14:36:24.455 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.453+0000 7f70b3a71740 1 -- 172.21.15.90:0/871561864 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f709c0ecb40 con 0x558c82a757d0 2024-04-02T14:36:24.455 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.454+0000 7f708be00640 1 -- 172.21.15.90:0/871561864 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f709801f420 con 0x558c82a757d0 2024-04-02T14:36:24.456 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.454+0000 7f70b3a71740 10 client.25898 fetch_fsmap learned FSMap version 27 2024-04-02T14:36:24.456 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.454+0000 7f70b3a71740 1 -- 172.21.15.90:0/871561864 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x558c82af4990 con 0x558c82a757d0 2024-04-02T14:36:24.457 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.455+0000 7f708be00640 1 -- 172.21.15.90:0/871561864 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f709c0ecb40 con 0x558c82a757d0 2024-04-02T14:36:24.457 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.455+0000 7f708be00640 1 -- 172.21.15.90:0/871561864 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f709802a6a0 con 0x558c82a757d0 2024-04-02T14:36:24.457 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.455+0000 7f70b3a71740 10 client.25898 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:36:24.457 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.455+0000 7f70b3a71740 10 client.25898 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-02T14:36:24.457 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.455+0000 7f70b3a71740 20 client.25898 populate_metadata read hostname 'smithi090' 2024-04-02T14:36:24.457 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.455+0000 7f70b3a71740 1 --2- 172.21.15.90:0/871561864 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x558c82a8e3d0 0x558c82af7160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:24.457 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.455+0000 7f70b3a71740 4 client.25898 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-02T14:36:24.457 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.455+0000 7f70b3a71740 1 -- 172.21.15.90:0/871561864 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "flush journal"}) -- 0x558c829e72b0 con 0x558c82a8e3d0 2024-04-02T14:36:24.457 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.455+0000 7f70a9e00640 1 --2- 172.21.15.90:0/871561864 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x558c82a8e3d0 0x558c82af7160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:24.458 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.456+0000 7f708be00640 10 client.25898 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:36:24.458 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.456+0000 7f70a9e00640 1 --2- 172.21.15.90:0/871561864 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x558c82a8e3d0 0x558c82af7160 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=94b6909ae8c781e4 server_cookie=b2fdcc221cd52c9e in_seq=0 out_seq=0 2024-04-02T14:36:24.479 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.478+0000 7f708be00640 1 -- 172.21.15.90:0/871561864 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x558c829e72b0 con 0x558c82a8e3d0 2024-04-02T14:36:24.479 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.478+0000 7f708be00640 10 client.25898 handle_command_reply: tid=0 2024-04-02T14:36:24.479 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.478+0000 7f70b3a71740 1 client.25898 shutdown 2024-04-02T14:36:24.480 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.478+0000 7f70b3a71740 1 -- 172.21.15.90:0/871561864 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558c82a757d0 msgr2=0x558c82a77f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:24.480 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.478+0000 7f70b3a71740 1 --2- 172.21.15.90:0/871561864 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558c82a757d0 0x558c82a77f10 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f70ac06b1e0 tx=0x7f7098004730 comp rx=0 tx=0).stop 2024-04-02T14:36:24.480 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.479+0000 7f70b3a71740 1 -- 172.21.15.90:0/871561864 shutdown_connections 2024-04-02T14:36:24.481 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.479+0000 7f70b3a71740 1 --2- 172.21.15.90:0/871561864 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x558c82a8e3d0 0x558c82af7160 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:24.481 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.479+0000 7f70b3a71740 1 --2- 172.21.15.90:0/871561864 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558c82a7e3b0 0x558c82a784b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:24.481 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.479+0000 7f70b3a71740 1 --2- 172.21.15.90:0/871561864 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558c82a757d0 0x558c82a77f10 secure :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f70ac06b1e0 tx=0x7f7098004730 comp rx=0 tx=0).stop 2024-04-02T14:36:24.481 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.479+0000 7f70b3a71740 1 --2- 172.21.15.90:0/871561864 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558c82a74eb0 0x558c82a86640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:24.481 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.479+0000 7f70b3a71740 1 -- 172.21.15.90:0/871561864 >> 172.21.15.90:0/871561864 conn(0x558c82a6f5e0 msgr2=0x558c82a71170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:36:24.481 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.479+0000 7f708be00640 0 client.25898 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:36:24.481 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.479+0000 7f70b3a71740 1 -- 172.21.15.90:0/871561864 shutdown_connections 2024-04-02T14:36:24.481 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.479+0000 7f70b3a71740 1 -- 172.21.15.90:0/871561864 wait complete. 2024-04-02T14:36:24.481 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.479+0000 7f70b3a71740 20 client.25898 trim_cache size 0 max 16384 2024-04-02T14:36:24.482 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:{ 2024-04-02T14:36:24.482 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "message": "", 2024-04-02T14:36:24.482 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "return_code": 0 2024-04-02T14:36:24.482 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:} 2024-04-02T14:36:24.483 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.480+0000 7f70b0e00640 1 -- 172.21.15.90:0/318888037 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f70900a4140 msgr2=0x7f70900a6600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:24.483 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.480+0000 7f70b0e00640 1 --2- 172.21.15.90:0/318888037 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f70900a4140 0x7f70900a6600 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f70ac06e670 tx=0x7f7098009260 comp rx=0 tx=0).stop 2024-04-02T14:36:24.483 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.480+0000 7f70b0e00640 1 -- 172.21.15.90:0/318888037 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f70ac108340 msgr2=0x7f70ac06d0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:24.483 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.480+0000 7f70b0e00640 1 --2- 172.21.15.90:0/318888037 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f70ac108340 0x7f70ac06d0e0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f709c0057c0 tx=0x7f709c004360 comp rx=0 tx=0).stop 2024-04-02T14:36:24.483 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.480+0000 7f70b0e00640 1 -- 172.21.15.90:0/318888037 shutdown_connections 2024-04-02T14:36:24.483 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.480+0000 7f70b0e00640 1 --2- 172.21.15.90:0/318888037 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f70900a4140 0x7f70900a6600 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:24.483 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.480+0000 7f70b0e00640 1 --2- 172.21.15.90:0/318888037 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f70ac071950 0x7f70ac071de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:24.483 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.480+0000 7f70b0e00640 1 --2- 172.21.15.90:0/318888037 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f70ac10d410 0x7f70ac06d620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:24.483 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.480+0000 7f70b0e00640 1 --2- 172.21.15.90:0/318888037 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f70ac108340 0x7f70ac06d0e0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:24.484 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.480+0000 7f70b0e00640 1 -- 172.21.15.90:0/318888037 >> 172.21.15.90:0/318888037 conn(0x7f70ac076070 msgr2=0x7f70ac078340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:36:24.484 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.481+0000 7f70b0e00640 1 -- 172.21.15.90:0/318888037 shutdown_connections 2024-04-02T14:36:24.484 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:24.481+0000 7f70b0e00640 1 -- 172.21.15.90:0/318888037 wait complete. 2024-04-02T14:36:25.686 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:36:25 smithi177 ceph-mon[36431]: pgmap v49: 97 pgs: 97 active+clean; 1.1 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 13 MiB/s wr, 1.09k op/s 2024-04-02T14:36:25.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:36:25 smithi177 ceph-mon[36431]: mgrmap e26: x(active, since 92s), standbys: z, y 2024-04-02T14:36:25.689 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:36:25 smithi090 ceph-mon[31674]: pgmap v49: 97 pgs: 97 active+clean; 1.1 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 13 MiB/s wr, 1.09k op/s 2024-04-02T14:36:25.689 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:36:25 smithi090 ceph-mon[31674]: mgrmap e26: x(active, since 92s), standbys: z, y 2024-04-02T14:36:25.861 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:36:25 smithi092 ceph-mon[36497]: pgmap v49: 97 pgs: 97 active+clean; 1.1 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 13 MiB/s wr, 1.09k op/s 2024-04-02T14:36:25.861 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:36:25 smithi092 ceph-mon[36497]: mgrmap e26: x(active, since 92s), standbys: z, y 2024-04-02T14:36:26.831 INFO:teuthology.orchestra.run.smithi090.stderr:progress: 55.0 s, 3648.8 tps, lat 27.332 ms stddev 29.919 2024-04-02T14:36:27.686 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:36:27 smithi177 ceph-mon[36431]: pgmap v50: 97 pgs: 97 active+clean; 1.1 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 8.5 MiB/s wr, 691 op/s 2024-04-02T14:36:27.688 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:36:27 smithi090 ceph-mon[31674]: pgmap v50: 97 pgs: 97 active+clean; 1.1 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 8.5 MiB/s wr, 691 op/s 2024-04-02T14:36:27.860 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:36:27 smithi092 ceph-mon[36497]: pgmap v50: 97 pgs: 97 active+clean; 1.1 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 8.5 MiB/s wr, 691 op/s 2024-04-02T14:36:29.686 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:36:29 smithi177 ceph-mon[36431]: pgmap v51: 97 pgs: 97 active+clean; 1.1 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 11 MiB/s wr, 926 op/s 2024-04-02T14:36:29.688 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:36:29 smithi090 ceph-mon[31674]: pgmap v51: 97 pgs: 97 active+clean; 1.1 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 11 MiB/s wr, 926 op/s 2024-04-02T14:36:29.860 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:36:29 smithi092 ceph-mon[36497]: pgmap v51: 97 pgs: 97 active+clean; 1.1 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 11 MiB/s wr, 926 op/s 2024-04-02T14:36:31.663 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:36:31 smithi092 ceph-mon[36497]: pgmap v52: 97 pgs: 97 active+clean; 1.1 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 10 MiB/s wr, 844 op/s 2024-04-02T14:36:31.686 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:36:31 smithi177 ceph-mon[36431]: pgmap v52: 97 pgs: 97 active+clean; 1.1 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 10 MiB/s wr, 844 op/s 2024-04-02T14:36:31.688 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:36:31 smithi090 ceph-mon[31674]: pgmap v52: 97 pgs: 97 active+clean; 1.1 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 10 MiB/s wr, 844 op/s 2024-04-02T14:36:31.832 INFO:teuthology.orchestra.run.smithi090.stderr:progress: 60.0 s, 3972.8 tps, lat 25.262 ms stddev 28.007 2024-04-02T14:36:33.686 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:36:33 smithi177 ceph-mon[36431]: pgmap v53: 97 pgs: 97 active+clean; 1.1 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 8.3 MiB/s wr, 682 op/s 2024-04-02T14:36:33.688 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:36:33 smithi090 ceph-mon[31674]: pgmap v53: 97 pgs: 97 active+clean; 1.1 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 8.3 MiB/s wr, 682 op/s 2024-04-02T14:36:33.860 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:36:33 smithi092 ceph-mon[36497]: pgmap v53: 97 pgs: 97 active+clean; 1.1 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 8.3 MiB/s wr, 682 op/s 2024-04-02T14:36:35.686 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:36:35 smithi177 ceph-mon[36431]: pgmap v54: 97 pgs: 97 active+clean; 1.1 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 14 MiB/s wr, 1.06k op/s 2024-04-02T14:36:35.689 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:36:35 smithi090 ceph-mon[31674]: pgmap v54: 97 pgs: 97 active+clean; 1.1 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 14 MiB/s wr, 1.06k op/s 2024-04-02T14:36:35.861 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:36:35 smithi092 ceph-mon[36497]: pgmap v54: 97 pgs: 97 active+clean; 1.1 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 14 MiB/s wr, 1.06k op/s 2024-04-02T14:36:36.832 INFO:teuthology.orchestra.run.smithi090.stderr:progress: 65.0 s, 3621.5 tps, lat 27.571 ms stddev 28.439 2024-04-02T14:36:37.669 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.666+0000 7fbd6b400640 1 -- 172.21.15.90:0/4232938552 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbd64076950 msgr2=0x7fbd64112430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:37.669 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.666+0000 7fbd6b400640 1 --2- 172.21.15.90:0/4232938552 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbd64076950 0x7fbd64112430 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fbd4c009890 tx=0x7fbd4c02bb90 comp rx=0 tx=0).stop 2024-04-02T14:36:37.669 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.666+0000 7fbd6b400640 1 -- 172.21.15.90:0/4232938552 shutdown_connections 2024-04-02T14:36:37.669 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.666+0000 7fbd6b400640 1 --2- 172.21.15.90:0/4232938552 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbd64112970 0x7fbd64114d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:37.669 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.666+0000 7fbd6b400640 1 --2- 172.21.15.90:0/4232938552 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbd64076950 0x7fbd64112430 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:37.669 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.666+0000 7fbd6b400640 1 --2- 172.21.15.90:0/4232938552 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbd64076030 0x7fbd64076410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:37.669 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.666+0000 7fbd6b400640 1 -- 172.21.15.90:0/4232938552 >> 172.21.15.90:0/4232938552 conn(0x7fbd640ff7c0 msgr2=0x7fbd64101be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:36:37.669 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.666+0000 7fbd6b400640 1 -- 172.21.15.90:0/4232938552 shutdown_connections 2024-04-02T14:36:37.669 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.666+0000 7fbd6b400640 1 -- 172.21.15.90:0/4232938552 wait complete. 2024-04-02T14:36:37.670 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.668+0000 7fbd6b400640 1 Processor -- start 2024-04-02T14:36:37.670 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.668+0000 7fbd6b400640 1 -- start start 2024-04-02T14:36:37.670 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.669+0000 7fbd6b400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbd64076030 0x7fbd64111e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:37.671 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.669+0000 7fbd6b400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbd64076950 0x7fbd641123d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:37.671 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.669+0000 7fbd6b400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbd64112970 0x7fbd6410b140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:37.671 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.669+0000 7fbd6b400640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fbd64118510 con 0x7fbd64076030 2024-04-02T14:36:37.671 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.669+0000 7fbd6b400640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fbd64118390 con 0x7fbd64076950 2024-04-02T14:36:37.671 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.669+0000 7fbd6b400640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fbd64118690 con 0x7fbd64112970 2024-04-02T14:36:37.672 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.669+0000 7fbd69400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbd64112970 0x7fbd6410b140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:37.672 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.669+0000 7fbd68a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbd64076030 0x7fbd64111e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:37.672 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.669+0000 7fbd69400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbd64112970 0x7fbd6410b140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:57820/0 (socket says 172.21.15.90:57820) 2024-04-02T14:36:37.672 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.669+0000 7fbd69400640 1 -- 172.21.15.90:0/3114203831 learned_addr learned my addr 172.21.15.90:0/3114203831 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:36:37.672 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.669+0000 7fbd63e00640 1 --2- 172.21.15.90:0/3114203831 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbd64076950 0x7fbd641123d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:37.672 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.670+0000 7fbd68a00640 1 -- 172.21.15.90:0/3114203831 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbd64112970 msgr2=0x7fbd6410b140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:37.672 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.670+0000 7fbd68a00640 1 --2- 172.21.15.90:0/3114203831 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbd64112970 0x7fbd6410b140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:37.672 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.670+0000 7fbd68a00640 1 -- 172.21.15.90:0/3114203831 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbd64076950 msgr2=0x7fbd641123d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:37.672 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.670+0000 7fbd68a00640 1 --2- 172.21.15.90:0/3114203831 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbd64076950 0x7fbd641123d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:37.672 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.670+0000 7fbd68a00640 1 -- 172.21.15.90:0/3114203831 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd6410ba00 con 0x7fbd64076030 2024-04-02T14:36:37.673 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.670+0000 7fbd63e00640 1 --2- 172.21.15.90:0/3114203831 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbd64076950 0x7fbd641123d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:36:37.673 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.671+0000 7fbd68a00640 1 --2- 172.21.15.90:0/3114203831 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbd64076030 0x7fbd64111e90 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fbd5c00ef60 tx=0x7fbd5c00ef90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:36:37.674 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.672+0000 7fbd61600640 1 -- 172.21.15.90:0/3114203831 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd5c01b070 con 0x7fbd64076030 2024-04-02T14:36:37.674 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.672+0000 7fbd61600640 1 -- 172.21.15.90:0/3114203831 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbd5c00ba50 con 0x7fbd64076030 2024-04-02T14:36:37.674 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.672+0000 7fbd61600640 1 -- 172.21.15.90:0/3114203831 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd5c016bf0 con 0x7fbd64076030 2024-04-02T14:36:37.674 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.672+0000 7fbd6b400640 1 -- 172.21.15.90:0/3114203831 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbd6410bc90 con 0x7fbd64076030 2024-04-02T14:36:37.674 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.672+0000 7fbd6b400640 1 -- 172.21.15.90:0/3114203831 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbd6410fbf0 con 0x7fbd64076030 2024-04-02T14:36:37.683 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.680+0000 7fbd6e16a740 1 Processor -- start 2024-04-02T14:36:37.683 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.681+0000 7fbd61600640 1 -- 172.21.15.90:0/3114203831 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fbd5c01e070 con 0x7fbd64076030 2024-04-02T14:36:37.683 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.681+0000 7fbd6e16a740 1 -- start start 2024-04-02T14:36:37.684 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.682+0000 7fbd6e16a740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56347cd6feb0 0x56347cd70290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:37.685 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.682+0000 7fbd6e16a740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56347cd707d0 0x56347cd78e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:37.685 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.682+0000 7fbd6e16a740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56347cd793b0 0x56347cd7b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:37.685 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.682+0000 7fbd6e16a740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x56347ccd4d70 con 0x56347cd793b0 2024-04-02T14:36:37.685 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.682+0000 7fbd6e16a740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x56347cc7d2a0 con 0x56347cd6feb0 2024-04-02T14:36:37.685 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.682+0000 7fbd6e16a740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x56347ccd0d20 con 0x56347cd707d0 2024-04-02T14:36:37.685 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.683+0000 7fbd69400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56347cd6feb0 0x56347cd70290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:37.686 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.683+0000 7fbd68a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56347cd707d0 0x56347cd78e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:37.686 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.683+0000 7fbd68a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56347cd707d0 0x56347cd78e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:57822/0 (socket says 172.21.15.90:57822) 2024-04-02T14:36:37.686 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.683+0000 7fbd68a00640 1 -- 172.21.15.90:0/1078464403 learned_addr learned my addr 172.21.15.90:0/1078464403 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:36:37.686 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.683+0000 7fbd63e00640 1 --2- 172.21.15.90:0/1078464403 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56347cd793b0 0x56347cd7b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:37.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:36:37 smithi177 ceph-mon[36431]: pgmap v55: 97 pgs: 97 active+clean; 1.1 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 8.7 MiB/s wr, 713 op/s 2024-04-02T14:36:37.688 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.684+0000 7fbd61600640 1 --2- 172.21.15.90:0/3114203831 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fbd480a3e90 0x7fbd480a6350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:37.688 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.684+0000 7fbd69400640 1 -- 172.21.15.90:0/1078464403 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56347cd707d0 msgr2=0x56347cd78e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:36:37.688 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.684+0000 7fbd69400640 1 --2- 172.21.15.90:0/1078464403 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56347cd707d0 0x56347cd78e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:37.688 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.684+0000 7fbd69400640 1 -- 172.21.15.90:0/1078464403 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56347cd793b0 msgr2=0x56347cd7b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:36:37.688 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.684+0000 7fbd69400640 1 --2- 172.21.15.90:0/1078464403 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56347cd793b0 0x56347cd7b7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:37.688 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.684+0000 7fbd69400640 1 -- 172.21.15.90:0/1078464403 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56347cc91840 con 0x56347cd6feb0 2024-04-02T14:36:37.688 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.685+0000 7fbd68a00640 1 --2- 172.21.15.90:0/1078464403 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56347cd707d0 0x56347cd78e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:36:37.688 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.685+0000 7fbd61600640 1 -- 172.21.15.90:0/3114203831 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7fbd5c0d0720 con 0x7fbd64076030 2024-04-02T14:36:37.688 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.685+0000 7fbd69400640 1 --2- 172.21.15.90:0/3114203831 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fbd480a3e90 0x7fbd480a6350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:37.689 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:36:37 smithi090 ceph-mon[31674]: pgmap v55: 97 pgs: 97 active+clean; 1.1 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 8.7 MiB/s wr, 713 op/s 2024-04-02T14:36:37.690 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.688+0000 7fbd69400640 1 --2- 172.21.15.90:0/1078464403 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56347cd6feb0 0x56347cd70290 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fbd5400bbe0 tx=0x7fbd540107a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8c231641768a4eb9 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:36:37.691 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.689+0000 7fbd69400640 1 --2- 172.21.15.90:0/3114203831 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fbd480a3e90 0x7fbd480a6350 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fbd54018000 tx=0x7fbd5401a000 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:36:37.691 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.689+0000 7fbd59c00640 1 -- 172.21.15.90:0/1078464403 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd5401c420 con 0x56347cd6feb0 2024-04-02T14:36:37.692 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.689+0000 7fbd59c00640 1 -- 172.21.15.90:0/1078464403 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbd5401caa0 con 0x56347cd6feb0 2024-04-02T14:36:37.692 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.689+0000 7fbd59c00640 1 -- 172.21.15.90:0/1078464403 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd5401ba80 con 0x56347cd6feb0 2024-04-02T14:36:37.692 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.690+0000 7fbd6e16a740 1 -- 172.21.15.90:0/1078464403 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56347cd6feb0 msgr2=0x56347cd70290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:37.692 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.690+0000 7fbd6e16a740 1 --2- 172.21.15.90:0/1078464403 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56347cd6feb0 0x56347cd70290 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fbd5400bbe0 tx=0x7fbd540107a0 comp rx=0 tx=0).stop 2024-04-02T14:36:37.693 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.690+0000 7fbd6e16a740 1 -- 172.21.15.90:0/1078464403 shutdown_connections 2024-04-02T14:36:37.693 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.690+0000 7fbd6e16a740 1 --2- 172.21.15.90:0/1078464403 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56347cd793b0 0x56347cd7b7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:37.693 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.690+0000 7fbd6e16a740 1 --2- 172.21.15.90:0/1078464403 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56347cd707d0 0x56347cd78e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:37.693 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.690+0000 7fbd6e16a740 1 --2- 172.21.15.90:0/1078464403 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56347cd6feb0 0x56347cd70290 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:37.693 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.690+0000 7fbd6e16a740 1 -- 172.21.15.90:0/1078464403 >> 172.21.15.90:0/1078464403 conn(0x56347cd6a5e0 msgr2=0x56347cd6a9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:36:37.693 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.691+0000 7fbd6e16a740 1 -- 172.21.15.90:0/1078464403 shutdown_connections 2024-04-02T14:36:37.693 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.691+0000 7fbd6e16a740 1 -- 172.21.15.90:0/1078464403 wait complete. 2024-04-02T14:36:37.694 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.692+0000 7fbd6e16a740 1 -- start start 2024-04-02T14:36:37.694 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.692+0000 7fbd6e16a740 1 Processor -- start 2024-04-02T14:36:37.696 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.693+0000 7fbd6e16a740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56347cd6feb0 0x56347cd76420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:37.696 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.693+0000 7fbd6e16a740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56347cd707d0 0x56347cd76960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:37.696 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.693+0000 7fbd6e16a740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56347cd793b0 0x56347cd76ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:37.696 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.693+0000 7fbd6e16a740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x56347cd7c1a0 con 0x56347cd793b0 2024-04-02T14:36:37.696 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.694+0000 7fbd6e16a740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x56347cc90660 con 0x56347cd707d0 2024-04-02T14:36:37.696 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.694+0000 7fbd6e16a740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x56347cd7c320 con 0x56347cd6feb0 2024-04-02T14:36:37.696 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.694+0000 7fbd6e16a740 10 client.0 _ll_register_callbacks cb 0x56347cc90050 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:36:37.696 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.694+0000 7fbd63e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56347cd707d0 0x56347cd76960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:37.698 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.694+0000 7fbd63e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56347cd707d0 0x56347cd76960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:57390/0 (socket says 172.21.15.90:57390) 2024-04-02T14:36:37.698 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.694+0000 7fbd63e00640 1 -- 172.21.15.90:0/319193250 learned_addr learned my addr 172.21.15.90:0/319193250 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:36:37.698 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.694+0000 7fbd63e00640 1 -- 172.21.15.90:0/319193250 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56347cd6feb0 msgr2=0x56347cd76420 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-02T14:36:37.698 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.694+0000 7fbd63e00640 1 --2- 172.21.15.90:0/319193250 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56347cd6feb0 0x56347cd76420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:37.699 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.694+0000 7fbd63e00640 1 -- 172.21.15.90:0/319193250 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56347cd793b0 msgr2=0x56347cd76ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:36:37.699 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.694+0000 7fbd63e00640 1 --2- 172.21.15.90:0/319193250 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56347cd793b0 0x56347cd76ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:37.699 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.694+0000 7fbd63e00640 1 -- 172.21.15.90:0/319193250 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56347cc9fe80 con 0x56347cd707d0 2024-04-02T14:36:37.699 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.695+0000 7fbd63e00640 1 --2- 172.21.15.90:0/319193250 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56347cd707d0 0x56347cd76960 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fbd4c009780 tx=0x7fbd4c034670 comp rx=0 tx=0).ready entity=mon.2 client_cookie=313dc6ac5886692c server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:36:37.699 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.695+0000 7fbd52000640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:36:37.699 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.695+0000 7fbd52000640 1 -- 172.21.15.90:0/319193250 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd4c039040 con 0x56347cd707d0 2024-04-02T14:36:37.699 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.695+0000 7fbd52000640 1 -- 172.21.15.90:0/319193250 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7fbd4c004250 con 0x56347cd707d0 2024-04-02T14:36:37.699 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.696+0000 7fbd6e16a740 1 -- 172.21.15.90:0/319193250 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbd5401caa0 con 0x56347cd707d0 2024-04-02T14:36:37.700 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.697+0000 7fbd52000640 1 -- 172.21.15.90:0/319193250 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fbd4c034860 con 0x56347cd707d0 2024-04-02T14:36:37.700 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.697+0000 7fbd6e16a740 10 client.25910 fetch_fsmap learned FSMap version 27 2024-04-02T14:36:37.700 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.697+0000 7fbd6e16a740 1 -- 172.21.15.90:0/319193250 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x56347c9e9210 con 0x56347cd707d0 2024-04-02T14:36:37.700 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.697+0000 7fbd52000640 1 -- 172.21.15.90:0/319193250 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbd5401caa0 con 0x56347cd707d0 2024-04-02T14:36:37.700 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.698+0000 7fbd52000640 1 -- 172.21.15.90:0/319193250 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fbd4c047030 con 0x56347cd707d0 2024-04-02T14:36:37.700 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.698+0000 7fbd6e16a740 10 client.25910 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:36:37.700 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.698+0000 7fbd6e16a740 10 client.25910 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-02T14:36:37.700 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.698+0000 7fbd6e16a740 20 client.25910 populate_metadata read hostname 'smithi090' 2024-04-02T14:36:37.701 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.698+0000 7fbd6e16a740 1 --2- 172.21.15.90:0/319193250 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x56347cd89820 0x56347cd8bc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:37.701 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.698+0000 7fbd6e16a740 4 client.25910 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:36:37.701 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.698+0000 7fbd6e16a740 1 -- 172.21.15.90:0/319193250 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56347cc84cd0 con 0x56347cd89820 2024-04-02T14:36:37.701 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.698+0000 7fbd68a00640 1 --2- 172.21.15.90:0/319193250 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x56347cd89820 0x56347cd8bc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:37.701 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.699+0000 7fbd52000640 10 client.25910 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:36:37.701 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.699+0000 7fbd68a00640 1 --2- 172.21.15.90:0/319193250 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x56347cd89820 0x56347cd8bc70 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=da3d5dad7cb04d00 server_cookie=b4b4c5dc3d83cdca in_seq=0 out_seq=0 2024-04-02T14:36:37.702 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.700+0000 7fbd52000640 1 -- 172.21.15.90:0/319193250 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56347cc84cd0 con 0x56347cd89820 2024-04-02T14:36:37.702 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.700+0000 7fbd52000640 10 client.25910 handle_command_reply: tid=0 2024-04-02T14:36:37.702 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.700+0000 7fbd6e16a740 1 client.25910 shutdown 2024-04-02T14:36:37.703 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.701+0000 7fbd6e16a740 1 -- 172.21.15.90:0/319193250 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56347cd707d0 msgr2=0x56347cd76960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:37.703 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.701+0000 7fbd6e16a740 1 --2- 172.21.15.90:0/319193250 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56347cd707d0 0x56347cd76960 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fbd4c009780 tx=0x7fbd4c034670 comp rx=0 tx=0).stop 2024-04-02T14:36:37.703 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.701+0000 7fbd6e16a740 1 -- 172.21.15.90:0/319193250 shutdown_connections 2024-04-02T14:36:37.703 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.701+0000 7fbd6e16a740 1 --2- 172.21.15.90:0/319193250 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x56347cd89820 0x56347cd8bc70 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:37.704 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.701+0000 7fbd6e16a740 1 --2- 172.21.15.90:0/319193250 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56347cd793b0 0x56347cd76ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:37.704 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.701+0000 7fbd6e16a740 1 --2- 172.21.15.90:0/319193250 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56347cd707d0 0x56347cd76960 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:37.704 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.701+0000 7fbd6e16a740 1 --2- 172.21.15.90:0/319193250 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56347cd6feb0 0x56347cd76420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:37.704 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.701+0000 7fbd6e16a740 1 -- 172.21.15.90:0/319193250 >> 172.21.15.90:0/319193250 conn(0x56347cd6a5e0 msgr2=0x56347cd6c330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:36:37.704 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.701+0000 7fbd52000640 0 client.25910 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:36:37.704 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.702+0000 7fbd6e16a740 1 -- 172.21.15.90:0/319193250 shutdown_connections 2024-04-02T14:36:37.704 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.702+0000 7fbd6e16a740 1 -- 172.21.15.90:0/319193250 wait complete. 2024-04-02T14:36:37.704 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.702+0000 7fbd6e16a740 20 client.25910 trim_cache size 0 max 16384 2024-04-02T14:36:37.734 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.731+0000 7fbd6e16a740 1 Processor -- start 2024-04-02T14:36:37.735 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.731+0000 7fbd6e16a740 1 -- start start 2024-04-02T14:36:37.735 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.732+0000 7fbd6e16a740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56347cd6feb0 0x56347cd850b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:37.735 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.732+0000 7fbd6e16a740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56347cd707d0 0x56347cd76cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:37.735 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.732+0000 7fbd6e16a740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56347cd793b0 0x56347cd91580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:37.735 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.732+0000 7fbd6e16a740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fbd641bc750 con 0x56347cd6feb0 2024-04-02T14:36:37.736 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.732+0000 7fbd6e16a740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fbd641be290 con 0x56347cd707d0 2024-04-02T14:36:37.736 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.732+0000 7fbd6e16a740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fbd641bce20 con 0x56347cd793b0 2024-04-02T14:36:37.736 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.732+0000 7fbd63e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56347cd707d0 0x56347cd76cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:37.736 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.732+0000 7fbd63e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56347cd707d0 0x56347cd76cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:57394/0 (socket says 172.21.15.90:57394) 2024-04-02T14:36:37.736 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.732+0000 7fbd63e00640 1 -- 172.21.15.90:0/4241490364 learned_addr learned my addr 172.21.15.90:0/4241490364 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:36:37.736 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.732+0000 7fbd69400640 1 --2- 172.21.15.90:0/4241490364 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56347cd793b0 0x56347cd91580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:37.740 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.733+0000 7fbd69400640 1 -- 172.21.15.90:0/4241490364 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56347cd707d0 msgr2=0x56347cd76cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:36:37.740 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.733+0000 7fbd69400640 1 --2- 172.21.15.90:0/4241490364 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56347cd707d0 0x56347cd76cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:37.741 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.733+0000 7fbd69400640 1 -- 172.21.15.90:0/4241490364 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56347cd6feb0 msgr2=0x56347cd850b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:36:37.741 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.733+0000 7fbd69400640 1 --2- 172.21.15.90:0/4241490364 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56347cd6feb0 0x56347cd850b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:37.741 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.733+0000 7fbd69400640 1 -- 172.21.15.90:0/4241490364 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56347cab5620 con 0x56347cd793b0 2024-04-02T14:36:37.741 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.734+0000 7fbd69400640 1 --2- 172.21.15.90:0/4241490364 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56347cd793b0 0x56347cd91580 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fbd5400bbe0 tx=0x7fbd54010fd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8cad55ee5d163e49 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:36:37.741 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.734+0000 7fbd59200640 1 -- 172.21.15.90:0/4241490364 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd5400b790 con 0x56347cd793b0 2024-04-02T14:36:37.742 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.734+0000 7fbd59200640 1 -- 172.21.15.90:0/4241490364 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbd54033070 con 0x56347cd793b0 2024-04-02T14:36:37.742 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.734+0000 7fbd59200640 1 -- 172.21.15.90:0/4241490364 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd5401bba0 con 0x56347cd793b0 2024-04-02T14:36:37.742 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.735+0000 7fbd6e16a740 1 -- 172.21.15.90:0/4241490364 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56347cd793b0 msgr2=0x56347cd91580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:37.742 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.735+0000 7fbd6e16a740 1 --2- 172.21.15.90:0/4241490364 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56347cd793b0 0x56347cd91580 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fbd5400bbe0 tx=0x7fbd54010fd0 comp rx=0 tx=0).stop 2024-04-02T14:36:37.742 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.736+0000 7fbd6e16a740 1 -- 172.21.15.90:0/4241490364 shutdown_connections 2024-04-02T14:36:37.742 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.736+0000 7fbd6e16a740 1 --2- 172.21.15.90:0/4241490364 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56347cd793b0 0x56347cd91580 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:37.742 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.736+0000 7fbd6e16a740 1 --2- 172.21.15.90:0/4241490364 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56347cd707d0 0x56347cd76cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:37.742 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.736+0000 7fbd6e16a740 1 --2- 172.21.15.90:0/4241490364 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56347cd6feb0 0x56347cd850b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:37.742 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.736+0000 7fbd6e16a740 1 -- 172.21.15.90:0/4241490364 >> 172.21.15.90:0/4241490364 conn(0x56347cd6a5e0 msgr2=0x56347cd6c2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:36:37.743 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.736+0000 7fbd6e16a740 1 -- 172.21.15.90:0/4241490364 shutdown_connections 2024-04-02T14:36:37.743 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.737+0000 7fbd6e16a740 1 -- 172.21.15.90:0/4241490364 wait complete. 2024-04-02T14:36:37.743 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.737+0000 7fbd6e16a740 1 -- start start 2024-04-02T14:36:37.743 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.738+0000 7fbd6e16a740 1 Processor -- start 2024-04-02T14:36:37.746 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.739+0000 7fbd6e16a740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56347cd6feb0 0x56347cd815c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:37.747 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.739+0000 7fbd6e16a740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56347cd707d0 0x56347cd81b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:37.747 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.739+0000 7fbd6e16a740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56347cd793b0 0x56347cdeadd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:37.747 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.739+0000 7fbd6e16a740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fbd641bdbc0 con 0x56347cd6feb0 2024-04-02T14:36:37.747 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.740+0000 7fbd69400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56347cd793b0 0x56347cdeadd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:37.747 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.740+0000 7fbd69400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56347cd793b0 0x56347cdeadd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:57840/0 (socket says 172.21.15.90:57840) 2024-04-02T14:36:37.747 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.740+0000 7fbd69400640 1 -- 172.21.15.90:0/2120991455 learned_addr learned my addr 172.21.15.90:0/2120991455 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:36:37.748 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.740+0000 7fbd63e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56347cd707d0 0x56347cd81b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:37.748 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.740+0000 7fbd6e16a740 1 -- 172.21.15.90:0/2120991455 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fbd641bd4f0 con 0x56347cd707d0 2024-04-02T14:36:37.748 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.740+0000 7fbd6e16a740 1 -- 172.21.15.90:0/2120991455 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fbd641ba730 con 0x56347cd793b0 2024-04-02T14:36:37.748 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.740+0000 7fbd68a00640 1 --2- 172.21.15.90:0/2120991455 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56347cd6feb0 0x56347cd815c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:37.748 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.740+0000 7fbd6e16a740 10 client.0 _ll_register_callbacks cb 0x7fbd4c035e70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:36:37.749 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.741+0000 7fbd69400640 1 -- 172.21.15.90:0/2120991455 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56347cd707d0 msgr2=0x56347cd81b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:36:37.749 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.741+0000 7fbd69400640 1 --2- 172.21.15.90:0/2120991455 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56347cd707d0 0x56347cd81b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:37.749 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.741+0000 7fbd69400640 1 -- 172.21.15.90:0/2120991455 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56347cd6feb0 msgr2=0x56347cd815c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:36:37.749 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.741+0000 7fbd69400640 1 --2- 172.21.15.90:0/2120991455 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56347cd6feb0 0x56347cd815c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:37.749 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.741+0000 7fbd69400640 1 -- 172.21.15.90:0/2120991455 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56347cdeb600 con 0x56347cd793b0 2024-04-02T14:36:37.749 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.741+0000 7fbd63e00640 1 --2- 172.21.15.90:0/2120991455 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56347cd707d0 0x56347cd81b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:36:37.750 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.741+0000 7fbd68a00640 1 --2- 172.21.15.90:0/2120991455 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56347cd6feb0 0x56347cd815c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:36:37.750 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.742+0000 7fbd69400640 1 --2- 172.21.15.90:0/2120991455 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56347cd793b0 0x56347cdeadd0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fbd5401b670 tx=0x7fbd5401b6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=96000a7d6163a62f server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:36:37.750 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.742+0000 7fbd52000640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:36:37.750 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.742+0000 7fbd52000640 1 -- 172.21.15.90:0/2120991455 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd54005310 con 0x56347cd793b0 2024-04-02T14:36:37.750 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.742+0000 7fbd52000640 1 -- 172.21.15.90:0/2120991455 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7fbd540054b0 con 0x56347cd793b0 2024-04-02T14:36:37.751 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.742+0000 7fbd6e16a740 1 -- 172.21.15.90:0/2120991455 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbd54033070 con 0x56347cd793b0 2024-04-02T14:36:37.751 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.743+0000 7fbd52000640 1 -- 172.21.15.90:0/2120991455 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fbd54045050 con 0x56347cd793b0 2024-04-02T14:36:37.751 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.743+0000 7fbd6e16a740 10 client.25837 fetch_fsmap learned FSMap version 27 2024-04-02T14:36:37.751 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.743+0000 7fbd6e16a740 1 -- 172.21.15.90:0/2120991455 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x56347cd73420 con 0x56347cd793b0 2024-04-02T14:36:37.751 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.743+0000 7fbd52000640 1 -- 172.21.15.90:0/2120991455 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbd54033070 con 0x56347cd793b0 2024-04-02T14:36:37.751 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.744+0000 7fbd52000640 1 -- 172.21.15.90:0/2120991455 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fbd54004a60 con 0x56347cd793b0 2024-04-02T14:36:37.751 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.744+0000 7fbd6e16a740 10 client.25837 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:36:37.752 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.744+0000 7fbd6e16a740 10 client.25837 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-02T14:36:37.752 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.744+0000 7fbd6e16a740 20 client.25837 populate_metadata read hostname 'smithi090' 2024-04-02T14:36:37.752 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.744+0000 7fbd6e16a740 1 --2- 172.21.15.90:0/2120991455 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x56347ce00e00 0x56347ce031f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:37.752 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.744+0000 7fbd6e16a740 4 client.25837 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-02T14:36:37.752 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.744+0000 7fbd6e16a740 1 -- 172.21.15.90:0/2120991455 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56347cce22b0 con 0x56347ce00e00 2024-04-02T14:36:37.752 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.744+0000 7fbd68a00640 1 --2- 172.21.15.90:0/2120991455 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x56347ce00e00 0x56347ce031f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:37.753 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.745+0000 7fbd52000640 10 client.25837 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:36:37.754 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.746+0000 7fbd68a00640 1 --2- 172.21.15.90:0/2120991455 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x56347ce00e00 0x56347ce031f0 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c8790aff63a14a7b server_cookie=8e34f429b4422de8 in_seq=0 out_seq=0 2024-04-02T14:36:37.764 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.762+0000 7fbd52000640 1 -- 172.21.15.90:0/2120991455 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56347cce22b0 con 0x56347ce00e00 2024-04-02T14:36:37.764 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.762+0000 7fbd52000640 10 client.25837 handle_command_reply: tid=0 2024-04-02T14:36:37.765 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.762+0000 7fbd6e16a740 1 client.25837 shutdown 2024-04-02T14:36:37.765 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.762+0000 7fbd6e16a740 1 -- 172.21.15.90:0/2120991455 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56347cd793b0 msgr2=0x56347cdeadd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:37.765 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.762+0000 7fbd6e16a740 1 --2- 172.21.15.90:0/2120991455 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56347cd793b0 0x56347cdeadd0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fbd5401b670 tx=0x7fbd5401b6a0 comp rx=0 tx=0).stop 2024-04-02T14:36:37.765 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.763+0000 7fbd6e16a740 1 -- 172.21.15.90:0/2120991455 shutdown_connections 2024-04-02T14:36:37.765 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.763+0000 7fbd6e16a740 1 --2- 172.21.15.90:0/2120991455 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x56347ce00e00 0x56347ce031f0 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:37.765 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.763+0000 7fbd6e16a740 1 --2- 172.21.15.90:0/2120991455 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56347cd793b0 0x56347cdeadd0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:37.765 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.763+0000 7fbd6e16a740 1 --2- 172.21.15.90:0/2120991455 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56347cd707d0 0x56347cd81b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:37.766 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.763+0000 7fbd6e16a740 1 --2- 172.21.15.90:0/2120991455 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56347cd6feb0 0x56347cd815c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:37.766 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.763+0000 7fbd52000640 0 client.25837 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:36:37.766 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.763+0000 7fbd6e16a740 1 -- 172.21.15.90:0/2120991455 >> 172.21.15.90:0/2120991455 conn(0x56347cd6a5e0 msgr2=0x56347cd68db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:36:37.766 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.763+0000 7fbd6e16a740 1 -- 172.21.15.90:0/2120991455 shutdown_connections 2024-04-02T14:36:37.767 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.764+0000 7fbd6e16a740 1 -- 172.21.15.90:0/2120991455 wait complete. 2024-04-02T14:36:37.767 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.764+0000 7fbd6e16a740 20 client.25837 trim_cache size 0 max 16384 2024-04-02T14:36:37.767 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:{ 2024-04-02T14:36:37.767 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "message": "", 2024-04-02T14:36:37.767 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "return_code": 0 2024-04-02T14:36:37.767 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:} 2024-04-02T14:36:37.768 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.765+0000 7fbd6b400640 1 -- 172.21.15.90:0/3114203831 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fbd480a3e90 msgr2=0x7fbd480a6350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:37.768 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.765+0000 7fbd6b400640 1 --2- 172.21.15.90:0/3114203831 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fbd480a3e90 0x7fbd480a6350 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fbd54018000 tx=0x7fbd5401a000 comp rx=0 tx=0).stop 2024-04-02T14:36:37.768 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.765+0000 7fbd6b400640 1 -- 172.21.15.90:0/3114203831 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbd64076030 msgr2=0x7fbd64111e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:37.768 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.765+0000 7fbd6b400640 1 --2- 172.21.15.90:0/3114203831 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbd64076030 0x7fbd64111e90 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fbd5c00ef60 tx=0x7fbd5c00ef90 comp rx=0 tx=0).stop 2024-04-02T14:36:37.768 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.766+0000 7fbd6b400640 1 -- 172.21.15.90:0/3114203831 shutdown_connections 2024-04-02T14:36:37.768 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.766+0000 7fbd6b400640 1 --2- 172.21.15.90:0/3114203831 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fbd480a3e90 0x7fbd480a6350 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:37.768 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.766+0000 7fbd6b400640 1 --2- 172.21.15.90:0/3114203831 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbd64112970 0x7fbd6410b140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:37.769 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.766+0000 7fbd6b400640 1 --2- 172.21.15.90:0/3114203831 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbd64076950 0x7fbd641123d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:37.769 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.766+0000 7fbd6b400640 1 --2- 172.21.15.90:0/3114203831 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbd64076030 0x7fbd64111e90 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:37.769 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.766+0000 7fbd6b400640 1 -- 172.21.15.90:0/3114203831 >> 172.21.15.90:0/3114203831 conn(0x7fbd640ff7c0 msgr2=0x7fbd64101bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:36:37.769 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.766+0000 7fbd6b400640 1 -- 172.21.15.90:0/3114203831 shutdown_connections 2024-04-02T14:36:37.769 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:37.766+0000 7fbd6b400640 1 -- 172.21.15.90:0/3114203831 wait complete. 2024-04-02T14:36:37.860 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:36:37 smithi092 ceph-mon[36497]: pgmap v55: 97 pgs: 97 active+clean; 1.1 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 8.7 MiB/s wr, 713 op/s 2024-04-02T14:36:39.686 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:36:39 smithi177 ceph-mon[36431]: pgmap v56: 97 pgs: 97 active+clean; 1.1 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 12 MiB/s wr, 939 op/s 2024-04-02T14:36:39.688 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:36:39 smithi090 ceph-mon[31674]: pgmap v56: 97 pgs: 97 active+clean; 1.1 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 12 MiB/s wr, 939 op/s 2024-04-02T14:36:39.860 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:36:39 smithi092 ceph-mon[36497]: pgmap v56: 97 pgs: 97 active+clean; 1.1 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 12 MiB/s wr, 939 op/s 2024-04-02T14:36:41.686 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:36:41 smithi177 ceph-mon[36431]: pgmap v57: 97 pgs: 97 active+clean; 1.2 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 10 MiB/s wr, 810 op/s 2024-04-02T14:36:41.688 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:36:41 smithi090 ceph-mon[31674]: pgmap v57: 97 pgs: 97 active+clean; 1.2 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 10 MiB/s wr, 810 op/s 2024-04-02T14:36:41.831 INFO:teuthology.orchestra.run.smithi090.stderr:progress: 70.0 s, 3993.3 tps, lat 24.999 ms stddev 29.210 2024-04-02T14:36:41.861 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:36:41 smithi092 ceph-mon[36497]: pgmap v57: 97 pgs: 97 active+clean; 1.2 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 10 MiB/s wr, 810 op/s 2024-04-02T14:36:42.532 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-02T14:36:42.532 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-02T14:36:42.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.817+0000 7fbdad200640 1 -- 172.21.15.90:0/3533507533 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbda8076030 msgr2=0x7fbda8076410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:42.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.817+0000 7fbdad200640 1 --2- 172.21.15.90:0/3533507533 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbda8076030 0x7fbda8076410 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fbd98009930 tx=0x7fbd9802bc30 comp rx=0 tx=0).stop 2024-04-02T14:36:42.820 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.817+0000 7fbdad200640 1 -- 172.21.15.90:0/3533507533 shutdown_connections 2024-04-02T14:36:42.820 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.817+0000 7fbdad200640 1 --2- 172.21.15.90:0/3533507533 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbda8112940 0x7fbda8114d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:42.820 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.817+0000 7fbdad200640 1 --2- 172.21.15.90:0/3533507533 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbda8076950 0x7fbda8112400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:42.820 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.817+0000 7fbdad200640 1 --2- 172.21.15.90:0/3533507533 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbda8076030 0x7fbda8076410 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:42.820 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.817+0000 7fbdad200640 1 -- 172.21.15.90:0/3533507533 >> 172.21.15.90:0/3533507533 conn(0x7fbda80ff770 msgr2=0x7fbda8101b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:36:42.820 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.818+0000 7fbdad200640 1 -- 172.21.15.90:0/3533507533 shutdown_connections 2024-04-02T14:36:42.821 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.818+0000 7fbdad200640 1 -- 172.21.15.90:0/3533507533 wait complete. 2024-04-02T14:36:42.821 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.819+0000 7fbdad200640 1 Processor -- start 2024-04-02T14:36:42.821 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.819+0000 7fbdad200640 1 -- start start 2024-04-02T14:36:42.822 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.820+0000 7fbdad200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbda8076030 0x7fbda81a46c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:42.822 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.820+0000 7fbdad200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbda8076950 0x7fbda81a4c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:42.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.820+0000 7fbdad200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbda8112940 0x7fbda819e8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:42.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.820+0000 7fbdad200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fbda81184e0 con 0x7fbda8112940 2024-04-02T14:36:42.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.820+0000 7fbdad200640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fbda8118360 con 0x7fbda8076030 2024-04-02T14:36:42.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.820+0000 7fbdad200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fbda8118660 con 0x7fbda8076950 2024-04-02T14:36:42.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.820+0000 7fbda5e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbda8076030 0x7fbda81a46c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:42.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.821+0000 7fbda5e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbda8076030 0x7fbda81a46c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:57424/0 (socket says 172.21.15.90:57424) 2024-04-02T14:36:42.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.821+0000 7fbda5e00640 1 -- 172.21.15.90:0/4045382748 learned_addr learned my addr 172.21.15.90:0/4045382748 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:36:42.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.821+0000 7fbda5400640 1 --2- 172.21.15.90:0/4045382748 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbda8076950 0x7fbda81a4c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:42.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.821+0000 7fbda6800640 1 --2- 172.21.15.90:0/4045382748 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbda8112940 0x7fbda819e8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:42.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.821+0000 7fbda5e00640 1 -- 172.21.15.90:0/4045382748 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbda8076950 msgr2=0x7fbda81a4c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:42.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.821+0000 7fbda5e00640 1 --2- 172.21.15.90:0/4045382748 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbda8076950 0x7fbda81a4c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:42.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.821+0000 7fbda5e00640 1 -- 172.21.15.90:0/4045382748 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbda8112940 msgr2=0x7fbda819e8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:42.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.821+0000 7fbda5e00640 1 --2- 172.21.15.90:0/4045382748 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbda8112940 0x7fbda819e8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:42.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.821+0000 7fbda5e00640 1 -- 172.21.15.90:0/4045382748 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbda819f040 con 0x7fbda8076030 2024-04-02T14:36:42.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.822+0000 7fbda5e00640 1 --2- 172.21.15.90:0/4045382748 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbda8076030 0x7fbda81a46c0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fbd98036040 tx=0x7fbd98004040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:36:42.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.822+0000 7fbd9ea00640 1 -- 172.21.15.90:0/4045382748 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd98043020 con 0x7fbda8076030 2024-04-02T14:36:42.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.822+0000 7fbd9ea00640 1 -- 172.21.15.90:0/4045382748 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbd98004470 con 0x7fbda8076030 2024-04-02T14:36:42.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.822+0000 7fbdad200640 1 -- 172.21.15.90:0/4045382748 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbda819f2d0 con 0x7fbda8076030 2024-04-02T14:36:42.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.822+0000 7fbd9ea00640 1 -- 172.21.15.90:0/4045382748 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd9803e450 con 0x7fbda8076030 2024-04-02T14:36:42.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.823+0000 7fbdad200640 1 -- 172.21.15.90:0/4045382748 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbda81ab580 con 0x7fbda8076030 2024-04-02T14:36:42.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.829+0000 7fbd9ea00640 1 -- 172.21.15.90:0/4045382748 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fbd98046020 con 0x7fbda8076030 2024-04-02T14:36:42.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.830+0000 7fbdaff6d740 1 Processor -- start 2024-04-02T14:36:42.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.831+0000 7fbd9ea00640 1 --2- 172.21.15.90:0/4045382748 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fbd8c0a3e70 0x7fbd8c0a6330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:42.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.831+0000 7fbd9ea00640 1 -- 172.21.15.90:0/4045382748 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7fbd980f2a80 con 0x7fbda8076030 2024-04-02T14:36:42.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.831+0000 7fbdaff6d740 1 -- start start 2024-04-02T14:36:42.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.832+0000 7fbdaff6d740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f210e507b0 0x55f210e50b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:42.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.832+0000 7fbdaff6d740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f210e510d0 0x55f210e59770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:42.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.832+0000 7fbdaff6d740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f210e59cb0 0x55f210e5c0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:42.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.832+0000 7fbdaff6d740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55f210db4770 con 0x55f210e507b0 2024-04-02T14:36:42.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.832+0000 7fbdaff6d740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55f210d5c700 con 0x55f210e59cb0 2024-04-02T14:36:42.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.832+0000 7fbdaff6d740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55f210db0720 con 0x55f210e510d0 2024-04-02T14:36:42.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.832+0000 7fbda6800640 1 --2- 172.21.15.90:0/4045382748 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fbd8c0a3e70 0x7fbd8c0a6330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:42.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.833+0000 7fbda5e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f210e507b0 0x55f210e50b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:42.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.833+0000 7fbda5e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f210e507b0 0x55f210e50b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:54030/0 (socket says 172.21.15.90:54030) 2024-04-02T14:36:42.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.832+0000 7fbda6800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f210e59cb0 0x55f210e5c0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:42.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.833+0000 7fbda5e00640 1 -- 172.21.15.90:0/350303608 learned_addr learned my addr 172.21.15.90:0/350303608 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:36:42.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.833+0000 7fbda5400640 1 --2- 172.21.15.90:0/350303608 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f210e510d0 0x55f210e59770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:42.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.833+0000 7fbda5e00640 1 -- 172.21.15.90:0/350303608 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f210e510d0 msgr2=0x55f210e59770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:36:42.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.833+0000 7fbda5e00640 1 --2- 172.21.15.90:0/350303608 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f210e510d0 0x55f210e59770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:42.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.833+0000 7fbda5e00640 1 -- 172.21.15.90:0/350303608 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f210e59cb0 msgr2=0x55f210e5c0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:36:42.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.833+0000 7fbda5e00640 1 --2- 172.21.15.90:0/350303608 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f210e59cb0 0x55f210e5c0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:42.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.834+0000 7fbda5e00640 1 -- 172.21.15.90:0/350303608 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f210b927d0 con 0x55f210e507b0 2024-04-02T14:36:42.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.835+0000 7fbda5e00640 1 --2- 172.21.15.90:0/350303608 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f210e507b0 0x55f210e50b90 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fbd980f0350 tx=0x7fbd980baf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ddd6403ac2db7ea server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:36:42.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.835+0000 7fbda6800640 1 --2- 172.21.15.90:0/4045382748 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fbd8c0a3e70 0x7fbd8c0a6330 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fbda806afb0 tx=0x7fbda0009d90 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:36:42.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.836+0000 7fbd93400640 1 -- 172.21.15.90:0/350303608 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd980d8070 con 0x55f210e507b0 2024-04-02T14:36:42.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.836+0000 7fbd93400640 1 -- 172.21.15.90:0/350303608 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbd980beb40 con 0x55f210e507b0 2024-04-02T14:36:42.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.836+0000 7fbd93400640 1 -- 172.21.15.90:0/350303608 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd980bb1b0 con 0x55f210e507b0 2024-04-02T14:36:42.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.837+0000 7fbdaff6d740 1 -- 172.21.15.90:0/350303608 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f210e507b0 msgr2=0x55f210e50b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:42.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.837+0000 7fbdaff6d740 1 --2- 172.21.15.90:0/350303608 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f210e507b0 0x55f210e50b90 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fbd980f0350 tx=0x7fbd980baf10 comp rx=0 tx=0).stop 2024-04-02T14:36:42.840 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.837+0000 7fbdaff6d740 1 -- 172.21.15.90:0/350303608 shutdown_connections 2024-04-02T14:36:42.840 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.837+0000 7fbdaff6d740 1 --2- 172.21.15.90:0/350303608 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f210e59cb0 0x55f210e5c0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:42.840 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.837+0000 7fbdaff6d740 1 --2- 172.21.15.90:0/350303608 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f210e510d0 0x55f210e59770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:42.840 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.837+0000 7fbdaff6d740 1 --2- 172.21.15.90:0/350303608 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f210e507b0 0x55f210e50b90 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:42.840 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.837+0000 7fbdaff6d740 1 -- 172.21.15.90:0/350303608 >> 172.21.15.90:0/350303608 conn(0x55f210e4aee0 msgr2=0x55f210e4b2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:36:42.841 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.837+0000 7fbdaff6d740 1 -- 172.21.15.90:0/350303608 shutdown_connections 2024-04-02T14:36:42.841 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.837+0000 7fbdaff6d740 1 -- 172.21.15.90:0/350303608 wait complete. 2024-04-02T14:36:42.841 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.838+0000 7fbdaff6d740 1 -- start start 2024-04-02T14:36:42.841 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.839+0000 7fbdaff6d740 1 Processor -- start 2024-04-02T14:36:42.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.840+0000 7fbdaff6d740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f210e507b0 0x55f210e56d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:42.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.840+0000 7fbdaff6d740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f210e510d0 0x55f210e57260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:42.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.841+0000 7fbdaff6d740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f210e59cb0 0x55f210e577a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:42.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.841+0000 7fbdaff6d740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55f210e5c860 con 0x55f210e59cb0 2024-04-02T14:36:42.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.841+0000 7fbdaff6d740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55f210d6fac0 con 0x55f210e510d0 2024-04-02T14:36:42.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.841+0000 7fbdaff6d740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55f210e5c9e0 con 0x55f210e507b0 2024-04-02T14:36:42.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.841+0000 7fbda5400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f210e510d0 0x55f210e57260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:42.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.841+0000 7fbda5400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f210e510d0 0x55f210e57260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:57440/0 (socket says 172.21.15.90:57440) 2024-04-02T14:36:42.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.841+0000 7fbda5400640 1 -- 172.21.15.90:0/22437152 learned_addr learned my addr 172.21.15.90:0/22437152 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:36:42.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.841+0000 7fbdaff6d740 10 client.0 _ll_register_callbacks cb 0x55f210d72180 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:36:42.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.842+0000 7fbda5e00640 1 --2- 172.21.15.90:0/22437152 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f210e507b0 0x55f210e56d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:42.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.842+0000 7fbda5400640 1 -- 172.21.15.90:0/22437152 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f210e507b0 msgr2=0x55f210e56d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:36:42.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.842+0000 7fbda6800640 1 --2- 172.21.15.90:0/22437152 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f210e59cb0 0x55f210e577a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:42.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.842+0000 7fbda5400640 1 --2- 172.21.15.90:0/22437152 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f210e507b0 0x55f210e56d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:42.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.842+0000 7fbda5400640 1 -- 172.21.15.90:0/22437152 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f210e59cb0 msgr2=0x55f210e577a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:36:42.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.842+0000 7fbda5400640 1 --2- 172.21.15.90:0/22437152 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f210e59cb0 0x55f210e577a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:42.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.842+0000 7fbda5400640 1 -- 172.21.15.90:0/22437152 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f210ac83f0 con 0x55f210e510d0 2024-04-02T14:36:42.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.842+0000 7fbda5e00640 1 --2- 172.21.15.90:0/22437152 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f210e507b0 0x55f210e56d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:36:42.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.843+0000 7fbda6800640 1 --2- 172.21.15.90:0/22437152 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f210e59cb0 0x55f210e577a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:36:42.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.844+0000 7fbda5400640 1 --2- 172.21.15.90:0/22437152 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f210e510d0 0x55f210e57260 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fbd94013740 tx=0x7fbd94013c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=da613c3961f0459d server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:36:42.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.844+0000 7fbd87e00640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:36:42.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.844+0000 7fbd87e00640 1 -- 172.21.15.90:0/22437152 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd94020060 con 0x55f210e510d0 2024-04-02T14:36:42.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.844+0000 7fbd87e00640 1 -- 172.21.15.90:0/22437152 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7fbd94011dd0 con 0x55f210e510d0 2024-04-02T14:36:42.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.844+0000 7fbdaff6d740 1 -- 172.21.15.90:0/22437152 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbd980beb40 con 0x55f210e510d0 2024-04-02T14:36:42.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.845+0000 7fbd87e00640 1 -- 172.21.15.90:0/22437152 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fbd9400f450 con 0x55f210e510d0 2024-04-02T14:36:42.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.845+0000 7fbdaff6d740 10 client.25928 fetch_fsmap learned FSMap version 27 2024-04-02T14:36:42.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.845+0000 7fbdaff6d740 1 -- 172.21.15.90:0/22437152 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55f210a880f0 con 0x55f210e510d0 2024-04-02T14:36:42.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.846+0000 7fbd87e00640 1 -- 172.21.15.90:0/22437152 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbd980beb40 con 0x55f210e510d0 2024-04-02T14:36:42.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.846+0000 7fbd87e00640 1 -- 172.21.15.90:0/22437152 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fbd9401a030 con 0x55f210e510d0 2024-04-02T14:36:42.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.847+0000 7fbdaff6d740 10 client.25928 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:36:42.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.847+0000 7fbdaff6d740 10 client.25928 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:36:42.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.847+0000 7fbdaff6d740 20 client.25928 populate_metadata read hostname 'smithi090' 2024-04-02T14:36:42.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.847+0000 7fbdaff6d740 1 --2- 172.21.15.90:0/22437152 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55f210e699b0 0x55f210e6be70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:42.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.847+0000 7fbdaff6d740 4 client.25928 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:36:42.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.847+0000 7fbdaff6d740 1 -- 172.21.15.90:0/22437152 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f210d55630 con 0x55f210e699b0 2024-04-02T14:36:42.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.847+0000 7fbda5e00640 1 --2- 172.21.15.90:0/22437152 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55f210e699b0 0x55f210e6be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:42.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.848+0000 7fbd87e00640 10 client.25928 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:36:42.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.849+0000 7fbda5e00640 1 --2- 172.21.15.90:0/22437152 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55f210e699b0 0x55f210e6be70 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=be2ef54b011b18f3 server_cookie=b68ffcff7418a8cf in_seq=0 out_seq=0 2024-04-02T14:36:42.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.851+0000 7fbd87e00640 1 -- 172.21.15.90:0/22437152 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f210d55630 con 0x55f210e699b0 2024-04-02T14:36:42.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.851+0000 7fbd87e00640 10 client.25928 handle_command_reply: tid=0 2024-04-02T14:36:42.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.851+0000 7fbdaff6d740 1 client.25928 shutdown 2024-04-02T14:36:42.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.852+0000 7fbdaff6d740 1 -- 172.21.15.90:0/22437152 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f210e510d0 msgr2=0x55f210e57260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:42.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.852+0000 7fbdaff6d740 1 --2- 172.21.15.90:0/22437152 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f210e510d0 0x55f210e57260 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fbd94013740 tx=0x7fbd94013c00 comp rx=0 tx=0).stop 2024-04-02T14:36:42.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.853+0000 7fbdaff6d740 1 -- 172.21.15.90:0/22437152 shutdown_connections 2024-04-02T14:36:42.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.853+0000 7fbdaff6d740 1 --2- 172.21.15.90:0/22437152 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55f210e699b0 0x55f210e6be70 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:42.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.853+0000 7fbdaff6d740 1 --2- 172.21.15.90:0/22437152 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f210e59cb0 0x55f210e577a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:42.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.853+0000 7fbdaff6d740 1 --2- 172.21.15.90:0/22437152 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f210e510d0 0x55f210e57260 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:42.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.853+0000 7fbdaff6d740 1 --2- 172.21.15.90:0/22437152 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f210e507b0 0x55f210e56d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:42.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.853+0000 7fbdaff6d740 1 -- 172.21.15.90:0/22437152 >> 172.21.15.90:0/22437152 conn(0x55f210e4aee0 msgr2=0x55f210e4cc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:36:42.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.853+0000 7fbd87e00640 0 client.25928 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:36:42.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.853+0000 7fbdaff6d740 1 -- 172.21.15.90:0/22437152 shutdown_connections 2024-04-02T14:36:42.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.853+0000 7fbdaff6d740 1 -- 172.21.15.90:0/22437152 wait complete. 2024-04-02T14:36:42.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.854+0000 7fbdaff6d740 20 client.25928 trim_cache size 0 max 16384 2024-04-02T14:36:42.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.884+0000 7fbdaff6d740 1 Processor -- start 2024-04-02T14:36:42.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.884+0000 7fbdaff6d740 1 -- start start 2024-04-02T14:36:42.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.884+0000 7fbdaff6d740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f210e507b0 0x55f210e59450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:42.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.884+0000 7fbdaff6d740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f210e510d0 0x55f210e65690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:42.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.884+0000 7fbdaff6d740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f210e59cb0 0x55f210e71780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:42.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.884+0000 7fbdaff6d740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fbda81b3e80 con 0x55f210e507b0 2024-04-02T14:36:42.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.884+0000 7fbdaff6d740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fbda81b59c0 con 0x55f210e59cb0 2024-04-02T14:36:42.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.884+0000 7fbdaff6d740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fbda81b4550 con 0x55f210e510d0 2024-04-02T14:36:42.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.885+0000 7fbda5400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f210e510d0 0x55f210e65690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:42.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.885+0000 7fbda5e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f210e507b0 0x55f210e59450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:42.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.885+0000 7fbda5400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f210e510d0 0x55f210e65690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:57886/0 (socket says 172.21.15.90:57886) 2024-04-02T14:36:42.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.885+0000 7fbda5400640 1 -- 172.21.15.90:0/1954308291 learned_addr learned my addr 172.21.15.90:0/1954308291 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:36:42.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.885+0000 7fbda6800640 1 --2- 172.21.15.90:0/1954308291 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f210e59cb0 0x55f210e71780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:42.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.886+0000 7fbda5e00640 1 -- 172.21.15.90:0/1954308291 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f210e510d0 msgr2=0x55f210e65690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:36:42.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.886+0000 7fbda5e00640 1 --2- 172.21.15.90:0/1954308291 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f210e510d0 0x55f210e65690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:42.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.886+0000 7fbda5e00640 1 -- 172.21.15.90:0/1954308291 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f210e59cb0 msgr2=0x55f210e71780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:36:42.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.886+0000 7fbda5e00640 1 --2- 172.21.15.90:0/1954308291 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f210e59cb0 0x55f210e71780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:42.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.886+0000 7fbda5e00640 1 -- 172.21.15.90:0/1954308291 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f210e4aa10 con 0x55f210e507b0 2024-04-02T14:36:42.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.886+0000 7fbda6800640 1 --2- 172.21.15.90:0/1954308291 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f210e59cb0 0x55f210e71780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:36:42.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.886+0000 7fbda5e00640 1 --2- 172.21.15.90:0/1954308291 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f210e507b0 0x55f210e59450 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fbd980c7970 tx=0x7fbd98005a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3fe0708c50b0d8b server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:36:42.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.887+0000 7fbd92a00640 1 -- 172.21.15.90:0/1954308291 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd980d8040 con 0x55f210e507b0 2024-04-02T14:36:42.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.887+0000 7fbd92a00640 1 -- 172.21.15.90:0/1954308291 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbd980beb40 con 0x55f210e507b0 2024-04-02T14:36:42.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.887+0000 7fbd92a00640 1 -- 172.21.15.90:0/1954308291 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd980bb1b0 con 0x55f210e507b0 2024-04-02T14:36:42.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.889+0000 7fbdaff6d740 1 -- 172.21.15.90:0/1954308291 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f210e507b0 msgr2=0x55f210e59450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:42.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.889+0000 7fbdaff6d740 1 --2- 172.21.15.90:0/1954308291 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f210e507b0 0x55f210e59450 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fbd980c7970 tx=0x7fbd98005a40 comp rx=0 tx=0).stop 2024-04-02T14:36:42.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.889+0000 7fbdaff6d740 1 -- 172.21.15.90:0/1954308291 shutdown_connections 2024-04-02T14:36:42.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.889+0000 7fbdaff6d740 1 --2- 172.21.15.90:0/1954308291 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f210e59cb0 0x55f210e71780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:42.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.889+0000 7fbdaff6d740 1 --2- 172.21.15.90:0/1954308291 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f210e510d0 0x55f210e65690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:42.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.889+0000 7fbdaff6d740 1 --2- 172.21.15.90:0/1954308291 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f210e507b0 0x55f210e59450 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:42.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.889+0000 7fbdaff6d740 1 -- 172.21.15.90:0/1954308291 >> 172.21.15.90:0/1954308291 conn(0x55f210e4aee0 msgr2=0x55f210e555f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:36:42.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.890+0000 7fbdaff6d740 1 -- 172.21.15.90:0/1954308291 shutdown_connections 2024-04-02T14:36:42.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.890+0000 7fbdaff6d740 1 -- 172.21.15.90:0/1954308291 wait complete. 2024-04-02T14:36:42.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.891+0000 7fbdaff6d740 1 -- start start 2024-04-02T14:36:42.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.891+0000 7fbdaff6d740 1 Processor -- start 2024-04-02T14:36:42.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.892+0000 7fbdaff6d740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f210e507b0 0x55f210e61cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:42.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.892+0000 7fbdaff6d740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f210e510d0 0x55f210e5d180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:42.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.892+0000 7fbdaff6d740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f210e59cb0 0x55f210e5d6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:42.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.892+0000 7fbdaff6d740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fbda81b52f0 con 0x55f210e510d0 2024-04-02T14:36:42.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.892+0000 7fbdaff6d740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fbda81b4c20 con 0x55f210e59cb0 2024-04-02T14:36:42.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.892+0000 7fbdaff6d740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fbda81b1e60 con 0x55f210e507b0 2024-04-02T14:36:42.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.892+0000 7fbdaff6d740 10 client.0 _ll_register_callbacks cb 0x7fbd94009650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:36:42.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.893+0000 7fbda5400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f210e510d0 0x55f210e5d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:42.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.893+0000 7fbda5e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f210e507b0 0x55f210e61cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:42.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.894+0000 7fbda6800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f210e59cb0 0x55f210e5d6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:42.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.894+0000 7fbda6800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f210e59cb0 0x55f210e5d6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:57458/0 (socket says 172.21.15.90:57458) 2024-04-02T14:36:42.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.894+0000 7fbda6800640 1 -- 172.21.15.90:0/2647021788 learned_addr learned my addr 172.21.15.90:0/2647021788 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:36:42.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.895+0000 7fbda5400640 1 -- 172.21.15.90:0/2647021788 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f210e507b0 msgr2=0x55f210e61cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:36:42.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.895+0000 7fbda5400640 1 --2- 172.21.15.90:0/2647021788 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f210e507b0 0x55f210e61cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:42.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.895+0000 7fbda5400640 1 -- 172.21.15.90:0/2647021788 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f210e59cb0 msgr2=0x55f210e5d6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:36:42.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.895+0000 7fbda5400640 1 --2- 172.21.15.90:0/2647021788 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f210e59cb0 0x55f210e5d6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:42.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.895+0000 7fbda5400640 1 -- 172.21.15.90:0/2647021788 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f210e5dc30 con 0x55f210e510d0 2024-04-02T14:36:42.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.895+0000 7fbda6800640 1 --2- 172.21.15.90:0/2647021788 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f210e59cb0 0x55f210e5d6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:36:42.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.895+0000 7fbda5400640 1 --2- 172.21.15.90:0/2647021788 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f210e510d0 0x55f210e5d180 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fbd9400e920 tx=0x7fbd9400e430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ec08ffb4d59af2c server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:36:42.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.896+0000 7fbd87e00640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-02T14:36:42.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.896+0000 7fbd87e00640 1 -- 172.21.15.90:0/2647021788 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd9402a080 con 0x55f210e510d0 2024-04-02T14:36:42.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.896+0000 7fbd87e00640 1 -- 172.21.15.90:0/2647021788 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7fbd9400ed10 con 0x55f210e510d0 2024-04-02T14:36:42.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.896+0000 7fbdaff6d740 1 -- 172.21.15.90:0/2647021788 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbd980beb40 con 0x55f210e510d0 2024-04-02T14:36:42.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.897+0000 7fbd87e00640 1 -- 172.21.15.90:0/2647021788 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fbd94020020 con 0x55f210e510d0 2024-04-02T14:36:42.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.897+0000 7fbdaff6d740 10 client.15717 fetch_fsmap learned FSMap version 27 2024-04-02T14:36:42.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.897+0000 7fbdaff6d740 1 -- 172.21.15.90:0/2647021788 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55f210ecfd80 con 0x55f210e510d0 2024-04-02T14:36:42.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.898+0000 7fbd87e00640 1 -- 172.21.15.90:0/2647021788 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbd980beb40 con 0x55f210e510d0 2024-04-02T14:36:42.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.898+0000 7fbd87e00640 1 -- 172.21.15.90:0/2647021788 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fbd9401a070 con 0x55f210e510d0 2024-04-02T14:36:42.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.898+0000 7fbdaff6d740 10 client.15717 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:36:42.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.898+0000 7fbdaff6d740 10 client.15717 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:36:42.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.898+0000 7fbdaff6d740 20 client.15717 populate_metadata read hostname 'smithi090' 2024-04-02T14:36:42.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.898+0000 7fbdaff6d740 1 --2- 172.21.15.90:0/2647021788 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55f210ed0470 0x55f210ed2890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:42.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.898+0000 7fbdaff6d740 4 client.15717 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-02T14:36:42.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.898+0000 7fbdaff6d740 1 -- 172.21.15.90:0/2647021788 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55f210dc1cb0 con 0x55f210ed0470 2024-04-02T14:36:42.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.898+0000 7fbda5e00640 1 --2- 172.21.15.90:0/2647021788 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55f210ed0470 0x55f210ed2890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:42.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.899+0000 7fbda5e00640 1 --2- 172.21.15.90:0/2647021788 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55f210ed0470 0x55f210ed2890 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a421b9c4b3d1b44 server_cookie=757e481a956827ec in_seq=0 out_seq=0 2024-04-02T14:36:42.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.899+0000 7fbd87e00640 10 client.15717 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:36:42.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.900+0000 7fbd87e00640 1 -- 172.21.15.90:0/2647021788 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55f210dc1cb0 con 0x55f210ed0470 2024-04-02T14:36:42.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.900+0000 7fbd87e00640 10 client.15717 handle_command_reply: tid=0 2024-04-02T14:36:42.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.900+0000 7fbdaff6d740 1 client.15717 shutdown 2024-04-02T14:36:42.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.900+0000 7fbdaff6d740 1 -- 172.21.15.90:0/2647021788 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f210e510d0 msgr2=0x55f210e5d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:42.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.900+0000 7fbdaff6d740 1 --2- 172.21.15.90:0/2647021788 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f210e510d0 0x55f210e5d180 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fbd9400e920 tx=0x7fbd9400e430 comp rx=0 tx=0).stop 2024-04-02T14:36:42.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.901+0000 7fbdaff6d740 1 -- 172.21.15.90:0/2647021788 shutdown_connections 2024-04-02T14:36:42.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.901+0000 7fbdaff6d740 1 --2- 172.21.15.90:0/2647021788 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55f210ed0470 0x55f210ed2890 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:42.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.901+0000 7fbdaff6d740 1 --2- 172.21.15.90:0/2647021788 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f210e59cb0 0x55f210e5d6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:42.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.901+0000 7fbdaff6d740 1 --2- 172.21.15.90:0/2647021788 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f210e510d0 0x55f210e5d180 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:42.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.901+0000 7fbdaff6d740 1 --2- 172.21.15.90:0/2647021788 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f210e507b0 0x55f210e61cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:42.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.901+0000 7fbd87e00640 0 client.15717 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:36:42.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.901+0000 7fbdaff6d740 1 -- 172.21.15.90:0/2647021788 >> 172.21.15.90:0/2647021788 conn(0x55f210e4aee0 msgr2=0x55f210e55a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:36:42.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.901+0000 7fbdaff6d740 1 -- 172.21.15.90:0/2647021788 shutdown_connections 2024-04-02T14:36:42.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.901+0000 7fbdaff6d740 1 -- 172.21.15.90:0/2647021788 wait complete. 2024-04-02T14:36:42.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.901+0000 7fbdaff6d740 20 client.15717 trim_cache size 0 max 16384 2024-04-02T14:36:42.904 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-02T14:36:42.904 INFO:teuthology.orchestra.run.smithi090.stdout: "status": "no active scrubs running", 2024-04-02T14:36:42.904 INFO:teuthology.orchestra.run.smithi090.stdout: "scrubs": {} 2024-04-02T14:36:42.905 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-02T14:36:42.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.902+0000 7fbdad200640 1 -- 172.21.15.90:0/4045382748 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fbd8c0a3e70 msgr2=0x7fbd8c0a6330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:42.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.902+0000 7fbdad200640 1 --2- 172.21.15.90:0/4045382748 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fbd8c0a3e70 0x7fbd8c0a6330 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fbda806afb0 tx=0x7fbda0009d90 comp rx=0 tx=0).stop 2024-04-02T14:36:42.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.902+0000 7fbdad200640 1 -- 172.21.15.90:0/4045382748 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbda8076030 msgr2=0x7fbda81a46c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:42.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.902+0000 7fbdad200640 1 --2- 172.21.15.90:0/4045382748 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbda8076030 0x7fbda81a46c0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fbd98036040 tx=0x7fbd98004040 comp rx=0 tx=0).stop 2024-04-02T14:36:42.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.902+0000 7fbdad200640 1 -- 172.21.15.90:0/4045382748 shutdown_connections 2024-04-02T14:36:42.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.902+0000 7fbdad200640 1 --2- 172.21.15.90:0/4045382748 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fbd8c0a3e70 0x7fbd8c0a6330 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:42.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.902+0000 7fbdad200640 1 --2- 172.21.15.90:0/4045382748 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbda8112940 0x7fbda819e8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:42.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.902+0000 7fbdad200640 1 --2- 172.21.15.90:0/4045382748 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbda8076950 0x7fbda81a4c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:42.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.902+0000 7fbdad200640 1 --2- 172.21.15.90:0/4045382748 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbda8076030 0x7fbda81a46c0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:42.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.902+0000 7fbdad200640 1 -- 172.21.15.90:0/4045382748 >> 172.21.15.90:0/4045382748 conn(0x7fbda80ff770 msgr2=0x7fbda81140d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:36:42.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.902+0000 7fbdad200640 1 -- 172.21.15.90:0/4045382748 shutdown_connections 2024-04-02T14:36:42.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:42.902+0000 7fbdad200640 1 -- 172.21.15.90:0/4045382748 wait complete. 2024-04-02T14:36:42.933 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-02T14:36:42.933 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T14:36:43.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:43.212+0000 7faa28600640 1 -- 172.21.15.90:0/2069625492 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faa20101a90 msgr2=0x7faa20101f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:43.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:43.212+0000 7faa28600640 1 --2- 172.21.15.90:0/2069625492 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faa20101a90 0x7faa20101f10 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7faa0c009890 tx=0x7faa0c02bb70 comp rx=0 tx=0).stop 2024-04-02T14:36:43.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:43.213+0000 7faa28600640 1 -- 172.21.15.90:0/2069625492 shutdown_connections 2024-04-02T14:36:43.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:43.213+0000 7faa28600640 1 --2- 172.21.15.90:0/2069625492 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faa20102450 0x7faa2010e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:43.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:43.213+0000 7faa28600640 1 --2- 172.21.15.90:0/2069625492 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faa20101a90 0x7faa20101f10 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:43.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:43.213+0000 7faa28600640 1 --2- 172.21.15.90:0/2069625492 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7faa20107aa0 0x7faa20107e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:43.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:43.213+0000 7faa28600640 1 -- 172.21.15.90:0/2069625492 >> 172.21.15.90:0/2069625492 conn(0x7faa20078f00 msgr2=0x7faa20079310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:36:43.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:43.213+0000 7faa28600640 1 -- 172.21.15.90:0/2069625492 shutdown_connections 2024-04-02T14:36:43.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:43.213+0000 7faa28600640 1 -- 172.21.15.90:0/2069625492 wait complete. 2024-04-02T14:36:43.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:43.214+0000 7faa28600640 1 Processor -- start 2024-04-02T14:36:43.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:43.214+0000 7faa28600640 1 -- start start 2024-04-02T14:36:43.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:43.214+0000 7faa28600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faa20101a90 0x7faa2019a5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:43.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:43.214+0000 7faa28600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faa20102450 0x7faa2019ab10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:43.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:43.214+0000 7faa28600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7faa20107aa0 0x7faa2019ee90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:43.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:43.214+0000 7faa28600640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7faa20111eb0 con 0x7faa20101a90 2024-04-02T14:36:43.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:43.214+0000 7faa28600640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7faa20111d30 con 0x7faa20107aa0 2024-04-02T14:36:43.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:43.214+0000 7faa28600640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7faa20112030 con 0x7faa20102450 2024-04-02T14:36:43.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:43.215+0000 7faa25c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faa20101a90 0x7faa2019a5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:43.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:43.215+0000 7faa25c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faa20101a90 0x7faa2019a5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:54072/0 (socket says 172.21.15.90:54072) 2024-04-02T14:36:43.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:43.215+0000 7faa25c00640 1 -- 172.21.15.90:0/798078648 learned_addr learned my addr 172.21.15.90:0/798078648 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:36:43.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:43.215+0000 7faa26600640 1 --2- 172.21.15.90:0/798078648 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7faa20107aa0 0x7faa2019ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:43.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:43.215+0000 7faa25200640 1 --2- 172.21.15.90:0/798078648 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faa20102450 0x7faa2019ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:43.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:43.215+0000 7faa26600640 1 -- 172.21.15.90:0/798078648 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faa20102450 msgr2=0x7faa2019ab10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:43.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:43.215+0000 7faa26600640 1 --2- 172.21.15.90:0/798078648 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faa20102450 0x7faa2019ab10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:43.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:43.215+0000 7faa26600640 1 -- 172.21.15.90:0/798078648 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faa20101a90 msgr2=0x7faa2019a5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:43.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:43.216+0000 7faa26600640 1 --2- 172.21.15.90:0/798078648 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faa20101a90 0x7faa2019a5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:43.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:43.216+0000 7faa26600640 1 -- 172.21.15.90:0/798078648 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa2019f5c0 con 0x7faa20107aa0 2024-04-02T14:36:43.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:43.216+0000 7faa26600640 1 --2- 172.21.15.90:0/798078648 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7faa20107aa0 0x7faa2019ee90 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7faa18009fd0 tx=0x7faa1800ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:36:43.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:43.216+0000 7faa25200640 1 --2- 172.21.15.90:0/798078648 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faa20102450 0x7faa2019ab10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:36:43.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:43.216+0000 7faa25c00640 1 --2- 172.21.15.90:0/798078648 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faa20101a90 0x7faa2019a5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:36:43.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:43.217+0000 7faa1e000640 1 -- 172.21.15.90:0/798078648 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa1800ce20 con 0x7faa20107aa0 2024-04-02T14:36:43.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:43.217+0000 7faa1e000640 1 -- 172.21.15.90:0/798078648 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faa18005020 con 0x7faa20107aa0 2024-04-02T14:36:43.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:43.217+0000 7faa28600640 1 -- 172.21.15.90:0/798078648 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa2019f8b0 con 0x7faa20107aa0 2024-04-02T14:36:43.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:43.217+0000 7faa1e000640 1 -- 172.21.15.90:0/798078648 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa18010720 con 0x7faa20107aa0 2024-04-02T14:36:43.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:43.217+0000 7faa28600640 1 -- 172.21.15.90:0/798078648 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa201a7140 con 0x7faa20107aa0 2024-04-02T14:36:43.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:43.219+0000 7faa28600640 1 -- 172.21.15.90:0/798078648 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faa201031d0 con 0x7faa20107aa0 2024-04-02T14:36:43.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:43.221+0000 7faa1e000640 1 -- 172.21.15.90:0/798078648 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7faa180051d0 con 0x7faa20107aa0 2024-04-02T14:36:43.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:43.223+0000 7faa1e000640 1 --2- 172.21.15.90:0/798078648 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7faa080a3ef0 0x7faa080a63b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:43.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:43.223+0000 7faa1e000640 1 -- 172.21.15.90:0/798078648 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7faa180d0c00 con 0x7faa20107aa0 2024-04-02T14:36:43.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:43.223+0000 7faa25c00640 1 --2- 172.21.15.90:0/798078648 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7faa080a3ef0 0x7faa080a63b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:43.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:43.224+0000 7faa25c00640 1 --2- 172.21.15.90:0/798078648 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7faa080a3ef0 0x7faa080a63b0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7faa140058c0 tx=0x7faa14009260 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:36:43.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:43.227+0000 7faa1e000640 1 -- 172.21.15.90:0/798078648 <== mon.2 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7faa1809d250 con 0x7faa20107aa0 2024-04-02T14:36:43.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:43.651+0000 7faa28600640 1 -- 172.21.15.90:0/798078648 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7faa201a7370 con 0x7faa20107aa0 2024-04-02T14:36:43.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:43.653+0000 7faa1e000640 1 -- 172.21.15.90:0/798078648 <== mon.2 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11927 (secure 0 0 0) 0x7faa1809cb50 con 0x7faa20107aa0 2024-04-02T14:36:43.656 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-02T14:36:43.656 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14412,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6837/2852160309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6836","nonce":2852160309},{"type":"v1","addr":"172.21.15.90:6837","nonce":2852160309}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14478,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6841/1919376858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6840","nonce":1919376858},{"type":"v1","addr":"172.21.15.90:6841","nonce":1919376858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-02T14:32:22.319083+0000","modified":"2024-04-02T14:34:52.266152+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24590,"mds_1":24560,"mds_2":24547,"mds_3":24539,"mds_4":24523},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24454":{"gid":24454,"name":"l","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6833/1564319431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":1564319431},{"type":"v1","addr":"172.21.15.177:6833","nonce":1564319431}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.92:6833/1570337372","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":1570337372},{"type":"v1","addr":"172.21.15.92:6833","nonce":1570337372}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"j","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.90:6835/1277124891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6834","nonce":1277124891},{"type":"v1","addr":"172.21.15.90:6835","nonce":1277124891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.92:6835/860721164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":860721164},{"type":"v1","addr":"172.21.15.92:6835","nonce":860721164}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24511":{"gid":24511,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6835/3177244988","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":3177244988},{"type":"v1","addr":"172.21.15.177:6835","nonce":3177244988}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"c","rank":4,"incarnation":25,"state":"up:active","state_seq":9,"addr":"172.21.15.177:6837/4223414858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":4223414858},{"type":"v1","addr":"172.21.15.177:6837","nonce":4223414858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24539":{"gid":24539,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":8,"addr":"172.21.15.92:6837/1270635939","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":1270635939},{"type":"v1","addr":"172.21.15.92:6837","nonce":1270635939}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24547":{"gid":24547,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":7,"addr":"172.21.15.177:6839/664895663","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":664895663},{"type":"v1","addr":"172.21.15.177:6839","nonce":664895663}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"g","rank":1,"incarnation":19,"state":"up:active","state_seq":5,"addr":"172.21.15.90:6839/1498915870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6838","nonce":1498915870},{"type":"v1","addr":"172.21.15.90:6839","nonce":1498915870}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24590":{"gid":24590,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.92:6839/1054888219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":1054888219},{"type":"v1","addr":"172.21.15.92:6839","nonce":1054888219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24590,"qdb_cluster":[24547,24590,24523,24539,24560]},"id":1}]} 2024-04-02T14:36:43.656 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 27 2024-04-02T14:36:43.658 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:43.655+0000 7faa28600640 1 -- 172.21.15.90:0/798078648 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7faa080a3ef0 msgr2=0x7faa080a63b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:43.658 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:43.655+0000 7faa28600640 1 --2- 172.21.15.90:0/798078648 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7faa080a3ef0 0x7faa080a63b0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7faa140058c0 tx=0x7faa14009260 comp rx=0 tx=0).stop 2024-04-02T14:36:43.658 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:43.655+0000 7faa28600640 1 -- 172.21.15.90:0/798078648 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7faa20107aa0 msgr2=0x7faa2019ee90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:43.658 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:43.655+0000 7faa28600640 1 --2- 172.21.15.90:0/798078648 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7faa20107aa0 0x7faa2019ee90 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7faa18009fd0 tx=0x7faa1800ef10 comp rx=0 tx=0).stop 2024-04-02T14:36:43.658 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:43.656+0000 7faa28600640 1 -- 172.21.15.90:0/798078648 shutdown_connections 2024-04-02T14:36:43.658 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:43.656+0000 7faa28600640 1 --2- 172.21.15.90:0/798078648 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7faa080a3ef0 0x7faa080a63b0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:43.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:43.656+0000 7faa28600640 1 --2- 172.21.15.90:0/798078648 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7faa20107aa0 0x7faa2019ee90 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:43.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:43.656+0000 7faa28600640 1 --2- 172.21.15.90:0/798078648 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faa20102450 0x7faa2019ab10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:43.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:43.656+0000 7faa28600640 1 --2- 172.21.15.90:0/798078648 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faa20101a90 0x7faa2019a5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:43.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:43.656+0000 7faa28600640 1 -- 172.21.15.90:0/798078648 >> 172.21.15.90:0/798078648 conn(0x7faa20078f00 msgr2=0x7faa20103620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:36:43.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:43.656+0000 7faa28600640 1 -- 172.21.15.90:0/798078648 shutdown_connections 2024-04-02T14:36:43.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:43.656+0000 7faa28600640 1 -- 172.21.15.90:0/798078648 wait complete. 2024-04-02T14:36:43.681 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:36:43 smithi090 ceph-mon[31674]: pgmap v58: 97 pgs: 97 active+clean; 1.2 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 9.5 MiB/s wr, 711 op/s 2024-04-02T14:36:43.684 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-04-02T14:36:43.686 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:36:43 smithi177 ceph-mon[36431]: pgmap v58: 97 pgs: 97 active+clean; 1.2 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 9.5 MiB/s wr, 711 op/s 2024-04-02T14:36:43.860 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:36:43 smithi092 ceph-mon[36497]: pgmap v58: 97 pgs: 97 active+clean; 1.2 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 9.5 MiB/s wr, 711 op/s 2024-04-02T14:36:43.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:43.983+0000 7f8c3d800640 1 -- 172.21.15.90:0/1084297737 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8c3806a070 msgr2=0x7f8c3806a4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:43.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:43.983+0000 7f8c3d800640 1 --2- 172.21.15.90:0/1084297737 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8c3806a070 0x7f8c3806a4f0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f8c2000dcc0 tx=0x7f8c2002fb90 comp rx=0 tx=0).stop 2024-04-02T14:36:43.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:43.983+0000 7f8c3d800640 1 -- 172.21.15.90:0/1084297737 shutdown_connections 2024-04-02T14:36:43.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:43.983+0000 7f8c3d800640 1 --2- 172.21.15.90:0/1084297737 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8c3806aa30 0x7f8c3810e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:43.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:43.983+0000 7f8c3d800640 1 --2- 172.21.15.90:0/1084297737 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8c3806a070 0x7f8c3806a4f0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:43.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:43.983+0000 7f8c3d800640 1 --2- 172.21.15.90:0/1084297737 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8c38107b30 0x7f8c38107f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:43.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:43.983+0000 7f8c3d800640 1 -- 172.21.15.90:0/1084297737 >> 172.21.15.90:0/1084297737 conn(0x7f8c380779e0 msgr2=0x7f8c38077df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:36:43.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:43.984+0000 7f8c3d800640 1 -- 172.21.15.90:0/1084297737 shutdown_connections 2024-04-02T14:36:43.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:43.984+0000 7f8c3d800640 1 -- 172.21.15.90:0/1084297737 wait complete. 2024-04-02T14:36:43.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:43.985+0000 7f8c3d800640 1 Processor -- start 2024-04-02T14:36:43.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:43.985+0000 7f8c3d800640 1 -- start start 2024-04-02T14:36:43.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:43.986+0000 7f8c3d800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8c3806a070 0x7f8c381a4640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:43.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:43.986+0000 7f8c3d800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8c3806aa30 0x7f8c381a4b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:43.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:43.986+0000 7f8c3d800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8c38107b30 0x7f8c3819e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:43.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:43.986+0000 7f8c3d800640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f8c3810fd00 con 0x7f8c3806a070 2024-04-02T14:36:43.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:43.986+0000 7f8c3d800640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f8c3810fb80 con 0x7f8c3806aa30 2024-04-02T14:36:43.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:43.986+0000 7f8c3d800640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f8c3810fe80 con 0x7f8c38107b30 2024-04-02T14:36:43.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:43.986+0000 7f8c37400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8c38107b30 0x7f8c3819e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:43.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:43.986+0000 7f8c36000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8c3806aa30 0x7f8c381a4b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:43.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:43.986+0000 7f8c36000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8c3806aa30 0x7f8c381a4b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:57322/0 (socket says 172.21.15.90:57322) 2024-04-02T14:36:43.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:43.986+0000 7f8c36000640 1 -- 172.21.15.90:0/2883546326 learned_addr learned my addr 172.21.15.90:0/2883546326 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:36:43.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:43.987+0000 7f8c36a00640 1 --2- 172.21.15.90:0/2883546326 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8c3806a070 0x7f8c381a4640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:43.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:43.987+0000 7f8c36000640 1 -- 172.21.15.90:0/2883546326 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8c38107b30 msgr2=0x7f8c3819e7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:43.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:43.987+0000 7f8c36000640 1 --2- 172.21.15.90:0/2883546326 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8c38107b30 0x7f8c3819e7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:43.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:43.987+0000 7f8c36000640 1 -- 172.21.15.90:0/2883546326 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8c3806a070 msgr2=0x7f8c381a4640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:43.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:43.987+0000 7f8c36000640 1 --2- 172.21.15.90:0/2883546326 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8c3806a070 0x7f8c381a4640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:43.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:43.987+0000 7f8c36000640 1 -- 172.21.15.90:0/2883546326 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c3819ef80 con 0x7f8c3806aa30 2024-04-02T14:36:43.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:43.987+0000 7f8c36a00640 1 --2- 172.21.15.90:0/2883546326 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8c3806a070 0x7f8c381a4640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:36:43.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:43.987+0000 7f8c37400640 1 --2- 172.21.15.90:0/2883546326 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8c38107b30 0x7f8c3819e7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:36:43.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:43.987+0000 7f8c36000640 1 --2- 172.21.15.90:0/2883546326 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8c3806aa30 0x7f8c381a4b80 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f8c20000c00 tx=0x7f8c200046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:36:43.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:43.988+0000 7f8c2f400640 1 -- 172.21.15.90:0/2883546326 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c20030530 con 0x7f8c3806aa30 2024-04-02T14:36:43.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:43.988+0000 7f8c2f400640 1 -- 172.21.15.90:0/2883546326 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8c20030bb0 con 0x7f8c3806aa30 2024-04-02T14:36:43.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:43.988+0000 7f8c2f400640 1 -- 172.21.15.90:0/2883546326 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c20038b90 con 0x7f8c3806aa30 2024-04-02T14:36:43.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:43.988+0000 7f8c3d800640 1 -- 172.21.15.90:0/2883546326 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c3819f210 con 0x7f8c3806aa30 2024-04-02T14:36:43.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:43.988+0000 7f8c3d800640 1 -- 172.21.15.90:0/2883546326 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c381ab490 con 0x7f8c3806aa30 2024-04-02T14:36:43.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:43.994+0000 7f8c405d4740 1 Processor -- start 2024-04-02T14:36:43.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:43.994+0000 7f8c405d4740 1 -- start start 2024-04-02T14:36:43.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:43.994+0000 7f8c405d4740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5580024b39f0 0x5580024b2740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:43.998 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:43.995+0000 7f8c405d4740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5580024b2d10 0x5580024b3190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:43.998 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:43.995+0000 7f8c405d4740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5580024bc080 0x5580024be470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:43.998 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:43.995+0000 7f8c405d4740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5580023f3400 con 0x5580024b39f0 2024-04-02T14:36:43.998 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:43.995+0000 7f8c405d4740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x5580023d7f10 con 0x5580024bc080 2024-04-02T14:36:43.998 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:43.995+0000 7f8c405d4740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5580024176f0 con 0x5580024b2d10 2024-04-02T14:36:43.998 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:43.995+0000 7f8c36a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5580024b2d10 0x5580024b3190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:43.998 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:43.995+0000 7f8c37400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5580024b39f0 0x5580024b2740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:43.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:43.995+0000 7f8c36a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5580024b2d10 0x5580024b3190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:41202/0 (socket says 172.21.15.90:41202) 2024-04-02T14:36:43.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:43.995+0000 7f8c36a00640 1 -- 172.21.15.90:0/1404786866 learned_addr learned my addr 172.21.15.90:0/1404786866 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:36:43.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:43.996+0000 7f8c36a00640 1 -- 172.21.15.90:0/1404786866 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5580024bc080 msgr2=0x5580024be470 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-02T14:36:43.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:43.996+0000 7f8c36a00640 1 --2- 172.21.15.90:0/1404786866 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5580024bc080 0x5580024be470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:43.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:43.996+0000 7f8c36a00640 1 -- 172.21.15.90:0/1404786866 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5580024b39f0 msgr2=0x5580024b2740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:36:43.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:43.996+0000 7f8c36a00640 1 --2- 172.21.15.90:0/1404786866 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5580024b39f0 0x5580024b2740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:43.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:43.996+0000 7f8c36a00640 1 -- 172.21.15.90:0/1404786866 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55800212b3f0 con 0x5580024b2d10 2024-04-02T14:36:44.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:43.996+0000 7f8c2f400640 1 -- 172.21.15.90:0/2883546326 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f8c20047060 con 0x7f8c3806aa30 2024-04-02T14:36:44.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:43.996+0000 7f8c37400640 1 --2- 172.21.15.90:0/1404786866 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5580024b39f0 0x5580024b2740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:36:44.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:43.996+0000 7f8c36a00640 1 --2- 172.21.15.90:0/1404786866 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5580024b2d10 0x5580024b3190 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f8c28009cb0 tx=0x7f8c280117b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9349c283d1de28c8 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:36:44.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:43.997+0000 7f8c27e00640 1 -- 172.21.15.90:0/1404786866 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c28012450 con 0x5580024b2d10 2024-04-02T14:36:44.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:43.997+0000 7f8c27e00640 1 -- 172.21.15.90:0/1404786866 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8c28012ad0 con 0x5580024b2d10 2024-04-02T14:36:44.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:43.997+0000 7f8c27e00640 1 -- 172.21.15.90:0/1404786866 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c2800fc30 con 0x5580024b2d10 2024-04-02T14:36:44.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:43.998+0000 7f8c2f400640 1 --2- 172.21.15.90:0/2883546326 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f8c1c0a3e70 0x7f8c1c0a6330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:44.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:43.998+0000 7f8c2f400640 1 -- 172.21.15.90:0/2883546326 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f8c200f1a70 con 0x7f8c3806aa30 2024-04-02T14:36:44.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:43.999+0000 7f8c37400640 1 --2- 172.21.15.90:0/2883546326 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f8c1c0a3e70 0x7f8c1c0a6330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:44.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:43.999+0000 7f8c405d4740 1 -- 172.21.15.90:0/1404786866 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5580024b2d10 msgr2=0x5580024b3190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:44.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:43.999+0000 7f8c405d4740 1 --2- 172.21.15.90:0/1404786866 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5580024b2d10 0x5580024b3190 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f8c28009cb0 tx=0x7f8c280117b0 comp rx=0 tx=0).stop 2024-04-02T14:36:44.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.001+0000 7f8c37400640 1 --2- 172.21.15.90:0/2883546326 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f8c1c0a3e70 0x7f8c1c0a6330 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f8c30005fd0 tx=0x7f8c30009500 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:36:44.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.001+0000 7f8c405d4740 1 -- 172.21.15.90:0/1404786866 shutdown_connections 2024-04-02T14:36:44.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.001+0000 7f8c405d4740 1 --2- 172.21.15.90:0/1404786866 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5580024bc080 0x5580024be470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:44.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.001+0000 7f8c405d4740 1 --2- 172.21.15.90:0/1404786866 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5580024b2d10 0x5580024b3190 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:44.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.001+0000 7f8c405d4740 1 --2- 172.21.15.90:0/1404786866 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5580024b39f0 0x5580024b2740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:44.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.001+0000 7f8c405d4740 1 -- 172.21.15.90:0/1404786866 >> 172.21.15.90:0/1404786866 conn(0x558002479a50 msgr2=0x5580024ad760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:36:44.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.001+0000 7f8c405d4740 1 -- 172.21.15.90:0/1404786866 shutdown_connections 2024-04-02T14:36:44.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.001+0000 7f8c405d4740 1 -- 172.21.15.90:0/1404786866 wait complete. 2024-04-02T14:36:44.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.002+0000 7f8c405d4740 1 -- start start 2024-04-02T14:36:44.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.002+0000 7f8c405d4740 1 Processor -- start 2024-04-02T14:36:44.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.004+0000 7f8c405d4740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5580024b2d10 0x5580024b9090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:44.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.004+0000 7f8c405d4740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5580024b39f0 0x5580024b95d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:44.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.004+0000 7f8c405d4740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5580024bc080 0x5580024b9b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:44.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.004+0000 7f8c405d4740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5580024136a0 con 0x5580024b39f0 2024-04-02T14:36:44.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.004+0000 7f8c405d4740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x5580023bf980 con 0x5580024b2d10 2024-04-02T14:36:44.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.004+0000 7f8c405d4740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5580023d2cb0 con 0x5580024bc080 2024-04-02T14:36:44.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.004+0000 7f8c405d4740 10 client.0 _ll_register_callbacks cb 0x5580023d26a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:36:44.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.004+0000 7f8c36a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5580024b2d10 0x5580024b9090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:44.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.004+0000 7f8c36a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5580024b2d10 0x5580024b9090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:57334/0 (socket says 172.21.15.90:57334) 2024-04-02T14:36:44.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.005+0000 7f8c36a00640 1 -- 172.21.15.90:0/1548881950 learned_addr learned my addr 172.21.15.90:0/1548881950 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:36:44.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.005+0000 7f8c36a00640 1 -- 172.21.15.90:0/1548881950 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5580024bc080 msgr2=0x5580024b9b10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-02T14:36:44.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.005+0000 7f8c36a00640 1 --2- 172.21.15.90:0/1548881950 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5580024bc080 0x5580024b9b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:44.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.005+0000 7f8c36a00640 1 -- 172.21.15.90:0/1548881950 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5580024b39f0 msgr2=0x5580024b95d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-02T14:36:44.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.006+0000 7f8c36000640 1 --2- 172.21.15.90:0/1548881950 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5580024b39f0 0x5580024b95d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:44.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.006+0000 7f8c36a00640 1 --2- 172.21.15.90:0/1548881950 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5580024b39f0 0x5580024b95d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:44.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.006+0000 7f8c36a00640 1 -- 172.21.15.90:0/1548881950 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5580021f57d0 con 0x5580024b2d10 2024-04-02T14:36:44.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.006+0000 7f8c36000640 1 --2- 172.21.15.90:0/1548881950 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5580024b39f0 0x5580024b95d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:36:44.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.007+0000 7f8c24c00640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:36:44.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.007+0000 7f8c36a00640 1 --2- 172.21.15.90:0/1548881950 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5580024b2d10 0x5580024b9090 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f8c2800f810 tx=0x7f8c2800f670 comp rx=0 tx=0).ready entity=mon.2 client_cookie=62b4629bd69bc0f0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:36:44.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.008+0000 7f8c24c00640 1 -- 172.21.15.90:0/1548881950 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c28039020 con 0x5580024b2d10 2024-04-02T14:36:44.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.008+0000 7f8c24c00640 1 -- 172.21.15.90:0/1548881950 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f8c28012720 con 0x5580024b2d10 2024-04-02T14:36:44.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.008+0000 7f8c405d4740 1 -- 172.21.15.90:0/1548881950 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8c28012ad0 con 0x5580024b2d10 2024-04-02T14:36:44.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.009+0000 7f8c24c00640 1 -- 172.21.15.90:0/1548881950 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f8c28012920 con 0x5580024b2d10 2024-04-02T14:36:44.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.009+0000 7f8c405d4740 10 client.25949 fetch_fsmap learned FSMap version 27 2024-04-02T14:36:44.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.009+0000 7f8c405d4740 1 -- 172.21.15.90:0/1548881950 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5580024caf60 con 0x5580024b2d10 2024-04-02T14:36:44.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.009+0000 7f8c24c00640 1 -- 172.21.15.90:0/1548881950 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8c28012ad0 con 0x5580024b2d10 2024-04-02T14:36:44.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.009+0000 7f8c24c00640 1 -- 172.21.15.90:0/1548881950 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f8c280046c0 con 0x5580024b2d10 2024-04-02T14:36:44.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.009+0000 7f8c405d4740 10 client.25949 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:36:44.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.009+0000 7f8c405d4740 10 client.25949 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.c 2024-04-02T14:36:44.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.009+0000 7f8c405d4740 20 client.25949 populate_metadata read hostname 'smithi090' 2024-04-02T14:36:44.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.009+0000 7f8c405d4740 1 --2- 172.21.15.90:0/1548881950 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x5580024cc310 0x5580024ce700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:44.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.010+0000 7f8c405d4740 4 client.25949 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:36:44.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.010+0000 7f8c405d4740 1 -- 172.21.15.90:0/1548881950 --> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5580023d7040 con 0x5580024cc310 2024-04-02T14:36:44.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.010+0000 7f8c36000640 1 --2- 172.21.15.90:0/1548881950 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x5580024cc310 0x5580024ce700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:44.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.012+0000 7f8c36000640 1 --2- 172.21.15.90:0/1548881950 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x5580024cc310 0x5580024ce700 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=c21aaf97c3949fe2 server_cookie=7140cb6664d1fb8b in_seq=0 out_seq=0 2024-04-02T14:36:44.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.012+0000 7f8c24c00640 10 client.25949 ms_handle_connect on v2:172.21.15.177:6836/4223414858 2024-04-02T14:36:44.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.014+0000 7f8c24c00640 1 -- 172.21.15.90:0/1548881950 <== mds.4 v2:172.21.15.177:6836/4223414858 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5580023d7040 con 0x5580024cc310 2024-04-02T14:36:44.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.014+0000 7f8c24c00640 10 client.25949 handle_command_reply: tid=0 2024-04-02T14:36:44.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.014+0000 7f8c405d4740 1 client.25949 shutdown 2024-04-02T14:36:44.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.014+0000 7f8c405d4740 1 -- 172.21.15.90:0/1548881950 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5580024b2d10 msgr2=0x5580024b9090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:44.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.014+0000 7f8c405d4740 1 --2- 172.21.15.90:0/1548881950 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5580024b2d10 0x5580024b9090 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f8c2800f810 tx=0x7f8c2800f670 comp rx=0 tx=0).stop 2024-04-02T14:36:44.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.015+0000 7f8c405d4740 1 -- 172.21.15.90:0/1548881950 shutdown_connections 2024-04-02T14:36:44.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.015+0000 7f8c405d4740 1 --2- 172.21.15.90:0/1548881950 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5580024bc080 0x5580024b9b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:44.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.015+0000 7f8c405d4740 1 --2- 172.21.15.90:0/1548881950 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x5580024cc310 0x5580024ce700 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:44.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.015+0000 7f8c405d4740 1 --2- 172.21.15.90:0/1548881950 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5580024b39f0 0x5580024b95d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:44.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.015+0000 7f8c405d4740 1 --2- 172.21.15.90:0/1548881950 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5580024b2d10 0x5580024b9090 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:44.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.015+0000 7f8c405d4740 1 -- 172.21.15.90:0/1548881950 >> 172.21.15.90:0/1548881950 conn(0x558002479a50 msgr2=0x5580024af4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:36:44.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.015+0000 7f8c24c00640 0 client.25949 ms_handle_reset on v2:172.21.15.177:6836/4223414858 2024-04-02T14:36:44.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.015+0000 7f8c405d4740 1 -- 172.21.15.90:0/1548881950 shutdown_connections 2024-04-02T14:36:44.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.015+0000 7f8c405d4740 1 -- 172.21.15.90:0/1548881950 wait complete. 2024-04-02T14:36:44.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.015+0000 7f8c405d4740 20 client.25949 trim_cache size 0 max 16384 2024-04-02T14:36:44.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.051+0000 7f8c405d4740 1 Processor -- start 2024-04-02T14:36:44.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.051+0000 7f8c405d4740 1 -- start start 2024-04-02T14:36:44.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.052+0000 7f8c405d4740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5580024b2d10 0x5580024bb7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:44.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.052+0000 7f8c405d4740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5580024b39f0 0x5580024c7b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:44.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.052+0000 7f8c405d4740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5580024bc080 0x5580024b8970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:44.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.052+0000 7f8c405d4740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f8c381ab660 con 0x5580024bc080 2024-04-02T14:36:44.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.052+0000 7f8c405d4740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f8c381ad170 con 0x5580024b2d10 2024-04-02T14:36:44.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.052+0000 7f8c405d4740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f8c381abd00 con 0x5580024b39f0 2024-04-02T14:36:44.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.052+0000 7f8c36a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5580024b2d10 0x5580024bb7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:44.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.053+0000 7f8c36a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5580024b2d10 0x5580024bb7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:57346/0 (socket says 172.21.15.90:57346) 2024-04-02T14:36:44.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.053+0000 7f8c36a00640 1 -- 172.21.15.90:0/1402520571 learned_addr learned my addr 172.21.15.90:0/1402520571 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:36:44.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.053+0000 7f8c36000640 1 --2- 172.21.15.90:0/1402520571 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5580024b39f0 0x5580024c7b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:44.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.053+0000 7f8c36000640 1 -- 172.21.15.90:0/1402520571 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5580024b2d10 msgr2=0x5580024bb7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:36:44.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.053+0000 7f8c36000640 1 --2- 172.21.15.90:0/1402520571 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5580024b2d10 0x5580024bb7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:44.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.053+0000 7f8c37400640 1 --2- 172.21.15.90:0/1402520571 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5580024bc080 0x5580024b8970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:44.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.054+0000 7f8c36000640 1 -- 172.21.15.90:0/1402520571 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5580024bc080 msgr2=0x5580024b8970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:36:44.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.054+0000 7f8c36000640 1 --2- 172.21.15.90:0/1402520571 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5580024bc080 0x5580024b8970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:44.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.054+0000 7f8c36000640 1 -- 172.21.15.90:0/1402520571 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5580024ad970 con 0x5580024b39f0 2024-04-02T14:36:44.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.054+0000 7f8c36a00640 1 --2- 172.21.15.90:0/1402520571 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5580024b2d10 0x5580024bb7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:36:44.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.054+0000 7f8c37400640 1 --2- 172.21.15.90:0/1402520571 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5580024bc080 0x5580024b8970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:36:44.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.055+0000 7f8c36000640 1 --2- 172.21.15.90:0/1402520571 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5580024b39f0 0x5580024c7b20 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f8c200ef280 tx=0x7f8c20032040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c1a524d4fb98fbb1 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:36:44.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.056+0000 7f8c27400640 1 -- 172.21.15.90:0/1402520571 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c200bdad0 con 0x5580024b39f0 2024-04-02T14:36:44.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.056+0000 7f8c27400640 1 -- 172.21.15.90:0/1402520571 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8c200d6070 con 0x5580024b39f0 2024-04-02T14:36:44.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.056+0000 7f8c27400640 1 -- 172.21.15.90:0/1402520571 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c200de140 con 0x5580024b39f0 2024-04-02T14:36:44.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.057+0000 7f8c405d4740 1 -- 172.21.15.90:0/1402520571 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5580024b39f0 msgr2=0x5580024c7b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:44.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.057+0000 7f8c405d4740 1 --2- 172.21.15.90:0/1402520571 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5580024b39f0 0x5580024c7b20 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f8c200ef280 tx=0x7f8c20032040 comp rx=0 tx=0).stop 2024-04-02T14:36:44.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.057+0000 7f8c405d4740 1 -- 172.21.15.90:0/1402520571 shutdown_connections 2024-04-02T14:36:44.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.057+0000 7f8c405d4740 1 --2- 172.21.15.90:0/1402520571 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5580024bc080 0x5580024b8970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:44.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.057+0000 7f8c405d4740 1 --2- 172.21.15.90:0/1402520571 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5580024b39f0 0x5580024c7b20 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:44.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.057+0000 7f8c405d4740 1 --2- 172.21.15.90:0/1402520571 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5580024b2d10 0x5580024bb7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:44.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.057+0000 7f8c405d4740 1 -- 172.21.15.90:0/1402520571 >> 172.21.15.90:0/1402520571 conn(0x558002479a50 msgr2=0x5580024af3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:36:44.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.057+0000 7f8c405d4740 1 -- 172.21.15.90:0/1402520571 shutdown_connections 2024-04-02T14:36:44.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.057+0000 7f8c405d4740 1 -- 172.21.15.90:0/1402520571 wait complete. 2024-04-02T14:36:44.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.058+0000 7f8c405d4740 1 -- start start 2024-04-02T14:36:44.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.058+0000 7f8c405d4740 1 Processor -- start 2024-04-02T14:36:44.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.059+0000 7f8c405d4740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5580024b2d10 0x5580024bdc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:44.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.059+0000 7f8c405d4740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5580024b39f0 0x5580024bc460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:44.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.059+0000 7f8c405d4740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5580024bc080 0x5580024bcae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:44.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.059+0000 7f8c405d4740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f8c381acaa0 con 0x5580024b2d10 2024-04-02T14:36:44.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.059+0000 7f8c405d4740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f8c381ac3d0 con 0x5580024bc080 2024-04-02T14:36:44.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.059+0000 7f8c405d4740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f8c38071cd0 con 0x5580024b39f0 2024-04-02T14:36:44.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.060+0000 7f8c37400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5580024bc080 0x5580024bcae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:44.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.060+0000 7f8c37400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5580024bc080 0x5580024bcae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:57352/0 (socket says 172.21.15.90:57352) 2024-04-02T14:36:44.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.060+0000 7f8c37400640 1 -- 172.21.15.90:0/2501750862 learned_addr learned my addr 172.21.15.90:0/2501750862 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:36:44.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.060+0000 7f8c36000640 1 --2- 172.21.15.90:0/2501750862 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5580024b39f0 0x5580024bc460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:44.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.059+0000 7f8c405d4740 10 client.0 _ll_register_callbacks cb 0x7f8c280097f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:36:44.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.060+0000 7f8c37400640 1 -- 172.21.15.90:0/2501750862 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5580024b39f0 msgr2=0x5580024bc460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:36:44.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.060+0000 7f8c37400640 1 --2- 172.21.15.90:0/2501750862 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5580024b39f0 0x5580024bc460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:44.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.060+0000 7f8c37400640 1 -- 172.21.15.90:0/2501750862 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5580024b2d10 msgr2=0x5580024bdc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:36:44.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.060+0000 7f8c36a00640 1 --2- 172.21.15.90:0/2501750862 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5580024b2d10 0x5580024bdc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:44.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.061+0000 7f8c37400640 1 --2- 172.21.15.90:0/2501750862 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5580024b2d10 0x5580024bdc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:44.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.061+0000 7f8c37400640 1 -- 172.21.15.90:0/2501750862 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5580024c3030 con 0x5580024bc080 2024-04-02T14:36:44.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.061+0000 7f8c36a00640 1 --2- 172.21.15.90:0/2501750862 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5580024b2d10 0x5580024bdc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:36:44.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.062+0000 7f8c37400640 1 --2- 172.21.15.90:0/2501750862 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5580024bc080 0x5580024bcae0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f8c30013fd0 tx=0x7f8c30004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=87f994f58561067c server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:36:44.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.062+0000 7f8c24c00640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:36:44.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.062+0000 7f8c24c00640 1 -- 172.21.15.90:0/2501750862 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c3002c060 con 0x5580024bc080 2024-04-02T14:36:44.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.062+0000 7f8c24c00640 1 -- 172.21.15.90:0/2501750862 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f8c3001ca00 con 0x5580024bc080 2024-04-02T14:36:44.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.062+0000 7f8c405d4740 1 -- 172.21.15.90:0/2501750862 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8c200d6070 con 0x5580024bc080 2024-04-02T14:36:44.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.062+0000 7f8c24c00640 1 -- 172.21.15.90:0/2501750862 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f8c3001cc00 con 0x5580024bc080 2024-04-02T14:36:44.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.062+0000 7f8c405d4740 10 client.25955 fetch_fsmap learned FSMap version 27 2024-04-02T14:36:44.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.062+0000 7f8c405d4740 1 -- 172.21.15.90:0/2501750862 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5580024c4300 con 0x5580024bc080 2024-04-02T14:36:44.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.063+0000 7f8c24c00640 1 -- 172.21.15.90:0/2501750862 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8c200d6070 con 0x5580024bc080 2024-04-02T14:36:44.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.063+0000 7f8c24c00640 1 -- 172.21.15.90:0/2501750862 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f8c3000e410 con 0x5580024bc080 2024-04-02T14:36:44.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.064+0000 7f8c405d4740 10 client.25955 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:36:44.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.064+0000 7f8c405d4740 10 client.25955 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.c 2024-04-02T14:36:44.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.064+0000 7f8c405d4740 20 client.25955 populate_metadata read hostname 'smithi090' 2024-04-02T14:36:44.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.064+0000 7f8c405d4740 1 --2- 172.21.15.90:0/2501750862 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x5580025325a0 0x558002534990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:44.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.064+0000 7f8c405d4740 4 client.25955 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-02T14:36:44.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.064+0000 7f8c405d4740 1 -- 172.21.15.90:0/2501750862 --> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] -- command(tid 0: {"prefix": "damage ls"}) -- 0x558002424c30 con 0x5580025325a0 2024-04-02T14:36:44.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.064+0000 7f8c36a00640 1 --2- 172.21.15.90:0/2501750862 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x5580025325a0 0x558002534990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:44.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.065+0000 7f8c24c00640 10 client.25955 ms_handle_connect on v2:172.21.15.177:6836/4223414858 2024-04-02T14:36:44.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.067+0000 7f8c36a00640 1 --2- 172.21.15.90:0/2501750862 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x5580025325a0 0x558002534990 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=a6385627e061d59b server_cookie=e3bec38673361420 in_seq=0 out_seq=0 2024-04-02T14:36:44.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.067+0000 7f8c24c00640 1 -- 172.21.15.90:0/2501750862 <== mds.4 v2:172.21.15.177:6836/4223414858 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x558002424c30 con 0x5580025325a0 2024-04-02T14:36:44.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.067+0000 7f8c24c00640 10 client.25955 handle_command_reply: tid=0 2024-04-02T14:36:44.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.067+0000 7f8c405d4740 1 client.25955 shutdown 2024-04-02T14:36:44.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.068+0000 7f8c405d4740 1 -- 172.21.15.90:0/2501750862 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5580024bc080 msgr2=0x5580024bcae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:44.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.068+0000 7f8c405d4740 1 --2- 172.21.15.90:0/2501750862 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5580024bc080 0x5580024bcae0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f8c30013fd0 tx=0x7f8c30004a40 comp rx=0 tx=0).stop 2024-04-02T14:36:44.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.068+0000 7f8c405d4740 1 -- 172.21.15.90:0/2501750862 shutdown_connections 2024-04-02T14:36:44.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.068+0000 7f8c405d4740 1 --2- 172.21.15.90:0/2501750862 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5580024bc080 0x5580024bcae0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:44.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.068+0000 7f8c405d4740 1 --2- 172.21.15.90:0/2501750862 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5580024b39f0 0x5580024bc460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:44.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.068+0000 7f8c405d4740 1 --2- 172.21.15.90:0/2501750862 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x5580025325a0 0x558002534990 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:44.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.068+0000 7f8c405d4740 1 --2- 172.21.15.90:0/2501750862 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5580024b2d10 0x5580024bdc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:44.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.068+0000 7f8c24c00640 0 client.25955 ms_handle_reset on v2:172.21.15.177:6836/4223414858 2024-04-02T14:36:44.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.068+0000 7f8c405d4740 1 -- 172.21.15.90:0/2501750862 >> 172.21.15.90:0/2501750862 conn(0x558002479a50 msgr2=0x5580024aea80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:36:44.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.069+0000 7f8c405d4740 1 -- 172.21.15.90:0/2501750862 shutdown_connections 2024-04-02T14:36:44.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.069+0000 7f8c405d4740 1 -- 172.21.15.90:0/2501750862 wait complete. 2024-04-02T14:36:44.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.069+0000 7f8c405d4740 20 client.25955 trim_cache size 0 max 16384 2024-04-02T14:36:44.072 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-02T14:36:44.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.070+0000 7f8c3d800640 1 -- 172.21.15.90:0/2883546326 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f8c1c0a3e70 msgr2=0x7f8c1c0a6330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:44.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.070+0000 7f8c3d800640 1 --2- 172.21.15.90:0/2883546326 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f8c1c0a3e70 0x7f8c1c0a6330 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f8c30005fd0 tx=0x7f8c30009500 comp rx=0 tx=0).stop 2024-04-02T14:36:44.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.070+0000 7f8c3d800640 1 -- 172.21.15.90:0/2883546326 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8c3806aa30 msgr2=0x7f8c381a4b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:44.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.070+0000 7f8c3d800640 1 --2- 172.21.15.90:0/2883546326 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8c3806aa30 0x7f8c381a4b80 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f8c20000c00 tx=0x7f8c200046c0 comp rx=0 tx=0).stop 2024-04-02T14:36:44.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.071+0000 7f8c3d800640 1 -- 172.21.15.90:0/2883546326 shutdown_connections 2024-04-02T14:36:44.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.071+0000 7f8c3d800640 1 --2- 172.21.15.90:0/2883546326 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f8c1c0a3e70 0x7f8c1c0a6330 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:44.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.071+0000 7f8c3d800640 1 --2- 172.21.15.90:0/2883546326 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8c38107b30 0x7f8c3819e7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:44.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.071+0000 7f8c3d800640 1 --2- 172.21.15.90:0/2883546326 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8c3806aa30 0x7f8c381a4b80 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:44.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.071+0000 7f8c3d800640 1 --2- 172.21.15.90:0/2883546326 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8c3806a070 0x7f8c381a4640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:44.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.071+0000 7f8c3d800640 1 -- 172.21.15.90:0/2883546326 >> 172.21.15.90:0/2883546326 conn(0x7f8c380779e0 msgr2=0x7f8c380ff840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:36:44.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.071+0000 7f8c3d800640 1 -- 172.21.15.90:0/2883546326 shutdown_connections 2024-04-02T14:36:44.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.071+0000 7f8c3d800640 1 -- 172.21.15.90:0/2883546326 wait complete. 2024-04-02T14:36:44.100 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-04-02T14:36:44.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.366+0000 7fed2d800640 1 -- 172.21.15.90:0/1093163966 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fed2806b420 msgr2=0x7fed2806b800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:44.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.366+0000 7fed2d800640 1 --2- 172.21.15.90:0/1093163966 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fed2806b420 0x7fed2806b800 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fed20009ca0 tx=0x7fed2002c190 comp rx=0 tx=0).stop 2024-04-02T14:36:44.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.366+0000 7fed2d800640 1 -- 172.21.15.90:0/1093163966 shutdown_connections 2024-04-02T14:36:44.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.366+0000 7fed2d800640 1 --2- 172.21.15.90:0/1093163966 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fed28116ad0 0x7fed28118f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:44.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.366+0000 7fed2d800640 1 --2- 172.21.15.90:0/1093163966 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fed2806bd40 0x7fed28116590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:44.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.366+0000 7fed2d800640 1 --2- 172.21.15.90:0/1093163966 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fed2806b420 0x7fed2806b800 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:44.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.366+0000 7fed2d800640 1 -- 172.21.15.90:0/1093163966 >> 172.21.15.90:0/1093163966 conn(0x7fed280730b0 msgr2=0x7fed280734c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:36:44.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.367+0000 7fed2d800640 1 -- 172.21.15.90:0/1093163966 shutdown_connections 2024-04-02T14:36:44.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.367+0000 7fed2d800640 1 -- 172.21.15.90:0/1093163966 wait complete. 2024-04-02T14:36:44.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.368+0000 7fed2d800640 1 Processor -- start 2024-04-02T14:36:44.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.368+0000 7fed2d800640 1 -- start start 2024-04-02T14:36:44.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.369+0000 7fed2d800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fed2806b420 0x7fed281a2d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:44.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.369+0000 7fed2d800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fed2806bd40 0x7fed281a3290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:44.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.369+0000 7fed2d800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fed28116ad0 0x7fed281a7610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:44.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.369+0000 7fed2d800640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fed2811c850 con 0x7fed2806b420 2024-04-02T14:36:44.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.369+0000 7fed2d800640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fed2811c6d0 con 0x7fed2806bd40 2024-04-02T14:36:44.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.369+0000 7fed2d800640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fed2811c9d0 con 0x7fed28116ad0 2024-04-02T14:36:44.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.369+0000 7fed26a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fed2806b420 0x7fed281a2d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:44.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.370+0000 7fed26a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fed2806b420 0x7fed281a2d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:49970/0 (socket says 172.21.15.90:49970) 2024-04-02T14:36:44.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.370+0000 7fed26a00640 1 -- 172.21.15.90:0/2164120341 learned_addr learned my addr 172.21.15.90:0/2164120341 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:36:44.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.370+0000 7fed27400640 1 --2- 172.21.15.90:0/2164120341 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fed28116ad0 0x7fed281a7610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:44.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.370+0000 7fed26000640 1 --2- 172.21.15.90:0/2164120341 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fed2806bd40 0x7fed281a3290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:44.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.370+0000 7fed26a00640 1 -- 172.21.15.90:0/2164120341 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fed28116ad0 msgr2=0x7fed281a7610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:44.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.371+0000 7fed26a00640 1 --2- 172.21.15.90:0/2164120341 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fed28116ad0 0x7fed281a7610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:44.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.371+0000 7fed26a00640 1 -- 172.21.15.90:0/2164120341 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fed2806bd40 msgr2=0x7fed281a3290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:44.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.371+0000 7fed26a00640 1 --2- 172.21.15.90:0/2164120341 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fed2806bd40 0x7fed281a3290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:44.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.371+0000 7fed26a00640 1 -- 172.21.15.90:0/2164120341 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed281a7d40 con 0x7fed2806b420 2024-04-02T14:36:44.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.372+0000 7fed26a00640 1 --2- 172.21.15.90:0/2164120341 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fed2806b420 0x7fed281a2d50 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fed20004af0 tx=0x7fed20035ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:36:44.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.372+0000 7fed1f400640 1 -- 172.21.15.90:0/2164120341 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed20039040 con 0x7fed2806b420 2024-04-02T14:36:44.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.372+0000 7fed2d800640 1 -- 172.21.15.90:0/2164120341 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed281a7fd0 con 0x7fed2806b420 2024-04-02T14:36:44.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.372+0000 7fed1f400640 1 -- 172.21.15.90:0/2164120341 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fed20004040 con 0x7fed2806b420 2024-04-02T14:36:44.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.372+0000 7fed1f400640 1 -- 172.21.15.90:0/2164120341 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed200328d0 con 0x7fed2806b420 2024-04-02T14:36:44.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.373+0000 7fed2d800640 1 -- 172.21.15.90:0/2164120341 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed2810a120 con 0x7fed2806b420 2024-04-02T14:36:44.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.376+0000 7fed3044e740 1 Processor -- start 2024-04-02T14:36:44.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.377+0000 7fed3044e740 1 -- start start 2024-04-02T14:36:44.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.377+0000 7fed1f400640 1 -- 172.21.15.90:0/2164120341 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fed200041f0 con 0x7fed2806b420 2024-04-02T14:36:44.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.377+0000 7fed3044e740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556cf7b0a9f0 0x556cf7b09740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:44.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.377+0000 7fed3044e740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556cf7b09d10 0x556cf7b0a190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:44.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.377+0000 7fed3044e740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556cf7b13080 0x556cf7b15470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:44.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.377+0000 7fed3044e740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x556cf7a4a400 con 0x556cf7b09d10 2024-04-02T14:36:44.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.377+0000 7fed3044e740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x556cf7a2ef10 con 0x556cf7b0a9f0 2024-04-02T14:36:44.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.377+0000 7fed3044e740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x556cf7a6e6f0 con 0x556cf7b13080 2024-04-02T14:36:44.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.378+0000 7fed26000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556cf7b13080 0x556cf7b15470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:44.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.378+0000 7fed26000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556cf7b13080 0x556cf7b15470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:41250/0 (socket says 172.21.15.90:41250) 2024-04-02T14:36:44.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.378+0000 7fed26000640 1 -- 172.21.15.90:0/4057814174 learned_addr learned my addr 172.21.15.90:0/4057814174 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:36:44.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.378+0000 7fed27400640 1 --2- 172.21.15.90:0/4057814174 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556cf7b0a9f0 0x556cf7b09740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:44.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.378+0000 7fed26000640 1 -- 172.21.15.90:0/4057814174 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556cf7b0a9f0 msgr2=0x556cf7b09740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:36:44.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.378+0000 7fed26000640 1 --2- 172.21.15.90:0/4057814174 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556cf7b0a9f0 0x556cf7b09740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:44.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.378+0000 7fed26000640 1 -- 172.21.15.90:0/4057814174 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556cf7b09d10 msgr2=0x556cf7b0a190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:36:44.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.378+0000 7fed26000640 1 --2- 172.21.15.90:0/4057814174 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556cf7b09d10 0x556cf7b0a190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:44.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.378+0000 7fed26000640 1 -- 172.21.15.90:0/4057814174 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556cf77823f0 con 0x556cf7b13080 2024-04-02T14:36:44.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.379+0000 7fed26000640 1 --2- 172.21.15.90:0/4057814174 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556cf7b13080 0x556cf7b15470 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fed1400cc40 tx=0x7fed1400e720 comp rx=0 tx=0).ready entity=mon.1 client_cookie=221203e51133f571 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:36:44.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.379+0000 7fed1f400640 1 --2- 172.21.15.90:0/2164120341 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fed0c0a3da0 0x7fed0c0a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:44.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.379+0000 7fed13e00640 1 -- 172.21.15.90:0/4057814174 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed14014070 con 0x556cf7b13080 2024-04-02T14:36:44.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.379+0000 7fed13e00640 1 -- 172.21.15.90:0/4057814174 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fed140047a0 con 0x556cf7b13080 2024-04-02T14:36:44.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.379+0000 7fed13e00640 1 -- 172.21.15.90:0/4057814174 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed14010760 con 0x556cf7b13080 2024-04-02T14:36:44.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.379+0000 7fed1f400640 1 -- 172.21.15.90:0/2164120341 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7fed200faf60 con 0x7fed2806b420 2024-04-02T14:36:44.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.380+0000 7fed27400640 1 --2- 172.21.15.90:0/2164120341 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fed0c0a3da0 0x7fed0c0a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:44.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.380+0000 7fed3044e740 1 -- 172.21.15.90:0/4057814174 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556cf7b13080 msgr2=0x556cf7b15470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:44.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.380+0000 7fed3044e740 1 --2- 172.21.15.90:0/4057814174 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556cf7b13080 0x556cf7b15470 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fed1400cc40 tx=0x7fed1400e720 comp rx=0 tx=0).stop 2024-04-02T14:36:44.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.381+0000 7fed3044e740 1 -- 172.21.15.90:0/4057814174 shutdown_connections 2024-04-02T14:36:44.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.381+0000 7fed3044e740 1 --2- 172.21.15.90:0/4057814174 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556cf7b13080 0x556cf7b15470 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:44.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.381+0000 7fed3044e740 1 --2- 172.21.15.90:0/4057814174 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556cf7b09d10 0x556cf7b0a190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:44.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.381+0000 7fed3044e740 1 --2- 172.21.15.90:0/4057814174 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556cf7b0a9f0 0x556cf7b09740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:44.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.381+0000 7fed3044e740 1 -- 172.21.15.90:0/4057814174 >> 172.21.15.90:0/4057814174 conn(0x556cf7ad0a50 msgr2=0x556cf7b04760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:36:44.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.382+0000 7fed3044e740 1 -- 172.21.15.90:0/4057814174 shutdown_connections 2024-04-02T14:36:44.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.382+0000 7fed27400640 1 --2- 172.21.15.90:0/2164120341 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fed0c0a3da0 0x7fed0c0a6260 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fed1800ead0 tx=0x7fed18005e50 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:36:44.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.382+0000 7fed3044e740 1 -- 172.21.15.90:0/4057814174 wait complete. 2024-04-02T14:36:44.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.383+0000 7fed3044e740 1 -- start start 2024-04-02T14:36:44.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.383+0000 7fed3044e740 1 Processor -- start 2024-04-02T14:36:44.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.384+0000 7fed3044e740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556cf7b09d10 0x556cf7b100c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:44.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.384+0000 7fed3044e740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556cf7b0a9f0 0x556cf7b10600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:44.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.384+0000 7fed3044e740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556cf7b13080 0x556cf7b10b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:44.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.384+0000 7fed3044e740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x556cf7a6a6a0 con 0x556cf7b13080 2024-04-02T14:36:44.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.384+0000 7fed3044e740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x556cf7a16980 con 0x556cf7b09d10 2024-04-02T14:36:44.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.384+0000 7fed3044e740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x556cf7a29cb0 con 0x556cf7b0a9f0 2024-04-02T14:36:44.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.384+0000 7fed3044e740 10 client.0 _ll_register_callbacks cb 0x556cf7a296a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:36:44.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.384+0000 7fed27400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556cf7b13080 0x556cf7b10b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:44.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.384+0000 7fed27400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556cf7b13080 0x556cf7b10b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:49986/0 (socket says 172.21.15.90:49986) 2024-04-02T14:36:44.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.384+0000 7fed26000640 1 --2- 172.21.15.90:0/2808908720 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556cf7b0a9f0 0x556cf7b10600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:44.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.384+0000 7fed27400640 1 -- 172.21.15.90:0/2808908720 learned_addr learned my addr 172.21.15.90:0/2808908720 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:36:44.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.385+0000 7fed26000640 1 -- 172.21.15.90:0/2808908720 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556cf7b09d10 msgr2=0x556cf7b100c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:36:44.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.385+0000 7fed26a00640 1 --2- 172.21.15.90:0/2808908720 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556cf7b09d10 0x556cf7b100c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:44.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.385+0000 7fed26000640 1 --2- 172.21.15.90:0/2808908720 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556cf7b09d10 0x556cf7b100c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:44.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.385+0000 7fed26000640 1 -- 172.21.15.90:0/2808908720 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556cf7b13080 msgr2=0x556cf7b10b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:36:44.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.385+0000 7fed26000640 1 --2- 172.21.15.90:0/2808908720 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556cf7b13080 0x556cf7b10b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:44.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.385+0000 7fed26000640 1 -- 172.21.15.90:0/2808908720 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556cf784c7d0 con 0x556cf7b0a9f0 2024-04-02T14:36:44.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.385+0000 7fed27400640 1 --2- 172.21.15.90:0/2808908720 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556cf7b13080 0x556cf7b10b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:36:44.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.385+0000 7fed26000640 1 --2- 172.21.15.90:0/2808908720 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556cf7b0a9f0 0x556cf7b10600 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fed14000c00 tx=0x7fed14005210 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7c3eab9afbefb37e server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:36:44.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.385+0000 7fed10c00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:36:44.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.386+0000 7fed10c00640 1 -- 172.21.15.90:0/2808908720 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed14027060 con 0x556cf7b0a9f0 2024-04-02T14:36:44.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.386+0000 7fed10c00640 1 -- 172.21.15.90:0/2808908720 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7fed140104a0 con 0x556cf7b0a9f0 2024-04-02T14:36:44.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.386+0000 7fed3044e740 1 -- 172.21.15.90:0/2808908720 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fed140047a0 con 0x556cf7b0a9f0 2024-04-02T14:36:44.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.386+0000 7fed10c00640 1 -- 172.21.15.90:0/2808908720 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fed14014070 con 0x556cf7b0a9f0 2024-04-02T14:36:44.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.386+0000 7fed3044e740 10 client.25882 fetch_fsmap learned FSMap version 27 2024-04-02T14:36:44.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.386+0000 7fed3044e740 1 -- 172.21.15.90:0/2808908720 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x556cf7b32f70 con 0x556cf7b0a9f0 2024-04-02T14:36:44.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.387+0000 7fed10c00640 1 -- 172.21.15.90:0/2808908720 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fed140047a0 con 0x556cf7b0a9f0 2024-04-02T14:36:44.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.388+0000 7fed10c00640 1 -- 172.21.15.90:0/2808908720 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fed140106e0 con 0x556cf7b0a9f0 2024-04-02T14:36:44.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.388+0000 7fed3044e740 10 client.25882 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:36:44.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.388+0000 7fed3044e740 10 client.25882 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-04-02T14:36:44.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.388+0000 7fed3044e740 20 client.25882 populate_metadata read hostname 'smithi090' 2024-04-02T14:36:44.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.388+0000 7fed3044e740 1 --2- 172.21.15.90:0/2808908720 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x556cf7b342c0 0x556cf7b366b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:44.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.388+0000 7fed3044e740 4 client.25882 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:36:44.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.388+0000 7fed3044e740 1 -- 172.21.15.90:0/2808908720 --> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556cf7a2e040 con 0x556cf7b342c0 2024-04-02T14:36:44.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.388+0000 7fed26a00640 1 --2- 172.21.15.90:0/2808908720 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x556cf7b342c0 0x556cf7b366b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:44.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.389+0000 7fed26a00640 1 --2- 172.21.15.90:0/2808908720 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x556cf7b342c0 0x556cf7b366b0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=f2805a3d474b5929 server_cookie=74227cac37c60ebc in_seq=0 out_seq=0 2024-04-02T14:36:44.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.389+0000 7fed10c00640 10 client.25882 ms_handle_connect on v2:172.21.15.92:6836/1270635939 2024-04-02T14:36:44.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.390+0000 7fed10c00640 1 -- 172.21.15.90:0/2808908720 <== mds.3 v2:172.21.15.92:6836/1270635939 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x556cf7a2e040 con 0x556cf7b342c0 2024-04-02T14:36:44.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.390+0000 7fed10c00640 10 client.25882 handle_command_reply: tid=0 2024-04-02T14:36:44.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.390+0000 7fed3044e740 1 client.25882 shutdown 2024-04-02T14:36:44.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.390+0000 7fed3044e740 1 -- 172.21.15.90:0/2808908720 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556cf7b0a9f0 msgr2=0x556cf7b10600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:44.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.390+0000 7fed3044e740 1 --2- 172.21.15.90:0/2808908720 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556cf7b0a9f0 0x556cf7b10600 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fed14000c00 tx=0x7fed14005210 comp rx=0 tx=0).stop 2024-04-02T14:36:44.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.391+0000 7fed3044e740 1 -- 172.21.15.90:0/2808908720 shutdown_connections 2024-04-02T14:36:44.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.391+0000 7fed3044e740 1 --2- 172.21.15.90:0/2808908720 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x556cf7b342c0 0x556cf7b366b0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:44.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.391+0000 7fed10c00640 0 client.25882 ms_handle_reset on v2:172.21.15.92:6836/1270635939 2024-04-02T14:36:44.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.391+0000 7fed3044e740 1 --2- 172.21.15.90:0/2808908720 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556cf7b13080 0x556cf7b10b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:44.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.391+0000 7fed3044e740 1 --2- 172.21.15.90:0/2808908720 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556cf7b0a9f0 0x556cf7b10600 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:44.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.391+0000 7fed3044e740 1 --2- 172.21.15.90:0/2808908720 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556cf7b09d10 0x556cf7b100c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:44.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.391+0000 7fed3044e740 1 -- 172.21.15.90:0/2808908720 >> 172.21.15.90:0/2808908720 conn(0x556cf7ad0a50 msgr2=0x556cf7b064b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:36:44.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.391+0000 7fed3044e740 1 -- 172.21.15.90:0/2808908720 shutdown_connections 2024-04-02T14:36:44.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.392+0000 7fed3044e740 1 -- 172.21.15.90:0/2808908720 wait complete. 2024-04-02T14:36:44.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.392+0000 7fed3044e740 20 client.25882 trim_cache size 0 max 16384 2024-04-02T14:36:44.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.413+0000 7fed3044e740 1 Processor -- start 2024-04-02T14:36:44.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.413+0000 7fed3044e740 1 -- start start 2024-04-02T14:36:44.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.413+0000 7fed3044e740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556cf7b09d10 0x556cf7b2fb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:44.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.414+0000 7fed3044e740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556cf7b0a9f0 0x556cf7b03720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:44.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.414+0000 7fed3044e740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556cf7b13080 0x556cf7b3bc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:44.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.414+0000 7fed3044e740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fed281b3d40 con 0x556cf7b0a9f0 2024-04-02T14:36:44.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.414+0000 7fed3044e740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fed281b5880 con 0x556cf7b13080 2024-04-02T14:36:44.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.414+0000 7fed3044e740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fed281b4410 con 0x556cf7b09d10 2024-04-02T14:36:44.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.414+0000 7fed27400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556cf7b13080 0x556cf7b3bc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:44.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.414+0000 7fed27400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556cf7b13080 0x556cf7b3bc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:57392/0 (socket says 172.21.15.90:57392) 2024-04-02T14:36:44.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.414+0000 7fed26a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556cf7b09d10 0x556cf7b2fb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:44.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.414+0000 7fed26000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556cf7b0a9f0 0x556cf7b03720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:44.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.414+0000 7fed26a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556cf7b09d10 0x556cf7b2fb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:41260/0 (socket says 172.21.15.90:41260) 2024-04-02T14:36:44.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.414+0000 7fed27400640 1 -- 172.21.15.90:0/4092768395 learned_addr learned my addr 172.21.15.90:0/4092768395 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:36:44.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.414+0000 7fed26000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556cf7b0a9f0 0x556cf7b03720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:49988/0 (socket says 172.21.15.90:49988) 2024-04-02T14:36:44.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.415+0000 7fed27400640 1 -- 172.21.15.90:0/4092768395 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556cf7b09d10 msgr2=0x556cf7b2fb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:36:44.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.415+0000 7fed27400640 1 --2- 172.21.15.90:0/4092768395 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556cf7b09d10 0x556cf7b2fb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:44.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.415+0000 7fed27400640 1 -- 172.21.15.90:0/4092768395 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556cf7b0a9f0 msgr2=0x556cf7b03720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:36:44.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.415+0000 7fed27400640 1 --2- 172.21.15.90:0/4092768395 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556cf7b0a9f0 0x556cf7b03720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:44.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.415+0000 7fed27400640 1 -- 172.21.15.90:0/4092768395 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556cf7b08d40 con 0x556cf7b13080 2024-04-02T14:36:44.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.415+0000 7fed27400640 1 --2- 172.21.15.90:0/4092768395 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556cf7b13080 0x556cf7b3bc80 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fed18010ee0 tx=0x7fed18013000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6e7615c5db4856c9 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:36:44.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.416+0000 7fed13400640 1 -- 172.21.15.90:0/4092768395 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed18005220 con 0x556cf7b13080 2024-04-02T14:36:44.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.416+0000 7fed13400640 1 -- 172.21.15.90:0/4092768395 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fed18024070 con 0x556cf7b13080 2024-04-02T14:36:44.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.416+0000 7fed13400640 1 -- 172.21.15.90:0/4092768395 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed18020670 con 0x556cf7b13080 2024-04-02T14:36:44.420 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:36:44 smithi090 ceph-mon[31674]: from='client.? 172.21.15.90:0/798078648' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-02T14:36:44.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.419+0000 7fed3044e740 1 -- 172.21.15.90:0/4092768395 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556cf7b13080 msgr2=0x556cf7b3bc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:44.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.419+0000 7fed3044e740 1 --2- 172.21.15.90:0/4092768395 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556cf7b13080 0x556cf7b3bc80 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fed18010ee0 tx=0x7fed18013000 comp rx=0 tx=0).stop 2024-04-02T14:36:44.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.420+0000 7fed3044e740 1 -- 172.21.15.90:0/4092768395 shutdown_connections 2024-04-02T14:36:44.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.420+0000 7fed3044e740 1 --2- 172.21.15.90:0/4092768395 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556cf7b13080 0x556cf7b3bc80 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:44.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.420+0000 7fed3044e740 1 --2- 172.21.15.90:0/4092768395 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556cf7b0a9f0 0x556cf7b03720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:44.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.420+0000 7fed3044e740 1 --2- 172.21.15.90:0/4092768395 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556cf7b09d10 0x556cf7b2fb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:44.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.420+0000 7fed3044e740 1 -- 172.21.15.90:0/4092768395 >> 172.21.15.90:0/4092768395 conn(0x556cf7ad0a50 msgr2=0x556cf7b03010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:36:44.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.420+0000 7fed3044e740 1 -- 172.21.15.90:0/4092768395 shutdown_connections 2024-04-02T14:36:44.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.421+0000 7fed3044e740 1 -- 172.21.15.90:0/4092768395 wait complete. 2024-04-02T14:36:44.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.422+0000 7fed3044e740 1 -- start start 2024-04-02T14:36:44.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.422+0000 7fed3044e740 1 Processor -- start 2024-04-02T14:36:44.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.425+0000 7fed3044e740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556cf7b09d10 0x556cf7b2c200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:44.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.425+0000 7fed3044e740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556cf7b0a9f0 0x556cf7b0c910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:44.429 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.425+0000 7fed3044e740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556cf7b13080 0x556cf7b0ce50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:44.429 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.425+0000 7fed3044e740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fed281b51b0 con 0x556cf7b09d10 2024-04-02T14:36:44.429 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.425+0000 7fed3044e740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fed281b4ae0 con 0x556cf7b13080 2024-04-02T14:36:44.429 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.425+0000 7fed3044e740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fed281b1d20 con 0x556cf7b0a9f0 2024-04-02T14:36:44.429 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.425+0000 7fed3044e740 10 client.0 _ll_register_callbacks cb 0x7fed14004df0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:36:44.429 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.426+0000 7fed27400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556cf7b13080 0x556cf7b0ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:44.429 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.426+0000 7fed27400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556cf7b13080 0x556cf7b0ce50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:57396/0 (socket says 172.21.15.90:57396) 2024-04-02T14:36:44.429 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.426+0000 7fed27400640 1 -- 172.21.15.90:0/3712529888 learned_addr learned my addr 172.21.15.90:0/3712529888 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:36:44.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.427+0000 7fed26a00640 1 --2- 172.21.15.90:0/3712529888 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556cf7b09d10 0x556cf7b2c200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:44.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.428+0000 7fed27400640 1 -- 172.21.15.90:0/3712529888 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556cf7b0a9f0 msgr2=0x556cf7b0c910 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-02T14:36:44.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.428+0000 7fed27400640 1 --2- 172.21.15.90:0/3712529888 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556cf7b0a9f0 0x556cf7b0c910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:44.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.428+0000 7fed27400640 1 -- 172.21.15.90:0/3712529888 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556cf7b09d10 msgr2=0x556cf7b2c200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:36:44.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.428+0000 7fed27400640 1 --2- 172.21.15.90:0/3712529888 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556cf7b09d10 0x556cf7b2c200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:44.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.428+0000 7fed27400640 1 -- 172.21.15.90:0/3712529888 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556cf7b0d390 con 0x556cf7b13080 2024-04-02T14:36:44.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.430+0000 7fed27400640 1 --2- 172.21.15.90:0/3712529888 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556cf7b13080 0x556cf7b0ce50 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fed18010eb0 tx=0x7fed18010fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e3b080b7a2e6d73b server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:36:44.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.430+0000 7fed10c00640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:36:44.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.431+0000 7fed10c00640 1 -- 172.21.15.90:0/3712529888 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed18037060 con 0x556cf7b13080 2024-04-02T14:36:44.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.431+0000 7fed3044e740 1 -- 172.21.15.90:0/3712529888 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fed18024070 con 0x556cf7b13080 2024-04-02T14:36:44.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.432+0000 7fed10c00640 1 -- 172.21.15.90:0/3712529888 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7fed18004c00 con 0x556cf7b13080 2024-04-02T14:36:44.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.432+0000 7fed10c00640 1 -- 172.21.15.90:0/3712529888 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fed18010b60 con 0x556cf7b13080 2024-04-02T14:36:44.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.432+0000 7fed3044e740 10 client.25970 fetch_fsmap learned FSMap version 27 2024-04-02T14:36:44.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.432+0000 7fed3044e740 1 -- 172.21.15.90:0/3712529888 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x556cf7b9a210 con 0x556cf7b13080 2024-04-02T14:36:44.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.432+0000 7fed10c00640 1 -- 172.21.15.90:0/3712529888 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fed18024070 con 0x556cf7b13080 2024-04-02T14:36:44.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.433+0000 7fed10c00640 1 -- 172.21.15.90:0/3712529888 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fed1800f040 con 0x556cf7b13080 2024-04-02T14:36:44.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.433+0000 7fed3044e740 10 client.25970 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:36:44.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.433+0000 7fed3044e740 10 client.25970 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-04-02T14:36:44.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.433+0000 7fed3044e740 20 client.25970 populate_metadata read hostname 'smithi090' 2024-04-02T14:36:44.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.433+0000 7fed3044e740 1 --2- 172.21.15.90:0/3712529888 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x556cf7b9a830 0x556cf7b9cc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:44.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.433+0000 7fed3044e740 4 client.25970 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-02T14:36:44.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.434+0000 7fed3044e740 1 -- 172.21.15.90:0/3712529888 --> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] -- command(tid 0: {"prefix": "damage ls"}) -- 0x556cf7a7bc30 con 0x556cf7b9a830 2024-04-02T14:36:44.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.435+0000 7fed26a00640 1 --2- 172.21.15.90:0/3712529888 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x556cf7b9a830 0x556cf7b9cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:44.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.436+0000 7fed26a00640 1 --2- 172.21.15.90:0/3712529888 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x556cf7b9a830 0x556cf7b9cc50 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=c31512a9aca77a17 server_cookie=b5dac7f70593fa20 in_seq=0 out_seq=0 2024-04-02T14:36:44.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.436+0000 7fed10c00640 10 client.25970 ms_handle_connect on v2:172.21.15.92:6836/1270635939 2024-04-02T14:36:44.440 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.436+0000 7fed10c00640 1 -- 172.21.15.90:0/3712529888 <== mds.3 v2:172.21.15.92:6836/1270635939 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x556cf7a7bc30 con 0x556cf7b9a830 2024-04-02T14:36:44.440 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.436+0000 7fed10c00640 10 client.25970 handle_command_reply: tid=0 2024-04-02T14:36:44.440 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.437+0000 7fed3044e740 1 client.25970 shutdown 2024-04-02T14:36:44.440 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.437+0000 7fed3044e740 1 -- 172.21.15.90:0/3712529888 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556cf7b13080 msgr2=0x556cf7b0ce50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:44.440 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.437+0000 7fed3044e740 1 --2- 172.21.15.90:0/3712529888 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556cf7b13080 0x556cf7b0ce50 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fed18010eb0 tx=0x7fed18010fd0 comp rx=0 tx=0).stop 2024-04-02T14:36:44.440 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.437+0000 7fed3044e740 1 -- 172.21.15.90:0/3712529888 shutdown_connections 2024-04-02T14:36:44.440 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.437+0000 7fed3044e740 1 --2- 172.21.15.90:0/3712529888 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x556cf7b9a830 0x556cf7b9cc50 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:44.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.437+0000 7fed3044e740 1 --2- 172.21.15.90:0/3712529888 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556cf7b13080 0x556cf7b0ce50 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:44.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.437+0000 7fed3044e740 1 --2- 172.21.15.90:0/3712529888 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556cf7b0a9f0 0x556cf7b0c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:44.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.437+0000 7fed3044e740 1 --2- 172.21.15.90:0/3712529888 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556cf7b09d10 0x556cf7b2c200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:44.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.437+0000 7fed3044e740 1 -- 172.21.15.90:0/3712529888 >> 172.21.15.90:0/3712529888 conn(0x556cf7ad0a50 msgr2=0x556cf7b038a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:36:44.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.437+0000 7fed10c00640 0 client.25970 ms_handle_reset on v2:172.21.15.92:6836/1270635939 2024-04-02T14:36:44.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.438+0000 7fed3044e740 1 -- 172.21.15.90:0/3712529888 shutdown_connections 2024-04-02T14:36:44.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.440+0000 7fed3044e740 1 -- 172.21.15.90:0/3712529888 wait complete. 2024-04-02T14:36:44.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.441+0000 7fed3044e740 20 client.25970 trim_cache size 0 max 16384 2024-04-02T14:36:44.444 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-02T14:36:44.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.442+0000 7fed2d800640 1 -- 172.21.15.90:0/2164120341 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fed0c0a3da0 msgr2=0x7fed0c0a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:44.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.442+0000 7fed2d800640 1 --2- 172.21.15.90:0/2164120341 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fed0c0a3da0 0x7fed0c0a6260 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fed1800ead0 tx=0x7fed18005e50 comp rx=0 tx=0).stop 2024-04-02T14:36:44.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.442+0000 7fed2d800640 1 -- 172.21.15.90:0/2164120341 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fed2806b420 msgr2=0x7fed281a2d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:44.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.442+0000 7fed2d800640 1 --2- 172.21.15.90:0/2164120341 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fed2806b420 0x7fed281a2d50 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fed20004af0 tx=0x7fed20035ed0 comp rx=0 tx=0).stop 2024-04-02T14:36:44.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.442+0000 7fed2d800640 1 -- 172.21.15.90:0/2164120341 shutdown_connections 2024-04-02T14:36:44.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.442+0000 7fed2d800640 1 --2- 172.21.15.90:0/2164120341 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fed0c0a3da0 0x7fed0c0a6260 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:44.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.442+0000 7fed2d800640 1 --2- 172.21.15.90:0/2164120341 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fed28116ad0 0x7fed281a7610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:44.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.442+0000 7fed2d800640 1 --2- 172.21.15.90:0/2164120341 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fed2806bd40 0x7fed281a3290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:44.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.442+0000 7fed2d800640 1 --2- 172.21.15.90:0/2164120341 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fed2806b420 0x7fed281a2d50 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:44.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.442+0000 7fed2d800640 1 -- 172.21.15.90:0/2164120341 >> 172.21.15.90:0/2164120341 conn(0x7fed280730b0 msgr2=0x7fed2806f740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:36:44.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.443+0000 7fed2d800640 1 -- 172.21.15.90:0/2164120341 shutdown_connections 2024-04-02T14:36:44.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.444+0000 7fed2d800640 1 -- 172.21.15.90:0/2164120341 wait complete. 2024-04-02T14:36:44.476 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-02T14:36:44.687 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:36:44 smithi177 ceph-mon[36431]: from='client.? 172.21.15.90:0/798078648' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-02T14:36:44.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.771+0000 7ff800a00640 1 -- 172.21.15.90:0/3719222530 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7f4005220 con 0x7ff7fc004cb0 2024-04-02T14:36:44.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.771+0000 7ff803200640 1 -- 172.21.15.90:0/3719222530 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff7fc004cb0 msgr2=0x7ff7fc005090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:44.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.771+0000 7ff803200640 1 --2- 172.21.15.90:0/3719222530 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff7fc004cb0 0x7ff7fc005090 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7ff7f4009900 tx=0x7ff7f402bc80 comp rx=0 tx=0).stop 2024-04-02T14:36:44.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.771+0000 7ff803200640 1 -- 172.21.15.90:0/3719222530 shutdown_connections 2024-04-02T14:36:44.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.771+0000 7ff803200640 1 --2- 172.21.15.90:0/3719222530 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff7fc0a2a00 0x7ff7fc0a4df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:44.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.771+0000 7ff803200640 1 --2- 172.21.15.90:0/3719222530 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff7fc0055d0 0x7ff7fc0a24c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:44.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.771+0000 7ff803200640 1 --2- 172.21.15.90:0/3719222530 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff7fc004cb0 0x7ff7fc005090 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:44.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.771+0000 7ff803200640 1 -- 172.21.15.90:0/3719222530 >> 172.21.15.90:0/3719222530 conn(0x7ff7fc00a740 msgr2=0x7ff7fc00ab50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:36:44.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.772+0000 7ff803200640 1 -- 172.21.15.90:0/3719222530 shutdown_connections 2024-04-02T14:36:44.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.772+0000 7ff803200640 1 -- 172.21.15.90:0/3719222530 wait complete. 2024-04-02T14:36:44.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.773+0000 7ff803200640 1 Processor -- start 2024-04-02T14:36:44.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.773+0000 7ff803200640 1 -- start start 2024-04-02T14:36:44.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.774+0000 7ff803200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff7fc004cb0 0x7ff7fc0a2330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:44.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.774+0000 7ff803200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff7fc0055d0 0x7ff7fc09f420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:44.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.774+0000 7ff803200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff7fc0a2a00 0x7ff7fc09f960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:44.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.775+0000 7ff803200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7ff7fc0a83f0 con 0x7ff7fc004cb0 2024-04-02T14:36:44.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.775+0000 7ff803200640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7ff7fc0a8270 con 0x7ff7fc0055d0 2024-04-02T14:36:44.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.775+0000 7ff803200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7ff7fc0a8570 con 0x7ff7fc0a2a00 2024-04-02T14:36:44.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.775+0000 7ff802800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff7fc0a2a00 0x7ff7fc09f960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:44.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.775+0000 7ff802800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff7fc0a2a00 0x7ff7fc09f960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:41286/0 (socket says 172.21.15.90:41286) 2024-04-02T14:36:44.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.775+0000 7ff802800640 1 -- 172.21.15.90:0/157314655 learned_addr learned my addr 172.21.15.90:0/157314655 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:36:44.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.775+0000 7ff801e00640 1 --2- 172.21.15.90:0/157314655 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff7fc004cb0 0x7ff7fc0a2330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:44.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.775+0000 7ff801400640 1 --2- 172.21.15.90:0/157314655 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff7fc0055d0 0x7ff7fc09f420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:44.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.776+0000 7ff802800640 1 -- 172.21.15.90:0/157314655 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff7fc0055d0 msgr2=0x7ff7fc09f420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:44.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.776+0000 7ff802800640 1 --2- 172.21.15.90:0/157314655 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff7fc0055d0 0x7ff7fc09f420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:44.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.776+0000 7ff802800640 1 -- 172.21.15.90:0/157314655 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff7fc004cb0 msgr2=0x7ff7fc0a2330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:44.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.776+0000 7ff802800640 1 --2- 172.21.15.90:0/157314655 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff7fc004cb0 0x7ff7fc0a2330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:44.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.777+0000 7ff802800640 1 -- 172.21.15.90:0/157314655 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7fc0a01f0 con 0x7ff7fc0a2a00 2024-04-02T14:36:44.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.777+0000 7ff801e00640 1 --2- 172.21.15.90:0/157314655 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff7fc004cb0 0x7ff7fc0a2330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:36:44.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.777+0000 7ff802800640 1 --2- 172.21.15.90:0/157314655 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff7fc0a2a00 0x7ff7fc09f960 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7ff804050610 tx=0x7ff804069cf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:36:44.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.777+0000 7ff7faa00640 1 -- 172.21.15.90:0/157314655 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8040798a0 con 0x7ff7fc0a2a00 2024-04-02T14:36:44.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.777+0000 7ff7faa00640 1 -- 172.21.15.90:0/157314655 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff80407f420 con 0x7ff7fc0a2a00 2024-04-02T14:36:44.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.778+0000 7ff803200640 1 -- 172.21.15.90:0/157314655 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff7fc13f1a0 con 0x7ff7fc0a2a00 2024-04-02T14:36:44.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.778+0000 7ff7faa00640 1 -- 172.21.15.90:0/157314655 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff80407e5e0 con 0x7ff7fc0a2a00 2024-04-02T14:36:44.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.778+0000 7ff803200640 1 -- 172.21.15.90:0/157314655 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7fc13f6a0 con 0x7ff7fc0a2a00 2024-04-02T14:36:44.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.782+0000 7ff7faa00640 1 -- 172.21.15.90:0/157314655 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7ff804086030 con 0x7ff7fc0a2a00 2024-04-02T14:36:44.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.786+0000 7ff7faa00640 1 --2- 172.21.15.90:0/157314655 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7ff7e80a3e20 0x7ff7e80a62e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:44.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.787+0000 7ff801e00640 1 --2- 172.21.15.90:0/157314655 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7ff7e80a3e20 0x7ff7e80a62e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:44.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.787+0000 7ff7faa00640 1 -- 172.21.15.90:0/157314655 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7ff80413b900 con 0x7ff7fc0a2a00 2024-04-02T14:36:44.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.789+0000 7ff801e00640 1 --2- 172.21.15.90:0/157314655 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7ff7e80a3e20 0x7ff7e80a62e0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7ff7f4006010 tx=0x7ff7f4005b70 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:36:44.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.792+0000 7ff80b851740 1 Processor -- start 2024-04-02T14:36:44.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.792+0000 7ff80b851740 1 -- start start 2024-04-02T14:36:44.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.793+0000 7ff80b851740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ed591f2ba0 0x55ed591f18f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:44.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.793+0000 7ff80b851740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ed591f1ec0 0x55ed591f2340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:44.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.793+0000 7ff80b851740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ed591fb230 0x55ed591fd620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:44.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.793+0000 7ff80b851740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55ed59132400 con 0x55ed591f1ec0 2024-04-02T14:36:44.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.793+0000 7ff80b851740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55ed59116f10 con 0x55ed591fb230 2024-04-02T14:36:44.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.793+0000 7ff80b851740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55ed591566f0 con 0x55ed591f2ba0 2024-04-02T14:36:44.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.793+0000 7ff802800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ed591fb230 0x55ed591fd620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:44.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.793+0000 7ff801e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ed591f2ba0 0x55ed591f18f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:44.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.793+0000 7ff802800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ed591fb230 0x55ed591fd620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:57426/0 (socket says 172.21.15.90:57426) 2024-04-02T14:36:44.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.793+0000 7ff802800640 1 -- 172.21.15.90:0/2814876711 learned_addr learned my addr 172.21.15.90:0/2814876711 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:36:44.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.794+0000 7ff802800640 1 -- 172.21.15.90:0/2814876711 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ed591f2ba0 msgr2=0x55ed591f18f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:36:44.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.794+0000 7ff802800640 1 --2- 172.21.15.90:0/2814876711 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ed591f2ba0 0x55ed591f18f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:44.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.794+0000 7ff802800640 1 -- 172.21.15.90:0/2814876711 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ed591f1ec0 msgr2=0x55ed591f2340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:36:44.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.794+0000 7ff802800640 1 --2- 172.21.15.90:0/2814876711 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ed591f1ec0 0x55ed591f2340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:44.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.794+0000 7ff802800640 1 -- 172.21.15.90:0/2814876711 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ed58e6a3f0 con 0x55ed591fb230 2024-04-02T14:36:44.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.794+0000 7ff802800640 1 --2- 172.21.15.90:0/2814876711 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ed591fb230 0x55ed591fd620 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7ff804103e50 tx=0x7ff804103d10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ef9db698f80adb26 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:36:44.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.794+0000 7ff7ef000640 1 -- 172.21.15.90:0/2814876711 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8041105d0 con 0x55ed591fb230 2024-04-02T14:36:44.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.795+0000 7ff7ef000640 1 -- 172.21.15.90:0/2814876711 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff804104060 con 0x55ed591fb230 2024-04-02T14:36:44.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.795+0000 7ff7ef000640 1 -- 172.21.15.90:0/2814876711 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff804110140 con 0x55ed591fb230 2024-04-02T14:36:44.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.795+0000 7ff80b851740 1 -- 172.21.15.90:0/2814876711 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ed591fb230 msgr2=0x55ed591fd620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:44.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.796+0000 7ff80b851740 1 --2- 172.21.15.90:0/2814876711 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ed591fb230 0x55ed591fd620 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7ff804103e50 tx=0x7ff804103d10 comp rx=0 tx=0).stop 2024-04-02T14:36:44.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.796+0000 7ff80b851740 1 -- 172.21.15.90:0/2814876711 shutdown_connections 2024-04-02T14:36:44.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.796+0000 7ff80b851740 1 --2- 172.21.15.90:0/2814876711 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ed591fb230 0x55ed591fd620 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:44.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.796+0000 7ff80b851740 1 --2- 172.21.15.90:0/2814876711 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ed591f1ec0 0x55ed591f2340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:44.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.796+0000 7ff80b851740 1 --2- 172.21.15.90:0/2814876711 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ed591f2ba0 0x55ed591f18f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:44.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.796+0000 7ff80b851740 1 -- 172.21.15.90:0/2814876711 >> 172.21.15.90:0/2814876711 conn(0x55ed591b8a50 msgr2=0x55ed591ec910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:36:44.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.796+0000 7ff80b851740 1 -- 172.21.15.90:0/2814876711 shutdown_connections 2024-04-02T14:36:44.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.796+0000 7ff80b851740 1 -- 172.21.15.90:0/2814876711 wait complete. 2024-04-02T14:36:44.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.797+0000 7ff80b851740 1 -- start start 2024-04-02T14:36:44.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.797+0000 7ff80b851740 1 Processor -- start 2024-04-02T14:36:44.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.798+0000 7ff80b851740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ed591f1ec0 0x55ed591f81f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:44.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.798+0000 7ff80b851740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ed591f2ba0 0x55ed591f8730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:44.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.798+0000 7ff80b851740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ed591fb230 0x55ed591f8c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:44.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.798+0000 7ff80b851740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55ed591526a0 con 0x55ed591fb230 2024-04-02T14:36:44.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.798+0000 7ff80b851740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55ed590fe980 con 0x55ed591f2ba0 2024-04-02T14:36:44.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.798+0000 7ff80b851740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55ed59111cb0 con 0x55ed591f1ec0 2024-04-02T14:36:44.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.798+0000 7ff80b851740 10 client.0 _ll_register_callbacks cb 0x55ed591116a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:36:44.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.798+0000 7ff801e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ed591f1ec0 0x55ed591f81f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:44.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.798+0000 7ff801e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ed591f1ec0 0x55ed591f81f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:41300/0 (socket says 172.21.15.90:41300) 2024-04-02T14:36:44.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.798+0000 7ff801e00640 1 -- 172.21.15.90:0/3116100240 learned_addr learned my addr 172.21.15.90:0/3116100240 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:36:44.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.799+0000 7ff802800640 1 --2- 172.21.15.90:0/3116100240 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ed591fb230 0x55ed591f8c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:44.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.799+0000 7ff801e00640 1 -- 172.21.15.90:0/3116100240 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ed591f2ba0 msgr2=0x55ed591f8730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:36:44.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.799+0000 7ff801400640 1 --2- 172.21.15.90:0/3116100240 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ed591f2ba0 0x55ed591f8730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:44.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.799+0000 7ff801e00640 1 --2- 172.21.15.90:0/3116100240 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ed591f2ba0 0x55ed591f8730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:44.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.799+0000 7ff801e00640 1 -- 172.21.15.90:0/3116100240 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ed591fb230 msgr2=0x55ed591f8c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:36:44.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.799+0000 7ff801e00640 1 --2- 172.21.15.90:0/3116100240 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ed591fb230 0x55ed591f8c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:44.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.799+0000 7ff801e00640 1 -- 172.21.15.90:0/3116100240 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ed58f347d0 con 0x55ed591f1ec0 2024-04-02T14:36:44.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.799+0000 7ff801400640 1 --2- 172.21.15.90:0/3116100240 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ed591f2ba0 0x55ed591f8730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:36:44.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.799+0000 7ff801e00640 1 --2- 172.21.15.90:0/3116100240 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ed591f1ec0 0x55ed591f81f0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7ff7f4009900 tx=0x7ff7f4009260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=17e8cca50a03f0df server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:36:44.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.800+0000 7ff7e3400640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:36:44.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.800+0000 7ff7e3400640 1 -- 172.21.15.90:0/3116100240 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7f4052030 con 0x55ed591f1ec0 2024-04-02T14:36:44.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.800+0000 7ff7e3400640 1 -- 172.21.15.90:0/3116100240 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7ff7f403a040 con 0x55ed591f1ec0 2024-04-02T14:36:44.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.801+0000 7ff80b851740 1 -- 172.21.15.90:0/3116100240 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff804104060 con 0x55ed591f1ec0 2024-04-02T14:36:44.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.802+0000 7ff7e3400640 1 -- 172.21.15.90:0/3116100240 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7ff7f4005490 con 0x55ed591f1ec0 2024-04-02T14:36:44.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.802+0000 7ff80b851740 10 client.25894 fetch_fsmap learned FSMap version 27 2024-04-02T14:36:44.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.802+0000 7ff80b851740 1 -- 172.21.15.90:0/3116100240 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55ed5920a150 con 0x55ed591f1ec0 2024-04-02T14:36:44.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.803+0000 7ff7e3400640 1 -- 172.21.15.90:0/3116100240 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff804104060 con 0x55ed591f1ec0 2024-04-02T14:36:44.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.803+0000 7ff7e3400640 1 -- 172.21.15.90:0/3116100240 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7ff7f404b050 con 0x55ed591f1ec0 2024-04-02T14:36:44.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.803+0000 7ff80b851740 10 client.25894 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:36:44.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.803+0000 7ff80b851740 10 client.25894 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-04-02T14:36:44.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.803+0000 7ff80b851740 20 client.25894 populate_metadata read hostname 'smithi090' 2024-04-02T14:36:44.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.803+0000 7ff80b851740 1 --2- 172.21.15.90:0/3116100240 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x55ed5920b6b0 0x55ed5920dae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:44.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.803+0000 7ff80b851740 4 client.25894 mds_command: new command op to 24547 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:36:44.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.803+0000 7ff80b851740 1 -- 172.21.15.90:0/3116100240 --> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ed59116040 con 0x55ed5920b6b0 2024-04-02T14:36:44.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.804+0000 7ff801400640 1 --2- 172.21.15.90:0/3116100240 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x55ed5920b6b0 0x55ed5920dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:44.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.805+0000 7ff801400640 1 --2- 172.21.15.90:0/3116100240 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x55ed5920b6b0 0x55ed5920dae0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=cf3262482cdea5d5 server_cookie=c25f82558f50c515 in_seq=0 out_seq=0 2024-04-02T14:36:44.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.806+0000 7ff7e3400640 10 client.25894 ms_handle_connect on v2:172.21.15.177:6838/664895663 2024-04-02T14:36:44.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.806+0000 7ff7e3400640 1 -- 172.21.15.90:0/3116100240 <== mds.2 v2:172.21.15.177:6838/664895663 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ed59116040 con 0x55ed5920b6b0 2024-04-02T14:36:44.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.806+0000 7ff7e3400640 10 client.25894 handle_command_reply: tid=0 2024-04-02T14:36:44.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.807+0000 7ff80b851740 1 client.25894 shutdown 2024-04-02T14:36:44.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.808+0000 7ff80b851740 1 -- 172.21.15.90:0/3116100240 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ed591f1ec0 msgr2=0x55ed591f81f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:44.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.808+0000 7ff80b851740 1 --2- 172.21.15.90:0/3116100240 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ed591f1ec0 0x55ed591f81f0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7ff7f4009900 tx=0x7ff7f4009260 comp rx=0 tx=0).stop 2024-04-02T14:36:44.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.808+0000 7ff80b851740 1 -- 172.21.15.90:0/3116100240 shutdown_connections 2024-04-02T14:36:44.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.808+0000 7ff80b851740 1 --2- 172.21.15.90:0/3116100240 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x55ed5920b6b0 0x55ed5920dae0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:44.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.808+0000 7ff80b851740 1 --2- 172.21.15.90:0/3116100240 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ed591fb230 0x55ed591f8c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:44.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.808+0000 7ff80b851740 1 --2- 172.21.15.90:0/3116100240 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ed591f2ba0 0x55ed591f8730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:44.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.808+0000 7ff7e3400640 0 client.25894 ms_handle_reset on v2:172.21.15.177:6838/664895663 2024-04-02T14:36:44.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.808+0000 7ff80b851740 1 --2- 172.21.15.90:0/3116100240 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ed591f1ec0 0x55ed591f81f0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:44.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.808+0000 7ff80b851740 1 -- 172.21.15.90:0/3116100240 >> 172.21.15.90:0/3116100240 conn(0x55ed591b8a50 msgr2=0x55ed591ee660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:36:44.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.809+0000 7ff80b851740 1 -- 172.21.15.90:0/3116100240 shutdown_connections 2024-04-02T14:36:44.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.809+0000 7ff80b851740 1 -- 172.21.15.90:0/3116100240 wait complete. 2024-04-02T14:36:44.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.809+0000 7ff80b851740 20 client.25894 trim_cache size 0 max 16384 2024-04-02T14:36:44.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.845+0000 7ff80b851740 1 Processor -- start 2024-04-02T14:36:44.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.845+0000 7ff80b851740 1 -- start start 2024-04-02T14:36:44.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.846+0000 7ff80b851740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ed591f1ec0 0x55ed59206d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:44.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.846+0000 7ff80b851740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ed591f2ba0 0x55ed591eb970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:44.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.846+0000 7ff80b851740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ed591fb230 0x55ed592133f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:44.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.846+0000 7ff80b851740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7ff7fc148100 con 0x55ed591f2ba0 2024-04-02T14:36:44.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.846+0000 7ff80b851740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7ff7fc149c40 con 0x55ed591f1ec0 2024-04-02T14:36:44.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.846+0000 7ff80b851740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7ff7fc1487d0 con 0x55ed591fb230 2024-04-02T14:36:44.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.847+0000 7ff802800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ed591fb230 0x55ed592133f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:44.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.847+0000 7ff802800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ed591fb230 0x55ed592133f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:41310/0 (socket says 172.21.15.90:41310) 2024-04-02T14:36:44.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.847+0000 7ff801400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ed591f2ba0 0x55ed591eb970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:44.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.847+0000 7ff802800640 1 -- 172.21.15.90:0/1224934126 learned_addr learned my addr 172.21.15.90:0/1224934126 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:36:44.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.847+0000 7ff801e00640 1 --2- 172.21.15.90:0/1224934126 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ed591f1ec0 0x55ed59206d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:44.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.848+0000 7ff801e00640 1 -- 172.21.15.90:0/1224934126 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ed591fb230 msgr2=0x55ed592133f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:36:44.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.848+0000 7ff801e00640 1 --2- 172.21.15.90:0/1224934126 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ed591fb230 0x55ed592133f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:44.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.848+0000 7ff801e00640 1 -- 172.21.15.90:0/1224934126 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ed591f2ba0 msgr2=0x55ed591eb970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:36:44.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.848+0000 7ff801e00640 1 --2- 172.21.15.90:0/1224934126 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ed591f2ba0 0x55ed591eb970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:44.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.848+0000 7ff801e00640 1 -- 172.21.15.90:0/1224934126 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ed5920e160 con 0x55ed591f1ec0 2024-04-02T14:36:44.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.848+0000 7ff801400640 1 --2- 172.21.15.90:0/1224934126 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ed591f2ba0 0x55ed591eb970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:36:44.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.848+0000 7ff802800640 1 --2- 172.21.15.90:0/1224934126 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ed591fb230 0x55ed592133f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:36:44.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.849+0000 7ff801e00640 1 --2- 172.21.15.90:0/1224934126 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ed591f1ec0 0x55ed59206d50 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7ff7f4009900 tx=0x7ff7f4009680 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e1738f894dde51fe server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:36:44.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.849+0000 7ff7ee600640 1 -- 172.21.15.90:0/1224934126 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7f4052070 con 0x55ed591f1ec0 2024-04-02T14:36:44.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.850+0000 7ff7ee600640 1 -- 172.21.15.90:0/1224934126 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff7f4046070 con 0x55ed591f1ec0 2024-04-02T14:36:44.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.850+0000 7ff7ee600640 1 -- 172.21.15.90:0/1224934126 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7f4003da0 con 0x55ed591f1ec0 2024-04-02T14:36:44.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.851+0000 7ff80b851740 1 -- 172.21.15.90:0/1224934126 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ed591f1ec0 msgr2=0x55ed59206d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:44.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.851+0000 7ff80b851740 1 --2- 172.21.15.90:0/1224934126 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ed591f1ec0 0x55ed59206d50 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7ff7f4009900 tx=0x7ff7f4009680 comp rx=0 tx=0).stop 2024-04-02T14:36:44.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.851+0000 7ff80b851740 1 -- 172.21.15.90:0/1224934126 shutdown_connections 2024-04-02T14:36:44.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.851+0000 7ff80b851740 1 --2- 172.21.15.90:0/1224934126 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ed591fb230 0x55ed592133f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:44.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.851+0000 7ff80b851740 1 --2- 172.21.15.90:0/1224934126 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ed591f2ba0 0x55ed591eb970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:44.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.851+0000 7ff80b851740 1 --2- 172.21.15.90:0/1224934126 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ed591f1ec0 0x55ed59206d50 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:44.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.851+0000 7ff80b851740 1 -- 172.21.15.90:0/1224934126 >> 172.21.15.90:0/1224934126 conn(0x55ed591b8a50 msgr2=0x55ed591eb260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:36:44.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.852+0000 7ff80b851740 1 -- 172.21.15.90:0/1224934126 shutdown_connections 2024-04-02T14:36:44.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.852+0000 7ff80b851740 1 -- 172.21.15.90:0/1224934126 wait complete. 2024-04-02T14:36:44.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.852+0000 7ff80b851740 1 -- start start 2024-04-02T14:36:44.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.852+0000 7ff80b851740 1 Processor -- start 2024-04-02T14:36:44.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.853+0000 7ff80b851740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ed591f1ec0 0x55ed59202260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:44.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.853+0000 7ff80b851740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ed591f2ba0 0x55ed592027a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:44.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.853+0000 7ff80b851740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ed591fb230 0x55ed59202ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:44.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.854+0000 7ff80b851740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7ff7fc149570 con 0x55ed591fb230 2024-04-02T14:36:44.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.854+0000 7ff80b851740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7ff7fc148ea0 con 0x55ed591f2ba0 2024-04-02T14:36:44.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.854+0000 7ff80b851740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7ff7fc1460e0 con 0x55ed591f1ec0 2024-04-02T14:36:44.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.854+0000 7ff80b851740 10 client.0 _ll_register_callbacks cb 0x7ff7f40041a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:36:44.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.854+0000 7ff802800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ed591fb230 0x55ed59202ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:44.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.854+0000 7ff801400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ed591f2ba0 0x55ed592027a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:44.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.854+0000 7ff801e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ed591f1ec0 0x55ed59202260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:44.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.855+0000 7ff801e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ed591f1ec0 0x55ed59202260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:41320/0 (socket says 172.21.15.90:41320) 2024-04-02T14:36:44.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.855+0000 7ff801e00640 1 -- 172.21.15.90:0/266946357 learned_addr learned my addr 172.21.15.90:0/266946357 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:36:44.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.855+0000 7ff801e00640 1 -- 172.21.15.90:0/266946357 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ed591f2ba0 msgr2=0x55ed592027a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:36:44.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.855+0000 7ff801e00640 1 --2- 172.21.15.90:0/266946357 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ed591f2ba0 0x55ed592027a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:44.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.855+0000 7ff801e00640 1 -- 172.21.15.90:0/266946357 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ed591fb230 msgr2=0x55ed59202ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:36:44.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.855+0000 7ff801e00640 1 --2- 172.21.15.90:0/266946357 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ed591fb230 0x55ed59202ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:44.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.855+0000 7ff801e00640 1 -- 172.21.15.90:0/266946357 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ed591f0ef0 con 0x55ed591f1ec0 2024-04-02T14:36:44.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.855+0000 7ff801400640 1 --2- 172.21.15.90:0/266946357 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ed591f2ba0 0x55ed592027a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:36:44.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.855+0000 7ff802800640 1 --2- 172.21.15.90:0/266946357 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ed591fb230 0x55ed59202ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:36:44.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.856+0000 7ff801e00640 1 --2- 172.21.15.90:0/266946357 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ed591f1ec0 0x55ed59202260 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7ff7f404e7e0 tx=0x7ff7f40045b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=af3bda79c81c5634 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:36:44.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.856+0000 7ff7e3400640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:36:44.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.856+0000 7ff7e3400640 1 -- 172.21.15.90:0/266946357 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7f4065070 con 0x55ed591f1ec0 2024-04-02T14:36:44.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.856+0000 7ff7e3400640 1 -- 172.21.15.90:0/266946357 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7ff7f40037f0 con 0x55ed591f1ec0 2024-04-02T14:36:44.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.856+0000 7ff80b851740 1 -- 172.21.15.90:0/266946357 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff7f4046070 con 0x55ed591f1ec0 2024-04-02T14:36:44.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.857+0000 7ff7e3400640 1 -- 172.21.15.90:0/266946357 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7ff7f4052040 con 0x55ed591f1ec0 2024-04-02T14:36:44.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.857+0000 7ff80b851740 10 client.25900 fetch_fsmap learned FSMap version 27 2024-04-02T14:36:44.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.857+0000 7ff80b851740 1 -- 172.21.15.90:0/266946357 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55ed592833f0 con 0x55ed591f1ec0 2024-04-02T14:36:44.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.858+0000 7ff7e3400640 1 -- 172.21.15.90:0/266946357 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff7f4046070 con 0x55ed591f1ec0 2024-04-02T14:36:44.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.858+0000 7ff7e3400640 1 -- 172.21.15.90:0/266946357 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7ff7f4068020 con 0x55ed591f1ec0 2024-04-02T14:36:44.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.858+0000 7ff80b851740 10 client.25900 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:36:44.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.858+0000 7ff80b851740 10 client.25900 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-04-02T14:36:44.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.858+0000 7ff80b851740 20 client.25900 populate_metadata read hostname 'smithi090' 2024-04-02T14:36:44.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.858+0000 7ff80b851740 1 --2- 172.21.15.90:0/266946357 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x55ed59283a50 0x55ed59285e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:44.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.858+0000 7ff80b851740 4 client.25900 mds_command: new command op to 24547 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-02T14:36:44.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.858+0000 7ff80b851740 1 -- 172.21.15.90:0/266946357 --> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55ed59163c30 con 0x55ed59283a50 2024-04-02T14:36:44.862 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:36:44 smithi092 ceph-mon[36497]: from='client.? 172.21.15.90:0/798078648' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-02T14:36:44.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.859+0000 7ff801400640 1 --2- 172.21.15.90:0/266946357 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x55ed59283a50 0x55ed59285e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:44.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.860+0000 7ff801400640 1 --2- 172.21.15.90:0/266946357 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x55ed59283a50 0x55ed59285e70 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=ee3e7c23d79a14ec server_cookie=601a597ec36b44e3 in_seq=0 out_seq=0 2024-04-02T14:36:44.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.860+0000 7ff7e3400640 10 client.25900 ms_handle_connect on v2:172.21.15.177:6838/664895663 2024-04-02T14:36:44.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.861+0000 7ff7e3400640 1 -- 172.21.15.90:0/266946357 <== mds.2 v2:172.21.15.177:6838/664895663 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55ed59163c30 con 0x55ed59283a50 2024-04-02T14:36:44.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.861+0000 7ff7e3400640 10 client.25900 handle_command_reply: tid=0 2024-04-02T14:36:44.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.861+0000 7ff80b851740 1 client.25900 shutdown 2024-04-02T14:36:44.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.862+0000 7ff80b851740 1 -- 172.21.15.90:0/266946357 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ed591f1ec0 msgr2=0x55ed59202260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:44.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.862+0000 7ff80b851740 1 --2- 172.21.15.90:0/266946357 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ed591f1ec0 0x55ed59202260 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7ff7f404e7e0 tx=0x7ff7f40045b0 comp rx=0 tx=0).stop 2024-04-02T14:36:44.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.862+0000 7ff80b851740 1 -- 172.21.15.90:0/266946357 shutdown_connections 2024-04-02T14:36:44.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.862+0000 7ff80b851740 1 --2- 172.21.15.90:0/266946357 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x55ed59283a50 0x55ed59285e70 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:44.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.862+0000 7ff80b851740 1 --2- 172.21.15.90:0/266946357 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ed591fb230 0x55ed59202ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:44.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.862+0000 7ff80b851740 1 --2- 172.21.15.90:0/266946357 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ed591f2ba0 0x55ed592027a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:44.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.862+0000 7ff80b851740 1 --2- 172.21.15.90:0/266946357 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ed591f1ec0 0x55ed59202260 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:44.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.862+0000 7ff80b851740 1 -- 172.21.15.90:0/266946357 >> 172.21.15.90:0/266946357 conn(0x55ed591b8a50 msgr2=0x55ed591f8a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:36:44.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.862+0000 7ff7e3400640 0 client.25900 ms_handle_reset on v2:172.21.15.177:6838/664895663 2024-04-02T14:36:44.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.863+0000 7ff80b851740 1 -- 172.21.15.90:0/266946357 shutdown_connections 2024-04-02T14:36:44.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.863+0000 7ff80b851740 1 -- 172.21.15.90:0/266946357 wait complete. 2024-04-02T14:36:44.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.863+0000 7ff80b851740 20 client.25900 trim_cache size 0 max 16384 2024-04-02T14:36:44.867 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-02T14:36:44.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.866+0000 7ff803200640 1 -- 172.21.15.90:0/157314655 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7ff7e80a3e20 msgr2=0x7ff7e80a62e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:44.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.866+0000 7ff803200640 1 --2- 172.21.15.90:0/157314655 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7ff7e80a3e20 0x7ff7e80a62e0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7ff7f4006010 tx=0x7ff7f4005b70 comp rx=0 tx=0).stop 2024-04-02T14:36:44.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.866+0000 7ff803200640 1 -- 172.21.15.90:0/157314655 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff7fc0a2a00 msgr2=0x7ff7fc09f960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:44.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.866+0000 7ff803200640 1 --2- 172.21.15.90:0/157314655 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff7fc0a2a00 0x7ff7fc09f960 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7ff804050610 tx=0x7ff804069cf0 comp rx=0 tx=0).stop 2024-04-02T14:36:44.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.867+0000 7ff803200640 1 -- 172.21.15.90:0/157314655 shutdown_connections 2024-04-02T14:36:44.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.867+0000 7ff803200640 1 --2- 172.21.15.90:0/157314655 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7ff7e80a3e20 0x7ff7e80a62e0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:44.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.867+0000 7ff803200640 1 --2- 172.21.15.90:0/157314655 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff7fc0a2a00 0x7ff7fc09f960 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:44.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.867+0000 7ff803200640 1 --2- 172.21.15.90:0/157314655 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff7fc0055d0 0x7ff7fc09f420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:44.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.867+0000 7ff803200640 1 --2- 172.21.15.90:0/157314655 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff7fc004cb0 0x7ff7fc0a2330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:44.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.867+0000 7ff803200640 1 -- 172.21.15.90:0/157314655 >> 172.21.15.90:0/157314655 conn(0x7ff7fc00a740 msgr2=0x7ff7fc0a3e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:36:44.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.867+0000 7ff803200640 1 -- 172.21.15.90:0/157314655 shutdown_connections 2024-04-02T14:36:44.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:44.868+0000 7ff803200640 1 -- 172.21.15.90:0/157314655 wait complete. 2024-04-02T14:36:44.906 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-02T14:36:45.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.232+0000 7fd2acc00640 1 -- 172.21.15.90:0/1489225978 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd2a8076030 msgr2=0x7fd2a8076410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:45.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.232+0000 7fd2acc00640 1 --2- 172.21.15.90:0/1489225978 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd2a8076030 0x7fd2a8076410 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fd290009930 tx=0x7fd29002bc20 comp rx=0 tx=0).stop 2024-04-02T14:36:45.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.232+0000 7fd2acc00640 1 -- 172.21.15.90:0/1489225978 shutdown_connections 2024-04-02T14:36:45.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.232+0000 7fd2acc00640 1 --2- 172.21.15.90:0/1489225978 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd2a8112940 0x7fd2a8114d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:45.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.232+0000 7fd2acc00640 1 --2- 172.21.15.90:0/1489225978 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd2a8076950 0x7fd2a8112400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:45.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.232+0000 7fd2acc00640 1 --2- 172.21.15.90:0/1489225978 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd2a8076030 0x7fd2a8076410 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:45.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.232+0000 7fd2acc00640 1 -- 172.21.15.90:0/1489225978 >> 172.21.15.90:0/1489225978 conn(0x7fd2a80ff770 msgr2=0x7fd2a8101b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:36:45.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.232+0000 7fd2acc00640 1 -- 172.21.15.90:0/1489225978 shutdown_connections 2024-04-02T14:36:45.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.232+0000 7fd2acc00640 1 -- 172.21.15.90:0/1489225978 wait complete. 2024-04-02T14:36:45.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.234+0000 7fd2acc00640 1 Processor -- start 2024-04-02T14:36:45.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.234+0000 7fd2acc00640 1 -- start start 2024-04-02T14:36:45.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.234+0000 7fd2acc00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd2a8076030 0x7fd2a81a46b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:45.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.234+0000 7fd2acc00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd2a8076950 0x7fd2a81a4bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:45.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.235+0000 7fd2acc00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd2a8112940 0x7fd2a819e820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:45.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.235+0000 7fd2a6800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd2a8112940 0x7fd2a819e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:45.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.235+0000 7fd2a5e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd2a8076030 0x7fd2a81a46b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:45.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.235+0000 7fd2a5400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd2a8076950 0x7fd2a81a4bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:45.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.235+0000 7fd2a5400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd2a8076950 0x7fd2a81a4bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:57460/0 (socket says 172.21.15.90:57460) 2024-04-02T14:36:45.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.235+0000 7fd2a5400640 1 -- 172.21.15.90:0/894841018 learned_addr learned my addr 172.21.15.90:0/894841018 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:36:45.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.235+0000 7fd2a6800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd2a8112940 0x7fd2a819e820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:50062/0 (socket says 172.21.15.90:50062) 2024-04-02T14:36:45.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.235+0000 7fd2acc00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fd2a81184e0 con 0x7fd2a8112940 2024-04-02T14:36:45.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.235+0000 7fd2acc00640 1 -- 172.21.15.90:0/894841018 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fd2a8118360 con 0x7fd2a8076950 2024-04-02T14:36:45.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.235+0000 7fd2acc00640 1 -- 172.21.15.90:0/894841018 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fd2a8118660 con 0x7fd2a8076030 2024-04-02T14:36:45.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.236+0000 7fd2a6800640 1 -- 172.21.15.90:0/894841018 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd2a8076030 msgr2=0x7fd2a81a46b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:45.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.236+0000 7fd2a6800640 1 --2- 172.21.15.90:0/894841018 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd2a8076030 0x7fd2a81a46b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:45.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.236+0000 7fd2a6800640 1 -- 172.21.15.90:0/894841018 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd2a8076950 msgr2=0x7fd2a81a4bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:45.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.236+0000 7fd2a6800640 1 --2- 172.21.15.90:0/894841018 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd2a8076950 0x7fd2a81a4bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:45.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.236+0000 7fd2a6800640 1 -- 172.21.15.90:0/894841018 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2a819eff0 con 0x7fd2a8112940 2024-04-02T14:36:45.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.237+0000 7fd2a6800640 1 --2- 172.21.15.90:0/894841018 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd2a8112940 0x7fd2a819e820 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fd2a000e940 tx=0x7fd2a000ee00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:36:45.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.237+0000 7fd29ea00640 1 -- 172.21.15.90:0/894841018 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2a000ccc0 con 0x7fd2a8112940 2024-04-02T14:36:45.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.238+0000 7fd2acc00640 1 -- 172.21.15.90:0/894841018 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2a819f2e0 con 0x7fd2a8112940 2024-04-02T14:36:45.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.238+0000 7fd2acc00640 1 -- 172.21.15.90:0/894841018 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2a81ab500 con 0x7fd2a8112940 2024-04-02T14:36:45.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.238+0000 7fd29ea00640 1 -- 172.21.15.90:0/894841018 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd2a0005020 con 0x7fd2a8112940 2024-04-02T14:36:45.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.238+0000 7fd29ea00640 1 -- 172.21.15.90:0/894841018 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2a0004a00 con 0x7fd2a8112940 2024-04-02T14:36:45.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.244+0000 7fd29ea00640 1 -- 172.21.15.90:0/894841018 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fd2a0004ba0 con 0x7fd2a8112940 2024-04-02T14:36:45.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.250+0000 7fd2af995740 1 Processor -- start 2024-04-02T14:36:45.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.251+0000 7fd29ea00640 1 --2- 172.21.15.90:0/894841018 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fd28c0a3e20 0x7fd28c0a62e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:45.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.252+0000 7fd2af995740 1 -- start start 2024-04-02T14:36:45.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.253+0000 7fd2a5400640 1 --2- 172.21.15.90:0/894841018 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fd28c0a3e20 0x7fd28c0a62e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:45.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.253+0000 7fd29ea00640 1 -- 172.21.15.90:0/894841018 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7fd2a00d06e0 con 0x7fd2a8112940 2024-04-02T14:36:45.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.254+0000 7fd2af995740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555d4d1ba9f0 0x555d4d1b9740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:45.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.254+0000 7fd2af995740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555d4d1b9d10 0x555d4d1ba190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:45.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.254+0000 7fd2af995740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555d4d1c3080 0x555d4d1c5470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:45.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.254+0000 7fd2af995740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x555d4d0fa400 con 0x555d4d1ba9f0 2024-04-02T14:36:45.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.254+0000 7fd2af995740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x555d4d0def10 con 0x555d4d1c3080 2024-04-02T14:36:45.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.254+0000 7fd2af995740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x555d4d11e6f0 con 0x555d4d1b9d10 2024-04-02T14:36:45.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.254+0000 7fd2a5e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555d4d1ba9f0 0x555d4d1b9740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:45.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.255+0000 7fd2a6800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555d4d1c3080 0x555d4d1c5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:45.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.255+0000 7fd2a6800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555d4d1c3080 0x555d4d1c5470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:57474/0 (socket says 172.21.15.90:57474) 2024-04-02T14:36:45.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.255+0000 7fd2a5e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555d4d1ba9f0 0x555d4d1b9740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:50064/0 (socket says 172.21.15.90:50064) 2024-04-02T14:36:45.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.255+0000 7fd2a5400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555d4d1b9d10 0x555d4d1ba190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:45.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.255+0000 7fd2a6800640 1 -- 172.21.15.90:0/983114645 learned_addr learned my addr 172.21.15.90:0/983114645 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:36:45.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.256+0000 7fd2a5400640 1 -- 172.21.15.90:0/983114645 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555d4d1c3080 msgr2=0x555d4d1c5470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:36:45.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.256+0000 7fd2a5400640 1 --2- 172.21.15.90:0/983114645 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555d4d1c3080 0x555d4d1c5470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:45.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.256+0000 7fd2a5400640 1 -- 172.21.15.90:0/983114645 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555d4d1ba9f0 msgr2=0x555d4d1b9740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:36:45.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.257+0000 7fd2a5400640 1 --2- 172.21.15.90:0/983114645 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555d4d1ba9f0 0x555d4d1b9740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:45.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.257+0000 7fd2a5400640 1 -- 172.21.15.90:0/983114645 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555d4ce323f0 con 0x555d4d1b9d10 2024-04-02T14:36:45.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.257+0000 7fd2a6800640 1 --2- 172.21.15.90:0/983114645 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555d4d1c3080 0x555d4d1c5470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:36:45.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.257+0000 7fd2a5e00640 1 --2- 172.21.15.90:0/983114645 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555d4d1ba9f0 0x555d4d1b9740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:36:45.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.258+0000 7fd2a5400640 1 --2- 172.21.15.90:0/894841018 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fd28c0a3e20 0x7fd28c0a62e0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fd2a8078bc0 tx=0x7fd298011ce0 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:36:45.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.258+0000 7fd2a5400640 1 --2- 172.21.15.90:0/983114645 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555d4d1b9d10 0x555d4d1ba190 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fd2a819fb70 tx=0x7fd29801b000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1347b274c30bca40 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:36:45.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.258+0000 7fd297000640 1 -- 172.21.15.90:0/983114645 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd29801a040 con 0x555d4d1b9d10 2024-04-02T14:36:45.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.258+0000 7fd297000640 1 -- 172.21.15.90:0/983114645 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd298018070 con 0x555d4d1b9d10 2024-04-02T14:36:45.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.258+0000 7fd297000640 1 -- 172.21.15.90:0/983114645 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2980044f0 con 0x555d4d1b9d10 2024-04-02T14:36:45.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.260+0000 7fd2af995740 1 -- 172.21.15.90:0/983114645 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555d4d1b9d10 msgr2=0x555d4d1ba190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:45.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.260+0000 7fd2af995740 1 --2- 172.21.15.90:0/983114645 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555d4d1b9d10 0x555d4d1ba190 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fd2a819fb70 tx=0x7fd29801b000 comp rx=0 tx=0).stop 2024-04-02T14:36:45.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.260+0000 7fd2af995740 1 -- 172.21.15.90:0/983114645 shutdown_connections 2024-04-02T14:36:45.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.260+0000 7fd2af995740 1 --2- 172.21.15.90:0/983114645 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555d4d1c3080 0x555d4d1c5470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:45.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.260+0000 7fd2af995740 1 --2- 172.21.15.90:0/983114645 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555d4d1b9d10 0x555d4d1ba190 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:45.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.260+0000 7fd2af995740 1 --2- 172.21.15.90:0/983114645 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555d4d1ba9f0 0x555d4d1b9740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:45.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.260+0000 7fd2af995740 1 -- 172.21.15.90:0/983114645 >> 172.21.15.90:0/983114645 conn(0x555d4d180a50 msgr2=0x555d4d1b4760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:36:45.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.260+0000 7fd2af995740 1 -- 172.21.15.90:0/983114645 shutdown_connections 2024-04-02T14:36:45.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.261+0000 7fd2af995740 1 -- 172.21.15.90:0/983114645 wait complete. 2024-04-02T14:36:45.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.261+0000 7fd2af995740 1 -- start start 2024-04-02T14:36:45.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.262+0000 7fd2af995740 1 Processor -- start 2024-04-02T14:36:45.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.264+0000 7fd2af995740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555d4d1b9d10 0x555d4d1bffa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:45.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.265+0000 7fd2af995740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555d4d1ba9f0 0x555d4d1c04e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:45.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.265+0000 7fd2af995740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555d4d1c3080 0x555d4d1c0a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:45.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.265+0000 7fd2af995740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x555d4d11a6a0 con 0x555d4d1b9d10 2024-04-02T14:36:45.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.265+0000 7fd2a5e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555d4d1b9d10 0x555d4d1bffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:45.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.265+0000 7fd2a6800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555d4d1c3080 0x555d4d1c0a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:45.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.265+0000 7fd2a6800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555d4d1c3080 0x555d4d1c0a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:41346/0 (socket says 172.21.15.90:41346) 2024-04-02T14:36:45.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.265+0000 7fd2a6800640 1 -- 172.21.15.90:0/573402882 learned_addr learned my addr 172.21.15.90:0/573402882 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:36:45.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.265+0000 7fd2a5400640 1 --2- 172.21.15.90:0/573402882 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555d4d1ba9f0 0x555d4d1c04e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:45.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.266+0000 7fd2af995740 1 -- 172.21.15.90:0/573402882 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x555d4d0c6980 con 0x555d4d1ba9f0 2024-04-02T14:36:45.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.266+0000 7fd2af995740 1 -- 172.21.15.90:0/573402882 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x555d4d0d9cb0 con 0x555d4d1c3080 2024-04-02T14:36:45.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.266+0000 7fd2af995740 10 client.0 _ll_register_callbacks cb 0x555d4d0d96a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:36:45.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.266+0000 7fd2a6800640 1 -- 172.21.15.90:0/573402882 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555d4d1ba9f0 msgr2=0x555d4d1c04e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:36:45.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.266+0000 7fd2a6800640 1 --2- 172.21.15.90:0/573402882 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555d4d1ba9f0 0x555d4d1c04e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:45.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.266+0000 7fd2a6800640 1 -- 172.21.15.90:0/573402882 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555d4d1b9d10 msgr2=0x555d4d1bffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:36:45.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.266+0000 7fd2a6800640 1 --2- 172.21.15.90:0/573402882 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555d4d1b9d10 0x555d4d1bffa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:45.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.266+0000 7fd2a6800640 1 -- 172.21.15.90:0/573402882 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555d4cefc7d0 con 0x555d4d1c3080 2024-04-02T14:36:45.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.266+0000 7fd2a5e00640 1 --2- 172.21.15.90:0/573402882 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555d4d1b9d10 0x555d4d1bffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:36:45.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.267+0000 7fd28b400640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:36:45.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.267+0000 7fd2a6800640 1 --2- 172.21.15.90:0/573402882 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555d4d1c3080 0x555d4d1c0a20 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fd2a00cdd50 tx=0x7fd2a009c0c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=45e070838c751ec9 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:36:45.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.267+0000 7fd28b400640 1 -- 172.21.15.90:0/573402882 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2a00b3030 con 0x555d4d1c3080 2024-04-02T14:36:45.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.267+0000 7fd28b400640 1 -- 172.21.15.90:0/573402882 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7fd2a00a4e10 con 0x555d4d1c3080 2024-04-02T14:36:45.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.268+0000 7fd2af995740 1 -- 172.21.15.90:0/573402882 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd298018070 con 0x555d4d1c3080 2024-04-02T14:36:45.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.268+0000 7fd28b400640 1 -- 172.21.15.90:0/573402882 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fd2a001c040 con 0x555d4d1c3080 2024-04-02T14:36:45.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.268+0000 7fd2af995740 10 client.25912 fetch_fsmap learned FSMap version 27 2024-04-02T14:36:45.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.268+0000 7fd2af995740 1 -- 172.21.15.90:0/573402882 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x555d4d1d1e40 con 0x555d4d1c3080 2024-04-02T14:36:45.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.269+0000 7fd28b400640 1 -- 172.21.15.90:0/573402882 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd298018070 con 0x555d4d1c3080 2024-04-02T14:36:45.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.269+0000 7fd28b400640 1 -- 172.21.15.90:0/573402882 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fd2a00bb2b0 con 0x555d4d1c3080 2024-04-02T14:36:45.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.269+0000 7fd2af995740 10 client.25912 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:36:45.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.269+0000 7fd2af995740 10 client.25912 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2024-04-02T14:36:45.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.269+0000 7fd2af995740 20 client.25912 populate_metadata read hostname 'smithi090' 2024-04-02T14:36:45.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.270+0000 7fd2af995740 1 --2- 172.21.15.90:0/573402882 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x555d4d1d33f0 0x555d4d1d5880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:45.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.270+0000 7fd2af995740 4 client.25912 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:36:45.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.270+0000 7fd2af995740 1 -- 172.21.15.90:0/573402882 --> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555d4d0de040 con 0x555d4d1d33f0 2024-04-02T14:36:45.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.270+0000 7fd2a5e00640 1 --2- 172.21.15.90:0/573402882 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x555d4d1d33f0 0x555d4d1d5880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:45.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.272+0000 7fd2a5e00640 1 --2- 172.21.15.90:0/573402882 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x555d4d1d33f0 0x555d4d1d5880 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=9485afb8d792da5e server_cookie=c8a37895a5f8ba02 in_seq=0 out_seq=0 2024-04-02T14:36:45.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.272+0000 7fd28b400640 10 client.25912 ms_handle_connect on v2:172.21.15.90:6838/1498915870 2024-04-02T14:36:45.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.273+0000 7fd28b400640 1 -- 172.21.15.90:0/573402882 <== mds.1 v2:172.21.15.90:6838/1498915870 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x555d4d0de040 con 0x555d4d1d33f0 2024-04-02T14:36:45.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.273+0000 7fd28b400640 10 client.25912 handle_command_reply: tid=0 2024-04-02T14:36:45.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.274+0000 7fd2af995740 1 client.25912 shutdown 2024-04-02T14:36:45.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.274+0000 7fd2af995740 1 -- 172.21.15.90:0/573402882 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555d4d1c3080 msgr2=0x555d4d1c0a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:45.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.274+0000 7fd2af995740 1 --2- 172.21.15.90:0/573402882 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555d4d1c3080 0x555d4d1c0a20 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fd2a00cdd50 tx=0x7fd2a009c0c0 comp rx=0 tx=0).stop 2024-04-02T14:36:45.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.274+0000 7fd2af995740 1 -- 172.21.15.90:0/573402882 shutdown_connections 2024-04-02T14:36:45.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.274+0000 7fd2af995740 1 --2- 172.21.15.90:0/573402882 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x555d4d1d33f0 0x555d4d1d5880 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:45.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.274+0000 7fd2af995740 1 --2- 172.21.15.90:0/573402882 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555d4d1c3080 0x555d4d1c0a20 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:45.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.274+0000 7fd2af995740 1 --2- 172.21.15.90:0/573402882 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555d4d1ba9f0 0x555d4d1c04e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:45.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.274+0000 7fd2af995740 1 --2- 172.21.15.90:0/573402882 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555d4d1b9d10 0x555d4d1bffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:45.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.274+0000 7fd2af995740 1 -- 172.21.15.90:0/573402882 >> 172.21.15.90:0/573402882 conn(0x555d4d180a50 msgr2=0x555d4d1b64b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:36:45.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.275+0000 7fd28b400640 0 client.25912 ms_handle_reset on v2:172.21.15.90:6838/1498915870 2024-04-02T14:36:45.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.275+0000 7fd2af995740 1 -- 172.21.15.90:0/573402882 shutdown_connections 2024-04-02T14:36:45.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.275+0000 7fd2af995740 1 -- 172.21.15.90:0/573402882 wait complete. 2024-04-02T14:36:45.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.275+0000 7fd2af995740 20 client.25912 trim_cache size 0 max 16384 2024-04-02T14:36:45.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.319+0000 7fd2af995740 1 Processor -- start 2024-04-02T14:36:45.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.320+0000 7fd2af995740 1 -- start start 2024-04-02T14:36:45.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.321+0000 7fd2af995740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555d4d1b9d10 0x555d4d1c26d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:45.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.321+0000 7fd2af995740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555d4d1ba9f0 0x555d4d1ceb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:45.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.321+0000 7fd2af995740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555d4d1c3080 0x555d4d1b4290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:45.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.321+0000 7fd2af995740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fd28400e750 con 0x555d4d1ba9f0 2024-04-02T14:36:45.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.321+0000 7fd2af995740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fd284010290 con 0x555d4d1b9d10 2024-04-02T14:36:45.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.321+0000 7fd2af995740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fd28400ee20 con 0x555d4d1c3080 2024-04-02T14:36:45.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.321+0000 7fd2a5e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555d4d1b9d10 0x555d4d1c26d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:45.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.322+0000 7fd2a5e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555d4d1b9d10 0x555d4d1c26d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:57492/0 (socket says 172.21.15.90:57492) 2024-04-02T14:36:45.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.322+0000 7fd2a5e00640 1 -- 172.21.15.90:0/4058658142 learned_addr learned my addr 172.21.15.90:0/4058658142 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:36:45.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.322+0000 7fd2a6800640 1 --2- 172.21.15.90:0/4058658142 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555d4d1c3080 0x555d4d1b4290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:45.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.322+0000 7fd2a5e00640 1 -- 172.21.15.90:0/4058658142 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555d4d1c3080 msgr2=0x555d4d1b4290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:36:45.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.322+0000 7fd2a5e00640 1 --2- 172.21.15.90:0/4058658142 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555d4d1c3080 0x555d4d1b4290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:45.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.322+0000 7fd2a5e00640 1 -- 172.21.15.90:0/4058658142 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555d4d1ba9f0 msgr2=0x555d4d1ceb20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-02T14:36:45.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.322+0000 7fd2a5400640 1 --2- 172.21.15.90:0/4058658142 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555d4d1ba9f0 0x555d4d1ceb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:45.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.322+0000 7fd2a5e00640 1 --2- 172.21.15.90:0/4058658142 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555d4d1ba9f0 0x555d4d1ceb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:45.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.322+0000 7fd2a5e00640 1 -- 172.21.15.90:0/4058658142 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555d4d1b4970 con 0x555d4d1b9d10 2024-04-02T14:36:45.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.323+0000 7fd2a5e00640 1 --2- 172.21.15.90:0/4058658142 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555d4d1b9d10 0x555d4d1c26d0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fd290009900 tx=0x7fd290005fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3d0802d3bdc40f2f server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:36:45.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.323+0000 7fd296600640 1 -- 172.21.15.90:0/4058658142 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd29004e070 con 0x555d4d1b9d10 2024-04-02T14:36:45.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.324+0000 7fd296600640 1 -- 172.21.15.90:0/4058658142 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x555d4d0de040 con 0x555d4d1b9d10 2024-04-02T14:36:45.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.324+0000 7fd296600640 1 -- 172.21.15.90:0/4058658142 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2900046c0 con 0x555d4d1b9d10 2024-04-02T14:36:45.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.325+0000 7fd2af995740 1 -- 172.21.15.90:0/4058658142 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555d4d1b9d10 msgr2=0x555d4d1c26d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:45.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.325+0000 7fd2af995740 1 --2- 172.21.15.90:0/4058658142 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555d4d1b9d10 0x555d4d1c26d0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fd290009900 tx=0x7fd290005fd0 comp rx=0 tx=0).stop 2024-04-02T14:36:45.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.325+0000 7fd2af995740 1 -- 172.21.15.90:0/4058658142 shutdown_connections 2024-04-02T14:36:45.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.325+0000 7fd2af995740 1 --2- 172.21.15.90:0/4058658142 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555d4d1c3080 0x555d4d1b4290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:45.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.325+0000 7fd2af995740 1 --2- 172.21.15.90:0/4058658142 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555d4d1ba9f0 0x555d4d1ceb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:45.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.325+0000 7fd2af995740 1 --2- 172.21.15.90:0/4058658142 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555d4d1b9d10 0x555d4d1c26d0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:45.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.325+0000 7fd2af995740 1 -- 172.21.15.90:0/4058658142 >> 172.21.15.90:0/4058658142 conn(0x555d4d180a50 msgr2=0x555d4d1b5bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:36:45.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.326+0000 7fd2af995740 1 -- 172.21.15.90:0/4058658142 shutdown_connections 2024-04-02T14:36:45.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.326+0000 7fd2af995740 1 -- 172.21.15.90:0/4058658142 wait complete. 2024-04-02T14:36:45.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.327+0000 7fd2af995740 1 -- start start 2024-04-02T14:36:45.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.327+0000 7fd2af995740 1 Processor -- start 2024-04-02T14:36:45.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.328+0000 7fd2af995740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555d4d1b9d10 0x555d4d1cb290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:45.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.328+0000 7fd2af995740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555d4d1ba9f0 0x555d4d1bc810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:45.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.328+0000 7fd2af995740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555d4d1c3080 0x555d4d1bcd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:45.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.328+0000 7fd2af995740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fd28400fbc0 con 0x555d4d1c3080 2024-04-02T14:36:45.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.328+0000 7fd2af995740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fd28400f4f0 con 0x555d4d1b9d10 2024-04-02T14:36:45.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.328+0000 7fd2af995740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fd28400c730 con 0x555d4d1ba9f0 2024-04-02T14:36:45.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.328+0000 7fd2af995740 10 client.0 _ll_register_callbacks cb 0x7fd2a00989a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:36:45.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.328+0000 7fd2a5400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555d4d1ba9f0 0x555d4d1bc810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:45.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.328+0000 7fd2a5400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555d4d1ba9f0 0x555d4d1bc810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:41368/0 (socket says 172.21.15.90:41368) 2024-04-02T14:36:45.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.328+0000 7fd2a5400640 1 -- 172.21.15.90:0/1680789644 learned_addr learned my addr 172.21.15.90:0/1680789644 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:36:45.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.328+0000 7fd2a5e00640 1 --2- 172.21.15.90:0/1680789644 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555d4d1b9d10 0x555d4d1cb290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:45.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.328+0000 7fd2a5400640 1 -- 172.21.15.90:0/1680789644 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555d4d1b9d10 msgr2=0x555d4d1cb290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:36:45.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.328+0000 7fd2a5400640 1 --2- 172.21.15.90:0/1680789644 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555d4d1b9d10 0x555d4d1cb290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:45.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.328+0000 7fd2a5400640 1 -- 172.21.15.90:0/1680789644 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555d4d1c3080 msgr2=0x555d4d1bcd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:36:45.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.328+0000 7fd2a5400640 1 --2- 172.21.15.90:0/1680789644 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555d4d1c3080 0x555d4d1bcd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:45.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.328+0000 7fd2a5400640 1 -- 172.21.15.90:0/1680789644 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555d4d1bd3f0 con 0x555d4d1ba9f0 2024-04-02T14:36:45.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.329+0000 7fd2a5400640 1 --2- 172.21.15.90:0/1680789644 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555d4d1ba9f0 0x555d4d1bc810 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x555d4d1baf40 tx=0x7fd298005e10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=df85935cf708a5a6 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:36:45.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.329+0000 7fd28b400640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:36:45.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.329+0000 7fd28b400640 1 -- 172.21.15.90:0/1680789644 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd29801a040 con 0x555d4d1ba9f0 2024-04-02T14:36:45.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.330+0000 7fd28b400640 1 -- 172.21.15.90:0/1680789644 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7fd298009050 con 0x555d4d1ba9f0 2024-04-02T14:36:45.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.330+0000 7fd2af995740 1 -- 172.21.15.90:0/1680789644 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x555d4d0de040 con 0x555d4d1ba9f0 2024-04-02T14:36:45.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.330+0000 7fd28b400640 1 -- 172.21.15.90:0/1680789644 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fd29801a040 con 0x555d4d1ba9f0 2024-04-02T14:36:45.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.330+0000 7fd2af995740 10 client.25918 fetch_fsmap learned FSMap version 27 2024-04-02T14:36:45.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.330+0000 7fd2af995740 1 -- 172.21.15.90:0/1680789644 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x555d4d239850 con 0x555d4d1ba9f0 2024-04-02T14:36:45.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.331+0000 7fd28b400640 1 -- 172.21.15.90:0/1680789644 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x555d4d0de040 con 0x555d4d1ba9f0 2024-04-02T14:36:45.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.331+0000 7fd28b400640 1 -- 172.21.15.90:0/1680789644 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fd29801d040 con 0x555d4d1ba9f0 2024-04-02T14:36:45.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.331+0000 7fd2af995740 10 client.25918 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:36:45.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.331+0000 7fd2af995740 10 client.25918 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2024-04-02T14:36:45.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.331+0000 7fd2af995740 20 client.25918 populate_metadata read hostname 'smithi090' 2024-04-02T14:36:45.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.331+0000 7fd2af995740 1 --2- 172.21.15.90:0/1680789644 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x555d4d239db0 0x555d4d23c1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:45.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.331+0000 7fd2af995740 4 client.25918 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-02T14:36:45.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.331+0000 7fd2af995740 1 -- 172.21.15.90:0/1680789644 --> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] -- command(tid 0: {"prefix": "damage ls"}) -- 0x555d4d12bc30 con 0x555d4d239db0 2024-04-02T14:36:45.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.332+0000 7fd2a5e00640 1 --2- 172.21.15.90:0/1680789644 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x555d4d239db0 0x555d4d23c1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:45.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.333+0000 7fd28b400640 10 client.25918 ms_handle_connect on v2:172.21.15.90:6838/1498915870 2024-04-02T14:36:45.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.333+0000 7fd2a5e00640 1 --2- 172.21.15.90:0/1680789644 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x555d4d239db0 0x555d4d23c1d0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=47f3d501927ca756 server_cookie=ba4855485c8f8cba in_seq=0 out_seq=0 2024-04-02T14:36:45.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.334+0000 7fd28b400640 1 -- 172.21.15.90:0/1680789644 <== mds.1 v2:172.21.15.90:6838/1498915870 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x555d4d12bc30 con 0x555d4d239db0 2024-04-02T14:36:45.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.334+0000 7fd28b400640 10 client.25918 handle_command_reply: tid=0 2024-04-02T14:36:45.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.334+0000 7fd2af995740 1 client.25918 shutdown 2024-04-02T14:36:45.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.335+0000 7fd2af995740 1 -- 172.21.15.90:0/1680789644 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555d4d1ba9f0 msgr2=0x555d4d1bc810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:45.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.335+0000 7fd2af995740 1 --2- 172.21.15.90:0/1680789644 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555d4d1ba9f0 0x555d4d1bc810 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x555d4d1baf40 tx=0x7fd298005e10 comp rx=0 tx=0).stop 2024-04-02T14:36:45.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.335+0000 7fd2af995740 1 -- 172.21.15.90:0/1680789644 shutdown_connections 2024-04-02T14:36:45.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.335+0000 7fd2af995740 1 --2- 172.21.15.90:0/1680789644 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x555d4d239db0 0x555d4d23c1d0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:45.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.335+0000 7fd2af995740 1 --2- 172.21.15.90:0/1680789644 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555d4d1c3080 0x555d4d1bcd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:45.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.335+0000 7fd2af995740 1 --2- 172.21.15.90:0/1680789644 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555d4d1ba9f0 0x555d4d1bc810 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:45.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.335+0000 7fd2af995740 1 --2- 172.21.15.90:0/1680789644 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555d4d1b9d10 0x555d4d1cb290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:45.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.335+0000 7fd28b400640 0 client.25918 ms_handle_reset on v2:172.21.15.90:6838/1498915870 2024-04-02T14:36:45.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.335+0000 7fd2af995740 1 -- 172.21.15.90:0/1680789644 >> 172.21.15.90:0/1680789644 conn(0x555d4d180a50 msgr2=0x555d4d1bff60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:36:45.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.336+0000 7fd2af995740 1 -- 172.21.15.90:0/1680789644 shutdown_connections 2024-04-02T14:36:45.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.336+0000 7fd2af995740 1 -- 172.21.15.90:0/1680789644 wait complete. 2024-04-02T14:36:45.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.337+0000 7fd2af995740 20 client.25918 trim_cache size 0 max 16384 2024-04-02T14:36:45.340 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-02T14:36:45.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.338+0000 7fd2acc00640 1 -- 172.21.15.90:0/894841018 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fd28c0a3e20 msgr2=0x7fd28c0a62e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:45.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.338+0000 7fd2acc00640 1 --2- 172.21.15.90:0/894841018 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fd28c0a3e20 0x7fd28c0a62e0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fd2a8078bc0 tx=0x7fd298011ce0 comp rx=0 tx=0).stop 2024-04-02T14:36:45.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.338+0000 7fd2acc00640 1 -- 172.21.15.90:0/894841018 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd2a8112940 msgr2=0x7fd2a819e820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:45.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.338+0000 7fd2acc00640 1 --2- 172.21.15.90:0/894841018 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd2a8112940 0x7fd2a819e820 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fd2a000e940 tx=0x7fd2a000ee00 comp rx=0 tx=0).stop 2024-04-02T14:36:45.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.339+0000 7fd2acc00640 1 -- 172.21.15.90:0/894841018 shutdown_connections 2024-04-02T14:36:45.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.339+0000 7fd2acc00640 1 --2- 172.21.15.90:0/894841018 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fd28c0a3e20 0x7fd28c0a62e0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:45.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.339+0000 7fd2acc00640 1 --2- 172.21.15.90:0/894841018 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd2a8112940 0x7fd2a819e820 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:45.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.339+0000 7fd2acc00640 1 --2- 172.21.15.90:0/894841018 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd2a8076950 0x7fd2a81a4bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:45.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.339+0000 7fd2acc00640 1 --2- 172.21.15.90:0/894841018 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd2a8076030 0x7fd2a81a46b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:45.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.339+0000 7fd2acc00640 1 -- 172.21.15.90:0/894841018 >> 172.21.15.90:0/894841018 conn(0x7fd2a80ff770 msgr2=0x7fd2a8101b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:36:45.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.339+0000 7fd2acc00640 1 -- 172.21.15.90:0/894841018 shutdown_connections 2024-04-02T14:36:45.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.340+0000 7fd2acc00640 1 -- 172.21.15.90:0/894841018 wait complete. 2024-04-02T14:36:45.372 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-02T14:36:45.686 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:36:45 smithi177 ceph-mon[36431]: pgmap v59: 97 pgs: 97 active+clean; 1.2 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 14 MiB/s wr, 1.05k op/s 2024-04-02T14:36:45.688 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:36:45 smithi090 ceph-mon[31674]: pgmap v59: 97 pgs: 97 active+clean; 1.2 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 14 MiB/s wr, 1.05k op/s 2024-04-02T14:36:45.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.705+0000 7f3568600640 1 -- 172.21.15.90:0/1005080532 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f35601048c0 msgr2=0x7f3560108900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:45.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.705+0000 7f3568600640 1 --2- 172.21.15.90:0/1005080532 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f35601048c0 0x7f3560108900 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f354c009900 tx=0x7f354c02bc30 comp rx=0 tx=0).stop 2024-04-02T14:36:45.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.706+0000 7f3568600640 1 -- 172.21.15.90:0/1005080532 shutdown_connections 2024-04-02T14:36:45.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.706+0000 7f3568600640 1 --2- 172.21.15.90:0/1005080532 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3560108fe0 0x7f3560110bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:45.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.706+0000 7f3568600640 1 --2- 172.21.15.90:0/1005080532 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f35601048c0 0x7f3560108900 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:45.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.706+0000 7f3568600640 1 --2- 172.21.15.90:0/1005080532 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3560103f10 0x7f35601042f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:45.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.706+0000 7f3568600640 1 -- 172.21.15.90:0/1005080532 >> 172.21.15.90:0/1005080532 conn(0x7f35600ff7b0 msgr2=0x7f3560101bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:36:45.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.706+0000 7f3568600640 1 -- 172.21.15.90:0/1005080532 shutdown_connections 2024-04-02T14:36:45.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.706+0000 7f3568600640 1 -- 172.21.15.90:0/1005080532 wait complete. 2024-04-02T14:36:45.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.707+0000 7f3568600640 1 Processor -- start 2024-04-02T14:36:45.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.708+0000 7f3568600640 1 -- start start 2024-04-02T14:36:45.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.709+0000 7f3568600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3560103f10 0x7f356019a640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:45.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.709+0000 7f3568600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3560108fe0 0x7f356019ab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:45.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.709+0000 7f3568600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f356019ef00 0x7f356019f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:45.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.709+0000 7f3568600640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f35601140c0 con 0x7f3560103f10 2024-04-02T14:36:45.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.709+0000 7f3568600640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f3560113f40 con 0x7f3560108fe0 2024-04-02T14:36:45.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.709+0000 7f3568600640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f3560114240 con 0x7f356019ef00 2024-04-02T14:36:45.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.709+0000 7f3566600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f356019ef00 0x7f356019f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:45.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.709+0000 7f3566600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f356019ef00 0x7f356019f3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:41388/0 (socket says 172.21.15.90:41388) 2024-04-02T14:36:45.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.709+0000 7f3566600640 1 -- 172.21.15.90:0/899548230 learned_addr learned my addr 172.21.15.90:0/899548230 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:36:45.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.709+0000 7f3565200640 1 --2- 172.21.15.90:0/899548230 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3560108fe0 0x7f356019ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:45.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.709+0000 7f3566600640 1 -- 172.21.15.90:0/899548230 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3560108fe0 msgr2=0x7f356019ab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:45.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.709+0000 7f3566600640 1 --2- 172.21.15.90:0/899548230 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3560108fe0 0x7f356019ab80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:45.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.709+0000 7f3566600640 1 -- 172.21.15.90:0/899548230 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3560103f10 msgr2=0x7f356019a640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:45.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.709+0000 7f3566600640 1 --2- 172.21.15.90:0/899548230 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3560103f10 0x7f356019a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:45.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.709+0000 7f3566600640 1 -- 172.21.15.90:0/899548230 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f356019f9f0 con 0x7f356019ef00 2024-04-02T14:36:45.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.710+0000 7f3566600640 1 --2- 172.21.15.90:0/899548230 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f356019ef00 0x7f356019f3b0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f355800efd0 tx=0x7f355800c6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:36:45.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.710+0000 7f355e000640 1 -- 172.21.15.90:0/899548230 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3558003b70 con 0x7f356019ef00 2024-04-02T14:36:45.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.710+0000 7f355e000640 1 -- 172.21.15.90:0/899548230 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3558003d10 con 0x7f356019ef00 2024-04-02T14:36:45.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.710+0000 7f355e000640 1 -- 172.21.15.90:0/899548230 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3558010960 con 0x7f356019ef00 2024-04-02T14:36:45.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.711+0000 7f3568600640 1 -- 172.21.15.90:0/899548230 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f356019fce0 con 0x7f356019ef00 2024-04-02T14:36:45.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.711+0000 7f3568600640 1 -- 172.21.15.90:0/899548230 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35600761e0 con 0x7f356019ef00 2024-04-02T14:36:45.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.718+0000 7f355e000640 1 -- 172.21.15.90:0/899548230 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f3558004020 con 0x7f356019ef00 2024-04-02T14:36:45.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.720+0000 7f356b3a0740 1 Processor -- start 2024-04-02T14:36:45.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.720+0000 7f356b3a0740 1 -- start start 2024-04-02T14:36:45.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.720+0000 7f356b3a0740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e52ad219f0 0x55e52ad20740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:45.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.720+0000 7f356b3a0740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e52ad20d10 0x55e52ad21190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:45.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.720+0000 7f356b3a0740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e52ad2a080 0x55e52ad2c470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:45.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.720+0000 7f356b3a0740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55e52ac61400 con 0x55e52ad20d10 2024-04-02T14:36:45.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.720+0000 7f356b3a0740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55e52ac45f10 con 0x55e52ad219f0 2024-04-02T14:36:45.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.720+0000 7f356b3a0740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55e52ac856f0 con 0x55e52ad2a080 2024-04-02T14:36:45.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.721+0000 7f3565200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e52ad219f0 0x55e52ad20740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:45.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.721+0000 7f3565200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e52ad219f0 0x55e52ad20740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:57532/0 (socket says 172.21.15.90:57532) 2024-04-02T14:36:45.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.721+0000 7f3565200640 1 -- 172.21.15.90:0/2541155226 learned_addr learned my addr 172.21.15.90:0/2541155226 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:36:45.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.721+0000 7f3565c00640 1 --2- 172.21.15.90:0/2541155226 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e52ad20d10 0x55e52ad21190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:45.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.721+0000 7f3565200640 1 --2- 172.21.15.90:0/2541155226 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e52ad2a080 0x55e52ad2c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:45.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.721+0000 7f3565200640 1 -- 172.21.15.90:0/2541155226 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e52ad2a080 msgr2=0x55e52ad2c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:36:45.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.721+0000 7f3565200640 1 --2- 172.21.15.90:0/2541155226 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e52ad2a080 0x55e52ad2c470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:45.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.721+0000 7f3565200640 1 -- 172.21.15.90:0/2541155226 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e52ad20d10 msgr2=0x55e52ad21190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:36:45.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.721+0000 7f3565200640 1 --2- 172.21.15.90:0/2541155226 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e52ad20d10 0x55e52ad21190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:45.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.721+0000 7f3565200640 1 -- 172.21.15.90:0/2541155226 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e52a9993f0 con 0x55e52ad219f0 2024-04-02T14:36:45.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.722+0000 7f3565200640 1 --2- 172.21.15.90:0/2541155226 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e52ad219f0 0x55e52ad20740 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f354c000c00 tx=0x7f354c005fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8b51da44285c3e1b server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:36:45.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.723+0000 7f3552a00640 1 -- 172.21.15.90:0/2541155226 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f354c02c650 con 0x55e52ad219f0 2024-04-02T14:36:45.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.723+0000 7f3552a00640 1 -- 172.21.15.90:0/2541155226 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f354c004390 con 0x55e52ad219f0 2024-04-02T14:36:45.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.723+0000 7f355e000640 1 --2- 172.21.15.90:0/899548230 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f35480a3e70 0x7f35480a6330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:45.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.724+0000 7f355e000640 1 -- 172.21.15.90:0/899548230 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f35580d0db0 con 0x7f356019ef00 2024-04-02T14:36:45.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.724+0000 7f3565c00640 1 --2- 172.21.15.90:0/899548230 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f35480a3e70 0x7f35480a6330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:45.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.724+0000 7f356b3a0740 1 -- 172.21.15.90:0/2541155226 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e52ad219f0 msgr2=0x55e52ad20740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:45.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.724+0000 7f356b3a0740 1 --2- 172.21.15.90:0/2541155226 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e52ad219f0 0x55e52ad20740 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f354c000c00 tx=0x7f354c005fd0 comp rx=0 tx=0).stop 2024-04-02T14:36:45.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.725+0000 7f356b3a0740 1 -- 172.21.15.90:0/2541155226 shutdown_connections 2024-04-02T14:36:45.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.725+0000 7f356b3a0740 1 --2- 172.21.15.90:0/2541155226 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e52ad2a080 0x55e52ad2c470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:45.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.725+0000 7f356b3a0740 1 --2- 172.21.15.90:0/2541155226 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e52ad20d10 0x55e52ad21190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:45.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.725+0000 7f356b3a0740 1 --2- 172.21.15.90:0/2541155226 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e52ad219f0 0x55e52ad20740 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:45.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.725+0000 7f356b3a0740 1 -- 172.21.15.90:0/2541155226 >> 172.21.15.90:0/2541155226 conn(0x55e52ace7a50 msgr2=0x55e52ad1b760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:36:45.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.726+0000 7f356b3a0740 1 -- 172.21.15.90:0/2541155226 shutdown_connections 2024-04-02T14:36:45.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.726+0000 7f3565c00640 1 --2- 172.21.15.90:0/899548230 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f35480a3e70 0x7f35480a6330 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f3554009460 tx=0x7f35540093b0 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:36:45.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.726+0000 7f356b3a0740 1 -- 172.21.15.90:0/2541155226 wait complete. 2024-04-02T14:36:45.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.727+0000 7f356b3a0740 1 -- start start 2024-04-02T14:36:45.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.727+0000 7f356b3a0740 1 Processor -- start 2024-04-02T14:36:45.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.728+0000 7f356b3a0740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e52ad20d10 0x55e52ad26f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:45.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.728+0000 7f356b3a0740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e52ad219f0 0x55e52ad27490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:45.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.728+0000 7f356b3a0740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e52ad2a080 0x55e52ad279d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:45.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.728+0000 7f3565200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e52ad219f0 0x55e52ad27490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:45.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.728+0000 7f3565200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e52ad219f0 0x55e52ad27490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:41410/0 (socket says 172.21.15.90:41410) 2024-04-02T14:36:45.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.728+0000 7f356b3a0740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55e52ac816a0 con 0x55e52ad2a080 2024-04-02T14:36:45.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.728+0000 7f3565200640 1 -- 172.21.15.90:0/2720649586 learned_addr learned my addr 172.21.15.90:0/2720649586 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:36:45.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.728+0000 7f356b3a0740 1 -- 172.21.15.90:0/2720649586 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55e52ac2d980 con 0x55e52ad20d10 2024-04-02T14:36:45.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.728+0000 7f356b3a0740 1 -- 172.21.15.90:0/2720649586 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55e52ac40cb0 con 0x55e52ad219f0 2024-04-02T14:36:45.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.728+0000 7f356b3a0740 10 client.0 _ll_register_callbacks cb 0x55e52ac406a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:36:45.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.729+0000 7f3565c00640 1 --2- 172.21.15.90:0/2720649586 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e52ad20d10 0x55e52ad26f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:45.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.729+0000 7f3566600640 1 --2- 172.21.15.90:0/2720649586 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e52ad2a080 0x55e52ad279d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:45.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.729+0000 7f3565c00640 1 -- 172.21.15.90:0/2720649586 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e52ad219f0 msgr2=0x55e52ad27490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:36:45.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.729+0000 7f3565c00640 1 --2- 172.21.15.90:0/2720649586 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e52ad219f0 0x55e52ad27490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:45.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.729+0000 7f3565c00640 1 -- 172.21.15.90:0/2720649586 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e52ad2a080 msgr2=0x55e52ad279d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:36:45.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.729+0000 7f3565c00640 1 --2- 172.21.15.90:0/2720649586 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e52ad2a080 0x55e52ad279d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:45.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.729+0000 7f3565c00640 1 -- 172.21.15.90:0/2720649586 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e52aa637d0 con 0x55e52ad20d10 2024-04-02T14:36:45.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.729+0000 7f3566600640 1 --2- 172.21.15.90:0/2720649586 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e52ad2a080 0x55e52ad279d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:36:45.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.730+0000 7f3565c00640 1 --2- 172.21.15.90:0/2720649586 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e52ad20d10 0x55e52ad26f50 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f3554010e20 tx=0x7f3554005f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6ae5cf2f7f74e820 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:36:45.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.730+0000 7f3547400640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:36:45.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.731+0000 7f3547400640 1 -- 172.21.15.90:0/2720649586 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3554023070 con 0x55e52ad20d10 2024-04-02T14:36:45.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.731+0000 7f3547400640 1 -- 172.21.15.90:0/2720649586 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f355401f860 con 0x55e52ad20d10 2024-04-02T14:36:45.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.731+0000 7f356b3a0740 1 -- 172.21.15.90:0/2720649586 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f354c004390 con 0x55e52ad20d10 2024-04-02T14:36:45.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.732+0000 7f3547400640 1 -- 172.21.15.90:0/2720649586 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f355402f050 con 0x55e52ad20d10 2024-04-02T14:36:45.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.732+0000 7f356b3a0740 10 client.26015 fetch_fsmap learned FSMap version 27 2024-04-02T14:36:45.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.732+0000 7f356b3a0740 1 -- 172.21.15.90:0/2720649586 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55e52ad38f70 con 0x55e52ad20d10 2024-04-02T14:36:45.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.733+0000 7f3547400640 1 -- 172.21.15.90:0/2720649586 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f354c004390 con 0x55e52ad20d10 2024-04-02T14:36:45.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.733+0000 7f3547400640 1 -- 172.21.15.90:0/2720649586 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f355400f040 con 0x55e52ad20d10 2024-04-02T14:36:45.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.733+0000 7f356b3a0740 10 client.26015 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:36:45.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.733+0000 7f356b3a0740 10 client.26015 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:36:45.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.733+0000 7f356b3a0740 20 client.26015 populate_metadata read hostname 'smithi090' 2024-04-02T14:36:45.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.733+0000 7f356b3a0740 1 --2- 172.21.15.90:0/2720649586 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55e52ad3a840 0x55e52ad3ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:45.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.733+0000 7f356b3a0740 4 client.26015 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:36:45.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.733+0000 7f356b3a0740 1 -- 172.21.15.90:0/2720649586 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e52ac45040 con 0x55e52ad3a840 2024-04-02T14:36:45.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.734+0000 7f3565200640 1 --2- 172.21.15.90:0/2720649586 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55e52ad3a840 0x55e52ad3ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:45.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.735+0000 7f3565200640 1 --2- 172.21.15.90:0/2720649586 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55e52ad3a840 0x55e52ad3ccc0 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=438e63fce4546861 server_cookie=b600ebda14f782e5 in_seq=0 out_seq=0 2024-04-02T14:36:45.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.735+0000 7f3547400640 10 client.26015 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:36:45.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.735+0000 7f3547400640 1 -- 172.21.15.90:0/2720649586 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e52ac45040 con 0x55e52ad3a840 2024-04-02T14:36:45.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.735+0000 7f3547400640 10 client.26015 handle_command_reply: tid=0 2024-04-02T14:36:45.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.736+0000 7f356b3a0740 1 client.26015 shutdown 2024-04-02T14:36:45.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.736+0000 7f356b3a0740 1 -- 172.21.15.90:0/2720649586 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e52ad20d10 msgr2=0x55e52ad26f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:45.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.736+0000 7f356b3a0740 1 --2- 172.21.15.90:0/2720649586 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e52ad20d10 0x55e52ad26f50 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f3554010e20 tx=0x7f3554005f90 comp rx=0 tx=0).stop 2024-04-02T14:36:45.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.736+0000 7f356b3a0740 1 -- 172.21.15.90:0/2720649586 shutdown_connections 2024-04-02T14:36:45.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.736+0000 7f356b3a0740 1 --2- 172.21.15.90:0/2720649586 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55e52ad3a840 0x55e52ad3ccc0 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:45.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.736+0000 7f356b3a0740 1 --2- 172.21.15.90:0/2720649586 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e52ad2a080 0x55e52ad279d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:45.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.736+0000 7f356b3a0740 1 --2- 172.21.15.90:0/2720649586 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e52ad219f0 0x55e52ad27490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:45.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.736+0000 7f356b3a0740 1 --2- 172.21.15.90:0/2720649586 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e52ad20d10 0x55e52ad26f50 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:45.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.736+0000 7f356b3a0740 1 -- 172.21.15.90:0/2720649586 >> 172.21.15.90:0/2720649586 conn(0x55e52ace7a50 msgr2=0x55e52ad1d4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:36:45.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.736+0000 7f3547400640 0 client.26015 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:36:45.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.737+0000 7f356b3a0740 1 -- 172.21.15.90:0/2720649586 shutdown_connections 2024-04-02T14:36:45.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.737+0000 7f356b3a0740 1 -- 172.21.15.90:0/2720649586 wait complete. 2024-04-02T14:36:45.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.737+0000 7f356b3a0740 20 client.26015 trim_cache size 0 max 16384 2024-04-02T14:36:45.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.770+0000 7f356b3a0740 1 Processor -- start 2024-04-02T14:36:45.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.770+0000 7f356b3a0740 1 -- start start 2024-04-02T14:36:45.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.770+0000 7f356b3a0740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e52ad20d10 0x55e52ad29680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:45.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.770+0000 7f356b3a0740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e52ad219f0 0x55e52ad35bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:45.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.771+0000 7f356b3a0740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e52ad2a080 0x55e52ad1d250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:45.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.771+0000 7f356b3a0740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f35601b3ee0 con 0x55e52ad219f0 2024-04-02T14:36:45.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.771+0000 7f356b3a0740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f35601b5a20 con 0x55e52ad20d10 2024-04-02T14:36:45.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.771+0000 7f356b3a0740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f35601b45b0 con 0x55e52ad2a080 2024-04-02T14:36:45.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.771+0000 7f3565c00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e52ad20d10 0x55e52ad29680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:45.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.771+0000 7f3565c00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e52ad20d10 0x55e52ad29680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:57544/0 (socket says 172.21.15.90:57544) 2024-04-02T14:36:45.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.771+0000 7f3565c00640 1 -- 172.21.15.90:0/1594501874 learned_addr learned my addr 172.21.15.90:0/1594501874 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:36:45.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.772+0000 7f3565200640 1 --2- 172.21.15.90:0/1594501874 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e52ad219f0 0x55e52ad35bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:45.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.771+0000 7f3566600640 1 --2- 172.21.15.90:0/1594501874 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e52ad2a080 0x55e52ad1d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:45.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.772+0000 7f3565200640 1 -- 172.21.15.90:0/1594501874 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e52ad2a080 msgr2=0x55e52ad1d250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:36:45.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.772+0000 7f3565200640 1 --2- 172.21.15.90:0/1594501874 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e52ad2a080 0x55e52ad1d250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:45.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.772+0000 7f3565200640 1 -- 172.21.15.90:0/1594501874 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e52ad20d10 msgr2=0x55e52ad29680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:36:45.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.772+0000 7f3565200640 1 --2- 172.21.15.90:0/1594501874 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e52ad20d10 0x55e52ad29680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:45.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.772+0000 7f3565200640 1 -- 172.21.15.90:0/1594501874 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e52ad1fd40 con 0x55e52ad219f0 2024-04-02T14:36:45.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.773+0000 7f3565200640 1 --2- 172.21.15.90:0/1594501874 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e52ad219f0 0x55e52ad35bb0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x55e52ad28520 tx=0x7f354c02bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e850363032edfa4e server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:36:45.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.774+0000 7f3552000640 1 -- 172.21.15.90:0/1594501874 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f354c0047c0 con 0x55e52ad219f0 2024-04-02T14:36:45.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.774+0000 7f3552000640 1 -- 172.21.15.90:0/1594501874 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f354c004960 con 0x55e52ad219f0 2024-04-02T14:36:45.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.774+0000 7f3552000640 1 -- 172.21.15.90:0/1594501874 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f354c003850 con 0x55e52ad219f0 2024-04-02T14:36:45.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.775+0000 7f356b3a0740 1 -- 172.21.15.90:0/1594501874 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e52ad219f0 msgr2=0x55e52ad35bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:45.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.775+0000 7f356b3a0740 1 --2- 172.21.15.90:0/1594501874 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e52ad219f0 0x55e52ad35bb0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x55e52ad28520 tx=0x7f354c02bc30 comp rx=0 tx=0).stop 2024-04-02T14:36:45.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.775+0000 7f356b3a0740 1 -- 172.21.15.90:0/1594501874 shutdown_connections 2024-04-02T14:36:45.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.775+0000 7f356b3a0740 1 --2- 172.21.15.90:0/1594501874 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e52ad2a080 0x55e52ad1d250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:45.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.775+0000 7f356b3a0740 1 --2- 172.21.15.90:0/1594501874 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e52ad219f0 0x55e52ad35bb0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:45.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.775+0000 7f356b3a0740 1 --2- 172.21.15.90:0/1594501874 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e52ad20d10 0x55e52ad29680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:45.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.776+0000 7f356b3a0740 1 -- 172.21.15.90:0/1594501874 >> 172.21.15.90:0/1594501874 conn(0x55e52ace7a50 msgr2=0x55e52ad1cb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:36:45.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.776+0000 7f356b3a0740 1 -- 172.21.15.90:0/1594501874 shutdown_connections 2024-04-02T14:36:45.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.776+0000 7f356b3a0740 1 -- 172.21.15.90:0/1594501874 wait complete. 2024-04-02T14:36:45.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.777+0000 7f356b3a0740 1 -- start start 2024-04-02T14:36:45.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.777+0000 7f356b3a0740 1 Processor -- start 2024-04-02T14:36:45.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.778+0000 7f356b3a0740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e52ad20d10 0x55e52ad322b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:45.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.778+0000 7f356b3a0740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e52ad219f0 0x55e52ad237e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:45.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.779+0000 7f356b3a0740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e52ad2a080 0x55e52ad23d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:45.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.779+0000 7f3566600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e52ad2a080 0x55e52ad23d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:45.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.779+0000 7f3566600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e52ad2a080 0x55e52ad23d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:41422/0 (socket says 172.21.15.90:41422) 2024-04-02T14:36:45.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.779+0000 7f3566600640 1 -- 172.21.15.90:0/4246731185 learned_addr learned my addr 172.21.15.90:0/4246731185 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:36:45.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.779+0000 7f356b3a0740 1 -- 172.21.15.90:0/4246731185 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f35601b5350 con 0x55e52ad219f0 2024-04-02T14:36:45.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.779+0000 7f356b3a0740 1 -- 172.21.15.90:0/4246731185 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f35601b4c80 con 0x55e52ad20d10 2024-04-02T14:36:45.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.779+0000 7f356b3a0740 1 -- 172.21.15.90:0/4246731185 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f35601b1ec0 con 0x55e52ad2a080 2024-04-02T14:36:45.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.780+0000 7f356b3a0740 10 client.0 _ll_register_callbacks cb 0x7f3554010ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:36:45.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.780+0000 7f3565200640 1 --2- 172.21.15.90:0/4246731185 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e52ad219f0 0x55e52ad237e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:45.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.780+0000 7f3566600640 1 -- 172.21.15.90:0/4246731185 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e52ad20d10 msgr2=0x55e52ad322b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:36:45.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.779+0000 7f3565c00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e52ad20d10 0x55e52ad322b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:45.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.780+0000 7f3566600640 1 --2- 172.21.15.90:0/4246731185 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e52ad20d10 0x55e52ad322b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:45.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.781+0000 7f3566600640 1 -- 172.21.15.90:0/4246731185 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e52ad219f0 msgr2=0x55e52ad237e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:36:45.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.781+0000 7f3566600640 1 --2- 172.21.15.90:0/4246731185 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e52ad219f0 0x55e52ad237e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:45.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.781+0000 7f3566600640 1 -- 172.21.15.90:0/4246731185 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e52ad24260 con 0x55e52ad2a080 2024-04-02T14:36:45.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.781+0000 7f3565200640 1 --2- 172.21.15.90:0/4246731185 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e52ad219f0 0x55e52ad237e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:36:45.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.781+0000 7f3565c00640 1 --2- 172.21.15.90:0/4246731185 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e52ad20d10 0x55e52ad322b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:36:45.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.782+0000 7f3566600640 1 --2- 172.21.15.90:0/4246731185 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e52ad2a080 0x55e52ad23d20 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f35580996d0 tx=0x7f35580098e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f65b037b491b0309 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:36:45.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.782+0000 7f3547400640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:36:45.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.783+0000 7f3547400640 1 -- 172.21.15.90:0/4246731185 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35580bb060 con 0x55e52ad2a080 2024-04-02T14:36:45.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.783+0000 7f3547400640 1 -- 172.21.15.90:0/4246731185 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f3558099370 con 0x55e52ad2a080 2024-04-02T14:36:45.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.783+0000 7f356b3a0740 1 -- 172.21.15.90:0/4246731185 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f354c004960 con 0x55e52ad2a080 2024-04-02T14:36:45.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.783+0000 7f3547400640 1 -- 172.21.15.90:0/4246731185 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f35580aa050 con 0x55e52ad2a080 2024-04-02T14:36:45.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.783+0000 7f356b3a0740 10 client.25936 fetch_fsmap learned FSMap version 27 2024-04-02T14:36:45.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.783+0000 7f356b3a0740 1 -- 172.21.15.90:0/4246731185 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55e52ada0ac0 con 0x55e52ad2a080 2024-04-02T14:36:45.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.783+0000 7f3547400640 1 -- 172.21.15.90:0/4246731185 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f354c004960 con 0x55e52ad2a080 2024-04-02T14:36:45.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.784+0000 7f3547400640 1 -- 172.21.15.90:0/4246731185 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f35580a5ce0 con 0x55e52ad2a080 2024-04-02T14:36:45.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.784+0000 7f356b3a0740 10 client.25936 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:36:45.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.785+0000 7f356b3a0740 10 client.25936 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:36:45.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.785+0000 7f356b3a0740 20 client.25936 populate_metadata read hostname 'smithi090' 2024-04-02T14:36:45.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.785+0000 7f356b3a0740 1 --2- 172.21.15.90:0/4246731185 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55e52ada1210 0x55e52ada3630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:45.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.785+0000 7f356b3a0740 4 client.25936 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-02T14:36:45.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.785+0000 7f356b3a0740 1 -- 172.21.15.90:0/4246731185 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55e52ac92c30 con 0x55e52ada1210 2024-04-02T14:36:45.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.785+0000 7f3565c00640 1 --2- 172.21.15.90:0/4246731185 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55e52ada1210 0x55e52ada3630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:45.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.786+0000 7f3565c00640 1 --2- 172.21.15.90:0/4246731185 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55e52ada1210 0x55e52ada3630 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7185f9acdf7f6fb3 server_cookie=ca78f325eb661b29 in_seq=0 out_seq=0 2024-04-02T14:36:45.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.786+0000 7f3547400640 10 client.25936 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:36:45.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.786+0000 7f3547400640 1 -- 172.21.15.90:0/4246731185 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55e52ac92c30 con 0x55e52ada1210 2024-04-02T14:36:45.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.786+0000 7f3547400640 10 client.25936 handle_command_reply: tid=0 2024-04-02T14:36:45.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.786+0000 7f356b3a0740 1 client.25936 shutdown 2024-04-02T14:36:45.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.787+0000 7f356b3a0740 1 -- 172.21.15.90:0/4246731185 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e52ad2a080 msgr2=0x55e52ad23d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:45.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.787+0000 7f356b3a0740 1 --2- 172.21.15.90:0/4246731185 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e52ad2a080 0x55e52ad23d20 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f35580996d0 tx=0x7f35580098e0 comp rx=0 tx=0).stop 2024-04-02T14:36:45.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.787+0000 7f356b3a0740 1 -- 172.21.15.90:0/4246731185 shutdown_connections 2024-04-02T14:36:45.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.787+0000 7f356b3a0740 1 --2- 172.21.15.90:0/4246731185 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55e52ada1210 0x55e52ada3630 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:45.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.787+0000 7f356b3a0740 1 --2- 172.21.15.90:0/4246731185 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e52ad2a080 0x55e52ad23d20 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:45.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.787+0000 7f356b3a0740 1 --2- 172.21.15.90:0/4246731185 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e52ad219f0 0x55e52ad237e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:45.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.787+0000 7f356b3a0740 1 --2- 172.21.15.90:0/4246731185 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e52ad20d10 0x55e52ad322b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:45.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.788+0000 7f356b3a0740 1 -- 172.21.15.90:0/4246731185 >> 172.21.15.90:0/4246731185 conn(0x55e52ace7a50 msgr2=0x55e52ad1a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:36:45.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.788+0000 7f3547400640 0 client.25936 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:36:45.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.788+0000 7f356b3a0740 1 -- 172.21.15.90:0/4246731185 shutdown_connections 2024-04-02T14:36:45.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.788+0000 7f356b3a0740 1 -- 172.21.15.90:0/4246731185 wait complete. 2024-04-02T14:36:45.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.788+0000 7f356b3a0740 20 client.25936 trim_cache size 0 max 16384 2024-04-02T14:36:45.791 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-02T14:36:45.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.790+0000 7f3568600640 1 -- 172.21.15.90:0/899548230 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f35480a3e70 msgr2=0x7f35480a6330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:45.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.790+0000 7f3568600640 1 --2- 172.21.15.90:0/899548230 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f35480a3e70 0x7f35480a6330 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f3554009460 tx=0x7f35540093b0 comp rx=0 tx=0).stop 2024-04-02T14:36:45.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.790+0000 7f3568600640 1 -- 172.21.15.90:0/899548230 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f356019ef00 msgr2=0x7f356019f3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:45.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.790+0000 7f3568600640 1 --2- 172.21.15.90:0/899548230 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f356019ef00 0x7f356019f3b0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f355800efd0 tx=0x7f355800c6a0 comp rx=0 tx=0).stop 2024-04-02T14:36:45.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.790+0000 7f3568600640 1 -- 172.21.15.90:0/899548230 shutdown_connections 2024-04-02T14:36:45.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.791+0000 7f3568600640 1 --2- 172.21.15.90:0/899548230 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f35480a3e70 0x7f35480a6330 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:45.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.791+0000 7f3568600640 1 --2- 172.21.15.90:0/899548230 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f356019ef00 0x7f356019f3b0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:45.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.791+0000 7f3568600640 1 --2- 172.21.15.90:0/899548230 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3560108fe0 0x7f356019ab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:45.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.791+0000 7f3568600640 1 --2- 172.21.15.90:0/899548230 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3560103f10 0x7f356019a640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:45.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.791+0000 7f3568600640 1 -- 172.21.15.90:0/899548230 >> 172.21.15.90:0/899548230 conn(0x7f35600ff7b0 msgr2=0x7f3560100ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:36:45.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.791+0000 7f3568600640 1 -- 172.21.15.90:0/899548230 shutdown_connections 2024-04-02T14:36:45.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:45.791+0000 7f3568600640 1 -- 172.21.15.90:0/899548230 wait complete. 2024-04-02T14:36:45.860 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:36:45 smithi092 ceph-mon[36497]: pgmap v59: 97 pgs: 97 active+clean; 1.2 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 14 MiB/s wr, 1.05k op/s 2024-04-02T14:36:46.824 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-02T14:36:46.824 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T14:36:46.824 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-02T14:36:46.832 INFO:teuthology.orchestra.run.smithi090.stderr:progress: 75.0 s, 3548.6 tps, lat 28.205 ms stddev 28.996 2024-04-02T14:36:47.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.141+0000 7f7fc0200640 1 -- 172.21.15.90:0/3528797786 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7fb8110320 msgr2=0x7f7fb810de10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:47.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.141+0000 7f7fc0200640 1 --2- 172.21.15.90:0/3528797786 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7fb8110320 0x7f7fb810de10 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f7fb0009900 tx=0x7f7fb002bc40 comp rx=0 tx=0).stop 2024-04-02T14:36:47.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.142+0000 7f7fc0200640 1 -- 172.21.15.90:0/3528797786 shutdown_connections 2024-04-02T14:36:47.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.142+0000 7f7fc0200640 1 --2- 172.21.15.90:0/3528797786 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7fb811b2e0 0x7f7fb811d6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:47.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.142+0000 7f7fc0200640 1 --2- 172.21.15.90:0/3528797786 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7fb810e350 0x7f7fb810e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:47.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.142+0000 7f7fc0200640 1 --2- 172.21.15.90:0/3528797786 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7fb8110320 0x7f7fb810de10 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:47.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.142+0000 7f7fc0200640 1 -- 172.21.15.90:0/3528797786 >> 172.21.15.90:0/3528797786 conn(0x7f7fb806cbe0 msgr2=0x7f7fb806cff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:36:47.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.142+0000 7f7fc0200640 1 -- 172.21.15.90:0/3528797786 shutdown_connections 2024-04-02T14:36:47.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.143+0000 7f7fc0200640 1 -- 172.21.15.90:0/3528797786 wait complete. 2024-04-02T14:36:47.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.144+0000 7f7fc0200640 1 Processor -- start 2024-04-02T14:36:47.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.144+0000 7f7fc0200640 1 -- start start 2024-04-02T14:36:47.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.145+0000 7f7fc0200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7fb810e350 0x7f7fb81a7110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:47.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.145+0000 7f7fc0200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7fb8110320 0x7f7fb81a7650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:47.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.145+0000 7f7fc0200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7fb811b2e0 0x7f7fb81ab980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:47.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.145+0000 7f7fc0200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f7fb8120c90 con 0x7f7fb8110320 2024-04-02T14:36:47.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.145+0000 7f7fc0200640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f7fb8120b10 con 0x7f7fb810e350 2024-04-02T14:36:47.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.145+0000 7f7fc0200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f7fb8120e10 con 0x7f7fb811b2e0 2024-04-02T14:36:47.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.145+0000 7f7fbd800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7fb810e350 0x7f7fb81a7110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:47.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.145+0000 7f7fbd800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7fb810e350 0x7f7fb81a7110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:57580/0 (socket says 172.21.15.90:57580) 2024-04-02T14:36:47.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.145+0000 7f7fbd800640 1 -- 172.21.15.90:0/3832901719 learned_addr learned my addr 172.21.15.90:0/3832901719 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:36:47.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.146+0000 7f7fbce00640 1 --2- 172.21.15.90:0/3832901719 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7fb8110320 0x7f7fb81a7650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:47.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.146+0000 7f7fbe200640 1 --2- 172.21.15.90:0/3832901719 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7fb811b2e0 0x7f7fb81ab980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:47.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.146+0000 7f7fbd800640 1 -- 172.21.15.90:0/3832901719 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7fb811b2e0 msgr2=0x7f7fb81ab980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:47.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.146+0000 7f7fbd800640 1 --2- 172.21.15.90:0/3832901719 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7fb811b2e0 0x7f7fb81ab980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:47.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.146+0000 7f7fbd800640 1 -- 172.21.15.90:0/3832901719 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7fb8110320 msgr2=0x7f7fb81a7650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:47.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.146+0000 7f7fbd800640 1 --2- 172.21.15.90:0/3832901719 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7fb8110320 0x7f7fb81a7650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:47.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.146+0000 7f7fbd800640 1 -- 172.21.15.90:0/3832901719 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7fb81ac0b0 con 0x7f7fb810e350 2024-04-02T14:36:47.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.146+0000 7f7fbce00640 1 --2- 172.21.15.90:0/3832901719 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7fb8110320 0x7f7fb81a7650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:36:47.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.146+0000 7f7fbd800640 1 --2- 172.21.15.90:0/3832901719 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7fb810e350 0x7f7fb81a7110 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f7fb0000c00 tx=0x7f7fb00045e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:36:47.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.147+0000 7f7fb6000640 1 -- 172.21.15.90:0/3832901719 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fb0041070 con 0x7f7fb810e350 2024-04-02T14:36:47.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.147+0000 7f7fb6000640 1 -- 172.21.15.90:0/3832901719 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7fb002cd20 con 0x7f7fb810e350 2024-04-02T14:36:47.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.147+0000 7f7fb6000640 1 -- 172.21.15.90:0/3832901719 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fb0034440 con 0x7f7fb810e350 2024-04-02T14:36:47.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.147+0000 7f7fc0200640 1 -- 172.21.15.90:0/3832901719 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7fb81ac340 con 0x7f7fb810e350 2024-04-02T14:36:47.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.147+0000 7f7fc0200640 1 -- 172.21.15.90:0/3832901719 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7fb81b3c30 con 0x7f7fb810e350 2024-04-02T14:36:47.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.152+0000 7f7fc2e29740 1 Processor -- start 2024-04-02T14:36:47.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.152+0000 7f7fc2e29740 1 -- start start 2024-04-02T14:36:47.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.153+0000 7f7fc2e29740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563c58657030 0x563c58655d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:47.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.153+0000 7f7fc2e29740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563c58656350 0x563c586567d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:47.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.153+0000 7f7fc2e29740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563c5865f6c0 0x563c58661ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:47.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.153+0000 7f7fc2e29740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x563c585ba310 con 0x563c5865f6c0 2024-04-02T14:36:47.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.153+0000 7f7fc2e29740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x563c58562690 con 0x563c58656350 2024-04-02T14:36:47.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.153+0000 7f7fc2e29740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x563c585b62c0 con 0x563c58657030 2024-04-02T14:36:47.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.155+0000 7f7fbe200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563c58657030 0x563c58655d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:47.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.155+0000 7f7fbe200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563c58657030 0x563c58655d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:41448/0 (socket says 172.21.15.90:41448) 2024-04-02T14:36:47.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.155+0000 7f7fbe200640 1 -- 172.21.15.90:0/738490372 learned_addr learned my addr 172.21.15.90:0/738490372 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:36:47.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.155+0000 7f7fb6000640 1 -- 172.21.15.90:0/3832901719 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f7fb0044070 con 0x7f7fb810e350 2024-04-02T14:36:47.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.156+0000 7f7fbe200640 1 -- 172.21.15.90:0/738490372 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563c58656350 msgr2=0x563c586567d0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-02T14:36:47.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.156+0000 7f7fbe200640 1 --2- 172.21.15.90:0/738490372 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563c58656350 0x563c586567d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:47.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.156+0000 7f7fbe200640 1 -- 172.21.15.90:0/738490372 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563c5865f6c0 msgr2=0x563c58661ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:36:47.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.156+0000 7f7fbe200640 1 --2- 172.21.15.90:0/738490372 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563c5865f6c0 0x563c58661ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:47.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.156+0000 7f7fbe200640 1 -- 172.21.15.90:0/738490372 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563c583967e0 con 0x563c58657030 2024-04-02T14:36:47.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.156+0000 7f7fbe200640 1 --2- 172.21.15.90:0/738490372 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563c58657030 0x563c58655d80 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f7fa8007dd0 tx=0x7f7fa800e720 comp rx=0 tx=0).ready entity=mon.1 client_cookie=51019b0c3ccf90ce server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:36:47.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.156+0000 7f7faea00640 1 -- 172.21.15.90:0/738490372 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fa8014070 con 0x563c58657030 2024-04-02T14:36:47.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.157+0000 7f7faea00640 1 -- 172.21.15.90:0/738490372 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7fa8004d20 con 0x563c58657030 2024-04-02T14:36:47.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.157+0000 7f7faea00640 1 -- 172.21.15.90:0/738490372 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fa8010910 con 0x563c58657030 2024-04-02T14:36:47.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.158+0000 7f7fc2e29740 1 -- 172.21.15.90:0/738490372 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563c58657030 msgr2=0x563c58655d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:47.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.158+0000 7f7fc2e29740 1 --2- 172.21.15.90:0/738490372 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563c58657030 0x563c58655d80 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f7fa8007dd0 tx=0x7f7fa800e720 comp rx=0 tx=0).stop 2024-04-02T14:36:47.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.158+0000 7f7fb6000640 1 --2- 172.21.15.90:0/3832901719 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f7f9c0a3de0 0x7f7f9c0a62a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:47.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.159+0000 7f7fb6000640 1 -- 172.21.15.90:0/3832901719 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f7fb00f2470 con 0x7f7fb810e350 2024-04-02T14:36:47.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.160+0000 7f7fc2e29740 1 -- 172.21.15.90:0/738490372 shutdown_connections 2024-04-02T14:36:47.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.160+0000 7f7fc2e29740 1 --2- 172.21.15.90:0/738490372 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563c5865f6c0 0x563c58661ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:47.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.160+0000 7f7fc2e29740 1 --2- 172.21.15.90:0/738490372 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563c58656350 0x563c586567d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:47.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.160+0000 7f7fc2e29740 1 --2- 172.21.15.90:0/738490372 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563c58657030 0x563c58655d80 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:47.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.160+0000 7f7fc2e29740 1 -- 172.21.15.90:0/738490372 >> 172.21.15.90:0/738490372 conn(0x563c585ea270 msgr2=0x563c58650da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:36:47.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.160+0000 7f7fbe200640 1 --2- 172.21.15.90:0/3832901719 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f7f9c0a3de0 0x7f7f9c0a62a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:47.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.160+0000 7f7fc2e29740 1 -- 172.21.15.90:0/738490372 shutdown_connections 2024-04-02T14:36:47.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.160+0000 7f7fc2e29740 1 -- 172.21.15.90:0/738490372 wait complete. 2024-04-02T14:36:47.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.161+0000 7f7fc2e29740 1 -- start start 2024-04-02T14:36:47.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.161+0000 7f7fc2e29740 1 Processor -- start 2024-04-02T14:36:47.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.162+0000 7f7fbe200640 1 --2- 172.21.15.90:0/3832901719 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f7f9c0a3de0 0x7f7f9c0a62a0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f7fa8010540 tx=0x7f7fa8000f40 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:36:47.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.162+0000 7f7fc2e29740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563c58656350 0x563c5865c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:47.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.162+0000 7f7fc2e29740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563c58657030 0x563c5865cc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:47.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.162+0000 7f7fc2e29740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563c5865f6c0 0x563c5865d140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:47.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.162+0000 7f7fc2e29740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x563c58662530 con 0x563c58657030 2024-04-02T14:36:47.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.162+0000 7f7fc2e29740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x563c58575a60 con 0x563c58656350 2024-04-02T14:36:47.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.162+0000 7f7fc2e29740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x563c586626b0 con 0x563c5865f6c0 2024-04-02T14:36:47.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.163+0000 7f7fc2e29740 10 client.0 _ll_register_callbacks cb 0x563c5857b030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:36:47.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.163+0000 7f7fbe200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563c5865f6c0 0x563c5865d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:47.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.163+0000 7f7fbe200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563c5865f6c0 0x563c5865d140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:41456/0 (socket says 172.21.15.90:41456) 2024-04-02T14:36:47.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.163+0000 7f7fbe200640 1 -- 172.21.15.90:0/1514902721 learned_addr learned my addr 172.21.15.90:0/1514902721 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:36:47.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.163+0000 7f7fbd800640 1 --2- 172.21.15.90:0/1514902721 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563c58656350 0x563c5865c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:47.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.163+0000 7f7fbd800640 1 -- 172.21.15.90:0/1514902721 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563c5865f6c0 msgr2=0x563c5865d140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:36:47.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.163+0000 7f7fbd800640 1 --2- 172.21.15.90:0/1514902721 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563c5865f6c0 0x563c5865d140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:47.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.163+0000 7f7fbd800640 1 -- 172.21.15.90:0/1514902721 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563c58657030 msgr2=0x563c5865cc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:36:47.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.163+0000 7f7fbce00640 1 --2- 172.21.15.90:0/1514902721 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563c58657030 0x563c5865cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:47.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.163+0000 7f7fbd800640 1 --2- 172.21.15.90:0/1514902721 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563c58657030 0x563c5865cc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:47.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.163+0000 7f7fbd800640 1 -- 172.21.15.90:0/1514902721 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563c583f6f70 con 0x563c58656350 2024-04-02T14:36:47.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.164+0000 7f7fbd800640 1 --2- 172.21.15.90:0/1514902721 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563c58656350 0x563c5865c6c0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f7fb00efce0 tx=0x7f7fb00ba800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cd2112cb3830b2a server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:36:47.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.164+0000 7f7fa3400640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:36:47.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.164+0000 7f7fa3400640 1 -- 172.21.15.90:0/1514902721 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fb004b070 con 0x563c58656350 2024-04-02T14:36:47.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.164+0000 7f7fa3400640 1 -- 172.21.15.90:0/1514902721 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f7fb00035f0 con 0x563c58656350 2024-04-02T14:36:47.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.164+0000 7f7fc2e29740 1 -- 172.21.15.90:0/1514902721 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7fa8004d20 con 0x563c58656350 2024-04-02T14:36:47.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.165+0000 7f7fa3400640 1 -- 172.21.15.90:0/1514902721 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f7fb00bc670 con 0x563c58656350 2024-04-02T14:36:47.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.165+0000 7f7fc2e29740 10 client.26027 fetch_fsmap learned FSMap version 27 2024-04-02T14:36:47.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.165+0000 7f7fc2e29740 1 -- 172.21.15.90:0/1514902721 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x563c5828e170 con 0x563c58656350 2024-04-02T14:36:47.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.165+0000 7f7fa3400640 1 -- 172.21.15.90:0/1514902721 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7fa8004d20 con 0x563c58656350 2024-04-02T14:36:47.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.165+0000 7f7fa3400640 1 -- 172.21.15.90:0/1514902721 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f7fb00bd980 con 0x563c58656350 2024-04-02T14:36:47.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.165+0000 7f7fc2e29740 10 client.26027 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:36:47.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.165+0000 7f7fc2e29740 10 client.26027 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:36:47.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.165+0000 7f7fc2e29740 20 client.26027 populate_metadata read hostname 'smithi090' 2024-04-02T14:36:47.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.165+0000 7f7fc2e29740 1 --2- 172.21.15.90:0/1514902721 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x563c58680c30 0x563c586830b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:47.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.165+0000 7f7fc2e29740 4 client.26027 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:36:47.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.165+0000 7f7fc2e29740 1 -- 172.21.15.90:0/1514902721 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563c585c7850 con 0x563c58680c30 2024-04-02T14:36:47.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.166+0000 7f7fbce00640 1 --2- 172.21.15.90:0/1514902721 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x563c58680c30 0x563c586830b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:47.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.167+0000 7f7fbce00640 1 --2- 172.21.15.90:0/1514902721 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x563c58680c30 0x563c586830b0 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=63edc83890327894 server_cookie=70a14426947f2267 in_seq=0 out_seq=0 2024-04-02T14:36:47.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.167+0000 7f7fa3400640 10 client.26027 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:36:47.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.169+0000 7f7fa3400640 1 -- 172.21.15.90:0/1514902721 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x563c585c7850 con 0x563c58680c30 2024-04-02T14:36:47.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.169+0000 7f7fa3400640 10 client.26027 handle_command_reply: tid=0 2024-04-02T14:36:47.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.169+0000 7f7fc2e29740 1 client.26027 shutdown 2024-04-02T14:36:47.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.169+0000 7f7fc2e29740 1 -- 172.21.15.90:0/1514902721 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563c58656350 msgr2=0x563c5865c6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:47.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.169+0000 7f7fc2e29740 1 --2- 172.21.15.90:0/1514902721 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563c58656350 0x563c5865c6c0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f7fb00efce0 tx=0x7f7fb00ba800 comp rx=0 tx=0).stop 2024-04-02T14:36:47.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.170+0000 7f7fc2e29740 1 -- 172.21.15.90:0/1514902721 shutdown_connections 2024-04-02T14:36:47.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.170+0000 7f7fc2e29740 1 --2- 172.21.15.90:0/1514902721 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x563c58680c30 0x563c586830b0 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:47.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.170+0000 7f7fc2e29740 1 --2- 172.21.15.90:0/1514902721 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563c5865f6c0 0x563c5865d140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:47.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.170+0000 7f7fa3400640 0 client.26027 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:36:47.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.170+0000 7f7fc2e29740 1 --2- 172.21.15.90:0/1514902721 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563c58657030 0x563c5865cc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:47.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.170+0000 7f7fc2e29740 1 --2- 172.21.15.90:0/1514902721 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563c58656350 0x563c5865c6c0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:47.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.170+0000 7f7fc2e29740 1 -- 172.21.15.90:0/1514902721 >> 172.21.15.90:0/1514902721 conn(0x563c585ea270 msgr2=0x563c58652af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:36:47.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.170+0000 7f7fc2e29740 1 -- 172.21.15.90:0/1514902721 shutdown_connections 2024-04-02T14:36:47.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.170+0000 7f7fc2e29740 1 -- 172.21.15.90:0/1514902721 wait complete. 2024-04-02T14:36:47.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.171+0000 7f7fc2e29740 20 client.26027 trim_cache size 0 max 16384 2024-04-02T14:36:47.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.217+0000 7f7fc2e29740 1 Processor -- start 2024-04-02T14:36:47.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.217+0000 7f7fc2e29740 1 -- start start 2024-04-02T14:36:47.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.218+0000 7f7fc2e29740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563c58656350 0x563c5864fc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:47.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.218+0000 7f7fc2e29740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563c58657030 0x563c5866b4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:47.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.218+0000 7f7fc2e29740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563c5865f6c0 0x563c58652900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:47.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.218+0000 7f7fbce00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563c58657030 0x563c5866b4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:47.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.218+0000 7f7fbce00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563c58657030 0x563c5866b4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:57602/0 (socket says 172.21.15.90:57602) 2024-04-02T14:36:47.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.218+0000 7f7fc2e29740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f7fb81b3e50 con 0x563c58656350 2024-04-02T14:36:47.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.218+0000 7f7fc2e29740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f7fb81b5990 con 0x563c58657030 2024-04-02T14:36:47.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.218+0000 7f7fc2e29740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f7fb81b4520 con 0x563c5865f6c0 2024-04-02T14:36:47.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.219+0000 7f7fbce00640 1 -- 172.21.15.90:0/336310605 learned_addr learned my addr 172.21.15.90:0/336310605 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:36:47.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.219+0000 7f7fbe200640 1 --2- 172.21.15.90:0/336310605 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563c5865f6c0 0x563c58652900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:47.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.219+0000 7f7fbce00640 1 -- 172.21.15.90:0/336310605 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563c5865f6c0 msgr2=0x563c58652900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:36:47.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.219+0000 7f7fbce00640 1 --2- 172.21.15.90:0/336310605 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563c5865f6c0 0x563c58652900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:47.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.219+0000 7f7fbce00640 1 -- 172.21.15.90:0/336310605 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563c58656350 msgr2=0x563c5864fc00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-02T14:36:47.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.220+0000 7f7fbd800640 1 --2- 172.21.15.90:0/336310605 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563c58656350 0x563c5864fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:47.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.221+0000 7f7fbce00640 1 --2- 172.21.15.90:0/336310605 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563c58656350 0x563c5864fc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:47.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.221+0000 7f7fbce00640 1 -- 172.21.15.90:0/336310605 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563c58683730 con 0x563c58657030 2024-04-02T14:36:47.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.221+0000 7f7fbd800640 1 --2- 172.21.15.90:0/336310605 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563c58656350 0x563c5864fc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:36:47.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.221+0000 7f7fbce00640 1 --2- 172.21.15.90:0/336310605 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563c58657030 0x563c5866b4b0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x563c5865dc90 tx=0x7f7fa4005c60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c98a09d0e4f028a2 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:36:47.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.222+0000 7f7fae000640 1 -- 172.21.15.90:0/336310605 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fa401c030 con 0x563c58657030 2024-04-02T14:36:47.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.222+0000 7f7fae000640 1 -- 172.21.15.90:0/336310605 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7fa40183c0 con 0x563c58657030 2024-04-02T14:36:47.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.222+0000 7f7fae000640 1 -- 172.21.15.90:0/336310605 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fa4005610 con 0x563c58657030 2024-04-02T14:36:47.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.224+0000 7f7fc2e29740 1 -- 172.21.15.90:0/336310605 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563c58657030 msgr2=0x563c5866b4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:47.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.224+0000 7f7fc2e29740 1 --2- 172.21.15.90:0/336310605 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563c58657030 0x563c5866b4b0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x563c5865dc90 tx=0x7f7fa4005c60 comp rx=0 tx=0).stop 2024-04-02T14:36:47.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.224+0000 7f7fc2e29740 1 -- 172.21.15.90:0/336310605 shutdown_connections 2024-04-02T14:36:47.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.224+0000 7f7fc2e29740 1 --2- 172.21.15.90:0/336310605 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563c5865f6c0 0x563c58652900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:47.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.224+0000 7f7fc2e29740 1 --2- 172.21.15.90:0/336310605 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563c58657030 0x563c5866b4b0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:47.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.224+0000 7f7fc2e29740 1 --2- 172.21.15.90:0/336310605 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563c58656350 0x563c5864fc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:47.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.224+0000 7f7fc2e29740 1 -- 172.21.15.90:0/336310605 >> 172.21.15.90:0/336310605 conn(0x563c585ea270 msgr2=0x563c586521f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:36:47.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.224+0000 7f7fc2e29740 1 -- 172.21.15.90:0/336310605 shutdown_connections 2024-04-02T14:36:47.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.225+0000 7f7fc2e29740 1 -- 172.21.15.90:0/336310605 wait complete. 2024-04-02T14:36:47.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.225+0000 7f7fc2e29740 1 -- start start 2024-04-02T14:36:47.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.225+0000 7f7fc2e29740 1 Processor -- start 2024-04-02T14:36:47.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.226+0000 7f7fc2e29740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563c58656350 0x563c58662980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:47.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.226+0000 7f7fc2e29740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563c58657030 0x563c58662ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:47.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.226+0000 7f7fc2e29740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563c5865f6c0 0x563c58663440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:47.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.226+0000 7f7fc2e29740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f7fb81b52c0 con 0x563c5865f6c0 2024-04-02T14:36:47.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.226+0000 7f7fc2e29740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f7fb81b4bf0 con 0x563c58657030 2024-04-02T14:36:47.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.226+0000 7f7fc2e29740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f7fb80688f0 con 0x563c58656350 2024-04-02T14:36:47.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.226+0000 7f7fc2e29740 10 client.0 _ll_register_callbacks cb 0x7f7fb0009610 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:36:47.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.226+0000 7f7fbd800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563c58656350 0x563c58662980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:47.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.226+0000 7f7fbd800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563c58656350 0x563c58662980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:41462/0 (socket says 172.21.15.90:41462) 2024-04-02T14:36:47.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.226+0000 7f7fbd800640 1 -- 172.21.15.90:0/2728603345 learned_addr learned my addr 172.21.15.90:0/2728603345 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:36:47.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.226+0000 7f7fbce00640 1 --2- 172.21.15.90:0/2728603345 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563c58657030 0x563c58662ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:47.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.227+0000 7f7fbd800640 1 -- 172.21.15.90:0/2728603345 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563c58657030 msgr2=0x563c58662ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:36:47.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.227+0000 7f7fbd800640 1 --2- 172.21.15.90:0/2728603345 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563c58657030 0x563c58662ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:47.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.227+0000 7f7fbd800640 1 -- 172.21.15.90:0/2728603345 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563c5865f6c0 msgr2=0x563c58663440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:36:47.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.227+0000 7f7fbd800640 1 --2- 172.21.15.90:0/2728603345 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563c5865f6c0 0x563c58663440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:47.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.227+0000 7f7fbd800640 1 -- 172.21.15.90:0/2728603345 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563c586e74d0 con 0x563c58656350 2024-04-02T14:36:47.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.228+0000 7f7fbd800640 1 --2- 172.21.15.90:0/2728603345 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563c58656350 0x563c58662980 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f7fb00efd90 tx=0x7f7fb00ba870 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fc9a60a5041833f0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:36:47.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.228+0000 7f7fa3400640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:36:47.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.228+0000 7f7fa3400640 1 -- 172.21.15.90:0/2728603345 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fb00bc670 con 0x563c58656350 2024-04-02T14:36:47.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.228+0000 7f7fa3400640 1 -- 172.21.15.90:0/2728603345 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f7fb00dd8f0 con 0x563c58656350 2024-04-02T14:36:47.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.228+0000 7f7fc2e29740 1 -- 172.21.15.90:0/2728603345 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7fa40183c0 con 0x563c58656350 2024-04-02T14:36:47.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.229+0000 7f7fa3400640 1 -- 172.21.15.90:0/2728603345 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f7fb00bdca0 con 0x563c58656350 2024-04-02T14:36:47.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.229+0000 7f7fc2e29740 10 client.25951 fetch_fsmap learned FSMap version 27 2024-04-02T14:36:47.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.229+0000 7f7fc2e29740 1 -- 172.21.15.90:0/2728603345 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x563c586e8780 con 0x563c58656350 2024-04-02T14:36:47.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.229+0000 7f7fa3400640 1 -- 172.21.15.90:0/2728603345 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7fa40183c0 con 0x563c58656350 2024-04-02T14:36:47.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.230+0000 7f7fa3400640 1 -- 172.21.15.90:0/2728603345 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f7fb00ddce0 con 0x563c58656350 2024-04-02T14:36:47.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.230+0000 7f7fc2e29740 10 client.25951 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:36:47.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.230+0000 7f7fc2e29740 10 client.25951 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:36:47.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.230+0000 7f7fc2e29740 20 client.25951 populate_metadata read hostname 'smithi090' 2024-04-02T14:36:47.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.230+0000 7f7fc2e29740 1 --2- 172.21.15.90:0/2728603345 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x563c586e8c50 0x563c586eb060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:47.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.230+0000 7f7fc2e29740 4 client.25951 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-02T14:36:47.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.230+0000 7f7fc2e29740 1 -- 172.21.15.90:0/2728603345 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x563c585c1ba0 con 0x563c586e8c50 2024-04-02T14:36:47.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.230+0000 7f7fbce00640 1 --2- 172.21.15.90:0/2728603345 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x563c586e8c50 0x563c586eb060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:47.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.231+0000 7f7fa3400640 10 client.25951 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:36:47.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.233+0000 7f7fbce00640 1 --2- 172.21.15.90:0/2728603345 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x563c586e8c50 0x563c586eb060 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8a8da79d89ba65b1 server_cookie=c0e7e4816cf966a6 in_seq=0 out_seq=0 2024-04-02T14:36:47.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.235+0000 7f7fa3400640 1 -- 172.21.15.90:0/2728603345 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x563c585c1ba0 con 0x563c586e8c50 2024-04-02T14:36:47.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.235+0000 7f7fa3400640 10 client.25951 handle_command_reply: tid=0 2024-04-02T14:36:47.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.235+0000 7f7fc2e29740 1 client.25951 shutdown 2024-04-02T14:36:47.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.235+0000 7f7fc2e29740 1 -- 172.21.15.90:0/2728603345 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563c58656350 msgr2=0x563c58662980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:47.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.235+0000 7f7fc2e29740 1 --2- 172.21.15.90:0/2728603345 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563c58656350 0x563c58662980 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f7fb00efd90 tx=0x7f7fb00ba870 comp rx=0 tx=0).stop 2024-04-02T14:36:47.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.235+0000 7f7fc2e29740 1 -- 172.21.15.90:0/2728603345 shutdown_connections 2024-04-02T14:36:47.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.235+0000 7f7fc2e29740 1 --2- 172.21.15.90:0/2728603345 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x563c586e8c50 0x563c586eb060 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:47.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.235+0000 7f7fc2e29740 1 --2- 172.21.15.90:0/2728603345 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563c5865f6c0 0x563c58663440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:47.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.235+0000 7f7fa3400640 0 client.25951 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:36:47.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.235+0000 7f7fc2e29740 1 --2- 172.21.15.90:0/2728603345 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563c58657030 0x563c58662ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:47.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.235+0000 7f7fc2e29740 1 --2- 172.21.15.90:0/2728603345 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563c58656350 0x563c58662980 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:47.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.235+0000 7f7fc2e29740 1 -- 172.21.15.90:0/2728603345 >> 172.21.15.90:0/2728603345 conn(0x563c585ea270 msgr2=0x563c5865cf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:36:47.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.236+0000 7f7fc2e29740 1 -- 172.21.15.90:0/2728603345 shutdown_connections 2024-04-02T14:36:47.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.236+0000 7f7fc2e29740 1 -- 172.21.15.90:0/2728603345 wait complete. 2024-04-02T14:36:47.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.236+0000 7f7fc2e29740 20 client.25951 trim_cache size 0 max 16384 2024-04-02T14:36:47.240 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-02T14:36:47.240 INFO:teuthology.orchestra.run.smithi090.stdout: "return_code": 0, 2024-04-02T14:36:47.240 INFO:teuthology.orchestra.run.smithi090.stdout: "scrub_tag": "a8d6deb9-b420-445f-82c9-65a84d385e84", 2024-04-02T14:36:47.240 INFO:teuthology.orchestra.run.smithi090.stdout: "mode": "asynchronous" 2024-04-02T14:36:47.240 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-02T14:36:47.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.238+0000 7f7fc0200640 1 -- 172.21.15.90:0/3832901719 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f7f9c0a3de0 msgr2=0x7f7f9c0a62a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:47.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.238+0000 7f7fc0200640 1 --2- 172.21.15.90:0/3832901719 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f7f9c0a3de0 0x7f7f9c0a62a0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f7fa8010540 tx=0x7f7fa8000f40 comp rx=0 tx=0).stop 2024-04-02T14:36:47.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.238+0000 7f7fc0200640 1 -- 172.21.15.90:0/3832901719 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7fb810e350 msgr2=0x7f7fb81a7110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:47.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.238+0000 7f7fc0200640 1 --2- 172.21.15.90:0/3832901719 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7fb810e350 0x7f7fb81a7110 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f7fb0000c00 tx=0x7f7fb00045e0 comp rx=0 tx=0).stop 2024-04-02T14:36:47.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.238+0000 7f7fc0200640 1 -- 172.21.15.90:0/3832901719 shutdown_connections 2024-04-02T14:36:47.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.238+0000 7f7fc0200640 1 --2- 172.21.15.90:0/3832901719 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f7f9c0a3de0 0x7f7f9c0a62a0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:47.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.238+0000 7f7fc0200640 1 --2- 172.21.15.90:0/3832901719 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7fb811b2e0 0x7f7fb81ab980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:47.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.238+0000 7f7fc0200640 1 --2- 172.21.15.90:0/3832901719 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7fb8110320 0x7f7fb81a7650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:47.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.238+0000 7f7fc0200640 1 --2- 172.21.15.90:0/3832901719 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7fb810e350 0x7f7fb81a7110 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:47.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.238+0000 7f7fc0200640 1 -- 172.21.15.90:0/3832901719 >> 172.21.15.90:0/3832901719 conn(0x7f7fb806cbe0 msgr2=0x7f7fb811c1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:36:47.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.238+0000 7f7fc0200640 1 -- 172.21.15.90:0/3832901719 shutdown_connections 2024-04-02T14:36:47.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.239+0000 7f7fc0200640 1 -- 172.21.15.90:0/3832901719 wait complete. 2024-04-02T14:36:47.269 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-02T14:36:47.269 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-02T14:36:47.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.595+0000 7fd857200640 1 -- 172.21.15.90:0/1470045299 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd850098160 msgr2=0x7fd850098540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:47.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.595+0000 7fd857200640 1 --2- 172.21.15.90:0/1470045299 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd850098160 0x7fd850098540 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fd848009860 tx=0x7fd84802bb70 comp rx=0 tx=0).stop 2024-04-02T14:36:47.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.596+0000 7fd857200640 1 -- 172.21.15.90:0/1470045299 shutdown_connections 2024-04-02T14:36:47.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.596+0000 7fd857200640 1 --2- 172.21.15.90:0/1470045299 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd85009d230 0x7fd8500a4e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:47.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.596+0000 7fd857200640 1 --2- 172.21.15.90:0/1470045299 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd850098b10 0x7fd85009cb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:47.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.596+0000 7fd857200640 1 --2- 172.21.15.90:0/1470045299 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd850098160 0x7fd850098540 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:47.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.596+0000 7fd857200640 1 -- 172.21.15.90:0/1470045299 >> 172.21.15.90:0/1470045299 conn(0x7fd8500939c0 msgr2=0x7fd850095de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:36:47.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.596+0000 7fd857200640 1 -- 172.21.15.90:0/1470045299 shutdown_connections 2024-04-02T14:36:47.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.596+0000 7fd857200640 1 -- 172.21.15.90:0/1470045299 wait complete. 2024-04-02T14:36:47.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.598+0000 7fd857200640 1 Processor -- start 2024-04-02T14:36:47.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.598+0000 7fd857200640 1 -- start start 2024-04-02T14:36:47.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.598+0000 7fd857200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd850098160 0x7fd8500a4d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:47.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.599+0000 7fd857200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd85009d230 0x7fd8500a2e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:47.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.599+0000 7fd857200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd8500a3520 0x7fd85013ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:47.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.599+0000 7fd855e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd850098160 0x7fd8500a4d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:47.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.599+0000 7fd857200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fd8500098d0 con 0x7fd8500a3520 2024-04-02T14:36:47.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.599+0000 7fd857200640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fd850009750 con 0x7fd850098160 2024-04-02T14:36:47.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.599+0000 7fd855400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd85009d230 0x7fd8500a2e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:47.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.599+0000 7fd855e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd850098160 0x7fd8500a4d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:57638/0 (socket says 172.21.15.90:57638) 2024-04-02T14:36:47.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.599+0000 7fd855e00640 1 -- 172.21.15.90:0/1128229906 learned_addr learned my addr 172.21.15.90:0/1128229906 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:36:47.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.599+0000 7fd857200640 1 -- 172.21.15.90:0/1128229906 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fd850009a50 con 0x7fd85009d230 2024-04-02T14:36:47.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.599+0000 7fd856800640 1 --2- 172.21.15.90:0/1128229906 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd8500a3520 0x7fd85013ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:47.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.600+0000 7fd855e00640 1 -- 172.21.15.90:0/1128229906 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd85009d230 msgr2=0x7fd8500a2e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:47.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.600+0000 7fd855e00640 1 --2- 172.21.15.90:0/1128229906 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd85009d230 0x7fd8500a2e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:47.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.600+0000 7fd855e00640 1 -- 172.21.15.90:0/1128229906 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd8500a3520 msgr2=0x7fd85013ef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:47.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.600+0000 7fd855e00640 1 --2- 172.21.15.90:0/1128229906 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd8500a3520 0x7fd85013ef70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:47.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.600+0000 7fd855e00640 1 -- 172.21.15.90:0/1128229906 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd85013f4b0 con 0x7fd850098160 2024-04-02T14:36:47.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.600+0000 7fd856800640 1 --2- 172.21.15.90:0/1128229906 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd8500a3520 0x7fd85013ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:36:47.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.601+0000 7fd855e00640 1 --2- 172.21.15.90:0/1128229906 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd850098160 0x7fd8500a4d80 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fd848037040 tx=0x7fd8480323b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:36:47.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.601+0000 7fd84ea00640 1 -- 172.21.15.90:0/1128229906 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd848041020 con 0x7fd850098160 2024-04-02T14:36:47.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.601+0000 7fd84ea00640 1 -- 172.21.15.90:0/1128229906 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd84802c9f0 con 0x7fd850098160 2024-04-02T14:36:47.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.601+0000 7fd84ea00640 1 -- 172.21.15.90:0/1128229906 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd848039040 con 0x7fd850098160 2024-04-02T14:36:47.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.601+0000 7fd857200640 1 -- 172.21.15.90:0/1128229906 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd85013f680 con 0x7fd850098160 2024-04-02T14:36:47.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.602+0000 7fd857200640 1 -- 172.21.15.90:0/1128229906 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd85013fa30 con 0x7fd850098160 2024-04-02T14:36:47.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.608+0000 7fd85fa22740 1 Processor -- start 2024-04-02T14:36:47.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.608+0000 7fd84ea00640 1 -- 172.21.15.90:0/1128229906 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fd848044020 con 0x7fd850098160 2024-04-02T14:36:47.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.608+0000 7fd85fa22740 1 -- start start 2024-04-02T14:36:47.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.609+0000 7fd85fa22740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d08c77f980 0x55d08c77fd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:47.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.610+0000 7fd85fa22740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d08c7802a0 0x55d08c788940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:47.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.611+0000 7fd85fa22740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d08c788e80 0x55d08c78b270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:47.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.611+0000 7fd85fa22740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55d08c6e3770 con 0x55d08c77f980 2024-04-02T14:36:47.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.611+0000 7fd855e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d08c7802a0 0x55d08c788940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:47.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.611+0000 7fd855e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d08c7802a0 0x55d08c788940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:41488/0 (socket says 172.21.15.90:41488) 2024-04-02T14:36:47.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.611+0000 7fd855e00640 1 -- 172.21.15.90:0/3642807452 learned_addr learned my addr 172.21.15.90:0/3642807452 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:36:47.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.611+0000 7fd856800640 1 --2- 172.21.15.90:0/3642807452 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d08c77f980 0x55d08c77fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:47.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.612+0000 7fd85fa22740 1 -- 172.21.15.90:0/3642807452 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55d08c68b700 con 0x55d08c788e80 2024-04-02T14:36:47.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.612+0000 7fd85fa22740 1 -- 172.21.15.90:0/3642807452 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55d08c6df720 con 0x55d08c7802a0 2024-04-02T14:36:47.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.612+0000 7fd855400640 1 --2- 172.21.15.90:0/3642807452 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d08c788e80 0x55d08c78b270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:47.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.613+0000 7fd84ea00640 1 --2- 172.21.15.90:0/1128229906 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fd83c0a3da0 0x7fd83c0a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:47.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.613+0000 7fd855e00640 1 -- 172.21.15.90:0/3642807452 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d08c788e80 msgr2=0x55d08c78b270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:36:47.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.613+0000 7fd855e00640 1 --2- 172.21.15.90:0/3642807452 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d08c788e80 0x55d08c78b270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:47.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.613+0000 7fd855e00640 1 -- 172.21.15.90:0/3642807452 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d08c77f980 msgr2=0x55d08c77fd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:36:47.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.613+0000 7fd855e00640 1 --2- 172.21.15.90:0/3642807452 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d08c77f980 0x55d08c77fd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:47.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.613+0000 7fd855e00640 1 -- 172.21.15.90:0/3642807452 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d08c4c17d0 con 0x55d08c7802a0 2024-04-02T14:36:47.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.614+0000 7fd84ea00640 1 -- 172.21.15.90:0/1128229906 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7fd8480f20d0 con 0x7fd850098160 2024-04-02T14:36:47.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.614+0000 7fd855400640 1 --2- 172.21.15.90:0/3642807452 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d08c788e80 0x55d08c78b270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:36:47.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.614+0000 7fd855e00640 1 --2- 172.21.15.90:0/3642807452 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d08c7802a0 0x55d08c788940 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fd8480ef980 tx=0x7fd84803d000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4f568020b29302df server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:36:47.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.614+0000 7fd856800640 1 --2- 172.21.15.90:0/3642807452 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d08c77f980 0x55d08c77fd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:36:47.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.615+0000 7fd843000640 1 -- 172.21.15.90:0/3642807452 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8480beb40 con 0x55d08c7802a0 2024-04-02T14:36:47.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.616+0000 7fd843000640 1 -- 172.21.15.90:0/3642807452 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd84804e350 con 0x55d08c7802a0 2024-04-02T14:36:47.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.616+0000 7fd843000640 1 -- 172.21.15.90:0/3642807452 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8480be960 con 0x55d08c7802a0 2024-04-02T14:36:47.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.616+0000 7fd856800640 1 --2- 172.21.15.90:0/1128229906 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fd83c0a3da0 0x7fd83c0a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:47.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.617+0000 7fd85fa22740 1 -- 172.21.15.90:0/3642807452 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d08c7802a0 msgr2=0x55d08c788940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:47.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.617+0000 7fd85fa22740 1 --2- 172.21.15.90:0/3642807452 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d08c7802a0 0x55d08c788940 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fd8480ef980 tx=0x7fd84803d000 comp rx=0 tx=0).stop 2024-04-02T14:36:47.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.617+0000 7fd85fa22740 1 -- 172.21.15.90:0/3642807452 shutdown_connections 2024-04-02T14:36:47.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.617+0000 7fd85fa22740 1 --2- 172.21.15.90:0/3642807452 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d08c788e80 0x55d08c78b270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:47.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.617+0000 7fd85fa22740 1 --2- 172.21.15.90:0/3642807452 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d08c7802a0 0x55d08c788940 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:47.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.617+0000 7fd85fa22740 1 --2- 172.21.15.90:0/3642807452 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d08c77f980 0x55d08c77fd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:47.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.617+0000 7fd85fa22740 1 -- 172.21.15.90:0/3642807452 >> 172.21.15.90:0/3642807452 conn(0x55d08c77a0b0 msgr2=0x55d08c77a490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:36:47.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.619+0000 7fd856800640 1 --2- 172.21.15.90:0/1128229906 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fd83c0a3da0 0x7fd83c0a6260 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fd858067230 tx=0x7fd858077040 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:36:47.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.620+0000 7fd85fa22740 1 -- 172.21.15.90:0/3642807452 shutdown_connections 2024-04-02T14:36:47.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.620+0000 7fd85fa22740 1 -- 172.21.15.90:0/3642807452 wait complete. 2024-04-02T14:36:47.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.621+0000 7fd85fa22740 1 -- start start 2024-04-02T14:36:47.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.621+0000 7fd85fa22740 1 Processor -- start 2024-04-02T14:36:47.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.626+0000 7fd85fa22740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d08c77f980 0x55d08c785ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:47.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.626+0000 7fd85fa22740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d08c7802a0 0x55d08c786400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:47.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.626+0000 7fd85fa22740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d08c788e80 0x55d08c786940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:47.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.627+0000 7fd856800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d08c788e80 0x55d08c786940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:47.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.627+0000 7fd856800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d08c788e80 0x55d08c786940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:41490/0 (socket says 172.21.15.90:41490) 2024-04-02T14:36:47.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.627+0000 7fd856800640 1 -- 172.21.15.90:0/2568631118 learned_addr learned my addr 172.21.15.90:0/2568631118 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:36:47.630 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.627+0000 7fd85fa22740 1 -- 172.21.15.90:0/2568631118 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55d08c78ba30 con 0x55d08c77f980 2024-04-02T14:36:47.630 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.627+0000 7fd85fa22740 1 -- 172.21.15.90:0/2568631118 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55d08c69eac0 con 0x55d08c7802a0 2024-04-02T14:36:47.630 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.627+0000 7fd85fa22740 1 -- 172.21.15.90:0/2568631118 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55d08c78bbb0 con 0x55d08c788e80 2024-04-02T14:36:47.630 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.627+0000 7fd855400640 1 --2- 172.21.15.90:0/2568631118 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d08c7802a0 0x55d08c786400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:47.630 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.627+0000 7fd85fa22740 10 client.0 _ll_register_callbacks cb 0x55d08c6a1180 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:36:47.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.628+0000 7fd856800640 1 -- 172.21.15.90:0/2568631118 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d08c7802a0 msgr2=0x55d08c786400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:36:47.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.628+0000 7fd855e00640 1 --2- 172.21.15.90:0/2568631118 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d08c77f980 0x55d08c785ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:47.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.628+0000 7fd856800640 1 --2- 172.21.15.90:0/2568631118 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d08c7802a0 0x55d08c786400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:47.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.628+0000 7fd856800640 1 -- 172.21.15.90:0/2568631118 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d08c77f980 msgr2=0x55d08c785ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:36:47.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.628+0000 7fd856800640 1 --2- 172.21.15.90:0/2568631118 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d08c77f980 0x55d08c785ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:47.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.628+0000 7fd856800640 1 -- 172.21.15.90:0/2568631118 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d08c3f73f0 con 0x55d08c788e80 2024-04-02T14:36:47.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.629+0000 7fd856800640 1 --2- 172.21.15.90:0/2568631118 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d08c788e80 0x55d08c786940 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fd858064ff0 tx=0x7fd858075040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6f986bc4c1366f03 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:36:47.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.629+0000 7fd837400640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:36:47.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.631+0000 7fd837400640 1 -- 172.21.15.90:0/2568631118 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd858093060 con 0x55d08c788e80 2024-04-02T14:36:47.634 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.631+0000 7fd837400640 1 -- 172.21.15.90:0/2568631118 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7fd858081cf0 con 0x55d08c788e80 2024-04-02T14:36:47.634 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.632+0000 7fd85fa22740 1 -- 172.21.15.90:0/2568631118 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd84804e350 con 0x55d08c788e80 2024-04-02T14:36:47.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.633+0000 7fd837400640 1 -- 172.21.15.90:0/2568631118 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fd85807e040 con 0x55d08c788e80 2024-04-02T14:36:47.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.634+0000 7fd85fa22740 10 client.25963 fetch_fsmap learned FSMap version 27 2024-04-02T14:36:47.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.634+0000 7fd85fa22740 1 -- 172.21.15.90:0/2568631118 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55d08c3b70f0 con 0x55d08c788e80 2024-04-02T14:36:47.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.635+0000 7fd837400640 1 -- 172.21.15.90:0/2568631118 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd84804e350 con 0x55d08c788e80 2024-04-02T14:36:47.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.635+0000 7fd837400640 1 -- 172.21.15.90:0/2568631118 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fd858081070 con 0x55d08c788e80 2024-04-02T14:36:47.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.636+0000 7fd85fa22740 10 client.25963 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:36:47.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.636+0000 7fd85fa22740 10 client.25963 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:36:47.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.636+0000 7fd85fa22740 20 client.25963 populate_metadata read hostname 'smithi090' 2024-04-02T14:36:47.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.636+0000 7fd85fa22740 1 --2- 172.21.15.90:0/2568631118 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55d08c798990 0x55d08c79adb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:47.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.636+0000 7fd85fa22740 4 client.25963 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:36:47.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.636+0000 7fd85fa22740 1 -- 172.21.15.90:0/2568631118 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d08c684630 con 0x55d08c798990 2024-04-02T14:36:47.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.637+0000 7fd855e00640 1 --2- 172.21.15.90:0/2568631118 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55d08c798990 0x55d08c79adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:47.640 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.639+0000 7fd837400640 10 client.25963 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:36:47.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.639+0000 7fd855e00640 1 --2- 172.21.15.90:0/2568631118 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55d08c798990 0x55d08c79adb0 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=63060423b3dd0a94 server_cookie=bfa778fd1955838f in_seq=0 out_seq=0 2024-04-02T14:36:47.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.640+0000 7fd837400640 1 -- 172.21.15.90:0/2568631118 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d08c684630 con 0x55d08c798990 2024-04-02T14:36:47.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.640+0000 7fd837400640 10 client.25963 handle_command_reply: tid=0 2024-04-02T14:36:47.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.640+0000 7fd85fa22740 1 client.25963 shutdown 2024-04-02T14:36:47.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.641+0000 7fd85fa22740 1 -- 172.21.15.90:0/2568631118 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d08c788e80 msgr2=0x55d08c786940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:47.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.641+0000 7fd85fa22740 1 --2- 172.21.15.90:0/2568631118 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d08c788e80 0x55d08c786940 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fd858064ff0 tx=0x7fd858075040 comp rx=0 tx=0).stop 2024-04-02T14:36:47.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.641+0000 7fd85fa22740 1 -- 172.21.15.90:0/2568631118 shutdown_connections 2024-04-02T14:36:47.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.641+0000 7fd85fa22740 1 --2- 172.21.15.90:0/2568631118 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55d08c798990 0x55d08c79adb0 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:47.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.641+0000 7fd85fa22740 1 --2- 172.21.15.90:0/2568631118 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d08c788e80 0x55d08c786940 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:47.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.641+0000 7fd85fa22740 1 --2- 172.21.15.90:0/2568631118 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d08c7802a0 0x55d08c786400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:47.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.641+0000 7fd85fa22740 1 --2- 172.21.15.90:0/2568631118 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d08c77f980 0x55d08c785ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:47.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.641+0000 7fd85fa22740 1 -- 172.21.15.90:0/2568631118 >> 172.21.15.90:0/2568631118 conn(0x55d08c77a0b0 msgr2=0x55d08c77be00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:36:47.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.642+0000 7fd837400640 0 client.25963 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:36:47.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.642+0000 7fd85fa22740 1 -- 172.21.15.90:0/2568631118 shutdown_connections 2024-04-02T14:36:47.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.642+0000 7fd85fa22740 1 -- 172.21.15.90:0/2568631118 wait complete. 2024-04-02T14:36:47.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.643+0000 7fd85fa22740 20 client.25963 trim_cache size 0 max 16384 2024-04-02T14:36:47.689 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:36:47 smithi090 ceph-mon[31674]: pgmap v60: 97 pgs: 97 active+clean; 1.2 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 8.7 MiB/s wr, 671 op/s 2024-04-02T14:36:47.690 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.686+0000 7fd85fa22740 1 Processor -- start 2024-04-02T14:36:47.690 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.687+0000 7fd85fa22740 1 -- start start 2024-04-02T14:36:47.690 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.687+0000 7fd85fa22740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d08c77f980 0x55d08c794840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:47.690 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.687+0000 7fd85fa22740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d08c7802a0 0x55d08c786760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:47.690 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.687+0000 7fd85fa22740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d08c788e80 0x55d08c779190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:47.690 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.687+0000 7fd85fa22740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fd850148160 con 0x55d08c77f980 2024-04-02T14:36:47.690 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.687+0000 7fd85fa22740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fd850149ca0 con 0x55d08c788e80 2024-04-02T14:36:47.691 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.687+0000 7fd85fa22740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fd850148830 con 0x55d08c7802a0 2024-04-02T14:36:47.691 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.688+0000 7fd855400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d08c7802a0 0x55d08c786760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:47.691 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.688+0000 7fd855400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d08c7802a0 0x55d08c786760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:41494/0 (socket says 172.21.15.90:41494) 2024-04-02T14:36:47.691 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.688+0000 7fd855400640 1 -- 172.21.15.90:0/2110861504 learned_addr learned my addr 172.21.15.90:0/2110861504 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:36:47.691 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.688+0000 7fd856800640 1 --2- 172.21.15.90:0/2110861504 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d08c788e80 0x55d08c779190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:47.691 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.688+0000 7fd855e00640 1 --2- 172.21.15.90:0/2110861504 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d08c77f980 0x55d08c794840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:47.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.689+0000 7fd855400640 1 -- 172.21.15.90:0/2110861504 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d08c788e80 msgr2=0x55d08c779190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:36:47.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.689+0000 7fd855400640 1 --2- 172.21.15.90:0/2110861504 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d08c788e80 0x55d08c779190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:47.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.689+0000 7fd855400640 1 -- 172.21.15.90:0/2110861504 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d08c77f980 msgr2=0x55d08c794840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:36:47.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.689+0000 7fd855400640 1 --2- 172.21.15.90:0/2110861504 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d08c77f980 0x55d08c794840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:47.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.689+0000 7fd855400640 1 -- 172.21.15.90:0/2110861504 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d08c78b270 con 0x55d08c7802a0 2024-04-02T14:36:47.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.689+0000 7fd856800640 1 --2- 172.21.15.90:0/2110861504 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d08c788e80 0x55d08c779190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:36:47.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.689+0000 7fd855e00640 1 --2- 172.21.15.90:0/2110861504 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d08c77f980 0x55d08c794840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:36:47.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.689+0000 7fd855400640 1 --2- 172.21.15.90:0/2110861504 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d08c7802a0 0x55d08c786760 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fd844009dc0 tx=0x7fd844010800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=47968994b9ec1274 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:36:47.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.690+0000 7fd842600640 1 -- 172.21.15.90:0/2110861504 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd844005220 con 0x55d08c7802a0 2024-04-02T14:36:47.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.690+0000 7fd842600640 1 -- 172.21.15.90:0/2110861504 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd8440047a0 con 0x55d08c7802a0 2024-04-02T14:36:47.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.690+0000 7fd842600640 1 -- 172.21.15.90:0/2110861504 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd84400f980 con 0x55d08c7802a0 2024-04-02T14:36:47.694 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.691+0000 7fd85fa22740 1 -- 172.21.15.90:0/2110861504 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d08c7802a0 msgr2=0x55d08c786760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:47.694 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.691+0000 7fd85fa22740 1 --2- 172.21.15.90:0/2110861504 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d08c7802a0 0x55d08c786760 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fd844009dc0 tx=0x7fd844010800 comp rx=0 tx=0).stop 2024-04-02T14:36:47.694 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.692+0000 7fd85fa22740 1 -- 172.21.15.90:0/2110861504 shutdown_connections 2024-04-02T14:36:47.694 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.692+0000 7fd85fa22740 1 --2- 172.21.15.90:0/2110861504 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d08c788e80 0x55d08c779190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:47.694 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.692+0000 7fd85fa22740 1 --2- 172.21.15.90:0/2110861504 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d08c7802a0 0x55d08c786760 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:47.694 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.692+0000 7fd85fa22740 1 --2- 172.21.15.90:0/2110861504 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d08c77f980 0x55d08c794840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:47.694 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.692+0000 7fd85fa22740 1 -- 172.21.15.90:0/2110861504 >> 172.21.15.90:0/2110861504 conn(0x55d08c77a0b0 msgr2=0x55d08c778a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:36:47.695 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.692+0000 7fd85fa22740 1 -- 172.21.15.90:0/2110861504 shutdown_connections 2024-04-02T14:36:47.695 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.692+0000 7fd85fa22740 1 -- 172.21.15.90:0/2110861504 wait complete. 2024-04-02T14:36:47.695 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.693+0000 7fd85fa22740 1 -- start start 2024-04-02T14:36:47.695 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.693+0000 7fd85fa22740 1 Processor -- start 2024-04-02T14:36:47.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.694+0000 7fd85fa22740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d08c77f980 0x55d08c78c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:47.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.694+0000 7fd85fa22740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d08c7802a0 0x55d08c78cde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:47.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.694+0000 7fd85fa22740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d08c788e80 0x55d08c78d320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:47.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.694+0000 7fd85fa22740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fd8501495d0 con 0x55d08c7802a0 2024-04-02T14:36:47.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.694+0000 7fd85fa22740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fd850148f00 con 0x55d08c77f980 2024-04-02T14:36:47.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.694+0000 7fd85fa22740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fd850146140 con 0x55d08c788e80 2024-04-02T14:36:47.697 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.694+0000 7fd85fa22740 10 client.0 _ll_register_callbacks cb 0x7fd85806b2e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:36:47.697 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.694+0000 7fd855e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d08c77f980 0x55d08c78c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:47.697 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.694+0000 7fd855e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d08c77f980 0x55d08c78c8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:57668/0 (socket says 172.21.15.90:57668) 2024-04-02T14:36:47.697 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.694+0000 7fd855e00640 1 -- 172.21.15.90:0/3408132290 learned_addr learned my addr 172.21.15.90:0/3408132290 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:36:47.697 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.694+0000 7fd856800640 1 --2- 172.21.15.90:0/3408132290 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d08c788e80 0x55d08c78d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:47.697 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.694+0000 7fd855400640 1 --2- 172.21.15.90:0/3408132290 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d08c7802a0 0x55d08c78cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:47.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.695+0000 7fd855e00640 1 -- 172.21.15.90:0/3408132290 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d08c788e80 msgr2=0x55d08c78d320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:36:47.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.695+0000 7fd855e00640 1 --2- 172.21.15.90:0/3408132290 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d08c788e80 0x55d08c78d320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:47.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.695+0000 7fd855e00640 1 -- 172.21.15.90:0/3408132290 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d08c7802a0 msgr2=0x55d08c78cde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:36:47.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.695+0000 7fd855e00640 1 --2- 172.21.15.90:0/3408132290 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d08c7802a0 0x55d08c78cde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:47.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.695+0000 7fd855e00640 1 -- 172.21.15.90:0/3408132290 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d08c7fe060 con 0x55d08c77f980 2024-04-02T14:36:47.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.695+0000 7fd855e00640 1 --2- 172.21.15.90:0/3408132290 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d08c77f980 0x55d08c78c8a0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fd8480efd30 tx=0x7fd848009790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8bcf7f3e1819dce0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:36:47.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.695+0000 7fd837400640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:36:47.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.696+0000 7fd837400640 1 -- 172.21.15.90:0/3408132290 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8480be960 con 0x55d08c77f980 2024-04-02T14:36:47.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.696+0000 7fd837400640 1 -- 172.21.15.90:0/3408132290 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7fd8480bd300 con 0x55d08c77f980 2024-04-02T14:36:47.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.696+0000 7fd85fa22740 1 -- 172.21.15.90:0/3408132290 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd8440047a0 con 0x55d08c77f980 2024-04-02T14:36:47.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.696+0000 7fd837400640 1 -- 172.21.15.90:0/3408132290 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fd8480bedd0 con 0x55d08c77f980 2024-04-02T14:36:47.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.697+0000 7fd85fa22740 10 client.26051 fetch_fsmap learned FSMap version 27 2024-04-02T14:36:47.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.697+0000 7fd85fa22740 1 -- 172.21.15.90:0/3408132290 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55d08c7ff320 con 0x55d08c77f980 2024-04-02T14:36:47.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.697+0000 7fd837400640 1 -- 172.21.15.90:0/3408132290 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd8480c6f30 con 0x55d08c77f980 2024-04-02T14:36:47.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.698+0000 7fd837400640 1 -- 172.21.15.90:0/3408132290 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fd84803c030 con 0x55d08c77f980 2024-04-02T14:36:47.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.698+0000 7fd85fa22740 10 client.26051 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:36:47.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.698+0000 7fd85fa22740 10 client.26051 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:36:47.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.698+0000 7fd85fa22740 20 client.26051 populate_metadata read hostname 'smithi090' 2024-04-02T14:36:47.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.698+0000 7fd85fa22740 1 --2- 172.21.15.90:0/3408132290 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55d08c7ff8e0 0x55d08c801d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:47.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.698+0000 7fd85fa22740 4 client.26051 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-02T14:36:47.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.698+0000 7fd85fa22740 1 -- 172.21.15.90:0/3408132290 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "scrub status"}) -- 0x7fd8440047a0 con 0x55d08c7ff8e0 2024-04-02T14:36:47.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.698+0000 7fd855400640 1 --2- 172.21.15.90:0/3408132290 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55d08c7ff8e0 0x55d08c801d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:47.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.699+0000 7fd855400640 1 --2- 172.21.15.90:0/3408132290 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55d08c7ff8e0 0x55d08c801d00 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6cbebb4488f7e717 server_cookie=3ff06e6b0401b276 in_seq=0 out_seq=0 2024-04-02T14:36:47.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.699+0000 7fd837400640 10 client.26051 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:36:47.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.700+0000 7fd837400640 1 -- 172.21.15.90:0/3408132290 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x7fd8440047a0 con 0x55d08c7ff8e0 2024-04-02T14:36:47.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.700+0000 7fd837400640 10 client.26051 handle_command_reply: tid=0 2024-04-02T14:36:47.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.700+0000 7fd85fa22740 1 client.26051 shutdown 2024-04-02T14:36:47.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.700+0000 7fd85fa22740 1 -- 172.21.15.90:0/3408132290 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d08c77f980 msgr2=0x55d08c78c8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:47.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.701+0000 7fd85fa22740 1 --2- 172.21.15.90:0/3408132290 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d08c77f980 0x55d08c78c8a0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fd8480efd30 tx=0x7fd848009790 comp rx=0 tx=0).stop 2024-04-02T14:36:47.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.701+0000 7fd85fa22740 1 -- 172.21.15.90:0/3408132290 shutdown_connections 2024-04-02T14:36:47.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.701+0000 7fd85fa22740 1 --2- 172.21.15.90:0/3408132290 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55d08c7ff8e0 0x55d08c801d00 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:47.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.701+0000 7fd837400640 0 client.26051 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:36:47.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.701+0000 7fd85fa22740 1 --2- 172.21.15.90:0/3408132290 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d08c788e80 0x55d08c78d320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:47.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.701+0000 7fd85fa22740 1 --2- 172.21.15.90:0/3408132290 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d08c7802a0 0x55d08c78cde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:47.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.701+0000 7fd85fa22740 1 --2- 172.21.15.90:0/3408132290 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d08c77f980 0x55d08c78c8a0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:47.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.701+0000 7fd85fa22740 1 -- 172.21.15.90:0/3408132290 >> 172.21.15.90:0/3408132290 conn(0x55d08c77a0b0 msgr2=0x55d08c779310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:36:47.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.702+0000 7fd85fa22740 1 -- 172.21.15.90:0/3408132290 shutdown_connections 2024-04-02T14:36:47.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.702+0000 7fd85fa22740 1 -- 172.21.15.90:0/3408132290 wait complete. 2024-04-02T14:36:47.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.702+0000 7fd85fa22740 20 client.26051 trim_cache size 0 max 16384 2024-04-02T14:36:47.705 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-02T14:36:47.706 INFO:teuthology.orchestra.run.smithi090.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-02T14:36:47.706 INFO:teuthology.orchestra.run.smithi090.stdout: "scrubs": { 2024-04-02T14:36:47.706 INFO:teuthology.orchestra.run.smithi090.stdout: "a8d6deb9-b420-445f-82c9-65a84d385e84": { 2024-04-02T14:36:47.706 INFO:teuthology.orchestra.run.smithi090.stdout: "path": "/", 2024-04-02T14:36:47.706 INFO:teuthology.orchestra.run.smithi090.stdout: "tag": "a8d6deb9-b420-445f-82c9-65a84d385e84", 2024-04-02T14:36:47.706 INFO:teuthology.orchestra.run.smithi090.stdout: "options": "recursive,force" 2024-04-02T14:36:47.706 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:36:47.706 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:36:47.706 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-02T14:36:47.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.703+0000 7fd857200640 1 -- 172.21.15.90:0/1128229906 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fd83c0a3da0 msgr2=0x7fd83c0a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:47.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.703+0000 7fd857200640 1 --2- 172.21.15.90:0/1128229906 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fd83c0a3da0 0x7fd83c0a6260 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fd858067230 tx=0x7fd858077040 comp rx=0 tx=0).stop 2024-04-02T14:36:47.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.703+0000 7fd857200640 1 -- 172.21.15.90:0/1128229906 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd850098160 msgr2=0x7fd8500a4d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:47.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.703+0000 7fd857200640 1 --2- 172.21.15.90:0/1128229906 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd850098160 0x7fd8500a4d80 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fd848037040 tx=0x7fd8480323b0 comp rx=0 tx=0).stop 2024-04-02T14:36:47.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.703+0000 7fd857200640 1 -- 172.21.15.90:0/1128229906 shutdown_connections 2024-04-02T14:36:47.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.703+0000 7fd857200640 1 --2- 172.21.15.90:0/1128229906 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fd83c0a3da0 0x7fd83c0a6260 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:47.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.703+0000 7fd857200640 1 --2- 172.21.15.90:0/1128229906 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd8500a3520 0x7fd85013ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:47.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.703+0000 7fd857200640 1 --2- 172.21.15.90:0/1128229906 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd85009d230 0x7fd8500a2e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:47.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.703+0000 7fd857200640 1 --2- 172.21.15.90:0/1128229906 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd850098160 0x7fd8500a4d80 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:47.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.703+0000 7fd857200640 1 -- 172.21.15.90:0/1128229906 >> 172.21.15.90:0/1128229906 conn(0x7fd8500939c0 msgr2=0x7fd850094db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:36:47.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.704+0000 7fd857200640 1 -- 172.21.15.90:0/1128229906 shutdown_connections 2024-04-02T14:36:47.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:36:47.704+0000 7fd857200640 1 -- 172.21.15.90:0/1128229906 wait complete. 2024-04-02T14:36:47.730 INFO:tasks.cephfs.filesystem:scrub status for tag:a8d6deb9-b420-445f-82c9-65a84d385e84 - {'path': '/', 'tag': 'a8d6deb9-b420-445f-82c9-65a84d385e84', 'options': 'recursive,force'} 2024-04-02T14:36:47.860 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:36:47 smithi092 ceph-mon[36497]: pgmap v60: 97 pgs: 97 active+clean; 1.2 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 8.7 MiB/s wr, 671 op/s 2024-04-02T14:36:47.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:36:47 smithi177 ceph-mon[36431]: pgmap v60: 97 pgs: 97 active+clean; 1.2 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 8.7 MiB/s wr, 671 op/s 2024-04-02T14:36:48.860 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:36:48 smithi092 ceph-mon[36497]: scrub queued for path: / 2024-04-02T14:36:48.861 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:36:48 smithi092 ceph-mon[36497]: scrub summary: idle+waiting paths [/] 2024-04-02T14:36:48.861 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:36:48 smithi092 ceph-mon[36497]: scrub summary: active paths [/] 2024-04-02T14:36:48.861 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:36:48 smithi092 ceph-mon[36497]: scrub summary: idle+waiting paths [/] 2024-04-02T14:36:48.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:36:48 smithi177 ceph-mon[36431]: scrub queued for path: / 2024-04-02T14:36:48.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:36:48 smithi177 ceph-mon[36431]: scrub summary: idle+waiting paths [/] 2024-04-02T14:36:48.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:36:48 smithi177 ceph-mon[36431]: scrub summary: active paths [/] 2024-04-02T14:36:48.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:36:48 smithi177 ceph-mon[36431]: scrub summary: idle+waiting paths [/] 2024-04-02T14:36:48.938 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:36:48 smithi090 ceph-mon[31674]: scrub queued for path: / 2024-04-02T14:36:48.938 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:36:48 smithi090 ceph-mon[31674]: scrub summary: idle+waiting paths [/] 2024-04-02T14:36:48.938 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:36:48 smithi090 ceph-mon[31674]: scrub summary: active paths [/] 2024-04-02T14:36:48.938 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:36:48 smithi090 ceph-mon[31674]: scrub summary: idle+waiting paths [/] 2024-04-02T14:36:49.860 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:36:49 smithi092 ceph-mon[36497]: pgmap v61: 97 pgs: 97 active+clean; 1.2 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 12 MiB/s wr, 1.03k op/s 2024-04-02T14:36:49.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:36:49 smithi177 ceph-mon[36431]: pgmap v61: 97 pgs: 97 active+clean; 1.2 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 12 MiB/s wr, 1.03k op/s 2024-04-02T14:36:49.938 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:36:49 smithi090 ceph-mon[31674]: pgmap v61: 97 pgs: 97 active+clean; 1.2 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 12 MiB/s wr, 1.03k op/s 2024-04-02T14:36:50.960 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:50.957+0000 7fda22c00640 1 -- 172.21.15.90:0/3920753204 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fda1c112950 msgr2=0x7fda1c114d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:50.961 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:50.957+0000 7fda22c00640 1 --2- 172.21.15.90:0/3920753204 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fda1c112950 0x7fda1c114d40 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fda14009cd0 tx=0x7fda1402c100 comp rx=0 tx=0).stop 2024-04-02T14:36:50.961 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:50.958+0000 7fda22c00640 1 -- 172.21.15.90:0/3920753204 shutdown_connections 2024-04-02T14:36:50.961 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:50.958+0000 7fda22c00640 1 --2- 172.21.15.90:0/3920753204 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fda1c112950 0x7fda1c114d40 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:50.961 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:50.958+0000 7fda22c00640 1 --2- 172.21.15.90:0/3920753204 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fda1c076950 0x7fda1c112410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:50.961 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:50.958+0000 7fda22c00640 1 --2- 172.21.15.90:0/3920753204 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fda1c076030 0x7fda1c076410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:50.961 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:50.958+0000 7fda22c00640 1 -- 172.21.15.90:0/3920753204 >> 172.21.15.90:0/3920753204 conn(0x7fda1c0ff780 msgr2=0x7fda1c101ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:36:50.961 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:50.958+0000 7fda22c00640 1 -- 172.21.15.90:0/3920753204 shutdown_connections 2024-04-02T14:36:50.962 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:50.958+0000 7fda22c00640 1 -- 172.21.15.90:0/3920753204 wait complete. 2024-04-02T14:36:50.962 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:50.959+0000 7fda22c00640 1 Processor -- start 2024-04-02T14:36:50.962 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:50.959+0000 7fda22c00640 1 -- start start 2024-04-02T14:36:50.964 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:50.959+0000 7fda22c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fda1c076030 0x7fda1c1a4690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:50.964 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:50.960+0000 7fda22c00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fda1c076950 0x7fda1c1a4bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:50.964 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:50.960+0000 7fda22c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fda1c112950 0x7fda1c19e750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:50.964 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:50.960+0000 7fda22c00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fda1c118470 con 0x7fda1c112950 2024-04-02T14:36:50.964 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:50.960+0000 7fda22c00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fda1c1182f0 con 0x7fda1c076950 2024-04-02T14:36:50.965 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:50.960+0000 7fda22c00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fda1c1185f0 con 0x7fda1c076030 2024-04-02T14:36:50.965 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:50.960+0000 7fda1b400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fda1c076950 0x7fda1c1a4bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:50.965 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:50.960+0000 7fda1b400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fda1c076950 0x7fda1c1a4bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:57686/0 (socket says 172.21.15.90:57686) 2024-04-02T14:36:50.965 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:50.960+0000 7fda1b400640 1 -- 172.21.15.90:0/4007083561 learned_addr learned my addr 172.21.15.90:0/4007083561 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:36:50.965 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:50.960+0000 7fda1be00640 1 --2- 172.21.15.90:0/4007083561 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fda1c076030 0x7fda1c1a4690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:50.965 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:50.961+0000 7fda1b400640 1 -- 172.21.15.90:0/4007083561 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fda1c076030 msgr2=0x7fda1c1a4690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:50.965 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:50.961+0000 7fda1b400640 1 --2- 172.21.15.90:0/4007083561 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fda1c076030 0x7fda1c1a4690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:50.965 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:50.961+0000 7fda1b400640 1 -- 172.21.15.90:0/4007083561 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fda1c112950 msgr2=0x7fda1c19e750 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-02T14:36:50.965 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:50.961+0000 7fda20c00640 1 --2- 172.21.15.90:0/4007083561 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fda1c112950 0x7fda1c19e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:50.965 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:50.961+0000 7fda1b400640 1 --2- 172.21.15.90:0/4007083561 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fda1c112950 0x7fda1c19e750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:50.966 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:50.961+0000 7fda1b400640 1 -- 172.21.15.90:0/4007083561 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda1c19f010 con 0x7fda1c076950 2024-04-02T14:36:50.966 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:50.961+0000 7fda1b400640 1 --2- 172.21.15.90:0/4007083561 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fda1c076950 0x7fda1c1a4bd0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fda0800d950 tx=0x7fda0800de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:36:50.966 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:50.962+0000 7fda18c00640 1 -- 172.21.15.90:0/4007083561 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda08028030 con 0x7fda1c076950 2024-04-02T14:36:50.966 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:50.962+0000 7fda18c00640 1 -- 172.21.15.90:0/4007083561 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fda080099d0 con 0x7fda1c076950 2024-04-02T14:36:50.966 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:50.962+0000 7fda18c00640 1 -- 172.21.15.90:0/4007083561 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda080047a0 con 0x7fda1c076950 2024-04-02T14:36:50.966 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:50.962+0000 7fda22c00640 1 -- 172.21.15.90:0/4007083561 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda1c19f2a0 con 0x7fda1c076950 2024-04-02T14:36:50.967 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:50.962+0000 7fda22c00640 1 -- 172.21.15.90:0/4007083561 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fda1c06b520 con 0x7fda1c076950 2024-04-02T14:36:50.969 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:50.967+0000 7fda18c00640 1 -- 172.21.15.90:0/4007083561 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fda08009b80 con 0x7fda1c076950 2024-04-02T14:36:50.971 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:50.968+0000 7fda18c00640 1 --2- 172.21.15.90:0/4007083561 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fda000a3e40 0x7fda000a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:50.971 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:50.969+0000 7fda18c00640 1 -- 172.21.15.90:0/4007083561 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7fda080e10f0 con 0x7fda1c076950 2024-04-02T14:36:50.973 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:50.969+0000 7fda1be00640 1 --2- 172.21.15.90:0/4007083561 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fda000a3e40 0x7fda000a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:50.973 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:50.970+0000 7fda259e5740 1 Processor -- start 2024-04-02T14:36:50.974 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:50.971+0000 7fda259e5740 1 -- start start 2024-04-02T14:36:50.975 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:50.971+0000 7fda259e5740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556d0a8c8eb0 0x556d0a8c9290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:50.976 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:50.971+0000 7fda259e5740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556d0a8c97d0 0x556d0a8d1e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:50.976 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:50.971+0000 7fda259e5740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556d0a8d23b0 0x556d0a8d47a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:50.976 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:50.971+0000 7fda259e5740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x556d0a82dd70 con 0x556d0a8c8eb0 2024-04-02T14:36:50.976 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:50.971+0000 7fda259e5740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x556d0a7d62a0 con 0x556d0a8d23b0 2024-04-02T14:36:50.977 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:50.971+0000 7fda259e5740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x556d0a829d20 con 0x556d0a8c97d0 2024-04-02T14:36:50.977 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:50.972+0000 7fda1b400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556d0a8c97d0 0x556d0a8d1e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:50.977 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:50.972+0000 7fda1b400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556d0a8c97d0 0x556d0a8d1e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:41550/0 (socket says 172.21.15.90:41550) 2024-04-02T14:36:50.977 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:50.972+0000 7fda1b400640 1 -- 172.21.15.90:0/3446408265 learned_addr learned my addr 172.21.15.90:0/3446408265 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:36:50.978 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:50.972+0000 7fda20c00640 1 --2- 172.21.15.90:0/3446408265 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556d0a8d23b0 0x556d0a8d47a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:50.978 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:50.972+0000 7fda1b400640 1 -- 172.21.15.90:0/3446408265 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556d0a8d23b0 msgr2=0x556d0a8d47a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:36:50.978 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:50.972+0000 7fda1b400640 1 --2- 172.21.15.90:0/3446408265 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556d0a8d23b0 0x556d0a8d47a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:50.979 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:50.972+0000 7fda1b400640 1 -- 172.21.15.90:0/3446408265 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556d0a8c8eb0 msgr2=0x556d0a8c9290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:36:50.979 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:50.972+0000 7fda1b400640 1 --2- 172.21.15.90:0/3446408265 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556d0a8c8eb0 0x556d0a8c9290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:50.979 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:50.972+0000 7fda1b400640 1 -- 172.21.15.90:0/3446408265 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556d0a7ea840 con 0x556d0a8c97d0 2024-04-02T14:36:50.979 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:50.973+0000 7fda1b400640 1 --2- 172.21.15.90:0/3446408265 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556d0a8c97d0 0x556d0a8d1e70 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fda080a9a70 tx=0x7fda080accf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a47c662ab43ffdb2 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:36:50.979 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:50.973+0000 7fda20c00640 1 --2- 172.21.15.90:0/3446408265 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556d0a8d23b0 0x556d0a8d47a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:36:50.979 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:50.973+0000 7fda11200640 1 -- 172.21.15.90:0/3446408265 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda080b5d10 con 0x556d0a8c97d0 2024-04-02T14:36:50.980 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:50.973+0000 7fda11200640 1 -- 172.21.15.90:0/3446408265 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fda080ab3a0 con 0x556d0a8c97d0 2024-04-02T14:36:50.980 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:50.973+0000 7fda11200640 1 -- 172.21.15.90:0/3446408265 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda080ac4f0 con 0x556d0a8c97d0 2024-04-02T14:36:50.987 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:50.974+0000 7fda259e5740 1 -- 172.21.15.90:0/3446408265 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556d0a8c97d0 msgr2=0x556d0a8d1e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:50.988 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:50.974+0000 7fda259e5740 1 --2- 172.21.15.90:0/3446408265 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556d0a8c97d0 0x556d0a8d1e70 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fda080a9a70 tx=0x7fda080accf0 comp rx=0 tx=0).stop 2024-04-02T14:36:50.988 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:50.974+0000 7fda1be00640 1 --2- 172.21.15.90:0/4007083561 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fda000a3e40 0x7fda000a6300 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fda0c00bbe0 tx=0x7fda0c00b390 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:36:50.988 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:50.974+0000 7fda259e5740 1 -- 172.21.15.90:0/3446408265 shutdown_connections 2024-04-02T14:36:50.988 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:50.974+0000 7fda259e5740 1 --2- 172.21.15.90:0/3446408265 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556d0a8d23b0 0x556d0a8d47a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:50.988 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:50.974+0000 7fda259e5740 1 --2- 172.21.15.90:0/3446408265 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556d0a8c97d0 0x556d0a8d1e70 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:50.988 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:50.974+0000 7fda259e5740 1 --2- 172.21.15.90:0/3446408265 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556d0a8c8eb0 0x556d0a8c9290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:50.988 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:50.974+0000 7fda259e5740 1 -- 172.21.15.90:0/3446408265 >> 172.21.15.90:0/3446408265 conn(0x556d0a8c35e0 msgr2=0x556d0a8c39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:36:50.988 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:50.974+0000 7fda259e5740 1 -- 172.21.15.90:0/3446408265 shutdown_connections 2024-04-02T14:36:50.989 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:50.974+0000 7fda259e5740 1 -- 172.21.15.90:0/3446408265 wait complete. 2024-04-02T14:36:50.989 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:50.975+0000 7fda259e5740 1 -- start start 2024-04-02T14:36:50.989 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:50.975+0000 7fda259e5740 1 Processor -- start 2024-04-02T14:36:50.989 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:50.976+0000 7fda259e5740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556d0a8c8eb0 0x556d0a8cf4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:50.989 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:50.976+0000 7fda259e5740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556d0a8c97d0 0x556d0a8cfa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:50.989 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:50.976+0000 7fda259e5740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556d0a8d23b0 0x556d0a8cff50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:50.989 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:50.976+0000 7fda259e5740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x556d0a8d51a0 con 0x556d0a8c97d0 2024-04-02T14:36:50.989 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:50.976+0000 7fda259e5740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x556d0a7e9660 con 0x556d0a8c8eb0 2024-04-02T14:36:50.989 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:50.976+0000 7fda259e5740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x556d0a8d5320 con 0x556d0a8d23b0 2024-04-02T14:36:50.989 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:50.976+0000 7fda259e5740 10 client.0 _ll_register_callbacks cb 0x556d0a7e9050 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:36:50.989 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:50.977+0000 7fda1be00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556d0a8c8eb0 0x556d0a8cf4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:50.990 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:50.977+0000 7fda1be00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556d0a8c8eb0 0x556d0a8cf4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:57710/0 (socket says 172.21.15.90:57710) 2024-04-02T14:36:50.990 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:50.977+0000 7fda1be00640 1 -- 172.21.15.90:0/1026663834 learned_addr learned my addr 172.21.15.90:0/1026663834 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:36:50.990 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:50.977+0000 7fda1b400640 1 --2- 172.21.15.90:0/1026663834 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556d0a8c97d0 0x556d0a8cfa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:50.990 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:50.977+0000 7fda20c00640 1 --2- 172.21.15.90:0/1026663834 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556d0a8d23b0 0x556d0a8cff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:50.990 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:50.978+0000 7fda20c00640 1 -- 172.21.15.90:0/1026663834 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556d0a8c8eb0 msgr2=0x556d0a8cf4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:36:50.990 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:50.978+0000 7fda20c00640 1 --2- 172.21.15.90:0/1026663834 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556d0a8c8eb0 0x556d0a8cf4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:50.990 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:50.978+0000 7fda20c00640 1 -- 172.21.15.90:0/1026663834 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556d0a8c97d0 msgr2=0x556d0a8cfa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:36:50.990 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:50.978+0000 7fda20c00640 1 --2- 172.21.15.90:0/1026663834 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556d0a8c97d0 0x556d0a8cfa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:50.990 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:50.978+0000 7fda20c00640 1 -- 172.21.15.90:0/1026663834 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556d0a7f8e80 con 0x556d0a8d23b0 2024-04-02T14:36:50.990 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:50.978+0000 7fda1b400640 1 --2- 172.21.15.90:0/1026663834 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556d0a8c97d0 0x556d0a8cfa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:36:50.990 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:50.978+0000 7fda20c00640 1 --2- 172.21.15.90:0/1026663834 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556d0a8d23b0 0x556d0a8cff50 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fda1402c710 tx=0x7fda14009a10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8aec74f22833fdff server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:36:50.991 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:50.978+0000 7fda05600640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:36:50.991 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:50.978+0000 7fda05600640 1 -- 172.21.15.90:0/1026663834 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda14043020 con 0x556d0a8d23b0 2024-04-02T14:36:50.991 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:50.979+0000 7fda259e5740 1 -- 172.21.15.90:0/1026663834 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fda080ab3a0 con 0x556d0a8d23b0 2024-04-02T14:36:50.991 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:50.979+0000 7fda05600640 1 -- 172.21.15.90:0/1026663834 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7fda140328f0 con 0x556d0a8d23b0 2024-04-02T14:36:50.991 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:50.979+0000 7fda05600640 1 -- 172.21.15.90:0/1026663834 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fda1404e050 con 0x556d0a8d23b0 2024-04-02T14:36:50.991 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:50.979+0000 7fda259e5740 10 client.25981 fetch_fsmap learned FSMap version 27 2024-04-02T14:36:50.992 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:50.979+0000 7fda259e5740 1 -- 172.21.15.90:0/1026663834 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x556d0a542210 con 0x556d0a8d23b0 2024-04-02T14:36:50.992 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:50.979+0000 7fda05600640 1 -- 172.21.15.90:0/1026663834 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fda080ab3a0 con 0x556d0a8d23b0 2024-04-02T14:36:50.992 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:50.979+0000 7fda05600640 1 -- 172.21.15.90:0/1026663834 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fda140046b0 con 0x556d0a8d23b0 2024-04-02T14:36:50.992 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:50.979+0000 7fda259e5740 10 client.25981 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:36:50.992 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:50.979+0000 7fda259e5740 10 client.25981 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-02T14:36:50.992 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:50.979+0000 7fda259e5740 20 client.25981 populate_metadata read hostname 'smithi090' 2024-04-02T14:36:50.993 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:50.979+0000 7fda259e5740 1 --2- 172.21.15.90:0/1026663834 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x556d0a8e2790 0x556d0a8e4b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:50.993 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:50.980+0000 7fda259e5740 4 client.25981 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:36:50.993 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:50.980+0000 7fda259e5740 1 -- 172.21.15.90:0/1026663834 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556d0a7ddcd0 con 0x556d0a8e2790 2024-04-02T14:36:50.993 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:50.980+0000 7fda1be00640 1 --2- 172.21.15.90:0/1026663834 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x556d0a8e2790 0x556d0a8e4b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:50.993 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:50.982+0000 7fda1be00640 1 --2- 172.21.15.90:0/1026663834 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x556d0a8e2790 0x556d0a8e4b80 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=47a64f5d50b0d7f9 server_cookie=cde12dedbd9e050 in_seq=0 out_seq=0 2024-04-02T14:36:50.993 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:50.982+0000 7fda05600640 10 client.25981 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:36:50.993 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:50.983+0000 7fda05600640 1 -- 172.21.15.90:0/1026663834 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x556d0a7ddcd0 con 0x556d0a8e2790 2024-04-02T14:36:50.993 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:50.983+0000 7fda05600640 10 client.25981 handle_command_reply: tid=0 2024-04-02T14:36:50.993 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:50.983+0000 7fda259e5740 1 client.25981 shutdown 2024-04-02T14:36:50.993 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:50.983+0000 7fda259e5740 1 -- 172.21.15.90:0/1026663834 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556d0a8d23b0 msgr2=0x556d0a8cff50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:50.993 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:50.983+0000 7fda259e5740 1 --2- 172.21.15.90:0/1026663834 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556d0a8d23b0 0x556d0a8cff50 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fda1402c710 tx=0x7fda14009a10 comp rx=0 tx=0).stop 2024-04-02T14:36:50.994 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:50.983+0000 7fda259e5740 1 -- 172.21.15.90:0/1026663834 shutdown_connections 2024-04-02T14:36:50.994 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:50.983+0000 7fda259e5740 1 --2- 172.21.15.90:0/1026663834 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x556d0a8e2790 0x556d0a8e4b80 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:50.994 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:50.983+0000 7fda259e5740 1 --2- 172.21.15.90:0/1026663834 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556d0a8d23b0 0x556d0a8cff50 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:50.994 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:50.983+0000 7fda259e5740 1 --2- 172.21.15.90:0/1026663834 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556d0a8c97d0 0x556d0a8cfa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:50.994 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:50.983+0000 7fda259e5740 1 --2- 172.21.15.90:0/1026663834 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556d0a8c8eb0 0x556d0a8cf4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:50.994 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:50.983+0000 7fda259e5740 1 -- 172.21.15.90:0/1026663834 >> 172.21.15.90:0/1026663834 conn(0x556d0a8c35e0 msgr2=0x556d0a8c5330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:36:50.994 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:50.983+0000 7fda05600640 0 client.25981 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:36:50.994 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:50.983+0000 7fda259e5740 1 -- 172.21.15.90:0/1026663834 shutdown_connections 2024-04-02T14:36:50.994 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:50.984+0000 7fda259e5740 1 -- 172.21.15.90:0/1026663834 wait complete. 2024-04-02T14:36:50.994 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:50.984+0000 7fda259e5740 20 client.25981 trim_cache size 0 max 16384 2024-04-02T14:36:51.012 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:51.010+0000 7fda259e5740 1 Processor -- start 2024-04-02T14:36:51.013 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:51.010+0000 7fda259e5740 1 -- start start 2024-04-02T14:36:51.013 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:51.010+0000 7fda259e5740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556d0a8c8eb0 0x556d0a8de060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:51.013 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:51.010+0000 7fda259e5740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556d0a8c97d0 0x556d0a8cfd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:51.013 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:51.010+0000 7fda259e5740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556d0a8d23b0 0x556d0a8ea420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:51.013 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:51.010+0000 7fda259e5740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fda1c1ab640 con 0x556d0a8d23b0 2024-04-02T14:36:51.013 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:51.010+0000 7fda259e5740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fda1c1ad180 con 0x556d0a8c97d0 2024-04-02T14:36:51.013 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:51.010+0000 7fda259e5740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fda1c1abd10 con 0x556d0a8c8eb0 2024-04-02T14:36:51.013 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:51.011+0000 7fda1b400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556d0a8c97d0 0x556d0a8cfd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:51.013 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:51.011+0000 7fda1b400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556d0a8c97d0 0x556d0a8cfd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:57720/0 (socket says 172.21.15.90:57720) 2024-04-02T14:36:51.013 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:51.011+0000 7fda1b400640 1 -- 172.21.15.90:0/1211926961 learned_addr learned my addr 172.21.15.90:0/1211926961 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:36:51.014 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:51.011+0000 7fda1be00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556d0a8c8eb0 0x556d0a8de060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:51.014 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:51.011+0000 7fda1be00640 1 -- 172.21.15.90:0/1211926961 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556d0a8c97d0 msgr2=0x556d0a8cfd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:36:51.015 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:51.011+0000 7fda1be00640 1 --2- 172.21.15.90:0/1211926961 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556d0a8c97d0 0x556d0a8cfd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:51.015 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:51.011+0000 7fda1be00640 1 -- 172.21.15.90:0/1211926961 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556d0a8d23b0 msgr2=0x556d0a8ea420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:36:51.015 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:51.011+0000 7fda1be00640 1 --2- 172.21.15.90:0/1211926961 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556d0a8d23b0 0x556d0a8ea420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:51.015 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:51.011+0000 7fda1be00640 1 -- 172.21.15.90:0/1211926961 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556d0a60e620 con 0x556d0a8c8eb0 2024-04-02T14:36:51.015 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:51.011+0000 7fda1b400640 1 --2- 172.21.15.90:0/1211926961 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556d0a8c97d0 0x556d0a8cfd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:36:51.015 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:51.012+0000 7fda1be00640 1 --2- 172.21.15.90:0/1211926961 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556d0a8c8eb0 0x556d0a8de060 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fda0c000c00 tx=0x7fda0c018000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=da0dea53a6f22871 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:36:51.015 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:51.012+0000 7fda07e00640 1 -- 172.21.15.90:0/1211926961 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda0c011bc0 con 0x556d0a8c8eb0 2024-04-02T14:36:51.015 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:51.012+0000 7fda07e00640 1 -- 172.21.15.90:0/1211926961 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fda0c02c070 con 0x556d0a8c8eb0 2024-04-02T14:36:51.015 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:51.012+0000 7fda07e00640 1 -- 172.21.15.90:0/1211926961 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda0c01f8a0 con 0x556d0a8c8eb0 2024-04-02T14:36:51.016 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:51.013+0000 7fda259e5740 1 -- 172.21.15.90:0/1211926961 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556d0a8c8eb0 msgr2=0x556d0a8de060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:51.016 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:51.013+0000 7fda259e5740 1 --2- 172.21.15.90:0/1211926961 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556d0a8c8eb0 0x556d0a8de060 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fda0c000c00 tx=0x7fda0c018000 comp rx=0 tx=0).stop 2024-04-02T14:36:51.016 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:51.014+0000 7fda259e5740 1 -- 172.21.15.90:0/1211926961 shutdown_connections 2024-04-02T14:36:51.016 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:51.014+0000 7fda259e5740 1 --2- 172.21.15.90:0/1211926961 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556d0a8d23b0 0x556d0a8ea420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:51.016 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:51.014+0000 7fda259e5740 1 --2- 172.21.15.90:0/1211926961 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556d0a8c97d0 0x556d0a8cfd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:51.016 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:51.014+0000 7fda259e5740 1 --2- 172.21.15.90:0/1211926961 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556d0a8c8eb0 0x556d0a8de060 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:51.017 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:51.014+0000 7fda259e5740 1 -- 172.21.15.90:0/1211926961 >> 172.21.15.90:0/1211926961 conn(0x556d0a8c35e0 msgr2=0x556d0a8c2640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:36:51.017 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:51.014+0000 7fda259e5740 1 -- 172.21.15.90:0/1211926961 shutdown_connections 2024-04-02T14:36:51.017 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:51.014+0000 7fda259e5740 1 -- 172.21.15.90:0/1211926961 wait complete. 2024-04-02T14:36:51.018 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:51.015+0000 7fda259e5740 1 -- start start 2024-04-02T14:36:51.018 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:51.015+0000 7fda259e5740 1 Processor -- start 2024-04-02T14:36:51.022 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:51.016+0000 7fda259e5740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556d0a8c8eb0 0x556d0a8da570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:51.022 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:51.016+0000 7fda259e5740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556d0a8c97d0 0x556d0a8daab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:51.022 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:51.016+0000 7fda259e5740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556d0a8d23b0 0x556d0a8cbd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:51.022 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:51.016+0000 7fda259e5740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fda1c1acab0 con 0x556d0a8c8eb0 2024-04-02T14:36:51.022 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:51.016+0000 7fda259e5740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fda1c1ac3e0 con 0x556d0a8c97d0 2024-04-02T14:36:51.023 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:51.016+0000 7fda259e5740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fda1c071f20 con 0x556d0a8d23b0 2024-04-02T14:36:51.023 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:51.017+0000 7fda20c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556d0a8d23b0 0x556d0a8cbd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:51.023 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:51.017+0000 7fda1b400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556d0a8c97d0 0x556d0a8daab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:51.023 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:51.017+0000 7fda1b400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556d0a8c97d0 0x556d0a8daab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:57732/0 (socket says 172.21.15.90:57732) 2024-04-02T14:36:51.023 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:51.017+0000 7fda1b400640 1 -- 172.21.15.90:0/2769797215 learned_addr learned my addr 172.21.15.90:0/2769797215 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:36:51.023 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:51.017+0000 7fda20c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556d0a8d23b0 0x556d0a8cbd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:41582/0 (socket says 172.21.15.90:41582) 2024-04-02T14:36:51.023 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:51.017+0000 7fda1be00640 1 --2- 172.21.15.90:0/2769797215 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556d0a8c8eb0 0x556d0a8da570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:51.023 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:51.017+0000 7fda1b400640 1 -- 172.21.15.90:0/2769797215 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556d0a8d23b0 msgr2=0x556d0a8cbd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:36:51.023 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:51.017+0000 7fda1b400640 1 --2- 172.21.15.90:0/2769797215 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556d0a8d23b0 0x556d0a8cbd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:51.024 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:51.017+0000 7fda1b400640 1 -- 172.21.15.90:0/2769797215 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556d0a8c8eb0 msgr2=0x556d0a8da570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:36:51.024 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:51.017+0000 7fda1b400640 1 --2- 172.21.15.90:0/2769797215 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556d0a8c8eb0 0x556d0a8da570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:51.024 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:51.017+0000 7fda1b400640 1 -- 172.21.15.90:0/2769797215 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556d0a8cc550 con 0x556d0a8c97d0 2024-04-02T14:36:51.024 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:51.017+0000 7fda259e5740 10 client.0 _ll_register_callbacks cb 0x7fda14032360 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:36:51.024 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:51.017+0000 7fda259e5740 1 -- 172.21.15.90:0/2769797215 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fda0c02c070 con 0x556d0a8c97d0 2024-04-02T14:36:51.024 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:51.017+0000 7fda1b400640 1 --2- 172.21.15.90:0/2769797215 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556d0a8c97d0 0x556d0a8daab0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fda080dee50 tx=0x7fda080ad220 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bc89c33e7ebe3a26 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:36:51.024 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:51.017+0000 7fda05600640 10 client.26066 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:36:51.024 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:51.018+0000 7fda05600640 1 -- 172.21.15.90:0/2769797215 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda080b10b0 con 0x556d0a8c97d0 2024-04-02T14:36:51.024 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:51.018+0000 7fda05600640 1 -- 172.21.15.90:0/2769797215 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7fda080c1b40 con 0x556d0a8c97d0 2024-04-02T14:36:51.024 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:51.018+0000 7fda05600640 1 -- 172.21.15.90:0/2769797215 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fda080b1250 con 0x556d0a8c97d0 2024-04-02T14:36:51.024 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:51.018+0000 7fda259e5740 10 client.26066 fetch_fsmap learned FSMap version 27 2024-04-02T14:36:51.024 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:51.018+0000 7fda259e5740 1 -- 172.21.15.90:0/2769797215 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({fsmap=27}) -- 0x556d0a948760 con 0x556d0a8c97d0 2024-04-02T14:36:51.025 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:51.018+0000 7fda05600640 1 -- 172.21.15.90:0/2769797215 <== mon.2 v2:172.21.15.92:3300/0 4 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fda080ac4f0 con 0x556d0a8c97d0 2024-04-02T14:36:51.025 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:51.019+0000 7fda259e5740 10 client.26066 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:36:51.025 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:51.019+0000 7fda259e5740 10 client.26066 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-02T14:36:51.025 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:51.019+0000 7fda259e5740 20 client.26066 populate_metadata read hostname 'smithi090' 2024-04-02T14:36:51.025 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:51.019+0000 7fda259e5740 1 --2- 172.21.15.90:0/2769797215 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x556d0a948e50 0x556d0a94b270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:36:51.025 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:51.019+0000 7fda259e5740 4 client.26066 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-02T14:36:51.025 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:51.019+0000 7fda259e5740 1 -- 172.21.15.90:0/2769797215 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "flush journal"}) -- 0x556d0a83b2b0 con 0x556d0a948e50 2024-04-02T14:36:51.025 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:51.019+0000 7fda1be00640 1 --2- 172.21.15.90:0/2769797215 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x556d0a948e50 0x556d0a94b270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:36:51.025 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:51.020+0000 7fda1be00640 1 --2- 172.21.15.90:0/2769797215 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x556d0a948e50 0x556d0a94b270 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=39a1c78cf1e1942c server_cookie=f469ae72bad51316 in_seq=0 out_seq=0 2024-04-02T14:36:51.025 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:51.020+0000 7fda05600640 10 client.26066 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:36:51.043 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:51.041+0000 7fda05600640 1 -- 172.21.15.90:0/2769797215 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x556d0a83b2b0 con 0x556d0a948e50 2024-04-02T14:36:51.043 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:51.041+0000 7fda05600640 10 client.26066 handle_command_reply: tid=0 2024-04-02T14:36:51.043 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:51.041+0000 7fda259e5740 1 client.26066 shutdown 2024-04-02T14:36:51.045 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:51.043+0000 7fda259e5740 1 -- 172.21.15.90:0/2769797215 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556d0a8c97d0 msgr2=0x556d0a8daab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:51.045 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:51.043+0000 7fda259e5740 1 --2- 172.21.15.90:0/2769797215 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556d0a8c97d0 0x556d0a8daab0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fda080dee50 tx=0x7fda080ad220 comp rx=0 tx=0).stop 2024-04-02T14:36:51.045 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:51.043+0000 7fda259e5740 1 -- 172.21.15.90:0/2769797215 shutdown_connections 2024-04-02T14:36:51.046 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:51.043+0000 7fda259e5740 1 --2- 172.21.15.90:0/2769797215 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x556d0a948e50 0x556d0a94b270 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:51.046 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:51.043+0000 7fda259e5740 1 --2- 172.21.15.90:0/2769797215 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556d0a8d23b0 0x556d0a8cbd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:51.046 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:51.043+0000 7fda259e5740 1 --2- 172.21.15.90:0/2769797215 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556d0a8c97d0 0x556d0a8daab0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:51.046 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:51.043+0000 7fda259e5740 1 --2- 172.21.15.90:0/2769797215 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556d0a8c8eb0 0x556d0a8da570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:51.046 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:51.043+0000 7fda259e5740 1 -- 172.21.15.90:0/2769797215 >> 172.21.15.90:0/2769797215 conn(0x556d0a8c35e0 msgr2=0x556d0a8c1db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:36:51.046 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:51.043+0000 7fda05600640 0 client.26066 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:36:51.046 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:51.043+0000 7fda259e5740 1 -- 172.21.15.90:0/2769797215 shutdown_connections 2024-04-02T14:36:51.046 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:51.043+0000 7fda259e5740 1 -- 172.21.15.90:0/2769797215 wait complete. 2024-04-02T14:36:51.046 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:51.043+0000 7fda259e5740 20 client.26066 trim_cache size 0 max 16384 2024-04-02T14:36:51.047 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:{ 2024-04-02T14:36:51.047 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "message": "", 2024-04-02T14:36:51.047 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "return_code": 0 2024-04-02T14:36:51.047 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:} 2024-04-02T14:36:51.047 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:51.045+0000 7fda22c00640 1 -- 172.21.15.90:0/4007083561 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fda000a3e40 msgr2=0x7fda000a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:51.048 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:51.045+0000 7fda22c00640 1 --2- 172.21.15.90:0/4007083561 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fda000a3e40 0x7fda000a6300 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fda0c00bbe0 tx=0x7fda0c00b390 comp rx=0 tx=0).stop 2024-04-02T14:36:51.048 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:51.045+0000 7fda22c00640 1 -- 172.21.15.90:0/4007083561 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fda1c076950 msgr2=0x7fda1c1a4bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:36:51.048 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:51.045+0000 7fda22c00640 1 --2- 172.21.15.90:0/4007083561 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fda1c076950 0x7fda1c1a4bd0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fda0800d950 tx=0x7fda0800de10 comp rx=0 tx=0).stop 2024-04-02T14:36:51.048 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:51.045+0000 7fda22c00640 1 -- 172.21.15.90:0/4007083561 shutdown_connections 2024-04-02T14:36:51.048 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:51.045+0000 7fda22c00640 1 --2- 172.21.15.90:0/4007083561 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fda000a3e40 0x7fda000a6300 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:51.048 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:51.045+0000 7fda22c00640 1 --2- 172.21.15.90:0/4007083561 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fda1c112950 0x7fda1c19e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:51.048 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:51.045+0000 7fda22c00640 1 --2- 172.21.15.90:0/4007083561 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fda1c076950 0x7fda1c1a4bd0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:51.048 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:51.045+0000 7fda22c00640 1 --2- 172.21.15.90:0/4007083561 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fda1c076030 0x7fda1c1a4690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:36:51.048 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:51.045+0000 7fda22c00640 1 -- 172.21.15.90:0/4007083561 >> 172.21.15.90:0/4007083561 conn(0x7fda1c0ff780 msgr2=0x7fda1c113f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:36:51.049 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:51.046+0000 7fda22c00640 1 -- 172.21.15.90:0/4007083561 shutdown_connections 2024-04-02T14:36:51.049 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:36:51.046+0000 7fda22c00640 1 -- 172.21.15.90:0/4007083561 wait complete. 2024-04-02T14:36:51.831 INFO:teuthology.orchestra.run.smithi090.stderr:progress: 80.0 s, 3792.5 tps, lat 26.309 ms stddev 27.303 2024-04-02T14:36:51.860 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:36:51 smithi092 ceph-mon[36497]: pgmap v62: 97 pgs: 97 active+clean; 1.2 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 9.3 MiB/s wr, 887 op/s 2024-04-02T14:36:51.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:36:51 smithi177 ceph-mon[36431]: pgmap v62: 97 pgs: 97 active+clean; 1.2 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 9.3 MiB/s wr, 887 op/s 2024-04-02T14:36:51.938 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:36:51 smithi090 ceph-mon[31674]: pgmap v62: 97 pgs: 97 active+clean; 1.2 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 9.3 MiB/s wr, 887 op/s 2024-04-02T14:36:52.860 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:36:52 smithi092 ceph-mon[36497]: scrub summary: idle 2024-04-02T14:36:52.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:36:52 smithi177 ceph-mon[36431]: scrub summary: idle 2024-04-02T14:36:52.939 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:36:52 smithi090 ceph-mon[31674]: scrub summary: idle 2024-04-02T14:36:53.837 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:36:53 smithi177 ceph-mon[36431]: pgmap v63: 97 pgs: 97 active+clean; 1.2 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 8.2 MiB/s wr, 781 op/s 2024-04-02T14:36:53.860 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:36:53 smithi092 ceph-mon[36497]: pgmap v63: 97 pgs: 97 active+clean; 1.2 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 8.2 MiB/s wr, 781 op/s 2024-04-02T14:36:53.938 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:36:53 smithi090 ceph-mon[31674]: pgmap v63: 97 pgs: 97 active+clean; 1.2 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 8.2 MiB/s wr, 781 op/s 2024-04-02T14:36:55.862 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:36:55 smithi092 ceph-mon[36497]: pgmap v64: 97 pgs: 97 active+clean; 1.2 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 13 MiB/s wr, 1.12k op/s 2024-04-02T14:36:55.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:36:55 smithi177 ceph-mon[36431]: pgmap v64: 97 pgs: 97 active+clean; 1.2 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 13 MiB/s wr, 1.12k op/s 2024-04-02T14:36:55.939 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:36:55 smithi090 ceph-mon[31674]: pgmap v64: 97 pgs: 97 active+clean; 1.2 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 13 MiB/s wr, 1.12k op/s 2024-04-02T14:36:56.831 INFO:teuthology.orchestra.run.smithi090.stderr:progress: 85.0 s, 3948.7 tps, lat 25.289 ms stddev 27.161 2024-04-02T14:36:57.861 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:36:57 smithi092 ceph-mon[36497]: pgmap v65: 97 pgs: 97 active+clean; 1.2 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 8.7 MiB/s wr, 782 op/s 2024-04-02T14:36:57.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:36:57 smithi177 ceph-mon[36431]: pgmap v65: 97 pgs: 97 active+clean; 1.2 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 8.7 MiB/s wr, 782 op/s 2024-04-02T14:36:57.938 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:36:57 smithi090 ceph-mon[31674]: pgmap v65: 97 pgs: 97 active+clean; 1.2 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 8.7 MiB/s wr, 782 op/s 2024-04-02T14:36:59.860 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:36:59 smithi092 ceph-mon[36497]: pgmap v66: 97 pgs: 97 active+clean; 1.3 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 13 MiB/s wr, 1.08k op/s 2024-04-02T14:36:59.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:36:59 smithi177 ceph-mon[36431]: pgmap v66: 97 pgs: 97 active+clean; 1.3 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 13 MiB/s wr, 1.08k op/s 2024-04-02T14:36:59.938 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:36:59 smithi090 ceph-mon[31674]: pgmap v66: 97 pgs: 97 active+clean; 1.3 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 13 MiB/s wr, 1.08k op/s 2024-04-02T14:37:01.832 INFO:teuthology.orchestra.run.smithi090.stderr:progress: 90.0 s, 2496.9 tps, lat 37.240 ms stddev 78.103 2024-04-02T14:37:01.861 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:37:01 smithi092 ceph-mon[36497]: pgmap v67: 97 pgs: 97 active+clean; 1.3 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 11 MiB/s wr, 767 op/s 2024-04-02T14:37:01.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:37:01 smithi177 ceph-mon[36431]: pgmap v67: 97 pgs: 97 active+clean; 1.3 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 11 MiB/s wr, 767 op/s 2024-04-02T14:37:01.938 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:37:01 smithi090 ceph-mon[31674]: pgmap v67: 97 pgs: 97 active+clean; 1.3 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 11 MiB/s wr, 767 op/s 2024-04-02T14:37:03.861 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:37:03 smithi092 ceph-mon[36497]: pgmap v68: 97 pgs: 97 active+clean; 1.3 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 10 MiB/s wr, 686 op/s 2024-04-02T14:37:03.861 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:37:03 smithi092 ceph-mon[36497]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T14:37:03.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:37:03 smithi177 ceph-mon[36431]: pgmap v68: 97 pgs: 97 active+clean; 1.3 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 10 MiB/s wr, 686 op/s 2024-04-02T14:37:03.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:37:03 smithi177 ceph-mon[36431]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T14:37:03.938 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:37:03 smithi090 ceph-mon[31674]: pgmap v68: 97 pgs: 97 active+clean; 1.3 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 10 MiB/s wr, 686 op/s 2024-04-02T14:37:03.938 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:37:03 smithi090 ceph-mon[31674]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T14:37:04.212 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.210+0000 7f0177200640 1 -- 172.21.15.90:0/552842929 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0170100c50 msgr2=0x7f01701010d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:04.212 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.210+0000 7f0177200640 1 --2- 172.21.15.90:0/552842929 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0170100c50 0x7f01701010d0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f015c009930 tx=0x7f015c02bc20 comp rx=0 tx=0).stop 2024-04-02T14:37:04.212 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.210+0000 7f0177200640 1 -- 172.21.15.90:0/552842929 shutdown_connections 2024-04-02T14:37:04.212 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.210+0000 7f0177200640 1 --2- 172.21.15.90:0/552842929 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0170101610 0x7f017010d830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:04.212 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.210+0000 7f0177200640 1 --2- 172.21.15.90:0/552842929 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0170100c50 0x7f01701010d0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:04.212 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.210+0000 7f0177200640 1 --2- 172.21.15.90:0/552842929 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f01701059a0 0x7f0170105d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:04.212 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.210+0000 7f0177200640 1 -- 172.21.15.90:0/552842929 >> 172.21.15.90:0/552842929 conn(0x7f0170077850 msgr2=0x7f0170077c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:37:04.212 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.211+0000 7f0177200640 1 -- 172.21.15.90:0/552842929 shutdown_connections 2024-04-02T14:37:04.213 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.211+0000 7f0177200640 1 -- 172.21.15.90:0/552842929 wait complete. 2024-04-02T14:37:04.213 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.212+0000 7f0177200640 1 Processor -- start 2024-04-02T14:37:04.213 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.212+0000 7f0177200640 1 -- start start 2024-04-02T14:37:04.214 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.213+0000 7f0177200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0170100c50 0x7f017019a5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:04.214 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.213+0000 7f0177200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0170101610 0x7f017019ab00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:04.214 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.213+0000 7f0177200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f01701059a0 0x7f017019ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:04.214 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.213+0000 7f0177200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f0170110f10 con 0x7f0170101610 2024-04-02T14:37:04.214 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.213+0000 7f0177200640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f0170110d90 con 0x7f0170100c50 2024-04-02T14:37:04.214 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.213+0000 7f0177200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f0170111090 con 0x7f01701059a0 2024-04-02T14:37:04.216 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.213+0000 7f016fe00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0170100c50 0x7f017019a5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:04.216 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.213+0000 7f016fe00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0170100c50 0x7f017019a5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:53556/0 (socket says 172.21.15.90:53556) 2024-04-02T14:37:04.216 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.213+0000 7f016fe00640 1 -- 172.21.15.90:0/4201893099 learned_addr learned my addr 172.21.15.90:0/4201893099 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:37:04.216 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.213+0000 7f016f400640 1 --2- 172.21.15.90:0/4201893099 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0170101610 0x7f017019ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:04.216 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.213+0000 7f0175200640 1 --2- 172.21.15.90:0/4201893099 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f01701059a0 0x7f017019ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:04.216 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.214+0000 7f016fe00640 1 -- 172.21.15.90:0/4201893099 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f01701059a0 msgr2=0x7f017019ee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:04.216 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.214+0000 7f016fe00640 1 --2- 172.21.15.90:0/4201893099 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f01701059a0 0x7f017019ee80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:04.216 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.214+0000 7f016fe00640 1 -- 172.21.15.90:0/4201893099 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0170101610 msgr2=0x7f017019ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:04.217 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.214+0000 7f016fe00640 1 --2- 172.21.15.90:0/4201893099 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0170101610 0x7f017019ab00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:04.217 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.214+0000 7f016fe00640 1 -- 172.21.15.90:0/4201893099 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f017019f5b0 con 0x7f0170100c50 2024-04-02T14:37:04.217 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.214+0000 7f016f400640 1 --2- 172.21.15.90:0/4201893099 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0170101610 0x7f017019ab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:37:04.217 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.214+0000 7f016fe00640 1 --2- 172.21.15.90:0/4201893099 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0170100c50 0x7f017019a5c0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f016000e9f0 tx=0x7f016000edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:37:04.217 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.214+0000 7f016cc00640 1 -- 172.21.15.90:0/4201893099 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f016000cd80 con 0x7f0170100c50 2024-04-02T14:37:04.217 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.215+0000 7f016cc00640 1 -- 172.21.15.90:0/4201893099 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0160005500 con 0x7f0170100c50 2024-04-02T14:37:04.217 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.215+0000 7f0177200640 1 -- 172.21.15.90:0/4201893099 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f017019f8a0 con 0x7f0170100c50 2024-04-02T14:37:04.217 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.215+0000 7f016cc00640 1 -- 172.21.15.90:0/4201893099 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0160010690 con 0x7f0170100c50 2024-04-02T14:37:04.217 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.215+0000 7f0177200640 1 -- 172.21.15.90:0/4201893099 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01701a7130 con 0x7f0170100c50 2024-04-02T14:37:04.219 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.218+0000 7f016cc00640 1 -- 172.21.15.90:0/4201893099 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f0160004f60 con 0x7f0170100c50 2024-04-02T14:37:04.221 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.220+0000 7f016cc00640 1 --2- 172.21.15.90:0/4201893099 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f01540a3e40 0x7f01540a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:04.221 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.220+0000 7f0179fc0740 1 Processor -- start 2024-04-02T14:37:04.221 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.221+0000 7f016cc00640 1 -- 172.21.15.90:0/4201893099 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f01600d0d00 con 0x7f0170100c50 2024-04-02T14:37:04.222 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.221+0000 7f0179fc0740 1 -- start start 2024-04-02T14:37:04.223 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.221+0000 7f016f400640 1 --2- 172.21.15.90:0/4201893099 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f01540a3e40 0x7f01540a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:04.223 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.221+0000 7f0179fc0740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b1c2db4eb0 0x55b1c2db5290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:04.223 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.221+0000 7f0179fc0740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b1c2db57d0 0x55b1c2dbde70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:04.224 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.221+0000 7f0179fc0740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b1c2dbe3b0 0x55b1c2dc07a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:04.224 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.221+0000 7f0179fc0740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55b1c2d19d70 con 0x55b1c2db57d0 2024-04-02T14:37:04.224 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.221+0000 7f0179fc0740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55b1c2cc22a0 con 0x55b1c2dbe3b0 2024-04-02T14:37:04.224 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.221+0000 7f0179fc0740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55b1c2d15d20 con 0x55b1c2db4eb0 2024-04-02T14:37:04.224 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.221+0000 7f016fe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b1c2db4eb0 0x55b1c2db5290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:04.224 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.221+0000 7f0175200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b1c2dbe3b0 0x55b1c2dc07a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:04.224 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.221+0000 7f016fe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b1c2db4eb0 0x55b1c2db5290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:58766/0 (socket says 172.21.15.90:58766) 2024-04-02T14:37:04.224 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.221+0000 7f016fe00640 1 -- 172.21.15.90:0/2543880978 learned_addr learned my addr 172.21.15.90:0/2543880978 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:37:04.224 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.221+0000 7f016f400640 1 --2- 172.21.15.90:0/2543880978 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b1c2db57d0 0x55b1c2dbde70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:04.224 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.222+0000 7f016fe00640 1 -- 172.21.15.90:0/2543880978 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b1c2dbe3b0 msgr2=0x55b1c2dc07a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:37:04.224 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.222+0000 7f016fe00640 1 --2- 172.21.15.90:0/2543880978 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b1c2dbe3b0 0x55b1c2dc07a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:04.224 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.222+0000 7f016fe00640 1 -- 172.21.15.90:0/2543880978 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b1c2db57d0 msgr2=0x55b1c2dbde70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:37:04.225 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.222+0000 7f016fe00640 1 --2- 172.21.15.90:0/2543880978 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b1c2db57d0 0x55b1c2dbde70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:04.225 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.222+0000 7f016f400640 1 --2- 172.21.15.90:0/4201893099 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f01540a3e40 0x7f01540a6300 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f015c009900 tx=0x7f015c02c6a0 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:37:04.225 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.222+0000 7f016fe00640 1 -- 172.21.15.90:0/2543880978 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b1c2cd6840 con 0x55b1c2db4eb0 2024-04-02T14:37:04.225 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.222+0000 7f0175200640 1 --2- 172.21.15.90:0/2543880978 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b1c2dbe3b0 0x55b1c2dc07a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:37:04.225 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.222+0000 7f016fe00640 1 --2- 172.21.15.90:0/2543880978 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b1c2db4eb0 0x55b1c2db5290 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f0160099620 tx=0x7f016009c5b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=59d65b8ea8d20716 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:37:04.225 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.222+0000 7f0165600640 1 -- 172.21.15.90:0/2543880978 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01600a1000 con 0x55b1c2db4eb0 2024-04-02T14:37:04.225 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.222+0000 7f0165600640 1 -- 172.21.15.90:0/2543880978 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f01600a58c0 con 0x55b1c2db4eb0 2024-04-02T14:37:04.226 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.222+0000 7f0165600640 1 -- 172.21.15.90:0/2543880978 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01600046c0 con 0x55b1c2db4eb0 2024-04-02T14:37:04.227 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.223+0000 7f0179fc0740 1 -- 172.21.15.90:0/2543880978 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b1c2db4eb0 msgr2=0x55b1c2db5290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:04.227 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.223+0000 7f0179fc0740 1 --2- 172.21.15.90:0/2543880978 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b1c2db4eb0 0x55b1c2db5290 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f0160099620 tx=0x7f016009c5b0 comp rx=0 tx=0).stop 2024-04-02T14:37:04.227 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.224+0000 7f0179fc0740 1 -- 172.21.15.90:0/2543880978 shutdown_connections 2024-04-02T14:37:04.227 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.224+0000 7f0179fc0740 1 --2- 172.21.15.90:0/2543880978 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b1c2dbe3b0 0x55b1c2dc07a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:04.227 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.224+0000 7f0179fc0740 1 --2- 172.21.15.90:0/2543880978 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b1c2db57d0 0x55b1c2dbde70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:04.227 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.224+0000 7f0179fc0740 1 --2- 172.21.15.90:0/2543880978 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b1c2db4eb0 0x55b1c2db5290 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:04.228 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.224+0000 7f0179fc0740 1 -- 172.21.15.90:0/2543880978 >> 172.21.15.90:0/2543880978 conn(0x55b1c2daf5e0 msgr2=0x55b1c2daf9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:37:04.228 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.224+0000 7f0179fc0740 1 -- 172.21.15.90:0/2543880978 shutdown_connections 2024-04-02T14:37:04.228 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.224+0000 7f0179fc0740 1 -- 172.21.15.90:0/2543880978 wait complete. 2024-04-02T14:37:04.228 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.225+0000 7f0179fc0740 1 -- start start 2024-04-02T14:37:04.228 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.225+0000 7f0179fc0740 1 Processor -- start 2024-04-02T14:37:04.228 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.226+0000 7f0179fc0740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b1c2db57d0 0x55b1c2dbb530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:04.228 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.226+0000 7f0179fc0740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b1c2dbe3b0 0x55b1c2dbba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:04.228 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.226+0000 7f0179fc0740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b1c2dbbfb0 0x55b1c2dca640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:04.228 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.226+0000 7f0179fc0740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55b1c2dc11a0 con 0x55b1c2dbbfb0 2024-04-02T14:37:04.228 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.226+0000 7f0179fc0740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55b1c2cd5660 con 0x55b1c2db57d0 2024-04-02T14:37:04.228 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.226+0000 7f0179fc0740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55b1c2dc1320 con 0x55b1c2dbe3b0 2024-04-02T14:37:04.228 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.226+0000 7f0179fc0740 10 client.0 _ll_register_callbacks cb 0x55b1c2cd5050 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:37:04.228 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.226+0000 7f016fe00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b1c2db57d0 0x55b1c2dbb530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:04.228 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.226+0000 7f0175200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b1c2dbbfb0 0x55b1c2dca640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:04.228 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.226+0000 7f016f400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b1c2dbe3b0 0x55b1c2dbba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:04.229 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.226+0000 7f016fe00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b1c2db57d0 0x55b1c2dbb530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:53562/0 (socket says 172.21.15.90:53562) 2024-04-02T14:37:04.229 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.226+0000 7f016fe00640 1 -- 172.21.15.90:0/3536297837 learned_addr learned my addr 172.21.15.90:0/3536297837 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:37:04.229 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.226+0000 7f016f400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b1c2dbe3b0 0x55b1c2dbba70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:58770/0 (socket says 172.21.15.90:58770) 2024-04-02T14:37:04.230 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.227+0000 7f016fe00640 1 -- 172.21.15.90:0/3536297837 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b1c2dbe3b0 msgr2=0x55b1c2dbba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:37:04.230 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.227+0000 7f016fe00640 1 --2- 172.21.15.90:0/3536297837 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b1c2dbe3b0 0x55b1c2dbba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:04.230 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.227+0000 7f016fe00640 1 -- 172.21.15.90:0/3536297837 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b1c2dbbfb0 msgr2=0x55b1c2dca640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:37:04.230 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.227+0000 7f016fe00640 1 --2- 172.21.15.90:0/3536297837 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b1c2dbbfb0 0x55b1c2dca640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:04.230 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.227+0000 7f016fe00640 1 -- 172.21.15.90:0/3536297837 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b1c2ce4e80 con 0x55b1c2db57d0 2024-04-02T14:37:04.231 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.227+0000 7f016f400640 1 --2- 172.21.15.90:0/3536297837 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b1c2dbe3b0 0x55b1c2dbba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:37:04.231 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.227+0000 7f016fe00640 1 --2- 172.21.15.90:0/3536297837 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b1c2db57d0 0x55b1c2dbb530 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f01600990b0 tx=0x7f016009cdd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4e82b6e715626ae7 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:37:04.231 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.227+0000 7f015a000640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:37:04.231 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.227+0000 7f015a000640 1 -- 172.21.15.90:0/3536297837 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01600a1000 con 0x55b1c2db57d0 2024-04-02T14:37:04.231 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.228+0000 7f015a000640 1 -- 172.21.15.90:0/3536297837 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f01600bd0b0 con 0x55b1c2db57d0 2024-04-02T14:37:04.231 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.228+0000 7f0179fc0740 1 -- 172.21.15.90:0/3536297837 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f01600a58c0 con 0x55b1c2db57d0 2024-04-02T14:37:04.231 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.229+0000 7f015a000640 1 -- 172.21.15.90:0/3536297837 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f01600bd2b0 con 0x55b1c2db57d0 2024-04-02T14:37:04.231 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.229+0000 7f0179fc0740 10 client.26078 fetch_fsmap learned FSMap version 27 2024-04-02T14:37:04.231 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.229+0000 7f0179fc0740 1 -- 172.21.15.90:0/3536297837 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55b1c2a2e210 con 0x55b1c2db57d0 2024-04-02T14:37:04.231 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.229+0000 7f015a000640 1 -- 172.21.15.90:0/3536297837 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f01600a58c0 con 0x55b1c2db57d0 2024-04-02T14:37:04.232 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.229+0000 7f015a000640 1 -- 172.21.15.90:0/3536297837 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f01600ceb50 con 0x55b1c2db57d0 2024-04-02T14:37:04.232 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.229+0000 7f0179fc0740 10 client.26078 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:37:04.232 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.229+0000 7f0179fc0740 10 client.26078 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-02T14:37:04.232 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.229+0000 7f0179fc0740 20 client.26078 populate_metadata read hostname 'smithi090' 2024-04-02T14:37:04.232 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.229+0000 7f0179fc0740 1 --2- 172.21.15.90:0/3536297837 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55b1c2dcebb0 0x55b1c2dd0fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:04.233 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.229+0000 7f0179fc0740 4 client.26078 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:37:04.233 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.229+0000 7f0179fc0740 1 -- 172.21.15.90:0/3536297837 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b1c2cc9cd0 con 0x55b1c2dcebb0 2024-04-02T14:37:04.233 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.230+0000 7f016f400640 1 --2- 172.21.15.90:0/3536297837 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55b1c2dcebb0 0x55b1c2dd0fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:04.233 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.231+0000 7f016f400640 1 --2- 172.21.15.90:0/3536297837 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55b1c2dcebb0 0x55b1c2dd0fa0 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=aed62ccda2d8dc6e server_cookie=f9f56238d236270 in_seq=0 out_seq=0 2024-04-02T14:37:04.233 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.231+0000 7f015a000640 10 client.26078 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:37:04.234 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.232+0000 7f015a000640 1 -- 172.21.15.90:0/3536297837 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b1c2cc9cd0 con 0x55b1c2dcebb0 2024-04-02T14:37:04.234 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.232+0000 7f015a000640 10 client.26078 handle_command_reply: tid=0 2024-04-02T14:37:04.234 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.232+0000 7f0179fc0740 1 client.26078 shutdown 2024-04-02T14:37:04.234 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.232+0000 7f0179fc0740 1 -- 172.21.15.90:0/3536297837 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b1c2db57d0 msgr2=0x55b1c2dbb530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:04.234 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.232+0000 7f0179fc0740 1 --2- 172.21.15.90:0/3536297837 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b1c2db57d0 0x55b1c2dbb530 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f01600990b0 tx=0x7f016009cdd0 comp rx=0 tx=0).stop 2024-04-02T14:37:04.234 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.232+0000 7f0179fc0740 1 -- 172.21.15.90:0/3536297837 shutdown_connections 2024-04-02T14:37:04.234 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.232+0000 7f0179fc0740 1 --2- 172.21.15.90:0/3536297837 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55b1c2dcebb0 0x55b1c2dd0fa0 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:04.234 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.232+0000 7f0179fc0740 1 --2- 172.21.15.90:0/3536297837 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b1c2dbbfb0 0x55b1c2dca640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:04.235 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.232+0000 7f0179fc0740 1 --2- 172.21.15.90:0/3536297837 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b1c2dbe3b0 0x55b1c2dbba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:04.235 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.232+0000 7f0179fc0740 1 --2- 172.21.15.90:0/3536297837 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b1c2db57d0 0x55b1c2dbb530 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:04.235 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.232+0000 7f015a000640 0 client.26078 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:37:04.235 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.232+0000 7f0179fc0740 1 -- 172.21.15.90:0/3536297837 >> 172.21.15.90:0/3536297837 conn(0x55b1c2daf5e0 msgr2=0x55b1c2db1330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:37:04.235 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.233+0000 7f0179fc0740 1 -- 172.21.15.90:0/3536297837 shutdown_connections 2024-04-02T14:37:04.235 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.233+0000 7f0179fc0740 1 -- 172.21.15.90:0/3536297837 wait complete. 2024-04-02T14:37:04.235 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.233+0000 7f0179fc0740 20 client.26078 trim_cache size 0 max 16384 2024-04-02T14:37:04.255 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.253+0000 7f0179fc0740 1 Processor -- start 2024-04-02T14:37:04.255 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.253+0000 7f0179fc0740 1 -- start start 2024-04-02T14:37:04.255 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.253+0000 7f0179fc0740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b1c2db57d0 0x55b1c2dca5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:04.255 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.253+0000 7f0179fc0740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b1c2dbbfb0 0x55b1c2db0ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:04.255 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.253+0000 7f0179fc0740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b1c2dbe3b0 0x55b1c2db1530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:04.255 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.253+0000 7f0179fc0740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f01701af9b0 con 0x55b1c2dbe3b0 2024-04-02T14:37:04.256 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.253+0000 7f0179fc0740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f01701b14f0 con 0x55b1c2db57d0 2024-04-02T14:37:04.256 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.253+0000 7f0179fc0740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f01701b0080 con 0x55b1c2dbbfb0 2024-04-02T14:37:04.256 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.254+0000 7f016f400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b1c2dbbfb0 0x55b1c2db0ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:04.256 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.254+0000 7f016fe00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b1c2db57d0 0x55b1c2dca5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:04.256 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.254+0000 7f016f400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b1c2dbbfb0 0x55b1c2db0ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:58778/0 (socket says 172.21.15.90:58778) 2024-04-02T14:37:04.256 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.254+0000 7f016fe00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b1c2db57d0 0x55b1c2dca5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:53568/0 (socket says 172.21.15.90:53568) 2024-04-02T14:37:04.256 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.254+0000 7f016f400640 1 -- 172.21.15.90:0/364866631 learned_addr learned my addr 172.21.15.90:0/364866631 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:37:04.256 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.254+0000 7f0175200640 1 --2- 172.21.15.90:0/364866631 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b1c2dbe3b0 0x55b1c2db1530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:04.256 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.254+0000 7f016f400640 1 -- 172.21.15.90:0/364866631 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b1c2db57d0 msgr2=0x55b1c2dca5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:37:04.256 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.254+0000 7f016f400640 1 --2- 172.21.15.90:0/364866631 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b1c2db57d0 0x55b1c2dca5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:04.256 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.254+0000 7f016f400640 1 -- 172.21.15.90:0/364866631 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b1c2dbe3b0 msgr2=0x55b1c2db1530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:37:04.256 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.254+0000 7f016f400640 1 --2- 172.21.15.90:0/364866631 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b1c2dbe3b0 0x55b1c2db1530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:04.257 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.254+0000 7f016f400640 1 -- 172.21.15.90:0/364866631 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b1c2afa620 con 0x55b1c2dbbfb0 2024-04-02T14:37:04.259 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.254+0000 7f016f400640 1 --2- 172.21.15.90:0/364866631 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b1c2dbbfb0 0x55b1c2db0ff0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f015c02c230 tx=0x7f015c045000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=af2c1447b121340a server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:37:04.259 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.255+0000 7f0164c00640 1 -- 172.21.15.90:0/364866631 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f015c02cba0 con 0x55b1c2dbbfb0 2024-04-02T14:37:04.259 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.255+0000 7f0164c00640 1 -- 172.21.15.90:0/364866631 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f015c058070 con 0x55b1c2dbbfb0 2024-04-02T14:37:04.259 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.255+0000 7f0164c00640 1 -- 172.21.15.90:0/364866631 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f015c053db0 con 0x55b1c2dbbfb0 2024-04-02T14:37:04.260 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.255+0000 7f0179fc0740 1 -- 172.21.15.90:0/364866631 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b1c2dbbfb0 msgr2=0x55b1c2db0ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:04.260 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.255+0000 7f0179fc0740 1 --2- 172.21.15.90:0/364866631 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b1c2dbbfb0 0x55b1c2db0ff0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f015c02c230 tx=0x7f015c045000 comp rx=0 tx=0).stop 2024-04-02T14:37:04.260 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.256+0000 7f0179fc0740 1 -- 172.21.15.90:0/364866631 shutdown_connections 2024-04-02T14:37:04.260 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.256+0000 7f0179fc0740 1 --2- 172.21.15.90:0/364866631 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b1c2dbe3b0 0x55b1c2db1530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:04.260 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.256+0000 7f0179fc0740 1 --2- 172.21.15.90:0/364866631 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b1c2dbbfb0 0x55b1c2db0ff0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:04.260 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.256+0000 7f0179fc0740 1 --2- 172.21.15.90:0/364866631 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b1c2db57d0 0x55b1c2dca5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:04.260 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.256+0000 7f0179fc0740 1 -- 172.21.15.90:0/364866631 >> 172.21.15.90:0/364866631 conn(0x55b1c2daf5e0 msgr2=0x55b1c2db08e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:37:04.260 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.256+0000 7f0179fc0740 1 -- 172.21.15.90:0/364866631 shutdown_connections 2024-04-02T14:37:04.260 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.256+0000 7f0179fc0740 1 -- 172.21.15.90:0/364866631 wait complete. 2024-04-02T14:37:04.260 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.256+0000 7f0179fc0740 1 -- start start 2024-04-02T14:37:04.260 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.256+0000 7f0179fc0740 1 Processor -- start 2024-04-02T14:37:04.260 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.257+0000 7f0179fc0740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b1c2db57d0 0x55b1c2dd4140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:04.261 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.257+0000 7f0179fc0740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b1c2dbbfb0 0x55b1c2dd15c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:04.261 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.257+0000 7f0179fc0740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b1c2dbe3b0 0x55b1c2dd1b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:04.261 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.257+0000 7f0179fc0740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f01701b0e20 con 0x55b1c2db57d0 2024-04-02T14:37:04.261 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.257+0000 7f0179fc0740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f01701b0750 con 0x55b1c2dbe3b0 2024-04-02T14:37:04.261 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.257+0000 7f0179fc0740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f01701ad990 con 0x55b1c2dbbfb0 2024-04-02T14:37:04.261 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.257+0000 7f0179fc0740 10 client.0 _ll_register_callbacks cb 0x7f016009cad0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:37:04.261 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.257+0000 7f0175200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b1c2dbe3b0 0x55b1c2dd1b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:04.261 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.257+0000 7f016fe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b1c2db57d0 0x55b1c2dd4140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:04.261 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.257+0000 7f016f400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b1c2dbbfb0 0x55b1c2dd15c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:04.261 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.257+0000 7f0175200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b1c2dbe3b0 0x55b1c2dd1b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:53570/0 (socket says 172.21.15.90:53570) 2024-04-02T14:37:04.261 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.257+0000 7f0175200640 1 -- 172.21.15.90:0/2479101009 learned_addr learned my addr 172.21.15.90:0/2479101009 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:37:04.261 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.258+0000 7f016f400640 1 -- 172.21.15.90:0/2479101009 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b1c2dbe3b0 msgr2=0x55b1c2dd1b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:37:04.261 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.258+0000 7f016f400640 1 --2- 172.21.15.90:0/2479101009 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b1c2dbe3b0 0x55b1c2dd1b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:04.262 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.258+0000 7f016f400640 1 -- 172.21.15.90:0/2479101009 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b1c2db57d0 msgr2=0x55b1c2dd4140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:37:04.262 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.258+0000 7f016f400640 1 --2- 172.21.15.90:0/2479101009 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b1c2db57d0 0x55b1c2dd4140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:04.262 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.258+0000 7f016f400640 1 -- 172.21.15.90:0/2479101009 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b1c2dd20c0 con 0x55b1c2dbbfb0 2024-04-02T14:37:04.262 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.258+0000 7f0175200640 1 --2- 172.21.15.90:0/2479101009 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b1c2dbe3b0 0x55b1c2dd1b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:37:04.262 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.258+0000 7f016fe00640 1 --2- 172.21.15.90:0/2479101009 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b1c2db57d0 0x55b1c2dd4140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:37:04.262 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.258+0000 7f016f400640 1 --2- 172.21.15.90:0/2479101009 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b1c2dbbfb0 0x55b1c2dd15c0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x55b1c2daf450 tx=0x7f015c02c8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5d82be64ef3aa21 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:37:04.262 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.258+0000 7f015a000640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:37:04.262 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.258+0000 7f015a000640 1 -- 172.21.15.90:0/2479101009 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f015c02cce0 con 0x55b1c2dbbfb0 2024-04-02T14:37:04.262 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.258+0000 7f0179fc0740 1 -- 172.21.15.90:0/2479101009 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f015c058070 con 0x55b1c2dbbfb0 2024-04-02T14:37:04.262 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.258+0000 7f015a000640 1 -- 172.21.15.90:0/2479101009 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f015c0046c0 con 0x55b1c2dbbfb0 2024-04-02T14:37:04.263 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.259+0000 7f015a000640 1 -- 172.21.15.90:0/2479101009 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f015c06a050 con 0x55b1c2dbbfb0 2024-04-02T14:37:04.263 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.259+0000 7f0179fc0740 10 client.26005 fetch_fsmap learned FSMap version 27 2024-04-02T14:37:04.263 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.259+0000 7f0179fc0740 1 -- 172.21.15.90:0/2479101009 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55b1c2e34b70 con 0x55b1c2dbbfb0 2024-04-02T14:37:04.263 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.259+0000 7f015a000640 1 -- 172.21.15.90:0/2479101009 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f015c058070 con 0x55b1c2dbbfb0 2024-04-02T14:37:04.263 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.259+0000 7f015a000640 1 -- 172.21.15.90:0/2479101009 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f015c003b40 con 0x55b1c2dbbfb0 2024-04-02T14:37:04.263 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.259+0000 7f0179fc0740 10 client.26005 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:37:04.263 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.259+0000 7f0179fc0740 10 client.26005 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-02T14:37:04.264 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.259+0000 7f0179fc0740 20 client.26005 populate_metadata read hostname 'smithi090' 2024-04-02T14:37:04.264 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.259+0000 7f0179fc0740 1 --2- 172.21.15.90:0/2479101009 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55b1c2e35240 0x55b1c2e37660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:04.264 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.259+0000 7f0179fc0740 4 client.26005 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-02T14:37:04.264 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.259+0000 7f0179fc0740 1 -- 172.21.15.90:0/2479101009 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55b1c2d272b0 con 0x55b1c2e35240 2024-04-02T14:37:04.264 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.260+0000 7f016fe00640 1 --2- 172.21.15.90:0/2479101009 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55b1c2e35240 0x55b1c2e37660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:04.264 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.260+0000 7f016fe00640 1 --2- 172.21.15.90:0/2479101009 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55b1c2e35240 0x55b1c2e37660 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=664aa201427b0875 server_cookie=6b7a20018fafbb6c in_seq=0 out_seq=0 2024-04-02T14:37:04.264 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.260+0000 7f015a000640 10 client.26005 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:37:04.282 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.281+0000 7f015a000640 1 -- 172.21.15.90:0/2479101009 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55b1c2d272b0 con 0x55b1c2e35240 2024-04-02T14:37:04.283 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.281+0000 7f015a000640 10 client.26005 handle_command_reply: tid=0 2024-04-02T14:37:04.283 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.281+0000 7f0179fc0740 1 client.26005 shutdown 2024-04-02T14:37:04.285 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.283+0000 7f0179fc0740 1 -- 172.21.15.90:0/2479101009 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b1c2dbbfb0 msgr2=0x55b1c2dd15c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:04.285 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.283+0000 7f0179fc0740 1 --2- 172.21.15.90:0/2479101009 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b1c2dbbfb0 0x55b1c2dd15c0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x55b1c2daf450 tx=0x7f015c02c8f0 comp rx=0 tx=0).stop 2024-04-02T14:37:04.285 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.283+0000 7f0179fc0740 1 -- 172.21.15.90:0/2479101009 shutdown_connections 2024-04-02T14:37:04.285 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.283+0000 7f0179fc0740 1 --2- 172.21.15.90:0/2479101009 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55b1c2e35240 0x55b1c2e37660 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:04.285 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.283+0000 7f0179fc0740 1 --2- 172.21.15.90:0/2479101009 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b1c2dbe3b0 0x55b1c2dd1b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:04.285 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.283+0000 7f0179fc0740 1 --2- 172.21.15.90:0/2479101009 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b1c2dbbfb0 0x55b1c2dd15c0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:04.285 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.283+0000 7f0179fc0740 1 --2- 172.21.15.90:0/2479101009 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b1c2db57d0 0x55b1c2dd4140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:04.285 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.283+0000 7f0179fc0740 1 -- 172.21.15.90:0/2479101009 >> 172.21.15.90:0/2479101009 conn(0x55b1c2daf5e0 msgr2=0x55b1c2db1170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:37:04.286 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.283+0000 7f015a000640 0 client.26005 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:37:04.286 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.283+0000 7f0179fc0740 1 -- 172.21.15.90:0/2479101009 shutdown_connections 2024-04-02T14:37:04.286 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.283+0000 7f0179fc0740 1 -- 172.21.15.90:0/2479101009 wait complete. 2024-04-02T14:37:04.286 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.284+0000 7f0179fc0740 20 client.26005 trim_cache size 0 max 16384 2024-04-02T14:37:04.286 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:{ 2024-04-02T14:37:04.286 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "message": "", 2024-04-02T14:37:04.286 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "return_code": 0 2024-04-02T14:37:04.287 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:} 2024-04-02T14:37:04.287 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.285+0000 7f0177200640 1 -- 172.21.15.90:0/4201893099 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f01540a3e40 msgr2=0x7f01540a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:04.287 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.285+0000 7f0177200640 1 --2- 172.21.15.90:0/4201893099 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f01540a3e40 0x7f01540a6300 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f015c009900 tx=0x7f015c02c6a0 comp rx=0 tx=0).stop 2024-04-02T14:37:04.287 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.285+0000 7f0177200640 1 -- 172.21.15.90:0/4201893099 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0170100c50 msgr2=0x7f017019a5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:04.287 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.285+0000 7f0177200640 1 --2- 172.21.15.90:0/4201893099 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0170100c50 0x7f017019a5c0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f016000e9f0 tx=0x7f016000edf0 comp rx=0 tx=0).stop 2024-04-02T14:37:04.287 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.285+0000 7f0177200640 1 -- 172.21.15.90:0/4201893099 shutdown_connections 2024-04-02T14:37:04.288 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.285+0000 7f0177200640 1 --2- 172.21.15.90:0/4201893099 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f01540a3e40 0x7f01540a6300 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:04.288 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.285+0000 7f0177200640 1 --2- 172.21.15.90:0/4201893099 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f01701059a0 0x7f017019ee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:04.288 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.285+0000 7f0177200640 1 --2- 172.21.15.90:0/4201893099 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0170101610 0x7f017019ab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:04.288 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.285+0000 7f0177200640 1 --2- 172.21.15.90:0/4201893099 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0170100c50 0x7f017019a5c0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:04.288 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.285+0000 7f0177200640 1 -- 172.21.15.90:0/4201893099 >> 172.21.15.90:0/4201893099 conn(0x7f0170077850 msgr2=0x7f0170069280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:37:04.288 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.285+0000 7f0177200640 1 -- 172.21.15.90:0/4201893099 shutdown_connections 2024-04-02T14:37:04.288 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:04.287+0000 7f0177200640 1 -- 172.21.15.90:0/4201893099 wait complete. 2024-04-02T14:37:04.862 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:37:04 smithi092 ceph-mon[36497]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:37:04.862 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:37:04 smithi092 ceph-mon[36497]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T14:37:04.862 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:37:04 smithi092 ceph-mon[36497]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' 2024-04-02T14:37:04.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:37:04 smithi177 ceph-mon[36431]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:37:04.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:37:04 smithi177 ceph-mon[36431]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T14:37:04.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:37:04 smithi177 ceph-mon[36431]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' 2024-04-02T14:37:04.938 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:37:04 smithi090 ceph-mon[31674]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:37:04.938 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:37:04 smithi090 ceph-mon[31674]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T14:37:04.938 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:37:04 smithi090 ceph-mon[31674]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' 2024-04-02T14:37:05.860 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:37:05 smithi092 ceph-mon[36497]: pgmap v69: 97 pgs: 97 active+clean; 1.3 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 12 MiB/s wr, 926 op/s 2024-04-02T14:37:05.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:37:05 smithi177 ceph-mon[36431]: pgmap v69: 97 pgs: 97 active+clean; 1.3 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 12 MiB/s wr, 926 op/s 2024-04-02T14:37:05.938 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:37:05 smithi090 ceph-mon[31674]: pgmap v69: 97 pgs: 97 active+clean; 1.3 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 12 MiB/s wr, 926 op/s 2024-04-02T14:37:06.832 INFO:teuthology.orchestra.run.smithi090.stderr:progress: 95.0 s, 212.4 tps, lat 455.845 ms stddev 283.012 2024-04-02T14:37:07.860 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:37:07 smithi092 ceph-mon[36497]: pgmap v70: 97 pgs: 97 active+clean; 1.3 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 6.5 MiB/s wr, 586 op/s 2024-04-02T14:37:07.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:37:07 smithi177 ceph-mon[36431]: pgmap v70: 97 pgs: 97 active+clean; 1.3 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 6.5 MiB/s wr, 586 op/s 2024-04-02T14:37:07.938 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:37:07 smithi090 ceph-mon[31674]: pgmap v70: 97 pgs: 97 active+clean; 1.3 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 6.5 MiB/s wr, 586 op/s 2024-04-02T14:37:09.860 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:37:09 smithi092 ceph-mon[36497]: pgmap v71: 97 pgs: 97 active+clean; 1.3 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 7.5 MiB/s wr, 739 op/s 2024-04-02T14:37:09.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:37:09 smithi177 ceph-mon[36431]: pgmap v71: 97 pgs: 97 active+clean; 1.3 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 7.5 MiB/s wr, 739 op/s 2024-04-02T14:37:09.938 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:37:09 smithi090 ceph-mon[31674]: pgmap v71: 97 pgs: 97 active+clean; 1.3 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 7.5 MiB/s wr, 739 op/s 2024-04-02T14:37:11.831 INFO:teuthology.orchestra.run.smithi090.stderr:progress: 100.0 s, 202.2 tps, lat 522.643 ms stddev 326.569 2024-04-02T14:37:11.860 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:37:11 smithi092 ceph-mon[36497]: pgmap v72: 97 pgs: 97 active+clean; 1.3 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 3.6 MiB/s wr, 469 op/s 2024-04-02T14:37:11.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:37:11 smithi177 ceph-mon[36431]: pgmap v72: 97 pgs: 97 active+clean; 1.3 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 3.6 MiB/s wr, 469 op/s 2024-04-02T14:37:11.939 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:37:11 smithi090 ceph-mon[31674]: pgmap v72: 97 pgs: 97 active+clean; 1.3 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 3.6 MiB/s wr, 469 op/s 2024-04-02T14:37:13.860 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:37:13 smithi092 ceph-mon[36497]: pgmap v73: 97 pgs: 97 active+clean; 1.3 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 2.9 MiB/s wr, 422 op/s 2024-04-02T14:37:13.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:37:13 smithi177 ceph-mon[36431]: pgmap v73: 97 pgs: 97 active+clean; 1.3 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 2.9 MiB/s wr, 422 op/s 2024-04-02T14:37:13.938 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:37:13 smithi090 ceph-mon[31674]: pgmap v73: 97 pgs: 97 active+clean; 1.3 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 2.9 MiB/s wr, 422 op/s 2024-04-02T14:37:15.861 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:37:15 smithi092 ceph-mon[36497]: pgmap v74: 97 pgs: 97 active+clean; 1.3 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 4.0 MiB/s wr, 576 op/s 2024-04-02T14:37:15.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:37:15 smithi177 ceph-mon[36431]: pgmap v74: 97 pgs: 97 active+clean; 1.3 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 4.0 MiB/s wr, 576 op/s 2024-04-02T14:37:15.938 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:37:15 smithi090 ceph-mon[31674]: pgmap v74: 97 pgs: 97 active+clean; 1.3 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 4.0 MiB/s wr, 576 op/s 2024-04-02T14:37:16.832 INFO:teuthology.orchestra.run.smithi090.stderr:progress: 105.0 s, 117.6 tps, lat 552.946 ms stddev 389.805 2024-04-02T14:37:17.450 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.448+0000 7f8a16200640 1 -- 172.21.15.90:0/3825397498 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8a10116b90 msgr2=0x7f8a10118fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:17.451 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.448+0000 7f8a16200640 1 --2- 172.21.15.90:0/3825397498 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8a10116b90 0x7f8a10118fa0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f89f8009900 tx=0x7f89f802bc60 comp rx=0 tx=0).stop 2024-04-02T14:37:17.451 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.449+0000 7f8a16200640 1 -- 172.21.15.90:0/3825397498 shutdown_connections 2024-04-02T14:37:17.451 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.449+0000 7f8a16200640 1 --2- 172.21.15.90:0/3825397498 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8a10116b90 0x7f8a10118fa0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:17.451 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.449+0000 7f8a16200640 1 --2- 172.21.15.90:0/3825397498 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8a10109040 0x7f8a101094a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:17.451 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.449+0000 7f8a16200640 1 --2- 172.21.15.90:0/3825397498 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8a1010c240 0x7f8a10108b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:17.451 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.449+0000 7f8a16200640 1 -- 172.21.15.90:0/3825397498 >> 172.21.15.90:0/3825397498 conn(0x7f8a1006cba0 msgr2=0x7f8a1006cfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:37:17.451 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.449+0000 7f8a16200640 1 -- 172.21.15.90:0/3825397498 shutdown_connections 2024-04-02T14:37:17.451 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.450+0000 7f8a16200640 1 -- 172.21.15.90:0/3825397498 wait complete. 2024-04-02T14:37:17.453 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.452+0000 7f8a16200640 1 Processor -- start 2024-04-02T14:37:17.454 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.453+0000 7f8a16200640 1 -- start start 2024-04-02T14:37:17.454 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.453+0000 7f8a16200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8a10109040 0x7f8a101a2e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:17.455 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.454+0000 7f8a16200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8a1010c240 0x7f8a101a33b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:17.455 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.454+0000 7f8a16200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8a10116b90 0x7f8a101a7730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:17.455 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.454+0000 7f8a16200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f8a1011ca30 con 0x7f8a1010c240 2024-04-02T14:37:17.455 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.454+0000 7f8a16200640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f8a1011c8b0 con 0x7f8a10109040 2024-04-02T14:37:17.455 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.454+0000 7f8a16200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f8a1011cbb0 con 0x7f8a10116b90 2024-04-02T14:37:17.455 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.454+0000 7f8a0fe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8a10116b90 0x7f8a101a7730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:17.455 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.454+0000 7f8a0fe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8a10116b90 0x7f8a101a7730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:58028/0 (socket says 172.21.15.90:58028) 2024-04-02T14:37:17.456 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.454+0000 7f8a0fe00640 1 -- 172.21.15.90:0/2756324201 learned_addr learned my addr 172.21.15.90:0/2756324201 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:37:17.456 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.454+0000 7f8a0ea00640 1 --2- 172.21.15.90:0/2756324201 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8a1010c240 0x7f8a101a33b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:17.456 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.454+0000 7f8a0f400640 1 --2- 172.21.15.90:0/2756324201 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8a10109040 0x7f8a101a2e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:17.456 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.455+0000 7f8a0f400640 1 -- 172.21.15.90:0/2756324201 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8a10116b90 msgr2=0x7f8a101a7730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:17.456 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.455+0000 7f8a0f400640 1 --2- 172.21.15.90:0/2756324201 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8a10116b90 0x7f8a101a7730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:17.456 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.455+0000 7f8a0f400640 1 -- 172.21.15.90:0/2756324201 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8a1010c240 msgr2=0x7f8a101a33b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:17.456 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.455+0000 7f8a0f400640 1 --2- 172.21.15.90:0/2756324201 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8a1010c240 0x7f8a101a33b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:17.456 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.455+0000 7f8a0f400640 1 -- 172.21.15.90:0/2756324201 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a101a7e60 con 0x7f8a10109040 2024-04-02T14:37:17.457 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.455+0000 7f8a0f400640 1 --2- 172.21.15.90:0/2756324201 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8a10109040 0x7f8a101a2e70 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f8a0800b3f0 tx=0x7f8a0800b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:37:17.457 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.455+0000 7f8a07e00640 1 -- 172.21.15.90:0/2756324201 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a08015020 con 0x7f8a10109040 2024-04-02T14:37:17.457 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.455+0000 7f8a07e00640 1 -- 172.21.15.90:0/2756324201 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8a08011d10 con 0x7f8a10109040 2024-04-02T14:37:17.457 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.456+0000 7f8a07e00640 1 -- 172.21.15.90:0/2756324201 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a0800f2e0 con 0x7f8a10109040 2024-04-02T14:37:17.457 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.456+0000 7f8a16200640 1 -- 172.21.15.90:0/2756324201 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a101a8150 con 0x7f8a10109040 2024-04-02T14:37:17.458 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.456+0000 7f8a16200640 1 -- 172.21.15.90:0/2756324201 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a101af9e0 con 0x7f8a10109040 2024-04-02T14:37:17.463 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.462+0000 7f8a18f94740 1 Processor -- start 2024-04-02T14:37:17.463 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.462+0000 7f8a18f94740 1 -- start start 2024-04-02T14:37:17.464 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.463+0000 7f8a07e00640 1 -- 172.21.15.90:0/2756324201 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f8a08018020 con 0x7f8a10109040 2024-04-02T14:37:17.464 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.463+0000 7f8a18f94740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eb7fd29cc0 0x55eb7fd2a0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:17.464 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.463+0000 7f8a18f94740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eb7fd2a670 0x55eb7fd3b680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:17.465 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.464+0000 7f8a18f94740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55eb7fd3bbc0 0x55eb7fd3dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:17.465 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.464+0000 7f8a18f94740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55eb7fc8ed70 con 0x55eb7fd2a670 2024-04-02T14:37:17.465 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.464+0000 7f8a18f94740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55eb7fc372a0 con 0x55eb7fd3bbc0 2024-04-02T14:37:17.465 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.464+0000 7f8a18f94740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55eb7fc8ad20 con 0x55eb7fd29cc0 2024-04-02T14:37:17.465 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.464+0000 7f8a0fe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eb7fd29cc0 0x55eb7fd2a0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:17.465 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.464+0000 7f8a0fe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eb7fd29cc0 0x55eb7fd2a0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:58034/0 (socket says 172.21.15.90:58034) 2024-04-02T14:37:17.466 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.464+0000 7f8a0fe00640 1 -- 172.21.15.90:0/3599842438 learned_addr learned my addr 172.21.15.90:0/3599842438 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:37:17.466 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.464+0000 7f8a0f400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eb7fd2a670 0x55eb7fd3b680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:17.466 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.464+0000 7f8a0fe00640 1 -- 172.21.15.90:0/3599842438 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55eb7fd3bbc0 msgr2=0x55eb7fd3dfb0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-02T14:37:17.466 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.464+0000 7f8a0fe00640 1 --2- 172.21.15.90:0/3599842438 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55eb7fd3bbc0 0x55eb7fd3dfb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:17.466 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.464+0000 7f8a0fe00640 1 -- 172.21.15.90:0/3599842438 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eb7fd2a670 msgr2=0x55eb7fd3b680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:37:17.466 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.464+0000 7f8a0fe00640 1 --2- 172.21.15.90:0/3599842438 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eb7fd2a670 0x55eb7fd3b680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:17.466 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.464+0000 7f8a0fe00640 1 -- 172.21.15.90:0/3599842438 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55eb7fc4b840 con 0x55eb7fd29cc0 2024-04-02T14:37:17.467 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.465+0000 7f8a0fe00640 1 --2- 172.21.15.90:0/3599842438 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eb7fd29cc0 0x55eb7fd2a0a0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f89f802c6b0 tx=0x7f89f802c6e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d5b69d88b96a104f server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:37:17.467 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.466+0000 7f8a07e00640 1 --2- 172.21.15.90:0/2756324201 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f89f40a3e40 0x7f89f40a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:17.468 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.466+0000 7f89ffe00640 1 -- 172.21.15.90:0/3599842438 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89f8004380 con 0x55eb7fd29cc0 2024-04-02T14:37:17.468 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.466+0000 7f89ffe00640 1 -- 172.21.15.90:0/3599842438 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f89f80046c0 con 0x55eb7fd29cc0 2024-04-02T14:37:17.468 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.466+0000 7f89ffe00640 1 -- 172.21.15.90:0/3599842438 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89f80052f0 con 0x55eb7fd29cc0 2024-04-02T14:37:17.468 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.467+0000 7f8a07e00640 1 -- 172.21.15.90:0/2756324201 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f8a080cfdd0 con 0x7f8a10109040 2024-04-02T14:37:17.468 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.467+0000 7f8a0f400640 1 --2- 172.21.15.90:0/2756324201 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f89f40a3e40 0x7f89f40a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:17.469 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.467+0000 7f8a18f94740 1 -- 172.21.15.90:0/3599842438 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eb7fd29cc0 msgr2=0x55eb7fd2a0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:17.469 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.467+0000 7f8a18f94740 1 --2- 172.21.15.90:0/3599842438 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eb7fd29cc0 0x55eb7fd2a0a0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f89f802c6b0 tx=0x7f89f802c6e0 comp rx=0 tx=0).stop 2024-04-02T14:37:17.469 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.468+0000 7f8a18f94740 1 -- 172.21.15.90:0/3599842438 shutdown_connections 2024-04-02T14:37:17.469 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.468+0000 7f8a18f94740 1 --2- 172.21.15.90:0/3599842438 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55eb7fd3bbc0 0x55eb7fd3dfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:17.469 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.468+0000 7f8a18f94740 1 --2- 172.21.15.90:0/3599842438 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eb7fd2a670 0x55eb7fd3b680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:17.469 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.468+0000 7f8a18f94740 1 --2- 172.21.15.90:0/3599842438 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eb7fd29cc0 0x55eb7fd2a0a0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:17.469 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.468+0000 7f8a18f94740 1 -- 172.21.15.90:0/3599842438 >> 172.21.15.90:0/3599842438 conn(0x55eb7fd245e0 msgr2=0x55eb7fd249c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:37:17.470 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.468+0000 7f8a18f94740 1 -- 172.21.15.90:0/3599842438 shutdown_connections 2024-04-02T14:37:17.470 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.469+0000 7f8a0f400640 1 --2- 172.21.15.90:0/2756324201 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f89f40a3e40 0x7f89f40a6300 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f8a080cd6f0 tx=0x7f8a08000bc0 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:37:17.470 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.469+0000 7f8a18f94740 1 -- 172.21.15.90:0/3599842438 wait complete. 2024-04-02T14:37:17.470 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.470+0000 7f8a18f94740 1 -- start start 2024-04-02T14:37:17.471 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.470+0000 7f8a18f94740 1 Processor -- start 2024-04-02T14:37:17.473 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.471+0000 7f8a18f94740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eb7fd29cc0 0x55eb7fd38c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:17.473 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.471+0000 7f8a18f94740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55eb7fd2a670 0x55eb7fd391c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:17.473 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.471+0000 7f8a18f94740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eb7fd3bbc0 0x55eb7fd39700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:17.473 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.471+0000 7f8a18f94740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55eb7fd3e920 con 0x55eb7fd3bbc0 2024-04-02T14:37:17.473 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.471+0000 7f8a18f94740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55eb7fc4a660 con 0x55eb7fd2a670 2024-04-02T14:37:17.473 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.471+0000 7f8a18f94740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55eb7fd3eaa0 con 0x55eb7fd29cc0 2024-04-02T14:37:17.473 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.471+0000 7f8a18f94740 10 client.0 _ll_register_callbacks cb 0x55eb7fc4a050 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:37:17.473 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.471+0000 7f8a0ea00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55eb7fd2a670 0x55eb7fd391c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:17.473 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.471+0000 7f8a0ea00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55eb7fd2a670 0x55eb7fd391c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:48494/0 (socket says 172.21.15.90:48494) 2024-04-02T14:37:17.473 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.471+0000 7f8a0ea00640 1 -- 172.21.15.90:0/1990946425 learned_addr learned my addr 172.21.15.90:0/1990946425 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:37:17.474 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.472+0000 7f8a0ea00640 1 -- 172.21.15.90:0/1990946425 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eb7fd29cc0 msgr2=0x55eb7fd38c80 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-02T14:37:17.474 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.472+0000 7f8a0ea00640 1 --2- 172.21.15.90:0/1990946425 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eb7fd29cc0 0x55eb7fd38c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:17.474 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.472+0000 7f8a0ea00640 1 -- 172.21.15.90:0/1990946425 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eb7fd3bbc0 msgr2=0x55eb7fd39700 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-02T14:37:17.475 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.472+0000 7f8a0fe00640 1 --2- 172.21.15.90:0/1990946425 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eb7fd3bbc0 0x55eb7fd39700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:17.475 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.472+0000 7f8a0ea00640 1 --2- 172.21.15.90:0/1990946425 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eb7fd3bbc0 0x55eb7fd39700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:17.475 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.472+0000 7f8a0ea00640 1 -- 172.21.15.90:0/1990946425 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55eb7fc59e80 con 0x55eb7fd2a670 2024-04-02T14:37:17.475 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.473+0000 7f8a0ea00640 1 --2- 172.21.15.90:0/1990946425 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55eb7fd2a670 0x55eb7fd391c0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f8a0000ab10 tx=0x7f8a0000e0c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b99f175f90f56bb2 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:37:17.475 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.473+0000 7f89fcc00640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:37:17.475 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.473+0000 7f89fcc00640 1 -- 172.21.15.90:0/1990946425 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a00018020 con 0x55eb7fd2a670 2024-04-02T14:37:17.475 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.473+0000 7f89fcc00640 1 -- 172.21.15.90:0/1990946425 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f8a0000ec50 con 0x55eb7fd2a670 2024-04-02T14:37:17.475 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.473+0000 7f8a18f94740 1 -- 172.21.15.90:0/1990946425 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f89f80046c0 con 0x55eb7fd2a670 2024-04-02T14:37:17.475 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.474+0000 7f89fcc00640 1 -- 172.21.15.90:0/1990946425 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f8a00023050 con 0x55eb7fd2a670 2024-04-02T14:37:17.476 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.474+0000 7f8a18f94740 10 client.26093 fetch_fsmap learned FSMap version 27 2024-04-02T14:37:17.476 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.474+0000 7f8a18f94740 1 -- 172.21.15.90:0/1990946425 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55eb7f9a3210 con 0x55eb7fd2a670 2024-04-02T14:37:17.476 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.475+0000 7f89fcc00640 1 -- 172.21.15.90:0/1990946425 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f89f80046c0 con 0x55eb7fd2a670 2024-04-02T14:37:17.476 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.475+0000 7f89fcc00640 1 -- 172.21.15.90:0/1990946425 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f8a0001c430 con 0x55eb7fd2a670 2024-04-02T14:37:17.477 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.475+0000 7f8a18f94740 10 client.26093 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:37:17.477 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.475+0000 7f8a18f94740 10 client.26093 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-02T14:37:17.477 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.475+0000 7f8a18f94740 20 client.26093 populate_metadata read hostname 'smithi090' 2024-04-02T14:37:17.477 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.475+0000 7f8a18f94740 1 --2- 172.21.15.90:0/1990946425 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55eb7fd4c0d0 0x55eb7fd4e500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:17.477 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.475+0000 7f8a18f94740 4 client.26093 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:37:17.477 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.475+0000 7f8a18f94740 1 -- 172.21.15.90:0/1990946425 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55eb7fc3ecd0 con 0x55eb7fd4c0d0 2024-04-02T14:37:17.477 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.476+0000 7f8a0fe00640 1 --2- 172.21.15.90:0/1990946425 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55eb7fd4c0d0 0x55eb7fd4e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:17.477 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.476+0000 7f8a0fe00640 1 --2- 172.21.15.90:0/1990946425 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55eb7fd4c0d0 0x55eb7fd4e500 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1ba49735b6f4b59e server_cookie=dc19a51cbeeb03a4 in_seq=0 out_seq=0 2024-04-02T14:37:17.477 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.476+0000 7f89fcc00640 10 client.26093 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:37:17.478 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.477+0000 7f89fcc00640 1 -- 172.21.15.90:0/1990946425 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55eb7fc3ecd0 con 0x55eb7fd4c0d0 2024-04-02T14:37:17.478 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.477+0000 7f89fcc00640 10 client.26093 handle_command_reply: tid=0 2024-04-02T14:37:17.478 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.477+0000 7f8a18f94740 1 client.26093 shutdown 2024-04-02T14:37:17.479 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.478+0000 7f8a18f94740 1 -- 172.21.15.90:0/1990946425 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55eb7fd2a670 msgr2=0x55eb7fd391c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:17.479 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.478+0000 7f8a18f94740 1 --2- 172.21.15.90:0/1990946425 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55eb7fd2a670 0x55eb7fd391c0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f8a0000ab10 tx=0x7f8a0000e0c0 comp rx=0 tx=0).stop 2024-04-02T14:37:17.479 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.478+0000 7f8a18f94740 1 -- 172.21.15.90:0/1990946425 shutdown_connections 2024-04-02T14:37:17.479 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.478+0000 7f8a18f94740 1 --2- 172.21.15.90:0/1990946425 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55eb7fd4c0d0 0x55eb7fd4e500 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:17.479 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.478+0000 7f8a18f94740 1 --2- 172.21.15.90:0/1990946425 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eb7fd3bbc0 0x55eb7fd39700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:17.479 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.478+0000 7f8a18f94740 1 --2- 172.21.15.90:0/1990946425 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55eb7fd2a670 0x55eb7fd391c0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:17.480 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.478+0000 7f8a18f94740 1 --2- 172.21.15.90:0/1990946425 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eb7fd29cc0 0x55eb7fd38c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:17.480 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.478+0000 7f8a18f94740 1 -- 172.21.15.90:0/1990946425 >> 172.21.15.90:0/1990946425 conn(0x55eb7fd245e0 msgr2=0x55eb7fd26330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:37:17.480 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.478+0000 7f89fcc00640 0 client.26093 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:37:17.480 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.479+0000 7f8a18f94740 1 -- 172.21.15.90:0/1990946425 shutdown_connections 2024-04-02T14:37:17.480 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.479+0000 7f8a18f94740 1 -- 172.21.15.90:0/1990946425 wait complete. 2024-04-02T14:37:17.480 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.479+0000 7f8a18f94740 20 client.26093 trim_cache size 0 max 16384 2024-04-02T14:37:17.508 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.507+0000 7f8a18f94740 1 Processor -- start 2024-04-02T14:37:17.509 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.507+0000 7f8a18f94740 1 -- start start 2024-04-02T14:37:17.509 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.507+0000 7f8a18f94740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55eb7fd29cc0 0x55eb7fd3b3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:17.509 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.507+0000 7f8a18f94740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eb7fd2a670 0x55eb7fd47870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:17.509 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.508+0000 7f8a0fe00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55eb7fd29cc0 0x55eb7fd3b3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:17.509 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.508+0000 7f8a0fe00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55eb7fd29cc0 0x55eb7fd3b3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:48504/0 (socket says 172.21.15.90:48504) 2024-04-02T14:37:17.509 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.508+0000 7f8a18f94740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eb7fd3bbc0 0x55eb7fd37550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:17.510 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.508+0000 7f8a18f94740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f8a101b3e00 con 0x55eb7fd2a670 2024-04-02T14:37:17.510 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.508+0000 7f8a18f94740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f8a101b5940 con 0x55eb7fd29cc0 2024-04-02T14:37:17.510 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.508+0000 7f8a18f94740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f8a101b44d0 con 0x55eb7fd3bbc0 2024-04-02T14:37:17.510 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.508+0000 7f8a0fe00640 1 -- 172.21.15.90:0/1290841090 learned_addr learned my addr 172.21.15.90:0/1290841090 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:37:17.510 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.509+0000 7f8a0fe00640 1 -- 172.21.15.90:0/1290841090 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eb7fd3bbc0 msgr2=0x55eb7fd37550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:37:17.511 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.509+0000 7f8a0fe00640 1 --2- 172.21.15.90:0/1290841090 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eb7fd3bbc0 0x55eb7fd37550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:17.511 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.509+0000 7f8a0fe00640 1 -- 172.21.15.90:0/1290841090 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eb7fd2a670 msgr2=0x55eb7fd47870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:37:17.511 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.509+0000 7f8a0fe00640 1 --2- 172.21.15.90:0/1290841090 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eb7fd2a670 0x55eb7fd47870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:17.511 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.509+0000 7f8a0fe00640 1 -- 172.21.15.90:0/1290841090 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55eb7fd37c30 con 0x55eb7fd29cc0 2024-04-02T14:37:17.511 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.509+0000 7f8a0fe00640 1 --2- 172.21.15.90:0/1290841090 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55eb7fd29cc0 0x55eb7fd3b3b0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f89f802c750 tx=0x7f89f8005de0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c8af0798137e5847 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:37:17.511 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.509+0000 7f89ff400640 1 -- 172.21.15.90:0/1290841090 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89f802c8b0 con 0x55eb7fd29cc0 2024-04-02T14:37:17.511 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.509+0000 7f89ff400640 1 -- 172.21.15.90:0/1290841090 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f89f8004ba0 con 0x55eb7fd29cc0 2024-04-02T14:37:17.511 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.509+0000 7f89ff400640 1 -- 172.21.15.90:0/1290841090 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89f80053e0 con 0x55eb7fd29cc0 2024-04-02T14:37:17.512 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.510+0000 7f8a18f94740 1 -- 172.21.15.90:0/1290841090 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55eb7fd29cc0 msgr2=0x55eb7fd3b3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:17.512 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.510+0000 7f8a18f94740 1 --2- 172.21.15.90:0/1290841090 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55eb7fd29cc0 0x55eb7fd3b3b0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f89f802c750 tx=0x7f89f8005de0 comp rx=0 tx=0).stop 2024-04-02T14:37:17.512 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.510+0000 7f8a18f94740 1 -- 172.21.15.90:0/1290841090 shutdown_connections 2024-04-02T14:37:17.512 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.510+0000 7f8a18f94740 1 --2- 172.21.15.90:0/1290841090 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eb7fd3bbc0 0x55eb7fd37550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:17.512 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.510+0000 7f8a18f94740 1 --2- 172.21.15.90:0/1290841090 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eb7fd2a670 0x55eb7fd47870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:17.512 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.510+0000 7f8a18f94740 1 --2- 172.21.15.90:0/1290841090 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55eb7fd29cc0 0x55eb7fd3b3b0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:17.512 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.510+0000 7f8a18f94740 1 -- 172.21.15.90:0/1290841090 >> 172.21.15.90:0/1290841090 conn(0x55eb7fd245e0 msgr2=0x55eb7fd23640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:37:17.512 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.511+0000 7f8a18f94740 1 -- 172.21.15.90:0/1290841090 shutdown_connections 2024-04-02T14:37:17.513 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.512+0000 7f8a18f94740 1 -- 172.21.15.90:0/1290841090 wait complete. 2024-04-02T14:37:17.513 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.512+0000 7f8a18f94740 1 -- start start 2024-04-02T14:37:17.513 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.513+0000 7f8a18f94740 1 Processor -- start 2024-04-02T14:37:17.517 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.516+0000 7f8a18f94740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55eb7fd29cc0 0x55eb7fd43d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:17.517 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.516+0000 7f8a18f94740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eb7fd2a670 0x55eb7fd442c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:17.517 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.516+0000 7f8a18f94740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eb7fd3bbc0 0x55eb7fd40d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:17.517 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.516+0000 7f8a18f94740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f8a101b5270 con 0x55eb7fd3bbc0 2024-04-02T14:37:17.517 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.516+0000 7f8a18f94740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f8a101b4ba0 con 0x55eb7fd29cc0 2024-04-02T14:37:17.518 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.516+0000 7f8a18f94740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f8a101b1e20 con 0x55eb7fd2a670 2024-04-02T14:37:17.518 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.516+0000 7f8a18f94740 10 client.0 _ll_register_callbacks cb 0x7f8a0000a770 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:37:17.518 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.517+0000 7f8a0ea00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eb7fd2a670 0x55eb7fd442c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:17.518 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.517+0000 7f8a0ea00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eb7fd2a670 0x55eb7fd442c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:58050/0 (socket says 172.21.15.90:58050) 2024-04-02T14:37:17.519 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.517+0000 7f8a0ea00640 1 -- 172.21.15.90:0/4201406939 learned_addr learned my addr 172.21.15.90:0/4201406939 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:37:17.519 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.517+0000 7f8a0fe00640 1 --2- 172.21.15.90:0/4201406939 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55eb7fd29cc0 0x55eb7fd43d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:17.519 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.517+0000 7f8a0fe00640 1 --2- 172.21.15.90:0/4201406939 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eb7fd3bbc0 0x55eb7fd40d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:17.519 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.517+0000 7f8a0ea00640 1 -- 172.21.15.90:0/4201406939 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55eb7fd29cc0 msgr2=0x55eb7fd43d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:37:17.519 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.517+0000 7f8a0ea00640 1 --2- 172.21.15.90:0/4201406939 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55eb7fd29cc0 0x55eb7fd43d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:17.519 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.517+0000 7f8a0ea00640 1 -- 172.21.15.90:0/4201406939 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eb7fd3bbc0 msgr2=0x55eb7fd40d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:37:17.519 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.517+0000 7f8a0ea00640 1 --2- 172.21.15.90:0/4201406939 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eb7fd3bbc0 0x55eb7fd40d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:17.519 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.517+0000 7f8a0ea00640 1 -- 172.21.15.90:0/4201406939 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55eb7fa6f620 con 0x55eb7fd2a670 2024-04-02T14:37:17.520 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.518+0000 7f8a0ea00640 1 --2- 172.21.15.90:0/4201406939 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eb7fd2a670 0x55eb7fd442c0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f8a0001ca40 tx=0x7f8a0001cd70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=152ebe938b08e884 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:37:17.520 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.519+0000 7f89fcc00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:37:17.520 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.519+0000 7f89fcc00640 1 -- 172.21.15.90:0/4201406939 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a00018070 con 0x55eb7fd2a670 2024-04-02T14:37:17.520 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.519+0000 7f89fcc00640 1 -- 172.21.15.90:0/4201406939 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f8a00003a10 con 0x55eb7fd2a670 2024-04-02T14:37:17.521 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.520+0000 7f8a18f94740 1 -- 172.21.15.90:0/4201406939 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f89f8004ba0 con 0x55eb7fd2a670 2024-04-02T14:37:17.521 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.520+0000 7f89fcc00640 1 -- 172.21.15.90:0/4201406939 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f8a00009390 con 0x55eb7fd2a670 2024-04-02T14:37:17.521 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.520+0000 7f8a18f94740 10 client.26014 fetch_fsmap learned FSMap version 27 2024-04-02T14:37:17.521 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.520+0000 7f8a18f94740 1 -- 172.21.15.90:0/4201406939 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55eb7fdb2ce0 con 0x55eb7fd2a670 2024-04-02T14:37:17.522 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.521+0000 7f89fcc00640 1 -- 172.21.15.90:0/4201406939 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f89f8004ba0 con 0x55eb7fd2a670 2024-04-02T14:37:17.522 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.521+0000 7f89fcc00640 1 -- 172.21.15.90:0/4201406939 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f8a0002e070 con 0x55eb7fd2a670 2024-04-02T14:37:17.523 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.521+0000 7f8a18f94740 10 client.26014 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:37:17.523 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.521+0000 7f8a18f94740 10 client.26014 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-02T14:37:17.523 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.521+0000 7f8a18f94740 20 client.26014 populate_metadata read hostname 'smithi090' 2024-04-02T14:37:17.523 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.521+0000 7f8a18f94740 1 --2- 172.21.15.90:0/4201406939 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55eb7fdb3410 0x55eb7fdb5830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:17.523 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.521+0000 7f8a18f94740 4 client.26014 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-02T14:37:17.523 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.521+0000 7f8a18f94740 1 -- 172.21.15.90:0/4201406939 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55eb7fc9c2b0 con 0x55eb7fdb3410 2024-04-02T14:37:17.523 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.522+0000 7f8a0fe00640 1 --2- 172.21.15.90:0/4201406939 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55eb7fdb3410 0x55eb7fdb5830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:17.524 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.523+0000 7f8a0fe00640 1 --2- 172.21.15.90:0/4201406939 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55eb7fdb3410 0x55eb7fdb5830 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9ad8ad85a370251d server_cookie=3e4ce38cdaabb0f6 in_seq=0 out_seq=0 2024-04-02T14:37:17.524 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.523+0000 7f89fcc00640 10 client.26014 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:37:17.540 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.539+0000 7f89fcc00640 1 -- 172.21.15.90:0/4201406939 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55eb7fc9c2b0 con 0x55eb7fdb3410 2024-04-02T14:37:17.540 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.539+0000 7f89fcc00640 10 client.26014 handle_command_reply: tid=0 2024-04-02T14:37:17.540 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.539+0000 7f8a18f94740 1 client.26014 shutdown 2024-04-02T14:37:17.541 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.539+0000 7f8a18f94740 1 -- 172.21.15.90:0/4201406939 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eb7fd2a670 msgr2=0x55eb7fd442c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:17.541 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.539+0000 7f8a18f94740 1 --2- 172.21.15.90:0/4201406939 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eb7fd2a670 0x55eb7fd442c0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f8a0001ca40 tx=0x7f8a0001cd70 comp rx=0 tx=0).stop 2024-04-02T14:37:17.541 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.540+0000 7f8a18f94740 1 -- 172.21.15.90:0/4201406939 shutdown_connections 2024-04-02T14:37:17.542 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.540+0000 7f8a18f94740 1 --2- 172.21.15.90:0/4201406939 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55eb7fdb3410 0x55eb7fdb5830 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:17.542 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.540+0000 7f8a18f94740 1 --2- 172.21.15.90:0/4201406939 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eb7fd3bbc0 0x55eb7fd40d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:17.542 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.540+0000 7f8a18f94740 1 --2- 172.21.15.90:0/4201406939 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eb7fd2a670 0x55eb7fd442c0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:17.542 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.540+0000 7f8a18f94740 1 --2- 172.21.15.90:0/4201406939 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55eb7fd29cc0 0x55eb7fd43d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:17.542 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.540+0000 7f8a18f94740 1 -- 172.21.15.90:0/4201406939 >> 172.21.15.90:0/4201406939 conn(0x55eb7fd245e0 msgr2=0x55eb7fd23640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:37:17.542 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.540+0000 7f89fcc00640 0 client.26014 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:37:17.542 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.540+0000 7f8a18f94740 1 -- 172.21.15.90:0/4201406939 shutdown_connections 2024-04-02T14:37:17.542 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.540+0000 7f8a18f94740 1 -- 172.21.15.90:0/4201406939 wait complete. 2024-04-02T14:37:17.542 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.540+0000 7f8a18f94740 20 client.26014 trim_cache size 0 max 16384 2024-04-02T14:37:17.543 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:{ 2024-04-02T14:37:17.543 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "message": "", 2024-04-02T14:37:17.543 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "return_code": 0 2024-04-02T14:37:17.543 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:} 2024-04-02T14:37:17.543 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.542+0000 7f8a16200640 1 -- 172.21.15.90:0/2756324201 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f89f40a3e40 msgr2=0x7f89f40a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:17.544 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.542+0000 7f8a16200640 1 --2- 172.21.15.90:0/2756324201 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f89f40a3e40 0x7f89f40a6300 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f8a080cd6f0 tx=0x7f8a08000bc0 comp rx=0 tx=0).stop 2024-04-02T14:37:17.544 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.542+0000 7f8a16200640 1 -- 172.21.15.90:0/2756324201 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8a10109040 msgr2=0x7f8a101a2e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:17.544 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.542+0000 7f8a16200640 1 --2- 172.21.15.90:0/2756324201 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8a10109040 0x7f8a101a2e70 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f8a0800b3f0 tx=0x7f8a0800b8b0 comp rx=0 tx=0).stop 2024-04-02T14:37:17.544 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.543+0000 7f8a16200640 1 -- 172.21.15.90:0/2756324201 shutdown_connections 2024-04-02T14:37:17.544 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.543+0000 7f8a16200640 1 --2- 172.21.15.90:0/2756324201 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f89f40a3e40 0x7f89f40a6300 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:17.545 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.543+0000 7f8a16200640 1 --2- 172.21.15.90:0/2756324201 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8a10116b90 0x7f8a101a7730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:17.545 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.543+0000 7f8a16200640 1 --2- 172.21.15.90:0/2756324201 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8a1010c240 0x7f8a101a33b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:17.545 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.543+0000 7f8a16200640 1 --2- 172.21.15.90:0/2756324201 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8a10109040 0x7f8a101a2e70 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:17.545 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.543+0000 7f8a16200640 1 -- 172.21.15.90:0/2756324201 >> 172.21.15.90:0/2756324201 conn(0x7f8a1006cba0 msgr2=0x7f8a10116f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:37:17.545 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.543+0000 7f8a16200640 1 -- 172.21.15.90:0/2756324201 shutdown_connections 2024-04-02T14:37:17.545 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:17.544+0000 7f8a16200640 1 -- 172.21.15.90:0/2756324201 wait complete. 2024-04-02T14:37:17.731 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-02T14:37:17.731 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-02T14:37:17.819 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:37:17 smithi090 ceph-mon[31674]: pgmap v75: 97 pgs: 97 active+clean; 1.3 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 2.3 MiB/s wr, 336 op/s 2024-04-02T14:37:17.860 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:37:17 smithi092 ceph-mon[36497]: pgmap v75: 97 pgs: 97 active+clean; 1.3 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 2.3 MiB/s wr, 336 op/s 2024-04-02T14:37:17.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:37:17 smithi177 ceph-mon[36431]: pgmap v75: 97 pgs: 97 active+clean; 1.3 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 2.3 MiB/s wr, 336 op/s 2024-04-02T14:37:18.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.000+0000 7fcdd0400640 1 -- 172.21.15.90:0/1611314502 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcdc8103f10 msgr2=0x7fcdc81042f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:18.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.000+0000 7fcdd0400640 1 --2- 172.21.15.90:0/1611314502 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcdc8103f10 0x7fcdc81042f0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fcdbc009900 tx=0x7fcdbc02bc40 comp rx=0 tx=0).stop 2024-04-02T14:37:18.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.001+0000 7fcdd0400640 1 -- 172.21.15.90:0/1611314502 shutdown_connections 2024-04-02T14:37:18.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.001+0000 7fcdd0400640 1 --2- 172.21.15.90:0/1611314502 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcdc8108fe0 0x7fcdc8110bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:18.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.001+0000 7fcdd0400640 1 --2- 172.21.15.90:0/1611314502 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcdc81048c0 0x7fcdc8108900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:18.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.001+0000 7fcdd0400640 1 --2- 172.21.15.90:0/1611314502 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcdc8103f10 0x7fcdc81042f0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:18.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.001+0000 7fcdd0400640 1 -- 172.21.15.90:0/1611314502 >> 172.21.15.90:0/1611314502 conn(0x7fcdc80ff7b0 msgr2=0x7fcdc8101bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:37:18.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.001+0000 7fcdd0400640 1 -- 172.21.15.90:0/1611314502 shutdown_connections 2024-04-02T14:37:18.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.002+0000 7fcdd0400640 1 -- 172.21.15.90:0/1611314502 wait complete. 2024-04-02T14:37:18.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.003+0000 7fcdd0400640 1 Processor -- start 2024-04-02T14:37:18.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.003+0000 7fcdd0400640 1 -- start start 2024-04-02T14:37:18.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.003+0000 7fcdd0400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcdc8103f10 0x7fcdc81163c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:18.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.003+0000 7fcdd0400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcdc8108fe0 0x7fcdc8116900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:18.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.003+0000 7fcdd0400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcdc810f9f0 0x7fcdc810fea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:18.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.003+0000 7fcdd0400640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fcdc80775d0 con 0x7fcdc8103f10 2024-04-02T14:37:18.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.003+0000 7fcdd0400640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fcdc8077450 con 0x7fcdc8108fe0 2024-04-02T14:37:18.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.003+0000 7fcdd0400640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fcdc8077750 con 0x7fcdc810f9f0 2024-04-02T14:37:18.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.004+0000 7fcdcda00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcdc8103f10 0x7fcdc81163c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:18.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.004+0000 7fcdcda00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcdc8103f10 0x7fcdc81163c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:36270/0 (socket says 172.21.15.90:36270) 2024-04-02T14:37:18.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.004+0000 7fcdcda00640 1 -- 172.21.15.90:0/230123659 learned_addr learned my addr 172.21.15.90:0/230123659 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:37:18.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.005+0000 7fcdcd000640 1 --2- 172.21.15.90:0/230123659 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcdc8108fe0 0x7fcdc8116900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:18.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.005+0000 7fcdce400640 1 --2- 172.21.15.90:0/230123659 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcdc810f9f0 0x7fcdc810fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:18.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.005+0000 7fcdcda00640 1 -- 172.21.15.90:0/230123659 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcdc810f9f0 msgr2=0x7fcdc810fea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:18.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.005+0000 7fcdcda00640 1 --2- 172.21.15.90:0/230123659 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcdc810f9f0 0x7fcdc810fea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:18.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.005+0000 7fcdcda00640 1 -- 172.21.15.90:0/230123659 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcdc8108fe0 msgr2=0x7fcdc8116900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:18.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.005+0000 7fcdcda00640 1 --2- 172.21.15.90:0/230123659 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcdc8108fe0 0x7fcdc8116900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:18.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.005+0000 7fcdcda00640 1 -- 172.21.15.90:0/230123659 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcdc81106b0 con 0x7fcdc8103f10 2024-04-02T14:37:18.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.005+0000 7fcdcd000640 1 --2- 172.21.15.90:0/230123659 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcdc8108fe0 0x7fcdc8116900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:37:18.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.006+0000 7fcdcda00640 1 --2- 172.21.15.90:0/230123659 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcdc8103f10 0x7fcdc81163c0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fcdbc036040 tx=0x7fcdbc02cb40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:37:18.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.006+0000 7fcdc6000640 1 -- 172.21.15.90:0/230123659 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdbc041020 con 0x7fcdc8103f10 2024-04-02T14:37:18.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.006+0000 7fcdd0400640 1 -- 172.21.15.90:0/230123659 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcdc811a0c0 con 0x7fcdc8103f10 2024-04-02T14:37:18.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.007+0000 7fcdd0400640 1 -- 172.21.15.90:0/230123659 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcdc811a600 con 0x7fcdc8103f10 2024-04-02T14:37:18.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.007+0000 7fcdc6000640 1 -- 172.21.15.90:0/230123659 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcdbc004c80 con 0x7fcdc8103f10 2024-04-02T14:37:18.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.007+0000 7fcdc6000640 1 -- 172.21.15.90:0/230123659 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdbc0038e0 con 0x7fcdc8103f10 2024-04-02T14:37:18.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.011+0000 7fcdc6000640 1 -- 172.21.15.90:0/230123659 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fcdbc044020 con 0x7fcdc8103f10 2024-04-02T14:37:18.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.013+0000 7fcdc6000640 1 --2- 172.21.15.90:0/230123659 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fcdb00a3da0 0x7fcdb00a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:18.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.013+0000 7fcdc6000640 1 -- 172.21.15.90:0/230123659 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7fcdbc0f1d80 con 0x7fcdc8103f10 2024-04-02T14:37:18.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.014+0000 7fcdcd000640 1 --2- 172.21.15.90:0/230123659 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fcdb00a3da0 0x7fcdb00a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:18.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.014+0000 7fcdd31c8740 1 Processor -- start 2024-04-02T14:37:18.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.015+0000 7fcdd31c8740 1 -- start start 2024-04-02T14:37:18.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.015+0000 7fcdcd000640 1 --2- 172.21.15.90:0/230123659 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fcdb00a3da0 0x7fcdb00a6260 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fcdc8112c90 tx=0x7fcdb4005f90 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:37:18.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.015+0000 7fcdd31c8740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5568d42f17b0 0x5568d42f1b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:18.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.015+0000 7fcdd31c8740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5568d42f20d0 0x5568d42fa770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:18.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.015+0000 7fcdd31c8740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5568d42facb0 0x5568d42fd0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:18.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.015+0000 7fcdd31c8740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5568d4255770 con 0x5568d42facb0 2024-04-02T14:37:18.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.015+0000 7fcdd31c8740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x5568d41fd700 con 0x5568d42f20d0 2024-04-02T14:37:18.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.015+0000 7fcdd31c8740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5568d4251720 con 0x5568d42f17b0 2024-04-02T14:37:18.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.015+0000 7fcdcda00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5568d42f17b0 0x5568d42f1b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:18.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.015+0000 7fcdcda00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5568d42f17b0 0x5568d42f1b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:58068/0 (socket says 172.21.15.90:58068) 2024-04-02T14:37:18.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.015+0000 7fcdcda00640 1 -- 172.21.15.90:0/2558417262 learned_addr learned my addr 172.21.15.90:0/2558417262 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:37:18.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.015+0000 7fcdcd000640 1 --2- 172.21.15.90:0/2558417262 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5568d42f20d0 0x5568d42fa770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:18.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.016+0000 7fcdcda00640 1 -- 172.21.15.90:0/2558417262 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5568d42f20d0 msgr2=0x5568d42fa770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:37:18.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.016+0000 7fcdcda00640 1 --2- 172.21.15.90:0/2558417262 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5568d42f20d0 0x5568d42fa770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:18.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.016+0000 7fcdcda00640 1 -- 172.21.15.90:0/2558417262 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5568d42facb0 msgr2=0x5568d42fd0a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-02T14:37:18.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.018+0000 7fcdce400640 1 --2- 172.21.15.90:0/2558417262 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5568d42facb0 0x5568d42fd0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:18.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.018+0000 7fcdcda00640 1 --2- 172.21.15.90:0/2558417262 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5568d42facb0 0x5568d42fd0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:18.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.018+0000 7fcdcda00640 1 -- 172.21.15.90:0/2558417262 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5568d40337d0 con 0x5568d42f17b0 2024-04-02T14:37:18.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.018+0000 7fcdcda00640 1 --2- 172.21.15.90:0/2558417262 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5568d42f17b0 0x5568d42f1b90 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fcdbc02c4f0 tx=0x7fcdbc0bd860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6aa30f78440e9650 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:37:18.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.019+0000 7fcdbaa00640 1 -- 172.21.15.90:0/2558417262 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdbc0d7070 con 0x5568d42f17b0 2024-04-02T14:37:18.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.019+0000 7fcdbaa00640 1 -- 172.21.15.90:0/2558417262 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcdbc0bde40 con 0x5568d42f17b0 2024-04-02T14:37:18.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.019+0000 7fcdbaa00640 1 -- 172.21.15.90:0/2558417262 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdbc0c0070 con 0x5568d42f17b0 2024-04-02T14:37:18.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.020+0000 7fcdd31c8740 1 -- 172.21.15.90:0/2558417262 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5568d42f17b0 msgr2=0x5568d42f1b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:18.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.020+0000 7fcdd31c8740 1 --2- 172.21.15.90:0/2558417262 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5568d42f17b0 0x5568d42f1b90 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fcdbc02c4f0 tx=0x7fcdbc0bd860 comp rx=0 tx=0).stop 2024-04-02T14:37:18.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.021+0000 7fcdd31c8740 1 -- 172.21.15.90:0/2558417262 shutdown_connections 2024-04-02T14:37:18.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.021+0000 7fcdd31c8740 1 --2- 172.21.15.90:0/2558417262 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5568d42facb0 0x5568d42fd0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:18.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.021+0000 7fcdd31c8740 1 --2- 172.21.15.90:0/2558417262 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5568d42f20d0 0x5568d42fa770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:18.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.021+0000 7fcdd31c8740 1 --2- 172.21.15.90:0/2558417262 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5568d42f17b0 0x5568d42f1b90 secure :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fcdbc02c4f0 tx=0x7fcdbc0bd860 comp rx=0 tx=0).stop 2024-04-02T14:37:18.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.021+0000 7fcdd31c8740 1 -- 172.21.15.90:0/2558417262 >> 172.21.15.90:0/2558417262 conn(0x5568d42ebee0 msgr2=0x5568d42ec2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:37:18.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.021+0000 7fcdd31c8740 1 -- 172.21.15.90:0/2558417262 shutdown_connections 2024-04-02T14:37:18.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.022+0000 7fcdd31c8740 1 -- 172.21.15.90:0/2558417262 wait complete. 2024-04-02T14:37:18.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.022+0000 7fcdd31c8740 1 -- start start 2024-04-02T14:37:18.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.023+0000 7fcdd31c8740 1 Processor -- start 2024-04-02T14:37:18.025 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.024+0000 7fcdd31c8740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5568d42f20d0 0x5568d42f7e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:18.025 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.024+0000 7fcdd31c8740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5568d42facb0 0x5568d42f8380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:18.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.024+0000 7fcdd31c8740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5568d42f88c0 0x5568d4306c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:18.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.024+0000 7fcdd31c8740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5568d42fd860 con 0x5568d42f88c0 2024-04-02T14:37:18.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.024+0000 7fcdd31c8740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x5568d4210ac0 con 0x5568d42facb0 2024-04-02T14:37:18.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.024+0000 7fcdd31c8740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5568d42fd9e0 con 0x5568d42f20d0 2024-04-02T14:37:18.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.024+0000 7fcdd31c8740 10 client.0 _ll_register_callbacks cb 0x5568d4213180 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:37:18.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.024+0000 7fcdcd000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5568d42facb0 0x5568d42f8380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:18.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.024+0000 7fcdcd000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5568d42facb0 0x5568d42f8380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:48562/0 (socket says 172.21.15.90:48562) 2024-04-02T14:37:18.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.024+0000 7fcdcd000640 1 -- 172.21.15.90:0/2162068334 learned_addr learned my addr 172.21.15.90:0/2162068334 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:37:18.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.024+0000 7fcdcda00640 1 --2- 172.21.15.90:0/2162068334 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5568d42f20d0 0x5568d42f7e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:18.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.025+0000 7fcdce400640 1 --2- 172.21.15.90:0/2162068334 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5568d42f88c0 0x5568d4306c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:18.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.025+0000 7fcdcda00640 1 -- 172.21.15.90:0/2162068334 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5568d42facb0 msgr2=0x5568d42f8380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:37:18.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.025+0000 7fcdcda00640 1 --2- 172.21.15.90:0/2162068334 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5568d42facb0 0x5568d42f8380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:18.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.025+0000 7fcdcda00640 1 -- 172.21.15.90:0/2162068334 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5568d42f88c0 msgr2=0x5568d4306c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:37:18.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.025+0000 7fcdcda00640 1 --2- 172.21.15.90:0/2162068334 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5568d42f88c0 0x5568d4306c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:18.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.025+0000 7fcdcda00640 1 -- 172.21.15.90:0/2162068334 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5568d3f693f0 con 0x5568d42f20d0 2024-04-02T14:37:18.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.025+0000 7fcdce400640 1 --2- 172.21.15.90:0/2162068334 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5568d42f88c0 0x5568d4306c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:37:18.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.026+0000 7fcdcda00640 1 --2- 172.21.15.90:0/2162068334 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5568d42f20d0 0x5568d42f7e40 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fcdbc02c4f0 tx=0x7fcdbc0095d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=914d887a2d4241da server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:37:18.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.026+0000 7fcdaf400640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:37:18.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.027+0000 7fcdaf400640 1 -- 172.21.15.90:0/2162068334 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdbc0c0070 con 0x5568d42f20d0 2024-04-02T14:37:18.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.027+0000 7fcdaf400640 1 -- 172.21.15.90:0/2162068334 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7fcdbc0be2b0 con 0x5568d42f20d0 2024-04-02T14:37:18.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.027+0000 7fcdd31c8740 1 -- 172.21.15.90:0/2162068334 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcdbc0bde40 con 0x5568d42f20d0 2024-04-02T14:37:18.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.027+0000 7fcdaf400640 1 -- 172.21.15.90:0/2162068334 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fcdbc0d7040 con 0x5568d42f20d0 2024-04-02T14:37:18.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.028+0000 7fcdd31c8740 10 client.26026 fetch_fsmap learned FSMap version 27 2024-04-02T14:37:18.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.028+0000 7fcdd31c8740 1 -- 172.21.15.90:0/2162068334 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5568d3f290f0 con 0x5568d42f20d0 2024-04-02T14:37:18.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.028+0000 7fcdaf400640 1 -- 172.21.15.90:0/2162068334 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcdbc0bde40 con 0x5568d42f20d0 2024-04-02T14:37:18.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.028+0000 7fcdaf400640 1 -- 172.21.15.90:0/2162068334 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fcdbc0bd290 con 0x5568d42f20d0 2024-04-02T14:37:18.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.028+0000 7fcdd31c8740 10 client.26026 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:37:18.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.028+0000 7fcdd31c8740 10 client.26026 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:37:18.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.028+0000 7fcdd31c8740 20 client.26026 populate_metadata read hostname 'smithi090' 2024-04-02T14:37:18.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.028+0000 7fcdd31c8740 1 --2- 172.21.15.90:0/2162068334 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5568d430ae80 0x5568d430d270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:18.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.028+0000 7fcdd31c8740 4 client.26026 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:37:18.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.028+0000 7fcdd31c8740 1 -- 172.21.15.90:0/2162068334 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5568d41f6630 con 0x5568d430ae80 2024-04-02T14:37:18.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.029+0000 7fcdcd000640 1 --2- 172.21.15.90:0/2162068334 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5568d430ae80 0x5568d430d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:18.031 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.030+0000 7fcdcd000640 1 --2- 172.21.15.90:0/2162068334 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5568d430ae80 0x5568d430d270 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=68bda36d5173b830 server_cookie=d1a55c10b7ec75cd in_seq=0 out_seq=0 2024-04-02T14:37:18.031 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.030+0000 7fcdaf400640 10 client.26026 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:37:18.032 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.031+0000 7fcdaf400640 1 -- 172.21.15.90:0/2162068334 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5568d41f6630 con 0x5568d430ae80 2024-04-02T14:37:18.032 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.031+0000 7fcdaf400640 10 client.26026 handle_command_reply: tid=0 2024-04-02T14:37:18.033 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.031+0000 7fcdd31c8740 1 client.26026 shutdown 2024-04-02T14:37:18.033 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.031+0000 7fcdd31c8740 1 -- 172.21.15.90:0/2162068334 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5568d42f20d0 msgr2=0x5568d42f7e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:18.033 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.031+0000 7fcdd31c8740 1 --2- 172.21.15.90:0/2162068334 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5568d42f20d0 0x5568d42f7e40 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fcdbc02c4f0 tx=0x7fcdbc0095d0 comp rx=0 tx=0).stop 2024-04-02T14:37:18.033 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.032+0000 7fcdd31c8740 1 -- 172.21.15.90:0/2162068334 shutdown_connections 2024-04-02T14:37:18.033 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.032+0000 7fcdd31c8740 1 --2- 172.21.15.90:0/2162068334 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5568d430ae80 0x5568d430d270 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:18.033 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.032+0000 7fcdaf400640 0 client.26026 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:37:18.033 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.032+0000 7fcdd31c8740 1 --2- 172.21.15.90:0/2162068334 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5568d42f88c0 0x5568d4306c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:18.033 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.032+0000 7fcdd31c8740 1 --2- 172.21.15.90:0/2162068334 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5568d42facb0 0x5568d42f8380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:18.033 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.032+0000 7fcdd31c8740 1 --2- 172.21.15.90:0/2162068334 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5568d42f20d0 0x5568d42f7e40 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:18.034 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.032+0000 7fcdd31c8740 1 -- 172.21.15.90:0/2162068334 >> 172.21.15.90:0/2162068334 conn(0x5568d42ebee0 msgr2=0x5568d42edc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:37:18.034 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.032+0000 7fcdd31c8740 1 -- 172.21.15.90:0/2162068334 shutdown_connections 2024-04-02T14:37:18.034 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.032+0000 7fcdd31c8740 1 -- 172.21.15.90:0/2162068334 wait complete. 2024-04-02T14:37:18.034 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.032+0000 7fcdd31c8740 20 client.26026 trim_cache size 0 max 16384 2024-04-02T14:37:18.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.065+0000 7fcdd31c8740 1 Processor -- start 2024-04-02T14:37:18.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.065+0000 7fcdd31c8740 1 -- start start 2024-04-02T14:37:18.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.065+0000 7fcdd31c8740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5568d42ebee0 0x5568d4306720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:18.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.065+0000 7fcdd31c8740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5568d42f20d0 0x5568d4306c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:18.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.065+0000 7fcdd31c8740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5568d42f88c0 0x5568d4312aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:18.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.065+0000 7fcdd31c8740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fcda800e660 con 0x5568d42f20d0 2024-04-02T14:37:18.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.065+0000 7fcdd31c8740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fcda80101a0 con 0x5568d42f88c0 2024-04-02T14:37:18.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.065+0000 7fcdd31c8740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fcda800ed30 con 0x5568d42ebee0 2024-04-02T14:37:18.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.066+0000 7fcdcda00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5568d42ebee0 0x5568d4306720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:18.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.066+0000 7fcdcda00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5568d42ebee0 0x5568d4306720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:58088/0 (socket says 172.21.15.90:58088) 2024-04-02T14:37:18.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.066+0000 7fcdcda00640 1 -- 172.21.15.90:0/3027957568 learned_addr learned my addr 172.21.15.90:0/3027957568 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:37:18.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.066+0000 7fcdce400640 1 --2- 172.21.15.90:0/3027957568 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5568d42f88c0 0x5568d4312aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:18.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.066+0000 7fcdcda00640 1 -- 172.21.15.90:0/3027957568 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5568d42f88c0 msgr2=0x5568d4312aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:37:18.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.066+0000 7fcdcda00640 1 --2- 172.21.15.90:0/3027957568 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5568d42f88c0 0x5568d4312aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:18.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.066+0000 7fcdcda00640 1 -- 172.21.15.90:0/3027957568 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5568d42f20d0 msgr2=0x5568d4306c60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-02T14:37:18.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.066+0000 7fcdcd000640 1 --2- 172.21.15.90:0/3027957568 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5568d42f20d0 0x5568d4306c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:18.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.067+0000 7fcdcda00640 1 --2- 172.21.15.90:0/3027957568 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5568d42f20d0 0x5568d4306c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:18.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.067+0000 7fcdcda00640 1 -- 172.21.15.90:0/3027957568 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5568d42ec820 con 0x5568d42ebee0 2024-04-02T14:37:18.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.067+0000 7fcdce400640 1 --2- 172.21.15.90:0/3027957568 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5568d42f88c0 0x5568d4312aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:37:18.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.067+0000 7fcdcd000640 1 --2- 172.21.15.90:0/3027957568 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5568d42f20d0 0x5568d4306c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:37:18.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.067+0000 7fcdcda00640 1 --2- 172.21.15.90:0/3027957568 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5568d42ebee0 0x5568d4306720 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fcdbc009430 tx=0x7fcdbc0ef6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b92a48d82982a335 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:37:18.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.068+0000 7fcdba000640 1 -- 172.21.15.90:0/3027957568 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdbc0c0070 con 0x5568d42ebee0 2024-04-02T14:37:18.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.068+0000 7fcdba000640 1 -- 172.21.15.90:0/3027957568 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcdbc0ef760 con 0x5568d42ebee0 2024-04-02T14:37:18.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.068+0000 7fcdba000640 1 -- 172.21.15.90:0/3027957568 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdbc0c6620 con 0x5568d42ebee0 2024-04-02T14:37:18.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.069+0000 7fcdd31c8740 1 -- 172.21.15.90:0/3027957568 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5568d42ebee0 msgr2=0x5568d4306720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:18.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.069+0000 7fcdd31c8740 1 --2- 172.21.15.90:0/3027957568 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5568d42ebee0 0x5568d4306720 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fcdbc009430 tx=0x7fcdbc0ef6a0 comp rx=0 tx=0).stop 2024-04-02T14:37:18.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.069+0000 7fcdd31c8740 1 -- 172.21.15.90:0/3027957568 shutdown_connections 2024-04-02T14:37:18.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.069+0000 7fcdd31c8740 1 --2- 172.21.15.90:0/3027957568 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5568d42f88c0 0x5568d4312aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:18.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.069+0000 7fcdd31c8740 1 --2- 172.21.15.90:0/3027957568 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5568d42f20d0 0x5568d4306c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:18.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.069+0000 7fcdd31c8740 1 --2- 172.21.15.90:0/3027957568 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5568d42ebee0 0x5568d4306720 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:18.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.069+0000 7fcdd31c8740 1 -- 172.21.15.90:0/3027957568 >> 172.21.15.90:0/3027957568 conn(0x5568d42f84e0 msgr2=0x5568d42f6e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:37:18.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.069+0000 7fcdd31c8740 1 -- 172.21.15.90:0/3027957568 shutdown_connections 2024-04-02T14:37:18.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.069+0000 7fcdd31c8740 1 -- 172.21.15.90:0/3027957568 wait complete. 2024-04-02T14:37:18.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.070+0000 7fcdd31c8740 1 -- start start 2024-04-02T14:37:18.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.070+0000 7fcdd31c8740 1 Processor -- start 2024-04-02T14:37:18.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.070+0000 7fcdd31c8740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5568d42ebee0 0x5568d4302c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:18.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.070+0000 7fcdd31c8740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5568d42f20d0 0x5568d4303170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:18.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.070+0000 7fcdd31c8740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5568d42f88c0 0x5568d42ffc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:18.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.070+0000 7fcdd31c8740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fcda800fad0 con 0x5568d42ebee0 2024-04-02T14:37:18.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.070+0000 7fcdd31c8740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fcda800f400 con 0x5568d42f88c0 2024-04-02T14:37:18.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.070+0000 7fcdd31c8740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fcda800c640 con 0x5568d42f20d0 2024-04-02T14:37:18.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.070+0000 7fcdd31c8740 10 client.0 _ll_register_callbacks cb 0x7fcdbc0d5070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:37:18.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.071+0000 7fcdce400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5568d42f88c0 0x5568d42ffc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:18.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.071+0000 7fcdce400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5568d42f88c0 0x5568d42ffc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:48578/0 (socket says 172.21.15.90:48578) 2024-04-02T14:37:18.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.071+0000 7fcdce400640 1 -- 172.21.15.90:0/2636889381 learned_addr learned my addr 172.21.15.90:0/2636889381 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:37:18.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.071+0000 7fcdcda00640 1 --2- 172.21.15.90:0/2636889381 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5568d42ebee0 0x5568d4302c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:18.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.071+0000 7fcdce400640 1 -- 172.21.15.90:0/2636889381 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5568d42f20d0 msgr2=0x5568d4303170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:37:18.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.071+0000 7fcdcd000640 1 --2- 172.21.15.90:0/2636889381 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5568d42f20d0 0x5568d4303170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:18.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.071+0000 7fcdce400640 1 --2- 172.21.15.90:0/2636889381 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5568d42f20d0 0x5568d4303170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:18.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.071+0000 7fcdce400640 1 -- 172.21.15.90:0/2636889381 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5568d42ebee0 msgr2=0x5568d4302c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:37:18.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.071+0000 7fcdce400640 1 --2- 172.21.15.90:0/2636889381 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5568d42ebee0 0x5568d4302c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:18.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.071+0000 7fcdce400640 1 -- 172.21.15.90:0/2636889381 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5568d43002c0 con 0x5568d42f88c0 2024-04-02T14:37:18.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.071+0000 7fcdcda00640 1 --2- 172.21.15.90:0/2636889381 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5568d42ebee0 0x5568d4302c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:37:18.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.071+0000 7fcdcd000640 1 --2- 172.21.15.90:0/2636889381 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5568d42f20d0 0x5568d4303170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:37:18.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.072+0000 7fcdce400640 1 --2- 172.21.15.90:0/2636889381 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5568d42f88c0 0x5568d42ffc00 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fcdc0013760 tx=0x7fcdc0013c20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=175cb76b32ab6e7 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:37:18.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.072+0000 7fcdaf400640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:37:18.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.072+0000 7fcdaf400640 1 -- 172.21.15.90:0/2636889381 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdc0011dd0 con 0x5568d42f88c0 2024-04-02T14:37:18.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.072+0000 7fcdaf400640 1 -- 172.21.15.90:0/2636889381 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7fcdc0004b20 con 0x5568d42f88c0 2024-04-02T14:37:18.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.072+0000 7fcdd31c8740 1 -- 172.21.15.90:0/2636889381 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcdbc0ef760 con 0x5568d42f88c0 2024-04-02T14:37:18.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.072+0000 7fcdaf400640 1 -- 172.21.15.90:0/2636889381 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fcdc0004d20 con 0x5568d42f88c0 2024-04-02T14:37:18.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.072+0000 7fcdd31c8740 10 client.26117 fetch_fsmap learned FSMap version 27 2024-04-02T14:37:18.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.072+0000 7fcdd31c8740 1 -- 172.21.15.90:0/2636889381 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5568d4371cc0 con 0x5568d42f88c0 2024-04-02T14:37:18.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.073+0000 7fcdaf400640 1 -- 172.21.15.90:0/2636889381 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcdbc0ef760 con 0x5568d42f88c0 2024-04-02T14:37:18.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.074+0000 7fcdaf400640 1 -- 172.21.15.90:0/2636889381 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fcdc0027470 con 0x5568d42f88c0 2024-04-02T14:37:18.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.074+0000 7fcdd31c8740 10 client.26117 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:37:18.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.074+0000 7fcdd31c8740 10 client.26117 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:37:18.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.074+0000 7fcdd31c8740 20 client.26117 populate_metadata read hostname 'smithi090' 2024-04-02T14:37:18.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.074+0000 7fcdd31c8740 1 --2- 172.21.15.90:0/2636889381 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5568d42facb0 0x5568d4374400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:18.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.074+0000 7fcdd31c8740 4 client.26117 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-02T14:37:18.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.074+0000 7fcdd31c8740 1 -- 172.21.15.90:0/2636889381 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5568d4262cb0 con 0x5568d42facb0 2024-04-02T14:37:18.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.074+0000 7fcdcda00640 1 --2- 172.21.15.90:0/2636889381 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5568d42facb0 0x5568d4374400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:18.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.075+0000 7fcdaf400640 10 client.26117 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:37:18.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.075+0000 7fcdcda00640 1 --2- 172.21.15.90:0/2636889381 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5568d42facb0 0x5568d4374400 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ef54d40697086502 server_cookie=10d0dacdbc56bacd in_seq=0 out_seq=0 2024-04-02T14:37:18.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.075+0000 7fcdaf400640 1 -- 172.21.15.90:0/2636889381 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5568d4262cb0 con 0x5568d42facb0 2024-04-02T14:37:18.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.075+0000 7fcdaf400640 10 client.26117 handle_command_reply: tid=0 2024-04-02T14:37:18.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.075+0000 7fcdd31c8740 1 client.26117 shutdown 2024-04-02T14:37:18.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.075+0000 7fcdd31c8740 1 -- 172.21.15.90:0/2636889381 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5568d42f88c0 msgr2=0x5568d42ffc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:18.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.075+0000 7fcdd31c8740 1 --2- 172.21.15.90:0/2636889381 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5568d42f88c0 0x5568d42ffc00 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fcdc0013760 tx=0x7fcdc0013c20 comp rx=0 tx=0).stop 2024-04-02T14:37:18.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.076+0000 7fcdd31c8740 1 -- 172.21.15.90:0/2636889381 shutdown_connections 2024-04-02T14:37:18.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.076+0000 7fcdd31c8740 1 --2- 172.21.15.90:0/2636889381 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5568d42facb0 0x5568d4374400 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:18.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.076+0000 7fcdd31c8740 1 --2- 172.21.15.90:0/2636889381 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5568d42f88c0 0x5568d42ffc00 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:18.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.076+0000 7fcdd31c8740 1 --2- 172.21.15.90:0/2636889381 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5568d42f20d0 0x5568d4303170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:18.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.076+0000 7fcdd31c8740 1 --2- 172.21.15.90:0/2636889381 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5568d42ebee0 0x5568d4302c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:18.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.076+0000 7fcdd31c8740 1 -- 172.21.15.90:0/2636889381 >> 172.21.15.90:0/2636889381 conn(0x5568d42f84e0 msgr2=0x5568d42ea6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:37:18.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.076+0000 7fcdaf400640 0 client.26117 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:37:18.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.076+0000 7fcdd31c8740 1 -- 172.21.15.90:0/2636889381 shutdown_connections 2024-04-02T14:37:18.078 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-02T14:37:18.078 INFO:teuthology.orchestra.run.smithi090.stdout: "status": "no active scrubs running", 2024-04-02T14:37:18.078 INFO:teuthology.orchestra.run.smithi090.stdout: "scrubs": {} 2024-04-02T14:37:18.078 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-02T14:37:18.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.076+0000 7fcdd31c8740 1 -- 172.21.15.90:0/2636889381 wait complete. 2024-04-02T14:37:18.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.076+0000 7fcdd31c8740 20 client.26117 trim_cache size 0 max 16384 2024-04-02T14:37:18.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.077+0000 7fcdd0400640 1 -- 172.21.15.90:0/230123659 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fcdb00a3da0 msgr2=0x7fcdb00a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:18.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.077+0000 7fcdd0400640 1 --2- 172.21.15.90:0/230123659 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fcdb00a3da0 0x7fcdb00a6260 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fcdc8112c90 tx=0x7fcdb4005f90 comp rx=0 tx=0).stop 2024-04-02T14:37:18.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.078+0000 7fcdd0400640 1 -- 172.21.15.90:0/230123659 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcdc8103f10 msgr2=0x7fcdc81163c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:18.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.078+0000 7fcdd0400640 1 --2- 172.21.15.90:0/230123659 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcdc8103f10 0x7fcdc81163c0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fcdbc036040 tx=0x7fcdbc02cb40 comp rx=0 tx=0).stop 2024-04-02T14:37:18.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.078+0000 7fcdd0400640 1 -- 172.21.15.90:0/230123659 shutdown_connections 2024-04-02T14:37:18.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.078+0000 7fcdd0400640 1 --2- 172.21.15.90:0/230123659 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fcdb00a3da0 0x7fcdb00a6260 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:18.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.078+0000 7fcdd0400640 1 --2- 172.21.15.90:0/230123659 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcdc810f9f0 0x7fcdc810fea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:18.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.078+0000 7fcdd0400640 1 --2- 172.21.15.90:0/230123659 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcdc8108fe0 0x7fcdc8116900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:18.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.078+0000 7fcdd0400640 1 --2- 172.21.15.90:0/230123659 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcdc8103f10 0x7fcdc81163c0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:18.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.078+0000 7fcdd0400640 1 -- 172.21.15.90:0/230123659 >> 172.21.15.90:0/230123659 conn(0x7fcdc80ff7b0 msgr2=0x7fcdc8101ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:37:18.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.078+0000 7fcdd0400640 1 -- 172.21.15.90:0/230123659 shutdown_connections 2024-04-02T14:37:18.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.078+0000 7fcdd0400640 1 -- 172.21.15.90:0/230123659 wait complete. 2024-04-02T14:37:18.110 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-02T14:37:18.110 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T14:37:18.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.387+0000 7f873b600640 1 -- 172.21.15.90:0/3161566038 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f873410a7d0 msgr2=0x7f87340ffbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:18.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.387+0000 7f873b600640 1 --2- 172.21.15.90:0/3161566038 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f873410a7d0 0x7f87340ffbd0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f8720009860 tx=0x7f872002bb70 comp rx=0 tx=0).stop 2024-04-02T14:37:18.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.388+0000 7f873b600640 1 -- 172.21.15.90:0/3161566038 shutdown_connections 2024-04-02T14:37:18.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.388+0000 7f873b600640 1 --2- 172.21.15.90:0/3161566038 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8734100300 0x7f8734110c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:18.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.388+0000 7f873b600640 1 --2- 172.21.15.90:0/3161566038 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f873410a7d0 0x7f87340ffbd0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:18.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.388+0000 7f873b600640 1 --2- 172.21.15.90:0/3161566038 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8734109e20 0x7f873410a200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:18.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.388+0000 7f873b600640 1 -- 172.21.15.90:0/3161566038 >> 172.21.15.90:0/3161566038 conn(0x7f8734076b60 msgr2=0x7f87340fdee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:37:18.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.388+0000 7f873b600640 1 -- 172.21.15.90:0/3161566038 shutdown_connections 2024-04-02T14:37:18.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.388+0000 7f873b600640 1 -- 172.21.15.90:0/3161566038 wait complete. 2024-04-02T14:37:18.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.391+0000 7f873b600640 1 Processor -- start 2024-04-02T14:37:18.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.391+0000 7f873b600640 1 -- start start 2024-04-02T14:37:18.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.392+0000 7f873b600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8734100300 0x7f8734110aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:18.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.392+0000 7f873b600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8734109e20 0x7f873410db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:18.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.392+0000 7f873b600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f873410e080 0x7f873410e530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:18.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.392+0000 7f873b600640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f8734114060 con 0x7f8734109e20 2024-04-02T14:37:18.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.392+0000 7f873b600640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f8734113ee0 con 0x7f8734100300 2024-04-02T14:37:18.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.392+0000 7f873b600640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f87341141e0 con 0x7f873410e080 2024-04-02T14:37:18.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.392+0000 7f8739600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f873410e080 0x7f873410e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:18.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.393+0000 7f8739600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f873410e080 0x7f873410e530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:58114/0 (socket says 172.21.15.90:58114) 2024-04-02T14:37:18.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.393+0000 7f8739600640 1 -- 172.21.15.90:0/1635187202 learned_addr learned my addr 172.21.15.90:0/1635187202 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:37:18.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.393+0000 7f8738c00640 1 --2- 172.21.15.90:0/1635187202 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8734100300 0x7f8734110aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:18.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.393+0000 7f8733e00640 1 --2- 172.21.15.90:0/1635187202 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8734109e20 0x7f873410db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:18.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.393+0000 7f8739600640 1 -- 172.21.15.90:0/1635187202 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8734100300 msgr2=0x7f8734110aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:18.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.393+0000 7f8739600640 1 --2- 172.21.15.90:0/1635187202 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8734100300 0x7f8734110aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:18.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.393+0000 7f8739600640 1 -- 172.21.15.90:0/1635187202 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8734109e20 msgr2=0x7f873410db40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:18.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.393+0000 7f8739600640 1 --2- 172.21.15.90:0/1635187202 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8734109e20 0x7f873410db40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:18.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.393+0000 7f8739600640 1 -- 172.21.15.90:0/1635187202 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87341aaf60 con 0x7f873410e080 2024-04-02T14:37:18.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.393+0000 7f8739600640 1 --2- 172.21.15.90:0/1635187202 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f873410e080 0x7f873410e530 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f872c009fd0 tx=0x7f872c00c800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:37:18.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.394+0000 7f8731600640 1 -- 172.21.15.90:0/1635187202 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f872c004b20 con 0x7f873410e080 2024-04-02T14:37:18.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.394+0000 7f8731600640 1 -- 172.21.15.90:0/1635187202 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f872c003e50 con 0x7f873410e080 2024-04-02T14:37:18.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.394+0000 7f8731600640 1 -- 172.21.15.90:0/1635187202 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f872c00b630 con 0x7f873410e080 2024-04-02T14:37:18.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.394+0000 7f873b600640 1 -- 172.21.15.90:0/1635187202 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87341ab250 con 0x7f873410e080 2024-04-02T14:37:18.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.394+0000 7f873b600640 1 -- 172.21.15.90:0/1635187202 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87341ab720 con 0x7f873410e080 2024-04-02T14:37:18.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.396+0000 7f873b600640 1 -- 172.21.15.90:0/1635187202 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8734100ff0 con 0x7f873410e080 2024-04-02T14:37:18.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.399+0000 7f8731600640 1 -- 172.21.15.90:0/1635187202 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f872c01d020 con 0x7f873410e080 2024-04-02T14:37:18.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.401+0000 7f8731600640 1 --2- 172.21.15.90:0/1635187202 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f871c0a3e20 0x7f871c0a62e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:18.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.401+0000 7f8731600640 1 -- 172.21.15.90:0/1635187202 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f872c0d08c0 con 0x7f873410e080 2024-04-02T14:37:18.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.401+0000 7f8738c00640 1 --2- 172.21.15.90:0/1635187202 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f871c0a3e20 0x7f871c0a62e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:18.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.403+0000 7f8738c00640 1 --2- 172.21.15.90:0/1635187202 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f871c0a3e20 0x7f871c0a62e0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f8728005f10 tx=0x7f8728005ea0 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:37:18.406 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.405+0000 7f8731600640 1 -- 172.21.15.90:0/1635187202 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7f872c09cf10 con 0x7f873410e080 2024-04-02T14:37:18.822 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.821+0000 7f873b600640 1 -- 172.21.15.90:0/1635187202 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f873410a7d0 con 0x7f873410e080 2024-04-02T14:37:18.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.822+0000 7f8731600640 1 -- 172.21.15.90:0/1635187202 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11927 (secure 0 0 0) 0x7f872c09c810 con 0x7f873410e080 2024-04-02T14:37:18.824 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-02T14:37:18.824 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14412,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6837/2852160309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6836","nonce":2852160309},{"type":"v1","addr":"172.21.15.90:6837","nonce":2852160309}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14478,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6841/1919376858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6840","nonce":1919376858},{"type":"v1","addr":"172.21.15.90:6841","nonce":1919376858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-02T14:32:22.319083+0000","modified":"2024-04-02T14:34:52.266152+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24590,"mds_1":24560,"mds_2":24547,"mds_3":24539,"mds_4":24523},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24454":{"gid":24454,"name":"l","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6833/1564319431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":1564319431},{"type":"v1","addr":"172.21.15.177:6833","nonce":1564319431}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.92:6833/1570337372","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":1570337372},{"type":"v1","addr":"172.21.15.92:6833","nonce":1570337372}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"j","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.90:6835/1277124891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6834","nonce":1277124891},{"type":"v1","addr":"172.21.15.90:6835","nonce":1277124891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.92:6835/860721164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":860721164},{"type":"v1","addr":"172.21.15.92:6835","nonce":860721164}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24511":{"gid":24511,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6835/3177244988","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":3177244988},{"type":"v1","addr":"172.21.15.177:6835","nonce":3177244988}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"c","rank":4,"incarnation":25,"state":"up:active","state_seq":9,"addr":"172.21.15.177:6837/4223414858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":4223414858},{"type":"v1","addr":"172.21.15.177:6837","nonce":4223414858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24539":{"gid":24539,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":8,"addr":"172.21.15.92:6837/1270635939","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":1270635939},{"type":"v1","addr":"172.21.15.92:6837","nonce":1270635939}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24547":{"gid":24547,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":7,"addr":"172.21.15.177:6839/664895663","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":664895663},{"type":"v1","addr":"172.21.15.177:6839","nonce":664895663}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"g","rank":1,"incarnation":19,"state":"up:active","state_seq":5,"addr":"172.21.15.90:6839/1498915870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6838","nonce":1498915870},{"type":"v1","addr":"172.21.15.90:6839","nonce":1498915870}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24590":{"gid":24590,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.92:6839/1054888219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":1054888219},{"type":"v1","addr":"172.21.15.92:6839","nonce":1054888219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24590,"qdb_cluster":[24547,24590,24523,24539,24560]},"id":1}]} 2024-04-02T14:37:18.824 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 27 2024-04-02T14:37:18.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.826+0000 7f873b600640 1 -- 172.21.15.90:0/1635187202 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f871c0a3e20 msgr2=0x7f871c0a62e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:18.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.826+0000 7f873b600640 1 --2- 172.21.15.90:0/1635187202 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f871c0a3e20 0x7f871c0a62e0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f8728005f10 tx=0x7f8728005ea0 comp rx=0 tx=0).stop 2024-04-02T14:37:18.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.826+0000 7f873b600640 1 -- 172.21.15.90:0/1635187202 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f873410e080 msgr2=0x7f873410e530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:18.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.826+0000 7f873b600640 1 --2- 172.21.15.90:0/1635187202 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f873410e080 0x7f873410e530 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f872c009fd0 tx=0x7f872c00c800 comp rx=0 tx=0).stop 2024-04-02T14:37:18.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.826+0000 7f873b600640 1 -- 172.21.15.90:0/1635187202 shutdown_connections 2024-04-02T14:37:18.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.826+0000 7f873b600640 1 --2- 172.21.15.90:0/1635187202 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f871c0a3e20 0x7f871c0a62e0 secure :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f8728005f10 tx=0x7f8728005ea0 comp rx=0 tx=0).stop 2024-04-02T14:37:18.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.826+0000 7f873b600640 1 --2- 172.21.15.90:0/1635187202 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f873410e080 0x7f873410e530 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:18.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.826+0000 7f873b600640 1 --2- 172.21.15.90:0/1635187202 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8734109e20 0x7f873410db40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:18.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.826+0000 7f873b600640 1 --2- 172.21.15.90:0/1635187202 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8734100300 0x7f8734110aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:18.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.826+0000 7f873b600640 1 -- 172.21.15.90:0/1635187202 >> 172.21.15.90:0/1635187202 conn(0x7f8734076b60 msgr2=0x7f87341018b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:37:18.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.826+0000 7f873b600640 1 -- 172.21.15.90:0/1635187202 shutdown_connections 2024-04-02T14:37:18.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:18.826+0000 7f873b600640 1 -- 172.21.15.90:0/1635187202 wait complete. 2024-04-02T14:37:18.852 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-04-02T14:37:19.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.139+0000 7f6f95000640 1 -- 172.21.15.90:0/4094099347 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6f90068d00 msgr2=0x7f6f900690e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:19.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.139+0000 7f6f95000640 1 --2- 172.21.15.90:0/4094099347 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6f90068d00 0x7f6f900690e0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f6f88009ca0 tx=0x7f6f8802c180 comp rx=0 tx=0).stop 2024-04-02T14:37:19.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.140+0000 7f6f95000640 1 -- 172.21.15.90:0/4094099347 shutdown_connections 2024-04-02T14:37:19.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.140+0000 7f6f95000640 1 --2- 172.21.15.90:0/4094099347 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6f9010e4a0 0x7f6f90110890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:19.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.140+0000 7f6f95000640 1 --2- 172.21.15.90:0/4094099347 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6f90069620 0x7f6f9010df60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:19.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.140+0000 7f6f95000640 1 --2- 172.21.15.90:0/4094099347 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6f90068d00 0x7f6f900690e0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:19.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.140+0000 7f6f95000640 1 -- 172.21.15.90:0/4094099347 >> 172.21.15.90:0/4094099347 conn(0x7f6f90077b00 msgr2=0x7f6f90077f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:37:19.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.141+0000 7f6f95000640 1 -- 172.21.15.90:0/4094099347 shutdown_connections 2024-04-02T14:37:19.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.141+0000 7f6f95000640 1 -- 172.21.15.90:0/4094099347 wait complete. 2024-04-02T14:37:19.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.142+0000 7f6f95000640 1 Processor -- start 2024-04-02T14:37:19.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.143+0000 7f6f95000640 1 -- start start 2024-04-02T14:37:19.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.143+0000 7f6f95000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6f90068d00 0x7f6f9010db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:19.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.144+0000 7f6f95000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6f90069620 0x7f6f9010bc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:19.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.144+0000 7f6f95000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6f9010e4a0 0x7f6f9010c170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:19.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.144+0000 7f6f95000640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f6f901030d0 con 0x7f6f90069620 2024-04-02T14:37:19.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.144+0000 7f6f95000640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f6f90102f50 con 0x7f6f90068d00 2024-04-02T14:37:19.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.144+0000 7f6f95000640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f6f90103250 con 0x7f6f9010e4a0 2024-04-02T14:37:19.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.144+0000 7f6f8de00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6f90068d00 0x7f6f9010db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:19.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.144+0000 7f6f8de00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6f90068d00 0x7f6f9010db30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:48614/0 (socket says 172.21.15.90:48614) 2024-04-02T14:37:19.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.144+0000 7f6f8de00640 1 -- 172.21.15.90:0/2160588634 learned_addr learned my addr 172.21.15.90:0/2160588634 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:37:19.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.144+0000 7f6f8e800640 1 --2- 172.21.15.90:0/2160588634 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6f9010e4a0 0x7f6f9010c170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:19.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.144+0000 7f6f8d400640 1 --2- 172.21.15.90:0/2160588634 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6f90069620 0x7f6f9010bc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:19.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.145+0000 7f6f8de00640 1 -- 172.21.15.90:0/2160588634 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6f9010e4a0 msgr2=0x7f6f9010c170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:19.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.145+0000 7f6f8de00640 1 --2- 172.21.15.90:0/2160588634 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6f9010e4a0 0x7f6f9010c170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:19.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.145+0000 7f6f8de00640 1 -- 172.21.15.90:0/2160588634 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6f90069620 msgr2=0x7f6f9010bc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:19.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.145+0000 7f6f8de00640 1 --2- 172.21.15.90:0/2160588634 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6f90069620 0x7f6f9010bc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:19.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.145+0000 7f6f8de00640 1 -- 172.21.15.90:0/2160588634 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f9010c810 con 0x7f6f90068d00 2024-04-02T14:37:19.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.145+0000 7f6f8de00640 1 --2- 172.21.15.90:0/2160588634 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6f90068d00 0x7f6f9010db30 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f6f88005e30 tx=0x7f6f88035ed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:37:19.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.146+0000 7f6f86a00640 1 -- 172.21.15.90:0/2160588634 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f88039040 con 0x7f6f90068d00 2024-04-02T14:37:19.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.146+0000 7f6f86a00640 1 -- 172.21.15.90:0/2160588634 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6f88044070 con 0x7f6f90068d00 2024-04-02T14:37:19.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.146+0000 7f6f86a00640 1 -- 172.21.15.90:0/2160588634 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f88032a90 con 0x7f6f90068d00 2024-04-02T14:37:19.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.146+0000 7f6f95000640 1 -- 172.21.15.90:0/2160588634 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f901ab030 con 0x7f6f90068d00 2024-04-02T14:37:19.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.147+0000 7f6f95000640 1 -- 172.21.15.90:0/2160588634 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f901ab3e0 con 0x7f6f90068d00 2024-04-02T14:37:19.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.152+0000 7f6f86a00640 1 -- 172.21.15.90:0/2160588634 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f6f88032c30 con 0x7f6f90068d00 2024-04-02T14:37:19.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.156+0000 7f6f97dde740 1 Processor -- start 2024-04-02T14:37:19.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.156+0000 7f6f97dde740 1 -- start start 2024-04-02T14:37:19.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.156+0000 7f6f86a00640 1 --2- 172.21.15.90:0/2160588634 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f6f740a3da0 0x7f6f740a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:19.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.157+0000 7f6f86a00640 1 -- 172.21.15.90:0/2160588634 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f6f880f2870 con 0x7f6f90068d00 2024-04-02T14:37:19.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.158+0000 7f6f8e800640 1 --2- 172.21.15.90:0/2160588634 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f6f740a3da0 0x7f6f740a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:19.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.159+0000 7f6f97dde740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5636d09429f0 0x5636d0941740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:19.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.160+0000 7f6f97dde740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5636d0941d10 0x5636d0942190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:19.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.161+0000 7f6f97dde740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5636d094b080 0x5636d094d470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:19.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.161+0000 7f6f97dde740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5636d0882400 con 0x5636d094b080 2024-04-02T14:37:19.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.161+0000 7f6f97dde740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x5636d0866f10 con 0x5636d0941d10 2024-04-02T14:37:19.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.161+0000 7f6f97dde740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5636d08a66f0 con 0x5636d09429f0 2024-04-02T14:37:19.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.160+0000 7f6f8e800640 1 --2- 172.21.15.90:0/2160588634 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f6f740a3da0 0x7f6f740a6260 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f6f80005fd0 tx=0x7f6f80005cc0 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:37:19.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.161+0000 7f6f8d400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5636d0941d10 0x5636d0942190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:19.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.161+0000 7f6f8d400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5636d0941d10 0x5636d0942190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:48624/0 (socket says 172.21.15.90:48624) 2024-04-02T14:37:19.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.161+0000 7f6f8d400640 1 -- 172.21.15.90:0/1291474685 learned_addr learned my addr 172.21.15.90:0/1291474685 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:37:19.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.161+0000 7f6f8de00640 1 --2- 172.21.15.90:0/1291474685 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5636d09429f0 0x5636d0941740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:19.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.161+0000 7f6f8e800640 1 --2- 172.21.15.90:0/1291474685 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5636d094b080 0x5636d094d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:19.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.161+0000 7f6f8d400640 1 -- 172.21.15.90:0/1291474685 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5636d09429f0 msgr2=0x5636d0941740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:37:19.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.162+0000 7f6f8d400640 1 --2- 172.21.15.90:0/1291474685 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5636d09429f0 0x5636d0941740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:19.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.162+0000 7f6f8d400640 1 -- 172.21.15.90:0/1291474685 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5636d094b080 msgr2=0x5636d094d470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:37:19.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.162+0000 7f6f8d400640 1 --2- 172.21.15.90:0/1291474685 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5636d094b080 0x5636d094d470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:19.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.162+0000 7f6f8d400640 1 -- 172.21.15.90:0/1291474685 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5636d05ba3f0 con 0x5636d0941d10 2024-04-02T14:37:19.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.162+0000 7f6f8e800640 1 --2- 172.21.15.90:0/1291474685 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5636d094b080 0x5636d094d470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:37:19.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.162+0000 7f6f8de00640 1 --2- 172.21.15.90:0/1291474685 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5636d09429f0 0x5636d0941740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:37:19.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.163+0000 7f6f8d400640 1 --2- 172.21.15.90:0/1291474685 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5636d0941d10 0x5636d0942190 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f6f7800cb90 tx=0x7f6f7800e720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2ff23246cbbd921 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:37:19.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.163+0000 7f6f7f400640 1 -- 172.21.15.90:0/1291474685 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f78010630 con 0x5636d0941d10 2024-04-02T14:37:19.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.163+0000 7f6f7f400640 1 -- 172.21.15.90:0/1291474685 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6f78010cb0 con 0x5636d0941d10 2024-04-02T14:37:19.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.164+0000 7f6f97dde740 1 -- 172.21.15.90:0/1291474685 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5636d0941d10 msgr2=0x5636d0942190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:19.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.164+0000 7f6f97dde740 1 --2- 172.21.15.90:0/1291474685 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5636d0941d10 0x5636d0942190 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f6f7800cb90 tx=0x7f6f7800e720 comp rx=0 tx=0).stop 2024-04-02T14:37:19.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.164+0000 7f6f97dde740 1 -- 172.21.15.90:0/1291474685 shutdown_connections 2024-04-02T14:37:19.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.164+0000 7f6f97dde740 1 --2- 172.21.15.90:0/1291474685 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5636d094b080 0x5636d094d470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:19.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.164+0000 7f6f97dde740 1 --2- 172.21.15.90:0/1291474685 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5636d0941d10 0x5636d0942190 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:19.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.164+0000 7f6f97dde740 1 --2- 172.21.15.90:0/1291474685 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5636d09429f0 0x5636d0941740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:19.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.164+0000 7f6f97dde740 1 -- 172.21.15.90:0/1291474685 >> 172.21.15.90:0/1291474685 conn(0x5636d0908a50 msgr2=0x5636d093c760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:37:19.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.164+0000 7f6f97dde740 1 -- 172.21.15.90:0/1291474685 shutdown_connections 2024-04-02T14:37:19.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.164+0000 7f6f97dde740 1 -- 172.21.15.90:0/1291474685 wait complete. 2024-04-02T14:37:19.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.165+0000 7f6f97dde740 1 -- start start 2024-04-02T14:37:19.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.165+0000 7f6f97dde740 1 Processor -- start 2024-04-02T14:37:19.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.166+0000 7f6f97dde740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5636d0941d10 0x5636d0947f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:19.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.166+0000 7f6f97dde740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5636d09429f0 0x5636d0948470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:19.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.166+0000 7f6f97dde740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5636d094b080 0x5636d09489b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:19.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.167+0000 7f6f8d400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5636d09429f0 0x5636d0948470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:19.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.167+0000 7f6f8d400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5636d09429f0 0x5636d0948470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:58158/0 (socket says 172.21.15.90:58158) 2024-04-02T14:37:19.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.167+0000 7f6f8d400640 1 -- 172.21.15.90:0/428075279 learned_addr learned my addr 172.21.15.90:0/428075279 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:37:19.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.167+0000 7f6f97dde740 1 -- 172.21.15.90:0/428075279 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5636d08a26a0 con 0x5636d094b080 2024-04-02T14:37:19.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.167+0000 7f6f97dde740 1 -- 172.21.15.90:0/428075279 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x5636d084e980 con 0x5636d0941d10 2024-04-02T14:37:19.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.167+0000 7f6f97dde740 1 -- 172.21.15.90:0/428075279 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5636d0861cb0 con 0x5636d09429f0 2024-04-02T14:37:19.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.167+0000 7f6f97dde740 10 client.0 _ll_register_callbacks cb 0x5636d08616a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:37:19.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.167+0000 7f6f8de00640 1 --2- 172.21.15.90:0/428075279 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5636d0941d10 0x5636d0947f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:19.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.167+0000 7f6f8e800640 1 --2- 172.21.15.90:0/428075279 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5636d094b080 0x5636d09489b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:19.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.167+0000 7f6f8de00640 1 -- 172.21.15.90:0/428075279 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5636d09429f0 msgr2=0x5636d0948470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:37:19.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.167+0000 7f6f8de00640 1 --2- 172.21.15.90:0/428075279 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5636d09429f0 0x5636d0948470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:19.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.168+0000 7f6f8de00640 1 -- 172.21.15.90:0/428075279 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5636d094b080 msgr2=0x5636d09489b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:37:19.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.170+0000 7f6f8de00640 1 --2- 172.21.15.90:0/428075279 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5636d094b080 0x5636d09489b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:19.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.170+0000 7f6f8de00640 1 -- 172.21.15.90:0/428075279 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5636d06847d0 con 0x5636d0941d10 2024-04-02T14:37:19.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.170+0000 7f6f8e800640 1 --2- 172.21.15.90:0/428075279 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5636d094b080 0x5636d09489b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:37:19.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.171+0000 7f6f73e00640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:37:19.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.171+0000 7f6f8de00640 1 --2- 172.21.15.90:0/428075279 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5636d0941d10 0x5636d0947f30 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f6f880f00e0 tx=0x7f6f88032fc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b281082ea07eed3b server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:37:19.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.172+0000 7f6f73e00640 1 -- 172.21.15.90:0/428075279 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f880bf030 con 0x5636d0941d10 2024-04-02T14:37:19.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.172+0000 7f6f97dde740 1 -- 172.21.15.90:0/428075279 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6f78010cb0 con 0x5636d0941d10 2024-04-02T14:37:19.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.172+0000 7f6f73e00640 1 -- 172.21.15.90:0/428075279 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f6f880a90b0 con 0x5636d0941d10 2024-04-02T14:37:19.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.173+0000 7f6f73e00640 1 -- 172.21.15.90:0/428075279 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f6f880bf030 con 0x5636d0941d10 2024-04-02T14:37:19.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.173+0000 7f6f97dde740 10 client.26135 fetch_fsmap learned FSMap version 27 2024-04-02T14:37:19.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.173+0000 7f6f97dde740 1 -- 172.21.15.90:0/428075279 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5636d0959eb0 con 0x5636d0941d10 2024-04-02T14:37:19.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.173+0000 7f6f73e00640 1 -- 172.21.15.90:0/428075279 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6f78010cb0 con 0x5636d0941d10 2024-04-02T14:37:19.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.174+0000 7f6f73e00640 1 -- 172.21.15.90:0/428075279 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f6f8803c030 con 0x5636d0941d10 2024-04-02T14:37:19.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.174+0000 7f6f97dde740 10 client.26135 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:37:19.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.176+0000 7f6f97dde740 10 client.26135 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.c 2024-04-02T14:37:19.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.176+0000 7f6f97dde740 20 client.26135 populate_metadata read hostname 'smithi090' 2024-04-02T14:37:19.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.176+0000 7f6f97dde740 1 --2- 172.21.15.90:0/428075279 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x5636d095b3b0 0x5636d095d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:19.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.176+0000 7f6f97dde740 4 client.26135 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:37:19.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.176+0000 7f6f97dde740 1 -- 172.21.15.90:0/428075279 --> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5636d0866040 con 0x5636d095b3b0 2024-04-02T14:37:19.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.177+0000 7f6f8d400640 1 --2- 172.21.15.90:0/428075279 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x5636d095b3b0 0x5636d095d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:19.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.181+0000 7f6f73e00640 10 client.26135 ms_handle_connect on v2:172.21.15.177:6836/4223414858 2024-04-02T14:37:19.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.181+0000 7f6f8d400640 1 --2- 172.21.15.90:0/428075279 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x5636d095b3b0 0x5636d095d7c0 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=73c771b886dc986 server_cookie=dbca2e7d7e3f1ab7 in_seq=0 out_seq=0 2024-04-02T14:37:19.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.183+0000 7f6f73e00640 1 -- 172.21.15.90:0/428075279 <== mds.4 v2:172.21.15.177:6836/4223414858 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5636d0866040 con 0x5636d095b3b0 2024-04-02T14:37:19.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.183+0000 7f6f73e00640 10 client.26135 handle_command_reply: tid=0 2024-04-02T14:37:19.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.183+0000 7f6f97dde740 1 client.26135 shutdown 2024-04-02T14:37:19.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.183+0000 7f6f97dde740 1 -- 172.21.15.90:0/428075279 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5636d0941d10 msgr2=0x5636d0947f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:19.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.183+0000 7f6f97dde740 1 --2- 172.21.15.90:0/428075279 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5636d0941d10 0x5636d0947f30 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f6f880f00e0 tx=0x7f6f88032fc0 comp rx=0 tx=0).stop 2024-04-02T14:37:19.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.184+0000 7f6f97dde740 1 -- 172.21.15.90:0/428075279 shutdown_connections 2024-04-02T14:37:19.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.184+0000 7f6f97dde740 1 --2- 172.21.15.90:0/428075279 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x5636d095b3b0 0x5636d095d7c0 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:19.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.184+0000 7f6f97dde740 1 --2- 172.21.15.90:0/428075279 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5636d094b080 0x5636d09489b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:19.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.184+0000 7f6f97dde740 1 --2- 172.21.15.90:0/428075279 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5636d09429f0 0x5636d0948470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:19.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.184+0000 7f6f97dde740 1 --2- 172.21.15.90:0/428075279 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5636d0941d10 0x5636d0947f30 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:19.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.184+0000 7f6f97dde740 1 -- 172.21.15.90:0/428075279 >> 172.21.15.90:0/428075279 conn(0x5636d0908a50 msgr2=0x5636d093e4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:37:19.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.184+0000 7f6f73e00640 0 client.26135 ms_handle_reset on v2:172.21.15.177:6836/4223414858 2024-04-02T14:37:19.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.185+0000 7f6f97dde740 1 -- 172.21.15.90:0/428075279 shutdown_connections 2024-04-02T14:37:19.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.185+0000 7f6f97dde740 1 -- 172.21.15.90:0/428075279 wait complete. 2024-04-02T14:37:19.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.185+0000 7f6f97dde740 20 client.26135 trim_cache size 0 max 16384 2024-04-02T14:37:19.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.221+0000 7f6f97dde740 1 Processor -- start 2024-04-02T14:37:19.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.221+0000 7f6f97dde740 1 -- start start 2024-04-02T14:37:19.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.222+0000 7f6f97dde740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5636d09429f0 0x5636d094a660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:19.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.222+0000 7f6f97dde740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5636d094b080 0x5636d0956c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:19.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.222+0000 7f6f97dde740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5636d093c290 0x5636d0947090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:19.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.222+0000 7f6f97dde740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f6f901b3ce0 con 0x5636d093c290 2024-04-02T14:37:19.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.222+0000 7f6f97dde740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f6f901b5820 con 0x5636d094b080 2024-04-02T14:37:19.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.222+0000 7f6f97dde740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f6f901b43b0 con 0x5636d09429f0 2024-04-02T14:37:19.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.222+0000 7f6f8de00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5636d09429f0 0x5636d094a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:19.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.222+0000 7f6f8d400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5636d094b080 0x5636d0956c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:19.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.222+0000 7f6f8de00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5636d09429f0 0x5636d094a660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:58174/0 (socket says 172.21.15.90:58174) 2024-04-02T14:37:19.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.222+0000 7f6f8d400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5636d094b080 0x5636d0956c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:48640/0 (socket says 172.21.15.90:48640) 2024-04-02T14:37:19.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.222+0000 7f6f8de00640 1 -- 172.21.15.90:0/3294345647 learned_addr learned my addr 172.21.15.90:0/3294345647 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:37:19.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.222+0000 7f6f8e800640 1 --2- 172.21.15.90:0/3294345647 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5636d093c290 0x5636d0947090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:19.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.222+0000 7f6f8d400640 1 -- 172.21.15.90:0/3294345647 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5636d09429f0 msgr2=0x5636d094a660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:37:19.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.222+0000 7f6f8d400640 1 --2- 172.21.15.90:0/3294345647 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5636d09429f0 0x5636d094a660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:19.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.222+0000 7f6f8d400640 1 -- 172.21.15.90:0/3294345647 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5636d093c290 msgr2=0x5636d0947090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:37:19.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.222+0000 7f6f8d400640 1 --2- 172.21.15.90:0/3294345647 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5636d093c290 0x5636d0947090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:19.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.222+0000 7f6f8d400640 1 -- 172.21.15.90:0/3294345647 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5636d093c970 con 0x5636d094b080 2024-04-02T14:37:19.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.223+0000 7f6f8d400640 1 --2- 172.21.15.90:0/3294345647 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5636d094b080 0x5636d0956c40 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f6f7801a880 tx=0x7f6f7800c480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=80e8831dbe504ecc server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:37:19.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.223+0000 7f6f7ea00640 1 -- 172.21.15.90:0/3294345647 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f7801a410 con 0x5636d094b080 2024-04-02T14:37:19.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.223+0000 7f6f7ea00640 1 -- 172.21.15.90:0/3294345647 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6f7801a5b0 con 0x5636d094b080 2024-04-02T14:37:19.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.223+0000 7f6f7ea00640 1 -- 172.21.15.90:0/3294345647 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f78005040 con 0x5636d094b080 2024-04-02T14:37:19.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.224+0000 7f6f97dde740 1 -- 172.21.15.90:0/3294345647 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5636d094b080 msgr2=0x5636d0956c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:19.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.224+0000 7f6f97dde740 1 --2- 172.21.15.90:0/3294345647 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5636d094b080 0x5636d0956c40 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f6f7801a880 tx=0x7f6f7800c480 comp rx=0 tx=0).stop 2024-04-02T14:37:19.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.226+0000 7f6f97dde740 1 -- 172.21.15.90:0/3294345647 shutdown_connections 2024-04-02T14:37:19.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.226+0000 7f6f97dde740 1 --2- 172.21.15.90:0/3294345647 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5636d093c290 0x5636d0947090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:19.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.226+0000 7f6f97dde740 1 --2- 172.21.15.90:0/3294345647 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5636d094b080 0x5636d0956c40 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:19.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.226+0000 7f6f97dde740 1 --2- 172.21.15.90:0/3294345647 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5636d09429f0 0x5636d094a660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:19.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.226+0000 7f6f97dde740 1 -- 172.21.15.90:0/3294345647 >> 172.21.15.90:0/3294345647 conn(0x5636d0908a50 msgr2=0x5636d093dbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:37:19.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.226+0000 7f6f97dde740 1 -- 172.21.15.90:0/3294345647 shutdown_connections 2024-04-02T14:37:19.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.226+0000 7f6f97dde740 1 -- 172.21.15.90:0/3294345647 wait complete. 2024-04-02T14:37:19.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.227+0000 7f6f97dde740 1 -- start start 2024-04-02T14:37:19.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.228+0000 7f6f97dde740 1 Processor -- start 2024-04-02T14:37:19.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.229+0000 7f6f97dde740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5636d093c290 0x5636d0944c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:19.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.229+0000 7f6f97dde740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5636d09429f0 0x5636d0945180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:19.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.230+0000 7f6f97dde740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5636d094b080 0x5636d09456c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:19.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.230+0000 7f6f97dde740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f6f901b5150 con 0x5636d093c290 2024-04-02T14:37:19.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.230+0000 7f6f97dde740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f6f901b4a80 con 0x5636d09429f0 2024-04-02T14:37:19.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.230+0000 7f6f97dde740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f6f901b1cc0 con 0x5636d094b080 2024-04-02T14:37:19.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.230+0000 7f6f97dde740 10 client.0 _ll_register_callbacks cb 0x7f6f880bac30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:37:19.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.230+0000 7f6f8d400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5636d09429f0 0x5636d0945180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:19.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.230+0000 7f6f8d400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5636d09429f0 0x5636d0945180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:48652/0 (socket says 172.21.15.90:48652) 2024-04-02T14:37:19.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.230+0000 7f6f8d400640 1 -- 172.21.15.90:0/3675593621 learned_addr learned my addr 172.21.15.90:0/3675593621 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:37:19.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.230+0000 7f6f8d400640 1 -- 172.21.15.90:0/3675593621 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5636d094b080 msgr2=0x5636d09456c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-02T14:37:19.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.230+0000 7f6f8d400640 1 --2- 172.21.15.90:0/3675593621 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5636d094b080 0x5636d09456c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:19.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.230+0000 7f6f8d400640 1 -- 172.21.15.90:0/3675593621 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5636d093c290 msgr2=0x5636d0944c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:37:19.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.230+0000 7f6f8d400640 1 --2- 172.21.15.90:0/3675593621 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5636d093c290 0x5636d0944c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:19.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.230+0000 7f6f8d400640 1 -- 172.21.15.90:0/3675593621 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5636d09c0950 con 0x5636d09429f0 2024-04-02T14:37:19.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.231+0000 7f6f8d400640 1 --2- 172.21.15.90:0/3675593621 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5636d09429f0 0x5636d0945180 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f6f7800cab0 tx=0x7f6f7801a7d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=40504fb0afffa316 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:37:19.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.231+0000 7f6f73e00640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:37:19.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.231+0000 7f6f73e00640 1 -- 172.21.15.90:0/3675593621 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f78033060 con 0x5636d09429f0 2024-04-02T14:37:19.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.231+0000 7f6f73e00640 1 -- 172.21.15.90:0/3675593621 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f6f780263c0 con 0x5636d09429f0 2024-04-02T14:37:19.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.231+0000 7f6f97dde740 1 -- 172.21.15.90:0/3675593621 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6f7801a5b0 con 0x5636d09429f0 2024-04-02T14:37:19.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.232+0000 7f6f73e00640 1 -- 172.21.15.90:0/3675593621 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f6f7801a410 con 0x5636d09429f0 2024-04-02T14:37:19.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.232+0000 7f6f97dde740 10 client.26141 fetch_fsmap learned FSMap version 27 2024-04-02T14:37:19.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.232+0000 7f6f97dde740 1 -- 172.21.15.90:0/3675593621 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5636d09c1c50 con 0x5636d09429f0 2024-04-02T14:37:19.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.232+0000 7f6f73e00640 1 -- 172.21.15.90:0/3675593621 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6f7801a5b0 con 0x5636d09429f0 2024-04-02T14:37:19.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.233+0000 7f6f73e00640 1 -- 172.21.15.90:0/3675593621 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f6f78004710 con 0x5636d09429f0 2024-04-02T14:37:19.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.233+0000 7f6f97dde740 10 client.26141 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:37:19.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.233+0000 7f6f97dde740 10 client.26141 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.c 2024-04-02T14:37:19.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.233+0000 7f6f97dde740 20 client.26141 populate_metadata read hostname 'smithi090' 2024-04-02T14:37:19.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.233+0000 7f6f97dde740 1 --2- 172.21.15.90:0/3675593621 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x5636d09c20d0 0x5636d09c44f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:19.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.233+0000 7f6f97dde740 4 client.26141 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-02T14:37:19.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.233+0000 7f6f97dde740 1 -- 172.21.15.90:0/3675593621 --> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5636d08b3c30 con 0x5636d09c20d0 2024-04-02T14:37:19.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.233+0000 7f6f8de00640 1 --2- 172.21.15.90:0/3675593621 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x5636d09c20d0 0x5636d09c44f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:19.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.234+0000 7f6f73e00640 10 client.26141 ms_handle_connect on v2:172.21.15.177:6836/4223414858 2024-04-02T14:37:19.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.234+0000 7f6f8de00640 1 --2- 172.21.15.90:0/3675593621 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x5636d09c20d0 0x5636d09c44f0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=6aaa82e2964572a1 server_cookie=bafda156d6e2f525 in_seq=0 out_seq=0 2024-04-02T14:37:19.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.235+0000 7f6f73e00640 1 -- 172.21.15.90:0/3675593621 <== mds.4 v2:172.21.15.177:6836/4223414858 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5636d08b3c30 con 0x5636d09c20d0 2024-04-02T14:37:19.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.235+0000 7f6f73e00640 10 client.26141 handle_command_reply: tid=0 2024-04-02T14:37:19.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.235+0000 7f6f97dde740 1 client.26141 shutdown 2024-04-02T14:37:19.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.235+0000 7f6f97dde740 1 -- 172.21.15.90:0/3675593621 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5636d09429f0 msgr2=0x5636d0945180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:19.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.235+0000 7f6f97dde740 1 --2- 172.21.15.90:0/3675593621 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5636d09429f0 0x5636d0945180 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f6f7800cab0 tx=0x7f6f7801a7d0 comp rx=0 tx=0).stop 2024-04-02T14:37:19.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.236+0000 7f6f97dde740 1 -- 172.21.15.90:0/3675593621 shutdown_connections 2024-04-02T14:37:19.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.236+0000 7f6f97dde740 1 --2- 172.21.15.90:0/3675593621 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5636d094b080 0x5636d09456c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:19.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.236+0000 7f6f97dde740 1 --2- 172.21.15.90:0/3675593621 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5636d09429f0 0x5636d0945180 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:19.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.236+0000 7f6f97dde740 1 --2- 172.21.15.90:0/3675593621 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x5636d09c20d0 0x5636d09c44f0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:19.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.236+0000 7f6f97dde740 1 --2- 172.21.15.90:0/3675593621 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5636d093c290 0x5636d0944c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:19.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.236+0000 7f6f97dde740 1 -- 172.21.15.90:0/3675593621 >> 172.21.15.90:0/3675593621 conn(0x5636d0908a50 msgr2=0x5636d093b010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:37:19.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.236+0000 7f6f73e00640 0 client.26141 ms_handle_reset on v2:172.21.15.177:6836/4223414858 2024-04-02T14:37:19.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.236+0000 7f6f97dde740 1 -- 172.21.15.90:0/3675593621 shutdown_connections 2024-04-02T14:37:19.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.237+0000 7f6f97dde740 1 -- 172.21.15.90:0/3675593621 wait complete. 2024-04-02T14:37:19.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.237+0000 7f6f97dde740 20 client.26141 trim_cache size 0 max 16384 2024-04-02T14:37:19.240 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-02T14:37:19.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.238+0000 7f6f95000640 1 -- 172.21.15.90:0/2160588634 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f6f740a3da0 msgr2=0x7f6f740a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:19.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.239+0000 7f6f95000640 1 --2- 172.21.15.90:0/2160588634 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f6f740a3da0 0x7f6f740a6260 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f6f80005fd0 tx=0x7f6f80005cc0 comp rx=0 tx=0).stop 2024-04-02T14:37:19.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.239+0000 7f6f95000640 1 -- 172.21.15.90:0/2160588634 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6f90068d00 msgr2=0x7f6f9010db30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:19.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.239+0000 7f6f95000640 1 --2- 172.21.15.90:0/2160588634 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6f90068d00 0x7f6f9010db30 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f6f88005e30 tx=0x7f6f88035ed0 comp rx=0 tx=0).stop 2024-04-02T14:37:19.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.239+0000 7f6f95000640 1 -- 172.21.15.90:0/2160588634 shutdown_connections 2024-04-02T14:37:19.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.239+0000 7f6f95000640 1 --2- 172.21.15.90:0/2160588634 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f6f740a3da0 0x7f6f740a6260 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:19.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.239+0000 7f6f95000640 1 --2- 172.21.15.90:0/2160588634 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6f9010e4a0 0x7f6f9010c170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:19.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.239+0000 7f6f95000640 1 --2- 172.21.15.90:0/2160588634 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6f90069620 0x7f6f9010bc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:19.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.239+0000 7f6f95000640 1 --2- 172.21.15.90:0/2160588634 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6f90068d00 0x7f6f9010db30 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:19.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.239+0000 7f6f95000640 1 -- 172.21.15.90:0/2160588634 >> 172.21.15.90:0/2160588634 conn(0x7f6f90077b00 msgr2=0x7f6f9010fa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:37:19.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.239+0000 7f6f95000640 1 -- 172.21.15.90:0/2160588634 shutdown_connections 2024-04-02T14:37:19.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.239+0000 7f6f95000640 1 -- 172.21.15.90:0/2160588634 wait complete. 2024-04-02T14:37:19.267 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-04-02T14:37:19.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.536+0000 7f0acf600640 1 -- 172.21.15.90:0/3010322319 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0ac8069620 msgr2=0x7f0ac810df60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:19.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.536+0000 7f0acf600640 1 --2- 172.21.15.90:0/3010322319 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0ac8069620 0x7f0ac810df60 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f0ab4009900 tx=0x7f0ab402bcc0 comp rx=0 tx=0).stop 2024-04-02T14:37:19.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.536+0000 7f0acf600640 1 -- 172.21.15.90:0/3010322319 shutdown_connections 2024-04-02T14:37:19.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.536+0000 7f0acf600640 1 --2- 172.21.15.90:0/3010322319 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0ac810e4a0 0x7f0ac8110890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:19.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.536+0000 7f0acf600640 1 --2- 172.21.15.90:0/3010322319 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0ac8069620 0x7f0ac810df60 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:19.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.536+0000 7f0acf600640 1 --2- 172.21.15.90:0/3010322319 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0ac8068d00 0x7f0ac80690e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:19.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.536+0000 7f0acf600640 1 -- 172.21.15.90:0/3010322319 >> 172.21.15.90:0/3010322319 conn(0x7f0ac8077b00 msgr2=0x7f0ac8077f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:37:19.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.537+0000 7f0acf600640 1 -- 172.21.15.90:0/3010322319 shutdown_connections 2024-04-02T14:37:19.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.537+0000 7f0acf600640 1 -- 172.21.15.90:0/3010322319 wait complete. 2024-04-02T14:37:19.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.538+0000 7f0acf600640 1 Processor -- start 2024-04-02T14:37:19.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.539+0000 7f0acf600640 1 -- start start 2024-04-02T14:37:19.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.539+0000 7f0acf600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0ac8068d00 0x7f0ac8073270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:19.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.539+0000 7f0acf600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0ac8069620 0x7f0ac80737b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:19.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.539+0000 7f0acf600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0ac810e4a0 0x7f0ac806c4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:19.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.539+0000 7f0acf600640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f0ac81140a0 con 0x7f0ac8068d00 2024-04-02T14:37:19.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.539+0000 7f0acf600640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f0ac8113f20 con 0x7f0ac810e4a0 2024-04-02T14:37:19.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.539+0000 7f0acf600640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f0ac8114220 con 0x7f0ac8069620 2024-04-02T14:37:19.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.540+0000 7f0ac7e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0ac8069620 0x7f0ac80737b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:19.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.540+0000 7f0acd600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0ac810e4a0 0x7f0ac806c4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:19.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.540+0000 7f0acd600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0ac810e4a0 0x7f0ac806c4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:48668/0 (socket says 172.21.15.90:48668) 2024-04-02T14:37:19.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.540+0000 7f0ac7e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0ac8069620 0x7f0ac80737b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:58192/0 (socket says 172.21.15.90:58192) 2024-04-02T14:37:19.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.540+0000 7f0acd600640 1 -- 172.21.15.90:0/2583082185 learned_addr learned my addr 172.21.15.90:0/2583082185 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:37:19.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.540+0000 7f0accc00640 1 --2- 172.21.15.90:0/2583082185 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0ac8068d00 0x7f0ac8073270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:19.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.540+0000 7f0acd600640 1 -- 172.21.15.90:0/2583082185 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0ac8069620 msgr2=0x7f0ac80737b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:19.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.540+0000 7f0acd600640 1 --2- 172.21.15.90:0/2583082185 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0ac8069620 0x7f0ac80737b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:19.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.540+0000 7f0acd600640 1 -- 172.21.15.90:0/2583082185 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0ac8068d00 msgr2=0x7f0ac8073270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:19.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.540+0000 7f0acd600640 1 --2- 172.21.15.90:0/2583082185 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0ac8068d00 0x7f0ac8073270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:19.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.540+0000 7f0acd600640 1 -- 172.21.15.90:0/2583082185 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ac806cd90 con 0x7f0ac810e4a0 2024-04-02T14:37:19.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.540+0000 7f0ac7e00640 1 --2- 172.21.15.90:0/2583082185 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0ac8069620 0x7f0ac80737b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello state changed while learned_addr, mark_down or replacing must be happened just now 2024-04-02T14:37:19.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.541+0000 7f0acd600640 1 --2- 172.21.15.90:0/2583082185 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0ac810e4a0 0x7f0ac806c4d0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f0ac000eaa0 tx=0x7f0ac000ef60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:37:19.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.544+0000 7f0ac5600640 1 -- 172.21.15.90:0/2583082185 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ac000cdb0 con 0x7f0ac810e4a0 2024-04-02T14:37:19.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.544+0000 7f0ac5600640 1 -- 172.21.15.90:0/2583082185 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0ac0005020 con 0x7f0ac810e4a0 2024-04-02T14:37:19.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.544+0000 7f0ac5600640 1 -- 172.21.15.90:0/2583082185 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ac00116b0 con 0x7f0ac810e4a0 2024-04-02T14:37:19.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.544+0000 7f0acf600640 1 -- 172.21.15.90:0/2583082185 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ac806d080 con 0x7f0ac810e4a0 2024-04-02T14:37:19.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.544+0000 7f0acf600640 1 -- 172.21.15.90:0/2583082185 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ac8071030 con 0x7f0ac810e4a0 2024-04-02T14:37:19.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.550+0000 7f0ad2246740 1 Processor -- start 2024-04-02T14:37:19.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.550+0000 7f0ac5600640 1 -- 172.21.15.90:0/2583082185 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f0ac0011850 con 0x7f0ac810e4a0 2024-04-02T14:37:19.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.551+0000 7f0ad2246740 1 -- start start 2024-04-02T14:37:19.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.551+0000 7f0ad2246740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562c4e1d29f0 0x562c4e1d1740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:19.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.551+0000 7f0ad2246740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562c4e1d1d10 0x562c4e1d2190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:19.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.551+0000 7f0ad2246740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x562c4e1db080 0x562c4e1dd470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:19.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.551+0000 7f0ad2246740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x562c4e112400 con 0x562c4e1d29f0 2024-04-02T14:37:19.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.551+0000 7f0ad2246740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x562c4e0f6f10 con 0x562c4e1db080 2024-04-02T14:37:19.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.551+0000 7f0ad2246740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x562c4e1366f0 con 0x562c4e1d1d10 2024-04-02T14:37:19.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.552+0000 7f0accc00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562c4e1d1d10 0x562c4e1d2190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:19.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.552+0000 7f0accc00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562c4e1d1d10 0x562c4e1d2190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:58194/0 (socket says 172.21.15.90:58194) 2024-04-02T14:37:19.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.552+0000 7f0accc00640 1 -- 172.21.15.90:0/3519699138 learned_addr learned my addr 172.21.15.90:0/3519699138 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:37:19.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.552+0000 7f0ac7e00640 1 --2- 172.21.15.90:0/3519699138 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562c4e1d29f0 0x562c4e1d1740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:19.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.553+0000 7f0accc00640 1 -- 172.21.15.90:0/3519699138 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x562c4e1db080 msgr2=0x562c4e1dd470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:37:19.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.553+0000 7f0accc00640 1 --2- 172.21.15.90:0/3519699138 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x562c4e1db080 0x562c4e1dd470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:19.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.553+0000 7f0accc00640 1 -- 172.21.15.90:0/3519699138 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562c4e1d29f0 msgr2=0x562c4e1d1740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:37:19.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.553+0000 7f0ac5600640 1 --2- 172.21.15.90:0/2583082185 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f0aac0a3e70 0x7f0aac0a6330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:19.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.553+0000 7f0ac5600640 1 -- 172.21.15.90:0/2583082185 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f0ac00d0970 con 0x7f0ac810e4a0 2024-04-02T14:37:19.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.553+0000 7f0accc00640 1 --2- 172.21.15.90:0/3519699138 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562c4e1d29f0 0x562c4e1d1740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:19.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.554+0000 7f0accc00640 1 -- 172.21.15.90:0/3519699138 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562c4de4a3f0 con 0x562c4e1d1d10 2024-04-02T14:37:19.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.554+0000 7f0ac7e00640 1 --2- 172.21.15.90:0/3519699138 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562c4e1d29f0 0x562c4e1d1740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:37:19.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.554+0000 7f0accc00640 1 --2- 172.21.15.90:0/3519699138 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562c4e1d1d10 0x562c4e1d2190 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f0ab800cc60 tx=0x7f0ab800e720 comp rx=0 tx=0).ready entity=mon.1 client_cookie=82b888a2b744f81a server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:37:19.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.554+0000 7f0abdc00640 1 -- 172.21.15.90:0/3519699138 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ab8014070 con 0x562c4e1d1d10 2024-04-02T14:37:19.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.554+0000 7f0ac7e00640 1 --2- 172.21.15.90:0/2583082185 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f0aac0a3e70 0x7f0aac0a6330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:19.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.554+0000 7f0abdc00640 1 -- 172.21.15.90:0/3519699138 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0ab80047a0 con 0x562c4e1d1d10 2024-04-02T14:37:19.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.554+0000 7f0abdc00640 1 -- 172.21.15.90:0/3519699138 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ab8010790 con 0x562c4e1d1d10 2024-04-02T14:37:19.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.556+0000 7f0ad2246740 1 -- 172.21.15.90:0/3519699138 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562c4e1d1d10 msgr2=0x562c4e1d2190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:19.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.556+0000 7f0ad2246740 1 --2- 172.21.15.90:0/3519699138 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562c4e1d1d10 0x562c4e1d2190 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f0ab800cc60 tx=0x7f0ab800e720 comp rx=0 tx=0).stop 2024-04-02T14:37:19.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.556+0000 7f0ac7e00640 1 --2- 172.21.15.90:0/2583082185 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f0aac0a3e70 0x7f0aac0a6330 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f0ab4036040 tx=0x7f0ab4009260 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:37:19.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.556+0000 7f0ad2246740 1 -- 172.21.15.90:0/3519699138 shutdown_connections 2024-04-02T14:37:19.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.556+0000 7f0ad2246740 1 --2- 172.21.15.90:0/3519699138 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x562c4e1db080 0x562c4e1dd470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:19.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.556+0000 7f0ad2246740 1 --2- 172.21.15.90:0/3519699138 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562c4e1d1d10 0x562c4e1d2190 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:19.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.556+0000 7f0ad2246740 1 --2- 172.21.15.90:0/3519699138 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562c4e1d29f0 0x562c4e1d1740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:19.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.556+0000 7f0ad2246740 1 -- 172.21.15.90:0/3519699138 >> 172.21.15.90:0/3519699138 conn(0x562c4e198a50 msgr2=0x562c4e1cc760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:37:19.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.557+0000 7f0ad2246740 1 -- 172.21.15.90:0/3519699138 shutdown_connections 2024-04-02T14:37:19.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.557+0000 7f0ad2246740 1 -- 172.21.15.90:0/3519699138 wait complete. 2024-04-02T14:37:19.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.557+0000 7f0ad2246740 1 -- start start 2024-04-02T14:37:19.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.558+0000 7f0ad2246740 1 Processor -- start 2024-04-02T14:37:19.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.559+0000 7f0ad2246740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x562c4e1d1d10 0x562c4e1d8070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:19.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.559+0000 7f0ad2246740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562c4e1d29f0 0x562c4e1d85b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:19.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.559+0000 7f0ad2246740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562c4e1db080 0x562c4e1d8af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:19.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.559+0000 7f0ad2246740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x562c4e1326a0 con 0x562c4e1d29f0 2024-04-02T14:37:19.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.559+0000 7f0ad2246740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x562c4e0de980 con 0x562c4e1d1d10 2024-04-02T14:37:19.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.559+0000 7f0ad2246740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x562c4e0f1cb0 con 0x562c4e1db080 2024-04-02T14:37:19.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.559+0000 7f0ad2246740 10 client.0 _ll_register_callbacks cb 0x562c4e0f16a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:37:19.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.559+0000 7f0accc00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x562c4e1d1d10 0x562c4e1d8070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:19.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.559+0000 7f0accc00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x562c4e1d1d10 0x562c4e1d8070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:48692/0 (socket says 172.21.15.90:48692) 2024-04-02T14:37:19.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.559+0000 7f0accc00640 1 -- 172.21.15.90:0/377204277 learned_addr learned my addr 172.21.15.90:0/377204277 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:37:19.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.559+0000 7f0acd600640 1 --2- 172.21.15.90:0/377204277 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562c4e1db080 0x562c4e1d8af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:19.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.560+0000 7f0accc00640 1 -- 172.21.15.90:0/377204277 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562c4e1db080 msgr2=0x562c4e1d8af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:37:19.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.560+0000 7f0accc00640 1 --2- 172.21.15.90:0/377204277 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562c4e1db080 0x562c4e1d8af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:19.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.560+0000 7f0ac7e00640 1 --2- 172.21.15.90:0/377204277 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562c4e1d29f0 0x562c4e1d85b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:19.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.560+0000 7f0accc00640 1 -- 172.21.15.90:0/377204277 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562c4e1d29f0 msgr2=0x562c4e1d85b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:37:19.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.560+0000 7f0accc00640 1 --2- 172.21.15.90:0/377204277 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562c4e1d29f0 0x562c4e1d85b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:19.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.560+0000 7f0accc00640 1 -- 172.21.15.90:0/377204277 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562c4df147d0 con 0x562c4e1d1d10 2024-04-02T14:37:19.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.560+0000 7f0ac7e00640 1 --2- 172.21.15.90:0/377204277 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562c4e1d29f0 0x562c4e1d85b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:37:19.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.560+0000 7f0accc00640 1 --2- 172.21.15.90:0/377204277 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x562c4e1d1d10 0x562c4e1d8070 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f0ab8000c00 tx=0x7f0ab8005210 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7d0b7ba60599787e server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:37:19.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.560+0000 7f0ab2000640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:37:19.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.561+0000 7f0ab2000640 1 -- 172.21.15.90:0/377204277 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ab8014070 con 0x562c4e1d1d10 2024-04-02T14:37:19.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.561+0000 7f0ab2000640 1 -- 172.21.15.90:0/377204277 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f0ab8004230 con 0x562c4e1d1d10 2024-04-02T14:37:19.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.561+0000 7f0ad2246740 1 -- 172.21.15.90:0/377204277 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0ab80047a0 con 0x562c4e1d1d10 2024-04-02T14:37:19.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.561+0000 7f0ab2000640 1 -- 172.21.15.90:0/377204277 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f0ab8004430 con 0x562c4e1d1d10 2024-04-02T14:37:19.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.561+0000 7f0ad2246740 10 client.26147 fetch_fsmap learned FSMap version 27 2024-04-02T14:37:19.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.561+0000 7f0ad2246740 1 -- 172.21.15.90:0/377204277 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x562c4e1f26a0 con 0x562c4e1d1d10 2024-04-02T14:37:19.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.562+0000 7f0ab2000640 1 -- 172.21.15.90:0/377204277 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0ab8003ad0 con 0x562c4e1d1d10 2024-04-02T14:37:19.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.562+0000 7f0ab2000640 1 -- 172.21.15.90:0/377204277 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f0ab8010740 con 0x562c4e1d1d10 2024-04-02T14:37:19.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.562+0000 7f0ad2246740 10 client.26147 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:37:19.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.562+0000 7f0ad2246740 10 client.26147 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-04-02T14:37:19.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.562+0000 7f0ad2246740 20 client.26147 populate_metadata read hostname 'smithi090' 2024-04-02T14:37:19.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.562+0000 7f0ad2246740 1 --2- 172.21.15.90:0/377204277 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x562c4e1f3aa0 0x562c4e1f5e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:19.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.562+0000 7f0ad2246740 4 client.26147 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:37:19.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.562+0000 7f0ad2246740 1 -- 172.21.15.90:0/377204277 --> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f0ab80047a0 con 0x562c4e1f3aa0 2024-04-02T14:37:19.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.562+0000 7f0ac7e00640 1 --2- 172.21.15.90:0/377204277 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x562c4e1f3aa0 0x562c4e1f5e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:19.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.563+0000 7f0ac7e00640 1 --2- 172.21.15.90:0/377204277 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x562c4e1f3aa0 0x562c4e1f5e90 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=8e1c700f21926ea6 server_cookie=d1fbd72591e7c938 in_seq=0 out_seq=0 2024-04-02T14:37:19.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.563+0000 7f0ab2000640 10 client.26147 ms_handle_connect on v2:172.21.15.92:6836/1270635939 2024-04-02T14:37:19.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.564+0000 7f0ab2000640 1 -- 172.21.15.90:0/377204277 <== mds.3 v2:172.21.15.92:6836/1270635939 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x7f0ab80047a0 con 0x562c4e1f3aa0 2024-04-02T14:37:19.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.564+0000 7f0ab2000640 10 client.26147 handle_command_reply: tid=0 2024-04-02T14:37:19.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.564+0000 7f0ad2246740 1 client.26147 shutdown 2024-04-02T14:37:19.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.564+0000 7f0ad2246740 1 -- 172.21.15.90:0/377204277 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x562c4e1d1d10 msgr2=0x562c4e1d8070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:19.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.564+0000 7f0ad2246740 1 --2- 172.21.15.90:0/377204277 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x562c4e1d1d10 0x562c4e1d8070 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f0ab8000c00 tx=0x7f0ab8005210 comp rx=0 tx=0).stop 2024-04-02T14:37:19.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.564+0000 7f0ad2246740 1 -- 172.21.15.90:0/377204277 shutdown_connections 2024-04-02T14:37:19.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.564+0000 7f0ad2246740 1 --2- 172.21.15.90:0/377204277 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x562c4e1f3aa0 0x562c4e1f5e90 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:19.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.564+0000 7f0ad2246740 1 --2- 172.21.15.90:0/377204277 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562c4e1db080 0x562c4e1d8af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:19.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.564+0000 7f0ad2246740 1 --2- 172.21.15.90:0/377204277 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562c4e1d29f0 0x562c4e1d85b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:19.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.564+0000 7f0ad2246740 1 --2- 172.21.15.90:0/377204277 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x562c4e1d1d10 0x562c4e1d8070 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:19.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.564+0000 7f0ad2246740 1 -- 172.21.15.90:0/377204277 >> 172.21.15.90:0/377204277 conn(0x562c4e198a50 msgr2=0x562c4e1ce4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:37:19.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.564+0000 7f0ab2000640 0 client.26147 ms_handle_reset on v2:172.21.15.92:6836/1270635939 2024-04-02T14:37:19.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.564+0000 7f0ad2246740 1 -- 172.21.15.90:0/377204277 shutdown_connections 2024-04-02T14:37:19.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.565+0000 7f0ad2246740 1 -- 172.21.15.90:0/377204277 wait complete. 2024-04-02T14:37:19.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.565+0000 7f0ad2246740 20 client.26147 trim_cache size 0 max 16384 2024-04-02T14:37:19.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.606+0000 7f0ad2246740 1 Processor -- start 2024-04-02T14:37:19.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.607+0000 7f0ad2246740 1 -- start start 2024-04-02T14:37:19.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.607+0000 7f0ad2246740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562c4e1d1d10 0x562c4e1da7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:19.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.607+0000 7f0ad2246740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x562c4e1d29f0 0x562c4e1ef300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:19.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.607+0000 7f0ad2246740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562c4e1db080 0x562c4e1fb4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:19.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.607+0000 7f0ad2246740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f0ac81b3f00 con 0x562c4e1db080 2024-04-02T14:37:19.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.607+0000 7f0ad2246740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f0ac81b5a40 con 0x562c4e1d29f0 2024-04-02T14:37:19.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.607+0000 7f0ad2246740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f0ac81b45d0 con 0x562c4e1d1d10 2024-04-02T14:37:19.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.608+0000 7f0acd600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562c4e1db080 0x562c4e1fb4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:19.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.608+0000 7f0accc00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562c4e1d1d10 0x562c4e1da7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:19.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.608+0000 7f0accc00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562c4e1d1d10 0x562c4e1da7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:58212/0 (socket says 172.21.15.90:58212) 2024-04-02T14:37:19.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.608+0000 7f0accc00640 1 -- 172.21.15.90:0/3221921784 learned_addr learned my addr 172.21.15.90:0/3221921784 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:37:19.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.609+0000 7f0ac7e00640 1 --2- 172.21.15.90:0/3221921784 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x562c4e1d29f0 0x562c4e1ef300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:19.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.609+0000 7f0ac7e00640 1 -- 172.21.15.90:0/3221921784 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562c4e1d1d10 msgr2=0x562c4e1da7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:37:19.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.609+0000 7f0ac7e00640 1 --2- 172.21.15.90:0/3221921784 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562c4e1d1d10 0x562c4e1da7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:19.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.609+0000 7f0ac7e00640 1 -- 172.21.15.90:0/3221921784 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562c4e1db080 msgr2=0x562c4e1fb4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:37:19.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.609+0000 7f0ac7e00640 1 --2- 172.21.15.90:0/3221921784 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562c4e1db080 0x562c4e1fb4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:19.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.609+0000 7f0ac7e00640 1 -- 172.21.15.90:0/3221921784 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562c4e1dd7e0 con 0x562c4e1d29f0 2024-04-02T14:37:19.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.609+0000 7f0ac7e00640 1 --2- 172.21.15.90:0/3221921784 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x562c4e1d29f0 0x562c4e1ef300 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f0ac806dab0 tx=0x7f0ab403d000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c334242d9b99165f server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:37:19.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.610+0000 7f0abd200640 1 -- 172.21.15.90:0/3221921784 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ab4005040 con 0x562c4e1d29f0 2024-04-02T14:37:19.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.610+0000 7f0abd200640 1 -- 172.21.15.90:0/3221921784 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0ab4042040 con 0x562c4e1d29f0 2024-04-02T14:37:19.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.610+0000 7f0abd200640 1 -- 172.21.15.90:0/3221921784 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ab4003850 con 0x562c4e1d29f0 2024-04-02T14:37:19.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.613+0000 7f0ad2246740 1 -- 172.21.15.90:0/3221921784 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x562c4e1d29f0 msgr2=0x562c4e1ef300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:19.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.613+0000 7f0ad2246740 1 --2- 172.21.15.90:0/3221921784 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x562c4e1d29f0 0x562c4e1ef300 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f0ac806dab0 tx=0x7f0ab403d000 comp rx=0 tx=0).stop 2024-04-02T14:37:19.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.613+0000 7f0ad2246740 1 -- 172.21.15.90:0/3221921784 shutdown_connections 2024-04-02T14:37:19.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.613+0000 7f0ad2246740 1 --2- 172.21.15.90:0/3221921784 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562c4e1db080 0x562c4e1fb4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:19.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.613+0000 7f0ad2246740 1 --2- 172.21.15.90:0/3221921784 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x562c4e1d29f0 0x562c4e1ef300 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:19.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.614+0000 7f0ad2246740 1 --2- 172.21.15.90:0/3221921784 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562c4e1d1d10 0x562c4e1da7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:19.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.614+0000 7f0ad2246740 1 -- 172.21.15.90:0/3221921784 >> 172.21.15.90:0/3221921784 conn(0x562c4e198a50 msgr2=0x562c4e1d6940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:37:19.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.614+0000 7f0ad2246740 1 -- 172.21.15.90:0/3221921784 shutdown_connections 2024-04-02T14:37:19.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.615+0000 7f0ad2246740 1 -- 172.21.15.90:0/3221921784 wait complete. 2024-04-02T14:37:19.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.616+0000 7f0ad2246740 1 -- start start 2024-04-02T14:37:19.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.617+0000 7f0ad2246740 1 Processor -- start 2024-04-02T14:37:19.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.620+0000 7f0ad2246740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562c4e1d1d10 0x562c4e1db5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:19.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.620+0000 7f0ad2246740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x562c4e1d29f0 0x562c4e1dbae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:19.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.620+0000 7f0ad2246740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562c4e1db080 0x562c4e1ea810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:19.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.620+0000 7f0ad2246740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f0ac81b5370 con 0x562c4e1d1d10 2024-04-02T14:37:19.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.620+0000 7f0ad2246740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f0ac81b4ca0 con 0x562c4e1d29f0 2024-04-02T14:37:19.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.620+0000 7f0ad2246740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f0ac8106d20 con 0x562c4e1db080 2024-04-02T14:37:19.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.620+0000 7f0ad2246740 10 client.0 _ll_register_callbacks cb 0x7f0ab8004d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:37:19.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.620+0000 7f0acd600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562c4e1db080 0x562c4e1ea810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:19.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.620+0000 7f0acd600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562c4e1db080 0x562c4e1ea810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:58220/0 (socket says 172.21.15.90:58220) 2024-04-02T14:37:19.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.620+0000 7f0acd600640 1 -- 172.21.15.90:0/756851454 learned_addr learned my addr 172.21.15.90:0/756851454 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:37:19.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.621+0000 7f0ac7e00640 1 --2- 172.21.15.90:0/756851454 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x562c4e1d29f0 0x562c4e1dbae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:19.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.621+0000 7f0accc00640 1 --2- 172.21.15.90:0/756851454 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562c4e1d1d10 0x562c4e1db5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:19.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.621+0000 7f0acd600640 1 -- 172.21.15.90:0/756851454 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x562c4e1d29f0 msgr2=0x562c4e1dbae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:37:19.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.621+0000 7f0acd600640 1 --2- 172.21.15.90:0/756851454 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x562c4e1d29f0 0x562c4e1dbae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:19.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.621+0000 7f0acd600640 1 -- 172.21.15.90:0/756851454 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562c4e1d1d10 msgr2=0x562c4e1db5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:37:19.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.621+0000 7f0acd600640 1 --2- 172.21.15.90:0/756851454 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562c4e1d1d10 0x562c4e1db5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:19.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.621+0000 7f0acd600640 1 -- 172.21.15.90:0/756851454 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562c4e1ead50 con 0x562c4e1db080 2024-04-02T14:37:19.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.622+0000 7f0acd600640 1 --2- 172.21.15.90:0/756851454 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562c4e1db080 0x562c4e1ea810 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f0ac0099290 tx=0x7f0ac009c980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=22eb09ffde0f4978 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:37:19.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.623+0000 7f0ab2000640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:37:19.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.623+0000 7f0ab2000640 1 -- 172.21.15.90:0/756851454 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ac001f040 con 0x562c4e1db080 2024-04-02T14:37:19.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.623+0000 7f0ab2000640 1 -- 172.21.15.90:0/756851454 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f0ac00a50a0 con 0x562c4e1db080 2024-04-02T14:37:19.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.623+0000 7f0ad2246740 1 -- 172.21.15.90:0/756851454 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0ab4042040 con 0x562c4e1db080 2024-04-02T14:37:19.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.624+0000 7f0ab2000640 1 -- 172.21.15.90:0/756851454 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f0ac00a52a0 con 0x562c4e1db080 2024-04-02T14:37:19.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.624+0000 7f0ad2246740 10 client.26059 fetch_fsmap learned FSMap version 27 2024-04-02T14:37:19.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.624+0000 7f0ad2246740 1 -- 172.21.15.90:0/756851454 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x562c4e1ec010 con 0x562c4e1db080 2024-04-02T14:37:19.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.624+0000 7f0ab2000640 1 -- 172.21.15.90:0/756851454 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0ab4042040 con 0x562c4e1db080 2024-04-02T14:37:19.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.625+0000 7f0ab2000640 1 -- 172.21.15.90:0/756851454 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f0ac000f040 con 0x562c4e1db080 2024-04-02T14:37:19.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.625+0000 7f0ad2246740 10 client.26059 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:37:19.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.625+0000 7f0ad2246740 10 client.26059 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-04-02T14:37:19.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.626+0000 7f0ad2246740 20 client.26059 populate_metadata read hostname 'smithi090' 2024-04-02T14:37:19.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.626+0000 7f0ad2246740 1 --2- 172.21.15.90:0/756851454 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x562c4e25a140 0x562c4e25c530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:19.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.626+0000 7f0ad2246740 4 client.26059 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-02T14:37:19.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.626+0000 7f0ad2246740 1 -- 172.21.15.90:0/756851454 --> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] -- command(tid 0: {"prefix": "damage ls"}) -- 0x562c4e0f6040 con 0x562c4e25a140 2024-04-02T14:37:19.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.626+0000 7f0accc00640 1 --2- 172.21.15.90:0/756851454 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x562c4e25a140 0x562c4e25c530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:19.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.627+0000 7f0accc00640 1 --2- 172.21.15.90:0/756851454 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x562c4e25a140 0x562c4e25c530 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=29c093d89d533435 server_cookie=9115a3ff032f4513 in_seq=0 out_seq=0 2024-04-02T14:37:19.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.627+0000 7f0ab2000640 10 client.26059 ms_handle_connect on v2:172.21.15.92:6836/1270635939 2024-04-02T14:37:19.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.627+0000 7f0ab2000640 1 -- 172.21.15.90:0/756851454 <== mds.3 v2:172.21.15.92:6836/1270635939 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x562c4e0f6040 con 0x562c4e25a140 2024-04-02T14:37:19.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.627+0000 7f0ab2000640 10 client.26059 handle_command_reply: tid=0 2024-04-02T14:37:19.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.627+0000 7f0ad2246740 1 client.26059 shutdown 2024-04-02T14:37:19.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.627+0000 7f0ad2246740 1 -- 172.21.15.90:0/756851454 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562c4e1db080 msgr2=0x562c4e1ea810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:19.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.627+0000 7f0ad2246740 1 --2- 172.21.15.90:0/756851454 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562c4e1db080 0x562c4e1ea810 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f0ac0099290 tx=0x7f0ac009c980 comp rx=0 tx=0).stop 2024-04-02T14:37:19.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.628+0000 7f0ad2246740 1 -- 172.21.15.90:0/756851454 shutdown_connections 2024-04-02T14:37:19.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.628+0000 7f0ad2246740 1 --2- 172.21.15.90:0/756851454 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x562c4e25a140 0x562c4e25c530 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:19.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.628+0000 7f0ad2246740 1 --2- 172.21.15.90:0/756851454 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562c4e1db080 0x562c4e1ea810 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:19.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.628+0000 7f0ad2246740 1 --2- 172.21.15.90:0/756851454 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x562c4e1d29f0 0x562c4e1dbae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:19.630 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.628+0000 7f0ad2246740 1 --2- 172.21.15.90:0/756851454 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562c4e1d1d10 0x562c4e1db5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:19.630 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.628+0000 7f0ad2246740 1 -- 172.21.15.90:0/756851454 >> 172.21.15.90:0/756851454 conn(0x562c4e198a50 msgr2=0x562c4e1cb010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:37:19.630 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.629+0000 7f0ab2000640 0 client.26059 ms_handle_reset on v2:172.21.15.92:6836/1270635939 2024-04-02T14:37:19.630 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.629+0000 7f0ad2246740 1 -- 172.21.15.90:0/756851454 shutdown_connections 2024-04-02T14:37:19.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.630+0000 7f0ad2246740 1 -- 172.21.15.90:0/756851454 wait complete. 2024-04-02T14:37:19.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.630+0000 7f0ad2246740 20 client.26059 trim_cache size 0 max 16384 2024-04-02T14:37:19.632 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-02T14:37:19.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.632+0000 7f0acf600640 1 -- 172.21.15.90:0/2583082185 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f0aac0a3e70 msgr2=0x7f0aac0a6330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:19.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.632+0000 7f0acf600640 1 --2- 172.21.15.90:0/2583082185 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f0aac0a3e70 0x7f0aac0a6330 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f0ab4036040 tx=0x7f0ab4009260 comp rx=0 tx=0).stop 2024-04-02T14:37:19.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.632+0000 7f0acf600640 1 -- 172.21.15.90:0/2583082185 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0ac810e4a0 msgr2=0x7f0ac806c4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:19.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.632+0000 7f0acf600640 1 --2- 172.21.15.90:0/2583082185 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0ac810e4a0 0x7f0ac806c4d0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f0ac000eaa0 tx=0x7f0ac000ef60 comp rx=0 tx=0).stop 2024-04-02T14:37:19.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.632+0000 7f0acf600640 1 -- 172.21.15.90:0/2583082185 shutdown_connections 2024-04-02T14:37:19.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.632+0000 7f0acf600640 1 --2- 172.21.15.90:0/2583082185 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f0aac0a3e70 0x7f0aac0a6330 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:19.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.632+0000 7f0acf600640 1 --2- 172.21.15.90:0/2583082185 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0ac810e4a0 0x7f0ac806c4d0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:19.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.632+0000 7f0acf600640 1 --2- 172.21.15.90:0/2583082185 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0ac8069620 0x7f0ac80737b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:19.634 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.632+0000 7f0acf600640 1 --2- 172.21.15.90:0/2583082185 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0ac8068d00 0x7f0ac8073270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:19.634 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.632+0000 7f0acf600640 1 -- 172.21.15.90:0/2583082185 >> 172.21.15.90:0/2583082185 conn(0x7f0ac8077b00 msgr2=0x7f0ac810f8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:37:19.634 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.633+0000 7f0acf600640 1 -- 172.21.15.90:0/2583082185 shutdown_connections 2024-04-02T14:37:19.634 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.633+0000 7f0acf600640 1 -- 172.21.15.90:0/2583082185 wait complete. 2024-04-02T14:37:19.658 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-02T14:37:19.860 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:37:19 smithi092 ceph-mon[36497]: pgmap v76: 97 pgs: 97 active+clean; 1.5 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 25 MiB/s wr, 996 op/s 2024-04-02T14:37:19.861 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:37:19 smithi092 ceph-mon[36497]: from='client.? 172.21.15.90:0/1635187202' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-02T14:37:19.929 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.928+0000 7f15a6800640 1 -- 172.21.15.90:0/28231284 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f15a0101a90 msgr2=0x7f15a0101f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:19.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.928+0000 7f15a6800640 1 --2- 172.21.15.90:0/28231284 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f15a0101a90 0x7f15a0101f10 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f158c009900 tx=0x7f158c02bc40 comp rx=0 tx=0).stop 2024-04-02T14:37:19.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.929+0000 7f15a6800640 1 -- 172.21.15.90:0/28231284 shutdown_connections 2024-04-02T14:37:19.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.929+0000 7f15a6800640 1 --2- 172.21.15.90:0/28231284 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f15a0102450 0x7f15a010e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:19.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.929+0000 7f15a6800640 1 --2- 172.21.15.90:0/28231284 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f15a0101a90 0x7f15a0101f10 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:19.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.929+0000 7f15a6800640 1 --2- 172.21.15.90:0/28231284 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f15a0107aa0 0x7f15a0107e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:19.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.929+0000 7f15a6800640 1 -- 172.21.15.90:0/28231284 >> 172.21.15.90:0/28231284 conn(0x7f15a0078f00 msgr2=0x7f15a0079310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:37:19.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.929+0000 7f15a6800640 1 -- 172.21.15.90:0/28231284 shutdown_connections 2024-04-02T14:37:19.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.929+0000 7f15a6800640 1 -- 172.21.15.90:0/28231284 wait complete. 2024-04-02T14:37:19.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.931+0000 7f15a6800640 1 Processor -- start 2024-04-02T14:37:19.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.931+0000 7f15a6800640 1 -- start start 2024-04-02T14:37:19.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.932+0000 7f15a6800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f15a0101a90 0x7f15a019a580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:19.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.932+0000 7f15a6800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f15a0102450 0x7f15a019aac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:19.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.933+0000 7f159f400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f15a0101a90 0x7f15a019a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:19.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.933+0000 7f159f400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f15a0101a90 0x7f15a019a580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:58238/0 (socket says 172.21.15.90:58238) 2024-04-02T14:37:19.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.933+0000 7f159f400640 1 -- 172.21.15.90:0/4148055458 learned_addr learned my addr 172.21.15.90:0/4148055458 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:37:19.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.933+0000 7f15a6800640 1 --2- 172.21.15.90:0/4148055458 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f15a0107aa0 0x7f15a019ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:19.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.933+0000 7f15a6800640 1 -- 172.21.15.90:0/4148055458 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f15a0075ae0 con 0x7f15a0102450 2024-04-02T14:37:19.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.933+0000 7f15a6800640 1 -- 172.21.15.90:0/4148055458 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f15a0075960 con 0x7f15a0107aa0 2024-04-02T14:37:19.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.933+0000 7f15a6800640 1 -- 172.21.15.90:0/4148055458 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f15a0075c60 con 0x7f15a0101a90 2024-04-02T14:37:19.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.933+0000 7f159ea00640 1 --2- 172.21.15.90:0/4148055458 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f15a0102450 0x7f15a019aac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:19.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.933+0000 7f159fe00640 1 --2- 172.21.15.90:0/4148055458 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f15a0107aa0 0x7f15a019ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:19.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.933+0000 7f159f400640 1 -- 172.21.15.90:0/4148055458 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f15a0107aa0 msgr2=0x7f15a019ee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:19.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.933+0000 7f159f400640 1 --2- 172.21.15.90:0/4148055458 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f15a0107aa0 0x7f15a019ee40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:19.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.933+0000 7f159f400640 1 -- 172.21.15.90:0/4148055458 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f15a0102450 msgr2=0x7f15a019aac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:19.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.934+0000 7f159f400640 1 --2- 172.21.15.90:0/4148055458 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f15a0102450 0x7f15a019aac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:19.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.934+0000 7f159f400640 1 -- 172.21.15.90:0/4148055458 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15a019f570 con 0x7f15a0101a90 2024-04-02T14:37:19.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.934+0000 7f159fe00640 1 --2- 172.21.15.90:0/4148055458 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f15a0107aa0 0x7f15a019ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:37:19.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.934+0000 7f159ea00640 1 --2- 172.21.15.90:0/4148055458 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f15a0102450 0x7f15a019aac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:37:19.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.934+0000 7f159f400640 1 --2- 172.21.15.90:0/4148055458 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f15a0101a90 0x7f15a019a580 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f159800ebc0 tx=0x7f159800c6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:37:19.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.934+0000 7f1597e00640 1 -- 172.21.15.90:0/4148055458 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1598010630 con 0x7f15a0101a90 2024-04-02T14:37:19.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.934+0000 7f1597e00640 1 -- 172.21.15.90:0/4148055458 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1598010cb0 con 0x7f15a0101a90 2024-04-02T14:37:19.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.934+0000 7f1597e00640 1 -- 172.21.15.90:0/4148055458 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1598018bd0 con 0x7f15a0101a90 2024-04-02T14:37:19.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:37:19 smithi177 ceph-mon[36431]: pgmap v76: 97 pgs: 97 active+clean; 1.5 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 25 MiB/s wr, 996 op/s 2024-04-02T14:37:19.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:37:19 smithi177 ceph-mon[36431]: from='client.? 172.21.15.90:0/1635187202' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-02T14:37:19.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.934+0000 7f15a6800640 1 -- 172.21.15.90:0/4148055458 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f15a019f860 con 0x7f15a0101a90 2024-04-02T14:37:19.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.935+0000 7f15a6800640 1 -- 172.21.15.90:0/4148055458 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15a01a70f0 con 0x7f15a0101a90 2024-04-02T14:37:19.938 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:37:19 smithi090 ceph-mon[31674]: pgmap v76: 97 pgs: 97 active+clean; 1.5 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 25 MiB/s wr, 996 op/s 2024-04-02T14:37:19.938 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:37:19 smithi090 ceph-mon[31674]: from='client.? 172.21.15.90:0/1635187202' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-02T14:37:19.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.940+0000 7f1597e00640 1 -- 172.21.15.90:0/4148055458 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f15980107d0 con 0x7f15a0101a90 2024-04-02T14:37:19.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.943+0000 7f15a95d3740 1 Processor -- start 2024-04-02T14:37:19.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.943+0000 7f1597e00640 1 --2- 172.21.15.90:0/4148055458 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f15840a3e60 0x7f15840a6320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:19.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.944+0000 7f1597e00640 1 -- 172.21.15.90:0/4148055458 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f15980d0ed0 con 0x7f15a0101a90 2024-04-02T14:37:19.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.945+0000 7f159fe00640 1 --2- 172.21.15.90:0/4148055458 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f15840a3e60 0x7f15840a6320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:19.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.945+0000 7f15a95d3740 1 -- start start 2024-04-02T14:37:19.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.945+0000 7f15a95d3740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561b7a19d9f0 0x561b7a19c740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:19.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.945+0000 7f15a95d3740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561b7a19cd10 0x561b7a19d190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:19.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.945+0000 7f15a95d3740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561b7a1ae890 0x561b7a1b0c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:19.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.945+0000 7f15a95d3740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x561b7a0dd400 con 0x561b7a19cd10 2024-04-02T14:37:19.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.945+0000 7f15a95d3740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x561b7a0c1f10 con 0x561b7a19d9f0 2024-04-02T14:37:19.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.945+0000 7f15a95d3740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x561b7a1016f0 con 0x561b7a1ae890 2024-04-02T14:37:19.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.945+0000 7f159fe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561b7a1ae890 0x561b7a1b0c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:19.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.946+0000 7f159fe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561b7a1ae890 0x561b7a1b0c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:58248/0 (socket says 172.21.15.90:58248) 2024-04-02T14:37:19.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.946+0000 7f159fe00640 1 -- 172.21.15.90:0/2863550514 learned_addr learned my addr 172.21.15.90:0/2863550514 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:37:19.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.946+0000 7f159f400640 1 --2- 172.21.15.90:0/2863550514 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561b7a19d9f0 0x561b7a19c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:19.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.946+0000 7f159f400640 1 -- 172.21.15.90:0/2863550514 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561b7a1ae890 msgr2=0x561b7a1b0c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:37:19.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.946+0000 7f159f400640 1 --2- 172.21.15.90:0/2863550514 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561b7a1ae890 0x561b7a1b0c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:19.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.946+0000 7f159f400640 1 -- 172.21.15.90:0/2863550514 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561b7a19cd10 msgr2=0x561b7a19d190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:37:19.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.946+0000 7f159ea00640 1 --2- 172.21.15.90:0/2863550514 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561b7a19cd10 0x561b7a19d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:19.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.946+0000 7f159f400640 1 --2- 172.21.15.90:0/2863550514 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561b7a19cd10 0x561b7a19d190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:19.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.946+0000 7f159f400640 1 -- 172.21.15.90:0/2863550514 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561b79e153f0 con 0x561b7a19d9f0 2024-04-02T14:37:19.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.946+0000 7f159ea00640 1 --2- 172.21.15.90:0/2863550514 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561b7a19cd10 0x561b7a19d190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:37:19.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.947+0000 7f159f400640 1 --2- 172.21.15.90:0/2863550514 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561b7a19d9f0 0x561b7a19c740 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f15980ce5d0 tx=0x7f159800cf90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fca20043ecfab4ee server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:37:19.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.947+0000 7f1594c00640 1 -- 172.21.15.90:0/2863550514 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15980a11d0 con 0x561b7a19d9f0 2024-04-02T14:37:19.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.947+0000 7f1594c00640 1 -- 172.21.15.90:0/2863550514 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f15980a5a90 con 0x561b7a19d9f0 2024-04-02T14:37:19.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.947+0000 7f1594c00640 1 -- 172.21.15.90:0/2863550514 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f159809c270 con 0x561b7a19d9f0 2024-04-02T14:37:19.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.948+0000 7f15a95d3740 1 -- 172.21.15.90:0/2863550514 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561b7a19d9f0 msgr2=0x561b7a19c740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:19.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.948+0000 7f15a95d3740 1 --2- 172.21.15.90:0/2863550514 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561b7a19d9f0 0x561b7a19c740 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f15980ce5d0 tx=0x7f159800cf90 comp rx=0 tx=0).stop 2024-04-02T14:37:19.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.949+0000 7f159fe00640 1 --2- 172.21.15.90:0/4148055458 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f15840a3e60 0x7f15840a6320 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f1590014460 tx=0x7f1590014920 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:37:19.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.949+0000 7f15a95d3740 1 -- 172.21.15.90:0/2863550514 shutdown_connections 2024-04-02T14:37:19.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.949+0000 7f15a95d3740 1 --2- 172.21.15.90:0/2863550514 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561b7a1ae890 0x561b7a1b0c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:19.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.949+0000 7f15a95d3740 1 --2- 172.21.15.90:0/2863550514 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561b7a19cd10 0x561b7a19d190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:19.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.949+0000 7f15a95d3740 1 --2- 172.21.15.90:0/2863550514 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561b7a19d9f0 0x561b7a19c740 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:19.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.949+0000 7f15a95d3740 1 -- 172.21.15.90:0/2863550514 >> 172.21.15.90:0/2863550514 conn(0x561b7a163a50 msgr2=0x561b7a197760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:37:19.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.950+0000 7f15a95d3740 1 -- 172.21.15.90:0/2863550514 shutdown_connections 2024-04-02T14:37:19.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.950+0000 7f15a95d3740 1 -- 172.21.15.90:0/2863550514 wait complete. 2024-04-02T14:37:19.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.951+0000 7f15a95d3740 1 -- start start 2024-04-02T14:37:19.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.951+0000 7f15a95d3740 1 Processor -- start 2024-04-02T14:37:19.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.954+0000 7f15a95d3740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561b7a19cd10 0x561b7a1ab840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:19.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.954+0000 7f15a95d3740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561b7a19d9f0 0x561b7a1abd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:19.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.954+0000 7f15a95d3740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561b7a1ae890 0x561b7a1ac2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:19.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.954+0000 7f15a95d3740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x561b7a0fd6a0 con 0x561b7a19cd10 2024-04-02T14:37:19.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.954+0000 7f15a95d3740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x561b7a0a9980 con 0x561b7a19d9f0 2024-04-02T14:37:19.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.954+0000 7f15a95d3740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x561b7a0bccb0 con 0x561b7a1ae890 2024-04-02T14:37:19.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.954+0000 7f15a95d3740 10 client.0 _ll_register_callbacks cb 0x561b7a0bc6a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:37:19.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.954+0000 7f159fe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561b7a1ae890 0x561b7a1ac2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:19.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.954+0000 7f159fe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561b7a1ae890 0x561b7a1ac2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:58262/0 (socket says 172.21.15.90:58262) 2024-04-02T14:37:19.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.954+0000 7f159fe00640 1 -- 172.21.15.90:0/3968195992 learned_addr learned my addr 172.21.15.90:0/3968195992 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:37:19.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.954+0000 7f159ea00640 1 --2- 172.21.15.90:0/3968195992 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561b7a19d9f0 0x561b7a1abd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:19.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.954+0000 7f159f400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561b7a19cd10 0x561b7a1ab840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:19.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.955+0000 7f159fe00640 1 -- 172.21.15.90:0/3968195992 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561b7a19d9f0 msgr2=0x561b7a1abd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:37:19.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.955+0000 7f159fe00640 1 --2- 172.21.15.90:0/3968195992 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561b7a19d9f0 0x561b7a1abd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:19.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.955+0000 7f159fe00640 1 -- 172.21.15.90:0/3968195992 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561b7a19cd10 msgr2=0x561b7a1ab840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:37:19.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.955+0000 7f159fe00640 1 --2- 172.21.15.90:0/3968195992 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561b7a19cd10 0x561b7a1ab840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:19.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.955+0000 7f159fe00640 1 -- 172.21.15.90:0/3968195992 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561b79edf7d0 con 0x561b7a1ae890 2024-04-02T14:37:19.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.956+0000 7f159fe00640 1 --2- 172.21.15.90:0/3968195992 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561b7a1ae890 0x561b7a1ac2c0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f1590014230 tx=0x7f159001a000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2e93850cf3bacfd5 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:37:19.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.956+0000 7f1589600640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:37:19.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.957+0000 7f1589600640 1 -- 172.21.15.90:0/3968195992 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1590012850 con 0x561b7a1ae890 2024-04-02T14:37:19.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.957+0000 7f1589600640 1 -- 172.21.15.90:0/3968195992 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f15900129f0 con 0x561b7a1ae890 2024-04-02T14:37:19.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.957+0000 7f15a95d3740 1 -- 172.21.15.90:0/3968195992 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f15980a5a90 con 0x561b7a1ae890 2024-04-02T14:37:19.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.958+0000 7f1589600640 1 -- 172.21.15.90:0/3968195992 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f1590012bf0 con 0x561b7a1ae890 2024-04-02T14:37:19.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.958+0000 7f15a95d3740 10 client.26071 fetch_fsmap learned FSMap version 27 2024-04-02T14:37:19.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.958+0000 7f15a95d3740 1 -- 172.21.15.90:0/3968195992 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x561b7a1bd620 con 0x561b7a1ae890 2024-04-02T14:37:19.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.959+0000 7f1589600640 1 -- 172.21.15.90:0/3968195992 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f15980a5a90 con 0x561b7a1ae890 2024-04-02T14:37:19.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.959+0000 7f1589600640 1 -- 172.21.15.90:0/3968195992 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f159000f040 con 0x561b7a1ae890 2024-04-02T14:37:19.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.959+0000 7f15a95d3740 10 client.26071 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:37:19.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.959+0000 7f15a95d3740 10 client.26071 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-04-02T14:37:19.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.959+0000 7f15a95d3740 20 client.26071 populate_metadata read hostname 'smithi090' 2024-04-02T14:37:19.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.959+0000 7f15a95d3740 1 --2- 172.21.15.90:0/3968195992 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x561b7a1beae0 0x561b7a1c0f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:19.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.959+0000 7f15a95d3740 4 client.26071 mds_command: new command op to 24547 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:37:19.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.959+0000 7f15a95d3740 1 -- 172.21.15.90:0/3968195992 --> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561b7a0c1040 con 0x561b7a1beae0 2024-04-02T14:37:19.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.959+0000 7f159f400640 1 --2- 172.21.15.90:0/3968195992 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x561b7a1beae0 0x561b7a1c0f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:19.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.960+0000 7f159f400640 1 --2- 172.21.15.90:0/3968195992 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x561b7a1beae0 0x561b7a1c0f00 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=7322602bdf280f10 server_cookie=8e4d34e824666092 in_seq=0 out_seq=0 2024-04-02T14:37:19.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.960+0000 7f1589600640 10 client.26071 ms_handle_connect on v2:172.21.15.177:6838/664895663 2024-04-02T14:37:19.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.961+0000 7f1589600640 1 -- 172.21.15.90:0/3968195992 <== mds.2 v2:172.21.15.177:6838/664895663 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x561b7a0c1040 con 0x561b7a1beae0 2024-04-02T14:37:19.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.961+0000 7f1589600640 10 client.26071 handle_command_reply: tid=0 2024-04-02T14:37:19.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.961+0000 7f15a95d3740 1 client.26071 shutdown 2024-04-02T14:37:19.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.962+0000 7f15a95d3740 1 -- 172.21.15.90:0/3968195992 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561b7a1ae890 msgr2=0x561b7a1ac2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:19.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.962+0000 7f15a95d3740 1 --2- 172.21.15.90:0/3968195992 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561b7a1ae890 0x561b7a1ac2c0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f1590014230 tx=0x7f159001a000 comp rx=0 tx=0).stop 2024-04-02T14:37:19.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.963+0000 7f15a95d3740 1 -- 172.21.15.90:0/3968195992 shutdown_connections 2024-04-02T14:37:19.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.963+0000 7f15a95d3740 1 --2- 172.21.15.90:0/3968195992 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x561b7a1beae0 0x561b7a1c0f00 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:19.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.963+0000 7f15a95d3740 1 --2- 172.21.15.90:0/3968195992 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561b7a1ae890 0x561b7a1ac2c0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:19.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.963+0000 7f15a95d3740 1 --2- 172.21.15.90:0/3968195992 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561b7a19d9f0 0x561b7a1abd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:19.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.963+0000 7f15a95d3740 1 --2- 172.21.15.90:0/3968195992 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561b7a19cd10 0x561b7a1ab840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:19.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.963+0000 7f15a95d3740 1 -- 172.21.15.90:0/3968195992 >> 172.21.15.90:0/3968195992 conn(0x561b7a163a50 msgr2=0x561b7a1994b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:37:19.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.963+0000 7f1589600640 0 client.26071 ms_handle_reset on v2:172.21.15.177:6838/664895663 2024-04-02T14:37:19.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.963+0000 7f15a95d3740 1 -- 172.21.15.90:0/3968195992 shutdown_connections 2024-04-02T14:37:19.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.963+0000 7f15a95d3740 1 -- 172.21.15.90:0/3968195992 wait complete. 2024-04-02T14:37:19.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.964+0000 7f15a95d3740 20 client.26071 trim_cache size 0 max 16384 2024-04-02T14:37:19.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.998+0000 7f15a95d3740 1 Processor -- start 2024-04-02T14:37:19.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.998+0000 7f15a95d3740 1 -- start start 2024-04-02T14:37:20.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.999+0000 7f15a95d3740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561b7a19cd10 0x561b7a1ba240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:20.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.999+0000 7f15a95d3740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561b7a19d9f0 0x561b7a1ad670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:20.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.999+0000 7f15a95d3740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561b7a1ae890 0x561b7a1c67d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:20.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.999+0000 7f15a95d3740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f15a01ab740 con 0x561b7a19cd10 2024-04-02T14:37:20.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.999+0000 7f15a95d3740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f15a01ad280 con 0x561b7a19d9f0 2024-04-02T14:37:20.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.999+0000 7f15a95d3740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f15a01abe10 con 0x561b7a1ae890 2024-04-02T14:37:20.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.999+0000 7f159ea00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561b7a19d9f0 0x561b7a1ad670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:20.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.999+0000 7f159ea00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561b7a19d9f0 0x561b7a1ad670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:48786/0 (socket says 172.21.15.90:48786) 2024-04-02T14:37:20.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:19.999+0000 7f159ea00640 1 -- 172.21.15.90:0/1576509979 learned_addr learned my addr 172.21.15.90:0/1576509979 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:37:20.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.000+0000 7f159fe00640 1 --2- 172.21.15.90:0/1576509979 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561b7a1ae890 0x561b7a1c67d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:20.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.000+0000 7f159ea00640 1 -- 172.21.15.90:0/1576509979 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561b7a1ae890 msgr2=0x561b7a1c67d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:37:20.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.000+0000 7f159ea00640 1 --2- 172.21.15.90:0/1576509979 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561b7a1ae890 0x561b7a1c67d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:20.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.000+0000 7f159ea00640 1 -- 172.21.15.90:0/1576509979 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561b7a19cd10 msgr2=0x561b7a1ba240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:37:20.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.000+0000 7f159f400640 1 --2- 172.21.15.90:0/1576509979 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561b7a19cd10 0x561b7a1ba240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:20.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.000+0000 7f159ea00640 1 --2- 172.21.15.90:0/1576509979 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561b7a19cd10 0x561b7a1ba240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:20.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.000+0000 7f159ea00640 1 -- 172.21.15.90:0/1576509979 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561b7a1a9d90 con 0x561b7a19d9f0 2024-04-02T14:37:20.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.000+0000 7f159f400640 1 --2- 172.21.15.90:0/1576509979 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561b7a19cd10 0x561b7a1ba240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:37:20.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.001+0000 7f159ea00640 1 --2- 172.21.15.90:0/1576509979 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561b7a19d9f0 0x561b7a1ad670 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f158c009340 tx=0x7f158c0095f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=40d5b05417013d62 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:37:20.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.002+0000 7f158be00640 1 -- 172.21.15.90:0/1576509979 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f158c042030 con 0x561b7a19d9f0 2024-04-02T14:37:20.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.002+0000 7f158be00640 1 -- 172.21.15.90:0/1576509979 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f158c02c660 con 0x561b7a19d9f0 2024-04-02T14:37:20.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.002+0000 7f158be00640 1 -- 172.21.15.90:0/1576509979 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f158c005410 con 0x561b7a19d9f0 2024-04-02T14:37:20.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.003+0000 7f15a95d3740 1 -- 172.21.15.90:0/1576509979 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561b7a19d9f0 msgr2=0x561b7a1ad670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:20.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.003+0000 7f15a95d3740 1 --2- 172.21.15.90:0/1576509979 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561b7a19d9f0 0x561b7a1ad670 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f158c009340 tx=0x7f158c0095f0 comp rx=0 tx=0).stop 2024-04-02T14:37:20.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.004+0000 7f15a95d3740 1 -- 172.21.15.90:0/1576509979 shutdown_connections 2024-04-02T14:37:20.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.004+0000 7f15a95d3740 1 --2- 172.21.15.90:0/1576509979 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561b7a1ae890 0x561b7a1c67d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:20.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.004+0000 7f15a95d3740 1 --2- 172.21.15.90:0/1576509979 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561b7a19d9f0 0x561b7a1ad670 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:20.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.004+0000 7f15a95d3740 1 --2- 172.21.15.90:0/1576509979 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561b7a19cd10 0x561b7a1ba240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:20.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.004+0000 7f15a95d3740 1 -- 172.21.15.90:0/1576509979 >> 172.21.15.90:0/1576509979 conn(0x561b7a163a50 msgr2=0x561b7a1acf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:37:20.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.004+0000 7f15a95d3740 1 -- 172.21.15.90:0/1576509979 shutdown_connections 2024-04-02T14:37:20.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.004+0000 7f15a95d3740 1 -- 172.21.15.90:0/1576509979 wait complete. 2024-04-02T14:37:20.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.005+0000 7f15a95d3740 1 -- start start 2024-04-02T14:37:20.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.005+0000 7f15a95d3740 1 Processor -- start 2024-04-02T14:37:20.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.006+0000 7f15a95d3740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561b7a19cd10 0x561b7a1b69c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:20.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.006+0000 7f15a95d3740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561b7a19d9f0 0x561b7a1af080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:20.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.006+0000 7f159f400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561b7a19cd10 0x561b7a1b69c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:20.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.006+0000 7f15a95d3740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561b7a1ae890 0x561b7a1af5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:20.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.006+0000 7f159f400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561b7a19cd10 0x561b7a1b69c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:58278/0 (socket says 172.21.15.90:58278) 2024-04-02T14:37:20.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.006+0000 7f15a95d3740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f15a01acbb0 con 0x561b7a19d9f0 2024-04-02T14:37:20.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.006+0000 7f159f400640 1 -- 172.21.15.90:0/2822483979 learned_addr learned my addr 172.21.15.90:0/2822483979 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:37:20.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.006+0000 7f15a95d3740 1 -- 172.21.15.90:0/2822483979 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f15a01ac4e0 con 0x561b7a1ae890 2024-04-02T14:37:20.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.006+0000 7f15a95d3740 1 -- 172.21.15.90:0/2822483979 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f15a01a9740 con 0x561b7a19cd10 2024-04-02T14:37:20.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.007+0000 7f15a95d3740 10 client.0 _ll_register_callbacks cb 0x7f1590014050 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:37:20.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.007+0000 7f159fe00640 1 --2- 172.21.15.90:0/2822483979 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561b7a1ae890 0x561b7a1af5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:20.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.007+0000 7f159f400640 1 -- 172.21.15.90:0/2822483979 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561b7a1ae890 msgr2=0x561b7a1af5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:37:20.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.007+0000 7f159f400640 1 --2- 172.21.15.90:0/2822483979 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561b7a1ae890 0x561b7a1af5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:20.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.007+0000 7f159f400640 1 -- 172.21.15.90:0/2822483979 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561b7a19d9f0 msgr2=0x561b7a1af080 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-02T14:37:20.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.007+0000 7f159ea00640 1 --2- 172.21.15.90:0/2822483979 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561b7a19d9f0 0x561b7a1af080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:20.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.008+0000 7f159f400640 1 --2- 172.21.15.90:0/2822483979 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561b7a19d9f0 0x561b7a1af080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:20.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.008+0000 7f159f400640 1 -- 172.21.15.90:0/2822483979 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561b7a1afc00 con 0x561b7a19cd10 2024-04-02T14:37:20.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.008+0000 7f159ea00640 1 --2- 172.21.15.90:0/2822483979 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561b7a19d9f0 0x561b7a1af080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:37:20.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.008+0000 7f159f400640 1 --2- 172.21.15.90:0/2822483979 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561b7a19cd10 0x561b7a1b69c0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f1598099280 tx=0x7f1598000f40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=290f6cc4431e6a12 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:37:20.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.008+0000 7f1589600640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:37:20.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.008+0000 7f1589600640 1 -- 172.21.15.90:0/2822483979 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1598010e60 con 0x561b7a19cd10 2024-04-02T14:37:20.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.008+0000 7f1589600640 1 -- 172.21.15.90:0/2822483979 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f159809b120 con 0x561b7a19cd10 2024-04-02T14:37:20.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.008+0000 7f15a95d3740 1 -- 172.21.15.90:0/2822483979 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f158c02c660 con 0x561b7a19cd10 2024-04-02T14:37:20.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.009+0000 7f1589600640 1 -- 172.21.15.90:0/2822483979 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f15980a5a90 con 0x561b7a19cd10 2024-04-02T14:37:20.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.009+0000 7f15a95d3740 10 client.26077 fetch_fsmap learned FSMap version 27 2024-04-02T14:37:20.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.009+0000 7f15a95d3740 1 -- 172.21.15.90:0/2822483979 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x561b7a1b6510 con 0x561b7a19cd10 2024-04-02T14:37:20.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.009+0000 7f1589600640 1 -- 172.21.15.90:0/2822483979 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f158c02c660 con 0x561b7a19cd10 2024-04-02T14:37:20.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.010+0000 7f1589600640 1 -- 172.21.15.90:0/2822483979 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f15980ce850 con 0x561b7a19cd10 2024-04-02T14:37:20.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.010+0000 7f15a95d3740 10 client.26077 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:37:20.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.010+0000 7f15a95d3740 10 client.26077 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-04-02T14:37:20.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.010+0000 7f15a95d3740 20 client.26077 populate_metadata read hostname 'smithi090' 2024-04-02T14:37:20.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.010+0000 7f15a95d3740 1 --2- 172.21.15.90:0/2822483979 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x561b7a225150 0x561b7a227540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:20.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.010+0000 7f15a95d3740 4 client.26077 mds_command: new command op to 24547 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-02T14:37:20.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.010+0000 7f15a95d3740 1 -- 172.21.15.90:0/2822483979 --> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] -- command(tid 0: {"prefix": "damage ls"}) -- 0x561b7a10ec30 con 0x561b7a225150 2024-04-02T14:37:20.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.010+0000 7f159ea00640 1 --2- 172.21.15.90:0/2822483979 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x561b7a225150 0x561b7a227540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:20.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.015+0000 7f159ea00640 1 --2- 172.21.15.90:0/2822483979 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x561b7a225150 0x561b7a227540 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=6138e35c36cc3d9 server_cookie=38ca93ff944b43e8 in_seq=0 out_seq=0 2024-04-02T14:37:20.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.015+0000 7f1589600640 10 client.26077 ms_handle_connect on v2:172.21.15.177:6838/664895663 2024-04-02T14:37:20.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.016+0000 7f1589600640 1 -- 172.21.15.90:0/2822483979 <== mds.2 v2:172.21.15.177:6838/664895663 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x561b7a10ec30 con 0x561b7a225150 2024-04-02T14:37:20.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.016+0000 7f1589600640 10 client.26077 handle_command_reply: tid=0 2024-04-02T14:37:20.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.016+0000 7f15a95d3740 1 client.26077 shutdown 2024-04-02T14:37:20.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.016+0000 7f15a95d3740 1 -- 172.21.15.90:0/2822483979 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561b7a19cd10 msgr2=0x561b7a1b69c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:20.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.016+0000 7f15a95d3740 1 --2- 172.21.15.90:0/2822483979 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561b7a19cd10 0x561b7a1b69c0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f1598099280 tx=0x7f1598000f40 comp rx=0 tx=0).stop 2024-04-02T14:37:20.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.016+0000 7f15a95d3740 1 -- 172.21.15.90:0/2822483979 shutdown_connections 2024-04-02T14:37:20.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.016+0000 7f15a95d3740 1 --2- 172.21.15.90:0/2822483979 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x561b7a225150 0x561b7a227540 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:20.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.016+0000 7f15a95d3740 1 --2- 172.21.15.90:0/2822483979 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561b7a1ae890 0x561b7a1af5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:20.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.016+0000 7f15a95d3740 1 --2- 172.21.15.90:0/2822483979 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561b7a19d9f0 0x561b7a1af080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:20.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.016+0000 7f1589600640 0 client.26077 ms_handle_reset on v2:172.21.15.177:6838/664895663 2024-04-02T14:37:20.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.016+0000 7f15a95d3740 1 --2- 172.21.15.90:0/2822483979 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561b7a19cd10 0x561b7a1b69c0 secure :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f1598099280 tx=0x7f1598000f40 comp rx=0 tx=0).stop 2024-04-02T14:37:20.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.016+0000 7f15a95d3740 1 -- 172.21.15.90:0/2822483979 >> 172.21.15.90:0/2822483979 conn(0x561b7a163a50 msgr2=0x561b7a1968a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:37:20.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.017+0000 7f15a95d3740 1 -- 172.21.15.90:0/2822483979 shutdown_connections 2024-04-02T14:37:20.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.018+0000 7f15a95d3740 1 -- 172.21.15.90:0/2822483979 wait complete. 2024-04-02T14:37:20.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.019+0000 7f15a95d3740 20 client.26077 trim_cache size 0 max 16384 2024-04-02T14:37:20.021 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-02T14:37:20.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.022+0000 7f15a6800640 1 -- 172.21.15.90:0/4148055458 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f15840a3e60 msgr2=0x7f15840a6320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:20.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.022+0000 7f15a6800640 1 --2- 172.21.15.90:0/4148055458 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f15840a3e60 0x7f15840a6320 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f1590014460 tx=0x7f1590014920 comp rx=0 tx=0).stop 2024-04-02T14:37:20.025 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.023+0000 7f15a6800640 1 -- 172.21.15.90:0/4148055458 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f15a0101a90 msgr2=0x7f15a019a580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:20.025 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.023+0000 7f15a6800640 1 --2- 172.21.15.90:0/4148055458 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f15a0101a90 0x7f15a019a580 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f159800ebc0 tx=0x7f159800c6a0 comp rx=0 tx=0).stop 2024-04-02T14:37:20.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.023+0000 7f15a6800640 1 -- 172.21.15.90:0/4148055458 shutdown_connections 2024-04-02T14:37:20.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.023+0000 7f15a6800640 1 --2- 172.21.15.90:0/4148055458 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f15840a3e60 0x7f15840a6320 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:20.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.023+0000 7f15a6800640 1 --2- 172.21.15.90:0/4148055458 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f15a0107aa0 0x7f15a019ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:20.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.023+0000 7f15a6800640 1 --2- 172.21.15.90:0/4148055458 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f15a0102450 0x7f15a019aac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:20.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.023+0000 7f15a6800640 1 --2- 172.21.15.90:0/4148055458 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f15a0101a90 0x7f15a019a580 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:20.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.023+0000 7f15a6800640 1 -- 172.21.15.90:0/4148055458 >> 172.21.15.90:0/4148055458 conn(0x7f15a0078f00 msgr2=0x7f15a0104360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:37:20.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.025+0000 7f15a6800640 1 -- 172.21.15.90:0/4148055458 shutdown_connections 2024-04-02T14:37:20.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.025+0000 7f15a6800640 1 -- 172.21.15.90:0/4148055458 wait complete. 2024-04-02T14:37:20.058 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-02T14:37:20.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.348+0000 7f91c1e00640 1 -- 172.21.15.90:0/3925737469 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f91bc101330 msgr2=0x7f91bc10d840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:20.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.348+0000 7f91c1e00640 1 --2- 172.21.15.90:0/3925737469 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f91bc101330 0x7f91bc10d840 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f91b4009930 tx=0x7f91b402bc20 comp rx=0 tx=0).stop 2024-04-02T14:37:20.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.350+0000 7f91c1e00640 1 -- 172.21.15.90:0/3925737469 shutdown_connections 2024-04-02T14:37:20.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.350+0000 7f91c1e00640 1 --2- 172.21.15.90:0/3925737469 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f91bc101330 0x7f91bc10d840 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:20.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.350+0000 7f91c1e00640 1 --2- 172.21.15.90:0/3925737469 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f91bc100970 0x7f91bc100df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:20.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.350+0000 7f91c1e00640 1 --2- 172.21.15.90:0/3925737469 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f91bc106980 0x7f91bc106d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:20.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.350+0000 7f91c1e00640 1 -- 172.21.15.90:0/3925737469 >> 172.21.15.90:0/3925737469 conn(0x7f91bc077b00 msgr2=0x7f91bc077f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:37:20.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.350+0000 7f91c1e00640 1 -- 172.21.15.90:0/3925737469 shutdown_connections 2024-04-02T14:37:20.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.351+0000 7f91c1e00640 1 -- 172.21.15.90:0/3925737469 wait complete. 2024-04-02T14:37:20.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.353+0000 7f91c1e00640 1 Processor -- start 2024-04-02T14:37:20.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.353+0000 7f91c1e00640 1 -- start start 2024-04-02T14:37:20.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.354+0000 7f91c1e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f91bc100970 0x7f91bc1a4690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:20.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.354+0000 7f91c1e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f91bc101330 0x7f91bc1a4bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:20.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.354+0000 7f91c1e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f91bc106980 0x7f91bc19ea00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:20.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.354+0000 7f91c1e00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f91bc110d60 con 0x7f91bc106980 2024-04-02T14:37:20.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.354+0000 7f91c1e00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f91bc110be0 con 0x7f91bc100970 2024-04-02T14:37:20.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.354+0000 7f91c1e00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f91bc110ee0 con 0x7f91bc101330 2024-04-02T14:37:20.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.354+0000 7f91baa00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f91bc100970 0x7f91bc1a4690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:20.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.354+0000 7f91baa00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f91bc100970 0x7f91bc1a4690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:48818/0 (socket says 172.21.15.90:48818) 2024-04-02T14:37:20.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.354+0000 7f91baa00640 1 -- 172.21.15.90:0/3972937747 learned_addr learned my addr 172.21.15.90:0/3972937747 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:37:20.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.354+0000 7f91ba000640 1 --2- 172.21.15.90:0/3972937747 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f91bc101330 0x7f91bc1a4bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:20.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.355+0000 7f91baa00640 1 -- 172.21.15.90:0/3972937747 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f91bc101330 msgr2=0x7f91bc1a4bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:20.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.355+0000 7f91baa00640 1 --2- 172.21.15.90:0/3972937747 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f91bc101330 0x7f91bc1a4bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:20.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.355+0000 7f91baa00640 1 -- 172.21.15.90:0/3972937747 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f91bc106980 msgr2=0x7f91bc19ea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:20.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.355+0000 7f91baa00640 1 --2- 172.21.15.90:0/3972937747 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f91bc106980 0x7f91bc19ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:20.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.355+0000 7f91baa00640 1 -- 172.21.15.90:0/3972937747 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91bc19f2c0 con 0x7f91bc100970 2024-04-02T14:37:20.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.355+0000 7f91ba000640 1 --2- 172.21.15.90:0/3972937747 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f91bc101330 0x7f91bc1a4bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:37:20.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.356+0000 7f91baa00640 1 --2- 172.21.15.90:0/3972937747 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f91bc100970 0x7f91bc1a4690 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f91a400e9f0 tx=0x7f91a400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:37:20.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.356+0000 7f91c0a00640 1 -- 172.21.15.90:0/3972937747 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91a400cd80 con 0x7f91bc100970 2024-04-02T14:37:20.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.356+0000 7f91c0a00640 1 -- 172.21.15.90:0/3972937747 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f91a4010910 con 0x7f91bc100970 2024-04-02T14:37:20.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.356+0000 7f91c0a00640 1 -- 172.21.15.90:0/3972937747 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91a4018980 con 0x7f91bc100970 2024-04-02T14:37:20.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.356+0000 7f91c1e00640 1 -- 172.21.15.90:0/3972937747 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f91bc19f550 con 0x7f91bc100970 2024-04-02T14:37:20.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.357+0000 7f91c1e00640 1 -- 172.21.15.90:0/3972937747 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f91bc1ab610 con 0x7f91bc100970 2024-04-02T14:37:20.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.365+0000 7f91c4a27740 1 Processor -- start 2024-04-02T14:37:20.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.367+0000 7f91c0a00640 1 -- 172.21.15.90:0/3972937747 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f91a4010ac0 con 0x7f91bc100970 2024-04-02T14:37:20.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.368+0000 7f91c4a27740 1 -- start start 2024-04-02T14:37:20.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.368+0000 7f91c4a27740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55869b2549f0 0x55869b253740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:20.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.368+0000 7f91c4a27740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55869b253d10 0x55869b254190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:20.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.368+0000 7f91c4a27740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55869b25d080 0x55869b25f470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:20.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.368+0000 7f91c4a27740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55869b194400 con 0x55869b2549f0 2024-04-02T14:37:20.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.368+0000 7f91c4a27740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55869b178f10 con 0x55869b253d10 2024-04-02T14:37:20.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.368+0000 7f91c4a27740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55869b1b86f0 con 0x55869b25d080 2024-04-02T14:37:20.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.369+0000 7f91bb400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55869b2549f0 0x55869b253740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:20.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.369+0000 7f91bb400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55869b2549f0 0x55869b253740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:36554/0 (socket says 172.21.15.90:36554) 2024-04-02T14:37:20.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.369+0000 7f91bb400640 1 -- 172.21.15.90:0/2160958247 learned_addr learned my addr 172.21.15.90:0/2160958247 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:37:20.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.369+0000 7f91ba000640 1 --2- 172.21.15.90:0/2160958247 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55869b25d080 0x55869b25f470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:20.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.369+0000 7f91ba000640 1 -- 172.21.15.90:0/2160958247 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55869b253d10 msgr2=0x55869b254190 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-02T14:37:20.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.370+0000 7f91ba000640 1 --2- 172.21.15.90:0/2160958247 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55869b253d10 0x55869b254190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:20.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.370+0000 7f91ba000640 1 -- 172.21.15.90:0/2160958247 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55869b2549f0 msgr2=0x55869b253740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:37:20.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.370+0000 7f91ba000640 1 --2- 172.21.15.90:0/2160958247 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55869b2549f0 0x55869b253740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:20.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.370+0000 7f91ba000640 1 -- 172.21.15.90:0/2160958247 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55869aecc3f0 con 0x55869b25d080 2024-04-02T14:37:20.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.370+0000 7f91bb400640 1 --2- 172.21.15.90:0/2160958247 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55869b2549f0 0x55869b253740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:37:20.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.370+0000 7f91ba000640 1 --2- 172.21.15.90:0/2160958247 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55869b25d080 0x55869b25f470 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f91bc19ff10 tx=0x7f91ac011c70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f45ba19e8634a01a server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:37:20.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.371+0000 7f91abe00640 1 -- 172.21.15.90:0/2160958247 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91ac004d20 con 0x55869b25d080 2024-04-02T14:37:20.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.371+0000 7f91abe00640 1 -- 172.21.15.90:0/2160958247 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f91ac003970 con 0x55869b25d080 2024-04-02T14:37:20.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.371+0000 7f91abe00640 1 -- 172.21.15.90:0/2160958247 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91ac004450 con 0x55869b25d080 2024-04-02T14:37:20.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.373+0000 7f91c0a00640 1 --2- 172.21.15.90:0/3972937747 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f91a00a3f20 0x7f91a00a63e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:20.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.373+0000 7f91c0a00640 1 -- 172.21.15.90:0/3972937747 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f91a40d0a70 con 0x7f91bc100970 2024-04-02T14:37:20.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.373+0000 7f91c4a27740 1 -- 172.21.15.90:0/2160958247 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55869b25d080 msgr2=0x55869b25f470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:20.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.373+0000 7f91c4a27740 1 --2- 172.21.15.90:0/2160958247 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55869b25d080 0x55869b25f470 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f91bc19ff10 tx=0x7f91ac011c70 comp rx=0 tx=0).stop 2024-04-02T14:37:20.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.374+0000 7f91c4a27740 1 -- 172.21.15.90:0/2160958247 shutdown_connections 2024-04-02T14:37:20.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.374+0000 7f91c4a27740 1 --2- 172.21.15.90:0/2160958247 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55869b25d080 0x55869b25f470 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:20.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.374+0000 7f91c4a27740 1 --2- 172.21.15.90:0/2160958247 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55869b253d10 0x55869b254190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:20.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.374+0000 7f91c4a27740 1 --2- 172.21.15.90:0/2160958247 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55869b2549f0 0x55869b253740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:20.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.374+0000 7f91c4a27740 1 -- 172.21.15.90:0/2160958247 >> 172.21.15.90:0/2160958247 conn(0x55869b21aa50 msgr2=0x55869b24e760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:37:20.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.374+0000 7f91bb400640 1 --2- 172.21.15.90:0/3972937747 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f91a00a3f20 0x7f91a00a63e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:20.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.375+0000 7f91c4a27740 1 -- 172.21.15.90:0/2160958247 shutdown_connections 2024-04-02T14:37:20.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.375+0000 7f91c4a27740 1 -- 172.21.15.90:0/2160958247 wait complete. 2024-04-02T14:37:20.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.375+0000 7f91c4a27740 1 -- start start 2024-04-02T14:37:20.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.376+0000 7f91c4a27740 1 Processor -- start 2024-04-02T14:37:20.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.376+0000 7f91bb400640 1 --2- 172.21.15.90:0/3972937747 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f91a00a3f20 0x7f91a00a63e0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f91b40096c0 tx=0x7f91b402e040 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:37:20.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.378+0000 7f91c4a27740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55869b253d10 0x55869b25a060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:20.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.378+0000 7f91c4a27740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55869b2549f0 0x55869b25a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:20.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.379+0000 7f91c4a27740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55869b25d080 0x55869b25aae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:20.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.379+0000 7f91baa00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55869b253d10 0x55869b25a060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:20.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.379+0000 7f91baa00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55869b253d10 0x55869b25a060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:58298/0 (socket says 172.21.15.90:58298) 2024-04-02T14:37:20.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.379+0000 7f91c4a27740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55869b1b46a0 con 0x55869b25d080 2024-04-02T14:37:20.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.379+0000 7f91c4a27740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55869b160980 con 0x55869b2549f0 2024-04-02T14:37:20.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.379+0000 7f91c4a27740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55869b173cb0 con 0x55869b253d10 2024-04-02T14:37:20.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.379+0000 7f91baa00640 1 -- 172.21.15.90:0/397739625 learned_addr learned my addr 172.21.15.90:0/397739625 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:37:20.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.379+0000 7f91c4a27740 10 client.0 _ll_register_callbacks cb 0x55869b1736a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:37:20.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.379+0000 7f91bb400640 1 --2- 172.21.15.90:0/397739625 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55869b25d080 0x55869b25aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:20.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.380+0000 7f91baa00640 1 -- 172.21.15.90:0/397739625 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55869b2549f0 msgr2=0x55869b25a5a0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-02T14:37:20.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.380+0000 7f91baa00640 1 --2- 172.21.15.90:0/397739625 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55869b2549f0 0x55869b25a5a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:20.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.380+0000 7f91baa00640 1 -- 172.21.15.90:0/397739625 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55869b25d080 msgr2=0x55869b25aae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:37:20.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.380+0000 7f91baa00640 1 --2- 172.21.15.90:0/397739625 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55869b25d080 0x55869b25aae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:20.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.380+0000 7f91baa00640 1 -- 172.21.15.90:0/397739625 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55869af967d0 con 0x55869b253d10 2024-04-02T14:37:20.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.380+0000 7f91baa00640 1 --2- 172.21.15.90:0/397739625 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55869b253d10 0x55869b25a060 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f91a400c6a0 tx=0x7f91a409c5f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=829c4f0a3da34868 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:37:20.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.380+0000 7f91a8c00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:37:20.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.381+0000 7f91a8c00640 1 -- 172.21.15.90:0/397739625 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91a40bbd80 con 0x55869b253d10 2024-04-02T14:37:20.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.381+0000 7f91a8c00640 1 -- 172.21.15.90:0/397739625 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f91a409be10 con 0x55869b253d10 2024-04-02T14:37:20.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.381+0000 7f91c4a27740 1 -- 172.21.15.90:0/397739625 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f91ac003970 con 0x55869b253d10 2024-04-02T14:37:20.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.382+0000 7f91a8c00640 1 -- 172.21.15.90:0/397739625 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f91a40a0d70 con 0x55869b253d10 2024-04-02T14:37:20.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.382+0000 7f91c4a27740 10 client.26089 fetch_fsmap learned FSMap version 27 2024-04-02T14:37:20.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.382+0000 7f91c4a27740 1 -- 172.21.15.90:0/397739625 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55869b26bf80 con 0x55869b253d10 2024-04-02T14:37:20.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.382+0000 7f91a8c00640 1 -- 172.21.15.90:0/397739625 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f91ac003970 con 0x55869b253d10 2024-04-02T14:37:20.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.383+0000 7f91a8c00640 1 -- 172.21.15.90:0/397739625 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f91a4010550 con 0x55869b253d10 2024-04-02T14:37:20.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.383+0000 7f91c4a27740 10 client.26089 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:37:20.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.383+0000 7f91c4a27740 10 client.26089 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2024-04-02T14:37:20.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.383+0000 7f91c4a27740 20 client.26089 populate_metadata read hostname 'smithi090' 2024-04-02T14:37:20.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.383+0000 7f91c4a27740 1 --2- 172.21.15.90:0/397739625 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x55869b26d4e0 0x55869b26f950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:20.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.383+0000 7f91c4a27740 4 client.26089 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:37:20.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.383+0000 7f91c4a27740 1 -- 172.21.15.90:0/397739625 --> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55869b178040 con 0x55869b26d4e0 2024-04-02T14:37:20.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.384+0000 7f91ba000640 1 --2- 172.21.15.90:0/397739625 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x55869b26d4e0 0x55869b26f950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:20.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.386+0000 7f91ba000640 1 --2- 172.21.15.90:0/397739625 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x55869b26d4e0 0x55869b26f950 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=372592392ab06cfd server_cookie=f710c2c27e85a349 in_seq=0 out_seq=0 2024-04-02T14:37:20.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.386+0000 7f91a8c00640 10 client.26089 ms_handle_connect on v2:172.21.15.90:6838/1498915870 2024-04-02T14:37:20.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.387+0000 7f91a8c00640 1 -- 172.21.15.90:0/397739625 <== mds.1 v2:172.21.15.90:6838/1498915870 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55869b178040 con 0x55869b26d4e0 2024-04-02T14:37:20.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.387+0000 7f91a8c00640 10 client.26089 handle_command_reply: tid=0 2024-04-02T14:37:20.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.387+0000 7f91c4a27740 1 client.26089 shutdown 2024-04-02T14:37:20.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.388+0000 7f91c4a27740 1 -- 172.21.15.90:0/397739625 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55869b253d10 msgr2=0x55869b25a060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:20.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.388+0000 7f91c4a27740 1 --2- 172.21.15.90:0/397739625 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55869b253d10 0x55869b25a060 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f91a400c6a0 tx=0x7f91a409c5f0 comp rx=0 tx=0).stop 2024-04-02T14:37:20.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.388+0000 7f91c4a27740 1 -- 172.21.15.90:0/397739625 shutdown_connections 2024-04-02T14:37:20.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.388+0000 7f91c4a27740 1 --2- 172.21.15.90:0/397739625 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x55869b26d4e0 0x55869b26f950 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:20.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.388+0000 7f91a8c00640 0 client.26089 ms_handle_reset on v2:172.21.15.90:6838/1498915870 2024-04-02T14:37:20.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.388+0000 7f91c4a27740 1 --2- 172.21.15.90:0/397739625 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55869b25d080 0x55869b25aae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:20.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.388+0000 7f91c4a27740 1 --2- 172.21.15.90:0/397739625 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55869b2549f0 0x55869b25a5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:20.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.388+0000 7f91c4a27740 1 --2- 172.21.15.90:0/397739625 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55869b253d10 0x55869b25a060 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:20.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.388+0000 7f91c4a27740 1 -- 172.21.15.90:0/397739625 >> 172.21.15.90:0/397739625 conn(0x55869b21aa50 msgr2=0x55869b2504b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:37:20.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.389+0000 7f91c4a27740 1 -- 172.21.15.90:0/397739625 shutdown_connections 2024-04-02T14:37:20.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.389+0000 7f91c4a27740 1 -- 172.21.15.90:0/397739625 wait complete. 2024-04-02T14:37:20.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.389+0000 7f91c4a27740 20 client.26089 trim_cache size 0 max 16384 2024-04-02T14:37:20.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.425+0000 7f91c4a27740 1 Processor -- start 2024-04-02T14:37:20.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.425+0000 7f91c4a27740 1 -- start start 2024-04-02T14:37:20.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.427+0000 7f91c4a27740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55869b253d10 0x55869b268b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:20.429 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.427+0000 7f91c4a27740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55869b2549f0 0x55869b25a900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:20.429 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.428+0000 7f91baa00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55869b253d10 0x55869b268b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:20.429 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.428+0000 7f91baa00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55869b253d10 0x55869b268b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:48860/0 (socket says 172.21.15.90:48860) 2024-04-02T14:37:20.429 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.428+0000 7f91c4a27740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55869b25d080 0x55869b24d720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:20.429 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.428+0000 7f91baa00640 1 -- 172.21.15.90:0/2905103018 learned_addr learned my addr 172.21.15.90:0/2905103018 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:37:20.429 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.428+0000 7f91c4a27740 1 -- 172.21.15.90:0/2905103018 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f91bc1b3f10 con 0x55869b25d080 2024-04-02T14:37:20.429 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.428+0000 7f91c4a27740 1 -- 172.21.15.90:0/2905103018 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f91bc1b5a50 con 0x55869b253d10 2024-04-02T14:37:20.429 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.428+0000 7f91c4a27740 1 -- 172.21.15.90:0/2905103018 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f91bc1b45e0 con 0x55869b2549f0 2024-04-02T14:37:20.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.428+0000 7f91ba000640 1 --2- 172.21.15.90:0/2905103018 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55869b2549f0 0x55869b25a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:20.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.429+0000 7f91bb400640 1 --2- 172.21.15.90:0/2905103018 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55869b25d080 0x55869b24d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:20.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.430+0000 7f91baa00640 1 -- 172.21.15.90:0/2905103018 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55869b2549f0 msgr2=0x55869b25a900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:37:20.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.430+0000 7f91baa00640 1 --2- 172.21.15.90:0/2905103018 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55869b2549f0 0x55869b25a900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:20.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.430+0000 7f91baa00640 1 -- 172.21.15.90:0/2905103018 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55869b25d080 msgr2=0x55869b24d720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:37:20.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.430+0000 7f91baa00640 1 --2- 172.21.15.90:0/2905103018 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55869b25d080 0x55869b24d720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:20.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.430+0000 7f91baa00640 1 -- 172.21.15.90:0/2905103018 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55869b252d40 con 0x55869b253d10 2024-04-02T14:37:20.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.430+0000 7f91bb400640 1 --2- 172.21.15.90:0/2905103018 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55869b25d080 0x55869b24d720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:37:20.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.430+0000 7f91baa00640 1 --2- 172.21.15.90:0/2905103018 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55869b253d10 0x55869b268b50 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f91a40a5100 tx=0x7f91a401f000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=565971663ae39a4d server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:37:20.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.432+0000 7f91ab400640 1 -- 172.21.15.90:0/2905103018 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91a40bbd80 con 0x55869b253d10 2024-04-02T14:37:20.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.432+0000 7f91ab400640 1 -- 172.21.15.90:0/2905103018 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f91a409d0c0 con 0x55869b253d10 2024-04-02T14:37:20.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.432+0000 7f91ab400640 1 -- 172.21.15.90:0/2905103018 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91a409b880 con 0x55869b253d10 2024-04-02T14:37:20.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.433+0000 7f91c4a27740 1 -- 172.21.15.90:0/2905103018 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55869b253d10 msgr2=0x55869b268b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:20.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.433+0000 7f91c4a27740 1 --2- 172.21.15.90:0/2905103018 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55869b253d10 0x55869b268b50 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f91a40a5100 tx=0x7f91a401f000 comp rx=0 tx=0).stop 2024-04-02T14:37:20.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.434+0000 7f91c4a27740 1 -- 172.21.15.90:0/2905103018 shutdown_connections 2024-04-02T14:37:20.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.434+0000 7f91c4a27740 1 --2- 172.21.15.90:0/2905103018 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55869b25d080 0x55869b24d720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:20.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.434+0000 7f91c4a27740 1 --2- 172.21.15.90:0/2905103018 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55869b2549f0 0x55869b25a900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:20.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.434+0000 7f91c4a27740 1 --2- 172.21.15.90:0/2905103018 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55869b253d10 0x55869b268b50 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:20.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.434+0000 7f91c4a27740 1 -- 172.21.15.90:0/2905103018 >> 172.21.15.90:0/2905103018 conn(0x55869b21aa50 msgr2=0x55869b24d010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:37:20.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.434+0000 7f91c4a27740 1 -- 172.21.15.90:0/2905103018 shutdown_connections 2024-04-02T14:37:20.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.434+0000 7f91c4a27740 1 -- 172.21.15.90:0/2905103018 wait complete. 2024-04-02T14:37:20.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.435+0000 7f91c4a27740 1 -- start start 2024-04-02T14:37:20.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.436+0000 7f91c4a27740 1 Processor -- start 2024-04-02T14:37:20.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.437+0000 7f91c4a27740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55869b253d10 0x55869b25d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:20.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.437+0000 7f91c4a27740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55869b2549f0 0x55869b25d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:20.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.437+0000 7f91c4a27740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55869b25d080 0x55869b25df20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:20.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.437+0000 7f91c4a27740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f91bc1b5380 con 0x55869b253d10 2024-04-02T14:37:20.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.437+0000 7f91c4a27740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f91bc1b4cb0 con 0x55869b2549f0 2024-04-02T14:37:20.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.437+0000 7f91c4a27740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f91bc1b1ef0 con 0x55869b25d080 2024-04-02T14:37:20.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.437+0000 7f91c4a27740 10 client.0 _ll_register_callbacks cb 0x7f91a4098d50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:37:20.440 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.438+0000 7f91baa00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55869b253d10 0x55869b25d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:20.440 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.438+0000 7f91baa00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55869b253d10 0x55869b25d4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:36584/0 (socket says 172.21.15.90:36584) 2024-04-02T14:37:20.440 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.438+0000 7f91baa00640 1 -- 172.21.15.90:0/3808217610 learned_addr learned my addr 172.21.15.90:0/3808217610 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:37:20.440 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.438+0000 7f91ba000640 1 --2- 172.21.15.90:0/3808217610 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55869b2549f0 0x55869b25d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:20.440 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.439+0000 7f91baa00640 1 -- 172.21.15.90:0/3808217610 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55869b25d080 msgr2=0x55869b25df20 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-02T14:37:20.440 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.439+0000 7f91baa00640 1 --2- 172.21.15.90:0/3808217610 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55869b25d080 0x55869b25df20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:20.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.439+0000 7f91baa00640 1 -- 172.21.15.90:0/3808217610 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55869b2549f0 msgr2=0x55869b25d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:37:20.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.439+0000 7f91baa00640 1 --2- 172.21.15.90:0/3808217610 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55869b2549f0 0x55869b25d9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:20.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.439+0000 7f91baa00640 1 -- 172.21.15.90:0/3808217610 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55869b264350 con 0x55869b253d10 2024-04-02T14:37:20.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.439+0000 7f91baa00640 1 --2- 172.21.15.90:0/3808217610 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55869b253d10 0x55869b25d4a0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f91a409cb80 tx=0x7f91a4018420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9cec73525b913d0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:37:20.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.439+0000 7f91a8c00640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-02T14:37:20.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.440+0000 7f91a8c00640 1 -- 172.21.15.90:0/3808217610 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91a40a9070 con 0x55869b253d10 2024-04-02T14:37:20.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.440+0000 7f91a8c00640 1 -- 172.21.15.90:0/3808217610 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f91a40bbd80 con 0x55869b253d10 2024-04-02T14:37:20.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.440+0000 7f91c4a27740 1 -- 172.21.15.90:0/3808217610 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f91a409d0c0 con 0x55869b253d10 2024-04-02T14:37:20.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.440+0000 7f91a8c00640 1 -- 172.21.15.90:0/3808217610 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f91a408c050 con 0x55869b253d10 2024-04-02T14:37:20.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.440+0000 7f91c4a27740 10 client.15873 fetch_fsmap learned FSMap version 27 2024-04-02T14:37:20.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.440+0000 7f91c4a27740 1 -- 172.21.15.90:0/3808217610 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55869b264350 con 0x55869b253d10 2024-04-02T14:37:20.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.441+0000 7f91a8c00640 1 -- 172.21.15.90:0/3808217610 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f91a409d0c0 con 0x55869b253d10 2024-04-02T14:37:20.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.442+0000 7f91a8c00640 1 -- 172.21.15.90:0/3808217610 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f91a409b880 con 0x55869b253d10 2024-04-02T14:37:20.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.442+0000 7f91c4a27740 10 client.15873 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:37:20.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.442+0000 7f91c4a27740 10 client.15873 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2024-04-02T14:37:20.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.442+0000 7f91c4a27740 20 client.15873 populate_metadata read hostname 'smithi090' 2024-04-02T14:37:20.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.442+0000 7f91c4a27740 1 --2- 172.21.15.90:0/3808217610 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x55869b26d4e0 0x55869b2d5aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:20.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.442+0000 7f91c4a27740 4 client.15873 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-02T14:37:20.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.442+0000 7f91c4a27740 1 -- 172.21.15.90:0/3808217610 --> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55869b1c5c30 con 0x55869b26d4e0 2024-04-02T14:37:20.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.445+0000 7f91ba000640 1 --2- 172.21.15.90:0/3808217610 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x55869b26d4e0 0x55869b2d5aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:20.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.447+0000 7f91ba000640 1 --2- 172.21.15.90:0/3808217610 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x55869b26d4e0 0x55869b2d5aa0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=9e95116c9f043eb8 server_cookie=67ba9533f64a6fad in_seq=0 out_seq=0 2024-04-02T14:37:20.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.447+0000 7f91a8c00640 10 client.15873 ms_handle_connect on v2:172.21.15.90:6838/1498915870 2024-04-02T14:37:20.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.449+0000 7f91a8c00640 1 -- 172.21.15.90:0/3808217610 <== mds.1 v2:172.21.15.90:6838/1498915870 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55869b1c5c30 con 0x55869b26d4e0 2024-04-02T14:37:20.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.449+0000 7f91a8c00640 10 client.15873 handle_command_reply: tid=0 2024-04-02T14:37:20.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.449+0000 7f91c4a27740 1 client.15873 shutdown 2024-04-02T14:37:20.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.449+0000 7f91c4a27740 1 -- 172.21.15.90:0/3808217610 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55869b253d10 msgr2=0x55869b25d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:20.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.449+0000 7f91c4a27740 1 --2- 172.21.15.90:0/3808217610 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55869b253d10 0x55869b25d4a0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f91a409cb80 tx=0x7f91a4018420 comp rx=0 tx=0).stop 2024-04-02T14:37:20.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.450+0000 7f91c4a27740 1 -- 172.21.15.90:0/3808217610 shutdown_connections 2024-04-02T14:37:20.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.450+0000 7f91c4a27740 1 --2- 172.21.15.90:0/3808217610 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x55869b26d4e0 0x55869b2d5aa0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:20.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.450+0000 7f91a8c00640 0 client.15873 ms_handle_reset on v2:172.21.15.90:6838/1498915870 2024-04-02T14:37:20.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.450+0000 7f91c4a27740 1 --2- 172.21.15.90:0/3808217610 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55869b25d080 0x55869b25df20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:20.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.450+0000 7f91c4a27740 1 --2- 172.21.15.90:0/3808217610 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55869b2549f0 0x55869b25d9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:20.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.450+0000 7f91c4a27740 1 --2- 172.21.15.90:0/3808217610 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55869b253d10 0x55869b25d4a0 secure :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f91a409cb80 tx=0x7f91a4018420 comp rx=0 tx=0).stop 2024-04-02T14:37:20.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.450+0000 7f91c4a27740 1 -- 172.21.15.90:0/3808217610 >> 172.21.15.90:0/3808217610 conn(0x55869b21aa50 msgr2=0x55869b24d8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:37:20.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.452+0000 7f91c4a27740 1 -- 172.21.15.90:0/3808217610 shutdown_connections 2024-04-02T14:37:20.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.452+0000 7f91c4a27740 1 -- 172.21.15.90:0/3808217610 wait complete. 2024-04-02T14:37:20.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.452+0000 7f91c4a27740 20 client.15873 trim_cache size 0 max 16384 2024-04-02T14:37:20.455 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-02T14:37:20.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.456+0000 7f91c1e00640 1 -- 172.21.15.90:0/3972937747 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f91a00a3f20 msgr2=0x7f91a00a63e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:20.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.456+0000 7f91c1e00640 1 --2- 172.21.15.90:0/3972937747 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f91a00a3f20 0x7f91a00a63e0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f91b40096c0 tx=0x7f91b402e040 comp rx=0 tx=0).stop 2024-04-02T14:37:20.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.456+0000 7f91c1e00640 1 -- 172.21.15.90:0/3972937747 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f91bc100970 msgr2=0x7f91bc1a4690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:20.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.456+0000 7f91c1e00640 1 --2- 172.21.15.90:0/3972937747 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f91bc100970 0x7f91bc1a4690 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f91a400e9f0 tx=0x7f91a400eeb0 comp rx=0 tx=0).stop 2024-04-02T14:37:20.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.457+0000 7f91c1e00640 1 -- 172.21.15.90:0/3972937747 shutdown_connections 2024-04-02T14:37:20.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.457+0000 7f91c1e00640 1 --2- 172.21.15.90:0/3972937747 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f91a00a3f20 0x7f91a00a63e0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:20.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.457+0000 7f91c1e00640 1 --2- 172.21.15.90:0/3972937747 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f91bc106980 0x7f91bc19ea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:20.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.457+0000 7f91c1e00640 1 --2- 172.21.15.90:0/3972937747 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f91bc101330 0x7f91bc1a4bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:20.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.457+0000 7f91c1e00640 1 --2- 172.21.15.90:0/3972937747 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f91bc100970 0x7f91bc1a4690 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:20.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.457+0000 7f91c1e00640 1 -- 172.21.15.90:0/3972937747 >> 172.21.15.90:0/3972937747 conn(0x7f91bc077b00 msgr2=0x7f91bc068d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:37:20.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.458+0000 7f91c1e00640 1 -- 172.21.15.90:0/3972937747 shutdown_connections 2024-04-02T14:37:20.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.458+0000 7f91c1e00640 1 -- 172.21.15.90:0/3972937747 wait complete. 2024-04-02T14:37:20.492 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-02T14:37:20.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.801+0000 7fd968800640 1 -- 172.21.15.90:0/3248916486 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd960076950 msgr2=0x7fd960112400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:20.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.801+0000 7fd968800640 1 --2- 172.21.15.90:0/3248916486 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd960076950 0x7fd960112400 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fd950009930 tx=0x7fd95002bc50 comp rx=0 tx=0).stop 2024-04-02T14:37:20.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.801+0000 7fd968800640 1 -- 172.21.15.90:0/3248916486 shutdown_connections 2024-04-02T14:37:20.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.801+0000 7fd968800640 1 --2- 172.21.15.90:0/3248916486 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd960112940 0x7fd960114d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:20.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.801+0000 7fd968800640 1 --2- 172.21.15.90:0/3248916486 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd960076950 0x7fd960112400 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:20.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.801+0000 7fd968800640 1 --2- 172.21.15.90:0/3248916486 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd960076030 0x7fd960076410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:20.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.801+0000 7fd968800640 1 -- 172.21.15.90:0/3248916486 >> 172.21.15.90:0/3248916486 conn(0x7fd9600ff7b0 msgr2=0x7fd960101bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:37:20.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.801+0000 7fd968800640 1 -- 172.21.15.90:0/3248916486 shutdown_connections 2024-04-02T14:37:20.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.801+0000 7fd968800640 1 -- 172.21.15.90:0/3248916486 wait complete. 2024-04-02T14:37:20.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.803+0000 7fd968800640 1 Processor -- start 2024-04-02T14:37:20.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.803+0000 7fd968800640 1 -- start start 2024-04-02T14:37:20.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.804+0000 7fd968800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd960076030 0x7fd9601a46c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:20.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.804+0000 7fd968800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd960076950 0x7fd9601a4c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:20.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.804+0000 7fd968800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd960112940 0x7fd96019e8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:20.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.804+0000 7fd968800640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fd9601184e0 con 0x7fd960076950 2024-04-02T14:37:20.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.804+0000 7fd968800640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fd960118360 con 0x7fd960076030 2024-04-02T14:37:20.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.804+0000 7fd968800640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fd960118660 con 0x7fd960112940 2024-04-02T14:37:20.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.804+0000 7fd966800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd960112940 0x7fd96019e8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:20.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.804+0000 7fd966800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd960112940 0x7fd96019e8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:58336/0 (socket says 172.21.15.90:58336) 2024-04-02T14:37:20.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.804+0000 7fd966800640 1 -- 172.21.15.90:0/3982949374 learned_addr learned my addr 172.21.15.90:0/3982949374 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:37:20.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.805+0000 7fd965400640 1 --2- 172.21.15.90:0/3982949374 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd960076950 0x7fd9601a4c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:20.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.804+0000 7fd965e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd960076030 0x7fd9601a46c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:20.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.805+0000 7fd966800640 1 -- 172.21.15.90:0/3982949374 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd960076030 msgr2=0x7fd9601a46c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:20.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.805+0000 7fd966800640 1 --2- 172.21.15.90:0/3982949374 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd960076030 0x7fd9601a46c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:20.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.805+0000 7fd966800640 1 -- 172.21.15.90:0/3982949374 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd960076950 msgr2=0x7fd9601a4c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:20.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.805+0000 7fd966800640 1 --2- 172.21.15.90:0/3982949374 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd960076950 0x7fd9601a4c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:20.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.805+0000 7fd966800640 1 -- 172.21.15.90:0/3982949374 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd96019f040 con 0x7fd960112940 2024-04-02T14:37:20.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.805+0000 7fd965400640 1 --2- 172.21.15.90:0/3982949374 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd960076950 0x7fd9601a4c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:37:20.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.805+0000 7fd965e00640 1 --2- 172.21.15.90:0/3982949374 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd960076030 0x7fd9601a46c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:37:20.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.806+0000 7fd966800640 1 --2- 172.21.15.90:0/3982949374 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd960112940 0x7fd96019e8c0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fd95c00e9d0 tx=0x7fd95c00ee90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:37:20.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.806+0000 7fd95aa00640 1 -- 172.21.15.90:0/3982949374 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd95c00cd80 con 0x7fd960112940 2024-04-02T14:37:20.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.806+0000 7fd95aa00640 1 -- 172.21.15.90:0/3982949374 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd95c005020 con 0x7fd960112940 2024-04-02T14:37:20.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.806+0000 7fd95aa00640 1 -- 172.21.15.90:0/3982949374 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd95c0116d0 con 0x7fd960112940 2024-04-02T14:37:20.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.806+0000 7fd968800640 1 -- 172.21.15.90:0/3982949374 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd96019f330 con 0x7fd960112940 2024-04-02T14:37:20.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.807+0000 7fd968800640 1 -- 172.21.15.90:0/3982949374 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9601ab580 con 0x7fd960112940 2024-04-02T14:37:20.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.811+0000 7fd95aa00640 1 -- 172.21.15.90:0/3982949374 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fd95c0051d0 con 0x7fd960112940 2024-04-02T14:37:20.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.815+0000 7fd95aa00640 1 --2- 172.21.15.90:0/3982949374 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fd9480a3e20 0x7fd9480a62e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:20.817 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.815+0000 7fd95aa00640 1 -- 172.21.15.90:0/3982949374 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7fd95c0d0db0 con 0x7fd960112940 2024-04-02T14:37:20.817 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.815+0000 7fd965e00640 1 --2- 172.21.15.90:0/3982949374 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fd9480a3e20 0x7fd9480a62e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:20.818 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.817+0000 7fd965e00640 1 --2- 172.21.15.90:0/3982949374 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fd9480a3e20 0x7fd9480a62e0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fd960102f50 tx=0x7fd954005f90 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:37:20.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.823+0000 7fd96b5f0740 1 Processor -- start 2024-04-02T14:37:20.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.824+0000 7fd96b5f0740 1 -- start start 2024-04-02T14:37:20.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.824+0000 7fd96b5f0740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c7623819f0 0x55c762380740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:20.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.824+0000 7fd96b5f0740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c762380d10 0x55c762381190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:20.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.824+0000 7fd96b5f0740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c76238a080 0x55c76238c470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:20.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.824+0000 7fd96b5f0740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55c7622c1400 con 0x55c7623819f0 2024-04-02T14:37:20.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.824+0000 7fd96b5f0740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55c7622a5f10 con 0x55c762380d10 2024-04-02T14:37:20.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.824+0000 7fd96b5f0740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55c7622e56f0 con 0x55c76238a080 2024-04-02T14:37:20.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.825+0000 7fd965400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c762380d10 0x55c762381190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:20.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.825+0000 7fd965400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c762380d10 0x55c762381190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:48904/0 (socket says 172.21.15.90:48904) 2024-04-02T14:37:20.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.825+0000 7fd965400640 1 -- 172.21.15.90:0/2319819918 learned_addr learned my addr 172.21.15.90:0/2319819918 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:37:20.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.825+0000 7fd965e00640 1 --2- 172.21.15.90:0/2319819918 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c7623819f0 0x55c762380740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:20.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.825+0000 7fd966800640 1 --2- 172.21.15.90:0/2319819918 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c76238a080 0x55c76238c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:20.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.826+0000 7fd965e00640 1 -- 172.21.15.90:0/2319819918 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c76238a080 msgr2=0x55c76238c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:37:20.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.826+0000 7fd965e00640 1 --2- 172.21.15.90:0/2319819918 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c76238a080 0x55c76238c470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:20.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.826+0000 7fd965e00640 1 -- 172.21.15.90:0/2319819918 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c762380d10 msgr2=0x55c762381190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:37:20.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.826+0000 7fd965e00640 1 --2- 172.21.15.90:0/2319819918 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c762380d10 0x55c762381190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:20.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.826+0000 7fd965e00640 1 -- 172.21.15.90:0/2319819918 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c761ff93f0 con 0x55c7623819f0 2024-04-02T14:37:20.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.826+0000 7fd965400640 1 --2- 172.21.15.90:0/2319819918 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c762380d10 0x55c762381190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:37:20.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.827+0000 7fd965e00640 1 --2- 172.21.15.90:0/2319819918 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c7623819f0 0x55c762380740 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fd954011870 tx=0x7fd954014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9046b37f624c80e4 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:37:20.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.828+0000 7fd94f000640 1 -- 172.21.15.90:0/2319819918 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd954012e60 con 0x55c7623819f0 2024-04-02T14:37:20.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.828+0000 7fd94f000640 1 -- 172.21.15.90:0/2319819918 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd95401fb00 con 0x55c7623819f0 2024-04-02T14:37:20.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.828+0000 7fd94f000640 1 -- 172.21.15.90:0/2319819918 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd954027c80 con 0x55c7623819f0 2024-04-02T14:37:20.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.830+0000 7fd96b5f0740 1 -- 172.21.15.90:0/2319819918 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c7623819f0 msgr2=0x55c762380740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:20.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.830+0000 7fd96b5f0740 1 --2- 172.21.15.90:0/2319819918 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c7623819f0 0x55c762380740 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fd954011870 tx=0x7fd954014000 comp rx=0 tx=0).stop 2024-04-02T14:37:20.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.831+0000 7fd96b5f0740 1 -- 172.21.15.90:0/2319819918 shutdown_connections 2024-04-02T14:37:20.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.831+0000 7fd96b5f0740 1 --2- 172.21.15.90:0/2319819918 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c76238a080 0x55c76238c470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:20.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.831+0000 7fd96b5f0740 1 --2- 172.21.15.90:0/2319819918 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c762380d10 0x55c762381190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:20.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.831+0000 7fd96b5f0740 1 --2- 172.21.15.90:0/2319819918 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c7623819f0 0x55c762380740 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:20.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.831+0000 7fd96b5f0740 1 -- 172.21.15.90:0/2319819918 >> 172.21.15.90:0/2319819918 conn(0x55c762347a50 msgr2=0x55c76237b760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:37:20.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.831+0000 7fd96b5f0740 1 -- 172.21.15.90:0/2319819918 shutdown_connections 2024-04-02T14:37:20.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.831+0000 7fd96b5f0740 1 -- 172.21.15.90:0/2319819918 wait complete. 2024-04-02T14:37:20.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.832+0000 7fd96b5f0740 1 -- start start 2024-04-02T14:37:20.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.832+0000 7fd96b5f0740 1 Processor -- start 2024-04-02T14:37:20.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.833+0000 7fd96b5f0740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c762380d10 0x55c7623870d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:20.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.833+0000 7fd96b5f0740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c7623819f0 0x55c762387610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:20.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.833+0000 7fd96b5f0740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c76238a080 0x55c762387b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:20.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.834+0000 7fd96b5f0740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55c7622e16a0 con 0x55c762380d10 2024-04-02T14:37:20.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.834+0000 7fd96b5f0740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55c76228d980 con 0x55c7623819f0 2024-04-02T14:37:20.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.834+0000 7fd96b5f0740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55c7622a0cb0 con 0x55c76238a080 2024-04-02T14:37:20.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.834+0000 7fd966800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c76238a080 0x55c762387b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:20.840 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.834+0000 7fd966800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c76238a080 0x55c762387b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:58348/0 (socket says 172.21.15.90:58348) 2024-04-02T14:37:20.840 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.834+0000 7fd966800640 1 -- 172.21.15.90:0/2187839894 learned_addr learned my addr 172.21.15.90:0/2187839894 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:37:20.840 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.834+0000 7fd965400640 1 --2- 172.21.15.90:0/2187839894 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c7623819f0 0x55c762387610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:20.840 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.834+0000 7fd966800640 1 -- 172.21.15.90:0/2187839894 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c7623819f0 msgr2=0x55c762387610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:37:20.840 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.834+0000 7fd966800640 1 --2- 172.21.15.90:0/2187839894 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c7623819f0 0x55c762387610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:20.840 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.834+0000 7fd966800640 1 -- 172.21.15.90:0/2187839894 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c762380d10 msgr2=0x55c7623870d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-02T14:37:20.840 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.835+0000 7fd96b5f0740 10 client.0 _ll_register_callbacks cb 0x55c7622a06a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:37:20.840 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.835+0000 7fd965e00640 1 --2- 172.21.15.90:0/2187839894 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c762380d10 0x55c7623870d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:20.840 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.835+0000 7fd966800640 1 --2- 172.21.15.90:0/2187839894 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c762380d10 0x55c7623870d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:20.840 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.835+0000 7fd966800640 1 -- 172.21.15.90:0/2187839894 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c7620c37d0 con 0x55c76238a080 2024-04-02T14:37:20.840 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.836+0000 7fd96b5f0740 1 -- 172.21.15.90:0/2187839894 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd95401fb00 con 0x55c76238a080 2024-04-02T14:37:20.840 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.836+0000 7fd965e00640 1 --2- 172.21.15.90:0/2187839894 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c762380d10 0x55c7623870d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:37:20.841 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.836+0000 7fd966800640 1 --2- 172.21.15.90:0/2187839894 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c76238a080 0x55c762387b50 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fd95c0ce510 tx=0x7fd95c09d530 comp rx=0 tx=0).ready entity=mon.1 client_cookie=964e24826f47baf1 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:37:20.841 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.836+0000 7fd943400640 10 client.26104 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:37:20.841 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.836+0000 7fd943400640 1 -- 172.21.15.90:0/2187839894 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd95c01f040 con 0x55c76238a080 2024-04-02T14:37:20.841 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.836+0000 7fd943400640 1 -- 172.21.15.90:0/2187839894 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7fd95c0ce730 con 0x55c76238a080 2024-04-02T14:37:20.841 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.836+0000 7fd943400640 1 -- 172.21.15.90:0/2187839894 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fd95c0a5970 con 0x55c76238a080 2024-04-02T14:37:20.841 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.836+0000 7fd96b5f0740 10 client.26104 fetch_fsmap learned FSMap version 27 2024-04-02T14:37:20.841 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.836+0000 7fd96b5f0740 1 -- 172.21.15.90:0/2187839894 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({fsmap=27}) -- 0x55c7623a9eb0 con 0x55c76238a080 2024-04-02T14:37:20.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.837+0000 7fd943400640 1 -- 172.21.15.90:0/2187839894 <== mon.1 v2:172.21.15.177:3300/0 4 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fd95c00f040 con 0x55c76238a080 2024-04-02T14:37:20.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.837+0000 7fd96b5f0740 10 client.26104 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:37:20.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.837+0000 7fd96b5f0740 10 client.26104 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:37:20.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.837+0000 7fd96b5f0740 20 client.26104 populate_metadata read hostname 'smithi090' 2024-04-02T14:37:20.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.837+0000 7fd96b5f0740 1 --2- 172.21.15.90:0/2187839894 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55c7623ab200 0x55c7623ad630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:20.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.837+0000 7fd96b5f0740 4 client.26104 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:37:20.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.837+0000 7fd96b5f0740 1 -- 172.21.15.90:0/2187839894 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c7622a5040 con 0x55c7623ab200 2024-04-02T14:37:20.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.838+0000 7fd965e00640 1 --2- 172.21.15.90:0/2187839894 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55c7623ab200 0x55c7623ad630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:20.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.840+0000 7fd943400640 10 client.26104 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:37:20.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.840+0000 7fd965e00640 1 --2- 172.21.15.90:0/2187839894 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55c7623ab200 0x55c7623ad630 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ec333c8efed072c1 server_cookie=ca02a6003fb95bb3 in_seq=0 out_seq=0 2024-04-02T14:37:20.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.842+0000 7fd943400640 1 -- 172.21.15.90:0/2187839894 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c7622a5040 con 0x55c7623ab200 2024-04-02T14:37:20.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.842+0000 7fd943400640 10 client.26104 handle_command_reply: tid=0 2024-04-02T14:37:20.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.842+0000 7fd96b5f0740 1 client.26104 shutdown 2024-04-02T14:37:20.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.842+0000 7fd96b5f0740 1 -- 172.21.15.90:0/2187839894 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c76238a080 msgr2=0x55c762387b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:20.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.842+0000 7fd96b5f0740 1 --2- 172.21.15.90:0/2187839894 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c76238a080 0x55c762387b50 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fd95c0ce510 tx=0x7fd95c09d530 comp rx=0 tx=0).stop 2024-04-02T14:37:20.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.843+0000 7fd96b5f0740 1 -- 172.21.15.90:0/2187839894 shutdown_connections 2024-04-02T14:37:20.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.843+0000 7fd96b5f0740 1 --2- 172.21.15.90:0/2187839894 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55c7623ab200 0x55c7623ad630 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:20.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.843+0000 7fd96b5f0740 1 --2- 172.21.15.90:0/2187839894 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c76238a080 0x55c762387b50 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:20.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.843+0000 7fd96b5f0740 1 --2- 172.21.15.90:0/2187839894 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c7623819f0 0x55c762387610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:20.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.843+0000 7fd96b5f0740 1 --2- 172.21.15.90:0/2187839894 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c762380d10 0x55c7623870d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:20.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.843+0000 7fd96b5f0740 1 -- 172.21.15.90:0/2187839894 >> 172.21.15.90:0/2187839894 conn(0x55c762347a50 msgr2=0x55c76237d4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:37:20.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.843+0000 7fd943400640 0 client.26104 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:37:20.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.843+0000 7fd96b5f0740 1 -- 172.21.15.90:0/2187839894 shutdown_connections 2024-04-02T14:37:20.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.844+0000 7fd96b5f0740 1 -- 172.21.15.90:0/2187839894 wait complete. 2024-04-02T14:37:20.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.844+0000 7fd96b5f0740 20 client.26104 trim_cache size 0 max 16384 2024-04-02T14:37:20.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.872+0000 7fd96b5f0740 1 Processor -- start 2024-04-02T14:37:20.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.872+0000 7fd96b5f0740 1 -- start start 2024-04-02T14:37:20.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.872+0000 7fd96b5f0740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c762380d10 0x55c762389800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:20.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.872+0000 7fd96b5f0740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c7623819f0 0x55c7623a6b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:20.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.872+0000 7fd96b5f0740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c76238a080 0x55c762387970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:20.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.873+0000 7fd96b5f0740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fd9601ab840 con 0x55c7623819f0 2024-04-02T14:37:20.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.873+0000 7fd96b5f0740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fd9601ad380 con 0x55c76238a080 2024-04-02T14:37:20.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.873+0000 7fd96b5f0740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fd9601abf10 con 0x55c762380d10 2024-04-02T14:37:20.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.873+0000 7fd966800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c76238a080 0x55c762387970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:20.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.873+0000 7fd966800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c76238a080 0x55c762387970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:48918/0 (socket says 172.21.15.90:48918) 2024-04-02T14:37:20.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.873+0000 7fd966800640 1 -- 172.21.15.90:0/3185545912 learned_addr learned my addr 172.21.15.90:0/3185545912 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:37:20.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.873+0000 7fd965e00640 1 --2- 172.21.15.90:0/3185545912 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c762380d10 0x55c762389800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:20.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.873+0000 7fd965400640 1 --2- 172.21.15.90:0/3185545912 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c7623819f0 0x55c7623a6b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:20.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.874+0000 7fd966800640 1 -- 172.21.15.90:0/3185545912 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c762380d10 msgr2=0x55c762389800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:37:20.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.874+0000 7fd966800640 1 --2- 172.21.15.90:0/3185545912 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c762380d10 0x55c762389800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:20.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.874+0000 7fd966800640 1 -- 172.21.15.90:0/3185545912 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c7623819f0 msgr2=0x55c7623a6b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:37:20.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.874+0000 7fd966800640 1 --2- 172.21.15.90:0/3185545912 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c7623819f0 0x55c7623a6b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:20.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.874+0000 7fd966800640 1 -- 172.21.15.90:0/3185545912 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c762387eb0 con 0x55c76238a080 2024-04-02T14:37:20.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.874+0000 7fd965e00640 1 --2- 172.21.15.90:0/3185545912 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c762380d10 0x55c762389800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:37:20.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.874+0000 7fd966800640 1 --2- 172.21.15.90:0/3185545912 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c76238a080 0x55c762387970 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fd95c00c640 tx=0x7fd95c0bd050 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5c33c93c26f3862a server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:37:20.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.875+0000 7fd94e600640 1 -- 172.21.15.90:0/3185545912 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd95c01f070 con 0x55c76238a080 2024-04-02T14:37:20.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.875+0000 7fd94e600640 1 -- 172.21.15.90:0/3185545912 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd95401fb00 con 0x55c76238a080 2024-04-02T14:37:20.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.875+0000 7fd94e600640 1 -- 172.21.15.90:0/3185545912 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd95c09c3b0 con 0x55c76238a080 2024-04-02T14:37:20.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.876+0000 7fd96b5f0740 1 -- 172.21.15.90:0/3185545912 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c76238a080 msgr2=0x55c762387970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:20.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.876+0000 7fd96b5f0740 1 --2- 172.21.15.90:0/3185545912 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c76238a080 0x55c762387970 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fd95c00c640 tx=0x7fd95c0bd050 comp rx=0 tx=0).stop 2024-04-02T14:37:20.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.876+0000 7fd96b5f0740 1 -- 172.21.15.90:0/3185545912 shutdown_connections 2024-04-02T14:37:20.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.876+0000 7fd96b5f0740 1 --2- 172.21.15.90:0/3185545912 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c76238a080 0x55c762387970 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:20.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.876+0000 7fd96b5f0740 1 --2- 172.21.15.90:0/3185545912 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c7623819f0 0x55c7623a6b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:20.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.876+0000 7fd96b5f0740 1 --2- 172.21.15.90:0/3185545912 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c762380d10 0x55c762389800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:20.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.876+0000 7fd96b5f0740 1 -- 172.21.15.90:0/3185545912 >> 172.21.15.90:0/3185545912 conn(0x55c762347a50 msgr2=0x55c762385870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:37:20.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.876+0000 7fd96b5f0740 1 -- 172.21.15.90:0/3185545912 shutdown_connections 2024-04-02T14:37:20.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.877+0000 7fd96b5f0740 1 -- 172.21.15.90:0/3185545912 wait complete. 2024-04-02T14:37:20.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.877+0000 7fd96b5f0740 1 -- start start 2024-04-02T14:37:20.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.877+0000 7fd96b5f0740 1 Processor -- start 2024-04-02T14:37:20.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.879+0000 7fd96b5f0740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c762380d10 0x55c7623a34c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:20.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.879+0000 7fd96b5f0740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c7623819f0 0x55c76238aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:20.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.879+0000 7fd96b5f0740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c76238a080 0x55c76238b020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:20.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.879+0000 7fd96b5f0740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fd9601accb0 con 0x55c7623819f0 2024-04-02T14:37:20.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.879+0000 7fd96b5f0740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fd9601ac5e0 con 0x55c76238a080 2024-04-02T14:37:20.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.879+0000 7fd96b5f0740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fd96010bdc0 con 0x55c762380d10 2024-04-02T14:37:20.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.879+0000 7fd96b5f0740 10 client.0 _ll_register_callbacks cb 0x7fd95c09c660 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:37:20.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.879+0000 7fd966800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c76238a080 0x55c76238b020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:20.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.879+0000 7fd966800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c76238a080 0x55c76238b020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:48924/0 (socket says 172.21.15.90:48924) 2024-04-02T14:37:20.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.879+0000 7fd966800640 1 -- 172.21.15.90:0/3634874466 learned_addr learned my addr 172.21.15.90:0/3634874466 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:37:20.882 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.880+0000 7fd965e00640 1 --2- 172.21.15.90:0/3634874466 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c762380d10 0x55c7623a34c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:20.882 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.880+0000 7fd966800640 1 -- 172.21.15.90:0/3634874466 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c762380d10 msgr2=0x55c7623a34c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:37:20.882 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.880+0000 7fd965400640 1 --2- 172.21.15.90:0/3634874466 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c7623819f0 0x55c76238aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:20.882 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.881+0000 7fd966800640 1 --2- 172.21.15.90:0/3634874466 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c762380d10 0x55c7623a34c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:20.882 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.881+0000 7fd966800640 1 -- 172.21.15.90:0/3634874466 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c7623819f0 msgr2=0x55c76238aaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:37:20.882 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.881+0000 7fd966800640 1 --2- 172.21.15.90:0/3634874466 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c7623819f0 0x55c76238aaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:20.882 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.881+0000 7fd966800640 1 -- 172.21.15.90:0/3634874466 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c76238b850 con 0x55c76238a080 2024-04-02T14:37:20.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.881+0000 7fd966800640 1 --2- 172.21.15.90:0/3634874466 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c76238a080 0x55c76238b020 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fd95c099160 tx=0x7fd95c0096d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cff5258b8f16a2a0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:37:20.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.881+0000 7fd943400640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:37:20.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.881+0000 7fd965400640 1 --2- 172.21.15.90:0/3634874466 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c7623819f0 0x55c76238aaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:37:20.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.882+0000 7fd943400640 1 -- 172.21.15.90:0/3634874466 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd95c01f070 con 0x55c76238a080 2024-04-02T14:37:20.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.882+0000 7fd965e00640 1 --2- 172.21.15.90:0/3634874466 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c762380d10 0x55c7623a34c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:37:20.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.882+0000 7fd943400640 1 -- 172.21.15.90:0/3634874466 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7fd95c09c150 con 0x55c76238a080 2024-04-02T14:37:20.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.882+0000 7fd96b5f0740 1 -- 172.21.15.90:0/3634874466 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd95401fb00 con 0x55c76238a080 2024-04-02T14:37:20.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.882+0000 7fd943400640 1 -- 172.21.15.90:0/3634874466 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fd95c086020 con 0x55c76238a080 2024-04-02T14:37:20.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.883+0000 7fd96b5f0740 10 client.26195 fetch_fsmap learned FSMap version 27 2024-04-02T14:37:20.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.883+0000 7fd96b5f0740 1 -- 172.21.15.90:0/3634874466 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55c762411830 con 0x55c76238a080 2024-04-02T14:37:20.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.883+0000 7fd943400640 1 -- 172.21.15.90:0/3634874466 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd95c0b6070 con 0x55c76238a080 2024-04-02T14:37:20.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.884+0000 7fd943400640 1 -- 172.21.15.90:0/3634874466 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fd95c0a2340 con 0x55c76238a080 2024-04-02T14:37:20.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.884+0000 7fd96b5f0740 10 client.26195 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:37:20.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.884+0000 7fd96b5f0740 10 client.26195 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:37:20.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.884+0000 7fd96b5f0740 20 client.26195 populate_metadata read hostname 'smithi090' 2024-04-02T14:37:20.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.884+0000 7fd96b5f0740 1 --2- 172.21.15.90:0/3634874466 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55c762411cb0 0x55c7624140a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:20.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.884+0000 7fd96b5f0740 4 client.26195 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-02T14:37:20.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.884+0000 7fd96b5f0740 1 -- 172.21.15.90:0/3634874466 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7fd95401fb00 con 0x55c762411cb0 2024-04-02T14:37:20.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.885+0000 7fd965e00640 1 --2- 172.21.15.90:0/3634874466 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55c762411cb0 0x55c7624140a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:20.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.887+0000 7fd943400640 10 client.26195 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:37:20.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.887+0000 7fd965e00640 1 --2- 172.21.15.90:0/3634874466 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55c762411cb0 0x55c7624140a0 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=14c2fd8648197354 server_cookie=809ac059dd2bb5d4 in_seq=0 out_seq=0 2024-04-02T14:37:20.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.887+0000 7fd943400640 1 -- 172.21.15.90:0/3634874466 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7fd95401fb00 con 0x55c762411cb0 2024-04-02T14:37:20.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.887+0000 7fd943400640 10 client.26195 handle_command_reply: tid=0 2024-04-02T14:37:20.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.887+0000 7fd96b5f0740 1 client.26195 shutdown 2024-04-02T14:37:20.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.888+0000 7fd96b5f0740 1 -- 172.21.15.90:0/3634874466 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c76238a080 msgr2=0x55c76238b020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:20.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.888+0000 7fd96b5f0740 1 --2- 172.21.15.90:0/3634874466 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c76238a080 0x55c76238b020 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fd95c099160 tx=0x7fd95c0096d0 comp rx=0 tx=0).stop 2024-04-02T14:37:20.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.888+0000 7fd96b5f0740 1 -- 172.21.15.90:0/3634874466 shutdown_connections 2024-04-02T14:37:20.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.888+0000 7fd96b5f0740 1 --2- 172.21.15.90:0/3634874466 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55c762411cb0 0x55c7624140a0 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:20.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.888+0000 7fd96b5f0740 1 --2- 172.21.15.90:0/3634874466 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c76238a080 0x55c76238b020 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:20.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.888+0000 7fd96b5f0740 1 --2- 172.21.15.90:0/3634874466 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c7623819f0 0x55c76238aaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:20.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.888+0000 7fd96b5f0740 1 --2- 172.21.15.90:0/3634874466 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c762380d10 0x55c7623a34c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:20.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.888+0000 7fd96b5f0740 1 -- 172.21.15.90:0/3634874466 >> 172.21.15.90:0/3634874466 conn(0x55c762347a50 msgr2=0x55c76237d370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:37:20.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.888+0000 7fd943400640 0 client.26195 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:37:20.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.888+0000 7fd96b5f0740 1 -- 172.21.15.90:0/3634874466 shutdown_connections 2024-04-02T14:37:20.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.889+0000 7fd96b5f0740 1 -- 172.21.15.90:0/3634874466 wait complete. 2024-04-02T14:37:20.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.889+0000 7fd96b5f0740 20 client.26195 trim_cache size 0 max 16384 2024-04-02T14:37:20.891 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-02T14:37:20.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.891+0000 7fd968800640 1 -- 172.21.15.90:0/3982949374 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fd9480a3e20 msgr2=0x7fd9480a62e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:20.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.891+0000 7fd968800640 1 --2- 172.21.15.90:0/3982949374 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fd9480a3e20 0x7fd9480a62e0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fd960102f50 tx=0x7fd954005f90 comp rx=0 tx=0).stop 2024-04-02T14:37:20.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.891+0000 7fd968800640 1 -- 172.21.15.90:0/3982949374 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd960112940 msgr2=0x7fd96019e8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:20.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.891+0000 7fd968800640 1 --2- 172.21.15.90:0/3982949374 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd960112940 0x7fd96019e8c0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fd95c00e9d0 tx=0x7fd95c00ee90 comp rx=0 tx=0).stop 2024-04-02T14:37:20.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.891+0000 7fd968800640 1 -- 172.21.15.90:0/3982949374 shutdown_connections 2024-04-02T14:37:20.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.891+0000 7fd968800640 1 --2- 172.21.15.90:0/3982949374 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fd9480a3e20 0x7fd9480a62e0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:20.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.891+0000 7fd968800640 1 --2- 172.21.15.90:0/3982949374 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd960112940 0x7fd96019e8c0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:20.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.891+0000 7fd968800640 1 --2- 172.21.15.90:0/3982949374 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd960076950 0x7fd9601a4c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:20.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.891+0000 7fd968800640 1 --2- 172.21.15.90:0/3982949374 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd960076030 0x7fd9601a46c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:20.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.891+0000 7fd968800640 1 -- 172.21.15.90:0/3982949374 >> 172.21.15.90:0/3982949374 conn(0x7fd9600ff7b0 msgr2=0x7fd960101ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:37:20.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.892+0000 7fd968800640 1 -- 172.21.15.90:0/3982949374 shutdown_connections 2024-04-02T14:37:20.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:20.892+0000 7fd968800640 1 -- 172.21.15.90:0/3982949374 wait complete. 2024-04-02T14:37:21.831 INFO:teuthology.orchestra.run.smithi090.stderr:progress: 110.0 s, 2922.9 tps, lat 47.738 ms stddev 224.369 2024-04-02T14:37:21.860 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:37:21 smithi092 ceph-mon[36497]: pgmap v77: 97 pgs: 97 active+clean; 1.5 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 33 MiB/s wr, 1.14k op/s 2024-04-02T14:37:21.931 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-02T14:37:21.931 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T14:37:21.931 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-02T14:37:21.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:37:21 smithi177 ceph-mon[36431]: pgmap v77: 97 pgs: 97 active+clean; 1.5 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 33 MiB/s wr, 1.14k op/s 2024-04-02T14:37:21.938 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:37:21 smithi090 ceph-mon[31674]: pgmap v77: 97 pgs: 97 active+clean; 1.5 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 33 MiB/s wr, 1.14k op/s 2024-04-02T14:37:22.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.272+0000 7f889c800640 1 -- 172.21.15.90:0/683123677 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8894100300 msgr2=0x7f8894110c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:22.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.272+0000 7f889c800640 1 --2- 172.21.15.90:0/683123677 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8894100300 0x7f8894110c60 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f8890009ca0 tx=0x7f889002c100 comp rx=0 tx=0).stop 2024-04-02T14:37:22.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.272+0000 7f889c800640 1 -- 172.21.15.90:0/683123677 shutdown_connections 2024-04-02T14:37:22.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.272+0000 7f889c800640 1 --2- 172.21.15.90:0/683123677 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8894100300 0x7f8894110c60 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:22.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.272+0000 7f889c800640 1 --2- 172.21.15.90:0/683123677 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f889410a7d0 0x7f88940ffbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:22.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.272+0000 7f889c800640 1 --2- 172.21.15.90:0/683123677 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8894109e20 0x7f889410a200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:22.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.272+0000 7f889c800640 1 -- 172.21.15.90:0/683123677 >> 172.21.15.90:0/683123677 conn(0x7f8894076b50 msgr2=0x7f88940fdee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:37:22.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.273+0000 7f889c800640 1 -- 172.21.15.90:0/683123677 shutdown_connections 2024-04-02T14:37:22.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.273+0000 7f889c800640 1 -- 172.21.15.90:0/683123677 wait complete. 2024-04-02T14:37:22.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.274+0000 7f889c800640 1 Processor -- start 2024-04-02T14:37:22.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.274+0000 7f889c800640 1 -- start start 2024-04-02T14:37:22.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.275+0000 7f889c800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8894100300 0x7f889419a500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:22.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.275+0000 7f889c800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8894109e20 0x7f889419aa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:22.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.275+0000 7f889c800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f889419edc0 0x7f889419f270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:22.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.275+0000 7f889c800640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f8894114080 con 0x7f889419edc0 2024-04-02T14:37:22.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.275+0000 7f889c800640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f8894113f00 con 0x7f8894100300 2024-04-02T14:37:22.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.275+0000 7f889c800640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f8894114200 con 0x7f8894109e20 2024-04-02T14:37:22.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.276+0000 7f8899e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8894100300 0x7f889419a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:22.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.276+0000 7f8899400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8894109e20 0x7f889419aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:22.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.276+0000 7f8899e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8894100300 0x7f889419a500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:48944/0 (socket says 172.21.15.90:48944) 2024-04-02T14:37:22.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.276+0000 7f8899e00640 1 -- 172.21.15.90:0/3323080989 learned_addr learned my addr 172.21.15.90:0/3323080989 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:37:22.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.276+0000 7f889a800640 1 --2- 172.21.15.90:0/3323080989 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f889419edc0 0x7f889419f270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:22.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.276+0000 7f8899e00640 1 -- 172.21.15.90:0/3323080989 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8894109e20 msgr2=0x7f889419aa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:22.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.276+0000 7f8899e00640 1 --2- 172.21.15.90:0/3323080989 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8894109e20 0x7f889419aa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:22.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.276+0000 7f8899e00640 1 -- 172.21.15.90:0/3323080989 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f889419edc0 msgr2=0x7f889419f270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:22.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.277+0000 7f8899e00640 1 --2- 172.21.15.90:0/3323080989 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f889419edc0 0x7f889419f270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:22.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.277+0000 7f8899e00640 1 -- 172.21.15.90:0/3323080989 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f889419f8b0 con 0x7f8894100300 2024-04-02T14:37:22.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.277+0000 7f8899400640 1 --2- 172.21.15.90:0/3323080989 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8894109e20 0x7f889419aa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:37:22.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.277+0000 7f889a800640 1 --2- 172.21.15.90:0/3323080989 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f889419edc0 0x7f889419f270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:37:22.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.277+0000 7f8899e00640 1 --2- 172.21.15.90:0/3323080989 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8894100300 0x7f889419a500 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f888800d950 tx=0x7f888800de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:37:22.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.278+0000 7f888ea00640 1 -- 172.21.15.90:0/3323080989 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8888017020 con 0x7f8894100300 2024-04-02T14:37:22.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.278+0000 7f888ea00640 1 -- 172.21.15.90:0/3323080989 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8888009c30 con 0x7f8894100300 2024-04-02T14:37:22.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.278+0000 7f888ea00640 1 -- 172.21.15.90:0/3323080989 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f888800f720 con 0x7f8894100300 2024-04-02T14:37:22.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.278+0000 7f889c800640 1 -- 172.21.15.90:0/3323080989 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f889419fba0 con 0x7f8894100300 2024-04-02T14:37:22.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.278+0000 7f889c800640 1 -- 172.21.15.90:0/3323080989 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88941a7340 con 0x7f8894100300 2024-04-02T14:37:22.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.284+0000 7f888ea00640 1 -- 172.21.15.90:0/3323080989 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f888801a020 con 0x7f8894100300 2024-04-02T14:37:22.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.287+0000 7f888ea00640 1 --2- 172.21.15.90:0/3323080989 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f887c0a3e70 0x7f887c0a6330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:22.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.287+0000 7f888ea00640 1 -- 172.21.15.90:0/3323080989 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f88880d07b0 con 0x7f8894100300 2024-04-02T14:37:22.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.288+0000 7f8899400640 1 --2- 172.21.15.90:0/3323080989 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f887c0a3e70 0x7f887c0a6330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:22.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.289+0000 7f8899400640 1 --2- 172.21.15.90:0/3323080989 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f887c0a3e70 0x7f887c0a6330 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f88940ff120 tx=0x7f88840094b0 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:37:22.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.289+0000 7f889f5b2740 1 Processor -- start 2024-04-02T14:37:22.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.290+0000 7f889f5b2740 1 -- start start 2024-04-02T14:37:22.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.290+0000 7f889f5b2740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ed70cbb840 0x55ed70cba590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:22.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.290+0000 7f889f5b2740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ed70cbab60 0x55ed70cbafe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:22.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.290+0000 7f889f5b2740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ed70cc3ed0 0x55ed70cc62c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:22.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.290+0000 7f889f5b2740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55ed70c16310 con 0x55ed70cc3ed0 2024-04-02T14:37:22.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.290+0000 7f889f5b2740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55ed70bbe690 con 0x55ed70cbb840 2024-04-02T14:37:22.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.290+0000 7f889f5b2740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55ed70c122c0 con 0x55ed70cbab60 2024-04-02T14:37:22.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.290+0000 7f8899400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ed70cbab60 0x55ed70cbafe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:22.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.290+0000 7f8899400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ed70cbab60 0x55ed70cbafe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:58388/0 (socket says 172.21.15.90:58388) 2024-04-02T14:37:22.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.290+0000 7f8899400640 1 -- 172.21.15.90:0/3945294511 learned_addr learned my addr 172.21.15.90:0/3945294511 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:37:22.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.291+0000 7f8899e00640 1 --2- 172.21.15.90:0/3945294511 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ed70cbb840 0x55ed70cba590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:22.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.291+0000 7f8899400640 1 -- 172.21.15.90:0/3945294511 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ed70cbb840 msgr2=0x55ed70cba590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:37:22.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.291+0000 7f8899400640 1 --2- 172.21.15.90:0/3945294511 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ed70cbb840 0x55ed70cba590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:22.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.291+0000 7f8899400640 1 -- 172.21.15.90:0/3945294511 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ed70cc3ed0 msgr2=0x55ed70cc62c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-02T14:37:22.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.291+0000 7f8899400640 1 --2- 172.21.15.90:0/3945294511 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ed70cc3ed0 0x55ed70cc62c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:22.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.291+0000 7f8899400640 1 -- 172.21.15.90:0/3945294511 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ed709f27e0 con 0x55ed70cbab60 2024-04-02T14:37:22.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.291+0000 7f8899e00640 1 --2- 172.21.15.90:0/3945294511 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ed70cbb840 0x55ed70cba590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:37:22.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.292+0000 7f8899400640 1 --2- 172.21.15.90:0/3945294511 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ed70cbab60 0x55ed70cbafe0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f888400eda0 tx=0x7f8884011000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9cdf7de8b60b0ccf server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:37:22.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.292+0000 7f8883000640 1 -- 172.21.15.90:0/3945294511 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f888401e440 con 0x55ed70cbab60 2024-04-02T14:37:22.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.292+0000 7f8883000640 1 -- 172.21.15.90:0/3945294511 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8884023070 con 0x55ed70cbab60 2024-04-02T14:37:22.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.292+0000 7f8883000640 1 -- 172.21.15.90:0/3945294511 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8884010950 con 0x55ed70cbab60 2024-04-02T14:37:22.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.294+0000 7f889f5b2740 1 -- 172.21.15.90:0/3945294511 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ed70cbab60 msgr2=0x55ed70cbafe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:22.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.294+0000 7f889f5b2740 1 --2- 172.21.15.90:0/3945294511 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ed70cbab60 0x55ed70cbafe0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f888400eda0 tx=0x7f8884011000 comp rx=0 tx=0).stop 2024-04-02T14:37:22.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.294+0000 7f889f5b2740 1 -- 172.21.15.90:0/3945294511 shutdown_connections 2024-04-02T14:37:22.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.294+0000 7f889f5b2740 1 --2- 172.21.15.90:0/3945294511 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ed70cc3ed0 0x55ed70cc62c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:22.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.294+0000 7f889f5b2740 1 --2- 172.21.15.90:0/3945294511 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ed70cbab60 0x55ed70cbafe0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:22.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.294+0000 7f889f5b2740 1 --2- 172.21.15.90:0/3945294511 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ed70cbb840 0x55ed70cba590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:22.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.294+0000 7f889f5b2740 1 -- 172.21.15.90:0/3945294511 >> 172.21.15.90:0/3945294511 conn(0x55ed70c46270 msgr2=0x55ed70cb55b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:37:22.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.295+0000 7f889f5b2740 1 -- 172.21.15.90:0/3945294511 shutdown_connections 2024-04-02T14:37:22.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.295+0000 7f889f5b2740 1 -- 172.21.15.90:0/3945294511 wait complete. 2024-04-02T14:37:22.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.296+0000 7f889f5b2740 1 -- start start 2024-04-02T14:37:22.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.297+0000 7f889f5b2740 1 Processor -- start 2024-04-02T14:37:22.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.298+0000 7f889f5b2740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ed70cbab60 0x55ed70cc0e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:22.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.298+0000 7f889f5b2740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ed70cbb840 0x55ed70cc13a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:22.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.298+0000 7f889f5b2740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ed70cc3ed0 0x55ed70cc18e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:22.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.299+0000 7f8899e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ed70cbab60 0x55ed70cc0e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:22.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.299+0000 7f8899e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ed70cbab60 0x55ed70cc0e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:48950/0 (socket says 172.21.15.90:48950) 2024-04-02T14:37:22.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.299+0000 7f8899e00640 1 -- 172.21.15.90:0/3312333457 learned_addr learned my addr 172.21.15.90:0/3312333457 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:37:22.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.299+0000 7f889f5b2740 1 -- 172.21.15.90:0/3312333457 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55ed70cc6ce0 con 0x55ed70cc3ed0 2024-04-02T14:37:22.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.299+0000 7f889f5b2740 1 -- 172.21.15.90:0/3312333457 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55ed70bd1a60 con 0x55ed70cbab60 2024-04-02T14:37:22.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.299+0000 7f889f5b2740 1 -- 172.21.15.90:0/3312333457 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55ed70cc6e60 con 0x55ed70cbb840 2024-04-02T14:37:22.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.299+0000 7f889f5b2740 10 client.0 _ll_register_callbacks cb 0x55ed70bd7030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:37:22.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.299+0000 7f8899400640 1 --2- 172.21.15.90:0/3312333457 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ed70cbb840 0x55ed70cc13a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:22.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.299+0000 7f8899e00640 1 -- 172.21.15.90:0/3312333457 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ed70cbb840 msgr2=0x55ed70cc13a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:37:22.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.299+0000 7f8899e00640 1 --2- 172.21.15.90:0/3312333457 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ed70cbb840 0x55ed70cc13a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:22.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.299+0000 7f8899e00640 1 -- 172.21.15.90:0/3312333457 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ed70cc3ed0 msgr2=0x55ed70cc18e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:37:22.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.299+0000 7f8899e00640 1 --2- 172.21.15.90:0/3312333457 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ed70cc3ed0 0x55ed70cc18e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:22.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.299+0000 7f8899e00640 1 -- 172.21.15.90:0/3312333457 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ed70a52f70 con 0x55ed70cbab60 2024-04-02T14:37:22.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.300+0000 7f889f5b2740 1 -- 172.21.15.90:0/3312333457 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8884023070 con 0x55ed70cbab60 2024-04-02T14:37:22.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.300+0000 7f8899e00640 1 --2- 172.21.15.90:0/3312333457 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ed70cbab60 0x55ed70cc0e60 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f8888005fd0 tx=0x7f88880a0ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9ef79425246271f2 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:37:22.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.300+0000 7f8877400640 10 client.26207 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:37:22.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.300+0000 7f8877400640 1 -- 172.21.15.90:0/3312333457 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f888809aac0 con 0x55ed70cbab60 2024-04-02T14:37:22.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.300+0000 7f8877400640 1 -- 172.21.15.90:0/3312333457 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f8888098d70 con 0x55ed70cbab60 2024-04-02T14:37:22.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.300+0000 7f8877400640 1 -- 172.21.15.90:0/3312333457 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f8888023bb0 con 0x55ed70cbab60 2024-04-02T14:37:22.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.301+0000 7f889f5b2740 10 client.26207 fetch_fsmap learned FSMap version 27 2024-04-02T14:37:22.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.301+0000 7f889f5b2740 1 -- 172.21.15.90:0/3312333457 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({fsmap=27}) -- 0x55ed708ea170 con 0x55ed70cbab60 2024-04-02T14:37:22.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.301+0000 7f8877400640 1 -- 172.21.15.90:0/3312333457 <== mon.2 v2:172.21.15.92:3300/0 4 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f88880b5030 con 0x55ed70cbab60 2024-04-02T14:37:22.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.301+0000 7f889f5b2740 10 client.26207 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:37:22.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.301+0000 7f889f5b2740 10 client.26207 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:37:22.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.301+0000 7f889f5b2740 20 client.26207 populate_metadata read hostname 'smithi090' 2024-04-02T14:37:22.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.301+0000 7f889f5b2740 1 --2- 172.21.15.90:0/3312333457 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55ed70cd4140 0x55ed70cd6530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:22.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.301+0000 7f889f5b2740 4 client.26207 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:37:22.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.301+0000 7f889f5b2740 1 -- 172.21.15.90:0/3312333457 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ed70c23850 con 0x55ed70cd4140 2024-04-02T14:37:22.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.302+0000 7f8899400640 1 --2- 172.21.15.90:0/3312333457 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55ed70cd4140 0x55ed70cd6530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:22.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.302+0000 7f8899400640 1 --2- 172.21.15.90:0/3312333457 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55ed70cd4140 0x55ed70cd6530 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cca4dd4084e3588c server_cookie=4013260251fd4ae6 in_seq=0 out_seq=0 2024-04-02T14:37:22.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.302+0000 7f8877400640 10 client.26207 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:37:22.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.304+0000 7f8877400640 1 -- 172.21.15.90:0/3312333457 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ed70c23850 con 0x55ed70cd4140 2024-04-02T14:37:22.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.304+0000 7f8877400640 10 client.26207 handle_command_reply: tid=0 2024-04-02T14:37:22.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.304+0000 7f889f5b2740 1 client.26207 shutdown 2024-04-02T14:37:22.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.305+0000 7f889f5b2740 1 -- 172.21.15.90:0/3312333457 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ed70cbab60 msgr2=0x55ed70cc0e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:22.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.305+0000 7f889f5b2740 1 --2- 172.21.15.90:0/3312333457 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ed70cbab60 0x55ed70cc0e60 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f8888005fd0 tx=0x7f88880a0ab0 comp rx=0 tx=0).stop 2024-04-02T14:37:22.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.306+0000 7f889f5b2740 1 -- 172.21.15.90:0/3312333457 shutdown_connections 2024-04-02T14:37:22.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.306+0000 7f889f5b2740 1 --2- 172.21.15.90:0/3312333457 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55ed70cd4140 0x55ed70cd6530 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:22.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.306+0000 7f889f5b2740 1 --2- 172.21.15.90:0/3312333457 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ed70cc3ed0 0x55ed70cc18e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:22.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.306+0000 7f889f5b2740 1 --2- 172.21.15.90:0/3312333457 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ed70cbb840 0x55ed70cc13a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:22.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.306+0000 7f889f5b2740 1 --2- 172.21.15.90:0/3312333457 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ed70cbab60 0x55ed70cc0e60 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:22.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.306+0000 7f8877400640 0 client.26207 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:37:22.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.306+0000 7f889f5b2740 1 -- 172.21.15.90:0/3312333457 >> 172.21.15.90:0/3312333457 conn(0x55ed70c46270 msgr2=0x55ed70cb7300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:37:22.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.307+0000 7f889f5b2740 1 -- 172.21.15.90:0/3312333457 shutdown_connections 2024-04-02T14:37:22.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.307+0000 7f889f5b2740 1 -- 172.21.15.90:0/3312333457 wait complete. 2024-04-02T14:37:22.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.308+0000 7f889f5b2740 20 client.26207 trim_cache size 0 max 16384 2024-04-02T14:37:22.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.349+0000 7f889f5b2740 1 Processor -- start 2024-04-02T14:37:22.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.349+0000 7f889f5b2740 1 -- start start 2024-04-02T14:37:22.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.350+0000 7f889f5b2740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ed70cbab60 0x55ed70ccfc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:22.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.350+0000 7f889f5b2740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ed70cbb840 0x55ed70cdbdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:22.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.350+0000 7f889f5b2740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ed70cc3ed0 0x55ed70cdc300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:22.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.350+0000 7f889f5b2740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f88941ab760 con 0x55ed70cbab60 2024-04-02T14:37:22.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.350+0000 7f8899400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ed70cbb840 0x55ed70cdbdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:22.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.350+0000 7f889f5b2740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f88941ad2a0 con 0x55ed70cc3ed0 2024-04-02T14:37:22.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.350+0000 7f889f5b2740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f88941abe30 con 0x55ed70cbb840 2024-04-02T14:37:22.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.350+0000 7f8899e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ed70cbab60 0x55ed70ccfc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:22.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.351+0000 7f8899400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ed70cbb840 0x55ed70cdbdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:58404/0 (socket says 172.21.15.90:58404) 2024-04-02T14:37:22.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.351+0000 7f8899400640 1 -- 172.21.15.90:0/3660677912 learned_addr learned my addr 172.21.15.90:0/3660677912 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:37:22.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.351+0000 7f889a800640 1 --2- 172.21.15.90:0/3660677912 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ed70cc3ed0 0x55ed70cdc300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:22.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.351+0000 7f8899400640 1 -- 172.21.15.90:0/3660677912 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ed70cc3ed0 msgr2=0x55ed70cdc300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:37:22.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.352+0000 7f8899400640 1 --2- 172.21.15.90:0/3660677912 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ed70cc3ed0 0x55ed70cdc300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:22.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.352+0000 7f8899400640 1 -- 172.21.15.90:0/3660677912 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ed70cbab60 msgr2=0x55ed70ccfc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:37:22.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.352+0000 7f8899400640 1 --2- 172.21.15.90:0/3660677912 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ed70cbab60 0x55ed70ccfc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:22.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.352+0000 7f8899400640 1 -- 172.21.15.90:0/3660677912 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ed70cd6b90 con 0x55ed70cbb840 2024-04-02T14:37:22.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.352+0000 7f889a800640 1 --2- 172.21.15.90:0/3660677912 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ed70cc3ed0 0x55ed70cdc300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:37:22.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.352+0000 7f8899e00640 1 --2- 172.21.15.90:0/3660677912 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ed70cbab60 0x55ed70ccfc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:37:22.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.352+0000 7f8899400640 1 --2- 172.21.15.90:0/3660677912 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ed70cbb840 0x55ed70cdbdc0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f8884010440 tx=0x7f8884010470 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3baa1334c6fe6cd2 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:37:22.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.352+0000 7f8882600640 1 -- 172.21.15.90:0/3660677912 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8884031540 con 0x55ed70cbb840 2024-04-02T14:37:22.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.353+0000 7f8882600640 1 -- 172.21.15.90:0/3660677912 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8884017030 con 0x55ed70cbb840 2024-04-02T14:37:22.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.353+0000 7f8882600640 1 -- 172.21.15.90:0/3660677912 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8884031540 con 0x55ed70cbb840 2024-04-02T14:37:22.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.353+0000 7f889f5b2740 1 -- 172.21.15.90:0/3660677912 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ed70cbb840 msgr2=0x55ed70cdbdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:22.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.353+0000 7f889f5b2740 1 --2- 172.21.15.90:0/3660677912 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ed70cbb840 0x55ed70cdbdc0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f8884010440 tx=0x7f8884010470 comp rx=0 tx=0).stop 2024-04-02T14:37:22.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.354+0000 7f889f5b2740 1 -- 172.21.15.90:0/3660677912 shutdown_connections 2024-04-02T14:37:22.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.354+0000 7f889f5b2740 1 --2- 172.21.15.90:0/3660677912 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ed70cc3ed0 0x55ed70cdc300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:22.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.354+0000 7f889f5b2740 1 --2- 172.21.15.90:0/3660677912 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ed70cbb840 0x55ed70cdbdc0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:22.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.354+0000 7f889f5b2740 1 --2- 172.21.15.90:0/3660677912 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ed70cbab60 0x55ed70ccfc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:22.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.354+0000 7f889f5b2740 1 -- 172.21.15.90:0/3660677912 >> 172.21.15.90:0/3660677912 conn(0x55ed70c46270 msgr2=0x55ed70cabe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:37:22.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.354+0000 7f889f5b2740 1 -- 172.21.15.90:0/3660677912 shutdown_connections 2024-04-02T14:37:22.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.354+0000 7f889f5b2740 1 -- 172.21.15.90:0/3660677912 wait complete. 2024-04-02T14:37:22.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.355+0000 7f889f5b2740 1 -- start start 2024-04-02T14:37:22.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.355+0000 7f889f5b2740 1 Processor -- start 2024-04-02T14:37:22.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.356+0000 7f889f5b2740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ed70cbab60 0x55ed70ccc540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:22.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.356+0000 7f889f5b2740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ed70cbb840 0x55ed70cc9120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:22.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.356+0000 7f889f5b2740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ed70cc3ed0 0x55ed70cc9660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:22.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.357+0000 7f8899400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ed70cbb840 0x55ed70cc9120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:22.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.357+0000 7f8899400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ed70cbb840 0x55ed70cc9120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:58420/0 (socket says 172.21.15.90:58420) 2024-04-02T14:37:22.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.357+0000 7f889a800640 1 --2- 172.21.15.90:0/3667283832 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ed70cc3ed0 0x55ed70cc9660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:22.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.357+0000 7f8899e00640 1 --2- 172.21.15.90:0/3667283832 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ed70cbab60 0x55ed70ccc540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:22.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.357+0000 7f8899400640 1 -- 172.21.15.90:0/3667283832 learned_addr learned my addr 172.21.15.90:0/3667283832 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:37:22.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.357+0000 7f889f5b2740 1 -- 172.21.15.90:0/3667283832 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f88941acbd0 con 0x55ed70cbab60 2024-04-02T14:37:22.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.357+0000 7f889f5b2740 1 -- 172.21.15.90:0/3667283832 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f88941ac500 con 0x55ed70cc3ed0 2024-04-02T14:37:22.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.357+0000 7f889f5b2740 1 -- 172.21.15.90:0/3667283832 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f88941a9740 con 0x55ed70cbb840 2024-04-02T14:37:22.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.357+0000 7f889f5b2740 10 client.0 _ll_register_callbacks cb 0x7f888809c060 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:37:22.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.358+0000 7f889a800640 1 -- 172.21.15.90:0/3667283832 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ed70cbb840 msgr2=0x55ed70cc9120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:37:22.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.358+0000 7f889a800640 1 --2- 172.21.15.90:0/3667283832 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ed70cbb840 0x55ed70cc9120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:22.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.358+0000 7f889a800640 1 -- 172.21.15.90:0/3667283832 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ed70cbab60 msgr2=0x55ed70ccc540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:37:22.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.358+0000 7f889a800640 1 --2- 172.21.15.90:0/3667283832 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ed70cbab60 0x55ed70ccc540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:22.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.358+0000 7f889a800640 1 -- 172.21.15.90:0/3667283832 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ed70cb9b90 con 0x55ed70cc3ed0 2024-04-02T14:37:22.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.358+0000 7f8899e00640 1 --2- 172.21.15.90:0/3667283832 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ed70cbab60 0x55ed70ccc540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:37:22.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.359+0000 7f889a800640 1 --2- 172.21.15.90:0/3667283832 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ed70cc3ed0 0x55ed70cc9660 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f8890009bf0 tx=0x7f8890009800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=37b05b89d7d2d050 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:37:22.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.359+0000 7f8877400640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:37:22.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.359+0000 7f8877400640 1 -- 172.21.15.90:0/3667283832 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8890044070 con 0x55ed70cc3ed0 2024-04-02T14:37:22.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.359+0000 7f8877400640 1 -- 172.21.15.90:0/3667283832 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f88900329a0 con 0x55ed70cc3ed0 2024-04-02T14:37:22.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.360+0000 7f889f5b2740 1 -- 172.21.15.90:0/3667283832 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8884017030 con 0x55ed70cc3ed0 2024-04-02T14:37:22.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.360+0000 7f8877400640 1 -- 172.21.15.90:0/3667283832 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f8890039040 con 0x55ed70cc3ed0 2024-04-02T14:37:22.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.360+0000 7f889f5b2740 10 client.26213 fetch_fsmap learned FSMap version 27 2024-04-02T14:37:22.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.360+0000 7f889f5b2740 1 -- 172.21.15.90:0/3667283832 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55ed70d3b4d0 con 0x55ed70cc3ed0 2024-04-02T14:37:22.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.360+0000 7f8877400640 1 -- 172.21.15.90:0/3667283832 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8884017030 con 0x55ed70cc3ed0 2024-04-02T14:37:22.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.361+0000 7f8877400640 1 -- 172.21.15.90:0/3667283832 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f889003f440 con 0x55ed70cc3ed0 2024-04-02T14:37:22.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.361+0000 7f889f5b2740 10 client.26213 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:37:22.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.361+0000 7f889f5b2740 10 client.26213 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:37:22.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.361+0000 7f889f5b2740 20 client.26213 populate_metadata read hostname 'smithi090' 2024-04-02T14:37:22.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.361+0000 7f889f5b2740 1 --2- 172.21.15.90:0/3667283832 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55ed70d3bbd0 0x55ed70d3e010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:22.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.361+0000 7f889f5b2740 4 client.26213 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-02T14:37:22.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.361+0000 7f889f5b2740 1 -- 172.21.15.90:0/3667283832 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55ed70c1dba0 con 0x55ed70d3bbd0 2024-04-02T14:37:22.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.361+0000 7f8899e00640 1 --2- 172.21.15.90:0/3667283832 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55ed70d3bbd0 0x55ed70d3e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:22.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.362+0000 7f8899e00640 1 --2- 172.21.15.90:0/3667283832 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55ed70d3bbd0 0x55ed70d3e010 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5abd339d8c1be7b3 server_cookie=72cd396833b48e10 in_seq=0 out_seq=0 2024-04-02T14:37:22.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.362+0000 7f8877400640 10 client.26213 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:37:22.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.362+0000 7f8877400640 1 -- 172.21.15.90:0/3667283832 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55ed70c1dba0 con 0x55ed70d3bbd0 2024-04-02T14:37:22.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.362+0000 7f8877400640 10 client.26213 handle_command_reply: tid=0 2024-04-02T14:37:22.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.363+0000 7f889f5b2740 1 client.26213 shutdown 2024-04-02T14:37:22.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.363+0000 7f889f5b2740 1 -- 172.21.15.90:0/3667283832 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ed70cc3ed0 msgr2=0x55ed70cc9660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:22.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.363+0000 7f889f5b2740 1 --2- 172.21.15.90:0/3667283832 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ed70cc3ed0 0x55ed70cc9660 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f8890009bf0 tx=0x7f8890009800 comp rx=0 tx=0).stop 2024-04-02T14:37:22.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.364+0000 7f889f5b2740 1 -- 172.21.15.90:0/3667283832 shutdown_connections 2024-04-02T14:37:22.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.364+0000 7f889f5b2740 1 --2- 172.21.15.90:0/3667283832 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55ed70d3bbd0 0x55ed70d3e010 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:22.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.364+0000 7f889f5b2740 1 --2- 172.21.15.90:0/3667283832 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ed70cc3ed0 0x55ed70cc9660 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:22.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.364+0000 7f889f5b2740 1 --2- 172.21.15.90:0/3667283832 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ed70cbb840 0x55ed70cc9120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:22.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.364+0000 7f889f5b2740 1 --2- 172.21.15.90:0/3667283832 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ed70cbab60 0x55ed70ccc540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:22.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.364+0000 7f889f5b2740 1 -- 172.21.15.90:0/3667283832 >> 172.21.15.90:0/3667283832 conn(0x55ed70c46270 msgr2=0x55ed70cc0e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:37:22.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.364+0000 7f8877400640 0 client.26213 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:37:22.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.364+0000 7f889f5b2740 1 -- 172.21.15.90:0/3667283832 shutdown_connections 2024-04-02T14:37:22.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.364+0000 7f889f5b2740 1 -- 172.21.15.90:0/3667283832 wait complete. 2024-04-02T14:37:22.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.365+0000 7f889f5b2740 20 client.26213 trim_cache size 0 max 16384 2024-04-02T14:37:22.369 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-02T14:37:22.370 INFO:teuthology.orchestra.run.smithi090.stdout: "return_code": 0, 2024-04-02T14:37:22.370 INFO:teuthology.orchestra.run.smithi090.stdout: "scrub_tag": "b9187b6d-35a7-4838-8e0e-1046f12b3662", 2024-04-02T14:37:22.370 INFO:teuthology.orchestra.run.smithi090.stdout: "mode": "asynchronous" 2024-04-02T14:37:22.370 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-02T14:37:22.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.366+0000 7f889c800640 1 -- 172.21.15.90:0/3323080989 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f887c0a3e70 msgr2=0x7f887c0a6330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:22.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.366+0000 7f889c800640 1 --2- 172.21.15.90:0/3323080989 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f887c0a3e70 0x7f887c0a6330 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f88940ff120 tx=0x7f88840094b0 comp rx=0 tx=0).stop 2024-04-02T14:37:22.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.366+0000 7f889c800640 1 -- 172.21.15.90:0/3323080989 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8894100300 msgr2=0x7f889419a500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:22.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.366+0000 7f889c800640 1 --2- 172.21.15.90:0/3323080989 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8894100300 0x7f889419a500 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f888800d950 tx=0x7f888800de10 comp rx=0 tx=0).stop 2024-04-02T14:37:22.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.366+0000 7f889c800640 1 -- 172.21.15.90:0/3323080989 shutdown_connections 2024-04-02T14:37:22.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.366+0000 7f889c800640 1 --2- 172.21.15.90:0/3323080989 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f887c0a3e70 0x7f887c0a6330 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:22.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.366+0000 7f889c800640 1 --2- 172.21.15.90:0/3323080989 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f889419edc0 0x7f889419f270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:22.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.366+0000 7f889c800640 1 --2- 172.21.15.90:0/3323080989 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8894109e20 0x7f889419aa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:22.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.366+0000 7f889c800640 1 --2- 172.21.15.90:0/3323080989 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8894100300 0x7f889419a500 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:22.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.366+0000 7f889c800640 1 -- 172.21.15.90:0/3323080989 >> 172.21.15.90:0/3323080989 conn(0x7f8894076b50 msgr2=0x7f8894101920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:37:22.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.367+0000 7f889c800640 1 -- 172.21.15.90:0/3323080989 shutdown_connections 2024-04-02T14:37:22.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.367+0000 7f889c800640 1 -- 172.21.15.90:0/3323080989 wait complete. 2024-04-02T14:37:22.397 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-02T14:37:22.397 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-02T14:37:22.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.733+0000 7fe145e00640 1 -- 172.21.15.90:0/1011991835 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe140073ea0 msgr2=0x7fe140074280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:22.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.733+0000 7fe145e00640 1 --2- 172.21.15.90:0/1011991835 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe140073ea0 0x7fe140074280 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fe130009900 tx=0x7fe13002bcd0 comp rx=0 tx=0).stop 2024-04-02T14:37:22.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.733+0000 7fe145e00640 1 -- 172.21.15.90:0/1011991835 shutdown_connections 2024-04-02T14:37:22.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.733+0000 7fe145e00640 1 --2- 172.21.15.90:0/1011991835 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe140112a90 0x7fe140114e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:22.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.733+0000 7fe145e00640 1 --2- 172.21.15.90:0/1011991835 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe1400747c0 0x7fe140112550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:22.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.733+0000 7fe145e00640 1 --2- 172.21.15.90:0/1011991835 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe140073ea0 0x7fe140074280 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:22.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.733+0000 7fe145e00640 1 -- 172.21.15.90:0/1011991835 >> 172.21.15.90:0/1011991835 conn(0x7fe1400fd590 msgr2=0x7fe1400ff9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:37:22.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.733+0000 7fe145e00640 1 -- 172.21.15.90:0/1011991835 shutdown_connections 2024-04-02T14:37:22.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.733+0000 7fe145e00640 1 -- 172.21.15.90:0/1011991835 wait complete. 2024-04-02T14:37:22.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.734+0000 7fe145e00640 1 Processor -- start 2024-04-02T14:37:22.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.735+0000 7fe145e00640 1 -- start start 2024-04-02T14:37:22.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.735+0000 7fe145e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe140073ea0 0x7fe1401a46d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:22.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.735+0000 7fe145e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe1400747c0 0x7fe1401a4c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:22.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.735+0000 7fe145e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe140112a90 0x7fe14019e880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:22.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.735+0000 7fe145e00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fe140118520 con 0x7fe1400747c0 2024-04-02T14:37:22.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.735+0000 7fe145e00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fe1401183a0 con 0x7fe140112a90 2024-04-02T14:37:22.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.735+0000 7fe145e00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fe1401186a0 con 0x7fe140073ea0 2024-04-02T14:37:22.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.736+0000 7fe13ea00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe140073ea0 0x7fe1401a46d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:22.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.736+0000 7fe13ea00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe140073ea0 0x7fe1401a46d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:58438/0 (socket says 172.21.15.90:58438) 2024-04-02T14:37:22.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.736+0000 7fe13ea00640 1 -- 172.21.15.90:0/3695923973 learned_addr learned my addr 172.21.15.90:0/3695923973 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:37:22.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.736+0000 7fe13f400640 1 --2- 172.21.15.90:0/3695923973 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe140112a90 0x7fe14019e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:22.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.736+0000 7fe13e000640 1 --2- 172.21.15.90:0/3695923973 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe1400747c0 0x7fe1401a4c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:22.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.736+0000 7fe13ea00640 1 -- 172.21.15.90:0/3695923973 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe140112a90 msgr2=0x7fe14019e880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:22.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.736+0000 7fe13ea00640 1 --2- 172.21.15.90:0/3695923973 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe140112a90 0x7fe14019e880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:22.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.736+0000 7fe13ea00640 1 -- 172.21.15.90:0/3695923973 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe1400747c0 msgr2=0x7fe1401a4c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:22.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.736+0000 7fe13ea00640 1 --2- 172.21.15.90:0/3695923973 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe1400747c0 0x7fe1401a4c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:22.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.737+0000 7fe13ea00640 1 -- 172.21.15.90:0/3695923973 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe14019f000 con 0x7fe140073ea0 2024-04-02T14:37:22.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.737+0000 7fe13ea00640 1 --2- 172.21.15.90:0/3695923973 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe140073ea0 0x7fe1401a46d0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fe130000c00 tx=0x7fe1300045e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:37:22.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.737+0000 7fe144a00640 1 -- 172.21.15.90:0/3695923973 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe130042020 con 0x7fe140073ea0 2024-04-02T14:37:22.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.738+0000 7fe144a00640 1 -- 172.21.15.90:0/3695923973 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe1300054e0 con 0x7fe140073ea0 2024-04-02T14:37:22.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.738+0000 7fe144a00640 1 -- 172.21.15.90:0/3695923973 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe130003c20 con 0x7fe140073ea0 2024-04-02T14:37:22.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.738+0000 7fe145e00640 1 -- 172.21.15.90:0/3695923973 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe14019f290 con 0x7fe140073ea0 2024-04-02T14:37:22.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.739+0000 7fe145e00640 1 -- 172.21.15.90:0/3695923973 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1401ab590 con 0x7fe140073ea0 2024-04-02T14:37:22.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.742+0000 7fe144a00640 1 -- 172.21.15.90:0/3695923973 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fe130045020 con 0x7fe140073ea0 2024-04-02T14:37:22.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.743+0000 7fe148bdc740 1 Processor -- start 2024-04-02T14:37:22.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.743+0000 7fe148bdc740 1 -- start start 2024-04-02T14:37:22.745 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.744+0000 7fe148bdc740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55baec5937b0 0x55baec593b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:22.745 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.744+0000 7fe148bdc740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55baec5940d0 0x55baec59c770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:22.745 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.744+0000 7fe148bdc740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55baec59ccb0 0x55baec59f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:22.745 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.744+0000 7fe148bdc740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55baec4f7770 con 0x55baec5937b0 2024-04-02T14:37:22.745 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.744+0000 7fe148bdc740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55baec49f700 con 0x55baec5940d0 2024-04-02T14:37:22.745 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.744+0000 7fe148bdc740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55baec4f3720 con 0x55baec59ccb0 2024-04-02T14:37:22.746 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.744+0000 7fe13ea00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55baec5940d0 0x55baec59c770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:22.746 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.744+0000 7fe13ea00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55baec5940d0 0x55baec59c770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:49002/0 (socket says 172.21.15.90:49002) 2024-04-02T14:37:22.746 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.744+0000 7fe13ea00640 1 -- 172.21.15.90:0/3203720700 learned_addr learned my addr 172.21.15.90:0/3203720700 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:37:22.746 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.745+0000 7fe13e000640 1 --2- 172.21.15.90:0/3203720700 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55baec59ccb0 0x55baec59f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:22.746 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.745+0000 7fe13ea00640 1 -- 172.21.15.90:0/3203720700 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55baec59ccb0 msgr2=0x55baec59f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:37:22.746 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.745+0000 7fe13ea00640 1 --2- 172.21.15.90:0/3203720700 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55baec59ccb0 0x55baec59f0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:22.746 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.745+0000 7fe13ea00640 1 -- 172.21.15.90:0/3203720700 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55baec5937b0 msgr2=0x55baec593b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:37:22.746 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.745+0000 7fe13ea00640 1 --2- 172.21.15.90:0/3203720700 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55baec5937b0 0x55baec593b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:22.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.745+0000 7fe13ea00640 1 -- 172.21.15.90:0/3203720700 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55baec2d57d0 con 0x55baec5940d0 2024-04-02T14:37:22.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.745+0000 7fe13ea00640 1 --2- 172.21.15.90:0/3203720700 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55baec5940d0 0x55baec59c770 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fe1300ef6d0 tx=0x7fe1300fa0e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=637a5f886043e398 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:37:22.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.745+0000 7fe12be00640 1 -- 172.21.15.90:0/3203720700 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1300fac90 con 0x55baec5940d0 2024-04-02T14:37:22.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.745+0000 7fe12be00640 1 -- 172.21.15.90:0/3203720700 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe1300ff070 con 0x55baec5940d0 2024-04-02T14:37:22.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.746+0000 7fe12be00640 1 -- 172.21.15.90:0/3203720700 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1301026e0 con 0x55baec5940d0 2024-04-02T14:37:22.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.746+0000 7fe148bdc740 1 -- 172.21.15.90:0/3203720700 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55baec5940d0 msgr2=0x55baec59c770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:22.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.746+0000 7fe148bdc740 1 --2- 172.21.15.90:0/3203720700 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55baec5940d0 0x55baec59c770 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fe1300ef6d0 tx=0x7fe1300fa0e0 comp rx=0 tx=0).stop 2024-04-02T14:37:22.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.747+0000 7fe148bdc740 1 -- 172.21.15.90:0/3203720700 shutdown_connections 2024-04-02T14:37:22.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.747+0000 7fe148bdc740 1 --2- 172.21.15.90:0/3203720700 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55baec59ccb0 0x55baec59f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:22.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.747+0000 7fe148bdc740 1 --2- 172.21.15.90:0/3203720700 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55baec5940d0 0x55baec59c770 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:22.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.747+0000 7fe148bdc740 1 --2- 172.21.15.90:0/3203720700 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55baec5937b0 0x55baec593b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:22.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.747+0000 7fe148bdc740 1 -- 172.21.15.90:0/3203720700 >> 172.21.15.90:0/3203720700 conn(0x55baec58dee0 msgr2=0x55baec58e2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:37:22.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.747+0000 7fe148bdc740 1 -- 172.21.15.90:0/3203720700 shutdown_connections 2024-04-02T14:37:22.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.747+0000 7fe148bdc740 1 -- 172.21.15.90:0/3203720700 wait complete. 2024-04-02T14:37:22.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.747+0000 7fe148bdc740 1 -- start start 2024-04-02T14:37:22.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.748+0000 7fe148bdc740 1 Processor -- start 2024-04-02T14:37:22.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.748+0000 7fe144a00640 1 --2- 172.21.15.90:0/3695923973 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fe1240a3db0 0x7fe1240a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:22.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.748+0000 7fe148bdc740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55baec5937b0 0x55baec599d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:22.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.748+0000 7fe148bdc740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55baec5940d0 0x55baec59a270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:22.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.748+0000 7fe148bdc740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55baec59ccb0 0x55baec59a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:22.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.748+0000 7fe148bdc740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55baec49f700 con 0x55baec5937b0 2024-04-02T14:37:22.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.748+0000 7fe148bdc740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55baec4b2ac0 con 0x55baec5940d0 2024-04-02T14:37:22.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.748+0000 7fe148bdc740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55baec59f860 con 0x55baec59ccb0 2024-04-02T14:37:22.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.748+0000 7fe148bdc740 10 client.0 _ll_register_callbacks cb 0x55baec4b5180 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:37:22.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.749+0000 7fe13e000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55baec5940d0 0x55baec59a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:22.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.749+0000 7fe13e000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55baec5940d0 0x55baec59a270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:49018/0 (socket says 172.21.15.90:49018) 2024-04-02T14:37:22.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.749+0000 7fe13e000640 1 -- 172.21.15.90:0/1029893268 learned_addr learned my addr 172.21.15.90:0/1029893268 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:37:22.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.749+0000 7fe13e000640 1 -- 172.21.15.90:0/1029893268 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55baec59ccb0 msgr2=0x55baec59a7b0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-02T14:37:22.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.749+0000 7fe13e000640 1 --2- 172.21.15.90:0/1029893268 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55baec59ccb0 0x55baec59a7b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:22.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.749+0000 7fe13e000640 1 -- 172.21.15.90:0/1029893268 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55baec5937b0 msgr2=0x55baec599d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:37:22.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.750+0000 7fe144a00640 1 -- 172.21.15.90:0/3695923973 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7fe1300f1ef0 con 0x7fe140073ea0 2024-04-02T14:37:22.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.750+0000 7fe13ea00640 1 --2- 172.21.15.90:0/1029893268 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55baec5937b0 0x55baec599d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:22.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.750+0000 7fe13f400640 1 --2- 172.21.15.90:0/3695923973 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fe1240a3db0 0x7fe1240a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:22.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.750+0000 7fe13e000640 1 --2- 172.21.15.90:0/1029893268 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55baec5937b0 0x55baec599d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:22.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.750+0000 7fe13e000640 1 -- 172.21.15.90:0/1029893268 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55baec20b3f0 con 0x55baec5940d0 2024-04-02T14:37:22.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.750+0000 7fe13ea00640 1 --2- 172.21.15.90:0/1029893268 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55baec5937b0 0x55baec599d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:37:22.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.751+0000 7fe13e000640 1 --2- 172.21.15.90:0/1029893268 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55baec5940d0 0x55baec59a270 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fe12c00be80 tx=0x7fe12c0117b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b017af7d85a8b498 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:37:22.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.751+0000 7fe128c00640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:37:22.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.751+0000 7fe128c00640 1 -- 172.21.15.90:0/1029893268 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe12c004230 con 0x55baec5940d0 2024-04-02T14:37:22.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.751+0000 7fe128c00640 1 -- 172.21.15.90:0/1029893268 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7fe12c0043d0 con 0x55baec5940d0 2024-04-02T14:37:22.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.751+0000 7fe148bdc740 1 -- 172.21.15.90:0/1029893268 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe1300ff070 con 0x55baec5940d0 2024-04-02T14:37:22.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.751+0000 7fe128c00640 1 -- 172.21.15.90:0/1029893268 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fe12c0035f0 con 0x55baec5940d0 2024-04-02T14:37:22.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.751+0000 7fe148bdc740 10 client.26225 fetch_fsmap learned FSMap version 27 2024-04-02T14:37:22.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.751+0000 7fe148bdc740 1 -- 172.21.15.90:0/1029893268 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55baec1cb0f0 con 0x55baec5940d0 2024-04-02T14:37:22.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.752+0000 7fe128c00640 1 -- 172.21.15.90:0/1029893268 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe1300ff070 con 0x55baec5940d0 2024-04-02T14:37:22.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.752+0000 7fe128c00640 1 -- 172.21.15.90:0/1029893268 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fe12c01ba10 con 0x55baec5940d0 2024-04-02T14:37:22.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.752+0000 7fe148bdc740 10 client.26225 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:37:22.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.752+0000 7fe148bdc740 10 client.26225 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:37:22.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.752+0000 7fe148bdc740 20 client.26225 populate_metadata read hostname 'smithi090' 2024-04-02T14:37:22.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.752+0000 7fe148bdc740 1 --2- 172.21.15.90:0/1029893268 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55baec5ac900 0x55baec5aed10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:22.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.752+0000 7fe148bdc740 4 client.26225 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:37:22.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.752+0000 7fe148bdc740 1 -- 172.21.15.90:0/1029893268 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55baec498630 con 0x55baec5ac900 2024-04-02T14:37:22.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.753+0000 7fe13ea00640 1 --2- 172.21.15.90:0/1029893268 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55baec5ac900 0x55baec5aed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:22.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.754+0000 7fe128c00640 10 client.26225 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:37:22.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.754+0000 7fe13ea00640 1 --2- 172.21.15.90:0/1029893268 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55baec5ac900 0x55baec5aed10 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d21811520bd55fb2 server_cookie=46960c532fb0e1f in_seq=0 out_seq=0 2024-04-02T14:37:22.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.754+0000 7fe13f400640 1 --2- 172.21.15.90:0/3695923973 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fe1240a3db0 0x7fe1240a6270 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fe14006b170 tx=0x7fe138009450 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:37:22.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.755+0000 7fe128c00640 1 -- 172.21.15.90:0/1029893268 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55baec498630 con 0x55baec5ac900 2024-04-02T14:37:22.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.755+0000 7fe128c00640 10 client.26225 handle_command_reply: tid=0 2024-04-02T14:37:22.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.755+0000 7fe148bdc740 1 client.26225 shutdown 2024-04-02T14:37:22.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.755+0000 7fe148bdc740 1 -- 172.21.15.90:0/1029893268 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55baec5940d0 msgr2=0x55baec59a270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:22.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.755+0000 7fe148bdc740 1 --2- 172.21.15.90:0/1029893268 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55baec5940d0 0x55baec59a270 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fe12c00be80 tx=0x7fe12c0117b0 comp rx=0 tx=0).stop 2024-04-02T14:37:22.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.755+0000 7fe148bdc740 1 -- 172.21.15.90:0/1029893268 shutdown_connections 2024-04-02T14:37:22.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.755+0000 7fe148bdc740 1 --2- 172.21.15.90:0/1029893268 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55baec5ac900 0x55baec5aed10 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:22.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.755+0000 7fe148bdc740 1 --2- 172.21.15.90:0/1029893268 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55baec59ccb0 0x55baec59a7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:22.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.755+0000 7fe148bdc740 1 --2- 172.21.15.90:0/1029893268 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55baec5940d0 0x55baec59a270 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:22.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.755+0000 7fe128c00640 0 client.26225 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:37:22.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.755+0000 7fe148bdc740 1 --2- 172.21.15.90:0/1029893268 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55baec5937b0 0x55baec599d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:22.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.755+0000 7fe148bdc740 1 -- 172.21.15.90:0/1029893268 >> 172.21.15.90:0/1029893268 conn(0x55baec58dee0 msgr2=0x55baec58fc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:37:22.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.756+0000 7fe148bdc740 1 -- 172.21.15.90:0/1029893268 shutdown_connections 2024-04-02T14:37:22.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.756+0000 7fe148bdc740 1 -- 172.21.15.90:0/1029893268 wait complete. 2024-04-02T14:37:22.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.756+0000 7fe148bdc740 20 client.26225 trim_cache size 0 max 16384 2024-04-02T14:37:22.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.788+0000 7fe148bdc740 1 Processor -- start 2024-04-02T14:37:22.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.788+0000 7fe148bdc740 1 -- start start 2024-04-02T14:37:22.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.788+0000 7fe148bdc740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55baec5937b0 0x55baec59c460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:22.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.788+0000 7fe148bdc740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55baec5940d0 0x55baec58e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:22.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.788+0000 7fe148bdc740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55baec59ccb0 0x55baec5a8480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:22.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.788+0000 7fe148bdc740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fe1401b3ed0 con 0x55baec5937b0 2024-04-02T14:37:22.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.789+0000 7fe148bdc740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fe1401b5a10 con 0x55baec5940d0 2024-04-02T14:37:22.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.789+0000 7fe148bdc740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fe1401b45a0 con 0x55baec59ccb0 2024-04-02T14:37:22.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.789+0000 7fe13e000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55baec5940d0 0x55baec58e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:22.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.789+0000 7fe13f400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55baec59ccb0 0x55baec5a8480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:22.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.789+0000 7fe13e000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55baec5940d0 0x55baec58e2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:49032/0 (socket says 172.21.15.90:49032) 2024-04-02T14:37:22.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.789+0000 7fe13e000640 1 -- 172.21.15.90:0/80707779 learned_addr learned my addr 172.21.15.90:0/80707779 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:37:22.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.789+0000 7fe13ea00640 1 --2- 172.21.15.90:0/80707779 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55baec5937b0 0x55baec59c460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:22.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.790+0000 7fe13e000640 1 -- 172.21.15.90:0/80707779 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55baec59ccb0 msgr2=0x55baec5a8480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:37:22.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.790+0000 7fe13e000640 1 --2- 172.21.15.90:0/80707779 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55baec59ccb0 0x55baec5a8480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:22.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.790+0000 7fe13e000640 1 -- 172.21.15.90:0/80707779 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55baec5937b0 msgr2=0x55baec59c460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:37:22.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.790+0000 7fe13e000640 1 --2- 172.21.15.90:0/80707779 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55baec5937b0 0x55baec59c460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:22.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.790+0000 7fe13e000640 1 -- 172.21.15.90:0/80707779 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55baec5a89c0 con 0x55baec5940d0 2024-04-02T14:37:22.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.790+0000 7fe13ea00640 1 --2- 172.21.15.90:0/80707779 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55baec5937b0 0x55baec59c460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:37:22.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.790+0000 7fe13e000640 1 --2- 172.21.15.90:0/80707779 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55baec5940d0 0x55baec58e2c0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fe12c005fa0 tx=0x7fe12c00b6e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=630c070bc917a05 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:37:22.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.790+0000 7fe12b400640 1 -- 172.21.15.90:0/80707779 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe12c031070 con 0x55baec5940d0 2024-04-02T14:37:22.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.790+0000 7fe12b400640 1 -- 172.21.15.90:0/80707779 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe12c01b470 con 0x55baec5940d0 2024-04-02T14:37:22.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.791+0000 7fe12b400640 1 -- 172.21.15.90:0/80707779 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe12c028040 con 0x55baec5940d0 2024-04-02T14:37:22.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.791+0000 7fe148bdc740 1 -- 172.21.15.90:0/80707779 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55baec5940d0 msgr2=0x55baec58e2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:22.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.791+0000 7fe148bdc740 1 --2- 172.21.15.90:0/80707779 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55baec5940d0 0x55baec58e2c0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fe12c005fa0 tx=0x7fe12c00b6e0 comp rx=0 tx=0).stop 2024-04-02T14:37:22.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.791+0000 7fe148bdc740 1 -- 172.21.15.90:0/80707779 shutdown_connections 2024-04-02T14:37:22.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.791+0000 7fe148bdc740 1 --2- 172.21.15.90:0/80707779 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55baec59ccb0 0x55baec5a8480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:22.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.791+0000 7fe148bdc740 1 --2- 172.21.15.90:0/80707779 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55baec5940d0 0x55baec58e2c0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:22.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.791+0000 7fe148bdc740 1 --2- 172.21.15.90:0/80707779 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55baec5937b0 0x55baec59c460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:22.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.791+0000 7fe148bdc740 1 -- 172.21.15.90:0/80707779 >> 172.21.15.90:0/80707779 conn(0x55baec58dee0 msgr2=0x55baec599cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:37:22.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.792+0000 7fe148bdc740 1 -- 172.21.15.90:0/80707779 shutdown_connections 2024-04-02T14:37:22.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.792+0000 7fe148bdc740 1 -- 172.21.15.90:0/80707779 wait complete. 2024-04-02T14:37:22.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.792+0000 7fe148bdc740 1 -- start start 2024-04-02T14:37:22.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.793+0000 7fe148bdc740 1 Processor -- start 2024-04-02T14:37:22.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.794+0000 7fe148bdc740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55baec5937b0 0x55baec596b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:22.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.794+0000 7fe148bdc740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55baec5940d0 0x55baec597070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:22.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.794+0000 7fe148bdc740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55baec59ccb0 0x55baec5975b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:22.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.794+0000 7fe148bdc740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fe1401b5340 con 0x55baec5940d0 2024-04-02T14:37:22.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.794+0000 7fe148bdc740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fe1401b4c70 con 0x55baec59ccb0 2024-04-02T14:37:22.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.794+0000 7fe148bdc740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fe1401b1eb0 con 0x55baec5937b0 2024-04-02T14:37:22.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.794+0000 7fe148bdc740 10 client.0 _ll_register_callbacks cb 0x7fe12c00b3a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:37:22.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.794+0000 7fe13e000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55baec5940d0 0x55baec597070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:22.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.794+0000 7fe13f400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55baec59ccb0 0x55baec5975b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:22.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.794+0000 7fe13f400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55baec59ccb0 0x55baec5975b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:49048/0 (socket says 172.21.15.90:49048) 2024-04-02T14:37:22.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.794+0000 7fe13f400640 1 -- 172.21.15.90:0/3814931742 learned_addr learned my addr 172.21.15.90:0/3814931742 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:37:22.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.795+0000 7fe13ea00640 1 --2- 172.21.15.90:0/3814931742 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55baec5937b0 0x55baec596b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:22.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.795+0000 7fe13e000640 1 -- 172.21.15.90:0/3814931742 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55baec5937b0 msgr2=0x55baec596b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:37:22.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.795+0000 7fe13e000640 1 --2- 172.21.15.90:0/3814931742 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55baec5937b0 0x55baec596b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:22.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.795+0000 7fe13e000640 1 -- 172.21.15.90:0/3814931742 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55baec59ccb0 msgr2=0x55baec5975b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:37:22.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.795+0000 7fe13e000640 1 --2- 172.21.15.90:0/3814931742 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55baec59ccb0 0x55baec5975b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:22.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.795+0000 7fe13e000640 1 -- 172.21.15.90:0/3814931742 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55baec611fb0 con 0x55baec5940d0 2024-04-02T14:37:22.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.795+0000 7fe13e000640 1 --2- 172.21.15.90:0/3814931742 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55baec5940d0 0x55baec597070 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x55baec592bb0 tx=0x7fe12c01baf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9f79138f0e2f16c server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:37:22.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.795+0000 7fe128c00640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-02T14:37:22.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.796+0000 7fe128c00640 1 -- 172.21.15.90:0/3814931742 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe12c028050 con 0x55baec5940d0 2024-04-02T14:37:22.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.796+0000 7fe148bdc740 1 -- 172.21.15.90:0/3814931742 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe12c01b470 con 0x55baec5940d0 2024-04-02T14:37:22.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.796+0000 7fe128c00640 1 -- 172.21.15.90:0/3814931742 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7fe12c005220 con 0x55baec5940d0 2024-04-02T14:37:22.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.796+0000 7fe128c00640 1 -- 172.21.15.90:0/3814931742 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fe12c031070 con 0x55baec5940d0 2024-04-02T14:37:22.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.796+0000 7fe148bdc740 10 client.15903 fetch_fsmap learned FSMap version 27 2024-04-02T14:37:22.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.796+0000 7fe148bdc740 1 -- 172.21.15.90:0/3814931742 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55baec6132a0 con 0x55baec5940d0 2024-04-02T14:37:22.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.797+0000 7fe128c00640 1 -- 172.21.15.90:0/3814931742 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe12c01b470 con 0x55baec5940d0 2024-04-02T14:37:22.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.797+0000 7fe128c00640 1 -- 172.21.15.90:0/3814931742 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fe12c02b070 con 0x55baec5940d0 2024-04-02T14:37:22.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.797+0000 7fe148bdc740 10 client.15903 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:37:22.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.797+0000 7fe148bdc740 10 client.15903 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:37:22.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.797+0000 7fe148bdc740 20 client.15903 populate_metadata read hostname 'smithi090' 2024-04-02T14:37:22.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.797+0000 7fe148bdc740 1 --2- 172.21.15.90:0/3814931742 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55baec613900 0x55baec615cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:22.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.797+0000 7fe148bdc740 4 client.15903 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-02T14:37:22.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.797+0000 7fe148bdc740 1 -- 172.21.15.90:0/3814931742 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55baec504cb0 con 0x55baec613900 2024-04-02T14:37:22.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.797+0000 7fe13ea00640 1 --2- 172.21.15.90:0/3814931742 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55baec613900 0x55baec615cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:22.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.798+0000 7fe128c00640 10 client.15903 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:37:22.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.798+0000 7fe13ea00640 1 --2- 172.21.15.90:0/3814931742 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55baec613900 0x55baec615cf0 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3b9c25fba293af4a server_cookie=adb969a818ae4f3 in_seq=0 out_seq=0 2024-04-02T14:37:22.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.799+0000 7fe128c00640 1 -- 172.21.15.90:0/3814931742 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55baec504cb0 con 0x55baec613900 2024-04-02T14:37:22.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.799+0000 7fe128c00640 10 client.15903 handle_command_reply: tid=0 2024-04-02T14:37:22.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.799+0000 7fe148bdc740 1 client.15903 shutdown 2024-04-02T14:37:22.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.799+0000 7fe148bdc740 1 -- 172.21.15.90:0/3814931742 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55baec5940d0 msgr2=0x55baec597070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:22.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.799+0000 7fe148bdc740 1 --2- 172.21.15.90:0/3814931742 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55baec5940d0 0x55baec597070 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x55baec592bb0 tx=0x7fe12c01baf0 comp rx=0 tx=0).stop 2024-04-02T14:37:22.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.799+0000 7fe148bdc740 1 -- 172.21.15.90:0/3814931742 shutdown_connections 2024-04-02T14:37:22.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.799+0000 7fe148bdc740 1 --2- 172.21.15.90:0/3814931742 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55baec613900 0x55baec615cf0 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:22.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.799+0000 7fe148bdc740 1 --2- 172.21.15.90:0/3814931742 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55baec59ccb0 0x55baec5975b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:22.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.799+0000 7fe148bdc740 1 --2- 172.21.15.90:0/3814931742 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55baec5940d0 0x55baec597070 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:22.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.799+0000 7fe148bdc740 1 --2- 172.21.15.90:0/3814931742 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55baec5937b0 0x55baec596b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:22.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.799+0000 7fe148bdc740 1 -- 172.21.15.90:0/3814931742 >> 172.21.15.90:0/3814931742 conn(0x55baec58dee0 msgr2=0x55baec59a560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:37:22.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.799+0000 7fe128c00640 0 client.15903 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:37:22.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.800+0000 7fe148bdc740 1 -- 172.21.15.90:0/3814931742 shutdown_connections 2024-04-02T14:37:22.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.800+0000 7fe148bdc740 1 -- 172.21.15.90:0/3814931742 wait complete. 2024-04-02T14:37:22.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.800+0000 7fe148bdc740 20 client.15903 trim_cache size 0 max 16384 2024-04-02T14:37:22.802 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-02T14:37:22.802 INFO:teuthology.orchestra.run.smithi090.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-02T14:37:22.802 INFO:teuthology.orchestra.run.smithi090.stdout: "scrubs": { 2024-04-02T14:37:22.802 INFO:teuthology.orchestra.run.smithi090.stdout: "b9187b6d-35a7-4838-8e0e-1046f12b3662": { 2024-04-02T14:37:22.803 INFO:teuthology.orchestra.run.smithi090.stdout: "path": "/", 2024-04-02T14:37:22.803 INFO:teuthology.orchestra.run.smithi090.stdout: "tag": "b9187b6d-35a7-4838-8e0e-1046f12b3662", 2024-04-02T14:37:22.803 INFO:teuthology.orchestra.run.smithi090.stdout: "options": "recursive,force" 2024-04-02T14:37:22.803 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:37:22.803 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:37:22.803 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-02T14:37:22.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.801+0000 7fe145e00640 1 -- 172.21.15.90:0/3695923973 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fe1240a3db0 msgr2=0x7fe1240a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:22.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.801+0000 7fe145e00640 1 --2- 172.21.15.90:0/3695923973 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fe1240a3db0 0x7fe1240a6270 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fe14006b170 tx=0x7fe138009450 comp rx=0 tx=0).stop 2024-04-02T14:37:22.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.801+0000 7fe145e00640 1 -- 172.21.15.90:0/3695923973 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe140073ea0 msgr2=0x7fe1401a46d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:22.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.801+0000 7fe145e00640 1 --2- 172.21.15.90:0/3695923973 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe140073ea0 0x7fe1401a46d0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fe130000c00 tx=0x7fe1300045e0 comp rx=0 tx=0).stop 2024-04-02T14:37:22.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.802+0000 7fe145e00640 1 -- 172.21.15.90:0/3695923973 shutdown_connections 2024-04-02T14:37:22.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.802+0000 7fe145e00640 1 --2- 172.21.15.90:0/3695923973 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fe1240a3db0 0x7fe1240a6270 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:22.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.802+0000 7fe145e00640 1 --2- 172.21.15.90:0/3695923973 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe140112a90 0x7fe14019e880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:22.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.802+0000 7fe145e00640 1 --2- 172.21.15.90:0/3695923973 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe1400747c0 0x7fe1401a4c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:22.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.802+0000 7fe145e00640 1 --2- 172.21.15.90:0/3695923973 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe140073ea0 0x7fe1401a46d0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:22.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.802+0000 7fe145e00640 1 -- 172.21.15.90:0/3695923973 >> 172.21.15.90:0/3695923973 conn(0x7fe1400fd590 msgr2=0x7fe1400fee20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:37:22.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.803+0000 7fe145e00640 1 -- 172.21.15.90:0/3695923973 shutdown_connections 2024-04-02T14:37:22.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:22.803+0000 7fe145e00640 1 -- 172.21.15.90:0/3695923973 wait complete. 2024-04-02T14:37:22.830 INFO:tasks.cephfs.filesystem:scrub status for tag:b9187b6d-35a7-4838-8e0e-1046f12b3662 - {'path': '/', 'tag': 'b9187b6d-35a7-4838-8e0e-1046f12b3662', 'options': 'recursive,force'} 2024-04-02T14:37:23.860 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:37:23 smithi092 ceph-mon[36497]: pgmap v78: 97 pgs: 97 active+clean; 1.5 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 33 MiB/s wr, 1.11k op/s 2024-04-02T14:37:23.861 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:37:23 smithi092 ceph-mon[36497]: scrub queued for path: / 2024-04-02T14:37:23.861 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:37:23 smithi092 ceph-mon[36497]: scrub summary: idle+waiting paths [/] 2024-04-02T14:37:23.861 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:37:23 smithi092 ceph-mon[36497]: scrub summary: active paths [/] 2024-04-02T14:37:23.861 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:37:23 smithi092 ceph-mon[36497]: scrub summary: idle+waiting paths [/] 2024-04-02T14:37:23.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:37:23 smithi177 ceph-mon[36431]: pgmap v78: 97 pgs: 97 active+clean; 1.5 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 33 MiB/s wr, 1.11k op/s 2024-04-02T14:37:23.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:37:23 smithi177 ceph-mon[36431]: scrub queued for path: / 2024-04-02T14:37:23.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:37:23 smithi177 ceph-mon[36431]: scrub summary: idle+waiting paths [/] 2024-04-02T14:37:23.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:37:23 smithi177 ceph-mon[36431]: scrub summary: active paths [/] 2024-04-02T14:37:23.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:37:23 smithi177 ceph-mon[36431]: scrub summary: idle+waiting paths [/] 2024-04-02T14:37:23.938 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:37:23 smithi090 ceph-mon[31674]: pgmap v78: 97 pgs: 97 active+clean; 1.5 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 33 MiB/s wr, 1.11k op/s 2024-04-02T14:37:23.938 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:37:23 smithi090 ceph-mon[31674]: scrub queued for path: / 2024-04-02T14:37:23.938 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:37:23 smithi090 ceph-mon[31674]: scrub summary: idle+waiting paths [/] 2024-04-02T14:37:23.938 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:37:23 smithi090 ceph-mon[31674]: scrub summary: active paths [/] 2024-04-02T14:37:23.938 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:37:23 smithi090 ceph-mon[31674]: scrub summary: idle+waiting paths [/] 2024-04-02T14:37:25.860 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:37:25 smithi092 ceph-mon[36497]: pgmap v79: 97 pgs: 97 active+clean; 1.6 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 36 MiB/s wr, 1.53k op/s 2024-04-02T14:37:25.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:37:25 smithi177 ceph-mon[36431]: pgmap v79: 97 pgs: 97 active+clean; 1.6 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 36 MiB/s wr, 1.53k op/s 2024-04-02T14:37:25.938 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:37:25 smithi090 ceph-mon[31674]: pgmap v79: 97 pgs: 97 active+clean; 1.6 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 36 MiB/s wr, 1.53k op/s 2024-04-02T14:37:26.832 INFO:teuthology.orchestra.run.smithi090.stderr:progress: 115.0 s, 3055.6 tps, lat 32.673 ms stddev 39.674 2024-04-02T14:37:27.860 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:37:27 smithi092 ceph-mon[36497]: pgmap v80: 97 pgs: 97 active+clean; 1.6 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 35 MiB/s wr, 1.37k op/s 2024-04-02T14:37:27.861 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:37:27 smithi092 ceph-mon[36497]: scrub summary: idle 2024-04-02T14:37:27.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:37:27 smithi177 ceph-mon[36431]: pgmap v80: 97 pgs: 97 active+clean; 1.6 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 35 MiB/s wr, 1.37k op/s 2024-04-02T14:37:27.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:37:27 smithi177 ceph-mon[36431]: scrub summary: idle 2024-04-02T14:37:27.940 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:37:27 smithi090 ceph-mon[31674]: pgmap v80: 97 pgs: 97 active+clean; 1.6 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 35 MiB/s wr, 1.37k op/s 2024-04-02T14:37:27.940 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:37:27 smithi090 ceph-mon[31674]: scrub summary: idle 2024-04-02T14:37:29.860 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:37:29 smithi092 ceph-mon[36497]: pgmap v81: 97 pgs: 97 active+clean; 1.6 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 37 MiB/s wr, 1.53k op/s 2024-04-02T14:37:29.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:37:29 smithi177 ceph-mon[36431]: pgmap v81: 97 pgs: 97 active+clean; 1.6 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 37 MiB/s wr, 1.53k op/s 2024-04-02T14:37:29.938 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:37:29 smithi090 ceph-mon[31674]: pgmap v81: 97 pgs: 97 active+clean; 1.6 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 37 MiB/s wr, 1.53k op/s 2024-04-02T14:37:30.745 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.744+0000 7f3e1ca00640 1 -- 172.21.15.90:0/2715843980 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3e18112790 msgr2=0x7f3e18114b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:30.745 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.744+0000 7f3e1ca00640 1 --2- 172.21.15.90:0/2715843980 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3e18112790 0x7f3e18114b80 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f3e10009ca0 tx=0x7f3e1002c100 comp rx=0 tx=0).stop 2024-04-02T14:37:30.746 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.744+0000 7f3e1ca00640 1 -- 172.21.15.90:0/2715843980 shutdown_connections 2024-04-02T14:37:30.746 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.744+0000 7f3e1ca00640 1 --2- 172.21.15.90:0/2715843980 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3e18112790 0x7f3e18114b80 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:30.746 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.744+0000 7f3e1ca00640 1 --2- 172.21.15.90:0/2715843980 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3e18076950 0x7f3e18112250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:30.746 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.744+0000 7f3e1ca00640 1 --2- 172.21.15.90:0/2715843980 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3e18076030 0x7f3e18076410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:30.746 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.744+0000 7f3e1ca00640 1 -- 172.21.15.90:0/2715843980 >> 172.21.15.90:0/2715843980 conn(0x7f3e180ff5e0 msgr2=0x7f3e18101a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:37:30.746 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.745+0000 7f3e1ca00640 1 -- 172.21.15.90:0/2715843980 shutdown_connections 2024-04-02T14:37:30.746 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.745+0000 7f3e1ca00640 1 -- 172.21.15.90:0/2715843980 wait complete. 2024-04-02T14:37:30.747 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.746+0000 7f3e1ca00640 1 Processor -- start 2024-04-02T14:37:30.747 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.746+0000 7f3e1ca00640 1 -- start start 2024-04-02T14:37:30.747 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.746+0000 7f3e1ca00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3e18076030 0x7f3e181a4620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:30.748 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.746+0000 7f3e1ca00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3e18076950 0x7f3e181a4b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:30.748 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.746+0000 7f3e1ca00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3e18112790 0x7f3e1819e730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:30.748 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.746+0000 7f3e1ca00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f3e181182b0 con 0x7f3e18076950 2024-04-02T14:37:30.748 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.746+0000 7f3e1ca00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f3e18118130 con 0x7f3e18076030 2024-04-02T14:37:30.748 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.746+0000 7f3e1ca00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f3e18118430 con 0x7f3e18112790 2024-04-02T14:37:30.749 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.747+0000 7f3e17400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3e18076030 0x7f3e181a4620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:30.749 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.747+0000 7f3e16a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3e18076950 0x7f3e181a4b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:30.750 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.747+0000 7f3e17400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3e18076030 0x7f3e181a4620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:39610/0 (socket says 172.21.15.90:39610) 2024-04-02T14:37:30.750 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.747+0000 7f3e17400640 1 -- 172.21.15.90:0/585243875 learned_addr learned my addr 172.21.15.90:0/585243875 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:37:30.750 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.747+0000 7f3e17e00640 1 --2- 172.21.15.90:0/585243875 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3e18112790 0x7f3e1819e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:30.750 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.747+0000 7f3e17400640 1 -- 172.21.15.90:0/585243875 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3e18112790 msgr2=0x7f3e1819e730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:30.750 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.747+0000 7f3e17400640 1 --2- 172.21.15.90:0/585243875 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3e18112790 0x7f3e1819e730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:30.750 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.747+0000 7f3e17400640 1 -- 172.21.15.90:0/585243875 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3e18076950 msgr2=0x7f3e181a4b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:30.750 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.747+0000 7f3e17400640 1 --2- 172.21.15.90:0/585243875 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3e18076950 0x7f3e181a4b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:30.750 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.747+0000 7f3e17400640 1 -- 172.21.15.90:0/585243875 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e1819eff0 con 0x7f3e18076030 2024-04-02T14:37:30.750 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.748+0000 7f3e17400640 1 --2- 172.21.15.90:0/585243875 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3e18076030 0x7f3e181a4620 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f3e0800d950 tx=0x7f3e0800de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:37:30.750 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.748+0000 7f3e0fe00640 1 -- 172.21.15.90:0/585243875 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e08017070 con 0x7f3e18076030 2024-04-02T14:37:30.751 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.748+0000 7f3e0fe00640 1 -- 172.21.15.90:0/585243875 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3e08009c30 con 0x7f3e18076030 2024-04-02T14:37:30.751 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.748+0000 7f3e1ca00640 1 -- 172.21.15.90:0/585243875 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e1819f2e0 con 0x7f3e18076030 2024-04-02T14:37:30.751 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.748+0000 7f3e0fe00640 1 -- 172.21.15.90:0/585243875 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e0800f6b0 con 0x7f3e18076030 2024-04-02T14:37:30.751 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.749+0000 7f3e1ca00640 1 -- 172.21.15.90:0/585243875 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e181ab5a0 con 0x7f3e18076030 2024-04-02T14:37:30.754 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.753+0000 7f3e20de6740 1 Processor -- start 2024-04-02T14:37:30.754 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.753+0000 7f3e20de6740 1 -- start start 2024-04-02T14:37:30.755 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.754+0000 7f3e20de6740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558011fe70e0 0x558011fe74c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:30.755 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.754+0000 7f3e20de6740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558011fe7a00 0x558011ff00a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:30.756 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.754+0000 7f3e20de6740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558011ff05e0 0x558011ff29d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:30.756 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.754+0000 7f3e20de6740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x558011f4bd70 con 0x558011ff05e0 2024-04-02T14:37:30.756 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.754+0000 7f3e20de6740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x558011ef42a0 con 0x558011fe70e0 2024-04-02T14:37:30.756 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.754+0000 7f3e20de6740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x558011f47d20 con 0x558011fe7a00 2024-04-02T14:37:30.756 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.754+0000 7f3e0fe00640 1 -- 172.21.15.90:0/585243875 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f3e080109d0 con 0x7f3e18076030 2024-04-02T14:37:30.756 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.754+0000 7f3e16a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558011ff05e0 0x558011ff29d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:30.756 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.754+0000 7f3e17e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558011fe70e0 0x558011fe74c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:30.756 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.754+0000 7f3e16a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558011ff05e0 0x558011ff29d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:46100/0 (socket says 172.21.15.90:46100) 2024-04-02T14:37:30.756 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.754+0000 7f3e16a00640 1 -- 172.21.15.90:0/317441020 learned_addr learned my addr 172.21.15.90:0/317441020 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:37:30.757 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.755+0000 7f3e17400640 1 --2- 172.21.15.90:0/317441020 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558011fe7a00 0x558011ff00a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:30.757 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.755+0000 7f3e16a00640 1 -- 172.21.15.90:0/317441020 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558011fe7a00 msgr2=0x558011ff00a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:37:30.757 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.755+0000 7f3e16a00640 1 --2- 172.21.15.90:0/317441020 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558011fe7a00 0x558011ff00a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:30.757 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.755+0000 7f3e16a00640 1 -- 172.21.15.90:0/317441020 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558011fe70e0 msgr2=0x558011fe74c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:37:30.757 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.755+0000 7f3e16a00640 1 --2- 172.21.15.90:0/317441020 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558011fe70e0 0x558011fe74c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:30.757 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.755+0000 7f3e16a00640 1 -- 172.21.15.90:0/317441020 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558011f08840 con 0x558011ff05e0 2024-04-02T14:37:30.757 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.756+0000 7f3e16a00640 1 --2- 172.21.15.90:0/317441020 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558011ff05e0 0x558011ff29d0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f3e0400bc30 tx=0x7f3e040107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8e89032b50736ba server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:37:30.758 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.757+0000 7f3e0fe00640 1 --2- 172.21.15.90:0/585243875 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f3dfc0a3d90 0x7f3dfc0a6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:30.758 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.757+0000 7f3e0fe00640 1 -- 172.21.15.90:0/585243875 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f3e080d02e0 con 0x7f3e18076030 2024-04-02T14:37:30.759 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.758+0000 7f3e17e00640 1 --2- 172.21.15.90:0/585243875 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f3dfc0a3d90 0x7f3dfc0a6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:30.759 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.758+0000 7f3e03e00640 1 -- 172.21.15.90:0/317441020 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e0401c480 con 0x558011ff05e0 2024-04-02T14:37:30.759 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.758+0000 7f3e03e00640 1 -- 172.21.15.90:0/317441020 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3e0401cb00 con 0x558011ff05e0 2024-04-02T14:37:30.759 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.758+0000 7f3e03e00640 1 -- 172.21.15.90:0/317441020 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e04011c00 con 0x558011ff05e0 2024-04-02T14:37:30.761 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.759+0000 7f3e17e00640 1 --2- 172.21.15.90:0/585243875 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f3dfc0a3d90 0x7f3dfc0a6250 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f3e1002c640 tx=0x7f3e10005d80 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:37:30.761 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.759+0000 7f3e20de6740 1 -- 172.21.15.90:0/317441020 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558011ff05e0 msgr2=0x558011ff29d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:30.761 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.759+0000 7f3e20de6740 1 --2- 172.21.15.90:0/317441020 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558011ff05e0 0x558011ff29d0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f3e0400bc30 tx=0x7f3e040107a0 comp rx=0 tx=0).stop 2024-04-02T14:37:30.761 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.760+0000 7f3e20de6740 1 -- 172.21.15.90:0/317441020 shutdown_connections 2024-04-02T14:37:30.761 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.760+0000 7f3e20de6740 1 --2- 172.21.15.90:0/317441020 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558011ff05e0 0x558011ff29d0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:30.761 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.760+0000 7f3e20de6740 1 --2- 172.21.15.90:0/317441020 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558011fe7a00 0x558011ff00a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:30.761 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.760+0000 7f3e20de6740 1 --2- 172.21.15.90:0/317441020 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558011fe70e0 0x558011fe74c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:30.761 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.760+0000 7f3e20de6740 1 -- 172.21.15.90:0/317441020 >> 172.21.15.90:0/317441020 conn(0x558011fe1810 msgr2=0x558011fe1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:37:30.761 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.760+0000 7f3e20de6740 1 -- 172.21.15.90:0/317441020 shutdown_connections 2024-04-02T14:37:30.762 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.760+0000 7f3e20de6740 1 -- 172.21.15.90:0/317441020 wait complete. 2024-04-02T14:37:30.762 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.760+0000 7f3e20de6740 1 -- start start 2024-04-02T14:37:30.762 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.762+0000 7f3e20de6740 1 Processor -- start 2024-04-02T14:37:30.763 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.762+0000 7f3e20de6740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558011fe70e0 0x558011fed560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:30.763 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.762+0000 7f3e20de6740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558011fe7a00 0x558011fedaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:30.764 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.763+0000 7f3e20de6740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558011ff05e0 0x558011fedfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:30.765 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.763+0000 7f3e20de6740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x558011ff33d0 con 0x558011fe70e0 2024-04-02T14:37:30.765 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.763+0000 7f3e20de6740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x558011f07660 con 0x558011fe7a00 2024-04-02T14:37:30.765 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.763+0000 7f3e20de6740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x558011ff3550 con 0x558011ff05e0 2024-04-02T14:37:30.765 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.763+0000 7f3e20de6740 10 client.0 _ll_register_callbacks cb 0x558011f07050 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:37:30.765 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.763+0000 7f3e16a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558011fe7a00 0x558011fedaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:30.765 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.764+0000 7f3e16a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558011fe7a00 0x558011fedaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:39632/0 (socket says 172.21.15.90:39632) 2024-04-02T14:37:30.765 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.764+0000 7f3e16a00640 1 -- 172.21.15.90:0/895486461 learned_addr learned my addr 172.21.15.90:0/895486461 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:37:30.765 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.764+0000 7f3e17400640 1 --2- 172.21.15.90:0/895486461 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558011fe70e0 0x558011fed560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:30.766 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.764+0000 7f3e17e00640 1 --2- 172.21.15.90:0/895486461 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558011ff05e0 0x558011fedfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:30.766 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.764+0000 7f3e16a00640 1 -- 172.21.15.90:0/895486461 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558011ff05e0 msgr2=0x558011fedfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:37:30.766 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.764+0000 7f3e16a00640 1 --2- 172.21.15.90:0/895486461 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558011ff05e0 0x558011fedfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:30.766 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.764+0000 7f3e16a00640 1 -- 172.21.15.90:0/895486461 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558011fe70e0 msgr2=0x558011fed560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:37:30.766 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.765+0000 7f3e16a00640 1 --2- 172.21.15.90:0/895486461 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558011fe70e0 0x558011fed560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:30.767 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.765+0000 7f3e16a00640 1 -- 172.21.15.90:0/895486461 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558011f16e80 con 0x558011fe7a00 2024-04-02T14:37:30.767 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.765+0000 7f3e17400640 1 --2- 172.21.15.90:0/895486461 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558011fe70e0 0x558011fed560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:37:30.767 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.765+0000 7f3e16a00640 1 --2- 172.21.15.90:0/895486461 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558011fe7a00 0x558011fedaa0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f3e04000c00 tx=0x7f3e0400b800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=714ed8ef19cebe8e server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:37:30.767 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.765+0000 7f3e00c00640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:37:30.768 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.766+0000 7f3e00c00640 1 -- 172.21.15.90:0/895486461 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e0401c890 con 0x558011fe7a00 2024-04-02T14:37:30.768 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.766+0000 7f3e00c00640 1 -- 172.21.15.90:0/895486461 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f3e04011900 con 0x558011fe7a00 2024-04-02T14:37:30.768 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.766+0000 7f3e20de6740 1 -- 172.21.15.90:0/895486461 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3e0401cb00 con 0x558011fe7a00 2024-04-02T14:37:30.768 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.766+0000 7f3e00c00640 1 -- 172.21.15.90:0/895486461 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f3e04032050 con 0x558011fe7a00 2024-04-02T14:37:30.768 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.766+0000 7f3e20de6740 10 client.26243 fetch_fsmap learned FSMap version 27 2024-04-02T14:37:30.768 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.766+0000 7f3e20de6740 1 -- 172.21.15.90:0/895486461 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x558011c60210 con 0x558011fe7a00 2024-04-02T14:37:30.768 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.767+0000 7f3e00c00640 1 -- 172.21.15.90:0/895486461 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3e0401cb00 con 0x558011fe7a00 2024-04-02T14:37:30.769 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.767+0000 7f3e00c00640 1 -- 172.21.15.90:0/895486461 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f3e0402d030 con 0x558011fe7a00 2024-04-02T14:37:30.769 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.767+0000 7f3e20de6740 10 client.26243 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:37:30.769 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.767+0000 7f3e20de6740 10 client.26243 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-02T14:37:30.769 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.767+0000 7f3e20de6740 20 client.26243 populate_metadata read hostname 'smithi090' 2024-04-02T14:37:30.769 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.767+0000 7f3e20de6740 1 --2- 172.21.15.90:0/895486461 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5580120007f0 0x558012002be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:30.769 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.767+0000 7f3e20de6740 4 client.26243 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:37:30.770 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.767+0000 7f3e20de6740 1 -- 172.21.15.90:0/895486461 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558011efbcd0 con 0x5580120007f0 2024-04-02T14:37:30.770 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.768+0000 7f3e17400640 1 --2- 172.21.15.90:0/895486461 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5580120007f0 0x558012002be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:30.770 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.769+0000 7f3e00c00640 10 client.26243 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:37:30.770 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.769+0000 7f3e17400640 1 --2- 172.21.15.90:0/895486461 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5580120007f0 0x558012002be0 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7099f15553003ef6 server_cookie=4e5412efa9ff5eac in_seq=0 out_seq=0 2024-04-02T14:37:30.771 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.770+0000 7f3e00c00640 1 -- 172.21.15.90:0/895486461 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x558011efbcd0 con 0x5580120007f0 2024-04-02T14:37:30.772 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.770+0000 7f3e00c00640 10 client.26243 handle_command_reply: tid=0 2024-04-02T14:37:30.772 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.770+0000 7f3e20de6740 1 client.26243 shutdown 2024-04-02T14:37:30.772 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.770+0000 7f3e20de6740 1 -- 172.21.15.90:0/895486461 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558011fe7a00 msgr2=0x558011fedaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:30.772 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.770+0000 7f3e20de6740 1 --2- 172.21.15.90:0/895486461 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558011fe7a00 0x558011fedaa0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f3e04000c00 tx=0x7f3e0400b800 comp rx=0 tx=0).stop 2024-04-02T14:37:30.772 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.770+0000 7f3e20de6740 1 -- 172.21.15.90:0/895486461 shutdown_connections 2024-04-02T14:37:30.772 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.770+0000 7f3e20de6740 1 --2- 172.21.15.90:0/895486461 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5580120007f0 0x558012002be0 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:30.772 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.770+0000 7f3e20de6740 1 --2- 172.21.15.90:0/895486461 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558011ff05e0 0x558011fedfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:30.772 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.770+0000 7f3e20de6740 1 --2- 172.21.15.90:0/895486461 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558011fe7a00 0x558011fedaa0 secure :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f3e04000c00 tx=0x7f3e0400b800 comp rx=0 tx=0).stop 2024-04-02T14:37:30.772 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.770+0000 7f3e20de6740 1 --2- 172.21.15.90:0/895486461 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558011fe70e0 0x558011fed560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:30.772 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.770+0000 7f3e20de6740 1 -- 172.21.15.90:0/895486461 >> 172.21.15.90:0/895486461 conn(0x558011fe1810 msgr2=0x558011fe3560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:37:30.772 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.770+0000 7f3e00c00640 0 client.26243 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:37:30.772 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.770+0000 7f3e20de6740 1 -- 172.21.15.90:0/895486461 shutdown_connections 2024-04-02T14:37:30.773 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.770+0000 7f3e20de6740 1 -- 172.21.15.90:0/895486461 wait complete. 2024-04-02T14:37:30.773 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.771+0000 7f3e20de6740 20 client.26243 trim_cache size 0 max 16384 2024-04-02T14:37:30.806 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.804+0000 7f3e20de6740 1 Processor -- start 2024-04-02T14:37:30.807 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.804+0000 7f3e20de6740 1 -- start start 2024-04-02T14:37:30.807 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.804+0000 7f3e20de6740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558011fe70e0 0x558011fefc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:30.807 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.804+0000 7f3e20de6740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558011ff05e0 0x558011ffc290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:30.807 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.804+0000 7f3e20de6740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558011fe3820 0x558012008270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:30.807 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.804+0000 7f3e20de6740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f3e181b3d70 con 0x558011ff05e0 2024-04-02T14:37:30.807 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.804+0000 7f3e20de6740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f3e181b58b0 con 0x558011fe70e0 2024-04-02T14:37:30.807 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.804+0000 7f3e20de6740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f3e181b4440 con 0x558011fe3820 2024-04-02T14:37:30.807 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.805+0000 7f3e17400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558011fe70e0 0x558011fefc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:30.807 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.805+0000 7f3e17400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558011fe70e0 0x558011fefc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:39638/0 (socket says 172.21.15.90:39638) 2024-04-02T14:37:30.807 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.805+0000 7f3e17400640 1 -- 172.21.15.90:0/3801833736 learned_addr learned my addr 172.21.15.90:0/3801833736 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:37:30.807 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.805+0000 7f3e17400640 1 -- 172.21.15.90:0/3801833736 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558011fe3820 msgr2=0x558012008270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:37:30.807 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.805+0000 7f3e17400640 1 --2- 172.21.15.90:0/3801833736 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558011fe3820 0x558012008270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:30.807 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.805+0000 7f3e17400640 1 -- 172.21.15.90:0/3801833736 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558011ff05e0 msgr2=0x558011ffc290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:37:30.808 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.805+0000 7f3e16a00640 1 --2- 172.21.15.90:0/3801833736 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558011ff05e0 0x558011ffc290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:30.808 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.805+0000 7f3e17400640 1 --2- 172.21.15.90:0/3801833736 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558011ff05e0 0x558011ffc290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:30.808 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.805+0000 7f3e17400640 1 -- 172.21.15.90:0/3801833736 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558011fe7830 con 0x558011fe70e0 2024-04-02T14:37:30.808 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.805+0000 7f3e16a00640 1 --2- 172.21.15.90:0/3801833736 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558011ff05e0 0x558011ffc290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:37:30.808 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.806+0000 7f3e17400640 1 --2- 172.21.15.90:0/3801833736 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558011fe70e0 0x558011fefc90 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f3e08009610 tx=0x7f3e080d3e30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=30813137a1885598 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:37:30.809 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.806+0000 7f3e03400640 1 -- 172.21.15.90:0/3801833736 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e08020040 con 0x558011fe70e0 2024-04-02T14:37:30.809 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.806+0000 7f3e03400640 1 -- 172.21.15.90:0/3801833736 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3e08009de0 con 0x558011fe70e0 2024-04-02T14:37:30.809 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.806+0000 7f3e03400640 1 -- 172.21.15.90:0/3801833736 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e080cdcc0 con 0x558011fe70e0 2024-04-02T14:37:30.809 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.807+0000 7f3e20de6740 1 -- 172.21.15.90:0/3801833736 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558011fe70e0 msgr2=0x558011fefc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:30.809 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.807+0000 7f3e20de6740 1 --2- 172.21.15.90:0/3801833736 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558011fe70e0 0x558011fefc90 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f3e08009610 tx=0x7f3e080d3e30 comp rx=0 tx=0).stop 2024-04-02T14:37:30.810 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.807+0000 7f3e20de6740 1 -- 172.21.15.90:0/3801833736 shutdown_connections 2024-04-02T14:37:30.810 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.807+0000 7f3e20de6740 1 --2- 172.21.15.90:0/3801833736 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558011fe3820 0x558012008270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:30.810 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.807+0000 7f3e20de6740 1 --2- 172.21.15.90:0/3801833736 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558011ff05e0 0x558011ffc290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:30.810 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.807+0000 7f3e20de6740 1 --2- 172.21.15.90:0/3801833736 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558011fe70e0 0x558011fefc90 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:30.810 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.807+0000 7f3e20de6740 1 -- 172.21.15.90:0/3801833736 >> 172.21.15.90:0/3801833736 conn(0x558011fe1810 msgr2=0x558011fe1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:37:30.810 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.807+0000 7f3e20de6740 1 -- 172.21.15.90:0/3801833736 shutdown_connections 2024-04-02T14:37:30.810 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.807+0000 7f3e20de6740 1 -- 172.21.15.90:0/3801833736 wait complete. 2024-04-02T14:37:30.810 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.807+0000 7f3e20de6740 1 -- start start 2024-04-02T14:37:30.810 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.808+0000 7f3e20de6740 1 Processor -- start 2024-04-02T14:37:30.811 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.809+0000 7f3e20de6740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558011fe3820 0x558011fea560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:30.812 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.809+0000 7f3e20de6740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558011fe70e0 0x558011feaaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:30.812 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.810+0000 7f3e20de6740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558011ff05e0 0x558012065ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:30.812 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.810+0000 7f3e17400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558011fe3820 0x558011fea560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:30.812 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.810+0000 7f3e17400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558011fe3820 0x558011fea560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:49876/0 (socket says 172.21.15.90:49876) 2024-04-02T14:37:30.812 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.810+0000 7f3e16a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558011fe70e0 0x558011feaaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:30.813 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.810+0000 7f3e16a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558011fe70e0 0x558011feaaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:39642/0 (socket says 172.21.15.90:39642) 2024-04-02T14:37:30.813 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.810+0000 7f3e16a00640 1 -- 172.21.15.90:0/1932167529 learned_addr learned my addr 172.21.15.90:0/1932167529 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:37:30.813 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.810+0000 7f3e20de6740 1 -- 172.21.15.90:0/1932167529 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f3e181b51e0 con 0x558011ff05e0 2024-04-02T14:37:30.813 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.810+0000 7f3e20de6740 1 -- 172.21.15.90:0/1932167529 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f3e181b4b10 con 0x558011fe70e0 2024-04-02T14:37:30.813 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.810+0000 7f3e20de6740 1 -- 172.21.15.90:0/1932167529 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f3e181b1d50 con 0x558011fe3820 2024-04-02T14:37:30.813 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.810+0000 7f3e20de6740 10 client.0 _ll_register_callbacks cb 0x7f3e0400ba80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:37:30.813 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.810+0000 7f3e17e00640 1 --2- 172.21.15.90:0/1932167529 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558011ff05e0 0x558012065ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:30.814 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.811+0000 7f3e17400640 1 -- 172.21.15.90:0/1932167529 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558011fe70e0 msgr2=0x558011feaaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:37:30.814 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.811+0000 7f3e17400640 1 --2- 172.21.15.90:0/1932167529 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558011fe70e0 0x558011feaaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:30.814 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.811+0000 7f3e17400640 1 -- 172.21.15.90:0/1932167529 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558011ff05e0 msgr2=0x558012065ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:37:30.814 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.811+0000 7f3e17400640 1 --2- 172.21.15.90:0/1932167529 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558011ff05e0 0x558012065ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:30.814 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.811+0000 7f3e17400640 1 -- 172.21.15.90:0/1932167529 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558011d2c620 con 0x558011fe3820 2024-04-02T14:37:30.815 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.811+0000 7f3e16a00640 1 --2- 172.21.15.90:0/1932167529 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558011fe70e0 0x558011feaaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:37:30.815 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.812+0000 7f3e17400640 1 --2- 172.21.15.90:0/1932167529 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558011fe3820 0x558011fea560 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f3e0809c2d0 tx=0x7f3e080d0270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8ca6cde20395cd7 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:37:30.815 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.812+0000 7f3e00c00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:37:30.815 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.812+0000 7f3e00c00640 1 -- 172.21.15.90:0/1932167529 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e08020070 con 0x558011fe3820 2024-04-02T14:37:30.815 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.812+0000 7f3e00c00640 1 -- 172.21.15.90:0/1932167529 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f3e080989b0 con 0x558011fe3820 2024-04-02T14:37:30.815 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.812+0000 7f3e20de6740 1 -- 172.21.15.90:0/1932167529 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3e08009de0 con 0x558011fe3820 2024-04-02T14:37:30.815 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.813+0000 7f3e00c00640 1 -- 172.21.15.90:0/1932167529 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f3e080a4f00 con 0x558011fe3820 2024-04-02T14:37:30.815 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.813+0000 7f3e20de6740 10 client.26149 fetch_fsmap learned FSMap version 27 2024-04-02T14:37:30.815 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.813+0000 7f3e20de6740 1 -- 172.21.15.90:0/1932167529 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x558012066ed0 con 0x558011fe3820 2024-04-02T14:37:30.816 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.813+0000 7f3e00c00640 1 -- 172.21.15.90:0/1932167529 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3e08009de0 con 0x558011fe3820 2024-04-02T14:37:30.816 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.814+0000 7f3e00c00640 1 -- 172.21.15.90:0/1932167529 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f3e0809b790 con 0x558011fe3820 2024-04-02T14:37:30.816 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.814+0000 7f3e20de6740 10 client.26149 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:37:30.816 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.814+0000 7f3e20de6740 10 client.26149 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-02T14:37:30.816 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.814+0000 7f3e20de6740 20 client.26149 populate_metadata read hostname 'smithi090' 2024-04-02T14:37:30.816 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.814+0000 7f3e20de6740 1 --2- 172.21.15.90:0/1932167529 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x558012067590 0x5580120699b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:30.817 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.814+0000 7f3e20de6740 4 client.26149 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-02T14:37:30.817 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.814+0000 7f3e20de6740 1 -- 172.21.15.90:0/1932167529 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "flush journal"}) -- 0x558011f592b0 con 0x558012067590 2024-04-02T14:37:30.817 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.814+0000 7f3e16a00640 1 --2- 172.21.15.90:0/1932167529 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x558012067590 0x5580120699b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:30.817 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.815+0000 7f3e00c00640 10 client.26149 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:37:30.817 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.815+0000 7f3e16a00640 1 --2- 172.21.15.90:0/1932167529 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x558012067590 0x5580120699b0 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=42cf747bf8946af9 server_cookie=37d044e235748d86 in_seq=0 out_seq=0 2024-04-02T14:37:30.845 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.844+0000 7f3e00c00640 1 -- 172.21.15.90:0/1932167529 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x558011f592b0 con 0x558012067590 2024-04-02T14:37:30.845 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.844+0000 7f3e00c00640 10 client.26149 handle_command_reply: tid=0 2024-04-02T14:37:30.845 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.844+0000 7f3e20de6740 1 client.26149 shutdown 2024-04-02T14:37:30.845 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.844+0000 7f3e20de6740 1 -- 172.21.15.90:0/1932167529 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558011fe3820 msgr2=0x558011fea560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:30.846 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.844+0000 7f3e20de6740 1 --2- 172.21.15.90:0/1932167529 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558011fe3820 0x558011fea560 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f3e0809c2d0 tx=0x7f3e080d0270 comp rx=0 tx=0).stop 2024-04-02T14:37:30.846 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.845+0000 7f3e20de6740 1 -- 172.21.15.90:0/1932167529 shutdown_connections 2024-04-02T14:37:30.847 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.845+0000 7f3e20de6740 1 --2- 172.21.15.90:0/1932167529 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x558012067590 0x5580120699b0 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:30.847 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.845+0000 7f3e20de6740 1 --2- 172.21.15.90:0/1932167529 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558011ff05e0 0x558012065ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:30.847 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.845+0000 7f3e20de6740 1 --2- 172.21.15.90:0/1932167529 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558011fe70e0 0x558011feaaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:30.847 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.845+0000 7f3e20de6740 1 --2- 172.21.15.90:0/1932167529 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558011fe3820 0x558011fea560 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:30.847 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.845+0000 7f3e20de6740 1 -- 172.21.15.90:0/1932167529 >> 172.21.15.90:0/1932167529 conn(0x558011fe1810 msgr2=0x558011fdffe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:37:30.847 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.845+0000 7f3e00c00640 0 client.26149 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:37:30.848 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.846+0000 7f3e20de6740 1 -- 172.21.15.90:0/1932167529 shutdown_connections 2024-04-02T14:37:30.848 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.846+0000 7f3e20de6740 1 -- 172.21.15.90:0/1932167529 wait complete. 2024-04-02T14:37:30.848 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.847+0000 7f3e20de6740 20 client.26149 trim_cache size 0 max 16384 2024-04-02T14:37:30.849 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:{ 2024-04-02T14:37:30.849 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "message": "", 2024-04-02T14:37:30.849 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "return_code": 0 2024-04-02T14:37:30.849 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:} 2024-04-02T14:37:30.849 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.848+0000 7f3e1ca00640 1 -- 172.21.15.90:0/585243875 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f3dfc0a3d90 msgr2=0x7f3dfc0a6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:30.849 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.848+0000 7f3e1ca00640 1 --2- 172.21.15.90:0/585243875 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f3dfc0a3d90 0x7f3dfc0a6250 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f3e1002c640 tx=0x7f3e10005d80 comp rx=0 tx=0).stop 2024-04-02T14:37:30.850 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.848+0000 7f3e1ca00640 1 -- 172.21.15.90:0/585243875 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3e18076030 msgr2=0x7f3e181a4620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:30.850 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.848+0000 7f3e1ca00640 1 --2- 172.21.15.90:0/585243875 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3e18076030 0x7f3e181a4620 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f3e0800d950 tx=0x7f3e0800de10 comp rx=0 tx=0).stop 2024-04-02T14:37:30.850 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.849+0000 7f3e1ca00640 1 -- 172.21.15.90:0/585243875 shutdown_connections 2024-04-02T14:37:30.851 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.849+0000 7f3e1ca00640 1 --2- 172.21.15.90:0/585243875 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f3dfc0a3d90 0x7f3dfc0a6250 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:30.851 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.849+0000 7f3e1ca00640 1 --2- 172.21.15.90:0/585243875 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3e18112790 0x7f3e1819e730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:30.851 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.849+0000 7f3e1ca00640 1 --2- 172.21.15.90:0/585243875 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3e18076950 0x7f3e181a4b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:30.851 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.849+0000 7f3e1ca00640 1 --2- 172.21.15.90:0/585243875 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3e18076030 0x7f3e181a4620 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:30.851 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.849+0000 7f3e1ca00640 1 -- 172.21.15.90:0/585243875 >> 172.21.15.90:0/585243875 conn(0x7f3e180ff5e0 msgr2=0x7f3e181019d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:37:30.851 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.849+0000 7f3e1ca00640 1 -- 172.21.15.90:0/585243875 shutdown_connections 2024-04-02T14:37:30.851 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:30.849+0000 7f3e1ca00640 1 -- 172.21.15.90:0/585243875 wait complete. 2024-04-02T14:37:31.831 INFO:teuthology.orchestra.run.smithi090.stderr:progress: 120.0 s, 2999.8 tps, lat 33.299 ms stddev 36.696 2024-04-02T14:37:31.860 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:37:31 smithi092 ceph-mon[36497]: pgmap v82: 97 pgs: 97 active+clean; 1.6 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 18 MiB/s wr, 1.06k op/s 2024-04-02T14:37:31.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:37:31 smithi177 ceph-mon[36431]: pgmap v82: 97 pgs: 97 active+clean; 1.6 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 18 MiB/s wr, 1.06k op/s 2024-04-02T14:37:31.938 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:37:31 smithi090 ceph-mon[31674]: pgmap v82: 97 pgs: 97 active+clean; 1.6 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 18 MiB/s wr, 1.06k op/s 2024-04-02T14:37:33.861 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:37:33 smithi092 ceph-mon[36497]: pgmap v83: 97 pgs: 97 active+clean; 1.6 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 8.5 MiB/s wr, 760 op/s 2024-04-02T14:37:33.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:37:33 smithi177 ceph-mon[36431]: pgmap v83: 97 pgs: 97 active+clean; 1.6 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 8.5 MiB/s wr, 760 op/s 2024-04-02T14:37:33.939 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:37:33 smithi090 ceph-mon[31674]: pgmap v83: 97 pgs: 97 active+clean; 1.6 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 8.5 MiB/s wr, 760 op/s 2024-04-02T14:37:35.860 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:37:35 smithi092 ceph-mon[36497]: pgmap v84: 97 pgs: 97 active+clean; 1.6 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 11 MiB/s wr, 1.07k op/s 2024-04-02T14:37:35.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:37:35 smithi177 ceph-mon[36431]: pgmap v84: 97 pgs: 97 active+clean; 1.6 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 11 MiB/s wr, 1.07k op/s 2024-04-02T14:37:35.938 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:37:35 smithi090 ceph-mon[31674]: pgmap v84: 97 pgs: 97 active+clean; 1.6 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 11 MiB/s wr, 1.07k op/s 2024-04-02T14:37:36.832 INFO:teuthology.orchestra.run.smithi090.stderr:progress: 125.0 s, 3177.2 tps, lat 31.535 ms stddev 38.039 2024-04-02T14:37:37.860 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:37:37 smithi092 ceph-mon[36497]: pgmap v85: 97 pgs: 97 active+clean; 1.6 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 8.2 MiB/s wr, 655 op/s 2024-04-02T14:37:37.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:37:37 smithi177 ceph-mon[36431]: pgmap v85: 97 pgs: 97 active+clean; 1.6 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 8.2 MiB/s wr, 655 op/s 2024-04-02T14:37:37.938 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:37:37 smithi090 ceph-mon[31674]: pgmap v85: 97 pgs: 97 active+clean; 1.6 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 8.2 MiB/s wr, 655 op/s 2024-04-02T14:37:39.861 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:37:39 smithi092 ceph-mon[36497]: pgmap v86: 97 pgs: 97 active+clean; 1.7 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 11 MiB/s wr, 888 op/s 2024-04-02T14:37:39.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:37:39 smithi177 ceph-mon[36431]: pgmap v86: 97 pgs: 97 active+clean; 1.7 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 11 MiB/s wr, 888 op/s 2024-04-02T14:37:39.939 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:37:39 smithi090 ceph-mon[31674]: pgmap v86: 97 pgs: 97 active+clean; 1.7 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 11 MiB/s wr, 888 op/s 2024-04-02T14:37:41.832 INFO:teuthology.orchestra.run.smithi090.stderr:progress: 130.0 s, 3563.2 tps, lat 27.771 ms stddev 29.061 2024-04-02T14:37:41.860 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:37:41 smithi092 ceph-mon[36497]: pgmap v87: 97 pgs: 97 active+clean; 1.7 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 12 MiB/s wr, 894 op/s 2024-04-02T14:37:41.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:37:41 smithi177 ceph-mon[36431]: pgmap v87: 97 pgs: 97 active+clean; 1.7 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 12 MiB/s wr, 894 op/s 2024-04-02T14:37:41.938 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:37:41 smithi090 ceph-mon[31674]: pgmap v87: 97 pgs: 97 active+clean; 1.7 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 12 MiB/s wr, 894 op/s 2024-04-02T14:37:43.860 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:37:43 smithi092 ceph-mon[36497]: pgmap v88: 97 pgs: 97 active+clean; 1.7 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 8.2 MiB/s wr, 704 op/s 2024-04-02T14:37:43.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:37:43 smithi177 ceph-mon[36431]: pgmap v88: 97 pgs: 97 active+clean; 1.7 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 8.2 MiB/s wr, 704 op/s 2024-04-02T14:37:43.938 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:37:43 smithi090 ceph-mon[31674]: pgmap v88: 97 pgs: 97 active+clean; 1.7 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 8.2 MiB/s wr, 704 op/s 2024-04-02T14:37:44.037 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.034+0000 7f7042200640 1 -- 172.21.15.90:0/2122387843 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f703c076950 msgr2=0x7f703c112410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:44.037 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.034+0000 7f7042200640 1 --2- 172.21.15.90:0/2122387843 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f703c076950 0x7f703c112410 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f7028009930 tx=0x7f702802bc30 comp rx=0 tx=0).stop 2024-04-02T14:37:44.037 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.034+0000 7f7042200640 1 -- 172.21.15.90:0/2122387843 shutdown_connections 2024-04-02T14:37:44.037 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.034+0000 7f7042200640 1 --2- 172.21.15.90:0/2122387843 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f703c112950 0x7f703c114d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:44.037 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.034+0000 7f7042200640 1 --2- 172.21.15.90:0/2122387843 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f703c076950 0x7f703c112410 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:44.037 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.034+0000 7f7042200640 1 --2- 172.21.15.90:0/2122387843 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f703c076030 0x7f703c076410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:44.037 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.034+0000 7f7042200640 1 -- 172.21.15.90:0/2122387843 >> 172.21.15.90:0/2122387843 conn(0x7f703c0ff780 msgr2=0x7f703c101ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:37:44.037 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.034+0000 7f7042200640 1 -- 172.21.15.90:0/2122387843 shutdown_connections 2024-04-02T14:37:44.037 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.034+0000 7f7042200640 1 -- 172.21.15.90:0/2122387843 wait complete. 2024-04-02T14:37:44.038 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.036+0000 7f7042200640 1 Processor -- start 2024-04-02T14:37:44.038 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.036+0000 7f7042200640 1 -- start start 2024-04-02T14:37:44.039 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.037+0000 7f7042200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f703c076030 0x7f703c1a4670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:44.040 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.037+0000 7f7042200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f703c076950 0x7f703c1a4bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:44.040 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.037+0000 7f7042200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f703c112950 0x7f703c19e730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:44.040 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.037+0000 7f7042200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f703c1184f0 con 0x7f703c076950 2024-04-02T14:37:44.040 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.037+0000 7f7042200640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f703c118370 con 0x7f703c112950 2024-04-02T14:37:44.040 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.037+0000 7f7042200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f703c118670 con 0x7f703c076030 2024-04-02T14:37:44.040 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.037+0000 7f703b400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f703c076030 0x7f703c1a4670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:44.040 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.037+0000 7f703aa00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f703c076950 0x7f703c1a4bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:44.040 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.038+0000 7f703aa00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f703c076950 0x7f703c1a4bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:37152/0 (socket says 172.21.15.90:37152) 2024-04-02T14:37:44.040 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.038+0000 7f703aa00640 1 -- 172.21.15.90:0/3794062998 learned_addr learned my addr 172.21.15.90:0/3794062998 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:37:44.040 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.038+0000 7f703b400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f703c076030 0x7f703c1a4670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:54156/0 (socket says 172.21.15.90:54156) 2024-04-02T14:37:44.041 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.038+0000 7f703b400640 1 -- 172.21.15.90:0/3794062998 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f703c112950 msgr2=0x7f703c19e730 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-02T14:37:44.042 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.038+0000 7f703b400640 1 --2- 172.21.15.90:0/3794062998 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f703c112950 0x7f703c19e730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:44.042 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.038+0000 7f703b400640 1 -- 172.21.15.90:0/3794062998 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f703c076950 msgr2=0x7f703c1a4bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:44.042 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.038+0000 7f703b400640 1 --2- 172.21.15.90:0/3794062998 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f703c076950 0x7f703c1a4bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:44.042 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.038+0000 7f703b400640 1 -- 172.21.15.90:0/3794062998 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f703c19eff0 con 0x7f703c076030 2024-04-02T14:37:44.042 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.038+0000 7f703aa00640 1 --2- 172.21.15.90:0/3794062998 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f703c076950 0x7f703c1a4bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:37:44.042 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.039+0000 7f703b400640 1 --2- 172.21.15.90:0/3794062998 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f703c076030 0x7f703c1a4670 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f702c00c970 tx=0x7f702c00ce30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:37:44.042 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.039+0000 7f7033e00640 1 -- 172.21.15.90:0/3794062998 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f702c007bb0 con 0x7f703c076030 2024-04-02T14:37:44.043 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.040+0000 7f7033e00640 1 -- 172.21.15.90:0/3794062998 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f702c007d50 con 0x7f703c076030 2024-04-02T14:37:44.043 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.040+0000 7f7042200640 1 -- 172.21.15.90:0/3794062998 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f703c19f2e0 con 0x7f703c076030 2024-04-02T14:37:44.043 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.040+0000 7f7033e00640 1 -- 172.21.15.90:0/3794062998 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f702c018820 con 0x7f703c076030 2024-04-02T14:37:44.043 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.040+0000 7f7042200640 1 -- 172.21.15.90:0/3794062998 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f703c1ab5f0 con 0x7f703c076030 2024-04-02T14:37:44.046 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.044+0000 7f7033e00640 1 -- 172.21.15.90:0/3794062998 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f702c0189c0 con 0x7f703c076030 2024-04-02T14:37:44.047 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.046+0000 7f7033e00640 1 --2- 172.21.15.90:0/3794062998 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f70200a3e40 0x7f70200a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:44.048 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.046+0000 7f7033e00640 1 -- 172.21.15.90:0/3794062998 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f702c0d0840 con 0x7f703c076030 2024-04-02T14:37:44.048 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.046+0000 7f703aa00640 1 --2- 172.21.15.90:0/3794062998 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f70200a3e40 0x7f70200a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:44.048 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.046+0000 7f7044fe8740 1 Processor -- start 2024-04-02T14:37:44.048 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.046+0000 7f7044fe8740 1 -- start start 2024-04-02T14:37:44.049 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.047+0000 7f7044fe8740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557a42afeeb0 0x557a42aff290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:44.049 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.047+0000 7f7044fe8740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557a42aff7d0 0x557a42b07e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:44.049 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.047+0000 7f7044fe8740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557a42b083b0 0x557a42b0a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:44.050 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.047+0000 7f7044fe8740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x557a42a63d70 con 0x557a42b083b0 2024-04-02T14:37:44.050 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.047+0000 7f7044fe8740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x557a42a0c2a0 con 0x557a42aff7d0 2024-04-02T14:37:44.050 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.047+0000 7f7044fe8740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x557a42a5fd20 con 0x557a42afeeb0 2024-04-02T14:37:44.050 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.047+0000 7f703b400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557a42afeeb0 0x557a42aff290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:44.051 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.047+0000 7f703b400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557a42afeeb0 0x557a42aff290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:54172/0 (socket says 172.21.15.90:54172) 2024-04-02T14:37:44.051 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.047+0000 7f703b400640 1 -- 172.21.15.90:0/4201217889 learned_addr learned my addr 172.21.15.90:0/4201217889 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:37:44.051 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.047+0000 7f703aa00640 1 --2- 172.21.15.90:0/3794062998 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f70200a3e40 0x7f70200a6300 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f70280096c0 tx=0x7f702802df90 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:37:44.051 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.047+0000 7f703be00640 1 --2- 172.21.15.90:0/4201217889 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557a42b083b0 0x557a42b0a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:44.051 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.047+0000 7f703b400640 1 -- 172.21.15.90:0/4201217889 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557a42aff7d0 msgr2=0x557a42b07e70 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-02T14:37:44.051 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.048+0000 7f703b400640 1 --2- 172.21.15.90:0/4201217889 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557a42aff7d0 0x557a42b07e70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:44.051 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.048+0000 7f703b400640 1 -- 172.21.15.90:0/4201217889 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557a42b083b0 msgr2=0x557a42b0a7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:37:44.051 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.048+0000 7f703b400640 1 --2- 172.21.15.90:0/4201217889 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557a42b083b0 0x557a42b0a7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:44.051 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.048+0000 7f703b400640 1 -- 172.21.15.90:0/4201217889 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557a42a20840 con 0x557a42afeeb0 2024-04-02T14:37:44.051 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.048+0000 7f703be00640 1 --2- 172.21.15.90:0/4201217889 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557a42b083b0 0x557a42b0a7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:37:44.052 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.048+0000 7f703b400640 1 --2- 172.21.15.90:0/4201217889 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557a42afeeb0 0x557a42aff290 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f702c099160 tx=0x7f702c09cf50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=af9c47043b86547a server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:37:44.052 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.048+0000 7f7027e00640 1 -- 172.21.15.90:0/4201217889 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f702c0a5400 con 0x557a42afeeb0 2024-04-02T14:37:44.052 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.049+0000 7f7027e00640 1 -- 172.21.15.90:0/4201217889 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f702c0870b0 con 0x557a42afeeb0 2024-04-02T14:37:44.053 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.050+0000 7f7044fe8740 1 -- 172.21.15.90:0/4201217889 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557a42afeeb0 msgr2=0x557a42aff290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:44.053 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.050+0000 7f7044fe8740 1 --2- 172.21.15.90:0/4201217889 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557a42afeeb0 0x557a42aff290 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f702c099160 tx=0x7f702c09cf50 comp rx=0 tx=0).stop 2024-04-02T14:37:44.053 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.050+0000 7f7044fe8740 1 -- 172.21.15.90:0/4201217889 shutdown_connections 2024-04-02T14:37:44.053 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.050+0000 7f7044fe8740 1 --2- 172.21.15.90:0/4201217889 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557a42b083b0 0x557a42b0a7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:44.053 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.050+0000 7f7044fe8740 1 --2- 172.21.15.90:0/4201217889 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557a42aff7d0 0x557a42b07e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:44.053 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.050+0000 7f7044fe8740 1 --2- 172.21.15.90:0/4201217889 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557a42afeeb0 0x557a42aff290 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:44.053 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.050+0000 7f7044fe8740 1 -- 172.21.15.90:0/4201217889 >> 172.21.15.90:0/4201217889 conn(0x557a42af95e0 msgr2=0x557a42af99c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:37:44.053 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.050+0000 7f7044fe8740 1 -- 172.21.15.90:0/4201217889 shutdown_connections 2024-04-02T14:37:44.053 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.050+0000 7f7044fe8740 1 -- 172.21.15.90:0/4201217889 wait complete. 2024-04-02T14:37:44.054 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.051+0000 7f7044fe8740 1 -- start start 2024-04-02T14:37:44.054 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.051+0000 7f7044fe8740 1 Processor -- start 2024-04-02T14:37:44.054 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.051+0000 7f7044fe8740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557a42afeeb0 0x557a42b053b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:44.054 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.051+0000 7f7044fe8740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557a42aff7d0 0x557a42b058f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:44.054 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.051+0000 7f7044fe8740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557a42b083b0 0x557a42b05e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:44.054 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.051+0000 7f7044fe8740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x557a42b0b240 con 0x557a42b083b0 2024-04-02T14:37:44.055 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.051+0000 7f7044fe8740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x557a42a1f660 con 0x557a42aff7d0 2024-04-02T14:37:44.055 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.051+0000 7f7044fe8740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x557a42b0b3c0 con 0x557a42afeeb0 2024-04-02T14:37:44.055 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.051+0000 7f7044fe8740 10 client.0 _ll_register_callbacks cb 0x557a42a1f050 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:37:44.055 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.052+0000 7f703b400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557a42afeeb0 0x557a42b053b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:44.055 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.052+0000 7f703b400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557a42afeeb0 0x557a42b053b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:54174/0 (socket says 172.21.15.90:54174) 2024-04-02T14:37:44.055 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.052+0000 7f703b400640 1 -- 172.21.15.90:0/1303438948 learned_addr learned my addr 172.21.15.90:0/1303438948 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:37:44.056 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.052+0000 7f703aa00640 1 --2- 172.21.15.90:0/1303438948 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557a42aff7d0 0x557a42b058f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:44.056 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.052+0000 7f703be00640 1 --2- 172.21.15.90:0/1303438948 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557a42b083b0 0x557a42b05e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:44.056 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.052+0000 7f703b400640 1 -- 172.21.15.90:0/1303438948 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557a42aff7d0 msgr2=0x557a42b058f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:37:44.056 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.052+0000 7f703b400640 1 --2- 172.21.15.90:0/1303438948 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557a42aff7d0 0x557a42b058f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:44.056 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.052+0000 7f703b400640 1 -- 172.21.15.90:0/1303438948 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557a42b083b0 msgr2=0x557a42b05e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:37:44.056 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.052+0000 7f703b400640 1 --2- 172.21.15.90:0/1303438948 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557a42b083b0 0x557a42b05e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:44.056 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.053+0000 7f703b400640 1 -- 172.21.15.90:0/1303438948 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557a42a2ee80 con 0x557a42afeeb0 2024-04-02T14:37:44.056 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.053+0000 7f703b400640 1 --2- 172.21.15.90:0/1303438948 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557a42afeeb0 0x557a42b053b0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f702c09c950 tx=0x7f702c00fde0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5e07c48427d002da server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:37:44.056 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.053+0000 7f7024c00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:37:44.058 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.053+0000 7f7024c00640 1 -- 172.21.15.90:0/1303438948 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f702c080060 con 0x557a42afeeb0 2024-04-02T14:37:44.058 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.053+0000 7f7024c00640 1 -- 172.21.15.90:0/1303438948 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f702c0a5170 con 0x557a42afeeb0 2024-04-02T14:37:44.058 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.054+0000 7f7044fe8740 1 -- 172.21.15.90:0/1303438948 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f702c0870b0 con 0x557a42afeeb0 2024-04-02T14:37:44.058 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.054+0000 7f7024c00640 1 -- 172.21.15.90:0/1303438948 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f702c0bb070 con 0x557a42afeeb0 2024-04-02T14:37:44.058 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.054+0000 7f7044fe8740 10 client.26158 fetch_fsmap learned FSMap version 27 2024-04-02T14:37:44.058 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.054+0000 7f7044fe8740 1 -- 172.21.15.90:0/1303438948 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x557a42778210 con 0x557a42afeeb0 2024-04-02T14:37:44.058 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.054+0000 7f7024c00640 1 -- 172.21.15.90:0/1303438948 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f702c0870b0 con 0x557a42afeeb0 2024-04-02T14:37:44.058 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.055+0000 7f7024c00640 1 -- 172.21.15.90:0/1303438948 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f702c09b650 con 0x557a42afeeb0 2024-04-02T14:37:44.058 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.055+0000 7f7044fe8740 10 client.26158 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:37:44.058 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.055+0000 7f7044fe8740 10 client.26158 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-02T14:37:44.059 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.055+0000 7f7044fe8740 20 client.26158 populate_metadata read hostname 'smithi090' 2024-04-02T14:37:44.059 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.055+0000 7f7044fe8740 1 --2- 172.21.15.90:0/1303438948 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x557a42b18610 0x557a42b1aa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:44.059 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.055+0000 7f7044fe8740 4 client.26158 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:37:44.059 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.055+0000 7f7044fe8740 1 -- 172.21.15.90:0/1303438948 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557a42a13cd0 con 0x557a42b18610 2024-04-02T14:37:44.059 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.055+0000 7f703aa00640 1 --2- 172.21.15.90:0/1303438948 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x557a42b18610 0x557a42b1aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:44.059 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.056+0000 7f703aa00640 1 --2- 172.21.15.90:0/1303438948 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x557a42b18610 0x557a42b1aa00 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=57c81f0a47bcb92d server_cookie=76a0099faf316a2d in_seq=0 out_seq=0 2024-04-02T14:37:44.059 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.056+0000 7f7024c00640 10 client.26158 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:37:44.060 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.057+0000 7f7024c00640 1 -- 172.21.15.90:0/1303438948 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x557a42a13cd0 con 0x557a42b18610 2024-04-02T14:37:44.060 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.057+0000 7f7024c00640 10 client.26158 handle_command_reply: tid=0 2024-04-02T14:37:44.060 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.057+0000 7f7044fe8740 1 client.26158 shutdown 2024-04-02T14:37:44.061 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.059+0000 7f7044fe8740 1 -- 172.21.15.90:0/1303438948 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557a42afeeb0 msgr2=0x557a42b053b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:44.061 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.059+0000 7f7044fe8740 1 --2- 172.21.15.90:0/1303438948 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557a42afeeb0 0x557a42b053b0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f702c09c950 tx=0x7f702c00fde0 comp rx=0 tx=0).stop 2024-04-02T14:37:44.062 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.059+0000 7f7044fe8740 1 -- 172.21.15.90:0/1303438948 shutdown_connections 2024-04-02T14:37:44.062 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.059+0000 7f7044fe8740 1 --2- 172.21.15.90:0/1303438948 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x557a42b18610 0x557a42b1aa00 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:44.062 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.059+0000 7f7044fe8740 1 --2- 172.21.15.90:0/1303438948 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557a42b083b0 0x557a42b05e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:44.062 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.059+0000 7f7044fe8740 1 --2- 172.21.15.90:0/1303438948 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557a42aff7d0 0x557a42b058f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:44.062 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.059+0000 7f7044fe8740 1 --2- 172.21.15.90:0/1303438948 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557a42afeeb0 0x557a42b053b0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:44.062 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.059+0000 7f7044fe8740 1 -- 172.21.15.90:0/1303438948 >> 172.21.15.90:0/1303438948 conn(0x557a42af95e0 msgr2=0x557a42afb330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:37:44.062 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.059+0000 7f7024c00640 0 client.26158 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:37:44.063 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.061+0000 7f7044fe8740 1 -- 172.21.15.90:0/1303438948 shutdown_connections 2024-04-02T14:37:44.063 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.061+0000 7f7044fe8740 1 -- 172.21.15.90:0/1303438948 wait complete. 2024-04-02T14:37:44.063 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.061+0000 7f7044fe8740 20 client.26158 trim_cache size 0 max 16384 2024-04-02T14:37:44.093 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.091+0000 7f7044fe8740 1 Processor -- start 2024-04-02T14:37:44.093 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.091+0000 7f7044fe8740 1 -- start start 2024-04-02T14:37:44.094 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.092+0000 7f7044fe8740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557a42afeeb0 0x557a42b13da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:44.094 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.092+0000 7f7044fe8740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557a42aff7d0 0x557a42b1d640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:44.095 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.092+0000 7f7044fe8740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557a42b083b0 0x557a42b1db80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:44.095 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.092+0000 7f7044fe8740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f703c1b3e30 con 0x557a42b083b0 2024-04-02T14:37:44.095 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.092+0000 7f7044fe8740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f703c1b5970 con 0x557a42aff7d0 2024-04-02T14:37:44.095 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.092+0000 7f7044fe8740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f703c1b4500 con 0x557a42afeeb0 2024-04-02T14:37:44.095 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.092+0000 7f703b400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557a42afeeb0 0x557a42b13da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:44.095 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.093+0000 7f703b400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557a42afeeb0 0x557a42b13da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:54176/0 (socket says 172.21.15.90:54176) 2024-04-02T14:37:44.095 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.093+0000 7f703b400640 1 -- 172.21.15.90:0/201230282 learned_addr learned my addr 172.21.15.90:0/201230282 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:37:44.096 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.093+0000 7f703aa00640 1 --2- 172.21.15.90:0/201230282 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557a42aff7d0 0x557a42b1d640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:44.096 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.093+0000 7f703b400640 1 -- 172.21.15.90:0/201230282 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557a42aff7d0 msgr2=0x557a42b1d640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:37:44.096 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.093+0000 7f703b400640 1 --2- 172.21.15.90:0/201230282 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557a42aff7d0 0x557a42b1d640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:44.096 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.093+0000 7f703b400640 1 -- 172.21.15.90:0/201230282 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557a42b083b0 msgr2=0x557a42b1db80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:37:44.096 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.093+0000 7f703b400640 1 --2- 172.21.15.90:0/201230282 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557a42b083b0 0x557a42b1db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:44.096 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.093+0000 7f703b400640 1 -- 172.21.15.90:0/201230282 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557a42844620 con 0x557a42afeeb0 2024-04-02T14:37:44.096 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.094+0000 7f703b400640 1 --2- 172.21.15.90:0/201230282 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557a42afeeb0 0x557a42b13da0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f702c09c490 tx=0x7f702c00fec0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7929ce236a93854a server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:37:44.097 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.094+0000 7f7027400640 1 -- 172.21.15.90:0/201230282 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f702c080050 con 0x557a42afeeb0 2024-04-02T14:37:44.097 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.094+0000 7f7027400640 1 -- 172.21.15.90:0/201230282 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f702c018e50 con 0x557a42afeeb0 2024-04-02T14:37:44.097 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.095+0000 7f7027400640 1 -- 172.21.15.90:0/201230282 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f702c0a1e40 con 0x557a42afeeb0 2024-04-02T14:37:44.097 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.095+0000 7f7044fe8740 1 -- 172.21.15.90:0/201230282 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557a42afeeb0 msgr2=0x557a42b13da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:44.098 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.095+0000 7f7044fe8740 1 --2- 172.21.15.90:0/201230282 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557a42afeeb0 0x557a42b13da0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f702c09c490 tx=0x7f702c00fec0 comp rx=0 tx=0).stop 2024-04-02T14:37:44.098 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.096+0000 7f7044fe8740 1 -- 172.21.15.90:0/201230282 shutdown_connections 2024-04-02T14:37:44.098 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.096+0000 7f7044fe8740 1 --2- 172.21.15.90:0/201230282 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557a42b083b0 0x557a42b1db80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:44.098 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.096+0000 7f7044fe8740 1 --2- 172.21.15.90:0/201230282 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557a42aff7d0 0x557a42b1d640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:44.098 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.096+0000 7f7044fe8740 1 --2- 172.21.15.90:0/201230282 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557a42afeeb0 0x557a42b13da0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:44.099 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.096+0000 7f7044fe8740 1 -- 172.21.15.90:0/201230282 >> 172.21.15.90:0/201230282 conn(0x557a42af95e0 msgr2=0x557a42b13690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:37:44.099 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.096+0000 7f7044fe8740 1 -- 172.21.15.90:0/201230282 shutdown_connections 2024-04-02T14:37:44.099 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.096+0000 7f7044fe8740 1 -- 172.21.15.90:0/201230282 wait complete. 2024-04-02T14:37:44.099 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.097+0000 7f7044fe8740 1 -- start start 2024-04-02T14:37:44.099 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.097+0000 7f7044fe8740 1 Processor -- start 2024-04-02T14:37:44.101 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.098+0000 7f7044fe8740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557a42afeeb0 0x557a42b1db60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:44.101 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.098+0000 7f7044fe8740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557a42aff7d0 0x557a42b1aef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:44.101 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.098+0000 7f7044fe8740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557a42b083b0 0x557a42b1b630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:44.101 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.098+0000 7f7044fe8740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f703c1b52a0 con 0x557a42aff7d0 2024-04-02T14:37:44.101 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.098+0000 7f7044fe8740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f703c1b4bd0 con 0x557a42b083b0 2024-04-02T14:37:44.101 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.098+0000 7f7044fe8740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f703c1b1e10 con 0x557a42afeeb0 2024-04-02T14:37:44.101 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.098+0000 7f7044fe8740 10 client.0 _ll_register_callbacks cb 0x7f702c087260 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:37:44.101 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.099+0000 7f703be00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557a42b083b0 0x557a42b1b630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:44.102 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.099+0000 7f703be00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557a42b083b0 0x557a42b1b630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:49858/0 (socket says 172.21.15.90:49858) 2024-04-02T14:37:44.102 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.099+0000 7f703be00640 1 -- 172.21.15.90:0/2814067033 learned_addr learned my addr 172.21.15.90:0/2814067033 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:37:44.102 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.099+0000 7f703aa00640 1 --2- 172.21.15.90:0/2814067033 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557a42aff7d0 0x557a42b1aef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:44.102 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.099+0000 7f703b400640 1 --2- 172.21.15.90:0/2814067033 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557a42afeeb0 0x557a42b1db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:44.102 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.100+0000 7f703be00640 1 -- 172.21.15.90:0/2814067033 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557a42afeeb0 msgr2=0x557a42b1db60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:37:44.102 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.100+0000 7f703be00640 1 --2- 172.21.15.90:0/2814067033 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557a42afeeb0 0x557a42b1db60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:44.103 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.100+0000 7f703be00640 1 -- 172.21.15.90:0/2814067033 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557a42aff7d0 msgr2=0x557a42b1aef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:37:44.103 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.100+0000 7f703be00640 1 --2- 172.21.15.90:0/2814067033 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557a42aff7d0 0x557a42b1aef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:44.103 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.100+0000 7f703be00640 1 -- 172.21.15.90:0/2814067033 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557a42b7d7f0 con 0x557a42b083b0 2024-04-02T14:37:44.103 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.100+0000 7f703aa00640 1 --2- 172.21.15.90:0/2814067033 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557a42aff7d0 0x557a42b1aef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:37:44.103 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.101+0000 7f703be00640 1 --2- 172.21.15.90:0/2814067033 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557a42b083b0 0x557a42b1b630 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f7034009ef0 tx=0x7f7034005410 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3bc247cfb0e8f2b9 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:37:44.103 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.101+0000 7f7024c00640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:37:44.103 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.101+0000 7f7024c00640 1 -- 172.21.15.90:0/2814067033 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7034012860 con 0x557a42b083b0 2024-04-02T14:37:44.104 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.101+0000 7f7024c00640 1 -- 172.21.15.90:0/2814067033 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f7034012a00 con 0x557a42b083b0 2024-04-02T14:37:44.104 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.101+0000 7f7044fe8740 1 -- 172.21.15.90:0/2814067033 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f702c018e50 con 0x557a42b083b0 2024-04-02T14:37:44.104 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.102+0000 7f7024c00640 1 -- 172.21.15.90:0/2814067033 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f7034012c00 con 0x557a42b083b0 2024-04-02T14:37:44.104 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.102+0000 7f7044fe8740 10 client.26261 fetch_fsmap learned FSMap version 27 2024-04-02T14:37:44.104 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.102+0000 7f7044fe8740 1 -- 172.21.15.90:0/2814067033 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x557a42b7ea00 con 0x557a42b083b0 2024-04-02T14:37:44.106 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.104+0000 7f7024c00640 1 -- 172.21.15.90:0/2814067033 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f702c018e50 con 0x557a42b083b0 2024-04-02T14:37:44.106 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.104+0000 7f7024c00640 1 -- 172.21.15.90:0/2814067033 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f703400f040 con 0x557a42b083b0 2024-04-02T14:37:44.107 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.104+0000 7f7044fe8740 10 client.26261 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:37:44.107 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.104+0000 7f7044fe8740 10 client.26261 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-02T14:37:44.107 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.104+0000 7f7044fe8740 20 client.26261 populate_metadata read hostname 'smithi090' 2024-04-02T14:37:44.107 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.104+0000 7f7044fe8740 1 --2- 172.21.15.90:0/2814067033 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x557a42b7f180 0x557a42b815a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:44.107 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.104+0000 7f7044fe8740 4 client.26261 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-02T14:37:44.107 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.104+0000 7f7044fe8740 1 -- 172.21.15.90:0/2814067033 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "flush journal"}) -- 0x557a42a712b0 con 0x557a42b7f180 2024-04-02T14:37:44.108 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.105+0000 7f703b400640 1 --2- 172.21.15.90:0/2814067033 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x557a42b7f180 0x557a42b815a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:44.108 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.105+0000 7f7024c00640 10 client.26261 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:37:44.108 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.105+0000 7f703b400640 1 --2- 172.21.15.90:0/2814067033 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x557a42b7f180 0x557a42b815a0 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=adf8c47116715b4e server_cookie=6575a6758831eec8 in_seq=0 out_seq=0 2024-04-02T14:37:44.124 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.122+0000 7f7024c00640 1 -- 172.21.15.90:0/2814067033 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x557a42a712b0 con 0x557a42b7f180 2024-04-02T14:37:44.124 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.122+0000 7f7024c00640 10 client.26261 handle_command_reply: tid=0 2024-04-02T14:37:44.124 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.122+0000 7f7044fe8740 1 client.26261 shutdown 2024-04-02T14:37:44.124 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.122+0000 7f7044fe8740 1 -- 172.21.15.90:0/2814067033 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557a42b083b0 msgr2=0x557a42b1b630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:44.125 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.122+0000 7f7044fe8740 1 --2- 172.21.15.90:0/2814067033 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557a42b083b0 0x557a42b1b630 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f7034009ef0 tx=0x7f7034005410 comp rx=0 tx=0).stop 2024-04-02T14:37:44.125 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.123+0000 7f7044fe8740 1 -- 172.21.15.90:0/2814067033 shutdown_connections 2024-04-02T14:37:44.125 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.123+0000 7f7044fe8740 1 --2- 172.21.15.90:0/2814067033 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x557a42b7f180 0x557a42b815a0 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:44.125 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.123+0000 7f7044fe8740 1 --2- 172.21.15.90:0/2814067033 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557a42b083b0 0x557a42b1b630 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:44.125 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.123+0000 7f7044fe8740 1 --2- 172.21.15.90:0/2814067033 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557a42aff7d0 0x557a42b1aef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:44.125 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.123+0000 7f7044fe8740 1 --2- 172.21.15.90:0/2814067033 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557a42afeeb0 0x557a42b1db60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:44.125 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.123+0000 7f7044fe8740 1 -- 172.21.15.90:0/2814067033 >> 172.21.15.90:0/2814067033 conn(0x557a42af95e0 msgr2=0x557a42b13f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:37:44.127 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.125+0000 7f7024c00640 0 client.26261 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:37:44.127 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.125+0000 7f7044fe8740 1 -- 172.21.15.90:0/2814067033 shutdown_connections 2024-04-02T14:37:44.129 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.127+0000 7f7044fe8740 1 -- 172.21.15.90:0/2814067033 wait complete. 2024-04-02T14:37:44.130 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.128+0000 7f7044fe8740 20 client.26261 trim_cache size 0 max 16384 2024-04-02T14:37:44.131 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:{ 2024-04-02T14:37:44.131 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "message": "", 2024-04-02T14:37:44.131 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "return_code": 0 2024-04-02T14:37:44.131 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:} 2024-04-02T14:37:44.132 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.131+0000 7f7042200640 1 -- 172.21.15.90:0/3794062998 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f70200a3e40 msgr2=0x7f70200a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:44.133 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.131+0000 7f7042200640 1 --2- 172.21.15.90:0/3794062998 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f70200a3e40 0x7f70200a6300 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f70280096c0 tx=0x7f702802df90 comp rx=0 tx=0).stop 2024-04-02T14:37:44.133 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.131+0000 7f7042200640 1 -- 172.21.15.90:0/3794062998 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f703c076030 msgr2=0x7f703c1a4670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:44.133 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.131+0000 7f7042200640 1 --2- 172.21.15.90:0/3794062998 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f703c076030 0x7f703c1a4670 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f702c00c970 tx=0x7f702c00ce30 comp rx=0 tx=0).stop 2024-04-02T14:37:44.133 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.131+0000 7f7042200640 1 -- 172.21.15.90:0/3794062998 shutdown_connections 2024-04-02T14:37:44.133 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.131+0000 7f7042200640 1 --2- 172.21.15.90:0/3794062998 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f70200a3e40 0x7f70200a6300 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:44.134 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.131+0000 7f7042200640 1 --2- 172.21.15.90:0/3794062998 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f703c112950 0x7f703c19e730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:44.134 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.131+0000 7f7042200640 1 --2- 172.21.15.90:0/3794062998 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f703c076950 0x7f703c1a4bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:44.134 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.131+0000 7f7042200640 1 --2- 172.21.15.90:0/3794062998 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f703c076030 0x7f703c1a4670 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:44.134 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.131+0000 7f7042200640 1 -- 172.21.15.90:0/3794062998 >> 172.21.15.90:0/3794062998 conn(0x7f703c0ff780 msgr2=0x7f703c114360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:37:44.135 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.132+0000 7f7042200640 1 -- 172.21.15.90:0/3794062998 shutdown_connections 2024-04-02T14:37:44.135 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:44.132+0000 7f7042200640 1 -- 172.21.15.90:0/3794062998 wait complete. 2024-04-02T14:37:45.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:37:45 smithi177 ceph-mon[36431]: pgmap v89: 97 pgs: 97 active+clean; 1.7 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 12 MiB/s wr, 1.01k op/s 2024-04-02T14:37:45.938 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:37:45 smithi090 ceph-mon[31674]: pgmap v89: 97 pgs: 97 active+clean; 1.7 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 12 MiB/s wr, 1.01k op/s 2024-04-02T14:37:46.110 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:37:45 smithi092 ceph-mon[36497]: pgmap v89: 97 pgs: 97 active+clean; 1.7 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 12 MiB/s wr, 1.01k op/s 2024-04-02T14:37:46.832 INFO:teuthology.orchestra.run.smithi090.stderr:progress: 135.0 s, 2806.9 tps, lat 35.812 ms stddev 41.520 2024-04-02T14:37:47.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:37:47 smithi177 ceph-mon[36431]: pgmap v90: 97 pgs: 97 active+clean; 1.7 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 9.1 MiB/s wr, 702 op/s 2024-04-02T14:37:47.938 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:37:47 smithi090 ceph-mon[31674]: pgmap v90: 97 pgs: 97 active+clean; 1.7 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 9.1 MiB/s wr, 702 op/s 2024-04-02T14:37:48.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:37:47 smithi092 ceph-mon[36497]: pgmap v90: 97 pgs: 97 active+clean; 1.7 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 9.1 MiB/s wr, 702 op/s 2024-04-02T14:37:49.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:37:49 smithi177 ceph-mon[36431]: pgmap v91: 97 pgs: 97 active+clean; 1.7 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 12 MiB/s wr, 856 op/s 2024-04-02T14:37:49.938 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:37:49 smithi090 ceph-mon[31674]: pgmap v91: 97 pgs: 97 active+clean; 1.7 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 12 MiB/s wr, 856 op/s 2024-04-02T14:37:50.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:37:49 smithi092 ceph-mon[36497]: pgmap v91: 97 pgs: 97 active+clean; 1.7 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 12 MiB/s wr, 856 op/s 2024-04-02T14:37:51.831 INFO:teuthology.orchestra.run.smithi090.stderr:progress: 140.0 s, 3494.8 tps, lat 28.703 ms stddev 33.785 2024-04-02T14:37:51.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:37:51 smithi177 ceph-mon[36431]: pgmap v92: 97 pgs: 97 active+clean; 1.7 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 11 MiB/s wr, 768 op/s 2024-04-02T14:37:51.938 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:37:51 smithi090 ceph-mon[31674]: pgmap v92: 97 pgs: 97 active+clean; 1.7 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 11 MiB/s wr, 768 op/s 2024-04-02T14:37:52.110 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:37:51 smithi092 ceph-mon[36497]: pgmap v92: 97 pgs: 97 active+clean; 1.7 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 11 MiB/s wr, 768 op/s 2024-04-02T14:37:52.832 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-02T14:37:52.832 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-02T14:37:53.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.128+0000 7f6e2e400640 1 -- 172.21.15.90:0/3564482789 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6e28108310 msgr2=0x7f6e281086f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:53.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.128+0000 7f6e2e400640 1 --2- 172.21.15.90:0/3564482789 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6e28108310 0x7f6e281086f0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f6e20009ca0 tx=0x7f6e2002c100 comp rx=0 tx=0).stop 2024-04-02T14:37:53.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.130+0000 7f6e2e400640 1 -- 172.21.15.90:0/3564482789 shutdown_connections 2024-04-02T14:37:53.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.130+0000 7f6e2e400640 1 --2- 172.21.15.90:0/3564482789 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6e2810d3e0 0x7f6e28114fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:53.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.130+0000 7f6e2e400640 1 --2- 172.21.15.90:0/3564482789 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6e28108cc0 0x7f6e2810cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:53.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.131+0000 7f6e2e400640 1 --2- 172.21.15.90:0/3564482789 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6e28108310 0x7f6e281086f0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:53.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.131+0000 7f6e2e400640 1 -- 172.21.15.90:0/3564482789 >> 172.21.15.90:0/3564482789 conn(0x7f6e28076040 msgr2=0x7f6e28078460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:37:53.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.131+0000 7f6e2e400640 1 -- 172.21.15.90:0/3564482789 shutdown_connections 2024-04-02T14:37:53.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.132+0000 7f6e2e400640 1 -- 172.21.15.90:0/3564482789 wait complete. 2024-04-02T14:37:53.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.133+0000 7f6e2e400640 1 Processor -- start 2024-04-02T14:37:53.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.134+0000 7f6e2e400640 1 -- start start 2024-04-02T14:37:53.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.135+0000 7f6e2e400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6e28108310 0x7f6e28073370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:53.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.135+0000 7f6e2e400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6e2810d3e0 0x7f6e2806c3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:53.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.135+0000 7f6e2e400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6e2806c930 0x7f6e2806cde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:53.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.135+0000 7f6e2e400640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f6e28118540 con 0x7f6e2810d3e0 2024-04-02T14:37:53.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.135+0000 7f6e2e400640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f6e281183c0 con 0x7f6e2806c930 2024-04-02T14:37:53.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.135+0000 7f6e2e400640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f6e281186c0 con 0x7f6e28108310 2024-04-02T14:37:53.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.136+0000 7f6e27400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6e28108310 0x7f6e28073370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:53.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.136+0000 7f6e27400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6e28108310 0x7f6e28073370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:54196/0 (socket says 172.21.15.90:54196) 2024-04-02T14:37:53.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.136+0000 7f6e27400640 1 -- 172.21.15.90:0/469384849 learned_addr learned my addr 172.21.15.90:0/469384849 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:37:53.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.136+0000 7f6e27e00640 1 --2- 172.21.15.90:0/469384849 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6e2806c930 0x7f6e2806cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:53.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.136+0000 7f6e27400640 1 -- 172.21.15.90:0/469384849 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6e2806c930 msgr2=0x7f6e2806cde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:53.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.136+0000 7f6e27400640 1 --2- 172.21.15.90:0/469384849 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6e2806c930 0x7f6e2806cde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:53.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.136+0000 7f6e27400640 1 -- 172.21.15.90:0/469384849 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6e2810d3e0 msgr2=0x7f6e2806c3f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T14:37:53.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.136+0000 7f6e26a00640 1 --2- 172.21.15.90:0/469384849 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6e2810d3e0 0x7f6e2806c3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:53.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.137+0000 7f6e27400640 1 --2- 172.21.15.90:0/469384849 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6e2810d3e0 0x7f6e2806c3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:53.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.137+0000 7f6e27400640 1 -- 172.21.15.90:0/469384849 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e28070e90 con 0x7f6e28108310 2024-04-02T14:37:53.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.137+0000 7f6e27e00640 1 --2- 172.21.15.90:0/469384849 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6e2806c930 0x7f6e2806cde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:37:53.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.137+0000 7f6e27400640 1 --2- 172.21.15.90:0/469384849 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6e28108310 0x7f6e28073370 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f6e2002c690 tx=0x7f6e20035d10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:37:53.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.138+0000 7f6e1fe00640 1 -- 172.21.15.90:0/469384849 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e20039040 con 0x7f6e28108310 2024-04-02T14:37:53.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.138+0000 7f6e1fe00640 1 -- 172.21.15.90:0/469384849 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6e20044070 con 0x7f6e28108310 2024-04-02T14:37:53.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.138+0000 7f6e1fe00640 1 -- 172.21.15.90:0/469384849 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e200043a0 con 0x7f6e28108310 2024-04-02T14:37:53.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.138+0000 7f6e2e400640 1 -- 172.21.15.90:0/469384849 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6e28071120 con 0x7f6e28108310 2024-04-02T14:37:53.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.139+0000 7f6e2e400640 1 -- 172.21.15.90:0/469384849 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6e280715c0 con 0x7f6e28108310 2024-04-02T14:37:53.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.144+0000 7f6e31179740 1 Processor -- start 2024-04-02T14:37:53.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.144+0000 7f6e31179740 1 -- start start 2024-04-02T14:37:53.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.145+0000 7f6e31179740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5636e34a37b0 0x5636e34a3b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:53.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.145+0000 7f6e31179740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5636e34a40d0 0x5636e34ac770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:53.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.145+0000 7f6e31179740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5636e34accb0 0x5636e34af0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:53.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.145+0000 7f6e31179740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5636e3407770 con 0x5636e34a40d0 2024-04-02T14:37:53.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.145+0000 7f6e31179740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x5636e33af700 con 0x5636e34a37b0 2024-04-02T14:37:53.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.145+0000 7f6e31179740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5636e3403720 con 0x5636e34accb0 2024-04-02T14:37:53.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.146+0000 7f6e27e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5636e34a37b0 0x5636e34a3b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:53.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.146+0000 7f6e27e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5636e34a37b0 0x5636e34a3b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:49882/0 (socket says 172.21.15.90:49882) 2024-04-02T14:37:53.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.146+0000 7f6e27e00640 1 -- 172.21.15.90:0/472063282 learned_addr learned my addr 172.21.15.90:0/472063282 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:37:53.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.146+0000 7f6e26a00640 1 --2- 172.21.15.90:0/472063282 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5636e34accb0 0x5636e34af0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:53.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.146+0000 7f6e27e00640 1 -- 172.21.15.90:0/472063282 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5636e34accb0 msgr2=0x5636e34af0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:37:53.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.146+0000 7f6e27e00640 1 --2- 172.21.15.90:0/472063282 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5636e34accb0 0x5636e34af0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:53.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.146+0000 7f6e27e00640 1 -- 172.21.15.90:0/472063282 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5636e34a40d0 msgr2=0x5636e34ac770 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-02T14:37:53.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.146+0000 7f6e27e00640 1 --2- 172.21.15.90:0/472063282 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5636e34a40d0 0x5636e34ac770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:53.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.146+0000 7f6e27e00640 1 -- 172.21.15.90:0/472063282 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5636e31e57d0 con 0x5636e34a37b0 2024-04-02T14:37:53.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.147+0000 7f6e27e00640 1 --2- 172.21.15.90:0/472063282 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5636e34a37b0 0x5636e34a3b90 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f6e14009bd0 tx=0x7f6e1400e6c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d5e19e0b471afdb4 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:37:53.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.147+0000 7f6e1fe00640 1 -- 172.21.15.90:0/469384849 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f6e20004540 con 0x7f6e28108310 2024-04-02T14:37:53.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.147+0000 7f6e1cc00640 1 -- 172.21.15.90:0/472063282 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e14016070 con 0x5636e34a37b0 2024-04-02T14:37:53.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.147+0000 7f6e1cc00640 1 -- 172.21.15.90:0/472063282 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6e14011950 con 0x5636e34a37b0 2024-04-02T14:37:53.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.147+0000 7f6e1cc00640 1 -- 172.21.15.90:0/472063282 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e14010980 con 0x5636e34a37b0 2024-04-02T14:37:53.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.150+0000 7f6e31179740 1 -- 172.21.15.90:0/472063282 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5636e34a37b0 msgr2=0x5636e34a3b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:53.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.150+0000 7f6e31179740 1 --2- 172.21.15.90:0/472063282 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5636e34a37b0 0x5636e34a3b90 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f6e14009bd0 tx=0x7f6e1400e6c0 comp rx=0 tx=0).stop 2024-04-02T14:37:53.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.150+0000 7f6e31179740 1 -- 172.21.15.90:0/472063282 shutdown_connections 2024-04-02T14:37:53.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.150+0000 7f6e31179740 1 --2- 172.21.15.90:0/472063282 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5636e34accb0 0x5636e34af0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:53.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.150+0000 7f6e31179740 1 --2- 172.21.15.90:0/472063282 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5636e34a40d0 0x5636e34ac770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:53.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.150+0000 7f6e31179740 1 --2- 172.21.15.90:0/472063282 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5636e34a37b0 0x5636e34a3b90 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:53.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.150+0000 7f6e31179740 1 -- 172.21.15.90:0/472063282 >> 172.21.15.90:0/472063282 conn(0x5636e349dee0 msgr2=0x5636e349e2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:37:53.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.151+0000 7f6e31179740 1 -- 172.21.15.90:0/472063282 shutdown_connections 2024-04-02T14:37:53.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.151+0000 7f6e1fe00640 1 --2- 172.21.15.90:0/469384849 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f6e0c0a3db0 0x7f6e0c0a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:53.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.151+0000 7f6e31179740 1 -- 172.21.15.90:0/472063282 wait complete. 2024-04-02T14:37:53.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.151+0000 7f6e1fe00640 1 -- 172.21.15.90:0/469384849 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f6e200f2d20 con 0x7f6e28108310 2024-04-02T14:37:53.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.152+0000 7f6e31179740 1 -- start start 2024-04-02T14:37:53.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.152+0000 7f6e26a00640 1 --2- 172.21.15.90:0/469384849 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f6e0c0a3db0 0x7f6e0c0a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:53.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.152+0000 7f6e31179740 1 Processor -- start 2024-04-02T14:37:53.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.153+0000 7f6e31179740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5636e34a37b0 0x5636e34a9d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:53.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.153+0000 7f6e31179740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5636e34a40d0 0x5636e34aa2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:53.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.153+0000 7f6e31179740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5636e34accb0 0x5636e34aa7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:53.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.153+0000 7f6e31179740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5636e34afaa0 con 0x5636e34a40d0 2024-04-02T14:37:53.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.153+0000 7f6e31179740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x5636e33c2ac0 con 0x5636e34accb0 2024-04-02T14:37:53.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.153+0000 7f6e31179740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5636e34afc20 con 0x5636e34a37b0 2024-04-02T14:37:53.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.153+0000 7f6e31179740 10 client.0 _ll_register_callbacks cb 0x5636e33c5180 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:37:53.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.154+0000 7f6e27400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5636e34a37b0 0x5636e34a9d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:53.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.154+0000 7f6e27400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5636e34a37b0 0x5636e34a9d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:54202/0 (socket says 172.21.15.90:54202) 2024-04-02T14:37:53.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.154+0000 7f6e27400640 1 -- 172.21.15.90:0/2807167894 learned_addr learned my addr 172.21.15.90:0/2807167894 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:37:53.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.154+0000 7f6e26a00640 1 --2- 172.21.15.90:0/2807167894 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5636e34a40d0 0x5636e34aa2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:53.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.154+0000 7f6e27e00640 1 --2- 172.21.15.90:0/2807167894 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5636e34accb0 0x5636e34aa7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:53.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.155+0000 7f6e27e00640 1 -- 172.21.15.90:0/2807167894 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5636e34a37b0 msgr2=0x5636e34a9d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:37:53.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.155+0000 7f6e27e00640 1 --2- 172.21.15.90:0/2807167894 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5636e34a37b0 0x5636e34a9d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:53.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.155+0000 7f6e27e00640 1 -- 172.21.15.90:0/2807167894 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5636e34a40d0 msgr2=0x5636e34aa2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:37:53.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.155+0000 7f6e27e00640 1 --2- 172.21.15.90:0/2807167894 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5636e34a40d0 0x5636e34aa2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:53.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.155+0000 7f6e27e00640 1 -- 172.21.15.90:0/2807167894 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5636e311b3f0 con 0x5636e34accb0 2024-04-02T14:37:53.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.155+0000 7f6e27400640 1 --2- 172.21.15.90:0/2807167894 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5636e34a37b0 0x5636e34a9d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:37:53.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.155+0000 7f6e26a00640 1 --2- 172.21.15.90:0/2807167894 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5636e34a40d0 0x5636e34aa2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:37:53.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.156+0000 7f6e27e00640 1 --2- 172.21.15.90:0/2807167894 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5636e34accb0 0x5636e34aa7e0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f6e14009bd0 tx=0x7f6e14009ef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a55e35650290d503 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:37:53.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.156+0000 7f6e11600640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:37:53.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.156+0000 7f6e11600640 1 -- 172.21.15.90:0/2807167894 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e14016050 con 0x5636e34accb0 2024-04-02T14:37:53.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.156+0000 7f6e11600640 1 -- 172.21.15.90:0/2807167894 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f6e14005040 con 0x5636e34accb0 2024-04-02T14:37:53.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.156+0000 7f6e31179740 1 -- 172.21.15.90:0/2807167894 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6e14011950 con 0x5636e34accb0 2024-04-02T14:37:53.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.157+0000 7f6e11600640 1 -- 172.21.15.90:0/2807167894 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f6e140097a0 con 0x5636e34accb0 2024-04-02T14:37:53.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.157+0000 7f6e31179740 10 client.26270 fetch_fsmap learned FSMap version 27 2024-04-02T14:37:53.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.157+0000 7f6e31179740 1 -- 172.21.15.90:0/2807167894 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5636e30db0f0 con 0x5636e34accb0 2024-04-02T14:37:53.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.158+0000 7f6e11600640 1 -- 172.21.15.90:0/2807167894 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6e14011950 con 0x5636e34accb0 2024-04-02T14:37:53.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.158+0000 7f6e11600640 1 -- 172.21.15.90:0/2807167894 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f6e1400f040 con 0x5636e34accb0 2024-04-02T14:37:53.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.158+0000 7f6e31179740 10 client.26270 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:37:53.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.158+0000 7f6e31179740 10 client.26270 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:37:53.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.158+0000 7f6e31179740 20 client.26270 populate_metadata read hostname 'smithi090' 2024-04-02T14:37:53.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.158+0000 7f6e31179740 1 --2- 172.21.15.90:0/2807167894 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5636e34ce1d0 0x5636e34d05f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:53.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.158+0000 7f6e31179740 4 client.26270 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:37:53.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.158+0000 7f6e31179740 1 -- 172.21.15.90:0/2807167894 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5636e33a8630 con 0x5636e34ce1d0 2024-04-02T14:37:53.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.159+0000 7f6e27400640 1 --2- 172.21.15.90:0/2807167894 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5636e34ce1d0 0x5636e34d05f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:53.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.161+0000 7f6e11600640 10 client.26270 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:37:53.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.162+0000 7f6e26a00640 1 --2- 172.21.15.90:0/469384849 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f6e0c0a3db0 0x7f6e0c0a6270 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x5636e34ab330 tx=0x7f6e1800c2c0 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:37:53.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.162+0000 7f6e27400640 1 --2- 172.21.15.90:0/2807167894 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5636e34ce1d0 0x5636e34d05f0 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8f2dc4a4cedda51b server_cookie=99048fc410e54328 in_seq=0 out_seq=0 2024-04-02T14:37:53.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.163+0000 7f6e11600640 1 -- 172.21.15.90:0/2807167894 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5636e33a8630 con 0x5636e34ce1d0 2024-04-02T14:37:53.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.163+0000 7f6e11600640 10 client.26270 handle_command_reply: tid=0 2024-04-02T14:37:53.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.163+0000 7f6e31179740 1 client.26270 shutdown 2024-04-02T14:37:53.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.164+0000 7f6e31179740 1 -- 172.21.15.90:0/2807167894 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5636e34accb0 msgr2=0x5636e34aa7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:53.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.164+0000 7f6e31179740 1 --2- 172.21.15.90:0/2807167894 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5636e34accb0 0x5636e34aa7e0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f6e14009bd0 tx=0x7f6e14009ef0 comp rx=0 tx=0).stop 2024-04-02T14:37:53.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.165+0000 7f6e31179740 1 -- 172.21.15.90:0/2807167894 shutdown_connections 2024-04-02T14:37:53.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.165+0000 7f6e31179740 1 --2- 172.21.15.90:0/2807167894 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5636e34ce1d0 0x5636e34d05f0 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:53.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.165+0000 7f6e31179740 1 --2- 172.21.15.90:0/2807167894 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5636e34accb0 0x5636e34aa7e0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:53.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.165+0000 7f6e31179740 1 --2- 172.21.15.90:0/2807167894 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5636e34a40d0 0x5636e34aa2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:53.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.165+0000 7f6e31179740 1 --2- 172.21.15.90:0/2807167894 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5636e34a37b0 0x5636e34a9d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:53.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.165+0000 7f6e31179740 1 -- 172.21.15.90:0/2807167894 >> 172.21.15.90:0/2807167894 conn(0x5636e349dee0 msgr2=0x5636e349fc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:37:53.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.165+0000 7f6e11600640 0 client.26270 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:37:53.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.166+0000 7f6e31179740 1 -- 172.21.15.90:0/2807167894 shutdown_connections 2024-04-02T14:37:53.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.166+0000 7f6e31179740 1 -- 172.21.15.90:0/2807167894 wait complete. 2024-04-02T14:37:53.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.166+0000 7f6e31179740 20 client.26270 trim_cache size 0 max 16384 2024-04-02T14:37:53.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.205+0000 7f6e31179740 1 Processor -- start 2024-04-02T14:37:53.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.205+0000 7f6e31179740 1 -- start start 2024-04-02T14:37:53.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.206+0000 7f6e31179740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5636e34a37b0 0x5636e34ac490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:53.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.206+0000 7f6e31179740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5636e34a40d0 0x5636e34b89c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:53.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.206+0000 7f6e27400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5636e34a37b0 0x5636e34ac490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:53.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.206+0000 7f6e31179740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5636e34accb0 0x5636e34d5ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:53.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.206+0000 7f6e27400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5636e34a37b0 0x5636e34ac490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:54214/0 (socket says 172.21.15.90:54214) 2024-04-02T14:37:53.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.206+0000 7f6e27400640 1 -- 172.21.15.90:0/3200206426 learned_addr learned my addr 172.21.15.90:0/3200206426 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:37:53.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.206+0000 7f6e31179740 1 -- 172.21.15.90:0/3200206426 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f6e28082190 con 0x5636e34a40d0 2024-04-02T14:37:53.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.206+0000 7f6e31179740 1 -- 172.21.15.90:0/3200206426 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f6e281b5bd0 con 0x5636e34accb0 2024-04-02T14:37:53.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.206+0000 7f6e31179740 1 -- 172.21.15.90:0/3200206426 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f6e28082860 con 0x5636e34a37b0 2024-04-02T14:37:53.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.206+0000 7f6e26a00640 1 --2- 172.21.15.90:0/3200206426 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5636e34a40d0 0x5636e34b89c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:53.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.207+0000 7f6e27e00640 1 --2- 172.21.15.90:0/3200206426 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5636e34accb0 0x5636e34d5ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:53.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.207+0000 7f6e26a00640 1 -- 172.21.15.90:0/3200206426 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5636e34a37b0 msgr2=0x5636e34ac490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:37:53.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.207+0000 7f6e26a00640 1 --2- 172.21.15.90:0/3200206426 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5636e34a37b0 0x5636e34ac490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:53.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.207+0000 7f6e26a00640 1 -- 172.21.15.90:0/3200206426 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5636e34accb0 msgr2=0x5636e34d5ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:37:53.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.207+0000 7f6e26a00640 1 --2- 172.21.15.90:0/3200206426 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5636e34accb0 0x5636e34d5ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:53.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.207+0000 7f6e26a00640 1 -- 172.21.15.90:0/3200206426 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5636e349da10 con 0x5636e34a40d0 2024-04-02T14:37:53.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.208+0000 7f6e26a00640 1 --2- 172.21.15.90:0/3200206426 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5636e34a40d0 0x5636e34b89c0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x5636e34a95c0 tx=0x7f6e180047d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f685f6e1d0193b9 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:37:53.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.209+0000 7f6e13e00640 1 -- 172.21.15.90:0/3200206426 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e18027070 con 0x5636e34a40d0 2024-04-02T14:37:53.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.209+0000 7f6e13e00640 1 -- 172.21.15.90:0/3200206426 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6e18015420 con 0x5636e34a40d0 2024-04-02T14:37:53.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.209+0000 7f6e13e00640 1 -- 172.21.15.90:0/3200206426 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e18022670 con 0x5636e34a40d0 2024-04-02T14:37:53.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.210+0000 7f6e31179740 1 -- 172.21.15.90:0/3200206426 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5636e34a40d0 msgr2=0x5636e34b89c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:53.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.210+0000 7f6e31179740 1 --2- 172.21.15.90:0/3200206426 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5636e34a40d0 0x5636e34b89c0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x5636e34a95c0 tx=0x7f6e180047d0 comp rx=0 tx=0).stop 2024-04-02T14:37:53.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.210+0000 7f6e31179740 1 -- 172.21.15.90:0/3200206426 shutdown_connections 2024-04-02T14:37:53.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.210+0000 7f6e31179740 1 --2- 172.21.15.90:0/3200206426 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5636e34accb0 0x5636e34d5ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:53.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.210+0000 7f6e31179740 1 --2- 172.21.15.90:0/3200206426 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5636e34a40d0 0x5636e34b89c0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:53.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.210+0000 7f6e31179740 1 --2- 172.21.15.90:0/3200206426 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5636e34a37b0 0x5636e34ac490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:53.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.210+0000 7f6e31179740 1 -- 172.21.15.90:0/3200206426 >> 172.21.15.90:0/3200206426 conn(0x5636e349dee0 msgr2=0x5636e34a8630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:37:53.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.211+0000 7f6e31179740 1 -- 172.21.15.90:0/3200206426 shutdown_connections 2024-04-02T14:37:53.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.211+0000 7f6e31179740 1 -- 172.21.15.90:0/3200206426 wait complete. 2024-04-02T14:37:53.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.211+0000 7f6e31179740 1 -- start start 2024-04-02T14:37:53.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.212+0000 7f6e31179740 1 Processor -- start 2024-04-02T14:37:53.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.212+0000 7f6e31179740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5636e34a37b0 0x5636e34afe90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:53.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.212+0000 7f6e31179740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5636e34a40d0 0x5636e34b03d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:53.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.212+0000 7f6e31179740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5636e34accb0 0x5636e3532720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:53.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.212+0000 7f6e31179740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f6e281b5500 con 0x5636e34accb0 2024-04-02T14:37:53.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.212+0000 7f6e31179740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f6e281b4e30 con 0x5636e34a37b0 2024-04-02T14:37:53.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.212+0000 7f6e31179740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f6e28080130 con 0x5636e34a40d0 2024-04-02T14:37:53.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.212+0000 7f6e31179740 10 client.0 _ll_register_callbacks cb 0x7f6e14009a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:37:53.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.212+0000 7f6e27400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5636e34a37b0 0x5636e34afe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:53.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.213+0000 7f6e27400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5636e34a37b0 0x5636e34afe90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:49926/0 (socket says 172.21.15.90:49926) 2024-04-02T14:37:53.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.213+0000 7f6e27400640 1 -- 172.21.15.90:0/3261298816 learned_addr learned my addr 172.21.15.90:0/3261298816 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:37:53.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.213+0000 7f6e27e00640 1 --2- 172.21.15.90:0/3261298816 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5636e34accb0 0x5636e3532720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:53.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.213+0000 7f6e26a00640 1 --2- 172.21.15.90:0/3261298816 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5636e34a40d0 0x5636e34b03d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:53.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.214+0000 7f6e26a00640 1 -- 172.21.15.90:0/3261298816 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5636e34a37b0 msgr2=0x5636e34afe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:37:53.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.214+0000 7f6e26a00640 1 --2- 172.21.15.90:0/3261298816 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5636e34a37b0 0x5636e34afe90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:53.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.214+0000 7f6e26a00640 1 -- 172.21.15.90:0/3261298816 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5636e34accb0 msgr2=0x5636e3532720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:37:53.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.214+0000 7f6e26a00640 1 --2- 172.21.15.90:0/3261298816 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5636e34accb0 0x5636e3532720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:53.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.214+0000 7f6e26a00640 1 -- 172.21.15.90:0/3261298816 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5636e3532c60 con 0x5636e34a40d0 2024-04-02T14:37:53.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.214+0000 7f6e27400640 1 --2- 172.21.15.90:0/3261298816 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5636e34a37b0 0x5636e34afe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:37:53.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.214+0000 7f6e27e00640 1 --2- 172.21.15.90:0/3261298816 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5636e34accb0 0x5636e3532720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:37:53.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.214+0000 7f6e11600640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:37:53.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.214+0000 7f6e26a00640 1 --2- 172.21.15.90:0/3261298816 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5636e34a40d0 0x5636e34b03d0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x5636e34cca50 tx=0x7f6e18005bf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=15ca7383e5a24829 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:37:53.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.215+0000 7f6e11600640 1 -- 172.21.15.90:0/3261298816 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e18027070 con 0x5636e34a40d0 2024-04-02T14:37:53.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.215+0000 7f6e11600640 1 -- 172.21.15.90:0/3261298816 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f6e180199b0 con 0x5636e34a40d0 2024-04-02T14:37:53.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.215+0000 7f6e31179740 1 -- 172.21.15.90:0/3261298816 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6e18015420 con 0x5636e34a40d0 2024-04-02T14:37:53.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.215+0000 7f6e11600640 1 -- 172.21.15.90:0/3261298816 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f6e18015e90 con 0x5636e34a40d0 2024-04-02T14:37:53.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.216+0000 7f6e31179740 10 client.26182 fetch_fsmap learned FSMap version 27 2024-04-02T14:37:53.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.216+0000 7f6e31179740 1 -- 172.21.15.90:0/3261298816 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5636e3533d80 con 0x5636e34a40d0 2024-04-02T14:37:53.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.216+0000 7f6e11600640 1 -- 172.21.15.90:0/3261298816 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6e18015420 con 0x5636e34a40d0 2024-04-02T14:37:53.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.216+0000 7f6e11600640 1 -- 172.21.15.90:0/3261298816 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f6e18016070 con 0x5636e34a40d0 2024-04-02T14:37:53.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.216+0000 7f6e31179740 10 client.26182 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:37:53.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.216+0000 7f6e31179740 10 client.26182 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:37:53.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.216+0000 7f6e31179740 20 client.26182 populate_metadata read hostname 'smithi090' 2024-04-02T14:37:53.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.216+0000 7f6e31179740 1 --2- 172.21.15.90:0/3261298816 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5636e3534400 0x5636e3536820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:53.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.216+0000 7f6e31179740 4 client.26182 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-02T14:37:53.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.216+0000 7f6e31179740 1 -- 172.21.15.90:0/3261298816 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5636e3414cb0 con 0x5636e3534400 2024-04-02T14:37:53.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.217+0000 7f6e27400640 1 --2- 172.21.15.90:0/3261298816 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5636e3534400 0x5636e3536820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:53.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.219+0000 7f6e11600640 10 client.26182 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:37:53.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.219+0000 7f6e27400640 1 --2- 172.21.15.90:0/3261298816 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5636e3534400 0x5636e3536820 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4849ad7a1912e403 server_cookie=50b36916a2a43d61 in_seq=0 out_seq=0 2024-04-02T14:37:53.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.220+0000 7f6e11600640 1 -- 172.21.15.90:0/3261298816 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5636e3414cb0 con 0x5636e3534400 2024-04-02T14:37:53.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.220+0000 7f6e11600640 10 client.26182 handle_command_reply: tid=0 2024-04-02T14:37:53.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.220+0000 7f6e31179740 1 client.26182 shutdown 2024-04-02T14:37:53.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.221+0000 7f6e31179740 1 -- 172.21.15.90:0/3261298816 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5636e34a40d0 msgr2=0x5636e34b03d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:53.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.221+0000 7f6e31179740 1 --2- 172.21.15.90:0/3261298816 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5636e34a40d0 0x5636e34b03d0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x5636e34cca50 tx=0x7f6e18005bf0 comp rx=0 tx=0).stop 2024-04-02T14:37:53.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.221+0000 7f6e31179740 1 -- 172.21.15.90:0/3261298816 shutdown_connections 2024-04-02T14:37:53.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.221+0000 7f6e31179740 1 --2- 172.21.15.90:0/3261298816 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5636e3534400 0x5636e3536820 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:53.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.221+0000 7f6e11600640 0 client.26182 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:37:53.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.221+0000 7f6e31179740 1 --2- 172.21.15.90:0/3261298816 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5636e34accb0 0x5636e3532720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:53.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.221+0000 7f6e31179740 1 --2- 172.21.15.90:0/3261298816 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5636e34a40d0 0x5636e34b03d0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:53.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.221+0000 7f6e31179740 1 --2- 172.21.15.90:0/3261298816 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5636e34a37b0 0x5636e34afe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:53.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.221+0000 7f6e31179740 1 -- 172.21.15.90:0/3261298816 >> 172.21.15.90:0/3261298816 conn(0x5636e349dee0 msgr2=0x5636e349c6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:37:53.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.222+0000 7f6e31179740 1 -- 172.21.15.90:0/3261298816 shutdown_connections 2024-04-02T14:37:53.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.222+0000 7f6e31179740 1 -- 172.21.15.90:0/3261298816 wait complete. 2024-04-02T14:37:53.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.222+0000 7f6e31179740 20 client.26182 trim_cache size 0 max 16384 2024-04-02T14:37:53.225 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-02T14:37:53.225 INFO:teuthology.orchestra.run.smithi090.stdout: "status": "no active scrubs running", 2024-04-02T14:37:53.225 INFO:teuthology.orchestra.run.smithi090.stdout: "scrubs": {} 2024-04-02T14:37:53.225 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-02T14:37:53.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.224+0000 7f6e2e400640 1 -- 172.21.15.90:0/469384849 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f6e0c0a3db0 msgr2=0x7f6e0c0a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:53.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.224+0000 7f6e2e400640 1 --2- 172.21.15.90:0/469384849 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f6e0c0a3db0 0x7f6e0c0a6270 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x5636e34ab330 tx=0x7f6e1800c2c0 comp rx=0 tx=0).stop 2024-04-02T14:37:53.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.224+0000 7f6e2e400640 1 -- 172.21.15.90:0/469384849 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6e28108310 msgr2=0x7f6e28073370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:53.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.224+0000 7f6e2e400640 1 --2- 172.21.15.90:0/469384849 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6e28108310 0x7f6e28073370 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f6e2002c690 tx=0x7f6e20035d10 comp rx=0 tx=0).stop 2024-04-02T14:37:53.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.225+0000 7f6e2e400640 1 -- 172.21.15.90:0/469384849 shutdown_connections 2024-04-02T14:37:53.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.225+0000 7f6e2e400640 1 --2- 172.21.15.90:0/469384849 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f6e0c0a3db0 0x7f6e0c0a6270 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:53.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.225+0000 7f6e2e400640 1 --2- 172.21.15.90:0/469384849 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6e2806c930 0x7f6e2806cde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:53.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.225+0000 7f6e2e400640 1 --2- 172.21.15.90:0/469384849 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6e2810d3e0 0x7f6e2806c3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:53.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.225+0000 7f6e2e400640 1 --2- 172.21.15.90:0/469384849 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6e28108310 0x7f6e28073370 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:53.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.225+0000 7f6e2e400640 1 -- 172.21.15.90:0/469384849 >> 172.21.15.90:0/469384849 conn(0x7f6e28076040 msgr2=0x7f6e28078390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:37:53.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.225+0000 7f6e2e400640 1 -- 172.21.15.90:0/469384849 shutdown_connections 2024-04-02T14:37:53.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.225+0000 7f6e2e400640 1 -- 172.21.15.90:0/469384849 wait complete. 2024-04-02T14:37:53.265 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-02T14:37:53.265 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T14:37:53.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.551+0000 7f2185000640 1 -- 172.21.15.90:0/1660102588 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f21801048c0 msgr2=0x7f2180108900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:53.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.551+0000 7f2185000640 1 --2- 172.21.15.90:0/1660102588 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f21801048c0 0x7f2180108900 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f216c009900 tx=0x7f216c02bc40 comp rx=0 tx=0).stop 2024-04-02T14:37:53.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.551+0000 7f2185000640 1 -- 172.21.15.90:0/1660102588 shutdown_connections 2024-04-02T14:37:53.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.551+0000 7f2185000640 1 --2- 172.21.15.90:0/1660102588 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2180108fe0 0x7f2180110bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:53.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.551+0000 7f2185000640 1 --2- 172.21.15.90:0/1660102588 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f21801048c0 0x7f2180108900 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:53.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.551+0000 7f2185000640 1 --2- 172.21.15.90:0/1660102588 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2180103f10 0x7f21801042f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:53.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.551+0000 7f2185000640 1 -- 172.21.15.90:0/1660102588 >> 172.21.15.90:0/1660102588 conn(0x7f21800ff750 msgr2=0x7f2180101b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:37:53.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.551+0000 7f2185000640 1 -- 172.21.15.90:0/1660102588 shutdown_connections 2024-04-02T14:37:53.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.552+0000 7f2185000640 1 -- 172.21.15.90:0/1660102588 wait complete. 2024-04-02T14:37:53.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.553+0000 7f2185000640 1 Processor -- start 2024-04-02T14:37:53.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.553+0000 7f2185000640 1 -- start start 2024-04-02T14:37:53.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.553+0000 7f2185000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2180103f10 0x7f2180073370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:53.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.553+0000 7f2185000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2180108fe0 0x7f218006d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:53.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.553+0000 7f2185000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f218006d920 0x7f218006ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:53.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.553+0000 7f2185000640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f218006bf70 con 0x7f2180103f10 2024-04-02T14:37:53.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.553+0000 7f2185000640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f218006bdf0 con 0x7f2180108fe0 2024-04-02T14:37:53.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.553+0000 7f2185000640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f218006c0f0 con 0x7f218006d920 2024-04-02T14:37:53.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.554+0000 7f217d400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2180108fe0 0x7f218006d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:53.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.554+0000 7f217d400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2180108fe0 0x7f218006d3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:49940/0 (socket says 172.21.15.90:49940) 2024-04-02T14:37:53.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.554+0000 7f217d400640 1 -- 172.21.15.90:0/2848626930 learned_addr learned my addr 172.21.15.90:0/2848626930 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:37:53.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.554+0000 7f217de00640 1 --2- 172.21.15.90:0/2848626930 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2180103f10 0x7f2180073370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:53.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.554+0000 7f217e800640 1 --2- 172.21.15.90:0/2848626930 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f218006d920 0x7f218006ddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:53.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.554+0000 7f217d400640 1 -- 172.21.15.90:0/2848626930 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f218006d920 msgr2=0x7f218006ddd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:53.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.554+0000 7f217d400640 1 --2- 172.21.15.90:0/2848626930 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f218006d920 0x7f218006ddd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:53.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.554+0000 7f217d400640 1 -- 172.21.15.90:0/2848626930 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2180103f10 msgr2=0x7f2180073370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:53.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.554+0000 7f217d400640 1 --2- 172.21.15.90:0/2848626930 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2180103f10 0x7f2180073370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:53.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.554+0000 7f217d400640 1 -- 172.21.15.90:0/2848626930 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21801af420 con 0x7f2180108fe0 2024-04-02T14:37:53.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.555+0000 7f217d400640 1 --2- 172.21.15.90:0/2848626930 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2180108fe0 0x7f218006d3e0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f216c02efd0 tx=0x7f216c02c890 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:37:53.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.555+0000 7f2176a00640 1 -- 172.21.15.90:0/2848626930 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f216c041020 con 0x7f2180108fe0 2024-04-02T14:37:53.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.555+0000 7f2176a00640 1 -- 172.21.15.90:0/2848626930 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f216c0054e0 con 0x7f2180108fe0 2024-04-02T14:37:53.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.555+0000 7f2176a00640 1 -- 172.21.15.90:0/2848626930 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f216c041020 con 0x7f2180108fe0 2024-04-02T14:37:53.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.555+0000 7f2185000640 1 -- 172.21.15.90:0/2848626930 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f21801af650 con 0x7f2180108fe0 2024-04-02T14:37:53.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.555+0000 7f2185000640 1 -- 172.21.15.90:0/2848626930 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21801afa60 con 0x7f2180108fe0 2024-04-02T14:37:53.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.557+0000 7f2185000640 1 -- 172.21.15.90:0/2848626930 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f21801053e0 con 0x7f2180108fe0 2024-04-02T14:37:53.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.559+0000 7f2176a00640 1 -- 172.21.15.90:0/2848626930 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f216c044020 con 0x7f2180108fe0 2024-04-02T14:37:53.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.561+0000 7f2176a00640 1 --2- 172.21.15.90:0/2848626930 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f21640a3fa0 0x7f21640a6460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:53.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.563+0000 7f2176a00640 1 -- 172.21.15.90:0/2848626930 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f216c0bc650 con 0x7f2180108fe0 2024-04-02T14:37:53.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.563+0000 7f217de00640 1 --2- 172.21.15.90:0/2848626930 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f21640a3fa0 0x7f21640a6460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:53.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.565+0000 7f217de00640 1 --2- 172.21.15.90:0/2848626930 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f21640a3fa0 0x7f21640a6460 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f2178005fd0 tx=0x7f2178009870 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:37:53.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:53.567+0000 7f2176a00640 1 -- 172.21.15.90:0/2848626930 <== mon.2 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7f216c04b020 con 0x7f2180108fe0 2024-04-02T14:37:53.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:37:53 smithi177 ceph-mon[36431]: pgmap v93: 97 pgs: 97 active+clean; 1.7 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 8.0 MiB/s wr, 606 op/s 2024-04-02T14:37:53.939 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:37:53 smithi090 ceph-mon[31674]: pgmap v93: 97 pgs: 97 active+clean; 1.7 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 8.0 MiB/s wr, 606 op/s 2024-04-02T14:37:54.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.033+0000 7f2185000640 1 -- 172.21.15.90:0/2848626930 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f21801048f0 con 0x7f2180108fe0 2024-04-02T14:37:54.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.034+0000 7f2176a00640 1 -- 172.21.15.90:0/2848626930 <== mon.2 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11927 (secure 0 0 0) 0x7f216c0becb0 con 0x7f2180108fe0 2024-04-02T14:37:54.038 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-02T14:37:54.038 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14412,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6837/2852160309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6836","nonce":2852160309},{"type":"v1","addr":"172.21.15.90:6837","nonce":2852160309}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14478,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6841/1919376858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6840","nonce":1919376858},{"type":"v1","addr":"172.21.15.90:6841","nonce":1919376858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-02T14:32:22.319083+0000","modified":"2024-04-02T14:34:52.266152+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24590,"mds_1":24560,"mds_2":24547,"mds_3":24539,"mds_4":24523},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24454":{"gid":24454,"name":"l","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6833/1564319431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":1564319431},{"type":"v1","addr":"172.21.15.177:6833","nonce":1564319431}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.92:6833/1570337372","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":1570337372},{"type":"v1","addr":"172.21.15.92:6833","nonce":1570337372}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"j","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.90:6835/1277124891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6834","nonce":1277124891},{"type":"v1","addr":"172.21.15.90:6835","nonce":1277124891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.92:6835/860721164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":860721164},{"type":"v1","addr":"172.21.15.92:6835","nonce":860721164}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24511":{"gid":24511,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6835/3177244988","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":3177244988},{"type":"v1","addr":"172.21.15.177:6835","nonce":3177244988}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"c","rank":4,"incarnation":25,"state":"up:active","state_seq":9,"addr":"172.21.15.177:6837/4223414858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":4223414858},{"type":"v1","addr":"172.21.15.177:6837","nonce":4223414858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24539":{"gid":24539,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":8,"addr":"172.21.15.92:6837/1270635939","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":1270635939},{"type":"v1","addr":"172.21.15.92:6837","nonce":1270635939}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24547":{"gid":24547,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":7,"addr":"172.21.15.177:6839/664895663","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":664895663},{"type":"v1","addr":"172.21.15.177:6839","nonce":664895663}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"g","rank":1,"incarnation":19,"state":"up:active","state_seq":5,"addr":"172.21.15.90:6839/1498915870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6838","nonce":1498915870},{"type":"v1","addr":"172.21.15.90:6839","nonce":1498915870}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24590":{"gid":24590,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.92:6839/1054888219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":1054888219},{"type":"v1","addr":"172.21.15.92:6839","nonce":1054888219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24590,"qdb_cluster":[24547,24590,24523,24539,24560]},"id":1}]} 2024-04-02T14:37:54.038 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 27 2024-04-02T14:37:54.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.038+0000 7f2185000640 1 -- 172.21.15.90:0/2848626930 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f21640a3fa0 msgr2=0x7f21640a6460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:54.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.038+0000 7f2185000640 1 --2- 172.21.15.90:0/2848626930 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f21640a3fa0 0x7f21640a6460 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f2178005fd0 tx=0x7f2178009870 comp rx=0 tx=0).stop 2024-04-02T14:37:54.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.038+0000 7f2185000640 1 -- 172.21.15.90:0/2848626930 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2180108fe0 msgr2=0x7f218006d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:54.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.038+0000 7f2185000640 1 --2- 172.21.15.90:0/2848626930 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2180108fe0 0x7f218006d3e0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f216c02efd0 tx=0x7f216c02c890 comp rx=0 tx=0).stop 2024-04-02T14:37:54.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.038+0000 7f2185000640 1 -- 172.21.15.90:0/2848626930 shutdown_connections 2024-04-02T14:37:54.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.038+0000 7f2185000640 1 --2- 172.21.15.90:0/2848626930 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f21640a3fa0 0x7f21640a6460 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:54.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.039+0000 7f2185000640 1 --2- 172.21.15.90:0/2848626930 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f218006d920 0x7f218006ddd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:54.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.039+0000 7f2185000640 1 --2- 172.21.15.90:0/2848626930 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2180108fe0 0x7f218006d3e0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:54.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.039+0000 7f2185000640 1 --2- 172.21.15.90:0/2848626930 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2180103f10 0x7f2180073370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:54.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.039+0000 7f2185000640 1 -- 172.21.15.90:0/2848626930 >> 172.21.15.90:0/2848626930 conn(0x7f21800ff750 msgr2=0x7f21801000b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:37:54.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.039+0000 7f2185000640 1 -- 172.21.15.90:0/2848626930 shutdown_connections 2024-04-02T14:37:54.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.039+0000 7f2185000640 1 -- 172.21.15.90:0/2848626930 wait complete. 2024-04-02T14:37:54.076 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-04-02T14:37:54.110 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:37:53 smithi092 ceph-mon[36497]: pgmap v93: 97 pgs: 97 active+clean; 1.7 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 8.0 MiB/s wr, 606 op/s 2024-04-02T14:37:54.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.376+0000 7f4037200640 1 -- 172.21.15.90:0/4115800320 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f403010e350 msgr2=0x7f403010e7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:54.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.377+0000 7f4037200640 1 --2- 172.21.15.90:0/4115800320 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f403010e350 0x7f403010e7d0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f401c009900 tx=0x7f401c02bc30 comp rx=0 tx=0).stop 2024-04-02T14:37:54.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.377+0000 7f4037200640 1 -- 172.21.15.90:0/4115800320 shutdown_connections 2024-04-02T14:37:54.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.377+0000 7f4037200640 1 --2- 172.21.15.90:0/4115800320 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f403011b2e0 0x7f403011d6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:54.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.377+0000 7f4037200640 1 --2- 172.21.15.90:0/4115800320 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f403010e350 0x7f403010e7d0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:54.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.377+0000 7f4037200640 1 --2- 172.21.15.90:0/4115800320 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4030110320 0x7f403010de10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:54.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.377+0000 7f4037200640 1 -- 172.21.15.90:0/4115800320 >> 172.21.15.90:0/4115800320 conn(0x7f403006cbe0 msgr2=0x7f403006cff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:37:54.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.378+0000 7f4037200640 1 -- 172.21.15.90:0/4115800320 shutdown_connections 2024-04-02T14:37:54.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.378+0000 7f4037200640 1 -- 172.21.15.90:0/4115800320 wait complete. 2024-04-02T14:37:54.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.380+0000 7f4037200640 1 Processor -- start 2024-04-02T14:37:54.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.380+0000 7f4037200640 1 -- start start 2024-04-02T14:37:54.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.381+0000 7f4037200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4030110320 0x7f40301a73c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:54.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.381+0000 7f4037200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f403011b2e0 0x7f40301a7900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:54.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.381+0000 7f4037200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f40301abc80 0x7f40301ac130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:54.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.381+0000 7f4037200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f4030120c90 con 0x7f40301abc80 2024-04-02T14:37:54.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.381+0000 7f4037200640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f4030120b10 con 0x7f403011b2e0 2024-04-02T14:37:54.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.381+0000 7f4037200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f4030120e10 con 0x7f4030110320 2024-04-02T14:37:54.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.381+0000 7f402f400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f403011b2e0 0x7f40301a7900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:54.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.381+0000 7f402f400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f403011b2e0 0x7f40301a7900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:34108/0 (socket says 172.21.15.90:34108) 2024-04-02T14:37:54.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.381+0000 7f402f400640 1 -- 172.21.15.90:0/3655545999 learned_addr learned my addr 172.21.15.90:0/3655545999 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:37:54.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.382+0000 7f402fe00640 1 --2- 172.21.15.90:0/3655545999 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4030110320 0x7f40301a73c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:54.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.382+0000 7f402f400640 1 -- 172.21.15.90:0/3655545999 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4030110320 msgr2=0x7f40301a73c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:54.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.382+0000 7f402f400640 1 --2- 172.21.15.90:0/3655545999 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4030110320 0x7f40301a73c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:54.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.382+0000 7f402f400640 1 -- 172.21.15.90:0/3655545999 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f40301abc80 msgr2=0x7f40301ac130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:54.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.382+0000 7f402f400640 1 --2- 172.21.15.90:0/3655545999 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f40301abc80 0x7f40301ac130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:54.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.382+0000 7f402f400640 1 -- 172.21.15.90:0/3655545999 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40301ac770 con 0x7f403011b2e0 2024-04-02T14:37:54.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.383+0000 7f402f400640 1 --2- 172.21.15.90:0/3655545999 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f403011b2e0 0x7f40301a7900 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f401c02c240 tx=0x7f401c004430 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:37:54.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.384+0000 7f402cc00640 1 -- 172.21.15.90:0/3655545999 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f401c043020 con 0x7f403011b2e0 2024-04-02T14:37:54.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.384+0000 7f402cc00640 1 -- 172.21.15.90:0/3655545999 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f401c0054e0 con 0x7f403011b2e0 2024-04-02T14:37:54.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.384+0000 7f402cc00640 1 -- 172.21.15.90:0/3655545999 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f401c003bd0 con 0x7f403011b2e0 2024-04-02T14:37:54.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.387+0000 7f4037200640 1 -- 172.21.15.90:0/3655545999 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40301aca00 con 0x7f403011b2e0 2024-04-02T14:37:54.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.387+0000 7f4037200640 1 -- 172.21.15.90:0/3655545999 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40301b4170 con 0x7f403011b2e0 2024-04-02T14:37:54.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.393+0000 7f4039de2740 1 Processor -- start 2024-04-02T14:37:54.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.394+0000 7f4039de2740 1 -- start start 2024-04-02T14:37:54.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.394+0000 7f4039de2740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556368b2a9f0 0x556368b29740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:54.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.394+0000 7f4039de2740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556368b29d10 0x556368b2a190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:54.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.394+0000 7f4039de2740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556368b33080 0x556368b35470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:54.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.394+0000 7f4039de2740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x556368a6a400 con 0x556368b2a9f0 2024-04-02T14:37:54.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.394+0000 7f4039de2740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x556368a4ef10 con 0x556368b29d10 2024-04-02T14:37:54.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.394+0000 7f4039de2740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x556368a8e6f0 con 0x556368b33080 2024-04-02T14:37:54.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.395+0000 7f402cc00640 1 -- 172.21.15.90:0/3655545999 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f401c046020 con 0x7f403011b2e0 2024-04-02T14:37:54.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.396+0000 7f402fe00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556368b29d10 0x556368b2a190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:54.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.396+0000 7f402fe00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556368b29d10 0x556368b2a190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:34114/0 (socket says 172.21.15.90:34114) 2024-04-02T14:37:54.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.396+0000 7f402fe00640 1 -- 172.21.15.90:0/3498951212 learned_addr learned my addr 172.21.15.90:0/3498951212 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:37:54.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.396+0000 7f402f400640 1 --2- 172.21.15.90:0/3498951212 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556368b33080 0x556368b35470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:54.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.397+0000 7f4035200640 1 --2- 172.21.15.90:0/3498951212 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556368b2a9f0 0x556368b29740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:54.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.398+0000 7f402fe00640 1 -- 172.21.15.90:0/3498951212 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556368b33080 msgr2=0x556368b35470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:37:54.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.398+0000 7f402fe00640 1 --2- 172.21.15.90:0/3498951212 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556368b33080 0x556368b35470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:54.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.399+0000 7f402fe00640 1 -- 172.21.15.90:0/3498951212 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556368b2a9f0 msgr2=0x556368b29740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:37:54.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.399+0000 7f402fe00640 1 --2- 172.21.15.90:0/3498951212 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556368b2a9f0 0x556368b29740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:54.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.400+0000 7f402fe00640 1 -- 172.21.15.90:0/3498951212 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5563687a23f0 con 0x556368b29d10 2024-04-02T14:37:54.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.400+0000 7f4035200640 1 --2- 172.21.15.90:0/3498951212 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556368b2a9f0 0x556368b29740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:37:54.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.401+0000 7f402fe00640 1 --2- 172.21.15.90:0/3498951212 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556368b29d10 0x556368b2a190 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f402000cbf0 tx=0x7f402000e720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=28306ef3e6c79b0e server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:37:54.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.402+0000 7f4025600640 1 -- 172.21.15.90:0/3498951212 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4020010630 con 0x556368b29d10 2024-04-02T14:37:54.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.402+0000 7f402cc00640 1 --2- 172.21.15.90:0/3655545999 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f40140a3da0 0x7f40140a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:54.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.402+0000 7f4025600640 1 -- 172.21.15.90:0/3498951212 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4020010cb0 con 0x556368b29d10 2024-04-02T14:37:54.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.402+0000 7f4025600640 1 -- 172.21.15.90:0/3498951212 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4020019c90 con 0x556368b29d10 2024-04-02T14:37:54.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.403+0000 7f402cc00640 1 -- 172.21.15.90:0/3655545999 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f401c0f20f0 con 0x7f403011b2e0 2024-04-02T14:37:54.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.403+0000 7f4035200640 1 --2- 172.21.15.90:0/3655545999 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f40140a3da0 0x7f40140a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:54.406 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.404+0000 7f4039de2740 1 -- 172.21.15.90:0/3498951212 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556368b29d10 msgr2=0x556368b2a190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:54.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.404+0000 7f4039de2740 1 --2- 172.21.15.90:0/3498951212 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556368b29d10 0x556368b2a190 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f402000cbf0 tx=0x7f402000e720 comp rx=0 tx=0).stop 2024-04-02T14:37:54.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.405+0000 7f4035200640 1 --2- 172.21.15.90:0/3655545999 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f40140a3da0 0x7f40140a6260 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f4028005c40 tx=0x7f4028009400 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:37:54.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.405+0000 7f4039de2740 1 -- 172.21.15.90:0/3498951212 shutdown_connections 2024-04-02T14:37:54.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.405+0000 7f4039de2740 1 --2- 172.21.15.90:0/3498951212 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556368b33080 0x556368b35470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:54.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.406+0000 7f4039de2740 1 --2- 172.21.15.90:0/3498951212 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556368b29d10 0x556368b2a190 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:54.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.406+0000 7f4039de2740 1 --2- 172.21.15.90:0/3498951212 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556368b2a9f0 0x556368b29740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:54.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.406+0000 7f4039de2740 1 -- 172.21.15.90:0/3498951212 >> 172.21.15.90:0/3498951212 conn(0x556368af0a50 msgr2=0x556368b24760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:37:54.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.406+0000 7f4039de2740 1 -- 172.21.15.90:0/3498951212 shutdown_connections 2024-04-02T14:37:54.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.406+0000 7f4039de2740 1 -- 172.21.15.90:0/3498951212 wait complete. 2024-04-02T14:37:54.410 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.408+0000 7f4039de2740 1 -- start start 2024-04-02T14:37:54.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.409+0000 7f4039de2740 1 Processor -- start 2024-04-02T14:37:54.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.410+0000 7f4039de2740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556368b29d10 0x556368b30040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:54.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.411+0000 7f4039de2740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556368b2a9f0 0x556368b30580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:54.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.411+0000 7f4039de2740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556368b33080 0x556368b30ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:54.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.412+0000 7f402f400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556368b2a9f0 0x556368b30580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:54.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.412+0000 7f402fe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556368b29d10 0x556368b30040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:54.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.412+0000 7f402f400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556368b2a9f0 0x556368b30580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:34126/0 (socket says 172.21.15.90:34126) 2024-04-02T14:37:54.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.412+0000 7f4035200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556368b33080 0x556368b30ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:54.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.412+0000 7f4035200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556368b33080 0x556368b30ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:40246/0 (socket says 172.21.15.90:40246) 2024-04-02T14:37:54.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.412+0000 7f402fe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556368b29d10 0x556368b30040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:54280/0 (socket says 172.21.15.90:54280) 2024-04-02T14:37:54.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.412+0000 7f4039de2740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x556368a8a6a0 con 0x556368b29d10 2024-04-02T14:37:54.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.412+0000 7f4039de2740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x556368a36980 con 0x556368b2a9f0 2024-04-02T14:37:54.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.412+0000 7f4039de2740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x556368a49cb0 con 0x556368b33080 2024-04-02T14:37:54.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.412+0000 7f4039de2740 10 client.0 _ll_register_callbacks cb 0x556368a496a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:37:54.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.413+0000 7f402f400640 1 -- 172.21.15.90:0/3995603382 learned_addr learned my addr 172.21.15.90:0/3995603382 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:37:54.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.414+0000 7f4035200640 1 -- 172.21.15.90:0/3995603382 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556368b2a9f0 msgr2=0x556368b30580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:37:54.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.414+0000 7f4035200640 1 --2- 172.21.15.90:0/3995603382 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556368b2a9f0 0x556368b30580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:54.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.414+0000 7f4035200640 1 -- 172.21.15.90:0/3995603382 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556368b29d10 msgr2=0x556368b30040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:37:54.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.416+0000 7f4035200640 1 --2- 172.21.15.90:0/3995603382 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556368b29d10 0x556368b30040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:54.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.416+0000 7f4035200640 1 -- 172.21.15.90:0/3995603382 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55636886c7d0 con 0x556368b33080 2024-04-02T14:37:54.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.416+0000 7f402fe00640 1 --2- 172.21.15.90:0/3995603382 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556368b29d10 0x556368b30040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:37:54.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.418+0000 7f4035200640 1 --2- 172.21.15.90:0/3995603382 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556368b33080 0x556368b30ac0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f40280116f0 tx=0x7f4028005bd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=faff56763711755a server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:37:54.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.418+0000 7f401a000640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:37:54.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.419+0000 7f401a000640 1 -- 172.21.15.90:0/3995603382 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4028022070 con 0x556368b33080 2024-04-02T14:37:54.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.419+0000 7f401a000640 1 -- 172.21.15.90:0/3995603382 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f4028004d20 con 0x556368b33080 2024-04-02T14:37:54.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.420+0000 7f4039de2740 1 -- 172.21.15.90:0/3995603382 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4020010cb0 con 0x556368b33080 2024-04-02T14:37:54.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.421+0000 7f401a000640 1 -- 172.21.15.90:0/3995603382 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f402801e660 con 0x556368b33080 2024-04-02T14:37:54.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.421+0000 7f4039de2740 10 client.26200 fetch_fsmap learned FSMap version 27 2024-04-02T14:37:54.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.421+0000 7f4039de2740 1 -- 172.21.15.90:0/3995603382 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x556368b41ef0 con 0x556368b33080 2024-04-02T14:37:54.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.423+0000 7f401a000640 1 -- 172.21.15.90:0/3995603382 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4020010cb0 con 0x556368b33080 2024-04-02T14:37:54.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.424+0000 7f401a000640 1 -- 172.21.15.90:0/3995603382 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f4028010ac0 con 0x556368b33080 2024-04-02T14:37:54.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.424+0000 7f4039de2740 10 client.26200 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:37:54.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.424+0000 7f4039de2740 10 client.26200 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.c 2024-04-02T14:37:54.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.424+0000 7f4039de2740 20 client.26200 populate_metadata read hostname 'smithi090' 2024-04-02T14:37:54.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.424+0000 7f4039de2740 1 --2- 172.21.15.90:0/3995603382 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x556368b43450 0x556368b45840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:54.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.424+0000 7f4039de2740 4 client.26200 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:37:54.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.424+0000 7f4039de2740 1 -- 172.21.15.90:0/3995603382 --> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556368a4e040 con 0x556368b43450 2024-04-02T14:37:54.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.425+0000 7f402fe00640 1 --2- 172.21.15.90:0/3995603382 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x556368b43450 0x556368b45840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:54.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.426+0000 7f402fe00640 1 --2- 172.21.15.90:0/3995603382 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x556368b43450 0x556368b45840 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=d2de109d96b4ced0 server_cookie=eebe763f799473e2 in_seq=0 out_seq=0 2024-04-02T14:37:54.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.426+0000 7f401a000640 10 client.26200 ms_handle_connect on v2:172.21.15.177:6836/4223414858 2024-04-02T14:37:54.429 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.427+0000 7f401a000640 1 -- 172.21.15.90:0/3995603382 <== mds.4 v2:172.21.15.177:6836/4223414858 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x556368a4e040 con 0x556368b43450 2024-04-02T14:37:54.429 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.427+0000 7f401a000640 10 client.26200 handle_command_reply: tid=0 2024-04-02T14:37:54.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.428+0000 7f4039de2740 1 client.26200 shutdown 2024-04-02T14:37:54.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.429+0000 7f4039de2740 1 -- 172.21.15.90:0/3995603382 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556368b33080 msgr2=0x556368b30ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:54.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.429+0000 7f4039de2740 1 --2- 172.21.15.90:0/3995603382 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556368b33080 0x556368b30ac0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f40280116f0 tx=0x7f4028005bd0 comp rx=0 tx=0).stop 2024-04-02T14:37:54.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.429+0000 7f4039de2740 1 -- 172.21.15.90:0/3995603382 shutdown_connections 2024-04-02T14:37:54.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.429+0000 7f4039de2740 1 --2- 172.21.15.90:0/3995603382 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556368b33080 0x556368b30ac0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:54.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.429+0000 7f4039de2740 1 --2- 172.21.15.90:0/3995603382 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556368b2a9f0 0x556368b30580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:54.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.429+0000 7f4039de2740 1 --2- 172.21.15.90:0/3995603382 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x556368b43450 0x556368b45840 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:54.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.429+0000 7f4039de2740 1 --2- 172.21.15.90:0/3995603382 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556368b29d10 0x556368b30040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:54.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.429+0000 7f401a000640 0 client.26200 ms_handle_reset on v2:172.21.15.177:6836/4223414858 2024-04-02T14:37:54.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.429+0000 7f4039de2740 1 -- 172.21.15.90:0/3995603382 >> 172.21.15.90:0/3995603382 conn(0x556368af0a50 msgr2=0x556368b264b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:37:54.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.429+0000 7f4039de2740 1 -- 172.21.15.90:0/3995603382 shutdown_connections 2024-04-02T14:37:54.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.430+0000 7f4039de2740 1 -- 172.21.15.90:0/3995603382 wait complete. 2024-04-02T14:37:54.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.430+0000 7f4039de2740 20 client.26200 trim_cache size 0 max 16384 2024-04-02T14:37:54.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.465+0000 7f4039de2740 1 Processor -- start 2024-04-02T14:37:54.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.466+0000 7f4039de2740 1 -- start start 2024-04-02T14:37:54.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.466+0000 7f4039de2740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556368b29d10 0x556368b32770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:54.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.466+0000 7f4039de2740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556368b2a9f0 0x556368b3eb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:54.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.467+0000 7f4039de2740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556368b33080 0x556368b2f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:54.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.467+0000 7f4039de2740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f40301bcc20 con 0x556368b2a9f0 2024-04-02T14:37:54.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.467+0000 7f4039de2740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f40301be760 con 0x556368b33080 2024-04-02T14:37:54.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.467+0000 7f4039de2740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f40301bd2f0 con 0x556368b29d10 2024-04-02T14:37:54.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.467+0000 7f4035200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556368b33080 0x556368b2f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:54.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.467+0000 7f402fe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556368b29d10 0x556368b32770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:54.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.467+0000 7f402fe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556368b29d10 0x556368b32770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:40256/0 (socket says 172.21.15.90:40256) 2024-04-02T14:37:54.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.467+0000 7f402fe00640 1 -- 172.21.15.90:0/3859571726 learned_addr learned my addr 172.21.15.90:0/3859571726 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:37:54.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.468+0000 7f402f400640 1 --2- 172.21.15.90:0/3859571726 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556368b2a9f0 0x556368b3eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:54.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.468+0000 7f4035200640 1 -- 172.21.15.90:0/3859571726 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556368b29d10 msgr2=0x556368b32770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:37:54.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.468+0000 7f4035200640 1 --2- 172.21.15.90:0/3859571726 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556368b29d10 0x556368b32770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:54.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.468+0000 7f4035200640 1 -- 172.21.15.90:0/3859571726 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556368b2a9f0 msgr2=0x556368b3eb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:37:54.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.468+0000 7f4035200640 1 --2- 172.21.15.90:0/3859571726 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556368b2a9f0 0x556368b3eb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:54.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.468+0000 7f4035200640 1 -- 172.21.15.90:0/3859571726 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556368b2e590 con 0x556368b33080 2024-04-02T14:37:54.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.468+0000 7f402f400640 1 --2- 172.21.15.90:0/3859571726 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556368b2a9f0 0x556368b3eb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:37:54.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.469+0000 7f4035200640 1 --2- 172.21.15.90:0/3859571726 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556368b33080 0x556368b2f920 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f4028005010 tx=0x7f4028011ec0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=72b2d98a2d063dc9 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:37:54.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.470+0000 7f4024c00640 1 -- 172.21.15.90:0/3859571726 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4028035070 con 0x556368b33080 2024-04-02T14:37:54.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.470+0000 7f4024c00640 1 -- 172.21.15.90:0/3859571726 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f40280059f0 con 0x556368b33080 2024-04-02T14:37:54.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.470+0000 7f4024c00640 1 -- 172.21.15.90:0/3859571726 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4028022040 con 0x556368b33080 2024-04-02T14:37:54.473 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.471+0000 7f4039de2740 1 -- 172.21.15.90:0/3859571726 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556368b33080 msgr2=0x556368b2f920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:54.473 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.471+0000 7f4039de2740 1 --2- 172.21.15.90:0/3859571726 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556368b33080 0x556368b2f920 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f4028005010 tx=0x7f4028011ec0 comp rx=0 tx=0).stop 2024-04-02T14:37:54.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.472+0000 7f4039de2740 1 -- 172.21.15.90:0/3859571726 shutdown_connections 2024-04-02T14:37:54.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.472+0000 7f4039de2740 1 --2- 172.21.15.90:0/3859571726 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556368b33080 0x556368b2f920 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:54.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.472+0000 7f4039de2740 1 --2- 172.21.15.90:0/3859571726 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556368b2a9f0 0x556368b3eb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:54.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.472+0000 7f4039de2740 1 --2- 172.21.15.90:0/3859571726 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556368b29d10 0x556368b32770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:54.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.472+0000 7f4039de2740 1 -- 172.21.15.90:0/3859571726 >> 172.21.15.90:0/3859571726 conn(0x556368af0a50 msgr2=0x556368b26440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:37:54.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.472+0000 7f4039de2740 1 -- 172.21.15.90:0/3859571726 shutdown_connections 2024-04-02T14:37:54.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.472+0000 7f4039de2740 1 -- 172.21.15.90:0/3859571726 wait complete. 2024-04-02T14:37:54.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.473+0000 7f4039de2740 1 -- start start 2024-04-02T14:37:54.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.474+0000 7f4039de2740 1 Processor -- start 2024-04-02T14:37:54.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.475+0000 7f4039de2740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556368b29d10 0x556368b3b570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:54.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.475+0000 7f4039de2740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556368b2a9f0 0x556368b2cb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:54.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.475+0000 7f4039de2740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556368b33080 0x556368b2d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:54.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.475+0000 7f4039de2740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f40301be090 con 0x556368b2a9f0 2024-04-02T14:37:54.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.475+0000 7f4039de2740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f40301bd9c0 con 0x556368b29d10 2024-04-02T14:37:54.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.475+0000 7f4039de2740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f40301bac00 con 0x556368b33080 2024-04-02T14:37:54.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.475+0000 7f4039de2740 10 client.0 _ll_register_callbacks cb 0x7f4028009d70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:37:54.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.476+0000 7f402fe00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556368b29d10 0x556368b3b570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:54.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.476+0000 7f402fe00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556368b29d10 0x556368b3b570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:34156/0 (socket says 172.21.15.90:34156) 2024-04-02T14:37:54.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.476+0000 7f402fe00640 1 -- 172.21.15.90:0/1152445418 learned_addr learned my addr 172.21.15.90:0/1152445418 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:37:54.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.476+0000 7f4035200640 1 --2- 172.21.15.90:0/1152445418 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556368b33080 0x556368b2d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:54.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.476+0000 7f4035200640 1 -- 172.21.15.90:0/1152445418 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556368b29d10 msgr2=0x556368b3b570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:37:54.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.476+0000 7f4035200640 1 --2- 172.21.15.90:0/1152445418 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556368b29d10 0x556368b3b570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:54.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.476+0000 7f4035200640 1 -- 172.21.15.90:0/1152445418 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556368b2a9f0 msgr2=0x556368b2cb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:37:54.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.476+0000 7f4035200640 1 --2- 172.21.15.90:0/1152445418 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556368b2a9f0 0x556368b2cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:54.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.476+0000 7f4035200640 1 -- 172.21.15.90:0/1152445418 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556368ba8980 con 0x556368b33080 2024-04-02T14:37:54.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.477+0000 7f4035200640 1 --2- 172.21.15.90:0/1152445418 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556368b33080 0x556368b2d2d0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f4028004f90 tx=0x7f4028005110 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3963dec731df3a56 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:37:54.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.477+0000 7f401a000640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:37:54.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.477+0000 7f401a000640 1 -- 172.21.15.90:0/1152445418 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4028040020 con 0x556368b33080 2024-04-02T14:37:54.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.478+0000 7f401a000640 1 -- 172.21.15.90:0/1152445418 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f402801e800 con 0x556368b33080 2024-04-02T14:37:54.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.478+0000 7f4039de2740 1 -- 172.21.15.90:0/1152445418 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f40280059f0 con 0x556368b33080 2024-04-02T14:37:54.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.479+0000 7f401a000640 1 -- 172.21.15.90:0/1152445418 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f4028022040 con 0x556368b33080 2024-04-02T14:37:54.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.479+0000 7f4039de2740 10 client.26206 fetch_fsmap learned FSMap version 27 2024-04-02T14:37:54.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.479+0000 7f4039de2740 1 -- 172.21.15.90:0/1152445418 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x556368ba9a90 con 0x556368b33080 2024-04-02T14:37:54.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.480+0000 7f401a000640 1 -- 172.21.15.90:0/1152445418 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f40280059f0 con 0x556368b33080 2024-04-02T14:37:54.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.480+0000 7f401a000640 1 -- 172.21.15.90:0/1152445418 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f4028010bb0 con 0x556368b33080 2024-04-02T14:37:54.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.480+0000 7f4039de2740 10 client.26206 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:37:54.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.480+0000 7f4039de2740 10 client.26206 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.c 2024-04-02T14:37:54.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.480+0000 7f4039de2740 20 client.26206 populate_metadata read hostname 'smithi090' 2024-04-02T14:37:54.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.481+0000 7f4039de2740 1 --2- 172.21.15.90:0/1152445418 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x556368baa0f0 0x556368bac4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:54.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.481+0000 7f4039de2740 4 client.26206 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-02T14:37:54.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.481+0000 7f4039de2740 1 -- 172.21.15.90:0/1152445418 --> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] -- command(tid 0: {"prefix": "damage ls"}) -- 0x556368a9bc30 con 0x556368baa0f0 2024-04-02T14:37:54.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.481+0000 7f402fe00640 1 --2- 172.21.15.90:0/1152445418 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x556368baa0f0 0x556368bac4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:54.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.482+0000 7f402fe00640 1 --2- 172.21.15.90:0/1152445418 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x556368baa0f0 0x556368bac4e0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=f1de3f84a4c2f408 server_cookie=19aa4c0f10260550 in_seq=0 out_seq=0 2024-04-02T14:37:54.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.482+0000 7f401a000640 10 client.26206 ms_handle_connect on v2:172.21.15.177:6836/4223414858 2024-04-02T14:37:54.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.483+0000 7f401a000640 1 -- 172.21.15.90:0/1152445418 <== mds.4 v2:172.21.15.177:6836/4223414858 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x556368a9bc30 con 0x556368baa0f0 2024-04-02T14:37:54.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.483+0000 7f401a000640 10 client.26206 handle_command_reply: tid=0 2024-04-02T14:37:54.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.483+0000 7f4039de2740 1 client.26206 shutdown 2024-04-02T14:37:54.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.483+0000 7f4039de2740 1 -- 172.21.15.90:0/1152445418 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556368b33080 msgr2=0x556368b2d2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:54.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.483+0000 7f4039de2740 1 --2- 172.21.15.90:0/1152445418 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556368b33080 0x556368b2d2d0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f4028004f90 tx=0x7f4028005110 comp rx=0 tx=0).stop 2024-04-02T14:37:54.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.484+0000 7f4039de2740 1 -- 172.21.15.90:0/1152445418 shutdown_connections 2024-04-02T14:37:54.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.484+0000 7f4039de2740 1 --2- 172.21.15.90:0/1152445418 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556368b33080 0x556368b2d2d0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:54.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.484+0000 7f4039de2740 1 --2- 172.21.15.90:0/1152445418 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x556368baa0f0 0x556368bac4e0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:54.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.484+0000 7f4039de2740 1 --2- 172.21.15.90:0/1152445418 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556368b2a9f0 0x556368b2cb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:54.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.484+0000 7f4039de2740 1 --2- 172.21.15.90:0/1152445418 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556368b29d10 0x556368b3b570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:54.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.484+0000 7f4039de2740 1 -- 172.21.15.90:0/1152445418 >> 172.21.15.90:0/1152445418 conn(0x556368af0a50 msgr2=0x556368b30000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:37:54.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.484+0000 7f401a000640 0 client.26206 ms_handle_reset on v2:172.21.15.177:6836/4223414858 2024-04-02T14:37:54.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.484+0000 7f4039de2740 1 -- 172.21.15.90:0/1152445418 shutdown_connections 2024-04-02T14:37:54.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.484+0000 7f4039de2740 1 -- 172.21.15.90:0/1152445418 wait complete. 2024-04-02T14:37:54.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.485+0000 7f4039de2740 20 client.26206 trim_cache size 0 max 16384 2024-04-02T14:37:54.488 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-02T14:37:54.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.487+0000 7f4037200640 1 -- 172.21.15.90:0/3655545999 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f40140a3da0 msgr2=0x7f40140a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:54.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.487+0000 7f4037200640 1 --2- 172.21.15.90:0/3655545999 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f40140a3da0 0x7f40140a6260 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f4028005c40 tx=0x7f4028009400 comp rx=0 tx=0).stop 2024-04-02T14:37:54.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.487+0000 7f4037200640 1 -- 172.21.15.90:0/3655545999 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f403011b2e0 msgr2=0x7f40301a7900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:54.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.487+0000 7f4037200640 1 --2- 172.21.15.90:0/3655545999 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f403011b2e0 0x7f40301a7900 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f401c02c240 tx=0x7f401c004430 comp rx=0 tx=0).stop 2024-04-02T14:37:54.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.487+0000 7f4037200640 1 -- 172.21.15.90:0/3655545999 shutdown_connections 2024-04-02T14:37:54.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.487+0000 7f4037200640 1 --2- 172.21.15.90:0/3655545999 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f40140a3da0 0x7f40140a6260 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:54.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.487+0000 7f4037200640 1 --2- 172.21.15.90:0/3655545999 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f40301abc80 0x7f40301ac130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:54.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.487+0000 7f4037200640 1 --2- 172.21.15.90:0/3655545999 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f403011b2e0 0x7f40301a7900 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:54.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.487+0000 7f4037200640 1 --2- 172.21.15.90:0/3655545999 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4030110320 0x7f40301a73c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:54.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.487+0000 7f4037200640 1 -- 172.21.15.90:0/3655545999 >> 172.21.15.90:0/3655545999 conn(0x7f403006cbe0 msgr2=0x7f40300725d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:37:54.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.488+0000 7f4037200640 1 -- 172.21.15.90:0/3655545999 shutdown_connections 2024-04-02T14:37:54.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.488+0000 7f4037200640 1 -- 172.21.15.90:0/3655545999 wait complete. 2024-04-02T14:37:54.522 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-04-02T14:37:54.822 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.820+0000 7f0c8dc00640 1 -- 172.21.15.90:0/1542825001 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0c88107aa0 msgr2=0x7f0c88107e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:54.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.820+0000 7f0c8dc00640 1 --2- 172.21.15.90:0/1542825001 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0c88107aa0 0x7f0c88107e80 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f0c74009900 tx=0x7f0c7402bc40 comp rx=0 tx=0).stop 2024-04-02T14:37:54.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.820+0000 7f0c8dc00640 1 -- 172.21.15.90:0/1542825001 shutdown_connections 2024-04-02T14:37:54.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.820+0000 7f0c8dc00640 1 --2- 172.21.15.90:0/1542825001 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0c88102450 0x7f0c8810e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:54.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.820+0000 7f0c8dc00640 1 --2- 172.21.15.90:0/1542825001 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0c88101a90 0x7f0c88101f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:54.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.820+0000 7f0c8dc00640 1 --2- 172.21.15.90:0/1542825001 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0c88107aa0 0x7f0c88107e80 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:54.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.820+0000 7f0c8dc00640 1 -- 172.21.15.90:0/1542825001 >> 172.21.15.90:0/1542825001 conn(0x7f0c88078f00 msgr2=0x7f0c88079310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:37:54.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.820+0000 7f0c8dc00640 1 -- 172.21.15.90:0/1542825001 shutdown_connections 2024-04-02T14:37:54.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.821+0000 7f0c8dc00640 1 -- 172.21.15.90:0/1542825001 wait complete. 2024-04-02T14:37:54.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.822+0000 7f0c8dc00640 1 Processor -- start 2024-04-02T14:37:54.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.823+0000 7f0c8dc00640 1 -- start start 2024-04-02T14:37:54.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.823+0000 7f0c8dc00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0c88101a90 0x7f0c881089c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:54.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.824+0000 7f0c8dc00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0c88102450 0x7f0c88108f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:54.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.824+0000 7f0c8dc00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0c88107aa0 0x7f0c88109440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:54.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.824+0000 7f0c8dc00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f0c88111ed0 con 0x7f0c88107aa0 2024-04-02T14:37:54.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.824+0000 7f0c8dc00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f0c88111d50 con 0x7f0c88102450 2024-04-02T14:37:54.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.824+0000 7f0c8dc00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f0c88112050 con 0x7f0c88101a90 2024-04-02T14:37:54.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.824+0000 7f0c86000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0c88102450 0x7f0c88108f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:54.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.824+0000 7f0c86000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0c88102450 0x7f0c88108f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:34174/0 (socket says 172.21.15.90:34174) 2024-04-02T14:37:54.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.824+0000 7f0c86000640 1 -- 172.21.15.90:0/288741834 learned_addr learned my addr 172.21.15.90:0/288741834 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:37:54.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.824+0000 7f0c86a00640 1 --2- 172.21.15.90:0/288741834 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0c88101a90 0x7f0c881089c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:54.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.824+0000 7f0c87400640 1 --2- 172.21.15.90:0/288741834 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0c88107aa0 0x7f0c88109440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:54.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.825+0000 7f0c86a00640 1 -- 172.21.15.90:0/288741834 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0c88102450 msgr2=0x7f0c88108f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:54.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.825+0000 7f0c86a00640 1 --2- 172.21.15.90:0/288741834 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0c88102450 0x7f0c88108f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:54.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.825+0000 7f0c86a00640 1 -- 172.21.15.90:0/288741834 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0c88107aa0 msgr2=0x7f0c88109440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:54.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.825+0000 7f0c86a00640 1 --2- 172.21.15.90:0/288741834 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0c88107aa0 0x7f0c88109440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:54.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.825+0000 7f0c86a00640 1 -- 172.21.15.90:0/288741834 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c881aac00 con 0x7f0c88101a90 2024-04-02T14:37:54.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.825+0000 7f0c87400640 1 --2- 172.21.15.90:0/288741834 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0c88107aa0 0x7f0c88109440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:37:54.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.825+0000 7f0c86000640 1 --2- 172.21.15.90:0/288741834 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0c88102450 0x7f0c88108f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:37:54.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.825+0000 7f0c86a00640 1 --2- 172.21.15.90:0/288741834 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0c88101a90 0x7f0c881089c0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f0c7402c180 tx=0x7f0c740043b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:37:54.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.826+0000 7f0c7f400640 1 -- 172.21.15.90:0/288741834 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c74041070 con 0x7f0c88101a90 2024-04-02T14:37:54.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.826+0000 7f0c7f400640 1 -- 172.21.15.90:0/288741834 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0c7402ccd0 con 0x7f0c88101a90 2024-04-02T14:37:54.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.826+0000 7f0c8dc00640 1 -- 172.21.15.90:0/288741834 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0c881aae90 con 0x7f0c88101a90 2024-04-02T14:37:54.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.826+0000 7f0c7f400640 1 -- 172.21.15.90:0/288741834 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c74003d10 con 0x7f0c88101a90 2024-04-02T14:37:54.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.826+0000 7f0c8dc00640 1 -- 172.21.15.90:0/288741834 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0c881ab350 con 0x7f0c88101a90 2024-04-02T14:37:54.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.833+0000 7f0c907ff740 1 Processor -- start 2024-04-02T14:37:54.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.833+0000 7f0c907ff740 1 -- start start 2024-04-02T14:37:54.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.833+0000 7f0c907ff740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d8cfcc69f0 0x55d8cfcc5740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:54.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.833+0000 7f0c907ff740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d8cfcc5d10 0x55d8cfcc6190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:54.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.833+0000 7f0c907ff740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d8cfccf080 0x55d8cfcd1470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:54.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.834+0000 7f0c907ff740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55d8cfc06400 con 0x55d8cfcc69f0 2024-04-02T14:37:54.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.834+0000 7f0c907ff740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55d8cfbeaf10 con 0x55d8cfccf080 2024-04-02T14:37:54.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.834+0000 7f0c907ff740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55d8cfc2a6f0 con 0x55d8cfcc5d10 2024-04-02T14:37:54.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.834+0000 7f0c87400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d8cfcc69f0 0x55d8cfcc5740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:54.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.834+0000 7f0c86000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d8cfccf080 0x55d8cfcd1470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:54.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.834+0000 7f0c86000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d8cfccf080 0x55d8cfcd1470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:34184/0 (socket says 172.21.15.90:34184) 2024-04-02T14:37:54.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.834+0000 7f0c86000640 1 -- 172.21.15.90:0/2218103851 learned_addr learned my addr 172.21.15.90:0/2218103851 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:37:54.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.834+0000 7f0c7f400640 1 -- 172.21.15.90:0/288741834 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f0c74044070 con 0x7f0c88101a90 2024-04-02T14:37:54.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.835+0000 7f0c86a00640 1 --2- 172.21.15.90:0/2218103851 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d8cfcc5d10 0x55d8cfcc6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:54.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.835+0000 7f0c86000640 1 -- 172.21.15.90:0/2218103851 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d8cfcc5d10 msgr2=0x55d8cfcc6190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:37:54.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.835+0000 7f0c86000640 1 --2- 172.21.15.90:0/2218103851 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d8cfcc5d10 0x55d8cfcc6190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:54.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.835+0000 7f0c86000640 1 -- 172.21.15.90:0/2218103851 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d8cfcc69f0 msgr2=0x55d8cfcc5740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:37:54.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.835+0000 7f0c86000640 1 --2- 172.21.15.90:0/2218103851 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d8cfcc69f0 0x55d8cfcc5740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:54.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.835+0000 7f0c86000640 1 -- 172.21.15.90:0/2218103851 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d8cf93e3f0 con 0x55d8cfccf080 2024-04-02T14:37:54.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.836+0000 7f0c86000640 1 --2- 172.21.15.90:0/2218103851 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d8cfccf080 0x55d8cfcd1470 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f0c7800f470 tx=0x7f0c7800f930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2cb0397f185d4cd9 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:37:54.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.837+0000 7f0c73e00640 1 -- 172.21.15.90:0/2218103851 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c7801d070 con 0x55d8cfccf080 2024-04-02T14:37:54.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.837+0000 7f0c73e00640 1 -- 172.21.15.90:0/2218103851 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0c78010e40 con 0x55d8cfccf080 2024-04-02T14:37:54.840 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.837+0000 7f0c73e00640 1 -- 172.21.15.90:0/2218103851 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c780165a0 con 0x55d8cfccf080 2024-04-02T14:37:54.840 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.838+0000 7f0c7f400640 1 --2- 172.21.15.90:0/288741834 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f0c6c0a3e20 0x7f0c6c0a62e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:54.840 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.838+0000 7f0c7f400640 1 -- 172.21.15.90:0/288741834 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f0c740f2180 con 0x7f0c88101a90 2024-04-02T14:37:54.841 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.838+0000 7f0c87400640 1 --2- 172.21.15.90:0/288741834 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f0c6c0a3e20 0x7f0c6c0a62e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:54.841 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.838+0000 7f0c907ff740 1 -- 172.21.15.90:0/2218103851 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d8cfccf080 msgr2=0x55d8cfcd1470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:54.841 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.838+0000 7f0c907ff740 1 --2- 172.21.15.90:0/2218103851 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d8cfccf080 0x55d8cfcd1470 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f0c7800f470 tx=0x7f0c7800f930 comp rx=0 tx=0).stop 2024-04-02T14:37:54.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.841+0000 7f0c87400640 1 --2- 172.21.15.90:0/288741834 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f0c6c0a3e20 0x7f0c6c0a62e0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f0c80005fd0 tx=0x7f0c800098d0 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:37:54.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.841+0000 7f0c907ff740 1 -- 172.21.15.90:0/2218103851 shutdown_connections 2024-04-02T14:37:54.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.841+0000 7f0c907ff740 1 --2- 172.21.15.90:0/2218103851 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d8cfccf080 0x55d8cfcd1470 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:54.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.841+0000 7f0c907ff740 1 --2- 172.21.15.90:0/2218103851 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d8cfcc5d10 0x55d8cfcc6190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:54.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.841+0000 7f0c907ff740 1 --2- 172.21.15.90:0/2218103851 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d8cfcc69f0 0x55d8cfcc5740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:54.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.841+0000 7f0c907ff740 1 -- 172.21.15.90:0/2218103851 >> 172.21.15.90:0/2218103851 conn(0x55d8cfc8ca50 msgr2=0x55d8cfcc0760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:37:54.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.842+0000 7f0c907ff740 1 -- 172.21.15.90:0/2218103851 shutdown_connections 2024-04-02T14:37:54.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.842+0000 7f0c907ff740 1 -- 172.21.15.90:0/2218103851 wait complete. 2024-04-02T14:37:54.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.842+0000 7f0c907ff740 1 -- start start 2024-04-02T14:37:54.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.843+0000 7f0c907ff740 1 Processor -- start 2024-04-02T14:37:54.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.843+0000 7f0c907ff740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d8cfcc5d10 0x55d8cfccc030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:54.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.843+0000 7f0c907ff740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d8cfcc69f0 0x55d8cfccc570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:54.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.843+0000 7f0c907ff740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d8cfccf080 0x55d8cfcccab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:54.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.843+0000 7f0c907ff740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55d8cfc266a0 con 0x55d8cfccf080 2024-04-02T14:37:54.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.843+0000 7f0c907ff740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55d8cfbd2980 con 0x55d8cfcc5d10 2024-04-02T14:37:54.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.843+0000 7f0c907ff740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55d8cfbe5cb0 con 0x55d8cfcc69f0 2024-04-02T14:37:54.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.843+0000 7f0c907ff740 10 client.0 _ll_register_callbacks cb 0x55d8cfbe56a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:37:54.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.845+0000 7f0c86a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d8cfcc5d10 0x55d8cfccc030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:54.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.845+0000 7f0c86a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d8cfcc5d10 0x55d8cfccc030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:34194/0 (socket says 172.21.15.90:34194) 2024-04-02T14:37:54.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.845+0000 7f0c86a00640 1 -- 172.21.15.90:0/3570876922 learned_addr learned my addr 172.21.15.90:0/3570876922 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:37:54.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.845+0000 7f0c87400640 1 --2- 172.21.15.90:0/3570876922 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d8cfccf080 0x55d8cfcccab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:54.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.845+0000 7f0c86a00640 1 -- 172.21.15.90:0/3570876922 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d8cfcc69f0 msgr2=0x55d8cfccc570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:37:54.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.845+0000 7f0c86000640 1 --2- 172.21.15.90:0/3570876922 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d8cfcc69f0 0x55d8cfccc570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:54.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.845+0000 7f0c86a00640 1 --2- 172.21.15.90:0/3570876922 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d8cfcc69f0 0x55d8cfccc570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:54.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.845+0000 7f0c86a00640 1 -- 172.21.15.90:0/3570876922 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d8cfccf080 msgr2=0x55d8cfcccab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:37:54.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.845+0000 7f0c86a00640 1 --2- 172.21.15.90:0/3570876922 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d8cfccf080 0x55d8cfcccab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:54.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.845+0000 7f0c86a00640 1 -- 172.21.15.90:0/3570876922 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d8cfa087d0 con 0x55d8cfcc5d10 2024-04-02T14:37:54.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.845+0000 7f0c86000640 1 --2- 172.21.15.90:0/3570876922 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d8cfcc69f0 0x55d8cfccc570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:37:54.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.845+0000 7f0c86a00640 1 --2- 172.21.15.90:0/3570876922 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d8cfcc5d10 0x55d8cfccc030 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f0c740be170 tx=0x7f0c74003eb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=155564f35d58206c server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:37:54.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.846+0000 7f0c70c00640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:37:54.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.846+0000 7f0c70c00640 1 -- 172.21.15.90:0/3570876922 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c7404b040 con 0x55d8cfcc5d10 2024-04-02T14:37:54.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.846+0000 7f0c70c00640 1 -- 172.21.15.90:0/3570876922 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f0c740ba850 con 0x55d8cfcc5d10 2024-04-02T14:37:54.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.848+0000 7f0c907ff740 1 -- 172.21.15.90:0/3570876922 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55d8cfbea040 con 0x55d8cfcc5d10 2024-04-02T14:37:54.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.848+0000 7f0c70c00640 1 -- 172.21.15.90:0/3570876922 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f0c740ba480 con 0x55d8cfcc5d10 2024-04-02T14:37:54.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.848+0000 7f0c907ff740 10 client.26312 fetch_fsmap learned FSMap version 27 2024-04-02T14:37:54.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.848+0000 7f0c907ff740 1 -- 172.21.15.90:0/3570876922 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55d8cfceef90 con 0x55d8cfcc5d10 2024-04-02T14:37:54.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.849+0000 7f0c70c00640 1 -- 172.21.15.90:0/3570876922 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55d8cfbea040 con 0x55d8cfcc5d10 2024-04-02T14:37:54.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.849+0000 7f0c70c00640 1 -- 172.21.15.90:0/3570876922 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f0c740c2150 con 0x55d8cfcc5d10 2024-04-02T14:37:54.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.849+0000 7f0c907ff740 10 client.26312 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:37:54.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.849+0000 7f0c907ff740 10 client.26312 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-04-02T14:37:54.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.849+0000 7f0c907ff740 20 client.26312 populate_metadata read hostname 'smithi090' 2024-04-02T14:37:54.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.849+0000 7f0c907ff740 1 --2- 172.21.15.90:0/3570876922 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x55d8cfcf06d0 0x55d8cfcf2b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:54.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.849+0000 7f0c907ff740 4 client.26312 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:37:54.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.849+0000 7f0c907ff740 1 -- 172.21.15.90:0/3570876922 --> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d8cfc37c30 con 0x55d8cfcf06d0 2024-04-02T14:37:54.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.850+0000 7f0c86000640 1 --2- 172.21.15.90:0/3570876922 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x55d8cfcf06d0 0x55d8cfcf2b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:54.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.852+0000 7f0c86000640 1 --2- 172.21.15.90:0/3570876922 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x55d8cfcf06d0 0x55d8cfcf2b50 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=444708e89618d957 server_cookie=60f39ff9b14a9650 in_seq=0 out_seq=0 2024-04-02T14:37:54.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.852+0000 7f0c70c00640 10 client.26312 ms_handle_connect on v2:172.21.15.92:6836/1270635939 2024-04-02T14:37:54.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.853+0000 7f0c70c00640 1 -- 172.21.15.90:0/3570876922 <== mds.3 v2:172.21.15.92:6836/1270635939 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d8cfc37c30 con 0x55d8cfcf06d0 2024-04-02T14:37:54.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.853+0000 7f0c70c00640 10 client.26312 handle_command_reply: tid=0 2024-04-02T14:37:54.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.853+0000 7f0c907ff740 1 client.26312 shutdown 2024-04-02T14:37:54.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.854+0000 7f0c907ff740 1 -- 172.21.15.90:0/3570876922 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d8cfcc5d10 msgr2=0x55d8cfccc030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:54.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.854+0000 7f0c907ff740 1 --2- 172.21.15.90:0/3570876922 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d8cfcc5d10 0x55d8cfccc030 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f0c740be170 tx=0x7f0c74003eb0 comp rx=0 tx=0).stop 2024-04-02T14:37:54.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.854+0000 7f0c907ff740 1 -- 172.21.15.90:0/3570876922 shutdown_connections 2024-04-02T14:37:54.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.854+0000 7f0c907ff740 1 --2- 172.21.15.90:0/3570876922 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x55d8cfcf06d0 0x55d8cfcf2b50 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:54.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.854+0000 7f0c907ff740 1 --2- 172.21.15.90:0/3570876922 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d8cfccf080 0x55d8cfcccab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:54.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.854+0000 7f0c907ff740 1 --2- 172.21.15.90:0/3570876922 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d8cfcc69f0 0x55d8cfccc570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:54.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.854+0000 7f0c70c00640 0 client.26312 ms_handle_reset on v2:172.21.15.92:6836/1270635939 2024-04-02T14:37:54.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.854+0000 7f0c907ff740 1 --2- 172.21.15.90:0/3570876922 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d8cfcc5d10 0x55d8cfccc030 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:54.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.854+0000 7f0c907ff740 1 -- 172.21.15.90:0/3570876922 >> 172.21.15.90:0/3570876922 conn(0x55d8cfc8ca50 msgr2=0x55d8cfcc24b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:37:54.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.854+0000 7f0c907ff740 1 -- 172.21.15.90:0/3570876922 shutdown_connections 2024-04-02T14:37:54.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.855+0000 7f0c907ff740 1 -- 172.21.15.90:0/3570876922 wait complete. 2024-04-02T14:37:54.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.855+0000 7f0c907ff740 20 client.26312 trim_cache size 0 max 16384 2024-04-02T14:37:54.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.882+0000 7f0c907ff740 1 Processor -- start 2024-04-02T14:37:54.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.882+0000 7f0c907ff740 1 -- start start 2024-04-02T14:37:54.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.883+0000 7f0c907ff740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d8cfcc5d10 0x55d8cfcdab80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:54.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.883+0000 7f0c907ff740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d8cfcc69f0 0x55d8cfcce690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:54.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.883+0000 7f0c907ff740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d8cfccf080 0x55d8cfcc22c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:54.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.883+0000 7f0c907ff740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f0c881ab6c0 con 0x55d8cfcc69f0 2024-04-02T14:37:54.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.883+0000 7f0c907ff740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f0c881ad200 con 0x55d8cfccf080 2024-04-02T14:37:54.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.883+0000 7f0c907ff740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f0c881abd90 con 0x55d8cfcc5d10 2024-04-02T14:37:54.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.883+0000 7f0c86a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d8cfcc5d10 0x55d8cfcdab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:54.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.883+0000 7f0c86a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d8cfcc5d10 0x55d8cfcdab80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:40288/0 (socket says 172.21.15.90:40288) 2024-04-02T14:37:54.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.883+0000 7f0c86a00640 1 -- 172.21.15.90:0/261511876 learned_addr learned my addr 172.21.15.90:0/261511876 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:37:54.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.883+0000 7f0c87400640 1 --2- 172.21.15.90:0/261511876 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d8cfccf080 0x55d8cfcc22c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:54.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.883+0000 7f0c86000640 1 --2- 172.21.15.90:0/261511876 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d8cfcc69f0 0x55d8cfcce690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:54.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.884+0000 7f0c86a00640 1 -- 172.21.15.90:0/261511876 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d8cfccf080 msgr2=0x55d8cfcc22c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:37:54.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.884+0000 7f0c86a00640 1 --2- 172.21.15.90:0/261511876 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d8cfccf080 0x55d8cfcc22c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:54.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.884+0000 7f0c86a00640 1 -- 172.21.15.90:0/261511876 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d8cfcc69f0 msgr2=0x55d8cfcce690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:37:54.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.884+0000 7f0c86a00640 1 --2- 172.21.15.90:0/261511876 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d8cfcc69f0 0x55d8cfcce690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:54.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.884+0000 7f0c86a00640 1 -- 172.21.15.90:0/261511876 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d8cfcc4f70 con 0x55d8cfcc5d10 2024-04-02T14:37:54.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.885+0000 7f0c86a00640 1 --2- 172.21.15.90:0/261511876 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d8cfcc5d10 0x55d8cfcdab80 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f0c740ba640 tx=0x7f0c74005fd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=751279294b5902d4 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:37:54.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.885+0000 7f0c73400640 1 -- 172.21.15.90:0/261511876 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c7404b070 con 0x55d8cfcc5d10 2024-04-02T14:37:54.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.885+0000 7f0c73400640 1 -- 172.21.15.90:0/261511876 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0c74004c80 con 0x55d8cfcc5d10 2024-04-02T14:37:54.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.885+0000 7f0c73400640 1 -- 172.21.15.90:0/261511876 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c740bdb40 con 0x55d8cfcc5d10 2024-04-02T14:37:54.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.886+0000 7f0c907ff740 1 -- 172.21.15.90:0/261511876 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d8cfcc5d10 msgr2=0x55d8cfcdab80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:54.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.886+0000 7f0c907ff740 1 --2- 172.21.15.90:0/261511876 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d8cfcc5d10 0x55d8cfcdab80 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f0c740ba640 tx=0x7f0c74005fd0 comp rx=0 tx=0).stop 2024-04-02T14:37:54.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.886+0000 7f0c907ff740 1 -- 172.21.15.90:0/261511876 shutdown_connections 2024-04-02T14:37:54.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.886+0000 7f0c907ff740 1 --2- 172.21.15.90:0/261511876 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d8cfccf080 0x55d8cfcc22c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:54.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.886+0000 7f0c907ff740 1 --2- 172.21.15.90:0/261511876 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d8cfcc69f0 0x55d8cfcce690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:54.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.886+0000 7f0c907ff740 1 --2- 172.21.15.90:0/261511876 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d8cfcc5d10 0x55d8cfcdab80 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:54.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.886+0000 7f0c907ff740 1 -- 172.21.15.90:0/261511876 >> 172.21.15.90:0/261511876 conn(0x55d8cfc8ca50 msgr2=0x55d8cfcc1bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:37:54.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.887+0000 7f0c907ff740 1 -- 172.21.15.90:0/261511876 shutdown_connections 2024-04-02T14:37:54.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.887+0000 7f0c907ff740 1 -- 172.21.15.90:0/261511876 wait complete. 2024-04-02T14:37:54.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.887+0000 7f0c907ff740 1 -- start start 2024-04-02T14:37:54.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.888+0000 7f0c907ff740 1 Processor -- start 2024-04-02T14:37:54.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.888+0000 7f0c907ff740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d8cfcc5d10 0x55d8cfcd02c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:54.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.888+0000 7f0c907ff740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d8cfcc69f0 0x55d8cfcd0800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:54.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.888+0000 7f0c907ff740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d8cfccf080 0x55d8cfcd56a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:54.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.888+0000 7f0c907ff740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f0c881acb30 con 0x55d8cfccf080 2024-04-02T14:37:54.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.888+0000 7f0c907ff740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f0c881ac460 con 0x55d8cfcc5d10 2024-04-02T14:37:54.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.888+0000 7f0c907ff740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f0c88068d40 con 0x55d8cfcc69f0 2024-04-02T14:37:54.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.888+0000 7f0c907ff740 10 client.0 _ll_register_callbacks cb 0x7f0c740f58a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:37:54.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.889+0000 7f0c86a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d8cfcc5d10 0x55d8cfcd02c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:54.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.889+0000 7f0c86a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d8cfcc5d10 0x55d8cfcd02c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:34216/0 (socket says 172.21.15.90:34216) 2024-04-02T14:37:54.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.889+0000 7f0c86a00640 1 -- 172.21.15.90:0/3365541225 learned_addr learned my addr 172.21.15.90:0/3365541225 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:37:54.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.889+0000 7f0c86000640 1 --2- 172.21.15.90:0/3365541225 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d8cfcc69f0 0x55d8cfcd0800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:54.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.889+0000 7f0c86a00640 1 -- 172.21.15.90:0/3365541225 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d8cfcc69f0 msgr2=0x55d8cfcd0800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:37:54.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.889+0000 7f0c86a00640 1 --2- 172.21.15.90:0/3365541225 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d8cfcc69f0 0x55d8cfcd0800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:54.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.889+0000 7f0c86a00640 1 -- 172.21.15.90:0/3365541225 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d8cfccf080 msgr2=0x55d8cfcd56a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:37:54.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.889+0000 7f0c86a00640 1 --2- 172.21.15.90:0/3365541225 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d8cfccf080 0x55d8cfcd56a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:54.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.889+0000 7f0c86a00640 1 -- 172.21.15.90:0/3365541225 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d8cfcd5be0 con 0x55d8cfcc5d10 2024-04-02T14:37:54.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.889+0000 7f0c86000640 1 --2- 172.21.15.90:0/3365541225 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d8cfcc69f0 0x55d8cfcd0800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:37:54.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.890+0000 7f0c86a00640 1 --2- 172.21.15.90:0/3365541225 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d8cfcc5d10 0x55d8cfcd02c0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f0c740c1040 tx=0x7f0c740bc6f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=324871d89b3368f5 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:37:54.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.890+0000 7f0c70c00640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:37:54.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.890+0000 7f0c70c00640 1 -- 172.21.15.90:0/3365541225 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c7404b070 con 0x55d8cfcc5d10 2024-04-02T14:37:54.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.890+0000 7f0c70c00640 1 -- 172.21.15.90:0/3365541225 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f0c740bd840 con 0x55d8cfcc5d10 2024-04-02T14:37:54.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.890+0000 7f0c907ff740 1 -- 172.21.15.90:0/3365541225 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0c74004c80 con 0x55d8cfcc5d10 2024-04-02T14:37:54.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.891+0000 7f0c70c00640 1 -- 172.21.15.90:0/3365541225 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f0c740d7040 con 0x55d8cfcc5d10 2024-04-02T14:37:54.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.891+0000 7f0c907ff740 10 client.26318 fetch_fsmap learned FSMap version 27 2024-04-02T14:37:54.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.891+0000 7f0c907ff740 1 -- 172.21.15.90:0/3365541225 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55d8cfcd5be0 con 0x55d8cfcc5d10 2024-04-02T14:37:54.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.891+0000 7f0c70c00640 1 -- 172.21.15.90:0/3365541225 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0c74004c80 con 0x55d8cfcc5d10 2024-04-02T14:37:54.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.891+0000 7f0c70c00640 1 -- 172.21.15.90:0/3365541225 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f0c740bd0a0 con 0x55d8cfcc5d10 2024-04-02T14:37:54.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.891+0000 7f0c907ff740 10 client.26318 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:37:54.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.891+0000 7f0c907ff740 10 client.26318 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-04-02T14:37:54.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.891+0000 7f0c907ff740 20 client.26318 populate_metadata read hostname 'smithi090' 2024-04-02T14:37:54.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.891+0000 7f0c907ff740 1 --2- 172.21.15.90:0/3365541225 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x55d8cfd57710 0x55d8cfd59b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:54.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.892+0000 7f0c907ff740 4 client.26318 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-02T14:37:54.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.892+0000 7f0c907ff740 1 -- 172.21.15.90:0/3365541225 --> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55d8cfc31f80 con 0x55d8cfd57710 2024-04-02T14:37:54.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.892+0000 7f0c86000640 1 --2- 172.21.15.90:0/3365541225 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x55d8cfd57710 0x55d8cfd59b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:54.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.893+0000 7f0c86000640 1 --2- 172.21.15.90:0/3365541225 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x55d8cfd57710 0x55d8cfd59b00 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=7f88da2be5355fc8 server_cookie=465c3d3d8a01bcc in_seq=0 out_seq=0 2024-04-02T14:37:54.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.893+0000 7f0c70c00640 10 client.26318 ms_handle_connect on v2:172.21.15.92:6836/1270635939 2024-04-02T14:37:54.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.893+0000 7f0c70c00640 1 -- 172.21.15.90:0/3365541225 <== mds.3 v2:172.21.15.92:6836/1270635939 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55d8cfc31f80 con 0x55d8cfd57710 2024-04-02T14:37:54.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.893+0000 7f0c70c00640 10 client.26318 handle_command_reply: tid=0 2024-04-02T14:37:54.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.893+0000 7f0c907ff740 1 client.26318 shutdown 2024-04-02T14:37:54.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.893+0000 7f0c907ff740 1 -- 172.21.15.90:0/3365541225 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d8cfcc5d10 msgr2=0x55d8cfcd02c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:54.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.893+0000 7f0c907ff740 1 --2- 172.21.15.90:0/3365541225 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d8cfcc5d10 0x55d8cfcd02c0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f0c740c1040 tx=0x7f0c740bc6f0 comp rx=0 tx=0).stop 2024-04-02T14:37:54.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.894+0000 7f0c907ff740 1 -- 172.21.15.90:0/3365541225 shutdown_connections 2024-04-02T14:37:54.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.894+0000 7f0c907ff740 1 --2- 172.21.15.90:0/3365541225 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x55d8cfd57710 0x55d8cfd59b00 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:54.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.894+0000 7f0c907ff740 1 --2- 172.21.15.90:0/3365541225 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d8cfccf080 0x55d8cfcd56a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:54.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.894+0000 7f0c907ff740 1 --2- 172.21.15.90:0/3365541225 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d8cfcc69f0 0x55d8cfcd0800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:54.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.894+0000 7f0c907ff740 1 --2- 172.21.15.90:0/3365541225 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d8cfcc5d10 0x55d8cfcd02c0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:54.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.894+0000 7f0c907ff740 1 -- 172.21.15.90:0/3365541225 >> 172.21.15.90:0/3365541225 conn(0x55d8cfc8ca50 msgr2=0x55d8cfccc880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:37:54.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.894+0000 7f0c70c00640 0 client.26318 ms_handle_reset on v2:172.21.15.92:6836/1270635939 2024-04-02T14:37:54.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.894+0000 7f0c907ff740 1 -- 172.21.15.90:0/3365541225 shutdown_connections 2024-04-02T14:37:54.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.895+0000 7f0c907ff740 1 -- 172.21.15.90:0/3365541225 wait complete. 2024-04-02T14:37:54.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.895+0000 7f0c907ff740 20 client.26318 trim_cache size 0 max 16384 2024-04-02T14:37:54.900 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-02T14:37:54.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.896+0000 7f0c8dc00640 1 -- 172.21.15.90:0/288741834 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f0c6c0a3e20 msgr2=0x7f0c6c0a62e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:54.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.896+0000 7f0c8dc00640 1 --2- 172.21.15.90:0/288741834 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f0c6c0a3e20 0x7f0c6c0a62e0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f0c80005fd0 tx=0x7f0c800098d0 comp rx=0 tx=0).stop 2024-04-02T14:37:54.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.896+0000 7f0c8dc00640 1 -- 172.21.15.90:0/288741834 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0c88101a90 msgr2=0x7f0c881089c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:54.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.896+0000 7f0c8dc00640 1 --2- 172.21.15.90:0/288741834 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0c88101a90 0x7f0c881089c0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f0c7402c180 tx=0x7f0c740043b0 comp rx=0 tx=0).stop 2024-04-02T14:37:54.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.897+0000 7f0c8dc00640 1 -- 172.21.15.90:0/288741834 shutdown_connections 2024-04-02T14:37:54.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.897+0000 7f0c8dc00640 1 --2- 172.21.15.90:0/288741834 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f0c6c0a3e20 0x7f0c6c0a62e0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:54.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.897+0000 7f0c8dc00640 1 --2- 172.21.15.90:0/288741834 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0c88107aa0 0x7f0c88109440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:54.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.897+0000 7f0c8dc00640 1 --2- 172.21.15.90:0/288741834 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0c88102450 0x7f0c88108f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:54.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.897+0000 7f0c8dc00640 1 --2- 172.21.15.90:0/288741834 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0c88101a90 0x7f0c881089c0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:54.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.897+0000 7f0c8dc00640 1 -- 172.21.15.90:0/288741834 >> 172.21.15.90:0/288741834 conn(0x7f0c88078f00 msgr2=0x7f0c8810d200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:37:54.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.897+0000 7f0c8dc00640 1 -- 172.21.15.90:0/288741834 shutdown_connections 2024-04-02T14:37:54.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:54.897+0000 7f0c8dc00640 1 -- 172.21.15.90:0/288741834 wait complete. 2024-04-02T14:37:54.924 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:37:54 smithi090 ceph-mon[31674]: from='client.? 172.21.15.90:0/2848626930' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-02T14:37:54.928 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-02T14:37:54.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:37:54 smithi177 ceph-mon[36431]: from='client.? 172.21.15.90:0/2848626930' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-02T14:37:55.110 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:37:54 smithi092 ceph-mon[36497]: from='client.? 172.21.15.90:0/2848626930' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-02T14:37:55.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.234+0000 7f51b3a00640 1 -- 172.21.15.90:0/2890465968 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f51ac114c50 msgr2=0x7f51ac117040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:55.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.234+0000 7f51b3a00640 1 --2- 172.21.15.90:0/2890465968 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f51ac114c50 0x7f51ac117040 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f51a4009900 tx=0x7f51a402bc30 comp rx=0 tx=0).stop 2024-04-02T14:37:55.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.234+0000 7f51b3a00640 1 -- 172.21.15.90:0/2890465968 shutdown_connections 2024-04-02T14:37:55.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.234+0000 7f51b3a00640 1 --2- 172.21.15.90:0/2890465968 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f51ac114c50 0x7f51ac117040 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:55.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.234+0000 7f51b3a00640 1 --2- 172.21.15.90:0/2890465968 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f51ac109600 0x7f51ac114710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:55.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.234+0000 7f51b3a00640 1 --2- 172.21.15.90:0/2890465968 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f51ac108ce0 0x7f51ac1090c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:55.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.234+0000 7f51b3a00640 1 -- 172.21.15.90:0/2890465968 >> 172.21.15.90:0/2890465968 conn(0x7f51ac06cbe0 msgr2=0x7f51ac06cff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:37:55.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.235+0000 7f51b3a00640 1 -- 172.21.15.90:0/2890465968 shutdown_connections 2024-04-02T14:37:55.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.235+0000 7f51b3a00640 1 -- 172.21.15.90:0/2890465968 wait complete. 2024-04-02T14:37:55.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.237+0000 7f51b3a00640 1 Processor -- start 2024-04-02T14:37:55.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.238+0000 7f51b3a00640 1 -- start start 2024-04-02T14:37:55.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.239+0000 7f51b3a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f51ac108ce0 0x7f51ac1a6800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:55.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.239+0000 7f51b3a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f51ac109600 0x7f51ac1a6d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:55.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.239+0000 7f51b3a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f51ac114c50 0x7f51ac1a09b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:55.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.239+0000 7f51b3a00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f51ac11a690 con 0x7f51ac108ce0 2024-04-02T14:37:55.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.239+0000 7f51b3a00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f51ac11a510 con 0x7f51ac114c50 2024-04-02T14:37:55.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.239+0000 7f51b3a00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f51ac11a810 con 0x7f51ac109600 2024-04-02T14:37:55.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.240+0000 7f51abe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f51ac109600 0x7f51ac1a6d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:55.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.240+0000 7f51abe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f51ac109600 0x7f51ac1a6d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:40324/0 (socket says 172.21.15.90:40324) 2024-04-02T14:37:55.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.240+0000 7f51abe00640 1 -- 172.21.15.90:0/2991185471 learned_addr learned my addr 172.21.15.90:0/2991185471 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:37:55.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.240+0000 7f51b1a00640 1 --2- 172.21.15.90:0/2991185471 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f51ac114c50 0x7f51ac1a09b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:55.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.240+0000 7f51abe00640 1 -- 172.21.15.90:0/2991185471 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f51ac114c50 msgr2=0x7f51ac1a09b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:55.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.240+0000 7f51abe00640 1 --2- 172.21.15.90:0/2991185471 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f51ac114c50 0x7f51ac1a09b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:55.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.240+0000 7f51abe00640 1 -- 172.21.15.90:0/2991185471 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f51ac108ce0 msgr2=0x7f51ac1a6800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:55.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.240+0000 7f51abe00640 1 --2- 172.21.15.90:0/2991185471 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f51ac108ce0 0x7f51ac1a6800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:55.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.240+0000 7f51abe00640 1 -- 172.21.15.90:0/2991185471 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51ac1a1130 con 0x7f51ac109600 2024-04-02T14:37:55.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.242+0000 7f51abe00640 1 --2- 172.21.15.90:0/2991185471 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f51ac109600 0x7f51ac1a6d40 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f519c00db80 tx=0x7f519c00b630 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:37:55.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.243+0000 7f51a9600640 1 -- 172.21.15.90:0/2991185471 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f519c017070 con 0x7f51ac109600 2024-04-02T14:37:55.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.243+0000 7f51a9600640 1 -- 172.21.15.90:0/2991185471 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f519c003b70 con 0x7f51ac109600 2024-04-02T14:37:55.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.243+0000 7f51a9600640 1 -- 172.21.15.90:0/2991185471 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f519c0106c0 con 0x7f51ac109600 2024-04-02T14:37:55.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.243+0000 7f51b3a00640 1 -- 172.21.15.90:0/2991185471 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51ac1a1420 con 0x7f51ac109600 2024-04-02T14:37:55.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.243+0000 7f51b3a00640 1 -- 172.21.15.90:0/2991185471 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51ac10a4a0 con 0x7f51ac109600 2024-04-02T14:37:55.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.250+0000 7f51a9600640 1 -- 172.21.15.90:0/2991185471 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f519c004020 con 0x7f51ac109600 2024-04-02T14:37:55.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.252+0000 7f51a9600640 1 --2- 172.21.15.90:0/2991185471 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f51900a8330 0x7f51900aa7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:55.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.253+0000 7f51b67fe740 1 Processor -- start 2024-04-02T14:37:55.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.253+0000 7f51b1000640 1 --2- 172.21.15.90:0/2991185471 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f51900a8330 0x7f51900aa7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:55.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.253+0000 7f51b67fe740 1 -- start start 2024-04-02T14:37:55.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.254+0000 7f51b67fe740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f3f03389f0 0x55f3f0337740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:55.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.254+0000 7f51b67fe740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f3f0337d10 0x55f3f0338190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:55.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.254+0000 7f51b67fe740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f3f0341080 0x55f3f0343470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:55.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.254+0000 7f51b67fe740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55f3f0278400 con 0x55f3f0341080 2024-04-02T14:37:55.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.254+0000 7f51b67fe740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55f3f025cf10 con 0x55f3f0337d10 2024-04-02T14:37:55.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.254+0000 7f51b67fe740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55f3f029c6f0 con 0x55f3f03389f0 2024-04-02T14:37:55.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.255+0000 7f51b1000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f3f03389f0 0x55f3f0337740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:55.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.255+0000 7f51b1000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f3f03389f0 0x55f3f0337740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:40332/0 (socket says 172.21.15.90:40332) 2024-04-02T14:37:55.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.255+0000 7f51b1000640 1 -- 172.21.15.90:0/2661133917 learned_addr learned my addr 172.21.15.90:0/2661133917 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:37:55.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.255+0000 7f51a9600640 1 -- 172.21.15.90:0/2991185471 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f519c0d0650 con 0x7f51ac109600 2024-04-02T14:37:55.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.256+0000 7f51b1a00640 1 --2- 172.21.15.90:0/2661133917 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f3f0341080 0x55f3f0343470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:55.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.256+0000 7f51abe00640 1 --2- 172.21.15.90:0/2661133917 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f3f0337d10 0x55f3f0338190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:55.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.256+0000 7f51abe00640 1 -- 172.21.15.90:0/2661133917 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f3f03389f0 msgr2=0x55f3f0337740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:37:55.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.256+0000 7f51abe00640 1 --2- 172.21.15.90:0/2661133917 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f3f03389f0 0x55f3f0337740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:55.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.256+0000 7f51abe00640 1 -- 172.21.15.90:0/2661133917 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f3f0341080 msgr2=0x55f3f0343470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:37:55.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.257+0000 7f51abe00640 1 --2- 172.21.15.90:0/2661133917 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f3f0341080 0x55f3f0343470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:55.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.257+0000 7f51abe00640 1 -- 172.21.15.90:0/2661133917 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f3effb03f0 con 0x55f3f0337d10 2024-04-02T14:37:55.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.257+0000 7f51b1a00640 1 --2- 172.21.15.90:0/2661133917 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f3f0341080 0x55f3f0343470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:37:55.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.257+0000 7f51b1000640 1 --2- 172.21.15.90:0/2991185471 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f51900a8330 0x7f51900aa7f0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f5194009d50 tx=0x7f51940125e0 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:37:55.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.258+0000 7f51abe00640 1 --2- 172.21.15.90:0/2661133917 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f3f0337d10 0x55f3f0338190 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f519c0cdd50 tx=0x7f519c005a50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=886a32f5c6c82fb server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:37:55.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.259+0000 7f51a2000640 1 -- 172.21.15.90:0/2661133917 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f519c014030 con 0x55f3f0337d10 2024-04-02T14:37:55.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.259+0000 7f51a2000640 1 -- 172.21.15.90:0/2661133917 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f519c09c5a0 con 0x55f3f0337d10 2024-04-02T14:37:55.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.259+0000 7f51a2000640 1 -- 172.21.15.90:0/2661133917 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f519c019310 con 0x55f3f0337d10 2024-04-02T14:37:55.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.260+0000 7f51b67fe740 1 -- 172.21.15.90:0/2661133917 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f3f0337d10 msgr2=0x55f3f0338190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:55.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.260+0000 7f51b67fe740 1 --2- 172.21.15.90:0/2661133917 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f3f0337d10 0x55f3f0338190 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f519c0cdd50 tx=0x7f519c005a50 comp rx=0 tx=0).stop 2024-04-02T14:37:55.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.260+0000 7f51b67fe740 1 -- 172.21.15.90:0/2661133917 shutdown_connections 2024-04-02T14:37:55.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.260+0000 7f51b67fe740 1 --2- 172.21.15.90:0/2661133917 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f3f0341080 0x55f3f0343470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:55.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.260+0000 7f51b67fe740 1 --2- 172.21.15.90:0/2661133917 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f3f0337d10 0x55f3f0338190 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:55.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.260+0000 7f51b67fe740 1 --2- 172.21.15.90:0/2661133917 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f3f03389f0 0x55f3f0337740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:55.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.260+0000 7f51b67fe740 1 -- 172.21.15.90:0/2661133917 >> 172.21.15.90:0/2661133917 conn(0x55f3f02fea50 msgr2=0x55f3f0332760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:37:55.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.261+0000 7f51b67fe740 1 -- 172.21.15.90:0/2661133917 shutdown_connections 2024-04-02T14:37:55.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.261+0000 7f51b67fe740 1 -- 172.21.15.90:0/2661133917 wait complete. 2024-04-02T14:37:55.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.262+0000 7f51b67fe740 1 -- start start 2024-04-02T14:37:55.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.263+0000 7f51b67fe740 1 Processor -- start 2024-04-02T14:37:55.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.264+0000 7f51b67fe740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f3f0337d10 0x55f3f033e150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:55.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.264+0000 7f51b67fe740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f3f03389f0 0x55f3f033e690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:55.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.264+0000 7f51b67fe740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f3f0341080 0x55f3f033ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:55.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.264+0000 7f51b67fe740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55f3f02986a0 con 0x55f3f0337d10 2024-04-02T14:37:55.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.264+0000 7f51b67fe740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55f3f0244980 con 0x55f3f0341080 2024-04-02T14:37:55.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.264+0000 7f51b67fe740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55f3f0257cb0 con 0x55f3f03389f0 2024-04-02T14:37:55.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.264+0000 7f51b67fe740 10 client.0 _ll_register_callbacks cb 0x55f3f02576a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:37:55.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.265+0000 7f51abe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f3f03389f0 0x55f3f033e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:55.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.265+0000 7f51abe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f3f03389f0 0x55f3f033e690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:40334/0 (socket says 172.21.15.90:40334) 2024-04-02T14:37:55.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.265+0000 7f51abe00640 1 -- 172.21.15.90:0/597817256 learned_addr learned my addr 172.21.15.90:0/597817256 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:37:55.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.265+0000 7f51b1a00640 1 --2- 172.21.15.90:0/597817256 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f3f0341080 0x55f3f033ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:55.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.266+0000 7f51abe00640 1 -- 172.21.15.90:0/597817256 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f3f0341080 msgr2=0x55f3f033ebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:37:55.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.266+0000 7f51abe00640 1 --2- 172.21.15.90:0/597817256 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f3f0341080 0x55f3f033ebd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:55.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.266+0000 7f51abe00640 1 -- 172.21.15.90:0/597817256 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f3f0337d10 msgr2=0x55f3f033e150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:37:55.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.266+0000 7f51b1000640 1 --2- 172.21.15.90:0/597817256 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f3f0337d10 0x55f3f033e150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:55.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.266+0000 7f51abe00640 1 --2- 172.21.15.90:0/597817256 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f3f0337d10 0x55f3f033e150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:55.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.266+0000 7f51abe00640 1 -- 172.21.15.90:0/597817256 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f3f007a7d0 con 0x55f3f03389f0 2024-04-02T14:37:55.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.267+0000 7f51b1000640 1 --2- 172.21.15.90:0/597817256 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f3f0337d10 0x55f3f033e150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:37:55.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.267+0000 7f51abe00640 1 --2- 172.21.15.90:0/597817256 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f3f03389f0 0x55f3f033e690 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f519c00bb30 tx=0x7f519c0a5290 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d720d8a9fe610c4c server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:37:55.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.267+0000 7f519aa00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:37:55.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.268+0000 7f519aa00640 1 -- 172.21.15.90:0/597817256 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f519c0aa060 con 0x55f3f03389f0 2024-04-02T14:37:55.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.268+0000 7f519aa00640 1 -- 172.21.15.90:0/597817256 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f519c0cdfd0 con 0x55f3f03389f0 2024-04-02T14:37:55.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.268+0000 7f51b67fe740 1 -- 172.21.15.90:0/597817256 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f519c09c5a0 con 0x55f3f03389f0 2024-04-02T14:37:55.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.269+0000 7f519aa00640 1 -- 172.21.15.90:0/597817256 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f519c014040 con 0x55f3f03389f0 2024-04-02T14:37:55.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.269+0000 7f51b67fe740 10 client.26233 fetch_fsmap learned FSMap version 27 2024-04-02T14:37:55.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.269+0000 7f51b67fe740 1 -- 172.21.15.90:0/597817256 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55f3f037a7c0 con 0x55f3f03389f0 2024-04-02T14:37:55.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.270+0000 7f519aa00640 1 -- 172.21.15.90:0/597817256 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f519c09c5a0 con 0x55f3f03389f0 2024-04-02T14:37:55.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.271+0000 7f519aa00640 1 -- 172.21.15.90:0/597817256 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f519c09b9f0 con 0x55f3f03389f0 2024-04-02T14:37:55.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.271+0000 7f51b67fe740 10 client.26233 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:37:55.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.271+0000 7f51b67fe740 10 client.26233 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-04-02T14:37:55.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.271+0000 7f51b67fe740 20 client.26233 populate_metadata read hostname 'smithi090' 2024-04-02T14:37:55.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.271+0000 7f51b67fe740 1 --2- 172.21.15.90:0/597817256 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x55f3f037bb10 0x55f3f037df00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:55.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.271+0000 7f51b67fe740 4 client.26233 mds_command: new command op to 24547 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:37:55.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.271+0000 7f51b67fe740 1 -- 172.21.15.90:0/597817256 --> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f3f025c040 con 0x55f3f037bb10 2024-04-02T14:37:55.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.271+0000 7f51b1000640 1 --2- 172.21.15.90:0/597817256 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x55f3f037bb10 0x55f3f037df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:55.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.272+0000 7f51b1000640 1 --2- 172.21.15.90:0/597817256 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x55f3f037bb10 0x55f3f037df00 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=45308a5ad6d59b3a server_cookie=a66cb7b83d72fb16 in_seq=0 out_seq=0 2024-04-02T14:37:55.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.272+0000 7f519aa00640 10 client.26233 ms_handle_connect on v2:172.21.15.177:6838/664895663 2024-04-02T14:37:55.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.273+0000 7f519aa00640 1 -- 172.21.15.90:0/597817256 <== mds.2 v2:172.21.15.177:6838/664895663 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f3f025c040 con 0x55f3f037bb10 2024-04-02T14:37:55.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.273+0000 7f519aa00640 10 client.26233 handle_command_reply: tid=0 2024-04-02T14:37:55.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.273+0000 7f51b67fe740 1 client.26233 shutdown 2024-04-02T14:37:55.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.275+0000 7f51b67fe740 1 -- 172.21.15.90:0/597817256 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f3f03389f0 msgr2=0x55f3f033e690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:55.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.275+0000 7f51b67fe740 1 --2- 172.21.15.90:0/597817256 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f3f03389f0 0x55f3f033e690 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f519c00bb30 tx=0x7f519c0a5290 comp rx=0 tx=0).stop 2024-04-02T14:37:55.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.275+0000 7f51b67fe740 1 -- 172.21.15.90:0/597817256 shutdown_connections 2024-04-02T14:37:55.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.275+0000 7f51b67fe740 1 --2- 172.21.15.90:0/597817256 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x55f3f037bb10 0x55f3f037df00 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:55.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.275+0000 7f51b67fe740 1 --2- 172.21.15.90:0/597817256 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f3f0341080 0x55f3f033ebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:55.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.275+0000 7f51b67fe740 1 --2- 172.21.15.90:0/597817256 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f3f03389f0 0x55f3f033e690 secure :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f519c00bb30 tx=0x7f519c0a5290 comp rx=0 tx=0).stop 2024-04-02T14:37:55.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.275+0000 7f51b67fe740 1 --2- 172.21.15.90:0/597817256 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f3f0337d10 0x55f3f033e150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:55.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.275+0000 7f519aa00640 0 client.26233 ms_handle_reset on v2:172.21.15.177:6838/664895663 2024-04-02T14:37:55.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.275+0000 7f51b67fe740 1 -- 172.21.15.90:0/597817256 >> 172.21.15.90:0/597817256 conn(0x55f3f02fea50 msgr2=0x55f3f03344b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:37:55.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.276+0000 7f51b67fe740 1 -- 172.21.15.90:0/597817256 shutdown_connections 2024-04-02T14:37:55.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.276+0000 7f51b67fe740 1 -- 172.21.15.90:0/597817256 wait complete. 2024-04-02T14:37:55.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.276+0000 7f51b67fe740 20 client.26233 trim_cache size 0 max 16384 2024-04-02T14:37:55.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.302+0000 7f51b67fe740 1 Processor -- start 2024-04-02T14:37:55.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.302+0000 7f51b67fe740 1 -- start start 2024-04-02T14:37:55.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.302+0000 7f51b67fe740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f3f0337d10 0x55f3f0340880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:55.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.303+0000 7f51b67fe740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f3f0341080 0x55f3f033e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:55.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.303+0000 7f51b67fe740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f3f033ed60 0x55f3f0342470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:55.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.303+0000 7f51b1000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f3f0337d10 0x55f3f0340880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:55.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.303+0000 7f51b1000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f3f0337d10 0x55f3f0340880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:34272/0 (socket says 172.21.15.90:34272) 2024-04-02T14:37:55.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.303+0000 7f51b1a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f3f033ed60 0x55f3f0342470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:55.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.303+0000 7f51b1a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f3f033ed60 0x55f3f0342470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:40340/0 (socket says 172.21.15.90:40340) 2024-04-02T14:37:55.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.303+0000 7f51b1000640 1 -- 172.21.15.90:0/1068807391 learned_addr learned my addr 172.21.15.90:0/1068807391 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:37:55.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.303+0000 7f51b67fe740 1 -- 172.21.15.90:0/1068807391 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f51ac07a5c0 con 0x55f3f0341080 2024-04-02T14:37:55.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.303+0000 7f51b67fe740 1 -- 172.21.15.90:0/1068807391 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f51ac07c100 con 0x55f3f0337d10 2024-04-02T14:37:55.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.303+0000 7f51b67fe740 1 -- 172.21.15.90:0/1068807391 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f51ac07ac90 con 0x55f3f033ed60 2024-04-02T14:37:55.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.304+0000 7f51b1a00640 1 -- 172.21.15.90:0/1068807391 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f3f0337d10 msgr2=0x55f3f0340880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:37:55.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.304+0000 7f51b1a00640 1 --2- 172.21.15.90:0/1068807391 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f3f0337d10 0x55f3f0340880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:55.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.304+0000 7f51b1a00640 1 -- 172.21.15.90:0/1068807391 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f3f0341080 msgr2=0x55f3f033e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:37:55.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.304+0000 7f51abe00640 1 --2- 172.21.15.90:0/1068807391 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f3f0341080 0x55f3f033e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:55.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.304+0000 7f51b1a00640 1 --2- 172.21.15.90:0/1068807391 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f3f0341080 0x55f3f033e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:55.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.304+0000 7f51b1a00640 1 -- 172.21.15.90:0/1068807391 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f3f032d960 con 0x55f3f033ed60 2024-04-02T14:37:55.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.304+0000 7f51abe00640 1 --2- 172.21.15.90:0/1068807391 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f3f0341080 0x55f3f033e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:37:55.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.305+0000 7f51b1a00640 1 --2- 172.21.15.90:0/1068807391 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f3f033ed60 0x55f3f0342470 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f51a4009230 tx=0x7f51a402c6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4b86acd3bf5f83b4 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:37:55.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.305+0000 7f51a1600640 1 -- 172.21.15.90:0/1068807391 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51a4004240 con 0x55f3f033ed60 2024-04-02T14:37:55.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.305+0000 7f51a1600640 1 -- 172.21.15.90:0/1068807391 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f51a40043e0 con 0x55f3f033ed60 2024-04-02T14:37:55.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.306+0000 7f51b67fe740 1 -- 172.21.15.90:0/1068807391 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f3f033ed60 msgr2=0x55f3f0342470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:55.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.307+0000 7f51a1600640 1 -- 172.21.15.90:0/1068807391 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51a4004240 con 0x55f3f033ed60 2024-04-02T14:37:55.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.307+0000 7f51b67fe740 1 --2- 172.21.15.90:0/1068807391 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f3f033ed60 0x55f3f0342470 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f51a4009230 tx=0x7f51a402c6a0 comp rx=0 tx=0).stop 2024-04-02T14:37:55.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.307+0000 7f51b67fe740 1 -- 172.21.15.90:0/1068807391 shutdown_connections 2024-04-02T14:37:55.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.307+0000 7f51b67fe740 1 --2- 172.21.15.90:0/1068807391 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f3f033ed60 0x55f3f0342470 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:55.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.307+0000 7f51b67fe740 1 --2- 172.21.15.90:0/1068807391 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f3f0341080 0x55f3f033e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:55.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.307+0000 7f51b67fe740 1 --2- 172.21.15.90:0/1068807391 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f3f0337d10 0x55f3f0340880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:55.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.307+0000 7f51b67fe740 1 -- 172.21.15.90:0/1068807391 >> 172.21.15.90:0/1068807391 conn(0x55f3f02fea50 msgr2=0x55f3f033e110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:37:55.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.308+0000 7f51b67fe740 1 -- 172.21.15.90:0/1068807391 shutdown_connections 2024-04-02T14:37:55.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.308+0000 7f51b67fe740 1 -- 172.21.15.90:0/1068807391 wait complete. 2024-04-02T14:37:55.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.309+0000 7f51b67fe740 1 -- start start 2024-04-02T14:37:55.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.309+0000 7f51b67fe740 1 Processor -- start 2024-04-02T14:37:55.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.309+0000 7f51b67fe740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f3f0337d10 0x55f3f03813c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:55.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.309+0000 7f51b67fe740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f3f033ed60 0x55f3f037d740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:55.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.309+0000 7f51b67fe740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f3f0341080 0x55f3f037dd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:55.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.309+0000 7f51b67fe740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f51ac07ba30 con 0x55f3f033ed60 2024-04-02T14:37:55.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.309+0000 7f51b67fe740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f51ac07b360 con 0x55f3f0341080 2024-04-02T14:37:55.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.309+0000 7f51b67fe740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f51ac0785a0 con 0x55f3f0337d10 2024-04-02T14:37:55.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.309+0000 7f51b67fe740 10 client.0 _ll_register_callbacks cb 0x7f5194009050 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:37:55.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.310+0000 7f51b1000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f3f0337d10 0x55f3f03813c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:55.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.310+0000 7f51b1000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f3f0337d10 0x55f3f03813c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:40342/0 (socket says 172.21.15.90:40342) 2024-04-02T14:37:55.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.310+0000 7f51b1000640 1 -- 172.21.15.90:0/3713706426 learned_addr learned my addr 172.21.15.90:0/3713706426 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:37:55.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.310+0000 7f51b1a00640 1 --2- 172.21.15.90:0/3713706426 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f3f0341080 0x55f3f037dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:55.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.310+0000 7f51abe00640 1 --2- 172.21.15.90:0/3713706426 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f3f033ed60 0x55f3f037d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:55.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.311+0000 7f51b1000640 1 -- 172.21.15.90:0/3713706426 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f3f0341080 msgr2=0x55f3f037dd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:37:55.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.311+0000 7f51b1000640 1 --2- 172.21.15.90:0/3713706426 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f3f0341080 0x55f3f037dd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:55.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.311+0000 7f51b1000640 1 -- 172.21.15.90:0/3713706426 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f3f033ed60 msgr2=0x55f3f037d740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:37:55.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.311+0000 7f51b1000640 1 --2- 172.21.15.90:0/3713706426 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f3f033ed60 0x55f3f037d740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:55.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.311+0000 7f51b1000640 1 -- 172.21.15.90:0/3713706426 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f3f037e550 con 0x55f3f0337d10 2024-04-02T14:37:55.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.311+0000 7f51abe00640 1 --2- 172.21.15.90:0/3713706426 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f3f033ed60 0x55f3f037d740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:37:55.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.312+0000 7f51b1000640 1 --2- 172.21.15.90:0/3713706426 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f3f0337d10 0x55f3f03813c0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f5194012ae0 tx=0x7f5194015000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9ce8c259f6bea882 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:37:55.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.312+0000 7f519aa00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:37:55.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.312+0000 7f519aa00640 1 -- 172.21.15.90:0/3713706426 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5194033060 con 0x55f3f0337d10 2024-04-02T14:37:55.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.313+0000 7f519aa00640 1 -- 172.21.15.90:0/3713706426 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f519400f040 con 0x55f3f0337d10 2024-04-02T14:37:55.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.313+0000 7f51b67fe740 1 -- 172.21.15.90:0/3713706426 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f51a40043e0 con 0x55f3f0337d10 2024-04-02T14:37:55.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.313+0000 7f519aa00640 1 -- 172.21.15.90:0/3713706426 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f5194033060 con 0x55f3f0337d10 2024-04-02T14:37:55.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.314+0000 7f51b67fe740 10 client.26239 fetch_fsmap learned FSMap version 27 2024-04-02T14:37:55.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.314+0000 7f51b67fe740 1 -- 172.21.15.90:0/3713706426 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55f3f03e2d90 con 0x55f3f0337d10 2024-04-02T14:37:55.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.314+0000 7f519aa00640 1 -- 172.21.15.90:0/3713706426 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f519401c030 con 0x55f3f0337d10 2024-04-02T14:37:55.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.314+0000 7f519aa00640 1 -- 172.21.15.90:0/3713706426 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f5194015070 con 0x55f3f0337d10 2024-04-02T14:37:55.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.314+0000 7f51b67fe740 10 client.26239 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:37:55.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.314+0000 7f51b67fe740 10 client.26239 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-04-02T14:37:55.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.314+0000 7f51b67fe740 20 client.26239 populate_metadata read hostname 'smithi090' 2024-04-02T14:37:55.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.315+0000 7f51b67fe740 1 --2- 172.21.15.90:0/3713706426 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x55f3f03e34f0 0x55f3f03e5910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:55.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.315+0000 7f51b67fe740 4 client.26239 mds_command: new command op to 24547 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-02T14:37:55.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.315+0000 7f51b67fe740 1 -- 172.21.15.90:0/3713706426 --> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7f51a40043e0 con 0x55f3f03e34f0 2024-04-02T14:37:55.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.315+0000 7f51abe00640 1 --2- 172.21.15.90:0/3713706426 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x55f3f03e34f0 0x55f3f03e5910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:55.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.315+0000 7f519aa00640 10 client.26239 ms_handle_connect on v2:172.21.15.177:6838/664895663 2024-04-02T14:37:55.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.317+0000 7f51abe00640 1 --2- 172.21.15.90:0/3713706426 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x55f3f03e34f0 0x55f3f03e5910 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=9413964d53957e56 server_cookie=ee331743ffb04c20 in_seq=0 out_seq=0 2024-04-02T14:37:55.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.318+0000 7f519aa00640 1 -- 172.21.15.90:0/3713706426 <== mds.2 v2:172.21.15.177:6838/664895663 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7f51a40043e0 con 0x55f3f03e34f0 2024-04-02T14:37:55.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.318+0000 7f519aa00640 10 client.26239 handle_command_reply: tid=0 2024-04-02T14:37:55.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.318+0000 7f51b67fe740 1 client.26239 shutdown 2024-04-02T14:37:55.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.319+0000 7f51b67fe740 1 -- 172.21.15.90:0/3713706426 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f3f0337d10 msgr2=0x55f3f03813c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:55.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.319+0000 7f51b67fe740 1 --2- 172.21.15.90:0/3713706426 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f3f0337d10 0x55f3f03813c0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f5194012ae0 tx=0x7f5194015000 comp rx=0 tx=0).stop 2024-04-02T14:37:55.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.319+0000 7f51b67fe740 1 -- 172.21.15.90:0/3713706426 shutdown_connections 2024-04-02T14:37:55.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.319+0000 7f51b67fe740 1 --2- 172.21.15.90:0/3713706426 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x55f3f03e34f0 0x55f3f03e5910 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:55.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.319+0000 7f51b67fe740 1 --2- 172.21.15.90:0/3713706426 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f3f0341080 0x55f3f037dd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:55.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.319+0000 7f51b67fe740 1 --2- 172.21.15.90:0/3713706426 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f3f033ed60 0x55f3f037d740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:55.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.319+0000 7f51b67fe740 1 --2- 172.21.15.90:0/3713706426 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f3f0337d10 0x55f3f03813c0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:55.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.319+0000 7f51b67fe740 1 -- 172.21.15.90:0/3713706426 >> 172.21.15.90:0/3713706426 conn(0x55f3f02fea50 msgr2=0x55f3f0376840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:37:55.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.319+0000 7f519aa00640 0 client.26239 ms_handle_reset on v2:172.21.15.177:6838/664895663 2024-04-02T14:37:55.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.320+0000 7f51b67fe740 1 -- 172.21.15.90:0/3713706426 shutdown_connections 2024-04-02T14:37:55.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.320+0000 7f51b67fe740 1 -- 172.21.15.90:0/3713706426 wait complete. 2024-04-02T14:37:55.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.320+0000 7f51b67fe740 20 client.26239 trim_cache size 0 max 16384 2024-04-02T14:37:55.323 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-02T14:37:55.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.321+0000 7f51b3a00640 1 -- 172.21.15.90:0/2991185471 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f51900a8330 msgr2=0x7f51900aa7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:55.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.321+0000 7f51b3a00640 1 --2- 172.21.15.90:0/2991185471 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f51900a8330 0x7f51900aa7f0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f5194009d50 tx=0x7f51940125e0 comp rx=0 tx=0).stop 2024-04-02T14:37:55.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.321+0000 7f51b3a00640 1 -- 172.21.15.90:0/2991185471 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f51ac109600 msgr2=0x7f51ac1a6d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:55.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.321+0000 7f51b3a00640 1 --2- 172.21.15.90:0/2991185471 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f51ac109600 0x7f51ac1a6d40 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f519c00db80 tx=0x7f519c00b630 comp rx=0 tx=0).stop 2024-04-02T14:37:55.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.322+0000 7f51b3a00640 1 -- 172.21.15.90:0/2991185471 shutdown_connections 2024-04-02T14:37:55.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.322+0000 7f51b3a00640 1 --2- 172.21.15.90:0/2991185471 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f51900a8330 0x7f51900aa7f0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:55.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.322+0000 7f51b3a00640 1 --2- 172.21.15.90:0/2991185471 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f51ac114c50 0x7f51ac1a09b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:55.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.322+0000 7f51b3a00640 1 --2- 172.21.15.90:0/2991185471 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f51ac109600 0x7f51ac1a6d40 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:55.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.322+0000 7f51b3a00640 1 --2- 172.21.15.90:0/2991185471 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f51ac108ce0 0x7f51ac1a6800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:55.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.322+0000 7f51b3a00640 1 -- 172.21.15.90:0/2991185471 >> 172.21.15.90:0/2991185471 conn(0x7f51ac06cbe0 msgr2=0x7f51ac070800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:37:55.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.322+0000 7f51b3a00640 1 -- 172.21.15.90:0/2991185471 shutdown_connections 2024-04-02T14:37:55.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.322+0000 7f51b3a00640 1 -- 172.21.15.90:0/2991185471 wait complete. 2024-04-02T14:37:55.350 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-02T14:37:55.646 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.643+0000 7f9aa1a00640 1 -- 172.21.15.90:0/2160777576 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9a9c108fe0 msgr2=0x7f9a9c110bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:55.646 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.643+0000 7f9aa1a00640 1 --2- 172.21.15.90:0/2160777576 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9a9c108fe0 0x7f9a9c110bb0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f9a8c009900 tx=0x7f9a8c02bc60 comp rx=0 tx=0).stop 2024-04-02T14:37:55.646 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.644+0000 7f9aa1a00640 1 -- 172.21.15.90:0/2160777576 shutdown_connections 2024-04-02T14:37:55.646 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.644+0000 7f9aa1a00640 1 --2- 172.21.15.90:0/2160777576 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9a9c108fe0 0x7f9a9c110bb0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:55.646 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.644+0000 7f9aa1a00640 1 --2- 172.21.15.90:0/2160777576 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9a9c1048c0 0x7f9a9c108900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:55.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.644+0000 7f9aa1a00640 1 --2- 172.21.15.90:0/2160777576 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9a9c103f10 0x7f9a9c1042f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:55.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.644+0000 7f9aa1a00640 1 -- 172.21.15.90:0/2160777576 >> 172.21.15.90:0/2160777576 conn(0x7f9a9c0ff7b0 msgr2=0x7f9a9c101bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:37:55.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.644+0000 7f9aa1a00640 1 -- 172.21.15.90:0/2160777576 shutdown_connections 2024-04-02T14:37:55.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.644+0000 7f9aa1a00640 1 -- 172.21.15.90:0/2160777576 wait complete. 2024-04-02T14:37:55.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.646+0000 7f9aa1a00640 1 Processor -- start 2024-04-02T14:37:55.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.647+0000 7f9aa1a00640 1 -- start start 2024-04-02T14:37:55.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.647+0000 7f9aa1a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9a9c103f10 0x7f9a9c076010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:55.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.647+0000 7f9aa1a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9a9c108fe0 0x7f9a9c076550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:55.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.647+0000 7f9aa1a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9a9c076a90 0x7f9a9c1aab30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:55.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.647+0000 7f9aa1a00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f9a9c114140 con 0x7f9a9c076a90 2024-04-02T14:37:55.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.647+0000 7f9aa1a00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f9a9c113fc0 con 0x7f9a9c103f10 2024-04-02T14:37:55.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.647+0000 7f9aa1a00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f9a9c1142c0 con 0x7f9a9c108fe0 2024-04-02T14:37:55.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.648+0000 7f9a9a000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9a9c108fe0 0x7f9a9c076550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:55.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.648+0000 7f9a9a000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9a9c108fe0 0x7f9a9c076550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:40366/0 (socket says 172.21.15.90:40366) 2024-04-02T14:37:55.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.648+0000 7f9a9a000640 1 -- 172.21.15.90:0/3233079441 learned_addr learned my addr 172.21.15.90:0/3233079441 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:37:55.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.648+0000 7f9a9aa00640 1 --2- 172.21.15.90:0/3233079441 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9a9c103f10 0x7f9a9c076010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:55.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.648+0000 7f9a9a000640 1 -- 172.21.15.90:0/3233079441 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9a9c103f10 msgr2=0x7f9a9c076010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:55.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.648+0000 7f9a9a000640 1 --2- 172.21.15.90:0/3233079441 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9a9c103f10 0x7f9a9c076010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:55.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.648+0000 7f9a9b400640 1 --2- 172.21.15.90:0/3233079441 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9a9c076a90 0x7f9a9c1aab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:55.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.648+0000 7f9a9a000640 1 -- 172.21.15.90:0/3233079441 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9a9c076a90 msgr2=0x7f9a9c1aab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:55.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.649+0000 7f9a9a000640 1 --2- 172.21.15.90:0/3233079441 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9a9c076a90 0x7f9a9c1aab30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:55.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.649+0000 7f9a9a000640 1 -- 172.21.15.90:0/3233079441 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a9c19a2d0 con 0x7f9a9c108fe0 2024-04-02T14:37:55.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.650+0000 7f9a9a000640 1 --2- 172.21.15.90:0/3233079441 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9a9c108fe0 0x7f9a9c076550 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f9a8800b4a0 tx=0x7f9a8800b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:37:55.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.651+0000 7f9a93400640 1 -- 172.21.15.90:0/3233079441 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a88014070 con 0x7f9a9c108fe0 2024-04-02T14:37:55.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.651+0000 7f9aa1a00640 1 -- 172.21.15.90:0/3233079441 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a9c19a560 con 0x7f9a9c108fe0 2024-04-02T14:37:55.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.651+0000 7f9a93400640 1 -- 172.21.15.90:0/3233079441 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9a88010af0 con 0x7f9a9c108fe0 2024-04-02T14:37:55.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.651+0000 7f9a93400640 1 -- 172.21.15.90:0/3233079441 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a88014070 con 0x7f9a9c108fe0 2024-04-02T14:37:55.658 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.656+0000 7f9aa1a00640 1 -- 172.21.15.90:0/3233079441 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a9c19aab0 con 0x7f9a9c108fe0 2024-04-02T14:37:55.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.659+0000 7f9a93400640 1 -- 172.21.15.90:0/3233079441 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f9a88017070 con 0x7f9a9c108fe0 2024-04-02T14:37:55.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.663+0000 7f9aa481d740 1 Processor -- start 2024-04-02T14:37:55.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.665+0000 7f9a93400640 1 --2- 172.21.15.90:0/3233079441 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f9a800a3fa0 0x7f9a800a6460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:55.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.665+0000 7f9a93400640 1 -- 172.21.15.90:0/3233079441 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f9a880d2170 con 0x7f9a9c108fe0 2024-04-02T14:37:55.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.666+0000 7f9a9b400640 1 --2- 172.21.15.90:0/3233079441 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f9a800a3fa0 0x7f9a800a6460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:55.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.666+0000 7f9aa481d740 1 -- start start 2024-04-02T14:37:55.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.666+0000 7f9aa481d740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556e3885f9f0 0x556e3885e740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:55.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.666+0000 7f9aa481d740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556e3885ed10 0x556e3885f190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:55.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.667+0000 7f9aa481d740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556e38868080 0x556e3886a470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:55.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.667+0000 7f9aa481d740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x556e3879f400 con 0x556e3885ed10 2024-04-02T14:37:55.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.667+0000 7f9a9aa00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556e3885f9f0 0x556e3885e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:55.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.667+0000 7f9aa481d740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x556e38783f10 con 0x556e3885f9f0 2024-04-02T14:37:55.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.667+0000 7f9aa481d740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x556e387c36f0 con 0x556e38868080 2024-04-02T14:37:55.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.667+0000 7f9a9aa00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556e3885f9f0 0x556e3885e740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:34316/0 (socket says 172.21.15.90:34316) 2024-04-02T14:37:55.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.667+0000 7f9a9aa00640 1 -- 172.21.15.90:0/3022958028 learned_addr learned my addr 172.21.15.90:0/3022958028 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:37:55.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.667+0000 7f9a9aa00640 1 -- 172.21.15.90:0/3022958028 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556e38868080 msgr2=0x556e3886a470 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-02T14:37:55.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.667+0000 7f9a9aa00640 1 --2- 172.21.15.90:0/3022958028 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556e38868080 0x556e3886a470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:55.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.667+0000 7f9a9aa00640 1 -- 172.21.15.90:0/3022958028 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556e3885ed10 msgr2=0x556e3885f190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:37:55.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.667+0000 7f9a9aa00640 1 --2- 172.21.15.90:0/3022958028 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556e3885ed10 0x556e3885f190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:55.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.667+0000 7f9a9aa00640 1 -- 172.21.15.90:0/3022958028 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556e384d73f0 con 0x556e3885f9f0 2024-04-02T14:37:55.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.668+0000 7f9a9aa00640 1 --2- 172.21.15.90:0/3022958028 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556e3885f9f0 0x556e3885e740 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f9a94007ed0 tx=0x7f9a9400e720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=465e71192b2dcb01 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:37:55.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.668+0000 7f9a87e00640 1 -- 172.21.15.90:0/3022958028 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a94014070 con 0x556e3885f9f0 2024-04-02T14:37:55.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.668+0000 7f9a87e00640 1 -- 172.21.15.90:0/3022958028 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9a94004d20 con 0x556e3885f9f0 2024-04-02T14:37:55.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.668+0000 7f9a87e00640 1 -- 172.21.15.90:0/3022958028 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a940054c0 con 0x556e3885f9f0 2024-04-02T14:37:55.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.669+0000 7f9aa481d740 1 -- 172.21.15.90:0/3022958028 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556e3885f9f0 msgr2=0x556e3885e740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:55.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.669+0000 7f9aa481d740 1 --2- 172.21.15.90:0/3022958028 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556e3885f9f0 0x556e3885e740 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f9a94007ed0 tx=0x7f9a9400e720 comp rx=0 tx=0).stop 2024-04-02T14:37:55.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.669+0000 7f9a9b400640 1 --2- 172.21.15.90:0/3233079441 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f9a800a3fa0 0x7f9a800a6460 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f9a8c02efd0 tx=0x7f9a8c005fd0 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:37:55.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.669+0000 7f9aa481d740 1 -- 172.21.15.90:0/3022958028 shutdown_connections 2024-04-02T14:37:55.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.669+0000 7f9aa481d740 1 --2- 172.21.15.90:0/3022958028 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556e38868080 0x556e3886a470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:55.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.669+0000 7f9aa481d740 1 --2- 172.21.15.90:0/3022958028 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556e3885ed10 0x556e3885f190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:55.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.669+0000 7f9aa481d740 1 --2- 172.21.15.90:0/3022958028 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556e3885f9f0 0x556e3885e740 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:55.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.669+0000 7f9aa481d740 1 -- 172.21.15.90:0/3022958028 >> 172.21.15.90:0/3022958028 conn(0x556e38825a50 msgr2=0x556e38859760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:37:55.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.670+0000 7f9aa481d740 1 -- 172.21.15.90:0/3022958028 shutdown_connections 2024-04-02T14:37:55.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.670+0000 7f9aa481d740 1 -- 172.21.15.90:0/3022958028 wait complete. 2024-04-02T14:37:55.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.671+0000 7f9aa481d740 1 -- start start 2024-04-02T14:37:55.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.671+0000 7f9aa481d740 1 Processor -- start 2024-04-02T14:37:55.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.671+0000 7f9aa481d740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556e3885ed10 0x556e38865130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:55.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.671+0000 7f9aa481d740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556e3885f9f0 0x556e38865670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:55.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.672+0000 7f9aa481d740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556e38868080 0x556e38865bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:55.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.672+0000 7f9a9b400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556e38868080 0x556e38865bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:55.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.672+0000 7f9a9b400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556e38868080 0x556e38865bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:34322/0 (socket says 172.21.15.90:34322) 2024-04-02T14:37:55.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.672+0000 7f9aa481d740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x556e387bf6a0 con 0x556e3885ed10 2024-04-02T14:37:55.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.672+0000 7f9a9aa00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556e3885ed10 0x556e38865130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:55.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.672+0000 7f9a9a000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556e3885f9f0 0x556e38865670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:55.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.672+0000 7f9a9b400640 1 -- 172.21.15.90:0/3568556063 learned_addr learned my addr 172.21.15.90:0/3568556063 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:37:55.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.672+0000 7f9aa481d740 1 -- 172.21.15.90:0/3568556063 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x556e3876b980 con 0x556e38868080 2024-04-02T14:37:55.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.672+0000 7f9aa481d740 1 -- 172.21.15.90:0/3568556063 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x556e3877ecb0 con 0x556e3885f9f0 2024-04-02T14:37:55.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.673+0000 7f9aa481d740 10 client.0 _ll_register_callbacks cb 0x556e3877e6a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:37:55.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.673+0000 7f9a9aa00640 1 -- 172.21.15.90:0/3568556063 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556e3885f9f0 msgr2=0x556e38865670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:37:55.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.673+0000 7f9a9aa00640 1 --2- 172.21.15.90:0/3568556063 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556e3885f9f0 0x556e38865670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:55.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.673+0000 7f9a9aa00640 1 -- 172.21.15.90:0/3568556063 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556e38868080 msgr2=0x556e38865bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:37:55.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.673+0000 7f9a9aa00640 1 --2- 172.21.15.90:0/3568556063 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556e38868080 0x556e38865bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:55.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.673+0000 7f9a9aa00640 1 -- 172.21.15.90:0/3568556063 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556e385a17d0 con 0x556e3885ed10 2024-04-02T14:37:55.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.674+0000 7f9a9aa00640 1 --2- 172.21.15.90:0/3568556063 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556e3885ed10 0x556e38865130 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f9a94000f80 tx=0x7f9a94007c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=929a8a3875693fc1 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:37:55.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.674+0000 7f9a84c00640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-02T14:37:55.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.674+0000 7f9a84c00640 1 -- 172.21.15.90:0/3568556063 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a94014070 con 0x556e3885ed10 2024-04-02T14:37:55.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.674+0000 7f9a84c00640 1 -- 172.21.15.90:0/3568556063 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f9a94003b70 con 0x556e3885ed10 2024-04-02T14:37:55.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.674+0000 7f9aa481d740 1 -- 172.21.15.90:0/3568556063 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9a94004d20 con 0x556e3885ed10 2024-04-02T14:37:55.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.675+0000 7f9a84c00640 1 -- 172.21.15.90:0/3568556063 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f9a94003d90 con 0x556e3885ed10 2024-04-02T14:37:55.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.675+0000 7f9aa481d740 10 client.15957 fetch_fsmap learned FSMap version 27 2024-04-02T14:37:55.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.675+0000 7f9aa481d740 1 -- 172.21.15.90:0/3568556063 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x556e38877010 con 0x556e3885ed10 2024-04-02T14:37:55.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.676+0000 7f9a84c00640 1 -- 172.21.15.90:0/3568556063 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9a94004d20 con 0x556e3885ed10 2024-04-02T14:37:55.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.676+0000 7f9a84c00640 1 -- 172.21.15.90:0/3568556063 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f9a940041f0 con 0x556e3885ed10 2024-04-02T14:37:55.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.676+0000 7f9aa481d740 10 client.15957 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:37:55.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.676+0000 7f9aa481d740 10 client.15957 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2024-04-02T14:37:55.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.676+0000 7f9aa481d740 20 client.15957 populate_metadata read hostname 'smithi090' 2024-04-02T14:37:55.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.676+0000 7f9aa481d740 1 --2- 172.21.15.90:0/3568556063 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x556e38878360 0x556e3887a750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:55.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.676+0000 7f9aa481d740 4 client.15957 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:37:55.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.676+0000 7f9aa481d740 1 -- 172.21.15.90:0/3568556063 --> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556e38783040 con 0x556e38878360 2024-04-02T14:37:55.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.677+0000 7f9a9a000640 1 --2- 172.21.15.90:0/3568556063 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x556e38878360 0x556e3887a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:55.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.678+0000 7f9a84c00640 10 client.15957 ms_handle_connect on v2:172.21.15.90:6838/1498915870 2024-04-02T14:37:55.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.678+0000 7f9a9a000640 1 --2- 172.21.15.90:0/3568556063 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x556e38878360 0x556e3887a750 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=3cc973e6a88b776d server_cookie=b1bf61ea507010cd in_seq=0 out_seq=0 2024-04-02T14:37:55.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.679+0000 7f9a84c00640 1 -- 172.21.15.90:0/3568556063 <== mds.1 v2:172.21.15.90:6838/1498915870 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x556e38783040 con 0x556e38878360 2024-04-02T14:37:55.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.679+0000 7f9a84c00640 10 client.15957 handle_command_reply: tid=0 2024-04-02T14:37:55.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.680+0000 7f9aa481d740 1 client.15957 shutdown 2024-04-02T14:37:55.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.680+0000 7f9aa481d740 1 -- 172.21.15.90:0/3568556063 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556e3885ed10 msgr2=0x556e38865130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:55.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.680+0000 7f9aa481d740 1 --2- 172.21.15.90:0/3568556063 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556e3885ed10 0x556e38865130 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f9a94000f80 tx=0x7f9a94007c10 comp rx=0 tx=0).stop 2024-04-02T14:37:55.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.680+0000 7f9aa481d740 1 -- 172.21.15.90:0/3568556063 shutdown_connections 2024-04-02T14:37:55.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.680+0000 7f9aa481d740 1 --2- 172.21.15.90:0/3568556063 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x556e38878360 0x556e3887a750 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:55.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.680+0000 7f9aa481d740 1 --2- 172.21.15.90:0/3568556063 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556e38868080 0x556e38865bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:55.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.680+0000 7f9aa481d740 1 --2- 172.21.15.90:0/3568556063 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556e3885f9f0 0x556e38865670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:55.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.680+0000 7f9aa481d740 1 --2- 172.21.15.90:0/3568556063 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556e3885ed10 0x556e38865130 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:55.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.680+0000 7f9aa481d740 1 -- 172.21.15.90:0/3568556063 >> 172.21.15.90:0/3568556063 conn(0x556e38825a50 msgr2=0x556e3885b4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:37:55.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.680+0000 7f9a84c00640 0 client.15957 ms_handle_reset on v2:172.21.15.90:6838/1498915870 2024-04-02T14:37:55.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.680+0000 7f9aa481d740 1 -- 172.21.15.90:0/3568556063 shutdown_connections 2024-04-02T14:37:55.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.681+0000 7f9aa481d740 1 -- 172.21.15.90:0/3568556063 wait complete. 2024-04-02T14:37:55.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.681+0000 7f9aa481d740 20 client.15957 trim_cache size 0 max 16384 2024-04-02T14:37:55.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.720+0000 7f9aa481d740 1 Processor -- start 2024-04-02T14:37:55.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.721+0000 7f9aa481d740 1 -- start start 2024-04-02T14:37:55.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.721+0000 7f9aa481d740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556e3885ed10 0x556e38867860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:55.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.721+0000 7f9aa481d740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556e3885f9f0 0x556e3885b680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:55.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.721+0000 7f9aa481d740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556e38868080 0x556e38864a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:55.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.721+0000 7f9aa481d740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f9a9c1bc830 con 0x556e38868080 2024-04-02T14:37:55.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.721+0000 7f9aa481d740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f9a9c1be370 con 0x556e3885f9f0 2024-04-02T14:37:55.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.721+0000 7f9aa481d740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f9a9c1bcf00 con 0x556e3885ed10 2024-04-02T14:37:55.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.722+0000 7f9a9b400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556e38868080 0x556e38864a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:55.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.722+0000 7f9a9b400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556e38868080 0x556e38864a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:54468/0 (socket says 172.21.15.90:54468) 2024-04-02T14:37:55.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.722+0000 7f9a9b400640 1 -- 172.21.15.90:0/1046551097 learned_addr learned my addr 172.21.15.90:0/1046551097 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:37:55.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.722+0000 7f9a9aa00640 1 --2- 172.21.15.90:0/1046551097 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556e3885ed10 0x556e38867860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:55.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.722+0000 7f9a9a000640 1 --2- 172.21.15.90:0/1046551097 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556e3885f9f0 0x556e3885b680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:55.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.723+0000 7f9a9b400640 1 -- 172.21.15.90:0/1046551097 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556e3885ed10 msgr2=0x556e38867860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:37:55.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.723+0000 7f9a9b400640 1 --2- 172.21.15.90:0/1046551097 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556e3885ed10 0x556e38867860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:55.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.723+0000 7f9a9b400640 1 -- 172.21.15.90:0/1046551097 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556e3885f9f0 msgr2=0x556e3885b680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:37:55.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.723+0000 7f9a9b400640 1 --2- 172.21.15.90:0/1046551097 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556e3885f9f0 0x556e3885b680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:55.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.723+0000 7f9a9b400640 1 -- 172.21.15.90:0/1046551097 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556e3885a900 con 0x556e38868080 2024-04-02T14:37:55.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.723+0000 7f9a9aa00640 1 --2- 172.21.15.90:0/1046551097 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556e3885ed10 0x556e38867860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:37:55.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.723+0000 7f9a9b400640 1 --2- 172.21.15.90:0/1046551097 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556e38868080 0x556e38864a10 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f9a8c036040 tx=0x7f9a8c03c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fbd774ae0d0098c2 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:37:55.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.724+0000 7f9a87400640 1 -- 172.21.15.90:0/1046551097 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a8c04e030 con 0x556e38868080 2024-04-02T14:37:55.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.724+0000 7f9a87400640 1 -- 172.21.15.90:0/1046551097 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9a8c005520 con 0x556e38868080 2024-04-02T14:37:55.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.724+0000 7f9a87400640 1 -- 172.21.15.90:0/1046551097 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a8c0035f0 con 0x556e38868080 2024-04-02T14:37:55.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.725+0000 7f9aa481d740 1 -- 172.21.15.90:0/1046551097 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556e38868080 msgr2=0x556e38864a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:55.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.725+0000 7f9aa481d740 1 --2- 172.21.15.90:0/1046551097 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556e38868080 0x556e38864a10 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f9a8c036040 tx=0x7f9a8c03c000 comp rx=0 tx=0).stop 2024-04-02T14:37:55.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.725+0000 7f9aa481d740 1 -- 172.21.15.90:0/1046551097 shutdown_connections 2024-04-02T14:37:55.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.725+0000 7f9aa481d740 1 --2- 172.21.15.90:0/1046551097 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556e38868080 0x556e38864a10 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:55.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.725+0000 7f9aa481d740 1 --2- 172.21.15.90:0/1046551097 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556e3885f9f0 0x556e3885b680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:55.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.725+0000 7f9aa481d740 1 --2- 172.21.15.90:0/1046551097 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556e3885ed10 0x556e38867860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:55.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.725+0000 7f9aa481d740 1 -- 172.21.15.90:0/1046551097 >> 172.21.15.90:0/1046551097 conn(0x556e38825a50 msgr2=0x556e388738c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:37:55.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.726+0000 7f9aa481d740 1 -- 172.21.15.90:0/1046551097 shutdown_connections 2024-04-02T14:37:55.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.726+0000 7f9aa481d740 1 -- 172.21.15.90:0/1046551097 wait complete. 2024-04-02T14:37:55.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.726+0000 7f9aa481d740 1 -- start start 2024-04-02T14:37:55.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.726+0000 7f9aa481d740 1 Processor -- start 2024-04-02T14:37:55.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.727+0000 7f9aa481d740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556e3885ed10 0x556e3887dac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:55.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.727+0000 7f9aa481d740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556e3885f9f0 0x556e38878e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:55.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.727+0000 7f9aa481d740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556e38868080 0x556e38879370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:55.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.727+0000 7f9aa481d740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f9a9c1bdca0 con 0x556e38868080 2024-04-02T14:37:55.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.727+0000 7f9aa481d740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f9a9c1bd5d0 con 0x556e3885f9f0 2024-04-02T14:37:55.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.727+0000 7f9aa481d740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f9a9c1a1270 con 0x556e3885ed10 2024-04-02T14:37:55.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.727+0000 7f9aa481d740 10 client.0 _ll_register_callbacks cb 0x7f9a94010b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:37:55.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.728+0000 7f9a9a000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556e3885f9f0 0x556e38878e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:55.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.728+0000 7f9a9a000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556e3885f9f0 0x556e38878e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:34336/0 (socket says 172.21.15.90:34336) 2024-04-02T14:37:55.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.728+0000 7f9a9b400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556e38868080 0x556e38879370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:55.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.728+0000 7f9a9aa00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556e3885ed10 0x556e3887dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:55.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.728+0000 7f9a9aa00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556e3885ed10 0x556e3887dac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:40392/0 (socket says 172.21.15.90:40392) 2024-04-02T14:37:55.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.728+0000 7f9a9aa00640 1 -- 172.21.15.90:0/566454573 learned_addr learned my addr 172.21.15.90:0/566454573 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:37:55.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.729+0000 7f9a9b400640 1 -- 172.21.15.90:0/566454573 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556e3885ed10 msgr2=0x556e3887dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:37:55.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.729+0000 7f9a9b400640 1 --2- 172.21.15.90:0/566454573 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556e3885ed10 0x556e3887dac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:55.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.729+0000 7f9a9b400640 1 -- 172.21.15.90:0/566454573 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556e3885f9f0 msgr2=0x556e38878e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:37:55.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.729+0000 7f9a9b400640 1 --2- 172.21.15.90:0/566454573 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556e3885f9f0 0x556e38878e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:55.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.729+0000 7f9a9b400640 1 -- 172.21.15.90:0/566454573 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556e38879b20 con 0x556e38868080 2024-04-02T14:37:55.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.729+0000 7f9a9b400640 1 --2- 172.21.15.90:0/566454573 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556e38868080 0x556e38879370 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f9a8c009900 tx=0x7f9a8c0045e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf0da54ad58a8d8e server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:37:55.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.730+0000 7f9a84c00640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-02T14:37:55.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.730+0000 7f9a84c00640 1 -- 172.21.15.90:0/566454573 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a8c04e040 con 0x556e38868080 2024-04-02T14:37:55.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.730+0000 7f9a84c00640 1 -- 172.21.15.90:0/566454573 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f9a8c005220 con 0x556e38868080 2024-04-02T14:37:55.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.730+0000 7f9aa481d740 1 -- 172.21.15.90:0/566454573 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9a8c005520 con 0x556e38868080 2024-04-02T14:37:55.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.731+0000 7f9a84c00640 1 -- 172.21.15.90:0/566454573 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f9a8c0035f0 con 0x556e38868080 2024-04-02T14:37:55.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.731+0000 7f9aa481d740 10 client.15963 fetch_fsmap learned FSMap version 27 2024-04-02T14:37:55.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.731+0000 7f9aa481d740 1 -- 172.21.15.90:0/566454573 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x556e388de4b0 con 0x556e38868080 2024-04-02T14:37:55.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.732+0000 7f9a84c00640 1 -- 172.21.15.90:0/566454573 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9a8c005520 con 0x556e38868080 2024-04-02T14:37:55.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.734+0000 7f9a84c00640 1 -- 172.21.15.90:0/566454573 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f9a8c03f4f0 con 0x556e38868080 2024-04-02T14:37:55.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.734+0000 7f9aa481d740 10 client.15963 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:37:55.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.734+0000 7f9aa481d740 10 client.15963 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2024-04-02T14:37:55.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.734+0000 7f9aa481d740 20 client.15963 populate_metadata read hostname 'smithi090' 2024-04-02T14:37:55.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.734+0000 7f9aa481d740 1 --2- 172.21.15.90:0/566454573 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x556e388deb70 0x556e388e0f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:55.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.734+0000 7f9aa481d740 4 client.15963 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-02T14:37:55.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.734+0000 7f9aa481d740 1 -- 172.21.15.90:0/566454573 --> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] -- command(tid 0: {"prefix": "damage ls"}) -- 0x556e387d0c30 con 0x556e388deb70 2024-04-02T14:37:55.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.735+0000 7f9a9aa00640 1 --2- 172.21.15.90:0/566454573 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x556e388deb70 0x556e388e0f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:55.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.736+0000 7f9a9aa00640 1 --2- 172.21.15.90:0/566454573 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x556e388deb70 0x556e388e0f60 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=8ff4beeb5a7411c8 server_cookie=78d25a3f90fe5908 in_seq=0 out_seq=0 2024-04-02T14:37:55.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.736+0000 7f9a84c00640 10 client.15963 ms_handle_connect on v2:172.21.15.90:6838/1498915870 2024-04-02T14:37:55.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.736+0000 7f9a84c00640 1 -- 172.21.15.90:0/566454573 <== mds.1 v2:172.21.15.90:6838/1498915870 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x556e387d0c30 con 0x556e388deb70 2024-04-02T14:37:55.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.736+0000 7f9a84c00640 10 client.15963 handle_command_reply: tid=0 2024-04-02T14:37:55.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.736+0000 7f9aa481d740 1 client.15963 shutdown 2024-04-02T14:37:55.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.736+0000 7f9aa481d740 1 -- 172.21.15.90:0/566454573 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556e38868080 msgr2=0x556e38879370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:55.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.736+0000 7f9aa481d740 1 --2- 172.21.15.90:0/566454573 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556e38868080 0x556e38879370 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f9a8c009900 tx=0x7f9a8c0045e0 comp rx=0 tx=0).stop 2024-04-02T14:37:55.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.737+0000 7f9aa481d740 1 -- 172.21.15.90:0/566454573 shutdown_connections 2024-04-02T14:37:55.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.737+0000 7f9aa481d740 1 --2- 172.21.15.90:0/566454573 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x556e388deb70 0x556e388e0f60 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:55.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.737+0000 7f9aa481d740 1 --2- 172.21.15.90:0/566454573 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556e38868080 0x556e38879370 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:55.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.737+0000 7f9aa481d740 1 --2- 172.21.15.90:0/566454573 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556e3885f9f0 0x556e38878e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:55.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.737+0000 7f9aa481d740 1 --2- 172.21.15.90:0/566454573 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556e3885ed10 0x556e3887dac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:55.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.737+0000 7f9a84c00640 0 client.15963 ms_handle_reset on v2:172.21.15.90:6838/1498915870 2024-04-02T14:37:55.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.737+0000 7f9aa481d740 1 -- 172.21.15.90:0/566454573 >> 172.21.15.90:0/566454573 conn(0x556e38825a50 msgr2=0x556e38858010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:37:55.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.737+0000 7f9aa481d740 1 -- 172.21.15.90:0/566454573 shutdown_connections 2024-04-02T14:37:55.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.737+0000 7f9aa481d740 1 -- 172.21.15.90:0/566454573 wait complete. 2024-04-02T14:37:55.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.737+0000 7f9aa481d740 20 client.15963 trim_cache size 0 max 16384 2024-04-02T14:37:55.740 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-02T14:37:55.741 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.739+0000 7f9aa1a00640 1 -- 172.21.15.90:0/3233079441 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f9a800a3fa0 msgr2=0x7f9a800a6460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:55.741 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.739+0000 7f9aa1a00640 1 --2- 172.21.15.90:0/3233079441 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f9a800a3fa0 0x7f9a800a6460 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f9a8c02efd0 tx=0x7f9a8c005fd0 comp rx=0 tx=0).stop 2024-04-02T14:37:55.741 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.739+0000 7f9aa1a00640 1 -- 172.21.15.90:0/3233079441 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9a9c108fe0 msgr2=0x7f9a9c076550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:55.741 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.739+0000 7f9aa1a00640 1 --2- 172.21.15.90:0/3233079441 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9a9c108fe0 0x7f9a9c076550 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f9a8800b4a0 tx=0x7f9a8800b960 comp rx=0 tx=0).stop 2024-04-02T14:37:55.741 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.739+0000 7f9aa1a00640 1 -- 172.21.15.90:0/3233079441 shutdown_connections 2024-04-02T14:37:55.741 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.739+0000 7f9aa1a00640 1 --2- 172.21.15.90:0/3233079441 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f9a800a3fa0 0x7f9a800a6460 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:55.741 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.739+0000 7f9aa1a00640 1 --2- 172.21.15.90:0/3233079441 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9a9c076a90 0x7f9a9c1aab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:55.741 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.739+0000 7f9aa1a00640 1 --2- 172.21.15.90:0/3233079441 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9a9c108fe0 0x7f9a9c076550 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:55.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.739+0000 7f9aa1a00640 1 --2- 172.21.15.90:0/3233079441 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9a9c103f10 0x7f9a9c076010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:55.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.739+0000 7f9aa1a00640 1 -- 172.21.15.90:0/3233079441 >> 172.21.15.90:0/3233079441 conn(0x7f9a9c0ff7b0 msgr2=0x7f9a9c1002f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:37:55.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.740+0000 7f9aa1a00640 1 -- 172.21.15.90:0/3233079441 shutdown_connections 2024-04-02T14:37:55.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:55.740+0000 7f9aa1a00640 1 -- 172.21.15.90:0/3233079441 wait complete. 2024-04-02T14:37:55.771 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-02T14:37:55.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:37:55 smithi177 ceph-mon[36431]: pgmap v94: 97 pgs: 97 active+clean; 1.7 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 11 MiB/s wr, 942 op/s 2024-04-02T14:37:55.938 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:37:55 smithi090 ceph-mon[31674]: pgmap v94: 97 pgs: 97 active+clean; 1.7 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 11 MiB/s wr, 942 op/s 2024-04-02T14:37:56.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.067+0000 7f5c31200640 1 -- 172.21.15.90:0/4231043942 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5c2c108bb0 msgr2=0x7f5c2c108f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:56.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.067+0000 7f5c31200640 1 --2- 172.21.15.90:0/4231043942 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5c2c108bb0 0x7f5c2c108f90 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f5c1c009860 tx=0x7f5c1c02bb70 comp rx=0 tx=0).stop 2024-04-02T14:37:56.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.067+0000 7f5c31200640 1 -- 172.21.15.90:0/4231043942 shutdown_connections 2024-04-02T14:37:56.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.067+0000 7f5c31200640 1 --2- 172.21.15.90:0/4231043942 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5c2c10d3f0 0x7f5c2c114fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:56.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.067+0000 7f5c31200640 1 --2- 172.21.15.90:0/4231043942 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5c2c109560 0x7f5c2c10cd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:56.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.067+0000 7f5c31200640 1 --2- 172.21.15.90:0/4231043942 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5c2c108bb0 0x7f5c2c108f90 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:56.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.067+0000 7f5c31200640 1 -- 172.21.15.90:0/4231043942 >> 172.21.15.90:0/4231043942 conn(0x7f5c2c0776c0 msgr2=0x7f5c2c079ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:37:56.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.067+0000 7f5c31200640 1 -- 172.21.15.90:0/4231043942 shutdown_connections 2024-04-02T14:37:56.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.067+0000 7f5c31200640 1 -- 172.21.15.90:0/4231043942 wait complete. 2024-04-02T14:37:56.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.069+0000 7f5c31200640 1 Processor -- start 2024-04-02T14:37:56.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.069+0000 7f5c31200640 1 -- start start 2024-04-02T14:37:56.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.070+0000 7f5c31200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5c2c108bb0 0x7f5c2c1a4760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:56.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.070+0000 7f5c31200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5c2c10d3f0 0x7f5c2c1a4ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:56.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.070+0000 7f5c31200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5c2c19e960 0x7f5c2c19ee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:56.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.070+0000 7f5c31200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f5c2c118500 con 0x7f5c2c10d3f0 2024-04-02T14:37:56.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.070+0000 7f5c31200640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f5c2c118380 con 0x7f5c2c19e960 2024-04-02T14:37:56.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.070+0000 7f5c31200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f5c2c118680 con 0x7f5c2c108bb0 2024-04-02T14:37:56.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.070+0000 7f5c29e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5c2c108bb0 0x7f5c2c1a4760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:56.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.070+0000 7f5c2a800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5c2c19e960 0x7f5c2c19ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:56.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.070+0000 7f5c2a800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5c2c19e960 0x7f5c2c19ee10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:34354/0 (socket says 172.21.15.90:34354) 2024-04-02T14:37:56.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.070+0000 7f5c2a800640 1 -- 172.21.15.90:0/4281722025 learned_addr learned my addr 172.21.15.90:0/4281722025 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:37:56.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.071+0000 7f5c29400640 1 --2- 172.21.15.90:0/4281722025 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5c2c10d3f0 0x7f5c2c1a4ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:56.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.071+0000 7f5c29e00640 1 -- 172.21.15.90:0/4281722025 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5c2c19e960 msgr2=0x7f5c2c19ee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:56.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.071+0000 7f5c29e00640 1 --2- 172.21.15.90:0/4281722025 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5c2c19e960 0x7f5c2c19ee10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:56.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.071+0000 7f5c29e00640 1 -- 172.21.15.90:0/4281722025 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5c2c10d3f0 msgr2=0x7f5c2c1a4ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:56.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.071+0000 7f5c29e00640 1 --2- 172.21.15.90:0/4281722025 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5c2c10d3f0 0x7f5c2c1a4ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:56.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.071+0000 7f5c29e00640 1 -- 172.21.15.90:0/4281722025 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c2c19f480 con 0x7f5c2c108bb0 2024-04-02T14:37:56.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.072+0000 7f5c29e00640 1 --2- 172.21.15.90:0/4281722025 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5c2c108bb0 0x7f5c2c1a4760 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f5c1c02c0b0 tx=0x7f5c1c0323b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:37:56.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.072+0000 7f5c22a00640 1 -- 172.21.15.90:0/4281722025 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c1c041020 con 0x7f5c2c108bb0 2024-04-02T14:37:56.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.072+0000 7f5c22a00640 1 -- 172.21.15.90:0/4281722025 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5c1c02c9f0 con 0x7f5c2c108bb0 2024-04-02T14:37:56.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.072+0000 7f5c22a00640 1 -- 172.21.15.90:0/4281722025 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c1c039040 con 0x7f5c2c108bb0 2024-04-02T14:37:56.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.072+0000 7f5c31200640 1 -- 172.21.15.90:0/4281722025 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5c2c19f710 con 0x7f5c2c108bb0 2024-04-02T14:37:56.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.072+0000 7f5c31200640 1 -- 172.21.15.90:0/4281722025 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5c2c1ab860 con 0x7f5c2c108bb0 2024-04-02T14:37:56.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.076+0000 7f5c22a00640 1 -- 172.21.15.90:0/4281722025 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f5c1c044020 con 0x7f5c2c108bb0 2024-04-02T14:37:56.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.079+0000 7f5c33e7c740 1 Processor -- start 2024-04-02T14:37:56.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.079+0000 7f5c22a00640 1 --2- 172.21.15.90:0/4281722025 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f5c100a3e70 0x7f5c100a6330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:56.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.079+0000 7f5c22a00640 1 -- 172.21.15.90:0/4281722025 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f5c1c0f1e50 con 0x7f5c2c108bb0 2024-04-02T14:37:56.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.079+0000 7f5c33e7c740 1 -- start start 2024-04-02T14:37:56.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.080+0000 7f5c2a800640 1 --2- 172.21.15.90:0/4281722025 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f5c100a3e70 0x7f5c100a6330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:56.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.080+0000 7f5c33e7c740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c5e76959f0 0x55c5e7694740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:56.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.080+0000 7f5c33e7c740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c5e7694d10 0x55c5e7695190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:56.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.080+0000 7f5c33e7c740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c5e769e080 0x55c5e76a0470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:56.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.080+0000 7f5c33e7c740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55c5e75d5400 con 0x55c5e769e080 2024-04-02T14:37:56.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.080+0000 7f5c33e7c740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55c5e75b9f10 con 0x55c5e76959f0 2024-04-02T14:37:56.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.080+0000 7f5c33e7c740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55c5e75f96f0 con 0x55c5e7694d10 2024-04-02T14:37:56.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.080+0000 7f5c29400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c5e7694d10 0x55c5e7695190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:56.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.080+0000 7f5c29400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c5e7694d10 0x55c5e7695190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:40412/0 (socket says 172.21.15.90:40412) 2024-04-02T14:37:56.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.080+0000 7f5c29400640 1 -- 172.21.15.90:0/73386368 learned_addr learned my addr 172.21.15.90:0/73386368 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:37:56.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.081+0000 7f5c29400640 1 -- 172.21.15.90:0/73386368 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c5e76959f0 msgr2=0x55c5e7694740 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-02T14:37:56.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.081+0000 7f5c29e00640 1 --2- 172.21.15.90:0/73386368 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c5e76959f0 0x55c5e7694740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:56.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.081+0000 7f5c2a800640 1 --2- 172.21.15.90:0/4281722025 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f5c100a3e70 0x7f5c100a6330 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f5c24009fd0 tx=0x7f5c2400b040 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:37:56.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.081+0000 7f5c2a800640 1 --2- 172.21.15.90:0/73386368 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c5e769e080 0x55c5e76a0470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:56.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.081+0000 7f5c29400640 1 --2- 172.21.15.90:0/73386368 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c5e76959f0 0x55c5e7694740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:56.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.082+0000 7f5c29400640 1 -- 172.21.15.90:0/73386368 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c5e769e080 msgr2=0x55c5e76a0470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:37:56.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.082+0000 7f5c29400640 1 --2- 172.21.15.90:0/73386368 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c5e769e080 0x55c5e76a0470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:56.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.082+0000 7f5c29400640 1 -- 172.21.15.90:0/73386368 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c5e730d3f0 con 0x55c5e7694d10 2024-04-02T14:37:56.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.082+0000 7f5c2a800640 1 --2- 172.21.15.90:0/73386368 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c5e769e080 0x55c5e76a0470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:37:56.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.082+0000 7f5c29400640 1 --2- 172.21.15.90:0/73386368 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c5e7694d10 0x55c5e7695190 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f5c18009c80 tx=0x7f5c180117b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=194413e523dc8c69 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:37:56.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.082+0000 7f5c29e00640 1 --2- 172.21.15.90:0/73386368 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c5e76959f0 0x55c5e7694740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:37:56.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.083+0000 7f5c17400640 1 -- 172.21.15.90:0/73386368 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c18012450 con 0x55c5e7694d10 2024-04-02T14:37:56.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.083+0000 7f5c17400640 1 -- 172.21.15.90:0/73386368 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5c18012ad0 con 0x55c5e7694d10 2024-04-02T14:37:56.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.083+0000 7f5c17400640 1 -- 172.21.15.90:0/73386368 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c1800fc40 con 0x55c5e7694d10 2024-04-02T14:37:56.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.084+0000 7f5c33e7c740 1 -- 172.21.15.90:0/73386368 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c5e7694d10 msgr2=0x55c5e7695190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:56.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.084+0000 7f5c33e7c740 1 --2- 172.21.15.90:0/73386368 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c5e7694d10 0x55c5e7695190 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f5c18009c80 tx=0x7f5c180117b0 comp rx=0 tx=0).stop 2024-04-02T14:37:56.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.084+0000 7f5c33e7c740 1 -- 172.21.15.90:0/73386368 shutdown_connections 2024-04-02T14:37:56.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.084+0000 7f5c33e7c740 1 --2- 172.21.15.90:0/73386368 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c5e769e080 0x55c5e76a0470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:56.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.084+0000 7f5c33e7c740 1 --2- 172.21.15.90:0/73386368 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c5e7694d10 0x55c5e7695190 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:56.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.084+0000 7f5c33e7c740 1 --2- 172.21.15.90:0/73386368 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c5e76959f0 0x55c5e7694740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:56.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.084+0000 7f5c33e7c740 1 -- 172.21.15.90:0/73386368 >> 172.21.15.90:0/73386368 conn(0x55c5e765ba50 msgr2=0x55c5e768f760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:37:56.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.084+0000 7f5c33e7c740 1 -- 172.21.15.90:0/73386368 shutdown_connections 2024-04-02T14:37:56.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.084+0000 7f5c33e7c740 1 -- 172.21.15.90:0/73386368 wait complete. 2024-04-02T14:37:56.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.085+0000 7f5c33e7c740 1 -- start start 2024-04-02T14:37:56.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.085+0000 7f5c33e7c740 1 Processor -- start 2024-04-02T14:37:56.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.086+0000 7f5c33e7c740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c5e7694d10 0x55c5e769b070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:56.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.086+0000 7f5c33e7c740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c5e76959f0 0x55c5e769b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:56.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.086+0000 7f5c33e7c740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c5e769e080 0x55c5e769baf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:56.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.086+0000 7f5c29400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c5e76959f0 0x55c5e769b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:56.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.086+0000 7f5c29400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c5e76959f0 0x55c5e769b5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:34386/0 (socket says 172.21.15.90:34386) 2024-04-02T14:37:56.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.087+0000 7f5c2a800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c5e769e080 0x55c5e769baf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:56.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.087+0000 7f5c2a800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c5e769e080 0x55c5e769baf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:40416/0 (socket says 172.21.15.90:40416) 2024-04-02T14:37:56.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.087+0000 7f5c29e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c5e7694d10 0x55c5e769b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:56.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.087+0000 7f5c29e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c5e7694d10 0x55c5e769b070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:54524/0 (socket says 172.21.15.90:54524) 2024-04-02T14:37:56.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.087+0000 7f5c29400640 1 -- 172.21.15.90:0/2328584159 learned_addr learned my addr 172.21.15.90:0/2328584159 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:37:56.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.087+0000 7f5c33e7c740 1 -- 172.21.15.90:0/2328584159 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55c5e75f56a0 con 0x55c5e7694d10 2024-04-02T14:37:56.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.087+0000 7f5c33e7c740 1 -- 172.21.15.90:0/2328584159 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55c5e75a1980 con 0x55c5e76959f0 2024-04-02T14:37:56.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.087+0000 7f5c33e7c740 1 -- 172.21.15.90:0/2328584159 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55c5e75b4cb0 con 0x55c5e769e080 2024-04-02T14:37:56.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.087+0000 7f5c33e7c740 10 client.0 _ll_register_callbacks cb 0x55c5e75b46a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:37:56.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.088+0000 7f5c2a800640 1 -- 172.21.15.90:0/2328584159 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c5e76959f0 msgr2=0x55c5e769b5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:37:56.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.088+0000 7f5c2a800640 1 --2- 172.21.15.90:0/2328584159 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c5e76959f0 0x55c5e769b5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:56.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.088+0000 7f5c2a800640 1 -- 172.21.15.90:0/2328584159 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c5e7694d10 msgr2=0x55c5e769b070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:37:56.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.088+0000 7f5c2a800640 1 --2- 172.21.15.90:0/2328584159 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c5e7694d10 0x55c5e769b070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:56.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.088+0000 7f5c2a800640 1 -- 172.21.15.90:0/2328584159 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c5e73d77d0 con 0x55c5e769e080 2024-04-02T14:37:56.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.088+0000 7f5c0be00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:37:56.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.088+0000 7f5c2a800640 1 --2- 172.21.15.90:0/2328584159 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c5e769e080 0x55c5e769baf0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f5c24015be0 tx=0x7f5c24004b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b237d80917b3d199 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:37:56.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.089+0000 7f5c0be00640 1 -- 172.21.15.90:0/2328584159 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c24024020 con 0x55c5e769e080 2024-04-02T14:37:56.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.089+0000 7f5c33e7c740 1 -- 172.21.15.90:0/2328584159 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5c18012ad0 con 0x55c5e769e080 2024-04-02T14:37:56.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.089+0000 7f5c0be00640 1 -- 172.21.15.90:0/2328584159 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f5c2401b050 con 0x55c5e769e080 2024-04-02T14:37:56.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.089+0000 7f5c0be00640 1 -- 172.21.15.90:0/2328584159 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f5c24024020 con 0x55c5e769e080 2024-04-02T14:37:56.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.090+0000 7f5c33e7c740 10 client.26266 fetch_fsmap learned FSMap version 27 2024-04-02T14:37:56.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.090+0000 7f5c33e7c740 1 -- 172.21.15.90:0/2328584159 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55c5e76acf40 con 0x55c5e769e080 2024-04-02T14:37:56.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.090+0000 7f5c0be00640 1 -- 172.21.15.90:0/2328584159 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5c18012ad0 con 0x55c5e769e080 2024-04-02T14:37:56.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.090+0000 7f5c0be00640 1 -- 172.21.15.90:0/2328584159 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f5c24031020 con 0x55c5e769e080 2024-04-02T14:37:56.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.090+0000 7f5c33e7c740 10 client.26266 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:37:56.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.090+0000 7f5c33e7c740 10 client.26266 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:37:56.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.090+0000 7f5c33e7c740 20 client.26266 populate_metadata read hostname 'smithi090' 2024-04-02T14:37:56.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.091+0000 7f5c33e7c740 1 --2- 172.21.15.90:0/2328584159 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55c5e76ae3a0 0x55c5e76b0790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:56.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.091+0000 7f5c33e7c740 4 client.26266 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:37:56.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.091+0000 7f5c33e7c740 1 -- 172.21.15.90:0/2328584159 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c5e75b9040 con 0x55c5e76ae3a0 2024-04-02T14:37:56.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.091+0000 7f5c29e00640 1 --2- 172.21.15.90:0/2328584159 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55c5e76ae3a0 0x55c5e76b0790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:56.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.091+0000 7f5c0be00640 10 client.26266 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:37:56.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.092+0000 7f5c29e00640 1 --2- 172.21.15.90:0/2328584159 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55c5e76ae3a0 0x55c5e76b0790 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d4403777b6f5dae5 server_cookie=2e68114cab09c0 in_seq=0 out_seq=0 2024-04-02T14:37:56.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.093+0000 7f5c0be00640 1 -- 172.21.15.90:0/2328584159 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c5e75b9040 con 0x55c5e76ae3a0 2024-04-02T14:37:56.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.093+0000 7f5c0be00640 10 client.26266 handle_command_reply: tid=0 2024-04-02T14:37:56.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.093+0000 7f5c33e7c740 1 client.26266 shutdown 2024-04-02T14:37:56.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.094+0000 7f5c33e7c740 1 -- 172.21.15.90:0/2328584159 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c5e769e080 msgr2=0x55c5e769baf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:56.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.094+0000 7f5c33e7c740 1 --2- 172.21.15.90:0/2328584159 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c5e769e080 0x55c5e769baf0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f5c24015be0 tx=0x7f5c24004b50 comp rx=0 tx=0).stop 2024-04-02T14:37:56.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.094+0000 7f5c33e7c740 1 -- 172.21.15.90:0/2328584159 shutdown_connections 2024-04-02T14:37:56.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.094+0000 7f5c33e7c740 1 --2- 172.21.15.90:0/2328584159 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55c5e76ae3a0 0x55c5e76b0790 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:56.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.094+0000 7f5c33e7c740 1 --2- 172.21.15.90:0/2328584159 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c5e769e080 0x55c5e769baf0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:56.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.094+0000 7f5c33e7c740 1 --2- 172.21.15.90:0/2328584159 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c5e76959f0 0x55c5e769b5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:56.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.094+0000 7f5c33e7c740 1 --2- 172.21.15.90:0/2328584159 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c5e7694d10 0x55c5e769b070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:56.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.094+0000 7f5c33e7c740 1 -- 172.21.15.90:0/2328584159 >> 172.21.15.90:0/2328584159 conn(0x55c5e765ba50 msgr2=0x55c5e76914b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:37:56.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.094+0000 7f5c0be00640 0 client.26266 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:37:56.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.094+0000 7f5c33e7c740 1 -- 172.21.15.90:0/2328584159 shutdown_connections 2024-04-02T14:37:56.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.094+0000 7f5c33e7c740 1 -- 172.21.15.90:0/2328584159 wait complete. 2024-04-02T14:37:56.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.095+0000 7f5c33e7c740 20 client.26266 trim_cache size 0 max 16384 2024-04-02T14:37:56.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:37:55 smithi092 ceph-mon[36497]: pgmap v94: 97 pgs: 97 active+clean; 1.7 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 11 MiB/s wr, 942 op/s 2024-04-02T14:37:56.120 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.118+0000 7f5c33e7c740 1 Processor -- start 2024-04-02T14:37:56.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.118+0000 7f5c33e7c740 1 -- start start 2024-04-02T14:37:56.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.119+0000 7f5c33e7c740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c5e7694d10 0x55c5e76a9af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:56.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.119+0000 7f5c33e7c740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c5e76959f0 0x55c5e769cea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:56.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.119+0000 7f5c33e7c740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c5e769e080 0x55c5e76b6060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:56.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.119+0000 7f5c29400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c5e76959f0 0x55c5e769cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:56.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.119+0000 7f5c29400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c5e76959f0 0x55c5e769cea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:40430/0 (socket says 172.21.15.90:40430) 2024-04-02T14:37:56.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.119+0000 7f5c29400640 1 -- 172.21.15.90:0/3071348788 learned_addr learned my addr 172.21.15.90:0/3071348788 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:37:56.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.119+0000 7f5c33e7c740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f5c2c1b3f60 con 0x55c5e7694d10 2024-04-02T14:37:56.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.120+0000 7f5c33e7c740 1 -- 172.21.15.90:0/3071348788 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f5c2c1b5aa0 con 0x55c5e769e080 2024-04-02T14:37:56.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.120+0000 7f5c33e7c740 1 -- 172.21.15.90:0/3071348788 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f5c2c1b4630 con 0x55c5e76959f0 2024-04-02T14:37:56.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.120+0000 7f5c2a800640 1 --2- 172.21.15.90:0/3071348788 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c5e769e080 0x55c5e76b6060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:56.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.120+0000 7f5c29e00640 1 --2- 172.21.15.90:0/3071348788 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c5e7694d10 0x55c5e76a9af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:56.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.120+0000 7f5c2a800640 1 -- 172.21.15.90:0/3071348788 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c5e76959f0 msgr2=0x55c5e769cea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:37:56.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.120+0000 7f5c2a800640 1 --2- 172.21.15.90:0/3071348788 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c5e76959f0 0x55c5e769cea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:56.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.120+0000 7f5c2a800640 1 -- 172.21.15.90:0/3071348788 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c5e7694d10 msgr2=0x55c5e76a9af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:37:56.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.120+0000 7f5c2a800640 1 --2- 172.21.15.90:0/3071348788 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c5e7694d10 0x55c5e76a9af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:56.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.120+0000 7f5c2a800640 1 -- 172.21.15.90:0/3071348788 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c5e76b0b00 con 0x55c5e769e080 2024-04-02T14:37:56.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.120+0000 7f5c29e00640 1 --2- 172.21.15.90:0/3071348788 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c5e7694d10 0x55c5e76a9af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:37:56.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.121+0000 7f5c2a800640 1 --2- 172.21.15.90:0/3071348788 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c5e769e080 0x55c5e76b6060 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f5c24020f40 tx=0x7f5c24009770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6cef9ea39b733dbf server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:37:56.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.121+0000 7f5c16a00640 1 -- 172.21.15.90:0/3071348788 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c24009d20 con 0x55c5e769e080 2024-04-02T14:37:56.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.121+0000 7f5c16a00640 1 -- 172.21.15.90:0/3071348788 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5c24020460 con 0x55c5e769e080 2024-04-02T14:37:56.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.121+0000 7f5c16a00640 1 -- 172.21.15.90:0/3071348788 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c2403a860 con 0x55c5e769e080 2024-04-02T14:37:56.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.123+0000 7f5c33e7c740 1 -- 172.21.15.90:0/3071348788 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c5e769e080 msgr2=0x55c5e76b6060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:56.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.123+0000 7f5c33e7c740 1 --2- 172.21.15.90:0/3071348788 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c5e769e080 0x55c5e76b6060 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f5c24020f40 tx=0x7f5c24009770 comp rx=0 tx=0).stop 2024-04-02T14:37:56.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.123+0000 7f5c33e7c740 1 -- 172.21.15.90:0/3071348788 shutdown_connections 2024-04-02T14:37:56.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.123+0000 7f5c33e7c740 1 --2- 172.21.15.90:0/3071348788 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c5e769e080 0x55c5e76b6060 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:56.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.123+0000 7f5c33e7c740 1 --2- 172.21.15.90:0/3071348788 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c5e76959f0 0x55c5e769cea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:56.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.123+0000 7f5c33e7c740 1 --2- 172.21.15.90:0/3071348788 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c5e7694d10 0x55c5e76a9af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:56.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.123+0000 7f5c33e7c740 1 -- 172.21.15.90:0/3071348788 >> 172.21.15.90:0/3071348788 conn(0x55c5e765ba50 msgr2=0x55c5e769c790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:37:56.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.124+0000 7f5c33e7c740 1 -- 172.21.15.90:0/3071348788 shutdown_connections 2024-04-02T14:37:56.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.124+0000 7f5c33e7c740 1 -- 172.21.15.90:0/3071348788 wait complete. 2024-04-02T14:37:56.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.125+0000 7f5c33e7c740 1 -- start start 2024-04-02T14:37:56.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.125+0000 7f5c33e7c740 1 Processor -- start 2024-04-02T14:37:56.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.126+0000 7f5c33e7c740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c5e7694d10 0x55c5e76a61f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:56.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.126+0000 7f5c33e7c740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c5e76959f0 0x55c5e76a2fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:56.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.126+0000 7f5c33e7c740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c5e769e080 0x55c5e76a3510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:56.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.126+0000 7f5c29e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c5e7694d10 0x55c5e76a61f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:56.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.126+0000 7f5c29e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c5e7694d10 0x55c5e76a61f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:40444/0 (socket says 172.21.15.90:40444) 2024-04-02T14:37:56.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.126+0000 7f5c33e7c740 1 -- 172.21.15.90:0/469023870 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f5c2c1b53d0 con 0x55c5e769e080 2024-04-02T14:37:56.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.126+0000 7f5c33e7c740 1 -- 172.21.15.90:0/469023870 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f5c2c1b4d00 con 0x55c5e76959f0 2024-04-02T14:37:56.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.126+0000 7f5c33e7c740 1 -- 172.21.15.90:0/469023870 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f5c2c1b1ee0 con 0x55c5e7694d10 2024-04-02T14:37:56.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.126+0000 7f5c33e7c740 10 client.0 _ll_register_callbacks cb 0x7f5c24015860 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:37:56.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.126+0000 7f5c29e00640 1 -- 172.21.15.90:0/469023870 learned_addr learned my addr 172.21.15.90:0/469023870 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:37:56.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.126+0000 7f5c29400640 1 --2- 172.21.15.90:0/469023870 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c5e76959f0 0x55c5e76a2fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:56.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.127+0000 7f5c2a800640 1 --2- 172.21.15.90:0/469023870 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c5e769e080 0x55c5e76a3510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:56.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.127+0000 7f5c29e00640 1 -- 172.21.15.90:0/469023870 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c5e76959f0 msgr2=0x55c5e76a2fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:37:56.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.127+0000 7f5c29e00640 1 --2- 172.21.15.90:0/469023870 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c5e76959f0 0x55c5e76a2fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:56.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.127+0000 7f5c29e00640 1 -- 172.21.15.90:0/469023870 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c5e769e080 msgr2=0x55c5e76a3510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:37:56.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.127+0000 7f5c29e00640 1 --2- 172.21.15.90:0/469023870 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c5e769e080 0x55c5e76a3510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:56.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.127+0000 7f5c29e00640 1 -- 172.21.15.90:0/469023870 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c5e76a3ad0 con 0x55c5e7694d10 2024-04-02T14:37:56.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.127+0000 7f5c29400640 1 --2- 172.21.15.90:0/469023870 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c5e76959f0 0x55c5e76a2fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:37:56.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.128+0000 7f5c2a800640 1 --2- 172.21.15.90:0/469023870 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c5e769e080 0x55c5e76a3510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:37:56.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.128+0000 7f5c0be00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:37:56.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.128+0000 7f5c29e00640 1 --2- 172.21.15.90:0/469023870 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c5e7694d10 0x55c5e76a61f0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f5c1c005e00 tx=0x7f5c1c032900 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a9263c3775e2d82 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:37:56.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.129+0000 7f5c0be00640 1 -- 172.21.15.90:0/469023870 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c1c0c2260 con 0x55c5e7694d10 2024-04-02T14:37:56.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.129+0000 7f5c0be00640 1 -- 172.21.15.90:0/469023870 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f5c1c0bc1b0 con 0x55c5e7694d10 2024-04-02T14:37:56.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.129+0000 7f5c33e7c740 1 -- 172.21.15.90:0/469023870 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5c24020460 con 0x55c5e7694d10 2024-04-02T14:37:56.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.130+0000 7f5c0be00640 1 -- 172.21.15.90:0/469023870 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f5c1c0c6b20 con 0x55c5e7694d10 2024-04-02T14:37:56.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.130+0000 7f5c33e7c740 10 client.26272 fetch_fsmap learned FSMap version 27 2024-04-02T14:37:56.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.130+0000 7f5c33e7c740 1 -- 172.21.15.90:0/469023870 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55c5e7715660 con 0x55c5e7694d10 2024-04-02T14:37:56.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.130+0000 7f5c0be00640 1 -- 172.21.15.90:0/469023870 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5c24020460 con 0x55c5e7694d10 2024-04-02T14:37:56.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.130+0000 7f5c0be00640 1 -- 172.21.15.90:0/469023870 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f5c1c03c030 con 0x55c5e7694d10 2024-04-02T14:37:56.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.131+0000 7f5c33e7c740 10 client.26272 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:37:56.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.131+0000 7f5c33e7c740 10 client.26272 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:37:56.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.131+0000 7f5c33e7c740 20 client.26272 populate_metadata read hostname 'smithi090' 2024-04-02T14:37:56.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.131+0000 7f5c33e7c740 1 --2- 172.21.15.90:0/469023870 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55c5e7715e20 0x55c5e7718240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:56.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.131+0000 7f5c33e7c740 4 client.26272 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-02T14:37:56.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.131+0000 7f5c33e7c740 1 -- 172.21.15.90:0/469023870 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55c5e7606c30 con 0x55c5e7715e20 2024-04-02T14:37:56.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.131+0000 7f5c29400640 1 --2- 172.21.15.90:0/469023870 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55c5e7715e20 0x55c5e7718240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:56.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.132+0000 7f5c29400640 1 --2- 172.21.15.90:0/469023870 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55c5e7715e20 0x55c5e7718240 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9823c75249593cf3 server_cookie=9e78979d847356e in_seq=0 out_seq=0 2024-04-02T14:37:56.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.132+0000 7f5c0be00640 10 client.26272 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:37:56.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.132+0000 7f5c0be00640 1 -- 172.21.15.90:0/469023870 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55c5e7606c30 con 0x55c5e7715e20 2024-04-02T14:37:56.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.132+0000 7f5c0be00640 10 client.26272 handle_command_reply: tid=0 2024-04-02T14:37:56.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.132+0000 7f5c33e7c740 1 client.26272 shutdown 2024-04-02T14:37:56.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.133+0000 7f5c33e7c740 1 -- 172.21.15.90:0/469023870 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c5e7694d10 msgr2=0x55c5e76a61f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:56.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.133+0000 7f5c33e7c740 1 --2- 172.21.15.90:0/469023870 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c5e7694d10 0x55c5e76a61f0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f5c1c005e00 tx=0x7f5c1c032900 comp rx=0 tx=0).stop 2024-04-02T14:37:56.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.133+0000 7f5c33e7c740 1 -- 172.21.15.90:0/469023870 shutdown_connections 2024-04-02T14:37:56.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.133+0000 7f5c33e7c740 1 --2- 172.21.15.90:0/469023870 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55c5e7715e20 0x55c5e7718240 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:56.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.133+0000 7f5c33e7c740 1 --2- 172.21.15.90:0/469023870 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c5e769e080 0x55c5e76a3510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:56.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.133+0000 7f5c33e7c740 1 --2- 172.21.15.90:0/469023870 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c5e76959f0 0x55c5e76a2fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:56.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.133+0000 7f5c33e7c740 1 --2- 172.21.15.90:0/469023870 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c5e7694d10 0x55c5e76a61f0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:56.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.133+0000 7f5c33e7c740 1 -- 172.21.15.90:0/469023870 >> 172.21.15.90:0/469023870 conn(0x55c5e765ba50 msgr2=0x55c5e7691380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:37:56.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.133+0000 7f5c0be00640 0 client.26272 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:37:56.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.134+0000 7f5c33e7c740 1 -- 172.21.15.90:0/469023870 shutdown_connections 2024-04-02T14:37:56.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.134+0000 7f5c33e7c740 1 -- 172.21.15.90:0/469023870 wait complete. 2024-04-02T14:37:56.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.134+0000 7f5c33e7c740 20 client.26272 trim_cache size 0 max 16384 2024-04-02T14:37:56.137 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-02T14:37:56.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.135+0000 7f5c31200640 1 -- 172.21.15.90:0/4281722025 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f5c100a3e70 msgr2=0x7f5c100a6330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:56.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.135+0000 7f5c31200640 1 --2- 172.21.15.90:0/4281722025 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f5c100a3e70 0x7f5c100a6330 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f5c24009fd0 tx=0x7f5c2400b040 comp rx=0 tx=0).stop 2024-04-02T14:37:56.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.135+0000 7f5c31200640 1 -- 172.21.15.90:0/4281722025 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5c2c108bb0 msgr2=0x7f5c2c1a4760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:56.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.135+0000 7f5c31200640 1 --2- 172.21.15.90:0/4281722025 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5c2c108bb0 0x7f5c2c1a4760 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f5c1c02c0b0 tx=0x7f5c1c0323b0 comp rx=0 tx=0).stop 2024-04-02T14:37:56.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.135+0000 7f5c31200640 1 -- 172.21.15.90:0/4281722025 shutdown_connections 2024-04-02T14:37:56.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.135+0000 7f5c31200640 1 --2- 172.21.15.90:0/4281722025 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f5c100a3e70 0x7f5c100a6330 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:56.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.135+0000 7f5c31200640 1 --2- 172.21.15.90:0/4281722025 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5c2c19e960 0x7f5c2c19ee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:56.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.135+0000 7f5c31200640 1 --2- 172.21.15.90:0/4281722025 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5c2c10d3f0 0x7f5c2c1a4ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:56.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.135+0000 7f5c31200640 1 --2- 172.21.15.90:0/4281722025 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5c2c108bb0 0x7f5c2c1a4760 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:56.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.135+0000 7f5c31200640 1 -- 172.21.15.90:0/4281722025 >> 172.21.15.90:0/4281722025 conn(0x7f5c2c0776c0 msgr2=0x7f5c2c079ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:37:56.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.136+0000 7f5c31200640 1 -- 172.21.15.90:0/4281722025 shutdown_connections 2024-04-02T14:37:56.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:56.136+0000 7f5c31200640 1 -- 172.21.15.90:0/4281722025 wait complete. 2024-04-02T14:37:56.832 INFO:teuthology.orchestra.run.smithi090.stderr:progress: 145.0 s, 3911.9 tps, lat 25.113 ms stddev 24.547 2024-04-02T14:37:57.166 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-02T14:37:57.166 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T14:37:57.167 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-02T14:37:57.382 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.379+0000 7fb231200640 1 -- 172.21.15.90:0/2481739051 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb22c109b90 msgr2=0x7fb22c109f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:57.382 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.379+0000 7fb231200640 1 --2- 172.21.15.90:0/2481739051 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb22c109b90 0x7fb22c109f70 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fb21c009900 tx=0x7fb21c02bc60 comp rx=0 tx=0).stop 2024-04-02T14:37:57.383 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.380+0000 7fb231200640 1 -- 172.21.15.90:0/2481739051 shutdown_connections 2024-04-02T14:37:57.383 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.380+0000 7fb231200640 1 --2- 172.21.15.90:0/2481739051 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb22c1044b0 0x7fb22c1109c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:57.383 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.380+0000 7fb231200640 1 --2- 172.21.15.90:0/2481739051 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb22c103af0 0x7fb22c103f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:57.383 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.380+0000 7fb231200640 1 --2- 172.21.15.90:0/2481739051 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb22c109b90 0x7fb22c109f70 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:57.383 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.380+0000 7fb231200640 1 -- 172.21.15.90:0/2481739051 >> 172.21.15.90:0/2481739051 conn(0x7fb22c0ff5e0 msgr2=0x7fb22c101a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:37:57.383 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.380+0000 7fb231200640 1 -- 172.21.15.90:0/2481739051 shutdown_connections 2024-04-02T14:37:57.384 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.380+0000 7fb231200640 1 -- 172.21.15.90:0/2481739051 wait complete. 2024-04-02T14:37:57.384 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.382+0000 7fb231200640 1 Processor -- start 2024-04-02T14:37:57.384 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.382+0000 7fb231200640 1 -- start start 2024-04-02T14:37:57.385 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.383+0000 7fb231200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb22c103af0 0x7fb22c10eb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:57.385 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.383+0000 7fb231200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb22c1044b0 0x7fb22c073580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:57.386 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.384+0000 7fb231200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb22c109b90 0x7fb22c10f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:57.386 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.384+0000 7fb231200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fb22c076970 con 0x7fb22c1044b0 2024-04-02T14:37:57.386 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.384+0000 7fb22b400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb22c103af0 0x7fb22c10eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:57.386 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.384+0000 7fb231200640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fb22c0767f0 con 0x7fb22c103af0 2024-04-02T14:37:57.387 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.384+0000 7fb231200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fb22c076af0 con 0x7fb22c109b90 2024-04-02T14:37:57.387 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.384+0000 7fb22b400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb22c103af0 0x7fb22c10eb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:34414/0 (socket says 172.21.15.90:34414) 2024-04-02T14:37:57.387 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.384+0000 7fb22b400640 1 -- 172.21.15.90:0/402996739 learned_addr learned my addr 172.21.15.90:0/402996739 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:37:57.387 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.385+0000 7fb22be00640 1 --2- 172.21.15.90:0/402996739 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb22c109b90 0x7fb22c10f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:57.388 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.385+0000 7fb22aa00640 1 --2- 172.21.15.90:0/402996739 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb22c1044b0 0x7fb22c073580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:57.388 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.385+0000 7fb22b400640 1 -- 172.21.15.90:0/402996739 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb22c109b90 msgr2=0x7fb22c10f0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:57.388 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.385+0000 7fb22b400640 1 --2- 172.21.15.90:0/402996739 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb22c109b90 0x7fb22c10f0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:57.388 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.385+0000 7fb22b400640 1 -- 172.21.15.90:0/402996739 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb22c1044b0 msgr2=0x7fb22c073580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:57.388 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.385+0000 7fb22b400640 1 --2- 172.21.15.90:0/402996739 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb22c1044b0 0x7fb22c073580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:57.388 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.385+0000 7fb22b400640 1 -- 172.21.15.90:0/402996739 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb22c10f5f0 con 0x7fb22c103af0 2024-04-02T14:37:57.388 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.386+0000 7fb22aa00640 1 --2- 172.21.15.90:0/402996739 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb22c1044b0 0x7fb22c073580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:37:57.389 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.386+0000 7fb22be00640 1 --2- 172.21.15.90:0/402996739 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb22c109b90 0x7fb22c10f0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:37:57.389 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.386+0000 7fb22b400640 1 --2- 172.21.15.90:0/402996739 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb22c103af0 0x7fb22c10eb70 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fb21c0058c0 tx=0x7fb21c02c990 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:37:57.389 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.387+0000 7fb223e00640 1 -- 172.21.15.90:0/402996739 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb21c043020 con 0x7fb22c103af0 2024-04-02T14:37:57.389 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.387+0000 7fb223e00640 1 -- 172.21.15.90:0/402996739 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb21c004390 con 0x7fb22c103af0 2024-04-02T14:37:57.389 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.387+0000 7fb223e00640 1 -- 172.21.15.90:0/402996739 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb21c003d70 con 0x7fb22c103af0 2024-04-02T14:37:57.389 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.387+0000 7fb231200640 1 -- 172.21.15.90:0/402996739 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb22c10f880 con 0x7fb22c103af0 2024-04-02T14:37:57.390 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.387+0000 7fb231200640 1 -- 172.21.15.90:0/402996739 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb22c1b3c70 con 0x7fb22c103af0 2024-04-02T14:37:57.394 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.391+0000 7fb2353ed740 1 Processor -- start 2024-04-02T14:37:57.395 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.392+0000 7fb2353ed740 1 -- start start 2024-04-02T14:37:57.395 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.392+0000 7fb2353ed740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558bec0790e0 0x558bec0794c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:57.395 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.393+0000 7fb2353ed740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558bec079a00 0x558bec0820a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:57.395 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.393+0000 7fb2353ed740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558bec0825e0 0x558bec0849d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:57.395 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.393+0000 7fb2353ed740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x558bebfddd70 con 0x558bec0825e0 2024-04-02T14:37:57.395 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.393+0000 7fb2353ed740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x558bebf862a0 con 0x558bec079a00 2024-04-02T14:37:57.395 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.393+0000 7fb2353ed740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x558bebfd9d20 con 0x558bec0790e0 2024-04-02T14:37:57.396 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.393+0000 7fb22aa00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558bec0825e0 0x558bec0849d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:57.396 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.394+0000 7fb22be00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558bec0790e0 0x558bec0794c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:57.396 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.394+0000 7fb22aa00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558bec0825e0 0x558bec0849d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:54574/0 (socket says 172.21.15.90:54574) 2024-04-02T14:37:57.396 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.394+0000 7fb22aa00640 1 -- 172.21.15.90:0/3266767778 learned_addr learned my addr 172.21.15.90:0/3266767778 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:37:57.397 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.395+0000 7fb223e00640 1 -- 172.21.15.90:0/402996739 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fb21c046020 con 0x7fb22c103af0 2024-04-02T14:37:57.397 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.395+0000 7fb22aa00640 1 -- 172.21.15.90:0/3266767778 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558bec0790e0 msgr2=0x558bec0794c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:37:57.398 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.395+0000 7fb22aa00640 1 --2- 172.21.15.90:0/3266767778 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558bec0790e0 0x558bec0794c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:57.398 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.395+0000 7fb22aa00640 1 -- 172.21.15.90:0/3266767778 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558bec079a00 msgr2=0x558bec0820a0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-02T14:37:57.398 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.395+0000 7fb22aa00640 1 --2- 172.21.15.90:0/3266767778 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558bec079a00 0x558bec0820a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:57.398 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.395+0000 7fb22aa00640 1 -- 172.21.15.90:0/3266767778 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558bebf9a840 con 0x558bec0825e0 2024-04-02T14:37:57.398 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.397+0000 7fb22aa00640 1 --2- 172.21.15.90:0/3266767778 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558bec0825e0 0x558bec0849d0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fb21800a980 tx=0x7fb21800ae40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2130ccca10e735bc server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:37:57.400 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.397+0000 7fb223e00640 1 --2- 172.21.15.90:0/402996739 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fb2100a3d10 0x7fb2100a61d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:57.400 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.397+0000 7fb223e00640 1 -- 172.21.15.90:0/402996739 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7fb21c0f2a60 con 0x7fb22c103af0 2024-04-02T14:37:57.400 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.397+0000 7fb220c00640 1 -- 172.21.15.90:0/3266767778 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb218015070 con 0x558bec0825e0 2024-04-02T14:37:57.400 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.397+0000 7fb220c00640 1 -- 172.21.15.90:0/3266767778 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb21800cc00 con 0x558bec0825e0 2024-04-02T14:37:57.400 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.397+0000 7fb220c00640 1 -- 172.21.15.90:0/3266767778 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb218018690 con 0x558bec0825e0 2024-04-02T14:37:57.401 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.398+0000 7fb22be00640 1 --2- 172.21.15.90:0/402996739 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fb2100a3d10 0x7fb2100a61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:57.401 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.398+0000 7fb2353ed740 1 -- 172.21.15.90:0/3266767778 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558bec0825e0 msgr2=0x558bec0849d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:57.401 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.398+0000 7fb2353ed740 1 --2- 172.21.15.90:0/3266767778 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558bec0825e0 0x558bec0849d0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fb21800a980 tx=0x7fb21800ae40 comp rx=0 tx=0).stop 2024-04-02T14:37:57.402 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.399+0000 7fb2353ed740 1 -- 172.21.15.90:0/3266767778 shutdown_connections 2024-04-02T14:37:57.402 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.399+0000 7fb2353ed740 1 --2- 172.21.15.90:0/3266767778 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558bec0825e0 0x558bec0849d0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:57.403 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.399+0000 7fb2353ed740 1 --2- 172.21.15.90:0/3266767778 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558bec079a00 0x558bec0820a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:57.403 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.399+0000 7fb2353ed740 1 --2- 172.21.15.90:0/3266767778 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558bec0790e0 0x558bec0794c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:57.403 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.399+0000 7fb2353ed740 1 -- 172.21.15.90:0/3266767778 >> 172.21.15.90:0/3266767778 conn(0x558bec073810 msgr2=0x558bec073bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:37:57.403 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.400+0000 7fb2353ed740 1 -- 172.21.15.90:0/3266767778 shutdown_connections 2024-04-02T14:37:57.404 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.401+0000 7fb2353ed740 1 -- 172.21.15.90:0/3266767778 wait complete. 2024-04-02T14:37:57.404 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.401+0000 7fb2353ed740 1 -- start start 2024-04-02T14:37:57.404 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.401+0000 7fb22be00640 1 --2- 172.21.15.90:0/402996739 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fb2100a3d10 0x7fb2100a61d0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fb22400af80 tx=0x7fb22400a730 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:37:57.404 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.402+0000 7fb2353ed740 1 Processor -- start 2024-04-02T14:37:57.405 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.402+0000 7fb2353ed740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558bec0790e0 0x558bec07f6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:57.405 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.402+0000 7fb2353ed740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558bec079a00 0x558bec07fc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:57.405 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.402+0000 7fb2353ed740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558bec0825e0 0x558bec080140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:57.405 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.403+0000 7fb2353ed740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x558bec0853d0 con 0x558bec0790e0 2024-04-02T14:37:57.405 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.403+0000 7fb2353ed740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x558bebf99660 con 0x558bec079a00 2024-04-02T14:37:57.405 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.403+0000 7fb2353ed740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x558bec085550 con 0x558bec0825e0 2024-04-02T14:37:57.405 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.403+0000 7fb2353ed740 10 client.0 _ll_register_callbacks cb 0x558bebf99050 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:37:57.406 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.403+0000 7fb22aa00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558bec079a00 0x558bec07fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:57.406 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.403+0000 7fb22aa00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558bec079a00 0x558bec07fc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:34430/0 (socket says 172.21.15.90:34430) 2024-04-02T14:37:57.406 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.403+0000 7fb22aa00640 1 -- 172.21.15.90:0/3980507551 learned_addr learned my addr 172.21.15.90:0/3980507551 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:37:57.406 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.403+0000 7fb22be00640 1 --2- 172.21.15.90:0/3980507551 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558bec0825e0 0x558bec080140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:57.406 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.403+0000 7fb22be00640 1 -- 172.21.15.90:0/3980507551 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558bec079a00 msgr2=0x558bec07fc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:37:57.406 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.403+0000 7fb22be00640 1 --2- 172.21.15.90:0/3980507551 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558bec079a00 0x558bec07fc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:57.406 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.403+0000 7fb22be00640 1 -- 172.21.15.90:0/3980507551 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558bec0790e0 msgr2=0x558bec07f6c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-02T14:37:57.407 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.404+0000 7fb22b400640 1 --2- 172.21.15.90:0/3980507551 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558bec0790e0 0x558bec07f6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:57.407 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.405+0000 7fb22be00640 1 --2- 172.21.15.90:0/3980507551 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558bec0790e0 0x558bec07f6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:57.407 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.405+0000 7fb22be00640 1 -- 172.21.15.90:0/3980507551 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558bebfa8e80 con 0x558bec0825e0 2024-04-02T14:37:57.408 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.406+0000 7fb22b400640 1 --2- 172.21.15.90:0/3980507551 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558bec0790e0 0x558bec07f6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:37:57.409 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.406+0000 7fb22be00640 1 --2- 172.21.15.90:0/3980507551 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558bec0825e0 0x558bec080140 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fb22400ad60 tx=0x7fb224004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=51bb2cd2680dd26 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:37:57.409 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.406+0000 7fb215600640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:37:57.409 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.407+0000 7fb215600640 1 -- 172.21.15.90:0/3980507551 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb224029060 con 0x558bec0825e0 2024-04-02T14:37:57.410 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.407+0000 7fb215600640 1 -- 172.21.15.90:0/3980507551 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7fb22401a670 con 0x558bec0825e0 2024-04-02T14:37:57.410 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.408+0000 7fb2353ed740 1 -- 172.21.15.90:0/3980507551 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb21800cc00 con 0x558bec0825e0 2024-04-02T14:37:57.411 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.409+0000 7fb215600640 1 -- 172.21.15.90:0/3980507551 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fb22401ce10 con 0x558bec0825e0 2024-04-02T14:37:57.411 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.409+0000 7fb2353ed740 10 client.26281 fetch_fsmap learned FSMap version 27 2024-04-02T14:37:57.411 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.409+0000 7fb2353ed740 1 -- 172.21.15.90:0/3980507551 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x558bebcf2210 con 0x558bec0825e0 2024-04-02T14:37:57.412 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.409+0000 7fb215600640 1 -- 172.21.15.90:0/3980507551 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb21800cc00 con 0x558bec0825e0 2024-04-02T14:37:57.412 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.409+0000 7fb215600640 1 -- 172.21.15.90:0/3980507551 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fb224018040 con 0x558bec0825e0 2024-04-02T14:37:57.412 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.410+0000 7fb2353ed740 10 client.26281 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:37:57.412 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.410+0000 7fb2353ed740 10 client.26281 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-02T14:37:57.412 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.410+0000 7fb2353ed740 20 client.26281 populate_metadata read hostname 'smithi090' 2024-04-02T14:37:57.413 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.410+0000 7fb2353ed740 1 --2- 172.21.15.90:0/3980507551 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x558bec092a30 0x558bec094e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:57.413 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.410+0000 7fb2353ed740 4 client.26281 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:37:57.413 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.410+0000 7fb2353ed740 1 -- 172.21.15.90:0/3980507551 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558bebf8dcd0 con 0x558bec092a30 2024-04-02T14:37:57.413 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.410+0000 7fb22b400640 1 --2- 172.21.15.90:0/3980507551 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x558bec092a30 0x558bec094e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:57.414 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.411+0000 7fb22b400640 1 --2- 172.21.15.90:0/3980507551 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x558bec092a30 0x558bec094e40 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=25eecf442b19e9d server_cookie=43535de99f28eda7 in_seq=0 out_seq=0 2024-04-02T14:37:57.415 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.412+0000 7fb215600640 10 client.26281 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:37:57.415 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.412+0000 7fb215600640 1 -- 172.21.15.90:0/3980507551 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x558bebf8dcd0 con 0x558bec092a30 2024-04-02T14:37:57.415 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.412+0000 7fb215600640 10 client.26281 handle_command_reply: tid=0 2024-04-02T14:37:57.415 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.413+0000 7fb2353ed740 1 client.26281 shutdown 2024-04-02T14:37:57.416 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.414+0000 7fb2353ed740 1 -- 172.21.15.90:0/3980507551 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558bec0825e0 msgr2=0x558bec080140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:57.417 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.414+0000 7fb2353ed740 1 --2- 172.21.15.90:0/3980507551 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558bec0825e0 0x558bec080140 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fb22400ad60 tx=0x7fb224004a40 comp rx=0 tx=0).stop 2024-04-02T14:37:57.417 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.414+0000 7fb2353ed740 1 -- 172.21.15.90:0/3980507551 shutdown_connections 2024-04-02T14:37:57.417 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.414+0000 7fb2353ed740 1 --2- 172.21.15.90:0/3980507551 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x558bec092a30 0x558bec094e40 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:57.417 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.414+0000 7fb2353ed740 1 --2- 172.21.15.90:0/3980507551 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558bec0825e0 0x558bec080140 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:57.417 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.414+0000 7fb2353ed740 1 --2- 172.21.15.90:0/3980507551 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558bec079a00 0x558bec07fc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:57.417 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.414+0000 7fb2353ed740 1 --2- 172.21.15.90:0/3980507551 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558bec0790e0 0x558bec07f6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:57.417 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.414+0000 7fb2353ed740 1 -- 172.21.15.90:0/3980507551 >> 172.21.15.90:0/3980507551 conn(0x558bec073810 msgr2=0x558bec075560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:37:57.418 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.414+0000 7fb215600640 0 client.26281 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:37:57.418 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.415+0000 7fb2353ed740 1 -- 172.21.15.90:0/3980507551 shutdown_connections 2024-04-02T14:37:57.419 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.415+0000 7fb2353ed740 1 -- 172.21.15.90:0/3980507551 wait complete. 2024-04-02T14:37:57.419 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.415+0000 7fb2353ed740 20 client.26281 trim_cache size 0 max 16384 2024-04-02T14:37:57.473 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.471+0000 7fb2353ed740 1 Processor -- start 2024-04-02T14:37:57.473 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.471+0000 7fb2353ed740 1 -- start start 2024-04-02T14:37:57.474 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.471+0000 7fb2353ed740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558bec0790e0 0x558bec081df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:57.474 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.472+0000 7fb2353ed740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558bec079a00 0x558bec08e290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:57.474 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.472+0000 7fb2353ed740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558bec0825e0 0x558bec09a750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:57.474 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.472+0000 7fb2353ed740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fb22c1bc5a0 con 0x558bec0825e0 2024-04-02T14:37:57.474 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.472+0000 7fb2353ed740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fb22c1be0e0 con 0x558bec0790e0 2024-04-02T14:37:57.474 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.472+0000 7fb2353ed740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fb22c1bcc70 con 0x558bec079a00 2024-04-02T14:37:57.475 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.472+0000 7fb22aa00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558bec079a00 0x558bec08e290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:57.475 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.472+0000 7fb22aa00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558bec079a00 0x558bec08e290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:40492/0 (socket says 172.21.15.90:40492) 2024-04-02T14:37:57.475 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.472+0000 7fb22aa00640 1 -- 172.21.15.90:0/1134462092 learned_addr learned my addr 172.21.15.90:0/1134462092 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:37:57.475 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.472+0000 7fb22b400640 1 --2- 172.21.15.90:0/1134462092 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558bec0790e0 0x558bec081df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:57.475 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.473+0000 7fb22be00640 1 --2- 172.21.15.90:0/1134462092 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558bec0825e0 0x558bec09a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:57.476 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.473+0000 7fb22aa00640 1 -- 172.21.15.90:0/1134462092 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558bec0790e0 msgr2=0x558bec081df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:37:57.476 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.473+0000 7fb22aa00640 1 --2- 172.21.15.90:0/1134462092 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558bec0790e0 0x558bec081df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:57.476 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.473+0000 7fb22aa00640 1 -- 172.21.15.90:0/1134462092 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558bec0825e0 msgr2=0x558bec09a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:37:57.476 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.473+0000 7fb22aa00640 1 --2- 172.21.15.90:0/1134462092 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558bec0825e0 0x558bec09a750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:57.476 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.473+0000 7fb22aa00640 1 -- 172.21.15.90:0/1134462092 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558bebdbe620 con 0x558bec079a00 2024-04-02T14:37:57.476 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.473+0000 7fb22be00640 1 --2- 172.21.15.90:0/1134462092 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558bec0825e0 0x558bec09a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:37:57.476 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.474+0000 7fb22aa00640 1 --2- 172.21.15.90:0/1134462092 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558bec079a00 0x558bec08e290 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fb218011e40 tx=0x7fb21800a660 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5fbd81e5264165a5 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:37:57.477 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.475+0000 7fb217e00640 1 -- 172.21.15.90:0/1134462092 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb218015050 con 0x558bec079a00 2024-04-02T14:37:57.477 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.475+0000 7fb217e00640 1 -- 172.21.15.90:0/1134462092 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb21800a220 con 0x558bec079a00 2024-04-02T14:37:57.477 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.475+0000 7fb217e00640 1 -- 172.21.15.90:0/1134462092 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb218003a50 con 0x558bec079a00 2024-04-02T14:37:57.478 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.476+0000 7fb2353ed740 1 -- 172.21.15.90:0/1134462092 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558bec079a00 msgr2=0x558bec08e290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:57.479 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.476+0000 7fb2353ed740 1 --2- 172.21.15.90:0/1134462092 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558bec079a00 0x558bec08e290 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fb218011e40 tx=0x7fb21800a660 comp rx=0 tx=0).stop 2024-04-02T14:37:57.479 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.477+0000 7fb2353ed740 1 -- 172.21.15.90:0/1134462092 shutdown_connections 2024-04-02T14:37:57.479 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.477+0000 7fb2353ed740 1 --2- 172.21.15.90:0/1134462092 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558bec0825e0 0x558bec09a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:57.479 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.477+0000 7fb2353ed740 1 --2- 172.21.15.90:0/1134462092 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558bec079a00 0x558bec08e290 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:57.480 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.477+0000 7fb2353ed740 1 --2- 172.21.15.90:0/1134462092 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558bec0790e0 0x558bec081df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:57.480 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.477+0000 7fb2353ed740 1 -- 172.21.15.90:0/1134462092 >> 172.21.15.90:0/1134462092 conn(0x558bec073810 msgr2=0x558bec073bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:37:57.480 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.477+0000 7fb2353ed740 1 -- 172.21.15.90:0/1134462092 shutdown_connections 2024-04-02T14:37:57.480 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.477+0000 7fb2353ed740 1 -- 172.21.15.90:0/1134462092 wait complete. 2024-04-02T14:37:57.480 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.478+0000 7fb2353ed740 1 -- start start 2024-04-02T14:37:57.481 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.479+0000 7fb2353ed740 1 Processor -- start 2024-04-02T14:37:57.482 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.479+0000 7fb2353ed740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558bec0790e0 0x558bec0860c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:57.482 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.479+0000 7fb2353ed740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558bec079a00 0x558bec086600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:57.482 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.479+0000 7fb2353ed740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558bec0825e0 0x558bec086b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:57.482 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.479+0000 7fb2353ed740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fb22c1bda10 con 0x558bec0825e0 2024-04-02T14:37:57.482 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.479+0000 7fb2353ed740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fb22c1bd340 con 0x558bec079a00 2024-04-02T14:37:57.482 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.479+0000 7fb2353ed740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fb22c1ba580 con 0x558bec0790e0 2024-04-02T14:37:57.482 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.480+0000 7fb2353ed740 10 client.0 _ll_register_callbacks cb 0x7fb22400fb00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:37:57.483 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.481+0000 7fb22b400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558bec0790e0 0x558bec0860c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:57.483 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.481+0000 7fb22b400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558bec0790e0 0x558bec0860c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:40498/0 (socket says 172.21.15.90:40498) 2024-04-02T14:37:57.483 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.481+0000 7fb22b400640 1 -- 172.21.15.90:0/3839928622 learned_addr learned my addr 172.21.15.90:0/3839928622 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:37:57.483 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.481+0000 7fb22aa00640 1 --2- 172.21.15.90:0/3839928622 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558bec079a00 0x558bec086600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:57.484 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.481+0000 7fb22be00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558bec0825e0 0x558bec086b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:57.484 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.482+0000 7fb22be00640 1 -- 172.21.15.90:0/3839928622 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558bec0790e0 msgr2=0x558bec0860c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:37:57.485 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.482+0000 7fb22be00640 1 --2- 172.21.15.90:0/3839928622 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558bec0790e0 0x558bec0860c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:57.485 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.482+0000 7fb22be00640 1 -- 172.21.15.90:0/3839928622 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558bec079a00 msgr2=0x558bec086600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:37:57.485 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.482+0000 7fb22be00640 1 --2- 172.21.15.90:0/3839928622 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558bec079a00 0x558bec086600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:57.485 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.483+0000 7fb22be00640 1 -- 172.21.15.90:0/3839928622 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558bec0f7fc0 con 0x558bec0825e0 2024-04-02T14:37:57.485 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.483+0000 7fb22b400640 1 --2- 172.21.15.90:0/3839928622 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558bec0790e0 0x558bec0860c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:37:57.486 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.483+0000 7fb22be00640 1 --2- 172.21.15.90:0/3839928622 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558bec0825e0 0x558bec086b40 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fb224024ca0 tx=0x7fb224024690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf0c49ad234437e0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:37:57.486 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.484+0000 7fb215600640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-02T14:37:57.486 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.484+0000 7fb215600640 1 -- 172.21.15.90:0/3839928622 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb224033080 con 0x558bec0825e0 2024-04-02T14:37:57.486 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.484+0000 7fb215600640 1 -- 172.21.15.90:0/3839928622 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7fb22401a870 con 0x558bec0825e0 2024-04-02T14:37:57.487 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.485+0000 7fb2353ed740 1 -- 172.21.15.90:0/3839928622 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x558bebfeb2b0 con 0x558bec0825e0 2024-04-02T14:37:57.487 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.485+0000 7fb215600640 1 -- 172.21.15.90:0/3839928622 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fb224029030 con 0x558bec0825e0 2024-04-02T14:37:57.488 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.486+0000 7fb2353ed740 10 client.15981 fetch_fsmap learned FSMap version 27 2024-04-02T14:37:57.488 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.486+0000 7fb2353ed740 1 -- 172.21.15.90:0/3839928622 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x558bec0f9270 con 0x558bec0825e0 2024-04-02T14:37:57.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.486+0000 7fbf36a00640 1 -- 172.21.15.90:0/490308410 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbf30076030 msgr2=0x7fbf30076410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:57.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.486+0000 7fbf36a00640 1 --2- 172.21.15.90:0/490308410 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbf30076030 0x7fbf30076410 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fbf20009930 tx=0x7fbf2002bc30 comp rx=0 tx=0).stop 2024-04-02T14:37:57.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.486+0000 7fbf36a00640 1 -- 172.21.15.90:0/490308410 shutdown_connections 2024-04-02T14:37:57.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.486+0000 7fbf36a00640 1 --2- 172.21.15.90:0/490308410 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbf30112940 0x7fbf30114d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:57.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.487+0000 7fbf36a00640 1 --2- 172.21.15.90:0/490308410 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbf30076950 0x7fbf30112400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:57.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.487+0000 7fbf36a00640 1 --2- 172.21.15.90:0/490308410 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbf30076030 0x7fbf30076410 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:57.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.487+0000 7fbf36a00640 1 -- 172.21.15.90:0/490308410 >> 172.21.15.90:0/490308410 conn(0x7fbf300ff770 msgr2=0x7fbf30101b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:37:57.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.487+0000 7fbf36a00640 1 -- 172.21.15.90:0/490308410 shutdown_connections 2024-04-02T14:37:57.490 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.487+0000 7fb215600640 1 -- 172.21.15.90:0/3839928622 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x558bebfeb2b0 con 0x558bec0825e0 2024-04-02T14:37:57.490 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.487+0000 7fb215600640 1 -- 172.21.15.90:0/3839928622 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fb224018040 con 0x558bec0825e0 2024-04-02T14:37:57.490 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.487+0000 7fb2353ed740 10 client.15981 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:37:57.490 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.488+0000 7fb2353ed740 10 client.15981 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-02T14:37:57.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.487+0000 7fbf36a00640 1 -- 172.21.15.90:0/490308410 wait complete. 2024-04-02T14:37:57.491 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.488+0000 7fb2353ed740 20 client.15981 populate_metadata read hostname 'smithi090' 2024-04-02T14:37:57.491 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.488+0000 7fb2353ed740 1 --2- 172.21.15.90:0/3839928622 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x558bec0f99c0 0x558bec0fbde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:57.491 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.488+0000 7fb2353ed740 4 client.15981 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-02T14:37:57.491 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.488+0000 7fb2353ed740 1 -- 172.21.15.90:0/3839928622 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "flush journal"}) -- 0x558bebfe5600 con 0x558bec0f99c0 2024-04-02T14:37:57.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.488+0000 7fbf36a00640 1 Processor -- start 2024-04-02T14:37:57.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.488+0000 7fbf36a00640 1 -- start start 2024-04-02T14:37:57.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.489+0000 7fbf36a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbf30076030 0x7fbf301a4640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:57.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.489+0000 7fbf36a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbf30076950 0x7fbf301a4b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:57.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.489+0000 7fbf36a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbf30112940 0x7fbf3019e800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:57.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.489+0000 7fbf36a00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fbf301184e0 con 0x7fbf30076030 2024-04-02T14:37:57.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.489+0000 7fbf36a00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fbf30118360 con 0x7fbf30076950 2024-04-02T14:37:57.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.489+0000 7fbf36a00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fbf30118660 con 0x7fbf30112940 2024-04-02T14:37:57.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.489+0000 7fbf34a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbf30112940 0x7fbf3019e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:57.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.489+0000 7fbf34a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbf30112940 0x7fbf3019e800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:40502/0 (socket says 172.21.15.90:40502) 2024-04-02T14:37:57.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.489+0000 7fbf34a00640 1 -- 172.21.15.90:0/3295335019 learned_addr learned my addr 172.21.15.90:0/3295335019 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:37:57.493 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.489+0000 7fb22b400640 1 --2- 172.21.15.90:0/3839928622 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x558bec0f99c0 0x558bec0fbde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:57.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.490+0000 7fbf34a00640 1 -- 172.21.15.90:0/3295335019 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbf30076950 msgr2=0x7fbf301a4b80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T14:37:57.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.490+0000 7fbf34a00640 1 --2- 172.21.15.90:0/3295335019 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbf30076950 0x7fbf301a4b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:57.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.490+0000 7fbf34a00640 1 -- 172.21.15.90:0/3295335019 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbf30076030 msgr2=0x7fbf301a4640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:57.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.490+0000 7fbf34a00640 1 --2- 172.21.15.90:0/3295335019 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbf30076030 0x7fbf301a4640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:57.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.490+0000 7fbf34a00640 1 -- 172.21.15.90:0/3295335019 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf3019efd0 con 0x7fbf30112940 2024-04-02T14:37:57.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.490+0000 7fbf34a00640 1 --2- 172.21.15.90:0/3295335019 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbf30112940 0x7fbf3019e800 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fbf2800e9d0 tx=0x7fbf2800ee90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:37:57.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.490+0000 7fbf2cc00640 1 -- 172.21.15.90:0/3295335019 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf2800cd80 con 0x7fbf30112940 2024-04-02T14:37:57.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.491+0000 7fbf2cc00640 1 -- 172.21.15.90:0/3295335019 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbf28005020 con 0x7fbf30112940 2024-04-02T14:37:57.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.491+0000 7fbf2cc00640 1 -- 172.21.15.90:0/3295335019 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf280116d0 con 0x7fbf30112940 2024-04-02T14:37:57.494 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.490+0000 7fb22b400640 1 --2- 172.21.15.90:0/3839928622 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x558bec0f99c0 0x558bec0fbde0 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1d0aafc6dd0a03f7 server_cookie=3afc79293efc024d in_seq=0 out_seq=0 2024-04-02T14:37:57.494 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.490+0000 7fb215600640 10 client.15981 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:37:57.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.491+0000 7fbf36a00640 1 -- 172.21.15.90:0/3295335019 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbf3019f2c0 con 0x7fbf30112940 2024-04-02T14:37:57.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.491+0000 7fbf36a00640 1 -- 172.21.15.90:0/3295335019 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbf30103bb0 con 0x7fbf30112940 2024-04-02T14:37:57.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.503+0000 7fbf3979d740 1 Processor -- start 2024-04-02T14:37:57.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.503+0000 7fbf2cc00640 1 -- 172.21.15.90:0/3295335019 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fbf28011930 con 0x7fbf30112940 2024-04-02T14:37:57.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.503+0000 7fbf3979d740 1 -- start start 2024-04-02T14:37:57.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.504+0000 7fbf3979d740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5594607ea030 0x5594607e8d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:57.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.504+0000 7fbf3979d740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5594607e9350 0x5594607e97d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:57.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.504+0000 7fbf3979d740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5594607f26c0 0x5594607f4ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:57.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.504+0000 7fbf3979d740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55946074d310 con 0x5594607ea030 2024-04-02T14:37:57.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.504+0000 7fbf3979d740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x5594606f5690 con 0x5594607f26c0 2024-04-02T14:37:57.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.504+0000 7fbf3979d740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5594607492c0 con 0x5594607e9350 2024-04-02T14:37:57.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.505+0000 7fbf2fe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5594607e9350 0x5594607e97d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:57.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.505+0000 7fbf2fe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5594607e9350 0x5594607e97d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:40514/0 (socket says 172.21.15.90:40514) 2024-04-02T14:37:57.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.505+0000 7fbf2fe00640 1 -- 172.21.15.90:0/3108433691 learned_addr learned my addr 172.21.15.90:0/3108433691 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:37:57.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.505+0000 7fbf2f400640 1 --2- 172.21.15.90:0/3108433691 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5594607ea030 0x5594607e8d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:57.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.506+0000 7fbf2f400640 1 -- 172.21.15.90:0/3108433691 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5594607e9350 msgr2=0x5594607e97d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:37:57.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.506+0000 7fbf2f400640 1 --2- 172.21.15.90:0/3108433691 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5594607e9350 0x5594607e97d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:57.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.506+0000 7fbf2f400640 1 -- 172.21.15.90:0/3108433691 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5594607f26c0 msgr2=0x5594607f4ab0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-02T14:37:57.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.506+0000 7fbf2f400640 1 --2- 172.21.15.90:0/3108433691 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5594607f26c0 0x5594607f4ab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:57.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.506+0000 7fbf2f400640 1 -- 172.21.15.90:0/3108433691 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5594605297e0 con 0x5594607ea030 2024-04-02T14:37:57.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.507+0000 7fbf2f400640 1 --2- 172.21.15.90:0/3108433691 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5594607ea030 0x5594607e8d80 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fbf3019fb00 tx=0x7fbf1c00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e2ff27122ed3e76 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:37:57.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.507+0000 7fbf2cc00640 1 --2- 172.21.15.90:0/3295335019 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fbf140a3de0 0x7fbf140a62a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:57.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.508+0000 7fbf25200640 1 -- 172.21.15.90:0/3108433691 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf1c00cca0 con 0x5594607ea030 2024-04-02T14:37:57.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.508+0000 7fbf2cc00640 1 -- 172.21.15.90:0/3295335019 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7fbf28005540 con 0x7fbf30112940 2024-04-02T14:37:57.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.508+0000 7fbf25200640 1 -- 172.21.15.90:0/3108433691 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbf1c00ce40 con 0x5594607ea030 2024-04-02T14:37:57.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.508+0000 7fbf25200640 1 -- 172.21.15.90:0/3108433691 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf1c018940 con 0x5594607ea030 2024-04-02T14:37:57.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.508+0000 7fbf2fe00640 1 --2- 172.21.15.90:0/3295335019 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fbf140a3de0 0x7fbf140a62a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:57.511 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.509+0000 7fb215600640 1 -- 172.21.15.90:0/3839928622 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x558bebfe5600 con 0x558bec0f99c0 2024-04-02T14:37:57.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.510+0000 7fbf3979d740 1 -- 172.21.15.90:0/3108433691 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5594607ea030 msgr2=0x5594607e8d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:57.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.510+0000 7fbf3979d740 1 --2- 172.21.15.90:0/3108433691 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5594607ea030 0x5594607e8d80 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fbf3019fb00 tx=0x7fbf1c00c2d0 comp rx=0 tx=0).stop 2024-04-02T14:37:57.512 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.509+0000 7fb215600640 10 client.15981 handle_command_reply: tid=0 2024-04-02T14:37:57.513 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.510+0000 7fb2353ed740 1 client.15981 shutdown 2024-04-02T14:37:57.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.510+0000 7fbf3979d740 1 -- 172.21.15.90:0/3108433691 shutdown_connections 2024-04-02T14:37:57.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.510+0000 7fbf3979d740 1 --2- 172.21.15.90:0/3108433691 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5594607f26c0 0x5594607f4ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:57.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.510+0000 7fbf3979d740 1 --2- 172.21.15.90:0/3108433691 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5594607e9350 0x5594607e97d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:57.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.510+0000 7fbf3979d740 1 --2- 172.21.15.90:0/3108433691 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5594607ea030 0x5594607e8d80 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:57.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.510+0000 7fbf3979d740 1 -- 172.21.15.90:0/3108433691 >> 172.21.15.90:0/3108433691 conn(0x55946077d270 msgr2=0x5594607e3da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:37:57.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.510+0000 7fbf2fe00640 1 --2- 172.21.15.90:0/3295335019 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fbf140a3de0 0x7fbf140a62a0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fbf20009900 tx=0x7fbf20000bc0 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:37:57.514 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.510+0000 7fb2353ed740 1 -- 172.21.15.90:0/3839928622 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558bec0825e0 msgr2=0x558bec086b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:57.514 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.510+0000 7fb2353ed740 1 --2- 172.21.15.90:0/3839928622 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558bec0825e0 0x558bec086b40 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fb224024ca0 tx=0x7fb224024690 comp rx=0 tx=0).stop 2024-04-02T14:37:57.514 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.510+0000 7fbf3979d740 1 -- 172.21.15.90:0/3108433691 shutdown_connections 2024-04-02T14:37:57.514 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.511+0000 7fbf3979d740 1 -- 172.21.15.90:0/3108433691 wait complete. 2024-04-02T14:37:57.514 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.511+0000 7fbf3979d740 1 -- start start 2024-04-02T14:37:57.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.511+0000 7fbf3979d740 1 Processor -- start 2024-04-02T14:37:57.515 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.511+0000 7fb2353ed740 1 -- 172.21.15.90:0/3839928622 shutdown_connections 2024-04-02T14:37:57.516 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.511+0000 7fb2353ed740 1 --2- 172.21.15.90:0/3839928622 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x558bec0f99c0 0x558bec0fbde0 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:57.516 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.511+0000 7fb2353ed740 1 --2- 172.21.15.90:0/3839928622 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558bec0825e0 0x558bec086b40 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:57.516 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.512+0000 7fb2353ed740 1 --2- 172.21.15.90:0/3839928622 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558bec079a00 0x558bec086600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:57.516 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.512+0000 7fb2353ed740 1 --2- 172.21.15.90:0/3839928622 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558bec0790e0 0x558bec0860c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:57.516 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.512+0000 7fb2353ed740 1 -- 172.21.15.90:0/3839928622 >> 172.21.15.90:0/3839928622 conn(0x558bec073810 msgr2=0x558bec07f680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:37:57.516 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.512+0000 7fb215600640 0 client.15981 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:37:57.516 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.512+0000 7fb2353ed740 1 -- 172.21.15.90:0/3839928622 shutdown_connections 2024-04-02T14:37:57.517 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.512+0000 7fb2353ed740 1 -- 172.21.15.90:0/3839928622 wait complete. 2024-04-02T14:37:57.517 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.512+0000 7fb2353ed740 20 client.15981 trim_cache size 0 max 16384 2024-04-02T14:37:57.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.513+0000 7fbf3979d740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5594607e9350 0x5594607ef6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:57.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.513+0000 7fbf3979d740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5594607ea030 0x5594607efc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:57.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.513+0000 7fbf3979d740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5594607f26c0 0x5594607f0170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:57.518 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:{ 2024-04-02T14:37:57.518 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "message": "", 2024-04-02T14:37:57.518 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "return_code": 0 2024-04-02T14:37:57.518 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:} 2024-04-02T14:37:57.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.513+0000 7fbf34a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5594607f26c0 0x5594607f0170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:57.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.513+0000 7fbf34a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5594607f26c0 0x5594607f0170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:34450/0 (socket says 172.21.15.90:34450) 2024-04-02T14:37:57.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.513+0000 7fbf34a00640 1 -- 172.21.15.90:0/1213561863 learned_addr learned my addr 172.21.15.90:0/1213561863 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:37:57.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.513+0000 7fbf3979d740 1 -- 172.21.15.90:0/1213561863 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5594607f5530 con 0x5594607ea030 2024-04-02T14:37:57.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.513+0000 7fbf3979d740 1 -- 172.21.15.90:0/1213561863 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x559460708a60 con 0x5594607f26c0 2024-04-02T14:37:57.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.513+0000 7fbf3979d740 1 -- 172.21.15.90:0/1213561863 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5594607f56b0 con 0x5594607e9350 2024-04-02T14:37:57.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.514+0000 7fbf2f400640 1 --2- 172.21.15.90:0/1213561863 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5594607ea030 0x5594607efc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:57.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.514+0000 7fbf2fe00640 1 --2- 172.21.15.90:0/1213561863 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5594607e9350 0x5594607ef6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:57.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.514+0000 7fbf3979d740 10 client.0 _ll_register_callbacks cb 0x55946070e030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:37:57.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.514+0000 7fbf34a00640 1 -- 172.21.15.90:0/1213561863 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5594607e9350 msgr2=0x5594607ef6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:37:57.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.514+0000 7fbf34a00640 1 --2- 172.21.15.90:0/1213561863 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5594607e9350 0x5594607ef6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:57.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.514+0000 7fbf34a00640 1 -- 172.21.15.90:0/1213561863 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5594607ea030 msgr2=0x5594607efc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:37:57.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.514+0000 7fbf34a00640 1 --2- 172.21.15.90:0/1213561863 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5594607ea030 0x5594607efc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:57.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.515+0000 7fbf34a00640 1 -- 172.21.15.90:0/1213561863 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559460589f70 con 0x5594607f26c0 2024-04-02T14:37:57.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.515+0000 7fbf2fe00640 1 --2- 172.21.15.90:0/1213561863 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5594607e9350 0x5594607ef6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:37:57.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.515+0000 7fbf2f400640 1 --2- 172.21.15.90:0/1213561863 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5594607ea030 0x5594607efc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:37:57.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.515+0000 7fbf34a00640 1 --2- 172.21.15.90:0/1213561863 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5594607f26c0 0x5594607f0170 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fbf2800cfd0 tx=0x7fbf2809dd50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d752428855d68250 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:37:57.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.515+0000 7fbf19600640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:37:57.521 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.515+0000 7fb231200640 1 -- 172.21.15.90:0/402996739 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fb2100a3d10 msgr2=0x7fb2100a61d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:57.521 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.515+0000 7fb231200640 1 --2- 172.21.15.90:0/402996739 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fb2100a3d10 0x7fb2100a61d0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fb22400af80 tx=0x7fb22400a730 comp rx=0 tx=0).stop 2024-04-02T14:37:57.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.515+0000 7fbf19600640 1 -- 172.21.15.90:0/1213561863 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf280b7060 con 0x5594607f26c0 2024-04-02T14:37:57.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.516+0000 7fbf19600640 1 -- 172.21.15.90:0/1213561863 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7fbf280a64b0 con 0x5594607f26c0 2024-04-02T14:37:57.522 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.516+0000 7fb231200640 1 -- 172.21.15.90:0/402996739 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb22c103af0 msgr2=0x7fb22c10eb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:57.522 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.516+0000 7fb231200640 1 --2- 172.21.15.90:0/402996739 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb22c103af0 0x7fb22c10eb70 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fb21c0058c0 tx=0x7fb21c02c990 comp rx=0 tx=0).stop 2024-04-02T14:37:57.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.516+0000 7fbf3979d740 1 -- 172.21.15.90:0/1213561863 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55946075a850 con 0x5594607f26c0 2024-04-02T14:37:57.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.516+0000 7fbf19600640 1 -- 172.21.15.90:0/1213561863 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fbf280bc050 con 0x5594607f26c0 2024-04-02T14:37:57.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.516+0000 7fbf3979d740 10 client.26387 fetch_fsmap learned FSMap version 27 2024-04-02T14:37:57.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.516+0000 7fbf3979d740 1 -- 172.21.15.90:0/1213561863 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x559460421170 con 0x5594607f26c0 2024-04-02T14:37:57.523 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.516+0000 7fb231200640 1 -- 172.21.15.90:0/402996739 shutdown_connections 2024-04-02T14:37:57.524 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.516+0000 7fb231200640 1 --2- 172.21.15.90:0/402996739 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fb2100a3d10 0x7fb2100a61d0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:57.524 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.516+0000 7fb231200640 1 --2- 172.21.15.90:0/402996739 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb22c109b90 0x7fb22c10f0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:57.524 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.516+0000 7fb231200640 1 --2- 172.21.15.90:0/402996739 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb22c1044b0 0x7fb22c073580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:57.524 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.516+0000 7fb231200640 1 --2- 172.21.15.90:0/402996739 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb22c103af0 0x7fb22c10eb70 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:57.524 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.516+0000 7fb231200640 1 -- 172.21.15.90:0/402996739 >> 172.21.15.90:0/402996739 conn(0x7fb22c0ff5e0 msgr2=0x7fb22c0ffe70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:37:57.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.516+0000 7fbf19600640 1 -- 172.21.15.90:0/1213561863 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55946075a850 con 0x5594607f26c0 2024-04-02T14:37:57.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.517+0000 7fbf19600640 1 -- 172.21.15.90:0/1213561863 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fbf2800f040 con 0x5594607f26c0 2024-04-02T14:37:57.525 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.517+0000 7fb231200640 1 -- 172.21.15.90:0/402996739 shutdown_connections 2024-04-02T14:37:57.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.517+0000 7fbf3979d740 10 client.26387 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:37:57.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.517+0000 7fbf3979d740 10 client.26387 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:37:57.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.517+0000 7fbf3979d740 20 client.26387 populate_metadata read hostname 'smithi090' 2024-04-02T14:37:57.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.517+0000 7fbf3979d740 1 --2- 172.21.15.90:0/1213561863 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x559460802c20 0x559460805010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:57.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.517+0000 7fbf3979d740 4 client.26387 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:37:57.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.517+0000 7fbf3979d740 1 -- 172.21.15.90:0/1213561863 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559460754ba0 con 0x559460802c20 2024-04-02T14:37:57.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.517+0000 7fbf2fe00640 1 --2- 172.21.15.90:0/1213561863 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x559460802c20 0x559460805010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:57.527 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:37:57.517+0000 7fb231200640 1 -- 172.21.15.90:0/402996739 wait complete. 2024-04-02T14:37:57.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.518+0000 7fbf19600640 10 client.26387 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:37:57.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.519+0000 7fbf2fe00640 1 --2- 172.21.15.90:0/1213561863 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x559460802c20 0x559460805010 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ff2442e0ceb84fe3 server_cookie=c96272a49722b01c in_seq=0 out_seq=0 2024-04-02T14:37:57.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.520+0000 7fbf19600640 1 -- 172.21.15.90:0/1213561863 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x559460754ba0 con 0x559460802c20 2024-04-02T14:37:57.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.520+0000 7fbf19600640 10 client.26387 handle_command_reply: tid=0 2024-04-02T14:37:57.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.520+0000 7fbf3979d740 1 client.26387 shutdown 2024-04-02T14:37:57.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.520+0000 7fbf3979d740 1 -- 172.21.15.90:0/1213561863 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5594607f26c0 msgr2=0x5594607f0170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:57.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.521+0000 7fbf3979d740 1 --2- 172.21.15.90:0/1213561863 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5594607f26c0 0x5594607f0170 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fbf2800cfd0 tx=0x7fbf2809dd50 comp rx=0 tx=0).stop 2024-04-02T14:37:57.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.521+0000 7fbf3979d740 1 -- 172.21.15.90:0/1213561863 shutdown_connections 2024-04-02T14:37:57.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.521+0000 7fbf3979d740 1 --2- 172.21.15.90:0/1213561863 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x559460802c20 0x559460805010 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:57.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.521+0000 7fbf3979d740 1 --2- 172.21.15.90:0/1213561863 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5594607f26c0 0x5594607f0170 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:57.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.521+0000 7fbf3979d740 1 --2- 172.21.15.90:0/1213561863 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5594607ea030 0x5594607efc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:57.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.521+0000 7fbf3979d740 1 --2- 172.21.15.90:0/1213561863 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5594607e9350 0x5594607ef6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:57.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.521+0000 7fbf3979d740 1 -- 172.21.15.90:0/1213561863 >> 172.21.15.90:0/1213561863 conn(0x55946077d270 msgr2=0x5594607e5260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:37:57.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.521+0000 7fbf19600640 0 client.26387 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:37:57.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.521+0000 7fbf3979d740 1 -- 172.21.15.90:0/1213561863 shutdown_connections 2024-04-02T14:37:57.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.522+0000 7fbf3979d740 1 -- 172.21.15.90:0/1213561863 wait complete. 2024-04-02T14:37:57.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.522+0000 7fbf3979d740 20 client.26387 trim_cache size 0 max 16384 2024-04-02T14:37:57.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.564+0000 7fbf3979d740 1 Processor -- start 2024-04-02T14:37:57.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.564+0000 7fbf3979d740 1 -- start start 2024-04-02T14:37:57.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.565+0000 7fbf3979d740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5594607e9350 0x5594607f1e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:57.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.565+0000 7fbf3979d740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5594607ea030 0x5594607fe430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:57.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.565+0000 7fbf3979d740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5594607f26c0 0x5594607e5890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:57.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.565+0000 7fbf3979d740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fbf301bc6b0 con 0x5594607e9350 2024-04-02T14:37:57.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.566+0000 7fbf3979d740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fbf301be1f0 con 0x5594607f26c0 2024-04-02T14:37:57.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.566+0000 7fbf3979d740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fbf301bcd80 con 0x5594607ea030 2024-04-02T14:37:57.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.565+0000 7fbf34a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5594607f26c0 0x5594607e5890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:57.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.566+0000 7fbf34a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5594607f26c0 0x5594607e5890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:34458/0 (socket says 172.21.15.90:34458) 2024-04-02T14:37:57.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.566+0000 7fbf34a00640 1 -- 172.21.15.90:0/4261700908 learned_addr learned my addr 172.21.15.90:0/4261700908 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:37:57.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.569+0000 7fbf34a00640 1 -- 172.21.15.90:0/4261700908 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5594607ea030 msgr2=0x5594607fe430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:37:57.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.566+0000 7fbf2fe00640 1 --2- 172.21.15.90:0/4261700908 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5594607e9350 0x5594607f1e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:57.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.567+0000 7fbf2f400640 1 --2- 172.21.15.90:0/4261700908 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5594607ea030 0x5594607fe430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:57.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.570+0000 7fbf34a00640 1 --2- 172.21.15.90:0/4261700908 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5594607ea030 0x5594607fe430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:57.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.570+0000 7fbf34a00640 1 -- 172.21.15.90:0/4261700908 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5594607e9350 msgr2=0x5594607f1e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:37:57.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.570+0000 7fbf34a00640 1 --2- 172.21.15.90:0/4261700908 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5594607e9350 0x5594607f1e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:57.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.570+0000 7fbf34a00640 1 -- 172.21.15.90:0/4261700908 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5594607edc40 con 0x5594607f26c0 2024-04-02T14:37:57.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.571+0000 7fbf2f400640 1 --2- 172.21.15.90:0/4261700908 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5594607ea030 0x5594607fe430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:37:57.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.571+0000 7fbf2fe00640 1 --2- 172.21.15.90:0/4261700908 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5594607e9350 0x5594607f1e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:37:57.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.571+0000 7fbf34a00640 1 --2- 172.21.15.90:0/4261700908 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5594607f26c0 0x5594607e5890 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fbf2800c640 tx=0x7fbf2809a030 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fd6703f776d2a21b server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:37:57.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.571+0000 7fbf1be00640 1 -- 172.21.15.90:0/4261700908 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf28081080 con 0x5594607f26c0 2024-04-02T14:37:57.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.572+0000 7fbf1be00640 1 -- 172.21.15.90:0/4261700908 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbf2809e3d0 con 0x5594607f26c0 2024-04-02T14:37:57.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.572+0000 7fbf1be00640 1 -- 172.21.15.90:0/4261700908 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf2801f030 con 0x5594607f26c0 2024-04-02T14:37:57.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.573+0000 7fbf3979d740 1 -- 172.21.15.90:0/4261700908 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5594607f26c0 msgr2=0x5594607e5890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:57.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.573+0000 7fbf3979d740 1 --2- 172.21.15.90:0/4261700908 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5594607f26c0 0x5594607e5890 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fbf2800c640 tx=0x7fbf2809a030 comp rx=0 tx=0).stop 2024-04-02T14:37:57.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.574+0000 7fbf3979d740 1 -- 172.21.15.90:0/4261700908 shutdown_connections 2024-04-02T14:37:57.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.574+0000 7fbf3979d740 1 --2- 172.21.15.90:0/4261700908 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5594607f26c0 0x5594607e5890 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:57.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.574+0000 7fbf3979d740 1 --2- 172.21.15.90:0/4261700908 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5594607ea030 0x5594607fe430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:57.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.574+0000 7fbf3979d740 1 --2- 172.21.15.90:0/4261700908 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5594607e9350 0x5594607f1e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:57.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.574+0000 7fbf3979d740 1 -- 172.21.15.90:0/4261700908 >> 172.21.15.90:0/4261700908 conn(0x55946077d270 msgr2=0x5594607e5180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:37:57.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.574+0000 7fbf3979d740 1 -- 172.21.15.90:0/4261700908 shutdown_connections 2024-04-02T14:37:57.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.574+0000 7fbf3979d740 1 -- 172.21.15.90:0/4261700908 wait complete. 2024-04-02T14:37:57.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.575+0000 7fbf3979d740 1 -- start start 2024-04-02T14:37:57.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.575+0000 7fbf3979d740 1 Processor -- start 2024-04-02T14:37:57.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.576+0000 7fbf3979d740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5594607e9350 0x5594607ecd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:57.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.576+0000 7fbf3979d740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5594607ea030 0x5594607ed240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:57.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.576+0000 7fbf3979d740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5594607f26c0 0x5594607f3520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:57.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.576+0000 7fbf2fe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5594607e9350 0x5594607ecd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:57.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.576+0000 7fbf2fe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5594607e9350 0x5594607ecd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:40534/0 (socket says 172.21.15.90:40534) 2024-04-02T14:37:57.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.577+0000 7fbf3979d740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fbf301bdb20 con 0x5594607f26c0 2024-04-02T14:37:57.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.577+0000 7fbf3979d740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fbf301bd450 con 0x5594607ea030 2024-04-02T14:37:57.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.577+0000 7fbf3979d740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fbf3010bee0 con 0x5594607e9350 2024-04-02T14:37:57.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.577+0000 7fbf2fe00640 1 -- 172.21.15.90:0/561416913 learned_addr learned my addr 172.21.15.90:0/561416913 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:37:57.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.577+0000 7fbf3979d740 10 client.0 _ll_register_callbacks cb 0x7fbf2809daa0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:37:57.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.577+0000 7fbf2f400640 1 --2- 172.21.15.90:0/561416913 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5594607ea030 0x5594607ed240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:57.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.577+0000 7fbf34a00640 1 --2- 172.21.15.90:0/561416913 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5594607f26c0 0x5594607f3520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:57.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.577+0000 7fbf2fe00640 1 -- 172.21.15.90:0/561416913 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5594607ea030 msgr2=0x5594607ed240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:37:57.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.577+0000 7fbf2fe00640 1 --2- 172.21.15.90:0/561416913 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5594607ea030 0x5594607ed240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:57.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.577+0000 7fbf2fe00640 1 -- 172.21.15.90:0/561416913 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5594607f26c0 msgr2=0x5594607f3520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:37:57.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.577+0000 7fbf2fe00640 1 --2- 172.21.15.90:0/561416913 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5594607f26c0 0x5594607f3520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:57.582 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.577+0000 7fbf2fe00640 1 -- 172.21.15.90:0/561416913 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5594607fb1b0 con 0x5594607e9350 2024-04-02T14:37:57.582 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.578+0000 7fbf2fe00640 1 --2- 172.21.15.90:0/561416913 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5594607e9350 0x5594607ecd00 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fbf20009930 tx=0x7fbf20009260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=34912580f8269382 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:37:57.582 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.578+0000 7fbf19600640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:37:57.582 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.579+0000 7fbf19600640 1 -- 172.21.15.90:0/561416913 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf20005450 con 0x5594607e9350 2024-04-02T14:37:57.582 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.579+0000 7fbf19600640 1 -- 172.21.15.90:0/561416913 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7fbf200055f0 con 0x5594607e9350 2024-04-02T14:37:57.582 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.579+0000 7fbf3979d740 1 -- 172.21.15.90:0/561416913 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbf2809e3d0 con 0x5594607e9350 2024-04-02T14:37:57.583 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.580+0000 7fbf19600640 1 -- 172.21.15.90:0/561416913 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fbf200532f0 con 0x5594607e9350 2024-04-02T14:37:57.583 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.580+0000 7fbf3979d740 10 client.26302 fetch_fsmap learned FSMap version 27 2024-04-02T14:37:57.583 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.580+0000 7fbf3979d740 1 -- 172.21.15.90:0/561416913 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5594608695c0 con 0x5594607e9350 2024-04-02T14:37:57.583 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.580+0000 7fbf19600640 1 -- 172.21.15.90:0/561416913 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbf2809e3d0 con 0x5594607e9350 2024-04-02T14:37:57.583 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.581+0000 7fbf19600640 1 -- 172.21.15.90:0/561416913 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fbf200035f0 con 0x5594607e9350 2024-04-02T14:37:57.583 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.581+0000 7fbf3979d740 10 client.26302 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:37:57.583 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.581+0000 7fbf3979d740 10 client.26302 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:37:57.583 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.581+0000 7fbf3979d740 20 client.26302 populate_metadata read hostname 'smithi090' 2024-04-02T14:37:57.583 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.581+0000 7fbf3979d740 1 --2- 172.21.15.90:0/561416913 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x559460869d40 0x55946086c160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:57.583 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.581+0000 7fbf3979d740 4 client.26302 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-02T14:37:57.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.581+0000 7fbf3979d740 1 -- 172.21.15.90:0/561416913 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5594606d2aa0 con 0x559460869d40 2024-04-02T14:37:57.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.581+0000 7fbf2f400640 1 --2- 172.21.15.90:0/561416913 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x559460869d40 0x55946086c160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:57.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.582+0000 7fbf19600640 10 client.26302 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:37:57.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.582+0000 7fbf2f400640 1 --2- 172.21.15.90:0/561416913 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x559460869d40 0x55946086c160 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5b4a6488f4f48ed7 server_cookie=f8d26458162ddacd in_seq=0 out_seq=0 2024-04-02T14:37:57.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.583+0000 7fbf19600640 1 -- 172.21.15.90:0/561416913 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5594606d2aa0 con 0x559460869d40 2024-04-02T14:37:57.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.583+0000 7fbf19600640 10 client.26302 handle_command_reply: tid=0 2024-04-02T14:37:57.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.583+0000 7fbf3979d740 1 client.26302 shutdown 2024-04-02T14:37:57.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.583+0000 7fbf3979d740 1 -- 172.21.15.90:0/561416913 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5594607e9350 msgr2=0x5594607ecd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:57.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.583+0000 7fbf3979d740 1 --2- 172.21.15.90:0/561416913 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5594607e9350 0x5594607ecd00 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fbf20009930 tx=0x7fbf20009260 comp rx=0 tx=0).stop 2024-04-02T14:37:57.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.583+0000 7fbf3979d740 1 -- 172.21.15.90:0/561416913 shutdown_connections 2024-04-02T14:37:57.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.583+0000 7fbf3979d740 1 --2- 172.21.15.90:0/561416913 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x559460869d40 0x55946086c160 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:57.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.583+0000 7fbf3979d740 1 --2- 172.21.15.90:0/561416913 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5594607f26c0 0x5594607f3520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:57.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.583+0000 7fbf3979d740 1 --2- 172.21.15.90:0/561416913 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5594607ea030 0x5594607ed240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:57.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.583+0000 7fbf3979d740 1 --2- 172.21.15.90:0/561416913 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5594607e9350 0x5594607ecd00 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:57.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.583+0000 7fbf19600640 0 client.26302 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:37:57.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.583+0000 7fbf3979d740 1 -- 172.21.15.90:0/561416913 >> 172.21.15.90:0/561416913 conn(0x55946077d270 msgr2=0x5594607e2590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:37:57.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.584+0000 7fbf3979d740 1 -- 172.21.15.90:0/561416913 shutdown_connections 2024-04-02T14:37:57.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.584+0000 7fbf3979d740 1 -- 172.21.15.90:0/561416913 wait complete. 2024-04-02T14:37:57.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.584+0000 7fbf3979d740 20 client.26302 trim_cache size 0 max 16384 2024-04-02T14:37:57.587 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-02T14:37:57.588 INFO:teuthology.orchestra.run.smithi090.stdout: "return_code": 0, 2024-04-02T14:37:57.588 INFO:teuthology.orchestra.run.smithi090.stdout: "scrub_tag": "b5d6821d-4252-4c22-a236-a34c809c0942", 2024-04-02T14:37:57.588 INFO:teuthology.orchestra.run.smithi090.stdout: "mode": "asynchronous" 2024-04-02T14:37:57.588 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-02T14:37:57.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.585+0000 7fbf36a00640 1 -- 172.21.15.90:0/3295335019 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fbf140a3de0 msgr2=0x7fbf140a62a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:57.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.585+0000 7fbf36a00640 1 --2- 172.21.15.90:0/3295335019 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fbf140a3de0 0x7fbf140a62a0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fbf20009900 tx=0x7fbf20000bc0 comp rx=0 tx=0).stop 2024-04-02T14:37:57.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.585+0000 7fbf36a00640 1 -- 172.21.15.90:0/3295335019 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbf30112940 msgr2=0x7fbf3019e800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:57.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.585+0000 7fbf36a00640 1 --2- 172.21.15.90:0/3295335019 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbf30112940 0x7fbf3019e800 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fbf2800e9d0 tx=0x7fbf2800ee90 comp rx=0 tx=0).stop 2024-04-02T14:37:57.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.586+0000 7fbf36a00640 1 -- 172.21.15.90:0/3295335019 shutdown_connections 2024-04-02T14:37:57.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.586+0000 7fbf36a00640 1 --2- 172.21.15.90:0/3295335019 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fbf140a3de0 0x7fbf140a62a0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:57.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.586+0000 7fbf36a00640 1 --2- 172.21.15.90:0/3295335019 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbf30112940 0x7fbf3019e800 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:57.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.586+0000 7fbf36a00640 1 --2- 172.21.15.90:0/3295335019 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbf30076950 0x7fbf301a4b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:57.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.586+0000 7fbf36a00640 1 --2- 172.21.15.90:0/3295335019 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbf30076030 0x7fbf301a4640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:57.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.586+0000 7fbf36a00640 1 -- 172.21.15.90:0/3295335019 >> 172.21.15.90:0/3295335019 conn(0x7fbf300ff770 msgr2=0x7fbf30113f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:37:57.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.586+0000 7fbf36a00640 1 -- 172.21.15.90:0/3295335019 shutdown_connections 2024-04-02T14:37:57.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.586+0000 7fbf36a00640 1 -- 172.21.15.90:0/3295335019 wait complete. 2024-04-02T14:37:57.617 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-02T14:37:57.617 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-02T14:37:57.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:37:57 smithi177 ceph-mon[36431]: pgmap v95: 97 pgs: 97 active+clean; 1.7 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 7.5 MiB/s wr, 634 op/s 2024-04-02T14:37:57.938 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:37:57 smithi090 ceph-mon[31674]: pgmap v95: 97 pgs: 97 active+clean; 1.7 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 7.5 MiB/s wr, 634 op/s 2024-04-02T14:37:57.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.946+0000 7f67ee200640 1 -- 172.21.15.90:0/3239653755 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f67e80ffbd0 msgr2=0x7f67e8100050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:57.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.946+0000 7f67ee200640 1 --2- 172.21.15.90:0/3239653755 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f67e80ffbd0 0x7f67e8100050 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f67d8009860 tx=0x7f67d802bb70 comp rx=0 tx=0).stop 2024-04-02T14:37:57.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.947+0000 7f67ee200640 1 -- 172.21.15.90:0/3239653755 shutdown_connections 2024-04-02T14:37:57.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.947+0000 7f67ee200640 1 --2- 172.21.15.90:0/3239653755 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f67e8100590 0x7f67e810e940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:57.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.947+0000 7f67ee200640 1 --2- 172.21.15.90:0/3239653755 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f67e80ffbd0 0x7f67e8100050 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:57.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.947+0000 7f67ee200640 1 --2- 172.21.15.90:0/3239653755 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f67e8107be0 0x7f67e8107fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:57.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.947+0000 7f67ee200640 1 -- 172.21.15.90:0/3239653755 >> 172.21.15.90:0/3239653755 conn(0x7f67e8076b50 msgr2=0x7f67e80fdee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:37:57.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.948+0000 7f67ee200640 1 -- 172.21.15.90:0/3239653755 shutdown_connections 2024-04-02T14:37:57.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.948+0000 7f67ee200640 1 -- 172.21.15.90:0/3239653755 wait complete. 2024-04-02T14:37:57.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.950+0000 7f67ee200640 1 Processor -- start 2024-04-02T14:37:57.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.950+0000 7f67ee200640 1 -- start start 2024-04-02T14:37:57.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.951+0000 7f67ee200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f67e80ffbd0 0x7f67e80732b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:57.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.951+0000 7f67ee200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f67e8100590 0x7f67e80737f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:57.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.951+0000 7f67ee200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f67e8107be0 0x7f67e806c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:57.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.951+0000 7f67ee200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f67e810fcc0 con 0x7f67e80ffbd0 2024-04-02T14:37:57.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.951+0000 7f67ee200640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f67e810fb40 con 0x7f67e8107be0 2024-04-02T14:37:57.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.951+0000 7f67ee200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f67e810fe40 con 0x7f67e8100590 2024-04-02T14:37:57.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.952+0000 7f67e7e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f67e8107be0 0x7f67e806c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:57.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.952+0000 7f67e7e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f67e8107be0 0x7f67e806c510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:34488/0 (socket says 172.21.15.90:34488) 2024-04-02T14:37:57.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.952+0000 7f67e7e00640 1 -- 172.21.15.90:0/2486305087 learned_addr learned my addr 172.21.15.90:0/2486305087 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:37:57.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.953+0000 7f67e7400640 1 --2- 172.21.15.90:0/2486305087 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f67e80ffbd0 0x7f67e80732b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:57.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.954+0000 7f67e7e00640 1 -- 172.21.15.90:0/2486305087 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f67e8100590 msgr2=0x7f67e80737f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:57.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.954+0000 7f67e7e00640 1 --2- 172.21.15.90:0/2486305087 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f67e8100590 0x7f67e80737f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:57.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.954+0000 7f67e7e00640 1 -- 172.21.15.90:0/2486305087 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f67e80ffbd0 msgr2=0x7f67e80732b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:57.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.954+0000 7f67e7e00640 1 --2- 172.21.15.90:0/2486305087 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f67e80ffbd0 0x7f67e80732b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:57.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.954+0000 7f67e7e00640 1 -- 172.21.15.90:0/2486305087 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67e806cdd0 con 0x7f67e8107be0 2024-04-02T14:37:57.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.954+0000 7f67e7400640 1 --2- 172.21.15.90:0/2486305087 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f67e80ffbd0 0x7f67e80732b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:37:57.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.956+0000 7f67e7e00640 1 --2- 172.21.15.90:0/2486305087 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f67e8107be0 0x7f67e806c510 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f67e000efd0 tx=0x7f67e000c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:37:57.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.957+0000 7f67dfe00640 1 -- 172.21.15.90:0/2486305087 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67e0003b70 con 0x7f67e8107be0 2024-04-02T14:37:57.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.957+0000 7f67dfe00640 1 -- 172.21.15.90:0/2486305087 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f67e0003d10 con 0x7f67e8107be0 2024-04-02T14:37:57.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.957+0000 7f67dfe00640 1 -- 172.21.15.90:0/2486305087 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67e00109f0 con 0x7f67e8107be0 2024-04-02T14:37:57.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.957+0000 7f67ee200640 1 -- 172.21.15.90:0/2486305087 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67e806d0c0 con 0x7f67e8107be0 2024-04-02T14:37:57.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.957+0000 7f67ee200640 1 -- 172.21.15.90:0/2486305087 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67e8071070 con 0x7f67e8107be0 2024-04-02T14:37:57.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.964+0000 7f67f0fd1740 1 Processor -- start 2024-04-02T14:37:57.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.965+0000 7f67f0fd1740 1 -- start start 2024-04-02T14:37:57.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.965+0000 7f67dfe00640 1 -- 172.21.15.90:0/2486305087 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f67e0010b90 con 0x7f67e8107be0 2024-04-02T14:37:57.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.965+0000 7f67f0fd1740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5619931485c0 0x5619931489a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:57.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.965+0000 7f67f0fd1740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561993148f70 0x561993159f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:57.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.965+0000 7f67f0fd1740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56199315a4c0 0x56199315c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:57.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.965+0000 7f67f0fd1740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5619930ac770 con 0x5619931485c0 2024-04-02T14:37:57.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.965+0000 7f67f0fd1740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x561993054700 con 0x56199315a4c0 2024-04-02T14:37:57.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.965+0000 7f67f0fd1740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5619930a8720 con 0x561993148f70 2024-04-02T14:37:57.969 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.967+0000 7f67e7400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561993148f70 0x561993159f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:57.970 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.967+0000 7f67e7400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561993148f70 0x561993159f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:40550/0 (socket says 172.21.15.90:40550) 2024-04-02T14:37:57.970 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.967+0000 7f67e7400640 1 -- 172.21.15.90:0/1693659824 learned_addr learned my addr 172.21.15.90:0/1693659824 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:37:57.970 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.967+0000 7f67dfe00640 1 --2- 172.21.15.90:0/2486305087 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f67cc0a3e20 0x7f67cc0a62e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:57.970 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.968+0000 7f67e6a00640 1 --2- 172.21.15.90:0/1693659824 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56199315a4c0 0x56199315c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:57.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.968+0000 7f67dfe00640 1 -- 172.21.15.90:0/2486305087 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f67e00d0940 con 0x7f67e8107be0 2024-04-02T14:37:57.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.968+0000 7f67e7e00640 1 --2- 172.21.15.90:0/2486305087 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f67cc0a3e20 0x7f67cc0a62e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:57.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.968+0000 7f67e7400640 1 -- 172.21.15.90:0/1693659824 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56199315a4c0 msgr2=0x56199315c8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:37:57.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.968+0000 7f67e7400640 1 --2- 172.21.15.90:0/1693659824 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56199315a4c0 0x56199315c8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:57.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.968+0000 7f67e7400640 1 -- 172.21.15.90:0/1693659824 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5619931485c0 msgr2=0x5619931489a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:37:57.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.968+0000 7f67e7400640 1 --2- 172.21.15.90:0/1693659824 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5619931485c0 0x5619931489a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:57.972 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.968+0000 7f67e7400640 1 -- 172.21.15.90:0/1693659824 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561992e8a7d0 con 0x561993148f70 2024-04-02T14:37:57.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.971+0000 7f67e7400640 1 --2- 172.21.15.90:0/1693659824 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561993148f70 0x561993159f80 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f67d000fcc0 tx=0x7f67d0009260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5a7d64ee52e69bdc server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:37:57.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.971+0000 7f67e7e00640 1 --2- 172.21.15.90:0/2486305087 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f67cc0a3e20 0x7f67cc0a62e0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f67e001a040 tx=0x7f67e0004920 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:37:57.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.972+0000 7f67d7e00640 1 -- 172.21.15.90:0/1693659824 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67d001b020 con 0x561993148f70 2024-04-02T14:37:57.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.972+0000 7f67d7e00640 1 -- 172.21.15.90:0/1693659824 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f67d0003610 con 0x561993148f70 2024-04-02T14:37:57.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.972+0000 7f67d7e00640 1 -- 172.21.15.90:0/1693659824 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67d0017640 con 0x561993148f70 2024-04-02T14:37:57.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.973+0000 7f67f0fd1740 1 -- 172.21.15.90:0/1693659824 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561993148f70 msgr2=0x561993159f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:57.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.973+0000 7f67f0fd1740 1 --2- 172.21.15.90:0/1693659824 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561993148f70 0x561993159f80 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f67d000fcc0 tx=0x7f67d0009260 comp rx=0 tx=0).stop 2024-04-02T14:37:57.977 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.975+0000 7f67f0fd1740 1 -- 172.21.15.90:0/1693659824 shutdown_connections 2024-04-02T14:37:57.977 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.975+0000 7f67f0fd1740 1 --2- 172.21.15.90:0/1693659824 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56199315a4c0 0x56199315c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:57.977 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.975+0000 7f67f0fd1740 1 --2- 172.21.15.90:0/1693659824 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561993148f70 0x561993159f80 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:57.977 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.975+0000 7f67f0fd1740 1 --2- 172.21.15.90:0/1693659824 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5619931485c0 0x5619931489a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:57.977 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.975+0000 7f67f0fd1740 1 -- 172.21.15.90:0/1693659824 >> 172.21.15.90:0/1693659824 conn(0x561993142ee0 msgr2=0x5619931432c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:37:57.978 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.977+0000 7f67f0fd1740 1 -- 172.21.15.90:0/1693659824 shutdown_connections 2024-04-02T14:37:57.979 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.977+0000 7f67f0fd1740 1 -- 172.21.15.90:0/1693659824 wait complete. 2024-04-02T14:37:57.980 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.978+0000 7f67f0fd1740 1 -- start start 2024-04-02T14:37:57.980 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.978+0000 7f67f0fd1740 1 Processor -- start 2024-04-02T14:37:57.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.980+0000 7f67f0fd1740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5619931485c0 0x561993154220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:57.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.980+0000 7f67f0fd1740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561993148f70 0x561993159f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:57.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.980+0000 7f67f0fd1740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56199315a4c0 0x56199314f620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:57.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.980+0000 7f67f0fd1740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x561993149b10 con 0x56199315a4c0 2024-04-02T14:37:57.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.980+0000 7f67f0fd1740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x561993067ac0 con 0x5619931485c0 2024-04-02T14:37:57.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.980+0000 7f67f0fd1740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x561993149c90 con 0x561993148f70 2024-04-02T14:37:57.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.980+0000 7f67f0fd1740 10 client.0 _ll_register_callbacks cb 0x56199306a180 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:37:57.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.981+0000 7f67e6a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5619931485c0 0x561993154220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:57.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.981+0000 7f67e6a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5619931485c0 0x561993154220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:34504/0 (socket says 172.21.15.90:34504) 2024-04-02T14:37:57.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.981+0000 7f67e6a00640 1 -- 172.21.15.90:0/1579446958 learned_addr learned my addr 172.21.15.90:0/1579446958 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:37:57.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.982+0000 7f67e7400640 1 --2- 172.21.15.90:0/1579446958 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561993148f70 0x561993159f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:57.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.983+0000 7f67e7400640 1 -- 172.21.15.90:0/1579446958 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5619931485c0 msgr2=0x561993154220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:37:57.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.983+0000 7f67e7400640 1 --2- 172.21.15.90:0/1579446958 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5619931485c0 0x561993154220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:57.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.983+0000 7f67e7400640 1 -- 172.21.15.90:0/1579446958 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56199315a4c0 msgr2=0x56199314f620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:37:57.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.983+0000 7f67e7400640 1 --2- 172.21.15.90:0/1579446958 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56199315a4c0 0x56199314f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:57.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.983+0000 7f67e7400640 1 -- 172.21.15.90:0/1579446958 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561992dc03f0 con 0x561993148f70 2024-04-02T14:37:57.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.983+0000 7f67e6a00640 1 --2- 172.21.15.90:0/1579446958 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5619931485c0 0x561993154220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:37:57.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.984+0000 7f67e7400640 1 --2- 172.21.15.90:0/1579446958 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561993148f70 0x561993159f40 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f67d0000c00 tx=0x7f67d000f8c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=487465788196217 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:37:57.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.984+0000 7f67d4c00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:37:57.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.984+0000 7f67d4c00640 1 -- 172.21.15.90:0/1579446958 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67d0025020 con 0x561993148f70 2024-04-02T14:37:57.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.984+0000 7f67d4c00640 1 -- 172.21.15.90:0/1579446958 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f67d00037c0 con 0x561993148f70 2024-04-02T14:37:57.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.984+0000 7f67f0fd1740 1 -- 172.21.15.90:0/1579446958 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f67d0003610 con 0x561993148f70 2024-04-02T14:37:57.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.986+0000 7f67d4c00640 1 -- 172.21.15.90:0/1579446958 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f67d0030050 con 0x561993148f70 2024-04-02T14:37:57.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.986+0000 7f67f0fd1740 10 client.26311 fetch_fsmap learned FSMap version 27 2024-04-02T14:37:57.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.986+0000 7f67f0fd1740 1 -- 172.21.15.90:0/1579446958 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x561992d800f0 con 0x561993148f70 2024-04-02T14:37:57.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.987+0000 7f67d4c00640 1 -- 172.21.15.90:0/1579446958 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f67d0003610 con 0x561993148f70 2024-04-02T14:37:57.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.987+0000 7f67d4c00640 1 -- 172.21.15.90:0/1579446958 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f67d0017400 con 0x561993148f70 2024-04-02T14:37:57.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.987+0000 7f67f0fd1740 10 client.26311 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:37:57.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.987+0000 7f67f0fd1740 10 client.26311 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:37:57.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.987+0000 7f67f0fd1740 20 client.26311 populate_metadata read hostname 'smithi090' 2024-04-02T14:37:57.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.988+0000 7f67f0fd1740 1 --2- 172.21.15.90:0/1579446958 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x561993172cd0 0x5619931750c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:57.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.988+0000 7f67f0fd1740 4 client.26311 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:37:57.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.988+0000 7f67f0fd1740 1 -- 172.21.15.90:0/1579446958 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56199304d630 con 0x561993172cd0 2024-04-02T14:37:57.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.988+0000 7f67e6a00640 1 --2- 172.21.15.90:0/1579446958 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x561993172cd0 0x5619931750c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:57.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.990+0000 7f67e6a00640 1 --2- 172.21.15.90:0/1579446958 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x561993172cd0 0x5619931750c0 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7f1f19b229258fbd server_cookie=109f3d544a6a4aa1 in_seq=0 out_seq=0 2024-04-02T14:37:57.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.990+0000 7f67d4c00640 10 client.26311 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:37:57.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.991+0000 7f67d4c00640 1 -- 172.21.15.90:0/1579446958 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56199304d630 con 0x561993172cd0 2024-04-02T14:37:57.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.991+0000 7f67d4c00640 10 client.26311 handle_command_reply: tid=0 2024-04-02T14:37:57.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.991+0000 7f67f0fd1740 1 client.26311 shutdown 2024-04-02T14:37:57.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.992+0000 7f67f0fd1740 1 -- 172.21.15.90:0/1579446958 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561993148f70 msgr2=0x561993159f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:57.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.992+0000 7f67f0fd1740 1 --2- 172.21.15.90:0/1579446958 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561993148f70 0x561993159f40 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f67d0000c00 tx=0x7f67d000f8c0 comp rx=0 tx=0).stop 2024-04-02T14:37:57.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.992+0000 7f67f0fd1740 1 -- 172.21.15.90:0/1579446958 shutdown_connections 2024-04-02T14:37:57.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.992+0000 7f67f0fd1740 1 --2- 172.21.15.90:0/1579446958 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x561993172cd0 0x5619931750c0 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:57.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.992+0000 7f67f0fd1740 1 --2- 172.21.15.90:0/1579446958 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56199315a4c0 0x56199314f620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:57.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.992+0000 7f67f0fd1740 1 --2- 172.21.15.90:0/1579446958 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561993148f70 0x561993159f40 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:57.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.992+0000 7f67f0fd1740 1 --2- 172.21.15.90:0/1579446958 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5619931485c0 0x561993154220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:57.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.992+0000 7f67f0fd1740 1 -- 172.21.15.90:0/1579446958 >> 172.21.15.90:0/1579446958 conn(0x561993142ee0 msgr2=0x561993144c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:37:57.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.993+0000 7f67d4c00640 0 client.26311 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:37:57.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.993+0000 7f67f0fd1740 1 -- 172.21.15.90:0/1579446958 shutdown_connections 2024-04-02T14:37:57.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.993+0000 7f67f0fd1740 1 -- 172.21.15.90:0/1579446958 wait complete. 2024-04-02T14:37:57.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:57.994+0000 7f67f0fd1740 20 client.26311 trim_cache size 0 max 16384 2024-04-02T14:37:58.034 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:58.031+0000 7f67f0fd1740 1 Processor -- start 2024-04-02T14:37:58.034 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:58.032+0000 7f67f0fd1740 1 -- start start 2024-04-02T14:37:58.034 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:58.032+0000 7f67f0fd1740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5619931485c0 0x561993174ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:58.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:58.032+0000 7f67f0fd1740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561993148f70 0x561993141de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:58.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:58.032+0000 7f67f0fd1740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56199315a4c0 0x56199317a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:58.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:58.032+0000 7f67f0fd1740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f67e81b3f30 con 0x5619931485c0 2024-04-02T14:37:58.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:58.032+0000 7f67f0fd1740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f67e81b5a70 con 0x561993148f70 2024-04-02T14:37:58.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:58.032+0000 7f67f0fd1740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f67e81b4600 con 0x56199315a4c0 2024-04-02T14:37:58.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:58.032+0000 7f67e7400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561993148f70 0x561993141de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:58.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:58.032+0000 7f67e7400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561993148f70 0x561993141de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:34510/0 (socket says 172.21.15.90:34510) 2024-04-02T14:37:58.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:58.032+0000 7f67e7400640 1 -- 172.21.15.90:0/1834837422 learned_addr learned my addr 172.21.15.90:0/1834837422 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:37:58.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:58.033+0000 7f67e6a00640 1 --2- 172.21.15.90:0/1834837422 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56199315a4c0 0x56199317a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:58.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:58.033+0000 7f67e7400640 1 -- 172.21.15.90:0/1834837422 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56199315a4c0 msgr2=0x56199317a6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:37:58.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:58.033+0000 7f67e7400640 1 --2- 172.21.15.90:0/1834837422 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56199315a4c0 0x56199317a6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:58.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:58.033+0000 7f67e7400640 1 -- 172.21.15.90:0/1834837422 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5619931485c0 msgr2=0x561993174ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:37:58.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:58.033+0000 7f67e7400640 1 --2- 172.21.15.90:0/1834837422 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5619931485c0 0x561993174ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:58.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:58.033+0000 7f67e7400640 1 -- 172.21.15.90:0/1834837422 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56199314e5d0 con 0x561993148f70 2024-04-02T14:37:58.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:58.033+0000 7f67e6a00640 1 --2- 172.21.15.90:0/1834837422 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56199315a4c0 0x56199317a6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:37:58.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:58.033+0000 7f67e7400640 1 --2- 172.21.15.90:0/1834837422 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561993148f70 0x561993141de0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f67d0003f70 tx=0x7f67d0005910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3f7be0efbdea99cb server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:37:58.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:58.033+0000 7f67d7400640 1 -- 172.21.15.90:0/1834837422 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67d0039070 con 0x561993148f70 2024-04-02T14:37:58.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:58.033+0000 7f67d7400640 1 -- 172.21.15.90:0/1834837422 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f67d0005550 con 0x561993148f70 2024-04-02T14:37:58.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:58.033+0000 7f67d7400640 1 -- 172.21.15.90:0/1834837422 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67d0025030 con 0x561993148f70 2024-04-02T14:37:58.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:58.034+0000 7f67f0fd1740 1 -- 172.21.15.90:0/1834837422 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561993148f70 msgr2=0x561993141de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:58.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:58.034+0000 7f67f0fd1740 1 --2- 172.21.15.90:0/1834837422 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561993148f70 0x561993141de0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f67d0003f70 tx=0x7f67d0005910 comp rx=0 tx=0).stop 2024-04-02T14:37:58.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:58.034+0000 7f67f0fd1740 1 -- 172.21.15.90:0/1834837422 shutdown_connections 2024-04-02T14:37:58.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:58.034+0000 7f67f0fd1740 1 --2- 172.21.15.90:0/1834837422 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56199315a4c0 0x56199317a6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:58.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:58.034+0000 7f67f0fd1740 1 --2- 172.21.15.90:0/1834837422 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561993148f70 0x561993141de0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:58.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:58.034+0000 7f67f0fd1740 1 --2- 172.21.15.90:0/1834837422 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5619931485c0 0x561993174ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:58.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:58.034+0000 7f67f0fd1740 1 -- 172.21.15.90:0/1834837422 >> 172.21.15.90:0/1834837422 conn(0x561993142ee0 msgr2=0x5619931416d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:37:58.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:58.035+0000 7f67f0fd1740 1 -- 172.21.15.90:0/1834837422 shutdown_connections 2024-04-02T14:37:58.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:58.035+0000 7f67f0fd1740 1 -- 172.21.15.90:0/1834837422 wait complete. 2024-04-02T14:37:58.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:58.035+0000 7f67f0fd1740 1 -- start start 2024-04-02T14:37:58.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:58.035+0000 7f67f0fd1740 1 Processor -- start 2024-04-02T14:37:58.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:58.036+0000 7f67f0fd1740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5619931485c0 0x561993181ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:58.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:58.036+0000 7f67f0fd1740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561993148f70 0x561993182010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:58.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:58.036+0000 7f67f0fd1740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56199315a4c0 0x5619931d7a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:58.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:58.036+0000 7f67f0fd1740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f67e81b53a0 con 0x561993148f70 2024-04-02T14:37:58.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:58.036+0000 7f67f0fd1740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f67e81b4cd0 con 0x5619931485c0 2024-04-02T14:37:58.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:58.036+0000 7f67f0fd1740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f67e81b1f10 con 0x56199315a4c0 2024-04-02T14:37:58.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:58.036+0000 7f67f0fd1740 10 client.0 _ll_register_callbacks cb 0x7f67d000fae0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:37:58.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:58.036+0000 7f67e6a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5619931485c0 0x561993181ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:58.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:58.036+0000 7f67e6a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5619931485c0 0x561993181ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:34522/0 (socket says 172.21.15.90:34522) 2024-04-02T14:37:58.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:58.036+0000 7f67e6a00640 1 -- 172.21.15.90:0/2907418097 learned_addr learned my addr 172.21.15.90:0/2907418097 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:37:58.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:58.036+0000 7f67e6a00640 1 --2- 172.21.15.90:0/2907418097 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56199315a4c0 0x5619931d7a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:58.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:58.037+0000 7f67e7400640 1 --2- 172.21.15.90:0/2907418097 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561993148f70 0x561993182010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:58.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:58.037+0000 7f67e6a00640 1 -- 172.21.15.90:0/2907418097 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56199315a4c0 msgr2=0x5619931d7a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:37:58.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:58.037+0000 7f67e6a00640 1 --2- 172.21.15.90:0/2907418097 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56199315a4c0 0x5619931d7a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:58.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:58.037+0000 7f67e6a00640 1 -- 172.21.15.90:0/2907418097 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561993148f70 msgr2=0x561993182010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:37:58.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:58.037+0000 7f67e6a00640 1 --2- 172.21.15.90:0/2907418097 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561993148f70 0x561993182010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:58.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:58.037+0000 7f67e6a00640 1 -- 172.21.15.90:0/2907418097 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561993182910 con 0x5619931485c0 2024-04-02T14:37:58.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:58.038+0000 7f67e6a00640 1 --2- 172.21.15.90:0/2907418097 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5619931485c0 0x561993181ad0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f67d8032570 tx=0x7f67d802e040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=86055e9b7cf94077 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:37:58.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:58.038+0000 7f67d4c00640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:37:58.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:58.038+0000 7f67d4c00640 1 -- 172.21.15.90:0/2907418097 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67d8039040 con 0x5619931485c0 2024-04-02T14:37:58.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:58.039+0000 7f67f0fd1740 1 -- 172.21.15.90:0/2907418097 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f67d0005550 con 0x5619931485c0 2024-04-02T14:37:58.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:58.038+0000 7f67d4c00640 1 -- 172.21.15.90:0/2907418097 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f67d8005220 con 0x5619931485c0 2024-04-02T14:37:58.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:58.039+0000 7f67d4c00640 1 -- 172.21.15.90:0/2907418097 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f67d8005420 con 0x5619931485c0 2024-04-02T14:37:58.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:58.039+0000 7f67f0fd1740 10 client.26408 fetch_fsmap learned FSMap version 27 2024-04-02T14:37:58.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:58.039+0000 7f67f0fd1740 1 -- 172.21.15.90:0/2907418097 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5619931d9030 con 0x5619931485c0 2024-04-02T14:37:58.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:58.039+0000 7f67d4c00640 1 -- 172.21.15.90:0/2907418097 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f67d0005550 con 0x5619931485c0 2024-04-02T14:37:58.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:58.041+0000 7f67d4c00640 1 -- 172.21.15.90:0/2907418097 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f67d80327e0 con 0x5619931485c0 2024-04-02T14:37:58.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:58.041+0000 7f67f0fd1740 10 client.26408 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:37:58.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:58.041+0000 7f67f0fd1740 10 client.26408 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:37:58.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:58.041+0000 7f67f0fd1740 20 client.26408 populate_metadata read hostname 'smithi090' 2024-04-02T14:37:58.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:58.041+0000 7f67f0fd1740 1 --2- 172.21.15.90:0/2907418097 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5619931d9700 0x5619931dbb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:37:58.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:58.041+0000 7f67f0fd1740 4 client.26408 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-02T14:37:58.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:58.041+0000 7f67f0fd1740 1 -- 172.21.15.90:0/2907418097 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5619930b9cb0 con 0x5619931d9700 2024-04-02T14:37:58.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:58.042+0000 7f67e7400640 1 --2- 172.21.15.90:0/2907418097 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5619931d9700 0x5619931dbb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:37:58.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:58.042+0000 7f67e7400640 1 --2- 172.21.15.90:0/2907418097 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5619931d9700 0x5619931dbb20 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2480f0768852ce7f server_cookie=58826125cd67e85f in_seq=0 out_seq=0 2024-04-02T14:37:58.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:58.042+0000 7f67d4c00640 10 client.26408 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:37:58.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:58.043+0000 7f67d4c00640 1 -- 172.21.15.90:0/2907418097 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x5619930b9cb0 con 0x5619931d9700 2024-04-02T14:37:58.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:58.043+0000 7f67d4c00640 10 client.26408 handle_command_reply: tid=0 2024-04-02T14:37:58.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:58.043+0000 7f67f0fd1740 1 client.26408 shutdown 2024-04-02T14:37:58.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:58.043+0000 7f67f0fd1740 1 -- 172.21.15.90:0/2907418097 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5619931485c0 msgr2=0x561993181ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:58.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:58.043+0000 7f67f0fd1740 1 --2- 172.21.15.90:0/2907418097 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5619931485c0 0x561993181ad0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f67d8032570 tx=0x7f67d802e040 comp rx=0 tx=0).stop 2024-04-02T14:37:58.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:58.043+0000 7f67f0fd1740 1 -- 172.21.15.90:0/2907418097 shutdown_connections 2024-04-02T14:37:58.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:58.043+0000 7f67f0fd1740 1 --2- 172.21.15.90:0/2907418097 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5619931d9700 0x5619931dbb20 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:58.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:58.043+0000 7f67f0fd1740 1 --2- 172.21.15.90:0/2907418097 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56199315a4c0 0x5619931d7a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:58.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:58.043+0000 7f67f0fd1740 1 --2- 172.21.15.90:0/2907418097 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561993148f70 0x561993182010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:58.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:58.043+0000 7f67f0fd1740 1 --2- 172.21.15.90:0/2907418097 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5619931485c0 0x561993181ad0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:58.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:58.043+0000 7f67d4c00640 0 client.26408 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:37:58.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:58.043+0000 7f67f0fd1740 1 -- 172.21.15.90:0/2907418097 >> 172.21.15.90:0/2907418097 conn(0x561993142ee0 msgr2=0x561993141f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:37:58.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:58.044+0000 7f67f0fd1740 1 -- 172.21.15.90:0/2907418097 shutdown_connections 2024-04-02T14:37:58.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:58.044+0000 7f67f0fd1740 1 -- 172.21.15.90:0/2907418097 wait complete. 2024-04-02T14:37:58.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:58.044+0000 7f67f0fd1740 20 client.26408 trim_cache size 0 max 16384 2024-04-02T14:37:58.048 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-02T14:37:58.048 INFO:teuthology.orchestra.run.smithi090.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-02T14:37:58.048 INFO:teuthology.orchestra.run.smithi090.stdout: "scrubs": { 2024-04-02T14:37:58.048 INFO:teuthology.orchestra.run.smithi090.stdout: "b5d6821d-4252-4c22-a236-a34c809c0942": { 2024-04-02T14:37:58.048 INFO:teuthology.orchestra.run.smithi090.stdout: "path": "/", 2024-04-02T14:37:58.048 INFO:teuthology.orchestra.run.smithi090.stdout: "tag": "b5d6821d-4252-4c22-a236-a34c809c0942", 2024-04-02T14:37:58.048 INFO:teuthology.orchestra.run.smithi090.stdout: "options": "recursive,force" 2024-04-02T14:37:58.048 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:37:58.049 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:37:58.049 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-02T14:37:58.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:58.045+0000 7f67ee200640 1 -- 172.21.15.90:0/2486305087 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f67cc0a3e20 msgr2=0x7f67cc0a62e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:58.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:58.045+0000 7f67ee200640 1 --2- 172.21.15.90:0/2486305087 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f67cc0a3e20 0x7f67cc0a62e0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f67e001a040 tx=0x7f67e0004920 comp rx=0 tx=0).stop 2024-04-02T14:37:58.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:58.045+0000 7f67ee200640 1 -- 172.21.15.90:0/2486305087 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f67e8107be0 msgr2=0x7f67e806c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:37:58.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:58.045+0000 7f67ee200640 1 --2- 172.21.15.90:0/2486305087 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f67e8107be0 0x7f67e806c510 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f67e000efd0 tx=0x7f67e000c6a0 comp rx=0 tx=0).stop 2024-04-02T14:37:58.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:58.045+0000 7f67ee200640 1 -- 172.21.15.90:0/2486305087 shutdown_connections 2024-04-02T14:37:58.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:58.045+0000 7f67ee200640 1 --2- 172.21.15.90:0/2486305087 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f67cc0a3e20 0x7f67cc0a62e0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:58.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:58.045+0000 7f67ee200640 1 --2- 172.21.15.90:0/2486305087 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f67e8107be0 0x7f67e806c510 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:58.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:58.045+0000 7f67ee200640 1 --2- 172.21.15.90:0/2486305087 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f67e8100590 0x7f67e80737f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:58.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:58.045+0000 7f67ee200640 1 --2- 172.21.15.90:0/2486305087 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f67e80ffbd0 0x7f67e80732b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:37:58.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:58.045+0000 7f67ee200640 1 -- 172.21.15.90:0/2486305087 >> 172.21.15.90:0/2486305087 conn(0x7f67e8076b50 msgr2=0x7f67e80fe380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:37:58.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:58.046+0000 7f67ee200640 1 -- 172.21.15.90:0/2486305087 shutdown_connections 2024-04-02T14:37:58.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:37:58.046+0000 7f67ee200640 1 -- 172.21.15.90:0/2486305087 wait complete. 2024-04-02T14:37:58.076 INFO:tasks.cephfs.filesystem:scrub status for tag:b5d6821d-4252-4c22-a236-a34c809c0942 - {'path': '/', 'tag': 'b5d6821d-4252-4c22-a236-a34c809c0942', 'options': 'recursive,force'} 2024-04-02T14:37:58.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:37:57 smithi092 ceph-mon[36497]: pgmap v95: 97 pgs: 97 active+clean; 1.7 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 7.5 MiB/s wr, 634 op/s 2024-04-02T14:37:58.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:37:58 smithi177 ceph-mon[36431]: scrub queued for path: / 2024-04-02T14:37:58.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:37:58 smithi177 ceph-mon[36431]: scrub summary: idle+waiting paths [/] 2024-04-02T14:37:58.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:37:58 smithi177 ceph-mon[36431]: scrub summary: active paths [/] 2024-04-02T14:37:58.938 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:37:58 smithi090 ceph-mon[31674]: scrub queued for path: / 2024-04-02T14:37:58.938 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:37:58 smithi090 ceph-mon[31674]: scrub summary: idle+waiting paths [/] 2024-04-02T14:37:58.938 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:37:58 smithi090 ceph-mon[31674]: scrub summary: active paths [/] 2024-04-02T14:37:59.110 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:37:58 smithi092 ceph-mon[36497]: scrub queued for path: / 2024-04-02T14:37:59.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:37:58 smithi092 ceph-mon[36497]: scrub summary: idle+waiting paths [/] 2024-04-02T14:37:59.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:37:58 smithi092 ceph-mon[36497]: scrub summary: active paths [/] 2024-04-02T14:37:59.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:37:59 smithi177 ceph-mon[36431]: scrub summary: idle+waiting paths [/] 2024-04-02T14:37:59.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:37:59 smithi177 ceph-mon[36431]: pgmap v96: 97 pgs: 97 active+clean; 1.8 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 12 MiB/s wr, 978 op/s 2024-04-02T14:37:59.938 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:37:59 smithi090 ceph-mon[31674]: scrub summary: idle+waiting paths [/] 2024-04-02T14:37:59.938 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:37:59 smithi090 ceph-mon[31674]: pgmap v96: 97 pgs: 97 active+clean; 1.8 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 12 MiB/s wr, 978 op/s 2024-04-02T14:38:00.110 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:37:59 smithi092 ceph-mon[36497]: scrub summary: idle+waiting paths [/] 2024-04-02T14:38:00.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:37:59 smithi092 ceph-mon[36497]: pgmap v96: 97 pgs: 97 active+clean; 1.8 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 12 MiB/s wr, 978 op/s 2024-04-02T14:38:01.832 INFO:teuthology.orchestra.run.smithi090.stderr:progress: 150.0 s, 2853.7 tps, lat 33.417 ms stddev 70.993 2024-04-02T14:38:01.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:38:01 smithi177 ceph-mon[36431]: pgmap v97: 97 pgs: 97 active+clean; 1.8 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 11 MiB/s wr, 939 op/s 2024-04-02T14:38:01.938 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:38:01 smithi090 ceph-mon[31674]: pgmap v97: 97 pgs: 97 active+clean; 1.8 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 11 MiB/s wr, 939 op/s 2024-04-02T14:38:02.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:38:01 smithi092 ceph-mon[36497]: pgmap v97: 97 pgs: 97 active+clean; 1.8 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 11 MiB/s wr, 939 op/s 2024-04-02T14:38:03.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:38:02 smithi092 ceph-mon[36497]: scrub summary: idle 2024-04-02T14:38:03.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:38:02 smithi177 ceph-mon[36431]: scrub summary: idle 2024-04-02T14:38:03.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:38:02 smithi090 ceph-mon[31674]: scrub summary: idle 2024-04-02T14:38:04.110 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:38:03 smithi092 ceph-mon[36497]: pgmap v98: 97 pgs: 97 active+clean; 1.8 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 9.0 MiB/s wr, 794 op/s 2024-04-02T14:38:04.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:38:03 smithi177 ceph-mon[36431]: pgmap v98: 97 pgs: 97 active+clean; 1.8 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 9.0 MiB/s wr, 794 op/s 2024-04-02T14:38:04.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:38:03 smithi090 ceph-mon[31674]: pgmap v98: 97 pgs: 97 active+clean; 1.8 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 9.0 MiB/s wr, 794 op/s 2024-04-02T14:38:05.110 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:38:04 smithi092 ceph-mon[36497]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T14:38:05.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:38:04 smithi177 ceph-mon[36431]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T14:38:05.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:38:04 smithi090 ceph-mon[31674]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T14:38:06.114 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:38:05 smithi092 ceph-mon[36497]: pgmap v99: 97 pgs: 97 active+clean; 1.8 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 11 MiB/s wr, 1.04k op/s 2024-04-02T14:38:06.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:38:05 smithi177 ceph-mon[36431]: pgmap v99: 97 pgs: 97 active+clean; 1.8 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 11 MiB/s wr, 1.04k op/s 2024-04-02T14:38:06.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:38:05 smithi090 ceph-mon[31674]: pgmap v99: 97 pgs: 97 active+clean; 1.8 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 11 MiB/s wr, 1.04k op/s 2024-04-02T14:38:06.832 INFO:teuthology.orchestra.run.smithi090.stderr:progress: 155.0 s, 189.0 tps, lat 518.642 ms stddev 301.531 2024-04-02T14:38:07.110 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:38:06 smithi092 ceph-mon[36497]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:38:07.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:38:06 smithi092 ceph-mon[36497]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T14:38:07.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:38:06 smithi092 ceph-mon[36497]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' 2024-04-02T14:38:07.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:38:06 smithi177 ceph-mon[36431]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:38:07.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:38:06 smithi177 ceph-mon[36431]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T14:38:07.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:38:06 smithi177 ceph-mon[36431]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' 2024-04-02T14:38:07.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:38:06 smithi090 ceph-mon[31674]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:38:07.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:38:06 smithi090 ceph-mon[31674]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T14:38:07.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:38:06 smithi090 ceph-mon[31674]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' 2024-04-02T14:38:08.110 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:38:07 smithi092 ceph-mon[36497]: pgmap v100: 97 pgs: 97 active+clean; 1.8 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 7.5 MiB/s wr, 703 op/s 2024-04-02T14:38:08.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:38:07 smithi177 ceph-mon[36431]: pgmap v100: 97 pgs: 97 active+clean; 1.8 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 7.5 MiB/s wr, 703 op/s 2024-04-02T14:38:08.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:38:07 smithi090 ceph-mon[31674]: pgmap v100: 97 pgs: 97 active+clean; 1.8 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 7.5 MiB/s wr, 703 op/s 2024-04-02T14:38:09.991 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:38:09 smithi090 ceph-mon[31674]: pgmap v101: 97 pgs: 97 active+clean; 1.8 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 8.2 MiB/s wr, 812 op/s 2024-04-02T14:38:10.110 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:38:09 smithi092 ceph-mon[36497]: pgmap v101: 97 pgs: 97 active+clean; 1.8 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 8.2 MiB/s wr, 812 op/s 2024-04-02T14:38:10.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:38:09 smithi177 ceph-mon[36431]: pgmap v101: 97 pgs: 97 active+clean; 1.8 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 8.2 MiB/s wr, 812 op/s 2024-04-02T14:38:10.712 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.710+0000 7fb7aa400640 1 -- 172.21.15.90:0/2939288626 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb7a4107ac0 msgr2=0x7fb7a4107ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:38:10.712 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.710+0000 7fb7aa400640 1 --2- 172.21.15.90:0/2939288626 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb7a4107ac0 0x7fb7a4107ea0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fb794009900 tx=0x7fb79402bc50 comp rx=0 tx=0).stop 2024-04-02T14:38:10.712 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.711+0000 7fb7aa400640 1 -- 172.21.15.90:0/2939288626 shutdown_connections 2024-04-02T14:38:10.712 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.711+0000 7fb7aa400640 1 --2- 172.21.15.90:0/2939288626 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb7a4102470 0x7fb7a410e980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:10.712 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.711+0000 7fb7aa400640 1 --2- 172.21.15.90:0/2939288626 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb7a4101ab0 0x7fb7a4101f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:10.713 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.711+0000 7fb7aa400640 1 --2- 172.21.15.90:0/2939288626 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb7a4107ac0 0x7fb7a4107ea0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:10.713 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.711+0000 7fb7aa400640 1 -- 172.21.15.90:0/2939288626 >> 172.21.15.90:0/2939288626 conn(0x7fb7a40fd560 msgr2=0x7fb7a40ff980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:38:10.713 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.711+0000 7fb7aa400640 1 -- 172.21.15.90:0/2939288626 shutdown_connections 2024-04-02T14:38:10.713 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.711+0000 7fb7aa400640 1 -- 172.21.15.90:0/2939288626 wait complete. 2024-04-02T14:38:10.715 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.712+0000 7fb7aa400640 1 Processor -- start 2024-04-02T14:38:10.715 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.712+0000 7fb7aa400640 1 -- start start 2024-04-02T14:38:10.715 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.712+0000 7fb7aa400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb7a4101ab0 0x7fb7a419a510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:10.715 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.712+0000 7fb7aa400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb7a4102470 0x7fb7a419aa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:10.715 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.712+0000 7fb7aa400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb7a4107ac0 0x7fb7a419edd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:10.715 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.712+0000 7fb7aa400640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fb7a4111ef0 con 0x7fb7a4102470 2024-04-02T14:38:10.715 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.712+0000 7fb7aa400640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fb7a4111d70 con 0x7fb7a4107ac0 2024-04-02T14:38:10.715 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.712+0000 7fb7aa400640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fb7a4112070 con 0x7fb7a4101ab0 2024-04-02T14:38:10.716 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.713+0000 7fb7a3400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb7a4101ab0 0x7fb7a419a510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:10.716 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.713+0000 7fb7a3e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb7a4107ac0 0x7fb7a419edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:10.716 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.713+0000 7fb7a3400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb7a4101ab0 0x7fb7a419a510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:52892/0 (socket says 172.21.15.90:52892) 2024-04-02T14:38:10.716 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.713+0000 7fb7a3400640 1 -- 172.21.15.90:0/13122311 learned_addr learned my addr 172.21.15.90:0/13122311 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:38:10.716 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.713+0000 7fb7a2a00640 1 --2- 172.21.15.90:0/13122311 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb7a4102470 0x7fb7a419aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:10.716 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.713+0000 7fb7a3400640 1 -- 172.21.15.90:0/13122311 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb7a4107ac0 msgr2=0x7fb7a419edd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:38:10.716 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.713+0000 7fb7a3400640 1 --2- 172.21.15.90:0/13122311 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb7a4107ac0 0x7fb7a419edd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:10.716 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.713+0000 7fb7a3400640 1 -- 172.21.15.90:0/13122311 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb7a4102470 msgr2=0x7fb7a419aa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:38:10.716 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.713+0000 7fb7a3400640 1 --2- 172.21.15.90:0/13122311 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb7a4102470 0x7fb7a419aa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:10.716 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.713+0000 7fb7a3400640 1 -- 172.21.15.90:0/13122311 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7a419f500 con 0x7fb7a4101ab0 2024-04-02T14:38:10.716 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.713+0000 7fb7a3e00640 1 --2- 172.21.15.90:0/13122311 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb7a4107ac0 0x7fb7a419edd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:38:10.716 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.713+0000 7fb7a3400640 1 --2- 172.21.15.90:0/13122311 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb7a4101ab0 0x7fb7a419a510 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fb794000c00 tx=0x7fb7940042f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:38:10.716 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.714+0000 7fb79be00640 1 -- 172.21.15.90:0/13122311 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb794041070 con 0x7fb7a4101ab0 2024-04-02T14:38:10.716 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.714+0000 7fb7aa400640 1 -- 172.21.15.90:0/13122311 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7a419f790 con 0x7fb7a4101ab0 2024-04-02T14:38:10.716 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.714+0000 7fb79be00640 1 -- 172.21.15.90:0/13122311 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb79402cce0 con 0x7fb7a4101ab0 2024-04-02T14:38:10.717 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.714+0000 7fb79be00640 1 -- 172.21.15.90:0/13122311 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb794003d10 con 0x7fb7a4101ab0 2024-04-02T14:38:10.717 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.714+0000 7fb7aa400640 1 -- 172.21.15.90:0/13122311 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7a41a7080 con 0x7fb7a4101ab0 2024-04-02T14:38:10.718 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.717+0000 7fb7ad1e6740 1 Processor -- start 2024-04-02T14:38:10.718 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.717+0000 7fb7ad1e6740 1 -- start start 2024-04-02T14:38:10.719 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.718+0000 7fb79be00640 1 -- 172.21.15.90:0/13122311 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fb794044070 con 0x7fb7a4101ab0 2024-04-02T14:38:10.719 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.718+0000 7fb7ad1e6740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56417c3feeb0 0x56417c3ff290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:10.719 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.718+0000 7fb7ad1e6740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56417c3ff7d0 0x56417c407e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:10.719 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.718+0000 7fb7ad1e6740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56417c4083b0 0x56417c40a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:10.720 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.718+0000 7fb7ad1e6740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x56417c363d70 con 0x56417c3feeb0 2024-04-02T14:38:10.720 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.718+0000 7fb7ad1e6740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x56417c30c2a0 con 0x56417c4083b0 2024-04-02T14:38:10.720 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.718+0000 7fb7ad1e6740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x56417c35fd20 con 0x56417c3ff7d0 2024-04-02T14:38:10.720 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.718+0000 7fb7a3400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56417c3ff7d0 0x56417c407e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:10.720 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.718+0000 7fb7a2a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56417c4083b0 0x56417c40a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:10.720 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.718+0000 7fb7a2a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56417c4083b0 0x56417c40a7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:45958/0 (socket says 172.21.15.90:45958) 2024-04-02T14:38:10.720 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.718+0000 7fb7a3e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56417c3feeb0 0x56417c3ff290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:10.720 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.718+0000 7fb7a2a00640 1 -- 172.21.15.90:0/1911301303 learned_addr learned my addr 172.21.15.90:0/1911301303 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:38:10.720 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.718+0000 7fb7a3400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56417c3ff7d0 0x56417c407e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:52908/0 (socket says 172.21.15.90:52908) 2024-04-02T14:38:10.721 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.719+0000 7fb7a2a00640 1 -- 172.21.15.90:0/1911301303 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56417c3ff7d0 msgr2=0x56417c407e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:38:10.721 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.719+0000 7fb7a2a00640 1 --2- 172.21.15.90:0/1911301303 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56417c3ff7d0 0x56417c407e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:10.722 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.719+0000 7fb7a2a00640 1 -- 172.21.15.90:0/1911301303 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56417c3feeb0 msgr2=0x56417c3ff290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:38:10.722 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.719+0000 7fb7a2a00640 1 --2- 172.21.15.90:0/1911301303 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56417c3feeb0 0x56417c3ff290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:10.722 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.719+0000 7fb7a2a00640 1 -- 172.21.15.90:0/1911301303 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56417c320840 con 0x56417c4083b0 2024-04-02T14:38:10.722 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.719+0000 7fb7a3400640 1 --2- 172.21.15.90:0/1911301303 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56417c3ff7d0 0x56417c407e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:38:10.722 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.719+0000 7fb7a2a00640 1 --2- 172.21.15.90:0/1911301303 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56417c4083b0 0x56417c40a7a0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fb790009c90 tx=0x7fb790011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9bbfa28f1596d749 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:38:10.722 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.719+0000 7fb798c00640 1 -- 172.21.15.90:0/1911301303 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb79001e070 con 0x56417c4083b0 2024-04-02T14:38:10.722 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.720+0000 7fb798c00640 1 -- 172.21.15.90:0/1911301303 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb790019b10 con 0x56417c4083b0 2024-04-02T14:38:10.722 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.720+0000 7fb798c00640 1 -- 172.21.15.90:0/1911301303 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb79000ea30 con 0x56417c4083b0 2024-04-02T14:38:10.722 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.720+0000 7fb79be00640 1 --2- 172.21.15.90:0/13122311 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fb7880a3dc0 0x7fb7880a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:10.722 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.720+0000 7fb79be00640 1 -- 172.21.15.90:0/13122311 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7fb7940f20c0 con 0x7fb7a4101ab0 2024-04-02T14:38:10.722 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.720+0000 7fb7a3e00640 1 --2- 172.21.15.90:0/13122311 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fb7880a3dc0 0x7fb7880a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:10.722 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.721+0000 7fb7ad1e6740 1 -- 172.21.15.90:0/1911301303 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56417c4083b0 msgr2=0x56417c40a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:38:10.722 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.721+0000 7fb7ad1e6740 1 --2- 172.21.15.90:0/1911301303 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56417c4083b0 0x56417c40a7a0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fb790009c90 tx=0x7fb790011810 comp rx=0 tx=0).stop 2024-04-02T14:38:10.726 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.721+0000 7fb7ad1e6740 1 -- 172.21.15.90:0/1911301303 shutdown_connections 2024-04-02T14:38:10.726 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.721+0000 7fb7ad1e6740 1 --2- 172.21.15.90:0/1911301303 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56417c4083b0 0x56417c40a7a0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:10.726 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.721+0000 7fb7ad1e6740 1 --2- 172.21.15.90:0/1911301303 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56417c3ff7d0 0x56417c407e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:10.726 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.721+0000 7fb7ad1e6740 1 --2- 172.21.15.90:0/1911301303 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56417c3feeb0 0x56417c3ff290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:10.726 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.721+0000 7fb7ad1e6740 1 -- 172.21.15.90:0/1911301303 >> 172.21.15.90:0/1911301303 conn(0x56417c3f95e0 msgr2=0x56417c3f99c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:38:10.726 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.721+0000 7fb7a3e00640 1 --2- 172.21.15.90:0/13122311 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fb7880a3dc0 0x7fb7880a6280 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fb79c009fd0 tx=0x7fb79c009500 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:38:10.726 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.721+0000 7fb7ad1e6740 1 -- 172.21.15.90:0/1911301303 shutdown_connections 2024-04-02T14:38:10.727 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.721+0000 7fb7ad1e6740 1 -- 172.21.15.90:0/1911301303 wait complete. 2024-04-02T14:38:10.727 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.722+0000 7fb7ad1e6740 1 -- start start 2024-04-02T14:38:10.727 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.722+0000 7fb7ad1e6740 1 Processor -- start 2024-04-02T14:38:10.727 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.722+0000 7fb7ad1e6740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56417c3feeb0 0x56417c405390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:10.727 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.722+0000 7fb7ad1e6740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56417c3ff7d0 0x56417c4058d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:10.727 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.722+0000 7fb7ad1e6740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56417c4083b0 0x56417c405e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:10.727 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.722+0000 7fb7ad1e6740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x56417c40b1a0 con 0x56417c3feeb0 2024-04-02T14:38:10.727 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.722+0000 7fb7ad1e6740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x56417c31f660 con 0x56417c4083b0 2024-04-02T14:38:10.727 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.722+0000 7fb7ad1e6740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x56417c40b320 con 0x56417c3ff7d0 2024-04-02T14:38:10.727 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.723+0000 7fb7ad1e6740 10 client.0 _ll_register_callbacks cb 0x56417c31f050 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:38:10.727 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.723+0000 7fb7a3e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56417c4083b0 0x56417c405e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:10.727 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.723+0000 7fb7a3400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56417c3feeb0 0x56417c405390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:10.727 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.723+0000 7fb7a3e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56417c4083b0 0x56417c405e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:45972/0 (socket says 172.21.15.90:45972) 2024-04-02T14:38:10.728 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.723+0000 7fb7a3e00640 1 -- 172.21.15.90:0/1188510402 learned_addr learned my addr 172.21.15.90:0/1188510402 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:38:10.728 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.723+0000 7fb7a2a00640 1 --2- 172.21.15.90:0/1188510402 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56417c3ff7d0 0x56417c4058d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:10.728 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.723+0000 7fb7a3e00640 1 -- 172.21.15.90:0/1188510402 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56417c3ff7d0 msgr2=0x56417c4058d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:38:10.728 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.723+0000 7fb7a3e00640 1 --2- 172.21.15.90:0/1188510402 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56417c3ff7d0 0x56417c4058d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:10.728 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.723+0000 7fb7a3e00640 1 -- 172.21.15.90:0/1188510402 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56417c3feeb0 msgr2=0x56417c405390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:38:10.728 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.723+0000 7fb7a3e00640 1 --2- 172.21.15.90:0/1188510402 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56417c3feeb0 0x56417c405390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:10.728 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.723+0000 7fb7a3e00640 1 -- 172.21.15.90:0/1188510402 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56417c32ee80 con 0x56417c4083b0 2024-04-02T14:38:10.728 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.723+0000 7fb7a2a00640 1 --2- 172.21.15.90:0/1188510402 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56417c3ff7d0 0x56417c4058d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:38:10.728 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.723+0000 7fb7a3400640 1 --2- 172.21.15.90:0/1188510402 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56417c3feeb0 0x56417c405390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:38:10.728 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.723+0000 7fb7a3e00640 1 --2- 172.21.15.90:0/1188510402 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56417c4083b0 0x56417c405e10 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fb79c013010 tx=0x7fb79c004ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d1709204a545ea1f server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:38:10.728 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.724+0000 7fb78d600640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:38:10.728 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.724+0000 7fb78d600640 1 -- 172.21.15.90:0/1188510402 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb79c0045c0 con 0x56417c4083b0 2024-04-02T14:38:10.728 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.724+0000 7fb78d600640 1 -- 172.21.15.90:0/1188510402 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7fb79c004760 con 0x56417c4083b0 2024-04-02T14:38:10.728 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.724+0000 7fb7ad1e6740 1 -- 172.21.15.90:0/1188510402 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb790019b10 con 0x56417c4083b0 2024-04-02T14:38:10.728 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.724+0000 7fb78d600640 1 -- 172.21.15.90:0/1188510402 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fb79c012020 con 0x56417c4083b0 2024-04-02T14:38:10.728 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.724+0000 7fb7ad1e6740 10 client.26420 fetch_fsmap learned FSMap version 27 2024-04-02T14:38:10.729 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.724+0000 7fb7ad1e6740 1 -- 172.21.15.90:0/1188510402 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x56417c078210 con 0x56417c4083b0 2024-04-02T14:38:10.729 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.724+0000 7fb78d600640 1 -- 172.21.15.90:0/1188510402 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb790019b10 con 0x56417c4083b0 2024-04-02T14:38:10.729 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.725+0000 7fb78d600640 1 -- 172.21.15.90:0/1188510402 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fb79c027070 con 0x56417c4083b0 2024-04-02T14:38:10.729 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.725+0000 7fb7ad1e6740 10 client.26420 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:38:10.729 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.725+0000 7fb7ad1e6740 10 client.26420 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-02T14:38:10.729 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.725+0000 7fb7ad1e6740 20 client.26420 populate_metadata read hostname 'smithi090' 2024-04-02T14:38:10.729 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.725+0000 7fb7ad1e6740 1 --2- 172.21.15.90:0/1188510402 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x56417c418830 0x56417c41ac80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:10.729 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.725+0000 7fb7ad1e6740 4 client.26420 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:38:10.729 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.725+0000 7fb7ad1e6740 1 -- 172.21.15.90:0/1188510402 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56417c313cd0 con 0x56417c418830 2024-04-02T14:38:10.729 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.725+0000 7fb7a3400640 1 --2- 172.21.15.90:0/1188510402 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x56417c418830 0x56417c41ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:10.730 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.726+0000 7fb7a3400640 1 --2- 172.21.15.90:0/1188510402 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x56417c418830 0x56417c41ac80 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3b3120fcc015c026 server_cookie=964405adbe9413fa in_seq=0 out_seq=0 2024-04-02T14:38:10.730 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.726+0000 7fb78d600640 10 client.26420 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:38:10.731 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.726+0000 7fb78d600640 1 -- 172.21.15.90:0/1188510402 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56417c313cd0 con 0x56417c418830 2024-04-02T14:38:10.731 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.726+0000 7fb78d600640 10 client.26420 handle_command_reply: tid=0 2024-04-02T14:38:10.731 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.726+0000 7fb7ad1e6740 1 client.26420 shutdown 2024-04-02T14:38:10.731 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.727+0000 7fb7ad1e6740 1 -- 172.21.15.90:0/1188510402 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56417c4083b0 msgr2=0x56417c405e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:38:10.731 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.727+0000 7fb7ad1e6740 1 --2- 172.21.15.90:0/1188510402 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56417c4083b0 0x56417c405e10 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fb79c013010 tx=0x7fb79c004ab0 comp rx=0 tx=0).stop 2024-04-02T14:38:10.731 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.727+0000 7fb7ad1e6740 1 -- 172.21.15.90:0/1188510402 shutdown_connections 2024-04-02T14:38:10.731 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.727+0000 7fb7ad1e6740 1 --2- 172.21.15.90:0/1188510402 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x56417c418830 0x56417c41ac80 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:10.731 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.727+0000 7fb7ad1e6740 1 --2- 172.21.15.90:0/1188510402 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56417c4083b0 0x56417c405e10 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:10.731 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.727+0000 7fb7ad1e6740 1 --2- 172.21.15.90:0/1188510402 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56417c3ff7d0 0x56417c4058d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:10.731 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.727+0000 7fb7ad1e6740 1 --2- 172.21.15.90:0/1188510402 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56417c3feeb0 0x56417c405390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:10.731 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.727+0000 7fb7ad1e6740 1 -- 172.21.15.90:0/1188510402 >> 172.21.15.90:0/1188510402 conn(0x56417c3f95e0 msgr2=0x56417c3fb330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:38:10.731 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.727+0000 7fb78d600640 0 client.26420 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:38:10.731 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.727+0000 7fb7ad1e6740 1 -- 172.21.15.90:0/1188510402 shutdown_connections 2024-04-02T14:38:10.732 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.727+0000 7fb7ad1e6740 1 -- 172.21.15.90:0/1188510402 wait complete. 2024-04-02T14:38:10.732 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.727+0000 7fb7ad1e6740 20 client.26420 trim_cache size 0 max 16384 2024-04-02T14:38:10.748 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.747+0000 7fb7ad1e6740 1 Processor -- start 2024-04-02T14:38:10.748 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.747+0000 7fb7ad1e6740 1 -- start start 2024-04-02T14:38:10.749 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.747+0000 7fb7ad1e6740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56417c3feeb0 0x56417c407ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:10.749 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.747+0000 7fb7ad1e6740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56417c3ff7d0 0x56417c414090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:10.749 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.747+0000 7fb7ad1e6740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56417c4083b0 0x56417c400730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:10.749 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.747+0000 7fb7ad1e6740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fb7a41af950 con 0x56417c4083b0 2024-04-02T14:38:10.749 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.747+0000 7fb7ad1e6740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fb7a41b1490 con 0x56417c3ff7d0 2024-04-02T14:38:10.749 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.747+0000 7fb7ad1e6740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fb7a41b0020 con 0x56417c3feeb0 2024-04-02T14:38:10.749 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.747+0000 7fb7a2a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56417c3ff7d0 0x56417c414090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:10.749 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.747+0000 7fb7a3e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56417c4083b0 0x56417c400730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:10.749 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.747+0000 7fb7a3400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56417c3feeb0 0x56417c407ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:10.749 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.747+0000 7fb7a2a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56417c3ff7d0 0x56417c414090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:45982/0 (socket says 172.21.15.90:45982) 2024-04-02T14:38:10.749 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.747+0000 7fb7a2a00640 1 -- 172.21.15.90:0/1555095606 learned_addr learned my addr 172.21.15.90:0/1555095606 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:38:10.751 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.748+0000 7fb7a3400640 1 -- 172.21.15.90:0/1555095606 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56417c3ff7d0 msgr2=0x56417c414090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:38:10.751 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.748+0000 7fb7a3400640 1 --2- 172.21.15.90:0/1555095606 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56417c3ff7d0 0x56417c414090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:10.751 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.748+0000 7fb7a3400640 1 -- 172.21.15.90:0/1555095606 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56417c4083b0 msgr2=0x56417c400730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:38:10.751 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.748+0000 7fb7a3400640 1 --2- 172.21.15.90:0/1555095606 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56417c4083b0 0x56417c400730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:10.751 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.748+0000 7fb7a3400640 1 -- 172.21.15.90:0/1555095606 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56417c3ff600 con 0x56417c3feeb0 2024-04-02T14:38:10.751 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.748+0000 7fb7a2a00640 1 --2- 172.21.15.90:0/1555095606 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56417c3ff7d0 0x56417c414090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:38:10.751 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.748+0000 7fb7a3e00640 1 --2- 172.21.15.90:0/1555095606 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56417c4083b0 0x56417c400730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:38:10.751 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.748+0000 7fb7a3400640 1 --2- 172.21.15.90:0/1555095606 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56417c3feeb0 0x56417c407ac0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fb7940ef930 tx=0x7fb7940f5c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6f63fd1ddd6f3f2d server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:38:10.751 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.748+0000 7fb78fe00640 1 -- 172.21.15.90:0/1555095606 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb79404b040 con 0x56417c3feeb0 2024-04-02T14:38:10.752 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.748+0000 7fb78fe00640 1 -- 172.21.15.90:0/1555095606 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb7940be230 con 0x56417c3feeb0 2024-04-02T14:38:10.752 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.748+0000 7fb78fe00640 1 -- 172.21.15.90:0/1555095606 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7940c2530 con 0x56417c3feeb0 2024-04-02T14:38:10.752 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.749+0000 7fb7ad1e6740 1 -- 172.21.15.90:0/1555095606 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56417c3feeb0 msgr2=0x56417c407ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:38:10.752 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.749+0000 7fb7ad1e6740 1 --2- 172.21.15.90:0/1555095606 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56417c3feeb0 0x56417c407ac0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fb7940ef930 tx=0x7fb7940f5c00 comp rx=0 tx=0).stop 2024-04-02T14:38:10.752 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.749+0000 7fb7ad1e6740 1 -- 172.21.15.90:0/1555095606 shutdown_connections 2024-04-02T14:38:10.752 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.749+0000 7fb7ad1e6740 1 --2- 172.21.15.90:0/1555095606 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56417c4083b0 0x56417c400730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:10.752 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.749+0000 7fb7ad1e6740 1 --2- 172.21.15.90:0/1555095606 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56417c3ff7d0 0x56417c414090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:10.752 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.749+0000 7fb7ad1e6740 1 --2- 172.21.15.90:0/1555095606 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56417c3feeb0 0x56417c407ac0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:10.752 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.749+0000 7fb7ad1e6740 1 -- 172.21.15.90:0/1555095606 >> 172.21.15.90:0/1555095606 conn(0x56417c3f95e0 msgr2=0x56417c3ffbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:38:10.752 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.749+0000 7fb7ad1e6740 1 -- 172.21.15.90:0/1555095606 shutdown_connections 2024-04-02T14:38:10.752 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.749+0000 7fb7ad1e6740 1 -- 172.21.15.90:0/1555095606 wait complete. 2024-04-02T14:38:10.752 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.750+0000 7fb7ad1e6740 1 -- start start 2024-04-02T14:38:10.752 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.750+0000 7fb7ad1e6740 1 Processor -- start 2024-04-02T14:38:10.755 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.750+0000 7fb7ad1e6740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56417c3feeb0 0x56417c4105a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:10.755 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.750+0000 7fb7ad1e6740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56417c3ff7d0 0x56417c410ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:10.755 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.750+0000 7fb7ad1e6740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56417c4083b0 0x56417c401bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:10.755 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.750+0000 7fb7ad1e6740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fb7a41b0dc0 con 0x56417c3feeb0 2024-04-02T14:38:10.755 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.750+0000 7fb7ad1e6740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fb7a41b06f0 con 0x56417c4083b0 2024-04-02T14:38:10.755 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.750+0000 7fb7ad1e6740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fb7a41ad930 con 0x56417c3ff7d0 2024-04-02T14:38:10.755 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.750+0000 7fb7ad1e6740 10 client.0 _ll_register_callbacks cb 0x7fb79c003e00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:38:10.755 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.751+0000 7fb7a3e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56417c4083b0 0x56417c401bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:10.756 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.751+0000 7fb7a2a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56417c3ff7d0 0x56417c410ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:10.756 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.751+0000 7fb7a3e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56417c4083b0 0x56417c401bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:45988/0 (socket says 172.21.15.90:45988) 2024-04-02T14:38:10.756 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.751+0000 7fb7a3e00640 1 -- 172.21.15.90:0/271599112 learned_addr learned my addr 172.21.15.90:0/271599112 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:38:10.756 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.751+0000 7fb7a3400640 1 --2- 172.21.15.90:0/271599112 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56417c3feeb0 0x56417c4105a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:10.756 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.751+0000 7fb7a2a00640 1 -- 172.21.15.90:0/271599112 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56417c4083b0 msgr2=0x56417c401bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:38:10.756 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.751+0000 7fb7a2a00640 1 --2- 172.21.15.90:0/271599112 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56417c4083b0 0x56417c401bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:10.756 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.751+0000 7fb7a2a00640 1 -- 172.21.15.90:0/271599112 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56417c3feeb0 msgr2=0x56417c4105a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:38:10.756 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.751+0000 7fb7a2a00640 1 --2- 172.21.15.90:0/271599112 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56417c3feeb0 0x56417c4105a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:10.756 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.751+0000 7fb7a2a00640 1 -- 172.21.15.90:0/271599112 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56417c144620 con 0x56417c3ff7d0 2024-04-02T14:38:10.756 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.751+0000 7fb7a3e00640 1 --2- 172.21.15.90:0/271599112 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56417c4083b0 0x56417c401bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:38:10.756 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.751+0000 7fb7a2a00640 1 --2- 172.21.15.90:0/271599112 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56417c3ff7d0 0x56417c410ae0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fb790000c00 tx=0x7fb7900056e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=99acaa1c46f82aea server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:38:10.756 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.751+0000 7fb78d600640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:38:10.756 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.752+0000 7fb78d600640 1 -- 172.21.15.90:0/271599112 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb790029070 con 0x56417c3ff7d0 2024-04-02T14:38:10.757 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.752+0000 7fb78d600640 1 -- 172.21.15.90:0/271599112 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7fb790019430 con 0x56417c3ff7d0 2024-04-02T14:38:10.757 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.752+0000 7fb7ad1e6740 1 -- 172.21.15.90:0/271599112 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb7940be230 con 0x56417c3ff7d0 2024-04-02T14:38:10.757 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.752+0000 7fb78d600640 1 -- 172.21.15.90:0/271599112 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fb790034050 con 0x56417c3ff7d0 2024-04-02T14:38:10.757 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.752+0000 7fb7ad1e6740 10 client.26332 fetch_fsmap learned FSMap version 27 2024-04-02T14:38:10.757 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.752+0000 7fb7ad1e6740 1 -- 172.21.15.90:0/271599112 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x56417c47e830 con 0x56417c3ff7d0 2024-04-02T14:38:10.757 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.752+0000 7fb78d600640 1 -- 172.21.15.90:0/271599112 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb7940be230 con 0x56417c3ff7d0 2024-04-02T14:38:10.757 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.753+0000 7fb78d600640 1 -- 172.21.15.90:0/271599112 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fb790024040 con 0x56417c3ff7d0 2024-04-02T14:38:10.757 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.753+0000 7fb7ad1e6740 10 client.26332 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:38:10.757 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.753+0000 7fb7ad1e6740 10 client.26332 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-02T14:38:10.757 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.753+0000 7fb7ad1e6740 20 client.26332 populate_metadata read hostname 'smithi090' 2024-04-02T14:38:10.757 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.753+0000 7fb7ad1e6740 1 --2- 172.21.15.90:0/271599112 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x56417c47edd0 0x56417c4811f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:10.757 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.753+0000 7fb7ad1e6740 4 client.26332 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-02T14:38:10.757 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.753+0000 7fb7ad1e6740 1 -- 172.21.15.90:0/271599112 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56417c3712b0 con 0x56417c47edd0 2024-04-02T14:38:10.757 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.753+0000 7fb7a3400640 1 --2- 172.21.15.90:0/271599112 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x56417c47edd0 0x56417c4811f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:10.757 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.753+0000 7fb7a3400640 1 --2- 172.21.15.90:0/271599112 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x56417c47edd0 0x56417c4811f0 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a22055a8e930afb2 server_cookie=66e939e2980b54c7 in_seq=0 out_seq=0 2024-04-02T14:38:10.758 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.753+0000 7fb78d600640 10 client.26332 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:38:10.773 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.772+0000 7fb78d600640 1 -- 172.21.15.90:0/271599112 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56417c3712b0 con 0x56417c47edd0 2024-04-02T14:38:10.773 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.772+0000 7fb78d600640 10 client.26332 handle_command_reply: tid=0 2024-04-02T14:38:10.773 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.772+0000 7fb7ad1e6740 1 client.26332 shutdown 2024-04-02T14:38:10.773 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.772+0000 7fb7ad1e6740 1 -- 172.21.15.90:0/271599112 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56417c3ff7d0 msgr2=0x56417c410ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:38:10.773 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.772+0000 7fb7ad1e6740 1 --2- 172.21.15.90:0/271599112 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56417c3ff7d0 0x56417c410ae0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fb790000c00 tx=0x7fb7900056e0 comp rx=0 tx=0).stop 2024-04-02T14:38:10.774 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.772+0000 7fb7ad1e6740 1 -- 172.21.15.90:0/271599112 shutdown_connections 2024-04-02T14:38:10.774 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.772+0000 7fb7ad1e6740 1 --2- 172.21.15.90:0/271599112 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x56417c47edd0 0x56417c4811f0 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:10.774 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.772+0000 7fb7ad1e6740 1 --2- 172.21.15.90:0/271599112 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56417c4083b0 0x56417c401bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:10.774 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.772+0000 7fb7ad1e6740 1 --2- 172.21.15.90:0/271599112 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56417c3ff7d0 0x56417c410ae0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:10.774 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.772+0000 7fb7ad1e6740 1 --2- 172.21.15.90:0/271599112 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56417c3feeb0 0x56417c4105a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:10.774 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.772+0000 7fb7ad1e6740 1 -- 172.21.15.90:0/271599112 >> 172.21.15.90:0/271599112 conn(0x56417c3f95e0 msgr2=0x56417c400730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:38:10.774 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.772+0000 7fb78d600640 0 client.26332 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:38:10.775 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.772+0000 7fb7ad1e6740 1 -- 172.21.15.90:0/271599112 shutdown_connections 2024-04-02T14:38:10.775 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.773+0000 7fb7ad1e6740 1 -- 172.21.15.90:0/271599112 wait complete. 2024-04-02T14:38:10.775 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.773+0000 7fb7ad1e6740 20 client.26332 trim_cache size 0 max 16384 2024-04-02T14:38:10.775 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:{ 2024-04-02T14:38:10.775 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "message": "", 2024-04-02T14:38:10.775 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "return_code": 0 2024-04-02T14:38:10.775 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:} 2024-04-02T14:38:10.776 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.773+0000 7fb7aa400640 1 -- 172.21.15.90:0/13122311 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fb7880a3dc0 msgr2=0x7fb7880a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:38:10.776 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.773+0000 7fb7aa400640 1 --2- 172.21.15.90:0/13122311 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fb7880a3dc0 0x7fb7880a6280 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fb79c009fd0 tx=0x7fb79c009500 comp rx=0 tx=0).stop 2024-04-02T14:38:10.776 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.773+0000 7fb7aa400640 1 -- 172.21.15.90:0/13122311 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb7a4101ab0 msgr2=0x7fb7a419a510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:38:10.776 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.773+0000 7fb7aa400640 1 --2- 172.21.15.90:0/13122311 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb7a4101ab0 0x7fb7a419a510 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fb794000c00 tx=0x7fb7940042f0 comp rx=0 tx=0).stop 2024-04-02T14:38:10.776 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.774+0000 7fb7aa400640 1 -- 172.21.15.90:0/13122311 shutdown_connections 2024-04-02T14:38:10.776 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.774+0000 7fb7aa400640 1 --2- 172.21.15.90:0/13122311 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fb7880a3dc0 0x7fb7880a6280 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:10.776 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.774+0000 7fb7aa400640 1 --2- 172.21.15.90:0/13122311 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb7a4107ac0 0x7fb7a419edd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:10.776 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.774+0000 7fb7aa400640 1 --2- 172.21.15.90:0/13122311 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb7a4102470 0x7fb7a419aa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:10.777 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.774+0000 7fb7aa400640 1 --2- 172.21.15.90:0/13122311 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb7a4101ab0 0x7fb7a419a510 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:10.777 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.774+0000 7fb7aa400640 1 -- 172.21.15.90:0/13122311 >> 172.21.15.90:0/13122311 conn(0x7fb7a40fd560 msgr2=0x7fb7a410d220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:38:10.777 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.774+0000 7fb7aa400640 1 -- 172.21.15.90:0/13122311 shutdown_connections 2024-04-02T14:38:10.777 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:10.774+0000 7fb7aa400640 1 -- 172.21.15.90:0/13122311 wait complete. 2024-04-02T14:38:11.832 INFO:teuthology.orchestra.run.smithi090.stderr:progress: 160.0 s, 172.6 tps, lat 585.530 ms stddev 360.119 2024-04-02T14:38:12.110 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:38:11 smithi092 ceph-mon[36497]: pgmap v102: 97 pgs: 97 active+clean; 1.8 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 4.0 MiB/s wr, 526 op/s 2024-04-02T14:38:12.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:38:11 smithi177 ceph-mon[36431]: pgmap v102: 97 pgs: 97 active+clean; 1.8 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 4.0 MiB/s wr, 526 op/s 2024-04-02T14:38:12.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:38:11 smithi090 ceph-mon[31674]: pgmap v102: 97 pgs: 97 active+clean; 1.8 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 4.0 MiB/s wr, 526 op/s 2024-04-02T14:38:14.110 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:38:13 smithi092 ceph-mon[36497]: pgmap v103: 97 pgs: 97 active+clean; 1.8 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 2.9 MiB/s wr, 411 op/s 2024-04-02T14:38:14.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:38:13 smithi177 ceph-mon[36431]: pgmap v103: 97 pgs: 97 active+clean; 1.8 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 2.9 MiB/s wr, 411 op/s 2024-04-02T14:38:14.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:38:13 smithi090 ceph-mon[31674]: pgmap v103: 97 pgs: 97 active+clean; 1.8 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 2.9 MiB/s wr, 411 op/s 2024-04-02T14:38:16.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:38:15 smithi092 ceph-mon[36497]: pgmap v104: 97 pgs: 97 active+clean; 1.8 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 4.1 MiB/s wr, 552 op/s 2024-04-02T14:38:16.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:38:15 smithi177 ceph-mon[36431]: pgmap v104: 97 pgs: 97 active+clean; 1.8 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 4.1 MiB/s wr, 552 op/s 2024-04-02T14:38:16.190 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:38:15 smithi090 ceph-mon[31674]: pgmap v104: 97 pgs: 97 active+clean; 1.8 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 4.1 MiB/s wr, 552 op/s 2024-04-02T14:38:16.832 INFO:teuthology.orchestra.run.smithi090.stderr:progress: 165.0 s, 131.0 tps, lat 596.360 ms stddev 353.521 2024-04-02T14:38:18.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:38:17 smithi092 ceph-mon[36497]: pgmap v105: 97 pgs: 97 active+clean; 1.8 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 2.3 MiB/s wr, 307 op/s 2024-04-02T14:38:18.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:38:17 smithi177 ceph-mon[36431]: pgmap v105: 97 pgs: 97 active+clean; 1.8 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 2.3 MiB/s wr, 307 op/s 2024-04-02T14:38:18.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:38:17 smithi090 ceph-mon[31674]: pgmap v105: 97 pgs: 97 active+clean; 1.8 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 2.3 MiB/s wr, 307 op/s 2024-04-02T14:38:20.110 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:38:19 smithi092 ceph-mon[36497]: pgmap v106: 97 pgs: 97 active+clean; 2.0 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 21 MiB/s wr, 928 op/s 2024-04-02T14:38:20.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:38:19 smithi177 ceph-mon[36431]: pgmap v106: 97 pgs: 97 active+clean; 2.0 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 21 MiB/s wr, 928 op/s 2024-04-02T14:38:20.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:38:19 smithi090 ceph-mon[31674]: pgmap v106: 97 pgs: 97 active+clean; 2.0 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 21 MiB/s wr, 928 op/s 2024-04-02T14:38:21.831 INFO:teuthology.orchestra.run.smithi090.stderr:progress: 170.0 s, 2312.4 tps, lat 55.855 ms stddev 236.746 2024-04-02T14:38:22.110 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:38:21 smithi092 ceph-mon[36497]: pgmap v107: 97 pgs: 97 active+clean; 2.1 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 32 MiB/s wr, 1.18k op/s 2024-04-02T14:38:22.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:38:21 smithi177 ceph-mon[36431]: pgmap v107: 97 pgs: 97 active+clean; 2.1 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 32 MiB/s wr, 1.18k op/s 2024-04-02T14:38:22.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:38:21 smithi090 ceph-mon[31674]: pgmap v107: 97 pgs: 97 active+clean; 2.1 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 32 MiB/s wr, 1.18k op/s 2024-04-02T14:38:23.942 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:23.941+0000 7f6d25a00640 1 -- 172.21.15.90:0/3953038216 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6d20109020 msgr2=0x7f6d20110bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:38:23.942 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:23.941+0000 7f6d25a00640 1 --2- 172.21.15.90:0/3953038216 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6d20109020 0x7f6d20110bf0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f6d18009be0 tx=0x7f6d1802bef0 comp rx=0 tx=0).stop 2024-04-02T14:38:23.942 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:23.941+0000 7f6d25a00640 1 -- 172.21.15.90:0/3953038216 shutdown_connections 2024-04-02T14:38:23.943 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:23.941+0000 7f6d25a00640 1 --2- 172.21.15.90:0/3953038216 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6d20109020 0x7f6d20110bf0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:23.943 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:23.941+0000 7f6d25a00640 1 --2- 172.21.15.90:0/3953038216 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6d20104900 0x7f6d20108940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:23.943 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:23.941+0000 7f6d25a00640 1 --2- 172.21.15.90:0/3953038216 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6d20103f50 0x7f6d20104330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:23.943 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:23.941+0000 7f6d25a00640 1 -- 172.21.15.90:0/3953038216 >> 172.21.15.90:0/3953038216 conn(0x7f6d200ff7b0 msgr2=0x7f6d20101bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:38:23.943 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:23.942+0000 7f6d25a00640 1 -- 172.21.15.90:0/3953038216 shutdown_connections 2024-04-02T14:38:23.943 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:23.942+0000 7f6d25a00640 1 -- 172.21.15.90:0/3953038216 wait complete. 2024-04-02T14:38:23.944 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:23.943+0000 7f6d25a00640 1 Processor -- start 2024-04-02T14:38:23.945 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:23.944+0000 7f6d25a00640 1 -- start start 2024-04-02T14:38:23.945 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:23.944+0000 7f6d25a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6d20103f50 0x7f6d2019a510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:23.945 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:23.944+0000 7f6d25a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6d20109020 0x7f6d2019aa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:23.945 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:23.944+0000 7f6d25a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6d2019ed80 0x7f6d2019f230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:23.945 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:23.944+0000 7f6d25a00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f6d201140d0 con 0x7f6d2019ed80 2024-04-02T14:38:23.945 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:23.944+0000 7f6d25a00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f6d20113f50 con 0x7f6d20109020 2024-04-02T14:38:23.946 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:23.944+0000 7f6d25a00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f6d20114250 con 0x7f6d20103f50 2024-04-02T14:38:23.946 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:23.944+0000 7f6d1ea00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6d20103f50 0x7f6d2019a510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:23.946 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:23.944+0000 7f6d1e000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6d20109020 0x7f6d2019aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:23.947 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:23.944+0000 7f6d1ea00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6d20103f50 0x7f6d2019a510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:37438/0 (socket says 172.21.15.90:37438) 2024-04-02T14:38:23.947 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:23.944+0000 7f6d1ea00640 1 -- 172.21.15.90:0/3924608110 learned_addr learned my addr 172.21.15.90:0/3924608110 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:38:23.947 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:23.945+0000 7f6d1ea00640 1 -- 172.21.15.90:0/3924608110 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6d20109020 msgr2=0x7f6d2019aa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:38:23.947 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:23.945+0000 7f6d1ea00640 1 --2- 172.21.15.90:0/3924608110 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6d20109020 0x7f6d2019aa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:23.947 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:23.945+0000 7f6d1ea00640 1 -- 172.21.15.90:0/3924608110 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6d2019ed80 msgr2=0x7f6d2019f230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:38:23.947 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:23.945+0000 7f6d1f400640 1 --2- 172.21.15.90:0/3924608110 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6d2019ed80 0x7f6d2019f230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:23.948 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:23.946+0000 7f6d1ea00640 1 --2- 172.21.15.90:0/3924608110 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6d2019ed80 0x7f6d2019f230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:23.948 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:23.946+0000 7f6d1ea00640 1 -- 172.21.15.90:0/3924608110 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d2019f870 con 0x7f6d20103f50 2024-04-02T14:38:23.948 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:23.946+0000 7f6d1e000640 1 --2- 172.21.15.90:0/3924608110 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6d20109020 0x7f6d2019aa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:38:23.948 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:23.946+0000 7f6d1ea00640 1 --2- 172.21.15.90:0/3924608110 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6d20103f50 0x7f6d2019a510 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f6d1000e9f0 tx=0x7f6d1000eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:38:23.948 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:23.946+0000 7f6d1f400640 1 --2- 172.21.15.90:0/3924608110 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6d2019ed80 0x7f6d2019f230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:38:23.948 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:23.946+0000 7f6d17400640 1 -- 172.21.15.90:0/3924608110 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d1000ccc0 con 0x7f6d20103f50 2024-04-02T14:38:23.948 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:23.946+0000 7f6d25a00640 1 -- 172.21.15.90:0/3924608110 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d2019fb60 con 0x7f6d20103f50 2024-04-02T14:38:23.948 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:23.946+0000 7f6d17400640 1 -- 172.21.15.90:0/3924608110 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6d10005500 con 0x7f6d20103f50 2024-04-02T14:38:23.948 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:23.946+0000 7f6d17400640 1 -- 172.21.15.90:0/3924608110 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d10010690 con 0x7f6d20103f50 2024-04-02T14:38:23.948 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:23.947+0000 7f6d25a00640 1 -- 172.21.15.90:0/3924608110 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d201a7350 con 0x7f6d20103f50 2024-04-02T14:38:23.951 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:23.950+0000 7f6d28633740 1 Processor -- start 2024-04-02T14:38:23.952 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:23.951+0000 7f6d28633740 1 -- start start 2024-04-02T14:38:23.954 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:23.951+0000 7f6d28633740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55df43e16cc0 0x55df43e170a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:23.955 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:23.951+0000 7f6d17400640 1 -- 172.21.15.90:0/3924608110 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f6d10004ef0 con 0x7f6d20103f50 2024-04-02T14:38:23.955 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:23.951+0000 7f6d28633740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55df43e17670 0x55df43e28680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:23.955 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:23.951+0000 7f6d28633740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55df43e28bc0 0x55df43e2afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:23.955 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:23.951+0000 7f6d28633740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55df43d7bd70 con 0x55df43e16cc0 2024-04-02T14:38:23.955 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:23.951+0000 7f6d28633740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55df43d242a0 con 0x55df43e17670 2024-04-02T14:38:23.955 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:23.951+0000 7f6d28633740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55df43d77d20 con 0x55df43e28bc0 2024-04-02T14:38:23.955 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:23.952+0000 7f6d1e000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55df43e28bc0 0x55df43e2afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:23.955 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:23.952+0000 7f6d1e000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55df43e28bc0 0x55df43e2afb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:37448/0 (socket says 172.21.15.90:37448) 2024-04-02T14:38:23.955 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:23.952+0000 7f6d1e000640 1 -- 172.21.15.90:0/3397578711 learned_addr learned my addr 172.21.15.90:0/3397578711 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:38:23.955 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:23.952+0000 7f6d1ea00640 1 --2- 172.21.15.90:0/3397578711 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55df43e17670 0x55df43e28680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:23.956 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:23.953+0000 7f6d1ea00640 1 -- 172.21.15.90:0/3397578711 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55df43e28bc0 msgr2=0x55df43e2afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:38:23.956 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:23.953+0000 7f6d1ea00640 1 --2- 172.21.15.90:0/3397578711 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55df43e28bc0 0x55df43e2afb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:23.956 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:23.953+0000 7f6d1ea00640 1 -- 172.21.15.90:0/3397578711 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55df43e16cc0 msgr2=0x55df43e170a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-02T14:38:23.956 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:23.953+0000 7f6d1ea00640 1 --2- 172.21.15.90:0/3397578711 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55df43e16cc0 0x55df43e170a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:23.956 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:23.953+0000 7f6d1ea00640 1 -- 172.21.15.90:0/3397578711 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55df43d38840 con 0x55df43e17670 2024-04-02T14:38:23.956 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:23.953+0000 7f6d1ea00640 1 --2- 172.21.15.90:0/3397578711 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55df43e17670 0x55df43e28680 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f6d100ce250 tx=0x7f6d100d4ba0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ef53b9b4063e91a8 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:38:23.956 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:23.954+0000 7f6d0be00640 1 -- 172.21.15.90:0/3397578711 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d100d8040 con 0x55df43e17670 2024-04-02T14:38:23.956 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:23.954+0000 7f6d0be00640 1 -- 172.21.15.90:0/3397578711 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6d100e2070 con 0x55df43e17670 2024-04-02T14:38:23.956 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:23.954+0000 7f6d0be00640 1 -- 172.21.15.90:0/3397578711 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d100ddc10 con 0x55df43e17670 2024-04-02T14:38:23.958 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:23.956+0000 7f6d17400640 1 --2- 172.21.15.90:0/3924608110 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f6d040a3e40 0x7f6d040a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:23.958 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:23.956+0000 7f6d28633740 1 -- 172.21.15.90:0/3397578711 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55df43e17670 msgr2=0x55df43e28680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:38:23.958 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:23.956+0000 7f6d28633740 1 --2- 172.21.15.90:0/3397578711 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55df43e17670 0x55df43e28680 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f6d100ce250 tx=0x7f6d100d4ba0 comp rx=0 tx=0).stop 2024-04-02T14:38:23.958 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:23.957+0000 7f6d17400640 1 -- 172.21.15.90:0/3924608110 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f6d100d0b80 con 0x7f6d20103f50 2024-04-02T14:38:23.958 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:23.957+0000 7f6d28633740 1 -- 172.21.15.90:0/3397578711 shutdown_connections 2024-04-02T14:38:23.959 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:23.957+0000 7f6d28633740 1 --2- 172.21.15.90:0/3397578711 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55df43e28bc0 0x55df43e2afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:23.959 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:23.957+0000 7f6d28633740 1 --2- 172.21.15.90:0/3397578711 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55df43e17670 0x55df43e28680 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:23.959 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:23.957+0000 7f6d28633740 1 --2- 172.21.15.90:0/3397578711 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55df43e16cc0 0x55df43e170a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:23.959 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:23.957+0000 7f6d28633740 1 -- 172.21.15.90:0/3397578711 >> 172.21.15.90:0/3397578711 conn(0x55df43e115e0 msgr2=0x55df43e119c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:38:23.959 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:23.958+0000 7f6d1f400640 1 --2- 172.21.15.90:0/3924608110 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f6d040a3e40 0x7f6d040a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:23.959 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:23.958+0000 7f6d28633740 1 -- 172.21.15.90:0/3397578711 shutdown_connections 2024-04-02T14:38:23.961 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:23.959+0000 7f6d28633740 1 -- 172.21.15.90:0/3397578711 wait complete. 2024-04-02T14:38:23.961 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:23.960+0000 7f6d1f400640 1 --2- 172.21.15.90:0/3924608110 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f6d040a3e40 0x7f6d040a6300 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f6d180099a0 tx=0x7f6d18005fd0 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:38:23.961 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:23.960+0000 7f6d28633740 1 -- start start 2024-04-02T14:38:23.962 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:23.961+0000 7f6d28633740 1 Processor -- start 2024-04-02T14:38:23.965 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:23.963+0000 7f6d28633740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55df43e16cc0 0x55df43e22710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:23.966 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:23.965+0000 7f6d28633740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55df43e17670 0x55df43e22c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:23.966 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:23.965+0000 7f6d28633740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55df43e28bc0 0x55df43e28430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:23.967 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:23.966+0000 7f6d28633740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55df43e17ff0 con 0x55df43e16cc0 2024-04-02T14:38:23.968 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:23.967+0000 7f6d1e000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55df43e17670 0x55df43e22c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:23.968 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:23.967+0000 7f6d28633740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55df43d37660 con 0x55df43e17670 2024-04-02T14:38:23.968 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:23.967+0000 7f6d28633740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55df43e18170 con 0x55df43e28bc0 2024-04-02T14:38:23.968 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:23.967+0000 7f6d28633740 10 client.0 _ll_register_callbacks cb 0x55df43d37050 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:38:23.968 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:23.967+0000 7f6d1e000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55df43e17670 0x55df43e22c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:47754/0 (socket says 172.21.15.90:47754) 2024-04-02T14:38:23.969 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:23.967+0000 7f6d1e000640 1 -- 172.21.15.90:0/588541194 learned_addr learned my addr 172.21.15.90:0/588541194 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:38:23.969 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:23.967+0000 7f6d1f400640 1 --2- 172.21.15.90:0/588541194 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55df43e28bc0 0x55df43e28430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:23.969 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:23.968+0000 7f6d1e000640 1 -- 172.21.15.90:0/588541194 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55df43e28bc0 msgr2=0x55df43e28430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:38:23.969 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:23.968+0000 7f6d1e000640 1 --2- 172.21.15.90:0/588541194 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55df43e28bc0 0x55df43e28430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:23.969 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:23.968+0000 7f6d1ea00640 1 --2- 172.21.15.90:0/588541194 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55df43e16cc0 0x55df43e22710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:23.969 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:23.968+0000 7f6d1e000640 1 -- 172.21.15.90:0/588541194 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55df43e16cc0 msgr2=0x55df43e22710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:38:23.971 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:23.969+0000 7f6d1e000640 1 --2- 172.21.15.90:0/588541194 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55df43e16cc0 0x55df43e22710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:23.971 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:23.969+0000 7f6d1e000640 1 -- 172.21.15.90:0/588541194 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55df43d46e80 con 0x55df43e17670 2024-04-02T14:38:23.971 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:23.969+0000 7f6d1ea00640 1 --2- 172.21.15.90:0/588541194 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55df43e16cc0 0x55df43e22710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:38:23.971 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:23.970+0000 7f6d1e000640 1 --2- 172.21.15.90:0/588541194 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55df43e17670 0x55df43e22c50 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f6d0c023d10 tx=0x7f6d0c00f790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c344099ec29a3c19 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:38:23.971 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:23.970+0000 7f6d08c00640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:38:23.973 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:23.972+0000 7f6d08c00640 1 -- 172.21.15.90:0/588541194 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d0c023110 con 0x55df43e17670 2024-04-02T14:38:23.973 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:23.972+0000 7f6d08c00640 1 -- 172.21.15.90:0/588541194 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f6d0c023e00 con 0x55df43e17670 2024-04-02T14:38:23.973 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:23.972+0000 7f6d28633740 1 -- 172.21.15.90:0/588541194 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6d100e2070 con 0x55df43e17670 2024-04-02T14:38:23.973 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:23.972+0000 7f6d08c00640 1 -- 172.21.15.90:0/588541194 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f6d0c0232b0 con 0x55df43e17670 2024-04-02T14:38:23.974 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:23.973+0000 7f6d28633740 10 client.26438 fetch_fsmap learned FSMap version 27 2024-04-02T14:38:23.975 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:23.974+0000 7f6d28633740 1 -- 172.21.15.90:0/588541194 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55df43a90210 con 0x55df43e17670 2024-04-02T14:38:23.976 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:23.975+0000 7f6d08c00640 1 -- 172.21.15.90:0/588541194 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6d100e2070 con 0x55df43e17670 2024-04-02T14:38:23.978 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:23.977+0000 7f6d08c00640 1 -- 172.21.15.90:0/588541194 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f6d0c036470 con 0x55df43e17670 2024-04-02T14:38:23.979 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:23.977+0000 7f6d28633740 10 client.26438 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:38:23.979 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:23.977+0000 7f6d28633740 10 client.26438 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-02T14:38:23.979 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:23.977+0000 7f6d28633740 20 client.26438 populate_metadata read hostname 'smithi090' 2024-04-02T14:38:23.979 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:23.977+0000 7f6d28633740 1 --2- 172.21.15.90:0/588541194 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55df43e30790 0x55df43e32ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:23.979 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:23.977+0000 7f6d28633740 4 client.26438 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:38:23.979 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:23.977+0000 7f6d28633740 1 -- 172.21.15.90:0/588541194 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55df43d2bcd0 con 0x55df43e30790 2024-04-02T14:38:23.979 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:23.978+0000 7f6d1ea00640 1 --2- 172.21.15.90:0/588541194 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55df43e30790 0x55df43e32ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:23.981 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:23.980+0000 7f6d08c00640 10 client.26438 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:38:23.981 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:23.980+0000 7f6d1ea00640 1 --2- 172.21.15.90:0/588541194 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55df43e30790 0x55df43e32ba0 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a48b6a8de7d5eb5f server_cookie=5a7af8db94e5a55b in_seq=0 out_seq=0 2024-04-02T14:38:23.982 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:23.981+0000 7f6d08c00640 1 -- 172.21.15.90:0/588541194 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55df43d2bcd0 con 0x55df43e30790 2024-04-02T14:38:23.983 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:23.981+0000 7f6d08c00640 10 client.26438 handle_command_reply: tid=0 2024-04-02T14:38:23.983 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:23.982+0000 7f6d28633740 1 client.26438 shutdown 2024-04-02T14:38:23.983 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:23.982+0000 7f6d28633740 1 -- 172.21.15.90:0/588541194 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55df43e17670 msgr2=0x55df43e22c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:38:23.983 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:23.982+0000 7f6d28633740 1 --2- 172.21.15.90:0/588541194 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55df43e17670 0x55df43e22c50 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f6d0c023d10 tx=0x7f6d0c00f790 comp rx=0 tx=0).stop 2024-04-02T14:38:23.983 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:23.982+0000 7f6d28633740 1 -- 172.21.15.90:0/588541194 shutdown_connections 2024-04-02T14:38:23.983 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:23.982+0000 7f6d28633740 1 --2- 172.21.15.90:0/588541194 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55df43e30790 0x55df43e32ba0 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:23.984 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:23.982+0000 7f6d28633740 1 --2- 172.21.15.90:0/588541194 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55df43e28bc0 0x55df43e28430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:23.984 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:23.982+0000 7f6d28633740 1 --2- 172.21.15.90:0/588541194 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55df43e17670 0x55df43e22c50 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:23.984 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:23.982+0000 7f6d28633740 1 --2- 172.21.15.90:0/588541194 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55df43e16cc0 0x55df43e22710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:23.984 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:23.982+0000 7f6d08c00640 0 client.26438 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:38:23.984 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:23.982+0000 7f6d28633740 1 -- 172.21.15.90:0/588541194 >> 172.21.15.90:0/588541194 conn(0x55df43e115e0 msgr2=0x55df43e13330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:38:23.984 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:23.983+0000 7f6d28633740 1 -- 172.21.15.90:0/588541194 shutdown_connections 2024-04-02T14:38:23.984 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:23.983+0000 7f6d28633740 1 -- 172.21.15.90:0/588541194 wait complete. 2024-04-02T14:38:23.984 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:23.983+0000 7f6d28633740 20 client.26438 trim_cache size 0 max 16384 2024-04-02T14:38:24.022 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:24.021+0000 7f6d28633740 1 Processor -- start 2024-04-02T14:38:24.023 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:24.021+0000 7f6d28633740 1 -- start start 2024-04-02T14:38:24.024 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:24.022+0000 7f6d28633740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55df43e16cc0 0x55df43e119c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:24.024 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:24.022+0000 7f6d28633740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55df43e17670 0x55df43e384b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:24.024 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:24.022+0000 7f6d28633740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55df43e28bc0 0x55df43e389f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:24.024 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:24.022+0000 7f6d28633740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f6d201ab560 con 0x55df43e28bc0 2024-04-02T14:38:24.024 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:24.022+0000 7f6d28633740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f6d201ad0a0 con 0x55df43e16cc0 2024-04-02T14:38:24.024 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:24.022+0000 7f6d28633740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f6d201abc30 con 0x55df43e17670 2024-04-02T14:38:24.025 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:24.022+0000 7f6d1e000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55df43e17670 0x55df43e384b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:24.025 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:24.022+0000 7f6d1e000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55df43e17670 0x55df43e384b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:37464/0 (socket says 172.21.15.90:37464) 2024-04-02T14:38:24.025 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:24.022+0000 7f6d1e000640 1 -- 172.21.15.90:0/1635983613 learned_addr learned my addr 172.21.15.90:0/1635983613 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:38:24.025 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:24.022+0000 7f6d1f400640 1 --2- 172.21.15.90:0/1635983613 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55df43e28bc0 0x55df43e389f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:24.025 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:24.022+0000 7f6d1ea00640 1 --2- 172.21.15.90:0/1635983613 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55df43e16cc0 0x55df43e119c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:24.025 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:24.022+0000 7f6d1e000640 1 -- 172.21.15.90:0/1635983613 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55df43e16cc0 msgr2=0x55df43e119c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:38:24.025 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:24.022+0000 7f6d1e000640 1 --2- 172.21.15.90:0/1635983613 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55df43e16cc0 0x55df43e119c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:24.025 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:24.022+0000 7f6d1e000640 1 -- 172.21.15.90:0/1635983613 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55df43e28bc0 msgr2=0x55df43e389f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:38:24.025 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:24.022+0000 7f6d1e000640 1 --2- 172.21.15.90:0/1635983613 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55df43e28bc0 0x55df43e389f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:24.025 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:24.022+0000 7f6d1e000640 1 -- 172.21.15.90:0/1635983613 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55df43e1cae0 con 0x55df43e17670 2024-04-02T14:38:24.025 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:24.023+0000 7f6d1e000640 1 --2- 172.21.15.90:0/1635983613 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55df43e17670 0x55df43e384b0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f6d0c009c80 tx=0x7f6d0c037ee0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=72a8ed281631c36b server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:38:24.025 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:24.023+0000 7f6d0b400640 1 -- 172.21.15.90:0/1635983613 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d0c03b030 con 0x55df43e17670 2024-04-02T14:38:24.025 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:24.023+0000 7f6d0b400640 1 -- 172.21.15.90:0/1635983613 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6d0c036530 con 0x55df43e17670 2024-04-02T14:38:24.026 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:24.023+0000 7f6d0b400640 1 -- 172.21.15.90:0/1635983613 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d0c0035f0 con 0x55df43e17670 2024-04-02T14:38:24.027 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:24.024+0000 7f6d28633740 1 -- 172.21.15.90:0/1635983613 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55df43e17670 msgr2=0x55df43e384b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:38:24.027 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:24.024+0000 7f6d28633740 1 --2- 172.21.15.90:0/1635983613 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55df43e17670 0x55df43e384b0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f6d0c009c80 tx=0x7f6d0c037ee0 comp rx=0 tx=0).stop 2024-04-02T14:38:24.027 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:24.024+0000 7f6d28633740 1 -- 172.21.15.90:0/1635983613 shutdown_connections 2024-04-02T14:38:24.027 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:24.024+0000 7f6d28633740 1 --2- 172.21.15.90:0/1635983613 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55df43e28bc0 0x55df43e389f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:24.027 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:24.024+0000 7f6d28633740 1 --2- 172.21.15.90:0/1635983613 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55df43e17670 0x55df43e384b0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:24.027 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:24.025+0000 7f6d28633740 1 --2- 172.21.15.90:0/1635983613 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55df43e16cc0 0x55df43e119c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:24.027 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:24.025+0000 7f6d28633740 1 -- 172.21.15.90:0/1635983613 >> 172.21.15.90:0/1635983613 conn(0x55df43e115e0 msgr2=0x55df43e10640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:38:24.027 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:24.025+0000 7f6d28633740 1 -- 172.21.15.90:0/1635983613 shutdown_connections 2024-04-02T14:38:24.027 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:24.025+0000 7f6d28633740 1 -- 172.21.15.90:0/1635983613 wait complete. 2024-04-02T14:38:24.027 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:24.025+0000 7f6d28633740 1 -- start start 2024-04-02T14:38:24.027 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:24.026+0000 7f6d28633740 1 Processor -- start 2024-04-02T14:38:24.028 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:24.026+0000 7f6d28633740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55df43e16cc0 0x55df43e42520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:24.028 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:24.026+0000 7f6d28633740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55df43e17670 0x55df43e42a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:24.028 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:24.026+0000 7f6d28633740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55df43e28bc0 0x55df43e42fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:24.029 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:24.026+0000 7f6d28633740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f6d201ac9d0 con 0x55df43e16cc0 2024-04-02T14:38:24.029 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:24.026+0000 7f6d28633740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f6d201ac300 con 0x55df43e28bc0 2024-04-02T14:38:24.029 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:24.026+0000 7f6d28633740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f6d201a9580 con 0x55df43e17670 2024-04-02T14:38:24.029 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:24.026+0000 7f6d28633740 10 client.0 _ll_register_callbacks cb 0x7f6d0c023920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:38:24.029 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:24.027+0000 7f6d1f400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55df43e28bc0 0x55df43e42fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:24.029 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:24.027+0000 7f6d1f400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55df43e28bc0 0x55df43e42fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:47770/0 (socket says 172.21.15.90:47770) 2024-04-02T14:38:24.029 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:24.027+0000 7f6d1f400640 1 -- 172.21.15.90:0/3736285659 learned_addr learned my addr 172.21.15.90:0/3736285659 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:38:24.029 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:24.027+0000 7f6d1e000640 1 --2- 172.21.15.90:0/3736285659 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55df43e17670 0x55df43e42a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:24.031 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:24.028+0000 7f6d1f400640 1 -- 172.21.15.90:0/3736285659 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55df43e17670 msgr2=0x55df43e42a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:38:24.031 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:24.028+0000 7f6d1f400640 1 --2- 172.21.15.90:0/3736285659 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55df43e17670 0x55df43e42a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:24.031 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:24.028+0000 7f6d1f400640 1 -- 172.21.15.90:0/3736285659 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55df43e16cc0 msgr2=0x55df43e42520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:38:24.031 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:24.028+0000 7f6d1f400640 1 --2- 172.21.15.90:0/3736285659 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55df43e16cc0 0x55df43e42520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:24.031 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:24.028+0000 7f6d1f400640 1 -- 172.21.15.90:0/3736285659 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55df43e17410 con 0x55df43e28bc0 2024-04-02T14:38:24.031 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:24.028+0000 7f6d1f400640 1 --2- 172.21.15.90:0/3736285659 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55df43e28bc0 0x55df43e42fa0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f6d18009be0 tx=0x7f6d18009500 comp rx=0 tx=0).ready entity=mon.2 client_cookie=69c0f61e4fba3364 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:38:24.031 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:24.028+0000 7f6d08c00640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:38:24.032 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:24.029+0000 7f6d08c00640 1 -- 172.21.15.90:0/3736285659 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d18004240 con 0x55df43e28bc0 2024-04-02T14:38:24.032 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:24.029+0000 7f6d08c00640 1 -- 172.21.15.90:0/3736285659 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f6d180043e0 con 0x55df43e28bc0 2024-04-02T14:38:24.032 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:24.029+0000 7f6d28633740 1 -- 172.21.15.90:0/3736285659 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6d0c036530 con 0x55df43e28bc0 2024-04-02T14:38:24.032 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:24.029+0000 7f6d08c00640 1 -- 172.21.15.90:0/3736285659 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f6d180045e0 con 0x55df43e28bc0 2024-04-02T14:38:24.032 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:24.029+0000 7f6d28633740 10 client.26444 fetch_fsmap learned FSMap version 27 2024-04-02T14:38:24.032 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:24.029+0000 7f6d28633740 1 -- 172.21.15.90:0/3736285659 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55df43b5c620 con 0x55df43e28bc0 2024-04-02T14:38:24.032 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:24.029+0000 7f6d08c00640 1 -- 172.21.15.90:0/3736285659 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6d0c036530 con 0x55df43e28bc0 2024-04-02T14:38:24.032 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:24.029+0000 7f6d08c00640 1 -- 172.21.15.90:0/3736285659 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f6d18034430 con 0x55df43e28bc0 2024-04-02T14:38:24.032 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:24.030+0000 7f6d28633740 10 client.26444 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:38:24.032 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:24.030+0000 7f6d28633740 10 client.26444 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-02T14:38:24.032 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:24.030+0000 7f6d28633740 20 client.26444 populate_metadata read hostname 'smithi090' 2024-04-02T14:38:24.032 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:24.030+0000 7f6d28633740 1 --2- 172.21.15.90:0/3736285659 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55df43e97f40 0x55df43e9a380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:24.032 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:24.030+0000 7f6d28633740 4 client.26444 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-02T14:38:24.032 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:24.030+0000 7f6d28633740 1 -- 172.21.15.90:0/3736285659 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55df43d892b0 con 0x55df43e97f40 2024-04-02T14:38:24.033 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:24.030+0000 7f6d1ea00640 1 --2- 172.21.15.90:0/3736285659 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55df43e97f40 0x55df43e9a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:24.033 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:24.030+0000 7f6d1ea00640 1 --2- 172.21.15.90:0/3736285659 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55df43e97f40 0x55df43e9a380 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=201e44c745f25ac4 server_cookie=c2f2aff58a34d5ec in_seq=0 out_seq=0 2024-04-02T14:38:24.033 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:24.030+0000 7f6d08c00640 10 client.26444 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:38:24.054 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:24.052+0000 7f6d08c00640 1 -- 172.21.15.90:0/3736285659 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55df43d892b0 con 0x55df43e97f40 2024-04-02T14:38:24.054 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:24.052+0000 7f6d08c00640 10 client.26444 handle_command_reply: tid=0 2024-04-02T14:38:24.054 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:24.052+0000 7f6d28633740 1 client.26444 shutdown 2024-04-02T14:38:24.054 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:24.052+0000 7f6d28633740 1 -- 172.21.15.90:0/3736285659 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55df43e28bc0 msgr2=0x55df43e42fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:38:24.054 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:24.052+0000 7f6d28633740 1 --2- 172.21.15.90:0/3736285659 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55df43e28bc0 0x55df43e42fa0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f6d18009be0 tx=0x7f6d18009500 comp rx=0 tx=0).stop 2024-04-02T14:38:24.054 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:24.052+0000 7f6d28633740 1 -- 172.21.15.90:0/3736285659 shutdown_connections 2024-04-02T14:38:24.054 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:24.052+0000 7f6d28633740 1 --2- 172.21.15.90:0/3736285659 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55df43e97f40 0x55df43e9a380 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:24.054 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:24.052+0000 7f6d28633740 1 --2- 172.21.15.90:0/3736285659 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55df43e28bc0 0x55df43e42fa0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:24.054 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:24.052+0000 7f6d28633740 1 --2- 172.21.15.90:0/3736285659 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55df43e17670 0x55df43e42a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:24.054 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:24.052+0000 7f6d08c00640 0 client.26444 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:38:24.055 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:24.052+0000 7f6d28633740 1 --2- 172.21.15.90:0/3736285659 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55df43e16cc0 0x55df43e42520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:24.055 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:24.052+0000 7f6d28633740 1 -- 172.21.15.90:0/3736285659 >> 172.21.15.90:0/3736285659 conn(0x55df43e115e0 msgr2=0x55df43e10640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:38:24.055 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:24.052+0000 7f6d28633740 1 -- 172.21.15.90:0/3736285659 shutdown_connections 2024-04-02T14:38:24.055 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:24.052+0000 7f6d28633740 1 -- 172.21.15.90:0/3736285659 wait complete. 2024-04-02T14:38:24.055 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:24.053+0000 7f6d28633740 20 client.26444 trim_cache size 0 max 16384 2024-04-02T14:38:24.055 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:{ 2024-04-02T14:38:24.055 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "message": "", 2024-04-02T14:38:24.056 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "return_code": 0 2024-04-02T14:38:24.056 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:} 2024-04-02T14:38:24.056 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:24.054+0000 7f6d25a00640 1 -- 172.21.15.90:0/3924608110 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f6d040a3e40 msgr2=0x7f6d040a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:38:24.056 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:24.054+0000 7f6d25a00640 1 --2- 172.21.15.90:0/3924608110 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f6d040a3e40 0x7f6d040a6300 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f6d180099a0 tx=0x7f6d18005fd0 comp rx=0 tx=0).stop 2024-04-02T14:38:24.056 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:24.054+0000 7f6d25a00640 1 -- 172.21.15.90:0/3924608110 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6d20103f50 msgr2=0x7f6d2019a510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:38:24.056 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:24.054+0000 7f6d25a00640 1 --2- 172.21.15.90:0/3924608110 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6d20103f50 0x7f6d2019a510 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f6d1000e9f0 tx=0x7f6d1000eeb0 comp rx=0 tx=0).stop 2024-04-02T14:38:24.056 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:24.054+0000 7f6d25a00640 1 -- 172.21.15.90:0/3924608110 shutdown_connections 2024-04-02T14:38:24.057 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:24.054+0000 7f6d25a00640 1 --2- 172.21.15.90:0/3924608110 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f6d040a3e40 0x7f6d040a6300 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:24.057 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:24.054+0000 7f6d25a00640 1 --2- 172.21.15.90:0/3924608110 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6d2019ed80 0x7f6d2019f230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:24.057 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:24.054+0000 7f6d25a00640 1 --2- 172.21.15.90:0/3924608110 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6d20109020 0x7f6d2019aa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:24.057 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:24.054+0000 7f6d25a00640 1 --2- 172.21.15.90:0/3924608110 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6d20103f50 0x7f6d2019a510 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:24.057 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:24.054+0000 7f6d25a00640 1 -- 172.21.15.90:0/3924608110 >> 172.21.15.90:0/3924608110 conn(0x7f6d200ff7b0 msgr2=0x7f6d201001f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:38:24.057 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:24.054+0000 7f6d25a00640 1 -- 172.21.15.90:0/3924608110 shutdown_connections 2024-04-02T14:38:24.057 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:24.054+0000 7f6d25a00640 1 -- 172.21.15.90:0/3924608110 wait complete. 2024-04-02T14:38:24.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:38:23 smithi092 ceph-mon[36497]: pgmap v108: 97 pgs: 97 active+clean; 2.1 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 32 MiB/s wr, 1.12k op/s 2024-04-02T14:38:24.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:38:23 smithi177 ceph-mon[36431]: pgmap v108: 97 pgs: 97 active+clean; 2.1 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 32 MiB/s wr, 1.12k op/s 2024-04-02T14:38:24.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:38:23 smithi090 ceph-mon[31674]: pgmap v108: 97 pgs: 97 active+clean; 2.1 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 32 MiB/s wr, 1.12k op/s 2024-04-02T14:38:26.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:38:25 smithi092 ceph-mon[36497]: pgmap v109: 97 pgs: 97 active+clean; 2.1 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 35 MiB/s wr, 1.45k op/s 2024-04-02T14:38:26.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:38:25 smithi177 ceph-mon[36431]: pgmap v109: 97 pgs: 97 active+clean; 2.1 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 35 MiB/s wr, 1.45k op/s 2024-04-02T14:38:26.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:38:25 smithi090 ceph-mon[31674]: pgmap v109: 97 pgs: 97 active+clean; 2.1 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 35 MiB/s wr, 1.45k op/s 2024-04-02T14:38:26.832 INFO:teuthology.orchestra.run.smithi090.stderr:progress: 175.0 s, 2921.5 tps, lat 33.982 ms stddev 44.471 2024-04-02T14:38:28.077 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-02T14:38:28.078 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-02T14:38:28.110 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:38:27 smithi092 ceph-mon[36497]: pgmap v110: 97 pgs: 97 active+clean; 2.1 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 34 MiB/s wr, 1.31k op/s 2024-04-02T14:38:28.174 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:38:27 smithi090 ceph-mon[31674]: pgmap v110: 97 pgs: 97 active+clean; 2.1 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 34 MiB/s wr, 1.31k op/s 2024-04-02T14:38:28.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:38:27 smithi177 ceph-mon[36431]: pgmap v110: 97 pgs: 97 active+clean; 2.1 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 34 MiB/s wr, 1.31k op/s 2024-04-02T14:38:28.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.372+0000 7f8d3d200640 1 -- 172.21.15.90:0/3765312212 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8d381045c0 msgr2=0x7f8d38104a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:38:28.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.372+0000 7f8d3d200640 1 --2- 172.21.15.90:0/3765312212 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8d381045c0 0x7f8d38104a40 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f8d30009cd0 tx=0x7f8d3002c100 comp rx=0 tx=0).stop 2024-04-02T14:38:28.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.373+0000 7f8d3d200640 1 -- 172.21.15.90:0/3765312212 shutdown_connections 2024-04-02T14:38:28.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.373+0000 7f8d3d200640 1 --2- 172.21.15.90:0/3765312212 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8d381045c0 0x7f8d38104a40 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:28.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.373+0000 7f8d3d200640 1 --2- 172.21.15.90:0/3765312212 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8d38069240 0x7f8d380696a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:28.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.373+0000 7f8d3d200640 1 --2- 172.21.15.90:0/3765312212 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8d38106a50 0x7f8d38068d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:28.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.373+0000 7f8d3d200640 1 -- 172.21.15.90:0/3765312212 >> 172.21.15.90:0/3765312212 conn(0x7f8d38077b00 msgr2=0x7f8d38077f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:38:28.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.373+0000 7f8d3d200640 1 -- 172.21.15.90:0/3765312212 shutdown_connections 2024-04-02T14:38:28.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.373+0000 7f8d3d200640 1 -- 172.21.15.90:0/3765312212 wait complete. 2024-04-02T14:38:28.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.374+0000 7f8d3d200640 1 Processor -- start 2024-04-02T14:38:28.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.376+0000 7f8d3d200640 1 -- start start 2024-04-02T14:38:28.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.377+0000 7f8d3d200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8d38069240 0x7f8d38114cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:28.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.377+0000 7f8d3d200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8d381045c0 0x7f8d3810ddc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:28.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.377+0000 7f8d35400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8d381045c0 0x7f8d3810ddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:28.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.377+0000 7f8d35400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8d381045c0 0x7f8d3810ddc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:47786/0 (socket says 172.21.15.90:47786) 2024-04-02T14:38:28.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.377+0000 7f8d35400640 1 -- 172.21.15.90:0/3964979143 learned_addr learned my addr 172.21.15.90:0/3964979143 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:38:28.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.377+0000 7f8d35e00640 1 --2- 172.21.15.90:0/3964979143 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8d38069240 0x7f8d38114cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:28.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.377+0000 7f8d3d200640 1 --2- 172.21.15.90:0/3964979143 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8d38106a50 0x7f8d3810e300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:28.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.378+0000 7f8d36800640 1 --2- 172.21.15.90:0/3964979143 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8d38106a50 0x7f8d3810e300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:28.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.378+0000 7f8d3d200640 1 -- 172.21.15.90:0/3964979143 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f8d381184b0 con 0x7f8d38069240 2024-04-02T14:38:28.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.378+0000 7f8d3d200640 1 -- 172.21.15.90:0/3964979143 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f8d38118330 con 0x7f8d381045c0 2024-04-02T14:38:28.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.378+0000 7f8d3d200640 1 -- 172.21.15.90:0/3964979143 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f8d38118630 con 0x7f8d38106a50 2024-04-02T14:38:28.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.378+0000 7f8d35400640 1 -- 172.21.15.90:0/3964979143 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8d38106a50 msgr2=0x7f8d3810e300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:38:28.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.379+0000 7f8d35400640 1 --2- 172.21.15.90:0/3964979143 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8d38106a50 0x7f8d3810e300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:28.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.379+0000 7f8d35400640 1 -- 172.21.15.90:0/3964979143 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8d38069240 msgr2=0x7f8d38114cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:38:28.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.379+0000 7f8d35400640 1 --2- 172.21.15.90:0/3964979143 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8d38069240 0x7f8d38114cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:28.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.379+0000 7f8d35400640 1 -- 172.21.15.90:0/3964979143 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d3810e8c0 con 0x7f8d381045c0 2024-04-02T14:38:28.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.379+0000 7f8d36800640 1 --2- 172.21.15.90:0/3964979143 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8d38106a50 0x7f8d3810e300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:38:28.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.379+0000 7f8d35e00640 1 --2- 172.21.15.90:0/3964979143 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8d38069240 0x7f8d38114cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:38:28.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.380+0000 7f8d35400640 1 --2- 172.21.15.90:0/3964979143 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8d381045c0 0x7f8d3810ddc0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f8d2400b4a0 tx=0x7f8d2400b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:38:28.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.381+0000 7f8d2ea00640 1 -- 172.21.15.90:0/3964979143 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d24014070 con 0x7f8d381045c0 2024-04-02T14:38:28.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.381+0000 7f8d2ea00640 1 -- 172.21.15.90:0/3964979143 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8d24010af0 con 0x7f8d381045c0 2024-04-02T14:38:28.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.381+0000 7f8d2ea00640 1 -- 172.21.15.90:0/3964979143 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d24017d10 con 0x7f8d381045c0 2024-04-02T14:38:28.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.381+0000 7f8d3d200640 1 -- 172.21.15.90:0/3964979143 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d3810ebb0 con 0x7f8d381045c0 2024-04-02T14:38:28.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.381+0000 7f8d3d200640 1 -- 172.21.15.90:0/3964979143 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d38112b40 con 0x7f8d381045c0 2024-04-02T14:38:28.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.386+0000 7f8d3ff96740 1 Processor -- start 2024-04-02T14:38:28.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.387+0000 7f8d3ff96740 1 -- start start 2024-04-02T14:38:28.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.387+0000 7f8d2ea00640 1 -- 172.21.15.90:0/3964979143 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f8d24017070 con 0x7f8d381045c0 2024-04-02T14:38:28.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.388+0000 7f8d3ff96740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5556631c57b0 0x5556631c5b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:28.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.388+0000 7f8d3ff96740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5556631c60d0 0x5556631ce770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:28.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.388+0000 7f8d3ff96740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5556631cecb0 0x5556631d10a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:28.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.388+0000 7f8d3ff96740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x555663129770 con 0x5556631c60d0 2024-04-02T14:38:28.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.388+0000 7f8d3ff96740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x5556630d1700 con 0x5556631cecb0 2024-04-02T14:38:28.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.388+0000 7f8d3ff96740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x555663125720 con 0x5556631c57b0 2024-04-02T14:38:28.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.389+0000 7f8d36800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5556631c57b0 0x5556631c5b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:28.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.389+0000 7f8d36800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5556631c57b0 0x5556631c5b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:37504/0 (socket says 172.21.15.90:37504) 2024-04-02T14:38:28.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.389+0000 7f8d36800640 1 -- 172.21.15.90:0/253163095 learned_addr learned my addr 172.21.15.90:0/253163095 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:38:28.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.389+0000 7f8d35e00640 1 --2- 172.21.15.90:0/253163095 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5556631c60d0 0x5556631ce770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:28.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.389+0000 7f8d35400640 1 --2- 172.21.15.90:0/253163095 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5556631cecb0 0x5556631d10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:28.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.390+0000 7f8d36800640 1 -- 172.21.15.90:0/253163095 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5556631cecb0 msgr2=0x5556631d10a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:38:28.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.390+0000 7f8d36800640 1 --2- 172.21.15.90:0/253163095 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5556631cecb0 0x5556631d10a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:28.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.390+0000 7f8d36800640 1 -- 172.21.15.90:0/253163095 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5556631c60d0 msgr2=0x5556631ce770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:38:28.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.390+0000 7f8d36800640 1 --2- 172.21.15.90:0/253163095 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5556631c60d0 0x5556631ce770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:28.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.390+0000 7f8d36800640 1 -- 172.21.15.90:0/253163095 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555662f077d0 con 0x5556631c57b0 2024-04-02T14:38:28.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.390+0000 7f8d35e00640 1 --2- 172.21.15.90:0/253163095 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5556631c60d0 0x5556631ce770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:38:28.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.391+0000 7f8d36800640 1 --2- 172.21.15.90:0/253163095 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5556631c57b0 0x5556631c5b90 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f8d30009cd0 tx=0x7f8d30009800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4dfb2b2391aef959 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:38:28.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.392+0000 7f8d2ea00640 1 --2- 172.21.15.90:0/3964979143 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f8d1c0a3ef0 0x7f8d1c0a63b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:28.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.393+0000 7f8d23400640 1 -- 172.21.15.90:0/253163095 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d30039040 con 0x5556631c57b0 2024-04-02T14:38:28.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.393+0000 7f8d23400640 1 -- 172.21.15.90:0/253163095 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8d30032df0 con 0x5556631c57b0 2024-04-02T14:38:28.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.393+0000 7f8d23400640 1 -- 172.21.15.90:0/253163095 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d3003fdd0 con 0x5556631c57b0 2024-04-02T14:38:28.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.394+0000 7f8d2ea00640 1 -- 172.21.15.90:0/3964979143 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f8d240cf990 con 0x7f8d381045c0 2024-04-02T14:38:28.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.394+0000 7f8d3ff96740 1 -- 172.21.15.90:0/253163095 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5556631c57b0 msgr2=0x5556631c5b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:38:28.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.394+0000 7f8d3ff96740 1 --2- 172.21.15.90:0/253163095 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5556631c57b0 0x5556631c5b90 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f8d30009cd0 tx=0x7f8d30009800 comp rx=0 tx=0).stop 2024-04-02T14:38:28.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.394+0000 7f8d35e00640 1 --2- 172.21.15.90:0/3964979143 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f8d1c0a3ef0 0x7f8d1c0a63b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:28.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.394+0000 7f8d3ff96740 1 -- 172.21.15.90:0/253163095 shutdown_connections 2024-04-02T14:38:28.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.394+0000 7f8d3ff96740 1 --2- 172.21.15.90:0/253163095 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5556631cecb0 0x5556631d10a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:28.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.394+0000 7f8d3ff96740 1 --2- 172.21.15.90:0/253163095 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5556631c60d0 0x5556631ce770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:28.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.394+0000 7f8d3ff96740 1 --2- 172.21.15.90:0/253163095 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5556631c57b0 0x5556631c5b90 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:28.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.394+0000 7f8d3ff96740 1 -- 172.21.15.90:0/253163095 >> 172.21.15.90:0/253163095 conn(0x5556631bfee0 msgr2=0x5556631c02c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:38:28.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.396+0000 7f8d3ff96740 1 -- 172.21.15.90:0/253163095 shutdown_connections 2024-04-02T14:38:28.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.397+0000 7f8d35e00640 1 --2- 172.21.15.90:0/3964979143 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f8d1c0a3ef0 0x7f8d1c0a63b0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x5556631c5280 tx=0x7f8d2800b390 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:38:28.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.397+0000 7f8d3ff96740 1 -- 172.21.15.90:0/253163095 wait complete. 2024-04-02T14:38:28.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.397+0000 7f8d3ff96740 1 -- start start 2024-04-02T14:38:28.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.398+0000 7f8d3ff96740 1 Processor -- start 2024-04-02T14:38:28.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.398+0000 7f8d3ff96740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5556631c57b0 0x5556631cbd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:28.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.398+0000 7f8d3ff96740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5556631c60d0 0x5556631cc2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:28.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.398+0000 7f8d3ff96740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5556631cecb0 0x5556631cc7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:28.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.398+0000 7f8d3ff96740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5556631d1a40 con 0x5556631cecb0 2024-04-02T14:38:28.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.398+0000 7f8d3ff96740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x5556630e4ac0 con 0x5556631c60d0 2024-04-02T14:38:28.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.398+0000 7f8d3ff96740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5556631d1bc0 con 0x5556631c57b0 2024-04-02T14:38:28.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.398+0000 7f8d3ff96740 10 client.0 _ll_register_callbacks cb 0x5556630e7180 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:38:28.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.398+0000 7f8d35400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5556631c60d0 0x5556631cc2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:28.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.398+0000 7f8d35400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5556631c60d0 0x5556631cc2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:47816/0 (socket says 172.21.15.90:47816) 2024-04-02T14:38:28.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.398+0000 7f8d35400640 1 -- 172.21.15.90:0/878994241 learned_addr learned my addr 172.21.15.90:0/878994241 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:38:28.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.398+0000 7f8d36800640 1 --2- 172.21.15.90:0/878994241 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5556631cecb0 0x5556631cc7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:28.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.399+0000 7f8d35400640 1 -- 172.21.15.90:0/878994241 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5556631c57b0 msgr2=0x5556631cbd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:38:28.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.399+0000 7f8d35e00640 1 --2- 172.21.15.90:0/878994241 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5556631c57b0 0x5556631cbd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:28.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.399+0000 7f8d35400640 1 --2- 172.21.15.90:0/878994241 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5556631c57b0 0x5556631cbd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:28.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.399+0000 7f8d35400640 1 -- 172.21.15.90:0/878994241 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5556631cecb0 msgr2=0x5556631cc7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:38:28.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.399+0000 7f8d35400640 1 --2- 172.21.15.90:0/878994241 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5556631cecb0 0x5556631cc7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:28.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.399+0000 7f8d35400640 1 -- 172.21.15.90:0/878994241 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555662e3d3f0 con 0x5556631c60d0 2024-04-02T14:38:28.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.399+0000 7f8d36800640 1 --2- 172.21.15.90:0/878994241 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5556631cecb0 0x5556631cc7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:38:28.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.399+0000 7f8d35e00640 1 --2- 172.21.15.90:0/878994241 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5556631c57b0 0x5556631cbd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:38:28.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.400+0000 7f8d35400640 1 --2- 172.21.15.90:0/878994241 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5556631c60d0 0x5556631cc2a0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f8d24000c00 tx=0x7f8d2409b350 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4ce5bb21492b5003 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:38:28.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.400+0000 7f8d17e00640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:38:28.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.400+0000 7f8d17e00640 1 -- 172.21.15.90:0/878994241 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d2401e050 con 0x5556631c60d0 2024-04-02T14:38:28.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.400+0000 7f8d17e00640 1 -- 172.21.15.90:0/878994241 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f8d24098060 con 0x5556631c60d0 2024-04-02T14:38:28.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.401+0000 7f8d3ff96740 1 -- 172.21.15.90:0/878994241 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8d30032df0 con 0x5556631c60d0 2024-04-02T14:38:28.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.401+0000 7f8d17e00640 1 -- 172.21.15.90:0/878994241 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f8d2409b9f0 con 0x5556631c60d0 2024-04-02T14:38:28.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.401+0000 7f8d3ff96740 10 client.26456 fetch_fsmap learned FSMap version 27 2024-04-02T14:38:28.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.401+0000 7f8d3ff96740 1 -- 172.21.15.90:0/878994241 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x555662dfd0f0 con 0x5556631c60d0 2024-04-02T14:38:28.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.402+0000 7f8d17e00640 1 -- 172.21.15.90:0/878994241 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8d30032df0 con 0x5556631c60d0 2024-04-02T14:38:28.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.402+0000 7f8d17e00640 1 -- 172.21.15.90:0/878994241 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f8d2409ae40 con 0x5556631c60d0 2024-04-02T14:38:28.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.402+0000 7f8d3ff96740 10 client.26456 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:38:28.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.402+0000 7f8d3ff96740 10 client.26456 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:38:28.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.402+0000 7f8d3ff96740 20 client.26456 populate_metadata read hostname 'smithi090' 2024-04-02T14:38:28.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.402+0000 7f8d3ff96740 1 --2- 172.21.15.90:0/878994241 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5556631ded50 0x5556631e1140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:28.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.402+0000 7f8d3ff96740 4 client.26456 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:38:28.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.402+0000 7f8d3ff96740 1 -- 172.21.15.90:0/878994241 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5556630ca630 con 0x5556631ded50 2024-04-02T14:38:28.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.403+0000 7f8d35e00640 1 --2- 172.21.15.90:0/878994241 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5556631ded50 0x5556631e1140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:28.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.404+0000 7f8d35e00640 1 --2- 172.21.15.90:0/878994241 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5556631ded50 0x5556631e1140 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6cad0b72f6e2dff6 server_cookie=9d52bb86cfe2b6f1 in_seq=0 out_seq=0 2024-04-02T14:38:28.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.404+0000 7f8d17e00640 10 client.26456 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:38:28.406 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.405+0000 7f8d17e00640 1 -- 172.21.15.90:0/878994241 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5556630ca630 con 0x5556631ded50 2024-04-02T14:38:28.406 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.405+0000 7f8d17e00640 10 client.26456 handle_command_reply: tid=0 2024-04-02T14:38:28.406 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.405+0000 7f8d3ff96740 1 client.26456 shutdown 2024-04-02T14:38:28.406 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.405+0000 7f8d3ff96740 1 -- 172.21.15.90:0/878994241 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5556631c60d0 msgr2=0x5556631cc2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:38:28.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.405+0000 7f8d3ff96740 1 --2- 172.21.15.90:0/878994241 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5556631c60d0 0x5556631cc2a0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f8d24000c00 tx=0x7f8d2409b350 comp rx=0 tx=0).stop 2024-04-02T14:38:28.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.405+0000 7f8d3ff96740 1 -- 172.21.15.90:0/878994241 shutdown_connections 2024-04-02T14:38:28.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.405+0000 7f8d3ff96740 1 --2- 172.21.15.90:0/878994241 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5556631ded50 0x5556631e1140 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:28.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.405+0000 7f8d3ff96740 1 --2- 172.21.15.90:0/878994241 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5556631cecb0 0x5556631cc7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:28.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.405+0000 7f8d3ff96740 1 --2- 172.21.15.90:0/878994241 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5556631c60d0 0x5556631cc2a0 secure :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f8d24000c00 tx=0x7f8d2409b350 comp rx=0 tx=0).stop 2024-04-02T14:38:28.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.405+0000 7f8d3ff96740 1 --2- 172.21.15.90:0/878994241 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5556631c57b0 0x5556631cbd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:28.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.405+0000 7f8d3ff96740 1 -- 172.21.15.90:0/878994241 >> 172.21.15.90:0/878994241 conn(0x5556631bfee0 msgr2=0x5556631c1c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:38:28.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.405+0000 7f8d17e00640 0 client.26456 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:38:28.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.406+0000 7f8d3ff96740 1 -- 172.21.15.90:0/878994241 shutdown_connections 2024-04-02T14:38:28.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.406+0000 7f8d3ff96740 1 -- 172.21.15.90:0/878994241 wait complete. 2024-04-02T14:38:28.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.406+0000 7f8d3ff96740 20 client.26456 trim_cache size 0 max 16384 2024-04-02T14:38:28.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.439+0000 7f8d3ff96740 1 Processor -- start 2024-04-02T14:38:28.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.440+0000 7f8d3ff96740 1 -- start start 2024-04-02T14:38:28.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.440+0000 7f8d3ff96740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5556631c57b0 0x5556631ce490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:28.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.440+0000 7f8d3ff96740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5556631cecb0 0x5556631da930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:28.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.440+0000 7f8d3ff96740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5556631e6930 0x5556631e6db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:28.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.440+0000 7f8d3ff96740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f8d381bc790 con 0x5556631c57b0 2024-04-02T14:38:28.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.440+0000 7f8d3ff96740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f8d381be2d0 con 0x5556631cecb0 2024-04-02T14:38:28.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.440+0000 7f8d3ff96740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f8d381bce60 con 0x5556631e6930 2024-04-02T14:38:28.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.440+0000 7f8d36800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5556631e6930 0x5556631e6db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:28.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.440+0000 7f8d36800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5556631e6930 0x5556631e6db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:37526/0 (socket says 172.21.15.90:37526) 2024-04-02T14:38:28.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.440+0000 7f8d36800640 1 -- 172.21.15.90:0/2362036969 learned_addr learned my addr 172.21.15.90:0/2362036969 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:38:28.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.441+0000 7f8d36800640 1 -- 172.21.15.90:0/2362036969 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5556631cecb0 msgr2=0x5556631da930 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-02T14:38:28.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.441+0000 7f8d35400640 1 --2- 172.21.15.90:0/2362036969 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5556631cecb0 0x5556631da930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:28.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.441+0000 7f8d36800640 1 --2- 172.21.15.90:0/2362036969 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5556631cecb0 0x5556631da930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:28.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.441+0000 7f8d36800640 1 -- 172.21.15.90:0/2362036969 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5556631c57b0 msgr2=0x5556631ce490 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-02T14:38:28.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.442+0000 7f8d35e00640 1 --2- 172.21.15.90:0/2362036969 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5556631c57b0 0x5556631ce490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:28.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.442+0000 7f8d36800640 1 --2- 172.21.15.90:0/2362036969 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5556631c57b0 0x5556631ce490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:28.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.443+0000 7f8d36800640 1 -- 172.21.15.90:0/2362036969 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5556631e14b0 con 0x5556631e6930 2024-04-02T14:38:28.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.443+0000 7f8d35e00640 1 --2- 172.21.15.90:0/2362036969 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5556631c57b0 0x5556631ce490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:38:28.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.443+0000 7f8d35400640 1 --2- 172.21.15.90:0/2362036969 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5556631cecb0 0x5556631da930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:38:28.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.443+0000 7f8d36800640 1 --2- 172.21.15.90:0/2362036969 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5556631e6930 0x5556631e6db0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f8d30009ca0 tx=0x7f8d300099f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=883995e8d85da5fb server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:38:28.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.444+0000 7f8d22a00640 1 -- 172.21.15.90:0/2362036969 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d30039040 con 0x5556631e6930 2024-04-02T14:38:28.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.444+0000 7f8d22a00640 1 -- 172.21.15.90:0/2362036969 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8d30032df0 con 0x5556631e6930 2024-04-02T14:38:28.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.444+0000 7f8d22a00640 1 -- 172.21.15.90:0/2362036969 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d300346a0 con 0x5556631e6930 2024-04-02T14:38:28.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.445+0000 7f8d3ff96740 1 -- 172.21.15.90:0/2362036969 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5556631e6930 msgr2=0x5556631e6db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:38:28.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.445+0000 7f8d3ff96740 1 --2- 172.21.15.90:0/2362036969 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5556631e6930 0x5556631e6db0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f8d30009ca0 tx=0x7f8d300099f0 comp rx=0 tx=0).stop 2024-04-02T14:38:28.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.445+0000 7f8d3ff96740 1 -- 172.21.15.90:0/2362036969 shutdown_connections 2024-04-02T14:38:28.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.445+0000 7f8d3ff96740 1 --2- 172.21.15.90:0/2362036969 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5556631e6930 0x5556631e6db0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:28.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.445+0000 7f8d3ff96740 1 --2- 172.21.15.90:0/2362036969 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5556631cecb0 0x5556631da930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:28.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.445+0000 7f8d3ff96740 1 --2- 172.21.15.90:0/2362036969 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5556631c57b0 0x5556631ce490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:28.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.445+0000 7f8d3ff96740 1 -- 172.21.15.90:0/2362036969 >> 172.21.15.90:0/2362036969 conn(0x5556631bfee0 msgr2=0x5556631c02c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:38:28.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.445+0000 7f8d3ff96740 1 -- 172.21.15.90:0/2362036969 shutdown_connections 2024-04-02T14:38:28.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.446+0000 7f8d3ff96740 1 -- 172.21.15.90:0/2362036969 wait complete. 2024-04-02T14:38:28.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.446+0000 7f8d3ff96740 1 -- start start 2024-04-02T14:38:28.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.446+0000 7f8d3ff96740 1 Processor -- start 2024-04-02T14:38:28.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.447+0000 7f8d3ff96740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5556631c57b0 0x5556631d2fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:28.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.447+0000 7f8d3ff96740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5556631cecb0 0x5556631d3510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:28.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.447+0000 7f8d3ff96740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5556631e6930 0x5556632441a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:28.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.447+0000 7f8d3ff96740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f8d381bdc00 con 0x5556631cecb0 2024-04-02T14:38:28.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.447+0000 7f8d3ff96740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f8d381bd530 con 0x5556631e6930 2024-04-02T14:38:28.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.447+0000 7f8d3ff96740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f8d381ba770 con 0x5556631c57b0 2024-04-02T14:38:28.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.447+0000 7f8d3ff96740 10 client.0 _ll_register_callbacks cb 0x7f8d2409b7c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:38:28.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.447+0000 7f8d35e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5556631c57b0 0x5556631d2fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:28.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.448+0000 7f8d35e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5556631c57b0 0x5556631d2fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:37530/0 (socket says 172.21.15.90:37530) 2024-04-02T14:38:28.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.448+0000 7f8d35e00640 1 -- 172.21.15.90:0/1569829333 learned_addr learned my addr 172.21.15.90:0/1569829333 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:38:28.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.448+0000 7f8d35400640 1 --2- 172.21.15.90:0/1569829333 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5556631cecb0 0x5556631d3510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:28.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.448+0000 7f8d36800640 1 --2- 172.21.15.90:0/1569829333 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5556631e6930 0x5556632441a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:28.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.448+0000 7f8d35e00640 1 -- 172.21.15.90:0/1569829333 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5556631e6930 msgr2=0x5556632441a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:38:28.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.448+0000 7f8d35e00640 1 --2- 172.21.15.90:0/1569829333 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5556631e6930 0x5556632441a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:28.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.448+0000 7f8d35e00640 1 -- 172.21.15.90:0/1569829333 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5556631cecb0 msgr2=0x5556631d3510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:38:28.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.448+0000 7f8d35e00640 1 --2- 172.21.15.90:0/1569829333 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5556631cecb0 0x5556631d3510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:28.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.448+0000 7f8d35e00640 1 -- 172.21.15.90:0/1569829333 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5556632446e0 con 0x5556631c57b0 2024-04-02T14:38:28.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.448+0000 7f8d35e00640 1 --2- 172.21.15.90:0/1569829333 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5556631c57b0 0x5556631d2fd0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f8d28005b60 tx=0x7f8d280046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ebfab9d38394b5d4 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:38:28.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.448+0000 7f8d17e00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:38:28.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.449+0000 7f8d17e00640 1 -- 172.21.15.90:0/1569829333 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d28004020 con 0x5556631c57b0 2024-04-02T14:38:28.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.449+0000 7f8d17e00640 1 -- 172.21.15.90:0/1569829333 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f8d28017030 con 0x5556631c57b0 2024-04-02T14:38:28.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.449+0000 7f8d3ff96740 1 -- 172.21.15.90:0/1569829333 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8d30032df0 con 0x5556631c57b0 2024-04-02T14:38:28.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.449+0000 7f8d17e00640 1 -- 172.21.15.90:0/1569829333 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f8d28037050 con 0x5556631c57b0 2024-04-02T14:38:28.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.449+0000 7f8d3ff96740 10 client.26365 fetch_fsmap learned FSMap version 27 2024-04-02T14:38:28.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.449+0000 7f8d3ff96740 1 -- 172.21.15.90:0/1569829333 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x555663245960 con 0x5556631c57b0 2024-04-02T14:38:28.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.450+0000 7f8d17e00640 1 -- 172.21.15.90:0/1569829333 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8d30032df0 con 0x5556631c57b0 2024-04-02T14:38:28.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.450+0000 7f8d17e00640 1 -- 172.21.15.90:0/1569829333 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f8d2801cbd0 con 0x5556631c57b0 2024-04-02T14:38:28.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.450+0000 7f8d3ff96740 10 client.26365 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:38:28.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.450+0000 7f8d3ff96740 10 client.26365 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:38:28.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.450+0000 7f8d3ff96740 20 client.26365 populate_metadata read hostname 'smithi090' 2024-04-02T14:38:28.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.450+0000 7f8d3ff96740 1 --2- 172.21.15.90:0/1569829333 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5556632460f0 0x555663248510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:28.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.450+0000 7f8d3ff96740 4 client.26365 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-02T14:38:28.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.450+0000 7f8d3ff96740 1 -- 172.21.15.90:0/1569829333 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "scrub status"}) -- 0x555663136cb0 con 0x5556632460f0 2024-04-02T14:38:28.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.451+0000 7f8d35400640 1 --2- 172.21.15.90:0/1569829333 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5556632460f0 0x555663248510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:28.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.452+0000 7f8d17e00640 10 client.26365 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:38:28.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.452+0000 7f8d35400640 1 --2- 172.21.15.90:0/1569829333 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5556632460f0 0x555663248510 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a896c948e4994439 server_cookie=a80353c444d63c2f in_seq=0 out_seq=0 2024-04-02T14:38:28.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.452+0000 7f8d17e00640 1 -- 172.21.15.90:0/1569829333 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x555663136cb0 con 0x5556632460f0 2024-04-02T14:38:28.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.452+0000 7f8d17e00640 10 client.26365 handle_command_reply: tid=0 2024-04-02T14:38:28.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.452+0000 7f8d3ff96740 1 client.26365 shutdown 2024-04-02T14:38:28.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.453+0000 7f8d3ff96740 1 -- 172.21.15.90:0/1569829333 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5556631c57b0 msgr2=0x5556631d2fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:38:28.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.453+0000 7f8d3ff96740 1 --2- 172.21.15.90:0/1569829333 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5556631c57b0 0x5556631d2fd0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f8d28005b60 tx=0x7f8d280046c0 comp rx=0 tx=0).stop 2024-04-02T14:38:28.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.453+0000 7f8d3ff96740 1 -- 172.21.15.90:0/1569829333 shutdown_connections 2024-04-02T14:38:28.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.453+0000 7f8d3ff96740 1 --2- 172.21.15.90:0/1569829333 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5556632460f0 0x555663248510 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:28.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.453+0000 7f8d3ff96740 1 --2- 172.21.15.90:0/1569829333 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5556631e6930 0x5556632441a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:28.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.453+0000 7f8d3ff96740 1 --2- 172.21.15.90:0/1569829333 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5556631cecb0 0x5556631d3510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:28.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.453+0000 7f8d3ff96740 1 --2- 172.21.15.90:0/1569829333 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5556631c57b0 0x5556631d2fd0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:28.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.453+0000 7f8d3ff96740 1 -- 172.21.15.90:0/1569829333 >> 172.21.15.90:0/1569829333 conn(0x5556631bfee0 msgr2=0x5556631e5920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:38:28.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.453+0000 7f8d17e00640 0 client.26365 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:38:28.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.453+0000 7f8d3ff96740 1 -- 172.21.15.90:0/1569829333 shutdown_connections 2024-04-02T14:38:28.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.453+0000 7f8d3ff96740 1 -- 172.21.15.90:0/1569829333 wait complete. 2024-04-02T14:38:28.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.454+0000 7f8d3ff96740 20 client.26365 trim_cache size 0 max 16384 2024-04-02T14:38:28.456 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-02T14:38:28.456 INFO:teuthology.orchestra.run.smithi090.stdout: "status": "no active scrubs running", 2024-04-02T14:38:28.456 INFO:teuthology.orchestra.run.smithi090.stdout: "scrubs": {} 2024-04-02T14:38:28.456 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-02T14:38:28.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.455+0000 7f8d3d200640 1 -- 172.21.15.90:0/3964979143 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f8d1c0a3ef0 msgr2=0x7f8d1c0a63b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:38:28.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.455+0000 7f8d3d200640 1 --2- 172.21.15.90:0/3964979143 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f8d1c0a3ef0 0x7f8d1c0a63b0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x5556631c5280 tx=0x7f8d2800b390 comp rx=0 tx=0).stop 2024-04-02T14:38:28.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.455+0000 7f8d3d200640 1 -- 172.21.15.90:0/3964979143 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8d381045c0 msgr2=0x7f8d3810ddc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:38:28.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.455+0000 7f8d3d200640 1 --2- 172.21.15.90:0/3964979143 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8d381045c0 0x7f8d3810ddc0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f8d2400b4a0 tx=0x7f8d2400b960 comp rx=0 tx=0).stop 2024-04-02T14:38:28.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.456+0000 7f8d3d200640 1 -- 172.21.15.90:0/3964979143 shutdown_connections 2024-04-02T14:38:28.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.456+0000 7f8d3d200640 1 --2- 172.21.15.90:0/3964979143 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f8d1c0a3ef0 0x7f8d1c0a63b0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:28.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.456+0000 7f8d3d200640 1 --2- 172.21.15.90:0/3964979143 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8d38106a50 0x7f8d3810e300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:28.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.456+0000 7f8d3d200640 1 --2- 172.21.15.90:0/3964979143 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8d381045c0 0x7f8d3810ddc0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:28.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.456+0000 7f8d3d200640 1 --2- 172.21.15.90:0/3964979143 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8d38069240 0x7f8d38114cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:28.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.456+0000 7f8d3d200640 1 -- 172.21.15.90:0/3964979143 >> 172.21.15.90:0/3964979143 conn(0x7f8d38077b00 msgr2=0x7f8d38101a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:38:28.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.456+0000 7f8d3d200640 1 -- 172.21.15.90:0/3964979143 shutdown_connections 2024-04-02T14:38:28.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.456+0000 7f8d3d200640 1 -- 172.21.15.90:0/3964979143 wait complete. 2024-04-02T14:38:28.482 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-02T14:38:28.483 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T14:38:28.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.770+0000 7f8667e00640 1 -- 172.21.15.90:0/1675571121 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f865809ad20 msgr2=0x7f8658004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:38:28.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.770+0000 7f8667e00640 1 --2- 172.21.15.90:0/1675571121 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f865809ad20 0x7f8658004c80 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f8654009900 tx=0x7f865402bc30 comp rx=0 tx=0).stop 2024-04-02T14:38:28.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.771+0000 7f8667e00640 1 -- 172.21.15.90:0/1675571121 shutdown_connections 2024-04-02T14:38:28.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.771+0000 7f8667e00640 1 --2- 172.21.15.90:0/1675571121 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8658098710 0x7f8658098b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:28.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.771+0000 7f8667e00640 1 --2- 172.21.15.90:0/1675571121 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f86580051c0 0x7f8658005640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:28.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.771+0000 7f8667e00640 1 --2- 172.21.15.90:0/1675571121 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f865809ad20 0x7f8658004c80 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:28.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.771+0000 7f8667e00640 1 -- 172.21.15.90:0/1675571121 >> 172.21.15.90:0/1675571121 conn(0x7f865800a770 msgr2=0x7f865800ab80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:38:28.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.772+0000 7f8667e00640 1 -- 172.21.15.90:0/1675571121 shutdown_connections 2024-04-02T14:38:28.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.772+0000 7f8667e00640 1 -- 172.21.15.90:0/1675571121 wait complete. 2024-04-02T14:38:28.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.773+0000 7f8667e00640 1 Processor -- start 2024-04-02T14:38:28.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.774+0000 7f8667e00640 1 -- start start 2024-04-02T14:38:28.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.774+0000 7f8667e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f86580051c0 0x7f8658149720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:28.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.774+0000 7f8667e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8658098710 0x7f8658149c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:28.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.774+0000 7f8667e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f865809ad20 0x7f8658143920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:28.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.774+0000 7f8667e00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f86580ac630 con 0x7f8658098710 2024-04-02T14:38:28.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.774+0000 7f8667e00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f86580ac4b0 con 0x7f865809ad20 2024-04-02T14:38:28.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.774+0000 7f8667e00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f86580ac7b0 con 0x7f86580051c0 2024-04-02T14:38:28.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.775+0000 7f8665400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f86580051c0 0x7f8658149720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:28.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.775+0000 7f8665400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f86580051c0 0x7f8658149720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:37560/0 (socket says 172.21.15.90:37560) 2024-04-02T14:38:28.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.775+0000 7f8665400640 1 -- 172.21.15.90:0/355244251 learned_addr learned my addr 172.21.15.90:0/355244251 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:38:28.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.775+0000 7f8664a00640 1 --2- 172.21.15.90:0/355244251 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8658098710 0x7f8658149c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:28.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.776+0000 7f8664a00640 1 -- 172.21.15.90:0/355244251 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f86580051c0 msgr2=0x7f8658149720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:38:28.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.776+0000 7f8664a00640 1 --2- 172.21.15.90:0/355244251 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f86580051c0 0x7f8658149720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:28.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.776+0000 7f8664a00640 1 -- 172.21.15.90:0/355244251 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f865809ad20 msgr2=0x7f8658143920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:38:28.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.776+0000 7f8664a00640 1 --2- 172.21.15.90:0/355244251 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f865809ad20 0x7f8658143920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:28.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.776+0000 7f8664a00640 1 -- 172.21.15.90:0/355244251 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86581440a0 con 0x7f8658098710 2024-04-02T14:38:28.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.779+0000 7f8664a00640 1 --2- 172.21.15.90:0/355244251 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8658098710 0x7f8658149c60 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f864c00e9f0 tx=0x7f864c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:38:28.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.781+0000 7f865e000640 1 -- 172.21.15.90:0/355244251 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f864c00cd80 con 0x7f8658098710 2024-04-02T14:38:28.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.781+0000 7f865e000640 1 -- 172.21.15.90:0/355244251 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f864c0035f0 con 0x7f8658098710 2024-04-02T14:38:28.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.781+0000 7f865e000640 1 -- 172.21.15.90:0/355244251 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f864c010690 con 0x7f8658098710 2024-04-02T14:38:28.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.781+0000 7f8667e00640 1 -- 172.21.15.90:0/355244251 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8658144390 con 0x7f8658098710 2024-04-02T14:38:28.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.781+0000 7f8667e00640 1 -- 172.21.15.90:0/355244251 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86581505e0 con 0x7f8658098710 2024-04-02T14:38:28.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.785+0000 7f8667e00640 1 -- 172.21.15.90:0/355244251 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8658004880 con 0x7f8658098710 2024-04-02T14:38:28.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.797+0000 7f865e000640 1 -- 172.21.15.90:0/355244251 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f864c003b30 con 0x7f8658098710 2024-04-02T14:38:28.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.804+0000 7f865e000640 1 --2- 172.21.15.90:0/355244251 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f86480a3de0 0x7f86480a62a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:28.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.804+0000 7f865e000640 1 -- 172.21.15.90:0/355244251 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f864c0d0b20 con 0x7f8658098710 2024-04-02T14:38:28.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.805+0000 7f8665400640 1 --2- 172.21.15.90:0/355244251 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f86480a3de0 0x7f86480a62a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:28.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.807+0000 7f8665400640 1 --2- 172.21.15.90:0/355244251 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f86480a3de0 0x7f86480a62a0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f86540096c0 tx=0x7f865402df90 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:38:28.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:28.812+0000 7f865e000640 1 -- 172.21.15.90:0/355244251 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7f864c09d260 con 0x7f8658098710 2024-04-02T14:38:29.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.241+0000 7f8667e00640 1 -- 172.21.15.90:0/355244251 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8658144e00 con 0x7f8658098710 2024-04-02T14:38:29.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.242+0000 7f865e000640 1 -- 172.21.15.90:0/355244251 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11927 (secure 0 0 0) 0x7f864c09cb60 con 0x7f8658098710 2024-04-02T14:38:29.247 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-02T14:38:29.247 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14412,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6837/2852160309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6836","nonce":2852160309},{"type":"v1","addr":"172.21.15.90:6837","nonce":2852160309}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14478,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6841/1919376858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6840","nonce":1919376858},{"type":"v1","addr":"172.21.15.90:6841","nonce":1919376858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-02T14:32:22.319083+0000","modified":"2024-04-02T14:34:52.266152+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24590,"mds_1":24560,"mds_2":24547,"mds_3":24539,"mds_4":24523},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24454":{"gid":24454,"name":"l","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6833/1564319431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":1564319431},{"type":"v1","addr":"172.21.15.177:6833","nonce":1564319431}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.92:6833/1570337372","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":1570337372},{"type":"v1","addr":"172.21.15.92:6833","nonce":1570337372}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"j","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.90:6835/1277124891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6834","nonce":1277124891},{"type":"v1","addr":"172.21.15.90:6835","nonce":1277124891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.92:6835/860721164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":860721164},{"type":"v1","addr":"172.21.15.92:6835","nonce":860721164}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24511":{"gid":24511,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6835/3177244988","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":3177244988},{"type":"v1","addr":"172.21.15.177:6835","nonce":3177244988}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"c","rank":4,"incarnation":25,"state":"up:active","state_seq":9,"addr":"172.21.15.177:6837/4223414858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":4223414858},{"type":"v1","addr":"172.21.15.177:6837","nonce":4223414858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24539":{"gid":24539,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":8,"addr":"172.21.15.92:6837/1270635939","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":1270635939},{"type":"v1","addr":"172.21.15.92:6837","nonce":1270635939}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24547":{"gid":24547,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":7,"addr":"172.21.15.177:6839/664895663","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":664895663},{"type":"v1","addr":"172.21.15.177:6839","nonce":664895663}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"g","rank":1,"incarnation":19,"state":"up:active","state_seq":5,"addr":"172.21.15.90:6839/1498915870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6838","nonce":1498915870},{"type":"v1","addr":"172.21.15.90:6839","nonce":1498915870}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24590":{"gid":24590,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.92:6839/1054888219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":1054888219},{"type":"v1","addr":"172.21.15.92:6839","nonce":1054888219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24590,"qdb_cluster":[24547,24590,24523,24539,24560]},"id":1}]} 2024-04-02T14:38:29.249 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 27 2024-04-02T14:38:29.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.246+0000 7f8667e00640 1 -- 172.21.15.90:0/355244251 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f86480a3de0 msgr2=0x7f86480a62a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:38:29.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.246+0000 7f8667e00640 1 --2- 172.21.15.90:0/355244251 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f86480a3de0 0x7f86480a62a0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f86540096c0 tx=0x7f865402df90 comp rx=0 tx=0).stop 2024-04-02T14:38:29.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.246+0000 7f8667e00640 1 -- 172.21.15.90:0/355244251 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8658098710 msgr2=0x7f8658149c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:38:29.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.246+0000 7f8667e00640 1 --2- 172.21.15.90:0/355244251 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8658098710 0x7f8658149c60 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f864c00e9f0 tx=0x7f864c00eeb0 comp rx=0 tx=0).stop 2024-04-02T14:38:29.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.247+0000 7f8667e00640 1 -- 172.21.15.90:0/355244251 shutdown_connections 2024-04-02T14:38:29.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.247+0000 7f8667e00640 1 --2- 172.21.15.90:0/355244251 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f86480a3de0 0x7f86480a62a0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:29.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.247+0000 7f8667e00640 1 --2- 172.21.15.90:0/355244251 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f865809ad20 0x7f8658143920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:29.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.247+0000 7f8667e00640 1 --2- 172.21.15.90:0/355244251 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8658098710 0x7f8658149c60 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:29.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.247+0000 7f8667e00640 1 --2- 172.21.15.90:0/355244251 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f86580051c0 0x7f8658149720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:29.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.247+0000 7f8667e00640 1 -- 172.21.15.90:0/355244251 >> 172.21.15.90:0/355244251 conn(0x7f865800a770 msgr2=0x7f86580976f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:38:29.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.247+0000 7f8667e00640 1 -- 172.21.15.90:0/355244251 shutdown_connections 2024-04-02T14:38:29.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.247+0000 7f8667e00640 1 -- 172.21.15.90:0/355244251 wait complete. 2024-04-02T14:38:29.275 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-04-02T14:38:29.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.547+0000 7f3ac3200640 1 -- 172.21.15.90:0/734393991 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3abc116b00 msgr2=0x7f3abc118f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:38:29.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.547+0000 7f3ac3200640 1 --2- 172.21.15.90:0/734393991 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3abc116b00 0x7f3abc118f10 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f3ab4009ca0 tx=0x7f3ab402c180 comp rx=0 tx=0).stop 2024-04-02T14:38:29.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.547+0000 7f3ac3200640 1 -- 172.21.15.90:0/734393991 shutdown_connections 2024-04-02T14:38:29.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.547+0000 7f3ac3200640 1 --2- 172.21.15.90:0/734393991 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3abc116b00 0x7f3abc118f10 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:29.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.547+0000 7f3ac3200640 1 --2- 172.21.15.90:0/734393991 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3abc109040 0x7f3abc1094a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:29.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.547+0000 7f3ac3200640 1 --2- 172.21.15.90:0/734393991 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3abc10c220 0x7f3abc108b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:29.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.547+0000 7f3ac3200640 1 -- 172.21.15.90:0/734393991 >> 172.21.15.90:0/734393991 conn(0x7f3abc06d040 msgr2=0x7f3abc06d450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:38:29.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.548+0000 7f3ac3200640 1 -- 172.21.15.90:0/734393991 shutdown_connections 2024-04-02T14:38:29.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.548+0000 7f3ac3200640 1 -- 172.21.15.90:0/734393991 wait complete. 2024-04-02T14:38:29.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.549+0000 7f3ac3200640 1 Processor -- start 2024-04-02T14:38:29.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.549+0000 7f3ac3200640 1 -- start start 2024-04-02T14:38:29.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.550+0000 7f3ac3200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3abc109040 0x7f3abc114b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:29.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.550+0000 7f3ac3200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3abc10c220 0x7f3abc1150a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:29.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.550+0000 7f3ac3200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3abc116b00 0x7f3abc1b2fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:29.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.550+0000 7f3ac3200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f3abc10d490 con 0x7f3abc116b00 2024-04-02T14:38:29.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.550+0000 7f3ac3200640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f3abc10d310 con 0x7f3abc10c220 2024-04-02T14:38:29.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.550+0000 7f3ac3200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f3abc10d610 con 0x7f3abc109040 2024-04-02T14:38:29.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.550+0000 7f3abb400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3abc10c220 0x7f3abc1150a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:29.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.550+0000 7f3abb400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3abc10c220 0x7f3abc1150a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:47874/0 (socket says 172.21.15.90:47874) 2024-04-02T14:38:29.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.550+0000 7f3abb400640 1 -- 172.21.15.90:0/879662405 learned_addr learned my addr 172.21.15.90:0/879662405 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:38:29.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.551+0000 7f3abbe00640 1 --2- 172.21.15.90:0/879662405 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3abc109040 0x7f3abc114b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:29.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.551+0000 7f3abb400640 1 -- 172.21.15.90:0/879662405 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3abc109040 msgr2=0x7f3abc114b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:38:29.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.551+0000 7f3abb400640 1 --2- 172.21.15.90:0/879662405 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3abc109040 0x7f3abc114b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:29.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.551+0000 7f3ac1200640 1 --2- 172.21.15.90:0/879662405 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3abc116b00 0x7f3abc1b2fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:29.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.551+0000 7f3abb400640 1 -- 172.21.15.90:0/879662405 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3abc116b00 msgr2=0x7f3abc1b2fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:38:29.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.551+0000 7f3abb400640 1 --2- 172.21.15.90:0/879662405 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3abc116b00 0x7f3abc1b2fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:29.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.551+0000 7f3abb400640 1 -- 172.21.15.90:0/879662405 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3abc1b3520 con 0x7f3abc10c220 2024-04-02T14:38:29.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.551+0000 7f3abb400640 1 --2- 172.21.15.90:0/879662405 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3abc10c220 0x7f3abc1150a0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f3aa800e9c0 tx=0x7f3aa800ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:38:29.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.552+0000 7f3ab8c00640 1 -- 172.21.15.90:0/879662405 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3aa800ccc0 con 0x7f3abc10c220 2024-04-02T14:38:29.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.552+0000 7f3ab8c00640 1 -- 172.21.15.90:0/879662405 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3aa80035f0 con 0x7f3abc10c220 2024-04-02T14:38:29.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.552+0000 7f3ab8c00640 1 -- 172.21.15.90:0/879662405 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3aa800ce60 con 0x7f3abc10c220 2024-04-02T14:38:29.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.552+0000 7f3ac3200640 1 -- 172.21.15.90:0/879662405 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3abc1b37b0 con 0x7f3abc10c220 2024-04-02T14:38:29.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.552+0000 7f3ac3200640 1 -- 172.21.15.90:0/879662405 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3abc1b3c80 con 0x7f3abc10c220 2024-04-02T14:38:29.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.559+0000 7f3ab8c00640 1 -- 172.21.15.90:0/879662405 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f3aa80037a0 con 0x7f3abc10c220 2024-04-02T14:38:29.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.562+0000 7f3ab8c00640 1 --2- 172.21.15.90:0/879662405 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f3aa00a3da0 0x7f3aa00a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:29.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.563+0000 7f3ab8c00640 1 -- 172.21.15.90:0/879662405 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f3aa80d0a40 con 0x7f3abc10c220 2024-04-02T14:38:29.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.563+0000 7f3abbe00640 1 --2- 172.21.15.90:0/879662405 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f3aa00a3da0 0x7f3aa00a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:29.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.563+0000 7f3ac5fc4740 1 Processor -- start 2024-04-02T14:38:29.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.564+0000 7f3ac5fc4740 1 -- start start 2024-04-02T14:38:29.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.564+0000 7f3ac5fc4740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b5344bd9f0 0x55b5344bc740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:29.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.564+0000 7f3ac5fc4740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b5344bcd10 0x55b5344bd190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:29.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.564+0000 7f3ac5fc4740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b5344c6080 0x55b5344c8470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:29.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.564+0000 7f3ac5fc4740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55b5343fd400 con 0x55b5344bd9f0 2024-04-02T14:38:29.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.564+0000 7f3ac5fc4740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55b5343e1f10 con 0x55b5344bcd10 2024-04-02T14:38:29.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.564+0000 7f3ac5fc4740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55b5344216f0 con 0x55b5344c6080 2024-04-02T14:38:29.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.564+0000 7f3abb400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b5344bcd10 0x55b5344bd190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:29.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.564+0000 7f3abb400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b5344bcd10 0x55b5344bd190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:47882/0 (socket says 172.21.15.90:47882) 2024-04-02T14:38:29.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.564+0000 7f3abb400640 1 -- 172.21.15.90:0/4050001270 learned_addr learned my addr 172.21.15.90:0/4050001270 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:38:29.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.565+0000 7f3ac1200640 1 --2- 172.21.15.90:0/4050001270 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b5344c6080 0x55b5344c8470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:29.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.565+0000 7f3abb400640 1 -- 172.21.15.90:0/4050001270 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b5344c6080 msgr2=0x55b5344c8470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:38:29.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.565+0000 7f3abb400640 1 --2- 172.21.15.90:0/4050001270 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b5344c6080 0x55b5344c8470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:29.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.565+0000 7f3abb400640 1 -- 172.21.15.90:0/4050001270 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b5344bd9f0 msgr2=0x55b5344bc740 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-02T14:38:29.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.566+0000 7f3abbe00640 1 --2- 172.21.15.90:0/4050001270 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b5344bd9f0 0x55b5344bc740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:29.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.566+0000 7f3abb400640 1 --2- 172.21.15.90:0/4050001270 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b5344bd9f0 0x55b5344bc740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:29.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.566+0000 7f3abb400640 1 -- 172.21.15.90:0/4050001270 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b5341353f0 con 0x55b5344bcd10 2024-04-02T14:38:29.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.566+0000 7f3abbe00640 1 --2- 172.21.15.90:0/4050001270 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b5344bd9f0 0x55b5344bc740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:38:29.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.566+0000 7f3abb400640 1 --2- 172.21.15.90:0/4050001270 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b5344bcd10 0x55b5344bd190 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f3aa80993c0 tx=0x7f3aa809c410 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7283e8491a003dcf server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:38:29.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.566+0000 7f3abbe00640 1 --2- 172.21.15.90:0/879662405 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f3aa00a3da0 0x7f3aa00a6260 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f3aac00bb60 tx=0x7f3aac00b390 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:38:29.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.566+0000 7f3ab1600640 1 -- 172.21.15.90:0/4050001270 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3aa80b7070 con 0x55b5344bcd10 2024-04-02T14:38:29.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.567+0000 7f3ab1600640 1 -- 172.21.15.90:0/4050001270 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3aa80a0da0 con 0x55b5344bcd10 2024-04-02T14:38:29.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.567+0000 7f3ab1600640 1 -- 172.21.15.90:0/4050001270 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3aa80a51d0 con 0x55b5344bcd10 2024-04-02T14:38:29.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.568+0000 7f3ac5fc4740 1 -- 172.21.15.90:0/4050001270 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b5344bcd10 msgr2=0x55b5344bd190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:38:29.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.568+0000 7f3ac5fc4740 1 --2- 172.21.15.90:0/4050001270 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b5344bcd10 0x55b5344bd190 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f3aa80993c0 tx=0x7f3aa809c410 comp rx=0 tx=0).stop 2024-04-02T14:38:29.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.568+0000 7f3ac5fc4740 1 -- 172.21.15.90:0/4050001270 shutdown_connections 2024-04-02T14:38:29.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.568+0000 7f3ac5fc4740 1 --2- 172.21.15.90:0/4050001270 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b5344c6080 0x55b5344c8470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:29.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.568+0000 7f3ac5fc4740 1 --2- 172.21.15.90:0/4050001270 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b5344bcd10 0x55b5344bd190 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:29.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.568+0000 7f3ac5fc4740 1 --2- 172.21.15.90:0/4050001270 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b5344bd9f0 0x55b5344bc740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:29.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.568+0000 7f3ac5fc4740 1 -- 172.21.15.90:0/4050001270 >> 172.21.15.90:0/4050001270 conn(0x55b534483a50 msgr2=0x55b5344b7760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:38:29.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.570+0000 7f3ac5fc4740 1 -- 172.21.15.90:0/4050001270 shutdown_connections 2024-04-02T14:38:29.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.570+0000 7f3ac5fc4740 1 -- 172.21.15.90:0/4050001270 wait complete. 2024-04-02T14:38:29.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.570+0000 7f3ac5fc4740 1 -- start start 2024-04-02T14:38:29.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.571+0000 7f3ac5fc4740 1 Processor -- start 2024-04-02T14:38:29.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.571+0000 7f3ac5fc4740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b5344bcd10 0x55b5344c3040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:29.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.571+0000 7f3ac5fc4740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b5344bd9f0 0x55b5344c3580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:29.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.571+0000 7f3ac5fc4740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b5344c6080 0x55b5344c3ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:29.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.571+0000 7f3ac5fc4740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55b53441d6a0 con 0x55b5344bcd10 2024-04-02T14:38:29.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.571+0000 7f3ac5fc4740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55b5343c9980 con 0x55b5344bd9f0 2024-04-02T14:38:29.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.571+0000 7f3ac5fc4740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55b5343dccb0 con 0x55b5344c6080 2024-04-02T14:38:29.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.571+0000 7f3ac5fc4740 10 client.0 _ll_register_callbacks cb 0x55b5343dc6a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:38:29.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.572+0000 7f3abb400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b5344bd9f0 0x55b5344c3580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:29.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.572+0000 7f3ac1200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b5344c6080 0x55b5344c3ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:29.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.572+0000 7f3abbe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b5344bcd10 0x55b5344c3040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:29.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.572+0000 7f3ac1200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b5344c6080 0x55b5344c3ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:37590/0 (socket says 172.21.15.90:37590) 2024-04-02T14:38:29.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.572+0000 7f3ac1200640 1 -- 172.21.15.90:0/899706749 learned_addr learned my addr 172.21.15.90:0/899706749 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:38:29.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.572+0000 7f3abb400640 1 -- 172.21.15.90:0/899706749 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b5344c6080 msgr2=0x55b5344c3ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:38:29.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.572+0000 7f3abb400640 1 --2- 172.21.15.90:0/899706749 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b5344c6080 0x55b5344c3ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:29.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.572+0000 7f3abb400640 1 -- 172.21.15.90:0/899706749 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b5344bcd10 msgr2=0x55b5344c3040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:38:29.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.573+0000 7f3abb400640 1 --2- 172.21.15.90:0/899706749 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b5344bcd10 0x55b5344c3040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:29.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.573+0000 7f3abb400640 1 -- 172.21.15.90:0/899706749 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b5341ff7d0 con 0x55b5344bd9f0 2024-04-02T14:38:29.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.573+0000 7f3abbe00640 1 --2- 172.21.15.90:0/899706749 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b5344bcd10 0x55b5344c3040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:38:29.576 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.573+0000 7f3abb400640 1 --2- 172.21.15.90:0/899706749 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b5344bd9f0 0x55b5344c3580 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x55b5344c8b40 tx=0x7f3aa809c480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b8522b64d3f80474 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:38:29.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.573+0000 7f3aa6000640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:38:29.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.574+0000 7f3aa6000640 1 -- 172.21.15.90:0/899706749 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3aa80b7070 con 0x55b5344bd9f0 2024-04-02T14:38:29.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.574+0000 7f3aa6000640 1 -- 172.21.15.90:0/899706749 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f3aa80a53e0 con 0x55b5344bd9f0 2024-04-02T14:38:29.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.574+0000 7f3ac5fc4740 1 -- 172.21.15.90:0/899706749 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3aa80a0da0 con 0x55b5344bd9f0 2024-04-02T14:38:29.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.574+0000 7f3aa6000640 1 -- 172.21.15.90:0/899706749 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f3aa80ce590 con 0x55b5344bd9f0 2024-04-02T14:38:29.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.574+0000 7f3ac5fc4740 10 client.26474 fetch_fsmap learned FSMap version 27 2024-04-02T14:38:29.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.574+0000 7f3ac5fc4740 1 -- 172.21.15.90:0/899706749 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55b5344d4e60 con 0x55b5344bd9f0 2024-04-02T14:38:29.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.574+0000 7f3aa6000640 1 -- 172.21.15.90:0/899706749 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3aa80a0da0 con 0x55b5344bd9f0 2024-04-02T14:38:29.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.574+0000 7f3aa6000640 1 -- 172.21.15.90:0/899706749 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f3aa8082050 con 0x55b5344bd9f0 2024-04-02T14:38:29.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.575+0000 7f3ac5fc4740 10 client.26474 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:38:29.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.575+0000 7f3ac5fc4740 10 client.26474 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.c 2024-04-02T14:38:29.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.575+0000 7f3ac5fc4740 20 client.26474 populate_metadata read hostname 'smithi090' 2024-04-02T14:38:29.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.575+0000 7f3ac5fc4740 1 --2- 172.21.15.90:0/899706749 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x55b5344d61b0 0x55b5344d85a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:29.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.575+0000 7f3ac5fc4740 4 client.26474 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:38:29.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.575+0000 7f3ac5fc4740 1 -- 172.21.15.90:0/899706749 --> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b5343e1040 con 0x55b5344d61b0 2024-04-02T14:38:29.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.575+0000 7f3abbe00640 1 --2- 172.21.15.90:0/899706749 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x55b5344d61b0 0x55b5344d85a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:29.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.576+0000 7f3abbe00640 1 --2- 172.21.15.90:0/899706749 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x55b5344d61b0 0x55b5344d85a0 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=5a7862e6c16a2f2c server_cookie=6eedf04cd2c0974a in_seq=0 out_seq=0 2024-04-02T14:38:29.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.576+0000 7f3aa6000640 10 client.26474 ms_handle_connect on v2:172.21.15.177:6836/4223414858 2024-04-02T14:38:29.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.577+0000 7f3aa6000640 1 -- 172.21.15.90:0/899706749 <== mds.4 v2:172.21.15.177:6836/4223414858 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b5343e1040 con 0x55b5344d61b0 2024-04-02T14:38:29.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.577+0000 7f3aa6000640 10 client.26474 handle_command_reply: tid=0 2024-04-02T14:38:29.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.577+0000 7f3ac5fc4740 1 client.26474 shutdown 2024-04-02T14:38:29.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.578+0000 7f3ac5fc4740 1 -- 172.21.15.90:0/899706749 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b5344bd9f0 msgr2=0x55b5344c3580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:38:29.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.578+0000 7f3ac5fc4740 1 --2- 172.21.15.90:0/899706749 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b5344bd9f0 0x55b5344c3580 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x55b5344c8b40 tx=0x7f3aa809c480 comp rx=0 tx=0).stop 2024-04-02T14:38:29.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.578+0000 7f3ac5fc4740 1 -- 172.21.15.90:0/899706749 shutdown_connections 2024-04-02T14:38:29.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.578+0000 7f3ac5fc4740 1 --2- 172.21.15.90:0/899706749 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b5344c6080 0x55b5344c3ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:29.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.578+0000 7f3ac5fc4740 1 --2- 172.21.15.90:0/899706749 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b5344bd9f0 0x55b5344c3580 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:29.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.578+0000 7f3ac5fc4740 1 --2- 172.21.15.90:0/899706749 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x55b5344d61b0 0x55b5344d85a0 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:29.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.578+0000 7f3ac5fc4740 1 --2- 172.21.15.90:0/899706749 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b5344bcd10 0x55b5344c3040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:29.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.578+0000 7f3aa6000640 0 client.26474 ms_handle_reset on v2:172.21.15.177:6836/4223414858 2024-04-02T14:38:29.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.578+0000 7f3ac5fc4740 1 -- 172.21.15.90:0/899706749 >> 172.21.15.90:0/899706749 conn(0x55b534483a50 msgr2=0x55b5344b94b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:38:29.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.579+0000 7f3ac5fc4740 1 -- 172.21.15.90:0/899706749 shutdown_connections 2024-04-02T14:38:29.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.579+0000 7f3ac5fc4740 1 -- 172.21.15.90:0/899706749 wait complete. 2024-04-02T14:38:29.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.579+0000 7f3ac5fc4740 20 client.26474 trim_cache size 0 max 16384 2024-04-02T14:38:29.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.607+0000 7f3ac5fc4740 1 Processor -- start 2024-04-02T14:38:29.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.607+0000 7f3ac5fc4740 1 -- start start 2024-04-02T14:38:29.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.608+0000 7f3ac5fc4740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b5344bcd10 0x55b5344c5770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:29.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.608+0000 7f3ac5fc4740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b5344bd9f0 0x55b5344c7470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:29.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.608+0000 7f3ac5fc4740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b5344c6080 0x55b5344c79b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:29.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.608+0000 7f3ac5fc4740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f3abc1bc730 con 0x55b5344bd9f0 2024-04-02T14:38:29.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.608+0000 7f3ac5fc4740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f3abc1be270 con 0x55b5344bcd10 2024-04-02T14:38:29.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.608+0000 7f3ac5fc4740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f3abc1bce00 con 0x55b5344c6080 2024-04-02T14:38:29.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.608+0000 7f3ac1200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b5344c6080 0x55b5344c79b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:29.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.608+0000 7f3ac1200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b5344c6080 0x55b5344c79b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:37600/0 (socket says 172.21.15.90:37600) 2024-04-02T14:38:29.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.608+0000 7f3ac1200640 1 -- 172.21.15.90:0/437196688 learned_addr learned my addr 172.21.15.90:0/437196688 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:38:29.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.609+0000 7f3abbe00640 1 --2- 172.21.15.90:0/437196688 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b5344bcd10 0x55b5344c5770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:29.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.609+0000 7f3abb400640 1 --2- 172.21.15.90:0/437196688 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b5344bd9f0 0x55b5344c7470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:29.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.609+0000 7f3ac1200640 1 -- 172.21.15.90:0/437196688 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b5344bcd10 msgr2=0x55b5344c5770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:38:29.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.609+0000 7f3ac1200640 1 --2- 172.21.15.90:0/437196688 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b5344bcd10 0x55b5344c5770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:29.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.609+0000 7f3ac1200640 1 -- 172.21.15.90:0/437196688 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b5344bd9f0 msgr2=0x55b5344c7470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:38:29.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.610+0000 7f3ac1200640 1 --2- 172.21.15.90:0/437196688 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b5344bd9f0 0x55b5344c7470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:29.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.610+0000 7f3ac1200640 1 -- 172.21.15.90:0/437196688 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b5344bbd40 con 0x55b5344c6080 2024-04-02T14:38:29.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.610+0000 7f3abb400640 1 --2- 172.21.15.90:0/437196688 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b5344bd9f0 0x55b5344c7470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:38:29.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.610+0000 7f3ac1200640 1 --2- 172.21.15.90:0/437196688 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b5344c6080 0x55b5344c79b0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f3ab4005e30 tx=0x7f3ab4009940 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f01272b5d60c1300 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:38:29.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.611+0000 7f3ab0c00640 1 -- 172.21.15.90:0/437196688 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ab4039040 con 0x55b5344c6080 2024-04-02T14:38:29.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.611+0000 7f3ab0c00640 1 -- 172.21.15.90:0/437196688 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3ab403a040 con 0x55b5344c6080 2024-04-02T14:38:29.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.612+0000 7f3ac5fc4740 1 -- 172.21.15.90:0/437196688 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b5344c6080 msgr2=0x55b5344c79b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:38:29.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.612+0000 7f3ac5fc4740 1 --2- 172.21.15.90:0/437196688 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b5344c6080 0x55b5344c79b0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f3ab4005e30 tx=0x7f3ab4009940 comp rx=0 tx=0).stop 2024-04-02T14:38:29.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.612+0000 7f3ac5fc4740 1 -- 172.21.15.90:0/437196688 shutdown_connections 2024-04-02T14:38:29.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.612+0000 7f3ac5fc4740 1 --2- 172.21.15.90:0/437196688 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b5344c6080 0x55b5344c79b0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:29.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.612+0000 7f3ac5fc4740 1 --2- 172.21.15.90:0/437196688 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b5344bd9f0 0x55b5344c7470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:29.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.612+0000 7f3ac5fc4740 1 --2- 172.21.15.90:0/437196688 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b5344bcd10 0x55b5344c5770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:29.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.612+0000 7f3ac5fc4740 1 -- 172.21.15.90:0/437196688 >> 172.21.15.90:0/437196688 conn(0x55b534483a50 msgr2=0x55b5344b7290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:38:29.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.613+0000 7f3ac5fc4740 1 -- 172.21.15.90:0/437196688 shutdown_connections 2024-04-02T14:38:29.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.613+0000 7f3ac5fc4740 1 -- 172.21.15.90:0/437196688 wait complete. 2024-04-02T14:38:29.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.613+0000 7f3ac5fc4740 1 -- start start 2024-04-02T14:38:29.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.614+0000 7f3ac5fc4740 1 Processor -- start 2024-04-02T14:38:29.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.615+0000 7f3ac5fc4740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b5344bcd10 0x55b5344d70c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:29.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.615+0000 7f3ac5fc4740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b5344bd9f0 0x55b5344d7600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:29.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.615+0000 7f3ac5fc4740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b5344c6080 0x55b5344d7b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:29.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.615+0000 7f3ac5fc4740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f3abc1bdba0 con 0x55b5344bd9f0 2024-04-02T14:38:29.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.615+0000 7f3ac5fc4740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f3abc1bd4d0 con 0x55b5344c6080 2024-04-02T14:38:29.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.615+0000 7f3ac5fc4740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f3abc1ba710 con 0x55b5344bcd10 2024-04-02T14:38:29.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.615+0000 7f3ac5fc4740 10 client.0 _ll_register_callbacks cb 0x7f3ab4037070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:38:29.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.616+0000 7f3abbe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b5344bcd10 0x55b5344d70c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:29.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.616+0000 7f3abbe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b5344bcd10 0x55b5344d70c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:37610/0 (socket says 172.21.15.90:37610) 2024-04-02T14:38:29.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.616+0000 7f3abbe00640 1 -- 172.21.15.90:0/3169626231 learned_addr learned my addr 172.21.15.90:0/3169626231 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:38:29.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.616+0000 7f3ac1200640 1 --2- 172.21.15.90:0/3169626231 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b5344c6080 0x55b5344d7b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:29.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.616+0000 7f3abbe00640 1 -- 172.21.15.90:0/3169626231 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b5344c6080 msgr2=0x55b5344d7b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:38:29.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.616+0000 7f3abbe00640 1 --2- 172.21.15.90:0/3169626231 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b5344c6080 0x55b5344d7b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:29.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.617+0000 7f3abbe00640 1 -- 172.21.15.90:0/3169626231 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b5344bd9f0 msgr2=0x55b5344d7600 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-02T14:38:29.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.617+0000 7f3abb400640 1 --2- 172.21.15.90:0/3169626231 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b5344bd9f0 0x55b5344d7600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:29.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.617+0000 7f3abbe00640 1 --2- 172.21.15.90:0/3169626231 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b5344bd9f0 0x55b5344d7600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:29.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.617+0000 7f3abbe00640 1 -- 172.21.15.90:0/3169626231 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b53454c410 con 0x55b5344bcd10 2024-04-02T14:38:29.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.617+0000 7f3ac1200640 1 --2- 172.21.15.90:0/3169626231 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b5344c6080 0x55b5344d7b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:38:29.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.618+0000 7f3abbe00640 1 --2- 172.21.15.90:0/3169626231 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b5344bcd10 0x55b5344d70c0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x55b5343dccb0 tx=0x7f3aac013070 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6ded20bd9e4e2847 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:38:29.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.618+0000 7f3aa6000640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:38:29.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.618+0000 7f3aa6000640 1 -- 172.21.15.90:0/3169626231 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3aac04c070 con 0x55b5344bcd10 2024-04-02T14:38:29.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.618+0000 7f3ac5fc4740 1 -- 172.21.15.90:0/3169626231 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3ab403a040 con 0x55b5344bcd10 2024-04-02T14:38:29.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.618+0000 7f3aa6000640 1 -- 172.21.15.90:0/3169626231 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f3aac048d70 con 0x55b5344bcd10 2024-04-02T14:38:29.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.619+0000 7f3aa6000640 1 -- 172.21.15.90:0/3169626231 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f3aac057050 con 0x55b5344bcd10 2024-04-02T14:38:29.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.619+0000 7f3ac5fc4740 10 client.26389 fetch_fsmap learned FSMap version 27 2024-04-02T14:38:29.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.619+0000 7f3ac5fc4740 1 -- 172.21.15.90:0/3169626231 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55b5345343d0 con 0x55b5344bcd10 2024-04-02T14:38:29.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.620+0000 7f3aa6000640 1 -- 172.21.15.90:0/3169626231 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3ab403a040 con 0x55b5344bcd10 2024-04-02T14:38:29.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.622+0000 7f3aa6000640 1 -- 172.21.15.90:0/3169626231 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f3aac026240 con 0x55b5344bcd10 2024-04-02T14:38:29.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.622+0000 7f3ac5fc4740 10 client.26389 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:38:29.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.622+0000 7f3ac5fc4740 10 client.26389 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.c 2024-04-02T14:38:29.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.622+0000 7f3ac5fc4740 20 client.26389 populate_metadata read hostname 'smithi090' 2024-04-02T14:38:29.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.622+0000 7f3ac5fc4740 1 --2- 172.21.15.90:0/3169626231 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x55b534534aa0 0x55b534536e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:29.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.622+0000 7f3ac5fc4740 4 client.26389 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-02T14:38:29.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.622+0000 7f3ac5fc4740 1 -- 172.21.15.90:0/3169626231 --> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55b53442ec30 con 0x55b534534aa0 2024-04-02T14:38:29.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.624+0000 7f3abb400640 1 --2- 172.21.15.90:0/3169626231 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x55b534534aa0 0x55b534536e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:29.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.626+0000 7f3abb400640 1 --2- 172.21.15.90:0/3169626231 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x55b534534aa0 0x55b534536e90 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=662676eeb5fc1d77 server_cookie=93359b6b7c09521e in_seq=0 out_seq=0 2024-04-02T14:38:29.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.626+0000 7f3aa6000640 10 client.26389 ms_handle_connect on v2:172.21.15.177:6836/4223414858 2024-04-02T14:38:29.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.628+0000 7f3aa6000640 1 -- 172.21.15.90:0/3169626231 <== mds.4 v2:172.21.15.177:6836/4223414858 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55b53442ec30 con 0x55b534534aa0 2024-04-02T14:38:29.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.628+0000 7f3aa6000640 10 client.26389 handle_command_reply: tid=0 2024-04-02T14:38:29.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.628+0000 7f3ac5fc4740 1 client.26389 shutdown 2024-04-02T14:38:29.630 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.629+0000 7f3ac5fc4740 1 -- 172.21.15.90:0/3169626231 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b5344bcd10 msgr2=0x55b5344d70c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:38:29.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.629+0000 7f3ac5fc4740 1 --2- 172.21.15.90:0/3169626231 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b5344bcd10 0x55b5344d70c0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x55b5343dccb0 tx=0x7f3aac013070 comp rx=0 tx=0).stop 2024-04-02T14:38:29.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.630+0000 7f3ac5fc4740 1 -- 172.21.15.90:0/3169626231 shutdown_connections 2024-04-02T14:38:29.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.630+0000 7f3ac5fc4740 1 --2- 172.21.15.90:0/3169626231 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b5344c6080 0x55b5344d7b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:29.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.630+0000 7f3ac5fc4740 1 --2- 172.21.15.90:0/3169626231 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x55b534534aa0 0x55b534536e90 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:29.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.630+0000 7f3ac5fc4740 1 --2- 172.21.15.90:0/3169626231 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b5344bd9f0 0x55b5344d7600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:29.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.630+0000 7f3ac5fc4740 1 --2- 172.21.15.90:0/3169626231 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b5344bcd10 0x55b5344d70c0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:29.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.630+0000 7f3aa6000640 0 client.26389 ms_handle_reset on v2:172.21.15.177:6836/4223414858 2024-04-02T14:38:29.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.630+0000 7f3ac5fc4740 1 -- 172.21.15.90:0/3169626231 >> 172.21.15.90:0/3169626231 conn(0x55b534483a50 msgr2=0x55b5344c2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:38:29.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.630+0000 7f3ac5fc4740 1 -- 172.21.15.90:0/3169626231 shutdown_connections 2024-04-02T14:38:29.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.630+0000 7f3ac5fc4740 1 -- 172.21.15.90:0/3169626231 wait complete. 2024-04-02T14:38:29.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.631+0000 7f3ac5fc4740 20 client.26389 trim_cache size 0 max 16384 2024-04-02T14:38:29.633 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-02T14:38:29.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.632+0000 7f3ac3200640 1 -- 172.21.15.90:0/879662405 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f3aa00a3da0 msgr2=0x7f3aa00a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:38:29.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.632+0000 7f3ac3200640 1 --2- 172.21.15.90:0/879662405 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f3aa00a3da0 0x7f3aa00a6260 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f3aac00bb60 tx=0x7f3aac00b390 comp rx=0 tx=0).stop 2024-04-02T14:38:29.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.632+0000 7f3ac3200640 1 -- 172.21.15.90:0/879662405 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3abc10c220 msgr2=0x7f3abc1150a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:38:29.634 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.632+0000 7f3ac3200640 1 --2- 172.21.15.90:0/879662405 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3abc10c220 0x7f3abc1150a0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f3aa800e9c0 tx=0x7f3aa800ee80 comp rx=0 tx=0).stop 2024-04-02T14:38:29.634 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.632+0000 7f3ac3200640 1 -- 172.21.15.90:0/879662405 shutdown_connections 2024-04-02T14:38:29.634 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.632+0000 7f3ac3200640 1 --2- 172.21.15.90:0/879662405 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f3aa00a3da0 0x7f3aa00a6260 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:29.634 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.632+0000 7f3ac3200640 1 --2- 172.21.15.90:0/879662405 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3abc116b00 0x7f3abc1b2fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:29.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.632+0000 7f3ac3200640 1 --2- 172.21.15.90:0/879662405 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3abc10c220 0x7f3abc1150a0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:29.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.632+0000 7f3ac3200640 1 --2- 172.21.15.90:0/879662405 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3abc109040 0x7f3abc114b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:29.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.632+0000 7f3ac3200640 1 -- 172.21.15.90:0/879662405 >> 172.21.15.90:0/879662405 conn(0x7f3abc06d040 msgr2=0x7f3abc116ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:38:29.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.633+0000 7f3ac3200640 1 -- 172.21.15.90:0/879662405 shutdown_connections 2024-04-02T14:38:29.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.633+0000 7f3ac3200640 1 -- 172.21.15.90:0/879662405 wait complete. 2024-04-02T14:38:29.663 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-04-02T14:38:29.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.930+0000 7f914b800640 1 -- 172.21.15.90:0/157385950 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9144108310 msgr2=0x7f91441086f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:38:29.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.930+0000 7f914b800640 1 --2- 172.21.15.90:0/157385950 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9144108310 0x7f91441086f0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f9138009900 tx=0x7f913802bc40 comp rx=0 tx=0).stop 2024-04-02T14:38:29.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.931+0000 7f914b800640 1 -- 172.21.15.90:0/157385950 shutdown_connections 2024-04-02T14:38:29.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.931+0000 7f914b800640 1 --2- 172.21.15.90:0/157385950 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f914410d3e0 0x7f9144114fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:29.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.931+0000 7f914b800640 1 --2- 172.21.15.90:0/157385950 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9144108cc0 0x7f914410cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:29.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.931+0000 7f914b800640 1 --2- 172.21.15.90:0/157385950 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9144108310 0x7f91441086f0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:29.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.931+0000 7f914b800640 1 -- 172.21.15.90:0/157385950 >> 172.21.15.90:0/157385950 conn(0x7f9144076040 msgr2=0x7f9144078460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:38:29.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.931+0000 7f914b800640 1 -- 172.21.15.90:0/157385950 shutdown_connections 2024-04-02T14:38:29.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.931+0000 7f914b800640 1 -- 172.21.15.90:0/157385950 wait complete. 2024-04-02T14:38:29.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.932+0000 7f914b800640 1 Processor -- start 2024-04-02T14:38:29.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.933+0000 7f914b800640 1 -- start start 2024-04-02T14:38:29.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.933+0000 7f914b800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9144108310 0x7f91441a4680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:29.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.933+0000 7f914b800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f914410d3e0 0x7f91441a4bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:29.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.934+0000 7f914b800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f914419e7f0 0x7f914419eca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:29.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.934+0000 7f914b800640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f9144118470 con 0x7f9144108310 2024-04-02T14:38:29.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.934+0000 7f914b800640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f91441182f0 con 0x7f914419e7f0 2024-04-02T14:38:29.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.934+0000 7f914b800640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f91441185f0 con 0x7f914410d3e0 2024-04-02T14:38:29.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.934+0000 7f9143e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f914410d3e0 0x7f91441a4bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:29.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.934+0000 7f9143e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f914410d3e0 0x7f91441a4bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:37634/0 (socket says 172.21.15.90:37634) 2024-04-02T14:38:29.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.934+0000 7f9143e00640 1 -- 172.21.15.90:0/1272140051 learned_addr learned my addr 172.21.15.90:0/1272140051 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:38:29.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.934+0000 7f9149800640 1 --2- 172.21.15.90:0/1272140051 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f914419e7f0 0x7f914419eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:29.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.934+0000 7f9148e00640 1 --2- 172.21.15.90:0/1272140051 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9144108310 0x7f91441a4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:29.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.934+0000 7f9143e00640 1 -- 172.21.15.90:0/1272140051 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f914419e7f0 msgr2=0x7f914419eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:38:29.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.934+0000 7f9143e00640 1 --2- 172.21.15.90:0/1272140051 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f914419e7f0 0x7f914419eca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:29.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.934+0000 7f9143e00640 1 -- 172.21.15.90:0/1272140051 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9144108310 msgr2=0x7f91441a4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:38:29.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.934+0000 7f9143e00640 1 --2- 172.21.15.90:0/1272140051 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9144108310 0x7f91441a4680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:29.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.934+0000 7f9143e00640 1 -- 172.21.15.90:0/1272140051 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f914419f380 con 0x7f914410d3e0 2024-04-02T14:38:29.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.934+0000 7f9148e00640 1 --2- 172.21.15.90:0/1272140051 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9144108310 0x7f91441a4680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:38:29.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.935+0000 7f9143e00640 1 --2- 172.21.15.90:0/1272140051 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f914410d3e0 0x7f91441a4bc0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f9130005fd0 tx=0x7f913000dde0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:38:29.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.935+0000 7f9141600640 1 -- 172.21.15.90:0/1272140051 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9130017070 con 0x7f914410d3e0 2024-04-02T14:38:29.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.935+0000 7f9141600640 1 -- 172.21.15.90:0/1272140051 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9130009c30 con 0x7f914410d3e0 2024-04-02T14:38:29.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.935+0000 7f9141600640 1 -- 172.21.15.90:0/1272140051 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9130010460 con 0x7f914410d3e0 2024-04-02T14:38:29.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.935+0000 7f914b800640 1 -- 172.21.15.90:0/1272140051 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f914419f670 con 0x7f914410d3e0 2024-04-02T14:38:29.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.935+0000 7f914b800640 1 -- 172.21.15.90:0/1272140051 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f91441ab800 con 0x7f914410d3e0 2024-04-02T14:38:29.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.940+0000 7f914e573740 1 Processor -- start 2024-04-02T14:38:29.942 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:38:29 smithi090 ceph-mon[31674]: pgmap v111: 97 pgs: 97 active+clean; 2.1 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 37 MiB/s wr, 1.50k op/s 2024-04-02T14:38:29.942 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:38:29 smithi090 ceph-mon[31674]: from='client.? 172.21.15.90:0/355244251' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-02T14:38:29.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.941+0000 7f9141600640 1 -- 172.21.15.90:0/1272140051 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f9130003e70 con 0x7f914410d3e0 2024-04-02T14:38:29.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.941+0000 7f914e573740 1 -- start start 2024-04-02T14:38:29.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.941+0000 7f914e573740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563a3b46f9f0 0x563a3b46e740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:29.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.941+0000 7f914e573740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563a3b46ed10 0x563a3b46f190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:29.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.941+0000 7f914e573740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563a3b478080 0x563a3b47a470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:29.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.941+0000 7f914e573740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x563a3b3af400 con 0x563a3b46ed10 2024-04-02T14:38:29.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.941+0000 7f914e573740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x563a3b393f10 con 0x563a3b478080 2024-04-02T14:38:29.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.942+0000 7f914e573740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x563a3b3d36f0 con 0x563a3b46f9f0 2024-04-02T14:38:29.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.942+0000 7f9149800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563a3b46f9f0 0x563a3b46e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:29.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.942+0000 7f9149800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563a3b46f9f0 0x563a3b46e740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:37646/0 (socket says 172.21.15.90:37646) 2024-04-02T14:38:29.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.942+0000 7f9149800640 1 -- 172.21.15.90:0/3520236176 learned_addr learned my addr 172.21.15.90:0/3520236176 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:38:29.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.942+0000 7f9141600640 1 --2- 172.21.15.90:0/1272140051 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f912c0a3e20 0x7f912c0a62e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:29.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.943+0000 7f9141600640 1 -- 172.21.15.90:0/1272140051 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f91300d0490 con 0x7f914410d3e0 2024-04-02T14:38:29.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.943+0000 7f9149800640 1 -- 172.21.15.90:0/3520236176 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563a3b478080 msgr2=0x563a3b47a470 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-02T14:38:29.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.943+0000 7f9149800640 1 --2- 172.21.15.90:0/3520236176 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563a3b478080 0x563a3b47a470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:29.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.943+0000 7f9149800640 1 -- 172.21.15.90:0/3520236176 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563a3b46ed10 msgr2=0x563a3b46f190 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-02T14:38:29.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.943+0000 7f9148e00640 1 --2- 172.21.15.90:0/3520236176 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563a3b46ed10 0x563a3b46f190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:29.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.943+0000 7f9149800640 1 --2- 172.21.15.90:0/3520236176 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563a3b46ed10 0x563a3b46f190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:29.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.943+0000 7f9149800640 1 -- 172.21.15.90:0/3520236176 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563a3b0e73f0 con 0x563a3b46f9f0 2024-04-02T14:38:29.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.943+0000 7f9148e00640 1 --2- 172.21.15.90:0/3520236176 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563a3b46ed10 0x563a3b46f190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:38:29.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.944+0000 7f9149800640 1 --2- 172.21.15.90:0/1272140051 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f912c0a3e20 0x7f912c0a62e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:29.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.944+0000 7f9149800640 1 --2- 172.21.15.90:0/3520236176 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563a3b46f9f0 0x563a3b46e740 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f913c011590 tx=0x7f913c011a50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=14226a376e4d70d9 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:38:29.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.945+0000 7f9149800640 1 --2- 172.21.15.90:0/1272140051 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f912c0a3e20 0x7f912c0a62e0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f913c011f50 tx=0x7f913c016000 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:38:29.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.945+0000 7f9136000640 1 -- 172.21.15.90:0/3520236176 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f913c021070 con 0x563a3b46f9f0 2024-04-02T14:38:29.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.945+0000 7f9136000640 1 -- 172.21.15.90:0/3520236176 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f913c00edd0 con 0x563a3b46f9f0 2024-04-02T14:38:29.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.945+0000 7f9136000640 1 -- 172.21.15.90:0/3520236176 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f913c017050 con 0x563a3b46f9f0 2024-04-02T14:38:29.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.946+0000 7f914e573740 1 -- 172.21.15.90:0/3520236176 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563a3b46f9f0 msgr2=0x563a3b46e740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:38:29.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.946+0000 7f914e573740 1 --2- 172.21.15.90:0/3520236176 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563a3b46f9f0 0x563a3b46e740 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f913c011590 tx=0x7f913c011a50 comp rx=0 tx=0).stop 2024-04-02T14:38:29.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.946+0000 7f914e573740 1 -- 172.21.15.90:0/3520236176 shutdown_connections 2024-04-02T14:38:29.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.947+0000 7f914e573740 1 --2- 172.21.15.90:0/3520236176 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563a3b478080 0x563a3b47a470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:29.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.947+0000 7f914e573740 1 --2- 172.21.15.90:0/3520236176 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563a3b46ed10 0x563a3b46f190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:29.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.947+0000 7f914e573740 1 --2- 172.21.15.90:0/3520236176 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563a3b46f9f0 0x563a3b46e740 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:29.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.947+0000 7f914e573740 1 -- 172.21.15.90:0/3520236176 >> 172.21.15.90:0/3520236176 conn(0x563a3b435a50 msgr2=0x563a3b469760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:38:29.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.947+0000 7f914e573740 1 -- 172.21.15.90:0/3520236176 shutdown_connections 2024-04-02T14:38:29.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.947+0000 7f914e573740 1 -- 172.21.15.90:0/3520236176 wait complete. 2024-04-02T14:38:29.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.948+0000 7f914e573740 1 -- start start 2024-04-02T14:38:29.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.948+0000 7f914e573740 1 Processor -- start 2024-04-02T14:38:29.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.949+0000 7f914e573740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563a3b46ed10 0x563a3b4750f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:29.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.950+0000 7f914e573740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563a3b46f9f0 0x563a3b475630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:29.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.950+0000 7f914e573740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563a3b478080 0x563a3b475b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:29.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.950+0000 7f914e573740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x563a3b3cf6a0 con 0x563a3b46ed10 2024-04-02T14:38:29.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.950+0000 7f914e573740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x563a3b37b980 con 0x563a3b46f9f0 2024-04-02T14:38:29.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.950+0000 7f914e573740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x563a3b38ecb0 con 0x563a3b478080 2024-04-02T14:38:29.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.950+0000 7f914e573740 10 client.0 _ll_register_callbacks cb 0x563a3b38e6a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:38:29.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.950+0000 7f9148e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563a3b46ed10 0x563a3b4750f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:29.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.951+0000 7f9149800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563a3b478080 0x563a3b475b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:29.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.951+0000 7f9143e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563a3b46f9f0 0x563a3b475630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:29.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.951+0000 7f9143e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563a3b46f9f0 0x563a3b475630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:47942/0 (socket says 172.21.15.90:47942) 2024-04-02T14:38:29.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.951+0000 7f9143e00640 1 -- 172.21.15.90:0/1363677597 learned_addr learned my addr 172.21.15.90:0/1363677597 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:38:29.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.951+0000 7f9143e00640 1 -- 172.21.15.90:0/1363677597 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563a3b478080 msgr2=0x563a3b475b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:38:29.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.951+0000 7f9143e00640 1 --2- 172.21.15.90:0/1363677597 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563a3b478080 0x563a3b475b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:29.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.951+0000 7f9143e00640 1 -- 172.21.15.90:0/1363677597 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563a3b46ed10 msgr2=0x563a3b4750f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:38:29.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.951+0000 7f9143e00640 1 --2- 172.21.15.90:0/1363677597 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563a3b46ed10 0x563a3b4750f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:29.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.951+0000 7f9143e00640 1 -- 172.21.15.90:0/1363677597 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563a3b1b17d0 con 0x563a3b46f9f0 2024-04-02T14:38:29.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.952+0000 7f9149800640 1 --2- 172.21.15.90:0/1363677597 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563a3b478080 0x563a3b475b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:38:29.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.952+0000 7f9148e00640 1 --2- 172.21.15.90:0/1363677597 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563a3b46ed10 0x563a3b4750f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:38:29.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.952+0000 7f9143e00640 1 --2- 172.21.15.90:0/1363677597 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563a3b46f9f0 0x563a3b475630 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f9130098db0 tx=0x7f913009be30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d15a077315d96779 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:38:29.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.952+0000 7f912aa00640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:38:29.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.952+0000 7f912aa00640 1 -- 172.21.15.90:0/1363677597 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91300ce010 con 0x563a3b46f9f0 2024-04-02T14:38:29.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.952+0000 7f912aa00640 1 -- 172.21.15.90:0/1363677597 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f913009b830 con 0x563a3b46f9f0 2024-04-02T14:38:29.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.952+0000 7f914e573740 1 -- 172.21.15.90:0/1363677597 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f913c00edd0 con 0x563a3b46f9f0 2024-04-02T14:38:29.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.953+0000 7f912aa00640 1 -- 172.21.15.90:0/1363677597 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f91300a0790 con 0x563a3b46f9f0 2024-04-02T14:38:29.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.953+0000 7f914e573740 10 client.26489 fetch_fsmap learned FSMap version 27 2024-04-02T14:38:29.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.953+0000 7f914e573740 1 -- 172.21.15.90:0/1363677597 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x563a3b497fb0 con 0x563a3b46f9f0 2024-04-02T14:38:29.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.954+0000 7f912aa00640 1 -- 172.21.15.90:0/1363677597 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f913c00edd0 con 0x563a3b46f9f0 2024-04-02T14:38:29.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.955+0000 7f912aa00640 1 -- 172.21.15.90:0/1363677597 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f91300b5b60 con 0x563a3b46f9f0 2024-04-02T14:38:29.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.955+0000 7f914e573740 10 client.26489 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:38:29.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.955+0000 7f914e573740 10 client.26489 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-04-02T14:38:29.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.955+0000 7f914e573740 20 client.26489 populate_metadata read hostname 'smithi090' 2024-04-02T14:38:29.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.955+0000 7f914e573740 1 --2- 172.21.15.90:0/1363677597 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x563a3b4993c0 0x563a3b49b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:29.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.956+0000 7f914e573740 4 client.26489 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:38:29.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.956+0000 7f914e573740 1 -- 172.21.15.90:0/1363677597 --> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563a3b393040 con 0x563a3b4993c0 2024-04-02T14:38:29.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.956+0000 7f9148e00640 1 --2- 172.21.15.90:0/1363677597 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x563a3b4993c0 0x563a3b49b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:29.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.957+0000 7f9148e00640 1 --2- 172.21.15.90:0/1363677597 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x563a3b4993c0 0x563a3b49b7b0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=d61f6c4af9831ace server_cookie=99b410fc92c087b1 in_seq=0 out_seq=0 2024-04-02T14:38:29.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.957+0000 7f912aa00640 10 client.26489 ms_handle_connect on v2:172.21.15.92:6836/1270635939 2024-04-02T14:38:29.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.958+0000 7f912aa00640 1 -- 172.21.15.90:0/1363677597 <== mds.3 v2:172.21.15.92:6836/1270635939 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x563a3b393040 con 0x563a3b4993c0 2024-04-02T14:38:29.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.958+0000 7f912aa00640 10 client.26489 handle_command_reply: tid=0 2024-04-02T14:38:29.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.958+0000 7f914e573740 1 client.26489 shutdown 2024-04-02T14:38:29.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.959+0000 7f914e573740 1 -- 172.21.15.90:0/1363677597 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563a3b46f9f0 msgr2=0x563a3b475630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:38:29.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.959+0000 7f914e573740 1 --2- 172.21.15.90:0/1363677597 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563a3b46f9f0 0x563a3b475630 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f9130098db0 tx=0x7f913009be30 comp rx=0 tx=0).stop 2024-04-02T14:38:29.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.959+0000 7f914e573740 1 -- 172.21.15.90:0/1363677597 shutdown_connections 2024-04-02T14:38:29.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.959+0000 7f914e573740 1 --2- 172.21.15.90:0/1363677597 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x563a3b4993c0 0x563a3b49b7b0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:29.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.959+0000 7f914e573740 1 --2- 172.21.15.90:0/1363677597 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563a3b478080 0x563a3b475b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:29.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.959+0000 7f914e573740 1 --2- 172.21.15.90:0/1363677597 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563a3b46f9f0 0x563a3b475630 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:29.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.959+0000 7f912aa00640 0 client.26489 ms_handle_reset on v2:172.21.15.92:6836/1270635939 2024-04-02T14:38:29.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.959+0000 7f914e573740 1 --2- 172.21.15.90:0/1363677597 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563a3b46ed10 0x563a3b4750f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:29.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.959+0000 7f914e573740 1 -- 172.21.15.90:0/1363677597 >> 172.21.15.90:0/1363677597 conn(0x563a3b435a50 msgr2=0x563a3b46b4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:38:29.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.960+0000 7f914e573740 1 -- 172.21.15.90:0/1363677597 shutdown_connections 2024-04-02T14:38:29.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.960+0000 7f914e573740 1 -- 172.21.15.90:0/1363677597 wait complete. 2024-04-02T14:38:29.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.960+0000 7f914e573740 20 client.26489 trim_cache size 0 max 16384 2024-04-02T14:38:29.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.982+0000 7f914e573740 1 Processor -- start 2024-04-02T14:38:29.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.982+0000 7f914e573740 1 -- start start 2024-04-02T14:38:29.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.982+0000 7f914e573740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563a3b46ed10 0x563a3b494b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:29.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.982+0000 7f914e573740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563a3b46f9f0 0x563a3b46b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:29.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.982+0000 7f914e573740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563a3b478080 0x563a3b4a10c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:29.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.982+0000 7f914e573740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f9144082660 con 0x563a3b478080 2024-04-02T14:38:29.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.982+0000 7f914e573740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f91441ad170 con 0x563a3b46f9f0 2024-04-02T14:38:29.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.982+0000 7f914e573740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f91441abd00 con 0x563a3b46ed10 2024-04-02T14:38:29.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.983+0000 7f9143e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563a3b46f9f0 0x563a3b46b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:29.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.983+0000 7f9143e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563a3b46f9f0 0x563a3b46b2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:47944/0 (socket says 172.21.15.90:47944) 2024-04-02T14:38:29.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.983+0000 7f9143e00640 1 -- 172.21.15.90:0/1063606679 learned_addr learned my addr 172.21.15.90:0/1063606679 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:38:29.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.983+0000 7f9149800640 1 --2- 172.21.15.90:0/1063606679 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563a3b478080 0x563a3b4a10c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:29.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.983+0000 7f9143e00640 1 -- 172.21.15.90:0/1063606679 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563a3b46ed10 msgr2=0x563a3b494b00 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-02T14:38:29.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.983+0000 7f9143e00640 1 --2- 172.21.15.90:0/1063606679 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563a3b46ed10 0x563a3b494b00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:29.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.983+0000 7f9143e00640 1 -- 172.21.15.90:0/1063606679 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563a3b478080 msgr2=0x563a3b4a10c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:38:29.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.983+0000 7f9143e00640 1 --2- 172.21.15.90:0/1063606679 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563a3b478080 0x563a3b4a10c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:29.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.983+0000 7f9143e00640 1 -- 172.21.15.90:0/1063606679 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563a3b473640 con 0x563a3b46f9f0 2024-04-02T14:38:29.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.983+0000 7f9143e00640 1 --2- 172.21.15.90:0/1063606679 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563a3b46f9f0 0x563a3b46b2c0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f9130000c00 tx=0x7f9130005be0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f358ed67d934a3d0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:38:29.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.983+0000 7f9135600640 1 -- 172.21.15.90:0/1063606679 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9130009de0 con 0x563a3b46f9f0 2024-04-02T14:38:29.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.983+0000 7f9135600640 1 -- 172.21.15.90:0/1063606679 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9130015070 con 0x563a3b46f9f0 2024-04-02T14:38:29.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.984+0000 7f9135600640 1 -- 172.21.15.90:0/1063606679 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91300b5b60 con 0x563a3b46f9f0 2024-04-02T14:38:29.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.984+0000 7f914e573740 1 -- 172.21.15.90:0/1063606679 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563a3b46f9f0 msgr2=0x563a3b46b2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:38:29.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.984+0000 7f914e573740 1 --2- 172.21.15.90:0/1063606679 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563a3b46f9f0 0x563a3b46b2c0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f9130000c00 tx=0x7f9130005be0 comp rx=0 tx=0).stop 2024-04-02T14:38:29.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.985+0000 7f914e573740 1 -- 172.21.15.90:0/1063606679 shutdown_connections 2024-04-02T14:38:29.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.985+0000 7f914e573740 1 --2- 172.21.15.90:0/1063606679 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563a3b478080 0x563a3b4a10c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:29.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.985+0000 7f914e573740 1 --2- 172.21.15.90:0/1063606679 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563a3b46f9f0 0x563a3b46b2c0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:29.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.985+0000 7f914e573740 1 --2- 172.21.15.90:0/1063606679 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563a3b46ed10 0x563a3b494b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:29.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.985+0000 7f914e573740 1 -- 172.21.15.90:0/1063606679 >> 172.21.15.90:0/1063606679 conn(0x563a3b435a50 msgr2=0x563a3b46abb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:38:29.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.985+0000 7f914e573740 1 -- 172.21.15.90:0/1063606679 shutdown_connections 2024-04-02T14:38:29.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.985+0000 7f914e573740 1 -- 172.21.15.90:0/1063606679 wait complete. 2024-04-02T14:38:29.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.986+0000 7f914e573740 1 -- start start 2024-04-02T14:38:29.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.986+0000 7f914e573740 1 Processor -- start 2024-04-02T14:38:29.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.987+0000 7f914e573740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563a3b46ed10 0x563a3b491200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:29.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.987+0000 7f914e573740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563a3b46f9f0 0x563a3b471360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:29.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.987+0000 7f914e573740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563a3b478080 0x563a3b4718a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:29.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.987+0000 7f914e573740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f91441acaa0 con 0x563a3b46ed10 2024-04-02T14:38:29.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.987+0000 7f914e573740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f91441ac3d0 con 0x563a3b46f9f0 2024-04-02T14:38:29.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.987+0000 7f914e573740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f9144080640 con 0x563a3b478080 2024-04-02T14:38:29.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.987+0000 7f914e573740 10 client.0 _ll_register_callbacks cb 0x7f913009bf90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:38:29.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.987+0000 7f9149800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563a3b478080 0x563a3b4718a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:29.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.987+0000 7f9149800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563a3b478080 0x563a3b4718a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:37692/0 (socket says 172.21.15.90:37692) 2024-04-02T14:38:29.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.987+0000 7f9149800640 1 -- 172.21.15.90:0/4291590131 learned_addr learned my addr 172.21.15.90:0/4291590131 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:38:29.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.988+0000 7f9143e00640 1 --2- 172.21.15.90:0/4291590131 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563a3b46f9f0 0x563a3b471360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:29.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.988+0000 7f9148e00640 1 --2- 172.21.15.90:0/4291590131 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563a3b46ed10 0x563a3b491200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:29.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.988+0000 7f9149800640 1 -- 172.21.15.90:0/4291590131 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563a3b46f9f0 msgr2=0x563a3b471360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:38:29.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.988+0000 7f9149800640 1 --2- 172.21.15.90:0/4291590131 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563a3b46f9f0 0x563a3b471360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:29.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.988+0000 7f9149800640 1 -- 172.21.15.90:0/4291590131 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563a3b46ed10 msgr2=0x563a3b491200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:38:29.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.988+0000 7f9149800640 1 --2- 172.21.15.90:0/4291590131 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563a3b46ed10 0x563a3b491200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:29.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.988+0000 7f9149800640 1 -- 172.21.15.90:0/4291590131 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563a3b471de0 con 0x563a3b478080 2024-04-02T14:38:29.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.988+0000 7f9148e00640 1 --2- 172.21.15.90:0/4291590131 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563a3b46ed10 0x563a3b491200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:38:29.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.989+0000 7f9149800640 1 --2- 172.21.15.90:0/4291590131 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563a3b478080 0x563a3b4718a0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f913c009bb0 tx=0x7f913c0054e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6b0f470043c13c0f server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:38:29.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.989+0000 7f912aa00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:38:29.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.989+0000 7f912aa00640 1 -- 172.21.15.90:0/4291590131 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f913c02a040 con 0x563a3b478080 2024-04-02T14:38:29.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.989+0000 7f912aa00640 1 -- 172.21.15.90:0/4291590131 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f913c00eaf0 con 0x563a3b478080 2024-04-02T14:38:29.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.989+0000 7f914e573740 1 -- 172.21.15.90:0/4291590131 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9130015070 con 0x563a3b478080 2024-04-02T14:38:29.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.990+0000 7f912aa00640 1 -- 172.21.15.90:0/4291590131 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f913c021040 con 0x563a3b478080 2024-04-02T14:38:29.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.990+0000 7f914e573740 10 client.26404 fetch_fsmap learned FSMap version 27 2024-04-02T14:38:29.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.990+0000 7f914e573740 1 -- 172.21.15.90:0/4291590131 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x563a3b473080 con 0x563a3b478080 2024-04-02T14:38:29.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.990+0000 7f912aa00640 1 -- 172.21.15.90:0/4291590131 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9130015070 con 0x563a3b478080 2024-04-02T14:38:29.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.990+0000 7f912aa00640 1 -- 172.21.15.90:0/4291590131 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f913c02d070 con 0x563a3b478080 2024-04-02T14:38:29.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.990+0000 7f914e573740 10 client.26404 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:38:29.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.990+0000 7f914e573740 10 client.26404 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-04-02T14:38:29.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.990+0000 7f914e573740 20 client.26404 populate_metadata read hostname 'smithi090' 2024-04-02T14:38:29.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.990+0000 7f914e573740 1 --2- 172.21.15.90:0/4291590131 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x563a3b500d30 0x563a3b503150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:29.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.990+0000 7f914e573740 4 client.26404 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-02T14:38:29.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.990+0000 7f914e573740 1 -- 172.21.15.90:0/4291590131 --> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] -- command(tid 0: {"prefix": "damage ls"}) -- 0x563a3b3e0c30 con 0x563a3b500d30 2024-04-02T14:38:29.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.991+0000 7f9148e00640 1 --2- 172.21.15.90:0/4291590131 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x563a3b500d30 0x563a3b503150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:29.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.991+0000 7f912aa00640 10 client.26404 ms_handle_connect on v2:172.21.15.92:6836/1270635939 2024-04-02T14:38:29.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.991+0000 7f9148e00640 1 --2- 172.21.15.90:0/4291590131 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x563a3b500d30 0x563a3b503150 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=6c63631c3073b184 server_cookie=203204bee6e37ee3 in_seq=0 out_seq=0 2024-04-02T14:38:29.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.992+0000 7f912aa00640 1 -- 172.21.15.90:0/4291590131 <== mds.3 v2:172.21.15.92:6836/1270635939 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x563a3b3e0c30 con 0x563a3b500d30 2024-04-02T14:38:29.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.992+0000 7f912aa00640 10 client.26404 handle_command_reply: tid=0 2024-04-02T14:38:29.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.992+0000 7f914e573740 1 client.26404 shutdown 2024-04-02T14:38:29.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.992+0000 7f914e573740 1 -- 172.21.15.90:0/4291590131 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563a3b478080 msgr2=0x563a3b4718a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:38:29.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.992+0000 7f914e573740 1 --2- 172.21.15.90:0/4291590131 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563a3b478080 0x563a3b4718a0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f913c009bb0 tx=0x7f913c0054e0 comp rx=0 tx=0).stop 2024-04-02T14:38:29.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.992+0000 7f914e573740 1 -- 172.21.15.90:0/4291590131 shutdown_connections 2024-04-02T14:38:29.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.992+0000 7f914e573740 1 --2- 172.21.15.90:0/4291590131 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x563a3b500d30 0x563a3b503150 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:29.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.992+0000 7f914e573740 1 --2- 172.21.15.90:0/4291590131 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563a3b478080 0x563a3b4718a0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:29.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.992+0000 7f914e573740 1 --2- 172.21.15.90:0/4291590131 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563a3b46f9f0 0x563a3b471360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:29.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.992+0000 7f914e573740 1 --2- 172.21.15.90:0/4291590131 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563a3b46ed10 0x563a3b491200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:29.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.992+0000 7f914e573740 1 -- 172.21.15.90:0/4291590131 >> 172.21.15.90:0/4291590131 conn(0x563a3b435a50 msgr2=0x563a3b4688a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:38:29.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.992+0000 7f912aa00640 0 client.26404 ms_handle_reset on v2:172.21.15.92:6836/1270635939 2024-04-02T14:38:29.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.992+0000 7f914e573740 1 -- 172.21.15.90:0/4291590131 shutdown_connections 2024-04-02T14:38:29.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.992+0000 7f914e573740 1 -- 172.21.15.90:0/4291590131 wait complete. 2024-04-02T14:38:29.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.993+0000 7f914e573740 20 client.26404 trim_cache size 0 max 16384 2024-04-02T14:38:29.996 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-02T14:38:29.997 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.993+0000 7f914b800640 1 -- 172.21.15.90:0/1272140051 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f912c0a3e20 msgr2=0x7f912c0a62e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:38:29.997 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.993+0000 7f914b800640 1 --2- 172.21.15.90:0/1272140051 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f912c0a3e20 0x7f912c0a62e0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f913c011f50 tx=0x7f913c016000 comp rx=0 tx=0).stop 2024-04-02T14:38:29.997 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.993+0000 7f914b800640 1 -- 172.21.15.90:0/1272140051 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f914410d3e0 msgr2=0x7f91441a4bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:38:29.997 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.993+0000 7f914b800640 1 --2- 172.21.15.90:0/1272140051 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f914410d3e0 0x7f91441a4bc0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f9130005fd0 tx=0x7f913000dde0 comp rx=0 tx=0).stop 2024-04-02T14:38:29.997 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.993+0000 7f914b800640 1 -- 172.21.15.90:0/1272140051 shutdown_connections 2024-04-02T14:38:29.997 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.993+0000 7f914b800640 1 --2- 172.21.15.90:0/1272140051 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f912c0a3e20 0x7f912c0a62e0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:29.997 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.993+0000 7f914b800640 1 --2- 172.21.15.90:0/1272140051 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f914419e7f0 0x7f914419eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:29.997 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.993+0000 7f914b800640 1 --2- 172.21.15.90:0/1272140051 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f914410d3e0 0x7f91441a4bc0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:29.998 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.994+0000 7f914b800640 1 --2- 172.21.15.90:0/1272140051 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9144108310 0x7f91441a4680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:29.998 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.994+0000 7f914b800640 1 -- 172.21.15.90:0/1272140051 >> 172.21.15.90:0/1272140051 conn(0x7f9144076040 msgr2=0x7f9144077370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:38:29.998 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.994+0000 7f914b800640 1 -- 172.21.15.90:0/1272140051 shutdown_connections 2024-04-02T14:38:29.998 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:29.994+0000 7f914b800640 1 -- 172.21.15.90:0/1272140051 wait complete. 2024-04-02T14:38:30.023 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-02T14:38:30.110 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:38:29 smithi092 ceph-mon[36497]: pgmap v111: 97 pgs: 97 active+clean; 2.1 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 37 MiB/s wr, 1.50k op/s 2024-04-02T14:38:30.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:38:29 smithi092 ceph-mon[36497]: from='client.? 172.21.15.90:0/355244251' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-02T14:38:30.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:38:29 smithi177 ceph-mon[36431]: pgmap v111: 97 pgs: 97 active+clean; 2.1 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 37 MiB/s wr, 1.50k op/s 2024-04-02T14:38:30.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:38:29 smithi177 ceph-mon[36431]: from='client.? 172.21.15.90:0/355244251' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-02T14:38:30.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.305+0000 7fdfdc400640 1 -- 172.21.15.90:0/3961048859 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdfd4108cc0 msgr2=0x7fdfd410cd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:38:30.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.305+0000 7fdfdc400640 1 --2- 172.21.15.90:0/3961048859 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdfd4108cc0 0x7fdfd410cd00 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fdfc4009900 tx=0x7fdfc402bc40 comp rx=0 tx=0).stop 2024-04-02T14:38:30.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.305+0000 7fdfdc400640 1 -- 172.21.15.90:0/3961048859 shutdown_connections 2024-04-02T14:38:30.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.305+0000 7fdfdc400640 1 --2- 172.21.15.90:0/3961048859 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdfd410d3e0 0x7fdfd4114fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:30.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.305+0000 7fdfdc400640 1 --2- 172.21.15.90:0/3961048859 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdfd4108cc0 0x7fdfd410cd00 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:30.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.305+0000 7fdfdc400640 1 --2- 172.21.15.90:0/3961048859 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdfd4108310 0x7fdfd41086f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:30.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.305+0000 7fdfdc400640 1 -- 172.21.15.90:0/3961048859 >> 172.21.15.90:0/3961048859 conn(0x7fdfd4076040 msgr2=0x7fdfd4078460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:38:30.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.305+0000 7fdfdc400640 1 -- 172.21.15.90:0/3961048859 shutdown_connections 2024-04-02T14:38:30.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.305+0000 7fdfdc400640 1 -- 172.21.15.90:0/3961048859 wait complete. 2024-04-02T14:38:30.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.307+0000 7fdfdc400640 1 Processor -- start 2024-04-02T14:38:30.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.307+0000 7fdfdc400640 1 -- start start 2024-04-02T14:38:30.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.307+0000 7fdfdc400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdfd4108310 0x7fdfd40821c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:30.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.307+0000 7fdfdc400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdfd410d3e0 0x7fdfd407b240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:30.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.307+0000 7fdfdc400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdfd407b780 0x7fdfd407bc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:30.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.308+0000 7fdfdc400640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fdfd4118520 con 0x7fdfd410d3e0 2024-04-02T14:38:30.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.308+0000 7fdfdc400640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fdfd41183a0 con 0x7fdfd4108310 2024-04-02T14:38:30.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.308+0000 7fdfdc400640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fdfd41186a0 con 0x7fdfd407b780 2024-04-02T14:38:30.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.308+0000 7fdfd9a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdfd4108310 0x7fdfd40821c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:30.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.308+0000 7fdfd9a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdfd4108310 0x7fdfd40821c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:47962/0 (socket says 172.21.15.90:47962) 2024-04-02T14:38:30.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.308+0000 7fdfd9a00640 1 -- 172.21.15.90:0/50047533 learned_addr learned my addr 172.21.15.90:0/50047533 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:38:30.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.308+0000 7fdfda400640 1 --2- 172.21.15.90:0/50047533 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdfd407b780 0x7fdfd407bc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:30.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.308+0000 7fdfd9000640 1 --2- 172.21.15.90:0/50047533 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdfd410d3e0 0x7fdfd407b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:30.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.308+0000 7fdfd9a00640 1 -- 172.21.15.90:0/50047533 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdfd407b780 msgr2=0x7fdfd407bc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:38:30.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.308+0000 7fdfd9a00640 1 --2- 172.21.15.90:0/50047533 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdfd407b780 0x7fdfd407bc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:30.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.308+0000 7fdfd9a00640 1 -- 172.21.15.90:0/50047533 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdfd410d3e0 msgr2=0x7fdfd407b240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:38:30.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.308+0000 7fdfd9a00640 1 --2- 172.21.15.90:0/50047533 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdfd410d3e0 0x7fdfd407b240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:30.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.309+0000 7fdfd9a00640 1 -- 172.21.15.90:0/50047533 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdfd407fce0 con 0x7fdfd4108310 2024-04-02T14:38:30.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.309+0000 7fdfd9a00640 1 --2- 172.21.15.90:0/50047533 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdfd4108310 0x7fdfd40821c0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fdfc800ea60 tx=0x7fdfc800ef20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:38:30.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.309+0000 7fdfd2000640 1 -- 172.21.15.90:0/50047533 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfc800cde0 con 0x7fdfd4108310 2024-04-02T14:38:30.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.310+0000 7fdfd2000640 1 -- 172.21.15.90:0/50047533 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdfc80035f0 con 0x7fdfd4108310 2024-04-02T14:38:30.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.310+0000 7fdfd2000640 1 -- 172.21.15.90:0/50047533 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfc8010690 con 0x7fdfd4108310 2024-04-02T14:38:30.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.310+0000 7fdfdc400640 1 -- 172.21.15.90:0/50047533 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdfd407ffd0 con 0x7fdfd4108310 2024-04-02T14:38:30.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.310+0000 7fdfdc400640 1 -- 172.21.15.90:0/50047533 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdfd40804d0 con 0x7fdfd4108310 2024-04-02T14:38:30.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.316+0000 7fdfdf1c3740 1 Processor -- start 2024-04-02T14:38:30.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.317+0000 7fdfdf1c3740 1 -- start start 2024-04-02T14:38:30.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.318+0000 7fdfdf1c3740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f67a36b9f0 0x55f67a36a740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:30.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.319+0000 7fdfdf1c3740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f67a36ad10 0x55f67a36b190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:30.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.319+0000 7fdfdf1c3740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f67a374080 0x55f67a376470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:30.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.320+0000 7fdfda400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f67a36b9f0 0x55f67a36a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:30.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.320+0000 7fdfdf1c3740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55f67a2ab400 con 0x55f67a36b9f0 2024-04-02T14:38:30.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.320+0000 7fdfdf1c3740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55f67a28ff10 con 0x55f67a374080 2024-04-02T14:38:30.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.320+0000 7fdfdf1c3740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55f67a2cf6f0 con 0x55f67a36ad10 2024-04-02T14:38:30.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.320+0000 7fdfda400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f67a36b9f0 0x55f67a36a740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:46036/0 (socket says 172.21.15.90:46036) 2024-04-02T14:38:30.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.320+0000 7fdfda400640 1 -- 172.21.15.90:0/334384432 learned_addr learned my addr 172.21.15.90:0/334384432 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:38:30.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.321+0000 7fdfd9000640 1 --2- 172.21.15.90:0/334384432 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f67a374080 0x55f67a376470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:30.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.321+0000 7fdfd2000640 1 -- 172.21.15.90:0/50047533 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fdfc8003fe0 con 0x7fdfd4108310 2024-04-02T14:38:30.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.322+0000 7fdfd9a00640 1 --2- 172.21.15.90:0/334384432 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f67a36ad10 0x55f67a36b190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:30.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.322+0000 7fdfda400640 1 -- 172.21.15.90:0/334384432 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f67a36ad10 msgr2=0x55f67a36b190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:38:30.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.322+0000 7fdfda400640 1 --2- 172.21.15.90:0/334384432 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f67a36ad10 0x55f67a36b190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:30.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.322+0000 7fdfda400640 1 -- 172.21.15.90:0/334384432 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f67a374080 msgr2=0x55f67a376470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:38:30.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.322+0000 7fdfda400640 1 --2- 172.21.15.90:0/334384432 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f67a374080 0x55f67a376470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:30.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.322+0000 7fdfda400640 1 -- 172.21.15.90:0/334384432 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f679fe33f0 con 0x55f67a36b9f0 2024-04-02T14:38:30.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.322+0000 7fdfd9000640 1 --2- 172.21.15.90:0/334384432 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f67a374080 0x55f67a376470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:38:30.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.322+0000 7fdfd9a00640 1 --2- 172.21.15.90:0/334384432 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f67a36ad10 0x55f67a36b190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:38:30.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.323+0000 7fdfda400640 1 --2- 172.21.15.90:0/334384432 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f67a36b9f0 0x55f67a36a740 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fdfcc009f20 tx=0x7fdfcc00c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b47639aea9d86a0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:38:30.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.324+0000 7fdfc2a00640 1 -- 172.21.15.90:0/334384432 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfcc01a070 con 0x55f67a36b9f0 2024-04-02T14:38:30.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.324+0000 7fdfc2a00640 1 -- 172.21.15.90:0/334384432 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdfcc0158f0 con 0x55f67a36b9f0 2024-04-02T14:38:30.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.325+0000 7fdfc2a00640 1 -- 172.21.15.90:0/334384432 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfcc00a900 con 0x55f67a36b9f0 2024-04-02T14:38:30.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.325+0000 7fdfdf1c3740 1 -- 172.21.15.90:0/334384432 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f67a36b9f0 msgr2=0x55f67a36a740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:38:30.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.325+0000 7fdfdf1c3740 1 --2- 172.21.15.90:0/334384432 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f67a36b9f0 0x55f67a36a740 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fdfcc009f20 tx=0x7fdfcc00c8b0 comp rx=0 tx=0).stop 2024-04-02T14:38:30.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.325+0000 7fdfd2000640 1 --2- 172.21.15.90:0/50047533 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fdfbc0a3e20 0x7fdfbc0a62e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:30.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.326+0000 7fdfd2000640 1 -- 172.21.15.90:0/50047533 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7fdfc80d0bb0 con 0x7fdfd4108310 2024-04-02T14:38:30.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.326+0000 7fdfdf1c3740 1 -- 172.21.15.90:0/334384432 shutdown_connections 2024-04-02T14:38:30.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.326+0000 7fdfdf1c3740 1 --2- 172.21.15.90:0/334384432 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f67a374080 0x55f67a376470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:30.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.326+0000 7fdfdf1c3740 1 --2- 172.21.15.90:0/334384432 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f67a36ad10 0x55f67a36b190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:30.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.326+0000 7fdfdf1c3740 1 --2- 172.21.15.90:0/334384432 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f67a36b9f0 0x55f67a36a740 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:30.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.326+0000 7fdfdf1c3740 1 -- 172.21.15.90:0/334384432 >> 172.21.15.90:0/334384432 conn(0x55f67a331a50 msgr2=0x55f67a365760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:38:30.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.326+0000 7fdfd9a00640 1 --2- 172.21.15.90:0/50047533 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fdfbc0a3e20 0x7fdfbc0a62e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:30.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.327+0000 7fdfdf1c3740 1 -- 172.21.15.90:0/334384432 shutdown_connections 2024-04-02T14:38:30.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.327+0000 7fdfdf1c3740 1 -- 172.21.15.90:0/334384432 wait complete. 2024-04-02T14:38:30.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.327+0000 7fdfd9a00640 1 --2- 172.21.15.90:0/50047533 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fdfbc0a3e20 0x7fdfbc0a62e0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x55f67a36bf40 tx=0x7fdfc8087040 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:38:30.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.328+0000 7fdfdf1c3740 1 -- start start 2024-04-02T14:38:30.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.328+0000 7fdfdf1c3740 1 Processor -- start 2024-04-02T14:38:30.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.329+0000 7fdfdf1c3740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f67a36ad10 0x55f67a370f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:30.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.329+0000 7fdfda400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f67a36ad10 0x55f67a370f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:30.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.329+0000 7fdfda400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f67a36ad10 0x55f67a370f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:37728/0 (socket says 172.21.15.90:37728) 2024-04-02T14:38:30.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.329+0000 7fdfdf1c3740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f67a36b9f0 0x55f67a371490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:30.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.329+0000 7fdfda400640 1 -- 172.21.15.90:0/3603295634 learned_addr learned my addr 172.21.15.90:0/3603295634 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:38:30.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.329+0000 7fdfdf1c3740 1 --2- 172.21.15.90:0/3603295634 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f67a374080 0x55f67a3719d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:30.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.329+0000 7fdfdf1c3740 1 -- 172.21.15.90:0/3603295634 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55f67a2cb6a0 con 0x55f67a36b9f0 2024-04-02T14:38:30.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.329+0000 7fdfdf1c3740 1 -- 172.21.15.90:0/3603295634 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55f67a277980 con 0x55f67a374080 2024-04-02T14:38:30.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.329+0000 7fdfdf1c3740 1 -- 172.21.15.90:0/3603295634 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55f67a28acb0 con 0x55f67a36ad10 2024-04-02T14:38:30.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.329+0000 7fdfdf1c3740 10 client.0 _ll_register_callbacks cb 0x55f67a28a6a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:38:30.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.330+0000 7fdfda400640 1 --2- 172.21.15.90:0/3603295634 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f67a374080 0x55f67a3719d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:30.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.330+0000 7fdfda400640 1 -- 172.21.15.90:0/3603295634 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f67a374080 msgr2=0x55f67a3719d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:38:30.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.330+0000 7fdfda400640 1 --2- 172.21.15.90:0/3603295634 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f67a374080 0x55f67a3719d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:30.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.330+0000 7fdfda400640 1 -- 172.21.15.90:0/3603295634 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f67a36b9f0 msgr2=0x55f67a371490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:38:30.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.330+0000 7fdfda400640 1 --2- 172.21.15.90:0/3603295634 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f67a36b9f0 0x55f67a371490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:30.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.330+0000 7fdfda400640 1 -- 172.21.15.90:0/3603295634 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f67a0ad7d0 con 0x55f67a36ad10 2024-04-02T14:38:30.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.331+0000 7fdfda400640 1 --2- 172.21.15.90:0/3603295634 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f67a36ad10 0x55f67a370f50 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fdfcc000f80 tx=0x7fdfcc0049c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a851bb5a7a823181 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:38:30.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.331+0000 7fdfb7400640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:38:30.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.331+0000 7fdfb7400640 1 -- 172.21.15.90:0/3603295634 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfcc028060 con 0x55f67a36ad10 2024-04-02T14:38:30.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.331+0000 7fdfb7400640 1 -- 172.21.15.90:0/3603295634 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7fdfcc00a700 con 0x55f67a36ad10 2024-04-02T14:38:30.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.331+0000 7fdfdf1c3740 1 -- 172.21.15.90:0/3603295634 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdfcc0158f0 con 0x55f67a36ad10 2024-04-02T14:38:30.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.332+0000 7fdfb7400640 1 -- 172.21.15.90:0/3603295634 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fdfcc01a040 con 0x55f67a36ad10 2024-04-02T14:38:30.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.332+0000 7fdfdf1c3740 10 client.26413 fetch_fsmap learned FSMap version 27 2024-04-02T14:38:30.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.332+0000 7fdfdf1c3740 1 -- 172.21.15.90:0/3603295634 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55f67a382ea0 con 0x55f67a36ad10 2024-04-02T14:38:30.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.333+0000 7fdfb7400640 1 -- 172.21.15.90:0/3603295634 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdfcc0158f0 con 0x55f67a36ad10 2024-04-02T14:38:30.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.333+0000 7fdfb7400640 1 -- 172.21.15.90:0/3603295634 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fdfcc020070 con 0x55f67a36ad10 2024-04-02T14:38:30.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.333+0000 7fdfdf1c3740 10 client.26413 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:38:30.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.333+0000 7fdfdf1c3740 10 client.26413 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-04-02T14:38:30.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.333+0000 7fdfdf1c3740 20 client.26413 populate_metadata read hostname 'smithi090' 2024-04-02T14:38:30.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.333+0000 7fdfdf1c3740 1 --2- 172.21.15.90:0/3603295634 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x55f67a384250 0x55f67a386640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:30.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.333+0000 7fdfdf1c3740 4 client.26413 mds_command: new command op to 24547 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:38:30.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.333+0000 7fdfdf1c3740 1 -- 172.21.15.90:0/3603295634 --> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f67a28f040 con 0x55f67a384250 2024-04-02T14:38:30.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.333+0000 7fdfd9000640 1 --2- 172.21.15.90:0/3603295634 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x55f67a384250 0x55f67a386640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:30.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.334+0000 7fdfd9000640 1 --2- 172.21.15.90:0/3603295634 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x55f67a384250 0x55f67a386640 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=21eb3413a9a3ae3a server_cookie=267abf786f76ad1a in_seq=0 out_seq=0 2024-04-02T14:38:30.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.334+0000 7fdfb7400640 10 client.26413 ms_handle_connect on v2:172.21.15.177:6838/664895663 2024-04-02T14:38:30.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.335+0000 7fdfb7400640 1 -- 172.21.15.90:0/3603295634 <== mds.2 v2:172.21.15.177:6838/664895663 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f67a28f040 con 0x55f67a384250 2024-04-02T14:38:30.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.335+0000 7fdfb7400640 10 client.26413 handle_command_reply: tid=0 2024-04-02T14:38:30.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.335+0000 7fdfdf1c3740 1 client.26413 shutdown 2024-04-02T14:38:30.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.336+0000 7fdfdf1c3740 1 -- 172.21.15.90:0/3603295634 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f67a36ad10 msgr2=0x55f67a370f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:38:30.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.336+0000 7fdfdf1c3740 1 --2- 172.21.15.90:0/3603295634 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f67a36ad10 0x55f67a370f50 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fdfcc000f80 tx=0x7fdfcc0049c0 comp rx=0 tx=0).stop 2024-04-02T14:38:30.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.336+0000 7fdfdf1c3740 1 -- 172.21.15.90:0/3603295634 shutdown_connections 2024-04-02T14:38:30.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.336+0000 7fdfdf1c3740 1 --2- 172.21.15.90:0/3603295634 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x55f67a384250 0x55f67a386640 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:30.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.336+0000 7fdfdf1c3740 1 --2- 172.21.15.90:0/3603295634 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f67a374080 0x55f67a3719d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:30.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.336+0000 7fdfdf1c3740 1 --2- 172.21.15.90:0/3603295634 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f67a36b9f0 0x55f67a371490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:30.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.336+0000 7fdfdf1c3740 1 --2- 172.21.15.90:0/3603295634 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f67a36ad10 0x55f67a370f50 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:30.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.336+0000 7fdfdf1c3740 1 -- 172.21.15.90:0/3603295634 >> 172.21.15.90:0/3603295634 conn(0x55f67a331a50 msgr2=0x55f67a3674b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:38:30.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.336+0000 7fdfb7400640 0 client.26413 ms_handle_reset on v2:172.21.15.177:6838/664895663 2024-04-02T14:38:30.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.337+0000 7fdfdf1c3740 1 -- 172.21.15.90:0/3603295634 shutdown_connections 2024-04-02T14:38:30.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.337+0000 7fdfdf1c3740 1 -- 172.21.15.90:0/3603295634 wait complete. 2024-04-02T14:38:30.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.338+0000 7fdfdf1c3740 20 client.26413 trim_cache size 0 max 16384 2024-04-02T14:38:30.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.363+0000 7fdfdf1c3740 1 Processor -- start 2024-04-02T14:38:30.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.363+0000 7fdfdf1c3740 1 -- start start 2024-04-02T14:38:30.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.363+0000 7fdfdf1c3740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f67a36ad10 0x55f67a373680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:30.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.363+0000 7fdfdf1c3740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f67a36b9f0 0x55f67a3717f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:30.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.363+0000 7fdfdf1c3740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f67a374080 0x55f67a389660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:30.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.363+0000 7fdfdf1c3740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fdfb800e850 con 0x55f67a374080 2024-04-02T14:38:30.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.363+0000 7fdfdf1c3740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fdfb8010390 con 0x55f67a36ad10 2024-04-02T14:38:30.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.363+0000 7fdfdf1c3740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fdfb800ef20 con 0x55f67a36b9f0 2024-04-02T14:38:30.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.364+0000 7fdfda400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f67a36ad10 0x55f67a373680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:30.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.364+0000 7fdfda400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f67a36ad10 0x55f67a373680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:47986/0 (socket says 172.21.15.90:47986) 2024-04-02T14:38:30.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.364+0000 7fdfda400640 1 -- 172.21.15.90:0/2535894682 learned_addr learned my addr 172.21.15.90:0/2535894682 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:38:30.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.364+0000 7fdfd9000640 1 --2- 172.21.15.90:0/2535894682 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f67a36b9f0 0x55f67a3717f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:30.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.365+0000 7fdfda400640 1 --2- 172.21.15.90:0/2535894682 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f67a374080 0x55f67a389660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:30.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.365+0000 7fdfd9000640 1 -- 172.21.15.90:0/2535894682 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f67a36ad10 msgr2=0x55f67a373680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:38:30.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.365+0000 7fdfd9000640 1 --2- 172.21.15.90:0/2535894682 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f67a36ad10 0x55f67a373680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:30.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.365+0000 7fdfd9000640 1 -- 172.21.15.90:0/2535894682 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f67a374080 msgr2=0x55f67a389660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:38:30.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.365+0000 7fdfd9000640 1 --2- 172.21.15.90:0/2535894682 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f67a374080 0x55f67a389660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:30.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.365+0000 7fdfd9000640 1 -- 172.21.15.90:0/2535894682 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f67a369d40 con 0x55f67a36b9f0 2024-04-02T14:38:30.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.365+0000 7fdfd9000640 1 --2- 172.21.15.90:0/2535894682 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f67a36b9f0 0x55f67a3717f0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fdfd407ca00 tx=0x7fdfc402cf90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=480269660ca2f9f2 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:38:30.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.366+0000 7fdfc2000640 1 -- 172.21.15.90:0/2535894682 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfc4045dc0 con 0x55f67a36b9f0 2024-04-02T14:38:30.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.366+0000 7fdfc2000640 1 -- 172.21.15.90:0/2535894682 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdfc4004200 con 0x55f67a36b9f0 2024-04-02T14:38:30.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.366+0000 7fdfc2000640 1 -- 172.21.15.90:0/2535894682 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfc4005020 con 0x55f67a36b9f0 2024-04-02T14:38:30.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.367+0000 7fdfdf1c3740 1 -- 172.21.15.90:0/2535894682 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f67a36b9f0 msgr2=0x55f67a3717f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:38:30.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.367+0000 7fdfdf1c3740 1 --2- 172.21.15.90:0/2535894682 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f67a36b9f0 0x55f67a3717f0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fdfd407ca00 tx=0x7fdfc402cf90 comp rx=0 tx=0).stop 2024-04-02T14:38:30.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.367+0000 7fdfdf1c3740 1 -- 172.21.15.90:0/2535894682 shutdown_connections 2024-04-02T14:38:30.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.367+0000 7fdfdf1c3740 1 --2- 172.21.15.90:0/2535894682 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f67a374080 0x55f67a389660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:30.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.367+0000 7fdfdf1c3740 1 --2- 172.21.15.90:0/2535894682 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f67a36b9f0 0x55f67a3717f0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:30.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.367+0000 7fdfdf1c3740 1 --2- 172.21.15.90:0/2535894682 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f67a36ad10 0x55f67a373680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:30.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.367+0000 7fdfdf1c3740 1 -- 172.21.15.90:0/2535894682 >> 172.21.15.90:0/2535894682 conn(0x55f67a331a50 msgr2=0x55f67a37f9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:38:30.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.368+0000 7fdfdf1c3740 1 -- 172.21.15.90:0/2535894682 shutdown_connections 2024-04-02T14:38:30.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.368+0000 7fdfdf1c3740 1 -- 172.21.15.90:0/2535894682 wait complete. 2024-04-02T14:38:30.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.368+0000 7fdfdf1c3740 1 -- start start 2024-04-02T14:38:30.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.369+0000 7fdfdf1c3740 1 Processor -- start 2024-04-02T14:38:30.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.371+0000 7fdfdf1c3740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f67a36ad10 0x55f67a387770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:30.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.372+0000 7fdfdf1c3740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f67a36b9f0 0x55f67a387cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:30.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.372+0000 7fdfdf1c3740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f67a374080 0x55f67a3e9430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:30.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.372+0000 7fdfdf1c3740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fdfb800fcc0 con 0x55f67a36ad10 2024-04-02T14:38:30.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.372+0000 7fdfdf1c3740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fdfb800f5f0 con 0x55f67a36b9f0 2024-04-02T14:38:30.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.372+0000 7fdfdf1c3740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fdfb800c7f0 con 0x55f67a374080 2024-04-02T14:38:30.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.372+0000 7fdfdf1c3740 10 client.0 _ll_register_callbacks cb 0x7fdfcc009df0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:38:30.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.372+0000 7fdfd9000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f67a36b9f0 0x55f67a387cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:30.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.372+0000 7fdfd9000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f67a36b9f0 0x55f67a387cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:47992/0 (socket says 172.21.15.90:47992) 2024-04-02T14:38:30.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.372+0000 7fdfd9000640 1 -- 172.21.15.90:0/1528255442 learned_addr learned my addr 172.21.15.90:0/1528255442 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:38:30.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.373+0000 7fdfd9000640 1 -- 172.21.15.90:0/1528255442 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f67a374080 msgr2=0x55f67a3e9430 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-02T14:38:30.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.373+0000 7fdfd9000640 1 --2- 172.21.15.90:0/1528255442 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f67a374080 0x55f67a3e9430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:30.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.373+0000 7fdfd9000640 1 -- 172.21.15.90:0/1528255442 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f67a36ad10 msgr2=0x55f67a387770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:38:30.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.373+0000 7fdfda400640 1 --2- 172.21.15.90:0/1528255442 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f67a36ad10 0x55f67a387770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:30.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.374+0000 7fdfd9000640 1 --2- 172.21.15.90:0/1528255442 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f67a36ad10 0x55f67a387770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:30.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.374+0000 7fdfd9000640 1 -- 172.21.15.90:0/1528255442 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f67a3e9970 con 0x55f67a36b9f0 2024-04-02T14:38:30.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.374+0000 7fdfda400640 1 --2- 172.21.15.90:0/1528255442 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f67a36ad10 0x55f67a387770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:38:30.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.375+0000 7fdfd9000640 1 --2- 172.21.15.90:0/1528255442 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f67a36b9f0 0x55f67a387cb0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fdfc402c180 tx=0x7fdfc4045610 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b303b5eca52d60ad server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:38:30.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.375+0000 7fdfb7400640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:38:30.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.375+0000 7fdfb7400640 1 -- 172.21.15.90:0/1528255442 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfc402ca90 con 0x55f67a36b9f0 2024-04-02T14:38:30.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.375+0000 7fdfb7400640 1 -- 172.21.15.90:0/1528255442 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7fdfc402cc30 con 0x55f67a36b9f0 2024-04-02T14:38:30.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.376+0000 7fdfdf1c3740 1 -- 172.21.15.90:0/1528255442 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdfc4004200 con 0x55f67a36b9f0 2024-04-02T14:38:30.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.376+0000 7fdfb7400640 1 -- 172.21.15.90:0/1528255442 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fdfc4005030 con 0x55f67a36b9f0 2024-04-02T14:38:30.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.376+0000 7fdfdf1c3740 10 client.26513 fetch_fsmap learned FSMap version 27 2024-04-02T14:38:30.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.376+0000 7fdfdf1c3740 1 -- 172.21.15.90:0/1528255442 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55f67a3eac10 con 0x55f67a36b9f0 2024-04-02T14:38:30.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.376+0000 7fdfb7400640 1 -- 172.21.15.90:0/1528255442 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdfc4004200 con 0x55f67a36b9f0 2024-04-02T14:38:30.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.377+0000 7fdfb7400640 1 -- 172.21.15.90:0/1528255442 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fdfc40046c0 con 0x55f67a36b9f0 2024-04-02T14:38:30.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.377+0000 7fdfdf1c3740 10 client.26513 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:38:30.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.377+0000 7fdfdf1c3740 10 client.26513 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-04-02T14:38:30.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.377+0000 7fdfdf1c3740 20 client.26513 populate_metadata read hostname 'smithi090' 2024-04-02T14:38:30.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.377+0000 7fdfdf1c3740 1 --2- 172.21.15.90:0/1528255442 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x55f67a3eb3c0 0x55f67a3ed820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:30.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.377+0000 7fdfdf1c3740 4 client.26513 mds_command: new command op to 24547 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-02T14:38:30.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.377+0000 7fdfdf1c3740 1 -- 172.21.15.90:0/1528255442 --> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55f67a2dcc30 con 0x55f67a3eb3c0 2024-04-02T14:38:30.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.378+0000 7fdfda400640 1 --2- 172.21.15.90:0/1528255442 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x55f67a3eb3c0 0x55f67a3ed820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:30.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.379+0000 7fdfb7400640 10 client.26513 ms_handle_connect on v2:172.21.15.177:6838/664895663 2024-04-02T14:38:30.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.379+0000 7fdfda400640 1 --2- 172.21.15.90:0/1528255442 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x55f67a3eb3c0 0x55f67a3ed820 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=2d7ad382ffbdded1 server_cookie=9f6be132496f65d in_seq=0 out_seq=0 2024-04-02T14:38:30.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.379+0000 7fdfb7400640 1 -- 172.21.15.90:0/1528255442 <== mds.2 v2:172.21.15.177:6838/664895663 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55f67a2dcc30 con 0x55f67a3eb3c0 2024-04-02T14:38:30.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.379+0000 7fdfb7400640 10 client.26513 handle_command_reply: tid=0 2024-04-02T14:38:30.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.379+0000 7fdfdf1c3740 1 client.26513 shutdown 2024-04-02T14:38:30.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.380+0000 7fdfdf1c3740 1 -- 172.21.15.90:0/1528255442 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f67a36b9f0 msgr2=0x55f67a387cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:38:30.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.380+0000 7fdfdf1c3740 1 --2- 172.21.15.90:0/1528255442 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f67a36b9f0 0x55f67a387cb0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fdfc402c180 tx=0x7fdfc4045610 comp rx=0 tx=0).stop 2024-04-02T14:38:30.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.380+0000 7fdfdf1c3740 1 -- 172.21.15.90:0/1528255442 shutdown_connections 2024-04-02T14:38:30.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.380+0000 7fdfdf1c3740 1 --2- 172.21.15.90:0/1528255442 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x55f67a3eb3c0 0x55f67a3ed820 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:30.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.380+0000 7fdfdf1c3740 1 --2- 172.21.15.90:0/1528255442 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f67a374080 0x55f67a3e9430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:30.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.380+0000 7fdfdf1c3740 1 --2- 172.21.15.90:0/1528255442 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f67a36b9f0 0x55f67a387cb0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:30.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.380+0000 7fdfdf1c3740 1 --2- 172.21.15.90:0/1528255442 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f67a36ad10 0x55f67a387770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:30.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.380+0000 7fdfdf1c3740 1 -- 172.21.15.90:0/1528255442 >> 172.21.15.90:0/1528255442 conn(0x55f67a331a50 msgr2=0x55f67a366b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:38:30.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.380+0000 7fdfb7400640 0 client.26513 ms_handle_reset on v2:172.21.15.177:6838/664895663 2024-04-02T14:38:30.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.381+0000 7fdfdf1c3740 1 -- 172.21.15.90:0/1528255442 shutdown_connections 2024-04-02T14:38:30.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.381+0000 7fdfdf1c3740 1 -- 172.21.15.90:0/1528255442 wait complete. 2024-04-02T14:38:30.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.381+0000 7fdfdf1c3740 20 client.26513 trim_cache size 0 max 16384 2024-04-02T14:38:30.383 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-02T14:38:30.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.382+0000 7fdfdc400640 1 -- 172.21.15.90:0/50047533 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fdfbc0a3e20 msgr2=0x7fdfbc0a62e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:38:30.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.382+0000 7fdfdc400640 1 --2- 172.21.15.90:0/50047533 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fdfbc0a3e20 0x7fdfbc0a62e0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x55f67a36bf40 tx=0x7fdfc8087040 comp rx=0 tx=0).stop 2024-04-02T14:38:30.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.382+0000 7fdfdc400640 1 -- 172.21.15.90:0/50047533 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdfd4108310 msgr2=0x7fdfd40821c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:38:30.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.382+0000 7fdfdc400640 1 --2- 172.21.15.90:0/50047533 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdfd4108310 0x7fdfd40821c0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fdfc800ea60 tx=0x7fdfc800ef20 comp rx=0 tx=0).stop 2024-04-02T14:38:30.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.382+0000 7fdfdc400640 1 -- 172.21.15.90:0/50047533 shutdown_connections 2024-04-02T14:38:30.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.382+0000 7fdfdc400640 1 --2- 172.21.15.90:0/50047533 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fdfbc0a3e20 0x7fdfbc0a62e0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:30.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.383+0000 7fdfdc400640 1 --2- 172.21.15.90:0/50047533 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdfd407b780 0x7fdfd407bc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:30.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.383+0000 7fdfdc400640 1 --2- 172.21.15.90:0/50047533 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdfd410d3e0 0x7fdfd407b240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:30.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.383+0000 7fdfdc400640 1 --2- 172.21.15.90:0/50047533 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdfd4108310 0x7fdfd40821c0 secure :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fdfc800ea60 tx=0x7fdfc800ef20 comp rx=0 tx=0).stop 2024-04-02T14:38:30.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.383+0000 7fdfdc400640 1 -- 172.21.15.90:0/50047533 >> 172.21.15.90:0/50047533 conn(0x7fdfd4076040 msgr2=0x7fdfd4078300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:38:30.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.383+0000 7fdfdc400640 1 -- 172.21.15.90:0/50047533 shutdown_connections 2024-04-02T14:38:30.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.383+0000 7fdfdc400640 1 -- 172.21.15.90:0/50047533 wait complete. 2024-04-02T14:38:30.412 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-02T14:38:30.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.692+0000 7fabac600640 1 -- 172.21.15.90:0/3411480726 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7faba4108ce0 msgr2=0x7faba41090c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:38:30.694 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.692+0000 7fabac600640 1 --2- 172.21.15.90:0/3411480726 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7faba4108ce0 0x7faba41090c0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fab9c009ca0 tx=0x7fab9c02c100 comp rx=0 tx=0).stop 2024-04-02T14:38:30.695 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.694+0000 7fabac600640 1 -- 172.21.15.90:0/3411480726 shutdown_connections 2024-04-02T14:38:30.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.694+0000 7fabac600640 1 --2- 172.21.15.90:0/3411480726 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faba4114c50 0x7faba4117040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:30.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.694+0000 7fabac600640 1 --2- 172.21.15.90:0/3411480726 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faba4109600 0x7faba4114710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:30.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.694+0000 7fabac600640 1 --2- 172.21.15.90:0/3411480726 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7faba4108ce0 0x7faba41090c0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:30.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.694+0000 7fabac600640 1 -- 172.21.15.90:0/3411480726 >> 172.21.15.90:0/3411480726 conn(0x7faba406cbe0 msgr2=0x7faba406cff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:38:30.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.695+0000 7fabac600640 1 -- 172.21.15.90:0/3411480726 shutdown_connections 2024-04-02T14:38:30.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.695+0000 7fabac600640 1 -- 172.21.15.90:0/3411480726 wait complete. 2024-04-02T14:38:30.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.697+0000 7fabac600640 1 Processor -- start 2024-04-02T14:38:30.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.698+0000 7fabac600640 1 -- start start 2024-04-02T14:38:30.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.698+0000 7fabac600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7faba4108ce0 0x7faba410a570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:30.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.698+0000 7fabac600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faba4109600 0x7faba410aab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:30.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.698+0000 7fabac600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faba4114c50 0x7faba410aff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:30.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.698+0000 7fabac600640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7faba411a6b0 con 0x7faba4114c50 2024-04-02T14:38:30.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.698+0000 7fabac600640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7faba411a530 con 0x7faba4108ce0 2024-04-02T14:38:30.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.698+0000 7fabac600640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7faba411a830 con 0x7faba4109600 2024-04-02T14:38:30.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.699+0000 7faba9200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faba4109600 0x7faba410aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:30.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.699+0000 7faba9200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faba4109600 0x7faba410aab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:37766/0 (socket says 172.21.15.90:37766) 2024-04-02T14:38:30.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.699+0000 7faba9200640 1 -- 172.21.15.90:0/2237013781 learned_addr learned my addr 172.21.15.90:0/2237013781 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:38:30.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.699+0000 7fabaa600640 1 --2- 172.21.15.90:0/2237013781 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faba4114c50 0x7faba410aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:30.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.699+0000 7faba9c00640 1 --2- 172.21.15.90:0/2237013781 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7faba4108ce0 0x7faba410a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:30.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.699+0000 7faba9200640 1 -- 172.21.15.90:0/2237013781 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7faba4108ce0 msgr2=0x7faba410a570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:38:30.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.699+0000 7faba9200640 1 --2- 172.21.15.90:0/2237013781 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7faba4108ce0 0x7faba410a570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:30.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.699+0000 7faba9200640 1 -- 172.21.15.90:0/2237013781 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faba4114c50 msgr2=0x7faba410aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:38:30.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.699+0000 7faba9200640 1 --2- 172.21.15.90:0/2237013781 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faba4114c50 0x7faba410aff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:30.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.699+0000 7faba9200640 1 -- 172.21.15.90:0/2237013781 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faba41af150 con 0x7faba4109600 2024-04-02T14:38:30.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.700+0000 7faba9200640 1 --2- 172.21.15.90:0/2237013781 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faba4109600 0x7faba410aab0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fab9000d950 tx=0x7fab9000de10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:38:30.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.700+0000 7faba2000640 1 -- 172.21.15.90:0/2237013781 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab90017030 con 0x7faba4109600 2024-04-02T14:38:30.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.700+0000 7faba2000640 1 -- 172.21.15.90:0/2237013781 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fab90004020 con 0x7faba4109600 2024-04-02T14:38:30.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.700+0000 7faba2000640 1 -- 172.21.15.90:0/2237013781 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab900047a0 con 0x7faba4109600 2024-04-02T14:38:30.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.701+0000 7fabac600640 1 -- 172.21.15.90:0/2237013781 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faba41a0a00 con 0x7faba4109600 2024-04-02T14:38:30.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.701+0000 7fabac600640 1 -- 172.21.15.90:0/2237013781 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faba41a0f00 con 0x7faba4109600 2024-04-02T14:38:30.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.706+0000 7faba2000640 1 -- 172.21.15.90:0/2237013781 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fab90004940 con 0x7faba4109600 2024-04-02T14:38:30.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.706+0000 7fabaf25c740 1 Processor -- start 2024-04-02T14:38:30.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.707+0000 7fabaf25c740 1 -- start start 2024-04-02T14:38:30.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.707+0000 7fabaf25c740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d69ed289f0 0x55d69ed27740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:30.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.707+0000 7fabaf25c740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d69ed27d10 0x55d69ed28190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:30.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.707+0000 7fabaf25c740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d69ed31080 0x55d69ed33470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:30.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.707+0000 7fabaf25c740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55d69ec68400 con 0x55d69ed31080 2024-04-02T14:38:30.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.707+0000 7fabaf25c740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55d69ec4cf10 con 0x55d69ed27d10 2024-04-02T14:38:30.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.707+0000 7fabaf25c740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55d69ec8c6f0 con 0x55d69ed289f0 2024-04-02T14:38:30.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.708+0000 7faba9c00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d69ed27d10 0x55d69ed28190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:30.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.708+0000 7faba9c00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d69ed27d10 0x55d69ed28190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:48028/0 (socket says 172.21.15.90:48028) 2024-04-02T14:38:30.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.708+0000 7faba9c00640 1 -- 172.21.15.90:0/1424181812 learned_addr learned my addr 172.21.15.90:0/1424181812 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:38:30.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.708+0000 7fabaa600640 1 --2- 172.21.15.90:0/1424181812 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d69ed289f0 0x55d69ed27740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:30.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.708+0000 7faba9200640 1 --2- 172.21.15.90:0/1424181812 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d69ed31080 0x55d69ed33470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:30.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.709+0000 7faba2000640 1 --2- 172.21.15.90:0/2237013781 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fab8c0a3da0 0x7fab8c0a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:30.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.709+0000 7faba9c00640 1 -- 172.21.15.90:0/1424181812 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d69ed289f0 msgr2=0x55d69ed27740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:38:30.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.709+0000 7faba9c00640 1 --2- 172.21.15.90:0/1424181812 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d69ed289f0 0x55d69ed27740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:30.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.709+0000 7faba2000640 1 -- 172.21.15.90:0/2237013781 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7fab900d0100 con 0x7faba4109600 2024-04-02T14:38:30.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.709+0000 7faba9c00640 1 -- 172.21.15.90:0/1424181812 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d69ed31080 msgr2=0x55d69ed33470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:38:30.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.709+0000 7faba9c00640 1 --2- 172.21.15.90:0/1424181812 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d69ed31080 0x55d69ed33470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:30.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.709+0000 7faba9c00640 1 -- 172.21.15.90:0/1424181812 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d69e9a03f0 con 0x55d69ed27d10 2024-04-02T14:38:30.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.710+0000 7faba9200640 1 --2- 172.21.15.90:0/1424181812 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d69ed31080 0x55d69ed33470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:38:30.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.710+0000 7fabaa600640 1 --2- 172.21.15.90:0/1424181812 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d69ed289f0 0x55d69ed27740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:38:30.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.710+0000 7faba9c00640 1 --2- 172.21.15.90:0/1424181812 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d69ed27d10 0x55d69ed28190 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fab9c037040 tx=0x7fab9c0099c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4a9a6f0e1779972b server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:38:30.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.710+0000 7fabaa600640 1 --2- 172.21.15.90:0/2237013781 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fab8c0a3da0 0x7fab8c0a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:30.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.710+0000 7fab96a00640 1 -- 172.21.15.90:0/1424181812 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab9c043070 con 0x55d69ed27d10 2024-04-02T14:38:30.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.711+0000 7fab96a00640 1 -- 172.21.15.90:0/1424181812 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fab9c003970 con 0x55d69ed27d10 2024-04-02T14:38:30.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.711+0000 7fab96a00640 1 -- 172.21.15.90:0/1424181812 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab9c039040 con 0x55d69ed27d10 2024-04-02T14:38:30.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.712+0000 7fabaf25c740 1 -- 172.21.15.90:0/1424181812 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d69ed27d10 msgr2=0x55d69ed28190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:38:30.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.712+0000 7fabaf25c740 1 --2- 172.21.15.90:0/1424181812 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d69ed27d10 0x55d69ed28190 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fab9c037040 tx=0x7fab9c0099c0 comp rx=0 tx=0).stop 2024-04-02T14:38:30.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.712+0000 7fabaa600640 1 --2- 172.21.15.90:0/2237013781 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fab8c0a3da0 0x7fab8c0a6260 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7faba406af20 tx=0x7fab98008040 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:38:30.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.712+0000 7fabaf25c740 1 -- 172.21.15.90:0/1424181812 shutdown_connections 2024-04-02T14:38:30.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.712+0000 7fabaf25c740 1 --2- 172.21.15.90:0/1424181812 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d69ed31080 0x55d69ed33470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:30.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.712+0000 7fabaf25c740 1 --2- 172.21.15.90:0/1424181812 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d69ed27d10 0x55d69ed28190 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:30.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.712+0000 7fabaf25c740 1 --2- 172.21.15.90:0/1424181812 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d69ed289f0 0x55d69ed27740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:30.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.712+0000 7fabaf25c740 1 -- 172.21.15.90:0/1424181812 >> 172.21.15.90:0/1424181812 conn(0x55d69eceea50 msgr2=0x55d69ed22760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:38:30.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.713+0000 7fabaf25c740 1 -- 172.21.15.90:0/1424181812 shutdown_connections 2024-04-02T14:38:30.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.713+0000 7fabaf25c740 1 -- 172.21.15.90:0/1424181812 wait complete. 2024-04-02T14:38:30.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.713+0000 7fabaf25c740 1 -- start start 2024-04-02T14:38:30.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.714+0000 7fabaf25c740 1 Processor -- start 2024-04-02T14:38:30.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.714+0000 7fabaf25c740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d69ed27d10 0x55d69ed2e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:30.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.714+0000 7fabaf25c740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d69ed289f0 0x55d69ed2e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:30.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.715+0000 7fabaf25c740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d69ed31080 0x55d69ed2eb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:30.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.715+0000 7fabaf25c740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55d69ec886a0 con 0x55d69ed31080 2024-04-02T14:38:30.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.715+0000 7fabaf25c740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55d69ec34980 con 0x55d69ed27d10 2024-04-02T14:38:30.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.715+0000 7fabaf25c740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55d69ec47cb0 con 0x55d69ed289f0 2024-04-02T14:38:30.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.715+0000 7fabaf25c740 10 client.0 _ll_register_callbacks cb 0x55d69ec476a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:38:30.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.715+0000 7faba9c00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d69ed27d10 0x55d69ed2e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:30.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.715+0000 7faba9200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d69ed289f0 0x55d69ed2e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:30.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.715+0000 7faba9c00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d69ed27d10 0x55d69ed2e0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:48032/0 (socket says 172.21.15.90:48032) 2024-04-02T14:38:30.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.715+0000 7faba9c00640 1 -- 172.21.15.90:0/1259469968 learned_addr learned my addr 172.21.15.90:0/1259469968 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:38:30.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.715+0000 7fabaa600640 1 --2- 172.21.15.90:0/1259469968 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d69ed31080 0x55d69ed2eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:30.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.715+0000 7faba9c00640 1 -- 172.21.15.90:0/1259469968 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d69ed289f0 msgr2=0x55d69ed2e5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:38:30.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.715+0000 7faba9c00640 1 --2- 172.21.15.90:0/1259469968 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d69ed289f0 0x55d69ed2e5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:30.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.715+0000 7faba9c00640 1 -- 172.21.15.90:0/1259469968 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d69ed31080 msgr2=0x55d69ed2eb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:38:30.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.715+0000 7faba9c00640 1 --2- 172.21.15.90:0/1259469968 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d69ed31080 0x55d69ed2eb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:30.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.715+0000 7faba9c00640 1 -- 172.21.15.90:0/1259469968 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d69ea6a7d0 con 0x55d69ed27d10 2024-04-02T14:38:30.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.715+0000 7faba9200640 1 --2- 172.21.15.90:0/1259469968 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d69ed289f0 0x55d69ed2e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:38:30.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.716+0000 7faba9c00640 1 --2- 172.21.15.90:0/1259469968 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d69ed27d10 0x55d69ed2e0a0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fab9c004af0 tx=0x7fab9c02c100 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f5f02b84fdb571da server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:38:30.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.716+0000 7fab8b400640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:38:30.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.716+0000 7fab8b400640 1 -- 172.21.15.90:0/1259469968 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab9c04c070 con 0x55d69ed27d10 2024-04-02T14:38:30.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.716+0000 7fab8b400640 1 -- 172.21.15.90:0/1259469968 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7fab9c032790 con 0x55d69ed27d10 2024-04-02T14:38:30.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.716+0000 7fabaf25c740 1 -- 172.21.15.90:0/1259469968 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fab9c003970 con 0x55d69ed27d10 2024-04-02T14:38:30.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.716+0000 7fab8b400640 1 -- 172.21.15.90:0/1259469968 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fab9c058050 con 0x55d69ed27d10 2024-04-02T14:38:30.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.717+0000 7fabaf25c740 10 client.26525 fetch_fsmap learned FSMap version 27 2024-04-02T14:38:30.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.717+0000 7fabaf25c740 1 -- 172.21.15.90:0/1259469968 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55d69ed50f10 con 0x55d69ed27d10 2024-04-02T14:38:30.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.717+0000 7fab8b400640 1 -- 172.21.15.90:0/1259469968 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fab9c003970 con 0x55d69ed27d10 2024-04-02T14:38:30.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.717+0000 7fab8b400640 1 -- 172.21.15.90:0/1259469968 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fab9c0488c0 con 0x55d69ed27d10 2024-04-02T14:38:30.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.717+0000 7fabaf25c740 10 client.26525 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:38:30.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.717+0000 7fabaf25c740 10 client.26525 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2024-04-02T14:38:30.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.717+0000 7fabaf25c740 20 client.26525 populate_metadata read hostname 'smithi090' 2024-04-02T14:38:30.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.717+0000 7fabaf25c740 1 --2- 172.21.15.90:0/1259469968 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x55d69ed52260 0x55d69ed54650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:30.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.718+0000 7fabaf25c740 4 client.26525 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:38:30.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.718+0000 7fabaf25c740 1 -- 172.21.15.90:0/1259469968 --> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d69ec4c040 con 0x55d69ed52260 2024-04-02T14:38:30.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.718+0000 7faba9200640 1 --2- 172.21.15.90:0/1259469968 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x55d69ed52260 0x55d69ed54650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:30.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.719+0000 7faba9200640 1 --2- 172.21.15.90:0/1259469968 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x55d69ed52260 0x55d69ed54650 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=fa9e1d75dacbb951 server_cookie=1f6bfd87d7f44332 in_seq=0 out_seq=0 2024-04-02T14:38:30.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.719+0000 7fab8b400640 10 client.26525 ms_handle_connect on v2:172.21.15.90:6838/1498915870 2024-04-02T14:38:30.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.720+0000 7fab8b400640 1 -- 172.21.15.90:0/1259469968 <== mds.1 v2:172.21.15.90:6838/1498915870 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d69ec4c040 con 0x55d69ed52260 2024-04-02T14:38:30.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.720+0000 7fab8b400640 10 client.26525 handle_command_reply: tid=0 2024-04-02T14:38:30.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.720+0000 7fabaf25c740 1 client.26525 shutdown 2024-04-02T14:38:30.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.721+0000 7fabaf25c740 1 -- 172.21.15.90:0/1259469968 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d69ed27d10 msgr2=0x55d69ed2e0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:38:30.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.721+0000 7fabaf25c740 1 --2- 172.21.15.90:0/1259469968 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d69ed27d10 0x55d69ed2e0a0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fab9c004af0 tx=0x7fab9c02c100 comp rx=0 tx=0).stop 2024-04-02T14:38:30.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.721+0000 7fabaf25c740 1 -- 172.21.15.90:0/1259469968 shutdown_connections 2024-04-02T14:38:30.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.721+0000 7fabaf25c740 1 --2- 172.21.15.90:0/1259469968 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x55d69ed52260 0x55d69ed54650 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:30.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.721+0000 7fabaf25c740 1 --2- 172.21.15.90:0/1259469968 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d69ed31080 0x55d69ed2eb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:30.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.721+0000 7fabaf25c740 1 --2- 172.21.15.90:0/1259469968 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d69ed289f0 0x55d69ed2e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:30.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.721+0000 7fabaf25c740 1 --2- 172.21.15.90:0/1259469968 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d69ed27d10 0x55d69ed2e0a0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:30.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.721+0000 7fabaf25c740 1 -- 172.21.15.90:0/1259469968 >> 172.21.15.90:0/1259469968 conn(0x55d69eceea50 msgr2=0x55d69ed244b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:38:30.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.721+0000 7fab8b400640 0 client.26525 ms_handle_reset on v2:172.21.15.90:6838/1498915870 2024-04-02T14:38:30.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.721+0000 7fabaf25c740 1 -- 172.21.15.90:0/1259469968 shutdown_connections 2024-04-02T14:38:30.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.721+0000 7fabaf25c740 1 -- 172.21.15.90:0/1259469968 wait complete. 2024-04-02T14:38:30.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.722+0000 7fabaf25c740 20 client.26525 trim_cache size 0 max 16384 2024-04-02T14:38:30.745 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.744+0000 7fabaf25c740 1 Processor -- start 2024-04-02T14:38:30.746 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.744+0000 7fabaf25c740 1 -- start start 2024-04-02T14:38:30.746 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.745+0000 7fabaf25c740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d69ed27d10 0x55d69ed307d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:30.746 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.745+0000 7fabaf25c740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d69ed289f0 0x55d69ed242c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:30.746 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.745+0000 7fabaf25c740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d69ed31080 0x55d69ed2e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:30.746 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.745+0000 7fabaf25c740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7faba41b3e70 con 0x55d69ed27d10 2024-04-02T14:38:30.746 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.745+0000 7fabaf25c740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7faba41b59b0 con 0x55d69ed289f0 2024-04-02T14:38:30.746 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.745+0000 7fabaf25c740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7faba41b4540 con 0x55d69ed31080 2024-04-02T14:38:30.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.745+0000 7faba9200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d69ed289f0 0x55d69ed242c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:30.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.745+0000 7faba9200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d69ed289f0 0x55d69ed242c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:48046/0 (socket says 172.21.15.90:48046) 2024-04-02T14:38:30.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.745+0000 7faba9c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d69ed27d10 0x55d69ed307d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:30.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.746+0000 7faba9200640 1 -- 172.21.15.90:0/3123294948 learned_addr learned my addr 172.21.15.90:0/3123294948 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:38:30.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.746+0000 7fabaa600640 1 --2- 172.21.15.90:0/3123294948 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d69ed31080 0x55d69ed2e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:30.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.746+0000 7faba9200640 1 -- 172.21.15.90:0/3123294948 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d69ed31080 msgr2=0x55d69ed2e5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:38:30.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.746+0000 7faba9200640 1 --2- 172.21.15.90:0/3123294948 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d69ed31080 0x55d69ed2e5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:30.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.746+0000 7faba9200640 1 -- 172.21.15.90:0/3123294948 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d69ed27d10 msgr2=0x55d69ed307d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:38:30.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.746+0000 7faba9200640 1 --2- 172.21.15.90:0/3123294948 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d69ed27d10 0x55d69ed307d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:30.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.746+0000 7faba9200640 1 -- 172.21.15.90:0/3123294948 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d69ed238e0 con 0x55d69ed289f0 2024-04-02T14:38:30.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.746+0000 7faba9200640 1 --2- 172.21.15.90:0/3123294948 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d69ed289f0 0x55d69ed242c0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fab90099a30 tx=0x7fab90000bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fe86c6428a6d6f96 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:38:30.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.746+0000 7fabaa600640 1 --2- 172.21.15.90:0/3123294948 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d69ed31080 0x55d69ed2e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:38:30.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.746+0000 7faba9c00640 1 --2- 172.21.15.90:0/3123294948 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d69ed27d10 0x55d69ed307d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:38:30.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.747+0000 7fab96000640 1 -- 172.21.15.90:0/3123294948 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab9009c9c0 con 0x55d69ed289f0 2024-04-02T14:38:30.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.747+0000 7fab96000640 1 -- 172.21.15.90:0/3123294948 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fab9009d760 con 0x55d69ed289f0 2024-04-02T14:38:30.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.747+0000 7fab96000640 1 -- 172.21.15.90:0/3123294948 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab900cdae0 con 0x55d69ed289f0 2024-04-02T14:38:30.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.748+0000 7fabaf25c740 1 -- 172.21.15.90:0/3123294948 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d69ed289f0 msgr2=0x55d69ed242c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:38:30.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.748+0000 7fabaf25c740 1 --2- 172.21.15.90:0/3123294948 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d69ed289f0 0x55d69ed242c0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fab90099a30 tx=0x7fab90000bc0 comp rx=0 tx=0).stop 2024-04-02T14:38:30.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.748+0000 7fabaf25c740 1 -- 172.21.15.90:0/3123294948 shutdown_connections 2024-04-02T14:38:30.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.748+0000 7fabaf25c740 1 --2- 172.21.15.90:0/3123294948 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d69ed31080 0x55d69ed2e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:30.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.748+0000 7fabaf25c740 1 --2- 172.21.15.90:0/3123294948 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d69ed289f0 0x55d69ed242c0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:30.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.748+0000 7fabaf25c740 1 --2- 172.21.15.90:0/3123294948 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d69ed27d10 0x55d69ed307d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:30.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.748+0000 7fabaf25c740 1 -- 172.21.15.90:0/3123294948 >> 172.21.15.90:0/3123294948 conn(0x55d69eceea50 msgr2=0x55d69ed23bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:38:30.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.748+0000 7fabaf25c740 1 -- 172.21.15.90:0/3123294948 shutdown_connections 2024-04-02T14:38:30.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.748+0000 7fabaf25c740 1 -- 172.21.15.90:0/3123294948 wait complete. 2024-04-02T14:38:30.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.749+0000 7fabaf25c740 1 -- start start 2024-04-02T14:38:30.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.749+0000 7fabaf25c740 1 Processor -- start 2024-04-02T14:38:30.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.750+0000 7fabaf25c740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d69ed27d10 0x55d69ed573d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:30.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.750+0000 7fabaf25c740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d69ed289f0 0x55d69ed52840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:30.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.750+0000 7fabaf25c740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d69ed31080 0x55d69ed52d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:30.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.750+0000 7faba9200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d69ed289f0 0x55d69ed52840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:30.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.750+0000 7fabaf25c740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7faba41b52e0 con 0x55d69ed31080 2024-04-02T14:38:30.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.750+0000 7fabaf25c740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7faba41b4c10 con 0x55d69ed27d10 2024-04-02T14:38:30.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.750+0000 7faba9200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d69ed289f0 0x55d69ed52840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:37820/0 (socket says 172.21.15.90:37820) 2024-04-02T14:38:30.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.750+0000 7faba9200640 1 -- 172.21.15.90:0/4209127264 learned_addr learned my addr 172.21.15.90:0/4209127264 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:38:30.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.750+0000 7fabaa600640 1 --2- 172.21.15.90:0/4209127264 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d69ed31080 0x55d69ed52d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:30.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.751+0000 7faba9c00640 1 --2- 172.21.15.90:0/4209127264 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d69ed27d10 0x55d69ed573d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:30.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.751+0000 7fabaf25c740 1 -- 172.21.15.90:0/4209127264 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7faba41b1e50 con 0x55d69ed289f0 2024-04-02T14:38:30.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.751+0000 7fabaf25c740 10 client.0 _ll_register_callbacks cb 0x7fab9c003e50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:38:30.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.751+0000 7faba9c00640 1 -- 172.21.15.90:0/4209127264 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d69ed289f0 msgr2=0x55d69ed52840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:38:30.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.751+0000 7faba9c00640 1 --2- 172.21.15.90:0/4209127264 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d69ed289f0 0x55d69ed52840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:30.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.751+0000 7faba9c00640 1 -- 172.21.15.90:0/4209127264 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d69ed31080 msgr2=0x55d69ed52d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:38:30.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.751+0000 7faba9c00640 1 --2- 172.21.15.90:0/4209127264 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d69ed31080 0x55d69ed52d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:30.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.751+0000 7faba9c00640 1 -- 172.21.15.90:0/4209127264 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d69ed26d40 con 0x55d69ed27d10 2024-04-02T14:38:30.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.751+0000 7fabaa600640 1 --2- 172.21.15.90:0/4209127264 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d69ed31080 0x55d69ed52d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:38:30.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.752+0000 7faba9c00640 1 --2- 172.21.15.90:0/4209127264 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d69ed27d10 0x55d69ed573d0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fab9c048e30 tx=0x7fab9c005bd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bf2c715b00dd2697 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:38:30.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.752+0000 7fab8b400640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:38:30.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.752+0000 7fab8b400640 1 -- 172.21.15.90:0/4209127264 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab9c04c070 con 0x55d69ed27d10 2024-04-02T14:38:30.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.752+0000 7fab8b400640 1 -- 172.21.15.90:0/4209127264 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7fab9c0329a0 con 0x55d69ed27d10 2024-04-02T14:38:30.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.752+0000 7fabaf25c740 1 -- 172.21.15.90:0/4209127264 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fab9009d760 con 0x55d69ed27d10 2024-04-02T14:38:30.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.752+0000 7fab8b400640 1 -- 172.21.15.90:0/4209127264 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fab9c043050 con 0x55d69ed27d10 2024-04-02T14:38:30.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.753+0000 7fabaf25c740 10 client.26531 fetch_fsmap learned FSMap version 27 2024-04-02T14:38:30.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.753+0000 7fabaf25c740 1 -- 172.21.15.90:0/4209127264 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55d69edb7f40 con 0x55d69ed27d10 2024-04-02T14:38:30.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.753+0000 7fab8b400640 1 -- 172.21.15.90:0/4209127264 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fab9009d760 con 0x55d69ed27d10 2024-04-02T14:38:30.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.754+0000 7fab8b400640 1 -- 172.21.15.90:0/4209127264 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fab9c048450 con 0x55d69ed27d10 2024-04-02T14:38:30.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.754+0000 7fabaf25c740 10 client.26531 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:38:30.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.754+0000 7fabaf25c740 10 client.26531 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2024-04-02T14:38:30.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.754+0000 7fabaf25c740 20 client.26531 populate_metadata read hostname 'smithi090' 2024-04-02T14:38:30.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.754+0000 7fabaf25c740 1 --2- 172.21.15.90:0/4209127264 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x55d69edb85a0 0x55d69edba990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:30.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.754+0000 7fabaf25c740 4 client.26531 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-02T14:38:30.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.754+0000 7fabaf25c740 1 -- 172.21.15.90:0/4209127264 --> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55d69ec99c30 con 0x55d69edb85a0 2024-04-02T14:38:30.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.754+0000 7faba9200640 1 --2- 172.21.15.90:0/4209127264 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x55d69edb85a0 0x55d69edba990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:30.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.755+0000 7faba9200640 1 --2- 172.21.15.90:0/4209127264 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x55d69edb85a0 0x55d69edba990 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=6992c25d9de3bc10 server_cookie=353333ac943eef3b in_seq=0 out_seq=0 2024-04-02T14:38:30.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.755+0000 7fab8b400640 10 client.26531 ms_handle_connect on v2:172.21.15.90:6838/1498915870 2024-04-02T14:38:30.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.755+0000 7fab8b400640 1 -- 172.21.15.90:0/4209127264 <== mds.1 v2:172.21.15.90:6838/1498915870 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55d69ec99c30 con 0x55d69edb85a0 2024-04-02T14:38:30.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.755+0000 7fab8b400640 10 client.26531 handle_command_reply: tid=0 2024-04-02T14:38:30.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.756+0000 7fabaf25c740 1 client.26531 shutdown 2024-04-02T14:38:30.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.757+0000 7fabaf25c740 1 -- 172.21.15.90:0/4209127264 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d69ed27d10 msgr2=0x55d69ed573d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:38:30.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.757+0000 7fabaf25c740 1 --2- 172.21.15.90:0/4209127264 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d69ed27d10 0x55d69ed573d0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fab9c048e30 tx=0x7fab9c005bd0 comp rx=0 tx=0).stop 2024-04-02T14:38:30.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.757+0000 7fabaf25c740 1 -- 172.21.15.90:0/4209127264 shutdown_connections 2024-04-02T14:38:30.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.757+0000 7fabaf25c740 1 --2- 172.21.15.90:0/4209127264 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x55d69edb85a0 0x55d69edba990 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:30.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.757+0000 7fabaf25c740 1 --2- 172.21.15.90:0/4209127264 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d69ed31080 0x55d69ed52d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:30.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.757+0000 7fabaf25c740 1 --2- 172.21.15.90:0/4209127264 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d69ed289f0 0x55d69ed52840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:30.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.757+0000 7fabaf25c740 1 --2- 172.21.15.90:0/4209127264 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d69ed27d10 0x55d69ed573d0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:30.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.757+0000 7fab8b400640 0 client.26531 ms_handle_reset on v2:172.21.15.90:6838/1498915870 2024-04-02T14:38:30.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.757+0000 7fabaf25c740 1 -- 172.21.15.90:0/4209127264 >> 172.21.15.90:0/4209127264 conn(0x55d69eceea50 msgr2=0x55d69ed218a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:38:30.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.758+0000 7fabaf25c740 1 -- 172.21.15.90:0/4209127264 shutdown_connections 2024-04-02T14:38:30.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.758+0000 7fabaf25c740 1 -- 172.21.15.90:0/4209127264 wait complete. 2024-04-02T14:38:30.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.758+0000 7fabaf25c740 20 client.26531 trim_cache size 0 max 16384 2024-04-02T14:38:30.760 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-02T14:38:30.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.759+0000 7fabac600640 1 -- 172.21.15.90:0/2237013781 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fab8c0a3da0 msgr2=0x7fab8c0a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:38:30.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.759+0000 7fabac600640 1 --2- 172.21.15.90:0/2237013781 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fab8c0a3da0 0x7fab8c0a6260 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7faba406af20 tx=0x7fab98008040 comp rx=0 tx=0).stop 2024-04-02T14:38:30.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.759+0000 7fabac600640 1 -- 172.21.15.90:0/2237013781 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faba4109600 msgr2=0x7faba410aab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:38:30.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.759+0000 7fabac600640 1 --2- 172.21.15.90:0/2237013781 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faba4109600 0x7faba410aab0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fab9000d950 tx=0x7fab9000de10 comp rx=0 tx=0).stop 2024-04-02T14:38:30.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.759+0000 7fabac600640 1 -- 172.21.15.90:0/2237013781 shutdown_connections 2024-04-02T14:38:30.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.759+0000 7fabac600640 1 --2- 172.21.15.90:0/2237013781 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fab8c0a3da0 0x7fab8c0a6260 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:30.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.759+0000 7fabac600640 1 --2- 172.21.15.90:0/2237013781 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faba4114c50 0x7faba410aff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:30.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.759+0000 7fabac600640 1 --2- 172.21.15.90:0/2237013781 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faba4109600 0x7faba410aab0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:30.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.759+0000 7fabac600640 1 --2- 172.21.15.90:0/2237013781 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7faba4108ce0 0x7faba410a570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:30.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.759+0000 7fabac600640 1 -- 172.21.15.90:0/2237013781 >> 172.21.15.90:0/2237013781 conn(0x7faba406cbe0 msgr2=0x7faba4071bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:38:30.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.760+0000 7fabac600640 1 -- 172.21.15.90:0/2237013781 shutdown_connections 2024-04-02T14:38:30.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:30.760+0000 7fabac600640 1 -- 172.21.15.90:0/2237013781 wait complete. 2024-04-02T14:38:30.788 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-02T14:38:31.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.062+0000 7f62f2200640 1 -- 172.21.15.90:0/701007717 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f62ec068d00 msgr2=0x7f62ec069180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:38:31.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.062+0000 7f62f2200640 1 --2- 172.21.15.90:0/701007717 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f62ec068d00 0x7f62ec069180 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f62d8009890 tx=0x7f62d802bb70 comp rx=0 tx=0).stop 2024-04-02T14:38:31.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.063+0000 7f62f2200640 1 -- 172.21.15.90:0/701007717 shutdown_connections 2024-04-02T14:38:31.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.063+0000 7f62f2200640 1 --2- 172.21.15.90:0/701007717 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f62ec0696c0 0x7f62ec10e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:31.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.063+0000 7f62f2200640 1 --2- 172.21.15.90:0/701007717 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f62ec068d00 0x7f62ec069180 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:31.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.063+0000 7f62f2200640 1 --2- 172.21.15.90:0/701007717 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f62ec107b50 0x7f62ec107f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:31.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.063+0000 7f62f2200640 1 -- 172.21.15.90:0/701007717 >> 172.21.15.90:0/701007717 conn(0x7f62ec077b00 msgr2=0x7f62ec077f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:38:31.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.063+0000 7f62f2200640 1 -- 172.21.15.90:0/701007717 shutdown_connections 2024-04-02T14:38:31.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.063+0000 7f62f2200640 1 -- 172.21.15.90:0/701007717 wait complete. 2024-04-02T14:38:31.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.065+0000 7f62f2200640 1 Processor -- start 2024-04-02T14:38:31.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.065+0000 7f62f2200640 1 -- start start 2024-04-02T14:38:31.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.066+0000 7f62f2200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f62ec068d00 0x7f62ec198360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:31.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.066+0000 7f62f2200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f62ec0696c0 0x7f62ec1988a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:31.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.066+0000 7f62f2200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f62ec107b50 0x7f62ec19cc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:31.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.066+0000 7f62f2200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f62ec111ea0 con 0x7f62ec068d00 2024-04-02T14:38:31.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.067+0000 7f62ebe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f62ec107b50 0x7f62ec19cc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:31.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.067+0000 7f62eaa00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f62ec0696c0 0x7f62ec1988a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:31.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.067+0000 7f62f2200640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f62ec111d20 con 0x7f62ec0696c0 2024-04-02T14:38:31.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.067+0000 7f62f2200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f62ec112020 con 0x7f62ec107b50 2024-04-02T14:38:31.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.067+0000 7f62eaa00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f62ec0696c0 0x7f62ec1988a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:48072/0 (socket says 172.21.15.90:48072) 2024-04-02T14:38:31.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.067+0000 7f62eaa00640 1 -- 172.21.15.90:0/4145206345 learned_addr learned my addr 172.21.15.90:0/4145206345 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:38:31.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.067+0000 7f62eb400640 1 --2- 172.21.15.90:0/4145206345 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f62ec068d00 0x7f62ec198360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:31.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.067+0000 7f62ebe00640 1 -- 172.21.15.90:0/4145206345 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f62ec0696c0 msgr2=0x7f62ec1988a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:38:31.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.067+0000 7f62ebe00640 1 --2- 172.21.15.90:0/4145206345 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f62ec0696c0 0x7f62ec1988a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:31.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.067+0000 7f62ebe00640 1 -- 172.21.15.90:0/4145206345 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f62ec068d00 msgr2=0x7f62ec198360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:38:31.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.067+0000 7f62ebe00640 1 --2- 172.21.15.90:0/4145206345 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f62ec068d00 0x7f62ec198360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:31.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.067+0000 7f62ebe00640 1 -- 172.21.15.90:0/4145206345 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62ec19d350 con 0x7f62ec107b50 2024-04-02T14:38:31.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.067+0000 7f62eaa00640 1 --2- 172.21.15.90:0/4145206345 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f62ec0696c0 0x7f62ec1988a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:38:31.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.068+0000 7f62ebe00640 1 --2- 172.21.15.90:0/4145206345 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f62ec107b50 0x7f62ec19cc20 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f62e4009fd0 tx=0x7f62e400c6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:38:31.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.068+0000 7f62e3e00640 1 -- 172.21.15.90:0/4145206345 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62e4004b20 con 0x7f62ec107b50 2024-04-02T14:38:31.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.068+0000 7f62f2200640 1 -- 172.21.15.90:0/4145206345 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62ec19d640 con 0x7f62ec107b50 2024-04-02T14:38:31.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.068+0000 7f62f2200640 1 -- 172.21.15.90:0/4145206345 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62ec1008f0 con 0x7f62ec107b50 2024-04-02T14:38:31.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.068+0000 7f62e3e00640 1 -- 172.21.15.90:0/4145206345 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f62e4018a00 con 0x7f62ec107b50 2024-04-02T14:38:31.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.068+0000 7f62e3e00640 1 -- 172.21.15.90:0/4145206345 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62e401abd0 con 0x7f62ec107b50 2024-04-02T14:38:31.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.074+0000 7f62e3e00640 1 -- 172.21.15.90:0/4145206345 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f62e4021020 con 0x7f62ec107b50 2024-04-02T14:38:31.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.076+0000 7f62f502a740 1 Processor -- start 2024-04-02T14:38:31.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.078+0000 7f62f502a740 1 -- start start 2024-04-02T14:38:31.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.078+0000 7f62f502a740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f4d9705200 0x55f4d9703f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:31.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.078+0000 7f62f502a740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f4d9704520 0x55f4d97049a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:31.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.078+0000 7f62f502a740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f4d970d890 0x55f4d970fc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:31.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.078+0000 7f62f502a740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55f4d963c400 con 0x55f4d9705200 2024-04-02T14:38:31.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.080+0000 7f62f502a740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55f4d9620f10 con 0x55f4d9704520 2024-04-02T14:38:31.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.080+0000 7f62f502a740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55f4d96606f0 con 0x55f4d970d890 2024-04-02T14:38:31.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.080+0000 7f62e3e00640 1 --2- 172.21.15.90:0/4145206345 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f62d00a3e20 0x7f62d00a62e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:31.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.081+0000 7f62e3e00640 1 -- 172.21.15.90:0/4145206345 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f62e40d46c0 con 0x7f62ec107b50 2024-04-02T14:38:31.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.081+0000 7f62ebe00640 1 --2- 172.21.15.90:0/4145206345 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f62d00a3e20 0x7f62d00a62e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:31.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.083+0000 7f62eb400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f4d9704520 0x55f4d97049a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:31.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.083+0000 7f62ebe00640 1 --2- 172.21.15.90:0/4145206345 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f62d00a3e20 0x7f62d00a62e0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f62e40a8d50 tx=0x7f62e4005f70 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:38:31.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.084+0000 7f62eaa00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f4d970d890 0x55f4d970fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:31.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.084+0000 7f62eaa00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f4d970d890 0x55f4d970fc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:37854/0 (socket says 172.21.15.90:37854) 2024-04-02T14:38:31.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.085+0000 7f62eaa00640 1 -- 172.21.15.90:0/2364826514 learned_addr learned my addr 172.21.15.90:0/2364826514 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:38:31.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.084+0000 7f62eb400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f4d9704520 0x55f4d97049a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:48084/0 (socket says 172.21.15.90:48084) 2024-04-02T14:38:31.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.085+0000 7f62eaa00640 1 --2- 172.21.15.90:0/2364826514 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f4d9705200 0x55f4d9703f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:31.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.086+0000 7f62eaa00640 1 -- 172.21.15.90:0/2364826514 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f4d9704520 msgr2=0x55f4d97049a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:38:31.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.087+0000 7f62eaa00640 1 --2- 172.21.15.90:0/2364826514 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f4d9704520 0x55f4d97049a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:31.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.087+0000 7f62eaa00640 1 -- 172.21.15.90:0/2364826514 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f4d9705200 msgr2=0x55f4d9703f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:38:31.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.087+0000 7f62eaa00640 1 --2- 172.21.15.90:0/2364826514 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f4d9705200 0x55f4d9703f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:31.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.087+0000 7f62eaa00640 1 -- 172.21.15.90:0/2364826514 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f4d93743f0 con 0x55f4d970d890 2024-04-02T14:38:31.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.087+0000 7f62eb400640 1 --2- 172.21.15.90:0/2364826514 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f4d9704520 0x55f4d97049a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:38:31.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.088+0000 7f62eaa00640 1 --2- 172.21.15.90:0/2364826514 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f4d970d890 0x55f4d970fc80 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f62d8032570 tx=0x7f62d8005ab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6e41c242056ba66b server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:38:31.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.089+0000 7f62d7e00640 1 -- 172.21.15.90:0/2364826514 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62d8048070 con 0x55f4d970d890 2024-04-02T14:38:31.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.089+0000 7f62d7e00640 1 -- 172.21.15.90:0/2364826514 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f62d803a040 con 0x55f4d970d890 2024-04-02T14:38:31.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.089+0000 7f62d7e00640 1 -- 172.21.15.90:0/2364826514 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62d8039070 con 0x55f4d970d890 2024-04-02T14:38:31.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.089+0000 7f62f502a740 1 -- 172.21.15.90:0/2364826514 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f4d970d890 msgr2=0x55f4d970fc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:38:31.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.089+0000 7f62f502a740 1 --2- 172.21.15.90:0/2364826514 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f4d970d890 0x55f4d970fc80 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f62d8032570 tx=0x7f62d8005ab0 comp rx=0 tx=0).stop 2024-04-02T14:38:31.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.090+0000 7f62f502a740 1 -- 172.21.15.90:0/2364826514 shutdown_connections 2024-04-02T14:38:31.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.090+0000 7f62f502a740 1 --2- 172.21.15.90:0/2364826514 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f4d970d890 0x55f4d970fc80 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:31.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.090+0000 7f62f502a740 1 --2- 172.21.15.90:0/2364826514 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f4d9704520 0x55f4d97049a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:31.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.090+0000 7f62f502a740 1 --2- 172.21.15.90:0/2364826514 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f4d9705200 0x55f4d9703f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:31.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.090+0000 7f62f502a740 1 -- 172.21.15.90:0/2364826514 >> 172.21.15.90:0/2364826514 conn(0x55f4d96c2a50 msgr2=0x55f4d96fef70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:38:31.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.091+0000 7f62f502a740 1 -- 172.21.15.90:0/2364826514 shutdown_connections 2024-04-02T14:38:31.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.091+0000 7f62f502a740 1 -- 172.21.15.90:0/2364826514 wait complete. 2024-04-02T14:38:31.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.092+0000 7f62f502a740 1 -- start start 2024-04-02T14:38:31.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.093+0000 7f62f502a740 1 Processor -- start 2024-04-02T14:38:31.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.094+0000 7f62f502a740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f4d9704520 0x55f4d970a860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:31.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.094+0000 7f62f502a740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f4d9705200 0x55f4d970ada0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:31.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.094+0000 7f62f502a740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f4d970d890 0x55f4d970b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:31.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.094+0000 7f62f502a740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55f4d965c6a0 con 0x55f4d970d890 2024-04-02T14:38:31.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.094+0000 7f62f502a740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55f4d9608980 con 0x55f4d9704520 2024-04-02T14:38:31.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.094+0000 7f62f502a740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55f4d961bcb0 con 0x55f4d9705200 2024-04-02T14:38:31.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.094+0000 7f62f502a740 10 client.0 _ll_register_callbacks cb 0x55f4d961b6a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:38:31.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.095+0000 7f62eaa00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f4d9704520 0x55f4d970a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:31.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.095+0000 7f62eaa00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f4d9704520 0x55f4d970a860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:48088/0 (socket says 172.21.15.90:48088) 2024-04-02T14:38:31.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.095+0000 7f62eaa00640 1 -- 172.21.15.90:0/1625838407 learned_addr learned my addr 172.21.15.90:0/1625838407 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:38:31.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.096+0000 7f62eaa00640 1 --2- 172.21.15.90:0/1625838407 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f4d970d890 0x55f4d970b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:31.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.096+0000 7f62eaa00640 1 -- 172.21.15.90:0/1625838407 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f4d9705200 msgr2=0x55f4d970ada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:38:31.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.096+0000 7f62eaa00640 1 --2- 172.21.15.90:0/1625838407 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f4d9705200 0x55f4d970ada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:31.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.096+0000 7f62eaa00640 1 -- 172.21.15.90:0/1625838407 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f4d970d890 msgr2=0x55f4d970b2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:38:31.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.096+0000 7f62eaa00640 1 --2- 172.21.15.90:0/1625838407 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f4d970d890 0x55f4d970b2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:31.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.096+0000 7f62eaa00640 1 -- 172.21.15.90:0/1625838407 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f4d943e7d0 con 0x55f4d9704520 2024-04-02T14:38:31.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.097+0000 7f62d4c00640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:38:31.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.097+0000 7f62eaa00640 1 --2- 172.21.15.90:0/1625838407 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f4d9704520 0x55f4d970a860 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f62d8034a70 tx=0x7f62d802c510 comp rx=0 tx=0).ready entity=mon.2 client_cookie=868d64883f52fcdf server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:38:31.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.097+0000 7f62d4c00640 1 -- 172.21.15.90:0/1625838407 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62d8048040 con 0x55f4d9704520 2024-04-02T14:38:31.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.097+0000 7f62d4c00640 1 -- 172.21.15.90:0/1625838407 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f62d8005220 con 0x55f4d9704520 2024-04-02T14:38:31.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.097+0000 7f62f502a740 1 -- 172.21.15.90:0/1625838407 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f62d803a040 con 0x55f4d9704520 2024-04-02T14:38:31.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.100+0000 7f62d4c00640 1 -- 172.21.15.90:0/1625838407 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f62d8039040 con 0x55f4d9704520 2024-04-02T14:38:31.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.100+0000 7f62f502a740 10 client.26543 fetch_fsmap learned FSMap version 27 2024-04-02T14:38:31.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.100+0000 7f62f502a740 1 -- 172.21.15.90:0/1625838407 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55f4d972d750 con 0x55f4d9704520 2024-04-02T14:38:31.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.100+0000 7f62d4c00640 1 -- 172.21.15.90:0/1625838407 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f62d803a040 con 0x55f4d9704520 2024-04-02T14:38:31.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.101+0000 7f62d4c00640 1 -- 172.21.15.90:0/1625838407 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f62d8004990 con 0x55f4d9704520 2024-04-02T14:38:31.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.101+0000 7f62f502a740 10 client.26543 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:38:31.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.101+0000 7f62f502a740 10 client.26543 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:38:31.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.101+0000 7f62f502a740 20 client.26543 populate_metadata read hostname 'smithi090' 2024-04-02T14:38:31.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.101+0000 7f62f502a740 1 --2- 172.21.15.90:0/1625838407 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55f4d972eaa0 0x55f4d9730e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:31.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.101+0000 7f62f502a740 4 client.26543 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:38:31.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.101+0000 7f62f502a740 1 -- 172.21.15.90:0/1625838407 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f4d9620040 con 0x55f4d972eaa0 2024-04-02T14:38:31.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.101+0000 7f62eb400640 1 --2- 172.21.15.90:0/1625838407 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55f4d972eaa0 0x55f4d9730e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:31.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.102+0000 7f62d4c00640 10 client.26543 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:38:31.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.104+0000 7f62eb400640 1 --2- 172.21.15.90:0/1625838407 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55f4d972eaa0 0x55f4d9730e90 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1507299b3e5cf0ac server_cookie=a07fb017bc09ee0 in_seq=0 out_seq=0 2024-04-02T14:38:31.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.105+0000 7f62d4c00640 1 -- 172.21.15.90:0/1625838407 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f4d9620040 con 0x55f4d972eaa0 2024-04-02T14:38:31.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.105+0000 7f62d4c00640 10 client.26543 handle_command_reply: tid=0 2024-04-02T14:38:31.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.106+0000 7f62f502a740 1 client.26543 shutdown 2024-04-02T14:38:31.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.107+0000 7f62f502a740 1 -- 172.21.15.90:0/1625838407 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f4d9704520 msgr2=0x55f4d970a860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:38:31.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.107+0000 7f62f502a740 1 --2- 172.21.15.90:0/1625838407 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f4d9704520 0x55f4d970a860 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f62d8034a70 tx=0x7f62d802c510 comp rx=0 tx=0).stop 2024-04-02T14:38:31.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.107+0000 7f62f502a740 1 -- 172.21.15.90:0/1625838407 shutdown_connections 2024-04-02T14:38:31.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.107+0000 7f62f502a740 1 --2- 172.21.15.90:0/1625838407 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55f4d972eaa0 0x55f4d9730e90 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:31.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.107+0000 7f62f502a740 1 --2- 172.21.15.90:0/1625838407 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f4d970d890 0x55f4d970b2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:31.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.107+0000 7f62f502a740 1 --2- 172.21.15.90:0/1625838407 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f4d9705200 0x55f4d970ada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:31.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.107+0000 7f62f502a740 1 --2- 172.21.15.90:0/1625838407 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f4d9704520 0x55f4d970a860 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:31.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.107+0000 7f62f502a740 1 -- 172.21.15.90:0/1625838407 >> 172.21.15.90:0/1625838407 conn(0x55f4d96c2a50 msgr2=0x55f4d9700cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:38:31.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.108+0000 7f62d4c00640 0 client.26543 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:38:31.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.108+0000 7f62f502a740 1 -- 172.21.15.90:0/1625838407 shutdown_connections 2024-04-02T14:38:31.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.108+0000 7f62f502a740 1 -- 172.21.15.90:0/1625838407 wait complete. 2024-04-02T14:38:31.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.108+0000 7f62f502a740 20 client.26543 trim_cache size 0 max 16384 2024-04-02T14:38:31.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.135+0000 7f62f502a740 1 Processor -- start 2024-04-02T14:38:31.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.135+0000 7f62f502a740 1 -- start start 2024-04-02T14:38:31.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.136+0000 7f62f502a740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f4d9704520 0x55f4d970cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:31.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.136+0000 7f62f502a740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f4d9705200 0x55f4d9705b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:31.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.136+0000 7f62f502a740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f4d970d890 0x55f4d97099c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:31.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.136+0000 7f62f502a740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f62cc00e810 con 0x55f4d9704520 2024-04-02T14:38:31.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.136+0000 7f62f502a740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f62cc010350 con 0x55f4d9705200 2024-04-02T14:38:31.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.136+0000 7f62f502a740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f62cc00eee0 con 0x55f4d970d890 2024-04-02T14:38:31.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.136+0000 7f62eb400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f4d9705200 0x55f4d9705b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:31.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.136+0000 7f62eb400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f4d9705200 0x55f4d9705b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:48092/0 (socket says 172.21.15.90:48092) 2024-04-02T14:38:31.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.136+0000 7f62eb400640 1 -- 172.21.15.90:0/1674650822 learned_addr learned my addr 172.21.15.90:0/1674650822 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:38:31.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.137+0000 7f62eb400640 1 -- 172.21.15.90:0/1674650822 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f4d970d890 msgr2=0x55f4d97099c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-02T14:38:31.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.137+0000 7f62eb400640 1 --2- 172.21.15.90:0/1674650822 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f4d970d890 0x55f4d97099c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:31.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.137+0000 7f62eb400640 1 -- 172.21.15.90:0/1674650822 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f4d9704520 msgr2=0x55f4d970cf90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-02T14:38:31.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.137+0000 7f62eb400640 1 --2- 172.21.15.90:0/1674650822 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f4d9704520 0x55f4d970cf90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:31.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.137+0000 7f62eb400640 1 -- 172.21.15.90:0/1674650822 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f4d9703550 con 0x55f4d9705200 2024-04-02T14:38:31.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.137+0000 7f62eb400640 1 --2- 172.21.15.90:0/1674650822 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f4d9705200 0x55f4d9705b60 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x55f4d9705750 tx=0x7f62dc014840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cc64352adc2fc193 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:38:31.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.138+0000 7f62d7400640 1 -- 172.21.15.90:0/1674650822 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62dc014110 con 0x55f4d9705200 2024-04-02T14:38:31.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.138+0000 7f62d7400640 1 -- 172.21.15.90:0/1674650822 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f62dc0142b0 con 0x55f4d9705200 2024-04-02T14:38:31.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.138+0000 7f62d7400640 1 -- 172.21.15.90:0/1674650822 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62dc011440 con 0x55f4d9705200 2024-04-02T14:38:31.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.139+0000 7f62f502a740 1 -- 172.21.15.90:0/1674650822 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f4d9705200 msgr2=0x55f4d9705b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:38:31.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.139+0000 7f62f502a740 1 --2- 172.21.15.90:0/1674650822 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f4d9705200 0x55f4d9705b60 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x55f4d9705750 tx=0x7f62dc014840 comp rx=0 tx=0).stop 2024-04-02T14:38:31.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.140+0000 7f62f502a740 1 -- 172.21.15.90:0/1674650822 shutdown_connections 2024-04-02T14:38:31.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.140+0000 7f62f502a740 1 --2- 172.21.15.90:0/1674650822 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f4d970d890 0x55f4d97099c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:31.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.140+0000 7f62f502a740 1 --2- 172.21.15.90:0/1674650822 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f4d9705200 0x55f4d9705b60 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:31.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.140+0000 7f62f502a740 1 --2- 172.21.15.90:0/1674650822 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f4d9704520 0x55f4d970cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:31.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.140+0000 7f62f502a740 1 -- 172.21.15.90:0/1674650822 >> 172.21.15.90:0/1674650822 conn(0x55f4d96c2a50 msgr2=0x55f4d96f5010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:38:31.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.141+0000 7f62f502a740 1 -- 172.21.15.90:0/1674650822 shutdown_connections 2024-04-02T14:38:31.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.141+0000 7f62f502a740 1 -- 172.21.15.90:0/1674650822 wait complete. 2024-04-02T14:38:31.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.142+0000 7f62f502a740 1 -- start start 2024-04-02T14:38:31.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.143+0000 7f62f502a740 1 Processor -- start 2024-04-02T14:38:31.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.144+0000 7f62f502a740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f4d9704520 0x55f4d9733ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:31.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.144+0000 7f62f502a740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f4d9705200 0x55f4d9734020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:31.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.144+0000 7f62f502a740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f4d970d890 0x55f4d9706fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:31.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.144+0000 7f62f502a740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f62cc00fc80 con 0x55f4d970d890 2024-04-02T14:38:31.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.144+0000 7f62f502a740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f62cc00f5b0 con 0x55f4d9704520 2024-04-02T14:38:31.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.144+0000 7f62f502a740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f62cc00c7f0 con 0x55f4d9705200 2024-04-02T14:38:31.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.144+0000 7f62f502a740 10 client.0 _ll_register_callbacks cb 0x7f62dc005ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:38:31.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.145+0000 7f62eaa00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f4d9704520 0x55f4d9733ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:31.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.145+0000 7f62eaa00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f4d9704520 0x55f4d9733ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:48106/0 (socket says 172.21.15.90:48106) 2024-04-02T14:38:31.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.145+0000 7f62eaa00640 1 -- 172.21.15.90:0/3505230149 learned_addr learned my addr 172.21.15.90:0/3505230149 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:38:31.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.145+0000 7f62eaa00640 1 --2- 172.21.15.90:0/3505230149 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f4d970d890 0x55f4d9706fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:31.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.145+0000 7f62eb400640 1 --2- 172.21.15.90:0/3505230149 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f4d9705200 0x55f4d9734020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:31.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.146+0000 7f62eaa00640 1 -- 172.21.15.90:0/3505230149 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f4d9705200 msgr2=0x55f4d9734020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:38:31.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.146+0000 7f62eaa00640 1 --2- 172.21.15.90:0/3505230149 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f4d9705200 0x55f4d9734020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:31.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.146+0000 7f62eaa00640 1 -- 172.21.15.90:0/3505230149 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f4d970d890 msgr2=0x55f4d9706fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:38:31.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.146+0000 7f62eaa00640 1 --2- 172.21.15.90:0/3505230149 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f4d970d890 0x55f4d9706fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:31.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.146+0000 7f62eaa00640 1 -- 172.21.15.90:0/3505230149 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f4d97077d0 con 0x55f4d9704520 2024-04-02T14:38:31.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.146+0000 7f62eb400640 1 --2- 172.21.15.90:0/3505230149 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f4d9705200 0x55f4d9734020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:38:31.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.146+0000 7f62d4c00640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:38:31.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.146+0000 7f62eaa00640 1 --2- 172.21.15.90:0/3505230149 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f4d9704520 0x55f4d9733ae0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f62d8034a40 tx=0x7f62d8005970 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3fe138c6b7290359 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:38:31.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.147+0000 7f62d4c00640 1 -- 172.21.15.90:0/3505230149 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62d8063060 con 0x55f4d9704520 2024-04-02T14:38:31.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.147+0000 7f62d4c00640 1 -- 172.21.15.90:0/3505230149 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f62d8032640 con 0x55f4d9704520 2024-04-02T14:38:31.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.148+0000 7f62f502a740 1 -- 172.21.15.90:0/3505230149 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55f4d966dc30 con 0x55f4d9704520 2024-04-02T14:38:31.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.148+0000 7f62d4c00640 1 -- 172.21.15.90:0/3505230149 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f62d8048070 con 0x55f4d9704520 2024-04-02T14:38:31.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.148+0000 7f62f502a740 10 client.26549 fetch_fsmap learned FSMap version 27 2024-04-02T14:38:31.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.148+0000 7f62f502a740 1 -- 172.21.15.90:0/3505230149 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55f4d97947f0 con 0x55f4d9704520 2024-04-02T14:38:31.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.148+0000 7f62d4c00640 1 -- 172.21.15.90:0/3505230149 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55f4d966dc30 con 0x55f4d9704520 2024-04-02T14:38:31.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.148+0000 7f62d4c00640 1 -- 172.21.15.90:0/3505230149 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f62d802c580 con 0x55f4d9704520 2024-04-02T14:38:31.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.149+0000 7f62f502a740 10 client.26549 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:38:31.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.149+0000 7f62f502a740 10 client.26549 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:38:31.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.149+0000 7f62f502a740 20 client.26549 populate_metadata read hostname 'smithi090' 2024-04-02T14:38:31.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.149+0000 7f62f502a740 1 --2- 172.21.15.90:0/3505230149 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55f4d9794f40 0x55f4d9797360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:31.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.149+0000 7f62f502a740 4 client.26549 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-02T14:38:31.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.149+0000 7f62f502a740 1 -- 172.21.15.90:0/3505230149 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55f4d9667f80 con 0x55f4d9794f40 2024-04-02T14:38:31.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.149+0000 7f62eb400640 1 --2- 172.21.15.90:0/3505230149 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55f4d9794f40 0x55f4d9797360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:31.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.150+0000 7f62eb400640 1 --2- 172.21.15.90:0/3505230149 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55f4d9794f40 0x55f4d9797360 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f46dc8d2d583d2da server_cookie=e9cb0c93fc2520bb in_seq=0 out_seq=0 2024-04-02T14:38:31.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.150+0000 7f62d4c00640 10 client.26549 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:38:31.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.150+0000 7f62d4c00640 1 -- 172.21.15.90:0/3505230149 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55f4d9667f80 con 0x55f4d9794f40 2024-04-02T14:38:31.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.150+0000 7f62d4c00640 10 client.26549 handle_command_reply: tid=0 2024-04-02T14:38:31.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.150+0000 7f62f502a740 1 client.26549 shutdown 2024-04-02T14:38:31.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.151+0000 7f62f502a740 1 -- 172.21.15.90:0/3505230149 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f4d9704520 msgr2=0x55f4d9733ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:38:31.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.151+0000 7f62f502a740 1 --2- 172.21.15.90:0/3505230149 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f4d9704520 0x55f4d9733ae0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f62d8034a40 tx=0x7f62d8005970 comp rx=0 tx=0).stop 2024-04-02T14:38:31.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.151+0000 7f62f502a740 1 -- 172.21.15.90:0/3505230149 shutdown_connections 2024-04-02T14:38:31.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.151+0000 7f62f502a740 1 --2- 172.21.15.90:0/3505230149 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55f4d9794f40 0x55f4d9797360 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:31.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.151+0000 7f62f502a740 1 --2- 172.21.15.90:0/3505230149 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f4d970d890 0x55f4d9706fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:31.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.151+0000 7f62f502a740 1 --2- 172.21.15.90:0/3505230149 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f4d9705200 0x55f4d9734020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:31.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.151+0000 7f62f502a740 1 --2- 172.21.15.90:0/3505230149 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f4d9704520 0x55f4d9733ae0 secure :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f62d8034a40 tx=0x7f62d8005970 comp rx=0 tx=0).stop 2024-04-02T14:38:31.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.151+0000 7f62f502a740 1 -- 172.21.15.90:0/3505230149 >> 172.21.15.90:0/3505230149 conn(0x55f4d96c2a50 msgr2=0x55f4d96f5010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:38:31.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.151+0000 7f62d4c00640 0 client.26549 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:38:31.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.151+0000 7f62f502a740 1 -- 172.21.15.90:0/3505230149 shutdown_connections 2024-04-02T14:38:31.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.151+0000 7f62f502a740 1 -- 172.21.15.90:0/3505230149 wait complete. 2024-04-02T14:38:31.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.152+0000 7f62f502a740 20 client.26549 trim_cache size 0 max 16384 2024-04-02T14:38:31.154 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-02T14:38:31.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.154+0000 7f62f2200640 1 -- 172.21.15.90:0/4145206345 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f62d00a3e20 msgr2=0x7f62d00a62e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:38:31.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.154+0000 7f62f2200640 1 --2- 172.21.15.90:0/4145206345 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f62d00a3e20 0x7f62d00a62e0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f62e40a8d50 tx=0x7f62e4005f70 comp rx=0 tx=0).stop 2024-04-02T14:38:31.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.154+0000 7f62f2200640 1 -- 172.21.15.90:0/4145206345 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f62ec107b50 msgr2=0x7f62ec19cc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:38:31.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.154+0000 7f62f2200640 1 --2- 172.21.15.90:0/4145206345 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f62ec107b50 0x7f62ec19cc20 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f62e4009fd0 tx=0x7f62e400c6a0 comp rx=0 tx=0).stop 2024-04-02T14:38:31.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.154+0000 7f62f2200640 1 -- 172.21.15.90:0/4145206345 shutdown_connections 2024-04-02T14:38:31.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.155+0000 7f62f2200640 1 --2- 172.21.15.90:0/4145206345 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f62d00a3e20 0x7f62d00a62e0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:31.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.155+0000 7f62f2200640 1 --2- 172.21.15.90:0/4145206345 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f62ec107b50 0x7f62ec19cc20 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:31.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.155+0000 7f62f2200640 1 --2- 172.21.15.90:0/4145206345 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f62ec0696c0 0x7f62ec1988a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:31.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.155+0000 7f62f2200640 1 --2- 172.21.15.90:0/4145206345 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f62ec068d00 0x7f62ec198360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:31.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.155+0000 7f62f2200640 1 -- 172.21.15.90:0/4145206345 >> 172.21.15.90:0/4145206345 conn(0x7f62ec077b00 msgr2=0x7f62ec0fe4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:38:31.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.155+0000 7f62f2200640 1 -- 172.21.15.90:0/4145206345 shutdown_connections 2024-04-02T14:38:31.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:31.155+0000 7f62f2200640 1 -- 172.21.15.90:0/4145206345 wait complete. 2024-04-02T14:38:31.831 INFO:teuthology.orchestra.run.smithi090.stderr:progress: 180.0 s, 2867.3 tps, lat 35.061 ms stddev 38.928 2024-04-02T14:38:32.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:38:31 smithi092 ceph-mon[36497]: pgmap v112: 97 pgs: 97 active+clean; 2.2 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 20 MiB/s wr, 1.07k op/s 2024-04-02T14:38:32.181 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-02T14:38:32.181 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T14:38:32.182 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-02T14:38:32.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:38:31 smithi177 ceph-mon[36431]: pgmap v112: 97 pgs: 97 active+clean; 2.2 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 20 MiB/s wr, 1.07k op/s 2024-04-02T14:38:32.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:38:31 smithi090 ceph-mon[31674]: pgmap v112: 97 pgs: 97 active+clean; 2.2 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 20 MiB/s wr, 1.07k op/s 2024-04-02T14:38:32.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.454+0000 7f69a8400640 1 -- 172.21.15.90:0/1622288164 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f69a010abe0 msgr2=0x7f69a010cfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:38:32.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.454+0000 7f69a8400640 1 --2- 172.21.15.90:0/1622288164 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f69a010abe0 0x7f69a010cfd0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f6998005fd0 tx=0x7f6998030190 comp rx=0 tx=0).stop 2024-04-02T14:38:32.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.455+0000 7f69a8400640 1 -- 172.21.15.90:0/1622288164 shutdown_connections 2024-04-02T14:38:32.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.455+0000 7f69a8400640 1 --2- 172.21.15.90:0/1622288164 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f69a010abe0 0x7f69a010cfd0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:32.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.455+0000 7f69a8400640 1 --2- 172.21.15.90:0/1622288164 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f69a0100870 0x7f69a0104e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:32.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.455+0000 7f69a8400640 1 --2- 172.21.15.90:0/1622288164 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f69a0068060 0x7f69a0104940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:32.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.455+0000 7f69a8400640 1 -- 172.21.15.90:0/1622288164 >> 172.21.15.90:0/1622288164 conn(0x7f69a0076b50 msgr2=0x7f69a00fdee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:38:32.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.455+0000 7f69a8400640 1 -- 172.21.15.90:0/1622288164 shutdown_connections 2024-04-02T14:38:32.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.455+0000 7f69a8400640 1 -- 172.21.15.90:0/1622288164 wait complete. 2024-04-02T14:38:32.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.457+0000 7f69a8400640 1 Processor -- start 2024-04-02T14:38:32.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.457+0000 7f69a8400640 1 -- start start 2024-04-02T14:38:32.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.457+0000 7f69a8400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f69a0068060 0x7f69a00733d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:32.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.457+0000 7f69a8400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f69a0100870 0x7f69a006c450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:32.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.457+0000 7f69a8400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f69a010abe0 0x7f69a006c990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:32.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.457+0000 7f69a8400640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f69a010fc70 con 0x7f69a0068060 2024-04-02T14:38:32.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.458+0000 7f69a8400640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f69a010faf0 con 0x7f69a010abe0 2024-04-02T14:38:32.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.458+0000 7f69a8400640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f69a010fdf0 con 0x7f69a0100870 2024-04-02T14:38:32.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.458+0000 7f69a6400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f69a010abe0 0x7f69a006c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:32.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.458+0000 7f69a6400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f69a010abe0 0x7f69a006c990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:48124/0 (socket says 172.21.15.90:48124) 2024-04-02T14:38:32.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.458+0000 7f69a6400640 1 -- 172.21.15.90:0/2880201341 learned_addr learned my addr 172.21.15.90:0/2880201341 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:38:32.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.458+0000 7f69a5000640 1 --2- 172.21.15.90:0/2880201341 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f69a0100870 0x7f69a006c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:32.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.458+0000 7f69a5a00640 1 --2- 172.21.15.90:0/2880201341 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f69a0068060 0x7f69a00733d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:32.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.458+0000 7f69a6400640 1 -- 172.21.15.90:0/2880201341 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f69a0100870 msgr2=0x7f69a006c450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:38:32.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.458+0000 7f69a6400640 1 --2- 172.21.15.90:0/2880201341 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f69a0100870 0x7f69a006c450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:32.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.458+0000 7f69a6400640 1 -- 172.21.15.90:0/2880201341 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f69a0068060 msgr2=0x7f69a00733d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:38:32.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.458+0000 7f69a6400640 1 --2- 172.21.15.90:0/2880201341 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f69a0068060 0x7f69a00733d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:32.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.458+0000 7f69a6400640 1 -- 172.21.15.90:0/2880201341 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69a006d030 con 0x7f69a010abe0 2024-04-02T14:38:32.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.459+0000 7f69a6400640 1 --2- 172.21.15.90:0/2880201341 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f69a010abe0 0x7f69a006c990 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f699803a040 tx=0x7f6998037c30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:38:32.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.459+0000 7f699e000640 1 -- 172.21.15.90:0/2880201341 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6998047020 con 0x7f69a010abe0 2024-04-02T14:38:32.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.459+0000 7f699e000640 1 -- 172.21.15.90:0/2880201341 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6998038640 con 0x7f69a010abe0 2024-04-02T14:38:32.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.459+0000 7f699e000640 1 -- 172.21.15.90:0/2880201341 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6998043660 con 0x7f69a010abe0 2024-04-02T14:38:32.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.459+0000 7f69a8400640 1 -- 172.21.15.90:0/2880201341 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f69a0070ef0 con 0x7f69a010abe0 2024-04-02T14:38:32.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.459+0000 7f69a8400640 1 -- 172.21.15.90:0/2880201341 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69a00713f0 con 0x7f69a010abe0 2024-04-02T14:38:32.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.464+0000 7f699e000640 1 -- 172.21.15.90:0/2880201341 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f699804a020 con 0x7f69a010abe0 2024-04-02T14:38:32.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.464+0000 7f69ab1aa740 1 Processor -- start 2024-04-02T14:38:32.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.464+0000 7f69ab1aa740 1 -- start start 2024-04-02T14:38:32.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.465+0000 7f69ab1aa740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5615400d3840 0x5615400d2590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:32.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.465+0000 7f69ab1aa740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5615400d2b60 0x5615400d2fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:32.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.465+0000 7f69ab1aa740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5615400dbed0 0x5615400de2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:32.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.465+0000 7f69ab1aa740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x56154002e310 con 0x5615400dbed0 2024-04-02T14:38:32.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.465+0000 7f69ab1aa740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x56153ffd6690 con 0x5615400d3840 2024-04-02T14:38:32.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.465+0000 7f69ab1aa740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x56154002a2c0 con 0x5615400d2b60 2024-04-02T14:38:32.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.465+0000 7f699e000640 1 --2- 172.21.15.90:0/2880201341 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f69880a3d20 0x7f69880a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:32.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.465+0000 7f69a5000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5615400d3840 0x5615400d2590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:32.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.465+0000 7f699e000640 1 -- 172.21.15.90:0/2880201341 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f69980f7620 con 0x7f69a010abe0 2024-04-02T14:38:32.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.465+0000 7f69a5000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5615400d3840 0x5615400d2590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:48126/0 (socket says 172.21.15.90:48126) 2024-04-02T14:38:32.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.465+0000 7f69a5000640 1 -- 172.21.15.90:0/777376650 learned_addr learned my addr 172.21.15.90:0/777376650 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:38:32.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.466+0000 7f69a5000640 1 --2- 172.21.15.90:0/777376650 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5615400dbed0 0x5615400de2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:32.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.466+0000 7f69a5a00640 1 --2- 172.21.15.90:0/777376650 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5615400d2b60 0x5615400d2fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:32.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.466+0000 7f69a6400640 1 --2- 172.21.15.90:0/2880201341 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f69880a3d20 0x7f69880a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:32.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.466+0000 7f69a5a00640 1 -- 172.21.15.90:0/777376650 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5615400d3840 msgr2=0x5615400d2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:38:32.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.466+0000 7f69a5a00640 1 --2- 172.21.15.90:0/777376650 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5615400d3840 0x5615400d2590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:32.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.466+0000 7f69a5a00640 1 -- 172.21.15.90:0/777376650 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5615400dbed0 msgr2=0x5615400de2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:38:32.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.466+0000 7f69a5a00640 1 --2- 172.21.15.90:0/777376650 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5615400dbed0 0x5615400de2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:32.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.466+0000 7f69a5a00640 1 -- 172.21.15.90:0/777376650 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56153fe0a7e0 con 0x5615400d2b60 2024-04-02T14:38:32.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.467+0000 7f69a5a00640 1 --2- 172.21.15.90:0/777376650 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5615400d2b60 0x5615400d2fe0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f699000d970 tx=0x7f699000de30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8e3007cd2de285a6 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:38:32.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.467+0000 7f69a6400640 1 --2- 172.21.15.90:0/2880201341 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f69880a3d20 0x7f69880a61e0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f69980c00b0 tx=0x7f6998030190 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:38:32.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.467+0000 7f698ea00640 1 -- 172.21.15.90:0/777376650 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f699001c070 con 0x5615400d2b60 2024-04-02T14:38:32.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.467+0000 7f698ea00640 1 -- 172.21.15.90:0/777376650 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6990009d00 con 0x5615400d2b60 2024-04-02T14:38:32.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.467+0000 7f698ea00640 1 -- 172.21.15.90:0/777376650 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6990016710 con 0x5615400d2b60 2024-04-02T14:38:32.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.468+0000 7f69ab1aa740 1 -- 172.21.15.90:0/777376650 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5615400d2b60 msgr2=0x5615400d2fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:38:32.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.468+0000 7f69ab1aa740 1 --2- 172.21.15.90:0/777376650 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5615400d2b60 0x5615400d2fe0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f699000d970 tx=0x7f699000de30 comp rx=0 tx=0).stop 2024-04-02T14:38:32.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.469+0000 7f69ab1aa740 1 -- 172.21.15.90:0/777376650 shutdown_connections 2024-04-02T14:38:32.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.469+0000 7f69ab1aa740 1 --2- 172.21.15.90:0/777376650 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5615400dbed0 0x5615400de2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:32.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.469+0000 7f69ab1aa740 1 --2- 172.21.15.90:0/777376650 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5615400d2b60 0x5615400d2fe0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:32.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.469+0000 7f69ab1aa740 1 --2- 172.21.15.90:0/777376650 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5615400d3840 0x5615400d2590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:32.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.469+0000 7f69ab1aa740 1 -- 172.21.15.90:0/777376650 >> 172.21.15.90:0/777376650 conn(0x56154005e270 msgr2=0x5615400cd5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:38:32.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.469+0000 7f69ab1aa740 1 -- 172.21.15.90:0/777376650 shutdown_connections 2024-04-02T14:38:32.473 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.470+0000 7f69ab1aa740 1 -- 172.21.15.90:0/777376650 wait complete. 2024-04-02T14:38:32.473 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.470+0000 7f69ab1aa740 1 -- start start 2024-04-02T14:38:32.473 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.470+0000 7f69ab1aa740 1 Processor -- start 2024-04-02T14:38:32.473 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.471+0000 7f69ab1aa740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5615400d2b60 0x5615400d8e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:32.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.471+0000 7f69ab1aa740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5615400d3840 0x5615400d9380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:32.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.471+0000 7f69ab1aa740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5615400dbed0 0x5615400d98c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:32.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.471+0000 7f69ab1aa740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5615400ded40 con 0x5615400dbed0 2024-04-02T14:38:32.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.471+0000 7f69ab1aa740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x56153ffe9a60 con 0x5615400d3840 2024-04-02T14:38:32.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.471+0000 7f69ab1aa740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5615400deec0 con 0x5615400d2b60 2024-04-02T14:38:32.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.471+0000 7f69ab1aa740 10 client.0 _ll_register_callbacks cb 0x56153ffef030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:38:32.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.471+0000 7f69a5000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5615400d2b60 0x5615400d8e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:32.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.471+0000 7f69a5a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5615400d3840 0x5615400d9380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:32.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.472+0000 7f69a5000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5615400d2b60 0x5615400d8e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:37914/0 (socket says 172.21.15.90:37914) 2024-04-02T14:38:32.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.472+0000 7f69a5000640 1 -- 172.21.15.90:0/2260491440 learned_addr learned my addr 172.21.15.90:0/2260491440 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:38:32.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.472+0000 7f69a5a00640 1 -- 172.21.15.90:0/2260491440 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5615400d2b60 msgr2=0x5615400d8e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:38:32.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.472+0000 7f69a5a00640 1 --2- 172.21.15.90:0/2260491440 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5615400d2b60 0x5615400d8e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:32.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.472+0000 7f69a5a00640 1 -- 172.21.15.90:0/2260491440 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5615400dbed0 msgr2=0x5615400d98c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:38:32.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.472+0000 7f69a5a00640 1 --2- 172.21.15.90:0/2260491440 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5615400dbed0 0x5615400d98c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:32.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.472+0000 7f69a5a00640 1 -- 172.21.15.90:0/2260491440 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56153fe6af70 con 0x5615400d3840 2024-04-02T14:38:32.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.473+0000 7f69a5a00640 1 --2- 172.21.15.90:0/2260491440 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5615400d3840 0x5615400d9380 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f6990018fd0 tx=0x7f6990005630 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ecf747a009255974 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:38:32.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.473+0000 7f6983400640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:38:32.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.474+0000 7f6983400640 1 -- 172.21.15.90:0/2260491440 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6990025020 con 0x5615400d3840 2024-04-02T14:38:32.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.474+0000 7f6983400640 1 -- 172.21.15.90:0/2260491440 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f69900173f0 con 0x5615400d3840 2024-04-02T14:38:32.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.474+0000 7f69ab1aa740 1 -- 172.21.15.90:0/2260491440 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6990009d00 con 0x5615400d3840 2024-04-02T14:38:32.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.474+0000 7f6983400640 1 -- 172.21.15.90:0/2260491440 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f699001c040 con 0x5615400d3840 2024-04-02T14:38:32.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.474+0000 7f69ab1aa740 10 client.26561 fetch_fsmap learned FSMap version 27 2024-04-02T14:38:32.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.474+0000 7f69ab1aa740 1 -- 172.21.15.90:0/2260491440 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x56153fd02170 con 0x5615400d3840 2024-04-02T14:38:32.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.474+0000 7f6983400640 1 -- 172.21.15.90:0/2260491440 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6990009d00 con 0x5615400d3840 2024-04-02T14:38:32.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.475+0000 7f6983400640 1 -- 172.21.15.90:0/2260491440 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f6990016070 con 0x5615400d3840 2024-04-02T14:38:32.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.475+0000 7f69ab1aa740 10 client.26561 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:38:32.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.475+0000 7f69ab1aa740 10 client.26561 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:38:32.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.475+0000 7f69ab1aa740 20 client.26561 populate_metadata read hostname 'smithi090' 2024-04-02T14:38:32.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.475+0000 7f69ab1aa740 1 --2- 172.21.15.90:0/2260491440 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5615400ec1f0 0x5615400ee5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:32.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.475+0000 7f69ab1aa740 4 client.26561 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:38:32.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.475+0000 7f69ab1aa740 1 -- 172.21.15.90:0/2260491440 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56154003b850 con 0x5615400ec1f0 2024-04-02T14:38:32.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.475+0000 7f69a5000640 1 --2- 172.21.15.90:0/2260491440 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5615400ec1f0 0x5615400ee5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:32.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.476+0000 7f69a5000640 1 --2- 172.21.15.90:0/2260491440 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5615400ec1f0 0x5615400ee5e0 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f0008faba9b21264 server_cookie=5bf7159085769233 in_seq=0 out_seq=0 2024-04-02T14:38:32.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.476+0000 7f6983400640 10 client.26561 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:38:32.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.477+0000 7f6983400640 1 -- 172.21.15.90:0/2260491440 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56154003b850 con 0x5615400ec1f0 2024-04-02T14:38:32.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.477+0000 7f6983400640 10 client.26561 handle_command_reply: tid=0 2024-04-02T14:38:32.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.477+0000 7f69ab1aa740 1 client.26561 shutdown 2024-04-02T14:38:32.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.478+0000 7f69ab1aa740 1 -- 172.21.15.90:0/2260491440 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5615400d3840 msgr2=0x5615400d9380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:38:32.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.478+0000 7f69ab1aa740 1 --2- 172.21.15.90:0/2260491440 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5615400d3840 0x5615400d9380 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f6990018fd0 tx=0x7f6990005630 comp rx=0 tx=0).stop 2024-04-02T14:38:32.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.478+0000 7f69ab1aa740 1 -- 172.21.15.90:0/2260491440 shutdown_connections 2024-04-02T14:38:32.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.478+0000 7f69ab1aa740 1 --2- 172.21.15.90:0/2260491440 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5615400ec1f0 0x5615400ee5e0 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:32.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.478+0000 7f69ab1aa740 1 --2- 172.21.15.90:0/2260491440 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5615400dbed0 0x5615400d98c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:32.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.478+0000 7f69ab1aa740 1 --2- 172.21.15.90:0/2260491440 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5615400d3840 0x5615400d9380 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:32.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.478+0000 7f69ab1aa740 1 --2- 172.21.15.90:0/2260491440 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5615400d2b60 0x5615400d8e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:32.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.478+0000 7f69ab1aa740 1 -- 172.21.15.90:0/2260491440 >> 172.21.15.90:0/2260491440 conn(0x56154005e270 msgr2=0x5615400cf300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:38:32.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.478+0000 7f6983400640 0 client.26561 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:38:32.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.479+0000 7f69ab1aa740 1 -- 172.21.15.90:0/2260491440 shutdown_connections 2024-04-02T14:38:32.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.479+0000 7f69ab1aa740 1 -- 172.21.15.90:0/2260491440 wait complete. 2024-04-02T14:38:32.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.479+0000 7f69ab1aa740 20 client.26561 trim_cache size 0 max 16384 2024-04-02T14:38:32.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.514+0000 7f69ab1aa740 1 Processor -- start 2024-04-02T14:38:32.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.514+0000 7f69ab1aa740 1 -- start start 2024-04-02T14:38:32.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.515+0000 7f69ab1aa740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5615400d2b60 0x5615400d96e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:32.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.515+0000 7f69ab1aa740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5615400d3840 0x5615400e78f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:32.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.515+0000 7f69ab1aa740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5615400dbed0 0x5615400d60e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:32.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.515+0000 7f69ab1aa740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f69a01a2f80 con 0x5615400d2b60 2024-04-02T14:38:32.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.515+0000 7f69ab1aa740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f69a01a4ac0 con 0x5615400dbed0 2024-04-02T14:38:32.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.515+0000 7f69ab1aa740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f69a01a3650 con 0x5615400d3840 2024-04-02T14:38:32.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.516+0000 7f69a5a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5615400d3840 0x5615400e78f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:32.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.517+0000 7f69a5a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5615400d3840 0x5615400e78f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:37918/0 (socket says 172.21.15.90:37918) 2024-04-02T14:38:32.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.517+0000 7f69a5a00640 1 -- 172.21.15.90:0/3472742636 learned_addr learned my addr 172.21.15.90:0/3472742636 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:38:32.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.517+0000 7f69a5000640 1 --2- 172.21.15.90:0/3472742636 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5615400dbed0 0x5615400d60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:32.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.518+0000 7f69a5000640 1 --2- 172.21.15.90:0/3472742636 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5615400d2b60 0x5615400d96e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:32.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.518+0000 7f69a5000640 1 -- 172.21.15.90:0/3472742636 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5615400d3840 msgr2=0x5615400e78f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:38:32.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.518+0000 7f69a5000640 1 --2- 172.21.15.90:0/3472742636 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5615400d3840 0x5615400e78f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:32.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.518+0000 7f69a5000640 1 -- 172.21.15.90:0/3472742636 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5615400d2b60 msgr2=0x5615400d96e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:38:32.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.518+0000 7f69a5000640 1 --2- 172.21.15.90:0/3472742636 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5615400d2b60 0x5615400d96e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:32.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.518+0000 7f69a5000640 1 -- 172.21.15.90:0/3472742636 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5615400d1b90 con 0x5615400dbed0 2024-04-02T14:38:32.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.518+0000 7f69a5000640 1 --2- 172.21.15.90:0/3472742636 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5615400dbed0 0x5615400d60e0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x5615400dee00 tx=0x7f6994009ac0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=297ff5f479d9e40b server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:38:32.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.519+0000 7f698e000640 1 -- 172.21.15.90:0/3472742636 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69940205a0 con 0x5615400dbed0 2024-04-02T14:38:32.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.519+0000 7f698e000640 1 -- 172.21.15.90:0/3472742636 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6994020740 con 0x5615400dbed0 2024-04-02T14:38:32.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.519+0000 7f698e000640 1 -- 172.21.15.90:0/3472742636 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6994003d00 con 0x5615400dbed0 2024-04-02T14:38:32.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.522+0000 7f69ab1aa740 1 -- 172.21.15.90:0/3472742636 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5615400dbed0 msgr2=0x5615400d60e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:38:32.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.522+0000 7f69ab1aa740 1 --2- 172.21.15.90:0/3472742636 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5615400dbed0 0x5615400d60e0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x5615400dee00 tx=0x7f6994009ac0 comp rx=0 tx=0).stop 2024-04-02T14:38:32.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.522+0000 7f69ab1aa740 1 -- 172.21.15.90:0/3472742636 shutdown_connections 2024-04-02T14:38:32.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.522+0000 7f69ab1aa740 1 --2- 172.21.15.90:0/3472742636 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5615400dbed0 0x5615400d60e0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:32.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.522+0000 7f69ab1aa740 1 --2- 172.21.15.90:0/3472742636 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5615400d3840 0x5615400e78f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:32.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.522+0000 7f69ab1aa740 1 --2- 172.21.15.90:0/3472742636 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5615400d2b60 0x5615400d96e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:32.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.522+0000 7f69ab1aa740 1 -- 172.21.15.90:0/3472742636 >> 172.21.15.90:0/3472742636 conn(0x56154005e270 msgr2=0x5615400e71e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:38:32.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.523+0000 7f69ab1aa740 1 -- 172.21.15.90:0/3472742636 shutdown_connections 2024-04-02T14:38:32.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.523+0000 7f69ab1aa740 1 -- 172.21.15.90:0/3472742636 wait complete. 2024-04-02T14:38:32.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.524+0000 7f69ab1aa740 1 -- start start 2024-04-02T14:38:32.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.524+0000 7f69ab1aa740 1 Processor -- start 2024-04-02T14:38:32.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.525+0000 7f69ab1aa740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5615400d2b60 0x5615400f1340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:32.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.525+0000 7f69ab1aa740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5615400d3840 0x5615400f1880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:32.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.525+0000 7f69ab1aa740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5615400dbed0 0x5615400ef8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:32.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.525+0000 7f69ab1aa740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f69a01a43f0 con 0x5615400dbed0 2024-04-02T14:38:32.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.525+0000 7f69ab1aa740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f69a01a3d20 con 0x5615400d2b60 2024-04-02T14:38:32.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.525+0000 7f69ab1aa740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f69a01a0f60 con 0x5615400d3840 2024-04-02T14:38:32.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.525+0000 7f69ab1aa740 10 client.0 _ll_register_callbacks cb 0x7f699001a070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:38:32.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.526+0000 7f69a5000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5615400d2b60 0x5615400f1340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:32.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.526+0000 7f69a5000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5615400d2b60 0x5615400f1340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:48152/0 (socket says 172.21.15.90:48152) 2024-04-02T14:38:32.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.526+0000 7f69a5000640 1 -- 172.21.15.90:0/88324210 learned_addr learned my addr 172.21.15.90:0/88324210 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:38:32.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.526+0000 7f69a5000640 1 --2- 172.21.15.90:0/88324210 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5615400dbed0 0x5615400ef8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:32.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.526+0000 7f69a5000640 1 -- 172.21.15.90:0/88324210 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5615400d3840 msgr2=0x5615400f1880 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-02T14:38:32.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.527+0000 7f69a5000640 1 --2- 172.21.15.90:0/88324210 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5615400d3840 0x5615400f1880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:32.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.527+0000 7f69a5000640 1 -- 172.21.15.90:0/88324210 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5615400dbed0 msgr2=0x5615400ef8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:38:32.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.527+0000 7f69a5000640 1 --2- 172.21.15.90:0/88324210 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5615400dbed0 0x5615400ef8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:32.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.527+0000 7f69a5000640 1 -- 172.21.15.90:0/88324210 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5615400effd0 con 0x5615400d2b60 2024-04-02T14:38:32.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.528+0000 7f69a5000640 1 --2- 172.21.15.90:0/88324210 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5615400d2b60 0x5615400f1340 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f69940097c0 tx=0x7f69940038e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ca426d8175404be4 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:38:32.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.528+0000 7f6983400640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:38:32.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.528+0000 7f6983400640 1 -- 172.21.15.90:0/88324210 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f699402a650 con 0x5615400d2b60 2024-04-02T14:38:32.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.529+0000 7f6983400640 1 -- 172.21.15.90:0/88324210 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f699402a7f0 con 0x5615400d2b60 2024-04-02T14:38:32.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.529+0000 7f69ab1aa740 1 -- 172.21.15.90:0/88324210 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6994020740 con 0x5615400d2b60 2024-04-02T14:38:32.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.530+0000 7f6983400640 1 -- 172.21.15.90:0/88324210 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f699402a9f0 con 0x5615400d2b60 2024-04-02T14:38:32.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.530+0000 7f69ab1aa740 10 client.26567 fetch_fsmap learned FSMap version 27 2024-04-02T14:38:32.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.530+0000 7f69ab1aa740 1 -- 172.21.15.90:0/88324210 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x561540152f70 con 0x5615400d2b60 2024-04-02T14:38:32.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.531+0000 7f6983400640 1 -- 172.21.15.90:0/88324210 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6994020740 con 0x5615400d2b60 2024-04-02T14:38:32.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.531+0000 7f6983400640 1 -- 172.21.15.90:0/88324210 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f6994004330 con 0x5615400d2b60 2024-04-02T14:38:32.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.531+0000 7f69ab1aa740 10 client.26567 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:38:32.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.531+0000 7f69ab1aa740 10 client.26567 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:38:32.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.531+0000 7f69ab1aa740 20 client.26567 populate_metadata read hostname 'smithi090' 2024-04-02T14:38:32.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.531+0000 7f69ab1aa740 1 --2- 172.21.15.90:0/88324210 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x561540153570 0x561540155990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:32.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.532+0000 7f69a5a00640 1 --2- 172.21.15.90:0/88324210 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x561540153570 0x561540155990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:32.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.532+0000 7f69ab1aa740 4 client.26567 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-02T14:38:32.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.532+0000 7f69ab1aa740 1 -- 172.21.15.90:0/88324210 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x561540035ba0 con 0x561540153570 2024-04-02T14:38:32.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.533+0000 7f69a5a00640 1 --2- 172.21.15.90:0/88324210 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x561540153570 0x561540155990 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dfc790b84ec1b4dd server_cookie=18267eccae03e8af in_seq=0 out_seq=0 2024-04-02T14:38:32.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.533+0000 7f6983400640 10 client.26567 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:38:32.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.534+0000 7f6983400640 1 -- 172.21.15.90:0/88324210 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x561540035ba0 con 0x561540153570 2024-04-02T14:38:32.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.534+0000 7f6983400640 10 client.26567 handle_command_reply: tid=0 2024-04-02T14:38:32.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.534+0000 7f69ab1aa740 1 client.26567 shutdown 2024-04-02T14:38:32.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.535+0000 7f69ab1aa740 1 -- 172.21.15.90:0/88324210 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5615400d2b60 msgr2=0x5615400f1340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:38:32.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.535+0000 7f69ab1aa740 1 --2- 172.21.15.90:0/88324210 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5615400d2b60 0x5615400f1340 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f69940097c0 tx=0x7f69940038e0 comp rx=0 tx=0).stop 2024-04-02T14:38:32.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.535+0000 7f69ab1aa740 1 -- 172.21.15.90:0/88324210 shutdown_connections 2024-04-02T14:38:32.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.535+0000 7f69ab1aa740 1 --2- 172.21.15.90:0/88324210 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x561540153570 0x561540155990 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:32.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.535+0000 7f69ab1aa740 1 --2- 172.21.15.90:0/88324210 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5615400dbed0 0x5615400ef8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:32.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.535+0000 7f69ab1aa740 1 --2- 172.21.15.90:0/88324210 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5615400d3840 0x5615400f1880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:32.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.535+0000 7f69ab1aa740 1 --2- 172.21.15.90:0/88324210 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5615400d2b60 0x5615400f1340 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:32.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.535+0000 7f69ab1aa740 1 -- 172.21.15.90:0/88324210 >> 172.21.15.90:0/88324210 conn(0x56154005e270 msgr2=0x5615400e7a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:38:32.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.535+0000 7f6983400640 0 client.26567 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:38:32.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.535+0000 7f69ab1aa740 1 -- 172.21.15.90:0/88324210 shutdown_connections 2024-04-02T14:38:32.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.536+0000 7f69ab1aa740 1 -- 172.21.15.90:0/88324210 wait complete. 2024-04-02T14:38:32.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.536+0000 7f69ab1aa740 20 client.26567 trim_cache size 0 max 16384 2024-04-02T14:38:32.538 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-02T14:38:32.538 INFO:teuthology.orchestra.run.smithi090.stdout: "return_code": 0, 2024-04-02T14:38:32.538 INFO:teuthology.orchestra.run.smithi090.stdout: "scrub_tag": "c2c73d6c-aa01-44af-8d8a-5251e737d405", 2024-04-02T14:38:32.538 INFO:teuthology.orchestra.run.smithi090.stdout: "mode": "asynchronous" 2024-04-02T14:38:32.538 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-02T14:38:32.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.536+0000 7f69a8400640 1 -- 172.21.15.90:0/2880201341 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f69880a3d20 msgr2=0x7f69880a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:38:32.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.536+0000 7f69a8400640 1 --2- 172.21.15.90:0/2880201341 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f69880a3d20 0x7f69880a61e0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f69980c00b0 tx=0x7f6998030190 comp rx=0 tx=0).stop 2024-04-02T14:38:32.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.536+0000 7f69a8400640 1 -- 172.21.15.90:0/2880201341 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f69a010abe0 msgr2=0x7f69a006c990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:38:32.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.536+0000 7f69a8400640 1 --2- 172.21.15.90:0/2880201341 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f69a010abe0 0x7f69a006c990 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f699803a040 tx=0x7f6998037c30 comp rx=0 tx=0).stop 2024-04-02T14:38:32.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.537+0000 7f69a8400640 1 -- 172.21.15.90:0/2880201341 shutdown_connections 2024-04-02T14:38:32.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.537+0000 7f69a8400640 1 --2- 172.21.15.90:0/2880201341 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f69880a3d20 0x7f69880a61e0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:32.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.537+0000 7f69a8400640 1 --2- 172.21.15.90:0/2880201341 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f69a010abe0 0x7f69a006c990 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:32.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.537+0000 7f69a8400640 1 --2- 172.21.15.90:0/2880201341 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f69a0100870 0x7f69a006c450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:32.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.537+0000 7f69a8400640 1 --2- 172.21.15.90:0/2880201341 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f69a0068060 0x7f69a00733d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:32.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.537+0000 7f69a8400640 1 -- 172.21.15.90:0/2880201341 >> 172.21.15.90:0/2880201341 conn(0x7f69a0076b50 msgr2=0x7f69a010c0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:38:32.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.538+0000 7f69a8400640 1 -- 172.21.15.90:0/2880201341 shutdown_connections 2024-04-02T14:38:32.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.538+0000 7f69a8400640 1 -- 172.21.15.90:0/2880201341 wait complete. 2024-04-02T14:38:32.569 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-02T14:38:32.569 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-02T14:38:32.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.891+0000 7fd174600640 1 -- 172.21.15.90:0/2531962879 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd16c073f30 msgr2=0x7fd16c074310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:38:32.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.891+0000 7fd174600640 1 --2- 172.21.15.90:0/2531962879 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd16c073f30 0x7fd16c074310 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fd15c009900 tx=0x7fd15c02bcf0 comp rx=0 tx=0).stop 2024-04-02T14:38:32.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.891+0000 7fd174600640 1 -- 172.21.15.90:0/2531962879 shutdown_connections 2024-04-02T14:38:32.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.891+0000 7fd174600640 1 --2- 172.21.15.90:0/2531962879 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd16c10e2e0 0x7fd16c110730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:32.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.891+0000 7fd174600640 1 --2- 172.21.15.90:0/2531962879 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd16c074850 0x7fd16c10dda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:32.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.891+0000 7fd174600640 1 --2- 172.21.15.90:0/2531962879 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd16c073f30 0x7fd16c074310 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:32.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.891+0000 7fd174600640 1 -- 172.21.15.90:0/2531962879 >> 172.21.15.90:0/2531962879 conn(0x7fd16c0fd530 msgr2=0x7fd16c0ff950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:38:32.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.891+0000 7fd174600640 1 -- 172.21.15.90:0/2531962879 shutdown_connections 2024-04-02T14:38:32.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.891+0000 7fd174600640 1 -- 172.21.15.90:0/2531962879 wait complete. 2024-04-02T14:38:32.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.893+0000 7fd174600640 1 Processor -- start 2024-04-02T14:38:32.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.893+0000 7fd174600640 1 -- start start 2024-04-02T14:38:32.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.893+0000 7fd174600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd16c074850 0x7fd16c10af00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:32.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.893+0000 7fd174600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd16c10e2e0 0x7fd16c10b440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:32.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.893+0000 7fd174600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd16c10b980 0x7fd16c1aade0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:32.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.893+0000 7fd174600640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fd16c1140a0 con 0x7fd16c10e2e0 2024-04-02T14:38:32.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.893+0000 7fd174600640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fd16c113f20 con 0x7fd16c074850 2024-04-02T14:38:32.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.893+0000 7fd174600640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fd16c114220 con 0x7fd16c10b980 2024-04-02T14:38:32.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.894+0000 7fd171c00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd16c074850 0x7fd16c10af00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:32.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.894+0000 7fd171c00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd16c074850 0x7fd16c10af00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:48176/0 (socket says 172.21.15.90:48176) 2024-04-02T14:38:32.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.894+0000 7fd171c00640 1 -- 172.21.15.90:0/3663269878 learned_addr learned my addr 172.21.15.90:0/3663269878 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:38:32.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.894+0000 7fd171200640 1 --2- 172.21.15.90:0/3663269878 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd16c10e2e0 0x7fd16c10b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:32.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.894+0000 7fd172600640 1 --2- 172.21.15.90:0/3663269878 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd16c10b980 0x7fd16c1aade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:32.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.895+0000 7fd171c00640 1 -- 172.21.15.90:0/3663269878 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd16c10b980 msgr2=0x7fd16c1aade0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:38:32.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.895+0000 7fd171c00640 1 --2- 172.21.15.90:0/3663269878 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd16c10b980 0x7fd16c1aade0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:32.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.895+0000 7fd171c00640 1 -- 172.21.15.90:0/3663269878 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd16c10e2e0 msgr2=0x7fd16c10b440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:38:32.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.895+0000 7fd171c00640 1 --2- 172.21.15.90:0/3663269878 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd16c10e2e0 0x7fd16c10b440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:32.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.895+0000 7fd171c00640 1 -- 172.21.15.90:0/3663269878 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd16c1ab320 con 0x7fd16c074850 2024-04-02T14:38:32.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.895+0000 7fd171200640 1 --2- 172.21.15.90:0/3663269878 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd16c10e2e0 0x7fd16c10b440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:38:32.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.895+0000 7fd171c00640 1 --2- 172.21.15.90:0/3663269878 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd16c074850 0x7fd16c10af00 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fd15c005ab0 tx=0x7fd15c02ec00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:38:32.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.895+0000 7fd16a000640 1 -- 172.21.15.90:0/3663269878 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd15c041020 con 0x7fd16c074850 2024-04-02T14:38:32.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.896+0000 7fd16a000640 1 -- 172.21.15.90:0/3663269878 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd15c02c690 con 0x7fd16c074850 2024-04-02T14:38:32.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.896+0000 7fd16a000640 1 -- 172.21.15.90:0/3663269878 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd15c041020 con 0x7fd16c074850 2024-04-02T14:38:32.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.896+0000 7fd174600640 1 -- 172.21.15.90:0/3663269878 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd16c1ab4f0 con 0x7fd16c074850 2024-04-02T14:38:32.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.896+0000 7fd174600640 1 -- 172.21.15.90:0/3663269878 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd16c1ab960 con 0x7fd16c074850 2024-04-02T14:38:32.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.904+0000 7fd1773fd740 1 Processor -- start 2024-04-02T14:38:32.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.905+0000 7fd1773fd740 1 -- start start 2024-04-02T14:38:32.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.905+0000 7fd16a000640 1 -- 172.21.15.90:0/3663269878 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fd15c044020 con 0x7fd16c074850 2024-04-02T14:38:32.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.905+0000 7fd1773fd740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ed0d6467b0 0x55ed0d646b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:32.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.905+0000 7fd1773fd740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ed0d6470d0 0x55ed0d64f770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:32.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.905+0000 7fd1773fd740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ed0d64fcb0 0x55ed0d6520a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:32.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.905+0000 7fd1773fd740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55ed0d5aa770 con 0x55ed0d6467b0 2024-04-02T14:38:32.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.905+0000 7fd1773fd740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55ed0d552700 con 0x55ed0d64fcb0 2024-04-02T14:38:32.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.905+0000 7fd1773fd740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55ed0d5a6720 con 0x55ed0d6470d0 2024-04-02T14:38:32.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.906+0000 7fd171c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ed0d6470d0 0x55ed0d64f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:32.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.906+0000 7fd171200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ed0d64fcb0 0x55ed0d6520a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:32.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.906+0000 7fd171200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ed0d64fcb0 0x55ed0d6520a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:48192/0 (socket says 172.21.15.90:48192) 2024-04-02T14:38:32.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.906+0000 7fd171200640 1 -- 172.21.15.90:0/1100743500 learned_addr learned my addr 172.21.15.90:0/1100743500 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:38:32.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.906+0000 7fd171200640 1 -- 172.21.15.90:0/1100743500 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ed0d6470d0 msgr2=0x55ed0d64f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:38:32.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.906+0000 7fd171200640 1 --2- 172.21.15.90:0/1100743500 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ed0d6470d0 0x55ed0d64f770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:32.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.906+0000 7fd171200640 1 -- 172.21.15.90:0/1100743500 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ed0d6467b0 msgr2=0x55ed0d646b90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-02T14:38:32.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.907+0000 7fd172600640 1 --2- 172.21.15.90:0/1100743500 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ed0d6467b0 0x55ed0d646b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:32.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.907+0000 7fd16a000640 1 --2- 172.21.15.90:0/3663269878 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fd1540a3f20 0x7fd1540a63e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:32.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.907+0000 7fd171200640 1 --2- 172.21.15.90:0/1100743500 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ed0d6467b0 0x55ed0d646b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:32.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.907+0000 7fd171200640 1 -- 172.21.15.90:0/1100743500 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ed0d3887d0 con 0x55ed0d64fcb0 2024-04-02T14:38:32.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.907+0000 7fd16a000640 1 -- 172.21.15.90:0/3663269878 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7fd15c0f0e30 con 0x7fd16c074850 2024-04-02T14:38:32.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.908+0000 7fd172600640 1 --2- 172.21.15.90:0/1100743500 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ed0d6467b0 0x55ed0d646b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:38:32.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.908+0000 7fd171c00640 1 --2- 172.21.15.90:0/3663269878 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fd1540a3f20 0x7fd1540a63e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:32.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.909+0000 7fd171200640 1 --2- 172.21.15.90:0/1100743500 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ed0d64fcb0 0x55ed0d6520a0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fd160010480 tx=0x7fd16000bf90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f3b4a66fe367387e server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:38:32.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.909+0000 7fd171c00640 1 --2- 172.21.15.90:0/3663269878 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fd1540a3f20 0x7fd1540a63e0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fd15c0ee750 tx=0x7fd15c0f4980 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:38:32.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.909+0000 7fd15aa00640 1 -- 172.21.15.90:0/1100743500 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd160011760 con 0x55ed0d64fcb0 2024-04-02T14:38:32.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.909+0000 7fd15aa00640 1 -- 172.21.15.90:0/1100743500 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd160011de0 con 0x55ed0d64fcb0 2024-04-02T14:38:32.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.909+0000 7fd15aa00640 1 -- 172.21.15.90:0/1100743500 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd16001ce60 con 0x55ed0d64fcb0 2024-04-02T14:38:32.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.910+0000 7fd1773fd740 1 -- 172.21.15.90:0/1100743500 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ed0d64fcb0 msgr2=0x55ed0d6520a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:38:32.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.910+0000 7fd1773fd740 1 --2- 172.21.15.90:0/1100743500 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ed0d64fcb0 0x55ed0d6520a0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fd160010480 tx=0x7fd16000bf90 comp rx=0 tx=0).stop 2024-04-02T14:38:32.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.910+0000 7fd1773fd740 1 -- 172.21.15.90:0/1100743500 shutdown_connections 2024-04-02T14:38:32.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.910+0000 7fd1773fd740 1 --2- 172.21.15.90:0/1100743500 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ed0d64fcb0 0x55ed0d6520a0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:32.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.910+0000 7fd1773fd740 1 --2- 172.21.15.90:0/1100743500 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ed0d6470d0 0x55ed0d64f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:32.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.910+0000 7fd1773fd740 1 --2- 172.21.15.90:0/1100743500 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ed0d6467b0 0x55ed0d646b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:32.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.910+0000 7fd1773fd740 1 -- 172.21.15.90:0/1100743500 >> 172.21.15.90:0/1100743500 conn(0x55ed0d640ee0 msgr2=0x55ed0d6412c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:38:32.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.910+0000 7fd1773fd740 1 -- 172.21.15.90:0/1100743500 shutdown_connections 2024-04-02T14:38:32.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.911+0000 7fd1773fd740 1 -- 172.21.15.90:0/1100743500 wait complete. 2024-04-02T14:38:32.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.912+0000 7fd1773fd740 1 -- start start 2024-04-02T14:38:32.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.912+0000 7fd1773fd740 1 Processor -- start 2024-04-02T14:38:32.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.913+0000 7fd1773fd740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ed0d6467b0 0x55ed0d64cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:32.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.913+0000 7fd1773fd740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ed0d6470d0 0x55ed0d64d290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:32.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.913+0000 7fd1773fd740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ed0d64fcb0 0x55ed0d64d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:32.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.913+0000 7fd1773fd740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55ed0d652aa0 con 0x55ed0d6470d0 2024-04-02T14:38:32.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.913+0000 7fd1773fd740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55ed0d565ac0 con 0x55ed0d64fcb0 2024-04-02T14:38:32.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.913+0000 7fd1773fd740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55ed0d652c20 con 0x55ed0d6467b0 2024-04-02T14:38:32.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.913+0000 7fd1773fd740 10 client.0 _ll_register_callbacks cb 0x55ed0d568180 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:38:32.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.914+0000 7fd171200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ed0d6470d0 0x55ed0d64d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:32.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.914+0000 7fd171200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ed0d6470d0 0x55ed0d64d290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:46240/0 (socket says 172.21.15.90:46240) 2024-04-02T14:38:32.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.914+0000 7fd171200640 1 -- 172.21.15.90:0/988654040 learned_addr learned my addr 172.21.15.90:0/988654040 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:38:32.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.914+0000 7fd172600640 1 --2- 172.21.15.90:0/988654040 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ed0d6467b0 0x55ed0d64cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:32.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.914+0000 7fd172600640 1 --2- 172.21.15.90:0/988654040 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ed0d64fcb0 0x55ed0d64d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:32.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.914+0000 7fd171200640 1 -- 172.21.15.90:0/988654040 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ed0d6467b0 msgr2=0x55ed0d64cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:38:32.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.914+0000 7fd171200640 1 --2- 172.21.15.90:0/988654040 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ed0d6467b0 0x55ed0d64cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:32.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.914+0000 7fd171200640 1 -- 172.21.15.90:0/988654040 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ed0d64fcb0 msgr2=0x55ed0d64d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:38:32.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.914+0000 7fd171200640 1 --2- 172.21.15.90:0/988654040 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ed0d64fcb0 0x55ed0d64d7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:32.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.914+0000 7fd171200640 1 -- 172.21.15.90:0/988654040 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ed0d2be3f0 con 0x55ed0d6470d0 2024-04-02T14:38:32.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.915+0000 7fd172600640 1 --2- 172.21.15.90:0/988654040 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ed0d64fcb0 0x55ed0d64d7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:38:32.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.915+0000 7fd171200640 1 --2- 172.21.15.90:0/988654040 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ed0d6470d0 0x55ed0d64d290 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fd16001c8d0 tx=0x7fd16001e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=458021bda84f072a server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:38:32.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.915+0000 7fd14f400640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-02T14:38:32.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.916+0000 7fd14f400640 1 -- 172.21.15.90:0/988654040 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd16002b020 con 0x55ed0d6470d0 2024-04-02T14:38:32.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.916+0000 7fd14f400640 1 -- 172.21.15.90:0/988654040 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7fd160004220 con 0x55ed0d6470d0 2024-04-02T14:38:32.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.916+0000 7fd1773fd740 1 -- 172.21.15.90:0/988654040 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd160011de0 con 0x55ed0d6470d0 2024-04-02T14:38:32.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.917+0000 7fd14f400640 1 -- 172.21.15.90:0/988654040 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fd16001c690 con 0x55ed0d6470d0 2024-04-02T14:38:32.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.917+0000 7fd1773fd740 10 client.16104 fetch_fsmap learned FSMap version 27 2024-04-02T14:38:32.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.917+0000 7fd1773fd740 1 -- 172.21.15.90:0/988654040 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55ed0d27e0f0 con 0x55ed0d6470d0 2024-04-02T14:38:32.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.918+0000 7fd14f400640 1 -- 172.21.15.90:0/988654040 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd160011de0 con 0x55ed0d6470d0 2024-04-02T14:38:32.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.918+0000 7fd14f400640 1 -- 172.21.15.90:0/988654040 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fd160003830 con 0x55ed0d6470d0 2024-04-02T14:38:32.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.919+0000 7fd1773fd740 10 client.16104 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:38:32.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.919+0000 7fd1773fd740 10 client.16104 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:38:32.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.919+0000 7fd1773fd740 20 client.16104 populate_metadata read hostname 'smithi090' 2024-04-02T14:38:32.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.919+0000 7fd1773fd740 1 --2- 172.21.15.90:0/988654040 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55ed0d670b70 0x55ed0d672f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:32.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.919+0000 7fd1773fd740 4 client.16104 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:38:32.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.919+0000 7fd1773fd740 1 -- 172.21.15.90:0/988654040 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ed0d54b630 con 0x55ed0d670b70 2024-04-02T14:38:32.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.919+0000 7fd172600640 1 --2- 172.21.15.90:0/988654040 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55ed0d670b70 0x55ed0d672f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:32.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.920+0000 7fd14f400640 10 client.16104 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:38:32.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.920+0000 7fd172600640 1 --2- 172.21.15.90:0/988654040 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55ed0d670b70 0x55ed0d672f60 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ea4174b535a81dbe server_cookie=5cf994ee975cc00e in_seq=0 out_seq=0 2024-04-02T14:38:32.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.921+0000 7fd14f400640 1 -- 172.21.15.90:0/988654040 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ed0d54b630 con 0x55ed0d670b70 2024-04-02T14:38:32.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.921+0000 7fd14f400640 10 client.16104 handle_command_reply: tid=0 2024-04-02T14:38:32.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.921+0000 7fd1773fd740 1 client.16104 shutdown 2024-04-02T14:38:32.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.922+0000 7fd1773fd740 1 -- 172.21.15.90:0/988654040 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ed0d6470d0 msgr2=0x55ed0d64d290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:38:32.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.922+0000 7fd1773fd740 1 --2- 172.21.15.90:0/988654040 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ed0d6470d0 0x55ed0d64d290 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fd16001c8d0 tx=0x7fd16001e000 comp rx=0 tx=0).stop 2024-04-02T14:38:32.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.922+0000 7fd1773fd740 1 -- 172.21.15.90:0/988654040 shutdown_connections 2024-04-02T14:38:32.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.922+0000 7fd1773fd740 1 --2- 172.21.15.90:0/988654040 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55ed0d670b70 0x55ed0d672f60 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:32.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.922+0000 7fd1773fd740 1 --2- 172.21.15.90:0/988654040 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ed0d64fcb0 0x55ed0d64d7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:32.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.922+0000 7fd1773fd740 1 --2- 172.21.15.90:0/988654040 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ed0d6470d0 0x55ed0d64d290 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:32.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.922+0000 7fd1773fd740 1 --2- 172.21.15.90:0/988654040 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ed0d6467b0 0x55ed0d64cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:32.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.922+0000 7fd1773fd740 1 -- 172.21.15.90:0/988654040 >> 172.21.15.90:0/988654040 conn(0x55ed0d640ee0 msgr2=0x55ed0d642c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:38:32.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.922+0000 7fd14f400640 0 client.16104 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:38:32.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.922+0000 7fd1773fd740 1 -- 172.21.15.90:0/988654040 shutdown_connections 2024-04-02T14:38:32.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.923+0000 7fd1773fd740 1 -- 172.21.15.90:0/988654040 wait complete. 2024-04-02T14:38:32.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.923+0000 7fd1773fd740 20 client.16104 trim_cache size 0 max 16384 2024-04-02T14:38:32.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.954+0000 7fd1773fd740 1 Processor -- start 2024-04-02T14:38:32.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.954+0000 7fd1773fd740 1 -- start start 2024-04-02T14:38:32.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.955+0000 7fd1773fd740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ed0d6467b0 0x55ed0d64f480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:32.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.955+0000 7fd1773fd740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ed0d6470d0 0x55ed0d65b960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:32.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.955+0000 7fd1773fd740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ed0d64fcb0 0x55ed0d6429b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:32.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.955+0000 7fd1773fd740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fd16c1abc80 con 0x55ed0d64fcb0 2024-04-02T14:38:32.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.955+0000 7fd1773fd740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fd16c1ad740 con 0x55ed0d6470d0 2024-04-02T14:38:32.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.955+0000 7fd1773fd740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fd16c1ac2d0 con 0x55ed0d6467b0 2024-04-02T14:38:32.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.956+0000 7fd172600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ed0d6467b0 0x55ed0d64f480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:32.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.956+0000 7fd172600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ed0d6467b0 0x55ed0d64f480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:37976/0 (socket says 172.21.15.90:37976) 2024-04-02T14:38:32.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.956+0000 7fd172600640 1 -- 172.21.15.90:0/3603511807 learned_addr learned my addr 172.21.15.90:0/3603511807 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:38:32.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.956+0000 7fd172600640 1 --2- 172.21.15.90:0/3603511807 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ed0d64fcb0 0x55ed0d6429b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:32.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.956+0000 7fd171200640 1 --2- 172.21.15.90:0/3603511807 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ed0d6470d0 0x55ed0d65b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:32.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.956+0000 7fd172600640 1 -- 172.21.15.90:0/3603511807 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ed0d6470d0 msgr2=0x55ed0d65b960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:38:32.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.956+0000 7fd172600640 1 --2- 172.21.15.90:0/3603511807 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ed0d6470d0 0x55ed0d65b960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:32.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.956+0000 7fd172600640 1 -- 172.21.15.90:0/3603511807 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ed0d64fcb0 msgr2=0x55ed0d6429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:38:32.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.956+0000 7fd172600640 1 --2- 172.21.15.90:0/3603511807 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ed0d64fcb0 0x55ed0d6429b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:32.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.956+0000 7fd172600640 1 -- 172.21.15.90:0/3603511807 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ed0d646f00 con 0x55ed0d6467b0 2024-04-02T14:38:32.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.957+0000 7fd172600640 1 --2- 172.21.15.90:0/3603511807 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ed0d6467b0 0x55ed0d64f480 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fd164013510 tx=0x7fd164004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cf720c55534ab67e server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:38:32.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.957+0000 7fd15a000640 1 -- 172.21.15.90:0/3603511807 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd16401f860 con 0x55ed0d6467b0 2024-04-02T14:38:32.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.957+0000 7fd15a000640 1 -- 172.21.15.90:0/3603511807 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd164005020 con 0x55ed0d6467b0 2024-04-02T14:38:32.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.958+0000 7fd15a000640 1 -- 172.21.15.90:0/3603511807 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd164010750 con 0x55ed0d6467b0 2024-04-02T14:38:32.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.958+0000 7fd1773fd740 1 -- 172.21.15.90:0/3603511807 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ed0d6467b0 msgr2=0x55ed0d64f480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:38:32.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.958+0000 7fd1773fd740 1 --2- 172.21.15.90:0/3603511807 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ed0d6467b0 0x55ed0d64f480 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fd164013510 tx=0x7fd164004a40 comp rx=0 tx=0).stop 2024-04-02T14:38:32.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.959+0000 7fd1773fd740 1 -- 172.21.15.90:0/3603511807 shutdown_connections 2024-04-02T14:38:32.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.959+0000 7fd1773fd740 1 --2- 172.21.15.90:0/3603511807 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ed0d64fcb0 0x55ed0d6429b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:32.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.959+0000 7fd1773fd740 1 --2- 172.21.15.90:0/3603511807 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ed0d6470d0 0x55ed0d65b960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:32.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.959+0000 7fd1773fd740 1 --2- 172.21.15.90:0/3603511807 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ed0d6467b0 0x55ed0d64f480 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:32.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.959+0000 7fd1773fd740 1 -- 172.21.15.90:0/3603511807 >> 172.21.15.90:0/3603511807 conn(0x55ed0d640ee0 msgr2=0x55ed0d6422a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:38:32.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.959+0000 7fd1773fd740 1 -- 172.21.15.90:0/3603511807 shutdown_connections 2024-04-02T14:38:32.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.959+0000 7fd1773fd740 1 -- 172.21.15.90:0/3603511807 wait complete. 2024-04-02T14:38:32.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.959+0000 7fd1773fd740 1 -- start start 2024-04-02T14:38:32.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.959+0000 7fd1773fd740 1 Processor -- start 2024-04-02T14:38:32.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.960+0000 7fd1773fd740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ed0d6467b0 0x55ed0d653c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:32.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.961+0000 7fd1773fd740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ed0d6470d0 0x55ed0d654190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:32.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.961+0000 7fd1773fd740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ed0d64fcb0 0x55ed0d6d5e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:32.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.961+0000 7fd1773fd740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fd16c1ad070 con 0x55ed0d6470d0 2024-04-02T14:38:32.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.961+0000 7fd1773fd740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fd16c1ac9a0 con 0x55ed0d6467b0 2024-04-02T14:38:32.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.961+0000 7fd1773fd740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fd16c071c80 con 0x55ed0d64fcb0 2024-04-02T14:38:32.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.961+0000 7fd172600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ed0d6467b0 0x55ed0d653c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:32.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.961+0000 7fd1773fd740 10 client.0 _ll_register_callbacks cb 0x7fd15c0f4850 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:38:32.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.961+0000 7fd172600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ed0d6467b0 0x55ed0d653c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:48218/0 (socket says 172.21.15.90:48218) 2024-04-02T14:38:32.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.961+0000 7fd172600640 1 -- 172.21.15.90:0/2719691333 learned_addr learned my addr 172.21.15.90:0/2719691333 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:38:32.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.961+0000 7fd171200640 1 --2- 172.21.15.90:0/2719691333 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ed0d6470d0 0x55ed0d654190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:32.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.961+0000 7fd172600640 1 --2- 172.21.15.90:0/2719691333 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ed0d64fcb0 0x55ed0d6d5e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:32.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.962+0000 7fd171200640 1 -- 172.21.15.90:0/2719691333 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ed0d64fcb0 msgr2=0x55ed0d6d5e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:38:32.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.962+0000 7fd171200640 1 --2- 172.21.15.90:0/2719691333 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ed0d64fcb0 0x55ed0d6d5e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:32.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.962+0000 7fd171200640 1 -- 172.21.15.90:0/2719691333 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ed0d6467b0 msgr2=0x55ed0d653c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:38:32.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.962+0000 7fd171200640 1 --2- 172.21.15.90:0/2719691333 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ed0d6467b0 0x55ed0d653c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:32.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.962+0000 7fd171200640 1 -- 172.21.15.90:0/2719691333 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ed0d6d63a0 con 0x55ed0d6470d0 2024-04-02T14:38:32.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.962+0000 7fd172600640 1 --2- 172.21.15.90:0/2719691333 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ed0d64fcb0 0x55ed0d6d5e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:38:32.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.963+0000 7fd14f400640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-02T14:38:32.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.963+0000 7fd171200640 1 --2- 172.21.15.90:0/2719691333 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ed0d6470d0 0x55ed0d654190 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fd16002c030 tx=0x7fd1600056f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b48c30c4ed7dae55 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:38:32.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.964+0000 7fd14f400640 1 -- 172.21.15.90:0/2719691333 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd16002b070 con 0x55ed0d6470d0 2024-04-02T14:38:32.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.964+0000 7fd14f400640 1 -- 172.21.15.90:0/2719691333 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7fd1600037f0 con 0x55ed0d6470d0 2024-04-02T14:38:32.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.964+0000 7fd1773fd740 1 -- 172.21.15.90:0/2719691333 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd164005020 con 0x55ed0d6470d0 2024-04-02T14:38:32.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.965+0000 7fd14f400640 1 -- 172.21.15.90:0/2719691333 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fd160048050 con 0x55ed0d6470d0 2024-04-02T14:38:32.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.965+0000 7fd1773fd740 10 client.16110 fetch_fsmap learned FSMap version 27 2024-04-02T14:38:32.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.965+0000 7fd1773fd740 1 -- 172.21.15.90:0/2719691333 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55ed0d6d7590 con 0x55ed0d6470d0 2024-04-02T14:38:32.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.965+0000 7fd14f400640 1 -- 172.21.15.90:0/2719691333 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd164005020 con 0x55ed0d6470d0 2024-04-02T14:38:32.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.966+0000 7fd14f400640 1 -- 172.21.15.90:0/2719691333 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fd160027770 con 0x55ed0d6470d0 2024-04-02T14:38:32.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.966+0000 7fd1773fd740 10 client.16110 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:38:32.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.966+0000 7fd1773fd740 10 client.16110 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:38:32.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.966+0000 7fd1773fd740 20 client.16110 populate_metadata read hostname 'smithi090' 2024-04-02T14:38:32.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.966+0000 7fd1773fd740 1 --2- 172.21.15.90:0/2719691333 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55ed0d6d7d90 0x55ed0d6da1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:32.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.966+0000 7fd1773fd740 4 client.16110 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-02T14:38:32.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.966+0000 7fd1773fd740 1 -- 172.21.15.90:0/2719691333 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55ed0d5b7cb0 con 0x55ed0d6d7d90 2024-04-02T14:38:32.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.966+0000 7fd172600640 1 --2- 172.21.15.90:0/2719691333 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55ed0d6d7d90 0x55ed0d6da1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:32.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.967+0000 7fd172600640 1 --2- 172.21.15.90:0/2719691333 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55ed0d6d7d90 0x55ed0d6da1b0 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eadef9c6fe48cb94 server_cookie=de28c657154b902a in_seq=0 out_seq=0 2024-04-02T14:38:32.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.967+0000 7fd14f400640 10 client.16110 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:38:32.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.967+0000 7fd14f400640 1 -- 172.21.15.90:0/2719691333 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55ed0d5b7cb0 con 0x55ed0d6d7d90 2024-04-02T14:38:32.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.967+0000 7fd14f400640 10 client.16110 handle_command_reply: tid=0 2024-04-02T14:38:32.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.967+0000 7fd1773fd740 1 client.16110 shutdown 2024-04-02T14:38:32.969 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.967+0000 7fd1773fd740 1 -- 172.21.15.90:0/2719691333 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ed0d6470d0 msgr2=0x55ed0d654190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:38:32.969 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.967+0000 7fd1773fd740 1 --2- 172.21.15.90:0/2719691333 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ed0d6470d0 0x55ed0d654190 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fd16002c030 tx=0x7fd1600056f0 comp rx=0 tx=0).stop 2024-04-02T14:38:32.969 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.968+0000 7fd1773fd740 1 -- 172.21.15.90:0/2719691333 shutdown_connections 2024-04-02T14:38:32.969 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.968+0000 7fd1773fd740 1 --2- 172.21.15.90:0/2719691333 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55ed0d6d7d90 0x55ed0d6da1b0 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:32.969 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.968+0000 7fd1773fd740 1 --2- 172.21.15.90:0/2719691333 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ed0d64fcb0 0x55ed0d6d5e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:32.970 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.968+0000 7fd1773fd740 1 --2- 172.21.15.90:0/2719691333 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ed0d6470d0 0x55ed0d654190 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:32.970 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.968+0000 7fd14f400640 0 client.16110 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:38:32.970 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.968+0000 7fd1773fd740 1 --2- 172.21.15.90:0/2719691333 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ed0d6467b0 0x55ed0d653c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:32.970 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.968+0000 7fd1773fd740 1 -- 172.21.15.90:0/2719691333 >> 172.21.15.90:0/2719691333 conn(0x55ed0d640ee0 msgr2=0x55ed0d642b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:38:32.970 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.968+0000 7fd1773fd740 1 -- 172.21.15.90:0/2719691333 shutdown_connections 2024-04-02T14:38:32.970 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.968+0000 7fd1773fd740 1 -- 172.21.15.90:0/2719691333 wait complete. 2024-04-02T14:38:32.970 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.969+0000 7fd1773fd740 20 client.16110 trim_cache size 0 max 16384 2024-04-02T14:38:32.970 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-02T14:38:32.970 INFO:teuthology.orchestra.run.smithi090.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-02T14:38:32.971 INFO:teuthology.orchestra.run.smithi090.stdout: "scrubs": { 2024-04-02T14:38:32.971 INFO:teuthology.orchestra.run.smithi090.stdout: "c2c73d6c-aa01-44af-8d8a-5251e737d405": { 2024-04-02T14:38:32.971 INFO:teuthology.orchestra.run.smithi090.stdout: "path": "/", 2024-04-02T14:38:32.971 INFO:teuthology.orchestra.run.smithi090.stdout: "tag": "c2c73d6c-aa01-44af-8d8a-5251e737d405", 2024-04-02T14:38:32.971 INFO:teuthology.orchestra.run.smithi090.stdout: "options": "recursive,force" 2024-04-02T14:38:32.971 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:38:32.971 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:38:32.971 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-02T14:38:32.972 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.969+0000 7fd174600640 1 -- 172.21.15.90:0/3663269878 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fd1540a3f20 msgr2=0x7fd1540a63e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:38:32.972 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.969+0000 7fd174600640 1 --2- 172.21.15.90:0/3663269878 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fd1540a3f20 0x7fd1540a63e0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fd15c0ee750 tx=0x7fd15c0f4980 comp rx=0 tx=0).stop 2024-04-02T14:38:32.972 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.969+0000 7fd174600640 1 -- 172.21.15.90:0/3663269878 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd16c074850 msgr2=0x7fd16c10af00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:38:32.972 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.969+0000 7fd174600640 1 --2- 172.21.15.90:0/3663269878 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd16c074850 0x7fd16c10af00 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fd15c005ab0 tx=0x7fd15c02ec00 comp rx=0 tx=0).stop 2024-04-02T14:38:32.972 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.970+0000 7fd174600640 1 -- 172.21.15.90:0/3663269878 shutdown_connections 2024-04-02T14:38:32.972 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.970+0000 7fd174600640 1 --2- 172.21.15.90:0/3663269878 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fd1540a3f20 0x7fd1540a63e0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:32.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.970+0000 7fd174600640 1 --2- 172.21.15.90:0/3663269878 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd16c10b980 0x7fd16c1aade0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:32.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.970+0000 7fd174600640 1 --2- 172.21.15.90:0/3663269878 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd16c10e2e0 0x7fd16c10b440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:32.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.970+0000 7fd174600640 1 --2- 172.21.15.90:0/3663269878 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd16c074850 0x7fd16c10af00 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:32.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.970+0000 7fd174600640 1 -- 172.21.15.90:0/3663269878 >> 172.21.15.90:0/3663269878 conn(0x7fd16c0fd530 msgr2=0x7fd16c0fe2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:38:32.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.970+0000 7fd174600640 1 -- 172.21.15.90:0/3663269878 shutdown_connections 2024-04-02T14:38:32.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:38:32.970+0000 7fd174600640 1 -- 172.21.15.90:0/3663269878 wait complete. 2024-04-02T14:38:32.998 INFO:tasks.cephfs.filesystem:scrub status for tag:c2c73d6c-aa01-44af-8d8a-5251e737d405 - {'path': '/', 'tag': 'c2c73d6c-aa01-44af-8d8a-5251e737d405', 'options': 'recursive,force'} 2024-04-02T14:38:33.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:38:32 smithi092 ceph-mon[36497]: scrub queued for path: / 2024-04-02T14:38:33.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:38:32 smithi092 ceph-mon[36497]: scrub summary: idle+waiting paths [/] 2024-04-02T14:38:33.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:38:32 smithi092 ceph-mon[36497]: scrub summary: active paths [/] 2024-04-02T14:38:33.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:38:32 smithi177 ceph-mon[36431]: scrub queued for path: / 2024-04-02T14:38:33.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:38:32 smithi177 ceph-mon[36431]: scrub summary: idle+waiting paths [/] 2024-04-02T14:38:33.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:38:32 smithi177 ceph-mon[36431]: scrub summary: active paths [/] 2024-04-02T14:38:33.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:38:32 smithi090 ceph-mon[31674]: scrub queued for path: / 2024-04-02T14:38:33.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:38:32 smithi090 ceph-mon[31674]: scrub summary: idle+waiting paths [/] 2024-04-02T14:38:33.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:38:32 smithi090 ceph-mon[31674]: scrub summary: active paths [/] 2024-04-02T14:38:34.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:38:33 smithi092 ceph-mon[36497]: pgmap v113: 97 pgs: 97 active+clean; 2.2 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 8.2 MiB/s wr, 706 op/s 2024-04-02T14:38:34.111 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:38:33 smithi092 ceph-mon[36497]: scrub summary: idle+waiting paths [/] 2024-04-02T14:38:34.189 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:38:33 smithi177 ceph-mon[36431]: pgmap v113: 97 pgs: 97 active+clean; 2.2 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 8.2 MiB/s wr, 706 op/s 2024-04-02T14:38:34.189 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:38:33 smithi177 ceph-mon[36431]: scrub summary: idle+waiting paths [/] 2024-04-02T14:38:34.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:38:33 smithi090 ceph-mon[31674]: pgmap v113: 97 pgs: 97 active+clean; 2.2 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 8.2 MiB/s wr, 706 op/s 2024-04-02T14:38:34.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:38:33 smithi090 ceph-mon[31674]: scrub summary: idle+waiting paths [/] 2024-04-02T14:38:36.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:38:35 smithi177 ceph-mon[36431]: pgmap v114: 97 pgs: 97 active+clean; 2.2 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 11 MiB/s wr, 1.09k op/s 2024-04-02T14:38:36.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:38:35 smithi090 ceph-mon[31674]: pgmap v114: 97 pgs: 97 active+clean; 2.2 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 11 MiB/s wr, 1.09k op/s 2024-04-02T14:38:36.362 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:38:35 smithi092 ceph-mon[36497]: pgmap v114: 97 pgs: 97 active+clean; 2.2 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 11 MiB/s wr, 1.09k op/s 2024-04-02T14:38:36.831 INFO:teuthology.orchestra.run.smithi090.stderr:progress: 185.0 s, 3599.5 tps, lat 27.700 ms stddev 33.556 2024-04-02T14:38:37.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:38:36 smithi177 ceph-mon[36431]: scrub summary: idle 2024-04-02T14:38:37.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:38:36 smithi090 ceph-mon[31674]: scrub summary: idle 2024-04-02T14:38:37.240 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.238+0000 7fa2f0e00640 1 -- 172.21.15.90:0/2266376749 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa2ec1048f0 msgr2=0x7fa2ec108930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:38:37.241 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.238+0000 7fa2f0e00640 1 --2- 172.21.15.90:0/2266376749 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa2ec1048f0 0x7fa2ec108930 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fa2d8009900 tx=0x7fa2d802bc30 comp rx=0 tx=0).stop 2024-04-02T14:38:37.241 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.239+0000 7fa2f0e00640 1 -- 172.21.15.90:0/2266376749 shutdown_connections 2024-04-02T14:38:37.241 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.239+0000 7fa2f0e00640 1 --2- 172.21.15.90:0/2266376749 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa2ec109010 0x7fa2ec110be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:37.241 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.239+0000 7fa2f0e00640 1 --2- 172.21.15.90:0/2266376749 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa2ec1048f0 0x7fa2ec108930 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:37.241 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.239+0000 7fa2f0e00640 1 --2- 172.21.15.90:0/2266376749 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa2ec103f40 0x7fa2ec104320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:37.241 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.239+0000 7fa2f0e00640 1 -- 172.21.15.90:0/2266376749 >> 172.21.15.90:0/2266376749 conn(0x7fa2ec0ff7c0 msgr2=0x7fa2ec101be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:38:37.242 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.240+0000 7fa2f0e00640 1 -- 172.21.15.90:0/2266376749 shutdown_connections 2024-04-02T14:38:37.242 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.240+0000 7fa2f0e00640 1 -- 172.21.15.90:0/2266376749 wait complete. 2024-04-02T14:38:37.243 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.242+0000 7fa2f0e00640 1 Processor -- start 2024-04-02T14:38:37.244 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.242+0000 7fa2f0e00640 1 -- start start 2024-04-02T14:38:37.244 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.242+0000 7fa2f0e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa2ec103f40 0x7fa2ec19a510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:37.244 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.242+0000 7fa2f0e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa2ec109010 0x7fa2ec19aa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:37.245 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.242+0000 7fa2f0e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa2ec19edd0 0x7fa2ec19f280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:37.245 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.242+0000 7fa2f0e00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fa2ec1140d0 con 0x7fa2ec103f40 2024-04-02T14:38:37.245 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.242+0000 7fa2f0e00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fa2ec113f50 con 0x7fa2ec109010 2024-04-02T14:38:37.245 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.242+0000 7fa2f0e00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fa2ec114250 con 0x7fa2ec19edd0 2024-04-02T14:38:37.245 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.243+0000 7fa2e9400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa2ec109010 0x7fa2ec19aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:37.245 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.243+0000 7fa2e9e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa2ec103f40 0x7fa2ec19a510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:37.246 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.243+0000 7fa2e9400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa2ec109010 0x7fa2ec19aa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:40144/0 (socket says 172.21.15.90:40144) 2024-04-02T14:38:37.246 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.243+0000 7fa2e9400640 1 -- 172.21.15.90:0/369862997 learned_addr learned my addr 172.21.15.90:0/369862997 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:38:37.246 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.243+0000 7fa2ea800640 1 --2- 172.21.15.90:0/369862997 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa2ec19edd0 0x7fa2ec19f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:37.246 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.244+0000 7fa2ea800640 1 -- 172.21.15.90:0/369862997 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa2ec109010 msgr2=0x7fa2ec19aa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:38:37.246 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.244+0000 7fa2ea800640 1 --2- 172.21.15.90:0/369862997 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa2ec109010 0x7fa2ec19aa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:37.246 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.244+0000 7fa2ea800640 1 -- 172.21.15.90:0/369862997 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa2ec103f40 msgr2=0x7fa2ec19a510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:38:37.246 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.244+0000 7fa2ea800640 1 --2- 172.21.15.90:0/369862997 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa2ec103f40 0x7fa2ec19a510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:37.247 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.244+0000 7fa2ea800640 1 -- 172.21.15.90:0/369862997 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2ec19f8c0 con 0x7fa2ec19edd0 2024-04-02T14:38:37.247 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.245+0000 7fa2ea800640 1 --2- 172.21.15.90:0/369862997 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa2ec19edd0 0x7fa2ec19f280 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fa2e400da10 tx=0x7fa2e400de10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:38:37.247 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.245+0000 7fa2e2a00640 1 -- 172.21.15.90:0/369862997 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2e400bd10 con 0x7fa2ec19edd0 2024-04-02T14:38:37.248 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.245+0000 7fa2e2a00640 1 -- 172.21.15.90:0/369862997 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa2e4003e50 con 0x7fa2ec19edd0 2024-04-02T14:38:37.248 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.245+0000 7fa2f0e00640 1 -- 172.21.15.90:0/369862997 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa2ec19fbb0 con 0x7fa2ec19edd0 2024-04-02T14:38:37.248 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.245+0000 7fa2e2a00640 1 -- 172.21.15.90:0/369862997 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2e40106c0 con 0x7fa2ec19edd0 2024-04-02T14:38:37.248 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.246+0000 7fa2f0e00640 1 -- 172.21.15.90:0/369862997 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2ec1a73a0 con 0x7fa2ec19edd0 2024-04-02T14:38:37.251 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.249+0000 7fa2f3a4b740 1 Processor -- start 2024-04-02T14:38:37.255 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.252+0000 7fa2e2a00640 1 -- 172.21.15.90:0/369862997 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fa2e4010860 con 0x7fa2ec19edd0 2024-04-02T14:38:37.256 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.254+0000 7fa2f3a4b740 1 -- start start 2024-04-02T14:38:37.257 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.255+0000 7fa2f3a4b740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5614d40e1eb0 0x5614d40e2290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:37.257 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.255+0000 7fa2f3a4b740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5614d40e27d0 0x5614d40eae70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:37.257 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.255+0000 7fa2f3a4b740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5614d40f3bc0 0x5614d40f5fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:37.257 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.255+0000 7fa2f3a4b740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5614d4046d70 con 0x5614d40e27d0 2024-04-02T14:38:37.257 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.255+0000 7fa2f3a4b740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x5614d3fef2a0 con 0x5614d40e1eb0 2024-04-02T14:38:37.257 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.255+0000 7fa2f3a4b740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5614d4042d20 con 0x5614d40f3bc0 2024-04-02T14:38:37.258 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.256+0000 7fa2e9400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5614d40e1eb0 0x5614d40e2290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:37.258 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.256+0000 7fa2e9400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5614d40e1eb0 0x5614d40e2290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:40148/0 (socket says 172.21.15.90:40148) 2024-04-02T14:38:37.259 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.256+0000 7fa2e9400640 1 -- 172.21.15.90:0/1281453134 learned_addr learned my addr 172.21.15.90:0/1281453134 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:38:37.259 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.256+0000 7fa2e2a00640 1 --2- 172.21.15.90:0/369862997 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fa2d00a3e90 0x7fa2d00a6350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:37.259 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.256+0000 7fa2e9e00640 1 --2- 172.21.15.90:0/1281453134 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5614d40e27d0 0x5614d40eae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:37.259 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.257+0000 7fa2e2a00640 1 -- 172.21.15.90:0/369862997 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7fa2e40d0a10 con 0x7fa2ec19edd0 2024-04-02T14:38:37.260 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.257+0000 7fa2e9400640 1 --2- 172.21.15.90:0/1281453134 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5614d40f3bc0 0x5614d40f5fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:37.260 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.257+0000 7fa2e9400640 1 -- 172.21.15.90:0/1281453134 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5614d40f3bc0 msgr2=0x5614d40f5fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:38:37.260 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.257+0000 7fa2e9400640 1 --2- 172.21.15.90:0/1281453134 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5614d40f3bc0 0x5614d40f5fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:37.260 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.257+0000 7fa2e9400640 1 -- 172.21.15.90:0/1281453134 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5614d40e27d0 msgr2=0x5614d40eae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:38:37.260 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.257+0000 7fa2e9400640 1 --2- 172.21.15.90:0/1281453134 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5614d40e27d0 0x5614d40eae70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:37.260 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.257+0000 7fa2e9400640 1 -- 172.21.15.90:0/1281453134 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5614d4003840 con 0x5614d40e1eb0 2024-04-02T14:38:37.260 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.257+0000 7fa2ea800640 1 --2- 172.21.15.90:0/369862997 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fa2d00a3e90 0x7fa2d00a6350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:37.260 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.257+0000 7fa2e9e00640 1 --2- 172.21.15.90:0/1281453134 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5614d40e27d0 0x5614d40eae70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:38:37.260 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.258+0000 7fa2e9400640 1 --2- 172.21.15.90:0/1281453134 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5614d40e1eb0 0x5614d40e2290 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fa2d80096c0 tx=0x7fa2d8005a20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=57d093711dc162e server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:38:37.261 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.258+0000 7fa2d7400640 1 -- 172.21.15.90:0/1281453134 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2d802c880 con 0x5614d40e1eb0 2024-04-02T14:38:37.261 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.258+0000 7fa2d7400640 1 -- 172.21.15.90:0/1281453134 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa2d8004ba0 con 0x5614d40e1eb0 2024-04-02T14:38:37.261 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.258+0000 7fa2d7400640 1 -- 172.21.15.90:0/1281453134 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2d80052c0 con 0x5614d40e1eb0 2024-04-02T14:38:37.261 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.259+0000 7fa2ea800640 1 --2- 172.21.15.90:0/369862997 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fa2d00a3e90 0x7fa2d00a6350 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fa2e40a5100 tx=0x7fa2e4004a40 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:38:37.263 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.261+0000 7fa2f3a4b740 1 -- 172.21.15.90:0/1281453134 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5614d40e1eb0 msgr2=0x5614d40e2290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:38:37.263 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.261+0000 7fa2f3a4b740 1 --2- 172.21.15.90:0/1281453134 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5614d40e1eb0 0x5614d40e2290 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fa2d80096c0 tx=0x7fa2d8005a20 comp rx=0 tx=0).stop 2024-04-02T14:38:37.263 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.261+0000 7fa2f3a4b740 1 -- 172.21.15.90:0/1281453134 shutdown_connections 2024-04-02T14:38:37.264 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.261+0000 7fa2f3a4b740 1 --2- 172.21.15.90:0/1281453134 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5614d40f3bc0 0x5614d40f5fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:37.264 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.261+0000 7fa2f3a4b740 1 --2- 172.21.15.90:0/1281453134 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5614d40e27d0 0x5614d40eae70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:37.264 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.261+0000 7fa2f3a4b740 1 --2- 172.21.15.90:0/1281453134 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5614d40e1eb0 0x5614d40e2290 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:37.264 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.261+0000 7fa2f3a4b740 1 -- 172.21.15.90:0/1281453134 >> 172.21.15.90:0/1281453134 conn(0x5614d40dc5e0 msgr2=0x5614d40dc9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:38:37.264 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.262+0000 7fa2f3a4b740 1 -- 172.21.15.90:0/1281453134 shutdown_connections 2024-04-02T14:38:37.264 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.262+0000 7fa2f3a4b740 1 -- 172.21.15.90:0/1281453134 wait complete. 2024-04-02T14:38:37.265 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.263+0000 7fa2f3a4b740 1 -- start start 2024-04-02T14:38:37.266 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.263+0000 7fa2f3a4b740 1 Processor -- start 2024-04-02T14:38:37.267 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.264+0000 7fa2f3a4b740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5614d40e1eb0 0x5614d40e8450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:37.267 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.264+0000 7fa2f3a4b740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5614d40e27d0 0x5614d40e8990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:37.267 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.264+0000 7fa2f3a4b740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5614d40f3bc0 0x5614d40e8ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:37.267 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.264+0000 7fa2f3a4b740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5614d40f6950 con 0x5614d40f3bc0 2024-04-02T14:38:37.267 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.264+0000 7fa2f3a4b740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x5614d4002660 con 0x5614d40e27d0 2024-04-02T14:38:37.267 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.264+0000 7fa2f3a4b740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5614d40f6ad0 con 0x5614d40e1eb0 2024-04-02T14:38:37.267 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.265+0000 7fa2f3a4b740 10 client.0 _ll_register_callbacks cb 0x5614d4002050 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:38:37.267 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.265+0000 7fa2e9400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5614d40e1eb0 0x5614d40e8450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:37.268 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.265+0000 7fa2e9400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5614d40e1eb0 0x5614d40e8450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:43488/0 (socket says 172.21.15.90:43488) 2024-04-02T14:38:37.268 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.265+0000 7fa2e9400640 1 -- 172.21.15.90:0/3469110755 learned_addr learned my addr 172.21.15.90:0/3469110755 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:38:37.268 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.265+0000 7fa2e9400640 1 --2- 172.21.15.90:0/3469110755 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5614d40f3bc0 0x5614d40e8ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:37.268 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.265+0000 7fa2e9e00640 1 --2- 172.21.15.90:0/3469110755 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5614d40e27d0 0x5614d40e8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:37.268 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.265+0000 7fa2e9400640 1 -- 172.21.15.90:0/3469110755 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5614d40e27d0 msgr2=0x5614d40e8990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:38:37.269 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.265+0000 7fa2e9400640 1 --2- 172.21.15.90:0/3469110755 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5614d40e27d0 0x5614d40e8990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:37.269 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.265+0000 7fa2e9400640 1 -- 172.21.15.90:0/3469110755 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5614d40f3bc0 msgr2=0x5614d40e8ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:38:37.269 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.265+0000 7fa2e9400640 1 --2- 172.21.15.90:0/3469110755 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5614d40f3bc0 0x5614d40e8ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:37.269 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.265+0000 7fa2e9400640 1 -- 172.21.15.90:0/3469110755 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5614d4011e80 con 0x5614d40e1eb0 2024-04-02T14:38:37.269 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.266+0000 7fa2e9400640 1 --2- 172.21.15.90:0/3469110755 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5614d40e1eb0 0x5614d40e8450 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fa2d8000c00 tx=0x7fa2d802df90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ce091b97b7c3e41f server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:38:37.269 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.266+0000 7fa2cbe00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:38:37.269 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.266+0000 7fa2cbe00640 1 -- 172.21.15.90:0/3469110755 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2d8005620 con 0x5614d40e1eb0 2024-04-02T14:38:37.269 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.267+0000 7fa2cbe00640 1 -- 172.21.15.90:0/3469110755 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7fa2d80046c0 con 0x5614d40e1eb0 2024-04-02T14:38:37.270 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.268+0000 7fa2f3a4b740 1 -- 172.21.15.90:0/3469110755 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa2d8004ba0 con 0x5614d40e1eb0 2024-04-02T14:38:37.271 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.268+0000 7fa2cbe00640 1 -- 172.21.15.90:0/3469110755 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fa2d80048c0 con 0x5614d40e1eb0 2024-04-02T14:38:37.271 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.268+0000 7fa2f3a4b740 10 client.26485 fetch_fsmap learned FSMap version 27 2024-04-02T14:38:37.271 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.268+0000 7fa2f3a4b740 1 -- 172.21.15.90:0/3469110755 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5614d3d5b210 con 0x5614d40e1eb0 2024-04-02T14:38:37.271 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.268+0000 7fa2cbe00640 1 -- 172.21.15.90:0/3469110755 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa2d8004ba0 con 0x5614d40e1eb0 2024-04-02T14:38:37.271 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.269+0000 7fa2cbe00640 1 -- 172.21.15.90:0/3469110755 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fa2d8003b90 con 0x5614d40e1eb0 2024-04-02T14:38:37.271 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.269+0000 7fa2f3a4b740 10 client.26485 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:38:37.271 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.269+0000 7fa2f3a4b740 10 client.26485 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-02T14:38:37.271 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.269+0000 7fa2f3a4b740 20 client.26485 populate_metadata read hostname 'smithi090' 2024-04-02T14:38:37.271 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.269+0000 7fa2f3a4b740 1 --2- 172.21.15.90:0/3469110755 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5614d4103dd0 0x5614d41061c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:37.271 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.269+0000 7fa2f3a4b740 4 client.26485 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:38:37.271 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.269+0000 7fa2f3a4b740 1 -- 172.21.15.90:0/3469110755 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5614d3ff6cd0 con 0x5614d4103dd0 2024-04-02T14:38:37.272 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.269+0000 7fa2e9e00640 1 --2- 172.21.15.90:0/3469110755 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5614d4103dd0 0x5614d41061c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:37.272 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.270+0000 7fa2e9e00640 1 --2- 172.21.15.90:0/3469110755 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5614d4103dd0 0x5614d41061c0 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6323c9fef8b78ea6 server_cookie=59d32977854385c7 in_seq=0 out_seq=0 2024-04-02T14:38:37.272 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.270+0000 7fa2cbe00640 10 client.26485 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:38:37.273 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.271+0000 7fa2cbe00640 1 -- 172.21.15.90:0/3469110755 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5614d3ff6cd0 con 0x5614d4103dd0 2024-04-02T14:38:37.273 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.271+0000 7fa2cbe00640 10 client.26485 handle_command_reply: tid=0 2024-04-02T14:38:37.274 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.271+0000 7fa2f3a4b740 1 client.26485 shutdown 2024-04-02T14:38:37.274 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.271+0000 7fa2f3a4b740 1 -- 172.21.15.90:0/3469110755 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5614d40e1eb0 msgr2=0x5614d40e8450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:38:37.274 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.271+0000 7fa2f3a4b740 1 --2- 172.21.15.90:0/3469110755 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5614d40e1eb0 0x5614d40e8450 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fa2d8000c00 tx=0x7fa2d802df90 comp rx=0 tx=0).stop 2024-04-02T14:38:37.274 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.271+0000 7fa2f3a4b740 1 -- 172.21.15.90:0/3469110755 shutdown_connections 2024-04-02T14:38:37.274 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.271+0000 7fa2f3a4b740 1 --2- 172.21.15.90:0/3469110755 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5614d4103dd0 0x5614d41061c0 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:37.274 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.271+0000 7fa2f3a4b740 1 --2- 172.21.15.90:0/3469110755 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5614d40f3bc0 0x5614d40e8ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:37.274 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.271+0000 7fa2f3a4b740 1 --2- 172.21.15.90:0/3469110755 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5614d40e27d0 0x5614d40e8990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:37.274 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.271+0000 7fa2f3a4b740 1 --2- 172.21.15.90:0/3469110755 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5614d40e1eb0 0x5614d40e8450 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:37.274 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.271+0000 7fa2f3a4b740 1 -- 172.21.15.90:0/3469110755 >> 172.21.15.90:0/3469110755 conn(0x5614d40dc5e0 msgr2=0x5614d40de330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:38:37.274 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.271+0000 7fa2cbe00640 0 client.26485 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:38:37.274 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.271+0000 7fa2f3a4b740 1 -- 172.21.15.90:0/3469110755 shutdown_connections 2024-04-02T14:38:37.274 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.271+0000 7fa2f3a4b740 1 -- 172.21.15.90:0/3469110755 wait complete. 2024-04-02T14:38:37.275 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.272+0000 7fa2f3a4b740 20 client.26485 trim_cache size 0 max 16384 2024-04-02T14:38:37.306 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.303+0000 7fa2f3a4b740 1 Processor -- start 2024-04-02T14:38:37.307 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.303+0000 7fa2f3a4b740 1 -- start start 2024-04-02T14:38:37.307 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.303+0000 7fa2f3a4b740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5614d40e1eb0 0x5614d40eab80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:37.307 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.303+0000 7fa2f3a4b740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5614d40e27d0 0x5614d40ff430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:37.307 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.303+0000 7fa2f3a4b740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5614d40f3bc0 0x5614d4108e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:37.307 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.303+0000 7fa2f3a4b740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fa2ec1ab4a0 con 0x5614d40e1eb0 2024-04-02T14:38:37.307 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.303+0000 7fa2f3a4b740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fa2ec1acfe0 con 0x5614d40f3bc0 2024-04-02T14:38:37.308 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.303+0000 7fa2f3a4b740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fa2ec1abb70 con 0x5614d40e27d0 2024-04-02T14:38:37.308 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.304+0000 7fa2e9400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5614d40f3bc0 0x5614d4108e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:37.308 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.305+0000 7fa2e9e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5614d40e27d0 0x5614d40ff430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:37.308 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.305+0000 7fa2e9e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5614d40e27d0 0x5614d40ff430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:43496/0 (socket says 172.21.15.90:43496) 2024-04-02T14:38:37.309 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.305+0000 7fa2e9e00640 1 -- 172.21.15.90:0/3362017152 learned_addr learned my addr 172.21.15.90:0/3362017152 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:38:37.309 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.306+0000 7fa2e9400640 1 --2- 172.21.15.90:0/3362017152 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5614d40e1eb0 0x5614d40eab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:37.309 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.306+0000 7fa2e9400640 1 -- 172.21.15.90:0/3362017152 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5614d40e27d0 msgr2=0x5614d40ff430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:38:37.309 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.306+0000 7fa2e9400640 1 --2- 172.21.15.90:0/3362017152 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5614d40e27d0 0x5614d40ff430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:37.309 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.306+0000 7fa2e9400640 1 -- 172.21.15.90:0/3362017152 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5614d40e1eb0 msgr2=0x5614d40eab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:38:37.309 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.306+0000 7fa2e9400640 1 --2- 172.21.15.90:0/3362017152 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5614d40e1eb0 0x5614d40eab80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:37.309 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.306+0000 7fa2e9400640 1 -- 172.21.15.90:0/3362017152 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5614d40e7400 con 0x5614d40f3bc0 2024-04-02T14:38:37.309 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.306+0000 7fa2e9400640 1 --2- 172.21.15.90:0/3362017152 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5614d40f3bc0 0x5614d4108e00 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fa2d80096c0 tx=0x7fa2d802bc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=90df30b1967147f4 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:38:37.310 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.308+0000 7fa2d6a00640 1 -- 172.21.15.90:0/3362017152 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2d8005250 con 0x5614d40f3bc0 2024-04-02T14:38:37.310 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.308+0000 7fa2d6a00640 1 -- 172.21.15.90:0/3362017152 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa2d8034690 con 0x5614d40f3bc0 2024-04-02T14:38:37.310 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.308+0000 7fa2d6a00640 1 -- 172.21.15.90:0/3362017152 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2d80041f0 con 0x5614d40f3bc0 2024-04-02T14:38:37.313 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.311+0000 7fa2f3a4b740 1 -- 172.21.15.90:0/3362017152 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5614d40f3bc0 msgr2=0x5614d4108e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:38:37.313 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.311+0000 7fa2f3a4b740 1 --2- 172.21.15.90:0/3362017152 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5614d40f3bc0 0x5614d4108e00 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fa2d80096c0 tx=0x7fa2d802bc30 comp rx=0 tx=0).stop 2024-04-02T14:38:37.314 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.311+0000 7fa2f3a4b740 1 -- 172.21.15.90:0/3362017152 shutdown_connections 2024-04-02T14:38:37.314 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.311+0000 7fa2f3a4b740 1 --2- 172.21.15.90:0/3362017152 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5614d40f3bc0 0x5614d4108e00 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:37.314 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.311+0000 7fa2f3a4b740 1 --2- 172.21.15.90:0/3362017152 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5614d40e27d0 0x5614d40ff430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:37.314 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.311+0000 7fa2f3a4b740 1 --2- 172.21.15.90:0/3362017152 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5614d40e1eb0 0x5614d40eab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:37.314 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.311+0000 7fa2f3a4b740 1 -- 172.21.15.90:0/3362017152 >> 172.21.15.90:0/3362017152 conn(0x5614d40dc5e0 msgr2=0x5614d40fed20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:38:37.314 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.311+0000 7fa2f3a4b740 1 -- 172.21.15.90:0/3362017152 shutdown_connections 2024-04-02T14:38:37.314 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.311+0000 7fa2f3a4b740 1 -- 172.21.15.90:0/3362017152 wait complete. 2024-04-02T14:38:37.315 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.312+0000 7fa2f3a4b740 1 -- start start 2024-04-02T14:38:37.316 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.313+0000 7fa2f3a4b740 1 Processor -- start 2024-04-02T14:38:37.319 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.317+0000 7fa2f3a4b740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5614d40e1eb0 0x5614d4106860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:37.320 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.318+0000 7fa2e9400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5614d40e1eb0 0x5614d4106860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:37.320 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.318+0000 7fa2f3a4b740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5614d40e27d0 0x5614d4106da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:37.320 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.318+0000 7fa2e9400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5614d40e1eb0 0x5614d4106860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:43506/0 (socket says 172.21.15.90:43506) 2024-04-02T14:38:37.321 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.318+0000 7fa2e9400640 1 -- 172.21.15.90:0/2207845485 learned_addr learned my addr 172.21.15.90:0/2207845485 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:38:37.321 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.319+0000 7fa2e9e00640 1 --2- 172.21.15.90:0/2207845485 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5614d40e27d0 0x5614d4106da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:37.322 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.320+0000 7fa2f3a4b740 1 --2- 172.21.15.90:0/2207845485 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5614d40f3bc0 0x5614d41072e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:37.322 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.320+0000 7fa2f3a4b740 1 -- 172.21.15.90:0/2207845485 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fa2ec1ac910 con 0x5614d40f3bc0 2024-04-02T14:38:37.322 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.320+0000 7fa2f3a4b740 1 -- 172.21.15.90:0/2207845485 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fa2ec1ac240 con 0x5614d40e27d0 2024-04-02T14:38:37.322 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.320+0000 7fa2f3a4b740 1 -- 172.21.15.90:0/2207845485 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fa2ec1a9480 con 0x5614d40e1eb0 2024-04-02T14:38:37.323 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.320+0000 7fa2f3a4b740 10 client.0 _ll_register_callbacks cb 0x7fa2d8004e50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:38:37.323 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.321+0000 7fa2e9e00640 1 -- 172.21.15.90:0/2207845485 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5614d40e1eb0 msgr2=0x5614d4106860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:38:37.324 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.321+0000 7fa2e9e00640 1 --2- 172.21.15.90:0/2207845485 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5614d40e1eb0 0x5614d4106860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:37.324 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.321+0000 7fa2e9e00640 1 -- 172.21.15.90:0/2207845485 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5614d40f3bc0 msgr2=0x5614d41072e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:38:37.324 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.321+0000 7fa2e9e00640 1 --2- 172.21.15.90:0/2207845485 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5614d40f3bc0 0x5614d41072e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:37.324 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.321+0000 7fa2e9e00640 1 -- 172.21.15.90:0/2207845485 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5614d3e27620 con 0x5614d40e27d0 2024-04-02T14:38:37.324 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.322+0000 7fa2e9e00640 1 --2- 172.21.15.90:0/2207845485 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5614d40e27d0 0x5614d4106da0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x5614d40e98e0 tx=0x7fa2dc01f900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=48390b281134023e server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:38:37.325 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.323+0000 7fa2cbe00640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:38:37.325 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.324+0000 7fa2cbe00640 1 -- 172.21.15.90:0/2207845485 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2dc005020 con 0x5614d40e27d0 2024-04-02T14:38:37.326 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.324+0000 7fa2f3a4b740 1 -- 172.21.15.90:0/2207845485 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa2d8034690 con 0x5614d40e27d0 2024-04-02T14:38:37.326 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.324+0000 7fa2cbe00640 1 -- 172.21.15.90:0/2207845485 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7fa2dc011d00 con 0x5614d40e27d0 2024-04-02T14:38:37.327 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.325+0000 7fa2cbe00640 1 -- 172.21.15.90:0/2207845485 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fa2dc005020 con 0x5614d40e27d0 2024-04-02T14:38:37.327 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.325+0000 7fa2f3a4b740 10 client.26603 fetch_fsmap learned FSMap version 27 2024-04-02T14:38:37.327 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.325+0000 7fa2f3a4b740 1 -- 172.21.15.90:0/2207845485 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5614d416a140 con 0x5614d40e27d0 2024-04-02T14:38:37.328 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.325+0000 7fa2cbe00640 1 -- 172.21.15.90:0/2207845485 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa2d8034690 con 0x5614d40e27d0 2024-04-02T14:38:37.328 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.326+0000 7fa2cbe00640 1 -- 172.21.15.90:0/2207845485 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fa2dc031420 con 0x5614d40e27d0 2024-04-02T14:38:37.328 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.326+0000 7fa2f3a4b740 10 client.26603 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:38:37.328 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.326+0000 7fa2f3a4b740 10 client.26603 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-02T14:38:37.328 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.326+0000 7fa2f3a4b740 20 client.26603 populate_metadata read hostname 'smithi090' 2024-04-02T14:38:37.328 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.326+0000 7fa2f3a4b740 1 --2- 172.21.15.90:0/2207845485 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5614d416a820 0x5614d416cc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:37.330 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.326+0000 7fa2f3a4b740 4 client.26603 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-02T14:38:37.331 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.326+0000 7fa2f3a4b740 1 -- 172.21.15.90:0/2207845485 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5614d40542b0 con 0x5614d416a820 2024-04-02T14:38:37.331 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.326+0000 7fa2e9400640 1 --2- 172.21.15.90:0/2207845485 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5614d416a820 0x5614d416cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:37.331 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.327+0000 7fa2e9400640 1 --2- 172.21.15.90:0/2207845485 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5614d416a820 0x5614d416cc40 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4dcec421cc236e3d server_cookie=b5fd7bd351a5cd81 in_seq=0 out_seq=0 2024-04-02T14:38:37.331 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.327+0000 7fa2cbe00640 10 client.26603 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:38:37.360 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:38:36 smithi092 ceph-mon[36497]: scrub summary: idle 2024-04-02T14:38:37.363 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.361+0000 7fa2cbe00640 1 -- 172.21.15.90:0/2207845485 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5614d40542b0 con 0x5614d416a820 2024-04-02T14:38:37.363 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.361+0000 7fa2cbe00640 10 client.26603 handle_command_reply: tid=0 2024-04-02T14:38:37.363 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.361+0000 7fa2f3a4b740 1 client.26603 shutdown 2024-04-02T14:38:37.364 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.361+0000 7fa2f3a4b740 1 -- 172.21.15.90:0/2207845485 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5614d40e27d0 msgr2=0x5614d4106da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:38:37.364 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.361+0000 7fa2f3a4b740 1 --2- 172.21.15.90:0/2207845485 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5614d40e27d0 0x5614d4106da0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x5614d40e98e0 tx=0x7fa2dc01f900 comp rx=0 tx=0).stop 2024-04-02T14:38:37.365 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.362+0000 7fa2f3a4b740 1 -- 172.21.15.90:0/2207845485 shutdown_connections 2024-04-02T14:38:37.365 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.362+0000 7fa2f3a4b740 1 --2- 172.21.15.90:0/2207845485 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5614d416a820 0x5614d416cc40 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:37.365 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.362+0000 7fa2f3a4b740 1 --2- 172.21.15.90:0/2207845485 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5614d40f3bc0 0x5614d41072e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:37.365 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.362+0000 7fa2f3a4b740 1 --2- 172.21.15.90:0/2207845485 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5614d40e27d0 0x5614d4106da0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:37.365 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.362+0000 7fa2f3a4b740 1 --2- 172.21.15.90:0/2207845485 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5614d40e1eb0 0x5614d4106860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:37.365 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.362+0000 7fa2f3a4b740 1 -- 172.21.15.90:0/2207845485 >> 172.21.15.90:0/2207845485 conn(0x5614d40dc5e0 msgr2=0x5614d40e8ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:38:37.365 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.362+0000 7fa2cbe00640 0 client.26603 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:38:37.365 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.362+0000 7fa2f3a4b740 1 -- 172.21.15.90:0/2207845485 shutdown_connections 2024-04-02T14:38:37.365 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.362+0000 7fa2f3a4b740 1 -- 172.21.15.90:0/2207845485 wait complete. 2024-04-02T14:38:37.365 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.363+0000 7fa2f3a4b740 20 client.26603 trim_cache size 0 max 16384 2024-04-02T14:38:37.366 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:{ 2024-04-02T14:38:37.366 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "message": "", 2024-04-02T14:38:37.366 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "return_code": 0 2024-04-02T14:38:37.366 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:} 2024-04-02T14:38:37.366 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.363+0000 7fa2f0e00640 1 -- 172.21.15.90:0/369862997 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fa2d00a3e90 msgr2=0x7fa2d00a6350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:38:37.366 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.363+0000 7fa2f0e00640 1 --2- 172.21.15.90:0/369862997 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fa2d00a3e90 0x7fa2d00a6350 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fa2e40a5100 tx=0x7fa2e4004a40 comp rx=0 tx=0).stop 2024-04-02T14:38:37.367 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.363+0000 7fa2f0e00640 1 -- 172.21.15.90:0/369862997 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa2ec19edd0 msgr2=0x7fa2ec19f280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:38:37.367 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.363+0000 7fa2f0e00640 1 --2- 172.21.15.90:0/369862997 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa2ec19edd0 0x7fa2ec19f280 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fa2e400da10 tx=0x7fa2e400de10 comp rx=0 tx=0).stop 2024-04-02T14:38:37.367 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.364+0000 7fa2f0e00640 1 -- 172.21.15.90:0/369862997 shutdown_connections 2024-04-02T14:38:37.367 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.364+0000 7fa2f0e00640 1 --2- 172.21.15.90:0/369862997 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fa2d00a3e90 0x7fa2d00a6350 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:37.367 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.364+0000 7fa2f0e00640 1 --2- 172.21.15.90:0/369862997 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa2ec19edd0 0x7fa2ec19f280 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:37.367 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.364+0000 7fa2f0e00640 1 --2- 172.21.15.90:0/369862997 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa2ec109010 0x7fa2ec19aa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:37.367 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.364+0000 7fa2f0e00640 1 --2- 172.21.15.90:0/369862997 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa2ec103f40 0x7fa2ec19a510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:37.367 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.364+0000 7fa2f0e00640 1 -- 172.21.15.90:0/369862997 >> 172.21.15.90:0/369862997 conn(0x7fa2ec0ff7c0 msgr2=0x7fa2ec100310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:38:37.367 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.364+0000 7fa2f0e00640 1 -- 172.21.15.90:0/369862997 shutdown_connections 2024-04-02T14:38:37.367 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:37.364+0000 7fa2f0e00640 1 -- 172.21.15.90:0/369862997 wait complete. 2024-04-02T14:38:38.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:38:37 smithi177 ceph-mon[36431]: pgmap v115: 97 pgs: 97 active+clean; 2.2 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 7.9 MiB/s wr, 769 op/s 2024-04-02T14:38:38.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:38:37 smithi090 ceph-mon[31674]: pgmap v115: 97 pgs: 97 active+clean; 2.2 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 7.9 MiB/s wr, 769 op/s 2024-04-02T14:38:38.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:38:37 smithi092 ceph-mon[36497]: pgmap v115: 97 pgs: 97 active+clean; 2.2 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 7.9 MiB/s wr, 769 op/s 2024-04-02T14:38:40.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:38:39 smithi177 ceph-mon[36431]: pgmap v116: 97 pgs: 97 active+clean; 2.2 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 12 MiB/s wr, 1.08k op/s 2024-04-02T14:38:40.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:38:39 smithi090 ceph-mon[31674]: pgmap v116: 97 pgs: 97 active+clean; 2.2 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 12 MiB/s wr, 1.08k op/s 2024-04-02T14:38:40.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:38:39 smithi092 ceph-mon[36497]: pgmap v116: 97 pgs: 97 active+clean; 2.2 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 12 MiB/s wr, 1.08k op/s 2024-04-02T14:38:41.831 INFO:teuthology.orchestra.run.smithi090.stderr:progress: 190.0 s, 3108.4 tps, lat 32.190 ms stddev 38.617 2024-04-02T14:38:42.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:38:41 smithi177 ceph-mon[36431]: pgmap v117: 97 pgs: 97 active+clean; 2.2 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 10 MiB/s wr, 985 op/s 2024-04-02T14:38:42.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:38:41 smithi090 ceph-mon[31674]: pgmap v117: 97 pgs: 97 active+clean; 2.2 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 10 MiB/s wr, 985 op/s 2024-04-02T14:38:42.360 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:38:41 smithi092 ceph-mon[36497]: pgmap v117: 97 pgs: 97 active+clean; 2.2 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 10 MiB/s wr, 985 op/s 2024-04-02T14:38:44.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:38:43 smithi177 ceph-mon[36431]: pgmap v118: 97 pgs: 97 active+clean; 2.2 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 8.1 MiB/s wr, 799 op/s 2024-04-02T14:38:44.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:38:43 smithi090 ceph-mon[31674]: pgmap v118: 97 pgs: 97 active+clean; 2.2 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 8.1 MiB/s wr, 799 op/s 2024-04-02T14:38:44.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:38:43 smithi092 ceph-mon[36497]: pgmap v118: 97 pgs: 97 active+clean; 2.2 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 8.1 MiB/s wr, 799 op/s 2024-04-02T14:38:46.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:38:45 smithi177 ceph-mon[36431]: pgmap v119: 97 pgs: 97 active+clean; 2.2 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 12 MiB/s wr, 1.11k op/s 2024-04-02T14:38:46.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:38:45 smithi090 ceph-mon[31674]: pgmap v119: 97 pgs: 97 active+clean; 2.2 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 12 MiB/s wr, 1.11k op/s 2024-04-02T14:38:46.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:38:45 smithi092 ceph-mon[36497]: pgmap v119: 97 pgs: 97 active+clean; 2.2 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 12 MiB/s wr, 1.11k op/s 2024-04-02T14:38:46.831 INFO:teuthology.orchestra.run.smithi090.stderr:progress: 195.0 s, 3062.6 tps, lat 32.526 ms stddev 38.200 2024-04-02T14:38:48.187 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:38:47 smithi177 ceph-mon[36431]: pgmap v120: 97 pgs: 97 active+clean; 2.2 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 9.0 MiB/s wr, 727 op/s 2024-04-02T14:38:49.338 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:38:47 smithi092 ceph-mon[36497]: pgmap v120: 97 pgs: 97 active+clean; 2.2 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 9.0 MiB/s wr, 727 op/s 2024-04-02T14:38:49.338 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:38:47 smithi090 ceph-mon[31674]: pgmap v120: 97 pgs: 97 active+clean; 2.2 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 9.0 MiB/s wr, 727 op/s 2024-04-02T14:38:50.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:38:49 smithi177 ceph-mon[36431]: pgmap v121: 97 pgs: 97 active+clean; 2.3 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 12 MiB/s wr, 873 op/s 2024-04-02T14:38:50.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:38:49 smithi090 ceph-mon[31674]: pgmap v121: 97 pgs: 97 active+clean; 2.3 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 12 MiB/s wr, 873 op/s 2024-04-02T14:38:50.360 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:38:49 smithi092 ceph-mon[36497]: pgmap v121: 97 pgs: 97 active+clean; 2.3 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 12 MiB/s wr, 873 op/s 2024-04-02T14:38:50.564 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.562+0000 7fcf6e800640 1 -- 172.21.15.90:0/2006738453 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcf68109420 msgr2=0x7fcf68114710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:38:50.565 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.562+0000 7fcf6e800640 1 --2- 172.21.15.90:0/2006738453 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcf68109420 0x7fcf68114710 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fcf50009930 tx=0x7fcf5002bcd0 comp rx=0 tx=0).stop 2024-04-02T14:38:50.565 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.562+0000 7fcf6e800640 1 -- 172.21.15.90:0/2006738453 shutdown_connections 2024-04-02T14:38:50.565 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.562+0000 7fcf6e800640 1 --2- 172.21.15.90:0/2006738453 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcf68114c50 0x7fcf68117040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:50.565 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.562+0000 7fcf6e800640 1 --2- 172.21.15.90:0/2006738453 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcf68109420 0x7fcf68114710 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:50.565 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.562+0000 7fcf6e800640 1 --2- 172.21.15.90:0/2006738453 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcf68108b00 0x7fcf68108ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:50.565 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.562+0000 7fcf6e800640 1 -- 172.21.15.90:0/2006738453 >> 172.21.15.90:0/2006738453 conn(0x7fcf6806d030 msgr2=0x7fcf6806d440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:38:50.566 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.563+0000 7fcf6e800640 1 -- 172.21.15.90:0/2006738453 shutdown_connections 2024-04-02T14:38:50.566 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.563+0000 7fcf6e800640 1 -- 172.21.15.90:0/2006738453 wait complete. 2024-04-02T14:38:50.567 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.565+0000 7fcf6e800640 1 Processor -- start 2024-04-02T14:38:50.567 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.565+0000 7fcf6e800640 1 -- start start 2024-04-02T14:38:50.568 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.566+0000 7fcf6e800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcf68108b00 0x7fcf681108c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:50.568 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.566+0000 7fcf6e800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcf68109420 0x7fcf68110e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:50.568 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.566+0000 7fcf6e800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcf68114c50 0x7fcf68111340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:50.568 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.566+0000 7fcf6e800640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fcf6810b150 con 0x7fcf68109420 2024-04-02T14:38:50.568 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.566+0000 7fcf6e800640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fcf6810afd0 con 0x7fcf68114c50 2024-04-02T14:38:50.568 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.566+0000 7fcf6e800640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fcf6810b2d0 con 0x7fcf68108b00 2024-04-02T14:38:50.570 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.567+0000 7fcf5ea00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcf68109420 0x7fcf68110e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:50.570 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.568+0000 7fcf67e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcf68114c50 0x7fcf68111340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:50.571 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.568+0000 7fcf5ea00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcf68109420 0x7fcf68110e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:55710/0 (socket says 172.21.15.90:55710) 2024-04-02T14:38:50.571 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.568+0000 7fcf5ea00640 1 -- 172.21.15.90:0/3927783890 learned_addr learned my addr 172.21.15.90:0/3927783890 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:38:50.572 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.570+0000 7fcf67400640 1 --2- 172.21.15.90:0/3927783890 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcf68108b00 0x7fcf681108c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:50.574 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.571+0000 7fcf67e00640 1 -- 172.21.15.90:0/3927783890 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcf68108b00 msgr2=0x7fcf681108c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:38:50.574 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.571+0000 7fcf67e00640 1 --2- 172.21.15.90:0/3927783890 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcf68108b00 0x7fcf681108c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:50.574 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.571+0000 7fcf67e00640 1 -- 172.21.15.90:0/3927783890 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcf68109420 msgr2=0x7fcf68110e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:38:50.574 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.571+0000 7fcf67e00640 1 --2- 172.21.15.90:0/3927783890 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcf68109420 0x7fcf68110e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:50.574 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.571+0000 7fcf67e00640 1 -- 172.21.15.90:0/3927783890 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf681b3480 con 0x7fcf68114c50 2024-04-02T14:38:50.575 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.573+0000 7fcf67e00640 1 --2- 172.21.15.90:0/3927783890 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcf68114c50 0x7fcf68111340 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fcf6000d680 tx=0x7fcf6000db40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:38:50.580 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.577+0000 7fcf64c00640 1 -- 172.21.15.90:0/3927783890 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf60017070 con 0x7fcf68114c50 2024-04-02T14:38:50.580 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.577+0000 7fcf64c00640 1 -- 172.21.15.90:0/3927783890 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcf60009820 con 0x7fcf68114c50 2024-04-02T14:38:50.580 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.577+0000 7fcf64c00640 1 -- 172.21.15.90:0/3927783890 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf60004ce0 con 0x7fcf68114c50 2024-04-02T14:38:50.580 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.578+0000 7fcf6e800640 1 -- 172.21.15.90:0/3927783890 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcf681b3770 con 0x7fcf68114c50 2024-04-02T14:38:50.582 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.578+0000 7fcf6e800640 1 -- 172.21.15.90:0/3927783890 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcf681b3c40 con 0x7fcf68114c50 2024-04-02T14:38:50.588 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.586+0000 7fcf64c00640 1 -- 172.21.15.90:0/3927783890 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fcf60010460 con 0x7fcf68114c50 2024-04-02T14:38:50.591 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.589+0000 7fcf715b2740 1 Processor -- start 2024-04-02T14:38:50.592 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.590+0000 7fcf64c00640 1 --2- 172.21.15.90:0/3927783890 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fcf4c0a3f20 0x7fcf4c0a63e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:50.593 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.590+0000 7fcf64c00640 1 -- 172.21.15.90:0/3927783890 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7fcf600d0f50 con 0x7fcf68114c50 2024-04-02T14:38:50.593 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.591+0000 7fcf5ea00640 1 --2- 172.21.15.90:0/3927783890 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fcf4c0a3f20 0x7fcf4c0a63e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:50.593 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.591+0000 7fcf715b2740 1 -- start start 2024-04-02T14:38:50.594 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.592+0000 7fcf715b2740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e61b391eb0 0x55e61b392290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:50.594 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.592+0000 7fcf715b2740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e61b3927d0 0x55e61b39ae70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:50.594 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.592+0000 7fcf715b2740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e61b39b3b0 0x55e61b39d7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:50.594 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.592+0000 7fcf715b2740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55e61b2f6d70 con 0x55e61b39b3b0 2024-04-02T14:38:50.594 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.592+0000 7fcf715b2740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55e61b29f2a0 con 0x55e61b391eb0 2024-04-02T14:38:50.594 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.592+0000 7fcf715b2740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55e61b2f2d20 con 0x55e61b3927d0 2024-04-02T14:38:50.595 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.592+0000 7fcf67400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e61b391eb0 0x55e61b392290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:50.595 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.592+0000 7fcf67400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e61b391eb0 0x55e61b392290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:58258/0 (socket says 172.21.15.90:58258) 2024-04-02T14:38:50.595 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.592+0000 7fcf67400640 1 -- 172.21.15.90:0/1896384574 learned_addr learned my addr 172.21.15.90:0/1896384574 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:38:50.595 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.592+0000 7fcf5ea00640 1 --2- 172.21.15.90:0/1896384574 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e61b3927d0 0x55e61b39ae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:50.595 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.593+0000 7fcf67e00640 1 --2- 172.21.15.90:0/1896384574 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e61b39b3b0 0x55e61b39d7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:50.597 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.594+0000 7fcf67e00640 1 -- 172.21.15.90:0/1896384574 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e61b3927d0 msgr2=0x55e61b39ae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:38:50.597 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.594+0000 7fcf67e00640 1 --2- 172.21.15.90:0/1896384574 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e61b3927d0 0x55e61b39ae70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:50.597 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.594+0000 7fcf67e00640 1 -- 172.21.15.90:0/1896384574 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e61b391eb0 msgr2=0x55e61b392290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:38:50.597 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.594+0000 7fcf67e00640 1 --2- 172.21.15.90:0/1896384574 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e61b391eb0 0x55e61b392290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:50.597 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.594+0000 7fcf67e00640 1 -- 172.21.15.90:0/1896384574 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e61b2b3840 con 0x55e61b39b3b0 2024-04-02T14:38:50.598 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.595+0000 7fcf67400640 1 --2- 172.21.15.90:0/1896384574 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e61b391eb0 0x55e61b392290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:38:50.598 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.595+0000 7fcf5ea00640 1 --2- 172.21.15.90:0/1896384574 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e61b3927d0 0x55e61b39ae70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:38:50.598 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.596+0000 7fcf5ea00640 1 --2- 172.21.15.90:0/3927783890 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fcf4c0a3f20 0x7fcf4c0a63e0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fcf68112110 tx=0x7fcf50005e10 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:38:50.599 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.597+0000 7fcf67e00640 1 --2- 172.21.15.90:0/1896384574 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e61b39b3b0 0x55e61b39d7a0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fcf600ce650 tx=0x7fcf6009cf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d0f893cd9b948b0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:38:50.600 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.597+0000 7fcf5cc00640 1 -- 172.21.15.90:0/1896384574 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf60014030 con 0x55e61b39b3b0 2024-04-02T14:38:50.600 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.598+0000 7fcf5cc00640 1 -- 172.21.15.90:0/1896384574 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcf600a5b10 con 0x55e61b39b3b0 2024-04-02T14:38:50.600 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.598+0000 7fcf5cc00640 1 -- 172.21.15.90:0/1896384574 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf60020040 con 0x55e61b39b3b0 2024-04-02T14:38:50.601 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.599+0000 7fcf715b2740 1 -- 172.21.15.90:0/1896384574 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e61b39b3b0 msgr2=0x55e61b39d7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:38:50.601 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.599+0000 7fcf715b2740 1 --2- 172.21.15.90:0/1896384574 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e61b39b3b0 0x55e61b39d7a0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fcf600ce650 tx=0x7fcf6009cf50 comp rx=0 tx=0).stop 2024-04-02T14:38:50.601 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.599+0000 7fcf715b2740 1 -- 172.21.15.90:0/1896384574 shutdown_connections 2024-04-02T14:38:50.601 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.599+0000 7fcf715b2740 1 --2- 172.21.15.90:0/1896384574 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e61b39b3b0 0x55e61b39d7a0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:50.601 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.599+0000 7fcf715b2740 1 --2- 172.21.15.90:0/1896384574 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e61b3927d0 0x55e61b39ae70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:50.601 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.599+0000 7fcf715b2740 1 --2- 172.21.15.90:0/1896384574 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e61b391eb0 0x55e61b392290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:50.601 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.599+0000 7fcf715b2740 1 -- 172.21.15.90:0/1896384574 >> 172.21.15.90:0/1896384574 conn(0x55e61b38c5e0 msgr2=0x55e61b38c9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:38:50.601 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.599+0000 7fcf715b2740 1 -- 172.21.15.90:0/1896384574 shutdown_connections 2024-04-02T14:38:50.602 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.599+0000 7fcf715b2740 1 -- 172.21.15.90:0/1896384574 wait complete. 2024-04-02T14:38:50.602 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.600+0000 7fcf715b2740 1 -- start start 2024-04-02T14:38:50.603 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.600+0000 7fcf715b2740 1 Processor -- start 2024-04-02T14:38:50.603 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.601+0000 7fcf715b2740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e61b391eb0 0x55e61b398440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:50.603 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.601+0000 7fcf715b2740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e61b3927d0 0x55e61b398980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:50.604 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.601+0000 7fcf715b2740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e61b39b3b0 0x55e61b398ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:50.604 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.601+0000 7fcf715b2740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55e61b39e140 con 0x55e61b39b3b0 2024-04-02T14:38:50.604 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.601+0000 7fcf715b2740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55e61b2b2660 con 0x55e61b3927d0 2024-04-02T14:38:50.604 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.601+0000 7fcf715b2740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55e61b39e2c0 con 0x55e61b391eb0 2024-04-02T14:38:50.605 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.601+0000 7fcf715b2740 10 client.0 _ll_register_callbacks cb 0x55e61b2b2050 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:38:50.605 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.602+0000 7fcf5ea00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e61b3927d0 0x55e61b398980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:50.605 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.602+0000 7fcf5ea00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e61b3927d0 0x55e61b398980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:58274/0 (socket says 172.21.15.90:58274) 2024-04-02T14:38:50.605 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.602+0000 7fcf5ea00640 1 -- 172.21.15.90:0/1045626143 learned_addr learned my addr 172.21.15.90:0/1045626143 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:38:50.605 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.602+0000 7fcf67400640 1 --2- 172.21.15.90:0/1045626143 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e61b391eb0 0x55e61b398440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:50.605 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.602+0000 7fcf67400640 1 -- 172.21.15.90:0/1045626143 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e61b3927d0 msgr2=0x55e61b398980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:38:50.605 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.602+0000 7fcf67400640 1 --2- 172.21.15.90:0/1045626143 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e61b3927d0 0x55e61b398980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:50.605 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.602+0000 7fcf67400640 1 -- 172.21.15.90:0/1045626143 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e61b39b3b0 msgr2=0x55e61b398ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:38:50.605 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.602+0000 7fcf67400640 1 --2- 172.21.15.90:0/1045626143 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e61b39b3b0 0x55e61b398ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:50.605 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.602+0000 7fcf67400640 1 -- 172.21.15.90:0/1045626143 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e61b2c1e80 con 0x55e61b391eb0 2024-04-02T14:38:50.606 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.602+0000 7fcf5ea00640 1 --2- 172.21.15.90:0/1045626143 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e61b3927d0 0x55e61b398980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:38:50.606 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.603+0000 7fcf67400640 1 --2- 172.21.15.90:0/1045626143 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e61b391eb0 0x55e61b398440 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fcf58011d00 tx=0x7fcf5800f790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d1894c09539c3eab server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:38:50.606 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.603+0000 7fcf55600640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:38:50.606 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.603+0000 7fcf55600640 1 -- 172.21.15.90:0/1045626143 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf58009d70 con 0x55e61b391eb0 2024-04-02T14:38:50.606 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.603+0000 7fcf55600640 1 -- 172.21.15.90:0/1045626143 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7fcf58003970 con 0x55e61b391eb0 2024-04-02T14:38:50.607 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.604+0000 7fcf715b2740 1 -- 172.21.15.90:0/1045626143 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcf600a5b10 con 0x55e61b391eb0 2024-04-02T14:38:50.607 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.604+0000 7fcf55600640 1 -- 172.21.15.90:0/1045626143 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fcf58003b70 con 0x55e61b391eb0 2024-04-02T14:38:50.607 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.604+0000 7fcf715b2740 10 client.26503 fetch_fsmap learned FSMap version 27 2024-04-02T14:38:50.607 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.604+0000 7fcf715b2740 1 -- 172.21.15.90:0/1045626143 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55e61b00b210 con 0x55e61b391eb0 2024-04-02T14:38:50.608 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.605+0000 7fcf55600640 1 -- 172.21.15.90:0/1045626143 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcf600a5b10 con 0x55e61b391eb0 2024-04-02T14:38:50.608 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.605+0000 7fcf55600640 1 -- 172.21.15.90:0/1045626143 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fcf58004630 con 0x55e61b391eb0 2024-04-02T14:38:50.608 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.605+0000 7fcf715b2740 10 client.26503 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:38:50.608 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.605+0000 7fcf715b2740 10 client.26503 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-02T14:38:50.608 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.605+0000 7fcf715b2740 20 client.26503 populate_metadata read hostname 'smithi090' 2024-04-02T14:38:50.608 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.605+0000 7fcf715b2740 1 --2- 172.21.15.90:0/1045626143 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55e61b3ab7d0 0x55e61b3adc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:50.608 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.605+0000 7fcf715b2740 4 client.26503 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:38:50.608 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.605+0000 7fcf715b2740 1 -- 172.21.15.90:0/1045626143 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e61b2a6cd0 con 0x55e61b3ab7d0 2024-04-02T14:38:50.608 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.606+0000 7fcf5ea00640 1 --2- 172.21.15.90:0/1045626143 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55e61b3ab7d0 0x55e61b3adc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:50.609 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.607+0000 7fcf55600640 10 client.26503 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:38:50.609 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.607+0000 7fcf5ea00640 1 --2- 172.21.15.90:0/1045626143 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55e61b3ab7d0 0x55e61b3adc70 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=812336ba59536c server_cookie=73a286b317ea4799 in_seq=0 out_seq=0 2024-04-02T14:38:50.610 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.608+0000 7fcf55600640 1 -- 172.21.15.90:0/1045626143 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e61b2a6cd0 con 0x55e61b3ab7d0 2024-04-02T14:38:50.610 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.608+0000 7fcf55600640 10 client.26503 handle_command_reply: tid=0 2024-04-02T14:38:50.610 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.608+0000 7fcf715b2740 1 client.26503 shutdown 2024-04-02T14:38:50.612 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.609+0000 7fcf715b2740 1 -- 172.21.15.90:0/1045626143 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e61b391eb0 msgr2=0x55e61b398440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:38:50.612 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.609+0000 7fcf715b2740 1 --2- 172.21.15.90:0/1045626143 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e61b391eb0 0x55e61b398440 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fcf58011d00 tx=0x7fcf5800f790 comp rx=0 tx=0).stop 2024-04-02T14:38:50.612 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.609+0000 7fcf715b2740 1 -- 172.21.15.90:0/1045626143 shutdown_connections 2024-04-02T14:38:50.612 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.609+0000 7fcf715b2740 1 --2- 172.21.15.90:0/1045626143 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55e61b3ab7d0 0x55e61b3adc70 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:50.612 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.609+0000 7fcf715b2740 1 --2- 172.21.15.90:0/1045626143 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e61b39b3b0 0x55e61b398ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:50.612 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.609+0000 7fcf715b2740 1 --2- 172.21.15.90:0/1045626143 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e61b3927d0 0x55e61b398980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:50.612 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.609+0000 7fcf715b2740 1 --2- 172.21.15.90:0/1045626143 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e61b391eb0 0x55e61b398440 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:50.612 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.609+0000 7fcf715b2740 1 -- 172.21.15.90:0/1045626143 >> 172.21.15.90:0/1045626143 conn(0x55e61b38c5e0 msgr2=0x55e61b38e330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:38:50.612 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.610+0000 7fcf55600640 0 client.26503 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:38:50.612 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.610+0000 7fcf715b2740 1 -- 172.21.15.90:0/1045626143 shutdown_connections 2024-04-02T14:38:50.613 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.610+0000 7fcf715b2740 1 -- 172.21.15.90:0/1045626143 wait complete. 2024-04-02T14:38:50.613 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.610+0000 7fcf715b2740 20 client.26503 trim_cache size 0 max 16384 2024-04-02T14:38:50.642 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.640+0000 7fcf715b2740 1 Processor -- start 2024-04-02T14:38:50.643 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.640+0000 7fcf715b2740 1 -- start start 2024-04-02T14:38:50.645 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.641+0000 7fcf715b2740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e61b391eb0 0x55e61b39ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:50.645 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.641+0000 7fcf715b2740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e61b3927d0 0x55e61b3a7000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:50.645 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.641+0000 7fcf715b2740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e61b39b3b0 0x55e61b3b3540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:50.646 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.641+0000 7fcf715b2740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fcf681bc750 con 0x55e61b391eb0 2024-04-02T14:38:50.646 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.641+0000 7fcf715b2740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fcf681be290 con 0x55e61b3927d0 2024-04-02T14:38:50.646 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.641+0000 7fcf715b2740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fcf681bce20 con 0x55e61b39b3b0 2024-04-02T14:38:50.646 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.641+0000 7fcf67e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e61b39b3b0 0x55e61b3b3540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:50.646 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.641+0000 7fcf67e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e61b39b3b0 0x55e61b3b3540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:53532/0 (socket says 172.21.15.90:53532) 2024-04-02T14:38:50.647 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.641+0000 7fcf67e00640 1 -- 172.21.15.90:0/211721235 learned_addr learned my addr 172.21.15.90:0/211721235 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:38:50.647 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.642+0000 7fcf67400640 1 --2- 172.21.15.90:0/211721235 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e61b391eb0 0x55e61b39ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:50.647 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.642+0000 7fcf5ea00640 1 --2- 172.21.15.90:0/211721235 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e61b3927d0 0x55e61b3a7000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:50.647 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.642+0000 7fcf67e00640 1 -- 172.21.15.90:0/211721235 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e61b3927d0 msgr2=0x55e61b3a7000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:38:50.647 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.642+0000 7fcf67e00640 1 --2- 172.21.15.90:0/211721235 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e61b3927d0 0x55e61b3a7000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:50.647 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.642+0000 7fcf67e00640 1 -- 172.21.15.90:0/211721235 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e61b391eb0 msgr2=0x55e61b39ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:38:50.648 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.642+0000 7fcf67e00640 1 --2- 172.21.15.90:0/211721235 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e61b391eb0 0x55e61b39ab70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:50.648 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.642+0000 7fcf67e00640 1 -- 172.21.15.90:0/211721235 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e61b0d7620 con 0x55e61b39b3b0 2024-04-02T14:38:50.648 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.642+0000 7fcf67400640 1 --2- 172.21.15.90:0/211721235 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e61b391eb0 0x55e61b39ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:38:50.648 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.642+0000 7fcf67e00640 1 --2- 172.21.15.90:0/211721235 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e61b39b3b0 0x55e61b3b3540 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fcf600047e0 tx=0x7fcf60004e80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5edd4105243c03ff server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:38:50.648 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.643+0000 7fcf57e00640 1 -- 172.21.15.90:0/211721235 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf600bb070 con 0x55e61b39b3b0 2024-04-02T14:38:50.648 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.643+0000 7fcf57e00640 1 -- 172.21.15.90:0/211721235 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcf600995c0 con 0x55e61b39b3b0 2024-04-02T14:38:50.649 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.643+0000 7fcf57e00640 1 -- 172.21.15.90:0/211721235 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf60014070 con 0x55e61b39b3b0 2024-04-02T14:38:50.655 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.644+0000 7fcf715b2740 1 -- 172.21.15.90:0/211721235 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e61b39b3b0 msgr2=0x55e61b3b3540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:38:50.655 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.644+0000 7fcf715b2740 1 --2- 172.21.15.90:0/211721235 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e61b39b3b0 0x55e61b3b3540 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fcf600047e0 tx=0x7fcf60004e80 comp rx=0 tx=0).stop 2024-04-02T14:38:50.656 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.644+0000 7fcf715b2740 1 -- 172.21.15.90:0/211721235 shutdown_connections 2024-04-02T14:38:50.656 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.644+0000 7fcf715b2740 1 --2- 172.21.15.90:0/211721235 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e61b39b3b0 0x55e61b3b3540 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:50.656 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.644+0000 7fcf715b2740 1 --2- 172.21.15.90:0/211721235 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e61b3927d0 0x55e61b3a7000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:50.656 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.644+0000 7fcf715b2740 1 --2- 172.21.15.90:0/211721235 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e61b391eb0 0x55e61b39ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:50.656 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.644+0000 7fcf715b2740 1 -- 172.21.15.90:0/211721235 >> 172.21.15.90:0/211721235 conn(0x55e61b38c5e0 msgr2=0x55e61b393730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:38:50.656 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.644+0000 7fcf715b2740 1 -- 172.21.15.90:0/211721235 shutdown_connections 2024-04-02T14:38:50.657 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.644+0000 7fcf715b2740 1 -- 172.21.15.90:0/211721235 wait complete. 2024-04-02T14:38:50.657 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.645+0000 7fcf715b2740 1 -- start start 2024-04-02T14:38:50.657 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.645+0000 7fcf715b2740 1 Processor -- start 2024-04-02T14:38:50.657 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.646+0000 7fcf715b2740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e61b391eb0 0x55e61b3a3960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:50.657 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.646+0000 7fcf715b2740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e61b3927d0 0x55e61b394ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:50.657 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.647+0000 7fcf715b2740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e61b39b3b0 0x55e61b395420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:50.658 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.647+0000 7fcf715b2740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fcf681bdbc0 con 0x55e61b39b3b0 2024-04-02T14:38:50.658 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.647+0000 7fcf715b2740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fcf681bd4f0 con 0x55e61b391eb0 2024-04-02T14:38:50.658 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.647+0000 7fcf715b2740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fcf681ba6f0 con 0x55e61b3927d0 2024-04-02T14:38:50.658 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.647+0000 7fcf715b2740 10 client.0 _ll_register_callbacks cb 0x7fcf58011940 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:38:50.658 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.647+0000 7fcf67400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e61b391eb0 0x55e61b3a3960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:50.659 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.647+0000 7fcf67400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e61b391eb0 0x55e61b3a3960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:58284/0 (socket says 172.21.15.90:58284) 2024-04-02T14:38:50.659 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.647+0000 7fcf67400640 1 -- 172.21.15.90:0/878816532 learned_addr learned my addr 172.21.15.90:0/878816532 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:38:50.659 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.647+0000 7fcf5ea00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e61b3927d0 0x55e61b394ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:50.659 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.647+0000 7fcf67e00640 1 --2- 172.21.15.90:0/878816532 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e61b39b3b0 0x55e61b395420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:50.659 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.648+0000 7fcf67400640 1 -- 172.21.15.90:0/878816532 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e61b3927d0 msgr2=0x55e61b394ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:38:50.659 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.648+0000 7fcf67400640 1 --2- 172.21.15.90:0/878816532 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e61b3927d0 0x55e61b394ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:50.660 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.648+0000 7fcf67400640 1 -- 172.21.15.90:0/878816532 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e61b39b3b0 msgr2=0x55e61b395420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:38:50.660 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.648+0000 7fcf67400640 1 --2- 172.21.15.90:0/878816532 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e61b39b3b0 0x55e61b395420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:50.660 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.648+0000 7fcf67400640 1 -- 172.21.15.90:0/878816532 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e61b395c50 con 0x55e61b391eb0 2024-04-02T14:38:50.660 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.649+0000 7fcf5ea00640 1 --2- 172.21.15.90:0/878816532 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e61b3927d0 0x55e61b394ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:38:50.660 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.649+0000 7fcf67400640 1 --2- 172.21.15.90:0/878816532 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e61b391eb0 0x55e61b3a3960 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fcf58002230 tx=0x7fcf580058e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e6203e00050faa30 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:38:50.660 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.650+0000 7fcf55600640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:38:50.661 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.650+0000 7fcf55600640 1 -- 172.21.15.90:0/878816532 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf58033060 con 0x55e61b391eb0 2024-04-02T14:38:50.661 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.650+0000 7fcf715b2740 1 -- 172.21.15.90:0/878816532 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcf600995c0 con 0x55e61b391eb0 2024-04-02T14:38:50.661 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.650+0000 7fcf55600640 1 -- 172.21.15.90:0/878816532 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7fcf58017040 con 0x55e61b391eb0 2024-04-02T14:38:50.661 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.651+0000 7fcf55600640 1 -- 172.21.15.90:0/878816532 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fcf58028040 con 0x55e61b391eb0 2024-04-02T14:38:50.661 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.651+0000 7fcf715b2740 10 client.26621 fetch_fsmap learned FSMap version 27 2024-04-02T14:38:50.661 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.651+0000 7fcf715b2740 1 -- 172.21.15.90:0/878816532 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55e61b411e50 con 0x55e61b391eb0 2024-04-02T14:38:50.662 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.651+0000 7fcf55600640 1 -- 172.21.15.90:0/878816532 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcf600995c0 con 0x55e61b391eb0 2024-04-02T14:38:50.662 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.651+0000 7fcf55600640 1 -- 172.21.15.90:0/878816532 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fcf58024430 con 0x55e61b391eb0 2024-04-02T14:38:50.662 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.651+0000 7fcf715b2740 10 client.26621 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:38:50.662 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.651+0000 7fcf715b2740 10 client.26621 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-02T14:38:50.662 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.651+0000 7fcf715b2740 20 client.26621 populate_metadata read hostname 'smithi090' 2024-04-02T14:38:50.662 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.651+0000 7fcf715b2740 1 --2- 172.21.15.90:0/878816532 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55e61b412520 0x55e61b414940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:38:50.663 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.652+0000 7fcf715b2740 4 client.26621 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-02T14:38:50.663 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.652+0000 7fcf715b2740 1 -- 172.21.15.90:0/878816532 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55e61b3042b0 con 0x55e61b412520 2024-04-02T14:38:50.663 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.652+0000 7fcf5ea00640 1 --2- 172.21.15.90:0/878816532 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55e61b412520 0x55e61b414940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:38:50.663 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.653+0000 7fcf55600640 10 client.26621 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:38:50.664 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.655+0000 7fcf5ea00640 1 --2- 172.21.15.90:0/878816532 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55e61b412520 0x55e61b414940 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3ed009649cf229ed server_cookie=ea64755e291cc9bc in_seq=0 out_seq=0 2024-04-02T14:38:50.672 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.669+0000 7fcf55600640 1 -- 172.21.15.90:0/878816532 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55e61b3042b0 con 0x55e61b412520 2024-04-02T14:38:50.672 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.669+0000 7fcf55600640 10 client.26621 handle_command_reply: tid=0 2024-04-02T14:38:50.672 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.669+0000 7fcf715b2740 1 client.26621 shutdown 2024-04-02T14:38:50.672 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.669+0000 7fcf715b2740 1 -- 172.21.15.90:0/878816532 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e61b391eb0 msgr2=0x55e61b3a3960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:38:50.673 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.669+0000 7fcf715b2740 1 --2- 172.21.15.90:0/878816532 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e61b391eb0 0x55e61b3a3960 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fcf58002230 tx=0x7fcf580058e0 comp rx=0 tx=0).stop 2024-04-02T14:38:50.673 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.669+0000 7fcf715b2740 1 -- 172.21.15.90:0/878816532 shutdown_connections 2024-04-02T14:38:50.673 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.669+0000 7fcf715b2740 1 --2- 172.21.15.90:0/878816532 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55e61b412520 0x55e61b414940 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:50.673 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.669+0000 7fcf715b2740 1 --2- 172.21.15.90:0/878816532 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e61b39b3b0 0x55e61b395420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:50.673 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.669+0000 7fcf715b2740 1 --2- 172.21.15.90:0/878816532 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e61b3927d0 0x55e61b394ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:50.673 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.669+0000 7fcf715b2740 1 --2- 172.21.15.90:0/878816532 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e61b391eb0 0x55e61b3a3960 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:50.674 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.669+0000 7fcf55600640 0 client.26621 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:38:50.674 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.669+0000 7fcf715b2740 1 -- 172.21.15.90:0/878816532 >> 172.21.15.90:0/878816532 conn(0x55e61b38c5e0 msgr2=0x55e61b38da10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:38:50.674 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.669+0000 7fcf715b2740 1 -- 172.21.15.90:0/878816532 shutdown_connections 2024-04-02T14:38:50.674 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.670+0000 7fcf715b2740 1 -- 172.21.15.90:0/878816532 wait complete. 2024-04-02T14:38:50.674 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.670+0000 7fcf715b2740 20 client.26621 trim_cache size 0 max 16384 2024-04-02T14:38:50.675 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:{ 2024-04-02T14:38:50.675 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "message": "", 2024-04-02T14:38:50.675 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "return_code": 0 2024-04-02T14:38:50.676 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:} 2024-04-02T14:38:50.677 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.672+0000 7fcf6e800640 1 -- 172.21.15.90:0/3927783890 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fcf4c0a3f20 msgr2=0x7fcf4c0a63e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:38:50.677 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.672+0000 7fcf6e800640 1 --2- 172.21.15.90:0/3927783890 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fcf4c0a3f20 0x7fcf4c0a63e0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fcf68112110 tx=0x7fcf50005e10 comp rx=0 tx=0).stop 2024-04-02T14:38:50.677 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.672+0000 7fcf6e800640 1 -- 172.21.15.90:0/3927783890 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcf68114c50 msgr2=0x7fcf68111340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:38:50.677 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.672+0000 7fcf6e800640 1 --2- 172.21.15.90:0/3927783890 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcf68114c50 0x7fcf68111340 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fcf6000d680 tx=0x7fcf6000db40 comp rx=0 tx=0).stop 2024-04-02T14:38:50.677 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.672+0000 7fcf6e800640 1 -- 172.21.15.90:0/3927783890 shutdown_connections 2024-04-02T14:38:50.678 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.672+0000 7fcf6e800640 1 --2- 172.21.15.90:0/3927783890 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fcf4c0a3f20 0x7fcf4c0a63e0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:50.678 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.672+0000 7fcf6e800640 1 --2- 172.21.15.90:0/3927783890 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcf68114c50 0x7fcf68111340 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:50.678 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.672+0000 7fcf6e800640 1 --2- 172.21.15.90:0/3927783890 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcf68109420 0x7fcf68110e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:50.678 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.672+0000 7fcf6e800640 1 --2- 172.21.15.90:0/3927783890 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcf68108b00 0x7fcf681108c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:38:50.678 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.672+0000 7fcf6e800640 1 -- 172.21.15.90:0/3927783890 >> 172.21.15.90:0/3927783890 conn(0x7fcf6806d030 msgr2=0x7fcf68116c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:38:50.678 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.672+0000 7fcf6e800640 1 -- 172.21.15.90:0/3927783890 shutdown_connections 2024-04-02T14:38:50.679 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:38:50.672+0000 7fcf6e800640 1 -- 172.21.15.90:0/3927783890 wait complete. 2024-04-02T14:38:51.833 INFO:teuthology.orchestra.run.smithi090.stderr:progress: 200.0 s, 3540.9 tps, lat 28.132 ms stddev 33.119 2024-04-02T14:38:52.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:38:51 smithi177 ceph-mon[36431]: pgmap v122: 97 pgs: 97 active+clean; 2.3 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 9.4 MiB/s wr, 719 op/s 2024-04-02T14:38:52.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:38:51 smithi090 ceph-mon[31674]: pgmap v122: 97 pgs: 97 active+clean; 2.3 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 9.4 MiB/s wr, 719 op/s 2024-04-02T14:38:52.360 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:38:51 smithi092 ceph-mon[36497]: pgmap v122: 97 pgs: 97 active+clean; 2.3 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 9.4 MiB/s wr, 719 op/s 2024-04-02T14:38:54.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:38:53 smithi177 ceph-mon[36431]: pgmap v123: 97 pgs: 97 active+clean; 2.3 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 7.9 MiB/s wr, 614 op/s 2024-04-02T14:38:54.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:38:53 smithi090 ceph-mon[31674]: pgmap v123: 97 pgs: 97 active+clean; 2.3 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 7.9 MiB/s wr, 614 op/s 2024-04-02T14:38:54.360 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:38:53 smithi092 ceph-mon[36497]: pgmap v123: 97 pgs: 97 active+clean; 2.3 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 7.9 MiB/s wr, 614 op/s 2024-04-02T14:38:56.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:38:55 smithi177 ceph-mon[36431]: pgmap v124: 97 pgs: 97 active+clean; 2.3 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 11 MiB/s wr, 944 op/s 2024-04-02T14:38:56.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:38:55 smithi090 ceph-mon[31674]: pgmap v124: 97 pgs: 97 active+clean; 2.3 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 11 MiB/s wr, 944 op/s 2024-04-02T14:38:56.360 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:38:55 smithi092 ceph-mon[36497]: pgmap v124: 97 pgs: 97 active+clean; 2.3 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 11 MiB/s wr, 944 op/s 2024-04-02T14:38:56.832 INFO:teuthology.orchestra.run.smithi090.stderr:progress: 205.0 s, 3657.2 tps, lat 27.585 ms stddev 32.874 2024-04-02T14:38:58.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:38:57 smithi177 ceph-mon[36431]: pgmap v125: 97 pgs: 97 active+clean; 2.3 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 7.4 MiB/s wr, 631 op/s 2024-04-02T14:38:58.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:38:57 smithi090 ceph-mon[31674]: pgmap v125: 97 pgs: 97 active+clean; 2.3 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 7.4 MiB/s wr, 631 op/s 2024-04-02T14:38:58.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:38:57 smithi092 ceph-mon[36497]: pgmap v125: 97 pgs: 97 active+clean; 2.3 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 7.4 MiB/s wr, 631 op/s 2024-04-02T14:39:00.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:38:59 smithi177 ceph-mon[36431]: pgmap v126: 97 pgs: 97 active+clean; 2.3 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 14 MiB/s wr, 933 op/s 2024-04-02T14:39:00.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:38:59 smithi090 ceph-mon[31674]: pgmap v126: 97 pgs: 97 active+clean; 2.3 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 14 MiB/s wr, 933 op/s 2024-04-02T14:39:00.360 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:38:59 smithi092 ceph-mon[36497]: pgmap v126: 97 pgs: 97 active+clean; 2.3 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 14 MiB/s wr, 933 op/s 2024-04-02T14:39:01.832 INFO:teuthology.orchestra.run.smithi090.stderr:progress: 210.0 s, 1808.0 tps, lat 51.910 ms stddev 85.285 2024-04-02T14:39:02.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:39:01 smithi177 ceph-mon[36431]: pgmap v127: 97 pgs: 97 active+clean; 2.3 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 13 MiB/s wr, 866 op/s 2024-04-02T14:39:02.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:39:01 smithi090 ceph-mon[31674]: pgmap v127: 97 pgs: 97 active+clean; 2.3 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 13 MiB/s wr, 866 op/s 2024-04-02T14:39:02.360 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:39:01 smithi092 ceph-mon[36497]: pgmap v127: 97 pgs: 97 active+clean; 2.3 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 13 MiB/s wr, 866 op/s 2024-04-02T14:39:02.999 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-02T14:39:03.000 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-02T14:39:03.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.222+0000 7f73ee400640 1 -- 172.21.15.90:0/3945985796 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f73e8104780 msgr2=0x7f73e81087c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:03.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.222+0000 7f73ee400640 1 --2- 172.21.15.90:0/3945985796 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f73e8104780 0x7f73e81087c0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f73d4009900 tx=0x7f73d402bc40 comp rx=0 tx=0).stop 2024-04-02T14:39:03.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.222+0000 7f73ee400640 1 -- 172.21.15.90:0/3945985796 shutdown_connections 2024-04-02T14:39:03.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.222+0000 7f73ee400640 1 --2- 172.21.15.90:0/3945985796 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f73e8108ea0 0x7f73e8110a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:03.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.222+0000 7f73ee400640 1 --2- 172.21.15.90:0/3945985796 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f73e8104780 0x7f73e81087c0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:03.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.222+0000 7f73ee400640 1 --2- 172.21.15.90:0/3945985796 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f73e8103dd0 0x7f73e81041b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:03.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.222+0000 7f73ee400640 1 -- 172.21.15.90:0/3945985796 >> 172.21.15.90:0/3945985796 conn(0x7f73e80ff610 msgr2=0x7f73e8101a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:39:03.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.222+0000 7f73ee400640 1 -- 172.21.15.90:0/3945985796 shutdown_connections 2024-04-02T14:39:03.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.222+0000 7f73ee400640 1 -- 172.21.15.90:0/3945985796 wait complete. 2024-04-02T14:39:03.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.223+0000 7f73ee400640 1 Processor -- start 2024-04-02T14:39:03.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.224+0000 7f73ee400640 1 -- start start 2024-04-02T14:39:03.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.224+0000 7f73ee400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f73e8103dd0 0x7f73e819a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:03.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.224+0000 7f73ee400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f73e8108ea0 0x7f73e819a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:03.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.224+0000 7f73ee400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f73e819ed40 0x7f73e819f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:03.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.224+0000 7f73ee400640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f73e8113fa0 con 0x7f73e8103dd0 2024-04-02T14:39:03.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.224+0000 7f73ee400640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f73e8113e20 con 0x7f73e8108ea0 2024-04-02T14:39:03.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.224+0000 7f73ee400640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f73e8114120 con 0x7f73e819ed40 2024-04-02T14:39:03.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.224+0000 7f73ed000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f73e8103dd0 0x7f73e819a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:03.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.224+0000 7f73eda00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f73e819ed40 0x7f73e819f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:03.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.224+0000 7f73e7e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f73e8108ea0 0x7f73e819a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:03.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.224+0000 7f73ed000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f73e8103dd0 0x7f73e819a480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:53776/0 (socket says 172.21.15.90:53776) 2024-04-02T14:39:03.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.224+0000 7f73ed000640 1 -- 172.21.15.90:0/3166179885 learned_addr learned my addr 172.21.15.90:0/3166179885 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:39:03.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.225+0000 7f73ed000640 1 -- 172.21.15.90:0/3166179885 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f73e819ed40 msgr2=0x7f73e819f1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:03.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.225+0000 7f73ed000640 1 --2- 172.21.15.90:0/3166179885 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f73e819ed40 0x7f73e819f1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:03.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.225+0000 7f73ed000640 1 -- 172.21.15.90:0/3166179885 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f73e8108ea0 msgr2=0x7f73e819a9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:03.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.225+0000 7f73ed000640 1 --2- 172.21.15.90:0/3166179885 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f73e8108ea0 0x7f73e819a9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:03.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.225+0000 7f73ed000640 1 -- 172.21.15.90:0/3166179885 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73e819f830 con 0x7f73e8103dd0 2024-04-02T14:39:03.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.225+0000 7f73eda00640 1 --2- 172.21.15.90:0/3166179885 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f73e819ed40 0x7f73e819f1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:39:03.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.225+0000 7f73e7e00640 1 --2- 172.21.15.90:0/3166179885 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f73e8108ea0 0x7f73e819a9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:39:03.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.225+0000 7f73ed000640 1 --2- 172.21.15.90:0/3166179885 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f73e8103dd0 0x7f73e819a480 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f73dc00ea60 tx=0x7f73dc00ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:39:03.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.225+0000 7f73e5600640 1 -- 172.21.15.90:0/3166179885 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73dc00cde0 con 0x7f73e8103dd0 2024-04-02T14:39:03.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.225+0000 7f73ee400640 1 -- 172.21.15.90:0/3166179885 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f73e819fb20 con 0x7f73e8103dd0 2024-04-02T14:39:03.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.225+0000 7f73e5600640 1 -- 172.21.15.90:0/3166179885 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f73dc0035f0 con 0x7f73e8103dd0 2024-04-02T14:39:03.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.225+0000 7f73e5600640 1 -- 172.21.15.90:0/3166179885 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73dc010670 con 0x7f73e8103dd0 2024-04-02T14:39:03.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.225+0000 7f73ee400640 1 -- 172.21.15.90:0/3166179885 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f73e81a7280 con 0x7f73e8103dd0 2024-04-02T14:39:03.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.229+0000 7f73e5600640 1 -- 172.21.15.90:0/3166179885 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f73dc0037a0 con 0x7f73e8103dd0 2024-04-02T14:39:03.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.230+0000 7f73f27d1740 1 Processor -- start 2024-04-02T14:39:03.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.230+0000 7f73f27d1740 1 -- start start 2024-04-02T14:39:03.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.230+0000 7f73f27d1740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55be7f714170 0x55be7f714550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:03.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.230+0000 7f73f27d1740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55be7f714a90 0x55be7f71d130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:03.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.230+0000 7f73f27d1740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55be7f71d670 0x55be7f71fa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:03.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.230+0000 7f73f27d1740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55be7f66f770 con 0x55be7f714a90 2024-04-02T14:39:03.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.230+0000 7f73f27d1740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55be7f617700 con 0x55be7f714170 2024-04-02T14:39:03.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.230+0000 7f73f27d1740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55be7f66b720 con 0x55be7f71d670 2024-04-02T14:39:03.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.231+0000 7f73eda00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55be7f714170 0x55be7f714550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:03.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.231+0000 7f73eda00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55be7f714170 0x55be7f714550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:52136/0 (socket says 172.21.15.90:52136) 2024-04-02T14:39:03.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.231+0000 7f73eda00640 1 -- 172.21.15.90:0/3060969113 learned_addr learned my addr 172.21.15.90:0/3060969113 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:39:03.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.231+0000 7f73ed000640 1 --2- 172.21.15.90:0/3060969113 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55be7f714a90 0x55be7f71d130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:03.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.231+0000 7f73e7e00640 1 --2- 172.21.15.90:0/3060969113 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55be7f71d670 0x55be7f71fa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:03.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.231+0000 7f73eda00640 1 -- 172.21.15.90:0/3060969113 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55be7f71d670 msgr2=0x55be7f71fa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:39:03.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.231+0000 7f73eda00640 1 --2- 172.21.15.90:0/3060969113 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55be7f71d670 0x55be7f71fa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:03.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.231+0000 7f73eda00640 1 -- 172.21.15.90:0/3060969113 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55be7f714a90 msgr2=0x55be7f71d130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:39:03.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.232+0000 7f73eda00640 1 --2- 172.21.15.90:0/3060969113 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55be7f714a90 0x55be7f71d130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:03.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.232+0000 7f73eda00640 1 -- 172.21.15.90:0/3060969113 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55be7f44d7d0 con 0x55be7f714170 2024-04-02T14:39:03.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.232+0000 7f73e7e00640 1 --2- 172.21.15.90:0/3060969113 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55be7f71d670 0x55be7f71fa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:39:03.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.232+0000 7f73ed000640 1 --2- 172.21.15.90:0/3060969113 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55be7f714a90 0x55be7f71d130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:39:03.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.232+0000 7f73e5600640 1 --2- 172.21.15.90:0/3166179885 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f73d00a3da0 0x7f73d00a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:03.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.232+0000 7f73ed000640 1 --2- 172.21.15.90:0/3166179885 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f73d00a3da0 0x7f73d00a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:03.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.232+0000 7f73eda00640 1 --2- 172.21.15.90:0/3060969113 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55be7f714170 0x55be7f714550 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f73e0011790 tx=0x7f73e0011c50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=96c6e55fdce0d3f4 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:39:03.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.232+0000 7f73e5600640 1 -- 172.21.15.90:0/3166179885 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f73dc0d0a90 con 0x7f73e8103dd0 2024-04-02T14:39:03.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.233+0000 7f73da000640 1 -- 172.21.15.90:0/3060969113 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73e0003c30 con 0x55be7f714170 2024-04-02T14:39:03.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.233+0000 7f73da000640 1 -- 172.21.15.90:0/3060969113 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f73e000fe80 con 0x55be7f714170 2024-04-02T14:39:03.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.233+0000 7f73da000640 1 -- 172.21.15.90:0/3060969113 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73e00193e0 con 0x55be7f714170 2024-04-02T14:39:03.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.234+0000 7f73f27d1740 1 -- 172.21.15.90:0/3060969113 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55be7f714170 msgr2=0x55be7f714550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:03.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.234+0000 7f73ed000640 1 --2- 172.21.15.90:0/3166179885 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f73d00a3da0 0x7f73d00a6260 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x55be7f713c40 tx=0x7f73dc003f30 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:39:03.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.234+0000 7f73f27d1740 1 --2- 172.21.15.90:0/3060969113 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55be7f714170 0x55be7f714550 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f73e0011790 tx=0x7f73e0011c50 comp rx=0 tx=0).stop 2024-04-02T14:39:03.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.234+0000 7f73f27d1740 1 -- 172.21.15.90:0/3060969113 shutdown_connections 2024-04-02T14:39:03.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.234+0000 7f73f27d1740 1 --2- 172.21.15.90:0/3060969113 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55be7f71d670 0x55be7f71fa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:03.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.234+0000 7f73f27d1740 1 --2- 172.21.15.90:0/3060969113 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55be7f714a90 0x55be7f71d130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:03.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.234+0000 7f73f27d1740 1 --2- 172.21.15.90:0/3060969113 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55be7f714170 0x55be7f714550 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:03.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.235+0000 7f73f27d1740 1 -- 172.21.15.90:0/3060969113 >> 172.21.15.90:0/3060969113 conn(0x55be7f70e8a0 msgr2=0x55be7f70ec80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:39:03.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.235+0000 7f73f27d1740 1 -- 172.21.15.90:0/3060969113 shutdown_connections 2024-04-02T14:39:03.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.235+0000 7f73f27d1740 1 -- 172.21.15.90:0/3060969113 wait complete. 2024-04-02T14:39:03.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.236+0000 7f73f27d1740 1 -- start start 2024-04-02T14:39:03.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.236+0000 7f73f27d1740 1 Processor -- start 2024-04-02T14:39:03.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.236+0000 7f73f27d1740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55be7f714170 0x55be7f71a770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:03.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.236+0000 7f73f27d1740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55be7f714a90 0x55be7f71acb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:03.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.236+0000 7f73f27d1740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55be7f71d670 0x55be7f71b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:03.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.236+0000 7f73f27d1740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55be7f720460 con 0x55be7f714170 2024-04-02T14:39:03.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.236+0000 7f73f27d1740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55be7f62aac0 con 0x55be7f71d670 2024-04-02T14:39:03.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.236+0000 7f73f27d1740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55be7f7205e0 con 0x55be7f714a90 2024-04-02T14:39:03.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.236+0000 7f73f27d1740 10 client.0 _ll_register_callbacks cb 0x55be7f62d180 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:39:03.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.237+0000 7f73e7e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55be7f714a90 0x55be7f71acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:03.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.237+0000 7f73eda00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55be7f714170 0x55be7f71a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:03.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.237+0000 7f73e7e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55be7f714a90 0x55be7f71acb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:36940/0 (socket says 172.21.15.90:36940) 2024-04-02T14:39:03.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.237+0000 7f73e7e00640 1 -- 172.21.15.90:0/2459898452 learned_addr learned my addr 172.21.15.90:0/2459898452 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:39:03.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.237+0000 7f73eda00640 1 --2- 172.21.15.90:0/2459898452 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55be7f71d670 0x55be7f71b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:03.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.237+0000 7f73e7e00640 1 -- 172.21.15.90:0/2459898452 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55be7f71d670 msgr2=0x55be7f71b1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:39:03.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.237+0000 7f73e7e00640 1 --2- 172.21.15.90:0/2459898452 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55be7f71d670 0x55be7f71b1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:03.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.237+0000 7f73e7e00640 1 -- 172.21.15.90:0/2459898452 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55be7f714170 msgr2=0x55be7f71a770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:39:03.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.237+0000 7f73e7e00640 1 --2- 172.21.15.90:0/2459898452 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55be7f714170 0x55be7f71a770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:03.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.237+0000 7f73e7e00640 1 -- 172.21.15.90:0/2459898452 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55be7f3833f0 con 0x55be7f714a90 2024-04-02T14:39:03.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.237+0000 7f73eda00640 1 --2- 172.21.15.90:0/2459898452 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55be7f714170 0x55be7f71a770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:39:03.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.238+0000 7f73e7e00640 1 --2- 172.21.15.90:0/2459898452 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55be7f714a90 0x55be7f71acb0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f73e806b180 tx=0x7f73d4004da0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=57dce1296d12b338 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:39:03.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.238+0000 7f73cea00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:39:03.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.238+0000 7f73cea00640 1 -- 172.21.15.90:0/2459898452 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73d4043070 con 0x55be7f714a90 2024-04-02T14:39:03.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.238+0000 7f73cea00640 1 -- 172.21.15.90:0/2459898452 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f73d402c890 con 0x55be7f714a90 2024-04-02T14:39:03.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.238+0000 7f73f27d1740 1 -- 172.21.15.90:0/2459898452 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55be7f610630 con 0x55be7f714a90 2024-04-02T14:39:03.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.239+0000 7f73cea00640 1 -- 172.21.15.90:0/2459898452 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f73d402ca90 con 0x55be7f714a90 2024-04-02T14:39:03.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.239+0000 7f73f27d1740 10 client.26518 fetch_fsmap learned FSMap version 27 2024-04-02T14:39:03.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.239+0000 7f73f27d1740 1 -- 172.21.15.90:0/2459898452 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55be7f3430f0 con 0x55be7f714a90 2024-04-02T14:39:03.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.239+0000 7f73cea00640 1 -- 172.21.15.90:0/2459898452 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55be7f610630 con 0x55be7f714a90 2024-04-02T14:39:03.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.239+0000 7f73cea00640 1 -- 172.21.15.90:0/2459898452 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f73d4003710 con 0x55be7f714a90 2024-04-02T14:39:03.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.239+0000 7f73f27d1740 10 client.26518 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:39:03.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.239+0000 7f73f27d1740 10 client.26518 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:39:03.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.239+0000 7f73f27d1740 20 client.26518 populate_metadata read hostname 'smithi090' 2024-04-02T14:39:03.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.239+0000 7f73f27d1740 1 --2- 172.21.15.90:0/2459898452 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55be7f72d6b0 0x55be7f72faa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:03.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.239+0000 7f73f27d1740 4 client.26518 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:39:03.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.239+0000 7f73f27d1740 1 -- 172.21.15.90:0/2459898452 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55be7f67ccb0 con 0x55be7f72d6b0 2024-04-02T14:39:03.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.240+0000 7f73eda00640 1 --2- 172.21.15.90:0/2459898452 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55be7f72d6b0 0x55be7f72faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:03.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.241+0000 7f73eda00640 1 --2- 172.21.15.90:0/2459898452 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55be7f72d6b0 0x55be7f72faa0 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6cc553566ab11252 server_cookie=99428b82daa2c59e in_seq=0 out_seq=0 2024-04-02T14:39:03.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.241+0000 7f73cea00640 10 client.26518 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:39:03.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.241+0000 7f73cea00640 1 -- 172.21.15.90:0/2459898452 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55be7f67ccb0 con 0x55be7f72d6b0 2024-04-02T14:39:03.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.241+0000 7f73cea00640 10 client.26518 handle_command_reply: tid=0 2024-04-02T14:39:03.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.241+0000 7f73f27d1740 1 client.26518 shutdown 2024-04-02T14:39:03.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.242+0000 7f73f27d1740 1 -- 172.21.15.90:0/2459898452 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55be7f714a90 msgr2=0x55be7f71acb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:03.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.242+0000 7f73f27d1740 1 --2- 172.21.15.90:0/2459898452 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55be7f714a90 0x55be7f71acb0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f73e806b180 tx=0x7f73d4004da0 comp rx=0 tx=0).stop 2024-04-02T14:39:03.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.242+0000 7f73f27d1740 1 -- 172.21.15.90:0/2459898452 shutdown_connections 2024-04-02T14:39:03.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.242+0000 7f73f27d1740 1 --2- 172.21.15.90:0/2459898452 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55be7f72d6b0 0x55be7f72faa0 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:03.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.242+0000 7f73f27d1740 1 --2- 172.21.15.90:0/2459898452 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55be7f71d670 0x55be7f71b1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:03.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.242+0000 7f73f27d1740 1 --2- 172.21.15.90:0/2459898452 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55be7f714a90 0x55be7f71acb0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:03.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.242+0000 7f73cea00640 0 client.26518 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:39:03.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.242+0000 7f73f27d1740 1 --2- 172.21.15.90:0/2459898452 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55be7f714170 0x55be7f71a770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:03.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.242+0000 7f73f27d1740 1 -- 172.21.15.90:0/2459898452 >> 172.21.15.90:0/2459898452 conn(0x55be7f70e8a0 msgr2=0x55be7f70fd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:39:03.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.242+0000 7f73f27d1740 1 -- 172.21.15.90:0/2459898452 shutdown_connections 2024-04-02T14:39:03.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.242+0000 7f73f27d1740 1 -- 172.21.15.90:0/2459898452 wait complete. 2024-04-02T14:39:03.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.242+0000 7f73f27d1740 20 client.26518 trim_cache size 0 max 16384 2024-04-02T14:39:03.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.262+0000 7f73f27d1740 1 Processor -- start 2024-04-02T14:39:03.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.262+0000 7f73f27d1740 1 -- start start 2024-04-02T14:39:03.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.263+0000 7f73f27d1740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55be7f714170 0x55be7f729320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:03.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.263+0000 7f73f27d1740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55be7f714a90 0x55be7f705310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:03.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.263+0000 7f73f27d1740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55be7f71d670 0x55be7f71ae40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:03.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.263+0000 7f73f27d1740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f73e81ab4a0 con 0x55be7f71d670 2024-04-02T14:39:03.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.263+0000 7f73f27d1740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f73e81acfe0 con 0x55be7f714170 2024-04-02T14:39:03.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.263+0000 7f73f27d1740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f73e81abb70 con 0x55be7f714a90 2024-04-02T14:39:03.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.263+0000 7f73e7e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55be7f714a90 0x55be7f705310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:03.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.263+0000 7f73e7e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55be7f714a90 0x55be7f705310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:36944/0 (socket says 172.21.15.90:36944) 2024-04-02T14:39:03.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.263+0000 7f73eda00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55be7f714170 0x55be7f729320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:03.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.263+0000 7f73e7e00640 1 -- 172.21.15.90:0/1277759207 learned_addr learned my addr 172.21.15.90:0/1277759207 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:39:03.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.263+0000 7f73e7e00640 1 -- 172.21.15.90:0/1277759207 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55be7f714170 msgr2=0x55be7f729320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:39:03.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.263+0000 7f73e7e00640 1 --2- 172.21.15.90:0/1277759207 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55be7f714170 0x55be7f729320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:03.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.263+0000 7f73e7e00640 1 -- 172.21.15.90:0/1277759207 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55be7f71d670 msgr2=0x55be7f71ae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:39:03.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.263+0000 7f73e7e00640 1 --2- 172.21.15.90:0/1277759207 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55be7f71d670 0x55be7f71ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:03.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.263+0000 7f73e7e00640 1 -- 172.21.15.90:0/1277759207 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55be7f7148c0 con 0x55be7f714a90 2024-04-02T14:39:03.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.263+0000 7f73eda00640 1 --2- 172.21.15.90:0/1277759207 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55be7f714170 0x55be7f729320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:39:03.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.264+0000 7f73e7e00640 1 --2- 172.21.15.90:0/1277759207 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55be7f714a90 0x55be7f705310 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f73d4006010 tx=0x7f73d4001e70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ab72575b91c3d000 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:39:03.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.264+0000 7f73d9600640 1 -- 172.21.15.90:0/1277759207 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73d4043040 con 0x55be7f714a90 2024-04-02T14:39:03.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.264+0000 7f73d9600640 1 -- 172.21.15.90:0/1277759207 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f73d4009520 con 0x55be7f714a90 2024-04-02T14:39:03.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.264+0000 7f73d9600640 1 -- 172.21.15.90:0/1277759207 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73d40056a0 con 0x55be7f714a90 2024-04-02T14:39:03.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.265+0000 7f73f27d1740 1 -- 172.21.15.90:0/1277759207 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55be7f714a90 msgr2=0x55be7f705310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:03.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.265+0000 7f73f27d1740 1 --2- 172.21.15.90:0/1277759207 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55be7f714a90 0x55be7f705310 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f73d4006010 tx=0x7f73d4001e70 comp rx=0 tx=0).stop 2024-04-02T14:39:03.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.265+0000 7f73f27d1740 1 -- 172.21.15.90:0/1277759207 shutdown_connections 2024-04-02T14:39:03.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.265+0000 7f73f27d1740 1 --2- 172.21.15.90:0/1277759207 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55be7f71d670 0x55be7f71ae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:03.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.265+0000 7f73f27d1740 1 --2- 172.21.15.90:0/1277759207 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55be7f714a90 0x55be7f705310 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:03.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.265+0000 7f73f27d1740 1 --2- 172.21.15.90:0/1277759207 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55be7f714170 0x55be7f729320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:03.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.265+0000 7f73f27d1740 1 -- 172.21.15.90:0/1277759207 >> 172.21.15.90:0/1277759207 conn(0x55be7f70e8a0 msgr2=0x55be7f71a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:39:03.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.265+0000 7f73f27d1740 1 -- 172.21.15.90:0/1277759207 shutdown_connections 2024-04-02T14:39:03.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.265+0000 7f73f27d1740 1 -- 172.21.15.90:0/1277759207 wait complete. 2024-04-02T14:39:03.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.265+0000 7f73f27d1740 1 -- start start 2024-04-02T14:39:03.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.266+0000 7f73f27d1740 1 Processor -- start 2024-04-02T14:39:03.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.266+0000 7f73f27d1740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55be7f714170 0x55be7f721650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:03.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.266+0000 7f73f27d1740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55be7f714a90 0x55be7f721b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:03.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.266+0000 7f73f27d1740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55be7f71d670 0x55be7f792c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:03.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.266+0000 7f73f27d1740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f73e81ac910 con 0x55be7f714a90 2024-04-02T14:39:03.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.266+0000 7f73f27d1740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f73e81ac240 con 0x55be7f714170 2024-04-02T14:39:03.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.266+0000 7f73f27d1740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f73e81a9480 con 0x55be7f71d670 2024-04-02T14:39:03.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.266+0000 7f73f27d1740 10 client.0 _ll_register_callbacks cb 0x7f73dc005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:39:03.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.266+0000 7f73e7e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55be7f714a90 0x55be7f721b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:03.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.266+0000 7f73eda00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55be7f714170 0x55be7f721650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:03.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.266+0000 7f73e7e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55be7f714a90 0x55be7f721b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:53816/0 (socket says 172.21.15.90:53816) 2024-04-02T14:39:03.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.266+0000 7f73e7e00640 1 -- 172.21.15.90:0/1717655164 learned_addr learned my addr 172.21.15.90:0/1717655164 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:39:03.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.266+0000 7f73eda00640 1 --2- 172.21.15.90:0/1717655164 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55be7f71d670 0x55be7f792c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:03.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.267+0000 7f73e7e00640 1 -- 172.21.15.90:0/1717655164 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55be7f71d670 msgr2=0x55be7f792c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:39:03.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.267+0000 7f73e7e00640 1 --2- 172.21.15.90:0/1717655164 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55be7f71d670 0x55be7f792c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:03.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.267+0000 7f73e7e00640 1 -- 172.21.15.90:0/1717655164 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55be7f714170 msgr2=0x55be7f721650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:39:03.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.267+0000 7f73e7e00640 1 --2- 172.21.15.90:0/1717655164 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55be7f714170 0x55be7f721650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:03.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.267+0000 7f73e7e00640 1 -- 172.21.15.90:0/1717655164 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55be7f793140 con 0x55be7f714a90 2024-04-02T14:39:03.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.267+0000 7f73e7e00640 1 --2- 172.21.15.90:0/1717655164 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55be7f714a90 0x55be7f721b90 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f73d4003b60 tx=0x7f73d4004d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=525626e5e2d04b8a server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:39:03.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.267+0000 7f73cea00640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-02T14:39:03.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.267+0000 7f73cea00640 1 -- 172.21.15.90:0/1717655164 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73d405f030 con 0x55be7f714a90 2024-04-02T14:39:03.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.267+0000 7f73cea00640 1 -- 172.21.15.90:0/1717655164 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f73d4004310 con 0x55be7f714a90 2024-04-02T14:39:03.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.267+0000 7f73f27d1740 1 -- 172.21.15.90:0/1717655164 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f73d4009520 con 0x55be7f714a90 2024-04-02T14:39:03.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.267+0000 7f73cea00640 1 -- 172.21.15.90:0/1717655164 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f73d4043040 con 0x55be7f714a90 2024-04-02T14:39:03.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.268+0000 7f73f27d1740 10 client.16149 fetch_fsmap learned FSMap version 27 2024-04-02T14:39:03.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.268+0000 7f73f27d1740 1 -- 172.21.15.90:0/1717655164 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55be7f7942f0 con 0x55be7f714a90 2024-04-02T14:39:03.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.268+0000 7f73cea00640 1 -- 172.21.15.90:0/1717655164 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f73d4009520 con 0x55be7f714a90 2024-04-02T14:39:03.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.268+0000 7f73cea00640 1 -- 172.21.15.90:0/1717655164 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f73d4005520 con 0x55be7f714a90 2024-04-02T14:39:03.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.268+0000 7f73f27d1740 10 client.16149 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:39:03.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.268+0000 7f73f27d1740 10 client.16149 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:39:03.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.268+0000 7f73f27d1740 20 client.16149 populate_metadata read hostname 'smithi090' 2024-04-02T14:39:03.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.268+0000 7f73f27d1740 1 --2- 172.21.15.90:0/1717655164 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55be7f7949b0 0x55be7f796dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:03.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.268+0000 7f73f27d1740 4 client.16149 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-02T14:39:03.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.268+0000 7f73f27d1740 1 -- 172.21.15.90:0/1717655164 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55be7f677000 con 0x55be7f7949b0 2024-04-02T14:39:03.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.268+0000 7f73eda00640 1 --2- 172.21.15.90:0/1717655164 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55be7f7949b0 0x55be7f796dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:03.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.269+0000 7f73eda00640 1 --2- 172.21.15.90:0/1717655164 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55be7f7949b0 0x55be7f796dd0 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bf50620b6f6b0b41 server_cookie=e5ed6117ef5116ec in_seq=0 out_seq=0 2024-04-02T14:39:03.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.269+0000 7f73cea00640 10 client.16149 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:39:03.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.269+0000 7f73cea00640 1 -- 172.21.15.90:0/1717655164 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55be7f677000 con 0x55be7f7949b0 2024-04-02T14:39:03.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.269+0000 7f73cea00640 10 client.16149 handle_command_reply: tid=0 2024-04-02T14:39:03.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.269+0000 7f73f27d1740 1 client.16149 shutdown 2024-04-02T14:39:03.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.269+0000 7f73f27d1740 1 -- 172.21.15.90:0/1717655164 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55be7f714a90 msgr2=0x55be7f721b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:03.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.269+0000 7f73f27d1740 1 --2- 172.21.15.90:0/1717655164 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55be7f714a90 0x55be7f721b90 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f73d4003b60 tx=0x7f73d4004d80 comp rx=0 tx=0).stop 2024-04-02T14:39:03.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.269+0000 7f73f27d1740 1 -- 172.21.15.90:0/1717655164 shutdown_connections 2024-04-02T14:39:03.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.269+0000 7f73f27d1740 1 --2- 172.21.15.90:0/1717655164 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55be7f7949b0 0x55be7f796dd0 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:03.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.269+0000 7f73f27d1740 1 --2- 172.21.15.90:0/1717655164 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55be7f71d670 0x55be7f792c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:03.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.269+0000 7f73f27d1740 1 --2- 172.21.15.90:0/1717655164 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55be7f714a90 0x55be7f721b90 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:03.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.269+0000 7f73cea00640 0 client.16149 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:39:03.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.269+0000 7f73f27d1740 1 --2- 172.21.15.90:0/1717655164 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55be7f714170 0x55be7f721650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:03.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.269+0000 7f73f27d1740 1 -- 172.21.15.90:0/1717655164 >> 172.21.15.90:0/1717655164 conn(0x55be7f70e8a0 msgr2=0x55be7f705190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:39:03.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.269+0000 7f73f27d1740 1 -- 172.21.15.90:0/1717655164 shutdown_connections 2024-04-02T14:39:03.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.269+0000 7f73f27d1740 1 -- 172.21.15.90:0/1717655164 wait complete. 2024-04-02T14:39:03.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.270+0000 7f73f27d1740 20 client.16149 trim_cache size 0 max 16384 2024-04-02T14:39:03.281 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-02T14:39:03.282 INFO:teuthology.orchestra.run.smithi090.stdout: "status": "no active scrubs running", 2024-04-02T14:39:03.282 INFO:teuthology.orchestra.run.smithi090.stdout: "scrubs": {} 2024-04-02T14:39:03.282 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-02T14:39:03.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.270+0000 7f73ee400640 1 -- 172.21.15.90:0/3166179885 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f73d00a3da0 msgr2=0x7f73d00a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:03.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.270+0000 7f73ee400640 1 --2- 172.21.15.90:0/3166179885 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f73d00a3da0 0x7f73d00a6260 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x55be7f713c40 tx=0x7f73dc003f30 comp rx=0 tx=0).stop 2024-04-02T14:39:03.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.270+0000 7f73ee400640 1 -- 172.21.15.90:0/3166179885 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f73e8103dd0 msgr2=0x7f73e819a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:03.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.270+0000 7f73ee400640 1 --2- 172.21.15.90:0/3166179885 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f73e8103dd0 0x7f73e819a480 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f73dc00ea60 tx=0x7f73dc00ef20 comp rx=0 tx=0).stop 2024-04-02T14:39:03.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.270+0000 7f73ee400640 1 -- 172.21.15.90:0/3166179885 shutdown_connections 2024-04-02T14:39:03.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.270+0000 7f73ee400640 1 --2- 172.21.15.90:0/3166179885 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f73d00a3da0 0x7f73d00a6260 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:03.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.270+0000 7f73ee400640 1 --2- 172.21.15.90:0/3166179885 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f73e819ed40 0x7f73e819f1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:03.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.270+0000 7f73ee400640 1 --2- 172.21.15.90:0/3166179885 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f73e8108ea0 0x7f73e819a9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:03.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.270+0000 7f73ee400640 1 --2- 172.21.15.90:0/3166179885 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f73e8103dd0 0x7f73e819a480 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:03.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.270+0000 7f73ee400640 1 -- 172.21.15.90:0/3166179885 >> 172.21.15.90:0/3166179885 conn(0x7f73e80ff610 msgr2=0x7f73e81018d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:39:03.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.271+0000 7f73ee400640 1 -- 172.21.15.90:0/3166179885 shutdown_connections 2024-04-02T14:39:03.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.271+0000 7f73ee400640 1 -- 172.21.15.90:0/3166179885 wait complete. 2024-04-02T14:39:03.290 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-02T14:39:03.291 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T14:39:03.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.506+0000 7fa75a800640 1 -- 172.21.15.90:0/1550737337 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa754069600 msgr2=0x7fa75410e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:03.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.506+0000 7fa75a800640 1 --2- 172.21.15.90:0/1550737337 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa754069600 0x7fa75410e7a0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fa74c009cd0 tx=0x7fa74c02c0e0 comp rx=0 tx=0).stop 2024-04-02T14:39:03.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.506+0000 7fa75a800640 1 -- 172.21.15.90:0/1550737337 shutdown_connections 2024-04-02T14:39:03.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.506+0000 7fa75a800640 1 --2- 172.21.15.90:0/1550737337 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa754069600 0x7fa75410e7a0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:03.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.506+0000 7fa75a800640 1 --2- 172.21.15.90:0/1550737337 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa754068c60 0x7fa7540690c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:03.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.507+0000 7fa75a800640 1 --2- 172.21.15.90:0/1550737337 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa754107950 0x7fa754107d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:03.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.507+0000 7fa75a800640 1 -- 172.21.15.90:0/1550737337 >> 172.21.15.90:0/1550737337 conn(0x7fa754076410 msgr2=0x7fa754076820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:39:03.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.507+0000 7fa75a800640 1 -- 172.21.15.90:0/1550737337 shutdown_connections 2024-04-02T14:39:03.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.507+0000 7fa75a800640 1 -- 172.21.15.90:0/1550737337 wait complete. 2024-04-02T14:39:03.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.508+0000 7fa75a800640 1 Processor -- start 2024-04-02T14:39:03.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.508+0000 7fa75a800640 1 -- start start 2024-04-02T14:39:03.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.508+0000 7fa75a800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa754068c60 0x7fa754198170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:03.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.508+0000 7fa75a800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa754069600 0x7fa7541986b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:03.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.508+0000 7fa75a800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa754107950 0x7fa75419ca30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:03.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.508+0000 7fa75a800640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fa754111d30 con 0x7fa754069600 2024-04-02T14:39:03.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.508+0000 7fa75a800640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fa754111bb0 con 0x7fa754068c60 2024-04-02T14:39:03.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.508+0000 7fa75a800640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fa754111eb0 con 0x7fa754107950 2024-04-02T14:39:03.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.508+0000 7fa759400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa754068c60 0x7fa754198170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:03.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.509+0000 7fa759400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa754068c60 0x7fa754198170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:52200/0 (socket says 172.21.15.90:52200) 2024-04-02T14:39:03.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.509+0000 7fa759400640 1 -- 172.21.15.90:0/3500657956 learned_addr learned my addr 172.21.15.90:0/3500657956 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:39:03.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.509+0000 7fa759e00640 1 --2- 172.21.15.90:0/3500657956 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa754107950 0x7fa75419ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:03.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.509+0000 7fa758a00640 1 --2- 172.21.15.90:0/3500657956 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa754069600 0x7fa7541986b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:03.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.509+0000 7fa759400640 1 -- 172.21.15.90:0/3500657956 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa754107950 msgr2=0x7fa75419ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:03.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.509+0000 7fa759400640 1 --2- 172.21.15.90:0/3500657956 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa754107950 0x7fa75419ca30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:03.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.509+0000 7fa759400640 1 -- 172.21.15.90:0/3500657956 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa754069600 msgr2=0x7fa7541986b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:03.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.509+0000 7fa759400640 1 --2- 172.21.15.90:0/3500657956 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa754069600 0x7fa7541986b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:03.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.509+0000 7fa759400640 1 -- 172.21.15.90:0/3500657956 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa75419d160 con 0x7fa754068c60 2024-04-02T14:39:03.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.509+0000 7fa759400640 1 --2- 172.21.15.90:0/3500657956 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa754068c60 0x7fa754198170 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fa74800e9f0 tx=0x7fa74800edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:39:03.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.509+0000 7fa752000640 1 -- 172.21.15.90:0/3500657956 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa74800cd80 con 0x7fa754068c60 2024-04-02T14:39:03.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.509+0000 7fa752000640 1 -- 172.21.15.90:0/3500657956 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa748005500 con 0x7fa754068c60 2024-04-02T14:39:03.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.509+0000 7fa75a800640 1 -- 172.21.15.90:0/3500657956 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa75419d450 con 0x7fa754068c60 2024-04-02T14:39:03.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.510+0000 7fa752000640 1 -- 172.21.15.90:0/3500657956 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa748010690 con 0x7fa754068c60 2024-04-02T14:39:03.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.510+0000 7fa75a800640 1 -- 172.21.15.90:0/3500657956 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7541a4ce0 con 0x7fa754068c60 2024-04-02T14:39:03.514 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.511+0000 7fa75a800640 1 -- 172.21.15.90:0/3500657956 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa754068810 con 0x7fa754068c60 2024-04-02T14:39:03.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.513+0000 7fa752000640 1 -- 172.21.15.90:0/3500657956 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fa7480046c0 con 0x7fa754068c60 2024-04-02T14:39:03.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.515+0000 7fa752000640 1 --2- 172.21.15.90:0/3500657956 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fa73c0a3d20 0x7fa73c0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:03.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.515+0000 7fa752000640 1 -- 172.21.15.90:0/3500657956 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7fa7480d0c80 con 0x7fa754068c60 2024-04-02T14:39:03.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.515+0000 7fa758a00640 1 --2- 172.21.15.90:0/3500657956 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fa73c0a3d20 0x7fa73c0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:03.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.516+0000 7fa758a00640 1 --2- 172.21.15.90:0/3500657956 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fa73c0a3d20 0x7fa73c0a61e0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fa7400095f0 tx=0x7fa740005f90 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:39:03.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.519+0000 7fa752000640 1 -- 172.21.15.90:0/3500657956 <== mon.2 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7fa74809d2d0 con 0x7fa754068c60 2024-04-02T14:39:03.822 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.819+0000 7f4b54000640 1 -- 172.21.15.90:0/2470230630 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4b4c103f40 msgr2=0x7f4b4c104320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:03.822 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.819+0000 7f4b54000640 1 --2- 172.21.15.90:0/2470230630 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4b4c103f40 0x7f4b4c104320 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f4b40009900 tx=0x7f4b4002bc40 comp rx=0 tx=0).stop 2024-04-02T14:39:03.822 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.819+0000 7f4b54000640 1 -- 172.21.15.90:0/2470230630 shutdown_connections 2024-04-02T14:39:03.822 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.819+0000 7f4b54000640 1 --2- 172.21.15.90:0/2470230630 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4b4c109010 0x7f4b4c110be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:03.822 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.819+0000 7f4b54000640 1 --2- 172.21.15.90:0/2470230630 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4b4c1048f0 0x7f4b4c108930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:03.822 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.819+0000 7f4b54000640 1 --2- 172.21.15.90:0/2470230630 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4b4c103f40 0x7f4b4c104320 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:03.822 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.819+0000 7f4b54000640 1 -- 172.21.15.90:0/2470230630 >> 172.21.15.90:0/2470230630 conn(0x7f4b4c0ff7c0 msgr2=0x7f4b4c101be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:39:03.823 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.819+0000 7f4b54000640 1 -- 172.21.15.90:0/2470230630 shutdown_connections 2024-04-02T14:39:03.823 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.820+0000 7f4b54000640 1 -- 172.21.15.90:0/2470230630 wait complete. 2024-04-02T14:39:03.825 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.820+0000 7f4b54000640 1 Processor -- start 2024-04-02T14:39:03.825 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.821+0000 7f4b54000640 1 -- start start 2024-04-02T14:39:03.825 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.821+0000 7f4b54000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4b4c103f40 0x7f4b4c19a5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:03.825 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.821+0000 7f4b54000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4b4c109010 0x7f4b4c19aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:03.825 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.821+0000 7f4b54000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4b4c19ee70 0x7f4b4c19f320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:03.825 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.821+0000 7f4b54000640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f4b4c1140b0 con 0x7f4b4c109010 2024-04-02T14:39:03.825 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.821+0000 7f4b54000640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f4b4c113f30 con 0x7f4b4c19ee70 2024-04-02T14:39:03.825 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.821+0000 7f4b54000640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f4b4c114230 con 0x7f4b4c103f40 2024-04-02T14:39:03.825 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.821+0000 7f4b51600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4b4c103f40 0x7f4b4c19a5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:03.825 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.821+0000 7f4b51600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4b4c103f40 0x7f4b4c19a5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:36996/0 (socket says 172.21.15.90:36996) 2024-04-02T14:39:03.825 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.821+0000 7f4b51600640 1 -- 172.21.15.90:0/3743300397 learned_addr learned my addr 172.21.15.90:0/3743300397 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:39:03.825 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.821+0000 7f4b52000640 1 --2- 172.21.15.90:0/3743300397 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4b4c19ee70 0x7f4b4c19f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:03.826 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.821+0000 7f4b50c00640 1 --2- 172.21.15.90:0/3743300397 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4b4c109010 0x7f4b4c19aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:03.826 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.822+0000 7f4b51600640 1 -- 172.21.15.90:0/3743300397 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4b4c19ee70 msgr2=0x7f4b4c19f320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:03.826 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.822+0000 7f4b51600640 1 --2- 172.21.15.90:0/3743300397 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4b4c19ee70 0x7f4b4c19f320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:03.826 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.822+0000 7f4b51600640 1 -- 172.21.15.90:0/3743300397 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4b4c109010 msgr2=0x7f4b4c19aaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:03.826 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.822+0000 7f4b51600640 1 --2- 172.21.15.90:0/3743300397 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4b4c109010 0x7f4b4c19aaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:03.826 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.822+0000 7f4b51600640 1 -- 172.21.15.90:0/3743300397 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b4c19f960 con 0x7f4b4c103f40 2024-04-02T14:39:03.826 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.822+0000 7f4b50c00640 1 --2- 172.21.15.90:0/3743300397 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4b4c109010 0x7f4b4c19aaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:39:03.826 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.822+0000 7f4b51600640 1 --2- 172.21.15.90:0/3743300397 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4b4c103f40 0x7f4b4c19a5b0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f4b40006010 tx=0x7f4b4002cb40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:39:03.826 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.822+0000 7f4b4a000640 1 -- 172.21.15.90:0/3743300397 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b40041070 con 0x7f4b4c103f40 2024-04-02T14:39:03.826 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.822+0000 7f4b4a000640 1 -- 172.21.15.90:0/3743300397 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4b40004450 con 0x7f4b4c103f40 2024-04-02T14:39:03.826 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.822+0000 7f4b4a000640 1 -- 172.21.15.90:0/3743300397 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b40003e30 con 0x7f4b4c103f40 2024-04-02T14:39:03.826 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.822+0000 7f4b54000640 1 -- 172.21.15.90:0/3743300397 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b4c19fbf0 con 0x7f4b4c103f40 2024-04-02T14:39:03.826 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.823+0000 7f4b54000640 1 -- 172.21.15.90:0/3743300397 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b4c1a7360 con 0x7f4b4c103f40 2024-04-02T14:39:03.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.823+0000 7fa75a800640 1 -- 172.21.15.90:0/3500657956 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa754109e50 con 0x7fa754068c60 2024-04-02T14:39:03.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.824+0000 7fa752000640 1 -- 172.21.15.90:0/3500657956 <== mon.2 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11927 (secure 0 0 0) 0x7fa74809cbd0 con 0x7fa754068c60 2024-04-02T14:39:03.827 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-02T14:39:03.828 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14412,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6837/2852160309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6836","nonce":2852160309},{"type":"v1","addr":"172.21.15.90:6837","nonce":2852160309}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14478,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6841/1919376858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6840","nonce":1919376858},{"type":"v1","addr":"172.21.15.90:6841","nonce":1919376858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-02T14:32:22.319083+0000","modified":"2024-04-02T14:34:52.266152+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24590,"mds_1":24560,"mds_2":24547,"mds_3":24539,"mds_4":24523},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24454":{"gid":24454,"name":"l","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6833/1564319431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":1564319431},{"type":"v1","addr":"172.21.15.177:6833","nonce":1564319431}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.92:6833/1570337372","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":1570337372},{"type":"v1","addr":"172.21.15.92:6833","nonce":1570337372}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"j","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.90:6835/1277124891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6834","nonce":1277124891},{"type":"v1","addr":"172.21.15.90:6835","nonce":1277124891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.92:6835/860721164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":860721164},{"type":"v1","addr":"172.21.15.92:6835","nonce":860721164}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24511":{"gid":24511,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6835/3177244988","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":3177244988},{"type":"v1","addr":"172.21.15.177:6835","nonce":3177244988}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"c","rank":4,"incarnation":25,"state":"up:active","state_seq":9,"addr":"172.21.15.177:6837/4223414858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":4223414858},{"type":"v1","addr":"172.21.15.177:6837","nonce":4223414858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24539":{"gid":24539,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":8,"addr":"172.21.15.92:6837/1270635939","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":1270635939},{"type":"v1","addr":"172.21.15.92:6837","nonce":1270635939}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24547":{"gid":24547,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":7,"addr":"172.21.15.177:6839/664895663","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":664895663},{"type":"v1","addr":"172.21.15.177:6839","nonce":664895663}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"g","rank":1,"incarnation":19,"state":"up:active","state_seq":5,"addr":"172.21.15.90:6839/1498915870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6838","nonce":1498915870},{"type":"v1","addr":"172.21.15.90:6839","nonce":1498915870}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24590":{"gid":24590,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.92:6839/1054888219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":1054888219},{"type":"v1","addr":"172.21.15.92:6839","nonce":1054888219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24590,"qdb_cluster":[24547,24590,24523,24539,24560]},"id":1}]} 2024-04-02T14:39:03.828 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 27 2024-04-02T14:39:03.829 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.826+0000 7f4b56d70740 1 Processor -- start 2024-04-02T14:39:03.829 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.826+0000 7f4b56d70740 1 -- start start 2024-04-02T14:39:03.829 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.826+0000 7f4b4a000640 1 -- 172.21.15.90:0/3743300397 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f4b40044070 con 0x7f4b4c103f40 2024-04-02T14:39:03.829 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.826+0000 7f4b56d70740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e800ccaeb0 0x55e800ccb290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:03.829 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.826+0000 7f4b56d70740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e800ccb7d0 0x55e800cd3e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:03.829 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.826+0000 7f4b56d70740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e800cd43b0 0x55e800cd67a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:03.829 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.826+0000 7f4b56d70740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55e800c2fd70 con 0x55e800cd43b0 2024-04-02T14:39:03.829 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.826+0000 7f4b56d70740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55e800bd82a0 con 0x55e800ccb7d0 2024-04-02T14:39:03.829 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.826+0000 7f4b56d70740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55e800c2bd20 con 0x55e800ccaeb0 2024-04-02T14:39:03.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.826+0000 7fa75a800640 1 -- 172.21.15.90:0/3500657956 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fa73c0a3d20 msgr2=0x7fa73c0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:03.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.827+0000 7fa75a800640 1 --2- 172.21.15.90:0/3500657956 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fa73c0a3d20 0x7fa73c0a61e0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fa7400095f0 tx=0x7fa740005f90 comp rx=0 tx=0).stop 2024-04-02T14:39:03.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.827+0000 7fa75a800640 1 -- 172.21.15.90:0/3500657956 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa754068c60 msgr2=0x7fa754198170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:03.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.827+0000 7fa75a800640 1 --2- 172.21.15.90:0/3500657956 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa754068c60 0x7fa754198170 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fa74800e9f0 tx=0x7fa74800edf0 comp rx=0 tx=0).stop 2024-04-02T14:39:03.830 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.827+0000 7f4b52000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e800ccaeb0 0x55e800ccb290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:03.831 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.827+0000 7f4b52000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e800ccaeb0 0x55e800ccb290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:37006/0 (socket says 172.21.15.90:37006) 2024-04-02T14:39:03.831 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.827+0000 7f4b51600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e800ccb7d0 0x55e800cd3e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:03.831 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.827+0000 7f4b52000640 1 -- 172.21.15.90:0/2225828429 learned_addr learned my addr 172.21.15.90:0/2225828429 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:39:03.831 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.827+0000 7f4b50c00640 1 --2- 172.21.15.90:0/2225828429 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e800cd43b0 0x55e800cd67a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:03.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.827+0000 7fa75a800640 1 -- 172.21.15.90:0/3500657956 shutdown_connections 2024-04-02T14:39:03.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.827+0000 7fa75a800640 1 --2- 172.21.15.90:0/3500657956 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fa73c0a3d20 0x7fa73c0a61e0 secure :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fa7400095f0 tx=0x7fa740005f90 comp rx=0 tx=0).stop 2024-04-02T14:39:03.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.827+0000 7fa75a800640 1 --2- 172.21.15.90:0/3500657956 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa754107950 0x7fa75419ca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:03.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.827+0000 7fa75a800640 1 --2- 172.21.15.90:0/3500657956 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa754069600 0x7fa7541986b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:03.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.827+0000 7fa75a800640 1 --2- 172.21.15.90:0/3500657956 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa754068c60 0x7fa754198170 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:03.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.827+0000 7fa75a800640 1 -- 172.21.15.90:0/3500657956 >> 172.21.15.90:0/3500657956 conn(0x7fa754076410 msgr2=0x7fa7540feac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:39:03.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.827+0000 7fa75a800640 1 -- 172.21.15.90:0/3500657956 shutdown_connections 2024-04-02T14:39:03.832 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.827+0000 7f4b52000640 1 -- 172.21.15.90:0/2225828429 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e800ccb7d0 msgr2=0x55e800cd3e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:39:03.832 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.827+0000 7f4b52000640 1 --2- 172.21.15.90:0/2225828429 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e800ccb7d0 0x55e800cd3e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:03.832 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.827+0000 7f4b52000640 1 -- 172.21.15.90:0/2225828429 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e800cd43b0 msgr2=0x55e800cd67a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:39:03.832 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.827+0000 7f4b52000640 1 --2- 172.21.15.90:0/2225828429 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e800cd43b0 0x55e800cd67a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:03.832 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.827+0000 7f4b52000640 1 -- 172.21.15.90:0/2225828429 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e800bec840 con 0x55e800ccaeb0 2024-04-02T14:39:03.833 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.827+0000 7f4b50c00640 1 --2- 172.21.15.90:0/2225828429 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e800cd43b0 0x55e800cd67a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:39:03.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:03.827+0000 7fa75a800640 1 -- 172.21.15.90:0/3500657956 wait complete. 2024-04-02T14:39:03.836 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.828+0000 7f4b52000640 1 --2- 172.21.15.90:0/2225828429 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e800ccaeb0 0x55e800ccb290 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f4b44009f50 tx=0x7f4b440118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=36958e0e4eed78b server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:39:03.836 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.828+0000 7f4b3e600640 1 -- 172.21.15.90:0/2225828429 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b44004d20 con 0x55e800ccaeb0 2024-04-02T14:39:03.836 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.828+0000 7f4b4a000640 1 --2- 172.21.15.90:0/3743300397 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f4b340a3e40 0x7f4b340a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:03.836 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.828+0000 7f4b3e600640 1 -- 172.21.15.90:0/2225828429 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4b44004050 con 0x55e800ccaeb0 2024-04-02T14:39:03.836 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.828+0000 7f4b3e600640 1 -- 172.21.15.90:0/2225828429 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b44005280 con 0x55e800ccaeb0 2024-04-02T14:39:03.836 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.828+0000 7f4b4a000640 1 -- 172.21.15.90:0/3743300397 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f4b400f24a0 con 0x7f4b4c103f40 2024-04-02T14:39:03.836 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.828+0000 7f4b51600640 1 --2- 172.21.15.90:0/3743300397 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f4b340a3e40 0x7f4b340a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:03.836 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.829+0000 7f4b56d70740 1 -- 172.21.15.90:0/2225828429 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e800ccaeb0 msgr2=0x55e800ccb290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:03.836 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.829+0000 7f4b51600640 1 --2- 172.21.15.90:0/3743300397 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f4b340a3e40 0x7f4b340a6300 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f4b400efdc0 tx=0x7f4b400f8060 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:39:03.836 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.829+0000 7f4b56d70740 1 --2- 172.21.15.90:0/2225828429 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e800ccaeb0 0x55e800ccb290 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f4b44009f50 tx=0x7f4b440118d0 comp rx=0 tx=0).stop 2024-04-02T14:39:03.837 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.829+0000 7f4b56d70740 1 -- 172.21.15.90:0/2225828429 shutdown_connections 2024-04-02T14:39:03.837 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.829+0000 7f4b56d70740 1 --2- 172.21.15.90:0/2225828429 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e800cd43b0 0x55e800cd67a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:03.837 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.829+0000 7f4b56d70740 1 --2- 172.21.15.90:0/2225828429 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e800ccb7d0 0x55e800cd3e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:03.837 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.829+0000 7f4b56d70740 1 --2- 172.21.15.90:0/2225828429 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e800ccaeb0 0x55e800ccb290 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:03.837 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.829+0000 7f4b56d70740 1 -- 172.21.15.90:0/2225828429 >> 172.21.15.90:0/2225828429 conn(0x55e800cc55e0 msgr2=0x55e800cc59c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:39:03.837 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.829+0000 7f4b56d70740 1 -- 172.21.15.90:0/2225828429 shutdown_connections 2024-04-02T14:39:03.837 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.829+0000 7f4b56d70740 1 -- 172.21.15.90:0/2225828429 wait complete. 2024-04-02T14:39:03.837 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.830+0000 7f4b56d70740 1 -- start start 2024-04-02T14:39:03.837 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.830+0000 7f4b56d70740 1 Processor -- start 2024-04-02T14:39:03.837 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.831+0000 7f4b56d70740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e800ccaeb0 0x55e800cd13e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:03.837 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.831+0000 7f4b56d70740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e800ccb7d0 0x55e800cd1920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:03.837 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.831+0000 7f4b56d70740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e800cd43b0 0x55e800cd1e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:03.837 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.831+0000 7f4b56d70740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55e800cd71a0 con 0x55e800cd43b0 2024-04-02T14:39:03.837 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.831+0000 7f4b56d70740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55e800beb660 con 0x55e800ccb7d0 2024-04-02T14:39:03.837 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.831+0000 7f4b56d70740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55e800cd7320 con 0x55e800ccaeb0 2024-04-02T14:39:03.838 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.831+0000 7f4b56d70740 10 client.0 _ll_register_callbacks cb 0x55e800beb050 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:39:03.838 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.831+0000 7f4b50c00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e800ccb7d0 0x55e800cd1920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:03.838 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.831+0000 7f4b50c00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e800ccb7d0 0x55e800cd1920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:41482/0 (socket says 172.21.15.90:41482) 2024-04-02T14:39:03.838 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.831+0000 7f4b50c00640 1 -- 172.21.15.90:0/3579715946 learned_addr learned my addr 172.21.15.90:0/3579715946 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:39:03.838 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.831+0000 7f4b52000640 1 --2- 172.21.15.90:0/3579715946 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e800ccaeb0 0x55e800cd13e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:03.838 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.831+0000 7f4b52000640 1 --2- 172.21.15.90:0/3579715946 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e800cd43b0 0x55e800cd1e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:03.838 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.831+0000 7f4b50c00640 1 -- 172.21.15.90:0/3579715946 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e800ccaeb0 msgr2=0x55e800cd13e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:39:03.838 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.831+0000 7f4b50c00640 1 --2- 172.21.15.90:0/3579715946 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e800ccaeb0 0x55e800cd13e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:03.838 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.831+0000 7f4b50c00640 1 -- 172.21.15.90:0/3579715946 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e800cd43b0 msgr2=0x55e800cd1e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:39:03.838 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.831+0000 7f4b50c00640 1 --2- 172.21.15.90:0/3579715946 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e800cd43b0 0x55e800cd1e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:03.838 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.831+0000 7f4b50c00640 1 -- 172.21.15.90:0/3579715946 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e800bfae80 con 0x55e800ccb7d0 2024-04-02T14:39:03.838 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.833+0000 7f4b50c00640 1 --2- 172.21.15.90:0/3579715946 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e800ccb7d0 0x55e800cd1920 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f4b38012780 tx=0x7f4b38012c40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=771f10275440a45a server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:39:03.838 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.833+0000 7f4b32a00640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:39:03.838 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.833+0000 7f4b32a00640 1 -- 172.21.15.90:0/3579715946 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b3801f060 con 0x55e800ccb7d0 2024-04-02T14:39:03.839 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.833+0000 7f4b32a00640 1 -- 172.21.15.90:0/3579715946 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f4b38010dc0 con 0x55e800ccb7d0 2024-04-02T14:39:03.839 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.833+0000 7f4b56d70740 1 -- 172.21.15.90:0/3579715946 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4b44004050 con 0x55e800ccb7d0 2024-04-02T14:39:03.839 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.833+0000 7f4b32a00640 1 -- 172.21.15.90:0/3579715946 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f4b3801a430 con 0x55e800ccb7d0 2024-04-02T14:39:03.839 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.833+0000 7f4b56d70740 10 client.26657 fetch_fsmap learned FSMap version 27 2024-04-02T14:39:03.839 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.833+0000 7f4b56d70740 1 -- 172.21.15.90:0/3579715946 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55e800944210 con 0x55e800ccb7d0 2024-04-02T14:39:03.839 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.834+0000 7f4b32a00640 1 -- 172.21.15.90:0/3579715946 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4b44004050 con 0x55e800ccb7d0 2024-04-02T14:39:03.841 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.834+0000 7f4b32a00640 1 -- 172.21.15.90:0/3579715946 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f4b380224c0 con 0x55e800ccb7d0 2024-04-02T14:39:03.841 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.834+0000 7f4b56d70740 10 client.26657 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:39:03.841 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.834+0000 7f4b56d70740 10 client.26657 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-02T14:39:03.841 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.834+0000 7f4b56d70740 20 client.26657 populate_metadata read hostname 'smithi090' 2024-04-02T14:39:03.841 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.834+0000 7f4b56d70740 1 --2- 172.21.15.90:0/3579715946 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55e800ce4750 0x55e800ce6b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:03.841 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.834+0000 7f4b56d70740 4 client.26657 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:39:03.841 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.834+0000 7f4b56d70740 1 -- 172.21.15.90:0/3579715946 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e800bdfcd0 con 0x55e800ce4750 2024-04-02T14:39:03.841 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.835+0000 7f4b52000640 1 --2- 172.21.15.90:0/3579715946 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55e800ce4750 0x55e800ce6b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:03.841 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.835+0000 7f4b52000640 1 --2- 172.21.15.90:0/3579715946 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55e800ce4750 0x55e800ce6b40 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d76e6a934677b69b server_cookie=8d499cb20278bdec in_seq=0 out_seq=0 2024-04-02T14:39:03.841 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.835+0000 7f4b32a00640 10 client.26657 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:39:03.841 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.836+0000 7f4b32a00640 1 -- 172.21.15.90:0/3579715946 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e800bdfcd0 con 0x55e800ce4750 2024-04-02T14:39:03.841 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.836+0000 7f4b32a00640 10 client.26657 handle_command_reply: tid=0 2024-04-02T14:39:03.841 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.836+0000 7f4b56d70740 1 client.26657 shutdown 2024-04-02T14:39:03.841 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.836+0000 7f4b56d70740 1 -- 172.21.15.90:0/3579715946 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e800ccb7d0 msgr2=0x55e800cd1920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:03.842 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.836+0000 7f4b56d70740 1 --2- 172.21.15.90:0/3579715946 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e800ccb7d0 0x55e800cd1920 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f4b38012780 tx=0x7f4b38012c40 comp rx=0 tx=0).stop 2024-04-02T14:39:03.842 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.836+0000 7f4b56d70740 1 -- 172.21.15.90:0/3579715946 shutdown_connections 2024-04-02T14:39:03.842 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.836+0000 7f4b56d70740 1 --2- 172.21.15.90:0/3579715946 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55e800ce4750 0x55e800ce6b40 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:03.842 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.836+0000 7f4b56d70740 1 --2- 172.21.15.90:0/3579715946 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e800cd43b0 0x55e800cd1e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:03.842 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.836+0000 7f4b56d70740 1 --2- 172.21.15.90:0/3579715946 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e800ccb7d0 0x55e800cd1920 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:03.842 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.836+0000 7f4b56d70740 1 --2- 172.21.15.90:0/3579715946 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e800ccaeb0 0x55e800cd13e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:03.842 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.836+0000 7f4b56d70740 1 -- 172.21.15.90:0/3579715946 >> 172.21.15.90:0/3579715946 conn(0x55e800cc55e0 msgr2=0x55e800cc7330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:39:03.842 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.836+0000 7f4b32a00640 0 client.26657 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:39:03.842 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.838+0000 7f4b56d70740 1 -- 172.21.15.90:0/3579715946 shutdown_connections 2024-04-02T14:39:03.842 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.838+0000 7f4b56d70740 1 -- 172.21.15.90:0/3579715946 wait complete. 2024-04-02T14:39:03.842 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.838+0000 7f4b56d70740 20 client.26657 trim_cache size 0 max 16384 2024-04-02T14:39:03.851 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-04-02T14:39:03.860 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.858+0000 7f4b56d70740 1 Processor -- start 2024-04-02T14:39:03.860 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.858+0000 7f4b56d70740 1 -- start start 2024-04-02T14:39:03.861 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.858+0000 7f4b56d70740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e800ccaeb0 0x55e800ce0060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:03.861 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.858+0000 7f4b56d70740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e800ccb7d0 0x55e800cd1c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:03.861 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.858+0000 7f4b56d70740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e800cd43b0 0x55e800cec430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:03.861 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.858+0000 7f4b56d70740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f4b4c1af9c0 con 0x55e800cd43b0 2024-04-02T14:39:03.861 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.858+0000 7f4b56d70740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f4b4c1b1500 con 0x55e800ccb7d0 2024-04-02T14:39:03.861 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.858+0000 7f4b56d70740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f4b4c1b0090 con 0x55e800ccaeb0 2024-04-02T14:39:03.862 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.858+0000 7f4b50c00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e800ccb7d0 0x55e800cd1c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:03.862 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.859+0000 7f4b50c00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e800ccb7d0 0x55e800cd1c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:41490/0 (socket says 172.21.15.90:41490) 2024-04-02T14:39:03.862 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.859+0000 7f4b50c00640 1 -- 172.21.15.90:0/1405190920 learned_addr learned my addr 172.21.15.90:0/1405190920 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:39:03.862 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.859+0000 7f4b52000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e800ccaeb0 0x55e800ce0060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:03.862 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.859+0000 7f4b52000640 1 --2- 172.21.15.90:0/1405190920 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e800cd43b0 0x55e800cec430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:03.862 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.859+0000 7f4b50c00640 1 -- 172.21.15.90:0/1405190920 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e800ccaeb0 msgr2=0x55e800ce0060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:39:03.862 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.859+0000 7f4b50c00640 1 --2- 172.21.15.90:0/1405190920 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e800ccaeb0 0x55e800ce0060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:03.862 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.859+0000 7f4b50c00640 1 -- 172.21.15.90:0/1405190920 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e800cd43b0 msgr2=0x55e800cec430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:39:03.862 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.859+0000 7f4b50c00640 1 --2- 172.21.15.90:0/1405190920 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e800cd43b0 0x55e800cec430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:03.862 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.859+0000 7f4b50c00640 1 -- 172.21.15.90:0/1405190920 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e800a10620 con 0x55e800ccb7d0 2024-04-02T14:39:03.862 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.859+0000 7f4b52000640 1 --2- 172.21.15.90:0/1405190920 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e800cd43b0 0x55e800cec430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:39:03.862 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.859+0000 7f4b50c00640 1 --2- 172.21.15.90:0/1405190920 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e800ccb7d0 0x55e800cd1c80 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f4b38001eb0 tx=0x7f4b38010820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d719916242ccdb1a server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:39:03.865 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.860+0000 7f4b3dc00640 1 -- 172.21.15.90:0/1405190920 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b38030070 con 0x55e800ccb7d0 2024-04-02T14:39:03.865 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.860+0000 7f4b3dc00640 1 -- 172.21.15.90:0/1405190920 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4b38010bc0 con 0x55e800ccb7d0 2024-04-02T14:39:03.865 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.860+0000 7f4b3dc00640 1 -- 172.21.15.90:0/1405190920 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b38003850 con 0x55e800ccb7d0 2024-04-02T14:39:03.865 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.860+0000 7f4b56d70740 1 -- 172.21.15.90:0/1405190920 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e800ccb7d0 msgr2=0x55e800cd1c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:03.865 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.860+0000 7f4b56d70740 1 --2- 172.21.15.90:0/1405190920 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e800ccb7d0 0x55e800cd1c80 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f4b38001eb0 tx=0x7f4b38010820 comp rx=0 tx=0).stop 2024-04-02T14:39:03.865 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.860+0000 7f4b56d70740 1 -- 172.21.15.90:0/1405190920 shutdown_connections 2024-04-02T14:39:03.865 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.860+0000 7f4b56d70740 1 --2- 172.21.15.90:0/1405190920 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e800cd43b0 0x55e800cec430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:03.865 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.860+0000 7f4b56d70740 1 --2- 172.21.15.90:0/1405190920 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e800ccb7d0 0x55e800cd1c80 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:03.865 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.860+0000 7f4b56d70740 1 --2- 172.21.15.90:0/1405190920 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e800ccaeb0 0x55e800ce0060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:03.865 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.860+0000 7f4b56d70740 1 -- 172.21.15.90:0/1405190920 >> 172.21.15.90:0/1405190920 conn(0x55e800cc55e0 msgr2=0x55e800cc7170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:39:03.866 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.861+0000 7f4b56d70740 1 -- 172.21.15.90:0/1405190920 shutdown_connections 2024-04-02T14:39:03.866 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.861+0000 7f4b56d70740 1 -- 172.21.15.90:0/1405190920 wait complete. 2024-04-02T14:39:03.866 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.861+0000 7f4b56d70740 1 -- start start 2024-04-02T14:39:03.866 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.861+0000 7f4b56d70740 1 Processor -- start 2024-04-02T14:39:03.866 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.862+0000 7f4b56d70740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e800ccaeb0 0x55e800ccdf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:03.866 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.862+0000 7f4b56d70740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e800ccb7d0 0x55e800cce4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:03.866 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.862+0000 7f4b56d70740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e800cd43b0 0x55e800ccea00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:03.866 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.862+0000 7f4b56d70740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f4b4c1b0e30 con 0x55e800ccaeb0 2024-04-02T14:39:03.866 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.862+0000 7f4b56d70740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f4b4c1b0760 con 0x55e800cd43b0 2024-04-02T14:39:03.866 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.862+0000 7f4b56d70740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f4b4c1ad9a0 con 0x55e800ccb7d0 2024-04-02T14:39:03.866 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.862+0000 7f4b56d70740 10 client.0 _ll_register_callbacks cb 0x7f4b38009cd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:39:03.866 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.862+0000 7f4b52000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e800ccaeb0 0x55e800ccdf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:03.866 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.862+0000 7f4b50c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e800ccb7d0 0x55e800cce4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:03.866 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.862+0000 7f4b52000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e800ccaeb0 0x55e800ccdf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:49488/0 (socket says 172.21.15.90:49488) 2024-04-02T14:39:03.867 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.862+0000 7f4b52000640 1 -- 172.21.15.90:0/2831464132 learned_addr learned my addr 172.21.15.90:0/2831464132 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:39:03.867 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.862+0000 7f4b52000640 1 --2- 172.21.15.90:0/2831464132 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e800cd43b0 0x55e800ccea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:03.867 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.863+0000 7f4b50c00640 1 -- 172.21.15.90:0/2831464132 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e800cd43b0 msgr2=0x55e800ccea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:39:03.867 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.863+0000 7f4b50c00640 1 --2- 172.21.15.90:0/2831464132 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e800cd43b0 0x55e800ccea00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:03.867 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.863+0000 7f4b50c00640 1 -- 172.21.15.90:0/2831464132 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e800ccaeb0 msgr2=0x55e800ccdf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:39:03.867 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.863+0000 7f4b50c00640 1 --2- 172.21.15.90:0/2831464132 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e800ccaeb0 0x55e800ccdf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:03.867 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.863+0000 7f4b50c00640 1 -- 172.21.15.90:0/2831464132 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e800d49ca0 con 0x55e800ccb7d0 2024-04-02T14:39:03.869 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.863+0000 7f4b50c00640 1 --2- 172.21.15.90:0/2831464132 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e800ccb7d0 0x55e800cce4c0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f4b38023000 tx=0x7f4b38005f30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3b51d8840774e24d server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:39:03.869 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.863+0000 7f4b32a00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:39:03.869 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.863+0000 7f4b32a00640 1 -- 172.21.15.90:0/2831464132 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b38030040 con 0x55e800ccb7d0 2024-04-02T14:39:03.870 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.863+0000 7f4b32a00640 1 -- 172.21.15.90:0/2831464132 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f4b38010d70 con 0x55e800ccb7d0 2024-04-02T14:39:03.870 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.864+0000 7f4b56d70740 1 -- 172.21.15.90:0/2831464132 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4b38010bc0 con 0x55e800ccb7d0 2024-04-02T14:39:03.870 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.864+0000 7f4b32a00640 1 -- 172.21.15.90:0/2831464132 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f4b380223a0 con 0x55e800ccb7d0 2024-04-02T14:39:03.870 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.864+0000 7f4b56d70740 10 client.26548 fetch_fsmap learned FSMap version 27 2024-04-02T14:39:03.870 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.864+0000 7f4b56d70740 1 -- 172.21.15.90:0/2831464132 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55e800d4aef0 con 0x55e800ccb7d0 2024-04-02T14:39:03.870 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.864+0000 7f4b32a00640 1 -- 172.21.15.90:0/2831464132 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4b38010bc0 con 0x55e800ccb7d0 2024-04-02T14:39:03.870 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.864+0000 7f4b32a00640 1 -- 172.21.15.90:0/2831464132 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f4b3801a5d0 con 0x55e800ccb7d0 2024-04-02T14:39:03.870 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.865+0000 7f4b56d70740 10 client.26548 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:39:03.870 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.865+0000 7f4b56d70740 10 client.26548 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-02T14:39:03.870 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.865+0000 7f4b56d70740 20 client.26548 populate_metadata read hostname 'smithi090' 2024-04-02T14:39:03.870 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.865+0000 7f4b56d70740 1 --2- 172.21.15.90:0/2831464132 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55e800d4b520 0x55e800d4d940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:03.870 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.865+0000 7f4b56d70740 4 client.26548 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-02T14:39:03.870 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.865+0000 7f4b56d70740 1 -- 172.21.15.90:0/2831464132 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55e800c3d2b0 con 0x55e800d4b520 2024-04-02T14:39:03.870 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.865+0000 7f4b52000640 1 --2- 172.21.15.90:0/2831464132 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55e800d4b520 0x55e800d4d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:03.871 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.866+0000 7f4b52000640 1 --2- 172.21.15.90:0/2831464132 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55e800d4b520 0x55e800d4d940 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8b692a179cf28bd6 server_cookie=b9303b653658e6fd in_seq=0 out_seq=0 2024-04-02T14:39:03.871 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.866+0000 7f4b32a00640 10 client.26548 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:39:03.892 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.889+0000 7f4b32a00640 1 -- 172.21.15.90:0/2831464132 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55e800c3d2b0 con 0x55e800d4b520 2024-04-02T14:39:03.892 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.889+0000 7f4b32a00640 10 client.26548 handle_command_reply: tid=0 2024-04-02T14:39:03.892 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.889+0000 7f4b56d70740 1 client.26548 shutdown 2024-04-02T14:39:03.892 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.889+0000 7f4b56d70740 1 -- 172.21.15.90:0/2831464132 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e800ccb7d0 msgr2=0x55e800cce4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:03.892 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.889+0000 7f4b56d70740 1 --2- 172.21.15.90:0/2831464132 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e800ccb7d0 0x55e800cce4c0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f4b38023000 tx=0x7f4b38005f30 comp rx=0 tx=0).stop 2024-04-02T14:39:03.892 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.889+0000 7f4b56d70740 1 -- 172.21.15.90:0/2831464132 shutdown_connections 2024-04-02T14:39:03.892 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.889+0000 7f4b56d70740 1 --2- 172.21.15.90:0/2831464132 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55e800d4b520 0x55e800d4d940 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:03.892 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.889+0000 7f4b56d70740 1 --2- 172.21.15.90:0/2831464132 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e800cd43b0 0x55e800ccea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:03.893 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.889+0000 7f4b56d70740 1 --2- 172.21.15.90:0/2831464132 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e800ccb7d0 0x55e800cce4c0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:03.893 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.889+0000 7f4b56d70740 1 --2- 172.21.15.90:0/2831464132 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e800ccaeb0 0x55e800ccdf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:03.893 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.889+0000 7f4b32a00640 0 client.26548 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:39:03.893 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.889+0000 7f4b56d70740 1 -- 172.21.15.90:0/2831464132 >> 172.21.15.90:0/2831464132 conn(0x55e800cc55e0 msgr2=0x55e800cc3db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:39:03.893 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.889+0000 7f4b56d70740 1 -- 172.21.15.90:0/2831464132 shutdown_connections 2024-04-02T14:39:03.893 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.889+0000 7f4b56d70740 1 -- 172.21.15.90:0/2831464132 wait complete. 2024-04-02T14:39:03.893 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.890+0000 7f4b56d70740 20 client.26548 trim_cache size 0 max 16384 2024-04-02T14:39:03.893 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:{ 2024-04-02T14:39:03.893 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "message": "", 2024-04-02T14:39:03.893 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "return_code": 0 2024-04-02T14:39:03.893 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:} 2024-04-02T14:39:03.894 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.890+0000 7f4b54000640 1 -- 172.21.15.90:0/3743300397 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f4b340a3e40 msgr2=0x7f4b340a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:03.894 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.890+0000 7f4b54000640 1 --2- 172.21.15.90:0/3743300397 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f4b340a3e40 0x7f4b340a6300 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f4b400efdc0 tx=0x7f4b400f8060 comp rx=0 tx=0).stop 2024-04-02T14:39:03.894 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.890+0000 7f4b54000640 1 -- 172.21.15.90:0/3743300397 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4b4c103f40 msgr2=0x7f4b4c19a5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:03.895 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.890+0000 7f4b54000640 1 --2- 172.21.15.90:0/3743300397 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4b4c103f40 0x7f4b4c19a5b0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f4b40006010 tx=0x7f4b4002cb40 comp rx=0 tx=0).stop 2024-04-02T14:39:03.895 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.890+0000 7f4b54000640 1 -- 172.21.15.90:0/3743300397 shutdown_connections 2024-04-02T14:39:03.895 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.890+0000 7f4b54000640 1 --2- 172.21.15.90:0/3743300397 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f4b340a3e40 0x7f4b340a6300 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:03.895 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.890+0000 7f4b54000640 1 --2- 172.21.15.90:0/3743300397 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4b4c19ee70 0x7f4b4c19f320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:03.895 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.891+0000 7f4b54000640 1 --2- 172.21.15.90:0/3743300397 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4b4c109010 0x7f4b4c19aaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:03.895 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.891+0000 7f4b54000640 1 --2- 172.21.15.90:0/3743300397 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4b4c103f40 0x7f4b4c19a5b0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:03.895 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.891+0000 7f4b54000640 1 -- 172.21.15.90:0/3743300397 >> 172.21.15.90:0/3743300397 conn(0x7f4b4c0ff7c0 msgr2=0x7f4b4c101b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:39:03.895 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.891+0000 7f4b54000640 1 -- 172.21.15.90:0/3743300397 shutdown_connections 2024-04-02T14:39:03.895 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:03.891+0000 7f4b54000640 1 -- 172.21.15.90:0/3743300397 wait complete. 2024-04-02T14:39:04.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.075+0000 7f0215a00640 1 -- 172.21.15.90:0/90867240 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0210073eb0 msgr2=0x7f0210074290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:04.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.075+0000 7f0215a00640 1 --2- 172.21.15.90:0/90867240 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0210073eb0 0x7f0210074290 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f0200009900 tx=0x7f020002bcd0 comp rx=0 tx=0).stop 2024-04-02T14:39:04.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.075+0000 7f0215a00640 1 -- 172.21.15.90:0/90867240 shutdown_connections 2024-04-02T14:39:04.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.075+0000 7f0215a00640 1 --2- 172.21.15.90:0/90867240 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f021010e360 0x7f0210110750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:04.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.076+0000 7f0215a00640 1 --2- 172.21.15.90:0/90867240 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f02100747d0 0x7f021010de20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:04.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.076+0000 7f0215a00640 1 --2- 172.21.15.90:0/90867240 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0210073eb0 0x7f0210074290 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:04.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.076+0000 7f0215a00640 1 -- 172.21.15.90:0/90867240 >> 172.21.15.90:0/90867240 conn(0x7f02100fd530 msgr2=0x7f02100ff950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:39:04.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.076+0000 7f0215a00640 1 -- 172.21.15.90:0/90867240 shutdown_connections 2024-04-02T14:39:04.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.076+0000 7f0215a00640 1 -- 172.21.15.90:0/90867240 wait complete. 2024-04-02T14:39:04.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.077+0000 7f0215a00640 1 Processor -- start 2024-04-02T14:39:04.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.077+0000 7f0215a00640 1 -- start start 2024-04-02T14:39:04.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.077+0000 7f0215a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0210073eb0 0x7f021019a5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:04.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.077+0000 7f0215a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f02100747d0 0x7f021019ab10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:04.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.077+0000 7f0215a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f021010e360 0x7f021019ee90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:04.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.077+0000 7f0215a00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f02101140b0 con 0x7f02100747d0 2024-04-02T14:39:04.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.077+0000 7f0215a00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f0210113f30 con 0x7f0210073eb0 2024-04-02T14:39:04.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.077+0000 7f0215a00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f0210114230 con 0x7f021010e360 2024-04-02T14:39:04.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.078+0000 7f020f400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f021010e360 0x7f021019ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:04.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.078+0000 7f020e000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f02100747d0 0x7f021019ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:04.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.078+0000 7f020ea00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0210073eb0 0x7f021019a5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:04.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.078+0000 7f020f400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f021010e360 0x7f021019ee90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:54720/0 (socket says 172.21.15.90:54720) 2024-04-02T14:39:04.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.078+0000 7f020f400640 1 -- 172.21.15.90:0/2865396505 learned_addr learned my addr 172.21.15.90:0/2865396505 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:39:04.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.078+0000 7f020f400640 1 -- 172.21.15.90:0/2865396505 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0210073eb0 msgr2=0x7f021019a5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:04.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.078+0000 7f020f400640 1 --2- 172.21.15.90:0/2865396505 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0210073eb0 0x7f021019a5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:04.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.078+0000 7f020f400640 1 -- 172.21.15.90:0/2865396505 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f02100747d0 msgr2=0x7f021019ab10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:04.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.078+0000 7f020f400640 1 --2- 172.21.15.90:0/2865396505 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f02100747d0 0x7f021019ab10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:04.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.078+0000 7f020f400640 1 -- 172.21.15.90:0/2865396505 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f021019f5c0 con 0x7f021010e360 2024-04-02T14:39:04.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.078+0000 7f020ea00640 1 --2- 172.21.15.90:0/2865396505 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0210073eb0 0x7f021019a5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:39:04.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.078+0000 7f020f400640 1 --2- 172.21.15.90:0/2865396505 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f021010e360 0x7f021019ee90 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f020800eb10 tx=0x7f020800c2d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:39:04.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.079+0000 7f0207400640 1 -- 172.21.15.90:0/2865396505 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0208004b20 con 0x7f021010e360 2024-04-02T14:39:04.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.079+0000 7f0207400640 1 -- 172.21.15.90:0/2865396505 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0208004cc0 con 0x7f021010e360 2024-04-02T14:39:04.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.079+0000 7f0207400640 1 -- 172.21.15.90:0/2865396505 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0208010760 con 0x7f021010e360 2024-04-02T14:39:04.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.079+0000 7f0215a00640 1 -- 172.21.15.90:0/2865396505 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f021019f8b0 con 0x7f021010e360 2024-04-02T14:39:04.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.079+0000 7f0215a00640 1 -- 172.21.15.90:0/2865396505 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02101a7140 con 0x7f021010e360 2024-04-02T14:39:04.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.083+0000 7f0218835740 1 Processor -- start 2024-04-02T14:39:04.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.083+0000 7f0207400640 1 -- 172.21.15.90:0/2865396505 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f0208005020 con 0x7f021010e360 2024-04-02T14:39:04.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.083+0000 7f0218835740 1 -- start start 2024-04-02T14:39:04.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.084+0000 7f0218835740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55da83d5f9f0 0x55da83d5e740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:04.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.084+0000 7f0218835740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55da83d5ed10 0x55da83d5f190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:04.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.084+0000 7f0218835740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55da83d68080 0x55da83d6a470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:04.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.084+0000 7f0218835740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55da83c9f400 con 0x55da83d5f9f0 2024-04-02T14:39:04.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.084+0000 7f0218835740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55da83c83f10 con 0x55da83d5ed10 2024-04-02T14:39:04.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.084+0000 7f0218835740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55da83cc36f0 con 0x55da83d68080 2024-04-02T14:39:04.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.084+0000 7f020ea00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55da83d5ed10 0x55da83d5f190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:04.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.084+0000 7f020ea00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55da83d5ed10 0x55da83d5f190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:41530/0 (socket says 172.21.15.90:41530) 2024-04-02T14:39:04.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.084+0000 7f020ea00640 1 -- 172.21.15.90:0/1926922648 learned_addr learned my addr 172.21.15.90:0/1926922648 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:39:04.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.084+0000 7f020e000640 1 --2- 172.21.15.90:0/1926922648 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55da83d5f9f0 0x55da83d5e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:04.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.084+0000 7f020ea00640 1 -- 172.21.15.90:0/1926922648 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55da83d68080 msgr2=0x55da83d6a470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:39:04.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.084+0000 7f020e000640 1 --2- 172.21.15.90:0/1926922648 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55da83d68080 0x55da83d6a470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:04.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.084+0000 7f020ea00640 1 --2- 172.21.15.90:0/1926922648 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55da83d68080 0x55da83d6a470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:04.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.084+0000 7f020ea00640 1 -- 172.21.15.90:0/1926922648 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55da83d5f9f0 msgr2=0x55da83d5e740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:39:04.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.084+0000 7f020ea00640 1 --2- 172.21.15.90:0/1926922648 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55da83d5f9f0 0x55da83d5e740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:04.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.084+0000 7f020ea00640 1 -- 172.21.15.90:0/1926922648 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55da839d73f0 con 0x55da83d5ed10 2024-04-02T14:39:04.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.084+0000 7f020e000640 1 --2- 172.21.15.90:0/1926922648 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55da83d68080 0x55da83d6a470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:39:04.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.085+0000 7f020ea00640 1 --2- 172.21.15.90:0/1926922648 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55da83d5ed10 0x55da83d5f190 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f0200009230 tx=0x7f0200009560 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ca19c1f219b445bf server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:39:04.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.085+0000 7f0207400640 1 --2- 172.21.15.90:0/2865396505 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f01f40a3da0 0x7f01f40a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:04.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.085+0000 7f01fbe00640 1 -- 172.21.15.90:0/1926922648 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0200043070 con 0x55da83d5ed10 2024-04-02T14:39:04.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.085+0000 7f01fbe00640 1 -- 172.21.15.90:0/1926922648 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f020002c6f0 con 0x55da83d5ed10 2024-04-02T14:39:04.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.085+0000 7f01fbe00640 1 -- 172.21.15.90:0/1926922648 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0200004ee0 con 0x55da83d5ed10 2024-04-02T14:39:04.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.085+0000 7f0207400640 1 -- 172.21.15.90:0/2865396505 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f02080d0d00 con 0x7f021010e360 2024-04-02T14:39:04.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.086+0000 7f020e000640 1 --2- 172.21.15.90:0/2865396505 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f01f40a3da0 0x7f01f40a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:04.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.086+0000 7f0218835740 1 -- 172.21.15.90:0/1926922648 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55da83d5ed10 msgr2=0x55da83d5f190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:04.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.086+0000 7f0218835740 1 --2- 172.21.15.90:0/1926922648 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55da83d5ed10 0x55da83d5f190 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f0200009230 tx=0x7f0200009560 comp rx=0 tx=0).stop 2024-04-02T14:39:04.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.086+0000 7f0218835740 1 -- 172.21.15.90:0/1926922648 shutdown_connections 2024-04-02T14:39:04.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.086+0000 7f0218835740 1 --2- 172.21.15.90:0/1926922648 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55da83d68080 0x55da83d6a470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:04.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.086+0000 7f0218835740 1 --2- 172.21.15.90:0/1926922648 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55da83d5ed10 0x55da83d5f190 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:04.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.086+0000 7f0218835740 1 --2- 172.21.15.90:0/1926922648 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55da83d5f9f0 0x55da83d5e740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:04.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.086+0000 7f0218835740 1 -- 172.21.15.90:0/1926922648 >> 172.21.15.90:0/1926922648 conn(0x55da83d25a50 msgr2=0x55da83d59760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:39:04.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.087+0000 7f0218835740 1 -- 172.21.15.90:0/1926922648 shutdown_connections 2024-04-02T14:39:04.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.087+0000 7f0218835740 1 -- 172.21.15.90:0/1926922648 wait complete. 2024-04-02T14:39:04.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.087+0000 7f020e000640 1 --2- 172.21.15.90:0/2865396505 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f01f40a3da0 0x7f01f40a6260 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f01fc009710 tx=0x7f01fc005f90 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:39:04.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.087+0000 7f0218835740 1 -- start start 2024-04-02T14:39:04.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.087+0000 7f0218835740 1 Processor -- start 2024-04-02T14:39:04.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.088+0000 7f0218835740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55da83d5ed10 0x55da83d65080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:04.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.088+0000 7f0218835740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55da83d5f9f0 0x55da83d655c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:04.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.088+0000 7f0218835740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55da83d68080 0x55da83d65b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:04.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.088+0000 7f0218835740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55da83cbf6a0 con 0x55da83d68080 2024-04-02T14:39:04.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.088+0000 7f0218835740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55da83c6b980 con 0x55da83d5f9f0 2024-04-02T14:39:04.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.088+0000 7f0218835740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55da83c7ecb0 con 0x55da83d5ed10 2024-04-02T14:39:04.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.088+0000 7f0218835740 10 client.0 _ll_register_callbacks cb 0x55da83c7e6a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:39:04.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.088+0000 7f020e000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55da83d5f9f0 0x55da83d655c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:04.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.088+0000 7f020ea00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55da83d5ed10 0x55da83d65080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:04.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.088+0000 7f020e000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55da83d5f9f0 0x55da83d655c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:41542/0 (socket says 172.21.15.90:41542) 2024-04-02T14:39:04.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.088+0000 7f020e000640 1 -- 172.21.15.90:0/2193573310 learned_addr learned my addr 172.21.15.90:0/2193573310 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:39:04.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.088+0000 7f020f400640 1 --2- 172.21.15.90:0/2193573310 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55da83d68080 0x55da83d65b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:04.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.089+0000 7f020e000640 1 -- 172.21.15.90:0/2193573310 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55da83d5ed10 msgr2=0x55da83d65080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:39:04.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.089+0000 7f020e000640 1 --2- 172.21.15.90:0/2193573310 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55da83d5ed10 0x55da83d65080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:04.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.089+0000 7f020e000640 1 -- 172.21.15.90:0/2193573310 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55da83d68080 msgr2=0x55da83d65b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:39:04.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.089+0000 7f020e000640 1 --2- 172.21.15.90:0/2193573310 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55da83d68080 0x55da83d65b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:04.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.089+0000 7f020e000640 1 -- 172.21.15.90:0/2193573310 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55da83aa17d0 con 0x55da83d5f9f0 2024-04-02T14:39:04.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.089+0000 7f020e000640 1 --2- 172.21.15.90:0/2193573310 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55da83d5f9f0 0x55da83d655c0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f01fc009650 tx=0x7f01fc004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=274756295bb68e70 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:39:04.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.089+0000 7f01f8c00640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:39:04.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.089+0000 7f01f8c00640 1 -- 172.21.15.90:0/2193573310 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01fc02e060 con 0x55da83d5f9f0 2024-04-02T14:39:04.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.089+0000 7f01f8c00640 1 -- 172.21.15.90:0/2193573310 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f01fc019030 con 0x55da83d5f9f0 2024-04-02T14:39:04.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.090+0000 7f0218835740 1 -- 172.21.15.90:0/2193573310 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f020002c6f0 con 0x55da83d5f9f0 2024-04-02T14:39:04.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.090+0000 7f01f8c00640 1 -- 172.21.15.90:0/2193573310 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f01fc009de0 con 0x55da83d5f9f0 2024-04-02T14:39:04.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.090+0000 7f0218835740 10 client.26675 fetch_fsmap learned FSMap version 27 2024-04-02T14:39:04.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.090+0000 7f0218835740 1 -- 172.21.15.90:0/2193573310 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55da83d76ea0 con 0x55da83d5f9f0 2024-04-02T14:39:04.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.090+0000 7f01f8c00640 1 -- 172.21.15.90:0/2193573310 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f020002c6f0 con 0x55da83d5f9f0 2024-04-02T14:39:04.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.090+0000 7f01f8c00640 1 -- 172.21.15.90:0/2193573310 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f01fc01d560 con 0x55da83d5f9f0 2024-04-02T14:39:04.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.090+0000 7f0218835740 10 client.26675 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:39:04.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.090+0000 7f0218835740 10 client.26675 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.c 2024-04-02T14:39:04.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.090+0000 7f0218835740 20 client.26675 populate_metadata read hostname 'smithi090' 2024-04-02T14:39:04.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.090+0000 7f0218835740 1 --2- 172.21.15.90:0/2193573310 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x55da83d78400 0x55da83d7a840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:04.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.090+0000 7f0218835740 4 client.26675 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:39:04.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.090+0000 7f0218835740 1 -- 172.21.15.90:0/2193573310 --> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55da83c83040 con 0x55da83d78400 2024-04-02T14:39:04.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.091+0000 7f020ea00640 1 --2- 172.21.15.90:0/2193573310 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x55da83d78400 0x55da83d7a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:04.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.091+0000 7f01f8c00640 10 client.26675 ms_handle_connect on v2:172.21.15.177:6836/4223414858 2024-04-02T14:39:04.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.091+0000 7f020ea00640 1 --2- 172.21.15.90:0/2193573310 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x55da83d78400 0x55da83d7a840 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=cf5654049bf5fbf7 server_cookie=f873c7119059508b in_seq=0 out_seq=0 2024-04-02T14:39:04.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.092+0000 7f01f8c00640 1 -- 172.21.15.90:0/2193573310 <== mds.4 v2:172.21.15.177:6836/4223414858 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55da83c83040 con 0x55da83d78400 2024-04-02T14:39:04.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.092+0000 7f01f8c00640 10 client.26675 handle_command_reply: tid=0 2024-04-02T14:39:04.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.092+0000 7f0218835740 1 client.26675 shutdown 2024-04-02T14:39:04.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.092+0000 7f0218835740 1 -- 172.21.15.90:0/2193573310 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55da83d5f9f0 msgr2=0x55da83d655c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:04.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.092+0000 7f0218835740 1 --2- 172.21.15.90:0/2193573310 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55da83d5f9f0 0x55da83d655c0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f01fc009650 tx=0x7f01fc004730 comp rx=0 tx=0).stop 2024-04-02T14:39:04.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.092+0000 7f0218835740 1 -- 172.21.15.90:0/2193573310 shutdown_connections 2024-04-02T14:39:04.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.092+0000 7f0218835740 1 --2- 172.21.15.90:0/2193573310 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x55da83d78400 0x55da83d7a840 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:04.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.092+0000 7f0218835740 1 --2- 172.21.15.90:0/2193573310 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55da83d68080 0x55da83d65b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:04.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.092+0000 7f0218835740 1 --2- 172.21.15.90:0/2193573310 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55da83d5f9f0 0x55da83d655c0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:04.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.092+0000 7f0218835740 1 --2- 172.21.15.90:0/2193573310 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55da83d5ed10 0x55da83d65080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:04.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.092+0000 7f01f8c00640 0 client.26675 ms_handle_reset on v2:172.21.15.177:6836/4223414858 2024-04-02T14:39:04.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.093+0000 7f0218835740 1 -- 172.21.15.90:0/2193573310 >> 172.21.15.90:0/2193573310 conn(0x55da83d25a50 msgr2=0x55da83d5b4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:39:04.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.093+0000 7f0218835740 1 -- 172.21.15.90:0/2193573310 shutdown_connections 2024-04-02T14:39:04.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.093+0000 7f0218835740 1 -- 172.21.15.90:0/2193573310 wait complete. 2024-04-02T14:39:04.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.093+0000 7f0218835740 20 client.26675 trim_cache size 0 max 16384 2024-04-02T14:39:04.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.112+0000 7f0218835740 1 Processor -- start 2024-04-02T14:39:04.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.112+0000 7f0218835740 1 -- start start 2024-04-02T14:39:04.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.112+0000 7f0218835740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55da83d5ed10 0x55da83d677b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:04.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.112+0000 7f0218835740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55da83d5f9f0 0x55da83d73a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:04.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.112+0000 7f0218835740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55da83d68080 0x55da83d5b230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:04.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.112+0000 7f0218835740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f02101af9a0 con 0x55da83d5ed10 2024-04-02T14:39:04.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.112+0000 7f0218835740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f02101b14e0 con 0x55da83d68080 2024-04-02T14:39:04.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.112+0000 7f0218835740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f02101b0070 con 0x55da83d5f9f0 2024-04-02T14:39:04.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.113+0000 7f020ea00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55da83d5ed10 0x55da83d677b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:04.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.113+0000 7f020ea00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55da83d5ed10 0x55da83d677b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:49528/0 (socket says 172.21.15.90:49528) 2024-04-02T14:39:04.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.113+0000 7f020ea00640 1 -- 172.21.15.90:0/846451169 learned_addr learned my addr 172.21.15.90:0/846451169 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:39:04.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.113+0000 7f020f400640 1 --2- 172.21.15.90:0/846451169 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55da83d68080 0x55da83d5b230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:04.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.113+0000 7f020e000640 1 --2- 172.21.15.90:0/846451169 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55da83d5f9f0 0x55da83d73a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:04.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.113+0000 7f020ea00640 1 -- 172.21.15.90:0/846451169 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55da83d5f9f0 msgr2=0x55da83d73a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:39:04.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.113+0000 7f020ea00640 1 --2- 172.21.15.90:0/846451169 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55da83d5f9f0 0x55da83d73a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:04.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.113+0000 7f020ea00640 1 -- 172.21.15.90:0/846451169 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55da83d68080 msgr2=0x55da83d5b230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:39:04.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.113+0000 7f020ea00640 1 --2- 172.21.15.90:0/846451169 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55da83d68080 0x55da83d5b230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:04.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.113+0000 7f020ea00640 1 -- 172.21.15.90:0/846451169 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55da83d5dd40 con 0x55da83d5ed10 2024-04-02T14:39:04.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.113+0000 7f020ea00640 1 --2- 172.21.15.90:0/846451169 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55da83d5ed10 0x55da83d677b0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f020002cc10 tx=0x7f0200041040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ea0bb3c3a468678 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:39:04.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.114+0000 7f01fb400640 1 -- 172.21.15.90:0/846451169 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0200043040 con 0x55da83d5ed10 2024-04-02T14:39:04.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.114+0000 7f01fb400640 1 -- 172.21.15.90:0/846451169 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0200004c50 con 0x55da83d5ed10 2024-04-02T14:39:04.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.114+0000 7f01fb400640 1 -- 172.21.15.90:0/846451169 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0200005300 con 0x55da83d5ed10 2024-04-02T14:39:04.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.114+0000 7f0218835740 1 -- 172.21.15.90:0/846451169 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55da83d5ed10 msgr2=0x55da83d677b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:04.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.114+0000 7f0218835740 1 --2- 172.21.15.90:0/846451169 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55da83d5ed10 0x55da83d677b0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f020002cc10 tx=0x7f0200041040 comp rx=0 tx=0).stop 2024-04-02T14:39:04.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.115+0000 7f0218835740 1 -- 172.21.15.90:0/846451169 shutdown_connections 2024-04-02T14:39:04.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.115+0000 7f0218835740 1 --2- 172.21.15.90:0/846451169 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55da83d68080 0x55da83d5b230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:04.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.115+0000 7f0218835740 1 --2- 172.21.15.90:0/846451169 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55da83d5f9f0 0x55da83d73a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:04.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.115+0000 7f0218835740 1 --2- 172.21.15.90:0/846451169 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55da83d5ed10 0x55da83d677b0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:04.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.115+0000 7f0218835740 1 -- 172.21.15.90:0/846451169 >> 172.21.15.90:0/846451169 conn(0x55da83d25a50 msgr2=0x55da83d5ab20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:39:04.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.115+0000 7f0218835740 1 -- 172.21.15.90:0/846451169 shutdown_connections 2024-04-02T14:39:04.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.115+0000 7f0218835740 1 -- 172.21.15.90:0/846451169 wait complete. 2024-04-02T14:39:04.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.115+0000 7f0218835740 1 -- start start 2024-04-02T14:39:04.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.115+0000 7f0218835740 1 Processor -- start 2024-04-02T14:39:04.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.116+0000 7f0218835740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55da83d5ed10 0x55da83d703f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:04.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.116+0000 7f0218835740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55da83d5f9f0 0x55da83d625d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:04.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.116+0000 7f0218835740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55da83d68080 0x55da83d62b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:04.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.116+0000 7f0218835740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f02101b0e10 con 0x55da83d68080 2024-04-02T14:39:04.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.116+0000 7f0218835740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f02101b0740 con 0x55da83d5ed10 2024-04-02T14:39:04.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.116+0000 7f0218835740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f02101ad980 con 0x55da83d5f9f0 2024-04-02T14:39:04.120 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.116+0000 7f0218835740 10 client.0 _ll_register_callbacks cb 0x7f01fc01e8a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:39:04.120 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.116+0000 7f020f400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55da83d68080 0x55da83d62b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:04.120 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.116+0000 7f020ea00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55da83d5ed10 0x55da83d703f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:04.120 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.116+0000 7f020f400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55da83d68080 0x55da83d62b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:49538/0 (socket says 172.21.15.90:49538) 2024-04-02T14:39:04.120 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.116+0000 7f020ea00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55da83d5ed10 0x55da83d703f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:41554/0 (socket says 172.21.15.90:41554) 2024-04-02T14:39:04.120 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.116+0000 7f020f400640 1 -- 172.21.15.90:0/970100693 learned_addr learned my addr 172.21.15.90:0/970100693 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:39:04.120 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.116+0000 7f020e000640 1 --2- 172.21.15.90:0/970100693 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55da83d5f9f0 0x55da83d625d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:04.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.117+0000 7f020ea00640 1 -- 172.21.15.90:0/970100693 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55da83d5f9f0 msgr2=0x55da83d625d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:39:04.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.117+0000 7f020ea00640 1 --2- 172.21.15.90:0/970100693 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55da83d5f9f0 0x55da83d625d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:04.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.117+0000 7f020ea00640 1 -- 172.21.15.90:0/970100693 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55da83d68080 msgr2=0x55da83d62b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:39:04.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.117+0000 7f020ea00640 1 --2- 172.21.15.90:0/970100693 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55da83d68080 0x55da83d62b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:04.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.117+0000 7f020ea00640 1 -- 172.21.15.90:0/970100693 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55da83d68de0 con 0x55da83d5ed10 2024-04-02T14:39:04.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.117+0000 7f020ea00640 1 --2- 172.21.15.90:0/970100693 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55da83d5ed10 0x55da83d703f0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f0200006010 tx=0x7f020002c670 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4547588388ec2e2c server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:39:04.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.117+0000 7f01f8c00640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:39:04.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.117+0000 7f01f8c00640 1 -- 172.21.15.90:0/970100693 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0200054020 con 0x55da83d5ed10 2024-04-02T14:39:04.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.117+0000 7f01f8c00640 1 -- 172.21.15.90:0/970100693 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f0200004300 con 0x55da83d5ed10 2024-04-02T14:39:04.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.117+0000 7f0218835740 1 -- 172.21.15.90:0/970100693 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0200004c50 con 0x55da83d5ed10 2024-04-02T14:39:04.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.117+0000 7f01f8c00640 1 -- 172.21.15.90:0/970100693 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f0200043040 con 0x55da83d5ed10 2024-04-02T14:39:04.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.117+0000 7f0218835740 10 client.26681 fetch_fsmap learned FSMap version 27 2024-04-02T14:39:04.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.117+0000 7f0218835740 1 -- 172.21.15.90:0/970100693 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55da83dded60 con 0x55da83d5ed10 2024-04-02T14:39:04.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.118+0000 7f01f8c00640 1 -- 172.21.15.90:0/970100693 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0200004c50 con 0x55da83d5ed10 2024-04-02T14:39:04.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.118+0000 7f01f8c00640 1 -- 172.21.15.90:0/970100693 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f0200003a30 con 0x55da83d5ed10 2024-04-02T14:39:04.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.118+0000 7f0218835740 10 client.26681 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:39:04.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.118+0000 7f0218835740 10 client.26681 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.c 2024-04-02T14:39:04.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.118+0000 7f0218835740 20 client.26681 populate_metadata read hostname 'smithi090' 2024-04-02T14:39:04.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.118+0000 7f0218835740 1 --2- 172.21.15.90:0/970100693 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x55da83ddefe0 0x55da83de13d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:04.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.118+0000 7f0218835740 4 client.26681 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-02T14:39:04.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.118+0000 7f0218835740 1 -- 172.21.15.90:0/970100693 --> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55da83cd0c30 con 0x55da83ddefe0 2024-04-02T14:39:04.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.118+0000 7f020e000640 1 --2- 172.21.15.90:0/970100693 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x55da83ddefe0 0x55da83de13d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:04.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.119+0000 7f020e000640 1 --2- 172.21.15.90:0/970100693 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x55da83ddefe0 0x55da83de13d0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=2109d0cf92b12fab server_cookie=911390d472c6fb48 in_seq=0 out_seq=0 2024-04-02T14:39:04.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.119+0000 7f01f8c00640 10 client.26681 ms_handle_connect on v2:172.21.15.177:6836/4223414858 2024-04-02T14:39:04.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.119+0000 7f01f8c00640 1 -- 172.21.15.90:0/970100693 <== mds.4 v2:172.21.15.177:6836/4223414858 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55da83cd0c30 con 0x55da83ddefe0 2024-04-02T14:39:04.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.119+0000 7f01f8c00640 10 client.26681 handle_command_reply: tid=0 2024-04-02T14:39:04.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.119+0000 7f0218835740 1 client.26681 shutdown 2024-04-02T14:39:04.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.119+0000 7f0218835740 1 -- 172.21.15.90:0/970100693 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55da83d5ed10 msgr2=0x55da83d703f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:04.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.119+0000 7f0218835740 1 --2- 172.21.15.90:0/970100693 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55da83d5ed10 0x55da83d703f0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f0200006010 tx=0x7f020002c670 comp rx=0 tx=0).stop 2024-04-02T14:39:04.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.119+0000 7f0218835740 1 -- 172.21.15.90:0/970100693 shutdown_connections 2024-04-02T14:39:04.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.119+0000 7f0218835740 1 --2- 172.21.15.90:0/970100693 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x55da83ddefe0 0x55da83de13d0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:04.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.119+0000 7f0218835740 1 --2- 172.21.15.90:0/970100693 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55da83d68080 0x55da83d62b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:04.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.119+0000 7f0218835740 1 --2- 172.21.15.90:0/970100693 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55da83d5f9f0 0x55da83d625d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:04.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.119+0000 7f0218835740 1 --2- 172.21.15.90:0/970100693 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55da83d5ed10 0x55da83d703f0 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:04.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.119+0000 7f01f8c00640 0 client.26681 ms_handle_reset on v2:172.21.15.177:6836/4223414858 2024-04-02T14:39:04.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.119+0000 7f0218835740 1 -- 172.21.15.90:0/970100693 >> 172.21.15.90:0/970100693 conn(0x55da83d25a50 msgr2=0x55da83d588a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:39:04.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.119+0000 7f0218835740 1 -- 172.21.15.90:0/970100693 shutdown_connections 2024-04-02T14:39:04.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.119+0000 7f0218835740 1 -- 172.21.15.90:0/970100693 wait complete. 2024-04-02T14:39:04.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.120+0000 7f0218835740 20 client.26681 trim_cache size 0 max 16384 2024-04-02T14:39:04.126 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-02T14:39:04.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.120+0000 7f0215a00640 1 -- 172.21.15.90:0/2865396505 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f01f40a3da0 msgr2=0x7f01f40a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:04.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.120+0000 7f0215a00640 1 --2- 172.21.15.90:0/2865396505 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f01f40a3da0 0x7f01f40a6260 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f01fc009710 tx=0x7f01fc005f90 comp rx=0 tx=0).stop 2024-04-02T14:39:04.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.120+0000 7f0215a00640 1 -- 172.21.15.90:0/2865396505 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f021010e360 msgr2=0x7f021019ee90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:04.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.120+0000 7f0215a00640 1 --2- 172.21.15.90:0/2865396505 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f021010e360 0x7f021019ee90 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f020800eb10 tx=0x7f020800c2d0 comp rx=0 tx=0).stop 2024-04-02T14:39:04.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.120+0000 7f0215a00640 1 -- 172.21.15.90:0/2865396505 shutdown_connections 2024-04-02T14:39:04.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.121+0000 7f0215a00640 1 --2- 172.21.15.90:0/2865396505 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f01f40a3da0 0x7f01f40a6260 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:04.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.121+0000 7f0215a00640 1 --2- 172.21.15.90:0/2865396505 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f021010e360 0x7f021019ee90 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:04.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.121+0000 7f0215a00640 1 --2- 172.21.15.90:0/2865396505 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f02100747d0 0x7f021019ab10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:04.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.121+0000 7f0215a00640 1 --2- 172.21.15.90:0/2865396505 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0210073eb0 0x7f021019a5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:04.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.121+0000 7f0215a00640 1 -- 172.21.15.90:0/2865396505 >> 172.21.15.90:0/2865396505 conn(0x7f02100fd530 msgr2=0x7f02100fe200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:39:04.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.121+0000 7f0215a00640 1 -- 172.21.15.90:0/2865396505 shutdown_connections 2024-04-02T14:39:04.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.121+0000 7f0215a00640 1 -- 172.21.15.90:0/2865396505 wait complete. 2024-04-02T14:39:04.138 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:39:03 smithi090 ceph-mon[31674]: pgmap v128: 97 pgs: 97 active+clean; 2.3 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 11 MiB/s wr, 711 op/s 2024-04-02T14:39:04.138 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:39:03 smithi090 ceph-mon[31674]: from='client.? 172.21.15.90:0/3500657956' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-02T14:39:04.142 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-04-02T14:39:04.360 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:39:03 smithi092 ceph-mon[36497]: pgmap v128: 97 pgs: 97 active+clean; 2.3 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 11 MiB/s wr, 711 op/s 2024-04-02T14:39:04.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:39:03 smithi092 ceph-mon[36497]: from='client.? 172.21.15.90:0/3500657956' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-02T14:39:04.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.400+0000 7f6798600640 1 -- 172.21.15.90:0/891775396 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f679006a070 msgr2=0x7f679006a4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:04.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.400+0000 7f6798600640 1 --2- 172.21.15.90:0/891775396 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f679006a070 0x7f679006a4b0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f677c009900 tx=0x7f677c02bc50 comp rx=0 tx=0).stop 2024-04-02T14:39:04.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.401+0000 7f6798600640 1 -- 172.21.15.90:0/891775396 shutdown_connections 2024-04-02T14:39:04.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.401+0000 7f6798600640 1 --2- 172.21.15.90:0/891775396 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f679006a9f0 0x7f679010e970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:04.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.401+0000 7f6798600640 1 --2- 172.21.15.90:0/891775396 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f679006a070 0x7f679006a4b0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:04.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.401+0000 7f6798600640 1 --2- 172.21.15.90:0/891775396 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6790107ab0 0x7f6790107e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:04.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.401+0000 7f6798600640 1 -- 172.21.15.90:0/891775396 >> 172.21.15.90:0/891775396 conn(0x7f67900779e0 msgr2=0x7f6790077df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:39:04.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.401+0000 7f6798600640 1 -- 172.21.15.90:0/891775396 shutdown_connections 2024-04-02T14:39:04.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.401+0000 7f6798600640 1 -- 172.21.15.90:0/891775396 wait complete. 2024-04-02T14:39:04.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.402+0000 7f6798600640 1 Processor -- start 2024-04-02T14:39:04.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.402+0000 7f6798600640 1 -- start start 2024-04-02T14:39:04.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.403+0000 7f6798600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f679006a070 0x7f679019a560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:04.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.403+0000 7f6798600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f679006a9f0 0x7f679019aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:04.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.403+0000 7f6798600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6790107ab0 0x7f679019ee20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:04.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.403+0000 7f6798600640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f6790111ed0 con 0x7f6790107ab0 2024-04-02T14:39:04.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.403+0000 7f6798600640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f6790111d50 con 0x7f679006a9f0 2024-04-02T14:39:04.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.403+0000 7f6798600640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f6790112050 con 0x7f679006a070 2024-04-02T14:39:04.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.403+0000 7f6795200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f679006a9f0 0x7f679019aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:04.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.403+0000 7f6795200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f679006a9f0 0x7f679019aaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:41566/0 (socket says 172.21.15.90:41566) 2024-04-02T14:39:04.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.403+0000 7f6795200640 1 -- 172.21.15.90:0/1807555968 learned_addr learned my addr 172.21.15.90:0/1807555968 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:39:04.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.403+0000 7f6795c00640 1 --2- 172.21.15.90:0/1807555968 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f679006a070 0x7f679019a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:04.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.403+0000 7f6796600640 1 --2- 172.21.15.90:0/1807555968 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6790107ab0 0x7f679019ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:04.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.404+0000 7f6795c00640 1 -- 172.21.15.90:0/1807555968 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f679006a9f0 msgr2=0x7f679019aaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:04.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.404+0000 7f6795c00640 1 --2- 172.21.15.90:0/1807555968 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f679006a9f0 0x7f679019aaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:04.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.404+0000 7f6795c00640 1 -- 172.21.15.90:0/1807555968 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6790107ab0 msgr2=0x7f679019ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:04.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.404+0000 7f6795c00640 1 --2- 172.21.15.90:0/1807555968 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6790107ab0 0x7f679019ee20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:04.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.404+0000 7f6795c00640 1 -- 172.21.15.90:0/1807555968 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f679019f550 con 0x7f679006a070 2024-04-02T14:39:04.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.404+0000 7f6795200640 1 --2- 172.21.15.90:0/1807555968 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f679006a9f0 0x7f679019aaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:39:04.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.404+0000 7f6795c00640 1 --2- 172.21.15.90:0/1807555968 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f679006a070 0x7f679019a560 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f678400e9f0 tx=0x7f678400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:39:04.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.404+0000 7f678e000640 1 -- 172.21.15.90:0/1807555968 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f678400ccc0 con 0x7f679006a070 2024-04-02T14:39:04.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.404+0000 7f678e000640 1 -- 172.21.15.90:0/1807555968 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6784010910 con 0x7f679006a070 2024-04-02T14:39:04.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.404+0000 7f6798600640 1 -- 172.21.15.90:0/1807555968 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f679019f840 con 0x7f679006a070 2024-04-02T14:39:04.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.404+0000 7f678e000640 1 -- 172.21.15.90:0/1807555968 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6784018960 con 0x7f679006a070 2024-04-02T14:39:04.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.404+0000 7f6798600640 1 -- 172.21.15.90:0/1807555968 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67901a70d0 con 0x7f679006a070 2024-04-02T14:39:04.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.408+0000 7f678e000640 1 -- 172.21.15.90:0/1807555968 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f6784010ac0 con 0x7f679006a070 2024-04-02T14:39:04.410 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.408+0000 7f679b377740 1 Processor -- start 2024-04-02T14:39:04.410 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.408+0000 7f679b377740 1 -- start start 2024-04-02T14:39:04.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.408+0000 7f679b377740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563261b759f0 0x563261b74740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:04.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.408+0000 7f679b377740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563261b74d10 0x563261b75190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:04.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.408+0000 7f679b377740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563261b7e080 0x563261b80470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:04.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.409+0000 7f679b377740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x563261ab5400 con 0x563261b759f0 2024-04-02T14:39:04.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.409+0000 7f679b377740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x563261a99f10 con 0x563261b7e080 2024-04-02T14:39:04.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.409+0000 7f679b377740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x563261ad96f0 con 0x563261b74d10 2024-04-02T14:39:04.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.409+0000 7f6795c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563261b74d10 0x563261b75190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:04.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.409+0000 7f6796600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563261b759f0 0x563261b74740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:04.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.409+0000 7f6795200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563261b7e080 0x563261b80470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:04.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.409+0000 7f6795c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563261b74d10 0x563261b75190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:54786/0 (socket says 172.21.15.90:54786) 2024-04-02T14:39:04.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.409+0000 7f6796600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563261b759f0 0x563261b74740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:49568/0 (socket says 172.21.15.90:49568) 2024-04-02T14:39:04.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.409+0000 7f6795c00640 1 -- 172.21.15.90:0/217188152 learned_addr learned my addr 172.21.15.90:0/217188152 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:39:04.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.409+0000 7f6796600640 1 -- 172.21.15.90:0/217188152 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563261b74d10 msgr2=0x563261b75190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:39:04.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.409+0000 7f6796600640 1 --2- 172.21.15.90:0/217188152 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563261b74d10 0x563261b75190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:04.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.409+0000 7f6796600640 1 -- 172.21.15.90:0/217188152 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563261b7e080 msgr2=0x563261b80470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:39:04.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.409+0000 7f6796600640 1 --2- 172.21.15.90:0/217188152 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563261b7e080 0x563261b80470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:04.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.409+0000 7f6796600640 1 -- 172.21.15.90:0/217188152 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5632617ed3f0 con 0x563261b759f0 2024-04-02T14:39:04.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.410+0000 7f678e000640 1 --2- 172.21.15.90:0/1807555968 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f67780a3da0 0x7f67780a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:04.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.410+0000 7f6796600640 1 --2- 172.21.15.90:0/217188152 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563261b759f0 0x563261b74740 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f6788009f00 tx=0x7f6788011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad2af7c70b3bc3e server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:39:04.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.410+0000 7f678e000640 1 -- 172.21.15.90:0/1807555968 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f67840d0a40 con 0x7f679006a070 2024-04-02T14:39:04.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.410+0000 7f6782a00640 1 -- 172.21.15.90:0/217188152 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6788004d20 con 0x563261b759f0 2024-04-02T14:39:04.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.410+0000 7f6795c00640 1 --2- 172.21.15.90:0/1807555968 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f67780a3da0 0x7f67780a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:04.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.410+0000 7f6782a00640 1 -- 172.21.15.90:0/217188152 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6788004030 con 0x563261b759f0 2024-04-02T14:39:04.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.410+0000 7f6782a00640 1 -- 172.21.15.90:0/217188152 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6788005240 con 0x563261b759f0 2024-04-02T14:39:04.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.411+0000 7f679b377740 1 -- 172.21.15.90:0/217188152 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563261b759f0 msgr2=0x563261b74740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:04.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.411+0000 7f6795c00640 1 --2- 172.21.15.90:0/1807555968 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f67780a3da0 0x7f67780a6260 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f67840a5130 tx=0x7f6784009700 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:39:04.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.411+0000 7f679b377740 1 --2- 172.21.15.90:0/217188152 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563261b759f0 0x563261b74740 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f6788009f00 tx=0x7f6788011810 comp rx=0 tx=0).stop 2024-04-02T14:39:04.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.412+0000 7f679b377740 1 -- 172.21.15.90:0/217188152 shutdown_connections 2024-04-02T14:39:04.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.412+0000 7f679b377740 1 --2- 172.21.15.90:0/217188152 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563261b7e080 0x563261b80470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:04.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.412+0000 7f679b377740 1 --2- 172.21.15.90:0/217188152 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563261b74d10 0x563261b75190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:04.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.412+0000 7f679b377740 1 --2- 172.21.15.90:0/217188152 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563261b759f0 0x563261b74740 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:04.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.412+0000 7f679b377740 1 -- 172.21.15.90:0/217188152 >> 172.21.15.90:0/217188152 conn(0x563261b3ba50 msgr2=0x563261b6f760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:39:04.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.412+0000 7f679b377740 1 -- 172.21.15.90:0/217188152 shutdown_connections 2024-04-02T14:39:04.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.412+0000 7f679b377740 1 -- 172.21.15.90:0/217188152 wait complete. 2024-04-02T14:39:04.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.413+0000 7f679b377740 1 -- start start 2024-04-02T14:39:04.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.413+0000 7f679b377740 1 Processor -- start 2024-04-02T14:39:04.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.413+0000 7f679b377740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563261b74d10 0x563261b7b060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:04.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.413+0000 7f679b377740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563261b759f0 0x563261b7b5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:04.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.413+0000 7f679b377740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563261b7e080 0x563261b7bae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:04.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.413+0000 7f679b377740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x563261ad56a0 con 0x563261b74d10 2024-04-02T14:39:04.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.413+0000 7f679b377740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x563261a81980 con 0x563261b7e080 2024-04-02T14:39:04.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.413+0000 7f679b377740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x563261a94cb0 con 0x563261b759f0 2024-04-02T14:39:04.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.414+0000 7f679b377740 10 client.0 _ll_register_callbacks cb 0x563261a946a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:39:04.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.414+0000 7f6795200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563261b759f0 0x563261b7b5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:04.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.414+0000 7f6796600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563261b74d10 0x563261b7b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:04.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.414+0000 7f6795200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563261b759f0 0x563261b7b5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:54788/0 (socket says 172.21.15.90:54788) 2024-04-02T14:39:04.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.414+0000 7f6795200640 1 -- 172.21.15.90:0/3845879601 learned_addr learned my addr 172.21.15.90:0/3845879601 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:39:04.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.414+0000 7f6796600640 1 --2- 172.21.15.90:0/3845879601 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563261b7e080 0x563261b7bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:04.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.414+0000 7f6795200640 1 -- 172.21.15.90:0/3845879601 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563261b7e080 msgr2=0x563261b7bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:39:04.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.414+0000 7f6795200640 1 --2- 172.21.15.90:0/3845879601 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563261b7e080 0x563261b7bae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:04.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.414+0000 7f6795200640 1 -- 172.21.15.90:0/3845879601 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563261b74d10 msgr2=0x563261b7b060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:39:04.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.414+0000 7f6795200640 1 --2- 172.21.15.90:0/3845879601 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563261b74d10 0x563261b7b060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:04.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.414+0000 7f6795200640 1 -- 172.21.15.90:0/3845879601 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5632618b77d0 con 0x563261b759f0 2024-04-02T14:39:04.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.414+0000 7f6796600640 1 --2- 172.21.15.90:0/3845879601 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563261b7e080 0x563261b7bae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:39:04.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.414+0000 7f6795200640 1 --2- 172.21.15.90:0/3845879601 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563261b759f0 0x563261b7b5a0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f677c036040 tx=0x7f677c02c820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1b62026c119254f5 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:39:04.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.415+0000 7f6777400640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:39:04.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.415+0000 7f6777400640 1 -- 172.21.15.90:0/3845879601 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f677c004040 con 0x563261b759f0 2024-04-02T14:39:04.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.415+0000 7f6777400640 1 -- 172.21.15.90:0/3845879601 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f677c0041e0 con 0x563261b759f0 2024-04-02T14:39:04.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.415+0000 7f679b377740 1 -- 172.21.15.90:0/3845879601 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6788004030 con 0x563261b759f0 2024-04-02T14:39:04.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.415+0000 7f6777400640 1 -- 172.21.15.90:0/3845879601 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f677c05a050 con 0x563261b759f0 2024-04-02T14:39:04.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.415+0000 7f679b377740 10 client.26575 fetch_fsmap learned FSMap version 27 2024-04-02T14:39:04.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.415+0000 7f679b377740 1 -- 172.21.15.90:0/3845879601 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x563261b8cf70 con 0x563261b759f0 2024-04-02T14:39:04.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.415+0000 7f6777400640 1 -- 172.21.15.90:0/3845879601 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6788004030 con 0x563261b759f0 2024-04-02T14:39:04.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.416+0000 7f6777400640 1 -- 172.21.15.90:0/3845879601 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f677c005430 con 0x563261b759f0 2024-04-02T14:39:04.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.416+0000 7f679b377740 10 client.26575 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:39:04.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.416+0000 7f679b377740 10 client.26575 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-04-02T14:39:04.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.416+0000 7f679b377740 20 client.26575 populate_metadata read hostname 'smithi090' 2024-04-02T14:39:04.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.416+0000 7f679b377740 1 --2- 172.21.15.90:0/3845879601 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x563261b8e430 0x563261b90890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:04.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.416+0000 7f679b377740 4 client.26575 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:39:04.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.416+0000 7f679b377740 1 -- 172.21.15.90:0/3845879601 --> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563261a99040 con 0x563261b8e430 2024-04-02T14:39:04.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.416+0000 7f6796600640 1 --2- 172.21.15.90:0/3845879601 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x563261b8e430 0x563261b90890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:04.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.417+0000 7f6796600640 1 --2- 172.21.15.90:0/3845879601 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x563261b8e430 0x563261b90890 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=e956366410c4a663 server_cookie=72cee6dc8ff34f3f in_seq=0 out_seq=0 2024-04-02T14:39:04.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.417+0000 7f6777400640 10 client.26575 ms_handle_connect on v2:172.21.15.92:6836/1270635939 2024-04-02T14:39:04.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.417+0000 7f6777400640 1 -- 172.21.15.90:0/3845879601 <== mds.3 v2:172.21.15.92:6836/1270635939 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x563261a99040 con 0x563261b8e430 2024-04-02T14:39:04.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.417+0000 7f6777400640 10 client.26575 handle_command_reply: tid=0 2024-04-02T14:39:04.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.417+0000 7f679b377740 1 client.26575 shutdown 2024-04-02T14:39:04.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.418+0000 7f679b377740 1 -- 172.21.15.90:0/3845879601 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563261b759f0 msgr2=0x563261b7b5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:04.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.418+0000 7f679b377740 1 --2- 172.21.15.90:0/3845879601 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563261b759f0 0x563261b7b5a0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f677c036040 tx=0x7f677c02c820 comp rx=0 tx=0).stop 2024-04-02T14:39:04.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.418+0000 7f679b377740 1 -- 172.21.15.90:0/3845879601 shutdown_connections 2024-04-02T14:39:04.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.418+0000 7f679b377740 1 --2- 172.21.15.90:0/3845879601 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x563261b8e430 0x563261b90890 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:04.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.418+0000 7f6777400640 0 client.26575 ms_handle_reset on v2:172.21.15.92:6836/1270635939 2024-04-02T14:39:04.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.418+0000 7f679b377740 1 --2- 172.21.15.90:0/3845879601 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563261b7e080 0x563261b7bae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:04.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.418+0000 7f679b377740 1 --2- 172.21.15.90:0/3845879601 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563261b759f0 0x563261b7b5a0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:04.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.418+0000 7f679b377740 1 --2- 172.21.15.90:0/3845879601 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563261b74d10 0x563261b7b060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:04.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.418+0000 7f679b377740 1 -- 172.21.15.90:0/3845879601 >> 172.21.15.90:0/3845879601 conn(0x563261b3ba50 msgr2=0x563261b714b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:39:04.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.418+0000 7f679b377740 1 -- 172.21.15.90:0/3845879601 shutdown_connections 2024-04-02T14:39:04.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.418+0000 7f679b377740 1 -- 172.21.15.90:0/3845879601 wait complete. 2024-04-02T14:39:04.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.418+0000 7f679b377740 20 client.26575 trim_cache size 0 max 16384 2024-04-02T14:39:04.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.434+0000 7f679b377740 1 Processor -- start 2024-04-02T14:39:04.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.434+0000 7f679b377740 1 -- start start 2024-04-02T14:39:04.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.434+0000 7f679b377740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563261b74d10 0x563261b7d790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:04.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.434+0000 7f679b377740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563261b759f0 0x563261b89af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:04.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.434+0000 7f679b377740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563261b7e080 0x563261b96160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:04.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.434+0000 7f679b377740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f67901ab570 con 0x563261b7e080 2024-04-02T14:39:04.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.434+0000 7f679b377740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f67901ad0b0 con 0x563261b759f0 2024-04-02T14:39:04.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.434+0000 7f679b377740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f67901abc40 con 0x563261b74d10 2024-04-02T14:39:04.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.434+0000 7f6796600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563261b74d10 0x563261b7d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:04.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.434+0000 7f6796600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563261b74d10 0x563261b7d790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:54796/0 (socket says 172.21.15.90:54796) 2024-04-02T14:39:04.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.434+0000 7f6796600640 1 -- 172.21.15.90:0/2034151665 learned_addr learned my addr 172.21.15.90:0/2034151665 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:39:04.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.434+0000 7f6795200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563261b759f0 0x563261b89af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:04.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.434+0000 7f6796600640 1 --2- 172.21.15.90:0/2034151665 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563261b7e080 0x563261b96160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:04.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.435+0000 7f6795200640 1 -- 172.21.15.90:0/2034151665 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563261b74d10 msgr2=0x563261b7d790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:39:04.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.435+0000 7f6795200640 1 --2- 172.21.15.90:0/2034151665 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563261b74d10 0x563261b7d790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:04.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.435+0000 7f6795200640 1 -- 172.21.15.90:0/2034151665 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563261b7e080 msgr2=0x563261b96160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:39:04.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.435+0000 7f6795200640 1 --2- 172.21.15.90:0/2034151665 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563261b7e080 0x563261b96160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:04.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.435+0000 7f6795200640 1 -- 172.21.15.90:0/2034151665 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563261b73d40 con 0x563261b759f0 2024-04-02T14:39:04.439 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:39:03 smithi177 ceph-mon[36431]: pgmap v128: 97 pgs: 97 active+clean; 2.3 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 11 MiB/s wr, 711 op/s 2024-04-02T14:39:04.439 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:39:03 smithi177 ceph-mon[36431]: from='client.? 172.21.15.90:0/3500657956' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-02T14:39:04.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.435+0000 7f6795200640 1 --2- 172.21.15.90:0/2034151665 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563261b759f0 0x563261b89af0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f677c02c260 tx=0x7f677c009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4d2c31aeec343f58 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:39:04.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.435+0000 7f6782000640 1 -- 172.21.15.90:0/2034151665 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f677c003a30 con 0x563261b759f0 2024-04-02T14:39:04.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.435+0000 7f6782000640 1 -- 172.21.15.90:0/2034151665 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f677c051070 con 0x563261b759f0 2024-04-02T14:39:04.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.435+0000 7f6782000640 1 -- 172.21.15.90:0/2034151665 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f677c005540 con 0x563261b759f0 2024-04-02T14:39:04.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.436+0000 7f679b377740 1 -- 172.21.15.90:0/2034151665 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563261b759f0 msgr2=0x563261b89af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:04.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.436+0000 7f679b377740 1 --2- 172.21.15.90:0/2034151665 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563261b759f0 0x563261b89af0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f677c02c260 tx=0x7f677c009260 comp rx=0 tx=0).stop 2024-04-02T14:39:04.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.436+0000 7f679b377740 1 -- 172.21.15.90:0/2034151665 shutdown_connections 2024-04-02T14:39:04.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.436+0000 7f679b377740 1 --2- 172.21.15.90:0/2034151665 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563261b7e080 0x563261b96160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:04.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.436+0000 7f679b377740 1 --2- 172.21.15.90:0/2034151665 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563261b759f0 0x563261b89af0 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:04.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.436+0000 7f679b377740 1 --2- 172.21.15.90:0/2034151665 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563261b74d10 0x563261b7d790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:04.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.436+0000 7f679b377740 1 -- 172.21.15.90:0/2034151665 >> 172.21.15.90:0/2034151665 conn(0x563261b3ba50 msgr2=0x563261b6f290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:39:04.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.436+0000 7f679b377740 1 -- 172.21.15.90:0/2034151665 shutdown_connections 2024-04-02T14:39:04.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.436+0000 7f679b377740 1 -- 172.21.15.90:0/2034151665 wait complete. 2024-04-02T14:39:04.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.437+0000 7f679b377740 1 -- start start 2024-04-02T14:39:04.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.437+0000 7f679b377740 1 Processor -- start 2024-04-02T14:39:04.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.437+0000 7f679b377740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563261b74d10 0x563261b7ef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:04.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.437+0000 7f679b377740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563261b759f0 0x563261b7f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:04.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.437+0000 7f679b377740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563261b7e080 0x563261b7fa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:04.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.437+0000 7f679b377740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f67901ac9e0 con 0x563261b7e080 2024-04-02T14:39:04.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.437+0000 7f679b377740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f67901ac310 con 0x563261b74d10 2024-04-02T14:39:04.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.437+0000 7f679b377740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f67901a9570 con 0x563261b759f0 2024-04-02T14:39:04.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.437+0000 7f679b377740 10 client.0 _ll_register_callbacks cb 0x7f677c005df0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:39:04.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.438+0000 7f6795200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563261b759f0 0x563261b7f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:04.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.438+0000 7f6795200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563261b759f0 0x563261b7f4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:54802/0 (socket says 172.21.15.90:54802) 2024-04-02T14:39:04.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.438+0000 7f6795200640 1 -- 172.21.15.90:0/1390310518 learned_addr learned my addr 172.21.15.90:0/1390310518 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:39:04.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.438+0000 7f6796600640 1 --2- 172.21.15.90:0/1390310518 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563261b74d10 0x563261b7ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:04.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.438+0000 7f6796600640 1 --2- 172.21.15.90:0/1390310518 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563261b7e080 0x563261b7fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:04.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.438+0000 7f6795200640 1 -- 172.21.15.90:0/1390310518 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563261b74d10 msgr2=0x563261b7ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:39:04.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.438+0000 7f6795200640 1 --2- 172.21.15.90:0/1390310518 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563261b74d10 0x563261b7ef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:04.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.438+0000 7f6795200640 1 -- 172.21.15.90:0/1390310518 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563261b7e080 msgr2=0x563261b7fa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:39:04.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.438+0000 7f6795200640 1 --2- 172.21.15.90:0/1390310518 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563261b7e080 0x563261b7fa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:04.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.438+0000 7f6795200640 1 -- 172.21.15.90:0/1390310518 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563261b846d0 con 0x563261b759f0 2024-04-02T14:39:04.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.438+0000 7f6795200640 1 --2- 172.21.15.90:0/1390310518 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563261b759f0 0x563261b7f4d0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f677c044010 tx=0x7f677c004e50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3d514f16d4fa63d8 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:39:04.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.438+0000 7f6777400640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:39:04.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.439+0000 7f6777400640 1 -- 172.21.15.90:0/1390310518 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f677c06c070 con 0x563261b759f0 2024-04-02T14:39:04.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.439+0000 7f679b377740 1 -- 172.21.15.90:0/1390310518 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f677c051070 con 0x563261b759f0 2024-04-02T14:39:04.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.439+0000 7f6777400640 1 -- 172.21.15.90:0/1390310518 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f677c0054c0 con 0x563261b759f0 2024-04-02T14:39:04.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.439+0000 7f6777400640 1 -- 172.21.15.90:0/1390310518 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f677c003a40 con 0x563261b759f0 2024-04-02T14:39:04.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.439+0000 7f679b377740 10 client.26581 fetch_fsmap learned FSMap version 27 2024-04-02T14:39:04.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.439+0000 7f679b377740 1 -- 172.21.15.90:0/1390310518 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x563261bf4f20 con 0x563261b759f0 2024-04-02T14:39:04.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.439+0000 7f6777400640 1 -- 172.21.15.90:0/1390310518 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f677c051070 con 0x563261b759f0 2024-04-02T14:39:04.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.439+0000 7f6777400640 1 -- 172.21.15.90:0/1390310518 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f677c004200 con 0x563261b759f0 2024-04-02T14:39:04.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.439+0000 7f679b377740 10 client.26581 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:39:04.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.439+0000 7f679b377740 10 client.26581 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-04-02T14:39:04.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.439+0000 7f679b377740 20 client.26581 populate_metadata read hostname 'smithi090' 2024-04-02T14:39:04.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.439+0000 7f679b377740 1 --2- 172.21.15.90:0/1390310518 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x563261b8e430 0x563261bf7540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:04.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.439+0000 7f679b377740 4 client.26581 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-02T14:39:04.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.439+0000 7f679b377740 1 -- 172.21.15.90:0/1390310518 --> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] -- command(tid 0: {"prefix": "damage ls"}) -- 0x563261ae6c30 con 0x563261b8e430 2024-04-02T14:39:04.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.440+0000 7f6796600640 1 --2- 172.21.15.90:0/1390310518 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x563261b8e430 0x563261bf7540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:04.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.440+0000 7f6796600640 1 --2- 172.21.15.90:0/1390310518 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x563261b8e430 0x563261bf7540 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=1b33587ea320ef44 server_cookie=d4a872ff2b8d2555 in_seq=0 out_seq=0 2024-04-02T14:39:04.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.440+0000 7f6777400640 10 client.26581 ms_handle_connect on v2:172.21.15.92:6836/1270635939 2024-04-02T14:39:04.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.440+0000 7f6777400640 1 -- 172.21.15.90:0/1390310518 <== mds.3 v2:172.21.15.92:6836/1270635939 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x563261ae6c30 con 0x563261b8e430 2024-04-02T14:39:04.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.440+0000 7f6777400640 10 client.26581 handle_command_reply: tid=0 2024-04-02T14:39:04.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.440+0000 7f679b377740 1 client.26581 shutdown 2024-04-02T14:39:04.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.441+0000 7f679b377740 1 -- 172.21.15.90:0/1390310518 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563261b759f0 msgr2=0x563261b7f4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:04.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.441+0000 7f679b377740 1 --2- 172.21.15.90:0/1390310518 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563261b759f0 0x563261b7f4d0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f677c044010 tx=0x7f677c004e50 comp rx=0 tx=0).stop 2024-04-02T14:39:04.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.441+0000 7f679b377740 1 -- 172.21.15.90:0/1390310518 shutdown_connections 2024-04-02T14:39:04.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.441+0000 7f679b377740 1 --2- 172.21.15.90:0/1390310518 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x563261b8e430 0x563261bf7540 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:04.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.441+0000 7f679b377740 1 --2- 172.21.15.90:0/1390310518 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563261b7e080 0x563261b7fa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:04.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.441+0000 7f679b377740 1 --2- 172.21.15.90:0/1390310518 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563261b759f0 0x563261b7f4d0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:04.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.441+0000 7f679b377740 1 --2- 172.21.15.90:0/1390310518 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563261b74d10 0x563261b7ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:04.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.441+0000 7f6777400640 0 client.26581 ms_handle_reset on v2:172.21.15.92:6836/1270635939 2024-04-02T14:39:04.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.441+0000 7f679b377740 1 -- 172.21.15.90:0/1390310518 >> 172.21.15.90:0/1390310518 conn(0x563261b3ba50 msgr2=0x563261b7b000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:39:04.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.441+0000 7f679b377740 1 -- 172.21.15.90:0/1390310518 shutdown_connections 2024-04-02T14:39:04.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.441+0000 7f679b377740 1 -- 172.21.15.90:0/1390310518 wait complete. 2024-04-02T14:39:04.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.441+0000 7f679b377740 20 client.26581 trim_cache size 0 max 16384 2024-04-02T14:39:04.448 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-02T14:39:04.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.442+0000 7f6798600640 1 -- 172.21.15.90:0/1807555968 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f67780a3da0 msgr2=0x7f67780a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:04.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.442+0000 7f6798600640 1 --2- 172.21.15.90:0/1807555968 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f67780a3da0 0x7f67780a6260 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f67840a5130 tx=0x7f6784009700 comp rx=0 tx=0).stop 2024-04-02T14:39:04.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.442+0000 7f6798600640 1 -- 172.21.15.90:0/1807555968 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f679006a070 msgr2=0x7f679019a560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:04.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.442+0000 7f6798600640 1 --2- 172.21.15.90:0/1807555968 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f679006a070 0x7f679019a560 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f678400e9f0 tx=0x7f678400eeb0 comp rx=0 tx=0).stop 2024-04-02T14:39:04.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.442+0000 7f6798600640 1 -- 172.21.15.90:0/1807555968 shutdown_connections 2024-04-02T14:39:04.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.442+0000 7f6798600640 1 --2- 172.21.15.90:0/1807555968 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f67780a3da0 0x7f67780a6260 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:04.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.442+0000 7f6798600640 1 --2- 172.21.15.90:0/1807555968 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6790107ab0 0x7f679019ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:04.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.442+0000 7f6798600640 1 --2- 172.21.15.90:0/1807555968 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f679006a9f0 0x7f679019aaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:04.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.442+0000 7f6798600640 1 --2- 172.21.15.90:0/1807555968 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f679006a070 0x7f679019a560 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:04.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.442+0000 7f6798600640 1 -- 172.21.15.90:0/1807555968 >> 172.21.15.90:0/1807555968 conn(0x7f67900779e0 msgr2=0x7f6790102070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:39:04.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.442+0000 7f6798600640 1 -- 172.21.15.90:0/1807555968 shutdown_connections 2024-04-02T14:39:04.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.442+0000 7f6798600640 1 -- 172.21.15.90:0/1807555968 wait complete. 2024-04-02T14:39:04.462 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-02T14:39:04.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.670+0000 7fda84c00640 1 -- 172.21.15.90:0/2606390091 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fda80073eb0 msgr2=0x7fda80074290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:04.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.670+0000 7fda84c00640 1 --2- 172.21.15.90:0/2606390091 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fda80073eb0 0x7fda80074290 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fda70009900 tx=0x7fda7002bcd0 comp rx=0 tx=0).stop 2024-04-02T14:39:04.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.670+0000 7fda84c00640 1 -- 172.21.15.90:0/2606390091 shutdown_connections 2024-04-02T14:39:04.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.670+0000 7fda84c00640 1 --2- 172.21.15.90:0/2606390091 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fda8010e230 0x7fda80110620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:04.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.670+0000 7fda84c00640 1 --2- 172.21.15.90:0/2606390091 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fda800747d0 0x7fda8010dcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:04.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.670+0000 7fda84c00640 1 --2- 172.21.15.90:0/2606390091 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fda80073eb0 0x7fda80074290 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:04.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.670+0000 7fda84c00640 1 -- 172.21.15.90:0/2606390091 >> 172.21.15.90:0/2606390091 conn(0x7fda800fd4a0 msgr2=0x7fda800ff8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:39:04.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.670+0000 7fda84c00640 1 -- 172.21.15.90:0/2606390091 shutdown_connections 2024-04-02T14:39:04.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.670+0000 7fda84c00640 1 -- 172.21.15.90:0/2606390091 wait complete. 2024-04-02T14:39:04.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.671+0000 7fda84c00640 1 Processor -- start 2024-04-02T14:39:04.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.671+0000 7fda84c00640 1 -- start start 2024-04-02T14:39:04.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.672+0000 7fda84c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fda80073eb0 0x7fda8019a440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:04.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.672+0000 7fda84c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fda800747d0 0x7fda8019a980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:04.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.672+0000 7fda84c00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fda8010e230 0x7fda8019ed00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:04.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.672+0000 7fda84c00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fda80113fa0 con 0x7fda800747d0 2024-04-02T14:39:04.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.672+0000 7fda84c00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fda80113e20 con 0x7fda8010e230 2024-04-02T14:39:04.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.672+0000 7fda84c00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fda80114120 con 0x7fda80073eb0 2024-04-02T14:39:04.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.672+0000 7fda7f400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fda80073eb0 0x7fda8019a440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:04.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.672+0000 7fda7f400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fda80073eb0 0x7fda8019a440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:54822/0 (socket says 172.21.15.90:54822) 2024-04-02T14:39:04.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.672+0000 7fda7fe00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fda8010e230 0x7fda8019ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:04.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.672+0000 7fda7f400640 1 -- 172.21.15.90:0/1159818832 learned_addr learned my addr 172.21.15.90:0/1159818832 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:39:04.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.672+0000 7fda7ea00640 1 --2- 172.21.15.90:0/1159818832 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fda800747d0 0x7fda8019a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:04.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.672+0000 7fda7fe00640 1 -- 172.21.15.90:0/1159818832 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fda80073eb0 msgr2=0x7fda8019a440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:04.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.672+0000 7fda7fe00640 1 --2- 172.21.15.90:0/1159818832 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fda80073eb0 0x7fda8019a440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:04.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.672+0000 7fda7fe00640 1 -- 172.21.15.90:0/1159818832 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fda800747d0 msgr2=0x7fda8019a980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:04.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.672+0000 7fda7fe00640 1 --2- 172.21.15.90:0/1159818832 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fda800747d0 0x7fda8019a980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:04.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.672+0000 7fda7fe00640 1 -- 172.21.15.90:0/1159818832 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda8019f430 con 0x7fda8010e230 2024-04-02T14:39:04.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.672+0000 7fda7f400640 1 --2- 172.21.15.90:0/1159818832 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fda80073eb0 0x7fda8019a440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:39:04.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.673+0000 7fda7fe00640 1 --2- 172.21.15.90:0/1159818832 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fda8010e230 0x7fda8019ed00 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fda78009fd0 tx=0x7fda7800ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:39:04.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.673+0000 7fda77e00640 1 -- 172.21.15.90:0/1159818832 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda7800ce20 con 0x7fda8010e230 2024-04-02T14:39:04.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.673+0000 7fda77e00640 1 -- 172.21.15.90:0/1159818832 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fda78005020 con 0x7fda8010e230 2024-04-02T14:39:04.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.673+0000 7fda77e00640 1 -- 172.21.15.90:0/1159818832 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda78010720 con 0x7fda8010e230 2024-04-02T14:39:04.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.673+0000 7fda84c00640 1 -- 172.21.15.90:0/1159818832 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda8019f720 con 0x7fda8010e230 2024-04-02T14:39:04.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.673+0000 7fda84c00640 1 -- 172.21.15.90:0/1159818832 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fda801a6fb0 con 0x7fda8010e230 2024-04-02T14:39:04.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.676+0000 7fda88e6b740 1 Processor -- start 2024-04-02T14:39:04.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.676+0000 7fda77e00640 1 -- 172.21.15.90:0/1159818832 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fda78005540 con 0x7fda8010e230 2024-04-02T14:39:04.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.676+0000 7fda88e6b740 1 -- start start 2024-04-02T14:39:04.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.677+0000 7fda88e6b740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c9f8ff0ba0 0x55c9f8fef8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:04.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.677+0000 7fda88e6b740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c9f8fefec0 0x55c9f8ff0340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:04.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.677+0000 7fda88e6b740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c9f8ff9230 0x55c9f8ffb620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:04.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.677+0000 7fda88e6b740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55c9f8f30400 con 0x55c9f8ff0ba0 2024-04-02T14:39:04.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.677+0000 7fda88e6b740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55c9f8f14f10 con 0x55c9f8ff9230 2024-04-02T14:39:04.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.677+0000 7fda88e6b740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55c9f8f546f0 con 0x55c9f8fefec0 2024-04-02T14:39:04.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.677+0000 7fda7f400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c9f8fefec0 0x55c9f8ff0340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:04.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.677+0000 7fda7ea00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c9f8ff0ba0 0x55c9f8fef8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:04.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.677+0000 7fda7f400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c9f8fefec0 0x55c9f8ff0340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:54824/0 (socket says 172.21.15.90:54824) 2024-04-02T14:39:04.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.677+0000 7fda7f400640 1 -- 172.21.15.90:0/3904688750 learned_addr learned my addr 172.21.15.90:0/3904688750 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:39:04.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.677+0000 7fda7ea00640 1 --2- 172.21.15.90:0/3904688750 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c9f8ff9230 0x55c9f8ffb620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:04.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.677+0000 7fda7f400640 1 -- 172.21.15.90:0/3904688750 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c9f8ff9230 msgr2=0x55c9f8ffb620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:39:04.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.677+0000 7fda7f400640 1 --2- 172.21.15.90:0/3904688750 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c9f8ff9230 0x55c9f8ffb620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:04.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.677+0000 7fda7f400640 1 -- 172.21.15.90:0/3904688750 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c9f8ff0ba0 msgr2=0x55c9f8fef8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:39:04.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.677+0000 7fda7f400640 1 --2- 172.21.15.90:0/3904688750 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c9f8ff0ba0 0x55c9f8fef8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:04.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.677+0000 7fda7f400640 1 -- 172.21.15.90:0/3904688750 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c9f8c683f0 con 0x55c9f8fefec0 2024-04-02T14:39:04.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.678+0000 7fda7f400640 1 --2- 172.21.15.90:0/3904688750 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c9f8fefec0 0x55c9f8ff0340 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fda70005c30 tx=0x7fda70009560 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8a31b25860a33061 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:39:04.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.678+0000 7fda77e00640 1 --2- 172.21.15.90:0/1159818832 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fda640a3d20 0x7fda640a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:04.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.678+0000 7fda77e00640 1 -- 172.21.15.90:0/1159818832 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7fda780d0e10 con 0x7fda8010e230 2024-04-02T14:39:04.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.678+0000 7fda7ea00640 1 --2- 172.21.15.90:0/1159818832 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fda640a3d20 0x7fda640a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:04.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.678+0000 7fda6be00640 1 -- 172.21.15.90:0/3904688750 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda70043070 con 0x55c9f8fefec0 2024-04-02T14:39:04.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.678+0000 7fda6be00640 1 -- 172.21.15.90:0/3904688750 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fda70004a70 con 0x55c9f8fefec0 2024-04-02T14:39:04.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.678+0000 7fda6be00640 1 -- 172.21.15.90:0/3904688750 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda700051a0 con 0x55c9f8fefec0 2024-04-02T14:39:04.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.679+0000 7fda88e6b740 1 -- 172.21.15.90:0/3904688750 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c9f8fefec0 msgr2=0x55c9f8ff0340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:04.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.679+0000 7fda7ea00640 1 --2- 172.21.15.90:0/1159818832 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fda640a3d20 0x7fda640a61e0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fda6c009730 tx=0x7fda6c015040 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:39:04.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.679+0000 7fda88e6b740 1 --2- 172.21.15.90:0/3904688750 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c9f8fefec0 0x55c9f8ff0340 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fda70005c30 tx=0x7fda70009560 comp rx=0 tx=0).stop 2024-04-02T14:39:04.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.679+0000 7fda88e6b740 1 -- 172.21.15.90:0/3904688750 shutdown_connections 2024-04-02T14:39:04.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.679+0000 7fda88e6b740 1 --2- 172.21.15.90:0/3904688750 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c9f8ff9230 0x55c9f8ffb620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:04.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.679+0000 7fda88e6b740 1 --2- 172.21.15.90:0/3904688750 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c9f8fefec0 0x55c9f8ff0340 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:04.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.679+0000 7fda88e6b740 1 --2- 172.21.15.90:0/3904688750 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c9f8ff0ba0 0x55c9f8fef8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:04.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.679+0000 7fda88e6b740 1 -- 172.21.15.90:0/3904688750 >> 172.21.15.90:0/3904688750 conn(0x55c9f8fb6a50 msgr2=0x55c9f8fea910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:39:04.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.679+0000 7fda88e6b740 1 -- 172.21.15.90:0/3904688750 shutdown_connections 2024-04-02T14:39:04.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.679+0000 7fda88e6b740 1 -- 172.21.15.90:0/3904688750 wait complete. 2024-04-02T14:39:04.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.680+0000 7fda88e6b740 1 -- start start 2024-04-02T14:39:04.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.680+0000 7fda88e6b740 1 Processor -- start 2024-04-02T14:39:04.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.680+0000 7fda88e6b740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c9f8fefec0 0x55c9f8ff61f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:04.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.680+0000 7fda88e6b740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c9f8ff0ba0 0x55c9f8ff6730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:04.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.680+0000 7fda88e6b740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c9f8ff9230 0x55c9f8ff6c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:04.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.680+0000 7fda88e6b740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55c9f8f506a0 con 0x55c9f8fefec0 2024-04-02T14:39:04.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.680+0000 7fda88e6b740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55c9f8efc980 con 0x55c9f8ff9230 2024-04-02T14:39:04.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.680+0000 7fda88e6b740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55c9f8f0fcb0 con 0x55c9f8ff0ba0 2024-04-02T14:39:04.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.681+0000 7fda88e6b740 10 client.0 _ll_register_callbacks cb 0x55c9f8f0f6a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:39:04.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.681+0000 7fda7fe00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c9f8ff9230 0x55c9f8ff6c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:04.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.681+0000 7fda7ea00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c9f8ff0ba0 0x55c9f8ff6730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:04.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.681+0000 7fda7ea00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c9f8ff0ba0 0x55c9f8ff6730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:54838/0 (socket says 172.21.15.90:54838) 2024-04-02T14:39:04.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.681+0000 7fda7fe00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c9f8ff9230 0x55c9f8ff6c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:41632/0 (socket says 172.21.15.90:41632) 2024-04-02T14:39:04.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.681+0000 7fda7ea00640 1 -- 172.21.15.90:0/4231034506 learned_addr learned my addr 172.21.15.90:0/4231034506 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:39:04.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.681+0000 7fda7f400640 1 --2- 172.21.15.90:0/4231034506 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c9f8fefec0 0x55c9f8ff61f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:04.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.681+0000 7fda7fe00640 1 -- 172.21.15.90:0/4231034506 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c9f8ff0ba0 msgr2=0x55c9f8ff6730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:39:04.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.681+0000 7fda7fe00640 1 --2- 172.21.15.90:0/4231034506 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c9f8ff0ba0 0x55c9f8ff6730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:04.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.681+0000 7fda7fe00640 1 -- 172.21.15.90:0/4231034506 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c9f8fefec0 msgr2=0x55c9f8ff61f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:39:04.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.681+0000 7fda7fe00640 1 --2- 172.21.15.90:0/4231034506 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c9f8fefec0 0x55c9f8ff61f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:04.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.681+0000 7fda7fe00640 1 -- 172.21.15.90:0/4231034506 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c9f8d327d0 con 0x55c9f8ff9230 2024-04-02T14:39:04.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.681+0000 7fda7f400640 1 --2- 172.21.15.90:0/4231034506 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c9f8fefec0 0x55c9f8ff61f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:39:04.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.681+0000 7fda7fe00640 1 --2- 172.21.15.90:0/4231034506 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c9f8ff9230 0x55c9f8ff6c70 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fda780ce510 tx=0x7fda7809cd60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=acf79064ae75a79 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:39:04.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.681+0000 7fda68c00640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:39:04.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.682+0000 7fda68c00640 1 -- 172.21.15.90:0/4231034506 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda780b6060 con 0x55c9f8ff9230 2024-04-02T14:39:04.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.682+0000 7fda68c00640 1 -- 172.21.15.90:0/4231034506 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7fda780993d0 con 0x55c9f8ff9230 2024-04-02T14:39:04.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.682+0000 7fda88e6b740 1 -- 172.21.15.90:0/4231034506 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fda70004a70 con 0x55c9f8ff9230 2024-04-02T14:39:04.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.682+0000 7fda68c00640 1 -- 172.21.15.90:0/4231034506 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fda780a59d0 con 0x55c9f8ff9230 2024-04-02T14:39:04.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.682+0000 7fda88e6b740 10 client.26711 fetch_fsmap learned FSMap version 27 2024-04-02T14:39:04.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.682+0000 7fda88e6b740 1 -- 172.21.15.90:0/4231034506 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55c9f90080a0 con 0x55c9f8ff9230 2024-04-02T14:39:04.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.682+0000 7fda68c00640 1 -- 172.21.15.90:0/4231034506 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fda70004a70 con 0x55c9f8ff9230 2024-04-02T14:39:04.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.682+0000 7fda68c00640 1 -- 172.21.15.90:0/4231034506 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fda7809c1b0 con 0x55c9f8ff9230 2024-04-02T14:39:04.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.683+0000 7fda88e6b740 10 client.26711 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:39:04.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.683+0000 7fda88e6b740 10 client.26711 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-04-02T14:39:04.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.683+0000 7fda88e6b740 20 client.26711 populate_metadata read hostname 'smithi090' 2024-04-02T14:39:04.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.683+0000 7fda88e6b740 1 --2- 172.21.15.90:0/4231034506 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x55c9f9009560 0x55c9f900b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:04.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.683+0000 7fda88e6b740 4 client.26711 mds_command: new command op to 24547 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:39:04.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.683+0000 7fda88e6b740 1 -- 172.21.15.90:0/4231034506 --> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c9f8f14040 con 0x55c9f9009560 2024-04-02T14:39:04.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.683+0000 7fda7f400640 1 --2- 172.21.15.90:0/4231034506 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x55c9f9009560 0x55c9f900b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:04.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.684+0000 7fda7f400640 1 --2- 172.21.15.90:0/4231034506 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x55c9f9009560 0x55c9f900b990 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=3025936aa54cd0f8 server_cookie=b1541348c82eb130 in_seq=0 out_seq=0 2024-04-02T14:39:04.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.684+0000 7fda68c00640 10 client.26711 ms_handle_connect on v2:172.21.15.177:6838/664895663 2024-04-02T14:39:04.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.684+0000 7fda68c00640 1 -- 172.21.15.90:0/4231034506 <== mds.2 v2:172.21.15.177:6838/664895663 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c9f8f14040 con 0x55c9f9009560 2024-04-02T14:39:04.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.684+0000 7fda68c00640 10 client.26711 handle_command_reply: tid=0 2024-04-02T14:39:04.690 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.684+0000 7fda88e6b740 1 client.26711 shutdown 2024-04-02T14:39:04.690 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.686+0000 7fda88e6b740 1 -- 172.21.15.90:0/4231034506 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c9f8ff9230 msgr2=0x55c9f8ff6c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:04.690 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.686+0000 7fda88e6b740 1 --2- 172.21.15.90:0/4231034506 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c9f8ff9230 0x55c9f8ff6c70 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fda780ce510 tx=0x7fda7809cd60 comp rx=0 tx=0).stop 2024-04-02T14:39:04.690 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.688+0000 7fda88e6b740 1 -- 172.21.15.90:0/4231034506 shutdown_connections 2024-04-02T14:39:04.691 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.688+0000 7fda88e6b740 1 --2- 172.21.15.90:0/4231034506 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x55c9f9009560 0x55c9f900b990 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:04.691 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.688+0000 7fda88e6b740 1 --2- 172.21.15.90:0/4231034506 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c9f8ff9230 0x55c9f8ff6c70 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:04.691 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.688+0000 7fda88e6b740 1 --2- 172.21.15.90:0/4231034506 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c9f8ff0ba0 0x55c9f8ff6730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:04.691 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.688+0000 7fda88e6b740 1 --2- 172.21.15.90:0/4231034506 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c9f8fefec0 0x55c9f8ff61f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:04.691 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.688+0000 7fda88e6b740 1 -- 172.21.15.90:0/4231034506 >> 172.21.15.90:0/4231034506 conn(0x55c9f8fb6a50 msgr2=0x55c9f8fec660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:39:04.691 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.689+0000 7fda68c00640 0 client.26711 ms_handle_reset on v2:172.21.15.177:6838/664895663 2024-04-02T14:39:04.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.690+0000 7fda88e6b740 1 -- 172.21.15.90:0/4231034506 shutdown_connections 2024-04-02T14:39:04.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.690+0000 7fda88e6b740 1 -- 172.21.15.90:0/4231034506 wait complete. 2024-04-02T14:39:04.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.690+0000 7fda88e6b740 20 client.26711 trim_cache size 0 max 16384 2024-04-02T14:39:04.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.712+0000 7fda88e6b740 1 Processor -- start 2024-04-02T14:39:04.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.712+0000 7fda88e6b740 1 -- start start 2024-04-02T14:39:04.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.713+0000 7fda88e6b740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c9f8fefec0 0x55c9f8ff8920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:04.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.713+0000 7fda88e6b740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c9f8ff0ba0 0x55c9f9004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:04.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.713+0000 7fda88e6b740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c9f8ff9230 0x55c9f8fec470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:04.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.713+0000 7fda88e6b740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fda801af7b0 con 0x55c9f8ff0ba0 2024-04-02T14:39:04.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.713+0000 7fda88e6b740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fda801b12f0 con 0x55c9f8fefec0 2024-04-02T14:39:04.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.713+0000 7fda88e6b740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fda801afe80 con 0x55c9f8ff9230 2024-04-02T14:39:04.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.713+0000 7fda7f400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c9f8fefec0 0x55c9f8ff8920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:04.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.713+0000 7fda7f400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c9f8fefec0 0x55c9f8ff8920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:41642/0 (socket says 172.21.15.90:41642) 2024-04-02T14:39:04.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.713+0000 7fda7f400640 1 -- 172.21.15.90:0/1745420387 learned_addr learned my addr 172.21.15.90:0/1745420387 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:39:04.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.713+0000 7fda7ea00640 1 --2- 172.21.15.90:0/1745420387 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c9f8ff0ba0 0x55c9f9004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:04.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.713+0000 7fda7fe00640 1 --2- 172.21.15.90:0/1745420387 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c9f8ff9230 0x55c9f8fec470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:04.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.714+0000 7fda7f400640 1 -- 172.21.15.90:0/1745420387 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c9f8ff9230 msgr2=0x55c9f8fec470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:39:04.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.714+0000 7fda7f400640 1 --2- 172.21.15.90:0/1745420387 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c9f8ff9230 0x55c9f8fec470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:04.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.714+0000 7fda7f400640 1 -- 172.21.15.90:0/1745420387 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c9f8ff0ba0 msgr2=0x55c9f9004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:39:04.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.714+0000 7fda7f400640 1 --2- 172.21.15.90:0/1745420387 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c9f8ff0ba0 0x55c9f9004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:04.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.714+0000 7fda7f400640 1 -- 172.21.15.90:0/1745420387 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c9f8feeef0 con 0x55c9f8fefec0 2024-04-02T14:39:04.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.714+0000 7fda7f400640 1 --2- 172.21.15.90:0/1745420387 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c9f8fefec0 0x55c9f8ff8920 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fda70036040 tx=0x7fda7002cc40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ddf48342c7dcae87 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:39:04.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.714+0000 7fda6b400640 1 -- 172.21.15.90:0/1745420387 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda70054070 con 0x55c9f8fefec0 2024-04-02T14:39:04.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.714+0000 7fda6b400640 1 -- 172.21.15.90:0/1745420387 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fda70004a70 con 0x55c9f8fefec0 2024-04-02T14:39:04.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.714+0000 7fda6b400640 1 -- 172.21.15.90:0/1745420387 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda70043040 con 0x55c9f8fefec0 2024-04-02T14:39:04.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.715+0000 7fda88e6b740 1 -- 172.21.15.90:0/1745420387 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c9f8fefec0 msgr2=0x55c9f8ff8920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:04.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.715+0000 7fda88e6b740 1 --2- 172.21.15.90:0/1745420387 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c9f8fefec0 0x55c9f8ff8920 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fda70036040 tx=0x7fda7002cc40 comp rx=0 tx=0).stop 2024-04-02T14:39:04.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.716+0000 7fda88e6b740 1 -- 172.21.15.90:0/1745420387 shutdown_connections 2024-04-02T14:39:04.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.716+0000 7fda88e6b740 1 --2- 172.21.15.90:0/1745420387 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c9f8ff9230 0x55c9f8fec470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:04.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.716+0000 7fda88e6b740 1 --2- 172.21.15.90:0/1745420387 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c9f8ff0ba0 0x55c9f9004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:04.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.716+0000 7fda88e6b740 1 --2- 172.21.15.90:0/1745420387 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c9f8fefec0 0x55c9f8ff8920 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:04.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.716+0000 7fda88e6b740 1 -- 172.21.15.90:0/1745420387 >> 172.21.15.90:0/1745420387 conn(0x55c9f8fb6a50 msgr2=0x55c9f8febd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:39:04.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.716+0000 7fda88e6b740 1 -- 172.21.15.90:0/1745420387 shutdown_connections 2024-04-02T14:39:04.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.716+0000 7fda88e6b740 1 -- 172.21.15.90:0/1745420387 wait complete. 2024-04-02T14:39:04.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.718+0000 7fda88e6b740 1 -- start start 2024-04-02T14:39:04.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.718+0000 7fda88e6b740 1 Processor -- start 2024-04-02T14:39:04.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.719+0000 7fda88e6b740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c9f8fefec0 0x55c9f8ffd200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:04.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.719+0000 7fda88e6b740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c9f8ff0ba0 0x55c9f8ffd740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:04.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.720+0000 7fda88e6b740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c9f8ff9230 0x55c9f9077330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:04.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.720+0000 7fda88e6b740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fda801b0c20 con 0x55c9f8ff0ba0 2024-04-02T14:39:04.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.720+0000 7fda88e6b740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fda801b0550 con 0x55c9f8fefec0 2024-04-02T14:39:04.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.720+0000 7fda88e6b740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fda801ad790 con 0x55c9f8ff9230 2024-04-02T14:39:04.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.720+0000 7fda7f400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c9f8fefec0 0x55c9f8ffd200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:04.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.720+0000 7fda88e6b740 10 client.0 _ll_register_callbacks cb 0x7fda7809c730 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:39:04.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.720+0000 7fda7f400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c9f8fefec0 0x55c9f8ffd200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:41644/0 (socket says 172.21.15.90:41644) 2024-04-02T14:39:04.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.720+0000 7fda7f400640 1 -- 172.21.15.90:0/2901857400 learned_addr learned my addr 172.21.15.90:0/2901857400 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:39:04.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.721+0000 7fda7f400640 1 -- 172.21.15.90:0/2901857400 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c9f8ff9230 msgr2=0x55c9f9077330 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-02T14:39:04.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.721+0000 7fda7f400640 1 --2- 172.21.15.90:0/2901857400 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c9f8ff9230 0x55c9f9077330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:04.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.721+0000 7fda7f400640 1 -- 172.21.15.90:0/2901857400 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c9f8ff0ba0 msgr2=0x55c9f8ffd740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:39:04.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.721+0000 7fda7f400640 1 --2- 172.21.15.90:0/2901857400 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c9f8ff0ba0 0x55c9f8ffd740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:04.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.721+0000 7fda7f400640 1 -- 172.21.15.90:0/2901857400 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c9f9077870 con 0x55c9f8fefec0 2024-04-02T14:39:04.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.722+0000 7fda7f400640 1 --2- 172.21.15.90:0/2901857400 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c9f8fefec0 0x55c9f8ffd200 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fda70052040 tx=0x7fda7002bcd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c74d3a65a1382ec3 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:39:04.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.722+0000 7fda68c00640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:39:04.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.722+0000 7fda68c00640 1 -- 172.21.15.90:0/2901857400 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda70054050 con 0x55c9f8fefec0 2024-04-02T14:39:04.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.722+0000 7fda88e6b740 1 -- 172.21.15.90:0/2901857400 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fda70004a70 con 0x55c9f8fefec0 2024-04-02T14:39:04.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.722+0000 7fda68c00640 1 -- 172.21.15.90:0/2901857400 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7fda700053b0 con 0x55c9f8fefec0 2024-04-02T14:39:04.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.723+0000 7fda68c00640 1 -- 172.21.15.90:0/2901857400 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fda70043040 con 0x55c9f8fefec0 2024-04-02T14:39:04.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.723+0000 7fda88e6b740 10 client.26717 fetch_fsmap learned FSMap version 27 2024-04-02T14:39:04.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.723+0000 7fda88e6b740 1 -- 172.21.15.90:0/2901857400 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55c9f9078ae0 con 0x55c9f8fefec0 2024-04-02T14:39:04.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.723+0000 7fda68c00640 1 -- 172.21.15.90:0/2901857400 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fda70004a70 con 0x55c9f8fefec0 2024-04-02T14:39:04.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.724+0000 7fda68c00640 1 -- 172.21.15.90:0/2901857400 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fda700345a0 con 0x55c9f8fefec0 2024-04-02T14:39:04.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.724+0000 7fda88e6b740 10 client.26717 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:39:04.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.724+0000 7fda88e6b740 10 client.26717 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-04-02T14:39:04.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.724+0000 7fda88e6b740 20 client.26717 populate_metadata read hostname 'smithi090' 2024-04-02T14:39:04.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.724+0000 7fda88e6b740 1 --2- 172.21.15.90:0/2901857400 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x55c9f9078d00 0x55c9f907b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:04.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.724+0000 7fda88e6b740 4 client.26717 mds_command: new command op to 24547 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-02T14:39:04.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.724+0000 7fda88e6b740 1 -- 172.21.15.90:0/2901857400 --> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55c9f8f61c30 con 0x55c9f9078d00 2024-04-02T14:39:04.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.724+0000 7fda7ea00640 1 --2- 172.21.15.90:0/2901857400 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x55c9f9078d00 0x55c9f907b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:04.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.725+0000 7fda7ea00640 1 --2- 172.21.15.90:0/2901857400 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x55c9f9078d00 0x55c9f907b0f0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=8c3e1cd252b667bf server_cookie=3f449c6765def09d in_seq=0 out_seq=0 2024-04-02T14:39:04.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.725+0000 7fda68c00640 10 client.26717 ms_handle_connect on v2:172.21.15.177:6838/664895663 2024-04-02T14:39:04.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.725+0000 7fda68c00640 1 -- 172.21.15.90:0/2901857400 <== mds.2 v2:172.21.15.177:6838/664895663 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55c9f8f61c30 con 0x55c9f9078d00 2024-04-02T14:39:04.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.725+0000 7fda68c00640 10 client.26717 handle_command_reply: tid=0 2024-04-02T14:39:04.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.726+0000 7fda88e6b740 1 client.26717 shutdown 2024-04-02T14:39:04.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.726+0000 7fda88e6b740 1 -- 172.21.15.90:0/2901857400 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c9f8fefec0 msgr2=0x55c9f8ffd200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:04.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.726+0000 7fda88e6b740 1 --2- 172.21.15.90:0/2901857400 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c9f8fefec0 0x55c9f8ffd200 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fda70052040 tx=0x7fda7002bcd0 comp rx=0 tx=0).stop 2024-04-02T14:39:04.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.726+0000 7fda88e6b740 1 -- 172.21.15.90:0/2901857400 shutdown_connections 2024-04-02T14:39:04.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.726+0000 7fda88e6b740 1 --2- 172.21.15.90:0/2901857400 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x55c9f9078d00 0x55c9f907b0f0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:04.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.726+0000 7fda88e6b740 1 --2- 172.21.15.90:0/2901857400 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c9f8ff9230 0x55c9f9077330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:04.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.726+0000 7fda88e6b740 1 --2- 172.21.15.90:0/2901857400 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c9f8ff0ba0 0x55c9f8ffd740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:04.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.726+0000 7fda88e6b740 1 --2- 172.21.15.90:0/2901857400 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c9f8fefec0 0x55c9f8ffd200 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:04.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.726+0000 7fda88e6b740 1 -- 172.21.15.90:0/2901857400 >> 172.21.15.90:0/2901857400 conn(0x55c9f8fb6a50 msgr2=0x55c9f8fe9af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:39:04.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.726+0000 7fda68c00640 0 client.26717 ms_handle_reset on v2:172.21.15.177:6838/664895663 2024-04-02T14:39:04.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.727+0000 7fda88e6b740 1 -- 172.21.15.90:0/2901857400 shutdown_connections 2024-04-02T14:39:04.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.727+0000 7fda88e6b740 1 -- 172.21.15.90:0/2901857400 wait complete. 2024-04-02T14:39:04.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.727+0000 7fda88e6b740 20 client.26717 trim_cache size 0 max 16384 2024-04-02T14:39:04.734 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-02T14:39:04.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.729+0000 7fda84c00640 1 -- 172.21.15.90:0/1159818832 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fda640a3d20 msgr2=0x7fda640a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:04.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.729+0000 7fda84c00640 1 --2- 172.21.15.90:0/1159818832 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fda640a3d20 0x7fda640a61e0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fda6c009730 tx=0x7fda6c015040 comp rx=0 tx=0).stop 2024-04-02T14:39:04.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.729+0000 7fda84c00640 1 -- 172.21.15.90:0/1159818832 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fda8010e230 msgr2=0x7fda8019ed00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:04.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.729+0000 7fda84c00640 1 --2- 172.21.15.90:0/1159818832 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fda8010e230 0x7fda8019ed00 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fda78009fd0 tx=0x7fda7800ef10 comp rx=0 tx=0).stop 2024-04-02T14:39:04.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.730+0000 7fda84c00640 1 -- 172.21.15.90:0/1159818832 shutdown_connections 2024-04-02T14:39:04.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.730+0000 7fda84c00640 1 --2- 172.21.15.90:0/1159818832 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fda640a3d20 0x7fda640a61e0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:04.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.730+0000 7fda84c00640 1 --2- 172.21.15.90:0/1159818832 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fda8010e230 0x7fda8019ed00 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:04.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.730+0000 7fda84c00640 1 --2- 172.21.15.90:0/1159818832 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fda800747d0 0x7fda8019a980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:04.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.730+0000 7fda84c00640 1 --2- 172.21.15.90:0/1159818832 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fda80073eb0 0x7fda8019a440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:04.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.730+0000 7fda84c00640 1 -- 172.21.15.90:0/1159818832 >> 172.21.15.90:0/1159818832 conn(0x7fda800fd4a0 msgr2=0x7fda800fe170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:39:04.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.732+0000 7fda84c00640 1 -- 172.21.15.90:0/1159818832 shutdown_connections 2024-04-02T14:39:04.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:04.732+0000 7fda84c00640 1 -- 172.21.15.90:0/1159818832 wait complete. 2024-04-02T14:39:04.758 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-02T14:39:05.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.055+0000 7f7ed6c00640 1 -- 172.21.15.90:0/2914411255 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7ed0100860 msgr2=0x7f7ed0100ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:05.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.055+0000 7f7ed6c00640 1 --2- 172.21.15.90:0/2914411255 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7ed0100860 0x7f7ed0100ce0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f7ebc00d940 tx=0x7f7ebc02dbb0 comp rx=0 tx=0).stop 2024-04-02T14:39:05.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.060+0000 7f7ed6c00640 1 -- 172.21.15.90:0/2914411255 shutdown_connections 2024-04-02T14:39:05.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.060+0000 7f7ed6c00640 1 --2- 172.21.15.90:0/2914411255 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7ed0101220 0x7f7ed010d730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:05.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.060+0000 7f7ed6c00640 1 --2- 172.21.15.90:0/2914411255 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7ed0100860 0x7f7ed0100ce0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:05.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.060+0000 7f7ed6c00640 1 --2- 172.21.15.90:0/2914411255 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7ed0106870 0x7f7ed0106c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:05.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.060+0000 7f7ed6c00640 1 -- 172.21.15.90:0/2914411255 >> 172.21.15.90:0/2914411255 conn(0x7f7ed00fc350 msgr2=0x7f7ed00fe770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:39:05.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.062+0000 7f7ed6c00640 1 -- 172.21.15.90:0/2914411255 shutdown_connections 2024-04-02T14:39:05.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.062+0000 7f7ed6c00640 1 -- 172.21.15.90:0/2914411255 wait complete. 2024-04-02T14:39:05.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.064+0000 7f7ed6c00640 1 Processor -- start 2024-04-02T14:39:05.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.064+0000 7f7ed6c00640 1 -- start start 2024-04-02T14:39:05.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.064+0000 7f7ed6c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7ed0100860 0x7f7ed0073470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:05.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.064+0000 7f7ed6c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7ed0101220 0x7f7ed0070820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:05.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.064+0000 7f7ed6c00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7ed0106870 0x7f7ed0070d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:05.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.064+0000 7f7ed6c00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f7ed010fbc0 con 0x7f7ed0100860 2024-04-02T14:39:05.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.065+0000 7f7ed4e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7ed0101220 0x7f7ed0070820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:05.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.065+0000 7f7ed6200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7ed0106870 0x7f7ed0070d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:05.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.065+0000 7f7ed4e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7ed0101220 0x7f7ed0070820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:54864/0 (socket says 172.21.15.90:54864) 2024-04-02T14:39:05.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.065+0000 7f7ed4e00640 1 -- 172.21.15.90:0/2215016861 learned_addr learned my addr 172.21.15.90:0/2215016861 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:39:05.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.065+0000 7f7ed6c00640 1 -- 172.21.15.90:0/2215016861 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f7ed010fa40 con 0x7f7ed0106870 2024-04-02T14:39:05.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.065+0000 7f7ed6c00640 1 -- 172.21.15.90:0/2215016861 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f7ed010fd40 con 0x7f7ed0101220 2024-04-02T14:39:05.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.065+0000 7f7ed5800640 1 --2- 172.21.15.90:0/2215016861 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7ed0100860 0x7f7ed0073470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:05.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.066+0000 7f7ed6200640 1 -- 172.21.15.90:0/2215016861 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7ed0101220 msgr2=0x7f7ed0070820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:05.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.066+0000 7f7ed6200640 1 --2- 172.21.15.90:0/2215016861 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7ed0101220 0x7f7ed0070820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:05.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.066+0000 7f7ed6200640 1 -- 172.21.15.90:0/2215016861 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7ed0100860 msgr2=0x7f7ed0073470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:05.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.066+0000 7f7ed6200640 1 --2- 172.21.15.90:0/2215016861 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7ed0100860 0x7f7ed0073470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:05.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.066+0000 7f7ed6200640 1 -- 172.21.15.90:0/2215016861 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ed00712a0 con 0x7f7ed0106870 2024-04-02T14:39:05.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.066+0000 7f7ed6200640 1 --2- 172.21.15.90:0/2215016861 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7ed0106870 0x7f7ed0070d60 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f7ec8009fd0 tx=0x7f7ec800ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:39:05.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.067+0000 7f7ece000640 1 -- 172.21.15.90:0/2215016861 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ec800ce20 con 0x7f7ed0106870 2024-04-02T14:39:05.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.067+0000 7f7ece000640 1 -- 172.21.15.90:0/2215016861 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7ec8005020 con 0x7f7ed0106870 2024-04-02T14:39:05.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.067+0000 7f7ece000640 1 -- 172.21.15.90:0/2215016861 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ec8010720 con 0x7f7ed0106870 2024-04-02T14:39:05.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.067+0000 7f7ed6c00640 1 -- 172.21.15.90:0/2215016861 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ed0071590 con 0x7f7ed0106870 2024-04-02T14:39:05.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.067+0000 7f7ed6c00640 1 -- 172.21.15.90:0/2215016861 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ed010a9b0 con 0x7f7ed0106870 2024-04-02T14:39:05.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.071+0000 7f7edae37740 1 Processor -- start 2024-04-02T14:39:05.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.073+0000 7f7ece000640 1 -- 172.21.15.90:0/2215016861 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f7ec8005540 con 0x7f7ed0106870 2024-04-02T14:39:05.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.073+0000 7f7edae37740 1 -- start start 2024-04-02T14:39:05.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.074+0000 7f7edae37740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f347d09ba0 0x55f347d088f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:05.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.074+0000 7f7edae37740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f347d08ec0 0x55f347d09340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:05.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.074+0000 7f7edae37740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f347d12230 0x55f347d14620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:05.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.074+0000 7f7edae37740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55f347c49400 con 0x55f347d12230 2024-04-02T14:39:05.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.074+0000 7f7edae37740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55f347c2df10 con 0x55f347d08ec0 2024-04-02T14:39:05.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.074+0000 7f7edae37740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55f347c6d6f0 con 0x55f347d09ba0 2024-04-02T14:39:05.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.076+0000 7f7ed4e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f347d09ba0 0x55f347d088f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:05.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.076+0000 7f7ed4e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f347d12230 0x55f347d14620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:05.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.076+0000 7f7ece000640 1 --2- 172.21.15.90:0/2215016861 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f7eb80a3de0 0x7f7eb80a62a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:05.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.076+0000 7f7ed4e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f347d09ba0 0x55f347d088f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:54876/0 (socket says 172.21.15.90:54876) 2024-04-02T14:39:05.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.076+0000 7f7ed4e00640 1 -- 172.21.15.90:0/2621283589 learned_addr learned my addr 172.21.15.90:0/2621283589 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:39:05.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.076+0000 7f7ece000640 1 -- 172.21.15.90:0/2215016861 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f7ec80d0ec0 con 0x7f7ed0106870 2024-04-02T14:39:05.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.076+0000 7f7ed5800640 1 --2- 172.21.15.90:0/2621283589 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f347d08ec0 0x55f347d09340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:05.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.077+0000 7f7ed4e00640 1 -- 172.21.15.90:0/2621283589 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f347d08ec0 msgr2=0x55f347d09340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:39:05.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.077+0000 7f7ed4e00640 1 --2- 172.21.15.90:0/2621283589 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f347d08ec0 0x55f347d09340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:05.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.077+0000 7f7ed4e00640 1 -- 172.21.15.90:0/2621283589 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f347d12230 msgr2=0x55f347d14620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:39:05.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.077+0000 7f7ed4e00640 1 --2- 172.21.15.90:0/2621283589 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f347d12230 0x55f347d14620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:05.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.077+0000 7f7ed4e00640 1 -- 172.21.15.90:0/2621283589 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f3479813f0 con 0x55f347d09ba0 2024-04-02T14:39:05.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.077+0000 7f7ed4e00640 1 --2- 172.21.15.90:0/2621283589 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f347d09ba0 0x55f347d088f0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f7ebc000c00 tx=0x7f7ebc009260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=db9359325f4011d1 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:39:05.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.078+0000 7f7ec2a00640 1 -- 172.21.15.90:0/2621283589 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ebc0049b0 con 0x55f347d09ba0 2024-04-02T14:39:05.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.078+0000 7f7ec2a00640 1 -- 172.21.15.90:0/2621283589 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7ebc032070 con 0x55f347d09ba0 2024-04-02T14:39:05.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.078+0000 7f7ec2a00640 1 -- 172.21.15.90:0/2621283589 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ebc004f50 con 0x55f347d09ba0 2024-04-02T14:39:05.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.079+0000 7f7edae37740 1 -- 172.21.15.90:0/2621283589 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f347d09ba0 msgr2=0x55f347d088f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:05.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.080+0000 7f7edae37740 1 --2- 172.21.15.90:0/2621283589 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f347d09ba0 0x55f347d088f0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f7ebc000c00 tx=0x7f7ebc009260 comp rx=0 tx=0).stop 2024-04-02T14:39:05.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.082+0000 7f7edae37740 1 -- 172.21.15.90:0/2621283589 shutdown_connections 2024-04-02T14:39:05.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.082+0000 7f7edae37740 1 --2- 172.21.15.90:0/2621283589 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f347d12230 0x55f347d14620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:05.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.082+0000 7f7edae37740 1 --2- 172.21.15.90:0/2621283589 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f347d08ec0 0x55f347d09340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:05.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.082+0000 7f7edae37740 1 --2- 172.21.15.90:0/2621283589 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f347d09ba0 0x55f347d088f0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:05.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.082+0000 7f7edae37740 1 -- 172.21.15.90:0/2621283589 >> 172.21.15.90:0/2621283589 conn(0x55f347ccfa50 msgr2=0x55f347d03910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:39:05.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.082+0000 7f7edae37740 1 -- 172.21.15.90:0/2621283589 shutdown_connections 2024-04-02T14:39:05.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.082+0000 7f7edae37740 1 -- 172.21.15.90:0/2621283589 wait complete. 2024-04-02T14:39:05.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.083+0000 7f7edae37740 1 -- start start 2024-04-02T14:39:05.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.083+0000 7f7edae37740 1 Processor -- start 2024-04-02T14:39:05.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.084+0000 7f7edae37740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f347d08ec0 0x55f347d0f2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:05.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.084+0000 7f7edae37740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f347d09ba0 0x55f347d0f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:05.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.084+0000 7f7edae37740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f347d12230 0x55f347d0fd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:05.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.084+0000 7f7edae37740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55f347c696a0 con 0x55f347d08ec0 2024-04-02T14:39:05.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.084+0000 7f7edae37740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55f347c15980 con 0x55f347d12230 2024-04-02T14:39:05.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.084+0000 7f7ed6200640 1 --2- 172.21.15.90:0/2215016861 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f7eb80a3de0 0x7f7eb80a62a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:05.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.084+0000 7f7edae37740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55f347c28cb0 con 0x55f347d09ba0 2024-04-02T14:39:05.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.085+0000 7f7edae37740 10 client.0 _ll_register_callbacks cb 0x55f347c286a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:39:05.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.085+0000 7f7ed4e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f347d08ec0 0x55f347d0f2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:05.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.085+0000 7f7ed5800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f347d09ba0 0x55f347d0f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:05.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.085+0000 7f7ed5800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f347d09ba0 0x55f347d0f7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:54878/0 (socket says 172.21.15.90:54878) 2024-04-02T14:39:05.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.085+0000 7f7ed5800640 1 -- 172.21.15.90:0/2689559093 learned_addr learned my addr 172.21.15.90:0/2689559093 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:39:05.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.086+0000 7f7ed4e00640 1 --2- 172.21.15.90:0/2689559093 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f347d12230 0x55f347d0fd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:05.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.086+0000 7f7ed6200640 1 --2- 172.21.15.90:0/2215016861 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f7eb80a3de0 0x7f7eb80a62a0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f7ec809cb10 tx=0x7f7ec8005f70 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:39:05.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.086+0000 7f7ed5800640 1 -- 172.21.15.90:0/2689559093 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f347d12230 msgr2=0x55f347d0fd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:39:05.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.086+0000 7f7ed5800640 1 --2- 172.21.15.90:0/2689559093 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f347d12230 0x55f347d0fd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:05.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.086+0000 7f7ed5800640 1 -- 172.21.15.90:0/2689559093 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f347d08ec0 msgr2=0x55f347d0f2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:39:05.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.086+0000 7f7ed5800640 1 --2- 172.21.15.90:0/2689559093 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f347d08ec0 0x55f347d0f2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:05.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.086+0000 7f7ed5800640 1 -- 172.21.15.90:0/2689559093 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f347a4b7d0 con 0x55f347d09ba0 2024-04-02T14:39:05.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.087+0000 7f7ed5800640 1 --2- 172.21.15.90:0/2689559093 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f347d09ba0 0x55f347d0f7f0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f7ec40126d0 tx=0x7f7ec4012b90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=685a31fe51a920b4 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:39:05.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.087+0000 7f7eb7400640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:39:05.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.088+0000 7f7eb7400640 1 -- 172.21.15.90:0/2689559093 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ec4004050 con 0x55f347d09ba0 2024-04-02T14:39:05.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.088+0000 7f7edae37740 1 -- 172.21.15.90:0/2689559093 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7ebc032070 con 0x55f347d09ba0 2024-04-02T14:39:05.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.088+0000 7f7eb7400640 1 -- 172.21.15.90:0/2689559093 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f7ec40041f0 con 0x55f347d09ba0 2024-04-02T14:39:05.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.088+0000 7f7eb7400640 1 -- 172.21.15.90:0/2689559093 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f7ec40043f0 con 0x55f347d09ba0 2024-04-02T14:39:05.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.088+0000 7f7edae37740 10 client.26608 fetch_fsmap learned FSMap version 27 2024-04-02T14:39:05.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.088+0000 7f7edae37740 1 -- 172.21.15.90:0/2689559093 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55f347d21180 con 0x55f347d09ba0 2024-04-02T14:39:05.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.089+0000 7f7eb7400640 1 -- 172.21.15.90:0/2689559093 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7ebc032070 con 0x55f347d09ba0 2024-04-02T14:39:05.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.089+0000 7f7eb7400640 1 -- 172.21.15.90:0/2689559093 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f7ec4003bc0 con 0x55f347d09ba0 2024-04-02T14:39:05.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.090+0000 7f7edae37740 10 client.26608 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:39:05.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.090+0000 7f7edae37740 10 client.26608 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2024-04-02T14:39:05.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.090+0000 7f7edae37740 20 client.26608 populate_metadata read hostname 'smithi090' 2024-04-02T14:39:05.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.090+0000 7f7edae37740 1 --2- 172.21.15.90:0/2689559093 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x55f347d22590 0x55f347d24980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:05.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.090+0000 7f7edae37740 4 client.26608 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:39:05.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.090+0000 7f7edae37740 1 -- 172.21.15.90:0/2689559093 --> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f347c2d040 con 0x55f347d22590 2024-04-02T14:39:05.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.091+0000 7f7ed4e00640 1 --2- 172.21.15.90:0/2689559093 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x55f347d22590 0x55f347d24980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:05.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.092+0000 7f7eb7400640 10 client.26608 ms_handle_connect on v2:172.21.15.90:6838/1498915870 2024-04-02T14:39:05.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.092+0000 7f7ed4e00640 1 --2- 172.21.15.90:0/2689559093 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x55f347d22590 0x55f347d24980 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=9e77f411b094314c server_cookie=28589ddd8266a736 in_seq=0 out_seq=0 2024-04-02T14:39:05.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.095+0000 7f7eb7400640 1 -- 172.21.15.90:0/2689559093 <== mds.1 v2:172.21.15.90:6838/1498915870 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f347c2d040 con 0x55f347d22590 2024-04-02T14:39:05.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.096+0000 7f7eb7400640 10 client.26608 handle_command_reply: tid=0 2024-04-02T14:39:05.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.096+0000 7f7edae37740 1 client.26608 shutdown 2024-04-02T14:39:05.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.096+0000 7f7edae37740 1 -- 172.21.15.90:0/2689559093 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f347d09ba0 msgr2=0x55f347d0f7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:05.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.096+0000 7f7edae37740 1 --2- 172.21.15.90:0/2689559093 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f347d09ba0 0x55f347d0f7f0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f7ec40126d0 tx=0x7f7ec4012b90 comp rx=0 tx=0).stop 2024-04-02T14:39:05.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.096+0000 7f7edae37740 1 -- 172.21.15.90:0/2689559093 shutdown_connections 2024-04-02T14:39:05.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.096+0000 7f7edae37740 1 --2- 172.21.15.90:0/2689559093 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x55f347d22590 0x55f347d24980 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:05.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.097+0000 7f7edae37740 1 --2- 172.21.15.90:0/2689559093 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f347d12230 0x55f347d0fd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:05.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.097+0000 7f7eb7400640 0 client.26608 ms_handle_reset on v2:172.21.15.90:6838/1498915870 2024-04-02T14:39:05.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.097+0000 7f7edae37740 1 --2- 172.21.15.90:0/2689559093 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f347d09ba0 0x55f347d0f7f0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:05.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.097+0000 7f7edae37740 1 --2- 172.21.15.90:0/2689559093 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f347d08ec0 0x55f347d0f2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:05.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.097+0000 7f7edae37740 1 -- 172.21.15.90:0/2689559093 >> 172.21.15.90:0/2689559093 conn(0x55f347ccfa50 msgr2=0x55f347d04dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:39:05.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.097+0000 7f7edae37740 1 -- 172.21.15.90:0/2689559093 shutdown_connections 2024-04-02T14:39:05.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.098+0000 7f7edae37740 1 -- 172.21.15.90:0/2689559093 wait complete. 2024-04-02T14:39:05.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.098+0000 7f7edae37740 20 client.26608 trim_cache size 0 max 16384 2024-04-02T14:39:05.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.128+0000 7f7edae37740 1 Processor -- start 2024-04-02T14:39:05.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.128+0000 7f7edae37740 1 -- start start 2024-04-02T14:39:05.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.128+0000 7f7edae37740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f347d08ec0 0x55f347d119e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:05.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.128+0000 7f7edae37740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f347d09ba0 0x55f347d1dc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:05.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.128+0000 7f7edae37740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f347d12230 0x55f347d05470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:05.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.128+0000 7f7edae37740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f7ed01adbf0 con 0x55f347d12230 2024-04-02T14:39:05.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.128+0000 7f7edae37740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f7ed01c6780 con 0x55f347d08ec0 2024-04-02T14:39:05.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.128+0000 7f7edae37740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f7ed01ae2c0 con 0x55f347d09ba0 2024-04-02T14:39:05.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.129+0000 7f7ed5800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f347d09ba0 0x55f347d1dc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:05.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.129+0000 7f7ed5800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f347d09ba0 0x55f347d1dc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:54882/0 (socket says 172.21.15.90:54882) 2024-04-02T14:39:05.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.129+0000 7f7ed5800640 1 -- 172.21.15.90:0/3687632715 learned_addr learned my addr 172.21.15.90:0/3687632715 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:39:05.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.129+0000 7f7ed4e00640 1 --2- 172.21.15.90:0/3687632715 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f347d08ec0 0x55f347d119e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:05.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.129+0000 7f7ed4e00640 1 --2- 172.21.15.90:0/3687632715 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f347d12230 0x55f347d05470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:05.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.129+0000 7f7ed5800640 1 -- 172.21.15.90:0/3687632715 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f347d08ec0 msgr2=0x55f347d119e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:39:05.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.129+0000 7f7ed5800640 1 --2- 172.21.15.90:0/3687632715 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f347d08ec0 0x55f347d119e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:05.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.129+0000 7f7ed5800640 1 -- 172.21.15.90:0/3687632715 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f347d12230 msgr2=0x55f347d05470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:39:05.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.129+0000 7f7ed5800640 1 --2- 172.21.15.90:0/3687632715 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f347d12230 0x55f347d05470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:05.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.129+0000 7f7ed5800640 1 -- 172.21.15.90:0/3687632715 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f347d0e210 con 0x55f347d09ba0 2024-04-02T14:39:05.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.130+0000 7f7ed4e00640 1 --2- 172.21.15.90:0/3687632715 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f347d12230 0x55f347d05470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:39:05.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.130+0000 7f7ed5800640 1 --2- 172.21.15.90:0/3687632715 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f347d09ba0 0x55f347d1dc10 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f7ec40105f0 tx=0x7f7ec40123a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a5e7cee613e437da server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:39:05.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.130+0000 7f7ec2000640 1 -- 172.21.15.90:0/3687632715 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ec4032070 con 0x55f347d09ba0 2024-04-02T14:39:05.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.130+0000 7f7ec2000640 1 -- 172.21.15.90:0/3687632715 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7ec400fe00 con 0x55f347d09ba0 2024-04-02T14:39:05.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.130+0000 7f7ec2000640 1 -- 172.21.15.90:0/3687632715 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ec4029040 con 0x55f347d09ba0 2024-04-02T14:39:05.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.132+0000 7f7edae37740 1 -- 172.21.15.90:0/3687632715 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f347d09ba0 msgr2=0x55f347d1dc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:05.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.132+0000 7f7edae37740 1 --2- 172.21.15.90:0/3687632715 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f347d09ba0 0x55f347d1dc10 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f7ec40105f0 tx=0x7f7ec40123a0 comp rx=0 tx=0).stop 2024-04-02T14:39:05.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.132+0000 7f7edae37740 1 -- 172.21.15.90:0/3687632715 shutdown_connections 2024-04-02T14:39:05.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.132+0000 7f7edae37740 1 --2- 172.21.15.90:0/3687632715 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f347d12230 0x55f347d05470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:05.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.132+0000 7f7edae37740 1 --2- 172.21.15.90:0/3687632715 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f347d09ba0 0x55f347d1dc10 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:05.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.132+0000 7f7edae37740 1 --2- 172.21.15.90:0/3687632715 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f347d08ec0 0x55f347d119e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:05.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.132+0000 7f7edae37740 1 -- 172.21.15.90:0/3687632715 >> 172.21.15.90:0/3687632715 conn(0x55f347ccfa50 msgr2=0x55f347d04d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:39:05.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.133+0000 7f7edae37740 1 -- 172.21.15.90:0/3687632715 shutdown_connections 2024-04-02T14:39:05.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.133+0000 7f7edae37740 1 -- 172.21.15.90:0/3687632715 wait complete. 2024-04-02T14:39:05.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.133+0000 7f7edae37740 1 -- start start 2024-04-02T14:39:05.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.133+0000 7f7edae37740 1 Processor -- start 2024-04-02T14:39:05.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.134+0000 7f7edae37740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f347d08ec0 0x55f347d1a120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:05.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.134+0000 7f7edae37740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f347d09ba0 0x55f347d1a660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:05.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.135+0000 7f7edae37740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f347d12230 0x55f347d0b540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:05.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.135+0000 7f7edae37740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f7ed01c60b0 con 0x55f347d08ec0 2024-04-02T14:39:05.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.135+0000 7f7edae37740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f7ed01c59e0 con 0x55f347d09ba0 2024-04-02T14:39:05.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.135+0000 7f7edae37740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f7ed01abbd0 con 0x55f347d12230 2024-04-02T14:39:05.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.135+0000 7f7edae37740 10 client.0 _ll_register_callbacks cb 0x7f7ec4009e10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:39:05.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.135+0000 7f7ed4e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f347d08ec0 0x55f347d1a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:05.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.135+0000 7f7ed5800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f347d09ba0 0x55f347d1a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:05.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.135+0000 7f7ed4e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f347d08ec0 0x55f347d1a120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:49740/0 (socket says 172.21.15.90:49740) 2024-04-02T14:39:05.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.135+0000 7f7ed4e00640 1 -- 172.21.15.90:0/378055117 learned_addr learned my addr 172.21.15.90:0/378055117 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:39:05.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.136+0000 7f7ed4e00640 1 -- 172.21.15.90:0/378055117 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f347d12230 msgr2=0x55f347d0b540 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-02T14:39:05.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.136+0000 7f7ed4e00640 1 --2- 172.21.15.90:0/378055117 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f347d12230 0x55f347d0b540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:05.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.136+0000 7f7ed4e00640 1 -- 172.21.15.90:0/378055117 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f347d09ba0 msgr2=0x55f347d1a660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:39:05.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.136+0000 7f7ed4e00640 1 --2- 172.21.15.90:0/378055117 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f347d09ba0 0x55f347d1a660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:05.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.136+0000 7f7ed4e00640 1 -- 172.21.15.90:0/378055117 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f347d0bd40 con 0x55f347d08ec0 2024-04-02T14:39:05.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.136+0000 7f7ed5800640 1 --2- 172.21.15.90:0/378055117 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f347d09ba0 0x55f347d1a660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:39:05.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.137+0000 7f7ed4e00640 1 --2- 172.21.15.90:0/378055117 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f347d08ec0 0x55f347d1a120 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f7ebc02e1c0 tx=0x7f7ebc00d3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f410efef37c8d13 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:39:05.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.137+0000 7f7eb7400640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-02T14:39:05.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.139+0000 7f7eb7400640 1 -- 172.21.15.90:0/378055117 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ebc02e750 con 0x55f347d08ec0 2024-04-02T14:39:05.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.139+0000 7f7eb7400640 1 -- 172.21.15.90:0/378055117 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f7ebc02e8f0 con 0x55f347d08ec0 2024-04-02T14:39:05.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.139+0000 7f7edae37740 1 -- 172.21.15.90:0/378055117 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7ec400fe00 con 0x55f347d08ec0 2024-04-02T14:39:05.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.141+0000 7f7eb7400640 1 -- 172.21.15.90:0/378055117 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f7ebc004af0 con 0x55f347d08ec0 2024-04-02T14:39:05.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.141+0000 7f7edae37740 10 client.16236 fetch_fsmap learned FSMap version 27 2024-04-02T14:39:05.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.141+0000 7f7edae37740 1 -- 172.21.15.90:0/378055117 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55f347d0cf20 con 0x55f347d08ec0 2024-04-02T14:39:05.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.141+0000 7f7eb7400640 1 -- 172.21.15.90:0/378055117 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7ec400fe00 con 0x55f347d08ec0 2024-04-02T14:39:05.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.141+0000 7f7eb7400640 1 -- 172.21.15.90:0/378055117 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f7ebc050040 con 0x55f347d08ec0 2024-04-02T14:39:05.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.141+0000 7f7edae37740 10 client.16236 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:39:05.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.141+0000 7f7edae37740 10 client.16236 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2024-04-02T14:39:05.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.141+0000 7f7edae37740 20 client.16236 populate_metadata read hostname 'smithi090' 2024-04-02T14:39:05.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.141+0000 7f7edae37740 1 --2- 172.21.15.90:0/378055117 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x55f347d22590 0x55f347d8ba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:05.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.141+0000 7f7edae37740 4 client.16236 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-02T14:39:05.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.141+0000 7f7edae37740 1 -- 172.21.15.90:0/378055117 --> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55f347c7ac30 con 0x55f347d22590 2024-04-02T14:39:05.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.142+0000 7f7ed5800640 1 --2- 172.21.15.90:0/378055117 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x55f347d22590 0x55f347d8ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:05.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.143+0000 7f7ed5800640 1 --2- 172.21.15.90:0/378055117 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x55f347d22590 0x55f347d8ba70 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=872588f8c5f31780 server_cookie=bcef7819a7b0d482 in_seq=0 out_seq=0 2024-04-02T14:39:05.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.143+0000 7f7eb7400640 10 client.16236 ms_handle_connect on v2:172.21.15.90:6838/1498915870 2024-04-02T14:39:05.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.143+0000 7f7eb7400640 1 -- 172.21.15.90:0/378055117 <== mds.1 v2:172.21.15.90:6838/1498915870 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55f347c7ac30 con 0x55f347d22590 2024-04-02T14:39:05.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.143+0000 7f7eb7400640 10 client.16236 handle_command_reply: tid=0 2024-04-02T14:39:05.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.143+0000 7f7edae37740 1 client.16236 shutdown 2024-04-02T14:39:05.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.143+0000 7f7edae37740 1 -- 172.21.15.90:0/378055117 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f347d08ec0 msgr2=0x55f347d1a120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:05.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.143+0000 7f7edae37740 1 --2- 172.21.15.90:0/378055117 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f347d08ec0 0x55f347d1a120 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f7ebc02e1c0 tx=0x7f7ebc00d3a0 comp rx=0 tx=0).stop 2024-04-02T14:39:05.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.143+0000 7f7edae37740 1 -- 172.21.15.90:0/378055117 shutdown_connections 2024-04-02T14:39:05.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.143+0000 7f7edae37740 1 --2- 172.21.15.90:0/378055117 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x55f347d22590 0x55f347d8ba70 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:05.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.143+0000 7f7edae37740 1 --2- 172.21.15.90:0/378055117 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f347d12230 0x55f347d0b540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:05.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.143+0000 7f7edae37740 1 --2- 172.21.15.90:0/378055117 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f347d09ba0 0x55f347d1a660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:05.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.143+0000 7f7edae37740 1 --2- 172.21.15.90:0/378055117 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f347d08ec0 0x55f347d1a120 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:05.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.143+0000 7f7edae37740 1 -- 172.21.15.90:0/378055117 >> 172.21.15.90:0/378055117 conn(0x55f347ccfa50 msgr2=0x55f347d02260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:39:05.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.143+0000 7f7eb7400640 0 client.16236 ms_handle_reset on v2:172.21.15.90:6838/1498915870 2024-04-02T14:39:05.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.144+0000 7f7edae37740 1 -- 172.21.15.90:0/378055117 shutdown_connections 2024-04-02T14:39:05.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.144+0000 7f7edae37740 1 -- 172.21.15.90:0/378055117 wait complete. 2024-04-02T14:39:05.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.144+0000 7f7edae37740 20 client.16236 trim_cache size 0 max 16384 2024-04-02T14:39:05.148 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-02T14:39:05.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.145+0000 7f7ed6c00640 1 -- 172.21.15.90:0/2215016861 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f7eb80a3de0 msgr2=0x7f7eb80a62a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:05.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.145+0000 7f7ed6c00640 1 --2- 172.21.15.90:0/2215016861 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f7eb80a3de0 0x7f7eb80a62a0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f7ec809cb10 tx=0x7f7ec8005f70 comp rx=0 tx=0).stop 2024-04-02T14:39:05.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.145+0000 7f7ed6c00640 1 -- 172.21.15.90:0/2215016861 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7ed0106870 msgr2=0x7f7ed0070d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:05.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.145+0000 7f7ed6c00640 1 --2- 172.21.15.90:0/2215016861 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7ed0106870 0x7f7ed0070d60 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f7ec8009fd0 tx=0x7f7ec800ef10 comp rx=0 tx=0).stop 2024-04-02T14:39:05.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.145+0000 7f7ed6c00640 1 -- 172.21.15.90:0/2215016861 shutdown_connections 2024-04-02T14:39:05.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.145+0000 7f7ed6c00640 1 --2- 172.21.15.90:0/2215016861 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f7eb80a3de0 0x7f7eb80a62a0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:05.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.145+0000 7f7ed6c00640 1 --2- 172.21.15.90:0/2215016861 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7ed0106870 0x7f7ed0070d60 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:05.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.145+0000 7f7ed6c00640 1 --2- 172.21.15.90:0/2215016861 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7ed0101220 0x7f7ed0070820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:05.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.145+0000 7f7ed6c00640 1 --2- 172.21.15.90:0/2215016861 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7ed0100860 0x7f7ed0073470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:05.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.145+0000 7f7ed6c00640 1 -- 172.21.15.90:0/2215016861 >> 172.21.15.90:0/2215016861 conn(0x7f7ed00fc350 msgr2=0x7f7ed00fc990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:39:05.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.145+0000 7f7ed6c00640 1 -- 172.21.15.90:0/2215016861 shutdown_connections 2024-04-02T14:39:05.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.145+0000 7f7ed6c00640 1 -- 172.21.15.90:0/2215016861 wait complete. 2024-04-02T14:39:05.171 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-02T14:39:05.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.496+0000 7f5c5e600640 1 -- 172.21.15.90:0/3816422487 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5c58102b30 msgr2=0x7f5c58102f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:05.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.496+0000 7f5c5e600640 1 --2- 172.21.15.90:0/3816422487 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5c58102b30 0x7f5c58102f10 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f5c4c009900 tx=0x7f5c4c02bcd0 comp rx=0 tx=0).stop 2024-04-02T14:39:05.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.496+0000 7f5c5e600640 1 -- 172.21.15.90:0/3816422487 shutdown_connections 2024-04-02T14:39:05.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.496+0000 7f5c5e600640 1 --2- 172.21.15.90:0/3816422487 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5c5810bf50 0x7f5c5810e3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:05.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.496+0000 7f5c5e600640 1 --2- 172.21.15.90:0/3816422487 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5c58103450 0x7f5c5810b930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:05.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.496+0000 7f5c5e600640 1 --2- 172.21.15.90:0/3816422487 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5c58102b30 0x7f5c58102f10 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:05.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.496+0000 7f5c5e600640 1 -- 172.21.15.90:0/3816422487 >> 172.21.15.90:0/3816422487 conn(0x7f5c580fc350 msgr2=0x7f5c580fe770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:39:05.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.496+0000 7f5c5e600640 1 -- 172.21.15.90:0/3816422487 shutdown_connections 2024-04-02T14:39:05.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.496+0000 7f5c5e600640 1 -- 172.21.15.90:0/3816422487 wait complete. 2024-04-02T14:39:05.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.498+0000 7f5c5e600640 1 Processor -- start 2024-04-02T14:39:05.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.498+0000 7f5c5e600640 1 -- start start 2024-04-02T14:39:05.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.499+0000 7f5c5e600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5c58102b30 0x7f5c5819a3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:05.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.499+0000 7f5c5e600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5c58103450 0x7f5c5819a930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:05.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.499+0000 7f5c5e600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5c5810bf50 0x7f5c5819ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:05.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.499+0000 7f5c5e600640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f5c58111da0 con 0x7f5c58103450 2024-04-02T14:39:05.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.499+0000 7f5c5e600640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f5c58111c20 con 0x7f5c58102b30 2024-04-02T14:39:05.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.499+0000 7f5c5e600640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f5c58111f20 con 0x7f5c5810bf50 2024-04-02T14:39:05.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.499+0000 7f5c57400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5c58102b30 0x7f5c5819a3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:05.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.499+0000 7f5c57400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5c58102b30 0x7f5c5819a3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:41714/0 (socket says 172.21.15.90:41714) 2024-04-02T14:39:05.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.499+0000 7f5c57400640 1 -- 172.21.15.90:0/1358858411 learned_addr learned my addr 172.21.15.90:0/1358858411 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:39:05.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.499+0000 7f5c57e00640 1 --2- 172.21.15.90:0/1358858411 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5c5810bf50 0x7f5c5819ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:05.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.500+0000 7f5c56a00640 1 --2- 172.21.15.90:0/1358858411 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5c58103450 0x7f5c5819a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:05.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.500+0000 7f5c57e00640 1 -- 172.21.15.90:0/1358858411 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5c58102b30 msgr2=0x7f5c5819a3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:05.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.500+0000 7f5c57e00640 1 --2- 172.21.15.90:0/1358858411 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5c58102b30 0x7f5c5819a3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:05.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.500+0000 7f5c57e00640 1 -- 172.21.15.90:0/1358858411 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5c58103450 msgr2=0x7f5c5819a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:05.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.500+0000 7f5c57e00640 1 --2- 172.21.15.90:0/1358858411 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5c58103450 0x7f5c5819a930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:05.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.500+0000 7f5c57e00640 1 -- 172.21.15.90:0/1358858411 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c5819f3e0 con 0x7f5c5810bf50 2024-04-02T14:39:05.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.501+0000 7f5c57e00640 1 --2- 172.21.15.90:0/1358858411 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5c5810bf50 0x7f5c5819ecb0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f5c50001440 tx=0x7f5c5000dd30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:39:05.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.501+0000 7f5c4be00640 1 -- 172.21.15.90:0/1358858411 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c50015070 con 0x7f5c5810bf50 2024-04-02T14:39:05.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.501+0000 7f5c4be00640 1 -- 172.21.15.90:0/1358858411 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5c5000a030 con 0x7f5c5810bf50 2024-04-02T14:39:05.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.501+0000 7f5c4be00640 1 -- 172.21.15.90:0/1358858411 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c50010600 con 0x7f5c5810bf50 2024-04-02T14:39:05.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.504+0000 7f5c5e600640 1 -- 172.21.15.90:0/1358858411 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5c5819f6d0 con 0x7f5c5810bf50 2024-04-02T14:39:05.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.504+0000 7f5c5e600640 1 -- 172.21.15.90:0/1358858411 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5c581a6f60 con 0x7f5c5810bf50 2024-04-02T14:39:05.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.509+0000 7f5c613d3740 1 Processor -- start 2024-04-02T14:39:05.514 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.511+0000 7f5c4be00640 1 -- 172.21.15.90:0/1358858411 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f5c50018070 con 0x7f5c5810bf50 2024-04-02T14:39:05.514 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.512+0000 7f5c613d3740 1 -- start start 2024-04-02T14:39:05.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.513+0000 7f5c613d3740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e56efe03b0 0x55e56efdf100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:05.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.513+0000 7f5c613d3740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e56efdf6d0 0x55e56efdfb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:05.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.513+0000 7f5c613d3740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e56efe8a40 0x55e56efeae30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:05.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.513+0000 7f5c613d3740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55e56ef17400 con 0x55e56efdf6d0 2024-04-02T14:39:05.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.513+0000 7f5c613d3740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55e56eefbf10 con 0x55e56efe03b0 2024-04-02T14:39:05.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.513+0000 7f5c613d3740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55e56ef3b6f0 con 0x55e56efe8a40 2024-04-02T14:39:05.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.514+0000 7f5c56a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e56efe8a40 0x55e56efeae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:05.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.514+0000 7f5c56a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e56efe8a40 0x55e56efeae30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:54906/0 (socket says 172.21.15.90:54906) 2024-04-02T14:39:05.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.514+0000 7f5c56a00640 1 -- 172.21.15.90:0/1479318188 learned_addr learned my addr 172.21.15.90:0/1479318188 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:39:05.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.515+0000 7f5c56a00640 1 --2- 172.21.15.90:0/1479318188 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e56efe03b0 0x55e56efdf100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:05.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.515+0000 7f5c4be00640 1 --2- 172.21.15.90:0/1358858411 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f5c400a3da0 0x7f5c400a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:05.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.515+0000 7f5c57400640 1 --2- 172.21.15.90:0/1479318188 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e56efdf6d0 0x55e56efdfb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:05.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.515+0000 7f5c4be00640 1 -- 172.21.15.90:0/1358858411 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f5c500d0660 con 0x7f5c5810bf50 2024-04-02T14:39:05.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.516+0000 7f5c57e00640 1 --2- 172.21.15.90:0/1358858411 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f5c400a3da0 0x7f5c400a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:05.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.516+0000 7f5c56a00640 1 -- 172.21.15.90:0/1479318188 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e56efe03b0 msgr2=0x55e56efdf100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:39:05.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.516+0000 7f5c56a00640 1 --2- 172.21.15.90:0/1479318188 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e56efe03b0 0x55e56efdf100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:05.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.516+0000 7f5c56a00640 1 -- 172.21.15.90:0/1479318188 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e56efdf6d0 msgr2=0x55e56efdfb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:39:05.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.516+0000 7f5c56a00640 1 --2- 172.21.15.90:0/1479318188 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e56efdf6d0 0x55e56efdfb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:05.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.517+0000 7f5c56a00640 1 -- 172.21.15.90:0/1479318188 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e56ec4f3f0 con 0x55e56efe8a40 2024-04-02T14:39:05.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.517+0000 7f5c56a00640 1 --2- 172.21.15.90:0/1479318188 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e56efe8a40 0x55e56efeae30 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f5c440124f0 tx=0x7f5c4400d420 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5a65ed2449d2626 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:39:05.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.517+0000 7f5c57e00640 1 --2- 172.21.15.90:0/1358858411 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f5c400a3da0 0x7f5c400a6260 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f5c500cddc0 tx=0x7f5c50021fc0 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:39:05.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.518+0000 7f5c48c00640 1 -- 172.21.15.90:0/1479318188 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c44005220 con 0x55e56efe8a40 2024-04-02T14:39:05.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.518+0000 7f5c48c00640 1 -- 172.21.15.90:0/1479318188 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5c44004c80 con 0x55e56efe8a40 2024-04-02T14:39:05.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.518+0000 7f5c48c00640 1 -- 172.21.15.90:0/1479318188 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c44010bd0 con 0x55e56efe8a40 2024-04-02T14:39:05.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.522+0000 7f5c613d3740 1 -- 172.21.15.90:0/1479318188 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e56efe8a40 msgr2=0x55e56efeae30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:05.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.522+0000 7f5c613d3740 1 --2- 172.21.15.90:0/1479318188 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e56efe8a40 0x55e56efeae30 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f5c440124f0 tx=0x7f5c4400d420 comp rx=0 tx=0).stop 2024-04-02T14:39:05.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.523+0000 7f5c613d3740 1 -- 172.21.15.90:0/1479318188 shutdown_connections 2024-04-02T14:39:05.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.523+0000 7f5c613d3740 1 --2- 172.21.15.90:0/1479318188 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e56efe8a40 0x55e56efeae30 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:05.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.523+0000 7f5c613d3740 1 --2- 172.21.15.90:0/1479318188 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e56efdf6d0 0x55e56efdfb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:05.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.523+0000 7f5c613d3740 1 --2- 172.21.15.90:0/1479318188 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e56efe03b0 0x55e56efdf100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:05.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.523+0000 7f5c613d3740 1 -- 172.21.15.90:0/1479318188 >> 172.21.15.90:0/1479318188 conn(0x55e56ef9da50 msgr2=0x55e56efda120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:39:05.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.524+0000 7f5c613d3740 1 -- 172.21.15.90:0/1479318188 shutdown_connections 2024-04-02T14:39:05.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.524+0000 7f5c613d3740 1 -- 172.21.15.90:0/1479318188 wait complete. 2024-04-02T14:39:05.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.526+0000 7f5c613d3740 1 -- start start 2024-04-02T14:39:05.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.527+0000 7f5c613d3740 1 Processor -- start 2024-04-02T14:39:05.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.528+0000 7f5c613d3740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e56efdf6d0 0x55e56efe5a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:05.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.529+0000 7f5c613d3740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e56efe03b0 0x55e56efe5fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:05.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.529+0000 7f5c613d3740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e56efe8a40 0x55e56efe64f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:05.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.529+0000 7f5c613d3740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55e56ef376a0 con 0x55e56efdf6d0 2024-04-02T14:39:05.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.529+0000 7f5c613d3740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55e56eee3980 con 0x55e56efe03b0 2024-04-02T14:39:05.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.529+0000 7f5c613d3740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55e56eef6cb0 con 0x55e56efe8a40 2024-04-02T14:39:05.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.530+0000 7f5c613d3740 10 client.0 _ll_register_callbacks cb 0x55e56eef66a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:39:05.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.530+0000 7f5c57400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e56efe03b0 0x55e56efe5fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:05.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.530+0000 7f5c57400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e56efe03b0 0x55e56efe5fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:41734/0 (socket says 172.21.15.90:41734) 2024-04-02T14:39:05.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.530+0000 7f5c57400640 1 -- 172.21.15.90:0/1586202091 learned_addr learned my addr 172.21.15.90:0/1586202091 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:39:05.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.531+0000 7f5c56a00640 1 --2- 172.21.15.90:0/1586202091 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e56efe8a40 0x55e56efe64f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:05.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.531+0000 7f5c57400640 1 -- 172.21.15.90:0/1586202091 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e56efe8a40 msgr2=0x55e56efe64f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:39:05.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.532+0000 7f5c57400640 1 --2- 172.21.15.90:0/1586202091 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e56efe8a40 0x55e56efe64f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:05.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.532+0000 7f5c57400640 1 -- 172.21.15.90:0/1586202091 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e56efdf6d0 msgr2=0x55e56efe5a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:39:05.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.532+0000 7f5c57400640 1 --2- 172.21.15.90:0/1586202091 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e56efdf6d0 0x55e56efe5a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:05.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.532+0000 7f5c57400640 1 -- 172.21.15.90:0/1586202091 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e56ed197d0 con 0x55e56efe03b0 2024-04-02T14:39:05.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.534+0000 7f5c3d600640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:39:05.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.534+0000 7f5c57400640 1 --2- 172.21.15.90:0/1586202091 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e56efe03b0 0x55e56efe5fb0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f5c4c036040 tx=0x7f5c4c02cb50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2121afbc50aaa5a0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:39:05.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.535+0000 7f5c3d600640 1 -- 172.21.15.90:0/1586202091 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c4c044070 con 0x55e56efe03b0 2024-04-02T14:39:05.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.535+0000 7f5c3d600640 1 -- 172.21.15.90:0/1586202091 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f5c4c0046c0 con 0x55e56efe03b0 2024-04-02T14:39:05.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.536+0000 7f5c613d3740 1 -- 172.21.15.90:0/1586202091 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5c44004c80 con 0x55e56efe03b0 2024-04-02T14:39:05.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.540+0000 7f5c3d600640 1 -- 172.21.15.90:0/1586202091 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f5c4c02c730 con 0x55e56efe03b0 2024-04-02T14:39:05.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.540+0000 7f5c613d3740 10 client.26747 fetch_fsmap learned FSMap version 27 2024-04-02T14:39:05.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.540+0000 7f5c613d3740 1 -- 172.21.15.90:0/1586202091 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55e56eff7970 con 0x55e56efe03b0 2024-04-02T14:39:05.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.542+0000 7f5c3d600640 1 -- 172.21.15.90:0/1586202091 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5c44004c80 con 0x55e56efe03b0 2024-04-02T14:39:05.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.543+0000 7f5c3d600640 1 -- 172.21.15.90:0/1586202091 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f5c4c0035f0 con 0x55e56efe03b0 2024-04-02T14:39:05.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.545+0000 7f5c613d3740 10 client.26747 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:39:05.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.545+0000 7f5c613d3740 10 client.26747 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:39:05.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.545+0000 7f5c613d3740 20 client.26747 populate_metadata read hostname 'smithi090' 2024-04-02T14:39:05.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.545+0000 7f5c613d3740 1 --2- 172.21.15.90:0/1586202091 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55e56eff8e30 0x55e56effb220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:05.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.545+0000 7f5c613d3740 4 client.26747 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:39:05.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.545+0000 7f5c613d3740 1 -- 172.21.15.90:0/1586202091 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e56eefb040 con 0x55e56eff8e30 2024-04-02T14:39:05.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.546+0000 7f5c56a00640 1 --2- 172.21.15.90:0/1586202091 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55e56eff8e30 0x55e56effb220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:05.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.547+0000 7f5c56a00640 1 --2- 172.21.15.90:0/1586202091 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55e56eff8e30 0x55e56effb220 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=96e507a1d11e8127 server_cookie=b625a7ffebdd5807 in_seq=0 out_seq=0 2024-04-02T14:39:05.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.547+0000 7f5c3d600640 10 client.26747 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:39:05.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.549+0000 7f5c3d600640 1 -- 172.21.15.90:0/1586202091 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e56eefb040 con 0x55e56eff8e30 2024-04-02T14:39:05.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.549+0000 7f5c3d600640 10 client.26747 handle_command_reply: tid=0 2024-04-02T14:39:05.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.549+0000 7f5c613d3740 1 client.26747 shutdown 2024-04-02T14:39:05.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.551+0000 7f5c613d3740 1 -- 172.21.15.90:0/1586202091 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e56efe03b0 msgr2=0x55e56efe5fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:05.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.551+0000 7f5c613d3740 1 --2- 172.21.15.90:0/1586202091 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e56efe03b0 0x55e56efe5fb0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f5c4c036040 tx=0x7f5c4c02cb50 comp rx=0 tx=0).stop 2024-04-02T14:39:05.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.551+0000 7f5c613d3740 1 -- 172.21.15.90:0/1586202091 shutdown_connections 2024-04-02T14:39:05.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.551+0000 7f5c613d3740 1 --2- 172.21.15.90:0/1586202091 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55e56eff8e30 0x55e56effb220 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:05.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.551+0000 7f5c613d3740 1 --2- 172.21.15.90:0/1586202091 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e56efe8a40 0x55e56efe64f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:05.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.551+0000 7f5c613d3740 1 --2- 172.21.15.90:0/1586202091 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e56efe03b0 0x55e56efe5fb0 secure :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f5c4c036040 tx=0x7f5c4c02cb50 comp rx=0 tx=0).stop 2024-04-02T14:39:05.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.551+0000 7f5c613d3740 1 --2- 172.21.15.90:0/1586202091 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e56efdf6d0 0x55e56efe5a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:05.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.551+0000 7f5c613d3740 1 -- 172.21.15.90:0/1586202091 >> 172.21.15.90:0/1586202091 conn(0x55e56ef9da50 msgr2=0x55e56efdb5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:39:05.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.551+0000 7f5c3d600640 0 client.26747 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:39:05.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.551+0000 7f5c613d3740 1 -- 172.21.15.90:0/1586202091 shutdown_connections 2024-04-02T14:39:05.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.552+0000 7f5c613d3740 1 -- 172.21.15.90:0/1586202091 wait complete. 2024-04-02T14:39:05.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.552+0000 7f5c613d3740 20 client.26747 trim_cache size 0 max 16384 2024-04-02T14:39:05.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.582+0000 7f5c613d3740 1 Processor -- start 2024-04-02T14:39:05.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.582+0000 7f5c613d3740 1 -- start start 2024-04-02T14:39:05.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.582+0000 7f5c613d3740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e56efdf6d0 0x55e56efe81a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:05.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.583+0000 7f5c613d3740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e56efe8a40 0x55e56efe6310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:05.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.583+0000 7f5c613d3740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e56efda360 0x55e56f000a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:05.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.583+0000 7f5c613d3740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f5c581ab5b0 con 0x55e56efdf6d0 2024-04-02T14:39:05.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.583+0000 7f5c613d3740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f5c581ad0f0 con 0x55e56efda360 2024-04-02T14:39:05.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.583+0000 7f5c613d3740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f5c581abc80 con 0x55e56efe8a40 2024-04-02T14:39:05.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.583+0000 7f5c57400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e56efe8a40 0x55e56efe6310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:05.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.583+0000 7f5c57400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e56efe8a40 0x55e56efe6310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:54928/0 (socket says 172.21.15.90:54928) 2024-04-02T14:39:05.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.583+0000 7f5c57400640 1 -- 172.21.15.90:0/1031731030 learned_addr learned my addr 172.21.15.90:0/1031731030 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:39:05.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.583+0000 7f5c56a00640 1 --2- 172.21.15.90:0/1031731030 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e56efdf6d0 0x55e56efe81a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:05.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.584+0000 7f5c57400640 1 -- 172.21.15.90:0/1031731030 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e56efda360 msgr2=0x55e56f000a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:39:05.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.584+0000 7f5c57400640 1 --2- 172.21.15.90:0/1031731030 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e56efda360 0x55e56f000a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:05.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.584+0000 7f5c57400640 1 -- 172.21.15.90:0/1031731030 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e56efdf6d0 msgr2=0x55e56efe81a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:39:05.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.584+0000 7f5c57400640 1 --2- 172.21.15.90:0/1031731030 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e56efdf6d0 0x55e56efe81a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:05.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.584+0000 7f5c57400640 1 -- 172.21.15.90:0/1031731030 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e56efde700 con 0x55e56efe8a40 2024-04-02T14:39:05.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.584+0000 7f5c57400640 1 --2- 172.21.15.90:0/1031731030 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e56efe8a40 0x55e56efe6310 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f5c4c034440 tx=0x7f5c4c02bcd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=63a7624b75b0baca server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:39:05.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.584+0000 7f5c3fe00640 1 -- 172.21.15.90:0/1031731030 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c4c044070 con 0x55e56efe8a40 2024-04-02T14:39:05.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.585+0000 7f5c3fe00640 1 -- 172.21.15.90:0/1031731030 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5c4c0037f0 con 0x55e56efe8a40 2024-04-02T14:39:05.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.585+0000 7f5c3fe00640 1 -- 172.21.15.90:0/1031731030 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c4c0051e0 con 0x55e56efe8a40 2024-04-02T14:39:05.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.585+0000 7f5c613d3740 1 -- 172.21.15.90:0/1031731030 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e56efe8a40 msgr2=0x55e56efe6310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:05.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.585+0000 7f5c613d3740 1 --2- 172.21.15.90:0/1031731030 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e56efe8a40 0x55e56efe6310 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f5c4c034440 tx=0x7f5c4c02bcd0 comp rx=0 tx=0).stop 2024-04-02T14:39:05.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.586+0000 7f5c613d3740 1 -- 172.21.15.90:0/1031731030 shutdown_connections 2024-04-02T14:39:05.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.586+0000 7f5c613d3740 1 --2- 172.21.15.90:0/1031731030 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e56efda360 0x55e56f000a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:05.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.586+0000 7f5c613d3740 1 --2- 172.21.15.90:0/1031731030 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e56efe8a40 0x55e56efe6310 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:05.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.586+0000 7f5c613d3740 1 --2- 172.21.15.90:0/1031731030 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e56efdf6d0 0x55e56efe81a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:05.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.586+0000 7f5c613d3740 1 -- 172.21.15.90:0/1031731030 >> 172.21.15.90:0/1031731030 conn(0x55e56ef9da50 msgr2=0x55e56efd9c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:39:05.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.586+0000 7f5c613d3740 1 -- 172.21.15.90:0/1031731030 shutdown_connections 2024-04-02T14:39:05.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.586+0000 7f5c613d3740 1 -- 172.21.15.90:0/1031731030 wait complete. 2024-04-02T14:39:05.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.586+0000 7f5c613d3740 1 -- start start 2024-04-02T14:39:05.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.587+0000 7f5c613d3740 1 Processor -- start 2024-04-02T14:39:05.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.587+0000 7f5c613d3740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e56efda360 0x55e56f0592d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:05.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.587+0000 7f5c613d3740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e56efdf6d0 0x55e56f059810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:05.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.587+0000 7f5c613d3740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e56efe8a40 0x55e56f059d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:05.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.587+0000 7f5c613d3740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f5c581aca20 con 0x55e56efe8a40 2024-04-02T14:39:05.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.587+0000 7f5c613d3740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f5c581ac350 con 0x55e56efdf6d0 2024-04-02T14:39:05.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.587+0000 7f5c613d3740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f5c581a95b0 con 0x55e56efda360 2024-04-02T14:39:05.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.587+0000 7f5c613d3740 10 client.0 _ll_register_callbacks cb 0x7f5c4c0043e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:39:05.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.588+0000 7f5c57400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e56efdf6d0 0x55e56f059810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:05.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.588+0000 7f5c57400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e56efdf6d0 0x55e56f059810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:41756/0 (socket says 172.21.15.90:41756) 2024-04-02T14:39:05.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.588+0000 7f5c57400640 1 -- 172.21.15.90:0/16500535 learned_addr learned my addr 172.21.15.90:0/16500535 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:39:05.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.588+0000 7f5c56a00640 1 --2- 172.21.15.90:0/16500535 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e56efda360 0x55e56f0592d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:05.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.588+0000 7f5c56a00640 1 --2- 172.21.15.90:0/16500535 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e56efe8a40 0x55e56f059d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:05.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.588+0000 7f5c57400640 1 -- 172.21.15.90:0/16500535 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e56efda360 msgr2=0x55e56f0592d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:39:05.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.588+0000 7f5c57400640 1 --2- 172.21.15.90:0/16500535 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e56efda360 0x55e56f0592d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:05.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.588+0000 7f5c57400640 1 -- 172.21.15.90:0/16500535 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e56efe8a40 msgr2=0x55e56f059d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:39:05.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.588+0000 7f5c57400640 1 --2- 172.21.15.90:0/16500535 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e56efe8a40 0x55e56f059d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:05.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.588+0000 7f5c57400640 1 -- 172.21.15.90:0/16500535 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e56efefa70 con 0x55e56efdf6d0 2024-04-02T14:39:05.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.588+0000 7f5c56a00640 1 --2- 172.21.15.90:0/16500535 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e56efe8a40 0x55e56f059d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:39:05.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.589+0000 7f5c57400640 1 --2- 172.21.15.90:0/16500535 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e56efdf6d0 0x55e56f059810 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f5c4c003ae0 tx=0x7f5c4c005d70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=10212c9e236c11de server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:39:05.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.589+0000 7f5c3d600640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:39:05.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.589+0000 7f5c3d600640 1 -- 172.21.15.90:0/16500535 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c4c044070 con 0x55e56efdf6d0 2024-04-02T14:39:05.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.589+0000 7f5c3d600640 1 -- 172.21.15.90:0/16500535 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f5c4c005190 con 0x55e56efdf6d0 2024-04-02T14:39:05.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.589+0000 7f5c613d3740 1 -- 172.21.15.90:0/16500535 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5c4c0037f0 con 0x55e56efdf6d0 2024-04-02T14:39:05.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.589+0000 7f5c3d600640 1 -- 172.21.15.90:0/16500535 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f5c4c069050 con 0x55e56efdf6d0 2024-04-02T14:39:05.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.590+0000 7f5c613d3740 10 client.26750 fetch_fsmap learned FSMap version 27 2024-04-02T14:39:05.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.590+0000 7f5c613d3740 1 -- 172.21.15.90:0/16500535 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55e56eff0d30 con 0x55e56efdf6d0 2024-04-02T14:39:05.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.590+0000 7f5c3d600640 1 -- 172.21.15.90:0/16500535 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5c4c0037f0 con 0x55e56efdf6d0 2024-04-02T14:39:05.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.590+0000 7f5c3d600640 1 -- 172.21.15.90:0/16500535 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f5c4c0349a0 con 0x55e56efdf6d0 2024-04-02T14:39:05.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.591+0000 7f5c613d3740 10 client.26750 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:39:05.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.591+0000 7f5c613d3740 10 client.26750 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:39:05.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.591+0000 7f5c613d3740 20 client.26750 populate_metadata read hostname 'smithi090' 2024-04-02T14:39:05.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.591+0000 7f5c613d3740 1 --2- 172.21.15.90:0/16500535 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55e56efd6290 0x55e56efd86b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:05.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.591+0000 7f5c613d3740 4 client.26750 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-02T14:39:05.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.591+0000 7f5c613d3740 1 -- 172.21.15.90:0/16500535 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55e56ef48c30 con 0x55e56efd6290 2024-04-02T14:39:05.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.592+0000 7f5c56a00640 1 --2- 172.21.15.90:0/16500535 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55e56efd6290 0x55e56efd86b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:05.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.592+0000 7f5c56a00640 1 --2- 172.21.15.90:0/16500535 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55e56efd6290 0x55e56efd86b0 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=545c169c0231ba39 server_cookie=574bb932fbaa01e2 in_seq=0 out_seq=0 2024-04-02T14:39:05.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.592+0000 7f5c3d600640 10 client.26750 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:39:05.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.593+0000 7f5c3d600640 1 -- 172.21.15.90:0/16500535 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55e56ef48c30 con 0x55e56efd6290 2024-04-02T14:39:05.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.593+0000 7f5c3d600640 10 client.26750 handle_command_reply: tid=0 2024-04-02T14:39:05.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.593+0000 7f5c613d3740 1 client.26750 shutdown 2024-04-02T14:39:05.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.593+0000 7f5c613d3740 1 -- 172.21.15.90:0/16500535 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e56efdf6d0 msgr2=0x55e56f059810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:05.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.593+0000 7f5c613d3740 1 --2- 172.21.15.90:0/16500535 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e56efdf6d0 0x55e56f059810 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f5c4c003ae0 tx=0x7f5c4c005d70 comp rx=0 tx=0).stop 2024-04-02T14:39:05.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.593+0000 7f5c613d3740 1 -- 172.21.15.90:0/16500535 shutdown_connections 2024-04-02T14:39:05.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.594+0000 7f5c613d3740 1 --2- 172.21.15.90:0/16500535 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55e56efd6290 0x55e56efd86b0 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:05.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.594+0000 7f5c613d3740 1 --2- 172.21.15.90:0/16500535 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e56efe8a40 0x55e56f059d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:05.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.594+0000 7f5c613d3740 1 --2- 172.21.15.90:0/16500535 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e56efdf6d0 0x55e56f059810 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:05.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.594+0000 7f5c613d3740 1 --2- 172.21.15.90:0/16500535 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e56efda360 0x55e56f0592d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:05.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.594+0000 7f5c613d3740 1 -- 172.21.15.90:0/16500535 >> 172.21.15.90:0/16500535 conn(0x55e56ef9da50 msgr2=0x55e56efe70f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:39:05.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.594+0000 7f5c3d600640 0 client.26750 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:39:05.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.594+0000 7f5c613d3740 1 -- 172.21.15.90:0/16500535 shutdown_connections 2024-04-02T14:39:05.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.594+0000 7f5c613d3740 1 -- 172.21.15.90:0/16500535 wait complete. 2024-04-02T14:39:05.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.594+0000 7f5c613d3740 20 client.26750 trim_cache size 0 max 16384 2024-04-02T14:39:05.597 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-02T14:39:05.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.595+0000 7f5c5e600640 1 -- 172.21.15.90:0/1358858411 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f5c400a3da0 msgr2=0x7f5c400a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:05.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.595+0000 7f5c5e600640 1 --2- 172.21.15.90:0/1358858411 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f5c400a3da0 0x7f5c400a6260 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f5c500cddc0 tx=0x7f5c50021fc0 comp rx=0 tx=0).stop 2024-04-02T14:39:05.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.595+0000 7f5c5e600640 1 -- 172.21.15.90:0/1358858411 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5c5810bf50 msgr2=0x7f5c5819ecb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:05.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.595+0000 7f5c5e600640 1 --2- 172.21.15.90:0/1358858411 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5c5810bf50 0x7f5c5819ecb0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f5c50001440 tx=0x7f5c5000dd30 comp rx=0 tx=0).stop 2024-04-02T14:39:05.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.598+0000 7f5c5e600640 1 -- 172.21.15.90:0/1358858411 shutdown_connections 2024-04-02T14:39:05.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.598+0000 7f5c5e600640 1 --2- 172.21.15.90:0/1358858411 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f5c400a3da0 0x7f5c400a6260 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:05.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.598+0000 7f5c5e600640 1 --2- 172.21.15.90:0/1358858411 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5c5810bf50 0x7f5c5819ecb0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:05.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.598+0000 7f5c5e600640 1 --2- 172.21.15.90:0/1358858411 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5c58103450 0x7f5c5819a930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:05.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.598+0000 7f5c5e600640 1 --2- 172.21.15.90:0/1358858411 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5c58102b30 0x7f5c5819a3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:05.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.598+0000 7f5c5e600640 1 -- 172.21.15.90:0/1358858411 >> 172.21.15.90:0/1358858411 conn(0x7f5c580fc350 msgr2=0x7f5c580fe740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:39:05.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.598+0000 7f5c5e600640 1 -- 172.21.15.90:0/1358858411 shutdown_connections 2024-04-02T14:39:05.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:05.598+0000 7f5c5e600640 1 -- 172.21.15.90:0/1358858411 wait complete. 2024-04-02T14:39:06.360 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:39:05 smithi092 ceph-mon[36497]: pgmap v129: 97 pgs: 97 active+clean; 2.4 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 19 MiB/s wr, 981 op/s 2024-04-02T14:39:06.436 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:39:05 smithi177 ceph-mon[36431]: pgmap v129: 97 pgs: 97 active+clean; 2.4 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 19 MiB/s wr, 981 op/s 2024-04-02T14:39:06.438 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:39:05 smithi090 ceph-mon[31674]: pgmap v129: 97 pgs: 97 active+clean; 2.4 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 19 MiB/s wr, 981 op/s 2024-04-02T14:39:06.628 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-02T14:39:06.629 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T14:39:06.629 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-02T14:39:06.834 INFO:teuthology.orchestra.run.smithi090.stderr:progress: 215.0 s, 202.8 tps, lat 486.777 ms stddev 476.299 2024-04-02T14:39:06.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.871+0000 7f53b3200640 1 -- 172.21.15.90:0/1281909762 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f53b410d300 msgr2=0x7f53b4114ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:06.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.871+0000 7f53b3200640 1 --2- 172.21.15.90:0/1281909762 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f53b410d300 0x7f53b4114ed0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f53ac009ca0 tx=0x7f53ac02c120 comp rx=0 tx=0).stop 2024-04-02T14:39:06.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.871+0000 7f53b3200640 1 -- 172.21.15.90:0/1281909762 shutdown_connections 2024-04-02T14:39:06.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.871+0000 7f53b3200640 1 --2- 172.21.15.90:0/1281909762 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f53b410d300 0x7f53b4114ed0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:06.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.871+0000 7f53b3200640 1 --2- 172.21.15.90:0/1281909762 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f53b4108c60 0x7f53b410cc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:06.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.871+0000 7f53b3200640 1 --2- 172.21.15.90:0/1281909762 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f53b41082b0 0x7f53b4108690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:06.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.871+0000 7f53b3200640 1 -- 172.21.15.90:0/1281909762 >> 172.21.15.90:0/1281909762 conn(0x7f53b4077920 msgr2=0x7f53b4106920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:39:06.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.872+0000 7f53b3200640 1 -- 172.21.15.90:0/1281909762 shutdown_connections 2024-04-02T14:39:06.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.872+0000 7f53b3200640 1 -- 172.21.15.90:0/1281909762 wait complete. 2024-04-02T14:39:06.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.874+0000 7f53b3200640 1 Processor -- start 2024-04-02T14:39:06.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.874+0000 7f53b3200640 1 -- start start 2024-04-02T14:39:06.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.874+0000 7f53b3200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f53b41082b0 0x7f53b41a4600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:06.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.874+0000 7f53b3200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f53b410d300 0x7f53b41a4b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:06.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.874+0000 7f53b3200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f53b419e710 0x7f53b419eb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:06.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.874+0000 7f53b3200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f53b4118410 con 0x7f53b410d300 2024-04-02T14:39:06.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.874+0000 7f53b3200640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f53b4118290 con 0x7f53b419e710 2024-04-02T14:39:06.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.874+0000 7f53b3200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f53b4118590 con 0x7f53b41082b0 2024-04-02T14:39:06.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.875+0000 7f53b1e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f53b41082b0 0x7f53b41a4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:06.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.875+0000 7f53b1400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f53b410d300 0x7f53b41a4b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:06.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.875+0000 7f53b1400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f53b410d300 0x7f53b41a4b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:49808/0 (socket says 172.21.15.90:49808) 2024-04-02T14:39:06.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.875+0000 7f53b1400640 1 -- 172.21.15.90:0/397102246 learned_addr learned my addr 172.21.15.90:0/397102246 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:39:06.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.875+0000 7f53b2800640 1 --2- 172.21.15.90:0/397102246 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f53b419e710 0x7f53b419eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:06.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.876+0000 7f53b1400640 1 -- 172.21.15.90:0/397102246 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f53b41082b0 msgr2=0x7f53b41a4600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:06.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.876+0000 7f53b1400640 1 --2- 172.21.15.90:0/397102246 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f53b41082b0 0x7f53b41a4600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:06.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.876+0000 7f53b1400640 1 -- 172.21.15.90:0/397102246 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f53b419e710 msgr2=0x7f53b419eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:06.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.876+0000 7f53b1400640 1 --2- 172.21.15.90:0/397102246 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f53b419e710 0x7f53b419eb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:06.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.876+0000 7f53b1400640 1 -- 172.21.15.90:0/397102246 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53b419f310 con 0x7f53b410d300 2024-04-02T14:39:06.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.876+0000 7f53b1400640 1 --2- 172.21.15.90:0/397102246 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f53b410d300 0x7f53b41a4b40 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f53a0006fd0 tx=0x7f53a000b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:39:06.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.877+0000 7f53aaa00640 1 -- 172.21.15.90:0/397102246 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53a0011830 con 0x7f53b410d300 2024-04-02T14:39:06.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.877+0000 7f53aaa00640 1 -- 172.21.15.90:0/397102246 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f53a0016070 con 0x7f53b410d300 2024-04-02T14:39:06.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.877+0000 7f53aaa00640 1 -- 172.21.15.90:0/397102246 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53a000f2e0 con 0x7f53b410d300 2024-04-02T14:39:06.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.878+0000 7f53b3200640 1 -- 172.21.15.90:0/397102246 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53b406b520 con 0x7f53b410d300 2024-04-02T14:39:06.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.878+0000 7f53b3200640 1 -- 172.21.15.90:0/397102246 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53b406b9f0 con 0x7f53b410d300 2024-04-02T14:39:06.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.889+0000 7f53aaa00640 1 -- 172.21.15.90:0/397102246 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f53a001a410 con 0x7f53b410d300 2024-04-02T14:39:06.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.891+0000 7f53aaa00640 1 --2- 172.21.15.90:0/397102246 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f53980a3e70 0x7f53980a6330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:06.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.891+0000 7f53bb7b0740 1 Processor -- start 2024-04-02T14:39:06.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.892+0000 7f53aaa00640 1 -- 172.21.15.90:0/397102246 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f53a00cf440 con 0x7f53b410d300 2024-04-02T14:39:06.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.892+0000 7f53b1e00640 1 --2- 172.21.15.90:0/397102246 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f53980a3e70 0x7f53980a6330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:06.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.892+0000 7f53bb7b0740 1 -- start start 2024-04-02T14:39:06.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.893+0000 7f53bb7b0740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5625a91ae1f0 0x5625a91acf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:06.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.893+0000 7f53bb7b0740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5625a91ad510 0x5625a91ad990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:06.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.893+0000 7f53bb7b0740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5625a91b6880 0x5625a91b8c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:06.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.893+0000 7f53bb7b0740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5625a9111310 con 0x5625a91b6880 2024-04-02T14:39:06.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.893+0000 7f53bb7b0740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x5625a90b9690 con 0x5625a91ae1f0 2024-04-02T14:39:06.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.893+0000 7f53bb7b0740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5625a910d2c0 con 0x5625a91ad510 2024-04-02T14:39:06.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.894+0000 7f53b1e00640 1 --2- 172.21.15.90:0/397102246 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f53980a3e70 0x7f53980a6330 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f53a4005fd0 tx=0x7f53a4009470 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:39:06.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.894+0000 7f53b1e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5625a91ae1f0 0x5625a91acf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:06.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.894+0000 7f53b1e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5625a91ae1f0 0x5625a91acf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:41796/0 (socket says 172.21.15.90:41796) 2024-04-02T14:39:06.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.894+0000 7f53b2800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5625a91b6880 0x5625a91b8c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:06.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.894+0000 7f53b1e00640 1 -- 172.21.15.90:0/409423490 learned_addr learned my addr 172.21.15.90:0/409423490 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:39:06.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.895+0000 7f53b1e00640 1 -- 172.21.15.90:0/409423490 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5625a91ad510 msgr2=0x5625a91ad990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:39:06.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.895+0000 7f53b1e00640 1 --2- 172.21.15.90:0/409423490 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5625a91ad510 0x5625a91ad990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:06.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.895+0000 7f53b1e00640 1 -- 172.21.15.90:0/409423490 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5625a91b6880 msgr2=0x5625a91b8c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:39:06.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.895+0000 7f53b1e00640 1 --2- 172.21.15.90:0/409423490 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5625a91b6880 0x5625a91b8c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:06.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.895+0000 7f53b1e00640 1 -- 172.21.15.90:0/409423490 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5625a8eed7e0 con 0x5625a91ae1f0 2024-04-02T14:39:06.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.895+0000 7f53b1e00640 1 --2- 172.21.15.90:0/409423490 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5625a91ae1f0 0x5625a91acf40 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f53a400ee00 tx=0x7f53a4011000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d644e6c3b2046cef server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:39:06.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.895+0000 7f539f000640 1 -- 172.21.15.90:0/409423490 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53a401e440 con 0x5625a91ae1f0 2024-04-02T14:39:06.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.895+0000 7f539f000640 1 -- 172.21.15.90:0/409423490 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f53a4023070 con 0x5625a91ae1f0 2024-04-02T14:39:06.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.895+0000 7f539f000640 1 -- 172.21.15.90:0/409423490 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53a4010a20 con 0x5625a91ae1f0 2024-04-02T14:39:06.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.896+0000 7f53bb7b0740 1 -- 172.21.15.90:0/409423490 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5625a91ae1f0 msgr2=0x5625a91acf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:06.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.896+0000 7f53bb7b0740 1 --2- 172.21.15.90:0/409423490 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5625a91ae1f0 0x5625a91acf40 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f53a400ee00 tx=0x7f53a4011000 comp rx=0 tx=0).stop 2024-04-02T14:39:06.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.897+0000 7f53bb7b0740 1 -- 172.21.15.90:0/409423490 shutdown_connections 2024-04-02T14:39:06.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.897+0000 7f53bb7b0740 1 --2- 172.21.15.90:0/409423490 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5625a91b6880 0x5625a91b8c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:06.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.897+0000 7f53bb7b0740 1 --2- 172.21.15.90:0/409423490 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5625a91ad510 0x5625a91ad990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:06.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.897+0000 7f53bb7b0740 1 --2- 172.21.15.90:0/409423490 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5625a91ae1f0 0x5625a91acf40 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:06.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.897+0000 7f53bb7b0740 1 -- 172.21.15.90:0/409423490 >> 172.21.15.90:0/409423490 conn(0x5625a9141270 msgr2=0x5625a91a7f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:39:06.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.897+0000 7f53bb7b0740 1 -- 172.21.15.90:0/409423490 shutdown_connections 2024-04-02T14:39:06.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.898+0000 7f53bb7b0740 1 -- 172.21.15.90:0/409423490 wait complete. 2024-04-02T14:39:06.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.898+0000 7f53bb7b0740 1 -- start start 2024-04-02T14:39:06.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.898+0000 7f53bb7b0740 1 Processor -- start 2024-04-02T14:39:06.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.899+0000 7f53bb7b0740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5625a91ad510 0x5625a91b3900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:06.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.899+0000 7f53bb7b0740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5625a91ae1f0 0x5625a91b3e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:06.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.899+0000 7f53bb7b0740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5625a91b6880 0x5625a91b4380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:06.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.899+0000 7f53bb7b0740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5625a91b9690 con 0x5625a91ae1f0 2024-04-02T14:39:06.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.899+0000 7f53bb7b0740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x5625a90cca60 con 0x5625a91b6880 2024-04-02T14:39:06.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.899+0000 7f53bb7b0740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5625a91b9810 con 0x5625a91ad510 2024-04-02T14:39:06.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.899+0000 7f53bb7b0740 10 client.0 _ll_register_callbacks cb 0x5625a90d2030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:39:06.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.900+0000 7f53b2800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5625a91b6880 0x5625a91b4380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:06.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.900+0000 7f53b2800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5625a91b6880 0x5625a91b4380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:41798/0 (socket says 172.21.15.90:41798) 2024-04-02T14:39:06.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.900+0000 7f53b2800640 1 -- 172.21.15.90:0/639173499 learned_addr learned my addr 172.21.15.90:0/639173499 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:39:06.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.900+0000 7f53b1e00640 1 --2- 172.21.15.90:0/639173499 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5625a91ad510 0x5625a91b3900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:06.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.900+0000 7f53b1400640 1 --2- 172.21.15.90:0/639173499 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5625a91ae1f0 0x5625a91b3e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:06.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.900+0000 7f53b2800640 1 -- 172.21.15.90:0/639173499 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5625a91ad510 msgr2=0x5625a91b3900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:39:06.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.900+0000 7f53b2800640 1 --2- 172.21.15.90:0/639173499 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5625a91ad510 0x5625a91b3900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:06.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.900+0000 7f53b2800640 1 -- 172.21.15.90:0/639173499 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5625a91ae1f0 msgr2=0x5625a91b3e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:39:06.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.900+0000 7f53b2800640 1 --2- 172.21.15.90:0/639173499 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5625a91ae1f0 0x5625a91b3e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:06.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.901+0000 7f53b2800640 1 -- 172.21.15.90:0/639173499 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5625a8f4df70 con 0x5625a91b6880 2024-04-02T14:39:06.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.902+0000 7f53b2800640 1 --2- 172.21.15.90:0/639173499 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5625a91b6880 0x5625a91b4380 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f53ac02c730 tx=0x7f53ac032830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c2e9700c4bd3dfe3 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:39:06.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.902+0000 7f5393400640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:39:06.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.902+0000 7f5393400640 1 -- 172.21.15.90:0/639173499 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53ac055070 con 0x5625a91b6880 2024-04-02T14:39:06.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.902+0000 7f5393400640 1 -- 172.21.15.90:0/639173499 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f53ac003b70 con 0x5625a91b6880 2024-04-02T14:39:06.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.902+0000 7f53bb7b0740 1 -- 172.21.15.90:0/639173499 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f53a4023070 con 0x5625a91b6880 2024-04-02T14:39:06.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.902+0000 7f5393400640 1 -- 172.21.15.90:0/639173499 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f53ac05f050 con 0x5625a91b6880 2024-04-02T14:39:06.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.902+0000 7f53bb7b0740 10 client.26759 fetch_fsmap learned FSMap version 27 2024-04-02T14:39:06.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.902+0000 7f53bb7b0740 1 -- 172.21.15.90:0/639173499 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5625a8de5170 con 0x5625a91b6880 2024-04-02T14:39:06.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.903+0000 7f5393400640 1 -- 172.21.15.90:0/639173499 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f53a4023070 con 0x5625a91b6880 2024-04-02T14:39:06.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.903+0000 7f5393400640 1 -- 172.21.15.90:0/639173499 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f53ac04f040 con 0x5625a91b6880 2024-04-02T14:39:06.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.903+0000 7f53bb7b0740 10 client.26759 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:39:06.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.903+0000 7f53bb7b0740 10 client.26759 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:39:06.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.903+0000 7f53bb7b0740 20 client.26759 populate_metadata read hostname 'smithi090' 2024-04-02T14:39:06.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.903+0000 7f53bb7b0740 1 --2- 172.21.15.90:0/639173499 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5625a91c6d90 0x5625a91c91c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:06.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.903+0000 7f53bb7b0740 4 client.26759 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:39:06.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.903+0000 7f53bb7b0740 1 -- 172.21.15.90:0/639173499 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5625a911e850 con 0x5625a91c6d90 2024-04-02T14:39:06.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.903+0000 7f53b1e00640 1 --2- 172.21.15.90:0/639173499 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5625a91c6d90 0x5625a91c91c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:06.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.904+0000 7f53b1e00640 1 --2- 172.21.15.90:0/639173499 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5625a91c6d90 0x5625a91c91c0 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e70cd14bd8d1e175 server_cookie=2678a8f8dd58ac17 in_seq=0 out_seq=0 2024-04-02T14:39:06.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.904+0000 7f5393400640 10 client.26759 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:39:06.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.905+0000 7f5393400640 1 -- 172.21.15.90:0/639173499 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5625a911e850 con 0x5625a91c6d90 2024-04-02T14:39:06.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.905+0000 7f5393400640 10 client.26759 handle_command_reply: tid=0 2024-04-02T14:39:06.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.905+0000 7f53bb7b0740 1 client.26759 shutdown 2024-04-02T14:39:06.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.905+0000 7f53bb7b0740 1 -- 172.21.15.90:0/639173499 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5625a91b6880 msgr2=0x5625a91b4380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:06.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.905+0000 7f53bb7b0740 1 --2- 172.21.15.90:0/639173499 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5625a91b6880 0x5625a91b4380 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f53ac02c730 tx=0x7f53ac032830 comp rx=0 tx=0).stop 2024-04-02T14:39:06.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.906+0000 7f53bb7b0740 1 -- 172.21.15.90:0/639173499 shutdown_connections 2024-04-02T14:39:06.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.906+0000 7f53bb7b0740 1 --2- 172.21.15.90:0/639173499 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5625a91c6d90 0x5625a91c91c0 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:06.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.906+0000 7f53bb7b0740 1 --2- 172.21.15.90:0/639173499 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5625a91b6880 0x5625a91b4380 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:06.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.906+0000 7f53bb7b0740 1 --2- 172.21.15.90:0/639173499 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5625a91ae1f0 0x5625a91b3e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:06.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.906+0000 7f53bb7b0740 1 --2- 172.21.15.90:0/639173499 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5625a91ad510 0x5625a91b3900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:06.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.906+0000 7f53bb7b0740 1 -- 172.21.15.90:0/639173499 >> 172.21.15.90:0/639173499 conn(0x5625a9141270 msgr2=0x5625a91a9cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:39:06.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.906+0000 7f5393400640 0 client.26759 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:39:06.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.906+0000 7f53bb7b0740 1 -- 172.21.15.90:0/639173499 shutdown_connections 2024-04-02T14:39:06.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.906+0000 7f53bb7b0740 1 -- 172.21.15.90:0/639173499 wait complete. 2024-04-02T14:39:06.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.906+0000 7f53bb7b0740 20 client.26759 trim_cache size 0 max 16384 2024-04-02T14:39:06.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.937+0000 7f53bb7b0740 1 Processor -- start 2024-04-02T14:39:06.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.937+0000 7f53bb7b0740 1 -- start start 2024-04-02T14:39:06.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.938+0000 7f53bb7b0740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5625a91ad510 0x5625a91b6030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:06.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.938+0000 7f53bb7b0740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5625a91ae1f0 0x5625a91c2590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:06.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.938+0000 7f53bb7b0740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5625a91b6880 0x5625a91af050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:06.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.938+0000 7f53bb7b0740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f53b41ab4e0 con 0x5625a91ad510 2024-04-02T14:39:06.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.938+0000 7f53bb7b0740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f53b41ad020 con 0x5625a91b6880 2024-04-02T14:39:06.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.938+0000 7f53bb7b0740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f53b41abbb0 con 0x5625a91ae1f0 2024-04-02T14:39:06.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.938+0000 7f53b2800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5625a91b6880 0x5625a91af050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:06.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.938+0000 7f53b2800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5625a91b6880 0x5625a91af050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:41808/0 (socket says 172.21.15.90:41808) 2024-04-02T14:39:06.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.938+0000 7f53b2800640 1 -- 172.21.15.90:0/2446437223 learned_addr learned my addr 172.21.15.90:0/2446437223 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:39:06.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.938+0000 7f53b1400640 1 --2- 172.21.15.90:0/2446437223 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5625a91ae1f0 0x5625a91c2590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:06.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.939+0000 7f53b2800640 1 -- 172.21.15.90:0/2446437223 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5625a91ae1f0 msgr2=0x5625a91c2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:39:06.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.939+0000 7f53b2800640 1 --2- 172.21.15.90:0/2446437223 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5625a91ae1f0 0x5625a91c2590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:06.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.939+0000 7f53b2800640 1 -- 172.21.15.90:0/2446437223 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5625a91ad510 msgr2=0x5625a91b6030 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-02T14:39:06.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.939+0000 7f53b1e00640 1 --2- 172.21.15.90:0/2446437223 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5625a91ad510 0x5625a91b6030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:06.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.939+0000 7f53b2800640 1 --2- 172.21.15.90:0/2446437223 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5625a91ad510 0x5625a91b6030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:06.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.939+0000 7f53b2800640 1 -- 172.21.15.90:0/2446437223 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5625a91a2190 con 0x5625a91b6880 2024-04-02T14:39:06.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.939+0000 7f53b1e00640 1 --2- 172.21.15.90:0/2446437223 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5625a91ad510 0x5625a91b6030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:39:06.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.940+0000 7f53b2800640 1 --2- 172.21.15.90:0/2446437223 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5625a91b6880 0x5625a91af050 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f53ac02c700 tx=0x7f53ac02c7a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=52146601aa199bb9 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:39:06.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.940+0000 7f539e600640 1 -- 172.21.15.90:0/2446437223 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53ac069070 con 0x5625a91b6880 2024-04-02T14:39:06.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.940+0000 7f539e600640 1 -- 172.21.15.90:0/2446437223 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f53ac04a040 con 0x5625a91b6880 2024-04-02T14:39:06.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.940+0000 7f539e600640 1 -- 172.21.15.90:0/2446437223 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53ac055040 con 0x5625a91b6880 2024-04-02T14:39:06.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.941+0000 7f53bb7b0740 1 -- 172.21.15.90:0/2446437223 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5625a91b6880 msgr2=0x5625a91af050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:06.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.941+0000 7f53bb7b0740 1 --2- 172.21.15.90:0/2446437223 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5625a91b6880 0x5625a91af050 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f53ac02c700 tx=0x7f53ac02c7a0 comp rx=0 tx=0).stop 2024-04-02T14:39:06.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.941+0000 7f53bb7b0740 1 -- 172.21.15.90:0/2446437223 shutdown_connections 2024-04-02T14:39:06.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.941+0000 7f53bb7b0740 1 --2- 172.21.15.90:0/2446437223 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5625a91b6880 0x5625a91af050 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:06.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.941+0000 7f53bb7b0740 1 --2- 172.21.15.90:0/2446437223 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5625a91ae1f0 0x5625a91c2590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:06.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.941+0000 7f53bb7b0740 1 --2- 172.21.15.90:0/2446437223 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5625a91ad510 0x5625a91b6030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:06.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.941+0000 7f53bb7b0740 1 -- 172.21.15.90:0/2446437223 >> 172.21.15.90:0/2446437223 conn(0x5625a9141270 msgr2=0x5625a91b20a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:39:06.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.941+0000 7f53bb7b0740 1 -- 172.21.15.90:0/2446437223 shutdown_connections 2024-04-02T14:39:06.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.941+0000 7f53bb7b0740 1 -- 172.21.15.90:0/2446437223 wait complete. 2024-04-02T14:39:06.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.941+0000 7f53bb7b0740 1 -- start start 2024-04-02T14:39:06.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.942+0000 7f53bb7b0740 1 Processor -- start 2024-04-02T14:39:06.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.942+0000 7f53bb7b0740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5625a91ad510 0x5625a922a870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:06.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.943+0000 7f53bb7b0740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5625a91ae1f0 0x5625a922adb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:06.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.943+0000 7f53bb7b0740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5625a91b6880 0x5625a91bec90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:06.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.943+0000 7f53bb7b0740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f53b41ac950 con 0x5625a91ae1f0 2024-04-02T14:39:06.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.943+0000 7f53bb7b0740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f53b41ac280 con 0x5625a91ad510 2024-04-02T14:39:06.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.943+0000 7f53bb7b0740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f53b4071e40 con 0x5625a91b6880 2024-04-02T14:39:06.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.943+0000 7f53bb7b0740 10 client.0 _ll_register_callbacks cb 0x7f53ac037070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:39:06.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.943+0000 7f53b1e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5625a91ad510 0x5625a922a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:06.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.943+0000 7f53b2800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5625a91b6880 0x5625a91bec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:06.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.943+0000 7f53b2800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5625a91b6880 0x5625a91bec90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:54988/0 (socket says 172.21.15.90:54988) 2024-04-02T14:39:06.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.943+0000 7f53b2800640 1 -- 172.21.15.90:0/1772962710 learned_addr learned my addr 172.21.15.90:0/1772962710 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:39:06.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.944+0000 7f53b1400640 1 --2- 172.21.15.90:0/1772962710 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5625a91ae1f0 0x5625a922adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:06.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.944+0000 7f53b2800640 1 -- 172.21.15.90:0/1772962710 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5625a91ad510 msgr2=0x5625a922a870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:39:06.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.944+0000 7f53b2800640 1 --2- 172.21.15.90:0/1772962710 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5625a91ad510 0x5625a922a870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:06.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.944+0000 7f53b2800640 1 -- 172.21.15.90:0/1772962710 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5625a91ae1f0 msgr2=0x5625a922adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:39:06.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.944+0000 7f53b2800640 1 --2- 172.21.15.90:0/1772962710 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5625a91ae1f0 0x5625a922adb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:06.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.944+0000 7f53b2800640 1 -- 172.21.15.90:0/1772962710 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5625a91b7360 con 0x5625a91b6880 2024-04-02T14:39:06.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.944+0000 7f53b1400640 1 --2- 172.21.15.90:0/1772962710 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5625a91ae1f0 0x5625a922adb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:39:06.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.944+0000 7f53b2800640 1 --2- 172.21.15.90:0/1772962710 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5625a91b6880 0x5625a91bec90 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f53ac006010 tx=0x7f53ac064650 comp rx=0 tx=0).ready entity=mon.1 client_cookie=96dc8947f601ed1a server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:39:06.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.944+0000 7f53b1e00640 1 --2- 172.21.15.90:0/1772962710 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5625a91ad510 0x5625a922a870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:39:06.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.944+0000 7f5393400640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:39:06.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.945+0000 7f5393400640 1 -- 172.21.15.90:0/1772962710 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53ac069060 con 0x5625a91b6880 2024-04-02T14:39:06.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.945+0000 7f5393400640 1 -- 172.21.15.90:0/1772962710 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f53ac003970 con 0x5625a91b6880 2024-04-02T14:39:06.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.945+0000 7f53bb7b0740 1 -- 172.21.15.90:0/1772962710 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f53ac04a040 con 0x5625a91b6880 2024-04-02T14:39:06.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.945+0000 7f5393400640 1 -- 172.21.15.90:0/1772962710 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f53ac055070 con 0x5625a91b6880 2024-04-02T14:39:06.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.945+0000 7f53bb7b0740 10 client.26638 fetch_fsmap learned FSMap version 27 2024-04-02T14:39:06.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.945+0000 7f53bb7b0740 1 -- 172.21.15.90:0/1772962710 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5625a923e4c0 con 0x5625a91b6880 2024-04-02T14:39:06.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.946+0000 7f5393400640 1 -- 172.21.15.90:0/1772962710 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f53ac04a040 con 0x5625a91b6880 2024-04-02T14:39:06.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.946+0000 7f5393400640 1 -- 172.21.15.90:0/1772962710 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f53ac06c050 con 0x5625a91b6880 2024-04-02T14:39:06.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.946+0000 7f53bb7b0740 10 client.26638 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:39:06.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.946+0000 7f53bb7b0740 10 client.26638 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:39:06.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.946+0000 7f53bb7b0740 20 client.26638 populate_metadata read hostname 'smithi090' 2024-04-02T14:39:06.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.946+0000 7f53bb7b0740 1 --2- 172.21.15.90:0/1772962710 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5625a923e9d0 0x5625a9240dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:06.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.946+0000 7f53bb7b0740 4 client.26638 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-02T14:39:06.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.946+0000 7f53bb7b0740 1 -- 172.21.15.90:0/1772962710 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5625a9118ba0 con 0x5625a923e9d0 2024-04-02T14:39:06.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.946+0000 7f53b1e00640 1 --2- 172.21.15.90:0/1772962710 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5625a923e9d0 0x5625a9240dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:06.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.947+0000 7f5393400640 10 client.26638 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:39:06.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.947+0000 7f53b1e00640 1 --2- 172.21.15.90:0/1772962710 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5625a923e9d0 0x5625a9240dc0 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=65fc02f7f05f748b server_cookie=e53b10a67c9bc0e6 in_seq=0 out_seq=0 2024-04-02T14:39:06.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.951+0000 7f5393400640 1 -- 172.21.15.90:0/1772962710 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5625a9118ba0 con 0x5625a923e9d0 2024-04-02T14:39:06.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.951+0000 7f5393400640 10 client.26638 handle_command_reply: tid=0 2024-04-02T14:39:06.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.951+0000 7f53bb7b0740 1 client.26638 shutdown 2024-04-02T14:39:06.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.951+0000 7f53bb7b0740 1 -- 172.21.15.90:0/1772962710 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5625a91b6880 msgr2=0x5625a91bec90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:06.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.951+0000 7f53bb7b0740 1 --2- 172.21.15.90:0/1772962710 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5625a91b6880 0x5625a91bec90 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f53ac006010 tx=0x7f53ac064650 comp rx=0 tx=0).stop 2024-04-02T14:39:06.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.952+0000 7f53bb7b0740 1 -- 172.21.15.90:0/1772962710 shutdown_connections 2024-04-02T14:39:06.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.952+0000 7f53bb7b0740 1 --2- 172.21.15.90:0/1772962710 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5625a923e9d0 0x5625a9240dc0 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:06.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.952+0000 7f53bb7b0740 1 --2- 172.21.15.90:0/1772962710 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5625a91b6880 0x5625a91bec90 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:06.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.952+0000 7f53bb7b0740 1 --2- 172.21.15.90:0/1772962710 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5625a91ae1f0 0x5625a922adb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:06.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.952+0000 7f53bb7b0740 1 --2- 172.21.15.90:0/1772962710 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5625a91ad510 0x5625a922a870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:06.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.952+0000 7f53bb7b0740 1 -- 172.21.15.90:0/1772962710 >> 172.21.15.90:0/1772962710 conn(0x5625a9141270 msgr2=0x5625a91b4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:39:06.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.952+0000 7f5393400640 0 client.26638 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:39:06.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.952+0000 7f53bb7b0740 1 -- 172.21.15.90:0/1772962710 shutdown_connections 2024-04-02T14:39:06.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.952+0000 7f53bb7b0740 1 -- 172.21.15.90:0/1772962710 wait complete. 2024-04-02T14:39:06.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.952+0000 7f53bb7b0740 20 client.26638 trim_cache size 0 max 16384 2024-04-02T14:39:06.955 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-02T14:39:06.955 INFO:teuthology.orchestra.run.smithi090.stdout: "return_code": 0, 2024-04-02T14:39:06.956 INFO:teuthology.orchestra.run.smithi090.stdout: "scrub_tag": "3cbf34cd-c7ad-4d88-b8e6-dcc476b035b6", 2024-04-02T14:39:06.956 INFO:teuthology.orchestra.run.smithi090.stdout: "mode": "asynchronous" 2024-04-02T14:39:06.956 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-02T14:39:06.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.953+0000 7f53b3200640 1 -- 172.21.15.90:0/397102246 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f53980a3e70 msgr2=0x7f53980a6330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:06.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.953+0000 7f53b3200640 1 --2- 172.21.15.90:0/397102246 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f53980a3e70 0x7f53980a6330 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f53a4005fd0 tx=0x7f53a4009470 comp rx=0 tx=0).stop 2024-04-02T14:39:06.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.953+0000 7f53b3200640 1 -- 172.21.15.90:0/397102246 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f53b410d300 msgr2=0x7f53b41a4b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:06.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.953+0000 7f53b3200640 1 --2- 172.21.15.90:0/397102246 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f53b410d300 0x7f53b41a4b40 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f53a0006fd0 tx=0x7f53a000b8e0 comp rx=0 tx=0).stop 2024-04-02T14:39:06.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.954+0000 7f53b3200640 1 -- 172.21.15.90:0/397102246 shutdown_connections 2024-04-02T14:39:06.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.954+0000 7f53b3200640 1 --2- 172.21.15.90:0/397102246 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f53980a3e70 0x7f53980a6330 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:06.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.954+0000 7f53b3200640 1 --2- 172.21.15.90:0/397102246 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f53b419e710 0x7f53b419eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:06.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.954+0000 7f53b3200640 1 --2- 172.21.15.90:0/397102246 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f53b410d300 0x7f53b41a4b40 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:06.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.954+0000 7f53b3200640 1 --2- 172.21.15.90:0/397102246 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f53b41082b0 0x7f53b41a4600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:06.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.954+0000 7f53b3200640 1 -- 172.21.15.90:0/397102246 >> 172.21.15.90:0/397102246 conn(0x7f53b4077920 msgr2=0x7f53b4077ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:39:06.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.954+0000 7f53b3200640 1 -- 172.21.15.90:0/397102246 shutdown_connections 2024-04-02T14:39:06.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:06.954+0000 7f53b3200640 1 -- 172.21.15.90:0/397102246 wait complete. 2024-04-02T14:39:06.983 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-02T14:39:06.983 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-02T14:39:07.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.318+0000 7f7500200640 1 -- 172.21.15.90:0/3544561908 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f74f8102450 msgr2=0x7f74f810e960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:07.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.318+0000 7f7500200640 1 --2- 172.21.15.90:0/3544561908 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f74f8102450 0x7f74f810e960 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f74f0009be0 tx=0x7f74f002bef0 comp rx=0 tx=0).stop 2024-04-02T14:39:07.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.319+0000 7f7500200640 1 -- 172.21.15.90:0/3544561908 shutdown_connections 2024-04-02T14:39:07.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.319+0000 7f7500200640 1 --2- 172.21.15.90:0/3544561908 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f74f8102450 0x7f74f810e960 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:07.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.319+0000 7f7500200640 1 --2- 172.21.15.90:0/3544561908 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f74f8101a90 0x7f74f8101f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:07.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.319+0000 7f7500200640 1 --2- 172.21.15.90:0/3544561908 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f74f8107aa0 0x7f74f8107e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:07.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.319+0000 7f7500200640 1 -- 172.21.15.90:0/3544561908 >> 172.21.15.90:0/3544561908 conn(0x7f74f8078f00 msgr2=0x7f74f8079310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:39:07.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.319+0000 7f7500200640 1 -- 172.21.15.90:0/3544561908 shutdown_connections 2024-04-02T14:39:07.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.319+0000 7f7500200640 1 -- 172.21.15.90:0/3544561908 wait complete. 2024-04-02T14:39:07.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.320+0000 7f7500200640 1 Processor -- start 2024-04-02T14:39:07.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.320+0000 7f7500200640 1 -- start start 2024-04-02T14:39:07.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.321+0000 7f7500200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f74f8101a90 0x7f74f8116100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:07.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.321+0000 7f7500200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f74f8102450 0x7f74f8116640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:07.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.321+0000 7f7500200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f74f8107aa0 0x7f74f8111210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:07.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.321+0000 7f7500200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f74f810fd00 con 0x7f74f8101a90 2024-04-02T14:39:07.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.321+0000 7f7500200640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f74f810fb80 con 0x7f74f8102450 2024-04-02T14:39:07.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.321+0000 7f7500200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f74f810fe80 con 0x7f74f8107aa0 2024-04-02T14:39:07.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.322+0000 7f74fd800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f74f8101a90 0x7f74f8116100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:07.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.322+0000 7f74fd800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f74f8101a90 0x7f74f8116100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:49876/0 (socket says 172.21.15.90:49876) 2024-04-02T14:39:07.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.322+0000 7f74fd800640 1 -- 172.21.15.90:0/430574541 learned_addr learned my addr 172.21.15.90:0/430574541 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:39:07.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.322+0000 7f74fe200640 1 --2- 172.21.15.90:0/430574541 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f74f8107aa0 0x7f74f8111210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:07.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.322+0000 7f74fce00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f74f8102450 0x7f74f8116640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:07.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.323+0000 7f74fd800640 1 -- 172.21.15.90:0/430574541 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f74f8107aa0 msgr2=0x7f74f8111210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:07.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.323+0000 7f74fd800640 1 --2- 172.21.15.90:0/430574541 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f74f8107aa0 0x7f74f8111210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:07.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.323+0000 7f74fd800640 1 -- 172.21.15.90:0/430574541 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f74f8102450 msgr2=0x7f74f8116640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:07.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.323+0000 7f74fd800640 1 --2- 172.21.15.90:0/430574541 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f74f8102450 0x7f74f8116640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:07.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.323+0000 7f74fd800640 1 -- 172.21.15.90:0/430574541 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74f8111ad0 con 0x7f74f8101a90 2024-04-02T14:39:07.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.323+0000 7f74fe200640 1 --2- 172.21.15.90:0/430574541 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f74f8107aa0 0x7f74f8111210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:39:07.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.324+0000 7f74fd800640 1 --2- 172.21.15.90:0/430574541 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f74f8101a90 0x7f74f8116100 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f74ec00e9c0 tx=0x7f74ec00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:39:07.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.324+0000 7f74f6000640 1 -- 172.21.15.90:0/430574541 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74ec00ccc0 con 0x7f74f8101a90 2024-04-02T14:39:07.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.324+0000 7f7500200640 1 -- 172.21.15.90:0/430574541 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74f8111dc0 con 0x7f74f8101a90 2024-04-02T14:39:07.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.324+0000 7f74f6000640 1 -- 172.21.15.90:0/430574541 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f74ec005500 con 0x7f74f8101a90 2024-04-02T14:39:07.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.324+0000 7f74f6000640 1 -- 172.21.15.90:0/430574541 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74ec010430 con 0x7f74f8101a90 2024-04-02T14:39:07.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.324+0000 7f7500200640 1 -- 172.21.15.90:0/430574541 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74f811e000 con 0x7f74f8101a90 2024-04-02T14:39:07.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.330+0000 7f7502f9b740 1 Processor -- start 2024-04-02T14:39:07.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.330+0000 7f7502f9b740 1 -- start start 2024-04-02T14:39:07.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.331+0000 7f7502f9b740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56405fdf17b0 0x56405fdf1b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:07.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.331+0000 7f7502f9b740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56405fdf20d0 0x56405fdfa770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:07.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.331+0000 7f7502f9b740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56405fdfacb0 0x56405fdfd0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:07.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.331+0000 7f7502f9b740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x56405fd55770 con 0x56405fdfacb0 2024-04-02T14:39:07.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.331+0000 7f7502f9b740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x56405fcfd700 con 0x56405fdf17b0 2024-04-02T14:39:07.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.331+0000 7f7502f9b740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x56405fd51720 con 0x56405fdf20d0 2024-04-02T14:39:07.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.331+0000 7f74f6000640 1 -- 172.21.15.90:0/430574541 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f74ec0046c0 con 0x7f74f8101a90 2024-04-02T14:39:07.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.331+0000 7f74fd800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56405fdf20d0 0x56405fdfa770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:07.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.332+0000 7f74fe200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56405fdf17b0 0x56405fdf1b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:07.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.332+0000 7f74fe200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56405fdf17b0 0x56405fdf1b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:41842/0 (socket says 172.21.15.90:41842) 2024-04-02T14:39:07.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.332+0000 7f74fd800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56405fdf20d0 0x56405fdfa770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:55024/0 (socket says 172.21.15.90:55024) 2024-04-02T14:39:07.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.332+0000 7f74fe200640 1 -- 172.21.15.90:0/443560547 learned_addr learned my addr 172.21.15.90:0/443560547 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:39:07.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.332+0000 7f74fce00640 1 --2- 172.21.15.90:0/443560547 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56405fdfacb0 0x56405fdfd0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:07.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.332+0000 7f74fe200640 1 -- 172.21.15.90:0/443560547 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56405fdf20d0 msgr2=0x56405fdfa770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:39:07.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.332+0000 7f74fe200640 1 --2- 172.21.15.90:0/443560547 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56405fdf20d0 0x56405fdfa770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:07.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.332+0000 7f74fe200640 1 -- 172.21.15.90:0/443560547 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56405fdfacb0 msgr2=0x56405fdfd0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:39:07.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.332+0000 7f74fe200640 1 --2- 172.21.15.90:0/443560547 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56405fdfacb0 0x56405fdfd0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:07.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.332+0000 7f74fe200640 1 -- 172.21.15.90:0/443560547 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56405fb337d0 con 0x56405fdf17b0 2024-04-02T14:39:07.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.333+0000 7f74fe200640 1 --2- 172.21.15.90:0/443560547 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56405fdf17b0 0x56405fdf1b90 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f74f0009be0 tx=0x7f74f0005ec0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2869c97750acce26 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:39:07.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.334+0000 7f74f6000640 1 --2- 172.21.15.90:0/430574541 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f74e00a3e00 0x7f74e00a62c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:07.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.334+0000 7f74f6000640 1 -- 172.21.15.90:0/430574541 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f74ec0d0b00 con 0x7f74f8101a90 2024-04-02T14:39:07.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.334+0000 7f74eaa00640 1 -- 172.21.15.90:0/443560547 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74f0043030 con 0x56405fdf17b0 2024-04-02T14:39:07.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.334+0000 7f74eaa00640 1 -- 172.21.15.90:0/443560547 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f74f0003970 con 0x56405fdf17b0 2024-04-02T14:39:07.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.334+0000 7f74eaa00640 1 -- 172.21.15.90:0/443560547 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74f002cd10 con 0x56405fdf17b0 2024-04-02T14:39:07.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.334+0000 7f74fd800640 1 --2- 172.21.15.90:0/430574541 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f74e00a3e00 0x7f74e00a62c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:07.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.335+0000 7f7502f9b740 1 -- 172.21.15.90:0/443560547 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56405fdf17b0 msgr2=0x56405fdf1b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:07.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.335+0000 7f7502f9b740 1 --2- 172.21.15.90:0/443560547 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56405fdf17b0 0x56405fdf1b90 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f74f0009be0 tx=0x7f74f0005ec0 comp rx=0 tx=0).stop 2024-04-02T14:39:07.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.336+0000 7f7502f9b740 1 -- 172.21.15.90:0/443560547 shutdown_connections 2024-04-02T14:39:07.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.336+0000 7f7502f9b740 1 --2- 172.21.15.90:0/443560547 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56405fdfacb0 0x56405fdfd0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:07.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.336+0000 7f7502f9b740 1 --2- 172.21.15.90:0/443560547 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56405fdf20d0 0x56405fdfa770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:07.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.336+0000 7f7502f9b740 1 --2- 172.21.15.90:0/443560547 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56405fdf17b0 0x56405fdf1b90 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:07.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.336+0000 7f7502f9b740 1 -- 172.21.15.90:0/443560547 >> 172.21.15.90:0/443560547 conn(0x56405fdebee0 msgr2=0x56405fdec2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:39:07.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.337+0000 7f74fd800640 1 --2- 172.21.15.90:0/430574541 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f74e00a3e00 0x7f74e00a62c0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x56405fdf1280 tx=0x7f74ec0bd040 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:39:07.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.337+0000 7f7502f9b740 1 -- 172.21.15.90:0/443560547 shutdown_connections 2024-04-02T14:39:07.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.337+0000 7f7502f9b740 1 -- 172.21.15.90:0/443560547 wait complete. 2024-04-02T14:39:07.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.338+0000 7f7502f9b740 1 -- start start 2024-04-02T14:39:07.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.338+0000 7f7502f9b740 1 Processor -- start 2024-04-02T14:39:07.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.338+0000 7f7502f9b740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56405fdf17b0 0x56405fdf7d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:07.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.338+0000 7f7502f9b740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56405fdf20d0 0x56405fdf8290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:07.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.339+0000 7f7502f9b740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56405fdfacb0 0x56405fdf87d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:07.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.339+0000 7f7502f9b740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x56405fdfda40 con 0x56405fdf17b0 2024-04-02T14:39:07.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.339+0000 7f7502f9b740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x56405fd10ac0 con 0x56405fdfacb0 2024-04-02T14:39:07.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.339+0000 7f7502f9b740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x56405fdfdbc0 con 0x56405fdf20d0 2024-04-02T14:39:07.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.339+0000 7f7502f9b740 10 client.0 _ll_register_callbacks cb 0x56405fd13180 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:39:07.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.339+0000 7f74fe200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56405fdf17b0 0x56405fdf7d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:07.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.339+0000 7f74fe200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56405fdf17b0 0x56405fdf7d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:49882/0 (socket says 172.21.15.90:49882) 2024-04-02T14:39:07.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.339+0000 7f74fe200640 1 -- 172.21.15.90:0/538832789 learned_addr learned my addr 172.21.15.90:0/538832789 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:39:07.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.339+0000 7f74fce00640 1 --2- 172.21.15.90:0/538832789 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56405fdf20d0 0x56405fdf8290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:07.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.339+0000 7f74fe200640 1 --2- 172.21.15.90:0/538832789 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56405fdfacb0 0x56405fdf87d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:07.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.339+0000 7f74fe200640 1 -- 172.21.15.90:0/538832789 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56405fdf20d0 msgr2=0x56405fdf8290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:39:07.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.339+0000 7f74fe200640 1 --2- 172.21.15.90:0/538832789 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56405fdf20d0 0x56405fdf8290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:07.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.339+0000 7f74fe200640 1 -- 172.21.15.90:0/538832789 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56405fdfacb0 msgr2=0x56405fdf87d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:39:07.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.339+0000 7f74fe200640 1 --2- 172.21.15.90:0/538832789 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56405fdfacb0 0x56405fdf87d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:07.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.339+0000 7f74fe200640 1 -- 172.21.15.90:0/538832789 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56405fa693f0 con 0x56405fdf17b0 2024-04-02T14:39:07.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.340+0000 7f74fe200640 1 --2- 172.21.15.90:0/538832789 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56405fdf17b0 0x56405fdf7d50 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f74f002cbb0 tx=0x7f74f002cab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=663e80b909f19199 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:39:07.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.340+0000 7f74df400640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-02T14:39:07.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.340+0000 7f74df400640 1 -- 172.21.15.90:0/538832789 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74f0043040 con 0x56405fdf17b0 2024-04-02T14:39:07.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.341+0000 7f7502f9b740 1 -- 172.21.15.90:0/538832789 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f74f0003970 con 0x56405fdf17b0 2024-04-02T14:39:07.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.341+0000 7f74df400640 1 -- 172.21.15.90:0/538832789 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f74f00408a0 con 0x56405fdf17b0 2024-04-02T14:39:07.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.341+0000 7f74df400640 1 -- 172.21.15.90:0/538832789 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f74f0004b40 con 0x56405fdf17b0 2024-04-02T14:39:07.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.341+0000 7f7502f9b740 10 client.16266 fetch_fsmap learned FSMap version 27 2024-04-02T14:39:07.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.341+0000 7f7502f9b740 1 -- 172.21.15.90:0/538832789 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x56405fa290f0 con 0x56405fdf17b0 2024-04-02T14:39:07.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.342+0000 7f74df400640 1 -- 172.21.15.90:0/538832789 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f74f0003970 con 0x56405fdf17b0 2024-04-02T14:39:07.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.343+0000 7f74df400640 1 -- 172.21.15.90:0/538832789 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f74f0053070 con 0x56405fdf17b0 2024-04-02T14:39:07.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.343+0000 7f7502f9b740 10 client.16266 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:39:07.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.343+0000 7f7502f9b740 10 client.16266 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:39:07.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.343+0000 7f7502f9b740 20 client.16266 populate_metadata read hostname 'smithi090' 2024-04-02T14:39:07.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.343+0000 7f7502f9b740 1 --2- 172.21.15.90:0/538832789 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x56405fe0ab70 0x56405fe0cf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:07.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.343+0000 7f7502f9b740 4 client.16266 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:39:07.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.343+0000 7f7502f9b740 1 -- 172.21.15.90:0/538832789 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56405fcf6630 con 0x56405fe0ab70 2024-04-02T14:39:07.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.343+0000 7f74fce00640 1 --2- 172.21.15.90:0/538832789 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x56405fe0ab70 0x56405fe0cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:07.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.344+0000 7f74fce00640 1 --2- 172.21.15.90:0/538832789 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x56405fe0ab70 0x56405fe0cf60 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d0b166071a8286d server_cookie=3bf5d572214a0b67 in_seq=0 out_seq=0 2024-04-02T14:39:07.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.344+0000 7f74df400640 10 client.16266 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:39:07.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.347+0000 7f74df400640 1 -- 172.21.15.90:0/538832789 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56405fcf6630 con 0x56405fe0ab70 2024-04-02T14:39:07.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.347+0000 7f74df400640 10 client.16266 handle_command_reply: tid=0 2024-04-02T14:39:07.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.347+0000 7f7502f9b740 1 client.16266 shutdown 2024-04-02T14:39:07.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.348+0000 7f7502f9b740 1 -- 172.21.15.90:0/538832789 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56405fdf17b0 msgr2=0x56405fdf7d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:07.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.348+0000 7f7502f9b740 1 --2- 172.21.15.90:0/538832789 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56405fdf17b0 0x56405fdf7d50 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f74f002cbb0 tx=0x7f74f002cab0 comp rx=0 tx=0).stop 2024-04-02T14:39:07.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.348+0000 7f7502f9b740 1 -- 172.21.15.90:0/538832789 shutdown_connections 2024-04-02T14:39:07.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.348+0000 7f7502f9b740 1 --2- 172.21.15.90:0/538832789 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x56405fe0ab70 0x56405fe0cf60 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:07.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.348+0000 7f7502f9b740 1 --2- 172.21.15.90:0/538832789 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56405fdfacb0 0x56405fdf87d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:07.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.348+0000 7f7502f9b740 1 --2- 172.21.15.90:0/538832789 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56405fdf20d0 0x56405fdf8290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:07.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.348+0000 7f7502f9b740 1 --2- 172.21.15.90:0/538832789 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56405fdf17b0 0x56405fdf7d50 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:07.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.348+0000 7f7502f9b740 1 -- 172.21.15.90:0/538832789 >> 172.21.15.90:0/538832789 conn(0x56405fdebee0 msgr2=0x56405fdedc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:39:07.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.348+0000 7f74df400640 0 client.16266 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:39:07.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.349+0000 7f7502f9b740 1 -- 172.21.15.90:0/538832789 shutdown_connections 2024-04-02T14:39:07.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.349+0000 7f7502f9b740 1 -- 172.21.15.90:0/538832789 wait complete. 2024-04-02T14:39:07.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.349+0000 7f7502f9b740 20 client.16266 trim_cache size 0 max 16384 2024-04-02T14:39:07.360 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:39:06 smithi092 ceph-mon[36497]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T14:39:07.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.384+0000 7f7502f9b740 1 Processor -- start 2024-04-02T14:39:07.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.385+0000 7f7502f9b740 1 -- start start 2024-04-02T14:39:07.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.385+0000 7f7502f9b740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56405fdf17b0 0x56405fdfa480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:07.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.385+0000 7f7502f9b740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56405fdf20d0 0x56405fdf6eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:07.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.385+0000 7f7502f9b740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56405fdfacb0 0x56405fe06580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:07.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.385+0000 7f7502f9b740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f74f81269c0 con 0x56405fdf20d0 2024-04-02T14:39:07.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.385+0000 7f7502f9b740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f74f8128500 con 0x56405fdfacb0 2024-04-02T14:39:07.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.385+0000 7f7502f9b740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f74f8127090 con 0x56405fdf17b0 2024-04-02T14:39:07.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.385+0000 7f74fe200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56405fdf17b0 0x56405fdfa480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:07.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.385+0000 7f74fe200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56405fdf17b0 0x56405fdfa480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:55048/0 (socket says 172.21.15.90:55048) 2024-04-02T14:39:07.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.385+0000 7f74fe200640 1 -- 172.21.15.90:0/2928237297 learned_addr learned my addr 172.21.15.90:0/2928237297 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:39:07.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.385+0000 7f74fce00640 1 --2- 172.21.15.90:0/2928237297 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56405fdf20d0 0x56405fdf6eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:07.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.385+0000 7f74fe200640 1 --2- 172.21.15.90:0/2928237297 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56405fdfacb0 0x56405fe06580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:07.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.386+0000 7f74fe200640 1 -- 172.21.15.90:0/2928237297 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56405fdfacb0 msgr2=0x56405fe06580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:39:07.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.386+0000 7f74fe200640 1 --2- 172.21.15.90:0/2928237297 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56405fdfacb0 0x56405fe06580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:07.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.386+0000 7f74fe200640 1 -- 172.21.15.90:0/2928237297 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56405fdf20d0 msgr2=0x56405fdf6eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:39:07.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.386+0000 7f74fe200640 1 --2- 172.21.15.90:0/2928237297 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56405fdf20d0 0x56405fdf6eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:07.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.386+0000 7f74fe200640 1 -- 172.21.15.90:0/2928237297 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56405fdfd980 con 0x56405fdf17b0 2024-04-02T14:39:07.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.386+0000 7f74fce00640 1 --2- 172.21.15.90:0/2928237297 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56405fdf20d0 0x56405fdf6eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:39:07.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.388+0000 7f74fe200640 1 --2- 172.21.15.90:0/2928237297 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56405fdf17b0 0x56405fdfa480 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f74f0005780 tx=0x7f74f002bef0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=706cdc76b44a1951 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:39:07.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.389+0000 7f74ea000640 1 -- 172.21.15.90:0/2928237297 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74f0004b40 con 0x56405fdf17b0 2024-04-02T14:39:07.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.389+0000 7f74ea000640 1 -- 172.21.15.90:0/2928237297 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f74f002c950 con 0x56405fdf17b0 2024-04-02T14:39:07.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.389+0000 7f74ea000640 1 -- 172.21.15.90:0/2928237297 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74f0040070 con 0x56405fdf17b0 2024-04-02T14:39:07.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.390+0000 7f7502f9b740 1 -- 172.21.15.90:0/2928237297 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56405fdf17b0 msgr2=0x56405fdfa480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:07.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.390+0000 7f7502f9b740 1 --2- 172.21.15.90:0/2928237297 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56405fdf17b0 0x56405fdfa480 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f74f0005780 tx=0x7f74f002bef0 comp rx=0 tx=0).stop 2024-04-02T14:39:07.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.390+0000 7f7502f9b740 1 -- 172.21.15.90:0/2928237297 shutdown_connections 2024-04-02T14:39:07.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.390+0000 7f7502f9b740 1 --2- 172.21.15.90:0/2928237297 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56405fdfacb0 0x56405fe06580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:07.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.390+0000 7f7502f9b740 1 --2- 172.21.15.90:0/2928237297 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56405fdf20d0 0x56405fdf6eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:07.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.390+0000 7f7502f9b740 1 --2- 172.21.15.90:0/2928237297 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56405fdf17b0 0x56405fdfa480 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:07.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.390+0000 7f7502f9b740 1 -- 172.21.15.90:0/2928237297 >> 172.21.15.90:0/2928237297 conn(0x56405fdebee0 msgr2=0x56405fe05e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:39:07.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.391+0000 7f7502f9b740 1 -- 172.21.15.90:0/2928237297 shutdown_connections 2024-04-02T14:39:07.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.391+0000 7f7502f9b740 1 -- 172.21.15.90:0/2928237297 wait complete. 2024-04-02T14:39:07.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.391+0000 7f7502f9b740 1 -- start start 2024-04-02T14:39:07.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.392+0000 7f7502f9b740 1 Processor -- start 2024-04-02T14:39:07.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.392+0000 7f7502f9b740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56405fdf17b0 0x56405fe69760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:07.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.392+0000 7f7502f9b740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56405fdf20d0 0x56405fe69ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:07.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.392+0000 7f7502f9b740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56405fdfacb0 0x56405fe6a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:07.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.392+0000 7f7502f9b740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f74f8127e30 con 0x56405fdf17b0 2024-04-02T14:39:07.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.392+0000 7f7502f9b740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f74f8127760 con 0x56405fdf20d0 2024-04-02T14:39:07.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.392+0000 7f7502f9b740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f74f8124940 con 0x56405fdfacb0 2024-04-02T14:39:07.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.392+0000 7f7502f9b740 10 client.0 _ll_register_callbacks cb 0x7f74f0009840 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:39:07.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.393+0000 7f74fe200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56405fdfacb0 0x56405fe6a1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:07.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.393+0000 7f74fce00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56405fdf20d0 0x56405fe69ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:07.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.393+0000 7f74fce00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56405fdf20d0 0x56405fe69ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:41866/0 (socket says 172.21.15.90:41866) 2024-04-02T14:39:07.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.393+0000 7f74fce00640 1 -- 172.21.15.90:0/654592634 learned_addr learned my addr 172.21.15.90:0/654592634 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:39:07.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.393+0000 7f74fe200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56405fdfacb0 0x56405fe6a1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:55058/0 (socket says 172.21.15.90:55058) 2024-04-02T14:39:07.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.393+0000 7f74fe200640 1 --2- 172.21.15.90:0/654592634 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56405fdf17b0 0x56405fe69760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:07.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.394+0000 7f74fce00640 1 -- 172.21.15.90:0/654592634 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56405fdfacb0 msgr2=0x56405fe6a1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:39:07.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.394+0000 7f74fce00640 1 --2- 172.21.15.90:0/654592634 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56405fdfacb0 0x56405fe6a1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:07.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.394+0000 7f74fce00640 1 -- 172.21.15.90:0/654592634 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56405fdf17b0 msgr2=0x56405fe69760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:39:07.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.394+0000 7f74fce00640 1 --2- 172.21.15.90:0/654592634 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56405fdf17b0 0x56405fe69760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:07.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.394+0000 7f74fce00640 1 -- 172.21.15.90:0/654592634 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56405fe0de00 con 0x56405fdf20d0 2024-04-02T14:39:07.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.394+0000 7f74fe200640 1 --2- 172.21.15.90:0/654592634 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56405fdfacb0 0x56405fe6a1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:39:07.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.394+0000 7f74fce00640 1 --2- 172.21.15.90:0/654592634 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56405fdf20d0 0x56405fe69ca0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f74e4000c00 tx=0x7f74e4014840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=56356d9ac2895ed5 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:39:07.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.394+0000 7f74df400640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:39:07.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.395+0000 7f74df400640 1 -- 172.21.15.90:0/654592634 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74e4004230 con 0x56405fdf20d0 2024-04-02T14:39:07.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.395+0000 7f74df400640 1 -- 172.21.15.90:0/654592634 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f74e40043d0 con 0x56405fdf20d0 2024-04-02T14:39:07.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.395+0000 7f7502f9b740 1 -- 172.21.15.90:0/654592634 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f74f002c950 con 0x56405fdf20d0 2024-04-02T14:39:07.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.395+0000 7f74df400640 1 -- 172.21.15.90:0/654592634 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f74e402a050 con 0x56405fdf20d0 2024-04-02T14:39:07.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.395+0000 7f7502f9b740 10 client.26783 fetch_fsmap learned FSMap version 27 2024-04-02T14:39:07.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.395+0000 7f7502f9b740 1 -- 172.21.15.90:0/654592634 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x56405fe820e0 con 0x56405fdf20d0 2024-04-02T14:39:07.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.396+0000 7f74df400640 1 -- 172.21.15.90:0/654592634 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f74e4020070 con 0x56405fdf20d0 2024-04-02T14:39:07.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.396+0000 7f74df400640 1 -- 172.21.15.90:0/654592634 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f74e40039c0 con 0x56405fdf20d0 2024-04-02T14:39:07.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.396+0000 7f7502f9b740 10 client.26783 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:39:07.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.396+0000 7f7502f9b740 10 client.26783 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:39:07.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.396+0000 7f7502f9b740 20 client.26783 populate_metadata read hostname 'smithi090' 2024-04-02T14:39:07.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.397+0000 7f7502f9b740 1 --2- 172.21.15.90:0/654592634 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x56405fe828e0 0x56405fe84d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:07.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.397+0000 7f7502f9b740 4 client.26783 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-02T14:39:07.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.397+0000 7f7502f9b740 1 -- 172.21.15.90:0/654592634 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "scrub status"}) -- 0x7f74f002c950 con 0x56405fe828e0 2024-04-02T14:39:07.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.397+0000 7f74fe200640 1 --2- 172.21.15.90:0/654592634 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x56405fe828e0 0x56405fe84d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:07.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.398+0000 7f74fe200640 1 --2- 172.21.15.90:0/654592634 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x56405fe828e0 0x56405fe84d00 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ffda27589735c34e server_cookie=51aa736ad456784c in_seq=0 out_seq=0 2024-04-02T14:39:07.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.398+0000 7f74df400640 10 client.26783 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:39:07.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.398+0000 7f74df400640 1 -- 172.21.15.90:0/654592634 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x7f74f002c950 con 0x56405fe828e0 2024-04-02T14:39:07.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.398+0000 7f74df400640 10 client.26783 handle_command_reply: tid=0 2024-04-02T14:39:07.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.398+0000 7f7502f9b740 1 client.26783 shutdown 2024-04-02T14:39:07.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.398+0000 7f7502f9b740 1 -- 172.21.15.90:0/654592634 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56405fdf20d0 msgr2=0x56405fe69ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:07.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.398+0000 7f7502f9b740 1 --2- 172.21.15.90:0/654592634 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56405fdf20d0 0x56405fe69ca0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f74e4000c00 tx=0x7f74e4014840 comp rx=0 tx=0).stop 2024-04-02T14:39:07.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.398+0000 7f7502f9b740 1 -- 172.21.15.90:0/654592634 shutdown_connections 2024-04-02T14:39:07.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.398+0000 7f7502f9b740 1 --2- 172.21.15.90:0/654592634 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x56405fe828e0 0x56405fe84d00 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:07.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.399+0000 7f7502f9b740 1 --2- 172.21.15.90:0/654592634 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56405fdfacb0 0x56405fe6a1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:07.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.399+0000 7f74df400640 0 client.26783 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:39:07.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.399+0000 7f7502f9b740 1 --2- 172.21.15.90:0/654592634 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56405fdf20d0 0x56405fe69ca0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:07.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.399+0000 7f7502f9b740 1 --2- 172.21.15.90:0/654592634 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56405fdf17b0 0x56405fe69760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:07.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.399+0000 7f7502f9b740 1 -- 172.21.15.90:0/654592634 >> 172.21.15.90:0/654592634 conn(0x56405fdebee0 msgr2=0x56405fe06700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:39:07.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.399+0000 7f7502f9b740 1 -- 172.21.15.90:0/654592634 shutdown_connections 2024-04-02T14:39:07.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.399+0000 7f7502f9b740 1 -- 172.21.15.90:0/654592634 wait complete. 2024-04-02T14:39:07.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.399+0000 7f7502f9b740 20 client.26783 trim_cache size 0 max 16384 2024-04-02T14:39:07.403 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-02T14:39:07.403 INFO:teuthology.orchestra.run.smithi090.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-02T14:39:07.404 INFO:teuthology.orchestra.run.smithi090.stdout: "scrubs": { 2024-04-02T14:39:07.404 INFO:teuthology.orchestra.run.smithi090.stdout: "3cbf34cd-c7ad-4d88-b8e6-dcc476b035b6": { 2024-04-02T14:39:07.404 INFO:teuthology.orchestra.run.smithi090.stdout: "path": "/", 2024-04-02T14:39:07.404 INFO:teuthology.orchestra.run.smithi090.stdout: "tag": "3cbf34cd-c7ad-4d88-b8e6-dcc476b035b6", 2024-04-02T14:39:07.404 INFO:teuthology.orchestra.run.smithi090.stdout: "options": "recursive,force" 2024-04-02T14:39:07.404 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:39:07.404 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:39:07.404 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-02T14:39:07.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.401+0000 7f7500200640 1 -- 172.21.15.90:0/430574541 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f74e00a3e00 msgr2=0x7f74e00a62c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:07.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.401+0000 7f7500200640 1 --2- 172.21.15.90:0/430574541 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f74e00a3e00 0x7f74e00a62c0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x56405fdf1280 tx=0x7f74ec0bd040 comp rx=0 tx=0).stop 2024-04-02T14:39:07.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.401+0000 7f7500200640 1 -- 172.21.15.90:0/430574541 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f74f8101a90 msgr2=0x7f74f8116100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:07.406 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.401+0000 7f7500200640 1 --2- 172.21.15.90:0/430574541 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f74f8101a90 0x7f74f8116100 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f74ec00e9c0 tx=0x7f74ec00ee80 comp rx=0 tx=0).stop 2024-04-02T14:39:07.406 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.401+0000 7f7500200640 1 -- 172.21.15.90:0/430574541 shutdown_connections 2024-04-02T14:39:07.406 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.401+0000 7f7500200640 1 --2- 172.21.15.90:0/430574541 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f74e00a3e00 0x7f74e00a62c0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:07.406 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.401+0000 7f7500200640 1 --2- 172.21.15.90:0/430574541 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f74f8107aa0 0x7f74f8111210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:07.406 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.401+0000 7f7500200640 1 --2- 172.21.15.90:0/430574541 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f74f8102450 0x7f74f8116640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:07.406 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.401+0000 7f7500200640 1 --2- 172.21.15.90:0/430574541 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f74f8101a90 0x7f74f8116100 secure :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f74ec00e9c0 tx=0x7f74ec00ee80 comp rx=0 tx=0).stop 2024-04-02T14:39:07.406 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.401+0000 7f7500200640 1 -- 172.21.15.90:0/430574541 >> 172.21.15.90:0/430574541 conn(0x7f74f8078f00 msgr2=0x7f74f81042c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:39:07.406 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.401+0000 7f7500200640 1 -- 172.21.15.90:0/430574541 shutdown_connections 2024-04-02T14:39:07.406 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:07.401+0000 7f7500200640 1 -- 172.21.15.90:0/430574541 wait complete. 2024-04-02T14:39:07.421 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:39:06 smithi090 ceph-mon[31674]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T14:39:07.429 INFO:tasks.cephfs.filesystem:scrub status for tag:3cbf34cd-c7ad-4d88-b8e6-dcc476b035b6 - {'path': '/', 'tag': '3cbf34cd-c7ad-4d88-b8e6-dcc476b035b6', 'options': 'recursive,force'} 2024-04-02T14:39:07.436 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:39:06 smithi177 ceph-mon[36431]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T14:39:08.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:39:07 smithi092 ceph-mon[36497]: pgmap v130: 97 pgs: 97 active+clean; 2.4 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 16 MiB/s wr, 651 op/s 2024-04-02T14:39:08.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:39:07 smithi092 ceph-mon[36497]: scrub queued for path: / 2024-04-02T14:39:08.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:39:07 smithi092 ceph-mon[36497]: scrub summary: idle+waiting paths [/] 2024-04-02T14:39:08.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:39:07 smithi092 ceph-mon[36497]: scrub summary: active paths [/] 2024-04-02T14:39:08.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:39:07 smithi092 ceph-mon[36497]: scrub summary: idle+waiting paths [/] 2024-04-02T14:39:08.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:39:07 smithi092 ceph-mon[36497]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:39:08.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:39:07 smithi092 ceph-mon[36497]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T14:39:08.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:39:07 smithi092 ceph-mon[36497]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' 2024-04-02T14:39:08.436 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:39:07 smithi177 ceph-mon[36431]: pgmap v130: 97 pgs: 97 active+clean; 2.4 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 16 MiB/s wr, 651 op/s 2024-04-02T14:39:08.437 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:39:07 smithi177 ceph-mon[36431]: scrub queued for path: / 2024-04-02T14:39:08.437 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:39:07 smithi177 ceph-mon[36431]: scrub summary: idle+waiting paths [/] 2024-04-02T14:39:08.437 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:39:07 smithi177 ceph-mon[36431]: scrub summary: active paths [/] 2024-04-02T14:39:08.437 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:39:07 smithi177 ceph-mon[36431]: scrub summary: idle+waiting paths [/] 2024-04-02T14:39:08.437 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:39:07 smithi177 ceph-mon[36431]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:39:08.437 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:39:07 smithi177 ceph-mon[36431]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T14:39:08.437 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:39:07 smithi177 ceph-mon[36431]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' 2024-04-02T14:39:08.438 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:39:07 smithi090 ceph-mon[31674]: pgmap v130: 97 pgs: 97 active+clean; 2.4 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 16 MiB/s wr, 651 op/s 2024-04-02T14:39:08.438 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:39:07 smithi090 ceph-mon[31674]: scrub queued for path: / 2024-04-02T14:39:08.438 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:39:07 smithi090 ceph-mon[31674]: scrub summary: idle+waiting paths [/] 2024-04-02T14:39:08.439 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:39:07 smithi090 ceph-mon[31674]: scrub summary: active paths [/] 2024-04-02T14:39:08.439 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:39:07 smithi090 ceph-mon[31674]: scrub summary: idle+waiting paths [/] 2024-04-02T14:39:08.439 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:39:07 smithi090 ceph-mon[31674]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:39:08.439 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:39:07 smithi090 ceph-mon[31674]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T14:39:08.439 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:39:07 smithi090 ceph-mon[31674]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' 2024-04-02T14:39:10.243 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:39:09 smithi090 ceph-mon[31674]: pgmap v131: 97 pgs: 97 active+clean; 2.6 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 40 MiB/s wr, 1.55k op/s 2024-04-02T14:39:10.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:39:09 smithi092 ceph-mon[36497]: pgmap v131: 97 pgs: 97 active+clean; 2.6 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 40 MiB/s wr, 1.55k op/s 2024-04-02T14:39:10.436 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:39:09 smithi177 ceph-mon[36431]: pgmap v131: 97 pgs: 97 active+clean; 2.6 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 40 MiB/s wr, 1.55k op/s 2024-04-02T14:39:11.831 INFO:teuthology.orchestra.run.smithi090.stderr:progress: 220.0 s, 2287.3 tps, lat 46.596 ms stddev 89.152 2024-04-02T14:39:12.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:39:11 smithi092 ceph-mon[36497]: pgmap v132: 97 pgs: 97 active+clean; 2.7 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 146 KiB/s rd, 42 MiB/s wr, 1.60k op/s 2024-04-02T14:39:12.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:39:11 smithi092 ceph-mon[36497]: scrub summary: idle 2024-04-02T14:39:12.437 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:39:11 smithi177 ceph-mon[36431]: pgmap v132: 97 pgs: 97 active+clean; 2.7 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 146 KiB/s rd, 42 MiB/s wr, 1.60k op/s 2024-04-02T14:39:12.437 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:39:11 smithi177 ceph-mon[36431]: scrub summary: idle 2024-04-02T14:39:12.438 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:39:11 smithi090 ceph-mon[31674]: pgmap v132: 97 pgs: 97 active+clean; 2.7 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 146 KiB/s rd, 42 MiB/s wr, 1.60k op/s 2024-04-02T14:39:12.438 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:39:11 smithi090 ceph-mon[31674]: scrub summary: idle 2024-04-02T14:39:14.360 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:39:13 smithi092 ceph-mon[36497]: pgmap v133: 97 pgs: 97 active+clean; 2.7 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 145 KiB/s rd, 40 MiB/s wr, 1.52k op/s 2024-04-02T14:39:14.436 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:39:13 smithi177 ceph-mon[36431]: pgmap v133: 97 pgs: 97 active+clean; 2.7 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 145 KiB/s rd, 40 MiB/s wr, 1.52k op/s 2024-04-02T14:39:14.438 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:39:13 smithi090 ceph-mon[31674]: pgmap v133: 97 pgs: 97 active+clean; 2.7 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 145 KiB/s rd, 40 MiB/s wr, 1.52k op/s 2024-04-02T14:39:16.360 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:39:15 smithi092 ceph-mon[36497]: pgmap v134: 97 pgs: 97 active+clean; 2.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 181 KiB/s rd, 55 MiB/s wr, 1.83k op/s 2024-04-02T14:39:16.436 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:39:15 smithi177 ceph-mon[36431]: pgmap v134: 97 pgs: 97 active+clean; 2.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 181 KiB/s rd, 55 MiB/s wr, 1.83k op/s 2024-04-02T14:39:16.440 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:39:15 smithi090 ceph-mon[31674]: pgmap v134: 97 pgs: 97 active+clean; 2.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 181 KiB/s rd, 55 MiB/s wr, 1.83k op/s 2024-04-02T14:39:16.832 INFO:teuthology.orchestra.run.smithi090.stderr:progress: 225.0 s, 2534.2 tps, lat 39.465 ms stddev 53.535 2024-04-02T14:39:17.078 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.077+0000 7f1166600640 1 -- 172.21.15.90:0/3749338435 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f11601048d0 msgr2=0x7f1160108910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:17.078 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.077+0000 7f1166600640 1 --2- 172.21.15.90:0/3749338435 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f11601048d0 0x7f1160108910 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f1148009900 tx=0x7f114802bc30 comp rx=0 tx=0).stop 2024-04-02T14:39:17.079 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.077+0000 7f1166600640 1 -- 172.21.15.90:0/3749338435 shutdown_connections 2024-04-02T14:39:17.079 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.077+0000 7f1166600640 1 --2- 172.21.15.90:0/3749338435 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1160108ff0 0x7f1160110bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:17.079 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.077+0000 7f1166600640 1 --2- 172.21.15.90:0/3749338435 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f11601048d0 0x7f1160108910 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:17.079 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.077+0000 7f1166600640 1 --2- 172.21.15.90:0/3749338435 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1160103f20 0x7f1160104300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:17.079 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.077+0000 7f1166600640 1 -- 172.21.15.90:0/3749338435 >> 172.21.15.90:0/3749338435 conn(0x7f11600ff780 msgr2=0x7f1160101ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:39:17.079 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.078+0000 7f1166600640 1 -- 172.21.15.90:0/3749338435 shutdown_connections 2024-04-02T14:39:17.079 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.078+0000 7f1166600640 1 -- 172.21.15.90:0/3749338435 wait complete. 2024-04-02T14:39:17.080 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.079+0000 7f1166600640 1 Processor -- start 2024-04-02T14:39:17.081 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.079+0000 7f1166600640 1 -- start start 2024-04-02T14:39:17.081 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.080+0000 7f1166600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1160103f20 0x7f116019a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:17.081 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.080+0000 7f1166600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1160108ff0 0x7f116019aae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:17.081 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.080+0000 7f1166600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f116019ee60 0x7f116019f310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:17.081 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.080+0000 7f1166600640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f11601140e0 con 0x7f1160103f20 2024-04-02T14:39:17.081 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.080+0000 7f1166600640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f1160113f60 con 0x7f116019ee60 2024-04-02T14:39:17.081 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.080+0000 7f1166600640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f1160114260 con 0x7f1160108ff0 2024-04-02T14:39:17.082 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.080+0000 7f115f400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1160103f20 0x7f116019a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:17.082 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.080+0000 7f115f400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1160103f20 0x7f116019a5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:60016/0 (socket says 172.21.15.90:60016) 2024-04-02T14:39:17.082 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.080+0000 7f115f400640 1 -- 172.21.15.90:0/1599894974 learned_addr learned my addr 172.21.15.90:0/1599894974 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:39:17.082 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.080+0000 7f1157e00640 1 --2- 172.21.15.90:0/1599894974 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1160108ff0 0x7f116019aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:17.082 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.080+0000 7f115fe00640 1 --2- 172.21.15.90:0/1599894974 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f116019ee60 0x7f116019f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:17.083 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.081+0000 7f1157e00640 1 -- 172.21.15.90:0/1599894974 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f116019ee60 msgr2=0x7f116019f310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:17.083 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.081+0000 7f1157e00640 1 --2- 172.21.15.90:0/1599894974 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f116019ee60 0x7f116019f310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:17.083 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.081+0000 7f1157e00640 1 -- 172.21.15.90:0/1599894974 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1160103f20 msgr2=0x7f116019a5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:17.083 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.081+0000 7f1157e00640 1 --2- 172.21.15.90:0/1599894974 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1160103f20 0x7f116019a5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:17.083 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.081+0000 7f1157e00640 1 -- 172.21.15.90:0/1599894974 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f116019f950 con 0x7f1160108ff0 2024-04-02T14:39:17.083 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.081+0000 7f1157e00640 1 --2- 172.21.15.90:0/1599894974 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1160108ff0 0x7f116019aae0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f1148036040 tx=0x7f114802cbf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:39:17.083 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.082+0000 7f115cc00640 1 -- 172.21.15.90:0/1599894974 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1148041070 con 0x7f1160108ff0 2024-04-02T14:39:17.083 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.082+0000 7f115cc00640 1 -- 172.21.15.90:0/1599894974 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1148004450 con 0x7f1160108ff0 2024-04-02T14:39:17.083 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.082+0000 7f1166600640 1 -- 172.21.15.90:0/1599894974 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f116019fbe0 con 0x7f1160108ff0 2024-04-02T14:39:17.084 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.082+0000 7f115cc00640 1 -- 172.21.15.90:0/1599894974 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1148003e30 con 0x7f1160108ff0 2024-04-02T14:39:17.084 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.082+0000 7f1166600640 1 -- 172.21.15.90:0/1599894974 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11601a7400 con 0x7f1160108ff0 2024-04-02T14:39:17.088 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.086+0000 7f1169380740 1 Processor -- start 2024-04-02T14:39:17.088 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.086+0000 7f1169380740 1 -- start start 2024-04-02T14:39:17.088 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.087+0000 7f1169380740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558eccf9aeb0 0x558eccf9b290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:17.090 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.087+0000 7f1169380740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558eccf9b7d0 0x558eccfa3e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:17.090 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.087+0000 7f1169380740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558eccfa43b0 0x558eccfa67a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:17.090 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.087+0000 7f1169380740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x558ecceffd70 con 0x558eccf9aeb0 2024-04-02T14:39:17.090 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.087+0000 7f1169380740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x558eccea82a0 con 0x558eccf9b7d0 2024-04-02T14:39:17.090 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.087+0000 7f1169380740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x558eccefbd20 con 0x558eccfa43b0 2024-04-02T14:39:17.090 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.087+0000 7f115fe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558eccf9aeb0 0x558eccf9b290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:17.091 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.087+0000 7f115cc00640 1 -- 172.21.15.90:0/1599894974 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f1148044070 con 0x7f1160108ff0 2024-04-02T14:39:17.091 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.087+0000 7f115fe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558eccf9aeb0 0x558eccf9b290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:60026/0 (socket says 172.21.15.90:60026) 2024-04-02T14:39:17.091 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.087+0000 7f115fe00640 1 -- 172.21.15.90:0/3336234631 learned_addr learned my addr 172.21.15.90:0/3336234631 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:39:17.091 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.088+0000 7f115fe00640 1 -- 172.21.15.90:0/3336234631 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558eccfa43b0 msgr2=0x558eccfa67a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:39:17.091 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.088+0000 7f115fe00640 1 --2- 172.21.15.90:0/3336234631 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558eccfa43b0 0x558eccfa67a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:17.091 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.088+0000 7f115fe00640 1 -- 172.21.15.90:0/3336234631 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558eccf9b7d0 msgr2=0x558eccfa3e70 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-02T14:39:17.092 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.088+0000 7f115fe00640 1 --2- 172.21.15.90:0/3336234631 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558eccf9b7d0 0x558eccfa3e70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:17.092 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.088+0000 7f115fe00640 1 -- 172.21.15.90:0/3336234631 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558eccebc840 con 0x558eccf9aeb0 2024-04-02T14:39:17.092 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.088+0000 7f115fe00640 1 --2- 172.21.15.90:0/3336234631 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558eccf9aeb0 0x558eccf9b290 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f115800a430 tx=0x7f1158009650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ebe8b3bfc542b7f9 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:39:17.100 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.089+0000 7f115cc00640 1 --2- 172.21.15.90:0/1599894974 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f11440a3e40 0x7f11440a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:17.100 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.089+0000 7f1154c00640 1 -- 172.21.15.90:0/3336234631 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1158013790 con 0x558eccf9aeb0 2024-04-02T14:39:17.100 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.089+0000 7f1154c00640 1 -- 172.21.15.90:0/3336234631 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1158019070 con 0x558eccf9aeb0 2024-04-02T14:39:17.100 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.089+0000 7f115cc00640 1 -- 172.21.15.90:0/1599894974 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f11480f2490 con 0x7f1160108ff0 2024-04-02T14:39:17.100 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.090+0000 7f1154c00640 1 -- 172.21.15.90:0/3336234631 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1158014e40 con 0x558eccf9aeb0 2024-04-02T14:39:17.100 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.090+0000 7f1169380740 1 -- 172.21.15.90:0/3336234631 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558eccf9aeb0 msgr2=0x558eccf9b290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:17.101 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.090+0000 7f1169380740 1 --2- 172.21.15.90:0/3336234631 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558eccf9aeb0 0x558eccf9b290 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f115800a430 tx=0x7f1158009650 comp rx=0 tx=0).stop 2024-04-02T14:39:17.101 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.090+0000 7f1169380740 1 -- 172.21.15.90:0/3336234631 shutdown_connections 2024-04-02T14:39:17.101 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.090+0000 7f1169380740 1 --2- 172.21.15.90:0/3336234631 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558eccfa43b0 0x558eccfa67a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:17.101 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.090+0000 7f1169380740 1 --2- 172.21.15.90:0/3336234631 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558eccf9b7d0 0x558eccfa3e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:17.101 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.090+0000 7f1169380740 1 --2- 172.21.15.90:0/3336234631 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558eccf9aeb0 0x558eccf9b290 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:17.102 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.090+0000 7f1169380740 1 -- 172.21.15.90:0/3336234631 >> 172.21.15.90:0/3336234631 conn(0x558eccf955e0 msgr2=0x558eccf959c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:39:17.102 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.090+0000 7f1169380740 1 -- 172.21.15.90:0/3336234631 shutdown_connections 2024-04-02T14:39:17.102 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.091+0000 7f1169380740 1 -- 172.21.15.90:0/3336234631 wait complete. 2024-04-02T14:39:17.102 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.091+0000 7f115f400640 1 --2- 172.21.15.90:0/1599894974 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f11440a3e40 0x7f11440a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:17.102 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.091+0000 7f1169380740 1 -- start start 2024-04-02T14:39:17.102 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.091+0000 7f115f400640 1 --2- 172.21.15.90:0/1599894974 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f11440a3e40 0x7f11440a6300 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x558eccf964e0 tx=0x7f1150005f90 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:39:17.103 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.091+0000 7f1169380740 1 Processor -- start 2024-04-02T14:39:17.103 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.092+0000 7f1169380740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558eccf9aeb0 0x558eccfa14f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:17.106 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.092+0000 7f1169380740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558eccfa43b0 0x558eccfa1a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:17.106 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.092+0000 7f1169380740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558eccfa1f70 0x558eccfb06f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:17.106 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.092+0000 7f1169380740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x558eccfa71f0 con 0x558eccfa43b0 2024-04-02T14:39:17.106 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.092+0000 7f1169380740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x558eccebb660 con 0x558eccfa1f70 2024-04-02T14:39:17.107 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.092+0000 7f1169380740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x558eccfa7370 con 0x558eccf9aeb0 2024-04-02T14:39:17.107 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.092+0000 7f1169380740 10 client.0 _ll_register_callbacks cb 0x558eccebb050 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:39:17.107 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.092+0000 7f115f400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558eccf9aeb0 0x558eccfa14f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:17.107 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.092+0000 7f115f400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558eccf9aeb0 0x558eccfa14f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:49366/0 (socket says 172.21.15.90:49366) 2024-04-02T14:39:17.107 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.092+0000 7f115f400640 1 -- 172.21.15.90:0/3739522497 learned_addr learned my addr 172.21.15.90:0/3739522497 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:39:17.107 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.092+0000 7f115fe00640 1 --2- 172.21.15.90:0/3739522497 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558eccfa1f70 0x558eccfb06f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:17.108 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.092+0000 7f115f400640 1 -- 172.21.15.90:0/3739522497 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558eccfa1f70 msgr2=0x558eccfb06f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:39:17.108 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.092+0000 7f115f400640 1 --2- 172.21.15.90:0/3739522497 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558eccfa1f70 0x558eccfb06f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:17.108 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.092+0000 7f115f400640 1 -- 172.21.15.90:0/3739522497 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558eccfa43b0 msgr2=0x558eccfa1a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:39:17.108 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.092+0000 7f115f400640 1 --2- 172.21.15.90:0/3739522497 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558eccfa43b0 0x558eccfa1a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:17.108 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.093+0000 7f115f400640 1 -- 172.21.15.90:0/3739522497 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558eccecae80 con 0x558eccf9aeb0 2024-04-02T14:39:17.108 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.093+0000 7f115f400640 1 --2- 172.21.15.90:0/3739522497 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558eccf9aeb0 0x558eccfa14f0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f115001c750 tx=0x7f1150004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7d831020d9255587 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:39:17.108 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.093+0000 7f114d600640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:39:17.109 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.093+0000 7f114d600640 1 -- 172.21.15.90:0/3739522497 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1150004220 con 0x558eccf9aeb0 2024-04-02T14:39:17.109 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.093+0000 7f114d600640 1 -- 172.21.15.90:0/3739522497 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f11500043c0 con 0x558eccf9aeb0 2024-04-02T14:39:17.109 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.093+0000 7f1169380740 1 -- 172.21.15.90:0/3739522497 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1158019070 con 0x558eccf9aeb0 2024-04-02T14:39:17.109 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.094+0000 7f114d600640 1 -- 172.21.15.90:0/3739522497 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f115001ce80 con 0x558eccf9aeb0 2024-04-02T14:39:17.109 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.094+0000 7f1169380740 10 client.26665 fetch_fsmap learned FSMap version 27 2024-04-02T14:39:17.109 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.094+0000 7f1169380740 1 -- 172.21.15.90:0/3739522497 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x558eccc14210 con 0x558eccf9aeb0 2024-04-02T14:39:17.110 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.094+0000 7f114d600640 1 -- 172.21.15.90:0/3739522497 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1158019070 con 0x558eccf9aeb0 2024-04-02T14:39:17.110 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.094+0000 7f114d600640 1 -- 172.21.15.90:0/3739522497 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f1150003ae0 con 0x558eccf9aeb0 2024-04-02T14:39:17.110 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.094+0000 7f1169380740 10 client.26665 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:39:17.110 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.094+0000 7f1169380740 10 client.26665 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-02T14:39:17.110 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.094+0000 7f1169380740 20 client.26665 populate_metadata read hostname 'smithi090' 2024-04-02T14:39:17.110 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.094+0000 7f1169380740 1 --2- 172.21.15.90:0/3739522497 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x558eccfb4e00 0x558eccfb7210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:17.111 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.094+0000 7f1169380740 4 client.26665 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:39:17.111 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.094+0000 7f1169380740 1 -- 172.21.15.90:0/3739522497 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558ecceafcd0 con 0x558eccfb4e00 2024-04-02T14:39:17.111 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.095+0000 7f1157e00640 1 --2- 172.21.15.90:0/3739522497 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x558eccfb4e00 0x558eccfb7210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:17.111 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.095+0000 7f1157e00640 1 --2- 172.21.15.90:0/3739522497 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x558eccfb4e00 0x558eccfb7210 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a8c6b9587a4c9765 server_cookie=5fa919ecaf272e3b in_seq=0 out_seq=0 2024-04-02T14:39:17.111 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.095+0000 7f114d600640 10 client.26665 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:39:17.111 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.096+0000 7f114d600640 1 -- 172.21.15.90:0/3739522497 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x558ecceafcd0 con 0x558eccfb4e00 2024-04-02T14:39:17.111 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.096+0000 7f114d600640 10 client.26665 handle_command_reply: tid=0 2024-04-02T14:39:17.111 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.096+0000 7f1169380740 1 client.26665 shutdown 2024-04-02T14:39:17.112 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.096+0000 7f1169380740 1 -- 172.21.15.90:0/3739522497 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558eccf9aeb0 msgr2=0x558eccfa14f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:17.112 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.096+0000 7f1169380740 1 --2- 172.21.15.90:0/3739522497 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558eccf9aeb0 0x558eccfa14f0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f115001c750 tx=0x7f1150004730 comp rx=0 tx=0).stop 2024-04-02T14:39:17.112 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.096+0000 7f1169380740 1 -- 172.21.15.90:0/3739522497 shutdown_connections 2024-04-02T14:39:17.112 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.096+0000 7f1169380740 1 --2- 172.21.15.90:0/3739522497 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x558eccfb4e00 0x558eccfb7210 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:17.112 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.096+0000 7f1169380740 1 --2- 172.21.15.90:0/3739522497 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558eccfa1f70 0x558eccfb06f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:17.112 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.096+0000 7f1169380740 1 --2- 172.21.15.90:0/3739522497 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558eccfa43b0 0x558eccfa1a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:17.112 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.096+0000 7f114d600640 0 client.26665 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:39:17.112 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.096+0000 7f1169380740 1 --2- 172.21.15.90:0/3739522497 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558eccf9aeb0 0x558eccfa14f0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:17.112 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.096+0000 7f1169380740 1 -- 172.21.15.90:0/3739522497 >> 172.21.15.90:0/3739522497 conn(0x558eccf955e0 msgr2=0x558eccf97330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:39:17.113 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.097+0000 7f1169380740 1 -- 172.21.15.90:0/3739522497 shutdown_connections 2024-04-02T14:39:17.113 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.097+0000 7f1169380740 1 -- 172.21.15.90:0/3739522497 wait complete. 2024-04-02T14:39:17.113 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.097+0000 7f1169380740 20 client.26665 trim_cache size 0 max 16384 2024-04-02T14:39:17.118 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.115+0000 7f1169380740 1 Processor -- start 2024-04-02T14:39:17.118 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.115+0000 7f1169380740 1 -- start start 2024-04-02T14:39:17.118 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.115+0000 7f1169380740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558eccf9aeb0 0x558eccfbcb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:17.118 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.115+0000 7f1169380740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558eccfa1f70 0x558eccfbd060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:17.118 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.115+0000 7f1169380740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558eccfa43b0 0x558eccfb0140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:17.118 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.115+0000 7f1169380740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f11601afa30 con 0x558eccfa1f70 2024-04-02T14:39:17.118 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.115+0000 7f1169380740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f11601b1570 con 0x558eccfa43b0 2024-04-02T14:39:17.118 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.115+0000 7f1169380740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f11601b0100 con 0x558eccf9aeb0 2024-04-02T14:39:17.118 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.115+0000 7f115f400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558eccf9aeb0 0x558eccfbcb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:17.119 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.115+0000 7f115f400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558eccf9aeb0 0x558eccfbcb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:49374/0 (socket says 172.21.15.90:49374) 2024-04-02T14:39:17.119 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.116+0000 7f115f400640 1 -- 172.21.15.90:0/229581582 learned_addr learned my addr 172.21.15.90:0/229581582 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:39:17.119 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.116+0000 7f115fe00640 1 --2- 172.21.15.90:0/229581582 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558eccfa43b0 0x558eccfb0140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:17.119 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.116+0000 7f1157e00640 1 --2- 172.21.15.90:0/229581582 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558eccfa1f70 0x558eccfbd060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:17.119 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.116+0000 7f115f400640 1 -- 172.21.15.90:0/229581582 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558eccfa43b0 msgr2=0x558eccfb0140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:39:17.119 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.116+0000 7f115f400640 1 --2- 172.21.15.90:0/229581582 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558eccfa43b0 0x558eccfb0140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:17.119 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.116+0000 7f115f400640 1 -- 172.21.15.90:0/229581582 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558eccfa1f70 msgr2=0x558eccfbd060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:39:17.120 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.116+0000 7f115f400640 1 --2- 172.21.15.90:0/229581582 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558eccfa1f70 0x558eccfbd060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:17.120 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.116+0000 7f115f400640 1 -- 172.21.15.90:0/229581582 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558eccce0620 con 0x558eccf9aeb0 2024-04-02T14:39:17.120 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.116+0000 7f115f400640 1 --2- 172.21.15.90:0/229581582 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558eccf9aeb0 0x558eccfbcb20 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f11500059f0 tx=0x7f11500275a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=159167d8242be436 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:39:17.120 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.116+0000 7f114fe00640 1 -- 172.21.15.90:0/229581582 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1150004200 con 0x558eccf9aeb0 2024-04-02T14:39:17.120 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.116+0000 7f114fe00640 1 -- 172.21.15.90:0/229581582 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f11500043a0 con 0x558eccf9aeb0 2024-04-02T14:39:17.120 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.116+0000 7f114fe00640 1 -- 172.21.15.90:0/229581582 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1150003bd0 con 0x558eccf9aeb0 2024-04-02T14:39:17.122 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.117+0000 7f1169380740 1 -- 172.21.15.90:0/229581582 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558eccf9aeb0 msgr2=0x558eccfbcb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:17.123 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.117+0000 7f1169380740 1 --2- 172.21.15.90:0/229581582 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558eccf9aeb0 0x558eccfbcb20 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f11500059f0 tx=0x7f11500275a0 comp rx=0 tx=0).stop 2024-04-02T14:39:17.123 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.117+0000 7f1169380740 1 -- 172.21.15.90:0/229581582 shutdown_connections 2024-04-02T14:39:17.123 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.117+0000 7f1169380740 1 --2- 172.21.15.90:0/229581582 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558eccfa43b0 0x558eccfb0140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:17.123 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.117+0000 7f1169380740 1 --2- 172.21.15.90:0/229581582 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558eccfa1f70 0x558eccfbd060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:17.123 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.117+0000 7f1169380740 1 --2- 172.21.15.90:0/229581582 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558eccf9aeb0 0x558eccfbcb20 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:17.123 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.117+0000 7f1169380740 1 -- 172.21.15.90:0/229581582 >> 172.21.15.90:0/229581582 conn(0x558eccf955e0 msgr2=0x558eccfa14b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:39:17.123 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.117+0000 7f1169380740 1 -- 172.21.15.90:0/229581582 shutdown_connections 2024-04-02T14:39:17.123 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.117+0000 7f1169380740 1 -- 172.21.15.90:0/229581582 wait complete. 2024-04-02T14:39:17.123 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.119+0000 7f1169380740 1 -- start start 2024-04-02T14:39:17.123 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.119+0000 7f1169380740 1 Processor -- start 2024-04-02T14:39:17.123 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.120+0000 7f1169380740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558eccf9aeb0 0x558eccf9e340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:17.123 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.120+0000 7f1169380740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558eccfa1f70 0x558eccf9e880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:17.123 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.120+0000 7f1169380740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558eccfa43b0 0x558eccf9edc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:17.123 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.120+0000 7f1169380740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f11601b0ea0 con 0x558eccfa1f70 2024-04-02T14:39:17.124 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.120+0000 7f1169380740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f11601b07d0 con 0x558eccf9aeb0 2024-04-02T14:39:17.124 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.120+0000 7f1169380740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f11601ada30 con 0x558eccfa43b0 2024-04-02T14:39:17.124 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.120+0000 7f1169380740 10 client.0 _ll_register_callbacks cb 0x7f1150011f30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:39:17.124 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.120+0000 7f1157e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558eccfa1f70 0x558eccf9e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:17.124 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.120+0000 7f1157e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558eccfa1f70 0x558eccf9e880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:60054/0 (socket says 172.21.15.90:60054) 2024-04-02T14:39:17.124 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.120+0000 7f1157e00640 1 -- 172.21.15.90:0/3610139539 learned_addr learned my addr 172.21.15.90:0/3610139539 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:39:17.124 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.120+0000 7f1157e00640 1 -- 172.21.15.90:0/3610139539 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558eccfa43b0 msgr2=0x558eccf9edc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:39:17.124 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.120+0000 7f1157e00640 1 --2- 172.21.15.90:0/3610139539 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558eccfa43b0 0x558eccf9edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:17.124 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.120+0000 7f1157e00640 1 -- 172.21.15.90:0/3610139539 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558eccf9aeb0 msgr2=0x558eccf9e340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:39:17.124 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.120+0000 7f1157e00640 1 --2- 172.21.15.90:0/3610139539 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558eccf9aeb0 0x558eccf9e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:17.124 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.120+0000 7f1157e00640 1 -- 172.21.15.90:0/3610139539 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558ecd01a560 con 0x558eccfa1f70 2024-04-02T14:39:17.124 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.121+0000 7f1157e00640 1 --2- 172.21.15.90:0/3610139539 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558eccfa1f70 0x558eccf9e880 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f11480be300 tx=0x7f114802c1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=758fa9eb7f7c6168 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:39:17.124 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.121+0000 7f114d600640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-02T14:39:17.124 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.121+0000 7f114d600640 1 -- 172.21.15.90:0/3610139539 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f114804b040 con 0x558eccfa1f70 2024-04-02T14:39:17.125 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.121+0000 7f1169380740 1 -- 172.21.15.90:0/3610139539 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f11500043a0 con 0x558eccfa1f70 2024-04-02T14:39:17.125 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.121+0000 7f114d600640 1 -- 172.21.15.90:0/3610139539 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f11480efe70 con 0x558eccfa1f70 2024-04-02T14:39:17.125 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.121+0000 7f114d600640 1 -- 172.21.15.90:0/3610139539 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f11480be140 con 0x558eccfa1f70 2024-04-02T14:39:17.125 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.121+0000 7f1169380740 10 client.16284 fetch_fsmap learned FSMap version 27 2024-04-02T14:39:17.125 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.121+0000 7f1169380740 1 -- 172.21.15.90:0/3610139539 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x558ecd01b890 con 0x558eccfa1f70 2024-04-02T14:39:17.126 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.122+0000 7f114d600640 1 -- 172.21.15.90:0/3610139539 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f11500043a0 con 0x558eccfa1f70 2024-04-02T14:39:17.126 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.122+0000 7f114d600640 1 -- 172.21.15.90:0/3610139539 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f11480be860 con 0x558eccfa1f70 2024-04-02T14:39:17.126 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.122+0000 7f1169380740 10 client.16284 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:39:17.126 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.122+0000 7f1169380740 10 client.16284 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-02T14:39:17.126 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.122+0000 7f1169380740 20 client.16284 populate_metadata read hostname 'smithi090' 2024-04-02T14:39:17.126 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.122+0000 7f1169380740 1 --2- 172.21.15.90:0/3610139539 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x558ecd01bfe0 0x558ecd01e460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:17.126 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.122+0000 7f1169380740 4 client.16284 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-02T14:39:17.126 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.122+0000 7f1169380740 1 -- 172.21.15.90:0/3610139539 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "flush journal"}) -- 0x558eccf0d2b0 con 0x558ecd01bfe0 2024-04-02T14:39:17.126 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.123+0000 7f115f400640 1 --2- 172.21.15.90:0/3610139539 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x558ecd01bfe0 0x558ecd01e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:17.127 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.126+0000 7f115f400640 1 --2- 172.21.15.90:0/3610139539 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x558ecd01bfe0 0x558ecd01e460 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7615fd65e30832db server_cookie=25e4a7f9f60e266f in_seq=0 out_seq=0 2024-04-02T14:39:17.127 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.126+0000 7f114d600640 10 client.16284 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:39:17.160 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.158+0000 7f114d600640 1 -- 172.21.15.90:0/3610139539 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x558eccf0d2b0 con 0x558ecd01bfe0 2024-04-02T14:39:17.160 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.158+0000 7f114d600640 10 client.16284 handle_command_reply: tid=0 2024-04-02T14:39:17.161 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.158+0000 7f1169380740 1 client.16284 shutdown 2024-04-02T14:39:17.161 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.159+0000 7f1169380740 1 -- 172.21.15.90:0/3610139539 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558eccfa1f70 msgr2=0x558eccf9e880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:17.161 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.159+0000 7f1169380740 1 --2- 172.21.15.90:0/3610139539 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558eccfa1f70 0x558eccf9e880 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f11480be300 tx=0x7f114802c1f0 comp rx=0 tx=0).stop 2024-04-02T14:39:17.161 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.159+0000 7f1169380740 1 -- 172.21.15.90:0/3610139539 shutdown_connections 2024-04-02T14:39:17.161 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.159+0000 7f1169380740 1 --2- 172.21.15.90:0/3610139539 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x558ecd01bfe0 0x558ecd01e460 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:17.161 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.159+0000 7f1169380740 1 --2- 172.21.15.90:0/3610139539 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558eccfa43b0 0x558eccf9edc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:17.161 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.159+0000 7f1169380740 1 --2- 172.21.15.90:0/3610139539 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558eccfa1f70 0x558eccf9e880 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:17.161 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.159+0000 7f114d600640 0 client.16284 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:39:17.161 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.159+0000 7f1169380740 1 --2- 172.21.15.90:0/3610139539 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558eccf9aeb0 0x558eccf9e340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:17.161 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.159+0000 7f1169380740 1 -- 172.21.15.90:0/3610139539 >> 172.21.15.90:0/3610139539 conn(0x558eccf955e0 msgr2=0x558eccf96a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:39:17.161 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.159+0000 7f1169380740 1 -- 172.21.15.90:0/3610139539 shutdown_connections 2024-04-02T14:39:17.161 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.159+0000 7f1169380740 1 -- 172.21.15.90:0/3610139539 wait complete. 2024-04-02T14:39:17.161 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.159+0000 7f1169380740 20 client.16284 trim_cache size 0 max 16384 2024-04-02T14:39:17.162 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:{ 2024-04-02T14:39:17.162 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "message": "", 2024-04-02T14:39:17.162 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "return_code": 0 2024-04-02T14:39:17.162 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:} 2024-04-02T14:39:17.163 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.160+0000 7f1166600640 1 -- 172.21.15.90:0/1599894974 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f11440a3e40 msgr2=0x7f11440a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:17.163 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.160+0000 7f1166600640 1 --2- 172.21.15.90:0/1599894974 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f11440a3e40 0x7f11440a6300 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x558eccf964e0 tx=0x7f1150005f90 comp rx=0 tx=0).stop 2024-04-02T14:39:17.163 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.160+0000 7f1166600640 1 -- 172.21.15.90:0/1599894974 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1160108ff0 msgr2=0x7f116019aae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:17.163 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.160+0000 7f1166600640 1 --2- 172.21.15.90:0/1599894974 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1160108ff0 0x7f116019aae0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f1148036040 tx=0x7f114802cbf0 comp rx=0 tx=0).stop 2024-04-02T14:39:17.163 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.160+0000 7f1166600640 1 -- 172.21.15.90:0/1599894974 shutdown_connections 2024-04-02T14:39:17.163 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.160+0000 7f1166600640 1 --2- 172.21.15.90:0/1599894974 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f11440a3e40 0x7f11440a6300 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:17.163 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.160+0000 7f1166600640 1 --2- 172.21.15.90:0/1599894974 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f116019ee60 0x7f116019f310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:17.163 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.160+0000 7f1166600640 1 --2- 172.21.15.90:0/1599894974 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1160108ff0 0x7f116019aae0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:17.164 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.160+0000 7f1166600640 1 --2- 172.21.15.90:0/1599894974 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1160103f20 0x7f116019a5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:17.164 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.160+0000 7f1166600640 1 -- 172.21.15.90:0/1599894974 >> 172.21.15.90:0/1599894974 conn(0x7f11600ff780 msgr2=0x7f1160101a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:39:17.164 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.160+0000 7f1166600640 1 -- 172.21.15.90:0/1599894974 shutdown_connections 2024-04-02T14:39:17.164 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:17.160+0000 7f1166600640 1 -- 172.21.15.90:0/1599894974 wait complete. 2024-04-02T14:39:18.360 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:39:17 smithi092 ceph-mon[36497]: pgmap v135: 97 pgs: 97 active+clean; 2.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 144 KiB/s rd, 47 MiB/s wr, 1.56k op/s 2024-04-02T14:39:18.437 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:39:17 smithi177 ceph-mon[36431]: pgmap v135: 97 pgs: 97 active+clean; 2.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 144 KiB/s rd, 47 MiB/s wr, 1.56k op/s 2024-04-02T14:39:18.438 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:39:17 smithi090 ceph-mon[31674]: pgmap v135: 97 pgs: 97 active+clean; 2.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 144 KiB/s rd, 47 MiB/s wr, 1.56k op/s 2024-04-02T14:39:20.360 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:39:19 smithi092 ceph-mon[36497]: pgmap v136: 97 pgs: 97 active+clean; 2.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 172 KiB/s rd, 58 MiB/s wr, 1.80k op/s 2024-04-02T14:39:20.436 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:39:19 smithi177 ceph-mon[36431]: pgmap v136: 97 pgs: 97 active+clean; 2.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 172 KiB/s rd, 58 MiB/s wr, 1.80k op/s 2024-04-02T14:39:20.438 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:39:19 smithi090 ceph-mon[31674]: pgmap v136: 97 pgs: 97 active+clean; 2.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 172 KiB/s rd, 58 MiB/s wr, 1.80k op/s 2024-04-02T14:39:21.831 INFO:teuthology.orchestra.run.smithi090.stderr:progress: 230.0 s, 2812.2 tps, lat 35.585 ms stddev 47.255 2024-04-02T14:39:22.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:39:21 smithi092 ceph-mon[36497]: pgmap v137: 97 pgs: 97 active+clean; 2.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 36 MiB/s wr, 964 op/s 2024-04-02T14:39:22.437 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:39:21 smithi177 ceph-mon[36431]: pgmap v137: 97 pgs: 97 active+clean; 2.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 36 MiB/s wr, 964 op/s 2024-04-02T14:39:22.438 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:39:21 smithi090 ceph-mon[31674]: pgmap v137: 97 pgs: 97 active+clean; 2.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 36 MiB/s wr, 964 op/s 2024-04-02T14:39:24.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:39:23 smithi092 ceph-mon[36497]: pgmap v138: 97 pgs: 97 active+clean; 2.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 28 MiB/s wr, 615 op/s 2024-04-02T14:39:24.437 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:39:23 smithi177 ceph-mon[36431]: pgmap v138: 97 pgs: 97 active+clean; 2.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 28 MiB/s wr, 615 op/s 2024-04-02T14:39:24.439 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:39:23 smithi090 ceph-mon[31674]: pgmap v138: 97 pgs: 97 active+clean; 2.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 28 MiB/s wr, 615 op/s 2024-04-02T14:39:26.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:39:26 smithi092 ceph-mon[36497]: pgmap v139: 97 pgs: 97 active+clean; 2.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 40 MiB/s wr, 918 op/s 2024-04-02T14:39:26.436 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:39:26 smithi177 ceph-mon[36431]: pgmap v139: 97 pgs: 97 active+clean; 2.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 40 MiB/s wr, 918 op/s 2024-04-02T14:39:26.438 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:39:26 smithi090 ceph-mon[31674]: pgmap v139: 97 pgs: 97 active+clean; 2.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 40 MiB/s wr, 918 op/s 2024-04-02T14:39:26.833 INFO:teuthology.orchestra.run.smithi090.stderr:progress: 235.0 s, 2593.0 tps, lat 38.459 ms stddev 47.594 2024-04-02T14:39:28.360 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:39:28 smithi092 ceph-mon[36497]: pgmap v140: 97 pgs: 97 active+clean; 2.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 25 MiB/s wr, 613 op/s 2024-04-02T14:39:28.436 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:39:28 smithi177 ceph-mon[36431]: pgmap v140: 97 pgs: 97 active+clean; 2.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 25 MiB/s wr, 613 op/s 2024-04-02T14:39:28.438 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:39:28 smithi090 ceph-mon[31674]: pgmap v140: 97 pgs: 97 active+clean; 2.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 25 MiB/s wr, 613 op/s 2024-04-02T14:39:30.355 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.353+0000 7fa9a1800640 1 -- 172.21.15.90:0/1595255821 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa99c0ffbd0 msgr2=0x7fa99c0fffb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:30.355 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.353+0000 7fa9a1800640 1 --2- 172.21.15.90:0/1595255821 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa99c0ffbd0 0x7fa99c0fffb0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fa994009900 tx=0x7fa99402bc30 comp rx=0 tx=0).stop 2024-04-02T14:39:30.355 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.354+0000 7fa9a1800640 1 -- 172.21.15.90:0/1595255821 shutdown_connections 2024-04-02T14:39:30.356 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.354+0000 7fa9a1800640 1 --2- 172.21.15.90:0/1595255821 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa99c1128f0 0x7fa99c114ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:30.356 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.354+0000 7fa9a1800640 1 --2- 172.21.15.90:0/1595255821 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa99c1004f0 0x7fa99c1123b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:30.356 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.354+0000 7fa9a1800640 1 --2- 172.21.15.90:0/1595255821 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa99c0ffbd0 0x7fa99c0fffb0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:30.356 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.354+0000 7fa9a1800640 1 -- 172.21.15.90:0/1595255821 >> 172.21.15.90:0/1595255821 conn(0x7fa99c076b90 msgr2=0x7fa99c0fdee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:39:30.356 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.354+0000 7fa9a1800640 1 -- 172.21.15.90:0/1595255821 shutdown_connections 2024-04-02T14:39:30.356 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.354+0000 7fa9a1800640 1 -- 172.21.15.90:0/1595255821 wait complete. 2024-04-02T14:39:30.358 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.357+0000 7fa9a1800640 1 Processor -- start 2024-04-02T14:39:30.358 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.357+0000 7fa9a1800640 1 -- start start 2024-04-02T14:39:30.359 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.358+0000 7fa9a1800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa99c0ffbd0 0x7fa99c1a4650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:30.359 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.358+0000 7fa9a1800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa99c1004f0 0x7fa99c1a4b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:30.359 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.358+0000 7fa9a1800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa99c1128f0 0x7fa99c19e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:30.360 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.358+0000 7fa99aa00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa99c0ffbd0 0x7fa99c1a4650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:30.360 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:39:30 smithi092 ceph-mon[36497]: pgmap v141: 97 pgs: 97 active+clean; 3.0 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 33 MiB/s wr, 872 op/s 2024-04-02T14:39:30.361 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.359+0000 7fa9a1800640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fa99c118560 con 0x7fa99c0ffbd0 2024-04-02T14:39:30.361 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.359+0000 7fa99b400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa99c1128f0 0x7fa99c19e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:30.361 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.359+0000 7fa99b400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa99c1128f0 0x7fa99c19e7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:44518/0 (socket says 172.21.15.90:44518) 2024-04-02T14:39:30.361 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.359+0000 7fa99b400640 1 -- 172.21.15.90:0/68145546 learned_addr learned my addr 172.21.15.90:0/68145546 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:39:30.362 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.359+0000 7fa99a000640 1 --2- 172.21.15.90:0/68145546 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa99c1004f0 0x7fa99c1a4b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:30.362 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.359+0000 7fa9a1800640 1 -- 172.21.15.90:0/68145546 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fa99c1183e0 con 0x7fa99c1004f0 2024-04-02T14:39:30.362 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.359+0000 7fa9a1800640 1 -- 172.21.15.90:0/68145546 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fa99c1186e0 con 0x7fa99c1128f0 2024-04-02T14:39:30.362 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.360+0000 7fa99b400640 1 -- 172.21.15.90:0/68145546 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa99c1004f0 msgr2=0x7fa99c1a4b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:30.362 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.360+0000 7fa99b400640 1 --2- 172.21.15.90:0/68145546 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa99c1004f0 0x7fa99c1a4b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:30.362 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.360+0000 7fa99b400640 1 -- 172.21.15.90:0/68145546 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa99c0ffbd0 msgr2=0x7fa99c1a4650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:30.362 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.360+0000 7fa99b400640 1 --2- 172.21.15.90:0/68145546 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa99c0ffbd0 0x7fa99c1a4650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:30.362 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.360+0000 7fa99b400640 1 -- 172.21.15.90:0/68145546 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa99c19ef90 con 0x7fa99c1128f0 2024-04-02T14:39:30.362 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.360+0000 7fa99aa00640 1 --2- 172.21.15.90:0/68145546 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa99c0ffbd0 0x7fa99c1a4650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:39:30.362 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.360+0000 7fa99a000640 1 --2- 172.21.15.90:0/68145546 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa99c1004f0 0x7fa99c1a4b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:39:30.362 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.360+0000 7fa99b400640 1 --2- 172.21.15.90:0/68145546 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa99c1128f0 0x7fa99c19e7c0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fa98c00e9d0 tx=0x7fa98c00ee90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:39:30.363 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.360+0000 7fa993400640 1 -- 172.21.15.90:0/68145546 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa98c00cd80 con 0x7fa99c1128f0 2024-04-02T14:39:30.363 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.361+0000 7fa993400640 1 -- 172.21.15.90:0/68145546 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa98c005020 con 0x7fa99c1128f0 2024-04-02T14:39:30.363 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.361+0000 7fa993400640 1 -- 172.21.15.90:0/68145546 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa98c011760 con 0x7fa99c1128f0 2024-04-02T14:39:30.363 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.361+0000 7fa9a1800640 1 -- 172.21.15.90:0/68145546 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa99c19f280 con 0x7fa99c1128f0 2024-04-02T14:39:30.363 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.362+0000 7fa9a1800640 1 -- 172.21.15.90:0/68145546 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa99c1ab4a0 con 0x7fa99c1128f0 2024-04-02T14:39:30.372 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.371+0000 7fa9a4563740 1 Processor -- start 2024-04-02T14:39:30.377 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.372+0000 7fa9a4563740 1 -- start start 2024-04-02T14:39:30.377 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.373+0000 7fa9a4563740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b120e9ceb0 0x55b120e9d290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:30.378 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.372+0000 7fa993400640 1 -- 172.21.15.90:0/68145546 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fa98c0051d0 con 0x7fa99c1128f0 2024-04-02T14:39:30.378 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.376+0000 7fa9a4563740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b120e9d7d0 0x55b120ea5e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:30.378 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.376+0000 7fa99aa00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b120e9d7d0 0x55b120ea5e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:30.378 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.377+0000 7fa99aa00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b120e9d7d0 0x55b120ea5e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:57410/0 (socket says 172.21.15.90:57410) 2024-04-02T14:39:30.382 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.378+0000 7fa993400640 1 --2- 172.21.15.90:0/68145546 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fa97c0a3ec0 0x7fa97c0a6380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:30.385 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.378+0000 7fa993400640 1 -- 172.21.15.90:0/68145546 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7fa98c0d0cb0 con 0x7fa99c1128f0 2024-04-02T14:39:30.385 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.381+0000 7fa99a000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b120e9ceb0 0x55b120e9d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:30.385 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.381+0000 7fa99a000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b120e9ceb0 0x55b120e9d290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:44084/0 (socket says 172.21.15.90:44084) 2024-04-02T14:39:30.385 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.381+0000 7fa9a4563740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b120ea63b0 0x55b120ea87a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:30.385 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.381+0000 7fa9a4563740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55b120e01d70 con 0x55b120e9ceb0 2024-04-02T14:39:30.385 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.381+0000 7fa9a4563740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55b120daa2a0 con 0x55b120e9d7d0 2024-04-02T14:39:30.385 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.381+0000 7fa9a4563740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55b120dfdd20 con 0x55b120ea63b0 2024-04-02T14:39:30.385 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.381+0000 7fa99aa00640 1 -- 172.21.15.90:0/1599309488 learned_addr learned my addr 172.21.15.90:0/1599309488 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:39:30.385 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.381+0000 7fa99b400640 1 --2- 172.21.15.90:0/68145546 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fa97c0a3ec0 0x7fa97c0a6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:30.385 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.382+0000 7fa99b400640 1 --2- 172.21.15.90:0/68145546 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fa97c0a3ec0 0x7fa97c0a6380 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fa98c005ba0 tx=0x7fa98c00c7f0 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:39:30.386 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.383+0000 7fa99a000640 1 --2- 172.21.15.90:0/1599309488 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b120ea63b0 0x55b120ea87a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:30.386 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.383+0000 7fa99aa00640 1 -- 172.21.15.90:0/1599309488 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b120ea63b0 msgr2=0x55b120ea87a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:39:30.386 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.383+0000 7fa99aa00640 1 --2- 172.21.15.90:0/1599309488 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b120ea63b0 0x55b120ea87a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:30.386 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.383+0000 7fa99aa00640 1 -- 172.21.15.90:0/1599309488 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b120e9ceb0 msgr2=0x55b120e9d290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:39:30.386 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.383+0000 7fa99aa00640 1 --2- 172.21.15.90:0/1599309488 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b120e9ceb0 0x55b120e9d290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:30.386 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.383+0000 7fa99aa00640 1 -- 172.21.15.90:0/1599309488 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b120dbe840 con 0x55b120e9d7d0 2024-04-02T14:39:30.386 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.384+0000 7fa99aa00640 1 --2- 172.21.15.90:0/1599309488 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b120e9d7d0 0x55b120ea5e70 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fa994009900 tx=0x7fa99402c650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dad421f6ceb6a75a server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:39:30.387 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.385+0000 7fa98be00640 1 -- 172.21.15.90:0/1599309488 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa994044020 con 0x55b120e9d7d0 2024-04-02T14:39:30.387 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.385+0000 7fa98be00640 1 -- 172.21.15.90:0/1599309488 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa994004380 con 0x55b120e9d7d0 2024-04-02T14:39:30.387 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.385+0000 7fa98be00640 1 -- 172.21.15.90:0/1599309488 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa994004920 con 0x55b120e9d7d0 2024-04-02T14:39:30.388 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.387+0000 7fa9a4563740 1 -- 172.21.15.90:0/1599309488 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b120e9d7d0 msgr2=0x55b120ea5e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:30.388 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.387+0000 7fa9a4563740 1 --2- 172.21.15.90:0/1599309488 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b120e9d7d0 0x55b120ea5e70 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fa994009900 tx=0x7fa99402c650 comp rx=0 tx=0).stop 2024-04-02T14:39:30.390 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.389+0000 7fa9a4563740 1 -- 172.21.15.90:0/1599309488 shutdown_connections 2024-04-02T14:39:30.390 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.389+0000 7fa9a4563740 1 --2- 172.21.15.90:0/1599309488 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b120ea63b0 0x55b120ea87a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:30.390 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.389+0000 7fa9a4563740 1 --2- 172.21.15.90:0/1599309488 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b120e9d7d0 0x55b120ea5e70 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:30.391 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.389+0000 7fa9a4563740 1 --2- 172.21.15.90:0/1599309488 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b120e9ceb0 0x55b120e9d290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:30.391 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.389+0000 7fa9a4563740 1 -- 172.21.15.90:0/1599309488 >> 172.21.15.90:0/1599309488 conn(0x55b120e975e0 msgr2=0x55b120e979c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:39:30.392 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.390+0000 7fa9a4563740 1 -- 172.21.15.90:0/1599309488 shutdown_connections 2024-04-02T14:39:30.392 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.390+0000 7fa9a4563740 1 -- 172.21.15.90:0/1599309488 wait complete. 2024-04-02T14:39:30.392 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.390+0000 7fa9a4563740 1 -- start start 2024-04-02T14:39:30.392 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.391+0000 7fa9a4563740 1 Processor -- start 2024-04-02T14:39:30.393 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.392+0000 7fa9a4563740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b120e9ceb0 0x55b120ea34a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:30.393 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.392+0000 7fa99a000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b120e9ceb0 0x55b120ea34a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:30.393 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.392+0000 7fa99a000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b120e9ceb0 0x55b120ea34a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:57412/0 (socket says 172.21.15.90:57412) 2024-04-02T14:39:30.394 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.392+0000 7fa9a4563740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b120e9d7d0 0x55b120ea39e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:30.395 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.393+0000 7fa9a4563740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b120ea63b0 0x55b120ea3f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:30.395 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.393+0000 7fa9a4563740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55b120ea9140 con 0x55b120ea63b0 2024-04-02T14:39:30.395 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.393+0000 7fa9a4563740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55b120dbd660 con 0x55b120e9ceb0 2024-04-02T14:39:30.395 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.393+0000 7fa9a4563740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55b120ea92c0 con 0x55b120e9d7d0 2024-04-02T14:39:30.395 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.393+0000 7fa9a4563740 10 client.0 _ll_register_callbacks cb 0x55b120dbd050 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:39:30.395 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.393+0000 7fa99a000640 1 -- 172.21.15.90:0/936693067 learned_addr learned my addr 172.21.15.90:0/936693067 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:39:30.395 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.394+0000 7fa99aa00640 1 --2- 172.21.15.90:0/936693067 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b120e9d7d0 0x55b120ea39e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:30.396 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.394+0000 7fa99a000640 1 --2- 172.21.15.90:0/936693067 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b120ea63b0 0x55b120ea3f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:30.396 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.395+0000 7fa99aa00640 1 -- 172.21.15.90:0/936693067 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b120e9ceb0 msgr2=0x55b120ea34a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:39:30.396 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.395+0000 7fa99aa00640 1 --2- 172.21.15.90:0/936693067 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b120e9ceb0 0x55b120ea34a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:30.396 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.395+0000 7fa99aa00640 1 -- 172.21.15.90:0/936693067 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b120ea63b0 msgr2=0x55b120ea3f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:39:30.397 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.395+0000 7fa99aa00640 1 --2- 172.21.15.90:0/936693067 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b120ea63b0 0x55b120ea3f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:30.397 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.395+0000 7fa99aa00640 1 -- 172.21.15.90:0/936693067 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b120dcce80 con 0x55b120e9d7d0 2024-04-02T14:39:30.397 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.395+0000 7fa99a000640 1 --2- 172.21.15.90:0/936693067 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b120ea63b0 0x55b120ea3f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:39:30.399 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.397+0000 7fa99aa00640 1 --2- 172.21.15.90:0/936693067 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b120e9d7d0 0x55b120ea39e0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fa994000c00 tx=0x7fa99402bc30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e67d44413a26bc59 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:39:30.399 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.397+0000 7fa988c00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:39:30.399 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.398+0000 7fa988c00640 1 -- 172.21.15.90:0/936693067 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa99404f020 con 0x55b120e9d7d0 2024-04-02T14:39:30.401 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.399+0000 7fa988c00640 1 -- 172.21.15.90:0/936693067 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7fa994003b30 con 0x55b120e9d7d0 2024-04-02T14:39:30.401 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.400+0000 7fa9a4563740 1 -- 172.21.15.90:0/936693067 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa994004380 con 0x55b120e9d7d0 2024-04-02T14:39:30.402 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.401+0000 7fa988c00640 1 -- 172.21.15.90:0/936693067 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fa994044040 con 0x55b120e9d7d0 2024-04-02T14:39:30.402 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.401+0000 7fa9a4563740 10 client.26680 fetch_fsmap learned FSMap version 27 2024-04-02T14:39:30.402 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.401+0000 7fa9a4563740 1 -- 172.21.15.90:0/936693067 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55b120b16210 con 0x55b120e9d7d0 2024-04-02T14:39:30.403 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.402+0000 7fa988c00640 1 -- 172.21.15.90:0/936693067 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa994004380 con 0x55b120e9d7d0 2024-04-02T14:39:30.404 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.402+0000 7fa988c00640 1 -- 172.21.15.90:0/936693067 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fa994040a20 con 0x55b120e9d7d0 2024-04-02T14:39:30.404 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.402+0000 7fa9a4563740 10 client.26680 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:39:30.404 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.403+0000 7fa9a4563740 10 client.26680 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-02T14:39:30.404 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.403+0000 7fa9a4563740 20 client.26680 populate_metadata read hostname 'smithi090' 2024-04-02T14:39:30.404 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.403+0000 7fa9a4563740 1 --2- 172.21.15.90:0/936693067 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55b120eb6620 0x55b120eb8a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:30.405 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.403+0000 7fa9a4563740 4 client.26680 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:39:30.405 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.403+0000 7fa9a4563740 1 -- 172.21.15.90:0/936693067 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b120db1cd0 con 0x55b120eb6620 2024-04-02T14:39:30.405 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.404+0000 7fa99a000640 1 --2- 172.21.15.90:0/936693067 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55b120eb6620 0x55b120eb8a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:30.407 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.405+0000 7fa99a000640 1 --2- 172.21.15.90:0/936693067 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55b120eb6620 0x55b120eb8a10 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cbdc8c2f497ff42f server_cookie=90bde5ce837f7934 in_seq=0 out_seq=0 2024-04-02T14:39:30.407 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.406+0000 7fa988c00640 10 client.26680 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:39:30.407 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.406+0000 7fa988c00640 1 -- 172.21.15.90:0/936693067 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b120db1cd0 con 0x55b120eb6620 2024-04-02T14:39:30.408 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.407+0000 7fa988c00640 10 client.26680 handle_command_reply: tid=0 2024-04-02T14:39:30.408 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.407+0000 7fa9a4563740 1 client.26680 shutdown 2024-04-02T14:39:30.408 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.407+0000 7fa9a4563740 1 -- 172.21.15.90:0/936693067 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b120e9d7d0 msgr2=0x55b120ea39e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:30.409 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.407+0000 7fa9a4563740 1 --2- 172.21.15.90:0/936693067 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b120e9d7d0 0x55b120ea39e0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fa994000c00 tx=0x7fa99402bc30 comp rx=0 tx=0).stop 2024-04-02T14:39:30.409 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.407+0000 7fa9a4563740 1 -- 172.21.15.90:0/936693067 shutdown_connections 2024-04-02T14:39:30.409 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.407+0000 7fa9a4563740 1 --2- 172.21.15.90:0/936693067 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55b120eb6620 0x55b120eb8a10 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:30.409 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.407+0000 7fa9a4563740 1 --2- 172.21.15.90:0/936693067 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b120ea63b0 0x55b120ea3f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:30.409 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.407+0000 7fa9a4563740 1 --2- 172.21.15.90:0/936693067 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b120e9d7d0 0x55b120ea39e0 secure :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fa994000c00 tx=0x7fa99402bc30 comp rx=0 tx=0).stop 2024-04-02T14:39:30.409 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.407+0000 7fa9a4563740 1 --2- 172.21.15.90:0/936693067 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b120e9ceb0 0x55b120ea34a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:30.409 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.407+0000 7fa9a4563740 1 -- 172.21.15.90:0/936693067 >> 172.21.15.90:0/936693067 conn(0x55b120e975e0 msgr2=0x55b120e99330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:39:30.409 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.407+0000 7fa988c00640 0 client.26680 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:39:30.410 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.408+0000 7fa9a4563740 1 -- 172.21.15.90:0/936693067 shutdown_connections 2024-04-02T14:39:30.410 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.409+0000 7fa9a4563740 1 -- 172.21.15.90:0/936693067 wait complete. 2024-04-02T14:39:30.410 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.409+0000 7fa9a4563740 20 client.26680 trim_cache size 0 max 16384 2024-04-02T14:39:30.436 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:39:30 smithi177 ceph-mon[36431]: pgmap v141: 97 pgs: 97 active+clean; 3.0 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 33 MiB/s wr, 872 op/s 2024-04-02T14:39:30.441 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:39:30 smithi090 ceph-mon[31674]: pgmap v141: 97 pgs: 97 active+clean; 3.0 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 33 MiB/s wr, 872 op/s 2024-04-02T14:39:30.458 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.457+0000 7fa9a4563740 1 Processor -- start 2024-04-02T14:39:30.459 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.458+0000 7fa9a4563740 1 -- start start 2024-04-02T14:39:30.461 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.459+0000 7fa9a4563740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b120e9ceb0 0x55b120ea5bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:30.461 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.459+0000 7fa9a4563740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b120ea63b0 0x55b120ea0720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:30.461 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.459+0000 7fa9a4563740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b120ea0c60 0x55b120ea10c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:30.461 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.459+0000 7fa9a4563740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fa99c1b3f40 con 0x55b120ea0c60 2024-04-02T14:39:30.461 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.459+0000 7fa9a4563740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fa99c1b5a80 con 0x55b120ea63b0 2024-04-02T14:39:30.461 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.459+0000 7fa9a4563740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fa99c1b4610 con 0x55b120e9ceb0 2024-04-02T14:39:30.462 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.460+0000 7fa99a000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b120e9ceb0 0x55b120ea5bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:30.462 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.460+0000 7fa99aa00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b120ea63b0 0x55b120ea0720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:30.463 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.462+0000 7fa99aa00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b120ea63b0 0x55b120ea0720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:57414/0 (socket says 172.21.15.90:57414) 2024-04-02T14:39:30.463 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.462+0000 7fa99aa00640 1 -- 172.21.15.90:0/1858689325 learned_addr learned my addr 172.21.15.90:0/1858689325 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:39:30.464 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.462+0000 7fa99a000640 1 --2- 172.21.15.90:0/1858689325 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b120ea0c60 0x55b120ea10c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:30.464 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.462+0000 7fa99aa00640 1 -- 172.21.15.90:0/1858689325 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b120e9ceb0 msgr2=0x55b120ea5bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:39:30.464 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.462+0000 7fa99aa00640 1 --2- 172.21.15.90:0/1858689325 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b120e9ceb0 0x55b120ea5bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:30.464 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.462+0000 7fa99aa00640 1 -- 172.21.15.90:0/1858689325 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b120ea0c60 msgr2=0x55b120ea10c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:39:30.464 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.462+0000 7fa99aa00640 1 --2- 172.21.15.90:0/1858689325 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b120ea0c60 0x55b120ea10c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:30.464 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.462+0000 7fa99aa00640 1 -- 172.21.15.90:0/1858689325 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b120ea8b10 con 0x55b120ea63b0 2024-04-02T14:39:30.464 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.463+0000 7fa99aa00640 1 --2- 172.21.15.90:0/1858689325 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b120ea63b0 0x55b120ea0720 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fa99402c170 tx=0x7fa99402c650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e2d870e6d45a50f3 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:39:30.467 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.466+0000 7fa98b400640 1 -- 172.21.15.90:0/1858689325 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa994062070 con 0x55b120ea63b0 2024-04-02T14:39:30.467 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.466+0000 7fa98b400640 1 -- 172.21.15.90:0/1858689325 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa99402cc80 con 0x55b120ea63b0 2024-04-02T14:39:30.467 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.466+0000 7fa98b400640 1 -- 172.21.15.90:0/1858689325 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa99404f040 con 0x55b120ea63b0 2024-04-02T14:39:30.470 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.469+0000 7fa9a4563740 1 -- 172.21.15.90:0/1858689325 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b120ea63b0 msgr2=0x55b120ea0720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:30.470 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.469+0000 7fa9a4563740 1 --2- 172.21.15.90:0/1858689325 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b120ea63b0 0x55b120ea0720 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fa99402c170 tx=0x7fa99402c650 comp rx=0 tx=0).stop 2024-04-02T14:39:30.470 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.469+0000 7fa9a4563740 1 -- 172.21.15.90:0/1858689325 shutdown_connections 2024-04-02T14:39:30.470 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.469+0000 7fa9a4563740 1 --2- 172.21.15.90:0/1858689325 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b120ea0c60 0x55b120ea10c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:30.471 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.470+0000 7fa9a4563740 1 --2- 172.21.15.90:0/1858689325 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b120ea63b0 0x55b120ea0720 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:30.471 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.470+0000 7fa9a4563740 1 --2- 172.21.15.90:0/1858689325 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b120e9ceb0 0x55b120ea5bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:30.471 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.470+0000 7fa9a4563740 1 -- 172.21.15.90:0/1858689325 >> 172.21.15.90:0/1858689325 conn(0x55b120e975e0 msgr2=0x55b120e979c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:39:30.472 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.470+0000 7fa9a4563740 1 -- 172.21.15.90:0/1858689325 shutdown_connections 2024-04-02T14:39:30.472 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.471+0000 7fa9a4563740 1 -- 172.21.15.90:0/1858689325 wait complete. 2024-04-02T14:39:30.473 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.472+0000 7fa9a4563740 1 -- start start 2024-04-02T14:39:30.474 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.473+0000 7fa9a4563740 1 Processor -- start 2024-04-02T14:39:30.476 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.475+0000 7fa9a4563740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b120e9ceb0 0x55b120eb98d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:30.476 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.475+0000 7fa9a4563740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b120ea0c60 0x55b120eb9e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:30.477 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.476+0000 7fa9a4563740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b120ea63b0 0x55b120f1b800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:30.477 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.476+0000 7fa9a4563740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fa99c1b53b0 con 0x55b120ea63b0 2024-04-02T14:39:30.477 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.476+0000 7fa9a4563740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fa99c1b4ce0 con 0x55b120e9ceb0 2024-04-02T14:39:30.477 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.476+0000 7fa9a4563740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fa99c1b1f20 con 0x55b120ea0c60 2024-04-02T14:39:30.478 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.476+0000 7fa9a4563740 10 client.0 _ll_register_callbacks cb 0x7fa99402cbd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:39:30.478 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.477+0000 7fa99a000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b120e9ceb0 0x55b120eb98d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:30.478 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.477+0000 7fa99a000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b120e9ceb0 0x55b120eb98d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:57420/0 (socket says 172.21.15.90:57420) 2024-04-02T14:39:30.478 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.477+0000 7fa99a000640 1 -- 172.21.15.90:0/4172590783 learned_addr learned my addr 172.21.15.90:0/4172590783 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:39:30.478 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.477+0000 7fa99aa00640 1 --2- 172.21.15.90:0/4172590783 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b120ea0c60 0x55b120eb9e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:30.479 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.478+0000 7fa99a000640 1 --2- 172.21.15.90:0/4172590783 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b120ea63b0 0x55b120f1b800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:30.480 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.479+0000 7fa99aa00640 1 -- 172.21.15.90:0/4172590783 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b120e9ceb0 msgr2=0x55b120eb98d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:39:30.480 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.479+0000 7fa99aa00640 1 --2- 172.21.15.90:0/4172590783 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b120e9ceb0 0x55b120eb98d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:30.480 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.479+0000 7fa99aa00640 1 -- 172.21.15.90:0/4172590783 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b120ea63b0 msgr2=0x55b120f1b800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:39:30.480 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.479+0000 7fa99aa00640 1 --2- 172.21.15.90:0/4172590783 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b120ea63b0 0x55b120f1b800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:30.481 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.479+0000 7fa99aa00640 1 -- 172.21.15.90:0/4172590783 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b120be2620 con 0x55b120ea0c60 2024-04-02T14:39:30.481 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.480+0000 7fa99aa00640 1 --2- 172.21.15.90:0/4172590783 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b120ea0c60 0x55b120eb9e10 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x55b120e889e0 tx=0x7fa99404ab00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f074aeed3741a04a server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:39:30.481 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.480+0000 7fa988c00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:39:30.481 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.480+0000 7fa988c00640 1 -- 172.21.15.90:0/4172590783 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa994062030 con 0x55b120ea0c60 2024-04-02T14:39:30.482 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.480+0000 7fa988c00640 1 -- 172.21.15.90:0/4172590783 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7fa994005570 con 0x55b120ea0c60 2024-04-02T14:39:30.482 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.481+0000 7fa9a4563740 1 -- 172.21.15.90:0/4172590783 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa99402cc80 con 0x55b120ea0c60 2024-04-02T14:39:30.483 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.481+0000 7fa988c00640 1 -- 172.21.15.90:0/4172590783 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fa99404f040 con 0x55b120ea0c60 2024-04-02T14:39:30.483 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.482+0000 7fa9a4563740 10 client.26686 fetch_fsmap learned FSMap version 27 2024-04-02T14:39:30.483 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.482+0000 7fa9a4563740 1 -- 172.21.15.90:0/4172590783 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55b120f1cd20 con 0x55b120ea0c60 2024-04-02T14:39:30.486 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.482+0000 7fa988c00640 1 -- 172.21.15.90:0/4172590783 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa99402cc80 con 0x55b120ea0c60 2024-04-02T14:39:30.486 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.484+0000 7fa988c00640 1 -- 172.21.15.90:0/4172590783 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fa994003d60 con 0x55b120ea0c60 2024-04-02T14:39:30.486 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.485+0000 7fa9a4563740 10 client.26686 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:39:30.486 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.485+0000 7fa9a4563740 10 client.26686 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-02T14:39:30.486 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.485+0000 7fa9a4563740 20 client.26686 populate_metadata read hostname 'smithi090' 2024-04-02T14:39:30.487 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.485+0000 7fa9a4563740 1 --2- 172.21.15.90:0/4172590783 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55b120f1d1a0 0x55b120f1f590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:30.487 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.485+0000 7fa9a4563740 4 client.26686 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-02T14:39:30.487 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.485+0000 7fa9a4563740 1 -- 172.21.15.90:0/4172590783 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55b120e0f2b0 con 0x55b120f1d1a0 2024-04-02T14:39:30.487 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.486+0000 7fa99a000640 1 --2- 172.21.15.90:0/4172590783 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55b120f1d1a0 0x55b120f1f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:30.488 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.487+0000 7fa99a000640 1 --2- 172.21.15.90:0/4172590783 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55b120f1d1a0 0x55b120f1f590 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=398cfa12c6b8c0d3 server_cookie=7ddd064ffc208c79 in_seq=0 out_seq=0 2024-04-02T14:39:30.489 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:30.487+0000 7fa988c00640 10 client.26686 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:39:31.066 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:31.064+0000 7fa988c00640 1 -- 172.21.15.90:0/4172590783 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55b120e0f2b0 con 0x55b120f1d1a0 2024-04-02T14:39:31.067 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:31.064+0000 7fa988c00640 10 client.26686 handle_command_reply: tid=0 2024-04-02T14:39:31.067 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:31.065+0000 7fa9a4563740 1 client.26686 shutdown 2024-04-02T14:39:31.067 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:31.065+0000 7fa9a4563740 1 -- 172.21.15.90:0/4172590783 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b120ea0c60 msgr2=0x55b120eb9e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:31.067 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:31.065+0000 7fa9a4563740 1 --2- 172.21.15.90:0/4172590783 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b120ea0c60 0x55b120eb9e10 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x55b120e889e0 tx=0x7fa99404ab00 comp rx=0 tx=0).stop 2024-04-02T14:39:31.068 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:31.067+0000 7fa9a4563740 1 -- 172.21.15.90:0/4172590783 shutdown_connections 2024-04-02T14:39:31.068 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:31.067+0000 7fa9a4563740 1 --2- 172.21.15.90:0/4172590783 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55b120f1d1a0 0x55b120f1f590 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:31.068 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:31.067+0000 7fa9a4563740 1 --2- 172.21.15.90:0/4172590783 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b120ea63b0 0x55b120f1b800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:31.068 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:31.067+0000 7fa9a4563740 1 --2- 172.21.15.90:0/4172590783 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b120ea0c60 0x55b120eb9e10 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:31.068 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:31.067+0000 7fa9a4563740 1 --2- 172.21.15.90:0/4172590783 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b120e9ceb0 0x55b120eb98d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:31.069 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:31.067+0000 7fa9a4563740 1 -- 172.21.15.90:0/4172590783 >> 172.21.15.90:0/4172590783 conn(0x55b120e975e0 msgr2=0x55b120eb1510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:39:31.069 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:31.068+0000 7fa988c00640 0 client.26686 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:39:31.069 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:31.068+0000 7fa9a4563740 1 -- 172.21.15.90:0/4172590783 shutdown_connections 2024-04-02T14:39:31.069 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:31.068+0000 7fa9a4563740 1 -- 172.21.15.90:0/4172590783 wait complete. 2024-04-02T14:39:31.069 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:31.068+0000 7fa9a4563740 20 client.26686 trim_cache size 0 max 16384 2024-04-02T14:39:31.074 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:{ 2024-04-02T14:39:31.074 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "message": "", 2024-04-02T14:39:31.074 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "return_code": 0 2024-04-02T14:39:31.074 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:} 2024-04-02T14:39:31.076 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:31.074+0000 7fa9a1800640 1 -- 172.21.15.90:0/68145546 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fa97c0a3ec0 msgr2=0x7fa97c0a6380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:31.076 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:31.074+0000 7fa9a1800640 1 --2- 172.21.15.90:0/68145546 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fa97c0a3ec0 0x7fa97c0a6380 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fa98c005ba0 tx=0x7fa98c00c7f0 comp rx=0 tx=0).stop 2024-04-02T14:39:31.076 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:31.074+0000 7fa9a1800640 1 -- 172.21.15.90:0/68145546 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa99c1128f0 msgr2=0x7fa99c19e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:31.076 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:31.074+0000 7fa9a1800640 1 --2- 172.21.15.90:0/68145546 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa99c1128f0 0x7fa99c19e7c0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fa98c00e9d0 tx=0x7fa98c00ee90 comp rx=0 tx=0).stop 2024-04-02T14:39:31.076 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:31.074+0000 7fa9a1800640 1 -- 172.21.15.90:0/68145546 shutdown_connections 2024-04-02T14:39:31.076 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:31.074+0000 7fa9a1800640 1 --2- 172.21.15.90:0/68145546 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fa97c0a3ec0 0x7fa97c0a6380 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:31.076 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:31.074+0000 7fa9a1800640 1 --2- 172.21.15.90:0/68145546 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa99c1128f0 0x7fa99c19e7c0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:31.076 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:31.074+0000 7fa9a1800640 1 --2- 172.21.15.90:0/68145546 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa99c1004f0 0x7fa99c1a4b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:31.076 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:31.074+0000 7fa9a1800640 1 --2- 172.21.15.90:0/68145546 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa99c0ffbd0 0x7fa99c1a4650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:31.076 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:31.074+0000 7fa9a1800640 1 -- 172.21.15.90:0/68145546 >> 172.21.15.90:0/68145546 conn(0x7fa99c076b90 msgr2=0x7fa99c111140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:39:31.077 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:31.075+0000 7fa9a1800640 1 -- 172.21.15.90:0/68145546 shutdown_connections 2024-04-02T14:39:31.077 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:31.075+0000 7fa9a1800640 1 -- 172.21.15.90:0/68145546 wait complete. 2024-04-02T14:39:31.832 INFO:teuthology.orchestra.run.smithi090.stderr:progress: 240.0 s, 2442.4 tps, lat 41.161 ms stddev 86.984 2024-04-02T14:39:32.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:39:32 smithi092 ceph-mon[36497]: pgmap v142: 97 pgs: 97 active+clean; 3.0 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 25 MiB/s wr, 765 op/s 2024-04-02T14:39:32.437 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:39:32 smithi177 ceph-mon[36431]: pgmap v142: 97 pgs: 97 active+clean; 3.0 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 25 MiB/s wr, 765 op/s 2024-04-02T14:39:32.438 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:39:32 smithi090 ceph-mon[31674]: pgmap v142: 97 pgs: 97 active+clean; 3.0 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 25 MiB/s wr, 765 op/s 2024-04-02T14:39:34.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:39:34 smithi092 ceph-mon[36497]: pgmap v143: 97 pgs: 97 active+clean; 3.0 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 23 MiB/s wr, 694 op/s 2024-04-02T14:39:34.436 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:39:34 smithi177 ceph-mon[36431]: pgmap v143: 97 pgs: 97 active+clean; 3.0 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 23 MiB/s wr, 694 op/s 2024-04-02T14:39:34.439 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:39:34 smithi090 ceph-mon[31674]: pgmap v143: 97 pgs: 97 active+clean; 3.0 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 23 MiB/s wr, 694 op/s 2024-04-02T14:39:36.362 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:39:36 smithi092 ceph-mon[36497]: pgmap v144: 97 pgs: 97 active+clean; 3.2 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 202 KiB/s rd, 49 MiB/s wr, 2.12k op/s 2024-04-02T14:39:36.436 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:39:36 smithi177 ceph-mon[36431]: pgmap v144: 97 pgs: 97 active+clean; 3.2 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 202 KiB/s rd, 49 MiB/s wr, 2.12k op/s 2024-04-02T14:39:36.438 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:39:36 smithi090 ceph-mon[31674]: pgmap v144: 97 pgs: 97 active+clean; 3.2 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 202 KiB/s rd, 49 MiB/s wr, 2.12k op/s 2024-04-02T14:39:36.831 INFO:teuthology.orchestra.run.smithi090.stderr:progress: 245.0 s, 3079.0 tps, lat 32.509 ms stddev 36.780 2024-04-02T14:39:37.430 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-02T14:39:37.430 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-02T14:39:37.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.720+0000 7feb0ec00640 1 -- 172.21.15.90:0/3782378435 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7feb081048c0 msgr2=0x7feb08108900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:37.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.720+0000 7feb0ec00640 1 --2- 172.21.15.90:0/3782378435 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7feb081048c0 0x7feb08108900 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7feaf4009860 tx=0x7feaf402bb70 comp rx=0 tx=0).stop 2024-04-02T14:39:37.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.720+0000 7feb0ec00640 1 -- 172.21.15.90:0/3782378435 shutdown_connections 2024-04-02T14:39:37.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.720+0000 7feb0ec00640 1 --2- 172.21.15.90:0/3782378435 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7feb08108fe0 0x7feb08110bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:37.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.720+0000 7feb0ec00640 1 --2- 172.21.15.90:0/3782378435 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7feb081048c0 0x7feb08108900 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:37.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.720+0000 7feb0ec00640 1 --2- 172.21.15.90:0/3782378435 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7feb08103f10 0x7feb081042f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:37.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.720+0000 7feb0ec00640 1 -- 172.21.15.90:0/3782378435 >> 172.21.15.90:0/3782378435 conn(0x7feb080ff7b0 msgr2=0x7feb08101bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:39:37.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.720+0000 7feb0ec00640 1 -- 172.21.15.90:0/3782378435 shutdown_connections 2024-04-02T14:39:37.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.720+0000 7feb0ec00640 1 -- 172.21.15.90:0/3782378435 wait complete. 2024-04-02T14:39:37.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.721+0000 7feb0ec00640 1 Processor -- start 2024-04-02T14:39:37.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.722+0000 7feb0ec00640 1 -- start start 2024-04-02T14:39:37.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.722+0000 7feb0ec00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7feb08103f10 0x7feb08076010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:37.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.722+0000 7feb0ec00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7feb08108fe0 0x7feb08076550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:37.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.723+0000 7feb0ec00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7feb08076a90 0x7feb081aa9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:37.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.723+0000 7feb0ec00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7feb08114090 con 0x7feb08076a90 2024-04-02T14:39:37.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.723+0000 7feb0ec00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7feb08113f10 con 0x7feb08103f10 2024-04-02T14:39:37.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.723+0000 7feb0ec00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7feb08114210 con 0x7feb08108fe0 2024-04-02T14:39:37.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.723+0000 7feb07e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7feb08103f10 0x7feb08076010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:37.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.723+0000 7feb07e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7feb08103f10 0x7feb08076010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:48156/0 (socket says 172.21.15.90:48156) 2024-04-02T14:39:37.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.723+0000 7feb07e00640 1 -- 172.21.15.90:0/4191964084 learned_addr learned my addr 172.21.15.90:0/4191964084 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:39:37.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.723+0000 7feb07400640 1 --2- 172.21.15.90:0/4191964084 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7feb08108fe0 0x7feb08076550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:37.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.724+0000 7feb0cc00640 1 --2- 172.21.15.90:0/4191964084 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7feb08076a90 0x7feb081aa9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:37.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.724+0000 7feb07e00640 1 -- 172.21.15.90:0/4191964084 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7feb08108fe0 msgr2=0x7feb08076550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:37.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.724+0000 7feb07e00640 1 --2- 172.21.15.90:0/4191964084 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7feb08108fe0 0x7feb08076550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:37.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.724+0000 7feb07e00640 1 -- 172.21.15.90:0/4191964084 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7feb08076a90 msgr2=0x7feb081aa9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:37.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.724+0000 7feb07e00640 1 --2- 172.21.15.90:0/4191964084 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7feb08076a90 0x7feb081aa9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:37.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.724+0000 7feb07e00640 1 -- 172.21.15.90:0/4191964084 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb081aaf20 con 0x7feb08103f10 2024-04-02T14:39:37.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.724+0000 7feb0cc00640 1 --2- 172.21.15.90:0/4191964084 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7feb08076a90 0x7feb081aa9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:39:37.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.725+0000 7feb07e00640 1 --2- 172.21.15.90:0/4191964084 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7feb08103f10 0x7feb08076010 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7feb0000efb0 tx=0x7feb0000c800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:39:37.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.725+0000 7feb04c00640 1 -- 172.21.15.90:0/4191964084 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb0001b070 con 0x7feb08103f10 2024-04-02T14:39:37.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.725+0000 7feb04c00640 1 -- 172.21.15.90:0/4191964084 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feb0000ba50 con 0x7feb08103f10 2024-04-02T14:39:37.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.725+0000 7feb04c00640 1 -- 172.21.15.90:0/4191964084 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb00016c00 con 0x7feb08103f10 2024-04-02T14:39:37.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.725+0000 7feb0ec00640 1 -- 172.21.15.90:0/4191964084 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feb081ab1b0 con 0x7feb08103f10 2024-04-02T14:39:37.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.726+0000 7feb0ec00640 1 -- 172.21.15.90:0/4191964084 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feb081ab680 con 0x7feb08103f10 2024-04-02T14:39:37.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.732+0000 7feb119bb740 1 Processor -- start 2024-04-02T14:39:37.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.732+0000 7feb04c00640 1 -- 172.21.15.90:0/4191964084 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7feb0001e070 con 0x7feb08103f10 2024-04-02T14:39:37.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.733+0000 7feb119bb740 1 -- start start 2024-04-02T14:39:37.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.733+0000 7feb119bb740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e0bd8697b0 0x55e0bd869b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:37.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.733+0000 7feb119bb740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e0bd86a0d0 0x55e0bd872770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:37.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.733+0000 7feb119bb740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e0bd872cb0 0x55e0bd8750a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:37.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.733+0000 7feb119bb740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55e0bd7cd770 con 0x55e0bd86a0d0 2024-04-02T14:39:37.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.733+0000 7feb119bb740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55e0bd775700 con 0x55e0bd8697b0 2024-04-02T14:39:37.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.733+0000 7feb119bb740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55e0bd7c9720 con 0x55e0bd872cb0 2024-04-02T14:39:37.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.734+0000 7feb07e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e0bd86a0d0 0x55e0bd872770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:37.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.734+0000 7feb0cc00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e0bd8697b0 0x55e0bd869b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:37.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.734+0000 7feb0cc00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e0bd8697b0 0x55e0bd869b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:48166/0 (socket says 172.21.15.90:48166) 2024-04-02T14:39:37.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.734+0000 7feb0cc00640 1 -- 172.21.15.90:0/1802938099 learned_addr learned my addr 172.21.15.90:0/1802938099 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:39:37.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.735+0000 7feb07400640 1 --2- 172.21.15.90:0/1802938099 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e0bd872cb0 0x55e0bd8750a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:37.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.735+0000 7feb07e00640 1 -- 172.21.15.90:0/1802938099 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e0bd872cb0 msgr2=0x55e0bd8750a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:39:37.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.735+0000 7feb07e00640 1 --2- 172.21.15.90:0/1802938099 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e0bd872cb0 0x55e0bd8750a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:37.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.735+0000 7feb07e00640 1 -- 172.21.15.90:0/1802938099 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e0bd8697b0 msgr2=0x55e0bd869b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:39:37.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.735+0000 7feb07e00640 1 --2- 172.21.15.90:0/1802938099 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e0bd8697b0 0x55e0bd869b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:37.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.735+0000 7feb07e00640 1 -- 172.21.15.90:0/1802938099 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e0bd5ab7d0 con 0x55e0bd86a0d0 2024-04-02T14:39:37.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.736+0000 7feb0cc00640 1 --2- 172.21.15.90:0/1802938099 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e0bd8697b0 0x55e0bd869b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:39:37.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.736+0000 7feb04c00640 1 --2- 172.21.15.90:0/4191964084 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7feaec0a3e70 0x7feaec0a6330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:37.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.736+0000 7feb07e00640 1 --2- 172.21.15.90:0/1802938099 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e0bd86a0d0 0x55e0bd872770 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7feb000cdda0 tx=0x7feb000d8690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d7fd11b8ef4366b server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:39:37.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.737+0000 7feb0cc00640 1 --2- 172.21.15.90:0/4191964084 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7feaec0a3e70 0x7feaec0a6330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:37.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.737+0000 7feafd200640 1 -- 172.21.15.90:0/1802938099 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb00098940 con 0x55e0bd86a0d0 2024-04-02T14:39:37.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.737+0000 7feafd200640 1 -- 172.21.15.90:0/1802938099 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feb0009c6a0 con 0x55e0bd86a0d0 2024-04-02T14:39:37.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.737+0000 7feafd200640 1 -- 172.21.15.90:0/1802938099 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb0009c000 con 0x55e0bd86a0d0 2024-04-02T14:39:37.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.737+0000 7feb04c00640 1 -- 172.21.15.90:0/4191964084 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7feb000d0640 con 0x7feb08103f10 2024-04-02T14:39:37.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.738+0000 7feb119bb740 1 -- 172.21.15.90:0/1802938099 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e0bd86a0d0 msgr2=0x55e0bd872770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:37.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.738+0000 7feb119bb740 1 --2- 172.21.15.90:0/1802938099 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e0bd86a0d0 0x55e0bd872770 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7feb000cdda0 tx=0x7feb000d8690 comp rx=0 tx=0).stop 2024-04-02T14:39:37.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.738+0000 7feb119bb740 1 -- 172.21.15.90:0/1802938099 shutdown_connections 2024-04-02T14:39:37.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.738+0000 7feb119bb740 1 --2- 172.21.15.90:0/1802938099 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e0bd872cb0 0x55e0bd8750a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:37.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.738+0000 7feb119bb740 1 --2- 172.21.15.90:0/1802938099 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e0bd86a0d0 0x55e0bd872770 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:37.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.738+0000 7feb119bb740 1 --2- 172.21.15.90:0/1802938099 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e0bd8697b0 0x55e0bd869b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:37.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.738+0000 7feb119bb740 1 -- 172.21.15.90:0/1802938099 >> 172.21.15.90:0/1802938099 conn(0x55e0bd863ee0 msgr2=0x55e0bd8642c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:39:37.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.738+0000 7feb0cc00640 1 --2- 172.21.15.90:0/4191964084 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7feaec0a3e70 0x7feaec0a6330 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7feaf80095f0 tx=0x7feaf8005f90 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:39:37.741 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.739+0000 7feb119bb740 1 -- 172.21.15.90:0/1802938099 shutdown_connections 2024-04-02T14:39:37.741 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.739+0000 7feb119bb740 1 -- 172.21.15.90:0/1802938099 wait complete. 2024-04-02T14:39:37.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.740+0000 7feb119bb740 1 -- start start 2024-04-02T14:39:37.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.740+0000 7feb119bb740 1 Processor -- start 2024-04-02T14:39:37.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.741+0000 7feb119bb740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e0bd8697b0 0x55e0bd86fd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:37.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.741+0000 7feb119bb740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e0bd86a0d0 0x55e0bd870290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:37.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.741+0000 7feb119bb740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e0bd872cb0 0x55e0bd8707d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:37.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.741+0000 7feb119bb740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55e0bd875aa0 con 0x55e0bd8697b0 2024-04-02T14:39:37.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.741+0000 7feb119bb740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55e0bd788ac0 con 0x55e0bd872cb0 2024-04-02T14:39:37.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.741+0000 7feb119bb740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55e0bd875c20 con 0x55e0bd86a0d0 2024-04-02T14:39:37.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.741+0000 7feb119bb740 10 client.0 _ll_register_callbacks cb 0x55e0bd78b180 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:39:37.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.741+0000 7feb07e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e0bd8697b0 0x55e0bd86fd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:37.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.741+0000 7feb07e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e0bd8697b0 0x55e0bd86fd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:48662/0 (socket says 172.21.15.90:48662) 2024-04-02T14:39:37.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.741+0000 7feb07e00640 1 -- 172.21.15.90:0/3610746945 learned_addr learned my addr 172.21.15.90:0/3610746945 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:39:37.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.741+0000 7feb07400640 1 --2- 172.21.15.90:0/3610746945 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e0bd86a0d0 0x55e0bd870290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:37.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.741+0000 7feb07e00640 1 -- 172.21.15.90:0/3610746945 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e0bd86a0d0 msgr2=0x55e0bd870290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:39:37.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.742+0000 7feb07e00640 1 --2- 172.21.15.90:0/3610746945 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e0bd86a0d0 0x55e0bd870290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:37.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.742+0000 7feb07e00640 1 -- 172.21.15.90:0/3610746945 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e0bd872cb0 msgr2=0x55e0bd8707d0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-02T14:39:37.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.742+0000 7feb0cc00640 1 --2- 172.21.15.90:0/3610746945 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e0bd872cb0 0x55e0bd8707d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:37.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.742+0000 7feb07e00640 1 --2- 172.21.15.90:0/3610746945 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e0bd872cb0 0x55e0bd8707d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:37.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.742+0000 7feb07e00640 1 -- 172.21.15.90:0/3610746945 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e0bd4e13f0 con 0x55e0bd8697b0 2024-04-02T14:39:37.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.742+0000 7feb0cc00640 1 --2- 172.21.15.90:0/3610746945 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e0bd872cb0 0x55e0bd8707d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:39:37.745 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.742+0000 7feaf1600640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-02T14:39:37.745 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.742+0000 7feb07e00640 1 --2- 172.21.15.90:0/3610746945 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e0bd8697b0 0x55e0bd86fd50 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7feb0000cd00 tx=0x7feb00009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f28f08bd4ed54a38 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:39:37.745 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.743+0000 7feaf1600640 1 -- 172.21.15.90:0/3610746945 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb0009c1e0 con 0x55e0bd8697b0 2024-04-02T14:39:37.745 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.743+0000 7feaf1600640 1 -- 172.21.15.90:0/3610746945 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7feb000ce020 con 0x55e0bd8697b0 2024-04-02T14:39:37.745 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.743+0000 7feb119bb740 1 -- 172.21.15.90:0/3610746945 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7feb0009c6a0 con 0x55e0bd8697b0 2024-04-02T14:39:37.745 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.744+0000 7feaf1600640 1 -- 172.21.15.90:0/3610746945 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7feb0009c470 con 0x55e0bd8697b0 2024-04-02T14:39:37.745 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.744+0000 7feb119bb740 10 client.16308 fetch_fsmap learned FSMap version 27 2024-04-02T14:39:37.745 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.744+0000 7feb119bb740 1 -- 172.21.15.90:0/3610746945 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55e0bd4a10f0 con 0x55e0bd8697b0 2024-04-02T14:39:37.746 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.744+0000 7feaf1600640 1 -- 172.21.15.90:0/3610746945 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feb0009c6a0 con 0x55e0bd8697b0 2024-04-02T14:39:37.746 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.744+0000 7feaf1600640 1 -- 172.21.15.90:0/3610746945 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7feb0000b040 con 0x55e0bd8697b0 2024-04-02T14:39:37.746 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.745+0000 7feb119bb740 10 client.16308 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:39:37.746 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.745+0000 7feb119bb740 10 client.16308 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:39:37.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.745+0000 7feb119bb740 20 client.16308 populate_metadata read hostname 'smithi090' 2024-04-02T14:39:37.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.745+0000 7feb119bb740 1 --2- 172.21.15.90:0/3610746945 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55e0bd882b90 0x55e0bd884f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:37.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.745+0000 7feb119bb740 4 client.16308 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:39:37.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.745+0000 7feb119bb740 1 -- 172.21.15.90:0/3610746945 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e0bd76e630 con 0x55e0bd882b90 2024-04-02T14:39:37.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.745+0000 7feb07400640 1 --2- 172.21.15.90:0/3610746945 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55e0bd882b90 0x55e0bd884f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:37.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.746+0000 7feb07400640 1 --2- 172.21.15.90:0/3610746945 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55e0bd882b90 0x55e0bd884f80 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=af1bee8769250a7a server_cookie=44adaaa038847f4b in_seq=0 out_seq=0 2024-04-02T14:39:37.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.746+0000 7feaf1600640 10 client.16308 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:39:37.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.747+0000 7feaf1600640 1 -- 172.21.15.90:0/3610746945 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e0bd76e630 con 0x55e0bd882b90 2024-04-02T14:39:37.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.747+0000 7feaf1600640 10 client.16308 handle_command_reply: tid=0 2024-04-02T14:39:37.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.748+0000 7feb119bb740 1 client.16308 shutdown 2024-04-02T14:39:37.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.748+0000 7feb119bb740 1 -- 172.21.15.90:0/3610746945 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e0bd8697b0 msgr2=0x55e0bd86fd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:37.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.748+0000 7feb119bb740 1 --2- 172.21.15.90:0/3610746945 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e0bd8697b0 0x55e0bd86fd50 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7feb0000cd00 tx=0x7feb00009780 comp rx=0 tx=0).stop 2024-04-02T14:39:37.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.748+0000 7feb119bb740 1 -- 172.21.15.90:0/3610746945 shutdown_connections 2024-04-02T14:39:37.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.748+0000 7feb119bb740 1 --2- 172.21.15.90:0/3610746945 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55e0bd882b90 0x55e0bd884f80 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:37.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.749+0000 7feb119bb740 1 --2- 172.21.15.90:0/3610746945 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e0bd872cb0 0x55e0bd8707d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:37.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.749+0000 7feaf1600640 0 client.16308 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:39:37.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.749+0000 7feb119bb740 1 --2- 172.21.15.90:0/3610746945 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e0bd86a0d0 0x55e0bd870290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:37.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.749+0000 7feb119bb740 1 --2- 172.21.15.90:0/3610746945 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e0bd8697b0 0x55e0bd86fd50 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:37.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.749+0000 7feb119bb740 1 -- 172.21.15.90:0/3610746945 >> 172.21.15.90:0/3610746945 conn(0x55e0bd863ee0 msgr2=0x55e0bd865c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:39:37.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.749+0000 7feb119bb740 1 -- 172.21.15.90:0/3610746945 shutdown_connections 2024-04-02T14:39:37.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.749+0000 7feb119bb740 1 -- 172.21.15.90:0/3610746945 wait complete. 2024-04-02T14:39:37.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.750+0000 7feb119bb740 20 client.16308 trim_cache size 0 max 16384 2024-04-02T14:39:37.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.793+0000 7feb119bb740 1 Processor -- start 2024-04-02T14:39:37.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.793+0000 7feb119bb740 1 -- start start 2024-04-02T14:39:37.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.793+0000 7feb119bb740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e0bd8697b0 0x55e0bd872480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:37.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.793+0000 7feb119bb740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e0bd86a0d0 0x55e0bd8642c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:37.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.793+0000 7feb119bb740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e0bd872cb0 0x55e0bd86eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:37.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.793+0000 7feb119bb740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7feb081b3d90 con 0x55e0bd86a0d0 2024-04-02T14:39:37.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.793+0000 7feb119bb740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7feb081b58d0 con 0x55e0bd8697b0 2024-04-02T14:39:37.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.793+0000 7feb119bb740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7feb081b4460 con 0x55e0bd872cb0 2024-04-02T14:39:37.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.794+0000 7feb0cc00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e0bd872cb0 0x55e0bd86eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:37.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.794+0000 7feb0cc00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e0bd872cb0 0x55e0bd86eeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:44398/0 (socket says 172.21.15.90:44398) 2024-04-02T14:39:37.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.794+0000 7feb0cc00640 1 -- 172.21.15.90:0/1544117265 learned_addr learned my addr 172.21.15.90:0/1544117265 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:39:37.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.795+0000 7feb07e00640 1 --2- 172.21.15.90:0/1544117265 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e0bd8697b0 0x55e0bd872480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:37.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.795+0000 7feb07400640 1 --2- 172.21.15.90:0/1544117265 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e0bd86a0d0 0x55e0bd8642c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:37.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.796+0000 7feb0cc00640 1 -- 172.21.15.90:0/1544117265 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e0bd8697b0 msgr2=0x55e0bd872480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:39:37.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.796+0000 7feb0cc00640 1 --2- 172.21.15.90:0/1544117265 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e0bd8697b0 0x55e0bd872480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:37.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.796+0000 7feb0cc00640 1 -- 172.21.15.90:0/1544117265 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e0bd86a0d0 msgr2=0x55e0bd8642c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:39:37.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.796+0000 7feb0cc00640 1 --2- 172.21.15.90:0/1544117265 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e0bd86a0d0 0x55e0bd8642c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:37.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.796+0000 7feb0cc00640 1 -- 172.21.15.90:0/1544117265 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e0bd85e3b0 con 0x55e0bd872cb0 2024-04-02T14:39:37.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.797+0000 7feb0cc00640 1 --2- 172.21.15.90:0/1544117265 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e0bd872cb0 0x55e0bd86eeb0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7feaf8009750 tx=0x7feaf8016000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f774b51dc06e84a3 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:39:37.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.797+0000 7feaf3e00640 1 -- 172.21.15.90:0/1544117265 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feaf8005020 con 0x55e0bd872cb0 2024-04-02T14:39:37.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.798+0000 7feaf3e00640 1 -- 172.21.15.90:0/1544117265 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feaf80047a0 con 0x55e0bd872cb0 2024-04-02T14:39:37.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.798+0000 7feaf3e00640 1 -- 172.21.15.90:0/1544117265 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feaf8026650 con 0x55e0bd872cb0 2024-04-02T14:39:37.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.799+0000 7feb119bb740 1 -- 172.21.15.90:0/1544117265 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e0bd872cb0 msgr2=0x55e0bd86eeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:37.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.799+0000 7feb119bb740 1 --2- 172.21.15.90:0/1544117265 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e0bd872cb0 0x55e0bd86eeb0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7feaf8009750 tx=0x7feaf8016000 comp rx=0 tx=0).stop 2024-04-02T14:39:37.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.799+0000 7feb119bb740 1 -- 172.21.15.90:0/1544117265 shutdown_connections 2024-04-02T14:39:37.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.799+0000 7feb119bb740 1 --2- 172.21.15.90:0/1544117265 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e0bd872cb0 0x55e0bd86eeb0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:37.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.799+0000 7feb119bb740 1 --2- 172.21.15.90:0/1544117265 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e0bd86a0d0 0x55e0bd8642c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:37.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.799+0000 7feb119bb740 1 --2- 172.21.15.90:0/1544117265 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e0bd8697b0 0x55e0bd872480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:37.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.799+0000 7feb119bb740 1 -- 172.21.15.90:0/1544117265 >> 172.21.15.90:0/1544117265 conn(0x55e0bd863ee0 msgr2=0x55e0bd865310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:39:37.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.799+0000 7feb119bb740 1 -- 172.21.15.90:0/1544117265 shutdown_connections 2024-04-02T14:39:37.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.800+0000 7feb119bb740 1 -- 172.21.15.90:0/1544117265 wait complete. 2024-04-02T14:39:37.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.800+0000 7feb119bb740 1 -- start start 2024-04-02T14:39:37.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.800+0000 7feb119bb740 1 Processor -- start 2024-04-02T14:39:37.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.804+0000 7feb119bb740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e0bd8697b0 0x55e0bd8883c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:37.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.804+0000 7feb119bb740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e0bd86a0d0 0x55e0bd885750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:37.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.804+0000 7feb119bb740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e0bd872cb0 0x55e0bd885c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:37.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.804+0000 7feb119bb740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7feb081b5200 con 0x55e0bd8697b0 2024-04-02T14:39:37.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.804+0000 7feb119bb740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7feb081b4b30 con 0x55e0bd86a0d0 2024-04-02T14:39:37.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.804+0000 7feb119bb740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7feb081b1d70 con 0x55e0bd872cb0 2024-04-02T14:39:37.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.804+0000 7feb119bb740 10 client.0 _ll_register_callbacks cb 0x7feaf8009490 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:39:37.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.805+0000 7feb07400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e0bd86a0d0 0x55e0bd885750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:37.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.805+0000 7feb07400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e0bd86a0d0 0x55e0bd885750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:48194/0 (socket says 172.21.15.90:48194) 2024-04-02T14:39:37.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.805+0000 7feb07400640 1 -- 172.21.15.90:0/3833502607 learned_addr learned my addr 172.21.15.90:0/3833502607 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:39:37.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.805+0000 7feb07e00640 1 --2- 172.21.15.90:0/3833502607 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e0bd8697b0 0x55e0bd8883c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:37.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.805+0000 7feb0cc00640 1 --2- 172.21.15.90:0/3833502607 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e0bd872cb0 0x55e0bd885c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:37.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.806+0000 7feb0cc00640 1 -- 172.21.15.90:0/3833502607 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e0bd86a0d0 msgr2=0x55e0bd885750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:39:37.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.806+0000 7feb0cc00640 1 --2- 172.21.15.90:0/3833502607 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e0bd86a0d0 0x55e0bd885750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:37.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.806+0000 7feb0cc00640 1 -- 172.21.15.90:0/3833502607 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e0bd8697b0 msgr2=0x55e0bd8883c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:39:37.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.806+0000 7feb0cc00640 1 --2- 172.21.15.90:0/3833502607 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e0bd8697b0 0x55e0bd8883c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:37.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.806+0000 7feb0cc00640 1 -- 172.21.15.90:0/3833502607 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e0bd8864c0 con 0x55e0bd872cb0 2024-04-02T14:39:37.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.807+0000 7feb07e00640 1 --2- 172.21.15.90:0/3833502607 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e0bd8697b0 0x55e0bd8883c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:39:37.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.807+0000 7feb0cc00640 1 --2- 172.21.15.90:0/3833502607 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e0bd872cb0 0x55e0bd885c90 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7feaf8009720 tx=0x7feaf8004c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fde8ac83f58d40bd server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:39:37.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.807+0000 7feaf1600640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:39:37.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.808+0000 7feaf1600640 1 -- 172.21.15.90:0/3833502607 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feaf8035020 con 0x55e0bd872cb0 2024-04-02T14:39:37.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.808+0000 7feaf1600640 1 -- 172.21.15.90:0/3833502607 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7feaf8023040 con 0x55e0bd872cb0 2024-04-02T14:39:37.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.808+0000 7feb119bb740 1 -- 172.21.15.90:0/3833502607 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7feaf80047a0 con 0x55e0bd872cb0 2024-04-02T14:39:37.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.809+0000 7feaf1600640 1 -- 172.21.15.90:0/3833502607 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7feaf8005210 con 0x55e0bd872cb0 2024-04-02T14:39:37.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.809+0000 7feb119bb740 10 client.26704 fetch_fsmap learned FSMap version 27 2024-04-02T14:39:37.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.809+0000 7feb119bb740 1 -- 172.21.15.90:0/3833502607 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55e0bd8e8de0 con 0x55e0bd872cb0 2024-04-02T14:39:37.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.810+0000 7feaf1600640 1 -- 172.21.15.90:0/3833502607 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feaf80047a0 con 0x55e0bd872cb0 2024-04-02T14:39:37.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.810+0000 7feaf1600640 1 -- 172.21.15.90:0/3833502607 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7feaf801a040 con 0x55e0bd872cb0 2024-04-02T14:39:37.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.810+0000 7feb119bb740 10 client.26704 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:39:37.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.810+0000 7feb119bb740 10 client.26704 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:39:37.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.810+0000 7feb119bb740 20 client.26704 populate_metadata read hostname 'smithi090' 2024-04-02T14:39:37.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.810+0000 7feb119bb740 1 --2- 172.21.15.90:0/3833502607 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55e0bd8e94a0 0x55e0bd8eb8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:37.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.810+0000 7feb119bb740 4 client.26704 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-02T14:39:37.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.810+0000 7feb119bb740 1 -- 172.21.15.90:0/3833502607 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55e0bd7dacb0 con 0x55e0bd8e94a0 2024-04-02T14:39:37.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.811+0000 7feb07e00640 1 --2- 172.21.15.90:0/3833502607 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55e0bd8e94a0 0x55e0bd8eb8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:37.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.811+0000 7feb07e00640 1 --2- 172.21.15.90:0/3833502607 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55e0bd8e94a0 0x55e0bd8eb8c0 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cae32d8765fae045 server_cookie=138d4852f2d9b6b3 in_seq=0 out_seq=0 2024-04-02T14:39:37.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.811+0000 7feaf1600640 10 client.26704 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:39:37.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.811+0000 7feaf1600640 1 -- 172.21.15.90:0/3833502607 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55e0bd7dacb0 con 0x55e0bd8e94a0 2024-04-02T14:39:37.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.811+0000 7feaf1600640 10 client.26704 handle_command_reply: tid=0 2024-04-02T14:39:37.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.811+0000 7feb119bb740 1 client.26704 shutdown 2024-04-02T14:39:37.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.812+0000 7feb119bb740 1 -- 172.21.15.90:0/3833502607 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e0bd872cb0 msgr2=0x55e0bd885c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:37.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.812+0000 7feb119bb740 1 --2- 172.21.15.90:0/3833502607 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e0bd872cb0 0x55e0bd885c90 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7feaf8009720 tx=0x7feaf8004c00 comp rx=0 tx=0).stop 2024-04-02T14:39:37.815 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.813+0000 7feb119bb740 1 -- 172.21.15.90:0/3833502607 shutdown_connections 2024-04-02T14:39:37.815 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.813+0000 7feb119bb740 1 --2- 172.21.15.90:0/3833502607 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55e0bd8e94a0 0x55e0bd8eb8c0 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:37.815 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.813+0000 7feaf1600640 0 client.26704 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:39:37.815 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.813+0000 7feb119bb740 1 --2- 172.21.15.90:0/3833502607 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e0bd872cb0 0x55e0bd885c90 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:37.815 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.813+0000 7feb119bb740 1 --2- 172.21.15.90:0/3833502607 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e0bd86a0d0 0x55e0bd885750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:37.815 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.813+0000 7feb119bb740 1 --2- 172.21.15.90:0/3833502607 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e0bd8697b0 0x55e0bd8883c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:37.815 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.813+0000 7feb119bb740 1 -- 172.21.15.90:0/3833502607 >> 172.21.15.90:0/3833502607 conn(0x55e0bd863ee0 msgr2=0x55e0bd87de70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:39:37.815 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.813+0000 7feb119bb740 1 -- 172.21.15.90:0/3833502607 shutdown_connections 2024-04-02T14:39:37.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.813+0000 7feb119bb740 1 -- 172.21.15.90:0/3833502607 wait complete. 2024-04-02T14:39:37.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.814+0000 7feb119bb740 20 client.26704 trim_cache size 0 max 16384 2024-04-02T14:39:37.816 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-02T14:39:37.816 INFO:teuthology.orchestra.run.smithi090.stdout: "status": "no active scrubs running", 2024-04-02T14:39:37.816 INFO:teuthology.orchestra.run.smithi090.stdout: "scrubs": {} 2024-04-02T14:39:37.816 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-02T14:39:37.817 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.815+0000 7feb0ec00640 1 -- 172.21.15.90:0/4191964084 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7feaec0a3e70 msgr2=0x7feaec0a6330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:37.817 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.815+0000 7feb0ec00640 1 --2- 172.21.15.90:0/4191964084 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7feaec0a3e70 0x7feaec0a6330 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7feaf80095f0 tx=0x7feaf8005f90 comp rx=0 tx=0).stop 2024-04-02T14:39:37.817 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.815+0000 7feb0ec00640 1 -- 172.21.15.90:0/4191964084 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7feb08103f10 msgr2=0x7feb08076010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:37.817 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.815+0000 7feb0ec00640 1 --2- 172.21.15.90:0/4191964084 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7feb08103f10 0x7feb08076010 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7feb0000efb0 tx=0x7feb0000c800 comp rx=0 tx=0).stop 2024-04-02T14:39:37.817 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.815+0000 7feb0ec00640 1 -- 172.21.15.90:0/4191964084 shutdown_connections 2024-04-02T14:39:37.817 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.815+0000 7feb0ec00640 1 --2- 172.21.15.90:0/4191964084 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7feaec0a3e70 0x7feaec0a6330 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:37.817 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.815+0000 7feb0ec00640 1 --2- 172.21.15.90:0/4191964084 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7feb08076a90 0x7feb081aa9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:37.817 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.815+0000 7feb0ec00640 1 --2- 172.21.15.90:0/4191964084 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7feb08108fe0 0x7feb08076550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:37.818 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.815+0000 7feb0ec00640 1 --2- 172.21.15.90:0/4191964084 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7feb08103f10 0x7feb08076010 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:37.818 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.815+0000 7feb0ec00640 1 -- 172.21.15.90:0/4191964084 >> 172.21.15.90:0/4191964084 conn(0x7feb080ff7b0 msgr2=0x7feb08100130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:39:37.818 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.816+0000 7feb0ec00640 1 -- 172.21.15.90:0/4191964084 shutdown_connections 2024-04-02T14:39:37.818 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:37.816+0000 7feb0ec00640 1 -- 172.21.15.90:0/4191964084 wait complete. 2024-04-02T14:39:37.840 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-02T14:39:37.840 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T14:39:38.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.123+0000 7f343de00640 1 -- 172.21.15.90:0/2228144145 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3438108ff0 msgr2=0x7f3438110bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:38.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.123+0000 7f343de00640 1 --2- 172.21.15.90:0/2228144145 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3438108ff0 0x7f3438110bc0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f3430009ca0 tx=0x7f343002c120 comp rx=0 tx=0).stop 2024-04-02T14:39:38.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.124+0000 7f343de00640 1 -- 172.21.15.90:0/2228144145 shutdown_connections 2024-04-02T14:39:38.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.124+0000 7f343de00640 1 --2- 172.21.15.90:0/2228144145 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3438108ff0 0x7f3438110bc0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:38.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.124+0000 7f343de00640 1 --2- 172.21.15.90:0/2228144145 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f34381048d0 0x7f3438108910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:38.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.124+0000 7f343de00640 1 --2- 172.21.15.90:0/2228144145 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3438103f20 0x7f3438104300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:38.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.124+0000 7f343de00640 1 -- 172.21.15.90:0/2228144145 >> 172.21.15.90:0/2228144145 conn(0x7f34380ff780 msgr2=0x7f3438101ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:39:38.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.124+0000 7f343de00640 1 -- 172.21.15.90:0/2228144145 shutdown_connections 2024-04-02T14:39:38.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.124+0000 7f343de00640 1 -- 172.21.15.90:0/2228144145 wait complete. 2024-04-02T14:39:38.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.125+0000 7f343de00640 1 Processor -- start 2024-04-02T14:39:38.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.125+0000 7f343de00640 1 -- start start 2024-04-02T14:39:38.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.126+0000 7f343de00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3438103f20 0x7f343819a620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:38.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.126+0000 7f343de00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3438108ff0 0x7f343819ab60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:38.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.126+0000 7f343de00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f343819eee0 0x7f343819f390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:38.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.126+0000 7f343de00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f3438114100 con 0x7f3438108ff0 2024-04-02T14:39:38.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.126+0000 7f343de00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f3438113f80 con 0x7f3438103f20 2024-04-02T14:39:38.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.126+0000 7f343de00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f3438114280 con 0x7f343819eee0 2024-04-02T14:39:38.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.126+0000 7f3436a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3438103f20 0x7f343819a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:38.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.126+0000 7f3436a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3438103f20 0x7f343819a620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:48212/0 (socket says 172.21.15.90:48212) 2024-04-02T14:39:38.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.126+0000 7f3436a00640 1 -- 172.21.15.90:0/3202212461 learned_addr learned my addr 172.21.15.90:0/3202212461 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:39:38.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.127+0000 7f3437400640 1 --2- 172.21.15.90:0/3202212461 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f343819eee0 0x7f343819f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:38.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.127+0000 7f3437400640 1 -- 172.21.15.90:0/3202212461 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3438103f20 msgr2=0x7f343819a620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:38.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.127+0000 7f3437400640 1 --2- 172.21.15.90:0/3202212461 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3438103f20 0x7f343819a620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:38.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.127+0000 7f3437400640 1 -- 172.21.15.90:0/3202212461 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3438108ff0 msgr2=0x7f343819ab60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T14:39:38.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.127+0000 7f3436000640 1 --2- 172.21.15.90:0/3202212461 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3438108ff0 0x7f343819ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:38.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.127+0000 7f3437400640 1 --2- 172.21.15.90:0/3202212461 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3438108ff0 0x7f343819ab60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:38.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.127+0000 7f3437400640 1 -- 172.21.15.90:0/3202212461 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f343819f9d0 con 0x7f343819eee0 2024-04-02T14:39:38.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.128+0000 7f3436a00640 1 --2- 172.21.15.90:0/3202212461 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3438103f20 0x7f343819a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:39:38.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.128+0000 7f3437400640 1 --2- 172.21.15.90:0/3202212461 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f343819eee0 0x7f343819f390 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f3430009bf0 tx=0x7f3430035d10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:39:38.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.128+0000 7f343ca00640 1 -- 172.21.15.90:0/3202212461 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3430039040 con 0x7f343819eee0 2024-04-02T14:39:38.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.128+0000 7f343ca00640 1 -- 172.21.15.90:0/3202212461 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3430003970 con 0x7f343819eee0 2024-04-02T14:39:38.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.128+0000 7f343ca00640 1 -- 172.21.15.90:0/3202212461 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34300045e0 con 0x7f343819eee0 2024-04-02T14:39:38.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.129+0000 7f343de00640 1 -- 172.21.15.90:0/3202212461 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f343819fc60 con 0x7f343819eee0 2024-04-02T14:39:38.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.129+0000 7f343de00640 1 -- 172.21.15.90:0/3202212461 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34381a7370 con 0x7f343819eee0 2024-04-02T14:39:38.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.131+0000 7f343de00640 1 -- 172.21.15.90:0/3202212461 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3438068d40 con 0x7f343819eee0 2024-04-02T14:39:38.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.134+0000 7f343ca00640 1 -- 172.21.15.90:0/3202212461 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f3430046030 con 0x7f343819eee0 2024-04-02T14:39:38.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.135+0000 7f343ca00640 1 --2- 172.21.15.90:0/3202212461 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f341c0a3ef0 0x7f341c0a63b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:38.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.135+0000 7f343ca00640 1 -- 172.21.15.90:0/3202212461 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f34300f2270 con 0x7f343819eee0 2024-04-02T14:39:38.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.136+0000 7f3436a00640 1 --2- 172.21.15.90:0/3202212461 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f341c0a3ef0 0x7f341c0a63b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:38.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.137+0000 7f3436a00640 1 --2- 172.21.15.90:0/3202212461 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f341c0a3ef0 0x7f341c0a63b0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f3428009e10 tx=0x7f34280091c0 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:39:38.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.139+0000 7f343ca00640 1 -- 172.21.15.90:0/3202212461 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7f34300bea60 con 0x7f343819eee0 2024-04-02T14:39:38.189 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:39:38 smithi090 ceph-mon[31674]: pgmap v145: 97 pgs: 97 active+clean; 3.2 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 37 MiB/s wr, 1.81k op/s 2024-04-02T14:39:38.360 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:39:38 smithi092 ceph-mon[36497]: pgmap v145: 97 pgs: 97 active+clean; 3.2 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 37 MiB/s wr, 1.81k op/s 2024-04-02T14:39:38.436 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:39:38 smithi177 ceph-mon[36431]: pgmap v145: 97 pgs: 97 active+clean; 3.2 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 37 MiB/s wr, 1.81k op/s 2024-04-02T14:39:38.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.550+0000 7f343de00640 1 -- 172.21.15.90:0/3202212461 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f34381048d0 con 0x7f343819eee0 2024-04-02T14:39:38.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.551+0000 7f343ca00640 1 -- 172.21.15.90:0/3202212461 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11927 (secure 0 0 0) 0x7f34300be360 con 0x7f343819eee0 2024-04-02T14:39:38.553 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-02T14:39:38.553 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14412,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6837/2852160309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6836","nonce":2852160309},{"type":"v1","addr":"172.21.15.90:6837","nonce":2852160309}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14478,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6841/1919376858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6840","nonce":1919376858},{"type":"v1","addr":"172.21.15.90:6841","nonce":1919376858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-02T14:32:22.319083+0000","modified":"2024-04-02T14:34:52.266152+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24590,"mds_1":24560,"mds_2":24547,"mds_3":24539,"mds_4":24523},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24454":{"gid":24454,"name":"l","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6833/1564319431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":1564319431},{"type":"v1","addr":"172.21.15.177:6833","nonce":1564319431}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.92:6833/1570337372","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":1570337372},{"type":"v1","addr":"172.21.15.92:6833","nonce":1570337372}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"j","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.90:6835/1277124891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6834","nonce":1277124891},{"type":"v1","addr":"172.21.15.90:6835","nonce":1277124891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.92:6835/860721164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":860721164},{"type":"v1","addr":"172.21.15.92:6835","nonce":860721164}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24511":{"gid":24511,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6835/3177244988","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":3177244988},{"type":"v1","addr":"172.21.15.177:6835","nonce":3177244988}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"c","rank":4,"incarnation":25,"state":"up:active","state_seq":9,"addr":"172.21.15.177:6837/4223414858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":4223414858},{"type":"v1","addr":"172.21.15.177:6837","nonce":4223414858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24539":{"gid":24539,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":8,"addr":"172.21.15.92:6837/1270635939","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":1270635939},{"type":"v1","addr":"172.21.15.92:6837","nonce":1270635939}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24547":{"gid":24547,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":7,"addr":"172.21.15.177:6839/664895663","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":664895663},{"type":"v1","addr":"172.21.15.177:6839","nonce":664895663}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"g","rank":1,"incarnation":19,"state":"up:active","state_seq":5,"addr":"172.21.15.90:6839/1498915870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6838","nonce":1498915870},{"type":"v1","addr":"172.21.15.90:6839","nonce":1498915870}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24590":{"gid":24590,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.92:6839/1054888219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":1054888219},{"type":"v1","addr":"172.21.15.92:6839","nonce":1054888219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24590,"qdb_cluster":[24547,24590,24523,24539,24560]},"id":1}]} 2024-04-02T14:39:38.553 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 27 2024-04-02T14:39:38.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.555+0000 7f343de00640 1 -- 172.21.15.90:0/3202212461 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f341c0a3ef0 msgr2=0x7f341c0a63b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:38.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.555+0000 7f343de00640 1 --2- 172.21.15.90:0/3202212461 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f341c0a3ef0 0x7f341c0a63b0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f3428009e10 tx=0x7f34280091c0 comp rx=0 tx=0).stop 2024-04-02T14:39:38.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.555+0000 7f343de00640 1 -- 172.21.15.90:0/3202212461 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f343819eee0 msgr2=0x7f343819f390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:38.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.555+0000 7f343de00640 1 --2- 172.21.15.90:0/3202212461 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f343819eee0 0x7f343819f390 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f3430009bf0 tx=0x7f3430035d10 comp rx=0 tx=0).stop 2024-04-02T14:39:38.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.557+0000 7f343de00640 1 -- 172.21.15.90:0/3202212461 shutdown_connections 2024-04-02T14:39:38.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.557+0000 7f343de00640 1 --2- 172.21.15.90:0/3202212461 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f341c0a3ef0 0x7f341c0a63b0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:38.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.557+0000 7f343de00640 1 --2- 172.21.15.90:0/3202212461 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f343819eee0 0x7f343819f390 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:38.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.557+0000 7f343de00640 1 --2- 172.21.15.90:0/3202212461 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3438108ff0 0x7f343819ab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:38.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.557+0000 7f343de00640 1 --2- 172.21.15.90:0/3202212461 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3438103f20 0x7f343819a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:38.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.557+0000 7f343de00640 1 -- 172.21.15.90:0/3202212461 >> 172.21.15.90:0/3202212461 conn(0x7f34380ff780 msgr2=0x7f3438101a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:39:38.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.557+0000 7f343de00640 1 -- 172.21.15.90:0/3202212461 shutdown_connections 2024-04-02T14:39:38.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.557+0000 7f343de00640 1 -- 172.21.15.90:0/3202212461 wait complete. 2024-04-02T14:39:38.584 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-04-02T14:39:38.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.873+0000 7f2a1f200640 1 -- 172.21.15.90:0/950558120 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2a1409be10 msgr2=0x7f2a1409c1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:38.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.873+0000 7f2a1f200640 1 --2- 172.21.15.90:0/950558120 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2a1409be10 0x7f2a1409c1f0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f2a0c009930 tx=0x7f2a0c02bc20 comp rx=0 tx=0).stop 2024-04-02T14:39:38.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.874+0000 7f2a1f200640 1 -- 172.21.15.90:0/950558120 shutdown_connections 2024-04-02T14:39:38.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.874+0000 7f2a1f200640 1 --2- 172.21.15.90:0/950558120 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2a14005670 0x7f2a140a2b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:38.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.874+0000 7f2a1f200640 1 --2- 172.21.15.90:0/950558120 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2a14004cb0 0x7f2a14005130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:38.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.874+0000 7f2a1f200640 1 --2- 172.21.15.90:0/950558120 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2a1409be10 0x7f2a1409c1f0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:38.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.874+0000 7f2a1f200640 1 -- 172.21.15.90:0/950558120 >> 172.21.15.90:0/950558120 conn(0x7f2a1400a740 msgr2=0x7f2a1400ab50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:39:38.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.874+0000 7f2a1f200640 1 -- 172.21.15.90:0/950558120 shutdown_connections 2024-04-02T14:39:38.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.874+0000 7f2a1f200640 1 -- 172.21.15.90:0/950558120 wait complete. 2024-04-02T14:39:38.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.876+0000 7f2a1f200640 1 Processor -- start 2024-04-02T14:39:38.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.876+0000 7f2a1f200640 1 -- start start 2024-04-02T14:39:38.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.877+0000 7f2a1f200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2a14004cb0 0x7f2a1412c620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:38.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.877+0000 7f2a1f200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2a14005670 0x7f2a1412cb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:38.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.877+0000 7f2a1f200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2a1409be10 0x7f2a14130ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:38.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.877+0000 7f2a1f200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f2a140a6100 con 0x7f2a1409be10 2024-04-02T14:39:38.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.877+0000 7f2a1f200640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f2a140a5f80 con 0x7f2a14004cb0 2024-04-02T14:39:38.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.877+0000 7f2a1f200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f2a140a6280 con 0x7f2a14005670 2024-04-02T14:39:38.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.877+0000 7f2a1d400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2a14005670 0x7f2a1412cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:38.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.877+0000 7f2a1de00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2a14004cb0 0x7f2a1412c620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:38.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.878+0000 7f2a1de00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2a14004cb0 0x7f2a1412c620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:48232/0 (socket says 172.21.15.90:48232) 2024-04-02T14:39:38.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.878+0000 7f2a1d400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2a14005670 0x7f2a1412cb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:44450/0 (socket says 172.21.15.90:44450) 2024-04-02T14:39:38.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.878+0000 7f2a1de00640 1 -- 172.21.15.90:0/2134755785 learned_addr learned my addr 172.21.15.90:0/2134755785 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:39:38.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.878+0000 7f2a1e800640 1 --2- 172.21.15.90:0/2134755785 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2a1409be10 0x7f2a14130ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:38.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.878+0000 7f2a1d400640 1 -- 172.21.15.90:0/2134755785 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2a14004cb0 msgr2=0x7f2a1412c620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:38.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.878+0000 7f2a1d400640 1 --2- 172.21.15.90:0/2134755785 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2a14004cb0 0x7f2a1412c620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:38.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.878+0000 7f2a1d400640 1 -- 172.21.15.90:0/2134755785 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2a1409be10 msgr2=0x7f2a14130ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:38.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.878+0000 7f2a1d400640 1 --2- 172.21.15.90:0/2134755785 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2a1409be10 0x7f2a14130ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:38.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.878+0000 7f2a1d400640 1 -- 172.21.15.90:0/2134755785 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a14131610 con 0x7f2a14005670 2024-04-02T14:39:38.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.879+0000 7f2a1d400640 1 --2- 172.21.15.90:0/2134755785 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2a14005670 0x7f2a1412cb60 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f2a0800b3f0 tx=0x7f2a0800b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:39:38.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.879+0000 7f2a12a00640 1 -- 172.21.15.90:0/2134755785 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a0801d070 con 0x7f2a14005670 2024-04-02T14:39:38.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.879+0000 7f2a1f200640 1 -- 172.21.15.90:0/2134755785 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a14131900 con 0x7f2a14005670 2024-04-02T14:39:38.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.879+0000 7f2a12a00640 1 -- 172.21.15.90:0/2134755785 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2a08018440 con 0x7f2a14005670 2024-04-02T14:39:38.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.879+0000 7f2a12a00640 1 -- 172.21.15.90:0/2134755785 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a0801d070 con 0x7f2a14005670 2024-04-02T14:39:38.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.879+0000 7f2a1f200640 1 -- 172.21.15.90:0/2134755785 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a14094e50 con 0x7f2a14005670 2024-04-02T14:39:38.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.883+0000 7f2a12a00640 1 -- 172.21.15.90:0/2134755785 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f2a08020070 con 0x7f2a14005670 2024-04-02T14:39:38.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.883+0000 7f2a23634740 1 Processor -- start 2024-04-02T14:39:38.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.884+0000 7f2a23634740 1 -- start start 2024-04-02T14:39:38.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.884+0000 7f2a23634740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5604ebbe6ba0 0x5604ebbe58f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:38.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.884+0000 7f2a23634740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5604ebbe5ec0 0x5604ebbe6340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:38.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.884+0000 7f2a23634740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5604ebbf7a40 0x5604ebbf9e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:38.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.884+0000 7f2a23634740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5604ebb26400 con 0x5604ebbf7a40 2024-04-02T14:39:38.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.884+0000 7f2a23634740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x5604ebb0af10 con 0x5604ebbe6ba0 2024-04-02T14:39:38.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.884+0000 7f2a23634740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5604ebb4a6f0 con 0x5604ebbe5ec0 2024-04-02T14:39:38.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.884+0000 7f2a1e800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5604ebbe6ba0 0x5604ebbe58f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:38.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.884+0000 7f2a1e800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5604ebbe6ba0 0x5604ebbe58f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:48236/0 (socket says 172.21.15.90:48236) 2024-04-02T14:39:38.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.884+0000 7f2a1e800640 1 -- 172.21.15.90:0/3773459020 learned_addr learned my addr 172.21.15.90:0/3773459020 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:39:38.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.884+0000 7f2a1de00640 1 --2- 172.21.15.90:0/3773459020 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5604ebbe5ec0 0x5604ebbe6340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:38.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.885+0000 7f2a1d400640 1 --2- 172.21.15.90:0/3773459020 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5604ebbf7a40 0x5604ebbf9e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:38.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.885+0000 7f2a1e800640 1 -- 172.21.15.90:0/3773459020 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5604ebbe5ec0 msgr2=0x5604ebbe6340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:39:38.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.885+0000 7f2a1e800640 1 --2- 172.21.15.90:0/3773459020 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5604ebbe5ec0 0x5604ebbe6340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:38.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.885+0000 7f2a1e800640 1 -- 172.21.15.90:0/3773459020 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5604ebbf7a40 msgr2=0x5604ebbf9e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:39:38.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.885+0000 7f2a1e800640 1 --2- 172.21.15.90:0/3773459020 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5604ebbf7a40 0x5604ebbf9e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:38.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.885+0000 7f2a1e800640 1 -- 172.21.15.90:0/3773459020 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5604eb85e3f0 con 0x5604ebbe6ba0 2024-04-02T14:39:38.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.885+0000 7f2a1d400640 1 --2- 172.21.15.90:0/3773459020 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5604ebbf7a40 0x5604ebbf9e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:39:38.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.885+0000 7f2a12a00640 1 --2- 172.21.15.90:0/2134755785 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f2a000a4120 0x7f2a000a65e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:38.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.885+0000 7f2a1de00640 1 --2- 172.21.15.90:0/3773459020 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5604ebbe5ec0 0x5604ebbe6340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:39:38.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.885+0000 7f2a12a00640 1 -- 172.21.15.90:0/2134755785 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f2a08010890 con 0x7f2a14005670 2024-04-02T14:39:38.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.886+0000 7f2a1e800640 1 --2- 172.21.15.90:0/3773459020 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5604ebbe6ba0 0x5604ebbe58f0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f2a18051160 tx=0x7f2a18075420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1ddfa13692241f32 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:39:38.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.886+0000 7f2a07000640 1 -- 172.21.15.90:0/3773459020 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a18086070 con 0x5604ebbe6ba0 2024-04-02T14:39:38.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.886+0000 7f2a07000640 1 -- 172.21.15.90:0/3773459020 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2a18075b60 con 0x5604ebbe6ba0 2024-04-02T14:39:38.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.886+0000 7f2a07000640 1 -- 172.21.15.90:0/3773459020 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a18086070 con 0x5604ebbe6ba0 2024-04-02T14:39:38.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.887+0000 7f2a23634740 1 -- 172.21.15.90:0/3773459020 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5604ebbe6ba0 msgr2=0x5604ebbe58f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:38.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.887+0000 7f2a23634740 1 --2- 172.21.15.90:0/3773459020 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5604ebbe6ba0 0x5604ebbe58f0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f2a18051160 tx=0x7f2a18075420 comp rx=0 tx=0).stop 2024-04-02T14:39:38.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.888+0000 7f2a1de00640 1 --2- 172.21.15.90:0/2134755785 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f2a000a4120 0x7f2a000a65e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:38.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.889+0000 7f2a23634740 1 -- 172.21.15.90:0/3773459020 shutdown_connections 2024-04-02T14:39:38.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.889+0000 7f2a23634740 1 --2- 172.21.15.90:0/3773459020 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5604ebbf7a40 0x5604ebbf9e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:38.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.889+0000 7f2a1de00640 1 --2- 172.21.15.90:0/2134755785 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f2a000a4120 0x7f2a000a65e0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x5604ebbe70f0 tx=0x7f2a0c02c910 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:39:38.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.889+0000 7f2a23634740 1 --2- 172.21.15.90:0/3773459020 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5604ebbe5ec0 0x5604ebbe6340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:38.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.889+0000 7f2a23634740 1 --2- 172.21.15.90:0/3773459020 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5604ebbe6ba0 0x5604ebbe58f0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:38.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.890+0000 7f2a23634740 1 -- 172.21.15.90:0/3773459020 >> 172.21.15.90:0/3773459020 conn(0x5604ebbaca50 msgr2=0x5604ebbe0910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:39:38.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.890+0000 7f2a23634740 1 -- 172.21.15.90:0/3773459020 shutdown_connections 2024-04-02T14:39:38.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.890+0000 7f2a23634740 1 -- 172.21.15.90:0/3773459020 wait complete. 2024-04-02T14:39:38.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.891+0000 7f2a23634740 1 -- start start 2024-04-02T14:39:38.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.891+0000 7f2a23634740 1 Processor -- start 2024-04-02T14:39:38.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.892+0000 7f2a23634740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5604ebbe5ec0 0x5604ebbf49b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:38.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.893+0000 7f2a23634740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5604ebbe6ba0 0x5604ebbf4ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:38.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.893+0000 7f2a23634740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5604ebbf7a40 0x5604ebbf5430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:38.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.893+0000 7f2a1de00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5604ebbe5ec0 0x5604ebbf49b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:38.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.893+0000 7f2a1d400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5604ebbe6ba0 0x5604ebbf4ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:38.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.893+0000 7f2a1d400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5604ebbe6ba0 0x5604ebbf4ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:44466/0 (socket says 172.21.15.90:44466) 2024-04-02T14:39:38.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.893+0000 7f2a1de00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5604ebbe5ec0 0x5604ebbf49b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:48240/0 (socket says 172.21.15.90:48240) 2024-04-02T14:39:38.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.893+0000 7f2a1e800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5604ebbf7a40 0x5604ebbf5430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:38.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.893+0000 7f2a1e800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5604ebbf7a40 0x5604ebbf5430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:48730/0 (socket says 172.21.15.90:48730) 2024-04-02T14:39:38.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.893+0000 7f2a1e800640 1 -- 172.21.15.90:0/949966138 learned_addr learned my addr 172.21.15.90:0/949966138 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:39:38.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.893+0000 7f2a23634740 1 -- 172.21.15.90:0/949966138 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5604ebb466a0 con 0x5604ebbf7a40 2024-04-02T14:39:38.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.894+0000 7f2a23634740 1 -- 172.21.15.90:0/949966138 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x5604ebaf2980 con 0x5604ebbe5ec0 2024-04-02T14:39:38.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.894+0000 7f2a23634740 1 -- 172.21.15.90:0/949966138 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5604ebb05cb0 con 0x5604ebbe6ba0 2024-04-02T14:39:38.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.894+0000 7f2a23634740 10 client.0 _ll_register_callbacks cb 0x5604ebb056a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:39:38.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.894+0000 7f2a1d400640 1 -- 172.21.15.90:0/949966138 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5604ebbe5ec0 msgr2=0x5604ebbf49b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:39:38.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.894+0000 7f2a1d400640 1 --2- 172.21.15.90:0/949966138 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5604ebbe5ec0 0x5604ebbf49b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:38.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.894+0000 7f2a1d400640 1 -- 172.21.15.90:0/949966138 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5604ebbf7a40 msgr2=0x5604ebbf5430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:39:38.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.894+0000 7f2a1d400640 1 --2- 172.21.15.90:0/949966138 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5604ebbf7a40 0x5604ebbf5430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:38.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.894+0000 7f2a1d400640 1 -- 172.21.15.90:0/949966138 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5604eb9287d0 con 0x5604ebbe6ba0 2024-04-02T14:39:38.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.894+0000 7f2a1d400640 1 --2- 172.21.15.90:0/949966138 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5604ebbe6ba0 0x5604ebbf4ef0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f2a080d0a30 tx=0x7f2a08018d80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=34c41f2d9ca60513 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:39:38.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.895+0000 7f29fb400640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:39:38.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.895+0000 7f29fb400640 1 -- 172.21.15.90:0/949966138 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a0801d070 con 0x5604ebbe6ba0 2024-04-02T14:39:38.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.895+0000 7f29fb400640 1 -- 172.21.15.90:0/949966138 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f2a0800e110 con 0x5604ebbe6ba0 2024-04-02T14:39:38.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.896+0000 7f2a23634740 1 -- 172.21.15.90:0/949966138 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2a18075b60 con 0x5604ebbe6ba0 2024-04-02T14:39:38.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.896+0000 7f29fb400640 1 -- 172.21.15.90:0/949966138 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f2a080a7ef0 con 0x5604ebbe6ba0 2024-04-02T14:39:38.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.896+0000 7f2a23634740 10 client.26722 fetch_fsmap learned FSMap version 27 2024-04-02T14:39:38.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.896+0000 7f2a23634740 1 -- 172.21.15.90:0/949966138 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5604ebc06860 con 0x5604ebbe6ba0 2024-04-02T14:39:38.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.896+0000 7f29fb400640 1 -- 172.21.15.90:0/949966138 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2a18075b60 con 0x5604ebbe6ba0 2024-04-02T14:39:38.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.897+0000 7f29fb400640 1 -- 172.21.15.90:0/949966138 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f2a080a3150 con 0x5604ebbe6ba0 2024-04-02T14:39:38.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.897+0000 7f2a23634740 10 client.26722 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:39:38.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.897+0000 7f2a23634740 10 client.26722 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.c 2024-04-02T14:39:38.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.897+0000 7f2a23634740 20 client.26722 populate_metadata read hostname 'smithi090' 2024-04-02T14:39:38.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.897+0000 7f2a23634740 1 --2- 172.21.15.90:0/949966138 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x5604ebc07d60 0x5604ebc0a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:38.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.897+0000 7f2a23634740 4 client.26722 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:39:38.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.897+0000 7f2a23634740 1 -- 172.21.15.90:0/949966138 --> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5604ebb0a040 con 0x5604ebc07d60 2024-04-02T14:39:38.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.897+0000 7f2a1de00640 1 --2- 172.21.15.90:0/949966138 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x5604ebc07d60 0x5604ebc0a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:38.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.899+0000 7f29fb400640 10 client.26722 ms_handle_connect on v2:172.21.15.177:6836/4223414858 2024-04-02T14:39:38.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.902+0000 7f2a1de00640 1 --2- 172.21.15.90:0/949966138 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x5604ebc07d60 0x5604ebc0a1c0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=b7d6657f59ec5f1e server_cookie=d6edc1954acce2ab in_seq=0 out_seq=0 2024-04-02T14:39:38.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.902+0000 7f29fb400640 1 -- 172.21.15.90:0/949966138 <== mds.4 v2:172.21.15.177:6836/4223414858 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5604ebb0a040 con 0x5604ebc07d60 2024-04-02T14:39:38.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.902+0000 7f29fb400640 10 client.26722 handle_command_reply: tid=0 2024-04-02T14:39:38.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.903+0000 7f2a23634740 1 client.26722 shutdown 2024-04-02T14:39:38.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.903+0000 7f2a23634740 1 -- 172.21.15.90:0/949966138 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5604ebbe6ba0 msgr2=0x5604ebbf4ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:38.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.903+0000 7f2a23634740 1 --2- 172.21.15.90:0/949966138 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5604ebbe6ba0 0x5604ebbf4ef0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f2a080d0a30 tx=0x7f2a08018d80 comp rx=0 tx=0).stop 2024-04-02T14:39:38.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.903+0000 7f2a23634740 1 -- 172.21.15.90:0/949966138 shutdown_connections 2024-04-02T14:39:38.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.903+0000 7f2a23634740 1 --2- 172.21.15.90:0/949966138 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x5604ebc07d60 0x5604ebc0a1c0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:38.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.903+0000 7f2a23634740 1 --2- 172.21.15.90:0/949966138 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5604ebbf7a40 0x5604ebbf5430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:38.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.903+0000 7f2a23634740 1 --2- 172.21.15.90:0/949966138 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5604ebbe6ba0 0x5604ebbf4ef0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:38.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.903+0000 7f2a23634740 1 --2- 172.21.15.90:0/949966138 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5604ebbe5ec0 0x5604ebbf49b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:38.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.903+0000 7f2a23634740 1 -- 172.21.15.90:0/949966138 >> 172.21.15.90:0/949966138 conn(0x5604ebbaca50 msgr2=0x5604ebbe2660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:39:38.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.903+0000 7f29fb400640 0 client.26722 ms_handle_reset on v2:172.21.15.177:6836/4223414858 2024-04-02T14:39:38.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.903+0000 7f2a23634740 1 -- 172.21.15.90:0/949966138 shutdown_connections 2024-04-02T14:39:38.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.903+0000 7f2a23634740 1 -- 172.21.15.90:0/949966138 wait complete. 2024-04-02T14:39:38.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.903+0000 7f2a23634740 20 client.26722 trim_cache size 0 max 16384 2024-04-02T14:39:38.929 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.925+0000 7f2a23634740 1 Processor -- start 2024-04-02T14:39:38.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.925+0000 7f2a23634740 1 -- start start 2024-04-02T14:39:38.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.926+0000 7f2a23634740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5604ebbe5ec0 0x5604ebbf70e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:38.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.926+0000 7f2a23634740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5604ebbe6ba0 0x5604ebbe0ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:38.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.926+0000 7f2a23634740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5604ebbf7a40 0x5604ebc03510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:38.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.926+0000 7f2a23634740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f2a1413f7d0 con 0x5604ebbe5ec0 2024-04-02T14:39:38.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.926+0000 7f2a23634740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f2a14141310 con 0x5604ebbe6ba0 2024-04-02T14:39:38.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.926+0000 7f2a23634740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f2a1413fea0 con 0x5604ebbf7a40 2024-04-02T14:39:38.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.926+0000 7f2a1d400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5604ebbe6ba0 0x5604ebbe0ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:38.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.926+0000 7f2a1e800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5604ebbf7a40 0x5604ebc03510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:38.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.927+0000 7f2a1d400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5604ebbe6ba0 0x5604ebbe0ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:48244/0 (socket says 172.21.15.90:48244) 2024-04-02T14:39:38.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.927+0000 7f2a1d400640 1 -- 172.21.15.90:0/814037613 learned_addr learned my addr 172.21.15.90:0/814037613 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:39:38.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.927+0000 7f2a1de00640 1 --2- 172.21.15.90:0/814037613 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5604ebbe5ec0 0x5604ebbf70e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:38.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.927+0000 7f2a1d400640 1 -- 172.21.15.90:0/814037613 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5604ebbf7a40 msgr2=0x5604ebc03510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:39:38.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.927+0000 7f2a1d400640 1 --2- 172.21.15.90:0/814037613 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5604ebbf7a40 0x5604ebc03510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:38.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.927+0000 7f2a1d400640 1 -- 172.21.15.90:0/814037613 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5604ebbe5ec0 msgr2=0x5604ebbf70e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:39:38.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.927+0000 7f2a1d400640 1 --2- 172.21.15.90:0/814037613 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5604ebbe5ec0 0x5604ebbf70e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:38.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.927+0000 7f2a1d400640 1 -- 172.21.15.90:0/814037613 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5604ebbe2c10 con 0x5604ebbe6ba0 2024-04-02T14:39:38.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.927+0000 7f2a1de00640 1 --2- 172.21.15.90:0/814037613 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5604ebbe5ec0 0x5604ebbf70e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:39:38.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.927+0000 7f2a1d400640 1 --2- 172.21.15.90:0/814037613 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5604ebbe6ba0 0x5604ebbe0ab0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f2a0809f2c0 tx=0x7f2a0809d840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d86aaf8126a96e6a server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:39:38.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.927+0000 7f2a06600640 1 -- 172.21.15.90:0/814037613 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a0801d070 con 0x5604ebbe6ba0 2024-04-02T14:39:38.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.927+0000 7f2a06600640 1 -- 172.21.15.90:0/814037613 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2a080bc020 con 0x5604ebbe6ba0 2024-04-02T14:39:38.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.927+0000 7f2a06600640 1 -- 172.21.15.90:0/814037613 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a0809b9a0 con 0x5604ebbe6ba0 2024-04-02T14:39:38.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.928+0000 7f2a23634740 1 -- 172.21.15.90:0/814037613 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5604ebbe6ba0 msgr2=0x5604ebbe0ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:38.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.928+0000 7f2a23634740 1 --2- 172.21.15.90:0/814037613 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5604ebbe6ba0 0x5604ebbe0ab0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f2a0809f2c0 tx=0x7f2a0809d840 comp rx=0 tx=0).stop 2024-04-02T14:39:38.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.928+0000 7f2a23634740 1 -- 172.21.15.90:0/814037613 shutdown_connections 2024-04-02T14:39:38.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.928+0000 7f2a23634740 1 --2- 172.21.15.90:0/814037613 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5604ebbf7a40 0x5604ebc03510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:38.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.928+0000 7f2a23634740 1 --2- 172.21.15.90:0/814037613 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5604ebbe6ba0 0x5604ebbe0ab0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:38.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.928+0000 7f2a23634740 1 --2- 172.21.15.90:0/814037613 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5604ebbe5ec0 0x5604ebbf70e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:38.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.928+0000 7f2a23634740 1 -- 172.21.15.90:0/814037613 >> 172.21.15.90:0/814037613 conn(0x5604ebbaca50 msgr2=0x5604ebbdf260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:39:38.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.928+0000 7f2a23634740 1 -- 172.21.15.90:0/814037613 shutdown_connections 2024-04-02T14:39:38.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.929+0000 7f2a23634740 1 -- 172.21.15.90:0/814037613 wait complete. 2024-04-02T14:39:38.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.929+0000 7f2a23634740 1 -- start start 2024-04-02T14:39:38.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.930+0000 7f2a23634740 1 Processor -- start 2024-04-02T14:39:38.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.930+0000 7f2a23634740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5604ebbe5ec0 0x5604ebbffe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:38.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.930+0000 7f2a23634740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5604ebbe6ba0 0x5604ebbf1d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:38.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.930+0000 7f2a23634740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5604ebbf7a40 0x5604ebbf22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:38.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.930+0000 7f2a23634740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f2a14140c40 con 0x5604ebbe5ec0 2024-04-02T14:39:38.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.930+0000 7f2a23634740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f2a14140570 con 0x5604ebbe6ba0 2024-04-02T14:39:38.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.930+0000 7f2a23634740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f2a1413d7b0 con 0x5604ebbf7a40 2024-04-02T14:39:38.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.930+0000 7f2a23634740 10 client.0 _ll_register_callbacks cb 0x7f2a0809ee70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:39:38.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.931+0000 7f2a1d400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5604ebbe6ba0 0x5604ebbf1d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:38.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.931+0000 7f2a1d400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5604ebbe6ba0 0x5604ebbf1d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:48254/0 (socket says 172.21.15.90:48254) 2024-04-02T14:39:38.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.931+0000 7f2a1d400640 1 -- 172.21.15.90:0/2730023511 learned_addr learned my addr 172.21.15.90:0/2730023511 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:39:38.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.931+0000 7f2a1e800640 1 --2- 172.21.15.90:0/2730023511 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5604ebbf7a40 0x5604ebbf22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:38.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.931+0000 7f2a1de00640 1 --2- 172.21.15.90:0/2730023511 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5604ebbe5ec0 0x5604ebbffe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:38.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.931+0000 7f2a1d400640 1 -- 172.21.15.90:0/2730023511 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5604ebbf7a40 msgr2=0x5604ebbf22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:39:38.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.931+0000 7f2a1d400640 1 --2- 172.21.15.90:0/2730023511 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5604ebbf7a40 0x5604ebbf22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:38.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.931+0000 7f2a1d400640 1 -- 172.21.15.90:0/2730023511 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5604ebbe5ec0 msgr2=0x5604ebbffe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:39:38.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.931+0000 7f2a1d400640 1 --2- 172.21.15.90:0/2730023511 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5604ebbe5ec0 0x5604ebbffe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:38.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.932+0000 7f2a1d400640 1 -- 172.21.15.90:0/2730023511 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5604ebbf85a0 con 0x5604ebbe6ba0 2024-04-02T14:39:38.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.932+0000 7f2a1e800640 1 --2- 172.21.15.90:0/2730023511 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5604ebbf7a40 0x5604ebbf22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:39:38.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.932+0000 7f2a1de00640 1 --2- 172.21.15.90:0/2730023511 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5604ebbe5ec0 0x5604ebbffe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:39:38.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.932+0000 7f2a1d400640 1 --2- 172.21.15.90:0/2730023511 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5604ebbe6ba0 0x5604ebbf1d10 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f2a0809d5e0 tx=0x7f2a080188b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ef5b975a8ef0d751 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:39:38.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.932+0000 7f29fb400640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:39:38.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.932+0000 7f29fb400640 1 -- 172.21.15.90:0/2730023511 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a0801d070 con 0x5604ebbe6ba0 2024-04-02T14:39:38.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.932+0000 7f29fb400640 1 -- 172.21.15.90:0/2730023511 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f2a080101f0 con 0x5604ebbe6ba0 2024-04-02T14:39:38.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.932+0000 7f2a23634740 1 -- 172.21.15.90:0/2730023511 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2a080bc020 con 0x5604ebbe6ba0 2024-04-02T14:39:38.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.932+0000 7f29fb400640 1 -- 172.21.15.90:0/2730023511 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f2a0809f330 con 0x5604ebbe6ba0 2024-04-02T14:39:38.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.932+0000 7f2a23634740 10 client.26849 fetch_fsmap learned FSMap version 27 2024-04-02T14:39:38.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.932+0000 7f2a23634740 1 -- 172.21.15.90:0/2730023511 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5604ebbf85a0 con 0x5604ebbe6ba0 2024-04-02T14:39:38.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.933+0000 7f29fb400640 1 -- 172.21.15.90:0/2730023511 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2a080bc020 con 0x5604ebbe6ba0 2024-04-02T14:39:38.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.933+0000 7f29fb400640 1 -- 172.21.15.90:0/2730023511 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f2a0809b1a0 con 0x5604ebbe6ba0 2024-04-02T14:39:38.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.933+0000 7f2a23634740 10 client.26849 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:39:38.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.933+0000 7f2a23634740 10 client.26849 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.c 2024-04-02T14:39:38.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.933+0000 7f2a23634740 20 client.26849 populate_metadata read hostname 'smithi090' 2024-04-02T14:39:38.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.933+0000 7f2a23634740 1 --2- 172.21.15.90:0/2730023511 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x5604ebbed540 0x5604ebbed950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:38.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.933+0000 7f2a23634740 4 client.26849 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-02T14:39:38.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.933+0000 7f2a23634740 1 -- 172.21.15.90:0/2730023511 --> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5604ebb57c30 con 0x5604ebbed540 2024-04-02T14:39:38.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.933+0000 7f2a1de00640 1 --2- 172.21.15.90:0/2730023511 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x5604ebbed540 0x5604ebbed950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:38.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.934+0000 7f2a1de00640 1 --2- 172.21.15.90:0/2730023511 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x5604ebbed540 0x5604ebbed950 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=4cd97f5d830cfef4 server_cookie=6575254f45f5e626 in_seq=0 out_seq=0 2024-04-02T14:39:38.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.934+0000 7f29fb400640 10 client.26849 ms_handle_connect on v2:172.21.15.177:6836/4223414858 2024-04-02T14:39:38.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.934+0000 7f29fb400640 1 -- 172.21.15.90:0/2730023511 <== mds.4 v2:172.21.15.177:6836/4223414858 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5604ebb57c30 con 0x5604ebbed540 2024-04-02T14:39:38.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.934+0000 7f29fb400640 10 client.26849 handle_command_reply: tid=0 2024-04-02T14:39:38.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.935+0000 7f2a23634740 1 client.26849 shutdown 2024-04-02T14:39:38.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.935+0000 7f2a23634740 1 -- 172.21.15.90:0/2730023511 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5604ebbe6ba0 msgr2=0x5604ebbf1d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:38.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.935+0000 7f2a23634740 1 --2- 172.21.15.90:0/2730023511 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5604ebbe6ba0 0x5604ebbf1d10 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f2a0809d5e0 tx=0x7f2a080188b0 comp rx=0 tx=0).stop 2024-04-02T14:39:38.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.936+0000 7f2a23634740 1 -- 172.21.15.90:0/2730023511 shutdown_connections 2024-04-02T14:39:38.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.936+0000 7f2a23634740 1 --2- 172.21.15.90:0/2730023511 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5604ebbf7a40 0x5604ebbf22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:38.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.936+0000 7f2a23634740 1 --2- 172.21.15.90:0/2730023511 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5604ebbe6ba0 0x5604ebbf1d10 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:38.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.936+0000 7f2a23634740 1 --2- 172.21.15.90:0/2730023511 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x5604ebbed540 0x5604ebbed950 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:38.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.936+0000 7f2a23634740 1 --2- 172.21.15.90:0/2730023511 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5604ebbe5ec0 0x5604ebbffe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:38.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.936+0000 7f2a23634740 1 -- 172.21.15.90:0/2730023511 >> 172.21.15.90:0/2730023511 conn(0x5604ebbaca50 msgr2=0x5604ebbdfaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:39:38.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.936+0000 7f29fb400640 0 client.26849 ms_handle_reset on v2:172.21.15.177:6836/4223414858 2024-04-02T14:39:38.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.936+0000 7f2a23634740 1 -- 172.21.15.90:0/2730023511 shutdown_connections 2024-04-02T14:39:38.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.936+0000 7f2a23634740 1 -- 172.21.15.90:0/2730023511 wait complete. 2024-04-02T14:39:38.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.936+0000 7f2a23634740 20 client.26849 trim_cache size 0 max 16384 2024-04-02T14:39:38.940 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-02T14:39:38.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.938+0000 7f2a1f200640 1 -- 172.21.15.90:0/2134755785 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f2a000a4120 msgr2=0x7f2a000a65e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:38.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.938+0000 7f2a1f200640 1 --2- 172.21.15.90:0/2134755785 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f2a000a4120 0x7f2a000a65e0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x5604ebbe70f0 tx=0x7f2a0c02c910 comp rx=0 tx=0).stop 2024-04-02T14:39:38.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.938+0000 7f2a1f200640 1 -- 172.21.15.90:0/2134755785 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2a14005670 msgr2=0x7f2a1412cb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:38.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.938+0000 7f2a1f200640 1 --2- 172.21.15.90:0/2134755785 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2a14005670 0x7f2a1412cb60 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f2a0800b3f0 tx=0x7f2a0800b8b0 comp rx=0 tx=0).stop 2024-04-02T14:39:38.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.938+0000 7f2a1f200640 1 -- 172.21.15.90:0/2134755785 shutdown_connections 2024-04-02T14:39:38.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.938+0000 7f2a1f200640 1 --2- 172.21.15.90:0/2134755785 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f2a000a4120 0x7f2a000a65e0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:38.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.938+0000 7f2a1f200640 1 --2- 172.21.15.90:0/2134755785 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2a1409be10 0x7f2a14130ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:38.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.938+0000 7f2a1f200640 1 --2- 172.21.15.90:0/2134755785 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2a14005670 0x7f2a1412cb60 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:38.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.938+0000 7f2a1f200640 1 --2- 172.21.15.90:0/2134755785 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2a14004cb0 0x7f2a1412c620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:38.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.938+0000 7f2a1f200640 1 -- 172.21.15.90:0/2134755785 >> 172.21.15.90:0/2134755785 conn(0x7f2a1400a740 msgr2=0x7f2a14093b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:39:38.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.938+0000 7f2a1f200640 1 -- 172.21.15.90:0/2134755785 shutdown_connections 2024-04-02T14:39:38.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:38.938+0000 7f2a1f200640 1 -- 172.21.15.90:0/2134755785 wait complete. 2024-04-02T14:39:38.962 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-04-02T14:39:39.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.220+0000 7f555e200640 1 -- 172.21.15.90:0/2374288383 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f55581048c0 msgr2=0x7f5558108900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:39.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.220+0000 7f555e200640 1 --2- 172.21.15.90:0/2374288383 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f55581048c0 0x7f5558108900 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f5544009900 tx=0x7f554402bc30 comp rx=0 tx=0).stop 2024-04-02T14:39:39.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.220+0000 7f555e200640 1 -- 172.21.15.90:0/2374288383 shutdown_connections 2024-04-02T14:39:39.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.220+0000 7f555e200640 1 --2- 172.21.15.90:0/2374288383 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5558108fe0 0x7f5558110bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:39.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.220+0000 7f555e200640 1 --2- 172.21.15.90:0/2374288383 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f55581048c0 0x7f5558108900 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:39.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.220+0000 7f555e200640 1 --2- 172.21.15.90:0/2374288383 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5558103f10 0x7f55581042f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:39.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.220+0000 7f555e200640 1 -- 172.21.15.90:0/2374288383 >> 172.21.15.90:0/2374288383 conn(0x7f55580ff7b0 msgr2=0x7f5558101bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:39:39.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.221+0000 7f555e200640 1 -- 172.21.15.90:0/2374288383 shutdown_connections 2024-04-02T14:39:39.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.221+0000 7f555e200640 1 -- 172.21.15.90:0/2374288383 wait complete. 2024-04-02T14:39:39.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.221+0000 7f555e200640 1 Processor -- start 2024-04-02T14:39:39.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.222+0000 7f555e200640 1 -- start start 2024-04-02T14:39:39.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.223+0000 7f555e200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5558103f10 0x7f555819a620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:39.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.223+0000 7f555e200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5558108fe0 0x7f555819ab60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:39.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.223+0000 7f555e200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f555819eee0 0x7f555819f390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:39.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.223+0000 7f555e200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f55581140c0 con 0x7f5558108fe0 2024-04-02T14:39:39.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.223+0000 7f555e200640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f5558113f40 con 0x7f5558103f10 2024-04-02T14:39:39.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.223+0000 7f555e200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f5558114240 con 0x7f555819eee0 2024-04-02T14:39:39.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.224+0000 7f5557400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5558103f10 0x7f555819a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:39.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.224+0000 7f5557400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5558103f10 0x7f555819a620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:48262/0 (socket says 172.21.15.90:48262) 2024-04-02T14:39:39.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.224+0000 7f5557400640 1 -- 172.21.15.90:0/663531451 learned_addr learned my addr 172.21.15.90:0/663531451 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:39:39.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.224+0000 7f5557e00640 1 --2- 172.21.15.90:0/663531451 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f555819eee0 0x7f555819f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:39.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.224+0000 7f5557400640 1 -- 172.21.15.90:0/663531451 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f555819eee0 msgr2=0x7f555819f390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:39.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.224+0000 7f5557400640 1 --2- 172.21.15.90:0/663531451 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f555819eee0 0x7f555819f390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:39.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.224+0000 7f5557400640 1 -- 172.21.15.90:0/663531451 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5558108fe0 msgr2=0x7f555819ab60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:39.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.224+0000 7f5557400640 1 --2- 172.21.15.90:0/663531451 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5558108fe0 0x7f555819ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:39.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.224+0000 7f5557400640 1 -- 172.21.15.90:0/663531451 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f555819f9d0 con 0x7f5558103f10 2024-04-02T14:39:39.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.225+0000 7f5557400640 1 --2- 172.21.15.90:0/663531451 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5558103f10 0x7f555819a620 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f554800e9f0 tx=0x7f554800edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:39:39.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.225+0000 7f554fe00640 1 -- 172.21.15.90:0/663531451 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f554800cd80 con 0x7f5558103f10 2024-04-02T14:39:39.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.225+0000 7f554fe00640 1 -- 172.21.15.90:0/663531451 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5548005500 con 0x7f5558103f10 2024-04-02T14:39:39.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.225+0000 7f555e200640 1 -- 172.21.15.90:0/663531451 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f555819fcc0 con 0x7f5558103f10 2024-04-02T14:39:39.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.226+0000 7f554fe00640 1 -- 172.21.15.90:0/663531451 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5548010690 con 0x7f5558103f10 2024-04-02T14:39:39.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.226+0000 7f555e200640 1 -- 172.21.15.90:0/663531451 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55581a74b0 con 0x7f5558103f10 2024-04-02T14:39:39.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.231+0000 7f554fe00640 1 -- 172.21.15.90:0/663531451 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f55480046c0 con 0x7f5558103f10 2024-04-02T14:39:39.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.231+0000 7f5560e5a740 1 Processor -- start 2024-04-02T14:39:39.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.232+0000 7f5560e5a740 1 -- start start 2024-04-02T14:39:39.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.232+0000 7f5560e5a740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559a2f9699f0 0x559a2f968740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:39.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.232+0000 7f5560e5a740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559a2f968d10 0x559a2f969190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:39.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.233+0000 7f5560e5a740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559a2f983090 0x559a2f985480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:39.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.233+0000 7f5560e5a740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x559a2f8a9400 con 0x559a2f968d10 2024-04-02T14:39:39.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.233+0000 7f5560e5a740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x559a2f88df10 con 0x559a2f983090 2024-04-02T14:39:39.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.233+0000 7f5560e5a740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x559a2f8cd6f0 con 0x559a2f9699f0 2024-04-02T14:39:39.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.233+0000 7f5557e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559a2f9699f0 0x559a2f968740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:39.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.233+0000 7f5557e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559a2f9699f0 0x559a2f968740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:44508/0 (socket says 172.21.15.90:44508) 2024-04-02T14:39:39.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.233+0000 7f5557e00640 1 -- 172.21.15.90:0/1678556884 learned_addr learned my addr 172.21.15.90:0/1678556884 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:39:39.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.233+0000 7f5556a00640 1 --2- 172.21.15.90:0/1678556884 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559a2f983090 0x559a2f985480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:39.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.234+0000 7f554fe00640 1 --2- 172.21.15.90:0/663531451 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f553c0a3e20 0x7f553c0a62e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:39.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.235+0000 7f5557e00640 1 -- 172.21.15.90:0/1678556884 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559a2f983090 msgr2=0x559a2f985480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:39:39.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.235+0000 7f5557e00640 1 --2- 172.21.15.90:0/1678556884 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559a2f983090 0x559a2f985480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:39.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.235+0000 7f5557e00640 1 -- 172.21.15.90:0/1678556884 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559a2f968d10 msgr2=0x559a2f969190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:39:39.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.235+0000 7f5557400640 1 --2- 172.21.15.90:0/1678556884 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559a2f968d10 0x559a2f969190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:39.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.235+0000 7f5557e00640 1 --2- 172.21.15.90:0/1678556884 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559a2f968d10 0x559a2f969190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:39.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.235+0000 7f5557e00640 1 -- 172.21.15.90:0/1678556884 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559a2f5e13f0 con 0x559a2f9699f0 2024-04-02T14:39:39.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.235+0000 7f554fe00640 1 -- 172.21.15.90:0/663531451 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f55480d0d30 con 0x7f5558103f10 2024-04-02T14:39:39.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.235+0000 7f5556a00640 1 --2- 172.21.15.90:0/1678556884 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559a2f983090 0x559a2f985480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:39:39.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.236+0000 7f5557e00640 1 --2- 172.21.15.90:0/663531451 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f553c0a3e20 0x7f553c0a62e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:39.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.237+0000 7f5557e00640 1 --2- 172.21.15.90:0/1678556884 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559a2f9699f0 0x559a2f968740 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f5550009f40 tx=0x7f5550010860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6618f97ecae58b09 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:39:39.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.238+0000 7f5557e00640 1 --2- 172.21.15.90:0/663531451 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f553c0a3e20 0x7f553c0a62e0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f5550018000 tx=0x7f555001a000 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:39:39.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.238+0000 7f5543e00640 1 -- 172.21.15.90:0/1678556884 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f555001c6e0 con 0x559a2f9699f0 2024-04-02T14:39:39.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.238+0000 7f5543e00640 1 -- 172.21.15.90:0/1678556884 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5550011500 con 0x559a2f9699f0 2024-04-02T14:39:39.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.239+0000 7f5543e00640 1 -- 172.21.15.90:0/1678556884 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f555000ee40 con 0x559a2f9699f0 2024-04-02T14:39:39.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.240+0000 7f5560e5a740 1 -- 172.21.15.90:0/1678556884 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559a2f9699f0 msgr2=0x559a2f968740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:39.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.240+0000 7f5560e5a740 1 --2- 172.21.15.90:0/1678556884 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559a2f9699f0 0x559a2f968740 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f5550009f40 tx=0x7f5550010860 comp rx=0 tx=0).stop 2024-04-02T14:39:39.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.240+0000 7f5560e5a740 1 -- 172.21.15.90:0/1678556884 shutdown_connections 2024-04-02T14:39:39.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.240+0000 7f5560e5a740 1 --2- 172.21.15.90:0/1678556884 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559a2f983090 0x559a2f985480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:39.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.240+0000 7f5560e5a740 1 --2- 172.21.15.90:0/1678556884 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559a2f968d10 0x559a2f969190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:39.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.240+0000 7f5560e5a740 1 --2- 172.21.15.90:0/1678556884 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559a2f9699f0 0x559a2f968740 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:39.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.240+0000 7f5560e5a740 1 -- 172.21.15.90:0/1678556884 >> 172.21.15.90:0/1678556884 conn(0x559a2f92fa50 msgr2=0x559a2f963760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:39:39.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.240+0000 7f5560e5a740 1 -- 172.21.15.90:0/1678556884 shutdown_connections 2024-04-02T14:39:39.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.240+0000 7f5560e5a740 1 -- 172.21.15.90:0/1678556884 wait complete. 2024-04-02T14:39:39.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.240+0000 7f5560e5a740 1 -- start start 2024-04-02T14:39:39.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.241+0000 7f5560e5a740 1 Processor -- start 2024-04-02T14:39:39.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.241+0000 7f5560e5a740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559a2f968d10 0x559a2f980040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:39.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.241+0000 7f5560e5a740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559a2f9699f0 0x559a2f980580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:39.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.241+0000 7f5560e5a740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559a2f983090 0x559a2f980ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:39.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.242+0000 7f5560e5a740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x559a2f8c96a0 con 0x559a2f9699f0 2024-04-02T14:39:39.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.242+0000 7f5560e5a740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x559a2f875980 con 0x559a2f983090 2024-04-02T14:39:39.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.242+0000 7f5560e5a740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x559a2f888cb0 con 0x559a2f968d10 2024-04-02T14:39:39.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.242+0000 7f5560e5a740 10 client.0 _ll_register_callbacks cb 0x559a2f8886a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:39:39.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.242+0000 7f5557e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559a2f983090 0x559a2f980ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:39.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.242+0000 7f5557400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559a2f968d10 0x559a2f980040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:39.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.242+0000 7f5557e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559a2f983090 0x559a2f980ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:48280/0 (socket says 172.21.15.90:48280) 2024-04-02T14:39:39.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.242+0000 7f5557400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559a2f968d10 0x559a2f980040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:44512/0 (socket says 172.21.15.90:44512) 2024-04-02T14:39:39.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.242+0000 7f5557e00640 1 -- 172.21.15.90:0/1291620788 learned_addr learned my addr 172.21.15.90:0/1291620788 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:39:39.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.243+0000 7f5557400640 1 -- 172.21.15.90:0/1291620788 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559a2f983090 msgr2=0x559a2f980ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:39:39.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.243+0000 7f5557400640 1 --2- 172.21.15.90:0/1291620788 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559a2f983090 0x559a2f980ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:39.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.243+0000 7f5557400640 1 -- 172.21.15.90:0/1291620788 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559a2f9699f0 msgr2=0x559a2f980580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:39:39.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.243+0000 7f5557400640 1 --2- 172.21.15.90:0/1291620788 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559a2f9699f0 0x559a2f980580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:39.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.243+0000 7f5557400640 1 -- 172.21.15.90:0/1291620788 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559a2f6ab7d0 con 0x559a2f968d10 2024-04-02T14:39:39.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.243+0000 7f5557400640 1 --2- 172.21.15.90:0/1291620788 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559a2f968d10 0x559a2f980040 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f55480ce430 tx=0x7f554809c7c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9b76c2bab61dff server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:39:39.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.243+0000 7f5540c00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:39:39.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.244+0000 7f5540c00640 1 -- 172.21.15.90:0/1291620788 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f554809cc80 con 0x559a2f968d10 2024-04-02T14:39:39.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.244+0000 7f5540c00640 1 -- 172.21.15.90:0/1291620788 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f554809cee0 con 0x559a2f968d10 2024-04-02T14:39:39.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.244+0000 7f5560e5a740 1 -- 172.21.15.90:0/1291620788 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5550011500 con 0x559a2f968d10 2024-04-02T14:39:39.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.244+0000 7f5540c00640 1 -- 172.21.15.90:0/1291620788 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f55480b6200 con 0x559a2f968d10 2024-04-02T14:39:39.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.244+0000 7f5560e5a740 10 client.26740 fetch_fsmap learned FSMap version 27 2024-04-02T14:39:39.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.244+0000 7f5560e5a740 1 -- 172.21.15.90:0/1291620788 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x559a2f99a800 con 0x559a2f968d10 2024-04-02T14:39:39.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.245+0000 7f5540c00640 1 -- 172.21.15.90:0/1291620788 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5550011500 con 0x559a2f968d10 2024-04-02T14:39:39.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.245+0000 7f5540c00640 1 -- 172.21.15.90:0/1291620788 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f55480a54f0 con 0x559a2f968d10 2024-04-02T14:39:39.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.245+0000 7f5560e5a740 10 client.26740 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:39:39.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.245+0000 7f5560e5a740 10 client.26740 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-04-02T14:39:39.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.245+0000 7f5560e5a740 20 client.26740 populate_metadata read hostname 'smithi090' 2024-04-02T14:39:39.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.245+0000 7f5560e5a740 1 --2- 172.21.15.90:0/1291620788 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x559a2f99bc10 0x559a2f99e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:39.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.245+0000 7f5560e5a740 4 client.26740 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:39:39.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.245+0000 7f5560e5a740 1 -- 172.21.15.90:0/1291620788 --> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559a2f88d040 con 0x559a2f99bc10 2024-04-02T14:39:39.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.246+0000 7f5556a00640 1 --2- 172.21.15.90:0/1291620788 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x559a2f99bc10 0x559a2f99e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:39.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.247+0000 7f5556a00640 1 --2- 172.21.15.90:0/1291620788 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x559a2f99bc10 0x559a2f99e0a0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=f1a8968f1fca7c5a server_cookie=889baca32659a7c9 in_seq=0 out_seq=0 2024-04-02T14:39:39.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.247+0000 7f5540c00640 10 client.26740 ms_handle_connect on v2:172.21.15.92:6836/1270635939 2024-04-02T14:39:39.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.248+0000 7f5540c00640 1 -- 172.21.15.90:0/1291620788 <== mds.3 v2:172.21.15.92:6836/1270635939 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x559a2f88d040 con 0x559a2f99bc10 2024-04-02T14:39:39.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.248+0000 7f5540c00640 10 client.26740 handle_command_reply: tid=0 2024-04-02T14:39:39.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.248+0000 7f5560e5a740 1 client.26740 shutdown 2024-04-02T14:39:39.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.248+0000 7f5560e5a740 1 -- 172.21.15.90:0/1291620788 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559a2f968d10 msgr2=0x559a2f980040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:39.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.248+0000 7f5560e5a740 1 --2- 172.21.15.90:0/1291620788 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559a2f968d10 0x559a2f980040 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f55480ce430 tx=0x7f554809c7c0 comp rx=0 tx=0).stop 2024-04-02T14:39:39.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.248+0000 7f5560e5a740 1 -- 172.21.15.90:0/1291620788 shutdown_connections 2024-04-02T14:39:39.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.248+0000 7f5560e5a740 1 --2- 172.21.15.90:0/1291620788 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x559a2f99bc10 0x559a2f99e0a0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:39.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.249+0000 7f5560e5a740 1 --2- 172.21.15.90:0/1291620788 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559a2f983090 0x559a2f980ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:39.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.249+0000 7f5560e5a740 1 --2- 172.21.15.90:0/1291620788 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559a2f9699f0 0x559a2f980580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:39.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.249+0000 7f5560e5a740 1 --2- 172.21.15.90:0/1291620788 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559a2f968d10 0x559a2f980040 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:39.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.249+0000 7f5560e5a740 1 -- 172.21.15.90:0/1291620788 >> 172.21.15.90:0/1291620788 conn(0x559a2f92fa50 msgr2=0x559a2f9654b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:39:39.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.249+0000 7f5540c00640 0 client.26740 ms_handle_reset on v2:172.21.15.92:6836/1270635939 2024-04-02T14:39:39.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.249+0000 7f5560e5a740 1 -- 172.21.15.90:0/1291620788 shutdown_connections 2024-04-02T14:39:39.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.249+0000 7f5560e5a740 1 -- 172.21.15.90:0/1291620788 wait complete. 2024-04-02T14:39:39.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.249+0000 7f5560e5a740 20 client.26740 trim_cache size 0 max 16384 2024-04-02T14:39:39.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.294+0000 7f5560e5a740 1 Processor -- start 2024-04-02T14:39:39.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.294+0000 7f5560e5a740 1 -- start start 2024-04-02T14:39:39.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.295+0000 7f5560e5a740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559a2f968d10 0x559a2f982770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:39.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.295+0000 7f5560e5a740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559a2f9699f0 0x559a2f98ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:39.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.295+0000 7f5560e5a740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559a2f983090 0x559a2f97f1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:39.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.295+0000 7f5560e5a740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f555806f5d0 con 0x559a2f9699f0 2024-04-02T14:39:39.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.295+0000 7f5560e5a740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f5558071110 con 0x559a2f983090 2024-04-02T14:39:39.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.295+0000 7f5560e5a740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f555806fca0 con 0x559a2f968d10 2024-04-02T14:39:39.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.295+0000 7f5557400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559a2f968d10 0x559a2f982770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:39.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.295+0000 7f5557400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559a2f968d10 0x559a2f982770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:44524/0 (socket says 172.21.15.90:44524) 2024-04-02T14:39:39.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.295+0000 7f5557400640 1 -- 172.21.15.90:0/33029443 learned_addr learned my addr 172.21.15.90:0/33029443 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:39:39.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.296+0000 7f5556a00640 1 --2- 172.21.15.90:0/33029443 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559a2f9699f0 0x559a2f98ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:39.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.296+0000 7f5557400640 1 -- 172.21.15.90:0/33029443 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559a2f983090 msgr2=0x559a2f97f1c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-02T14:39:39.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.296+0000 7f5557400640 1 --2- 172.21.15.90:0/33029443 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559a2f983090 0x559a2f97f1c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:39.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.296+0000 7f5557400640 1 -- 172.21.15.90:0/33029443 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559a2f9699f0 msgr2=0x559a2f98ebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:39:39.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.296+0000 7f5557400640 1 --2- 172.21.15.90:0/33029443 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559a2f9699f0 0x559a2f98ebf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:39.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.296+0000 7f5557400640 1 -- 172.21.15.90:0/33029443 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559a2f964760 con 0x559a2f968d10 2024-04-02T14:39:39.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.296+0000 7f5557400640 1 --2- 172.21.15.90:0/33029443 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559a2f968d10 0x559a2f982770 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f554809cc10 tx=0x7f5548017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=90453a6f20ec109d server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:39:39.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.296+0000 7f5543400640 1 -- 172.21.15.90:0/33029443 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f554809cc80 con 0x559a2f968d10 2024-04-02T14:39:39.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.296+0000 7f5543400640 1 -- 172.21.15.90:0/33029443 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f55480d3070 con 0x559a2f968d10 2024-04-02T14:39:39.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.297+0000 7f5543400640 1 -- 172.21.15.90:0/33029443 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55480a5460 con 0x559a2f968d10 2024-04-02T14:39:39.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.298+0000 7f5560e5a740 1 -- 172.21.15.90:0/33029443 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559a2f968d10 msgr2=0x559a2f982770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:39.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.298+0000 7f5560e5a740 1 --2- 172.21.15.90:0/33029443 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559a2f968d10 0x559a2f982770 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f554809cc10 tx=0x7f5548017000 comp rx=0 tx=0).stop 2024-04-02T14:39:39.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.299+0000 7f5560e5a740 1 -- 172.21.15.90:0/33029443 shutdown_connections 2024-04-02T14:39:39.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.299+0000 7f5560e5a740 1 --2- 172.21.15.90:0/33029443 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559a2f983090 0x559a2f97f1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:39.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.299+0000 7f5560e5a740 1 --2- 172.21.15.90:0/33029443 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559a2f9699f0 0x559a2f98ebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:39.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.299+0000 7f5560e5a740 1 --2- 172.21.15.90:0/33029443 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559a2f968d10 0x559a2f982770 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:39.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.300+0000 7f5560e5a740 1 -- 172.21.15.90:0/33029443 >> 172.21.15.90:0/33029443 conn(0x559a2f92fa50 msgr2=0x559a2f964b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:39:39.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.300+0000 7f5560e5a740 1 -- 172.21.15.90:0/33029443 shutdown_connections 2024-04-02T14:39:39.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.300+0000 7f5560e5a740 1 -- 172.21.15.90:0/33029443 wait complete. 2024-04-02T14:39:39.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.301+0000 7f5560e5a740 1 -- start start 2024-04-02T14:39:39.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.301+0000 7f5560e5a740 1 Processor -- start 2024-04-02T14:39:39.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.302+0000 7f5560e5a740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559a2f968d10 0x559a2f98b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:39.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.302+0000 7f5560e5a740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559a2f9699f0 0x559a2f97d600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:39.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.302+0000 7f5560e5a740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559a2f983090 0x559a2f97dc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:39.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.302+0000 7f5560e5a740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f5558070a40 con 0x559a2f983090 2024-04-02T14:39:39.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.302+0000 7f5560e5a740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f5558070370 con 0x559a2f968d10 2024-04-02T14:39:39.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.302+0000 7f5560e5a740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f555806d5b0 con 0x559a2f9699f0 2024-04-02T14:39:39.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.302+0000 7f5560e5a740 10 client.0 _ll_register_callbacks cb 0x7f5548004ae0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:39:39.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.303+0000 7f5557400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559a2f968d10 0x559a2f98b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:39.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.303+0000 7f5557400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559a2f968d10 0x559a2f98b5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:48294/0 (socket says 172.21.15.90:48294) 2024-04-02T14:39:39.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.303+0000 7f5557400640 1 -- 172.21.15.90:0/781435517 learned_addr learned my addr 172.21.15.90:0/781435517 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:39:39.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.303+0000 7f5557400640 1 -- 172.21.15.90:0/781435517 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559a2f9699f0 msgr2=0x559a2f97d600 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-02T14:39:39.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.303+0000 7f5557400640 1 --2- 172.21.15.90:0/781435517 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559a2f9699f0 0x559a2f97d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:39.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.303+0000 7f5557400640 1 -- 172.21.15.90:0/781435517 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559a2f983090 msgr2=0x559a2f97dc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:39:39.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.303+0000 7f5557400640 1 --2- 172.21.15.90:0/781435517 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559a2f983090 0x559a2f97dc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:39.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.303+0000 7f5557400640 1 -- 172.21.15.90:0/781435517 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559a2f984020 con 0x559a2f968d10 2024-04-02T14:39:39.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.304+0000 7f5540c00640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:39:39.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.304+0000 7f5557400640 1 --2- 172.21.15.90:0/781435517 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559a2f968d10 0x559a2f98b5d0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f55480cef50 tx=0x7f554809c8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ee21090c3d3187dd server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:39:39.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.305+0000 7f5540c00640 1 -- 172.21.15.90:0/781435517 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f554809cc80 con 0x559a2f968d10 2024-04-02T14:39:39.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.305+0000 7f5540c00640 1 -- 172.21.15.90:0/781435517 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f55480a5670 con 0x559a2f968d10 2024-04-02T14:39:39.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.305+0000 7f5560e5a740 1 -- 172.21.15.90:0/781435517 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f55480d3070 con 0x559a2f968d10 2024-04-02T14:39:39.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.305+0000 7f5540c00640 1 -- 172.21.15.90:0/781435517 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f554809c5e0 con 0x559a2f968d10 2024-04-02T14:39:39.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.306+0000 7f5560e5a740 10 client.26864 fetch_fsmap learned FSMap version 27 2024-04-02T14:39:39.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.306+0000 7f5560e5a740 1 -- 172.21.15.90:0/781435517 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x559a2f98f690 con 0x559a2f968d10 2024-04-02T14:39:39.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.306+0000 7f5540c00640 1 -- 172.21.15.90:0/781435517 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f55480d3070 con 0x559a2f968d10 2024-04-02T14:39:39.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.306+0000 7f5540c00640 1 -- 172.21.15.90:0/781435517 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f554809bb40 con 0x559a2f968d10 2024-04-02T14:39:39.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.306+0000 7f5560e5a740 10 client.26864 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:39:39.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.306+0000 7f5560e5a740 10 client.26864 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-04-02T14:39:39.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.306+0000 7f5560e5a740 20 client.26864 populate_metadata read hostname 'smithi090' 2024-04-02T14:39:39.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.306+0000 7f5560e5a740 1 --2- 172.21.15.90:0/781435517 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x559a2f98fb00 0x559a2f991f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:39.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.306+0000 7f5560e5a740 4 client.26864 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-02T14:39:39.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.306+0000 7f5560e5a740 1 -- 172.21.15.90:0/781435517 --> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] -- command(tid 0: {"prefix": "damage ls"}) -- 0x559a2f8dac30 con 0x559a2f98fb00 2024-04-02T14:39:39.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.307+0000 7f5556a00640 1 --2- 172.21.15.90:0/781435517 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x559a2f98fb00 0x559a2f991f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:39.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.307+0000 7f5540c00640 10 client.26864 ms_handle_connect on v2:172.21.15.92:6836/1270635939 2024-04-02T14:39:39.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.307+0000 7f5556a00640 1 --2- 172.21.15.90:0/781435517 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x559a2f98fb00 0x559a2f991f20 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=406c8be4aaf7a63b server_cookie=e6337fa2e536cabc in_seq=0 out_seq=0 2024-04-02T14:39:39.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.307+0000 7f5540c00640 1 -- 172.21.15.90:0/781435517 <== mds.3 v2:172.21.15.92:6836/1270635939 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x559a2f8dac30 con 0x559a2f98fb00 2024-04-02T14:39:39.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.307+0000 7f5540c00640 10 client.26864 handle_command_reply: tid=0 2024-04-02T14:39:39.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.307+0000 7f5560e5a740 1 client.26864 shutdown 2024-04-02T14:39:39.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.307+0000 7f5560e5a740 1 -- 172.21.15.90:0/781435517 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559a2f968d10 msgr2=0x559a2f98b5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:39.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.307+0000 7f5560e5a740 1 --2- 172.21.15.90:0/781435517 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559a2f968d10 0x559a2f98b5d0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f55480cef50 tx=0x7f554809c8f0 comp rx=0 tx=0).stop 2024-04-02T14:39:39.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.308+0000 7f5560e5a740 1 -- 172.21.15.90:0/781435517 shutdown_connections 2024-04-02T14:39:39.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.308+0000 7f5560e5a740 1 --2- 172.21.15.90:0/781435517 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x559a2f98fb00 0x559a2f991f20 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:39.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.308+0000 7f5560e5a740 1 --2- 172.21.15.90:0/781435517 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559a2f983090 0x559a2f97dc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:39.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.308+0000 7f5560e5a740 1 --2- 172.21.15.90:0/781435517 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559a2f9699f0 0x559a2f97d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:39.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.308+0000 7f5560e5a740 1 --2- 172.21.15.90:0/781435517 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559a2f968d10 0x559a2f98b5d0 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:39.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.308+0000 7f5540c00640 0 client.26864 ms_handle_reset on v2:172.21.15.92:6836/1270635939 2024-04-02T14:39:39.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.308+0000 7f5560e5a740 1 -- 172.21.15.90:0/781435517 >> 172.21.15.90:0/781435517 conn(0x559a2f92fa50 msgr2=0x559a2f97ffe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:39:39.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.308+0000 7f5560e5a740 1 -- 172.21.15.90:0/781435517 shutdown_connections 2024-04-02T14:39:39.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.308+0000 7f5560e5a740 1 -- 172.21.15.90:0/781435517 wait complete. 2024-04-02T14:39:39.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.308+0000 7f5560e5a740 20 client.26864 trim_cache size 0 max 16384 2024-04-02T14:39:39.312 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-02T14:39:39.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.309+0000 7f555e200640 1 -- 172.21.15.90:0/663531451 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f553c0a3e20 msgr2=0x7f553c0a62e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:39.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.309+0000 7f555e200640 1 --2- 172.21.15.90:0/663531451 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f553c0a3e20 0x7f553c0a62e0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f5550018000 tx=0x7f555001a000 comp rx=0 tx=0).stop 2024-04-02T14:39:39.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.309+0000 7f555e200640 1 -- 172.21.15.90:0/663531451 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5558103f10 msgr2=0x7f555819a620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:39.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.309+0000 7f555e200640 1 --2- 172.21.15.90:0/663531451 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5558103f10 0x7f555819a620 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f554800e9f0 tx=0x7f554800edf0 comp rx=0 tx=0).stop 2024-04-02T14:39:39.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.309+0000 7f555e200640 1 -- 172.21.15.90:0/663531451 shutdown_connections 2024-04-02T14:39:39.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.309+0000 7f555e200640 1 --2- 172.21.15.90:0/663531451 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f553c0a3e20 0x7f553c0a62e0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:39.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.309+0000 7f555e200640 1 --2- 172.21.15.90:0/663531451 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f555819eee0 0x7f555819f390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:39.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.309+0000 7f555e200640 1 --2- 172.21.15.90:0/663531451 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5558108fe0 0x7f555819ab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:39.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.309+0000 7f555e200640 1 --2- 172.21.15.90:0/663531451 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5558103f10 0x7f555819a620 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:39.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.309+0000 7f555e200640 1 -- 172.21.15.90:0/663531451 >> 172.21.15.90:0/663531451 conn(0x7f55580ff7b0 msgr2=0x7f5558100af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:39:39.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.310+0000 7f555e200640 1 -- 172.21.15.90:0/663531451 shutdown_connections 2024-04-02T14:39:39.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.310+0000 7f555e200640 1 -- 172.21.15.90:0/663531451 wait complete. 2024-04-02T14:39:39.331 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:39:39 smithi090 ceph-mon[31674]: from='client.? 172.21.15.90:0/3202212461' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-02T14:39:39.335 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-02T14:39:39.360 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:39:39 smithi092 ceph-mon[36497]: from='client.? 172.21.15.90:0/3202212461' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-02T14:39:39.436 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:39:39 smithi177 ceph-mon[36431]: from='client.? 172.21.15.90:0/3202212461' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-02T14:39:39.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.589+0000 7f3b4ae00640 1 -- 172.21.15.90:0/839188466 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3b44107a80 msgr2=0x7f3b44107e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:39.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.589+0000 7f3b4ae00640 1 --2- 172.21.15.90:0/839188466 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3b44107a80 0x7f3b44107e60 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f3b30009930 tx=0x7f3b3002bc30 comp rx=0 tx=0).stop 2024-04-02T14:39:39.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.589+0000 7f3b4ae00640 1 -- 172.21.15.90:0/839188466 shutdown_connections 2024-04-02T14:39:39.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.589+0000 7f3b4ae00640 1 --2- 172.21.15.90:0/839188466 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3b44102430 0x7f3b4410e940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:39.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.589+0000 7f3b4ae00640 1 --2- 172.21.15.90:0/839188466 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3b44101a70 0x7f3b44101ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:39.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.589+0000 7f3b4ae00640 1 --2- 172.21.15.90:0/839188466 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3b44107a80 0x7f3b44107e60 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:39.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.589+0000 7f3b4ae00640 1 -- 172.21.15.90:0/839188466 >> 172.21.15.90:0/839188466 conn(0x7f3b440fd560 msgr2=0x7f3b440ff980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:39:39.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.590+0000 7f3b4ae00640 1 -- 172.21.15.90:0/839188466 shutdown_connections 2024-04-02T14:39:39.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.590+0000 7f3b4ae00640 1 -- 172.21.15.90:0/839188466 wait complete. 2024-04-02T14:39:39.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.591+0000 7f3b4ae00640 1 Processor -- start 2024-04-02T14:39:39.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.592+0000 7f3b4ae00640 1 -- start start 2024-04-02T14:39:39.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.593+0000 7f3b4ae00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3b44101a70 0x7f3b4419a5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:39.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.593+0000 7f3b4ae00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3b44102430 0x7f3b4419ab00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:39.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.593+0000 7f3b4ae00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3b44107a80 0x7f3b4419ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:39.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.593+0000 7f3b4ae00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f3b44111eb0 con 0x7f3b44101a70 2024-04-02T14:39:39.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.593+0000 7f3b4ae00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f3b44111d30 con 0x7f3b44107a80 2024-04-02T14:39:39.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.593+0000 7f3b4ae00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f3b44112030 con 0x7f3b44102430 2024-04-02T14:39:39.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.593+0000 7f3b48e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3b44107a80 0x7f3b4419ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:39.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.593+0000 7f3b48e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3b44107a80 0x7f3b4419ee80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:48322/0 (socket says 172.21.15.90:48322) 2024-04-02T14:39:39.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.593+0000 7f3b48e00640 1 -- 172.21.15.90:0/2077625413 learned_addr learned my addr 172.21.15.90:0/2077625413 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:39:39.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.593+0000 7f3b43e00640 1 --2- 172.21.15.90:0/2077625413 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3b44101a70 0x7f3b4419a5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:39.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.593+0000 7f3b43400640 1 --2- 172.21.15.90:0/2077625413 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3b44102430 0x7f3b4419ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:39.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.593+0000 7f3b48e00640 1 -- 172.21.15.90:0/2077625413 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3b44102430 msgr2=0x7f3b4419ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:39.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.593+0000 7f3b48e00640 1 --2- 172.21.15.90:0/2077625413 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3b44102430 0x7f3b4419ab00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:39.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.593+0000 7f3b48e00640 1 -- 172.21.15.90:0/2077625413 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3b44101a70 msgr2=0x7f3b4419a5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:39.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.593+0000 7f3b48e00640 1 --2- 172.21.15.90:0/2077625413 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3b44101a70 0x7f3b4419a5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:39.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.593+0000 7f3b48e00640 1 -- 172.21.15.90:0/2077625413 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b4419f5b0 con 0x7f3b44107a80 2024-04-02T14:39:39.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.593+0000 7f3b43400640 1 --2- 172.21.15.90:0/2077625413 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3b44102430 0x7f3b4419ab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:39:39.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.594+0000 7f3b48e00640 1 --2- 172.21.15.90:0/2077625413 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3b44107a80 0x7f3b4419ee80 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f3b3c009fd0 tx=0x7f3b3c00b260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:39:39.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.594+0000 7f3b40c00640 1 -- 172.21.15.90:0/2077625413 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b3c017070 con 0x7f3b44107a80 2024-04-02T14:39:39.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.594+0000 7f3b4ae00640 1 -- 172.21.15.90:0/2077625413 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b4419f8a0 con 0x7f3b44107a80 2024-04-02T14:39:39.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.594+0000 7f3b40c00640 1 -- 172.21.15.90:0/2077625413 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3b3c003e50 con 0x7f3b44107a80 2024-04-02T14:39:39.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.594+0000 7f3b40c00640 1 -- 172.21.15.90:0/2077625413 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b3c0106c0 con 0x7f3b44107a80 2024-04-02T14:39:39.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.594+0000 7f3b4ae00640 1 -- 172.21.15.90:0/2077625413 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b441a7130 con 0x7f3b44107a80 2024-04-02T14:39:39.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.599+0000 7f3b40c00640 1 -- 172.21.15.90:0/2077625413 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f3b3c010860 con 0x7f3b44107a80 2024-04-02T14:39:39.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.599+0000 7f3b4da64740 1 Processor -- start 2024-04-02T14:39:39.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.600+0000 7f3b4da64740 1 -- start start 2024-04-02T14:39:39.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.600+0000 7f3b4da64740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5628ddd989f0 0x5628ddd97740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:39.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.600+0000 7f3b40c00640 1 --2- 172.21.15.90:0/2077625413 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f3b280a3da0 0x7f3b280a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:39.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.600+0000 7f3b4da64740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5628ddd97d10 0x5628ddd98190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:39.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.600+0000 7f3b4da64740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5628ddda9890 0x5628dddabc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:39.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.600+0000 7f3b4da64740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5628ddcd8400 con 0x5628ddd97d10 2024-04-02T14:39:39.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.600+0000 7f3b4da64740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x5628ddcbcf10 con 0x5628ddda9890 2024-04-02T14:39:39.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.600+0000 7f3b4da64740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5628ddcfc6f0 con 0x5628ddd989f0 2024-04-02T14:39:39.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.600+0000 7f3b40c00640 1 -- 172.21.15.90:0/2077625413 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f3b3c0d0a30 con 0x7f3b44107a80 2024-04-02T14:39:39.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.601+0000 7f3b48e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5628ddda9890 0x5628dddabc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:39.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.601+0000 7f3b48e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5628ddda9890 0x5628dddabc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:48334/0 (socket says 172.21.15.90:48334) 2024-04-02T14:39:39.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.601+0000 7f3b48e00640 1 -- 172.21.15.90:0/1900792029 learned_addr learned my addr 172.21.15.90:0/1900792029 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:39:39.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.601+0000 7f3b48e00640 1 -- 172.21.15.90:0/1900792029 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5628ddd989f0 msgr2=0x5628ddd97740 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-02T14:39:39.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.601+0000 7f3b48e00640 1 --2- 172.21.15.90:0/1900792029 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5628ddd989f0 0x5628ddd97740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:39.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.601+0000 7f3b48e00640 1 -- 172.21.15.90:0/1900792029 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5628ddd97d10 msgr2=0x5628ddd98190 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-02T14:39:39.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.601+0000 7f3b48e00640 1 --2- 172.21.15.90:0/1900792029 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5628ddd97d10 0x5628ddd98190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:39.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.601+0000 7f3b48e00640 1 -- 172.21.15.90:0/1900792029 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5628dda103f0 con 0x5628ddda9890 2024-04-02T14:39:39.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.602+0000 7f3b48e00640 1 --2- 172.21.15.90:0/1900792029 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5628ddda9890 0x5628dddabc80 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f3b3c099c70 tx=0x7f3b3c015000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ff23935ff4b69198 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:39:39.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.602+0000 7f3b43400640 1 --2- 172.21.15.90:0/2077625413 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f3b280a3da0 0x7f3b280a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:39.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.602+0000 7f3b39600640 1 -- 172.21.15.90:0/1900792029 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b3c020040 con 0x5628ddda9890 2024-04-02T14:39:39.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.602+0000 7f3b39600640 1 -- 172.21.15.90:0/1900792029 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3b3c09df10 con 0x5628ddda9890 2024-04-02T14:39:39.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.602+0000 7f3b39600640 1 -- 172.21.15.90:0/1900792029 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b3c09bbe0 con 0x5628ddda9890 2024-04-02T14:39:39.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.604+0000 7f3b4da64740 1 -- 172.21.15.90:0/1900792029 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5628ddda9890 msgr2=0x5628dddabc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:39.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.604+0000 7f3b4da64740 1 --2- 172.21.15.90:0/1900792029 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5628ddda9890 0x5628dddabc80 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f3b3c099c70 tx=0x7f3b3c015000 comp rx=0 tx=0).stop 2024-04-02T14:39:39.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.604+0000 7f3b43400640 1 --2- 172.21.15.90:0/2077625413 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f3b280a3da0 0x7f3b280a6260 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x5628ddd98f40 tx=0x7f3b34005f90 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:39:39.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.604+0000 7f3b4da64740 1 -- 172.21.15.90:0/1900792029 shutdown_connections 2024-04-02T14:39:39.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.604+0000 7f3b4da64740 1 --2- 172.21.15.90:0/1900792029 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5628ddda9890 0x5628dddabc80 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:39.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.604+0000 7f3b4da64740 1 --2- 172.21.15.90:0/1900792029 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5628ddd97d10 0x5628ddd98190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:39.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.604+0000 7f3b4da64740 1 --2- 172.21.15.90:0/1900792029 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5628ddd989f0 0x5628ddd97740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:39.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.604+0000 7f3b4da64740 1 -- 172.21.15.90:0/1900792029 >> 172.21.15.90:0/1900792029 conn(0x5628ddd5ea50 msgr2=0x5628ddd92760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:39:39.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.605+0000 7f3b4da64740 1 -- 172.21.15.90:0/1900792029 shutdown_connections 2024-04-02T14:39:39.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.605+0000 7f3b4da64740 1 -- 172.21.15.90:0/1900792029 wait complete. 2024-04-02T14:39:39.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.605+0000 7f3b4da64740 1 -- start start 2024-04-02T14:39:39.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.606+0000 7f3b4da64740 1 Processor -- start 2024-04-02T14:39:39.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.606+0000 7f3b4da64740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5628ddd97d10 0x5628ddda68d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:39.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.606+0000 7f3b4da64740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5628ddd989f0 0x5628ddda6e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:39.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.606+0000 7f3b4da64740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5628ddda9890 0x5628ddda7350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:39.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.606+0000 7f3b4da64740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5628ddcf86a0 con 0x5628ddd97d10 2024-04-02T14:39:39.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.606+0000 7f3b4da64740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x5628ddca4980 con 0x5628ddd989f0 2024-04-02T14:39:39.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.606+0000 7f3b4da64740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5628ddcb7cb0 con 0x5628ddda9890 2024-04-02T14:39:39.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.606+0000 7f3b4da64740 10 client.0 _ll_register_callbacks cb 0x5628ddcb76a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:39:39.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.607+0000 7f3b43400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5628ddd989f0 0x5628ddda6e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:39.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.607+0000 7f3b43400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5628ddd989f0 0x5628ddda6e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:48346/0 (socket says 172.21.15.90:48346) 2024-04-02T14:39:39.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.607+0000 7f3b43400640 1 -- 172.21.15.90:0/2596433765 learned_addr learned my addr 172.21.15.90:0/2596433765 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:39:39.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.607+0000 7f3b43e00640 1 --2- 172.21.15.90:0/2596433765 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5628ddd97d10 0x5628ddda68d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:39.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.607+0000 7f3b48e00640 1 --2- 172.21.15.90:0/2596433765 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5628ddda9890 0x5628ddda7350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:39.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.607+0000 7f3b43e00640 1 -- 172.21.15.90:0/2596433765 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5628ddda9890 msgr2=0x5628ddda7350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:39:39.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.607+0000 7f3b43e00640 1 --2- 172.21.15.90:0/2596433765 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5628ddda9890 0x5628ddda7350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:39.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.607+0000 7f3b43e00640 1 -- 172.21.15.90:0/2596433765 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5628ddd989f0 msgr2=0x5628ddda6e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:39:39.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.607+0000 7f3b43e00640 1 --2- 172.21.15.90:0/2596433765 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5628ddd989f0 0x5628ddda6e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:39.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.607+0000 7f3b43e00640 1 -- 172.21.15.90:0/2596433765 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5628ddada7d0 con 0x5628ddd97d10 2024-04-02T14:39:39.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.608+0000 7f3b48e00640 1 --2- 172.21.15.90:0/2596433765 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5628ddda9890 0x5628ddda7350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:39:39.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.608+0000 7f3b43e00640 1 --2- 172.21.15.90:0/2596433765 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5628ddd97d10 0x5628ddda68d0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f3b30036040 tx=0x7f3b300045d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f5a4c77253d5dfc server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:39:39.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.608+0000 7f3b2e000640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-02T14:39:39.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.608+0000 7f3b2e000640 1 -- 172.21.15.90:0/2596433765 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b30044020 con 0x5628ddd97d10 2024-04-02T14:39:39.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.608+0000 7f3b2e000640 1 -- 172.21.15.90:0/2596433765 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f3b3002cc90 con 0x5628ddd97d10 2024-04-02T14:39:39.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.609+0000 7f3b4da64740 1 -- 172.21.15.90:0/2596433765 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3b3c09df10 con 0x5628ddd97d10 2024-04-02T14:39:39.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.609+0000 7f3b2e000640 1 -- 172.21.15.90:0/2596433765 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f3b3004f050 con 0x5628ddd97d10 2024-04-02T14:39:39.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.609+0000 7f3b4da64740 10 client.16332 fetch_fsmap learned FSMap version 27 2024-04-02T14:39:39.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.609+0000 7f3b4da64740 1 -- 172.21.15.90:0/2596433765 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5628dddc9780 con 0x5628ddd97d10 2024-04-02T14:39:39.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.610+0000 7f3b2e000640 1 -- 172.21.15.90:0/2596433765 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3b3c09df10 con 0x5628ddd97d10 2024-04-02T14:39:39.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.610+0000 7f3b2e000640 1 -- 172.21.15.90:0/2596433765 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f3b30004ca0 con 0x5628ddd97d10 2024-04-02T14:39:39.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.610+0000 7f3b4da64740 10 client.16332 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:39:39.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.610+0000 7f3b4da64740 10 client.16332 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-04-02T14:39:39.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.610+0000 7f3b4da64740 20 client.16332 populate_metadata read hostname 'smithi090' 2024-04-02T14:39:39.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.610+0000 7f3b4da64740 1 --2- 172.21.15.90:0/2596433765 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x5628dddcac90 0x5628dddcd0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:39.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.610+0000 7f3b4da64740 4 client.16332 mds_command: new command op to 24547 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:39:39.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.610+0000 7f3b4da64740 1 -- 172.21.15.90:0/2596433765 --> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5628ddcbc040 con 0x5628dddcac90 2024-04-02T14:39:39.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.611+0000 7f3b43400640 1 --2- 172.21.15.90:0/2596433765 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x5628dddcac90 0x5628dddcd0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:39.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.612+0000 7f3b43400640 1 --2- 172.21.15.90:0/2596433765 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x5628dddcac90 0x5628dddcd0e0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=d52cde0e949d06d8 server_cookie=e8c089634df9207f in_seq=0 out_seq=0 2024-04-02T14:39:39.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.612+0000 7f3b2e000640 10 client.16332 ms_handle_connect on v2:172.21.15.177:6838/664895663 2024-04-02T14:39:39.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.614+0000 7f3b2e000640 1 -- 172.21.15.90:0/2596433765 <== mds.2 v2:172.21.15.177:6838/664895663 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5628ddcbc040 con 0x5628dddcac90 2024-04-02T14:39:39.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.614+0000 7f3b2e000640 10 client.16332 handle_command_reply: tid=0 2024-04-02T14:39:39.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.614+0000 7f3b4da64740 1 client.16332 shutdown 2024-04-02T14:39:39.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.614+0000 7f3b4da64740 1 -- 172.21.15.90:0/2596433765 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5628ddd97d10 msgr2=0x5628ddda68d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:39.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.614+0000 7f3b4da64740 1 --2- 172.21.15.90:0/2596433765 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5628ddd97d10 0x5628ddda68d0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f3b30036040 tx=0x7f3b300045d0 comp rx=0 tx=0).stop 2024-04-02T14:39:39.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.614+0000 7f3b4da64740 1 -- 172.21.15.90:0/2596433765 shutdown_connections 2024-04-02T14:39:39.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.614+0000 7f3b4da64740 1 --2- 172.21.15.90:0/2596433765 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x5628dddcac90 0x5628dddcd0e0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:39.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.614+0000 7f3b4da64740 1 --2- 172.21.15.90:0/2596433765 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5628ddda9890 0x5628ddda7350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:39.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.614+0000 7f3b4da64740 1 --2- 172.21.15.90:0/2596433765 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5628ddd989f0 0x5628ddda6e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:39.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.614+0000 7f3b4da64740 1 --2- 172.21.15.90:0/2596433765 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5628ddd97d10 0x5628ddda68d0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:39.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.614+0000 7f3b2e000640 0 client.16332 ms_handle_reset on v2:172.21.15.177:6838/664895663 2024-04-02T14:39:39.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.614+0000 7f3b4da64740 1 -- 172.21.15.90:0/2596433765 >> 172.21.15.90:0/2596433765 conn(0x5628ddd5ea50 msgr2=0x5628ddd944b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:39:39.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.615+0000 7f3b4da64740 1 -- 172.21.15.90:0/2596433765 shutdown_connections 2024-04-02T14:39:39.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.615+0000 7f3b4da64740 1 -- 172.21.15.90:0/2596433765 wait complete. 2024-04-02T14:39:39.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.616+0000 7f3b4da64740 20 client.16332 trim_cache size 0 max 16384 2024-04-02T14:39:39.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.640+0000 7f3b4da64740 1 Processor -- start 2024-04-02T14:39:39.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.640+0000 7f3b4da64740 1 -- start start 2024-04-02T14:39:39.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.641+0000 7f3b4da64740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5628ddd97d10 0x5628ddda9000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:39.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.641+0000 7f3b4da64740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5628ddd989f0 0x5628dddb5300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:39.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.641+0000 7f3b4da64740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5628ddda9890 0x5628ddda5a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:39.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.641+0000 7f3b4da64740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f3b4406f8e0 con 0x5628ddd989f0 2024-04-02T14:39:39.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.641+0000 7f3b4da64740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f3b44071420 con 0x5628ddd97d10 2024-04-02T14:39:39.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.641+0000 7f3b48e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5628ddda9890 0x5628ddda5a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:39.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.641+0000 7f3b48e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5628ddda9890 0x5628ddda5a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:44570/0 (socket says 172.21.15.90:44570) 2024-04-02T14:39:39.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.641+0000 7f3b48e00640 1 -- 172.21.15.90:0/1805152184 learned_addr learned my addr 172.21.15.90:0/1805152184 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:39:39.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.641+0000 7f3b4da64740 1 -- 172.21.15.90:0/1805152184 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f3b4406ffb0 con 0x5628ddda9890 2024-04-02T14:39:39.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.641+0000 7f3b43e00640 1 --2- 172.21.15.90:0/1805152184 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5628ddd97d10 0x5628ddda9000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:39.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.642+0000 7f3b43400640 1 --2- 172.21.15.90:0/1805152184 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5628ddd989f0 0x5628dddb5300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:39.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.642+0000 7f3b48e00640 1 -- 172.21.15.90:0/1805152184 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5628ddd97d10 msgr2=0x5628ddda9000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:39:39.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.642+0000 7f3b48e00640 1 --2- 172.21.15.90:0/1805152184 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5628ddd97d10 0x5628ddda9000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:39.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.642+0000 7f3b48e00640 1 -- 172.21.15.90:0/1805152184 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5628ddd989f0 msgr2=0x5628dddb5300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:39:39.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.642+0000 7f3b48e00640 1 --2- 172.21.15.90:0/1805152184 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5628ddd989f0 0x5628dddb5300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:39.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.642+0000 7f3b48e00640 1 -- 172.21.15.90:0/1805152184 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5628ddd96d40 con 0x5628ddda9890 2024-04-02T14:39:39.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.642+0000 7f3b43400640 1 --2- 172.21.15.90:0/1805152184 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5628ddd989f0 0x5628dddb5300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:39:39.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.643+0000 7f3b48e00640 1 --2- 172.21.15.90:0/1805152184 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5628ddda9890 0x5628ddda5a30 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f3b3c005c40 tx=0x7f3b3c00bf10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3e0858d6b3e07614 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:39:39.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.643+0000 7f3b38c00640 1 -- 172.21.15.90:0/1805152184 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b3c020040 con 0x5628ddda9890 2024-04-02T14:39:39.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.643+0000 7f3b38c00640 1 -- 172.21.15.90:0/1805152184 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3b3c09df10 con 0x5628ddda9890 2024-04-02T14:39:39.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.643+0000 7f3b38c00640 1 -- 172.21.15.90:0/1805152184 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b3c0ce410 con 0x5628ddda9890 2024-04-02T14:39:39.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.644+0000 7f3b4da64740 1 -- 172.21.15.90:0/1805152184 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5628ddda9890 msgr2=0x5628ddda5a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:39.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.644+0000 7f3b4da64740 1 --2- 172.21.15.90:0/1805152184 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5628ddda9890 0x5628ddda5a30 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f3b3c005c40 tx=0x7f3b3c00bf10 comp rx=0 tx=0).stop 2024-04-02T14:39:39.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.644+0000 7f3b4da64740 1 -- 172.21.15.90:0/1805152184 shutdown_connections 2024-04-02T14:39:39.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.644+0000 7f3b4da64740 1 --2- 172.21.15.90:0/1805152184 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5628ddda9890 0x5628ddda5a30 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:39.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.644+0000 7f3b4da64740 1 --2- 172.21.15.90:0/1805152184 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5628ddd989f0 0x5628dddb5300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:39.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.644+0000 7f3b4da64740 1 --2- 172.21.15.90:0/1805152184 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5628ddd97d10 0x5628ddda9000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:39.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.644+0000 7f3b4da64740 1 -- 172.21.15.90:0/1805152184 >> 172.21.15.90:0/1805152184 conn(0x5628ddd5ea50 msgr2=0x5628ddd93b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:39:39.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.644+0000 7f3b4da64740 1 -- 172.21.15.90:0/1805152184 shutdown_connections 2024-04-02T14:39:39.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.644+0000 7f3b4da64740 1 -- 172.21.15.90:0/1805152184 wait complete. 2024-04-02T14:39:39.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.644+0000 7f3b4da64740 1 -- start start 2024-04-02T14:39:39.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.645+0000 7f3b4da64740 1 Processor -- start 2024-04-02T14:39:39.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.645+0000 7f3b4da64740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5628ddd97d10 0x5628ddda40c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:39.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.645+0000 7f3b4da64740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5628ddd989f0 0x5628ddda4600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:39.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.645+0000 7f3b4da64740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5628ddda9890 0x5628dde2e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:39.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.645+0000 7f3b4da64740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f3b44070d50 con 0x5628ddda9890 2024-04-02T14:39:39.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.645+0000 7f3b4da64740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f3b44070680 con 0x5628ddd989f0 2024-04-02T14:39:39.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.645+0000 7f3b4da64740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f3b4406d8e0 con 0x5628ddd97d10 2024-04-02T14:39:39.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.645+0000 7f3b4da64740 10 client.0 _ll_register_callbacks cb 0x7f3b30004490 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:39:39.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.646+0000 7f3b43400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5628ddd989f0 0x5628ddda4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:39.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.646+0000 7f3b48e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5628ddda9890 0x5628dde2e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:39.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.646+0000 7f3b43e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5628ddd97d10 0x5628ddda40c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:39.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.646+0000 7f3b43400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5628ddd989f0 0x5628ddda4600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:48354/0 (socket says 172.21.15.90:48354) 2024-04-02T14:39:39.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.646+0000 7f3b43400640 1 -- 172.21.15.90:0/672485900 learned_addr learned my addr 172.21.15.90:0/672485900 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:39:39.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.647+0000 7f3b43400640 1 -- 172.21.15.90:0/672485900 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5628ddd97d10 msgr2=0x5628ddda40c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:39:39.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.647+0000 7f3b43400640 1 --2- 172.21.15.90:0/672485900 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5628ddd97d10 0x5628ddda40c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:39.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.647+0000 7f3b43400640 1 -- 172.21.15.90:0/672485900 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5628ddda9890 msgr2=0x5628dde2e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:39:39.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.647+0000 7f3b43400640 1 --2- 172.21.15.90:0/672485900 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5628ddda9890 0x5628dde2e3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:39.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.647+0000 7f3b43400640 1 -- 172.21.15.90:0/672485900 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5628dde2ebe0 con 0x5628ddd989f0 2024-04-02T14:39:39.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.647+0000 7f3b43400640 1 --2- 172.21.15.90:0/672485900 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5628ddd989f0 0x5628ddda4600 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f3b340098a0 tx=0x7f3b34004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5b7c1ef5f5ba398c server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:39:39.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.647+0000 7f3b2e000640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:39:39.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.648+0000 7f3b2e000640 1 -- 172.21.15.90:0/672485900 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b34012df0 con 0x5628ddd989f0 2024-04-02T14:39:39.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.648+0000 7f3b2e000640 1 -- 172.21.15.90:0/672485900 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f3b340293a0 con 0x5628ddd989f0 2024-04-02T14:39:39.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.648+0000 7f3b4da64740 1 -- 172.21.15.90:0/672485900 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3b3c09df10 con 0x5628ddd989f0 2024-04-02T14:39:39.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.648+0000 7f3b2e000640 1 -- 172.21.15.90:0/672485900 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f3b340295a0 con 0x5628ddd989f0 2024-04-02T14:39:39.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.648+0000 7f3b4da64740 10 client.26882 fetch_fsmap learned FSMap version 27 2024-04-02T14:39:39.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.648+0000 7f3b4da64740 1 -- 172.21.15.90:0/672485900 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5628dddab020 con 0x5628ddd989f0 2024-04-02T14:39:39.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.649+0000 7f3b2e000640 1 -- 172.21.15.90:0/672485900 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3b3c09df10 con 0x5628ddd989f0 2024-04-02T14:39:39.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.649+0000 7f3b2e000640 1 -- 172.21.15.90:0/672485900 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f3b3400f040 con 0x5628ddd989f0 2024-04-02T14:39:39.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.650+0000 7f3b4da64740 10 client.26882 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:39:39.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.650+0000 7f3b4da64740 10 client.26882 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-04-02T14:39:39.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.650+0000 7f3b4da64740 20 client.26882 populate_metadata read hostname 'smithi090' 2024-04-02T14:39:39.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.650+0000 7f3b4da64740 1 --2- 172.21.15.90:0/672485900 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x5628dddb0c90 0x5628dddb10a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:39.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.650+0000 7f3b4da64740 4 client.26882 mds_command: new command op to 24547 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-02T14:39:39.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.650+0000 7f3b4da64740 1 -- 172.21.15.90:0/672485900 --> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5628ddd09c30 con 0x5628dddb0c90 2024-04-02T14:39:39.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.650+0000 7f3b43e00640 1 --2- 172.21.15.90:0/672485900 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x5628dddb0c90 0x5628dddb10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:39.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.650+0000 7f3b43e00640 1 --2- 172.21.15.90:0/672485900 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x5628dddb0c90 0x5628dddb10a0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=76d00fdd5044b0a5 server_cookie=1189435e4f27abd in_seq=0 out_seq=0 2024-04-02T14:39:39.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.650+0000 7f3b2e000640 10 client.26882 ms_handle_connect on v2:172.21.15.177:6838/664895663 2024-04-02T14:39:39.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.651+0000 7f3b2e000640 1 -- 172.21.15.90:0/672485900 <== mds.2 v2:172.21.15.177:6838/664895663 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5628ddd09c30 con 0x5628dddb0c90 2024-04-02T14:39:39.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.651+0000 7f3b2e000640 10 client.26882 handle_command_reply: tid=0 2024-04-02T14:39:39.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.651+0000 7f3b4da64740 1 client.26882 shutdown 2024-04-02T14:39:39.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.651+0000 7f3b4da64740 1 -- 172.21.15.90:0/672485900 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5628ddd989f0 msgr2=0x5628ddda4600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:39.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.651+0000 7f3b4da64740 1 --2- 172.21.15.90:0/672485900 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5628ddd989f0 0x5628ddda4600 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f3b340098a0 tx=0x7f3b34004730 comp rx=0 tx=0).stop 2024-04-02T14:39:39.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.651+0000 7f3b4da64740 1 -- 172.21.15.90:0/672485900 shutdown_connections 2024-04-02T14:39:39.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.651+0000 7f3b4da64740 1 --2- 172.21.15.90:0/672485900 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x5628dddb0c90 0x5628dddb10a0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:39.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.652+0000 7f3b4da64740 1 --2- 172.21.15.90:0/672485900 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5628ddda9890 0x5628dde2e3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:39.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.652+0000 7f3b4da64740 1 --2- 172.21.15.90:0/672485900 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5628ddd989f0 0x5628ddda4600 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:39.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.652+0000 7f3b4da64740 1 --2- 172.21.15.90:0/672485900 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5628ddd97d10 0x5628ddda40c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:39.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.652+0000 7f3b4da64740 1 -- 172.21.15.90:0/672485900 >> 172.21.15.90:0/672485900 conn(0x5628ddd5ea50 msgr2=0x5628ddd918a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:39:39.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.652+0000 7f3b2e000640 0 client.26882 ms_handle_reset on v2:172.21.15.177:6838/664895663 2024-04-02T14:39:39.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.652+0000 7f3b4da64740 1 -- 172.21.15.90:0/672485900 shutdown_connections 2024-04-02T14:39:39.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.652+0000 7f3b4da64740 1 -- 172.21.15.90:0/672485900 wait complete. 2024-04-02T14:39:39.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.652+0000 7f3b4da64740 20 client.26882 trim_cache size 0 max 16384 2024-04-02T14:39:39.657 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-02T14:39:39.657 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.653+0000 7f3b4ae00640 1 -- 172.21.15.90:0/2077625413 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f3b280a3da0 msgr2=0x7f3b280a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:39.657 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.653+0000 7f3b4ae00640 1 --2- 172.21.15.90:0/2077625413 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f3b280a3da0 0x7f3b280a6260 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x5628ddd98f40 tx=0x7f3b34005f90 comp rx=0 tx=0).stop 2024-04-02T14:39:39.658 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.653+0000 7f3b4ae00640 1 -- 172.21.15.90:0/2077625413 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3b44107a80 msgr2=0x7f3b4419ee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:39.658 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.653+0000 7f3b4ae00640 1 --2- 172.21.15.90:0/2077625413 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3b44107a80 0x7f3b4419ee80 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f3b3c009fd0 tx=0x7f3b3c00b260 comp rx=0 tx=0).stop 2024-04-02T14:39:39.658 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.654+0000 7f3b4ae00640 1 -- 172.21.15.90:0/2077625413 shutdown_connections 2024-04-02T14:39:39.658 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.654+0000 7f3b4ae00640 1 --2- 172.21.15.90:0/2077625413 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f3b280a3da0 0x7f3b280a6260 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:39.658 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.654+0000 7f3b4ae00640 1 --2- 172.21.15.90:0/2077625413 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3b44107a80 0x7f3b4419ee80 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:39.658 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.654+0000 7f3b4ae00640 1 --2- 172.21.15.90:0/2077625413 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3b44102430 0x7f3b4419ab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:39.658 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.654+0000 7f3b4ae00640 1 --2- 172.21.15.90:0/2077625413 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3b44101a70 0x7f3b4419a5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:39.658 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.654+0000 7f3b4ae00640 1 -- 172.21.15.90:0/2077625413 >> 172.21.15.90:0/2077625413 conn(0x7f3b440fd560 msgr2=0x7f3b4410cce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:39:39.658 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.654+0000 7f3b4ae00640 1 -- 172.21.15.90:0/2077625413 shutdown_connections 2024-04-02T14:39:39.658 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.654+0000 7f3b4ae00640 1 -- 172.21.15.90:0/2077625413 wait complete. 2024-04-02T14:39:39.683 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-02T14:39:39.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.955+0000 7f814c200640 1 -- 172.21.15.90:0/2673942111 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8144103f10 msgr2=0x7f81441042f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:39.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.955+0000 7f814c200640 1 --2- 172.21.15.90:0/2673942111 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8144103f10 0x7f81441042f0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f8138009900 tx=0x7f813802bc40 comp rx=0 tx=0).stop 2024-04-02T14:39:39.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.955+0000 7f814c200640 1 -- 172.21.15.90:0/2673942111 shutdown_connections 2024-04-02T14:39:39.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.955+0000 7f814c200640 1 --2- 172.21.15.90:0/2673942111 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8144108fe0 0x7f8144110bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:39.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.955+0000 7f814c200640 1 --2- 172.21.15.90:0/2673942111 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f81441048c0 0x7f8144108900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:39.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.955+0000 7f814c200640 1 --2- 172.21.15.90:0/2673942111 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8144103f10 0x7f81441042f0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:39.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.955+0000 7f814c200640 1 -- 172.21.15.90:0/2673942111 >> 172.21.15.90:0/2673942111 conn(0x7f81440ff750 msgr2=0x7f8144101b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:39:39.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.955+0000 7f814c200640 1 -- 172.21.15.90:0/2673942111 shutdown_connections 2024-04-02T14:39:39.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.955+0000 7f814c200640 1 -- 172.21.15.90:0/2673942111 wait complete. 2024-04-02T14:39:39.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.956+0000 7f814c200640 1 Processor -- start 2024-04-02T14:39:39.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.956+0000 7f814c200640 1 -- start start 2024-04-02T14:39:39.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.957+0000 7f814c200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8144103f10 0x7f814419a570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:39.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.957+0000 7f814c200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8144108fe0 0x7f814419aab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:39.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.957+0000 7f814c200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f814419ede0 0x7f814419f290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:39.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.957+0000 7f814c200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f81441140d0 con 0x7f8144103f10 2024-04-02T14:39:39.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.957+0000 7f814c200640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f8144113f50 con 0x7f8144108fe0 2024-04-02T14:39:39.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.957+0000 7f814c200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f8144114250 con 0x7f814419ede0 2024-04-02T14:39:39.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.958+0000 7f8149800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8144103f10 0x7f814419a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:39.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.958+0000 7f8148e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8144108fe0 0x7f814419aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:39.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.958+0000 7f814a200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f814419ede0 0x7f814419f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:39.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.958+0000 7f8148e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8144108fe0 0x7f814419aab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:48372/0 (socket says 172.21.15.90:48372) 2024-04-02T14:39:39.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.958+0000 7f8148e00640 1 -- 172.21.15.90:0/2771544579 learned_addr learned my addr 172.21.15.90:0/2771544579 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:39:39.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.958+0000 7f8149800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8144103f10 0x7f814419a570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:48872/0 (socket says 172.21.15.90:48872) 2024-04-02T14:39:39.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.958+0000 7f814a200640 1 -- 172.21.15.90:0/2771544579 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8144108fe0 msgr2=0x7f814419aab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:39.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.958+0000 7f814a200640 1 --2- 172.21.15.90:0/2771544579 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8144108fe0 0x7f814419aab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:39.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.958+0000 7f814a200640 1 -- 172.21.15.90:0/2771544579 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8144103f10 msgr2=0x7f814419a570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:39.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.958+0000 7f814a200640 1 --2- 172.21.15.90:0/2771544579 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8144103f10 0x7f814419a570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:39.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.958+0000 7f814a200640 1 -- 172.21.15.90:0/2771544579 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f814419f8d0 con 0x7f814419ede0 2024-04-02T14:39:39.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.959+0000 7f814a200640 1 --2- 172.21.15.90:0/2771544579 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f814419ede0 0x7f814419f290 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f813c00cab0 tx=0x7f813c00cf70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:39:39.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.959+0000 7f8142000640 1 -- 172.21.15.90:0/2771544579 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f813c00ae20 con 0x7f814419ede0 2024-04-02T14:39:39.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.959+0000 7f8142000640 1 -- 172.21.15.90:0/2771544579 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f813c00f930 con 0x7f814419ede0 2024-04-02T14:39:39.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.959+0000 7f814c200640 1 -- 172.21.15.90:0/2771544579 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f814419fbc0 con 0x7f814419ede0 2024-04-02T14:39:39.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.959+0000 7f8142000640 1 -- 172.21.15.90:0/2771544579 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f813c018b30 con 0x7f814419ede0 2024-04-02T14:39:39.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.959+0000 7f814c200640 1 -- 172.21.15.90:0/2771544579 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8144076260 con 0x7f814419ede0 2024-04-02T14:39:39.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.964+0000 7f814ef70740 1 Processor -- start 2024-04-02T14:39:39.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.966+0000 7f8142000640 1 -- 172.21.15.90:0/2771544579 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f813c00fae0 con 0x7f814419ede0 2024-04-02T14:39:39.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.966+0000 7f814ef70740 1 -- start start 2024-04-02T14:39:39.969 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.967+0000 7f814ef70740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fc3cf549f0 0x55fc3cf53740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:39.969 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.967+0000 7f814ef70740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fc3cf53d10 0x55fc3cf54190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:39.969 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.967+0000 7f814ef70740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fc3cf5d080 0x55fc3cf5f470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:39.970 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.967+0000 7f814ef70740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55fc3ce94400 con 0x55fc3cf549f0 2024-04-02T14:39:39.970 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.967+0000 7f814ef70740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55fc3ce78f10 con 0x55fc3cf53d10 2024-04-02T14:39:39.970 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.967+0000 7f814ef70740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55fc3ceb86f0 con 0x55fc3cf5d080 2024-04-02T14:39:39.970 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.968+0000 7f8149800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fc3cf53d10 0x55fc3cf54190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:39.970 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.968+0000 7f8148e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fc3cf5d080 0x55fc3cf5f470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:39.972 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.968+0000 7f8148e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fc3cf5d080 0x55fc3cf5f470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:44602/0 (socket says 172.21.15.90:44602) 2024-04-02T14:39:39.972 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.968+0000 7f8148e00640 1 -- 172.21.15.90:0/241743486 learned_addr learned my addr 172.21.15.90:0/241743486 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:39:39.972 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.968+0000 7f8149800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fc3cf53d10 0x55fc3cf54190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:48380/0 (socket says 172.21.15.90:48380) 2024-04-02T14:39:39.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.968+0000 7f8142000640 1 --2- 172.21.15.90:0/2771544579 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f812c0a3da0 0x7f812c0a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:39.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.969+0000 7f8142000640 1 -- 172.21.15.90:0/2771544579 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f813c0d0e90 con 0x7f814419ede0 2024-04-02T14:39:39.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.969+0000 7f8148e00640 1 --2- 172.21.15.90:0/241743486 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fc3cf549f0 0x55fc3cf53740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:39.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.969+0000 7f8148e00640 1 -- 172.21.15.90:0/241743486 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fc3cf53d10 msgr2=0x55fc3cf54190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:39:39.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.969+0000 7f8148e00640 1 --2- 172.21.15.90:0/241743486 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fc3cf53d10 0x55fc3cf54190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:39.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.969+0000 7f8148e00640 1 -- 172.21.15.90:0/241743486 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fc3cf549f0 msgr2=0x55fc3cf53740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:39:39.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.969+0000 7f8148e00640 1 --2- 172.21.15.90:0/241743486 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fc3cf549f0 0x55fc3cf53740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:39.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.969+0000 7f8148e00640 1 -- 172.21.15.90:0/241743486 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fc3cbcc3f0 con 0x55fc3cf5d080 2024-04-02T14:39:39.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.969+0000 7f814a200640 1 --2- 172.21.15.90:0/2771544579 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f812c0a3da0 0x7f812c0a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:39.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.970+0000 7f8148e00640 1 --2- 172.21.15.90:0/241743486 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fc3cf5d080 0x55fc3cf5f470 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f8130011510 tx=0x7f8130009910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=24d3bca4febd9fc5 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:39:39.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.970+0000 7f8136a00640 1 -- 172.21.15.90:0/241743486 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8130005020 con 0x55fc3cf5d080 2024-04-02T14:39:39.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.970+0000 7f8136a00640 1 -- 172.21.15.90:0/241743486 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f81300047a0 con 0x55fc3cf5d080 2024-04-02T14:39:39.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.970+0000 7f8136a00640 1 -- 172.21.15.90:0/241743486 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f813000f7e0 con 0x55fc3cf5d080 2024-04-02T14:39:39.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.971+0000 7f814ef70740 1 -- 172.21.15.90:0/241743486 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fc3cf5d080 msgr2=0x55fc3cf5f470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:39.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.971+0000 7f814ef70740 1 --2- 172.21.15.90:0/241743486 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fc3cf5d080 0x55fc3cf5f470 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f8130011510 tx=0x7f8130009910 comp rx=0 tx=0).stop 2024-04-02T14:39:39.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.971+0000 7f814a200640 1 --2- 172.21.15.90:0/2771544579 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f812c0a3da0 0x7f812c0a6260 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f813c0992a0 tx=0x7f813c00a890 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:39:39.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.972+0000 7f814ef70740 1 -- 172.21.15.90:0/241743486 shutdown_connections 2024-04-02T14:39:39.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.972+0000 7f814ef70740 1 --2- 172.21.15.90:0/241743486 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fc3cf5d080 0x55fc3cf5f470 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:39.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.972+0000 7f814ef70740 1 --2- 172.21.15.90:0/241743486 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fc3cf53d10 0x55fc3cf54190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:39.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.972+0000 7f814ef70740 1 --2- 172.21.15.90:0/241743486 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fc3cf549f0 0x55fc3cf53740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:39.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.972+0000 7f814ef70740 1 -- 172.21.15.90:0/241743486 >> 172.21.15.90:0/241743486 conn(0x55fc3cf1aa50 msgr2=0x55fc3cf4e760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:39:39.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.972+0000 7f814ef70740 1 -- 172.21.15.90:0/241743486 shutdown_connections 2024-04-02T14:39:39.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.972+0000 7f814ef70740 1 -- 172.21.15.90:0/241743486 wait complete. 2024-04-02T14:39:39.976 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.972+0000 7f814ef70740 1 -- start start 2024-04-02T14:39:39.976 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.972+0000 7f814ef70740 1 Processor -- start 2024-04-02T14:39:39.976 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.974+0000 7f814ef70740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fc3cf53d10 0x55fc3cf5a160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:39.976 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.974+0000 7f814ef70740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fc3cf549f0 0x55fc3cf5a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:39.976 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.974+0000 7f814ef70740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fc3cf5d080 0x55fc3cf5abe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:39.976 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.974+0000 7f814ef70740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55fc3ceb46a0 con 0x55fc3cf53d10 2024-04-02T14:39:39.976 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.974+0000 7f814ef70740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55fc3ce60980 con 0x55fc3cf5d080 2024-04-02T14:39:39.976 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.974+0000 7f814ef70740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55fc3ce73cb0 con 0x55fc3cf549f0 2024-04-02T14:39:39.977 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.974+0000 7f814ef70740 10 client.0 _ll_register_callbacks cb 0x55fc3ce736a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:39:39.977 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.974+0000 7f8149800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fc3cf549f0 0x55fc3cf5a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:39.977 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.974+0000 7f8149800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fc3cf549f0 0x55fc3cf5a6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:44614/0 (socket says 172.21.15.90:44614) 2024-04-02T14:39:39.977 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.974+0000 7f8149800640 1 -- 172.21.15.90:0/3875588045 learned_addr learned my addr 172.21.15.90:0/3875588045 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:39:39.977 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.975+0000 7f8149800640 1 -- 172.21.15.90:0/3875588045 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fc3cf5d080 msgr2=0x55fc3cf5abe0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-02T14:39:39.977 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.975+0000 7f8149800640 1 --2- 172.21.15.90:0/3875588045 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fc3cf5d080 0x55fc3cf5abe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:39.977 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.975+0000 7f8149800640 1 -- 172.21.15.90:0/3875588045 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fc3cf53d10 msgr2=0x55fc3cf5a160 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-02T14:39:39.979 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.976+0000 7f8148e00640 1 --2- 172.21.15.90:0/3875588045 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fc3cf53d10 0x55fc3cf5a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:39.979 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.977+0000 7f8149800640 1 --2- 172.21.15.90:0/3875588045 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fc3cf53d10 0x55fc3cf5a160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:39.979 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.977+0000 7f8149800640 1 -- 172.21.15.90:0/3875588045 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fc3cc967d0 con 0x55fc3cf549f0 2024-04-02T14:39:39.979 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.977+0000 7f8148e00640 1 --2- 172.21.15.90:0/3875588045 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fc3cf53d10 0x55fc3cf5a160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:39:39.980 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.978+0000 7f8149800640 1 --2- 172.21.15.90:0/3875588045 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fc3cf549f0 0x55fc3cf5a6a0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f8138009230 tx=0x7f81380045b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e4e4726b10994f4 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:39:39.980 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.978+0000 7f812b400640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:39:39.980 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.978+0000 7f812b400640 1 -- 172.21.15.90:0/3875588045 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8138042070 con 0x55fc3cf549f0 2024-04-02T14:39:39.980 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.978+0000 7f812b400640 1 -- 172.21.15.90:0/3875588045 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f8138005030 con 0x55fc3cf549f0 2024-04-02T14:39:39.980 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.978+0000 7f814ef70740 1 -- 172.21.15.90:0/3875588045 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f81300047a0 con 0x55fc3cf549f0 2024-04-02T14:39:39.980 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.978+0000 7f812b400640 1 -- 172.21.15.90:0/3875588045 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f8138005230 con 0x55fc3cf549f0 2024-04-02T14:39:39.980 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.978+0000 7f814ef70740 10 client.26767 fetch_fsmap learned FSMap version 27 2024-04-02T14:39:39.980 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.978+0000 7f814ef70740 1 -- 172.21.15.90:0/3875588045 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55fc3cf6c010 con 0x55fc3cf549f0 2024-04-02T14:39:39.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.979+0000 7f812b400640 1 -- 172.21.15.90:0/3875588045 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f81380053d0 con 0x55fc3cf549f0 2024-04-02T14:39:39.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.979+0000 7f812b400640 1 -- 172.21.15.90:0/3875588045 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f81380037b0 con 0x55fc3cf549f0 2024-04-02T14:39:39.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.979+0000 7f814ef70740 10 client.26767 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:39:39.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.979+0000 7f814ef70740 10 client.26767 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2024-04-02T14:39:39.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.979+0000 7f814ef70740 20 client.26767 populate_metadata read hostname 'smithi090' 2024-04-02T14:39:39.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.979+0000 7f814ef70740 1 --2- 172.21.15.90:0/3875588045 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x55fc3cf6d520 0x55fc3cf6f950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:39.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.979+0000 7f814ef70740 4 client.26767 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:39:39.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.979+0000 7f814ef70740 1 -- 172.21.15.90:0/3875588045 --> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f81300047a0 con 0x55fc3cf6d520 2024-04-02T14:39:39.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.979+0000 7f8148e00640 1 --2- 172.21.15.90:0/3875588045 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x55fc3cf6d520 0x55fc3cf6f950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:39.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.980+0000 7f8148e00640 1 --2- 172.21.15.90:0/3875588045 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x55fc3cf6d520 0x55fc3cf6f950 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=16c5405e04f50e2e server_cookie=37e5b103b1faf3cb in_seq=0 out_seq=0 2024-04-02T14:39:39.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.980+0000 7f812b400640 10 client.26767 ms_handle_connect on v2:172.21.15.90:6838/1498915870 2024-04-02T14:39:39.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.981+0000 7f812b400640 1 -- 172.21.15.90:0/3875588045 <== mds.1 v2:172.21.15.90:6838/1498915870 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x7f81300047a0 con 0x55fc3cf6d520 2024-04-02T14:39:39.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.981+0000 7f812b400640 10 client.26767 handle_command_reply: tid=0 2024-04-02T14:39:39.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.982+0000 7f814ef70740 1 client.26767 shutdown 2024-04-02T14:39:39.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.982+0000 7f814ef70740 1 -- 172.21.15.90:0/3875588045 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fc3cf549f0 msgr2=0x55fc3cf5a6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:39.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.982+0000 7f814ef70740 1 --2- 172.21.15.90:0/3875588045 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fc3cf549f0 0x55fc3cf5a6a0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f8138009230 tx=0x7f81380045b0 comp rx=0 tx=0).stop 2024-04-02T14:39:39.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.982+0000 7f814ef70740 1 -- 172.21.15.90:0/3875588045 shutdown_connections 2024-04-02T14:39:39.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.982+0000 7f814ef70740 1 --2- 172.21.15.90:0/3875588045 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x55fc3cf6d520 0x55fc3cf6f950 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:39.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.982+0000 7f814ef70740 1 --2- 172.21.15.90:0/3875588045 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fc3cf5d080 0x55fc3cf5abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:39.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.982+0000 7f814ef70740 1 --2- 172.21.15.90:0/3875588045 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fc3cf549f0 0x55fc3cf5a6a0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:39.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.982+0000 7f814ef70740 1 --2- 172.21.15.90:0/3875588045 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fc3cf53d10 0x55fc3cf5a160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:39.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.982+0000 7f814ef70740 1 -- 172.21.15.90:0/3875588045 >> 172.21.15.90:0/3875588045 conn(0x55fc3cf1aa50 msgr2=0x55fc3cf504b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:39:39.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.982+0000 7f812b400640 0 client.26767 ms_handle_reset on v2:172.21.15.90:6838/1498915870 2024-04-02T14:39:39.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.983+0000 7f814ef70740 1 -- 172.21.15.90:0/3875588045 shutdown_connections 2024-04-02T14:39:39.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.983+0000 7f814ef70740 1 -- 172.21.15.90:0/3875588045 wait complete. 2024-04-02T14:39:39.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:39.983+0000 7f814ef70740 20 client.26767 trim_cache size 0 max 16384 2024-04-02T14:39:40.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.011+0000 7f814ef70740 1 Processor -- start 2024-04-02T14:39:40.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.012+0000 7f814ef70740 1 -- start start 2024-04-02T14:39:40.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.012+0000 7f814ef70740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fc3cf53d10 0x55fc3cf5c890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:40.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.012+0000 7f814ef70740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fc3cf549f0 0x55fc3cf68af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:40.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.013+0000 7f814ef70740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fc3cf5d080 0x55fc3cf75260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:40.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.013+0000 7f814ef70740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f81441b3dd0 con 0x55fc3cf5d080 2024-04-02T14:39:40.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.013+0000 7f814ef70740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f81441b5910 con 0x55fc3cf549f0 2024-04-02T14:39:40.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.013+0000 7f814ef70740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f81441b44a0 con 0x55fc3cf53d10 2024-04-02T14:39:40.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.013+0000 7f8148e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fc3cf53d10 0x55fc3cf5c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:40.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.013+0000 7f8148e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fc3cf53d10 0x55fc3cf5c890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:44624/0 (socket says 172.21.15.90:44624) 2024-04-02T14:39:40.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.013+0000 7f8148e00640 1 -- 172.21.15.90:0/318271427 learned_addr learned my addr 172.21.15.90:0/318271427 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:39:40.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.013+0000 7f8148e00640 1 --2- 172.21.15.90:0/318271427 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fc3cf5d080 0x55fc3cf75260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:40.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.013+0000 7f8149800640 1 --2- 172.21.15.90:0/318271427 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fc3cf549f0 0x55fc3cf68af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:40.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.014+0000 7f8149800640 1 -- 172.21.15.90:0/318271427 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fc3cf53d10 msgr2=0x55fc3cf5c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:39:40.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.014+0000 7f8149800640 1 --2- 172.21.15.90:0/318271427 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fc3cf53d10 0x55fc3cf5c890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:40.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.014+0000 7f8149800640 1 -- 172.21.15.90:0/318271427 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fc3cf5d080 msgr2=0x55fc3cf75260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:39:40.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.014+0000 7f8149800640 1 --2- 172.21.15.90:0/318271427 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fc3cf5d080 0x55fc3cf75260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:40.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.014+0000 7f8149800640 1 -- 172.21.15.90:0/318271427 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fc3cf69ac0 con 0x55fc3cf549f0 2024-04-02T14:39:40.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.014+0000 7f8148e00640 1 --2- 172.21.15.90:0/318271427 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fc3cf5d080 0x55fc3cf75260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:39:40.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.014+0000 7f8149800640 1 --2- 172.21.15.90:0/318271427 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fc3cf549f0 0x55fc3cf68af0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f8138003cf0 tx=0x7f813802bc40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3bc428a7a782792b server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:39:40.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.015+0000 7f8136000640 1 -- 172.21.15.90:0/318271427 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8138053030 con 0x55fc3cf549f0 2024-04-02T14:39:40.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.015+0000 7f8136000640 1 -- 172.21.15.90:0/318271427 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8138005000 con 0x55fc3cf549f0 2024-04-02T14:39:40.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.015+0000 7f8136000640 1 -- 172.21.15.90:0/318271427 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8138042070 con 0x55fc3cf549f0 2024-04-02T14:39:40.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.016+0000 7f814ef70740 1 -- 172.21.15.90:0/318271427 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fc3cf549f0 msgr2=0x55fc3cf68af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:40.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.016+0000 7f814ef70740 1 --2- 172.21.15.90:0/318271427 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fc3cf549f0 0x55fc3cf68af0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f8138003cf0 tx=0x7f813802bc40 comp rx=0 tx=0).stop 2024-04-02T14:39:40.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.016+0000 7f814ef70740 1 -- 172.21.15.90:0/318271427 shutdown_connections 2024-04-02T14:39:40.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.016+0000 7f814ef70740 1 --2- 172.21.15.90:0/318271427 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fc3cf5d080 0x55fc3cf75260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:40.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.016+0000 7f814ef70740 1 --2- 172.21.15.90:0/318271427 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fc3cf549f0 0x55fc3cf68af0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:40.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.016+0000 7f814ef70740 1 --2- 172.21.15.90:0/318271427 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fc3cf53d10 0x55fc3cf5c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:40.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.016+0000 7f814ef70740 1 -- 172.21.15.90:0/318271427 >> 172.21.15.90:0/318271427 conn(0x55fc3cf1aa50 msgr2=0x55fc3cf4e290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:39:40.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.017+0000 7f814ef70740 1 -- 172.21.15.90:0/318271427 shutdown_connections 2024-04-02T14:39:40.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.017+0000 7f814ef70740 1 -- 172.21.15.90:0/318271427 wait complete. 2024-04-02T14:39:40.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.018+0000 7f814ef70740 1 -- start start 2024-04-02T14:39:40.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.018+0000 7f814ef70740 1 Processor -- start 2024-04-02T14:39:40.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.019+0000 7f814ef70740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fc3cf53d10 0x55fc3cf654e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:40.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.019+0000 7f814ef70740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fc3cf549f0 0x55fc3cf57450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:40.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.019+0000 7f814ef70740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fc3cf5d080 0x55fc3cf57a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:40.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.019+0000 7f814ef70740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f81441b5240 con 0x55fc3cf5d080 2024-04-02T14:39:40.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.019+0000 7f814ef70740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f81441b4b70 con 0x55fc3cf549f0 2024-04-02T14:39:40.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.019+0000 7f814ef70740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f81441b1dd0 con 0x55fc3cf53d10 2024-04-02T14:39:40.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.019+0000 7f814ef70740 10 client.0 _ll_register_callbacks cb 0x7f8138004470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:39:40.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.020+0000 7f8149800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fc3cf549f0 0x55fc3cf57450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:40.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.020+0000 7f8149800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fc3cf549f0 0x55fc3cf57450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:48396/0 (socket says 172.21.15.90:48396) 2024-04-02T14:39:40.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.020+0000 7f8149800640 1 -- 172.21.15.90:0/2884843853 learned_addr learned my addr 172.21.15.90:0/2884843853 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:39:40.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.020+0000 7f8148e00640 1 --2- 172.21.15.90:0/2884843853 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fc3cf53d10 0x55fc3cf654e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:40.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.020+0000 7f8149800640 1 -- 172.21.15.90:0/2884843853 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fc3cf53d10 msgr2=0x55fc3cf654e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:39:40.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.020+0000 7f8149800640 1 --2- 172.21.15.90:0/2884843853 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fc3cf53d10 0x55fc3cf654e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:40.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.020+0000 7f8149800640 1 -- 172.21.15.90:0/2884843853 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fc3cf5d080 msgr2=0x55fc3cf57a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:39:40.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.020+0000 7f8149800640 1 --2- 172.21.15.90:0/2884843853 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fc3cf5d080 0x55fc3cf57a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:40.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.020+0000 7f8149800640 1 -- 172.21.15.90:0/2884843853 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fc3cf5dc10 con 0x55fc3cf549f0 2024-04-02T14:39:40.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.021+0000 7f8149800640 1 --2- 172.21.15.90:0/2884843853 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fc3cf549f0 0x55fc3cf57450 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f8138034480 tx=0x7f813802c200 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b8ebca409caa84aa server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:39:40.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.021+0000 7f812b400640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:39:40.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.021+0000 7f812b400640 1 -- 172.21.15.90:0/2884843853 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8138053070 con 0x55fc3cf549f0 2024-04-02T14:39:40.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.021+0000 7f812b400640 1 -- 172.21.15.90:0/2884843853 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f81380046c0 con 0x55fc3cf549f0 2024-04-02T14:39:40.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.021+0000 7f814ef70740 1 -- 172.21.15.90:0/2884843853 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8138005000 con 0x55fc3cf549f0 2024-04-02T14:39:40.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.022+0000 7f812b400640 1 -- 172.21.15.90:0/2884843853 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f8138042040 con 0x55fc3cf549f0 2024-04-02T14:39:40.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.022+0000 7f814ef70740 10 client.26894 fetch_fsmap learned FSMap version 27 2024-04-02T14:39:40.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.022+0000 7f814ef70740 1 -- 172.21.15.90:0/2884843853 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55fc3cfd3cb0 con 0x55fc3cf549f0 2024-04-02T14:39:40.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.022+0000 7f812b400640 1 -- 172.21.15.90:0/2884843853 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8138005000 con 0x55fc3cf549f0 2024-04-02T14:39:40.025 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.022+0000 7f812b400640 1 -- 172.21.15.90:0/2884843853 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f8138034640 con 0x55fc3cf549f0 2024-04-02T14:39:40.025 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.022+0000 7f814ef70740 10 client.26894 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:39:40.025 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.022+0000 7f814ef70740 10 client.26894 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2024-04-02T14:39:40.025 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.022+0000 7f814ef70740 20 client.26894 populate_metadata read hostname 'smithi090' 2024-04-02T14:39:40.025 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.022+0000 7f814ef70740 1 --2- 172.21.15.90:0/2884843853 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x55fc3cfd42b0 0x55fc3cfd66a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:40.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.022+0000 7f814ef70740 4 client.26894 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-02T14:39:40.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.022+0000 7f814ef70740 1 -- 172.21.15.90:0/2884843853 --> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55fc3ce78040 con 0x55fc3cfd42b0 2024-04-02T14:39:40.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.023+0000 7f8148e00640 1 --2- 172.21.15.90:0/2884843853 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x55fc3cfd42b0 0x55fc3cfd66a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:40.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.024+0000 7f8148e00640 1 --2- 172.21.15.90:0/2884843853 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x55fc3cfd42b0 0x55fc3cfd66a0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=d49c7c5508deda25 server_cookie=67eb33a2656ac750 in_seq=0 out_seq=0 2024-04-02T14:39:40.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.024+0000 7f812b400640 10 client.26894 ms_handle_connect on v2:172.21.15.90:6838/1498915870 2024-04-02T14:39:40.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.024+0000 7f812b400640 1 -- 172.21.15.90:0/2884843853 <== mds.1 v2:172.21.15.90:6838/1498915870 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55fc3ce78040 con 0x55fc3cfd42b0 2024-04-02T14:39:40.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.024+0000 7f812b400640 10 client.26894 handle_command_reply: tid=0 2024-04-02T14:39:40.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.025+0000 7f814ef70740 1 client.26894 shutdown 2024-04-02T14:39:40.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.025+0000 7f814ef70740 1 -- 172.21.15.90:0/2884843853 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fc3cf549f0 msgr2=0x55fc3cf57450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:40.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.025+0000 7f814ef70740 1 --2- 172.21.15.90:0/2884843853 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fc3cf549f0 0x55fc3cf57450 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f8138034480 tx=0x7f813802c200 comp rx=0 tx=0).stop 2024-04-02T14:39:40.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.025+0000 7f814ef70740 1 -- 172.21.15.90:0/2884843853 shutdown_connections 2024-04-02T14:39:40.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.025+0000 7f814ef70740 1 --2- 172.21.15.90:0/2884843853 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x55fc3cfd42b0 0x55fc3cfd66a0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:40.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.025+0000 7f814ef70740 1 --2- 172.21.15.90:0/2884843853 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fc3cf5d080 0x55fc3cf57a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:40.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.025+0000 7f814ef70740 1 --2- 172.21.15.90:0/2884843853 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fc3cf549f0 0x55fc3cf57450 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:40.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.025+0000 7f814ef70740 1 --2- 172.21.15.90:0/2884843853 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fc3cf53d10 0x55fc3cf654e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:40.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.025+0000 7f812b400640 0 client.26894 ms_handle_reset on v2:172.21.15.90:6838/1498915870 2024-04-02T14:39:40.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.025+0000 7f814ef70740 1 -- 172.21.15.90:0/2884843853 >> 172.21.15.90:0/2884843853 conn(0x55fc3cf1aa50 msgr2=0x55fc3cf4d010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:39:40.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.025+0000 7f814ef70740 1 -- 172.21.15.90:0/2884843853 shutdown_connections 2024-04-02T14:39:40.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.025+0000 7f814ef70740 1 -- 172.21.15.90:0/2884843853 wait complete. 2024-04-02T14:39:40.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.026+0000 7f814ef70740 20 client.26894 trim_cache size 0 max 16384 2024-04-02T14:39:40.029 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-02T14:39:40.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.026+0000 7f814c200640 1 -- 172.21.15.90:0/2771544579 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f812c0a3da0 msgr2=0x7f812c0a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:40.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.026+0000 7f814c200640 1 --2- 172.21.15.90:0/2771544579 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f812c0a3da0 0x7f812c0a6260 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f813c0992a0 tx=0x7f813c00a890 comp rx=0 tx=0).stop 2024-04-02T14:39:40.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.026+0000 7f814c200640 1 -- 172.21.15.90:0/2771544579 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f814419ede0 msgr2=0x7f814419f290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:40.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.026+0000 7f814c200640 1 --2- 172.21.15.90:0/2771544579 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f814419ede0 0x7f814419f290 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f813c00cab0 tx=0x7f813c00cf70 comp rx=0 tx=0).stop 2024-04-02T14:39:40.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.027+0000 7f814c200640 1 -- 172.21.15.90:0/2771544579 shutdown_connections 2024-04-02T14:39:40.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.027+0000 7f814c200640 1 --2- 172.21.15.90:0/2771544579 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f812c0a3da0 0x7f812c0a6260 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:40.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.027+0000 7f814c200640 1 --2- 172.21.15.90:0/2771544579 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f814419ede0 0x7f814419f290 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:40.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.027+0000 7f814c200640 1 --2- 172.21.15.90:0/2771544579 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8144108fe0 0x7f814419aab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:40.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.027+0000 7f814c200640 1 --2- 172.21.15.90:0/2771544579 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8144103f10 0x7f814419a570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:40.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.027+0000 7f814c200640 1 -- 172.21.15.90:0/2771544579 >> 172.21.15.90:0/2771544579 conn(0x7f81440ff750 msgr2=0x7f81441002c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:39:40.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.027+0000 7f814c200640 1 -- 172.21.15.90:0/2771544579 shutdown_connections 2024-04-02T14:39:40.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.028+0000 7f814c200640 1 -- 172.21.15.90:0/2771544579 wait complete. 2024-04-02T14:39:40.055 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-02T14:39:40.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.322+0000 7f10f7200640 1 -- 172.21.15.90:0/187454382 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f10f8112980 msgr2=0x7f10f8114d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:40.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.322+0000 7f10f7200640 1 --2- 172.21.15.90:0/187454382 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f10f8112980 0x7f10f8114d70 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f10e0009900 tx=0x7f10e002bcd0 comp rx=0 tx=0).stop 2024-04-02T14:39:40.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.322+0000 7f10f7200640 1 -- 172.21.15.90:0/187454382 shutdown_connections 2024-04-02T14:39:40.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.322+0000 7f10f7200640 1 --2- 172.21.15.90:0/187454382 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f10f8112980 0x7f10f8114d70 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:40.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.323+0000 7f10f7200640 1 --2- 172.21.15.90:0/187454382 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f10f80747c0 0x7f10f8112440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:40.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.323+0000 7f10f7200640 1 --2- 172.21.15.90:0/187454382 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f10f8073ea0 0x7f10f8074280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:40.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.323+0000 7f10f7200640 1 -- 172.21.15.90:0/187454382 >> 172.21.15.90:0/187454382 conn(0x7f10f80fd4a0 msgr2=0x7f10f80ff8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:39:40.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.323+0000 7f10f7200640 1 -- 172.21.15.90:0/187454382 shutdown_connections 2024-04-02T14:39:40.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.323+0000 7f10f7200640 1 -- 172.21.15.90:0/187454382 wait complete. 2024-04-02T14:39:40.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.324+0000 7f10f7200640 1 Processor -- start 2024-04-02T14:39:40.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.324+0000 7f10f7200640 1 -- start start 2024-04-02T14:39:40.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.325+0000 7f10f7200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f10f8073ea0 0x7f10f819e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:40.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.325+0000 7f10f7200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f10f80747c0 0x7f10f819ec20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:40.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.325+0000 7f10f7200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f10f8112980 0x7f10f81a2fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:40.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.325+0000 7f10f7200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f10f8118320 con 0x7f10f80747c0 2024-04-02T14:39:40.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.325+0000 7f10f7200640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f10f81181a0 con 0x7f10f8073ea0 2024-04-02T14:39:40.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.325+0000 7f10f7200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f10f81184a0 con 0x7f10f8112980 2024-04-02T14:39:40.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.325+0000 7f10f6800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f10f8112980 0x7f10f81a2fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:40.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.325+0000 7f10f5e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f10f8073ea0 0x7f10f819e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:40.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.325+0000 7f10f6800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f10f8112980 0x7f10f81a2fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:44660/0 (socket says 172.21.15.90:44660) 2024-04-02T14:39:40.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.325+0000 7f10f6800640 1 -- 172.21.15.90:0/2570044433 learned_addr learned my addr 172.21.15.90:0/2570044433 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:39:40.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.326+0000 7f10f5400640 1 --2- 172.21.15.90:0/2570044433 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f10f80747c0 0x7f10f819ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:40.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.326+0000 7f10f5e00640 1 -- 172.21.15.90:0/2570044433 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f10f8112980 msgr2=0x7f10f81a2fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:40.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.326+0000 7f10f5e00640 1 --2- 172.21.15.90:0/2570044433 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f10f8112980 0x7f10f81a2fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:40.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.326+0000 7f10f5e00640 1 -- 172.21.15.90:0/2570044433 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f10f80747c0 msgr2=0x7f10f819ec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:40.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.326+0000 7f10f5e00640 1 --2- 172.21.15.90:0/2570044433 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f10f80747c0 0x7f10f819ec20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:40.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.326+0000 7f10f5e00640 1 -- 172.21.15.90:0/2570044433 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10f81a36d0 con 0x7f10f8073ea0 2024-04-02T14:39:40.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.326+0000 7f10f5e00640 1 --2- 172.21.15.90:0/2570044433 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f10f8073ea0 0x7f10f819e6e0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f10f000c970 tx=0x7f10f000ce30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:39:40.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.326+0000 7f10eea00640 1 -- 172.21.15.90:0/2570044433 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10f0007bb0 con 0x7f10f8073ea0 2024-04-02T14:39:40.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.327+0000 7f10f7200640 1 -- 172.21.15.90:0/2570044433 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10f81a39c0 con 0x7f10f8073ea0 2024-04-02T14:39:40.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.327+0000 7f10eea00640 1 -- 172.21.15.90:0/2570044433 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f10f0007d50 con 0x7f10f8073ea0 2024-04-02T14:39:40.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.327+0000 7f10eea00640 1 -- 172.21.15.90:0/2570044433 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10f0018820 con 0x7f10f8073ea0 2024-04-02T14:39:40.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.327+0000 7f10f7200640 1 -- 172.21.15.90:0/2570044433 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10f81ab250 con 0x7f10f8073ea0 2024-04-02T14:39:40.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.331+0000 7f10ff9a8740 1 Processor -- start 2024-04-02T14:39:40.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.331+0000 7f10ff9a8740 1 -- start start 2024-04-02T14:39:40.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.332+0000 7f10ff9a8740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ecbaee4ba0 0x55ecbaee38f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:40.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.332+0000 7f10ff9a8740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ecbaee3ec0 0x55ecbaee4340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:40.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.332+0000 7f10ff9a8740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ecbaeed230 0x55ecbaeef620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:40.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.332+0000 7f10ff9a8740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55ecbae24400 con 0x55ecbaeed230 2024-04-02T14:39:40.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.332+0000 7f10ff9a8740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55ecbae08f10 con 0x55ecbaee4ba0 2024-04-02T14:39:40.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.332+0000 7f10ff9a8740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55ecbae486f0 con 0x55ecbaee3ec0 2024-04-02T14:39:40.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.332+0000 7f10f6800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ecbaee4ba0 0x55ecbaee38f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:40.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.333+0000 7f10f6800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ecbaee4ba0 0x55ecbaee38f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:48432/0 (socket says 172.21.15.90:48432) 2024-04-02T14:39:40.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.333+0000 7f10f6800640 1 -- 172.21.15.90:0/2399297910 learned_addr learned my addr 172.21.15.90:0/2399297910 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:39:40.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.333+0000 7f10f5400640 1 --2- 172.21.15.90:0/2399297910 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ecbaeed230 0x55ecbaeef620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:40.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.333+0000 7f10f6800640 1 -- 172.21.15.90:0/2399297910 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ecbaee3ec0 msgr2=0x55ecbaee4340 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-02T14:39:40.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.333+0000 7f10f6800640 1 --2- 172.21.15.90:0/2399297910 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ecbaee3ec0 0x55ecbaee4340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:40.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.333+0000 7f10f6800640 1 -- 172.21.15.90:0/2399297910 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ecbaeed230 msgr2=0x55ecbaeef620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:39:40.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.333+0000 7f10f6800640 1 --2- 172.21.15.90:0/2399297910 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ecbaeed230 0x55ecbaeef620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:40.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.333+0000 7f10f6800640 1 -- 172.21.15.90:0/2399297910 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ecbab5c3f0 con 0x55ecbaee4ba0 2024-04-02T14:39:40.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.333+0000 7f10f5400640 1 --2- 172.21.15.90:0/2399297910 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ecbaeed230 0x55ecbaeef620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:39:40.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.334+0000 7f10eea00640 1 -- 172.21.15.90:0/2570044433 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f10f0018a80 con 0x7f10f8073ea0 2024-04-02T14:39:40.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.334+0000 7f10f6800640 1 --2- 172.21.15.90:0/2399297910 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ecbaee4ba0 0x55ecbaee38f0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f10e0005e80 tx=0x7f10e002c670 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c452c18818f7e32e server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:39:40.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.334+0000 7f10e7000640 1 -- 172.21.15.90:0/2399297910 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10e0042030 con 0x55ecbaee4ba0 2024-04-02T14:39:40.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.334+0000 7f10e7000640 1 -- 172.21.15.90:0/2399297910 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f10e0004040 con 0x55ecbaee4ba0 2024-04-02T14:39:40.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.335+0000 7f10ff9a8740 1 -- 172.21.15.90:0/2399297910 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ecbaee4ba0 msgr2=0x55ecbaee38f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:40.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.335+0000 7f10e7000640 1 -- 172.21.15.90:0/2399297910 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10e0004f10 con 0x55ecbaee4ba0 2024-04-02T14:39:40.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.335+0000 7f10ff9a8740 1 --2- 172.21.15.90:0/2399297910 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ecbaee4ba0 0x55ecbaee38f0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f10e0005e80 tx=0x7f10e002c670 comp rx=0 tx=0).stop 2024-04-02T14:39:40.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.335+0000 7f10ff9a8740 1 -- 172.21.15.90:0/2399297910 shutdown_connections 2024-04-02T14:39:40.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.335+0000 7f10eea00640 1 --2- 172.21.15.90:0/2570044433 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f10dc0a3da0 0x7f10dc0a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:40.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.335+0000 7f10ff9a8740 1 --2- 172.21.15.90:0/2399297910 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ecbaeed230 0x55ecbaeef620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:40.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.335+0000 7f10ff9a8740 1 --2- 172.21.15.90:0/2399297910 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ecbaee3ec0 0x55ecbaee4340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:40.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.335+0000 7f10ff9a8740 1 --2- 172.21.15.90:0/2399297910 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ecbaee4ba0 0x55ecbaee38f0 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:40.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.335+0000 7f10ff9a8740 1 -- 172.21.15.90:0/2399297910 >> 172.21.15.90:0/2399297910 conn(0x55ecbaeaaa50 msgr2=0x55ecbaede910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:39:40.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.335+0000 7f10eea00640 1 -- 172.21.15.90:0/2570044433 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f10f00d08a0 con 0x7f10f8073ea0 2024-04-02T14:39:40.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.335+0000 7f10f6800640 1 --2- 172.21.15.90:0/2570044433 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f10dc0a3da0 0x7f10dc0a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:40.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.336+0000 7f10ff9a8740 1 -- 172.21.15.90:0/2399297910 shutdown_connections 2024-04-02T14:39:40.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.336+0000 7f10ff9a8740 1 -- 172.21.15.90:0/2399297910 wait complete. 2024-04-02T14:39:40.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.337+0000 7f10ff9a8740 1 -- start start 2024-04-02T14:39:40.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.337+0000 7f10ff9a8740 1 Processor -- start 2024-04-02T14:39:40.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.337+0000 7f10f6800640 1 --2- 172.21.15.90:0/2570044433 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f10dc0a3da0 0x7f10dc0a6260 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f10e002f020 tx=0x7f10e0040040 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:39:40.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.338+0000 7f10ff9a8740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ecbaee3ec0 0x55ecbaeea150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:40.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.338+0000 7f10ff9a8740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ecbaee4ba0 0x55ecbaeea690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:40.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.338+0000 7f10ff9a8740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ecbaeed230 0x55ecbaeeabd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:40.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.339+0000 7f10ff9a8740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55ecbae446a0 con 0x55ecbaee3ec0 2024-04-02T14:39:40.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.339+0000 7f10f5400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ecbaee4ba0 0x55ecbaeea690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:40.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.339+0000 7f10ff9a8740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55ecbadf0980 con 0x55ecbaee4ba0 2024-04-02T14:39:40.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.339+0000 7f10ff9a8740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55ecbae03cb0 con 0x55ecbaeed230 2024-04-02T14:39:40.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.339+0000 7f10f6800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ecbaeed230 0x55ecbaeeabd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:40.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.339+0000 7f10f5e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ecbaee3ec0 0x55ecbaeea150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:40.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.339+0000 7f10f6800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ecbaeed230 0x55ecbaeeabd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:44670/0 (socket says 172.21.15.90:44670) 2024-04-02T14:39:40.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.339+0000 7f10f6800640 1 -- 172.21.15.90:0/3803980840 learned_addr learned my addr 172.21.15.90:0/3803980840 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:39:40.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.339+0000 7f10f5400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ecbaee4ba0 0x55ecbaeea690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:48442/0 (socket says 172.21.15.90:48442) 2024-04-02T14:39:40.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.339+0000 7f10ff9a8740 10 client.0 _ll_register_callbacks cb 0x55ecbae036a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:39:40.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.340+0000 7f10f5e00640 1 -- 172.21.15.90:0/3803980840 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ecbaeed230 msgr2=0x55ecbaeeabd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:39:40.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.340+0000 7f10f5e00640 1 --2- 172.21.15.90:0/3803980840 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ecbaeed230 0x55ecbaeeabd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:40.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.340+0000 7f10f5e00640 1 -- 172.21.15.90:0/3803980840 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ecbaee4ba0 msgr2=0x55ecbaeea690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:39:40.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.340+0000 7f10f5e00640 1 --2- 172.21.15.90:0/3803980840 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ecbaee4ba0 0x55ecbaeea690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:40.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.340+0000 7f10f5e00640 1 -- 172.21.15.90:0/3803980840 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ecbac267d0 con 0x55ecbaee3ec0 2024-04-02T14:39:40.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.340+0000 7f10f6800640 1 --2- 172.21.15.90:0/3803980840 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ecbaeed230 0x55ecbaeeabd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:39:40.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.340+0000 7f10f5e00640 1 --2- 172.21.15.90:0/3803980840 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ecbaee3ec0 0x55ecbaeea150 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f10f00ce000 tx=0x7f10f009c1b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5bdfc43c901b13d3 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:39:40.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.340+0000 7f10db400640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-02T14:39:40.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.340+0000 7f10db400640 1 -- 172.21.15.90:0/3803980840 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10f00a0c00 con 0x55ecbaee3ec0 2024-04-02T14:39:40.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.340+0000 7f10db400640 1 -- 172.21.15.90:0/3803980840 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f10f000fac0 con 0x55ecbaee3ec0 2024-04-02T14:39:40.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.340+0000 7f10ff9a8740 1 -- 172.21.15.90:0/3803980840 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f10e0004040 con 0x55ecbaee3ec0 2024-04-02T14:39:40.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.341+0000 7f10db400640 1 -- 172.21.15.90:0/3803980840 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f10f00a54c0 con 0x55ecbaee3ec0 2024-04-02T14:39:40.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.341+0000 7f10ff9a8740 10 client.16356 fetch_fsmap learned FSMap version 27 2024-04-02T14:39:40.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.341+0000 7f10ff9a8740 1 -- 172.21.15.90:0/3803980840 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55ecbaefbe70 con 0x55ecbaee3ec0 2024-04-02T14:39:40.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.341+0000 7f10db400640 1 -- 172.21.15.90:0/3803980840 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f10e0004040 con 0x55ecbaee3ec0 2024-04-02T14:39:40.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.341+0000 7f10db400640 1 -- 172.21.15.90:0/3803980840 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f10f00b2070 con 0x55ecbaee3ec0 2024-04-02T14:39:40.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.342+0000 7f10ff9a8740 10 client.16356 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:39:40.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.342+0000 7f10ff9a8740 10 client.16356 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:39:40.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.342+0000 7f10ff9a8740 20 client.16356 populate_metadata read hostname 'smithi090' 2024-04-02T14:39:40.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.342+0000 7f10ff9a8740 1 --2- 172.21.15.90:0/3803980840 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55ecbaefd380 0x55ecbaeff770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:40.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.342+0000 7f10ff9a8740 4 client.16356 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:39:40.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.342+0000 7f10ff9a8740 1 -- 172.21.15.90:0/3803980840 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ecbae08040 con 0x55ecbaefd380 2024-04-02T14:39:40.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.342+0000 7f10f5400640 1 --2- 172.21.15.90:0/3803980840 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55ecbaefd380 0x55ecbaeff770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:40.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.343+0000 7f10f5400640 1 --2- 172.21.15.90:0/3803980840 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55ecbaefd380 0x55ecbaeff770 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bfa0803106013490 server_cookie=91d0ed660fe23299 in_seq=0 out_seq=0 2024-04-02T14:39:40.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.343+0000 7f10db400640 10 client.16356 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:39:40.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.344+0000 7f10db400640 1 -- 172.21.15.90:0/3803980840 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ecbae08040 con 0x55ecbaefd380 2024-04-02T14:39:40.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.344+0000 7f10db400640 10 client.16356 handle_command_reply: tid=0 2024-04-02T14:39:40.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.344+0000 7f10ff9a8740 1 client.16356 shutdown 2024-04-02T14:39:40.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.344+0000 7f10ff9a8740 1 -- 172.21.15.90:0/3803980840 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ecbaee3ec0 msgr2=0x55ecbaeea150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:40.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.344+0000 7f10ff9a8740 1 --2- 172.21.15.90:0/3803980840 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ecbaee3ec0 0x55ecbaeea150 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f10f00ce000 tx=0x7f10f009c1b0 comp rx=0 tx=0).stop 2024-04-02T14:39:40.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.345+0000 7f10ff9a8740 1 -- 172.21.15.90:0/3803980840 shutdown_connections 2024-04-02T14:39:40.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.345+0000 7f10ff9a8740 1 --2- 172.21.15.90:0/3803980840 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55ecbaefd380 0x55ecbaeff770 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:40.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.345+0000 7f10ff9a8740 1 --2- 172.21.15.90:0/3803980840 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ecbaeed230 0x55ecbaeeabd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:40.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.345+0000 7f10ff9a8740 1 --2- 172.21.15.90:0/3803980840 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ecbaee4ba0 0x55ecbaeea690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:40.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.345+0000 7f10ff9a8740 1 --2- 172.21.15.90:0/3803980840 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ecbaee3ec0 0x55ecbaeea150 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:40.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.345+0000 7f10ff9a8740 1 -- 172.21.15.90:0/3803980840 >> 172.21.15.90:0/3803980840 conn(0x55ecbaeaaa50 msgr2=0x55ecbaee0660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:39:40.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.345+0000 7f10db400640 0 client.16356 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:39:40.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.345+0000 7f10ff9a8740 1 -- 172.21.15.90:0/3803980840 shutdown_connections 2024-04-02T14:39:40.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.345+0000 7f10ff9a8740 1 -- 172.21.15.90:0/3803980840 wait complete. 2024-04-02T14:39:40.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.345+0000 7f10ff9a8740 20 client.16356 trim_cache size 0 max 16384 2024-04-02T14:39:40.360 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:39:40 smithi092 ceph-mon[36497]: pgmap v146: 97 pgs: 97 active+clean; 3.3 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 175 KiB/s rd, 40 MiB/s wr, 2.02k op/s 2024-04-02T14:39:40.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.368+0000 7f10ff9a8740 1 Processor -- start 2024-04-02T14:39:40.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.368+0000 7f10ff9a8740 1 -- start start 2024-04-02T14:39:40.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.369+0000 7f10ff9a8740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ecbaee3ec0 0x55ecbaeec880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:40.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.369+0000 7f10ff9a8740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ecbaee4ba0 0x55ecbaef8d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:40.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.369+0000 7f10ff9a8740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ecbaeed230 0x55ecbaf05040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:40.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.369+0000 7f10ff9a8740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f10f81b3a20 con 0x55ecbaee4ba0 2024-04-02T14:39:40.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.369+0000 7f10ff9a8740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f10f81b5560 con 0x55ecbaee3ec0 2024-04-02T14:39:40.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.369+0000 7f10ff9a8740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f10f81b40f0 con 0x55ecbaeed230 2024-04-02T14:39:40.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.369+0000 7f10f6800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ecbaeed230 0x55ecbaf05040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:40.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.369+0000 7f10f6800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ecbaeed230 0x55ecbaf05040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:44684/0 (socket says 172.21.15.90:44684) 2024-04-02T14:39:40.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.369+0000 7f10f6800640 1 -- 172.21.15.90:0/806908912 learned_addr learned my addr 172.21.15.90:0/806908912 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:39:40.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.369+0000 7f10f5e00640 1 --2- 172.21.15.90:0/806908912 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ecbaee3ec0 0x55ecbaeec880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:40.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.369+0000 7f10f5400640 1 --2- 172.21.15.90:0/806908912 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ecbaee4ba0 0x55ecbaef8d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:40.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.370+0000 7f10f6800640 1 -- 172.21.15.90:0/806908912 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ecbaee3ec0 msgr2=0x55ecbaeec880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:39:40.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.370+0000 7f10f6800640 1 --2- 172.21.15.90:0/806908912 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ecbaee3ec0 0x55ecbaeec880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:40.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.370+0000 7f10f6800640 1 -- 172.21.15.90:0/806908912 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ecbaee4ba0 msgr2=0x55ecbaef8d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:39:40.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.370+0000 7f10f6800640 1 --2- 172.21.15.90:0/806908912 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ecbaee4ba0 0x55ecbaef8d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:40.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.370+0000 7f10f6800640 1 -- 172.21.15.90:0/806908912 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ecbaeeff80 con 0x55ecbaeed230 2024-04-02T14:39:40.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.370+0000 7f10f5400640 1 --2- 172.21.15.90:0/806908912 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ecbaee4ba0 0x55ecbaef8d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:39:40.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.370+0000 7f10f6800640 1 --2- 172.21.15.90:0/806908912 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ecbaeed230 0x55ecbaf05040 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f10e00095b0 tx=0x7f10e0043000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5e76fa12305d2858 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:39:40.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.371+0000 7f10e6600640 1 -- 172.21.15.90:0/806908912 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10e0042040 con 0x55ecbaeed230 2024-04-02T14:39:40.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.371+0000 7f10e6600640 1 -- 172.21.15.90:0/806908912 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f10e004e030 con 0x55ecbaeed230 2024-04-02T14:39:40.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.371+0000 7f10e6600640 1 -- 172.21.15.90:0/806908912 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10e0056d60 con 0x55ecbaeed230 2024-04-02T14:39:40.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.373+0000 7f10ff9a8740 1 -- 172.21.15.90:0/806908912 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ecbaeed230 msgr2=0x55ecbaf05040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:40.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.373+0000 7f10ff9a8740 1 --2- 172.21.15.90:0/806908912 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ecbaeed230 0x55ecbaf05040 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f10e00095b0 tx=0x7f10e0043000 comp rx=0 tx=0).stop 2024-04-02T14:39:40.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.373+0000 7f10ff9a8740 1 -- 172.21.15.90:0/806908912 shutdown_connections 2024-04-02T14:39:40.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.373+0000 7f10ff9a8740 1 --2- 172.21.15.90:0/806908912 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ecbaeed230 0x55ecbaf05040 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:40.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.373+0000 7f10ff9a8740 1 --2- 172.21.15.90:0/806908912 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ecbaee4ba0 0x55ecbaef8d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:40.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.373+0000 7f10ff9a8740 1 --2- 172.21.15.90:0/806908912 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ecbaee3ec0 0x55ecbaeec880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:40.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.373+0000 7f10ff9a8740 1 -- 172.21.15.90:0/806908912 >> 172.21.15.90:0/806908912 conn(0x55ecbaeaaa50 msgr2=0x55ecbaede440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:39:40.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.373+0000 7f10ff9a8740 1 -- 172.21.15.90:0/806908912 shutdown_connections 2024-04-02T14:39:40.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.374+0000 7f10ff9a8740 1 -- 172.21.15.90:0/806908912 wait complete. 2024-04-02T14:39:40.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.374+0000 7f10ff9a8740 1 -- start start 2024-04-02T14:39:40.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.374+0000 7f10ff9a8740 1 Processor -- start 2024-04-02T14:39:40.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.375+0000 7f10ff9a8740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ecbaee3ec0 0x55ecbaef5220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:40.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.375+0000 7f10ff9a8740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ecbaee4ba0 0x55ecbaef5760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:40.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.375+0000 7f10ff9a8740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ecbaeed230 0x55ecbaee6440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:40.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.375+0000 7f10ff9a8740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f10f81b4e90 con 0x55ecbaee3ec0 2024-04-02T14:39:40.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.375+0000 7f10ff9a8740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f10f81b47c0 con 0x55ecbaee4ba0 2024-04-02T14:39:40.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.375+0000 7f10ff9a8740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f10f81b1a00 con 0x55ecbaeed230 2024-04-02T14:39:40.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.375+0000 7f10ff9a8740 10 client.0 _ll_register_callbacks cb 0x7f10f0098af0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:39:40.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.375+0000 7f10f5400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ecbaee4ba0 0x55ecbaef5760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:40.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.375+0000 7f10f5400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ecbaee4ba0 0x55ecbaef5760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:48470/0 (socket says 172.21.15.90:48470) 2024-04-02T14:39:40.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.375+0000 7f10f5400640 1 -- 172.21.15.90:0/4080587831 learned_addr learned my addr 172.21.15.90:0/4080587831 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:39:40.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.375+0000 7f10f5e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ecbaee3ec0 0x55ecbaef5220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:40.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.375+0000 7f10f6800640 1 --2- 172.21.15.90:0/4080587831 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ecbaeed230 0x55ecbaee6440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:40.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.376+0000 7f10f6800640 1 -- 172.21.15.90:0/4080587831 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ecbaee4ba0 msgr2=0x55ecbaef5760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:39:40.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.376+0000 7f10f6800640 1 --2- 172.21.15.90:0/4080587831 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ecbaee4ba0 0x55ecbaef5760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:40.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.376+0000 7f10f6800640 1 -- 172.21.15.90:0/4080587831 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ecbaee3ec0 msgr2=0x55ecbaef5220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:39:40.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.376+0000 7f10f6800640 1 --2- 172.21.15.90:0/4080587831 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ecbaee3ec0 0x55ecbaef5220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:40.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.376+0000 7f10f6800640 1 -- 172.21.15.90:0/4080587831 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ecbaee2ef0 con 0x55ecbaeed230 2024-04-02T14:39:40.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.376+0000 7f10f6800640 1 --2- 172.21.15.90:0/4080587831 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ecbaeed230 0x55ecbaee6440 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f10e0005940 tx=0x7f10e002bcd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d12eac1f5997124 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:39:40.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.376+0000 7f10db400640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:39:40.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.376+0000 7f10db400640 1 -- 172.21.15.90:0/4080587831 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10e0042040 con 0x55ecbaeed230 2024-04-02T14:39:40.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.377+0000 7f10ff9a8740 1 -- 172.21.15.90:0/4080587831 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f10e004e030 con 0x55ecbaeed230 2024-04-02T14:39:40.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.377+0000 7f10db400640 1 -- 172.21.15.90:0/4080587831 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f10e0005320 con 0x55ecbaeed230 2024-04-02T14:39:40.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.377+0000 7f10db400640 1 -- 172.21.15.90:0/4080587831 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f10e0004c80 con 0x55ecbaeed230 2024-04-02T14:39:40.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.377+0000 7f10ff9a8740 10 client.26785 fetch_fsmap learned FSMap version 27 2024-04-02T14:39:40.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.377+0000 7f10ff9a8740 1 -- 172.21.15.90:0/4080587831 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55ecbaf62ee0 con 0x55ecbaeed230 2024-04-02T14:39:40.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.378+0000 7f10db400640 1 -- 172.21.15.90:0/4080587831 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f10e0005560 con 0x55ecbaeed230 2024-04-02T14:39:40.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.378+0000 7f10db400640 1 -- 172.21.15.90:0/4080587831 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f10e0053050 con 0x55ecbaeed230 2024-04-02T14:39:40.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.378+0000 7f10ff9a8740 10 client.26785 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:39:40.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.378+0000 7f10ff9a8740 10 client.26785 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:39:40.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.378+0000 7f10ff9a8740 20 client.26785 populate_metadata read hostname 'smithi090' 2024-04-02T14:39:40.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.378+0000 7f10ff9a8740 1 --2- 172.21.15.90:0/4080587831 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55ecbaf63110 0x55ecbaf65500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:40.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.378+0000 7f10ff9a8740 4 client.26785 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-02T14:39:40.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.379+0000 7f10ff9a8740 1 -- 172.21.15.90:0/4080587831 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7f10e004e030 con 0x55ecbaf63110 2024-04-02T14:39:40.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.379+0000 7f10f5e00640 1 --2- 172.21.15.90:0/4080587831 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55ecbaf63110 0x55ecbaf65500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:40.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.380+0000 7f10f5e00640 1 --2- 172.21.15.90:0/4080587831 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55ecbaf63110 0x55ecbaf65500 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bcc8c3034dd5f859 server_cookie=4ee47605b670459c in_seq=0 out_seq=0 2024-04-02T14:39:40.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.380+0000 7f10db400640 10 client.26785 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:39:40.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.380+0000 7f10db400640 1 -- 172.21.15.90:0/4080587831 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7f10e004e030 con 0x55ecbaf63110 2024-04-02T14:39:40.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.380+0000 7f10db400640 10 client.26785 handle_command_reply: tid=0 2024-04-02T14:39:40.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.380+0000 7f10ff9a8740 1 client.26785 shutdown 2024-04-02T14:39:40.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.380+0000 7f10ff9a8740 1 -- 172.21.15.90:0/4080587831 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ecbaeed230 msgr2=0x55ecbaee6440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:40.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.380+0000 7f10ff9a8740 1 --2- 172.21.15.90:0/4080587831 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ecbaeed230 0x55ecbaee6440 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f10e0005940 tx=0x7f10e002bcd0 comp rx=0 tx=0).stop 2024-04-02T14:39:40.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.381+0000 7f10ff9a8740 1 -- 172.21.15.90:0/4080587831 shutdown_connections 2024-04-02T14:39:40.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.381+0000 7f10ff9a8740 1 --2- 172.21.15.90:0/4080587831 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55ecbaf63110 0x55ecbaf65500 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:40.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.381+0000 7f10db400640 0 client.26785 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:39:40.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.381+0000 7f10ff9a8740 1 --2- 172.21.15.90:0/4080587831 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ecbaeed230 0x55ecbaee6440 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:40.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.381+0000 7f10ff9a8740 1 --2- 172.21.15.90:0/4080587831 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ecbaee4ba0 0x55ecbaef5760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:40.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.381+0000 7f10ff9a8740 1 --2- 172.21.15.90:0/4080587831 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ecbaee3ec0 0x55ecbaef5220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:40.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.381+0000 7f10ff9a8740 1 -- 172.21.15.90:0/4080587831 >> 172.21.15.90:0/4080587831 conn(0x55ecbaeaaa50 msgr2=0x55ecbaee8960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:39:40.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.382+0000 7f10ff9a8740 1 -- 172.21.15.90:0/4080587831 shutdown_connections 2024-04-02T14:39:40.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.382+0000 7f10ff9a8740 1 -- 172.21.15.90:0/4080587831 wait complete. 2024-04-02T14:39:40.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.382+0000 7f10ff9a8740 20 client.26785 trim_cache size 0 max 16384 2024-04-02T14:39:40.385 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-02T14:39:40.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.383+0000 7f10f7200640 1 -- 172.21.15.90:0/2570044433 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f10dc0a3da0 msgr2=0x7f10dc0a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:40.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.383+0000 7f10f7200640 1 --2- 172.21.15.90:0/2570044433 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f10dc0a3da0 0x7f10dc0a6260 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f10e002f020 tx=0x7f10e0040040 comp rx=0 tx=0).stop 2024-04-02T14:39:40.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.383+0000 7f10f7200640 1 -- 172.21.15.90:0/2570044433 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f10f8073ea0 msgr2=0x7f10f819e6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:40.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.383+0000 7f10f7200640 1 --2- 172.21.15.90:0/2570044433 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f10f8073ea0 0x7f10f819e6e0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f10f000c970 tx=0x7f10f000ce30 comp rx=0 tx=0).stop 2024-04-02T14:39:40.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.384+0000 7f10f7200640 1 -- 172.21.15.90:0/2570044433 shutdown_connections 2024-04-02T14:39:40.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.384+0000 7f10f7200640 1 --2- 172.21.15.90:0/2570044433 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f10dc0a3da0 0x7f10dc0a6260 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:40.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.384+0000 7f10f7200640 1 --2- 172.21.15.90:0/2570044433 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f10f8112980 0x7f10f81a2fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:40.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.384+0000 7f10f7200640 1 --2- 172.21.15.90:0/2570044433 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f10f80747c0 0x7f10f819ec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:40.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.384+0000 7f10f7200640 1 --2- 172.21.15.90:0/2570044433 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f10f8073ea0 0x7f10f819e6e0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:40.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.384+0000 7f10f7200640 1 -- 172.21.15.90:0/2570044433 >> 172.21.15.90:0/2570044433 conn(0x7f10f80fd4a0 msgr2=0x7f10f80fe9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:39:40.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.384+0000 7f10f7200640 1 -- 172.21.15.90:0/2570044433 shutdown_connections 2024-04-02T14:39:40.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:40.385+0000 7f10f7200640 1 -- 172.21.15.90:0/2570044433 wait complete. 2024-04-02T14:39:40.406 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:39:40 smithi090 ceph-mon[31674]: pgmap v146: 97 pgs: 97 active+clean; 3.3 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 175 KiB/s rd, 40 MiB/s wr, 2.02k op/s 2024-04-02T14:39:40.436 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:39:40 smithi177 ceph-mon[36431]: pgmap v146: 97 pgs: 97 active+clean; 3.3 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 175 KiB/s rd, 40 MiB/s wr, 2.02k op/s 2024-04-02T14:39:41.410 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-02T14:39:41.410 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T14:39:41.410 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-02T14:39:41.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.685+0000 7fafd4e00640 1 -- 172.21.15.90:0/470253868 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fafd0103f10 msgr2=0x7fafd01042f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:41.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.685+0000 7fafd4e00640 1 --2- 172.21.15.90:0/470253868 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fafd0103f10 0x7fafd01042f0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fafc0009900 tx=0x7fafc002bc30 comp rx=0 tx=0).stop 2024-04-02T14:39:41.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.685+0000 7fafd4e00640 1 -- 172.21.15.90:0/470253868 shutdown_connections 2024-04-02T14:39:41.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.685+0000 7fafd4e00640 1 --2- 172.21.15.90:0/470253868 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fafd0108fe0 0x7fafd0110bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:41.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.685+0000 7fafd4e00640 1 --2- 172.21.15.90:0/470253868 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fafd01048c0 0x7fafd0108900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:41.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.685+0000 7fafd4e00640 1 --2- 172.21.15.90:0/470253868 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fafd0103f10 0x7fafd01042f0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:41.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.685+0000 7fafd4e00640 1 -- 172.21.15.90:0/470253868 >> 172.21.15.90:0/470253868 conn(0x7fafd00ff770 msgr2=0x7fafd0101b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:39:41.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.685+0000 7fafd4e00640 1 -- 172.21.15.90:0/470253868 shutdown_connections 2024-04-02T14:39:41.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.685+0000 7fafd4e00640 1 -- 172.21.15.90:0/470253868 wait complete. 2024-04-02T14:39:41.690 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.686+0000 7fafd4e00640 1 Processor -- start 2024-04-02T14:39:41.690 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.687+0000 7fafd4e00640 1 -- start start 2024-04-02T14:39:41.690 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.687+0000 7fafd4e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fafd0103f10 0x7fafd0110a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:41.690 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.687+0000 7fafd4e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fafd0108fe0 0x7fafd010dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:41.690 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.687+0000 7fafd4e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fafd010e110 0x7fafd010e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:41.690 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.688+0000 7fafd4e00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fafd01140d0 con 0x7fafd0108fe0 2024-04-02T14:39:41.690 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.688+0000 7fafd4e00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fafd0113f50 con 0x7fafd0103f10 2024-04-02T14:39:41.690 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.688+0000 7fafd4e00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fafd0114250 con 0x7fafd010e110 2024-04-02T14:39:41.691 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.688+0000 7fafcd400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fafd0108fe0 0x7fafd010dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:41.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.688+0000 7fafce800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fafd010e110 0x7fafd010e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:41.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.688+0000 7fafce800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fafd010e110 0x7fafd010e5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:44728/0 (socket says 172.21.15.90:44728) 2024-04-02T14:39:41.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.688+0000 7fafce800640 1 -- 172.21.15.90:0/4005327843 learned_addr learned my addr 172.21.15.90:0/4005327843 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:39:41.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.688+0000 7fafcde00640 1 --2- 172.21.15.90:0/4005327843 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fafd0103f10 0x7fafd0110a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:41.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.688+0000 7fafce800640 1 -- 172.21.15.90:0/4005327843 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fafd0103f10 msgr2=0x7fafd0110a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:41.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.688+0000 7fafce800640 1 --2- 172.21.15.90:0/4005327843 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fafd0103f10 0x7fafd0110a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:41.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.688+0000 7fafce800640 1 -- 172.21.15.90:0/4005327843 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fafd0108fe0 msgr2=0x7fafd010dbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:41.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.688+0000 7fafce800640 1 --2- 172.21.15.90:0/4005327843 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fafd0108fe0 0x7fafd010dbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:41.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.689+0000 7fafce800640 1 -- 172.21.15.90:0/4005327843 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fafd01ab050 con 0x7fafd010e110 2024-04-02T14:39:41.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.689+0000 7fafce800640 1 --2- 172.21.15.90:0/4005327843 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fafd010e110 0x7fafd010e5a0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fafc800eb10 tx=0x7fafc800c2d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:39:41.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.689+0000 7fafc6a00640 1 -- 172.21.15.90:0/4005327843 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafc8004b20 con 0x7fafd010e110 2024-04-02T14:39:41.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.689+0000 7fafc6a00640 1 -- 172.21.15.90:0/4005327843 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fafc8004cc0 con 0x7fafd010e110 2024-04-02T14:39:41.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.689+0000 7fafc6a00640 1 -- 172.21.15.90:0/4005327843 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafc8010760 con 0x7fafd010e110 2024-04-02T14:39:41.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.690+0000 7fafd4e00640 1 -- 172.21.15.90:0/4005327843 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fafd01ab340 con 0x7fafd010e110 2024-04-02T14:39:41.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.690+0000 7fafd4e00640 1 -- 172.21.15.90:0/4005327843 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fafd01ab840 con 0x7fafd010e110 2024-04-02T14:39:41.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.697+0000 7fafd7b86740 1 Processor -- start 2024-04-02T14:39:41.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.698+0000 7fafc6a00640 1 -- 172.21.15.90:0/4005327843 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fafc8005020 con 0x7fafd010e110 2024-04-02T14:39:41.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.698+0000 7fafd7b86740 1 -- start start 2024-04-02T14:39:41.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.698+0000 7fafd7b86740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555895381030 0x55589537fd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:41.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.698+0000 7fafd7b86740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555895380350 0x5558953807d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:41.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.698+0000 7fafd7b86740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5558953896c0 0x55589538bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:41.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.698+0000 7fafd7b86740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5558952e4310 con 0x555895380350 2024-04-02T14:39:41.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.698+0000 7fafd7b86740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55589528c690 con 0x5558953896c0 2024-04-02T14:39:41.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.698+0000 7fafd7b86740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5558952e02c0 con 0x555895381030 2024-04-02T14:39:41.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.699+0000 7fafcd400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555895381030 0x55589537fd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:41.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.699+0000 7fafcd400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555895381030 0x55589537fd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:44742/0 (socket says 172.21.15.90:44742) 2024-04-02T14:39:41.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.699+0000 7fafcd400640 1 -- 172.21.15.90:0/3903066735 learned_addr learned my addr 172.21.15.90:0/3903066735 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:39:41.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.699+0000 7fafcde00640 1 --2- 172.21.15.90:0/3903066735 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555895380350 0x5558953807d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:41.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.699+0000 7fafcd400640 1 --2- 172.21.15.90:0/3903066735 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5558953896c0 0x55589538bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:41.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.699+0000 7fafcd400640 1 -- 172.21.15.90:0/3903066735 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555895381030 msgr2=0x55589537fd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:39:41.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.699+0000 7fafcd400640 1 --2- 172.21.15.90:0/3903066735 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555895381030 0x55589537fd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:41.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.699+0000 7fafcd400640 1 -- 172.21.15.90:0/3903066735 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555895380350 msgr2=0x5558953807d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:39:41.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.699+0000 7fafcd400640 1 --2- 172.21.15.90:0/3903066735 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555895380350 0x5558953807d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:41.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.699+0000 7fafcd400640 1 -- 172.21.15.90:0/3903066735 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5558950c07e0 con 0x5558953896c0 2024-04-02T14:39:41.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.699+0000 7fafcde00640 1 --2- 172.21.15.90:0/3903066735 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555895380350 0x5558953807d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:39:41.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.700+0000 7fafc6a00640 1 --2- 172.21.15.90:0/4005327843 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fafb40a3e70 0x7fafb40a6330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:41.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.700+0000 7fafcd400640 1 --2- 172.21.15.90:0/3903066735 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5558953896c0 0x55589538bab0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fafbc011710 tx=0x7fafbc005f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6403a42ec612efc6 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:39:41.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.700+0000 7fafbb400640 1 -- 172.21.15.90:0/3903066735 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafbc011d50 con 0x5558953896c0 2024-04-02T14:39:41.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.700+0000 7fafbb400640 1 -- 172.21.15.90:0/3903066735 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fafbc00fd20 con 0x5558953896c0 2024-04-02T14:39:41.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.700+0000 7fafbb400640 1 -- 172.21.15.90:0/3903066735 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafbc004a00 con 0x5558953896c0 2024-04-02T14:39:41.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.700+0000 7fafcde00640 1 --2- 172.21.15.90:0/4005327843 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fafb40a3e70 0x7fafb40a6330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:41.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.700+0000 7fafc6a00640 1 -- 172.21.15.90:0/4005327843 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7fafc80d0cd0 con 0x7fafd010e110 2024-04-02T14:39:41.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.701+0000 7fafd7b86740 1 -- 172.21.15.90:0/3903066735 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5558953896c0 msgr2=0x55589538bab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:41.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.701+0000 7fafd7b86740 1 --2- 172.21.15.90:0/3903066735 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5558953896c0 0x55589538bab0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fafbc011710 tx=0x7fafbc005f90 comp rx=0 tx=0).stop 2024-04-02T14:39:41.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.702+0000 7fafd7b86740 1 -- 172.21.15.90:0/3903066735 shutdown_connections 2024-04-02T14:39:41.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.702+0000 7fafd7b86740 1 --2- 172.21.15.90:0/3903066735 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5558953896c0 0x55589538bab0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:41.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.702+0000 7fafd7b86740 1 --2- 172.21.15.90:0/3903066735 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555895380350 0x5558953807d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:41.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.702+0000 7fafd7b86740 1 --2- 172.21.15.90:0/3903066735 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555895381030 0x55589537fd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:41.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.702+0000 7fafd7b86740 1 -- 172.21.15.90:0/3903066735 >> 172.21.15.90:0/3903066735 conn(0x555895314270 msgr2=0x55589537ada0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:39:41.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.703+0000 7fafd7b86740 1 -- 172.21.15.90:0/3903066735 shutdown_connections 2024-04-02T14:39:41.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.703+0000 7fafd7b86740 1 -- 172.21.15.90:0/3903066735 wait complete. 2024-04-02T14:39:41.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.703+0000 7fafcde00640 1 --2- 172.21.15.90:0/4005327843 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fafb40a3e70 0x7fafb40a6330 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fafc0006010 tx=0x7fafc002df90 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:39:41.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.704+0000 7fafd7b86740 1 -- start start 2024-04-02T14:39:41.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.704+0000 7fafd7b86740 1 Processor -- start 2024-04-02T14:39:41.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.705+0000 7fafd7b86740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555895380350 0x5558953866b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:41.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.705+0000 7fafd7b86740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555895381030 0x555895386bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:41.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.705+0000 7fafd7b86740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5558953896c0 0x555895387130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:41.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.705+0000 7fafd7b86740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55589538c530 con 0x555895380350 2024-04-02T14:39:41.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.705+0000 7fafd7b86740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55589529fa60 con 0x5558953896c0 2024-04-02T14:39:41.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.705+0000 7fafcde00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555895380350 0x5558953866b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:41.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.705+0000 7fafcd400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555895381030 0x555895386bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:41.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.705+0000 7fafd7b86740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55589538c6b0 con 0x555895381030 2024-04-02T14:39:41.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.705+0000 7fafcd400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555895381030 0x555895386bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:44754/0 (socket says 172.21.15.90:44754) 2024-04-02T14:39:41.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.705+0000 7fafcd400640 1 -- 172.21.15.90:0/2106349798 learned_addr learned my addr 172.21.15.90:0/2106349798 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:39:41.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.705+0000 7fafce800640 1 --2- 172.21.15.90:0/2106349798 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5558953896c0 0x555895387130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:41.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.706+0000 7fafd7b86740 10 client.0 _ll_register_callbacks cb 0x5558952a5030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:39:41.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.706+0000 7fafcd400640 1 -- 172.21.15.90:0/2106349798 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5558953896c0 msgr2=0x555895387130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:39:41.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.706+0000 7fafcd400640 1 --2- 172.21.15.90:0/2106349798 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5558953896c0 0x555895387130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:41.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.706+0000 7fafcd400640 1 -- 172.21.15.90:0/2106349798 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555895380350 msgr2=0x5558953866b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:39:41.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.707+0000 7fafcd400640 1 --2- 172.21.15.90:0/2106349798 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555895380350 0x5558953866b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:41.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.707+0000 7fafcd400640 1 -- 172.21.15.90:0/2106349798 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555895120f70 con 0x555895381030 2024-04-02T14:39:41.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.707+0000 7fafcde00640 1 --2- 172.21.15.90:0/2106349798 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555895380350 0x5558953866b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:39:41.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.707+0000 7fafcd400640 1 --2- 172.21.15.90:0/2106349798 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555895381030 0x555895386bf0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fafbc005bc0 tx=0x7fafbc009b90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7b6275070c5d855 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:39:41.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.707+0000 7fafafe00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:39:41.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.708+0000 7fafafe00640 1 -- 172.21.15.90:0/2106349798 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafbc011270 con 0x555895381030 2024-04-02T14:39:41.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.708+0000 7fafafe00640 1 -- 172.21.15.90:0/2106349798 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7fafbc004030 con 0x555895381030 2024-04-02T14:39:41.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.708+0000 7fafd7b86740 1 -- 172.21.15.90:0/2106349798 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fafbc00fd20 con 0x555895381030 2024-04-02T14:39:41.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.708+0000 7fafafe00640 1 -- 172.21.15.90:0/2106349798 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fafbc004230 con 0x555895381030 2024-04-02T14:39:41.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.708+0000 7fafd7b86740 10 client.26797 fetch_fsmap learned FSMap version 27 2024-04-02T14:39:41.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.708+0000 7fafd7b86740 1 -- 172.21.15.90:0/2106349798 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x555894fb8170 con 0x555895381030 2024-04-02T14:39:41.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.708+0000 7fafafe00640 1 -- 172.21.15.90:0/2106349798 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fafbc00fd20 con 0x555895381030 2024-04-02T14:39:41.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.709+0000 7fafafe00640 1 -- 172.21.15.90:0/2106349798 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fafbc0035f0 con 0x555895381030 2024-04-02T14:39:41.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.709+0000 7fafd7b86740 10 client.26797 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:39:41.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.709+0000 7fafd7b86740 10 client.26797 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:39:41.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.709+0000 7fafd7b86740 20 client.26797 populate_metadata read hostname 'smithi090' 2024-04-02T14:39:41.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.709+0000 7fafd7b86740 1 --2- 172.21.15.90:0/2106349798 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5558953999e0 0x55589539bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:41.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.709+0000 7fafd7b86740 4 client.26797 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:39:41.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.709+0000 7fafd7b86740 1 -- 172.21.15.90:0/2106349798 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5558952f1850 con 0x5558953999e0 2024-04-02T14:39:41.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.709+0000 7fafcde00640 1 --2- 172.21.15.90:0/2106349798 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5558953999e0 0x55589539bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:41.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.710+0000 7fafafe00640 10 client.26797 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:39:41.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.710+0000 7fafcde00640 1 --2- 172.21.15.90:0/2106349798 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5558953999e0 0x55589539bdd0 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3bce1c3d420862ec server_cookie=dfdacf8c2d785a39 in_seq=0 out_seq=0 2024-04-02T14:39:41.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.714+0000 7fafafe00640 1 -- 172.21.15.90:0/2106349798 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5558952f1850 con 0x5558953999e0 2024-04-02T14:39:41.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.714+0000 7fafafe00640 10 client.26797 handle_command_reply: tid=0 2024-04-02T14:39:41.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.714+0000 7fafd7b86740 1 client.26797 shutdown 2024-04-02T14:39:41.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.715+0000 7fafd7b86740 1 -- 172.21.15.90:0/2106349798 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555895381030 msgr2=0x555895386bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:41.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.715+0000 7fafd7b86740 1 --2- 172.21.15.90:0/2106349798 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555895381030 0x555895386bf0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fafbc005bc0 tx=0x7fafbc009b90 comp rx=0 tx=0).stop 2024-04-02T14:39:41.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.715+0000 7fafd7b86740 1 -- 172.21.15.90:0/2106349798 shutdown_connections 2024-04-02T14:39:41.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.715+0000 7fafd7b86740 1 --2- 172.21.15.90:0/2106349798 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5558953999e0 0x55589539bdd0 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:41.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.715+0000 7fafd7b86740 1 --2- 172.21.15.90:0/2106349798 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5558953896c0 0x555895387130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:41.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.715+0000 7fafd7b86740 1 --2- 172.21.15.90:0/2106349798 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555895381030 0x555895386bf0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:41.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.715+0000 7fafd7b86740 1 --2- 172.21.15.90:0/2106349798 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555895380350 0x5558953866b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:41.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.715+0000 7fafd7b86740 1 -- 172.21.15.90:0/2106349798 >> 172.21.15.90:0/2106349798 conn(0x555895314270 msgr2=0x55589537caf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:39:41.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.715+0000 7fafafe00640 0 client.26797 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:39:41.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.715+0000 7fafd7b86740 1 -- 172.21.15.90:0/2106349798 shutdown_connections 2024-04-02T14:39:41.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.715+0000 7fafd7b86740 1 -- 172.21.15.90:0/2106349798 wait complete. 2024-04-02T14:39:41.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.716+0000 7fafd7b86740 20 client.26797 trim_cache size 0 max 16384 2024-04-02T14:39:41.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.760+0000 7fafd7b86740 1 Processor -- start 2024-04-02T14:39:41.764 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.760+0000 7fafd7b86740 1 -- start start 2024-04-02T14:39:41.764 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.761+0000 7fafd7b86740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555895380350 0x555895395430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:41.764 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.761+0000 7fafd7b86740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555895381030 0x5558953a1480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:41.764 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.761+0000 7fafd7b86740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5558953896c0 0x5558953a19c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:41.764 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.761+0000 7fafd7b86740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fafd01b3e80 con 0x555895380350 2024-04-02T14:39:41.764 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.761+0000 7fafd7b86740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fafd01b59c0 con 0x5558953896c0 2024-04-02T14:39:41.764 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.761+0000 7fafd7b86740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fafd01b4550 con 0x555895381030 2024-04-02T14:39:41.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.762+0000 7fafcd400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555895381030 0x5558953a1480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:41.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.762+0000 7fafce800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5558953896c0 0x5558953a19c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:41.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.762+0000 7fafcd400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555895381030 0x5558953a1480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:44770/0 (socket says 172.21.15.90:44770) 2024-04-02T14:39:41.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.762+0000 7fafcd400640 1 -- 172.21.15.90:0/3235006292 learned_addr learned my addr 172.21.15.90:0/3235006292 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:39:41.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.762+0000 7fafcde00640 1 --2- 172.21.15.90:0/3235006292 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555895380350 0x555895395430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:41.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.763+0000 7fafce800640 1 -- 172.21.15.90:0/3235006292 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555895381030 msgr2=0x5558953a1480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:39:41.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.763+0000 7fafce800640 1 --2- 172.21.15.90:0/3235006292 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555895381030 0x5558953a1480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:41.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.763+0000 7fafce800640 1 -- 172.21.15.90:0/3235006292 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555895380350 msgr2=0x555895395430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:39:41.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.763+0000 7fafce800640 1 --2- 172.21.15.90:0/3235006292 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555895380350 0x555895395430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:41.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.763+0000 7fafce800640 1 -- 172.21.15.90:0/3235006292 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55589537f380 con 0x5558953896c0 2024-04-02T14:39:41.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.763+0000 7fafce800640 1 --2- 172.21.15.90:0/3235006292 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5558953896c0 0x5558953a19c0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fafc80ce430 tx=0x7fafc809cb10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fc785f6d13a47343 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:39:41.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.764+0000 7fafbaa00640 1 -- 172.21.15.90:0/3235006292 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafc809cc80 con 0x5558953896c0 2024-04-02T14:39:41.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.764+0000 7fafbaa00640 1 -- 172.21.15.90:0/3235006292 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fafc80a1030 con 0x5558953896c0 2024-04-02T14:39:41.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.764+0000 7fafbaa00640 1 -- 172.21.15.90:0/3235006292 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafc80bd0c0 con 0x5558953896c0 2024-04-02T14:39:41.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.765+0000 7fafd7b86740 1 -- 172.21.15.90:0/3235006292 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5558953896c0 msgr2=0x5558953a19c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:41.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.766+0000 7fafd7b86740 1 --2- 172.21.15.90:0/3235006292 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5558953896c0 0x5558953a19c0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fafc80ce430 tx=0x7fafc809cb10 comp rx=0 tx=0).stop 2024-04-02T14:39:41.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.766+0000 7fafd7b86740 1 -- 172.21.15.90:0/3235006292 shutdown_connections 2024-04-02T14:39:41.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.766+0000 7fafd7b86740 1 --2- 172.21.15.90:0/3235006292 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5558953896c0 0x5558953a19c0 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:41.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.766+0000 7fafd7b86740 1 --2- 172.21.15.90:0/3235006292 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555895381030 0x5558953a1480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:41.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.766+0000 7fafd7b86740 1 --2- 172.21.15.90:0/3235006292 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555895380350 0x555895395430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:41.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.766+0000 7fafd7b86740 1 -- 172.21.15.90:0/3235006292 >> 172.21.15.90:0/3235006292 conn(0x555895314270 msgr2=0x555895379590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:39:41.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.766+0000 7fafd7b86740 1 -- 172.21.15.90:0/3235006292 shutdown_connections 2024-04-02T14:39:41.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.766+0000 7fafd7b86740 1 -- 172.21.15.90:0/3235006292 wait complete. 2024-04-02T14:39:41.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.767+0000 7fafd7b86740 1 -- start start 2024-04-02T14:39:41.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.767+0000 7fafd7b86740 1 Processor -- start 2024-04-02T14:39:41.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.768+0000 7fafd7b86740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555895380350 0x555895391e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:41.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.768+0000 7fafd7b86740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555895381030 0x555895382920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:41.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.768+0000 7fafd7b86740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5558953896c0 0x555895382ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:41.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.768+0000 7fafd7b86740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fafd01b52f0 con 0x5558953896c0 2024-04-02T14:39:41.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.768+0000 7fafd7b86740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fafd01b4c20 con 0x555895381030 2024-04-02T14:39:41.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.768+0000 7fafd7b86740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fafd01b1e60 con 0x555895380350 2024-04-02T14:39:41.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.768+0000 7fafd7b86740 10 client.0 _ll_register_callbacks cb 0x7fafc80093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:39:41.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.769+0000 7fafcde00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555895380350 0x555895391e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:41.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.769+0000 7fafcd400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555895381030 0x555895382920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:41.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.769+0000 7fafcd400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555895381030 0x555895382920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:48524/0 (socket says 172.21.15.90:48524) 2024-04-02T14:39:41.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.769+0000 7fafcde00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555895380350 0x555895391e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:44778/0 (socket says 172.21.15.90:44778) 2024-04-02T14:39:41.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.769+0000 7fafcd400640 1 -- 172.21.15.90:0/505818115 learned_addr learned my addr 172.21.15.90:0/505818115 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:39:41.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.769+0000 7fafce800640 1 --2- 172.21.15.90:0/505818115 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5558953896c0 0x555895382ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:41.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.769+0000 7fafcd400640 1 -- 172.21.15.90:0/505818115 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555895380350 msgr2=0x555895391e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:39:41.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.769+0000 7fafcd400640 1 --2- 172.21.15.90:0/505818115 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555895380350 0x555895391e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:41.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.769+0000 7fafcd400640 1 -- 172.21.15.90:0/505818115 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5558953896c0 msgr2=0x555895382ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:39:41.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.769+0000 7fafcd400640 1 --2- 172.21.15.90:0/505818115 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5558953896c0 0x555895382ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:41.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.769+0000 7fafcd400640 1 -- 172.21.15.90:0/505818115 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5558953836f0 con 0x555895381030 2024-04-02T14:39:41.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.770+0000 7fafafe00640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:39:41.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.770+0000 7fafcd400640 1 --2- 172.21.15.90:0/505818115 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555895381030 0x555895382920 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fafbc005250 tx=0x7fafbc003b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=df36a64afce7e821 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:39:41.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.770+0000 7fafafe00640 1 -- 172.21.15.90:0/505818115 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafbc005370 con 0x555895381030 2024-04-02T14:39:41.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.770+0000 7fafafe00640 1 -- 172.21.15.90:0/505818115 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7fafbc003f70 con 0x555895381030 2024-04-02T14:39:41.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.770+0000 7fafd7b86740 1 -- 172.21.15.90:0/505818115 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fafc80a1030 con 0x555895381030 2024-04-02T14:39:41.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.772+0000 7fafafe00640 1 -- 172.21.15.90:0/505818115 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fafbc041050 con 0x555895381030 2024-04-02T14:39:41.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.772+0000 7fafd7b86740 10 client.26927 fetch_fsmap learned FSMap version 27 2024-04-02T14:39:41.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.772+0000 7fafd7b86740 1 -- 172.21.15.90:0/505818115 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x555895411dd0 con 0x555895381030 2024-04-02T14:39:41.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.772+0000 7fafafe00640 1 -- 172.21.15.90:0/505818115 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fafc80a1030 con 0x555895381030 2024-04-02T14:39:41.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.773+0000 7fafafe00640 1 -- 172.21.15.90:0/505818115 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fafbc044020 con 0x555895381030 2024-04-02T14:39:41.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.773+0000 7fafd7b86740 10 client.26927 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:39:41.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.773+0000 7fafd7b86740 10 client.26927 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:39:41.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.773+0000 7fafd7b86740 20 client.26927 populate_metadata read hostname 'smithi090' 2024-04-02T14:39:41.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.773+0000 7fafd7b86740 1 --2- 172.21.15.90:0/505818115 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x555895412430 0x555895414850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:41.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.773+0000 7fafd7b86740 4 client.26927 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-02T14:39:41.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.773+0000 7fafd7b86740 1 -- 172.21.15.90:0/505818115 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5558952ebba0 con 0x555895412430 2024-04-02T14:39:41.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.774+0000 7fafcde00640 1 --2- 172.21.15.90:0/505818115 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x555895412430 0x555895414850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:41.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.774+0000 7fafcde00640 1 --2- 172.21.15.90:0/505818115 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x555895412430 0x555895414850 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=49a293e0faac342b server_cookie=1e1789213dfa28b4 in_seq=0 out_seq=0 2024-04-02T14:39:41.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.774+0000 7fafafe00640 10 client.26927 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:39:41.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.775+0000 7fafafe00640 1 -- 172.21.15.90:0/505818115 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5558952ebba0 con 0x555895412430 2024-04-02T14:39:41.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.775+0000 7fafafe00640 10 client.26927 handle_command_reply: tid=0 2024-04-02T14:39:41.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.775+0000 7fafd7b86740 1 client.26927 shutdown 2024-04-02T14:39:41.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.776+0000 7fafd7b86740 1 -- 172.21.15.90:0/505818115 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555895381030 msgr2=0x555895382920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:41.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.776+0000 7fafd7b86740 1 --2- 172.21.15.90:0/505818115 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555895381030 0x555895382920 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fafbc005250 tx=0x7fafbc003b80 comp rx=0 tx=0).stop 2024-04-02T14:39:41.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.776+0000 7fafd7b86740 1 -- 172.21.15.90:0/505818115 shutdown_connections 2024-04-02T14:39:41.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.776+0000 7fafd7b86740 1 --2- 172.21.15.90:0/505818115 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x555895412430 0x555895414850 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:41.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.776+0000 7fafd7b86740 1 --2- 172.21.15.90:0/505818115 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5558953896c0 0x555895382ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:41.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.776+0000 7fafafe00640 0 client.26927 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:39:41.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.776+0000 7fafd7b86740 1 --2- 172.21.15.90:0/505818115 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555895381030 0x555895382920 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:41.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.776+0000 7fafd7b86740 1 --2- 172.21.15.90:0/505818115 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555895380350 0x555895391e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:41.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.776+0000 7fafd7b86740 1 -- 172.21.15.90:0/505818115 >> 172.21.15.90:0/505818115 conn(0x555895314270 msgr2=0x555895379e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:39:41.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.777+0000 7fafd7b86740 1 -- 172.21.15.90:0/505818115 shutdown_connections 2024-04-02T14:39:41.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.777+0000 7fafd7b86740 1 -- 172.21.15.90:0/505818115 wait complete. 2024-04-02T14:39:41.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.778+0000 7fafd7b86740 20 client.26927 trim_cache size 0 max 16384 2024-04-02T14:39:41.780 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-02T14:39:41.781 INFO:teuthology.orchestra.run.smithi090.stdout: "return_code": 0, 2024-04-02T14:39:41.781 INFO:teuthology.orchestra.run.smithi090.stdout: "scrub_tag": "77fb932f-0f67-4c0a-b524-e1eb04fb36d6", 2024-04-02T14:39:41.781 INFO:teuthology.orchestra.run.smithi090.stdout: "mode": "asynchronous" 2024-04-02T14:39:41.781 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-02T14:39:41.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.779+0000 7fafd4e00640 1 -- 172.21.15.90:0/4005327843 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fafb40a3e70 msgr2=0x7fafb40a6330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:41.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.779+0000 7fafd4e00640 1 --2- 172.21.15.90:0/4005327843 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fafb40a3e70 0x7fafb40a6330 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fafc0006010 tx=0x7fafc002df90 comp rx=0 tx=0).stop 2024-04-02T14:39:41.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.779+0000 7fafd4e00640 1 -- 172.21.15.90:0/4005327843 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fafd010e110 msgr2=0x7fafd010e5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:41.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.779+0000 7fafd4e00640 1 --2- 172.21.15.90:0/4005327843 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fafd010e110 0x7fafd010e5a0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fafc800eb10 tx=0x7fafc800c2d0 comp rx=0 tx=0).stop 2024-04-02T14:39:41.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.779+0000 7fafd4e00640 1 -- 172.21.15.90:0/4005327843 shutdown_connections 2024-04-02T14:39:41.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.779+0000 7fafd4e00640 1 --2- 172.21.15.90:0/4005327843 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fafb40a3e70 0x7fafb40a6330 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:41.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.779+0000 7fafd4e00640 1 --2- 172.21.15.90:0/4005327843 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fafd010e110 0x7fafd010e5a0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:41.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.779+0000 7fafd4e00640 1 --2- 172.21.15.90:0/4005327843 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fafd0108fe0 0x7fafd010dbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:41.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.779+0000 7fafd4e00640 1 --2- 172.21.15.90:0/4005327843 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fafd0103f10 0x7fafd0110a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:41.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.779+0000 7fafd4e00640 1 -- 172.21.15.90:0/4005327843 >> 172.21.15.90:0/4005327843 conn(0x7fafd00ff770 msgr2=0x7fafd0100b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:39:41.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.780+0000 7fafd4e00640 1 -- 172.21.15.90:0/4005327843 shutdown_connections 2024-04-02T14:39:41.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:41.780+0000 7fafd4e00640 1 -- 172.21.15.90:0/4005327843 wait complete. 2024-04-02T14:39:41.806 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-02T14:39:41.806 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-02T14:39:41.832 INFO:teuthology.orchestra.run.smithi090.stderr:progress: 250.0 s, 2339.4 tps, lat 42.534 ms stddev 50.084 2024-04-02T14:39:42.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.072+0000 7f450e000640 1 -- 172.21.15.90:0/3999459367 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f45081047a0 msgr2=0x7f45081087e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:42.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.072+0000 7f450e000640 1 --2- 172.21.15.90:0/3999459367 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f45081047a0 0x7f45081087e0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f44f4009900 tx=0x7f44f402bc30 comp rx=0 tx=0).stop 2024-04-02T14:39:42.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.073+0000 7f450e000640 1 -- 172.21.15.90:0/3999459367 shutdown_connections 2024-04-02T14:39:42.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.073+0000 7f450e000640 1 --2- 172.21.15.90:0/3999459367 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4508108ec0 0x7f4508110a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:42.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.073+0000 7f450e000640 1 --2- 172.21.15.90:0/3999459367 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f45081047a0 0x7f45081087e0 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:42.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.073+0000 7f450e000640 1 --2- 172.21.15.90:0/3999459367 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4508103df0 0x7f45081041d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:42.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.073+0000 7f450e000640 1 -- 172.21.15.90:0/3999459367 >> 172.21.15.90:0/3999459367 conn(0x7f45080ff610 msgr2=0x7f4508101a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:39:42.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.073+0000 7f450e000640 1 -- 172.21.15.90:0/3999459367 shutdown_connections 2024-04-02T14:39:42.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.073+0000 7f450e000640 1 -- 172.21.15.90:0/3999459367 wait complete. 2024-04-02T14:39:42.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.074+0000 7f450e000640 1 Processor -- start 2024-04-02T14:39:42.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.074+0000 7f450e000640 1 -- start start 2024-04-02T14:39:42.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.075+0000 7f450e000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4508103df0 0x7f450819a510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:42.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.075+0000 7f450e000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4508108ec0 0x7f450819aa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:42.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.075+0000 7f450e000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f450819ed80 0x7f450819f230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:42.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.075+0000 7f450e000640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f4508113f70 con 0x7f4508103df0 2024-04-02T14:39:42.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.075+0000 7f450e000640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f4508113df0 con 0x7f450819ed80 2024-04-02T14:39:42.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.075+0000 7f450e000640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f45081140f0 con 0x7f4508108ec0 2024-04-02T14:39:42.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.075+0000 7f450d600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f450819ed80 0x7f450819f230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:42.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.075+0000 7f450d600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f450819ed80 0x7f450819f230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:48548/0 (socket says 172.21.15.90:48548) 2024-04-02T14:39:42.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.075+0000 7f450d600640 1 -- 172.21.15.90:0/636342577 learned_addr learned my addr 172.21.15.90:0/636342577 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:39:42.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.075+0000 7f4507e00640 1 --2- 172.21.15.90:0/636342577 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4508108ec0 0x7f450819aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:42.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.076+0000 7f450d600640 1 -- 172.21.15.90:0/636342577 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4508108ec0 msgr2=0x7f450819aa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:42.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.076+0000 7f450d600640 1 --2- 172.21.15.90:0/636342577 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4508108ec0 0x7f450819aa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:42.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.076+0000 7f450d600640 1 -- 172.21.15.90:0/636342577 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4508103df0 msgr2=0x7f450819a510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:42.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.076+0000 7f450cc00640 1 --2- 172.21.15.90:0/636342577 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4508103df0 0x7f450819a510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:42.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.076+0000 7f450d600640 1 --2- 172.21.15.90:0/636342577 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4508103df0 0x7f450819a510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:42.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.076+0000 7f450d600640 1 -- 172.21.15.90:0/636342577 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f450819f870 con 0x7f450819ed80 2024-04-02T14:39:42.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.076+0000 7f450cc00640 1 --2- 172.21.15.90:0/636342577 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4508103df0 0x7f450819a510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:39:42.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.076+0000 7f450d600640 1 --2- 172.21.15.90:0/636342577 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f450819ed80 0x7f450819f230 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f44f800e970 tx=0x7f44f800ed70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:39:42.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.077+0000 7f4505600640 1 -- 172.21.15.90:0/636342577 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44f800ccc0 con 0x7f450819ed80 2024-04-02T14:39:42.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.078+0000 7f4505600640 1 -- 172.21.15.90:0/636342577 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f44f800ce60 con 0x7f450819ed80 2024-04-02T14:39:42.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.078+0000 7f450e000640 1 -- 172.21.15.90:0/636342577 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f450819fb60 con 0x7f450819ed80 2024-04-02T14:39:42.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.078+0000 7f4505600640 1 -- 172.21.15.90:0/636342577 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44f8010490 con 0x7f450819ed80 2024-04-02T14:39:42.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.078+0000 7f450e000640 1 -- 172.21.15.90:0/636342577 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45081a7320 con 0x7f450819ed80 2024-04-02T14:39:42.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.081+0000 7f4505600640 1 -- 172.21.15.90:0/636342577 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f44f80106b0 con 0x7f450819ed80 2024-04-02T14:39:42.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.082+0000 7f451221c740 1 Processor -- start 2024-04-02T14:39:42.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.083+0000 7f451221c740 1 -- start start 2024-04-02T14:39:42.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.083+0000 7f4505600640 1 --2- 172.21.15.90:0/636342577 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f44ec0a3ef0 0x7f44ec0a63b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:42.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.083+0000 7f4505600640 1 -- 172.21.15.90:0/636342577 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f44f80d0c60 con 0x7f450819ed80 2024-04-02T14:39:42.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.083+0000 7f451221c740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5642d269c960 0x5642d269cd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:42.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.083+0000 7f451221c740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5642d269d280 0x5642d26a5920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:42.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.083+0000 7f451221c740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5642d26a5e60 0x5642d26a8250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:42.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.083+0000 7f451221c740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5642d2600770 con 0x5642d269d280 2024-04-02T14:39:42.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.083+0000 7f451221c740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x5642d25a8700 con 0x5642d269c960 2024-04-02T14:39:42.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.083+0000 7f451221c740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5642d25fc720 con 0x5642d26a5e60 2024-04-02T14:39:42.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.083+0000 7f4507e00640 1 --2- 172.21.15.90:0/636342577 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f44ec0a3ef0 0x7f44ec0a63b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:42.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.083+0000 7f450cc00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5642d269c960 0x5642d269cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:42.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.083+0000 7f450cc00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5642d269c960 0x5642d269cd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:48552/0 (socket says 172.21.15.90:48552) 2024-04-02T14:39:42.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.083+0000 7f450cc00640 1 -- 172.21.15.90:0/1023380241 learned_addr learned my addr 172.21.15.90:0/1023380241 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:39:42.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.084+0000 7f450d600640 1 --2- 172.21.15.90:0/1023380241 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5642d26a5e60 0x5642d26a8250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:42.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.084+0000 7f4507e00640 1 --2- 172.21.15.90:0/1023380241 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5642d269d280 0x5642d26a5920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:42.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.084+0000 7f450cc00640 1 -- 172.21.15.90:0/1023380241 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5642d26a5e60 msgr2=0x5642d26a8250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:39:42.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.084+0000 7f450cc00640 1 --2- 172.21.15.90:0/1023380241 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5642d26a5e60 0x5642d26a8250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:42.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.084+0000 7f450cc00640 1 -- 172.21.15.90:0/1023380241 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5642d269d280 msgr2=0x5642d26a5920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:39:42.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.085+0000 7f450cc00640 1 --2- 172.21.15.90:0/1023380241 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5642d269d280 0x5642d26a5920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:42.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.085+0000 7f450cc00640 1 -- 172.21.15.90:0/1023380241 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5642d23de7d0 con 0x5642d269c960 2024-04-02T14:39:42.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.086+0000 7f450cc00640 1 --2- 172.21.15.90:0/1023380241 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5642d269c960 0x5642d269cd40 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f450000f3e0 tx=0x7f4500009f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=81c361e77d961411 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:39:42.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.086+0000 7f4507e00640 1 --2- 172.21.15.90:0/636342577 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f44ec0a3ef0 0x7f44ec0a63b0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f44f4009230 tx=0x7f44f402ca00 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:39:42.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.087+0000 7f44fdc00640 1 -- 172.21.15.90:0/1023380241 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4500004d20 con 0x5642d269c960 2024-04-02T14:39:42.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.087+0000 7f44fdc00640 1 -- 172.21.15.90:0/1023380241 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f45000119e0 con 0x5642d269c960 2024-04-02T14:39:42.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.087+0000 7f44fdc00640 1 -- 172.21.15.90:0/1023380241 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45000055c0 con 0x5642d269c960 2024-04-02T14:39:42.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.088+0000 7f451221c740 1 -- 172.21.15.90:0/1023380241 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5642d269c960 msgr2=0x5642d269cd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:42.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.088+0000 7f451221c740 1 --2- 172.21.15.90:0/1023380241 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5642d269c960 0x5642d269cd40 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f450000f3e0 tx=0x7f4500009f90 comp rx=0 tx=0).stop 2024-04-02T14:39:42.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.089+0000 7f451221c740 1 -- 172.21.15.90:0/1023380241 shutdown_connections 2024-04-02T14:39:42.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.089+0000 7f451221c740 1 --2- 172.21.15.90:0/1023380241 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5642d26a5e60 0x5642d26a8250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:42.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.089+0000 7f451221c740 1 --2- 172.21.15.90:0/1023380241 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5642d269d280 0x5642d26a5920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:42.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.089+0000 7f451221c740 1 --2- 172.21.15.90:0/1023380241 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5642d269c960 0x5642d269cd40 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:42.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.089+0000 7f451221c740 1 -- 172.21.15.90:0/1023380241 >> 172.21.15.90:0/1023380241 conn(0x5642d2697090 msgr2=0x5642d2697470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:39:42.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.089+0000 7f451221c740 1 -- 172.21.15.90:0/1023380241 shutdown_connections 2024-04-02T14:39:42.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.089+0000 7f451221c740 1 -- 172.21.15.90:0/1023380241 wait complete. 2024-04-02T14:39:42.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.090+0000 7f451221c740 1 -- start start 2024-04-02T14:39:42.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.091+0000 7f451221c740 1 Processor -- start 2024-04-02T14:39:42.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.092+0000 7f451221c740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5642d269c960 0x5642d26a2f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:42.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.092+0000 7f451221c740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5642d269d280 0x5642d26a3460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:42.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.092+0000 7f451221c740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5642d26a5e60 0x5642d26a39a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:42.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.092+0000 7f451221c740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5642d26a8c50 con 0x5642d269d280 2024-04-02T14:39:42.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.092+0000 7f451221c740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x5642d25bbac0 con 0x5642d269c960 2024-04-02T14:39:42.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.092+0000 7f451221c740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5642d26a8dd0 con 0x5642d26a5e60 2024-04-02T14:39:42.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.092+0000 7f451221c740 10 client.0 _ll_register_callbacks cb 0x5642d25be180 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:39:42.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.092+0000 7f450cc00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5642d269c960 0x5642d26a2f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:42.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.092+0000 7f450cc00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5642d269c960 0x5642d26a2f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:48564/0 (socket says 172.21.15.90:48564) 2024-04-02T14:39:42.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.092+0000 7f450cc00640 1 -- 172.21.15.90:0/1778677805 learned_addr learned my addr 172.21.15.90:0/1778677805 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:39:42.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.093+0000 7f4507e00640 1 --2- 172.21.15.90:0/1778677805 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5642d269d280 0x5642d26a3460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:42.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.093+0000 7f450d600640 1 --2- 172.21.15.90:0/1778677805 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5642d26a5e60 0x5642d26a39a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:42.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.093+0000 7f450cc00640 1 -- 172.21.15.90:0/1778677805 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5642d26a5e60 msgr2=0x5642d26a39a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:39:42.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.093+0000 7f450cc00640 1 --2- 172.21.15.90:0/1778677805 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5642d26a5e60 0x5642d26a39a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:42.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.093+0000 7f450cc00640 1 -- 172.21.15.90:0/1778677805 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5642d269d280 msgr2=0x5642d26a3460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:39:42.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.093+0000 7f450cc00640 1 --2- 172.21.15.90:0/1778677805 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5642d269d280 0x5642d26a3460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:42.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.093+0000 7f450cc00640 1 -- 172.21.15.90:0/1778677805 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5642d23143f0 con 0x5642d269c960 2024-04-02T14:39:42.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.093+0000 7f4507e00640 1 --2- 172.21.15.90:0/1778677805 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5642d269d280 0x5642d26a3460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:39:42.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.093+0000 7f450d600640 1 --2- 172.21.15.90:0/1778677805 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5642d26a5e60 0x5642d26a39a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:39:42.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.093+0000 7f450cc00640 1 --2- 172.21.15.90:0/1778677805 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5642d269c960 0x5642d26a2f20 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f45000092f0 tx=0x7f4500011dc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7f9ae19f243ba1be server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:39:42.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.093+0000 7f44f2000640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:39:42.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.093+0000 7f44f2000640 1 -- 172.21.15.90:0/1778677805 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4500011710 con 0x5642d269c960 2024-04-02T14:39:42.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.094+0000 7f44f2000640 1 -- 172.21.15.90:0/1778677805 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f4500003970 con 0x5642d269c960 2024-04-02T14:39:42.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.094+0000 7f451221c740 1 -- 172.21.15.90:0/1778677805 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f45000119e0 con 0x5642d269c960 2024-04-02T14:39:42.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.094+0000 7f44f2000640 1 -- 172.21.15.90:0/1778677805 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f4500003b70 con 0x5642d269c960 2024-04-02T14:39:42.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.094+0000 7f451221c740 10 client.26939 fetch_fsmap learned FSMap version 27 2024-04-02T14:39:42.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.094+0000 7f451221c740 1 -- 172.21.15.90:0/1778677805 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5642d22d40f0 con 0x5642d269c960 2024-04-02T14:39:42.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.094+0000 7f44f2000640 1 -- 172.21.15.90:0/1778677805 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f45000119e0 con 0x5642d269c960 2024-04-02T14:39:42.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.095+0000 7f44f2000640 1 -- 172.21.15.90:0/1778677805 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f4500005570 con 0x5642d269c960 2024-04-02T14:39:42.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.095+0000 7f451221c740 10 client.26939 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:39:42.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.095+0000 7f451221c740 10 client.26939 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:39:42.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.095+0000 7f451221c740 20 client.26939 populate_metadata read hostname 'smithi090' 2024-04-02T14:39:42.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.095+0000 7f451221c740 1 --2- 172.21.15.90:0/1778677805 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5642d26be690 0x5642d26c0a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:42.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.095+0000 7f451221c740 4 client.26939 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:39:42.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.095+0000 7f451221c740 1 -- 172.21.15.90:0/1778677805 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5642d25a1630 con 0x5642d26be690 2024-04-02T14:39:42.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.095+0000 7f4507e00640 1 --2- 172.21.15.90:0/1778677805 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5642d26be690 0x5642d26c0a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:42.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.097+0000 7f4507e00640 1 --2- 172.21.15.90:0/1778677805 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5642d26be690 0x5642d26c0a80 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eec333d467f67ff9 server_cookie=f020eaecc1bef7f5 in_seq=0 out_seq=0 2024-04-02T14:39:42.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.097+0000 7f44f2000640 10 client.26939 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:39:42.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.098+0000 7f44f2000640 1 -- 172.21.15.90:0/1778677805 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5642d25a1630 con 0x5642d26be690 2024-04-02T14:39:42.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.098+0000 7f44f2000640 10 client.26939 handle_command_reply: tid=0 2024-04-02T14:39:42.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.098+0000 7f451221c740 1 client.26939 shutdown 2024-04-02T14:39:42.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.098+0000 7f451221c740 1 -- 172.21.15.90:0/1778677805 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5642d269c960 msgr2=0x5642d26a2f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:42.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.098+0000 7f451221c740 1 --2- 172.21.15.90:0/1778677805 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5642d269c960 0x5642d26a2f20 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f45000092f0 tx=0x7f4500011dc0 comp rx=0 tx=0).stop 2024-04-02T14:39:42.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.098+0000 7f451221c740 1 -- 172.21.15.90:0/1778677805 shutdown_connections 2024-04-02T14:39:42.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.098+0000 7f451221c740 1 --2- 172.21.15.90:0/1778677805 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5642d26be690 0x5642d26c0a80 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:42.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.098+0000 7f451221c740 1 --2- 172.21.15.90:0/1778677805 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5642d26a5e60 0x5642d26a39a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:42.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.098+0000 7f451221c740 1 --2- 172.21.15.90:0/1778677805 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5642d269d280 0x5642d26a3460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:42.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.098+0000 7f451221c740 1 --2- 172.21.15.90:0/1778677805 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5642d269c960 0x5642d26a2f20 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:42.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.098+0000 7f451221c740 1 -- 172.21.15.90:0/1778677805 >> 172.21.15.90:0/1778677805 conn(0x5642d2697090 msgr2=0x5642d2698de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:39:42.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.098+0000 7f44f2000640 0 client.26939 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:39:42.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.099+0000 7f451221c740 1 -- 172.21.15.90:0/1778677805 shutdown_connections 2024-04-02T14:39:42.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.099+0000 7f451221c740 1 -- 172.21.15.90:0/1778677805 wait complete. 2024-04-02T14:39:42.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.099+0000 7f451221c740 20 client.26939 trim_cache size 0 max 16384 2024-04-02T14:39:42.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.133+0000 7f451221c740 1 Processor -- start 2024-04-02T14:39:42.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.133+0000 7f451221c740 1 -- start start 2024-04-02T14:39:42.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.133+0000 7f451221c740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5642d269c960 0x5642d26a5650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:42.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.133+0000 7f451221c740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5642d269d280 0x5642d26b1b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:42.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.133+0000 7f451221c740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5642d26a5e60 0x5642d2697470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:42.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.133+0000 7f451221c740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f45081ab480 con 0x5642d269c960 2024-04-02T14:39:42.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.133+0000 7f451221c740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f45081acfc0 con 0x5642d26a5e60 2024-04-02T14:39:42.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.133+0000 7f451221c740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f45081abb50 con 0x5642d269d280 2024-04-02T14:39:42.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.134+0000 7f450cc00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5642d269c960 0x5642d26a5650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:42.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.134+0000 7f450cc00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5642d269c960 0x5642d26a5650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:49042/0 (socket says 172.21.15.90:49042) 2024-04-02T14:39:42.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.134+0000 7f450cc00640 1 -- 172.21.15.90:0/2946618708 learned_addr learned my addr 172.21.15.90:0/2946618708 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:39:42.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.134+0000 7f450d600640 1 --2- 172.21.15.90:0/2946618708 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5642d26a5e60 0x5642d2697470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:42.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.134+0000 7f4507e00640 1 --2- 172.21.15.90:0/2946618708 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5642d269d280 0x5642d26b1b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:42.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.134+0000 7f450cc00640 1 -- 172.21.15.90:0/2946618708 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5642d269d280 msgr2=0x5642d26b1b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:39:42.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.134+0000 7f450cc00640 1 --2- 172.21.15.90:0/2946618708 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5642d269d280 0x5642d26b1b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:42.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.134+0000 7f450cc00640 1 -- 172.21.15.90:0/2946618708 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5642d26a5e60 msgr2=0x5642d2697470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:39:42.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.134+0000 7f450cc00640 1 --2- 172.21.15.90:0/2946618708 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5642d26a5e60 0x5642d2697470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:42.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.134+0000 7f450cc00640 1 -- 172.21.15.90:0/2946618708 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5642d26a85c0 con 0x5642d269c960 2024-04-02T14:39:42.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.135+0000 7f450cc00640 1 --2- 172.21.15.90:0/2946618708 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5642d269c960 0x5642d26a5650 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f450000f3e0 tx=0x7f4500011f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6624f55ff040bec5 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:39:42.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.135+0000 7f44fd200640 1 -- 172.21.15.90:0/2946618708 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4500009bb0 con 0x5642d269c960 2024-04-02T14:39:42.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.135+0000 7f44fd200640 1 -- 172.21.15.90:0/2946618708 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4500011710 con 0x5642d269c960 2024-04-02T14:39:42.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.135+0000 7f44fd200640 1 -- 172.21.15.90:0/2946618708 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f450001b450 con 0x5642d269c960 2024-04-02T14:39:42.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.136+0000 7f451221c740 1 -- 172.21.15.90:0/2946618708 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5642d269c960 msgr2=0x5642d26a5650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:42.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.136+0000 7f451221c740 1 --2- 172.21.15.90:0/2946618708 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5642d269c960 0x5642d26a5650 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f450000f3e0 tx=0x7f4500011f30 comp rx=0 tx=0).stop 2024-04-02T14:39:42.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.136+0000 7f451221c740 1 -- 172.21.15.90:0/2946618708 shutdown_connections 2024-04-02T14:39:42.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.136+0000 7f451221c740 1 --2- 172.21.15.90:0/2946618708 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5642d26a5e60 0x5642d2697470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:42.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.136+0000 7f451221c740 1 --2- 172.21.15.90:0/2946618708 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5642d269d280 0x5642d26b1b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:42.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.136+0000 7f451221c740 1 --2- 172.21.15.90:0/2946618708 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5642d269c960 0x5642d26a5650 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:42.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.136+0000 7f451221c740 1 -- 172.21.15.90:0/2946618708 >> 172.21.15.90:0/2946618708 conn(0x5642d2697090 msgr2=0x5642d2695a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:39:42.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.137+0000 7f451221c740 1 -- 172.21.15.90:0/2946618708 shutdown_connections 2024-04-02T14:39:42.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.137+0000 7f451221c740 1 -- 172.21.15.90:0/2946618708 wait complete. 2024-04-02T14:39:42.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.137+0000 7f451221c740 1 -- start start 2024-04-02T14:39:42.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.139+0000 7f451221c740 1 Processor -- start 2024-04-02T14:39:42.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.140+0000 7f451221c740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5642d269c960 0x5642d26ae520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:42.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.140+0000 7f451221c740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5642d269d280 0x5642d269fa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:42.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.141+0000 7f451221c740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5642d26a5e60 0x5642d26a01d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:42.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.141+0000 7f451221c740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f45081ac8f0 con 0x5642d269d280 2024-04-02T14:39:42.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.141+0000 7f451221c740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f45081ac220 con 0x5642d26a5e60 2024-04-02T14:39:42.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.141+0000 7f451221c740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f45081a9480 con 0x5642d269c960 2024-04-02T14:39:42.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.141+0000 7f451221c740 10 client.0 _ll_register_callbacks cb 0x7f4500015070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:39:42.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.141+0000 7f4507e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5642d269d280 0x5642d269fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:42.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.141+0000 7f450cc00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5642d269c960 0x5642d26ae520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:42.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.141+0000 7f450cc00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5642d269c960 0x5642d26ae520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:44836/0 (socket says 172.21.15.90:44836) 2024-04-02T14:39:42.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.141+0000 7f4507e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5642d269d280 0x5642d269fa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:49054/0 (socket says 172.21.15.90:49054) 2024-04-02T14:39:42.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.141+0000 7f450cc00640 1 -- 172.21.15.90:0/3721679120 learned_addr learned my addr 172.21.15.90:0/3721679120 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:39:42.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.141+0000 7f450d600640 1 --2- 172.21.15.90:0/3721679120 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5642d26a5e60 0x5642d26a01d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:42.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.141+0000 7f450cc00640 1 -- 172.21.15.90:0/3721679120 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5642d26a5e60 msgr2=0x5642d26a01d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:39:42.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.141+0000 7f450cc00640 1 --2- 172.21.15.90:0/3721679120 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5642d26a5e60 0x5642d26a01d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:42.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.141+0000 7f450cc00640 1 -- 172.21.15.90:0/3721679120 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5642d269d280 msgr2=0x5642d269fa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:39:42.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.141+0000 7f450cc00640 1 --2- 172.21.15.90:0/3721679120 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5642d269d280 0x5642d269fa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:42.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.141+0000 7f450cc00640 1 -- 172.21.15.90:0/3721679120 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5642d2723950 con 0x5642d269c960 2024-04-02T14:39:42.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.141+0000 7f450d600640 1 --2- 172.21.15.90:0/3721679120 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5642d26a5e60 0x5642d26a01d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:39:42.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.142+0000 7f4507e00640 1 --2- 172.21.15.90:0/3721679120 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5642d269d280 0x5642d269fa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:39:42.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.142+0000 7f450cc00640 1 --2- 172.21.15.90:0/3721679120 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5642d269c960 0x5642d26ae520 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f4500006010 tx=0x7f4500009240 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a28223ee92042253 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:39:42.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.142+0000 7f44f2000640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:39:42.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.142+0000 7f44f2000640 1 -- 172.21.15.90:0/3721679120 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f450003b020 con 0x5642d269c960 2024-04-02T14:39:42.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.142+0000 7f451221c740 1 -- 172.21.15.90:0/3721679120 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4500011710 con 0x5642d269c960 2024-04-02T14:39:42.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.142+0000 7f44f2000640 1 -- 172.21.15.90:0/3721679120 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f4500033040 con 0x5642d269c960 2024-04-02T14:39:42.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.143+0000 7f44f2000640 1 -- 172.21.15.90:0/3721679120 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f4500046050 con 0x5642d269c960 2024-04-02T14:39:42.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.143+0000 7f451221c740 10 client.26818 fetch_fsmap learned FSMap version 27 2024-04-02T14:39:42.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.143+0000 7f451221c740 1 -- 172.21.15.90:0/3721679120 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5642d2724ad0 con 0x5642d269c960 2024-04-02T14:39:42.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.143+0000 7f44f2000640 1 -- 172.21.15.90:0/3721679120 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4500011710 con 0x5642d269c960 2024-04-02T14:39:42.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.143+0000 7f44f2000640 1 -- 172.21.15.90:0/3721679120 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f4500037420 con 0x5642d269c960 2024-04-02T14:39:42.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.144+0000 7f451221c740 10 client.26818 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:39:42.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.144+0000 7f451221c740 10 client.26818 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:39:42.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.144+0000 7f451221c740 20 client.26818 populate_metadata read hostname 'smithi090' 2024-04-02T14:39:42.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.144+0000 7f451221c740 1 --2- 172.21.15.90:0/3721679120 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5642d2725240 0x5642d2727660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:42.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.144+0000 7f451221c740 4 client.26818 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-02T14:39:42.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.144+0000 7f451221c740 1 -- 172.21.15.90:0/3721679120 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5642d260dcb0 con 0x5642d2725240 2024-04-02T14:39:42.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.145+0000 7f4507e00640 1 --2- 172.21.15.90:0/3721679120 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5642d2725240 0x5642d2727660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:42.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.151+0000 7f44f2000640 10 client.26818 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:39:42.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.151+0000 7f4507e00640 1 --2- 172.21.15.90:0/3721679120 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5642d2725240 0x5642d2727660 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=29ba5da16dc96349 server_cookie=c28cd912cfce8ee5 in_seq=0 out_seq=0 2024-04-02T14:39:42.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.156+0000 7f44f2000640 1 -- 172.21.15.90:0/3721679120 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+265 (crc 0 0 0) 0x5642d260dcb0 con 0x5642d2725240 2024-04-02T14:39:42.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.156+0000 7f44f2000640 10 client.26818 handle_command_reply: tid=0 2024-04-02T14:39:42.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.156+0000 7f451221c740 1 client.26818 shutdown 2024-04-02T14:39:42.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.156+0000 7f451221c740 1 -- 172.21.15.90:0/3721679120 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5642d269c960 msgr2=0x5642d26ae520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:42.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.156+0000 7f451221c740 1 --2- 172.21.15.90:0/3721679120 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5642d269c960 0x5642d26ae520 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f4500006010 tx=0x7f4500009240 comp rx=0 tx=0).stop 2024-04-02T14:39:42.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.157+0000 7f451221c740 1 -- 172.21.15.90:0/3721679120 shutdown_connections 2024-04-02T14:39:42.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.157+0000 7f451221c740 1 --2- 172.21.15.90:0/3721679120 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5642d2725240 0x5642d2727660 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:42.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.157+0000 7f451221c740 1 --2- 172.21.15.90:0/3721679120 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5642d26a5e60 0x5642d26a01d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:42.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.157+0000 7f451221c740 1 --2- 172.21.15.90:0/3721679120 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5642d269d280 0x5642d269fa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:42.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.157+0000 7f451221c740 1 --2- 172.21.15.90:0/3721679120 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5642d269c960 0x5642d26ae520 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:42.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.157+0000 7f451221c740 1 -- 172.21.15.90:0/3721679120 >> 172.21.15.90:0/3721679120 conn(0x5642d2697090 msgr2=0x5642d2696310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:39:42.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.157+0000 7f44f2000640 0 client.26818 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:39:42.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.158+0000 7f451221c740 1 -- 172.21.15.90:0/3721679120 shutdown_connections 2024-04-02T14:39:42.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.158+0000 7f451221c740 1 -- 172.21.15.90:0/3721679120 wait complete. 2024-04-02T14:39:42.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.159+0000 7f451221c740 20 client.26818 trim_cache size 0 max 16384 2024-04-02T14:39:42.162 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-02T14:39:42.162 INFO:teuthology.orchestra.run.smithi090.stdout: "status": "scrub active (1 inodes in the stack)", 2024-04-02T14:39:42.162 INFO:teuthology.orchestra.run.smithi090.stdout: "scrubs": { 2024-04-02T14:39:42.162 INFO:teuthology.orchestra.run.smithi090.stdout: "77fb932f-0f67-4c0a-b524-e1eb04fb36d6": { 2024-04-02T14:39:42.162 INFO:teuthology.orchestra.run.smithi090.stdout: "path": "/", 2024-04-02T14:39:42.162 INFO:teuthology.orchestra.run.smithi090.stdout: "tag": "77fb932f-0f67-4c0a-b524-e1eb04fb36d6", 2024-04-02T14:39:42.163 INFO:teuthology.orchestra.run.smithi090.stdout: "options": "recursive,force" 2024-04-02T14:39:42.163 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:39:42.163 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:39:42.163 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-02T14:39:42.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.160+0000 7f450e000640 1 -- 172.21.15.90:0/636342577 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f44ec0a3ef0 msgr2=0x7f44ec0a63b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:42.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.160+0000 7f450e000640 1 --2- 172.21.15.90:0/636342577 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f44ec0a3ef0 0x7f44ec0a63b0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f44f4009230 tx=0x7f44f402ca00 comp rx=0 tx=0).stop 2024-04-02T14:39:42.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.160+0000 7f450e000640 1 -- 172.21.15.90:0/636342577 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f450819ed80 msgr2=0x7f450819f230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:42.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.160+0000 7f450e000640 1 --2- 172.21.15.90:0/636342577 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f450819ed80 0x7f450819f230 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f44f800e970 tx=0x7f44f800ed70 comp rx=0 tx=0).stop 2024-04-02T14:39:42.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.160+0000 7f450e000640 1 -- 172.21.15.90:0/636342577 shutdown_connections 2024-04-02T14:39:42.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.161+0000 7f450e000640 1 --2- 172.21.15.90:0/636342577 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f44ec0a3ef0 0x7f44ec0a63b0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:42.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.161+0000 7f450e000640 1 --2- 172.21.15.90:0/636342577 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f450819ed80 0x7f450819f230 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:42.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.161+0000 7f450e000640 1 --2- 172.21.15.90:0/636342577 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4508108ec0 0x7f450819aa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:42.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.161+0000 7f450e000640 1 --2- 172.21.15.90:0/636342577 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4508103df0 0x7f450819a510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:42.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.161+0000 7f450e000640 1 -- 172.21.15.90:0/636342577 >> 172.21.15.90:0/636342577 conn(0x7f45080ff610 msgr2=0x7f4508100930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:39:42.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.161+0000 7f450e000640 1 -- 172.21.15.90:0/636342577 shutdown_connections 2024-04-02T14:39:42.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:39:42.161+0000 7f450e000640 1 -- 172.21.15.90:0/636342577 wait complete. 2024-04-02T14:39:42.188 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:39:42 smithi090 ceph-mon[31674]: pgmap v147: 97 pgs: 97 active+clean; 3.3 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 156 KiB/s rd, 34 MiB/s wr, 1.90k op/s 2024-04-02T14:39:42.192 INFO:tasks.cephfs.filesystem:scrub status for tag:77fb932f-0f67-4c0a-b524-e1eb04fb36d6 - {'path': '/', 'tag': '77fb932f-0f67-4c0a-b524-e1eb04fb36d6', 'options': 'recursive,force'} 2024-04-02T14:39:42.360 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:39:42 smithi092 ceph-mon[36497]: pgmap v147: 97 pgs: 97 active+clean; 3.3 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 156 KiB/s rd, 34 MiB/s wr, 1.90k op/s 2024-04-02T14:39:42.436 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:39:42 smithi177 ceph-mon[36431]: pgmap v147: 97 pgs: 97 active+clean; 3.3 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 156 KiB/s rd, 34 MiB/s wr, 1.90k op/s 2024-04-02T14:39:43.360 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:39:43 smithi092 ceph-mon[36497]: scrub queued for path: / 2024-04-02T14:39:43.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:39:43 smithi092 ceph-mon[36497]: scrub summary: idle+waiting paths [/] 2024-04-02T14:39:43.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:39:43 smithi092 ceph-mon[36497]: scrub summary: active paths [/] 2024-04-02T14:39:43.437 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:39:43 smithi177 ceph-mon[36431]: scrub queued for path: / 2024-04-02T14:39:43.437 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:39:43 smithi177 ceph-mon[36431]: scrub summary: idle+waiting paths [/] 2024-04-02T14:39:43.437 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:39:43 smithi177 ceph-mon[36431]: scrub summary: active paths [/] 2024-04-02T14:39:43.439 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:39:43 smithi090 ceph-mon[31674]: scrub queued for path: / 2024-04-02T14:39:43.439 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:39:43 smithi090 ceph-mon[31674]: scrub summary: idle+waiting paths [/] 2024-04-02T14:39:43.439 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:39:43 smithi090 ceph-mon[31674]: scrub summary: active paths [/] 2024-04-02T14:39:44.272 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.270+0000 7f4a0b200640 1 -- 172.21.15.90:0/1994972737 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4a040fff50 msgr2=0x7f4a04100330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:44.272 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.270+0000 7f4a0b200640 1 --2- 172.21.15.90:0/1994972737 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4a040fff50 0x7f4a04100330 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f49f401a910 tx=0x7f49f403cc60 comp rx=0 tx=0).stop 2024-04-02T14:39:44.273 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.270+0000 7f4a0b200640 1 -- 172.21.15.90:0/1994972737 shutdown_connections 2024-04-02T14:39:44.273 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.270+0000 7f4a0b200640 1 --2- 172.21.15.90:0/1994972737 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4a04105020 0x7f4a0410cbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:44.273 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.270+0000 7f4a0b200640 1 --2- 172.21.15.90:0/1994972737 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4a04100900 0x7f4a04104940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:44.273 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.270+0000 7f4a0b200640 1 --2- 172.21.15.90:0/1994972737 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4a040fff50 0x7f4a04100330 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:44.274 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.270+0000 7f4a0b200640 1 -- 172.21.15.90:0/1994972737 >> 172.21.15.90:0/1994972737 conn(0x7f4a04076b90 msgr2=0x7f4a040fdee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:39:44.274 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.272+0000 7f4a0b200640 1 -- 172.21.15.90:0/1994972737 shutdown_connections 2024-04-02T14:39:44.274 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.272+0000 7f4a0b200640 1 -- 172.21.15.90:0/1994972737 wait complete. 2024-04-02T14:39:44.275 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.273+0000 7f4a0b200640 1 Processor -- start 2024-04-02T14:39:44.275 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.273+0000 7f4a0b200640 1 -- start start 2024-04-02T14:39:44.276 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.274+0000 7f4a0b200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4a040fff50 0x7f4a04106d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:44.276 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.274+0000 7f4a0b200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4a04105020 0x7f4a04107260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:44.276 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.274+0000 7f4a0b200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4a0419bc70 0x7f4a0419c120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:44.276 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.274+0000 7f4a0b200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f4a0406dab0 con 0x7f4a040fff50 2024-04-02T14:39:44.277 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.274+0000 7f4a0b200640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f4a0406d930 con 0x7f4a04105020 2024-04-02T14:39:44.277 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.274+0000 7f4a0b200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f4a0406dc30 con 0x7f4a0419bc70 2024-04-02T14:39:44.277 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.274+0000 7f4a03400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4a04105020 0x7f4a04107260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:44.277 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.274+0000 7f4a03400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4a04105020 0x7f4a04107260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:38752/0 (socket says 172.21.15.90:38752) 2024-04-02T14:39:44.277 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.274+0000 7f4a03400640 1 -- 172.21.15.90:0/2026460957 learned_addr learned my addr 172.21.15.90:0/2026460957 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:39:44.277 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.275+0000 7f4a03e00640 1 --2- 172.21.15.90:0/2026460957 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4a040fff50 0x7f4a04106d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:44.278 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.275+0000 7f4a09200640 1 --2- 172.21.15.90:0/2026460957 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4a0419bc70 0x7f4a0419c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:44.278 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.275+0000 7f4a03400640 1 -- 172.21.15.90:0/2026460957 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4a0419bc70 msgr2=0x7f4a0419c120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:44.278 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.275+0000 7f4a03400640 1 --2- 172.21.15.90:0/2026460957 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4a0419bc70 0x7f4a0419c120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:44.278 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.275+0000 7f4a03400640 1 -- 172.21.15.90:0/2026460957 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4a040fff50 msgr2=0x7f4a04106d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:44.278 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.275+0000 7f4a03400640 1 --2- 172.21.15.90:0/2026460957 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4a040fff50 0x7f4a04106d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:44.278 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.275+0000 7f4a03400640 1 -- 172.21.15.90:0/2026460957 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a0419c760 con 0x7f4a04105020 2024-04-02T14:39:44.279 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.275+0000 7f4a09200640 1 --2- 172.21.15.90:0/2026460957 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4a0419bc70 0x7f4a0419c120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:39:44.279 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.276+0000 7f4a03400640 1 --2- 172.21.15.90:0/2026460957 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4a04105020 0x7f4a04107260 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f49f000d770 tx=0x7f49f000dc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:39:44.279 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.276+0000 7f4a00c00640 1 -- 172.21.15.90:0/2026460957 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49f0017070 con 0x7f4a04105020 2024-04-02T14:39:44.279 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.276+0000 7f4a00c00640 1 -- 172.21.15.90:0/2026460957 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f49f0009c30 con 0x7f4a04105020 2024-04-02T14:39:44.279 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.276+0000 7f4a00c00640 1 -- 172.21.15.90:0/2026460957 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49f000f6b0 con 0x7f4a04105020 2024-04-02T14:39:44.279 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.276+0000 7f4a0b200640 1 -- 172.21.15.90:0/2026460957 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a0419c9f0 con 0x7f4a04105020 2024-04-02T14:39:44.279 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.277+0000 7f4a0b200640 1 -- 172.21.15.90:0/2026460957 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a041a2fc0 con 0x7f4a04105020 2024-04-02T14:39:44.285 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.282+0000 7f4a0dfc7740 1 Processor -- start 2024-04-02T14:39:44.285 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.282+0000 7f4a0dfc7740 1 -- start start 2024-04-02T14:39:44.285 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.282+0000 7f4a0dfc7740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5604441d0eb0 0x5604441d1290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:44.286 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.284+0000 7f4a09200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5604441d0eb0 0x5604441d1290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:44.286 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.284+0000 7f4a09200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5604441d0eb0 0x5604441d1290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:55962/0 (socket says 172.21.15.90:55962) 2024-04-02T14:39:44.287 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.285+0000 7f4a00c00640 1 -- 172.21.15.90:0/2026460957 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f49f001a070 con 0x7f4a04105020 2024-04-02T14:39:44.287 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.285+0000 7f4a09200640 1 -- 172.21.15.90:0/1016972250 learned_addr learned my addr 172.21.15.90:0/1016972250 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:39:44.288 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.287+0000 7f4a0dfc7740 1 --2- 172.21.15.90:0/1016972250 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5604441d17d0 0x5604441d9e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:44.289 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.287+0000 7f4a0dfc7740 1 --2- 172.21.15.90:0/1016972250 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5604441da3b0 0x5604441dc7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:44.289 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.287+0000 7f4a0dfc7740 1 -- 172.21.15.90:0/1016972250 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x560444135d70 con 0x5604441d17d0 2024-04-02T14:39:44.289 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.287+0000 7f4a0dfc7740 1 -- 172.21.15.90:0/1016972250 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x5604440de2a0 con 0x5604441da3b0 2024-04-02T14:39:44.290 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.287+0000 7f4a0dfc7740 1 -- 172.21.15.90:0/1016972250 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x560444131d20 con 0x5604441d0eb0 2024-04-02T14:39:44.290 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.287+0000 7f4a09200640 1 -- 172.21.15.90:0/1016972250 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5604441da3b0 msgr2=0x5604441dc7a0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-02T14:39:44.290 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.287+0000 7f4a09200640 1 --2- 172.21.15.90:0/1016972250 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5604441da3b0 0x5604441dc7a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:44.290 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.287+0000 7f4a09200640 1 -- 172.21.15.90:0/1016972250 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5604441d17d0 msgr2=0x5604441d9e70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-02T14:39:44.290 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.287+0000 7f4a09200640 1 --2- 172.21.15.90:0/1016972250 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5604441d17d0 0x5604441d9e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:44.290 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.287+0000 7f4a09200640 1 -- 172.21.15.90:0/1016972250 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5604440f2840 con 0x5604441d0eb0 2024-04-02T14:39:44.290 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.287+0000 7f4a00c00640 1 --2- 172.21.15.90:0/2026460957 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f49e80a3f70 0x7f49e80a6430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:44.290 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.287+0000 7f4a09200640 1 --2- 172.21.15.90:0/1016972250 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5604441d0eb0 0x5604441d1290 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f49fc00ac90 tx=0x7f49fc008600 comp rx=0 tx=0).ready entity=mon.1 client_cookie=20730ca040b82ad5 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:39:44.290 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.288+0000 7f4a00c00640 1 -- 172.21.15.90:0/2026460957 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f49f00cfef0 con 0x7f4a04105020 2024-04-02T14:39:44.291 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.288+0000 7f49f9600640 1 -- 172.21.15.90:0/1016972250 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49fc004b20 con 0x5604441d0eb0 2024-04-02T14:39:44.291 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.288+0000 7f49f9600640 1 -- 172.21.15.90:0/1016972250 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f49fc004cc0 con 0x5604441d0eb0 2024-04-02T14:39:44.291 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.288+0000 7f49f9600640 1 -- 172.21.15.90:0/1016972250 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49fc014770 con 0x5604441d0eb0 2024-04-02T14:39:44.291 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.288+0000 7f4a03e00640 1 --2- 172.21.15.90:0/2026460957 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f49e80a3f70 0x7f49e80a6430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:44.293 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.291+0000 7f4a0dfc7740 1 -- 172.21.15.90:0/1016972250 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5604441d0eb0 msgr2=0x5604441d1290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:44.293 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.291+0000 7f4a0dfc7740 1 --2- 172.21.15.90:0/1016972250 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5604441d0eb0 0x5604441d1290 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f49fc00ac90 tx=0x7f49fc008600 comp rx=0 tx=0).stop 2024-04-02T14:39:44.293 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.291+0000 7f4a0dfc7740 1 -- 172.21.15.90:0/1016972250 shutdown_connections 2024-04-02T14:39:44.293 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.291+0000 7f4a0dfc7740 1 --2- 172.21.15.90:0/1016972250 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5604441da3b0 0x5604441dc7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:44.293 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.291+0000 7f4a0dfc7740 1 --2- 172.21.15.90:0/1016972250 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5604441d17d0 0x5604441d9e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:44.293 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.291+0000 7f4a0dfc7740 1 --2- 172.21.15.90:0/1016972250 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5604441d0eb0 0x5604441d1290 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:44.293 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.291+0000 7f4a0dfc7740 1 -- 172.21.15.90:0/1016972250 >> 172.21.15.90:0/1016972250 conn(0x5604441cb5e0 msgr2=0x5604441cb9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:39:44.294 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.291+0000 7f4a0dfc7740 1 -- 172.21.15.90:0/1016972250 shutdown_connections 2024-04-02T14:39:44.294 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.292+0000 7f4a0dfc7740 1 -- 172.21.15.90:0/1016972250 wait complete. 2024-04-02T14:39:44.294 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.292+0000 7f4a03e00640 1 --2- 172.21.15.90:0/2026460957 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f49e80a3f70 0x7f49e80a6430 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f49f4006010 tx=0x7f49f403f040 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:39:44.295 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.293+0000 7f4a0dfc7740 1 -- start start 2024-04-02T14:39:44.295 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.293+0000 7f4a0dfc7740 1 Processor -- start 2024-04-02T14:39:44.296 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.293+0000 7f4a0dfc7740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5604441d0eb0 0x5604441d7550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:44.296 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.293+0000 7f4a0dfc7740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5604441d17d0 0x5604441d7a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:44.296 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.293+0000 7f4a0dfc7740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5604441da3b0 0x5604441d7fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:44.296 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.293+0000 7f4a0dfc7740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5604441dd1a0 con 0x5604441d0eb0 2024-04-02T14:39:44.296 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.293+0000 7f4a0dfc7740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x5604440f1660 con 0x5604441d17d0 2024-04-02T14:39:44.296 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.293+0000 7f4a0dfc7740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5604441dd320 con 0x5604441da3b0 2024-04-02T14:39:44.296 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.294+0000 7f4a0dfc7740 10 client.0 _ll_register_callbacks cb 0x5604440f1050 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:39:44.297 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.294+0000 7f4a03400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5604441d17d0 0x5604441d7a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:44.297 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.294+0000 7f4a03400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5604441d17d0 0x5604441d7a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:38774/0 (socket says 172.21.15.90:38774) 2024-04-02T14:39:44.297 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.294+0000 7f4a03400640 1 -- 172.21.15.90:0/3965610456 learned_addr learned my addr 172.21.15.90:0/3965610456 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:39:44.297 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.294+0000 7f4a09200640 1 --2- 172.21.15.90:0/3965610456 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5604441da3b0 0x5604441d7fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:44.297 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.295+0000 7f4a03400640 1 -- 172.21.15.90:0/3965610456 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5604441da3b0 msgr2=0x5604441d7fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:39:44.297 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.295+0000 7f4a03400640 1 --2- 172.21.15.90:0/3965610456 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5604441da3b0 0x5604441d7fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:44.297 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.295+0000 7f4a03400640 1 -- 172.21.15.90:0/3965610456 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5604441d0eb0 msgr2=0x5604441d7550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:39:44.298 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.295+0000 7f4a03400640 1 --2- 172.21.15.90:0/3965610456 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5604441d0eb0 0x5604441d7550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:44.298 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.295+0000 7f4a03400640 1 -- 172.21.15.90:0/3965610456 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560444100e80 con 0x5604441d17d0 2024-04-02T14:39:44.298 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.295+0000 7f4a09200640 1 --2- 172.21.15.90:0/3965610456 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5604441da3b0 0x5604441d7fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:39:44.298 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.295+0000 7f4a03400640 1 --2- 172.21.15.90:0/3965610456 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5604441d17d0 0x5604441d7a90 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f49f00cdc50 tx=0x7f49f009baa0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=55c008010d9c31d8 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:39:44.298 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.295+0000 7f49ee000640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:39:44.299 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.296+0000 7f49ee000640 1 -- 172.21.15.90:0/3965610456 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49f0021040 con 0x5604441d17d0 2024-04-02T14:39:44.299 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.296+0000 7f49ee000640 1 -- 172.21.15.90:0/3965610456 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f49f0098510 con 0x5604441d17d0 2024-04-02T14:39:44.299 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.296+0000 7f4a0dfc7740 1 -- 172.21.15.90:0/3965610456 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f49fc004cc0 con 0x5604441d17d0 2024-04-02T14:39:44.299 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.296+0000 7f49ee000640 1 -- 172.21.15.90:0/3965610456 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f49f009bea0 con 0x5604441d17d0 2024-04-02T14:39:44.299 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.296+0000 7f4a0dfc7740 10 client.26951 fetch_fsmap learned FSMap version 27 2024-04-02T14:39:44.299 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.296+0000 7f4a0dfc7740 1 -- 172.21.15.90:0/3965610456 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x560443e4a210 con 0x5604441d17d0 2024-04-02T14:39:44.299 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.297+0000 7f49ee000640 1 -- 172.21.15.90:0/3965610456 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f49fc004cc0 con 0x5604441d17d0 2024-04-02T14:39:44.300 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.298+0000 7f49ee000640 1 -- 172.21.15.90:0/3965610456 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f49f0024060 con 0x5604441d17d0 2024-04-02T14:39:44.300 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.298+0000 7f4a0dfc7740 10 client.26951 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:39:44.300 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.298+0000 7f4a0dfc7740 10 client.26951 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-02T14:39:44.300 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.298+0000 7f4a0dfc7740 20 client.26951 populate_metadata read hostname 'smithi090' 2024-04-02T14:39:44.300 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.298+0000 7f4a0dfc7740 1 --2- 172.21.15.90:0/3965610456 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5604441ea860 0x5604441eccb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:44.300 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.298+0000 7f4a0dfc7740 4 client.26951 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:39:44.300 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.298+0000 7f4a0dfc7740 1 -- 172.21.15.90:0/3965610456 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5604440e5cd0 con 0x5604441ea860 2024-04-02T14:39:44.300 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.298+0000 7f4a03e00640 1 --2- 172.21.15.90:0/3965610456 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5604441ea860 0x5604441eccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:44.301 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.299+0000 7f49ee000640 10 client.26951 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:39:44.301 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.299+0000 7f4a03e00640 1 --2- 172.21.15.90:0/3965610456 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5604441ea860 0x5604441eccb0 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=58128b59f2acb6f4 server_cookie=9e0d1dcee60df880 in_seq=0 out_seq=0 2024-04-02T14:39:44.301 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.300+0000 7f49ee000640 1 -- 172.21.15.90:0/3965610456 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5604440e5cd0 con 0x5604441ea860 2024-04-02T14:39:44.301 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.300+0000 7f49ee000640 10 client.26951 handle_command_reply: tid=0 2024-04-02T14:39:44.302 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.300+0000 7f4a0dfc7740 1 client.26951 shutdown 2024-04-02T14:39:44.302 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.300+0000 7f4a0dfc7740 1 -- 172.21.15.90:0/3965610456 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5604441d17d0 msgr2=0x5604441d7a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:44.302 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.300+0000 7f4a0dfc7740 1 --2- 172.21.15.90:0/3965610456 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5604441d17d0 0x5604441d7a90 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f49f00cdc50 tx=0x7f49f009baa0 comp rx=0 tx=0).stop 2024-04-02T14:39:44.303 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.300+0000 7f4a0dfc7740 1 -- 172.21.15.90:0/3965610456 shutdown_connections 2024-04-02T14:39:44.303 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.300+0000 7f4a0dfc7740 1 --2- 172.21.15.90:0/3965610456 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5604441ea860 0x5604441eccb0 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:44.303 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.300+0000 7f4a0dfc7740 1 --2- 172.21.15.90:0/3965610456 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5604441da3b0 0x5604441d7fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:44.303 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.300+0000 7f4a0dfc7740 1 --2- 172.21.15.90:0/3965610456 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5604441d17d0 0x5604441d7a90 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:44.303 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.300+0000 7f4a0dfc7740 1 --2- 172.21.15.90:0/3965610456 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5604441d0eb0 0x5604441d7550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:44.303 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.300+0000 7f49ee000640 0 client.26951 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:39:44.303 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.300+0000 7f4a0dfc7740 1 -- 172.21.15.90:0/3965610456 >> 172.21.15.90:0/3965610456 conn(0x5604441cb5e0 msgr2=0x5604441cd330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:39:44.303 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.301+0000 7f4a0dfc7740 1 -- 172.21.15.90:0/3965610456 shutdown_connections 2024-04-02T14:39:44.303 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.301+0000 7f4a0dfc7740 1 -- 172.21.15.90:0/3965610456 wait complete. 2024-04-02T14:39:44.303 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.301+0000 7f4a0dfc7740 20 client.26951 trim_cache size 0 max 16384 2024-04-02T14:39:44.328 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.326+0000 7f4a0dfc7740 1 Processor -- start 2024-04-02T14:39:44.328 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.326+0000 7f4a0dfc7740 1 -- start start 2024-04-02T14:39:44.328 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.326+0000 7f4a0dfc7740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5604441d0eb0 0x5604441d9c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:44.328 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.326+0000 7f4a0dfc7740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5604441d17d0 0x5604441e6060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:44.329 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.326+0000 7f4a0dfc7740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5604441da3b0 0x5604441d7c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:44.329 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.326+0000 7f4a0dfc7740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f4a041ab560 con 0x5604441d0eb0 2024-04-02T14:39:44.329 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.326+0000 7f4a0dfc7740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f4a041ad0a0 con 0x5604441da3b0 2024-04-02T14:39:44.329 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.326+0000 7f4a0dfc7740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f4a041abc30 con 0x5604441d17d0 2024-04-02T14:39:44.330 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.327+0000 7f4a03400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5604441d17d0 0x5604441e6060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:44.330 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.327+0000 7f4a03400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5604441d17d0 0x5604441e6060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:55980/0 (socket says 172.21.15.90:55980) 2024-04-02T14:39:44.330 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.327+0000 7f4a03400640 1 -- 172.21.15.90:0/608707145 learned_addr learned my addr 172.21.15.90:0/608707145 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:39:44.330 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.327+0000 7f4a03e00640 1 --2- 172.21.15.90:0/608707145 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5604441d0eb0 0x5604441d9c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:44.330 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.327+0000 7f4a09200640 1 --2- 172.21.15.90:0/608707145 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5604441da3b0 0x5604441d7c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:44.330 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.327+0000 7f4a03400640 1 -- 172.21.15.90:0/608707145 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5604441da3b0 msgr2=0x5604441d7c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:39:44.330 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.327+0000 7f4a03400640 1 --2- 172.21.15.90:0/608707145 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5604441da3b0 0x5604441d7c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:44.330 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.327+0000 7f4a03400640 1 -- 172.21.15.90:0/608707145 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5604441d0eb0 msgr2=0x5604441d9c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:39:44.330 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.327+0000 7f4a03400640 1 --2- 172.21.15.90:0/608707145 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5604441d0eb0 0x5604441d9c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:44.331 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.328+0000 7f4a03400640 1 -- 172.21.15.90:0/608707145 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560443f16620 con 0x5604441d17d0 2024-04-02T14:39:44.331 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.328+0000 7f4a09200640 1 --2- 172.21.15.90:0/608707145 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5604441da3b0 0x5604441d7c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:39:44.331 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.328+0000 7f4a03e00640 1 --2- 172.21.15.90:0/608707145 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5604441d0eb0 0x5604441d9c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:39:44.331 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.328+0000 7f4a03400640 1 --2- 172.21.15.90:0/608707145 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5604441d17d0 0x5604441e6060 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f49f009bba0 tx=0x7f49f009a3b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5bb169a74a5aa592 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:39:44.331 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.328+0000 7f49f8c00640 1 -- 172.21.15.90:0/608707145 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49f0021070 con 0x5604441d17d0 2024-04-02T14:39:44.331 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.328+0000 7f49f8c00640 1 -- 172.21.15.90:0/608707145 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f49f00a0250 con 0x5604441d17d0 2024-04-02T14:39:44.331 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.328+0000 7f49f8c00640 1 -- 172.21.15.90:0/608707145 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49f0097e50 con 0x5604441d17d0 2024-04-02T14:39:44.332 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.329+0000 7f4a0dfc7740 1 -- 172.21.15.90:0/608707145 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5604441d17d0 msgr2=0x5604441e6060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:44.332 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.329+0000 7f4a0dfc7740 1 --2- 172.21.15.90:0/608707145 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5604441d17d0 0x5604441e6060 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f49f009bba0 tx=0x7f49f009a3b0 comp rx=0 tx=0).stop 2024-04-02T14:39:44.332 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.329+0000 7f4a0dfc7740 1 -- 172.21.15.90:0/608707145 shutdown_connections 2024-04-02T14:39:44.332 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.329+0000 7f4a0dfc7740 1 --2- 172.21.15.90:0/608707145 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5604441da3b0 0x5604441d7c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:44.332 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.329+0000 7f4a0dfc7740 1 --2- 172.21.15.90:0/608707145 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5604441d17d0 0x5604441e6060 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:44.332 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.329+0000 7f4a0dfc7740 1 --2- 172.21.15.90:0/608707145 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5604441d0eb0 0x5604441d9c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:44.332 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.329+0000 7f4a0dfc7740 1 -- 172.21.15.90:0/608707145 >> 172.21.15.90:0/608707145 conn(0x5604441cb5e0 msgr2=0x5604441d7510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:39:44.332 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.330+0000 7f4a0dfc7740 1 -- 172.21.15.90:0/608707145 shutdown_connections 2024-04-02T14:39:44.333 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.330+0000 7f4a0dfc7740 1 -- 172.21.15.90:0/608707145 wait complete. 2024-04-02T14:39:44.333 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.331+0000 7f4a0dfc7740 1 -- start start 2024-04-02T14:39:44.333 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.331+0000 7f4a0dfc7740 1 Processor -- start 2024-04-02T14:39:44.333 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.332+0000 7f4a0dfc7740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5604441d0eb0 0x5604441de0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:44.333 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.332+0000 7f4a0dfc7740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5604441d17d0 0x5604441de620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:44.334 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.332+0000 7f4a0dfc7740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5604441da3b0 0x5604441deb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:44.334 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.332+0000 7f4a0dfc7740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f4a041ac9d0 con 0x5604441d17d0 2024-04-02T14:39:44.334 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.332+0000 7f4a0dfc7740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f4a041ac300 con 0x5604441d0eb0 2024-04-02T14:39:44.334 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.332+0000 7f4a0dfc7740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f4a041a9540 con 0x5604441da3b0 2024-04-02T14:39:44.334 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.332+0000 7f4a0dfc7740 10 client.0 _ll_register_callbacks cb 0x7f49f0098140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:39:44.335 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.332+0000 7f4a03400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5604441d17d0 0x5604441de620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:44.335 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.332+0000 7f4a03400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5604441d17d0 0x5604441de620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:37046/0 (socket says 172.21.15.90:37046) 2024-04-02T14:39:44.335 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.332+0000 7f4a03e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5604441d0eb0 0x5604441de0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:44.335 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.332+0000 7f4a03400640 1 -- 172.21.15.90:0/4003104068 learned_addr learned my addr 172.21.15.90:0/4003104068 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:39:44.335 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.333+0000 7f4a03e00640 1 -- 172.21.15.90:0/4003104068 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5604441da3b0 msgr2=0x5604441deb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:39:44.335 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.333+0000 7f4a09200640 1 --2- 172.21.15.90:0/4003104068 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5604441da3b0 0x5604441deb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:44.335 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.333+0000 7f4a03e00640 1 --2- 172.21.15.90:0/4003104068 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5604441da3b0 0x5604441deb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:44.335 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.333+0000 7f4a03e00640 1 -- 172.21.15.90:0/4003104068 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5604441d17d0 msgr2=0x5604441de620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:39:44.336 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.333+0000 7f4a03e00640 1 --2- 172.21.15.90:0/4003104068 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5604441d17d0 0x5604441de620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:44.336 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.333+0000 7f4a03e00640 1 -- 172.21.15.90:0/4003104068 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56044424ffe0 con 0x5604441d0eb0 2024-04-02T14:39:44.336 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.333+0000 7f4a09200640 1 --2- 172.21.15.90:0/4003104068 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5604441da3b0 0x5604441deb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:39:44.336 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.333+0000 7f4a03400640 1 --2- 172.21.15.90:0/4003104068 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5604441d17d0 0x5604441de620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:39:44.336 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.334+0000 7f4a03e00640 1 --2- 172.21.15.90:0/4003104068 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5604441d0eb0 0x5604441de0e0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f49f403d270 tx=0x7f49f40044e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=99fdc3f95ea747 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:39:44.337 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.334+0000 7f49ee000640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:39:44.337 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.334+0000 7f49ee000640 1 -- 172.21.15.90:0/4003104068 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49f403de60 con 0x5604441d0eb0 2024-04-02T14:39:44.337 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.334+0000 7f49ee000640 1 -- 172.21.15.90:0/4003104068 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f49f4061350 con 0x5604441d0eb0 2024-04-02T14:39:44.337 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.334+0000 7f4a0dfc7740 1 -- 172.21.15.90:0/4003104068 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f49f00a0250 con 0x5604441d0eb0 2024-04-02T14:39:44.338 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.334+0000 7f49ee000640 1 -- 172.21.15.90:0/4003104068 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f49f4071050 con 0x5604441d0eb0 2024-04-02T14:39:44.338 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.335+0000 7f4a0dfc7740 10 client.26954 fetch_fsmap learned FSMap version 27 2024-04-02T14:39:44.338 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.335+0000 7f4a0dfc7740 1 -- 172.21.15.90:0/4003104068 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x560444251240 con 0x5604441d0eb0 2024-04-02T14:39:44.339 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.335+0000 7f49ee000640 1 -- 172.21.15.90:0/4003104068 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f49f00a0250 con 0x5604441d0eb0 2024-04-02T14:39:44.339 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.335+0000 7f49ee000640 1 -- 172.21.15.90:0/4003104068 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f49f4003c80 con 0x5604441d0eb0 2024-04-02T14:39:44.339 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.335+0000 7f4a0dfc7740 10 client.26954 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:39:44.339 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.335+0000 7f4a0dfc7740 10 client.26954 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-02T14:39:44.339 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.335+0000 7f4a0dfc7740 20 client.26954 populate_metadata read hostname 'smithi090' 2024-04-02T14:39:44.339 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.335+0000 7f4a0dfc7740 1 --2- 172.21.15.90:0/4003104068 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x560444251910 0x560444253d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:39:44.339 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.335+0000 7f4a0dfc7740 4 client.26954 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-02T14:39:44.339 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.335+0000 7f4a0dfc7740 1 -- 172.21.15.90:0/4003104068 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5604441432b0 con 0x560444251910 2024-04-02T14:39:44.339 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.335+0000 7f4a03400640 1 --2- 172.21.15.90:0/4003104068 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x560444251910 0x560444253d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:39:44.339 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.336+0000 7f4a03400640 1 --2- 172.21.15.90:0/4003104068 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x560444251910 0x560444253d30 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6a9c91376f931625 server_cookie=c20dd7f52d86235c in_seq=0 out_seq=0 2024-04-02T14:39:44.339 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:44.336+0000 7f49ee000640 10 client.26954 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:39:44.360 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:39:44 smithi092 ceph-mon[36497]: pgmap v148: 97 pgs: 97 active+clean; 3.3 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 155 KiB/s rd, 31 MiB/s wr, 1.77k op/s 2024-04-02T14:39:44.436 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:39:44 smithi177 ceph-mon[36431]: pgmap v148: 97 pgs: 97 active+clean; 3.3 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 155 KiB/s rd, 31 MiB/s wr, 1.77k op/s 2024-04-02T14:39:44.439 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:39:44 smithi090 ceph-mon[31674]: pgmap v148: 97 pgs: 97 active+clean; 3.3 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 155 KiB/s rd, 31 MiB/s wr, 1.77k op/s 2024-04-02T14:39:46.360 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:39:46 smithi092 ceph-mon[36497]: pgmap v149: 97 pgs: 97 active+clean; 3.3 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 284 KiB/s rd, 35 MiB/s wr, 2.12k op/s 2024-04-02T14:39:46.436 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:39:46 smithi177 ceph-mon[36431]: pgmap v149: 97 pgs: 97 active+clean; 3.3 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 284 KiB/s rd, 35 MiB/s wr, 2.12k op/s 2024-04-02T14:39:46.438 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:39:46 smithi090 ceph-mon[31674]: pgmap v149: 97 pgs: 97 active+clean; 3.3 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 284 KiB/s rd, 35 MiB/s wr, 2.12k op/s 2024-04-02T14:39:46.831 INFO:teuthology.orchestra.run.smithi090.stderr:progress: 255.0 s, 2730.4 tps, lat 36.333 ms stddev 43.800 2024-04-02T14:39:48.360 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:39:48 smithi092 ceph-mon[36497]: pgmap v150: 97 pgs: 97 active+clean; 3.3 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 147 KiB/s rd, 9.5 MiB/s wr, 702 op/s 2024-04-02T14:39:48.436 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:39:48 smithi177 ceph-mon[36431]: pgmap v150: 97 pgs: 97 active+clean; 3.3 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 147 KiB/s rd, 9.5 MiB/s wr, 702 op/s 2024-04-02T14:39:48.438 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:39:48 smithi090 ceph-mon[31674]: pgmap v150: 97 pgs: 97 active+clean; 3.3 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 147 KiB/s rd, 9.5 MiB/s wr, 702 op/s 2024-04-02T14:39:50.360 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:39:50 smithi092 ceph-mon[36497]: pgmap v151: 97 pgs: 97 active+clean; 3.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 11 MiB/s wr, 862 op/s 2024-04-02T14:39:50.436 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:39:50 smithi177 ceph-mon[36431]: pgmap v151: 97 pgs: 97 active+clean; 3.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 11 MiB/s wr, 862 op/s 2024-04-02T14:39:50.438 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:39:50 smithi090 ceph-mon[31674]: pgmap v151: 97 pgs: 97 active+clean; 3.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 11 MiB/s wr, 862 op/s 2024-04-02T14:39:51.751 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:51.749+0000 7f49ee000640 1 -- 172.21.15.90:0/4003104068 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5604441432b0 con 0x560444251910 2024-04-02T14:39:51.751 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:51.749+0000 7f49ee000640 10 client.26954 handle_command_reply: tid=0 2024-04-02T14:39:51.751 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:51.749+0000 7f4a0dfc7740 1 client.26954 shutdown 2024-04-02T14:39:51.752 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:51.750+0000 7f4a0dfc7740 1 -- 172.21.15.90:0/4003104068 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5604441d0eb0 msgr2=0x5604441de0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:51.752 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:51.750+0000 7f4a0dfc7740 1 --2- 172.21.15.90:0/4003104068 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5604441d0eb0 0x5604441de0e0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f49f403d270 tx=0x7f49f40044e0 comp rx=0 tx=0).stop 2024-04-02T14:39:51.753 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:51.750+0000 7f4a0dfc7740 1 -- 172.21.15.90:0/4003104068 shutdown_connections 2024-04-02T14:39:51.753 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:51.750+0000 7f4a0dfc7740 1 --2- 172.21.15.90:0/4003104068 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x560444251910 0x560444253d30 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:51.753 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:51.750+0000 7f4a0dfc7740 1 --2- 172.21.15.90:0/4003104068 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5604441da3b0 0x5604441deb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:51.753 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:51.750+0000 7f4a0dfc7740 1 --2- 172.21.15.90:0/4003104068 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5604441d17d0 0x5604441de620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:51.753 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:51.750+0000 7f4a0dfc7740 1 --2- 172.21.15.90:0/4003104068 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5604441d0eb0 0x5604441de0e0 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:51.753 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:51.750+0000 7f4a0dfc7740 1 -- 172.21.15.90:0/4003104068 >> 172.21.15.90:0/4003104068 conn(0x5604441cb5e0 msgr2=0x5604441cd2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:39:51.753 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:51.750+0000 7f49ee000640 0 client.26954 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:39:51.753 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:51.750+0000 7f4a0dfc7740 1 -- 172.21.15.90:0/4003104068 shutdown_connections 2024-04-02T14:39:51.753 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:51.750+0000 7f4a0dfc7740 1 -- 172.21.15.90:0/4003104068 wait complete. 2024-04-02T14:39:51.753 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:51.751+0000 7f4a0dfc7740 20 client.26954 trim_cache size 0 max 16384 2024-04-02T14:39:51.754 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:{ 2024-04-02T14:39:51.754 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "message": "", 2024-04-02T14:39:51.754 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "return_code": 0 2024-04-02T14:39:51.754 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:} 2024-04-02T14:39:51.755 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:51.751+0000 7f4a0b200640 1 -- 172.21.15.90:0/2026460957 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f49e80a3f70 msgr2=0x7f49e80a6430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:51.755 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:51.751+0000 7f4a0b200640 1 --2- 172.21.15.90:0/2026460957 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f49e80a3f70 0x7f49e80a6430 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f49f4006010 tx=0x7f49f403f040 comp rx=0 tx=0).stop 2024-04-02T14:39:51.755 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:51.751+0000 7f4a0b200640 1 -- 172.21.15.90:0/2026460957 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4a04105020 msgr2=0x7f4a04107260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:39:51.755 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:51.751+0000 7f4a0b200640 1 --2- 172.21.15.90:0/2026460957 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4a04105020 0x7f4a04107260 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f49f000d770 tx=0x7f49f000dc30 comp rx=0 tx=0).stop 2024-04-02T14:39:51.755 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:51.752+0000 7f4a0b200640 1 -- 172.21.15.90:0/2026460957 shutdown_connections 2024-04-02T14:39:51.755 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:51.752+0000 7f4a0b200640 1 --2- 172.21.15.90:0/2026460957 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f49e80a3f70 0x7f49e80a6430 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:51.755 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:51.752+0000 7f4a0b200640 1 --2- 172.21.15.90:0/2026460957 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4a0419bc70 0x7f4a0419c120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:51.755 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:51.752+0000 7f4a0b200640 1 --2- 172.21.15.90:0/2026460957 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4a04105020 0x7f4a04107260 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:51.755 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:51.752+0000 7f4a0b200640 1 --2- 172.21.15.90:0/2026460957 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4a040fff50 0x7f4a04106d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:39:51.755 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:51.752+0000 7f4a0b200640 1 -- 172.21.15.90:0/2026460957 >> 172.21.15.90:0/2026460957 conn(0x7f4a04076b90 msgr2=0x7f4a04102ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:39:51.755 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:51.752+0000 7f4a0b200640 1 -- 172.21.15.90:0/2026460957 shutdown_connections 2024-04-02T14:39:51.755 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:39:51.752+0000 7f4a0b200640 1 -- 172.21.15.90:0/2026460957 wait complete. 2024-04-02T14:39:51.831 INFO:teuthology.orchestra.run.smithi090.stderr:progress: 260.0 s, 1548.6 tps, lat 60.906 ms stddev 184.834 2024-04-02T14:39:52.360 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:39:52 smithi092 ceph-mon[36497]: pgmap v152: 97 pgs: 97 active+clean; 3.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 10 MiB/s wr, 802 op/s 2024-04-02T14:39:52.436 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:39:52 smithi177 ceph-mon[36431]: pgmap v152: 97 pgs: 97 active+clean; 3.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 10 MiB/s wr, 802 op/s 2024-04-02T14:39:52.438 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:39:52 smithi090 ceph-mon[31674]: pgmap v152: 97 pgs: 97 active+clean; 3.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 10 MiB/s wr, 802 op/s 2024-04-02T14:39:53.360 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:39:53 smithi092 ceph-mon[36497]: scrub summary: idle+waiting paths [/] 2024-04-02T14:39:53.436 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:39:53 smithi177 ceph-mon[36431]: scrub summary: idle+waiting paths [/] 2024-04-02T14:39:53.438 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:39:53 smithi090 ceph-mon[31674]: scrub summary: idle+waiting paths [/] 2024-04-02T14:39:54.363 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:39:54 smithi092 ceph-mon[36497]: pgmap v153: 97 pgs: 97 active+clean; 3.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 8.1 MiB/s wr, 665 op/s 2024-04-02T14:39:54.436 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:39:54 smithi177 ceph-mon[36431]: pgmap v153: 97 pgs: 97 active+clean; 3.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 8.1 MiB/s wr, 665 op/s 2024-04-02T14:39:54.438 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:39:54 smithi090 ceph-mon[31674]: pgmap v153: 97 pgs: 97 active+clean; 3.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 8.1 MiB/s wr, 665 op/s 2024-04-02T14:39:56.360 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:39:56 smithi092 ceph-mon[36497]: pgmap v154: 97 pgs: 97 active+clean; 3.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 165 KiB/s rd, 11 MiB/s wr, 927 op/s 2024-04-02T14:39:56.436 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:39:56 smithi177 ceph-mon[36431]: pgmap v154: 97 pgs: 97 active+clean; 3.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 165 KiB/s rd, 11 MiB/s wr, 927 op/s 2024-04-02T14:39:56.438 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:39:56 smithi090 ceph-mon[31674]: pgmap v154: 97 pgs: 97 active+clean; 3.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 165 KiB/s rd, 11 MiB/s wr, 927 op/s 2024-04-02T14:39:56.831 INFO:teuthology.orchestra.run.smithi090.stderr:progress: 265.0 s, 2793.4 tps, lat 36.565 ms stddev 79.490 2024-04-02T14:39:57.436 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:39:57 smithi177 ceph-mon[36431]: scrub summary: idle 2024-04-02T14:39:57.438 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:39:57 smithi090 ceph-mon[31674]: scrub summary: idle 2024-04-02T14:39:57.610 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:39:57 smithi092 ceph-mon[36497]: scrub summary: idle 2024-04-02T14:39:58.436 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:39:58 smithi177 ceph-mon[36431]: pgmap v155: 97 pgs: 97 active+clean; 3.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 6.9 MiB/s wr, 567 op/s 2024-04-02T14:39:58.438 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:39:58 smithi090 ceph-mon[31674]: pgmap v155: 97 pgs: 97 active+clean; 3.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 6.9 MiB/s wr, 567 op/s 2024-04-02T14:39:58.610 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:39:58 smithi092 ceph-mon[36497]: pgmap v155: 97 pgs: 97 active+clean; 3.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 6.9 MiB/s wr, 567 op/s 2024-04-02T14:40:00.436 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:40:00 smithi177 ceph-mon[36431]: pgmap v156: 97 pgs: 97 active+clean; 3.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 9.0 MiB/s wr, 752 op/s 2024-04-02T14:40:00.436 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:40:00 smithi177 ceph-mon[36431]: overall HEALTH_OK 2024-04-02T14:40:00.438 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:40:00 smithi090 ceph-mon[31674]: pgmap v156: 97 pgs: 97 active+clean; 3.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 9.0 MiB/s wr, 752 op/s 2024-04-02T14:40:00.438 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:40:00 smithi090 ceph-mon[31674]: overall HEALTH_OK 2024-04-02T14:40:00.610 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:40:00 smithi092 ceph-mon[36497]: pgmap v156: 97 pgs: 97 active+clean; 3.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 9.0 MiB/s wr, 752 op/s 2024-04-02T14:40:00.610 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:40:00 smithi092 ceph-mon[36497]: overall HEALTH_OK 2024-04-02T14:40:01.831 INFO:teuthology.orchestra.run.smithi090.stderr:progress: 270.0 s, 2201.1 tps, lat 43.962 ms stddev 68.081 2024-04-02T14:40:02.412 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:40:02 smithi092 ceph-mon[36497]: pgmap v157: 97 pgs: 97 active+clean; 3.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 9.4 MiB/s wr, 741 op/s 2024-04-02T14:40:02.435 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:40:02 smithi177 ceph-mon[36431]: pgmap v157: 97 pgs: 97 active+clean; 3.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 9.4 MiB/s wr, 741 op/s 2024-04-02T14:40:02.438 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:40:02 smithi090 ceph-mon[31674]: pgmap v157: 97 pgs: 97 active+clean; 3.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 9.4 MiB/s wr, 741 op/s 2024-04-02T14:40:04.435 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:40:04 smithi177 ceph-mon[36431]: pgmap v158: 97 pgs: 97 active+clean; 3.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 7.0 MiB/s wr, 594 op/s 2024-04-02T14:40:04.438 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:40:04 smithi090 ceph-mon[31674]: pgmap v158: 97 pgs: 97 active+clean; 3.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 7.0 MiB/s wr, 594 op/s 2024-04-02T14:40:04.611 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:40:04 smithi092 ceph-mon[36497]: pgmap v158: 97 pgs: 97 active+clean; 3.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 7.0 MiB/s wr, 594 op/s 2024-04-02T14:40:04.929 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.927+0000 7f80d1200640 1 -- 172.21.15.90:0/4156637863 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f80cc074850 msgr2=0x7f80cc10ddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:04.929 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.927+0000 7f80d1200640 1 --2- 172.21.15.90:0/4156637863 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f80cc074850 0x7f80cc10ddb0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f80b8009900 tx=0x7f80b802bcc0 comp rx=0 tx=0).stop 2024-04-02T14:40:04.929 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.927+0000 7f80d1200640 1 -- 172.21.15.90:0/4156637863 shutdown_connections 2024-04-02T14:40:04.929 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.927+0000 7f80d1200640 1 --2- 172.21.15.90:0/4156637863 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f80cc10e2f0 0x7f80cc110740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:04.929 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.927+0000 7f80d1200640 1 --2- 172.21.15.90:0/4156637863 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f80cc074850 0x7f80cc10ddb0 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:04.929 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.927+0000 7f80d1200640 1 --2- 172.21.15.90:0/4156637863 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f80cc073f30 0x7f80cc074310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:04.929 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.927+0000 7f80d1200640 1 -- 172.21.15.90:0/4156637863 >> 172.21.15.90:0/4156637863 conn(0x7f80cc0fd5c0 msgr2=0x7f80cc0ff9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:40:04.930 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.927+0000 7f80d1200640 1 -- 172.21.15.90:0/4156637863 shutdown_connections 2024-04-02T14:40:04.930 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.927+0000 7f80d1200640 1 -- 172.21.15.90:0/4156637863 wait complete. 2024-04-02T14:40:04.931 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.928+0000 7f80d1200640 1 Processor -- start 2024-04-02T14:40:04.931 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.929+0000 7f80d1200640 1 -- start start 2024-04-02T14:40:04.931 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.929+0000 7f80d1200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f80cc073f30 0x7f80cc19a5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:04.931 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.929+0000 7f80d1200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f80cc074850 0x7f80cc19ab00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:04.931 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.929+0000 7f80d1200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f80cc10e2f0 0x7f80cc19ee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:04.931 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.929+0000 7f80d1200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f80cc1140a0 con 0x7f80cc073f30 2024-04-02T14:40:04.932 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.929+0000 7f80d1200640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f80cc113f20 con 0x7f80cc074850 2024-04-02T14:40:04.932 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.929+0000 7f80d1200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f80cc114220 con 0x7f80cc10e2f0 2024-04-02T14:40:04.932 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.929+0000 7f80c9e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f80cc073f30 0x7f80cc19a5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:04.932 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.929+0000 7f80c9400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f80cc074850 0x7f80cc19ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:04.932 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.929+0000 7f80c9e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f80cc073f30 0x7f80cc19a5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43824/0 (socket says 172.21.15.90:43824) 2024-04-02T14:40:04.932 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.929+0000 7f80ca800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f80cc10e2f0 0x7f80cc19ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:04.932 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.929+0000 7f80c9400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f80cc074850 0x7f80cc19ab00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:44590/0 (socket says 172.21.15.90:44590) 2024-04-02T14:40:04.932 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.929+0000 7f80c9e00640 1 -- 172.21.15.90:0/925462656 learned_addr learned my addr 172.21.15.90:0/925462656 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:40:04.932 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.930+0000 7f80c9400640 1 -- 172.21.15.90:0/925462656 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f80cc10e2f0 msgr2=0x7f80cc19ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:04.932 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.930+0000 7f80c9400640 1 --2- 172.21.15.90:0/925462656 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f80cc10e2f0 0x7f80cc19ee30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:04.932 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.930+0000 7f80c9400640 1 -- 172.21.15.90:0/925462656 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f80cc073f30 msgr2=0x7f80cc19a5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:04.932 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.930+0000 7f80c9400640 1 --2- 172.21.15.90:0/925462656 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f80cc073f30 0x7f80cc19a5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:04.932 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.930+0000 7f80c9400640 1 -- 172.21.15.90:0/925462656 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80cc19f560 con 0x7f80cc074850 2024-04-02T14:40:04.932 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.930+0000 7f80c9400640 1 --2- 172.21.15.90:0/925462656 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f80cc074850 0x7f80cc19ab00 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f80b802c200 tx=0x7f80b8005e80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:40:04.933 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.930+0000 7f80c2a00640 1 -- 172.21.15.90:0/925462656 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80b8042020 con 0x7f80cc074850 2024-04-02T14:40:04.933 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.930+0000 7f80c2a00640 1 -- 172.21.15.90:0/925462656 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f80b802cdf0 con 0x7f80cc074850 2024-04-02T14:40:04.933 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.930+0000 7f80c2a00640 1 -- 172.21.15.90:0/925462656 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80b8003a50 con 0x7f80cc074850 2024-04-02T14:40:04.933 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.930+0000 7f80d1200640 1 -- 172.21.15.90:0/925462656 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80cc19f7f0 con 0x7f80cc074850 2024-04-02T14:40:04.933 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.931+0000 7f80d1200640 1 -- 172.21.15.90:0/925462656 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80cc1a70e0 con 0x7f80cc074850 2024-04-02T14:40:04.935 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.934+0000 7f80d3fae740 1 Processor -- start 2024-04-02T14:40:04.935 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.934+0000 7f80c2a00640 1 -- 172.21.15.90:0/925462656 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f80b8045020 con 0x7f80cc074850 2024-04-02T14:40:04.935 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.934+0000 7f80d3fae740 1 -- start start 2024-04-02T14:40:04.937 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.934+0000 7f80d3fae740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5644b4c07eb0 0x5644b4c08290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:04.937 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.934+0000 7f80d3fae740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5644b4c087d0 0x5644b4c10e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:04.937 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.934+0000 7f80d3fae740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5644b4c113b0 0x5644b4c137a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:04.937 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.934+0000 7f80d3fae740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5644b4b6cd70 con 0x5644b4c087d0 2024-04-02T14:40:04.937 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.934+0000 7f80d3fae740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x5644b4b152a0 con 0x5644b4c07eb0 2024-04-02T14:40:04.937 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.934+0000 7f80d3fae740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5644b4b68d20 con 0x5644b4c113b0 2024-04-02T14:40:04.938 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.934+0000 7f80ca800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5644b4c07eb0 0x5644b4c08290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:04.938 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.934+0000 7f80ca800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5644b4c07eb0 0x5644b4c08290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:44592/0 (socket says 172.21.15.90:44592) 2024-04-02T14:40:04.938 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.934+0000 7f80ca800640 1 -- 172.21.15.90:0/645027733 learned_addr learned my addr 172.21.15.90:0/645027733 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:40:04.938 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.935+0000 7f80c9e00640 1 --2- 172.21.15.90:0/645027733 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5644b4c087d0 0x5644b4c10e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:04.938 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.935+0000 7f80c9400640 1 --2- 172.21.15.90:0/645027733 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5644b4c113b0 0x5644b4c137a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:04.938 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.935+0000 7f80ca800640 1 -- 172.21.15.90:0/645027733 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5644b4c113b0 msgr2=0x5644b4c137a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:40:04.938 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.935+0000 7f80ca800640 1 --2- 172.21.15.90:0/645027733 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5644b4c113b0 0x5644b4c137a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:04.938 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.935+0000 7f80ca800640 1 -- 172.21.15.90:0/645027733 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5644b4c087d0 msgr2=0x5644b4c10e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:40:04.938 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.935+0000 7f80ca800640 1 --2- 172.21.15.90:0/645027733 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5644b4c087d0 0x5644b4c10e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:04.938 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.935+0000 7f80ca800640 1 -- 172.21.15.90:0/645027733 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5644b4b29840 con 0x5644b4c07eb0 2024-04-02T14:40:04.938 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.935+0000 7f80c9400640 1 --2- 172.21.15.90:0/645027733 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5644b4c113b0 0x5644b4c137a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:40:04.938 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.935+0000 7f80ca800640 1 --2- 172.21.15.90:0/645027733 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5644b4c07eb0 0x5644b4c08290 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f80c4009f50 tx=0x7f80c40118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=47602f935a27494d server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:40:04.938 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.935+0000 7f80c2a00640 1 --2- 172.21.15.90:0/925462656 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f80b00a3e40 0x7f80b00a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:04.939 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.935+0000 7f80b7400640 1 -- 172.21.15.90:0/645027733 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80c4004d20 con 0x5644b4c07eb0 2024-04-02T14:40:04.939 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.936+0000 7f80b7400640 1 -- 172.21.15.90:0/645027733 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f80c4004050 con 0x5644b4c07eb0 2024-04-02T14:40:04.939 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.936+0000 7f80b7400640 1 -- 172.21.15.90:0/645027733 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80c4005240 con 0x5644b4c07eb0 2024-04-02T14:40:04.939 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.936+0000 7f80c2a00640 1 -- 172.21.15.90:0/925462656 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f80b80f2170 con 0x7f80cc074850 2024-04-02T14:40:04.939 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.936+0000 7f80c9e00640 1 --2- 172.21.15.90:0/925462656 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f80b00a3e40 0x7f80b00a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:04.943 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.936+0000 7f80d3fae740 1 -- 172.21.15.90:0/645027733 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5644b4c07eb0 msgr2=0x5644b4c08290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:04.943 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.936+0000 7f80d3fae740 1 --2- 172.21.15.90:0/645027733 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5644b4c07eb0 0x5644b4c08290 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f80c4009f50 tx=0x7f80c40118d0 comp rx=0 tx=0).stop 2024-04-02T14:40:04.943 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.936+0000 7f80c9e00640 1 --2- 172.21.15.90:0/925462656 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f80b00a3e40 0x7f80b00a6300 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x5644b4c07980 tx=0x7f80bc005f90 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:40:04.943 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.936+0000 7f80d3fae740 1 -- 172.21.15.90:0/645027733 shutdown_connections 2024-04-02T14:40:04.944 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.936+0000 7f80d3fae740 1 --2- 172.21.15.90:0/645027733 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5644b4c113b0 0x5644b4c137a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:04.944 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.937+0000 7f80d3fae740 1 --2- 172.21.15.90:0/645027733 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5644b4c087d0 0x5644b4c10e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:04.944 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.937+0000 7f80d3fae740 1 --2- 172.21.15.90:0/645027733 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5644b4c07eb0 0x5644b4c08290 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:04.944 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.937+0000 7f80d3fae740 1 -- 172.21.15.90:0/645027733 >> 172.21.15.90:0/645027733 conn(0x5644b4c025e0 msgr2=0x5644b4c029c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:40:04.944 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.937+0000 7f80d3fae740 1 -- 172.21.15.90:0/645027733 shutdown_connections 2024-04-02T14:40:04.944 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.937+0000 7f80d3fae740 1 -- 172.21.15.90:0/645027733 wait complete. 2024-04-02T14:40:04.944 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.937+0000 7f80d3fae740 1 -- start start 2024-04-02T14:40:04.944 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.937+0000 7f80d3fae740 1 Processor -- start 2024-04-02T14:40:04.944 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.938+0000 7f80d3fae740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5644b4c07eb0 0x5644b4c0e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:04.944 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.938+0000 7f80d3fae740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5644b4c087d0 0x5644b4c0e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:04.944 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.938+0000 7f80d3fae740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5644b4c113b0 0x5644b4c0ee70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:04.944 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.938+0000 7f80d3fae740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5644b4c141a0 con 0x5644b4c087d0 2024-04-02T14:40:04.944 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.938+0000 7f80d3fae740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x5644b4b28660 con 0x5644b4c07eb0 2024-04-02T14:40:04.944 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.938+0000 7f80d3fae740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5644b4c14320 con 0x5644b4c113b0 2024-04-02T14:40:04.945 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.938+0000 7f80d3fae740 10 client.0 _ll_register_callbacks cb 0x5644b4b28050 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:40:04.945 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.938+0000 7f80c9e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5644b4c07eb0 0x5644b4c0e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:04.945 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.938+0000 7f80ca800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5644b4c113b0 0x5644b4c0ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:04.945 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.938+0000 7f80c9400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5644b4c087d0 0x5644b4c0e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:04.945 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.938+0000 7f80c9e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5644b4c07eb0 0x5644b4c0e3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:44608/0 (socket says 172.21.15.90:44608) 2024-04-02T14:40:04.945 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.938+0000 7f80c9e00640 1 -- 172.21.15.90:0/3139929569 learned_addr learned my addr 172.21.15.90:0/3139929569 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:40:04.945 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.938+0000 7f80c9e00640 1 -- 172.21.15.90:0/3139929569 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5644b4c113b0 msgr2=0x5644b4c0ee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:40:04.945 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.938+0000 7f80c9e00640 1 --2- 172.21.15.90:0/3139929569 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5644b4c113b0 0x5644b4c0ee70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:04.945 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.938+0000 7f80c9e00640 1 -- 172.21.15.90:0/3139929569 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5644b4c087d0 msgr2=0x5644b4c0e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:40:04.945 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.938+0000 7f80c9e00640 1 --2- 172.21.15.90:0/3139929569 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5644b4c087d0 0x5644b4c0e930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:04.945 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.938+0000 7f80c9e00640 1 -- 172.21.15.90:0/3139929569 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5644b4b37e80 con 0x5644b4c07eb0 2024-04-02T14:40:04.945 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.938+0000 7f80c9400640 1 --2- 172.21.15.90:0/3139929569 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5644b4c087d0 0x5644b4c0e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:40:04.945 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.939+0000 7f80c9e00640 1 --2- 172.21.15.90:0/3139929569 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5644b4c07eb0 0x5644b4c0e3f0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f80bc01e9f0 tx=0x7f80bc004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e2460ce940deb096 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:40:04.945 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.939+0000 7f80abe00640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:40:04.946 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.939+0000 7f80abe00640 1 -- 172.21.15.90:0/3139929569 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80bc02e060 con 0x5644b4c07eb0 2024-04-02T14:40:04.946 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.939+0000 7f80abe00640 1 -- 172.21.15.90:0/3139929569 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f80bc018030 con 0x5644b4c07eb0 2024-04-02T14:40:04.946 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.939+0000 7f80d3fae740 1 -- 172.21.15.90:0/3139929569 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f80c4004050 con 0x5644b4c07eb0 2024-04-02T14:40:04.946 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.939+0000 7f80abe00640 1 -- 172.21.15.90:0/3139929569 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f80bc00f480 con 0x5644b4c07eb0 2024-04-02T14:40:04.946 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.939+0000 7f80d3fae740 10 client.26966 fetch_fsmap learned FSMap version 27 2024-04-02T14:40:04.946 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.939+0000 7f80d3fae740 1 -- 172.21.15.90:0/3139929569 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5644b4881210 con 0x5644b4c07eb0 2024-04-02T14:40:04.946 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.939+0000 7f80abe00640 1 -- 172.21.15.90:0/3139929569 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f80c4004050 con 0x5644b4c07eb0 2024-04-02T14:40:04.946 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.940+0000 7f80abe00640 1 -- 172.21.15.90:0/3139929569 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f80bc0316e0 con 0x5644b4c07eb0 2024-04-02T14:40:04.946 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.940+0000 7f80d3fae740 10 client.26966 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:40:04.946 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.940+0000 7f80d3fae740 10 client.26966 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-02T14:40:04.946 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.940+0000 7f80d3fae740 20 client.26966 populate_metadata read hostname 'smithi090' 2024-04-02T14:40:04.946 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.940+0000 7f80d3fae740 1 --2- 172.21.15.90:0/3139929569 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5644b4c217e0 0x5644b4c23c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:04.947 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.940+0000 7f80d3fae740 4 client.26966 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:40:04.947 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.940+0000 7f80d3fae740 1 -- 172.21.15.90:0/3139929569 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5644b4b1ccd0 con 0x5644b4c217e0 2024-04-02T14:40:04.947 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.940+0000 7f80c9400640 1 --2- 172.21.15.90:0/3139929569 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5644b4c217e0 0x5644b4c23c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:04.947 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.940+0000 7f80c9400640 1 --2- 172.21.15.90:0/3139929569 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5644b4c217e0 0x5644b4c23c50 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5a687b0299277b34 server_cookie=412ee2bad59442de in_seq=0 out_seq=0 2024-04-02T14:40:04.947 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.940+0000 7f80abe00640 10 client.26966 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:40:04.947 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.941+0000 7f80abe00640 1 -- 172.21.15.90:0/3139929569 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5644b4b1ccd0 con 0x5644b4c217e0 2024-04-02T14:40:04.947 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.941+0000 7f80abe00640 10 client.26966 handle_command_reply: tid=0 2024-04-02T14:40:04.947 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.941+0000 7f80d3fae740 1 client.26966 shutdown 2024-04-02T14:40:04.947 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.942+0000 7f80d3fae740 1 -- 172.21.15.90:0/3139929569 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5644b4c07eb0 msgr2=0x5644b4c0e3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:04.947 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.942+0000 7f80d3fae740 1 --2- 172.21.15.90:0/3139929569 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5644b4c07eb0 0x5644b4c0e3f0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f80bc01e9f0 tx=0x7f80bc004730 comp rx=0 tx=0).stop 2024-04-02T14:40:04.947 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.942+0000 7f80d3fae740 1 -- 172.21.15.90:0/3139929569 shutdown_connections 2024-04-02T14:40:04.947 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.942+0000 7f80d3fae740 1 --2- 172.21.15.90:0/3139929569 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5644b4c217e0 0x5644b4c23c50 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:04.947 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.942+0000 7f80d3fae740 1 --2- 172.21.15.90:0/3139929569 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5644b4c113b0 0x5644b4c0ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:04.947 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.942+0000 7f80d3fae740 1 --2- 172.21.15.90:0/3139929569 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5644b4c087d0 0x5644b4c0e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:04.948 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.942+0000 7f80d3fae740 1 --2- 172.21.15.90:0/3139929569 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5644b4c07eb0 0x5644b4c0e3f0 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:04.948 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.942+0000 7f80abe00640 0 client.26966 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:40:04.948 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.942+0000 7f80d3fae740 1 -- 172.21.15.90:0/3139929569 >> 172.21.15.90:0/3139929569 conn(0x5644b4c025e0 msgr2=0x5644b4c04330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:40:04.948 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.942+0000 7f80d3fae740 1 -- 172.21.15.90:0/3139929569 shutdown_connections 2024-04-02T14:40:04.948 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.942+0000 7f80d3fae740 1 -- 172.21.15.90:0/3139929569 wait complete. 2024-04-02T14:40:04.948 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.942+0000 7f80d3fae740 20 client.26966 trim_cache size 0 max 16384 2024-04-02T14:40:04.962 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.959+0000 7f80d3fae740 1 Processor -- start 2024-04-02T14:40:04.962 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.959+0000 7f80d3fae740 1 -- start start 2024-04-02T14:40:04.962 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.959+0000 7f80d3fae740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5644b4c07eb0 0x5644b4c10b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:04.962 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.959+0000 7f80d3fae740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5644b4c087d0 0x5644b4c1d090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:04.962 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.959+0000 7f80d3fae740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5644b4c113b0 0x5644b4c29560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:04.962 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.959+0000 7f80d3fae740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f80cc1af990 con 0x5644b4c07eb0 2024-04-02T14:40:04.962 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.959+0000 7f80d3fae740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f80cc1b14d0 con 0x5644b4c113b0 2024-04-02T14:40:04.962 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.959+0000 7f80d3fae740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f80cc1b0060 con 0x5644b4c087d0 2024-04-02T14:40:04.962 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.960+0000 7f80c9e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5644b4c07eb0 0x5644b4c10b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:04.962 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.960+0000 7f80c9400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5644b4c087d0 0x5644b4c1d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:04.962 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.960+0000 7f80c9e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5644b4c07eb0 0x5644b4c10b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43844/0 (socket says 172.21.15.90:43844) 2024-04-02T14:40:04.962 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.960+0000 7f80c9e00640 1 -- 172.21.15.90:0/2978862300 learned_addr learned my addr 172.21.15.90:0/2978862300 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:40:04.963 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.960+0000 7f80ca800640 1 --2- 172.21.15.90:0/2978862300 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5644b4c113b0 0x5644b4c29560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:04.963 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.960+0000 7f80c9e00640 1 -- 172.21.15.90:0/2978862300 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5644b4c087d0 msgr2=0x5644b4c1d090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:40:04.963 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.960+0000 7f80c9e00640 1 --2- 172.21.15.90:0/2978862300 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5644b4c087d0 0x5644b4c1d090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:04.963 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.960+0000 7f80c9e00640 1 -- 172.21.15.90:0/2978862300 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5644b4c113b0 msgr2=0x5644b4c29560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:40:04.963 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.960+0000 7f80c9e00640 1 --2- 172.21.15.90:0/2978862300 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5644b4c113b0 0x5644b4c29560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:04.963 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.960+0000 7f80c9e00640 1 -- 172.21.15.90:0/2978862300 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5644b4c13b10 con 0x5644b4c07eb0 2024-04-02T14:40:04.963 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.960+0000 7f80c9e00640 1 --2- 172.21.15.90:0/2978862300 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5644b4c07eb0 0x5644b4c10b20 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f80bc0098e0 tx=0x7f80bc031f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=610eaf1b91db5c40 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:40:04.966 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.961+0000 7f80b6a00640 1 -- 172.21.15.90:0/2978862300 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80bc0313a0 con 0x5644b4c07eb0 2024-04-02T14:40:04.967 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.961+0000 7f80b6a00640 1 -- 172.21.15.90:0/2978862300 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f80bc00f620 con 0x5644b4c07eb0 2024-04-02T14:40:04.967 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.961+0000 7f80b6a00640 1 -- 172.21.15.90:0/2978862300 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80bc029680 con 0x5644b4c07eb0 2024-04-02T14:40:04.967 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.961+0000 7f80d3fae740 1 -- 172.21.15.90:0/2978862300 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5644b4c07eb0 msgr2=0x5644b4c10b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:04.967 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.961+0000 7f80d3fae740 1 --2- 172.21.15.90:0/2978862300 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5644b4c07eb0 0x5644b4c10b20 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f80bc0098e0 tx=0x7f80bc031f00 comp rx=0 tx=0).stop 2024-04-02T14:40:04.967 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.961+0000 7f80d3fae740 1 -- 172.21.15.90:0/2978862300 shutdown_connections 2024-04-02T14:40:04.967 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.961+0000 7f80d3fae740 1 --2- 172.21.15.90:0/2978862300 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5644b4c113b0 0x5644b4c29560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:04.967 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.961+0000 7f80d3fae740 1 --2- 172.21.15.90:0/2978862300 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5644b4c087d0 0x5644b4c1d090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:04.967 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.961+0000 7f80d3fae740 1 --2- 172.21.15.90:0/2978862300 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5644b4c07eb0 0x5644b4c10b20 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:04.967 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.961+0000 7f80d3fae740 1 -- 172.21.15.90:0/2978862300 >> 172.21.15.90:0/2978862300 conn(0x5644b4c025e0 msgr2=0x5644b4c01640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:40:04.967 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.961+0000 7f80d3fae740 1 -- 172.21.15.90:0/2978862300 shutdown_connections 2024-04-02T14:40:04.967 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.962+0000 7f80d3fae740 1 -- 172.21.15.90:0/2978862300 wait complete. 2024-04-02T14:40:04.967 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.962+0000 7f80d3fae740 1 -- start start 2024-04-02T14:40:04.967 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.962+0000 7f80d3fae740 1 Processor -- start 2024-04-02T14:40:04.968 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.962+0000 7f80d3fae740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5644b4c07eb0 0x5644b4c195a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:04.968 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.962+0000 7f80d3fae740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5644b4c087d0 0x5644b4c19ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:04.968 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.962+0000 7f80d3fae740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5644b4c113b0 0x5644b4c16590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:04.968 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.962+0000 7f80d3fae740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f80cc1b0e00 con 0x5644b4c113b0 2024-04-02T14:40:04.968 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.962+0000 7f80d3fae740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f80cc1b0730 con 0x5644b4c07eb0 2024-04-02T14:40:04.968 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.963+0000 7f80d3fae740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f80cc1ad970 con 0x5644b4c087d0 2024-04-02T14:40:04.968 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.963+0000 7f80d3fae740 10 client.0 _ll_register_callbacks cb 0x7f80bc01e8b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:40:04.968 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.963+0000 7f80c9e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5644b4c07eb0 0x5644b4c195a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:04.968 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.963+0000 7f80c9e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5644b4c07eb0 0x5644b4c195a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:44626/0 (socket says 172.21.15.90:44626) 2024-04-02T14:40:04.968 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.963+0000 7f80c9e00640 1 -- 172.21.15.90:0/4077800572 learned_addr learned my addr 172.21.15.90:0/4077800572 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:40:04.968 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.963+0000 7f80ca800640 1 --2- 172.21.15.90:0/4077800572 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5644b4c113b0 0x5644b4c16590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:04.968 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.963+0000 7f80c9400640 1 --2- 172.21.15.90:0/4077800572 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5644b4c087d0 0x5644b4c19ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:04.968 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.963+0000 7f80c9e00640 1 -- 172.21.15.90:0/4077800572 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5644b4c087d0 msgr2=0x5644b4c19ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:40:04.968 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.963+0000 7f80c9e00640 1 --2- 172.21.15.90:0/4077800572 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5644b4c087d0 0x5644b4c19ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:04.969 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.963+0000 7f80c9e00640 1 -- 172.21.15.90:0/4077800572 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5644b4c113b0 msgr2=0x5644b4c16590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:40:04.969 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.963+0000 7f80c9e00640 1 --2- 172.21.15.90:0/4077800572 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5644b4c113b0 0x5644b4c16590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:04.969 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.963+0000 7f80c9e00640 1 -- 172.21.15.90:0/4077800572 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5644b494d620 con 0x5644b4c07eb0 2024-04-02T14:40:04.969 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.963+0000 7f80ca800640 1 --2- 172.21.15.90:0/4077800572 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5644b4c113b0 0x5644b4c16590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:40:04.969 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.963+0000 7f80c9e00640 1 --2- 172.21.15.90:0/4077800572 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5644b4c07eb0 0x5644b4c195a0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f80bc01e880 tx=0x7f80bc031810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cc73573f1effd76a server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:40:04.969 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.963+0000 7f80abe00640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:40:04.969 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.964+0000 7f80abe00640 1 -- 172.21.15.90:0/4077800572 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80bc049060 con 0x5644b4c07eb0 2024-04-02T14:40:04.969 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.964+0000 7f80abe00640 1 -- 172.21.15.90:0/4077800572 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f80bc00fbe0 con 0x5644b4c07eb0 2024-04-02T14:40:04.969 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.964+0000 7f80d3fae740 1 -- 172.21.15.90:0/4077800572 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f80bc00f620 con 0x5644b4c07eb0 2024-04-02T14:40:04.969 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.964+0000 7f80abe00640 1 -- 172.21.15.90:0/4077800572 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f80bc0050b0 con 0x5644b4c07eb0 2024-04-02T14:40:04.969 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.964+0000 7f80d3fae740 10 client.26972 fetch_fsmap learned FSMap version 27 2024-04-02T14:40:04.969 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.964+0000 7f80d3fae740 1 -- 172.21.15.90:0/4077800572 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5644b4c88770 con 0x5644b4c07eb0 2024-04-02T14:40:04.969 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.964+0000 7f80abe00640 1 -- 172.21.15.90:0/4077800572 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f80bc00f620 con 0x5644b4c07eb0 2024-04-02T14:40:04.969 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.964+0000 7f80abe00640 1 -- 172.21.15.90:0/4077800572 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f80bc03b070 con 0x5644b4c07eb0 2024-04-02T14:40:04.969 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.965+0000 7f80d3fae740 10 client.26972 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:40:04.970 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.965+0000 7f80d3fae740 10 client.26972 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-02T14:40:04.970 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.965+0000 7f80d3fae740 20 client.26972 populate_metadata read hostname 'smithi090' 2024-04-02T14:40:04.970 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.965+0000 7f80d3fae740 1 --2- 172.21.15.90:0/4077800572 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5644b4c88d80 0x5644b4c8b170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:04.970 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.965+0000 7f80d3fae740 4 client.26972 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-02T14:40:04.970 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.965+0000 7f80d3fae740 1 -- 172.21.15.90:0/4077800572 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5644b4b7a2b0 con 0x5644b4c88d80 2024-04-02T14:40:04.970 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.965+0000 7f80c9400640 1 --2- 172.21.15.90:0/4077800572 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5644b4c88d80 0x5644b4c8b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:04.970 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.965+0000 7f80c9400640 1 --2- 172.21.15.90:0/4077800572 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5644b4c88d80 0x5644b4c8b170 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=16d63497b6d55504 server_cookie=77150ae030a7c8ca in_seq=0 out_seq=0 2024-04-02T14:40:04.970 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.965+0000 7f80abe00640 10 client.26972 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:40:04.993 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.991+0000 7f80abe00640 1 -- 172.21.15.90:0/4077800572 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5644b4b7a2b0 con 0x5644b4c88d80 2024-04-02T14:40:04.993 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.991+0000 7f80abe00640 10 client.26972 handle_command_reply: tid=0 2024-04-02T14:40:04.993 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.991+0000 7f80d3fae740 1 client.26972 shutdown 2024-04-02T14:40:04.993 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.991+0000 7f80d3fae740 1 -- 172.21.15.90:0/4077800572 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5644b4c07eb0 msgr2=0x5644b4c195a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:04.993 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.991+0000 7f80d3fae740 1 --2- 172.21.15.90:0/4077800572 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5644b4c07eb0 0x5644b4c195a0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f80bc01e880 tx=0x7f80bc031810 comp rx=0 tx=0).stop 2024-04-02T14:40:04.993 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.991+0000 7f80d3fae740 1 -- 172.21.15.90:0/4077800572 shutdown_connections 2024-04-02T14:40:04.993 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.991+0000 7f80d3fae740 1 --2- 172.21.15.90:0/4077800572 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5644b4c88d80 0x5644b4c8b170 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:04.993 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.991+0000 7f80d3fae740 1 --2- 172.21.15.90:0/4077800572 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5644b4c113b0 0x5644b4c16590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:04.993 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.991+0000 7f80d3fae740 1 --2- 172.21.15.90:0/4077800572 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5644b4c087d0 0x5644b4c19ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:04.994 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.991+0000 7f80abe00640 0 client.26972 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:40:04.994 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.991+0000 7f80d3fae740 1 --2- 172.21.15.90:0/4077800572 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5644b4c07eb0 0x5644b4c195a0 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:04.994 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.991+0000 7f80d3fae740 1 -- 172.21.15.90:0/4077800572 >> 172.21.15.90:0/4077800572 conn(0x5644b4c025e0 msgr2=0x5644b4c038e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:40:04.994 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.991+0000 7f80d3fae740 1 -- 172.21.15.90:0/4077800572 shutdown_connections 2024-04-02T14:40:04.994 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.991+0000 7f80d3fae740 1 -- 172.21.15.90:0/4077800572 wait complete. 2024-04-02T14:40:04.994 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.991+0000 7f80d3fae740 20 client.26972 trim_cache size 0 max 16384 2024-04-02T14:40:04.994 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:{ 2024-04-02T14:40:04.994 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "message": "", 2024-04-02T14:40:04.995 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "return_code": 0 2024-04-02T14:40:04.995 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:} 2024-04-02T14:40:04.995 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.993+0000 7f80d1200640 1 -- 172.21.15.90:0/925462656 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f80b00a3e40 msgr2=0x7f80b00a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:04.995 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.993+0000 7f80d1200640 1 --2- 172.21.15.90:0/925462656 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f80b00a3e40 0x7f80b00a6300 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x5644b4c07980 tx=0x7f80bc005f90 comp rx=0 tx=0).stop 2024-04-02T14:40:04.995 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.993+0000 7f80d1200640 1 -- 172.21.15.90:0/925462656 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f80cc074850 msgr2=0x7f80cc19ab00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:04.995 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.993+0000 7f80d1200640 1 --2- 172.21.15.90:0/925462656 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f80cc074850 0x7f80cc19ab00 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f80b802c200 tx=0x7f80b8005e80 comp rx=0 tx=0).stop 2024-04-02T14:40:04.996 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.993+0000 7f80d1200640 1 -- 172.21.15.90:0/925462656 shutdown_connections 2024-04-02T14:40:04.996 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.993+0000 7f80d1200640 1 --2- 172.21.15.90:0/925462656 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f80b00a3e40 0x7f80b00a6300 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:04.996 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.993+0000 7f80d1200640 1 --2- 172.21.15.90:0/925462656 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f80cc10e2f0 0x7f80cc19ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:04.996 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.993+0000 7f80d1200640 1 --2- 172.21.15.90:0/925462656 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f80cc074850 0x7f80cc19ab00 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:04.996 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.993+0000 7f80d1200640 1 --2- 172.21.15.90:0/925462656 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f80cc073f30 0x7f80cc19a5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:04.996 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.993+0000 7f80d1200640 1 -- 172.21.15.90:0/925462656 >> 172.21.15.90:0/925462656 conn(0x7f80cc0fd5c0 msgr2=0x7f80cc0fec70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:40:04.996 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.994+0000 7f80d1200640 1 -- 172.21.15.90:0/925462656 shutdown_connections 2024-04-02T14:40:04.996 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:04.994+0000 7f80d1200640 1 -- 172.21.15.90:0/925462656 wait complete. 2024-04-02T14:40:06.436 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:40:06 smithi177 ceph-mon[36431]: pgmap v159: 97 pgs: 97 active+clean; 3.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 9.1 MiB/s wr, 818 op/s 2024-04-02T14:40:06.437 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:40:06 smithi090 ceph-mon[31674]: pgmap v159: 97 pgs: 97 active+clean; 3.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 9.1 MiB/s wr, 818 op/s 2024-04-02T14:40:06.610 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:40:06 smithi092 ceph-mon[36497]: pgmap v159: 97 pgs: 97 active+clean; 3.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 9.1 MiB/s wr, 818 op/s 2024-04-02T14:40:06.831 INFO:teuthology.orchestra.run.smithi090.stderr:progress: 275.0 s, 213.0 tps, lat 436.243 ms stddev 259.184 2024-04-02T14:40:07.610 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:40:07 smithi092 ceph-mon[36497]: pgmap v160: 97 pgs: 97 active+clean; 3.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 6.3 MiB/s wr, 556 op/s 2024-04-02T14:40:07.685 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:40:07 smithi177 ceph-mon[36431]: pgmap v160: 97 pgs: 97 active+clean; 3.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 6.3 MiB/s wr, 556 op/s 2024-04-02T14:40:07.688 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:40:07 smithi090 ceph-mon[31674]: pgmap v160: 97 pgs: 97 active+clean; 3.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 6.3 MiB/s wr, 556 op/s 2024-04-02T14:40:08.610 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:40:08 smithi092 ceph-mon[36497]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T14:40:08.685 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:40:08 smithi177 ceph-mon[36431]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T14:40:08.688 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:40:08 smithi090 ceph-mon[31674]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T14:40:09.610 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:40:09 smithi092 ceph-mon[36497]: pgmap v161: 97 pgs: 97 active+clean; 3.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 17 MiB/s wr, 1.17k op/s 2024-04-02T14:40:09.685 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:40:09 smithi177 ceph-mon[36431]: pgmap v161: 97 pgs: 97 active+clean; 3.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 17 MiB/s wr, 1.17k op/s 2024-04-02T14:40:09.687 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:40:09 smithi090 ceph-mon[31674]: pgmap v161: 97 pgs: 97 active+clean; 3.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 17 MiB/s wr, 1.17k op/s 2024-04-02T14:40:10.589 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:40:10 smithi090 ceph-mon[31674]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:40:10.589 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:40:10 smithi090 ceph-mon[31674]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T14:40:10.589 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:40:10 smithi090 ceph-mon[31674]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' 2024-04-02T14:40:10.610 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:40:10 smithi092 ceph-mon[36497]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:40:10.610 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:40:10 smithi092 ceph-mon[36497]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T14:40:10.610 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:40:10 smithi092 ceph-mon[36497]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' 2024-04-02T14:40:10.686 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:40:10 smithi177 ceph-mon[36431]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:40:10.686 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:40:10 smithi177 ceph-mon[36431]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T14:40:10.686 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:40:10 smithi177 ceph-mon[36431]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' 2024-04-02T14:40:11.610 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:40:11 smithi092 ceph-mon[36497]: pgmap v162: 97 pgs: 97 active+clean; 3.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 23 MiB/s wr, 1.46k op/s 2024-04-02T14:40:11.685 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:40:11 smithi177 ceph-mon[36431]: pgmap v162: 97 pgs: 97 active+clean; 3.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 23 MiB/s wr, 1.46k op/s 2024-04-02T14:40:11.688 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:40:11 smithi090 ceph-mon[31674]: pgmap v162: 97 pgs: 97 active+clean; 3.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 23 MiB/s wr, 1.46k op/s 2024-04-02T14:40:11.831 INFO:teuthology.orchestra.run.smithi090.stderr:progress: 280.0 s, 1328.6 tps, lat 86.473 ms stddev 389.289 2024-04-02T14:40:12.193 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-02T14:40:12.194 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-02T14:40:12.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.496+0000 7f92ca200640 1 -- 172.21.15.90:0/3393681833 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f92c411b2e0 msgr2=0x7f92c411d6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:12.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.496+0000 7f92ca200640 1 --2- 172.21.15.90:0/3393681833 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f92c411b2e0 0x7f92c411d6d0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f92bc009be0 tx=0x7f92bc02bef0 comp rx=0 tx=0).stop 2024-04-02T14:40:12.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.497+0000 7f92ca200640 1 -- 172.21.15.90:0/3393681833 shutdown_connections 2024-04-02T14:40:12.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.497+0000 7f92ca200640 1 --2- 172.21.15.90:0/3393681833 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f92c411b2e0 0x7f92c411d6d0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:12.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.497+0000 7f92ca200640 1 --2- 172.21.15.90:0/3393681833 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f92c410e350 0x7f92c410e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:12.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.497+0000 7f92ca200640 1 --2- 172.21.15.90:0/3393681833 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f92c4110320 0x7f92c410de10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:12.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.497+0000 7f92ca200640 1 -- 172.21.15.90:0/3393681833 >> 172.21.15.90:0/3393681833 conn(0x7f92c406cbe0 msgr2=0x7f92c406cff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:40:12.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.498+0000 7f92ca200640 1 -- 172.21.15.90:0/3393681833 shutdown_connections 2024-04-02T14:40:12.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.498+0000 7f92ca200640 1 -- 172.21.15.90:0/3393681833 wait complete. 2024-04-02T14:40:12.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.500+0000 7f92ca200640 1 Processor -- start 2024-04-02T14:40:12.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.501+0000 7f92ca200640 1 -- start start 2024-04-02T14:40:12.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.502+0000 7f92ca200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f92c410e350 0x7f92c4118890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:12.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.502+0000 7f92ca200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f92c4110320 0x7f92c4111960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:12.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.502+0000 7f92ca200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f92c411b2e0 0x7f92c4111ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:12.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.502+0000 7f92ca200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f92c4120c40 con 0x7f92c4110320 2024-04-02T14:40:12.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.502+0000 7f92ca200640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f92c4120ac0 con 0x7f92c411b2e0 2024-04-02T14:40:12.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.502+0000 7f92ca200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f92c4120dc0 con 0x7f92c410e350 2024-04-02T14:40:12.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.503+0000 7f92c3400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f92c410e350 0x7f92c4118890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:12.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.503+0000 7f92c3400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f92c410e350 0x7f92c4118890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:48918/0 (socket says 172.21.15.90:48918) 2024-04-02T14:40:12.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.503+0000 7f92c3400640 1 -- 172.21.15.90:0/4116968037 learned_addr learned my addr 172.21.15.90:0/4116968037 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:40:12.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.503+0000 7f92bbe00640 1 --2- 172.21.15.90:0/4116968037 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f92c4110320 0x7f92c4111960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:12.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.503+0000 7f92c3e00640 1 --2- 172.21.15.90:0/4116968037 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f92c411b2e0 0x7f92c4111ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:12.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.503+0000 7f92c3e00640 1 -- 172.21.15.90:0/4116968037 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f92c410e350 msgr2=0x7f92c4118890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:12.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.503+0000 7f92c3e00640 1 --2- 172.21.15.90:0/4116968037 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f92c410e350 0x7f92c4118890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:12.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.503+0000 7f92c3e00640 1 -- 172.21.15.90:0/4116968037 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f92c4110320 msgr2=0x7f92c4111960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:12.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.503+0000 7f92c3e00640 1 --2- 172.21.15.90:0/4116968037 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f92c4110320 0x7f92c4111960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:12.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.503+0000 7f92c3e00640 1 -- 172.21.15.90:0/4116968037 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92c4112460 con 0x7f92c411b2e0 2024-04-02T14:40:12.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.504+0000 7f92c3e00640 1 --2- 172.21.15.90:0/4116968037 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f92c411b2e0 0x7f92c4111ea0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f92bc009b30 tx=0x7f92bc0336b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:40:12.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.505+0000 7f92c0c00640 1 -- 172.21.15.90:0/4116968037 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92bc043020 con 0x7f92c411b2e0 2024-04-02T14:40:12.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.505+0000 7f92c0c00640 1 -- 172.21.15.90:0/4116968037 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f92bc003e50 con 0x7f92c411b2e0 2024-04-02T14:40:12.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.505+0000 7f92c0c00640 1 -- 172.21.15.90:0/4116968037 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92bc03e6b0 con 0x7f92c411b2e0 2024-04-02T14:40:12.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.506+0000 7f92ca200640 1 -- 172.21.15.90:0/4116968037 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f92c41126f0 con 0x7f92c411b2e0 2024-04-02T14:40:12.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.506+0000 7f92ca200640 1 -- 172.21.15.90:0/4116968037 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f92c4116590 con 0x7f92c411b2e0 2024-04-02T14:40:12.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.513+0000 7f92c0c00640 1 -- 172.21.15.90:0/4116968037 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f92bc046020 con 0x7f92c411b2e0 2024-04-02T14:40:12.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.515+0000 7f92ccfd7740 1 Processor -- start 2024-04-02T14:40:12.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.516+0000 7f92ccfd7740 1 -- start start 2024-04-02T14:40:12.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.516+0000 7f92c0c00640 1 --2- 172.21.15.90:0/4116968037 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f92a80a3de0 0x7f92a80a62a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:12.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.516+0000 7f92ccfd7740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f8271407b0 0x55f827140b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:12.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.516+0000 7f92ccfd7740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f8271410d0 0x55f827149770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:12.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.516+0000 7f92ccfd7740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f827149cb0 0x55f82714c0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:12.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.516+0000 7f92c0c00640 1 -- 172.21.15.90:0/4116968037 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f92bc0f2ad0 con 0x7f92c411b2e0 2024-04-02T14:40:12.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.517+0000 7f92c3400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f8271407b0 0x55f827140b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:12.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.517+0000 7f92c3e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f827149cb0 0x55f82714c0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:12.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.517+0000 7f92c3e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f827149cb0 0x55f82714c0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:48934/0 (socket says 172.21.15.90:48934) 2024-04-02T14:40:12.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.517+0000 7f92c3400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f8271407b0 0x55f827140b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:44664/0 (socket says 172.21.15.90:44664) 2024-04-02T14:40:12.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.517+0000 7f92c3e00640 1 -- 172.21.15.90:0/796895488 learned_addr learned my addr 172.21.15.90:0/796895488 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:40:12.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.517+0000 7f92ccfd7740 1 -- 172.21.15.90:0/796895488 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55f8270a4770 con 0x55f8271410d0 2024-04-02T14:40:12.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.518+0000 7f92bbe00640 1 --2- 172.21.15.90:0/4116968037 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f92a80a3de0 0x7f92a80a62a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:12.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.518+0000 7f92ccfd7740 1 -- 172.21.15.90:0/796895488 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55f82704c700 con 0x55f8271407b0 2024-04-02T14:40:12.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.518+0000 7f92ccfd7740 1 -- 172.21.15.90:0/796895488 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55f8270a0720 con 0x55f827149cb0 2024-04-02T14:40:12.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.518+0000 7f92bbe00640 1 --2- 172.21.15.90:0/796895488 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f8271410d0 0x55f827149770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:12.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.519+0000 7f92c3400640 1 -- 172.21.15.90:0/796895488 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f827149cb0 msgr2=0x55f82714c0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:40:12.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.519+0000 7f92c3400640 1 --2- 172.21.15.90:0/796895488 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f827149cb0 0x55f82714c0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:12.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.519+0000 7f92c3400640 1 -- 172.21.15.90:0/796895488 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f8271410d0 msgr2=0x55f827149770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:40:12.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.519+0000 7f92c3400640 1 --2- 172.21.15.90:0/796895488 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f8271410d0 0x55f827149770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:12.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.519+0000 7f92c3400640 1 -- 172.21.15.90:0/796895488 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f826e827d0 con 0x55f8271407b0 2024-04-02T14:40:12.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.520+0000 7f92c3400640 1 --2- 172.21.15.90:0/796895488 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f8271407b0 0x55f827140b90 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f92ac005fd0 tx=0x7f92ac00bbb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a63b6698f2aeaaee server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:40:12.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.520+0000 7f92bbe00640 1 --2- 172.21.15.90:0/4116968037 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f92a80a3de0 0x7f92a80a62a0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x55f827140280 tx=0x7f92b4009580 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:40:12.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.520+0000 7f92b3e00640 1 -- 172.21.15.90:0/796895488 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92ac003b50 con 0x55f8271407b0 2024-04-02T14:40:12.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.520+0000 7f92b3e00640 1 -- 172.21.15.90:0/796895488 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f92ac015070 con 0x55f8271407b0 2024-04-02T14:40:12.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.520+0000 7f92b3e00640 1 -- 172.21.15.90:0/796895488 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92ac004e40 con 0x55f8271407b0 2024-04-02T14:40:12.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.522+0000 7f92ccfd7740 1 -- 172.21.15.90:0/796895488 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f8271407b0 msgr2=0x55f827140b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:12.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.522+0000 7f92ccfd7740 1 --2- 172.21.15.90:0/796895488 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f8271407b0 0x55f827140b90 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f92ac005fd0 tx=0x7f92ac00bbb0 comp rx=0 tx=0).stop 2024-04-02T14:40:12.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.522+0000 7f92ccfd7740 1 -- 172.21.15.90:0/796895488 shutdown_connections 2024-04-02T14:40:12.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.522+0000 7f92ccfd7740 1 --2- 172.21.15.90:0/796895488 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f827149cb0 0x55f82714c0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:12.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.523+0000 7f92ccfd7740 1 --2- 172.21.15.90:0/796895488 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f8271410d0 0x55f827149770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:12.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.523+0000 7f92ccfd7740 1 --2- 172.21.15.90:0/796895488 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f8271407b0 0x55f827140b90 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:12.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.523+0000 7f92ccfd7740 1 -- 172.21.15.90:0/796895488 >> 172.21.15.90:0/796895488 conn(0x55f82713aee0 msgr2=0x55f82713b2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:40:12.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.523+0000 7f92ccfd7740 1 -- 172.21.15.90:0/796895488 shutdown_connections 2024-04-02T14:40:12.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.523+0000 7f92ccfd7740 1 -- 172.21.15.90:0/796895488 wait complete. 2024-04-02T14:40:12.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.524+0000 7f92ccfd7740 1 -- start start 2024-04-02T14:40:12.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.524+0000 7f92ccfd7740 1 Processor -- start 2024-04-02T14:40:12.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.525+0000 7f92ccfd7740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f8271407b0 0x55f827146d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:12.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.525+0000 7f92ccfd7740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f8271410d0 0x55f827147270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:12.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.525+0000 7f92ccfd7740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f827149cb0 0x55f8271477b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:12.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.525+0000 7f92bbe00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f8271410d0 0x55f827147270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:12.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.525+0000 7f92c3400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f8271407b0 0x55f827146d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:12.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.525+0000 7f92bbe00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f8271410d0 0x55f827147270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:44668/0 (socket says 172.21.15.90:44668) 2024-04-02T14:40:12.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.525+0000 7f92c3400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f8271407b0 0x55f827146d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:48948/0 (socket says 172.21.15.90:48948) 2024-04-02T14:40:12.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.526+0000 7f92bbe00640 1 -- 172.21.15.90:0/3648987613 learned_addr learned my addr 172.21.15.90:0/3648987613 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:40:12.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.526+0000 7f92c3e00640 1 --2- 172.21.15.90:0/3648987613 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f827149cb0 0x55f8271477b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:12.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.526+0000 7f92ccfd7740 1 -- 172.21.15.90:0/3648987613 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55f82714caa0 con 0x55f827149cb0 2024-04-02T14:40:12.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.526+0000 7f92ccfd7740 1 -- 172.21.15.90:0/3648987613 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55f82705fac0 con 0x55f8271410d0 2024-04-02T14:40:12.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.526+0000 7f92ccfd7740 1 -- 172.21.15.90:0/3648987613 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55f82714cc20 con 0x55f8271407b0 2024-04-02T14:40:12.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.526+0000 7f92ccfd7740 10 client.0 _ll_register_callbacks cb 0x55f827062180 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:40:12.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.527+0000 7f92c3e00640 1 -- 172.21.15.90:0/3648987613 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f8271407b0 msgr2=0x55f827146d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:40:12.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.527+0000 7f92c3e00640 1 --2- 172.21.15.90:0/3648987613 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f8271407b0 0x55f827146d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:12.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.527+0000 7f92c3e00640 1 -- 172.21.15.90:0/3648987613 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f8271410d0 msgr2=0x55f827147270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:40:12.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.527+0000 7f92c3e00640 1 --2- 172.21.15.90:0/3648987613 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f8271410d0 0x55f827147270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:12.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.527+0000 7f92c3e00640 1 -- 172.21.15.90:0/3648987613 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f826db83f0 con 0x55f827149cb0 2024-04-02T14:40:12.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.527+0000 7f92c3400640 1 --2- 172.21.15.90:0/3648987613 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f8271407b0 0x55f827146d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:40:12.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.528+0000 7f92bbe00640 1 --2- 172.21.15.90:0/3648987613 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f8271410d0 0x55f827147270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:40:12.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.528+0000 7f92c3e00640 1 --2- 172.21.15.90:0/3648987613 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f827149cb0 0x55f8271477b0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f92bc0c72d0 tx=0x7f92bc0be9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4d912957b453a1f server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:40:12.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.528+0000 7f92b0c00640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-02T14:40:12.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.530+0000 7f92b0c00640 1 -- 172.21.15.90:0/3648987613 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92bc0bf290 con 0x55f827149cb0 2024-04-02T14:40:12.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.530+0000 7f92b0c00640 1 -- 172.21.15.90:0/3648987613 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f92bc004220 con 0x55f827149cb0 2024-04-02T14:40:12.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.530+0000 7f92ccfd7740 1 -- 172.21.15.90:0/3648987613 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f92ac015070 con 0x55f827149cb0 2024-04-02T14:40:12.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.530+0000 7f92b0c00640 1 -- 172.21.15.90:0/3648987613 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f92bc0beb90 con 0x55f827149cb0 2024-04-02T14:40:12.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.531+0000 7f92ccfd7740 10 client.16431 fetch_fsmap learned FSMap version 27 2024-04-02T14:40:12.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.531+0000 7f92ccfd7740 1 -- 172.21.15.90:0/3648987613 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55f826d780f0 con 0x55f827149cb0 2024-04-02T14:40:12.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.531+0000 7f92b0c00640 1 -- 172.21.15.90:0/3648987613 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f92ac015070 con 0x55f827149cb0 2024-04-02T14:40:12.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.532+0000 7f92b0c00640 1 -- 172.21.15.90:0/3648987613 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f92bc0c7370 con 0x55f827149cb0 2024-04-02T14:40:12.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.532+0000 7f92ccfd7740 10 client.16431 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:40:12.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.532+0000 7f92ccfd7740 10 client.16431 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:40:12.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.532+0000 7f92ccfd7740 20 client.16431 populate_metadata read hostname 'smithi090' 2024-04-02T14:40:12.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.532+0000 7f92ccfd7740 1 --2- 172.21.15.90:0/3648987613 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55f827159c70 0x55f82715c0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:12.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.532+0000 7f92ccfd7740 4 client.16431 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:40:12.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.532+0000 7f92ccfd7740 1 -- 172.21.15.90:0/3648987613 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f827045630 con 0x55f827159c70 2024-04-02T14:40:12.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.533+0000 7f92c3400640 1 --2- 172.21.15.90:0/3648987613 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55f827159c70 0x55f82715c0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:12.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.533+0000 7f92c3400640 1 --2- 172.21.15.90:0/3648987613 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55f827159c70 0x55f82715c0d0 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6bddc20128624e2d server_cookie=a45c3ad134c4a932 in_seq=0 out_seq=0 2024-04-02T14:40:12.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.533+0000 7f92b0c00640 10 client.16431 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:40:12.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.534+0000 7f92b0c00640 1 -- 172.21.15.90:0/3648987613 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f827045630 con 0x55f827159c70 2024-04-02T14:40:12.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.534+0000 7f92b0c00640 10 client.16431 handle_command_reply: tid=0 2024-04-02T14:40:12.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.534+0000 7f92ccfd7740 1 client.16431 shutdown 2024-04-02T14:40:12.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.535+0000 7f92ccfd7740 1 -- 172.21.15.90:0/3648987613 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f827149cb0 msgr2=0x55f8271477b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:12.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.535+0000 7f92ccfd7740 1 --2- 172.21.15.90:0/3648987613 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f827149cb0 0x55f8271477b0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f92bc0c72d0 tx=0x7f92bc0be9e0 comp rx=0 tx=0).stop 2024-04-02T14:40:12.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.535+0000 7f92ccfd7740 1 -- 172.21.15.90:0/3648987613 shutdown_connections 2024-04-02T14:40:12.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.535+0000 7f92ccfd7740 1 --2- 172.21.15.90:0/3648987613 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55f827159c70 0x55f82715c0d0 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:12.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.535+0000 7f92ccfd7740 1 --2- 172.21.15.90:0/3648987613 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f827149cb0 0x55f8271477b0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:12.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.535+0000 7f92ccfd7740 1 --2- 172.21.15.90:0/3648987613 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f8271410d0 0x55f827147270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:12.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.535+0000 7f92ccfd7740 1 --2- 172.21.15.90:0/3648987613 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f8271407b0 0x55f827146d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:12.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.535+0000 7f92ccfd7740 1 -- 172.21.15.90:0/3648987613 >> 172.21.15.90:0/3648987613 conn(0x55f82713aee0 msgr2=0x55f82713cc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:40:12.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.535+0000 7f92b0c00640 0 client.16431 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:40:12.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.536+0000 7f92ccfd7740 1 -- 172.21.15.90:0/3648987613 shutdown_connections 2024-04-02T14:40:12.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.536+0000 7f92ccfd7740 1 -- 172.21.15.90:0/3648987613 wait complete. 2024-04-02T14:40:12.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.536+0000 7f92ccfd7740 20 client.16431 trim_cache size 0 max 16384 2024-04-02T14:40:12.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.570+0000 7f92ccfd7740 1 Processor -- start 2024-04-02T14:40:12.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.570+0000 7f92ccfd7740 1 -- start start 2024-04-02T14:40:12.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.571+0000 7f92ccfd7740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f8271407b0 0x55f827149460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:12.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.571+0000 7f92ccfd7740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f8271410d0 0x55f827155990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:12.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.571+0000 7f92ccfd7740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f827149cb0 0x55f82713ca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:12.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.571+0000 7f92ccfd7740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f92c41c4e70 con 0x55f8271407b0 2024-04-02T14:40:12.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.571+0000 7f92ccfd7740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f92c41c69b0 con 0x55f8271410d0 2024-04-02T14:40:12.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.571+0000 7f92ccfd7740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f92c41c5540 con 0x55f827149cb0 2024-04-02T14:40:12.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.571+0000 7f92c3400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f8271407b0 0x55f827149460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:12.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.572+0000 7f92c3400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f8271407b0 0x55f827149460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43900/0 (socket says 172.21.15.90:43900) 2024-04-02T14:40:12.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.572+0000 7f92c3400640 1 -- 172.21.15.90:0/2113400793 learned_addr learned my addr 172.21.15.90:0/2113400793 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:40:12.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.572+0000 7f92bbe00640 1 --2- 172.21.15.90:0/2113400793 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f8271410d0 0x55f827155990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:12.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.572+0000 7f92c3e00640 1 --2- 172.21.15.90:0/2113400793 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f827149cb0 0x55f82713ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:12.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.573+0000 7f92c3400640 1 -- 172.21.15.90:0/2113400793 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f827149cb0 msgr2=0x55f82713ca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:40:12.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.573+0000 7f92c3400640 1 --2- 172.21.15.90:0/2113400793 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f827149cb0 0x55f82713ca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:12.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.573+0000 7f92c3400640 1 -- 172.21.15.90:0/2113400793 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f8271410d0 msgr2=0x55f827155990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:40:12.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.573+0000 7f92c3400640 1 --2- 172.21.15.90:0/2113400793 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f8271410d0 0x55f827155990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:12.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.573+0000 7f92c3400640 1 -- 172.21.15.90:0/2113400793 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f827145280 con 0x55f8271407b0 2024-04-02T14:40:12.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.573+0000 7f92c3e00640 1 --2- 172.21.15.90:0/2113400793 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f827149cb0 0x55f82713ca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:40:12.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.574+0000 7f92c3400640 1 --2- 172.21.15.90:0/2113400793 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f8271407b0 0x55f827149460 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f92ac000c00 tx=0x7f92ac0042c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5342075cdb12f4b server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:40:12.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.574+0000 7f92b3400640 1 -- 172.21.15.90:0/2113400793 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92ac004680 con 0x55f8271407b0 2024-04-02T14:40:12.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.574+0000 7f92b3400640 1 -- 172.21.15.90:0/2113400793 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f92ac015040 con 0x55f8271407b0 2024-04-02T14:40:12.576 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.574+0000 7f92b3400640 1 -- 172.21.15.90:0/2113400793 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92ac029730 con 0x55f8271407b0 2024-04-02T14:40:12.576 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.575+0000 7f92ccfd7740 1 -- 172.21.15.90:0/2113400793 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f8271407b0 msgr2=0x55f827149460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:12.576 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.575+0000 7f92ccfd7740 1 --2- 172.21.15.90:0/2113400793 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f8271407b0 0x55f827149460 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f92ac000c00 tx=0x7f92ac0042c0 comp rx=0 tx=0).stop 2024-04-02T14:40:12.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.575+0000 7f92ccfd7740 1 -- 172.21.15.90:0/2113400793 shutdown_connections 2024-04-02T14:40:12.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.575+0000 7f92ccfd7740 1 --2- 172.21.15.90:0/2113400793 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f827149cb0 0x55f82713ca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:12.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.575+0000 7f92ccfd7740 1 --2- 172.21.15.90:0/2113400793 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f8271410d0 0x55f827155990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:12.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.575+0000 7f92ccfd7740 1 --2- 172.21.15.90:0/2113400793 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f8271407b0 0x55f827149460 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:12.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.576+0000 7f92ccfd7740 1 -- 172.21.15.90:0/2113400793 >> 172.21.15.90:0/2113400793 conn(0x55f82713aee0 msgr2=0x55f82713c310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:40:12.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.576+0000 7f92ccfd7740 1 -- 172.21.15.90:0/2113400793 shutdown_connections 2024-04-02T14:40:12.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.576+0000 7f92ccfd7740 1 -- 172.21.15.90:0/2113400793 wait complete. 2024-04-02T14:40:12.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.577+0000 7f92ccfd7740 1 -- start start 2024-04-02T14:40:12.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.577+0000 7f92ccfd7740 1 Processor -- start 2024-04-02T14:40:12.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.578+0000 7f92ccfd7740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f8271407b0 0x55f82714d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:12.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.578+0000 7f92ccfd7740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f8271410d0 0x55f82714de20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:12.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.578+0000 7f92ccfd7740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f827149cb0 0x55f82714e360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:12.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.578+0000 7f92ccfd7740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f92c41c62e0 con 0x55f8271407b0 2024-04-02T14:40:12.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.578+0000 7f92ccfd7740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f92c41c5c10 con 0x55f8271410d0 2024-04-02T14:40:12.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.578+0000 7f92ccfd7740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f92c41c2e50 con 0x55f827149cb0 2024-04-02T14:40:12.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.579+0000 7f92ccfd7740 10 client.0 _ll_register_callbacks cb 0x7f92bc0baee0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:40:12.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.579+0000 7f92c3e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f827149cb0 0x55f82714e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:12.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.579+0000 7f92c3e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f827149cb0 0x55f82714e360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:48974/0 (socket says 172.21.15.90:48974) 2024-04-02T14:40:12.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.579+0000 7f92c3e00640 1 -- 172.21.15.90:0/1692053675 learned_addr learned my addr 172.21.15.90:0/1692053675 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:40:12.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.579+0000 7f92c3400640 1 --2- 172.21.15.90:0/1692053675 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f8271407b0 0x55f82714d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:12.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.579+0000 7f92bbe00640 1 --2- 172.21.15.90:0/1692053675 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f8271410d0 0x55f82714de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:12.582 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.580+0000 7f92c3e00640 1 -- 172.21.15.90:0/1692053675 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f8271410d0 msgr2=0x55f82714de20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:40:12.582 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.580+0000 7f92c3e00640 1 --2- 172.21.15.90:0/1692053675 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f8271410d0 0x55f82714de20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:12.582 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.580+0000 7f92c3e00640 1 -- 172.21.15.90:0/1692053675 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f8271407b0 msgr2=0x55f82714d8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:40:12.582 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.580+0000 7f92c3e00640 1 --2- 172.21.15.90:0/1692053675 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f8271407b0 0x55f82714d8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:12.582 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.580+0000 7f92c3e00640 1 -- 172.21.15.90:0/1692053675 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f8271bf280 con 0x55f827149cb0 2024-04-02T14:40:12.582 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.580+0000 7f92c3400640 1 --2- 172.21.15.90:0/1692053675 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f8271407b0 0x55f82714d8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:40:12.582 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.580+0000 7f92bbe00640 1 --2- 172.21.15.90:0/1692053675 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f8271410d0 0x55f82714de20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:40:12.582 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.580+0000 7f92c3e00640 1 --2- 172.21.15.90:0/1692053675 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f827149cb0 0x55f82714e360 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f92bc0f0830 tx=0x7f92bc0df2c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=34999136c9b3d827 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:40:12.583 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.580+0000 7f92b0c00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:40:12.583 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.581+0000 7f92b0c00640 1 -- 172.21.15.90:0/1692053675 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92bc0be1e0 con 0x55f827149cb0 2024-04-02T14:40:12.583 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.581+0000 7f92b0c00640 1 -- 172.21.15.90:0/1692053675 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f92bc003970 con 0x55f827149cb0 2024-04-02T14:40:12.583 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.581+0000 7f92ccfd7740 1 -- 172.21.15.90:0/1692053675 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f92ac015040 con 0x55f827149cb0 2024-04-02T14:40:12.583 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.582+0000 7f92b0c00640 1 -- 172.21.15.90:0/1692053675 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f92bc0bf290 con 0x55f827149cb0 2024-04-02T14:40:12.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.582+0000 7f92ccfd7740 10 client.26866 fetch_fsmap learned FSMap version 27 2024-04-02T14:40:12.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.582+0000 7f92ccfd7740 1 -- 172.21.15.90:0/1692053675 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55f8271c05a0 con 0x55f827149cb0 2024-04-02T14:40:12.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.582+0000 7f92b0c00640 1 -- 172.21.15.90:0/1692053675 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f92ac015040 con 0x55f827149cb0 2024-04-02T14:40:12.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.583+0000 7f92b0c00640 1 -- 172.21.15.90:0/1692053675 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f92bc0bda50 con 0x55f827149cb0 2024-04-02T14:40:12.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.583+0000 7f92ccfd7740 10 client.26866 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:40:12.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.583+0000 7f92ccfd7740 10 client.26866 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:40:12.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.583+0000 7f92ccfd7740 20 client.26866 populate_metadata read hostname 'smithi090' 2024-04-02T14:40:12.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.583+0000 7f92ccfd7740 1 --2- 172.21.15.90:0/1692053675 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55f8271c0b90 0x55f8271c2fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:12.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.583+0000 7f92ccfd7740 4 client.26866 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-02T14:40:12.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.583+0000 7f92ccfd7740 1 -- 172.21.15.90:0/1692053675 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55f8270b1cb0 con 0x55f8271c0b90 2024-04-02T14:40:12.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.584+0000 7f92c3400640 1 --2- 172.21.15.90:0/1692053675 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55f8271c0b90 0x55f8271c2fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:12.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.585+0000 7f92c3400640 1 --2- 172.21.15.90:0/1692053675 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55f8271c0b90 0x55f8271c2fb0 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8fa012bfede00a5e server_cookie=d339f956395cbfa9 in_seq=0 out_seq=0 2024-04-02T14:40:12.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.585+0000 7f92b0c00640 10 client.26866 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:40:12.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.585+0000 7f92b0c00640 1 -- 172.21.15.90:0/1692053675 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55f8270b1cb0 con 0x55f8271c0b90 2024-04-02T14:40:12.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.585+0000 7f92b0c00640 10 client.26866 handle_command_reply: tid=0 2024-04-02T14:40:12.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.585+0000 7f92ccfd7740 1 client.26866 shutdown 2024-04-02T14:40:12.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.586+0000 7f92ccfd7740 1 -- 172.21.15.90:0/1692053675 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f827149cb0 msgr2=0x55f82714e360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:12.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.586+0000 7f92ccfd7740 1 --2- 172.21.15.90:0/1692053675 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f827149cb0 0x55f82714e360 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f92bc0f0830 tx=0x7f92bc0df2c0 comp rx=0 tx=0).stop 2024-04-02T14:40:12.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.586+0000 7f92ccfd7740 1 -- 172.21.15.90:0/1692053675 shutdown_connections 2024-04-02T14:40:12.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.586+0000 7f92ccfd7740 1 --2- 172.21.15.90:0/1692053675 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55f8271c0b90 0x55f8271c2fb0 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:12.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.586+0000 7f92ccfd7740 1 --2- 172.21.15.90:0/1692053675 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f827149cb0 0x55f82714e360 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:12.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.586+0000 7f92ccfd7740 1 --2- 172.21.15.90:0/1692053675 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f8271410d0 0x55f82714de20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:12.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.586+0000 7f92ccfd7740 1 --2- 172.21.15.90:0/1692053675 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f8271407b0 0x55f82714d8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:12.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.586+0000 7f92ccfd7740 1 -- 172.21.15.90:0/1692053675 >> 172.21.15.90:0/1692053675 conn(0x55f82713aee0 msgr2=0x55f827139f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:40:12.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.586+0000 7f92b0c00640 0 client.26866 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:40:12.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.586+0000 7f92ccfd7740 1 -- 172.21.15.90:0/1692053675 shutdown_connections 2024-04-02T14:40:12.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.587+0000 7f92ccfd7740 1 -- 172.21.15.90:0/1692053675 wait complete. 2024-04-02T14:40:12.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.587+0000 7f92ccfd7740 20 client.26866 trim_cache size 0 max 16384 2024-04-02T14:40:12.589 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-02T14:40:12.589 INFO:teuthology.orchestra.run.smithi090.stdout: "status": "no active scrubs running", 2024-04-02T14:40:12.589 INFO:teuthology.orchestra.run.smithi090.stdout: "scrubs": {} 2024-04-02T14:40:12.589 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-02T14:40:12.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.588+0000 7f92ca200640 1 -- 172.21.15.90:0/4116968037 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f92a80a3de0 msgr2=0x7f92a80a62a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:12.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.588+0000 7f92ca200640 1 --2- 172.21.15.90:0/4116968037 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f92a80a3de0 0x7f92a80a62a0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x55f827140280 tx=0x7f92b4009580 comp rx=0 tx=0).stop 2024-04-02T14:40:12.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.588+0000 7f92ca200640 1 -- 172.21.15.90:0/4116968037 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f92c411b2e0 msgr2=0x7f92c4111ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:12.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.588+0000 7f92ca200640 1 --2- 172.21.15.90:0/4116968037 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f92c411b2e0 0x7f92c4111ea0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f92bc009b30 tx=0x7f92bc0336b0 comp rx=0 tx=0).stop 2024-04-02T14:40:12.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.588+0000 7f92ca200640 1 -- 172.21.15.90:0/4116968037 shutdown_connections 2024-04-02T14:40:12.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.588+0000 7f92ca200640 1 --2- 172.21.15.90:0/4116968037 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f92a80a3de0 0x7f92a80a62a0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:12.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.588+0000 7f92ca200640 1 --2- 172.21.15.90:0/4116968037 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f92c411b2e0 0x7f92c4111ea0 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:12.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.588+0000 7f92ca200640 1 --2- 172.21.15.90:0/4116968037 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f92c4110320 0x7f92c4111960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:12.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.588+0000 7f92ca200640 1 --2- 172.21.15.90:0/4116968037 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f92c410e350 0x7f92c4118890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:12.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.588+0000 7f92ca200640 1 -- 172.21.15.90:0/4116968037 >> 172.21.15.90:0/4116968037 conn(0x7f92c406cbe0 msgr2=0x7f92c40725d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:40:12.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.589+0000 7f92ca200640 1 -- 172.21.15.90:0/4116968037 shutdown_connections 2024-04-02T14:40:12.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.589+0000 7f92ca200640 1 -- 172.21.15.90:0/4116968037 wait complete. 2024-04-02T14:40:12.620 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-02T14:40:12.621 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T14:40:12.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.903+0000 7fb2a7200640 1 -- 172.21.15.90:0/4115502235 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb2a01044b0 msgr2=0x7fb2a01109c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:12.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.903+0000 7fb2a7200640 1 --2- 172.21.15.90:0/4115502235 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb2a01044b0 0x7fb2a01109c0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fb29c009cd0 tx=0x7fb29c02c0f0 comp rx=0 tx=0).stop 2024-04-02T14:40:12.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.904+0000 7fb2a7200640 1 -- 172.21.15.90:0/4115502235 shutdown_connections 2024-04-02T14:40:12.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.904+0000 7fb2a7200640 1 --2- 172.21.15.90:0/4115502235 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb2a01044b0 0x7fb2a01109c0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:12.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.904+0000 7fb2a7200640 1 --2- 172.21.15.90:0/4115502235 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb2a0103af0 0x7fb2a0103f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:12.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.904+0000 7fb2a7200640 1 --2- 172.21.15.90:0/4115502235 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb2a0109b90 0x7fb2a0109f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:12.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.904+0000 7fb2a7200640 1 -- 172.21.15.90:0/4115502235 >> 172.21.15.90:0/4115502235 conn(0x7fb2a00ff5a0 msgr2=0x7fb2a01019c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:40:12.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.904+0000 7fb2a7200640 1 -- 172.21.15.90:0/4115502235 shutdown_connections 2024-04-02T14:40:12.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.905+0000 7fb2a7200640 1 -- 172.21.15.90:0/4115502235 wait complete. 2024-04-02T14:40:12.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.907+0000 7fb2a7200640 1 Processor -- start 2024-04-02T14:40:12.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.907+0000 7fb2a7200640 1 -- start start 2024-04-02T14:40:12.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.907+0000 7fb2a7200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb2a0103af0 0x7fb2a0110830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:12.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.907+0000 7fb2a7200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb2a01044b0 0x7fb2a010d920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:12.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.907+0000 7fb2a7200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb2a0109b90 0x7fb2a010de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:12.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.907+0000 7fb2a7200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fb2a0113ee0 con 0x7fb2a0103af0 2024-04-02T14:40:12.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.908+0000 7fb2a7200640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fb2a0113d60 con 0x7fb2a01044b0 2024-04-02T14:40:12.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.908+0000 7fb2a7200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fb2a0114060 con 0x7fb2a0109b90 2024-04-02T14:40:12.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.908+0000 7fb2a6800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb2a0109b90 0x7fb2a010de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:12.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.908+0000 7fb2a6800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb2a0109b90 0x7fb2a010de80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:48990/0 (socket says 172.21.15.90:48990) 2024-04-02T14:40:12.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.908+0000 7fb2a6800640 1 -- 172.21.15.90:0/1516673928 learned_addr learned my addr 172.21.15.90:0/1516673928 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:40:12.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.908+0000 7fb2a5400640 1 --2- 172.21.15.90:0/1516673928 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb2a01044b0 0x7fb2a010d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:12.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.908+0000 7fb2a6800640 1 -- 172.21.15.90:0/1516673928 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb2a01044b0 msgr2=0x7fb2a010d920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:12.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.908+0000 7fb2a6800640 1 --2- 172.21.15.90:0/1516673928 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb2a01044b0 0x7fb2a010d920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:12.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.908+0000 7fb2a6800640 1 -- 172.21.15.90:0/1516673928 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb2a0103af0 msgr2=0x7fb2a0110830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:12.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.908+0000 7fb2a6800640 1 --2- 172.21.15.90:0/1516673928 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb2a0103af0 0x7fb2a0110830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:12.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.908+0000 7fb2a6800640 1 -- 172.21.15.90:0/1516673928 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2a010e740 con 0x7fb2a0109b90 2024-04-02T14:40:12.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.909+0000 7fb2a6800640 1 --2- 172.21.15.90:0/1516673928 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb2a0109b90 0x7fb2a010de80 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fb29c009bf0 tx=0x7fb29c032ec0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:40:12.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.910+0000 7fb29aa00640 1 -- 172.21.15.90:0/1516673928 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb29c041020 con 0x7fb2a0109b90 2024-04-02T14:40:12.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.911+0000 7fb29aa00640 1 -- 172.21.15.90:0/1516673928 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb29c034cd0 con 0x7fb2a0109b90 2024-04-02T14:40:12.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.911+0000 7fb29aa00640 1 -- 172.21.15.90:0/1516673928 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb29c039040 con 0x7fb2a0109b90 2024-04-02T14:40:12.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.911+0000 7fb2a7200640 1 -- 172.21.15.90:0/1516673928 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2a006b5e0 con 0x7fb2a0109b90 2024-04-02T14:40:12.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.912+0000 7fb2a7200640 1 -- 172.21.15.90:0/1516673928 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2a006b9c0 con 0x7fb2a0109b90 2024-04-02T14:40:12.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.915+0000 7fb2a7200640 1 -- 172.21.15.90:0/1516673928 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb2a0068810 con 0x7fb2a0109b90 2024-04-02T14:40:12.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.919+0000 7fb29aa00640 1 -- 172.21.15.90:0/1516673928 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fb29c044020 con 0x7fb2a0109b90 2024-04-02T14:40:12.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.922+0000 7fb29aa00640 1 --2- 172.21.15.90:0/1516673928 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fb2880a3ea0 0x7fb2880a6360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:12.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.922+0000 7fb29aa00640 1 -- 172.21.15.90:0/1516673928 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7fb29c0f1d60 con 0x7fb2a0109b90 2024-04-02T14:40:12.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.923+0000 7fb2a5e00640 1 --2- 172.21.15.90:0/1516673928 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fb2880a3ea0 0x7fb2880a6360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:12.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.925+0000 7fb2a5e00640 1 --2- 172.21.15.90:0/1516673928 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fb2880a3ea0 0x7fb2880a6360 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fb294009ad0 tx=0x7fb294009590 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:40:12.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:12.925+0000 7fb29aa00640 1 -- 172.21.15.90:0/1516673928 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7fb29c04b070 con 0x7fb2a0109b90 2024-04-02T14:40:13.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.356+0000 7fb2a7200640 1 -- 172.21.15.90:0/1516673928 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb2a006bcb0 con 0x7fb2a0109b90 2024-04-02T14:40:13.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.357+0000 7fb29aa00640 1 -- 172.21.15.90:0/1516673928 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11927 (secure 0 0 0) 0x7fb29c0be520 con 0x7fb2a0109b90 2024-04-02T14:40:13.359 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-02T14:40:13.359 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14412,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6837/2852160309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6836","nonce":2852160309},{"type":"v1","addr":"172.21.15.90:6837","nonce":2852160309}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14478,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6841/1919376858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6840","nonce":1919376858},{"type":"v1","addr":"172.21.15.90:6841","nonce":1919376858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-02T14:32:22.319083+0000","modified":"2024-04-02T14:34:52.266152+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24590,"mds_1":24560,"mds_2":24547,"mds_3":24539,"mds_4":24523},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24454":{"gid":24454,"name":"l","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6833/1564319431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":1564319431},{"type":"v1","addr":"172.21.15.177:6833","nonce":1564319431}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.92:6833/1570337372","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":1570337372},{"type":"v1","addr":"172.21.15.92:6833","nonce":1570337372}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"j","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.90:6835/1277124891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6834","nonce":1277124891},{"type":"v1","addr":"172.21.15.90:6835","nonce":1277124891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.92:6835/860721164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":860721164},{"type":"v1","addr":"172.21.15.92:6835","nonce":860721164}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24511":{"gid":24511,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6835/3177244988","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":3177244988},{"type":"v1","addr":"172.21.15.177:6835","nonce":3177244988}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"c","rank":4,"incarnation":25,"state":"up:active","state_seq":9,"addr":"172.21.15.177:6837/4223414858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":4223414858},{"type":"v1","addr":"172.21.15.177:6837","nonce":4223414858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24539":{"gid":24539,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":8,"addr":"172.21.15.92:6837/1270635939","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":1270635939},{"type":"v1","addr":"172.21.15.92:6837","nonce":1270635939}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24547":{"gid":24547,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":7,"addr":"172.21.15.177:6839/664895663","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":664895663},{"type":"v1","addr":"172.21.15.177:6839","nonce":664895663}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"g","rank":1,"incarnation":19,"state":"up:active","state_seq":5,"addr":"172.21.15.90:6839/1498915870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6838","nonce":1498915870},{"type":"v1","addr":"172.21.15.90:6839","nonce":1498915870}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24590":{"gid":24590,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.92:6839/1054888219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":1054888219},{"type":"v1","addr":"172.21.15.92:6839","nonce":1054888219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24590,"qdb_cluster":[24547,24590,24523,24539,24560]},"id":1}]} 2024-04-02T14:40:13.360 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 27 2024-04-02T14:40:13.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.361+0000 7fb2a7200640 1 -- 172.21.15.90:0/1516673928 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fb2880a3ea0 msgr2=0x7fb2880a6360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:13.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.361+0000 7fb2a7200640 1 --2- 172.21.15.90:0/1516673928 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fb2880a3ea0 0x7fb2880a6360 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fb294009ad0 tx=0x7fb294009590 comp rx=0 tx=0).stop 2024-04-02T14:40:13.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.361+0000 7fb2a7200640 1 -- 172.21.15.90:0/1516673928 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb2a0109b90 msgr2=0x7fb2a010de80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:13.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.361+0000 7fb2a7200640 1 --2- 172.21.15.90:0/1516673928 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb2a0109b90 0x7fb2a010de80 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fb29c009bf0 tx=0x7fb29c032ec0 comp rx=0 tx=0).stop 2024-04-02T14:40:13.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.361+0000 7fb2a7200640 1 -- 172.21.15.90:0/1516673928 shutdown_connections 2024-04-02T14:40:13.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.361+0000 7fb2a7200640 1 --2- 172.21.15.90:0/1516673928 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fb2880a3ea0 0x7fb2880a6360 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:13.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.361+0000 7fb2a7200640 1 --2- 172.21.15.90:0/1516673928 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb2a0109b90 0x7fb2a010de80 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:13.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.361+0000 7fb2a7200640 1 --2- 172.21.15.90:0/1516673928 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb2a01044b0 0x7fb2a010d920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:13.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.361+0000 7fb2a7200640 1 --2- 172.21.15.90:0/1516673928 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb2a0103af0 0x7fb2a0110830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:13.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.361+0000 7fb2a7200640 1 -- 172.21.15.90:0/1516673928 >> 172.21.15.90:0/1516673928 conn(0x7fb2a00ff5a0 msgr2=0x7fb2a00ffac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:40:13.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.361+0000 7fb2a7200640 1 -- 172.21.15.90:0/1516673928 shutdown_connections 2024-04-02T14:40:13.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.361+0000 7fb2a7200640 1 -- 172.21.15.90:0/1516673928 wait complete. 2024-04-02T14:40:13.395 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-04-02T14:40:13.690 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:40:13 smithi090 ceph-mon[31674]: pgmap v163: 97 pgs: 97 active+clean; 3.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 21 MiB/s wr, 1.31k op/s 2024-04-02T14:40:13.690 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:40:13 smithi090 ceph-mon[31674]: from='client.? 172.21.15.90:0/1516673928' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-02T14:40:13.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.691+0000 7f139de00640 1 -- 172.21.15.90:0/426148513 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1398102450 msgr2=0x7f139810e960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:13.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.691+0000 7f139de00640 1 --2- 172.21.15.90:0/426148513 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1398102450 0x7f139810e960 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f1390009cd0 tx=0x7f139002c0e0 comp rx=0 tx=0).stop 2024-04-02T14:40:13.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.691+0000 7f139de00640 1 -- 172.21.15.90:0/426148513 shutdown_connections 2024-04-02T14:40:13.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.691+0000 7f139de00640 1 --2- 172.21.15.90:0/426148513 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1398102450 0x7f139810e960 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:13.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.691+0000 7f139de00640 1 --2- 172.21.15.90:0/426148513 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1398101a90 0x7f1398101f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:13.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.691+0000 7f139de00640 1 --2- 172.21.15.90:0/426148513 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1398107aa0 0x7f1398107e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:13.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.691+0000 7f139de00640 1 -- 172.21.15.90:0/426148513 >> 172.21.15.90:0/426148513 conn(0x7f1398078f00 msgr2=0x7f1398079310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:40:13.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.692+0000 7f139de00640 1 -- 172.21.15.90:0/426148513 shutdown_connections 2024-04-02T14:40:13.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.692+0000 7f139de00640 1 -- 172.21.15.90:0/426148513 wait complete. 2024-04-02T14:40:13.694 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.693+0000 7f139de00640 1 Processor -- start 2024-04-02T14:40:13.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.695+0000 7f139de00640 1 -- start start 2024-04-02T14:40:13.697 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.695+0000 7f139de00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1398101a90 0x7f1398073e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:13.697 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.695+0000 7f139de00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1398102450 0x7f1398074340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:13.697 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.695+0000 7f139de00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1398107aa0 0x7f1398074880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:13.697 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.695+0000 7f139de00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f1398111eb0 con 0x7f1398107aa0 2024-04-02T14:40:13.697 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.695+0000 7f139de00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f1398111d30 con 0x7f1398102450 2024-04-02T14:40:13.697 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.695+0000 7f139de00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f1398112030 con 0x7f1398101a90 2024-04-02T14:40:13.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.696+0000 7f1396a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1398101a90 0x7f1398073e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:13.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.696+0000 7f1396a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1398101a90 0x7f1398073e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:49000/0 (socket says 172.21.15.90:49000) 2024-04-02T14:40:13.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.696+0000 7f1396a00640 1 -- 172.21.15.90:0/1482349326 learned_addr learned my addr 172.21.15.90:0/1482349326 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:40:13.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.696+0000 7f1396a00640 1 -- 172.21.15.90:0/1482349326 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1398102450 msgr2=0x7f1398074340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:13.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.696+0000 7f1396a00640 1 --2- 172.21.15.90:0/1482349326 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1398102450 0x7f1398074340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:13.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.696+0000 7f1396a00640 1 -- 172.21.15.90:0/1482349326 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1398107aa0 msgr2=0x7f1398074880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:13.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.696+0000 7f1396a00640 1 --2- 172.21.15.90:0/1482349326 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1398107aa0 0x7f1398074880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:13.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.697+0000 7f1396a00640 1 -- 172.21.15.90:0/1482349326 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13981a68f0 con 0x7f1398101a90 2024-04-02T14:40:13.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.697+0000 7f1396a00640 1 --2- 172.21.15.90:0/1482349326 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1398101a90 0x7f1398073e00 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f138800d950 tx=0x7f138800de10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:40:13.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.698+0000 7f139ca00640 1 -- 172.21.15.90:0/1482349326 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1388017070 con 0x7f1398101a90 2024-04-02T14:40:13.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.698+0000 7f139ca00640 1 -- 172.21.15.90:0/1482349326 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1388009c30 con 0x7f1398101a90 2024-04-02T14:40:13.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.698+0000 7f139ca00640 1 -- 172.21.15.90:0/1482349326 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1388017070 con 0x7f1398101a90 2024-04-02T14:40:13.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.698+0000 7f139de00640 1 -- 172.21.15.90:0/1482349326 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13981a6be0 con 0x7f1398101a90 2024-04-02T14:40:13.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.699+0000 7f139de00640 1 -- 172.21.15.90:0/1482349326 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1398198320 con 0x7f1398101a90 2024-04-02T14:40:13.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.704+0000 7f13a09e1740 1 Processor -- start 2024-04-02T14:40:13.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.704+0000 7f13a09e1740 1 -- start start 2024-04-02T14:40:13.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.705+0000 7f13a09e1740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559225a199f0 0x559225a18740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:13.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.705+0000 7f13a09e1740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559225a18d10 0x559225a19190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:13.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.705+0000 7f13a09e1740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559225a22080 0x559225a24470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:13.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.705+0000 7f13a09e1740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x559225959400 con 0x559225a199f0 2024-04-02T14:40:13.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.705+0000 7f13a09e1740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55922593df10 con 0x559225a18d10 2024-04-02T14:40:13.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.705+0000 7f13a09e1740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55922597d6f0 con 0x559225a22080 2024-04-02T14:40:13.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.705+0000 7f1396000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559225a22080 0x559225a24470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:13.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.705+0000 7f1396000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559225a22080 0x559225a24470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:49004/0 (socket says 172.21.15.90:49004) 2024-04-02T14:40:13.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.705+0000 7f1396000640 1 -- 172.21.15.90:0/3279881674 learned_addr learned my addr 172.21.15.90:0/3279881674 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:40:13.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.706+0000 7f1397400640 1 --2- 172.21.15.90:0/3279881674 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559225a199f0 0x559225a18740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:13.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.706+0000 7f1396000640 1 -- 172.21.15.90:0/3279881674 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559225a18d10 msgr2=0x559225a19190 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-02T14:40:13.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.706+0000 7f1396000640 1 --2- 172.21.15.90:0/3279881674 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559225a18d10 0x559225a19190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:13.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.706+0000 7f1396000640 1 -- 172.21.15.90:0/3279881674 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559225a199f0 msgr2=0x559225a18740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:40:13.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.706+0000 7f1396000640 1 --2- 172.21.15.90:0/3279881674 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559225a199f0 0x559225a18740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:13.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.706+0000 7f1396000640 1 -- 172.21.15.90:0/3279881674 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5592256913f0 con 0x559225a22080 2024-04-02T14:40:13.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.706+0000 7f1396000640 1 --2- 172.21.15.90:0/3279881674 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559225a22080 0x559225a24470 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f13840097f0 tx=0x7f1384009cb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9f2afdaf02405065 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:40:13.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.707+0000 7f1383e00640 1 -- 172.21.15.90:0/3279881674 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1384015020 con 0x559225a22080 2024-04-02T14:40:13.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.707+0000 7f139ca00640 1 -- 172.21.15.90:0/1482349326 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f138800f4b0 con 0x7f1398101a90 2024-04-02T14:40:13.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.707+0000 7f1383e00640 1 -- 172.21.15.90:0/3279881674 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1384011420 con 0x559225a22080 2024-04-02T14:40:13.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.707+0000 7f1383e00640 1 -- 172.21.15.90:0/3279881674 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13840105e0 con 0x559225a22080 2024-04-02T14:40:13.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.708+0000 7f13a09e1740 1 -- 172.21.15.90:0/3279881674 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559225a22080 msgr2=0x559225a24470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:13.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.708+0000 7f13a09e1740 1 --2- 172.21.15.90:0/3279881674 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559225a22080 0x559225a24470 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f13840097f0 tx=0x7f1384009cb0 comp rx=0 tx=0).stop 2024-04-02T14:40:13.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.708+0000 7f13a09e1740 1 -- 172.21.15.90:0/3279881674 shutdown_connections 2024-04-02T14:40:13.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.708+0000 7f13a09e1740 1 --2- 172.21.15.90:0/3279881674 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559225a22080 0x559225a24470 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:13.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.708+0000 7f13a09e1740 1 --2- 172.21.15.90:0/3279881674 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559225a18d10 0x559225a19190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:13.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.708+0000 7f13a09e1740 1 --2- 172.21.15.90:0/3279881674 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559225a199f0 0x559225a18740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:13.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.708+0000 7f13a09e1740 1 -- 172.21.15.90:0/3279881674 >> 172.21.15.90:0/3279881674 conn(0x5592259dfa50 msgr2=0x559225a13760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:40:13.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.709+0000 7f13a09e1740 1 -- 172.21.15.90:0/3279881674 shutdown_connections 2024-04-02T14:40:13.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.709+0000 7f13a09e1740 1 -- 172.21.15.90:0/3279881674 wait complete. 2024-04-02T14:40:13.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.709+0000 7f139ca00640 1 --2- 172.21.15.90:0/1482349326 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f137c0a3eb0 0x7f137c0a6370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:13.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.709+0000 7f139ca00640 1 -- 172.21.15.90:0/1482349326 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f13880d0420 con 0x7f1398101a90 2024-04-02T14:40:13.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.710+0000 7f13a09e1740 1 -- start start 2024-04-02T14:40:13.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.710+0000 7f1397400640 1 --2- 172.21.15.90:0/1482349326 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f137c0a3eb0 0x7f137c0a6370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:13.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.711+0000 7f13a09e1740 1 Processor -- start 2024-04-02T14:40:13.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.712+0000 7f1397400640 1 --2- 172.21.15.90:0/1482349326 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f137c0a3eb0 0x7f137c0a6370 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f139002c620 tx=0x7f1390004a80 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:40:13.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.712+0000 7f13a09e1740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559225a18d10 0x559225a1f080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:13.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.712+0000 7f13a09e1740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559225a199f0 0x559225a1f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:13.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.712+0000 7f13a09e1740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559225a22080 0x559225a1fb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:13.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.712+0000 7f13a09e1740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5592259796a0 con 0x559225a22080 2024-04-02T14:40:13.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.712+0000 7f13a09e1740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x559225925980 con 0x559225a18d10 2024-04-02T14:40:13.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.712+0000 7f13a09e1740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x559225938cb0 con 0x559225a199f0 2024-04-02T14:40:13.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.712+0000 7f13a09e1740 10 client.0 _ll_register_callbacks cb 0x5592259386a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:40:13.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.714+0000 7f1396a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559225a18d10 0x559225a1f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:13.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.714+0000 7f1396a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559225a18d10 0x559225a1f080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:44720/0 (socket says 172.21.15.90:44720) 2024-04-02T14:40:13.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.714+0000 7f1396a00640 1 -- 172.21.15.90:0/3511868929 learned_addr learned my addr 172.21.15.90:0/3511868929 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:40:13.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.714+0000 7f1396a00640 1 -- 172.21.15.90:0/3511868929 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559225a199f0 msgr2=0x559225a1f5c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-02T14:40:13.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.714+0000 7f1397400640 1 --2- 172.21.15.90:0/3511868929 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559225a22080 0x559225a1fb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:13.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.714+0000 7f1396000640 1 --2- 172.21.15.90:0/3511868929 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559225a199f0 0x559225a1f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:13.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.714+0000 7f1396a00640 1 --2- 172.21.15.90:0/3511868929 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559225a199f0 0x559225a1f5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:13.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.714+0000 7f1396a00640 1 -- 172.21.15.90:0/3511868929 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559225a22080 msgr2=0x559225a1fb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:40:13.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.714+0000 7f1396a00640 1 --2- 172.21.15.90:0/3511868929 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559225a22080 0x559225a1fb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:13.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.714+0000 7f1396a00640 1 -- 172.21.15.90:0/3511868929 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55922575b7d0 con 0x559225a18d10 2024-04-02T14:40:13.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.714+0000 7f1397400640 1 --2- 172.21.15.90:0/3511868929 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559225a22080 0x559225a1fb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:40:13.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.714+0000 7f1396a00640 1 --2- 172.21.15.90:0/3511868929 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559225a18d10 0x559225a1f080 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f13880cdbe0 tx=0x7f138809cc40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f7db81f778f9086b server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:40:13.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.714+0000 7f1380c00640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:40:13.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.714+0000 7f1380c00640 1 -- 172.21.15.90:0/3511868929 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1388020070 con 0x559225a18d10 2024-04-02T14:40:13.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.714+0000 7f1380c00640 1 -- 172.21.15.90:0/3511868929 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f1388098a40 con 0x559225a18d10 2024-04-02T14:40:13.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.714+0000 7f13a09e1740 1 -- 172.21.15.90:0/3511868929 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1384011420 con 0x559225a18d10 2024-04-02T14:40:13.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.715+0000 7f1380c00640 1 -- 172.21.15.90:0/3511868929 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f13880a5040 con 0x559225a18d10 2024-04-02T14:40:13.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.715+0000 7f13a09e1740 10 client.26996 fetch_fsmap learned FSMap version 27 2024-04-02T14:40:13.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.715+0000 7f13a09e1740 1 -- 172.21.15.90:0/3511868929 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x559225a30ff0 con 0x559225a18d10 2024-04-02T14:40:13.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.715+0000 7f1380c00640 1 -- 172.21.15.90:0/3511868929 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1384011420 con 0x559225a18d10 2024-04-02T14:40:13.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.715+0000 7f1380c00640 1 -- 172.21.15.90:0/3511868929 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f13880b5b50 con 0x559225a18d10 2024-04-02T14:40:13.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.715+0000 7f13a09e1740 10 client.26996 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:40:13.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.715+0000 7f13a09e1740 10 client.26996 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.c 2024-04-02T14:40:13.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.715+0000 7f13a09e1740 20 client.26996 populate_metadata read hostname 'smithi090' 2024-04-02T14:40:13.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.715+0000 7f13a09e1740 1 --2- 172.21.15.90:0/3511868929 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x559225a32500 0x559225a34930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:13.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.716+0000 7f13a09e1740 4 client.26996 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:40:13.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.716+0000 7f13a09e1740 1 -- 172.21.15.90:0/3511868929 --> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55922593d040 con 0x559225a32500 2024-04-02T14:40:13.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.717+0000 7f1396000640 1 --2- 172.21.15.90:0/3511868929 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x559225a32500 0x559225a34930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:13.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.718+0000 7f1380c00640 10 client.26996 ms_handle_connect on v2:172.21.15.177:6836/4223414858 2024-04-02T14:40:13.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.720+0000 7f1396000640 1 --2- 172.21.15.90:0/3511868929 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x559225a32500 0x559225a34930 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=90c31ef73141b86e server_cookie=bf4b568a5912b4f2 in_seq=0 out_seq=0 2024-04-02T14:40:13.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.721+0000 7f1380c00640 1 -- 172.21.15.90:0/3511868929 <== mds.4 v2:172.21.15.177:6836/4223414858 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55922593d040 con 0x559225a32500 2024-04-02T14:40:13.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.721+0000 7f1380c00640 10 client.26996 handle_command_reply: tid=0 2024-04-02T14:40:13.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.721+0000 7f13a09e1740 1 client.26996 shutdown 2024-04-02T14:40:13.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.723+0000 7f13a09e1740 1 -- 172.21.15.90:0/3511868929 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559225a18d10 msgr2=0x559225a1f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:13.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.723+0000 7f13a09e1740 1 --2- 172.21.15.90:0/3511868929 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559225a18d10 0x559225a1f080 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f13880cdbe0 tx=0x7f138809cc40 comp rx=0 tx=0).stop 2024-04-02T14:40:13.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.723+0000 7f13a09e1740 1 -- 172.21.15.90:0/3511868929 shutdown_connections 2024-04-02T14:40:13.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.723+0000 7f13a09e1740 1 --2- 172.21.15.90:0/3511868929 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x559225a32500 0x559225a34930 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:13.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.723+0000 7f13a09e1740 1 --2- 172.21.15.90:0/3511868929 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559225a22080 0x559225a1fb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:13.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.723+0000 7f13a09e1740 1 --2- 172.21.15.90:0/3511868929 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559225a199f0 0x559225a1f5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:13.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.723+0000 7f13a09e1740 1 --2- 172.21.15.90:0/3511868929 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559225a18d10 0x559225a1f080 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:13.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.723+0000 7f13a09e1740 1 -- 172.21.15.90:0/3511868929 >> 172.21.15.90:0/3511868929 conn(0x5592259dfa50 msgr2=0x559225a154b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:40:13.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.723+0000 7f1380c00640 0 client.26996 ms_handle_reset on v2:172.21.15.177:6836/4223414858 2024-04-02T14:40:13.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.724+0000 7f13a09e1740 1 -- 172.21.15.90:0/3511868929 shutdown_connections 2024-04-02T14:40:13.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.724+0000 7f13a09e1740 1 -- 172.21.15.90:0/3511868929 wait complete. 2024-04-02T14:40:13.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.724+0000 7f13a09e1740 20 client.26996 trim_cache size 0 max 16384 2024-04-02T14:40:13.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.747+0000 7f13a09e1740 1 Processor -- start 2024-04-02T14:40:13.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.747+0000 7f13a09e1740 1 -- start start 2024-04-02T14:40:13.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.748+0000 7f13a09e1740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559225a18d10 0x559225a217b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:13.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.748+0000 7f13a09e1740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559225a199f0 0x559225a2db50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:13.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.748+0000 7f13a09e1740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559225a22080 0x559225a3a200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:13.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.748+0000 7f1396000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559225a199f0 0x559225a2db50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:13.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.748+0000 7f13a09e1740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f13981b3fd0 con 0x559225a22080 2024-04-02T14:40:13.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.748+0000 7f13a09e1740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f13981b5b10 con 0x559225a18d10 2024-04-02T14:40:13.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.748+0000 7f13a09e1740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f13981b46a0 con 0x559225a199f0 2024-04-02T14:40:13.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.748+0000 7f1396000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559225a199f0 0x559225a2db50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:49020/0 (socket says 172.21.15.90:49020) 2024-04-02T14:40:13.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.748+0000 7f1396000640 1 -- 172.21.15.90:0/3251475177 learned_addr learned my addr 172.21.15.90:0/3251475177 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:40:13.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.748+0000 7f1396a00640 1 --2- 172.21.15.90:0/3251475177 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559225a18d10 0x559225a217b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:13.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.749+0000 7f1397400640 1 --2- 172.21.15.90:0/3251475177 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559225a22080 0x559225a3a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:13.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.749+0000 7f1396000640 1 -- 172.21.15.90:0/3251475177 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559225a18d10 msgr2=0x559225a217b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:40:13.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.749+0000 7f1396000640 1 --2- 172.21.15.90:0/3251475177 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559225a18d10 0x559225a217b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:13.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.749+0000 7f1396000640 1 -- 172.21.15.90:0/3251475177 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559225a22080 msgr2=0x559225a3a200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:40:13.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.749+0000 7f1396000640 1 --2- 172.21.15.90:0/3251475177 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559225a22080 0x559225a3a200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:13.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.749+0000 7f1396000640 1 -- 172.21.15.90:0/3251475177 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559225a17d40 con 0x559225a199f0 2024-04-02T14:40:13.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.749+0000 7f1396a00640 1 --2- 172.21.15.90:0/3251475177 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559225a18d10 0x559225a217b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:40:13.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.750+0000 7f1396000640 1 --2- 172.21.15.90:0/3251475177 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559225a199f0 0x559225a2db50 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f13840097c0 tx=0x7f1384009190 comp rx=0 tx=0).ready entity=mon.1 client_cookie=62f40ebf7e39bd server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:40:13.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.751+0000 7f1383400640 1 -- 172.21.15.90:0/3251475177 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1384015070 con 0x559225a199f0 2024-04-02T14:40:13.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.751+0000 7f1383400640 1 -- 172.21.15.90:0/3251475177 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1384025910 con 0x559225a199f0 2024-04-02T14:40:13.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.751+0000 7f1383400640 1 -- 172.21.15.90:0/3251475177 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1384011680 con 0x559225a199f0 2024-04-02T14:40:13.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.752+0000 7f13a09e1740 1 -- 172.21.15.90:0/3251475177 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559225a199f0 msgr2=0x559225a2db50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:13.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.752+0000 7f13a09e1740 1 --2- 172.21.15.90:0/3251475177 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559225a199f0 0x559225a2db50 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f13840097c0 tx=0x7f1384009190 comp rx=0 tx=0).stop 2024-04-02T14:40:13.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.752+0000 7f13a09e1740 1 -- 172.21.15.90:0/3251475177 shutdown_connections 2024-04-02T14:40:13.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.752+0000 7f13a09e1740 1 --2- 172.21.15.90:0/3251475177 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559225a22080 0x559225a3a200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:13.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.752+0000 7f13a09e1740 1 --2- 172.21.15.90:0/3251475177 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559225a199f0 0x559225a2db50 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:13.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.752+0000 7f13a09e1740 1 --2- 172.21.15.90:0/3251475177 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559225a18d10 0x559225a217b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:13.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.752+0000 7f13a09e1740 1 -- 172.21.15.90:0/3251475177 >> 172.21.15.90:0/3251475177 conn(0x5592259dfa50 msgr2=0x559225a13290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:40:13.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.752+0000 7f13a09e1740 1 -- 172.21.15.90:0/3251475177 shutdown_connections 2024-04-02T14:40:13.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.752+0000 7f13a09e1740 1 -- 172.21.15.90:0/3251475177 wait complete. 2024-04-02T14:40:13.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.753+0000 7f13a09e1740 1 -- start start 2024-04-02T14:40:13.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.753+0000 7f13a09e1740 1 Processor -- start 2024-04-02T14:40:13.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.754+0000 7f13a09e1740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559225a18d10 0x559225a2a250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:13.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.754+0000 7f13a09e1740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559225a199f0 0x559225a1c300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:13.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.754+0000 7f13a09e1740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559225a22080 0x559225a1c840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:13.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.754+0000 7f13a09e1740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f13981b5440 con 0x559225a199f0 2024-04-02T14:40:13.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.754+0000 7f13a09e1740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f13981b4d70 con 0x559225a22080 2024-04-02T14:40:13.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.754+0000 7f13a09e1740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f13981b1f70 con 0x559225a18d10 2024-04-02T14:40:13.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.754+0000 7f13a09e1740 10 client.0 _ll_register_callbacks cb 0x7f138809bf90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:40:13.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.754+0000 7f1396a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559225a18d10 0x559225a2a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:13.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.755+0000 7f1396a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559225a18d10 0x559225a2a250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:49032/0 (socket says 172.21.15.90:49032) 2024-04-02T14:40:13.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.755+0000 7f1396a00640 1 -- 172.21.15.90:0/3610768524 learned_addr learned my addr 172.21.15.90:0/3610768524 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:40:13.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.755+0000 7f1397400640 1 --2- 172.21.15.90:0/3610768524 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559225a22080 0x559225a1c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:13.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.755+0000 7f1396000640 1 --2- 172.21.15.90:0/3610768524 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559225a199f0 0x559225a1c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:13.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.755+0000 7f1396a00640 1 -- 172.21.15.90:0/3610768524 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559225a22080 msgr2=0x559225a1c840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:40:13.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.755+0000 7f1396a00640 1 --2- 172.21.15.90:0/3610768524 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559225a22080 0x559225a1c840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:13.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.755+0000 7f1396a00640 1 -- 172.21.15.90:0/3610768524 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559225a199f0 msgr2=0x559225a1c300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:40:13.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.755+0000 7f1396a00640 1 --2- 172.21.15.90:0/3610768524 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559225a199f0 0x559225a1c300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:13.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.755+0000 7f1396a00640 1 -- 172.21.15.90:0/3610768524 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559225a1ac80 con 0x559225a18d10 2024-04-02T14:40:13.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.755+0000 7f1396000640 1 --2- 172.21.15.90:0/3610768524 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559225a199f0 0x559225a1c300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:40:13.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.756+0000 7f1396a00640 1 --2- 172.21.15.90:0/3610768524 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559225a18d10 0x559225a2a250 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f13880a4b10 tx=0x7f1388005880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=be624a01544af969 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:40:13.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.756+0000 7f1380c00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:40:13.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.756+0000 7f1380c00640 1 -- 172.21.15.90:0/3610768524 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13880a5040 con 0x559225a18d10 2024-04-02T14:40:13.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.756+0000 7f1380c00640 1 -- 172.21.15.90:0/3610768524 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f13880109d0 con 0x559225a18d10 2024-04-02T14:40:13.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.756+0000 7f13a09e1740 1 -- 172.21.15.90:0/3610768524 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1384025910 con 0x559225a18d10 2024-04-02T14:40:13.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.757+0000 7f1380c00640 1 -- 172.21.15.90:0/3610768524 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f138809bb40 con 0x559225a18d10 2024-04-02T14:40:13.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.757+0000 7f13a09e1740 10 client.26887 fetch_fsmap learned FSMap version 27 2024-04-02T14:40:13.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.757+0000 7f13a09e1740 1 -- 172.21.15.90:0/3610768524 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x559225a1ac80 con 0x559225a18d10 2024-04-02T14:40:13.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.757+0000 7f1380c00640 1 -- 172.21.15.90:0/3610768524 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1384025910 con 0x559225a18d10 2024-04-02T14:40:13.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.757+0000 7f1380c00640 1 -- 172.21.15.90:0/3610768524 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f1388010460 con 0x559225a18d10 2024-04-02T14:40:13.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.757+0000 7f13a09e1740 10 client.26887 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:40:13.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.757+0000 7f13a09e1740 10 client.26887 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.c 2024-04-02T14:40:13.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.757+0000 7f13a09e1740 20 client.26887 populate_metadata read hostname 'smithi090' 2024-04-02T14:40:13.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.757+0000 7f13a09e1740 1 --2- 172.21.15.90:0/3610768524 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x559225a98a70 0x559225a9ae60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:13.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.757+0000 7f13a09e1740 4 client.26887 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-02T14:40:13.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.757+0000 7f13a09e1740 1 -- 172.21.15.90:0/3610768524 --> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55922598ac30 con 0x559225a98a70 2024-04-02T14:40:13.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.758+0000 7f1396000640 1 --2- 172.21.15.90:0/3610768524 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x559225a98a70 0x559225a9ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:13.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.759+0000 7f1396000640 1 --2- 172.21.15.90:0/3610768524 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x559225a98a70 0x559225a9ae60 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=e1ca60ae53fcb609 server_cookie=256c9dbce61db7ac in_seq=0 out_seq=0 2024-04-02T14:40:13.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.759+0000 7f1380c00640 10 client.26887 ms_handle_connect on v2:172.21.15.177:6836/4223414858 2024-04-02T14:40:13.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.759+0000 7f1380c00640 1 -- 172.21.15.90:0/3610768524 <== mds.4 v2:172.21.15.177:6836/4223414858 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55922598ac30 con 0x559225a98a70 2024-04-02T14:40:13.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.759+0000 7f1380c00640 10 client.26887 handle_command_reply: tid=0 2024-04-02T14:40:13.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.759+0000 7f13a09e1740 1 client.26887 shutdown 2024-04-02T14:40:13.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.759+0000 7f13a09e1740 1 -- 172.21.15.90:0/3610768524 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559225a18d10 msgr2=0x559225a2a250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:13.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.759+0000 7f13a09e1740 1 --2- 172.21.15.90:0/3610768524 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559225a18d10 0x559225a2a250 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f13880a4b10 tx=0x7f1388005880 comp rx=0 tx=0).stop 2024-04-02T14:40:13.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.759+0000 7f13a09e1740 1 -- 172.21.15.90:0/3610768524 shutdown_connections 2024-04-02T14:40:13.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.759+0000 7f13a09e1740 1 --2- 172.21.15.90:0/3610768524 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559225a22080 0x559225a1c840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:13.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.759+0000 7f13a09e1740 1 --2- 172.21.15.90:0/3610768524 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x559225a98a70 0x559225a9ae60 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:13.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.759+0000 7f13a09e1740 1 --2- 172.21.15.90:0/3610768524 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559225a199f0 0x559225a1c300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:13.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.759+0000 7f13a09e1740 1 --2- 172.21.15.90:0/3610768524 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559225a18d10 0x559225a2a250 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:13.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.759+0000 7f1380c00640 0 client.26887 ms_handle_reset on v2:172.21.15.177:6836/4223414858 2024-04-02T14:40:13.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.759+0000 7f13a09e1740 1 -- 172.21.15.90:0/3610768524 >> 172.21.15.90:0/3610768524 conn(0x5592259dfa50 msgr2=0x559225a1d8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:40:13.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.761+0000 7f13a09e1740 1 -- 172.21.15.90:0/3610768524 shutdown_connections 2024-04-02T14:40:13.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.761+0000 7f13a09e1740 1 -- 172.21.15.90:0/3610768524 wait complete. 2024-04-02T14:40:13.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.761+0000 7f13a09e1740 20 client.26887 trim_cache size 0 max 16384 2024-04-02T14:40:13.763 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-02T14:40:13.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.761+0000 7f139de00640 1 -- 172.21.15.90:0/1482349326 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f137c0a3eb0 msgr2=0x7f137c0a6370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:13.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.761+0000 7f139de00640 1 --2- 172.21.15.90:0/1482349326 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f137c0a3eb0 0x7f137c0a6370 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f139002c620 tx=0x7f1390004a80 comp rx=0 tx=0).stop 2024-04-02T14:40:13.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.761+0000 7f139de00640 1 -- 172.21.15.90:0/1482349326 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1398101a90 msgr2=0x7f1398073e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:13.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.761+0000 7f139de00640 1 --2- 172.21.15.90:0/1482349326 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1398101a90 0x7f1398073e00 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f138800d950 tx=0x7f138800de10 comp rx=0 tx=0).stop 2024-04-02T14:40:13.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.762+0000 7f139de00640 1 -- 172.21.15.90:0/1482349326 shutdown_connections 2024-04-02T14:40:13.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.762+0000 7f139de00640 1 --2- 172.21.15.90:0/1482349326 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f137c0a3eb0 0x7f137c0a6370 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:13.764 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.762+0000 7f139de00640 1 --2- 172.21.15.90:0/1482349326 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1398107aa0 0x7f1398074880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:13.764 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.762+0000 7f139de00640 1 --2- 172.21.15.90:0/1482349326 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1398102450 0x7f1398074340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:13.764 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.762+0000 7f139de00640 1 --2- 172.21.15.90:0/1482349326 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1398101a90 0x7f1398073e00 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:13.764 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.762+0000 7f139de00640 1 -- 172.21.15.90:0/1482349326 >> 172.21.15.90:0/1482349326 conn(0x7f1398078f00 msgr2=0x7f1398104360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:40:13.764 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.763+0000 7f139de00640 1 -- 172.21.15.90:0/1482349326 shutdown_connections 2024-04-02T14:40:13.764 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:13.763+0000 7f139de00640 1 -- 172.21.15.90:0/1482349326 wait complete. 2024-04-02T14:40:13.793 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-04-02T14:40:13.860 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:40:13 smithi092 ceph-mon[36497]: pgmap v163: 97 pgs: 97 active+clean; 3.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 21 MiB/s wr, 1.31k op/s 2024-04-02T14:40:13.860 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:40:13 smithi092 ceph-mon[36497]: from='client.? 172.21.15.90:0/1516673928' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-02T14:40:13.935 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:40:13 smithi177 ceph-mon[36431]: pgmap v163: 97 pgs: 97 active+clean; 3.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 21 MiB/s wr, 1.31k op/s 2024-04-02T14:40:13.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:40:13 smithi177 ceph-mon[36431]: from='client.? 172.21.15.90:0/1516673928' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-02T14:40:14.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.078+0000 7fecbb200640 1 -- 172.21.15.90:0/1218357569 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fecb4068d00 msgr2=0x7fecb4069180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:14.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.078+0000 7fecbb200640 1 --2- 172.21.15.90:0/1218357569 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fecb4068d00 0x7fecb4069180 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7feca4009890 tx=0x7feca402bb70 comp rx=0 tx=0).stop 2024-04-02T14:40:14.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.078+0000 7fecbb200640 1 -- 172.21.15.90:0/1218357569 shutdown_connections 2024-04-02T14:40:14.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.079+0000 7fecbb200640 1 --2- 172.21.15.90:0/1218357569 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fecb40696c0 0x7fecb410e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:14.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.079+0000 7fecbb200640 1 --2- 172.21.15.90:0/1218357569 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fecb4068d00 0x7fecb4069180 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:14.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.079+0000 7fecbb200640 1 --2- 172.21.15.90:0/1218357569 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fecb4107b50 0x7fecb4107f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:14.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.079+0000 7fecbb200640 1 -- 172.21.15.90:0/1218357569 >> 172.21.15.90:0/1218357569 conn(0x7fecb4077b00 msgr2=0x7fecb4077f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:40:14.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.079+0000 7fecbb200640 1 -- 172.21.15.90:0/1218357569 shutdown_connections 2024-04-02T14:40:14.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.079+0000 7fecbb200640 1 -- 172.21.15.90:0/1218357569 wait complete. 2024-04-02T14:40:14.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.080+0000 7fecbb200640 1 Processor -- start 2024-04-02T14:40:14.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.080+0000 7fecbb200640 1 -- start start 2024-04-02T14:40:14.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.080+0000 7fecbb200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fecb4068d00 0x7fecb41a46b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:14.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.080+0000 7fecbb200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fecb40696c0 0x7fecb41a4bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:14.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.080+0000 7fecbb200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fecb4107b50 0x7fecb419e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:14.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.081+0000 7fecbb200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fecb410fd00 con 0x7fecb4107b50 2024-04-02T14:40:14.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.081+0000 7fecbb200640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fecb410fb80 con 0x7fecb4068d00 2024-04-02T14:40:14.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.081+0000 7fecbb200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fecb410fe80 con 0x7fecb40696c0 2024-04-02T14:40:14.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.081+0000 7fecb3e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fecb4068d00 0x7fecb41a46b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:14.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.081+0000 7fecb3e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fecb4068d00 0x7fecb41a46b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:39154/0 (socket says 172.21.15.90:39154) 2024-04-02T14:40:14.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.081+0000 7fecb3e00640 1 -- 172.21.15.90:0/2542116354 learned_addr learned my addr 172.21.15.90:0/2542116354 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:40:14.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.082+0000 7fecb3e00640 1 -- 172.21.15.90:0/2542116354 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fecb40696c0 msgr2=0x7fecb41a4bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:14.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.082+0000 7fecb3400640 1 --2- 172.21.15.90:0/2542116354 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fecb40696c0 0x7fecb41a4bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:14.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.082+0000 7fecb3e00640 1 --2- 172.21.15.90:0/2542116354 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fecb40696c0 0x7fecb41a4bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:14.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.082+0000 7fecb3e00640 1 -- 172.21.15.90:0/2542116354 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fecb4107b50 msgr2=0x7fecb419e770 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T14:40:14.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.082+0000 7fecb9200640 1 --2- 172.21.15.90:0/2542116354 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fecb4107b50 0x7fecb419e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:14.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.082+0000 7fecb3e00640 1 --2- 172.21.15.90:0/2542116354 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fecb4107b50 0x7fecb419e770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:14.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.082+0000 7fecb3e00640 1 -- 172.21.15.90:0/2542116354 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fecb419f030 con 0x7fecb4068d00 2024-04-02T14:40:14.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.082+0000 7fecb3400640 1 --2- 172.21.15.90:0/2542116354 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fecb40696c0 0x7fecb41a4bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:40:14.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.083+0000 7fecb3e00640 1 --2- 172.21.15.90:0/2542116354 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fecb4068d00 0x7fecb41a46b0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fec9c00e9c0 tx=0x7fec9c00ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:40:14.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.083+0000 7fecb0c00640 1 -- 172.21.15.90:0/2542116354 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec9c00cde0 con 0x7fecb4068d00 2024-04-02T14:40:14.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.083+0000 7fecb0c00640 1 -- 172.21.15.90:0/2542116354 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fec9c0035f0 con 0x7fecb4068d00 2024-04-02T14:40:14.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.083+0000 7fecbb200640 1 -- 172.21.15.90:0/2542116354 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fecb419f320 con 0x7fecb4068d00 2024-04-02T14:40:14.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.083+0000 7fecb0c00640 1 -- 172.21.15.90:0/2542116354 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec9c010690 con 0x7fecb4068d00 2024-04-02T14:40:14.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.083+0000 7fecbb200640 1 -- 172.21.15.90:0/2542116354 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fecb41ab630 con 0x7fecb4068d00 2024-04-02T14:40:14.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.088+0000 7fecb0c00640 1 -- 172.21.15.90:0/2542116354 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fec9c010830 con 0x7fecb4068d00 2024-04-02T14:40:14.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.090+0000 7fecbdfd2740 1 Processor -- start 2024-04-02T14:40:14.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.090+0000 7fecbdfd2740 1 -- start start 2024-04-02T14:40:14.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.090+0000 7fecb0c00640 1 --2- 172.21.15.90:0/2542116354 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fec980a3df0 0x7fec980a62b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:14.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.090+0000 7fecbdfd2740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ad080e89f0 0x55ad080e7740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:14.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.090+0000 7fecbdfd2740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ad080e7d10 0x55ad080e8190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:14.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.090+0000 7fecbdfd2740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ad080f1080 0x55ad080f3470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:14.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.090+0000 7fecbdfd2740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55ad08028400 con 0x55ad080f1080 2024-04-02T14:40:14.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.090+0000 7fecbdfd2740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55ad0800cf10 con 0x55ad080e89f0 2024-04-02T14:40:14.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.090+0000 7fecbdfd2740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55ad0804c6f0 con 0x55ad080e7d10 2024-04-02T14:40:14.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.091+0000 7fecb0c00640 1 -- 172.21.15.90:0/2542116354 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7fec9c0d0c20 con 0x7fecb4068d00 2024-04-02T14:40:14.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.091+0000 7fecb3400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ad080e7d10 0x55ad080e8190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:14.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.091+0000 7fecb3400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ad080e7d10 0x55ad080e8190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:53830/0 (socket says 172.21.15.90:53830) 2024-04-02T14:40:14.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.091+0000 7fecb3400640 1 -- 172.21.15.90:0/3738857759 learned_addr learned my addr 172.21.15.90:0/3738857759 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:40:14.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.091+0000 7fecb9200640 1 --2- 172.21.15.90:0/2542116354 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fec980a3df0 0x7fec980a62b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:14.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.092+0000 7fecb3400640 1 -- 172.21.15.90:0/3738857759 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ad080e89f0 msgr2=0x55ad080e7740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:40:14.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.091+0000 7fecb3e00640 1 --2- 172.21.15.90:0/3738857759 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ad080e89f0 0x55ad080e7740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:14.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.092+0000 7fecb3400640 1 --2- 172.21.15.90:0/3738857759 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ad080e89f0 0x55ad080e7740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:14.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.092+0000 7fecb3400640 1 -- 172.21.15.90:0/3738857759 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ad080f1080 msgr2=0x55ad080f3470 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-02T14:40:14.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.092+0000 7fecb9200640 1 --2- 172.21.15.90:0/3738857759 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ad080f1080 0x55ad080f3470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:14.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.093+0000 7fecb3400640 1 --2- 172.21.15.90:0/3738857759 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ad080f1080 0x55ad080f3470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:14.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.093+0000 7fecb3400640 1 -- 172.21.15.90:0/3738857759 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ad07d603f0 con 0x55ad080e7d10 2024-04-02T14:40:14.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.093+0000 7fecb3e00640 1 --2- 172.21.15.90:0/3738857759 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ad080e89f0 0x55ad080e7740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:40:14.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.093+0000 7fecb9200640 1 --2- 172.21.15.90:0/3738857759 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ad080f1080 0x55ad080f3470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:40:14.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.094+0000 7fecb3400640 1 --2- 172.21.15.90:0/3738857759 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ad080e7d10 0x55ad080e8190 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7feca4000c00 tx=0x7feca4032340 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6927708c94002238 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:40:14.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.094+0000 7feca9600640 1 -- 172.21.15.90:0/3738857759 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feca4039070 con 0x55ad080e7d10 2024-04-02T14:40:14.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.094+0000 7feca9600640 1 -- 172.21.15.90:0/3738857759 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feca403a040 con 0x55ad080e7d10 2024-04-02T14:40:14.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.094+0000 7feca9600640 1 -- 172.21.15.90:0/3738857759 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feca40052a0 con 0x55ad080e7d10 2024-04-02T14:40:14.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.094+0000 7fecb9200640 1 --2- 172.21.15.90:0/2542116354 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fec980a3df0 0x7fec980a62b0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fecac005c40 tx=0x7fecac005bd0 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:40:14.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.096+0000 7fecbdfd2740 1 -- 172.21.15.90:0/3738857759 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ad080e7d10 msgr2=0x55ad080e8190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:14.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.096+0000 7fecbdfd2740 1 --2- 172.21.15.90:0/3738857759 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ad080e7d10 0x55ad080e8190 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7feca4000c00 tx=0x7feca4032340 comp rx=0 tx=0).stop 2024-04-02T14:40:14.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.097+0000 7fecbdfd2740 1 -- 172.21.15.90:0/3738857759 shutdown_connections 2024-04-02T14:40:14.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.097+0000 7fecbdfd2740 1 --2- 172.21.15.90:0/3738857759 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ad080f1080 0x55ad080f3470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:14.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.097+0000 7fecbdfd2740 1 --2- 172.21.15.90:0/3738857759 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ad080e7d10 0x55ad080e8190 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:14.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.097+0000 7fecbdfd2740 1 --2- 172.21.15.90:0/3738857759 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ad080e89f0 0x55ad080e7740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:14.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.097+0000 7fecbdfd2740 1 -- 172.21.15.90:0/3738857759 >> 172.21.15.90:0/3738857759 conn(0x55ad080aea50 msgr2=0x55ad080e2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:40:14.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.098+0000 7fecbdfd2740 1 -- 172.21.15.90:0/3738857759 shutdown_connections 2024-04-02T14:40:14.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.098+0000 7fecbdfd2740 1 -- 172.21.15.90:0/3738857759 wait complete. 2024-04-02T14:40:14.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.098+0000 7fecbdfd2740 1 -- start start 2024-04-02T14:40:14.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.098+0000 7fecbdfd2740 1 Processor -- start 2024-04-02T14:40:14.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.099+0000 7fecbdfd2740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ad080e7d10 0x55ad080ee000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:14.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.099+0000 7fecbdfd2740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ad080e89f0 0x55ad080ee540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:14.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.099+0000 7fecbdfd2740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ad080f1080 0x55ad080eea80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:14.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.099+0000 7fecbdfd2740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55ad080486a0 con 0x55ad080e89f0 2024-04-02T14:40:14.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.099+0000 7fecbdfd2740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55ad07ff4980 con 0x55ad080e7d10 2024-04-02T14:40:14.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.099+0000 7fecbdfd2740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55ad08007cb0 con 0x55ad080f1080 2024-04-02T14:40:14.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.099+0000 7fecbdfd2740 10 client.0 _ll_register_callbacks cb 0x55ad080076a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:40:14.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.099+0000 7fecb3e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ad080e7d10 0x55ad080ee000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:14.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.099+0000 7fecb3e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ad080e7d10 0x55ad080ee000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:39172/0 (socket says 172.21.15.90:39172) 2024-04-02T14:40:14.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.099+0000 7fecb3e00640 1 -- 172.21.15.90:0/208226379 learned_addr learned my addr 172.21.15.90:0/208226379 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:40:14.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.100+0000 7fecb9200640 1 --2- 172.21.15.90:0/208226379 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ad080f1080 0x55ad080eea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:14.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.100+0000 7fecb3e00640 1 -- 172.21.15.90:0/208226379 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ad080f1080 msgr2=0x55ad080eea80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:40:14.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.100+0000 7fecb3e00640 1 --2- 172.21.15.90:0/208226379 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ad080f1080 0x55ad080eea80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:14.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.100+0000 7fecb3e00640 1 -- 172.21.15.90:0/208226379 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ad080e89f0 msgr2=0x55ad080ee540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:40:14.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.100+0000 7fecb3e00640 1 --2- 172.21.15.90:0/208226379 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ad080e89f0 0x55ad080ee540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:14.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.100+0000 7fecb3e00640 1 -- 172.21.15.90:0/208226379 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ad07e2a7d0 con 0x55ad080e7d10 2024-04-02T14:40:14.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.100+0000 7fecb3e00640 1 --2- 172.21.15.90:0/208226379 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ad080e7d10 0x55ad080ee000 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fec9c0995a0 tx=0x7fec9c09cd10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9348abdf42f76fba server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:40:14.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.101+0000 7feca2000640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:40:14.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.101+0000 7feca2000640 1 -- 172.21.15.90:0/208226379 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec9c0b6060 con 0x55ad080e7d10 2024-04-02T14:40:14.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.101+0000 7feca2000640 1 -- 172.21.15.90:0/208226379 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7fec9c099240 con 0x55ad080e7d10 2024-04-02T14:40:14.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.101+0000 7fecbdfd2740 1 -- 172.21.15.90:0/208226379 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7feca403a040 con 0x55ad080e7d10 2024-04-02T14:40:14.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.101+0000 7feca2000640 1 -- 172.21.15.90:0/208226379 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fec9c0037a0 con 0x55ad080e7d10 2024-04-02T14:40:14.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.101+0000 7fecbdfd2740 10 client.27011 fetch_fsmap learned FSMap version 27 2024-04-02T14:40:14.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.101+0000 7fecbdfd2740 1 -- 172.21.15.90:0/208226379 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55ad080ffe50 con 0x55ad080e7d10 2024-04-02T14:40:14.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.102+0000 7feca2000640 1 -- 172.21.15.90:0/208226379 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feca403a040 con 0x55ad080e7d10 2024-04-02T14:40:14.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.102+0000 7feca2000640 1 -- 172.21.15.90:0/208226379 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fec9c004000 con 0x55ad080e7d10 2024-04-02T14:40:14.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.102+0000 7fecbdfd2740 10 client.27011 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:40:14.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.102+0000 7fecbdfd2740 10 client.27011 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-04-02T14:40:14.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.102+0000 7fecbdfd2740 20 client.27011 populate_metadata read hostname 'smithi090' 2024-04-02T14:40:14.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.102+0000 7fecbdfd2740 1 --2- 172.21.15.90:0/208226379 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x55ad08101310 0x55ad081037a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:14.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.102+0000 7fecbdfd2740 4 client.27011 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:40:14.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.102+0000 7fecbdfd2740 1 -- 172.21.15.90:0/208226379 --> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ad0800c040 con 0x55ad08101310 2024-04-02T14:40:14.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.103+0000 7fecb3400640 1 --2- 172.21.15.90:0/208226379 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x55ad08101310 0x55ad081037a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:14.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.103+0000 7fecb3400640 1 --2- 172.21.15.90:0/208226379 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x55ad08101310 0x55ad081037a0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=7bbf6046b39b463 server_cookie=c034b0da05be02a9 in_seq=0 out_seq=0 2024-04-02T14:40:14.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.103+0000 7feca2000640 10 client.27011 ms_handle_connect on v2:172.21.15.92:6836/1270635939 2024-04-02T14:40:14.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.104+0000 7feca2000640 1 -- 172.21.15.90:0/208226379 <== mds.3 v2:172.21.15.92:6836/1270635939 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ad0800c040 con 0x55ad08101310 2024-04-02T14:40:14.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.104+0000 7feca2000640 10 client.27011 handle_command_reply: tid=0 2024-04-02T14:40:14.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.105+0000 7fecbdfd2740 1 client.27011 shutdown 2024-04-02T14:40:14.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.105+0000 7fecbdfd2740 1 -- 172.21.15.90:0/208226379 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ad080e7d10 msgr2=0x55ad080ee000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:14.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.105+0000 7fecbdfd2740 1 --2- 172.21.15.90:0/208226379 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ad080e7d10 0x55ad080ee000 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fec9c0995a0 tx=0x7fec9c09cd10 comp rx=0 tx=0).stop 2024-04-02T14:40:14.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.106+0000 7fecbdfd2740 1 -- 172.21.15.90:0/208226379 shutdown_connections 2024-04-02T14:40:14.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.106+0000 7fecbdfd2740 1 --2- 172.21.15.90:0/208226379 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x55ad08101310 0x55ad081037a0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:14.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.106+0000 7fecbdfd2740 1 --2- 172.21.15.90:0/208226379 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ad080f1080 0x55ad080eea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:14.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.106+0000 7fecbdfd2740 1 --2- 172.21.15.90:0/208226379 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ad080e89f0 0x55ad080ee540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:14.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.106+0000 7fecbdfd2740 1 --2- 172.21.15.90:0/208226379 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ad080e7d10 0x55ad080ee000 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:14.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.107+0000 7fecbdfd2740 1 -- 172.21.15.90:0/208226379 >> 172.21.15.90:0/208226379 conn(0x55ad080aea50 msgr2=0x55ad080e44b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:40:14.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.107+0000 7feca2000640 0 client.27011 ms_handle_reset on v2:172.21.15.92:6836/1270635939 2024-04-02T14:40:14.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.108+0000 7fecbdfd2740 1 -- 172.21.15.90:0/208226379 shutdown_connections 2024-04-02T14:40:14.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.108+0000 7fecbdfd2740 1 -- 172.21.15.90:0/208226379 wait complete. 2024-04-02T14:40:14.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.109+0000 7fecbdfd2740 20 client.27011 trim_cache size 0 max 16384 2024-04-02T14:40:14.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.134+0000 7fecbdfd2740 1 Processor -- start 2024-04-02T14:40:14.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.135+0000 7fecbdfd2740 1 -- start start 2024-04-02T14:40:14.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.135+0000 7fecbdfd2740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ad080e7d10 0x55ad080f0730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:14.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.135+0000 7fecbdfd2740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ad080e89f0 0x55ad080fcac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:14.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.135+0000 7fecbdfd2740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ad080f1080 0x55ad08109070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:14.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.135+0000 7fecbdfd2740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fecb41b3fa0 con 0x55ad080e7d10 2024-04-02T14:40:14.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.135+0000 7fecbdfd2740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fecb41b5ae0 con 0x55ad080e89f0 2024-04-02T14:40:14.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.135+0000 7fecbdfd2740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fecb41b4670 con 0x55ad080f1080 2024-04-02T14:40:14.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.135+0000 7fecb3400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ad080e89f0 0x55ad080fcac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:14.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.136+0000 7fecb3400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ad080e89f0 0x55ad080fcac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:39188/0 (socket says 172.21.15.90:39188) 2024-04-02T14:40:14.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.136+0000 7fecb3400640 1 -- 172.21.15.90:0/844717428 learned_addr learned my addr 172.21.15.90:0/844717428 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:40:14.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.136+0000 7fecb9200640 1 --2- 172.21.15.90:0/844717428 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ad080f1080 0x55ad08109070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:14.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.136+0000 7fecb3e00640 1 --2- 172.21.15.90:0/844717428 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ad080e7d10 0x55ad080f0730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:14.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.136+0000 7fecb9200640 1 -- 172.21.15.90:0/844717428 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ad080e89f0 msgr2=0x55ad080fcac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:40:14.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.136+0000 7fecb9200640 1 --2- 172.21.15.90:0/844717428 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ad080e89f0 0x55ad080fcac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:14.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.136+0000 7fecb9200640 1 -- 172.21.15.90:0/844717428 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ad080e7d10 msgr2=0x55ad080f0730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:40:14.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.136+0000 7fecb9200640 1 --2- 172.21.15.90:0/844717428 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ad080e7d10 0x55ad080f0730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:14.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.136+0000 7fecb9200640 1 -- 172.21.15.90:0/844717428 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ad080e9bc0 con 0x55ad080f1080 2024-04-02T14:40:14.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.137+0000 7fecb9200640 1 --2- 172.21.15.90:0/844717428 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ad080f1080 0x55ad08109070 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fecac005fd0 tx=0x7fecac010000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e3f6ecc2c08bb3b1 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:40:14.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.138+0000 7feca8c00640 1 -- 172.21.15.90:0/844717428 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecac010460 con 0x55ad080f1080 2024-04-02T14:40:14.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.138+0000 7feca8c00640 1 -- 172.21.15.90:0/844717428 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fecac025070 con 0x55ad080f1080 2024-04-02T14:40:14.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.138+0000 7feca8c00640 1 -- 172.21.15.90:0/844717428 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecac0208c0 con 0x55ad080f1080 2024-04-02T14:40:14.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.139+0000 7fecbdfd2740 1 -- 172.21.15.90:0/844717428 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ad080f1080 msgr2=0x55ad08109070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:14.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.139+0000 7fecbdfd2740 1 --2- 172.21.15.90:0/844717428 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ad080f1080 0x55ad08109070 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fecac005fd0 tx=0x7fecac010000 comp rx=0 tx=0).stop 2024-04-02T14:40:14.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.139+0000 7fecbdfd2740 1 -- 172.21.15.90:0/844717428 shutdown_connections 2024-04-02T14:40:14.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.139+0000 7fecbdfd2740 1 --2- 172.21.15.90:0/844717428 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ad080f1080 0x55ad08109070 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:14.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.139+0000 7fecbdfd2740 1 --2- 172.21.15.90:0/844717428 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ad080e89f0 0x55ad080fcac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:14.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.139+0000 7fecbdfd2740 1 --2- 172.21.15.90:0/844717428 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ad080e7d10 0x55ad080f0730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:14.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.139+0000 7fecbdfd2740 1 -- 172.21.15.90:0/844717428 >> 172.21.15.90:0/844717428 conn(0x55ad080aea50 msgr2=0x55ad080e2290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:40:14.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.140+0000 7fecbdfd2740 1 -- 172.21.15.90:0/844717428 shutdown_connections 2024-04-02T14:40:14.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.140+0000 7fecbdfd2740 1 -- 172.21.15.90:0/844717428 wait complete. 2024-04-02T14:40:14.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.141+0000 7fecbdfd2740 1 -- start start 2024-04-02T14:40:14.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.141+0000 7fecbdfd2740 1 Processor -- start 2024-04-02T14:40:14.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.142+0000 7fecbdfd2740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ad080e7d10 0x55ad080f8f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:14.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.142+0000 7fecbdfd2740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ad080e89f0 0x55ad080f9480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:14.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.142+0000 7fecbdfd2740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ad080f1080 0x55ad080f45f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:14.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.143+0000 7fecb3400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ad080e89f0 0x55ad080f9480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:14.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.143+0000 7fecb3400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ad080e89f0 0x55ad080f9480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:53856/0 (socket says 172.21.15.90:53856) 2024-04-02T14:40:14.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.143+0000 7fecb3e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ad080e7d10 0x55ad080f8f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:14.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.143+0000 7fecbdfd2740 1 -- 172.21.15.90:0/3322698130 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fecb41b5410 con 0x55ad080f1080 2024-04-02T14:40:14.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.143+0000 7fecbdfd2740 1 -- 172.21.15.90:0/3322698130 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fecb41b4d40 con 0x55ad080e7d10 2024-04-02T14:40:14.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.143+0000 7fecbdfd2740 1 -- 172.21.15.90:0/3322698130 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fecb41b1f80 con 0x55ad080e89f0 2024-04-02T14:40:14.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.143+0000 7fecb3400640 1 -- 172.21.15.90:0/3322698130 learned_addr learned my addr 172.21.15.90:0/3322698130 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:40:14.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.143+0000 7fecbdfd2740 10 client.0 _ll_register_callbacks cb 0x7fec9c09cbd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:40:14.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.144+0000 7fecb3400640 1 -- 172.21.15.90:0/3322698130 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ad080e7d10 msgr2=0x55ad080f8f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:40:14.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.144+0000 7fecb3400640 1 --2- 172.21.15.90:0/3322698130 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ad080e7d10 0x55ad080f8f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:14.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.144+0000 7fecb3400640 1 -- 172.21.15.90:0/3322698130 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ad080f1080 msgr2=0x55ad080f45f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:40:14.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.144+0000 7fecb3400640 1 --2- 172.21.15.90:0/3322698130 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ad080f1080 0x55ad080f45f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:14.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.144+0000 7fecb3400640 1 -- 172.21.15.90:0/3322698130 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ad080f4e20 con 0x55ad080e89f0 2024-04-02T14:40:14.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.144+0000 7fecb3e00640 1 --2- 172.21.15.90:0/3322698130 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ad080e7d10 0x55ad080f8f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:40:14.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.144+0000 7fecb3400640 1 --2- 172.21.15.90:0/3322698130 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ad080e89f0 0x55ad080f9480 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7feca4000c00 tx=0x7feca402bb70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=38cfbb4db8e5cc1f server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:40:14.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.144+0000 7feca2000640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:40:14.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.145+0000 7feca2000640 1 -- 172.21.15.90:0/3322698130 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feca4039040 con 0x55ad080e89f0 2024-04-02T14:40:14.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.145+0000 7feca2000640 1 -- 172.21.15.90:0/3322698130 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7feca4034ce0 con 0x55ad080e89f0 2024-04-02T14:40:14.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.145+0000 7fecbdfd2740 1 -- 172.21.15.90:0/3322698130 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fecac025070 con 0x55ad080e89f0 2024-04-02T14:40:14.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.146+0000 7feca2000640 1 -- 172.21.15.90:0/3322698130 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7feca4005c90 con 0x55ad080e89f0 2024-04-02T14:40:14.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.146+0000 7fecbdfd2740 10 client.26902 fetch_fsmap learned FSMap version 27 2024-04-02T14:40:14.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.146+0000 7fecbdfd2740 1 -- 172.21.15.90:0/3322698130 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55ad081674f0 con 0x55ad080e89f0 2024-04-02T14:40:14.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.147+0000 7feca2000640 1 -- 172.21.15.90:0/3322698130 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fecac025070 con 0x55ad080e89f0 2024-04-02T14:40:14.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.148+0000 7feca2000640 1 -- 172.21.15.90:0/3322698130 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7feca404b4a0 con 0x55ad080e89f0 2024-04-02T14:40:14.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.148+0000 7fecbdfd2740 10 client.26902 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:40:14.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.148+0000 7fecbdfd2740 10 client.26902 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-04-02T14:40:14.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.148+0000 7fecbdfd2740 20 client.26902 populate_metadata read hostname 'smithi090' 2024-04-02T14:40:14.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.148+0000 7fecbdfd2740 1 --2- 172.21.15.90:0/3322698130 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x55ad08167860 0x55ad08169c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:14.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.148+0000 7fecbdfd2740 4 client.26902 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-02T14:40:14.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.148+0000 7fecbdfd2740 1 -- 172.21.15.90:0/3322698130 --> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55ad08059c30 con 0x55ad08167860 2024-04-02T14:40:14.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.149+0000 7fecb3e00640 1 --2- 172.21.15.90:0/3322698130 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x55ad08167860 0x55ad08169c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:14.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.150+0000 7feca2000640 10 client.26902 ms_handle_connect on v2:172.21.15.92:6836/1270635939 2024-04-02T14:40:14.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.150+0000 7fecb3e00640 1 --2- 172.21.15.90:0/3322698130 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x55ad08167860 0x55ad08169c50 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=8a32a8a35ef2c310 server_cookie=3424148f88f418c3 in_seq=0 out_seq=0 2024-04-02T14:40:14.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.151+0000 7feca2000640 1 -- 172.21.15.90:0/3322698130 <== mds.3 v2:172.21.15.92:6836/1270635939 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55ad08059c30 con 0x55ad08167860 2024-04-02T14:40:14.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.151+0000 7feca2000640 10 client.26902 handle_command_reply: tid=0 2024-04-02T14:40:14.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.151+0000 7fecbdfd2740 1 client.26902 shutdown 2024-04-02T14:40:14.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.151+0000 7fecbdfd2740 1 -- 172.21.15.90:0/3322698130 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ad080e89f0 msgr2=0x55ad080f9480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:14.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.151+0000 7fecbdfd2740 1 --2- 172.21.15.90:0/3322698130 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ad080e89f0 0x55ad080f9480 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7feca4000c00 tx=0x7feca402bb70 comp rx=0 tx=0).stop 2024-04-02T14:40:14.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.151+0000 7fecbdfd2740 1 -- 172.21.15.90:0/3322698130 shutdown_connections 2024-04-02T14:40:14.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.151+0000 7fecbdfd2740 1 --2- 172.21.15.90:0/3322698130 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x55ad08167860 0x55ad08169c50 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:14.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.151+0000 7fecbdfd2740 1 --2- 172.21.15.90:0/3322698130 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ad080f1080 0x55ad080f45f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:14.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.151+0000 7fecbdfd2740 1 --2- 172.21.15.90:0/3322698130 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ad080e89f0 0x55ad080f9480 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:14.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.151+0000 7fecbdfd2740 1 --2- 172.21.15.90:0/3322698130 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ad080e7d10 0x55ad080f8f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:14.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.151+0000 7fecbdfd2740 1 -- 172.21.15.90:0/3322698130 >> 172.21.15.90:0/3322698130 conn(0x55ad080aea50 msgr2=0x55ad080e1010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:40:14.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.151+0000 7feca2000640 0 client.26902 ms_handle_reset on v2:172.21.15.92:6836/1270635939 2024-04-02T14:40:14.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.152+0000 7fecbdfd2740 1 -- 172.21.15.90:0/3322698130 shutdown_connections 2024-04-02T14:40:14.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.152+0000 7fecbdfd2740 1 -- 172.21.15.90:0/3322698130 wait complete. 2024-04-02T14:40:14.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.152+0000 7fecbdfd2740 20 client.26902 trim_cache size 0 max 16384 2024-04-02T14:40:14.153 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-02T14:40:14.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.154+0000 7fecbb200640 1 -- 172.21.15.90:0/2542116354 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fec980a3df0 msgr2=0x7fec980a62b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:14.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.154+0000 7fecbb200640 1 --2- 172.21.15.90:0/2542116354 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fec980a3df0 0x7fec980a62b0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fecac005c40 tx=0x7fecac005bd0 comp rx=0 tx=0).stop 2024-04-02T14:40:14.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.154+0000 7fecbb200640 1 -- 172.21.15.90:0/2542116354 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fecb4068d00 msgr2=0x7fecb41a46b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:14.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.154+0000 7fecbb200640 1 --2- 172.21.15.90:0/2542116354 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fecb4068d00 0x7fecb41a46b0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fec9c00e9c0 tx=0x7fec9c00ee80 comp rx=0 tx=0).stop 2024-04-02T14:40:14.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.154+0000 7fecbb200640 1 -- 172.21.15.90:0/2542116354 shutdown_connections 2024-04-02T14:40:14.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.154+0000 7fecbb200640 1 --2- 172.21.15.90:0/2542116354 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fec980a3df0 0x7fec980a62b0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:14.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.154+0000 7fecbb200640 1 --2- 172.21.15.90:0/2542116354 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fecb4107b50 0x7fecb419e770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:14.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.154+0000 7fecbb200640 1 --2- 172.21.15.90:0/2542116354 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fecb40696c0 0x7fecb41a4bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:14.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.154+0000 7fecbb200640 1 --2- 172.21.15.90:0/2542116354 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fecb4068d00 0x7fecb41a46b0 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:14.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.154+0000 7fecbb200640 1 -- 172.21.15.90:0/2542116354 >> 172.21.15.90:0/2542116354 conn(0x7fecb4077b00 msgr2=0x7fecb40fe4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:40:14.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.157+0000 7fecbb200640 1 -- 172.21.15.90:0/2542116354 shutdown_connections 2024-04-02T14:40:14.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.157+0000 7fecbb200640 1 -- 172.21.15.90:0/2542116354 wait complete. 2024-04-02T14:40:14.182 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-02T14:40:14.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.462+0000 7f7bf1a00640 1 -- 172.21.15.90:0/3975701226 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7bec10e590 msgr2=0x7f7bec10e970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:14.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.462+0000 7f7bf1a00640 1 --2- 172.21.15.90:0/3975701226 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7bec10e590 0x7f7bec10e970 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f7bd8009930 tx=0x7f7bd802bc30 comp rx=0 tx=0).stop 2024-04-02T14:40:14.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.463+0000 7f7bf1a00640 1 -- 172.21.15.90:0/3975701226 shutdown_connections 2024-04-02T14:40:14.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.463+0000 7f7bf1a00640 1 --2- 172.21.15.90:0/3975701226 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7bec1094c0 0x7f7bec115350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:14.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.463+0000 7f7bf1a00640 1 --2- 172.21.15.90:0/3975701226 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7bec108b00 0x7f7bec108f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:14.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.463+0000 7f7bf1a00640 1 --2- 172.21.15.90:0/3975701226 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7bec10e590 0x7f7bec10e970 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:14.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.463+0000 7f7bf1a00640 1 -- 172.21.15.90:0/3975701226 >> 172.21.15.90:0/3975701226 conn(0x7f7bec06d040 msgr2=0x7f7bec06d450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:40:14.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.463+0000 7f7bf1a00640 1 -- 172.21.15.90:0/3975701226 shutdown_connections 2024-04-02T14:40:14.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.463+0000 7f7bf1a00640 1 -- 172.21.15.90:0/3975701226 wait complete. 2024-04-02T14:40:14.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.464+0000 7f7bf1a00640 1 Processor -- start 2024-04-02T14:40:14.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.464+0000 7f7bf1a00640 1 -- start start 2024-04-02T14:40:14.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.465+0000 7f7bf1a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7bec108b00 0x7f7bec114130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:14.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.465+0000 7f7bf1a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7bec1094c0 0x7f7bec114670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:14.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.465+0000 7f7bf1a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7bec10e590 0x7f7bec1abaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:14.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.465+0000 7f7bf1a00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f7bec117b70 con 0x7f7bec10e590 2024-04-02T14:40:14.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.465+0000 7f7bf1a00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f7bec1179f0 con 0x7f7bec108b00 2024-04-02T14:40:14.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.465+0000 7f7bf1a00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f7bec117cf0 con 0x7f7bec1094c0 2024-04-02T14:40:14.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.465+0000 7f7beaa00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7bec108b00 0x7f7bec114130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:14.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.465+0000 7f7bea000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7bec1094c0 0x7f7bec114670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:14.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.465+0000 7f7bea000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7bec1094c0 0x7f7bec114670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:53880/0 (socket says 172.21.15.90:53880) 2024-04-02T14:40:14.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.465+0000 7f7beb400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7bec10e590 0x7f7bec1abaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:14.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.465+0000 7f7bea000640 1 -- 172.21.15.90:0/1049773296 learned_addr learned my addr 172.21.15.90:0/1049773296 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:40:14.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.466+0000 7f7beaa00640 1 -- 172.21.15.90:0/1049773296 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7bec1094c0 msgr2=0x7f7bec114670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:14.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.466+0000 7f7beaa00640 1 --2- 172.21.15.90:0/1049773296 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7bec1094c0 0x7f7bec114670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:14.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.466+0000 7f7beaa00640 1 -- 172.21.15.90:0/1049773296 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7bec10e590 msgr2=0x7f7bec1abaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:14.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.466+0000 7f7beaa00640 1 --2- 172.21.15.90:0/1049773296 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7bec10e590 0x7f7bec1abaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:14.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.466+0000 7f7beaa00640 1 -- 172.21.15.90:0/1049773296 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7bec1ac1d0 con 0x7f7bec108b00 2024-04-02T14:40:14.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.466+0000 7f7beaa00640 1 --2- 172.21.15.90:0/1049773296 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7bec108b00 0x7f7bec114130 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f7bd8036040 tx=0x7f7bd8005e80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:40:14.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.467+0000 7f7be3400640 1 -- 172.21.15.90:0/1049773296 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bd8043020 con 0x7f7bec108b00 2024-04-02T14:40:14.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.467+0000 7f7be3400640 1 -- 172.21.15.90:0/1049773296 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7bd8005000 con 0x7f7bec108b00 2024-04-02T14:40:14.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.467+0000 7f7be3400640 1 -- 172.21.15.90:0/1049773296 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bd8003af0 con 0x7f7bec108b00 2024-04-02T14:40:14.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.467+0000 7f7bf1a00640 1 -- 172.21.15.90:0/1049773296 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7bec1ac460 con 0x7f7bec108b00 2024-04-02T14:40:14.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.467+0000 7f7bf1a00640 1 -- 172.21.15.90:0/1049773296 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7bec1b3d90 con 0x7f7bec108b00 2024-04-02T14:40:14.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.472+0000 7f7be3400640 1 -- 172.21.15.90:0/1049773296 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f7bd8046020 con 0x7f7bec108b00 2024-04-02T14:40:14.473 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.473+0000 7f7bf4794740 1 Processor -- start 2024-04-02T14:40:14.473 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.473+0000 7f7bf4794740 1 -- start start 2024-04-02T14:40:14.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.474+0000 7f7bf4794740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5584c2d219f0 0x5584c2d20740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:14.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.474+0000 7f7bf4794740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5584c2d20d10 0x5584c2d21190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:14.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.474+0000 7f7bf4794740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5584c2d2a080 0x5584c2d2c470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:14.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.474+0000 7f7bf4794740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5584c2c61400 con 0x5584c2d219f0 2024-04-02T14:40:14.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.474+0000 7f7bf4794740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x5584c2c45f10 con 0x5584c2d20d10 2024-04-02T14:40:14.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.474+0000 7f7bf4794740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5584c2c856f0 con 0x5584c2d2a080 2024-04-02T14:40:14.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.475+0000 7f7be3400640 1 --2- 172.21.15.90:0/1049773296 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f7bd00a3ed0 0x7f7bd00a6390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:14.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.475+0000 7f7bea000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5584c2d2a080 0x5584c2d2c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:14.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.475+0000 7f7be3400640 1 -- 172.21.15.90:0/1049773296 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f7bd80f26a0 con 0x7f7bec108b00 2024-04-02T14:40:14.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.475+0000 7f7beb400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5584c2d219f0 0x5584c2d20740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:14.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.475+0000 7f7beb400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5584c2d219f0 0x5584c2d20740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:52900/0 (socket says 172.21.15.90:52900) 2024-04-02T14:40:14.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.475+0000 7f7beb400640 1 -- 172.21.15.90:0/3805510004 learned_addr learned my addr 172.21.15.90:0/3805510004 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:40:14.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.476+0000 7f7beaa00640 1 --2- 172.21.15.90:0/3805510004 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5584c2d20d10 0x5584c2d21190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:14.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.476+0000 7f7beb400640 1 --2- 172.21.15.90:0/1049773296 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f7bd00a3ed0 0x7f7bd00a6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:14.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.476+0000 7f7beaa00640 1 -- 172.21.15.90:0/3805510004 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5584c2d2a080 msgr2=0x5584c2d2c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:40:14.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.476+0000 7f7beaa00640 1 --2- 172.21.15.90:0/3805510004 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5584c2d2a080 0x5584c2d2c470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:14.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.476+0000 7f7beaa00640 1 -- 172.21.15.90:0/3805510004 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5584c2d219f0 msgr2=0x5584c2d20740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:40:14.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.476+0000 7f7beaa00640 1 --2- 172.21.15.90:0/3805510004 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5584c2d219f0 0x5584c2d20740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:14.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.476+0000 7f7beaa00640 1 -- 172.21.15.90:0/3805510004 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5584c29993f0 con 0x5584c2d20d10 2024-04-02T14:40:14.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.477+0000 7f7beaa00640 1 --2- 172.21.15.90:0/3805510004 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5584c2d20d10 0x5584c2d21190 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f7bd80efeb0 tx=0x7f7bd80be590 comp rx=0 tx=0).ready entity=mon.2 client_cookie=416afc49a807d372 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:40:14.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.477+0000 7f7bd7e00640 1 -- 172.21.15.90:0/3805510004 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bd80be700 con 0x5584c2d20d10 2024-04-02T14:40:14.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.477+0000 7f7bd7e00640 1 -- 172.21.15.90:0/3805510004 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7bd80c2ab0 con 0x5584c2d20d10 2024-04-02T14:40:14.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.477+0000 7f7bd7e00640 1 -- 172.21.15.90:0/3805510004 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bd8003c90 con 0x5584c2d20d10 2024-04-02T14:40:14.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.478+0000 7f7bf4794740 1 -- 172.21.15.90:0/3805510004 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5584c2d20d10 msgr2=0x5584c2d21190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:14.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.478+0000 7f7bf4794740 1 --2- 172.21.15.90:0/3805510004 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5584c2d20d10 0x5584c2d21190 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f7bd80efeb0 tx=0x7f7bd80be590 comp rx=0 tx=0).stop 2024-04-02T14:40:14.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.478+0000 7f7bf4794740 1 -- 172.21.15.90:0/3805510004 shutdown_connections 2024-04-02T14:40:14.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.478+0000 7f7beb400640 1 --2- 172.21.15.90:0/1049773296 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f7bd00a3ed0 0x7f7bd00a6390 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f7be4009e20 tx=0x7f7be4004ab0 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:40:14.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.479+0000 7f7bf4794740 1 --2- 172.21.15.90:0/3805510004 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5584c2d2a080 0x5584c2d2c470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:14.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.479+0000 7f7bf4794740 1 --2- 172.21.15.90:0/3805510004 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5584c2d20d10 0x5584c2d21190 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:14.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.479+0000 7f7bf4794740 1 --2- 172.21.15.90:0/3805510004 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5584c2d219f0 0x5584c2d20740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:14.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.479+0000 7f7bf4794740 1 -- 172.21.15.90:0/3805510004 >> 172.21.15.90:0/3805510004 conn(0x5584c2ce7a50 msgr2=0x5584c2d1b760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:40:14.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.479+0000 7f7bf4794740 1 -- 172.21.15.90:0/3805510004 shutdown_connections 2024-04-02T14:40:14.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.479+0000 7f7bf4794740 1 -- 172.21.15.90:0/3805510004 wait complete. 2024-04-02T14:40:14.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.480+0000 7f7bf4794740 1 -- start start 2024-04-02T14:40:14.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.481+0000 7f7bf4794740 1 Processor -- start 2024-04-02T14:40:14.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.481+0000 7f7bf4794740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5584c2d20d10 0x5584c2d26f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:14.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.481+0000 7f7bf4794740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5584c2d219f0 0x5584c2d274c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:14.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.481+0000 7f7bf4794740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5584c2d2a080 0x5584c2d27a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:14.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.482+0000 7f7beaa00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5584c2d20d10 0x5584c2d26f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:14.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.482+0000 7f7beaa00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5584c2d20d10 0x5584c2d26f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:53886/0 (socket says 172.21.15.90:53886) 2024-04-02T14:40:14.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.482+0000 7f7beaa00640 1 -- 172.21.15.90:0/2957479363 learned_addr learned my addr 172.21.15.90:0/2957479363 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:40:14.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.482+0000 7f7bf4794740 1 -- 172.21.15.90:0/2957479363 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5584c2c816a0 con 0x5584c2d219f0 2024-04-02T14:40:14.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.482+0000 7f7beb400640 1 --2- 172.21.15.90:0/2957479363 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5584c2d2a080 0x5584c2d27a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:14.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.483+0000 7f7bf4794740 1 -- 172.21.15.90:0/2957479363 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x5584c2c2d980 con 0x5584c2d2a080 2024-04-02T14:40:14.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.483+0000 7f7bea000640 1 --2- 172.21.15.90:0/2957479363 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5584c2d219f0 0x5584c2d274c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:14.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.483+0000 7f7bf4794740 1 -- 172.21.15.90:0/2957479363 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5584c2c40cb0 con 0x5584c2d20d10 2024-04-02T14:40:14.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.483+0000 7f7bf4794740 10 client.0 _ll_register_callbacks cb 0x5584c2c406a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:40:14.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.484+0000 7f7beaa00640 1 -- 172.21.15.90:0/2957479363 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5584c2d2a080 msgr2=0x5584c2d27a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:40:14.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.484+0000 7f7beaa00640 1 --2- 172.21.15.90:0/2957479363 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5584c2d2a080 0x5584c2d27a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:14.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.484+0000 7f7beaa00640 1 -- 172.21.15.90:0/2957479363 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5584c2d219f0 msgr2=0x5584c2d274c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:40:14.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.484+0000 7f7beaa00640 1 --2- 172.21.15.90:0/2957479363 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5584c2d219f0 0x5584c2d274c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:14.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.484+0000 7f7beaa00640 1 -- 172.21.15.90:0/2957479363 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5584c2a637d0 con 0x5584c2d20d10 2024-04-02T14:40:14.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.484+0000 7f7bea000640 1 --2- 172.21.15.90:0/2957479363 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5584c2d219f0 0x5584c2d274c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:40:14.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.484+0000 7f7beaa00640 1 --2- 172.21.15.90:0/2957479363 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5584c2d20d10 0x5584c2d26f80 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f7bd8006010 tx=0x7f7bd80beec0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f225a897442d96c server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:40:14.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.484+0000 7f7bd4c00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:40:14.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.487+0000 7f7bd4c00640 1 -- 172.21.15.90:0/2957479363 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bd80be700 con 0x5584c2d20d10 2024-04-02T14:40:14.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.487+0000 7f7bd4c00640 1 -- 172.21.15.90:0/2957479363 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f7bd80051b0 con 0x5584c2d20d10 2024-04-02T14:40:14.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.487+0000 7f7bf4794740 1 -- 172.21.15.90:0/2957479363 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7bd80c2ab0 con 0x5584c2d20d10 2024-04-02T14:40:14.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.489+0000 7f7bd4c00640 1 -- 172.21.15.90:0/2957479363 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f7bd80c6ee0 con 0x5584c2d20d10 2024-04-02T14:40:14.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.489+0000 7f7bf4794740 10 client.26914 fetch_fsmap learned FSMap version 27 2024-04-02T14:40:14.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.489+0000 7f7bf4794740 1 -- 172.21.15.90:0/2957479363 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5584c2d38b90 con 0x5584c2d20d10 2024-04-02T14:40:14.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.490+0000 7f7bd4c00640 1 -- 172.21.15.90:0/2957479363 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7bd80c2ab0 con 0x5584c2d20d10 2024-04-02T14:40:14.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.491+0000 7f7bd4c00640 1 -- 172.21.15.90:0/2957479363 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f7bd80d7070 con 0x5584c2d20d10 2024-04-02T14:40:14.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.491+0000 7f7bf4794740 10 client.26914 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:40:14.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.491+0000 7f7bf4794740 10 client.26914 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-04-02T14:40:14.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.491+0000 7f7bf4794740 20 client.26914 populate_metadata read hostname 'smithi090' 2024-04-02T14:40:14.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.491+0000 7f7bf4794740 1 --2- 172.21.15.90:0/2957479363 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x5584c2d39d10 0x5584c2d3c100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:14.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.491+0000 7f7bf4794740 4 client.26914 mds_command: new command op to 24547 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:40:14.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.491+0000 7f7bf4794740 1 -- 172.21.15.90:0/2957479363 --> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5584c2c45040 con 0x5584c2d39d10 2024-04-02T14:40:14.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.491+0000 7f7bea000640 1 --2- 172.21.15.90:0/2957479363 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x5584c2d39d10 0x5584c2d3c100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:14.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.492+0000 7f7bea000640 1 --2- 172.21.15.90:0/2957479363 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x5584c2d39d10 0x5584c2d3c100 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=296a30724f7d04bb server_cookie=a4d7eb8823f36397 in_seq=0 out_seq=0 2024-04-02T14:40:14.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.492+0000 7f7bd4c00640 10 client.26914 ms_handle_connect on v2:172.21.15.177:6838/664895663 2024-04-02T14:40:14.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.493+0000 7f7bd4c00640 1 -- 172.21.15.90:0/2957479363 <== mds.2 v2:172.21.15.177:6838/664895663 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5584c2c45040 con 0x5584c2d39d10 2024-04-02T14:40:14.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.493+0000 7f7bd4c00640 10 client.26914 handle_command_reply: tid=0 2024-04-02T14:40:14.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.493+0000 7f7bf4794740 1 client.26914 shutdown 2024-04-02T14:40:14.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.494+0000 7f7bf4794740 1 -- 172.21.15.90:0/2957479363 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5584c2d20d10 msgr2=0x5584c2d26f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:14.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.494+0000 7f7bf4794740 1 --2- 172.21.15.90:0/2957479363 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5584c2d20d10 0x5584c2d26f80 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f7bd8006010 tx=0x7f7bd80beec0 comp rx=0 tx=0).stop 2024-04-02T14:40:14.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.494+0000 7f7bf4794740 1 -- 172.21.15.90:0/2957479363 shutdown_connections 2024-04-02T14:40:14.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.494+0000 7f7bf4794740 1 --2- 172.21.15.90:0/2957479363 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x5584c2d39d10 0x5584c2d3c100 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:14.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.494+0000 7f7bf4794740 1 --2- 172.21.15.90:0/2957479363 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5584c2d2a080 0x5584c2d27a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:14.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.494+0000 7f7bf4794740 1 --2- 172.21.15.90:0/2957479363 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5584c2d219f0 0x5584c2d274c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:14.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.494+0000 7f7bf4794740 1 --2- 172.21.15.90:0/2957479363 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5584c2d20d10 0x5584c2d26f80 secure :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f7bd8006010 tx=0x7f7bd80beec0 comp rx=0 tx=0).stop 2024-04-02T14:40:14.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.494+0000 7f7bf4794740 1 -- 172.21.15.90:0/2957479363 >> 172.21.15.90:0/2957479363 conn(0x5584c2ce7a50 msgr2=0x5584c2d1d4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:40:14.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.494+0000 7f7bd4c00640 0 client.26914 ms_handle_reset on v2:172.21.15.177:6838/664895663 2024-04-02T14:40:14.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.495+0000 7f7bf4794740 1 -- 172.21.15.90:0/2957479363 shutdown_connections 2024-04-02T14:40:14.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.495+0000 7f7bf4794740 1 -- 172.21.15.90:0/2957479363 wait complete. 2024-04-02T14:40:14.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.495+0000 7f7bf4794740 20 client.26914 trim_cache size 0 max 16384 2024-04-02T14:40:14.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.521+0000 7f7bf4794740 1 Processor -- start 2024-04-02T14:40:14.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.522+0000 7f7bf4794740 1 -- start start 2024-04-02T14:40:14.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.522+0000 7f7bf4794740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5584c2d219f0 0x5584c2d296b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:14.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.522+0000 7f7bf4794740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5584c2d2a080 0x5584c2d35870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:14.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.522+0000 7f7bf4794740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5584c2d1b6f0 0x5584c2d1d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:14.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.523+0000 7f7bf4794740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f7bec1bc590 con 0x5584c2d219f0 2024-04-02T14:40:14.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.523+0000 7f7bf4794740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f7bec1be0d0 con 0x5584c2d1b6f0 2024-04-02T14:40:14.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.523+0000 7f7bf4794740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f7bec1bcc60 con 0x5584c2d2a080 2024-04-02T14:40:14.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.524+0000 7f7beb400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5584c2d1b6f0 0x5584c2d1d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:14.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.524+0000 7f7beb400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5584c2d1b6f0 0x5584c2d1d2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:39236/0 (socket says 172.21.15.90:39236) 2024-04-02T14:40:14.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.524+0000 7f7beb400640 1 -- 172.21.15.90:0/4160756943 learned_addr learned my addr 172.21.15.90:0/4160756943 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:40:14.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.525+0000 7f7bea000640 1 --2- 172.21.15.90:0/4160756943 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5584c2d2a080 0x5584c2d35870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:14.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.525+0000 7f7beb400640 1 -- 172.21.15.90:0/4160756943 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5584c2d2a080 msgr2=0x5584c2d35870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:40:14.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.525+0000 7f7beb400640 1 --2- 172.21.15.90:0/4160756943 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5584c2d2a080 0x5584c2d35870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:14.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.525+0000 7f7beb400640 1 -- 172.21.15.90:0/4160756943 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5584c2d219f0 msgr2=0x5584c2d296b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:40:14.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.525+0000 7f7beb400640 1 --2- 172.21.15.90:0/4160756943 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5584c2d219f0 0x5584c2d296b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:14.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.525+0000 7f7beb400640 1 -- 172.21.15.90:0/4160756943 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5584c2d17020 con 0x5584c2d1b6f0 2024-04-02T14:40:14.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.526+0000 7f7beb400640 1 --2- 172.21.15.90:0/4160756943 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5584c2d1b6f0 0x5584c2d1d2c0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f7be40093d0 tx=0x7f7be4009040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=764963fb3f625ac7 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:40:14.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.527+0000 7f7bd7400640 1 -- 172.21.15.90:0/4160756943 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7be401be60 con 0x5584c2d1b6f0 2024-04-02T14:40:14.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.527+0000 7f7bd7400640 1 -- 172.21.15.90:0/4160756943 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7be401b0d0 con 0x5584c2d1b6f0 2024-04-02T14:40:14.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.527+0000 7f7bd7400640 1 -- 172.21.15.90:0/4160756943 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7be401be60 con 0x5584c2d1b6f0 2024-04-02T14:40:14.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.528+0000 7f7bf4794740 1 -- 172.21.15.90:0/4160756943 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5584c2d1b6f0 msgr2=0x5584c2d1d2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:14.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.528+0000 7f7bf4794740 1 --2- 172.21.15.90:0/4160756943 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5584c2d1b6f0 0x5584c2d1d2c0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f7be40093d0 tx=0x7f7be4009040 comp rx=0 tx=0).stop 2024-04-02T14:40:14.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.531+0000 7f7bf4794740 1 -- 172.21.15.90:0/4160756943 shutdown_connections 2024-04-02T14:40:14.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.531+0000 7f7bf4794740 1 --2- 172.21.15.90:0/4160756943 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5584c2d1b6f0 0x5584c2d1d2c0 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:14.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.531+0000 7f7bf4794740 1 --2- 172.21.15.90:0/4160756943 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5584c2d2a080 0x5584c2d35870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:14.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.531+0000 7f7bf4794740 1 --2- 172.21.15.90:0/4160756943 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5584c2d219f0 0x5584c2d296b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:14.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.531+0000 7f7bf4794740 1 -- 172.21.15.90:0/4160756943 >> 172.21.15.90:0/4160756943 conn(0x5584c2ce7a50 msgr2=0x5584c2d1cbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:40:14.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.531+0000 7f7bf4794740 1 -- 172.21.15.90:0/4160756943 shutdown_connections 2024-04-02T14:40:14.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.532+0000 7f7bf4794740 1 -- 172.21.15.90:0/4160756943 wait complete. 2024-04-02T14:40:14.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.532+0000 7f7bf4794740 1 -- start start 2024-04-02T14:40:14.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.532+0000 7f7bf4794740 1 Processor -- start 2024-04-02T14:40:14.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.534+0000 7f7bf4794740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5584c2d1b6f0 0x5584c2d23fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:14.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.534+0000 7f7bf4794740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5584c2d219f0 0x5584c2d244f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:14.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.534+0000 7f7bf4794740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5584c2d2a080 0x5584c2d9f150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:14.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.534+0000 7f7beb400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5584c2d2a080 0x5584c2d9f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:14.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.535+0000 7f7beb400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5584c2d2a080 0x5584c2d9f150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:39240/0 (socket says 172.21.15.90:39240) 2024-04-02T14:40:14.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.535+0000 7f7beb400640 1 -- 172.21.15.90:0/2628349335 learned_addr learned my addr 172.21.15.90:0/2628349335 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:40:14.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.535+0000 7f7beaa00640 1 --2- 172.21.15.90:0/2628349335 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5584c2d1b6f0 0x5584c2d23fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:14.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.535+0000 7f7bf4794740 1 -- 172.21.15.90:0/2628349335 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f7bec1bda00 con 0x5584c2d1b6f0 2024-04-02T14:40:14.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.535+0000 7f7bea000640 1 --2- 172.21.15.90:0/2628349335 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5584c2d219f0 0x5584c2d244f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:14.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.535+0000 7f7bf4794740 1 -- 172.21.15.90:0/2628349335 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f7bec1bd330 con 0x5584c2d2a080 2024-04-02T14:40:14.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.535+0000 7f7bf4794740 1 -- 172.21.15.90:0/2628349335 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f7bec1ba620 con 0x5584c2d219f0 2024-04-02T14:40:14.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.535+0000 7f7bf4794740 10 client.0 _ll_register_callbacks cb 0x7f7bd80ba9a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:40:14.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.536+0000 7f7beb400640 1 -- 172.21.15.90:0/2628349335 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5584c2d219f0 msgr2=0x5584c2d244f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:40:14.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.536+0000 7f7beb400640 1 --2- 172.21.15.90:0/2628349335 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5584c2d219f0 0x5584c2d244f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:14.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.536+0000 7f7beb400640 1 -- 172.21.15.90:0/2628349335 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5584c2d1b6f0 msgr2=0x5584c2d23fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:40:14.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.536+0000 7f7beb400640 1 --2- 172.21.15.90:0/2628349335 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5584c2d1b6f0 0x5584c2d23fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:14.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.536+0000 7f7beb400640 1 -- 172.21.15.90:0/2628349335 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5584c2d24d80 con 0x5584c2d2a080 2024-04-02T14:40:14.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.536+0000 7f7beaa00640 1 --2- 172.21.15.90:0/2628349335 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5584c2d1b6f0 0x5584c2d23fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:40:14.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.537+0000 7f7beb400640 1 --2- 172.21.15.90:0/2628349335 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5584c2d2a080 0x5584c2d9f150 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f7be40096b0 tx=0x7f7be4009400 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a256cde95878740f server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:40:14.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.537+0000 7f7bd4c00640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:40:14.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.538+0000 7f7bd4c00640 1 -- 172.21.15.90:0/2628349335 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7be401be60 con 0x5584c2d2a080 2024-04-02T14:40:14.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.538+0000 7f7bf4794740 1 -- 172.21.15.90:0/2628349335 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7be401b0d0 con 0x5584c2d2a080 2024-04-02T14:40:14.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.538+0000 7f7bd4c00640 1 -- 172.21.15.90:0/2628349335 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f7be402c040 con 0x5584c2d2a080 2024-04-02T14:40:14.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.539+0000 7f7bd4c00640 1 -- 172.21.15.90:0/2628349335 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f7be403d050 con 0x5584c2d2a080 2024-04-02T14:40:14.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.539+0000 7f7bf4794740 10 client.27035 fetch_fsmap learned FSMap version 27 2024-04-02T14:40:14.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.540+0000 7f7bf4794740 1 -- 172.21.15.90:0/2628349335 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5584c2da0760 con 0x5584c2d2a080 2024-04-02T14:40:14.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.540+0000 7f7bd4c00640 1 -- 172.21.15.90:0/2628349335 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7be401b0d0 con 0x5584c2d2a080 2024-04-02T14:40:14.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.540+0000 7f7bd4c00640 1 -- 172.21.15.90:0/2628349335 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f7be4026690 con 0x5584c2d2a080 2024-04-02T14:40:14.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.541+0000 7f7bf4794740 10 client.27035 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:40:14.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.541+0000 7f7bf4794740 10 client.27035 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-04-02T14:40:14.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.541+0000 7f7bf4794740 20 client.27035 populate_metadata read hostname 'smithi090' 2024-04-02T14:40:14.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.541+0000 7f7bf4794740 1 --2- 172.21.15.90:0/2628349335 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x5584c2da0d80 0x5584c2da3170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:14.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.541+0000 7f7bf4794740 4 client.27035 mds_command: new command op to 24547 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-02T14:40:14.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.541+0000 7f7bf4794740 1 -- 172.21.15.90:0/2628349335 --> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5584c2c92c30 con 0x5584c2da0d80 2024-04-02T14:40:14.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.541+0000 7f7beaa00640 1 --2- 172.21.15.90:0/2628349335 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x5584c2da0d80 0x5584c2da3170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:14.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.544+0000 7f7beaa00640 1 --2- 172.21.15.90:0/2628349335 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x5584c2da0d80 0x5584c2da3170 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=e7c9317ef481d73 server_cookie=6aeb359f446f92e0 in_seq=0 out_seq=0 2024-04-02T14:40:14.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.544+0000 7f7bd4c00640 10 client.27035 ms_handle_connect on v2:172.21.15.177:6838/664895663 2024-04-02T14:40:14.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.544+0000 7f7bd4c00640 1 -- 172.21.15.90:0/2628349335 <== mds.2 v2:172.21.15.177:6838/664895663 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5584c2c92c30 con 0x5584c2da0d80 2024-04-02T14:40:14.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.546+0000 7f7bd4c00640 10 client.27035 handle_command_reply: tid=0 2024-04-02T14:40:14.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.546+0000 7f7bf4794740 1 client.27035 shutdown 2024-04-02T14:40:14.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.548+0000 7f7bf4794740 1 -- 172.21.15.90:0/2628349335 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5584c2d2a080 msgr2=0x5584c2d9f150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:14.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.548+0000 7f7bf4794740 1 --2- 172.21.15.90:0/2628349335 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5584c2d2a080 0x5584c2d9f150 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f7be40096b0 tx=0x7f7be4009400 comp rx=0 tx=0).stop 2024-04-02T14:40:14.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.548+0000 7f7bf4794740 1 -- 172.21.15.90:0/2628349335 shutdown_connections 2024-04-02T14:40:14.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.548+0000 7f7bf4794740 1 --2- 172.21.15.90:0/2628349335 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x5584c2da0d80 0x5584c2da3170 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:14.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.548+0000 7f7bf4794740 1 --2- 172.21.15.90:0/2628349335 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5584c2d2a080 0x5584c2d9f150 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:14.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.548+0000 7f7bf4794740 1 --2- 172.21.15.90:0/2628349335 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5584c2d219f0 0x5584c2d244f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:14.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.548+0000 7f7bf4794740 1 --2- 172.21.15.90:0/2628349335 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5584c2d1b6f0 0x5584c2d23fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:14.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.548+0000 7f7bd4c00640 0 client.27035 ms_handle_reset on v2:172.21.15.177:6838/664895663 2024-04-02T14:40:14.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.548+0000 7f7bf4794740 1 -- 172.21.15.90:0/2628349335 >> 172.21.15.90:0/2628349335 conn(0x5584c2ce7a50 msgr2=0x5584c2d1a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:40:14.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.549+0000 7f7bf4794740 1 -- 172.21.15.90:0/2628349335 shutdown_connections 2024-04-02T14:40:14.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.549+0000 7f7bf4794740 1 -- 172.21.15.90:0/2628349335 wait complete. 2024-04-02T14:40:14.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.549+0000 7f7bf4794740 20 client.27035 trim_cache size 0 max 16384 2024-04-02T14:40:14.550 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-02T14:40:14.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.550+0000 7f7bf1a00640 1 -- 172.21.15.90:0/1049773296 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f7bd00a3ed0 msgr2=0x7f7bd00a6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:14.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.550+0000 7f7bf1a00640 1 --2- 172.21.15.90:0/1049773296 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f7bd00a3ed0 0x7f7bd00a6390 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f7be4009e20 tx=0x7f7be4004ab0 comp rx=0 tx=0).stop 2024-04-02T14:40:14.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.551+0000 7f7bf1a00640 1 -- 172.21.15.90:0/1049773296 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7bec108b00 msgr2=0x7f7bec114130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:14.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.551+0000 7f7bf1a00640 1 --2- 172.21.15.90:0/1049773296 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7bec108b00 0x7f7bec114130 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f7bd8036040 tx=0x7f7bd8005e80 comp rx=0 tx=0).stop 2024-04-02T14:40:14.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.551+0000 7f7bf1a00640 1 -- 172.21.15.90:0/1049773296 shutdown_connections 2024-04-02T14:40:14.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.551+0000 7f7bf1a00640 1 --2- 172.21.15.90:0/1049773296 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f7bd00a3ed0 0x7f7bd00a6390 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:14.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.551+0000 7f7bf1a00640 1 --2- 172.21.15.90:0/1049773296 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7bec10e590 0x7f7bec1abaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:14.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.551+0000 7f7bf1a00640 1 --2- 172.21.15.90:0/1049773296 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7bec1094c0 0x7f7bec114670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:14.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.551+0000 7f7bf1a00640 1 --2- 172.21.15.90:0/1049773296 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7bec108b00 0x7f7bec114130 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:14.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.551+0000 7f7bf1a00640 1 -- 172.21.15.90:0/1049773296 >> 172.21.15.90:0/1049773296 conn(0x7f7bec06d040 msgr2=0x7f7bec0705c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:40:14.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.552+0000 7f7bf1a00640 1 -- 172.21.15.90:0/1049773296 shutdown_connections 2024-04-02T14:40:14.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.552+0000 7f7bf1a00640 1 -- 172.21.15.90:0/1049773296 wait complete. 2024-04-02T14:40:14.584 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-02T14:40:14.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.864+0000 7fa6d9e00640 1 -- 172.21.15.90:0/878089213 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa6d410d3e0 msgr2=0x7fa6d4114fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:14.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.864+0000 7fa6d9e00640 1 --2- 172.21.15.90:0/878089213 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa6d410d3e0 0x7fa6d4114fb0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fa6cc009be0 tx=0x7fa6cc02bef0 comp rx=0 tx=0).stop 2024-04-02T14:40:14.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.864+0000 7fa6d9e00640 1 -- 172.21.15.90:0/878089213 shutdown_connections 2024-04-02T14:40:14.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.864+0000 7fa6d9e00640 1 --2- 172.21.15.90:0/878089213 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa6d410d3e0 0x7fa6d4114fb0 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:14.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.864+0000 7fa6d9e00640 1 --2- 172.21.15.90:0/878089213 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa6d4108cc0 0x7fa6d410cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:14.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.864+0000 7fa6d9e00640 1 --2- 172.21.15.90:0/878089213 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa6d4108310 0x7fa6d41086f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:14.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.864+0000 7fa6d9e00640 1 -- 172.21.15.90:0/878089213 >> 172.21.15.90:0/878089213 conn(0x7fa6d4076040 msgr2=0x7fa6d4078460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:40:14.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.864+0000 7fa6d9e00640 1 -- 172.21.15.90:0/878089213 shutdown_connections 2024-04-02T14:40:14.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.865+0000 7fa6d9e00640 1 -- 172.21.15.90:0/878089213 wait complete. 2024-04-02T14:40:14.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.867+0000 7fa6d9e00640 1 Processor -- start 2024-04-02T14:40:14.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.867+0000 7fa6d9e00640 1 -- start start 2024-04-02T14:40:14.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.868+0000 7fa6d9e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa6d4108310 0x7fa6d41a4630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:14.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.868+0000 7fa6d9e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa6d410d3e0 0x7fa6d41a4b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:14.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.868+0000 7fa6d9e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa6d419e7f0 0x7fa6d419eca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:14.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.868+0000 7fa6d2000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa6d410d3e0 0x7fa6d41a4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:14.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.868+0000 7fa6d2a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa6d4108310 0x7fa6d41a4630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:14.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.868+0000 7fa6d9e00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fa6d41184f0 con 0x7fa6d419e7f0 2024-04-02T14:40:14.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.868+0000 7fa6d9e00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fa6d4118370 con 0x7fa6d410d3e0 2024-04-02T14:40:14.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.868+0000 7fa6d9e00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fa6d4118670 con 0x7fa6d4108310 2024-04-02T14:40:14.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.868+0000 7fa6d2000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa6d410d3e0 0x7fa6d41a4b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:39254/0 (socket says 172.21.15.90:39254) 2024-04-02T14:40:14.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.868+0000 7fa6d2a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa6d4108310 0x7fa6d41a4630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:53930/0 (socket says 172.21.15.90:53930) 2024-04-02T14:40:14.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.868+0000 7fa6d2000640 1 -- 172.21.15.90:0/894492857 learned_addr learned my addr 172.21.15.90:0/894492857 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:40:14.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.869+0000 7fa6d3400640 1 --2- 172.21.15.90:0/894492857 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa6d419e7f0 0x7fa6d419eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:14.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.869+0000 7fa6d2a00640 1 -- 172.21.15.90:0/894492857 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa6d410d3e0 msgr2=0x7fa6d41a4b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:14.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.869+0000 7fa6d2a00640 1 --2- 172.21.15.90:0/894492857 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa6d410d3e0 0x7fa6d41a4b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:14.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.869+0000 7fa6d2a00640 1 -- 172.21.15.90:0/894492857 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa6d419e7f0 msgr2=0x7fa6d419eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:14.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.869+0000 7fa6d2a00640 1 --2- 172.21.15.90:0/894492857 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa6d419e7f0 0x7fa6d419eca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:14.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.869+0000 7fa6d2a00640 1 -- 172.21.15.90:0/894492857 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6d419f380 con 0x7fa6d4108310 2024-04-02T14:40:14.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.869+0000 7fa6d2a00640 1 --2- 172.21.15.90:0/894492857 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa6d4108310 0x7fa6d41a4630 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fa6c400e9f0 tx=0x7fa6c400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:40:14.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.870+0000 7fa6d8a00640 1 -- 172.21.15.90:0/894492857 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6c400ccc0 con 0x7fa6d4108310 2024-04-02T14:40:14.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.870+0000 7fa6d8a00640 1 -- 172.21.15.90:0/894492857 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa6c4010910 con 0x7fa6d4108310 2024-04-02T14:40:14.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.870+0000 7fa6d8a00640 1 -- 172.21.15.90:0/894492857 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6c4018960 con 0x7fa6d4108310 2024-04-02T14:40:14.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.870+0000 7fa6d9e00640 1 -- 172.21.15.90:0/894492857 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6d419f670 con 0x7fa6d4108310 2024-04-02T14:40:14.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.870+0000 7fa6d9e00640 1 -- 172.21.15.90:0/894492857 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6d41ab7b0 con 0x7fa6d4108310 2024-04-02T14:40:14.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.875+0000 7fa6dcb64740 1 Processor -- start 2024-04-02T14:40:14.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.875+0000 7fa6dcb64740 1 -- start start 2024-04-02T14:40:14.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.876+0000 7fa6dcb64740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557d7f0969f0 0x557d7f095740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:14.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.876+0000 7fa6dcb64740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557d7f095d10 0x557d7f096190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:14.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.876+0000 7fa6dcb64740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557d7f09f080 0x557d7f0a1470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:14.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.876+0000 7fa6dcb64740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x557d7efd6400 con 0x557d7f09f080 2024-04-02T14:40:14.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.876+0000 7fa6dcb64740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x557d7efbaf10 con 0x557d7f0969f0 2024-04-02T14:40:14.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.876+0000 7fa6dcb64740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x557d7effa6f0 con 0x557d7f095d10 2024-04-02T14:40:14.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.876+0000 7fa6d8a00640 1 -- 172.21.15.90:0/894492857 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fa6c4018b00 con 0x7fa6d4108310 2024-04-02T14:40:14.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.877+0000 7fa6d2a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557d7f095d10 0x557d7f096190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:14.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.877+0000 7fa6d2a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557d7f095d10 0x557d7f096190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:53936/0 (socket says 172.21.15.90:53936) 2024-04-02T14:40:14.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.877+0000 7fa6d2a00640 1 -- 172.21.15.90:0/2327902241 learned_addr learned my addr 172.21.15.90:0/2327902241 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:40:14.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.877+0000 7fa6d2000640 1 --2- 172.21.15.90:0/2327902241 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557d7f09f080 0x557d7f0a1470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:14.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.877+0000 7fa6d2a00640 1 -- 172.21.15.90:0/2327902241 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557d7f0969f0 msgr2=0x557d7f095740 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-02T14:40:14.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.877+0000 7fa6d2a00640 1 --2- 172.21.15.90:0/2327902241 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557d7f0969f0 0x557d7f095740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:14.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.877+0000 7fa6d2a00640 1 -- 172.21.15.90:0/2327902241 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557d7f09f080 msgr2=0x557d7f0a1470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:40:14.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.877+0000 7fa6d2a00640 1 --2- 172.21.15.90:0/2327902241 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557d7f09f080 0x557d7f0a1470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:14.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.877+0000 7fa6d2a00640 1 -- 172.21.15.90:0/2327902241 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557d7ed0e3f0 con 0x557d7f095d10 2024-04-02T14:40:14.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.878+0000 7fa6d2a00640 1 --2- 172.21.15.90:0/2327902241 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557d7f095d10 0x557d7f096190 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fa6c40ce1c0 tx=0x7fa6c40d95d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f8379c9735389b0a server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:40:14.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.879+0000 7fa6d8a00640 1 --2- 172.21.15.90:0/894492857 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fa6b80a3e20 0x7fa6b80a62e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:14.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.879+0000 7fa6d8a00640 1 -- 172.21.15.90:0/894492857 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7fa6c40d0b80 con 0x7fa6d4108310 2024-04-02T14:40:14.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.879+0000 7fa6bfe00640 1 -- 172.21.15.90:0/2327902241 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6c40d4e60 con 0x557d7f095d10 2024-04-02T14:40:14.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.879+0000 7fa6d3400640 1 --2- 172.21.15.90:0/894492857 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fa6b80a3e20 0x7fa6b80a62e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:14.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.880+0000 7fa6bfe00640 1 -- 172.21.15.90:0/2327902241 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa6c40e2070 con 0x557d7f095d10 2024-04-02T14:40:14.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.880+0000 7fa6bfe00640 1 -- 172.21.15.90:0/2327902241 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6c40dc6f0 con 0x557d7f095d10 2024-04-02T14:40:14.882 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.881+0000 7fa6dcb64740 1 -- 172.21.15.90:0/2327902241 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557d7f095d10 msgr2=0x557d7f096190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:14.882 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.881+0000 7fa6d3400640 1 --2- 172.21.15.90:0/894492857 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fa6b80a3e20 0x7fa6b80a62e0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fa6cc004af0 tx=0x7fa6cc005fd0 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:40:14.882 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.881+0000 7fa6dcb64740 1 --2- 172.21.15.90:0/2327902241 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557d7f095d10 0x557d7f096190 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fa6c40ce1c0 tx=0x7fa6c40d95d0 comp rx=0 tx=0).stop 2024-04-02T14:40:14.882 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.881+0000 7fa6dcb64740 1 -- 172.21.15.90:0/2327902241 shutdown_connections 2024-04-02T14:40:14.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.881+0000 7fa6dcb64740 1 --2- 172.21.15.90:0/2327902241 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557d7f09f080 0x557d7f0a1470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:14.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.881+0000 7fa6dcb64740 1 --2- 172.21.15.90:0/2327902241 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557d7f095d10 0x557d7f096190 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:14.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.881+0000 7fa6dcb64740 1 --2- 172.21.15.90:0/2327902241 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557d7f0969f0 0x557d7f095740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:14.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.881+0000 7fa6dcb64740 1 -- 172.21.15.90:0/2327902241 >> 172.21.15.90:0/2327902241 conn(0x557d7f05ca50 msgr2=0x557d7f090760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:40:14.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.881+0000 7fa6dcb64740 1 -- 172.21.15.90:0/2327902241 shutdown_connections 2024-04-02T14:40:14.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.881+0000 7fa6dcb64740 1 -- 172.21.15.90:0/2327902241 wait complete. 2024-04-02T14:40:14.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.882+0000 7fa6dcb64740 1 -- start start 2024-04-02T14:40:14.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.882+0000 7fa6dcb64740 1 Processor -- start 2024-04-02T14:40:14.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.883+0000 7fa6dcb64740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557d7f095d10 0x557d7f09c120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:14.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.883+0000 7fa6dcb64740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557d7f0969f0 0x557d7f09c660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:14.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.883+0000 7fa6dcb64740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557d7f09f080 0x557d7f09cba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:14.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.884+0000 7fa6d2a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557d7f095d10 0x557d7f09c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:14.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.884+0000 7fa6d2000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557d7f0969f0 0x557d7f09c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:14.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.884+0000 7fa6dcb64740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x557d7eff66a0 con 0x557d7f095d10 2024-04-02T14:40:14.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.884+0000 7fa6d2a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557d7f095d10 0x557d7f09c120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:52958/0 (socket says 172.21.15.90:52958) 2024-04-02T14:40:14.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.884+0000 7fa6d2a00640 1 -- 172.21.15.90:0/3302156287 learned_addr learned my addr 172.21.15.90:0/3302156287 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:40:14.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.884+0000 7fa6dcb64740 1 -- 172.21.15.90:0/3302156287 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x557d7efa2980 con 0x557d7f0969f0 2024-04-02T14:40:14.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.884+0000 7fa6dcb64740 1 -- 172.21.15.90:0/3302156287 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x557d7efb5cb0 con 0x557d7f09f080 2024-04-02T14:40:14.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.884+0000 7fa6d3400640 1 --2- 172.21.15.90:0/3302156287 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557d7f09f080 0x557d7f09cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:14.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.885+0000 7fa6dcb64740 10 client.0 _ll_register_callbacks cb 0x557d7efb56a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:40:14.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.885+0000 7fa6d2000640 1 -- 172.21.15.90:0/3302156287 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557d7f09f080 msgr2=0x557d7f09cba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:40:14.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.885+0000 7fa6d2000640 1 --2- 172.21.15.90:0/3302156287 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557d7f09f080 0x557d7f09cba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:14.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.885+0000 7fa6d2000640 1 -- 172.21.15.90:0/3302156287 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557d7f095d10 msgr2=0x557d7f09c120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:40:14.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.885+0000 7fa6d2000640 1 --2- 172.21.15.90:0/3302156287 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557d7f095d10 0x557d7f09c120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:14.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.885+0000 7fa6d2000640 1 -- 172.21.15.90:0/3302156287 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557d7edd87d0 con 0x557d7f0969f0 2024-04-02T14:40:14.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.885+0000 7fa6dcb64740 1 -- 172.21.15.90:0/3302156287 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa6c40e2070 con 0x557d7f0969f0 2024-04-02T14:40:14.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.885+0000 7fa6d2a00640 1 --2- 172.21.15.90:0/3302156287 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557d7f095d10 0x557d7f09c120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:40:14.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.885+0000 7fa6d3400640 1 --2- 172.21.15.90:0/3302156287 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557d7f09f080 0x557d7f09cba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:40:14.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.886+0000 7fa6d2000640 1 --2- 172.21.15.90:0/3302156287 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557d7f0969f0 0x557d7f09c660 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fa6d419faf0 tx=0x7fa6c0012bd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9fe2d37fc9ce128c server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:40:14.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.886+0000 7fa6bcc00640 10 client.27044 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:40:14.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.886+0000 7fa6bcc00640 1 -- 172.21.15.90:0/3302156287 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6c000f890 con 0x557d7f0969f0 2024-04-02T14:40:14.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.886+0000 7fa6bcc00640 1 -- 172.21.15.90:0/3302156287 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7fa6c000fa30 con 0x557d7f0969f0 2024-04-02T14:40:14.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.886+0000 7fa6bcc00640 1 -- 172.21.15.90:0/3302156287 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fa6c000fc90 con 0x557d7f0969f0 2024-04-02T14:40:14.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.886+0000 7fa6dcb64740 10 client.27044 fetch_fsmap learned FSMap version 27 2024-04-02T14:40:14.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.886+0000 7fa6dcb64740 1 -- 172.21.15.90:0/3302156287 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({fsmap=27}) -- 0x557d7f0bef80 con 0x557d7f0969f0 2024-04-02T14:40:14.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.887+0000 7fa6bcc00640 1 -- 172.21.15.90:0/3302156287 <== mon.2 v2:172.21.15.92:3300/0 4 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fa6c0026470 con 0x557d7f0969f0 2024-04-02T14:40:14.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.887+0000 7fa6dcb64740 10 client.27044 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:40:14.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.887+0000 7fa6dcb64740 10 client.27044 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2024-04-02T14:40:14.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.887+0000 7fa6dcb64740 20 client.27044 populate_metadata read hostname 'smithi090' 2024-04-02T14:40:14.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.887+0000 7fa6dcb64740 1 --2- 172.21.15.90:0/3302156287 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x557d7f0c03d0 0x557d7f0c27c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:14.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.887+0000 7fa6dcb64740 4 client.27044 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:40:14.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.887+0000 7fa6dcb64740 1 -- 172.21.15.90:0/3302156287 --> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557d7efba040 con 0x557d7f0c03d0 2024-04-02T14:40:14.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.887+0000 7fa6d2a00640 1 --2- 172.21.15.90:0/3302156287 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x557d7f0c03d0 0x557d7f0c27c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:14.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.888+0000 7fa6d2a00640 1 --2- 172.21.15.90:0/3302156287 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x557d7f0c03d0 0x557d7f0c27c0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=7aa16cdc0d5fd1a0 server_cookie=f68a2829f680e92a in_seq=0 out_seq=0 2024-04-02T14:40:14.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.888+0000 7fa6bcc00640 10 client.27044 ms_handle_connect on v2:172.21.15.90:6838/1498915870 2024-04-02T14:40:14.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.888+0000 7fa6bcc00640 1 -- 172.21.15.90:0/3302156287 <== mds.1 v2:172.21.15.90:6838/1498915870 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x557d7efba040 con 0x557d7f0c03d0 2024-04-02T14:40:14.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.888+0000 7fa6bcc00640 10 client.27044 handle_command_reply: tid=0 2024-04-02T14:40:14.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.889+0000 7fa6dcb64740 1 client.27044 shutdown 2024-04-02T14:40:14.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.889+0000 7fa6dcb64740 1 -- 172.21.15.90:0/3302156287 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557d7f0969f0 msgr2=0x557d7f09c660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:14.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.889+0000 7fa6dcb64740 1 --2- 172.21.15.90:0/3302156287 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557d7f0969f0 0x557d7f09c660 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fa6d419faf0 tx=0x7fa6c0012bd0 comp rx=0 tx=0).stop 2024-04-02T14:40:14.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.889+0000 7fa6dcb64740 1 -- 172.21.15.90:0/3302156287 shutdown_connections 2024-04-02T14:40:14.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.889+0000 7fa6dcb64740 1 --2- 172.21.15.90:0/3302156287 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x557d7f0c03d0 0x557d7f0c27c0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:14.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.889+0000 7fa6dcb64740 1 --2- 172.21.15.90:0/3302156287 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557d7f09f080 0x557d7f09cba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:14.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.889+0000 7fa6dcb64740 1 --2- 172.21.15.90:0/3302156287 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557d7f0969f0 0x557d7f09c660 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:14.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.889+0000 7fa6dcb64740 1 --2- 172.21.15.90:0/3302156287 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557d7f095d10 0x557d7f09c120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:14.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.889+0000 7fa6dcb64740 1 -- 172.21.15.90:0/3302156287 >> 172.21.15.90:0/3302156287 conn(0x557d7f05ca50 msgr2=0x557d7f0924b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:40:14.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.889+0000 7fa6bcc00640 0 client.27044 ms_handle_reset on v2:172.21.15.90:6838/1498915870 2024-04-02T14:40:14.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.889+0000 7fa6dcb64740 1 -- 172.21.15.90:0/3302156287 shutdown_connections 2024-04-02T14:40:14.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.890+0000 7fa6dcb64740 1 -- 172.21.15.90:0/3302156287 wait complete. 2024-04-02T14:40:14.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.890+0000 7fa6dcb64740 20 client.27044 trim_cache size 0 max 16384 2024-04-02T14:40:14.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.915+0000 7fa6dcb64740 1 Processor -- start 2024-04-02T14:40:14.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.915+0000 7fa6dcb64740 1 -- start start 2024-04-02T14:40:14.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.915+0000 7fa6dcb64740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557d7f095d10 0x557d7f0bbaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:14.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.915+0000 7fa6dcb64740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557d7f09f080 0x557d7f09c9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:14.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.915+0000 7fa6dcb64740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557d7f0922c0 0x557d7f0c8050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:14.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.915+0000 7fa6dcb64740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fa6d40827c0 con 0x557d7f095d10 2024-04-02T14:40:14.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.915+0000 7fa6dcb64740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fa6d41ad1b0 con 0x557d7f0922c0 2024-04-02T14:40:14.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.915+0000 7fa6dcb64740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fa6d41abd40 con 0x557d7f09f080 2024-04-02T14:40:14.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.916+0000 7fa6d3400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557d7f0922c0 0x557d7f0c8050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:14.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.916+0000 7fa6d3400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557d7f0922c0 0x557d7f0c8050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:39294/0 (socket says 172.21.15.90:39294) 2024-04-02T14:40:14.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.916+0000 7fa6d3400640 1 -- 172.21.15.90:0/2718882461 learned_addr learned my addr 172.21.15.90:0/2718882461 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:40:14.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.916+0000 7fa6d2000640 1 --2- 172.21.15.90:0/2718882461 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557d7f09f080 0x557d7f09c9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:14.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.916+0000 7fa6d2a00640 1 --2- 172.21.15.90:0/2718882461 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557d7f095d10 0x557d7f0bbaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:14.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.917+0000 7fa6d2000640 1 -- 172.21.15.90:0/2718882461 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557d7f0922c0 msgr2=0x557d7f0c8050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:40:14.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.917+0000 7fa6d2000640 1 --2- 172.21.15.90:0/2718882461 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557d7f0922c0 0x557d7f0c8050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:14.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.917+0000 7fa6d2000640 1 -- 172.21.15.90:0/2718882461 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557d7f095d10 msgr2=0x557d7f0bbaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:40:14.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.917+0000 7fa6d2000640 1 --2- 172.21.15.90:0/2718882461 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557d7f095d10 0x557d7f0bbaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:14.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.917+0000 7fa6d2000640 1 -- 172.21.15.90:0/2718882461 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557d7f0bca20 con 0x557d7f09f080 2024-04-02T14:40:14.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.917+0000 7fa6d2a00640 1 --2- 172.21.15.90:0/2718882461 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557d7f095d10 0x557d7f0bbaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:40:14.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.917+0000 7fa6d2000640 1 --2- 172.21.15.90:0/2718882461 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557d7f09f080 0x557d7f09c9c0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x557d7f09d1b0 tx=0x7fa6c0014000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4b6a67e8bfef56a9 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:40:14.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.918+0000 7fa6bf400640 1 -- 172.21.15.90:0/2718882461 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6c000f790 con 0x557d7f09f080 2024-04-02T14:40:14.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.918+0000 7fa6bf400640 1 -- 172.21.15.90:0/2718882461 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa6c002c070 con 0x557d7f09f080 2024-04-02T14:40:14.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.918+0000 7fa6bf400640 1 -- 172.21.15.90:0/2718882461 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6c00053a0 con 0x557d7f09f080 2024-04-02T14:40:14.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.919+0000 7fa6dcb64740 1 -- 172.21.15.90:0/2718882461 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557d7f09f080 msgr2=0x557d7f09c9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:14.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.919+0000 7fa6dcb64740 1 --2- 172.21.15.90:0/2718882461 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557d7f09f080 0x557d7f09c9c0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x557d7f09d1b0 tx=0x7fa6c0014000 comp rx=0 tx=0).stop 2024-04-02T14:40:14.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.919+0000 7fa6dcb64740 1 -- 172.21.15.90:0/2718882461 shutdown_connections 2024-04-02T14:40:14.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.919+0000 7fa6dcb64740 1 --2- 172.21.15.90:0/2718882461 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557d7f0922c0 0x557d7f0c8050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:14.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.919+0000 7fa6dcb64740 1 --2- 172.21.15.90:0/2718882461 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557d7f09f080 0x557d7f09c9c0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:14.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.919+0000 7fa6dcb64740 1 --2- 172.21.15.90:0/2718882461 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557d7f095d10 0x557d7f0bbaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:14.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.919+0000 7fa6dcb64740 1 -- 172.21.15.90:0/2718882461 >> 172.21.15.90:0/2718882461 conn(0x557d7f05ca50 msgr2=0x557d7f091bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:40:14.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.919+0000 7fa6dcb64740 1 -- 172.21.15.90:0/2718882461 shutdown_connections 2024-04-02T14:40:14.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.919+0000 7fa6dcb64740 1 -- 172.21.15.90:0/2718882461 wait complete. 2024-04-02T14:40:14.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.919+0000 7fa6dcb64740 1 -- start start 2024-04-02T14:40:14.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.920+0000 7fa6dcb64740 1 Processor -- start 2024-04-02T14:40:14.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.920+0000 7fa6dcb64740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557d7f0922c0 0x557d7f09fca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:14.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.920+0000 7fa6dcb64740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557d7f095d10 0x557d7f0a01e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:14.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.920+0000 7fa6dcb64740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557d7f09f080 0x557d7f0a0720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:14.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.920+0000 7fa6dcb64740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fa6d41acae0 con 0x557d7f0922c0 2024-04-02T14:40:14.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.920+0000 7fa6dcb64740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fa6d41ac410 con 0x557d7f09f080 2024-04-02T14:40:14.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.920+0000 7fa6dcb64740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fa6d40807a0 con 0x557d7f095d10 2024-04-02T14:40:14.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.920+0000 7fa6dcb64740 10 client.0 _ll_register_callbacks cb 0x7fa6c0009ed0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:40:14.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.921+0000 7fa6d3400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557d7f09f080 0x557d7f0a0720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:14.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.921+0000 7fa6d3400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557d7f09f080 0x557d7f0a0720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:39310/0 (socket says 172.21.15.90:39310) 2024-04-02T14:40:14.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.921+0000 7fa6d3400640 1 -- 172.21.15.90:0/855292584 learned_addr learned my addr 172.21.15.90:0/855292584 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:40:14.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.921+0000 7fa6d2000640 1 --2- 172.21.15.90:0/855292584 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557d7f095d10 0x557d7f0a01e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:14.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.921+0000 7fa6d2a00640 1 --2- 172.21.15.90:0/855292584 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557d7f0922c0 0x557d7f09fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:14.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.922+0000 7fa6d3400640 1 -- 172.21.15.90:0/855292584 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557d7f095d10 msgr2=0x557d7f0a01e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:40:14.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.922+0000 7fa6d3400640 1 --2- 172.21.15.90:0/855292584 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557d7f095d10 0x557d7f0a01e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:14.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.922+0000 7fa6d3400640 1 -- 172.21.15.90:0/855292584 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557d7f0922c0 msgr2=0x557d7f09fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:40:14.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.922+0000 7fa6d3400640 1 --2- 172.21.15.90:0/855292584 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557d7f0922c0 0x557d7f09fca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:14.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.922+0000 7fa6d3400640 1 -- 172.21.15.90:0/855292584 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557d7f0b60f0 con 0x557d7f09f080 2024-04-02T14:40:14.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.922+0000 7fa6bcc00640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:40:14.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.922+0000 7fa6d3400640 1 --2- 172.21.15.90:0/855292584 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557d7f09f080 0x557d7f0a0720 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fa6cc036040 tx=0x7fa6cc009500 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4cec7fdd41cc6262 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:40:14.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.923+0000 7fa6bcc00640 1 -- 172.21.15.90:0/855292584 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6cc051070 con 0x557d7f09f080 2024-04-02T14:40:14.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.923+0000 7fa6bcc00640 1 -- 172.21.15.90:0/855292584 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7fa6cc03a040 con 0x557d7f09f080 2024-04-02T14:40:14.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.923+0000 7fa6dcb64740 1 -- 172.21.15.90:0/855292584 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa6c002c070 con 0x557d7f09f080 2024-04-02T14:40:14.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.923+0000 7fa6bcc00640 1 -- 172.21.15.90:0/855292584 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fa6cc05b050 con 0x557d7f09f080 2024-04-02T14:40:14.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.924+0000 7fa6dcb64740 10 client.27050 fetch_fsmap learned FSMap version 27 2024-04-02T14:40:14.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.924+0000 7fa6dcb64740 1 -- 172.21.15.90:0/855292584 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x557d7f126b50 con 0x557d7f09f080 2024-04-02T14:40:14.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.924+0000 7fa6bcc00640 1 -- 172.21.15.90:0/855292584 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa6c002c070 con 0x557d7f09f080 2024-04-02T14:40:14.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.925+0000 7fa6bcc00640 1 -- 172.21.15.90:0/855292584 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fa6cc02caa0 con 0x557d7f09f080 2024-04-02T14:40:14.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.926+0000 7fa6dcb64740 10 client.27050 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:40:14.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.926+0000 7fa6dcb64740 10 client.27050 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2024-04-02T14:40:14.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.926+0000 7fa6dcb64740 20 client.27050 populate_metadata read hostname 'smithi090' 2024-04-02T14:40:14.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.926+0000 7fa6dcb64740 1 --2- 172.21.15.90:0/855292584 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x557d7f127270 0x557d7f129660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:14.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.926+0000 7fa6dcb64740 4 client.27050 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-02T14:40:14.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.926+0000 7fa6dcb64740 1 -- 172.21.15.90:0/855292584 --> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] -- command(tid 0: {"prefix": "damage ls"}) -- 0x557d7f007c30 con 0x557d7f127270 2024-04-02T14:40:14.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.927+0000 7fa6d2a00640 1 --2- 172.21.15.90:0/855292584 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x557d7f127270 0x557d7f129660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:14.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.927+0000 7fa6bcc00640 10 client.27050 ms_handle_connect on v2:172.21.15.90:6838/1498915870 2024-04-02T14:40:14.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.927+0000 7fa6d2a00640 1 --2- 172.21.15.90:0/855292584 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x557d7f127270 0x557d7f129660 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=8e463ac41061a623 server_cookie=5259ded6cb03ca38 in_seq=0 out_seq=0 2024-04-02T14:40:14.929 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.928+0000 7fa6bcc00640 1 -- 172.21.15.90:0/855292584 <== mds.1 v2:172.21.15.90:6838/1498915870 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x557d7f007c30 con 0x557d7f127270 2024-04-02T14:40:14.929 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.928+0000 7fa6bcc00640 10 client.27050 handle_command_reply: tid=0 2024-04-02T14:40:14.929 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.928+0000 7fa6dcb64740 1 client.27050 shutdown 2024-04-02T14:40:14.929 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.929+0000 7fa6dcb64740 1 -- 172.21.15.90:0/855292584 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557d7f09f080 msgr2=0x557d7f0a0720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:14.929 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.929+0000 7fa6dcb64740 1 --2- 172.21.15.90:0/855292584 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557d7f09f080 0x557d7f0a0720 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fa6cc036040 tx=0x7fa6cc009500 comp rx=0 tx=0).stop 2024-04-02T14:40:14.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.929+0000 7fa6dcb64740 1 -- 172.21.15.90:0/855292584 shutdown_connections 2024-04-02T14:40:14.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.929+0000 7fa6dcb64740 1 --2- 172.21.15.90:0/855292584 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x557d7f127270 0x557d7f129660 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:14.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.929+0000 7fa6dcb64740 1 --2- 172.21.15.90:0/855292584 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557d7f09f080 0x557d7f0a0720 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:14.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.929+0000 7fa6dcb64740 1 --2- 172.21.15.90:0/855292584 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557d7f095d10 0x557d7f0a01e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:14.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.929+0000 7fa6dcb64740 1 --2- 172.21.15.90:0/855292584 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557d7f0922c0 0x557d7f09fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:14.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.929+0000 7fa6dcb64740 1 -- 172.21.15.90:0/855292584 >> 172.21.15.90:0/855292584 conn(0x557d7f05ca50 msgr2=0x557d7f08f010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:40:14.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.929+0000 7fa6bcc00640 0 client.27050 ms_handle_reset on v2:172.21.15.90:6838/1498915870 2024-04-02T14:40:14.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.929+0000 7fa6dcb64740 1 -- 172.21.15.90:0/855292584 shutdown_connections 2024-04-02T14:40:14.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.930+0000 7fa6dcb64740 1 -- 172.21.15.90:0/855292584 wait complete. 2024-04-02T14:40:14.932 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-02T14:40:14.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.930+0000 7fa6dcb64740 20 client.27050 trim_cache size 0 max 16384 2024-04-02T14:40:14.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.931+0000 7fa6d9e00640 1 -- 172.21.15.90:0/894492857 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fa6b80a3e20 msgr2=0x7fa6b80a62e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:14.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.931+0000 7fa6d9e00640 1 --2- 172.21.15.90:0/894492857 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fa6b80a3e20 0x7fa6b80a62e0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fa6cc004af0 tx=0x7fa6cc005fd0 comp rx=0 tx=0).stop 2024-04-02T14:40:14.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.931+0000 7fa6d9e00640 1 -- 172.21.15.90:0/894492857 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa6d4108310 msgr2=0x7fa6d41a4630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:14.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.931+0000 7fa6d9e00640 1 --2- 172.21.15.90:0/894492857 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa6d4108310 0x7fa6d41a4630 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fa6c400e9f0 tx=0x7fa6c400eeb0 comp rx=0 tx=0).stop 2024-04-02T14:40:14.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.931+0000 7fa6d9e00640 1 -- 172.21.15.90:0/894492857 shutdown_connections 2024-04-02T14:40:14.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.931+0000 7fa6d9e00640 1 --2- 172.21.15.90:0/894492857 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fa6b80a3e20 0x7fa6b80a62e0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:14.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.931+0000 7fa6d9e00640 1 --2- 172.21.15.90:0/894492857 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa6d419e7f0 0x7fa6d419eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:14.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.931+0000 7fa6d9e00640 1 --2- 172.21.15.90:0/894492857 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa6d410d3e0 0x7fa6d41a4b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:14.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.931+0000 7fa6d9e00640 1 --2- 172.21.15.90:0/894492857 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa6d4108310 0x7fa6d41a4630 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:14.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.931+0000 7fa6d9e00640 1 -- 172.21.15.90:0/894492857 >> 172.21.15.90:0/894492857 conn(0x7fa6d4076040 msgr2=0x7fa6d4078340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:40:14.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.931+0000 7fa6d9e00640 1 -- 172.21.15.90:0/894492857 shutdown_connections 2024-04-02T14:40:14.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:14.931+0000 7fa6d9e00640 1 -- 172.21.15.90:0/894492857 wait complete. 2024-04-02T14:40:14.972 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-02T14:40:15.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.255+0000 7fdb96c00640 1 -- 172.21.15.90:0/3812827565 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdb901048c0 msgr2=0x7fdb90108900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:15.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.255+0000 7fdb96c00640 1 --2- 172.21.15.90:0/3812827565 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdb901048c0 0x7fdb90108900 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fdb7c009900 tx=0x7fdb7c02bc50 comp rx=0 tx=0).stop 2024-04-02T14:40:15.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.255+0000 7fdb96c00640 1 -- 172.21.15.90:0/3812827565 shutdown_connections 2024-04-02T14:40:15.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.255+0000 7fdb96c00640 1 --2- 172.21.15.90:0/3812827565 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdb90108fe0 0x7fdb90110bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:15.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.255+0000 7fdb96c00640 1 --2- 172.21.15.90:0/3812827565 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdb901048c0 0x7fdb90108900 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:15.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.255+0000 7fdb96c00640 1 --2- 172.21.15.90:0/3812827565 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdb90103f10 0x7fdb901042f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:15.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.255+0000 7fdb96c00640 1 -- 172.21.15.90:0/3812827565 >> 172.21.15.90:0/3812827565 conn(0x7fdb900ff7b0 msgr2=0x7fdb90101bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:40:15.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.256+0000 7fdb96c00640 1 -- 172.21.15.90:0/3812827565 shutdown_connections 2024-04-02T14:40:15.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.256+0000 7fdb96c00640 1 -- 172.21.15.90:0/3812827565 wait complete. 2024-04-02T14:40:15.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.258+0000 7fdb96c00640 1 Processor -- start 2024-04-02T14:40:15.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.258+0000 7fdb96c00640 1 -- start start 2024-04-02T14:40:15.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.258+0000 7fdb96c00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdb90103f10 0x7fdb9019a620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:15.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.258+0000 7fdb96c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdb90108fe0 0x7fdb9019ab60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:15.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.258+0000 7fdb96c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdb9019eee0 0x7fdb9019f390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:15.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.258+0000 7fdb96c00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fdb901140c0 con 0x7fdb9019eee0 2024-04-02T14:40:15.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.258+0000 7fdb96c00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fdb90113f40 con 0x7fdb90103f10 2024-04-02T14:40:15.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.258+0000 7fdb96c00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fdb90114240 con 0x7fdb90108fe0 2024-04-02T14:40:15.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.259+0000 7fdb8fe00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdb90103f10 0x7fdb9019a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:15.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.259+0000 7fdb8fe00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdb90103f10 0x7fdb9019a620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:39328/0 (socket says 172.21.15.90:39328) 2024-04-02T14:40:15.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.259+0000 7fdb8f400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdb90108fe0 0x7fdb9019ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:15.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.259+0000 7fdb8fe00640 1 -- 172.21.15.90:0/841489669 learned_addr learned my addr 172.21.15.90:0/841489669 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:40:15.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.259+0000 7fdb94c00640 1 --2- 172.21.15.90:0/841489669 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdb9019eee0 0x7fdb9019f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:15.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.259+0000 7fdb8f400640 1 -- 172.21.15.90:0/841489669 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdb90103f10 msgr2=0x7fdb9019a620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:15.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.259+0000 7fdb8f400640 1 --2- 172.21.15.90:0/841489669 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdb90103f10 0x7fdb9019a620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:15.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.259+0000 7fdb8f400640 1 -- 172.21.15.90:0/841489669 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdb9019eee0 msgr2=0x7fdb9019f390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:15.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.259+0000 7fdb8f400640 1 --2- 172.21.15.90:0/841489669 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdb9019eee0 0x7fdb9019f390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:15.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.259+0000 7fdb8f400640 1 -- 172.21.15.90:0/841489669 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb9019f9d0 con 0x7fdb90108fe0 2024-04-02T14:40:15.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.261+0000 7fdb8f400640 1 --2- 172.21.15.90:0/841489669 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdb90108fe0 0x7fdb9019ab60 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fdb7c036040 tx=0x7fdb7c0042e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:40:15.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.262+0000 7fdb8cc00640 1 -- 172.21.15.90:0/841489669 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb7c041070 con 0x7fdb90108fe0 2024-04-02T14:40:15.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.262+0000 7fdb8cc00640 1 -- 172.21.15.90:0/841489669 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdb7c0044d0 con 0x7fdb90108fe0 2024-04-02T14:40:15.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.262+0000 7fdb8cc00640 1 -- 172.21.15.90:0/841489669 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb7c003d00 con 0x7fdb90108fe0 2024-04-02T14:40:15.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.263+0000 7fdb96c00640 1 -- 172.21.15.90:0/841489669 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb9019fc60 con 0x7fdb90108fe0 2024-04-02T14:40:15.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.263+0000 7fdb96c00640 1 -- 172.21.15.90:0/841489669 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb901a73d0 con 0x7fdb90108fe0 2024-04-02T14:40:15.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.267+0000 7fdb9987e740 1 Processor -- start 2024-04-02T14:40:15.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.267+0000 7fdb8cc00640 1 -- 172.21.15.90:0/841489669 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fdb7c044070 con 0x7fdb90108fe0 2024-04-02T14:40:15.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.268+0000 7fdb9987e740 1 -- start start 2024-04-02T14:40:15.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.269+0000 7fdb9987e740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c849d6a9f0 0x55c849d69740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:15.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.269+0000 7fdb9987e740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c849d69d10 0x55c849d6a190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:15.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.269+0000 7fdb9987e740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c849d73080 0x55c849d75470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:15.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.269+0000 7fdb9987e740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55c849caa400 con 0x55c849d69d10 2024-04-02T14:40:15.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.269+0000 7fdb9987e740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55c849c8ef10 con 0x55c849d73080 2024-04-02T14:40:15.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.269+0000 7fdb9987e740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55c849cce6f0 con 0x55c849d6a9f0 2024-04-02T14:40:15.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.269+0000 7fdb8cc00640 1 --2- 172.21.15.90:0/841489669 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fdb740a3e60 0x7fdb740a6320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:15.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.269+0000 7fdb8f400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c849d73080 0x55c849d75470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:15.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.269+0000 7fdb8f400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c849d73080 0x55c849d75470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:39330/0 (socket says 172.21.15.90:39330) 2024-04-02T14:40:15.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.269+0000 7fdb8f400640 1 -- 172.21.15.90:0/2610368545 learned_addr learned my addr 172.21.15.90:0/2610368545 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:40:15.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.269+0000 7fdb8fe00640 1 --2- 172.21.15.90:0/2610368545 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c849d69d10 0x55c849d6a190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:15.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.269+0000 7fdb8cc00640 1 -- 172.21.15.90:0/841489669 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7fdb7c0f23a0 con 0x7fdb90108fe0 2024-04-02T14:40:15.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.269+0000 7fdb94c00640 1 --2- 172.21.15.90:0/841489669 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fdb740a3e60 0x7fdb740a6320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:15.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.270+0000 7fdb8f400640 1 -- 172.21.15.90:0/2610368545 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c849d6a9f0 msgr2=0x55c849d69740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:40:15.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.270+0000 7fdb94c00640 1 --2- 172.21.15.90:0/2610368545 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c849d6a9f0 0x55c849d69740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:15.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.270+0000 7fdb8f400640 1 --2- 172.21.15.90:0/2610368545 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c849d6a9f0 0x55c849d69740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:15.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.270+0000 7fdb8f400640 1 -- 172.21.15.90:0/2610368545 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c849d69d10 msgr2=0x55c849d6a190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:40:15.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.270+0000 7fdb8f400640 1 --2- 172.21.15.90:0/2610368545 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c849d69d10 0x55c849d6a190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:15.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.270+0000 7fdb8f400640 1 -- 172.21.15.90:0/2610368545 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c8499e23f0 con 0x55c849d73080 2024-04-02T14:40:15.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.270+0000 7fdb94c00640 1 --2- 172.21.15.90:0/2610368545 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c849d6a9f0 0x55c849d69740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:40:15.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.270+0000 7fdb8fe00640 1 --2- 172.21.15.90:0/2610368545 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c849d69d10 0x55c849d6a190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:40:15.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.270+0000 7fdb8f400640 1 --2- 172.21.15.90:0/2610368545 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c849d73080 0x55c849d75470 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fdb7c0c6b40 tx=0x7fdb7c090f10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cdcb3e0bfd83770d server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:40:15.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.270+0000 7fdb85200640 1 -- 172.21.15.90:0/2610368545 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb7c04b020 con 0x55c849d73080 2024-04-02T14:40:15.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.270+0000 7fdb85200640 1 -- 172.21.15.90:0/2610368545 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdb7c0035f0 con 0x55c849d73080 2024-04-02T14:40:15.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.270+0000 7fdb85200640 1 -- 172.21.15.90:0/2610368545 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb7c0baa90 con 0x55c849d73080 2024-04-02T14:40:15.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.272+0000 7fdb9987e740 1 -- 172.21.15.90:0/2610368545 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c849d73080 msgr2=0x55c849d75470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:15.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.272+0000 7fdb94c00640 1 --2- 172.21.15.90:0/841489669 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fdb740a3e60 0x7fdb740a6320 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fdb88005fd0 tx=0x7fdb88009b60 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:40:15.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.272+0000 7fdb9987e740 1 --2- 172.21.15.90:0/2610368545 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c849d73080 0x55c849d75470 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fdb7c0c6b40 tx=0x7fdb7c090f10 comp rx=0 tx=0).stop 2024-04-02T14:40:15.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.272+0000 7fdb9987e740 1 -- 172.21.15.90:0/2610368545 shutdown_connections 2024-04-02T14:40:15.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.272+0000 7fdb9987e740 1 --2- 172.21.15.90:0/2610368545 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c849d73080 0x55c849d75470 secure :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fdb7c0c6b40 tx=0x7fdb7c090f10 comp rx=0 tx=0).stop 2024-04-02T14:40:15.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.272+0000 7fdb9987e740 1 --2- 172.21.15.90:0/2610368545 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c849d69d10 0x55c849d6a190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:15.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.272+0000 7fdb9987e740 1 --2- 172.21.15.90:0/2610368545 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c849d6a9f0 0x55c849d69740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:15.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.273+0000 7fdb9987e740 1 -- 172.21.15.90:0/2610368545 >> 172.21.15.90:0/2610368545 conn(0x55c849d30a50 msgr2=0x55c849d64760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:40:15.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.273+0000 7fdb9987e740 1 -- 172.21.15.90:0/2610368545 shutdown_connections 2024-04-02T14:40:15.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.273+0000 7fdb9987e740 1 -- 172.21.15.90:0/2610368545 wait complete. 2024-04-02T14:40:15.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.273+0000 7fdb9987e740 1 -- start start 2024-04-02T14:40:15.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.274+0000 7fdb9987e740 1 Processor -- start 2024-04-02T14:40:15.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.274+0000 7fdb9987e740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c849d69d10 0x55c849d70130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:15.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.274+0000 7fdb9987e740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c849d6a9f0 0x55c849d70670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:15.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.274+0000 7fdb9987e740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c849d70bb0 0x55c849d7eda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:15.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.274+0000 7fdb9987e740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55c849cca6a0 con 0x55c849d69d10 2024-04-02T14:40:15.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.274+0000 7fdb9987e740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55c849c76980 con 0x55c849d6a9f0 2024-04-02T14:40:15.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.274+0000 7fdb9987e740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55c849c89cb0 con 0x55c849d70bb0 2024-04-02T14:40:15.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.274+0000 7fdb9987e740 10 client.0 _ll_register_callbacks cb 0x55c849c896a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:40:15.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.275+0000 7fdb94c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c849d70bb0 0x55c849d7eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:15.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.275+0000 7fdb94c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c849d70bb0 0x55c849d7eda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:54010/0 (socket says 172.21.15.90:54010) 2024-04-02T14:40:15.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.275+0000 7fdb94c00640 1 -- 172.21.15.90:0/3755888823 learned_addr learned my addr 172.21.15.90:0/3755888823 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:40:15.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.275+0000 7fdb8f400640 1 --2- 172.21.15.90:0/3755888823 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c849d6a9f0 0x55c849d70670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:15.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.275+0000 7fdb94c00640 1 -- 172.21.15.90:0/3755888823 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c849d6a9f0 msgr2=0x55c849d70670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:40:15.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.275+0000 7fdb94c00640 1 --2- 172.21.15.90:0/3755888823 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c849d6a9f0 0x55c849d70670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:15.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.275+0000 7fdb94c00640 1 -- 172.21.15.90:0/3755888823 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c849d69d10 msgr2=0x55c849d70130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:40:15.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.275+0000 7fdb94c00640 1 --2- 172.21.15.90:0/3755888823 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c849d69d10 0x55c849d70130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:15.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.275+0000 7fdb94c00640 1 -- 172.21.15.90:0/3755888823 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c849aac7d0 con 0x55c849d70bb0 2024-04-02T14:40:15.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.275+0000 7fdb8f400640 1 --2- 172.21.15.90:0/3755888823 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c849d6a9f0 0x55c849d70670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:40:15.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.276+0000 7fdb79600640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:40:15.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.276+0000 7fdb94c00640 1 --2- 172.21.15.90:0/3755888823 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c849d70bb0 0x55c849d7eda0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fdb8800fcb0 tx=0x7fdb88004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e8993a7425380241 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:40:15.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.276+0000 7fdb79600640 1 -- 172.21.15.90:0/3755888823 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb88018070 con 0x55c849d70bb0 2024-04-02T14:40:15.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.276+0000 7fdb79600640 1 -- 172.21.15.90:0/3755888823 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7fdb88029480 con 0x55c849d70bb0 2024-04-02T14:40:15.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.277+0000 7fdb9987e740 1 -- 172.21.15.90:0/3755888823 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55c849c8e040 con 0x55c849d70bb0 2024-04-02T14:40:15.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.277+0000 7fdb79600640 1 -- 172.21.15.90:0/3755888823 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fdb88018070 con 0x55c849d70bb0 2024-04-02T14:40:15.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.277+0000 7fdb9987e740 10 client.26944 fetch_fsmap learned FSMap version 27 2024-04-02T14:40:15.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.277+0000 7fdb9987e740 1 -- 172.21.15.90:0/3755888823 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55c849d8a940 con 0x55c849d70bb0 2024-04-02T14:40:15.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.277+0000 7fdb79600640 1 -- 172.21.15.90:0/3755888823 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdb88005020 con 0x55c849d70bb0 2024-04-02T14:40:15.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.278+0000 7fdb79600640 1 -- 172.21.15.90:0/3755888823 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fdb88033420 con 0x55c849d70bb0 2024-04-02T14:40:15.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.278+0000 7fdb9987e740 10 client.26944 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:40:15.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.278+0000 7fdb9987e740 10 client.26944 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:40:15.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.278+0000 7fdb9987e740 20 client.26944 populate_metadata read hostname 'smithi090' 2024-04-02T14:40:15.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.278+0000 7fdb9987e740 1 --2- 172.21.15.90:0/3755888823 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55c849d8be50 0x55c849d8e2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:15.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.278+0000 7fdb9987e740 4 client.26944 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:40:15.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.278+0000 7fdb9987e740 1 -- 172.21.15.90:0/3755888823 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c849c8e040 con 0x55c849d8be50 2024-04-02T14:40:15.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.279+0000 7fdb8fe00640 1 --2- 172.21.15.90:0/3755888823 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55c849d8be50 0x55c849d8e2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:15.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.279+0000 7fdb8fe00640 1 --2- 172.21.15.90:0/3755888823 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55c849d8be50 0x55c849d8e2d0 crc :-1 s=READY pgs=154 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=68e658b431fb73e0 server_cookie=2fb0edb252c7c151 in_seq=0 out_seq=0 2024-04-02T14:40:15.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.279+0000 7fdb79600640 10 client.26944 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:40:15.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.280+0000 7fdb79600640 1 -- 172.21.15.90:0/3755888823 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c849c8e040 con 0x55c849d8be50 2024-04-02T14:40:15.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.280+0000 7fdb79600640 10 client.26944 handle_command_reply: tid=0 2024-04-02T14:40:15.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.280+0000 7fdb9987e740 1 client.26944 shutdown 2024-04-02T14:40:15.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.280+0000 7fdb9987e740 1 -- 172.21.15.90:0/3755888823 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c849d70bb0 msgr2=0x55c849d7eda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:15.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.280+0000 7fdb9987e740 1 --2- 172.21.15.90:0/3755888823 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c849d70bb0 0x55c849d7eda0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fdb8800fcb0 tx=0x7fdb88004a40 comp rx=0 tx=0).stop 2024-04-02T14:40:15.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.280+0000 7fdb9987e740 1 -- 172.21.15.90:0/3755888823 shutdown_connections 2024-04-02T14:40:15.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.281+0000 7fdb9987e740 1 --2- 172.21.15.90:0/3755888823 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55c849d8be50 0x55c849d8e2d0 crc :-1 s=READY pgs=154 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:15.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.281+0000 7fdb9987e740 1 --2- 172.21.15.90:0/3755888823 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c849d70bb0 0x55c849d7eda0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:15.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.281+0000 7fdb9987e740 1 --2- 172.21.15.90:0/3755888823 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c849d6a9f0 0x55c849d70670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:15.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.281+0000 7fdb79600640 0 client.26944 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:40:15.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.281+0000 7fdb9987e740 1 --2- 172.21.15.90:0/3755888823 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c849d69d10 0x55c849d70130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:15.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.281+0000 7fdb9987e740 1 -- 172.21.15.90:0/3755888823 >> 172.21.15.90:0/3755888823 conn(0x55c849d30a50 msgr2=0x55c849d664b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:40:15.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.281+0000 7fdb9987e740 1 -- 172.21.15.90:0/3755888823 shutdown_connections 2024-04-02T14:40:15.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.281+0000 7fdb9987e740 1 -- 172.21.15.90:0/3755888823 wait complete. 2024-04-02T14:40:15.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.281+0000 7fdb9987e740 20 client.26944 trim_cache size 0 max 16384 2024-04-02T14:40:15.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.303+0000 7fdb9987e740 1 Processor -- start 2024-04-02T14:40:15.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.303+0000 7fdb9987e740 1 -- start start 2024-04-02T14:40:15.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.303+0000 7fdb9987e740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c849d69d10 0x55c849d7e870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:15.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.303+0000 7fdb9987e740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c849d6a9f0 0x55c849d63720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:15.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.303+0000 7fdb9987e740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c849d70bb0 0x55c849d93be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:15.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.304+0000 7fdb9987e740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fdb901ab690 con 0x55c849d69d10 2024-04-02T14:40:15.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.304+0000 7fdb9987e740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fdb901ad1d0 con 0x55c849d6a9f0 2024-04-02T14:40:15.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.304+0000 7fdb9987e740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fdb901abd60 con 0x55c849d70bb0 2024-04-02T14:40:15.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.304+0000 7fdb8f400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c849d6a9f0 0x55c849d63720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:15.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.304+0000 7fdb94c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c849d70bb0 0x55c849d93be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:15.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.304+0000 7fdb8f400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c849d6a9f0 0x55c849d63720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:39354/0 (socket says 172.21.15.90:39354) 2024-04-02T14:40:15.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.304+0000 7fdb94c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c849d70bb0 0x55c849d93be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:54018/0 (socket says 172.21.15.90:54018) 2024-04-02T14:40:15.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.304+0000 7fdb8f400640 1 -- 172.21.15.90:0/401142538 learned_addr learned my addr 172.21.15.90:0/401142538 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:40:15.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.304+0000 7fdb8fe00640 1 --2- 172.21.15.90:0/401142538 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c849d69d10 0x55c849d7e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:15.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.304+0000 7fdb8f400640 1 -- 172.21.15.90:0/401142538 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c849d70bb0 msgr2=0x55c849d93be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:40:15.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.304+0000 7fdb8f400640 1 --2- 172.21.15.90:0/401142538 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c849d70bb0 0x55c849d93be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:15.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.304+0000 7fdb8f400640 1 -- 172.21.15.90:0/401142538 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c849d69d10 msgr2=0x55c849d7e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:40:15.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.304+0000 7fdb8f400640 1 --2- 172.21.15.90:0/401142538 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c849d69d10 0x55c849d7e870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:15.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.304+0000 7fdb8f400640 1 -- 172.21.15.90:0/401142538 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c849d8e950 con 0x55c849d6a9f0 2024-04-02T14:40:15.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.304+0000 7fdb8fe00640 1 --2- 172.21.15.90:0/401142538 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c849d69d10 0x55c849d7e870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:40:15.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.305+0000 7fdb8f400640 1 --2- 172.21.15.90:0/401142538 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c849d6a9f0 0x55c849d63720 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fdb7c0be1b0 tx=0x7fdb7c02c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9718721daf78bd34 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:40:15.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.305+0000 7fdb7be00640 1 -- 172.21.15.90:0/401142538 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb7c04b040 con 0x55c849d6a9f0 2024-04-02T14:40:15.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.305+0000 7fdb7be00640 1 -- 172.21.15.90:0/401142538 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdb7c0c27b0 con 0x55c849d6a9f0 2024-04-02T14:40:15.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.305+0000 7fdb7be00640 1 -- 172.21.15.90:0/401142538 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb7c0be710 con 0x55c849d6a9f0 2024-04-02T14:40:15.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.306+0000 7fdb9987e740 1 -- 172.21.15.90:0/401142538 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c849d6a9f0 msgr2=0x55c849d63720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:15.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.306+0000 7fdb9987e740 1 --2- 172.21.15.90:0/401142538 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c849d6a9f0 0x55c849d63720 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fdb7c0be1b0 tx=0x7fdb7c02c2d0 comp rx=0 tx=0).stop 2024-04-02T14:40:15.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.306+0000 7fdb9987e740 1 -- 172.21.15.90:0/401142538 shutdown_connections 2024-04-02T14:40:15.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.306+0000 7fdb9987e740 1 --2- 172.21.15.90:0/401142538 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c849d70bb0 0x55c849d93be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:15.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.306+0000 7fdb9987e740 1 --2- 172.21.15.90:0/401142538 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c849d6a9f0 0x55c849d63720 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:15.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.306+0000 7fdb9987e740 1 --2- 172.21.15.90:0/401142538 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c849d69d10 0x55c849d7e870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:15.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.306+0000 7fdb9987e740 1 -- 172.21.15.90:0/401142538 >> 172.21.15.90:0/401142538 conn(0x55c849d30a50 msgr2=0x55c849d63010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:40:15.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.307+0000 7fdb9987e740 1 -- 172.21.15.90:0/401142538 shutdown_connections 2024-04-02T14:40:15.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.307+0000 7fdb9987e740 1 -- 172.21.15.90:0/401142538 wait complete. 2024-04-02T14:40:15.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.307+0000 7fdb9987e740 1 -- start start 2024-04-02T14:40:15.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.307+0000 7fdb9987e740 1 Processor -- start 2024-04-02T14:40:15.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.308+0000 7fdb9987e740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c849d69d10 0x55c849d7b250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:15.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.308+0000 7fdb9987e740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c849d6a9f0 0x55c849d6d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:15.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.308+0000 7fdb9987e740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c849d70bb0 0x55c849d6dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:15.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.309+0000 7fdb8f400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c849d6a9f0 0x55c849d6d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:15.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.309+0000 7fdb9987e740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fdb901acb00 con 0x55c849d70bb0 2024-04-02T14:40:15.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.309+0000 7fdb9987e740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fdb901ac430 con 0x55c849d6a9f0 2024-04-02T14:40:15.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.309+0000 7fdb9987e740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fdb901a9670 con 0x55c849d69d10 2024-04-02T14:40:15.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.309+0000 7fdb8f400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c849d6a9f0 0x55c849d6d6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:39366/0 (socket says 172.21.15.90:39366) 2024-04-02T14:40:15.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.309+0000 7fdb8f400640 1 -- 172.21.15.90:0/1870835063 learned_addr learned my addr 172.21.15.90:0/1870835063 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:40:15.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.309+0000 7fdb9987e740 10 client.0 _ll_register_callbacks cb 0x7fdb8800f900 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:40:15.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.309+0000 7fdb94c00640 1 --2- 172.21.15.90:0/1870835063 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c849d70bb0 0x55c849d6dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:15.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.309+0000 7fdb8f400640 1 -- 172.21.15.90:0/1870835063 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c849d69d10 msgr2=0x55c849d7b250 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-02T14:40:15.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.309+0000 7fdb8f400640 1 --2- 172.21.15.90:0/1870835063 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c849d69d10 0x55c849d7b250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:15.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.309+0000 7fdb8f400640 1 -- 172.21.15.90:0/1870835063 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c849d70bb0 msgr2=0x55c849d6dcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:40:15.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.309+0000 7fdb8f400640 1 --2- 172.21.15.90:0/1870835063 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c849d70bb0 0x55c849d6dcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:15.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.309+0000 7fdb8f400640 1 -- 172.21.15.90:0/1870835063 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c849d73f20 con 0x55c849d6a9f0 2024-04-02T14:40:15.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.309+0000 7fdb94c00640 1 --2- 172.21.15.90:0/1870835063 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c849d70bb0 0x55c849d6dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:40:15.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.309+0000 7fdb8f400640 1 --2- 172.21.15.90:0/1870835063 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c849d6a9f0 0x55c849d6d6a0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7fdb7c0096c0 tx=0x7fdb7c0be070 comp rx=0 tx=0).ready entity=mon.2 client_cookie=14282189dcc3a638 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:40:15.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.310+0000 7fdb79600640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:40:15.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.310+0000 7fdb79600640 1 -- 172.21.15.90:0/1870835063 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb7c04b070 con 0x55c849d6a9f0 2024-04-02T14:40:15.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.310+0000 7fdb79600640 1 -- 172.21.15.90:0/1870835063 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7fdb7c0c6be0 con 0x55c849d6a9f0 2024-04-02T14:40:15.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.310+0000 7fdb9987e740 1 -- 172.21.15.90:0/1870835063 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdb7c0c27b0 con 0x55c849d6a9f0 2024-04-02T14:40:15.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.310+0000 7fdb79600640 1 -- 172.21.15.90:0/1870835063 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fdb7c0c6e60 con 0x55c849d6a9f0 2024-04-02T14:40:15.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.310+0000 7fdb9987e740 10 client.27065 fetch_fsmap learned FSMap version 27 2024-04-02T14:40:15.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.310+0000 7fdb9987e740 1 -- 172.21.15.90:0/1870835063 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55c849df2820 con 0x55c849d6a9f0 2024-04-02T14:40:15.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.311+0000 7fdb79600640 1 -- 172.21.15.90:0/1870835063 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdb7c0c27b0 con 0x55c849d6a9f0 2024-04-02T14:40:15.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.311+0000 7fdb79600640 1 -- 172.21.15.90:0/1870835063 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fdb7c0bd900 con 0x55c849d6a9f0 2024-04-02T14:40:15.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.311+0000 7fdb9987e740 10 client.27065 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:40:15.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.311+0000 7fdb9987e740 10 client.27065 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:40:15.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.311+0000 7fdb9987e740 20 client.27065 populate_metadata read hostname 'smithi090' 2024-04-02T14:40:15.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.311+0000 7fdb9987e740 1 --2- 172.21.15.90:0/1870835063 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55c849df2ac0 0x55c849df4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:15.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.311+0000 7fdb9987e740 4 client.27065 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-02T14:40:15.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.311+0000 7fdb9987e740 1 -- 172.21.15.90:0/1870835063 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55c849cdbc30 con 0x55c849df2ac0 2024-04-02T14:40:15.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.313+0000 7fdb8fe00640 1 --2- 172.21.15.90:0/1870835063 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55c849df2ac0 0x55c849df4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:15.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.315+0000 7fdb8fe00640 1 --2- 172.21.15.90:0/1870835063 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55c849df2ac0 0x55c849df4eb0 crc :-1 s=READY pgs=155 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=58949a3325de4ed9 server_cookie=ec07b8c1adca30ad in_seq=0 out_seq=0 2024-04-02T14:40:15.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.315+0000 7fdb79600640 10 client.27065 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:40:15.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.315+0000 7fdb79600640 1 -- 172.21.15.90:0/1870835063 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55c849cdbc30 con 0x55c849df2ac0 2024-04-02T14:40:15.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.315+0000 7fdb79600640 10 client.27065 handle_command_reply: tid=0 2024-04-02T14:40:15.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.316+0000 7fdb9987e740 1 client.27065 shutdown 2024-04-02T14:40:15.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.316+0000 7fdb9987e740 1 -- 172.21.15.90:0/1870835063 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c849d6a9f0 msgr2=0x55c849d6d6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:15.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.316+0000 7fdb9987e740 1 --2- 172.21.15.90:0/1870835063 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c849d6a9f0 0x55c849d6d6a0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7fdb7c0096c0 tx=0x7fdb7c0be070 comp rx=0 tx=0).stop 2024-04-02T14:40:15.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.316+0000 7fdb9987e740 1 -- 172.21.15.90:0/1870835063 shutdown_connections 2024-04-02T14:40:15.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.316+0000 7fdb9987e740 1 --2- 172.21.15.90:0/1870835063 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55c849df2ac0 0x55c849df4eb0 crc :-1 s=READY pgs=155 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:15.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.317+0000 7fdb9987e740 1 --2- 172.21.15.90:0/1870835063 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c849d70bb0 0x55c849d6dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:15.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.317+0000 7fdb9987e740 1 --2- 172.21.15.90:0/1870835063 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c849d6a9f0 0x55c849d6d6a0 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:15.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.317+0000 7fdb9987e740 1 --2- 172.21.15.90:0/1870835063 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c849d69d10 0x55c849d7b250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:15.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.317+0000 7fdb9987e740 1 -- 172.21.15.90:0/1870835063 >> 172.21.15.90:0/1870835063 conn(0x55c849d30a50 msgr2=0x55c849d638a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:40:15.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.317+0000 7fdb79600640 0 client.27065 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:40:15.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.318+0000 7fdb9987e740 1 -- 172.21.15.90:0/1870835063 shutdown_connections 2024-04-02T14:40:15.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.318+0000 7fdb9987e740 1 -- 172.21.15.90:0/1870835063 wait complete. 2024-04-02T14:40:15.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.318+0000 7fdb9987e740 20 client.27065 trim_cache size 0 max 16384 2024-04-02T14:40:15.319 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-02T14:40:15.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.319+0000 7fdb96c00640 1 -- 172.21.15.90:0/841489669 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fdb740a3e60 msgr2=0x7fdb740a6320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:15.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.319+0000 7fdb96c00640 1 --2- 172.21.15.90:0/841489669 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fdb740a3e60 0x7fdb740a6320 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fdb88005fd0 tx=0x7fdb88009b60 comp rx=0 tx=0).stop 2024-04-02T14:40:15.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.319+0000 7fdb96c00640 1 -- 172.21.15.90:0/841489669 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdb90108fe0 msgr2=0x7fdb9019ab60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:15.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.319+0000 7fdb96c00640 1 --2- 172.21.15.90:0/841489669 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdb90108fe0 0x7fdb9019ab60 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fdb7c036040 tx=0x7fdb7c0042e0 comp rx=0 tx=0).stop 2024-04-02T14:40:15.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.320+0000 7fdb96c00640 1 -- 172.21.15.90:0/841489669 shutdown_connections 2024-04-02T14:40:15.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.320+0000 7fdb96c00640 1 --2- 172.21.15.90:0/841489669 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fdb740a3e60 0x7fdb740a6320 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:15.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.320+0000 7fdb96c00640 1 --2- 172.21.15.90:0/841489669 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdb9019eee0 0x7fdb9019f390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:15.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.320+0000 7fdb96c00640 1 --2- 172.21.15.90:0/841489669 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdb90108fe0 0x7fdb9019ab60 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:15.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.320+0000 7fdb96c00640 1 --2- 172.21.15.90:0/841489669 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdb90103f10 0x7fdb9019a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:15.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.320+0000 7fdb96c00640 1 -- 172.21.15.90:0/841489669 >> 172.21.15.90:0/841489669 conn(0x7fdb900ff7b0 msgr2=0x7fdb90100af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:40:15.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.320+0000 7fdb96c00640 1 -- 172.21.15.90:0/841489669 shutdown_connections 2024-04-02T14:40:15.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:15.320+0000 7fdb96c00640 1 -- 172.21.15.90:0/841489669 wait complete. 2024-04-02T14:40:15.687 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:40:15 smithi090 ceph-mon[31674]: pgmap v164: 97 pgs: 97 active+clean; 3.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 26 MiB/s wr, 1.73k op/s 2024-04-02T14:40:15.859 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:40:15 smithi092 ceph-mon[36497]: pgmap v164: 97 pgs: 97 active+clean; 3.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 26 MiB/s wr, 1.73k op/s 2024-04-02T14:40:15.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:40:15 smithi177 ceph-mon[36431]: pgmap v164: 97 pgs: 97 active+clean; 3.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 26 MiB/s wr, 1.73k op/s 2024-04-02T14:40:16.348 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-02T14:40:16.349 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T14:40:16.349 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-02T14:40:16.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.637+0000 7feadba00640 1 -- 172.21.15.90:0/919303376 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fead4106980 msgr2=0x7fead4106d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:16.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.637+0000 7feadba00640 1 --2- 172.21.15.90:0/919303376 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fead4106980 0x7fead4106d60 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7feac8009860 tx=0x7feac802bb70 comp rx=0 tx=0).stop 2024-04-02T14:40:16.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.638+0000 7feadba00640 1 -- 172.21.15.90:0/919303376 shutdown_connections 2024-04-02T14:40:16.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.638+0000 7feadba00640 1 --2- 172.21.15.90:0/919303376 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fead4101330 0x7fead410d840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:16.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.638+0000 7feadba00640 1 --2- 172.21.15.90:0/919303376 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fead4100970 0x7fead4100df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:16.640 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.638+0000 7feadba00640 1 --2- 172.21.15.90:0/919303376 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fead4106980 0x7fead4106d60 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:16.640 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.638+0000 7feadba00640 1 -- 172.21.15.90:0/919303376 >> 172.21.15.90:0/919303376 conn(0x7fead4077b00 msgr2=0x7fead4077f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:40:16.640 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.638+0000 7feadba00640 1 -- 172.21.15.90:0/919303376 shutdown_connections 2024-04-02T14:40:16.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.638+0000 7feadba00640 1 -- 172.21.15.90:0/919303376 wait complete. 2024-04-02T14:40:16.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.641+0000 7feadba00640 1 Processor -- start 2024-04-02T14:40:16.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.641+0000 7feadba00640 1 -- start start 2024-04-02T14:40:16.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.642+0000 7feadba00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fead4100970 0x7fead41961b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:16.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.642+0000 7feadba00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fead4101330 0x7fead41966f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:16.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.642+0000 7feadba00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fead4106980 0x7fead419aa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:16.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.642+0000 7feadba00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fead410fcd0 con 0x7fead4100970 2024-04-02T14:40:16.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.642+0000 7feadba00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fead410fb50 con 0x7fead4101330 2024-04-02T14:40:16.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.642+0000 7feadba00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fead410fe50 con 0x7fead4106980 2024-04-02T14:40:16.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.643+0000 7fead9a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fead4106980 0x7fead419aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:16.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.643+0000 7fead9a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fead4106980 0x7fead419aa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:54042/0 (socket says 172.21.15.90:54042) 2024-04-02T14:40:16.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.643+0000 7fead9a00640 1 -- 172.21.15.90:0/1390556431 learned_addr learned my addr 172.21.15.90:0/1390556431 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:40:16.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.643+0000 7fead9000640 1 --2- 172.21.15.90:0/1390556431 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fead4100970 0x7fead41961b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:16.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.643+0000 7fead3e00640 1 --2- 172.21.15.90:0/1390556431 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fead4101330 0x7fead41966f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:16.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.643+0000 7fead9a00640 1 -- 172.21.15.90:0/1390556431 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fead4101330 msgr2=0x7fead41966f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:16.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.643+0000 7fead9a00640 1 --2- 172.21.15.90:0/1390556431 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fead4101330 0x7fead41966f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:16.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.643+0000 7fead9a00640 1 -- 172.21.15.90:0/1390556431 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fead4100970 msgr2=0x7fead41961b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:16.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.643+0000 7fead9a00640 1 --2- 172.21.15.90:0/1390556431 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fead4100970 0x7fead41961b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:16.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.643+0000 7fead9a00640 1 -- 172.21.15.90:0/1390556431 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fead419b1a0 con 0x7fead4106980 2024-04-02T14:40:16.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.643+0000 7fead9a00640 1 --2- 172.21.15.90:0/1390556431 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fead4106980 0x7fead419aa70 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7feacc009fd0 tx=0x7feacc00c800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:40:16.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.644+0000 7fead9000640 1 --2- 172.21.15.90:0/1390556431 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fead4100970 0x7fead41961b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:40:16.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.644+0000 7fead1600640 1 -- 172.21.15.90:0/1390556431 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feacc004b20 con 0x7fead4106980 2024-04-02T14:40:16.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.644+0000 7fead1600640 1 -- 172.21.15.90:0/1390556431 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feacc003e50 con 0x7fead4106980 2024-04-02T14:40:16.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.644+0000 7fead1600640 1 -- 172.21.15.90:0/1390556431 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feacc00b630 con 0x7fead4106980 2024-04-02T14:40:16.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.644+0000 7feadba00640 1 -- 172.21.15.90:0/1390556431 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fead419b490 con 0x7fead4106980 2024-04-02T14:40:16.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.645+0000 7feadba00640 1 -- 172.21.15.90:0/1390556431 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fead41a2d20 con 0x7fead4106980 2024-04-02T14:40:16.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.651+0000 7fead1600640 1 -- 172.21.15.90:0/1390556431 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7feacc01d020 con 0x7fead4106980 2024-04-02T14:40:16.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.653+0000 7feade834740 1 Processor -- start 2024-04-02T14:40:16.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.654+0000 7feade834740 1 -- start start 2024-04-02T14:40:16.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.656+0000 7feade834740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560fabca4030 0x560fabca2d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:16.657 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.656+0000 7feade834740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560fabca3350 0x560fabca37d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:16.657 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.656+0000 7feade834740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560fabcac6c0 0x560fabcaeab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:16.657 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.656+0000 7feade834740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x560fabc07310 con 0x560fabca3350 2024-04-02T14:40:16.657 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.656+0000 7feade834740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x560fabbaf690 con 0x560fabcac6c0 2024-04-02T14:40:16.657 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.656+0000 7feade834740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x560fabc032c0 con 0x560fabca4030 2024-04-02T14:40:16.658 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.656+0000 7fead1600640 1 --2- 172.21.15.90:0/1390556431 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7feabc0a8280 0x7feabc0aa740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:16.658 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.657+0000 7fead1600640 1 -- 172.21.15.90:0/1390556431 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7feacc0d0770 con 0x7fead4106980 2024-04-02T14:40:16.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.658+0000 7fead9000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560fabca3350 0x560fabca37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:16.660 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.658+0000 7fead9a00640 1 --2- 172.21.15.90:0/1390556431 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7feabc0a8280 0x7feabc0aa740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:16.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.658+0000 7fead3e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560fabca4030 0x560fabca2d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:16.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.660+0000 7fead3e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560fabca4030 0x560fabca2d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:54058/0 (socket says 172.21.15.90:54058) 2024-04-02T14:40:16.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.660+0000 7fead3e00640 1 -- 172.21.15.90:0/927703816 learned_addr learned my addr 172.21.15.90:0/927703816 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:40:16.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.660+0000 7fead3e00640 1 --2- 172.21.15.90:0/927703816 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560fabcac6c0 0x560fabcaeab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:16.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.662+0000 7fead3e00640 1 -- 172.21.15.90:0/927703816 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560fabcac6c0 msgr2=0x560fabcaeab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:40:16.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.662+0000 7fead3e00640 1 --2- 172.21.15.90:0/927703816 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560fabcac6c0 0x560fabcaeab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:16.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.662+0000 7fead3e00640 1 -- 172.21.15.90:0/927703816 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560fabca3350 msgr2=0x560fabca37d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:40:16.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.662+0000 7fead3e00640 1 --2- 172.21.15.90:0/927703816 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560fabca3350 0x560fabca37d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:16.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.662+0000 7fead3e00640 1 -- 172.21.15.90:0/927703816 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560fab9e37e0 con 0x560fabca4030 2024-04-02T14:40:16.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.664+0000 7fead9a00640 1 --2- 172.21.15.90:0/1390556431 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7feabc0a8280 0x7feabc0aa740 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7feacc0cded0 tx=0x7feacc009a00 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:40:16.666 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.665+0000 7fead3e00640 1 --2- 172.21.15.90:0/927703816 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560fabca4030 0x560fabca2d80 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7feac00149c0 tx=0x7feac000c4d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1675f993dcd13979 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:40:16.666 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.665+0000 7feac6000640 1 -- 172.21.15.90:0/927703816 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feac00142d0 con 0x560fabca4030 2024-04-02T14:40:16.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.666+0000 7feac6000640 1 -- 172.21.15.90:0/927703816 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feac000fc40 con 0x560fabca4030 2024-04-02T14:40:16.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.666+0000 7feac6000640 1 -- 172.21.15.90:0/927703816 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feac0012b50 con 0x560fabca4030 2024-04-02T14:40:16.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.667+0000 7feade834740 1 -- 172.21.15.90:0/927703816 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560fabca4030 msgr2=0x560fabca2d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:16.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.667+0000 7feade834740 1 --2- 172.21.15.90:0/927703816 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560fabca4030 0x560fabca2d80 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7feac00149c0 tx=0x7feac000c4d0 comp rx=0 tx=0).stop 2024-04-02T14:40:16.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.668+0000 7feade834740 1 -- 172.21.15.90:0/927703816 shutdown_connections 2024-04-02T14:40:16.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.668+0000 7feade834740 1 --2- 172.21.15.90:0/927703816 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560fabcac6c0 0x560fabcaeab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:16.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.668+0000 7feade834740 1 --2- 172.21.15.90:0/927703816 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560fabca3350 0x560fabca37d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:16.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.668+0000 7feade834740 1 --2- 172.21.15.90:0/927703816 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560fabca4030 0x560fabca2d80 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:16.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.668+0000 7feade834740 1 -- 172.21.15.90:0/927703816 >> 172.21.15.90:0/927703816 conn(0x560fabc37270 msgr2=0x560fabc9dda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:40:16.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.669+0000 7feade834740 1 -- 172.21.15.90:0/927703816 shutdown_connections 2024-04-02T14:40:16.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.669+0000 7feade834740 1 -- 172.21.15.90:0/927703816 wait complete. 2024-04-02T14:40:16.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.670+0000 7feade834740 1 -- start start 2024-04-02T14:40:16.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.670+0000 7feade834740 1 Processor -- start 2024-04-02T14:40:16.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.673+0000 7feade834740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560fabca3350 0x560fabca9690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:16.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.673+0000 7feade834740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560fabca4030 0x560fabca9bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:16.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.673+0000 7feade834740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560fabcac6c0 0x560fabcaa110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:16.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.673+0000 7feade834740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x560fabcaf530 con 0x560fabca3350 2024-04-02T14:40:16.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.673+0000 7feade834740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x560fabbc2a60 con 0x560fabca4030 2024-04-02T14:40:16.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.673+0000 7feade834740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x560fabcaf6b0 con 0x560fabcac6c0 2024-04-02T14:40:16.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.673+0000 7feade834740 10 client.0 _ll_register_callbacks cb 0x560fabbc8030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:40:16.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.674+0000 7fead9000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560fabca4030 0x560fabca9bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:16.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.674+0000 7fead9000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560fabca4030 0x560fabca9bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:39380/0 (socket says 172.21.15.90:39380) 2024-04-02T14:40:16.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.674+0000 7fead9000640 1 -- 172.21.15.90:0/454766842 learned_addr learned my addr 172.21.15.90:0/454766842 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:40:16.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.675+0000 7fead3e00640 1 --2- 172.21.15.90:0/454766842 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560fabca3350 0x560fabca9690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:16.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.675+0000 7fead9000640 1 -- 172.21.15.90:0/454766842 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560fabcac6c0 msgr2=0x560fabcaa110 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-02T14:40:16.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.675+0000 7fead9000640 1 --2- 172.21.15.90:0/454766842 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560fabcac6c0 0x560fabcaa110 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:16.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.675+0000 7fead9000640 1 -- 172.21.15.90:0/454766842 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560fabca3350 msgr2=0x560fabca9690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:40:16.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.675+0000 7fead9000640 1 --2- 172.21.15.90:0/454766842 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560fabca3350 0x560fabca9690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:16.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.676+0000 7fead9000640 1 -- 172.21.15.90:0/454766842 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560faba43f70 con 0x560fabca4030 2024-04-02T14:40:16.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.676+0000 7feabaa00640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:40:16.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.677+0000 7fead9000640 1 --2- 172.21.15.90:0/454766842 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560fabca4030 0x560fabca9bd0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7feac8036000 tx=0x7feac80323d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e56b03f313f83ab4 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:40:16.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.677+0000 7feabaa00640 1 -- 172.21.15.90:0/454766842 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feac8039040 con 0x560fabca4030 2024-04-02T14:40:16.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.678+0000 7feade834740 1 -- 172.21.15.90:0/454766842 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7feac000fc40 con 0x560fabca4030 2024-04-02T14:40:16.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.678+0000 7feabaa00640 1 -- 172.21.15.90:0/454766842 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7feac8004260 con 0x560fabca4030 2024-04-02T14:40:16.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.679+0000 7feabaa00640 1 -- 172.21.15.90:0/454766842 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7feac804c050 con 0x560fabca4030 2024-04-02T14:40:16.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.679+0000 7feade834740 10 client.27077 fetch_fsmap learned FSMap version 27 2024-04-02T14:40:16.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.680+0000 7feade834740 1 -- 172.21.15.90:0/454766842 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x560fab8db170 con 0x560fabca4030 2024-04-02T14:40:16.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.680+0000 7feabaa00640 1 -- 172.21.15.90:0/454766842 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feac000fc40 con 0x560fabca4030 2024-04-02T14:40:16.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.681+0000 7feabaa00640 1 -- 172.21.15.90:0/454766842 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7feac8047030 con 0x560fabca4030 2024-04-02T14:40:16.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.682+0000 7feade834740 10 client.27077 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:40:16.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.682+0000 7feade834740 10 client.27077 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:40:16.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.682+0000 7feade834740 20 client.27077 populate_metadata read hostname 'smithi090' 2024-04-02T14:40:16.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.682+0000 7feade834740 1 --2- 172.21.15.90:0/454766842 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x560fabcb3ac0 0x560fabcb5f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:16.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.682+0000 7feade834740 4 client.27077 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:40:16.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.682+0000 7feade834740 1 -- 172.21.15.90:0/454766842 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560fabc14850 con 0x560fabcb3ac0 2024-04-02T14:40:16.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.683+0000 7fead3e00640 1 --2- 172.21.15.90:0/454766842 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x560fabcb3ac0 0x560fabcb5f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:16.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.685+0000 7feabaa00640 10 client.27077 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:40:16.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.685+0000 7fead3e00640 1 --2- 172.21.15.90:0/454766842 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x560fabcb3ac0 0x560fabcb5f30 crc :-1 s=READY pgs=156 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=37d08b0dd9711899 server_cookie=3fffe643f2ce209a in_seq=0 out_seq=0 2024-04-02T14:40:16.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.686+0000 7feabaa00640 1 -- 172.21.15.90:0/454766842 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x560fabc14850 con 0x560fabcb3ac0 2024-04-02T14:40:16.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.686+0000 7feabaa00640 10 client.27077 handle_command_reply: tid=0 2024-04-02T14:40:16.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.687+0000 7feade834740 1 client.27077 shutdown 2024-04-02T14:40:16.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.688+0000 7feade834740 1 -- 172.21.15.90:0/454766842 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560fabca4030 msgr2=0x560fabca9bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:16.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.688+0000 7feade834740 1 --2- 172.21.15.90:0/454766842 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560fabca4030 0x560fabca9bd0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7feac8036000 tx=0x7feac80323d0 comp rx=0 tx=0).stop 2024-04-02T14:40:16.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.688+0000 7feade834740 1 -- 172.21.15.90:0/454766842 shutdown_connections 2024-04-02T14:40:16.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.689+0000 7feade834740 1 --2- 172.21.15.90:0/454766842 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x560fabcb3ac0 0x560fabcb5f30 crc :-1 s=READY pgs=156 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:16.690 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.689+0000 7feade834740 1 --2- 172.21.15.90:0/454766842 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560fabcac6c0 0x560fabcaa110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:16.690 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.689+0000 7feade834740 1 --2- 172.21.15.90:0/454766842 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560fabca4030 0x560fabca9bd0 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:16.690 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.689+0000 7feade834740 1 --2- 172.21.15.90:0/454766842 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560fabca3350 0x560fabca9690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:16.690 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.689+0000 7feade834740 1 -- 172.21.15.90:0/454766842 >> 172.21.15.90:0/454766842 conn(0x560fabc37270 msgr2=0x560fabc9faf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:40:16.690 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.689+0000 7feabaa00640 0 client.27077 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:40:16.691 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.689+0000 7feade834740 1 -- 172.21.15.90:0/454766842 shutdown_connections 2024-04-02T14:40:16.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.689+0000 7feade834740 1 -- 172.21.15.90:0/454766842 wait complete. 2024-04-02T14:40:16.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.690+0000 7feade834740 20 client.27077 trim_cache size 0 max 16384 2024-04-02T14:40:16.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.723+0000 7feade834740 1 Processor -- start 2024-04-02T14:40:16.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.723+0000 7feade834740 1 -- start start 2024-04-02T14:40:16.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.723+0000 7feade834740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560fabca3350 0x560fabcabdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:16.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.723+0000 7feade834740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560fabca4030 0x560fabcc0970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:16.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.723+0000 7feade834740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560fabcac6c0 0x560fabc9f890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:16.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.723+0000 7feade834740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fead41ab6e0 con 0x560fabca3350 2024-04-02T14:40:16.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.723+0000 7feade834740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fead41ad220 con 0x560fabca4030 2024-04-02T14:40:16.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.723+0000 7feade834740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fead41abdb0 con 0x560fabcac6c0 2024-04-02T14:40:16.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.724+0000 7fead3e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560fabca3350 0x560fabcabdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:16.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.724+0000 7fead3e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560fabcac6c0 0x560fabc9f890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:16.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.724+0000 7fead3e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560fabcac6c0 0x560fabc9f890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:54086/0 (socket says 172.21.15.90:54086) 2024-04-02T14:40:16.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.724+0000 7fead3e00640 1 -- 172.21.15.90:0/4042408240 learned_addr learned my addr 172.21.15.90:0/4042408240 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:40:16.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.724+0000 7fead9000640 1 --2- 172.21.15.90:0/4042408240 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560fabca4030 0x560fabcc0970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:16.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.724+0000 7fead9000640 1 -- 172.21.15.90:0/4042408240 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560fabcac6c0 msgr2=0x560fabc9f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:40:16.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.724+0000 7fead9000640 1 --2- 172.21.15.90:0/4042408240 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560fabcac6c0 0x560fabc9f890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:16.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.724+0000 7fead9000640 1 -- 172.21.15.90:0/4042408240 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560fabca3350 msgr2=0x560fabcabdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:40:16.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.725+0000 7fead9000640 1 --2- 172.21.15.90:0/4042408240 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560fabca3350 0x560fabcabdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:16.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.725+0000 7fead9000640 1 -- 172.21.15.90:0/4042408240 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560fabca2380 con 0x560fabca4030 2024-04-02T14:40:16.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.725+0000 7fead3e00640 1 --2- 172.21.15.90:0/4042408240 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560fabca3350 0x560fabcabdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:40:16.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.726+0000 7fead9000640 1 --2- 172.21.15.90:0/4042408240 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560fabca4030 0x560fabcc0970 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7feac8005660 tx=0x7feac80094b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=875a97147b817fcf server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:40:16.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.726+0000 7feac5600640 1 -- 172.21.15.90:0/4042408240 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feac8054020 con 0x560fabca4030 2024-04-02T14:40:16.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.726+0000 7feac5600640 1 -- 172.21.15.90:0/4042408240 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feac802c630 con 0x560fabca4030 2024-04-02T14:40:16.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.726+0000 7feac5600640 1 -- 172.21.15.90:0/4042408240 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feac8039040 con 0x560fabca4030 2024-04-02T14:40:16.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.727+0000 7feade834740 1 -- 172.21.15.90:0/4042408240 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560fabca4030 msgr2=0x560fabcc0970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:16.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.727+0000 7feade834740 1 --2- 172.21.15.90:0/4042408240 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560fabca4030 0x560fabcc0970 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7feac8005660 tx=0x7feac80094b0 comp rx=0 tx=0).stop 2024-04-02T14:40:16.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.728+0000 7feade834740 1 -- 172.21.15.90:0/4042408240 shutdown_connections 2024-04-02T14:40:16.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.728+0000 7feade834740 1 --2- 172.21.15.90:0/4042408240 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560fabcac6c0 0x560fabc9f890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:16.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.728+0000 7feade834740 1 --2- 172.21.15.90:0/4042408240 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560fabca4030 0x560fabcc0970 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:16.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.728+0000 7feade834740 1 --2- 172.21.15.90:0/4042408240 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560fabca3350 0x560fabcabdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:16.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.728+0000 7feade834740 1 -- 172.21.15.90:0/4042408240 >> 172.21.15.90:0/4042408240 conn(0x560fabc37270 msgr2=0x560fabc9f180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:40:16.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.729+0000 7feade834740 1 -- 172.21.15.90:0/4042408240 shutdown_connections 2024-04-02T14:40:16.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.729+0000 7feade834740 1 -- 172.21.15.90:0/4042408240 wait complete. 2024-04-02T14:40:16.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.730+0000 7feade834740 1 -- start start 2024-04-02T14:40:16.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.730+0000 7feade834740 1 Processor -- start 2024-04-02T14:40:16.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.730+0000 7feade834740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560fabca3350 0x560fabcbb760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:16.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.730+0000 7feade834740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560fabca4030 0x560fabcbbca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:16.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.730+0000 7feade834740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560fabcac6c0 0x560fabca5ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:16.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.731+0000 7feade834740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fead41acb50 con 0x560fabcac6c0 2024-04-02T14:40:16.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.731+0000 7feade834740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fead41ac480 con 0x560fabca4030 2024-04-02T14:40:16.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.731+0000 7feade834740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fead41a96c0 con 0x560fabca3350 2024-04-02T14:40:16.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.731+0000 7feade834740 10 client.0 _ll_register_callbacks cb 0x7feac8037070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:40:16.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.731+0000 7fead3e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560fabca3350 0x560fabcbb760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:16.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.731+0000 7fead3e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560fabca3350 0x560fabcbb760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:54102/0 (socket says 172.21.15.90:54102) 2024-04-02T14:40:16.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.731+0000 7fead3e00640 1 -- 172.21.15.90:0/1972866228 learned_addr learned my addr 172.21.15.90:0/1972866228 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:40:16.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.731+0000 7fead9000640 1 --2- 172.21.15.90:0/1972866228 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560fabca4030 0x560fabcbbca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:16.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.731+0000 7fead3e00640 1 --2- 172.21.15.90:0/1972866228 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560fabcac6c0 0x560fabca5ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:16.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.731+0000 7fead9000640 1 -- 172.21.15.90:0/1972866228 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560fabca3350 msgr2=0x560fabcbb760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:40:16.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.731+0000 7fead9000640 1 --2- 172.21.15.90:0/1972866228 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560fabca3350 0x560fabcbb760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:16.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.731+0000 7fead9000640 1 -- 172.21.15.90:0/1972866228 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560fabcac6c0 msgr2=0x560fabca5ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:40:16.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.732+0000 7fead9000640 1 --2- 172.21.15.90:0/1972866228 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560fabcac6c0 0x560fabca5ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:16.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.732+0000 7fead9000640 1 -- 172.21.15.90:0/1972866228 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560fabca61f0 con 0x560fabca4030 2024-04-02T14:40:16.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.732+0000 7fead9000640 1 --2- 172.21.15.90:0/1972866228 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560fabca4030 0x560fabcbbca0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7feac8037040 tx=0x7feac8032340 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6ba38b634d9baa45 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:40:16.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.732+0000 7feabaa00640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:40:16.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.732+0000 7feabaa00640 1 -- 172.21.15.90:0/1972866228 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feac8054030 con 0x560fabca4030 2024-04-02T14:40:16.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.732+0000 7feabaa00640 1 -- 172.21.15.90:0/1972866228 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7feac8034440 con 0x560fabca4030 2024-04-02T14:40:16.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.733+0000 7feade834740 1 -- 172.21.15.90:0/1972866228 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7feac802c630 con 0x560fabca4030 2024-04-02T14:40:16.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.733+0000 7feabaa00640 1 -- 172.21.15.90:0/1972866228 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7feac8039040 con 0x560fabca4030 2024-04-02T14:40:16.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.733+0000 7feade834740 10 client.27083 fetch_fsmap learned FSMap version 27 2024-04-02T14:40:16.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.733+0000 7feade834740 1 -- 172.21.15.90:0/1972866228 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x560fabd354a0 con 0x560fabca4030 2024-04-02T14:40:16.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.733+0000 7feabaa00640 1 -- 172.21.15.90:0/1972866228 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feac802c630 con 0x560fabca4030 2024-04-02T14:40:16.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.733+0000 7feabaa00640 1 -- 172.21.15.90:0/1972866228 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7feac8047040 con 0x560fabca4030 2024-04-02T14:40:16.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.734+0000 7feade834740 10 client.27083 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:40:16.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.734+0000 7feade834740 10 client.27083 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:40:16.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.734+0000 7feade834740 20 client.27083 populate_metadata read hostname 'smithi090' 2024-04-02T14:40:16.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.734+0000 7feade834740 1 --2- 172.21.15.90:0/1972866228 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x560fabd35ab0 0x560fabd37ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:16.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.734+0000 7feade834740 4 client.27083 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-02T14:40:16.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.734+0000 7feade834740 1 -- 172.21.15.90:0/1972866228 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x560fabc0eba0 con 0x560fabd35ab0 2024-04-02T14:40:16.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.734+0000 7fead3e00640 1 --2- 172.21.15.90:0/1972866228 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x560fabd35ab0 0x560fabd37ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:16.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.736+0000 7fead3e00640 1 --2- 172.21.15.90:0/1972866228 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x560fabd35ab0 0x560fabd37ed0 crc :-1 s=READY pgs=157 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=14434dde4c4db441 server_cookie=5a4f4f75b097bd58 in_seq=0 out_seq=0 2024-04-02T14:40:16.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.736+0000 7feabaa00640 10 client.27083 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:40:16.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.737+0000 7feabaa00640 1 -- 172.21.15.90:0/1972866228 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x560fabc0eba0 con 0x560fabd35ab0 2024-04-02T14:40:16.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.737+0000 7feabaa00640 10 client.27083 handle_command_reply: tid=0 2024-04-02T14:40:16.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.737+0000 7feade834740 1 client.27083 shutdown 2024-04-02T14:40:16.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.738+0000 7feade834740 1 -- 172.21.15.90:0/1972866228 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560fabca4030 msgr2=0x560fabcbbca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:16.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.738+0000 7feade834740 1 --2- 172.21.15.90:0/1972866228 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560fabca4030 0x560fabcbbca0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7feac8037040 tx=0x7feac8032340 comp rx=0 tx=0).stop 2024-04-02T14:40:16.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.738+0000 7feade834740 1 -- 172.21.15.90:0/1972866228 shutdown_connections 2024-04-02T14:40:16.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.738+0000 7feade834740 1 --2- 172.21.15.90:0/1972866228 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x560fabd35ab0 0x560fabd37ed0 crc :-1 s=READY pgs=157 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:16.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.738+0000 7feade834740 1 --2- 172.21.15.90:0/1972866228 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560fabcac6c0 0x560fabca5ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:16.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.738+0000 7feade834740 1 --2- 172.21.15.90:0/1972866228 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560fabca4030 0x560fabcbbca0 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:16.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.738+0000 7feade834740 1 --2- 172.21.15.90:0/1972866228 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560fabca3350 0x560fabcbb760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:16.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.738+0000 7feabaa00640 0 client.27083 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:40:16.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.738+0000 7feade834740 1 -- 172.21.15.90:0/1972866228 >> 172.21.15.90:0/1972866228 conn(0x560fabc37270 msgr2=0x560fabca9e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:40:16.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.738+0000 7feade834740 1 -- 172.21.15.90:0/1972866228 shutdown_connections 2024-04-02T14:40:16.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.739+0000 7feade834740 1 -- 172.21.15.90:0/1972866228 wait complete. 2024-04-02T14:40:16.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.739+0000 7feade834740 20 client.27083 trim_cache size 0 max 16384 2024-04-02T14:40:16.740 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-02T14:40:16.740 INFO:teuthology.orchestra.run.smithi090.stdout: "return_code": 0, 2024-04-02T14:40:16.740 INFO:teuthology.orchestra.run.smithi090.stdout: "scrub_tag": "81a61248-ad60-4fb8-a6a1-6f88274484e0", 2024-04-02T14:40:16.740 INFO:teuthology.orchestra.run.smithi090.stdout: "mode": "asynchronous" 2024-04-02T14:40:16.741 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-02T14:40:16.741 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.739+0000 7feadba00640 1 -- 172.21.15.90:0/1390556431 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7feabc0a8280 msgr2=0x7feabc0aa740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:16.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.739+0000 7feadba00640 1 --2- 172.21.15.90:0/1390556431 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7feabc0a8280 0x7feabc0aa740 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7feacc0cded0 tx=0x7feacc009a00 comp rx=0 tx=0).stop 2024-04-02T14:40:16.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.739+0000 7feadba00640 1 -- 172.21.15.90:0/1390556431 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fead4106980 msgr2=0x7fead419aa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:16.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.739+0000 7feadba00640 1 --2- 172.21.15.90:0/1390556431 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fead4106980 0x7fead419aa70 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7feacc009fd0 tx=0x7feacc00c800 comp rx=0 tx=0).stop 2024-04-02T14:40:16.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.740+0000 7feadba00640 1 -- 172.21.15.90:0/1390556431 shutdown_connections 2024-04-02T14:40:16.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.740+0000 7feadba00640 1 --2- 172.21.15.90:0/1390556431 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7feabc0a8280 0x7feabc0aa740 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:16.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.740+0000 7feadba00640 1 --2- 172.21.15.90:0/1390556431 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fead4106980 0x7fead419aa70 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:16.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.740+0000 7feadba00640 1 --2- 172.21.15.90:0/1390556431 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fead4101330 0x7fead41966f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:16.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.740+0000 7feadba00640 1 --2- 172.21.15.90:0/1390556431 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fead4100970 0x7fead41961b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:16.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.740+0000 7feadba00640 1 -- 172.21.15.90:0/1390556431 >> 172.21.15.90:0/1390556431 conn(0x7fead4077b00 msgr2=0x7fead4103090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:40:16.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.741+0000 7feadba00640 1 -- 172.21.15.90:0/1390556431 shutdown_connections 2024-04-02T14:40:16.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:16.741+0000 7feadba00640 1 -- 172.21.15.90:0/1390556431 wait complete. 2024-04-02T14:40:16.768 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-02T14:40:16.768 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-02T14:40:16.831 INFO:teuthology.orchestra.run.smithi090.stderr:progress: 285.0 s, 2781.9 tps, lat 35.828 ms stddev 40.856 2024-04-02T14:40:17.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.077+0000 7fb6ca600640 1 -- 172.21.15.90:0/3376651516 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb6c4107aa0 msgr2=0x7fb6c4107e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:17.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.077+0000 7fb6ca600640 1 --2- 172.21.15.90:0/3376651516 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb6c4107aa0 0x7fb6c4107e80 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fb6b4009cd0 tx=0x7fb6b402c0e0 comp rx=0 tx=0).stop 2024-04-02T14:40:17.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.078+0000 7fb6ca600640 1 -- 172.21.15.90:0/3376651516 shutdown_connections 2024-04-02T14:40:17.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.078+0000 7fb6ca600640 1 --2- 172.21.15.90:0/3376651516 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb6c4102450 0x7fb6c410e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:17.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.078+0000 7fb6ca600640 1 --2- 172.21.15.90:0/3376651516 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb6c4101a90 0x7fb6c4101f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:17.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.078+0000 7fb6ca600640 1 --2- 172.21.15.90:0/3376651516 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb6c4107aa0 0x7fb6c4107e80 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:17.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.078+0000 7fb6ca600640 1 -- 172.21.15.90:0/3376651516 >> 172.21.15.90:0/3376651516 conn(0x7fb6c4078f00 msgr2=0x7fb6c4079310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:40:17.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.078+0000 7fb6ca600640 1 -- 172.21.15.90:0/3376651516 shutdown_connections 2024-04-02T14:40:17.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.078+0000 7fb6ca600640 1 -- 172.21.15.90:0/3376651516 wait complete. 2024-04-02T14:40:17.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.080+0000 7fb6ca600640 1 Processor -- start 2024-04-02T14:40:17.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.081+0000 7fb6ca600640 1 -- start start 2024-04-02T14:40:17.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.081+0000 7fb6ca600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb6c4101a90 0x7fb6c419f090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:17.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.081+0000 7fb6ca600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb6c4102450 0x7fb6c419f5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:17.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.081+0000 7fb6ca600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb6c4107aa0 0x7fb6c410cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:17.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.081+0000 7fb6ca600640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fb6c4111e80 con 0x7fb6c4101a90 2024-04-02T14:40:17.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.081+0000 7fb6ca600640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fb6c4111d00 con 0x7fb6c4102450 2024-04-02T14:40:17.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.081+0000 7fb6ca600640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fb6c4112000 con 0x7fb6c4107aa0 2024-04-02T14:40:17.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.081+0000 7fb6c3e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb6c4107aa0 0x7fb6c410cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:17.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.081+0000 7fb6c3e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb6c4107aa0 0x7fb6c410cbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:54122/0 (socket says 172.21.15.90:54122) 2024-04-02T14:40:17.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.082+0000 7fb6c3e00640 1 -- 172.21.15.90:0/3575788135 learned_addr learned my addr 172.21.15.90:0/3575788135 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:40:17.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.082+0000 7fb6c3400640 1 --2- 172.21.15.90:0/3575788135 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb6c4101a90 0x7fb6c419f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:17.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.082+0000 7fb6baa00640 1 --2- 172.21.15.90:0/3575788135 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb6c4102450 0x7fb6c419f5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:17.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.083+0000 7fb6c3e00640 1 -- 172.21.15.90:0/3575788135 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb6c4102450 msgr2=0x7fb6c419f5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:17.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.083+0000 7fb6c3e00640 1 --2- 172.21.15.90:0/3575788135 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb6c4102450 0x7fb6c419f5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:17.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.083+0000 7fb6c3e00640 1 -- 172.21.15.90:0/3575788135 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb6c4101a90 msgr2=0x7fb6c419f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:17.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.083+0000 7fb6c3e00640 1 --2- 172.21.15.90:0/3575788135 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb6c4101a90 0x7fb6c419f090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:17.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.083+0000 7fb6c3e00640 1 -- 172.21.15.90:0/3575788135 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6c410d470 con 0x7fb6c4107aa0 2024-04-02T14:40:17.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.083+0000 7fb6c3e00640 1 --2- 172.21.15.90:0/3575788135 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb6c4107aa0 0x7fb6c410cbb0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fb6bc00e970 tx=0x7fb6bc00ed70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:40:17.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.084+0000 7fb6c0c00640 1 -- 172.21.15.90:0/3575788135 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6bc00ccc0 con 0x7fb6c4107aa0 2024-04-02T14:40:17.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.084+0000 7fb6c0c00640 1 -- 172.21.15.90:0/3575788135 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb6bc00ce60 con 0x7fb6c4107aa0 2024-04-02T14:40:17.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.084+0000 7fb6ca600640 1 -- 172.21.15.90:0/3575788135 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6c410d760 con 0x7fb6c4107aa0 2024-04-02T14:40:17.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.084+0000 7fb6ca600640 1 -- 172.21.15.90:0/3575788135 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6c419ac40 con 0x7fb6c4107aa0 2024-04-02T14:40:17.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.085+0000 7fb6c0c00640 1 -- 172.21.15.90:0/3575788135 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6bc010490 con 0x7fb6c4107aa0 2024-04-02T14:40:17.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.089+0000 7fb6c0c00640 1 -- 172.21.15.90:0/3575788135 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fb6bc0046c0 con 0x7fb6c4107aa0 2024-04-02T14:40:17.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.089+0000 7fb6cd241740 1 Processor -- start 2024-04-02T14:40:17.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.090+0000 7fb6cd241740 1 -- start start 2024-04-02T14:40:17.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.091+0000 7fb6cd241740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56440975b7b0 0x56440975bb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:17.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.091+0000 7fb6cd241740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56440975c0d0 0x564409764770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:17.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.091+0000 7fb6cd241740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564409764cb0 0x5644097670a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:17.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.091+0000 7fb6cd241740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5644096bf770 con 0x564409764cb0 2024-04-02T14:40:17.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.091+0000 7fb6cd241740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x564409667700 con 0x56440975b7b0 2024-04-02T14:40:17.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.091+0000 7fb6cd241740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5644096bb720 con 0x56440975c0d0 2024-04-02T14:40:17.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.091+0000 7fb6c0c00640 1 --2- 172.21.15.90:0/3575788135 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fb6a80a3e70 0x7fb6a80a6330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:17.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.091+0000 7fb6c3400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56440975c0d0 0x564409764770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:17.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.091+0000 7fb6c3400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56440975c0d0 0x564409764770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:54138/0 (socket says 172.21.15.90:54138) 2024-04-02T14:40:17.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.091+0000 7fb6c3400640 1 -- 172.21.15.90:0/211350629 learned_addr learned my addr 172.21.15.90:0/211350629 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:40:17.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.091+0000 7fb6baa00640 1 --2- 172.21.15.90:0/211350629 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564409764cb0 0x5644097670a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:17.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.091+0000 7fb6baa00640 1 --2- 172.21.15.90:0/211350629 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56440975b7b0 0x56440975bb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:17.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.091+0000 7fb6c0c00640 1 -- 172.21.15.90:0/3575788135 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7fb6bc0d0a50 con 0x7fb6c4107aa0 2024-04-02T14:40:17.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.092+0000 7fb6c3400640 1 -- 172.21.15.90:0/211350629 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56440975b7b0 msgr2=0x56440975bb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:40:17.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.092+0000 7fb6c3400640 1 --2- 172.21.15.90:0/211350629 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56440975b7b0 0x56440975bb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:17.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.092+0000 7fb6c3400640 1 -- 172.21.15.90:0/211350629 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564409764cb0 msgr2=0x5644097670a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:40:17.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.092+0000 7fb6c3400640 1 --2- 172.21.15.90:0/211350629 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564409764cb0 0x5644097670a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:17.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.092+0000 7fb6c3400640 1 -- 172.21.15.90:0/211350629 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56440949d7d0 con 0x56440975c0d0 2024-04-02T14:40:17.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.092+0000 7fb6baa00640 1 --2- 172.21.15.90:0/211350629 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56440975b7b0 0x56440975bb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:40:17.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.093+0000 7fb6c3e00640 1 --2- 172.21.15.90:0/3575788135 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fb6a80a3e70 0x7fb6a80a6330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:17.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.095+0000 7fb6c3e00640 1 --2- 172.21.15.90:0/3575788135 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fb6a80a3e70 0x7fb6a80a6330 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fb6bc018000 tx=0x7fb6bc009790 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:40:17.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.096+0000 7fb6c3400640 1 --2- 172.21.15.90:0/211350629 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56440975c0d0 0x564409764770 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fb6b4004af0 tx=0x7fb6b40323b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=70648138eb2f4102 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:40:17.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.098+0000 7fb6b8c00640 1 -- 172.21.15.90:0/211350629 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6b4043070 con 0x56440975c0d0 2024-04-02T14:40:17.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.099+0000 7fb6b8c00640 1 -- 172.21.15.90:0/211350629 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb6b4032c70 con 0x56440975c0d0 2024-04-02T14:40:17.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.099+0000 7fb6b8c00640 1 -- 172.21.15.90:0/211350629 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6b4039040 con 0x56440975c0d0 2024-04-02T14:40:17.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.101+0000 7fb6cd241740 1 -- 172.21.15.90:0/211350629 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56440975c0d0 msgr2=0x564409764770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:17.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.101+0000 7fb6cd241740 1 --2- 172.21.15.90:0/211350629 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56440975c0d0 0x564409764770 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fb6b4004af0 tx=0x7fb6b40323b0 comp rx=0 tx=0).stop 2024-04-02T14:40:17.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.102+0000 7fb6cd241740 1 -- 172.21.15.90:0/211350629 shutdown_connections 2024-04-02T14:40:17.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.102+0000 7fb6cd241740 1 --2- 172.21.15.90:0/211350629 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564409764cb0 0x5644097670a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:17.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.102+0000 7fb6cd241740 1 --2- 172.21.15.90:0/211350629 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56440975c0d0 0x564409764770 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:17.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.102+0000 7fb6cd241740 1 --2- 172.21.15.90:0/211350629 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56440975b7b0 0x56440975bb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:17.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.102+0000 7fb6cd241740 1 -- 172.21.15.90:0/211350629 >> 172.21.15.90:0/211350629 conn(0x564409755ee0 msgr2=0x5644097562c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:40:17.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.102+0000 7fb6cd241740 1 -- 172.21.15.90:0/211350629 shutdown_connections 2024-04-02T14:40:17.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.103+0000 7fb6cd241740 1 -- 172.21.15.90:0/211350629 wait complete. 2024-04-02T14:40:17.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.104+0000 7fb6cd241740 1 -- start start 2024-04-02T14:40:17.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.104+0000 7fb6cd241740 1 Processor -- start 2024-04-02T14:40:17.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.108+0000 7fb6cd241740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56440975b7b0 0x564409761d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:17.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.109+0000 7fb6cd241740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56440975c0d0 0x564409762260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:17.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.109+0000 7fb6cd241740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564409764cb0 0x5644097627a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:17.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.109+0000 7fb6cd241740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x564409767a40 con 0x56440975b7b0 2024-04-02T14:40:17.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.109+0000 7fb6cd241740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x56440967aac0 con 0x56440975c0d0 2024-04-02T14:40:17.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.109+0000 7fb6cd241740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x564409767bc0 con 0x564409764cb0 2024-04-02T14:40:17.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.109+0000 7fb6cd241740 10 client.0 _ll_register_callbacks cb 0x56440967d180 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:40:17.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.110+0000 7fb6c3400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56440975c0d0 0x564409762260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:17.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.110+0000 7fb6c3400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56440975c0d0 0x564409762260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:39432/0 (socket says 172.21.15.90:39432) 2024-04-02T14:40:17.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.110+0000 7fb6c3400640 1 -- 172.21.15.90:0/2933597120 learned_addr learned my addr 172.21.15.90:0/2933597120 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:40:17.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.111+0000 7fb6baa00640 1 --2- 172.21.15.90:0/2933597120 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564409764cb0 0x5644097627a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:17.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.112+0000 7fb6c3400640 1 -- 172.21.15.90:0/2933597120 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564409764cb0 msgr2=0x5644097627a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:40:17.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.112+0000 7fb6c3400640 1 --2- 172.21.15.90:0/2933597120 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564409764cb0 0x5644097627a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:17.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.112+0000 7fb6c3400640 1 -- 172.21.15.90:0/2933597120 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56440975b7b0 msgr2=0x564409761d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:40:17.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.112+0000 7fb6c3400640 1 --2- 172.21.15.90:0/2933597120 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56440975b7b0 0x564409761d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:17.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.112+0000 7fb6c3400640 1 -- 172.21.15.90:0/2933597120 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5644093d33f0 con 0x56440975c0d0 2024-04-02T14:40:17.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.113+0000 7fb6c3400640 1 --2- 172.21.15.90:0/2933597120 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56440975c0d0 0x564409762260 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fb6b4037040 tx=0x7fb6b4005bd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c692a28b6f93bb00 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:40:17.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.113+0000 7fb6b1600640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:40:17.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.115+0000 7fb6b1600640 1 -- 172.21.15.90:0/2933597120 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6b404c070 con 0x56440975c0d0 2024-04-02T14:40:17.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.115+0000 7fb6b1600640 1 -- 172.21.15.90:0/2933597120 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7fb6b40329a0 con 0x56440975c0d0 2024-04-02T14:40:17.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.116+0000 7fb6cd241740 1 -- 172.21.15.90:0/2933597120 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb6b4032c70 con 0x56440975c0d0 2024-04-02T14:40:17.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.117+0000 7fb6b1600640 1 -- 172.21.15.90:0/2933597120 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fb6b4056050 con 0x56440975c0d0 2024-04-02T14:40:17.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.117+0000 7fb6cd241740 10 client.27092 fetch_fsmap learned FSMap version 27 2024-04-02T14:40:17.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.117+0000 7fb6cd241740 1 -- 172.21.15.90:0/2933597120 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5644093930f0 con 0x56440975c0d0 2024-04-02T14:40:17.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.118+0000 7fb6b1600640 1 -- 172.21.15.90:0/2933597120 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb6b4032c70 con 0x56440975c0d0 2024-04-02T14:40:17.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.118+0000 7fb6b1600640 1 -- 172.21.15.90:0/2933597120 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fb6b4048780 con 0x56440975c0d0 2024-04-02T14:40:17.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.118+0000 7fb6cd241740 10 client.27092 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:40:17.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.118+0000 7fb6cd241740 10 client.27092 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:40:17.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.118+0000 7fb6cd241740 20 client.27092 populate_metadata read hostname 'smithi090' 2024-04-02T14:40:17.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.118+0000 7fb6cd241740 1 --2- 172.21.15.90:0/2933597120 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x564409774b20 0x564409776f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:17.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.118+0000 7fb6cd241740 4 client.27092 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:40:17.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.118+0000 7fb6cd241740 1 -- 172.21.15.90:0/2933597120 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564409660630 con 0x564409774b20 2024-04-02T14:40:17.120 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.119+0000 7fb6baa00640 1 --2- 172.21.15.90:0/2933597120 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x564409774b20 0x564409776f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:17.120 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.119+0000 7fb6baa00640 1 --2- 172.21.15.90:0/2933597120 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x564409774b20 0x564409776f10 crc :-1 s=READY pgs=158 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e66dbc81c332806e server_cookie=542a58faffc7867c in_seq=0 out_seq=0 2024-04-02T14:40:17.120 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.119+0000 7fb6b1600640 10 client.27092 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:40:17.120 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.120+0000 7fb6b1600640 1 -- 172.21.15.90:0/2933597120 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x564409660630 con 0x564409774b20 2024-04-02T14:40:17.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.120+0000 7fb6b1600640 10 client.27092 handle_command_reply: tid=0 2024-04-02T14:40:17.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.120+0000 7fb6cd241740 1 client.27092 shutdown 2024-04-02T14:40:17.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.121+0000 7fb6cd241740 1 -- 172.21.15.90:0/2933597120 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56440975c0d0 msgr2=0x564409762260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:17.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.121+0000 7fb6cd241740 1 --2- 172.21.15.90:0/2933597120 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56440975c0d0 0x564409762260 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fb6b4037040 tx=0x7fb6b4005bd0 comp rx=0 tx=0).stop 2024-04-02T14:40:17.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.121+0000 7fb6cd241740 1 -- 172.21.15.90:0/2933597120 shutdown_connections 2024-04-02T14:40:17.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.121+0000 7fb6cd241740 1 --2- 172.21.15.90:0/2933597120 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x564409774b20 0x564409776f10 crc :-1 s=READY pgs=158 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:17.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.121+0000 7fb6cd241740 1 --2- 172.21.15.90:0/2933597120 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564409764cb0 0x5644097627a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:17.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.121+0000 7fb6cd241740 1 --2- 172.21.15.90:0/2933597120 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56440975c0d0 0x564409762260 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:17.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.121+0000 7fb6cd241740 1 --2- 172.21.15.90:0/2933597120 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56440975b7b0 0x564409761d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:17.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.121+0000 7fb6cd241740 1 -- 172.21.15.90:0/2933597120 >> 172.21.15.90:0/2933597120 conn(0x564409755ee0 msgr2=0x564409757c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:40:17.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.122+0000 7fb6b1600640 0 client.27092 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:40:17.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.122+0000 7fb6cd241740 1 -- 172.21.15.90:0/2933597120 shutdown_connections 2024-04-02T14:40:17.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.122+0000 7fb6cd241740 1 -- 172.21.15.90:0/2933597120 wait complete. 2024-04-02T14:40:17.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.123+0000 7fb6cd241740 20 client.27092 trim_cache size 0 max 16384 2024-04-02T14:40:17.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.162+0000 7fb6cd241740 1 Processor -- start 2024-04-02T14:40:17.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.163+0000 7fb6cd241740 1 -- start start 2024-04-02T14:40:17.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.163+0000 7fb6cd241740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56440975b7b0 0x564409770990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:17.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.163+0000 7fb6cd241740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56440975c0d0 0x5644097625c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:17.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.163+0000 7fb6cd241740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564409764cb0 0x564409757940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:17.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.163+0000 7fb6cd241740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fb6a000e810 con 0x56440975b7b0 2024-04-02T14:40:17.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.163+0000 7fb6cd241740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fb6a0010350 con 0x56440975c0d0 2024-04-02T14:40:17.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.163+0000 7fb6cd241740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fb6a000eee0 con 0x564409764cb0 2024-04-02T14:40:17.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.164+0000 7fb6c3400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56440975c0d0 0x5644097625c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:17.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.164+0000 7fb6c3400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56440975c0d0 0x5644097625c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:39436/0 (socket says 172.21.15.90:39436) 2024-04-02T14:40:17.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.164+0000 7fb6c3400640 1 -- 172.21.15.90:0/1168730544 learned_addr learned my addr 172.21.15.90:0/1168730544 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:40:17.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.164+0000 7fb6baa00640 1 --2- 172.21.15.90:0/1168730544 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56440975b7b0 0x564409770990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:17.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.164+0000 7fb6baa00640 1 --2- 172.21.15.90:0/1168730544 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564409764cb0 0x564409757940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:17.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.164+0000 7fb6c3400640 1 -- 172.21.15.90:0/1168730544 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564409764cb0 msgr2=0x564409757940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:40:17.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.164+0000 7fb6c3400640 1 --2- 172.21.15.90:0/1168730544 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564409764cb0 0x564409757940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:17.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.164+0000 7fb6c3400640 1 -- 172.21.15.90:0/1168730544 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56440975b7b0 msgr2=0x564409770990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:40:17.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.165+0000 7fb6c3400640 1 --2- 172.21.15.90:0/1168730544 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56440975b7b0 0x564409770990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:17.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.165+0000 7fb6c3400640 1 -- 172.21.15.90:0/1168730544 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5644097556e0 con 0x56440975c0d0 2024-04-02T14:40:17.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.165+0000 7fb6c3400640 1 --2- 172.21.15.90:0/1168730544 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56440975c0d0 0x5644097625c0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fb6b4048cf0 tx=0x7fb6b4009ba0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=49dbc3b94eaf90d5 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:40:17.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.165+0000 7fb6b3e00640 1 -- 172.21.15.90:0/1168730544 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6b4060070 con 0x56440975c0d0 2024-04-02T14:40:17.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.165+0000 7fb6b3e00640 1 -- 172.21.15.90:0/1168730544 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb6b4048930 con 0x56440975c0d0 2024-04-02T14:40:17.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.165+0000 7fb6b3e00640 1 -- 172.21.15.90:0/1168730544 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6b404c070 con 0x56440975c0d0 2024-04-02T14:40:17.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.167+0000 7fb6cd241740 1 -- 172.21.15.90:0/1168730544 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56440975c0d0 msgr2=0x5644097625c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:17.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.167+0000 7fb6cd241740 1 --2- 172.21.15.90:0/1168730544 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56440975c0d0 0x5644097625c0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fb6b4048cf0 tx=0x7fb6b4009ba0 comp rx=0 tx=0).stop 2024-04-02T14:40:17.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.167+0000 7fb6cd241740 1 -- 172.21.15.90:0/1168730544 shutdown_connections 2024-04-02T14:40:17.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.167+0000 7fb6cd241740 1 --2- 172.21.15.90:0/1168730544 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564409764cb0 0x564409757940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:17.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.167+0000 7fb6cd241740 1 --2- 172.21.15.90:0/1168730544 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56440975c0d0 0x5644097625c0 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:17.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.167+0000 7fb6cd241740 1 --2- 172.21.15.90:0/1168730544 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56440975b7b0 0x564409770990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:17.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.167+0000 7fb6cd241740 1 -- 172.21.15.90:0/1168730544 >> 172.21.15.90:0/1168730544 conn(0x564409755ee0 msgr2=0x564409757230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:40:17.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.167+0000 7fb6cd241740 1 -- 172.21.15.90:0/1168730544 shutdown_connections 2024-04-02T14:40:17.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.167+0000 7fb6cd241740 1 -- 172.21.15.90:0/1168730544 wait complete. 2024-04-02T14:40:17.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.169+0000 7fb6cd241740 1 -- start start 2024-04-02T14:40:17.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.169+0000 7fb6cd241740 1 Processor -- start 2024-04-02T14:40:17.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.170+0000 7fb6cd241740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56440975b7b0 0x56440976d170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:17.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.170+0000 7fb6cd241740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56440975c0d0 0x56440975e610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:17.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.170+0000 7fb6cd241740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564409764cb0 0x56440975eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:17.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.170+0000 7fb6cd241740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fb6a000fc80 con 0x56440975b7b0 2024-04-02T14:40:17.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.170+0000 7fb6cd241740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fb6a000f5b0 con 0x564409764cb0 2024-04-02T14:40:17.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.170+0000 7fb6cd241740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fb6a000c7f0 con 0x56440975c0d0 2024-04-02T14:40:17.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.171+0000 7fb6cd241740 10 client.0 _ll_register_callbacks cb 0x7fb6ac005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:40:17.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.171+0000 7fb6baa00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56440975b7b0 0x56440976d170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:17.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.172+0000 7fb6c3400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56440975c0d0 0x56440975e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:17.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.172+0000 7fb6c3400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56440975c0d0 0x56440975e610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:54164/0 (socket says 172.21.15.90:54164) 2024-04-02T14:40:17.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.172+0000 7fb6c3400640 1 -- 172.21.15.90:0/3751838225 learned_addr learned my addr 172.21.15.90:0/3751838225 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:40:17.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.173+0000 7fb6c3400640 1 -- 172.21.15.90:0/3751838225 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564409764cb0 msgr2=0x56440975eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:40:17.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.173+0000 7fb6c3400640 1 --2- 172.21.15.90:0/3751838225 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564409764cb0 0x56440975eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:17.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.173+0000 7fb6c3400640 1 -- 172.21.15.90:0/3751838225 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56440975b7b0 msgr2=0x56440976d170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:40:17.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.173+0000 7fb6c3400640 1 --2- 172.21.15.90:0/3751838225 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56440975b7b0 0x56440976d170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:17.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.173+0000 7fb6c3400640 1 -- 172.21.15.90:0/3751838225 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56440975f210 con 0x56440975c0d0 2024-04-02T14:40:17.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.173+0000 7fb6baa00640 1 --2- 172.21.15.90:0/3751838225 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56440975b7b0 0x56440976d170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:40:17.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.173+0000 7fb6c3400640 1 --2- 172.21.15.90:0/3751838225 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56440975c0d0 0x56440975e610 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fb6b4048cc0 tx=0x7fb6b40099f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b2ec2f6e948e0629 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:40:17.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.173+0000 7fb6b1600640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:40:17.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.174+0000 7fb6b1600640 1 -- 172.21.15.90:0/3751838225 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6b404c070 con 0x56440975c0d0 2024-04-02T14:40:17.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.174+0000 7fb6b1600640 1 -- 172.21.15.90:0/3751838225 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7fb6b4003f60 con 0x56440975c0d0 2024-04-02T14:40:17.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.174+0000 7fb6cd241740 1 -- 172.21.15.90:0/3751838225 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb6b4048930 con 0x56440975c0d0 2024-04-02T14:40:17.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.174+0000 7fb6b1600640 1 -- 172.21.15.90:0/3751838225 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fb6b4060070 con 0x56440975c0d0 2024-04-02T14:40:17.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.174+0000 7fb6cd241740 10 client.26980 fetch_fsmap learned FSMap version 27 2024-04-02T14:40:17.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.174+0000 7fb6cd241740 1 -- 172.21.15.90:0/3751838225 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5644097dac60 con 0x56440975c0d0 2024-04-02T14:40:17.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.175+0000 7fb6b1600640 1 -- 172.21.15.90:0/3751838225 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb6b4048930 con 0x56440975c0d0 2024-04-02T14:40:17.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.175+0000 7fb6b1600640 1 -- 172.21.15.90:0/3751838225 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fb6b40517c0 con 0x56440975c0d0 2024-04-02T14:40:17.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.176+0000 7fb6cd241740 10 client.26980 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:40:17.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.176+0000 7fb6cd241740 10 client.26980 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:40:17.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.176+0000 7fb6cd241740 20 client.26980 populate_metadata read hostname 'smithi090' 2024-04-02T14:40:17.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.176+0000 7fb6cd241740 1 --2- 172.21.15.90:0/3751838225 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5644097db190 0x5644097dd5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:17.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.176+0000 7fb6cd241740 4 client.26980 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-02T14:40:17.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.176+0000 7fb6cd241740 1 -- 172.21.15.90:0/3751838225 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5644096cccb0 con 0x5644097db190 2024-04-02T14:40:17.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.177+0000 7fb6baa00640 1 --2- 172.21.15.90:0/3751838225 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5644097db190 0x5644097dd5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:17.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.180+0000 7fb6b1600640 10 client.26980 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:40:17.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.180+0000 7fb6baa00640 1 --2- 172.21.15.90:0/3751838225 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5644097db190 0x5644097dd5b0 crc :-1 s=READY pgs=159 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=91b378eff4c6265f server_cookie=4822f363bca795da in_seq=0 out_seq=0 2024-04-02T14:40:17.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.180+0000 7fb6b1600640 1 -- 172.21.15.90:0/3751838225 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x5644096cccb0 con 0x5644097db190 2024-04-02T14:40:17.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.180+0000 7fb6b1600640 10 client.26980 handle_command_reply: tid=0 2024-04-02T14:40:17.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.180+0000 7fb6cd241740 1 client.26980 shutdown 2024-04-02T14:40:17.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.181+0000 7fb6cd241740 1 -- 172.21.15.90:0/3751838225 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56440975c0d0 msgr2=0x56440975e610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:17.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.181+0000 7fb6cd241740 1 --2- 172.21.15.90:0/3751838225 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56440975c0d0 0x56440975e610 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fb6b4048cc0 tx=0x7fb6b40099f0 comp rx=0 tx=0).stop 2024-04-02T14:40:17.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.181+0000 7fb6cd241740 1 -- 172.21.15.90:0/3751838225 shutdown_connections 2024-04-02T14:40:17.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.181+0000 7fb6cd241740 1 --2- 172.21.15.90:0/3751838225 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5644097db190 0x5644097dd5b0 crc :-1 s=READY pgs=159 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:17.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.181+0000 7fb6cd241740 1 --2- 172.21.15.90:0/3751838225 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564409764cb0 0x56440975eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:17.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.181+0000 7fb6cd241740 1 --2- 172.21.15.90:0/3751838225 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56440975c0d0 0x56440975e610 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:17.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.181+0000 7fb6cd241740 1 --2- 172.21.15.90:0/3751838225 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56440975b7b0 0x56440976d170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:17.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.181+0000 7fb6cd241740 1 -- 172.21.15.90:0/3751838225 >> 172.21.15.90:0/3751838225 conn(0x564409755ee0 msgr2=0x564409754f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:40:17.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.181+0000 7fb6b1600640 0 client.26980 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:40:17.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.181+0000 7fb6cd241740 1 -- 172.21.15.90:0/3751838225 shutdown_connections 2024-04-02T14:40:17.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.182+0000 7fb6cd241740 1 -- 172.21.15.90:0/3751838225 wait complete. 2024-04-02T14:40:17.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.182+0000 7fb6cd241740 20 client.26980 trim_cache size 0 max 16384 2024-04-02T14:40:17.183 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-02T14:40:17.183 INFO:teuthology.orchestra.run.smithi090.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-02T14:40:17.183 INFO:teuthology.orchestra.run.smithi090.stdout: "scrubs": { 2024-04-02T14:40:17.183 INFO:teuthology.orchestra.run.smithi090.stdout: "81a61248-ad60-4fb8-a6a1-6f88274484e0": { 2024-04-02T14:40:17.183 INFO:teuthology.orchestra.run.smithi090.stdout: "path": "/", 2024-04-02T14:40:17.183 INFO:teuthology.orchestra.run.smithi090.stdout: "tag": "81a61248-ad60-4fb8-a6a1-6f88274484e0", 2024-04-02T14:40:17.183 INFO:teuthology.orchestra.run.smithi090.stdout: "options": "recursive,force" 2024-04-02T14:40:17.183 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:40:17.184 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:40:17.184 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-02T14:40:17.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.183+0000 7fb6ca600640 1 -- 172.21.15.90:0/3575788135 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fb6a80a3e70 msgr2=0x7fb6a80a6330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:17.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.183+0000 7fb6ca600640 1 --2- 172.21.15.90:0/3575788135 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fb6a80a3e70 0x7fb6a80a6330 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fb6bc018000 tx=0x7fb6bc009790 comp rx=0 tx=0).stop 2024-04-02T14:40:17.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.183+0000 7fb6ca600640 1 -- 172.21.15.90:0/3575788135 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb6c4107aa0 msgr2=0x7fb6c410cbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:17.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.183+0000 7fb6ca600640 1 --2- 172.21.15.90:0/3575788135 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb6c4107aa0 0x7fb6c410cbb0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fb6bc00e970 tx=0x7fb6bc00ed70 comp rx=0 tx=0).stop 2024-04-02T14:40:17.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.183+0000 7fb6ca600640 1 -- 172.21.15.90:0/3575788135 shutdown_connections 2024-04-02T14:40:17.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.183+0000 7fb6ca600640 1 --2- 172.21.15.90:0/3575788135 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fb6a80a3e70 0x7fb6a80a6330 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:17.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.183+0000 7fb6ca600640 1 --2- 172.21.15.90:0/3575788135 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb6c4107aa0 0x7fb6c410cbb0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:17.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.183+0000 7fb6ca600640 1 --2- 172.21.15.90:0/3575788135 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb6c4102450 0x7fb6c419f5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:17.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.183+0000 7fb6ca600640 1 --2- 172.21.15.90:0/3575788135 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb6c4101a90 0x7fb6c419f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:17.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.183+0000 7fb6ca600640 1 -- 172.21.15.90:0/3575788135 >> 172.21.15.90:0/3575788135 conn(0x7fb6c4078f00 msgr2=0x7fb6c4074210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:40:17.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.183+0000 7fb6ca600640 1 -- 172.21.15.90:0/3575788135 shutdown_connections 2024-04-02T14:40:17.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:17.183+0000 7fb6ca600640 1 -- 172.21.15.90:0/3575788135 wait complete. 2024-04-02T14:40:17.208 INFO:tasks.cephfs.filesystem:scrub status for tag:81a61248-ad60-4fb8-a6a1-6f88274484e0 - {'path': '/', 'tag': '81a61248-ad60-4fb8-a6a1-6f88274484e0', 'options': 'recursive,force'} 2024-04-02T14:40:17.860 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:40:17 smithi092 ceph-mon[36497]: pgmap v165: 97 pgs: 97 active+clean; 3.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 24 MiB/s wr, 1.50k op/s 2024-04-02T14:40:17.935 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:40:17 smithi177 ceph-mon[36431]: pgmap v165: 97 pgs: 97 active+clean; 3.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 24 MiB/s wr, 1.50k op/s 2024-04-02T14:40:17.937 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:40:17 smithi090 ceph-mon[31674]: pgmap v165: 97 pgs: 97 active+clean; 3.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 24 MiB/s wr, 1.50k op/s 2024-04-02T14:40:18.177 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.177+0000 7fed56000640 1 -- 172.21.15.90:0/3999424312 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fed50116b90 msgr2=0x7fed50118f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:18.177 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.177+0000 7fed56000640 1 --2- 172.21.15.90:0/3999424312 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fed50116b90 0x7fed50118f80 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fed40009900 tx=0x7fed4002bcc0 comp rx=0 tx=0).stop 2024-04-02T14:40:18.178 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.177+0000 7fed56000640 1 -- 172.21.15.90:0/3999424312 shutdown_connections 2024-04-02T14:40:18.178 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.177+0000 7fed56000640 1 --2- 172.21.15.90:0/3999424312 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fed50116b90 0x7fed50118f80 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:18.178 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.177+0000 7fed56000640 1 --2- 172.21.15.90:0/3999424312 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fed50109040 0x7fed501094c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:18.178 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.177+0000 7fed56000640 1 --2- 172.21.15.90:0/3999424312 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fed5010c230 0x7fed50108b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:18.179 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.177+0000 7fed56000640 1 -- 172.21.15.90:0/3999424312 >> 172.21.15.90:0/3999424312 conn(0x7fed5006d030 msgr2=0x7fed5006d440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:40:18.179 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.177+0000 7fed56000640 1 -- 172.21.15.90:0/3999424312 shutdown_connections 2024-04-02T14:40:18.179 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.178+0000 7fed56000640 1 -- 172.21.15.90:0/3999424312 wait complete. 2024-04-02T14:40:18.179 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.179+0000 7fed56000640 1 Processor -- start 2024-04-02T14:40:18.179 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.179+0000 7fed56000640 1 -- start start 2024-04-02T14:40:18.180 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.180+0000 7fed56000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fed50109040 0x7fed50116450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:18.180 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.180+0000 7fed4f400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fed50109040 0x7fed50116450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:18.181 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.180+0000 7fed4f400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fed50109040 0x7fed50116450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:39474/0 (socket says 172.21.15.90:39474) 2024-04-02T14:40:18.181 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.181+0000 7fed56000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fed5010c230 0x7fed50113590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:18.181 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.181+0000 7fed4f400640 1 -- 172.21.15.90:0/2606905318 learned_addr learned my addr 172.21.15.90:0/2606905318 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:40:18.181 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.181+0000 7fed56000640 1 --2- 172.21.15.90:0/2606905318 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fed50116b90 0x7fed50113ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:18.182 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.181+0000 7fed56000640 1 -- 172.21.15.90:0/2606905318 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fed5011c8c0 con 0x7fed50116b90 2024-04-02T14:40:18.182 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.181+0000 7fed56000640 1 -- 172.21.15.90:0/2606905318 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fed5011c740 con 0x7fed50109040 2024-04-02T14:40:18.182 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.181+0000 7fed56000640 1 -- 172.21.15.90:0/2606905318 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fed5011ca40 con 0x7fed5010c230 2024-04-02T14:40:18.182 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.182+0000 7fed4f400640 1 -- 172.21.15.90:0/2606905318 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fed5010c230 msgr2=0x7fed50113590 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-02T14:40:18.182 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.182+0000 7fed4f400640 1 --2- 172.21.15.90:0/2606905318 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fed5010c230 0x7fed50113590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:18.182 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.182+0000 7fed4f400640 1 -- 172.21.15.90:0/2606905318 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fed50116b90 msgr2=0x7fed50113ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:18.182 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.182+0000 7fed4f400640 1 --2- 172.21.15.90:0/2606905318 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fed50116b90 0x7fed50113ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:18.183 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.182+0000 7fed4f400640 1 -- 172.21.15.90:0/2606905318 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed501b3600 con 0x7fed50109040 2024-04-02T14:40:18.183 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.182+0000 7fed4f400640 1 --2- 172.21.15.90:0/2606905318 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fed50109040 0x7fed50116450 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fed4800eab0 tx=0x7fed4800ef70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:40:18.183 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.183+0000 7fed47e00640 1 -- 172.21.15.90:0/2606905318 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed4800ce40 con 0x7fed50109040 2024-04-02T14:40:18.184 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.183+0000 7fed47e00640 1 -- 172.21.15.90:0/2606905318 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fed48011950 con 0x7fed50109040 2024-04-02T14:40:18.184 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.183+0000 7fed56000640 1 -- 172.21.15.90:0/2606905318 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed501b37d0 con 0x7fed50109040 2024-04-02T14:40:18.184 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.183+0000 7fed47e00640 1 -- 172.21.15.90:0/2606905318 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed4801aad0 con 0x7fed50109040 2024-04-02T14:40:18.184 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.183+0000 7fed56000640 1 -- 172.21.15.90:0/2606905318 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed501b3c30 con 0x7fed50109040 2024-04-02T14:40:18.191 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.190+0000 7fed47e00640 1 -- 172.21.15.90:0/2606905318 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fed4801ac70 con 0x7fed50109040 2024-04-02T14:40:18.193 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.193+0000 7fed47e00640 1 --2- 172.21.15.90:0/2606905318 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fed340a3f20 0x7fed340a63e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:18.194 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.193+0000 7fed47e00640 1 -- 172.21.15.90:0/2606905318 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7fed480d1120 con 0x7fed50109040 2024-04-02T14:40:18.195 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.195+0000 7fed4ea00640 1 --2- 172.21.15.90:0/2606905318 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fed340a3f20 0x7fed340a63e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:18.197 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.197+0000 7fed58de9740 1 Processor -- start 2024-04-02T14:40:18.197 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.197+0000 7fed58de9740 1 -- start start 2024-04-02T14:40:18.198 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.197+0000 7fed58de9740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559cc2f3beb0 0x559cc2f3c290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:18.198 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.197+0000 7fed58de9740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559cc2f3c7d0 0x559cc2f44e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:18.198 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.197+0000 7fed58de9740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559cc2f453b0 0x559cc2f477a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:18.198 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.197+0000 7fed58de9740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x559cc2ea0d70 con 0x559cc2f453b0 2024-04-02T14:40:18.198 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.197+0000 7fed58de9740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x559cc2e492a0 con 0x559cc2f3beb0 2024-04-02T14:40:18.198 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.197+0000 7fed58de9740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x559cc2e9cd20 con 0x559cc2f3c7d0 2024-04-02T14:40:18.198 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.197+0000 7fed4f400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559cc2f3beb0 0x559cc2f3c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:18.199 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.198+0000 7fed4f400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559cc2f3beb0 0x559cc2f3c290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:39478/0 (socket says 172.21.15.90:39478) 2024-04-02T14:40:18.199 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.198+0000 7fed4f400640 1 -- 172.21.15.90:0/2854585527 learned_addr learned my addr 172.21.15.90:0/2854585527 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:40:18.199 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.198+0000 7fed4f400640 1 -- 172.21.15.90:0/2854585527 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559cc2f3c7d0 msgr2=0x559cc2f44e70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-02T14:40:18.199 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.198+0000 7fed4f400640 1 --2- 172.21.15.90:0/2854585527 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559cc2f3c7d0 0x559cc2f44e70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:18.199 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.198+0000 7fed4f400640 1 -- 172.21.15.90:0/2854585527 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559cc2f453b0 msgr2=0x559cc2f477a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-02T14:40:18.200 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.198+0000 7fed4f400640 1 --2- 172.21.15.90:0/2854585527 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559cc2f453b0 0x559cc2f477a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:18.200 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.198+0000 7fed4f400640 1 -- 172.21.15.90:0/2854585527 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559cc2e5d840 con 0x559cc2f3beb0 2024-04-02T14:40:18.200 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.198+0000 7fed4f400640 1 --2- 172.21.15.90:0/2854585527 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559cc2f3beb0 0x559cc2f3c290 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fed480cee80 tx=0x7fed4809cfd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dd9afa995d0c70ac server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:40:18.200 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.199+0000 7fed3fe00640 1 -- 172.21.15.90:0/2854585527 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed480b5020 con 0x559cc2f3beb0 2024-04-02T14:40:18.200 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.199+0000 7fed3fe00640 1 -- 172.21.15.90:0/2854585527 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fed4809d7d0 con 0x559cc2f3beb0 2024-04-02T14:40:18.200 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.199+0000 7fed3fe00640 1 -- 172.21.15.90:0/2854585527 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed48099740 con 0x559cc2f3beb0 2024-04-02T14:40:18.201 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.200+0000 7fed58de9740 1 -- 172.21.15.90:0/2854585527 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559cc2f3beb0 msgr2=0x559cc2f3c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:18.201 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.200+0000 7fed58de9740 1 --2- 172.21.15.90:0/2854585527 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559cc2f3beb0 0x559cc2f3c290 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fed480cee80 tx=0x7fed4809cfd0 comp rx=0 tx=0).stop 2024-04-02T14:40:18.201 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.200+0000 7fed58de9740 1 -- 172.21.15.90:0/2854585527 shutdown_connections 2024-04-02T14:40:18.201 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.200+0000 7fed58de9740 1 --2- 172.21.15.90:0/2854585527 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559cc2f453b0 0x559cc2f477a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:18.201 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.200+0000 7fed58de9740 1 --2- 172.21.15.90:0/2854585527 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559cc2f3c7d0 0x559cc2f44e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:18.201 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.200+0000 7fed58de9740 1 --2- 172.21.15.90:0/2854585527 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559cc2f3beb0 0x559cc2f3c290 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:18.201 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.200+0000 7fed58de9740 1 -- 172.21.15.90:0/2854585527 >> 172.21.15.90:0/2854585527 conn(0x559cc2f365e0 msgr2=0x559cc2f369c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:40:18.201 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.200+0000 7fed4ea00640 1 --2- 172.21.15.90:0/2606905318 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fed340a3f20 0x7fed340a63e0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fed50114cb0 tx=0x7fed38005c00 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:40:18.202 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.201+0000 7fed58de9740 1 -- 172.21.15.90:0/2854585527 shutdown_connections 2024-04-02T14:40:18.202 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.201+0000 7fed58de9740 1 -- 172.21.15.90:0/2854585527 wait complete. 2024-04-02T14:40:18.203 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.203+0000 7fed58de9740 1 -- start start 2024-04-02T14:40:18.203 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.203+0000 7fed58de9740 1 Processor -- start 2024-04-02T14:40:18.206 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.205+0000 7fed58de9740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559cc2f3beb0 0x559cc2f424b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:18.206 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.206+0000 7fed58de9740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559cc2f3c7d0 0x559cc2f429f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:18.207 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.206+0000 7fed4f400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559cc2f3beb0 0x559cc2f424b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:18.207 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.206+0000 7fed4f400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559cc2f3beb0 0x559cc2f424b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:54190/0 (socket says 172.21.15.90:54190) 2024-04-02T14:40:18.207 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.206+0000 7fed4ea00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559cc2f3c7d0 0x559cc2f429f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:18.207 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.206+0000 7fed4ea00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559cc2f3c7d0 0x559cc2f429f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:39480/0 (socket says 172.21.15.90:39480) 2024-04-02T14:40:18.207 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.206+0000 7fed58de9740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559cc2f453b0 0x559cc2f42f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:18.207 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.206+0000 7fed58de9740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x559cc2f481a0 con 0x559cc2f453b0 2024-04-02T14:40:18.207 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.206+0000 7fed58de9740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x559cc2e5c660 con 0x559cc2f3c7d0 2024-04-02T14:40:18.207 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.206+0000 7fed58de9740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x559cc2f48320 con 0x559cc2f3beb0 2024-04-02T14:40:18.208 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.206+0000 7fed58de9740 10 client.0 _ll_register_callbacks cb 0x559cc2e5c050 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:40:18.208 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.206+0000 7fed4f400640 1 -- 172.21.15.90:0/1541484953 learned_addr learned my addr 172.21.15.90:0/1541484953 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:40:18.208 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.207+0000 7fed4fe00640 1 --2- 172.21.15.90:0/1541484953 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559cc2f453b0 0x559cc2f42f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:18.208 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.207+0000 7fed4ea00640 1 -- 172.21.15.90:0/1541484953 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559cc2f3beb0 msgr2=0x559cc2f424b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:40:18.208 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.207+0000 7fed4ea00640 1 --2- 172.21.15.90:0/1541484953 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559cc2f3beb0 0x559cc2f424b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:18.209 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.207+0000 7fed4ea00640 1 -- 172.21.15.90:0/1541484953 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559cc2f453b0 msgr2=0x559cc2f42f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:40:18.209 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.207+0000 7fed4ea00640 1 --2- 172.21.15.90:0/1541484953 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559cc2f453b0 0x559cc2f42f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:18.209 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.207+0000 7fed4ea00640 1 -- 172.21.15.90:0/1541484953 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559cc2e6be80 con 0x559cc2f3c7d0 2024-04-02T14:40:18.209 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.208+0000 7fed4ea00640 1 --2- 172.21.15.90:0/1541484953 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559cc2f3c7d0 0x559cc2f429f0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fed38005fd0 tx=0x7fed38004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c6b0ced5e8f938b server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:40:18.209 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.208+0000 7fed3cc00640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:40:18.209 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.208+0000 7fed3cc00640 1 -- 172.21.15.90:0/1541484953 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed38020070 con 0x559cc2f3c7d0 2024-04-02T14:40:18.209 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.208+0000 7fed3cc00640 1 -- 172.21.15.90:0/1541484953 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7fed38005220 con 0x559cc2f3c7d0 2024-04-02T14:40:18.210 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.208+0000 7fed58de9740 1 -- 172.21.15.90:0/1541484953 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fed4809d7d0 con 0x559cc2f3c7d0 2024-04-02T14:40:18.210 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.209+0000 7fed3cc00640 1 -- 172.21.15.90:0/1541484953 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fed3802b050 con 0x559cc2f3c7d0 2024-04-02T14:40:18.210 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.209+0000 7fed58de9740 10 client.27107 fetch_fsmap learned FSMap version 27 2024-04-02T14:40:18.210 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.209+0000 7fed58de9740 1 -- 172.21.15.90:0/1541484953 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x559cc2bb5210 con 0x559cc2f3c7d0 2024-04-02T14:40:18.210 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.209+0000 7fed3cc00640 1 -- 172.21.15.90:0/1541484953 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fed38016070 con 0x559cc2f3c7d0 2024-04-02T14:40:18.211 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.210+0000 7fed3cc00640 1 -- 172.21.15.90:0/1541484953 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fed3801b070 con 0x559cc2f3c7d0 2024-04-02T14:40:18.211 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.210+0000 7fed58de9740 10 client.27107 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:40:18.211 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.210+0000 7fed58de9740 10 client.27107 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-02T14:40:18.211 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.210+0000 7fed58de9740 20 client.27107 populate_metadata read hostname 'smithi090' 2024-04-02T14:40:18.211 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.210+0000 7fed58de9740 1 --2- 172.21.15.90:0/1541484953 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x559cc2f55820 0x559cc2f57c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:18.211 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.210+0000 7fed58de9740 4 client.27107 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:40:18.211 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.210+0000 7fed58de9740 1 -- 172.21.15.90:0/1541484953 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fed4809d7d0 con 0x559cc2f55820 2024-04-02T14:40:18.211 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.210+0000 7fed4f400640 1 --2- 172.21.15.90:0/1541484953 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x559cc2f55820 0x559cc2f57c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:18.212 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.211+0000 7fed3cc00640 10 client.27107 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:40:18.212 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.212+0000 7fed4f400640 1 --2- 172.21.15.90:0/1541484953 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x559cc2f55820 0x559cc2f57c50 crc :-1 s=READY pgs=160 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=80bb89acadab60f4 server_cookie=e538bcf04b192aa2 in_seq=0 out_seq=0 2024-04-02T14:40:18.212 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.212+0000 7fed3cc00640 1 -- 172.21.15.90:0/1541484953 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x7fed4809d7d0 con 0x559cc2f55820 2024-04-02T14:40:18.212 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.212+0000 7fed3cc00640 10 client.27107 handle_command_reply: tid=0 2024-04-02T14:40:18.213 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.213+0000 7fed58de9740 1 client.27107 shutdown 2024-04-02T14:40:18.214 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.213+0000 7fed58de9740 1 -- 172.21.15.90:0/1541484953 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559cc2f3c7d0 msgr2=0x559cc2f429f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:18.214 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.213+0000 7fed58de9740 1 --2- 172.21.15.90:0/1541484953 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559cc2f3c7d0 0x559cc2f429f0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fed38005fd0 tx=0x7fed38004730 comp rx=0 tx=0).stop 2024-04-02T14:40:18.214 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.213+0000 7fed58de9740 1 -- 172.21.15.90:0/1541484953 shutdown_connections 2024-04-02T14:40:18.214 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.213+0000 7fed58de9740 1 --2- 172.21.15.90:0/1541484953 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x559cc2f55820 0x559cc2f57c50 crc :-1 s=READY pgs=160 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:18.214 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.213+0000 7fed58de9740 1 --2- 172.21.15.90:0/1541484953 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559cc2f453b0 0x559cc2f42f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:18.214 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.213+0000 7fed58de9740 1 --2- 172.21.15.90:0/1541484953 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559cc2f3c7d0 0x559cc2f429f0 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:18.214 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.213+0000 7fed58de9740 1 --2- 172.21.15.90:0/1541484953 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559cc2f3beb0 0x559cc2f424b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:18.214 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.213+0000 7fed58de9740 1 -- 172.21.15.90:0/1541484953 >> 172.21.15.90:0/1541484953 conn(0x559cc2f365e0 msgr2=0x559cc2f38330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:40:18.215 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.213+0000 7fed3cc00640 0 client.27107 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:40:18.215 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.213+0000 7fed58de9740 1 -- 172.21.15.90:0/1541484953 shutdown_connections 2024-04-02T14:40:18.215 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.214+0000 7fed58de9740 1 -- 172.21.15.90:0/1541484953 wait complete. 2024-04-02T14:40:18.215 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.214+0000 7fed58de9740 20 client.27107 trim_cache size 0 max 16384 2024-04-02T14:40:18.245 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.244+0000 7fed58de9740 1 Processor -- start 2024-04-02T14:40:18.245 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.244+0000 7fed58de9740 1 -- start start 2024-04-02T14:40:18.245 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.244+0000 7fed58de9740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559cc2f3beb0 0x559cc2f51060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:18.245 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.244+0000 7fed58de9740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559cc2f3c7d0 0x559cc2f3d790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:18.245 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.244+0000 7fed58de9740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559cc2f453b0 0x559cc2f42d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:18.245 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.244+0000 7fed58de9740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fed501bc690 con 0x559cc2f453b0 2024-04-02T14:40:18.245 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.244+0000 7fed58de9740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fed501be1d0 con 0x559cc2f3c7d0 2024-04-02T14:40:18.245 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.244+0000 7fed58de9740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fed501bcd60 con 0x559cc2f3beb0 2024-04-02T14:40:18.246 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.245+0000 7fed4ea00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559cc2f3c7d0 0x559cc2f3d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:18.246 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.245+0000 7fed4ea00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559cc2f3c7d0 0x559cc2f3d790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:39484/0 (socket says 172.21.15.90:39484) 2024-04-02T14:40:18.246 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.245+0000 7fed4ea00640 1 -- 172.21.15.90:0/2274507331 learned_addr learned my addr 172.21.15.90:0/2274507331 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:40:18.246 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.245+0000 7fed4f400640 1 --2- 172.21.15.90:0/2274507331 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559cc2f3beb0 0x559cc2f51060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:18.246 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.245+0000 7fed4fe00640 1 --2- 172.21.15.90:0/2274507331 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559cc2f453b0 0x559cc2f42d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:18.246 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.246+0000 7fed4ea00640 1 -- 172.21.15.90:0/2274507331 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559cc2f3beb0 msgr2=0x559cc2f51060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:40:18.247 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.246+0000 7fed4ea00640 1 --2- 172.21.15.90:0/2274507331 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559cc2f3beb0 0x559cc2f51060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:18.247 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.246+0000 7fed4ea00640 1 -- 172.21.15.90:0/2274507331 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559cc2f453b0 msgr2=0x559cc2f42d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:40:18.247 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.246+0000 7fed4ea00640 1 --2- 172.21.15.90:0/2274507331 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559cc2f453b0 0x559cc2f42d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:18.247 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.246+0000 7fed4ea00640 1 -- 172.21.15.90:0/2274507331 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559cc2c81620 con 0x559cc2f3c7d0 2024-04-02T14:40:18.247 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.246+0000 7fed4fe00640 1 --2- 172.21.15.90:0/2274507331 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559cc2f453b0 0x559cc2f42d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:40:18.247 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.246+0000 7fed4ea00640 1 --2- 172.21.15.90:0/2274507331 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559cc2f3c7d0 0x559cc2f3d790 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7fed3800e9e0 tx=0x7fed3800c840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=34f0db010db18ae4 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:40:18.247 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.247+0000 7fed3f400640 1 -- 172.21.15.90:0/2274507331 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed38034070 con 0x559cc2f3c7d0 2024-04-02T14:40:18.247 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.247+0000 7fed3f400640 1 -- 172.21.15.90:0/2274507331 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fed38016070 con 0x559cc2f3c7d0 2024-04-02T14:40:18.248 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.247+0000 7fed3f400640 1 -- 172.21.15.90:0/2274507331 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed38020040 con 0x559cc2f3c7d0 2024-04-02T14:40:18.249 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.249+0000 7fed58de9740 1 -- 172.21.15.90:0/2274507331 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559cc2f3c7d0 msgr2=0x559cc2f3d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:18.249 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.249+0000 7fed58de9740 1 --2- 172.21.15.90:0/2274507331 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559cc2f3c7d0 0x559cc2f3d790 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7fed3800e9e0 tx=0x7fed3800c840 comp rx=0 tx=0).stop 2024-04-02T14:40:18.249 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.249+0000 7fed58de9740 1 -- 172.21.15.90:0/2274507331 shutdown_connections 2024-04-02T14:40:18.250 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.249+0000 7fed58de9740 1 --2- 172.21.15.90:0/2274507331 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559cc2f453b0 0x559cc2f42d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:18.250 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.249+0000 7fed58de9740 1 --2- 172.21.15.90:0/2274507331 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559cc2f3c7d0 0x559cc2f3d790 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:18.250 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.249+0000 7fed58de9740 1 --2- 172.21.15.90:0/2274507331 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559cc2f3beb0 0x559cc2f51060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:18.250 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.249+0000 7fed58de9740 1 -- 172.21.15.90:0/2274507331 >> 172.21.15.90:0/2274507331 conn(0x559cc2f365e0 msgr2=0x559cc2f35640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:40:18.250 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.249+0000 7fed58de9740 1 -- 172.21.15.90:0/2274507331 shutdown_connections 2024-04-02T14:40:18.250 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.249+0000 7fed58de9740 1 -- 172.21.15.90:0/2274507331 wait complete. 2024-04-02T14:40:18.250 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.250+0000 7fed58de9740 1 -- start start 2024-04-02T14:40:18.251 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.250+0000 7fed58de9740 1 Processor -- start 2024-04-02T14:40:18.252 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.252+0000 7fed58de9740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559cc2f3beb0 0x559cc2f490f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:18.252 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.252+0000 7fed58de9740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559cc2f3c7d0 0x559cc2f49630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:18.252 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.252+0000 7fed58de9740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559cc2f453b0 0x559cc2f49b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:18.252 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.252+0000 7fed4f400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559cc2f3beb0 0x559cc2f490f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:18.253 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.252+0000 7fed58de9740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fed501bdb00 con 0x559cc2f3c7d0 2024-04-02T14:40:18.253 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.252+0000 7fed58de9740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fed501bd430 con 0x559cc2f453b0 2024-04-02T14:40:18.253 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.252+0000 7fed58de9740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fed501ba670 con 0x559cc2f3beb0 2024-04-02T14:40:18.253 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.252+0000 7fed4ea00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559cc2f3c7d0 0x559cc2f49630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:18.253 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.252+0000 7fed4fe00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559cc2f453b0 0x559cc2f49b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:18.253 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.252+0000 7fed4f400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559cc2f3beb0 0x559cc2f490f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:54220/0 (socket says 172.21.15.90:54220) 2024-04-02T14:40:18.253 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.252+0000 7fed4f400640 1 -- 172.21.15.90:0/3453059696 learned_addr learned my addr 172.21.15.90:0/3453059696 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:40:18.253 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.252+0000 7fed58de9740 10 client.0 _ll_register_callbacks cb 0x7fed3800f450 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:40:18.254 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.253+0000 7fed4fe00640 1 -- 172.21.15.90:0/3453059696 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559cc2f3beb0 msgr2=0x559cc2f490f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:40:18.254 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.253+0000 7fed4fe00640 1 --2- 172.21.15.90:0/3453059696 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559cc2f3beb0 0x559cc2f490f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:18.254 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.253+0000 7fed4fe00640 1 -- 172.21.15.90:0/3453059696 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559cc2f3c7d0 msgr2=0x559cc2f49630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:40:18.254 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.253+0000 7fed4fe00640 1 --2- 172.21.15.90:0/3453059696 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559cc2f3c7d0 0x559cc2f49630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:18.254 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.253+0000 7fed4fe00640 1 -- 172.21.15.90:0/3453059696 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559cc2fbaf30 con 0x559cc2f453b0 2024-04-02T14:40:18.254 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.253+0000 7fed4f400640 1 --2- 172.21.15.90:0/3453059696 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559cc2f3beb0 0x559cc2f490f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:40:18.254 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.253+0000 7fed4ea00640 1 --2- 172.21.15.90:0/3453059696 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559cc2f3c7d0 0x559cc2f49630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:40:18.254 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.254+0000 7fed4fe00640 1 --2- 172.21.15.90:0/3453059696 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559cc2f453b0 0x559cc2f49b70 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fed50118470 tx=0x7fed4002c910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1f8ad5f7299fe20d server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:40:18.255 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.254+0000 7fed3cc00640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:40:18.255 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.254+0000 7fed3cc00640 1 -- 172.21.15.90:0/3453059696 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed40005030 con 0x559cc2f453b0 2024-04-02T14:40:18.255 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.254+0000 7fed3cc00640 1 -- 172.21.15.90:0/3453059696 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7fed400051d0 con 0x559cc2f453b0 2024-04-02T14:40:18.255 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.254+0000 7fed58de9740 1 -- 172.21.15.90:0/3453059696 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fed38016070 con 0x559cc2f453b0 2024-04-02T14:40:18.255 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.255+0000 7fed3cc00640 1 -- 172.21.15.90:0/3453059696 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fed4004d050 con 0x559cc2f453b0 2024-04-02T14:40:18.255 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.255+0000 7fed58de9740 10 client.27113 fetch_fsmap learned FSMap version 27 2024-04-02T14:40:18.256 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.255+0000 7fed58de9740 1 -- 172.21.15.90:0/3453059696 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x559cc2fbc200 con 0x559cc2f453b0 2024-04-02T14:40:18.256 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.255+0000 7fed3cc00640 1 -- 172.21.15.90:0/3453059696 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fed38016070 con 0x559cc2f453b0 2024-04-02T14:40:18.257 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.256+0000 7fed3cc00640 1 -- 172.21.15.90:0/3453059696 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fed400035f0 con 0x559cc2f453b0 2024-04-02T14:40:18.257 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.256+0000 7fed58de9740 10 client.27113 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:40:18.257 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.256+0000 7fed58de9740 10 client.27113 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-02T14:40:18.257 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.256+0000 7fed58de9740 20 client.27113 populate_metadata read hostname 'smithi090' 2024-04-02T14:40:18.257 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.256+0000 7fed58de9740 1 --2- 172.21.15.90:0/3453059696 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x559cc2fbc920 0x559cc2fbed10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:18.257 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.256+0000 7fed58de9740 4 client.27113 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-02T14:40:18.257 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.256+0000 7fed58de9740 1 -- 172.21.15.90:0/3453059696 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "flush journal"}) -- 0x559cc2e50cd0 con 0x559cc2fbc920 2024-04-02T14:40:18.257 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.257+0000 7fed4f400640 1 --2- 172.21.15.90:0/3453059696 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x559cc2fbc920 0x559cc2fbed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:18.258 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.258+0000 7fed4f400640 1 --2- 172.21.15.90:0/3453059696 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x559cc2fbc920 0x559cc2fbed10 crc :-1 s=READY pgs=161 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5c135c6f05e6ea21 server_cookie=7cbf187658c37df5 in_seq=0 out_seq=0 2024-04-02T14:40:18.258 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.258+0000 7fed3cc00640 10 client.27113 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:40:18.282 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.279+0000 7fed3cc00640 1 -- 172.21.15.90:0/3453059696 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x559cc2e50cd0 con 0x559cc2fbc920 2024-04-02T14:40:18.282 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.279+0000 7fed3cc00640 10 client.27113 handle_command_reply: tid=0 2024-04-02T14:40:18.282 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.280+0000 7fed58de9740 1 client.27113 shutdown 2024-04-02T14:40:18.282 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.280+0000 7fed58de9740 1 -- 172.21.15.90:0/3453059696 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559cc2f453b0 msgr2=0x559cc2f49b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:18.282 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.280+0000 7fed58de9740 1 --2- 172.21.15.90:0/3453059696 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559cc2f453b0 0x559cc2f49b70 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fed50118470 tx=0x7fed4002c910 comp rx=0 tx=0).stop 2024-04-02T14:40:18.283 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.280+0000 7fed58de9740 1 -- 172.21.15.90:0/3453059696 shutdown_connections 2024-04-02T14:40:18.283 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.280+0000 7fed58de9740 1 --2- 172.21.15.90:0/3453059696 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x559cc2fbc920 0x559cc2fbed10 crc :-1 s=READY pgs=161 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:18.283 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.280+0000 7fed58de9740 1 --2- 172.21.15.90:0/3453059696 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559cc2f453b0 0x559cc2f49b70 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:18.283 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.280+0000 7fed58de9740 1 --2- 172.21.15.90:0/3453059696 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559cc2f3c7d0 0x559cc2f49630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:18.283 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.280+0000 7fed58de9740 1 --2- 172.21.15.90:0/3453059696 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559cc2f3beb0 0x559cc2f490f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:18.283 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.280+0000 7fed58de9740 1 -- 172.21.15.90:0/3453059696 >> 172.21.15.90:0/3453059696 conn(0x559cc2f365e0 msgr2=0x559cc2f378c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:40:18.284 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.280+0000 7fed3cc00640 0 client.27113 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:40:18.284 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.281+0000 7fed58de9740 1 -- 172.21.15.90:0/3453059696 shutdown_connections 2024-04-02T14:40:18.284 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.281+0000 7fed58de9740 1 -- 172.21.15.90:0/3453059696 wait complete. 2024-04-02T14:40:18.284 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.281+0000 7fed58de9740 20 client.27113 trim_cache size 0 max 16384 2024-04-02T14:40:18.285 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:{ 2024-04-02T14:40:18.285 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "message": "", 2024-04-02T14:40:18.285 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "return_code": 0 2024-04-02T14:40:18.285 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:} 2024-04-02T14:40:18.286 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.283+0000 7fed56000640 1 -- 172.21.15.90:0/2606905318 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fed340a3f20 msgr2=0x7fed340a63e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:18.286 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.283+0000 7fed56000640 1 --2- 172.21.15.90:0/2606905318 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fed340a3f20 0x7fed340a63e0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fed50114cb0 tx=0x7fed38005c00 comp rx=0 tx=0).stop 2024-04-02T14:40:18.287 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.283+0000 7fed56000640 1 -- 172.21.15.90:0/2606905318 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fed50109040 msgr2=0x7fed50116450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:18.287 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.283+0000 7fed56000640 1 --2- 172.21.15.90:0/2606905318 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fed50109040 0x7fed50116450 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fed4800eab0 tx=0x7fed4800ef70 comp rx=0 tx=0).stop 2024-04-02T14:40:18.287 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.283+0000 7fed56000640 1 -- 172.21.15.90:0/2606905318 shutdown_connections 2024-04-02T14:40:18.287 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.283+0000 7fed56000640 1 --2- 172.21.15.90:0/2606905318 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fed340a3f20 0x7fed340a63e0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:18.287 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.283+0000 7fed56000640 1 --2- 172.21.15.90:0/2606905318 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fed50116b90 0x7fed50113ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:18.287 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.283+0000 7fed56000640 1 --2- 172.21.15.90:0/2606905318 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fed5010c230 0x7fed50113590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:18.288 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.283+0000 7fed56000640 1 --2- 172.21.15.90:0/2606905318 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fed50109040 0x7fed50116450 secure :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fed4800eab0 tx=0x7fed4800ef70 comp rx=0 tx=0).stop 2024-04-02T14:40:18.288 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.283+0000 7fed56000640 1 -- 172.21.15.90:0/2606905318 >> 172.21.15.90:0/2606905318 conn(0x7fed5006d030 msgr2=0x7fed50116f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:40:18.288 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.284+0000 7fed56000640 1 -- 172.21.15.90:0/2606905318 shutdown_connections 2024-04-02T14:40:18.288 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:18.284+0000 7fed56000640 1 -- 172.21.15.90:0/2606905318 wait complete. 2024-04-02T14:40:18.865 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:40:18 smithi092 ceph-mon[36497]: scrub queued for path: / 2024-04-02T14:40:18.865 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:40:18 smithi092 ceph-mon[36497]: scrub summary: idle+waiting paths [/] 2024-04-02T14:40:18.865 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:40:18 smithi092 ceph-mon[36497]: scrub summary: active paths [/] 2024-04-02T14:40:18.865 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:40:18 smithi092 ceph-mon[36497]: scrub summary: idle+waiting paths [/] 2024-04-02T14:40:18.935 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:40:18 smithi177 ceph-mon[36431]: scrub queued for path: / 2024-04-02T14:40:18.935 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:40:18 smithi177 ceph-mon[36431]: scrub summary: idle+waiting paths [/] 2024-04-02T14:40:18.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:40:18 smithi177 ceph-mon[36431]: scrub summary: active paths [/] 2024-04-02T14:40:18.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:40:18 smithi177 ceph-mon[36431]: scrub summary: idle+waiting paths [/] 2024-04-02T14:40:18.937 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:40:18 smithi090 ceph-mon[31674]: scrub queued for path: / 2024-04-02T14:40:18.938 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:40:18 smithi090 ceph-mon[31674]: scrub summary: idle+waiting paths [/] 2024-04-02T14:40:18.938 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:40:18 smithi090 ceph-mon[31674]: scrub summary: active paths [/] 2024-04-02T14:40:18.938 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:40:18 smithi090 ceph-mon[31674]: scrub summary: idle+waiting paths [/] 2024-04-02T14:40:19.859 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:40:19 smithi092 ceph-mon[36497]: pgmap v166: 97 pgs: 97 active+clean; 3.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 26 MiB/s wr, 1.73k op/s 2024-04-02T14:40:19.935 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:40:19 smithi177 ceph-mon[36431]: pgmap v166: 97 pgs: 97 active+clean; 3.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 26 MiB/s wr, 1.73k op/s 2024-04-02T14:40:19.937 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:40:19 smithi090 ceph-mon[31674]: pgmap v166: 97 pgs: 97 active+clean; 3.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 26 MiB/s wr, 1.73k op/s 2024-04-02T14:40:21.831 INFO:teuthology.orchestra.run.smithi090.stderr:progress: 290.0 s, 2744.3 tps, lat 36.314 ms stddev 42.305 2024-04-02T14:40:21.859 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:40:21 smithi092 ceph-mon[36497]: pgmap v167: 97 pgs: 97 active+clean; 3.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 17 MiB/s wr, 1.27k op/s 2024-04-02T14:40:21.935 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:40:21 smithi177 ceph-mon[36431]: pgmap v167: 97 pgs: 97 active+clean; 3.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 17 MiB/s wr, 1.27k op/s 2024-04-02T14:40:21.937 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:40:21 smithi090 ceph-mon[31674]: pgmap v167: 97 pgs: 97 active+clean; 3.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 17 MiB/s wr, 1.27k op/s 2024-04-02T14:40:22.859 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:40:22 smithi092 ceph-mon[36497]: scrub summary: idle 2024-04-02T14:40:22.935 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:40:22 smithi177 ceph-mon[36431]: scrub summary: idle 2024-04-02T14:40:22.937 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:40:22 smithi090 ceph-mon[31674]: scrub summary: idle 2024-04-02T14:40:23.859 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:40:23 smithi092 ceph-mon[36497]: pgmap v168: 97 pgs: 97 active+clean; 3.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 9.0 MiB/s wr, 801 op/s 2024-04-02T14:40:23.935 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:40:23 smithi177 ceph-mon[36431]: pgmap v168: 97 pgs: 97 active+clean; 3.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 9.0 MiB/s wr, 801 op/s 2024-04-02T14:40:23.937 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:40:23 smithi090 ceph-mon[31674]: pgmap v168: 97 pgs: 97 active+clean; 3.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 9.0 MiB/s wr, 801 op/s 2024-04-02T14:40:25.859 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:40:25 smithi092 ceph-mon[36497]: pgmap v169: 97 pgs: 97 active+clean; 3.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 12 MiB/s wr, 1.10k op/s 2024-04-02T14:40:25.935 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:40:25 smithi177 ceph-mon[36431]: pgmap v169: 97 pgs: 97 active+clean; 3.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 12 MiB/s wr, 1.10k op/s 2024-04-02T14:40:25.937 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:40:25 smithi090 ceph-mon[31674]: pgmap v169: 97 pgs: 97 active+clean; 3.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 12 MiB/s wr, 1.10k op/s 2024-04-02T14:40:26.830 INFO:teuthology.orchestra.run.smithi090.stderr:progress: 295.0 s, 3503.6 tps, lat 28.626 ms stddev 33.855 2024-04-02T14:40:27.859 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:40:27 smithi092 ceph-mon[36497]: pgmap v170: 97 pgs: 97 active+clean; 3.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 6.8 MiB/s wr, 676 op/s 2024-04-02T14:40:27.935 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:40:27 smithi177 ceph-mon[36431]: pgmap v170: 97 pgs: 97 active+clean; 3.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 6.8 MiB/s wr, 676 op/s 2024-04-02T14:40:27.937 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:40:27 smithi090 ceph-mon[31674]: pgmap v170: 97 pgs: 97 active+clean; 3.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 6.8 MiB/s wr, 676 op/s 2024-04-02T14:40:29.859 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:40:29 smithi092 ceph-mon[36497]: pgmap v171: 97 pgs: 97 active+clean; 3.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 11 MiB/s wr, 917 op/s 2024-04-02T14:40:29.935 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:40:29 smithi177 ceph-mon[36431]: pgmap v171: 97 pgs: 97 active+clean; 3.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 11 MiB/s wr, 917 op/s 2024-04-02T14:40:29.937 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:40:29 smithi090 ceph-mon[31674]: pgmap v171: 97 pgs: 97 active+clean; 3.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 11 MiB/s wr, 917 op/s 2024-04-02T14:40:31.475 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.473+0000 7ff981200640 1 -- 172.21.15.90:0/3345695552 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff97c0ffec0 msgr2=0x7ff97c1002a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:31.475 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.473+0000 7ff981200640 1 --2- 172.21.15.90:0/3345695552 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff97c0ffec0 0x7ff97c1002a0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7ff96c009930 tx=0x7ff96c02bc20 comp rx=0 tx=0).stop 2024-04-02T14:40:31.475 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.474+0000 7ff981200640 1 -- 172.21.15.90:0/3345695552 shutdown_connections 2024-04-02T14:40:31.476 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.474+0000 7ff981200640 1 --2- 172.21.15.90:0/3345695552 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff97c104e80 0x7ff97c105330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:31.476 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.474+0000 7ff981200640 1 --2- 172.21.15.90:0/3345695552 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff97c1007e0 0x7ff97c104940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:31.476 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.474+0000 7ff981200640 1 --2- 172.21.15.90:0/3345695552 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff97c0ffec0 0x7ff97c1002a0 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:31.476 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.474+0000 7ff981200640 1 -- 172.21.15.90:0/3345695552 >> 172.21.15.90:0/3345695552 conn(0x7ff97c076b90 msgr2=0x7ff97c0fdee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:40:31.476 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.475+0000 7ff981200640 1 -- 172.21.15.90:0/3345695552 shutdown_connections 2024-04-02T14:40:31.476 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.475+0000 7ff981200640 1 -- 172.21.15.90:0/3345695552 wait complete. 2024-04-02T14:40:31.476 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.476+0000 7ff981200640 1 Processor -- start 2024-04-02T14:40:31.478 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.478+0000 7ff981200640 1 -- start start 2024-04-02T14:40:31.479 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.478+0000 7ff981200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff97c0ffec0 0x7ff97c1a46a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:31.479 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.478+0000 7ff981200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff97c1007e0 0x7ff97c1a4be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:31.479 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.478+0000 7ff981200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff97c104e80 0x7ff97c10c890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:31.479 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.478+0000 7ff981200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7ff97c111fe0 con 0x7ff97c0ffec0 2024-04-02T14:40:31.479 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.478+0000 7ff981200640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7ff97c107a90 con 0x7ff97c1007e0 2024-04-02T14:40:31.479 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.479+0000 7ff981200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7ff97c112160 con 0x7ff97c104e80 2024-04-02T14:40:31.479 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.479+0000 7ff979400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff97c1007e0 0x7ff97c1a4be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:31.480 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.479+0000 7ff979400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff97c1007e0 0x7ff97c1a4be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:51660/0 (socket says 172.21.15.90:51660) 2024-04-02T14:40:31.480 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.479+0000 7ff97a800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff97c104e80 0x7ff97c10c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:31.480 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.479+0000 7ff979400640 1 -- 172.21.15.90:0/3152904745 learned_addr learned my addr 172.21.15.90:0/3152904745 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:40:31.480 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.479+0000 7ff979e00640 1 --2- 172.21.15.90:0/3152904745 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff97c0ffec0 0x7ff97c1a46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:31.480 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.479+0000 7ff979400640 1 -- 172.21.15.90:0/3152904745 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff97c104e80 msgr2=0x7ff97c10c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:31.481 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.479+0000 7ff979400640 1 --2- 172.21.15.90:0/3152904745 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff97c104e80 0x7ff97c10c890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:31.481 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.479+0000 7ff979400640 1 -- 172.21.15.90:0/3152904745 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff97c0ffec0 msgr2=0x7ff97c1a46a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:31.481 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.479+0000 7ff979400640 1 --2- 172.21.15.90:0/3152904745 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff97c0ffec0 0x7ff97c1a46a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:31.481 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.479+0000 7ff979400640 1 -- 172.21.15.90:0/3152904745 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff97c10d150 con 0x7ff97c1007e0 2024-04-02T14:40:31.481 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.479+0000 7ff97a800640 1 --2- 172.21.15.90:0/3152904745 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff97c104e80 0x7ff97c10c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:40:31.481 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.480+0000 7ff979400640 1 --2- 172.21.15.90:0/3152904745 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff97c1007e0 0x7ff97c1a4be0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7ff96800e9f0 tx=0x7ff96800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:40:31.481 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.480+0000 7ff972a00640 1 -- 172.21.15.90:0/3152904745 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff96800ccc0 con 0x7ff97c1007e0 2024-04-02T14:40:31.481 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.480+0000 7ff981200640 1 -- 172.21.15.90:0/3152904745 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff97c10d440 con 0x7ff97c1007e0 2024-04-02T14:40:31.481 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.480+0000 7ff972a00640 1 -- 172.21.15.90:0/3152904745 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff96800ce60 con 0x7ff97c1007e0 2024-04-02T14:40:31.481 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.480+0000 7ff972a00640 1 -- 172.21.15.90:0/3152904745 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff968010690 con 0x7ff97c1007e0 2024-04-02T14:40:31.481 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.480+0000 7ff981200640 1 -- 172.21.15.90:0/3152904745 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff97c1ab620 con 0x7ff97c1007e0 2024-04-02T14:40:31.485 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.485+0000 7ff983e15740 1 Processor -- start 2024-04-02T14:40:31.485 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.485+0000 7ff983e15740 1 -- start start 2024-04-02T14:40:31.486 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.485+0000 7ff972a00640 1 -- 172.21.15.90:0/3152904745 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7ff968010830 con 0x7ff97c1007e0 2024-04-02T14:40:31.486 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.486+0000 7ff983e15740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cfea515eb0 0x55cfea516290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:31.486 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.486+0000 7ff983e15740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55cfea5167d0 0x55cfea51ee70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:31.486 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.486+0000 7ff983e15740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cfea51f3b0 0x55cfea5217a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:31.486 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.486+0000 7ff983e15740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55cfea47ad70 con 0x55cfea515eb0 2024-04-02T14:40:31.486 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.486+0000 7ff983e15740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55cfea4232a0 con 0x55cfea5167d0 2024-04-02T14:40:31.486 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.486+0000 7ff983e15740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55cfea476d20 con 0x55cfea51f3b0 2024-04-02T14:40:31.486 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.486+0000 7ff979e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55cfea5167d0 0x55cfea51ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:31.486 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.486+0000 7ff979e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55cfea5167d0 0x55cfea51ee70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:51664/0 (socket says 172.21.15.90:51664) 2024-04-02T14:40:31.487 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.486+0000 7ff979e00640 1 -- 172.21.15.90:0/4077274660 learned_addr learned my addr 172.21.15.90:0/4077274660 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:40:31.487 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.487+0000 7ff97a800640 1 --2- 172.21.15.90:0/4077274660 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cfea515eb0 0x55cfea516290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:31.487 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.487+0000 7ff979400640 1 --2- 172.21.15.90:0/4077274660 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cfea51f3b0 0x55cfea5217a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:31.487 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.487+0000 7ff979e00640 1 -- 172.21.15.90:0/4077274660 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cfea51f3b0 msgr2=0x55cfea5217a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:40:31.487 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.487+0000 7ff979e00640 1 --2- 172.21.15.90:0/4077274660 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cfea51f3b0 0x55cfea5217a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:31.489 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.487+0000 7ff979e00640 1 -- 172.21.15.90:0/4077274660 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cfea515eb0 msgr2=0x55cfea516290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:40:31.489 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.488+0000 7ff979e00640 1 --2- 172.21.15.90:0/4077274660 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cfea515eb0 0x55cfea516290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:31.490 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.488+0000 7ff972a00640 1 --2- 172.21.15.90:0/3152904745 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7ff9600a3e10 0x7ff9600a62d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:31.490 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.488+0000 7ff979e00640 1 -- 172.21.15.90:0/4077274660 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cfea437840 con 0x55cfea5167d0 2024-04-02T14:40:31.490 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.488+0000 7ff972a00640 1 -- 172.21.15.90:0/3152904745 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7ff9680d0bd0 con 0x7ff97c1007e0 2024-04-02T14:40:31.490 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.488+0000 7ff979400640 1 --2- 172.21.15.90:0/4077274660 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cfea51f3b0 0x55cfea5217a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:40:31.490 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.488+0000 7ff97a800640 1 --2- 172.21.15.90:0/4077274660 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cfea515eb0 0x55cfea516290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:40:31.490 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.489+0000 7ff979400640 1 --2- 172.21.15.90:0/3152904745 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7ff9600a3e10 0x7ff9600a62d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:31.490 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.489+0000 7ff979e00640 1 --2- 172.21.15.90:0/4077274660 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55cfea5167d0 0x55cfea51ee70 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7ff96c02c1b0 tx=0x7ff96c0095f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=45016505a91a5178 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:40:31.490 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.489+0000 7ff967400640 1 -- 172.21.15.90:0/4077274660 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff96c044020 con 0x55cfea5167d0 2024-04-02T14:40:31.490 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.489+0000 7ff967400640 1 -- 172.21.15.90:0/4077274660 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff96c02c870 con 0x55cfea5167d0 2024-04-02T14:40:31.490 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.489+0000 7ff967400640 1 -- 172.21.15.90:0/4077274660 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff96c0046c0 con 0x55cfea5167d0 2024-04-02T14:40:31.490 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.489+0000 7ff979400640 1 --2- 172.21.15.90:0/3152904745 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7ff9600a3e10 0x7ff9600a62d0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7ff9680ce3e0 tx=0x7ff9680d4720 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:40:31.491 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.490+0000 7ff983e15740 1 -- 172.21.15.90:0/4077274660 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55cfea5167d0 msgr2=0x55cfea51ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:31.492 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.490+0000 7ff983e15740 1 --2- 172.21.15.90:0/4077274660 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55cfea5167d0 0x55cfea51ee70 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7ff96c02c1b0 tx=0x7ff96c0095f0 comp rx=0 tx=0).stop 2024-04-02T14:40:31.492 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.490+0000 7ff983e15740 1 -- 172.21.15.90:0/4077274660 shutdown_connections 2024-04-02T14:40:31.492 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.490+0000 7ff983e15740 1 --2- 172.21.15.90:0/4077274660 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cfea51f3b0 0x55cfea5217a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:31.492 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.490+0000 7ff983e15740 1 --2- 172.21.15.90:0/4077274660 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55cfea5167d0 0x55cfea51ee70 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:31.492 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.490+0000 7ff983e15740 1 --2- 172.21.15.90:0/4077274660 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cfea515eb0 0x55cfea516290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:31.492 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.490+0000 7ff983e15740 1 -- 172.21.15.90:0/4077274660 >> 172.21.15.90:0/4077274660 conn(0x55cfea5105e0 msgr2=0x55cfea5109c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:40:31.492 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.491+0000 7ff983e15740 1 -- 172.21.15.90:0/4077274660 shutdown_connections 2024-04-02T14:40:31.492 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.491+0000 7ff983e15740 1 -- 172.21.15.90:0/4077274660 wait complete. 2024-04-02T14:40:31.492 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.491+0000 7ff983e15740 1 -- start start 2024-04-02T14:40:31.492 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.491+0000 7ff983e15740 1 Processor -- start 2024-04-02T14:40:31.493 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.492+0000 7ff983e15740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55cfea515eb0 0x55cfea51c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:31.493 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.492+0000 7ff983e15740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cfea5167d0 0x55cfea51c960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:31.493 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.492+0000 7ff983e15740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cfea51f3b0 0x55cfea51cea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:31.493 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.492+0000 7ff983e15740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55cfea522140 con 0x55cfea51f3b0 2024-04-02T14:40:31.494 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.492+0000 7ff983e15740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55cfea436660 con 0x55cfea515eb0 2024-04-02T14:40:31.494 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.492+0000 7ff983e15740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55cfea5222c0 con 0x55cfea5167d0 2024-04-02T14:40:31.494 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.492+0000 7ff983e15740 10 client.0 _ll_register_callbacks cb 0x55cfea436050 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:40:31.494 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.493+0000 7ff97a800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55cfea515eb0 0x55cfea51c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:31.494 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.493+0000 7ff97a800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55cfea515eb0 0x55cfea51c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:51676/0 (socket says 172.21.15.90:51676) 2024-04-02T14:40:31.494 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.493+0000 7ff97a800640 1 -- 172.21.15.90:0/1745613708 learned_addr learned my addr 172.21.15.90:0/1745613708 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:40:31.494 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.493+0000 7ff97a800640 1 --2- 172.21.15.90:0/1745613708 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cfea51f3b0 0x55cfea51cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:31.494 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.493+0000 7ff97a800640 1 -- 172.21.15.90:0/1745613708 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cfea5167d0 msgr2=0x55cfea51c960 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-02T14:40:31.495 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.493+0000 7ff97a800640 1 --2- 172.21.15.90:0/1745613708 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cfea5167d0 0x55cfea51c960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:31.495 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.493+0000 7ff97a800640 1 -- 172.21.15.90:0/1745613708 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cfea51f3b0 msgr2=0x55cfea51cea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:40:31.495 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.493+0000 7ff97a800640 1 --2- 172.21.15.90:0/1745613708 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cfea51f3b0 0x55cfea51cea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:31.495 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.493+0000 7ff97a800640 1 -- 172.21.15.90:0/1745613708 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cfea445e80 con 0x55cfea515eb0 2024-04-02T14:40:31.497 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.494+0000 7ff95be00640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:40:31.497 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.494+0000 7ff97a800640 1 --2- 172.21.15.90:0/1745613708 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55cfea515eb0 0x55cfea51c420 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7ff97400f6d0 tx=0x7ff974005bd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3cbc25a0d5a50523 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:40:31.497 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.494+0000 7ff95be00640 1 -- 172.21.15.90:0/1745613708 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff974019040 con 0x55cfea515eb0 2024-04-02T14:40:31.497 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.494+0000 7ff95be00640 1 -- 172.21.15.90:0/1745613708 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7ff97400fd10 con 0x55cfea515eb0 2024-04-02T14:40:31.497 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.494+0000 7ff983e15740 1 -- 172.21.15.90:0/1745613708 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff96c02c870 con 0x55cfea515eb0 2024-04-02T14:40:31.497 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.495+0000 7ff95be00640 1 -- 172.21.15.90:0/1745613708 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7ff974019040 con 0x55cfea515eb0 2024-04-02T14:40:31.497 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.495+0000 7ff983e15740 10 client.27125 fetch_fsmap learned FSMap version 27 2024-04-02T14:40:31.497 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.495+0000 7ff983e15740 1 -- 172.21.15.90:0/1745613708 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55cfea18f210 con 0x55cfea515eb0 2024-04-02T14:40:31.498 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.495+0000 7ff95be00640 1 -- 172.21.15.90:0/1745613708 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff96c02c870 con 0x55cfea515eb0 2024-04-02T14:40:31.498 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.495+0000 7ff95be00640 1 -- 172.21.15.90:0/1745613708 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7ff974027a70 con 0x55cfea515eb0 2024-04-02T14:40:31.499 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.496+0000 7ff983e15740 10 client.27125 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:40:31.499 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.496+0000 7ff983e15740 10 client.27125 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-02T14:40:31.499 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.496+0000 7ff983e15740 20 client.27125 populate_metadata read hostname 'smithi090' 2024-04-02T14:40:31.499 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.496+0000 7ff983e15740 1 --2- 172.21.15.90:0/1745613708 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55cfea52f730 0x55cfea531b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:31.499 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.496+0000 7ff983e15740 4 client.27125 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:40:31.499 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.496+0000 7ff983e15740 1 -- 172.21.15.90:0/1745613708 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cfea42acd0 con 0x55cfea52f730 2024-04-02T14:40:31.499 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.496+0000 7ff979e00640 1 --2- 172.21.15.90:0/1745613708 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55cfea52f730 0x55cfea531b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:31.499 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.496+0000 7ff979e00640 1 --2- 172.21.15.90:0/1745613708 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55cfea52f730 0x55cfea531b40 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8320b56da44dcc33 server_cookie=5f2e5e92bdcdf7b9 in_seq=0 out_seq=0 2024-04-02T14:40:31.499 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.496+0000 7ff95be00640 10 client.27125 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:40:31.500 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.497+0000 7ff95be00640 1 -- 172.21.15.90:0/1745613708 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55cfea42acd0 con 0x55cfea52f730 2024-04-02T14:40:31.500 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.497+0000 7ff95be00640 10 client.27125 handle_command_reply: tid=0 2024-04-02T14:40:31.500 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.497+0000 7ff983e15740 1 client.27125 shutdown 2024-04-02T14:40:31.502 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.498+0000 7ff983e15740 1 -- 172.21.15.90:0/1745613708 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55cfea515eb0 msgr2=0x55cfea51c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:31.502 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.498+0000 7ff983e15740 1 --2- 172.21.15.90:0/1745613708 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55cfea515eb0 0x55cfea51c420 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7ff97400f6d0 tx=0x7ff974005bd0 comp rx=0 tx=0).stop 2024-04-02T14:40:31.502 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.498+0000 7ff983e15740 1 -- 172.21.15.90:0/1745613708 shutdown_connections 2024-04-02T14:40:31.503 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.498+0000 7ff983e15740 1 --2- 172.21.15.90:0/1745613708 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55cfea52f730 0x55cfea531b40 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:31.503 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.498+0000 7ff983e15740 1 --2- 172.21.15.90:0/1745613708 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cfea51f3b0 0x55cfea51cea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:31.503 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.498+0000 7ff983e15740 1 --2- 172.21.15.90:0/1745613708 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cfea5167d0 0x55cfea51c960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:31.503 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.498+0000 7ff983e15740 1 --2- 172.21.15.90:0/1745613708 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55cfea515eb0 0x55cfea51c420 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:31.503 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.498+0000 7ff983e15740 1 -- 172.21.15.90:0/1745613708 >> 172.21.15.90:0/1745613708 conn(0x55cfea5105e0 msgr2=0x55cfea512330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:40:31.503 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.498+0000 7ff95be00640 0 client.27125 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:40:31.504 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.499+0000 7ff983e15740 1 -- 172.21.15.90:0/1745613708 shutdown_connections 2024-04-02T14:40:31.504 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.499+0000 7ff983e15740 1 -- 172.21.15.90:0/1745613708 wait complete. 2024-04-02T14:40:31.504 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.499+0000 7ff983e15740 20 client.27125 trim_cache size 0 max 16384 2024-04-02T14:40:31.525 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.523+0000 7ff983e15740 1 Processor -- start 2024-04-02T14:40:31.525 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.523+0000 7ff983e15740 1 -- start start 2024-04-02T14:40:31.525 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.524+0000 7ff983e15740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55cfea515eb0 0x55cfea52b000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:31.526 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.524+0000 7ff983e15740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cfea5167d0 0x55cfea51ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:31.526 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.524+0000 7ff983e15740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cfea51f3b0 0x55cfea537450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:31.526 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.524+0000 7ff983e15740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7ff97c1b3e50 con 0x55cfea5167d0 2024-04-02T14:40:31.526 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.524+0000 7ff983e15740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7ff97c1b5990 con 0x55cfea515eb0 2024-04-02T14:40:31.526 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.524+0000 7ff983e15740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7ff97c1b4520 con 0x55cfea51f3b0 2024-04-02T14:40:31.526 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.524+0000 7ff97a800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55cfea515eb0 0x55cfea52b000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:31.526 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.524+0000 7ff97a800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55cfea515eb0 0x55cfea52b000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:51692/0 (socket says 172.21.15.90:51692) 2024-04-02T14:40:31.526 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.524+0000 7ff979e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cfea5167d0 0x55cfea51ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:31.527 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.524+0000 7ff97a800640 1 -- 172.21.15.90:0/3662218605 learned_addr learned my addr 172.21.15.90:0/3662218605 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:40:31.527 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.524+0000 7ff97a800640 1 --2- 172.21.15.90:0/3662218605 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cfea51f3b0 0x55cfea537450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:31.527 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.525+0000 7ff97a800640 1 -- 172.21.15.90:0/3662218605 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cfea51f3b0 msgr2=0x55cfea537450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:40:31.527 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.525+0000 7ff97a800640 1 --2- 172.21.15.90:0/3662218605 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cfea51f3b0 0x55cfea537450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:31.527 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.525+0000 7ff97a800640 1 -- 172.21.15.90:0/3662218605 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cfea5167d0 msgr2=0x55cfea51ccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:40:31.528 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.525+0000 7ff97a800640 1 --2- 172.21.15.90:0/3662218605 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cfea5167d0 0x55cfea51ccc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:31.528 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.525+0000 7ff97a800640 1 -- 172.21.15.90:0/3662218605 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cfea25b620 con 0x55cfea515eb0 2024-04-02T14:40:31.528 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.525+0000 7ff97a800640 1 --2- 172.21.15.90:0/3662218605 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55cfea515eb0 0x55cfea52b000 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7ff974010e90 tx=0x7ff974000f40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d221c36604c31584 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:40:31.532 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.525+0000 7ff966a00640 1 -- 172.21.15.90:0/3662218605 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff974004c40 con 0x55cfea515eb0 2024-04-02T14:40:31.532 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.525+0000 7ff966a00640 1 -- 172.21.15.90:0/3662218605 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff974004de0 con 0x55cfea515eb0 2024-04-02T14:40:31.532 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.525+0000 7ff966a00640 1 -- 172.21.15.90:0/3662218605 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff974027890 con 0x55cfea515eb0 2024-04-02T14:40:31.532 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.526+0000 7ff983e15740 1 -- 172.21.15.90:0/3662218605 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55cfea515eb0 msgr2=0x55cfea52b000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:31.532 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.526+0000 7ff983e15740 1 --2- 172.21.15.90:0/3662218605 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55cfea515eb0 0x55cfea52b000 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7ff974010e90 tx=0x7ff974000f40 comp rx=0 tx=0).stop 2024-04-02T14:40:31.532 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.527+0000 7ff983e15740 1 -- 172.21.15.90:0/3662218605 shutdown_connections 2024-04-02T14:40:31.532 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.527+0000 7ff983e15740 1 --2- 172.21.15.90:0/3662218605 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cfea51f3b0 0x55cfea537450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:31.533 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.527+0000 7ff983e15740 1 --2- 172.21.15.90:0/3662218605 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cfea5167d0 0x55cfea51ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:31.533 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.527+0000 7ff983e15740 1 --2- 172.21.15.90:0/3662218605 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55cfea515eb0 0x55cfea52b000 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:31.533 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.527+0000 7ff983e15740 1 -- 172.21.15.90:0/3662218605 >> 172.21.15.90:0/3662218605 conn(0x55cfea5105e0 msgr2=0x55cfea5122a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:40:31.533 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.527+0000 7ff983e15740 1 -- 172.21.15.90:0/3662218605 shutdown_connections 2024-04-02T14:40:31.533 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.527+0000 7ff983e15740 1 -- 172.21.15.90:0/3662218605 wait complete. 2024-04-02T14:40:31.533 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.528+0000 7ff983e15740 1 -- start start 2024-04-02T14:40:31.533 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.528+0000 7ff983e15740 1 Processor -- start 2024-04-02T14:40:31.533 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.529+0000 7ff983e15740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cfea515eb0 0x55cfea527430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:31.533 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.529+0000 7ff983e15740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cfea5167d0 0x55cfea527970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:31.534 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.529+0000 7ff983e15740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55cfea51f3b0 0x55cfea5228e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:31.534 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.529+0000 7ff983e15740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7ff97c1b52c0 con 0x55cfea5167d0 2024-04-02T14:40:31.534 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.529+0000 7ff983e15740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7ff97c1b4bf0 con 0x55cfea51f3b0 2024-04-02T14:40:31.534 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.529+0000 7ff983e15740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7ff97c1b1e30 con 0x55cfea515eb0 2024-04-02T14:40:31.534 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.529+0000 7ff983e15740 10 client.0 _ll_register_callbacks cb 0x7ff974010f50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:40:31.534 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.529+0000 7ff97a800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55cfea51f3b0 0x55cfea5228e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:31.534 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.529+0000 7ff979e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cfea5167d0 0x55cfea527970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:31.534 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.529+0000 7ff97a800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55cfea51f3b0 0x55cfea5228e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:51696/0 (socket says 172.21.15.90:51696) 2024-04-02T14:40:31.534 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.529+0000 7ff97a800640 1 -- 172.21.15.90:0/4203159868 learned_addr learned my addr 172.21.15.90:0/4203159868 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:40:31.534 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.529+0000 7ff97a800640 1 --2- 172.21.15.90:0/4203159868 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cfea515eb0 0x55cfea527430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:31.534 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.530+0000 7ff97a800640 1 -- 172.21.15.90:0/4203159868 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cfea515eb0 msgr2=0x55cfea527430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:40:31.534 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.530+0000 7ff97a800640 1 --2- 172.21.15.90:0/4203159868 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cfea515eb0 0x55cfea527430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:31.534 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.530+0000 7ff97a800640 1 -- 172.21.15.90:0/4203159868 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cfea5167d0 msgr2=0x55cfea527970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:40:31.535 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.530+0000 7ff97a800640 1 --2- 172.21.15.90:0/4203159868 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cfea5167d0 0x55cfea527970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:31.535 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.530+0000 7ff97a800640 1 -- 172.21.15.90:0/4203159868 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cfea523110 con 0x55cfea51f3b0 2024-04-02T14:40:31.535 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.530+0000 7ff979e00640 1 --2- 172.21.15.90:0/4203159868 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cfea5167d0 0x55cfea527970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:40:31.535 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.530+0000 7ff97a800640 1 --2- 172.21.15.90:0/4203159868 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55cfea51f3b0 0x55cfea5228e0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7ff97400ffd0 tx=0x7ff974027470 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e6db949f988a538b server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:40:31.535 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.530+0000 7ff95be00640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:40:31.535 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.530+0000 7ff95be00640 1 -- 172.21.15.90:0/4203159868 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9740278e0 con 0x55cfea51f3b0 2024-04-02T14:40:31.535 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.530+0000 7ff95be00640 1 -- 172.21.15.90:0/4203159868 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7ff974027a80 con 0x55cfea51f3b0 2024-04-02T14:40:31.535 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.530+0000 7ff983e15740 1 -- 172.21.15.90:0/4203159868 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff974004de0 con 0x55cfea51f3b0 2024-04-02T14:40:31.535 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.531+0000 7ff95be00640 1 -- 172.21.15.90:0/4203159868 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7ff974027c80 con 0x55cfea51f3b0 2024-04-02T14:40:31.535 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.531+0000 7ff983e15740 10 client.27131 fetch_fsmap learned FSMap version 27 2024-04-02T14:40:31.535 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.531+0000 7ff983e15740 1 -- 172.21.15.90:0/4203159868 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55cfea595830 con 0x55cfea51f3b0 2024-04-02T14:40:31.535 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.531+0000 7ff95be00640 1 -- 172.21.15.90:0/4203159868 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff974004de0 con 0x55cfea51f3b0 2024-04-02T14:40:31.535 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.532+0000 7ff95be00640 1 -- 172.21.15.90:0/4203159868 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7ff974034070 con 0x55cfea51f3b0 2024-04-02T14:40:31.535 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.532+0000 7ff983e15740 10 client.27131 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:40:31.536 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.532+0000 7ff983e15740 10 client.27131 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-02T14:40:31.536 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.532+0000 7ff983e15740 20 client.27131 populate_metadata read hostname 'smithi090' 2024-04-02T14:40:31.536 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.532+0000 7ff983e15740 1 --2- 172.21.15.90:0/4203159868 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55cfea595d40 0x55cfea598130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:31.536 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.532+0000 7ff983e15740 4 client.27131 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-02T14:40:31.536 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.532+0000 7ff983e15740 1 -- 172.21.15.90:0/4203159868 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55cfea4882b0 con 0x55cfea595d40 2024-04-02T14:40:31.536 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.533+0000 7ff979e00640 1 --2- 172.21.15.90:0/4203159868 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55cfea595d40 0x55cfea598130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:31.537 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.533+0000 7ff95be00640 10 client.27131 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:40:31.537 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.533+0000 7ff979e00640 1 --2- 172.21.15.90:0/4203159868 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55cfea595d40 0x55cfea598130 crc :-1 s=READY pgs=163 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=550a1557bdd79144 server_cookie=1a48a40f44f3b798 in_seq=0 out_seq=0 2024-04-02T14:40:31.564 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.564+0000 7ff95be00640 1 -- 172.21.15.90:0/4203159868 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55cfea4882b0 con 0x55cfea595d40 2024-04-02T14:40:31.564 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.564+0000 7ff95be00640 10 client.27131 handle_command_reply: tid=0 2024-04-02T14:40:31.565 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.564+0000 7ff983e15740 1 client.27131 shutdown 2024-04-02T14:40:31.565 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.566+0000 7ff983e15740 1 -- 172.21.15.90:0/4203159868 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55cfea51f3b0 msgr2=0x55cfea5228e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:31.566 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.566+0000 7ff983e15740 1 --2- 172.21.15.90:0/4203159868 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55cfea51f3b0 0x55cfea5228e0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7ff97400ffd0 tx=0x7ff974027470 comp rx=0 tx=0).stop 2024-04-02T14:40:31.566 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.566+0000 7ff983e15740 1 -- 172.21.15.90:0/4203159868 shutdown_connections 2024-04-02T14:40:31.566 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.566+0000 7ff983e15740 1 --2- 172.21.15.90:0/4203159868 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55cfea595d40 0x55cfea598130 crc :-1 s=READY pgs=163 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:31.567 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.566+0000 7ff983e15740 1 --2- 172.21.15.90:0/4203159868 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55cfea51f3b0 0x55cfea5228e0 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:31.567 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.566+0000 7ff983e15740 1 --2- 172.21.15.90:0/4203159868 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cfea5167d0 0x55cfea527970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:31.567 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.566+0000 7ff983e15740 1 --2- 172.21.15.90:0/4203159868 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cfea515eb0 0x55cfea527430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:31.567 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.566+0000 7ff983e15740 1 -- 172.21.15.90:0/4203159868 >> 172.21.15.90:0/4203159868 conn(0x55cfea5105e0 msgr2=0x55cfea50edb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:40:31.567 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.567+0000 7ff95be00640 0 client.27131 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:40:31.567 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.567+0000 7ff983e15740 1 -- 172.21.15.90:0/4203159868 shutdown_connections 2024-04-02T14:40:31.567 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.567+0000 7ff983e15740 1 -- 172.21.15.90:0/4203159868 wait complete. 2024-04-02T14:40:31.568 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.568+0000 7ff983e15740 20 client.27131 trim_cache size 0 max 16384 2024-04-02T14:40:31.568 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:{ 2024-04-02T14:40:31.568 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "message": "", 2024-04-02T14:40:31.568 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "return_code": 0 2024-04-02T14:40:31.568 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:} 2024-04-02T14:40:31.569 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.569+0000 7ff981200640 1 -- 172.21.15.90:0/3152904745 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7ff9600a3e10 msgr2=0x7ff9600a62d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:31.569 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.569+0000 7ff981200640 1 --2- 172.21.15.90:0/3152904745 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7ff9600a3e10 0x7ff9600a62d0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7ff9680ce3e0 tx=0x7ff9680d4720 comp rx=0 tx=0).stop 2024-04-02T14:40:31.569 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.569+0000 7ff981200640 1 -- 172.21.15.90:0/3152904745 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff97c1007e0 msgr2=0x7ff97c1a4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:31.569 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.569+0000 7ff981200640 1 --2- 172.21.15.90:0/3152904745 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff97c1007e0 0x7ff97c1a4be0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7ff96800e9f0 tx=0x7ff96800eeb0 comp rx=0 tx=0).stop 2024-04-02T14:40:31.569 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.569+0000 7ff981200640 1 -- 172.21.15.90:0/3152904745 shutdown_connections 2024-04-02T14:40:31.570 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.569+0000 7ff981200640 1 --2- 172.21.15.90:0/3152904745 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7ff9600a3e10 0x7ff9600a62d0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:31.570 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.569+0000 7ff981200640 1 --2- 172.21.15.90:0/3152904745 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff97c104e80 0x7ff97c10c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:31.570 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.569+0000 7ff981200640 1 --2- 172.21.15.90:0/3152904745 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff97c1007e0 0x7ff97c1a4be0 secure :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7ff96800e9f0 tx=0x7ff96800eeb0 comp rx=0 tx=0).stop 2024-04-02T14:40:31.570 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.569+0000 7ff981200640 1 --2- 172.21.15.90:0/3152904745 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff97c0ffec0 0x7ff97c1a46a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:31.570 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.569+0000 7ff981200640 1 -- 172.21.15.90:0/3152904745 >> 172.21.15.90:0/3152904745 conn(0x7ff97c076b90 msgr2=0x7ff97c102470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:40:31.570 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.570+0000 7ff981200640 1 -- 172.21.15.90:0/3152904745 shutdown_connections 2024-04-02T14:40:31.570 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:31.570+0000 7ff981200640 1 -- 172.21.15.90:0/3152904745 wait complete. 2024-04-02T14:40:31.833 INFO:teuthology.orchestra.run.smithi090.stderr:progress: 300.0 s, 2813.6 tps, lat 35.454 ms stddev 41.823 2024-04-02T14:40:31.859 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:40:31 smithi092 ceph-mon[36497]: pgmap v172: 97 pgs: 97 active+clean; 3.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 10 MiB/s wr, 796 op/s 2024-04-02T14:40:31.935 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:40:31 smithi177 ceph-mon[36431]: pgmap v172: 97 pgs: 97 active+clean; 3.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 10 MiB/s wr, 796 op/s 2024-04-02T14:40:31.937 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:40:31 smithi090 ceph-mon[31674]: pgmap v172: 97 pgs: 97 active+clean; 3.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 10 MiB/s wr, 796 op/s 2024-04-02T14:40:33.859 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:40:33 smithi092 ceph-mon[36497]: pgmap v173: 97 pgs: 97 active+clean; 3.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 8.7 MiB/s wr, 638 op/s 2024-04-02T14:40:33.935 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:40:33 smithi177 ceph-mon[36431]: pgmap v173: 97 pgs: 97 active+clean; 3.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 8.7 MiB/s wr, 638 op/s 2024-04-02T14:40:33.937 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:40:33 smithi090 ceph-mon[31674]: pgmap v173: 97 pgs: 97 active+clean; 3.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 8.7 MiB/s wr, 638 op/s 2024-04-02T14:40:35.859 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:40:35 smithi092 ceph-mon[36497]: pgmap v174: 97 pgs: 97 active+clean; 3.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 12 MiB/s wr, 955 op/s 2024-04-02T14:40:35.935 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:40:35 smithi177 ceph-mon[36431]: pgmap v174: 97 pgs: 97 active+clean; 3.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 12 MiB/s wr, 955 op/s 2024-04-02T14:40:35.937 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:40:35 smithi090 ceph-mon[31674]: pgmap v174: 97 pgs: 97 active+clean; 3.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 12 MiB/s wr, 955 op/s 2024-04-02T14:40:36.830 INFO:teuthology.orchestra.run.smithi090.stderr:progress: 305.0 s, 2954.8 tps, lat 33.769 ms stddev 38.534 2024-04-02T14:40:37.859 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:40:37 smithi092 ceph-mon[36497]: pgmap v175: 97 pgs: 97 active+clean; 3.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 8.8 MiB/s wr, 657 op/s 2024-04-02T14:40:37.935 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:40:37 smithi177 ceph-mon[36431]: pgmap v175: 97 pgs: 97 active+clean; 3.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 8.8 MiB/s wr, 657 op/s 2024-04-02T14:40:37.937 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:40:37 smithi090 ceph-mon[31674]: pgmap v175: 97 pgs: 97 active+clean; 3.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 8.8 MiB/s wr, 657 op/s 2024-04-02T14:40:39.859 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:40:39 smithi092 ceph-mon[36497]: pgmap v176: 97 pgs: 97 active+clean; 3.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 9.8 MiB/s wr, 781 op/s 2024-04-02T14:40:39.935 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:40:39 smithi177 ceph-mon[36431]: pgmap v176: 97 pgs: 97 active+clean; 3.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 9.8 MiB/s wr, 781 op/s 2024-04-02T14:40:39.937 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:40:39 smithi090 ceph-mon[31674]: pgmap v176: 97 pgs: 97 active+clean; 3.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 9.8 MiB/s wr, 781 op/s 2024-04-02T14:40:41.830 INFO:teuthology.orchestra.run.smithi090.stderr:progress: 310.0 s, 2625.2 tps, lat 37.882 ms stddev 44.342 2024-04-02T14:40:41.861 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:40:41 smithi092 ceph-mon[36497]: pgmap v177: 97 pgs: 97 active+clean; 3.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 7.9 MiB/s wr, 718 op/s 2024-04-02T14:40:41.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:40:41 smithi177 ceph-mon[36431]: pgmap v177: 97 pgs: 97 active+clean; 3.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 7.9 MiB/s wr, 718 op/s 2024-04-02T14:40:41.937 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:40:41 smithi090 ceph-mon[31674]: pgmap v177: 97 pgs: 97 active+clean; 3.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 7.9 MiB/s wr, 718 op/s 2024-04-02T14:40:43.859 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:40:43 smithi092 ceph-mon[36497]: pgmap v178: 97 pgs: 97 active+clean; 3.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 6.6 MiB/s wr, 617 op/s 2024-04-02T14:40:43.935 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:40:43 smithi177 ceph-mon[36431]: pgmap v178: 97 pgs: 97 active+clean; 3.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 6.6 MiB/s wr, 617 op/s 2024-04-02T14:40:43.938 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:40:43 smithi090 ceph-mon[31674]: pgmap v178: 97 pgs: 97 active+clean; 3.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 6.6 MiB/s wr, 617 op/s 2024-04-02T14:40:44.763 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.760+0000 7f671ba00640 1 -- 172.21.15.90:0/1912627327 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6714101af0 msgr2=0x7f6714101f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:44.763 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.760+0000 7f671ba00640 1 --2- 172.21.15.90:0/1912627327 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6714101af0 0x7f6714101f70 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f6700009890 tx=0x7f670002bb70 comp rx=0 tx=0).stop 2024-04-02T14:40:44.763 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.762+0000 7f671ba00640 1 -- 172.21.15.90:0/1912627327 shutdown_connections 2024-04-02T14:40:44.763 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.762+0000 7f671ba00640 1 --2- 172.21.15.90:0/1912627327 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f67141024b0 0x7f671410e9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:44.763 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.762+0000 7f671ba00640 1 --2- 172.21.15.90:0/1912627327 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6714101af0 0x7f6714101f70 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:44.764 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.762+0000 7f671ba00640 1 --2- 172.21.15.90:0/1912627327 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6714107b90 0x7f6714107f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:44.764 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.762+0000 7f671ba00640 1 -- 172.21.15.90:0/1912627327 >> 172.21.15.90:0/1912627327 conn(0x7f67140fd5c0 msgr2=0x7f67140ff9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:40:44.764 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.762+0000 7f671ba00640 1 -- 172.21.15.90:0/1912627327 shutdown_connections 2024-04-02T14:40:44.764 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.763+0000 7f671ba00640 1 -- 172.21.15.90:0/1912627327 wait complete. 2024-04-02T14:40:44.765 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.764+0000 7f671ba00640 1 Processor -- start 2024-04-02T14:40:44.765 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.764+0000 7f671ba00640 1 -- start start 2024-04-02T14:40:44.766 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.765+0000 7f671ba00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6714101af0 0x7f6714116210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:44.766 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.765+0000 7f671ba00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f67141024b0 0x7f6714116750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:44.766 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.765+0000 7f671ba00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6714107b90 0x7f671411bf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:44.766 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.766+0000 7f671ba00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f67140759f0 con 0x7f67141024b0 2024-04-02T14:40:44.767 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.766+0000 7f671ba00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f6714075870 con 0x7f6714101af0 2024-04-02T14:40:44.767 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.766+0000 7f671ba00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f6714075b70 con 0x7f6714107b90 2024-04-02T14:40:44.767 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.767+0000 7f6719000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6714101af0 0x7f6714116210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:44.768 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.767+0000 7f6719000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6714101af0 0x7f6714116210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:33984/0 (socket says 172.21.15.90:33984) 2024-04-02T14:40:44.768 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.767+0000 7f6719000640 1 -- 172.21.15.90:0/1443204311 learned_addr learned my addr 172.21.15.90:0/1443204311 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:40:44.768 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.767+0000 7f6719000640 1 -- 172.21.15.90:0/1443204311 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6714107b90 msgr2=0x7f671411bf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:44.768 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.767+0000 7f6719000640 1 --2- 172.21.15.90:0/1443204311 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6714107b90 0x7f671411bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:44.769 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.767+0000 7f6719000640 1 -- 172.21.15.90:0/1443204311 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f67141024b0 msgr2=0x7f6714116750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:44.769 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.767+0000 7f6719000640 1 --2- 172.21.15.90:0/1443204311 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f67141024b0 0x7f6714116750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:44.769 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.767+0000 7f6719000640 1 -- 172.21.15.90:0/1443204311 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f671411c7e0 con 0x7f6714101af0 2024-04-02T14:40:44.769 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.769+0000 7f6719000640 1 --2- 172.21.15.90:0/1443204311 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6714101af0 0x7f6714116210 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f670800e9c0 tx=0x7f670800ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:40:44.770 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.770+0000 7f6711600640 1 -- 172.21.15.90:0/1443204311 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f670800ccc0 con 0x7f6714101af0 2024-04-02T14:40:44.772 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.770+0000 7f6711600640 1 -- 172.21.15.90:0/1443204311 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6708005500 con 0x7f6714101af0 2024-04-02T14:40:44.772 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.770+0000 7f6711600640 1 -- 172.21.15.90:0/1443204311 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6708010430 con 0x7f6714101af0 2024-04-02T14:40:44.772 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.770+0000 7f671ba00640 1 -- 172.21.15.90:0/1443204311 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f671411cad0 con 0x7f6714101af0 2024-04-02T14:40:44.773 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.770+0000 7f671ba00640 1 -- 172.21.15.90:0/1443204311 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67141115a0 con 0x7f6714101af0 2024-04-02T14:40:44.778 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.777+0000 7f6711600640 1 -- 172.21.15.90:0/1443204311 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f67080046c0 con 0x7f6714101af0 2024-04-02T14:40:44.778 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.778+0000 7f671e5ee740 1 Processor -- start 2024-04-02T14:40:44.782 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.779+0000 7f671e5ee740 1 -- start start 2024-04-02T14:40:44.782 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.780+0000 7f671e5ee740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563fd1879eb0 0x563fd187a290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:44.782 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.780+0000 7f671e5ee740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563fd187a7d0 0x563fd1882e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:44.782 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.780+0000 7f671e5ee740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563fd18833b0 0x563fd18857a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:44.782 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.780+0000 7f671e5ee740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x563fd17ded70 con 0x563fd187a7d0 2024-04-02T14:40:44.782 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.780+0000 7f671e5ee740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x563fd17872a0 con 0x563fd1879eb0 2024-04-02T14:40:44.782 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.780+0000 7f671e5ee740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x563fd17dad20 con 0x563fd18833b0 2024-04-02T14:40:44.782 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.780+0000 7f6719a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563fd1879eb0 0x563fd187a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:44.783 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.780+0000 7f6719a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563fd1879eb0 0x563fd187a290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:34000/0 (socket says 172.21.15.90:34000) 2024-04-02T14:40:44.783 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.781+0000 7f6719a00640 1 -- 172.21.15.90:0/1888924636 learned_addr learned my addr 172.21.15.90:0/1888924636 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:40:44.783 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.781+0000 7f6719a00640 1 -- 172.21.15.90:0/1888924636 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563fd18833b0 msgr2=0x563fd18857a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-02T14:40:44.783 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.781+0000 7f6719a00640 1 --2- 172.21.15.90:0/1888924636 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563fd18833b0 0x563fd18857a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:44.783 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.781+0000 7f6719a00640 1 -- 172.21.15.90:0/1888924636 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563fd187a7d0 msgr2=0x563fd1882e70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-02T14:40:44.783 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.781+0000 7f6719a00640 1 --2- 172.21.15.90:0/1888924636 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563fd187a7d0 0x563fd1882e70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:44.783 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.781+0000 7f6719a00640 1 -- 172.21.15.90:0/1888924636 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563fd179b840 con 0x563fd1879eb0 2024-04-02T14:40:44.784 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.781+0000 7f6711600640 1 --2- 172.21.15.90:0/1443204311 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f66fc0a3f70 0x7f66fc0a6430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:44.784 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.782+0000 7f6711600640 1 -- 172.21.15.90:0/1443204311 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f67080d0b70 con 0x7f6714101af0 2024-04-02T14:40:44.784 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.782+0000 7f6719a00640 1 --2- 172.21.15.90:0/1888924636 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563fd1879eb0 0x563fd187a290 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f670c009e70 tx=0x7f670c00e720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=41ecef344ce60b40 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:40:44.784 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.782+0000 7f6719000640 1 --2- 172.21.15.90:0/1443204311 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f66fc0a3f70 0x7f66fc0a6430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:44.785 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.785+0000 7f6706000640 1 -- 172.21.15.90:0/1888924636 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f670c004d20 con 0x563fd1879eb0 2024-04-02T14:40:44.786 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.785+0000 7f6706000640 1 -- 172.21.15.90:0/1888924636 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f670c00ee70 con 0x563fd1879eb0 2024-04-02T14:40:44.786 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.785+0000 7f6706000640 1 -- 172.21.15.90:0/1888924636 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f670c005280 con 0x563fd1879eb0 2024-04-02T14:40:44.787 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.786+0000 7f6719000640 1 --2- 172.21.15.90:0/1443204311 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f66fc0a3f70 0x7f66fc0a6430 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x563fd1879980 tx=0x7f670800ce60 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:40:44.787 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.786+0000 7f671e5ee740 1 -- 172.21.15.90:0/1888924636 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563fd1879eb0 msgr2=0x563fd187a290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:44.787 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.786+0000 7f671e5ee740 1 --2- 172.21.15.90:0/1888924636 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563fd1879eb0 0x563fd187a290 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f670c009e70 tx=0x7f670c00e720 comp rx=0 tx=0).stop 2024-04-02T14:40:44.789 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.788+0000 7f671e5ee740 1 -- 172.21.15.90:0/1888924636 shutdown_connections 2024-04-02T14:40:44.789 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.788+0000 7f671e5ee740 1 --2- 172.21.15.90:0/1888924636 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563fd18833b0 0x563fd18857a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:44.789 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.788+0000 7f671e5ee740 1 --2- 172.21.15.90:0/1888924636 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563fd187a7d0 0x563fd1882e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:44.789 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.788+0000 7f671e5ee740 1 --2- 172.21.15.90:0/1888924636 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563fd1879eb0 0x563fd187a290 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:44.789 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.788+0000 7f671e5ee740 1 -- 172.21.15.90:0/1888924636 >> 172.21.15.90:0/1888924636 conn(0x563fd18745e0 msgr2=0x563fd18749c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:40:44.790 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.789+0000 7f671e5ee740 1 -- 172.21.15.90:0/1888924636 shutdown_connections 2024-04-02T14:40:44.790 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.789+0000 7f671e5ee740 1 -- 172.21.15.90:0/1888924636 wait complete. 2024-04-02T14:40:44.790 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.790+0000 7f671e5ee740 1 -- start start 2024-04-02T14:40:44.791 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.790+0000 7f671e5ee740 1 Processor -- start 2024-04-02T14:40:44.792 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.791+0000 7f671e5ee740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563fd1879eb0 0x563fd18803b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:44.792 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.791+0000 7f671e5ee740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563fd187a7d0 0x563fd18808f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:44.792 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.791+0000 7f671e5ee740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563fd18833b0 0x563fd1880e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:44.792 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.791+0000 7f671e5ee740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x563fd18971b0 con 0x563fd1879eb0 2024-04-02T14:40:44.792 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.791+0000 7f671e5ee740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x563fd179a660 con 0x563fd18833b0 2024-04-02T14:40:44.792 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.791+0000 7f671e5ee740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x563fd1897330 con 0x563fd187a7d0 2024-04-02T14:40:44.792 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.791+0000 7f671e5ee740 10 client.0 _ll_register_callbacks cb 0x563fd179a050 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:40:44.793 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.792+0000 7f6713e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563fd187a7d0 0x563fd18808f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:44.793 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.792+0000 7f6713e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563fd187a7d0 0x563fd18808f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:49692/0 (socket says 172.21.15.90:49692) 2024-04-02T14:40:44.793 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.792+0000 7f6713e00640 1 -- 172.21.15.90:0/1451065915 learned_addr learned my addr 172.21.15.90:0/1451065915 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:40:44.793 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.792+0000 7f6719a00640 1 --2- 172.21.15.90:0/1451065915 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563fd18833b0 0x563fd1880e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:44.793 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.792+0000 7f6719a00640 1 --2- 172.21.15.90:0/1451065915 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563fd1879eb0 0x563fd18803b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:44.794 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.793+0000 7f6719a00640 1 -- 172.21.15.90:0/1451065915 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563fd187a7d0 msgr2=0x563fd18808f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:40:44.794 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.793+0000 7f6719a00640 1 --2- 172.21.15.90:0/1451065915 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563fd187a7d0 0x563fd18808f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:44.794 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.793+0000 7f6719a00640 1 -- 172.21.15.90:0/1451065915 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563fd1879eb0 msgr2=0x563fd18803b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:40:44.794 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.793+0000 7f6719a00640 1 --2- 172.21.15.90:0/1451065915 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563fd1879eb0 0x563fd18803b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:44.794 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.793+0000 7f6719a00640 1 -- 172.21.15.90:0/1451065915 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563fd17a9e80 con 0x563fd18833b0 2024-04-02T14:40:44.795 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.794+0000 7f6719a00640 1 --2- 172.21.15.90:0/1451065915 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563fd18833b0 0x563fd1880e30 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f670c004780 tx=0x7f670c004680 comp rx=0 tx=0).ready entity=mon.2 client_cookie=21f7f5ff348f3eda server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:40:44.795 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.794+0000 7f66faa00640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:40:44.795 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.795+0000 7f66faa00640 1 -- 172.21.15.90:0/1451065915 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f670c022040 con 0x563fd18833b0 2024-04-02T14:40:44.797 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.797+0000 7f66faa00640 1 -- 172.21.15.90:0/1451065915 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f670c003e90 con 0x563fd18833b0 2024-04-02T14:40:44.797 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.797+0000 7f671e5ee740 1 -- 172.21.15.90:0/1451065915 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f670c00ee70 con 0x563fd18833b0 2024-04-02T14:40:44.798 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.797+0000 7f66faa00640 1 -- 172.21.15.90:0/1451065915 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f670c004b20 con 0x563fd18833b0 2024-04-02T14:40:44.798 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.797+0000 7f671e5ee740 10 client.27143 fetch_fsmap learned FSMap version 27 2024-04-02T14:40:44.798 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.797+0000 7f671e5ee740 1 -- 172.21.15.90:0/1451065915 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x563fd14f3210 con 0x563fd18833b0 2024-04-02T14:40:44.798 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.798+0000 7f66faa00640 1 -- 172.21.15.90:0/1451065915 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f670c019b50 con 0x563fd18833b0 2024-04-02T14:40:44.800 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.798+0000 7f66faa00640 1 -- 172.21.15.90:0/1451065915 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f670c028970 con 0x563fd18833b0 2024-04-02T14:40:44.800 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.799+0000 7f671e5ee740 10 client.27143 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:40:44.800 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.799+0000 7f671e5ee740 10 client.27143 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-02T14:40:44.800 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.799+0000 7f671e5ee740 20 client.27143 populate_metadata read hostname 'smithi090' 2024-04-02T14:40:44.800 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.799+0000 7f671e5ee740 1 --2- 172.21.15.90:0/1451065915 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x563fd18a46d0 0x563fd18a6ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:44.800 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.799+0000 7f671e5ee740 4 client.27143 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:40:44.800 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.799+0000 7f671e5ee740 1 -- 172.21.15.90:0/1451065915 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f670c00ee70 con 0x563fd18a46d0 2024-04-02T14:40:44.800 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.799+0000 7f6713e00640 1 --2- 172.21.15.90:0/1451065915 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x563fd18a46d0 0x563fd18a6ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:44.803 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.802+0000 7f6713e00640 1 --2- 172.21.15.90:0/1451065915 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x563fd18a46d0 0x563fd18a6ac0 crc :-1 s=READY pgs=164 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2bc6d491664096c3 server_cookie=b11a03ef818b7495 in_seq=0 out_seq=0 2024-04-02T14:40:44.803 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.802+0000 7f66faa00640 10 client.27143 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:40:44.804 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.803+0000 7f66faa00640 1 -- 172.21.15.90:0/1451065915 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x7f670c00ee70 con 0x563fd18a46d0 2024-04-02T14:40:44.804 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.803+0000 7f66faa00640 10 client.27143 handle_command_reply: tid=0 2024-04-02T14:40:44.804 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.804+0000 7f671e5ee740 1 client.27143 shutdown 2024-04-02T14:40:44.805 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.804+0000 7f671e5ee740 1 -- 172.21.15.90:0/1451065915 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563fd18833b0 msgr2=0x563fd1880e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:44.805 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.804+0000 7f671e5ee740 1 --2- 172.21.15.90:0/1451065915 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563fd18833b0 0x563fd1880e30 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f670c004780 tx=0x7f670c004680 comp rx=0 tx=0).stop 2024-04-02T14:40:44.805 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.805+0000 7f671e5ee740 1 -- 172.21.15.90:0/1451065915 shutdown_connections 2024-04-02T14:40:44.805 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.805+0000 7f671e5ee740 1 --2- 172.21.15.90:0/1451065915 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x563fd18a46d0 0x563fd18a6ac0 crc :-1 s=READY pgs=164 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:44.805 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.805+0000 7f671e5ee740 1 --2- 172.21.15.90:0/1451065915 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563fd18833b0 0x563fd1880e30 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:44.805 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.805+0000 7f671e5ee740 1 --2- 172.21.15.90:0/1451065915 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563fd187a7d0 0x563fd18808f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:44.806 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.805+0000 7f66faa00640 0 client.27143 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:40:44.806 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.805+0000 7f671e5ee740 1 --2- 172.21.15.90:0/1451065915 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563fd1879eb0 0x563fd18803b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:44.806 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.805+0000 7f671e5ee740 1 -- 172.21.15.90:0/1451065915 >> 172.21.15.90:0/1451065915 conn(0x563fd18745e0 msgr2=0x563fd1876330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:40:44.806 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.805+0000 7f671e5ee740 1 -- 172.21.15.90:0/1451065915 shutdown_connections 2024-04-02T14:40:44.806 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.805+0000 7f671e5ee740 1 -- 172.21.15.90:0/1451065915 wait complete. 2024-04-02T14:40:44.806 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.806+0000 7f671e5ee740 20 client.27143 trim_cache size 0 max 16384 2024-04-02T14:40:44.840 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.839+0000 7f671e5ee740 1 Processor -- start 2024-04-02T14:40:44.840 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.839+0000 7f671e5ee740 1 -- start start 2024-04-02T14:40:44.840 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.840+0000 7f671e5ee740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563fd1879eb0 0x563fd18a00a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:44.841 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.840+0000 7f671e5ee740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563fd187a7d0 0x563fd1880c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:44.841 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.840+0000 7f671e5ee740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563fd18833b0 0x563fd18ac1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:44.841 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.840+0000 7f671e5ee740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f67140730d0 con 0x563fd18833b0 2024-04-02T14:40:44.841 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.840+0000 7f671e5ee740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f67141b5aa0 con 0x563fd1879eb0 2024-04-02T14:40:44.842 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.840+0000 7f671e5ee740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f67140737a0 con 0x563fd187a7d0 2024-04-02T14:40:44.842 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.840+0000 7f6713e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563fd187a7d0 0x563fd1880c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:44.842 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.840+0000 7f6713e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563fd187a7d0 0x563fd1880c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:49700/0 (socket says 172.21.15.90:49700) 2024-04-02T14:40:44.842 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.840+0000 7f6713e00640 1 -- 172.21.15.90:0/4071040298 learned_addr learned my addr 172.21.15.90:0/4071040298 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:40:44.842 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.840+0000 7f6719a00640 1 --2- 172.21.15.90:0/4071040298 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563fd1879eb0 0x563fd18a00a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:44.842 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.840+0000 7f6719a00640 1 --2- 172.21.15.90:0/4071040298 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563fd18833b0 0x563fd18ac1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:44.842 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.841+0000 7f6719a00640 1 -- 172.21.15.90:0/4071040298 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563fd187a7d0 msgr2=0x563fd1880c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:40:44.842 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.841+0000 7f6719a00640 1 --2- 172.21.15.90:0/4071040298 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563fd187a7d0 0x563fd1880c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:44.842 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.841+0000 7f6719a00640 1 -- 172.21.15.90:0/4071040298 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563fd18833b0 msgr2=0x563fd18ac1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:40:44.842 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.841+0000 7f6719a00640 1 --2- 172.21.15.90:0/4071040298 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563fd18833b0 0x563fd18ac1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:44.842 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.841+0000 7f6719a00640 1 -- 172.21.15.90:0/4071040298 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563fd15bf620 con 0x563fd1879eb0 2024-04-02T14:40:44.842 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.841+0000 7f6713e00640 1 --2- 172.21.15.90:0/4071040298 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563fd187a7d0 0x563fd1880c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:40:44.843 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.841+0000 7f6719a00640 1 --2- 172.21.15.90:0/4071040298 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563fd1879eb0 0x563fd18a00a0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f670c00bc80 tx=0x7f670c00e720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fca43f7713f85454 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:40:44.843 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.841+0000 7f6705600640 1 -- 172.21.15.90:0/4071040298 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f670c004b20 con 0x563fd1879eb0 2024-04-02T14:40:44.844 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.841+0000 7f6705600640 1 -- 172.21.15.90:0/4071040298 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f670c00ee70 con 0x563fd1879eb0 2024-04-02T14:40:44.844 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.841+0000 7f6705600640 1 -- 172.21.15.90:0/4071040298 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f670c028c90 con 0x563fd1879eb0 2024-04-02T14:40:44.844 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.842+0000 7f671e5ee740 1 -- 172.21.15.90:0/4071040298 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563fd1879eb0 msgr2=0x563fd18a00a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:44.844 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.842+0000 7f671e5ee740 1 --2- 172.21.15.90:0/4071040298 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563fd1879eb0 0x563fd18a00a0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f670c00bc80 tx=0x7f670c00e720 comp rx=0 tx=0).stop 2024-04-02T14:40:44.844 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.843+0000 7f671e5ee740 1 -- 172.21.15.90:0/4071040298 shutdown_connections 2024-04-02T14:40:44.844 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.843+0000 7f671e5ee740 1 --2- 172.21.15.90:0/4071040298 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563fd18833b0 0x563fd18ac1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:44.844 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.843+0000 7f671e5ee740 1 --2- 172.21.15.90:0/4071040298 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563fd187a7d0 0x563fd1880c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:44.844 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.843+0000 7f671e5ee740 1 --2- 172.21.15.90:0/4071040298 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563fd1879eb0 0x563fd18a00a0 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:44.844 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.843+0000 7f671e5ee740 1 -- 172.21.15.90:0/4071040298 >> 172.21.15.90:0/4071040298 conn(0x563fd18745e0 msgr2=0x563fd1873640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:40:44.844 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.843+0000 7f671e5ee740 1 -- 172.21.15.90:0/4071040298 shutdown_connections 2024-04-02T14:40:44.844 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.843+0000 7f671e5ee740 1 -- 172.21.15.90:0/4071040298 wait complete. 2024-04-02T14:40:44.845 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.843+0000 7f671e5ee740 1 -- start start 2024-04-02T14:40:44.845 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.844+0000 7f671e5ee740 1 Processor -- start 2024-04-02T14:40:44.845 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.844+0000 7f671e5ee740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563fd1879eb0 0x563fd189ca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:44.845 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.844+0000 7f671e5ee740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563fd187a7d0 0x563fd187ce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:44.846 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.844+0000 7f671e5ee740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563fd18833b0 0x563fd187d3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:44.846 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.844+0000 7f671e5ee740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f67141b53d0 con 0x563fd18833b0 2024-04-02T14:40:44.846 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.844+0000 7f671e5ee740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f67141b4d00 con 0x563fd1879eb0 2024-04-02T14:40:44.846 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.844+0000 7f671e5ee740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f6714071070 con 0x563fd187a7d0 2024-04-02T14:40:44.846 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.844+0000 7f671e5ee740 10 client.0 _ll_register_callbacks cb 0x7f670c003d50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:40:44.847 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.845+0000 7f6713e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563fd187a7d0 0x563fd187ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:44.847 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.845+0000 7f6713e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563fd187a7d0 0x563fd187ce40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:49708/0 (socket says 172.21.15.90:49708) 2024-04-02T14:40:44.847 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.845+0000 7f6713e00640 1 -- 172.21.15.90:0/3806281875 learned_addr learned my addr 172.21.15.90:0/3806281875 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:40:44.847 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.845+0000 7f6713e00640 1 -- 172.21.15.90:0/3806281875 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563fd1879eb0 msgr2=0x563fd189ca80 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-02T14:40:44.847 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.845+0000 7f6713e00640 1 --2- 172.21.15.90:0/3806281875 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563fd1879eb0 0x563fd189ca80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:44.847 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.845+0000 7f6713e00640 1 -- 172.21.15.90:0/3806281875 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563fd18833b0 msgr2=0x563fd187d3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:40:44.847 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.845+0000 7f6719a00640 1 --2- 172.21.15.90:0/3806281875 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563fd18833b0 0x563fd187d3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:44.847 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.845+0000 7f6713e00640 1 --2- 172.21.15.90:0/3806281875 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563fd18833b0 0x563fd187d3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:44.848 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.846+0000 7f6713e00640 1 -- 172.21.15.90:0/3806281875 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563fd187dbd0 con 0x563fd187a7d0 2024-04-02T14:40:44.848 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.846+0000 7f6719a00640 1 --2- 172.21.15.90:0/3806281875 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563fd18833b0 0x563fd187d3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:40:44.848 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.846+0000 7f6713e00640 1 --2- 172.21.15.90:0/3806281875 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563fd187a7d0 0x563fd187ce40 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f670002c180 tx=0x7f6700034d80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d1dcca4525572eb3 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:40:44.849 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.846+0000 7f66faa00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:40:44.849 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.847+0000 7f66faa00640 1 -- 172.21.15.90:0/3806281875 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f670004a020 con 0x563fd187a7d0 2024-04-02T14:40:44.849 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.847+0000 7f66faa00640 1 -- 172.21.15.90:0/3806281875 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f6700004100 con 0x563fd187a7d0 2024-04-02T14:40:44.849 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.848+0000 7f671e5ee740 1 -- 172.21.15.90:0/3806281875 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f670c00ee70 con 0x563fd187a7d0 2024-04-02T14:40:44.850 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.849+0000 7f66faa00640 1 -- 172.21.15.90:0/3806281875 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f670004a020 con 0x563fd187a7d0 2024-04-02T14:40:44.850 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.849+0000 7f671e5ee740 10 client.27019 fetch_fsmap learned FSMap version 27 2024-04-02T14:40:44.850 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.849+0000 7f671e5ee740 1 -- 172.21.15.90:0/3806281875 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x563fd190ab80 con 0x563fd187a7d0 2024-04-02T14:40:44.850 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.849+0000 7f66faa00640 1 -- 172.21.15.90:0/3806281875 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f670c00ee70 con 0x563fd187a7d0 2024-04-02T14:40:44.850 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.850+0000 7f66faa00640 1 -- 172.21.15.90:0/3806281875 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f67000461d0 con 0x563fd187a7d0 2024-04-02T14:40:44.851 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.850+0000 7f671e5ee740 10 client.27019 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:40:44.851 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.850+0000 7f671e5ee740 10 client.27019 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-02T14:40:44.851 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.850+0000 7f671e5ee740 20 client.27019 populate_metadata read hostname 'smithi090' 2024-04-02T14:40:44.851 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.850+0000 7f671e5ee740 1 --2- 172.21.15.90:0/3806281875 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x563fd190b330 0x563fd190d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:44.851 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.850+0000 7f671e5ee740 4 client.27019 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-02T14:40:44.851 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.850+0000 7f671e5ee740 1 -- 172.21.15.90:0/3806281875 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "flush journal"}) -- 0x563fd178ecd0 con 0x563fd190b330 2024-04-02T14:40:44.852 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.851+0000 7f6719a00640 1 --2- 172.21.15.90:0/3806281875 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x563fd190b330 0x563fd190d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:44.852 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.851+0000 7f6719a00640 1 --2- 172.21.15.90:0/3806281875 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x563fd190b330 0x563fd190d7b0 crc :-1 s=READY pgs=165 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4322848d91d004dd server_cookie=55ff2e1af05c839d in_seq=0 out_seq=0 2024-04-02T14:40:44.852 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.851+0000 7f66faa00640 10 client.27019 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:40:44.885 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.884+0000 7f66faa00640 1 -- 172.21.15.90:0/3806281875 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x563fd178ecd0 con 0x563fd190b330 2024-04-02T14:40:44.885 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.884+0000 7f66faa00640 10 client.27019 handle_command_reply: tid=0 2024-04-02T14:40:44.885 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.884+0000 7f671e5ee740 1 client.27019 shutdown 2024-04-02T14:40:44.885 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.884+0000 7f671e5ee740 1 -- 172.21.15.90:0/3806281875 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563fd187a7d0 msgr2=0x563fd187ce40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:44.885 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.884+0000 7f671e5ee740 1 --2- 172.21.15.90:0/3806281875 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563fd187a7d0 0x563fd187ce40 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f670002c180 tx=0x7f6700034d80 comp rx=0 tx=0).stop 2024-04-02T14:40:44.885 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.884+0000 7f671e5ee740 1 -- 172.21.15.90:0/3806281875 shutdown_connections 2024-04-02T14:40:44.885 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.884+0000 7f671e5ee740 1 --2- 172.21.15.90:0/3806281875 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x563fd190b330 0x563fd190d7b0 crc :-1 s=READY pgs=165 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:44.886 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.884+0000 7f671e5ee740 1 --2- 172.21.15.90:0/3806281875 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563fd18833b0 0x563fd187d3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:44.886 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.884+0000 7f671e5ee740 1 --2- 172.21.15.90:0/3806281875 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563fd187a7d0 0x563fd187ce40 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:44.886 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.884+0000 7f671e5ee740 1 --2- 172.21.15.90:0/3806281875 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563fd1879eb0 0x563fd189ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:44.886 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.884+0000 7f671e5ee740 1 -- 172.21.15.90:0/3806281875 >> 172.21.15.90:0/3806281875 conn(0x563fd18745e0 msgr2=0x563fd18758e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:40:44.886 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.884+0000 7f66faa00640 0 client.27019 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:40:44.887 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.885+0000 7f671e5ee740 1 -- 172.21.15.90:0/3806281875 shutdown_connections 2024-04-02T14:40:44.887 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.885+0000 7f671e5ee740 1 -- 172.21.15.90:0/3806281875 wait complete. 2024-04-02T14:40:44.887 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.885+0000 7f671e5ee740 20 client.27019 trim_cache size 0 max 16384 2024-04-02T14:40:44.887 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:{ 2024-04-02T14:40:44.887 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "message": "", 2024-04-02T14:40:44.887 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "return_code": 0 2024-04-02T14:40:44.887 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:} 2024-04-02T14:40:44.888 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.886+0000 7f671ba00640 1 -- 172.21.15.90:0/1443204311 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f66fc0a3f70 msgr2=0x7f66fc0a6430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:44.888 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.886+0000 7f671ba00640 1 --2- 172.21.15.90:0/1443204311 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f66fc0a3f70 0x7f66fc0a6430 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x563fd1879980 tx=0x7f670800ce60 comp rx=0 tx=0).stop 2024-04-02T14:40:44.888 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.886+0000 7f671ba00640 1 -- 172.21.15.90:0/1443204311 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6714101af0 msgr2=0x7f6714116210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:44.888 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.886+0000 7f671ba00640 1 --2- 172.21.15.90:0/1443204311 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6714101af0 0x7f6714116210 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f670800e9c0 tx=0x7f670800ee80 comp rx=0 tx=0).stop 2024-04-02T14:40:44.888 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.886+0000 7f671ba00640 1 -- 172.21.15.90:0/1443204311 shutdown_connections 2024-04-02T14:40:44.888 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.886+0000 7f671ba00640 1 --2- 172.21.15.90:0/1443204311 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f66fc0a3f70 0x7f66fc0a6430 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:44.888 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.886+0000 7f671ba00640 1 --2- 172.21.15.90:0/1443204311 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6714107b90 0x7f671411bf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:44.888 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.886+0000 7f671ba00640 1 --2- 172.21.15.90:0/1443204311 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f67141024b0 0x7f6714116750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:44.888 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.886+0000 7f671ba00640 1 --2- 172.21.15.90:0/1443204311 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6714101af0 0x7f6714116210 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:44.888 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.886+0000 7f671ba00640 1 -- 172.21.15.90:0/1443204311 >> 172.21.15.90:0/1443204311 conn(0x7f67140fd5c0 msgr2=0x7f67140fdba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:40:44.888 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.887+0000 7f671ba00640 1 -- 172.21.15.90:0/1443204311 shutdown_connections 2024-04-02T14:40:44.889 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:44.887+0000 7f671ba00640 1 -- 172.21.15.90:0/1443204311 wait complete. 2024-04-02T14:40:45.859 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:40:45 smithi092 ceph-mon[36497]: pgmap v179: 97 pgs: 97 active+clean; 3.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 9.9 MiB/s wr, 933 op/s 2024-04-02T14:40:45.935 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:40:45 smithi177 ceph-mon[36431]: pgmap v179: 97 pgs: 97 active+clean; 3.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 9.9 MiB/s wr, 933 op/s 2024-04-02T14:40:45.937 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:40:45 smithi090 ceph-mon[31674]: pgmap v179: 97 pgs: 97 active+clean; 3.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 9.9 MiB/s wr, 933 op/s 2024-04-02T14:40:46.830 INFO:teuthology.orchestra.run.smithi090.stderr:progress: 315.0 s, 3500.6 tps, lat 28.783 ms stddev 33.538 2024-04-02T14:40:47.210 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-02T14:40:47.210 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-02T14:40:47.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.501+0000 7f9967800640 1 -- 172.21.15.90:0/1369329051 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f99600747d0 msgr2=0x7f996010de10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:47.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.501+0000 7f9967800640 1 --2- 172.21.15.90:0/1369329051 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f99600747d0 0x7f996010de10 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f994c009900 tx=0x7f994c02bcc0 comp rx=0 tx=0).stop 2024-04-02T14:40:47.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.501+0000 7f9967800640 1 -- 172.21.15.90:0/1369329051 shutdown_connections 2024-04-02T14:40:47.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.501+0000 7f9967800640 1 --2- 172.21.15.90:0/1369329051 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f996010e350 0x7f9960110740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:47.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.501+0000 7f9967800640 1 --2- 172.21.15.90:0/1369329051 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f99600747d0 0x7f996010de10 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:47.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.501+0000 7f9967800640 1 --2- 172.21.15.90:0/1369329051 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9960073eb0 0x7f9960074290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:47.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.501+0000 7f9967800640 1 -- 172.21.15.90:0/1369329051 >> 172.21.15.90:0/1369329051 conn(0x7f9960078f00 msgr2=0x7f9960079310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:40:47.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.502+0000 7f9967800640 1 -- 172.21.15.90:0/1369329051 shutdown_connections 2024-04-02T14:40:47.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.502+0000 7f9967800640 1 -- 172.21.15.90:0/1369329051 wait complete. 2024-04-02T14:40:47.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.503+0000 7f9967800640 1 Processor -- start 2024-04-02T14:40:47.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.504+0000 7f9967800640 1 -- start start 2024-04-02T14:40:47.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.505+0000 7f9967800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9960073eb0 0x7f9960105ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:47.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.505+0000 7f9967800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f99600747d0 0x7f9960106430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:47.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.505+0000 7f9967800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f996010e350 0x7f9960106970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:47.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.505+0000 7f9964e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9960073eb0 0x7f9960105ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:47.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.505+0000 7f9964e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9960073eb0 0x7f9960105ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:49728/0 (socket says 172.21.15.90:49728) 2024-04-02T14:40:47.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.505+0000 7f9964e00640 1 -- 172.21.15.90:0/2562806078 learned_addr learned my addr 172.21.15.90:0/2562806078 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:40:47.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.506+0000 7f9965800640 1 --2- 172.21.15.90:0/2562806078 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f996010e350 0x7f9960106970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:47.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.506+0000 7f9967800640 1 -- 172.21.15.90:0/2562806078 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f9960104bd0 con 0x7f99600747d0 2024-04-02T14:40:47.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.506+0000 7f9967800640 1 -- 172.21.15.90:0/2562806078 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f9960104a50 con 0x7f996010e350 2024-04-02T14:40:47.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.506+0000 7f9967800640 1 -- 172.21.15.90:0/2562806078 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f9960104d50 con 0x7f9960073eb0 2024-04-02T14:40:47.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.507+0000 7f9965800640 1 -- 172.21.15.90:0/2562806078 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9960073eb0 msgr2=0x7f9960105ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:47.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.507+0000 7f9965800640 1 --2- 172.21.15.90:0/2562806078 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9960073eb0 0x7f9960105ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:47.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.507+0000 7f9965800640 1 -- 172.21.15.90:0/2562806078 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f99600747d0 msgr2=0x7f9960106430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:47.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.507+0000 7f9965800640 1 --2- 172.21.15.90:0/2562806078 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f99600747d0 0x7f9960106430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:47.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.507+0000 7f9965800640 1 -- 172.21.15.90:0/2562806078 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99601b3450 con 0x7f996010e350 2024-04-02T14:40:47.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.507+0000 7f9965800640 1 --2- 172.21.15.90:0/2562806078 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f996010e350 0x7f9960106970 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f995800cab0 tx=0x7f995800cf70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:40:47.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.509+0000 7f995d600640 1 -- 172.21.15.90:0/2562806078 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f995800ae20 con 0x7f996010e350 2024-04-02T14:40:47.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.509+0000 7f9967800640 1 -- 172.21.15.90:0/2562806078 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99601b3740 con 0x7f996010e350 2024-04-02T14:40:47.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.509+0000 7f9967800640 1 -- 172.21.15.90:0/2562806078 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99601b3c90 con 0x7f996010e350 2024-04-02T14:40:47.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.510+0000 7f995d600640 1 -- 172.21.15.90:0/2562806078 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f995800f930 con 0x7f996010e350 2024-04-02T14:40:47.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.510+0000 7f995d600640 1 -- 172.21.15.90:0/2562806078 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f995800ae20 con 0x7f996010e350 2024-04-02T14:40:47.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.517+0000 7f996a62d740 1 Processor -- start 2024-04-02T14:40:47.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.517+0000 7f995d600640 1 -- 172.21.15.90:0/2562806078 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f995800fae0 con 0x7f996010e350 2024-04-02T14:40:47.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.520+0000 7f996a62d740 1 -- start start 2024-04-02T14:40:47.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.520+0000 7f996a62d740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a14a9107b0 0x55a14a910b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:47.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.520+0000 7f996a62d740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a14a9110d0 0x55a14a919770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:47.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.520+0000 7f996a62d740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a14a919cb0 0x55a14a91c0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:47.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.520+0000 7f996a62d740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55a14a874770 con 0x55a14a9107b0 2024-04-02T14:40:47.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.520+0000 7f996a62d740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55a14a81c700 con 0x55a14a9110d0 2024-04-02T14:40:47.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.520+0000 7f996a62d740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55a14a870720 con 0x55a14a919cb0 2024-04-02T14:40:47.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.521+0000 7f995d600640 1 --2- 172.21.15.90:0/2562806078 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f99480a4050 0x7f99480a6510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:47.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.521+0000 7f9964e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a14a9110d0 0x55a14a919770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:47.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.522+0000 7f9964e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a14a9110d0 0x55a14a919770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:34050/0 (socket says 172.21.15.90:34050) 2024-04-02T14:40:47.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.522+0000 7f9964e00640 1 -- 172.21.15.90:0/2202250723 learned_addr learned my addr 172.21.15.90:0/2202250723 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:40:47.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.522+0000 7f995d600640 1 -- 172.21.15.90:0/2562806078 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f99580d0be0 con 0x7f996010e350 2024-04-02T14:40:47.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.522+0000 7f995fe00640 1 --2- 172.21.15.90:0/2202250723 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a14a9107b0 0x55a14a910b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:47.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.523+0000 7f995fe00640 1 --2- 172.21.15.90:0/2202250723 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a14a919cb0 0x55a14a91c0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:47.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.523+0000 7f9965800640 1 --2- 172.21.15.90:0/2562806078 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f99480a4050 0x7f99480a6510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:47.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.524+0000 7f9965800640 1 --2- 172.21.15.90:0/2562806078 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f99480a4050 0x7f99480a6510 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f995809cac0 tx=0x7f99580072a0 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:40:47.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.524+0000 7f9964e00640 1 -- 172.21.15.90:0/2202250723 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a14a919cb0 msgr2=0x55a14a91c0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:40:47.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.525+0000 7f9964e00640 1 --2- 172.21.15.90:0/2202250723 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a14a919cb0 0x55a14a91c0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:47.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.525+0000 7f9964e00640 1 -- 172.21.15.90:0/2202250723 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a14a9107b0 msgr2=0x55a14a910b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:40:47.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.525+0000 7f9964e00640 1 --2- 172.21.15.90:0/2202250723 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a14a9107b0 0x55a14a910b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:47.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.525+0000 7f9964e00640 1 -- 172.21.15.90:0/2202250723 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a14a6527d0 con 0x55a14a9110d0 2024-04-02T14:40:47.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.525+0000 7f995fe00640 1 --2- 172.21.15.90:0/2202250723 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a14a9107b0 0x55a14a910b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:40:47.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.526+0000 7f9964e00640 1 --2- 172.21.15.90:0/2202250723 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a14a9110d0 0x55a14a919770 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f9954009b20 tx=0x7f995400e1d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=92280352b946d075 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:40:47.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.526+0000 7f9952000640 1 -- 172.21.15.90:0/2202250723 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9954017070 con 0x55a14a9110d0 2024-04-02T14:40:47.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.527+0000 7f9952000640 1 -- 172.21.15.90:0/2202250723 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f995400ec10 con 0x55a14a9110d0 2024-04-02T14:40:47.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.527+0000 7f9952000640 1 -- 172.21.15.90:0/2202250723 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9954012630 con 0x55a14a9110d0 2024-04-02T14:40:47.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.528+0000 7f996a62d740 1 -- 172.21.15.90:0/2202250723 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a14a9110d0 msgr2=0x55a14a919770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:47.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.528+0000 7f996a62d740 1 --2- 172.21.15.90:0/2202250723 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a14a9110d0 0x55a14a919770 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f9954009b20 tx=0x7f995400e1d0 comp rx=0 tx=0).stop 2024-04-02T14:40:47.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.529+0000 7f996a62d740 1 -- 172.21.15.90:0/2202250723 shutdown_connections 2024-04-02T14:40:47.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.529+0000 7f996a62d740 1 --2- 172.21.15.90:0/2202250723 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a14a919cb0 0x55a14a91c0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:47.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.529+0000 7f996a62d740 1 --2- 172.21.15.90:0/2202250723 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a14a9110d0 0x55a14a919770 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:47.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.529+0000 7f996a62d740 1 --2- 172.21.15.90:0/2202250723 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a14a9107b0 0x55a14a910b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:47.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.529+0000 7f996a62d740 1 -- 172.21.15.90:0/2202250723 >> 172.21.15.90:0/2202250723 conn(0x55a14a90aee0 msgr2=0x55a14a90b2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:40:47.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.529+0000 7f996a62d740 1 -- 172.21.15.90:0/2202250723 shutdown_connections 2024-04-02T14:40:47.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.529+0000 7f996a62d740 1 -- 172.21.15.90:0/2202250723 wait complete. 2024-04-02T14:40:47.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.530+0000 7f996a62d740 1 -- start start 2024-04-02T14:40:47.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.530+0000 7f996a62d740 1 Processor -- start 2024-04-02T14:40:47.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.531+0000 7f996a62d740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a14a9107b0 0x55a14a916c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:47.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.531+0000 7f996a62d740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a14a9110d0 0x55a14a9171a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:47.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.531+0000 7f996a62d740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a14a919cb0 0x55a14a9176e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:47.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.531+0000 7f996a62d740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55a14a91caa0 con 0x55a14a9107b0 2024-04-02T14:40:47.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.531+0000 7f996a62d740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55a14a82fac0 con 0x55a14a9110d0 2024-04-02T14:40:47.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.531+0000 7f996a62d740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55a14a91cc20 con 0x55a14a919cb0 2024-04-02T14:40:47.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.531+0000 7f996a62d740 10 client.0 _ll_register_callbacks cb 0x55a14a832180 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:40:47.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.531+0000 7f9964e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a14a9110d0 0x55a14a9171a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:47.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.531+0000 7f995fe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a14a919cb0 0x55a14a9176e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:47.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.532+0000 7f9964e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a14a9110d0 0x55a14a9171a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:34056/0 (socket says 172.21.15.90:34056) 2024-04-02T14:40:47.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.532+0000 7f9964e00640 1 -- 172.21.15.90:0/3329902427 learned_addr learned my addr 172.21.15.90:0/3329902427 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:40:47.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.532+0000 7f995fe00640 1 --2- 172.21.15.90:0/3329902427 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a14a9107b0 0x55a14a916c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:47.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.532+0000 7f9964e00640 1 -- 172.21.15.90:0/3329902427 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a14a919cb0 msgr2=0x55a14a9176e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:40:47.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.532+0000 7f9964e00640 1 --2- 172.21.15.90:0/3329902427 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a14a919cb0 0x55a14a9176e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:47.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.532+0000 7f9964e00640 1 -- 172.21.15.90:0/3329902427 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a14a9107b0 msgr2=0x55a14a916c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:40:47.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.533+0000 7f9964e00640 1 --2- 172.21.15.90:0/3329902427 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a14a9107b0 0x55a14a916c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:47.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.533+0000 7f9964e00640 1 -- 172.21.15.90:0/3329902427 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a14a5883f0 con 0x55a14a9110d0 2024-04-02T14:40:47.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.533+0000 7f995fe00640 1 --2- 172.21.15.90:0/3329902427 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a14a9107b0 0x55a14a916c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:40:47.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.534+0000 7f9964e00640 1 --2- 172.21.15.90:0/3329902427 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a14a9110d0 0x55a14a9171a0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f9954000c00 tx=0x7f9954012600 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9a6a438c034a9467 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:40:47.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.534+0000 7f9946a00640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:40:47.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.536+0000 7f9946a00640 1 -- 172.21.15.90:0/3329902427 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9954022070 con 0x55a14a9110d0 2024-04-02T14:40:47.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.536+0000 7f9946a00640 1 -- 172.21.15.90:0/3329902427 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f9954012a30 con 0x55a14a9110d0 2024-04-02T14:40:47.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.536+0000 7f996a62d740 1 -- 172.21.15.90:0/3329902427 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f995400ec10 con 0x55a14a9110d0 2024-04-02T14:40:47.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.537+0000 7f9946a00640 1 -- 172.21.15.90:0/3329902427 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f9954017040 con 0x55a14a9110d0 2024-04-02T14:40:47.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.537+0000 7f996a62d740 10 client.27158 fetch_fsmap learned FSMap version 27 2024-04-02T14:40:47.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.537+0000 7f996a62d740 1 -- 172.21.15.90:0/3329902427 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55a14a5480f0 con 0x55a14a9110d0 2024-04-02T14:40:47.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.537+0000 7f9946a00640 1 -- 172.21.15.90:0/3329902427 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f995400ec10 con 0x55a14a9110d0 2024-04-02T14:40:47.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.537+0000 7f9946a00640 1 -- 172.21.15.90:0/3329902427 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f9954030960 con 0x55a14a9110d0 2024-04-02T14:40:47.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.537+0000 7f996a62d740 10 client.27158 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:40:47.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.537+0000 7f996a62d740 10 client.27158 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:40:47.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.537+0000 7f996a62d740 20 client.27158 populate_metadata read hostname 'smithi090' 2024-04-02T14:40:47.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.537+0000 7f996a62d740 1 --2- 172.21.15.90:0/3329902427 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55a14a93ab20 0x55a14a93cf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:47.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.537+0000 7f996a62d740 4 client.27158 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:40:47.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.538+0000 7f996a62d740 1 -- 172.21.15.90:0/3329902427 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a14a815630 con 0x55a14a93ab20 2024-04-02T14:40:47.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.540+0000 7f995fe00640 1 --2- 172.21.15.90:0/3329902427 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55a14a93ab20 0x55a14a93cf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:47.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.540+0000 7f9946a00640 10 client.27158 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:40:47.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.540+0000 7f995fe00640 1 --2- 172.21.15.90:0/3329902427 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55a14a93ab20 0x55a14a93cf10 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2651ef6601311017 server_cookie=efff5c188bf4f0e5 in_seq=0 out_seq=0 2024-04-02T14:40:47.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.541+0000 7f9946a00640 1 -- 172.21.15.90:0/3329902427 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a14a815630 con 0x55a14a93ab20 2024-04-02T14:40:47.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.541+0000 7f9946a00640 10 client.27158 handle_command_reply: tid=0 2024-04-02T14:40:47.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.541+0000 7f996a62d740 1 client.27158 shutdown 2024-04-02T14:40:47.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.542+0000 7f996a62d740 1 -- 172.21.15.90:0/3329902427 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a14a9110d0 msgr2=0x55a14a9171a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:47.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.542+0000 7f996a62d740 1 --2- 172.21.15.90:0/3329902427 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a14a9110d0 0x55a14a9171a0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f9954000c00 tx=0x7f9954012600 comp rx=0 tx=0).stop 2024-04-02T14:40:47.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.542+0000 7f996a62d740 1 -- 172.21.15.90:0/3329902427 shutdown_connections 2024-04-02T14:40:47.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.542+0000 7f996a62d740 1 --2- 172.21.15.90:0/3329902427 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55a14a93ab20 0x55a14a93cf10 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:47.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.542+0000 7f996a62d740 1 --2- 172.21.15.90:0/3329902427 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a14a919cb0 0x55a14a9176e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:47.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.542+0000 7f996a62d740 1 --2- 172.21.15.90:0/3329902427 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a14a9110d0 0x55a14a9171a0 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:47.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.542+0000 7f996a62d740 1 --2- 172.21.15.90:0/3329902427 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a14a9107b0 0x55a14a916c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:47.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.542+0000 7f996a62d740 1 -- 172.21.15.90:0/3329902427 >> 172.21.15.90:0/3329902427 conn(0x55a14a90aee0 msgr2=0x55a14a90cc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:40:47.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.542+0000 7f9946a00640 0 client.27158 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:40:47.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.542+0000 7f996a62d740 1 -- 172.21.15.90:0/3329902427 shutdown_connections 2024-04-02T14:40:47.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.542+0000 7f996a62d740 1 -- 172.21.15.90:0/3329902427 wait complete. 2024-04-02T14:40:47.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.542+0000 7f996a62d740 20 client.27158 trim_cache size 0 max 16384 2024-04-02T14:40:47.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.570+0000 7f996a62d740 1 Processor -- start 2024-04-02T14:40:47.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.571+0000 7f996a62d740 1 -- start start 2024-04-02T14:40:47.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.571+0000 7f996a62d740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a14a9107b0 0x55a14a9259c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:47.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.571+0000 7f996a62d740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a14a9110d0 0x55a14a915ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:47.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.571+0000 7f996a62d740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a14a919cb0 0x55a14a917500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:47.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.571+0000 7f996a62d740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f994000e750 con 0x55a14a9107b0 2024-04-02T14:40:47.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.571+0000 7f996a62d740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f9940010290 con 0x55a14a919cb0 2024-04-02T14:40:47.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.571+0000 7f996a62d740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f994000ee20 con 0x55a14a9110d0 2024-04-02T14:40:47.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.572+0000 7f9964e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a14a9110d0 0x55a14a915ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:47.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.572+0000 7f9964e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a14a9110d0 0x55a14a915ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:49768/0 (socket says 172.21.15.90:49768) 2024-04-02T14:40:47.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.572+0000 7f9964e00640 1 -- 172.21.15.90:0/2596172066 learned_addr learned my addr 172.21.15.90:0/2596172066 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:40:47.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.572+0000 7f995fe00640 1 --2- 172.21.15.90:0/2596172066 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a14a919cb0 0x55a14a917500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:47.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.572+0000 7f995fe00640 1 --2- 172.21.15.90:0/2596172066 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a14a9107b0 0x55a14a9259c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:47.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.572+0000 7f9964e00640 1 -- 172.21.15.90:0/2596172066 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a14a919cb0 msgr2=0x55a14a917500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:40:47.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.572+0000 7f9964e00640 1 --2- 172.21.15.90:0/2596172066 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a14a919cb0 0x55a14a917500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:47.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.573+0000 7f9964e00640 1 -- 172.21.15.90:0/2596172066 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a14a9107b0 msgr2=0x55a14a9259c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:40:47.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.573+0000 7f9964e00640 1 --2- 172.21.15.90:0/2596172066 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a14a9107b0 0x55a14a9259c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:47.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.573+0000 7f9964e00640 1 -- 172.21.15.90:0/2596172066 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a14a9102d0 con 0x55a14a9110d0 2024-04-02T14:40:47.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.573+0000 7f995fe00640 1 --2- 172.21.15.90:0/2596172066 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a14a919cb0 0x55a14a917500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:40:47.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.573+0000 7f9964e00640 1 --2- 172.21.15.90:0/2596172066 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a14a9110d0 0x55a14a915ba0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f9954009030 tx=0x7f995400e6d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ff0047695aa378b3 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:40:47.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.573+0000 7f9951600640 1 -- 172.21.15.90:0/2596172066 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9954022040 con 0x55a14a9110d0 2024-04-02T14:40:47.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.573+0000 7f9951600640 1 -- 172.21.15.90:0/2596172066 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9954012670 con 0x55a14a9110d0 2024-04-02T14:40:47.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.573+0000 7f9951600640 1 -- 172.21.15.90:0/2596172066 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9954030870 con 0x55a14a9110d0 2024-04-02T14:40:47.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.574+0000 7f996a62d740 1 -- 172.21.15.90:0/2596172066 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a14a9110d0 msgr2=0x55a14a915ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:47.576 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.574+0000 7f996a62d740 1 --2- 172.21.15.90:0/2596172066 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a14a9110d0 0x55a14a915ba0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f9954009030 tx=0x7f995400e6d0 comp rx=0 tx=0).stop 2024-04-02T14:40:47.576 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.574+0000 7f996a62d740 1 -- 172.21.15.90:0/2596172066 shutdown_connections 2024-04-02T14:40:47.576 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.574+0000 7f996a62d740 1 --2- 172.21.15.90:0/2596172066 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a14a919cb0 0x55a14a917500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:47.576 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.574+0000 7f996a62d740 1 --2- 172.21.15.90:0/2596172066 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a14a9110d0 0x55a14a915ba0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:47.576 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.574+0000 7f996a62d740 1 --2- 172.21.15.90:0/2596172066 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a14a9107b0 0x55a14a9259c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:47.576 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.574+0000 7f996a62d740 1 -- 172.21.15.90:0/2596172066 >> 172.21.15.90:0/2596172066 conn(0x55a14a90aee0 msgr2=0x55a14a909f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:40:47.576 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.575+0000 7f996a62d740 1 -- 172.21.15.90:0/2596172066 shutdown_connections 2024-04-02T14:40:47.576 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.575+0000 7f996a62d740 1 -- 172.21.15.90:0/2596172066 wait complete. 2024-04-02T14:40:47.576 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.575+0000 7f996a62d740 1 -- start start 2024-04-02T14:40:47.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.575+0000 7f996a62d740 1 Processor -- start 2024-04-02T14:40:47.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.576+0000 7f996a62d740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a14a9107b0 0x55a14a913770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:47.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.576+0000 7f996a62d740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a14a9110d0 0x55a14a913cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:47.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.576+0000 7f996a62d740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a14a919cb0 0x55a14a9141f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:47.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.576+0000 7f996a62d740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f994000fbc0 con 0x55a14a9107b0 2024-04-02T14:40:47.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.576+0000 7f996a62d740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f994000f4f0 con 0x55a14a919cb0 2024-04-02T14:40:47.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.576+0000 7f996a62d740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f994000c730 con 0x55a14a9110d0 2024-04-02T14:40:47.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.576+0000 7f996a62d740 10 client.0 _ll_register_callbacks cb 0x7f9954013f50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:40:47.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.576+0000 7f995fe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a14a9107b0 0x55a14a913770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:47.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.576+0000 7f9964e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a14a9110d0 0x55a14a913cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:47.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.576+0000 7f9964e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a14a9110d0 0x55a14a913cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:49780/0 (socket says 172.21.15.90:49780) 2024-04-02T14:40:47.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.576+0000 7f9964e00640 1 -- 172.21.15.90:0/197603986 learned_addr learned my addr 172.21.15.90:0/197603986 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:40:47.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.577+0000 7f995fe00640 1 --2- 172.21.15.90:0/197603986 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a14a919cb0 0x55a14a9141f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:47.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.577+0000 7f9964e00640 1 -- 172.21.15.90:0/197603986 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a14a919cb0 msgr2=0x55a14a9141f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:40:47.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.577+0000 7f9964e00640 1 --2- 172.21.15.90:0/197603986 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a14a919cb0 0x55a14a9141f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:47.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.577+0000 7f9964e00640 1 -- 172.21.15.90:0/197603986 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a14a9107b0 msgr2=0x55a14a913770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:40:47.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.577+0000 7f9964e00640 1 --2- 172.21.15.90:0/197603986 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a14a9107b0 0x55a14a913770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:47.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.577+0000 7f9964e00640 1 -- 172.21.15.90:0/197603986 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a14a91c2d0 con 0x55a14a9110d0 2024-04-02T14:40:47.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.577+0000 7f995fe00640 1 --2- 172.21.15.90:0/197603986 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a14a919cb0 0x55a14a9141f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:40:47.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.577+0000 7f9964e00640 1 --2- 172.21.15.90:0/197603986 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a14a9110d0 0x55a14a913cb0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f99540047d0 tx=0x7f9954013c40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bd7c0c31401baa5d server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:40:47.582 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.577+0000 7f9946a00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:40:47.582 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.577+0000 7f9946a00640 1 -- 172.21.15.90:0/197603986 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9954022040 con 0x55a14a9110d0 2024-04-02T14:40:47.582 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.577+0000 7f9946a00640 1 -- 172.21.15.90:0/197603986 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f9954030480 con 0x55a14a9110d0 2024-04-02T14:40:47.582 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.578+0000 7f996a62d740 1 -- 172.21.15.90:0/197603986 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9954012670 con 0x55a14a9110d0 2024-04-02T14:40:47.582 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.578+0000 7f9946a00640 1 -- 172.21.15.90:0/197603986 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f9954030680 con 0x55a14a9110d0 2024-04-02T14:40:47.582 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.578+0000 7f996a62d740 10 client.27034 fetch_fsmap learned FSMap version 27 2024-04-02T14:40:47.582 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.578+0000 7f996a62d740 1 -- 172.21.15.90:0/197603986 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55a14a9a0d60 con 0x55a14a9110d0 2024-04-02T14:40:47.582 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.578+0000 7f9946a00640 1 -- 172.21.15.90:0/197603986 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9954012670 con 0x55a14a9110d0 2024-04-02T14:40:47.582 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.578+0000 7f9946a00640 1 -- 172.21.15.90:0/197603986 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f9954039410 con 0x55a14a9110d0 2024-04-02T14:40:47.582 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.578+0000 7f996a62d740 10 client.27034 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:40:47.582 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.578+0000 7f996a62d740 10 client.27034 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:40:47.583 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.578+0000 7f996a62d740 20 client.27034 populate_metadata read hostname 'smithi090' 2024-04-02T14:40:47.583 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.578+0000 7f996a62d740 1 --2- 172.21.15.90:0/197603986 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55a14a9a12d0 0x55a14a9a3710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:47.583 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.578+0000 7f996a62d740 4 client.27034 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-02T14:40:47.583 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.578+0000 7f996a62d740 1 -- 172.21.15.90:0/197603986 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55a14a881cb0 con 0x55a14a9a12d0 2024-04-02T14:40:47.583 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.579+0000 7f995fe00640 1 --2- 172.21.15.90:0/197603986 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55a14a9a12d0 0x55a14a9a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:47.583 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.579+0000 7f995fe00640 1 --2- 172.21.15.90:0/197603986 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55a14a9a12d0 0x55a14a9a3710 crc :-1 s=READY pgs=167 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=81dcc59b02f1c08e server_cookie=102325b12672ca5e in_seq=0 out_seq=0 2024-04-02T14:40:47.583 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.579+0000 7f9946a00640 10 client.27034 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:40:47.583 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.580+0000 7f9946a00640 1 -- 172.21.15.90:0/197603986 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55a14a881cb0 con 0x55a14a9a12d0 2024-04-02T14:40:47.583 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.580+0000 7f9946a00640 10 client.27034 handle_command_reply: tid=0 2024-04-02T14:40:47.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.582+0000 7f996a62d740 1 client.27034 shutdown 2024-04-02T14:40:47.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.582+0000 7f996a62d740 1 -- 172.21.15.90:0/197603986 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a14a9110d0 msgr2=0x55a14a913cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:47.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.582+0000 7f996a62d740 1 --2- 172.21.15.90:0/197603986 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a14a9110d0 0x55a14a913cb0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f99540047d0 tx=0x7f9954013c40 comp rx=0 tx=0).stop 2024-04-02T14:40:47.584 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:40:47 smithi090 ceph-mon[31674]: pgmap v180: 97 pgs: 97 active+clean; 3.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 6.3 MiB/s wr, 617 op/s 2024-04-02T14:40:47.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.582+0000 7f996a62d740 1 -- 172.21.15.90:0/197603986 shutdown_connections 2024-04-02T14:40:47.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.582+0000 7f996a62d740 1 --2- 172.21.15.90:0/197603986 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55a14a9a12d0 0x55a14a9a3710 crc :-1 s=READY pgs=167 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:47.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.583+0000 7f996a62d740 1 --2- 172.21.15.90:0/197603986 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a14a919cb0 0x55a14a9141f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:47.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.583+0000 7f996a62d740 1 --2- 172.21.15.90:0/197603986 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a14a9110d0 0x55a14a913cb0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:47.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.583+0000 7f996a62d740 1 --2- 172.21.15.90:0/197603986 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a14a9107b0 0x55a14a913770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:47.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.583+0000 7f996a62d740 1 -- 172.21.15.90:0/197603986 >> 172.21.15.90:0/197603986 conn(0x55a14a90aee0 msgr2=0x55a14a90bee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:40:47.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.583+0000 7f9946a00640 0 client.27034 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:40:47.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.583+0000 7f996a62d740 1 -- 172.21.15.90:0/197603986 shutdown_connections 2024-04-02T14:40:47.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.584+0000 7f996a62d740 1 -- 172.21.15.90:0/197603986 wait complete. 2024-04-02T14:40:47.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.584+0000 7f996a62d740 20 client.27034 trim_cache size 0 max 16384 2024-04-02T14:40:47.587 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-02T14:40:47.587 INFO:teuthology.orchestra.run.smithi090.stdout: "status": "no active scrubs running", 2024-04-02T14:40:47.587 INFO:teuthology.orchestra.run.smithi090.stdout: "scrubs": {} 2024-04-02T14:40:47.587 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-02T14:40:47.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.586+0000 7f9967800640 1 -- 172.21.15.90:0/2562806078 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f99480a4050 msgr2=0x7f99480a6510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:47.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.586+0000 7f9967800640 1 --2- 172.21.15.90:0/2562806078 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f99480a4050 0x7f99480a6510 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f995809cac0 tx=0x7f99580072a0 comp rx=0 tx=0).stop 2024-04-02T14:40:47.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.586+0000 7f9967800640 1 -- 172.21.15.90:0/2562806078 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f996010e350 msgr2=0x7f9960106970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:47.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.586+0000 7f9967800640 1 --2- 172.21.15.90:0/2562806078 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f996010e350 0x7f9960106970 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f995800cab0 tx=0x7f995800cf70 comp rx=0 tx=0).stop 2024-04-02T14:40:47.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.586+0000 7f9967800640 1 -- 172.21.15.90:0/2562806078 shutdown_connections 2024-04-02T14:40:47.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.586+0000 7f9967800640 1 --2- 172.21.15.90:0/2562806078 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f99480a4050 0x7f99480a6510 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:47.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.586+0000 7f9967800640 1 --2- 172.21.15.90:0/2562806078 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f996010e350 0x7f9960106970 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:47.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.586+0000 7f9967800640 1 --2- 172.21.15.90:0/2562806078 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f99600747d0 0x7f9960106430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:47.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.586+0000 7f9967800640 1 --2- 172.21.15.90:0/2562806078 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9960073eb0 0x7f9960105ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:47.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.586+0000 7f9967800640 1 -- 172.21.15.90:0/2562806078 >> 172.21.15.90:0/2562806078 conn(0x7f9960078f00 msgr2=0x7f99600fe920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:40:47.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.586+0000 7f9967800640 1 -- 172.21.15.90:0/2562806078 shutdown_connections 2024-04-02T14:40:47.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.587+0000 7f9967800640 1 -- 172.21.15.90:0/2562806078 wait complete. 2024-04-02T14:40:47.612 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-02T14:40:47.612 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T14:40:47.859 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:40:47 smithi092 ceph-mon[36497]: pgmap v180: 97 pgs: 97 active+clean; 3.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 6.3 MiB/s wr, 617 op/s 2024-04-02T14:40:47.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.882+0000 7f0c14400640 1 -- 172.21.15.90:0/1214484657 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0c0c106e70 msgr2=0x7f0c0c06a610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:47.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.882+0000 7f0c14400640 1 --2- 172.21.15.90:0/1214484657 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0c0c106e70 0x7f0c0c06a610 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f0c00009860 tx=0x7f0c0002bb70 comp rx=0 tx=0).stop 2024-04-02T14:40:47.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.882+0000 7f0c14400640 1 -- 172.21.15.90:0/1214484657 shutdown_connections 2024-04-02T14:40:47.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.883+0000 7f0c14400640 1 --2- 172.21.15.90:0/1214484657 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0c0c06acf0 0x7f0c0c104500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:47.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.883+0000 7f0c14400640 1 --2- 172.21.15.90:0/1214484657 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0c0c106e70 0x7f0c0c06a610 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:47.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.883+0000 7f0c14400640 1 --2- 172.21.15.90:0/1214484657 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0c0c106a00 0x7f0c0c06a0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:47.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.883+0000 7f0c14400640 1 -- 172.21.15.90:0/1214484657 >> 172.21.15.90:0/1214484657 conn(0x7f0c0c0779e0 msgr2=0x7f0c0c077df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:40:47.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.883+0000 7f0c14400640 1 -- 172.21.15.90:0/1214484657 shutdown_connections 2024-04-02T14:40:47.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.883+0000 7f0c14400640 1 -- 172.21.15.90:0/1214484657 wait complete. 2024-04-02T14:40:47.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.884+0000 7f0c14400640 1 Processor -- start 2024-04-02T14:40:47.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.885+0000 7f0c14400640 1 -- start start 2024-04-02T14:40:47.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.885+0000 7f0c14400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0c0c106a00 0x7f0c0c1a4640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:47.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.885+0000 7f0c14400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0c0c106e70 0x7f0c0c1a4b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:47.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.885+0000 7f0c14400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0c0c19e7b0 0x7f0c0c19ec60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:47.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.885+0000 7f0c14400640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f0c0c118490 con 0x7f0c0c106a00 2024-04-02T14:40:47.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.885+0000 7f0c14400640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f0c0c118310 con 0x7f0c0c106e70 2024-04-02T14:40:47.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.885+0000 7f0c14400640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f0c0c118610 con 0x7f0c0c19e7b0 2024-04-02T14:40:47.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.886+0000 7f0c12400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0c0c19e7b0 0x7f0c0c19ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:47.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.886+0000 7f0c12400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0c0c19e7b0 0x7f0c0c19ec60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:49800/0 (socket says 172.21.15.90:49800) 2024-04-02T14:40:47.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.886+0000 7f0c12400640 1 -- 172.21.15.90:0/42471831 learned_addr learned my addr 172.21.15.90:0/42471831 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:40:47.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.886+0000 7f0c11a00640 1 --2- 172.21.15.90:0/42471831 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0c0c106a00 0x7f0c0c1a4640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:47.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.886+0000 7f0c11000640 1 --2- 172.21.15.90:0/42471831 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0c0c106e70 0x7f0c0c1a4b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:47.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.886+0000 7f0c12400640 1 -- 172.21.15.90:0/42471831 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0c0c106e70 msgr2=0x7f0c0c1a4b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:47.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.886+0000 7f0c12400640 1 --2- 172.21.15.90:0/42471831 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0c0c106e70 0x7f0c0c1a4b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:47.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.886+0000 7f0c12400640 1 -- 172.21.15.90:0/42471831 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0c0c106a00 msgr2=0x7f0c0c1a4640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:47.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.886+0000 7f0c12400640 1 --2- 172.21.15.90:0/42471831 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0c0c106a00 0x7f0c0c1a4640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:47.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.886+0000 7f0c12400640 1 -- 172.21.15.90:0/42471831 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c0c19f340 con 0x7f0c0c19e7b0 2024-04-02T14:40:47.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.887+0000 7f0c12400640 1 --2- 172.21.15.90:0/42471831 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0c0c19e7b0 0x7f0c0c19ec60 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f0c04009fa0 tx=0x7f0c0400c800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:40:47.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.887+0000 7f0c0a000640 1 -- 172.21.15.90:0/42471831 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c04004b20 con 0x7f0c0c19e7b0 2024-04-02T14:40:47.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.887+0000 7f0c0a000640 1 -- 172.21.15.90:0/42471831 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0c04003e50 con 0x7f0c0c19e7b0 2024-04-02T14:40:47.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.887+0000 7f0c0a000640 1 -- 172.21.15.90:0/42471831 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c0400b570 con 0x7f0c0c19e7b0 2024-04-02T14:40:47.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.887+0000 7f0c14400640 1 -- 172.21.15.90:0/42471831 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0c0c19f630 con 0x7f0c0c19e7b0 2024-04-02T14:40:47.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.887+0000 7f0c14400640 1 -- 172.21.15.90:0/42471831 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0c0c1ab6d0 con 0x7f0c0c19e7b0 2024-04-02T14:40:47.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.889+0000 7f0c14400640 1 -- 172.21.15.90:0/42471831 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0c0c100fe0 con 0x7f0c0c19e7b0 2024-04-02T14:40:47.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.891+0000 7f0c0a000640 1 -- 172.21.15.90:0/42471831 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f0c0401d020 con 0x7f0c0c19e7b0 2024-04-02T14:40:47.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.893+0000 7f0c0a000640 1 --2- 172.21.15.90:0/42471831 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f0bf40a3da0 0x7f0bf40a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:47.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.893+0000 7f0c0a000640 1 -- 172.21.15.90:0/42471831 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f0c040d0840 con 0x7f0c0c19e7b0 2024-04-02T14:40:47.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.894+0000 7f0c11a00640 1 --2- 172.21.15.90:0/42471831 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f0bf40a3da0 0x7f0bf40a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:47.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.895+0000 7f0c11a00640 1 --2- 172.21.15.90:0/42471831 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f0bf40a3da0 0x7f0bf40a6260 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f0bfc009ae0 tx=0x7f0bfc005f90 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:40:47.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:47.897+0000 7f0c0a000640 1 -- 172.21.15.90:0/42471831 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7f0c0c19f630 con 0x7f0c0c19e7b0 2024-04-02T14:40:47.935 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:40:47 smithi177 ceph-mon[36431]: pgmap v180: 97 pgs: 97 active+clean; 3.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 6.3 MiB/s wr, 617 op/s 2024-04-02T14:40:48.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.327+0000 7f0c14400640 1 -- 172.21.15.90:0/42471831 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0c0c19fce0 con 0x7f0c0c19e7b0 2024-04-02T14:40:48.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.328+0000 7f0c0a000640 1 -- 172.21.15.90:0/42471831 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11927 (secure 0 0 0) 0x7f0c0401d310 con 0x7f0c0c19e7b0 2024-04-02T14:40:48.329 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-02T14:40:48.329 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14412,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6837/2852160309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6836","nonce":2852160309},{"type":"v1","addr":"172.21.15.90:6837","nonce":2852160309}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14478,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6841/1919376858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6840","nonce":1919376858},{"type":"v1","addr":"172.21.15.90:6841","nonce":1919376858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-02T14:32:22.319083+0000","modified":"2024-04-02T14:34:52.266152+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24590,"mds_1":24560,"mds_2":24547,"mds_3":24539,"mds_4":24523},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24454":{"gid":24454,"name":"l","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6833/1564319431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":1564319431},{"type":"v1","addr":"172.21.15.177:6833","nonce":1564319431}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.92:6833/1570337372","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":1570337372},{"type":"v1","addr":"172.21.15.92:6833","nonce":1570337372}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"j","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.90:6835/1277124891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6834","nonce":1277124891},{"type":"v1","addr":"172.21.15.90:6835","nonce":1277124891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.92:6835/860721164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":860721164},{"type":"v1","addr":"172.21.15.92:6835","nonce":860721164}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24511":{"gid":24511,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6835/3177244988","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":3177244988},{"type":"v1","addr":"172.21.15.177:6835","nonce":3177244988}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"c","rank":4,"incarnation":25,"state":"up:active","state_seq":9,"addr":"172.21.15.177:6837/4223414858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":4223414858},{"type":"v1","addr":"172.21.15.177:6837","nonce":4223414858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24539":{"gid":24539,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":8,"addr":"172.21.15.92:6837/1270635939","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":1270635939},{"type":"v1","addr":"172.21.15.92:6837","nonce":1270635939}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24547":{"gid":24547,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":7,"addr":"172.21.15.177:6839/664895663","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":664895663},{"type":"v1","addr":"172.21.15.177:6839","nonce":664895663}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"g","rank":1,"incarnation":19,"state":"up:active","state_seq":5,"addr":"172.21.15.90:6839/1498915870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6838","nonce":1498915870},{"type":"v1","addr":"172.21.15.90:6839","nonce":1498915870}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24590":{"gid":24590,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.92:6839/1054888219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":1054888219},{"type":"v1","addr":"172.21.15.92:6839","nonce":1054888219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24590,"qdb_cluster":[24547,24590,24523,24539,24560]},"id":1}]} 2024-04-02T14:40:48.329 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 27 2024-04-02T14:40:48.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.332+0000 7f0c14400640 1 -- 172.21.15.90:0/42471831 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f0bf40a3da0 msgr2=0x7f0bf40a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:48.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.332+0000 7f0c14400640 1 --2- 172.21.15.90:0/42471831 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f0bf40a3da0 0x7f0bf40a6260 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f0bfc009ae0 tx=0x7f0bfc005f90 comp rx=0 tx=0).stop 2024-04-02T14:40:48.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.332+0000 7f0c14400640 1 -- 172.21.15.90:0/42471831 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0c0c19e7b0 msgr2=0x7f0c0c19ec60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:48.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.332+0000 7f0c14400640 1 --2- 172.21.15.90:0/42471831 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0c0c19e7b0 0x7f0c0c19ec60 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f0c04009fa0 tx=0x7f0c0400c800 comp rx=0 tx=0).stop 2024-04-02T14:40:48.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.332+0000 7f0c14400640 1 -- 172.21.15.90:0/42471831 shutdown_connections 2024-04-02T14:40:48.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.332+0000 7f0c14400640 1 --2- 172.21.15.90:0/42471831 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f0bf40a3da0 0x7f0bf40a6260 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:48.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.332+0000 7f0c14400640 1 --2- 172.21.15.90:0/42471831 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0c0c19e7b0 0x7f0c0c19ec60 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:48.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.332+0000 7f0c14400640 1 --2- 172.21.15.90:0/42471831 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0c0c106e70 0x7f0c0c1a4b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:48.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.332+0000 7f0c14400640 1 --2- 172.21.15.90:0/42471831 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0c0c106a00 0x7f0c0c1a4640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:48.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.332+0000 7f0c14400640 1 -- 172.21.15.90:0/42471831 >> 172.21.15.90:0/42471831 conn(0x7f0c0c0779e0 msgr2=0x7f0c0c0fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:40:48.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.332+0000 7f0c14400640 1 -- 172.21.15.90:0/42471831 shutdown_connections 2024-04-02T14:40:48.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.333+0000 7f0c14400640 1 -- 172.21.15.90:0/42471831 wait complete. 2024-04-02T14:40:48.365 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-04-02T14:40:48.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.646+0000 7fce69000640 1 -- 172.21.15.90:0/3548534568 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fce64108b00 msgr2=0x7fce64108ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:48.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.646+0000 7fce69000640 1 --2- 172.21.15.90:0/3548534568 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fce64108b00 0x7fce64108ee0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fce54009900 tx=0x7fce5402bcd0 comp rx=0 tx=0).stop 2024-04-02T14:40:48.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.646+0000 7fce69000640 1 -- 172.21.15.90:0/3548534568 shutdown_connections 2024-04-02T14:40:48.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.646+0000 7fce69000640 1 --2- 172.21.15.90:0/3548534568 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce64114c50 0x7fce64117040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:48.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.646+0000 7fce69000640 1 --2- 172.21.15.90:0/3548534568 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fce64109420 0x7fce64114710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:48.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.646+0000 7fce69000640 1 --2- 172.21.15.90:0/3548534568 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fce64108b00 0x7fce64108ee0 secure :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fce54009900 tx=0x7fce5402bcd0 comp rx=0 tx=0).stop 2024-04-02T14:40:48.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.646+0000 7fce69000640 1 -- 172.21.15.90:0/3548534568 >> 172.21.15.90:0/3548534568 conn(0x7fce6406d040 msgr2=0x7fce6406d450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:40:48.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.647+0000 7fce69000640 1 -- 172.21.15.90:0/3548534568 shutdown_connections 2024-04-02T14:40:48.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.647+0000 7fce69000640 1 -- 172.21.15.90:0/3548534568 wait complete. 2024-04-02T14:40:48.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.649+0000 7fce69000640 1 Processor -- start 2024-04-02T14:40:48.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.649+0000 7fce69000640 1 -- start start 2024-04-02T14:40:48.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.650+0000 7fce69000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fce64109420 0x7fce64111870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:48.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.650+0000 7fce69000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fce64114c50 0x7fce64111db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:48.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.651+0000 7fce69000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce641122f0 0x7fce641a8dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:48.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.651+0000 7fce69000640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fce6411c8e0 con 0x7fce641122f0 2024-04-02T14:40:48.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.651+0000 7fce69000640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fce6411c760 con 0x7fce64109420 2024-04-02T14:40:48.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.651+0000 7fce69000640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fce6411ca60 con 0x7fce64114c50 2024-04-02T14:40:48.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.651+0000 7fce61400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fce64114c50 0x7fce64111db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:48.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.651+0000 7fce61400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fce64114c50 0x7fce64111db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:49824/0 (socket says 172.21.15.90:49824) 2024-04-02T14:40:48.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.651+0000 7fce61400640 1 -- 172.21.15.90:0/3283524981 learned_addr learned my addr 172.21.15.90:0/3283524981 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:40:48.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.651+0000 7fce61400640 1 -- 172.21.15.90:0/3283524981 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fce64109420 msgr2=0x7fce64111870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:48.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.651+0000 7fce61400640 1 --2- 172.21.15.90:0/3283524981 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fce64109420 0x7fce64111870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:48.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.651+0000 7fce61400640 1 -- 172.21.15.90:0/3283524981 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce641122f0 msgr2=0x7fce641a8dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:48.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.651+0000 7fce61400640 1 --2- 172.21.15.90:0/3283524981 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce641122f0 0x7fce641a8dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:48.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.651+0000 7fce61400640 1 -- 172.21.15.90:0/3283524981 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce641a9300 con 0x7fce64114c50 2024-04-02T14:40:48.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.652+0000 7fce61400640 1 --2- 172.21.15.90:0/3283524981 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fce64114c50 0x7fce64111db0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fce5000e9f0 tx=0x7fce5000eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:40:48.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.652+0000 7fce5aa00640 1 -- 172.21.15.90:0/3283524981 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce5000cd80 con 0x7fce64114c50 2024-04-02T14:40:48.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.652+0000 7fce5aa00640 1 -- 172.21.15.90:0/3283524981 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fce50010910 con 0x7fce64114c50 2024-04-02T14:40:48.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.652+0000 7fce69000640 1 -- 172.21.15.90:0/3283524981 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fce641a95f0 con 0x7fce64114c50 2024-04-02T14:40:48.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.652+0000 7fce69000640 1 -- 172.21.15.90:0/3283524981 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fce641a9b90 con 0x7fce64114c50 2024-04-02T14:40:48.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.653+0000 7fce5aa00640 1 -- 172.21.15.90:0/3283524981 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce50018950 con 0x7fce64114c50 2024-04-02T14:40:48.657 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.657+0000 7fce6bde5740 1 Processor -- start 2024-04-02T14:40:48.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.657+0000 7fce6bde5740 1 -- start start 2024-04-02T14:40:48.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.658+0000 7fce6bde5740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556fa04e79f0 0x556fa04e6740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:48.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.658+0000 7fce6bde5740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556fa04e6d10 0x556fa04e7190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:48.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.658+0000 7fce6bde5740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556fa04f0080 0x556fa04f2470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:48.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.658+0000 7fce6bde5740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x556fa0427400 con 0x556fa04f0080 2024-04-02T14:40:48.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.658+0000 7fce6bde5740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x556fa040bf10 con 0x556fa04e79f0 2024-04-02T14:40:48.660 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.658+0000 7fce6bde5740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x556fa044b6f0 con 0x556fa04e6d10 2024-04-02T14:40:48.660 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.658+0000 7fce61e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556fa04e6d10 0x556fa04e7190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:48.660 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.658+0000 7fce62800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556fa04e79f0 0x556fa04e6740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:48.660 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.658+0000 7fce61e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556fa04e6d10 0x556fa04e7190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:49832/0 (socket says 172.21.15.90:49832) 2024-04-02T14:40:48.660 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.658+0000 7fce61e00640 1 -- 172.21.15.90:0/2823994633 learned_addr learned my addr 172.21.15.90:0/2823994633 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:40:48.660 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.658+0000 7fce62800640 1 --2- 172.21.15.90:0/2823994633 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556fa04e79f0 0x556fa04e6740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:34112/0 (socket says 172.21.15.90:34112) 2024-04-02T14:40:48.660 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.659+0000 7fce5aa00640 1 -- 172.21.15.90:0/3283524981 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fce50018b10 con 0x7fce64114c50 2024-04-02T14:40:48.660 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.659+0000 7fce61e00640 1 -- 172.21.15.90:0/2823994633 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556fa04e79f0 msgr2=0x556fa04e6740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:40:48.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.659+0000 7fce61e00640 1 --2- 172.21.15.90:0/2823994633 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556fa04e79f0 0x556fa04e6740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:48.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.659+0000 7fce61e00640 1 -- 172.21.15.90:0/2823994633 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556fa04f0080 msgr2=0x556fa04f2470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:40:48.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.659+0000 7fce61e00640 1 --2- 172.21.15.90:0/2823994633 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556fa04f0080 0x556fa04f2470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:48.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.659+0000 7fce61e00640 1 -- 172.21.15.90:0/2823994633 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556fa015f3f0 con 0x556fa04e6d10 2024-04-02T14:40:48.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.660+0000 7fce61e00640 1 --2- 172.21.15.90:0/2823994633 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556fa04e6d10 0x556fa04e7190 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fce54009230 tx=0x7fce54005780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=99c2639feb2e1583 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:40:48.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.660+0000 7fce4f400640 1 -- 172.21.15.90:0/2823994633 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce54042020 con 0x556fa04e6d10 2024-04-02T14:40:48.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.660+0000 7fce4f400640 1 -- 172.21.15.90:0/2823994633 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fce5402c950 con 0x556fa04e6d10 2024-04-02T14:40:48.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.660+0000 7fce4f400640 1 -- 172.21.15.90:0/2823994633 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce54042020 con 0x556fa04e6d10 2024-04-02T14:40:48.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.661+0000 7fce5aa00640 1 --2- 172.21.15.90:0/3283524981 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fce480a3e60 0x7fce480a6320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:48.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.661+0000 7fce5aa00640 1 -- 172.21.15.90:0/3283524981 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7fce500d0b00 con 0x7fce64114c50 2024-04-02T14:40:48.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.662+0000 7fce6bde5740 1 -- 172.21.15.90:0/2823994633 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556fa04e6d10 msgr2=0x556fa04e7190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:48.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.662+0000 7fce6bde5740 1 --2- 172.21.15.90:0/2823994633 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556fa04e6d10 0x556fa04e7190 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fce54009230 tx=0x7fce54005780 comp rx=0 tx=0).stop 2024-04-02T14:40:48.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.662+0000 7fce6bde5740 1 -- 172.21.15.90:0/2823994633 shutdown_connections 2024-04-02T14:40:48.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.662+0000 7fce6bde5740 1 --2- 172.21.15.90:0/2823994633 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556fa04f0080 0x556fa04f2470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:48.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.662+0000 7fce6bde5740 1 --2- 172.21.15.90:0/2823994633 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556fa04e6d10 0x556fa04e7190 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:48.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.662+0000 7fce6bde5740 1 --2- 172.21.15.90:0/2823994633 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556fa04e79f0 0x556fa04e6740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:48.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.662+0000 7fce6bde5740 1 -- 172.21.15.90:0/2823994633 >> 172.21.15.90:0/2823994633 conn(0x556fa04ada50 msgr2=0x556fa04e1760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:40:48.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.663+0000 7fce62800640 1 --2- 172.21.15.90:0/3283524981 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fce480a3e60 0x7fce480a6320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:48.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.663+0000 7fce6bde5740 1 -- 172.21.15.90:0/2823994633 shutdown_connections 2024-04-02T14:40:48.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.664+0000 7fce6bde5740 1 -- 172.21.15.90:0/2823994633 wait complete. 2024-04-02T14:40:48.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.664+0000 7fce6bde5740 1 -- start start 2024-04-02T14:40:48.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.665+0000 7fce62800640 1 --2- 172.21.15.90:0/3283524981 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fce480a3e60 0x7fce480a6320 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fce5c009270 tx=0x7fce5c009200 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:40:48.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.665+0000 7fce6bde5740 1 Processor -- start 2024-04-02T14:40:48.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.666+0000 7fce6bde5740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556fa04e6d10 0x556fa04ed070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:48.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.666+0000 7fce6bde5740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556fa04e79f0 0x556fa04ed5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:48.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.666+0000 7fce6bde5740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556fa04f0080 0x556fa04edaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:48.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.666+0000 7fce6bde5740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x556fa04476a0 con 0x556fa04f0080 2024-04-02T14:40:48.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.666+0000 7fce6bde5740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x556fa03f3980 con 0x556fa04e6d10 2024-04-02T14:40:48.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.666+0000 7fce6bde5740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x556fa0406cb0 con 0x556fa04e79f0 2024-04-02T14:40:48.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.666+0000 7fce6bde5740 10 client.0 _ll_register_callbacks cb 0x556fa04066a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:40:48.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.666+0000 7fce61e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556fa04e6d10 0x556fa04ed070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:48.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.666+0000 7fce61e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556fa04e6d10 0x556fa04ed070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:34122/0 (socket says 172.21.15.90:34122) 2024-04-02T14:40:48.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.666+0000 7fce61e00640 1 -- 172.21.15.90:0/3125285478 learned_addr learned my addr 172.21.15.90:0/3125285478 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:40:48.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.667+0000 7fce62800640 1 --2- 172.21.15.90:0/3125285478 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556fa04f0080 0x556fa04edaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:48.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.667+0000 7fce61e00640 1 -- 172.21.15.90:0/3125285478 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556fa04e79f0 msgr2=0x556fa04ed5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:40:48.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.667+0000 7fce61400640 1 --2- 172.21.15.90:0/3125285478 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556fa04e79f0 0x556fa04ed5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:48.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.667+0000 7fce61e00640 1 --2- 172.21.15.90:0/3125285478 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556fa04e79f0 0x556fa04ed5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:48.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.667+0000 7fce61e00640 1 -- 172.21.15.90:0/3125285478 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556fa04f0080 msgr2=0x556fa04edaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:40:48.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.667+0000 7fce61e00640 1 --2- 172.21.15.90:0/3125285478 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556fa04f0080 0x556fa04edaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:48.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.667+0000 7fce61e00640 1 -- 172.21.15.90:0/3125285478 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556fa02297d0 con 0x556fa04e6d10 2024-04-02T14:40:48.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.667+0000 7fce61400640 1 --2- 172.21.15.90:0/3125285478 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556fa04e79f0 0x556fa04ed5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:40:48.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.667+0000 7fce61e00640 1 --2- 172.21.15.90:0/3125285478 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556fa04e6d10 0x556fa04ed070 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fce5402c8a0 tx=0x7fce5402c240 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8236d46fb290c7d7 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:40:48.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.668+0000 7fce43e00640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:40:48.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.668+0000 7fce43e00640 1 -- 172.21.15.90:0/3125285478 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce54042030 con 0x556fa04e6d10 2024-04-02T14:40:48.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.668+0000 7fce43e00640 1 -- 172.21.15.90:0/3125285478 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7fce540036f0 con 0x556fa04e6d10 2024-04-02T14:40:48.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.669+0000 7fce6bde5740 1 -- 172.21.15.90:0/3125285478 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fce5402c950 con 0x556fa04e6d10 2024-04-02T14:40:48.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.669+0000 7fce43e00640 1 -- 172.21.15.90:0/3125285478 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fce54004450 con 0x556fa04e6d10 2024-04-02T14:40:48.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.671+0000 7fce6bde5740 10 client.27179 fetch_fsmap learned FSMap version 27 2024-04-02T14:40:48.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.671+0000 7fce6bde5740 1 -- 172.21.15.90:0/3125285478 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x556fa04fef50 con 0x556fa04e6d10 2024-04-02T14:40:48.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.672+0000 7fce43e00640 1 -- 172.21.15.90:0/3125285478 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fce5402c950 con 0x556fa04e6d10 2024-04-02T14:40:48.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.672+0000 7fce43e00640 1 -- 172.21.15.90:0/3125285478 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fce54052030 con 0x556fa04e6d10 2024-04-02T14:40:48.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.672+0000 7fce6bde5740 10 client.27179 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:40:48.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.673+0000 7fce6bde5740 10 client.27179 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.c 2024-04-02T14:40:48.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.673+0000 7fce6bde5740 20 client.27179 populate_metadata read hostname 'smithi090' 2024-04-02T14:40:48.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.673+0000 7fce6bde5740 1 --2- 172.21.15.90:0/3125285478 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x556fa05002a0 0x556fa0502690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:48.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.673+0000 7fce6bde5740 4 client.27179 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:40:48.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.673+0000 7fce6bde5740 1 -- 172.21.15.90:0/3125285478 --> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556fa040b040 con 0x556fa05002a0 2024-04-02T14:40:48.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.675+0000 7fce61400640 1 --2- 172.21.15.90:0/3125285478 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x556fa05002a0 0x556fa0502690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:48.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.675+0000 7fce61400640 1 --2- 172.21.15.90:0/3125285478 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x556fa05002a0 0x556fa0502690 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=87bbfd9b6f203839 server_cookie=61cab1bf7d8e0c69 in_seq=0 out_seq=0 2024-04-02T14:40:48.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.675+0000 7fce43e00640 10 client.27179 ms_handle_connect on v2:172.21.15.177:6836/4223414858 2024-04-02T14:40:48.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.676+0000 7fce43e00640 1 -- 172.21.15.90:0/3125285478 <== mds.4 v2:172.21.15.177:6836/4223414858 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x556fa040b040 con 0x556fa05002a0 2024-04-02T14:40:48.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.676+0000 7fce43e00640 10 client.27179 handle_command_reply: tid=0 2024-04-02T14:40:48.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.677+0000 7fce6bde5740 1 client.27179 shutdown 2024-04-02T14:40:48.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.677+0000 7fce6bde5740 1 -- 172.21.15.90:0/3125285478 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556fa04e6d10 msgr2=0x556fa04ed070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:48.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.677+0000 7fce6bde5740 1 --2- 172.21.15.90:0/3125285478 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556fa04e6d10 0x556fa04ed070 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fce5402c8a0 tx=0x7fce5402c240 comp rx=0 tx=0).stop 2024-04-02T14:40:48.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.678+0000 7fce6bde5740 1 -- 172.21.15.90:0/3125285478 shutdown_connections 2024-04-02T14:40:48.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.678+0000 7fce6bde5740 1 --2- 172.21.15.90:0/3125285478 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x556fa05002a0 0x556fa0502690 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:48.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.678+0000 7fce6bde5740 1 --2- 172.21.15.90:0/3125285478 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556fa04f0080 0x556fa04edaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:48.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.678+0000 7fce6bde5740 1 --2- 172.21.15.90:0/3125285478 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556fa04e79f0 0x556fa04ed5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:48.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.678+0000 7fce6bde5740 1 --2- 172.21.15.90:0/3125285478 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556fa04e6d10 0x556fa04ed070 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:48.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.678+0000 7fce6bde5740 1 -- 172.21.15.90:0/3125285478 >> 172.21.15.90:0/3125285478 conn(0x556fa04ada50 msgr2=0x556fa04e34b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:40:48.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.678+0000 7fce43e00640 0 client.27179 ms_handle_reset on v2:172.21.15.177:6836/4223414858 2024-04-02T14:40:48.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.678+0000 7fce6bde5740 1 -- 172.21.15.90:0/3125285478 shutdown_connections 2024-04-02T14:40:48.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.679+0000 7fce6bde5740 1 -- 172.21.15.90:0/3125285478 wait complete. 2024-04-02T14:40:48.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.679+0000 7fce6bde5740 20 client.27179 trim_cache size 0 max 16384 2024-04-02T14:40:48.687 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:40:48 smithi090 ceph-mon[31674]: from='client.? 172.21.15.90:0/42471831' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-02T14:40:48.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.704+0000 7fce6bde5740 1 Processor -- start 2024-04-02T14:40:48.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.704+0000 7fce6bde5740 1 -- start start 2024-04-02T14:40:48.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.704+0000 7fce6bde5740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556fa04e6d10 0x556fa04fbb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:48.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.705+0000 7fce6bde5740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556fa04e79f0 0x556fa0507c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:48.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.705+0000 7fce6bde5740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556fa04f0080 0x556fa05081a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:48.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.705+0000 7fce6bde5740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fce4400e860 con 0x556fa04e79f0 2024-04-02T14:40:48.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.705+0000 7fce6bde5740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fce440103a0 con 0x556fa04f0080 2024-04-02T14:40:48.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.705+0000 7fce6bde5740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fce4400ef30 con 0x556fa04e6d10 2024-04-02T14:40:48.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.705+0000 7fce61e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556fa04e6d10 0x556fa04fbb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:48.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.705+0000 7fce61e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556fa04e6d10 0x556fa04fbb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:49848/0 (socket says 172.21.15.90:49848) 2024-04-02T14:40:48.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.705+0000 7fce61e00640 1 -- 172.21.15.90:0/1145096568 learned_addr learned my addr 172.21.15.90:0/1145096568 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:40:48.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.705+0000 7fce62800640 1 --2- 172.21.15.90:0/1145096568 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556fa04f0080 0x556fa05081a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:48.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.705+0000 7fce61400640 1 --2- 172.21.15.90:0/1145096568 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556fa04e79f0 0x556fa0507c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:48.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.705+0000 7fce61e00640 1 -- 172.21.15.90:0/1145096568 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556fa04f0080 msgr2=0x556fa05081a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:40:48.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.705+0000 7fce61e00640 1 --2- 172.21.15.90:0/1145096568 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556fa04f0080 0x556fa05081a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:48.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.705+0000 7fce61e00640 1 -- 172.21.15.90:0/1145096568 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556fa04e79f0 msgr2=0x556fa0507c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:40:48.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.705+0000 7fce61e00640 1 --2- 172.21.15.90:0/1145096568 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556fa04e79f0 0x556fa0507c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:48.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.705+0000 7fce61e00640 1 -- 172.21.15.90:0/1145096568 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556fa04e5d40 con 0x556fa04e6d10 2024-04-02T14:40:48.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.706+0000 7fce61e00640 1 --2- 172.21.15.90:0/1145096568 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556fa04e6d10 0x556fa04fbb20 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fce5402c600 tx=0x7fce54005970 comp rx=0 tx=0).ready entity=mon.1 client_cookie=56f4df7171210b1a server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:40:48.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.706+0000 7fce4ea00640 1 -- 172.21.15.90:0/1145096568 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce5405b020 con 0x556fa04e6d10 2024-04-02T14:40:48.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.706+0000 7fce4ea00640 1 -- 172.21.15.90:0/1145096568 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fce54009520 con 0x556fa04e6d10 2024-04-02T14:40:48.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.706+0000 7fce4ea00640 1 -- 172.21.15.90:0/1145096568 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce54034be0 con 0x556fa04e6d10 2024-04-02T14:40:48.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.706+0000 7fce6bde5740 1 -- 172.21.15.90:0/1145096568 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556fa04e6d10 msgr2=0x556fa04fbb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:48.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.706+0000 7fce6bde5740 1 --2- 172.21.15.90:0/1145096568 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556fa04e6d10 0x556fa04fbb20 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fce5402c600 tx=0x7fce54005970 comp rx=0 tx=0).stop 2024-04-02T14:40:48.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.707+0000 7fce6bde5740 1 -- 172.21.15.90:0/1145096568 shutdown_connections 2024-04-02T14:40:48.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.707+0000 7fce6bde5740 1 --2- 172.21.15.90:0/1145096568 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556fa04f0080 0x556fa05081a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:48.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.707+0000 7fce6bde5740 1 --2- 172.21.15.90:0/1145096568 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556fa04e79f0 0x556fa0507c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:48.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.707+0000 7fce6bde5740 1 --2- 172.21.15.90:0/1145096568 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556fa04e6d10 0x556fa04fbb20 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:48.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.707+0000 7fce6bde5740 1 -- 172.21.15.90:0/1145096568 >> 172.21.15.90:0/1145096568 conn(0x556fa04ada50 msgr2=0x556fa04eb940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:40:48.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.707+0000 7fce6bde5740 1 -- 172.21.15.90:0/1145096568 shutdown_connections 2024-04-02T14:40:48.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.707+0000 7fce6bde5740 1 -- 172.21.15.90:0/1145096568 wait complete. 2024-04-02T14:40:48.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.707+0000 7fce6bde5740 1 -- start start 2024-04-02T14:40:48.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.707+0000 7fce6bde5740 1 Processor -- start 2024-04-02T14:40:48.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.708+0000 7fce6bde5740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556fa04e6d10 0x556fa04f8030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:48.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.708+0000 7fce6bde5740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556fa04e79f0 0x556fa04f8570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:48.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.708+0000 7fce6bde5740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556fa04f0080 0x556fa04e92e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:48.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.708+0000 7fce6bde5740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fce4400fcd0 con 0x556fa04f0080 2024-04-02T14:40:48.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.708+0000 7fce6bde5740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fce4400f600 con 0x556fa04e6d10 2024-04-02T14:40:48.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.708+0000 7fce6bde5740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fce4400c840 con 0x556fa04e79f0 2024-04-02T14:40:48.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.708+0000 7fce6bde5740 10 client.0 _ll_register_callbacks cb 0x7fce54009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:40:48.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.708+0000 7fce61e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556fa04e6d10 0x556fa04f8030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:48.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.708+0000 7fce61e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556fa04e6d10 0x556fa04f8030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:34142/0 (socket says 172.21.15.90:34142) 2024-04-02T14:40:48.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.708+0000 7fce61e00640 1 -- 172.21.15.90:0/3070987869 learned_addr learned my addr 172.21.15.90:0/3070987869 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:40:48.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.708+0000 7fce62800640 1 --2- 172.21.15.90:0/3070987869 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556fa04f0080 0x556fa04e92e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:48.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.708+0000 7fce61400640 1 --2- 172.21.15.90:0/3070987869 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556fa04e79f0 0x556fa04f8570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:48.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.708+0000 7fce61e00640 1 -- 172.21.15.90:0/3070987869 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556fa04e79f0 msgr2=0x556fa04f8570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:40:48.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.708+0000 7fce61e00640 1 --2- 172.21.15.90:0/3070987869 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556fa04e79f0 0x556fa04f8570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:48.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.708+0000 7fce61e00640 1 -- 172.21.15.90:0/3070987869 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556fa04f0080 msgr2=0x556fa04e92e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:40:48.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.708+0000 7fce61e00640 1 --2- 172.21.15.90:0/3070987869 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556fa04f0080 0x556fa04e92e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:48.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.708+0000 7fce61e00640 1 -- 172.21.15.90:0/3070987869 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556fa04e9820 con 0x556fa04e6d10 2024-04-02T14:40:48.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.708+0000 7fce61400640 1 --2- 172.21.15.90:0/3070987869 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556fa04e79f0 0x556fa04f8570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:40:48.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.709+0000 7fce61e00640 1 --2- 172.21.15.90:0/3070987869 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556fa04e6d10 0x556fa04f8030 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fce5402ca50 tx=0x7fce54047680 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6756263574761e35 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:40:48.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.709+0000 7fce43e00640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:40:48.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.709+0000 7fce43e00640 1 -- 172.21.15.90:0/3070987869 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce5405b030 con 0x556fa04e6d10 2024-04-02T14:40:48.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.709+0000 7fce43e00640 1 -- 172.21.15.90:0/3070987869 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7fce54034b80 con 0x556fa04e6d10 2024-04-02T14:40:48.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.709+0000 7fce6bde5740 1 -- 172.21.15.90:0/3070987869 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fce54009520 con 0x556fa04e6d10 2024-04-02T14:40:48.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.710+0000 7fce43e00640 1 -- 172.21.15.90:0/3070987869 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fce54034da0 con 0x556fa04e6d10 2024-04-02T14:40:48.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.710+0000 7fce6bde5740 10 client.27185 fetch_fsmap learned FSMap version 27 2024-04-02T14:40:48.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.710+0000 7fce6bde5740 1 -- 172.21.15.90:0/3070987869 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x556fa0565cd0 con 0x556fa04e6d10 2024-04-02T14:40:48.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.711+0000 7fce43e00640 1 -- 172.21.15.90:0/3070987869 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fce54009520 con 0x556fa04e6d10 2024-04-02T14:40:48.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.712+0000 7fce43e00640 1 -- 172.21.15.90:0/3070987869 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fce54052040 con 0x556fa04e6d10 2024-04-02T14:40:48.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.713+0000 7fce6bde5740 10 client.27185 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:40:48.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.713+0000 7fce6bde5740 10 client.27185 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.c 2024-04-02T14:40:48.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.713+0000 7fce6bde5740 20 client.27185 populate_metadata read hostname 'smithi090' 2024-04-02T14:40:48.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.713+0000 7fce6bde5740 1 --2- 172.21.15.90:0/3070987869 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x556fa05662d0 0x556fa05686f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:48.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.713+0000 7fce6bde5740 4 client.27185 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-02T14:40:48.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.713+0000 7fce6bde5740 1 -- 172.21.15.90:0/3070987869 --> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] -- command(tid 0: {"prefix": "damage ls"}) -- 0x556fa0458c30 con 0x556fa05662d0 2024-04-02T14:40:48.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.714+0000 7fce61400640 1 --2- 172.21.15.90:0/3070987869 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x556fa05662d0 0x556fa05686f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:48.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.715+0000 7fce61400640 1 --2- 172.21.15.90:0/3070987869 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x556fa05662d0 0x556fa05686f0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=cde4c0ff92e5ec0a server_cookie=2fd15e9af5ef5e27 in_seq=0 out_seq=0 2024-04-02T14:40:48.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.715+0000 7fce43e00640 10 client.27185 ms_handle_connect on v2:172.21.15.177:6836/4223414858 2024-04-02T14:40:48.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.715+0000 7fce43e00640 1 -- 172.21.15.90:0/3070987869 <== mds.4 v2:172.21.15.177:6836/4223414858 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x556fa0458c30 con 0x556fa05662d0 2024-04-02T14:40:48.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.715+0000 7fce43e00640 10 client.27185 handle_command_reply: tid=0 2024-04-02T14:40:48.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.715+0000 7fce6bde5740 1 client.27185 shutdown 2024-04-02T14:40:48.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.715+0000 7fce6bde5740 1 -- 172.21.15.90:0/3070987869 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556fa04e6d10 msgr2=0x556fa04f8030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:48.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.715+0000 7fce6bde5740 1 --2- 172.21.15.90:0/3070987869 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556fa04e6d10 0x556fa04f8030 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fce5402ca50 tx=0x7fce54047680 comp rx=0 tx=0).stop 2024-04-02T14:40:48.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.715+0000 7fce6bde5740 1 -- 172.21.15.90:0/3070987869 shutdown_connections 2024-04-02T14:40:48.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.715+0000 7fce6bde5740 1 --2- 172.21.15.90:0/3070987869 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x556fa05662d0 0x556fa05686f0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:48.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.715+0000 7fce6bde5740 1 --2- 172.21.15.90:0/3070987869 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556fa04f0080 0x556fa04e92e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:48.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.715+0000 7fce6bde5740 1 --2- 172.21.15.90:0/3070987869 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556fa04e79f0 0x556fa04f8570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:48.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.715+0000 7fce6bde5740 1 --2- 172.21.15.90:0/3070987869 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556fa04e6d10 0x556fa04f8030 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:48.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.715+0000 7fce6bde5740 1 -- 172.21.15.90:0/3070987869 >> 172.21.15.90:0/3070987869 conn(0x556fa04ada50 msgr2=0x556fa04e08a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:40:48.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.715+0000 7fce43e00640 0 client.27185 ms_handle_reset on v2:172.21.15.177:6836/4223414858 2024-04-02T14:40:48.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.715+0000 7fce6bde5740 1 -- 172.21.15.90:0/3070987869 shutdown_connections 2024-04-02T14:40:48.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.716+0000 7fce6bde5740 1 -- 172.21.15.90:0/3070987869 wait complete. 2024-04-02T14:40:48.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.716+0000 7fce6bde5740 20 client.27185 trim_cache size 0 max 16384 2024-04-02T14:40:48.717 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-02T14:40:48.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.716+0000 7fce69000640 1 -- 172.21.15.90:0/3283524981 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fce480a3e60 msgr2=0x7fce480a6320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:48.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.716+0000 7fce69000640 1 --2- 172.21.15.90:0/3283524981 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fce480a3e60 0x7fce480a6320 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fce5c009270 tx=0x7fce5c009200 comp rx=0 tx=0).stop 2024-04-02T14:40:48.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.716+0000 7fce69000640 1 -- 172.21.15.90:0/3283524981 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fce64114c50 msgr2=0x7fce64111db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:48.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.716+0000 7fce69000640 1 --2- 172.21.15.90:0/3283524981 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fce64114c50 0x7fce64111db0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fce5000e9f0 tx=0x7fce5000eeb0 comp rx=0 tx=0).stop 2024-04-02T14:40:48.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.716+0000 7fce69000640 1 -- 172.21.15.90:0/3283524981 shutdown_connections 2024-04-02T14:40:48.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.717+0000 7fce69000640 1 --2- 172.21.15.90:0/3283524981 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fce480a3e60 0x7fce480a6320 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:48.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.717+0000 7fce69000640 1 --2- 172.21.15.90:0/3283524981 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce641122f0 0x7fce641a8dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:48.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.717+0000 7fce69000640 1 --2- 172.21.15.90:0/3283524981 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fce64114c50 0x7fce64111db0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:48.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.717+0000 7fce69000640 1 --2- 172.21.15.90:0/3283524981 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fce64109420 0x7fce64111870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:48.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.717+0000 7fce69000640 1 -- 172.21.15.90:0/3283524981 >> 172.21.15.90:0/3283524981 conn(0x7fce6406d040 msgr2=0x7fce64070840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:40:48.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.717+0000 7fce69000640 1 -- 172.21.15.90:0/3283524981 shutdown_connections 2024-04-02T14:40:48.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:48.717+0000 7fce69000640 1 -- 172.21.15.90:0/3283524981 wait complete. 2024-04-02T14:40:48.733 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-04-02T14:40:48.859 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:40:48 smithi092 ceph-mon[36497]: from='client.? 172.21.15.90:0/42471831' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-02T14:40:48.935 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:40:48 smithi177 ceph-mon[36431]: from='client.? 172.21.15.90:0/42471831' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-02T14:40:49.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.014+0000 7f987aa00640 1 -- 172.21.15.90:0/2879846427 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9874100b30 msgr2=0x7f9874100f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:49.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.014+0000 7f987aa00640 1 --2- 172.21.15.90:0/2879846427 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9874100b30 0x7f9874100f10 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f9864009930 tx=0x7f986402bcb0 comp rx=0 tx=0).stop 2024-04-02T14:40:49.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.015+0000 7f987aa00640 1 -- 172.21.15.90:0/2879846427 shutdown_connections 2024-04-02T14:40:49.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.015+0000 7f987aa00640 1 --2- 172.21.15.90:0/2879846427 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9874105f30 0x7f98741063e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:49.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.015+0000 7f987aa00640 1 --2- 172.21.15.90:0/2879846427 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9874101450 0x7f9874105910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:49.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.015+0000 7f987aa00640 1 --2- 172.21.15.90:0/2879846427 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9874100b30 0x7f9874100f10 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:49.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.015+0000 7f987aa00640 1 -- 172.21.15.90:0/2879846427 >> 172.21.15.90:0/2879846427 conn(0x7f9874076d80 msgr2=0x7f9874077190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:40:49.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.016+0000 7f987aa00640 1 -- 172.21.15.90:0/2879846427 shutdown_connections 2024-04-02T14:40:49.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.016+0000 7f987aa00640 1 -- 172.21.15.90:0/2879846427 wait complete. 2024-04-02T14:40:49.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.017+0000 7f987aa00640 1 Processor -- start 2024-04-02T14:40:49.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.017+0000 7f987aa00640 1 -- start start 2024-04-02T14:40:49.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.018+0000 7f987aa00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9874100b30 0x7f98741a46d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:49.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.018+0000 7f987aa00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9874101450 0x7f98741a4c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:49.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.018+0000 7f987aa00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9874105f30 0x7f987419e890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:49.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.018+0000 7f987aa00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f9874118550 con 0x7f9874105f30 2024-04-02T14:40:49.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.018+0000 7f987aa00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f98741183d0 con 0x7f9874101450 2024-04-02T14:40:49.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.018+0000 7f987aa00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f98741186d0 con 0x7f9874100b30 2024-04-02T14:40:49.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.018+0000 7f9873e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9874100b30 0x7f98741a46d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:49.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.018+0000 7f9873e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9874100b30 0x7f98741a46d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:49864/0 (socket says 172.21.15.90:49864) 2024-04-02T14:40:49.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.018+0000 7f9873e00640 1 -- 172.21.15.90:0/3842915585 learned_addr learned my addr 172.21.15.90:0/3842915585 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:40:49.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.019+0000 7f9878a00640 1 --2- 172.21.15.90:0/3842915585 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9874105f30 0x7f987419e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:49.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.019+0000 7f9873400640 1 --2- 172.21.15.90:0/3842915585 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9874101450 0x7f98741a4c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:49.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.019+0000 7f9873e00640 1 -- 172.21.15.90:0/3842915585 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9874101450 msgr2=0x7f98741a4c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:49.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.019+0000 7f9873e00640 1 --2- 172.21.15.90:0/3842915585 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9874101450 0x7f98741a4c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:49.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.019+0000 7f9873e00640 1 -- 172.21.15.90:0/3842915585 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9874105f30 msgr2=0x7f987419e890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:49.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.019+0000 7f9873e00640 1 --2- 172.21.15.90:0/3842915585 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9874105f30 0x7f987419e890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:49.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.019+0000 7f9873e00640 1 -- 172.21.15.90:0/3842915585 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f987419f060 con 0x7f9874100b30 2024-04-02T14:40:49.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.020+0000 7f9873e00640 1 --2- 172.21.15.90:0/3842915585 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9874100b30 0x7f98741a46d0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f9864000c00 tx=0x7f98640043b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:40:49.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.020+0000 7f9870c00640 1 -- 172.21.15.90:0/3842915585 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9864042020 con 0x7f9874100b30 2024-04-02T14:40:49.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.020+0000 7f9870c00640 1 -- 172.21.15.90:0/3842915585 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9864005000 con 0x7f9874100b30 2024-04-02T14:40:49.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.020+0000 7f9870c00640 1 -- 172.21.15.90:0/3842915585 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9864003a10 con 0x7f9874100b30 2024-04-02T14:40:49.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.021+0000 7f987aa00640 1 -- 172.21.15.90:0/3842915585 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f987419f2f0 con 0x7f9874100b30 2024-04-02T14:40:49.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.022+0000 7f987aa00640 1 -- 172.21.15.90:0/3842915585 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f98741ab520 con 0x7f9874100b30 2024-04-02T14:40:49.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.026+0000 7f9870c00640 1 -- 172.21.15.90:0/3842915585 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f9864045020 con 0x7f9874100b30 2024-04-02T14:40:49.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.026+0000 7f987d7ca740 1 Processor -- start 2024-04-02T14:40:49.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.026+0000 7f987d7ca740 1 -- start start 2024-04-02T14:40:49.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.027+0000 7f987d7ca740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ff2616a9f0 0x55ff26169740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:49.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.027+0000 7f987d7ca740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ff26169d10 0x55ff2616a190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:49.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.027+0000 7f987d7ca740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ff26173080 0x55ff26175470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:49.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.027+0000 7f987d7ca740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55ff260aa400 con 0x55ff26169d10 2024-04-02T14:40:49.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.027+0000 7f987d7ca740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55ff2608ef10 con 0x55ff26173080 2024-04-02T14:40:49.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.027+0000 7f987d7ca740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55ff260ce6f0 con 0x55ff2616a9f0 2024-04-02T14:40:49.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.028+0000 7f9873400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ff26173080 0x55ff26175470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:49.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.028+0000 7f9878a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ff2616a9f0 0x55ff26169740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:49.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.028+0000 7f9878a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ff2616a9f0 0x55ff26169740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:49868/0 (socket says 172.21.15.90:49868) 2024-04-02T14:40:49.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.028+0000 7f9878a00640 1 -- 172.21.15.90:0/990396199 learned_addr learned my addr 172.21.15.90:0/990396199 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:40:49.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.028+0000 7f9878a00640 1 -- 172.21.15.90:0/990396199 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ff26173080 msgr2=0x55ff26175470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:40:49.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.028+0000 7f9878a00640 1 --2- 172.21.15.90:0/990396199 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ff26173080 0x55ff26175470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:49.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.028+0000 7f9878a00640 1 -- 172.21.15.90:0/990396199 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ff26169d10 msgr2=0x55ff2616a190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:40:49.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.028+0000 7f9878a00640 1 --2- 172.21.15.90:0/990396199 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ff26169d10 0x55ff2616a190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:49.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.028+0000 7f9878a00640 1 -- 172.21.15.90:0/990396199 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ff25de23f0 con 0x55ff2616a9f0 2024-04-02T14:40:49.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.028+0000 7f9878a00640 1 --2- 172.21.15.90:0/990396199 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ff2616a9f0 0x55ff26169740 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f986c009fc0 tx=0x7f986c010860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ae7106acc318f37c server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:40:49.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.029+0000 7f9870c00640 1 --2- 172.21.15.90:0/3842915585 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f98580a3de0 0x7f98580a62a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:49.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.029+0000 7f9873e00640 1 --2- 172.21.15.90:0/3842915585 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f98580a3de0 0x7f98580a62a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:49.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.030+0000 7f9870c00640 1 -- 172.21.15.90:0/3842915585 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f98640f20c0 con 0x7f9874100b30 2024-04-02T14:40:49.031 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.031+0000 7f9873e00640 1 --2- 172.21.15.90:0/3842915585 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f98580a3de0 0x7f98580a62a0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x55ff2616af40 tx=0x7f986402c1d0 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:40:49.032 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.031+0000 7f9869200640 1 -- 172.21.15.90:0/990396199 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f986c003b70 con 0x55ff2616a9f0 2024-04-02T14:40:49.032 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.031+0000 7f9869200640 1 -- 172.21.15.90:0/990396199 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f986c003d10 con 0x55ff2616a9f0 2024-04-02T14:40:49.032 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.031+0000 7f9869200640 1 -- 172.21.15.90:0/990396199 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f986c011aa0 con 0x55ff2616a9f0 2024-04-02T14:40:49.033 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.032+0000 7f987d7ca740 1 -- 172.21.15.90:0/990396199 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ff2616a9f0 msgr2=0x55ff26169740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:49.033 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.032+0000 7f987d7ca740 1 --2- 172.21.15.90:0/990396199 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ff2616a9f0 0x55ff26169740 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f986c009fc0 tx=0x7f986c010860 comp rx=0 tx=0).stop 2024-04-02T14:40:49.033 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.033+0000 7f987d7ca740 1 -- 172.21.15.90:0/990396199 shutdown_connections 2024-04-02T14:40:49.033 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.033+0000 7f987d7ca740 1 --2- 172.21.15.90:0/990396199 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ff26173080 0x55ff26175470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:49.033 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.033+0000 7f987d7ca740 1 --2- 172.21.15.90:0/990396199 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ff26169d10 0x55ff2616a190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:49.033 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.033+0000 7f987d7ca740 1 --2- 172.21.15.90:0/990396199 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ff2616a9f0 0x55ff26169740 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:49.034 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.033+0000 7f987d7ca740 1 -- 172.21.15.90:0/990396199 >> 172.21.15.90:0/990396199 conn(0x55ff26130a50 msgr2=0x55ff26164760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:40:49.034 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.033+0000 7f987d7ca740 1 -- 172.21.15.90:0/990396199 shutdown_connections 2024-04-02T14:40:49.034 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.033+0000 7f987d7ca740 1 -- 172.21.15.90:0/990396199 wait complete. 2024-04-02T14:40:49.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.034+0000 7f987d7ca740 1 -- start start 2024-04-02T14:40:49.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.035+0000 7f987d7ca740 1 Processor -- start 2024-04-02T14:40:49.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.036+0000 7f987d7ca740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ff26169d10 0x55ff26170190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:49.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.036+0000 7f987d7ca740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ff2616a9f0 0x55ff261706d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:49.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.036+0000 7f987d7ca740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ff26173080 0x55ff26170c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:49.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.036+0000 7f987d7ca740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55ff260ca6a0 con 0x55ff26169d10 2024-04-02T14:40:49.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.036+0000 7f987d7ca740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55ff26076980 con 0x55ff2616a9f0 2024-04-02T14:40:49.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.036+0000 7f987d7ca740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55ff26089cb0 con 0x55ff26173080 2024-04-02T14:40:49.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.036+0000 7f987d7ca740 10 client.0 _ll_register_callbacks cb 0x55ff260896a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:40:49.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.037+0000 7f9878a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ff26173080 0x55ff26170c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:49.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.037+0000 7f9878a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ff26173080 0x55ff26170c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:49874/0 (socket says 172.21.15.90:49874) 2024-04-02T14:40:49.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.037+0000 7f9878a00640 1 -- 172.21.15.90:0/2675084628 learned_addr learned my addr 172.21.15.90:0/2675084628 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:40:49.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.037+0000 7f9878a00640 1 -- 172.21.15.90:0/2675084628 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ff2616a9f0 msgr2=0x55ff261706d0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-02T14:40:49.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.037+0000 7f9878a00640 1 --2- 172.21.15.90:0/2675084628 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ff2616a9f0 0x55ff261706d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:49.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.037+0000 7f9878a00640 1 -- 172.21.15.90:0/2675084628 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ff26169d10 msgr2=0x55ff26170190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:40:49.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.037+0000 7f9878a00640 1 --2- 172.21.15.90:0/2675084628 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ff26169d10 0x55ff26170190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:49.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.037+0000 7f9878a00640 1 -- 172.21.15.90:0/2675084628 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ff25eac7d0 con 0x55ff26173080 2024-04-02T14:40:49.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.038+0000 7f9878a00640 1 --2- 172.21.15.90:0/2675084628 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ff26173080 0x55ff26170c10 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f986c009350 tx=0x7f986c009bd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e45497c94a237772 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:40:49.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.038+0000 7f985d600640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:40:49.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.038+0000 7f985d600640 1 -- 172.21.15.90:0/2675084628 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f986c004a40 con 0x55ff26173080 2024-04-02T14:40:49.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.038+0000 7f985d600640 1 -- 172.21.15.90:0/2675084628 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f986c004be0 con 0x55ff26173080 2024-04-02T14:40:49.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.038+0000 7f987d7ca740 1 -- 172.21.15.90:0/2675084628 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f986c003d10 con 0x55ff26173080 2024-04-02T14:40:49.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.038+0000 7f985d600640 1 -- 172.21.15.90:0/2675084628 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f986c003ec0 con 0x55ff26173080 2024-04-02T14:40:49.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.038+0000 7f987d7ca740 10 client.27067 fetch_fsmap learned FSMap version 27 2024-04-02T14:40:49.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.039+0000 7f987d7ca740 1 -- 172.21.15.90:0/2675084628 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55ff261820a0 con 0x55ff26173080 2024-04-02T14:40:49.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.039+0000 7f985d600640 1 -- 172.21.15.90:0/2675084628 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f986c003d10 con 0x55ff26173080 2024-04-02T14:40:49.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.039+0000 7f985d600640 1 -- 172.21.15.90:0/2675084628 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f986c027950 con 0x55ff26173080 2024-04-02T14:40:49.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.039+0000 7f987d7ca740 10 client.27067 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:40:49.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.039+0000 7f987d7ca740 10 client.27067 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-04-02T14:40:49.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.039+0000 7f987d7ca740 20 client.27067 populate_metadata read hostname 'smithi090' 2024-04-02T14:40:49.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.039+0000 7f987d7ca740 1 --2- 172.21.15.90:0/2675084628 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x55ff26183450 0x55ff26185840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:49.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.039+0000 7f987d7ca740 4 client.27067 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:40:49.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.039+0000 7f987d7ca740 1 -- 172.21.15.90:0/2675084628 --> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ff2608e040 con 0x55ff26183450 2024-04-02T14:40:49.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.040+0000 7f9873400640 1 --2- 172.21.15.90:0/2675084628 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x55ff26183450 0x55ff26185840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:49.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.040+0000 7f9873400640 1 --2- 172.21.15.90:0/2675084628 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x55ff26183450 0x55ff26185840 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=d88ed771318cc9ef server_cookie=3b17e5e5cf8de666 in_seq=0 out_seq=0 2024-04-02T14:40:49.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.040+0000 7f985d600640 10 client.27067 ms_handle_connect on v2:172.21.15.92:6836/1270635939 2024-04-02T14:40:49.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.041+0000 7f985d600640 1 -- 172.21.15.90:0/2675084628 <== mds.3 v2:172.21.15.92:6836/1270635939 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ff2608e040 con 0x55ff26183450 2024-04-02T14:40:49.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.041+0000 7f985d600640 10 client.27067 handle_command_reply: tid=0 2024-04-02T14:40:49.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.041+0000 7f987d7ca740 1 client.27067 shutdown 2024-04-02T14:40:49.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.042+0000 7f987d7ca740 1 -- 172.21.15.90:0/2675084628 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ff26173080 msgr2=0x55ff26170c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:49.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.042+0000 7f987d7ca740 1 --2- 172.21.15.90:0/2675084628 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ff26173080 0x55ff26170c10 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f986c009350 tx=0x7f986c009bd0 comp rx=0 tx=0).stop 2024-04-02T14:40:49.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.042+0000 7f987d7ca740 1 -- 172.21.15.90:0/2675084628 shutdown_connections 2024-04-02T14:40:49.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.042+0000 7f987d7ca740 1 --2- 172.21.15.90:0/2675084628 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x55ff26183450 0x55ff26185840 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:49.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.042+0000 7f987d7ca740 1 --2- 172.21.15.90:0/2675084628 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ff26173080 0x55ff26170c10 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:49.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.042+0000 7f987d7ca740 1 --2- 172.21.15.90:0/2675084628 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ff2616a9f0 0x55ff261706d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:49.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.042+0000 7f987d7ca740 1 --2- 172.21.15.90:0/2675084628 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ff26169d10 0x55ff26170190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:49.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.042+0000 7f987d7ca740 1 -- 172.21.15.90:0/2675084628 >> 172.21.15.90:0/2675084628 conn(0x55ff26130a50 msgr2=0x55ff261664b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:40:49.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.042+0000 7f985d600640 0 client.27067 ms_handle_reset on v2:172.21.15.92:6836/1270635939 2024-04-02T14:40:49.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.042+0000 7f987d7ca740 1 -- 172.21.15.90:0/2675084628 shutdown_connections 2024-04-02T14:40:49.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.043+0000 7f987d7ca740 1 -- 172.21.15.90:0/2675084628 wait complete. 2024-04-02T14:40:49.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.043+0000 7f987d7ca740 20 client.27067 trim_cache size 0 max 16384 2024-04-02T14:40:49.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.069+0000 7f987d7ca740 1 Processor -- start 2024-04-02T14:40:49.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.069+0000 7f987d7ca740 1 -- start start 2024-04-02T14:40:49.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.069+0000 7f987d7ca740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ff26169d10 0x55ff261728c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:49.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.069+0000 7f987d7ca740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ff2616a9f0 0x55ff2617eb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:49.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.069+0000 7f987d7ca740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ff26173080 0x55ff26170860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:49.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.070+0000 7f987d7ca740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f98741b3e50 con 0x55ff26169d10 2024-04-02T14:40:49.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.070+0000 7f987d7ca740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f98741b5990 con 0x55ff2616a9f0 2024-04-02T14:40:49.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.070+0000 7f987d7ca740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f98741b4520 con 0x55ff26173080 2024-04-02T14:40:49.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.070+0000 7f9873400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ff2616a9f0 0x55ff2617eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:49.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.070+0000 7f9873400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ff2616a9f0 0x55ff2617eb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:34182/0 (socket says 172.21.15.90:34182) 2024-04-02T14:40:49.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.070+0000 7f9873400640 1 -- 172.21.15.90:0/4090614414 learned_addr learned my addr 172.21.15.90:0/4090614414 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:40:49.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.070+0000 7f9878a00640 1 --2- 172.21.15.90:0/4090614414 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ff26173080 0x55ff26170860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:49.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.070+0000 7f9878a00640 1 --2- 172.21.15.90:0/4090614414 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ff26169d10 0x55ff261728c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:49.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.070+0000 7f9873400640 1 -- 172.21.15.90:0/4090614414 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ff26173080 msgr2=0x55ff26170860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:40:49.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.070+0000 7f9873400640 1 --2- 172.21.15.90:0/4090614414 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ff26173080 0x55ff26170860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:49.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.070+0000 7f9873400640 1 -- 172.21.15.90:0/4090614414 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ff26169d10 msgr2=0x55ff261728c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:40:49.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.070+0000 7f9873400640 1 --2- 172.21.15.90:0/4090614414 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ff26169d10 0x55ff261728c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:49.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.070+0000 7f9873400640 1 -- 172.21.15.90:0/4090614414 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ff26168f70 con 0x55ff2616a9f0 2024-04-02T14:40:49.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.070+0000 7f9878a00640 1 --2- 172.21.15.90:0/4090614414 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ff26169d10 0x55ff261728c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:40:49.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.071+0000 7f9873400640 1 --2- 172.21.15.90:0/4090614414 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ff2616a9f0 0x55ff2617eb40 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f9860006e10 tx=0x7f9860009110 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4628f33ebc509a21 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:40:49.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.071+0000 7f985fe00640 1 -- 172.21.15.90:0/4090614414 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9860016a40 con 0x55ff2616a9f0 2024-04-02T14:40:49.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.071+0000 7f985fe00640 1 -- 172.21.15.90:0/4090614414 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f986001f030 con 0x55ff2616a9f0 2024-04-02T14:40:49.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.071+0000 7f985fe00640 1 -- 172.21.15.90:0/4090614414 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98600143b0 con 0x55ff2616a9f0 2024-04-02T14:40:49.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.072+0000 7f987d7ca740 1 -- 172.21.15.90:0/4090614414 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ff2616a9f0 msgr2=0x55ff2617eb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:49.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.072+0000 7f987d7ca740 1 --2- 172.21.15.90:0/4090614414 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ff2616a9f0 0x55ff2617eb40 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f9860006e10 tx=0x7f9860009110 comp rx=0 tx=0).stop 2024-04-02T14:40:49.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.072+0000 7f987d7ca740 1 -- 172.21.15.90:0/4090614414 shutdown_connections 2024-04-02T14:40:49.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.072+0000 7f987d7ca740 1 --2- 172.21.15.90:0/4090614414 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ff26173080 0x55ff26170860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:49.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.072+0000 7f987d7ca740 1 --2- 172.21.15.90:0/4090614414 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ff2616a9f0 0x55ff2617eb40 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:49.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.072+0000 7f987d7ca740 1 --2- 172.21.15.90:0/4090614414 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ff26169d10 0x55ff261728c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:49.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.072+0000 7f987d7ca740 1 -- 172.21.15.90:0/4090614414 >> 172.21.15.90:0/4090614414 conn(0x55ff26130a50 msgr2=0x55ff26170150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:40:49.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.073+0000 7f987d7ca740 1 -- 172.21.15.90:0/4090614414 shutdown_connections 2024-04-02T14:40:49.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.073+0000 7f987d7ca740 1 -- 172.21.15.90:0/4090614414 wait complete. 2024-04-02T14:40:49.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.073+0000 7f987d7ca740 1 -- start start 2024-04-02T14:40:49.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.073+0000 7f987d7ca740 1 Processor -- start 2024-04-02T14:40:49.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.074+0000 7f987d7ca740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ff26169d10 0x55ff2617b060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:49.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.074+0000 7f987d7ca740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ff2616a9f0 0x55ff2617b5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:49.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.074+0000 7f987d7ca740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ff26173080 0x55ff26176010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:49.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.074+0000 7f987d7ca740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f98741b52c0 con 0x55ff2616a9f0 2024-04-02T14:40:49.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.074+0000 7f987d7ca740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f98741b4bf0 con 0x55ff26173080 2024-04-02T14:40:49.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.074+0000 7f987d7ca740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f98741b1e30 con 0x55ff26169d10 2024-04-02T14:40:49.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.074+0000 7f987d7ca740 10 client.0 _ll_register_callbacks cb 0x7f986c009c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:40:49.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.074+0000 7f9878a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ff26169d10 0x55ff2617b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:49.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.075+0000 7f9878a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ff26169d10 0x55ff2617b060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:49890/0 (socket says 172.21.15.90:49890) 2024-04-02T14:40:49.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.075+0000 7f9878a00640 1 -- 172.21.15.90:0/2622666943 learned_addr learned my addr 172.21.15.90:0/2622666943 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:40:49.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.075+0000 7f9878a00640 1 --2- 172.21.15.90:0/2622666943 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ff26173080 0x55ff26176010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:49.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.075+0000 7f9873400640 1 --2- 172.21.15.90:0/2622666943 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ff2616a9f0 0x55ff2617b5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:49.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.075+0000 7f9878a00640 1 -- 172.21.15.90:0/2622666943 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ff26173080 msgr2=0x55ff26176010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:40:49.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.075+0000 7f9878a00640 1 --2- 172.21.15.90:0/2622666943 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ff26173080 0x55ff26176010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:49.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.075+0000 7f9878a00640 1 -- 172.21.15.90:0/2622666943 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ff2616a9f0 msgr2=0x55ff2617b5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:40:49.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.075+0000 7f9878a00640 1 --2- 172.21.15.90:0/2622666943 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ff2616a9f0 0x55ff2617b5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:49.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.075+0000 7f9878a00640 1 -- 172.21.15.90:0/2622666943 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ff261767e0 con 0x55ff26169d10 2024-04-02T14:40:49.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.076+0000 7f9878a00640 1 --2- 172.21.15.90:0/2622666943 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ff26169d10 0x55ff2617b060 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f986c005d70 tx=0x7f986c011640 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1d20c77131891ad7 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:40:49.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.076+0000 7f985d600640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:40:49.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.076+0000 7f985d600640 1 -- 172.21.15.90:0/2622666943 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f986c003d10 con 0x55ff26169d10 2024-04-02T14:40:49.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.076+0000 7f985d600640 1 -- 172.21.15.90:0/2622666943 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f986c004060 con 0x55ff26169d10 2024-04-02T14:40:49.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.076+0000 7f987d7ca740 1 -- 172.21.15.90:0/2622666943 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f986001f030 con 0x55ff26169d10 2024-04-02T14:40:49.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.076+0000 7f985d600640 1 -- 172.21.15.90:0/2622666943 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f986c042050 con 0x55ff26169d10 2024-04-02T14:40:49.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.076+0000 7f987d7ca740 10 client.27073 fetch_fsmap learned FSMap version 27 2024-04-02T14:40:49.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.076+0000 7f987d7ca740 1 -- 172.21.15.90:0/2622666943 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55ff261ea200 con 0x55ff26169d10 2024-04-02T14:40:49.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.077+0000 7f985d600640 1 -- 172.21.15.90:0/2622666943 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f986001f030 con 0x55ff26169d10 2024-04-02T14:40:49.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.077+0000 7f985d600640 1 -- 172.21.15.90:0/2622666943 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f986c03c020 con 0x55ff26169d10 2024-04-02T14:40:49.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.077+0000 7f987d7ca740 10 client.27073 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:40:49.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.077+0000 7f987d7ca740 10 client.27073 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-04-02T14:40:49.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.077+0000 7f987d7ca740 20 client.27073 populate_metadata read hostname 'smithi090' 2024-04-02T14:40:49.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.077+0000 7f987d7ca740 1 --2- 172.21.15.90:0/2622666943 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x55ff261ea860 0x55ff261ecc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:49.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.077+0000 7f987d7ca740 4 client.27073 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-02T14:40:49.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.077+0000 7f987d7ca740 1 -- 172.21.15.90:0/2622666943 --> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55ff260dbc30 con 0x55ff261ea860 2024-04-02T14:40:49.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.079+0000 7f9873400640 1 --2- 172.21.15.90:0/2622666943 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x55ff261ea860 0x55ff261ecc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:49.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.079+0000 7f9873400640 1 --2- 172.21.15.90:0/2622666943 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x55ff261ea860 0x55ff261ecc50 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=c0822ca5c27cee4b server_cookie=ca80cd2cf23cb637 in_seq=0 out_seq=0 2024-04-02T14:40:49.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.079+0000 7f985d600640 10 client.27073 ms_handle_connect on v2:172.21.15.92:6836/1270635939 2024-04-02T14:40:49.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.080+0000 7f985d600640 1 -- 172.21.15.90:0/2622666943 <== mds.3 v2:172.21.15.92:6836/1270635939 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55ff260dbc30 con 0x55ff261ea860 2024-04-02T14:40:49.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.080+0000 7f985d600640 10 client.27073 handle_command_reply: tid=0 2024-04-02T14:40:49.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.080+0000 7f987d7ca740 1 client.27073 shutdown 2024-04-02T14:40:49.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.080+0000 7f987d7ca740 1 -- 172.21.15.90:0/2622666943 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ff26169d10 msgr2=0x55ff2617b060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:49.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.080+0000 7f987d7ca740 1 --2- 172.21.15.90:0/2622666943 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ff26169d10 0x55ff2617b060 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f986c005d70 tx=0x7f986c011640 comp rx=0 tx=0).stop 2024-04-02T14:40:49.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.080+0000 7f987d7ca740 1 -- 172.21.15.90:0/2622666943 shutdown_connections 2024-04-02T14:40:49.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.080+0000 7f987d7ca740 1 --2- 172.21.15.90:0/2622666943 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x55ff261ea860 0x55ff261ecc50 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:49.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.080+0000 7f987d7ca740 1 --2- 172.21.15.90:0/2622666943 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ff26173080 0x55ff26176010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:49.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.080+0000 7f987d7ca740 1 --2- 172.21.15.90:0/2622666943 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ff2616a9f0 0x55ff2617b5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:49.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.080+0000 7f987d7ca740 1 --2- 172.21.15.90:0/2622666943 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ff26169d10 0x55ff2617b060 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:49.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.080+0000 7f987d7ca740 1 -- 172.21.15.90:0/2622666943 >> 172.21.15.90:0/2622666943 conn(0x55ff26130a50 msgr2=0x55ff261638a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:40:49.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.080+0000 7f985d600640 0 client.27073 ms_handle_reset on v2:172.21.15.92:6836/1270635939 2024-04-02T14:40:49.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.080+0000 7f987d7ca740 1 -- 172.21.15.90:0/2622666943 shutdown_connections 2024-04-02T14:40:49.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.081+0000 7f987d7ca740 1 -- 172.21.15.90:0/2622666943 wait complete. 2024-04-02T14:40:49.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.081+0000 7f987d7ca740 20 client.27073 trim_cache size 0 max 16384 2024-04-02T14:40:49.083 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-02T14:40:49.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.082+0000 7f987aa00640 1 -- 172.21.15.90:0/3842915585 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f98580a3de0 msgr2=0x7f98580a62a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:49.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.082+0000 7f987aa00640 1 --2- 172.21.15.90:0/3842915585 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f98580a3de0 0x7f98580a62a0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x55ff2616af40 tx=0x7f986402c1d0 comp rx=0 tx=0).stop 2024-04-02T14:40:49.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.082+0000 7f987aa00640 1 -- 172.21.15.90:0/3842915585 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9874100b30 msgr2=0x7f98741a46d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:49.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.082+0000 7f987aa00640 1 --2- 172.21.15.90:0/3842915585 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9874100b30 0x7f98741a46d0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f9864000c00 tx=0x7f98640043b0 comp rx=0 tx=0).stop 2024-04-02T14:40:49.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.082+0000 7f987aa00640 1 -- 172.21.15.90:0/3842915585 shutdown_connections 2024-04-02T14:40:49.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.082+0000 7f987aa00640 1 --2- 172.21.15.90:0/3842915585 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f98580a3de0 0x7f98580a62a0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:49.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.082+0000 7f987aa00640 1 --2- 172.21.15.90:0/3842915585 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9874105f30 0x7f987419e890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:49.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.083+0000 7f987aa00640 1 --2- 172.21.15.90:0/3842915585 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9874101450 0x7f98741a4c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:49.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.083+0000 7f987aa00640 1 --2- 172.21.15.90:0/3842915585 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9874100b30 0x7f98741a46d0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:49.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.083+0000 7f987aa00640 1 -- 172.21.15.90:0/3842915585 >> 172.21.15.90:0/3842915585 conn(0x7f9874076d80 msgr2=0x7f98741038e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:40:49.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.083+0000 7f987aa00640 1 -- 172.21.15.90:0/3842915585 shutdown_connections 2024-04-02T14:40:49.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.083+0000 7f987aa00640 1 -- 172.21.15.90:0/3842915585 wait complete. 2024-04-02T14:40:49.106 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-02T14:40:49.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.369+0000 7f726a800640 1 -- 172.21.15.90:0/2322616568 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7264108310 msgr2=0x7f72641086f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:49.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.369+0000 7f726a800640 1 --2- 172.21.15.90:0/2322616568 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7264108310 0x7f72641086f0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f7254009900 tx=0x7f725402bc40 comp rx=0 tx=0).stop 2024-04-02T14:40:49.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.369+0000 7f726a800640 1 -- 172.21.15.90:0/2322616568 shutdown_connections 2024-04-02T14:40:49.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.369+0000 7f726a800640 1 --2- 172.21.15.90:0/2322616568 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f726410d3e0 0x7f7264114fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:49.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.369+0000 7f726a800640 1 --2- 172.21.15.90:0/2322616568 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7264108cc0 0x7f726410cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:49.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.369+0000 7f726a800640 1 --2- 172.21.15.90:0/2322616568 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7264108310 0x7f72641086f0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:49.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.369+0000 7f726a800640 1 -- 172.21.15.90:0/2322616568 >> 172.21.15.90:0/2322616568 conn(0x7f7264076040 msgr2=0x7f7264078460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:40:49.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.369+0000 7f726a800640 1 -- 172.21.15.90:0/2322616568 shutdown_connections 2024-04-02T14:40:49.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.370+0000 7f726a800640 1 -- 172.21.15.90:0/2322616568 wait complete. 2024-04-02T14:40:49.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.371+0000 7f726a800640 1 Processor -- start 2024-04-02T14:40:49.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.371+0000 7f726a800640 1 -- start start 2024-04-02T14:40:49.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.371+0000 7f726a800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7264108310 0x7f72641a46d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:49.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.371+0000 7f726a800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f726410d3e0 0x7f72641a4c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:49.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.371+0000 7f726a800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f726419e880 0x7f726419ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:49.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.371+0000 7f726a800640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f7264118560 con 0x7f726419e880 2024-04-02T14:40:49.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.371+0000 7f726a800640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f72641183e0 con 0x7f7264108310 2024-04-02T14:40:49.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.371+0000 7f726a800640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f72641186e0 con 0x7f726410d3e0 2024-04-02T14:40:49.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.372+0000 7f7263400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7264108310 0x7f72641a46d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:49.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.372+0000 7f7262a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f726410d3e0 0x7f72641a4c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:49.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.372+0000 7f7263400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7264108310 0x7f72641a46d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:34198/0 (socket says 172.21.15.90:34198) 2024-04-02T14:40:49.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.372+0000 7f7263400640 1 -- 172.21.15.90:0/3576699231 learned_addr learned my addr 172.21.15.90:0/3576699231 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:40:49.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.372+0000 7f7263e00640 1 --2- 172.21.15.90:0/3576699231 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f726419e880 0x7f726419ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:49.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.372+0000 7f7263400640 1 -- 172.21.15.90:0/3576699231 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f726410d3e0 msgr2=0x7f72641a4c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:49.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.372+0000 7f7263400640 1 --2- 172.21.15.90:0/3576699231 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f726410d3e0 0x7f72641a4c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:49.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.372+0000 7f7263400640 1 -- 172.21.15.90:0/3576699231 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f726419e880 msgr2=0x7f726419ed30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:49.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.372+0000 7f7263400640 1 --2- 172.21.15.90:0/3576699231 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f726419e880 0x7f726419ed30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:49.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.372+0000 7f7263400640 1 -- 172.21.15.90:0/3576699231 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f726419f3a0 con 0x7f7264108310 2024-04-02T14:40:49.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.372+0000 7f7263e00640 1 --2- 172.21.15.90:0/3576699231 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f726419e880 0x7f726419ed30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:40:49.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.373+0000 7f7263400640 1 --2- 172.21.15.90:0/3576699231 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7264108310 0x7f72641a46d0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f7254005ab0 tx=0x7f7254004360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:40:49.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.373+0000 7f725be00640 1 -- 172.21.15.90:0/3576699231 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7254041070 con 0x7f7264108310 2024-04-02T14:40:49.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.373+0000 7f725be00640 1 -- 172.21.15.90:0/3576699231 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f725402cca0 con 0x7f7264108310 2024-04-02T14:40:49.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.373+0000 7f725be00640 1 -- 172.21.15.90:0/3576699231 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7254003d90 con 0x7f7264108310 2024-04-02T14:40:49.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.373+0000 7f726a800640 1 -- 172.21.15.90:0/3576699231 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f726419f630 con 0x7f7264108310 2024-04-02T14:40:49.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.373+0000 7f726a800640 1 -- 172.21.15.90:0/3576699231 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72641ab7d0 con 0x7f7264108310 2024-04-02T14:40:49.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.379+0000 7f726d578740 1 Processor -- start 2024-04-02T14:40:49.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.379+0000 7f726d578740 1 -- start start 2024-04-02T14:40:49.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.380+0000 7f725be00640 1 -- 172.21.15.90:0/3576699231 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f7254044070 con 0x7f7264108310 2024-04-02T14:40:49.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.380+0000 7f726d578740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55881de559f0 0x55881de54740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:49.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.380+0000 7f726d578740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55881de54d10 0x55881de55190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:49.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.380+0000 7f726d578740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55881de5e080 0x55881de60470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:49.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.380+0000 7f726d578740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55881dd95400 con 0x55881de559f0 2024-04-02T14:40:49.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.380+0000 7f726d578740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55881dd79f10 con 0x55881de5e080 2024-04-02T14:40:49.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.380+0000 7f726d578740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55881ddb96f0 con 0x55881de54d10 2024-04-02T14:40:49.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.381+0000 7f7262a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55881de5e080 0x55881de60470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:49.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.381+0000 7f7262a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55881de5e080 0x55881de60470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:34202/0 (socket says 172.21.15.90:34202) 2024-04-02T14:40:49.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.381+0000 7f7262a00640 1 -- 172.21.15.90:0/2640913300 learned_addr learned my addr 172.21.15.90:0/2640913300 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:40:49.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.381+0000 7f7263400640 1 --2- 172.21.15.90:0/2640913300 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55881de54d10 0x55881de55190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:49.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.381+0000 7f7263e00640 1 --2- 172.21.15.90:0/2640913300 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55881de559f0 0x55881de54740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:49.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.381+0000 7f7262a00640 1 -- 172.21.15.90:0/2640913300 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55881de54d10 msgr2=0x55881de55190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:40:49.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.381+0000 7f7262a00640 1 --2- 172.21.15.90:0/2640913300 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55881de54d10 0x55881de55190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:49.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.381+0000 7f7262a00640 1 -- 172.21.15.90:0/2640913300 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55881de559f0 msgr2=0x55881de54740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:40:49.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.382+0000 7f7262a00640 1 --2- 172.21.15.90:0/2640913300 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55881de559f0 0x55881de54740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:49.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.382+0000 7f7262a00640 1 -- 172.21.15.90:0/2640913300 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55881dacd3f0 con 0x55881de5e080 2024-04-02T14:40:49.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.382+0000 7f7263e00640 1 --2- 172.21.15.90:0/2640913300 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55881de559f0 0x55881de54740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:40:49.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.382+0000 7f725be00640 1 --2- 172.21.15.90:0/3576699231 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f72480a3d20 0x7f72480a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:49.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.382+0000 7f725be00640 1 -- 172.21.15.90:0/3576699231 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f72540f21d0 con 0x7f7264108310 2024-04-02T14:40:49.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.383+0000 7f7263e00640 1 --2- 172.21.15.90:0/3576699231 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f72480a3d20 0x7f72480a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:49.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.383+0000 7f7262a00640 1 --2- 172.21.15.90:0/2640913300 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55881de5e080 0x55881de60470 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f72500099d0 tx=0x7f7250009e90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9180dbeeffd0131 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:40:49.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.383+0000 7f7258c00640 1 -- 172.21.15.90:0/2640913300 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7250017070 con 0x55881de5e080 2024-04-02T14:40:49.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.384+0000 7f7258c00640 1 -- 172.21.15.90:0/2640913300 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7250013470 con 0x55881de5e080 2024-04-02T14:40:49.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.384+0000 7f7258c00640 1 -- 172.21.15.90:0/2640913300 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7250012610 con 0x55881de5e080 2024-04-02T14:40:49.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.387+0000 7f726d578740 1 -- 172.21.15.90:0/2640913300 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55881de5e080 msgr2=0x55881de60470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:49.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.387+0000 7f726d578740 1 --2- 172.21.15.90:0/2640913300 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55881de5e080 0x55881de60470 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f72500099d0 tx=0x7f7250009e90 comp rx=0 tx=0).stop 2024-04-02T14:40:49.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.386+0000 7f7263e00640 1 --2- 172.21.15.90:0/3576699231 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f72480a3d20 0x7f72480a61e0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f725c005fd0 tx=0x7f725c009500 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:40:49.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.387+0000 7f726d578740 1 -- 172.21.15.90:0/2640913300 shutdown_connections 2024-04-02T14:40:49.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.387+0000 7f726d578740 1 --2- 172.21.15.90:0/2640913300 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55881de5e080 0x55881de60470 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:49.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.387+0000 7f726d578740 1 --2- 172.21.15.90:0/2640913300 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55881de54d10 0x55881de55190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:49.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.387+0000 7f726d578740 1 --2- 172.21.15.90:0/2640913300 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55881de559f0 0x55881de54740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:49.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.387+0000 7f726d578740 1 -- 172.21.15.90:0/2640913300 >> 172.21.15.90:0/2640913300 conn(0x55881de1ba50 msgr2=0x55881de4f760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:40:49.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.388+0000 7f726d578740 1 -- 172.21.15.90:0/2640913300 shutdown_connections 2024-04-02T14:40:49.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.388+0000 7f726d578740 1 -- 172.21.15.90:0/2640913300 wait complete. 2024-04-02T14:40:49.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.388+0000 7f726d578740 1 -- start start 2024-04-02T14:40:49.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.389+0000 7f726d578740 1 Processor -- start 2024-04-02T14:40:49.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.389+0000 7f726d578740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55881de54d10 0x55881de5b030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:49.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.389+0000 7f726d578740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55881de559f0 0x55881de5b570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:49.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.389+0000 7f726d578740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55881de5e080 0x55881de5bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:49.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.390+0000 7f726d578740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55881ddb56a0 con 0x55881de5e080 2024-04-02T14:40:49.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.390+0000 7f726d578740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55881dd61980 con 0x55881de559f0 2024-04-02T14:40:49.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.390+0000 7f726d578740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55881dd74cb0 con 0x55881de54d10 2024-04-02T14:40:49.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.390+0000 7f726d578740 10 client.0 _ll_register_callbacks cb 0x55881dd746a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:40:49.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.390+0000 7f7263e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55881de5e080 0x55881de5bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:49.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.391+0000 7f7263e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55881de5e080 0x55881de5bab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41576/0 (socket says 172.21.15.90:41576) 2024-04-02T14:40:49.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.391+0000 7f7263e00640 1 -- 172.21.15.90:0/1436306822 learned_addr learned my addr 172.21.15.90:0/1436306822 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:40:49.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.391+0000 7f7263400640 1 --2- 172.21.15.90:0/1436306822 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55881de54d10 0x55881de5b030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:49.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.391+0000 7f7262a00640 1 --2- 172.21.15.90:0/1436306822 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55881de559f0 0x55881de5b570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:49.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.391+0000 7f7263e00640 1 -- 172.21.15.90:0/1436306822 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55881de54d10 msgr2=0x55881de5b030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:40:49.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.391+0000 7f7263e00640 1 --2- 172.21.15.90:0/1436306822 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55881de54d10 0x55881de5b030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:49.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.391+0000 7f7263e00640 1 -- 172.21.15.90:0/1436306822 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55881de559f0 msgr2=0x55881de5b570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:40:49.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.391+0000 7f7263e00640 1 --2- 172.21.15.90:0/1436306822 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55881de559f0 0x55881de5b570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:49.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.391+0000 7f7263e00640 1 -- 172.21.15.90:0/1436306822 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55881db977d0 con 0x55881de5e080 2024-04-02T14:40:49.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.392+0000 7f7263e00640 1 --2- 172.21.15.90:0/1436306822 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55881de5e080 0x55881de5bab0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f725c010f60 tx=0x7f725c004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f2da340fdbd2e13 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:40:49.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.392+0000 7f724d600640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-02T14:40:49.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.393+0000 7f724d600640 1 -- 172.21.15.90:0/1436306822 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f725c011e70 con 0x55881de5e080 2024-04-02T14:40:49.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.393+0000 7f724d600640 1 -- 172.21.15.90:0/1436306822 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f725c01c600 con 0x55881de5e080 2024-04-02T14:40:49.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.393+0000 7f726d578740 1 -- 172.21.15.90:0/1436306822 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7250013470 con 0x55881de5e080 2024-04-02T14:40:49.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.394+0000 7f724d600640 1 -- 172.21.15.90:0/1436306822 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f725c01c800 con 0x55881de5e080 2024-04-02T14:40:49.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.394+0000 7f726d578740 10 client.16593 fetch_fsmap learned FSMap version 27 2024-04-02T14:40:49.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.394+0000 7f726d578740 1 -- 172.21.15.90:0/1436306822 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55881de6cf50 con 0x55881de5e080 2024-04-02T14:40:49.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.395+0000 7f724d600640 1 -- 172.21.15.90:0/1436306822 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7250013470 con 0x55881de5e080 2024-04-02T14:40:49.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.395+0000 7f724d600640 1 -- 172.21.15.90:0/1436306822 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f725c026070 con 0x55881de5e080 2024-04-02T14:40:49.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.395+0000 7f726d578740 10 client.16593 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:40:49.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.395+0000 7f726d578740 10 client.16593 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-04-02T14:40:49.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.395+0000 7f726d578740 20 client.16593 populate_metadata read hostname 'smithi090' 2024-04-02T14:40:49.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.395+0000 7f726d578740 1 --2- 172.21.15.90:0/1436306822 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x55881de6e500 0x55881de70970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:49.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.395+0000 7f726d578740 4 client.16593 mds_command: new command op to 24547 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:40:49.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.395+0000 7f726d578740 1 -- 172.21.15.90:0/1436306822 --> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55881dd79040 con 0x55881de6e500 2024-04-02T14:40:49.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.396+0000 7f7263400640 1 --2- 172.21.15.90:0/1436306822 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x55881de6e500 0x55881de70970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:49.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.398+0000 7f724d600640 10 client.16593 ms_handle_connect on v2:172.21.15.177:6838/664895663 2024-04-02T14:40:49.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.400+0000 7f7263400640 1 --2- 172.21.15.90:0/1436306822 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x55881de6e500 0x55881de70970 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=39d520eac774838d server_cookie=e0a54cc006122811 in_seq=0 out_seq=0 2024-04-02T14:40:49.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.401+0000 7f724d600640 1 -- 172.21.15.90:0/1436306822 <== mds.2 v2:172.21.15.177:6838/664895663 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55881dd79040 con 0x55881de6e500 2024-04-02T14:40:49.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.401+0000 7f724d600640 10 client.16593 handle_command_reply: tid=0 2024-04-02T14:40:49.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.401+0000 7f726d578740 1 client.16593 shutdown 2024-04-02T14:40:49.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.402+0000 7f726d578740 1 -- 172.21.15.90:0/1436306822 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55881de5e080 msgr2=0x55881de5bab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:49.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.402+0000 7f726d578740 1 --2- 172.21.15.90:0/1436306822 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55881de5e080 0x55881de5bab0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f725c010f60 tx=0x7f725c004ab0 comp rx=0 tx=0).stop 2024-04-02T14:40:49.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.402+0000 7f726d578740 1 -- 172.21.15.90:0/1436306822 shutdown_connections 2024-04-02T14:40:49.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.402+0000 7f726d578740 1 --2- 172.21.15.90:0/1436306822 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x55881de6e500 0x55881de70970 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:49.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.402+0000 7f726d578740 1 --2- 172.21.15.90:0/1436306822 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55881de5e080 0x55881de5bab0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:49.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.402+0000 7f726d578740 1 --2- 172.21.15.90:0/1436306822 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55881de559f0 0x55881de5b570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:49.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.402+0000 7f726d578740 1 --2- 172.21.15.90:0/1436306822 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55881de54d10 0x55881de5b030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:49.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.402+0000 7f726d578740 1 -- 172.21.15.90:0/1436306822 >> 172.21.15.90:0/1436306822 conn(0x55881de1ba50 msgr2=0x55881de514b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:40:49.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.402+0000 7f724d600640 0 client.16593 ms_handle_reset on v2:172.21.15.177:6838/664895663 2024-04-02T14:40:49.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.402+0000 7f726d578740 1 -- 172.21.15.90:0/1436306822 shutdown_connections 2024-04-02T14:40:49.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.402+0000 7f726d578740 1 -- 172.21.15.90:0/1436306822 wait complete. 2024-04-02T14:40:49.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.403+0000 7f726d578740 20 client.16593 trim_cache size 0 max 16384 2024-04-02T14:40:49.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.452+0000 7f726d578740 1 Processor -- start 2024-04-02T14:40:49.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.452+0000 7f726d578740 1 -- start start 2024-04-02T14:40:49.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.452+0000 7f726d578740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55881de54d10 0x55881de5d760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:49.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.452+0000 7f726d578740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55881de559f0 0x55881de69af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:49.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.452+0000 7f726d578740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55881de5e080 0x55881de5a010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:49.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.452+0000 7f726d578740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f72641b3ed0 con 0x55881de5e080 2024-04-02T14:40:49.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.452+0000 7f726d578740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f72641b5a10 con 0x55881de54d10 2024-04-02T14:40:49.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.452+0000 7f726d578740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f72641b45a0 con 0x55881de559f0 2024-04-02T14:40:49.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.453+0000 7f7262a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55881de559f0 0x55881de69af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:49.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.453+0000 7f7263400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55881de54d10 0x55881de5d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:49.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.453+0000 7f7262a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55881de559f0 0x55881de69af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:49942/0 (socket says 172.21.15.90:49942) 2024-04-02T14:40:49.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.453+0000 7f7262a00640 1 -- 172.21.15.90:0/3595219215 learned_addr learned my addr 172.21.15.90:0/3595219215 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:40:49.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.453+0000 7f7263e00640 1 --2- 172.21.15.90:0/3595219215 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55881de5e080 0x55881de5a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:49.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.454+0000 7f7262a00640 1 -- 172.21.15.90:0/3595219215 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55881de54d10 msgr2=0x55881de5d760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:40:49.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.454+0000 7f7262a00640 1 --2- 172.21.15.90:0/3595219215 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55881de54d10 0x55881de5d760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:49.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.454+0000 7f7262a00640 1 -- 172.21.15.90:0/3595219215 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55881de5e080 msgr2=0x55881de5a010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:40:49.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.454+0000 7f7262a00640 1 --2- 172.21.15.90:0/3595219215 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55881de5e080 0x55881de5a010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:49.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.454+0000 7f7262a00640 1 -- 172.21.15.90:0/3595219215 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55881de5a550 con 0x55881de559f0 2024-04-02T14:40:49.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.454+0000 7f7262a00640 1 --2- 172.21.15.90:0/3595219215 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55881de559f0 0x55881de69af0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f7250009530 tx=0x7f7250009560 comp rx=0 tx=0).ready entity=mon.1 client_cookie=34b0ad41a3bb52c7 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:40:49.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.455+0000 7f724fe00640 1 -- 172.21.15.90:0/3595219215 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7250017050 con 0x55881de559f0 2024-04-02T14:40:49.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.455+0000 7f724fe00640 1 -- 172.21.15.90:0/3595219215 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f725000ece0 con 0x55881de559f0 2024-04-02T14:40:49.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.455+0000 7f724fe00640 1 -- 172.21.15.90:0/3595219215 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7250013e40 con 0x55881de559f0 2024-04-02T14:40:49.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.456+0000 7f726d578740 1 -- 172.21.15.90:0/3595219215 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55881de559f0 msgr2=0x55881de69af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:49.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.456+0000 7f726d578740 1 --2- 172.21.15.90:0/3595219215 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55881de559f0 0x55881de69af0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f7250009530 tx=0x7f7250009560 comp rx=0 tx=0).stop 2024-04-02T14:40:49.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.458+0000 7f726d578740 1 -- 172.21.15.90:0/3595219215 shutdown_connections 2024-04-02T14:40:49.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.458+0000 7f726d578740 1 --2- 172.21.15.90:0/3595219215 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55881de5e080 0x55881de5a010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:49.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.458+0000 7f726d578740 1 --2- 172.21.15.90:0/3595219215 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55881de559f0 0x55881de69af0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:49.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.458+0000 7f726d578740 1 --2- 172.21.15.90:0/3595219215 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55881de54d10 0x55881de5d760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:49.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.458+0000 7f726d578740 1 -- 172.21.15.90:0/3595219215 >> 172.21.15.90:0/3595219215 conn(0x55881de1ba50 msgr2=0x55881de59900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:40:49.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.459+0000 7f726d578740 1 -- 172.21.15.90:0/3595219215 shutdown_connections 2024-04-02T14:40:49.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.459+0000 7f726d578740 1 -- 172.21.15.90:0/3595219215 wait complete. 2024-04-02T14:40:49.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.459+0000 7f726d578740 1 -- start start 2024-04-02T14:40:49.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.460+0000 7f726d578740 1 Processor -- start 2024-04-02T14:40:49.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.460+0000 7f726d578740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55881de54d10 0x55881de5e460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:49.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.460+0000 7f726d578740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55881de559f0 0x55881de5e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:49.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.460+0000 7f726d578740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55881de5e080 0x55881de5eee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:49.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.460+0000 7f726d578740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f72641b5340 con 0x55881de5e080 2024-04-02T14:40:49.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.460+0000 7f726d578740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f72641b4c70 con 0x55881de54d10 2024-04-02T14:40:49.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.460+0000 7f726d578740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f72641b1eb0 con 0x55881de559f0 2024-04-02T14:40:49.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.460+0000 7f726d578740 10 client.0 _ll_register_callbacks cb 0x7f725c010e40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:40:49.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.461+0000 7f7263400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55881de54d10 0x55881de5e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:49.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.461+0000 7f7262a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55881de559f0 0x55881de5e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:49.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.461+0000 7f7262a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55881de559f0 0x55881de5e9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:49958/0 (socket says 172.21.15.90:49958) 2024-04-02T14:40:49.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.461+0000 7f7262a00640 1 -- 172.21.15.90:0/1671209364 learned_addr learned my addr 172.21.15.90:0/1671209364 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:40:49.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.461+0000 7f7263e00640 1 --2- 172.21.15.90:0/1671209364 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55881de5e080 0x55881de5eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:49.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.461+0000 7f7262a00640 1 -- 172.21.15.90:0/1671209364 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55881de54d10 msgr2=0x55881de5e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:40:49.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.461+0000 7f7262a00640 1 --2- 172.21.15.90:0/1671209364 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55881de54d10 0x55881de5e460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:49.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.461+0000 7f7262a00640 1 -- 172.21.15.90:0/1671209364 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55881de5e080 msgr2=0x55881de5eee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:40:49.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.461+0000 7f7262a00640 1 --2- 172.21.15.90:0/1671209364 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55881de5e080 0x55881de5eee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:49.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.461+0000 7f7262a00640 1 -- 172.21.15.90:0/1671209364 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55881de65270 con 0x55881de559f0 2024-04-02T14:40:49.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.461+0000 7f7262a00640 1 --2- 172.21.15.90:0/1671209364 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55881de559f0 0x55881de5e9a0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f7250004810 tx=0x7f7250014f60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f4a4bb0f835ace0f server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:40:49.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.461+0000 7f724d600640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:40:49.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.462+0000 7f724d600640 1 -- 172.21.15.90:0/1671209364 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7250032060 con 0x55881de559f0 2024-04-02T14:40:49.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.462+0000 7f724d600640 1 -- 172.21.15.90:0/1671209364 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f725001c660 con 0x55881de559f0 2024-04-02T14:40:49.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.462+0000 7f726d578740 1 -- 172.21.15.90:0/1671209364 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f725000ece0 con 0x55881de559f0 2024-04-02T14:40:49.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.462+0000 7f724d600640 1 -- 172.21.15.90:0/1671209364 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f725001c880 con 0x55881de559f0 2024-04-02T14:40:49.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.462+0000 7f726d578740 10 client.27091 fetch_fsmap learned FSMap version 27 2024-04-02T14:40:49.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.462+0000 7f726d578740 1 -- 172.21.15.90:0/1671209364 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55881de66510 con 0x55881de559f0 2024-04-02T14:40:49.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.463+0000 7f724d600640 1 -- 172.21.15.90:0/1671209364 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f725000ece0 con 0x55881de559f0 2024-04-02T14:40:49.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.463+0000 7f724d600640 1 -- 172.21.15.90:0/1671209364 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f7250023070 con 0x55881de559f0 2024-04-02T14:40:49.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.463+0000 7f726d578740 10 client.27091 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:40:49.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.463+0000 7f726d578740 10 client.27091 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-04-02T14:40:49.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.463+0000 7f726d578740 20 client.27091 populate_metadata read hostname 'smithi090' 2024-04-02T14:40:49.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.463+0000 7f726d578740 1 --2- 172.21.15.90:0/1671209364 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x55881ded4af0 0x55881ded6ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:49.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.463+0000 7f726d578740 4 client.27091 mds_command: new command op to 24547 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-02T14:40:49.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.463+0000 7f726d578740 1 -- 172.21.15.90:0/1671209364 --> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55881ddc6c30 con 0x55881ded4af0 2024-04-02T14:40:49.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.463+0000 7f7263400640 1 --2- 172.21.15.90:0/1671209364 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x55881ded4af0 0x55881ded6ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:49.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.464+0000 7f7263400640 1 --2- 172.21.15.90:0/1671209364 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x55881ded4af0 0x55881ded6ee0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=2c2e4d89771076dd server_cookie=569f9aaab64995e2 in_seq=0 out_seq=0 2024-04-02T14:40:49.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.464+0000 7f724d600640 10 client.27091 ms_handle_connect on v2:172.21.15.177:6838/664895663 2024-04-02T14:40:49.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.464+0000 7f724d600640 1 -- 172.21.15.90:0/1671209364 <== mds.2 v2:172.21.15.177:6838/664895663 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55881ddc6c30 con 0x55881ded4af0 2024-04-02T14:40:49.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.464+0000 7f724d600640 10 client.27091 handle_command_reply: tid=0 2024-04-02T14:40:49.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.464+0000 7f726d578740 1 client.27091 shutdown 2024-04-02T14:40:49.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.465+0000 7f726d578740 1 -- 172.21.15.90:0/1671209364 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55881de559f0 msgr2=0x55881de5e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:49.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.465+0000 7f726d578740 1 --2- 172.21.15.90:0/1671209364 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55881de559f0 0x55881de5e9a0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f7250004810 tx=0x7f7250014f60 comp rx=0 tx=0).stop 2024-04-02T14:40:49.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.465+0000 7f726d578740 1 -- 172.21.15.90:0/1671209364 shutdown_connections 2024-04-02T14:40:49.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.465+0000 7f726d578740 1 --2- 172.21.15.90:0/1671209364 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x55881ded4af0 0x55881ded6ee0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:49.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.465+0000 7f726d578740 1 --2- 172.21.15.90:0/1671209364 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55881de5e080 0x55881de5eee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:49.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.465+0000 7f724d600640 0 client.27091 ms_handle_reset on v2:172.21.15.177:6838/664895663 2024-04-02T14:40:49.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.465+0000 7f726d578740 1 --2- 172.21.15.90:0/1671209364 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55881de559f0 0x55881de5e9a0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:49.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.465+0000 7f726d578740 1 --2- 172.21.15.90:0/1671209364 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55881de54d10 0x55881de5e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:49.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.465+0000 7f726d578740 1 -- 172.21.15.90:0/1671209364 >> 172.21.15.90:0/1671209364 conn(0x55881de1ba50 msgr2=0x55881de50a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:40:49.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.466+0000 7f726d578740 1 -- 172.21.15.90:0/1671209364 shutdown_connections 2024-04-02T14:40:49.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.466+0000 7f726d578740 1 -- 172.21.15.90:0/1671209364 wait complete. 2024-04-02T14:40:49.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.466+0000 7f726d578740 20 client.27091 trim_cache size 0 max 16384 2024-04-02T14:40:49.469 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-02T14:40:49.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.468+0000 7f726a800640 1 -- 172.21.15.90:0/3576699231 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f72480a3d20 msgr2=0x7f72480a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:49.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.468+0000 7f726a800640 1 --2- 172.21.15.90:0/3576699231 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f72480a3d20 0x7f72480a61e0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f725c005fd0 tx=0x7f725c009500 comp rx=0 tx=0).stop 2024-04-02T14:40:49.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.468+0000 7f726a800640 1 -- 172.21.15.90:0/3576699231 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7264108310 msgr2=0x7f72641a46d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:49.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.468+0000 7f726a800640 1 --2- 172.21.15.90:0/3576699231 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7264108310 0x7f72641a46d0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f7254005ab0 tx=0x7f7254004360 comp rx=0 tx=0).stop 2024-04-02T14:40:49.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.468+0000 7f726a800640 1 -- 172.21.15.90:0/3576699231 shutdown_connections 2024-04-02T14:40:49.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.468+0000 7f726a800640 1 --2- 172.21.15.90:0/3576699231 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f72480a3d20 0x7f72480a61e0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:49.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.468+0000 7f726a800640 1 --2- 172.21.15.90:0/3576699231 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f726419e880 0x7f726419ed30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:49.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.468+0000 7f726a800640 1 --2- 172.21.15.90:0/3576699231 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f726410d3e0 0x7f72641a4c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:49.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.468+0000 7f726a800640 1 --2- 172.21.15.90:0/3576699231 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7264108310 0x7f72641a46d0 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:49.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.468+0000 7f726a800640 1 -- 172.21.15.90:0/3576699231 >> 172.21.15.90:0/3576699231 conn(0x7f7264076040 msgr2=0x7f72640783b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:40:49.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.469+0000 7f726a800640 1 -- 172.21.15.90:0/3576699231 shutdown_connections 2024-04-02T14:40:49.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.469+0000 7f726a800640 1 -- 172.21.15.90:0/3576699231 wait complete. 2024-04-02T14:40:49.495 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-02T14:40:49.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.752+0000 7f8a82800640 1 -- 172.21.15.90:0/151108575 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8a7c112940 msgr2=0x7f8a7c114d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:49.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.752+0000 7f8a82800640 1 --2- 172.21.15.90:0/151108575 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8a7c112940 0x7f8a7c114d30 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f8a6c009930 tx=0x7f8a6c02bc30 comp rx=0 tx=0).stop 2024-04-02T14:40:49.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.752+0000 7f8a82800640 1 -- 172.21.15.90:0/151108575 shutdown_connections 2024-04-02T14:40:49.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.752+0000 7f8a82800640 1 --2- 172.21.15.90:0/151108575 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8a7c112940 0x7f8a7c114d30 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:49.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.752+0000 7f8a82800640 1 --2- 172.21.15.90:0/151108575 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8a7c076950 0x7f8a7c112400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:49.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.752+0000 7f8a82800640 1 --2- 172.21.15.90:0/151108575 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8a7c076030 0x7f8a7c076410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:49.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.752+0000 7f8a82800640 1 -- 172.21.15.90:0/151108575 >> 172.21.15.90:0/151108575 conn(0x7f8a7c0ff7b0 msgr2=0x7f8a7c101bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:40:49.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.752+0000 7f8a82800640 1 -- 172.21.15.90:0/151108575 shutdown_connections 2024-04-02T14:40:49.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.752+0000 7f8a82800640 1 -- 172.21.15.90:0/151108575 wait complete. 2024-04-02T14:40:49.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.753+0000 7f8a82800640 1 Processor -- start 2024-04-02T14:40:49.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.753+0000 7f8a82800640 1 -- start start 2024-04-02T14:40:49.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.754+0000 7f8a82800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8a7c076030 0x7f8a7c1a46d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:49.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.754+0000 7f8a82800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8a7c076950 0x7f8a7c1a4c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:49.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.754+0000 7f8a82800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8a7c112940 0x7f8a7c19e840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:49.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.754+0000 7f8a82800640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f8a7c118460 con 0x7f8a7c076030 2024-04-02T14:40:49.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.754+0000 7f8a82800640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f8a7c1182e0 con 0x7f8a7c076950 2024-04-02T14:40:49.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.754+0000 7f8a82800640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f8a7c1185e0 con 0x7f8a7c112940 2024-04-02T14:40:49.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.754+0000 7f8a7aa00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8a7c076950 0x7f8a7c1a4c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:49.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.754+0000 7f8a7aa00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8a7c076950 0x7f8a7c1a4c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:34268/0 (socket says 172.21.15.90:34268) 2024-04-02T14:40:49.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.754+0000 7f8a7aa00640 1 -- 172.21.15.90:0/4124957103 learned_addr learned my addr 172.21.15.90:0/4124957103 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:40:49.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.754+0000 7f8a7be00640 1 --2- 172.21.15.90:0/4124957103 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8a7c112940 0x7f8a7c19e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:49.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.755+0000 7f8a7aa00640 1 -- 172.21.15.90:0/4124957103 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8a7c112940 msgr2=0x7f8a7c19e840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:49.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.755+0000 7f8a7aa00640 1 --2- 172.21.15.90:0/4124957103 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8a7c112940 0x7f8a7c19e840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:49.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.755+0000 7f8a7aa00640 1 -- 172.21.15.90:0/4124957103 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8a7c076030 msgr2=0x7f8a7c1a46d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T14:40:49.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.755+0000 7f8a7b400640 1 --2- 172.21.15.90:0/4124957103 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8a7c076030 0x7f8a7c1a46d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:49.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.756+0000 7f8a7aa00640 1 --2- 172.21.15.90:0/4124957103 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8a7c076030 0x7f8a7c1a46d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:49.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.756+0000 7f8a7aa00640 1 -- 172.21.15.90:0/4124957103 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a7c19f010 con 0x7f8a7c076950 2024-04-02T14:40:49.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.756+0000 7f8a7be00640 1 --2- 172.21.15.90:0/4124957103 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8a7c112940 0x7f8a7c19e840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:40:49.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.756+0000 7f8a7b400640 1 --2- 172.21.15.90:0/4124957103 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8a7c076030 0x7f8a7c1a46d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:40:49.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.756+0000 7f8a7aa00640 1 --2- 172.21.15.90:0/4124957103 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8a7c076950 0x7f8a7c1a4c10 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f8a6800e9f0 tx=0x7f8a6800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:40:49.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.756+0000 7f8a73e00640 1 -- 172.21.15.90:0/4124957103 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a6800cd80 con 0x7f8a7c076950 2024-04-02T14:40:49.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.756+0000 7f8a73e00640 1 -- 172.21.15.90:0/4124957103 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8a68010910 con 0x7f8a7c076950 2024-04-02T14:40:49.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.756+0000 7f8a73e00640 1 -- 172.21.15.90:0/4124957103 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a68018990 con 0x7f8a7c076950 2024-04-02T14:40:49.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.757+0000 7f8a82800640 1 -- 172.21.15.90:0/4124957103 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a7c19f300 con 0x7f8a7c076950 2024-04-02T14:40:49.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.757+0000 7f8a82800640 1 -- 172.21.15.90:0/4124957103 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a7c1ab520 con 0x7f8a7c076950 2024-04-02T14:40:49.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.761+0000 7f8a85587740 1 Processor -- start 2024-04-02T14:40:49.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.762+0000 7f8a85587740 1 -- start start 2024-04-02T14:40:49.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.762+0000 7f8a73e00640 1 -- 172.21.15.90:0/4124957103 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f8a68010ac0 con 0x7f8a7c076950 2024-04-02T14:40:49.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.762+0000 7f8a85587740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a940bb49f0 0x55a940bb3740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:49.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.762+0000 7f8a85587740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a940bb3d10 0x55a940bb4190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:49.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.762+0000 7f8a85587740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a940bbd080 0x55a940bbf470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:49.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.762+0000 7f8a85587740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55a940af4400 con 0x55a940bb3d10 2024-04-02T14:40:49.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.762+0000 7f8a85587740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55a940ad8f10 con 0x55a940bbd080 2024-04-02T14:40:49.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.762+0000 7f8a85587740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55a940b186f0 con 0x55a940bb49f0 2024-04-02T14:40:49.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.763+0000 7f8a7be00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a940bb49f0 0x55a940bb3740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:49.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.763+0000 7f8a7be00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a940bb49f0 0x55a940bb3740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:49970/0 (socket says 172.21.15.90:49970) 2024-04-02T14:40:49.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.763+0000 7f8a7be00640 1 -- 172.21.15.90:0/715486814 learned_addr learned my addr 172.21.15.90:0/715486814 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:40:49.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.763+0000 7f8a7aa00640 1 --2- 172.21.15.90:0/715486814 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a940bbd080 0x55a940bbf470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:49.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.763+0000 7f8a7be00640 1 -- 172.21.15.90:0/715486814 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a940bbd080 msgr2=0x55a940bbf470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:40:49.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.763+0000 7f8a7be00640 1 --2- 172.21.15.90:0/715486814 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a940bbd080 0x55a940bbf470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:49.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.763+0000 7f8a7be00640 1 -- 172.21.15.90:0/715486814 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a940bb3d10 msgr2=0x55a940bb4190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:40:49.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.763+0000 7f8a7be00640 1 --2- 172.21.15.90:0/715486814 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a940bb3d10 0x55a940bb4190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:49.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.763+0000 7f8a7be00640 1 -- 172.21.15.90:0/715486814 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a94082c3f0 con 0x55a940bb49f0 2024-04-02T14:40:49.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.763+0000 7f8a7be00640 1 --2- 172.21.15.90:0/715486814 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a940bb49f0 0x55a940bb3740 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f8a6c036040 tx=0x7f8a6c009630 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8ee0a378409806c9 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:40:49.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.764+0000 7f8a70c00640 1 -- 172.21.15.90:0/715486814 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a6c045070 con 0x55a940bb49f0 2024-04-02T14:40:49.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.764+0000 7f8a70c00640 1 -- 172.21.15.90:0/715486814 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8a6c02c880 con 0x55a940bb49f0 2024-04-02T14:40:49.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.764+0000 7f8a70c00640 1 -- 172.21.15.90:0/715486814 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a6c003bb0 con 0x55a940bb49f0 2024-04-02T14:40:49.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.764+0000 7f8a73e00640 1 --2- 172.21.15.90:0/4124957103 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f8a600a3ea0 0x7f8a600a6360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:49.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.764+0000 7f8a73e00640 1 -- 172.21.15.90:0/4124957103 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f8a680d0a90 con 0x7f8a7c076950 2024-04-02T14:40:49.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.764+0000 7f8a7b400640 1 --2- 172.21.15.90:0/4124957103 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f8a600a3ea0 0x7f8a600a6360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:49.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.765+0000 7f8a85587740 1 -- 172.21.15.90:0/715486814 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a940bb49f0 msgr2=0x55a940bb3740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:49.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.765+0000 7f8a85587740 1 --2- 172.21.15.90:0/715486814 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a940bb49f0 0x55a940bb3740 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f8a6c036040 tx=0x7f8a6c009630 comp rx=0 tx=0).stop 2024-04-02T14:40:49.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.765+0000 7f8a7b400640 1 --2- 172.21.15.90:0/4124957103 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f8a600a3ea0 0x7f8a600a6360 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f8a74005c40 tx=0x7f8a74005bd0 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:40:49.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.765+0000 7f8a85587740 1 -- 172.21.15.90:0/715486814 shutdown_connections 2024-04-02T14:40:49.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.765+0000 7f8a85587740 1 --2- 172.21.15.90:0/715486814 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a940bbd080 0x55a940bbf470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:49.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.765+0000 7f8a85587740 1 --2- 172.21.15.90:0/715486814 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a940bb3d10 0x55a940bb4190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:49.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.766+0000 7f8a85587740 1 --2- 172.21.15.90:0/715486814 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a940bb49f0 0x55a940bb3740 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:49.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.766+0000 7f8a85587740 1 -- 172.21.15.90:0/715486814 >> 172.21.15.90:0/715486814 conn(0x55a940b7aa50 msgr2=0x55a940bae760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:40:49.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.766+0000 7f8a85587740 1 -- 172.21.15.90:0/715486814 shutdown_connections 2024-04-02T14:40:49.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.766+0000 7f8a85587740 1 -- 172.21.15.90:0/715486814 wait complete. 2024-04-02T14:40:49.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.766+0000 7f8a85587740 1 -- start start 2024-04-02T14:40:49.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.767+0000 7f8a85587740 1 Processor -- start 2024-04-02T14:40:49.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.768+0000 7f8a85587740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a940bb3d10 0x55a940bba0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:49.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.768+0000 7f8a85587740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a940bb49f0 0x55a940bba620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:49.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.768+0000 7f8a85587740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a940bbd080 0x55a940bbab60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:49.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.768+0000 7f8a85587740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55a940b146a0 con 0x55a940bb3d10 2024-04-02T14:40:49.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.768+0000 7f8a85587740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55a940ac0980 con 0x55a940bb49f0 2024-04-02T14:40:49.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.768+0000 7f8a85587740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55a940ad3cb0 con 0x55a940bbd080 2024-04-02T14:40:49.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.768+0000 7f8a7aa00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a940bb49f0 0x55a940bba620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:49.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.768+0000 7f8a7be00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a940bbd080 0x55a940bbab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:49.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.768+0000 7f8a7aa00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a940bb49f0 0x55a940bba620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:34294/0 (socket says 172.21.15.90:34294) 2024-04-02T14:40:49.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.768+0000 7f8a7aa00640 1 -- 172.21.15.90:0/3904812280 learned_addr learned my addr 172.21.15.90:0/3904812280 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:40:49.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.768+0000 7f8a7b400640 1 --2- 172.21.15.90:0/3904812280 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a940bb3d10 0x55a940bba0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:49.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.768+0000 7f8a85587740 10 client.0 _ll_register_callbacks cb 0x55a940ad36a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:40:49.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.769+0000 7f8a7be00640 1 -- 172.21.15.90:0/3904812280 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a940bb49f0 msgr2=0x55a940bba620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:40:49.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.769+0000 7f8a7be00640 1 --2- 172.21.15.90:0/3904812280 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a940bb49f0 0x55a940bba620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:49.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.769+0000 7f8a7be00640 1 -- 172.21.15.90:0/3904812280 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a940bb3d10 msgr2=0x55a940bba0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:40:49.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.769+0000 7f8a7be00640 1 --2- 172.21.15.90:0/3904812280 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a940bb3d10 0x55a940bba0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:49.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.769+0000 7f8a7be00640 1 -- 172.21.15.90:0/3904812280 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a9408f67d0 con 0x55a940bbd080 2024-04-02T14:40:49.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.769+0000 7f8a7be00640 1 --2- 172.21.15.90:0/3904812280 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a940bbd080 0x55a940bbab60 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f8a6c0096c0 tx=0x7f8a6c040a70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=20e4adfb8722a23d server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:40:49.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.769+0000 7f8a65600640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:40:49.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.769+0000 7f8a65600640 1 -- 172.21.15.90:0/3904812280 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a6c04e070 con 0x55a940bbd080 2024-04-02T14:40:49.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.769+0000 7f8a65600640 1 -- 172.21.15.90:0/3904812280 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f8a6c04a860 con 0x55a940bbd080 2024-04-02T14:40:49.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.769+0000 7f8a85587740 1 -- 172.21.15.90:0/3904812280 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8a6c02c880 con 0x55a940bbd080 2024-04-02T14:40:49.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.770+0000 7f8a65600640 1 -- 172.21.15.90:0/3904812280 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f8a6c045070 con 0x55a940bbd080 2024-04-02T14:40:49.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.770+0000 7f8a85587740 10 client.27103 fetch_fsmap learned FSMap version 27 2024-04-02T14:40:49.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.770+0000 7f8a85587740 1 -- 172.21.15.90:0/3904812280 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55a940bcbd30 con 0x55a940bbd080 2024-04-02T14:40:49.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.771+0000 7f8a65600640 1 -- 172.21.15.90:0/3904812280 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8a6c02c880 con 0x55a940bbd080 2024-04-02T14:40:49.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.772+0000 7f8a65600640 1 -- 172.21.15.90:0/3904812280 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f8a6c05bb60 con 0x55a940bbd080 2024-04-02T14:40:49.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.772+0000 7f8a85587740 10 client.27103 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:40:49.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.772+0000 7f8a85587740 10 client.27103 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2024-04-02T14:40:49.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.772+0000 7f8a85587740 20 client.27103 populate_metadata read hostname 'smithi090' 2024-04-02T14:40:49.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.772+0000 7f8a85587740 1 --2- 172.21.15.90:0/3904812280 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x55a940bcd080 0x55a940bcf470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:49.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.772+0000 7f8a85587740 4 client.27103 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:40:49.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.772+0000 7f8a85587740 1 -- 172.21.15.90:0/3904812280 --> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a940ad8040 con 0x55a940bcd080 2024-04-02T14:40:49.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.773+0000 7f8a7b400640 1 --2- 172.21.15.90:0/3904812280 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x55a940bcd080 0x55a940bcf470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:49.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.773+0000 7f8a7b400640 1 --2- 172.21.15.90:0/3904812280 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x55a940bcd080 0x55a940bcf470 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=767d83bdfab410b5 server_cookie=1b4044c443c4cbe8 in_seq=0 out_seq=0 2024-04-02T14:40:49.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.773+0000 7f8a65600640 10 client.27103 ms_handle_connect on v2:172.21.15.90:6838/1498915870 2024-04-02T14:40:49.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.774+0000 7f8a65600640 1 -- 172.21.15.90:0/3904812280 <== mds.1 v2:172.21.15.90:6838/1498915870 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a940ad8040 con 0x55a940bcd080 2024-04-02T14:40:49.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.774+0000 7f8a65600640 10 client.27103 handle_command_reply: tid=0 2024-04-02T14:40:49.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.774+0000 7f8a85587740 1 client.27103 shutdown 2024-04-02T14:40:49.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.775+0000 7f8a85587740 1 -- 172.21.15.90:0/3904812280 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a940bbd080 msgr2=0x55a940bbab60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:49.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.775+0000 7f8a85587740 1 --2- 172.21.15.90:0/3904812280 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a940bbd080 0x55a940bbab60 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f8a6c0096c0 tx=0x7f8a6c040a70 comp rx=0 tx=0).stop 2024-04-02T14:40:49.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.775+0000 7f8a85587740 1 -- 172.21.15.90:0/3904812280 shutdown_connections 2024-04-02T14:40:49.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.775+0000 7f8a85587740 1 --2- 172.21.15.90:0/3904812280 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x55a940bcd080 0x55a940bcf470 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:49.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.775+0000 7f8a85587740 1 --2- 172.21.15.90:0/3904812280 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a940bbd080 0x55a940bbab60 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:49.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.775+0000 7f8a85587740 1 --2- 172.21.15.90:0/3904812280 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a940bb49f0 0x55a940bba620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:49.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.775+0000 7f8a65600640 0 client.27103 ms_handle_reset on v2:172.21.15.90:6838/1498915870 2024-04-02T14:40:49.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.775+0000 7f8a85587740 1 --2- 172.21.15.90:0/3904812280 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a940bb3d10 0x55a940bba0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:49.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.775+0000 7f8a85587740 1 -- 172.21.15.90:0/3904812280 >> 172.21.15.90:0/3904812280 conn(0x55a940b7aa50 msgr2=0x55a940bb04b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:40:49.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.775+0000 7f8a85587740 1 -- 172.21.15.90:0/3904812280 shutdown_connections 2024-04-02T14:40:49.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.775+0000 7f8a85587740 1 -- 172.21.15.90:0/3904812280 wait complete. 2024-04-02T14:40:49.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.775+0000 7f8a85587740 20 client.27103 trim_cache size 0 max 16384 2024-04-02T14:40:49.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.799+0000 7f8a85587740 1 Processor -- start 2024-04-02T14:40:49.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.800+0000 7f8a85587740 1 -- start start 2024-04-02T14:40:49.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.800+0000 7f8a85587740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a940bb3d10 0x55a940bc8860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:49.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.800+0000 7f8a85587740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a940bb49f0 0x55a940bb0250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:49.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.800+0000 7f8a85587740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a940bbd080 0x55a940bbe470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:49.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.800+0000 7f8a85587740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f8a7c1b3e60 con 0x55a940bbd080 2024-04-02T14:40:49.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.800+0000 7f8a85587740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f8a7c1b59a0 con 0x55a940bb49f0 2024-04-02T14:40:49.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.800+0000 7f8a85587740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f8a7c1b4530 con 0x55a940bb3d10 2024-04-02T14:40:49.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.801+0000 7f8a7b400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a940bb3d10 0x55a940bc8860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:49.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.801+0000 7f8a7be00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a940bbd080 0x55a940bbe470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:49.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.801+0000 7f8a7b400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a940bb3d10 0x55a940bc8860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:49992/0 (socket says 172.21.15.90:49992) 2024-04-02T14:40:49.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.801+0000 7f8a7b400640 1 -- 172.21.15.90:0/3546107635 learned_addr learned my addr 172.21.15.90:0/3546107635 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:40:49.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.801+0000 7f8a7aa00640 1 --2- 172.21.15.90:0/3546107635 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a940bb49f0 0x55a940bb0250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:49.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.801+0000 7f8a7aa00640 1 -- 172.21.15.90:0/3546107635 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a940bb3d10 msgr2=0x55a940bc8860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:40:49.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.801+0000 7f8a7aa00640 1 --2- 172.21.15.90:0/3546107635 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a940bb3d10 0x55a940bc8860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:49.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.801+0000 7f8a7aa00640 1 -- 172.21.15.90:0/3546107635 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a940bbd080 msgr2=0x55a940bbe470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:40:49.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.801+0000 7f8a7aa00640 1 --2- 172.21.15.90:0/3546107635 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a940bbd080 0x55a940bbe470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:49.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.801+0000 7f8a7aa00640 1 -- 172.21.15.90:0/3546107635 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a940bb2d40 con 0x55a940bb49f0 2024-04-02T14:40:49.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.802+0000 7f8a7b400640 1 --2- 172.21.15.90:0/3546107635 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a940bb3d10 0x55a940bc8860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:40:49.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.802+0000 7f8a7aa00640 1 --2- 172.21.15.90:0/3546107635 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a940bb49f0 0x55a940bb0250 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f8a68000c00 tx=0x7f8a68018420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f7df1117cc3e0954 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:40:49.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.802+0000 7f8a67e00640 1 -- 172.21.15.90:0/3546107635 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a6809c9e0 con 0x55a940bb49f0 2024-04-02T14:40:49.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.803+0000 7f8a67e00640 1 -- 172.21.15.90:0/3546107635 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8a680a0d90 con 0x55a940bb49f0 2024-04-02T14:40:49.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.803+0000 7f8a67e00640 1 -- 172.21.15.90:0/3546107635 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a680bbe00 con 0x55a940bb49f0 2024-04-02T14:40:49.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.803+0000 7f8a85587740 1 -- 172.21.15.90:0/3546107635 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a940bb49f0 msgr2=0x55a940bb0250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:49.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.804+0000 7f8a85587740 1 --2- 172.21.15.90:0/3546107635 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a940bb49f0 0x55a940bb0250 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f8a68000c00 tx=0x7f8a68018420 comp rx=0 tx=0).stop 2024-04-02T14:40:49.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.804+0000 7f8a85587740 1 -- 172.21.15.90:0/3546107635 shutdown_connections 2024-04-02T14:40:49.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.804+0000 7f8a85587740 1 --2- 172.21.15.90:0/3546107635 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a940bbd080 0x55a940bbe470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:49.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.804+0000 7f8a85587740 1 --2- 172.21.15.90:0/3546107635 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a940bb49f0 0x55a940bb0250 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:49.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.804+0000 7f8a85587740 1 --2- 172.21.15.90:0/3546107635 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a940bb3d10 0x55a940bc8860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:49.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.804+0000 7f8a85587740 1 -- 172.21.15.90:0/3546107635 >> 172.21.15.90:0/3546107635 conn(0x55a940b7aa50 msgr2=0x55a940bafb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:40:49.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.805+0000 7f8a85587740 1 -- 172.21.15.90:0/3546107635 shutdown_connections 2024-04-02T14:40:49.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.806+0000 7f8a85587740 1 -- 172.21.15.90:0/3546107635 wait complete. 2024-04-02T14:40:49.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.806+0000 7f8a85587740 1 -- start start 2024-04-02T14:40:49.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.807+0000 7f8a85587740 1 Processor -- start 2024-04-02T14:40:49.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.807+0000 7f8a85587740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a940bb3d10 0x55a940bd2600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:49.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.807+0000 7f8a85587740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a940bb49f0 0x55a940c2e240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:49.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.807+0000 7f8a85587740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a940bbd080 0x55a940c2e780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:49.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.807+0000 7f8a85587740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f8a7c1b52d0 con 0x55a940bb49f0 2024-04-02T14:40:49.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.807+0000 7f8a85587740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f8a7c1b4c00 con 0x55a940bb3d10 2024-04-02T14:40:49.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.807+0000 7f8a85587740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f8a7c1b1e40 con 0x55a940bbd080 2024-04-02T14:40:49.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.807+0000 7f8a85587740 10 client.0 _ll_register_callbacks cb 0x7f8a6c040680 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:40:49.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.807+0000 7f8a7be00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a940bbd080 0x55a940c2e780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:49.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.808+0000 7f8a7be00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a940bbd080 0x55a940c2e780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:50000/0 (socket says 172.21.15.90:50000) 2024-04-02T14:40:49.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.808+0000 7f8a7be00640 1 -- 172.21.15.90:0/128894111 learned_addr learned my addr 172.21.15.90:0/128894111 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:40:49.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.808+0000 7f8a7aa00640 1 --2- 172.21.15.90:0/128894111 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a940bb49f0 0x55a940c2e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:49.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.808+0000 7f8a7b400640 1 --2- 172.21.15.90:0/128894111 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a940bb3d10 0x55a940bd2600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:49.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.808+0000 7f8a7be00640 1 -- 172.21.15.90:0/128894111 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a940bb3d10 msgr2=0x55a940bd2600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:40:49.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.808+0000 7f8a7be00640 1 --2- 172.21.15.90:0/128894111 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a940bb3d10 0x55a940bd2600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:49.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.808+0000 7f8a7be00640 1 -- 172.21.15.90:0/128894111 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a940bb49f0 msgr2=0x55a940c2e240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:40:49.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.808+0000 7f8a7be00640 1 --2- 172.21.15.90:0/128894111 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a940bb49f0 0x55a940c2e240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:49.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.808+0000 7f8a7be00640 1 -- 172.21.15.90:0/128894111 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a940c2ed40 con 0x55a940bbd080 2024-04-02T14:40:49.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.808+0000 7f8a7aa00640 1 --2- 172.21.15.90:0/128894111 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a940bb49f0 0x55a940c2e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:40:49.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.810+0000 7f8a7be00640 1 --2- 172.21.15.90:0/128894111 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a940bbd080 0x55a940c2e780 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f8a6c005a90 tx=0x7f8a6c02c3d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e3e593045a9b470e server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:40:49.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.810+0000 7f8a65600640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:40:49.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.813+0000 7f8a65600640 1 -- 172.21.15.90:0/128894111 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a6c060030 con 0x55a940bbd080 2024-04-02T14:40:49.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.813+0000 7f8a65600640 1 -- 172.21.15.90:0/128894111 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f8a6c040cf0 con 0x55a940bbd080 2024-04-02T14:40:49.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.813+0000 7f8a85587740 1 -- 172.21.15.90:0/128894111 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8a680a0d90 con 0x55a940bbd080 2024-04-02T14:40:49.817 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.816+0000 7f8a65600640 1 -- 172.21.15.90:0/128894111 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f8a6c04e040 con 0x55a940bbd080 2024-04-02T14:40:49.817 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.816+0000 7f8a85587740 10 client.27109 fetch_fsmap learned FSMap version 27 2024-04-02T14:40:49.817 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.816+0000 7f8a85587740 1 -- 172.21.15.90:0/128894111 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55a940bd0790 con 0x55a940bbd080 2024-04-02T14:40:49.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.818+0000 7f8a65600640 1 -- 172.21.15.90:0/128894111 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8a680a0d90 con 0x55a940bbd080 2024-04-02T14:40:49.820 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.819+0000 7f8a65600640 1 -- 172.21.15.90:0/128894111 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f8a6c04ab40 con 0x55a940bbd080 2024-04-02T14:40:49.820 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.819+0000 7f8a85587740 10 client.27109 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:40:49.820 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.819+0000 7f8a85587740 10 client.27109 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2024-04-02T14:40:49.820 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.819+0000 7f8a85587740 20 client.27109 populate_metadata read hostname 'smithi090' 2024-04-02T14:40:49.820 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.819+0000 7f8a85587740 1 --2- 172.21.15.90:0/128894111 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x55a940c55700 0x55a940c57b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:49.820 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.820+0000 7f8a85587740 4 client.27109 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-02T14:40:49.820 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.820+0000 7f8a85587740 1 -- 172.21.15.90:0/128894111 --> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55a940b25c30 con 0x55a940c55700 2024-04-02T14:40:49.821 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.820+0000 7f8a7b400640 1 --2- 172.21.15.90:0/128894111 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x55a940c55700 0x55a940c57b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:49.822 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.821+0000 7f8a7b400640 1 --2- 172.21.15.90:0/128894111 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x55a940c55700 0x55a940c57b20 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=d44c2fb85a749f16 server_cookie=2518f53e7eb688eb in_seq=0 out_seq=0 2024-04-02T14:40:49.822 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.821+0000 7f8a65600640 10 client.27109 ms_handle_connect on v2:172.21.15.90:6838/1498915870 2024-04-02T14:40:49.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.823+0000 7f8a65600640 1 -- 172.21.15.90:0/128894111 <== mds.1 v2:172.21.15.90:6838/1498915870 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55a940b25c30 con 0x55a940c55700 2024-04-02T14:40:49.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.823+0000 7f8a65600640 10 client.27109 handle_command_reply: tid=0 2024-04-02T14:40:49.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.824+0000 7f8a85587740 1 client.27109 shutdown 2024-04-02T14:40:49.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.824+0000 7f8a85587740 1 -- 172.21.15.90:0/128894111 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a940bbd080 msgr2=0x55a940c2e780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:49.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.824+0000 7f8a85587740 1 --2- 172.21.15.90:0/128894111 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a940bbd080 0x55a940c2e780 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f8a6c005a90 tx=0x7f8a6c02c3d0 comp rx=0 tx=0).stop 2024-04-02T14:40:49.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.825+0000 7f8a85587740 1 -- 172.21.15.90:0/128894111 shutdown_connections 2024-04-02T14:40:49.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.825+0000 7f8a85587740 1 --2- 172.21.15.90:0/128894111 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x55a940c55700 0x55a940c57b20 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:49.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.825+0000 7f8a85587740 1 --2- 172.21.15.90:0/128894111 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a940bbd080 0x55a940c2e780 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:49.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.825+0000 7f8a85587740 1 --2- 172.21.15.90:0/128894111 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a940bb49f0 0x55a940c2e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:49.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.825+0000 7f8a85587740 1 --2- 172.21.15.90:0/128894111 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a940bb3d10 0x55a940bd2600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:49.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.825+0000 7f8a85587740 1 -- 172.21.15.90:0/128894111 >> 172.21.15.90:0/128894111 conn(0x55a940b7aa50 msgr2=0x55a940bc86e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:40:49.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.825+0000 7f8a65600640 0 client.27109 ms_handle_reset on v2:172.21.15.90:6838/1498915870 2024-04-02T14:40:49.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.828+0000 7f8a85587740 1 -- 172.21.15.90:0/128894111 shutdown_connections 2024-04-02T14:40:49.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.830+0000 7f8a85587740 1 -- 172.21.15.90:0/128894111 wait complete. 2024-04-02T14:40:49.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.830+0000 7f8a85587740 20 client.27109 trim_cache size 0 max 16384 2024-04-02T14:40:49.831 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-02T14:40:49.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.831+0000 7f8a82800640 1 -- 172.21.15.90:0/4124957103 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f8a600a3ea0 msgr2=0x7f8a600a6360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:49.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.831+0000 7f8a82800640 1 --2- 172.21.15.90:0/4124957103 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f8a600a3ea0 0x7f8a600a6360 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f8a74005c40 tx=0x7f8a74005bd0 comp rx=0 tx=0).stop 2024-04-02T14:40:49.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.831+0000 7f8a82800640 1 -- 172.21.15.90:0/4124957103 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8a7c076950 msgr2=0x7f8a7c1a4c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:49.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.831+0000 7f8a82800640 1 --2- 172.21.15.90:0/4124957103 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8a7c076950 0x7f8a7c1a4c10 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f8a6800e9f0 tx=0x7f8a6800eeb0 comp rx=0 tx=0).stop 2024-04-02T14:40:49.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.831+0000 7f8a82800640 1 -- 172.21.15.90:0/4124957103 shutdown_connections 2024-04-02T14:40:49.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.831+0000 7f8a82800640 1 --2- 172.21.15.90:0/4124957103 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f8a600a3ea0 0x7f8a600a6360 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:49.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.831+0000 7f8a82800640 1 --2- 172.21.15.90:0/4124957103 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8a7c112940 0x7f8a7c19e840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:49.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.831+0000 7f8a82800640 1 --2- 172.21.15.90:0/4124957103 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8a7c076950 0x7f8a7c1a4c10 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:49.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.831+0000 7f8a82800640 1 --2- 172.21.15.90:0/4124957103 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8a7c076030 0x7f8a7c1a46d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:49.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.831+0000 7f8a82800640 1 -- 172.21.15.90:0/4124957103 >> 172.21.15.90:0/4124957103 conn(0x7f8a7c0ff7b0 msgr2=0x7f8a7c1135b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:40:49.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.832+0000 7f8a82800640 1 -- 172.21.15.90:0/4124957103 shutdown_connections 2024-04-02T14:40:49.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:49.832+0000 7f8a82800640 1 -- 172.21.15.90:0/4124957103 wait complete. 2024-04-02T14:40:49.855 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:40:49 smithi090 ceph-mon[31674]: pgmap v181: 97 pgs: 97 active+clean; 3.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 8.3 MiB/s wr, 816 op/s 2024-04-02T14:40:49.858 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-02T14:40:49.859 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:40:49 smithi092 ceph-mon[36497]: pgmap v181: 97 pgs: 97 active+clean; 3.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 8.3 MiB/s wr, 816 op/s 2024-04-02T14:40:49.935 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:40:49 smithi177 ceph-mon[36431]: pgmap v181: 97 pgs: 97 active+clean; 3.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 8.3 MiB/s wr, 816 op/s 2024-04-02T14:40:50.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.124+0000 7fe7dcc00640 1 -- 172.21.15.90:0/371177874 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe7d8102440 msgr2=0x7fe7d810e950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:50.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.124+0000 7fe7dcc00640 1 --2- 172.21.15.90:0/371177874 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe7d8102440 0x7fe7d810e950 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fe7d0009cd0 tx=0x7fe7d002c0e0 comp rx=0 tx=0).stop 2024-04-02T14:40:50.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.124+0000 7fe7dcc00640 1 -- 172.21.15.90:0/371177874 shutdown_connections 2024-04-02T14:40:50.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.124+0000 7fe7dcc00640 1 --2- 172.21.15.90:0/371177874 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe7d8102440 0x7fe7d810e950 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:50.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.124+0000 7fe7dcc00640 1 --2- 172.21.15.90:0/371177874 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe7d8101a80 0x7fe7d8101f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:50.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.124+0000 7fe7dcc00640 1 --2- 172.21.15.90:0/371177874 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe7d8107a90 0x7fe7d8107e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:50.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.124+0000 7fe7dcc00640 1 -- 172.21.15.90:0/371177874 >> 172.21.15.90:0/371177874 conn(0x7fe7d80fd590 msgr2=0x7fe7d80ff9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:40:50.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.124+0000 7fe7dcc00640 1 -- 172.21.15.90:0/371177874 shutdown_connections 2024-04-02T14:40:50.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.125+0000 7fe7dcc00640 1 -- 172.21.15.90:0/371177874 wait complete. 2024-04-02T14:40:50.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.125+0000 7fe7dcc00640 1 Processor -- start 2024-04-02T14:40:50.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.126+0000 7fe7dcc00640 1 -- start start 2024-04-02T14:40:50.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.127+0000 7fe7dcc00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe7d8101a80 0x7fe7d81961d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:50.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.127+0000 7fe7dcc00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe7d8102440 0x7fe7d8196710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:50.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.127+0000 7fe7dcc00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe7d8107a90 0x7fe7d819aa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:50.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.127+0000 7fe7dcc00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fe7d810fc90 con 0x7fe7d8102440 2024-04-02T14:40:50.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.127+0000 7fe7dcc00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fe7d810fb10 con 0x7fe7d8107a90 2024-04-02T14:40:50.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.127+0000 7fe7dcc00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fe7d810fe10 con 0x7fe7d8101a80 2024-04-02T14:40:50.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.127+0000 7fe7d5400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe7d8102440 0x7fe7d8196710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:50.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.127+0000 7fe7d5400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe7d8102440 0x7fe7d8196710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41672/0 (socket says 172.21.15.90:41672) 2024-04-02T14:40:50.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.127+0000 7fe7d5400640 1 -- 172.21.15.90:0/1067219792 learned_addr learned my addr 172.21.15.90:0/1067219792 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:40:50.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.127+0000 7fe7d6800640 1 --2- 172.21.15.90:0/1067219792 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe7d8107a90 0x7fe7d819aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:50.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.128+0000 7fe7d5e00640 1 --2- 172.21.15.90:0/1067219792 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe7d8101a80 0x7fe7d81961d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:50.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.128+0000 7fe7d5400640 1 -- 172.21.15.90:0/1067219792 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe7d8101a80 msgr2=0x7fe7d81961d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:50.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.128+0000 7fe7d5400640 1 --2- 172.21.15.90:0/1067219792 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe7d8101a80 0x7fe7d81961d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:50.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.128+0000 7fe7d5400640 1 -- 172.21.15.90:0/1067219792 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe7d8107a90 msgr2=0x7fe7d819aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:50.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.128+0000 7fe7d5400640 1 --2- 172.21.15.90:0/1067219792 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe7d8107a90 0x7fe7d819aa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:50.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.128+0000 7fe7d5400640 1 -- 172.21.15.90:0/1067219792 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7d819b1c0 con 0x7fe7d8102440 2024-04-02T14:40:50.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.128+0000 7fe7d5e00640 1 --2- 172.21.15.90:0/1067219792 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe7d8101a80 0x7fe7d81961d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:40:50.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.129+0000 7fe7d5400640 1 --2- 172.21.15.90:0/1067219792 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe7d8102440 0x7fe7d8196710 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fe7c400d950 tx=0x7fe7c400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:40:50.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.129+0000 7fe7cea00640 1 -- 172.21.15.90:0/1067219792 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7c4017070 con 0x7fe7d8102440 2024-04-02T14:40:50.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.129+0000 7fe7cea00640 1 -- 172.21.15.90:0/1067219792 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe7c4009bb0 con 0x7fe7d8102440 2024-04-02T14:40:50.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.129+0000 7fe7cea00640 1 -- 172.21.15.90:0/1067219792 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7c400f600 con 0x7fe7d8102440 2024-04-02T14:40:50.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.129+0000 7fe7dcc00640 1 -- 172.21.15.90:0/1067219792 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe7d819b4b0 con 0x7fe7d8102440 2024-04-02T14:40:50.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.129+0000 7fe7dcc00640 1 -- 172.21.15.90:0/1067219792 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7d81a2d40 con 0x7fe7d8102440 2024-04-02T14:40:50.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.135+0000 7fe7df9ad740 1 Processor -- start 2024-04-02T14:40:50.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.135+0000 7fe7df9ad740 1 -- start start 2024-04-02T14:40:50.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.135+0000 7fe7df9ad740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5609da673200 0x5609da671f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:50.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.135+0000 7fe7df9ad740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5609da672520 0x5609da6729a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:50.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.135+0000 7fe7df9ad740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5609da67b890 0x5609da67dc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:50.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.135+0000 7fe7df9ad740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5609da5aa400 con 0x5609da67b890 2024-04-02T14:40:50.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.135+0000 7fe7df9ad740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x5609da58ef10 con 0x5609da672520 2024-04-02T14:40:50.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.135+0000 7fe7df9ad740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5609da5ce6f0 con 0x5609da673200 2024-04-02T14:40:50.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.135+0000 7fe7cea00640 1 -- 172.21.15.90:0/1067219792 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fe7c4010460 con 0x7fe7d8102440 2024-04-02T14:40:50.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.135+0000 7fe7d6800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5609da673200 0x5609da671f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:50.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.135+0000 7fe7d5e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5609da672520 0x5609da6729a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:50.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.135+0000 7fe7d5e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5609da672520 0x5609da6729a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:34342/0 (socket says 172.21.15.90:34342) 2024-04-02T14:40:50.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.135+0000 7fe7d5e00640 1 -- 172.21.15.90:0/2788267314 learned_addr learned my addr 172.21.15.90:0/2788267314 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:40:50.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.136+0000 7fe7d5e00640 1 -- 172.21.15.90:0/2788267314 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5609da673200 msgr2=0x5609da671f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:40:50.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.136+0000 7fe7d5e00640 1 --2- 172.21.15.90:0/2788267314 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5609da673200 0x5609da671f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:50.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.136+0000 7fe7d5e00640 1 -- 172.21.15.90:0/2788267314 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5609da67b890 msgr2=0x5609da67dc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:40:50.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.136+0000 7fe7d5e00640 1 --2- 172.21.15.90:0/2788267314 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5609da67b890 0x5609da67dc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:50.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.136+0000 7fe7d5e00640 1 -- 172.21.15.90:0/2788267314 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5609da2e23f0 con 0x5609da672520 2024-04-02T14:40:50.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.137+0000 7fe7d5e00640 1 --2- 172.21.15.90:0/2788267314 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5609da672520 0x5609da6729a0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7fe7c800dc40 tx=0x7fe7c800f730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d5fb64537f05269e server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:40:50.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.137+0000 7fe7c3000640 1 -- 172.21.15.90:0/2788267314 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7c8005020 con 0x5609da672520 2024-04-02T14:40:50.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.137+0000 7fe7c3000640 1 -- 172.21.15.90:0/2788267314 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe7c80047a0 con 0x5609da672520 2024-04-02T14:40:50.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.137+0000 7fe7c3000640 1 -- 172.21.15.90:0/2788267314 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7c80116a0 con 0x5609da672520 2024-04-02T14:40:50.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.138+0000 7fe7df9ad740 1 -- 172.21.15.90:0/2788267314 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5609da672520 msgr2=0x5609da6729a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:50.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.138+0000 7fe7df9ad740 1 --2- 172.21.15.90:0/2788267314 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5609da672520 0x5609da6729a0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7fe7c800dc40 tx=0x7fe7c800f730 comp rx=0 tx=0).stop 2024-04-02T14:40:50.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.139+0000 7fe7df9ad740 1 -- 172.21.15.90:0/2788267314 shutdown_connections 2024-04-02T14:40:50.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.139+0000 7fe7df9ad740 1 --2- 172.21.15.90:0/2788267314 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5609da67b890 0x5609da67dc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:50.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.139+0000 7fe7df9ad740 1 --2- 172.21.15.90:0/2788267314 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5609da672520 0x5609da6729a0 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:50.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.139+0000 7fe7df9ad740 1 --2- 172.21.15.90:0/2788267314 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5609da673200 0x5609da671f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:50.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.139+0000 7fe7df9ad740 1 -- 172.21.15.90:0/2788267314 >> 172.21.15.90:0/2788267314 conn(0x5609da630a50 msgr2=0x5609da66cf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:40:50.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.140+0000 7fe7cea00640 1 --2- 172.21.15.90:0/1067219792 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fe7bc0a3eb0 0x7fe7bc0a6370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:50.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.140+0000 7fe7df9ad740 1 -- 172.21.15.90:0/2788267314 shutdown_connections 2024-04-02T14:40:50.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.140+0000 7fe7cea00640 1 -- 172.21.15.90:0/1067219792 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7fe7c40d0300 con 0x7fe7d8102440 2024-04-02T14:40:50.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.140+0000 7fe7d5e00640 1 --2- 172.21.15.90:0/1067219792 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fe7bc0a3eb0 0x7fe7bc0a6370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:50.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.141+0000 7fe7df9ad740 1 -- 172.21.15.90:0/2788267314 wait complete. 2024-04-02T14:40:50.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.141+0000 7fe7df9ad740 1 -- start start 2024-04-02T14:40:50.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.142+0000 7fe7d5e00640 1 --2- 172.21.15.90:0/1067219792 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fe7bc0a3eb0 0x7fe7bc0a6370 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fe7c8005e00 tx=0x7fe7c800d780 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:40:50.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.142+0000 7fe7df9ad740 1 Processor -- start 2024-04-02T14:40:50.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.143+0000 7fe7df9ad740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5609da672520 0x5609da6788f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:50.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.143+0000 7fe7df9ad740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5609da673200 0x5609da678e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:50.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.143+0000 7fe7df9ad740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5609da67b890 0x5609da679370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:50.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.143+0000 7fe7df9ad740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5609da5ca6a0 con 0x5609da67b890 2024-04-02T14:40:50.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.143+0000 7fe7df9ad740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x5609da576980 con 0x5609da672520 2024-04-02T14:40:50.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.143+0000 7fe7df9ad740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5609da589cb0 con 0x5609da673200 2024-04-02T14:40:50.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.143+0000 7fe7df9ad740 10 client.0 _ll_register_callbacks cb 0x5609da5896a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:40:50.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.143+0000 7fe7d5e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5609da672520 0x5609da6788f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:50.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.143+0000 7fe7d5e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5609da672520 0x5609da6788f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:34354/0 (socket says 172.21.15.90:34354) 2024-04-02T14:40:50.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.143+0000 7fe7d5e00640 1 -- 172.21.15.90:0/1775475172 learned_addr learned my addr 172.21.15.90:0/1775475172 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:40:50.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.143+0000 7fe7d5400640 1 --2- 172.21.15.90:0/1775475172 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5609da673200 0x5609da678e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:50.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.144+0000 7fe7d5e00640 1 -- 172.21.15.90:0/1775475172 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5609da673200 msgr2=0x5609da678e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:40:50.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.144+0000 7fe7d6800640 1 --2- 172.21.15.90:0/1775475172 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5609da67b890 0x5609da679370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:50.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.144+0000 7fe7d5e00640 1 --2- 172.21.15.90:0/1775475172 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5609da673200 0x5609da678e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:50.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.144+0000 7fe7d5e00640 1 -- 172.21.15.90:0/1775475172 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5609da67b890 msgr2=0x5609da679370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:40:50.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.144+0000 7fe7d5e00640 1 --2- 172.21.15.90:0/1775475172 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5609da67b890 0x5609da679370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:50.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.144+0000 7fe7d5e00640 1 -- 172.21.15.90:0/1775475172 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5609da3ac7d0 con 0x5609da672520 2024-04-02T14:40:50.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.144+0000 7fe7d5400640 1 --2- 172.21.15.90:0/1775475172 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5609da673200 0x5609da678e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:40:50.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.144+0000 7fe7d5e00640 1 --2- 172.21.15.90:0/1775475172 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5609da672520 0x5609da6788f0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fe7c8005c70 tx=0x7fe7c800f730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=36c7359d4605d384 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:40:50.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.144+0000 7fe7b7400640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:40:50.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.144+0000 7fe7b7400640 1 -- 172.21.15.90:0/1775475172 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7c8004310 con 0x5609da672520 2024-04-02T14:40:50.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.145+0000 7fe7b7400640 1 -- 172.21.15.90:0/1775475172 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7fe7c80044b0 con 0x5609da672520 2024-04-02T14:40:50.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.145+0000 7fe7df9ad740 1 -- 172.21.15.90:0/1775475172 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe7c80047a0 con 0x5609da672520 2024-04-02T14:40:50.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.145+0000 7fe7b7400640 1 -- 172.21.15.90:0/1775475172 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fe7c80035f0 con 0x5609da672520 2024-04-02T14:40:50.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.145+0000 7fe7df9ad740 10 client.27245 fetch_fsmap learned FSMap version 27 2024-04-02T14:40:50.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.145+0000 7fe7df9ad740 1 -- 172.21.15.90:0/1775475172 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5609da68a7a0 con 0x5609da672520 2024-04-02T14:40:50.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.146+0000 7fe7b7400640 1 -- 172.21.15.90:0/1775475172 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe7c80047a0 con 0x5609da672520 2024-04-02T14:40:50.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.147+0000 7fe7b7400640 1 -- 172.21.15.90:0/1775475172 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fe7c8031070 con 0x5609da672520 2024-04-02T14:40:50.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.147+0000 7fe7df9ad740 10 client.27245 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:40:50.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.147+0000 7fe7df9ad740 10 client.27245 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:40:50.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.147+0000 7fe7df9ad740 20 client.27245 populate_metadata read hostname 'smithi090' 2024-04-02T14:40:50.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.147+0000 7fe7df9ad740 1 --2- 172.21.15.90:0/1775475172 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5609da68bb50 0x5609da68df40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:50.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.147+0000 7fe7df9ad740 4 client.27245 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:40:50.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.147+0000 7fe7df9ad740 1 -- 172.21.15.90:0/1775475172 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5609da58e040 con 0x5609da68bb50 2024-04-02T14:40:50.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.147+0000 7fe7d5400640 1 --2- 172.21.15.90:0/1775475172 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5609da68bb50 0x5609da68df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:50.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.148+0000 7fe7d5400640 1 --2- 172.21.15.90:0/1775475172 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5609da68bb50 0x5609da68df40 crc :-1 s=READY pgs=168 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5591872b343db372 server_cookie=193239c47c149bcc in_seq=0 out_seq=0 2024-04-02T14:40:50.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.148+0000 7fe7b7400640 10 client.27245 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:40:50.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.149+0000 7fe7b7400640 1 -- 172.21.15.90:0/1775475172 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5609da58e040 con 0x5609da68bb50 2024-04-02T14:40:50.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.149+0000 7fe7b7400640 10 client.27245 handle_command_reply: tid=0 2024-04-02T14:40:50.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.149+0000 7fe7df9ad740 1 client.27245 shutdown 2024-04-02T14:40:50.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.150+0000 7fe7df9ad740 1 -- 172.21.15.90:0/1775475172 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5609da672520 msgr2=0x5609da6788f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:50.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.150+0000 7fe7df9ad740 1 --2- 172.21.15.90:0/1775475172 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5609da672520 0x5609da6788f0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fe7c8005c70 tx=0x7fe7c800f730 comp rx=0 tx=0).stop 2024-04-02T14:40:50.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.150+0000 7fe7df9ad740 1 -- 172.21.15.90:0/1775475172 shutdown_connections 2024-04-02T14:40:50.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.150+0000 7fe7df9ad740 1 --2- 172.21.15.90:0/1775475172 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5609da68bb50 0x5609da68df40 crc :-1 s=READY pgs=168 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:50.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.150+0000 7fe7df9ad740 1 --2- 172.21.15.90:0/1775475172 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5609da67b890 0x5609da679370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:50.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.150+0000 7fe7df9ad740 1 --2- 172.21.15.90:0/1775475172 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5609da673200 0x5609da678e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:50.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.150+0000 7fe7df9ad740 1 --2- 172.21.15.90:0/1775475172 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5609da672520 0x5609da6788f0 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:50.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.150+0000 7fe7df9ad740 1 -- 172.21.15.90:0/1775475172 >> 172.21.15.90:0/1775475172 conn(0x5609da630a50 msgr2=0x5609da66ecc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:40:50.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.150+0000 7fe7b7400640 0 client.27245 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:40:50.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.150+0000 7fe7df9ad740 1 -- 172.21.15.90:0/1775475172 shutdown_connections 2024-04-02T14:40:50.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.150+0000 7fe7df9ad740 1 -- 172.21.15.90:0/1775475172 wait complete. 2024-04-02T14:40:50.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.151+0000 7fe7df9ad740 20 client.27245 trim_cache size 0 max 16384 2024-04-02T14:40:50.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.175+0000 7fe7df9ad740 1 Processor -- start 2024-04-02T14:40:50.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.175+0000 7fe7df9ad740 1 -- start start 2024-04-02T14:40:50.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.175+0000 7fe7df9ad740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5609da672520 0x5609da67b020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:50.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.176+0000 7fe7df9ad740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5609da673200 0x5609da687300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:50.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.176+0000 7fe7df9ad740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5609da67b890 0x5609da663720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:50.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.176+0000 7fe7df9ad740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fe7d81ab780 con 0x5609da67b890 2024-04-02T14:40:50.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.176+0000 7fe7df9ad740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fe7d81ad2c0 con 0x5609da673200 2024-04-02T14:40:50.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.176+0000 7fe7df9ad740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fe7d81abe50 con 0x5609da672520 2024-04-02T14:40:50.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.176+0000 7fe7d5400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5609da673200 0x5609da687300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:50.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.176+0000 7fe7d5400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5609da673200 0x5609da687300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:34364/0 (socket says 172.21.15.90:34364) 2024-04-02T14:40:50.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.176+0000 7fe7d5400640 1 -- 172.21.15.90:0/116527208 learned_addr learned my addr 172.21.15.90:0/116527208 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:40:50.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.176+0000 7fe7d5e00640 1 --2- 172.21.15.90:0/116527208 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5609da672520 0x5609da67b020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:50.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.176+0000 7fe7d6800640 1 --2- 172.21.15.90:0/116527208 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5609da67b890 0x5609da663720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:50.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.176+0000 7fe7d5400640 1 -- 172.21.15.90:0/116527208 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5609da672520 msgr2=0x5609da67b020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:40:50.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.176+0000 7fe7d5400640 1 --2- 172.21.15.90:0/116527208 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5609da672520 0x5609da67b020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:50.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.176+0000 7fe7d5400640 1 -- 172.21.15.90:0/116527208 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5609da67b890 msgr2=0x5609da663720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:40:50.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.176+0000 7fe7d5400640 1 --2- 172.21.15.90:0/116527208 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5609da67b890 0x5609da663720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:50.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.176+0000 7fe7d5400640 1 -- 172.21.15.90:0/116527208 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5609da68e4d0 con 0x5609da673200 2024-04-02T14:40:50.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.176+0000 7fe7d5e00640 1 --2- 172.21.15.90:0/116527208 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5609da672520 0x5609da67b020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:40:50.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.177+0000 7fe7d5400640 1 --2- 172.21.15.90:0/116527208 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5609da673200 0x5609da687300 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fe7c409bf50 tx=0x7fe7c4010ce0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=be9bc52c4eef9512 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:40:50.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.177+0000 7fe7c2600640 1 -- 172.21.15.90:0/116527208 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7c4020040 con 0x5609da673200 2024-04-02T14:40:50.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.177+0000 7fe7c2600640 1 -- 172.21.15.90:0/116527208 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe7c4015070 con 0x5609da673200 2024-04-02T14:40:50.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.177+0000 7fe7c2600640 1 -- 172.21.15.90:0/116527208 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7c40988c0 con 0x5609da673200 2024-04-02T14:40:50.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.178+0000 7fe7df9ad740 1 -- 172.21.15.90:0/116527208 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5609da673200 msgr2=0x5609da687300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:50.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.178+0000 7fe7df9ad740 1 --2- 172.21.15.90:0/116527208 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5609da673200 0x5609da687300 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fe7c409bf50 tx=0x7fe7c4010ce0 comp rx=0 tx=0).stop 2024-04-02T14:40:50.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.178+0000 7fe7df9ad740 1 -- 172.21.15.90:0/116527208 shutdown_connections 2024-04-02T14:40:50.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.178+0000 7fe7df9ad740 1 --2- 172.21.15.90:0/116527208 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5609da67b890 0x5609da663720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:50.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.178+0000 7fe7df9ad740 1 --2- 172.21.15.90:0/116527208 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5609da673200 0x5609da687300 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:50.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.178+0000 7fe7df9ad740 1 --2- 172.21.15.90:0/116527208 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5609da672520 0x5609da67b020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:50.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.178+0000 7fe7df9ad740 1 -- 172.21.15.90:0/116527208 >> 172.21.15.90:0/116527208 conn(0x5609da630a50 msgr2=0x5609da663010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:40:50.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.179+0000 7fe7df9ad740 1 -- 172.21.15.90:0/116527208 shutdown_connections 2024-04-02T14:40:50.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.179+0000 7fe7df9ad740 1 -- 172.21.15.90:0/116527208 wait complete. 2024-04-02T14:40:50.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.179+0000 7fe7df9ad740 1 -- start start 2024-04-02T14:40:50.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.180+0000 7fe7df9ad740 1 Processor -- start 2024-04-02T14:40:50.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.180+0000 7fe7df9ad740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5609da672520 0x5609da683cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:50.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.180+0000 7fe7df9ad740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5609da673200 0x5609da675e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:50.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.180+0000 7fe7df9ad740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5609da67b890 0x5609da6764c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:50.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.180+0000 7fe7df9ad740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fe7d81acbf0 con 0x5609da672520 2024-04-02T14:40:50.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.180+0000 7fe7df9ad740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fe7d81ac520 con 0x5609da67b890 2024-04-02T14:40:50.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.180+0000 7fe7df9ad740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fe7d81a9760 con 0x5609da673200 2024-04-02T14:40:50.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.180+0000 7fe7df9ad740 10 client.0 _ll_register_callbacks cb 0x7fe7c800dac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:40:50.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.181+0000 7fe7d5400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5609da673200 0x5609da675e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:50.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.181+0000 7fe7d5400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5609da673200 0x5609da675e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:50060/0 (socket says 172.21.15.90:50060) 2024-04-02T14:40:50.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.181+0000 7fe7d5400640 1 -- 172.21.15.90:0/3698951394 learned_addr learned my addr 172.21.15.90:0/3698951394 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:40:50.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.181+0000 7fe7d5e00640 1 --2- 172.21.15.90:0/3698951394 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5609da672520 0x5609da683cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:50.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.181+0000 7fe7d6800640 1 --2- 172.21.15.90:0/3698951394 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5609da67b890 0x5609da6764c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:50.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.182+0000 7fe7d5400640 1 -- 172.21.15.90:0/3698951394 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5609da67b890 msgr2=0x5609da6764c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:40:50.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.182+0000 7fe7d5400640 1 --2- 172.21.15.90:0/3698951394 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5609da67b890 0x5609da6764c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:50.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.182+0000 7fe7d5400640 1 -- 172.21.15.90:0/3698951394 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5609da672520 msgr2=0x5609da683cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:40:50.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.182+0000 7fe7d5400640 1 --2- 172.21.15.90:0/3698951394 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5609da672520 0x5609da683cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:50.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.182+0000 7fe7d5400640 1 -- 172.21.15.90:0/3698951394 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5609da671550 con 0x5609da673200 2024-04-02T14:40:50.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.182+0000 7fe7d6800640 1 --2- 172.21.15.90:0/3698951394 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5609da67b890 0x5609da6764c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:40:50.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.182+0000 7fe7d5400640 1 --2- 172.21.15.90:0/3698951394 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5609da673200 0x5609da675e90 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fe7c409c1e0 tx=0x7fe7c40d0620 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e67ca317e7802b7f server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:40:50.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.182+0000 7fe7b7400640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:40:50.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.183+0000 7fe7b7400640 1 -- 172.21.15.90:0/3698951394 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7c4020070 con 0x5609da673200 2024-04-02T14:40:50.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.183+0000 7fe7b7400640 1 -- 172.21.15.90:0/3698951394 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7fe7c40cdc80 con 0x5609da673200 2024-04-02T14:40:50.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.183+0000 7fe7df9ad740 1 -- 172.21.15.90:0/3698951394 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe7c4015070 con 0x5609da673200 2024-04-02T14:40:50.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.183+0000 7fe7b7400640 1 -- 172.21.15.90:0/3698951394 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fe7c40cde80 con 0x5609da673200 2024-04-02T14:40:50.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.183+0000 7fe7df9ad740 10 client.27127 fetch_fsmap learned FSMap version 27 2024-04-02T14:40:50.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.183+0000 7fe7df9ad740 1 -- 172.21.15.90:0/3698951394 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5609da664260 con 0x5609da673200 2024-04-02T14:40:50.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.184+0000 7fe7b7400640 1 -- 172.21.15.90:0/3698951394 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe7c40cde80 con 0x5609da673200 2024-04-02T14:40:50.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.184+0000 7fe7b7400640 1 -- 172.21.15.90:0/3698951394 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fe7c409b7a0 con 0x5609da673200 2024-04-02T14:40:50.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.184+0000 7fe7df9ad740 10 client.27127 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:40:50.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.184+0000 7fe7df9ad740 10 client.27127 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:40:50.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.184+0000 7fe7df9ad740 20 client.27127 populate_metadata read hostname 'smithi090' 2024-04-02T14:40:50.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.184+0000 7fe7df9ad740 1 --2- 172.21.15.90:0/3698951394 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5609da664a00 0x5609da666e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:50.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.184+0000 7fe7df9ad740 4 client.27127 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-02T14:40:50.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.184+0000 7fe7df9ad740 1 -- 172.21.15.90:0/3698951394 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5609da5dbc30 con 0x5609da664a00 2024-04-02T14:40:50.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.184+0000 7fe7d5e00640 1 --2- 172.21.15.90:0/3698951394 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5609da664a00 0x5609da666e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:50.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.185+0000 7fe7b7400640 10 client.27127 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:40:50.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.185+0000 7fe7d5e00640 1 --2- 172.21.15.90:0/3698951394 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5609da664a00 0x5609da666e20 crc :-1 s=READY pgs=169 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d0feff936681c201 server_cookie=22eea1133567ef9f in_seq=0 out_seq=0 2024-04-02T14:40:50.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.185+0000 7fe7b7400640 1 -- 172.21.15.90:0/3698951394 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5609da5dbc30 con 0x5609da664a00 2024-04-02T14:40:50.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.185+0000 7fe7b7400640 10 client.27127 handle_command_reply: tid=0 2024-04-02T14:40:50.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.185+0000 7fe7df9ad740 1 client.27127 shutdown 2024-04-02T14:40:50.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.186+0000 7fe7df9ad740 1 -- 172.21.15.90:0/3698951394 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5609da673200 msgr2=0x5609da675e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:50.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.186+0000 7fe7df9ad740 1 --2- 172.21.15.90:0/3698951394 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5609da673200 0x5609da675e90 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fe7c409c1e0 tx=0x7fe7c40d0620 comp rx=0 tx=0).stop 2024-04-02T14:40:50.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.186+0000 7fe7df9ad740 1 -- 172.21.15.90:0/3698951394 shutdown_connections 2024-04-02T14:40:50.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.186+0000 7fe7df9ad740 1 --2- 172.21.15.90:0/3698951394 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5609da664a00 0x5609da666e20 crc :-1 s=READY pgs=169 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:50.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.186+0000 7fe7df9ad740 1 --2- 172.21.15.90:0/3698951394 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5609da67b890 0x5609da6764c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:50.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.186+0000 7fe7df9ad740 1 --2- 172.21.15.90:0/3698951394 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5609da673200 0x5609da675e90 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:50.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.186+0000 7fe7df9ad740 1 --2- 172.21.15.90:0/3698951394 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5609da672520 0x5609da683cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:50.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.186+0000 7fe7df9ad740 1 -- 172.21.15.90:0/3698951394 >> 172.21.15.90:0/3698951394 conn(0x5609da630a50 msgr2=0x5609da679140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:40:50.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.186+0000 7fe7b7400640 0 client.27127 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:40:50.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.187+0000 7fe7df9ad740 1 -- 172.21.15.90:0/3698951394 shutdown_connections 2024-04-02T14:40:50.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.187+0000 7fe7df9ad740 1 -- 172.21.15.90:0/3698951394 wait complete. 2024-04-02T14:40:50.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.187+0000 7fe7df9ad740 20 client.27127 trim_cache size 0 max 16384 2024-04-02T14:40:50.191 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-02T14:40:50.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.188+0000 7fe7dcc00640 1 -- 172.21.15.90:0/1067219792 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fe7bc0a3eb0 msgr2=0x7fe7bc0a6370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:50.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.188+0000 7fe7dcc00640 1 --2- 172.21.15.90:0/1067219792 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fe7bc0a3eb0 0x7fe7bc0a6370 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fe7c8005e00 tx=0x7fe7c800d780 comp rx=0 tx=0).stop 2024-04-02T14:40:50.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.188+0000 7fe7dcc00640 1 -- 172.21.15.90:0/1067219792 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe7d8102440 msgr2=0x7fe7d8196710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:50.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.188+0000 7fe7dcc00640 1 --2- 172.21.15.90:0/1067219792 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe7d8102440 0x7fe7d8196710 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fe7c400d950 tx=0x7fe7c400de10 comp rx=0 tx=0).stop 2024-04-02T14:40:50.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.188+0000 7fe7dcc00640 1 -- 172.21.15.90:0/1067219792 shutdown_connections 2024-04-02T14:40:50.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.188+0000 7fe7dcc00640 1 --2- 172.21.15.90:0/1067219792 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fe7bc0a3eb0 0x7fe7bc0a6370 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:50.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.188+0000 7fe7dcc00640 1 --2- 172.21.15.90:0/1067219792 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe7d8107a90 0x7fe7d819aa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:50.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.188+0000 7fe7dcc00640 1 --2- 172.21.15.90:0/1067219792 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe7d8102440 0x7fe7d8196710 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:50.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.188+0000 7fe7dcc00640 1 --2- 172.21.15.90:0/1067219792 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe7d8101a80 0x7fe7d81961d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:50.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.188+0000 7fe7dcc00640 1 -- 172.21.15.90:0/1067219792 >> 172.21.15.90:0/1067219792 conn(0x7fe7d80fd590 msgr2=0x7fe7d80fdd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:40:50.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.189+0000 7fe7dcc00640 1 -- 172.21.15.90:0/1067219792 shutdown_connections 2024-04-02T14:40:50.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:50.189+0000 7fe7dcc00640 1 -- 172.21.15.90:0/1067219792 wait complete. 2024-04-02T14:40:51.212 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-02T14:40:51.212 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T14:40:51.212 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-02T14:40:51.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.495+0000 7f9ddea00640 1 -- 172.21.15.90:0/2973323181 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9dd8068ce0 msgr2=0x7f9dd8069160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:51.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.495+0000 7f9ddea00640 1 --2- 172.21.15.90:0/2973323181 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9dd8068ce0 0x7f9dd8069160 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f9dc4009900 tx=0x7f9dc402bc30 comp rx=0 tx=0).stop 2024-04-02T14:40:51.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.496+0000 7f9ddea00640 1 -- 172.21.15.90:0/2973323181 shutdown_connections 2024-04-02T14:40:51.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.496+0000 7f9ddea00640 1 --2- 172.21.15.90:0/2973323181 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9dd80696a0 0x7f9dd810e940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:51.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.496+0000 7f9ddea00640 1 --2- 172.21.15.90:0/2973323181 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9dd8068ce0 0x7f9dd8069160 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:51.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.496+0000 7f9ddea00640 1 --2- 172.21.15.90:0/2973323181 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9dd8107a80 0x7f9dd8107e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:51.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.496+0000 7f9ddea00640 1 -- 172.21.15.90:0/2973323181 >> 172.21.15.90:0/2973323181 conn(0x7f9dd8077b00 msgr2=0x7f9dd8077f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:40:51.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.496+0000 7f9ddea00640 1 -- 172.21.15.90:0/2973323181 shutdown_connections 2024-04-02T14:40:51.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.496+0000 7f9ddea00640 1 -- 172.21.15.90:0/2973323181 wait complete. 2024-04-02T14:40:51.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.498+0000 7f9ddea00640 1 Processor -- start 2024-04-02T14:40:51.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.498+0000 7f9ddea00640 1 -- start start 2024-04-02T14:40:51.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.499+0000 7f9ddea00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9dd8068ce0 0x7f9dd8073120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:51.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.499+0000 7f9ddea00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9dd80696a0 0x7f9dd8073660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:51.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.499+0000 7f9dd7e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9dd8068ce0 0x7f9dd8073120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:51.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.499+0000 7f9dd7e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9dd8068ce0 0x7f9dd8073120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:34384/0 (socket says 172.21.15.90:34384) 2024-04-02T14:40:51.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.499+0000 7f9dd7400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9dd80696a0 0x7f9dd8073660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:51.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.499+0000 7f9dd7400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9dd80696a0 0x7f9dd8073660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41712/0 (socket says 172.21.15.90:41712) 2024-04-02T14:40:51.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.500+0000 7f9dd7e00640 1 -- 172.21.15.90:0/1293789658 learned_addr learned my addr 172.21.15.90:0/1293789658 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:40:51.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.500+0000 7f9ddea00640 1 --2- 172.21.15.90:0/1293789658 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9dd8107a80 0x7f9dd810dc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:51.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.500+0000 7f9ddea00640 1 -- 172.21.15.90:0/1293789658 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f9dd8111f00 con 0x7f9dd80696a0 2024-04-02T14:40:51.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.500+0000 7f9ddea00640 1 -- 172.21.15.90:0/1293789658 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f9dd8111d80 con 0x7f9dd8068ce0 2024-04-02T14:40:51.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.500+0000 7f9ddea00640 1 -- 172.21.15.90:0/1293789658 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f9dd8112080 con 0x7f9dd8107a80 2024-04-02T14:40:51.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.500+0000 7f9ddca00640 1 --2- 172.21.15.90:0/1293789658 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9dd8107a80 0x7f9dd810dc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:51.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.500+0000 7f9dd7e00640 1 -- 172.21.15.90:0/1293789658 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9dd8107a80 msgr2=0x7f9dd810dc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:51.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.500+0000 7f9dd7e00640 1 --2- 172.21.15.90:0/1293789658 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9dd8107a80 0x7f9dd810dc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:51.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.500+0000 7f9dd7e00640 1 -- 172.21.15.90:0/1293789658 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9dd80696a0 msgr2=0x7f9dd8073660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:51.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.500+0000 7f9dd7e00640 1 --2- 172.21.15.90:0/1293789658 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9dd80696a0 0x7f9dd8073660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:51.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.500+0000 7f9dd7e00640 1 -- 172.21.15.90:0/1293789658 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9dd810e4d0 con 0x7f9dd8068ce0 2024-04-02T14:40:51.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.501+0000 7f9dd7e00640 1 --2- 172.21.15.90:0/1293789658 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9dd8068ce0 0x7f9dd8073120 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f9dc800b4a0 tx=0x7f9dc800b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:40:51.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.501+0000 7f9dd4c00640 1 -- 172.21.15.90:0/1293789658 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9dc8014070 con 0x7f9dd8068ce0 2024-04-02T14:40:51.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.501+0000 7f9dd4c00640 1 -- 172.21.15.90:0/1293789658 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9dc8010af0 con 0x7f9dd8068ce0 2024-04-02T14:40:51.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.501+0000 7f9dd4c00640 1 -- 172.21.15.90:0/1293789658 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9dc8017d10 con 0x7f9dd8068ce0 2024-04-02T14:40:51.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.501+0000 7f9ddea00640 1 -- 172.21.15.90:0/1293789658 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9dd810e7c0 con 0x7f9dd8068ce0 2024-04-02T14:40:51.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.502+0000 7f9ddea00640 1 -- 172.21.15.90:0/1293789658 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9dd806ed40 con 0x7f9dd8068ce0 2024-04-02T14:40:51.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.508+0000 7f9de179e740 1 Processor -- start 2024-04-02T14:40:51.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.509+0000 7f9de179e740 1 -- start start 2024-04-02T14:40:51.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.510+0000 7f9de179e740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5614c401f030 0x5614c401dd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:51.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.510+0000 7f9de179e740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5614c401e350 0x5614c401e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:51.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.510+0000 7f9de179e740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5614c40276c0 0x5614c4029ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:51.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.510+0000 7f9de179e740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5614c3f82310 con 0x5614c401f030 2024-04-02T14:40:51.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.510+0000 7f9de179e740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x5614c3f2a690 con 0x5614c40276c0 2024-04-02T14:40:51.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.510+0000 7f9de179e740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5614c3f7e2c0 con 0x5614c401e350 2024-04-02T14:40:51.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.511+0000 7f9dd7400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5614c40276c0 0x5614c4029ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:51.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.511+0000 7f9dd7400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5614c40276c0 0x5614c4029ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:34400/0 (socket says 172.21.15.90:34400) 2024-04-02T14:40:51.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.511+0000 7f9dd7400640 1 -- 172.21.15.90:0/4025944725 learned_addr learned my addr 172.21.15.90:0/4025944725 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:40:51.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.511+0000 7f9ddca00640 1 --2- 172.21.15.90:0/4025944725 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5614c401f030 0x5614c401dd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:51.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.511+0000 7f9dd4c00640 1 -- 172.21.15.90:0/1293789658 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f9dc8017070 con 0x7f9dd8068ce0 2024-04-02T14:40:51.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.511+0000 7f9dd7400640 1 -- 172.21.15.90:0/4025944725 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5614c401e350 msgr2=0x5614c401e7d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-02T14:40:51.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.511+0000 7f9dd7400640 1 --2- 172.21.15.90:0/4025944725 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5614c401e350 0x5614c401e7d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:51.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.511+0000 7f9dd7400640 1 -- 172.21.15.90:0/4025944725 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5614c401f030 msgr2=0x5614c401dd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:40:51.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.512+0000 7f9dd7400640 1 --2- 172.21.15.90:0/4025944725 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5614c401f030 0x5614c401dd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:51.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.512+0000 7f9dd7400640 1 -- 172.21.15.90:0/4025944725 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5614c3d5e7e0 con 0x5614c40276c0 2024-04-02T14:40:51.514 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.513+0000 7f9ddca00640 1 --2- 172.21.15.90:0/4025944725 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5614c401f030 0x5614c401dd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:40:51.514 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.513+0000 7f9dd7400640 1 --2- 172.21.15.90:0/4025944725 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5614c40276c0 0x5614c4029ab0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f9dc4009900 tx=0x7f9dc402c710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=26be2b26030d9606 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:40:51.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.514+0000 7f9dcd200640 1 -- 172.21.15.90:0/4025944725 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9dc4042070 con 0x5614c40276c0 2024-04-02T14:40:51.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.514+0000 7f9dcd200640 1 -- 172.21.15.90:0/4025944725 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9dc4004380 con 0x5614c40276c0 2024-04-02T14:40:51.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.514+0000 7f9dcd200640 1 -- 172.21.15.90:0/4025944725 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9dc40050f0 con 0x5614c40276c0 2024-04-02T14:40:51.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.515+0000 7f9de179e740 1 -- 172.21.15.90:0/4025944725 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5614c40276c0 msgr2=0x5614c4029ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:51.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.515+0000 7f9de179e740 1 --2- 172.21.15.90:0/4025944725 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5614c40276c0 0x5614c4029ab0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f9dc4009900 tx=0x7f9dc402c710 comp rx=0 tx=0).stop 2024-04-02T14:40:51.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.516+0000 7f9dd4c00640 1 --2- 172.21.15.90:0/1293789658 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f9dbc0a3de0 0x7f9dbc0a62a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:51.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.516+0000 7f9dd4c00640 1 -- 172.21.15.90:0/1293789658 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f9dc80cfa10 con 0x7f9dd8068ce0 2024-04-02T14:40:51.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.518+0000 7f9de179e740 1 -- 172.21.15.90:0/4025944725 shutdown_connections 2024-04-02T14:40:51.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.518+0000 7f9de179e740 1 --2- 172.21.15.90:0/4025944725 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5614c40276c0 0x5614c4029ab0 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:51.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.518+0000 7f9de179e740 1 --2- 172.21.15.90:0/4025944725 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5614c401e350 0x5614c401e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:51.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.518+0000 7f9de179e740 1 --2- 172.21.15.90:0/4025944725 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5614c401f030 0x5614c401dd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:51.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.518+0000 7f9de179e740 1 -- 172.21.15.90:0/4025944725 >> 172.21.15.90:0/4025944725 conn(0x5614c3fb2270 msgr2=0x5614c4018da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:40:51.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.518+0000 7f9ddca00640 1 --2- 172.21.15.90:0/1293789658 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f9dbc0a3de0 0x7f9dbc0a62a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:51.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.519+0000 7f9de179e740 1 -- 172.21.15.90:0/4025944725 shutdown_connections 2024-04-02T14:40:51.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.519+0000 7f9de179e740 1 -- 172.21.15.90:0/4025944725 wait complete. 2024-04-02T14:40:51.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.520+0000 7f9de179e740 1 -- start start 2024-04-02T14:40:51.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.520+0000 7f9de179e740 1 Processor -- start 2024-04-02T14:40:51.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.521+0000 7f9ddca00640 1 --2- 172.21.15.90:0/1293789658 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f9dbc0a3de0 0x7f9dbc0a62a0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f9dd0005fd0 tx=0x7f9dd0009620 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:40:51.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.521+0000 7f9de179e740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5614c401e350 0x5614c40246e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:51.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.522+0000 7f9de179e740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5614c401f030 0x5614c4024c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:51.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.522+0000 7f9de179e740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5614c40276c0 0x5614c4025160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:51.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.522+0000 7f9de179e740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5614c402a4d0 con 0x5614c40276c0 2024-04-02T14:40:51.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.522+0000 7f9de179e740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x5614c3f3da60 con 0x5614c401f030 2024-04-02T14:40:51.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.522+0000 7f9de179e740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5614c402a650 con 0x5614c401e350 2024-04-02T14:40:51.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.522+0000 7f9de179e740 10 client.0 _ll_register_callbacks cb 0x5614c3f43030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:40:51.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.523+0000 7f9ddca00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5614c40276c0 0x5614c4025160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:51.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.523+0000 7f9ddca00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5614c40276c0 0x5614c4025160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41722/0 (socket says 172.21.15.90:41722) 2024-04-02T14:40:51.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.523+0000 7f9ddca00640 1 -- 172.21.15.90:0/4259177951 learned_addr learned my addr 172.21.15.90:0/4259177951 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:40:51.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.523+0000 7f9dd7400640 1 --2- 172.21.15.90:0/4259177951 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5614c401f030 0x5614c4024c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:51.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.524+0000 7f9dd7e00640 1 --2- 172.21.15.90:0/4259177951 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5614c401e350 0x5614c40246e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:51.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.524+0000 7f9dd7400640 1 -- 172.21.15.90:0/4259177951 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5614c401e350 msgr2=0x5614c40246e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:40:51.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.524+0000 7f9dd7400640 1 --2- 172.21.15.90:0/4259177951 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5614c401e350 0x5614c40246e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:51.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.524+0000 7f9dd7400640 1 -- 172.21.15.90:0/4259177951 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5614c40276c0 msgr2=0x5614c4025160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:40:51.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.524+0000 7f9dd7400640 1 --2- 172.21.15.90:0/4259177951 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5614c40276c0 0x5614c4025160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:51.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.524+0000 7f9dd7400640 1 -- 172.21.15.90:0/4259177951 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5614c3dbef70 con 0x5614c401f030 2024-04-02T14:40:51.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.524+0000 7f9dd7e00640 1 --2- 172.21.15.90:0/4259177951 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5614c401e350 0x5614c40246e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:40:51.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.524+0000 7f9dd7400640 1 --2- 172.21.15.90:0/4259177951 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5614c401f030 0x5614c4024c20 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f9dc4006010 tx=0x7f9dc402cae0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=44b69908cd18e309 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:40:51.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.524+0000 7f9dc1600640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:40:51.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.524+0000 7f9dc1600640 1 -- 172.21.15.90:0/4259177951 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9dc4042040 con 0x5614c401f030 2024-04-02T14:40:51.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.525+0000 7f9de179e740 1 -- 172.21.15.90:0/4259177951 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9dc4004380 con 0x5614c401f030 2024-04-02T14:40:51.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.525+0000 7f9dc1600640 1 -- 172.21.15.90:0/4259177951 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f9dc4004e90 con 0x5614c401f030 2024-04-02T14:40:51.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.525+0000 7f9dc1600640 1 -- 172.21.15.90:0/4259177951 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f9dc4005090 con 0x5614c401f030 2024-04-02T14:40:51.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.525+0000 7f9de179e740 10 client.27260 fetch_fsmap learned FSMap version 27 2024-04-02T14:40:51.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.525+0000 7f9de179e740 1 -- 172.21.15.90:0/4259177951 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5614c3c56170 con 0x5614c401f030 2024-04-02T14:40:51.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.525+0000 7f9dc1600640 1 -- 172.21.15.90:0/4259177951 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9dc4004380 con 0x5614c401f030 2024-04-02T14:40:51.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.526+0000 7f9dc1600640 1 -- 172.21.15.90:0/4259177951 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f9dc4003cf0 con 0x5614c401f030 2024-04-02T14:40:51.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.526+0000 7f9de179e740 10 client.27260 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:40:51.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.526+0000 7f9de179e740 10 client.27260 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:40:51.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.526+0000 7f9de179e740 20 client.27260 populate_metadata read hostname 'smithi090' 2024-04-02T14:40:51.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.526+0000 7f9de179e740 1 --2- 172.21.15.90:0/4259177951 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5614c4037a10 0x5614c4039e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:51.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.526+0000 7f9de179e740 4 client.27260 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:40:51.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.526+0000 7f9de179e740 1 -- 172.21.15.90:0/4259177951 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5614c3f8f850 con 0x5614c4037a10 2024-04-02T14:40:51.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.526+0000 7f9dd7e00640 1 --2- 172.21.15.90:0/4259177951 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5614c4037a10 0x5614c4039e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:51.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.527+0000 7f9dc1600640 10 client.27260 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:40:51.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.527+0000 7f9dd7e00640 1 --2- 172.21.15.90:0/4259177951 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5614c4037a10 0x5614c4039e00 crc :-1 s=READY pgs=170 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4e036103479fea7c server_cookie=b03113140af581b3 in_seq=0 out_seq=0 2024-04-02T14:40:51.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.528+0000 7f9dc1600640 1 -- 172.21.15.90:0/4259177951 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5614c3f8f850 con 0x5614c4037a10 2024-04-02T14:40:51.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.528+0000 7f9dc1600640 10 client.27260 handle_command_reply: tid=0 2024-04-02T14:40:51.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.528+0000 7f9de179e740 1 client.27260 shutdown 2024-04-02T14:40:51.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.528+0000 7f9de179e740 1 -- 172.21.15.90:0/4259177951 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5614c401f030 msgr2=0x5614c4024c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:51.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.528+0000 7f9de179e740 1 --2- 172.21.15.90:0/4259177951 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5614c401f030 0x5614c4024c20 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f9dc4006010 tx=0x7f9dc402cae0 comp rx=0 tx=0).stop 2024-04-02T14:40:51.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.529+0000 7f9de179e740 1 -- 172.21.15.90:0/4259177951 shutdown_connections 2024-04-02T14:40:51.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.529+0000 7f9de179e740 1 --2- 172.21.15.90:0/4259177951 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5614c4037a10 0x5614c4039e00 crc :-1 s=READY pgs=170 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:51.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.529+0000 7f9de179e740 1 --2- 172.21.15.90:0/4259177951 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5614c40276c0 0x5614c4025160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:51.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.529+0000 7f9de179e740 1 --2- 172.21.15.90:0/4259177951 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5614c401f030 0x5614c4024c20 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:51.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.529+0000 7f9dc1600640 0 client.27260 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:40:51.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.529+0000 7f9de179e740 1 --2- 172.21.15.90:0/4259177951 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5614c401e350 0x5614c40246e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:51.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.529+0000 7f9de179e740 1 -- 172.21.15.90:0/4259177951 >> 172.21.15.90:0/4259177951 conn(0x5614c3fb2270 msgr2=0x5614c401aaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:40:51.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.529+0000 7f9de179e740 1 -- 172.21.15.90:0/4259177951 shutdown_connections 2024-04-02T14:40:51.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.529+0000 7f9de179e740 1 -- 172.21.15.90:0/4259177951 wait complete. 2024-04-02T14:40:51.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.529+0000 7f9de179e740 20 client.27260 trim_cache size 0 max 16384 2024-04-02T14:40:51.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.564+0000 7f9de179e740 1 Processor -- start 2024-04-02T14:40:51.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.564+0000 7f9de179e740 1 -- start start 2024-04-02T14:40:51.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.565+0000 7f9de179e740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5614c401e350 0x5614c4023e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:51.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.565+0000 7f9de179e740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5614c401f030 0x5614c4024db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:51.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.565+0000 7f9de179e740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5614c40276c0 0x5614c403ce20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:51.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.565+0000 7f9de179e740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f9dd81c4dc0 con 0x5614c401f030 2024-04-02T14:40:51.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.565+0000 7f9de179e740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f9dd81c6900 con 0x5614c401e350 2024-04-02T14:40:51.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.565+0000 7f9de179e740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f9dd81c5490 con 0x5614c40276c0 2024-04-02T14:40:51.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.565+0000 7f9dd7e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5614c401e350 0x5614c4023e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:51.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.565+0000 7f9dd7e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5614c401e350 0x5614c4023e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:34408/0 (socket says 172.21.15.90:34408) 2024-04-02T14:40:51.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.565+0000 7f9dd7e00640 1 -- 172.21.15.90:0/1469996534 learned_addr learned my addr 172.21.15.90:0/1469996534 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:40:51.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.565+0000 7f9ddca00640 1 --2- 172.21.15.90:0/1469996534 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5614c40276c0 0x5614c403ce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:51.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.566+0000 7f9dd7400640 1 --2- 172.21.15.90:0/1469996534 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5614c401f030 0x5614c4024db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:51.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.566+0000 7f9dd7e00640 1 -- 172.21.15.90:0/1469996534 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5614c40276c0 msgr2=0x5614c403ce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:40:51.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.566+0000 7f9dd7e00640 1 --2- 172.21.15.90:0/1469996534 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5614c40276c0 0x5614c403ce20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:51.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.566+0000 7f9dd7e00640 1 -- 172.21.15.90:0/1469996534 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5614c401f030 msgr2=0x5614c4024db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:40:51.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.566+0000 7f9dd7e00640 1 --2- 172.21.15.90:0/1469996534 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5614c401f030 0x5614c4024db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:51.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.566+0000 7f9dd7e00640 1 -- 172.21.15.90:0/1469996534 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5614c402a580 con 0x5614c401e350 2024-04-02T14:40:51.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.566+0000 7f9ddca00640 1 --2- 172.21.15.90:0/1469996534 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5614c40276c0 0x5614c403ce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:40:51.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.566+0000 7f9dd7e00640 1 --2- 172.21.15.90:0/1469996534 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5614c401e350 0x5614c4023e90 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f9dc80984a0 tx=0x7f9dc800bf90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7674f89e190a796c server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:40:51.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.567+0000 7f9dc3e00640 1 -- 172.21.15.90:0/1469996534 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9dc801e040 con 0x5614c401e350 2024-04-02T14:40:51.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.567+0000 7f9dc3e00640 1 -- 172.21.15.90:0/1469996534 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9dc8099b70 con 0x5614c401e350 2024-04-02T14:40:51.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.567+0000 7f9dc3e00640 1 -- 172.21.15.90:0/1469996534 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9dc80cd3f0 con 0x5614c401e350 2024-04-02T14:40:51.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.568+0000 7f9de179e740 1 -- 172.21.15.90:0/1469996534 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5614c401e350 msgr2=0x5614c4023e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:51.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.568+0000 7f9de179e740 1 --2- 172.21.15.90:0/1469996534 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5614c401e350 0x5614c4023e90 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f9dc80984a0 tx=0x7f9dc800bf90 comp rx=0 tx=0).stop 2024-04-02T14:40:51.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.568+0000 7f9de179e740 1 -- 172.21.15.90:0/1469996534 shutdown_connections 2024-04-02T14:40:51.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.568+0000 7f9de179e740 1 --2- 172.21.15.90:0/1469996534 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5614c40276c0 0x5614c403ce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:51.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.568+0000 7f9de179e740 1 --2- 172.21.15.90:0/1469996534 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5614c401f030 0x5614c4024db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:51.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.568+0000 7f9de179e740 1 --2- 172.21.15.90:0/1469996534 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5614c401e350 0x5614c4023e90 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:51.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.568+0000 7f9de179e740 1 -- 172.21.15.90:0/1469996534 >> 172.21.15.90:0/1469996534 conn(0x5614c3fb2270 msgr2=0x5614c40246a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:40:51.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.568+0000 7f9de179e740 1 -- 172.21.15.90:0/1469996534 shutdown_connections 2024-04-02T14:40:51.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.568+0000 7f9de179e740 1 -- 172.21.15.90:0/1469996534 wait complete. 2024-04-02T14:40:51.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.568+0000 7f9de179e740 1 -- start start 2024-04-02T14:40:51.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.569+0000 7f9de179e740 1 Processor -- start 2024-04-02T14:40:51.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.569+0000 7f9de179e740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5614c401e350 0x5614c403aa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:51.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.569+0000 7f9de179e740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5614c401f030 0x5614c403afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:51.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.569+0000 7f9de179e740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5614c40276c0 0x5614c403b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:51.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.569+0000 7f9de179e740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f9dd81c6230 con 0x5614c401e350 2024-04-02T14:40:51.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.569+0000 7f9de179e740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f9dd81c5b60 con 0x5614c40276c0 2024-04-02T14:40:51.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.569+0000 7f9de179e740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f9dd81c2da0 con 0x5614c401f030 2024-04-02T14:40:51.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.569+0000 7f9de179e740 10 client.0 _ll_register_callbacks cb 0x7f9dd00094c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:40:51.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.569+0000 7f9ddca00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5614c40276c0 0x5614c403b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:51.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.569+0000 7f9ddca00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5614c40276c0 0x5614c403b4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:34418/0 (socket says 172.21.15.90:34418) 2024-04-02T14:40:51.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.569+0000 7f9ddca00640 1 -- 172.21.15.90:0/1977028231 learned_addr learned my addr 172.21.15.90:0/1977028231 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:40:51.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.569+0000 7f9dd7e00640 1 --2- 172.21.15.90:0/1977028231 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5614c401e350 0x5614c403aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:51.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.570+0000 7f9dd7400640 1 --2- 172.21.15.90:0/1977028231 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5614c401f030 0x5614c403afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:51.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.570+0000 7f9ddca00640 1 -- 172.21.15.90:0/1977028231 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5614c401f030 msgr2=0x5614c403afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:40:51.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.570+0000 7f9ddca00640 1 --2- 172.21.15.90:0/1977028231 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5614c401f030 0x5614c403afb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:51.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.570+0000 7f9ddca00640 1 -- 172.21.15.90:0/1977028231 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5614c401e350 msgr2=0x5614c403aa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:40:51.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.570+0000 7f9ddca00640 1 --2- 172.21.15.90:0/1977028231 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5614c401e350 0x5614c403aa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:51.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.570+0000 7f9ddca00640 1 -- 172.21.15.90:0/1977028231 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5614c409ce40 con 0x5614c40276c0 2024-04-02T14:40:51.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.570+0000 7f9dd7e00640 1 --2- 172.21.15.90:0/1977028231 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5614c401e350 0x5614c403aa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:40:51.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.571+0000 7f9dd7400640 1 --2- 172.21.15.90:0/1977028231 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5614c401f030 0x5614c403afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:40:51.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.571+0000 7f9ddca00640 1 --2- 172.21.15.90:0/1977028231 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5614c40276c0 0x5614c403b4f0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f9dd0003e40 tx=0x7f9dd0004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=14904090c2bd33a9 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:40:51.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.571+0000 7f9dc1600640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:40:51.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.572+0000 7f9dc1600640 1 -- 172.21.15.90:0/1977028231 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9dd002e060 con 0x5614c40276c0 2024-04-02T14:40:51.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.572+0000 7f9dc1600640 1 -- 172.21.15.90:0/1977028231 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f9dd00219d0 con 0x5614c40276c0 2024-04-02T14:40:51.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.572+0000 7f9de179e740 1 -- 172.21.15.90:0/1977028231 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9dc8099b70 con 0x5614c40276c0 2024-04-02T14:40:51.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.573+0000 7f9dc1600640 1 -- 172.21.15.90:0/1977028231 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f9dd0021bd0 con 0x5614c40276c0 2024-04-02T14:40:51.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.573+0000 7f9de179e740 10 client.27266 fetch_fsmap learned FSMap version 27 2024-04-02T14:40:51.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.573+0000 7f9de179e740 1 -- 172.21.15.90:0/1977028231 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5614c409e180 con 0x5614c40276c0 2024-04-02T14:40:51.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.573+0000 7f9dc1600640 1 -- 172.21.15.90:0/1977028231 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9dc8099b70 con 0x5614c40276c0 2024-04-02T14:40:51.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.574+0000 7f9dc1600640 1 -- 172.21.15.90:0/1977028231 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f9dd0028030 con 0x5614c40276c0 2024-04-02T14:40:51.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.574+0000 7f9de179e740 10 client.27266 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:40:51.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.574+0000 7f9de179e740 10 client.27266 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:40:51.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.574+0000 7f9de179e740 20 client.27266 populate_metadata read hostname 'smithi090' 2024-04-02T14:40:51.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.574+0000 7f9de179e740 1 --2- 172.21.15.90:0/1977028231 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5614c409e8b0 0x5614c40a0cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:51.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.574+0000 7f9de179e740 4 client.27266 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-02T14:40:51.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.574+0000 7f9de179e740 1 -- 172.21.15.90:0/1977028231 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5614c3f89ba0 con 0x5614c409e8b0 2024-04-02T14:40:51.576 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.575+0000 7f9dd7e00640 1 --2- 172.21.15.90:0/1977028231 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5614c409e8b0 0x5614c40a0cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:51.576 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.576+0000 7f9dd7e00640 1 --2- 172.21.15.90:0/1977028231 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5614c409e8b0 0x5614c40a0cd0 crc :-1 s=READY pgs=171 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b6174a3a8dbaf16 server_cookie=3db6f1bf7497a275 in_seq=0 out_seq=0 2024-04-02T14:40:51.576 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.576+0000 7f9dc1600640 10 client.27266 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:40:51.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.577+0000 7f9dc1600640 1 -- 172.21.15.90:0/1977028231 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5614c3f89ba0 con 0x5614c409e8b0 2024-04-02T14:40:51.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.577+0000 7f9dc1600640 10 client.27266 handle_command_reply: tid=0 2024-04-02T14:40:51.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.577+0000 7f9de179e740 1 client.27266 shutdown 2024-04-02T14:40:51.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.578+0000 7f9de179e740 1 -- 172.21.15.90:0/1977028231 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5614c40276c0 msgr2=0x5614c403b4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:51.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.578+0000 7f9de179e740 1 --2- 172.21.15.90:0/1977028231 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5614c40276c0 0x5614c403b4f0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f9dd0003e40 tx=0x7f9dd0004a40 comp rx=0 tx=0).stop 2024-04-02T14:40:51.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.578+0000 7f9de179e740 1 -- 172.21.15.90:0/1977028231 shutdown_connections 2024-04-02T14:40:51.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.578+0000 7f9de179e740 1 --2- 172.21.15.90:0/1977028231 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5614c409e8b0 0x5614c40a0cd0 crc :-1 s=READY pgs=171 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:51.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.578+0000 7f9de179e740 1 --2- 172.21.15.90:0/1977028231 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5614c40276c0 0x5614c403b4f0 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:51.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.578+0000 7f9dc1600640 0 client.27266 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:40:51.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.578+0000 7f9de179e740 1 --2- 172.21.15.90:0/1977028231 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5614c401f030 0x5614c403afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:51.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.578+0000 7f9de179e740 1 --2- 172.21.15.90:0/1977028231 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5614c401e350 0x5614c403aa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:51.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.578+0000 7f9de179e740 1 -- 172.21.15.90:0/1977028231 >> 172.21.15.90:0/1977028231 conn(0x5614c3fb2270 msgr2=0x5614c40331d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:40:51.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.578+0000 7f9de179e740 1 -- 172.21.15.90:0/1977028231 shutdown_connections 2024-04-02T14:40:51.582 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.578+0000 7f9de179e740 1 -- 172.21.15.90:0/1977028231 wait complete. 2024-04-02T14:40:51.582 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.579+0000 7f9de179e740 20 client.27266 trim_cache size 0 max 16384 2024-04-02T14:40:51.583 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-02T14:40:51.583 INFO:teuthology.orchestra.run.smithi090.stdout: "return_code": 0, 2024-04-02T14:40:51.583 INFO:teuthology.orchestra.run.smithi090.stdout: "scrub_tag": "a2f61ea9-3210-4998-a3a7-85919b48980a", 2024-04-02T14:40:51.583 INFO:teuthology.orchestra.run.smithi090.stdout: "mode": "asynchronous" 2024-04-02T14:40:51.583 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-02T14:40:51.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.581+0000 7f9ddea00640 1 -- 172.21.15.90:0/1293789658 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f9dbc0a3de0 msgr2=0x7f9dbc0a62a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:51.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.581+0000 7f9ddea00640 1 --2- 172.21.15.90:0/1293789658 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f9dbc0a3de0 0x7f9dbc0a62a0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f9dd0005fd0 tx=0x7f9dd0009620 comp rx=0 tx=0).stop 2024-04-02T14:40:51.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.581+0000 7f9ddea00640 1 -- 172.21.15.90:0/1293789658 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9dd8068ce0 msgr2=0x7f9dd8073120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:51.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.581+0000 7f9ddea00640 1 --2- 172.21.15.90:0/1293789658 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9dd8068ce0 0x7f9dd8073120 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f9dc800b4a0 tx=0x7f9dc800b960 comp rx=0 tx=0).stop 2024-04-02T14:40:51.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.581+0000 7f9ddea00640 1 -- 172.21.15.90:0/1293789658 shutdown_connections 2024-04-02T14:40:51.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.581+0000 7f9ddea00640 1 --2- 172.21.15.90:0/1293789658 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f9dbc0a3de0 0x7f9dbc0a62a0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:51.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.581+0000 7f9ddea00640 1 --2- 172.21.15.90:0/1293789658 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9dd8107a80 0x7f9dd810dc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:51.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.581+0000 7f9ddea00640 1 --2- 172.21.15.90:0/1293789658 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9dd80696a0 0x7f9dd8073660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:51.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.581+0000 7f9ddea00640 1 --2- 172.21.15.90:0/1293789658 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9dd8068ce0 0x7f9dd8073120 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:51.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.581+0000 7f9ddea00640 1 -- 172.21.15.90:0/1293789658 >> 172.21.15.90:0/1293789658 conn(0x7f9dd8077b00 msgr2=0x7f9dd8103570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:40:51.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.582+0000 7f9ddea00640 1 -- 172.21.15.90:0/1293789658 shutdown_connections 2024-04-02T14:40:51.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.582+0000 7f9ddea00640 1 -- 172.21.15.90:0/1293789658 wait complete. 2024-04-02T14:40:51.608 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-02T14:40:51.608 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-02T14:40:51.830 INFO:teuthology.orchestra.run.smithi090.stderr:progress: 320.0 s, 2299.4 tps, lat 43.307 ms stddev 50.244 2024-04-02T14:40:51.859 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:40:51 smithi092 ceph-mon[36497]: pgmap v182: 97 pgs: 97 active+clean; 3.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 8.4 MiB/s wr, 807 op/s 2024-04-02T14:40:51.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.876+0000 7fb94ce00640 1 -- 172.21.15.90:0/3646166189 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb948107980 msgr2=0x7fb948107d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:51.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.876+0000 7fb94ce00640 1 --2- 172.21.15.90:0/3646166189 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb948107980 0x7fb948107d60 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fb938009930 tx=0x7fb93802bc30 comp rx=0 tx=0).stop 2024-04-02T14:40:51.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.876+0000 7fb94ce00640 1 -- 172.21.15.90:0/3646166189 shutdown_connections 2024-04-02T14:40:51.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.876+0000 7fb94ce00640 1 --2- 172.21.15.90:0/3646166189 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb948102330 0x7fb94810e840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:51.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.876+0000 7fb94ce00640 1 --2- 172.21.15.90:0/3646166189 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb948101970 0x7fb948101df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:51.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.876+0000 7fb94ce00640 1 --2- 172.21.15.90:0/3646166189 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb948107980 0x7fb948107d60 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:51.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.876+0000 7fb94ce00640 1 -- 172.21.15.90:0/3646166189 >> 172.21.15.90:0/3646166189 conn(0x7fb9480fd4a0 msgr2=0x7fb9480ff8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:40:51.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.876+0000 7fb94ce00640 1 -- 172.21.15.90:0/3646166189 shutdown_connections 2024-04-02T14:40:51.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.876+0000 7fb94ce00640 1 -- 172.21.15.90:0/3646166189 wait complete. 2024-04-02T14:40:51.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.877+0000 7fb94ce00640 1 Processor -- start 2024-04-02T14:40:51.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.878+0000 7fb94ce00640 1 -- start start 2024-04-02T14:40:51.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.879+0000 7fb94ce00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb948101970 0x7fb948075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:51.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.879+0000 7fb94ce00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb948102330 0x7fb9480707f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:51.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.879+0000 7fb94ce00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb948107980 0x7fb948070d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:51.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.879+0000 7fb94ce00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fb948111db0 con 0x7fb948107980 2024-04-02T14:40:51.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.879+0000 7fb94ce00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fb948111c30 con 0x7fb948101970 2024-04-02T14:40:51.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.879+0000 7fb94ce00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fb948111f30 con 0x7fb948102330 2024-04-02T14:40:51.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.879+0000 7fb947400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb948101970 0x7fb948075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:51.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.879+0000 7fb947400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb948101970 0x7fb948075710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:34442/0 (socket says 172.21.15.90:34442) 2024-04-02T14:40:51.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.879+0000 7fb947400640 1 -- 172.21.15.90:0/1989206085 learned_addr learned my addr 172.21.15.90:0/1989206085 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:40:51.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.879+0000 7fb947e00640 1 --2- 172.21.15.90:0/1989206085 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb948107980 0x7fb948070d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:51.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.879+0000 7fb946a00640 1 --2- 172.21.15.90:0/1989206085 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb948102330 0x7fb9480707f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:51.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.880+0000 7fb947400640 1 -- 172.21.15.90:0/1989206085 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb948102330 msgr2=0x7fb9480707f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:51.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.880+0000 7fb947400640 1 --2- 172.21.15.90:0/1989206085 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb948102330 0x7fb9480707f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:51.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.880+0000 7fb947400640 1 -- 172.21.15.90:0/1989206085 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb948107980 msgr2=0x7fb948070d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:51.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.880+0000 7fb947400640 1 --2- 172.21.15.90:0/1989206085 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb948107980 0x7fb948070d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:51.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.880+0000 7fb947400640 1 -- 172.21.15.90:0/1989206085 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9480714b0 con 0x7fb948101970 2024-04-02T14:40:51.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.880+0000 7fb946a00640 1 --2- 172.21.15.90:0/1989206085 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb948102330 0x7fb9480707f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:40:51.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.880+0000 7fb947400640 1 --2- 172.21.15.90:0/1989206085 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb948101970 0x7fb948075710 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fb938009900 tx=0x7fb938004360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:40:51.882 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.880+0000 7fb93fe00640 1 -- 172.21.15.90:0/1989206085 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb938043020 con 0x7fb948101970 2024-04-02T14:40:51.882 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.881+0000 7fb93fe00640 1 -- 172.21.15.90:0/1989206085 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb9380054e0 con 0x7fb948101970 2024-04-02T14:40:51.882 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.881+0000 7fb93fe00640 1 -- 172.21.15.90:0/1989206085 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb938003b80 con 0x7fb948101970 2024-04-02T14:40:51.882 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.881+0000 7fb94ce00640 1 -- 172.21.15.90:0/1989206085 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb9481af290 con 0x7fb948101970 2024-04-02T14:40:51.882 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.881+0000 7fb94ce00640 1 -- 172.21.15.90:0/1989206085 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9481af700 con 0x7fb948101970 2024-04-02T14:40:51.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.885+0000 7fb95116d740 1 Processor -- start 2024-04-02T14:40:51.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.885+0000 7fb95116d740 1 -- start start 2024-04-02T14:40:51.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.886+0000 7fb95116d740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564836c87e20 0x564836c88200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:51.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.886+0000 7fb95116d740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564836c88740 0x564836c819c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:51.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.886+0000 7fb95116d740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564836c9a0e0 0x564836c9c4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:51.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.886+0000 7fb95116d740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x564836be3770 con 0x564836c9a0e0 2024-04-02T14:40:51.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.886+0000 7fb95116d740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x564836b8b700 con 0x564836c87e20 2024-04-02T14:40:51.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.886+0000 7fb95116d740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x564836bdf720 con 0x564836c88740 2024-04-02T14:40:51.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.885+0000 7fb93fe00640 1 -- 172.21.15.90:0/1989206085 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fb938046020 con 0x7fb948101970 2024-04-02T14:40:51.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.886+0000 7fb947400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564836c88740 0x564836c819c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:51.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.886+0000 7fb947400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564836c88740 0x564836c819c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:50122/0 (socket says 172.21.15.90:50122) 2024-04-02T14:40:51.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.886+0000 7fb947400640 1 -- 172.21.15.90:0/1687366097 learned_addr learned my addr 172.21.15.90:0/1687366097 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:40:51.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.886+0000 7fb947e00640 1 --2- 172.21.15.90:0/1687366097 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564836c87e20 0x564836c88200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:51.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.887+0000 7fb947400640 1 -- 172.21.15.90:0/1687366097 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564836c87e20 msgr2=0x564836c88200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:40:51.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.887+0000 7fb947400640 1 --2- 172.21.15.90:0/1687366097 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564836c87e20 0x564836c88200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:51.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.887+0000 7fb947400640 1 -- 172.21.15.90:0/1687366097 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564836c9a0e0 msgr2=0x564836c9c4d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-02T14:40:51.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.889+0000 7fb946a00640 1 --2- 172.21.15.90:0/1687366097 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564836c9a0e0 0x564836c9c4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:51.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.889+0000 7fb947400640 1 --2- 172.21.15.90:0/1687366097 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564836c9a0e0 0x564836c9c4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:51.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.889+0000 7fb947400640 1 -- 172.21.15.90:0/1687366097 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5648369c17d0 con 0x564836c88740 2024-04-02T14:40:51.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.889+0000 7fb946a00640 1 --2- 172.21.15.90:0/1687366097 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564836c9a0e0 0x564836c9c4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:40:51.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.890+0000 7fb947400640 1 --2- 172.21.15.90:0/1687366097 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564836c88740 0x564836c819c0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fb9380f0050 tx=0x7fb9380f8b40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6bfdd63d1c02dba6 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:40:51.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.890+0000 7fb93fe00640 1 --2- 172.21.15.90:0/1989206085 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fb92c0a3da0 0x7fb92c0a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:51.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.891+0000 7fb93fe00640 1 -- 172.21.15.90:0/1989206085 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7fb9380f2870 con 0x7fb948101970 2024-04-02T14:40:51.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.891+0000 7fb93cc00640 1 -- 172.21.15.90:0/1687366097 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb938101070 con 0x564836c88740 2024-04-02T14:40:51.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.891+0000 7fb93cc00640 1 -- 172.21.15.90:0/1687366097 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb938004c80 con 0x564836c88740 2024-04-02T14:40:51.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.891+0000 7fb93cc00640 1 -- 172.21.15.90:0/1687366097 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb938101070 con 0x564836c88740 2024-04-02T14:40:51.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.892+0000 7fb95116d740 1 -- 172.21.15.90:0/1687366097 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564836c88740 msgr2=0x564836c819c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:51.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.892+0000 7fb95116d740 1 --2- 172.21.15.90:0/1687366097 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564836c88740 0x564836c819c0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fb9380f0050 tx=0x7fb9380f8b40 comp rx=0 tx=0).stop 2024-04-02T14:40:51.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.894+0000 7fb947e00640 1 --2- 172.21.15.90:0/1989206085 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fb92c0a3da0 0x7fb92c0a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:51.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.895+0000 7fb95116d740 1 -- 172.21.15.90:0/1687366097 shutdown_connections 2024-04-02T14:40:51.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.895+0000 7fb95116d740 1 --2- 172.21.15.90:0/1687366097 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564836c9a0e0 0x564836c9c4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:51.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.895+0000 7fb95116d740 1 --2- 172.21.15.90:0/1687366097 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564836c88740 0x564836c819c0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:51.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.895+0000 7fb95116d740 1 --2- 172.21.15.90:0/1687366097 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564836c87e20 0x564836c88200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:51.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.895+0000 7fb95116d740 1 -- 172.21.15.90:0/1687366097 >> 172.21.15.90:0/1687366097 conn(0x564836c828a0 msgr2=0x564836c82c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:40:51.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.895+0000 7fb95116d740 1 -- 172.21.15.90:0/1687366097 shutdown_connections 2024-04-02T14:40:51.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.895+0000 7fb95116d740 1 -- 172.21.15.90:0/1687366097 wait complete. 2024-04-02T14:40:51.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.896+0000 7fb95116d740 1 -- start start 2024-04-02T14:40:51.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.897+0000 7fb95116d740 1 Processor -- start 2024-04-02T14:40:51.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.898+0000 7fb95116d740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564836c87e20 0x564836c7efb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:51.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.898+0000 7fb95116d740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564836c88740 0x564836c7f4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:51.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.898+0000 7fb95116d740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564836c9a0e0 0x564836c7fa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:51.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.898+0000 7fb95116d740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x564836c9cad0 con 0x564836c9a0e0 2024-04-02T14:40:51.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.898+0000 7fb95116d740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x564836b9eac0 con 0x564836c88740 2024-04-02T14:40:51.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.898+0000 7fb95116d740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x564836c9cc50 con 0x564836c87e20 2024-04-02T14:40:51.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.898+0000 7fb95116d740 10 client.0 _ll_register_callbacks cb 0x564836ba1180 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:40:51.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.899+0000 7fb947400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564836c87e20 0x564836c7efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:51.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.900+0000 7fb946a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564836c88740 0x564836c7f4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:51.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.900+0000 7fb946a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564836c88740 0x564836c7f4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:34460/0 (socket says 172.21.15.90:34460) 2024-04-02T14:40:51.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.900+0000 7fb946a00640 1 -- 172.21.15.90:0/4157764324 learned_addr learned my addr 172.21.15.90:0/4157764324 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:40:51.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.901+0000 7fb946a00640 1 -- 172.21.15.90:0/4157764324 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564836c87e20 msgr2=0x564836c7efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:40:51.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.901+0000 7fb946a00640 1 --2- 172.21.15.90:0/4157764324 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564836c87e20 0x564836c7efb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:51.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.901+0000 7fb946a00640 1 -- 172.21.15.90:0/4157764324 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564836c9a0e0 msgr2=0x564836c7fa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:40:51.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.901+0000 7fb946a00640 1 --2- 172.21.15.90:0/4157764324 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564836c9a0e0 0x564836c7fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:51.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.901+0000 7fb946a00640 1 -- 172.21.15.90:0/4157764324 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5648368f73f0 con 0x564836c88740 2024-04-02T14:40:51.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.902+0000 7fb947e00640 1 --2- 172.21.15.90:0/1989206085 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fb92c0a3da0 0x7fb92c0a6260 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fb940005fd0 tx=0x7fb940009500 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:40:51.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.903+0000 7fb946a00640 1 --2- 172.21.15.90:0/4157764324 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564836c88740 0x564836c7f4f0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fb93400e6b0 tx=0x7fb93400eb70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a104fddd995c6ce2 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:40:51.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.903+0000 7fb931600640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:40:51.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.903+0000 7fb931600640 1 -- 172.21.15.90:0/4157764324 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb93401c060 con 0x564836c88740 2024-04-02T14:40:51.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.903+0000 7fb95116d740 1 -- 172.21.15.90:0/4157764324 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb938004c80 con 0x564836c88740 2024-04-02T14:40:51.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.903+0000 7fb931600640 1 -- 172.21.15.90:0/4157764324 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7fb934009a80 con 0x564836c88740 2024-04-02T14:40:51.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.904+0000 7fb931600640 1 -- 172.21.15.90:0/4157764324 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fb934009c80 con 0x564836c88740 2024-04-02T14:40:51.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.904+0000 7fb95116d740 10 client.27278 fetch_fsmap learned FSMap version 27 2024-04-02T14:40:51.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.904+0000 7fb95116d740 1 -- 172.21.15.90:0/4157764324 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5648368b70f0 con 0x564836c88740 2024-04-02T14:40:51.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.905+0000 7fb931600640 1 -- 172.21.15.90:0/4157764324 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb938004c80 con 0x564836c88740 2024-04-02T14:40:51.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.905+0000 7fb931600640 1 -- 172.21.15.90:0/4157764324 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fb93401f430 con 0x564836c88740 2024-04-02T14:40:51.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.906+0000 7fb95116d740 10 client.27278 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:40:51.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.906+0000 7fb95116d740 10 client.27278 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:40:51.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.906+0000 7fb95116d740 20 client.27278 populate_metadata read hostname 'smithi090' 2024-04-02T14:40:51.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.906+0000 7fb95116d740 1 --2- 172.21.15.90:0/4157764324 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x564836ca9a70 0x564836cabe90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:51.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.906+0000 7fb95116d740 4 client.27278 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:40:51.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.906+0000 7fb95116d740 1 -- 172.21.15.90:0/4157764324 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564836b84630 con 0x564836ca9a70 2024-04-02T14:40:51.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.910+0000 7fb947400640 1 --2- 172.21.15.90:0/4157764324 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x564836ca9a70 0x564836cabe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:51.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.913+0000 7fb947400640 1 --2- 172.21.15.90:0/4157764324 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x564836ca9a70 0x564836cabe90 crc :-1 s=READY pgs=172 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=95e3c3f0611d1a50 server_cookie=e9bd865a0c8ca5e9 in_seq=0 out_seq=0 2024-04-02T14:40:51.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.913+0000 7fb931600640 10 client.27278 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:40:51.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.918+0000 7fb931600640 1 -- 172.21.15.90:0/4157764324 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x564836b84630 con 0x564836ca9a70 2024-04-02T14:40:51.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.918+0000 7fb931600640 10 client.27278 handle_command_reply: tid=0 2024-04-02T14:40:51.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.919+0000 7fb95116d740 1 client.27278 shutdown 2024-04-02T14:40:51.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.920+0000 7fb95116d740 1 -- 172.21.15.90:0/4157764324 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564836c88740 msgr2=0x564836c7f4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:51.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.920+0000 7fb95116d740 1 --2- 172.21.15.90:0/4157764324 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564836c88740 0x564836c7f4f0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fb93400e6b0 tx=0x7fb93400eb70 comp rx=0 tx=0).stop 2024-04-02T14:40:51.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.920+0000 7fb95116d740 1 -- 172.21.15.90:0/4157764324 shutdown_connections 2024-04-02T14:40:51.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.920+0000 7fb95116d740 1 --2- 172.21.15.90:0/4157764324 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x564836ca9a70 0x564836cabe90 crc :-1 s=READY pgs=172 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:51.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.920+0000 7fb95116d740 1 --2- 172.21.15.90:0/4157764324 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564836c9a0e0 0x564836c7fa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:51.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.920+0000 7fb95116d740 1 --2- 172.21.15.90:0/4157764324 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564836c88740 0x564836c7f4f0 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:51.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.920+0000 7fb95116d740 1 --2- 172.21.15.90:0/4157764324 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564836c87e20 0x564836c7efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:51.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.920+0000 7fb95116d740 1 -- 172.21.15.90:0/4157764324 >> 172.21.15.90:0/4157764324 conn(0x564836c828a0 msgr2=0x564836c845f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:40:51.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.920+0000 7fb931600640 0 client.27278 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:40:51.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.920+0000 7fb95116d740 1 -- 172.21.15.90:0/4157764324 shutdown_connections 2024-04-02T14:40:51.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.920+0000 7fb95116d740 1 -- 172.21.15.90:0/4157764324 wait complete. 2024-04-02T14:40:51.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.920+0000 7fb95116d740 20 client.27278 trim_cache size 0 max 16384 2024-04-02T14:40:51.934 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:40:51 smithi177 ceph-mon[36431]: pgmap v182: 97 pgs: 97 active+clean; 3.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 8.4 MiB/s wr, 807 op/s 2024-04-02T14:40:51.937 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:40:51 smithi090 ceph-mon[31674]: pgmap v182: 97 pgs: 97 active+clean; 3.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 8.4 MiB/s wr, 807 op/s 2024-04-02T14:40:51.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.960+0000 7fb95116d740 1 Processor -- start 2024-04-02T14:40:51.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.960+0000 7fb95116d740 1 -- start start 2024-04-02T14:40:51.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.961+0000 7fb95116d740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564836c828a0 0x564836ca57d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:51.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.961+0000 7fb95116d740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564836c87e20 0x564836c7f850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:51.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.961+0000 7fb95116d740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564836c88740 0x564836cb1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:51.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.961+0000 7fb95116d740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fb9481b3cc0 con 0x564836c87e20 2024-04-02T14:40:51.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.961+0000 7fb95116d740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fb9481b5800 con 0x564836c828a0 2024-04-02T14:40:51.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.961+0000 7fb95116d740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fb9481b4390 con 0x564836c88740 2024-04-02T14:40:51.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.962+0000 7fb947400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564836c828a0 0x564836ca57d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:51.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.962+0000 7fb947400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564836c828a0 0x564836ca57d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:34474/0 (socket says 172.21.15.90:34474) 2024-04-02T14:40:51.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.962+0000 7fb947400640 1 -- 172.21.15.90:0/2867698086 learned_addr learned my addr 172.21.15.90:0/2867698086 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:40:51.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.962+0000 7fb947e00640 1 --2- 172.21.15.90:0/2867698086 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564836c88740 0x564836cb1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:51.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.962+0000 7fb946a00640 1 --2- 172.21.15.90:0/2867698086 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564836c87e20 0x564836c7f850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:51.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.963+0000 7fb947e00640 1 -- 172.21.15.90:0/2867698086 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564836c828a0 msgr2=0x564836ca57d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:40:51.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.963+0000 7fb947e00640 1 --2- 172.21.15.90:0/2867698086 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564836c828a0 0x564836ca57d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:51.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.963+0000 7fb947e00640 1 -- 172.21.15.90:0/2867698086 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564836c87e20 msgr2=0x564836c7f850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:40:51.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.963+0000 7fb947e00640 1 --2- 172.21.15.90:0/2867698086 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564836c87e20 0x564836c7f850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:51.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.963+0000 7fb947e00640 1 -- 172.21.15.90:0/2867698086 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564836ca6a90 con 0x564836c88740 2024-04-02T14:40:51.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.963+0000 7fb947400640 1 --2- 172.21.15.90:0/2867698086 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564836c828a0 0x564836ca57d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:40:51.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.963+0000 7fb946a00640 1 --2- 172.21.15.90:0/2867698086 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564836c87e20 0x564836c7f850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:40:51.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.964+0000 7fb947e00640 1 --2- 172.21.15.90:0/2867698086 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564836c88740 0x564836cb1780 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fb940004410 tx=0x7fb940004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=585c9129e90bd560 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:40:51.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.965+0000 7fb933e00640 1 -- 172.21.15.90:0/2867698086 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb940028440 con 0x564836c88740 2024-04-02T14:40:51.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.965+0000 7fb933e00640 1 -- 172.21.15.90:0/2867698086 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb940011920 con 0x564836c88740 2024-04-02T14:40:51.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.965+0000 7fb933e00640 1 -- 172.21.15.90:0/2867698086 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb94001caa0 con 0x564836c88740 2024-04-02T14:40:51.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.966+0000 7fb95116d740 1 -- 172.21.15.90:0/2867698086 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564836c88740 msgr2=0x564836cb1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:51.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.966+0000 7fb95116d740 1 --2- 172.21.15.90:0/2867698086 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564836c88740 0x564836cb1780 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fb940004410 tx=0x7fb940004a40 comp rx=0 tx=0).stop 2024-04-02T14:40:51.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.966+0000 7fb95116d740 1 -- 172.21.15.90:0/2867698086 shutdown_connections 2024-04-02T14:40:51.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.966+0000 7fb95116d740 1 --2- 172.21.15.90:0/2867698086 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564836c88740 0x564836cb1780 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:51.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.966+0000 7fb95116d740 1 --2- 172.21.15.90:0/2867698086 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564836c87e20 0x564836c7f850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:51.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.966+0000 7fb95116d740 1 --2- 172.21.15.90:0/2867698086 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564836c828a0 0x564836ca57d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:51.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.966+0000 7fb95116d740 1 -- 172.21.15.90:0/2867698086 >> 172.21.15.90:0/2867698086 conn(0x564836c79a40 msgr2=0x564836c78a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:40:51.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.967+0000 7fb95116d740 1 -- 172.21.15.90:0/2867698086 shutdown_connections 2024-04-02T14:40:51.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.967+0000 7fb95116d740 1 -- 172.21.15.90:0/2867698086 wait complete. 2024-04-02T14:40:51.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.967+0000 7fb95116d740 1 -- start start 2024-04-02T14:40:51.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.968+0000 7fb95116d740 1 Processor -- start 2024-04-02T14:40:51.969 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.969+0000 7fb95116d740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564836c828a0 0x564836ca1ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:51.970 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.969+0000 7fb95116d740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564836c87e20 0x564836ca2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:51.970 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.969+0000 7fb95116d740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564836c88740 0x564836c9ee60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:51.970 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.969+0000 7fb95116d740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fb9481b5130 con 0x564836c828a0 2024-04-02T14:40:51.970 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.969+0000 7fb95116d740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fb9481b4a60 con 0x564836c88740 2024-04-02T14:40:51.970 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.969+0000 7fb95116d740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fb9481b1ca0 con 0x564836c87e20 2024-04-02T14:40:51.970 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.969+0000 7fb95116d740 10 client.0 _ll_register_callbacks cb 0x7fb93400c330 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:40:51.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.970+0000 7fb947e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564836c88740 0x564836c9ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:51.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.970+0000 7fb947e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564836c88740 0x564836c9ee60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:34490/0 (socket says 172.21.15.90:34490) 2024-04-02T14:40:51.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.970+0000 7fb947e00640 1 -- 172.21.15.90:0/1972359717 learned_addr learned my addr 172.21.15.90:0/1972359717 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:40:51.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.970+0000 7fb947400640 1 --2- 172.21.15.90:0/1972359717 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564836c828a0 0x564836ca1ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:51.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.970+0000 7fb946a00640 1 --2- 172.21.15.90:0/1972359717 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564836c87e20 0x564836ca2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:51.972 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.971+0000 7fb947e00640 1 -- 172.21.15.90:0/1972359717 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564836c87e20 msgr2=0x564836ca2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:40:51.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.972+0000 7fb947e00640 1 --2- 172.21.15.90:0/1972359717 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564836c87e20 0x564836ca2220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:51.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.972+0000 7fb947e00640 1 -- 172.21.15.90:0/1972359717 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564836c828a0 msgr2=0x564836ca1ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:40:51.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.972+0000 7fb947e00640 1 --2- 172.21.15.90:0/1972359717 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564836c828a0 0x564836ca1ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:51.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.972+0000 7fb947e00640 1 -- 172.21.15.90:0/1972359717 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564836c9f660 con 0x564836c88740 2024-04-02T14:40:51.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.972+0000 7fb946a00640 1 --2- 172.21.15.90:0/1972359717 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564836c87e20 0x564836ca2220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:40:51.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.972+0000 7fb947400640 1 --2- 172.21.15.90:0/1972359717 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564836c828a0 0x564836ca1ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:40:51.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.973+0000 7fb947e00640 1 --2- 172.21.15.90:0/1972359717 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564836c88740 0x564836c9ee60 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fb940018040 tx=0x7fb9400046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d4ae3b32a2aba07 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:40:51.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.973+0000 7fb931600640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:40:51.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.973+0000 7fb931600640 1 -- 172.21.15.90:0/1972359717 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb940004cc0 con 0x564836c88740 2024-04-02T14:40:51.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.973+0000 7fb95116d740 1 -- 172.21.15.90:0/1972359717 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb940011920 con 0x564836c88740 2024-04-02T14:40:51.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.973+0000 7fb931600640 1 -- 172.21.15.90:0/1972359717 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7fb940011ad0 con 0x564836c88740 2024-04-02T14:40:51.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.973+0000 7fb931600640 1 -- 172.21.15.90:0/1972359717 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fb94003d050 con 0x564836c88740 2024-04-02T14:40:51.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.973+0000 7fb95116d740 10 client.27284 fetch_fsmap learned FSMap version 27 2024-04-02T14:40:51.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.973+0000 7fb95116d740 1 -- 172.21.15.90:0/1972359717 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x564836d10bb0 con 0x564836c88740 2024-04-02T14:40:51.977 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.974+0000 7fb931600640 1 -- 172.21.15.90:0/1972359717 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb940011920 con 0x564836c88740 2024-04-02T14:40:51.977 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.974+0000 7fb931600640 1 -- 172.21.15.90:0/1972359717 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fb940038030 con 0x564836c88740 2024-04-02T14:40:51.977 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.975+0000 7fb95116d740 10 client.27284 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:40:51.977 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.975+0000 7fb95116d740 10 client.27284 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:40:51.977 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.975+0000 7fb95116d740 20 client.27284 populate_metadata read hostname 'smithi090' 2024-04-02T14:40:51.977 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.975+0000 7fb95116d740 1 --2- 172.21.15.90:0/1972359717 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x564836c9a0e0 0x564836d132c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:51.977 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.975+0000 7fb95116d740 4 client.27284 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-02T14:40:51.977 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.975+0000 7fb95116d740 1 -- 172.21.15.90:0/1972359717 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "scrub status"}) -- 0x564836bf0cb0 con 0x564836c9a0e0 2024-04-02T14:40:51.977 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.975+0000 7fb947400640 1 --2- 172.21.15.90:0/1972359717 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x564836c9a0e0 0x564836d132c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:51.977 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.975+0000 7fb947400640 1 --2- 172.21.15.90:0/1972359717 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x564836c9a0e0 0x564836d132c0 crc :-1 s=READY pgs=173 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7758b85965fcd909 server_cookie=c6d11e4efaf88c43 in_seq=0 out_seq=0 2024-04-02T14:40:51.977 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.975+0000 7fb931600640 10 client.27284 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:40:51.977 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.975+0000 7fb931600640 1 -- 172.21.15.90:0/1972359717 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x564836bf0cb0 con 0x564836c9a0e0 2024-04-02T14:40:51.977 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.976+0000 7fb931600640 10 client.27284 handle_command_reply: tid=0 2024-04-02T14:40:51.978 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.976+0000 7fb95116d740 1 client.27284 shutdown 2024-04-02T14:40:51.978 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.976+0000 7fb95116d740 1 -- 172.21.15.90:0/1972359717 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564836c88740 msgr2=0x564836c9ee60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:51.978 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.976+0000 7fb95116d740 1 --2- 172.21.15.90:0/1972359717 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564836c88740 0x564836c9ee60 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fb940018040 tx=0x7fb9400046c0 comp rx=0 tx=0).stop 2024-04-02T14:40:51.978 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.976+0000 7fb95116d740 1 -- 172.21.15.90:0/1972359717 shutdown_connections 2024-04-02T14:40:51.978 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.976+0000 7fb95116d740 1 --2- 172.21.15.90:0/1972359717 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x564836c9a0e0 0x564836d132c0 crc :-1 s=READY pgs=173 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:51.978 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.976+0000 7fb95116d740 1 --2- 172.21.15.90:0/1972359717 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564836c88740 0x564836c9ee60 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:51.978 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.976+0000 7fb95116d740 1 --2- 172.21.15.90:0/1972359717 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564836c87e20 0x564836ca2220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:51.978 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.976+0000 7fb95116d740 1 --2- 172.21.15.90:0/1972359717 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564836c828a0 0x564836ca1ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:51.978 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.976+0000 7fb95116d740 1 -- 172.21.15.90:0/1972359717 >> 172.21.15.90:0/1972359717 conn(0x564836c79a40 msgr2=0x564836c79310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:40:51.978 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.976+0000 7fb931600640 0 client.27284 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:40:51.978 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.976+0000 7fb95116d740 1 -- 172.21.15.90:0/1972359717 shutdown_connections 2024-04-02T14:40:51.978 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.976+0000 7fb95116d740 1 -- 172.21.15.90:0/1972359717 wait complete. 2024-04-02T14:40:51.978 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.976+0000 7fb95116d740 20 client.27284 trim_cache size 0 max 16384 2024-04-02T14:40:51.979 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-02T14:40:51.979 INFO:teuthology.orchestra.run.smithi090.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-02T14:40:51.979 INFO:teuthology.orchestra.run.smithi090.stdout: "scrubs": { 2024-04-02T14:40:51.979 INFO:teuthology.orchestra.run.smithi090.stdout: "a2f61ea9-3210-4998-a3a7-85919b48980a": { 2024-04-02T14:40:51.979 INFO:teuthology.orchestra.run.smithi090.stdout: "path": "/", 2024-04-02T14:40:51.979 INFO:teuthology.orchestra.run.smithi090.stdout: "tag": "a2f61ea9-3210-4998-a3a7-85919b48980a", 2024-04-02T14:40:51.979 INFO:teuthology.orchestra.run.smithi090.stdout: "options": "recursive,force" 2024-04-02T14:40:51.979 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:40:51.979 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:40:51.980 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-02T14:40:51.980 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.978+0000 7fb94ce00640 1 -- 172.21.15.90:0/1989206085 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fb92c0a3da0 msgr2=0x7fb92c0a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:51.980 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.978+0000 7fb94ce00640 1 --2- 172.21.15.90:0/1989206085 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fb92c0a3da0 0x7fb92c0a6260 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fb940005fd0 tx=0x7fb940009500 comp rx=0 tx=0).stop 2024-04-02T14:40:51.980 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.978+0000 7fb94ce00640 1 -- 172.21.15.90:0/1989206085 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb948101970 msgr2=0x7fb948075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:51.980 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.978+0000 7fb94ce00640 1 --2- 172.21.15.90:0/1989206085 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb948101970 0x7fb948075710 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fb938009900 tx=0x7fb938004360 comp rx=0 tx=0).stop 2024-04-02T14:40:51.980 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.978+0000 7fb94ce00640 1 -- 172.21.15.90:0/1989206085 shutdown_connections 2024-04-02T14:40:51.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.978+0000 7fb94ce00640 1 --2- 172.21.15.90:0/1989206085 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fb92c0a3da0 0x7fb92c0a6260 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:51.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.978+0000 7fb94ce00640 1 --2- 172.21.15.90:0/1989206085 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb948107980 0x7fb948070d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:51.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.978+0000 7fb94ce00640 1 --2- 172.21.15.90:0/1989206085 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb948102330 0x7fb9480707f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:51.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.978+0000 7fb94ce00640 1 --2- 172.21.15.90:0/1989206085 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb948101970 0x7fb948075710 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:51.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.978+0000 7fb94ce00640 1 -- 172.21.15.90:0/1989206085 >> 172.21.15.90:0/1989206085 conn(0x7fb9480fd4a0 msgr2=0x7fb9480fdc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:40:51.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.978+0000 7fb94ce00640 1 -- 172.21.15.90:0/1989206085 shutdown_connections 2024-04-02T14:40:51.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:40:51.979+0000 7fb94ce00640 1 -- 172.21.15.90:0/1989206085 wait complete. 2024-04-02T14:40:52.001 INFO:tasks.cephfs.filesystem:scrub status for tag:a2f61ea9-3210-4998-a3a7-85919b48980a - {'path': '/', 'tag': 'a2f61ea9-3210-4998-a3a7-85919b48980a', 'options': 'recursive,force'} 2024-04-02T14:40:52.935 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:40:52 smithi177 ceph-mon[36431]: scrub queued for path: / 2024-04-02T14:40:52.935 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:40:52 smithi177 ceph-mon[36431]: scrub summary: idle+waiting paths [/] 2024-04-02T14:40:52.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:40:52 smithi177 ceph-mon[36431]: scrub summary: active paths [/] 2024-04-02T14:40:52.937 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:40:52 smithi090 ceph-mon[31674]: scrub queued for path: / 2024-04-02T14:40:52.937 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:40:52 smithi090 ceph-mon[31674]: scrub summary: idle+waiting paths [/] 2024-04-02T14:40:52.937 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:40:52 smithi090 ceph-mon[31674]: scrub summary: active paths [/] 2024-04-02T14:40:53.109 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:40:52 smithi092 ceph-mon[36497]: scrub queued for path: / 2024-04-02T14:40:53.109 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:40:52 smithi092 ceph-mon[36497]: scrub summary: idle+waiting paths [/] 2024-04-02T14:40:53.109 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:40:52 smithi092 ceph-mon[36497]: scrub summary: active paths [/] 2024-04-02T14:40:53.935 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:40:53 smithi177 ceph-mon[36431]: scrub summary: idle+waiting paths [/] 2024-04-02T14:40:53.935 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:40:53 smithi177 ceph-mon[36431]: pgmap v183: 97 pgs: 97 active+clean; 3.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 6.3 MiB/s wr, 630 op/s 2024-04-02T14:40:53.937 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:40:53 smithi090 ceph-mon[31674]: scrub summary: idle+waiting paths [/] 2024-04-02T14:40:53.937 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:40:53 smithi090 ceph-mon[31674]: pgmap v183: 97 pgs: 97 active+clean; 3.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 6.3 MiB/s wr, 630 op/s 2024-04-02T14:40:54.109 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:40:53 smithi092 ceph-mon[36497]: scrub summary: idle+waiting paths [/] 2024-04-02T14:40:54.109 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:40:53 smithi092 ceph-mon[36497]: pgmap v183: 97 pgs: 97 active+clean; 3.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 6.3 MiB/s wr, 630 op/s 2024-04-02T14:40:55.935 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:40:55 smithi177 ceph-mon[36431]: pgmap v184: 97 pgs: 97 active+clean; 3.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 9.9 MiB/s wr, 1k op/s 2024-04-02T14:40:55.937 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:40:55 smithi090 ceph-mon[31674]: pgmap v184: 97 pgs: 97 active+clean; 3.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 9.9 MiB/s wr, 1k op/s 2024-04-02T14:40:56.109 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:40:55 smithi092 ceph-mon[36497]: pgmap v184: 97 pgs: 97 active+clean; 3.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 9.9 MiB/s wr, 1k op/s 2024-04-02T14:40:56.830 INFO:teuthology.orchestra.run.smithi090.stderr:progress: 325.0 s, 3302.2 tps, lat 30.366 ms stddev 36.761 2024-04-02T14:40:57.935 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:40:57 smithi177 ceph-mon[36431]: pgmap v185: 97 pgs: 97 active+clean; 3.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 6.7 MiB/s wr, 683 op/s 2024-04-02T14:40:57.935 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:40:57 smithi177 ceph-mon[36431]: scrub summary: idle 2024-04-02T14:40:57.936 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:40:57 smithi090 ceph-mon[31674]: pgmap v185: 97 pgs: 97 active+clean; 3.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 6.7 MiB/s wr, 683 op/s 2024-04-02T14:40:57.937 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:40:57 smithi090 ceph-mon[31674]: scrub summary: idle 2024-04-02T14:40:58.075 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.074+0000 7f3e55800640 1 -- 172.21.15.90:0/1874738650 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3e5010d410 msgr2=0x7f3e50114fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:58.075 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.074+0000 7f3e55800640 1 --2- 172.21.15.90:0/1874738650 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3e5010d410 0x7f3e50114fe0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f3e48009ca0 tx=0x7f3e4802c0f0 comp rx=0 tx=0).stop 2024-04-02T14:40:58.075 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.074+0000 7f3e55800640 1 -- 172.21.15.90:0/1874738650 shutdown_connections 2024-04-02T14:40:58.075 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.074+0000 7f3e55800640 1 --2- 172.21.15.90:0/1874738650 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3e5010d410 0x7f3e50114fe0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:58.075 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.074+0000 7f3e55800640 1 --2- 172.21.15.90:0/1874738650 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3e50108cf0 0x7f3e5010cd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:58.075 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.074+0000 7f3e55800640 1 --2- 172.21.15.90:0/1874738650 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3e50108340 0x7f3e50108720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:58.076 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.074+0000 7f3e55800640 1 -- 172.21.15.90:0/1874738650 >> 172.21.15.90:0/1874738650 conn(0x7f3e50076070 msgr2=0x7f3e50078490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:40:58.076 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.075+0000 7f3e55800640 1 -- 172.21.15.90:0/1874738650 shutdown_connections 2024-04-02T14:40:58.076 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.075+0000 7f3e55800640 1 -- 172.21.15.90:0/1874738650 wait complete. 2024-04-02T14:40:58.076 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.076+0000 7f3e55800640 1 Processor -- start 2024-04-02T14:40:58.077 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.076+0000 7f3e55800640 1 -- start start 2024-04-02T14:40:58.077 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.076+0000 7f3e55800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3e50108340 0x7f3e501a45f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:58.077 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.076+0000 7f3e55800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3e5010d410 0x7f3e501a4b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:58.077 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.076+0000 7f3e55800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3e5019e6b0 0x7f3e5019eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:58.077 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.076+0000 7f3e55800640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f3e501184b0 con 0x7f3e50108340 2024-04-02T14:40:58.077 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.076+0000 7f3e55800640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f3e50118330 con 0x7f3e5010d410 2024-04-02T14:40:58.077 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.076+0000 7f3e55800640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f3e50118630 con 0x7f3e5019e6b0 2024-04-02T14:40:58.078 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.077+0000 7f3e4f400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3e5019e6b0 0x7f3e5019eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:58.078 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.077+0000 7f3e4ea00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3e50108340 0x7f3e501a45f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:58.078 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.077+0000 7f3e4ea00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3e50108340 0x7f3e501a45f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:56910/0 (socket says 172.21.15.90:56910) 2024-04-02T14:40:58.078 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.077+0000 7f3e4ea00640 1 -- 172.21.15.90:0/1904184532 learned_addr learned my addr 172.21.15.90:0/1904184532 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:40:58.078 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.077+0000 7f3e46000640 1 --2- 172.21.15.90:0/1904184532 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3e5010d410 0x7f3e501a4b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:58.079 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.078+0000 7f3e4f400640 1 -- 172.21.15.90:0/1904184532 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3e5010d410 msgr2=0x7f3e501a4b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:58.079 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.078+0000 7f3e4f400640 1 --2- 172.21.15.90:0/1904184532 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3e5010d410 0x7f3e501a4b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:58.079 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.078+0000 7f3e4f400640 1 -- 172.21.15.90:0/1904184532 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3e50108340 msgr2=0x7f3e501a45f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:58.079 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.078+0000 7f3e4f400640 1 --2- 172.21.15.90:0/1904184532 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3e50108340 0x7f3e501a45f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:58.079 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.078+0000 7f3e4f400640 1 -- 172.21.15.90:0/1904184532 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e5019f280 con 0x7f3e5019e6b0 2024-04-02T14:40:58.079 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.078+0000 7f3e4ea00640 1 --2- 172.21.15.90:0/1904184532 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3e50108340 0x7f3e501a45f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:40:58.079 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.079+0000 7f3e4f400640 1 --2- 172.21.15.90:0/1904184532 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3e5019e6b0 0x7f3e5019eb30 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f3e48037040 tx=0x7f3e48032ea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:40:58.080 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.079+0000 7f3e47e00640 1 -- 172.21.15.90:0/1904184532 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e48041020 con 0x7f3e5019e6b0 2024-04-02T14:40:58.080 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.079+0000 7f3e47e00640 1 -- 172.21.15.90:0/1904184532 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3e48034cd0 con 0x7f3e5019e6b0 2024-04-02T14:40:58.080 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.079+0000 7f3e47e00640 1 -- 172.21.15.90:0/1904184532 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e48039040 con 0x7f3e5019e6b0 2024-04-02T14:40:58.080 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.079+0000 7f3e55800640 1 -- 172.21.15.90:0/1904184532 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e5019f510 con 0x7f3e5019e6b0 2024-04-02T14:40:58.080 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.080+0000 7f3e55800640 1 -- 172.21.15.90:0/1904184532 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e5006b7c0 con 0x7f3e5019e6b0 2024-04-02T14:40:58.085 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.084+0000 7f3e58402740 1 Processor -- start 2024-04-02T14:40:58.086 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.085+0000 7f3e47e00640 1 -- 172.21.15.90:0/1904184532 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f3e48044020 con 0x7f3e5019e6b0 2024-04-02T14:40:58.086 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.085+0000 7f3e58402740 1 -- start start 2024-04-02T14:40:58.088 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.086+0000 7f3e58402740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563a28bb7eb0 0x563a28bb8290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:58.088 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.086+0000 7f3e58402740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563a28bb87d0 0x563a28bc0e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:58.088 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.086+0000 7f3e58402740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563a28bc13b0 0x563a28bc37a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:58.088 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.086+0000 7f3e58402740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x563a28b1cd70 con 0x563a28bc13b0 2024-04-02T14:40:58.088 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.086+0000 7f3e58402740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x563a28ac52a0 con 0x563a28bb7eb0 2024-04-02T14:40:58.089 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.086+0000 7f3e58402740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x563a28b18d20 con 0x563a28bb87d0 2024-04-02T14:40:58.089 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.087+0000 7f3e47e00640 1 --2- 172.21.15.90:0/1904184532 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f3e300a3fa0 0x7f3e300a6460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:58.089 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.087+0000 7f3e4ea00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563a28bb87d0 0x563a28bc0e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:58.089 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.087+0000 7f3e4ea00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563a28bb87d0 0x563a28bc0e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:35368/0 (socket says 172.21.15.90:35368) 2024-04-02T14:40:58.089 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.087+0000 7f3e4ea00640 1 -- 172.21.15.90:0/2290534546 learned_addr learned my addr 172.21.15.90:0/2290534546 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:40:58.089 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.087+0000 7f3e46000640 1 --2- 172.21.15.90:0/2290534546 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563a28bc13b0 0x563a28bc37a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:58.089 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.087+0000 7f3e47e00640 1 -- 172.21.15.90:0/1904184532 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f3e480f2330 con 0x7f3e5019e6b0 2024-04-02T14:40:58.089 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.087+0000 7f3e46000640 1 --2- 172.21.15.90:0/2290534546 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563a28bb7eb0 0x563a28bb8290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:58.089 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.087+0000 7f3e4ea00640 1 -- 172.21.15.90:0/2290534546 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563a28bb7eb0 msgr2=0x563a28bb8290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:40:58.089 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.087+0000 7f3e4ea00640 1 --2- 172.21.15.90:0/2290534546 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563a28bb7eb0 0x563a28bb8290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:58.089 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.087+0000 7f3e4ea00640 1 -- 172.21.15.90:0/2290534546 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563a28bc13b0 msgr2=0x563a28bc37a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:40:58.089 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.087+0000 7f3e4ea00640 1 --2- 172.21.15.90:0/2290534546 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563a28bc13b0 0x563a28bc37a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:58.089 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.087+0000 7f3e4ea00640 1 -- 172.21.15.90:0/2290534546 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563a28ad9840 con 0x563a28bb87d0 2024-04-02T14:40:58.089 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.087+0000 7f3e4f400640 1 --2- 172.21.15.90:0/1904184532 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f3e300a3fa0 0x7f3e300a6460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:58.089 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.087+0000 7f3e46000640 1 --2- 172.21.15.90:0/2290534546 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563a28bc13b0 0x563a28bc37a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:40:58.090 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.088+0000 7f3e4ea00640 1 --2- 172.21.15.90:0/2290534546 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563a28bb87d0 0x563a28bc0e70 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f3e38011650 tx=0x7f3e38009f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d3ca23b6a8b81ffc server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:40:58.090 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.088+0000 7f3e3fe00640 1 -- 172.21.15.90:0/2290534546 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e3801e070 con 0x563a28bb87d0 2024-04-02T14:40:58.090 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.088+0000 7f3e3fe00640 1 -- 172.21.15.90:0/2290534546 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3e3800fba0 con 0x563a28bb87d0 2024-04-02T14:40:58.090 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.088+0000 7f3e3fe00640 1 -- 172.21.15.90:0/2290534546 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e38019cd0 con 0x563a28bb87d0 2024-04-02T14:40:58.091 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.088+0000 7f3e4f400640 1 --2- 172.21.15.90:0/1904184532 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f3e300a3fa0 0x7f3e300a6460 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f3e480ba7f0 tx=0x7f3e480095c0 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:40:58.091 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.089+0000 7f3e58402740 1 -- 172.21.15.90:0/2290534546 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563a28bb87d0 msgr2=0x563a28bc0e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:58.091 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.089+0000 7f3e58402740 1 --2- 172.21.15.90:0/2290534546 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563a28bb87d0 0x563a28bc0e70 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f3e38011650 tx=0x7f3e38009f90 comp rx=0 tx=0).stop 2024-04-02T14:40:58.091 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.090+0000 7f3e58402740 1 -- 172.21.15.90:0/2290534546 shutdown_connections 2024-04-02T14:40:58.091 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.090+0000 7f3e58402740 1 --2- 172.21.15.90:0/2290534546 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563a28bc13b0 0x563a28bc37a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:58.091 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.090+0000 7f3e58402740 1 --2- 172.21.15.90:0/2290534546 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563a28bb87d0 0x563a28bc0e70 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:58.091 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.090+0000 7f3e58402740 1 --2- 172.21.15.90:0/2290534546 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563a28bb7eb0 0x563a28bb8290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:58.091 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.090+0000 7f3e58402740 1 -- 172.21.15.90:0/2290534546 >> 172.21.15.90:0/2290534546 conn(0x563a28bb25e0 msgr2=0x563a28bb29c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:40:58.092 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.090+0000 7f3e58402740 1 -- 172.21.15.90:0/2290534546 shutdown_connections 2024-04-02T14:40:58.092 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.091+0000 7f3e58402740 1 -- 172.21.15.90:0/2290534546 wait complete. 2024-04-02T14:40:58.092 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.091+0000 7f3e58402740 1 -- start start 2024-04-02T14:40:58.092 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.092+0000 7f3e58402740 1 Processor -- start 2024-04-02T14:40:58.092 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.092+0000 7f3e58402740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563a28bb7eb0 0x563a28bbe3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:58.093 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.092+0000 7f3e58402740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563a28bb87d0 0x563a28bbe920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:58.093 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.092+0000 7f3e58402740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563a28bc13b0 0x563a28bbee60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:58.093 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.092+0000 7f3e58402740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x563a28bc41a0 con 0x563a28bb7eb0 2024-04-02T14:40:58.093 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.092+0000 7f3e58402740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x563a28ad8660 con 0x563a28bc13b0 2024-04-02T14:40:58.093 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.092+0000 7f3e58402740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x563a28bc4320 con 0x563a28bb87d0 2024-04-02T14:40:58.093 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.092+0000 7f3e58402740 10 client.0 _ll_register_callbacks cb 0x563a28ad8050 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:40:58.093 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.093+0000 7f3e46000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563a28bb7eb0 0x563a28bbe3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:58.094 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.093+0000 7f3e46000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563a28bb7eb0 0x563a28bbe3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:56924/0 (socket says 172.21.15.90:56924) 2024-04-02T14:40:58.094 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.093+0000 7f3e46000640 1 -- 172.21.15.90:0/2781215363 learned_addr learned my addr 172.21.15.90:0/2781215363 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:40:58.094 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.093+0000 7f3e4ea00640 1 --2- 172.21.15.90:0/2781215363 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563a28bb87d0 0x563a28bbe920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:58.094 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.093+0000 7f3e46000640 1 --2- 172.21.15.90:0/2781215363 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563a28bc13b0 0x563a28bbee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:58.094 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.094+0000 7f3e46000640 1 -- 172.21.15.90:0/2781215363 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563a28bb87d0 msgr2=0x563a28bbe920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:40:58.095 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.094+0000 7f3e46000640 1 --2- 172.21.15.90:0/2781215363 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563a28bb87d0 0x563a28bbe920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:58.095 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.094+0000 7f3e46000640 1 -- 172.21.15.90:0/2781215363 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563a28bc13b0 msgr2=0x563a28bbee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:40:58.095 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.094+0000 7f3e46000640 1 --2- 172.21.15.90:0/2781215363 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563a28bc13b0 0x563a28bbee60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:58.095 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.094+0000 7f3e46000640 1 -- 172.21.15.90:0/2781215363 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563a28ae7e80 con 0x563a28bb7eb0 2024-04-02T14:40:58.095 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.095+0000 7f3e46000640 1 --2- 172.21.15.90:0/2781215363 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563a28bb7eb0 0x563a28bbe3e0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f3e4000bf70 tx=0x7f3e4000b5b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7984c8a0130da324 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:40:58.096 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.095+0000 7f3e3cc00640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-02T14:40:58.096 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.096+0000 7f3e3cc00640 1 -- 172.21.15.90:0/2781215363 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e4001b040 con 0x563a28bb7eb0 2024-04-02T14:40:58.097 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.096+0000 7f3e3cc00640 1 -- 172.21.15.90:0/2781215363 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f3e40012850 con 0x563a28bb7eb0 2024-04-02T14:40:58.097 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.096+0000 7f3e58402740 1 -- 172.21.15.90:0/2781215363 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3e3800fba0 con 0x563a28bb7eb0 2024-04-02T14:40:58.097 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.097+0000 7f3e3cc00640 1 -- 172.21.15.90:0/2781215363 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f3e40012a50 con 0x563a28bb7eb0 2024-04-02T14:40:58.097 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.097+0000 7f3e58402740 10 client.16656 fetch_fsmap learned FSMap version 27 2024-04-02T14:40:58.097 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.097+0000 7f3e58402740 1 -- 172.21.15.90:0/2781215363 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x563a28831210 con 0x563a28bb7eb0 2024-04-02T14:40:58.099 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.097+0000 7f3e3cc00640 1 -- 172.21.15.90:0/2781215363 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3e3800fba0 con 0x563a28bb7eb0 2024-04-02T14:40:58.099 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.098+0000 7f3e3cc00640 1 -- 172.21.15.90:0/2781215363 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f3e4002ac30 con 0x563a28bb7eb0 2024-04-02T14:40:58.099 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.098+0000 7f3e58402740 10 client.16656 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:40:58.099 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.098+0000 7f3e58402740 10 client.16656 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-02T14:40:58.099 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.098+0000 7f3e58402740 20 client.16656 populate_metadata read hostname 'smithi090' 2024-04-02T14:40:58.099 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.098+0000 7f3e58402740 1 --2- 172.21.15.90:0/2781215363 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x563a28bd1740 0x563a28bd3b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:58.099 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.098+0000 7f3e58402740 4 client.16656 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:40:58.099 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.098+0000 7f3e58402740 1 -- 172.21.15.90:0/2781215363 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563a28acccd0 con 0x563a28bd1740 2024-04-02T14:40:58.099 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.098+0000 7f3e4ea00640 1 --2- 172.21.15.90:0/2781215363 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x563a28bd1740 0x563a28bd3b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:58.100 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.100+0000 7f3e4ea00640 1 --2- 172.21.15.90:0/2781215363 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x563a28bd1740 0x563a28bd3b90 crc :-1 s=READY pgs=174 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9f1e24c16b881331 server_cookie=be94766d0f8a6a5 in_seq=0 out_seq=0 2024-04-02T14:40:58.100 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.100+0000 7f3e3cc00640 10 client.16656 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:40:58.101 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.101+0000 7f3e3cc00640 1 -- 172.21.15.90:0/2781215363 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x563a28acccd0 con 0x563a28bd1740 2024-04-02T14:40:58.101 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.101+0000 7f3e3cc00640 10 client.16656 handle_command_reply: tid=0 2024-04-02T14:40:58.102 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.101+0000 7f3e58402740 1 client.16656 shutdown 2024-04-02T14:40:58.103 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.102+0000 7f3e58402740 1 -- 172.21.15.90:0/2781215363 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563a28bb7eb0 msgr2=0x563a28bbe3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:58.103 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.102+0000 7f3e58402740 1 --2- 172.21.15.90:0/2781215363 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563a28bb7eb0 0x563a28bbe3e0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f3e4000bf70 tx=0x7f3e4000b5b0 comp rx=0 tx=0).stop 2024-04-02T14:40:58.103 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.102+0000 7f3e58402740 1 -- 172.21.15.90:0/2781215363 shutdown_connections 2024-04-02T14:40:58.103 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.102+0000 7f3e58402740 1 --2- 172.21.15.90:0/2781215363 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x563a28bd1740 0x563a28bd3b90 crc :-1 s=READY pgs=174 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:58.103 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.102+0000 7f3e58402740 1 --2- 172.21.15.90:0/2781215363 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563a28bc13b0 0x563a28bbee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:58.103 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.102+0000 7f3e58402740 1 --2- 172.21.15.90:0/2781215363 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563a28bb87d0 0x563a28bbe920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:58.103 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.102+0000 7f3e58402740 1 --2- 172.21.15.90:0/2781215363 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563a28bb7eb0 0x563a28bbe3e0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:58.103 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.102+0000 7f3e58402740 1 -- 172.21.15.90:0/2781215363 >> 172.21.15.90:0/2781215363 conn(0x563a28bb25e0 msgr2=0x563a28bb4330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:40:58.103 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.102+0000 7f3e3cc00640 0 client.16656 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:40:58.104 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.103+0000 7f3e58402740 1 -- 172.21.15.90:0/2781215363 shutdown_connections 2024-04-02T14:40:58.104 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.103+0000 7f3e58402740 1 -- 172.21.15.90:0/2781215363 wait complete. 2024-04-02T14:40:58.104 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.103+0000 7f3e58402740 20 client.16656 trim_cache size 0 max 16384 2024-04-02T14:40:58.109 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:40:57 smithi092 ceph-mon[36497]: pgmap v185: 97 pgs: 97 active+clean; 3.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 6.7 MiB/s wr, 683 op/s 2024-04-02T14:40:58.109 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:40:57 smithi092 ceph-mon[36497]: scrub summary: idle 2024-04-02T14:40:58.128 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.127+0000 7f3e58402740 1 Processor -- start 2024-04-02T14:40:58.128 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.127+0000 7f3e58402740 1 -- start start 2024-04-02T14:40:58.128 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.128+0000 7f3e58402740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563a28bb7eb0 0x563a28bc0b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:58.128 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.128+0000 7f3e58402740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563a28bb87d0 0x563a28bccc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:58.129 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.128+0000 7f3e58402740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563a28bc13b0 0x563a28bd91a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:58.129 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.128+0000 7f3e58402740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f3e501bc560 con 0x563a28bb7eb0 2024-04-02T14:40:58.129 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.128+0000 7f3e58402740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f3e501be0a0 con 0x563a28bc13b0 2024-04-02T14:40:58.129 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.128+0000 7f3e58402740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f3e501bcc30 con 0x563a28bb87d0 2024-04-02T14:40:58.130 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.129+0000 7f3e46000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563a28bb7eb0 0x563a28bc0b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:58.130 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.129+0000 7f3e46000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563a28bb7eb0 0x563a28bc0b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:56932/0 (socket says 172.21.15.90:56932) 2024-04-02T14:40:58.130 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.129+0000 7f3e46000640 1 -- 172.21.15.90:0/2500001762 learned_addr learned my addr 172.21.15.90:0/2500001762 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:40:58.130 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.129+0000 7f3e46000640 1 --2- 172.21.15.90:0/2500001762 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563a28bc13b0 0x563a28bd91a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:58.130 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.129+0000 7f3e4ea00640 1 --2- 172.21.15.90:0/2500001762 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563a28bb87d0 0x563a28bccc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:58.130 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.130+0000 7f3e46000640 1 -- 172.21.15.90:0/2500001762 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563a28bb87d0 msgr2=0x563a28bccc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:40:58.131 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.130+0000 7f3e46000640 1 --2- 172.21.15.90:0/2500001762 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563a28bb87d0 0x563a28bccc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:58.131 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.130+0000 7f3e46000640 1 -- 172.21.15.90:0/2500001762 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563a28bb7eb0 msgr2=0x563a28bc0b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:40:58.131 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.130+0000 7f3e46000640 1 --2- 172.21.15.90:0/2500001762 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563a28bb7eb0 0x563a28bc0b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:58.131 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.130+0000 7f3e46000640 1 -- 172.21.15.90:0/2500001762 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563a288fd620 con 0x563a28bc13b0 2024-04-02T14:40:58.131 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.130+0000 7f3e46000640 1 --2- 172.21.15.90:0/2500001762 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563a28bc13b0 0x563a28bd91a0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f3e4000bbd0 tx=0x7f3e40000bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8a61edc41cfc6025 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:40:58.131 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.131+0000 7f3e3f400640 1 -- 172.21.15.90:0/2500001762 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e40005070 con 0x563a28bc13b0 2024-04-02T14:40:58.131 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.131+0000 7f3e3f400640 1 -- 172.21.15.90:0/2500001762 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3e40031070 con 0x563a28bc13b0 2024-04-02T14:40:58.132 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.131+0000 7f3e3f400640 1 -- 172.21.15.90:0/2500001762 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e40021590 con 0x563a28bc13b0 2024-04-02T14:40:58.132 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.132+0000 7f3e58402740 1 -- 172.21.15.90:0/2500001762 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563a28bc13b0 msgr2=0x563a28bd91a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:58.132 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.132+0000 7f3e58402740 1 --2- 172.21.15.90:0/2500001762 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563a28bc13b0 0x563a28bd91a0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f3e4000bbd0 tx=0x7f3e40000bc0 comp rx=0 tx=0).stop 2024-04-02T14:40:58.133 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.132+0000 7f3e58402740 1 -- 172.21.15.90:0/2500001762 shutdown_connections 2024-04-02T14:40:58.133 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.132+0000 7f3e58402740 1 --2- 172.21.15.90:0/2500001762 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563a28bc13b0 0x563a28bd91a0 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:58.133 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.132+0000 7f3e58402740 1 --2- 172.21.15.90:0/2500001762 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563a28bb87d0 0x563a28bccc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:58.133 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.132+0000 7f3e58402740 1 --2- 172.21.15.90:0/2500001762 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563a28bb7eb0 0x563a28bc0b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:58.133 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.132+0000 7f3e58402740 1 -- 172.21.15.90:0/2500001762 >> 172.21.15.90:0/2500001762 conn(0x563a28bb25e0 msgr2=0x563a28bcc570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:40:58.133 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.132+0000 7f3e58402740 1 -- 172.21.15.90:0/2500001762 shutdown_connections 2024-04-02T14:40:58.133 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.132+0000 7f3e58402740 1 -- 172.21.15.90:0/2500001762 wait complete. 2024-04-02T14:40:58.135 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.132+0000 7f3e58402740 1 -- start start 2024-04-02T14:40:58.135 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.133+0000 7f3e58402740 1 Processor -- start 2024-04-02T14:40:58.135 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.133+0000 7f3e58402740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563a28bb7eb0 0x563a28bd9620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:58.135 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.133+0000 7f3e58402740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563a28bb87d0 0x563a28bd69b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:58.135 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.133+0000 7f3e58402740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563a28bc13b0 0x563a28bd6ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:58.135 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.133+0000 7f3e58402740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f3e501bd9d0 con 0x563a28bb7eb0 2024-04-02T14:40:58.135 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.133+0000 7f3e58402740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f3e501bd300 con 0x563a28bb87d0 2024-04-02T14:40:58.135 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.133+0000 7f3e58402740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f3e50071ec0 con 0x563a28bc13b0 2024-04-02T14:40:58.135 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.133+0000 7f3e58402740 10 client.0 _ll_register_callbacks cb 0x7f3e4000bdd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:40:58.135 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.133+0000 7f3e4ea00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563a28bb87d0 0x563a28bd69b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:58.136 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.133+0000 7f3e4ea00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563a28bb87d0 0x563a28bd69b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:53752/0 (socket says 172.21.15.90:53752) 2024-04-02T14:40:58.136 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.133+0000 7f3e4ea00640 1 -- 172.21.15.90:0/2905438294 learned_addr learned my addr 172.21.15.90:0/2905438294 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:40:58.136 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.133+0000 7f3e46000640 1 --2- 172.21.15.90:0/2905438294 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563a28bb7eb0 0x563a28bd9620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:58.136 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.134+0000 7f3e4ea00640 1 -- 172.21.15.90:0/2905438294 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563a28bc13b0 msgr2=0x563a28bd6ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:40:58.136 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.134+0000 7f3e46000640 1 --2- 172.21.15.90:0/2905438294 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563a28bc13b0 0x563a28bd6ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:58.136 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.134+0000 7f3e4ea00640 1 --2- 172.21.15.90:0/2905438294 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563a28bc13b0 0x563a28bd6ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:58.136 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.134+0000 7f3e4ea00640 1 -- 172.21.15.90:0/2905438294 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563a28bb7eb0 msgr2=0x563a28bd9620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:40:58.136 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.134+0000 7f3e4ea00640 1 --2- 172.21.15.90:0/2905438294 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563a28bb7eb0 0x563a28bd9620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:58.136 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.134+0000 7f3e4ea00640 1 -- 172.21.15.90:0/2905438294 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563a28bd75e0 con 0x563a28bb87d0 2024-04-02T14:40:58.136 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.134+0000 7f3e46000640 1 --2- 172.21.15.90:0/2905438294 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563a28bc13b0 0x563a28bd6ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:40:58.136 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.134+0000 7f3e4ea00640 1 --2- 172.21.15.90:0/2905438294 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563a28bb87d0 0x563a28bd69b0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f3e3801c040 tx=0x7f3e3800ffb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eeb95936da3eb5e8 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:40:58.136 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.134+0000 7f3e3cc00640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:40:58.137 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.135+0000 7f3e3cc00640 1 -- 172.21.15.90:0/2905438294 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e3802f060 con 0x563a28bb87d0 2024-04-02T14:40:58.138 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.135+0000 7f3e3cc00640 1 -- 172.21.15.90:0/2905438294 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f3e38009c10 con 0x563a28bb87d0 2024-04-02T14:40:58.138 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.135+0000 7f3e58402740 1 -- 172.21.15.90:0/2905438294 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3e40031070 con 0x563a28bb87d0 2024-04-02T14:40:58.138 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.136+0000 7f3e3cc00640 1 -- 172.21.15.90:0/2905438294 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f3e3801e040 con 0x563a28bb87d0 2024-04-02T14:40:58.138 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.136+0000 7f3e58402740 10 client.27299 fetch_fsmap learned FSMap version 27 2024-04-02T14:40:58.138 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.136+0000 7f3e58402740 1 -- 172.21.15.90:0/2905438294 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x563a28c37b10 con 0x563a28bb87d0 2024-04-02T14:40:58.138 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.137+0000 7f3e3cc00640 1 -- 172.21.15.90:0/2905438294 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3e40031070 con 0x563a28bb87d0 2024-04-02T14:40:58.138 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.137+0000 7f3e3cc00640 1 -- 172.21.15.90:0/2905438294 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f3e38005070 con 0x563a28bb87d0 2024-04-02T14:40:58.138 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.137+0000 7f3e58402740 10 client.27299 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:40:58.138 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.137+0000 7f3e58402740 10 client.27299 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-02T14:40:58.138 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.137+0000 7f3e58402740 20 client.27299 populate_metadata read hostname 'smithi090' 2024-04-02T14:40:58.138 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.137+0000 7f3e58402740 1 --2- 172.21.15.90:0/2905438294 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x563a28c380e0 0x563a28c3a500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:40:58.138 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.137+0000 7f3e58402740 4 client.27299 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-02T14:40:58.138 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.137+0000 7f3e58402740 1 -- 172.21.15.90:0/2905438294 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "flush journal"}) -- 0x563a28b2a2b0 con 0x563a28c380e0 2024-04-02T14:40:58.139 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.138+0000 7f3e46000640 1 --2- 172.21.15.90:0/2905438294 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x563a28c380e0 0x563a28c3a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:40:58.139 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.138+0000 7f3e46000640 1 --2- 172.21.15.90:0/2905438294 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x563a28c380e0 0x563a28c3a500 crc :-1 s=READY pgs=175 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=af11d518bb37aef server_cookie=7da94d5b7587f817 in_seq=0 out_seq=0 2024-04-02T14:40:58.139 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.138+0000 7f3e3cc00640 10 client.27299 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:40:58.159 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.159+0000 7f3e3cc00640 1 -- 172.21.15.90:0/2905438294 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x563a28b2a2b0 con 0x563a28c380e0 2024-04-02T14:40:58.159 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.159+0000 7f3e3cc00640 10 client.27299 handle_command_reply: tid=0 2024-04-02T14:40:58.160 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.159+0000 7f3e58402740 1 client.27299 shutdown 2024-04-02T14:40:58.161 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.159+0000 7f3e58402740 1 -- 172.21.15.90:0/2905438294 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563a28bb87d0 msgr2=0x563a28bd69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:58.162 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.159+0000 7f3e58402740 1 --2- 172.21.15.90:0/2905438294 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563a28bb87d0 0x563a28bd69b0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f3e3801c040 tx=0x7f3e3800ffb0 comp rx=0 tx=0).stop 2024-04-02T14:40:58.162 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.160+0000 7f3e58402740 1 -- 172.21.15.90:0/2905438294 shutdown_connections 2024-04-02T14:40:58.162 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.160+0000 7f3e58402740 1 --2- 172.21.15.90:0/2905438294 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x563a28c380e0 0x563a28c3a500 crc :-1 s=READY pgs=175 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:58.162 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.160+0000 7f3e58402740 1 --2- 172.21.15.90:0/2905438294 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563a28bc13b0 0x563a28bd6ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:58.162 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.160+0000 7f3e3cc00640 0 client.27299 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:40:58.162 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.160+0000 7f3e58402740 1 --2- 172.21.15.90:0/2905438294 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563a28bb87d0 0x563a28bd69b0 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:58.162 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.160+0000 7f3e58402740 1 --2- 172.21.15.90:0/2905438294 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563a28bb7eb0 0x563a28bd9620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:58.162 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.160+0000 7f3e58402740 1 -- 172.21.15.90:0/2905438294 >> 172.21.15.90:0/2905438294 conn(0x563a28bb25e0 msgr2=0x563a28bbc870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:40:58.163 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.160+0000 7f3e58402740 1 -- 172.21.15.90:0/2905438294 shutdown_connections 2024-04-02T14:40:58.163 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.160+0000 7f3e58402740 1 -- 172.21.15.90:0/2905438294 wait complete. 2024-04-02T14:40:58.163 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.160+0000 7f3e58402740 20 client.27299 trim_cache size 0 max 16384 2024-04-02T14:40:58.163 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:{ 2024-04-02T14:40:58.163 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "message": "", 2024-04-02T14:40:58.164 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "return_code": 0 2024-04-02T14:40:58.164 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:} 2024-04-02T14:40:58.165 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.161+0000 7f3e55800640 1 -- 172.21.15.90:0/1904184532 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f3e300a3fa0 msgr2=0x7f3e300a6460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:58.165 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.161+0000 7f3e55800640 1 --2- 172.21.15.90:0/1904184532 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f3e300a3fa0 0x7f3e300a6460 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f3e480ba7f0 tx=0x7f3e480095c0 comp rx=0 tx=0).stop 2024-04-02T14:40:58.165 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.161+0000 7f3e55800640 1 -- 172.21.15.90:0/1904184532 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3e5019e6b0 msgr2=0x7f3e5019eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:40:58.165 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.161+0000 7f3e55800640 1 --2- 172.21.15.90:0/1904184532 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3e5019e6b0 0x7f3e5019eb30 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f3e48037040 tx=0x7f3e48032ea0 comp rx=0 tx=0).stop 2024-04-02T14:40:58.165 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.161+0000 7f3e55800640 1 -- 172.21.15.90:0/1904184532 shutdown_connections 2024-04-02T14:40:58.165 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.161+0000 7f3e55800640 1 --2- 172.21.15.90:0/1904184532 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f3e300a3fa0 0x7f3e300a6460 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:58.165 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.161+0000 7f3e55800640 1 --2- 172.21.15.90:0/1904184532 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3e5019e6b0 0x7f3e5019eb30 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:58.166 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.162+0000 7f3e55800640 1 --2- 172.21.15.90:0/1904184532 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3e5010d410 0x7f3e501a4b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:58.166 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.162+0000 7f3e55800640 1 --2- 172.21.15.90:0/1904184532 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3e50108340 0x7f3e501a45f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:40:58.166 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.162+0000 7f3e55800640 1 -- 172.21.15.90:0/1904184532 >> 172.21.15.90:0/1904184532 conn(0x7f3e50076070 msgr2=0x7f3e50076b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:40:58.166 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.162+0000 7f3e55800640 1 -- 172.21.15.90:0/1904184532 shutdown_connections 2024-04-02T14:40:58.166 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:40:58.162+0000 7f3e55800640 1 -- 172.21.15.90:0/1904184532 wait complete. 2024-04-02T14:40:59.935 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:40:59 smithi177 ceph-mon[36431]: pgmap v186: 97 pgs: 97 active+clean; 3.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 8.5 MiB/s wr, 873 op/s 2024-04-02T14:40:59.937 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:40:59 smithi090 ceph-mon[31674]: pgmap v186: 97 pgs: 97 active+clean; 3.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 8.5 MiB/s wr, 873 op/s 2024-04-02T14:41:00.109 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:40:59 smithi092 ceph-mon[36497]: pgmap v186: 97 pgs: 97 active+clean; 3.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 8.5 MiB/s wr, 873 op/s 2024-04-02T14:41:01.830 INFO:teuthology.orchestra.run.smithi090.stderr:progress: 330.0 s, 1909.4 tps, lat 49.991 ms stddev 82.836 2024-04-02T14:41:01.934 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:41:01 smithi177 ceph-mon[36431]: pgmap v187: 97 pgs: 97 active+clean; 3.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 7.9 MiB/s wr, 812 op/s 2024-04-02T14:41:01.937 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:41:01 smithi090 ceph-mon[31674]: pgmap v187: 97 pgs: 97 active+clean; 3.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 7.9 MiB/s wr, 812 op/s 2024-04-02T14:41:02.109 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:41:01 smithi092 ceph-mon[36497]: pgmap v187: 97 pgs: 97 active+clean; 3.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 7.9 MiB/s wr, 812 op/s 2024-04-02T14:41:03.911 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:41:03 smithi092 ceph-mon[36497]: pgmap v188: 97 pgs: 97 active+clean; 3.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 6.8 MiB/s wr, 697 op/s 2024-04-02T14:41:03.934 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:41:03 smithi177 ceph-mon[36431]: pgmap v188: 97 pgs: 97 active+clean; 3.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 6.8 MiB/s wr, 697 op/s 2024-04-02T14:41:03.936 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:41:03 smithi090 ceph-mon[31674]: pgmap v188: 97 pgs: 97 active+clean; 3.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 6.8 MiB/s wr, 697 op/s 2024-04-02T14:41:05.934 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:41:05 smithi177 ceph-mon[36431]: pgmap v189: 97 pgs: 97 active+clean; 3.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 19 MiB/s wr, 1.17k op/s 2024-04-02T14:41:05.937 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:41:05 smithi090 ceph-mon[31674]: pgmap v189: 97 pgs: 97 active+clean; 3.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 19 MiB/s wr, 1.17k op/s 2024-04-02T14:41:06.109 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:41:05 smithi092 ceph-mon[36497]: pgmap v189: 97 pgs: 97 active+clean; 3.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 19 MiB/s wr, 1.17k op/s 2024-04-02T14:41:06.830 INFO:teuthology.orchestra.run.smithi090.stderr:progress: 335.0 s, 896.2 tps, lat 116.192 ms stddev 367.912 2024-04-02T14:41:07.934 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:41:07 smithi177 ceph-mon[36431]: pgmap v190: 97 pgs: 97 active+clean; 3.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 15 MiB/s wr, 794 op/s 2024-04-02T14:41:07.937 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:41:07 smithi090 ceph-mon[31674]: pgmap v190: 97 pgs: 97 active+clean; 3.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 15 MiB/s wr, 794 op/s 2024-04-02T14:41:08.109 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:41:07 smithi092 ceph-mon[36497]: pgmap v190: 97 pgs: 97 active+clean; 3.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 15 MiB/s wr, 794 op/s 2024-04-02T14:41:09.935 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:41:09 smithi177 ceph-mon[36431]: pgmap v191: 97 pgs: 97 active+clean; 4.0 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 30 MiB/s wr, 1.31k op/s 2024-04-02T14:41:09.935 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:41:09 smithi177 ceph-mon[36431]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T14:41:09.936 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:41:09 smithi090 ceph-mon[31674]: pgmap v191: 97 pgs: 97 active+clean; 4.0 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 30 MiB/s wr, 1.31k op/s 2024-04-02T14:41:09.936 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:41:09 smithi090 ceph-mon[31674]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T14:41:10.109 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:41:09 smithi092 ceph-mon[36497]: pgmap v191: 97 pgs: 97 active+clean; 4.0 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 30 MiB/s wr, 1.31k op/s 2024-04-02T14:41:10.109 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:41:09 smithi092 ceph-mon[36497]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T14:41:11.351 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.351+0000 7f7372a00640 1 -- 172.21.15.90:0/1095488269 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f736c101ab0 msgr2=0x7f736c101f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:41:11.351 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.351+0000 7f7372a00640 1 --2- 172.21.15.90:0/1095488269 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f736c101ab0 0x7f736c101f30 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f7358009900 tx=0x7f735802bc40 comp rx=0 tx=0).stop 2024-04-02T14:41:11.351 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.351+0000 7f7372a00640 1 -- 172.21.15.90:0/1095488269 shutdown_connections 2024-04-02T14:41:11.352 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.351+0000 7f7372a00640 1 --2- 172.21.15.90:0/1095488269 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f736c102470 0x7f736c10e980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:11.352 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.351+0000 7f7372a00640 1 --2- 172.21.15.90:0/1095488269 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f736c101ab0 0x7f736c101f30 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:11.352 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.351+0000 7f7372a00640 1 --2- 172.21.15.90:0/1095488269 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f736c107ac0 0x7f736c107ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:11.352 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.351+0000 7f7372a00640 1 -- 172.21.15.90:0/1095488269 >> 172.21.15.90:0/1095488269 conn(0x7f736c0fd5c0 msgr2=0x7f736c0ff9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:41:11.352 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.351+0000 7f7372a00640 1 -- 172.21.15.90:0/1095488269 shutdown_connections 2024-04-02T14:41:11.352 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.351+0000 7f7372a00640 1 -- 172.21.15.90:0/1095488269 wait complete. 2024-04-02T14:41:11.353 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.352+0000 7f7372a00640 1 Processor -- start 2024-04-02T14:41:11.353 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.352+0000 7f7372a00640 1 -- start start 2024-04-02T14:41:11.353 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.353+0000 7f7372a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f736c101ab0 0x7f736c198330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:11.354 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.353+0000 7f7372a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f736c102470 0x7f736c198870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:11.354 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.353+0000 7f7372a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f736c107ac0 0x7f736c19cba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:11.354 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.353+0000 7f7372a00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f736c111ef0 con 0x7f736c107ac0 2024-04-02T14:41:11.354 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.353+0000 7f7372a00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f736c111d70 con 0x7f736c101ab0 2024-04-02T14:41:11.354 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.353+0000 7f7372a00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f736c112070 con 0x7f736c102470 2024-04-02T14:41:11.355 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.353+0000 7f736be00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f736c101ab0 0x7f736c198330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:11.355 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.354+0000 7f736b400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f736c102470 0x7f736c198870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:11.355 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.354+0000 7f736be00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f736c101ab0 0x7f736c198330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:45618/0 (socket says 172.21.15.90:45618) 2024-04-02T14:41:11.355 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.354+0000 7f736b400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f736c102470 0x7f736c198870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:58252/0 (socket says 172.21.15.90:58252) 2024-04-02T14:41:11.355 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.354+0000 7f736be00640 1 -- 172.21.15.90:0/315606410 learned_addr learned my addr 172.21.15.90:0/315606410 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:41:11.355 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.355+0000 7f736be00640 1 -- 172.21.15.90:0/315606410 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f736c102470 msgr2=0x7f736c198870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:41:11.356 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.355+0000 7f736be00640 1 --2- 172.21.15.90:0/315606410 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f736c102470 0x7f736c198870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:11.356 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.355+0000 7f736be00640 1 -- 172.21.15.90:0/315606410 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f736c107ac0 msgr2=0x7f736c19cba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:41:11.356 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.355+0000 7f736be00640 1 --2- 172.21.15.90:0/315606410 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f736c107ac0 0x7f736c19cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:11.356 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.355+0000 7f736be00640 1 -- 172.21.15.90:0/315606410 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f736c19d2d0 con 0x7f736c101ab0 2024-04-02T14:41:11.356 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.355+0000 7f736b400640 1 --2- 172.21.15.90:0/315606410 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f736c102470 0x7f736c198870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:41:11.357 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.356+0000 7f736be00640 1 --2- 172.21.15.90:0/315606410 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f736c101ab0 0x7f736c198330 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f735c007b20 tx=0x7f735c010050 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:41:11.357 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.356+0000 7f7368c00640 1 -- 172.21.15.90:0/315606410 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f735c010c90 con 0x7f736c101ab0 2024-04-02T14:41:11.357 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.356+0000 7f7372a00640 1 -- 172.21.15.90:0/315606410 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f736c19d5c0 con 0x7f736c101ab0 2024-04-02T14:41:11.357 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.356+0000 7f7368c00640 1 -- 172.21.15.90:0/315606410 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f735c010e30 con 0x7f736c101ab0 2024-04-02T14:41:11.357 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.356+0000 7f7368c00640 1 -- 172.21.15.90:0/315606410 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f735c00c440 con 0x7f736c101ab0 2024-04-02T14:41:11.357 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.357+0000 7f7372a00640 1 -- 172.21.15.90:0/315606410 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f736c073d30 con 0x7f736c101ab0 2024-04-02T14:41:11.360 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.360+0000 7f7368c00640 1 -- 172.21.15.90:0/315606410 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f735c016070 con 0x7f736c101ab0 2024-04-02T14:41:11.360 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.360+0000 7f73757e7740 1 Processor -- start 2024-04-02T14:41:11.361 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.361+0000 7f73757e7740 1 -- start start 2024-04-02T14:41:11.362 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.361+0000 7f73757e7740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e19f82aeb0 0x55e19f82b290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:11.362 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.362+0000 7f73757e7740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e19f82b7d0 0x55e19f833e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:11.362 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.362+0000 7f7370a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e19f82aeb0 0x55e19f82b290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:11.362 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.362+0000 7f7370a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e19f82aeb0 0x55e19f82b290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:58254/0 (socket says 172.21.15.90:58254) 2024-04-02T14:41:11.363 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.362+0000 7f73757e7740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e19f8343b0 0x55e19f8367a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:11.363 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.362+0000 7f73757e7740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55e19f78fd70 con 0x55e19f8343b0 2024-04-02T14:41:11.363 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.362+0000 7f73757e7740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55e19f7382a0 con 0x55e19f82b7d0 2024-04-02T14:41:11.363 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.362+0000 7f73757e7740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55e19f78bd20 con 0x55e19f82aeb0 2024-04-02T14:41:11.364 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.363+0000 7f7370a00640 1 -- 172.21.15.90:0/4256428035 learned_addr learned my addr 172.21.15.90:0/4256428035 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:41:11.364 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.363+0000 7f736be00640 1 --2- 172.21.15.90:0/4256428035 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e19f82b7d0 0x55e19f833e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:11.365 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.363+0000 7f736b400640 1 --2- 172.21.15.90:0/4256428035 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e19f8343b0 0x55e19f8367a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:11.365 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.363+0000 7f736be00640 1 -- 172.21.15.90:0/4256428035 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e19f82aeb0 msgr2=0x55e19f82b290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:41:11.365 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.363+0000 7f736be00640 1 --2- 172.21.15.90:0/4256428035 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e19f82aeb0 0x55e19f82b290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:11.365 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.363+0000 7f736be00640 1 -- 172.21.15.90:0/4256428035 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e19f8343b0 msgr2=0x55e19f8367a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:41:11.365 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.363+0000 7f736be00640 1 --2- 172.21.15.90:0/4256428035 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e19f8343b0 0x55e19f8367a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:11.365 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.363+0000 7f736be00640 1 -- 172.21.15.90:0/4256428035 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e19f74c840 con 0x55e19f82b7d0 2024-04-02T14:41:11.365 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.364+0000 7f736be00640 1 --2- 172.21.15.90:0/4256428035 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e19f82b7d0 0x55e19f833e70 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f735c0ce280 tx=0x7f735c0d4c60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=66ab58225ba29162 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:41:11.365 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.364+0000 7f7361200640 1 -- 172.21.15.90:0/4256428035 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f735c0db840 con 0x55e19f82b7d0 2024-04-02T14:41:11.365 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.364+0000 7f7361200640 1 -- 172.21.15.90:0/4256428035 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f735c0e1070 con 0x55e19f82b7d0 2024-04-02T14:41:11.365 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.364+0000 7f7361200640 1 -- 172.21.15.90:0/4256428035 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f735c0d9cf0 con 0x55e19f82b7d0 2024-04-02T14:41:11.366 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.364+0000 7f7368c00640 1 --2- 172.21.15.90:0/315606410 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f73500a3dc0 0x7f73500a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:11.366 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.365+0000 7f7368c00640 1 -- 172.21.15.90:0/315606410 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f735c0d0b20 con 0x7f736c101ab0 2024-04-02T14:41:11.366 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.365+0000 7f73757e7740 1 -- 172.21.15.90:0/4256428035 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e19f82b7d0 msgr2=0x55e19f833e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:41:11.366 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.365+0000 7f73757e7740 1 --2- 172.21.15.90:0/4256428035 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e19f82b7d0 0x55e19f833e70 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f735c0ce280 tx=0x7f735c0d4c60 comp rx=0 tx=0).stop 2024-04-02T14:41:11.366 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.365+0000 7f73757e7740 1 -- 172.21.15.90:0/4256428035 shutdown_connections 2024-04-02T14:41:11.366 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.365+0000 7f73757e7740 1 --2- 172.21.15.90:0/4256428035 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e19f8343b0 0x55e19f8367a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:11.367 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.365+0000 7f73757e7740 1 --2- 172.21.15.90:0/4256428035 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e19f82b7d0 0x55e19f833e70 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:11.367 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.365+0000 7f73757e7740 1 --2- 172.21.15.90:0/4256428035 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e19f82aeb0 0x55e19f82b290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:11.367 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.366+0000 7f73757e7740 1 -- 172.21.15.90:0/4256428035 >> 172.21.15.90:0/4256428035 conn(0x55e19f8255e0 msgr2=0x55e19f8259c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:41:11.367 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.366+0000 7f7370a00640 1 --2- 172.21.15.90:0/315606410 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f73500a3dc0 0x7f73500a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:11.367 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.366+0000 7f73757e7740 1 -- 172.21.15.90:0/4256428035 shutdown_connections 2024-04-02T14:41:11.367 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.366+0000 7f73757e7740 1 -- 172.21.15.90:0/4256428035 wait complete. 2024-04-02T14:41:11.367 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.366+0000 7f73757e7740 1 -- start start 2024-04-02T14:41:11.367 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.366+0000 7f73757e7740 1 Processor -- start 2024-04-02T14:41:11.368 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.367+0000 7f7370a00640 1 --2- 172.21.15.90:0/315606410 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f73500a3dc0 0x7f73500a6280 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f7364009270 tx=0x7f7364009680 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:41:11.369 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.367+0000 7f73757e7740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e19f82aeb0 0x55e19f8313e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:11.369 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.367+0000 7f73757e7740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e19f82b7d0 0x55e19f831920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:11.370 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.367+0000 7f73757e7740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e19f8343b0 0x55e19f831e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:11.370 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.367+0000 7f73757e7740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55e19f8371a0 con 0x55e19f82aeb0 2024-04-02T14:41:11.370 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.367+0000 7f73757e7740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55e19f74b660 con 0x55e19f8343b0 2024-04-02T14:41:11.370 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.367+0000 7f73757e7740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55e19f837320 con 0x55e19f82b7d0 2024-04-02T14:41:11.370 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.368+0000 7f73757e7740 10 client.0 _ll_register_callbacks cb 0x55e19f74b050 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:41:11.370 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.368+0000 7f7370a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e19f8343b0 0x55e19f831e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:11.370 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.368+0000 7f7370a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e19f8343b0 0x55e19f831e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:45626/0 (socket says 172.21.15.90:45626) 2024-04-02T14:41:11.370 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.368+0000 7f736b400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e19f82b7d0 0x55e19f831920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:11.370 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.369+0000 7f736be00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e19f82aeb0 0x55e19f8313e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:11.370 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.369+0000 7f736be00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e19f82aeb0 0x55e19f8313e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:47020/0 (socket says 172.21.15.90:47020) 2024-04-02T14:41:11.370 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.369+0000 7f736be00640 1 -- 172.21.15.90:0/3787572482 learned_addr learned my addr 172.21.15.90:0/3787572482 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:41:11.371 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.369+0000 7f736be00640 1 -- 172.21.15.90:0/3787572482 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e19f82b7d0 msgr2=0x55e19f831920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:41:11.371 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.369+0000 7f736be00640 1 --2- 172.21.15.90:0/3787572482 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e19f82b7d0 0x55e19f831920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:11.371 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.369+0000 7f736be00640 1 -- 172.21.15.90:0/3787572482 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e19f8343b0 msgr2=0x55e19f831e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:41:11.372 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.370+0000 7f736be00640 1 --2- 172.21.15.90:0/3787572482 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e19f8343b0 0x55e19f831e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:11.372 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.370+0000 7f736be00640 1 -- 172.21.15.90:0/3787572482 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e19f75ae80 con 0x55e19f82aeb0 2024-04-02T14:41:11.372 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.370+0000 7f736b400640 1 --2- 172.21.15.90:0/3787572482 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e19f82b7d0 0x55e19f831920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:41:11.372 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.370+0000 7f7370a00640 1 --2- 172.21.15.90:0/3787572482 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e19f8343b0 0x55e19f831e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:41:11.372 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.370+0000 7f736be00640 1 --2- 172.21.15.90:0/3787572482 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e19f82aeb0 0x55e19f8313e0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f735c09c7d0 tx=0x7f735c010700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=574127ef9049cf5 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:41:11.372 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.370+0000 7f7355600640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-02T14:41:11.372 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.370+0000 7f7355600640 1 -- 172.21.15.90:0/3787572482 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f735c01d070 con 0x55e19f82aeb0 2024-04-02T14:41:11.372 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.370+0000 7f7355600640 1 -- 172.21.15.90:0/3787572482 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f735c09add0 con 0x55e19f82aeb0 2024-04-02T14:41:11.372 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.371+0000 7f73757e7740 1 -- 172.21.15.90:0/3787572482 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f735c0e1070 con 0x55e19f82aeb0 2024-04-02T14:41:11.372 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.371+0000 7f7355600640 1 -- 172.21.15.90:0/3787572482 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f735c098d70 con 0x55e19f82aeb0 2024-04-02T14:41:11.372 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.371+0000 7f73757e7740 10 client.16668 fetch_fsmap learned FSMap version 27 2024-04-02T14:41:11.372 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.371+0000 7f73757e7740 1 -- 172.21.15.90:0/3787572482 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55e19f4a4210 con 0x55e19f82aeb0 2024-04-02T14:41:11.373 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.371+0000 7f7355600640 1 -- 172.21.15.90:0/3787572482 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f735c0e1070 con 0x55e19f82aeb0 2024-04-02T14:41:11.373 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.372+0000 7f7355600640 1 -- 172.21.15.90:0/3787572482 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f735c099140 con 0x55e19f82aeb0 2024-04-02T14:41:11.373 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.372+0000 7f73757e7740 10 client.16668 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:41:11.373 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.372+0000 7f73757e7740 10 client.16668 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-02T14:41:11.373 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.372+0000 7f73757e7740 20 client.16668 populate_metadata read hostname 'smithi090' 2024-04-02T14:41:11.373 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.372+0000 7f73757e7740 1 --2- 172.21.15.90:0/3787572482 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55e19f8555d0 0x55e19f8579c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:11.373 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.372+0000 7f73757e7740 4 client.16668 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:41:11.373 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.372+0000 7f73757e7740 1 -- 172.21.15.90:0/3787572482 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e19f73fcd0 con 0x55e19f8555d0 2024-04-02T14:41:11.375 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.375+0000 7f736b400640 1 --2- 172.21.15.90:0/3787572482 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55e19f8555d0 0x55e19f8579c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:11.382 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.380+0000 7f7355600640 10 client.16668 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:41:11.382 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.380+0000 7f736b400640 1 --2- 172.21.15.90:0/3787572482 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55e19f8555d0 0x55e19f8579c0 crc :-1 s=READY pgs=176 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9bce32410d543526 server_cookie=31a30870c672f9e1 in_seq=0 out_seq=0 2024-04-02T14:41:11.382 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.382+0000 7f7355600640 1 -- 172.21.15.90:0/3787572482 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e19f73fcd0 con 0x55e19f8555d0 2024-04-02T14:41:11.382 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.382+0000 7f7355600640 10 client.16668 handle_command_reply: tid=0 2024-04-02T14:41:11.382 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.382+0000 7f73757e7740 1 client.16668 shutdown 2024-04-02T14:41:11.383 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.382+0000 7f73757e7740 1 -- 172.21.15.90:0/3787572482 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e19f82aeb0 msgr2=0x55e19f8313e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:41:11.383 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.382+0000 7f73757e7740 1 --2- 172.21.15.90:0/3787572482 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e19f82aeb0 0x55e19f8313e0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f735c09c7d0 tx=0x7f735c010700 comp rx=0 tx=0).stop 2024-04-02T14:41:11.383 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.383+0000 7f73757e7740 1 -- 172.21.15.90:0/3787572482 shutdown_connections 2024-04-02T14:41:11.383 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.383+0000 7f73757e7740 1 --2- 172.21.15.90:0/3787572482 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55e19f8555d0 0x55e19f8579c0 crc :-1 s=READY pgs=176 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:11.383 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.383+0000 7f73757e7740 1 --2- 172.21.15.90:0/3787572482 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e19f8343b0 0x55e19f831e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:11.384 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.383+0000 7f73757e7740 1 --2- 172.21.15.90:0/3787572482 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e19f82b7d0 0x55e19f831920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:11.384 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.383+0000 7f73757e7740 1 --2- 172.21.15.90:0/3787572482 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e19f82aeb0 0x55e19f8313e0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:11.384 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.383+0000 7f73757e7740 1 -- 172.21.15.90:0/3787572482 >> 172.21.15.90:0/3787572482 conn(0x55e19f8255e0 msgr2=0x55e19f827330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:41:11.384 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.383+0000 7f7355600640 0 client.16668 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:41:11.384 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.383+0000 7f73757e7740 1 -- 172.21.15.90:0/3787572482 shutdown_connections 2024-04-02T14:41:11.385 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.385+0000 7f73757e7740 1 -- 172.21.15.90:0/3787572482 wait complete. 2024-04-02T14:41:11.385 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.385+0000 7f73757e7740 20 client.16668 trim_cache size 0 max 16384 2024-04-02T14:41:11.420 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.419+0000 7f73757e7740 1 Processor -- start 2024-04-02T14:41:11.420 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.419+0000 7f73757e7740 1 -- start start 2024-04-02T14:41:11.420 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.419+0000 7f73757e7740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e19f82aeb0 0x55e19f833b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:11.420 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.419+0000 7f73757e7740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e19f82b7d0 0x55e19f824640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:11.420 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.419+0000 7f73757e7740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e19f8343b0 0x55e19f85a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:11.420 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.419+0000 7f73757e7740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f736c1ab650 con 0x55e19f82aeb0 2024-04-02T14:41:11.420 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.419+0000 7f73757e7740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f736c1ad190 con 0x55e19f82b7d0 2024-04-02T14:41:11.420 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.419+0000 7f73757e7740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f736c1abd20 con 0x55e19f8343b0 2024-04-02T14:41:11.421 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.420+0000 7f736b400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e19f82b7d0 0x55e19f824640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:11.422 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.420+0000 7f736b400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e19f82b7d0 0x55e19f824640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:45638/0 (socket says 172.21.15.90:45638) 2024-04-02T14:41:11.422 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.420+0000 7f736b400640 1 -- 172.21.15.90:0/2437804805 learned_addr learned my addr 172.21.15.90:0/2437804805 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:41:11.422 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.420+0000 7f7370a00640 1 --2- 172.21.15.90:0/2437804805 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e19f8343b0 0x55e19f85a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:11.422 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.421+0000 7f736b400640 1 -- 172.21.15.90:0/2437804805 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e19f8343b0 msgr2=0x55e19f85a9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:41:11.422 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.421+0000 7f736b400640 1 --2- 172.21.15.90:0/2437804805 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e19f8343b0 0x55e19f85a9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:11.422 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.421+0000 7f736b400640 1 -- 172.21.15.90:0/2437804805 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e19f82aeb0 msgr2=0x55e19f833b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:41:11.422 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.421+0000 7f736b400640 1 --2- 172.21.15.90:0/2437804805 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e19f82aeb0 0x55e19f833b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:11.422 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.421+0000 7f736b400640 1 -- 172.21.15.90:0/2437804805 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e19f830410 con 0x55e19f82b7d0 2024-04-02T14:41:11.422 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.421+0000 7f736b400640 1 --2- 172.21.15.90:0/2437804805 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e19f82b7d0 0x55e19f824640 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f7358009850 tx=0x7f7358043040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dd9e0d7570967847 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:41:11.423 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.422+0000 7f7357e00640 1 -- 172.21.15.90:0/2437804805 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f735804f070 con 0x55e19f82b7d0 2024-04-02T14:41:11.424 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.422+0000 7f7357e00640 1 -- 172.21.15.90:0/2437804805 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f73580044a0 con 0x55e19f82b7d0 2024-04-02T14:41:11.424 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.422+0000 7f7357e00640 1 -- 172.21.15.90:0/2437804805 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f735804b5a0 con 0x55e19f82b7d0 2024-04-02T14:41:11.424 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.423+0000 7f73757e7740 1 -- 172.21.15.90:0/2437804805 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e19f82b7d0 msgr2=0x55e19f824640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:41:11.424 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.423+0000 7f73757e7740 1 --2- 172.21.15.90:0/2437804805 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e19f82b7d0 0x55e19f824640 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f7358009850 tx=0x7f7358043040 comp rx=0 tx=0).stop 2024-04-02T14:41:11.425 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.423+0000 7f73757e7740 1 -- 172.21.15.90:0/2437804805 shutdown_connections 2024-04-02T14:41:11.425 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.423+0000 7f73757e7740 1 --2- 172.21.15.90:0/2437804805 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e19f8343b0 0x55e19f85a9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:11.425 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.423+0000 7f73757e7740 1 --2- 172.21.15.90:0/2437804805 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e19f82b7d0 0x55e19f824640 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:11.425 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.423+0000 7f73757e7740 1 --2- 172.21.15.90:0/2437804805 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e19f82aeb0 0x55e19f833b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:11.425 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.423+0000 7f73757e7740 1 -- 172.21.15.90:0/2437804805 >> 172.21.15.90:0/2437804805 conn(0x55e19f8255e0 msgr2=0x55e19f8259c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:41:11.425 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.423+0000 7f73757e7740 1 -- 172.21.15.90:0/2437804805 shutdown_connections 2024-04-02T14:41:11.425 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.423+0000 7f73757e7740 1 -- 172.21.15.90:0/2437804805 wait complete. 2024-04-02T14:41:11.426 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.424+0000 7f73757e7740 1 -- start start 2024-04-02T14:41:11.426 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.424+0000 7f73757e7740 1 Processor -- start 2024-04-02T14:41:11.426 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.425+0000 7f73757e7740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e19f82aeb0 0x55e19f858c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:11.426 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.425+0000 7f73757e7740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e19f82b7d0 0x55e19f859160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:11.427 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.425+0000 7f73757e7740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e19f8343b0 0x55e19f8ba7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:11.427 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.425+0000 7f73757e7740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f736c1acac0 con 0x55e19f82b7d0 2024-04-02T14:41:11.427 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.425+0000 7f73757e7740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f736c1ac3f0 con 0x55e19f82aeb0 2024-04-02T14:41:11.427 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.425+0000 7f73757e7740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f736c1a95f0 con 0x55e19f8343b0 2024-04-02T14:41:11.427 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.425+0000 7f73757e7740 10 client.0 _ll_register_callbacks cb 0x7f735c0d4a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:41:11.427 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.425+0000 7f7370a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e19f8343b0 0x55e19f8ba7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:11.427 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.425+0000 7f7370a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e19f8343b0 0x55e19f8ba7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:58282/0 (socket says 172.21.15.90:58282) 2024-04-02T14:41:11.427 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.425+0000 7f7370a00640 1 -- 172.21.15.90:0/2109450686 learned_addr learned my addr 172.21.15.90:0/2109450686 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:41:11.427 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.426+0000 7f736be00640 1 --2- 172.21.15.90:0/2109450686 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e19f82aeb0 0x55e19f858c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:11.427 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.427+0000 7f7370a00640 1 -- 172.21.15.90:0/2109450686 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e19f82aeb0 msgr2=0x55e19f858c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:41:11.428 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.427+0000 7f736b400640 1 --2- 172.21.15.90:0/2109450686 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e19f82b7d0 0x55e19f859160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:11.428 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.427+0000 7f7370a00640 1 --2- 172.21.15.90:0/2109450686 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e19f82aeb0 0x55e19f858c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:11.428 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.427+0000 7f7370a00640 1 -- 172.21.15.90:0/2109450686 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e19f82b7d0 msgr2=0x55e19f859160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:41:11.428 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.427+0000 7f7370a00640 1 --2- 172.21.15.90:0/2109450686 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e19f82b7d0 0x55e19f859160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:11.428 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.427+0000 7f7370a00640 1 -- 172.21.15.90:0/2109450686 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e19f570620 con 0x55e19f8343b0 2024-04-02T14:41:11.428 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.427+0000 7f736be00640 1 --2- 172.21.15.90:0/2109450686 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e19f82aeb0 0x55e19f858c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:41:11.429 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.428+0000 7f736b400640 1 --2- 172.21.15.90:0/2109450686 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e19f82b7d0 0x55e19f859160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:41:11.429 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.428+0000 7f7370a00640 1 --2- 172.21.15.90:0/2109450686 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e19f8343b0 0x55e19f8ba7b0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f7364005c40 tx=0x7f7364004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=56e9d21ec61df9b6 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:41:11.429 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.428+0000 7f7355600640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:41:11.429 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.428+0000 7f7355600640 1 -- 172.21.15.90:0/2109450686 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7364004b20 con 0x55e19f8343b0 2024-04-02T14:41:11.429 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.428+0000 7f7355600640 1 -- 172.21.15.90:0/2109450686 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f7364004cc0 con 0x55e19f8343b0 2024-04-02T14:41:11.429 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.429+0000 7f73757e7740 1 -- 172.21.15.90:0/2109450686 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f73580044a0 con 0x55e19f8343b0 2024-04-02T14:41:11.430 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.430+0000 7f7355600640 1 -- 172.21.15.90:0/2109450686 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f736401f6f0 con 0x55e19f8343b0 2024-04-02T14:41:11.431 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.431+0000 7f73757e7740 10 client.27184 fetch_fsmap learned FSMap version 27 2024-04-02T14:41:11.431 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.431+0000 7f73757e7740 1 -- 172.21.15.90:0/2109450686 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55e19f8bbf90 con 0x55e19f8343b0 2024-04-02T14:41:11.431 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.431+0000 7f7355600640 1 -- 172.21.15.90:0/2109450686 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f73580044a0 con 0x55e19f8343b0 2024-04-02T14:41:11.432 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.432+0000 7f7355600640 1 -- 172.21.15.90:0/2109450686 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f736400f040 con 0x55e19f8343b0 2024-04-02T14:41:11.432 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.432+0000 7f73757e7740 10 client.27184 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:41:11.432 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.432+0000 7f73757e7740 10 client.27184 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-02T14:41:11.432 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.432+0000 7f73757e7740 20 client.27184 populate_metadata read hostname 'smithi090' 2024-04-02T14:41:11.432 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.432+0000 7f73757e7740 1 --2- 172.21.15.90:0/2109450686 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55e19f8bc6c0 0x55e19f8beb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:11.432 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.432+0000 7f73757e7740 4 client.27184 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-02T14:41:11.432 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.432+0000 7f73757e7740 1 -- 172.21.15.90:0/2109450686 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55e19f79d2b0 con 0x55e19f8bc6c0 2024-04-02T14:41:11.433 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.433+0000 7f736be00640 1 --2- 172.21.15.90:0/2109450686 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55e19f8bc6c0 0x55e19f8beb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:11.434 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.434+0000 7f736be00640 1 --2- 172.21.15.90:0/2109450686 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55e19f8bc6c0 0x55e19f8beb40 crc :-1 s=READY pgs=177 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a914b93d7acabee7 server_cookie=8bde4110a98c9b48 in_seq=0 out_seq=0 2024-04-02T14:41:11.434 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.434+0000 7f7355600640 10 client.27184 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:41:11.462 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.461+0000 7f7355600640 1 -- 172.21.15.90:0/2109450686 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55e19f79d2b0 con 0x55e19f8bc6c0 2024-04-02T14:41:11.462 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.461+0000 7f7355600640 10 client.27184 handle_command_reply: tid=0 2024-04-02T14:41:11.462 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.462+0000 7f73757e7740 1 client.27184 shutdown 2024-04-02T14:41:11.462 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.462+0000 7f73757e7740 1 -- 172.21.15.90:0/2109450686 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e19f8343b0 msgr2=0x55e19f8ba7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:41:11.463 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.462+0000 7f73757e7740 1 --2- 172.21.15.90:0/2109450686 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e19f8343b0 0x55e19f8ba7b0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f7364005c40 tx=0x7f7364004a40 comp rx=0 tx=0).stop 2024-04-02T14:41:11.463 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.462+0000 7f73757e7740 1 -- 172.21.15.90:0/2109450686 shutdown_connections 2024-04-02T14:41:11.463 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.462+0000 7f73757e7740 1 --2- 172.21.15.90:0/2109450686 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55e19f8bc6c0 0x55e19f8beb40 crc :-1 s=READY pgs=177 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:11.463 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.462+0000 7f73757e7740 1 --2- 172.21.15.90:0/2109450686 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e19f8343b0 0x55e19f8ba7b0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:11.463 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.462+0000 7f73757e7740 1 --2- 172.21.15.90:0/2109450686 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e19f82b7d0 0x55e19f859160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:11.463 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.462+0000 7f73757e7740 1 --2- 172.21.15.90:0/2109450686 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e19f82aeb0 0x55e19f858c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:11.463 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.462+0000 7f73757e7740 1 -- 172.21.15.90:0/2109450686 >> 172.21.15.90:0/2109450686 conn(0x55e19f8255e0 msgr2=0x55e19f83fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:41:11.463 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.462+0000 7f7355600640 0 client.27184 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:41:11.464 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.462+0000 7f73757e7740 1 -- 172.21.15.90:0/2109450686 shutdown_connections 2024-04-02T14:41:11.464 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.463+0000 7f73757e7740 1 -- 172.21.15.90:0/2109450686 wait complete. 2024-04-02T14:41:11.464 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.463+0000 7f73757e7740 20 client.27184 trim_cache size 0 max 16384 2024-04-02T14:41:11.464 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:{ 2024-04-02T14:41:11.464 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "message": "", 2024-04-02T14:41:11.464 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "return_code": 0 2024-04-02T14:41:11.464 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:} 2024-04-02T14:41:11.465 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.463+0000 7f7372a00640 1 -- 172.21.15.90:0/315606410 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f73500a3dc0 msgr2=0x7f73500a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:41:11.465 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.463+0000 7f7372a00640 1 --2- 172.21.15.90:0/315606410 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f73500a3dc0 0x7f73500a6280 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f7364009270 tx=0x7f7364009680 comp rx=0 tx=0).stop 2024-04-02T14:41:11.465 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.463+0000 7f7372a00640 1 -- 172.21.15.90:0/315606410 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f736c101ab0 msgr2=0x7f736c198330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:41:11.465 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.463+0000 7f7372a00640 1 --2- 172.21.15.90:0/315606410 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f736c101ab0 0x7f736c198330 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f735c007b20 tx=0x7f735c010050 comp rx=0 tx=0).stop 2024-04-02T14:41:11.465 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.464+0000 7f7372a00640 1 -- 172.21.15.90:0/315606410 shutdown_connections 2024-04-02T14:41:11.465 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.464+0000 7f7372a00640 1 --2- 172.21.15.90:0/315606410 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f73500a3dc0 0x7f73500a6280 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:11.466 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.464+0000 7f7372a00640 1 --2- 172.21.15.90:0/315606410 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f736c107ac0 0x7f736c19cba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:11.466 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.464+0000 7f7372a00640 1 --2- 172.21.15.90:0/315606410 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f736c102470 0x7f736c198870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:11.466 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.464+0000 7f7372a00640 1 --2- 172.21.15.90:0/315606410 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f736c101ab0 0x7f736c198330 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:11.466 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.464+0000 7f7372a00640 1 -- 172.21.15.90:0/315606410 >> 172.21.15.90:0/315606410 conn(0x7f736c0fd5c0 msgr2=0x7f736c0fdae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:41:11.466 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.464+0000 7f7372a00640 1 -- 172.21.15.90:0/315606410 shutdown_connections 2024-04-02T14:41:11.466 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:11.464+0000 7f7372a00640 1 -- 172.21.15.90:0/315606410 wait complete. 2024-04-02T14:41:11.830 INFO:teuthology.orchestra.run.smithi090.stderr:progress: 340.0 s, 2741.8 tps, lat 36.315 ms stddev 46.298 2024-04-02T14:41:11.935 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:41:11 smithi177 ceph-mon[36431]: pgmap v192: 97 pgs: 97 active+clean; 4.0 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 36 MiB/s wr, 1.44k op/s 2024-04-02T14:41:11.935 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:41:11 smithi177 ceph-mon[36431]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:41:11.935 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:41:11 smithi177 ceph-mon[36431]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T14:41:11.935 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:41:11 smithi177 ceph-mon[36431]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' 2024-04-02T14:41:11.936 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:41:11 smithi090 ceph-mon[31674]: pgmap v192: 97 pgs: 97 active+clean; 4.0 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 36 MiB/s wr, 1.44k op/s 2024-04-02T14:41:11.936 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:41:11 smithi090 ceph-mon[31674]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:41:11.936 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:41:11 smithi090 ceph-mon[31674]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T14:41:11.937 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:41:11 smithi090 ceph-mon[31674]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' 2024-04-02T14:41:12.109 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:41:11 smithi092 ceph-mon[36497]: pgmap v192: 97 pgs: 97 active+clean; 4.0 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 36 MiB/s wr, 1.44k op/s 2024-04-02T14:41:12.109 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:41:11 smithi092 ceph-mon[36497]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:41:12.109 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:41:11 smithi092 ceph-mon[36497]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T14:41:12.109 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:41:11 smithi092 ceph-mon[36497]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' 2024-04-02T14:41:13.934 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:41:13 smithi177 ceph-mon[36431]: pgmap v193: 97 pgs: 97 active+clean; 4.0 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 34 MiB/s wr, 1.30k op/s 2024-04-02T14:41:13.936 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:41:13 smithi090 ceph-mon[31674]: pgmap v193: 97 pgs: 97 active+clean; 4.0 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 34 MiB/s wr, 1.30k op/s 2024-04-02T14:41:14.109 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:41:13 smithi092 ceph-mon[36497]: pgmap v193: 97 pgs: 97 active+clean; 4.0 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 34 MiB/s wr, 1.30k op/s 2024-04-02T14:41:15.934 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:41:15 smithi177 ceph-mon[36431]: pgmap v194: 97 pgs: 97 active+clean; 4.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 38 MiB/s wr, 1.64k op/s 2024-04-02T14:41:15.936 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:41:15 smithi090 ceph-mon[31674]: pgmap v194: 97 pgs: 97 active+clean; 4.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 38 MiB/s wr, 1.64k op/s 2024-04-02T14:41:16.109 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:41:15 smithi092 ceph-mon[36497]: pgmap v194: 97 pgs: 97 active+clean; 4.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 38 MiB/s wr, 1.64k op/s 2024-04-02T14:41:16.830 INFO:teuthology.orchestra.run.smithi090.stderr:progress: 345.0 s, 2837.6 tps, lat 35.347 ms stddev 44.602 2024-04-02T14:41:17.934 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:41:17 smithi177 ceph-mon[36431]: pgmap v195: 97 pgs: 97 active+clean; 4.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 26 MiB/s wr, 1.17k op/s 2024-04-02T14:41:17.936 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:41:17 smithi090 ceph-mon[31674]: pgmap v195: 97 pgs: 97 active+clean; 4.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 26 MiB/s wr, 1.17k op/s 2024-04-02T14:41:18.110 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:41:17 smithi092 ceph-mon[36497]: pgmap v195: 97 pgs: 97 active+clean; 4.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 26 MiB/s wr, 1.17k op/s 2024-04-02T14:41:20.109 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:41:19 smithi092 ceph-mon[36497]: pgmap v196: 97 pgs: 97 active+clean; 4.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 28 MiB/s wr, 1.38k op/s 2024-04-02T14:41:20.184 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:41:19 smithi177 ceph-mon[36431]: pgmap v196: 97 pgs: 97 active+clean; 4.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 28 MiB/s wr, 1.38k op/s 2024-04-02T14:41:20.186 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:41:19 smithi090 ceph-mon[31674]: pgmap v196: 97 pgs: 97 active+clean; 4.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 28 MiB/s wr, 1.38k op/s 2024-04-02T14:41:21.829 INFO:teuthology.orchestra.run.smithi090.stderr:progress: 350.0 s, 2325.0 tps, lat 43.096 ms stddev 50.497 2024-04-02T14:41:22.005 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-02T14:41:22.006 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-02T14:41:22.088 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:41:21 smithi090 ceph-mon[31674]: pgmap v197: 97 pgs: 97 active+clean; 4.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 14 MiB/s wr, 953 op/s 2024-04-02T14:41:22.108 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:41:21 smithi092 ceph-mon[36497]: pgmap v197: 97 pgs: 97 active+clean; 4.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 14 MiB/s wr, 953 op/s 2024-04-02T14:41:22.184 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:41:21 smithi177 ceph-mon[36431]: pgmap v197: 97 pgs: 97 active+clean; 4.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 14 MiB/s wr, 953 op/s 2024-04-02T14:41:22.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.278+0000 7f6dfd800640 1 -- 172.21.15.90:0/165107142 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6df81048c0 msgr2=0x7f6df8108900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:41:22.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.278+0000 7f6dfd800640 1 --2- 172.21.15.90:0/165107142 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6df81048c0 0x7f6df8108900 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f6de4013150 tx=0x7f6de4033fc0 comp rx=0 tx=0).stop 2024-04-02T14:41:22.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.278+0000 7f6dfd800640 1 -- 172.21.15.90:0/165107142 shutdown_connections 2024-04-02T14:41:22.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.278+0000 7f6dfd800640 1 --2- 172.21.15.90:0/165107142 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6df8108fe0 0x7f6df8110bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:22.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.278+0000 7f6dfd800640 1 --2- 172.21.15.90:0/165107142 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6df81048c0 0x7f6df8108900 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:22.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.278+0000 7f6dfd800640 1 --2- 172.21.15.90:0/165107142 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6df8103f10 0x7f6df81042f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:22.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.278+0000 7f6dfd800640 1 -- 172.21.15.90:0/165107142 >> 172.21.15.90:0/165107142 conn(0x7f6df80ff7b0 msgr2=0x7f6df8101bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:41:22.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.278+0000 7f6dfd800640 1 -- 172.21.15.90:0/165107142 shutdown_connections 2024-04-02T14:41:22.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.278+0000 7f6dfd800640 1 -- 172.21.15.90:0/165107142 wait complete. 2024-04-02T14:41:22.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.279+0000 7f6dfd800640 1 Processor -- start 2024-04-02T14:41:22.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.280+0000 7f6dfd800640 1 -- start start 2024-04-02T14:41:22.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.280+0000 7f6dfd800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6df8103f10 0x7f6df81965f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:22.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.280+0000 7f6dfd800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6df8108fe0 0x7f6df8196b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:22.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.280+0000 7f6dfd800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6df819aaa0 0x7f6df819af50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:22.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.280+0000 7f6dfd800640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f6df80788d0 con 0x7f6df8108fe0 2024-04-02T14:41:22.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.280+0000 7f6dfd800640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f6df8078750 con 0x7f6df8103f10 2024-04-02T14:41:22.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.280+0000 7f6dfd800640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f6df8078a50 con 0x7f6df819aaa0 2024-04-02T14:41:22.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.280+0000 7f6df6a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6df8103f10 0x7f6df81965f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:22.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.281+0000 7f6df6a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6df8103f10 0x7f6df81965f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:57530/0 (socket says 172.21.15.90:57530) 2024-04-02T14:41:22.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.281+0000 7f6df6a00640 1 -- 172.21.15.90:0/869176656 learned_addr learned my addr 172.21.15.90:0/869176656 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:41:22.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.281+0000 7f6df7400640 1 --2- 172.21.15.90:0/869176656 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6df819aaa0 0x7f6df819af50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:22.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.281+0000 7f6df6a00640 1 -- 172.21.15.90:0/869176656 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6df819aaa0 msgr2=0x7f6df819af50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:41:22.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.281+0000 7f6df6a00640 1 --2- 172.21.15.90:0/869176656 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6df819aaa0 0x7f6df819af50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:22.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.281+0000 7f6df6000640 1 --2- 172.21.15.90:0/869176656 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6df8108fe0 0x7f6df8196b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:22.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.281+0000 7f6df6a00640 1 -- 172.21.15.90:0/869176656 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6df8108fe0 msgr2=0x7f6df8196b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:41:22.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.281+0000 7f6df6a00640 1 --2- 172.21.15.90:0/869176656 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6df8108fe0 0x7f6df8196b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:22.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.281+0000 7f6df6a00640 1 -- 172.21.15.90:0/869176656 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6df819b590 con 0x7f6df8103f10 2024-04-02T14:41:22.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.282+0000 7f6df6000640 1 --2- 172.21.15.90:0/869176656 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6df8108fe0 0x7f6df8196b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:41:22.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.282+0000 7f6df6a00640 1 --2- 172.21.15.90:0/869176656 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6df8103f10 0x7f6df81965f0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f6de800d950 tx=0x7f6de800de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:41:22.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.282+0000 7f6def400640 1 -- 172.21.15.90:0/869176656 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6de8017070 con 0x7f6df8103f10 2024-04-02T14:41:22.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.282+0000 7f6def400640 1 -- 172.21.15.90:0/869176656 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6de8009c30 con 0x7f6df8103f10 2024-04-02T14:41:22.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.282+0000 7f6def400640 1 -- 172.21.15.90:0/869176656 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6de800f690 con 0x7f6df8103f10 2024-04-02T14:41:22.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.282+0000 7f6dfd800640 1 -- 172.21.15.90:0/869176656 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6df819b880 con 0x7f6df8103f10 2024-04-02T14:41:22.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.282+0000 7f6dfd800640 1 -- 172.21.15.90:0/869176656 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6df81a3020 con 0x7f6df8103f10 2024-04-02T14:41:22.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.286+0000 7f6e00587740 1 Processor -- start 2024-04-02T14:41:22.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.286+0000 7f6e00587740 1 -- start start 2024-04-02T14:41:22.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.287+0000 7f6e00587740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e887d7d7b0 0x55e887d7db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:22.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.287+0000 7f6e00587740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e887d7e0d0 0x55e887d86770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:22.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.287+0000 7f6e00587740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e887d86cb0 0x55e887d890a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:22.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.287+0000 7f6e00587740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55e887ce1770 con 0x55e887d7d7b0 2024-04-02T14:41:22.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.287+0000 7f6e00587740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55e887c89700 con 0x55e887d86cb0 2024-04-02T14:41:22.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.287+0000 7f6e00587740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55e887cdd720 con 0x55e887d7e0d0 2024-04-02T14:41:22.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.287+0000 7f6df7400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e887d7d7b0 0x55e887d7db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:22.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.287+0000 7f6df7400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e887d7d7b0 0x55e887d7db90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:59148/0 (socket says 172.21.15.90:59148) 2024-04-02T14:41:22.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.287+0000 7f6df7400640 1 -- 172.21.15.90:0/4184346407 learned_addr learned my addr 172.21.15.90:0/4184346407 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:41:22.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.288+0000 7f6df6000640 1 --2- 172.21.15.90:0/4184346407 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e887d86cb0 0x55e887d890a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:22.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.288+0000 7f6df7400640 1 -- 172.21.15.90:0/4184346407 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e887d7e0d0 msgr2=0x55e887d86770 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-02T14:41:22.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.288+0000 7f6def400640 1 -- 172.21.15.90:0/869176656 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f6de80109d0 con 0x7f6df8103f10 2024-04-02T14:41:22.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.290+0000 7f6def400640 1 --2- 172.21.15.90:0/869176656 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f6ddc0a3da0 0x7f6ddc0a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:22.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.290+0000 7f6def400640 1 -- 172.21.15.90:0/869176656 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f6de80d0110 con 0x7f6df8103f10 2024-04-02T14:41:22.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.289+0000 7f6df6a00640 1 --2- 172.21.15.90:0/4184346407 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e887d7e0d0 0x55e887d86770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:22.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.291+0000 7f6df7400640 1 --2- 172.21.15.90:0/4184346407 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e887d7e0d0 0x55e887d86770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:22.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.291+0000 7f6df7400640 1 -- 172.21.15.90:0/4184346407 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e887d86cb0 msgr2=0x55e887d890a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:41:22.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.291+0000 7f6df7400640 1 --2- 172.21.15.90:0/4184346407 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e887d86cb0 0x55e887d890a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:22.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.291+0000 7f6df7400640 1 -- 172.21.15.90:0/4184346407 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e887abf7d0 con 0x55e887d7d7b0 2024-04-02T14:41:22.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.291+0000 7f6df6000640 1 --2- 172.21.15.90:0/4184346407 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e887d86cb0 0x55e887d890a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:41:22.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.291+0000 7f6df6a00640 1 --2- 172.21.15.90:0/4184346407 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e887d7e0d0 0x55e887d86770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:41:22.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.292+0000 7f6df7400640 1 --2- 172.21.15.90:0/4184346407 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e887d7d7b0 0x55e887d7db90 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f6df0009e60 tx=0x7f6df0010740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f600f3a998fa7d5 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:41:22.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.292+0000 7f6df7400640 1 --2- 172.21.15.90:0/869176656 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f6ddc0a3da0 0x7f6ddc0a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:22.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.292+0000 7f6de3e00640 1 -- 172.21.15.90:0/4184346407 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6df0016070 con 0x55e887d7d7b0 2024-04-02T14:41:22.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.292+0000 7f6de3e00640 1 -- 172.21.15.90:0/4184346407 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6df0012bd0 con 0x55e887d7d7b0 2024-04-02T14:41:22.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.292+0000 7f6de3e00640 1 -- 172.21.15.90:0/4184346407 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6df00113e0 con 0x55e887d7d7b0 2024-04-02T14:41:22.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.295+0000 7f6df7400640 1 --2- 172.21.15.90:0/869176656 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f6ddc0a3da0 0x7f6ddc0a6260 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f6df0012fd0 tx=0x7f6df001af70 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:41:22.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.295+0000 7f6e00587740 1 -- 172.21.15.90:0/4184346407 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e887d7d7b0 msgr2=0x55e887d7db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:41:22.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.295+0000 7f6e00587740 1 --2- 172.21.15.90:0/4184346407 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e887d7d7b0 0x55e887d7db90 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f6df0009e60 tx=0x7f6df0010740 comp rx=0 tx=0).stop 2024-04-02T14:41:22.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.295+0000 7f6e00587740 1 -- 172.21.15.90:0/4184346407 shutdown_connections 2024-04-02T14:41:22.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.295+0000 7f6e00587740 1 --2- 172.21.15.90:0/4184346407 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e887d86cb0 0x55e887d890a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:22.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.295+0000 7f6e00587740 1 --2- 172.21.15.90:0/4184346407 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e887d7e0d0 0x55e887d86770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:22.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.295+0000 7f6e00587740 1 --2- 172.21.15.90:0/4184346407 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e887d7d7b0 0x55e887d7db90 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:22.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.295+0000 7f6e00587740 1 -- 172.21.15.90:0/4184346407 >> 172.21.15.90:0/4184346407 conn(0x55e887d77ee0 msgr2=0x55e887d782c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:41:22.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.295+0000 7f6e00587740 1 -- 172.21.15.90:0/4184346407 shutdown_connections 2024-04-02T14:41:22.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.295+0000 7f6e00587740 1 -- 172.21.15.90:0/4184346407 wait complete. 2024-04-02T14:41:22.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.296+0000 7f6e00587740 1 -- start start 2024-04-02T14:41:22.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.296+0000 7f6e00587740 1 Processor -- start 2024-04-02T14:41:22.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.297+0000 7f6e00587740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e887d7d7b0 0x55e887d83d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:22.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.297+0000 7f6e00587740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e887d7e0d0 0x55e887d842d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:22.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.297+0000 7f6e00587740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e887d86cb0 0x55e887d84810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:22.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.297+0000 7f6e00587740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55e887d89aa0 con 0x55e887d7e0d0 2024-04-02T14:41:22.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.297+0000 7f6e00587740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55e887c9cac0 con 0x55e887d7d7b0 2024-04-02T14:41:22.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.297+0000 7f6e00587740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55e887d89c20 con 0x55e887d86cb0 2024-04-02T14:41:22.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.297+0000 7f6e00587740 10 client.0 _ll_register_callbacks cb 0x55e887c9f180 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:41:22.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.297+0000 7f6df6000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e887d7e0d0 0x55e887d842d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:22.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.297+0000 7f6df6000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e887d7e0d0 0x55e887d842d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:59162/0 (socket says 172.21.15.90:59162) 2024-04-02T14:41:22.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.297+0000 7f6df6000640 1 -- 172.21.15.90:0/2221839291 learned_addr learned my addr 172.21.15.90:0/2221839291 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:41:22.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.297+0000 7f6df6a00640 1 --2- 172.21.15.90:0/2221839291 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e887d7d7b0 0x55e887d83d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:22.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.298+0000 7f6df7400640 1 --2- 172.21.15.90:0/2221839291 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e887d86cb0 0x55e887d84810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:22.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.298+0000 7f6df6000640 1 -- 172.21.15.90:0/2221839291 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e887d86cb0 msgr2=0x55e887d84810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:41:22.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.298+0000 7f6df6000640 1 --2- 172.21.15.90:0/2221839291 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e887d86cb0 0x55e887d84810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:22.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.298+0000 7f6df6000640 1 -- 172.21.15.90:0/2221839291 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e887d7d7b0 msgr2=0x55e887d83d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:41:22.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.298+0000 7f6df6000640 1 --2- 172.21.15.90:0/2221839291 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e887d7d7b0 0x55e887d83d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:22.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.298+0000 7f6df6000640 1 -- 172.21.15.90:0/2221839291 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e8879f53f0 con 0x55e887d7e0d0 2024-04-02T14:41:22.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.298+0000 7f6df6a00640 1 --2- 172.21.15.90:0/2221839291 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e887d7d7b0 0x55e887d83d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:41:22.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.298+0000 7f6df6000640 1 --2- 172.21.15.90:0/2221839291 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e887d7e0d0 0x55e887d842d0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f6de4034500 tx=0x7f6de4005de0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9beb50d0fc62bae4 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:41:22.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.298+0000 7f6de0c00640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-02T14:41:22.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.299+0000 7f6de0c00640 1 -- 172.21.15.90:0/2221839291 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6de404b070 con 0x55e887d7e0d0 2024-04-02T14:41:22.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.299+0000 7f6e00587740 1 -- 172.21.15.90:0/2221839291 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6df0012bd0 con 0x55e887d7e0d0 2024-04-02T14:41:22.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.299+0000 7f6de0c00640 1 -- 172.21.15.90:0/2221839291 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f6de4034960 con 0x55e887d7e0d0 2024-04-02T14:41:22.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.300+0000 7f6de0c00640 1 -- 172.21.15.90:0/2221839291 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f6de4034b60 con 0x55e887d7e0d0 2024-04-02T14:41:22.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.300+0000 7f6e00587740 10 client.16677 fetch_fsmap learned FSMap version 27 2024-04-02T14:41:22.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.300+0000 7f6e00587740 1 -- 172.21.15.90:0/2221839291 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55e8879b50f0 con 0x55e887d7e0d0 2024-04-02T14:41:22.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.301+0000 7f6de0c00640 1 -- 172.21.15.90:0/2221839291 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6df0012bd0 con 0x55e887d7e0d0 2024-04-02T14:41:22.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.301+0000 7f6de0c00640 1 -- 172.21.15.90:0/2221839291 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f6de4003810 con 0x55e887d7e0d0 2024-04-02T14:41:22.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.301+0000 7f6e00587740 10 client.16677 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:41:22.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.301+0000 7f6e00587740 10 client.16677 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:41:22.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.301+0000 7f6e00587740 20 client.16677 populate_metadata read hostname 'smithi090' 2024-04-02T14:41:22.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.302+0000 7f6e00587740 1 --2- 172.21.15.90:0/2221839291 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55e887d96ab0 0x55e887d98ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:22.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.302+0000 7f6e00587740 4 client.16677 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:41:22.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.302+0000 7f6e00587740 1 -- 172.21.15.90:0/2221839291 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e887c82630 con 0x55e887d96ab0 2024-04-02T14:41:22.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.305+0000 7f6df6a00640 1 --2- 172.21.15.90:0/2221839291 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55e887d96ab0 0x55e887d98ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:22.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.309+0000 7f6df6a00640 1 --2- 172.21.15.90:0/2221839291 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55e887d96ab0 0x55e887d98ed0 crc :-1 s=READY pgs=178 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=16c9ce24dad2deb9 server_cookie=1219ff6409345320 in_seq=0 out_seq=0 2024-04-02T14:41:22.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.309+0000 7f6de0c00640 10 client.16677 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:41:22.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.311+0000 7f6de0c00640 1 -- 172.21.15.90:0/2221839291 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e887c82630 con 0x55e887d96ab0 2024-04-02T14:41:22.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.311+0000 7f6de0c00640 10 client.16677 handle_command_reply: tid=0 2024-04-02T14:41:22.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.311+0000 7f6e00587740 1 client.16677 shutdown 2024-04-02T14:41:22.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.314+0000 7f6e00587740 1 -- 172.21.15.90:0/2221839291 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e887d7e0d0 msgr2=0x55e887d842d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:41:22.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.314+0000 7f6e00587740 1 --2- 172.21.15.90:0/2221839291 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e887d7e0d0 0x55e887d842d0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f6de4034500 tx=0x7f6de4005de0 comp rx=0 tx=0).stop 2024-04-02T14:41:22.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.314+0000 7f6e00587740 1 -- 172.21.15.90:0/2221839291 shutdown_connections 2024-04-02T14:41:22.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.314+0000 7f6e00587740 1 --2- 172.21.15.90:0/2221839291 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55e887d96ab0 0x55e887d98ed0 crc :-1 s=READY pgs=178 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:22.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.314+0000 7f6e00587740 1 --2- 172.21.15.90:0/2221839291 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e887d86cb0 0x55e887d84810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:22.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.314+0000 7f6e00587740 1 --2- 172.21.15.90:0/2221839291 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e887d7e0d0 0x55e887d842d0 secure :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f6de4034500 tx=0x7f6de4005de0 comp rx=0 tx=0).stop 2024-04-02T14:41:22.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.314+0000 7f6e00587740 1 --2- 172.21.15.90:0/2221839291 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e887d7d7b0 0x55e887d83d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:22.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.314+0000 7f6de0c00640 0 client.16677 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:41:22.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.314+0000 7f6e00587740 1 -- 172.21.15.90:0/2221839291 >> 172.21.15.90:0/2221839291 conn(0x55e887d77ee0 msgr2=0x55e887d79c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:41:22.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.315+0000 7f6e00587740 1 -- 172.21.15.90:0/2221839291 shutdown_connections 2024-04-02T14:41:22.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.315+0000 7f6e00587740 1 -- 172.21.15.90:0/2221839291 wait complete. 2024-04-02T14:41:22.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.315+0000 7f6e00587740 20 client.16677 trim_cache size 0 max 16384 2024-04-02T14:41:22.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.365+0000 7f6e00587740 1 Processor -- start 2024-04-02T14:41:22.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.365+0000 7f6e00587740 1 -- start start 2024-04-02T14:41:22.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.365+0000 7f6e00587740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e887d7d7b0 0x55e887d82cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:22.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.365+0000 7f6e00587740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e887d86cb0 0x55e887d92960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:22.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.365+0000 7f6e00587740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e887d84630 0x55e887d9e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:22.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.365+0000 7f6e00587740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f6df81ab800 con 0x55e887d84630 2024-04-02T14:41:22.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.365+0000 7f6e00587740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f6df81ad340 con 0x55e887d7d7b0 2024-04-02T14:41:22.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.365+0000 7f6e00587740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f6df81abed0 con 0x55e887d86cb0 2024-04-02T14:41:22.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.366+0000 7f6df6a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e887d7d7b0 0x55e887d82cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:22.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.366+0000 7f6df6a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e887d7d7b0 0x55e887d82cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:57556/0 (socket says 172.21.15.90:57556) 2024-04-02T14:41:22.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.366+0000 7f6df6a00640 1 -- 172.21.15.90:0/3930815371 learned_addr learned my addr 172.21.15.90:0/3930815371 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:41:22.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.366+0000 7f6df6a00640 1 -- 172.21.15.90:0/3930815371 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e887d86cb0 msgr2=0x55e887d92960 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-02T14:41:22.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.366+0000 7f6df6a00640 1 --2- 172.21.15.90:0/3930815371 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e887d86cb0 0x55e887d92960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:22.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.366+0000 7f6df6a00640 1 -- 172.21.15.90:0/3930815371 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e887d84630 msgr2=0x55e887d9e7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:41:22.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.366+0000 7f6df6a00640 1 --2- 172.21.15.90:0/3930815371 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e887d84630 0x55e887d9e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:22.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.366+0000 7f6df6a00640 1 -- 172.21.15.90:0/3930815371 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e887d7df00 con 0x55e887d7d7b0 2024-04-02T14:41:22.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.366+0000 7f6df6a00640 1 --2- 172.21.15.90:0/3930815371 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e887d7d7b0 0x55e887d82cd0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f6de8005fd0 tx=0x7f6de80d2620 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1b1359cc3a8bf80a server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:41:22.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.367+0000 7f6de3400640 1 -- 172.21.15.90:0/3930815371 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6de80b5070 con 0x55e887d7d7b0 2024-04-02T14:41:22.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.367+0000 7f6de3400640 1 -- 172.21.15.90:0/3930815371 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6de80a4d30 con 0x55e887d7d7b0 2024-04-02T14:41:22.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.367+0000 7f6de3400640 1 -- 172.21.15.90:0/3930815371 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6de8020040 con 0x55e887d7d7b0 2024-04-02T14:41:22.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.369+0000 7f6e00587740 1 -- 172.21.15.90:0/3930815371 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e887d7d7b0 msgr2=0x55e887d82cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:41:22.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.369+0000 7f6e00587740 1 --2- 172.21.15.90:0/3930815371 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e887d7d7b0 0x55e887d82cd0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f6de8005fd0 tx=0x7f6de80d2620 comp rx=0 tx=0).stop 2024-04-02T14:41:22.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.370+0000 7f6e00587740 1 -- 172.21.15.90:0/3930815371 shutdown_connections 2024-04-02T14:41:22.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.370+0000 7f6e00587740 1 --2- 172.21.15.90:0/3930815371 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e887d84630 0x55e887d9e7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:22.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.370+0000 7f6e00587740 1 --2- 172.21.15.90:0/3930815371 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e887d86cb0 0x55e887d92960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:22.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.370+0000 7f6e00587740 1 --2- 172.21.15.90:0/3930815371 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e887d7d7b0 0x55e887d82cd0 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:22.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.370+0000 7f6e00587740 1 -- 172.21.15.90:0/3930815371 >> 172.21.15.90:0/3930815371 conn(0x55e887d77ee0 msgr2=0x55e887d76f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:41:22.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.371+0000 7f6e00587740 1 -- 172.21.15.90:0/3930815371 shutdown_connections 2024-04-02T14:41:22.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.371+0000 7f6e00587740 1 -- 172.21.15.90:0/3930815371 wait complete. 2024-04-02T14:41:22.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.371+0000 7f6e00587740 1 -- start start 2024-04-02T14:41:22.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.372+0000 7f6e00587740 1 Processor -- start 2024-04-02T14:41:22.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.374+0000 7f6e00587740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e887d7d7b0 0x55e887d8ab40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:22.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.374+0000 7f6e00587740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e887d84630 0x55e887d8b080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:22.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.374+0000 7f6e00587740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e887d86cb0 0x55e887d8b5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:22.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.374+0000 7f6e00587740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f6df81acc70 con 0x55e887d7d7b0 2024-04-02T14:41:22.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.374+0000 7f6e00587740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f6df81ac5a0 con 0x55e887d86cb0 2024-04-02T14:41:22.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.374+0000 7f6e00587740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f6df81a97e0 con 0x55e887d84630 2024-04-02T14:41:22.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.374+0000 7f6e00587740 10 client.0 _ll_register_callbacks cb 0x7f6de40046c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:41:22.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.375+0000 7f6df6000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e887d84630 0x55e887d8b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:22.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.375+0000 7f6df6000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e887d84630 0x55e887d8b080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:42536/0 (socket says 172.21.15.90:42536) 2024-04-02T14:41:22.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.375+0000 7f6df6000640 1 -- 172.21.15.90:0/1870936676 learned_addr learned my addr 172.21.15.90:0/1870936676 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:41:22.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.375+0000 7f6df7400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e887d86cb0 0x55e887d8b5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:22.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.376+0000 7f6df6a00640 1 --2- 172.21.15.90:0/1870936676 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e887d7d7b0 0x55e887d8ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:22.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.376+0000 7f6df6000640 1 -- 172.21.15.90:0/1870936676 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e887d86cb0 msgr2=0x55e887d8b5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:41:22.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.376+0000 7f6df6000640 1 --2- 172.21.15.90:0/1870936676 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e887d86cb0 0x55e887d8b5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:22.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.376+0000 7f6df6000640 1 -- 172.21.15.90:0/1870936676 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e887d7d7b0 msgr2=0x55e887d8ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:41:22.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.376+0000 7f6df6000640 1 --2- 172.21.15.90:0/1870936676 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e887d7d7b0 0x55e887d8ab40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:22.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.376+0000 7f6df6000640 1 -- 172.21.15.90:0/1870936676 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e887dfc330 con 0x55e887d84630 2024-04-02T14:41:22.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.376+0000 7f6df6000640 1 --2- 172.21.15.90:0/1870936676 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e887d84630 0x55e887d8b080 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f6de4001eb0 tx=0x7f6de4004770 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b7caee35789105ae server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:41:22.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.377+0000 7f6de0c00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:41:22.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.377+0000 7f6de0c00640 1 -- 172.21.15.90:0/1870936676 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6de404b040 con 0x55e887d84630 2024-04-02T14:41:22.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.377+0000 7f6de0c00640 1 -- 172.21.15.90:0/1870936676 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f6de4034960 con 0x55e887d84630 2024-04-02T14:41:22.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.377+0000 7f6e00587740 1 -- 172.21.15.90:0/1870936676 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6de80a4d30 con 0x55e887d84630 2024-04-02T14:41:22.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.377+0000 7f6de0c00640 1 -- 172.21.15.90:0/1870936676 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f6de4057030 con 0x55e887d84630 2024-04-02T14:41:22.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.377+0000 7f6e00587740 10 client.27196 fetch_fsmap learned FSMap version 27 2024-04-02T14:41:22.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.377+0000 7f6e00587740 1 -- 172.21.15.90:0/1870936676 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55e887dfd5e0 con 0x55e887d84630 2024-04-02T14:41:22.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.378+0000 7f6de0c00640 1 -- 172.21.15.90:0/1870936676 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6de80a4d30 con 0x55e887d84630 2024-04-02T14:41:22.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.378+0000 7f6de0c00640 1 -- 172.21.15.90:0/1870936676 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f6de4063030 con 0x55e887d84630 2024-04-02T14:41:22.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.378+0000 7f6e00587740 10 client.27196 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:41:22.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.378+0000 7f6e00587740 10 client.27196 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:41:22.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.378+0000 7f6e00587740 20 client.27196 populate_metadata read hostname 'smithi090' 2024-04-02T14:41:22.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.378+0000 7f6e00587740 1 --2- 172.21.15.90:0/1870936676 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55e887dfdca0 0x55e887e000c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:22.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.378+0000 7f6e00587740 4 client.27196 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-02T14:41:22.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.378+0000 7f6e00587740 1 -- 172.21.15.90:0/1870936676 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55e887ceecb0 con 0x55e887dfdca0 2024-04-02T14:41:22.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.379+0000 7f6df6a00640 1 --2- 172.21.15.90:0/1870936676 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55e887dfdca0 0x55e887e000c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:22.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.380+0000 7f6df6a00640 1 --2- 172.21.15.90:0/1870936676 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55e887dfdca0 0x55e887e000c0 crc :-1 s=READY pgs=179 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=19bd38584487ef23 server_cookie=64a5e01a79a206a5 in_seq=0 out_seq=0 2024-04-02T14:41:22.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.380+0000 7f6de0c00640 10 client.27196 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:41:22.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.380+0000 7f6de0c00640 1 -- 172.21.15.90:0/1870936676 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55e887ceecb0 con 0x55e887dfdca0 2024-04-02T14:41:22.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.380+0000 7f6de0c00640 10 client.27196 handle_command_reply: tid=0 2024-04-02T14:41:22.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.381+0000 7f6e00587740 1 client.27196 shutdown 2024-04-02T14:41:22.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.381+0000 7f6e00587740 1 -- 172.21.15.90:0/1870936676 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e887d84630 msgr2=0x55e887d8b080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:41:22.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.381+0000 7f6e00587740 1 --2- 172.21.15.90:0/1870936676 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e887d84630 0x55e887d8b080 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f6de4001eb0 tx=0x7f6de4004770 comp rx=0 tx=0).stop 2024-04-02T14:41:22.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.381+0000 7f6e00587740 1 -- 172.21.15.90:0/1870936676 shutdown_connections 2024-04-02T14:41:22.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.381+0000 7f6e00587740 1 --2- 172.21.15.90:0/1870936676 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55e887dfdca0 0x55e887e000c0 crc :-1 s=READY pgs=179 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:22.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.381+0000 7f6e00587740 1 --2- 172.21.15.90:0/1870936676 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e887d86cb0 0x55e887d8b5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:22.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.382+0000 7f6e00587740 1 --2- 172.21.15.90:0/1870936676 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e887d84630 0x55e887d8b080 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:22.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.382+0000 7f6e00587740 1 --2- 172.21.15.90:0/1870936676 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e887d7d7b0 0x55e887d8ab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:22.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.382+0000 7f6e00587740 1 -- 172.21.15.90:0/1870936676 >> 172.21.15.90:0/1870936676 conn(0x55e887d77ee0 msgr2=0x55e887d83620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:41:22.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.382+0000 7f6de0c00640 0 client.27196 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:41:22.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.383+0000 7f6e00587740 1 -- 172.21.15.90:0/1870936676 shutdown_connections 2024-04-02T14:41:22.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.383+0000 7f6e00587740 1 -- 172.21.15.90:0/1870936676 wait complete. 2024-04-02T14:41:22.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.383+0000 7f6e00587740 20 client.27196 trim_cache size 0 max 16384 2024-04-02T14:41:22.383 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-02T14:41:22.384 INFO:teuthology.orchestra.run.smithi090.stdout: "status": "no active scrubs running", 2024-04-02T14:41:22.384 INFO:teuthology.orchestra.run.smithi090.stdout: "scrubs": {} 2024-04-02T14:41:22.384 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-02T14:41:22.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.385+0000 7f6dfd800640 1 -- 172.21.15.90:0/869176656 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f6ddc0a3da0 msgr2=0x7f6ddc0a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:41:22.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.385+0000 7f6dfd800640 1 --2- 172.21.15.90:0/869176656 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f6ddc0a3da0 0x7f6ddc0a6260 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f6df0012fd0 tx=0x7f6df001af70 comp rx=0 tx=0).stop 2024-04-02T14:41:22.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.385+0000 7f6dfd800640 1 -- 172.21.15.90:0/869176656 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6df8103f10 msgr2=0x7f6df81965f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:41:22.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.385+0000 7f6dfd800640 1 --2- 172.21.15.90:0/869176656 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6df8103f10 0x7f6df81965f0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f6de800d950 tx=0x7f6de800de10 comp rx=0 tx=0).stop 2024-04-02T14:41:22.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.385+0000 7f6dfd800640 1 -- 172.21.15.90:0/869176656 shutdown_connections 2024-04-02T14:41:22.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.385+0000 7f6dfd800640 1 --2- 172.21.15.90:0/869176656 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f6ddc0a3da0 0x7f6ddc0a6260 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:22.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.385+0000 7f6dfd800640 1 --2- 172.21.15.90:0/869176656 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6df819aaa0 0x7f6df819af50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:22.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.385+0000 7f6dfd800640 1 --2- 172.21.15.90:0/869176656 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6df8108fe0 0x7f6df8196b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:22.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.385+0000 7f6dfd800640 1 --2- 172.21.15.90:0/869176656 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6df8103f10 0x7f6df81965f0 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:22.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.385+0000 7f6dfd800640 1 -- 172.21.15.90:0/869176656 >> 172.21.15.90:0/869176656 conn(0x7f6df80ff7b0 msgr2=0x7f6df81078e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:41:22.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.386+0000 7f6dfd800640 1 -- 172.21.15.90:0/869176656 shutdown_connections 2024-04-02T14:41:22.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.386+0000 7f6dfd800640 1 -- 172.21.15.90:0/869176656 wait complete. 2024-04-02T14:41:22.414 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-02T14:41:22.415 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T14:41:22.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.679+0000 7fd88ba00640 1 -- 172.21.15.90:0/281452072 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd884073ed0 msgr2=0x7fd8840742b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:41:22.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.679+0000 7fd88ba00640 1 --2- 172.21.15.90:0/281452072 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd884073ed0 0x7fd8840742b0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fd87c009be0 tx=0x7fd87c02bf80 comp rx=0 tx=0).stop 2024-04-02T14:41:22.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.679+0000 7fd88ba00640 1 -- 172.21.15.90:0/281452072 shutdown_connections 2024-04-02T14:41:22.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.679+0000 7fd88ba00640 1 --2- 172.21.15.90:0/281452072 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd88410e330 0x7fd884110740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:22.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.679+0000 7fd88ba00640 1 --2- 172.21.15.90:0/281452072 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd8840747f0 0x7fd88410ddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:22.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.679+0000 7fd88ba00640 1 --2- 172.21.15.90:0/281452072 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd884073ed0 0x7fd8840742b0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:22.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.679+0000 7fd88ba00640 1 -- 172.21.15.90:0/281452072 >> 172.21.15.90:0/281452072 conn(0x7fd8840fd5a0 msgr2=0x7fd8840ff9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:41:22.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.679+0000 7fd88ba00640 1 -- 172.21.15.90:0/281452072 shutdown_connections 2024-04-02T14:41:22.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.679+0000 7fd88ba00640 1 -- 172.21.15.90:0/281452072 wait complete. 2024-04-02T14:41:22.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.680+0000 7fd88ba00640 1 Processor -- start 2024-04-02T14:41:22.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.681+0000 7fd88ba00640 1 -- start start 2024-04-02T14:41:22.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.682+0000 7fd88ba00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd884073ed0 0x7fd884107450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:22.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.682+0000 7fd88ba00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd8840747f0 0x7fd884107990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:22.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.682+0000 7fd88ba00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd88410e330 0x7fd8841025b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:22.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.682+0000 7fd88ba00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fd8841140e0 con 0x7fd88410e330 2024-04-02T14:41:22.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.682+0000 7fd88ba00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fd884113f60 con 0x7fd884073ed0 2024-04-02T14:41:22.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.682+0000 7fd88ba00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fd884114260 con 0x7fd8840747f0 2024-04-02T14:41:22.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.682+0000 7fd883e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd8840747f0 0x7fd884107990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:22.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.683+0000 7fd883e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd8840747f0 0x7fd884107990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:42554/0 (socket says 172.21.15.90:42554) 2024-04-02T14:41:22.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.683+0000 7fd883e00640 1 -- 172.21.15.90:0/2857449706 learned_addr learned my addr 172.21.15.90:0/2857449706 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:41:22.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.683+0000 7fd883e00640 1 -- 172.21.15.90:0/2857449706 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd884073ed0 msgr2=0x7fd884107450 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-02T14:41:22.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.683+0000 7fd883e00640 1 --2- 172.21.15.90:0/2857449706 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd884073ed0 0x7fd884107450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:22.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.683+0000 7fd883e00640 1 -- 172.21.15.90:0/2857449706 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd88410e330 msgr2=0x7fd8841025b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T14:41:22.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.683+0000 7fd883e00640 1 --2- 172.21.15.90:0/2857449706 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd88410e330 0x7fd8841025b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:22.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.683+0000 7fd883e00640 1 -- 172.21.15.90:0/2857449706 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd884102e70 con 0x7fd8840747f0 2024-04-02T14:41:22.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.684+0000 7fd883e00640 1 --2- 172.21.15.90:0/2857449706 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd8840747f0 0x7fd884107990 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fd87400e9c0 tx=0x7fd87400ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:41:22.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.684+0000 7fd881600640 1 -- 172.21.15.90:0/2857449706 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd87400ccc0 con 0x7fd8840747f0 2024-04-02T14:41:22.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.684+0000 7fd881600640 1 -- 172.21.15.90:0/2857449706 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd874005500 con 0x7fd8840747f0 2024-04-02T14:41:22.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.684+0000 7fd881600640 1 -- 172.21.15.90:0/2857449706 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd874010430 con 0x7fd8840747f0 2024-04-02T14:41:22.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.684+0000 7fd88ba00640 1 -- 172.21.15.90:0/2857449706 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd884103160 con 0x7fd8840747f0 2024-04-02T14:41:22.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.685+0000 7fd88ba00640 1 -- 172.21.15.90:0/2857449706 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd88410d340 con 0x7fd8840747f0 2024-04-02T14:41:22.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.686+0000 7fd88ba00640 1 -- 172.21.15.90:0/2857449706 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd8841033c0 con 0x7fd8840747f0 2024-04-02T14:41:22.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.690+0000 7fd881600640 1 -- 172.21.15.90:0/2857449706 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fd8740046c0 con 0x7fd8840747f0 2024-04-02T14:41:22.691 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.692+0000 7fd881600640 1 --2- 172.21.15.90:0/2857449706 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fd8680a3ef0 0x7fd8680a63b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:22.691 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.693+0000 7fd889000640 1 --2- 172.21.15.90:0/2857449706 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fd8680a3ef0 0x7fd8680a63b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:22.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.693+0000 7fd881600640 1 -- 172.21.15.90:0/2857449706 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7fd8740d0b80 con 0x7fd8840747f0 2024-04-02T14:41:22.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.694+0000 7fd881600640 1 -- 172.21.15.90:0/2857449706 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7fd874016330 con 0x7fd8840747f0 2024-04-02T14:41:22.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:22.694+0000 7fd889000640 1 --2- 172.21.15.90:0/2857449706 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fd8680a3ef0 0x7fd8680a63b0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fd87c0099a0 tx=0x7fd87c005fd0 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:41:23.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.104+0000 7fd88ba00640 1 -- 172.21.15.90:0/2857449706 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd884075580 con 0x7fd8840747f0 2024-04-02T14:41:23.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.105+0000 7fd881600640 1 -- 172.21.15.90:0/2857449706 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11927 (secure 0 0 0) 0x7fd87409d250 con 0x7fd8840747f0 2024-04-02T14:41:23.105 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-02T14:41:23.105 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14412,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6837/2852160309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6836","nonce":2852160309},{"type":"v1","addr":"172.21.15.90:6837","nonce":2852160309}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14478,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6841/1919376858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6840","nonce":1919376858},{"type":"v1","addr":"172.21.15.90:6841","nonce":1919376858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-02T14:32:22.319083+0000","modified":"2024-04-02T14:34:52.266152+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24590,"mds_1":24560,"mds_2":24547,"mds_3":24539,"mds_4":24523},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24454":{"gid":24454,"name":"l","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6833/1564319431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":1564319431},{"type":"v1","addr":"172.21.15.177:6833","nonce":1564319431}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.92:6833/1570337372","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":1570337372},{"type":"v1","addr":"172.21.15.92:6833","nonce":1570337372}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"j","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.90:6835/1277124891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6834","nonce":1277124891},{"type":"v1","addr":"172.21.15.90:6835","nonce":1277124891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.92:6835/860721164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":860721164},{"type":"v1","addr":"172.21.15.92:6835","nonce":860721164}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24511":{"gid":24511,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6835/3177244988","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":3177244988},{"type":"v1","addr":"172.21.15.177:6835","nonce":3177244988}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"c","rank":4,"incarnation":25,"state":"up:active","state_seq":9,"addr":"172.21.15.177:6837/4223414858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":4223414858},{"type":"v1","addr":"172.21.15.177:6837","nonce":4223414858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24539":{"gid":24539,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":8,"addr":"172.21.15.92:6837/1270635939","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":1270635939},{"type":"v1","addr":"172.21.15.92:6837","nonce":1270635939}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24547":{"gid":24547,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":7,"addr":"172.21.15.177:6839/664895663","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":664895663},{"type":"v1","addr":"172.21.15.177:6839","nonce":664895663}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"g","rank":1,"incarnation":19,"state":"up:active","state_seq":5,"addr":"172.21.15.90:6839/1498915870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6838","nonce":1498915870},{"type":"v1","addr":"172.21.15.90:6839","nonce":1498915870}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24590":{"gid":24590,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.92:6839/1054888219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":1054888219},{"type":"v1","addr":"172.21.15.92:6839","nonce":1054888219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24590,"qdb_cluster":[24547,24590,24523,24539,24560]},"id":1}]} 2024-04-02T14:41:23.105 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 27 2024-04-02T14:41:23.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.108+0000 7fd88ba00640 1 -- 172.21.15.90:0/2857449706 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fd8680a3ef0 msgr2=0x7fd8680a63b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:41:23.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.108+0000 7fd88ba00640 1 --2- 172.21.15.90:0/2857449706 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fd8680a3ef0 0x7fd8680a63b0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fd87c0099a0 tx=0x7fd87c005fd0 comp rx=0 tx=0).stop 2024-04-02T14:41:23.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.108+0000 7fd88ba00640 1 -- 172.21.15.90:0/2857449706 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd8840747f0 msgr2=0x7fd884107990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:41:23.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.108+0000 7fd88ba00640 1 --2- 172.21.15.90:0/2857449706 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd8840747f0 0x7fd884107990 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fd87400e9c0 tx=0x7fd87400ee80 comp rx=0 tx=0).stop 2024-04-02T14:41:23.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.108+0000 7fd88ba00640 1 -- 172.21.15.90:0/2857449706 shutdown_connections 2024-04-02T14:41:23.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.108+0000 7fd88ba00640 1 --2- 172.21.15.90:0/2857449706 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fd8680a3ef0 0x7fd8680a63b0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:23.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.108+0000 7fd88ba00640 1 --2- 172.21.15.90:0/2857449706 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd88410e330 0x7fd8841025b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:23.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.108+0000 7fd88ba00640 1 --2- 172.21.15.90:0/2857449706 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd8840747f0 0x7fd884107990 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:23.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.108+0000 7fd88ba00640 1 --2- 172.21.15.90:0/2857449706 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd884073ed0 0x7fd884107450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:23.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.108+0000 7fd88ba00640 1 -- 172.21.15.90:0/2857449706 >> 172.21.15.90:0/2857449706 conn(0x7fd8840fd5a0 msgr2=0x7fd8840fec80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:41:23.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.109+0000 7fd88ba00640 1 -- 172.21.15.90:0/2857449706 shutdown_connections 2024-04-02T14:41:23.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.109+0000 7fd88ba00640 1 -- 172.21.15.90:0/2857449706 wait complete. 2024-04-02T14:41:23.131 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-04-02T14:41:23.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.442+0000 7fa636c00640 1 -- 172.21.15.90:0/1504853563 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa630073ea0 msgr2=0x7fa630074280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:41:23.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.442+0000 7fa636c00640 1 --2- 172.21.15.90:0/1504853563 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa630073ea0 0x7fa630074280 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fa620009860 tx=0x7fa62002bc10 comp rx=0 tx=0).stop 2024-04-02T14:41:23.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.443+0000 7fa636c00640 1 -- 172.21.15.90:0/1504853563 shutdown_connections 2024-04-02T14:41:23.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.443+0000 7fa636c00640 1 --2- 172.21.15.90:0/1504853563 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa630112a90 0x7fa630114e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:23.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.443+0000 7fa636c00640 1 --2- 172.21.15.90:0/1504853563 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa6300747c0 0x7fa630112550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:23.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.443+0000 7fa636c00640 1 --2- 172.21.15.90:0/1504853563 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa630073ea0 0x7fa630074280 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:23.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.443+0000 7fa636c00640 1 -- 172.21.15.90:0/1504853563 >> 172.21.15.90:0/1504853563 conn(0x7fa6300fd590 msgr2=0x7fa6300ff9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:41:23.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.443+0000 7fa636c00640 1 -- 172.21.15.90:0/1504853563 shutdown_connections 2024-04-02T14:41:23.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.443+0000 7fa636c00640 1 -- 172.21.15.90:0/1504853563 wait complete. 2024-04-02T14:41:23.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.445+0000 7fa636c00640 1 Processor -- start 2024-04-02T14:41:23.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.445+0000 7fa636c00640 1 -- start start 2024-04-02T14:41:23.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.445+0000 7fa636c00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa630073ea0 0x7fa6301a4610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:23.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.445+0000 7fa636c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa6300747c0 0x7fa6301a4b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:23.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.445+0000 7fa636c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa630112a90 0x7fa63019e780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:23.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.445+0000 7fa636c00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fa630118460 con 0x7fa6300747c0 2024-04-02T14:41:23.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.445+0000 7fa636c00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fa6301182e0 con 0x7fa630073ea0 2024-04-02T14:41:23.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.445+0000 7fa636c00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fa6301185e0 con 0x7fa630112a90 2024-04-02T14:41:23.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.446+0000 7fa634c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa630112a90 0x7fa63019e780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:23.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.446+0000 7fa62fe00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa630073ea0 0x7fa6301a4610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:23.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.446+0000 7fa634c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa630112a90 0x7fa63019e780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:42570/0 (socket says 172.21.15.90:42570) 2024-04-02T14:41:23.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.446+0000 7fa634c00640 1 -- 172.21.15.90:0/2170308035 learned_addr learned my addr 172.21.15.90:0/2170308035 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:41:23.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.446+0000 7fa62f400640 1 --2- 172.21.15.90:0/2170308035 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa6300747c0 0x7fa6301a4b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:23.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.446+0000 7fa62fe00640 1 -- 172.21.15.90:0/2170308035 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa630112a90 msgr2=0x7fa63019e780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:41:23.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.446+0000 7fa62fe00640 1 --2- 172.21.15.90:0/2170308035 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa630112a90 0x7fa63019e780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:23.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.446+0000 7fa62fe00640 1 -- 172.21.15.90:0/2170308035 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa6300747c0 msgr2=0x7fa6301a4b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:41:23.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.446+0000 7fa62fe00640 1 --2- 172.21.15.90:0/2170308035 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa6300747c0 0x7fa6301a4b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:23.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.446+0000 7fa62fe00640 1 -- 172.21.15.90:0/2170308035 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa63019ef50 con 0x7fa630073ea0 2024-04-02T14:41:23.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.447+0000 7fa62f400640 1 --2- 172.21.15.90:0/2170308035 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa6300747c0 0x7fa6301a4b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:41:23.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.447+0000 7fa62fe00640 1 --2- 172.21.15.90:0/2170308035 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa630073ea0 0x7fa6301a4610 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fa62002c150 tx=0x7fa6200346f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:41:23.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.447+0000 7fa62cc00640 1 -- 172.21.15.90:0/2170308035 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa620041020 con 0x7fa630073ea0 2024-04-02T14:41:23.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.447+0000 7fa636c00640 1 -- 172.21.15.90:0/2170308035 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa63019f1e0 con 0x7fa630073ea0 2024-04-02T14:41:23.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.447+0000 7fa62cc00640 1 -- 172.21.15.90:0/2170308035 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa62002ca90 con 0x7fa630073ea0 2024-04-02T14:41:23.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.447+0000 7fa62cc00640 1 -- 172.21.15.90:0/2170308035 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa620039040 con 0x7fa630073ea0 2024-04-02T14:41:23.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.447+0000 7fa636c00640 1 -- 172.21.15.90:0/2170308035 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6301ab460 con 0x7fa630073ea0 2024-04-02T14:41:23.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.452+0000 7fa639984740 1 Processor -- start 2024-04-02T14:41:23.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.453+0000 7fa639984740 1 -- start start 2024-04-02T14:41:23.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.454+0000 7fa639984740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556175c1a9f0 0x556175c19740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:23.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.454+0000 7fa639984740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556175c19d10 0x556175c1a190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:23.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.454+0000 7fa639984740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556175c23080 0x556175c25470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:23.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.454+0000 7fa639984740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x556175b5a400 con 0x556175c19d10 2024-04-02T14:41:23.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.454+0000 7fa639984740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x556175b3ef10 con 0x556175c23080 2024-04-02T14:41:23.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.454+0000 7fa639984740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x556175b7e6f0 con 0x556175c1a9f0 2024-04-02T14:41:23.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.454+0000 7fa62f400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556175c23080 0x556175c25470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:23.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.455+0000 7fa634c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556175c1a9f0 0x556175c19740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:23.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.455+0000 7fa634c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556175c1a9f0 0x556175c19740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:42580/0 (socket says 172.21.15.90:42580) 2024-04-02T14:41:23.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.455+0000 7fa634c00640 1 -- 172.21.15.90:0/16998457 learned_addr learned my addr 172.21.15.90:0/16998457 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:41:23.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.455+0000 7fa62cc00640 1 -- 172.21.15.90:0/2170308035 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fa620044020 con 0x7fa630073ea0 2024-04-02T14:41:23.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.456+0000 7fa634c00640 1 -- 172.21.15.90:0/16998457 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556175c23080 msgr2=0x556175c25470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:41:23.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.456+0000 7fa634c00640 1 --2- 172.21.15.90:0/16998457 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556175c23080 0x556175c25470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:23.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.456+0000 7fa634c00640 1 -- 172.21.15.90:0/16998457 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556175c19d10 msgr2=0x556175c1a190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:41:23.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.456+0000 7fa634c00640 1 --2- 172.21.15.90:0/16998457 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556175c19d10 0x556175c1a190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:23.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.456+0000 7fa634c00640 1 -- 172.21.15.90:0/16998457 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5561758923f0 con 0x556175c1a9f0 2024-04-02T14:41:23.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.456+0000 7fa62f400640 1 --2- 172.21.15.90:0/16998457 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556175c23080 0x556175c25470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:41:23.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.457+0000 7fa634c00640 1 --2- 172.21.15.90:0/16998457 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556175c1a9f0 0x556175c19740 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fa628007f10 tx=0x7fa62800f790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=958103a5bc2b64e2 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:41:23.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.458+0000 7fa625200640 1 -- 172.21.15.90:0/16998457 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa628004d20 con 0x556175c1a9f0 2024-04-02T14:41:23.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.458+0000 7fa625200640 1 -- 172.21.15.90:0/16998457 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa628010b20 con 0x556175c1a9f0 2024-04-02T14:41:23.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.458+0000 7fa625200640 1 -- 172.21.15.90:0/16998457 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa628005750 con 0x556175c1a9f0 2024-04-02T14:41:23.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.458+0000 7fa62cc00640 1 --2- 172.21.15.90:0/2170308035 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fa6140a3da0 0x7fa6140a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:23.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.459+0000 7fa62cc00640 1 -- 172.21.15.90:0/2170308035 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7fa6200f1ec0 con 0x7fa630073ea0 2024-04-02T14:41:23.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.459+0000 7fa639984740 1 -- 172.21.15.90:0/16998457 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556175c1a9f0 msgr2=0x556175c19740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:41:23.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.459+0000 7fa639984740 1 --2- 172.21.15.90:0/16998457 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556175c1a9f0 0x556175c19740 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fa628007f10 tx=0x7fa62800f790 comp rx=0 tx=0).stop 2024-04-02T14:41:23.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.460+0000 7fa62fe00640 1 --2- 172.21.15.90:0/2170308035 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fa6140a3da0 0x7fa6140a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:23.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.460+0000 7fa639984740 1 -- 172.21.15.90:0/16998457 shutdown_connections 2024-04-02T14:41:23.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.460+0000 7fa639984740 1 --2- 172.21.15.90:0/16998457 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556175c23080 0x556175c25470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:23.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.460+0000 7fa639984740 1 --2- 172.21.15.90:0/16998457 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556175c19d10 0x556175c1a190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:23.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.460+0000 7fa639984740 1 --2- 172.21.15.90:0/16998457 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556175c1a9f0 0x556175c19740 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:23.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.460+0000 7fa639984740 1 -- 172.21.15.90:0/16998457 >> 172.21.15.90:0/16998457 conn(0x556175be0a50 msgr2=0x556175c14760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:41:23.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.461+0000 7fa639984740 1 -- 172.21.15.90:0/16998457 shutdown_connections 2024-04-02T14:41:23.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.461+0000 7fa62fe00640 1 --2- 172.21.15.90:0/2170308035 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fa6140a3da0 0x7fa6140a6260 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x556175c1af40 tx=0x7fa62002bc10 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:41:23.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.461+0000 7fa639984740 1 -- 172.21.15.90:0/16998457 wait complete. 2024-04-02T14:41:23.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.462+0000 7fa639984740 1 -- start start 2024-04-02T14:41:23.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.462+0000 7fa639984740 1 Processor -- start 2024-04-02T14:41:23.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.463+0000 7fa639984740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556175c19d10 0x556175c20040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:23.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.463+0000 7fa639984740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556175c1a9f0 0x556175c20580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:23.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.463+0000 7fa639984740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556175c23080 0x556175c20ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:23.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.463+0000 7fa639984740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x556175b7a6a0 con 0x556175c23080 2024-04-02T14:41:23.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.463+0000 7fa639984740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x556175b26980 con 0x556175c1a9f0 2024-04-02T14:41:23.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.463+0000 7fa639984740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x556175b39cb0 con 0x556175c19d10 2024-04-02T14:41:23.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.463+0000 7fa639984740 10 client.0 _ll_register_callbacks cb 0x556175b396a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:41:23.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.463+0000 7fa62f400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556175c1a9f0 0x556175c20580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:23.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.463+0000 7fa62f400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556175c1a9f0 0x556175c20580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:57614/0 (socket says 172.21.15.90:57614) 2024-04-02T14:41:23.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.463+0000 7fa62f400640 1 -- 172.21.15.90:0/2195265586 learned_addr learned my addr 172.21.15.90:0/2195265586 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:41:23.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.464+0000 7fa62f400640 1 -- 172.21.15.90:0/2195265586 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556175c19d10 msgr2=0x556175c20040 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-02T14:41:23.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.464+0000 7fa634c00640 1 --2- 172.21.15.90:0/2195265586 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556175c23080 0x556175c20ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:23.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.464+0000 7fa62f400640 1 --2- 172.21.15.90:0/2195265586 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556175c19d10 0x556175c20040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:23.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.464+0000 7fa62f400640 1 -- 172.21.15.90:0/2195265586 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556175c23080 msgr2=0x556175c20ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:41:23.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.465+0000 7fa62f400640 1 --2- 172.21.15.90:0/2195265586 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556175c23080 0x556175c20ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:23.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.465+0000 7fa62f400640 1 -- 172.21.15.90:0/2195265586 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55617595c7d0 con 0x556175c1a9f0 2024-04-02T14:41:23.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.465+0000 7fa62f400640 1 --2- 172.21.15.90:0/2195265586 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556175c1a9f0 0x556175c20580 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fa61c00bf20 tx=0x7fa61c0128e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6860d2f837f396ff server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:41:23.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.465+0000 7fa634c00640 1 --2- 172.21.15.90:0/2195265586 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556175c23080 0x556175c20ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:41:23.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.465+0000 7fa619600640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:41:23.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.465+0000 7fa619600640 1 -- 172.21.15.90:0/2195265586 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa61c004230 con 0x556175c1a9f0 2024-04-02T14:41:23.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.465+0000 7fa619600640 1 -- 172.21.15.90:0/2195265586 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7fa61c0043d0 con 0x556175c1a9f0 2024-04-02T14:41:23.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.466+0000 7fa639984740 1 -- 172.21.15.90:0/2195265586 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa628010b20 con 0x556175c1a9f0 2024-04-02T14:41:23.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.466+0000 7fa619600640 1 -- 172.21.15.90:0/2195265586 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fa61c0106a0 con 0x556175c1a9f0 2024-04-02T14:41:23.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.466+0000 7fa639984740 10 client.27344 fetch_fsmap learned FSMap version 27 2024-04-02T14:41:23.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.466+0000 7fa639984740 1 -- 172.21.15.90:0/2195265586 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x556175c31fb0 con 0x556175c1a9f0 2024-04-02T14:41:23.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.467+0000 7fa619600640 1 -- 172.21.15.90:0/2195265586 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa628010b20 con 0x556175c1a9f0 2024-04-02T14:41:23.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.467+0000 7fa619600640 1 -- 172.21.15.90:0/2195265586 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fa61c026470 con 0x556175c1a9f0 2024-04-02T14:41:23.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.467+0000 7fa639984740 10 client.27344 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:41:23.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.467+0000 7fa639984740 10 client.27344 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.c 2024-04-02T14:41:23.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.467+0000 7fa639984740 20 client.27344 populate_metadata read hostname 'smithi090' 2024-04-02T14:41:23.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.468+0000 7fa639984740 1 --2- 172.21.15.90:0/2195265586 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x556175c33860 0x556175c35cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:23.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.468+0000 7fa639984740 4 client.27344 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:41:23.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.468+0000 7fa639984740 1 -- 172.21.15.90:0/2195265586 --> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556175b3e040 con 0x556175c33860 2024-04-02T14:41:23.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.468+0000 7fa634c00640 1 --2- 172.21.15.90:0/2195265586 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x556175c33860 0x556175c35cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:23.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.469+0000 7fa634c00640 1 --2- 172.21.15.90:0/2195265586 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x556175c33860 0x556175c35cc0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=6e3a223c1c5452a7 server_cookie=79afa7158f4811c7 in_seq=0 out_seq=0 2024-04-02T14:41:23.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.469+0000 7fa619600640 10 client.27344 ms_handle_connect on v2:172.21.15.177:6836/4223414858 2024-04-02T14:41:23.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.469+0000 7fa619600640 1 -- 172.21.15.90:0/2195265586 <== mds.4 v2:172.21.15.177:6836/4223414858 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x556175b3e040 con 0x556175c33860 2024-04-02T14:41:23.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.469+0000 7fa619600640 10 client.27344 handle_command_reply: tid=0 2024-04-02T14:41:23.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.470+0000 7fa639984740 1 client.27344 shutdown 2024-04-02T14:41:23.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.470+0000 7fa639984740 1 -- 172.21.15.90:0/2195265586 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556175c1a9f0 msgr2=0x556175c20580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:41:23.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.470+0000 7fa639984740 1 --2- 172.21.15.90:0/2195265586 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556175c1a9f0 0x556175c20580 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fa61c00bf20 tx=0x7fa61c0128e0 comp rx=0 tx=0).stop 2024-04-02T14:41:23.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.470+0000 7fa639984740 1 -- 172.21.15.90:0/2195265586 shutdown_connections 2024-04-02T14:41:23.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.470+0000 7fa639984740 1 --2- 172.21.15.90:0/2195265586 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x556175c33860 0x556175c35cc0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:23.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.470+0000 7fa639984740 1 --2- 172.21.15.90:0/2195265586 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556175c23080 0x556175c20ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:23.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.470+0000 7fa639984740 1 --2- 172.21.15.90:0/2195265586 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556175c1a9f0 0x556175c20580 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:23.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.470+0000 7fa639984740 1 --2- 172.21.15.90:0/2195265586 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556175c19d10 0x556175c20040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:23.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.470+0000 7fa639984740 1 -- 172.21.15.90:0/2195265586 >> 172.21.15.90:0/2195265586 conn(0x556175be0a50 msgr2=0x556175c164b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:41:23.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.470+0000 7fa619600640 0 client.27344 ms_handle_reset on v2:172.21.15.177:6836/4223414858 2024-04-02T14:41:23.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.471+0000 7fa639984740 1 -- 172.21.15.90:0/2195265586 shutdown_connections 2024-04-02T14:41:23.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.471+0000 7fa639984740 1 -- 172.21.15.90:0/2195265586 wait complete. 2024-04-02T14:41:23.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.471+0000 7fa639984740 20 client.27344 trim_cache size 0 max 16384 2024-04-02T14:41:23.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.494+0000 7fa639984740 1 Processor -- start 2024-04-02T14:41:23.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.494+0000 7fa639984740 1 -- start start 2024-04-02T14:41:23.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.494+0000 7fa639984740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556175c19d10 0x556175c22770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:23.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.494+0000 7fa639984740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556175c23080 0x556175c2eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:23.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.494+0000 7fa639984740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556175c20710 0x556175c3b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:23.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.494+0000 7fa639984740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fa6301b3f80 con 0x556175c23080 2024-04-02T14:41:23.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.494+0000 7fa639984740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fa6301b5ac0 con 0x556175c20710 2024-04-02T14:41:23.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.494+0000 7fa639984740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fa6301b4650 con 0x556175c19d10 2024-04-02T14:41:23.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.494+0000 7fa634c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556175c19d10 0x556175c22770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:23.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.494+0000 7fa634c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556175c19d10 0x556175c22770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:42590/0 (socket says 172.21.15.90:42590) 2024-04-02T14:41:23.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.494+0000 7fa634c00640 1 -- 172.21.15.90:0/266204592 learned_addr learned my addr 172.21.15.90:0/266204592 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:41:23.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.494+0000 7fa634c00640 1 --2- 172.21.15.90:0/266204592 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556175c20710 0x556175c3b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:23.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.495+0000 7fa62f400640 1 --2- 172.21.15.90:0/266204592 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556175c23080 0x556175c2eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:23.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.495+0000 7fa634c00640 1 -- 172.21.15.90:0/266204592 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556175c20710 msgr2=0x556175c3b5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:41:23.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.495+0000 7fa634c00640 1 --2- 172.21.15.90:0/266204592 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556175c20710 0x556175c3b5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:23.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.495+0000 7fa634c00640 1 -- 172.21.15.90:0/266204592 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556175c23080 msgr2=0x556175c2eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:41:23.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.495+0000 7fa634c00640 1 --2- 172.21.15.90:0/266204592 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556175c23080 0x556175c2eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:23.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.495+0000 7fa634c00640 1 -- 172.21.15.90:0/266204592 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556175c18d40 con 0x556175c19d10 2024-04-02T14:41:23.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.496+0000 7fa634c00640 1 --2- 172.21.15.90:0/266204592 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556175c19d10 0x556175c22770 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fa628007e60 tx=0x7fa628007ad0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=50b469eba65b79f4 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:41:23.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.497+0000 7fa61be00640 1 -- 172.21.15.90:0/266204592 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa62801d040 con 0x556175c19d10 2024-04-02T14:41:23.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.497+0000 7fa61be00640 1 -- 172.21.15.90:0/266204592 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa628027e50 con 0x556175c19d10 2024-04-02T14:41:23.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.497+0000 7fa61be00640 1 -- 172.21.15.90:0/266204592 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa628005760 con 0x556175c19d10 2024-04-02T14:41:23.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.498+0000 7fa639984740 1 -- 172.21.15.90:0/266204592 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556175c19d10 msgr2=0x556175c22770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:41:23.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.498+0000 7fa639984740 1 --2- 172.21.15.90:0/266204592 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556175c19d10 0x556175c22770 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fa628007e60 tx=0x7fa628007ad0 comp rx=0 tx=0).stop 2024-04-02T14:41:23.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.498+0000 7fa639984740 1 -- 172.21.15.90:0/266204592 shutdown_connections 2024-04-02T14:41:23.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.498+0000 7fa639984740 1 --2- 172.21.15.90:0/266204592 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556175c20710 0x556175c3b5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:23.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.498+0000 7fa639984740 1 --2- 172.21.15.90:0/266204592 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556175c23080 0x556175c2eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:23.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.498+0000 7fa639984740 1 --2- 172.21.15.90:0/266204592 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556175c19d10 0x556175c22770 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:23.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.498+0000 7fa639984740 1 -- 172.21.15.90:0/266204592 >> 172.21.15.90:0/266204592 conn(0x556175be0a50 msgr2=0x556175c20000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:41:23.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.498+0000 7fa639984740 1 -- 172.21.15.90:0/266204592 shutdown_connections 2024-04-02T14:41:23.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.498+0000 7fa639984740 1 -- 172.21.15.90:0/266204592 wait complete. 2024-04-02T14:41:23.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.499+0000 7fa639984740 1 -- start start 2024-04-02T14:41:23.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.499+0000 7fa639984740 1 Processor -- start 2024-04-02T14:41:23.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.500+0000 7fa639984740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556175c19d10 0x556175c24240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:23.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.500+0000 7fa639984740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556175c20710 0x556175c24780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:23.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.500+0000 7fa639984740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556175c23080 0x556175c295d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:23.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.500+0000 7fa639984740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fa6301b53f0 con 0x556175c19d10 2024-04-02T14:41:23.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.500+0000 7fa62f400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556175c20710 0x556175c24780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:23.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.500+0000 7fa639984740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fa6301b4d20 con 0x556175c20710 2024-04-02T14:41:23.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.500+0000 7fa639984740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fa6301b1f60 con 0x556175c23080 2024-04-02T14:41:23.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.500+0000 7fa634c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556175c19d10 0x556175c24240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:23.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.500+0000 7fa62f400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556175c20710 0x556175c24780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:57634/0 (socket says 172.21.15.90:57634) 2024-04-02T14:41:23.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.500+0000 7fa62f400640 1 -- 172.21.15.90:0/1586134158 learned_addr learned my addr 172.21.15.90:0/1586134158 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:41:23.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.500+0000 7fa639984740 10 client.0 _ll_register_callbacks cb 0x7fa61c00bb30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:41:23.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.500+0000 7fa634c00640 1 --2- 172.21.15.90:0/1586134158 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556175c23080 0x556175c295d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:23.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.501+0000 7fa62f400640 1 -- 172.21.15.90:0/1586134158 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556175c23080 msgr2=0x556175c295d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:41:23.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.501+0000 7fa62f400640 1 --2- 172.21.15.90:0/1586134158 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556175c23080 0x556175c295d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:23.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.501+0000 7fa62f400640 1 -- 172.21.15.90:0/1586134158 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556175c19d10 msgr2=0x556175c24240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:41:23.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.501+0000 7fa62f400640 1 --2- 172.21.15.90:0/1586134158 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556175c19d10 0x556175c24240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:23.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.501+0000 7fa62f400640 1 -- 172.21.15.90:0/1586134158 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556175c29b10 con 0x556175c20710 2024-04-02T14:41:23.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.501+0000 7fa634c00640 1 --2- 172.21.15.90:0/1586134158 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556175c19d10 0x556175c24240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:41:23.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.501+0000 7fa62f400640 1 --2- 172.21.15.90:0/1586134158 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556175c20710 0x556175c24780 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fa61c012de0 tx=0x7fa61c026cb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ae07e9c75aed69df server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:41:23.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.501+0000 7fa619600640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:41:23.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.502+0000 7fa619600640 1 -- 172.21.15.90:0/1586134158 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa61c010cd0 con 0x556175c20710 2024-04-02T14:41:23.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.502+0000 7fa619600640 1 -- 172.21.15.90:0/1586134158 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7fa61c0050d0 con 0x556175c20710 2024-04-02T14:41:23.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.502+0000 7fa639984740 1 -- 172.21.15.90:0/1586134158 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa628027e50 con 0x556175c20710 2024-04-02T14:41:23.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.502+0000 7fa619600640 1 -- 172.21.15.90:0/1586134158 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fa61c010e70 con 0x556175c20710 2024-04-02T14:41:23.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.502+0000 7fa639984740 10 client.27350 fetch_fsmap learned FSMap version 27 2024-04-02T14:41:23.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.502+0000 7fa639984740 1 -- 172.21.15.90:0/1586134158 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x556175c9a4c0 con 0x556175c20710 2024-04-02T14:41:23.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.503+0000 7fa619600640 1 -- 172.21.15.90:0/1586134158 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa628027e50 con 0x556175c20710 2024-04-02T14:41:23.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.503+0000 7fa619600640 1 -- 172.21.15.90:0/1586134158 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fa61c02f450 con 0x556175c20710 2024-04-02T14:41:23.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.503+0000 7fa639984740 10 client.27350 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:41:23.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.503+0000 7fa639984740 10 client.27350 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.c 2024-04-02T14:41:23.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.503+0000 7fa639984740 20 client.27350 populate_metadata read hostname 'smithi090' 2024-04-02T14:41:23.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.503+0000 7fa639984740 1 --2- 172.21.15.90:0/1586134158 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x556175c9ab30 0x556175c9cf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:23.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.503+0000 7fa639984740 4 client.27350 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-02T14:41:23.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.503+0000 7fa639984740 1 -- 172.21.15.90:0/1586134158 --> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] -- command(tid 0: {"prefix": "damage ls"}) -- 0x556175b8bc30 con 0x556175c9ab30 2024-04-02T14:41:23.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.504+0000 7fa634c00640 1 --2- 172.21.15.90:0/1586134158 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x556175c9ab30 0x556175c9cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:23.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.504+0000 7fa634c00640 1 --2- 172.21.15.90:0/1586134158 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x556175c9ab30 0x556175c9cf50 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=6bf7935fe5fc70be server_cookie=78431e1a9bd006c3 in_seq=0 out_seq=0 2024-04-02T14:41:23.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.504+0000 7fa619600640 10 client.27350 ms_handle_connect on v2:172.21.15.177:6836/4223414858 2024-04-02T14:41:23.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.505+0000 7fa619600640 1 -- 172.21.15.90:0/1586134158 <== mds.4 v2:172.21.15.177:6836/4223414858 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x556175b8bc30 con 0x556175c9ab30 2024-04-02T14:41:23.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.505+0000 7fa619600640 10 client.27350 handle_command_reply: tid=0 2024-04-02T14:41:23.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.505+0000 7fa639984740 1 client.27350 shutdown 2024-04-02T14:41:23.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.505+0000 7fa639984740 1 -- 172.21.15.90:0/1586134158 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556175c20710 msgr2=0x556175c24780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:41:23.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.505+0000 7fa639984740 1 --2- 172.21.15.90:0/1586134158 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556175c20710 0x556175c24780 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fa61c012de0 tx=0x7fa61c026cb0 comp rx=0 tx=0).stop 2024-04-02T14:41:23.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.506+0000 7fa639984740 1 -- 172.21.15.90:0/1586134158 shutdown_connections 2024-04-02T14:41:23.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.506+0000 7fa639984740 1 --2- 172.21.15.90:0/1586134158 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556175c23080 0x556175c295d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:23.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.506+0000 7fa639984740 1 --2- 172.21.15.90:0/1586134158 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556175c20710 0x556175c24780 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:23.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.506+0000 7fa639984740 1 --2- 172.21.15.90:0/1586134158 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x556175c9ab30 0x556175c9cf50 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:23.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.506+0000 7fa639984740 1 --2- 172.21.15.90:0/1586134158 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556175c19d10 0x556175c24240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:23.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.506+0000 7fa619600640 0 client.27350 ms_handle_reset on v2:172.21.15.177:6836/4223414858 2024-04-02T14:41:23.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.506+0000 7fa639984740 1 -- 172.21.15.90:0/1586134158 >> 172.21.15.90:0/1586134158 conn(0x556175be0a50 msgr2=0x556175c13010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:41:23.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.506+0000 7fa639984740 1 -- 172.21.15.90:0/1586134158 shutdown_connections 2024-04-02T14:41:23.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.506+0000 7fa639984740 1 -- 172.21.15.90:0/1586134158 wait complete. 2024-04-02T14:41:23.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.506+0000 7fa639984740 20 client.27350 trim_cache size 0 max 16384 2024-04-02T14:41:23.506 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-02T14:41:23.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.508+0000 7fa636c00640 1 -- 172.21.15.90:0/2170308035 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fa6140a3da0 msgr2=0x7fa6140a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:41:23.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.508+0000 7fa636c00640 1 --2- 172.21.15.90:0/2170308035 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fa6140a3da0 0x7fa6140a6260 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x556175c1af40 tx=0x7fa62002bc10 comp rx=0 tx=0).stop 2024-04-02T14:41:23.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.508+0000 7fa636c00640 1 -- 172.21.15.90:0/2170308035 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa630073ea0 msgr2=0x7fa6301a4610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:41:23.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.508+0000 7fa636c00640 1 --2- 172.21.15.90:0/2170308035 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa630073ea0 0x7fa6301a4610 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fa62002c150 tx=0x7fa6200346f0 comp rx=0 tx=0).stop 2024-04-02T14:41:23.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.508+0000 7fa636c00640 1 -- 172.21.15.90:0/2170308035 shutdown_connections 2024-04-02T14:41:23.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.508+0000 7fa636c00640 1 --2- 172.21.15.90:0/2170308035 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fa6140a3da0 0x7fa6140a6260 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:23.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.508+0000 7fa636c00640 1 --2- 172.21.15.90:0/2170308035 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa630112a90 0x7fa63019e780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:23.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.508+0000 7fa636c00640 1 --2- 172.21.15.90:0/2170308035 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa6300747c0 0x7fa6301a4b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:23.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.508+0000 7fa636c00640 1 --2- 172.21.15.90:0/2170308035 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa630073ea0 0x7fa6301a4610 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:23.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.508+0000 7fa636c00640 1 -- 172.21.15.90:0/2170308035 >> 172.21.15.90:0/2170308035 conn(0x7fa6300fd590 msgr2=0x7fa6300fed20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:41:23.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.509+0000 7fa636c00640 1 -- 172.21.15.90:0/2170308035 shutdown_connections 2024-04-02T14:41:23.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.509+0000 7fa636c00640 1 -- 172.21.15.90:0/2170308035 wait complete. 2024-04-02T14:41:23.534 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-04-02T14:41:23.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.851+0000 7f0f52600640 1 -- 172.21.15.90:0/491770532 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0f4c10d440 msgr2=0x7f0f4c079e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:41:23.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.851+0000 7f0f52600640 1 --2- 172.21.15.90:0/491770532 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0f4c10d440 0x7f0f4c079e10 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f0f4400afd0 tx=0x7f0f4402cc50 comp rx=0 tx=0).stop 2024-04-02T14:41:23.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.852+0000 7f0f52600640 1 -- 172.21.15.90:0/491770532 shutdown_connections 2024-04-02T14:41:23.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.852+0000 7f0f52600640 1 --2- 172.21.15.90:0/491770532 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0f4c10d440 0x7f0f4c079e10 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:23.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.852+0000 7f0f52600640 1 --2- 172.21.15.90:0/491770532 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0f4c109720 0x7f0f4c10cd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:23.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.852+0000 7f0f52600640 1 --2- 172.21.15.90:0/491770532 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0f4c108d70 0x7f0f4c109150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:23.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.852+0000 7f0f52600640 1 -- 172.21.15.90:0/491770532 >> 172.21.15.90:0/491770532 conn(0x7f0f4c06cbe0 msgr2=0x7f0f4c06cff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:41:23.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.852+0000 7f0f52600640 1 -- 172.21.15.90:0/491770532 shutdown_connections 2024-04-02T14:41:23.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.852+0000 7f0f52600640 1 -- 172.21.15.90:0/491770532 wait complete. 2024-04-02T14:41:23.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.853+0000 7f0f52600640 1 Processor -- start 2024-04-02T14:41:23.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.853+0000 7f0f52600640 1 -- start start 2024-04-02T14:41:23.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.854+0000 7f0f52600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0f4c108d70 0x7f0f4c109d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:23.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.855+0000 7f0f52600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0f4c10d440 0x7f0f4c10a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:23.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.855+0000 7f0f52600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0f4c10a7e0 0x7f0f4c1bfcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:23.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.855+0000 7f0f52600640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f0f4c07d230 con 0x7f0f4c10d440 2024-04-02T14:41:23.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.855+0000 7f0f52600640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f0f4c07d0b0 con 0x7f0f4c10a7e0 2024-04-02T14:41:23.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.855+0000 7f0f52600640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f0f4c07d3b0 con 0x7f0f4c108d70 2024-04-02T14:41:23.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.855+0000 7f0f4b400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0f4c108d70 0x7f0f4c109d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:23.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.855+0000 7f0f4b400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0f4c108d70 0x7f0f4c109d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:42414/0 (socket says 172.21.15.90:42414) 2024-04-02T14:41:23.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.855+0000 7f0f4b400640 1 -- 172.21.15.90:0/1598266783 learned_addr learned my addr 172.21.15.90:0/1598266783 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:41:23.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.855+0000 7f0f4be00640 1 --2- 172.21.15.90:0/1598266783 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0f4c10a7e0 0x7f0f4c1bfcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:23.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.856+0000 7f0f4b400640 1 -- 172.21.15.90:0/1598266783 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0f4c10a7e0 msgr2=0x7f0f4c1bfcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:41:23.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.856+0000 7f0f4b400640 1 --2- 172.21.15.90:0/1598266783 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0f4c10a7e0 0x7f0f4c1bfcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:23.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.856+0000 7f0f4b400640 1 -- 172.21.15.90:0/1598266783 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0f4c10d440 msgr2=0x7f0f4c10a2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:41:23.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.856+0000 7f0f4b400640 1 --2- 172.21.15.90:0/1598266783 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0f4c10d440 0x7f0f4c10a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:23.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.856+0000 7f0f4b400640 1 -- 172.21.15.90:0/1598266783 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f4c1c0370 con 0x7f0f4c108d70 2024-04-02T14:41:23.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.856+0000 7f0f4b400640 1 --2- 172.21.15.90:0/1598266783 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0f4c108d70 0x7f0f4c109d60 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f0f3c00e9f0 tx=0x7f0f3c00eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:41:23.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.858+0000 7f0f43e00640 1 -- 172.21.15.90:0/1598266783 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f3c00cd80 con 0x7f0f4c108d70 2024-04-02T14:41:23.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.858+0000 7f0f43e00640 1 -- 172.21.15.90:0/1598266783 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0f3c010910 con 0x7f0f4c108d70 2024-04-02T14:41:23.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.858+0000 7f0f43e00640 1 -- 172.21.15.90:0/1598266783 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f3c018980 con 0x7f0f4c108d70 2024-04-02T14:41:23.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.858+0000 7f0f52600640 1 -- 172.21.15.90:0/1598266783 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f4c1c0660 con 0x7f0f4c108d70 2024-04-02T14:41:23.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.858+0000 7f0f52600640 1 -- 172.21.15.90:0/1598266783 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f4c1c0b60 con 0x7f0f4c108d70 2024-04-02T14:41:23.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.864+0000 7f0f43e00640 1 -- 172.21.15.90:0/1598266783 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f0f3c010ac0 con 0x7f0f4c108d70 2024-04-02T14:41:23.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.867+0000 7f0f43e00640 1 --2- 172.21.15.90:0/1598266783 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f0f300a3da0 0x7f0f300a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:23.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.867+0000 7f0f43e00640 1 -- 172.21.15.90:0/1598266783 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f0f3c09aa20 con 0x7f0f4c108d70 2024-04-02T14:41:23.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.869+0000 7f0f4aa00640 1 --2- 172.21.15.90:0/1598266783 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f0f300a3da0 0x7f0f300a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:23.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.872+0000 7f0f4aa00640 1 --2- 172.21.15.90:0/1598266783 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f0f300a3da0 0x7f0f300a6260 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f0f38005d80 tx=0x7f0f38005c90 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:41:23.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.875+0000 7f0f55394740 1 Processor -- start 2024-04-02T14:41:23.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.877+0000 7f0f55394740 1 -- start start 2024-04-02T14:41:23.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.879+0000 7f0f55394740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557b07f589f0 0x557b07f57740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:23.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.879+0000 7f0f55394740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557b07f57d10 0x557b07f58190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:23.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.879+0000 7f0f55394740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557b07f61080 0x557b07f63470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:23.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.879+0000 7f0f55394740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x557b07e98400 con 0x557b07f589f0 2024-04-02T14:41:23.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.879+0000 7f0f55394740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x557b07e7cf10 con 0x557b07f57d10 2024-04-02T14:41:23.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.879+0000 7f0f55394740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x557b07ebc6f0 con 0x557b07f61080 2024-04-02T14:41:23.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.880+0000 7f0f4aa00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557b07f57d10 0x557b07f58190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:23.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.880+0000 7f0f4aa00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557b07f57d10 0x557b07f58190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:35228/0 (socket says 172.21.15.90:35228) 2024-04-02T14:41:23.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.880+0000 7f0f4aa00640 1 -- 172.21.15.90:0/2493518922 learned_addr learned my addr 172.21.15.90:0/2493518922 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:41:23.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.880+0000 7f0f4b400640 1 --2- 172.21.15.90:0/2493518922 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557b07f589f0 0x557b07f57740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:23.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.880+0000 7f0f4be00640 1 --2- 172.21.15.90:0/2493518922 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557b07f61080 0x557b07f63470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:23.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.881+0000 7f0f4b400640 1 -- 172.21.15.90:0/2493518922 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557b07f61080 msgr2=0x557b07f63470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:41:23.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.881+0000 7f0f4b400640 1 --2- 172.21.15.90:0/2493518922 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557b07f61080 0x557b07f63470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:23.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.881+0000 7f0f4b400640 1 -- 172.21.15.90:0/2493518922 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557b07f57d10 msgr2=0x557b07f58190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:41:23.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.881+0000 7f0f4b400640 1 --2- 172.21.15.90:0/2493518922 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557b07f57d10 0x557b07f58190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:23.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.881+0000 7f0f4b400640 1 -- 172.21.15.90:0/2493518922 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557b07bd03f0 con 0x557b07f589f0 2024-04-02T14:41:23.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.881+0000 7f0f4b400640 1 --2- 172.21.15.90:0/2493518922 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557b07f589f0 0x557b07f57740 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f0f3c0ce130 tx=0x7f0f3c00cf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33af19b28ac47d69 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:41:23.882 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.882+0000 7f0f40c00640 1 -- 172.21.15.90:0/2493518922 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f3c0d5060 con 0x557b07f589f0 2024-04-02T14:41:23.882 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.882+0000 7f0f40c00640 1 -- 172.21.15.90:0/2493518922 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0f3c0a0d30 con 0x557b07f589f0 2024-04-02T14:41:23.882 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.882+0000 7f0f40c00640 1 -- 172.21.15.90:0/2493518922 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f3c0bbb40 con 0x557b07f589f0 2024-04-02T14:41:23.882 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.883+0000 7f0f55394740 1 -- 172.21.15.90:0/2493518922 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557b07f589f0 msgr2=0x557b07f57740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:41:23.882 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.883+0000 7f0f55394740 1 --2- 172.21.15.90:0/2493518922 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557b07f589f0 0x557b07f57740 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f0f3c0ce130 tx=0x7f0f3c00cf20 comp rx=0 tx=0).stop 2024-04-02T14:41:23.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.884+0000 7f0f55394740 1 -- 172.21.15.90:0/2493518922 shutdown_connections 2024-04-02T14:41:23.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.884+0000 7f0f55394740 1 --2- 172.21.15.90:0/2493518922 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557b07f61080 0x557b07f63470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:23.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.884+0000 7f0f55394740 1 --2- 172.21.15.90:0/2493518922 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557b07f57d10 0x557b07f58190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:23.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.884+0000 7f0f55394740 1 --2- 172.21.15.90:0/2493518922 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557b07f589f0 0x557b07f57740 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:23.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.884+0000 7f0f55394740 1 -- 172.21.15.90:0/2493518922 >> 172.21.15.90:0/2493518922 conn(0x557b07f1ea50 msgr2=0x557b07f52760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:41:23.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.884+0000 7f0f55394740 1 -- 172.21.15.90:0/2493518922 shutdown_connections 2024-04-02T14:41:23.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.885+0000 7f0f55394740 1 -- 172.21.15.90:0/2493518922 wait complete. 2024-04-02T14:41:23.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.885+0000 7f0f55394740 1 -- start start 2024-04-02T14:41:23.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.885+0000 7f0f55394740 1 Processor -- start 2024-04-02T14:41:23.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.886+0000 7f0f55394740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557b07f57d10 0x557b07f5e0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:23.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.886+0000 7f0f55394740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557b07f589f0 0x557b07f5e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:23.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.886+0000 7f0f55394740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557b07f61080 0x557b07f5eb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:23.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.886+0000 7f0f55394740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x557b07eb86a0 con 0x557b07f61080 2024-04-02T14:41:23.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.886+0000 7f0f55394740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x557b07e64980 con 0x557b07f589f0 2024-04-02T14:41:23.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.886+0000 7f0f55394740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x557b07e77cb0 con 0x557b07f57d10 2024-04-02T14:41:23.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.886+0000 7f0f55394740 10 client.0 _ll_register_callbacks cb 0x557b07e776a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:41:23.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.887+0000 7f0f4aa00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557b07f589f0 0x557b07f5e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:23.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.887+0000 7f0f4aa00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557b07f589f0 0x557b07f5e5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:35238/0 (socket says 172.21.15.90:35238) 2024-04-02T14:41:23.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.887+0000 7f0f4aa00640 1 -- 172.21.15.90:0/2460078637 learned_addr learned my addr 172.21.15.90:0/2460078637 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:41:23.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.887+0000 7f0f4b400640 1 --2- 172.21.15.90:0/2460078637 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557b07f57d10 0x557b07f5e0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:23.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.887+0000 7f0f4be00640 1 --2- 172.21.15.90:0/2460078637 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557b07f61080 0x557b07f5eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:23.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.888+0000 7f0f4b400640 1 -- 172.21.15.90:0/2460078637 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557b07f589f0 msgr2=0x557b07f5e5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:41:23.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.888+0000 7f0f4b400640 1 --2- 172.21.15.90:0/2460078637 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557b07f589f0 0x557b07f5e5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:23.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.888+0000 7f0f4b400640 1 -- 172.21.15.90:0/2460078637 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557b07f61080 msgr2=0x557b07f5eb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:41:23.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.888+0000 7f0f4b400640 1 --2- 172.21.15.90:0/2460078637 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557b07f61080 0x557b07f5eb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:23.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.888+0000 7f0f4b400640 1 -- 172.21.15.90:0/2460078637 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557b07c9a7d0 con 0x557b07f57d10 2024-04-02T14:41:23.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.889+0000 7f0f4b400640 1 --2- 172.21.15.90:0/2460078637 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557b07f57d10 0x557b07f5e0b0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f0f3c09c680 tx=0x7f0f3c098d50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=84350163570a62c5 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:41:23.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.889+0000 7f0f35600640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:41:23.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.889+0000 7f0f35600640 1 -- 172.21.15.90:0/2460078637 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f3c0bbd40 con 0x557b07f57d10 2024-04-02T14:41:23.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.889+0000 7f0f35600640 1 -- 172.21.15.90:0/2460078637 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f0f3c09bdd0 con 0x557b07f57d10 2024-04-02T14:41:23.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.889+0000 7f0f55394740 1 -- 172.21.15.90:0/2460078637 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0f3c0a0d30 con 0x557b07f57d10 2024-04-02T14:41:23.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.890+0000 7f0f35600640 1 -- 172.21.15.90:0/2460078637 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f0f3c0a55f0 con 0x557b07f57d10 2024-04-02T14:41:23.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.891+0000 7f0f55394740 10 client.27229 fetch_fsmap learned FSMap version 27 2024-04-02T14:41:23.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.891+0000 7f0f55394740 1 -- 172.21.15.90:0/2460078637 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x557b07f6ffa0 con 0x557b07f57d10 2024-04-02T14:41:23.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.892+0000 7f0f35600640 1 -- 172.21.15.90:0/2460078637 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0f3c0a0d30 con 0x557b07f57d10 2024-04-02T14:41:23.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.892+0000 7f0f35600640 1 -- 172.21.15.90:0/2460078637 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f0f3c0987e0 con 0x557b07f57d10 2024-04-02T14:41:23.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.892+0000 7f0f55394740 10 client.27229 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:41:23.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.892+0000 7f0f55394740 10 client.27229 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-04-02T14:41:23.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.892+0000 7f0f55394740 20 client.27229 populate_metadata read hostname 'smithi090' 2024-04-02T14:41:23.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.892+0000 7f0f55394740 1 --2- 172.21.15.90:0/2460078637 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x557b07f712f0 0x557b07f736e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:23.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.892+0000 7f0f55394740 4 client.27229 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:41:23.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.892+0000 7f0f55394740 1 -- 172.21.15.90:0/2460078637 --> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557b07e7c040 con 0x557b07f712f0 2024-04-02T14:41:23.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.893+0000 7f0f4aa00640 1 --2- 172.21.15.90:0/2460078637 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x557b07f712f0 0x557b07f736e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:23.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.896+0000 7f0f4aa00640 1 --2- 172.21.15.90:0/2460078637 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x557b07f712f0 0x557b07f736e0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=ec50bfc4217c7f31 server_cookie=78ccb6deacc2bd9e in_seq=0 out_seq=0 2024-04-02T14:41:23.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.896+0000 7f0f35600640 10 client.27229 ms_handle_connect on v2:172.21.15.92:6836/1270635939 2024-04-02T14:41:23.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.897+0000 7f0f35600640 1 -- 172.21.15.90:0/2460078637 <== mds.3 v2:172.21.15.92:6836/1270635939 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x557b07e7c040 con 0x557b07f712f0 2024-04-02T14:41:23.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.897+0000 7f0f35600640 10 client.27229 handle_command_reply: tid=0 2024-04-02T14:41:23.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.897+0000 7f0f55394740 1 client.27229 shutdown 2024-04-02T14:41:23.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.897+0000 7f0f55394740 1 -- 172.21.15.90:0/2460078637 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557b07f57d10 msgr2=0x557b07f5e0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:41:23.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.897+0000 7f0f55394740 1 --2- 172.21.15.90:0/2460078637 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557b07f57d10 0x557b07f5e0b0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f0f3c09c680 tx=0x7f0f3c098d50 comp rx=0 tx=0).stop 2024-04-02T14:41:23.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.897+0000 7f0f55394740 1 -- 172.21.15.90:0/2460078637 shutdown_connections 2024-04-02T14:41:23.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.897+0000 7f0f55394740 1 --2- 172.21.15.90:0/2460078637 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x557b07f712f0 0x557b07f736e0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:23.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.897+0000 7f0f55394740 1 --2- 172.21.15.90:0/2460078637 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557b07f61080 0x557b07f5eb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:23.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.897+0000 7f0f55394740 1 --2- 172.21.15.90:0/2460078637 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557b07f589f0 0x557b07f5e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:23.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.897+0000 7f0f55394740 1 --2- 172.21.15.90:0/2460078637 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557b07f57d10 0x557b07f5e0b0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:23.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.897+0000 7f0f55394740 1 -- 172.21.15.90:0/2460078637 >> 172.21.15.90:0/2460078637 conn(0x557b07f1ea50 msgr2=0x557b07f544b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:41:23.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.898+0000 7f0f35600640 0 client.27229 ms_handle_reset on v2:172.21.15.92:6836/1270635939 2024-04-02T14:41:23.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.898+0000 7f0f55394740 1 -- 172.21.15.90:0/2460078637 shutdown_connections 2024-04-02T14:41:23.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.898+0000 7f0f55394740 1 -- 172.21.15.90:0/2460078637 wait complete. 2024-04-02T14:41:23.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.898+0000 7f0f55394740 20 client.27229 trim_cache size 0 max 16384 2024-04-02T14:41:23.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.931+0000 7f0f55394740 1 Processor -- start 2024-04-02T14:41:23.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.931+0000 7f0f55394740 1 -- start start 2024-04-02T14:41:23.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.932+0000 7f0f55394740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557b07f57d10 0x557b07f5d610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:23.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.932+0000 7f0f55394740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557b07f589f0 0x557b07f5b330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:23.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.932+0000 7f0f55394740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557b07f61080 0x557b07f5b870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:23.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.932+0000 7f0f55394740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f0f4c1c9330 con 0x557b07f57d10 2024-04-02T14:41:23.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.932+0000 7f0f55394740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f0f4c1cae70 con 0x557b07f589f0 2024-04-02T14:41:23.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.932+0000 7f0f55394740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f0f4c1c9a00 con 0x557b07f61080 2024-04-02T14:41:23.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.932+0000 7f0f4be00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557b07f61080 0x557b07f5b870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:23.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.932+0000 7f0f4be00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557b07f61080 0x557b07f5b870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:42442/0 (socket says 172.21.15.90:42442) 2024-04-02T14:41:23.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.932+0000 7f0f4be00640 1 -- 172.21.15.90:0/1817342265 learned_addr learned my addr 172.21.15.90:0/1817342265 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:41:23.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.932+0000 7f0f4aa00640 1 --2- 172.21.15.90:0/1817342265 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557b07f589f0 0x557b07f5b330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:23.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.933+0000 7f0f4b400640 1 --2- 172.21.15.90:0/1817342265 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557b07f57d10 0x557b07f5d610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:23.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.933+0000 7f0f4be00640 1 -- 172.21.15.90:0/1817342265 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557b07f589f0 msgr2=0x557b07f5b330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:41:23.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.933+0000 7f0f4be00640 1 --2- 172.21.15.90:0/1817342265 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557b07f589f0 0x557b07f5b330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:23.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.933+0000 7f0f4be00640 1 -- 172.21.15.90:0/1817342265 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557b07f57d10 msgr2=0x557b07f5d610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:41:23.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.933+0000 7f0f4be00640 1 --2- 172.21.15.90:0/1817342265 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557b07f57d10 0x557b07f5d610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:23.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.933+0000 7f0f4be00640 1 -- 172.21.15.90:0/1817342265 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557b07f56d40 con 0x557b07f61080 2024-04-02T14:41:23.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.934+0000 7f0f4b400640 1 --2- 172.21.15.90:0/1817342265 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557b07f57d10 0x557b07f5d610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:41:23.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.934+0000 7f0f4aa00640 1 --2- 172.21.15.90:0/1817342265 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557b07f589f0 0x557b07f5b330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:41:23.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.935+0000 7f0f4be00640 1 --2- 172.21.15.90:0/1817342265 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557b07f61080 0x557b07f5b870 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f0f4402f010 tx=0x7f0f4402f040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b8daf6ad7b8c36f server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:41:23.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.935+0000 7f0f37e00640 1 -- 172.21.15.90:0/1817342265 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f440343a0 con 0x557b07f61080 2024-04-02T14:41:23.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.935+0000 7f0f37e00640 1 -- 172.21.15.90:0/1817342265 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0f44034540 con 0x557b07f61080 2024-04-02T14:41:23.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.935+0000 7f0f37e00640 1 -- 172.21.15.90:0/1817342265 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f44034d60 con 0x557b07f61080 2024-04-02T14:41:23.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.937+0000 7f0f55394740 1 -- 172.21.15.90:0/1817342265 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557b07f61080 msgr2=0x557b07f5b870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:41:23.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.937+0000 7f0f55394740 1 --2- 172.21.15.90:0/1817342265 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557b07f61080 0x557b07f5b870 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f0f4402f010 tx=0x7f0f4402f040 comp rx=0 tx=0).stop 2024-04-02T14:41:23.937 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:41:23 smithi090 ceph-mon[31674]: pgmap v198: 97 pgs: 97 active+clean; 4.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 6.0 MiB/s wr, 632 op/s 2024-04-02T14:41:23.937 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:41:23 smithi090 ceph-mon[31674]: from='client.? 172.21.15.90:0/2857449706' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-02T14:41:23.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.937+0000 7f0f55394740 1 -- 172.21.15.90:0/1817342265 shutdown_connections 2024-04-02T14:41:23.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.937+0000 7f0f55394740 1 --2- 172.21.15.90:0/1817342265 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557b07f61080 0x557b07f5b870 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:23.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.937+0000 7f0f55394740 1 --2- 172.21.15.90:0/1817342265 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557b07f589f0 0x557b07f5b330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:23.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.937+0000 7f0f55394740 1 --2- 172.21.15.90:0/1817342265 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557b07f57d10 0x557b07f5d610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:23.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.937+0000 7f0f55394740 1 -- 172.21.15.90:0/1817342265 >> 172.21.15.90:0/1817342265 conn(0x557b07f1ea50 msgr2=0x557b07f518a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:41:23.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.938+0000 7f0f55394740 1 -- 172.21.15.90:0/1817342265 shutdown_connections 2024-04-02T14:41:23.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.939+0000 7f0f55394740 1 -- 172.21.15.90:0/1817342265 wait complete. 2024-04-02T14:41:23.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.940+0000 7f0f55394740 1 -- start start 2024-04-02T14:41:23.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.940+0000 7f0f55394740 1 Processor -- start 2024-04-02T14:41:23.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.941+0000 7f0f55394740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557b07f57d10 0x557b07fcfa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:23.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.942+0000 7f0f55394740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557b07f589f0 0x557b07fcff90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:23.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.942+0000 7f0f55394740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557b07f61080 0x557b07f75330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:23.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.942+0000 7f0f55394740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f0f4c1ca7a0 con 0x557b07f61080 2024-04-02T14:41:23.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.942+0000 7f0f55394740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f0f4c1ca0d0 con 0x557b07f57d10 2024-04-02T14:41:23.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.942+0000 7f0f55394740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f0f4c1c7310 con 0x557b07f589f0 2024-04-02T14:41:23.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.942+0000 7f0f55394740 10 client.0 _ll_register_callbacks cb 0x7f0f3c09c590 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:41:23.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.942+0000 7f0f4b400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557b07f57d10 0x557b07fcfa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:23.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.942+0000 7f0f4b400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557b07f57d10 0x557b07fcfa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:35254/0 (socket says 172.21.15.90:35254) 2024-04-02T14:41:23.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.942+0000 7f0f4b400640 1 -- 172.21.15.90:0/2422196705 learned_addr learned my addr 172.21.15.90:0/2422196705 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:41:23.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.942+0000 7f0f4aa00640 1 --2- 172.21.15.90:0/2422196705 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557b07f589f0 0x557b07fcff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:23.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.943+0000 7f0f4be00640 1 --2- 172.21.15.90:0/2422196705 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557b07f61080 0x557b07f75330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:23.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.943+0000 7f0f4b400640 1 -- 172.21.15.90:0/2422196705 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557b07f589f0 msgr2=0x557b07fcff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:41:23.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.943+0000 7f0f4b400640 1 --2- 172.21.15.90:0/2422196705 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557b07f589f0 0x557b07fcff90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:23.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.943+0000 7f0f4b400640 1 -- 172.21.15.90:0/2422196705 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557b07f61080 msgr2=0x557b07f75330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:41:23.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.943+0000 7f0f4b400640 1 --2- 172.21.15.90:0/2422196705 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557b07f61080 0x557b07f75330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:23.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.943+0000 7f0f4b400640 1 -- 172.21.15.90:0/2422196705 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557b07f75870 con 0x557b07f57d10 2024-04-02T14:41:23.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.944+0000 7f0f4aa00640 1 --2- 172.21.15.90:0/2422196705 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557b07f589f0 0x557b07fcff90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:41:23.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.945+0000 7f0f4b400640 1 --2- 172.21.15.90:0/2422196705 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557b07f57d10 0x557b07fcfa50 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f0f3c09c910 tx=0x7f0f3c09ae90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7f40dd72fa14ef5 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:41:23.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.945+0000 7f0f35600640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:41:23.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.945+0000 7f0f35600640 1 -- 172.21.15.90:0/2422196705 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f3c09bfd0 con 0x557b07f57d10 2024-04-02T14:41:23.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.946+0000 7f0f35600640 1 -- 172.21.15.90:0/2422196705 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f0f3c018b20 con 0x557b07f57d10 2024-04-02T14:41:23.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.946+0000 7f0f55394740 1 -- 172.21.15.90:0/2422196705 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0f44034540 con 0x557b07f57d10 2024-04-02T14:41:23.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.947+0000 7f0f35600640 1 -- 172.21.15.90:0/2422196705 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f0f3c0a0d30 con 0x557b07f57d10 2024-04-02T14:41:23.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.947+0000 7f0f55394740 10 client.27368 fetch_fsmap learned FSMap version 27 2024-04-02T14:41:23.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.947+0000 7f0f55394740 1 -- 172.21.15.90:0/2422196705 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x557b07f76a50 con 0x557b07f57d10 2024-04-02T14:41:23.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.948+0000 7f0f35600640 1 -- 172.21.15.90:0/2422196705 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0f44034540 con 0x557b07f57d10 2024-04-02T14:41:23.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.948+0000 7f0f35600640 1 -- 172.21.15.90:0/2422196705 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f0f3c09b840 con 0x557b07f57d10 2024-04-02T14:41:23.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.949+0000 7f0f55394740 10 client.27368 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:41:23.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.949+0000 7f0f55394740 10 client.27368 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-04-02T14:41:23.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.949+0000 7f0f55394740 20 client.27368 populate_metadata read hostname 'smithi090' 2024-04-02T14:41:23.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.949+0000 7f0f55394740 1 --2- 172.21.15.90:0/2422196705 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x557b07fdfeb0 0x557b07fe22d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:23.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.949+0000 7f0f55394740 4 client.27368 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-02T14:41:23.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.949+0000 7f0f55394740 1 -- 172.21.15.90:0/2422196705 --> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] -- command(tid 0: {"prefix": "damage ls"}) -- 0x557b07ec9c30 con 0x557b07fdfeb0 2024-04-02T14:41:23.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.949+0000 7f0f4aa00640 1 --2- 172.21.15.90:0/2422196705 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x557b07fdfeb0 0x557b07fe22d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:23.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.951+0000 7f0f4aa00640 1 --2- 172.21.15.90:0/2422196705 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x557b07fdfeb0 0x557b07fe22d0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=3325b93316ad5981 server_cookie=96dfe817b07fc53e in_seq=0 out_seq=0 2024-04-02T14:41:23.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.951+0000 7f0f35600640 10 client.27368 ms_handle_connect on v2:172.21.15.92:6836/1270635939 2024-04-02T14:41:23.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.952+0000 7f0f35600640 1 -- 172.21.15.90:0/2422196705 <== mds.3 v2:172.21.15.92:6836/1270635939 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x557b07ec9c30 con 0x557b07fdfeb0 2024-04-02T14:41:23.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.952+0000 7f0f35600640 10 client.27368 handle_command_reply: tid=0 2024-04-02T14:41:23.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.952+0000 7f0f55394740 1 client.27368 shutdown 2024-04-02T14:41:23.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.952+0000 7f0f55394740 1 -- 172.21.15.90:0/2422196705 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557b07f57d10 msgr2=0x557b07fcfa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:41:23.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.952+0000 7f0f55394740 1 --2- 172.21.15.90:0/2422196705 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557b07f57d10 0x557b07fcfa50 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f0f3c09c910 tx=0x7f0f3c09ae90 comp rx=0 tx=0).stop 2024-04-02T14:41:23.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.953+0000 7f0f55394740 1 -- 172.21.15.90:0/2422196705 shutdown_connections 2024-04-02T14:41:23.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.953+0000 7f0f55394740 1 --2- 172.21.15.90:0/2422196705 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x557b07fdfeb0 0x557b07fe22d0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:23.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.953+0000 7f0f55394740 1 --2- 172.21.15.90:0/2422196705 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557b07f61080 0x557b07f75330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:23.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.953+0000 7f0f55394740 1 --2- 172.21.15.90:0/2422196705 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557b07f589f0 0x557b07fcff90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:23.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.953+0000 7f0f35600640 0 client.27368 ms_handle_reset on v2:172.21.15.92:6836/1270635939 2024-04-02T14:41:23.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.953+0000 7f0f55394740 1 --2- 172.21.15.90:0/2422196705 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557b07f57d10 0x557b07fcfa50 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:23.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.953+0000 7f0f55394740 1 -- 172.21.15.90:0/2422196705 >> 172.21.15.90:0/2422196705 conn(0x557b07f1ea50 msgr2=0x557b07f5e610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:41:23.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.953+0000 7f0f55394740 1 -- 172.21.15.90:0/2422196705 shutdown_connections 2024-04-02T14:41:23.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.954+0000 7f0f55394740 1 -- 172.21.15.90:0/2422196705 wait complete. 2024-04-02T14:41:23.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.954+0000 7f0f55394740 20 client.27368 trim_cache size 0 max 16384 2024-04-02T14:41:23.953 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-02T14:41:23.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.955+0000 7f0f52600640 1 -- 172.21.15.90:0/1598266783 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f0f300a3da0 msgr2=0x7f0f300a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:41:23.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.955+0000 7f0f52600640 1 --2- 172.21.15.90:0/1598266783 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f0f300a3da0 0x7f0f300a6260 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f0f38005d80 tx=0x7f0f38005c90 comp rx=0 tx=0).stop 2024-04-02T14:41:23.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.955+0000 7f0f52600640 1 -- 172.21.15.90:0/1598266783 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0f4c108d70 msgr2=0x7f0f4c109d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:41:23.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.955+0000 7f0f52600640 1 --2- 172.21.15.90:0/1598266783 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0f4c108d70 0x7f0f4c109d60 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f0f3c00e9f0 tx=0x7f0f3c00eeb0 comp rx=0 tx=0).stop 2024-04-02T14:41:23.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.956+0000 7f0f52600640 1 -- 172.21.15.90:0/1598266783 shutdown_connections 2024-04-02T14:41:23.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.956+0000 7f0f52600640 1 --2- 172.21.15.90:0/1598266783 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f0f300a3da0 0x7f0f300a6260 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:23.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.956+0000 7f0f52600640 1 --2- 172.21.15.90:0/1598266783 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0f4c10a7e0 0x7f0f4c1bfcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:23.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.956+0000 7f0f52600640 1 --2- 172.21.15.90:0/1598266783 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0f4c10d440 0x7f0f4c10a2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:23.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.956+0000 7f0f52600640 1 --2- 172.21.15.90:0/1598266783 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0f4c108d70 0x7f0f4c109d60 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:23.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.956+0000 7f0f52600640 1 -- 172.21.15.90:0/1598266783 >> 172.21.15.90:0/1598266783 conn(0x7f0f4c06cbe0 msgr2=0x7f0f4c10c250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:41:23.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.956+0000 7f0f52600640 1 -- 172.21.15.90:0/1598266783 shutdown_connections 2024-04-02T14:41:23.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:23.956+0000 7f0f52600640 1 -- 172.21.15.90:0/1598266783 wait complete. 2024-04-02T14:41:23.983 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-02T14:41:24.108 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:41:23 smithi092 ceph-mon[36497]: pgmap v198: 97 pgs: 97 active+clean; 4.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 6.0 MiB/s wr, 632 op/s 2024-04-02T14:41:24.108 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:41:23 smithi092 ceph-mon[36497]: from='client.? 172.21.15.90:0/2857449706' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-02T14:41:24.184 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:41:23 smithi177 ceph-mon[36431]: pgmap v198: 97 pgs: 97 active+clean; 4.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 6.0 MiB/s wr, 632 op/s 2024-04-02T14:41:24.184 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:41:23 smithi177 ceph-mon[36431]: from='client.? 172.21.15.90:0/2857449706' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-02T14:41:24.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.294+0000 7fdc60600640 1 -- 172.21.15.90:0/1321118288 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdc58108cc0 msgr2=0x7fdc5810cd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:41:24.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.294+0000 7fdc60600640 1 --2- 172.21.15.90:0/1321118288 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdc58108cc0 0x7fdc5810cd00 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fdc44009900 tx=0x7fdc4402bc60 comp rx=0 tx=0).stop 2024-04-02T14:41:24.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.295+0000 7fdc60600640 1 -- 172.21.15.90:0/1321118288 shutdown_connections 2024-04-02T14:41:24.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.295+0000 7fdc60600640 1 --2- 172.21.15.90:0/1321118288 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdc5810d3e0 0x7fdc58114fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:24.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.295+0000 7fdc60600640 1 --2- 172.21.15.90:0/1321118288 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdc58108cc0 0x7fdc5810cd00 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:24.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.295+0000 7fdc60600640 1 --2- 172.21.15.90:0/1321118288 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdc58108310 0x7fdc581086f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:24.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.295+0000 7fdc60600640 1 -- 172.21.15.90:0/1321118288 >> 172.21.15.90:0/1321118288 conn(0x7fdc58076040 msgr2=0x7fdc58078460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:41:24.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.295+0000 7fdc60600640 1 -- 172.21.15.90:0/1321118288 shutdown_connections 2024-04-02T14:41:24.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.295+0000 7fdc60600640 1 -- 172.21.15.90:0/1321118288 wait complete. 2024-04-02T14:41:24.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.296+0000 7fdc60600640 1 Processor -- start 2024-04-02T14:41:24.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.297+0000 7fdc60600640 1 -- start start 2024-04-02T14:41:24.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.298+0000 7fdc60600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdc58108310 0x7fdc580733a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:24.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.298+0000 7fdc60600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdc5810d3e0 0x7fdc5806c470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:24.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.298+0000 7fdc5dc00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdc58108310 0x7fdc580733a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:24.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.298+0000 7fdc5dc00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdc58108310 0x7fdc580733a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:35278/0 (socket says 172.21.15.90:35278) 2024-04-02T14:41:24.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.298+0000 7fdc5dc00640 1 -- 172.21.15.90:0/275648356 learned_addr learned my addr 172.21.15.90:0/275648356 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:41:24.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.298+0000 7fdc60600640 1 --2- 172.21.15.90:0/275648356 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdc5806c9b0 0x7fdc5806ce60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:24.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.299+0000 7fdc60600640 1 -- 172.21.15.90:0/275648356 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fdc581184a0 con 0x7fdc5810d3e0 2024-04-02T14:41:24.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.299+0000 7fdc60600640 1 -- 172.21.15.90:0/275648356 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fdc58118320 con 0x7fdc58108310 2024-04-02T14:41:24.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.299+0000 7fdc60600640 1 -- 172.21.15.90:0/275648356 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fdc58118620 con 0x7fdc5806c9b0 2024-04-02T14:41:24.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.299+0000 7fdc5e600640 1 --2- 172.21.15.90:0/275648356 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdc5806c9b0 0x7fdc5806ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:24.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.299+0000 7fdc5dc00640 1 -- 172.21.15.90:0/275648356 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdc5806c9b0 msgr2=0x7fdc5806ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:41:24.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.299+0000 7fdc5dc00640 1 --2- 172.21.15.90:0/275648356 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdc5806c9b0 0x7fdc5806ce60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:24.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.299+0000 7fdc5dc00640 1 -- 172.21.15.90:0/275648356 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdc5810d3e0 msgr2=0x7fdc5806c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:41:24.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.299+0000 7fdc5dc00640 1 --2- 172.21.15.90:0/275648356 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdc5810d3e0 0x7fdc5806c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:24.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.299+0000 7fdc5dc00640 1 -- 172.21.15.90:0/275648356 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc58070ec0 con 0x7fdc58108310 2024-04-02T14:41:24.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.299+0000 7fdc5e600640 1 --2- 172.21.15.90:0/275648356 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdc5806c9b0 0x7fdc5806ce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:41:24.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.300+0000 7fdc5dc00640 1 --2- 172.21.15.90:0/275648356 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdc58108310 0x7fdc580733a0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7fdc4c00ea60 tx=0x7fdc4c00ef20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:41:24.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.300+0000 7fdc56000640 1 -- 172.21.15.90:0/275648356 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc4c00cde0 con 0x7fdc58108310 2024-04-02T14:41:24.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.300+0000 7fdc56000640 1 -- 172.21.15.90:0/275648356 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdc4c0035f0 con 0x7fdc58108310 2024-04-02T14:41:24.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.300+0000 7fdc56000640 1 -- 172.21.15.90:0/275648356 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc4c010670 con 0x7fdc58108310 2024-04-02T14:41:24.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.300+0000 7fdc60600640 1 -- 172.21.15.90:0/275648356 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc580711b0 con 0x7fdc58108310 2024-04-02T14:41:24.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.300+0000 7fdc60600640 1 -- 172.21.15.90:0/275648356 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc580716b0 con 0x7fdc58108310 2024-04-02T14:41:24.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.309+0000 7fdc56000640 1 -- 172.21.15.90:0/275648356 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fdc4c0037a0 con 0x7fdc58108310 2024-04-02T14:41:24.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.311+0000 7fdc56000640 1 --2- 172.21.15.90:0/275648356 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fdc400a3f20 0x7fdc400a63e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:24.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.311+0000 7fdc56000640 1 -- 172.21.15.90:0/275648356 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7fdc4c0d0bc0 con 0x7fdc58108310 2024-04-02T14:41:24.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.312+0000 7fdc5d200640 1 --2- 172.21.15.90:0/275648356 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fdc400a3f20 0x7fdc400a63e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:24.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.312+0000 7fdc633ad740 1 Processor -- start 2024-04-02T14:41:24.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.313+0000 7fdc633ad740 1 -- start start 2024-04-02T14:41:24.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.313+0000 7fdc633ad740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5650aca939f0 0x5650aca92740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:24.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.313+0000 7fdc633ad740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5650aca92d10 0x5650aca93190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:24.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.313+0000 7fdc633ad740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5650aca9c080 0x5650aca9e470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:24.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.313+0000 7fdc633ad740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5650ac9d3400 con 0x5650aca92d10 2024-04-02T14:41:24.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.313+0000 7fdc633ad740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x5650ac9b7f10 con 0x5650aca9c080 2024-04-02T14:41:24.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.313+0000 7fdc633ad740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5650ac9f76f0 con 0x5650aca939f0 2024-04-02T14:41:24.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.314+0000 7fdc5e600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5650aca9c080 0x5650aca9e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:24.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.314+0000 7fdc5e600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5650aca9c080 0x5650aca9e470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:35282/0 (socket says 172.21.15.90:35282) 2024-04-02T14:41:24.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.314+0000 7fdc5e600640 1 -- 172.21.15.90:0/396520840 learned_addr learned my addr 172.21.15.90:0/396520840 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:41:24.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.314+0000 7fdc5dc00640 1 --2- 172.21.15.90:0/396520840 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5650aca939f0 0x5650aca92740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:24.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.315+0000 7fdc5e600640 1 -- 172.21.15.90:0/396520840 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5650aca939f0 msgr2=0x5650aca92740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:41:24.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.315+0000 7fdc5e600640 1 --2- 172.21.15.90:0/396520840 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5650aca939f0 0x5650aca92740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:24.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.315+0000 7fdc5e600640 1 -- 172.21.15.90:0/396520840 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5650aca92d10 msgr2=0x5650aca93190 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-02T14:41:24.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.315+0000 7fdc5d200640 1 --2- 172.21.15.90:0/275648356 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fdc400a3f20 0x7fdc400a63e0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fdc44009850 tx=0x7fdc44000bc0 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:41:24.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.315+0000 7fdc5e600640 1 --2- 172.21.15.90:0/396520840 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5650aca92d10 0x5650aca93190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:24.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.315+0000 7fdc5e600640 1 -- 172.21.15.90:0/396520840 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5650ac70b3f0 con 0x5650aca9c080 2024-04-02T14:41:24.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.316+0000 7fdc5e600640 1 --2- 172.21.15.90:0/396520840 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5650aca9c080 0x5650aca9e470 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7fdc50009f10 tx=0x7fdc5000f730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d9cc53d383f8e588 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:41:24.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.316+0000 7fdc4aa00640 1 -- 172.21.15.90:0/396520840 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc50003b70 con 0x5650aca9c080 2024-04-02T14:41:24.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.316+0000 7fdc4aa00640 1 -- 172.21.15.90:0/396520840 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdc50003d10 con 0x5650aca9c080 2024-04-02T14:41:24.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.316+0000 7fdc4aa00640 1 -- 172.21.15.90:0/396520840 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc50011a70 con 0x5650aca9c080 2024-04-02T14:41:24.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.318+0000 7fdc633ad740 1 -- 172.21.15.90:0/396520840 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5650aca9c080 msgr2=0x5650aca9e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:41:24.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.318+0000 7fdc633ad740 1 --2- 172.21.15.90:0/396520840 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5650aca9c080 0x5650aca9e470 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7fdc50009f10 tx=0x7fdc5000f730 comp rx=0 tx=0).stop 2024-04-02T14:41:24.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.318+0000 7fdc633ad740 1 -- 172.21.15.90:0/396520840 shutdown_connections 2024-04-02T14:41:24.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.318+0000 7fdc633ad740 1 --2- 172.21.15.90:0/396520840 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5650aca9c080 0x5650aca9e470 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:24.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.318+0000 7fdc633ad740 1 --2- 172.21.15.90:0/396520840 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5650aca92d10 0x5650aca93190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:24.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.318+0000 7fdc633ad740 1 --2- 172.21.15.90:0/396520840 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5650aca939f0 0x5650aca92740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:24.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.318+0000 7fdc633ad740 1 -- 172.21.15.90:0/396520840 >> 172.21.15.90:0/396520840 conn(0x5650aca59a50 msgr2=0x5650aca8d760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:41:24.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.320+0000 7fdc633ad740 1 -- 172.21.15.90:0/396520840 shutdown_connections 2024-04-02T14:41:24.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.320+0000 7fdc633ad740 1 -- 172.21.15.90:0/396520840 wait complete. 2024-04-02T14:41:24.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.321+0000 7fdc633ad740 1 -- start start 2024-04-02T14:41:24.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.321+0000 7fdc633ad740 1 Processor -- start 2024-04-02T14:41:24.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.322+0000 7fdc633ad740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5650aca92d10 0x5650aca99160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:24.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.322+0000 7fdc633ad740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5650aca939f0 0x5650aca996a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:24.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.322+0000 7fdc633ad740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5650aca9c080 0x5650aca99be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:24.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.323+0000 7fdc633ad740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5650ac9f36a0 con 0x5650aca939f0 2024-04-02T14:41:24.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.323+0000 7fdc633ad740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x5650ac99f980 con 0x5650aca92d10 2024-04-02T14:41:24.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.323+0000 7fdc633ad740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5650ac9b2cb0 con 0x5650aca9c080 2024-04-02T14:41:24.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.323+0000 7fdc5d200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5650aca939f0 0x5650aca996a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:24.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.323+0000 7fdc633ad740 10 client.0 _ll_register_callbacks cb 0x5650ac9b26a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:41:24.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.323+0000 7fdc5dc00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5650aca92d10 0x5650aca99160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:24.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.323+0000 7fdc5dc00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5650aca92d10 0x5650aca99160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:35294/0 (socket says 172.21.15.90:35294) 2024-04-02T14:41:24.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.323+0000 7fdc5dc00640 1 -- 172.21.15.90:0/1189780195 learned_addr learned my addr 172.21.15.90:0/1189780195 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:41:24.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.324+0000 7fdc5e600640 1 --2- 172.21.15.90:0/1189780195 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5650aca9c080 0x5650aca99be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:24.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.324+0000 7fdc5dc00640 1 -- 172.21.15.90:0/1189780195 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5650aca9c080 msgr2=0x5650aca99be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:41:24.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.324+0000 7fdc5dc00640 1 --2- 172.21.15.90:0/1189780195 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5650aca9c080 0x5650aca99be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:24.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.324+0000 7fdc5dc00640 1 -- 172.21.15.90:0/1189780195 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5650aca939f0 msgr2=0x5650aca996a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:41:24.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.325+0000 7fdc5dc00640 1 --2- 172.21.15.90:0/1189780195 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5650aca939f0 0x5650aca996a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:24.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.325+0000 7fdc5dc00640 1 -- 172.21.15.90:0/1189780195 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5650ac7d57d0 con 0x5650aca92d10 2024-04-02T14:41:24.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.326+0000 7fdc5d200640 1 --2- 172.21.15.90:0/1189780195 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5650aca939f0 0x5650aca996a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:41:24.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.326+0000 7fdc5dc00640 1 --2- 172.21.15.90:0/1189780195 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5650aca92d10 0x5650aca99160 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fdc4c098f70 tx=0x7fdc4c09c760 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f6cafb2435341556 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:41:24.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.326+0000 7fdc3f400640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:41:24.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.326+0000 7fdc3f400640 1 -- 172.21.15.90:0/1189780195 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc4c0b5070 con 0x5650aca92d10 2024-04-02T14:41:24.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.326+0000 7fdc3f400640 1 -- 172.21.15.90:0/1189780195 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7fdc4c003b30 con 0x5650aca92d10 2024-04-02T14:41:24.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.326+0000 7fdc633ad740 1 -- 172.21.15.90:0/1189780195 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdc50003d10 con 0x5650aca92d10 2024-04-02T14:41:24.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.327+0000 7fdc3f400640 1 -- 172.21.15.90:0/1189780195 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fdc4c099180 con 0x5650aca92d10 2024-04-02T14:41:24.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.327+0000 7fdc633ad740 10 client.27380 fetch_fsmap learned FSMap version 27 2024-04-02T14:41:24.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.327+0000 7fdc633ad740 1 -- 172.21.15.90:0/1189780195 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5650acaab070 con 0x5650aca92d10 2024-04-02T14:41:24.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.328+0000 7fdc3f400640 1 -- 172.21.15.90:0/1189780195 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdc50003d10 con 0x5650aca92d10 2024-04-02T14:41:24.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.328+0000 7fdc3f400640 1 -- 172.21.15.90:0/1189780195 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fdc4c09bf60 con 0x5650aca92d10 2024-04-02T14:41:24.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.329+0000 7fdc633ad740 10 client.27380 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:41:24.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.329+0000 7fdc633ad740 10 client.27380 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-04-02T14:41:24.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.329+0000 7fdc633ad740 20 client.27380 populate_metadata read hostname 'smithi090' 2024-04-02T14:41:24.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.329+0000 7fdc633ad740 1 --2- 172.21.15.90:0/1189780195 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x5650acaac4d0 0x5650acaae940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:24.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.329+0000 7fdc633ad740 4 client.27380 mds_command: new command op to 24547 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:41:24.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.329+0000 7fdc633ad740 1 -- 172.21.15.90:0/1189780195 --> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5650ac9b7040 con 0x5650acaac4d0 2024-04-02T14:41:24.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.331+0000 7fdc5d200640 1 --2- 172.21.15.90:0/1189780195 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x5650acaac4d0 0x5650acaae940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:24.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.332+0000 7fdc5d200640 1 --2- 172.21.15.90:0/1189780195 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x5650acaac4d0 0x5650acaae940 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=8bd0a95db0fe287c server_cookie=9c5559cadda14195 in_seq=0 out_seq=0 2024-04-02T14:41:24.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.333+0000 7fdc3f400640 10 client.27380 ms_handle_connect on v2:172.21.15.177:6838/664895663 2024-04-02T14:41:24.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.333+0000 7fdc3f400640 1 -- 172.21.15.90:0/1189780195 <== mds.2 v2:172.21.15.177:6838/664895663 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5650ac9b7040 con 0x5650acaac4d0 2024-04-02T14:41:24.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.334+0000 7fdc3f400640 10 client.27380 handle_command_reply: tid=0 2024-04-02T14:41:24.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.334+0000 7fdc633ad740 1 client.27380 shutdown 2024-04-02T14:41:24.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.335+0000 7fdc633ad740 1 -- 172.21.15.90:0/1189780195 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5650aca92d10 msgr2=0x5650aca99160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:41:24.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.335+0000 7fdc633ad740 1 --2- 172.21.15.90:0/1189780195 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5650aca92d10 0x5650aca99160 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fdc4c098f70 tx=0x7fdc4c09c760 comp rx=0 tx=0).stop 2024-04-02T14:41:24.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.336+0000 7fdc633ad740 1 -- 172.21.15.90:0/1189780195 shutdown_connections 2024-04-02T14:41:24.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.336+0000 7fdc633ad740 1 --2- 172.21.15.90:0/1189780195 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x5650acaac4d0 0x5650acaae940 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:24.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.336+0000 7fdc633ad740 1 --2- 172.21.15.90:0/1189780195 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5650aca9c080 0x5650aca99be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:24.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.336+0000 7fdc633ad740 1 --2- 172.21.15.90:0/1189780195 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5650aca939f0 0x5650aca996a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:24.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.336+0000 7fdc633ad740 1 --2- 172.21.15.90:0/1189780195 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5650aca92d10 0x5650aca99160 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:24.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.336+0000 7fdc3f400640 0 client.27380 ms_handle_reset on v2:172.21.15.177:6838/664895663 2024-04-02T14:41:24.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.336+0000 7fdc633ad740 1 -- 172.21.15.90:0/1189780195 >> 172.21.15.90:0/1189780195 conn(0x5650aca59a50 msgr2=0x5650aca8f4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:41:24.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.336+0000 7fdc633ad740 1 -- 172.21.15.90:0/1189780195 shutdown_connections 2024-04-02T14:41:24.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.336+0000 7fdc633ad740 1 -- 172.21.15.90:0/1189780195 wait complete. 2024-04-02T14:41:24.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.337+0000 7fdc633ad740 20 client.27380 trim_cache size 0 max 16384 2024-04-02T14:41:24.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.370+0000 7fdc633ad740 1 Processor -- start 2024-04-02T14:41:24.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.370+0000 7fdc633ad740 1 -- start start 2024-04-02T14:41:24.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.370+0000 7fdc633ad740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5650aca92d10 0x5650aca9b890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:24.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.370+0000 7fdc633ad740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5650aca939f0 0x5650acaa7af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:24.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.370+0000 7fdc633ad740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5650aca9c080 0x5650aca982c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:24.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.370+0000 7fdc633ad740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fdc581bc710 con 0x5650aca9c080 2024-04-02T14:41:24.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.370+0000 7fdc633ad740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fdc581be250 con 0x5650aca939f0 2024-04-02T14:41:24.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.370+0000 7fdc633ad740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fdc581bcde0 con 0x5650aca92d10 2024-04-02T14:41:24.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.371+0000 7fdc5dc00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5650aca92d10 0x5650aca9b890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:24.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.371+0000 7fdc5dc00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5650aca92d10 0x5650aca9b890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:42476/0 (socket says 172.21.15.90:42476) 2024-04-02T14:41:24.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.371+0000 7fdc5dc00640 1 -- 172.21.15.90:0/3052129540 learned_addr learned my addr 172.21.15.90:0/3052129540 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:41:24.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.371+0000 7fdc5e600640 1 --2- 172.21.15.90:0/3052129540 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5650aca9c080 0x5650aca982c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:24.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.371+0000 7fdc5d200640 1 --2- 172.21.15.90:0/3052129540 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5650aca939f0 0x5650acaa7af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:24.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.372+0000 7fdc5dc00640 1 -- 172.21.15.90:0/3052129540 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5650aca939f0 msgr2=0x5650acaa7af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:41:24.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.372+0000 7fdc5dc00640 1 --2- 172.21.15.90:0/3052129540 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5650aca939f0 0x5650acaa7af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:24.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.372+0000 7fdc5dc00640 1 -- 172.21.15.90:0/3052129540 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5650aca9c080 msgr2=0x5650aca982c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:41:24.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.372+0000 7fdc5dc00640 1 --2- 172.21.15.90:0/3052129540 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5650aca9c080 0x5650aca982c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:24.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.372+0000 7fdc5dc00640 1 -- 172.21.15.90:0/3052129540 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5650aca91d40 con 0x5650aca92d10 2024-04-02T14:41:24.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.372+0000 7fdc5e600640 1 --2- 172.21.15.90:0/3052129540 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5650aca9c080 0x5650aca982c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:41:24.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.372+0000 7fdc5dc00640 1 --2- 172.21.15.90:0/3052129540 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5650aca92d10 0x5650aca9b890 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fdc4c09caa0 tx=0x7fdc4c09b020 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d97b9b119e1878c7 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:41:24.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.373+0000 7fdc4a000640 1 -- 172.21.15.90:0/3052129540 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc4c099320 con 0x5650aca92d10 2024-04-02T14:41:24.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.373+0000 7fdc4a000640 1 -- 172.21.15.90:0/3052129540 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdc4c098950 con 0x5650aca92d10 2024-04-02T14:41:24.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.373+0000 7fdc4a000640 1 -- 172.21.15.90:0/3052129540 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc4c098be0 con 0x5650aca92d10 2024-04-02T14:41:24.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.374+0000 7fdc633ad740 1 -- 172.21.15.90:0/3052129540 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5650aca92d10 msgr2=0x5650aca9b890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:41:24.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.374+0000 7fdc633ad740 1 --2- 172.21.15.90:0/3052129540 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5650aca92d10 0x5650aca9b890 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fdc4c09caa0 tx=0x7fdc4c09b020 comp rx=0 tx=0).stop 2024-04-02T14:41:24.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.374+0000 7fdc633ad740 1 -- 172.21.15.90:0/3052129540 shutdown_connections 2024-04-02T14:41:24.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.374+0000 7fdc633ad740 1 --2- 172.21.15.90:0/3052129540 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5650aca9c080 0x5650aca982c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:24.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.374+0000 7fdc633ad740 1 --2- 172.21.15.90:0/3052129540 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5650aca939f0 0x5650acaa7af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:24.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.374+0000 7fdc633ad740 1 --2- 172.21.15.90:0/3052129540 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5650aca92d10 0x5650aca9b890 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:24.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.374+0000 7fdc633ad740 1 -- 172.21.15.90:0/3052129540 >> 172.21.15.90:0/3052129540 conn(0x5650aca59a50 msgr2=0x5650aca99120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:41:24.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.374+0000 7fdc633ad740 1 -- 172.21.15.90:0/3052129540 shutdown_connections 2024-04-02T14:41:24.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.374+0000 7fdc633ad740 1 -- 172.21.15.90:0/3052129540 wait complete. 2024-04-02T14:41:24.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.375+0000 7fdc633ad740 1 -- start start 2024-04-02T14:41:24.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.375+0000 7fdc633ad740 1 Processor -- start 2024-04-02T14:41:24.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.376+0000 7fdc633ad740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5650aca92d10 0x5650acaa42d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:24.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.376+0000 7fdc633ad740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5650aca939f0 0x5650aca963e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:24.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.376+0000 7fdc633ad740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5650aca9c080 0x5650aca96920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:24.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.376+0000 7fdc633ad740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fdc581bdb80 con 0x5650aca92d10 2024-04-02T14:41:24.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.376+0000 7fdc633ad740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fdc581bd4b0 con 0x5650aca9c080 2024-04-02T14:41:24.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.376+0000 7fdc633ad740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fdc581ba6f0 con 0x5650aca939f0 2024-04-02T14:41:24.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.376+0000 7fdc633ad740 10 client.0 _ll_register_callbacks cb 0x7fdc4c09d210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:41:24.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.376+0000 7fdc5e600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5650aca9c080 0x5650aca96920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:24.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.376+0000 7fdc5d200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5650aca939f0 0x5650aca963e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:24.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.377+0000 7fdc5d200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5650aca939f0 0x5650aca963e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:42484/0 (socket says 172.21.15.90:42484) 2024-04-02T14:41:24.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.377+0000 7fdc5d200640 1 -- 172.21.15.90:0/2983789425 learned_addr learned my addr 172.21.15.90:0/2983789425 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:41:24.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.377+0000 7fdc5d200640 1 -- 172.21.15.90:0/2983789425 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5650aca9c080 msgr2=0x5650aca96920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:41:24.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.377+0000 7fdc5d200640 1 --2- 172.21.15.90:0/2983789425 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5650aca9c080 0x5650aca96920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:24.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.377+0000 7fdc5dc00640 1 --2- 172.21.15.90:0/2983789425 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5650aca92d10 0x5650acaa42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:24.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.377+0000 7fdc5d200640 1 -- 172.21.15.90:0/2983789425 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5650aca92d10 msgr2=0x5650acaa42d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:41:24.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.377+0000 7fdc5d200640 1 --2- 172.21.15.90:0/2983789425 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5650aca92d10 0x5650acaa42d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:24.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.377+0000 7fdc5d200640 1 -- 172.21.15.90:0/2983789425 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5650aca96fc0 con 0x5650aca939f0 2024-04-02T14:41:24.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.377+0000 7fdc5dc00640 1 --2- 172.21.15.90:0/2983789425 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5650aca92d10 0x5650acaa42d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:41:24.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.378+0000 7fdc5d200640 1 --2- 172.21.15.90:0/2983789425 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5650aca939f0 0x5650aca963e0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fdc44004150 tx=0x7fdc44005fd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9e240f922a66e59b server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:41:24.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.378+0000 7fdc3f400640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:41:24.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.378+0000 7fdc3f400640 1 -- 172.21.15.90:0/2983789425 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc44039040 con 0x5650aca939f0 2024-04-02T14:41:24.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.378+0000 7fdc633ad740 1 -- 172.21.15.90:0/2983789425 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdc4c098950 con 0x5650aca939f0 2024-04-02T14:41:24.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.378+0000 7fdc3f400640 1 -- 172.21.15.90:0/2983789425 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7fdc4404e430 con 0x5650aca939f0 2024-04-02T14:41:24.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.378+0000 7fdc3f400640 1 -- 172.21.15.90:0/2983789425 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fdc44039040 con 0x5650aca939f0 2024-04-02T14:41:24.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.379+0000 7fdc633ad740 10 client.27250 fetch_fsmap learned FSMap version 27 2024-04-02T14:41:24.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.379+0000 7fdc633ad740 1 -- 172.21.15.90:0/2983789425 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5650acb12a80 con 0x5650aca939f0 2024-04-02T14:41:24.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.379+0000 7fdc3f400640 1 -- 172.21.15.90:0/2983789425 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdc4c098950 con 0x5650aca939f0 2024-04-02T14:41:24.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.379+0000 7fdc3f400640 1 -- 172.21.15.90:0/2983789425 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fdc4403c030 con 0x5650aca939f0 2024-04-02T14:41:24.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.380+0000 7fdc633ad740 10 client.27250 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:41:24.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.380+0000 7fdc633ad740 10 client.27250 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-04-02T14:41:24.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.380+0000 7fdc633ad740 20 client.27250 populate_metadata read hostname 'smithi090' 2024-04-02T14:41:24.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.380+0000 7fdc633ad740 1 --2- 172.21.15.90:0/2983789425 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x5650acb130a0 0x5650acb154c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:24.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.380+0000 7fdc633ad740 4 client.27250 mds_command: new command op to 24547 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-02T14:41:24.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.380+0000 7fdc633ad740 1 -- 172.21.15.90:0/2983789425 --> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5650aca04c30 con 0x5650acb130a0 2024-04-02T14:41:24.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.380+0000 7fdc5dc00640 1 --2- 172.21.15.90:0/2983789425 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x5650acb130a0 0x5650acb154c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:24.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.381+0000 7fdc3f400640 10 client.27250 ms_handle_connect on v2:172.21.15.177:6838/664895663 2024-04-02T14:41:24.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.382+0000 7fdc5dc00640 1 --2- 172.21.15.90:0/2983789425 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x5650acb130a0 0x5650acb154c0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=c7bfb331d61f7e30 server_cookie=756db685161e9224 in_seq=0 out_seq=0 2024-04-02T14:41:24.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.383+0000 7fdc3f400640 1 -- 172.21.15.90:0/2983789425 <== mds.2 v2:172.21.15.177:6838/664895663 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5650aca04c30 con 0x5650acb130a0 2024-04-02T14:41:24.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.383+0000 7fdc3f400640 10 client.27250 handle_command_reply: tid=0 2024-04-02T14:41:24.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.383+0000 7fdc633ad740 1 client.27250 shutdown 2024-04-02T14:41:24.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.383+0000 7fdc633ad740 1 -- 172.21.15.90:0/2983789425 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5650aca939f0 msgr2=0x5650aca963e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:41:24.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.383+0000 7fdc633ad740 1 --2- 172.21.15.90:0/2983789425 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5650aca939f0 0x5650aca963e0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fdc44004150 tx=0x7fdc44005fd0 comp rx=0 tx=0).stop 2024-04-02T14:41:24.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.383+0000 7fdc633ad740 1 -- 172.21.15.90:0/2983789425 shutdown_connections 2024-04-02T14:41:24.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.383+0000 7fdc633ad740 1 --2- 172.21.15.90:0/2983789425 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x5650acb130a0 0x5650acb154c0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:24.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.383+0000 7fdc633ad740 1 --2- 172.21.15.90:0/2983789425 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5650aca9c080 0x5650aca96920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:24.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.383+0000 7fdc633ad740 1 --2- 172.21.15.90:0/2983789425 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5650aca939f0 0x5650aca963e0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:24.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.383+0000 7fdc633ad740 1 --2- 172.21.15.90:0/2983789425 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5650aca92d10 0x5650acaa42d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:24.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.383+0000 7fdc3f400640 0 client.27250 ms_handle_reset on v2:172.21.15.177:6838/664895663 2024-04-02T14:41:24.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.383+0000 7fdc633ad740 1 -- 172.21.15.90:0/2983789425 >> 172.21.15.90:0/2983789425 conn(0x5650aca59a50 msgr2=0x5650aca8c010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:41:24.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.383+0000 7fdc633ad740 1 -- 172.21.15.90:0/2983789425 shutdown_connections 2024-04-02T14:41:24.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.384+0000 7fdc633ad740 1 -- 172.21.15.90:0/2983789425 wait complete. 2024-04-02T14:41:24.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.384+0000 7fdc633ad740 20 client.27250 trim_cache size 0 max 16384 2024-04-02T14:41:24.384 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-02T14:41:24.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.385+0000 7fdc60600640 1 -- 172.21.15.90:0/275648356 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fdc400a3f20 msgr2=0x7fdc400a63e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:41:24.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.385+0000 7fdc60600640 1 --2- 172.21.15.90:0/275648356 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fdc400a3f20 0x7fdc400a63e0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fdc44009850 tx=0x7fdc44000bc0 comp rx=0 tx=0).stop 2024-04-02T14:41:24.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.385+0000 7fdc60600640 1 -- 172.21.15.90:0/275648356 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdc58108310 msgr2=0x7fdc580733a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:41:24.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.385+0000 7fdc60600640 1 --2- 172.21.15.90:0/275648356 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdc58108310 0x7fdc580733a0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7fdc4c00ea60 tx=0x7fdc4c00ef20 comp rx=0 tx=0).stop 2024-04-02T14:41:24.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.386+0000 7fdc60600640 1 -- 172.21.15.90:0/275648356 shutdown_connections 2024-04-02T14:41:24.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.386+0000 7fdc60600640 1 --2- 172.21.15.90:0/275648356 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fdc400a3f20 0x7fdc400a63e0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:24.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.386+0000 7fdc60600640 1 --2- 172.21.15.90:0/275648356 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdc5806c9b0 0x7fdc5806ce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:24.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.386+0000 7fdc60600640 1 --2- 172.21.15.90:0/275648356 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdc5810d3e0 0x7fdc5806c470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:24.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.386+0000 7fdc60600640 1 --2- 172.21.15.90:0/275648356 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdc58108310 0x7fdc580733a0 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:24.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.386+0000 7fdc60600640 1 -- 172.21.15.90:0/275648356 >> 172.21.15.90:0/275648356 conn(0x7fdc58076040 msgr2=0x7fdc58077350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:41:24.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.386+0000 7fdc60600640 1 -- 172.21.15.90:0/275648356 shutdown_connections 2024-04-02T14:41:24.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.386+0000 7fdc60600640 1 -- 172.21.15.90:0/275648356 wait complete. 2024-04-02T14:41:24.413 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-02T14:41:24.652 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.653+0000 7f4959400640 1 -- 172.21.15.90:0/1446316077 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f49541059a0 msgr2=0x7f4954105d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:41:24.652 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.653+0000 7f4959400640 1 --2- 172.21.15.90:0/1446316077 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f49541059a0 0x7f4954105d80 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f494c009cd0 tx=0x7f494c02c0e0 comp rx=0 tx=0).stop 2024-04-02T14:41:24.653 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.654+0000 7f4959400640 1 -- 172.21.15.90:0/1446316077 shutdown_connections 2024-04-02T14:41:24.653 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.654+0000 7f4959400640 1 --2- 172.21.15.90:0/1446316077 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4954101610 0x7f495410d830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:24.653 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.654+0000 7f4959400640 1 --2- 172.21.15.90:0/1446316077 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4954100c50 0x7f49541010d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:24.653 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.654+0000 7f4959400640 1 --2- 172.21.15.90:0/1446316077 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f49541059a0 0x7f4954105d80 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:24.653 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.654+0000 7f4959400640 1 -- 172.21.15.90:0/1446316077 >> 172.21.15.90:0/1446316077 conn(0x7f4954077850 msgr2=0x7f4954077c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:41:24.653 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.654+0000 7f4959400640 1 -- 172.21.15.90:0/1446316077 shutdown_connections 2024-04-02T14:41:24.654 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.655+0000 7f4959400640 1 -- 172.21.15.90:0/1446316077 wait complete. 2024-04-02T14:41:24.656 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.657+0000 7f4959400640 1 Processor -- start 2024-04-02T14:41:24.657 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.657+0000 7f4959400640 1 -- start start 2024-04-02T14:41:24.657 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.658+0000 7f4959400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4954100c50 0x7f49541a4650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:24.657 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.658+0000 7f4959400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4954101610 0x7f49541a4b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:24.657 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.658+0000 7f4959400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f49541059a0 0x7f495419e810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:24.661 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.659+0000 7f4959400640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f495410fdc0 con 0x7f4954100c50 2024-04-02T14:41:24.661 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.659+0000 7f4959400640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f495410fc40 con 0x7f49541059a0 2024-04-02T14:41:24.661 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.659+0000 7f4959400640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f495410ff40 con 0x7f4954101610 2024-04-02T14:41:24.661 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.660+0000 7f4951e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4954101610 0x7f49541a4b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:24.662 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.660+0000 7f4951e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4954101610 0x7f49541a4b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:42504/0 (socket says 172.21.15.90:42504) 2024-04-02T14:41:24.662 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.660+0000 7f4951e00640 1 -- 172.21.15.90:0/98318128 learned_addr learned my addr 172.21.15.90:0/98318128 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:41:24.662 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.660+0000 7f4952800640 1 --2- 172.21.15.90:0/98318128 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4954100c50 0x7f49541a4650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:24.662 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.661+0000 7f4952800640 1 -- 172.21.15.90:0/98318128 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4954101610 msgr2=0x7f49541a4b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:41:24.662 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.661+0000 7f4952800640 1 --2- 172.21.15.90:0/98318128 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4954101610 0x7f49541a4b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:24.662 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.661+0000 7f4952800640 1 -- 172.21.15.90:0/98318128 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f49541059a0 msgr2=0x7f495419e810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:41:24.662 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.659+0000 7f4953200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f49541059a0 0x7f495419e810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:24.662 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.662+0000 7f4952800640 1 --2- 172.21.15.90:0/98318128 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f49541059a0 0x7f495419e810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:24.662 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.662+0000 7f4952800640 1 -- 172.21.15.90:0/98318128 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f495419efe0 con 0x7f4954100c50 2024-04-02T14:41:24.662 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.663+0000 7f4953200640 1 --2- 172.21.15.90:0/98318128 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f49541059a0 0x7f495419e810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:41:24.666 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.667+0000 7f4952800640 1 --2- 172.21.15.90:0/98318128 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4954100c50 0x7f49541a4650 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f494c009bf0 tx=0x7f494c035d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:41:24.667 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.668+0000 7f494b400640 1 -- 172.21.15.90:0/98318128 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f494c039040 con 0x7f4954100c50 2024-04-02T14:41:24.668 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.669+0000 7f494b400640 1 -- 172.21.15.90:0/98318128 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f494c044070 con 0x7f4954100c50 2024-04-02T14:41:24.668 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.669+0000 7f494b400640 1 -- 172.21.15.90:0/98318128 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f494c004330 con 0x7f4954100c50 2024-04-02T14:41:24.669 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.669+0000 7f4959400640 1 -- 172.21.15.90:0/98318128 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f495419f270 con 0x7f4954100c50 2024-04-02T14:41:24.669 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.669+0000 7f4959400640 1 -- 172.21.15.90:0/98318128 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49541ab4a0 con 0x7f4954100c50 2024-04-02T14:41:24.679 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.678+0000 7f494b400640 1 -- 172.21.15.90:0/98318128 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f494c0044d0 con 0x7f4954100c50 2024-04-02T14:41:24.680 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.681+0000 7f494b400640 1 --2- 172.21.15.90:0/98318128 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f49380a3fa0 0x7f49380a6460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:24.680 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.682+0000 7f494b400640 1 -- 172.21.15.90:0/98318128 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f494c0f2520 con 0x7f4954100c50 2024-04-02T14:41:24.682 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.683+0000 7f4951e00640 1 --2- 172.21.15.90:0/98318128 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f49380a3fa0 0x7f49380a6460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:24.683 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.684+0000 7f4951e00640 1 --2- 172.21.15.90:0/98318128 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f49380a3fa0 0x7f49380a6460 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f4954077e40 tx=0x7f494400b4a0 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:41:24.683 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.684+0000 7f495c063740 1 Processor -- start 2024-04-02T14:41:24.683 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.684+0000 7f495c063740 1 -- start start 2024-04-02T14:41:24.684 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.685+0000 7f495c063740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557ae75c6eb0 0x557ae75c7290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:24.685 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.685+0000 7f495c063740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557ae75c77d0 0x557ae75cfe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:24.685 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.685+0000 7f495c063740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ae75d03b0 0x557ae75d27a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:24.685 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.685+0000 7f495c063740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x557ae752bd70 con 0x557ae75c77d0 2024-04-02T14:41:24.685 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.685+0000 7f495c063740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x557ae74d42a0 con 0x557ae75c6eb0 2024-04-02T14:41:24.685 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.685+0000 7f495c063740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x557ae7527d20 con 0x557ae75d03b0 2024-04-02T14:41:24.685 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.685+0000 7f4952800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557ae75c6eb0 0x557ae75c7290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:24.685 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.685+0000 7f4952800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557ae75c6eb0 0x557ae75c7290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:35338/0 (socket says 172.21.15.90:35338) 2024-04-02T14:41:24.685 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.685+0000 7f4952800640 1 -- 172.21.15.90:0/1844428598 learned_addr learned my addr 172.21.15.90:0/1844428598 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:41:24.685 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.686+0000 7f4953200640 1 --2- 172.21.15.90:0/1844428598 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ae75d03b0 0x557ae75d27a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:24.685 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.686+0000 7f4951e00640 1 --2- 172.21.15.90:0/1844428598 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557ae75c77d0 0x557ae75cfe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:24.686 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.686+0000 7f4953200640 1 -- 172.21.15.90:0/1844428598 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557ae75c6eb0 msgr2=0x557ae75c7290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:41:24.686 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.686+0000 7f4953200640 1 --2- 172.21.15.90:0/1844428598 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557ae75c6eb0 0x557ae75c7290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:24.686 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.686+0000 7f4953200640 1 -- 172.21.15.90:0/1844428598 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557ae75c77d0 msgr2=0x557ae75cfe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:41:24.686 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.686+0000 7f4953200640 1 --2- 172.21.15.90:0/1844428598 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557ae75c77d0 0x557ae75cfe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:24.686 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.686+0000 7f4953200640 1 -- 172.21.15.90:0/1844428598 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557ae74e8840 con 0x557ae75d03b0 2024-04-02T14:41:24.686 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.687+0000 7f4953200640 1 --2- 172.21.15.90:0/1844428598 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ae75d03b0 0x557ae75d27a0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f494000bbb0 tx=0x7f4940005f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=20d78975c9a58173 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:41:24.686 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.687+0000 7f493f400640 1 -- 172.21.15.90:0/1844428598 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f494001b070 con 0x557ae75d03b0 2024-04-02T14:41:24.687 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.687+0000 7f493f400640 1 -- 172.21.15.90:0/1844428598 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f494000abc0 con 0x557ae75d03b0 2024-04-02T14:41:24.687 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.687+0000 7f493f400640 1 -- 172.21.15.90:0/1844428598 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4940003e20 con 0x557ae75d03b0 2024-04-02T14:41:24.687 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.688+0000 7f495c063740 1 -- 172.21.15.90:0/1844428598 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ae75d03b0 msgr2=0x557ae75d27a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:41:24.687 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.688+0000 7f495c063740 1 --2- 172.21.15.90:0/1844428598 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ae75d03b0 0x557ae75d27a0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f494000bbb0 tx=0x7f4940005f90 comp rx=0 tx=0).stop 2024-04-02T14:41:24.688 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.690+0000 7f495c063740 1 -- 172.21.15.90:0/1844428598 shutdown_connections 2024-04-02T14:41:24.689 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.690+0000 7f495c063740 1 --2- 172.21.15.90:0/1844428598 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ae75d03b0 0x557ae75d27a0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:24.689 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.690+0000 7f495c063740 1 --2- 172.21.15.90:0/1844428598 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557ae75c77d0 0x557ae75cfe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:24.689 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.690+0000 7f495c063740 1 --2- 172.21.15.90:0/1844428598 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557ae75c6eb0 0x557ae75c7290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:24.689 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.690+0000 7f495c063740 1 -- 172.21.15.90:0/1844428598 >> 172.21.15.90:0/1844428598 conn(0x557ae75c15e0 msgr2=0x557ae75c19c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:41:24.689 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.690+0000 7f495c063740 1 -- 172.21.15.90:0/1844428598 shutdown_connections 2024-04-02T14:41:24.692 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.691+0000 7f495c063740 1 -- 172.21.15.90:0/1844428598 wait complete. 2024-04-02T14:41:24.692 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.691+0000 7f495c063740 1 -- start start 2024-04-02T14:41:24.692 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.692+0000 7f495c063740 1 Processor -- start 2024-04-02T14:41:24.694 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.695+0000 7f495c063740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557ae75c6eb0 0x557ae75cd470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:24.694 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.695+0000 7f495c063740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557ae75c77d0 0x557ae75cd9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:24.696 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.696+0000 7f495c063740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ae75d03b0 0x557ae75cdef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:24.696 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.696+0000 7f4952800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557ae75c6eb0 0x557ae75cd470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:24.696 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.696+0000 7f4952800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557ae75c6eb0 0x557ae75cd470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:35354/0 (socket says 172.21.15.90:35354) 2024-04-02T14:41:24.696 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.697+0000 7f4951e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557ae75c77d0 0x557ae75cd9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:24.696 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.697+0000 7f4953200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ae75d03b0 0x557ae75cdef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:24.696 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.697+0000 7f4953200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ae75d03b0 0x557ae75cdef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:42530/0 (socket says 172.21.15.90:42530) 2024-04-02T14:41:24.696 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.697+0000 7f4951e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557ae75c77d0 0x557ae75cd9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:57266/0 (socket says 172.21.15.90:57266) 2024-04-02T14:41:24.696 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.697+0000 7f4951e00640 1 -- 172.21.15.90:0/1210873699 learned_addr learned my addr 172.21.15.90:0/1210873699 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:41:24.697 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.697+0000 7f495c063740 1 -- 172.21.15.90:0/1210873699 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x557ae75d31a0 con 0x557ae75c77d0 2024-04-02T14:41:24.697 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.697+0000 7f495c063740 1 -- 172.21.15.90:0/1210873699 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x557ae74e7660 con 0x557ae75c6eb0 2024-04-02T14:41:24.697 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.697+0000 7f495c063740 1 -- 172.21.15.90:0/1210873699 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x557ae75d3320 con 0x557ae75d03b0 2024-04-02T14:41:24.697 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.697+0000 7f495c063740 10 client.0 _ll_register_callbacks cb 0x557ae74e7050 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:41:24.698 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.699+0000 7f4953200640 1 -- 172.21.15.90:0/1210873699 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557ae75c6eb0 msgr2=0x557ae75cd470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:41:24.698 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.699+0000 7f4953200640 1 --2- 172.21.15.90:0/1210873699 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557ae75c6eb0 0x557ae75cd470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:24.698 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.699+0000 7f4953200640 1 -- 172.21.15.90:0/1210873699 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557ae75c77d0 msgr2=0x557ae75cd9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:41:24.698 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.699+0000 7f4953200640 1 --2- 172.21.15.90:0/1210873699 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557ae75c77d0 0x557ae75cd9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:24.698 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.699+0000 7f4953200640 1 -- 172.21.15.90:0/1210873699 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557ae74f6e80 con 0x557ae75d03b0 2024-04-02T14:41:24.699 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.699+0000 7f4953200640 1 --2- 172.21.15.90:0/1210873699 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ae75d03b0 0x557ae75cdef0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f4940003630 tx=0x7f49400042e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a96a35ddd246a57e server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:41:24.699 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.699+0000 7f4933e00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:41:24.699 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.700+0000 7f4933e00640 1 -- 172.21.15.90:0/1210873699 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f494001b070 con 0x557ae75d03b0 2024-04-02T14:41:24.699 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.700+0000 7f4933e00640 1 -- 172.21.15.90:0/1210873699 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f49400043d0 con 0x557ae75d03b0 2024-04-02T14:41:24.699 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.700+0000 7f495c063740 1 -- 172.21.15.90:0/1210873699 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f494000abc0 con 0x557ae75d03b0 2024-04-02T14:41:24.699 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.700+0000 7f4933e00640 1 -- 172.21.15.90:0/1210873699 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f494000b800 con 0x557ae75d03b0 2024-04-02T14:41:24.699 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.700+0000 7f495c063740 10 client.27262 fetch_fsmap learned FSMap version 27 2024-04-02T14:41:24.700 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.700+0000 7f495c063740 1 -- 172.21.15.90:0/1210873699 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x557ae7240210 con 0x557ae75d03b0 2024-04-02T14:41:24.700 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.701+0000 7f4933e00640 1 -- 172.21.15.90:0/1210873699 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f494001c040 con 0x557ae75d03b0 2024-04-02T14:41:24.700 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.701+0000 7f4933e00640 1 -- 172.21.15.90:0/1210873699 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f4940004b50 con 0x557ae75d03b0 2024-04-02T14:41:24.700 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.701+0000 7f495c063740 10 client.27262 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:41:24.700 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.701+0000 7f495c063740 10 client.27262 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-02T14:41:24.700 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.701+0000 7f495c063740 20 client.27262 populate_metadata read hostname 'smithi090' 2024-04-02T14:41:24.701 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.701+0000 7f495c063740 1 --2- 172.21.15.90:0/1210873699 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x557ae75e0710 0x557ae75e2b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:24.701 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.701+0000 7f495c063740 4 client.27262 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:41:24.701 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.701+0000 7f495c063740 1 -- 172.21.15.90:0/1210873699 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f494000abc0 con 0x557ae75e0710 2024-04-02T14:41:24.701 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.702+0000 7f4952800640 1 --2- 172.21.15.90:0/1210873699 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x557ae75e0710 0x557ae75e2b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:24.703 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.705+0000 7f4933e00640 10 client.27262 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:41:24.704 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.705+0000 7f4952800640 1 --2- 172.21.15.90:0/1210873699 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x557ae75e0710 0x557ae75e2b00 crc :-1 s=READY pgs=180 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2528394e52f423f5 server_cookie=23f649ba6cec25f3 in_seq=0 out_seq=0 2024-04-02T14:41:24.705 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.706+0000 7f4933e00640 1 -- 172.21.15.90:0/1210873699 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x7f494000abc0 con 0x557ae75e0710 2024-04-02T14:41:24.705 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.706+0000 7f4933e00640 10 client.27262 handle_command_reply: tid=0 2024-04-02T14:41:24.705 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.706+0000 7f495c063740 1 client.27262 shutdown 2024-04-02T14:41:24.706 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.707+0000 7f495c063740 1 -- 172.21.15.90:0/1210873699 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ae75d03b0 msgr2=0x557ae75cdef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:41:24.706 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.707+0000 7f495c063740 1 --2- 172.21.15.90:0/1210873699 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ae75d03b0 0x557ae75cdef0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f4940003630 tx=0x7f49400042e0 comp rx=0 tx=0).stop 2024-04-02T14:41:24.706 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.707+0000 7f495c063740 1 -- 172.21.15.90:0/1210873699 shutdown_connections 2024-04-02T14:41:24.706 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.707+0000 7f495c063740 1 --2- 172.21.15.90:0/1210873699 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x557ae75e0710 0x557ae75e2b00 crc :-1 s=READY pgs=180 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:24.707 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.707+0000 7f495c063740 1 --2- 172.21.15.90:0/1210873699 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ae75d03b0 0x557ae75cdef0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:24.707 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.707+0000 7f495c063740 1 --2- 172.21.15.90:0/1210873699 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557ae75c77d0 0x557ae75cd9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:24.707 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.707+0000 7f4933e00640 0 client.27262 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:41:24.707 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.707+0000 7f495c063740 1 --2- 172.21.15.90:0/1210873699 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557ae75c6eb0 0x557ae75cd470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:24.707 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.707+0000 7f495c063740 1 -- 172.21.15.90:0/1210873699 >> 172.21.15.90:0/1210873699 conn(0x557ae75c15e0 msgr2=0x557ae75c3330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:41:24.707 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.708+0000 7f495c063740 1 -- 172.21.15.90:0/1210873699 shutdown_connections 2024-04-02T14:41:24.707 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.708+0000 7f495c063740 1 -- 172.21.15.90:0/1210873699 wait complete. 2024-04-02T14:41:24.709 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.710+0000 7f495c063740 20 client.27262 trim_cache size 0 max 16384 2024-04-02T14:41:24.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.744+0000 7f929b200640 1 -- 172.21.15.90:0/1741780571 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9294107aa0 msgr2=0x7f9294107e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:41:24.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.744+0000 7f929b200640 1 --2- 172.21.15.90:0/1741780571 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9294107aa0 0x7f9294107e80 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f928c009ca0 tx=0x7f928c02c100 comp rx=0 tx=0).stop 2024-04-02T14:41:24.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.745+0000 7f929b200640 1 -- 172.21.15.90:0/1741780571 shutdown_connections 2024-04-02T14:41:24.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.745+0000 7f929b200640 1 --2- 172.21.15.90:0/1741780571 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9294102450 0x7f929410e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:24.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.745+0000 7f929b200640 1 --2- 172.21.15.90:0/1741780571 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9294101a90 0x7f9294101f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:24.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.745+0000 7f929b200640 1 --2- 172.21.15.90:0/1741780571 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9294107aa0 0x7f9294107e80 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:24.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.745+0000 7f929b200640 1 -- 172.21.15.90:0/1741780571 >> 172.21.15.90:0/1741780571 conn(0x7f9294078f00 msgr2=0x7f9294079310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:41:24.745 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.746+0000 7f929b200640 1 -- 172.21.15.90:0/1741780571 shutdown_connections 2024-04-02T14:41:24.745 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.746+0000 7f929b200640 1 -- 172.21.15.90:0/1741780571 wait complete. 2024-04-02T14:41:24.745 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.746+0000 7f495c063740 1 Processor -- start 2024-04-02T14:41:24.746 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.746+0000 7f495c063740 1 -- start start 2024-04-02T14:41:24.746 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.747+0000 7f495c063740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557ae75c6eb0 0x557ae75cfba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:24.746 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.747+0000 7f495c063740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ae75c77d0 0x557ae75dc060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:24.746 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.747+0000 7f495c063740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557ae75d03b0 0x557ae75e8270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:24.746 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.747+0000 7f495c063740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f49541b3cc0 con 0x557ae75d03b0 2024-04-02T14:41:24.746 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.747+0000 7f495c063740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f49541b5800 con 0x557ae75c6eb0 2024-04-02T14:41:24.746 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.747+0000 7f495c063740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f49541b4390 con 0x557ae75c77d0 2024-04-02T14:41:24.746 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.747+0000 7f4952800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557ae75c6eb0 0x557ae75cfba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:24.746 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.747+0000 7f4952800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557ae75c6eb0 0x557ae75cfba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:35374/0 (socket says 172.21.15.90:35374) 2024-04-02T14:41:24.747 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.747+0000 7f4952800640 1 -- 172.21.15.90:0/1915366815 learned_addr learned my addr 172.21.15.90:0/1915366815 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:41:24.747 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.747+0000 7f4951e00640 1 --2- 172.21.15.90:0/1915366815 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ae75c77d0 0x557ae75dc060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:24.747 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.748+0000 7f4952800640 1 -- 172.21.15.90:0/1915366815 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ae75c77d0 msgr2=0x557ae75dc060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:41:24.747 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.748+0000 7f4952800640 1 --2- 172.21.15.90:0/1915366815 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ae75c77d0 0x557ae75dc060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:24.747 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.748+0000 7f4952800640 1 -- 172.21.15.90:0/1915366815 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557ae75d03b0 msgr2=0x557ae75e8270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:41:24.747 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.748+0000 7f4952800640 1 --2- 172.21.15.90:0/1915366815 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557ae75d03b0 0x557ae75e8270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:24.747 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.748+0000 7f4952800640 1 -- 172.21.15.90:0/1915366815 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557ae75d2b10 con 0x557ae75c6eb0 2024-04-02T14:41:24.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.748+0000 7f929b200640 1 Processor -- start 2024-04-02T14:41:24.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.749+0000 7f929b200640 1 -- start start 2024-04-02T14:41:24.749 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.749+0000 7f4952800640 1 --2- 172.21.15.90:0/1915366815 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557ae75c6eb0 0x557ae75cfba0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f494c0baf50 tx=0x7f494c000f40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a0557ace968fdcd8 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:41:24.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.750+0000 7f929b200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9294101a90 0x7f92941a4620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:24.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.750+0000 7f929b200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9294102450 0x7f92941a4b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:24.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.750+0000 7f929b200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9294107aa0 0x7f929419e790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:24.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.750+0000 7f929b200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f929410fd00 con 0x7f9294107aa0 2024-04-02T14:41:24.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.750+0000 7f929b200640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f929410fb80 con 0x7f9294102450 2024-04-02T14:41:24.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.750+0000 7f929b200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f929410fe80 con 0x7f9294101a90 2024-04-02T14:41:24.750 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.750+0000 7f493ea00640 1 -- 172.21.15.90:0/1915366815 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f494c0be580 con 0x557ae75c6eb0 2024-04-02T14:41:24.750 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.750+0000 7f493ea00640 1 -- 172.21.15.90:0/1915366815 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f494c0c2930 con 0x557ae75c6eb0 2024-04-02T14:41:24.750 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.750+0000 7f493ea00640 1 -- 172.21.15.90:0/1915366815 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f494c009800 con 0x557ae75c6eb0 2024-04-02T14:41:24.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.750+0000 7f9293e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9294101a90 0x7f92941a4620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:24.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.751+0000 7f9293e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9294101a90 0x7f92941a4620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:42560/0 (socket says 172.21.15.90:42560) 2024-04-02T14:41:24.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.751+0000 7f9293e00640 1 -- 172.21.15.90:0/4268017949 learned_addr learned my addr 172.21.15.90:0/4268017949 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:41:24.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.751+0000 7f9293400640 1 --2- 172.21.15.90:0/4268017949 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9294102450 0x7f92941a4b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:24.752 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.751+0000 7f495c063740 1 -- 172.21.15.90:0/1915366815 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557ae75c6eb0 msgr2=0x557ae75cfba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:41:24.752 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.751+0000 7f495c063740 1 --2- 172.21.15.90:0/1915366815 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557ae75c6eb0 0x557ae75cfba0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f494c0baf50 tx=0x7f494c000f40 comp rx=0 tx=0).stop 2024-04-02T14:41:24.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.752+0000 7f9293400640 1 -- 172.21.15.90:0/4268017949 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9294101a90 msgr2=0x7f92941a4620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:41:24.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.752+0000 7f9293400640 1 --2- 172.21.15.90:0/4268017949 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9294101a90 0x7f92941a4620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:24.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.752+0000 7f9293400640 1 -- 172.21.15.90:0/4268017949 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9294107aa0 msgr2=0x7f929419e790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:41:24.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.752+0000 7f9293400640 1 --2- 172.21.15.90:0/4268017949 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9294107aa0 0x7f929419e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:24.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.752+0000 7f9293400640 1 -- 172.21.15.90:0/4268017949 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f929419ef60 con 0x7f9294102450 2024-04-02T14:41:24.753 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.752+0000 7f495c063740 1 -- 172.21.15.90:0/1915366815 shutdown_connections 2024-04-02T14:41:24.753 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.752+0000 7f495c063740 1 --2- 172.21.15.90:0/1915366815 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557ae75d03b0 0x557ae75e8270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:24.753 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.752+0000 7f495c063740 1 --2- 172.21.15.90:0/1915366815 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ae75c77d0 0x557ae75dc060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:24.753 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.752+0000 7f495c063740 1 --2- 172.21.15.90:0/1915366815 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557ae75c6eb0 0x557ae75cfba0 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:24.753 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.752+0000 7f495c063740 1 -- 172.21.15.90:0/1915366815 >> 172.21.15.90:0/1915366815 conn(0x557ae75c15e0 msgr2=0x557ae75c31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:41:24.753 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.753+0000 7f495c063740 1 -- 172.21.15.90:0/1915366815 shutdown_connections 2024-04-02T14:41:24.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.752+0000 7f9293400640 1 --2- 172.21.15.90:0/4268017949 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9294102450 0x7f92941a4b60 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f928400c970 tx=0x7f928400ce30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:41:24.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.752+0000 7f9290c00640 1 -- 172.21.15.90:0/4268017949 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9284007bb0 con 0x7f9294102450 2024-04-02T14:41:24.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.752+0000 7f9290c00640 1 -- 172.21.15.90:0/4268017949 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9284007d50 con 0x7f9294102450 2024-04-02T14:41:24.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.752+0000 7f9290c00640 1 -- 172.21.15.90:0/4268017949 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9284004e60 con 0x7f9294102450 2024-04-02T14:41:24.754 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.753+0000 7f495c063740 1 -- 172.21.15.90:0/1915366815 wait complete. 2024-04-02T14:41:24.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.753+0000 7f929b200640 1 -- 172.21.15.90:0/4268017949 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f929419f250 con 0x7f9294102450 2024-04-02T14:41:24.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.754+0000 7f929b200640 1 -- 172.21.15.90:0/4268017949 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f92941ab470 con 0x7f9294102450 2024-04-02T14:41:24.755 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.754+0000 7f495c063740 1 -- start start 2024-04-02T14:41:24.755 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.755+0000 7f495c063740 1 Processor -- start 2024-04-02T14:41:24.755 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.756+0000 7f495c063740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ae75c6eb0 0x557ae75d8570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:24.755 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.756+0000 7f495c063740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557ae75c77d0 0x557ae75d8ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:24.755 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.756+0000 7f495c063740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557ae75d03b0 0x557ae75d5540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:24.756 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.756+0000 7f495c063740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f49541b5130 con 0x557ae75c77d0 2024-04-02T14:41:24.756 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.756+0000 7f495c063740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f49541b4a60 con 0x557ae75d03b0 2024-04-02T14:41:24.756 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.756+0000 7f495c063740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f49541b1ca0 con 0x557ae75c6eb0 2024-04-02T14:41:24.756 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.756+0000 7f495c063740 10 client.0 _ll_register_callbacks cb 0x7f4940003f70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:41:24.756 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.757+0000 7f4951e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557ae75c77d0 0x557ae75d8ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:24.756 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.757+0000 7f4953200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557ae75d03b0 0x557ae75d5540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:24.756 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.757+0000 7f4953200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557ae75d03b0 0x557ae75d5540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:35386/0 (socket says 172.21.15.90:35386) 2024-04-02T14:41:24.757 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.757+0000 7f4953200640 1 -- 172.21.15.90:0/3894883330 learned_addr learned my addr 172.21.15.90:0/3894883330 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:41:24.757 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.757+0000 7f4952800640 1 --2- 172.21.15.90:0/3894883330 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ae75c6eb0 0x557ae75d8570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:24.757 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.758+0000 7f4953200640 1 -- 172.21.15.90:0/3894883330 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ae75c6eb0 msgr2=0x557ae75d8570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:41:24.757 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.758+0000 7f4953200640 1 --2- 172.21.15.90:0/3894883330 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ae75c6eb0 0x557ae75d8570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:24.758 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.758+0000 7f4953200640 1 -- 172.21.15.90:0/3894883330 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557ae75c77d0 msgr2=0x557ae75d8ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:41:24.758 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.759+0000 7f4953200640 1 --2- 172.21.15.90:0/3894883330 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557ae75c77d0 0x557ae75d8ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:24.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.759+0000 7f9290c00640 1 -- 172.21.15.90:0/4268017949 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f9284005000 con 0x7f9294102450 2024-04-02T14:41:24.759 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.759+0000 7f4953200640 1 -- 172.21.15.90:0/3894883330 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557ae730c620 con 0x557ae75d03b0 2024-04-02T14:41:24.759 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.759+0000 7f4952800640 1 --2- 172.21.15.90:0/3894883330 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ae75c6eb0 0x557ae75d8570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:41:24.759 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.759+0000 7f4951e00640 1 --2- 172.21.15.90:0/3894883330 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557ae75c77d0 0x557ae75d8ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:41:24.762 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.763+0000 7f4953200640 1 --2- 172.21.15.90:0/3894883330 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557ae75d03b0 0x557ae75d5540 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f494000acc0 tx=0x7f494000acf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1f3d6d80f6cb0bfc server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:41:24.762 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.763+0000 7f4933e00640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:41:24.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.763+0000 7f9290c00640 1 --2- 172.21.15.90:0/4268017949 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f92780a3e70 0x7f92780a6330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:24.762 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.763+0000 7f4933e00640 1 -- 172.21.15.90:0/3894883330 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f494002c070 con 0x557ae75d03b0 2024-04-02T14:41:24.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.763+0000 7f9290c00640 1 -- 172.21.15.90:0/4268017949 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f92840d00d0 con 0x7f9294102450 2024-04-02T14:41:24.763 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.763+0000 7f4933e00640 1 -- 172.21.15.90:0/3894883330 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f4940013070 con 0x557ae75d03b0 2024-04-02T14:41:24.763 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.763+0000 7f495c063740 1 -- 172.21.15.90:0/3894883330 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f494c0c2930 con 0x557ae75d03b0 2024-04-02T14:41:24.763 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.764+0000 7f4933e00640 1 -- 172.21.15.90:0/3894883330 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f494001b030 con 0x557ae75d03b0 2024-04-02T14:41:24.763 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.764+0000 7f495c063740 10 client.27407 fetch_fsmap learned FSMap version 27 2024-04-02T14:41:24.763 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.764+0000 7f495c063740 1 -- 172.21.15.90:0/3894883330 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x557ae7647300 con 0x557ae75d03b0 2024-04-02T14:41:24.764 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.763+0000 7f929df99740 1 Processor -- start 2024-04-02T14:41:24.764 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.763+0000 7f929df99740 1 -- start start 2024-04-02T14:41:24.764 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.764+0000 7f929df99740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556b308fba10 0x556b308fa760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:24.764 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.764+0000 7f929df99740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556b308fad30 0x556b308fb1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:24.764 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.764+0000 7f929df99740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556b309040a0 0x556b30906490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:24.764 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.764+0000 7f929df99740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x556b3083b400 con 0x556b308fad30 2024-04-02T14:41:24.764 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.764+0000 7f929df99740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x556b3081ff10 con 0x556b308fba10 2024-04-02T14:41:24.764 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.764+0000 7f929df99740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x556b3085f6f0 con 0x556b309040a0 2024-04-02T14:41:24.764 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.764+0000 7f9293e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556b308fba10 0x556b308fa760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:24.764 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.764+0000 7f9293e00640 1 --2- 172.21.15.90:0/4268017949 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f92780a3e70 0x7f92780a6330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:24.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.764+0000 7f9299200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556b309040a0 0x556b30906490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:24.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.765+0000 7f9299200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556b309040a0 0x556b30906490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:42592/0 (socket says 172.21.15.90:42592) 2024-04-02T14:41:24.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.765+0000 7f9299200640 1 -- 172.21.15.90:0/512141119 learned_addr learned my addr 172.21.15.90:0/512141119 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:41:24.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.765+0000 7f9299200640 1 -- 172.21.15.90:0/512141119 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556b308fba10 msgr2=0x556b308fa760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:41:24.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.765+0000 7f9299200640 1 --2- 172.21.15.90:0/512141119 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556b308fba10 0x556b308fa760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:24.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.765+0000 7f9299200640 1 -- 172.21.15.90:0/512141119 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556b308fad30 msgr2=0x556b308fb1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:41:24.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.765+0000 7f9299200640 1 --2- 172.21.15.90:0/512141119 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556b308fad30 0x556b308fb1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:24.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.765+0000 7f9299200640 1 -- 172.21.15.90:0/512141119 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556b305733f0 con 0x556b309040a0 2024-04-02T14:41:24.766 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.764+0000 7f4933e00640 1 -- 172.21.15.90:0/3894883330 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f494c0c2930 con 0x557ae75d03b0 2024-04-02T14:41:24.766 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.766+0000 7f4933e00640 1 -- 172.21.15.90:0/3894883330 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f494002f020 con 0x557ae75d03b0 2024-04-02T14:41:24.766 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.766+0000 7f495c063740 10 client.27407 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:41:24.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.766+0000 7f9299200640 1 --2- 172.21.15.90:0/512141119 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556b309040a0 0x556b30906490 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f928000ba00 tx=0x7f928000bec0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dfc2525c9e54dbe5 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:41:24.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.766+0000 7f9289600640 1 -- 172.21.15.90:0/512141119 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9280016070 con 0x556b309040a0 2024-04-02T14:41:24.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.766+0000 7f9289600640 1 -- 172.21.15.90:0/512141119 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9280005500 con 0x556b309040a0 2024-04-02T14:41:24.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.766+0000 7f9289600640 1 -- 172.21.15.90:0/512141119 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9280010460 con 0x556b309040a0 2024-04-02T14:41:24.767 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.766+0000 7f495c063740 10 client.27407 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-02T14:41:24.767 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.766+0000 7f495c063740 20 client.27407 populate_metadata read hostname 'smithi090' 2024-04-02T14:41:24.767 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.767+0000 7f495c063740 1 --2- 172.21.15.90:0/3894883330 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x557ae7647a70 0x557ae7649e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:24.767 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.767+0000 7f495c063740 4 client.27407 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-02T14:41:24.767 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.767+0000 7f495c063740 1 -- 172.21.15.90:0/3894883330 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "flush journal"}) -- 0x557ae74dbcd0 con 0x557ae7647a70 2024-04-02T14:41:24.767 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.767+0000 7f4952800640 1 --2- 172.21.15.90:0/3894883330 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x557ae7647a70 0x557ae7649e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:24.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.767+0000 7f929df99740 1 -- 172.21.15.90:0/512141119 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556b309040a0 msgr2=0x556b30906490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:41:24.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.767+0000 7f929df99740 1 --2- 172.21.15.90:0/512141119 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556b309040a0 0x556b30906490 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f928000ba00 tx=0x7f928000bec0 comp rx=0 tx=0).stop 2024-04-02T14:41:24.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.767+0000 7f9293e00640 1 --2- 172.21.15.90:0/4268017949 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f92780a3e70 0x7f92780a6330 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f928c000f80 tx=0x7f928c005ef0 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:41:24.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.768+0000 7f929df99740 1 -- 172.21.15.90:0/512141119 shutdown_connections 2024-04-02T14:41:24.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.768+0000 7f929df99740 1 --2- 172.21.15.90:0/512141119 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556b309040a0 0x556b30906490 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:24.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.768+0000 7f929df99740 1 --2- 172.21.15.90:0/512141119 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556b308fad30 0x556b308fb1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:24.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.768+0000 7f929df99740 1 --2- 172.21.15.90:0/512141119 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556b308fba10 0x556b308fa760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:24.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.768+0000 7f929df99740 1 -- 172.21.15.90:0/512141119 >> 172.21.15.90:0/512141119 conn(0x556b308c1a50 msgr2=0x556b308f5780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:41:24.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.768+0000 7f929df99740 1 -- 172.21.15.90:0/512141119 shutdown_connections 2024-04-02T14:41:24.768 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.769+0000 7f4952800640 1 --2- 172.21.15.90:0/3894883330 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x557ae7647a70 0x557ae7649e90 crc :-1 s=READY pgs=181 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5846175062324fea server_cookie=2db9ecf6dc01969a in_seq=0 out_seq=0 2024-04-02T14:41:24.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.768+0000 7f929df99740 1 -- 172.21.15.90:0/512141119 wait complete. 2024-04-02T14:41:24.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.769+0000 7f929df99740 1 -- start start 2024-04-02T14:41:24.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.769+0000 7f929df99740 1 Processor -- start 2024-04-02T14:41:24.769 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.769+0000 7f4933e00640 10 client.27407 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:41:24.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.770+0000 7f929df99740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556b308fad30 0x556b30901120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:24.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.770+0000 7f929df99740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556b308fba10 0x556b30901660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:24.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.770+0000 7f929df99740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556b309040a0 0x556b30901ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:24.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.771+0000 7f9299200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556b309040a0 0x556b30901ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:24.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.771+0000 7f9299200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556b309040a0 0x556b30901ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:42594/0 (socket says 172.21.15.90:42594) 2024-04-02T14:41:24.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.771+0000 7f9299200640 1 -- 172.21.15.90:0/3120923488 learned_addr learned my addr 172.21.15.90:0/3120923488 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:41:24.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.771+0000 7f929df99740 1 -- 172.21.15.90:0/3120923488 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x556b3085b6a0 con 0x556b308fad30 2024-04-02T14:41:24.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.771+0000 7f929df99740 1 -- 172.21.15.90:0/3120923488 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x556b30807980 con 0x556b308fba10 2024-04-02T14:41:24.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.771+0000 7f929df99740 1 -- 172.21.15.90:0/3120923488 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x556b3081acb0 con 0x556b309040a0 2024-04-02T14:41:24.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.771+0000 7f9293400640 1 --2- 172.21.15.90:0/3120923488 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556b308fba10 0x556b30901660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:24.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.771+0000 7f9293e00640 1 --2- 172.21.15.90:0/3120923488 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556b308fad30 0x556b30901120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:24.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.771+0000 7f9299200640 1 -- 172.21.15.90:0/3120923488 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556b308fba10 msgr2=0x556b30901660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:41:24.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.771+0000 7f929df99740 10 client.0 _ll_register_callbacks cb 0x556b3081a6a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:41:24.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.772+0000 7f9299200640 1 --2- 172.21.15.90:0/3120923488 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556b308fba10 0x556b30901660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:24.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.772+0000 7f9299200640 1 -- 172.21.15.90:0/3120923488 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556b308fad30 msgr2=0x556b30901120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:41:24.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.772+0000 7f9299200640 1 --2- 172.21.15.90:0/3120923488 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556b308fad30 0x556b30901120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:24.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.772+0000 7f9299200640 1 -- 172.21.15.90:0/3120923488 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556b3063d7d0 con 0x556b309040a0 2024-04-02T14:41:24.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.772+0000 7f9293e00640 1 --2- 172.21.15.90:0/3120923488 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556b308fad30 0x556b30901120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:41:24.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.772+0000 7f9293400640 1 --2- 172.21.15.90:0/3120923488 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556b308fba10 0x556b30901660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:41:24.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.772+0000 7f929df99740 1 -- 172.21.15.90:0/3120923488 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9280005500 con 0x556b309040a0 2024-04-02T14:41:24.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.772+0000 7f9299200640 1 --2- 172.21.15.90:0/3120923488 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556b309040a0 0x556b30901ba0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f9280004bf0 tx=0x7f9280004ec0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=61424e230b282b20 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:41:24.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.773+0000 7f927e000640 10 client.27277 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:41:24.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.773+0000 7f927e000640 1 -- 172.21.15.90:0/3120923488 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f928002a050 con 0x556b309040a0 2024-04-02T14:41:24.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.773+0000 7f927e000640 1 -- 172.21.15.90:0/3120923488 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f9280004040 con 0x556b309040a0 2024-04-02T14:41:24.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.773+0000 7f927e000640 1 -- 172.21.15.90:0/3120923488 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f9280016070 con 0x556b309040a0 2024-04-02T14:41:24.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.774+0000 7f929df99740 10 client.27277 fetch_fsmap learned FSMap version 27 2024-04-02T14:41:24.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.774+0000 7f929df99740 1 -- 172.21.15.90:0/3120923488 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({fsmap=27}) -- 0x556b30912f40 con 0x556b309040a0 2024-04-02T14:41:24.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.776+0000 7f927e000640 1 -- 172.21.15.90:0/3120923488 <== mon.1 v2:172.21.15.177:3300/0 4 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f9280004280 con 0x556b309040a0 2024-04-02T14:41:24.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.776+0000 7f929df99740 10 client.27277 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:41:24.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.776+0000 7f929df99740 10 client.27277 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2024-04-02T14:41:24.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.776+0000 7f929df99740 20 client.27277 populate_metadata read hostname 'smithi090' 2024-04-02T14:41:24.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.776+0000 7f929df99740 1 --2- 172.21.15.90:0/3120923488 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x556b30914290 0x556b30916680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:24.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.776+0000 7f929df99740 4 client.27277 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:41:24.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.776+0000 7f929df99740 1 -- 172.21.15.90:0/3120923488 --> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556b3081f040 con 0x556b30914290 2024-04-02T14:41:24.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.777+0000 7f9293e00640 1 --2- 172.21.15.90:0/3120923488 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x556b30914290 0x556b30916680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:24.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.779+0000 7f927e000640 10 client.27277 ms_handle_connect on v2:172.21.15.90:6838/1498915870 2024-04-02T14:41:24.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.779+0000 7f9293e00640 1 --2- 172.21.15.90:0/3120923488 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x556b30914290 0x556b30916680 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=2f4e02ff9816502d server_cookie=735b0086e994fa98 in_seq=0 out_seq=0 2024-04-02T14:41:24.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.784+0000 7f927e000640 1 -- 172.21.15.90:0/3120923488 <== mds.1 v2:172.21.15.90:6838/1498915870 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x556b3081f040 con 0x556b30914290 2024-04-02T14:41:24.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.784+0000 7f927e000640 10 client.27277 handle_command_reply: tid=0 2024-04-02T14:41:24.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.784+0000 7f929df99740 1 client.27277 shutdown 2024-04-02T14:41:24.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.785+0000 7f929df99740 1 -- 172.21.15.90:0/3120923488 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556b309040a0 msgr2=0x556b30901ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:41:24.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.785+0000 7f929df99740 1 --2- 172.21.15.90:0/3120923488 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556b309040a0 0x556b30901ba0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f9280004bf0 tx=0x7f9280004ec0 comp rx=0 tx=0).stop 2024-04-02T14:41:24.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.785+0000 7f929df99740 1 -- 172.21.15.90:0/3120923488 shutdown_connections 2024-04-02T14:41:24.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.786+0000 7f929df99740 1 --2- 172.21.15.90:0/3120923488 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x556b30914290 0x556b30916680 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:24.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.786+0000 7f929df99740 1 --2- 172.21.15.90:0/3120923488 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556b309040a0 0x556b30901ba0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:24.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.786+0000 7f929df99740 1 --2- 172.21.15.90:0/3120923488 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556b308fba10 0x556b30901660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:24.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.786+0000 7f929df99740 1 --2- 172.21.15.90:0/3120923488 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556b308fad30 0x556b30901120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:24.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.786+0000 7f929df99740 1 -- 172.21.15.90:0/3120923488 >> 172.21.15.90:0/3120923488 conn(0x556b308c1a50 msgr2=0x556b308f74d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:41:24.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.786+0000 7f927e000640 0 client.27277 ms_handle_reset on v2:172.21.15.90:6838/1498915870 2024-04-02T14:41:24.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.786+0000 7f929df99740 1 -- 172.21.15.90:0/3120923488 shutdown_connections 2024-04-02T14:41:24.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.786+0000 7f929df99740 1 -- 172.21.15.90:0/3120923488 wait complete. 2024-04-02T14:41:24.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.787+0000 7f929df99740 20 client.27277 trim_cache size 0 max 16384 2024-04-02T14:41:24.825 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.826+0000 7f4933e00640 1 -- 172.21.15.90:0/3894883330 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x557ae74dbcd0 con 0x557ae7647a70 2024-04-02T14:41:24.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.826+0000 7f929df99740 1 Processor -- start 2024-04-02T14:41:24.825 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.826+0000 7f4933e00640 10 client.27407 handle_command_reply: tid=0 2024-04-02T14:41:24.825 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.826+0000 7f495c063740 1 client.27407 shutdown 2024-04-02T14:41:24.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.826+0000 7f929df99740 1 -- start start 2024-04-02T14:41:24.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.826+0000 7f929df99740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556b308fad30 0x556b3090fb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:24.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.826+0000 7f929df99740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556b308fba10 0x556b30902f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:24.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.826+0000 7f929df99740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556b309040a0 0x556b3091bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:24.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.826+0000 7f929df99740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f92941b3f50 con 0x556b308fba10 2024-04-02T14:41:24.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.826+0000 7f929df99740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f92941b5a90 con 0x556b308fad30 2024-04-02T14:41:24.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.826+0000 7f929df99740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f92941b4620 con 0x556b309040a0 2024-04-02T14:41:24.827 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.827+0000 7f495c063740 1 -- 172.21.15.90:0/3894883330 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557ae75d03b0 msgr2=0x557ae75d5540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:41:24.827 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.827+0000 7f495c063740 1 --2- 172.21.15.90:0/3894883330 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557ae75d03b0 0x557ae75d5540 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f494000acc0 tx=0x7f494000acf0 comp rx=0 tx=0).stop 2024-04-02T14:41:24.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.827+0000 7f9299200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556b309040a0 0x556b3091bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:24.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.827+0000 7f9293400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556b308fba10 0x556b30902f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:24.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.827+0000 7f9293400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556b308fba10 0x556b30902f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:57342/0 (socket says 172.21.15.90:57342) 2024-04-02T14:41:24.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.827+0000 7f9293400640 1 -- 172.21.15.90:0/1302667376 learned_addr learned my addr 172.21.15.90:0/1302667376 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:41:24.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.827+0000 7f9293e00640 1 --2- 172.21.15.90:0/1302667376 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556b308fad30 0x556b3090fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:24.828 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.828+0000 7f495c063740 1 -- 172.21.15.90:0/3894883330 shutdown_connections 2024-04-02T14:41:24.828 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.828+0000 7f495c063740 1 --2- 172.21.15.90:0/3894883330 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x557ae7647a70 0x557ae7649e90 crc :-1 s=READY pgs=181 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:24.828 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.828+0000 7f495c063740 1 --2- 172.21.15.90:0/3894883330 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557ae75d03b0 0x557ae75d5540 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:24.828 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.828+0000 7f495c063740 1 --2- 172.21.15.90:0/3894883330 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557ae75c77d0 0x557ae75d8ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:24.828 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.828+0000 7f495c063740 1 --2- 172.21.15.90:0/3894883330 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ae75c6eb0 0x557ae75d8570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:24.828 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.828+0000 7f4933e00640 0 client.27407 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:41:24.828 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.828+0000 7f495c063740 1 -- 172.21.15.90:0/3894883330 >> 172.21.15.90:0/3894883330 conn(0x557ae75c15e0 msgr2=0x557ae75cd430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:41:24.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.828+0000 7f9293e00640 1 -- 172.21.15.90:0/1302667376 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556b309040a0 msgr2=0x556b3091bcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:41:24.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.828+0000 7f9293e00640 1 --2- 172.21.15.90:0/1302667376 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556b309040a0 0x556b3091bcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:24.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.828+0000 7f9293e00640 1 -- 172.21.15.90:0/1302667376 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556b308fba10 msgr2=0x556b30902f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:41:24.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.828+0000 7f9293e00640 1 --2- 172.21.15.90:0/1302667376 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556b308fba10 0x556b30902f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:24.829 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.828+0000 7f495c063740 1 -- 172.21.15.90:0/3894883330 shutdown_connections 2024-04-02T14:41:24.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.828+0000 7f9293e00640 1 -- 172.21.15.90:0/1302667376 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556b308f9d60 con 0x556b308fad30 2024-04-02T14:41:24.830 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.828+0000 7f495c063740 1 -- 172.21.15.90:0/3894883330 wait complete. 2024-04-02T14:41:24.830 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.829+0000 7f495c063740 20 client.27407 trim_cache size 0 max 16384 2024-04-02T14:41:24.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.829+0000 7f9293e00640 1 --2- 172.21.15.90:0/1302667376 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556b308fad30 0x556b3090fb70 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f928c02c690 tx=0x7f928c039000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4ddf5e8c4ff49097 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:41:24.831 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:{ 2024-04-02T14:41:24.831 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "message": "", 2024-04-02T14:41:24.831 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "return_code": 0 2024-04-02T14:41:24.831 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:} 2024-04-02T14:41:24.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.829+0000 7f9288c00640 1 -- 172.21.15.90:0/1302667376 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f928c058070 con 0x556b308fad30 2024-04-02T14:41:24.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.829+0000 7f9288c00640 1 -- 172.21.15.90:0/1302667376 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f928c03a040 con 0x556b308fad30 2024-04-02T14:41:24.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.829+0000 7f9288c00640 1 -- 172.21.15.90:0/1302667376 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f928c039070 con 0x556b308fad30 2024-04-02T14:41:24.832 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.829+0000 7f4959400640 1 -- 172.21.15.90:0/98318128 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f49380a3fa0 msgr2=0x7f49380a6460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:41:24.832 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.829+0000 7f4959400640 1 --2- 172.21.15.90:0/98318128 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f49380a3fa0 0x7f49380a6460 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f4954077e40 tx=0x7f494400b4a0 comp rx=0 tx=0).stop 2024-04-02T14:41:24.832 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.829+0000 7f4959400640 1 -- 172.21.15.90:0/98318128 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4954100c50 msgr2=0x7f49541a4650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:41:24.832 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.829+0000 7f4959400640 1 --2- 172.21.15.90:0/98318128 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4954100c50 0x7f49541a4650 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f494c009bf0 tx=0x7f494c035d10 comp rx=0 tx=0).stop 2024-04-02T14:41:24.832 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.830+0000 7f4959400640 1 -- 172.21.15.90:0/98318128 shutdown_connections 2024-04-02T14:41:24.832 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.830+0000 7f4959400640 1 --2- 172.21.15.90:0/98318128 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f49380a3fa0 0x7f49380a6460 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:24.833 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.830+0000 7f4959400640 1 --2- 172.21.15.90:0/98318128 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f49541059a0 0x7f495419e810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:24.833 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.830+0000 7f4959400640 1 --2- 172.21.15.90:0/98318128 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4954101610 0x7f49541a4b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:24.833 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.830+0000 7f4959400640 1 --2- 172.21.15.90:0/98318128 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4954100c50 0x7f49541a4650 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:24.833 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.830+0000 7f4959400640 1 -- 172.21.15.90:0/98318128 >> 172.21.15.90:0/98318128 conn(0x7f4954077850 msgr2=0x7f49540fec00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:41:24.833 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.830+0000 7f4959400640 1 -- 172.21.15.90:0/98318128 shutdown_connections 2024-04-02T14:41:24.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.830+0000 7f929df99740 1 -- 172.21.15.90:0/1302667376 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556b308fad30 msgr2=0x556b3090fb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:41:24.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.830+0000 7f929df99740 1 --2- 172.21.15.90:0/1302667376 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556b308fad30 0x556b3090fb70 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f928c02c690 tx=0x7f928c039000 comp rx=0 tx=0).stop 2024-04-02T14:41:24.834 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:24.830+0000 7f4959400640 1 -- 172.21.15.90:0/98318128 wait complete. 2024-04-02T14:41:24.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.831+0000 7f929df99740 1 -- 172.21.15.90:0/1302667376 shutdown_connections 2024-04-02T14:41:24.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.831+0000 7f929df99740 1 --2- 172.21.15.90:0/1302667376 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556b309040a0 0x556b3091bcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:24.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.831+0000 7f929df99740 1 --2- 172.21.15.90:0/1302667376 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556b308fba10 0x556b30902f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:24.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.831+0000 7f929df99740 1 --2- 172.21.15.90:0/1302667376 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556b308fad30 0x556b3090fb70 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:24.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.831+0000 7f929df99740 1 -- 172.21.15.90:0/1302667376 >> 172.21.15.90:0/1302667376 conn(0x556b308c1a50 msgr2=0x556b30902840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:41:24.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.831+0000 7f929df99740 1 -- 172.21.15.90:0/1302667376 shutdown_connections 2024-04-02T14:41:24.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.831+0000 7f929df99740 1 -- 172.21.15.90:0/1302667376 wait complete. 2024-04-02T14:41:24.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.832+0000 7f929df99740 1 -- start start 2024-04-02T14:41:24.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.832+0000 7f929df99740 1 Processor -- start 2024-04-02T14:41:24.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.833+0000 7f929df99740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556b308fad30 0x556b3090c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:24.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.833+0000 7f929df99740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556b308fba10 0x556b308fe650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:24.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.833+0000 7f929df99740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556b309040a0 0x556b308fed40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:24.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.833+0000 7f929df99740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f92941b53c0 con 0x556b308fad30 2024-04-02T14:41:24.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.833+0000 7f929df99740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f92941b4cf0 con 0x556b309040a0 2024-04-02T14:41:24.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.833+0000 7f929df99740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f92941b1f30 con 0x556b308fba10 2024-04-02T14:41:24.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.834+0000 7f929df99740 10 client.0 _ll_register_callbacks cb 0x7f92800056d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:41:24.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.834+0000 7f9299200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556b309040a0 0x556b308fed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:24.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.834+0000 7f9299200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556b309040a0 0x556b308fed40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:35414/0 (socket says 172.21.15.90:35414) 2024-04-02T14:41:24.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.835+0000 7f9299200640 1 -- 172.21.15.90:0/45867378 learned_addr learned my addr 172.21.15.90:0/45867378 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:41:24.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.835+0000 7f9293e00640 1 --2- 172.21.15.90:0/45867378 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556b308fad30 0x556b3090c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:24.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.835+0000 7f9293400640 1 --2- 172.21.15.90:0/45867378 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556b308fba10 0x556b308fe650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:24.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.836+0000 7f9299200640 1 -- 172.21.15.90:0/45867378 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556b308fba10 msgr2=0x556b308fe650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:41:24.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.836+0000 7f9299200640 1 --2- 172.21.15.90:0/45867378 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556b308fba10 0x556b308fe650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:24.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.836+0000 7f9299200640 1 -- 172.21.15.90:0/45867378 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556b308fad30 msgr2=0x556b3090c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:41:24.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.836+0000 7f9299200640 1 --2- 172.21.15.90:0/45867378 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556b308fad30 0x556b3090c5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:24.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.836+0000 7f9299200640 1 -- 172.21.15.90:0/45867378 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556b30904e60 con 0x556b309040a0 2024-04-02T14:41:24.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.837+0000 7f9293e00640 1 --2- 172.21.15.90:0/45867378 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556b308fad30 0x556b3090c5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:41:24.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.837+0000 7f9293400640 1 --2- 172.21.15.90:0/45867378 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556b308fba10 0x556b308fe650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:41:24.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.837+0000 7f9299200640 1 --2- 172.21.15.90:0/45867378 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556b309040a0 0x556b308fed40 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f9280004c60 tx=0x7f928000b310 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f685b2b52d1c57f9 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:41:24.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.837+0000 7f927e000640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:41:24.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.838+0000 7f927e000640 1 -- 172.21.15.90:0/45867378 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f928002a040 con 0x556b309040a0 2024-04-02T14:41:24.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.838+0000 7f927e000640 1 -- 172.21.15.90:0/45867378 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f928000dd10 con 0x556b309040a0 2024-04-02T14:41:24.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.838+0000 7f929df99740 1 -- 172.21.15.90:0/45867378 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f928c03a040 con 0x556b309040a0 2024-04-02T14:41:24.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.838+0000 7f927e000640 1 -- 172.21.15.90:0/45867378 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f928003e050 con 0x556b309040a0 2024-04-02T14:41:24.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.839+0000 7f929df99740 10 client.27416 fetch_fsmap learned FSMap version 27 2024-04-02T14:41:24.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.839+0000 7f929df99740 1 -- 172.21.15.90:0/45867378 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x556b3098b8b0 con 0x556b309040a0 2024-04-02T14:41:24.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.839+0000 7f927e000640 1 -- 172.21.15.90:0/45867378 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f928c03a040 con 0x556b309040a0 2024-04-02T14:41:24.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.839+0000 7f927e000640 1 -- 172.21.15.90:0/45867378 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f9280003c90 con 0x556b309040a0 2024-04-02T14:41:24.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.839+0000 7f929df99740 10 client.27416 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:41:24.840 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.839+0000 7f929df99740 10 client.27416 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2024-04-02T14:41:24.840 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.839+0000 7f929df99740 20 client.27416 populate_metadata read hostname 'smithi090' 2024-04-02T14:41:24.840 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.839+0000 7f929df99740 1 --2- 172.21.15.90:0/45867378 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x556b3098bf70 0x556b3098e360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:24.840 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.839+0000 7f929df99740 4 client.27416 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-02T14:41:24.840 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.839+0000 7f929df99740 1 -- 172.21.15.90:0/45867378 --> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] -- command(tid 0: {"prefix": "damage ls"}) -- 0x556b3086cc30 con 0x556b3098bf70 2024-04-02T14:41:24.840 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.840+0000 7f9293e00640 1 --2- 172.21.15.90:0/45867378 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x556b3098bf70 0x556b3098e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:24.840 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.841+0000 7f927e000640 10 client.27416 ms_handle_connect on v2:172.21.15.90:6838/1498915870 2024-04-02T14:41:24.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.843+0000 7f9293e00640 1 --2- 172.21.15.90:0/45867378 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x556b3098bf70 0x556b3098e360 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=6b6b40926f0d5847 server_cookie=75f41ed284586195 in_seq=0 out_seq=0 2024-04-02T14:41:24.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.844+0000 7f927e000640 1 -- 172.21.15.90:0/45867378 <== mds.1 v2:172.21.15.90:6838/1498915870 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x556b3086cc30 con 0x556b3098bf70 2024-04-02T14:41:24.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.844+0000 7f927e000640 10 client.27416 handle_command_reply: tid=0 2024-04-02T14:41:24.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.844+0000 7f929df99740 1 client.27416 shutdown 2024-04-02T14:41:24.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.844+0000 7f929df99740 1 -- 172.21.15.90:0/45867378 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556b309040a0 msgr2=0x556b308fed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:41:24.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.844+0000 7f929df99740 1 --2- 172.21.15.90:0/45867378 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556b309040a0 0x556b308fed40 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f9280004c60 tx=0x7f928000b310 comp rx=0 tx=0).stop 2024-04-02T14:41:24.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.845+0000 7f929df99740 1 -- 172.21.15.90:0/45867378 shutdown_connections 2024-04-02T14:41:24.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.845+0000 7f929df99740 1 --2- 172.21.15.90:0/45867378 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x556b3098bf70 0x556b3098e360 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:24.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.845+0000 7f929df99740 1 --2- 172.21.15.90:0/45867378 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556b309040a0 0x556b308fed40 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:24.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.845+0000 7f929df99740 1 --2- 172.21.15.90:0/45867378 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556b308fba10 0x556b308fe650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:24.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.845+0000 7f929df99740 1 --2- 172.21.15.90:0/45867378 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556b308fad30 0x556b3090c5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:24.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.845+0000 7f929df99740 1 -- 172.21.15.90:0/45867378 >> 172.21.15.90:0/45867378 conn(0x556b308c1a50 msgr2=0x556b308f73d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:41:24.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.845+0000 7f927e000640 0 client.27416 ms_handle_reset on v2:172.21.15.90:6838/1498915870 2024-04-02T14:41:24.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.847+0000 7f929df99740 1 -- 172.21.15.90:0/45867378 shutdown_connections 2024-04-02T14:41:24.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.848+0000 7f929df99740 1 -- 172.21.15.90:0/45867378 wait complete. 2024-04-02T14:41:24.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.849+0000 7f929df99740 20 client.27416 trim_cache size 0 max 16384 2024-04-02T14:41:24.848 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-02T14:41:24.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.850+0000 7f929b200640 1 -- 172.21.15.90:0/4268017949 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f92780a3e70 msgr2=0x7f92780a6330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:41:24.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.850+0000 7f929b200640 1 --2- 172.21.15.90:0/4268017949 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f92780a3e70 0x7f92780a6330 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f928c000f80 tx=0x7f928c005ef0 comp rx=0 tx=0).stop 2024-04-02T14:41:24.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.850+0000 7f929b200640 1 -- 172.21.15.90:0/4268017949 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9294102450 msgr2=0x7f92941a4b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:41:24.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.850+0000 7f929b200640 1 --2- 172.21.15.90:0/4268017949 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9294102450 0x7f92941a4b60 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f928400c970 tx=0x7f928400ce30 comp rx=0 tx=0).stop 2024-04-02T14:41:24.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.850+0000 7f929b200640 1 -- 172.21.15.90:0/4268017949 shutdown_connections 2024-04-02T14:41:24.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.850+0000 7f929b200640 1 --2- 172.21.15.90:0/4268017949 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f92780a3e70 0x7f92780a6330 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:24.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.850+0000 7f929b200640 1 --2- 172.21.15.90:0/4268017949 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9294107aa0 0x7f929419e790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:24.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.850+0000 7f929b200640 1 --2- 172.21.15.90:0/4268017949 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9294102450 0x7f92941a4b60 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:24.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.850+0000 7f929b200640 1 --2- 172.21.15.90:0/4268017949 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9294101a90 0x7f92941a4620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:24.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.850+0000 7f929b200640 1 -- 172.21.15.90:0/4268017949 >> 172.21.15.90:0/4268017949 conn(0x7f9294078f00 msgr2=0x7f92941043a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:41:24.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.850+0000 7f929b200640 1 -- 172.21.15.90:0/4268017949 shutdown_connections 2024-04-02T14:41:24.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:24.850+0000 7f929b200640 1 -- 172.21.15.90:0/4268017949 wait complete. 2024-04-02T14:41:24.880 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-02T14:41:25.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.195+0000 7fec97600640 1 -- 172.21.15.90:0/1776776064 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fec90113340 msgr2=0x7fec90115730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:41:25.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.195+0000 7fec97600640 1 --2- 172.21.15.90:0/1776776064 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fec90113340 0x7fec90115730 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fec8800acc0 tx=0x7fec8802bec0 comp rx=0 tx=0).stop 2024-04-02T14:41:25.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.195+0000 7fec97600640 1 -- 172.21.15.90:0/1776776064 shutdown_connections 2024-04-02T14:41:25.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.195+0000 7fec97600640 1 --2- 172.21.15.90:0/1776776064 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fec90113340 0x7fec90115730 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:25.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.195+0000 7fec97600640 1 --2- 172.21.15.90:0/1776776064 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fec90109040 0x7fec901094c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:25.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.195+0000 7fec97600640 1 --2- 172.21.15.90:0/1776776064 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fec900695f0 0x7fec90108b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:25.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.195+0000 7fec97600640 1 -- 172.21.15.90:0/1776776064 >> 172.21.15.90:0/1776776064 conn(0x7fec9006d040 msgr2=0x7fec9006d450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:41:25.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.196+0000 7fec97600640 1 -- 172.21.15.90:0/1776776064 shutdown_connections 2024-04-02T14:41:25.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.197+0000 7fec97600640 1 -- 172.21.15.90:0/1776776064 wait complete. 2024-04-02T14:41:25.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.198+0000 7fec97600640 1 Processor -- start 2024-04-02T14:41:25.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.198+0000 7fec97600640 1 -- start start 2024-04-02T14:41:25.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.199+0000 7fec97600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fec900695f0 0x7fec901a2dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:25.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.199+0000 7fec97600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fec90109040 0x7fec901a3300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:25.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.199+0000 7fec97600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fec90113340 0x7fec901a7680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:25.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.199+0000 7fec97600640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fec90118450 con 0x7fec90109040 2024-04-02T14:41:25.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.199+0000 7fec97600640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fec901182d0 con 0x7fec900695f0 2024-04-02T14:41:25.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.199+0000 7fec97600640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fec901185d0 con 0x7fec90113340 2024-04-02T14:41:25.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.199+0000 7fec94c00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fec900695f0 0x7fec901a2dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:25.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.199+0000 7fec95600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fec90113340 0x7fec901a7680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:25.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.199+0000 7fec95600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fec90113340 0x7fec901a7680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:42624/0 (socket says 172.21.15.90:42624) 2024-04-02T14:41:25.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.199+0000 7fec95600640 1 -- 172.21.15.90:0/3808585973 learned_addr learned my addr 172.21.15.90:0/3808585973 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:41:25.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.200+0000 7fec8fe00640 1 --2- 172.21.15.90:0/3808585973 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fec90109040 0x7fec901a3300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:25.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.200+0000 7fec95600640 1 -- 172.21.15.90:0/3808585973 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fec900695f0 msgr2=0x7fec901a2dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:41:25.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.200+0000 7fec95600640 1 --2- 172.21.15.90:0/3808585973 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fec900695f0 0x7fec901a2dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:25.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.200+0000 7fec95600640 1 -- 172.21.15.90:0/3808585973 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fec90109040 msgr2=0x7fec901a3300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:41:25.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.200+0000 7fec95600640 1 --2- 172.21.15.90:0/3808585973 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fec90109040 0x7fec901a3300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:25.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.200+0000 7fec95600640 1 -- 172.21.15.90:0/3808585973 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec901a7db0 con 0x7fec90113340 2024-04-02T14:41:25.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.201+0000 7fec95600640 1 --2- 172.21.15.90:0/3808585973 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fec90113340 0x7fec901a7680 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fec8802c4d0 tx=0x7fec88032870 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:41:25.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.201+0000 7fec8d600640 1 -- 172.21.15.90:0/3808585973 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec88039040 con 0x7fec90113340 2024-04-02T14:41:25.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.201+0000 7fec8d600640 1 -- 172.21.15.90:0/3808585973 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fec88044070 con 0x7fec90113340 2024-04-02T14:41:25.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.201+0000 7fec8d600640 1 -- 172.21.15.90:0/3808585973 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec88034440 con 0x7fec90113340 2024-04-02T14:41:25.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.201+0000 7fec97600640 1 -- 172.21.15.90:0/3808585973 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec901a8040 con 0x7fec90113340 2024-04-02T14:41:25.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.202+0000 7fec97600640 1 -- 172.21.15.90:0/3808585973 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fec901af930 con 0x7fec90113340 2024-04-02T14:41:25.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.208+0000 7fec8d600640 1 -- 172.21.15.90:0/3808585973 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fec88004140 con 0x7fec90113340 2024-04-02T14:41:25.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.210+0000 7fec9a38f740 1 Processor -- start 2024-04-02T14:41:25.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.211+0000 7fec8d600640 1 --2- 172.21.15.90:0/3808585973 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fec780a3e60 0x7fec780a6320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:25.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.211+0000 7fec8d600640 1 -- 172.21.15.90:0/3808585973 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7fec880f2900 con 0x7fec90113340 2024-04-02T14:41:25.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.212+0000 7fec9a38f740 1 -- start start 2024-04-02T14:41:25.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.213+0000 7fec9a38f740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e7bad479f0 0x55e7bad46740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:25.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.213+0000 7fec9a38f740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e7bad46d10 0x55e7bad47190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:25.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.213+0000 7fec9a38f740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e7bad58890 0x55e7bad5ac80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:25.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.213+0000 7fec9a38f740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55e7bac87400 con 0x55e7bad479f0 2024-04-02T14:41:25.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.213+0000 7fec9a38f740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55e7bac6bf10 con 0x55e7bad46d10 2024-04-02T14:41:25.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.213+0000 7fec9a38f740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55e7bacab6f0 con 0x55e7bad58890 2024-04-02T14:41:25.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.213+0000 7fec95600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e7bad58890 0x55e7bad5ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:25.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.213+0000 7fec95600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e7bad58890 0x55e7bad5ac80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:42626/0 (socket says 172.21.15.90:42626) 2024-04-02T14:41:25.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.213+0000 7fec95600640 1 -- 172.21.15.90:0/3441185299 learned_addr learned my addr 172.21.15.90:0/3441185299 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:41:25.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.214+0000 7fec8fe00640 1 --2- 172.21.15.90:0/3441185299 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e7bad46d10 0x55e7bad47190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:25.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.214+0000 7fec8fe00640 1 --2- 172.21.15.90:0/3808585973 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fec780a3e60 0x7fec780a6320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:25.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.214+0000 7fec94c00640 1 --2- 172.21.15.90:0/3441185299 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e7bad479f0 0x55e7bad46740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:25.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.214+0000 7fec95600640 1 -- 172.21.15.90:0/3441185299 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e7bad46d10 msgr2=0x55e7bad47190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:41:25.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.214+0000 7fec95600640 1 --2- 172.21.15.90:0/3441185299 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e7bad46d10 0x55e7bad47190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:25.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.214+0000 7fec95600640 1 -- 172.21.15.90:0/3441185299 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e7bad479f0 msgr2=0x55e7bad46740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:41:25.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.214+0000 7fec95600640 1 --2- 172.21.15.90:0/3441185299 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e7bad479f0 0x55e7bad46740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:25.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.214+0000 7fec95600640 1 -- 172.21.15.90:0/3441185299 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e7ba9bf3f0 con 0x55e7bad58890 2024-04-02T14:41:25.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.214+0000 7fec8fe00640 1 --2- 172.21.15.90:0/3441185299 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e7bad46d10 0x55e7bad47190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:41:25.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.215+0000 7fec8fe00640 1 --2- 172.21.15.90:0/3808585973 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fec780a3e60 0x7fec780a6320 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x55e7bad47f40 tx=0x7fec7c00d040 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:41:25.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.215+0000 7fec95600640 1 --2- 172.21.15.90:0/3441185299 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e7bad58890 0x55e7bad5ac80 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fec880be8f0 tx=0x7fec880be730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9e33794211f1893 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:41:25.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.215+0000 7fec81c00640 1 -- 172.21.15.90:0/3441185299 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec880bf060 con 0x55e7bad58890 2024-04-02T14:41:25.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.215+0000 7fec81c00640 1 -- 172.21.15.90:0/3441185299 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fec880be960 con 0x55e7bad58890 2024-04-02T14:41:25.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.215+0000 7fec81c00640 1 -- 172.21.15.90:0/3441185299 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec880bebc0 con 0x55e7bad58890 2024-04-02T14:41:25.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.216+0000 7fec9a38f740 1 -- 172.21.15.90:0/3441185299 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e7bad58890 msgr2=0x55e7bad5ac80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:41:25.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.216+0000 7fec9a38f740 1 --2- 172.21.15.90:0/3441185299 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e7bad58890 0x55e7bad5ac80 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fec880be8f0 tx=0x7fec880be730 comp rx=0 tx=0).stop 2024-04-02T14:41:25.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.216+0000 7fec9a38f740 1 -- 172.21.15.90:0/3441185299 shutdown_connections 2024-04-02T14:41:25.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.216+0000 7fec9a38f740 1 --2- 172.21.15.90:0/3441185299 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e7bad58890 0x55e7bad5ac80 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:25.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.216+0000 7fec9a38f740 1 --2- 172.21.15.90:0/3441185299 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e7bad46d10 0x55e7bad47190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:25.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.216+0000 7fec9a38f740 1 --2- 172.21.15.90:0/3441185299 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e7bad479f0 0x55e7bad46740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:25.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.216+0000 7fec9a38f740 1 -- 172.21.15.90:0/3441185299 >> 172.21.15.90:0/3441185299 conn(0x55e7bad0da50 msgr2=0x55e7bad41760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:41:25.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.217+0000 7fec9a38f740 1 -- 172.21.15.90:0/3441185299 shutdown_connections 2024-04-02T14:41:25.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.217+0000 7fec9a38f740 1 -- 172.21.15.90:0/3441185299 wait complete. 2024-04-02T14:41:25.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.218+0000 7fec9a38f740 1 -- start start 2024-04-02T14:41:25.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.218+0000 7fec9a38f740 1 Processor -- start 2024-04-02T14:41:25.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.220+0000 7fec9a38f740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e7bad46d10 0x55e7bad558c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:25.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.220+0000 7fec9a38f740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e7bad479f0 0x55e7bad55e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:25.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.220+0000 7fec9a38f740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e7bad58890 0x55e7bad56340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:25.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.220+0000 7fec9a38f740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55e7baca76a0 con 0x55e7bad46d10 2024-04-02T14:41:25.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.220+0000 7fec9a38f740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55e7bac53980 con 0x55e7bad479f0 2024-04-02T14:41:25.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.220+0000 7fec9a38f740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55e7bac66cb0 con 0x55e7bad58890 2024-04-02T14:41:25.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.220+0000 7fec9a38f740 10 client.0 _ll_register_callbacks cb 0x55e7bac666a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:41:25.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.220+0000 7fec8fe00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e7bad479f0 0x55e7bad55e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:25.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.220+0000 7fec8fe00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e7bad479f0 0x55e7bad55e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:35476/0 (socket says 172.21.15.90:35476) 2024-04-02T14:41:25.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.220+0000 7fec8fe00640 1 -- 172.21.15.90:0/3208386950 learned_addr learned my addr 172.21.15.90:0/3208386950 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:41:25.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.220+0000 7fec94c00640 1 --2- 172.21.15.90:0/3208386950 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e7bad46d10 0x55e7bad558c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:25.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.220+0000 7fec95600640 1 --2- 172.21.15.90:0/3208386950 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e7bad58890 0x55e7bad56340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:25.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.221+0000 7fec8fe00640 1 -- 172.21.15.90:0/3208386950 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e7bad58890 msgr2=0x55e7bad56340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:41:25.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.221+0000 7fec8fe00640 1 --2- 172.21.15.90:0/3208386950 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e7bad58890 0x55e7bad56340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:25.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.221+0000 7fec8fe00640 1 -- 172.21.15.90:0/3208386950 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e7bad46d10 msgr2=0x55e7bad558c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:41:25.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.221+0000 7fec8fe00640 1 --2- 172.21.15.90:0/3208386950 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e7bad46d10 0x55e7bad558c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:25.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.221+0000 7fec8fe00640 1 -- 172.21.15.90:0/3208386950 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e7baa897d0 con 0x55e7bad479f0 2024-04-02T14:41:25.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.221+0000 7fec94c00640 1 --2- 172.21.15.90:0/3208386950 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e7bad46d10 0x55e7bad558c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:41:25.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.222+0000 7fec8fe00640 1 --2- 172.21.15.90:0/3208386950 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e7bad479f0 0x55e7bad55e00 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7fec7c004550 tx=0x7fec7c004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6e326c0fa7e0e442 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:41:25.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.222+0000 7fec76000640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:41:25.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.222+0000 7fec76000640 1 -- 172.21.15.90:0/3208386950 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec7c016040 con 0x55e7bad479f0 2024-04-02T14:41:25.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.222+0000 7fec76000640 1 -- 172.21.15.90:0/3208386950 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7fec7c003d80 con 0x55e7bad479f0 2024-04-02T14:41:25.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.223+0000 7fec9a38f740 1 -- 172.21.15.90:0/3208386950 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fec880be960 con 0x55e7bad479f0 2024-04-02T14:41:25.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.223+0000 7fec76000640 1 -- 172.21.15.90:0/3208386950 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fec7c027470 con 0x55e7bad479f0 2024-04-02T14:41:25.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.223+0000 7fec9a38f740 10 client.27428 fetch_fsmap learned FSMap version 27 2024-04-02T14:41:25.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.223+0000 7fec9a38f740 1 -- 172.21.15.90:0/3208386950 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55e7bad674d0 con 0x55e7bad479f0 2024-04-02T14:41:25.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.224+0000 7fec76000640 1 -- 172.21.15.90:0/3208386950 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fec880be960 con 0x55e7bad479f0 2024-04-02T14:41:25.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.224+0000 7fec76000640 1 -- 172.21.15.90:0/3208386950 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fec7c01d580 con 0x55e7bad479f0 2024-04-02T14:41:25.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.225+0000 7fec9a38f740 10 client.27428 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:41:25.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.225+0000 7fec9a38f740 10 client.27428 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:41:25.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.225+0000 7fec9a38f740 20 client.27428 populate_metadata read hostname 'smithi090' 2024-04-02T14:41:25.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.225+0000 7fec9a38f740 1 --2- 172.21.15.90:0/3208386950 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55e7bad68990 0x55e7bad6ad80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:25.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.225+0000 7fec9a38f740 4 client.27428 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:41:25.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.225+0000 7fec9a38f740 1 -- 172.21.15.90:0/3208386950 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e7bac6b040 con 0x55e7bad68990 2024-04-02T14:41:25.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.225+0000 7fec94c00640 1 --2- 172.21.15.90:0/3208386950 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55e7bad68990 0x55e7bad6ad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:25.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.226+0000 7fec94c00640 1 --2- 172.21.15.90:0/3208386950 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55e7bad68990 0x55e7bad6ad80 crc :-1 s=READY pgs=182 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=75c35f350433a71 server_cookie=35a58d80b944a49e in_seq=0 out_seq=0 2024-04-02T14:41:25.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.226+0000 7fec76000640 10 client.27428 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:41:25.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.227+0000 7fec76000640 1 -- 172.21.15.90:0/3208386950 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e7bac6b040 con 0x55e7bad68990 2024-04-02T14:41:25.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.227+0000 7fec76000640 10 client.27428 handle_command_reply: tid=0 2024-04-02T14:41:25.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.227+0000 7fec9a38f740 1 client.27428 shutdown 2024-04-02T14:41:25.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.227+0000 7fec9a38f740 1 -- 172.21.15.90:0/3208386950 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e7bad479f0 msgr2=0x55e7bad55e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:41:25.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.227+0000 7fec9a38f740 1 --2- 172.21.15.90:0/3208386950 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e7bad479f0 0x55e7bad55e00 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7fec7c004550 tx=0x7fec7c004730 comp rx=0 tx=0).stop 2024-04-02T14:41:25.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.228+0000 7fec9a38f740 1 -- 172.21.15.90:0/3208386950 shutdown_connections 2024-04-02T14:41:25.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.228+0000 7fec9a38f740 1 --2- 172.21.15.90:0/3208386950 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55e7bad68990 0x55e7bad6ad80 crc :-1 s=READY pgs=182 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:25.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.228+0000 7fec9a38f740 1 --2- 172.21.15.90:0/3208386950 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e7bad58890 0x55e7bad56340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:25.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.228+0000 7fec9a38f740 1 --2- 172.21.15.90:0/3208386950 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e7bad479f0 0x55e7bad55e00 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:25.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.228+0000 7fec9a38f740 1 --2- 172.21.15.90:0/3208386950 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e7bad46d10 0x55e7bad558c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:25.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.228+0000 7fec9a38f740 1 -- 172.21.15.90:0/3208386950 >> 172.21.15.90:0/3208386950 conn(0x55e7bad0da50 msgr2=0x55e7bad434b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:41:25.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.228+0000 7fec76000640 0 client.27428 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:41:25.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.228+0000 7fec9a38f740 1 -- 172.21.15.90:0/3208386950 shutdown_connections 2024-04-02T14:41:25.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.228+0000 7fec9a38f740 1 -- 172.21.15.90:0/3208386950 wait complete. 2024-04-02T14:41:25.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.229+0000 7fec9a38f740 20 client.27428 trim_cache size 0 max 16384 2024-04-02T14:41:25.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.254+0000 7fec9a38f740 1 Processor -- start 2024-04-02T14:41:25.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.254+0000 7fec9a38f740 1 -- start start 2024-04-02T14:41:25.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.255+0000 7fec9a38f740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e7bad46d10 0x55e7bad57ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:25.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.255+0000 7fec9a38f740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e7bad479f0 0x55e7bad70650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:25.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.255+0000 7fec9a38f740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e7bad58890 0x55e7bad70b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:25.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.255+0000 7fec9a38f740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fec901b3e50 con 0x55e7bad479f0 2024-04-02T14:41:25.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.255+0000 7fec9a38f740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fec901b5990 con 0x55e7bad58890 2024-04-02T14:41:25.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.255+0000 7fec9a38f740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fec901b4520 con 0x55e7bad46d10 2024-04-02T14:41:25.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.255+0000 7fec94c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e7bad46d10 0x55e7bad57ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:25.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.255+0000 7fec95600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e7bad58890 0x55e7bad70b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:25.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.255+0000 7fec94c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e7bad46d10 0x55e7bad57ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:42650/0 (socket says 172.21.15.90:42650) 2024-04-02T14:41:25.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.255+0000 7fec95600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e7bad58890 0x55e7bad70b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:35492/0 (socket says 172.21.15.90:35492) 2024-04-02T14:41:25.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.255+0000 7fec94c00640 1 -- 172.21.15.90:0/3371290167 learned_addr learned my addr 172.21.15.90:0/3371290167 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:41:25.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.256+0000 7fec8fe00640 1 --2- 172.21.15.90:0/3371290167 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e7bad479f0 0x55e7bad70650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:25.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.256+0000 7fec95600640 1 -- 172.21.15.90:0/3371290167 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e7bad46d10 msgr2=0x55e7bad57ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:41:25.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.256+0000 7fec95600640 1 --2- 172.21.15.90:0/3371290167 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e7bad46d10 0x55e7bad57ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:25.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.256+0000 7fec95600640 1 -- 172.21.15.90:0/3371290167 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e7bad479f0 msgr2=0x55e7bad70650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:41:25.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.256+0000 7fec95600640 1 --2- 172.21.15.90:0/3371290167 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e7bad479f0 0x55e7bad70650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:25.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.256+0000 7fec95600640 1 -- 172.21.15.90:0/3371290167 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e7bad41b20 con 0x55e7bad58890 2024-04-02T14:41:25.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.256+0000 7fec94c00640 1 --2- 172.21.15.90:0/3371290167 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e7bad46d10 0x55e7bad57ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:41:25.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.256+0000 7fec8fe00640 1 --2- 172.21.15.90:0/3371290167 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e7bad479f0 0x55e7bad70650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:41:25.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.257+0000 7fec95600640 1 --2- 172.21.15.90:0/3371290167 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e7bad58890 0x55e7bad70b90 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7fec880c2ed0 tx=0x7fec880be380 comp rx=0 tx=0).ready entity=mon.2 client_cookie=447d1cb751dd3ebf server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:41:25.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.257+0000 7fec81200640 1 -- 172.21.15.90:0/3371290167 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec880c75d0 con 0x55e7bad58890 2024-04-02T14:41:25.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.257+0000 7fec81200640 1 -- 172.21.15.90:0/3371290167 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fec880bf060 con 0x55e7bad58890 2024-04-02T14:41:25.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.257+0000 7fec81200640 1 -- 172.21.15.90:0/3371290167 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec880c7140 con 0x55e7bad58890 2024-04-02T14:41:25.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.258+0000 7fec9a38f740 1 -- 172.21.15.90:0/3371290167 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e7bad58890 msgr2=0x55e7bad70b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:41:25.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.258+0000 7fec9a38f740 1 --2- 172.21.15.90:0/3371290167 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e7bad58890 0x55e7bad70b90 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7fec880c2ed0 tx=0x7fec880be380 comp rx=0 tx=0).stop 2024-04-02T14:41:25.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.259+0000 7fec9a38f740 1 -- 172.21.15.90:0/3371290167 shutdown_connections 2024-04-02T14:41:25.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.259+0000 7fec9a38f740 1 --2- 172.21.15.90:0/3371290167 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e7bad58890 0x55e7bad70b90 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:25.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.259+0000 7fec9a38f740 1 --2- 172.21.15.90:0/3371290167 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e7bad479f0 0x55e7bad70650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:25.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.259+0000 7fec9a38f740 1 --2- 172.21.15.90:0/3371290167 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e7bad46d10 0x55e7bad57ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:25.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.259+0000 7fec9a38f740 1 -- 172.21.15.90:0/3371290167 >> 172.21.15.90:0/3371290167 conn(0x55e7bad0da50 msgr2=0x55e7bad54190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:41:25.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.259+0000 7fec9a38f740 1 -- 172.21.15.90:0/3371290167 shutdown_connections 2024-04-02T14:41:25.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.259+0000 7fec9a38f740 1 -- 172.21.15.90:0/3371290167 wait complete. 2024-04-02T14:41:25.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.259+0000 7fec9a38f740 1 -- start start 2024-04-02T14:41:25.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.260+0000 7fec9a38f740 1 Processor -- start 2024-04-02T14:41:25.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.260+0000 7fec9a38f740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e7bad46d10 0x55e7bad60960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:25.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.261+0000 7fec9a38f740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e7bad479f0 0x55e7bad52270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:25.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.261+0000 7fec94c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e7bad46d10 0x55e7bad60960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:25.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.261+0000 7fec94c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e7bad46d10 0x55e7bad60960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:57380/0 (socket says 172.21.15.90:57380) 2024-04-02T14:41:25.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.261+0000 7fec9a38f740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e7bad58890 0x55e7bad527b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:25.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.261+0000 7fec9a38f740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fec901b52c0 con 0x55e7bad46d10 2024-04-02T14:41:25.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.261+0000 7fec9a38f740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fec901b4bf0 con 0x55e7bad58890 2024-04-02T14:41:25.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.261+0000 7fec9a38f740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fec901b1e70 con 0x55e7bad479f0 2024-04-02T14:41:25.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.261+0000 7fec9a38f740 10 client.0 _ll_register_callbacks cb 0x7fec7c004430 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:41:25.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.261+0000 7fec94c00640 1 -- 172.21.15.90:0/3349527145 learned_addr learned my addr 172.21.15.90:0/3349527145 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:41:25.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.262+0000 7fec8fe00640 1 --2- 172.21.15.90:0/3349527145 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e7bad479f0 0x55e7bad52270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:25.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.262+0000 7fec95600640 1 --2- 172.21.15.90:0/3349527145 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e7bad58890 0x55e7bad527b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:25.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.262+0000 7fec94c00640 1 -- 172.21.15.90:0/3349527145 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e7bad479f0 msgr2=0x55e7bad52270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:41:25.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.262+0000 7fec94c00640 1 --2- 172.21.15.90:0/3349527145 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e7bad479f0 0x55e7bad52270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:25.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.262+0000 7fec94c00640 1 -- 172.21.15.90:0/3349527145 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e7bad58890 msgr2=0x55e7bad527b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:41:25.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.262+0000 7fec94c00640 1 --2- 172.21.15.90:0/3349527145 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e7bad58890 0x55e7bad527b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:25.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.262+0000 7fec94c00640 1 -- 172.21.15.90:0/3349527145 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e7bad52fe0 con 0x55e7bad46d10 2024-04-02T14:41:25.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.263+0000 7fec94c00640 1 --2- 172.21.15.90:0/3349527145 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e7bad46d10 0x55e7bad60960 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fec8400bea0 tx=0x7fec84004f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a32c433b5135549e server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:41:25.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.263+0000 7fec76000640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-02T14:41:25.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.263+0000 7fec76000640 1 -- 172.21.15.90:0/3349527145 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec8401d820 con 0x55e7bad46d10 2024-04-02T14:41:25.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.263+0000 7fec76000640 1 -- 172.21.15.90:0/3349527145 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7fec84004020 con 0x55e7bad46d10 2024-04-02T14:41:25.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.263+0000 7fec9a38f740 1 -- 172.21.15.90:0/3349527145 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fec880bf060 con 0x55e7bad46d10 2024-04-02T14:41:25.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.264+0000 7fec76000640 1 -- 172.21.15.90:0/3349527145 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fec8402d050 con 0x55e7bad46d10 2024-04-02T14:41:25.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.265+0000 7fec9a38f740 10 client.16755 fetch_fsmap learned FSMap version 27 2024-04-02T14:41:25.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.265+0000 7fec9a38f740 1 -- 172.21.15.90:0/3349527145 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55e7badceeb0 con 0x55e7bad46d10 2024-04-02T14:41:25.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.266+0000 7fec76000640 1 -- 172.21.15.90:0/3349527145 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fec880bf060 con 0x55e7bad46d10 2024-04-02T14:41:25.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.266+0000 7fec76000640 1 -- 172.21.15.90:0/3349527145 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fec84011440 con 0x55e7bad46d10 2024-04-02T14:41:25.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.266+0000 7fec9a38f740 10 client.16755 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:41:25.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.266+0000 7fec9a38f740 10 client.16755 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:41:25.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.266+0000 7fec9a38f740 20 client.16755 populate_metadata read hostname 'smithi090' 2024-04-02T14:41:25.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.266+0000 7fec9a38f740 1 --2- 172.21.15.90:0/3349527145 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55e7badcf530 0x55e7badd1950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:25.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.266+0000 7fec9a38f740 4 client.16755 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-02T14:41:25.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.266+0000 7fec9a38f740 1 -- 172.21.15.90:0/3349527145 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55e7bacb8c30 con 0x55e7badcf530 2024-04-02T14:41:25.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.266+0000 7fec8fe00640 1 --2- 172.21.15.90:0/3349527145 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55e7badcf530 0x55e7badd1950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:25.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.267+0000 7fec76000640 10 client.16755 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:41:25.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.268+0000 7fec8fe00640 1 --2- 172.21.15.90:0/3349527145 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55e7badcf530 0x55e7badd1950 crc :-1 s=READY pgs=183 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dd7ba52e4b47fe9e server_cookie=d0b093742e826215 in_seq=0 out_seq=0 2024-04-02T14:41:25.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.268+0000 7fec76000640 1 -- 172.21.15.90:0/3349527145 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55e7bacb8c30 con 0x55e7badcf530 2024-04-02T14:41:25.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.268+0000 7fec76000640 10 client.16755 handle_command_reply: tid=0 2024-04-02T14:41:25.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.268+0000 7fec9a38f740 1 client.16755 shutdown 2024-04-02T14:41:25.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.269+0000 7fec9a38f740 1 -- 172.21.15.90:0/3349527145 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e7bad46d10 msgr2=0x55e7bad60960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:41:25.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.269+0000 7fec9a38f740 1 --2- 172.21.15.90:0/3349527145 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e7bad46d10 0x55e7bad60960 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fec8400bea0 tx=0x7fec84004f00 comp rx=0 tx=0).stop 2024-04-02T14:41:25.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.269+0000 7fec9a38f740 1 -- 172.21.15.90:0/3349527145 shutdown_connections 2024-04-02T14:41:25.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.269+0000 7fec9a38f740 1 --2- 172.21.15.90:0/3349527145 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55e7badcf530 0x55e7badd1950 crc :-1 s=READY pgs=183 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:25.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.269+0000 7fec9a38f740 1 --2- 172.21.15.90:0/3349527145 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e7bad58890 0x55e7bad527b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:25.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.269+0000 7fec9a38f740 1 --2- 172.21.15.90:0/3349527145 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e7bad479f0 0x55e7bad52270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:25.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.269+0000 7fec76000640 0 client.16755 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:41:25.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.269+0000 7fec9a38f740 1 --2- 172.21.15.90:0/3349527145 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e7bad46d10 0x55e7bad60960 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:25.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.269+0000 7fec9a38f740 1 -- 172.21.15.90:0/3349527145 >> 172.21.15.90:0/3349527145 conn(0x55e7bad0da50 msgr2=0x55e7bad40010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:41:25.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.269+0000 7fec9a38f740 1 -- 172.21.15.90:0/3349527145 shutdown_connections 2024-04-02T14:41:25.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.270+0000 7fec9a38f740 1 -- 172.21.15.90:0/3349527145 wait complete. 2024-04-02T14:41:25.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.270+0000 7fec9a38f740 20 client.16755 trim_cache size 0 max 16384 2024-04-02T14:41:25.269 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-02T14:41:25.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.271+0000 7fec97600640 1 -- 172.21.15.90:0/3808585973 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fec780a3e60 msgr2=0x7fec780a6320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:41:25.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.271+0000 7fec97600640 1 --2- 172.21.15.90:0/3808585973 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fec780a3e60 0x7fec780a6320 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x55e7bad47f40 tx=0x7fec7c00d040 comp rx=0 tx=0).stop 2024-04-02T14:41:25.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.271+0000 7fec97600640 1 -- 172.21.15.90:0/3808585973 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fec90113340 msgr2=0x7fec901a7680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:41:25.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.271+0000 7fec97600640 1 --2- 172.21.15.90:0/3808585973 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fec90113340 0x7fec901a7680 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fec8802c4d0 tx=0x7fec88032870 comp rx=0 tx=0).stop 2024-04-02T14:41:25.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.272+0000 7fec97600640 1 -- 172.21.15.90:0/3808585973 shutdown_connections 2024-04-02T14:41:25.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.272+0000 7fec97600640 1 --2- 172.21.15.90:0/3808585973 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fec780a3e60 0x7fec780a6320 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:25.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.272+0000 7fec97600640 1 --2- 172.21.15.90:0/3808585973 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fec90113340 0x7fec901a7680 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:25.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.272+0000 7fec97600640 1 --2- 172.21.15.90:0/3808585973 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fec90109040 0x7fec901a3300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:25.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.272+0000 7fec97600640 1 --2- 172.21.15.90:0/3808585973 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fec900695f0 0x7fec901a2dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:25.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.272+0000 7fec97600640 1 -- 172.21.15.90:0/3808585973 >> 172.21.15.90:0/3808585973 conn(0x7fec9006d040 msgr2=0x7fec900705c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:41:25.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.272+0000 7fec97600640 1 -- 172.21.15.90:0/3808585973 shutdown_connections 2024-04-02T14:41:25.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:25.272+0000 7fec97600640 1 -- 172.21.15.90:0/3808585973 wait complete. 2024-04-02T14:41:25.986 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:41:25 smithi177 ceph-mon[36431]: pgmap v199: 97 pgs: 97 active+clean; 4.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 9.0 MiB/s wr, 917 op/s 2024-04-02T14:41:26.108 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:41:25 smithi092 ceph-mon[36497]: pgmap v199: 97 pgs: 97 active+clean; 4.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 9.0 MiB/s wr, 917 op/s 2024-04-02T14:41:26.186 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:41:25 smithi090 ceph-mon[31674]: pgmap v199: 97 pgs: 97 active+clean; 4.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 9.0 MiB/s wr, 917 op/s 2024-04-02T14:41:26.297 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-02T14:41:26.297 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T14:41:26.297 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-02T14:41:26.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.605+0000 7f6bbd600640 1 -- 172.21.15.90:0/3983173394 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6bb8109040 msgr2=0x7f6bb81094c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:41:26.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.605+0000 7f6bbd600640 1 --2- 172.21.15.90:0/3983173394 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6bb8109040 0x7f6bb81094c0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f6ba4009900 tx=0x7f6ba402bc30 comp rx=0 tx=0).stop 2024-04-02T14:41:26.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.609+0000 7f6bbd600640 1 -- 172.21.15.90:0/3983173394 shutdown_connections 2024-04-02T14:41:26.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.609+0000 7f6bbd600640 1 --2- 172.21.15.90:0/3983173394 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6bb8113340 0x7f6bb8115730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:26.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.609+0000 7f6bbd600640 1 --2- 172.21.15.90:0/3983173394 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6bb8109040 0x7f6bb81094c0 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:26.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.609+0000 7f6bbd600640 1 --2- 172.21.15.90:0/3983173394 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6bb810ebb0 0x7f6bb8108b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:26.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.609+0000 7f6bbd600640 1 -- 172.21.15.90:0/3983173394 >> 172.21.15.90:0/3983173394 conn(0x7f6bb806d040 msgr2=0x7f6bb806d450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:41:26.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.610+0000 7f6bbd600640 1 -- 172.21.15.90:0/3983173394 shutdown_connections 2024-04-02T14:41:26.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.610+0000 7f6bbd600640 1 -- 172.21.15.90:0/3983173394 wait complete. 2024-04-02T14:41:26.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.611+0000 7f6bbd600640 1 Processor -- start 2024-04-02T14:41:26.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.612+0000 7f6bbd600640 1 -- start start 2024-04-02T14:41:26.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.612+0000 7f6bbd600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6bb8109040 0x7f6bb811e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:26.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.612+0000 7f6bbd600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6bb810ebb0 0x7f6bb811eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:26.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.612+0000 7f6bbd600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6bb8113340 0x7f6bb8119ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:26.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.612+0000 7f6bbd600640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f6bb81184b0 con 0x7f6bb8113340 2024-04-02T14:41:26.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.612+0000 7f6bbd600640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f6bb8118330 con 0x7f6bb810ebb0 2024-04-02T14:41:26.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.612+0000 7f6bbd600640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f6bb8118630 con 0x7f6bb8109040 2024-04-02T14:41:26.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.613+0000 7f6bb6a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6bb8109040 0x7f6bb811e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:26.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.613+0000 7f6bb6a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6bb8109040 0x7f6bb811e9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:42670/0 (socket says 172.21.15.90:42670) 2024-04-02T14:41:26.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.613+0000 7f6bb6a00640 1 -- 172.21.15.90:0/3671408369 learned_addr learned my addr 172.21.15.90:0/3671408369 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:41:26.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.613+0000 7f6bb6000640 1 --2- 172.21.15.90:0/3671408369 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6bb810ebb0 0x7f6bb811eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:26.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.613+0000 7f6bb7400640 1 --2- 172.21.15.90:0/3671408369 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6bb8113340 0x7f6bb8119ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:26.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.614+0000 7f6bb6a00640 1 -- 172.21.15.90:0/3671408369 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6bb810ebb0 msgr2=0x7f6bb811eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:41:26.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.614+0000 7f6bb6a00640 1 --2- 172.21.15.90:0/3671408369 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6bb810ebb0 0x7f6bb811eef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:26.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.614+0000 7f6bb6a00640 1 -- 172.21.15.90:0/3671408369 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6bb8113340 msgr2=0x7f6bb8119ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:41:26.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.614+0000 7f6bb6a00640 1 --2- 172.21.15.90:0/3671408369 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6bb8113340 0x7f6bb8119ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:26.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.614+0000 7f6bb6a00640 1 -- 172.21.15.90:0/3671408369 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6bb811a380 con 0x7f6bb8109040 2024-04-02T14:41:26.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.614+0000 7f6bb6000640 1 --2- 172.21.15.90:0/3671408369 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6bb810ebb0 0x7f6bb811eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:41:26.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.615+0000 7f6bb6a00640 1 --2- 172.21.15.90:0/3671408369 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6bb8109040 0x7f6bb811e9b0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f6ba800ea60 tx=0x7f6ba800ef20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:41:26.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.615+0000 7f6baf400640 1 -- 172.21.15.90:0/3671408369 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ba800cde0 con 0x7f6bb8109040 2024-04-02T14:41:26.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.615+0000 7f6baf400640 1 -- 172.21.15.90:0/3671408369 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6ba80035f0 con 0x7f6bb8109040 2024-04-02T14:41:26.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.615+0000 7f6baf400640 1 -- 172.21.15.90:0/3671408369 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ba8010690 con 0x7f6bb8109040 2024-04-02T14:41:26.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.615+0000 7f6bbd600640 1 -- 172.21.15.90:0/3671408369 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6bb811a670 con 0x7f6bb8109040 2024-04-02T14:41:26.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.616+0000 7f6bbd600640 1 -- 172.21.15.90:0/3671408369 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6bb81248a0 con 0x7f6bb8109040 2024-04-02T14:41:26.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.623+0000 7f6baf400640 1 -- 172.21.15.90:0/3671408369 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f6ba80037a0 con 0x7f6bb8109040 2024-04-02T14:41:26.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.626+0000 7f6bc0367740 1 Processor -- start 2024-04-02T14:41:26.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.626+0000 7f6bc0367740 1 -- start start 2024-04-02T14:41:26.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.628+0000 7f6baf400640 1 --2- 172.21.15.90:0/3671408369 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f6b9c0a3da0 0x7f6b9c0a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:26.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.628+0000 7f6baf400640 1 -- 172.21.15.90:0/3671408369 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f6ba80d0b30 con 0x7f6bb8109040 2024-04-02T14:41:26.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.629+0000 7f6bc0367740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ee93f4a030 0x55ee93f48d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:26.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.629+0000 7f6bb7400640 1 --2- 172.21.15.90:0/3671408369 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f6b9c0a3da0 0x7f6b9c0a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:26.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.630+0000 7f6bc0367740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ee93f49350 0x55ee93f497d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:26.630 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.630+0000 7f6bc0367740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ee93f526c0 0x55ee93f54ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:26.630 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.630+0000 7f6bc0367740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55ee93ead310 con 0x55ee93f526c0 2024-04-02T14:41:26.630 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.630+0000 7f6bc0367740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55ee93e55690 con 0x55ee93f4a030 2024-04-02T14:41:26.630 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.630+0000 7f6bc0367740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55ee93ea92c0 con 0x55ee93f49350 2024-04-02T14:41:26.630 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.630+0000 7f6bb6a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ee93f4a030 0x55ee93f48d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:26.630 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.630+0000 7f6bb6a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ee93f4a030 0x55ee93f48d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:35544/0 (socket says 172.21.15.90:35544) 2024-04-02T14:41:26.630 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.630+0000 7f6bb6a00640 1 -- 172.21.15.90:0/2491081753 learned_addr learned my addr 172.21.15.90:0/2491081753 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:41:26.630 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.630+0000 7f6bb6000640 1 --2- 172.21.15.90:0/2491081753 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ee93f49350 0x55ee93f497d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:26.630 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.631+0000 7f6bb7400640 1 --2- 172.21.15.90:0/3671408369 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f6b9c0a3da0 0x7f6b9c0a6260 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f6bb8072c80 tx=0x7f6bb0009500 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:41:26.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.631+0000 7f6bb6a00640 1 -- 172.21.15.90:0/2491081753 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ee93f49350 msgr2=0x55ee93f497d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:41:26.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.631+0000 7f6bb6a00640 1 --2- 172.21.15.90:0/2491081753 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ee93f49350 0x55ee93f497d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:26.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.631+0000 7f6bb6a00640 1 -- 172.21.15.90:0/2491081753 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ee93f526c0 msgr2=0x55ee93f54ab0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-02T14:41:26.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.634+0000 7f6bb7400640 1 --2- 172.21.15.90:0/2491081753 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ee93f526c0 0x55ee93f54ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:26.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.634+0000 7f6bb6a00640 1 --2- 172.21.15.90:0/2491081753 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ee93f526c0 0x55ee93f54ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:26.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.634+0000 7f6bb6a00640 1 -- 172.21.15.90:0/2491081753 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ee93c897e0 con 0x55ee93f4a030 2024-04-02T14:41:26.634 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.635+0000 7f6bb6a00640 1 --2- 172.21.15.90:0/2491081753 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ee93f4a030 0x55ee93f48d80 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f6ba80994b0 tx=0x7f6ba809c500 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f491e6afdb3c8be1 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:41:26.634 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.635+0000 7f6ba3400640 1 -- 172.21.15.90:0/2491081753 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ba80a5750 con 0x55ee93f4a030 2024-04-02T14:41:26.634 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.636+0000 7f6ba3400640 1 -- 172.21.15.90:0/2491081753 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6ba80bd070 con 0x55ee93f4a030 2024-04-02T14:41:26.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.637+0000 7f6bc0367740 1 -- 172.21.15.90:0/2491081753 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ee93f4a030 msgr2=0x55ee93f48d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:41:26.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.637+0000 7f6bc0367740 1 --2- 172.21.15.90:0/2491081753 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ee93f4a030 0x55ee93f48d80 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f6ba80994b0 tx=0x7f6ba809c500 comp rx=0 tx=0).stop 2024-04-02T14:41:26.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.637+0000 7f6bc0367740 1 -- 172.21.15.90:0/2491081753 shutdown_connections 2024-04-02T14:41:26.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.637+0000 7f6bc0367740 1 --2- 172.21.15.90:0/2491081753 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ee93f526c0 0x55ee93f54ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:26.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.637+0000 7f6bc0367740 1 --2- 172.21.15.90:0/2491081753 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ee93f49350 0x55ee93f497d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:26.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.637+0000 7f6bc0367740 1 --2- 172.21.15.90:0/2491081753 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ee93f4a030 0x55ee93f48d80 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:26.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.637+0000 7f6bc0367740 1 -- 172.21.15.90:0/2491081753 >> 172.21.15.90:0/2491081753 conn(0x55ee93edd270 msgr2=0x55ee93f43da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:41:26.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.637+0000 7f6bc0367740 1 -- 172.21.15.90:0/2491081753 shutdown_connections 2024-04-02T14:41:26.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.637+0000 7f6bc0367740 1 -- 172.21.15.90:0/2491081753 wait complete. 2024-04-02T14:41:26.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.638+0000 7f6bc0367740 1 -- start start 2024-04-02T14:41:26.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.638+0000 7f6bc0367740 1 Processor -- start 2024-04-02T14:41:26.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.639+0000 7f6bc0367740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ee93f49350 0x55ee93f4f610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:26.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.639+0000 7f6bc0367740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ee93f4a030 0x55ee93f4fb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:26.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.639+0000 7f6bc0367740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ee93f526c0 0x55ee93f50090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:26.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.639+0000 7f6bc0367740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55ee93f55580 con 0x55ee93f49350 2024-04-02T14:41:26.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.639+0000 7f6bc0367740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55ee93e68a60 con 0x55ee93f4a030 2024-04-02T14:41:26.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.639+0000 7f6bc0367740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55ee93f55700 con 0x55ee93f526c0 2024-04-02T14:41:26.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.639+0000 7f6bc0367740 10 client.0 _ll_register_callbacks cb 0x55ee93e6e030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:41:26.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.639+0000 7f6bb7400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ee93f526c0 0x55ee93f50090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:26.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.639+0000 7f6bb7400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ee93f526c0 0x55ee93f50090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:42684/0 (socket says 172.21.15.90:42684) 2024-04-02T14:41:26.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.639+0000 7f6bb7400640 1 -- 172.21.15.90:0/168625862 learned_addr learned my addr 172.21.15.90:0/168625862 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:41:26.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.640+0000 7f6bb7400640 1 -- 172.21.15.90:0/168625862 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ee93f4a030 msgr2=0x55ee93f4fb50 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-02T14:41:26.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.640+0000 7f6bb7400640 1 --2- 172.21.15.90:0/168625862 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ee93f4a030 0x55ee93f4fb50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:26.640 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.640+0000 7f6bb7400640 1 -- 172.21.15.90:0/168625862 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ee93f49350 msgr2=0x55ee93f4f610 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-02T14:41:26.640 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.640+0000 7f6bb6a00640 1 --2- 172.21.15.90:0/168625862 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ee93f49350 0x55ee93f4f610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:26.640 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.640+0000 7f6bb7400640 1 --2- 172.21.15.90:0/168625862 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ee93f49350 0x55ee93f4f610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:26.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.640+0000 7f6bb7400640 1 -- 172.21.15.90:0/168625862 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ee93ce9f70 con 0x55ee93f526c0 2024-04-02T14:41:26.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.641+0000 7f6bb7400640 1 --2- 172.21.15.90:0/168625862 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ee93f526c0 0x55ee93f50090 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f6bb0011c50 tx=0x7f6bb0004ab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a97035056597349 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:41:26.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.641+0000 7f6b97e00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:41:26.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.641+0000 7f6b97e00640 1 -- 172.21.15.90:0/168625862 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bb0003970 con 0x55ee93f526c0 2024-04-02T14:41:26.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.641+0000 7f6b97e00640 1 -- 172.21.15.90:0/168625862 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f6bb0003b10 con 0x55ee93f526c0 2024-04-02T14:41:26.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.641+0000 7f6bc0367740 1 -- 172.21.15.90:0/168625862 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6ba80bd070 con 0x55ee93f526c0 2024-04-02T14:41:26.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.643+0000 7f6b97e00640 1 -- 172.21.15.90:0/168625862 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f6bb0003d10 con 0x55ee93f526c0 2024-04-02T14:41:26.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.644+0000 7f6bc0367740 10 client.27313 fetch_fsmap learned FSMap version 27 2024-04-02T14:41:26.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.644+0000 7f6bc0367740 1 -- 172.21.15.90:0/168625862 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55ee93b81170 con 0x55ee93f526c0 2024-04-02T14:41:26.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.644+0000 7f6b97e00640 1 -- 172.21.15.90:0/168625862 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6ba80bd070 con 0x55ee93f526c0 2024-04-02T14:41:26.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.644+0000 7f6b97e00640 1 -- 172.21.15.90:0/168625862 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f6bb0030750 con 0x55ee93f526c0 2024-04-02T14:41:26.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.645+0000 7f6bc0367740 10 client.27313 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:41:26.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.645+0000 7f6bc0367740 10 client.27313 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:41:26.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.645+0000 7f6bc0367740 20 client.27313 populate_metadata read hostname 'smithi090' 2024-04-02T14:41:26.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.645+0000 7f6bc0367740 1 --2- 172.21.15.90:0/168625862 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55ee93f62b50 0x55ee93f64fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:26.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.645+0000 7f6bc0367740 4 client.27313 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:41:26.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.645+0000 7f6bc0367740 1 -- 172.21.15.90:0/168625862 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ee93eba850 con 0x55ee93f62b50 2024-04-02T14:41:26.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.645+0000 7f6bb6a00640 1 --2- 172.21.15.90:0/168625862 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55ee93f62b50 0x55ee93f64fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:26.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.646+0000 7f6b97e00640 10 client.27313 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:41:26.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.649+0000 7f6bb6a00640 1 --2- 172.21.15.90:0/168625862 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55ee93f62b50 0x55ee93f64fe0 crc :-1 s=READY pgs=184 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8b3961b91b9a44d4 server_cookie=f1f132675c8752ba in_seq=0 out_seq=0 2024-04-02T14:41:26.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.650+0000 7f6b97e00640 1 -- 172.21.15.90:0/168625862 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ee93eba850 con 0x55ee93f62b50 2024-04-02T14:41:26.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.650+0000 7f6b97e00640 10 client.27313 handle_command_reply: tid=0 2024-04-02T14:41:26.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.651+0000 7f6bc0367740 1 client.27313 shutdown 2024-04-02T14:41:26.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.651+0000 7f6bc0367740 1 -- 172.21.15.90:0/168625862 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ee93f526c0 msgr2=0x55ee93f50090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:41:26.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.651+0000 7f6bc0367740 1 --2- 172.21.15.90:0/168625862 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ee93f526c0 0x55ee93f50090 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f6bb0011c50 tx=0x7f6bb0004ab0 comp rx=0 tx=0).stop 2024-04-02T14:41:26.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.651+0000 7f6bc0367740 1 -- 172.21.15.90:0/168625862 shutdown_connections 2024-04-02T14:41:26.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.651+0000 7f6bc0367740 1 --2- 172.21.15.90:0/168625862 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55ee93f62b50 0x55ee93f64fe0 crc :-1 s=READY pgs=184 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:26.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.651+0000 7f6bc0367740 1 --2- 172.21.15.90:0/168625862 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ee93f526c0 0x55ee93f50090 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:26.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.651+0000 7f6bc0367740 1 --2- 172.21.15.90:0/168625862 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ee93f4a030 0x55ee93f4fb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:26.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.651+0000 7f6bc0367740 1 --2- 172.21.15.90:0/168625862 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ee93f49350 0x55ee93f4f610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:26.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.651+0000 7f6bc0367740 1 -- 172.21.15.90:0/168625862 >> 172.21.15.90:0/168625862 conn(0x55ee93edd270 msgr2=0x55ee93f45af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:41:26.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.652+0000 7f6b97e00640 0 client.27313 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:41:26.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.652+0000 7f6bc0367740 1 -- 172.21.15.90:0/168625862 shutdown_connections 2024-04-02T14:41:26.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.652+0000 7f6bc0367740 1 -- 172.21.15.90:0/168625862 wait complete. 2024-04-02T14:41:26.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.652+0000 7f6bc0367740 20 client.27313 trim_cache size 0 max 16384 2024-04-02T14:41:26.695 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.696+0000 7f6bc0367740 1 Processor -- start 2024-04-02T14:41:26.695 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.696+0000 7f6bc0367740 1 -- start start 2024-04-02T14:41:26.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.696+0000 7f6bc0367740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ee93f49350 0x55ee93f51d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:26.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.696+0000 7f6bc0367740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ee93f4a030 0x55ee93f5e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:26.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.697+0000 7f6bc0367740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ee93f526c0 0x55ee93f6a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:26.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.697+0000 7f6bc0367740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f6bb81c4fd0 con 0x55ee93f49350 2024-04-02T14:41:26.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.697+0000 7f6bc0367740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f6bb81c6b10 con 0x55ee93f526c0 2024-04-02T14:41:26.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.697+0000 7f6bc0367740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f6bb81c56a0 con 0x55ee93f4a030 2024-04-02T14:41:26.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.697+0000 7f6bb6000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ee93f4a030 0x55ee93f5e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:26.697 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.697+0000 7f6bb6000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ee93f4a030 0x55ee93f5e4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:42692/0 (socket says 172.21.15.90:42692) 2024-04-02T14:41:26.697 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.697+0000 7f6bb6000640 1 -- 172.21.15.90:0/2739576182 learned_addr learned my addr 172.21.15.90:0/2739576182 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:41:26.697 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.697+0000 7f6bb7400640 1 --2- 172.21.15.90:0/2739576182 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ee93f526c0 0x55ee93f6a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:26.697 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.698+0000 7f6bb6000640 1 -- 172.21.15.90:0/2739576182 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ee93f526c0 msgr2=0x55ee93f6a8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:41:26.697 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.698+0000 7f6bb6000640 1 --2- 172.21.15.90:0/2739576182 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ee93f526c0 0x55ee93f6a8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:26.697 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.698+0000 7f6bb6000640 1 -- 172.21.15.90:0/2739576182 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ee93f49350 msgr2=0x55ee93f51d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:41:26.697 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.698+0000 7f6bb6000640 1 --2- 172.21.15.90:0/2739576182 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ee93f49350 0x55ee93f51d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:26.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.698+0000 7f6bb6000640 1 -- 172.21.15.90:0/2739576182 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ee93f617a0 con 0x55ee93f4a030 2024-04-02T14:41:26.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.698+0000 7f6bb6000640 1 --2- 172.21.15.90:0/2739576182 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ee93f4a030 0x55ee93f5e4d0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f6ba4009230 tx=0x7f6ba402c5d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2afbbbc33ddbcc0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:41:26.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.699+0000 7f6ba2a00640 1 -- 172.21.15.90:0/2739576182 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ba4004a70 con 0x55ee93f4a030 2024-04-02T14:41:26.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.699+0000 7f6ba2a00640 1 -- 172.21.15.90:0/2739576182 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6ba4004c10 con 0x55ee93f4a030 2024-04-02T14:41:26.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.699+0000 7f6ba2a00640 1 -- 172.21.15.90:0/2739576182 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ba40052e0 con 0x55ee93f4a030 2024-04-02T14:41:26.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.701+0000 7f6bc0367740 1 -- 172.21.15.90:0/2739576182 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ee93f4a030 msgr2=0x55ee93f5e4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:41:26.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.701+0000 7f6bc0367740 1 --2- 172.21.15.90:0/2739576182 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ee93f4a030 0x55ee93f5e4d0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f6ba4009230 tx=0x7f6ba402c5d0 comp rx=0 tx=0).stop 2024-04-02T14:41:26.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.702+0000 7f6bc0367740 1 -- 172.21.15.90:0/2739576182 shutdown_connections 2024-04-02T14:41:26.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.702+0000 7f6bc0367740 1 --2- 172.21.15.90:0/2739576182 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ee93f526c0 0x55ee93f6a8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:26.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.702+0000 7f6bc0367740 1 --2- 172.21.15.90:0/2739576182 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ee93f4a030 0x55ee93f5e4d0 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:26.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.702+0000 7f6bc0367740 1 --2- 172.21.15.90:0/2739576182 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ee93f49350 0x55ee93f51d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:26.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.702+0000 7f6bc0367740 1 -- 172.21.15.90:0/2739576182 >> 172.21.15.90:0/2739576182 conn(0x55ee93edd270 msgr2=0x55ee93f45a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:41:26.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.703+0000 7f6bc0367740 1 -- 172.21.15.90:0/2739576182 shutdown_connections 2024-04-02T14:41:26.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.703+0000 7f6bc0367740 1 -- 172.21.15.90:0/2739576182 wait complete. 2024-04-02T14:41:26.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.704+0000 7f6bc0367740 1 -- start start 2024-04-02T14:41:26.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.704+0000 7f6bc0367740 1 Processor -- start 2024-04-02T14:41:26.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.705+0000 7f6bc0367740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ee93f49350 0x55ee93f5ac40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:26.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.705+0000 7f6bc0367740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ee93f4a030 0x55ee93f579b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:26.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.705+0000 7f6bc0367740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ee93f526c0 0x55ee93f57ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:26.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.706+0000 7f6bb6a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ee93f49350 0x55ee93f5ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:26.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.706+0000 7f6bb6a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ee93f49350 0x55ee93f5ac40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:57448/0 (socket says 172.21.15.90:57448) 2024-04-02T14:41:26.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.706+0000 7f6bb6a00640 1 -- 172.21.15.90:0/213619823 learned_addr learned my addr 172.21.15.90:0/213619823 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:41:26.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.706+0000 7f6bc0367740 1 -- 172.21.15.90:0/213619823 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f6bb81c6440 con 0x55ee93f49350 2024-04-02T14:41:26.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.706+0000 7f6bc0367740 1 -- 172.21.15.90:0/213619823 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f6bb81c5d70 con 0x55ee93f4a030 2024-04-02T14:41:26.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.706+0000 7f6bc0367740 1 -- 172.21.15.90:0/213619823 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f6bb81c2fb0 con 0x55ee93f526c0 2024-04-02T14:41:26.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.706+0000 7f6bb7400640 1 --2- 172.21.15.90:0/213619823 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ee93f526c0 0x55ee93f57ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:26.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.706+0000 7f6bb6000640 1 --2- 172.21.15.90:0/213619823 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ee93f4a030 0x55ee93f579b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:26.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.706+0000 7f6bb7400640 1 -- 172.21.15.90:0/213619823 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ee93f4a030 msgr2=0x55ee93f579b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:41:26.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.706+0000 7f6bb7400640 1 --2- 172.21.15.90:0/213619823 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ee93f4a030 0x55ee93f579b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:26.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.706+0000 7f6bb7400640 1 -- 172.21.15.90:0/213619823 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ee93f49350 msgr2=0x55ee93f5ac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:41:26.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.706+0000 7f6bb7400640 1 --2- 172.21.15.90:0/213619823 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ee93f49350 0x55ee93f5ac40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:26.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.706+0000 7f6bb7400640 1 -- 172.21.15.90:0/213619823 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ee93f48380 con 0x55ee93f526c0 2024-04-02T14:41:26.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.707+0000 7f6bc0367740 10 client.0 _ll_register_callbacks cb 0x7f6bb0010ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:41:26.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.707+0000 7f6bc0367740 1 -- 172.21.15.90:0/213619823 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6ba4004c10 con 0x55ee93f526c0 2024-04-02T14:41:26.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.707+0000 7f6bb7400640 1 --2- 172.21.15.90:0/213619823 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ee93f526c0 0x55ee93f57ef0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f6bb0030b20 tx=0x7f6bb0011860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=11052b2f1097dbb2 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:41:26.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.707+0000 7f6b97e00640 10 client.27319 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:41:26.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.708+0000 7f6b97e00640 1 -- 172.21.15.90:0/213619823 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bb0030670 con 0x55ee93f526c0 2024-04-02T14:41:26.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.708+0000 7f6b97e00640 1 -- 172.21.15.90:0/213619823 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f6bb0030810 con 0x55ee93f526c0 2024-04-02T14:41:26.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.709+0000 7f6b97e00640 1 -- 172.21.15.90:0/213619823 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f6bb0004000 con 0x55ee93f526c0 2024-04-02T14:41:26.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.709+0000 7f6bc0367740 10 client.27319 fetch_fsmap learned FSMap version 27 2024-04-02T14:41:26.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.709+0000 7f6bc0367740 1 -- 172.21.15.90:0/213619823 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({fsmap=27}) -- 0x55ee93fc9ca0 con 0x55ee93f526c0 2024-04-02T14:41:26.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.710+0000 7f6b97e00640 1 -- 172.21.15.90:0/213619823 <== mon.1 v2:172.21.15.177:3300/0 4 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f6bb00041a0 con 0x55ee93f526c0 2024-04-02T14:41:26.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.710+0000 7f6bc0367740 10 client.27319 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:41:26.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.710+0000 7f6bc0367740 10 client.27319 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:41:26.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.710+0000 7f6bc0367740 20 client.27319 populate_metadata read hostname 'smithi090' 2024-04-02T14:41:26.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.710+0000 7f6bc0367740 1 --2- 172.21.15.90:0/213619823 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55ee93fca260 0x55ee93fcc6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:26.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.710+0000 7f6bc0367740 4 client.27319 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-02T14:41:26.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.710+0000 7f6bc0367740 1 -- 172.21.15.90:0/213619823 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55ee93eb4ba0 con 0x55ee93fca260 2024-04-02T14:41:26.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.711+0000 7f6bb6a00640 1 --2- 172.21.15.90:0/213619823 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55ee93fca260 0x55ee93fcc6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:26.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.712+0000 7f6bb6a00640 1 --2- 172.21.15.90:0/213619823 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55ee93fca260 0x55ee93fcc6a0 crc :-1 s=READY pgs=185 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=28f5e24c29e0fb48 server_cookie=6100dc3f7af8e87 in_seq=0 out_seq=0 2024-04-02T14:41:26.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.712+0000 7f6b97e00640 10 client.27319 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:41:26.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.713+0000 7f6b97e00640 1 -- 172.21.15.90:0/213619823 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55ee93eb4ba0 con 0x55ee93fca260 2024-04-02T14:41:26.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.713+0000 7f6b97e00640 10 client.27319 handle_command_reply: tid=0 2024-04-02T14:41:26.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.713+0000 7f6bc0367740 1 client.27319 shutdown 2024-04-02T14:41:26.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.714+0000 7f6bc0367740 1 -- 172.21.15.90:0/213619823 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ee93f526c0 msgr2=0x55ee93f57ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:41:26.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.714+0000 7f6bc0367740 1 --2- 172.21.15.90:0/213619823 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ee93f526c0 0x55ee93f57ef0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f6bb0030b20 tx=0x7f6bb0011860 comp rx=0 tx=0).stop 2024-04-02T14:41:26.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.714+0000 7f6bc0367740 1 -- 172.21.15.90:0/213619823 shutdown_connections 2024-04-02T14:41:26.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.714+0000 7f6bc0367740 1 --2- 172.21.15.90:0/213619823 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55ee93fca260 0x55ee93fcc6a0 crc :-1 s=READY pgs=185 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:26.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.714+0000 7f6bc0367740 1 --2- 172.21.15.90:0/213619823 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ee93f526c0 0x55ee93f57ef0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:26.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.714+0000 7f6bc0367740 1 --2- 172.21.15.90:0/213619823 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ee93f4a030 0x55ee93f579b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:26.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.714+0000 7f6bc0367740 1 --2- 172.21.15.90:0/213619823 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ee93f49350 0x55ee93f5ac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:26.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.714+0000 7f6bc0367740 1 -- 172.21.15.90:0/213619823 >> 172.21.15.90:0/213619823 conn(0x55ee93edd270 msgr2=0x55ee93f45180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:41:26.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.714+0000 7f6b97e00640 0 client.27319 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:41:26.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.715+0000 7f6bc0367740 1 -- 172.21.15.90:0/213619823 shutdown_connections 2024-04-02T14:41:26.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.715+0000 7f6bc0367740 1 -- 172.21.15.90:0/213619823 wait complete. 2024-04-02T14:41:26.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.715+0000 7f6bc0367740 20 client.27319 trim_cache size 0 max 16384 2024-04-02T14:41:26.715 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-02T14:41:26.715 INFO:teuthology.orchestra.run.smithi090.stdout: "return_code": 0, 2024-04-02T14:41:26.715 INFO:teuthology.orchestra.run.smithi090.stdout: "scrub_tag": "ed86a8cb-4ca9-4377-9358-bfa61a387de0", 2024-04-02T14:41:26.715 INFO:teuthology.orchestra.run.smithi090.stdout: "mode": "asynchronous" 2024-04-02T14:41:26.715 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-02T14:41:26.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.717+0000 7f6bbd600640 1 -- 172.21.15.90:0/3671408369 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f6b9c0a3da0 msgr2=0x7f6b9c0a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:41:26.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.717+0000 7f6bbd600640 1 --2- 172.21.15.90:0/3671408369 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f6b9c0a3da0 0x7f6b9c0a6260 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f6bb8072c80 tx=0x7f6bb0009500 comp rx=0 tx=0).stop 2024-04-02T14:41:26.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.717+0000 7f6bbd600640 1 -- 172.21.15.90:0/3671408369 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6bb8109040 msgr2=0x7f6bb811e9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:41:26.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.717+0000 7f6bbd600640 1 --2- 172.21.15.90:0/3671408369 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6bb8109040 0x7f6bb811e9b0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f6ba800ea60 tx=0x7f6ba800ef20 comp rx=0 tx=0).stop 2024-04-02T14:41:26.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.717+0000 7f6bbd600640 1 -- 172.21.15.90:0/3671408369 shutdown_connections 2024-04-02T14:41:26.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.717+0000 7f6bbd600640 1 --2- 172.21.15.90:0/3671408369 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f6b9c0a3da0 0x7f6b9c0a6260 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:26.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.717+0000 7f6bbd600640 1 --2- 172.21.15.90:0/3671408369 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6bb8113340 0x7f6bb8119ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:26.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.717+0000 7f6bbd600640 1 --2- 172.21.15.90:0/3671408369 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6bb810ebb0 0x7f6bb811eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:26.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.717+0000 7f6bbd600640 1 --2- 172.21.15.90:0/3671408369 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6bb8109040 0x7f6bb811e9b0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:26.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.717+0000 7f6bbd600640 1 -- 172.21.15.90:0/3671408369 >> 172.21.15.90:0/3671408369 conn(0x7f6bb806d040 msgr2=0x7f6bb80705c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:41:26.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.717+0000 7f6bbd600640 1 -- 172.21.15.90:0/3671408369 shutdown_connections 2024-04-02T14:41:26.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:26.717+0000 7f6bbd600640 1 -- 172.21.15.90:0/3671408369 wait complete. 2024-04-02T14:41:26.744 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-02T14:41:26.744 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-02T14:41:26.832 INFO:teuthology.orchestra.run.smithi090.stderr:progress: 355.0 s, 3084.8 tps, lat 32.449 ms stddev 37.572 2024-04-02T14:41:27.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.079+0000 7fb4d9a00640 1 -- 172.21.15.90:0/599584674 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb4d41044a0 msgr2=0x7fb4d41109b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:41:27.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.079+0000 7fb4d9a00640 1 --2- 172.21.15.90:0/599584674 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb4d41044a0 0x7fb4d41109b0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fb4cc009c10 tx=0x7fb4cc02bf00 comp rx=0 tx=0).stop 2024-04-02T14:41:27.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.079+0000 7fb4d9a00640 1 -- 172.21.15.90:0/599584674 shutdown_connections 2024-04-02T14:41:27.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.079+0000 7fb4d9a00640 1 --2- 172.21.15.90:0/599584674 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb4d41044a0 0x7fb4d41109b0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:27.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.079+0000 7fb4d9a00640 1 --2- 172.21.15.90:0/599584674 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb4d4103ae0 0x7fb4d4103f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:27.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.079+0000 7fb4d9a00640 1 --2- 172.21.15.90:0/599584674 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb4d4109b80 0x7fb4d4109f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:27.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.079+0000 7fb4d9a00640 1 -- 172.21.15.90:0/599584674 >> 172.21.15.90:0/599584674 conn(0x7fb4d40ff590 msgr2=0x7fb4d41019b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:41:27.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.080+0000 7fb4d9a00640 1 -- 172.21.15.90:0/599584674 shutdown_connections 2024-04-02T14:41:27.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.080+0000 7fb4d9a00640 1 -- 172.21.15.90:0/599584674 wait complete. 2024-04-02T14:41:27.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.082+0000 7fb4d9a00640 1 Processor -- start 2024-04-02T14:41:27.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.082+0000 7fb4d9a00640 1 -- start start 2024-04-02T14:41:27.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.084+0000 7fb4d9a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb4d4103ae0 0x7fb4d4073370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:27.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.084+0000 7fb4d9a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb4d41044a0 0x7fb4d406c3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:27.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.084+0000 7fb4d9a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb4d4109b80 0x7fb4d406c930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:27.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.084+0000 7fb4d9a00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fb4d4113ed0 con 0x7fb4d4109b80 2024-04-02T14:41:27.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.084+0000 7fb4d9a00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fb4d4113d50 con 0x7fb4d41044a0 2024-04-02T14:41:27.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.084+0000 7fb4d9a00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fb4d4114050 con 0x7fb4d4103ae0 2024-04-02T14:41:27.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.084+0000 7fb4d2a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb4d4103ae0 0x7fb4d4073370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:27.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.084+0000 7fb4d2a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb4d4103ae0 0x7fb4d4073370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:42728/0 (socket says 172.21.15.90:42728) 2024-04-02T14:41:27.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.084+0000 7fb4d2a00640 1 -- 172.21.15.90:0/2105663690 learned_addr learned my addr 172.21.15.90:0/2105663690 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:41:27.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.084+0000 7fb4d2000640 1 --2- 172.21.15.90:0/2105663690 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb4d41044a0 0x7fb4d406c3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:27.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.084+0000 7fb4d2a00640 1 -- 172.21.15.90:0/2105663690 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb4d41044a0 msgr2=0x7fb4d406c3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:41:27.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.084+0000 7fb4d2a00640 1 --2- 172.21.15.90:0/2105663690 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb4d41044a0 0x7fb4d406c3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:27.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.084+0000 7fb4d2a00640 1 -- 172.21.15.90:0/2105663690 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb4d4109b80 msgr2=0x7fb4d406c930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:41:27.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.084+0000 7fb4d2a00640 1 --2- 172.21.15.90:0/2105663690 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb4d4109b80 0x7fb4d406c930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:27.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.084+0000 7fb4d2a00640 1 -- 172.21.15.90:0/2105663690 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4d406cf70 con 0x7fb4d4103ae0 2024-04-02T14:41:27.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.085+0000 7fb4d2000640 1 --2- 172.21.15.90:0/2105663690 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb4d41044a0 0x7fb4d406c3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:41:27.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.086+0000 7fb4d2a00640 1 --2- 172.21.15.90:0/2105663690 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb4d4103ae0 0x7fb4d4073370 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fb4bc00e9c0 tx=0x7fb4bc00ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:41:27.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.086+0000 7fb4cb400640 1 -- 172.21.15.90:0/2105663690 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4bc00cd80 con 0x7fb4d4103ae0 2024-04-02T14:41:27.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.086+0000 7fb4cb400640 1 -- 172.21.15.90:0/2105663690 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb4bc0035f0 con 0x7fb4d4103ae0 2024-04-02T14:41:27.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.086+0000 7fb4cb400640 1 -- 172.21.15.90:0/2105663690 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4bc010690 con 0x7fb4d4103ae0 2024-04-02T14:41:27.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.087+0000 7fb4d9a00640 1 -- 172.21.15.90:0/2105663690 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb4d4070e90 con 0x7fb4d4103ae0 2024-04-02T14:41:27.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.087+0000 7fb4d9a00640 1 -- 172.21.15.90:0/2105663690 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4d4071390 con 0x7fb4d4103ae0 2024-04-02T14:41:27.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.094+0000 7fb4cb400640 1 -- 172.21.15.90:0/2105663690 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fb4bc0037a0 con 0x7fb4d4103ae0 2024-04-02T14:41:27.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.097+0000 7fb4cb400640 1 --2- 172.21.15.90:0/2105663690 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fb4b80a3e60 0x7fb4b80a6320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:27.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.099+0000 7fb4d2000640 1 --2- 172.21.15.90:0/2105663690 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fb4b80a3e60 0x7fb4b80a6320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:27.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.100+0000 7fb4cb400640 1 -- 172.21.15.90:0/2105663690 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7fb4bc0d04c0 con 0x7fb4d4103ae0 2024-04-02T14:41:27.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.101+0000 7fb4dc7ae740 1 Processor -- start 2024-04-02T14:41:27.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.102+0000 7fb4dc7ae740 1 -- start start 2024-04-02T14:41:27.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.103+0000 7fb4dc7ae740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f80d666960 0x55f80d666d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:27.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.103+0000 7fb4dc7ae740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f80d667280 0x55f80d66f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:27.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.103+0000 7fb4dc7ae740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f80d66fe60 0x55f80d672250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:27.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.103+0000 7fb4dc7ae740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55f80d5ca770 con 0x55f80d666960 2024-04-02T14:41:27.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.103+0000 7fb4dc7ae740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55f80d572700 con 0x55f80d667280 2024-04-02T14:41:27.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.103+0000 7fb4dc7ae740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55f80d5c6720 con 0x55f80d66fe60 2024-04-02T14:41:27.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.102+0000 7fb4d2000640 1 --2- 172.21.15.90:0/2105663690 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fb4b80a3e60 0x7fb4b80a6320 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fb4c40058c0 tx=0x7fb4c4005850 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:41:27.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.103+0000 7fb4d2a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f80d666960 0x55f80d666d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:27.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.104+0000 7fb4d3400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f80d66fe60 0x55f80d672250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:27.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.104+0000 7fb4d3400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f80d66fe60 0x55f80d672250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:42730/0 (socket says 172.21.15.90:42730) 2024-04-02T14:41:27.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.104+0000 7fb4d3400640 1 -- 172.21.15.90:0/2714984336 learned_addr learned my addr 172.21.15.90:0/2714984336 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:41:27.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.104+0000 7fb4d2000640 1 --2- 172.21.15.90:0/2714984336 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f80d667280 0x55f80d66f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:27.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.105+0000 7fb4d3400640 1 -- 172.21.15.90:0/2714984336 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f80d667280 msgr2=0x55f80d66f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:41:27.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.105+0000 7fb4d3400640 1 --2- 172.21.15.90:0/2714984336 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f80d667280 0x55f80d66f920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:27.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.105+0000 7fb4d3400640 1 -- 172.21.15.90:0/2714984336 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f80d666960 msgr2=0x55f80d666d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:41:27.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.105+0000 7fb4d3400640 1 --2- 172.21.15.90:0/2714984336 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f80d666960 0x55f80d666d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:27.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.105+0000 7fb4d3400640 1 -- 172.21.15.90:0/2714984336 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f80d3a87d0 con 0x55f80d66fe60 2024-04-02T14:41:27.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.105+0000 7fb4d2a00640 1 --2- 172.21.15.90:0/2714984336 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f80d666960 0x55f80d666d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:41:27.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.105+0000 7fb4d2000640 1 --2- 172.21.15.90:0/2714984336 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f80d667280 0x55f80d66f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:41:27.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.106+0000 7fb4d3400640 1 --2- 172.21.15.90:0/2714984336 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f80d66fe60 0x55f80d672250 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fb4cc036040 tx=0x7fb4cc009940 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9697b8e12b793bb3 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:41:27.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.107+0000 7fb4c3e00640 1 -- 172.21.15.90:0/2714984336 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4cc042020 con 0x55f80d66fe60 2024-04-02T14:41:27.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.108+0000 7fb4c3e00640 1 -- 172.21.15.90:0/2714984336 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb4cc033b90 con 0x55f80d66fe60 2024-04-02T14:41:27.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.108+0000 7fb4c3e00640 1 -- 172.21.15.90:0/2714984336 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4cc042020 con 0x55f80d66fe60 2024-04-02T14:41:27.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.110+0000 7fb4dc7ae740 1 -- 172.21.15.90:0/2714984336 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f80d66fe60 msgr2=0x55f80d672250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:41:27.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.110+0000 7fb4dc7ae740 1 --2- 172.21.15.90:0/2714984336 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f80d66fe60 0x55f80d672250 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fb4cc036040 tx=0x7fb4cc009940 comp rx=0 tx=0).stop 2024-04-02T14:41:27.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.112+0000 7fb4dc7ae740 1 -- 172.21.15.90:0/2714984336 shutdown_connections 2024-04-02T14:41:27.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.112+0000 7fb4dc7ae740 1 --2- 172.21.15.90:0/2714984336 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f80d66fe60 0x55f80d672250 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:27.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.112+0000 7fb4dc7ae740 1 --2- 172.21.15.90:0/2714984336 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f80d667280 0x55f80d66f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:27.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.112+0000 7fb4dc7ae740 1 --2- 172.21.15.90:0/2714984336 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f80d666960 0x55f80d666d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:27.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.112+0000 7fb4dc7ae740 1 -- 172.21.15.90:0/2714984336 >> 172.21.15.90:0/2714984336 conn(0x55f80d661090 msgr2=0x55f80d661470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:41:27.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.112+0000 7fb4dc7ae740 1 -- 172.21.15.90:0/2714984336 shutdown_connections 2024-04-02T14:41:27.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.113+0000 7fb4dc7ae740 1 -- 172.21.15.90:0/2714984336 wait complete. 2024-04-02T14:41:27.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.113+0000 7fb4dc7ae740 1 -- start start 2024-04-02T14:41:27.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.114+0000 7fb4dc7ae740 1 Processor -- start 2024-04-02T14:41:27.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.117+0000 7fb4dc7ae740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f80d666960 0x55f80d66ced0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:27.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.121+0000 7fb4dc7ae740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f80d667280 0x55f80d66d410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:27.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.121+0000 7fb4dc7ae740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f80d66fe60 0x55f80d66d950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:27.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.121+0000 7fb4dc7ae740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55f80d672bf0 con 0x55f80d667280 2024-04-02T14:41:27.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.121+0000 7fb4dc7ae740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55f80d585ac0 con 0x55f80d666960 2024-04-02T14:41:27.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.121+0000 7fb4dc7ae740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55f80d672d70 con 0x55f80d66fe60 2024-04-02T14:41:27.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.121+0000 7fb4dc7ae740 10 client.0 _ll_register_callbacks cb 0x55f80d588180 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:41:27.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.121+0000 7fb4d2a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f80d666960 0x55f80d66ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:27.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.121+0000 7fb4d2a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f80d666960 0x55f80d66ced0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:35608/0 (socket says 172.21.15.90:35608) 2024-04-02T14:41:27.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.121+0000 7fb4d2a00640 1 -- 172.21.15.90:0/3486801602 learned_addr learned my addr 172.21.15.90:0/3486801602 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:41:27.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.122+0000 7fb4d3400640 1 --2- 172.21.15.90:0/3486801602 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f80d66fe60 0x55f80d66d950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:27.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.122+0000 7fb4d2000640 1 --2- 172.21.15.90:0/3486801602 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f80d667280 0x55f80d66d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:27.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.122+0000 7fb4d2a00640 1 -- 172.21.15.90:0/3486801602 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f80d66fe60 msgr2=0x55f80d66d950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:41:27.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.122+0000 7fb4d2a00640 1 --2- 172.21.15.90:0/3486801602 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f80d66fe60 0x55f80d66d950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:27.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.122+0000 7fb4d2a00640 1 -- 172.21.15.90:0/3486801602 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f80d667280 msgr2=0x55f80d66d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:41:27.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.123+0000 7fb4d2a00640 1 --2- 172.21.15.90:0/3486801602 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f80d667280 0x55f80d66d410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:27.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.123+0000 7fb4d2a00640 1 -- 172.21.15.90:0/3486801602 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f80d2de3f0 con 0x55f80d666960 2024-04-02T14:41:27.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.123+0000 7fb4d2000640 1 --2- 172.21.15.90:0/3486801602 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f80d667280 0x55f80d66d410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:41:27.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.124+0000 7fb4d2a00640 1 --2- 172.21.15.90:0/3486801602 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f80d666960 0x55f80d66ced0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7fb4bc098de0 tx=0x7fb4bc09cc50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3bf55316de6f8a73 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:41:27.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.124+0000 7fb4c0c00640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:41:27.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.125+0000 7fb4c0c00640 1 -- 172.21.15.90:0/3486801602 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4bc0cde40 con 0x55f80d666960 2024-04-02T14:41:27.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.125+0000 7fb4c0c00640 1 -- 172.21.15.90:0/3486801602 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7fb4bc0cdfe0 con 0x55f80d666960 2024-04-02T14:41:27.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.126+0000 7fb4dc7ae740 1 -- 172.21.15.90:0/3486801602 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb4cc033b90 con 0x55f80d666960 2024-04-02T14:41:27.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.127+0000 7fb4c0c00640 1 -- 172.21.15.90:0/3486801602 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fb4bc0bb280 con 0x55f80d666960 2024-04-02T14:41:27.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.127+0000 7fb4dc7ae740 10 client.27461 fetch_fsmap learned FSMap version 27 2024-04-02T14:41:27.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.128+0000 7fb4dc7ae740 1 -- 172.21.15.90:0/3486801602 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55f80d29e0f0 con 0x55f80d666960 2024-04-02T14:41:27.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.128+0000 7fb4c0c00640 1 -- 172.21.15.90:0/3486801602 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb4cc033b90 con 0x55f80d666960 2024-04-02T14:41:27.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.129+0000 7fb4c0c00640 1 -- 172.21.15.90:0/3486801602 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fb4bc09b980 con 0x55f80d666960 2024-04-02T14:41:27.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.129+0000 7fb4dc7ae740 10 client.27461 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:41:27.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.129+0000 7fb4dc7ae740 10 client.27461 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:41:27.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.129+0000 7fb4dc7ae740 20 client.27461 populate_metadata read hostname 'smithi090' 2024-04-02T14:41:27.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.130+0000 7fb4dc7ae740 1 --2- 172.21.15.90:0/3486801602 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55f80d67fea0 0x55f80d682360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:27.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.130+0000 7fb4dc7ae740 4 client.27461 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:41:27.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.130+0000 7fb4dc7ae740 1 -- 172.21.15.90:0/3486801602 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f80d56b630 con 0x55f80d67fea0 2024-04-02T14:41:27.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.131+0000 7fb4d2000640 1 --2- 172.21.15.90:0/3486801602 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55f80d67fea0 0x55f80d682360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:27.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.132+0000 7fb4d2000640 1 --2- 172.21.15.90:0/3486801602 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55f80d67fea0 0x55f80d682360 crc :-1 s=READY pgs=186 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b08aec221f006a1b server_cookie=b8e7ef01a25be770 in_seq=0 out_seq=0 2024-04-02T14:41:27.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.132+0000 7fb4c0c00640 10 client.27461 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:41:27.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.133+0000 7fb4c0c00640 1 -- 172.21.15.90:0/3486801602 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f80d56b630 con 0x55f80d67fea0 2024-04-02T14:41:27.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.133+0000 7fb4c0c00640 10 client.27461 handle_command_reply: tid=0 2024-04-02T14:41:27.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.133+0000 7fb4dc7ae740 1 client.27461 shutdown 2024-04-02T14:41:27.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.134+0000 7fb4dc7ae740 1 -- 172.21.15.90:0/3486801602 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f80d666960 msgr2=0x55f80d66ced0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:41:27.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.134+0000 7fb4dc7ae740 1 --2- 172.21.15.90:0/3486801602 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f80d666960 0x55f80d66ced0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7fb4bc098de0 tx=0x7fb4bc09cc50 comp rx=0 tx=0).stop 2024-04-02T14:41:27.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.134+0000 7fb4dc7ae740 1 -- 172.21.15.90:0/3486801602 shutdown_connections 2024-04-02T14:41:27.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.134+0000 7fb4dc7ae740 1 --2- 172.21.15.90:0/3486801602 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55f80d67fea0 0x55f80d682360 crc :-1 s=READY pgs=186 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:27.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.134+0000 7fb4dc7ae740 1 --2- 172.21.15.90:0/3486801602 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f80d66fe60 0x55f80d66d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:27.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.134+0000 7fb4dc7ae740 1 --2- 172.21.15.90:0/3486801602 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f80d667280 0x55f80d66d410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:27.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.134+0000 7fb4dc7ae740 1 --2- 172.21.15.90:0/3486801602 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f80d666960 0x55f80d66ced0 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:27.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.135+0000 7fb4dc7ae740 1 -- 172.21.15.90:0/3486801602 >> 172.21.15.90:0/3486801602 conn(0x55f80d661090 msgr2=0x55f80d660310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:41:27.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.135+0000 7fb4c0c00640 0 client.27461 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:41:27.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.135+0000 7fb4dc7ae740 1 -- 172.21.15.90:0/3486801602 shutdown_connections 2024-04-02T14:41:27.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.135+0000 7fb4dc7ae740 1 -- 172.21.15.90:0/3486801602 wait complete. 2024-04-02T14:41:27.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.136+0000 7fb4dc7ae740 20 client.27461 trim_cache size 0 max 16384 2024-04-02T14:41:27.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.173+0000 7fb4dc7ae740 1 Processor -- start 2024-04-02T14:41:27.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.173+0000 7fb4dc7ae740 1 -- start start 2024-04-02T14:41:27.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.173+0000 7fb4dc7ae740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f80d667280 0x55f80d66f600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:27.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.174+0000 7fb4dc7ae740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f80d66fe60 0x55f80d67bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:27.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.174+0000 7fb4dc7ae740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f80d660190 0x55f80d687c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:27.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.174+0000 7fb4dc7ae740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fb4d41b81d0 con 0x55f80d660190 2024-04-02T14:41:27.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.174+0000 7fb4dc7ae740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fb4d41b9d10 con 0x55f80d66fe60 2024-04-02T14:41:27.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.174+0000 7fb4dc7ae740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fb4d41b88a0 con 0x55f80d667280 2024-04-02T14:41:27.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.174+0000 7fb4d2a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f80d667280 0x55f80d66f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:27.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.174+0000 7fb4d2000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f80d66fe60 0x55f80d67bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:27.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.174+0000 7fb4d2a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f80d667280 0x55f80d66f600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:42754/0 (socket says 172.21.15.90:42754) 2024-04-02T14:41:27.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.174+0000 7fb4d2a00640 1 -- 172.21.15.90:0/3907739188 learned_addr learned my addr 172.21.15.90:0/3907739188 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:41:27.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.175+0000 7fb4d3400640 1 --2- 172.21.15.90:0/3907739188 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f80d660190 0x55f80d687c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:27.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.175+0000 7fb4d2000640 1 -- 172.21.15.90:0/3907739188 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f80d667280 msgr2=0x55f80d66f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:41:27.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.175+0000 7fb4d2000640 1 --2- 172.21.15.90:0/3907739188 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f80d667280 0x55f80d66f600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:27.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.175+0000 7fb4d2000640 1 -- 172.21.15.90:0/3907739188 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f80d660190 msgr2=0x55f80d687c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:41:27.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.175+0000 7fb4d2000640 1 --2- 172.21.15.90:0/3907739188 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f80d660190 0x55f80d687c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:27.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.175+0000 7fb4d2000640 1 -- 172.21.15.90:0/3907739188 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f80d666100 con 0x55f80d66fe60 2024-04-02T14:41:27.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.175+0000 7fb4d3400640 1 --2- 172.21.15.90:0/3907739188 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f80d660190 0x55f80d687c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:41:27.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.176+0000 7fb4d2000640 1 --2- 172.21.15.90:0/3907739188 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f80d66fe60 0x55f80d67bab0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x55f80d682bd0 tx=0x7fb4c4015000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=912cb19755ce3cc0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:41:27.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.176+0000 7fb4c3400640 1 -- 172.21.15.90:0/3907739188 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4c4028530 con 0x55f80d66fe60 2024-04-02T14:41:27.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.176+0000 7fb4c3400640 1 -- 172.21.15.90:0/3907739188 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb4c40286d0 con 0x55f80d66fe60 2024-04-02T14:41:27.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.176+0000 7fb4c3400640 1 -- 172.21.15.90:0/3907739188 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4c4028530 con 0x55f80d66fe60 2024-04-02T14:41:27.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.177+0000 7fb4dc7ae740 1 -- 172.21.15.90:0/3907739188 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f80d66fe60 msgr2=0x55f80d67bab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:41:27.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.177+0000 7fb4dc7ae740 1 --2- 172.21.15.90:0/3907739188 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f80d66fe60 0x55f80d67bab0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x55f80d682bd0 tx=0x7fb4c4015000 comp rx=0 tx=0).stop 2024-04-02T14:41:27.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.178+0000 7fb4dc7ae740 1 -- 172.21.15.90:0/3907739188 shutdown_connections 2024-04-02T14:41:27.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.178+0000 7fb4dc7ae740 1 --2- 172.21.15.90:0/3907739188 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f80d660190 0x55f80d687c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:27.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.178+0000 7fb4dc7ae740 1 --2- 172.21.15.90:0/3907739188 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f80d66fe60 0x55f80d67bab0 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:27.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.178+0000 7fb4dc7ae740 1 --2- 172.21.15.90:0/3907739188 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f80d667280 0x55f80d66f600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:27.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.178+0000 7fb4dc7ae740 1 -- 172.21.15.90:0/3907739188 >> 172.21.15.90:0/3907739188 conn(0x55f80d661090 msgr2=0x55f80d65fa80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:41:27.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.178+0000 7fb4dc7ae740 1 -- 172.21.15.90:0/3907739188 shutdown_connections 2024-04-02T14:41:27.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.178+0000 7fb4dc7ae740 1 -- 172.21.15.90:0/3907739188 wait complete. 2024-04-02T14:41:27.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.179+0000 7fb4dc7ae740 1 -- start start 2024-04-02T14:41:27.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.179+0000 7fb4dc7ae740 1 Processor -- start 2024-04-02T14:41:27.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.181+0000 7fb4dc7ae740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f80d660190 0x55f80d678230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:27.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.181+0000 7fb4dc7ae740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f80d667280 0x55f80d676fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:27.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.181+0000 7fb4dc7ae740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f80d66fe60 0x55f80d677500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:27.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.181+0000 7fb4dc7ae740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fb4d41b9640 con 0x55f80d66fe60 2024-04-02T14:41:27.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.181+0000 7fb4dc7ae740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fb4d41b8f70 con 0x55f80d660190 2024-04-02T14:41:27.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.181+0000 7fb4dc7ae740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fb4d41b61b0 con 0x55f80d667280 2024-04-02T14:41:27.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.181+0000 7fb4dc7ae740 10 client.0 _ll_register_callbacks cb 0x7fb4bc009640 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:41:27.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.182+0000 7fb4d3400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f80d66fe60 0x55f80d677500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:27.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.182+0000 7fb4d3400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f80d66fe60 0x55f80d677500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:57510/0 (socket says 172.21.15.90:57510) 2024-04-02T14:41:27.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.182+0000 7fb4d3400640 1 -- 172.21.15.90:0/3126511733 learned_addr learned my addr 172.21.15.90:0/3126511733 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:41:27.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.182+0000 7fb4d3400640 1 -- 172.21.15.90:0/3126511733 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f80d667280 msgr2=0x55f80d676fc0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-02T14:41:27.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.182+0000 7fb4d3400640 1 --2- 172.21.15.90:0/3126511733 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f80d667280 0x55f80d676fc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:27.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.182+0000 7fb4d3400640 1 -- 172.21.15.90:0/3126511733 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f80d660190 msgr2=0x55f80d678230 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-02T14:41:27.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.182+0000 7fb4d3400640 1 --2- 172.21.15.90:0/3126511733 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f80d660190 0x55f80d678230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:27.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.182+0000 7fb4d3400640 1 -- 172.21.15.90:0/3126511733 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f80d677ba0 con 0x55f80d66fe60 2024-04-02T14:41:27.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.183+0000 7fb4c0c00640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-02T14:41:27.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.183+0000 7fb4d3400640 1 --2- 172.21.15.90:0/3126511733 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f80d66fe60 0x55f80d677500 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x55f80d662220 tx=0x7fb4cc003bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f93f5dd5796d32a server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:41:27.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.183+0000 7fb4c0c00640 1 -- 172.21.15.90:0/3126511733 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4cc04f070 con 0x55f80d66fe60 2024-04-02T14:41:27.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.183+0000 7fb4c0c00640 1 -- 172.21.15.90:0/3126511733 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7fb4cc0333e0 con 0x55f80d66fe60 2024-04-02T14:41:27.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.184+0000 7fb4dc7ae740 1 -- 172.21.15.90:0/3126511733 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb4c40286d0 con 0x55f80d66fe60 2024-04-02T14:41:27.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.184+0000 7fb4c0c00640 1 -- 172.21.15.90:0/3126511733 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fb4cc042040 con 0x55f80d66fe60 2024-04-02T14:41:27.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.184+0000 7fb4dc7ae740 10 client.16776 fetch_fsmap learned FSMap version 27 2024-04-02T14:41:27.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.184+0000 7fb4dc7ae740 1 -- 172.21.15.90:0/3126511733 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55f80d6efb10 con 0x55f80d66fe60 2024-04-02T14:41:27.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.185+0000 7fb4c0c00640 1 -- 172.21.15.90:0/3126511733 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb4c40286d0 con 0x55f80d66fe60 2024-04-02T14:41:27.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.185+0000 7fb4c0c00640 1 -- 172.21.15.90:0/3126511733 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fb4cc004240 con 0x55f80d66fe60 2024-04-02T14:41:27.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.185+0000 7fb4dc7ae740 10 client.16776 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:41:27.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.185+0000 7fb4dc7ae740 10 client.16776 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:41:27.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.185+0000 7fb4dc7ae740 20 client.16776 populate_metadata read hostname 'smithi090' 2024-04-02T14:41:27.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.185+0000 7fb4dc7ae740 1 --2- 172.21.15.90:0/3126511733 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55f80d6efea0 0x55f80d6f2300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:27.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.185+0000 7fb4dc7ae740 4 client.16776 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-02T14:41:27.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.185+0000 7fb4dc7ae740 1 -- 172.21.15.90:0/3126511733 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55f80d5d7cb0 con 0x55f80d6efea0 2024-04-02T14:41:27.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.190+0000 7fb4d2a00640 1 --2- 172.21.15.90:0/3126511733 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55f80d6efea0 0x55f80d6f2300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:27.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.191+0000 7fb4d2a00640 1 --2- 172.21.15.90:0/3126511733 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55f80d6efea0 0x55f80d6f2300 crc :-1 s=READY pgs=187 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=aa806733fdf02d1d server_cookie=ba32ec9ee1d08f25 in_seq=0 out_seq=0 2024-04-02T14:41:27.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.191+0000 7fb4c0c00640 10 client.16776 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:41:27.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.194+0000 7fb4c0c00640 1 -- 172.21.15.90:0/3126511733 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55f80d5d7cb0 con 0x55f80d6efea0 2024-04-02T14:41:27.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.194+0000 7fb4c0c00640 10 client.16776 handle_command_reply: tid=0 2024-04-02T14:41:27.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.194+0000 7fb4dc7ae740 1 client.16776 shutdown 2024-04-02T14:41:27.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.195+0000 7fb4dc7ae740 1 -- 172.21.15.90:0/3126511733 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f80d66fe60 msgr2=0x55f80d677500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:41:27.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.195+0000 7fb4dc7ae740 1 --2- 172.21.15.90:0/3126511733 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f80d66fe60 0x55f80d677500 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x55f80d662220 tx=0x7fb4cc003bd0 comp rx=0 tx=0).stop 2024-04-02T14:41:27.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.195+0000 7fb4dc7ae740 1 -- 172.21.15.90:0/3126511733 shutdown_connections 2024-04-02T14:41:27.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.195+0000 7fb4dc7ae740 1 --2- 172.21.15.90:0/3126511733 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55f80d6efea0 0x55f80d6f2300 crc :-1 s=READY pgs=187 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:27.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.195+0000 7fb4dc7ae740 1 --2- 172.21.15.90:0/3126511733 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f80d66fe60 0x55f80d677500 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:27.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.195+0000 7fb4dc7ae740 1 --2- 172.21.15.90:0/3126511733 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f80d667280 0x55f80d676fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:27.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.195+0000 7fb4dc7ae740 1 --2- 172.21.15.90:0/3126511733 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f80d660190 0x55f80d678230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:27.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.195+0000 7fb4dc7ae740 1 -- 172.21.15.90:0/3126511733 >> 172.21.15.90:0/3126511733 conn(0x55f80d661090 msgr2=0x55f80d6624c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:41:27.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.195+0000 7fb4c0c00640 0 client.16776 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:41:27.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.195+0000 7fb4dc7ae740 1 -- 172.21.15.90:0/3126511733 shutdown_connections 2024-04-02T14:41:27.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.196+0000 7fb4dc7ae740 1 -- 172.21.15.90:0/3126511733 wait complete. 2024-04-02T14:41:27.196 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-02T14:41:27.196 INFO:teuthology.orchestra.run.smithi090.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-02T14:41:27.196 INFO:teuthology.orchestra.run.smithi090.stdout: "scrubs": { 2024-04-02T14:41:27.196 INFO:teuthology.orchestra.run.smithi090.stdout: "ed86a8cb-4ca9-4377-9358-bfa61a387de0": { 2024-04-02T14:41:27.196 INFO:teuthology.orchestra.run.smithi090.stdout: "path": "/", 2024-04-02T14:41:27.196 INFO:teuthology.orchestra.run.smithi090.stdout: "tag": "ed86a8cb-4ca9-4377-9358-bfa61a387de0", 2024-04-02T14:41:27.196 INFO:teuthology.orchestra.run.smithi090.stdout: "options": "recursive,force" 2024-04-02T14:41:27.196 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:41:27.196 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:41:27.196 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-02T14:41:27.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.196+0000 7fb4dc7ae740 20 client.16776 trim_cache size 0 max 16384 2024-04-02T14:41:27.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.198+0000 7fb4d9a00640 1 -- 172.21.15.90:0/2105663690 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fb4b80a3e60 msgr2=0x7fb4b80a6320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:41:27.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.198+0000 7fb4d9a00640 1 --2- 172.21.15.90:0/2105663690 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fb4b80a3e60 0x7fb4b80a6320 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fb4c40058c0 tx=0x7fb4c4005850 comp rx=0 tx=0).stop 2024-04-02T14:41:27.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.198+0000 7fb4d9a00640 1 -- 172.21.15.90:0/2105663690 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb4d4103ae0 msgr2=0x7fb4d4073370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:41:27.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.198+0000 7fb4d9a00640 1 --2- 172.21.15.90:0/2105663690 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb4d4103ae0 0x7fb4d4073370 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fb4bc00e9c0 tx=0x7fb4bc00ee80 comp rx=0 tx=0).stop 2024-04-02T14:41:27.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.201+0000 7fb4d9a00640 1 -- 172.21.15.90:0/2105663690 shutdown_connections 2024-04-02T14:41:27.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.201+0000 7fb4d9a00640 1 --2- 172.21.15.90:0/2105663690 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fb4b80a3e60 0x7fb4b80a6320 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:27.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.201+0000 7fb4d9a00640 1 --2- 172.21.15.90:0/2105663690 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb4d4109b80 0x7fb4d406c930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:27.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.201+0000 7fb4d9a00640 1 --2- 172.21.15.90:0/2105663690 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb4d41044a0 0x7fb4d406c3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:27.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.201+0000 7fb4d9a00640 1 --2- 172.21.15.90:0/2105663690 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb4d4103ae0 0x7fb4d4073370 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:27.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.201+0000 7fb4d9a00640 1 -- 172.21.15.90:0/2105663690 >> 172.21.15.90:0/2105663690 conn(0x7fb4d40ff590 msgr2=0x7fb4d40ffc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:41:27.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.201+0000 7fb4d9a00640 1 -- 172.21.15.90:0/2105663690 shutdown_connections 2024-04-02T14:41:27.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:27.201+0000 7fb4d9a00640 1 -- 172.21.15.90:0/2105663690 wait complete. 2024-04-02T14:41:27.239 INFO:tasks.cephfs.filesystem:scrub status for tag:ed86a8cb-4ca9-4377-9358-bfa61a387de0 - {'path': '/', 'tag': 'ed86a8cb-4ca9-4377-9358-bfa61a387de0', 'options': 'recursive,force'} 2024-04-02T14:41:28.108 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:41:27 smithi092 ceph-mon[36497]: pgmap v200: 97 pgs: 97 active+clean; 4.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 5.9 MiB/s wr, 582 op/s 2024-04-02T14:41:28.109 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:41:27 smithi092 ceph-mon[36497]: scrub queued for path: / 2024-04-02T14:41:28.109 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:41:27 smithi092 ceph-mon[36497]: scrub summary: idle+waiting paths [/] 2024-04-02T14:41:28.109 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:41:27 smithi092 ceph-mon[36497]: scrub summary: active paths [/] 2024-04-02T14:41:28.184 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:41:27 smithi177 ceph-mon[36431]: pgmap v200: 97 pgs: 97 active+clean; 4.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 5.9 MiB/s wr, 582 op/s 2024-04-02T14:41:28.184 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:41:27 smithi177 ceph-mon[36431]: scrub queued for path: / 2024-04-02T14:41:28.185 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:41:27 smithi177 ceph-mon[36431]: scrub summary: idle+waiting paths [/] 2024-04-02T14:41:28.185 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:41:27 smithi177 ceph-mon[36431]: scrub summary: active paths [/] 2024-04-02T14:41:28.186 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:41:27 smithi090 ceph-mon[31674]: pgmap v200: 97 pgs: 97 active+clean; 4.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 5.9 MiB/s wr, 582 op/s 2024-04-02T14:41:28.186 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:41:27 smithi090 ceph-mon[31674]: scrub queued for path: / 2024-04-02T14:41:28.186 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:41:27 smithi090 ceph-mon[31674]: scrub summary: idle+waiting paths [/] 2024-04-02T14:41:28.187 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:41:27 smithi090 ceph-mon[31674]: scrub summary: active paths [/] 2024-04-02T14:41:29.108 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:41:28 smithi092 ceph-mon[36497]: scrub summary: idle+waiting paths [/] 2024-04-02T14:41:29.184 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:41:28 smithi177 ceph-mon[36431]: scrub summary: idle+waiting paths [/] 2024-04-02T14:41:29.186 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:41:28 smithi090 ceph-mon[31674]: scrub summary: idle+waiting paths [/] 2024-04-02T14:41:30.108 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:41:29 smithi092 ceph-mon[36497]: pgmap v201: 97 pgs: 97 active+clean; 4.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 8.4 MiB/s wr, 880 op/s 2024-04-02T14:41:30.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:41:29 smithi177 ceph-mon[36431]: pgmap v201: 97 pgs: 97 active+clean; 4.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 8.4 MiB/s wr, 880 op/s 2024-04-02T14:41:30.186 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:41:29 smithi090 ceph-mon[31674]: pgmap v201: 97 pgs: 97 active+clean; 4.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 8.4 MiB/s wr, 880 op/s 2024-04-02T14:41:31.829 INFO:teuthology.orchestra.run.smithi090.stderr:progress: 360.0 s, 2759.2 tps, lat 36.057 ms stddev 43.577 2024-04-02T14:41:32.108 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:41:31 smithi092 ceph-mon[36497]: pgmap v202: 97 pgs: 97 active+clean; 4.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 150 KiB/s rd, 6.9 MiB/s wr, 797 op/s 2024-04-02T14:41:32.184 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:41:31 smithi177 ceph-mon[36431]: pgmap v202: 97 pgs: 97 active+clean; 4.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 150 KiB/s rd, 6.9 MiB/s wr, 797 op/s 2024-04-02T14:41:32.186 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:41:31 smithi090 ceph-mon[31674]: pgmap v202: 97 pgs: 97 active+clean; 4.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 150 KiB/s rd, 6.9 MiB/s wr, 797 op/s 2024-04-02T14:41:33.009 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:41:32 smithi092 ceph-mon[36497]: scrub summary: idle 2024-04-02T14:41:33.184 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:41:32 smithi177 ceph-mon[36431]: scrub summary: idle 2024-04-02T14:41:33.186 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:41:32 smithi090 ceph-mon[31674]: scrub summary: idle 2024-04-02T14:41:34.108 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:41:33 smithi092 ceph-mon[36497]: pgmap v203: 97 pgs: 97 active+clean; 4.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 6.2 MiB/s wr, 707 op/s 2024-04-02T14:41:34.184 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:41:33 smithi177 ceph-mon[36431]: pgmap v203: 97 pgs: 97 active+clean; 4.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 6.2 MiB/s wr, 707 op/s 2024-04-02T14:41:34.186 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:41:33 smithi090 ceph-mon[31674]: pgmap v203: 97 pgs: 97 active+clean; 4.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 6.2 MiB/s wr, 707 op/s 2024-04-02T14:41:36.108 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:41:35 smithi092 ceph-mon[36497]: pgmap v204: 97 pgs: 97 active+clean; 4.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 149 KiB/s rd, 9.2 MiB/s wr, 1.01k op/s 2024-04-02T14:41:36.184 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:41:35 smithi177 ceph-mon[36431]: pgmap v204: 97 pgs: 97 active+clean; 4.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 149 KiB/s rd, 9.2 MiB/s wr, 1.01k op/s 2024-04-02T14:41:36.186 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:41:35 smithi090 ceph-mon[31674]: pgmap v204: 97 pgs: 97 active+clean; 4.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 149 KiB/s rd, 9.2 MiB/s wr, 1.01k op/s 2024-04-02T14:41:36.829 INFO:teuthology.orchestra.run.smithi090.stderr:progress: 365.0 s, 2509.8 tps, lat 40.042 ms stddev 51.957 2024-04-02T14:41:38.022 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.022+0000 7f4dab400640 1 -- 172.21.15.90:0/3877189945 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4da4108340 msgr2=0x7f4da4108720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:41:38.022 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.022+0000 7f4dab400640 1 --2- 172.21.15.90:0/3877189945 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4da4108340 0x7f4da4108720 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f4d94009900 tx=0x7f4d9402bc40 comp rx=0 tx=0).stop 2024-04-02T14:41:38.022 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.023+0000 7f4dab400640 1 -- 172.21.15.90:0/3877189945 shutdown_connections 2024-04-02T14:41:38.022 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.023+0000 7f4dab400640 1 --2- 172.21.15.90:0/3877189945 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4da410d410 0x7f4da4114fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:38.022 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.023+0000 7f4dab400640 1 --2- 172.21.15.90:0/3877189945 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4da4108cf0 0x7f4da410cd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:38.022 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.023+0000 7f4dab400640 1 --2- 172.21.15.90:0/3877189945 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4da4108340 0x7f4da4108720 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:38.022 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.023+0000 7f4dab400640 1 -- 172.21.15.90:0/3877189945 >> 172.21.15.90:0/3877189945 conn(0x7f4da4076070 msgr2=0x7f4da4078490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:41:38.023 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.023+0000 7f4dab400640 1 -- 172.21.15.90:0/3877189945 shutdown_connections 2024-04-02T14:41:38.023 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.023+0000 7f4dab400640 1 -- 172.21.15.90:0/3877189945 wait complete. 2024-04-02T14:41:38.024 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.024+0000 7f4dab400640 1 Processor -- start 2024-04-02T14:41:38.024 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.024+0000 7f4dab400640 1 -- start start 2024-04-02T14:41:38.024 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.024+0000 7f4dab400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4da4108340 0x7f4da41a4610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:38.024 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.024+0000 7f4dab400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4da410d410 0x7f4da41a4b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:38.024 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.025+0000 7f4dab400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4da419e7d0 0x7f4da419ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:38.024 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.025+0000 7f4dab400640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f4da4118570 con 0x7f4da4108340 2024-04-02T14:41:38.024 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.025+0000 7f4dab400640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f4da41183f0 con 0x7f4da410d410 2024-04-02T14:41:38.024 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.025+0000 7f4dab400640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f4da41186f0 con 0x7f4da419e7d0 2024-04-02T14:41:38.024 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.025+0000 7f4da3e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4da410d410 0x7f4da41a4b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:38.024 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.025+0000 7f4da8a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4da4108340 0x7f4da41a4610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:38.024 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.025+0000 7f4da3e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4da410d410 0x7f4da41a4b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:51996/0 (socket says 172.21.15.90:51996) 2024-04-02T14:41:38.024 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.025+0000 7f4da3e00640 1 -- 172.21.15.90:0/1150630967 learned_addr learned my addr 172.21.15.90:0/1150630967 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:41:38.024 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.025+0000 7f4da9400640 1 --2- 172.21.15.90:0/1150630967 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4da419e7d0 0x7f4da419ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:38.025 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.025+0000 7f4da3e00640 1 -- 172.21.15.90:0/1150630967 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4da419e7d0 msgr2=0x7f4da419ec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:41:38.026 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.025+0000 7f4da3e00640 1 --2- 172.21.15.90:0/1150630967 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4da419e7d0 0x7f4da419ec80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:38.026 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.025+0000 7f4da3e00640 1 -- 172.21.15.90:0/1150630967 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4da4108340 msgr2=0x7f4da41a4610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:41:38.026 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.025+0000 7f4da3e00640 1 --2- 172.21.15.90:0/1150630967 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4da4108340 0x7f4da41a4610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:38.026 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.025+0000 7f4da3e00640 1 -- 172.21.15.90:0/1150630967 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4da419f360 con 0x7f4da410d410 2024-04-02T14:41:38.026 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.025+0000 7f4da9400640 1 --2- 172.21.15.90:0/1150630967 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4da419e7d0 0x7f4da419ec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:41:38.026 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.025+0000 7f4da8a00640 1 --2- 172.21.15.90:0/1150630967 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4da4108340 0x7f4da41a4610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:41:38.026 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.026+0000 7f4da3e00640 1 --2- 172.21.15.90:0/1150630967 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4da410d410 0x7f4da41a4b50 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f4d9800e9c0 tx=0x7f4d9800ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:41:38.026 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.026+0000 7f4da1600640 1 -- 172.21.15.90:0/1150630967 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d9800cd80 con 0x7f4da410d410 2024-04-02T14:41:38.026 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.026+0000 7f4da1600640 1 -- 172.21.15.90:0/1150630967 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4d98005500 con 0x7f4da410d410 2024-04-02T14:41:38.026 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.026+0000 7f4da1600640 1 -- 172.21.15.90:0/1150630967 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d98010690 con 0x7f4da410d410 2024-04-02T14:41:38.026 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.026+0000 7f4dab400640 1 -- 172.21.15.90:0/1150630967 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4da419f650 con 0x7f4da410d410 2024-04-02T14:41:38.026 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.026+0000 7f4dab400640 1 -- 172.21.15.90:0/1150630967 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4da41ab790 con 0x7f4da410d410 2024-04-02T14:41:38.029 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.030+0000 7f4da1600640 1 -- 172.21.15.90:0/1150630967 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f4d980046c0 con 0x7f4da410d410 2024-04-02T14:41:38.029 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.030+0000 7f4dae21e740 1 Processor -- start 2024-04-02T14:41:38.029 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.031+0000 7f4dae21e740 1 -- start start 2024-04-02T14:41:38.032 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.031+0000 7f4dae21e740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56094e104eb0 0x56094e105290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:38.032 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.031+0000 7f4dae21e740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56094e1057d0 0x56094e10de70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:38.032 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.031+0000 7f4dae21e740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56094e10e3b0 0x56094e1107a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:38.032 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.031+0000 7f4dae21e740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x56094e069d70 con 0x56094e1057d0 2024-04-02T14:41:38.032 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.031+0000 7f4dae21e740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x56094e0122a0 con 0x56094e10e3b0 2024-04-02T14:41:38.033 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.031+0000 7f4dae21e740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x56094e065d20 con 0x56094e104eb0 2024-04-02T14:41:38.033 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.031+0000 7f4da9400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56094e104eb0 0x56094e105290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:38.033 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.031+0000 7f4da3e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56094e10e3b0 0x56094e1107a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:38.033 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.031+0000 7f4da9400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56094e104eb0 0x56094e105290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:49758/0 (socket says 172.21.15.90:49758) 2024-04-02T14:41:38.033 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.031+0000 7f4da9400640 1 -- 172.21.15.90:0/4000639185 learned_addr learned my addr 172.21.15.90:0/4000639185 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:41:38.033 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.032+0000 7f4da8a00640 1 --2- 172.21.15.90:0/4000639185 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56094e1057d0 0x56094e10de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:38.033 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.032+0000 7f4da1600640 1 --2- 172.21.15.90:0/1150630967 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f4d8c0a3e90 0x7f4d8c0a6350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:38.033 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.032+0000 7f4da1600640 1 -- 172.21.15.90:0/1150630967 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f4d980d0bf0 con 0x7f4da410d410 2024-04-02T14:41:38.033 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.032+0000 7f4da3e00640 1 -- 172.21.15.90:0/4000639185 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56094e104eb0 msgr2=0x56094e105290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:41:38.033 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.032+0000 7f4da3e00640 1 --2- 172.21.15.90:0/4000639185 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56094e104eb0 0x56094e105290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:38.033 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.032+0000 7f4da3e00640 1 -- 172.21.15.90:0/4000639185 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56094e1057d0 msgr2=0x56094e10de70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:41:38.033 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.032+0000 7f4da3e00640 1 --2- 172.21.15.90:0/4000639185 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56094e1057d0 0x56094e10de70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:38.033 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.032+0000 7f4da3e00640 1 -- 172.21.15.90:0/4000639185 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56094e026840 con 0x56094e10e3b0 2024-04-02T14:41:38.033 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.032+0000 7f4da9400640 1 --2- 172.21.15.90:0/1150630967 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f4d8c0a3e90 0x7f4d8c0a6350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:38.034 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.032+0000 7f4da3e00640 1 --2- 172.21.15.90:0/4000639185 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56094e10e3b0 0x56094e1107a0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f4d980ce2f0 tx=0x7f4d98004b30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1eee4c04e1ca322 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:41:38.034 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.033+0000 7f4d91c00640 1 -- 172.21.15.90:0/4000639185 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d9809c910 con 0x56094e10e3b0 2024-04-02T14:41:38.034 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.033+0000 7f4d91c00640 1 -- 172.21.15.90:0/4000639185 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4d9809d240 con 0x56094e10e3b0 2024-04-02T14:41:38.034 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.033+0000 7f4d91c00640 1 -- 172.21.15.90:0/4000639185 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d9809c4a0 con 0x56094e10e3b0 2024-04-02T14:41:38.035 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.034+0000 7f4dae21e740 1 -- 172.21.15.90:0/4000639185 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56094e10e3b0 msgr2=0x56094e1107a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:41:38.035 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.034+0000 7f4dae21e740 1 --2- 172.21.15.90:0/4000639185 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56094e10e3b0 0x56094e1107a0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f4d980ce2f0 tx=0x7f4d98004b30 comp rx=0 tx=0).stop 2024-04-02T14:41:38.035 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.034+0000 7f4da9400640 1 --2- 172.21.15.90:0/1150630967 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f4d8c0a3e90 0x7f4d8c0a6350 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f4d9c00a660 tx=0x7f4d9c00e040 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:41:38.035 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.034+0000 7f4dae21e740 1 -- 172.21.15.90:0/4000639185 shutdown_connections 2024-04-02T14:41:38.036 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.034+0000 7f4dae21e740 1 --2- 172.21.15.90:0/4000639185 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56094e10e3b0 0x56094e1107a0 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:38.036 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.034+0000 7f4dae21e740 1 --2- 172.21.15.90:0/4000639185 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56094e1057d0 0x56094e10de70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:38.036 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.034+0000 7f4dae21e740 1 --2- 172.21.15.90:0/4000639185 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56094e104eb0 0x56094e105290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:38.036 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.034+0000 7f4dae21e740 1 -- 172.21.15.90:0/4000639185 >> 172.21.15.90:0/4000639185 conn(0x56094e0ff5e0 msgr2=0x56094e0ff9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:41:38.036 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.034+0000 7f4dae21e740 1 -- 172.21.15.90:0/4000639185 shutdown_connections 2024-04-02T14:41:38.036 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.035+0000 7f4dae21e740 1 -- 172.21.15.90:0/4000639185 wait complete. 2024-04-02T14:41:38.036 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.035+0000 7f4dae21e740 1 -- start start 2024-04-02T14:41:38.036 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.035+0000 7f4dae21e740 1 Processor -- start 2024-04-02T14:41:38.036 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.036+0000 7f4dae21e740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56094e104eb0 0x56094e10b470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:38.036 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.036+0000 7f4dae21e740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56094e1057d0 0x56094e10b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:38.036 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.036+0000 7f4dae21e740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56094e10e3b0 0x56094e10bef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:38.036 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.036+0000 7f4dae21e740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x56094e1110e0 con 0x56094e10e3b0 2024-04-02T14:41:38.036 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.036+0000 7f4dae21e740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x56094e025660 con 0x56094e104eb0 2024-04-02T14:41:38.036 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.036+0000 7f4dae21e740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x56094e111260 con 0x56094e1057d0 2024-04-02T14:41:38.037 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.037+0000 7f4dae21e740 10 client.0 _ll_register_callbacks cb 0x56094e025050 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:41:38.037 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.037+0000 7f4da8a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56094e104eb0 0x56094e10b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:38.037 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.037+0000 7f4da9400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56094e10e3b0 0x56094e10bef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:38.037 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.037+0000 7f4da9400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56094e10e3b0 0x56094e10bef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41614/0 (socket says 172.21.15.90:41614) 2024-04-02T14:41:38.038 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.037+0000 7f4da9400640 1 -- 172.21.15.90:0/2364775017 learned_addr learned my addr 172.21.15.90:0/2364775017 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:41:38.038 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.038+0000 7f4da3e00640 1 --2- 172.21.15.90:0/2364775017 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56094e1057d0 0x56094e10b9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:38.038 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.038+0000 7f4da9400640 1 -- 172.21.15.90:0/2364775017 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56094e1057d0 msgr2=0x56094e10b9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:41:38.038 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.038+0000 7f4da9400640 1 --2- 172.21.15.90:0/2364775017 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56094e1057d0 0x56094e10b9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:38.038 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.038+0000 7f4da9400640 1 -- 172.21.15.90:0/2364775017 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56094e104eb0 msgr2=0x56094e10b470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:41:38.038 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.038+0000 7f4da9400640 1 --2- 172.21.15.90:0/2364775017 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56094e104eb0 0x56094e10b470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:38.038 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.038+0000 7f4da9400640 1 -- 172.21.15.90:0/2364775017 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56094e034e80 con 0x56094e10e3b0 2024-04-02T14:41:38.039 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.039+0000 7f4da9400640 1 --2- 172.21.15.90:0/2364775017 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56094e10e3b0 0x56094e10bef0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f4d9c007b80 tx=0x7f4d9c009960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1370b405cab37154 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:41:38.039 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.039+0000 7f4d86000640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-02T14:41:38.039 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.040+0000 7f4d86000640 1 -- 172.21.15.90:0/2364775017 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d9c028060 con 0x56094e10e3b0 2024-04-02T14:41:38.039 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.040+0000 7f4dae21e740 1 -- 172.21.15.90:0/2364775017 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4d9809d240 con 0x56094e10e3b0 2024-04-02T14:41:38.039 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.040+0000 7f4d86000640 1 -- 172.21.15.90:0/2364775017 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f4d9c018e00 con 0x56094e10e3b0 2024-04-02T14:41:38.040 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.041+0000 7f4d86000640 1 -- 172.21.15.90:0/2364775017 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f4d9c023440 con 0x56094e10e3b0 2024-04-02T14:41:38.040 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.041+0000 7f4dae21e740 10 client.16785 fetch_fsmap learned FSMap version 27 2024-04-02T14:41:38.040 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.041+0000 7f4dae21e740 1 -- 172.21.15.90:0/2364775017 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x56094dd7e210 con 0x56094e10e3b0 2024-04-02T14:41:38.041 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.041+0000 7f4d86000640 1 -- 172.21.15.90:0/2364775017 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4d9809d240 con 0x56094e10e3b0 2024-04-02T14:41:38.041 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.042+0000 7f4d86000640 1 -- 172.21.15.90:0/2364775017 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f4d9c0167c0 con 0x56094e10e3b0 2024-04-02T14:41:38.041 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.042+0000 7f4dae21e740 10 client.16785 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:41:38.041 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.042+0000 7f4dae21e740 10 client.16785 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-02T14:41:38.041 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.042+0000 7f4dae21e740 20 client.16785 populate_metadata read hostname 'smithi090' 2024-04-02T14:41:38.041 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.042+0000 7f4dae21e740 1 --2- 172.21.15.90:0/2364775017 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x56094e11e7b0 0x56094e120ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:38.041 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.042+0000 7f4dae21e740 4 client.16785 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:41:38.041 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.042+0000 7f4dae21e740 1 -- 172.21.15.90:0/2364775017 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56094e019cd0 con 0x56094e11e7b0 2024-04-02T14:41:38.042 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.043+0000 7f4da8a00640 1 --2- 172.21.15.90:0/2364775017 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x56094e11e7b0 0x56094e120ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:38.042 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.044+0000 7f4da8a00640 1 --2- 172.21.15.90:0/2364775017 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x56094e11e7b0 0x56094e120ba0 crc :-1 s=READY pgs=188 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b7b8ede53d33fe7c server_cookie=e15e7bdd7eb5dfd6 in_seq=0 out_seq=0 2024-04-02T14:41:38.042 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.044+0000 7f4d86000640 10 client.16785 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:41:38.044 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.044+0000 7f4d86000640 1 -- 172.21.15.90:0/2364775017 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56094e019cd0 con 0x56094e11e7b0 2024-04-02T14:41:38.044 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.044+0000 7f4d86000640 10 client.16785 handle_command_reply: tid=0 2024-04-02T14:41:38.044 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.044+0000 7f4dae21e740 1 client.16785 shutdown 2024-04-02T14:41:38.044 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.045+0000 7f4dae21e740 1 -- 172.21.15.90:0/2364775017 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56094e10e3b0 msgr2=0x56094e10bef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:41:38.044 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.045+0000 7f4dae21e740 1 --2- 172.21.15.90:0/2364775017 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56094e10e3b0 0x56094e10bef0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f4d9c007b80 tx=0x7f4d9c009960 comp rx=0 tx=0).stop 2024-04-02T14:41:38.044 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.045+0000 7f4dae21e740 1 -- 172.21.15.90:0/2364775017 shutdown_connections 2024-04-02T14:41:38.044 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.045+0000 7f4dae21e740 1 --2- 172.21.15.90:0/2364775017 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x56094e11e7b0 0x56094e120ba0 crc :-1 s=READY pgs=188 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:38.044 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.045+0000 7f4dae21e740 1 --2- 172.21.15.90:0/2364775017 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56094e10e3b0 0x56094e10bef0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:38.044 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.045+0000 7f4dae21e740 1 --2- 172.21.15.90:0/2364775017 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56094e1057d0 0x56094e10b9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:38.044 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.045+0000 7f4dae21e740 1 --2- 172.21.15.90:0/2364775017 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56094e104eb0 0x56094e10b470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:38.044 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.045+0000 7f4dae21e740 1 -- 172.21.15.90:0/2364775017 >> 172.21.15.90:0/2364775017 conn(0x56094e0ff5e0 msgr2=0x56094e101330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:41:38.045 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.045+0000 7f4d86000640 0 client.16785 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:41:38.045 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.045+0000 7f4dae21e740 1 -- 172.21.15.90:0/2364775017 shutdown_connections 2024-04-02T14:41:38.045 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.045+0000 7f4dae21e740 1 -- 172.21.15.90:0/2364775017 wait complete. 2024-04-02T14:41:38.045 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.045+0000 7f4dae21e740 20 client.16785 trim_cache size 0 max 16384 2024-04-02T14:41:38.075 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.075+0000 7f4dae21e740 1 Processor -- start 2024-04-02T14:41:38.075 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.075+0000 7f4dae21e740 1 -- start start 2024-04-02T14:41:38.075 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.075+0000 7f4dae21e740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56094e104eb0 0x56094e10dba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:38.075 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.075+0000 7f4dae21e740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56094e1057d0 0x56094e119fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:38.075 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.075+0000 7f4dae21e740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56094e10e3b0 0x56094e101120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:38.075 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.075+0000 7f4dae21e740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f4da41b3db0 con 0x56094e10e3b0 2024-04-02T14:41:38.076 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.075+0000 7f4dae21e740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f4da41b58f0 con 0x56094e104eb0 2024-04-02T14:41:38.076 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.075+0000 7f4dae21e740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f4da41b4480 con 0x56094e1057d0 2024-04-02T14:41:38.076 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.076+0000 7f4da3e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56094e1057d0 0x56094e119fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:38.076 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.076+0000 7f4da8a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56094e104eb0 0x56094e10dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:38.076 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.076+0000 7f4da3e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56094e1057d0 0x56094e119fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:49782/0 (socket says 172.21.15.90:49782) 2024-04-02T14:41:38.076 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.076+0000 7f4da3e00640 1 -- 172.21.15.90:0/2325718265 learned_addr learned my addr 172.21.15.90:0/2325718265 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:41:38.076 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.076+0000 7f4da9400640 1 --2- 172.21.15.90:0/2325718265 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56094e10e3b0 0x56094e101120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:38.077 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.077+0000 7f4da3e00640 1 -- 172.21.15.90:0/2325718265 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56094e104eb0 msgr2=0x56094e10dba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:41:38.078 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.077+0000 7f4da3e00640 1 --2- 172.21.15.90:0/2325718265 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56094e104eb0 0x56094e10dba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:38.078 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.077+0000 7f4da3e00640 1 -- 172.21.15.90:0/2325718265 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56094e10e3b0 msgr2=0x56094e101120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:41:38.078 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.077+0000 7f4da3e00640 1 --2- 172.21.15.90:0/2325718265 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56094e10e3b0 0x56094e101120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:38.078 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.077+0000 7f4da3e00640 1 -- 172.21.15.90:0/2325718265 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56094de4a620 con 0x56094e1057d0 2024-04-02T14:41:38.078 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.077+0000 7f4da9400640 1 --2- 172.21.15.90:0/2325718265 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56094e10e3b0 0x56094e101120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:41:38.078 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.077+0000 7f4da3e00640 1 --2- 172.21.15.90:0/2325718265 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56094e1057d0 0x56094e119fa0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f4d9809adf0 tx=0x7f4d980d0e90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dc5dffe7cd232398 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:41:38.078 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.077+0000 7f4d91200640 1 -- 172.21.15.90:0/2325718265 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d9809c680 con 0x56094e1057d0 2024-04-02T14:41:38.078 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.078+0000 7f4d91200640 1 -- 172.21.15.90:0/2325718265 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4d980d44f0 con 0x56094e1057d0 2024-04-02T14:41:38.078 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.078+0000 7f4d91200640 1 -- 172.21.15.90:0/2325718265 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d980991b0 con 0x56094e1057d0 2024-04-02T14:41:38.078 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.078+0000 7f4dae21e740 1 -- 172.21.15.90:0/2325718265 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56094e1057d0 msgr2=0x56094e119fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:41:38.078 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.078+0000 7f4dae21e740 1 --2- 172.21.15.90:0/2325718265 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56094e1057d0 0x56094e119fa0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f4d9809adf0 tx=0x7f4d980d0e90 comp rx=0 tx=0).stop 2024-04-02T14:41:38.078 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.078+0000 7f4dae21e740 1 -- 172.21.15.90:0/2325718265 shutdown_connections 2024-04-02T14:41:38.079 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.078+0000 7f4dae21e740 1 --2- 172.21.15.90:0/2325718265 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56094e10e3b0 0x56094e101120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:38.079 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.078+0000 7f4dae21e740 1 --2- 172.21.15.90:0/2325718265 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56094e1057d0 0x56094e119fa0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:38.079 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.078+0000 7f4dae21e740 1 --2- 172.21.15.90:0/2325718265 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56094e104eb0 0x56094e10dba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:38.079 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.078+0000 7f4dae21e740 1 -- 172.21.15.90:0/2325718265 >> 172.21.15.90:0/2325718265 conn(0x56094e0ff5e0 msgr2=0x56094e100a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:41:38.079 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.078+0000 7f4dae21e740 1 -- 172.21.15.90:0/2325718265 shutdown_connections 2024-04-02T14:41:38.079 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.079+0000 7f4dae21e740 1 -- 172.21.15.90:0/2325718265 wait complete. 2024-04-02T14:41:38.079 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.079+0000 7f4dae21e740 1 -- start start 2024-04-02T14:41:38.079 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.079+0000 7f4dae21e740 1 Processor -- start 2024-04-02T14:41:38.081 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.080+0000 7f4dae21e740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56094e104eb0 0x56094e108440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:38.081 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.080+0000 7f4dae21e740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56094e1057d0 0x56094e108980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:38.081 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.080+0000 7f4dae21e740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56094e10e3b0 0x56094e183ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:38.081 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.080+0000 7f4dae21e740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f4da41b5220 con 0x56094e1057d0 2024-04-02T14:41:38.081 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.080+0000 7f4dae21e740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f4da41b4b50 con 0x56094e10e3b0 2024-04-02T14:41:38.081 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.080+0000 7f4dae21e740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f4da41b1d90 con 0x56094e104eb0 2024-04-02T14:41:38.081 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.080+0000 7f4dae21e740 10 client.0 _ll_register_callbacks cb 0x7f4d9c007830 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:41:38.082 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.080+0000 7f4da3e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56094e1057d0 0x56094e108980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:38.082 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.080+0000 7f4da9400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56094e10e3b0 0x56094e183ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:38.082 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.080+0000 7f4da9400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56094e10e3b0 0x56094e183ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:52038/0 (socket says 172.21.15.90:52038) 2024-04-02T14:41:38.082 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.080+0000 7f4da9400640 1 -- 172.21.15.90:0/3309961469 learned_addr learned my addr 172.21.15.90:0/3309961469 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:41:38.082 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.080+0000 7f4da8a00640 1 --2- 172.21.15.90:0/3309961469 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56094e104eb0 0x56094e108440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:38.082 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.080+0000 7f4da3e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56094e1057d0 0x56094e108980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41624/0 (socket says 172.21.15.90:41624) 2024-04-02T14:41:38.082 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.081+0000 7f4da9400640 1 -- 172.21.15.90:0/3309961469 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56094e104eb0 msgr2=0x56094e108440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:41:38.082 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.081+0000 7f4da9400640 1 --2- 172.21.15.90:0/3309961469 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56094e104eb0 0x56094e108440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:38.082 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.081+0000 7f4da9400640 1 -- 172.21.15.90:0/3309961469 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56094e1057d0 msgr2=0x56094e108980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:41:38.082 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.081+0000 7f4da9400640 1 --2- 172.21.15.90:0/3309961469 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56094e1057d0 0x56094e108980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:38.082 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.081+0000 7f4da9400640 1 -- 172.21.15.90:0/3309961469 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56094e109210 con 0x56094e10e3b0 2024-04-02T14:41:38.082 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.081+0000 7f4da9400640 1 --2- 172.21.15.90:0/3309961469 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56094e10e3b0 0x56094e183ce0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f4d9c009dc0 tx=0x7f4d9c016b10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=34525350c06f4162 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:41:38.082 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.081+0000 7f4d86000640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:41:38.082 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.082+0000 7f4d86000640 1 -- 172.21.15.90:0/3309961469 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d9c0237e0 con 0x56094e10e3b0 2024-04-02T14:41:38.083 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.082+0000 7f4d86000640 1 -- 172.21.15.90:0/3309961469 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f4d9c023980 con 0x56094e10e3b0 2024-04-02T14:41:38.083 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.082+0000 7f4dae21e740 1 -- 172.21.15.90:0/3309961469 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4d980d44f0 con 0x56094e10e3b0 2024-04-02T14:41:38.083 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.082+0000 7f4d86000640 1 -- 172.21.15.90:0/3309961469 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f4d9c023b80 con 0x56094e10e3b0 2024-04-02T14:41:38.084 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.083+0000 7f4dae21e740 10 client.27482 fetch_fsmap learned FSMap version 27 2024-04-02T14:41:38.084 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.083+0000 7f4dae21e740 1 -- 172.21.15.90:0/3309961469 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x56094e185300 con 0x56094e10e3b0 2024-04-02T14:41:38.084 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.083+0000 7f4d86000640 1 -- 172.21.15.90:0/3309961469 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4d980d44f0 con 0x56094e10e3b0 2024-04-02T14:41:38.084 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.083+0000 7f4d86000640 1 -- 172.21.15.90:0/3309961469 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f4d9c035430 con 0x56094e10e3b0 2024-04-02T14:41:38.084 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.084+0000 7f4dae21e740 10 client.27482 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:41:38.084 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.084+0000 7f4dae21e740 10 client.27482 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-02T14:41:38.084 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.084+0000 7f4dae21e740 20 client.27482 populate_metadata read hostname 'smithi090' 2024-04-02T14:41:38.084 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.084+0000 7f4dae21e740 1 --2- 172.21.15.90:0/3309961469 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x56094e185a20 0x56094e187e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:38.084 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.084+0000 7f4dae21e740 4 client.27482 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-02T14:41:38.084 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.084+0000 7f4dae21e740 1 -- 172.21.15.90:0/3309961469 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56094e0772b0 con 0x56094e185a20 2024-04-02T14:41:38.084 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.085+0000 7f4da8a00640 1 --2- 172.21.15.90:0/3309961469 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x56094e185a20 0x56094e187e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:38.084 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.085+0000 7f4d86000640 10 client.27482 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:41:38.085 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.085+0000 7f4da8a00640 1 --2- 172.21.15.90:0/3309961469 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x56094e185a20 0x56094e187e40 crc :-1 s=READY pgs=189 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b85ef5f5e355d94 server_cookie=dde2c9b9b0cbf1db in_seq=0 out_seq=0 2024-04-02T14:41:38.104 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.105+0000 7f4d86000640 1 -- 172.21.15.90:0/3309961469 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56094e0772b0 con 0x56094e185a20 2024-04-02T14:41:38.105 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.105+0000 7f4d86000640 10 client.27482 handle_command_reply: tid=0 2024-04-02T14:41:38.105 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.106+0000 7f4dae21e740 1 client.27482 shutdown 2024-04-02T14:41:38.105 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.106+0000 7f4dae21e740 1 -- 172.21.15.90:0/3309961469 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56094e10e3b0 msgr2=0x56094e183ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:41:38.105 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.106+0000 7f4dae21e740 1 --2- 172.21.15.90:0/3309961469 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56094e10e3b0 0x56094e183ce0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f4d9c009dc0 tx=0x7f4d9c016b10 comp rx=0 tx=0).stop 2024-04-02T14:41:38.106 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.106+0000 7f4dae21e740 1 -- 172.21.15.90:0/3309961469 shutdown_connections 2024-04-02T14:41:38.106 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.106+0000 7f4dae21e740 1 --2- 172.21.15.90:0/3309961469 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x56094e185a20 0x56094e187e40 crc :-1 s=READY pgs=189 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:38.106 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.106+0000 7f4dae21e740 1 --2- 172.21.15.90:0/3309961469 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56094e10e3b0 0x56094e183ce0 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:38.106 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.106+0000 7f4d86000640 0 client.27482 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:41:38.106 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.106+0000 7f4dae21e740 1 --2- 172.21.15.90:0/3309961469 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56094e1057d0 0x56094e108980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:38.106 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.106+0000 7f4dae21e740 1 --2- 172.21.15.90:0/3309961469 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56094e104eb0 0x56094e108440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:38.106 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.106+0000 7f4dae21e740 1 -- 172.21.15.90:0/3309961469 >> 172.21.15.90:0/3309961469 conn(0x56094e0ff5e0 msgr2=0x56094e0fe640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:41:38.107 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.107+0000 7f4dae21e740 1 -- 172.21.15.90:0/3309961469 shutdown_connections 2024-04-02T14:41:38.107 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.107+0000 7f4dae21e740 1 -- 172.21.15.90:0/3309961469 wait complete. 2024-04-02T14:41:38.107 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.108+0000 7f4dae21e740 20 client.27482 trim_cache size 0 max 16384 2024-04-02T14:41:38.108 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:{ 2024-04-02T14:41:38.108 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "message": "", 2024-04-02T14:41:38.108 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "return_code": 0 2024-04-02T14:41:38.108 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:} 2024-04-02T14:41:38.109 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.109+0000 7f4dab400640 1 -- 172.21.15.90:0/1150630967 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f4d8c0a3e90 msgr2=0x7f4d8c0a6350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:41:38.109 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.109+0000 7f4dab400640 1 --2- 172.21.15.90:0/1150630967 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f4d8c0a3e90 0x7f4d8c0a6350 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f4d9c00a660 tx=0x7f4d9c00e040 comp rx=0 tx=0).stop 2024-04-02T14:41:38.109 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.109+0000 7f4dab400640 1 -- 172.21.15.90:0/1150630967 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4da410d410 msgr2=0x7f4da41a4b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:41:38.109 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.109+0000 7f4dab400640 1 --2- 172.21.15.90:0/1150630967 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4da410d410 0x7f4da41a4b50 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f4d9800e9c0 tx=0x7f4d9800ee80 comp rx=0 tx=0).stop 2024-04-02T14:41:38.109 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.109+0000 7f4dab400640 1 -- 172.21.15.90:0/1150630967 shutdown_connections 2024-04-02T14:41:38.109 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.109+0000 7f4dab400640 1 --2- 172.21.15.90:0/1150630967 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f4d8c0a3e90 0x7f4d8c0a6350 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:38.110 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.109+0000 7f4dab400640 1 --2- 172.21.15.90:0/1150630967 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4da419e7d0 0x7f4da419ec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:38.110 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.109+0000 7f4dab400640 1 --2- 172.21.15.90:0/1150630967 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4da410d410 0x7f4da41a4b50 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:38.110 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.109+0000 7f4dab400640 1 --2- 172.21.15.90:0/1150630967 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4da4108340 0x7f4da41a4610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:38.110 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.109+0000 7f4dab400640 1 -- 172.21.15.90:0/1150630967 >> 172.21.15.90:0/1150630967 conn(0x7f4da4076070 msgr2=0x7f4da4076be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:41:38.110 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.109+0000 7f4dab400640 1 -- 172.21.15.90:0/1150630967 shutdown_connections 2024-04-02T14:41:38.110 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:38.109+0000 7f4dab400640 1 -- 172.21.15.90:0/1150630967 wait complete. 2024-04-02T14:41:38.110 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:41:37 smithi092 ceph-mon[36497]: pgmap v205: 97 pgs: 97 active+clean; 4.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 6.2 MiB/s wr, 723 op/s 2024-04-02T14:41:38.184 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:41:37 smithi177 ceph-mon[36431]: pgmap v205: 97 pgs: 97 active+clean; 4.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 6.2 MiB/s wr, 723 op/s 2024-04-02T14:41:38.186 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:41:37 smithi090 ceph-mon[31674]: pgmap v205: 97 pgs: 97 active+clean; 4.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 6.2 MiB/s wr, 723 op/s 2024-04-02T14:41:40.108 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:41:39 smithi092 ceph-mon[36497]: pgmap v206: 97 pgs: 97 active+clean; 4.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 8.3 MiB/s wr, 931 op/s 2024-04-02T14:41:40.184 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:41:39 smithi177 ceph-mon[36431]: pgmap v206: 97 pgs: 97 active+clean; 4.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 8.3 MiB/s wr, 931 op/s 2024-04-02T14:41:40.187 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:41:39 smithi090 ceph-mon[31674]: pgmap v206: 97 pgs: 97 active+clean; 4.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 8.3 MiB/s wr, 931 op/s 2024-04-02T14:41:41.829 INFO:teuthology.orchestra.run.smithi090.stderr:progress: 370.0 s, 2272.4 tps, lat 44.017 ms stddev 54.969 2024-04-02T14:41:42.108 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:41:41 smithi092 ceph-mon[36497]: pgmap v207: 97 pgs: 97 active+clean; 4.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 6.5 MiB/s wr, 715 op/s 2024-04-02T14:41:42.184 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:41:41 smithi177 ceph-mon[36431]: pgmap v207: 97 pgs: 97 active+clean; 4.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 6.5 MiB/s wr, 715 op/s 2024-04-02T14:41:42.186 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:41:41 smithi090 ceph-mon[31674]: pgmap v207: 97 pgs: 97 active+clean; 4.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 6.5 MiB/s wr, 715 op/s 2024-04-02T14:41:44.108 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:41:43 smithi092 ceph-mon[36497]: pgmap v208: 97 pgs: 97 active+clean; 4.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 5.7 MiB/s wr, 592 op/s 2024-04-02T14:41:44.186 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:41:43 smithi177 ceph-mon[36431]: pgmap v208: 97 pgs: 97 active+clean; 4.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 5.7 MiB/s wr, 592 op/s 2024-04-02T14:41:44.187 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:41:43 smithi090 ceph-mon[31674]: pgmap v208: 97 pgs: 97 active+clean; 4.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 5.7 MiB/s wr, 592 op/s 2024-04-02T14:41:46.108 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:41:45 smithi092 ceph-mon[36497]: pgmap v209: 97 pgs: 97 active+clean; 4.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 8.8 MiB/s wr, 913 op/s 2024-04-02T14:41:46.184 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:41:45 smithi177 ceph-mon[36431]: pgmap v209: 97 pgs: 97 active+clean; 4.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 8.8 MiB/s wr, 913 op/s 2024-04-02T14:41:46.186 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:41:45 smithi090 ceph-mon[31674]: pgmap v209: 97 pgs: 97 active+clean; 4.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 8.8 MiB/s wr, 913 op/s 2024-04-02T14:41:46.829 INFO:teuthology.orchestra.run.smithi090.stderr:progress: 375.0 s, 2839.1 tps, lat 35.031 ms stddev 39.245 2024-04-02T14:41:48.108 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:41:47 smithi092 ceph-mon[36497]: pgmap v210: 97 pgs: 97 active+clean; 4.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 5.8 MiB/s wr, 611 op/s 2024-04-02T14:41:48.184 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:41:47 smithi177 ceph-mon[36431]: pgmap v210: 97 pgs: 97 active+clean; 4.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 5.8 MiB/s wr, 611 op/s 2024-04-02T14:41:48.185 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:41:47 smithi090 ceph-mon[31674]: pgmap v210: 97 pgs: 97 active+clean; 4.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 5.8 MiB/s wr, 611 op/s 2024-04-02T14:41:50.108 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:41:49 smithi092 ceph-mon[36497]: pgmap v211: 97 pgs: 97 active+clean; 4.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 7.1 MiB/s wr, 759 op/s 2024-04-02T14:41:50.184 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:41:49 smithi177 ceph-mon[36431]: pgmap v211: 97 pgs: 97 active+clean; 4.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 7.1 MiB/s wr, 759 op/s 2024-04-02T14:41:50.185 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:41:49 smithi090 ceph-mon[31674]: pgmap v211: 97 pgs: 97 active+clean; 4.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 7.1 MiB/s wr, 759 op/s 2024-04-02T14:41:51.292 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.292+0000 7f042b000640 1 -- 172.21.15.90:0/2152360422 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0424073ed0 msgr2=0x7f04240742b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:41:51.292 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.292+0000 7f042b000640 1 --2- 172.21.15.90:0/2152360422 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0424073ed0 0x7f04240742b0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f0410009860 tx=0x7f041002bc30 comp rx=0 tx=0).stop 2024-04-02T14:41:51.292 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.293+0000 7f042b000640 1 -- 172.21.15.90:0/2152360422 shutdown_connections 2024-04-02T14:41:51.293 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.293+0000 7f042b000640 1 --2- 172.21.15.90:0/2152360422 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f042410e350 0x7f0424110760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:51.293 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.293+0000 7f042b000640 1 --2- 172.21.15.90:0/2152360422 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f04240747f0 0x7f042410de10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:51.293 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.293+0000 7f042b000640 1 --2- 172.21.15.90:0/2152360422 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0424073ed0 0x7f04240742b0 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:51.293 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.293+0000 7f042b000640 1 -- 172.21.15.90:0/2152360422 >> 172.21.15.90:0/2152360422 conn(0x7f04240fd5c0 msgr2=0x7f04240ff9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:41:51.293 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.293+0000 7f042b000640 1 -- 172.21.15.90:0/2152360422 shutdown_connections 2024-04-02T14:41:51.293 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.293+0000 7f042b000640 1 -- 172.21.15.90:0/2152360422 wait complete. 2024-04-02T14:41:51.296 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.297+0000 7f042b000640 1 Processor -- start 2024-04-02T14:41:51.296 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.297+0000 7f042b000640 1 -- start start 2024-04-02T14:41:51.297 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.298+0000 7f042b000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0424073ed0 0x7f042419a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:51.297 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.298+0000 7f042b000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f04240747f0 0x7f042419ab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:51.297 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.298+0000 7f042b000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f042410e350 0x7f042419ee50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:51.297 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.298+0000 7f042b000640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f0424114100 con 0x7f0424073ed0 2024-04-02T14:41:51.297 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.298+0000 7f042b000640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f0424113f80 con 0x7f042410e350 2024-04-02T14:41:51.297 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.298+0000 7f042b000640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f0424114280 con 0x7f04240747f0 2024-04-02T14:41:51.298 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.298+0000 7f0423400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f04240747f0 0x7f042419ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:51.298 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.298+0000 7f0423400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f04240747f0 0x7f042419ab20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:49850/0 (socket says 172.21.15.90:49850) 2024-04-02T14:41:51.298 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.298+0000 7f0423400640 1 -- 172.21.15.90:0/1886203004 learned_addr learned my addr 172.21.15.90:0/1886203004 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:41:51.298 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.298+0000 7f0429000640 1 --2- 172.21.15.90:0/1886203004 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f042410e350 0x7f042419ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:51.299 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.299+0000 7f0429000640 1 -- 172.21.15.90:0/1886203004 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f04240747f0 msgr2=0x7f042419ab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:41:51.299 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.299+0000 7f0429000640 1 --2- 172.21.15.90:0/1886203004 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f04240747f0 0x7f042419ab20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:51.299 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.299+0000 7f0429000640 1 -- 172.21.15.90:0/1886203004 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0424073ed0 msgr2=0x7f042419a5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:41:51.299 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.299+0000 7f0429000640 1 --2- 172.21.15.90:0/1886203004 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0424073ed0 0x7f042419a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:51.299 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.299+0000 7f0429000640 1 -- 172.21.15.90:0/1886203004 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f042419f580 con 0x7f042410e350 2024-04-02T14:41:51.299 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.300+0000 7f0429000640 1 --2- 172.21.15.90:0/1886203004 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f042410e350 0x7f042419ee50 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f0424074e20 tx=0x7f041c00edd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:41:51.300 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.300+0000 7f0420c00640 1 -- 172.21.15.90:0/1886203004 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f041c004d20 con 0x7f042410e350 2024-04-02T14:41:51.301 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.300+0000 7f0420c00640 1 -- 172.21.15.90:0/1886203004 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f041c005020 con 0x7f042410e350 2024-04-02T14:41:51.301 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.300+0000 7f0420c00640 1 -- 172.21.15.90:0/1886203004 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f041c0116d0 con 0x7f042410e350 2024-04-02T14:41:51.301 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.300+0000 7f042b000640 1 -- 172.21.15.90:0/1886203004 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f042419f870 con 0x7f042410e350 2024-04-02T14:41:51.301 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.301+0000 7f042b000640 1 -- 172.21.15.90:0/1886203004 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04241a7100 con 0x7f042410e350 2024-04-02T14:41:51.305 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.305+0000 7f0420c00640 1 -- 172.21.15.90:0/1886203004 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f041c0051d0 con 0x7f042410e350 2024-04-02T14:41:51.307 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.308+0000 7f0420c00640 1 --2- 172.21.15.90:0/1886203004 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f04080a3f10 0x7f04080a63d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:51.307 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.308+0000 7f0420c00640 1 -- 172.21.15.90:0/1886203004 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f041c0d0a30 con 0x7f042410e350 2024-04-02T14:41:51.308 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.309+0000 7f042dc42740 1 Processor -- start 2024-04-02T14:41:51.309 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.309+0000 7f042dc42740 1 -- start start 2024-04-02T14:41:51.310 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.310+0000 7f0423e00640 1 --2- 172.21.15.90:0/1886203004 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f04080a3f10 0x7f04080a63d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:51.310 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.310+0000 7f042dc42740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55622c3d9eb0 0x55622c3da290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:51.311 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.310+0000 7f042dc42740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55622c3da7d0 0x55622c3eb680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:51.311 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.310+0000 7f042dc42740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55622c3ebbc0 0x55622c3edfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:51.311 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.310+0000 7f042dc42740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55622c33ed70 con 0x55622c3da7d0 2024-04-02T14:41:51.311 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.310+0000 7f042dc42740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55622c2e72a0 con 0x55622c3ebbc0 2024-04-02T14:41:51.311 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.310+0000 7f042dc42740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55622c33ad20 con 0x55622c3d9eb0 2024-04-02T14:41:51.311 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.310+0000 7f0429000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55622c3ebbc0 0x55622c3edfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:51.311 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.310+0000 7f0429000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55622c3ebbc0 0x55622c3edfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:51938/0 (socket says 172.21.15.90:51938) 2024-04-02T14:41:51.311 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.311+0000 7f0429000640 1 -- 172.21.15.90:0/2685159213 learned_addr learned my addr 172.21.15.90:0/2685159213 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:41:51.311 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.311+0000 7f0423e00640 1 --2- 172.21.15.90:0/2685159213 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55622c3d9eb0 0x55622c3da290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:51.313 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.311+0000 7f0423400640 1 --2- 172.21.15.90:0/2685159213 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55622c3da7d0 0x55622c3eb680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:51.313 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.312+0000 7f0423e00640 1 --2- 172.21.15.90:0/1886203004 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f04080a3f10 0x7f04080a63d0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f041002c240 tx=0x7f0410009490 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:41:51.313 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.312+0000 7f0423e00640 1 -- 172.21.15.90:0/2685159213 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55622c3ebbc0 msgr2=0x55622c3edfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:41:51.313 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.312+0000 7f0423e00640 1 --2- 172.21.15.90:0/2685159213 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55622c3ebbc0 0x55622c3edfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:51.313 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.312+0000 7f0423e00640 1 -- 172.21.15.90:0/2685159213 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55622c3da7d0 msgr2=0x55622c3eb680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:41:51.313 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.312+0000 7f0423e00640 1 --2- 172.21.15.90:0/2685159213 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55622c3da7d0 0x55622c3eb680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:51.313 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.312+0000 7f0423e00640 1 -- 172.21.15.90:0/2685159213 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55622c2fb840 con 0x55622c3d9eb0 2024-04-02T14:41:51.313 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.312+0000 7f0423e00640 1 --2- 172.21.15.90:0/2685159213 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55622c3d9eb0 0x55622c3da290 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f0410009780 tx=0x7f041003b000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=20e9d0a6a95c28b0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:41:51.313 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.313+0000 7f0419600640 1 -- 172.21.15.90:0/2685159213 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f041004f070 con 0x55622c3d9eb0 2024-04-02T14:41:51.313 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.313+0000 7f0419600640 1 -- 172.21.15.90:0/2685159213 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0410005040 con 0x55622c3d9eb0 2024-04-02T14:41:51.314 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.313+0000 7f0419600640 1 -- 172.21.15.90:0/2685159213 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0410039040 con 0x55622c3d9eb0 2024-04-02T14:41:51.315 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.314+0000 7f042dc42740 1 -- 172.21.15.90:0/2685159213 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55622c3d9eb0 msgr2=0x55622c3da290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:41:51.315 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.314+0000 7f042dc42740 1 --2- 172.21.15.90:0/2685159213 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55622c3d9eb0 0x55622c3da290 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f0410009780 tx=0x7f041003b000 comp rx=0 tx=0).stop 2024-04-02T14:41:51.315 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.314+0000 7f042dc42740 1 -- 172.21.15.90:0/2685159213 shutdown_connections 2024-04-02T14:41:51.315 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.314+0000 7f042dc42740 1 --2- 172.21.15.90:0/2685159213 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55622c3ebbc0 0x55622c3edfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:51.315 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.315+0000 7f042dc42740 1 --2- 172.21.15.90:0/2685159213 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55622c3da7d0 0x55622c3eb680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:51.315 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.315+0000 7f042dc42740 1 --2- 172.21.15.90:0/2685159213 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55622c3d9eb0 0x55622c3da290 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:51.316 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.315+0000 7f042dc42740 1 -- 172.21.15.90:0/2685159213 >> 172.21.15.90:0/2685159213 conn(0x55622c3d45e0 msgr2=0x55622c3d49c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:41:51.316 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.315+0000 7f042dc42740 1 -- 172.21.15.90:0/2685159213 shutdown_connections 2024-04-02T14:41:51.316 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.315+0000 7f042dc42740 1 -- 172.21.15.90:0/2685159213 wait complete. 2024-04-02T14:41:51.316 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.316+0000 7f042dc42740 1 -- start start 2024-04-02T14:41:51.316 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.316+0000 7f042dc42740 1 Processor -- start 2024-04-02T14:41:51.317 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.317+0000 7f042dc42740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55622c3d9eb0 0x55622c3e57a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:51.318 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.317+0000 7f042dc42740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55622c3da7d0 0x55622c3eb4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:51.318 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.317+0000 7f042dc42740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55622c3ebbc0 0x55622c3e5ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:51.318 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.317+0000 7f042dc42740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55622c3df170 con 0x55622c3ebbc0 2024-04-02T14:41:51.318 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.317+0000 7f042dc42740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55622c2fa660 con 0x55622c3da7d0 2024-04-02T14:41:51.318 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.317+0000 7f042dc42740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55622c3df2f0 con 0x55622c3d9eb0 2024-04-02T14:41:51.318 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.318+0000 7f0429000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55622c3ebbc0 0x55622c3e5ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:51.318 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.318+0000 7f0429000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55622c3ebbc0 0x55622c3e5ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41370/0 (socket says 172.21.15.90:41370) 2024-04-02T14:41:51.318 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.318+0000 7f0429000640 1 -- 172.21.15.90:0/2087715547 learned_addr learned my addr 172.21.15.90:0/2087715547 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:41:51.319 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.318+0000 7f0423e00640 1 --2- 172.21.15.90:0/2087715547 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55622c3d9eb0 0x55622c3e57a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:51.319 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.319+0000 7f0429000640 1 -- 172.21.15.90:0/2087715547 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55622c3d9eb0 msgr2=0x55622c3e57a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:41:51.319 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.319+0000 7f0429000640 1 --2- 172.21.15.90:0/2087715547 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55622c3d9eb0 0x55622c3e57a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:51.319 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.319+0000 7f0423400640 1 --2- 172.21.15.90:0/2087715547 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55622c3da7d0 0x55622c3eb4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:51.319 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.319+0000 7f0429000640 1 -- 172.21.15.90:0/2087715547 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55622c3da7d0 msgr2=0x55622c3eb4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:41:51.319 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.319+0000 7f0429000640 1 --2- 172.21.15.90:0/2087715547 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55622c3da7d0 0x55622c3eb4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:51.320 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.319+0000 7f0429000640 1 -- 172.21.15.90:0/2087715547 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55622c309e80 con 0x55622c3ebbc0 2024-04-02T14:41:51.320 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.319+0000 7f042dc42740 10 client.0 _ll_register_callbacks cb 0x55622c2fa050 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:41:51.320 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.319+0000 7f042dc42740 1 -- 172.21.15.90:0/2087715547 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0410005040 con 0x55622c3ebbc0 2024-04-02T14:41:51.320 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.319+0000 7f0429000640 1 --2- 172.21.15.90:0/2087715547 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55622c3ebbc0 0x55622c3e5ce0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f041c098d90 tx=0x7f041c09ca40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=547c4bfb86d10675 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:41:51.320 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.319+0000 7f040e000640 10 client.16800 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-02T14:41:51.320 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.320+0000 7f040e000640 1 -- 172.21.15.90:0/2087715547 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f041c01f040 con 0x55622c3ebbc0 2024-04-02T14:41:51.320 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.320+0000 7f040e000640 1 -- 172.21.15.90:0/2087715547 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f041c0870b0 con 0x55622c3ebbc0 2024-04-02T14:41:51.321 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.320+0000 7f040e000640 1 -- 172.21.15.90:0/2087715547 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f041c0872b0 con 0x55622c3ebbc0 2024-04-02T14:41:51.321 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.320+0000 7f042dc42740 10 client.16800 fetch_fsmap learned FSMap version 27 2024-04-02T14:41:51.321 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.320+0000 7f042dc42740 1 -- 172.21.15.90:0/2087715547 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({fsmap=27}) -- 0x55622c053210 con 0x55622c3ebbc0 2024-04-02T14:41:51.321 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.321+0000 7f040e000640 1 -- 172.21.15.90:0/2087715547 <== mon.0 v2:172.21.15.90:3300/0 4 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f041c00f040 con 0x55622c3ebbc0 2024-04-02T14:41:51.321 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.321+0000 7f042dc42740 10 client.16800 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:41:51.321 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.321+0000 7f042dc42740 10 client.16800 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-02T14:41:51.321 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.321+0000 7f042dc42740 20 client.16800 populate_metadata read hostname 'smithi090' 2024-04-02T14:41:51.321 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.321+0000 7f042dc42740 1 --2- 172.21.15.90:0/2087715547 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55622c404580 0x55622c406a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:51.321 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.321+0000 7f042dc42740 4 client.16800 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:41:51.321 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.321+0000 7f042dc42740 1 -- 172.21.15.90:0/2087715547 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55622c2eecd0 con 0x55622c404580 2024-04-02T14:41:51.321 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.321+0000 7f0423e00640 1 --2- 172.21.15.90:0/2087715547 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55622c404580 0x55622c406a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:51.322 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.322+0000 7f0423e00640 1 --2- 172.21.15.90:0/2087715547 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55622c404580 0x55622c406a10 crc :-1 s=READY pgs=190 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8efd7f2f0d5e2ecd server_cookie=5b27a6162374ea1b in_seq=0 out_seq=0 2024-04-02T14:41:51.322 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.322+0000 7f040e000640 10 client.16800 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:41:51.322 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.323+0000 7f040e000640 1 -- 172.21.15.90:0/2087715547 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55622c2eecd0 con 0x55622c404580 2024-04-02T14:41:51.322 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.323+0000 7f040e000640 10 client.16800 handle_command_reply: tid=0 2024-04-02T14:41:51.323 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.323+0000 7f042dc42740 1 client.16800 shutdown 2024-04-02T14:41:51.323 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.323+0000 7f042dc42740 1 -- 172.21.15.90:0/2087715547 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55622c3ebbc0 msgr2=0x55622c3e5ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:41:51.323 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.323+0000 7f042dc42740 1 --2- 172.21.15.90:0/2087715547 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55622c3ebbc0 0x55622c3e5ce0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f041c098d90 tx=0x7f041c09ca40 comp rx=0 tx=0).stop 2024-04-02T14:41:51.323 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.323+0000 7f042dc42740 1 -- 172.21.15.90:0/2087715547 shutdown_connections 2024-04-02T14:41:51.323 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.323+0000 7f042dc42740 1 --2- 172.21.15.90:0/2087715547 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55622c404580 0x55622c406a10 crc :-1 s=READY pgs=190 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:51.323 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.323+0000 7f042dc42740 1 --2- 172.21.15.90:0/2087715547 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55622c3ebbc0 0x55622c3e5ce0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:51.323 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.323+0000 7f042dc42740 1 --2- 172.21.15.90:0/2087715547 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55622c3da7d0 0x55622c3eb4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:51.323 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.323+0000 7f042dc42740 1 --2- 172.21.15.90:0/2087715547 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55622c3d9eb0 0x55622c3e57a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:51.324 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.323+0000 7f040e000640 0 client.16800 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:41:51.324 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.323+0000 7f042dc42740 1 -- 172.21.15.90:0/2087715547 >> 172.21.15.90:0/2087715547 conn(0x55622c3d45e0 msgr2=0x55622c3d6330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:41:51.324 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.324+0000 7f042dc42740 1 -- 172.21.15.90:0/2087715547 shutdown_connections 2024-04-02T14:41:51.324 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.324+0000 7f042dc42740 1 -- 172.21.15.90:0/2087715547 wait complete. 2024-04-02T14:41:51.324 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.324+0000 7f042dc42740 20 client.16800 trim_cache size 0 max 16384 2024-04-02T14:41:51.351 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.352+0000 7f042dc42740 1 Processor -- start 2024-04-02T14:41:51.351 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.352+0000 7f042dc42740 1 -- start start 2024-04-02T14:41:51.352 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.352+0000 7f042dc42740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55622c3d9eb0 0x55622c40c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:51.352 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.352+0000 7f042dc42740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55622c3da7d0 0x55622c40c830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:51.352 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.352+0000 7f042dc42740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55622c3ebbc0 0x55622c3dd6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:51.352 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.352+0000 7f042dc42740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f04241ab6a0 con 0x55622c3ebbc0 2024-04-02T14:41:51.352 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.352+0000 7f042dc42740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f04241ad1e0 con 0x55622c3da7d0 2024-04-02T14:41:51.352 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.352+0000 7f042dc42740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f04241abd70 con 0x55622c3d9eb0 2024-04-02T14:41:51.353 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.353+0000 7f0423400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55622c3da7d0 0x55622c40c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:51.353 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.353+0000 7f0423400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55622c3da7d0 0x55622c40c830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:51952/0 (socket says 172.21.15.90:51952) 2024-04-02T14:41:51.353 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.353+0000 7f0423400640 1 -- 172.21.15.90:0/2214709315 learned_addr learned my addr 172.21.15.90:0/2214709315 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:41:51.354 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.353+0000 7f0423e00640 1 --2- 172.21.15.90:0/2214709315 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55622c3d9eb0 0x55622c40c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:51.354 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.353+0000 7f0429000640 1 --2- 172.21.15.90:0/2214709315 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55622c3ebbc0 0x55622c3dd6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:51.354 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.354+0000 7f0423400640 1 -- 172.21.15.90:0/2214709315 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55622c3d9eb0 msgr2=0x55622c40c2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:41:51.354 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.354+0000 7f0423400640 1 --2- 172.21.15.90:0/2214709315 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55622c3d9eb0 0x55622c40c2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:51.354 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.354+0000 7f0423400640 1 -- 172.21.15.90:0/2214709315 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55622c3ebbc0 msgr2=0x55622c3dd6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:41:51.354 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.354+0000 7f0423400640 1 --2- 172.21.15.90:0/2214709315 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55622c3ebbc0 0x55622c3dd6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:51.354 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.354+0000 7f0423400640 1 -- 172.21.15.90:0/2214709315 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55622c11f620 con 0x55622c3da7d0 2024-04-02T14:41:51.354 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.354+0000 7f0429000640 1 --2- 172.21.15.90:0/2214709315 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55622c3ebbc0 0x55622c3dd6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:41:51.355 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.355+0000 7f0423400640 1 --2- 172.21.15.90:0/2214709315 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55622c3da7d0 0x55622c40c830 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f0414009c10 tx=0x7f041400f7f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dfabf87fa7e2caec server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:41:51.355 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.355+0000 7f0418c00640 1 -- 172.21.15.90:0/2214709315 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0414009d00 con 0x55622c3da7d0 2024-04-02T14:41:51.355 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.355+0000 7f0418c00640 1 -- 172.21.15.90:0/2214709315 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0414013940 con 0x55622c3da7d0 2024-04-02T14:41:51.355 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.355+0000 7f0418c00640 1 -- 172.21.15.90:0/2214709315 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04140119d0 con 0x55622c3da7d0 2024-04-02T14:41:51.355 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.356+0000 7f042dc42740 1 -- 172.21.15.90:0/2214709315 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55622c3da7d0 msgr2=0x55622c40c830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:41:51.355 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.356+0000 7f042dc42740 1 --2- 172.21.15.90:0/2214709315 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55622c3da7d0 0x55622c40c830 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f0414009c10 tx=0x7f041400f7f0 comp rx=0 tx=0).stop 2024-04-02T14:41:51.356 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.356+0000 7f042dc42740 1 -- 172.21.15.90:0/2214709315 shutdown_connections 2024-04-02T14:41:51.356 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.356+0000 7f042dc42740 1 --2- 172.21.15.90:0/2214709315 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55622c3ebbc0 0x55622c3dd6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:51.356 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.356+0000 7f042dc42740 1 --2- 172.21.15.90:0/2214709315 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55622c3da7d0 0x55622c40c830 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:51.356 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.356+0000 7f042dc42740 1 --2- 172.21.15.90:0/2214709315 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55622c3d9eb0 0x55622c40c2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:51.356 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.356+0000 7f042dc42740 1 -- 172.21.15.90:0/2214709315 >> 172.21.15.90:0/2214709315 conn(0x55622c3d45e0 msgr2=0x55622c3d62a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:41:51.356 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.356+0000 7f042dc42740 1 -- 172.21.15.90:0/2214709315 shutdown_connections 2024-04-02T14:41:51.356 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.356+0000 7f042dc42740 1 -- 172.21.15.90:0/2214709315 wait complete. 2024-04-02T14:41:51.356 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.357+0000 7f042dc42740 1 -- start start 2024-04-02T14:41:51.357 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.357+0000 7f042dc42740 1 Processor -- start 2024-04-02T14:41:51.357 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.357+0000 7f042dc42740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55622c3d9eb0 0x55622c4178a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:51.357 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.357+0000 7f042dc42740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55622c3da7d0 0x55622c417de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:51.357 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.357+0000 7f042dc42740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55622c3ebbc0 0x55622c415e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:51.357 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.357+0000 7f042dc42740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f04241acb10 con 0x55622c3ebbc0 2024-04-02T14:41:51.357 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.357+0000 7f042dc42740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f04241ac440 con 0x55622c3da7d0 2024-04-02T14:41:51.357 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.357+0000 7f042dc42740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f04241a9680 con 0x55622c3d9eb0 2024-04-02T14:41:51.358 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.358+0000 7f042dc42740 10 client.0 _ll_register_callbacks cb 0x7f041c09d0f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:41:51.358 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.358+0000 7f0423e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55622c3d9eb0 0x55622c4178a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:51.358 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.358+0000 7f0423e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55622c3d9eb0 0x55622c4178a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:49888/0 (socket says 172.21.15.90:49888) 2024-04-02T14:41:51.358 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.358+0000 7f0423e00640 1 -- 172.21.15.90:0/2916311130 learned_addr learned my addr 172.21.15.90:0/2916311130 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:41:51.358 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.359+0000 7f0423400640 1 --2- 172.21.15.90:0/2916311130 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55622c3da7d0 0x55622c417de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:51.359 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.359+0000 7f0423400640 1 -- 172.21.15.90:0/2916311130 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55622c3d9eb0 msgr2=0x55622c4178a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:41:51.359 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.359+0000 7f0423400640 1 --2- 172.21.15.90:0/2916311130 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55622c3d9eb0 0x55622c4178a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:51.359 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.359+0000 7f0423400640 1 -- 172.21.15.90:0/2916311130 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55622c3ebbc0 msgr2=0x55622c415e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:41:51.359 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.359+0000 7f0423400640 1 --2- 172.21.15.90:0/2916311130 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55622c3ebbc0 0x55622c415e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:51.359 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.359+0000 7f0423400640 1 -- 172.21.15.90:0/2916311130 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55622c416640 con 0x55622c3da7d0 2024-04-02T14:41:51.360 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.361+0000 7f0423400640 1 --2- 172.21.15.90:0/2916311130 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55622c3da7d0 0x55622c417de0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x55622c3d4410 tx=0x7f0414013690 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8d2e070c5cebcb79 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:41:51.360 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.361+0000 7f040e000640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:41:51.361 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.362+0000 7f040e000640 1 -- 172.21.15.90:0/2916311130 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0414013d00 con 0x55622c3da7d0 2024-04-02T14:41:51.361 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.362+0000 7f040e000640 1 -- 172.21.15.90:0/2916311130 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f0414018070 con 0x55622c3da7d0 2024-04-02T14:41:51.362 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.362+0000 7f042dc42740 1 -- 172.21.15.90:0/2916311130 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0414013940 con 0x55622c3da7d0 2024-04-02T14:41:51.362 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.363+0000 7f040e000640 1 -- 172.21.15.90:0/2916311130 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f0414011710 con 0x55622c3da7d0 2024-04-02T14:41:51.362 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.363+0000 7f042dc42740 10 client.27497 fetch_fsmap learned FSMap version 27 2024-04-02T14:41:51.362 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.363+0000 7f042dc42740 1 -- 172.21.15.90:0/2916311130 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55622c46b1c0 con 0x55622c3da7d0 2024-04-02T14:41:51.363 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.363+0000 7f040e000640 1 -- 172.21.15.90:0/2916311130 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0414013940 con 0x55622c3da7d0 2024-04-02T14:41:51.363 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.364+0000 7f040e000640 1 -- 172.21.15.90:0/2916311130 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f0414024720 con 0x55622c3da7d0 2024-04-02T14:41:51.363 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.364+0000 7f042dc42740 10 client.27497 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:41:51.363 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.364+0000 7f042dc42740 10 client.27497 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-02T14:41:51.363 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.364+0000 7f042dc42740 20 client.27497 populate_metadata read hostname 'smithi090' 2024-04-02T14:41:51.364 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.364+0000 7f042dc42740 1 --2- 172.21.15.90:0/2916311130 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55622c46b780 0x55622c46db70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:51.364 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.364+0000 7f042dc42740 4 client.27497 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-02T14:41:51.364 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.364+0000 7f042dc42740 1 -- 172.21.15.90:0/2916311130 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55622c34c2b0 con 0x55622c46b780 2024-04-02T14:41:51.364 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.364+0000 7f0423e00640 1 --2- 172.21.15.90:0/2916311130 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55622c46b780 0x55622c46db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:51.365 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.366+0000 7f040e000640 10 client.27497 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:41:51.365 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.366+0000 7f0423e00640 1 --2- 172.21.15.90:0/2916311130 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55622c46b780 0x55622c46db70 crc :-1 s=READY pgs=191 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cbc40b333869b73e server_cookie=bd5a0cb5d08b9e07 in_seq=0 out_seq=0 2024-04-02T14:41:51.385 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.385+0000 7f040e000640 1 -- 172.21.15.90:0/2916311130 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55622c34c2b0 con 0x55622c46b780 2024-04-02T14:41:51.385 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.385+0000 7f040e000640 10 client.27497 handle_command_reply: tid=0 2024-04-02T14:41:51.385 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.385+0000 7f042dc42740 1 client.27497 shutdown 2024-04-02T14:41:51.386 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.387+0000 7f042dc42740 1 -- 172.21.15.90:0/2916311130 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55622c3da7d0 msgr2=0x55622c417de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:41:51.387 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.387+0000 7f042dc42740 1 --2- 172.21.15.90:0/2916311130 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55622c3da7d0 0x55622c417de0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x55622c3d4410 tx=0x7f0414013690 comp rx=0 tx=0).stop 2024-04-02T14:41:51.387 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.387+0000 7f042dc42740 1 -- 172.21.15.90:0/2916311130 shutdown_connections 2024-04-02T14:41:51.387 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.387+0000 7f042dc42740 1 --2- 172.21.15.90:0/2916311130 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55622c46b780 0x55622c46db70 crc :-1 s=READY pgs=191 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:51.387 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.387+0000 7f042dc42740 1 --2- 172.21.15.90:0/2916311130 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55622c3ebbc0 0x55622c415e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:51.387 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.387+0000 7f042dc42740 1 --2- 172.21.15.90:0/2916311130 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55622c3da7d0 0x55622c417de0 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:51.387 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.387+0000 7f042dc42740 1 --2- 172.21.15.90:0/2916311130 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55622c3d9eb0 0x55622c4178a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:51.387 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.387+0000 7f040e000640 0 client.27497 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:41:51.387 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.387+0000 7f042dc42740 1 -- 172.21.15.90:0/2916311130 >> 172.21.15.90:0/2916311130 conn(0x55622c3d45e0 msgr2=0x55622c3d2db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:41:51.387 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.387+0000 7f042dc42740 1 -- 172.21.15.90:0/2916311130 shutdown_connections 2024-04-02T14:41:51.387 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.387+0000 7f042dc42740 1 -- 172.21.15.90:0/2916311130 wait complete. 2024-04-02T14:41:51.387 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.387+0000 7f042dc42740 20 client.27497 trim_cache size 0 max 16384 2024-04-02T14:41:51.388 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:{ 2024-04-02T14:41:51.388 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "message": "", 2024-04-02T14:41:51.388 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "return_code": 0 2024-04-02T14:41:51.388 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:} 2024-04-02T14:41:51.389 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.389+0000 7f042b000640 1 -- 172.21.15.90:0/1886203004 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f04080a3f10 msgr2=0x7f04080a63d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:41:51.389 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.389+0000 7f042b000640 1 --2- 172.21.15.90:0/1886203004 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f04080a3f10 0x7f04080a63d0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f041002c240 tx=0x7f0410009490 comp rx=0 tx=0).stop 2024-04-02T14:41:51.389 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.389+0000 7f042b000640 1 -- 172.21.15.90:0/1886203004 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f042410e350 msgr2=0x7f042419ee50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:41:51.389 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.389+0000 7f042b000640 1 --2- 172.21.15.90:0/1886203004 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f042410e350 0x7f042419ee50 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f0424074e20 tx=0x7f041c00edd0 comp rx=0 tx=0).stop 2024-04-02T14:41:51.389 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.389+0000 7f042b000640 1 -- 172.21.15.90:0/1886203004 shutdown_connections 2024-04-02T14:41:51.389 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.389+0000 7f042b000640 1 --2- 172.21.15.90:0/1886203004 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f04080a3f10 0x7f04080a63d0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:51.389 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.389+0000 7f042b000640 1 --2- 172.21.15.90:0/1886203004 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f042410e350 0x7f042419ee50 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:51.389 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.389+0000 7f042b000640 1 --2- 172.21.15.90:0/1886203004 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f04240747f0 0x7f042419ab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:51.389 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.389+0000 7f042b000640 1 --2- 172.21.15.90:0/1886203004 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0424073ed0 0x7f042419a5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:51.389 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.389+0000 7f042b000640 1 -- 172.21.15.90:0/1886203004 >> 172.21.15.90:0/1886203004 conn(0x7f04240fd5c0 msgr2=0x7f04240fece0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:41:51.389 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.389+0000 7f042b000640 1 -- 172.21.15.90:0/1886203004 shutdown_connections 2024-04-02T14:41:51.389 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:41:51.389+0000 7f042b000640 1 -- 172.21.15.90:0/1886203004 wait complete. 2024-04-02T14:41:51.829 INFO:teuthology.orchestra.run.smithi090.stderr:progress: 380.0 s, 2076.5 tps, lat 48.228 ms stddev 57.157 2024-04-02T14:41:52.108 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:41:51 smithi092 ceph-mon[36497]: pgmap v212: 97 pgs: 97 active+clean; 4.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 6.6 MiB/s wr, 705 op/s 2024-04-02T14:41:52.184 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:41:51 smithi177 ceph-mon[36431]: pgmap v212: 97 pgs: 97 active+clean; 4.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 6.6 MiB/s wr, 705 op/s 2024-04-02T14:41:52.185 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:41:51 smithi090 ceph-mon[31674]: pgmap v212: 97 pgs: 97 active+clean; 4.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 6.6 MiB/s wr, 705 op/s 2024-04-02T14:41:54.108 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:41:53 smithi092 ceph-mon[36497]: pgmap v213: 97 pgs: 97 active+clean; 4.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 6.0 MiB/s wr, 622 op/s 2024-04-02T14:41:54.184 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:41:53 smithi177 ceph-mon[36431]: pgmap v213: 97 pgs: 97 active+clean; 4.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 6.0 MiB/s wr, 622 op/s 2024-04-02T14:41:54.186 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:41:53 smithi090 ceph-mon[31674]: pgmap v213: 97 pgs: 97 active+clean; 4.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 6.0 MiB/s wr, 622 op/s 2024-04-02T14:41:56.108 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:41:55 smithi092 ceph-mon[36497]: pgmap v214: 97 pgs: 97 active+clean; 4.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 9.0 MiB/s wr, 913 op/s 2024-04-02T14:41:56.184 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:41:55 smithi177 ceph-mon[36431]: pgmap v214: 97 pgs: 97 active+clean; 4.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 9.0 MiB/s wr, 913 op/s 2024-04-02T14:41:56.185 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:41:55 smithi090 ceph-mon[31674]: pgmap v214: 97 pgs: 97 active+clean; 4.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 9.0 MiB/s wr, 913 op/s 2024-04-02T14:41:56.829 INFO:teuthology.orchestra.run.smithi090.stderr:progress: 385.0 s, 3899.4 tps, lat 25.590 ms stddev 29.478 2024-04-02T14:41:57.240 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-02T14:41:57.240 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-02T14:41:57.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.525+0000 7f3e55200640 1 -- 172.21.15.90:0/3337414832 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3e50108310 msgr2=0x7f3e501086f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:41:57.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.525+0000 7f3e55200640 1 --2- 172.21.15.90:0/3337414832 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3e50108310 0x7f3e501086f0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f3e40009900 tx=0x7f3e4002bc30 comp rx=0 tx=0).stop 2024-04-02T14:41:57.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.526+0000 7f3e55200640 1 -- 172.21.15.90:0/3337414832 shutdown_connections 2024-04-02T14:41:57.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.526+0000 7f3e55200640 1 --2- 172.21.15.90:0/3337414832 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3e5010d3e0 0x7f3e50114fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:57.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.526+0000 7f3e55200640 1 --2- 172.21.15.90:0/3337414832 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3e50108cc0 0x7f3e5010cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:57.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.526+0000 7f3e55200640 1 --2- 172.21.15.90:0/3337414832 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3e50108310 0x7f3e501086f0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:57.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.526+0000 7f3e55200640 1 -- 172.21.15.90:0/3337414832 >> 172.21.15.90:0/3337414832 conn(0x7f3e50076040 msgr2=0x7f3e50078460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:41:57.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.526+0000 7f3e55200640 1 -- 172.21.15.90:0/3337414832 shutdown_connections 2024-04-02T14:41:57.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.526+0000 7f3e55200640 1 -- 172.21.15.90:0/3337414832 wait complete. 2024-04-02T14:41:57.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.527+0000 7f3e55200640 1 Processor -- start 2024-04-02T14:41:57.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.528+0000 7f3e55200640 1 -- start start 2024-04-02T14:41:57.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.528+0000 7f3e55200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3e50108310 0x7f3e501a46c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:57.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.528+0000 7f3e55200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3e5010d3e0 0x7f3e501a4c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:57.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.528+0000 7f3e55200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3e5019e830 0x7f3e5019ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:57.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.528+0000 7f3e55200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f3e501184f0 con 0x7f3e5019e830 2024-04-02T14:41:57.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.528+0000 7f3e55200640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f3e50118370 con 0x7f3e5010d3e0 2024-04-02T14:41:57.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.528+0000 7f3e55200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f3e50118670 con 0x7f3e50108310 2024-04-02T14:41:57.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.529+0000 7f3e47e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3e5010d3e0 0x7f3e501a4c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:57.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.529+0000 7f3e47e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3e5010d3e0 0x7f3e501a4c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:36428/0 (socket says 172.21.15.90:36428) 2024-04-02T14:41:57.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.529+0000 7f3e47e00640 1 -- 172.21.15.90:0/258093501 learned_addr learned my addr 172.21.15.90:0/258093501 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:41:57.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.529+0000 7f3e4e800640 1 --2- 172.21.15.90:0/258093501 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3e5019e830 0x7f3e5019ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:57.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.529+0000 7f3e47e00640 1 -- 172.21.15.90:0/258093501 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3e50108310 msgr2=0x7f3e501a46c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:41:57.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.529+0000 7f3e4de00640 1 --2- 172.21.15.90:0/258093501 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3e50108310 0x7f3e501a46c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:57.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.529+0000 7f3e47e00640 1 --2- 172.21.15.90:0/258093501 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3e50108310 0x7f3e501a46c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:57.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.529+0000 7f3e47e00640 1 -- 172.21.15.90:0/258093501 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3e5019e830 msgr2=0x7f3e5019ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:41:57.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.529+0000 7f3e47e00640 1 --2- 172.21.15.90:0/258093501 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3e5019e830 0x7f3e5019ece0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:57.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.529+0000 7f3e47e00640 1 -- 172.21.15.90:0/258093501 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e5019f3c0 con 0x7f3e5010d3e0 2024-04-02T14:41:57.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.529+0000 7f3e4e800640 1 --2- 172.21.15.90:0/258093501 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3e5019e830 0x7f3e5019ece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:41:57.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.530+0000 7f3e47e00640 1 --2- 172.21.15.90:0/258093501 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3e5010d3e0 0x7f3e501a4c00 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f3e38005fd0 tx=0x7f3e3800dde0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:41:57.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.530+0000 7f3e46a00640 1 -- 172.21.15.90:0/258093501 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e38017070 con 0x7f3e5010d3e0 2024-04-02T14:41:57.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.530+0000 7f3e55200640 1 -- 172.21.15.90:0/258093501 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e5019f6b0 con 0x7f3e5010d3e0 2024-04-02T14:41:57.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.530+0000 7f3e46a00640 1 -- 172.21.15.90:0/258093501 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3e38009c30 con 0x7f3e5010d3e0 2024-04-02T14:41:57.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.530+0000 7f3e46a00640 1 -- 172.21.15.90:0/258093501 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e38003e40 con 0x7f3e5010d3e0 2024-04-02T14:41:57.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.530+0000 7f3e55200640 1 -- 172.21.15.90:0/258093501 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e501ab8e0 con 0x7f3e5010d3e0 2024-04-02T14:41:57.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.534+0000 7f3e57fd1740 1 Processor -- start 2024-04-02T14:41:57.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.534+0000 7f3e57fd1740 1 -- start start 2024-04-02T14:41:57.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.535+0000 7f3e57fd1740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5606c74c97b0 0x5606c74c9b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:57.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.535+0000 7f3e57fd1740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5606c74ca0d0 0x5606c74d2770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:57.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.535+0000 7f3e57fd1740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5606c74d2cb0 0x5606c74d50a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:57.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.535+0000 7f3e57fd1740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5606c742d770 con 0x5606c74d2cb0 2024-04-02T14:41:57.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.535+0000 7f3e57fd1740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x5606c73d5700 con 0x5606c74ca0d0 2024-04-02T14:41:57.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.535+0000 7f3e57fd1740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5606c7429720 con 0x5606c74c97b0 2024-04-02T14:41:57.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.535+0000 7f3e4e800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5606c74c97b0 0x5606c74c9b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:57.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.535+0000 7f3e4e800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5606c74c97b0 0x5606c74c9b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:56352/0 (socket says 172.21.15.90:56352) 2024-04-02T14:41:57.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.535+0000 7f3e4e800640 1 -- 172.21.15.90:0/1990305446 learned_addr learned my addr 172.21.15.90:0/1990305446 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:41:57.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.535+0000 7f3e4de00640 1 --2- 172.21.15.90:0/1990305446 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5606c74ca0d0 0x5606c74d2770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:57.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.536+0000 7f3e4e800640 1 -- 172.21.15.90:0/1990305446 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5606c74ca0d0 msgr2=0x5606c74d2770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:41:57.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.536+0000 7f3e4e800640 1 --2- 172.21.15.90:0/1990305446 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5606c74ca0d0 0x5606c74d2770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:57.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.536+0000 7f3e4e800640 1 -- 172.21.15.90:0/1990305446 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5606c74d2cb0 msgr2=0x5606c74d50a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-02T14:41:57.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.536+0000 7f3e4e800640 1 --2- 172.21.15.90:0/1990305446 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5606c74d2cb0 0x5606c74d50a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:57.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.536+0000 7f3e4e800640 1 -- 172.21.15.90:0/1990305446 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5606c720b7d0 con 0x5606c74c97b0 2024-04-02T14:41:57.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.536+0000 7f3e4e800640 1 --2- 172.21.15.90:0/1990305446 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5606c74c97b0 0x5606c74c9b90 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f3e48009e40 tx=0x7f3e4800f730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fb800076b340db83 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:41:57.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.536+0000 7f3e46a00640 1 -- 172.21.15.90:0/258093501 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f3e38004020 con 0x7f3e5010d3e0 2024-04-02T14:41:57.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.537+0000 7f3e3f400640 1 -- 172.21.15.90:0/1990305446 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e48004d20 con 0x5606c74c97b0 2024-04-02T14:41:57.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.537+0000 7f3e3f400640 1 -- 172.21.15.90:0/1990305446 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3e48005500 con 0x5606c74c97b0 2024-04-02T14:41:57.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.538+0000 7f3e3f400640 1 -- 172.21.15.90:0/1990305446 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e480118e0 con 0x5606c74c97b0 2024-04-02T14:41:57.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.539+0000 7f3e46a00640 1 --2- 172.21.15.90:0/258093501 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f3e340a3da0 0x7f3e340a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:57.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.539+0000 7f3e46a00640 1 -- 172.21.15.90:0/258093501 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f3e380d04f0 con 0x7f3e5010d3e0 2024-04-02T14:41:57.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.539+0000 7f3e57fd1740 1 -- 172.21.15.90:0/1990305446 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5606c74c97b0 msgr2=0x5606c74c9b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:41:57.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.539+0000 7f3e57fd1740 1 --2- 172.21.15.90:0/1990305446 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5606c74c97b0 0x5606c74c9b90 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f3e48009e40 tx=0x7f3e4800f730 comp rx=0 tx=0).stop 2024-04-02T14:41:57.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.539+0000 7f3e57fd1740 1 -- 172.21.15.90:0/1990305446 shutdown_connections 2024-04-02T14:41:57.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.539+0000 7f3e57fd1740 1 --2- 172.21.15.90:0/1990305446 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5606c74d2cb0 0x5606c74d50a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:57.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.539+0000 7f3e57fd1740 1 --2- 172.21.15.90:0/1990305446 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5606c74ca0d0 0x5606c74d2770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:57.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.539+0000 7f3e57fd1740 1 --2- 172.21.15.90:0/1990305446 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5606c74c97b0 0x5606c74c9b90 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:57.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.540+0000 7f3e57fd1740 1 -- 172.21.15.90:0/1990305446 >> 172.21.15.90:0/1990305446 conn(0x5606c74c3ee0 msgr2=0x5606c74c42c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:41:57.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.540+0000 7f3e4de00640 1 --2- 172.21.15.90:0/258093501 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f3e340a3da0 0x7f3e340a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:57.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.540+0000 7f3e57fd1740 1 -- 172.21.15.90:0/1990305446 shutdown_connections 2024-04-02T14:41:57.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.540+0000 7f3e57fd1740 1 -- 172.21.15.90:0/1990305446 wait complete. 2024-04-02T14:41:57.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.540+0000 7f3e4de00640 1 --2- 172.21.15.90:0/258093501 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f3e340a3da0 0x7f3e340a6260 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f3e40009230 tx=0x7f3e4002df90 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:41:57.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.540+0000 7f3e57fd1740 1 -- start start 2024-04-02T14:41:57.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.541+0000 7f3e57fd1740 1 Processor -- start 2024-04-02T14:41:57.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.542+0000 7f3e57fd1740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5606c74c97b0 0x5606c74cfde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:57.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.542+0000 7f3e57fd1740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5606c74ca0d0 0x5606c74d0320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:57.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.542+0000 7f3e57fd1740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5606c74d2cb0 0x5606c74d0860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:57.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.542+0000 7f3e57fd1740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5606c74d5aa0 con 0x5606c74d2cb0 2024-04-02T14:41:57.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.543+0000 7f3e57fd1740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x5606c73e8ac0 con 0x5606c74c97b0 2024-04-02T14:41:57.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.543+0000 7f3e57fd1740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5606c74d5c20 con 0x5606c74ca0d0 2024-04-02T14:41:57.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.543+0000 7f3e57fd1740 10 client.0 _ll_register_callbacks cb 0x5606c73eb180 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:41:57.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.543+0000 7f3e4de00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5606c74c97b0 0x5606c74cfde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:57.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.543+0000 7f3e4de00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5606c74c97b0 0x5606c74cfde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:36452/0 (socket says 172.21.15.90:36452) 2024-04-02T14:41:57.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.543+0000 7f3e4de00640 1 -- 172.21.15.90:0/553465333 learned_addr learned my addr 172.21.15.90:0/553465333 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:41:57.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.543+0000 7f3e47e00640 1 --2- 172.21.15.90:0/553465333 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5606c74ca0d0 0x5606c74d0320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:57.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.544+0000 7f3e4de00640 1 -- 172.21.15.90:0/553465333 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5606c74ca0d0 msgr2=0x5606c74d0320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:41:57.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.544+0000 7f3e4de00640 1 --2- 172.21.15.90:0/553465333 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5606c74ca0d0 0x5606c74d0320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:57.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.544+0000 7f3e4de00640 1 -- 172.21.15.90:0/553465333 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5606c74d2cb0 msgr2=0x5606c74d0860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:41:57.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.544+0000 7f3e4de00640 1 --2- 172.21.15.90:0/553465333 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5606c74d2cb0 0x5606c74d0860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:57.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.544+0000 7f3e4de00640 1 -- 172.21.15.90:0/553465333 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5606c71413f0 con 0x5606c74c97b0 2024-04-02T14:41:57.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.544+0000 7f3e47e00640 1 --2- 172.21.15.90:0/553465333 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5606c74ca0d0 0x5606c74d0320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:41:57.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.544+0000 7f3e4de00640 1 --2- 172.21.15.90:0/553465333 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5606c74c97b0 0x5606c74cfde0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f3e40009900 tx=0x7f3e4002bc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aba7a0e365cba987 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:41:57.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.544+0000 7f3e33e00640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:41:57.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.544+0000 7f3e33e00640 1 -- 172.21.15.90:0/553465333 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e4004f020 con 0x5606c74c97b0 2024-04-02T14:41:57.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.544+0000 7f3e33e00640 1 -- 172.21.15.90:0/553465333 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f3e400046c0 con 0x5606c74c97b0 2024-04-02T14:41:57.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.544+0000 7f3e57fd1740 1 -- 172.21.15.90:0/553465333 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3e48005500 con 0x5606c74c97b0 2024-04-02T14:41:57.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.545+0000 7f3e33e00640 1 -- 172.21.15.90:0/553465333 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f3e400048c0 con 0x5606c74c97b0 2024-04-02T14:41:57.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.545+0000 7f3e57fd1740 10 client.27509 fetch_fsmap learned FSMap version 27 2024-04-02T14:41:57.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.545+0000 7f3e57fd1740 1 -- 172.21.15.90:0/553465333 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5606c71010f0 con 0x5606c74c97b0 2024-04-02T14:41:57.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.545+0000 7f3e33e00640 1 -- 172.21.15.90:0/553465333 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3e48005500 con 0x5606c74c97b0 2024-04-02T14:41:57.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.547+0000 7f3e33e00640 1 -- 172.21.15.90:0/553465333 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f3e40034440 con 0x5606c74c97b0 2024-04-02T14:41:57.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.547+0000 7f3e57fd1740 10 client.27509 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:41:57.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.547+0000 7f3e57fd1740 10 client.27509 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:41:57.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.547+0000 7f3e57fd1740 20 client.27509 populate_metadata read hostname 'smithi090' 2024-04-02T14:41:57.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.547+0000 7f3e57fd1740 1 --2- 172.21.15.90:0/553465333 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5606c74e2e60 0x5606c74e52d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:57.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.547+0000 7f3e57fd1740 4 client.27509 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:41:57.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.547+0000 7f3e57fd1740 1 -- 172.21.15.90:0/553465333 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5606c73ce630 con 0x5606c74e2e60 2024-04-02T14:41:57.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.548+0000 7f3e47e00640 1 --2- 172.21.15.90:0/553465333 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5606c74e2e60 0x5606c74e52d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:57.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.549+0000 7f3e47e00640 1 --2- 172.21.15.90:0/553465333 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5606c74e2e60 0x5606c74e52d0 crc :-1 s=READY pgs=192 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=99f9f206691bb2dd server_cookie=29eb01a55fc6d8a3 in_seq=0 out_seq=0 2024-04-02T14:41:57.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.549+0000 7f3e33e00640 10 client.27509 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:41:57.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.550+0000 7f3e33e00640 1 -- 172.21.15.90:0/553465333 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5606c73ce630 con 0x5606c74e2e60 2024-04-02T14:41:57.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.550+0000 7f3e33e00640 10 client.27509 handle_command_reply: tid=0 2024-04-02T14:41:57.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.550+0000 7f3e57fd1740 1 client.27509 shutdown 2024-04-02T14:41:57.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.551+0000 7f3e57fd1740 1 -- 172.21.15.90:0/553465333 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5606c74c97b0 msgr2=0x5606c74cfde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:41:57.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.551+0000 7f3e57fd1740 1 --2- 172.21.15.90:0/553465333 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5606c74c97b0 0x5606c74cfde0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f3e40009900 tx=0x7f3e4002bc30 comp rx=0 tx=0).stop 2024-04-02T14:41:57.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.551+0000 7f3e57fd1740 1 -- 172.21.15.90:0/553465333 shutdown_connections 2024-04-02T14:41:57.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.551+0000 7f3e57fd1740 1 --2- 172.21.15.90:0/553465333 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5606c74e2e60 0x5606c74e52d0 crc :-1 s=READY pgs=192 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:57.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.551+0000 7f3e57fd1740 1 --2- 172.21.15.90:0/553465333 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5606c74d2cb0 0x5606c74d0860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:57.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.551+0000 7f3e57fd1740 1 --2- 172.21.15.90:0/553465333 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5606c74ca0d0 0x5606c74d0320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:57.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.551+0000 7f3e57fd1740 1 --2- 172.21.15.90:0/553465333 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5606c74c97b0 0x5606c74cfde0 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:57.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.551+0000 7f3e57fd1740 1 -- 172.21.15.90:0/553465333 >> 172.21.15.90:0/553465333 conn(0x5606c74c3ee0 msgr2=0x5606c74c5c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:41:57.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.551+0000 7f3e33e00640 0 client.27509 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:41:57.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.552+0000 7f3e57fd1740 1 -- 172.21.15.90:0/553465333 shutdown_connections 2024-04-02T14:41:57.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.552+0000 7f3e57fd1740 1 -- 172.21.15.90:0/553465333 wait complete. 2024-04-02T14:41:57.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.552+0000 7f3e57fd1740 20 client.27509 trim_cache size 0 max 16384 2024-04-02T14:41:57.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.586+0000 7f3e57fd1740 1 Processor -- start 2024-04-02T14:41:57.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.586+0000 7f3e57fd1740 1 -- start start 2024-04-02T14:41:57.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.586+0000 7f3e57fd1740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5606c74c97b0 0x5606c74d2510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:57.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.586+0000 7f3e57fd1740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5606c74ca0d0 0x5606c74de9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:57.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.586+0000 7f3e57fd1740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5606c74d2cb0 0x5606c74c2de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:57.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.586+0000 7f3e57fd1740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f3e501b3f30 con 0x5606c74c97b0 2024-04-02T14:41:57.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.586+0000 7f3e57fd1740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f3e501b5a70 con 0x5606c74ca0d0 2024-04-02T14:41:57.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.586+0000 7f3e57fd1740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f3e501b4600 con 0x5606c74d2cb0 2024-04-02T14:41:57.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.586+0000 7f3e47e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5606c74ca0d0 0x5606c74de9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:57.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.586+0000 7f3e47e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5606c74ca0d0 0x5606c74de9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:36468/0 (socket says 172.21.15.90:36468) 2024-04-02T14:41:57.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.586+0000 7f3e47e00640 1 -- 172.21.15.90:0/3622314961 learned_addr learned my addr 172.21.15.90:0/3622314961 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:41:57.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.587+0000 7f3e4e800640 1 --2- 172.21.15.90:0/3622314961 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5606c74d2cb0 0x5606c74c2de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:57.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.587+0000 7f3e4de00640 1 --2- 172.21.15.90:0/3622314961 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5606c74c97b0 0x5606c74d2510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:57.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.587+0000 7f3e4e800640 1 -- 172.21.15.90:0/3622314961 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5606c74ca0d0 msgr2=0x5606c74de9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:41:57.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.587+0000 7f3e4e800640 1 --2- 172.21.15.90:0/3622314961 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5606c74ca0d0 0x5606c74de9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:57.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.587+0000 7f3e4e800640 1 -- 172.21.15.90:0/3622314961 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5606c74c97b0 msgr2=0x5606c74d2510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:41:57.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.587+0000 7f3e4e800640 1 --2- 172.21.15.90:0/3622314961 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5606c74c97b0 0x5606c74d2510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:57.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.587+0000 7f3e4e800640 1 -- 172.21.15.90:0/3622314961 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5606c74e5640 con 0x5606c74d2cb0 2024-04-02T14:41:57.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.587+0000 7f3e4e800640 1 --2- 172.21.15.90:0/3622314961 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5606c74d2cb0 0x5606c74c2de0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f3e48009e40 tx=0x7f3e48005eb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d466de3809764e8c server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:41:57.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.588+0000 7f3e3ea00640 1 -- 172.21.15.90:0/3622314961 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e48025450 con 0x5606c74d2cb0 2024-04-02T14:41:57.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.588+0000 7f3e3ea00640 1 -- 172.21.15.90:0/3622314961 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3e48025ad0 con 0x5606c74d2cb0 2024-04-02T14:41:57.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.588+0000 7f3e3ea00640 1 -- 172.21.15.90:0/3622314961 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e48011af0 con 0x5606c74d2cb0 2024-04-02T14:41:57.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.589+0000 7f3e57fd1740 1 -- 172.21.15.90:0/3622314961 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5606c74d2cb0 msgr2=0x5606c74c2de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:41:57.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.589+0000 7f3e57fd1740 1 --2- 172.21.15.90:0/3622314961 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5606c74d2cb0 0x5606c74c2de0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f3e48009e40 tx=0x7f3e48005eb0 comp rx=0 tx=0).stop 2024-04-02T14:41:57.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.589+0000 7f3e57fd1740 1 -- 172.21.15.90:0/3622314961 shutdown_connections 2024-04-02T14:41:57.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.589+0000 7f3e57fd1740 1 --2- 172.21.15.90:0/3622314961 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5606c74d2cb0 0x5606c74c2de0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:57.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.589+0000 7f3e57fd1740 1 --2- 172.21.15.90:0/3622314961 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5606c74ca0d0 0x5606c74de9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:57.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.589+0000 7f3e57fd1740 1 --2- 172.21.15.90:0/3622314961 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5606c74c97b0 0x5606c74d2510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:57.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.589+0000 7f3e57fd1740 1 -- 172.21.15.90:0/3622314961 >> 172.21.15.90:0/3622314961 conn(0x5606c74c3ee0 msgr2=0x5606c74c26d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:41:57.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.589+0000 7f3e57fd1740 1 -- 172.21.15.90:0/3622314961 shutdown_connections 2024-04-02T14:41:57.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.589+0000 7f3e57fd1740 1 -- 172.21.15.90:0/3622314961 wait complete. 2024-04-02T14:41:57.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.590+0000 7f3e57fd1740 1 -- start start 2024-04-02T14:41:57.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.590+0000 7f3e57fd1740 1 Processor -- start 2024-04-02T14:41:57.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.590+0000 7f3e57fd1740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5606c74c97b0 0x5606c74ccc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:57.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.590+0000 7f3e57fd1740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5606c74ca0d0 0x5606c74cd150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:57.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.590+0000 7f3e57fd1740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5606c74d2cb0 0x5606c74cd690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:57.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.590+0000 7f3e57fd1740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f3e501b53a0 con 0x5606c74c97b0 2024-04-02T14:41:57.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.590+0000 7f3e57fd1740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f3e501b4cd0 con 0x5606c74ca0d0 2024-04-02T14:41:57.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.590+0000 7f3e57fd1740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f3e501b1f10 con 0x5606c74d2cb0 2024-04-02T14:41:57.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.591+0000 7f3e57fd1740 10 client.0 _ll_register_callbacks cb 0x7f3e40009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:41:57.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.591+0000 7f3e4de00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5606c74c97b0 0x5606c74ccc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:57.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.591+0000 7f3e4de00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5606c74c97b0 0x5606c74ccc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:37996/0 (socket says 172.21.15.90:37996) 2024-04-02T14:41:57.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.591+0000 7f3e4de00640 1 -- 172.21.15.90:0/4099668613 learned_addr learned my addr 172.21.15.90:0/4099668613 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:41:57.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.591+0000 7f3e4e800640 1 --2- 172.21.15.90:0/4099668613 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5606c74d2cb0 0x5606c74cd690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:57.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.591+0000 7f3e47e00640 1 --2- 172.21.15.90:0/4099668613 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5606c74ca0d0 0x5606c74cd150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:57.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.592+0000 7f3e4de00640 1 -- 172.21.15.90:0/4099668613 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5606c74d2cb0 msgr2=0x5606c74cd690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:41:57.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.592+0000 7f3e4de00640 1 --2- 172.21.15.90:0/4099668613 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5606c74d2cb0 0x5606c74cd690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:57.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.592+0000 7f3e4de00640 1 -- 172.21.15.90:0/4099668613 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5606c74ca0d0 msgr2=0x5606c74cd150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:41:57.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.592+0000 7f3e4de00640 1 --2- 172.21.15.90:0/4099668613 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5606c74ca0d0 0x5606c74cd150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:57.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.592+0000 7f3e4de00640 1 -- 172.21.15.90:0/4099668613 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5606c7548590 con 0x5606c74c97b0 2024-04-02T14:41:57.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.592+0000 7f3e47e00640 1 --2- 172.21.15.90:0/4099668613 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5606c74ca0d0 0x5606c74cd150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:41:57.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.592+0000 7f3e4e800640 1 --2- 172.21.15.90:0/4099668613 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5606c74d2cb0 0x5606c74cd690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:41:57.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.592+0000 7f3e4de00640 1 --2- 172.21.15.90:0/4099668613 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5606c74c97b0 0x5606c74ccc10 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f3e4002c240 tx=0x7f3e4004b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96fdb44b789b26bf server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:41:57.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.592+0000 7f3e33e00640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-02T14:41:57.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.593+0000 7f3e33e00640 1 -- 172.21.15.90:0/4099668613 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e40063070 con 0x5606c74c97b0 2024-04-02T14:41:57.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.593+0000 7f3e33e00640 1 -- 172.21.15.90:0/4099668613 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f3e400046c0 con 0x5606c74c97b0 2024-04-02T14:41:57.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.593+0000 7f3e57fd1740 1 -- 172.21.15.90:0/4099668613 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3e48025ad0 con 0x5606c74c97b0 2024-04-02T14:41:57.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.593+0000 7f3e33e00640 1 -- 172.21.15.90:0/4099668613 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f3e4006d050 con 0x5606c74c97b0 2024-04-02T14:41:57.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.593+0000 7f3e57fd1740 10 client.16809 fetch_fsmap learned FSMap version 27 2024-04-02T14:41:57.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.594+0000 7f3e57fd1740 1 -- 172.21.15.90:0/4099668613 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5606c7549860 con 0x5606c74c97b0 2024-04-02T14:41:57.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.594+0000 7f3e33e00640 1 -- 172.21.15.90:0/4099668613 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3e48025ad0 con 0x5606c74c97b0 2024-04-02T14:41:57.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.594+0000 7f3e33e00640 1 -- 172.21.15.90:0/4099668613 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f3e40003a80 con 0x5606c74c97b0 2024-04-02T14:41:57.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.594+0000 7f3e57fd1740 10 client.16809 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:41:57.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.594+0000 7f3e57fd1740 10 client.16809 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:41:57.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.594+0000 7f3e57fd1740 20 client.16809 populate_metadata read hostname 'smithi090' 2024-04-02T14:41:57.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.594+0000 7f3e57fd1740 1 --2- 172.21.15.90:0/4099668613 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5606c7549fb0 0x5606c754c3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:57.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.594+0000 7f3e57fd1740 4 client.16809 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-02T14:41:57.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.594+0000 7f3e57fd1740 1 -- 172.21.15.90:0/4099668613 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5606c743acb0 con 0x5606c7549fb0 2024-04-02T14:41:57.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.595+0000 7f3e47e00640 1 --2- 172.21.15.90:0/4099668613 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5606c7549fb0 0x5606c754c3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:57.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.596+0000 7f3e33e00640 10 client.16809 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:41:57.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.596+0000 7f3e47e00640 1 --2- 172.21.15.90:0/4099668613 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5606c7549fb0 0x5606c754c3d0 crc :-1 s=READY pgs=193 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=73f0a69628d0a887 server_cookie=2b022bf58b2004e1 in_seq=0 out_seq=0 2024-04-02T14:41:57.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.597+0000 7f3e33e00640 1 -- 172.21.15.90:0/4099668613 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5606c743acb0 con 0x5606c7549fb0 2024-04-02T14:41:57.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.597+0000 7f3e33e00640 10 client.16809 handle_command_reply: tid=0 2024-04-02T14:41:57.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.597+0000 7f3e57fd1740 1 client.16809 shutdown 2024-04-02T14:41:57.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.597+0000 7f3e57fd1740 1 -- 172.21.15.90:0/4099668613 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5606c74c97b0 msgr2=0x5606c74ccc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:41:57.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.597+0000 7f3e57fd1740 1 --2- 172.21.15.90:0/4099668613 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5606c74c97b0 0x5606c74ccc10 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f3e4002c240 tx=0x7f3e4004b000 comp rx=0 tx=0).stop 2024-04-02T14:41:57.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.598+0000 7f3e57fd1740 1 -- 172.21.15.90:0/4099668613 shutdown_connections 2024-04-02T14:41:57.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.598+0000 7f3e57fd1740 1 --2- 172.21.15.90:0/4099668613 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5606c7549fb0 0x5606c754c3d0 crc :-1 s=READY pgs=193 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:57.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.598+0000 7f3e57fd1740 1 --2- 172.21.15.90:0/4099668613 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5606c74d2cb0 0x5606c74cd690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:57.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.598+0000 7f3e57fd1740 1 --2- 172.21.15.90:0/4099668613 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5606c74ca0d0 0x5606c74cd150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:57.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.598+0000 7f3e33e00640 0 client.16809 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:41:57.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.598+0000 7f3e57fd1740 1 --2- 172.21.15.90:0/4099668613 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5606c74c97b0 0x5606c74ccc10 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:57.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.598+0000 7f3e57fd1740 1 -- 172.21.15.90:0/4099668613 >> 172.21.15.90:0/4099668613 conn(0x5606c74c3ee0 msgr2=0x5606c74c5b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:41:57.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.598+0000 7f3e57fd1740 1 -- 172.21.15.90:0/4099668613 shutdown_connections 2024-04-02T14:41:57.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.598+0000 7f3e57fd1740 1 -- 172.21.15.90:0/4099668613 wait complete. 2024-04-02T14:41:57.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.598+0000 7f3e57fd1740 20 client.16809 trim_cache size 0 max 16384 2024-04-02T14:41:57.599 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-02T14:41:57.599 INFO:teuthology.orchestra.run.smithi090.stdout: "status": "no active scrubs running", 2024-04-02T14:41:57.599 INFO:teuthology.orchestra.run.smithi090.stdout: "scrubs": {} 2024-04-02T14:41:57.599 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-02T14:41:57.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.600+0000 7f3e55200640 1 -- 172.21.15.90:0/258093501 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f3e340a3da0 msgr2=0x7f3e340a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:41:57.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.600+0000 7f3e55200640 1 --2- 172.21.15.90:0/258093501 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f3e340a3da0 0x7f3e340a6260 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f3e40009230 tx=0x7f3e4002df90 comp rx=0 tx=0).stop 2024-04-02T14:41:57.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.600+0000 7f3e55200640 1 -- 172.21.15.90:0/258093501 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3e5010d3e0 msgr2=0x7f3e501a4c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:41:57.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.600+0000 7f3e55200640 1 --2- 172.21.15.90:0/258093501 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3e5010d3e0 0x7f3e501a4c00 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f3e38005fd0 tx=0x7f3e3800dde0 comp rx=0 tx=0).stop 2024-04-02T14:41:57.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.600+0000 7f3e55200640 1 -- 172.21.15.90:0/258093501 shutdown_connections 2024-04-02T14:41:57.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.600+0000 7f3e55200640 1 --2- 172.21.15.90:0/258093501 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f3e340a3da0 0x7f3e340a6260 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:57.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.600+0000 7f3e55200640 1 --2- 172.21.15.90:0/258093501 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3e5019e830 0x7f3e5019ece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:57.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.600+0000 7f3e55200640 1 --2- 172.21.15.90:0/258093501 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3e5010d3e0 0x7f3e501a4c00 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:57.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.600+0000 7f3e55200640 1 --2- 172.21.15.90:0/258093501 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3e50108310 0x7f3e501a46c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:57.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.600+0000 7f3e55200640 1 -- 172.21.15.90:0/258093501 >> 172.21.15.90:0/258093501 conn(0x7f3e50076040 msgr2=0x7f3e50078390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:41:57.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.601+0000 7f3e55200640 1 -- 172.21.15.90:0/258093501 shutdown_connections 2024-04-02T14:41:57.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.601+0000 7f3e55200640 1 -- 172.21.15.90:0/258093501 wait complete. 2024-04-02T14:41:57.631 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-02T14:41:57.631 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T14:41:57.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.930+0000 7f1097400640 1 -- 172.21.15.90:0/3442728381 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1090112940 msgr2=0x7f1090114d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:41:57.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.930+0000 7f1097400640 1 --2- 172.21.15.90:0/3442728381 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1090112940 0x7f1090114d30 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f1088009cd0 tx=0x7f108802c100 comp rx=0 tx=0).stop 2024-04-02T14:41:57.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.930+0000 7f1097400640 1 -- 172.21.15.90:0/3442728381 shutdown_connections 2024-04-02T14:41:57.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.930+0000 7f1097400640 1 --2- 172.21.15.90:0/3442728381 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1090112940 0x7f1090114d30 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:57.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.931+0000 7f1097400640 1 --2- 172.21.15.90:0/3442728381 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1090076950 0x7f1090112400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:57.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.931+0000 7f1097400640 1 --2- 172.21.15.90:0/3442728381 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1090076030 0x7f1090076410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:57.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.931+0000 7f1097400640 1 -- 172.21.15.90:0/3442728381 >> 172.21.15.90:0/3442728381 conn(0x7f10900ff750 msgr2=0x7f1090101b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:41:57.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.932+0000 7f1097400640 1 -- 172.21.15.90:0/3442728381 shutdown_connections 2024-04-02T14:41:57.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.932+0000 7f1097400640 1 -- 172.21.15.90:0/3442728381 wait complete. 2024-04-02T14:41:57.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.933+0000 7f1097400640 1 Processor -- start 2024-04-02T14:41:57.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.934+0000 7f1097400640 1 -- start start 2024-04-02T14:41:57.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.934+0000 7f1097400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1090076030 0x7f1090111de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:57.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.934+0000 7f1097400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1090076950 0x7f1090112320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:57.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.935+0000 7f1097400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1090112940 0x7f109010af60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:57.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.935+0000 7f1097400640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f1090118460 con 0x7f1090076950 2024-04-02T14:41:57.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.935+0000 7f1097400640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f10901182e0 con 0x7f1090112940 2024-04-02T14:41:57.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.935+0000 7f1097400640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f10901185e0 con 0x7f1090076030 2024-04-02T14:41:57.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.935+0000 7f1094a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1090076030 0x7f1090111de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:57.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.935+0000 7f1094a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1090076030 0x7f1090111de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:56398/0 (socket says 172.21.15.90:56398) 2024-04-02T14:41:57.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.935+0000 7f1094a00640 1 -- 172.21.15.90:0/571773655 learned_addr learned my addr 172.21.15.90:0/571773655 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:41:57.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.935+0000 7f108fe00640 1 --2- 172.21.15.90:0/571773655 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1090076950 0x7f1090112320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:57.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.935+0000 7f1095400640 1 --2- 172.21.15.90:0/571773655 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1090112940 0x7f109010af60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:57.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.936+0000 7f1094a00640 1 -- 172.21.15.90:0/571773655 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1090112940 msgr2=0x7f109010af60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:41:57.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.936+0000 7f1094a00640 1 --2- 172.21.15.90:0/571773655 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1090112940 0x7f109010af60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:57.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.936+0000 7f1094a00640 1 -- 172.21.15.90:0/571773655 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1090076950 msgr2=0x7f1090112320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:41:57.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.936+0000 7f1094a00640 1 --2- 172.21.15.90:0/571773655 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1090076950 0x7f1090112320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:57.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.936+0000 7f1094a00640 1 -- 172.21.15.90:0/571773655 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f109010b820 con 0x7f1090076030 2024-04-02T14:41:57.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.936+0000 7f108fe00640 1 --2- 172.21.15.90:0/571773655 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1090076950 0x7f1090112320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:41:57.937 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:41:57 smithi090 ceph-mon[31674]: pgmap v215: 97 pgs: 97 active+clean; 4.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 5.9 MiB/s wr, 592 op/s 2024-04-02T14:41:57.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.937+0000 7f1094a00640 1 --2- 172.21.15.90:0/571773655 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1090076030 0x7f1090111de0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f108400c920 tx=0x7f108400cde0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:41:57.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.938+0000 7f108d600640 1 -- 172.21.15.90:0/571773655 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1084007c70 con 0x7f1090076030 2024-04-02T14:41:57.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.938+0000 7f108d600640 1 -- 172.21.15.90:0/571773655 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1084007e10 con 0x7f1090076030 2024-04-02T14:41:57.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.938+0000 7f108d600640 1 -- 172.21.15.90:0/571773655 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1084018890 con 0x7f1090076030 2024-04-02T14:41:57.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.938+0000 7f1097400640 1 -- 172.21.15.90:0/571773655 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f109010bb10 con 0x7f1090076030 2024-04-02T14:41:57.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.938+0000 7f1097400640 1 -- 172.21.15.90:0/571773655 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f109010fae0 con 0x7f1090076030 2024-04-02T14:41:57.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.940+0000 7f1097400640 1 -- 172.21.15.90:0/571773655 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1090077770 con 0x7f1090076030 2024-04-02T14:41:57.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.946+0000 7f108d600640 1 -- 172.21.15.90:0/571773655 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f1084018a70 con 0x7f1090076030 2024-04-02T14:41:57.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.948+0000 7f108d600640 1 --2- 172.21.15.90:0/571773655 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f10780a3da0 0x7f10780a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:57.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.948+0000 7f108d600640 1 -- 172.21.15.90:0/571773655 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f10840d08d0 con 0x7f1090076030 2024-04-02T14:41:57.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.948+0000 7f108fe00640 1 --2- 172.21.15.90:0/571773655 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f10780a3da0 0x7f10780a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:57.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.949+0000 7f108fe00640 1 --2- 172.21.15.90:0/571773655 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f10780a3da0 0x7f10780a6260 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f107c005e10 tx=0x7f107c005da0 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:41:57.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:57.951+0000 7f108d600640 1 -- 172.21.15.90:0/571773655 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7f1084014030 con 0x7f1090076030 2024-04-02T14:41:58.108 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:41:57 smithi092 ceph-mon[36497]: pgmap v215: 97 pgs: 97 active+clean; 4.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 5.9 MiB/s wr, 592 op/s 2024-04-02T14:41:58.184 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:41:57 smithi177 ceph-mon[36431]: pgmap v215: 97 pgs: 97 active+clean; 4.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 5.9 MiB/s wr, 592 op/s 2024-04-02T14:41:58.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.377+0000 7f1097400640 1 -- 172.21.15.90:0/571773655 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f109010fe40 con 0x7f1090076030 2024-04-02T14:41:58.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.378+0000 7f108d600640 1 -- 172.21.15.90:0/571773655 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11927 (secure 0 0 0) 0x7f108409cf80 con 0x7f1090076030 2024-04-02T14:41:58.378 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-02T14:41:58.378 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14412,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6837/2852160309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6836","nonce":2852160309},{"type":"v1","addr":"172.21.15.90:6837","nonce":2852160309}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14478,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6841/1919376858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6840","nonce":1919376858},{"type":"v1","addr":"172.21.15.90:6841","nonce":1919376858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-02T14:32:22.319083+0000","modified":"2024-04-02T14:34:52.266152+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24590,"mds_1":24560,"mds_2":24547,"mds_3":24539,"mds_4":24523},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24454":{"gid":24454,"name":"l","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6833/1564319431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":1564319431},{"type":"v1","addr":"172.21.15.177:6833","nonce":1564319431}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.92:6833/1570337372","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":1570337372},{"type":"v1","addr":"172.21.15.92:6833","nonce":1570337372}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"j","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.90:6835/1277124891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6834","nonce":1277124891},{"type":"v1","addr":"172.21.15.90:6835","nonce":1277124891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.92:6835/860721164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":860721164},{"type":"v1","addr":"172.21.15.92:6835","nonce":860721164}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24511":{"gid":24511,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6835/3177244988","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":3177244988},{"type":"v1","addr":"172.21.15.177:6835","nonce":3177244988}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"c","rank":4,"incarnation":25,"state":"up:active","state_seq":9,"addr":"172.21.15.177:6837/4223414858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":4223414858},{"type":"v1","addr":"172.21.15.177:6837","nonce":4223414858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24539":{"gid":24539,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":8,"addr":"172.21.15.92:6837/1270635939","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":1270635939},{"type":"v1","addr":"172.21.15.92:6837","nonce":1270635939}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24547":{"gid":24547,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":7,"addr":"172.21.15.177:6839/664895663","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":664895663},{"type":"v1","addr":"172.21.15.177:6839","nonce":664895663}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"g","rank":1,"incarnation":19,"state":"up:active","state_seq":5,"addr":"172.21.15.90:6839/1498915870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6838","nonce":1498915870},{"type":"v1","addr":"172.21.15.90:6839","nonce":1498915870}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24590":{"gid":24590,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.92:6839/1054888219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":1054888219},{"type":"v1","addr":"172.21.15.92:6839","nonce":1054888219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24590,"qdb_cluster":[24547,24590,24523,24539,24560]},"id":1}]} 2024-04-02T14:41:58.378 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 27 2024-04-02T14:41:58.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.382+0000 7f1097400640 1 -- 172.21.15.90:0/571773655 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f10780a3da0 msgr2=0x7f10780a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:41:58.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.382+0000 7f1097400640 1 --2- 172.21.15.90:0/571773655 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f10780a3da0 0x7f10780a6260 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f107c005e10 tx=0x7f107c005da0 comp rx=0 tx=0).stop 2024-04-02T14:41:58.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.382+0000 7f1097400640 1 -- 172.21.15.90:0/571773655 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1090076030 msgr2=0x7f1090111de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:41:58.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.382+0000 7f1097400640 1 --2- 172.21.15.90:0/571773655 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1090076030 0x7f1090111de0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f108400c920 tx=0x7f108400cde0 comp rx=0 tx=0).stop 2024-04-02T14:41:58.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.382+0000 7f1097400640 1 -- 172.21.15.90:0/571773655 shutdown_connections 2024-04-02T14:41:58.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.382+0000 7f1097400640 1 --2- 172.21.15.90:0/571773655 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f10780a3da0 0x7f10780a6260 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:58.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.382+0000 7f1097400640 1 --2- 172.21.15.90:0/571773655 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1090112940 0x7f109010af60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:58.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.382+0000 7f1097400640 1 --2- 172.21.15.90:0/571773655 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1090076950 0x7f1090112320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:58.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.382+0000 7f1097400640 1 --2- 172.21.15.90:0/571773655 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1090076030 0x7f1090111de0 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:58.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.382+0000 7f1097400640 1 -- 172.21.15.90:0/571773655 >> 172.21.15.90:0/571773655 conn(0x7f10900ff750 msgr2=0x7f1090114330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:41:58.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.382+0000 7f1097400640 1 -- 172.21.15.90:0/571773655 shutdown_connections 2024-04-02T14:41:58.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.382+0000 7f1097400640 1 -- 172.21.15.90:0/571773655 wait complete. 2024-04-02T14:41:58.407 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-04-02T14:41:58.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.707+0000 7f4287800640 1 -- 172.21.15.90:0/1094762265 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4280109420 msgr2=0x7f4280114710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:41:58.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.707+0000 7f4287800640 1 --2- 172.21.15.90:0/1094762265 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4280109420 0x7f4280114710 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f4268009900 tx=0x7f426802bcd0 comp rx=0 tx=0).stop 2024-04-02T14:41:58.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.708+0000 7f4287800640 1 -- 172.21.15.90:0/1094762265 shutdown_connections 2024-04-02T14:41:58.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.709+0000 7f4287800640 1 --2- 172.21.15.90:0/1094762265 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4280114c50 0x7f4280117040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:58.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.709+0000 7f4287800640 1 --2- 172.21.15.90:0/1094762265 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4280109420 0x7f4280114710 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:58.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.709+0000 7f4287800640 1 --2- 172.21.15.90:0/1094762265 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4280108b00 0x7f4280108ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:58.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.709+0000 7f4287800640 1 -- 172.21.15.90:0/1094762265 >> 172.21.15.90:0/1094762265 conn(0x7f428006d040 msgr2=0x7f428006d450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:41:58.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.709+0000 7f4287800640 1 -- 172.21.15.90:0/1094762265 shutdown_connections 2024-04-02T14:41:58.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.710+0000 7f4287800640 1 -- 172.21.15.90:0/1094762265 wait complete. 2024-04-02T14:41:58.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.711+0000 7f4287800640 1 Processor -- start 2024-04-02T14:41:58.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.712+0000 7f4287800640 1 -- start start 2024-04-02T14:41:58.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.713+0000 7f4287800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4280108b00 0x7f4280114050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:58.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.713+0000 7f4287800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4280109420 0x7f4280114590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:58.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.713+0000 7f4287800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4280114c50 0x7f428010b090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:58.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.713+0000 7f4287800640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f428011a6b0 con 0x7f4280108b00 2024-04-02T14:41:58.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.713+0000 7f4287800640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f428011a530 con 0x7f4280114c50 2024-04-02T14:41:58.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.713+0000 7f4287800640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f428011a830 con 0x7f4280109420 2024-04-02T14:41:58.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.713+0000 7f427fe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4280109420 0x7f4280114590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:58.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.713+0000 7f427fe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4280109420 0x7f4280114590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:56418/0 (socket says 172.21.15.90:56418) 2024-04-02T14:41:58.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.713+0000 7f427fe00640 1 -- 172.21.15.90:0/1466843201 learned_addr learned my addr 172.21.15.90:0/1466843201 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:41:58.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.713+0000 7f4285800640 1 --2- 172.21.15.90:0/1466843201 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4280114c50 0x7f428010b090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:58.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.714+0000 7f4284e00640 1 --2- 172.21.15.90:0/1466843201 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4280108b00 0x7f4280114050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:58.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.714+0000 7f427fe00640 1 -- 172.21.15.90:0/1466843201 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4280114c50 msgr2=0x7f428010b090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:41:58.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.714+0000 7f427fe00640 1 --2- 172.21.15.90:0/1466843201 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4280114c50 0x7f428010b090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:58.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.714+0000 7f427fe00640 1 -- 172.21.15.90:0/1466843201 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4280108b00 msgr2=0x7f4280114050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:41:58.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.714+0000 7f427fe00640 1 --2- 172.21.15.90:0/1466843201 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4280108b00 0x7f4280114050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:58.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.714+0000 7f427fe00640 1 -- 172.21.15.90:0/1466843201 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f428010b950 con 0x7f4280109420 2024-04-02T14:41:58.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.714+0000 7f4285800640 1 --2- 172.21.15.90:0/1466843201 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4280114c50 0x7f428010b090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:41:58.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.714+0000 7f4284e00640 1 --2- 172.21.15.90:0/1466843201 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4280108b00 0x7f4280114050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:41:58.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.715+0000 7f427fe00640 1 --2- 172.21.15.90:0/1466843201 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4280109420 0x7f4280114590 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f426802c2e0 tx=0x7f426802c7b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:41:58.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.715+0000 7f427d600640 1 -- 172.21.15.90:0/1466843201 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4268043020 con 0x7f4280109420 2024-04-02T14:41:58.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.715+0000 7f427d600640 1 -- 172.21.15.90:0/1466843201 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f42680054e0 con 0x7f4280109420 2024-04-02T14:41:58.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.715+0000 7f427d600640 1 -- 172.21.15.90:0/1466843201 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4268003cf0 con 0x7f4280109420 2024-04-02T14:41:58.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.717+0000 7f4287800640 1 -- 172.21.15.90:0/1466843201 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f428010bbe0 con 0x7f4280109420 2024-04-02T14:41:58.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.717+0000 7f4287800640 1 -- 172.21.15.90:0/1466843201 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f428010fc70 con 0x7f4280109420 2024-04-02T14:41:58.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.723+0000 7f428a57b740 1 Processor -- start 2024-04-02T14:41:58.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.724+0000 7f428a57b740 1 -- start start 2024-04-02T14:41:58.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.724+0000 7f428a57b740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5609b8ce19f0 0x5609b8ce0740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:58.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.724+0000 7f428a57b740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5609b8ce0d10 0x5609b8ce1190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:58.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.724+0000 7f428a57b740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5609b8cea080 0x5609b8cec470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:58.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.724+0000 7f428a57b740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5609b8c21400 con 0x5609b8cea080 2024-04-02T14:41:58.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.724+0000 7f428a57b740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x5609b8c05f10 con 0x5609b8ce19f0 2024-04-02T14:41:58.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.724+0000 7f428a57b740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5609b8c456f0 con 0x5609b8ce0d10 2024-04-02T14:41:58.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.725+0000 7f4285800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5609b8ce19f0 0x5609b8ce0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:58.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.725+0000 7f4285800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5609b8ce19f0 0x5609b8ce0740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:36510/0 (socket says 172.21.15.90:36510) 2024-04-02T14:41:58.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.725+0000 7f4285800640 1 -- 172.21.15.90:0/3747983827 learned_addr learned my addr 172.21.15.90:0/3747983827 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:41:58.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.725+0000 7f4284e00640 1 --2- 172.21.15.90:0/3747983827 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5609b8ce0d10 0x5609b8ce1190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:58.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.726+0000 7f4285800640 1 -- 172.21.15.90:0/3747983827 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5609b8ce0d10 msgr2=0x5609b8ce1190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:41:58.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.726+0000 7f4285800640 1 --2- 172.21.15.90:0/3747983827 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5609b8ce0d10 0x5609b8ce1190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:58.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.726+0000 7f4285800640 1 -- 172.21.15.90:0/3747983827 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5609b8cea080 msgr2=0x5609b8cec470 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-02T14:41:58.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.726+0000 7f427d600640 1 -- 172.21.15.90:0/1466843201 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f4268046020 con 0x7f4280109420 2024-04-02T14:41:58.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.727+0000 7f427fe00640 1 --2- 172.21.15.90:0/3747983827 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5609b8cea080 0x5609b8cec470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:58.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.727+0000 7f4285800640 1 --2- 172.21.15.90:0/3747983827 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5609b8cea080 0x5609b8cec470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:58.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.727+0000 7f4285800640 1 -- 172.21.15.90:0/3747983827 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5609b89593f0 con 0x5609b8ce19f0 2024-04-02T14:41:58.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.728+0000 7f4285800640 1 --2- 172.21.15.90:0/3747983827 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5609b8ce19f0 0x5609b8ce0740 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f427800bab0 tx=0x7f427800bf70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=81148eb9d4fc7e11 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:41:58.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.730+0000 7f427d600640 1 --2- 172.21.15.90:0/1466843201 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f42640a3e20 0x7f42640a62e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:58.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.730+0000 7f427d600640 1 -- 172.21.15.90:0/1466843201 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f42680f2420 con 0x7f4280109420 2024-04-02T14:41:58.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.730+0000 7f4276000640 1 -- 172.21.15.90:0/3747983827 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4278017420 con 0x5609b8ce19f0 2024-04-02T14:41:58.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.730+0000 7f4276000640 1 -- 172.21.15.90:0/3747983827 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4278017aa0 con 0x5609b8ce19f0 2024-04-02T14:41:58.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.730+0000 7f4276000640 1 -- 172.21.15.90:0/3747983827 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f427800fb60 con 0x5609b8ce19f0 2024-04-02T14:41:58.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.731+0000 7f4284e00640 1 --2- 172.21.15.90:0/1466843201 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f42640a3e20 0x7f42640a62e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:58.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.732+0000 7f428a57b740 1 -- 172.21.15.90:0/3747983827 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5609b8ce19f0 msgr2=0x5609b8ce0740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:41:58.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.732+0000 7f4284e00640 1 --2- 172.21.15.90:0/1466843201 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f42640a3e20 0x7f42640a62e0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x5609b8ce1f40 tx=0x7f4270008590 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:41:58.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.732+0000 7f428a57b740 1 --2- 172.21.15.90:0/3747983827 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5609b8ce19f0 0x5609b8ce0740 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f427800bab0 tx=0x7f427800bf70 comp rx=0 tx=0).stop 2024-04-02T14:41:58.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.732+0000 7f428a57b740 1 -- 172.21.15.90:0/3747983827 shutdown_connections 2024-04-02T14:41:58.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.732+0000 7f428a57b740 1 --2- 172.21.15.90:0/3747983827 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5609b8cea080 0x5609b8cec470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:58.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.732+0000 7f428a57b740 1 --2- 172.21.15.90:0/3747983827 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5609b8ce0d10 0x5609b8ce1190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:58.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.732+0000 7f428a57b740 1 --2- 172.21.15.90:0/3747983827 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5609b8ce19f0 0x5609b8ce0740 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:58.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.732+0000 7f428a57b740 1 -- 172.21.15.90:0/3747983827 >> 172.21.15.90:0/3747983827 conn(0x5609b8ca7a50 msgr2=0x5609b8cdb760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:41:58.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.732+0000 7f428a57b740 1 -- 172.21.15.90:0/3747983827 shutdown_connections 2024-04-02T14:41:58.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.732+0000 7f428a57b740 1 -- 172.21.15.90:0/3747983827 wait complete. 2024-04-02T14:41:58.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.733+0000 7f428a57b740 1 -- start start 2024-04-02T14:41:58.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.733+0000 7f428a57b740 1 Processor -- start 2024-04-02T14:41:58.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.733+0000 7f428a57b740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5609b8ce0d10 0x5609b8ce7090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:58.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.734+0000 7f428a57b740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5609b8ce19f0 0x5609b8ce75d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:58.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.734+0000 7f428a57b740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5609b8cea080 0x5609b8ce7b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:58.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.734+0000 7f428a57b740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5609b8c416a0 con 0x5609b8ce19f0 2024-04-02T14:41:58.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.734+0000 7f428a57b740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x5609b8bed980 con 0x5609b8ce0d10 2024-04-02T14:41:58.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.734+0000 7f428a57b740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5609b8c00cb0 con 0x5609b8cea080 2024-04-02T14:41:58.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.734+0000 7f428a57b740 10 client.0 _ll_register_callbacks cb 0x5609b8c006a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:41:58.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.734+0000 7f4285800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5609b8cea080 0x5609b8ce7b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:58.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.734+0000 7f4285800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5609b8cea080 0x5609b8ce7b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:56436/0 (socket says 172.21.15.90:56436) 2024-04-02T14:41:58.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.734+0000 7f4285800640 1 -- 172.21.15.90:0/3288092750 learned_addr learned my addr 172.21.15.90:0/3288092750 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:41:58.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.734+0000 7f4284e00640 1 --2- 172.21.15.90:0/3288092750 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5609b8ce0d10 0x5609b8ce7090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:58.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.735+0000 7f427fe00640 1 --2- 172.21.15.90:0/3288092750 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5609b8ce19f0 0x5609b8ce75d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:58.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.735+0000 7f4284e00640 1 -- 172.21.15.90:0/3288092750 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5609b8cea080 msgr2=0x5609b8ce7b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:41:58.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.735+0000 7f4284e00640 1 --2- 172.21.15.90:0/3288092750 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5609b8cea080 0x5609b8ce7b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:58.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.735+0000 7f4284e00640 1 -- 172.21.15.90:0/3288092750 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5609b8ce19f0 msgr2=0x5609b8ce75d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:41:58.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.735+0000 7f4284e00640 1 --2- 172.21.15.90:0/3288092750 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5609b8ce19f0 0x5609b8ce75d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:58.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.735+0000 7f4284e00640 1 -- 172.21.15.90:0/3288092750 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5609b8a237d0 con 0x5609b8ce0d10 2024-04-02T14:41:58.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.735+0000 7f427fe00640 1 --2- 172.21.15.90:0/3288092750 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5609b8ce19f0 0x5609b8ce75d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:41:58.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.736+0000 7f4284e00640 1 --2- 172.21.15.90:0/3288092750 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5609b8ce0d10 0x5609b8ce7090 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f4270010ec0 tx=0x7f4270004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5b2c95a9ab7aa543 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:41:58.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.736+0000 7f426ea00640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:41:58.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.736+0000 7f426ea00640 1 -- 172.21.15.90:0/3288092750 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f427001d620 con 0x5609b8ce0d10 2024-04-02T14:41:58.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.736+0000 7f426ea00640 1 -- 172.21.15.90:0/3288092750 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f427001d7c0 con 0x5609b8ce0d10 2024-04-02T14:41:58.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.736+0000 7f428a57b740 1 -- 172.21.15.90:0/3288092750 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4278017aa0 con 0x5609b8ce0d10 2024-04-02T14:41:58.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.737+0000 7f426ea00640 1 -- 172.21.15.90:0/3288092750 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f427001d9c0 con 0x5609b8ce0d10 2024-04-02T14:41:58.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.737+0000 7f428a57b740 10 client.27524 fetch_fsmap learned FSMap version 27 2024-04-02T14:41:58.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.737+0000 7f428a57b740 1 -- 172.21.15.90:0/3288092750 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5609b8cf8fe0 con 0x5609b8ce0d10 2024-04-02T14:41:58.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.737+0000 7f426ea00640 1 -- 172.21.15.90:0/3288092750 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4278017aa0 con 0x5609b8ce0d10 2024-04-02T14:41:58.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.737+0000 7f426ea00640 1 -- 172.21.15.90:0/3288092750 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f427001b3b0 con 0x5609b8ce0d10 2024-04-02T14:41:58.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.737+0000 7f428a57b740 10 client.27524 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:41:58.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.737+0000 7f428a57b740 10 client.27524 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.c 2024-04-02T14:41:58.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.737+0000 7f428a57b740 20 client.27524 populate_metadata read hostname 'smithi090' 2024-04-02T14:41:58.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.737+0000 7f428a57b740 1 --2- 172.21.15.90:0/3288092750 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x5609b8cfa4a0 0x5609b8cfc8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:58.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.738+0000 7f428a57b740 4 client.27524 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:41:58.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.738+0000 7f428a57b740 1 -- 172.21.15.90:0/3288092750 --> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5609b8c05040 con 0x5609b8cfa4a0 2024-04-02T14:41:58.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.738+0000 7f427fe00640 1 --2- 172.21.15.90:0/3288092750 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x5609b8cfa4a0 0x5609b8cfc8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:58.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.739+0000 7f427fe00640 1 --2- 172.21.15.90:0/3288092750 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x5609b8cfa4a0 0x5609b8cfc8b0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=da688d622295b67f server_cookie=7d8edcb2663cec22 in_seq=0 out_seq=0 2024-04-02T14:41:58.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.739+0000 7f426ea00640 10 client.27524 ms_handle_connect on v2:172.21.15.177:6836/4223414858 2024-04-02T14:41:58.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.740+0000 7f426ea00640 1 -- 172.21.15.90:0/3288092750 <== mds.4 v2:172.21.15.177:6836/4223414858 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5609b8c05040 con 0x5609b8cfa4a0 2024-04-02T14:41:58.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.740+0000 7f426ea00640 10 client.27524 handle_command_reply: tid=0 2024-04-02T14:41:58.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.740+0000 7f428a57b740 1 client.27524 shutdown 2024-04-02T14:41:58.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.741+0000 7f428a57b740 1 -- 172.21.15.90:0/3288092750 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5609b8ce0d10 msgr2=0x5609b8ce7090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:41:58.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.741+0000 7f428a57b740 1 --2- 172.21.15.90:0/3288092750 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5609b8ce0d10 0x5609b8ce7090 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f4270010ec0 tx=0x7f4270004a40 comp rx=0 tx=0).stop 2024-04-02T14:41:58.741 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.741+0000 7f428a57b740 1 -- 172.21.15.90:0/3288092750 shutdown_connections 2024-04-02T14:41:58.741 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.741+0000 7f428a57b740 1 --2- 172.21.15.90:0/3288092750 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5609b8cea080 0x5609b8ce7b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:58.741 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.741+0000 7f428a57b740 1 --2- 172.21.15.90:0/3288092750 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x5609b8cfa4a0 0x5609b8cfc8b0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:58.741 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.741+0000 7f428a57b740 1 --2- 172.21.15.90:0/3288092750 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5609b8ce19f0 0x5609b8ce75d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:58.741 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.741+0000 7f428a57b740 1 --2- 172.21.15.90:0/3288092750 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5609b8ce0d10 0x5609b8ce7090 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:58.741 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.741+0000 7f426ea00640 0 client.27524 ms_handle_reset on v2:172.21.15.177:6836/4223414858 2024-04-02T14:41:58.741 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.741+0000 7f428a57b740 1 -- 172.21.15.90:0/3288092750 >> 172.21.15.90:0/3288092750 conn(0x5609b8ca7a50 msgr2=0x5609b8cdd4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:41:58.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.742+0000 7f428a57b740 1 -- 172.21.15.90:0/3288092750 shutdown_connections 2024-04-02T14:41:58.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.742+0000 7f428a57b740 1 -- 172.21.15.90:0/3288092750 wait complete. 2024-04-02T14:41:58.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.742+0000 7f428a57b740 20 client.27524 trim_cache size 0 max 16384 2024-04-02T14:41:58.764 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.764+0000 7f428a57b740 1 Processor -- start 2024-04-02T14:41:58.764 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.765+0000 7f428a57b740 1 -- start start 2024-04-02T14:41:58.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.765+0000 7f428a57b740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5609b8ce0d10 0x5609b8ce97c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:58.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.765+0000 7f428a57b740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5609b8ce19f0 0x5609b8cf5b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:58.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.765+0000 7f428a57b740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5609b8cea080 0x5609b8cda720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:58.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.765+0000 7f428a57b740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f42801c4db0 con 0x5609b8ce19f0 2024-04-02T14:41:58.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.765+0000 7f428a57b740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f42801c68f0 con 0x5609b8ce0d10 2024-04-02T14:41:58.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.765+0000 7f428a57b740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f42801c5480 con 0x5609b8cea080 2024-04-02T14:41:58.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.765+0000 7f4284e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5609b8ce0d10 0x5609b8ce97c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:58.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.765+0000 7f4284e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5609b8ce0d10 0x5609b8ce97c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:36538/0 (socket says 172.21.15.90:36538) 2024-04-02T14:41:58.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.765+0000 7f4284e00640 1 -- 172.21.15.90:0/3133703266 learned_addr learned my addr 172.21.15.90:0/3133703266 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:41:58.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.765+0000 7f427fe00640 1 --2- 172.21.15.90:0/3133703266 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5609b8ce19f0 0x5609b8cf5b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:58.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.765+0000 7f4285800640 1 --2- 172.21.15.90:0/3133703266 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5609b8cea080 0x5609b8cda720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:58.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.766+0000 7f4284e00640 1 -- 172.21.15.90:0/3133703266 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5609b8cea080 msgr2=0x5609b8cda720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:41:58.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.766+0000 7f4284e00640 1 --2- 172.21.15.90:0/3133703266 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5609b8cea080 0x5609b8cda720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:58.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.766+0000 7f4284e00640 1 -- 172.21.15.90:0/3133703266 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5609b8ce19f0 msgr2=0x5609b8cf5b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:41:58.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.766+0000 7f4284e00640 1 --2- 172.21.15.90:0/3133703266 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5609b8ce19f0 0x5609b8cf5b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:58.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.766+0000 7f4284e00640 1 -- 172.21.15.90:0/3133703266 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5609b8cd7020 con 0x5609b8ce0d10 2024-04-02T14:41:58.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.766+0000 7f4284e00640 1 --2- 172.21.15.90:0/3133703266 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5609b8ce0d10 0x5609b8ce97c0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f4270004af0 tx=0x7f4270008ce0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d848090461735724 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:41:58.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.766+0000 7f4275600640 1 -- 172.21.15.90:0/3133703266 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4270032070 con 0x5609b8ce0d10 2024-04-02T14:41:58.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.766+0000 7f4275600640 1 -- 172.21.15.90:0/3133703266 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4270033040 con 0x5609b8ce0d10 2024-04-02T14:41:58.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.766+0000 7f4275600640 1 -- 172.21.15.90:0/3133703266 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f427002e420 con 0x5609b8ce0d10 2024-04-02T14:41:58.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.767+0000 7f428a57b740 1 -- 172.21.15.90:0/3133703266 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5609b8ce0d10 msgr2=0x5609b8ce97c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:41:58.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.767+0000 7f428a57b740 1 --2- 172.21.15.90:0/3133703266 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5609b8ce0d10 0x5609b8ce97c0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f4270004af0 tx=0x7f4270008ce0 comp rx=0 tx=0).stop 2024-04-02T14:41:58.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.767+0000 7f428a57b740 1 -- 172.21.15.90:0/3133703266 shutdown_connections 2024-04-02T14:41:58.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.767+0000 7f428a57b740 1 --2- 172.21.15.90:0/3133703266 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5609b8cea080 0x5609b8cda720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:58.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.767+0000 7f428a57b740 1 --2- 172.21.15.90:0/3133703266 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5609b8ce19f0 0x5609b8cf5b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:58.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.767+0000 7f428a57b740 1 --2- 172.21.15.90:0/3133703266 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5609b8ce0d10 0x5609b8ce97c0 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:58.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.767+0000 7f428a57b740 1 -- 172.21.15.90:0/3133703266 >> 172.21.15.90:0/3133703266 conn(0x5609b8ca7a50 msgr2=0x5609b8cda010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:41:58.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.768+0000 7f428a57b740 1 -- 172.21.15.90:0/3133703266 shutdown_connections 2024-04-02T14:41:58.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.768+0000 7f428a57b740 1 -- 172.21.15.90:0/3133703266 wait complete. 2024-04-02T14:41:58.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.769+0000 7f428a57b740 1 -- start start 2024-04-02T14:41:58.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.769+0000 7f428a57b740 1 Processor -- start 2024-04-02T14:41:58.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.769+0000 7f428a57b740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5609b8ce0d10 0x5609b8cf2050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:58.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.770+0000 7f428a57b740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5609b8ce19f0 0x5609b8cf2590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:58.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.770+0000 7f428a57b740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5609b8cea080 0x5609b8ce3360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:58.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.770+0000 7f428a57b740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f42801c6220 con 0x5609b8ce0d10 2024-04-02T14:41:58.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.770+0000 7f428a57b740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f42801c5b50 con 0x5609b8cea080 2024-04-02T14:41:58.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.770+0000 7f428a57b740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f42801c2d90 con 0x5609b8ce19f0 2024-04-02T14:41:58.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.770+0000 7f428a57b740 10 client.0 _ll_register_callbacks cb 0x7f4270010890 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:41:58.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.770+0000 7f4285800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5609b8cea080 0x5609b8ce3360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:58.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.770+0000 7f427fe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5609b8ce19f0 0x5609b8cf2590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:58.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.770+0000 7f4285800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5609b8cea080 0x5609b8ce3360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:36550/0 (socket says 172.21.15.90:36550) 2024-04-02T14:41:58.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.770+0000 7f4285800640 1 -- 172.21.15.90:0/1509126849 learned_addr learned my addr 172.21.15.90:0/1509126849 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:41:58.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.770+0000 7f4284e00640 1 --2- 172.21.15.90:0/1509126849 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5609b8ce0d10 0x5609b8cf2050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:58.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.770+0000 7f4285800640 1 -- 172.21.15.90:0/1509126849 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5609b8ce19f0 msgr2=0x5609b8cf2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:41:58.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.770+0000 7f4285800640 1 --2- 172.21.15.90:0/1509126849 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5609b8ce19f0 0x5609b8cf2590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:58.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.770+0000 7f4285800640 1 -- 172.21.15.90:0/1509126849 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5609b8ce0d10 msgr2=0x5609b8cf2050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:41:58.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.770+0000 7f4285800640 1 --2- 172.21.15.90:0/1509126849 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5609b8ce0d10 0x5609b8cf2050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:58.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.770+0000 7f4285800640 1 -- 172.21.15.90:0/1509126849 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5609b8ce3b90 con 0x5609b8cea080 2024-04-02T14:41:58.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.771+0000 7f4285800640 1 --2- 172.21.15.90:0/1509126849 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5609b8cea080 0x5609b8ce3360 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f427800f730 tx=0x7f427800b7d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=61ac1b0bcd986041 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:41:58.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.771+0000 7f426ea00640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:41:58.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.771+0000 7f426ea00640 1 -- 172.21.15.90:0/1509126849 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4278005040 con 0x5609b8cea080 2024-04-02T14:41:58.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.771+0000 7f426ea00640 1 -- 172.21.15.90:0/1509126849 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f42780051e0 con 0x5609b8cea080 2024-04-02T14:41:58.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.771+0000 7f428a57b740 1 -- 172.21.15.90:0/1509126849 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4270033040 con 0x5609b8cea080 2024-04-02T14:41:58.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.772+0000 7f426ea00640 1 -- 172.21.15.90:0/1509126849 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f42780053e0 con 0x5609b8cea080 2024-04-02T14:41:58.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.772+0000 7f428a57b740 10 client.27530 fetch_fsmap learned FSMap version 27 2024-04-02T14:41:58.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.772+0000 7f428a57b740 1 -- 172.21.15.90:0/1509126849 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5609b8ce4e90 con 0x5609b8cea080 2024-04-02T14:41:58.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.772+0000 7f426ea00640 1 -- 172.21.15.90:0/1509126849 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4270033040 con 0x5609b8cea080 2024-04-02T14:41:58.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.773+0000 7f426ea00640 1 -- 172.21.15.90:0/1509126849 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f4278022440 con 0x5609b8cea080 2024-04-02T14:41:58.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.773+0000 7f428a57b740 10 client.27530 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:41:58.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.773+0000 7f428a57b740 10 client.27530 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.c 2024-04-02T14:41:58.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.773+0000 7f428a57b740 20 client.27530 populate_metadata read hostname 'smithi090' 2024-04-02T14:41:58.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.773+0000 7f428a57b740 1 --2- 172.21.15.90:0/1509126849 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x5609b8d61a40 0x5609b8d63e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:58.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.773+0000 7f428a57b740 4 client.27530 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-02T14:41:58.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.773+0000 7f428a57b740 1 -- 172.21.15.90:0/1509126849 --> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5609b8c52c30 con 0x5609b8d61a40 2024-04-02T14:41:58.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.774+0000 7f4284e00640 1 --2- 172.21.15.90:0/1509126849 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x5609b8d61a40 0x5609b8d63e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:58.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.774+0000 7f4284e00640 1 --2- 172.21.15.90:0/1509126849 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x5609b8d61a40 0x5609b8d63e30 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=3f8d7d71c6185d71 server_cookie=eea502ba6c1c0056 in_seq=0 out_seq=0 2024-04-02T14:41:58.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.774+0000 7f426ea00640 10 client.27530 ms_handle_connect on v2:172.21.15.177:6836/4223414858 2024-04-02T14:41:58.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.774+0000 7f426ea00640 1 -- 172.21.15.90:0/1509126849 <== mds.4 v2:172.21.15.177:6836/4223414858 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5609b8c52c30 con 0x5609b8d61a40 2024-04-02T14:41:58.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.774+0000 7f426ea00640 10 client.27530 handle_command_reply: tid=0 2024-04-02T14:41:58.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.774+0000 7f428a57b740 1 client.27530 shutdown 2024-04-02T14:41:58.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.775+0000 7f428a57b740 1 -- 172.21.15.90:0/1509126849 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5609b8cea080 msgr2=0x5609b8ce3360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:41:58.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.775+0000 7f428a57b740 1 --2- 172.21.15.90:0/1509126849 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5609b8cea080 0x5609b8ce3360 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f427800f730 tx=0x7f427800b7d0 comp rx=0 tx=0).stop 2024-04-02T14:41:58.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.775+0000 7f428a57b740 1 -- 172.21.15.90:0/1509126849 shutdown_connections 2024-04-02T14:41:58.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.775+0000 7f428a57b740 1 --2- 172.21.15.90:0/1509126849 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5609b8cea080 0x5609b8ce3360 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:58.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.775+0000 7f428a57b740 1 --2- 172.21.15.90:0/1509126849 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5609b8ce19f0 0x5609b8cf2590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:58.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.775+0000 7f428a57b740 1 --2- 172.21.15.90:0/1509126849 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x5609b8d61a40 0x5609b8d63e30 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:58.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.775+0000 7f428a57b740 1 --2- 172.21.15.90:0/1509126849 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5609b8ce0d10 0x5609b8cf2050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:58.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.775+0000 7f428a57b740 1 -- 172.21.15.90:0/1509126849 >> 172.21.15.90:0/1509126849 conn(0x5609b8ca7a50 msgr2=0x5609b8cda8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:41:58.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.775+0000 7f426ea00640 0 client.27530 ms_handle_reset on v2:172.21.15.177:6836/4223414858 2024-04-02T14:41:58.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.775+0000 7f428a57b740 1 -- 172.21.15.90:0/1509126849 shutdown_connections 2024-04-02T14:41:58.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.775+0000 7f428a57b740 1 -- 172.21.15.90:0/1509126849 wait complete. 2024-04-02T14:41:58.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.775+0000 7f428a57b740 20 client.27530 trim_cache size 0 max 16384 2024-04-02T14:41:58.777 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-02T14:41:58.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.777+0000 7f4287800640 1 -- 172.21.15.90:0/1466843201 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f42640a3e20 msgr2=0x7f42640a62e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:41:58.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.777+0000 7f4287800640 1 --2- 172.21.15.90:0/1466843201 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f42640a3e20 0x7f42640a62e0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x5609b8ce1f40 tx=0x7f4270008590 comp rx=0 tx=0).stop 2024-04-02T14:41:58.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.777+0000 7f4287800640 1 -- 172.21.15.90:0/1466843201 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4280109420 msgr2=0x7f4280114590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:41:58.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.777+0000 7f4287800640 1 --2- 172.21.15.90:0/1466843201 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4280109420 0x7f4280114590 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f426802c2e0 tx=0x7f426802c7b0 comp rx=0 tx=0).stop 2024-04-02T14:41:58.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.778+0000 7f4287800640 1 -- 172.21.15.90:0/1466843201 shutdown_connections 2024-04-02T14:41:58.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.778+0000 7f4287800640 1 --2- 172.21.15.90:0/1466843201 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f42640a3e20 0x7f42640a62e0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:58.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.778+0000 7f4287800640 1 --2- 172.21.15.90:0/1466843201 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4280114c50 0x7f428010b090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:58.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.778+0000 7f4287800640 1 --2- 172.21.15.90:0/1466843201 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4280109420 0x7f4280114590 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:58.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.778+0000 7f4287800640 1 --2- 172.21.15.90:0/1466843201 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4280108b00 0x7f4280114050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:58.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.778+0000 7f4287800640 1 -- 172.21.15.90:0/1466843201 >> 172.21.15.90:0/1466843201 conn(0x7f428006d040 msgr2=0x7f4280070700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:41:58.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.778+0000 7f4287800640 1 -- 172.21.15.90:0/1466843201 shutdown_connections 2024-04-02T14:41:58.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:58.778+0000 7f4287800640 1 -- 172.21.15.90:0/1466843201 wait complete. 2024-04-02T14:41:58.802 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-04-02T14:41:59.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.085+0000 7fd48f600640 1 -- 172.21.15.90:0/4209745247 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd488108fe0 msgr2=0x7fd488110bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:41:59.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.085+0000 7fd48f600640 1 --2- 172.21.15.90:0/4209745247 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd488108fe0 0x7fd488110bb0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7fd480013490 tx=0x7fd480034370 comp rx=0 tx=0).stop 2024-04-02T14:41:59.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.086+0000 7fd48f600640 1 -- 172.21.15.90:0/4209745247 shutdown_connections 2024-04-02T14:41:59.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.086+0000 7fd48f600640 1 --2- 172.21.15.90:0/4209745247 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd488108fe0 0x7fd488110bb0 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:59.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.086+0000 7fd48f600640 1 --2- 172.21.15.90:0/4209745247 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd4881048c0 0x7fd488108900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:59.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.086+0000 7fd48f600640 1 --2- 172.21.15.90:0/4209745247 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd488103f10 0x7fd4881042f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:59.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.086+0000 7fd48f600640 1 -- 172.21.15.90:0/4209745247 >> 172.21.15.90:0/4209745247 conn(0x7fd4880ff770 msgr2=0x7fd488101b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:41:59.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.086+0000 7fd48f600640 1 -- 172.21.15.90:0/4209745247 shutdown_connections 2024-04-02T14:41:59.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.086+0000 7fd48f600640 1 -- 172.21.15.90:0/4209745247 wait complete. 2024-04-02T14:41:59.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.087+0000 7fd48f600640 1 Processor -- start 2024-04-02T14:41:59.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.089+0000 7fd48f600640 1 -- start start 2024-04-02T14:41:59.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.090+0000 7fd48f600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd488103f10 0x7fd4881965f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:59.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.090+0000 7fd48f600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd488108fe0 0x7fd488196b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:59.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.090+0000 7fd48f600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd48819ac00 0x7fd48819b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:59.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.090+0000 7fd48f600640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fd4880789d0 con 0x7fd48819ac00 2024-04-02T14:41:59.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.090+0000 7fd48f600640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fd488078850 con 0x7fd488108fe0 2024-04-02T14:41:59.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.090+0000 7fd48f600640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fd488078b50 con 0x7fd488103f10 2024-04-02T14:41:59.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.090+0000 7fd48cc00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd488103f10 0x7fd4881965f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:59.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.090+0000 7fd487e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd488108fe0 0x7fd488196b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:59.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.090+0000 7fd487e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd488108fe0 0x7fd488196b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:36582/0 (socket says 172.21.15.90:36582) 2024-04-02T14:41:59.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.090+0000 7fd48cc00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd488103f10 0x7fd4881965f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:56464/0 (socket says 172.21.15.90:56464) 2024-04-02T14:41:59.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.090+0000 7fd487e00640 1 -- 172.21.15.90:0/2276317867 learned_addr learned my addr 172.21.15.90:0/2276317867 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:41:59.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.090+0000 7fd48d600640 1 --2- 172.21.15.90:0/2276317867 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd48819ac00 0x7fd48819b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:59.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.091+0000 7fd487e00640 1 -- 172.21.15.90:0/2276317867 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd488103f10 msgr2=0x7fd4881965f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:41:59.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.091+0000 7fd487e00640 1 --2- 172.21.15.90:0/2276317867 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd488103f10 0x7fd4881965f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:59.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.091+0000 7fd487e00640 1 -- 172.21.15.90:0/2276317867 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd48819ac00 msgr2=0x7fd48819b0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:41:59.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.091+0000 7fd487e00640 1 --2- 172.21.15.90:0/2276317867 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd48819ac00 0x7fd48819b0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:59.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.091+0000 7fd487e00640 1 -- 172.21.15.90:0/2276317867 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd48819b6f0 con 0x7fd488108fe0 2024-04-02T14:41:59.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.091+0000 7fd48cc00640 1 --2- 172.21.15.90:0/2276317867 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd488103f10 0x7fd4881965f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello state changed while learned_addr, mark_down or replacing must be happened just now 2024-04-02T14:41:59.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.091+0000 7fd487e00640 1 --2- 172.21.15.90:0/2276317867 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd488108fe0 0x7fd488196b30 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7fd47400b470 tx=0x7fd47400b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:41:59.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.092+0000 7fd485600640 1 -- 172.21.15.90:0/2276317867 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd474015020 con 0x7fd488108fe0 2024-04-02T14:41:59.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.092+0000 7fd485600640 1 -- 172.21.15.90:0/2276317867 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd474011d70 con 0x7fd488108fe0 2024-04-02T14:41:59.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.092+0000 7fd485600640 1 -- 172.21.15.90:0/2276317867 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd47400f340 con 0x7fd488108fe0 2024-04-02T14:41:59.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.094+0000 7fd48f600640 1 -- 172.21.15.90:0/2276317867 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd48819b9e0 con 0x7fd488108fe0 2024-04-02T14:41:59.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.095+0000 7fd48f600640 1 -- 172.21.15.90:0/2276317867 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd4881a3110 con 0x7fd488108fe0 2024-04-02T14:41:59.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.101+0000 7fd4923e9740 1 Processor -- start 2024-04-02T14:41:59.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.101+0000 7fd485600640 1 -- 172.21.15.90:0/2276317867 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fd474018020 con 0x7fd488108fe0 2024-04-02T14:41:59.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.103+0000 7fd485600640 1 --2- 172.21.15.90:0/2276317867 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fd4700a3e70 0x7fd4700a6330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:59.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.104+0000 7fd4923e9740 1 -- start start 2024-04-02T14:41:59.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.104+0000 7fd48d600640 1 --2- 172.21.15.90:0/2276317867 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fd4700a3e70 0x7fd4700a6330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:59.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.104+0000 7fd485600640 1 -- 172.21.15.90:0/2276317867 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7fd4740cffe0 con 0x7fd488108fe0 2024-04-02T14:41:59.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.105+0000 7fd4923e9740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eb5e25f9f0 0x55eb5e25e740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:59.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.105+0000 7fd4923e9740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eb5e25ed10 0x55eb5e25f190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:59.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.105+0000 7fd4923e9740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55eb5e268080 0x55eb5e26a470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:59.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.105+0000 7fd4923e9740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55eb5e19f400 con 0x55eb5e25ed10 2024-04-02T14:41:59.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.105+0000 7fd4923e9740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55eb5e183f10 con 0x55eb5e268080 2024-04-02T14:41:59.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.105+0000 7fd4923e9740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55eb5e1c36f0 con 0x55eb5e25f9f0 2024-04-02T14:41:59.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.105+0000 7fd48cc00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eb5e25f9f0 0x55eb5e25e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:59.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.105+0000 7fd48cc00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eb5e25f9f0 0x55eb5e25e740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:56470/0 (socket says 172.21.15.90:56470) 2024-04-02T14:41:59.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.105+0000 7fd48cc00640 1 -- 172.21.15.90:0/2534396704 learned_addr learned my addr 172.21.15.90:0/2534396704 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:41:59.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.105+0000 7fd48d600640 1 --2- 172.21.15.90:0/2534396704 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55eb5e268080 0x55eb5e26a470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:59.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.105+0000 7fd487e00640 1 --2- 172.21.15.90:0/2534396704 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eb5e25ed10 0x55eb5e25f190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:59.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.106+0000 7fd48d600640 1 -- 172.21.15.90:0/2534396704 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eb5e25f9f0 msgr2=0x55eb5e25e740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:41:59.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.106+0000 7fd48d600640 1 --2- 172.21.15.90:0/2534396704 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eb5e25f9f0 0x55eb5e25e740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:59.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.106+0000 7fd48d600640 1 -- 172.21.15.90:0/2534396704 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eb5e25ed10 msgr2=0x55eb5e25f190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:41:59.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.106+0000 7fd48d600640 1 --2- 172.21.15.90:0/2534396704 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eb5e25ed10 0x55eb5e25f190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:59.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.106+0000 7fd48d600640 1 -- 172.21.15.90:0/2534396704 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55eb5ded73f0 con 0x55eb5e268080 2024-04-02T14:41:59.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.106+0000 7fd48d600640 1 --2- 172.21.15.90:0/2276317867 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fd4700a3e70 0x7fd4700a6330 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fd480006010 tx=0x7fd48000ff90 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:41:59.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.106+0000 7fd48d600640 1 --2- 172.21.15.90:0/2534396704 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55eb5e268080 0x55eb5e26a470 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7fd480013460 tx=0x7fd480050000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5afe1691d300a023 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:41:59.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.106+0000 7fd479c00640 1 -- 172.21.15.90:0/2534396704 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd480050070 con 0x55eb5e268080 2024-04-02T14:41:59.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.107+0000 7fd479c00640 1 -- 172.21.15.90:0/2534396704 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd480003e50 con 0x55eb5e268080 2024-04-02T14:41:59.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.107+0000 7fd479c00640 1 -- 172.21.15.90:0/2534396704 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd480040040 con 0x55eb5e268080 2024-04-02T14:41:59.108 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:41:58 smithi092 ceph-mon[36497]: from='client.? 172.21.15.90:0/571773655' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-02T14:41:59.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.108+0000 7fd4923e9740 1 -- 172.21.15.90:0/2534396704 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55eb5e268080 msgr2=0x55eb5e26a470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:41:59.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.108+0000 7fd4923e9740 1 --2- 172.21.15.90:0/2534396704 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55eb5e268080 0x55eb5e26a470 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7fd480013460 tx=0x7fd480050000 comp rx=0 tx=0).stop 2024-04-02T14:41:59.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.108+0000 7fd4923e9740 1 -- 172.21.15.90:0/2534396704 shutdown_connections 2024-04-02T14:41:59.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.108+0000 7fd4923e9740 1 --2- 172.21.15.90:0/2534396704 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55eb5e268080 0x55eb5e26a470 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:59.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.108+0000 7fd4923e9740 1 --2- 172.21.15.90:0/2534396704 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eb5e25ed10 0x55eb5e25f190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:59.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.108+0000 7fd4923e9740 1 --2- 172.21.15.90:0/2534396704 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eb5e25f9f0 0x55eb5e25e740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:59.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.108+0000 7fd4923e9740 1 -- 172.21.15.90:0/2534396704 >> 172.21.15.90:0/2534396704 conn(0x55eb5e225a50 msgr2=0x55eb5e259760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:41:59.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.109+0000 7fd4923e9740 1 -- 172.21.15.90:0/2534396704 shutdown_connections 2024-04-02T14:41:59.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.110+0000 7fd4923e9740 1 -- 172.21.15.90:0/2534396704 wait complete. 2024-04-02T14:41:59.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.112+0000 7fd4923e9740 1 -- start start 2024-04-02T14:41:59.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.112+0000 7fd4923e9740 1 Processor -- start 2024-04-02T14:41:59.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.115+0000 7fd4923e9740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eb5e25ed10 0x55eb5e2650b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:59.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.115+0000 7fd4923e9740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eb5e25f9f0 0x55eb5e2655f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:59.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.115+0000 7fd4923e9740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55eb5e268080 0x55eb5e265b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:59.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.115+0000 7fd4923e9740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55eb5e1bf6a0 con 0x55eb5e25f9f0 2024-04-02T14:41:59.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.115+0000 7fd4923e9740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55eb5e16b980 con 0x55eb5e268080 2024-04-02T14:41:59.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.115+0000 7fd4923e9740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55eb5e17ecb0 con 0x55eb5e25ed10 2024-04-02T14:41:59.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.115+0000 7fd4923e9740 10 client.0 _ll_register_callbacks cb 0x55eb5e17e6a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:41:59.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.115+0000 7fd48cc00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eb5e25ed10 0x55eb5e2650b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:59.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.115+0000 7fd48cc00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eb5e25ed10 0x55eb5e2650b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:56472/0 (socket says 172.21.15.90:56472) 2024-04-02T14:41:59.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.115+0000 7fd48cc00640 1 -- 172.21.15.90:0/61603554 learned_addr learned my addr 172.21.15.90:0/61603554 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:41:59.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.116+0000 7fd48cc00640 1 -- 172.21.15.90:0/61603554 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55eb5e268080 msgr2=0x55eb5e265b30 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-02T14:41:59.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.116+0000 7fd48cc00640 1 --2- 172.21.15.90:0/61603554 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55eb5e268080 0x55eb5e265b30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:59.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.116+0000 7fd48cc00640 1 -- 172.21.15.90:0/61603554 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eb5e25f9f0 msgr2=0x55eb5e2655f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-02T14:41:59.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.116+0000 7fd48cc00640 1 --2- 172.21.15.90:0/61603554 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eb5e25f9f0 0x55eb5e2655f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:59.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.116+0000 7fd48cc00640 1 -- 172.21.15.90:0/61603554 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55eb5dfa17d0 con 0x55eb5e25ed10 2024-04-02T14:41:59.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.117+0000 7fd48cc00640 1 --2- 172.21.15.90:0/61603554 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eb5e25ed10 0x55eb5e2650b0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fd47c00f530 tx=0x7fd47c00f9f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=306720afc38a37d3 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:41:59.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.117+0000 7fd46e000640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:41:59.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.117+0000 7fd46e000640 1 -- 172.21.15.90:0/61603554 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd47c013850 con 0x55eb5e25ed10 2024-04-02T14:41:59.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.118+0000 7fd46e000640 1 -- 172.21.15.90:0/61603554 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7fd47c0139f0 con 0x55eb5e25ed10 2024-04-02T14:41:59.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.118+0000 7fd4923e9740 1 -- 172.21.15.90:0/61603554 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd480003e50 con 0x55eb5e25ed10 2024-04-02T14:41:59.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.119+0000 7fd46e000640 1 -- 172.21.15.90:0/61603554 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fd47c013bf0 con 0x55eb5e25ed10 2024-04-02T14:41:59.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.119+0000 7fd4923e9740 10 client.27409 fetch_fsmap learned FSMap version 27 2024-04-02T14:41:59.120 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.120+0000 7fd4923e9740 1 -- 172.21.15.90:0/61603554 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55eb5e276f60 con 0x55eb5e25ed10 2024-04-02T14:41:59.120 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.120+0000 7fd46e000640 1 -- 172.21.15.90:0/61603554 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd480003e50 con 0x55eb5e25ed10 2024-04-02T14:41:59.120 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.120+0000 7fd46e000640 1 -- 172.21.15.90:0/61603554 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fd47c0113e0 con 0x55eb5e25ed10 2024-04-02T14:41:59.120 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.120+0000 7fd4923e9740 10 client.27409 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:41:59.120 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.120+0000 7fd4923e9740 10 client.27409 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-04-02T14:41:59.120 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.120+0000 7fd4923e9740 20 client.27409 populate_metadata read hostname 'smithi090' 2024-04-02T14:41:59.120 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.120+0000 7fd4923e9740 1 --2- 172.21.15.90:0/61603554 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x55eb5e278420 0x55eb5e27a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:59.120 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.120+0000 7fd4923e9740 4 client.27409 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:41:59.120 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.120+0000 7fd4923e9740 1 -- 172.21.15.90:0/61603554 --> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55eb5e183040 con 0x55eb5e278420 2024-04-02T14:41:59.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.121+0000 7fd487e00640 1 --2- 172.21.15.90:0/61603554 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x55eb5e278420 0x55eb5e27a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:59.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.121+0000 7fd487e00640 1 --2- 172.21.15.90:0/61603554 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x55eb5e278420 0x55eb5e27a8c0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=465d325987b5a925 server_cookie=4167273d30e6d851 in_seq=0 out_seq=0 2024-04-02T14:41:59.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.121+0000 7fd46e000640 10 client.27409 ms_handle_connect on v2:172.21.15.92:6836/1270635939 2024-04-02T14:41:59.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.122+0000 7fd46e000640 1 -- 172.21.15.90:0/61603554 <== mds.3 v2:172.21.15.92:6836/1270635939 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55eb5e183040 con 0x55eb5e278420 2024-04-02T14:41:59.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.122+0000 7fd46e000640 10 client.27409 handle_command_reply: tid=0 2024-04-02T14:41:59.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.123+0000 7fd4923e9740 1 client.27409 shutdown 2024-04-02T14:41:59.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.124+0000 7fd4923e9740 1 -- 172.21.15.90:0/61603554 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eb5e25ed10 msgr2=0x55eb5e2650b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:41:59.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.124+0000 7fd4923e9740 1 --2- 172.21.15.90:0/61603554 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eb5e25ed10 0x55eb5e2650b0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fd47c00f530 tx=0x7fd47c00f9f0 comp rx=0 tx=0).stop 2024-04-02T14:41:59.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.124+0000 7fd4923e9740 1 -- 172.21.15.90:0/61603554 shutdown_connections 2024-04-02T14:41:59.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.124+0000 7fd4923e9740 1 --2- 172.21.15.90:0/61603554 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x55eb5e278420 0x55eb5e27a8c0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:59.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.124+0000 7fd4923e9740 1 --2- 172.21.15.90:0/61603554 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55eb5e268080 0x55eb5e265b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:59.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.124+0000 7fd4923e9740 1 --2- 172.21.15.90:0/61603554 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eb5e25f9f0 0x55eb5e2655f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:59.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.124+0000 7fd4923e9740 1 --2- 172.21.15.90:0/61603554 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eb5e25ed10 0x55eb5e2650b0 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:59.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.124+0000 7fd4923e9740 1 -- 172.21.15.90:0/61603554 >> 172.21.15.90:0/61603554 conn(0x55eb5e225a50 msgr2=0x55eb5e25b4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:41:59.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.124+0000 7fd46e000640 0 client.27409 ms_handle_reset on v2:172.21.15.92:6836/1270635939 2024-04-02T14:41:59.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.125+0000 7fd4923e9740 1 -- 172.21.15.90:0/61603554 shutdown_connections 2024-04-02T14:41:59.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.125+0000 7fd4923e9740 1 -- 172.21.15.90:0/61603554 wait complete. 2024-04-02T14:41:59.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.125+0000 7fd4923e9740 20 client.27409 trim_cache size 0 max 16384 2024-04-02T14:41:59.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.155+0000 7fd4923e9740 1 Processor -- start 2024-04-02T14:41:59.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.155+0000 7fd4923e9740 1 -- start start 2024-04-02T14:41:59.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.155+0000 7fd4923e9740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eb5e25ed10 0x55eb5e2677e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:59.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.155+0000 7fd4923e9740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55eb5e25f9f0 0x55eb5e273a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:59.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.155+0000 7fd4923e9740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eb5e268080 0x55eb5e260390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:59.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.155+0000 7fd4923e9740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fd4881ab7c0 con 0x55eb5e25ed10 2024-04-02T14:41:59.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.155+0000 7fd4923e9740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fd4881ad300 con 0x55eb5e25f9f0 2024-04-02T14:41:59.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.155+0000 7fd4923e9740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fd4881abe90 con 0x55eb5e268080 2024-04-02T14:41:59.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.155+0000 7fd48d600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eb5e268080 0x55eb5e260390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:59.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.156+0000 7fd48d600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eb5e268080 0x55eb5e260390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:56474/0 (socket says 172.21.15.90:56474) 2024-04-02T14:41:59.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.156+0000 7fd48d600640 1 -- 172.21.15.90:0/1603277258 learned_addr learned my addr 172.21.15.90:0/1603277258 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:41:59.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.156+0000 7fd48cc00640 1 --2- 172.21.15.90:0/1603277258 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eb5e25ed10 0x55eb5e2677e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:59.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.156+0000 7fd487e00640 1 --2- 172.21.15.90:0/1603277258 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55eb5e25f9f0 0x55eb5e273a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:59.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.156+0000 7fd48d600640 1 -- 172.21.15.90:0/1603277258 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55eb5e25f9f0 msgr2=0x55eb5e273a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:41:59.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.156+0000 7fd48d600640 1 --2- 172.21.15.90:0/1603277258 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55eb5e25f9f0 0x55eb5e273a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:59.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.156+0000 7fd48d600640 1 -- 172.21.15.90:0/1603277258 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eb5e25ed10 msgr2=0x55eb5e2677e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:41:59.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.156+0000 7fd48d600640 1 --2- 172.21.15.90:0/1603277258 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eb5e25ed10 0x55eb5e2677e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:59.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.156+0000 7fd48d600640 1 -- 172.21.15.90:0/1603277258 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55eb5e263600 con 0x55eb5e268080 2024-04-02T14:41:59.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.157+0000 7fd48d600640 1 --2- 172.21.15.90:0/1603277258 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eb5e268080 0x55eb5e260390 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fd480013490 tx=0x7fd48000fc50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c1201ffa43a982da server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:41:59.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.157+0000 7fd479200640 1 -- 172.21.15.90:0/1603277258 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd480058030 con 0x55eb5e268080 2024-04-02T14:41:59.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.157+0000 7fd479200640 1 -- 172.21.15.90:0/1603277258 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd48003b580 con 0x55eb5e268080 2024-04-02T14:41:59.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.157+0000 7fd479200640 1 -- 172.21.15.90:0/1603277258 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd480040040 con 0x55eb5e268080 2024-04-02T14:41:59.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.158+0000 7fd4923e9740 1 -- 172.21.15.90:0/1603277258 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eb5e268080 msgr2=0x55eb5e260390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:41:59.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.158+0000 7fd4923e9740 1 --2- 172.21.15.90:0/1603277258 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eb5e268080 0x55eb5e260390 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fd480013490 tx=0x7fd48000fc50 comp rx=0 tx=0).stop 2024-04-02T14:41:59.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.159+0000 7fd4923e9740 1 -- 172.21.15.90:0/1603277258 shutdown_connections 2024-04-02T14:41:59.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.159+0000 7fd4923e9740 1 --2- 172.21.15.90:0/1603277258 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eb5e268080 0x55eb5e260390 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:59.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.159+0000 7fd4923e9740 1 --2- 172.21.15.90:0/1603277258 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55eb5e25f9f0 0x55eb5e273a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:59.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.159+0000 7fd4923e9740 1 --2- 172.21.15.90:0/1603277258 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eb5e25ed10 0x55eb5e2677e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:59.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.159+0000 7fd4923e9740 1 -- 172.21.15.90:0/1603277258 >> 172.21.15.90:0/1603277258 conn(0x55eb5e225a50 msgr2=0x55eb5e258010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:41:59.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.159+0000 7fd4923e9740 1 -- 172.21.15.90:0/1603277258 shutdown_connections 2024-04-02T14:41:59.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.159+0000 7fd4923e9740 1 -- 172.21.15.90:0/1603277258 wait complete. 2024-04-02T14:41:59.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.160+0000 7fd4923e9740 1 -- start start 2024-04-02T14:41:59.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.160+0000 7fd4923e9740 1 Processor -- start 2024-04-02T14:41:59.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.162+0000 7fd4923e9740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55eb5e25ed10 0x55eb5e2685a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:59.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.162+0000 7fd4923e9740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eb5e25f9f0 0x55eb5e268ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:59.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.162+0000 7fd4923e9740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eb5e268080 0x55eb5e26efa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:59.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.162+0000 7fd48cc00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55eb5e25ed10 0x55eb5e2685a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:59.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.162+0000 7fd48cc00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55eb5e25ed10 0x55eb5e2685a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:36634/0 (socket says 172.21.15.90:36634) 2024-04-02T14:41:59.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.162+0000 7fd487e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eb5e25f9f0 0x55eb5e268ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:59.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.162+0000 7fd4923e9740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fd4881acc30 con 0x55eb5e268080 2024-04-02T14:41:59.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.162+0000 7fd4923e9740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fd4881ac560 con 0x55eb5e25ed10 2024-04-02T14:41:59.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.162+0000 7fd4923e9740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fd4881a97a0 con 0x55eb5e25f9f0 2024-04-02T14:41:59.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.162+0000 7fd4923e9740 10 client.0 _ll_register_callbacks cb 0x7fd47c00aef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:41:59.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.162+0000 7fd48d600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eb5e268080 0x55eb5e26efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:59.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.162+0000 7fd487e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eb5e25f9f0 0x55eb5e268ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:56486/0 (socket says 172.21.15.90:56486) 2024-04-02T14:41:59.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.162+0000 7fd487e00640 1 -- 172.21.15.90:0/3361729921 learned_addr learned my addr 172.21.15.90:0/3361729921 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:41:59.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.163+0000 7fd48cc00640 1 -- 172.21.15.90:0/3361729921 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eb5e25f9f0 msgr2=0x55eb5e268ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:41:59.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.163+0000 7fd48cc00640 1 --2- 172.21.15.90:0/3361729921 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eb5e25f9f0 0x55eb5e268ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:59.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.163+0000 7fd48cc00640 1 -- 172.21.15.90:0/3361729921 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eb5e268080 msgr2=0x55eb5e26efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:41:59.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.163+0000 7fd48cc00640 1 --2- 172.21.15.90:0/3361729921 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eb5e268080 0x55eb5e26efa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:59.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.163+0000 7fd48cc00640 1 -- 172.21.15.90:0/3361729921 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55eb5e26f4e0 con 0x55eb5e25ed10 2024-04-02T14:41:59.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.163+0000 7fd487e00640 1 --2- 172.21.15.90:0/3361729921 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eb5e25f9f0 0x55eb5e268ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:41:59.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.164+0000 7fd48cc00640 1 --2- 172.21.15.90:0/3361729921 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55eb5e25ed10 0x55eb5e2685a0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7fd47c000c00 tx=0x7fd47c013650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cd276af8f925b9e7 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:41:59.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.164+0000 7fd46e000640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:41:59.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.164+0000 7fd46e000640 1 -- 172.21.15.90:0/3361729921 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd47c030020 con 0x55eb5e25ed10 2024-04-02T14:41:59.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.164+0000 7fd46e000640 1 -- 172.21.15.90:0/3361729921 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7fd47c0119f0 con 0x55eb5e25ed10 2024-04-02T14:41:59.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.164+0000 7fd4923e9740 1 -- 172.21.15.90:0/3361729921 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd48003b580 con 0x55eb5e25ed10 2024-04-02T14:41:59.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.165+0000 7fd46e000640 1 -- 172.21.15.90:0/3361729921 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fd47c028040 con 0x55eb5e25ed10 2024-04-02T14:41:59.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.165+0000 7fd4923e9740 10 client.27545 fetch_fsmap learned FSMap version 27 2024-04-02T14:41:59.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.165+0000 7fd4923e9740 1 -- 172.21.15.90:0/3361729921 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55eb5e2707e0 con 0x55eb5e25ed10 2024-04-02T14:41:59.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.166+0000 7fd46e000640 1 -- 172.21.15.90:0/3361729921 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd48003b580 con 0x55eb5e25ed10 2024-04-02T14:41:59.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.166+0000 7fd46e000640 1 -- 172.21.15.90:0/3361729921 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fd47c02b070 con 0x55eb5e25ed10 2024-04-02T14:41:59.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.166+0000 7fd4923e9740 10 client.27545 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:41:59.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.166+0000 7fd4923e9740 10 client.27545 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-04-02T14:41:59.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.166+0000 7fd4923e9740 20 client.27545 populate_metadata read hostname 'smithi090' 2024-04-02T14:41:59.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.166+0000 7fd4923e9740 1 --2- 172.21.15.90:0/3361729921 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x55eb5e2ded90 0x55eb5e2e1180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:59.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.166+0000 7fd4923e9740 4 client.27545 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-02T14:41:59.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.166+0000 7fd4923e9740 1 -- 172.21.15.90:0/3361729921 --> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55eb5e1d0c30 con 0x55eb5e2ded90 2024-04-02T14:41:59.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.167+0000 7fd487e00640 1 --2- 172.21.15.90:0/3361729921 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x55eb5e2ded90 0x55eb5e2e1180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:59.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.167+0000 7fd487e00640 1 --2- 172.21.15.90:0/3361729921 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x55eb5e2ded90 0x55eb5e2e1180 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=2bb4621b70591e94 server_cookie=681cbc004c840b09 in_seq=0 out_seq=0 2024-04-02T14:41:59.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.167+0000 7fd46e000640 10 client.27545 ms_handle_connect on v2:172.21.15.92:6836/1270635939 2024-04-02T14:41:59.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.167+0000 7fd46e000640 1 -- 172.21.15.90:0/3361729921 <== mds.3 v2:172.21.15.92:6836/1270635939 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55eb5e1d0c30 con 0x55eb5e2ded90 2024-04-02T14:41:59.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.167+0000 7fd46e000640 10 client.27545 handle_command_reply: tid=0 2024-04-02T14:41:59.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.167+0000 7fd4923e9740 1 client.27545 shutdown 2024-04-02T14:41:59.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.168+0000 7fd4923e9740 1 -- 172.21.15.90:0/3361729921 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55eb5e25ed10 msgr2=0x55eb5e2685a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:41:59.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.168+0000 7fd4923e9740 1 --2- 172.21.15.90:0/3361729921 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55eb5e25ed10 0x55eb5e2685a0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7fd47c000c00 tx=0x7fd47c013650 comp rx=0 tx=0).stop 2024-04-02T14:41:59.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.168+0000 7fd4923e9740 1 -- 172.21.15.90:0/3361729921 shutdown_connections 2024-04-02T14:41:59.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.169+0000 7fd4923e9740 1 --2- 172.21.15.90:0/3361729921 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x55eb5e2ded90 0x55eb5e2e1180 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:59.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.169+0000 7fd4923e9740 1 --2- 172.21.15.90:0/3361729921 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eb5e268080 0x55eb5e26efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:59.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.169+0000 7fd4923e9740 1 --2- 172.21.15.90:0/3361729921 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eb5e25f9f0 0x55eb5e268ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:59.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.169+0000 7fd4923e9740 1 --2- 172.21.15.90:0/3361729921 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55eb5e25ed10 0x55eb5e2685a0 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:59.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.169+0000 7fd4923e9740 1 -- 172.21.15.90:0/3361729921 >> 172.21.15.90:0/3361729921 conn(0x55eb5e225a50 msgr2=0x55eb5e258010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:41:59.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.169+0000 7fd46e000640 0 client.27545 ms_handle_reset on v2:172.21.15.92:6836/1270635939 2024-04-02T14:41:59.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.169+0000 7fd4923e9740 1 -- 172.21.15.90:0/3361729921 shutdown_connections 2024-04-02T14:41:59.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.169+0000 7fd4923e9740 1 -- 172.21.15.90:0/3361729921 wait complete. 2024-04-02T14:41:59.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.169+0000 7fd4923e9740 20 client.27545 trim_cache size 0 max 16384 2024-04-02T14:41:59.170 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-02T14:41:59.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.170+0000 7fd48f600640 1 -- 172.21.15.90:0/2276317867 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fd4700a3e70 msgr2=0x7fd4700a6330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:41:59.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.170+0000 7fd48f600640 1 --2- 172.21.15.90:0/2276317867 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fd4700a3e70 0x7fd4700a6330 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fd480006010 tx=0x7fd48000ff90 comp rx=0 tx=0).stop 2024-04-02T14:41:59.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.170+0000 7fd48f600640 1 -- 172.21.15.90:0/2276317867 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd488108fe0 msgr2=0x7fd488196b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:41:59.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.170+0000 7fd48f600640 1 --2- 172.21.15.90:0/2276317867 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd488108fe0 0x7fd488196b30 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7fd47400b470 tx=0x7fd47400b930 comp rx=0 tx=0).stop 2024-04-02T14:41:59.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.171+0000 7fd48f600640 1 -- 172.21.15.90:0/2276317867 shutdown_connections 2024-04-02T14:41:59.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.171+0000 7fd48f600640 1 --2- 172.21.15.90:0/2276317867 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fd4700a3e70 0x7fd4700a6330 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:59.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.171+0000 7fd48f600640 1 --2- 172.21.15.90:0/2276317867 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd48819ac00 0x7fd48819b0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:59.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.171+0000 7fd48f600640 1 --2- 172.21.15.90:0/2276317867 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd488108fe0 0x7fd488196b30 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:59.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.171+0000 7fd48f600640 1 --2- 172.21.15.90:0/2276317867 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd488103f10 0x7fd4881965f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:59.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.171+0000 7fd48f600640 1 -- 172.21.15.90:0/2276317867 >> 172.21.15.90:0/2276317867 conn(0x7fd4880ff770 msgr2=0x7fd4881078e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:41:59.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.171+0000 7fd48f600640 1 -- 172.21.15.90:0/2276317867 shutdown_connections 2024-04-02T14:41:59.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.171+0000 7fd48f600640 1 -- 172.21.15.90:0/2276317867 wait complete. 2024-04-02T14:41:59.183 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:41:58 smithi177 ceph-mon[36431]: from='client.? 172.21.15.90:0/571773655' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-02T14:41:59.185 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:41:58 smithi090 ceph-mon[31674]: from='client.? 172.21.15.90:0/571773655' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-02T14:41:59.193 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-02T14:41:59.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.477+0000 7fea2ba00640 1 -- 172.21.15.90:0/3631774085 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fea24108cc0 msgr2=0x7fea2410cd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:41:59.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.477+0000 7fea2ba00640 1 --2- 172.21.15.90:0/3631774085 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fea24108cc0 0x7fea2410cd00 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7fea10009900 tx=0x7fea1002bc30 comp rx=0 tx=0).stop 2024-04-02T14:41:59.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.477+0000 7fea2ba00640 1 -- 172.21.15.90:0/3631774085 shutdown_connections 2024-04-02T14:41:59.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.477+0000 7fea2ba00640 1 --2- 172.21.15.90:0/3631774085 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fea2410d3e0 0x7fea24114fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:59.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.477+0000 7fea2ba00640 1 --2- 172.21.15.90:0/3631774085 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fea24108cc0 0x7fea2410cd00 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:59.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.477+0000 7fea2ba00640 1 --2- 172.21.15.90:0/3631774085 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fea24108310 0x7fea241086f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:59.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.477+0000 7fea2ba00640 1 -- 172.21.15.90:0/3631774085 >> 172.21.15.90:0/3631774085 conn(0x7fea24076040 msgr2=0x7fea24078460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:41:59.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.478+0000 7fea2ba00640 1 -- 172.21.15.90:0/3631774085 shutdown_connections 2024-04-02T14:41:59.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.478+0000 7fea2ba00640 1 -- 172.21.15.90:0/3631774085 wait complete. 2024-04-02T14:41:59.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.479+0000 7fea2ba00640 1 Processor -- start 2024-04-02T14:41:59.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.479+0000 7fea2ba00640 1 -- start start 2024-04-02T14:41:59.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.480+0000 7fea2ba00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fea24108310 0x7fea241a4470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:59.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.480+0000 7fea2ba00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fea2410d3e0 0x7fea241a49b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:59.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.480+0000 7fea2ba00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fea2419e620 0x7fea2419ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:59.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.480+0000 7fea2ba00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fea241184a0 con 0x7fea2419e620 2024-04-02T14:41:59.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.480+0000 7fea2ba00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fea24118320 con 0x7fea2410d3e0 2024-04-02T14:41:59.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.480+0000 7fea2ba00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fea24118620 con 0x7fea24108310 2024-04-02T14:41:59.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.480+0000 7fea29000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fea24108310 0x7fea241a4470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:59.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.480+0000 7fea23e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fea2410d3e0 0x7fea241a49b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:59.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.480+0000 7fea29000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fea24108310 0x7fea241a4470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:56506/0 (socket says 172.21.15.90:56506) 2024-04-02T14:41:59.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.480+0000 7fea29000640 1 -- 172.21.15.90:0/696511514 learned_addr learned my addr 172.21.15.90:0/696511514 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:41:59.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.480+0000 7fea29a00640 1 --2- 172.21.15.90:0/696511514 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fea2419e620 0x7fea2419ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:59.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.480+0000 7fea29000640 1 -- 172.21.15.90:0/696511514 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fea2410d3e0 msgr2=0x7fea241a49b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:41:59.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.480+0000 7fea29000640 1 --2- 172.21.15.90:0/696511514 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fea2410d3e0 0x7fea241a49b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:59.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.480+0000 7fea29000640 1 -- 172.21.15.90:0/696511514 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fea2419e620 msgr2=0x7fea2419ead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:41:59.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.480+0000 7fea29000640 1 --2- 172.21.15.90:0/696511514 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fea2419e620 0x7fea2419ead0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:59.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.480+0000 7fea29000640 1 -- 172.21.15.90:0/696511514 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea2419f140 con 0x7fea24108310 2024-04-02T14:41:59.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.480+0000 7fea29a00640 1 --2- 172.21.15.90:0/696511514 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fea2419e620 0x7fea2419ead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:41:59.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.481+0000 7fea29000640 1 --2- 172.21.15.90:0/696511514 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fea24108310 0x7fea241a4470 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fea1800e9f0 tx=0x7fea1800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:41:59.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.481+0000 7fea21600640 1 -- 172.21.15.90:0/696511514 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea1800cd80 con 0x7fea24108310 2024-04-02T14:41:59.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.481+0000 7fea21600640 1 -- 172.21.15.90:0/696511514 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fea18010910 con 0x7fea24108310 2024-04-02T14:41:59.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.481+0000 7fea21600640 1 -- 172.21.15.90:0/696511514 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea18018990 con 0x7fea24108310 2024-04-02T14:41:59.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.481+0000 7fea2ba00640 1 -- 172.21.15.90:0/696511514 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea2419f430 con 0x7fea24108310 2024-04-02T14:41:59.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.481+0000 7fea2ba00640 1 -- 172.21.15.90:0/696511514 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fea241ab6d0 con 0x7fea24108310 2024-04-02T14:41:59.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.486+0000 7fea2e81b740 1 Processor -- start 2024-04-02T14:41:59.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.487+0000 7fea2e81b740 1 -- start start 2024-04-02T14:41:59.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.488+0000 7fea2e81b740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5597637429f0 0x559763741740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:59.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.488+0000 7fea2e81b740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559763741d10 0x559763742190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:59.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.488+0000 7fea2e81b740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55976374b080 0x55976374d470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:59.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.488+0000 7fea2e81b740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x559763682400 con 0x55976374b080 2024-04-02T14:41:59.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.488+0000 7fea2e81b740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x559763666f10 con 0x5597637429f0 2024-04-02T14:41:59.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.488+0000 7fea2e81b740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5597636a66f0 con 0x559763741d10 2024-04-02T14:41:59.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.488+0000 7fea29000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559763741d10 0x559763742190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:59.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.488+0000 7fea29000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559763741d10 0x559763742190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:56512/0 (socket says 172.21.15.90:56512) 2024-04-02T14:41:59.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.488+0000 7fea29000640 1 -- 172.21.15.90:0/3271458385 learned_addr learned my addr 172.21.15.90:0/3271458385 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:41:59.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.488+0000 7fea29a00640 1 --2- 172.21.15.90:0/3271458385 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5597637429f0 0x559763741740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:59.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.489+0000 7fea21600640 1 -- 172.21.15.90:0/696511514 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fea18010ac0 con 0x7fea24108310 2024-04-02T14:41:59.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.489+0000 7fea29000640 1 -- 172.21.15.90:0/3271458385 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5597637429f0 msgr2=0x559763741740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:41:59.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.489+0000 7fea29000640 1 --2- 172.21.15.90:0/3271458385 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5597637429f0 0x559763741740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:59.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.489+0000 7fea29000640 1 -- 172.21.15.90:0/3271458385 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55976374b080 msgr2=0x55976374d470 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-02T14:41:59.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.490+0000 7fea23e00640 1 --2- 172.21.15.90:0/3271458385 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55976374b080 0x55976374d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:59.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.490+0000 7fea29000640 1 --2- 172.21.15.90:0/3271458385 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55976374b080 0x55976374d470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:59.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.490+0000 7fea29000640 1 -- 172.21.15.90:0/3271458385 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5597633ba3f0 con 0x559763741d10 2024-04-02T14:41:59.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.490+0000 7fea23e00640 1 --2- 172.21.15.90:0/3271458385 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55976374b080 0x55976374d470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:41:59.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.491+0000 7fea29000640 1 --2- 172.21.15.90:0/3271458385 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559763741d10 0x559763742190 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fea180ce1c0 tx=0x7fea180d9680 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c2dd01a66bbdc45a server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:41:59.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.494+0000 7fea16000640 1 -- 172.21.15.90:0/3271458385 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea180e0020 con 0x559763741d10 2024-04-02T14:41:59.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.494+0000 7fea16000640 1 -- 172.21.15.90:0/3271458385 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fea180dcb10 con 0x559763741d10 2024-04-02T14:41:59.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.494+0000 7fea16000640 1 -- 172.21.15.90:0/3271458385 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea180dbae0 con 0x559763741d10 2024-04-02T14:41:59.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.498+0000 7fea2e81b740 1 -- 172.21.15.90:0/3271458385 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559763741d10 msgr2=0x559763742190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:41:59.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.498+0000 7fea2e81b740 1 --2- 172.21.15.90:0/3271458385 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559763741d10 0x559763742190 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fea180ce1c0 tx=0x7fea180d9680 comp rx=0 tx=0).stop 2024-04-02T14:41:59.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.499+0000 7fea21600640 1 --2- 172.21.15.90:0/696511514 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fea0c0a3e20 0x7fea0c0a62e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:59.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.499+0000 7fea21600640 1 -- 172.21.15.90:0/696511514 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7fea180d0b80 con 0x7fea24108310 2024-04-02T14:41:59.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.501+0000 7fea2e81b740 1 -- 172.21.15.90:0/3271458385 shutdown_connections 2024-04-02T14:41:59.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.501+0000 7fea2e81b740 1 --2- 172.21.15.90:0/3271458385 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55976374b080 0x55976374d470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:59.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.501+0000 7fea2e81b740 1 --2- 172.21.15.90:0/3271458385 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559763741d10 0x559763742190 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:59.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.501+0000 7fea29a00640 1 --2- 172.21.15.90:0/696511514 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fea0c0a3e20 0x7fea0c0a62e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:59.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.501+0000 7fea2e81b740 1 --2- 172.21.15.90:0/3271458385 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5597637429f0 0x559763741740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:59.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.501+0000 7fea2e81b740 1 -- 172.21.15.90:0/3271458385 >> 172.21.15.90:0/3271458385 conn(0x559763708a50 msgr2=0x55976373c760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:41:59.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.504+0000 7fea2e81b740 1 -- 172.21.15.90:0/3271458385 shutdown_connections 2024-04-02T14:41:59.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.505+0000 7fea2e81b740 1 -- 172.21.15.90:0/3271458385 wait complete. 2024-04-02T14:41:59.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.505+0000 7fea29a00640 1 --2- 172.21.15.90:0/696511514 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fea0c0a3e20 0x7fea0c0a62e0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fea1c005fd0 tx=0x7fea1c009820 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:41:59.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.505+0000 7fea2e81b740 1 -- start start 2024-04-02T14:41:59.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.506+0000 7fea2e81b740 1 Processor -- start 2024-04-02T14:41:59.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.506+0000 7fea2e81b740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559763741d10 0x559763748110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:59.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.506+0000 7fea2e81b740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5597637429f0 0x559763748650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:59.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.506+0000 7fea2e81b740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55976374b080 0x559763748b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:59.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.506+0000 7fea2e81b740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5597636a26a0 con 0x559763741d10 2024-04-02T14:41:59.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.507+0000 7fea2e81b740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55976364e980 con 0x55976374b080 2024-04-02T14:41:59.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.507+0000 7fea2e81b740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x559763661cb0 con 0x5597637429f0 2024-04-02T14:41:59.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.507+0000 7fea2e81b740 10 client.0 _ll_register_callbacks cb 0x5597636616a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:41:59.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.507+0000 7fea29a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55976374b080 0x559763748b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:59.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.507+0000 7fea29a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55976374b080 0x559763748b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:36658/0 (socket says 172.21.15.90:36658) 2024-04-02T14:41:59.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.507+0000 7fea29a00640 1 -- 172.21.15.90:0/3723542330 learned_addr learned my addr 172.21.15.90:0/3723542330 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:41:59.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.507+0000 7fea29a00640 1 -- 172.21.15.90:0/3723542330 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5597637429f0 msgr2=0x559763748650 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-02T14:41:59.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.507+0000 7fea29a00640 1 --2- 172.21.15.90:0/3723542330 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5597637429f0 0x559763748650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:59.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.507+0000 7fea29a00640 1 -- 172.21.15.90:0/3723542330 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559763741d10 msgr2=0x559763748110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:41:59.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.507+0000 7fea29a00640 1 --2- 172.21.15.90:0/3723542330 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559763741d10 0x559763748110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:59.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.507+0000 7fea29a00640 1 -- 172.21.15.90:0/3723542330 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5597634847d0 con 0x55976374b080 2024-04-02T14:41:59.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.508+0000 7fea29a00640 1 --2- 172.21.15.90:0/3723542330 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55976374b080 0x559763748b90 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7fea1c010a40 tx=0x7fea1c004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=25b73728a8305531 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:41:59.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.508+0000 7fea0aa00640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:41:59.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.509+0000 7fea0aa00640 1 -- 172.21.15.90:0/3723542330 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea1c01d8f0 con 0x55976374b080 2024-04-02T14:41:59.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.509+0000 7fea0aa00640 1 -- 172.21.15.90:0/3723542330 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7fea1c01da90 con 0x55976374b080 2024-04-02T14:41:59.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.509+0000 7fea2e81b740 1 -- 172.21.15.90:0/3723542330 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fea180dcb10 con 0x55976374b080 2024-04-02T14:41:59.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.510+0000 7fea0aa00640 1 -- 172.21.15.90:0/3723542330 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fea1c01dc90 con 0x55976374b080 2024-04-02T14:41:59.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.510+0000 7fea2e81b740 10 client.27557 fetch_fsmap learned FSMap version 27 2024-04-02T14:41:59.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.510+0000 7fea2e81b740 1 -- 172.21.15.90:0/3723542330 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x559763759f90 con 0x55976374b080 2024-04-02T14:41:59.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.510+0000 7fea0aa00640 1 -- 172.21.15.90:0/3723542330 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fea180dcb10 con 0x55976374b080 2024-04-02T14:41:59.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.510+0000 7fea0aa00640 1 -- 172.21.15.90:0/3723542330 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fea1c01e620 con 0x55976374b080 2024-04-02T14:41:59.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.513+0000 7fea2e81b740 10 client.27557 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:41:59.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.513+0000 7fea2e81b740 10 client.27557 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-04-02T14:41:59.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.513+0000 7fea2e81b740 20 client.27557 populate_metadata read hostname 'smithi090' 2024-04-02T14:41:59.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.513+0000 7fea2e81b740 1 --2- 172.21.15.90:0/3723542330 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x55976375b450 0x55976375d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:59.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.513+0000 7fea2e81b740 4 client.27557 mds_command: new command op to 24547 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:41:59.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.513+0000 7fea2e81b740 1 -- 172.21.15.90:0/3723542330 --> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559763666040 con 0x55976375b450 2024-04-02T14:41:59.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.513+0000 7fea29000640 1 --2- 172.21.15.90:0/3723542330 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x55976375b450 0x55976375d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:59.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.514+0000 7fea29000640 1 --2- 172.21.15.90:0/3723542330 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x55976375b450 0x55976375d8c0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=ceefae49777b6280 server_cookie=243a0d2e2d8701c1 in_seq=0 out_seq=0 2024-04-02T14:41:59.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.514+0000 7fea0aa00640 10 client.27557 ms_handle_connect on v2:172.21.15.177:6838/664895663 2024-04-02T14:41:59.514 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.515+0000 7fea0aa00640 1 -- 172.21.15.90:0/3723542330 <== mds.2 v2:172.21.15.177:6838/664895663 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x559763666040 con 0x55976375b450 2024-04-02T14:41:59.514 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.515+0000 7fea0aa00640 10 client.27557 handle_command_reply: tid=0 2024-04-02T14:41:59.514 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.515+0000 7fea2e81b740 1 client.27557 shutdown 2024-04-02T14:41:59.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.516+0000 7fea2e81b740 1 -- 172.21.15.90:0/3723542330 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55976374b080 msgr2=0x559763748b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:41:59.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.516+0000 7fea2e81b740 1 --2- 172.21.15.90:0/3723542330 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55976374b080 0x559763748b90 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7fea1c010a40 tx=0x7fea1c004a40 comp rx=0 tx=0).stop 2024-04-02T14:41:59.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.517+0000 7fea2e81b740 1 -- 172.21.15.90:0/3723542330 shutdown_connections 2024-04-02T14:41:59.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.517+0000 7fea2e81b740 1 --2- 172.21.15.90:0/3723542330 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x55976375b450 0x55976375d8c0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:59.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.517+0000 7fea2e81b740 1 --2- 172.21.15.90:0/3723542330 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55976374b080 0x559763748b90 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:59.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.517+0000 7fea2e81b740 1 --2- 172.21.15.90:0/3723542330 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5597637429f0 0x559763748650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:59.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.517+0000 7fea2e81b740 1 --2- 172.21.15.90:0/3723542330 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559763741d10 0x559763748110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:59.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.517+0000 7fea2e81b740 1 -- 172.21.15.90:0/3723542330 >> 172.21.15.90:0/3723542330 conn(0x559763708a50 msgr2=0x55976373e4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:41:59.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.517+0000 7fea0aa00640 0 client.27557 ms_handle_reset on v2:172.21.15.177:6838/664895663 2024-04-02T14:41:59.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.517+0000 7fea2e81b740 1 -- 172.21.15.90:0/3723542330 shutdown_connections 2024-04-02T14:41:59.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.517+0000 7fea2e81b740 1 -- 172.21.15.90:0/3723542330 wait complete. 2024-04-02T14:41:59.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.517+0000 7fea2e81b740 20 client.27557 trim_cache size 0 max 16384 2024-04-02T14:41:59.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.544+0000 7fea2e81b740 1 Processor -- start 2024-04-02T14:41:59.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.545+0000 7fea2e81b740 1 -- start start 2024-04-02T14:41:59.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.545+0000 7fea2e81b740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559763741d10 0x55976374a840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:59.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.545+0000 7fea2e81b740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5597637429f0 0x559763756af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:59.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.545+0000 7fea2e81b740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55976374b080 0x5597637479f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:59.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.545+0000 7fea2e81b740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fea241b3df0 con 0x5597637429f0 2024-04-02T14:41:59.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.545+0000 7fea2e81b740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fea241b5930 con 0x559763741d10 2024-04-02T14:41:59.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.545+0000 7fea2e81b740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fea241b44c0 con 0x55976374b080 2024-04-02T14:41:59.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.546+0000 7fea29000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559763741d10 0x55976374a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:59.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.546+0000 7fea29a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55976374b080 0x5597637479f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:59.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.546+0000 7fea29a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55976374b080 0x5597637479f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:56520/0 (socket says 172.21.15.90:56520) 2024-04-02T14:41:59.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.546+0000 7fea29a00640 1 -- 172.21.15.90:0/3601883022 learned_addr learned my addr 172.21.15.90:0/3601883022 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:41:59.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.546+0000 7fea23e00640 1 --2- 172.21.15.90:0/3601883022 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5597637429f0 0x559763756af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:59.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.546+0000 7fea29000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559763741d10 0x55976374a840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:36666/0 (socket says 172.21.15.90:36666) 2024-04-02T14:41:59.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.546+0000 7fea29a00640 1 -- 172.21.15.90:0/3601883022 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559763741d10 msgr2=0x55976374a840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:41:59.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.546+0000 7fea29a00640 1 --2- 172.21.15.90:0/3601883022 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559763741d10 0x55976374a840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:59.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.546+0000 7fea29a00640 1 -- 172.21.15.90:0/3601883022 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5597637429f0 msgr2=0x559763756af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:41:59.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.546+0000 7fea29a00640 1 --2- 172.21.15.90:0/3601883022 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5597637429f0 0x559763756af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:59.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.546+0000 7fea29a00640 1 -- 172.21.15.90:0/3601883022 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559763746660 con 0x55976374b080 2024-04-02T14:41:59.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.546+0000 7fea23e00640 1 --2- 172.21.15.90:0/3601883022 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5597637429f0 0x559763756af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:41:59.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.547+0000 7fea29a00640 1 --2- 172.21.15.90:0/3601883022 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55976374b080 0x5597637479f0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fea1c01ea70 tx=0x7fea1c009e50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f94ee6bff0e6c55d server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:41:59.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.547+0000 7fea15600640 1 -- 172.21.15.90:0/3601883022 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea1c004d70 con 0x55976374b080 2024-04-02T14:41:59.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.548+0000 7fea15600640 1 -- 172.21.15.90:0/3601883022 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fea1c01d6f0 con 0x55976374b080 2024-04-02T14:41:59.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.548+0000 7fea15600640 1 -- 172.21.15.90:0/3601883022 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea1c005630 con 0x55976374b080 2024-04-02T14:41:59.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.549+0000 7fea2e81b740 1 -- 172.21.15.90:0/3601883022 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55976374b080 msgr2=0x5597637479f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:41:59.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.549+0000 7fea2e81b740 1 --2- 172.21.15.90:0/3601883022 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55976374b080 0x5597637479f0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fea1c01ea70 tx=0x7fea1c009e50 comp rx=0 tx=0).stop 2024-04-02T14:41:59.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.550+0000 7fea2e81b740 1 -- 172.21.15.90:0/3601883022 shutdown_connections 2024-04-02T14:41:59.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.550+0000 7fea2e81b740 1 --2- 172.21.15.90:0/3601883022 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55976374b080 0x5597637479f0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:59.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.550+0000 7fea2e81b740 1 --2- 172.21.15.90:0/3601883022 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5597637429f0 0x559763756af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:59.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.550+0000 7fea2e81b740 1 --2- 172.21.15.90:0/3601883022 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559763741d10 0x55976374a840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:59.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.550+0000 7fea2e81b740 1 -- 172.21.15.90:0/3601883022 >> 172.21.15.90:0/3601883022 conn(0x559763708a50 msgr2=0x55976373b8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:41:59.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.551+0000 7fea2e81b740 1 -- 172.21.15.90:0/3601883022 shutdown_connections 2024-04-02T14:41:59.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.551+0000 7fea2e81b740 1 -- 172.21.15.90:0/3601883022 wait complete. 2024-04-02T14:41:59.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.552+0000 7fea2e81b740 1 -- start start 2024-04-02T14:41:59.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.552+0000 7fea2e81b740 1 Processor -- start 2024-04-02T14:41:59.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.552+0000 7fea2e81b740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559763741d10 0x55976374b460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:59.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.552+0000 7fea2e81b740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5597637429f0 0x55976374b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:59.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.552+0000 7fea2e81b740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55976374b080 0x55976374bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:59.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.553+0000 7fea2e81b740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fea241b5260 con 0x5597637429f0 2024-04-02T14:41:59.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.553+0000 7fea2e81b740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fea241b4b90 con 0x559763741d10 2024-04-02T14:41:59.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.553+0000 7fea2e81b740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fea241b1dd0 con 0x55976374b080 2024-04-02T14:41:59.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.553+0000 7fea2e81b740 10 client.0 _ll_register_callbacks cb 0x7fea1c0106f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:41:59.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.553+0000 7fea29000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559763741d10 0x55976374b460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:59.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.553+0000 7fea29000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559763741d10 0x55976374b460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:36670/0 (socket says 172.21.15.90:36670) 2024-04-02T14:41:59.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.553+0000 7fea29000640 1 -- 172.21.15.90:0/649157198 learned_addr learned my addr 172.21.15.90:0/649157198 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:41:59.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.553+0000 7fea29a00640 1 --2- 172.21.15.90:0/649157198 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55976374b080 0x55976374bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:59.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.553+0000 7fea23e00640 1 --2- 172.21.15.90:0/649157198 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5597637429f0 0x55976374b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:59.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.553+0000 7fea29000640 1 -- 172.21.15.90:0/649157198 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55976374b080 msgr2=0x55976374bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:41:59.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.553+0000 7fea29000640 1 --2- 172.21.15.90:0/649157198 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55976374b080 0x55976374bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:59.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.553+0000 7fea29000640 1 -- 172.21.15.90:0/649157198 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5597637429f0 msgr2=0x55976374b9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:41:59.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.553+0000 7fea29000640 1 --2- 172.21.15.90:0/649157198 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5597637429f0 0x55976374b9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:59.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.553+0000 7fea29000640 1 -- 172.21.15.90:0/649157198 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559763752000 con 0x559763741d10 2024-04-02T14:41:59.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.554+0000 7fea0aa00640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:41:59.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.554+0000 7fea29000640 1 --2- 172.21.15.90:0/649157198 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559763741d10 0x55976374b460 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fea1800c6a0 tx=0x7fea180d48b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=49d0b89424ca6918 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:41:59.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.555+0000 7fea0aa00640 1 -- 172.21.15.90:0/649157198 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea180e0040 con 0x559763741d10 2024-04-02T14:41:59.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.555+0000 7fea0aa00640 1 -- 172.21.15.90:0/649157198 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7fea180d4920 con 0x559763741d10 2024-04-02T14:41:59.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.555+0000 7fea2e81b740 1 -- 172.21.15.90:0/649157198 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fea1c01d6f0 con 0x559763741d10 2024-04-02T14:41:59.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.555+0000 7fea0aa00640 1 -- 172.21.15.90:0/649157198 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fea18098d70 con 0x559763741d10 2024-04-02T14:41:59.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.555+0000 7fea2e81b740 10 client.27563 fetch_fsmap learned FSMap version 27 2024-04-02T14:41:59.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.555+0000 7fea2e81b740 1 -- 172.21.15.90:0/649157198 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x559763753300 con 0x559763741d10 2024-04-02T14:41:59.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.555+0000 7fea0aa00640 1 -- 172.21.15.90:0/649157198 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fea1c01d6f0 con 0x559763741d10 2024-04-02T14:41:59.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.555+0000 7fea0aa00640 1 -- 172.21.15.90:0/649157198 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fea18099140 con 0x559763741d10 2024-04-02T14:41:59.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.555+0000 7fea2e81b740 10 client.27563 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:41:59.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.555+0000 7fea2e81b740 10 client.27563 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-04-02T14:41:59.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.555+0000 7fea2e81b740 20 client.27563 populate_metadata read hostname 'smithi090' 2024-04-02T14:41:59.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.555+0000 7fea2e81b740 1 --2- 172.21.15.90:0/649157198 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x5597637535f0 0x5597637c39e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:59.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.555+0000 7fea2e81b740 4 client.27563 mds_command: new command op to 24547 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-02T14:41:59.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.555+0000 7fea2e81b740 1 -- 172.21.15.90:0/649157198 --> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5597636b3c30 con 0x5597637535f0 2024-04-02T14:41:59.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.556+0000 7fea23e00640 1 --2- 172.21.15.90:0/649157198 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x5597637535f0 0x5597637c39e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:59.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.558+0000 7fea0aa00640 10 client.27563 ms_handle_connect on v2:172.21.15.177:6838/664895663 2024-04-02T14:41:59.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.558+0000 7fea23e00640 1 --2- 172.21.15.90:0/649157198 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x5597637535f0 0x5597637c39e0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=6844fcd671548cb4 server_cookie=fa333400866d1c6c in_seq=0 out_seq=0 2024-04-02T14:41:59.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.558+0000 7fea0aa00640 1 -- 172.21.15.90:0/649157198 <== mds.2 v2:172.21.15.177:6838/664895663 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5597636b3c30 con 0x5597637535f0 2024-04-02T14:41:59.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.558+0000 7fea0aa00640 10 client.27563 handle_command_reply: tid=0 2024-04-02T14:41:59.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.559+0000 7fea2e81b740 1 client.27563 shutdown 2024-04-02T14:41:59.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.559+0000 7fea2e81b740 1 -- 172.21.15.90:0/649157198 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559763741d10 msgr2=0x55976374b460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:41:59.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.559+0000 7fea2e81b740 1 --2- 172.21.15.90:0/649157198 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559763741d10 0x55976374b460 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fea1800c6a0 tx=0x7fea180d48b0 comp rx=0 tx=0).stop 2024-04-02T14:41:59.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.559+0000 7fea2e81b740 1 -- 172.21.15.90:0/649157198 shutdown_connections 2024-04-02T14:41:59.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.559+0000 7fea2e81b740 1 --2- 172.21.15.90:0/649157198 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x5597637535f0 0x5597637c39e0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:59.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.559+0000 7fea2e81b740 1 --2- 172.21.15.90:0/649157198 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55976374b080 0x55976374bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:59.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.559+0000 7fea2e81b740 1 --2- 172.21.15.90:0/649157198 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5597637429f0 0x55976374b9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:59.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.559+0000 7fea0aa00640 0 client.27563 ms_handle_reset on v2:172.21.15.177:6838/664895663 2024-04-02T14:41:59.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.559+0000 7fea2e81b740 1 --2- 172.21.15.90:0/649157198 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559763741d10 0x55976374b460 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:59.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.559+0000 7fea2e81b740 1 -- 172.21.15.90:0/649157198 >> 172.21.15.90:0/649157198 conn(0x559763708a50 msgr2=0x55976373b010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:41:59.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.560+0000 7fea2e81b740 1 -- 172.21.15.90:0/649157198 shutdown_connections 2024-04-02T14:41:59.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.560+0000 7fea2e81b740 1 -- 172.21.15.90:0/649157198 wait complete. 2024-04-02T14:41:59.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.560+0000 7fea2e81b740 20 client.27563 trim_cache size 0 max 16384 2024-04-02T14:41:59.560 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-02T14:41:59.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.561+0000 7fea2ba00640 1 -- 172.21.15.90:0/696511514 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fea0c0a3e20 msgr2=0x7fea0c0a62e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:41:59.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.561+0000 7fea2ba00640 1 --2- 172.21.15.90:0/696511514 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fea0c0a3e20 0x7fea0c0a62e0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fea1c005fd0 tx=0x7fea1c009820 comp rx=0 tx=0).stop 2024-04-02T14:41:59.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.561+0000 7fea2ba00640 1 -- 172.21.15.90:0/696511514 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fea24108310 msgr2=0x7fea241a4470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:41:59.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.561+0000 7fea2ba00640 1 --2- 172.21.15.90:0/696511514 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fea24108310 0x7fea241a4470 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fea1800e9f0 tx=0x7fea1800eeb0 comp rx=0 tx=0).stop 2024-04-02T14:41:59.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.561+0000 7fea2ba00640 1 -- 172.21.15.90:0/696511514 shutdown_connections 2024-04-02T14:41:59.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.561+0000 7fea2ba00640 1 --2- 172.21.15.90:0/696511514 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fea0c0a3e20 0x7fea0c0a62e0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:59.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.561+0000 7fea2ba00640 1 --2- 172.21.15.90:0/696511514 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fea2419e620 0x7fea2419ead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:59.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.561+0000 7fea2ba00640 1 --2- 172.21.15.90:0/696511514 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fea2410d3e0 0x7fea241a49b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:59.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.561+0000 7fea2ba00640 1 --2- 172.21.15.90:0/696511514 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fea24108310 0x7fea241a4470 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:59.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.561+0000 7fea2ba00640 1 -- 172.21.15.90:0/696511514 >> 172.21.15.90:0/696511514 conn(0x7fea24076040 msgr2=0x7fea24076960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:41:59.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.561+0000 7fea2ba00640 1 -- 172.21.15.90:0/696511514 shutdown_connections 2024-04-02T14:41:59.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.561+0000 7fea2ba00640 1 -- 172.21.15.90:0/696511514 wait complete. 2024-04-02T14:41:59.584 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-02T14:41:59.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.850+0000 7fa6b2800640 1 -- 172.21.15.90:0/1496998794 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa6ac107b50 msgr2=0x7fa6ac107f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:41:59.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.850+0000 7fa6b2800640 1 --2- 172.21.15.90:0/1496998794 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa6ac107b50 0x7fa6ac107f30 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fa69c009890 tx=0x7fa69c02bb70 comp rx=0 tx=0).stop 2024-04-02T14:41:59.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.851+0000 7fa6b2800640 1 -- 172.21.15.90:0/1496998794 shutdown_connections 2024-04-02T14:41:59.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.851+0000 7fa6b2800640 1 --2- 172.21.15.90:0/1496998794 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa6ac0696c0 0x7fa6ac10e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:59.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.851+0000 7fa6b2800640 1 --2- 172.21.15.90:0/1496998794 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa6ac068d00 0x7fa6ac069180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:59.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.851+0000 7fa6b2800640 1 --2- 172.21.15.90:0/1496998794 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa6ac107b50 0x7fa6ac107f30 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:59.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.851+0000 7fa6b2800640 1 -- 172.21.15.90:0/1496998794 >> 172.21.15.90:0/1496998794 conn(0x7fa6ac077b00 msgr2=0x7fa6ac077f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:41:59.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.852+0000 7fa6b2800640 1 -- 172.21.15.90:0/1496998794 shutdown_connections 2024-04-02T14:41:59.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.852+0000 7fa6b2800640 1 -- 172.21.15.90:0/1496998794 wait complete. 2024-04-02T14:41:59.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.852+0000 7fa6b2800640 1 Processor -- start 2024-04-02T14:41:59.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.853+0000 7fa6b2800640 1 -- start start 2024-04-02T14:41:59.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.853+0000 7fa6b2800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa6ac068d00 0x7fa6ac1089c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:59.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.853+0000 7fa6b2800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa6ac0696c0 0x7fa6ac108f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:59.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.853+0000 7fa6b2800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa6ac107b50 0x7fa6ac109440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:59.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.854+0000 7fa6b2800640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fa6ac111ea0 con 0x7fa6ac0696c0 2024-04-02T14:41:59.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.854+0000 7fa6b2800640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fa6ac111d20 con 0x7fa6ac068d00 2024-04-02T14:41:59.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.854+0000 7fa6b2800640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fa6ac112020 con 0x7fa6ac107b50 2024-04-02T14:41:59.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.854+0000 7fa6ab400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa6ac068d00 0x7fa6ac1089c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:59.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.854+0000 7fa6aaa00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa6ac0696c0 0x7fa6ac108f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:59.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.854+0000 7fa6ab400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa6ac068d00 0x7fa6ac1089c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:36686/0 (socket says 172.21.15.90:36686) 2024-04-02T14:41:59.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.854+0000 7fa6ab400640 1 -- 172.21.15.90:0/2911905732 learned_addr learned my addr 172.21.15.90:0/2911905732 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:41:59.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.854+0000 7fa6abe00640 1 --2- 172.21.15.90:0/2911905732 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa6ac107b50 0x7fa6ac109440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:59.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.855+0000 7fa6abe00640 1 -- 172.21.15.90:0/2911905732 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa6ac068d00 msgr2=0x7fa6ac1089c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:41:59.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.855+0000 7fa6abe00640 1 --2- 172.21.15.90:0/2911905732 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa6ac068d00 0x7fa6ac1089c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:59.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.855+0000 7fa6abe00640 1 -- 172.21.15.90:0/2911905732 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa6ac0696c0 msgr2=0x7fa6ac108f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:41:59.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.855+0000 7fa6abe00640 1 --2- 172.21.15.90:0/2911905732 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa6ac0696c0 0x7fa6ac108f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:59.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.855+0000 7fa6abe00640 1 -- 172.21.15.90:0/2911905732 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6ac1aabd0 con 0x7fa6ac107b50 2024-04-02T14:41:59.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.855+0000 7fa6aaa00640 1 --2- 172.21.15.90:0/2911905732 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa6ac0696c0 0x7fa6ac108f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:41:59.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.855+0000 7fa6ab400640 1 --2- 172.21.15.90:0/2911905732 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa6ac068d00 0x7fa6ac1089c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:41:59.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.856+0000 7fa6abe00640 1 --2- 172.21.15.90:0/2911905732 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa6ac107b50 0x7fa6ac109440 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fa6a400cab0 tx=0x7fa6a400ceb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:41:59.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.856+0000 7fa6a3e00640 1 -- 172.21.15.90:0/2911905732 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6a400ad60 con 0x7fa6ac107b50 2024-04-02T14:41:59.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.856+0000 7fa6b2800640 1 -- 172.21.15.90:0/2911905732 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6ac1aaec0 con 0x7fa6ac107b50 2024-04-02T14:41:59.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.856+0000 7fa6b2800640 1 -- 172.21.15.90:0/2911905732 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6ac1ab3e0 con 0x7fa6ac107b50 2024-04-02T14:41:59.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.857+0000 7fa6a3e00640 1 -- 172.21.15.90:0/2911905732 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa6a400f930 con 0x7fa6ac107b50 2024-04-02T14:41:59.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.857+0000 7fa6a3e00640 1 -- 172.21.15.90:0/2911905732 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6a400ad60 con 0x7fa6ac107b50 2024-04-02T14:41:59.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.860+0000 7fa6b55d7740 1 Processor -- start 2024-04-02T14:41:59.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.860+0000 7fa6a3e00640 1 -- 172.21.15.90:0/2911905732 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fa6a400e780 con 0x7fa6ac107b50 2024-04-02T14:41:59.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.861+0000 7fa6b55d7740 1 -- start start 2024-04-02T14:41:59.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.862+0000 7fa6b55d7740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5613e17b99f0 0x5613e17b8740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:59.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.862+0000 7fa6b55d7740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5613e17b8d10 0x5613e17b9190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:59.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.862+0000 7fa6b55d7740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5613e17c2080 0x5613e17c4470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:59.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.862+0000 7fa6b55d7740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5613e16f9400 con 0x5613e17b99f0 2024-04-02T14:41:59.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.862+0000 7fa6b55d7740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x5613e16ddf10 con 0x5613e17c2080 2024-04-02T14:41:59.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.862+0000 7fa6b55d7740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5613e171d6f0 con 0x5613e17b8d10 2024-04-02T14:41:59.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.863+0000 7fa6ab400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5613e17b8d10 0x5613e17b9190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:59.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.863+0000 7fa6aaa00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5613e17b99f0 0x5613e17b8740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:59.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.863+0000 7fa6a3e00640 1 --2- 172.21.15.90:0/2911905732 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fa6900a4020 0x7fa6900a64e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:59.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.863+0000 7fa6aaa00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5613e17b99f0 0x5613e17b8740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38196/0 (socket says 172.21.15.90:38196) 2024-04-02T14:41:59.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.863+0000 7fa6aaa00640 1 -- 172.21.15.90:0/2329146670 learned_addr learned my addr 172.21.15.90:0/2329146670 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:41:59.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.863+0000 7fa6aaa00640 1 --2- 172.21.15.90:0/2329146670 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5613e17c2080 0x5613e17c4470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:59.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.863+0000 7fa6a3e00640 1 -- 172.21.15.90:0/2911905732 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7fa6a40d0a10 con 0x7fa6ac107b50 2024-04-02T14:41:59.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.863+0000 7fa6abe00640 1 --2- 172.21.15.90:0/2911905732 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fa6900a4020 0x7fa6900a64e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:59.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.863+0000 7fa6aaa00640 1 -- 172.21.15.90:0/2329146670 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5613e17b8d10 msgr2=0x5613e17b9190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:41:59.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.863+0000 7fa6aaa00640 1 --2- 172.21.15.90:0/2329146670 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5613e17b8d10 0x5613e17b9190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:59.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.863+0000 7fa6aaa00640 1 -- 172.21.15.90:0/2329146670 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5613e17b99f0 msgr2=0x5613e17b8740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:41:59.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.863+0000 7fa6aaa00640 1 --2- 172.21.15.90:0/2329146670 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5613e17b99f0 0x5613e17b8740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:59.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.863+0000 7fa6aaa00640 1 -- 172.21.15.90:0/2329146670 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5613e14313f0 con 0x5613e17c2080 2024-04-02T14:41:59.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.864+0000 7fa6aaa00640 1 --2- 172.21.15.90:0/2329146670 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5613e17c2080 0x5613e17c4470 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7fa698013be0 tx=0x7fa698004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c7d08929787b3ff2 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:41:59.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.864+0000 7fa6a0c00640 1 -- 172.21.15.90:0/2329146670 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa69800ec40 con 0x5613e17c2080 2024-04-02T14:41:59.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.864+0000 7fa6a0c00640 1 -- 172.21.15.90:0/2329146670 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa69800ede0 con 0x5613e17c2080 2024-04-02T14:41:59.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.864+0000 7fa6a0c00640 1 -- 172.21.15.90:0/2329146670 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa698010460 con 0x5613e17c2080 2024-04-02T14:41:59.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.867+0000 7fa6b55d7740 1 -- 172.21.15.90:0/2329146670 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5613e17c2080 msgr2=0x5613e17c4470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:41:59.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.867+0000 7fa6abe00640 1 --2- 172.21.15.90:0/2911905732 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fa6900a4020 0x7fa6900a64e0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fa6a401f000 tx=0x7fa6a400a7d0 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:41:59.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.867+0000 7fa6b55d7740 1 --2- 172.21.15.90:0/2329146670 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5613e17c2080 0x5613e17c4470 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7fa698013be0 tx=0x7fa698004730 comp rx=0 tx=0).stop 2024-04-02T14:41:59.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.867+0000 7fa6b55d7740 1 -- 172.21.15.90:0/2329146670 shutdown_connections 2024-04-02T14:41:59.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.867+0000 7fa6b55d7740 1 --2- 172.21.15.90:0/2329146670 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5613e17c2080 0x5613e17c4470 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:59.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.867+0000 7fa6b55d7740 1 --2- 172.21.15.90:0/2329146670 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5613e17b8d10 0x5613e17b9190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:59.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.867+0000 7fa6b55d7740 1 --2- 172.21.15.90:0/2329146670 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5613e17b99f0 0x5613e17b8740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:59.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.867+0000 7fa6b55d7740 1 -- 172.21.15.90:0/2329146670 >> 172.21.15.90:0/2329146670 conn(0x5613e177fa50 msgr2=0x5613e17b3760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:41:59.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.867+0000 7fa6b55d7740 1 -- 172.21.15.90:0/2329146670 shutdown_connections 2024-04-02T14:41:59.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.867+0000 7fa6b55d7740 1 -- 172.21.15.90:0/2329146670 wait complete. 2024-04-02T14:41:59.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.868+0000 7fa6b55d7740 1 -- start start 2024-04-02T14:41:59.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.868+0000 7fa6b55d7740 1 Processor -- start 2024-04-02T14:41:59.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.868+0000 7fa6b55d7740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5613e17b8d10 0x5613e17bf100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:59.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.869+0000 7fa6b55d7740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5613e17b99f0 0x5613e17bf640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:59.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.869+0000 7fa6b55d7740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5613e17bfb80 0x5613e17df140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:59.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.869+0000 7fa6b55d7740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5613e17196a0 con 0x5613e17b99f0 2024-04-02T14:41:59.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.869+0000 7fa6b55d7740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x5613e16c5980 con 0x5613e17b8d10 2024-04-02T14:41:59.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.869+0000 7fa6b55d7740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5613e16d8cb0 con 0x5613e17bfb80 2024-04-02T14:41:59.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.869+0000 7fa6b55d7740 10 client.0 _ll_register_callbacks cb 0x5613e16d86a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:41:59.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.869+0000 7fa6aaa00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5613e17b8d10 0x5613e17bf100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:59.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.869+0000 7fa6aaa00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5613e17b8d10 0x5613e17bf100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:36718/0 (socket says 172.21.15.90:36718) 2024-04-02T14:41:59.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.869+0000 7fa6aaa00640 1 -- 172.21.15.90:0/3154192878 learned_addr learned my addr 172.21.15.90:0/3154192878 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:41:59.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.869+0000 7fa6aaa00640 1 --2- 172.21.15.90:0/3154192878 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5613e17bfb80 0x5613e17df140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:59.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.870+0000 7fa6ab400640 1 --2- 172.21.15.90:0/3154192878 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5613e17b99f0 0x5613e17bf640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:59.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.870+0000 7fa6aaa00640 1 -- 172.21.15.90:0/3154192878 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5613e17bfb80 msgr2=0x5613e17df140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:41:59.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.870+0000 7fa6aaa00640 1 --2- 172.21.15.90:0/3154192878 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5613e17bfb80 0x5613e17df140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:59.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.870+0000 7fa6aaa00640 1 -- 172.21.15.90:0/3154192878 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5613e17b99f0 msgr2=0x5613e17bf640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:41:59.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.870+0000 7fa6aaa00640 1 --2- 172.21.15.90:0/3154192878 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5613e17b99f0 0x5613e17bf640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:59.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.870+0000 7fa6aaa00640 1 -- 172.21.15.90:0/3154192878 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5613e14fb7d0 con 0x5613e17b8d10 2024-04-02T14:41:59.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.870+0000 7fa6ab400640 1 --2- 172.21.15.90:0/3154192878 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5613e17b99f0 0x5613e17bf640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:41:59.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.870+0000 7fa6aaa00640 1 --2- 172.21.15.90:0/3154192878 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5613e17b8d10 0x5613e17bf100 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7fa698009bd0 tx=0x7fa698003a90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f2a2ac7be14d1add server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:41:59.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.871+0000 7fa695600640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:41:59.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.871+0000 7fa695600640 1 -- 172.21.15.90:0/3154192878 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6980035f0 con 0x5613e17b8d10 2024-04-02T14:41:59.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.871+0000 7fa695600640 1 -- 172.21.15.90:0/3154192878 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7fa698005020 con 0x5613e17b8d10 2024-04-02T14:41:59.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.871+0000 7fa6b55d7740 1 -- 172.21.15.90:0/3154192878 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa69800ede0 con 0x5613e17b8d10 2024-04-02T14:41:59.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.872+0000 7fa695600640 1 -- 172.21.15.90:0/3154192878 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fa698034050 con 0x5613e17b8d10 2024-04-02T14:41:59.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.872+0000 7fa6b55d7740 10 client.27575 fetch_fsmap learned FSMap version 27 2024-04-02T14:41:59.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.872+0000 7fa6b55d7740 1 -- 172.21.15.90:0/3154192878 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5613e17e24c0 con 0x5613e17b8d10 2024-04-02T14:41:59.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.872+0000 7fa695600640 1 -- 172.21.15.90:0/3154192878 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa69800ede0 con 0x5613e17b8d10 2024-04-02T14:41:59.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.872+0000 7fa695600640 1 -- 172.21.15.90:0/3154192878 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fa698004810 con 0x5613e17b8d10 2024-04-02T14:41:59.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.873+0000 7fa6b55d7740 10 client.27575 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:41:59.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.873+0000 7fa6b55d7740 10 client.27575 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2024-04-02T14:41:59.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.873+0000 7fa6b55d7740 20 client.27575 populate_metadata read hostname 'smithi090' 2024-04-02T14:41:59.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.873+0000 7fa6b55d7740 1 --2- 172.21.15.90:0/3154192878 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x5613e17e3810 0x5613e17e5c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:59.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.873+0000 7fa6b55d7740 4 client.27575 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:41:59.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.873+0000 7fa6b55d7740 1 -- 172.21.15.90:0/3154192878 --> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5613e16dd040 con 0x5613e17e3810 2024-04-02T14:41:59.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.873+0000 7fa6ab400640 1 --2- 172.21.15.90:0/3154192878 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x5613e17e3810 0x5613e17e5c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:59.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.875+0000 7fa6ab400640 1 --2- 172.21.15.90:0/3154192878 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x5613e17e3810 0x5613e17e5c00 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=58e490b6a820dbae server_cookie=619e6c17491cb006 in_seq=0 out_seq=0 2024-04-02T14:41:59.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.875+0000 7fa695600640 10 client.27575 ms_handle_connect on v2:172.21.15.90:6838/1498915870 2024-04-02T14:41:59.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.876+0000 7fa695600640 1 -- 172.21.15.90:0/3154192878 <== mds.1 v2:172.21.15.90:6838/1498915870 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5613e16dd040 con 0x5613e17e3810 2024-04-02T14:41:59.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.876+0000 7fa695600640 10 client.27575 handle_command_reply: tid=0 2024-04-02T14:41:59.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.876+0000 7fa6b55d7740 1 client.27575 shutdown 2024-04-02T14:41:59.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.876+0000 7fa6b55d7740 1 -- 172.21.15.90:0/3154192878 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5613e17b8d10 msgr2=0x5613e17bf100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:41:59.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.876+0000 7fa6b55d7740 1 --2- 172.21.15.90:0/3154192878 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5613e17b8d10 0x5613e17bf100 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7fa698009bd0 tx=0x7fa698003a90 comp rx=0 tx=0).stop 2024-04-02T14:41:59.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.876+0000 7fa6b55d7740 1 -- 172.21.15.90:0/3154192878 shutdown_connections 2024-04-02T14:41:59.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.876+0000 7fa6b55d7740 1 --2- 172.21.15.90:0/3154192878 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x5613e17e3810 0x5613e17e5c00 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:59.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.876+0000 7fa6b55d7740 1 --2- 172.21.15.90:0/3154192878 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5613e17bfb80 0x5613e17df140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:59.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.876+0000 7fa6b55d7740 1 --2- 172.21.15.90:0/3154192878 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5613e17b99f0 0x5613e17bf640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:59.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.876+0000 7fa6b55d7740 1 --2- 172.21.15.90:0/3154192878 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5613e17b8d10 0x5613e17bf100 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:59.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.876+0000 7fa6b55d7740 1 -- 172.21.15.90:0/3154192878 >> 172.21.15.90:0/3154192878 conn(0x5613e177fa50 msgr2=0x5613e17b54b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:41:59.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.876+0000 7fa695600640 0 client.27575 ms_handle_reset on v2:172.21.15.90:6838/1498915870 2024-04-02T14:41:59.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.877+0000 7fa6b55d7740 1 -- 172.21.15.90:0/3154192878 shutdown_connections 2024-04-02T14:41:59.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.877+0000 7fa6b55d7740 1 -- 172.21.15.90:0/3154192878 wait complete. 2024-04-02T14:41:59.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.877+0000 7fa6b55d7740 20 client.27575 trim_cache size 0 max 16384 2024-04-02T14:41:59.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.906+0000 7fa6b55d7740 1 Processor -- start 2024-04-02T14:41:59.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.906+0000 7fa6b55d7740 1 -- start start 2024-04-02T14:41:59.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.906+0000 7fa6b55d7740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5613e17b8d10 0x5613e17b2720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:59.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.906+0000 7fa6b55d7740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5613e17b99f0 0x5613e17eb3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:59.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.906+0000 7fa6b55d7740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5613e17bfb80 0x5613e17eb910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:59.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.906+0000 7fa6b55d7740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fa68c00e810 con 0x5613e17b8d10 2024-04-02T14:41:59.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.906+0000 7fa6b55d7740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fa68c010350 con 0x5613e17bfb80 2024-04-02T14:41:59.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.906+0000 7fa6b55d7740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fa68c00eee0 con 0x5613e17b99f0 2024-04-02T14:41:59.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.907+0000 7fa6aaa00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5613e17b8d10 0x5613e17b2720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:59.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.907+0000 7fa6aaa00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5613e17b8d10 0x5613e17b2720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38218/0 (socket says 172.21.15.90:38218) 2024-04-02T14:41:59.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.907+0000 7fa6aaa00640 1 -- 172.21.15.90:0/3247831304 learned_addr learned my addr 172.21.15.90:0/3247831304 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:41:59.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.907+0000 7fa6aaa00640 1 --2- 172.21.15.90:0/3247831304 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5613e17bfb80 0x5613e17eb910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:59.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.907+0000 7fa6ab400640 1 --2- 172.21.15.90:0/3247831304 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5613e17b99f0 0x5613e17eb3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:59.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.907+0000 7fa6aaa00640 1 -- 172.21.15.90:0/3247831304 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5613e17b99f0 msgr2=0x5613e17eb3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:41:59.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.907+0000 7fa6aaa00640 1 --2- 172.21.15.90:0/3247831304 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5613e17b99f0 0x5613e17eb3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:59.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.907+0000 7fa6aaa00640 1 -- 172.21.15.90:0/3247831304 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5613e17b8d10 msgr2=0x5613e17b2720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:41:59.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.907+0000 7fa6aaa00640 1 --2- 172.21.15.90:0/3247831304 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5613e17b8d10 0x5613e17b2720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:59.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.907+0000 7fa6aaa00640 1 -- 172.21.15.90:0/3247831304 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5613e17b7d40 con 0x5613e17bfb80 2024-04-02T14:41:59.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.907+0000 7fa6ab400640 1 --2- 172.21.15.90:0/3247831304 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5613e17b99f0 0x5613e17eb3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:41:59.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.908+0000 7fa6aaa00640 1 --2- 172.21.15.90:0/3247831304 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5613e17bfb80 0x5613e17eb910 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7fa698004c10 tx=0x7fa698000bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c2fdb2b0afc95105 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:41:59.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.908+0000 7fa697e00640 1 -- 172.21.15.90:0/3247831304 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa698003b00 con 0x5613e17bfb80 2024-04-02T14:41:59.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.908+0000 7fa697e00640 1 -- 172.21.15.90:0/3247831304 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa6980038c0 con 0x5613e17bfb80 2024-04-02T14:41:59.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.908+0000 7fa697e00640 1 -- 172.21.15.90:0/3247831304 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa69800ec10 con 0x5613e17bfb80 2024-04-02T14:41:59.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.910+0000 7fa6b55d7740 1 -- 172.21.15.90:0/3247831304 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5613e17bfb80 msgr2=0x5613e17eb910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:41:59.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.910+0000 7fa6b55d7740 1 --2- 172.21.15.90:0/3247831304 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5613e17bfb80 0x5613e17eb910 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7fa698004c10 tx=0x7fa698000bc0 comp rx=0 tx=0).stop 2024-04-02T14:41:59.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.910+0000 7fa6b55d7740 1 -- 172.21.15.90:0/3247831304 shutdown_connections 2024-04-02T14:41:59.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.910+0000 7fa6b55d7740 1 --2- 172.21.15.90:0/3247831304 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5613e17bfb80 0x5613e17eb910 secure :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7fa698004c10 tx=0x7fa698000bc0 comp rx=0 tx=0).stop 2024-04-02T14:41:59.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.910+0000 7fa6b55d7740 1 --2- 172.21.15.90:0/3247831304 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5613e17b99f0 0x5613e17eb3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:59.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.910+0000 7fa6b55d7740 1 --2- 172.21.15.90:0/3247831304 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5613e17b8d10 0x5613e17b2720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:59.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.910+0000 7fa6b55d7740 1 -- 172.21.15.90:0/3247831304 >> 172.21.15.90:0/3247831304 conn(0x5613e177fa50 msgr2=0x5613e17b2010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:41:59.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.911+0000 7fa6b55d7740 1 -- 172.21.15.90:0/3247831304 shutdown_connections 2024-04-02T14:41:59.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.911+0000 7fa6b55d7740 1 -- 172.21.15.90:0/3247831304 wait complete. 2024-04-02T14:41:59.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.912+0000 7fa6b55d7740 1 -- start start 2024-04-02T14:41:59.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.912+0000 7fa6b55d7740 1 Processor -- start 2024-04-02T14:41:59.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.914+0000 7fa6b55d7740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5613e17b8d10 0x5613e17c2a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:59.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.914+0000 7fa6b55d7740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5613e17b99f0 0x5613e17c2fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:59.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.914+0000 7fa6b55d7740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5613e17da0a0 0x5613e17da4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:59.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.914+0000 7fa6b55d7740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fa68c00fc80 con 0x5613e17b99f0 2024-04-02T14:41:59.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.914+0000 7fa6b55d7740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fa68c00f5b0 con 0x5613e17b8d10 2024-04-02T14:41:59.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.914+0000 7fa6b55d7740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fa68c00c7f0 con 0x5613e17da0a0 2024-04-02T14:41:59.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.914+0000 7fa6b55d7740 10 client.0 _ll_register_callbacks cb 0x7fa698004140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:41:59.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.914+0000 7fa6ab400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5613e17b99f0 0x5613e17c2fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:59.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.914+0000 7fa6aaa00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5613e17b8d10 0x5613e17c2a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:59.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.914+0000 7fa6aaa00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5613e17b8d10 0x5613e17c2a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:36744/0 (socket says 172.21.15.90:36744) 2024-04-02T14:41:59.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.914+0000 7fa6aaa00640 1 -- 172.21.15.90:0/4055667497 learned_addr learned my addr 172.21.15.90:0/4055667497 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:41:59.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.915+0000 7fa6aaa00640 1 --2- 172.21.15.90:0/4055667497 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5613e17da0a0 0x5613e17da4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:59.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.915+0000 7fa6aaa00640 1 -- 172.21.15.90:0/4055667497 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5613e17da0a0 msgr2=0x5613e17da4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:41:59.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.915+0000 7fa6aaa00640 1 --2- 172.21.15.90:0/4055667497 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5613e17da0a0 0x5613e17da4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:59.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.915+0000 7fa6aaa00640 1 -- 172.21.15.90:0/4055667497 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5613e17b99f0 msgr2=0x5613e17c2fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:41:59.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.915+0000 7fa6aaa00640 1 --2- 172.21.15.90:0/4055667497 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5613e17b99f0 0x5613e17c2fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:59.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.915+0000 7fa6aaa00640 1 -- 172.21.15.90:0/4055667497 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5613e17daa20 con 0x5613e17b8d10 2024-04-02T14:41:59.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.915+0000 7fa6ab400640 1 --2- 172.21.15.90:0/4055667497 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5613e17b99f0 0x5613e17c2fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:41:59.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.916+0000 7fa6aaa00640 1 --2- 172.21.15.90:0/4055667497 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5613e17b8d10 0x5613e17c2a80 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7fa698009490 tx=0x7fa698005850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d5551d0a387a2fda server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:41:59.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.916+0000 7fa695600640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:41:59.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.916+0000 7fa695600640 1 -- 172.21.15.90:0/4055667497 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa69800ee10 con 0x5613e17b8d10 2024-04-02T14:41:59.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.916+0000 7fa695600640 1 -- 172.21.15.90:0/4055667497 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7fa698004490 con 0x5613e17b8d10 2024-04-02T14:41:59.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.916+0000 7fa6b55d7740 1 -- 172.21.15.90:0/4055667497 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa6980038c0 con 0x5613e17b8d10 2024-04-02T14:41:59.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.916+0000 7fa695600640 1 -- 172.21.15.90:0/4055667497 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fa6980035f0 con 0x5613e17b8d10 2024-04-02T14:41:59.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.917+0000 7fa6b55d7740 10 client.27581 fetch_fsmap learned FSMap version 27 2024-04-02T14:41:59.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.917+0000 7fa6b55d7740 1 -- 172.21.15.90:0/4055667497 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5613e1849d90 con 0x5613e17b8d10 2024-04-02T14:41:59.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.917+0000 7fa695600640 1 -- 172.21.15.90:0/4055667497 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa6980038c0 con 0x5613e17b8d10 2024-04-02T14:41:59.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.917+0000 7fa695600640 1 -- 172.21.15.90:0/4055667497 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fa6980106e0 con 0x5613e17b8d10 2024-04-02T14:41:59.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.917+0000 7fa6b55d7740 10 client.27581 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:41:59.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.917+0000 7fa6b55d7740 10 client.27581 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2024-04-02T14:41:59.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.917+0000 7fa6b55d7740 20 client.27581 populate_metadata read hostname 'smithi090' 2024-04-02T14:41:59.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.917+0000 7fa6b55d7740 1 --2- 172.21.15.90:0/4055667497 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x5613e184a250 0x5613e184c640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:41:59.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.917+0000 7fa6b55d7740 4 client.27581 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-02T14:41:59.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.917+0000 7fa6b55d7740 1 -- 172.21.15.90:0/4055667497 --> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5613e172ac30 con 0x5613e184a250 2024-04-02T14:41:59.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.918+0000 7fa6ab400640 1 --2- 172.21.15.90:0/4055667497 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x5613e184a250 0x5613e184c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:41:59.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.919+0000 7fa6ab400640 1 --2- 172.21.15.90:0/4055667497 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x5613e184a250 0x5613e184c640 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=7c614b613db47683 server_cookie=fe829016617f0e7 in_seq=0 out_seq=0 2024-04-02T14:41:59.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.919+0000 7fa695600640 10 client.27581 ms_handle_connect on v2:172.21.15.90:6838/1498915870 2024-04-02T14:41:59.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.920+0000 7fa695600640 1 -- 172.21.15.90:0/4055667497 <== mds.1 v2:172.21.15.90:6838/1498915870 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5613e172ac30 con 0x5613e184a250 2024-04-02T14:41:59.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.920+0000 7fa695600640 10 client.27581 handle_command_reply: tid=0 2024-04-02T14:41:59.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.920+0000 7fa6b55d7740 1 client.27581 shutdown 2024-04-02T14:41:59.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.921+0000 7fa6b55d7740 1 -- 172.21.15.90:0/4055667497 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5613e17b8d10 msgr2=0x5613e17c2a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:41:59.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.921+0000 7fa6b55d7740 1 --2- 172.21.15.90:0/4055667497 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5613e17b8d10 0x5613e17c2a80 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7fa698009490 tx=0x7fa698005850 comp rx=0 tx=0).stop 2024-04-02T14:41:59.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.921+0000 7fa6b55d7740 1 -- 172.21.15.90:0/4055667497 shutdown_connections 2024-04-02T14:41:59.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.921+0000 7fa6b55d7740 1 --2- 172.21.15.90:0/4055667497 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x5613e184a250 0x5613e184c640 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:59.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.921+0000 7fa6b55d7740 1 --2- 172.21.15.90:0/4055667497 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5613e17da0a0 0x5613e17da4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:59.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.921+0000 7fa6b55d7740 1 --2- 172.21.15.90:0/4055667497 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5613e17b99f0 0x5613e17c2fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:59.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.921+0000 7fa6b55d7740 1 --2- 172.21.15.90:0/4055667497 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5613e17b8d10 0x5613e17c2a80 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:59.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.921+0000 7fa6b55d7740 1 -- 172.21.15.90:0/4055667497 >> 172.21.15.90:0/4055667497 conn(0x5613e177fa50 msgr2=0x5613e17b28a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:41:59.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.921+0000 7fa695600640 0 client.27581 ms_handle_reset on v2:172.21.15.90:6838/1498915870 2024-04-02T14:41:59.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.922+0000 7fa6b55d7740 1 -- 172.21.15.90:0/4055667497 shutdown_connections 2024-04-02T14:41:59.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.922+0000 7fa6b55d7740 1 -- 172.21.15.90:0/4055667497 wait complete. 2024-04-02T14:41:59.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.922+0000 7fa6b55d7740 20 client.27581 trim_cache size 0 max 16384 2024-04-02T14:41:59.923 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-02T14:41:59.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.924+0000 7fa6b2800640 1 -- 172.21.15.90:0/2911905732 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fa6900a4020 msgr2=0x7fa6900a64e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:41:59.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.924+0000 7fa6b2800640 1 --2- 172.21.15.90:0/2911905732 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fa6900a4020 0x7fa6900a64e0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fa6a401f000 tx=0x7fa6a400a7d0 comp rx=0 tx=0).stop 2024-04-02T14:41:59.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.924+0000 7fa6b2800640 1 -- 172.21.15.90:0/2911905732 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa6ac107b50 msgr2=0x7fa6ac109440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:41:59.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.924+0000 7fa6b2800640 1 --2- 172.21.15.90:0/2911905732 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa6ac107b50 0x7fa6ac109440 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fa6a400cab0 tx=0x7fa6a400ceb0 comp rx=0 tx=0).stop 2024-04-02T14:41:59.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.925+0000 7fa6b2800640 1 -- 172.21.15.90:0/2911905732 shutdown_connections 2024-04-02T14:41:59.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.925+0000 7fa6b2800640 1 --2- 172.21.15.90:0/2911905732 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fa6900a4020 0x7fa6900a64e0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:59.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.925+0000 7fa6b2800640 1 --2- 172.21.15.90:0/2911905732 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa6ac107b50 0x7fa6ac109440 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:59.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.925+0000 7fa6b2800640 1 --2- 172.21.15.90:0/2911905732 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa6ac0696c0 0x7fa6ac108f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:59.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.925+0000 7fa6b2800640 1 --2- 172.21.15.90:0/2911905732 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa6ac068d00 0x7fa6ac1089c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:41:59.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.925+0000 7fa6b2800640 1 -- 172.21.15.90:0/2911905732 >> 172.21.15.90:0/2911905732 conn(0x7fa6ac077b00 msgr2=0x7fa6ac10e630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:41:59.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.926+0000 7fa6b2800640 1 -- 172.21.15.90:0/2911905732 shutdown_connections 2024-04-02T14:41:59.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:41:59.926+0000 7fa6b2800640 1 -- 172.21.15.90:0/2911905732 wait complete. 2024-04-02T14:41:59.937 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:41:59 smithi090 ceph-mon[31674]: pgmap v216: 97 pgs: 97 active+clean; 4.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 8.3 MiB/s wr, 806 op/s 2024-04-02T14:41:59.961 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-02T14:42:00.108 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:41:59 smithi092 ceph-mon[36497]: pgmap v216: 97 pgs: 97 active+clean; 4.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 8.3 MiB/s wr, 806 op/s 2024-04-02T14:42:00.184 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:41:59 smithi177 ceph-mon[36431]: pgmap v216: 97 pgs: 97 active+clean; 4.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 8.3 MiB/s wr, 806 op/s 2024-04-02T14:42:00.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.230+0000 7fad25400640 1 -- 172.21.15.90:0/3572038541 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fad20100d30 msgr2=0x7fad20104d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:00.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.230+0000 7fad25400640 1 --2- 172.21.15.90:0/3572038541 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fad20100d30 0x7fad20104d70 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fad0c00b9f0 tx=0x7fad0c02d8f0 comp rx=0 tx=0).stop 2024-04-02T14:42:00.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.230+0000 7fad25400640 1 -- 172.21.15.90:0/3572038541 shutdown_connections 2024-04-02T14:42:00.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.230+0000 7fad25400640 1 --2- 172.21.15.90:0/3572038541 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fad20105450 0x7fad2010d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:00.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.230+0000 7fad25400640 1 --2- 172.21.15.90:0/3572038541 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fad20100d30 0x7fad20104d70 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:00.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.230+0000 7fad25400640 1 --2- 172.21.15.90:0/3572038541 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fad20100380 0x7fad20100760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:00.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.230+0000 7fad25400640 1 -- 172.21.15.90:0/3572038541 >> 172.21.15.90:0/3572038541 conn(0x7fad20076f20 msgr2=0x7fad200fe2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:42:00.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.230+0000 7fad25400640 1 -- 172.21.15.90:0/3572038541 shutdown_connections 2024-04-02T14:42:00.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.230+0000 7fad25400640 1 -- 172.21.15.90:0/3572038541 wait complete. 2024-04-02T14:42:00.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.232+0000 7fad25400640 1 Processor -- start 2024-04-02T14:42:00.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.232+0000 7fad25400640 1 -- start start 2024-04-02T14:42:00.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.232+0000 7fad25400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fad20100380 0x7fad201070a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:00.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.232+0000 7fad25400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fad20105450 0x7fad201075e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:00.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.232+0000 7fad25400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fad20107b20 0x7fad201aabc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:00.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.232+0000 7fad25400640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fad2010fcd0 con 0x7fad20100380 2024-04-02T14:42:00.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.232+0000 7fad25400640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fad2010fb50 con 0x7fad20105450 2024-04-02T14:42:00.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.232+0000 7fad25400640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fad2010fe50 con 0x7fad20107b20 2024-04-02T14:42:00.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.233+0000 7fad1e800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fad20100380 0x7fad201070a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:00.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.233+0000 7fad1e800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fad20100380 0x7fad201070a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38240/0 (socket says 172.21.15.90:38240) 2024-04-02T14:42:00.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.233+0000 7fad1e800640 1 -- 172.21.15.90:0/186631147 learned_addr learned my addr 172.21.15.90:0/186631147 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:42:00.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.233+0000 7fad1de00640 1 --2- 172.21.15.90:0/186631147 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fad20105450 0x7fad201075e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:00.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.233+0000 7fad1f200640 1 --2- 172.21.15.90:0/186631147 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fad20107b20 0x7fad201aabc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:00.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.233+0000 7fad1e800640 1 -- 172.21.15.90:0/186631147 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fad20107b20 msgr2=0x7fad201aabc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:00.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.233+0000 7fad1e800640 1 --2- 172.21.15.90:0/186631147 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fad20107b20 0x7fad201aabc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:00.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.233+0000 7fad1e800640 1 -- 172.21.15.90:0/186631147 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fad20105450 msgr2=0x7fad201075e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:00.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.233+0000 7fad1e800640 1 --2- 172.21.15.90:0/186631147 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fad20105450 0x7fad201075e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:00.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.233+0000 7fad1e800640 1 -- 172.21.15.90:0/186631147 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad201ab100 con 0x7fad20100380 2024-04-02T14:42:00.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.234+0000 7fad1e800640 1 --2- 172.21.15.90:0/186631147 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fad20100380 0x7fad201070a0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fad1000fdc0 tx=0x7fad1000d7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:42:00.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.234+0000 7fad17400640 1 -- 172.21.15.90:0/186631147 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad10004220 con 0x7fad20100380 2024-04-02T14:42:00.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.234+0000 7fad17400640 1 -- 172.21.15.90:0/186631147 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fad100043c0 con 0x7fad20100380 2024-04-02T14:42:00.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.234+0000 7fad17400640 1 -- 172.21.15.90:0/186631147 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad1000bad0 con 0x7fad20100380 2024-04-02T14:42:00.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.235+0000 7fad25400640 1 -- 172.21.15.90:0/186631147 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad201ab390 con 0x7fad20100380 2024-04-02T14:42:00.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.235+0000 7fad25400640 1 -- 172.21.15.90:0/186631147 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fad201ab860 con 0x7fad20100380 2024-04-02T14:42:00.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.241+0000 7fad17400640 1 -- 172.21.15.90:0/186631147 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fad1000bc70 con 0x7fad20100380 2024-04-02T14:42:00.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.243+0000 7fad28194740 1 Processor -- start 2024-04-02T14:42:00.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.243+0000 7fad28194740 1 -- start start 2024-04-02T14:42:00.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.244+0000 7fad17400640 1 --2- 172.21.15.90:0/186631147 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fad040a3da0 0x7fad040a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:00.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.244+0000 7fad28194740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55bef1a369f0 0x55bef1a35740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:00.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.244+0000 7fad28194740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bef1a35d10 0x55bef1a36190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:00.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.244+0000 7fad28194740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bef1a3f080 0x55bef1a41470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:00.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.244+0000 7fad28194740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55bef1976400 con 0x55bef1a3f080 2024-04-02T14:42:00.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.244+0000 7fad28194740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55bef195af10 con 0x55bef1a369f0 2024-04-02T14:42:00.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.244+0000 7fad28194740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55bef199a6f0 con 0x55bef1a35d10 2024-04-02T14:42:00.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.244+0000 7fad17400640 1 -- 172.21.15.90:0/186631147 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7fad100d0ae0 con 0x7fad20100380 2024-04-02T14:42:00.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.244+0000 7fad1e800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55bef1a369f0 0x55bef1a35740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:00.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.244+0000 7fad1e800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55bef1a369f0 0x55bef1a35740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:36774/0 (socket says 172.21.15.90:36774) 2024-04-02T14:42:00.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.244+0000 7fad1e800640 1 -- 172.21.15.90:0/41712909 learned_addr learned my addr 172.21.15.90:0/41712909 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:42:00.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.244+0000 7fad1e800640 1 -- 172.21.15.90:0/41712909 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bef1a35d10 msgr2=0x55bef1a36190 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-02T14:42:00.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.245+0000 7fad1e800640 1 --2- 172.21.15.90:0/41712909 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bef1a35d10 0x55bef1a36190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:00.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.245+0000 7fad1e800640 1 -- 172.21.15.90:0/41712909 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bef1a3f080 msgr2=0x55bef1a41470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:42:00.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.245+0000 7fad1e800640 1 --2- 172.21.15.90:0/41712909 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bef1a3f080 0x55bef1a41470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:00.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.245+0000 7fad1e800640 1 -- 172.21.15.90:0/41712909 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bef16ae3f0 con 0x55bef1a369f0 2024-04-02T14:42:00.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.245+0000 7fad1f200640 1 --2- 172.21.15.90:0/186631147 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fad040a3da0 0x7fad040a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:00.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.245+0000 7fad1e800640 1 --2- 172.21.15.90:0/41712909 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55bef1a369f0 0x55bef1a35740 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7fad100a51d0 tx=0x7fad1009c700 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c57b29421bba0e22 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:42:00.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.245+0000 7fad0b400640 1 -- 172.21.15.90:0/41712909 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad100a0e40 con 0x55bef1a369f0 2024-04-02T14:42:00.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.245+0000 7fad0b400640 1 -- 172.21.15.90:0/41712909 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fad100a5700 con 0x55bef1a369f0 2024-04-02T14:42:00.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.245+0000 7fad0b400640 1 -- 172.21.15.90:0/41712909 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad10099100 con 0x55bef1a369f0 2024-04-02T14:42:00.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.246+0000 7fad28194740 1 -- 172.21.15.90:0/41712909 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55bef1a369f0 msgr2=0x55bef1a35740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:00.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.246+0000 7fad28194740 1 --2- 172.21.15.90:0/41712909 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55bef1a369f0 0x55bef1a35740 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7fad100a51d0 tx=0x7fad1009c700 comp rx=0 tx=0).stop 2024-04-02T14:42:00.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.246+0000 7fad1f200640 1 --2- 172.21.15.90:0/186631147 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fad040a3da0 0x7fad040a6260 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fad18009990 tx=0x7fad18009920 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:42:00.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.246+0000 7fad28194740 1 -- 172.21.15.90:0/41712909 shutdown_connections 2024-04-02T14:42:00.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.246+0000 7fad28194740 1 --2- 172.21.15.90:0/41712909 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bef1a3f080 0x55bef1a41470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:00.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.246+0000 7fad28194740 1 --2- 172.21.15.90:0/41712909 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bef1a35d10 0x55bef1a36190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:00.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.246+0000 7fad28194740 1 --2- 172.21.15.90:0/41712909 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55bef1a369f0 0x55bef1a35740 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:00.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.246+0000 7fad28194740 1 -- 172.21.15.90:0/41712909 >> 172.21.15.90:0/41712909 conn(0x55bef19fca50 msgr2=0x55bef1a30760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:42:00.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.247+0000 7fad28194740 1 -- 172.21.15.90:0/41712909 shutdown_connections 2024-04-02T14:42:00.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.247+0000 7fad28194740 1 -- 172.21.15.90:0/41712909 wait complete. 2024-04-02T14:42:00.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.247+0000 7fad28194740 1 -- start start 2024-04-02T14:42:00.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.247+0000 7fad28194740 1 Processor -- start 2024-04-02T14:42:00.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.248+0000 7fad28194740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bef1a35d10 0x55bef1a3c110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:00.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.248+0000 7fad28194740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bef1a369f0 0x55bef1a3c650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:00.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.248+0000 7fad28194740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55bef1a3f080 0x55bef1a3cb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:00.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.248+0000 7fad28194740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55bef19966a0 con 0x55bef1a35d10 2024-04-02T14:42:00.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.248+0000 7fad28194740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55bef1942980 con 0x55bef1a3f080 2024-04-02T14:42:00.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.248+0000 7fad28194740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55bef1955cb0 con 0x55bef1a369f0 2024-04-02T14:42:00.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.248+0000 7fad28194740 10 client.0 _ll_register_callbacks cb 0x55bef19556a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:42:00.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.248+0000 7fad1de00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bef1a369f0 0x55bef1a3c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:00.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.248+0000 7fad1de00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bef1a369f0 0x55bef1a3c650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:56620/0 (socket says 172.21.15.90:56620) 2024-04-02T14:42:00.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.248+0000 7fad1de00640 1 -- 172.21.15.90:0/205625403 learned_addr learned my addr 172.21.15.90:0/205625403 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:42:00.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.249+0000 7fad1e800640 1 --2- 172.21.15.90:0/205625403 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bef1a35d10 0x55bef1a3c110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:00.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.249+0000 7fad1de00640 1 -- 172.21.15.90:0/205625403 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55bef1a3f080 msgr2=0x55bef1a3cb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:42:00.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.249+0000 7fad1de00640 1 --2- 172.21.15.90:0/205625403 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55bef1a3f080 0x55bef1a3cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:00.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.249+0000 7fad1de00640 1 -- 172.21.15.90:0/205625403 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bef1a35d10 msgr2=0x55bef1a3c110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:42:00.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.249+0000 7fad1de00640 1 --2- 172.21.15.90:0/205625403 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bef1a35d10 0x55bef1a3c110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:00.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.249+0000 7fad1de00640 1 -- 172.21.15.90:0/205625403 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bef17787d0 con 0x55bef1a369f0 2024-04-02T14:42:00.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.249+0000 7fad1de00640 1 --2- 172.21.15.90:0/205625403 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bef1a369f0 0x55bef1a3c650 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fad20074220 tx=0x7fad0c009450 comp rx=0 tx=0).ready entity=mon.1 client_cookie=580efc711f86bd2 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:42:00.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.249+0000 7facffe00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:42:00.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.249+0000 7facffe00640 1 -- 172.21.15.90:0/205625403 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad0c037040 con 0x55bef1a369f0 2024-04-02T14:42:00.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.250+0000 7facffe00640 1 -- 172.21.15.90:0/205625403 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7fad0c009920 con 0x55bef1a369f0 2024-04-02T14:42:00.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.250+0000 7fad28194740 1 -- 172.21.15.90:0/205625403 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fad100a5700 con 0x55bef1a369f0 2024-04-02T14:42:00.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.250+0000 7facffe00640 1 -- 172.21.15.90:0/205625403 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fad0c009b20 con 0x55bef1a369f0 2024-04-02T14:42:00.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.250+0000 7fad28194740 10 client.27457 fetch_fsmap learned FSMap version 27 2024-04-02T14:42:00.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.250+0000 7fad28194740 1 -- 172.21.15.90:0/205625403 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55bef1a4dfd0 con 0x55bef1a369f0 2024-04-02T14:42:00.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.250+0000 7facffe00640 1 -- 172.21.15.90:0/205625403 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fad100a5700 con 0x55bef1a369f0 2024-04-02T14:42:00.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.251+0000 7facffe00640 1 -- 172.21.15.90:0/205625403 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fad0c044070 con 0x55bef1a369f0 2024-04-02T14:42:00.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.251+0000 7fad28194740 10 client.27457 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:42:00.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.251+0000 7fad28194740 10 client.27457 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:42:00.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.251+0000 7fad28194740 20 client.27457 populate_metadata read hostname 'smithi090' 2024-04-02T14:42:00.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.251+0000 7fad28194740 1 --2- 172.21.15.90:0/205625403 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55bef1a4f530 0x55bef1a519f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:00.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.251+0000 7fad28194740 4 client.27457 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:42:00.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.251+0000 7fad28194740 1 -- 172.21.15.90:0/205625403 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bef195a040 con 0x55bef1a4f530 2024-04-02T14:42:00.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.252+0000 7fad1e800640 1 --2- 172.21.15.90:0/205625403 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55bef1a4f530 0x55bef1a519f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:00.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.252+0000 7fad1e800640 1 --2- 172.21.15.90:0/205625403 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55bef1a4f530 0x55bef1a519f0 crc :-1 s=READY pgs=194 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9c1a70e1a76ab2ff server_cookie=e815ba13348342b0 in_seq=0 out_seq=0 2024-04-02T14:42:00.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.252+0000 7facffe00640 10 client.27457 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:42:00.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.253+0000 7facffe00640 1 -- 172.21.15.90:0/205625403 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55bef195a040 con 0x55bef1a4f530 2024-04-02T14:42:00.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.253+0000 7facffe00640 10 client.27457 handle_command_reply: tid=0 2024-04-02T14:42:00.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.254+0000 7fad28194740 1 client.27457 shutdown 2024-04-02T14:42:00.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.254+0000 7fad28194740 1 -- 172.21.15.90:0/205625403 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bef1a369f0 msgr2=0x55bef1a3c650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:00.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.254+0000 7fad28194740 1 --2- 172.21.15.90:0/205625403 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bef1a369f0 0x55bef1a3c650 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fad20074220 tx=0x7fad0c009450 comp rx=0 tx=0).stop 2024-04-02T14:42:00.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.255+0000 7fad28194740 1 -- 172.21.15.90:0/205625403 shutdown_connections 2024-04-02T14:42:00.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.255+0000 7fad28194740 1 --2- 172.21.15.90:0/205625403 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55bef1a4f530 0x55bef1a519f0 crc :-1 s=READY pgs=194 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:00.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.255+0000 7fad28194740 1 --2- 172.21.15.90:0/205625403 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55bef1a3f080 0x55bef1a3cb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:00.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.255+0000 7fad28194740 1 --2- 172.21.15.90:0/205625403 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bef1a369f0 0x55bef1a3c650 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:00.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.255+0000 7facffe00640 0 client.27457 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:42:00.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.255+0000 7fad28194740 1 --2- 172.21.15.90:0/205625403 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bef1a35d10 0x55bef1a3c110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:00.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.255+0000 7fad28194740 1 -- 172.21.15.90:0/205625403 >> 172.21.15.90:0/205625403 conn(0x55bef19fca50 msgr2=0x55bef1a324b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:42:00.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.255+0000 7fad28194740 1 -- 172.21.15.90:0/205625403 shutdown_connections 2024-04-02T14:42:00.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.255+0000 7fad28194740 1 -- 172.21.15.90:0/205625403 wait complete. 2024-04-02T14:42:00.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.255+0000 7fad28194740 20 client.27457 trim_cache size 0 max 16384 2024-04-02T14:42:00.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.279+0000 7fad28194740 1 Processor -- start 2024-04-02T14:42:00.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.279+0000 7fad28194740 1 -- start start 2024-04-02T14:42:00.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.279+0000 7fad28194740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bef1a35d10 0x55bef1a4aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:00.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.279+0000 7fad28194740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55bef1a369f0 0x55bef1a3b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:00.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.279+0000 7fad28194740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bef1a3f080 0x55bef1a3df40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:00.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.279+0000 7fad28194740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fad201b3e90 con 0x55bef1a35d10 2024-04-02T14:42:00.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.279+0000 7fad28194740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fad201b59d0 con 0x55bef1a369f0 2024-04-02T14:42:00.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.279+0000 7fad28194740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fad201b4560 con 0x55bef1a3f080 2024-04-02T14:42:00.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.280+0000 7fad1de00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55bef1a369f0 0x55bef1a3b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:00.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.280+0000 7fad1e800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bef1a35d10 0x55bef1a4aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:00.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.280+0000 7fad1de00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55bef1a369f0 0x55bef1a3b8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:36784/0 (socket says 172.21.15.90:36784) 2024-04-02T14:42:00.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.280+0000 7fad1de00640 1 -- 172.21.15.90:0/182215398 learned_addr learned my addr 172.21.15.90:0/182215398 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:42:00.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.280+0000 7fad1f200640 1 --2- 172.21.15.90:0/182215398 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bef1a3f080 0x55bef1a3df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:00.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.280+0000 7fad1de00640 1 -- 172.21.15.90:0/182215398 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bef1a3f080 msgr2=0x55bef1a3df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:42:00.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.280+0000 7fad1de00640 1 --2- 172.21.15.90:0/182215398 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bef1a3f080 0x55bef1a3df40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:00.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.280+0000 7fad1de00640 1 -- 172.21.15.90:0/182215398 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bef1a35d10 msgr2=0x55bef1a4aaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:42:00.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.280+0000 7fad1de00640 1 --2- 172.21.15.90:0/182215398 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bef1a35d10 0x55bef1a4aaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:00.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.280+0000 7fad1de00640 1 -- 172.21.15.90:0/182215398 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bef1a34d40 con 0x55bef1a369f0 2024-04-02T14:42:00.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.280+0000 7fad1e800640 1 --2- 172.21.15.90:0/182215398 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bef1a35d10 0x55bef1a4aaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:42:00.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.281+0000 7fad1de00640 1 --2- 172.21.15.90:0/182215398 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55bef1a369f0 0x55bef1a3b8c0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7fad0c001eb0 tx=0x7fad0c02d8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8accb6be60b2a87c server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:42:00.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.281+0000 7fad0aa00640 1 -- 172.21.15.90:0/182215398 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad0c052070 con 0x55bef1a369f0 2024-04-02T14:42:00.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.281+0000 7fad0aa00640 1 -- 172.21.15.90:0/182215398 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fad0c009cc0 con 0x55bef1a369f0 2024-04-02T14:42:00.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.281+0000 7fad0aa00640 1 -- 172.21.15.90:0/182215398 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad0c037030 con 0x55bef1a369f0 2024-04-02T14:42:00.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.282+0000 7fad28194740 1 -- 172.21.15.90:0/182215398 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55bef1a369f0 msgr2=0x55bef1a3b8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:00.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.282+0000 7fad28194740 1 --2- 172.21.15.90:0/182215398 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55bef1a369f0 0x55bef1a3b8c0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7fad0c001eb0 tx=0x7fad0c02d8f0 comp rx=0 tx=0).stop 2024-04-02T14:42:00.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.283+0000 7fad28194740 1 -- 172.21.15.90:0/182215398 shutdown_connections 2024-04-02T14:42:00.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.283+0000 7fad28194740 1 --2- 172.21.15.90:0/182215398 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bef1a3f080 0x55bef1a3df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:00.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.283+0000 7fad28194740 1 --2- 172.21.15.90:0/182215398 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55bef1a369f0 0x55bef1a3b8c0 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:00.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.283+0000 7fad28194740 1 --2- 172.21.15.90:0/182215398 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bef1a35d10 0x55bef1a4aaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:00.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.283+0000 7fad28194740 1 -- 172.21.15.90:0/182215398 >> 172.21.15.90:0/182215398 conn(0x55bef19fca50 msgr2=0x55bef1a3d830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:42:00.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.283+0000 7fad28194740 1 -- 172.21.15.90:0/182215398 shutdown_connections 2024-04-02T14:42:00.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.283+0000 7fad28194740 1 -- 172.21.15.90:0/182215398 wait complete. 2024-04-02T14:42:00.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.284+0000 7fad28194740 1 -- start start 2024-04-02T14:42:00.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.284+0000 7fad28194740 1 Processor -- start 2024-04-02T14:42:00.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.284+0000 7fad28194740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bef1a35d10 0x55bef1a47530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:00.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.284+0000 7fad28194740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55bef1a369f0 0x55bef1a3b940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:00.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.284+0000 7fad28194740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bef1a3f080 0x55bef1a3c080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:00.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.284+0000 7fad28194740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fad201b5300 con 0x55bef1a35d10 2024-04-02T14:42:00.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.284+0000 7fad28194740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fad201b4c30 con 0x55bef1a369f0 2024-04-02T14:42:00.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.284+0000 7fad28194740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fad201b1e70 con 0x55bef1a3f080 2024-04-02T14:42:00.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.285+0000 7fad28194740 10 client.0 _ll_register_callbacks cb 0x7fad0c004520 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:42:00.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.285+0000 7fad1de00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55bef1a369f0 0x55bef1a3b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:00.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.285+0000 7fad1e800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bef1a35d10 0x55bef1a47530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:00.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.285+0000 7fad1e800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bef1a35d10 0x55bef1a47530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38294/0 (socket says 172.21.15.90:38294) 2024-04-02T14:42:00.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.285+0000 7fad1e800640 1 -- 172.21.15.90:0/4206132322 learned_addr learned my addr 172.21.15.90:0/4206132322 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:42:00.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.285+0000 7fad1f200640 1 --2- 172.21.15.90:0/4206132322 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bef1a3f080 0x55bef1a3c080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:00.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.286+0000 7fad1e800640 1 -- 172.21.15.90:0/4206132322 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bef1a3f080 msgr2=0x55bef1a3c080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:42:00.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.286+0000 7fad1e800640 1 --2- 172.21.15.90:0/4206132322 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bef1a3f080 0x55bef1a3c080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:00.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.286+0000 7fad1e800640 1 -- 172.21.15.90:0/4206132322 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55bef1a369f0 msgr2=0x55bef1a3b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:42:00.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.286+0000 7fad1e800640 1 --2- 172.21.15.90:0/4206132322 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55bef1a369f0 0x55bef1a3b940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:00.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.286+0000 7fad1e800640 1 -- 172.21.15.90:0/4206132322 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bef1a44ff0 con 0x55bef1a35d10 2024-04-02T14:42:00.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.287+0000 7fad1de00640 1 --2- 172.21.15.90:0/4206132322 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55bef1a369f0 0x55bef1a3b940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:42:00.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.287+0000 7fad1f200640 1 --2- 172.21.15.90:0/4206132322 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bef1a3f080 0x55bef1a3c080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:42:00.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.287+0000 7fad1e800640 1 --2- 172.21.15.90:0/4206132322 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bef1a35d10 0x55bef1a47530 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fad100ce2f0 tx=0x7fad10004570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c67cc7cf3a540aec server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:42:00.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.288+0000 7facffe00640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-02T14:42:00.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.288+0000 7facffe00640 1 -- 172.21.15.90:0/4206132322 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad10098d30 con 0x55bef1a35d10 2024-04-02T14:42:00.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.288+0000 7facffe00640 1 -- 172.21.15.90:0/4206132322 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7fad100ce4c0 con 0x55bef1a35d10 2024-04-02T14:42:00.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.288+0000 7fad28194740 1 -- 172.21.15.90:0/4206132322 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fad0c009cc0 con 0x55bef1a35d10 2024-04-02T14:42:00.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.289+0000 7facffe00640 1 -- 172.21.15.90:0/4206132322 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fad100a5700 con 0x55bef1a35d10 2024-04-02T14:42:00.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.289+0000 7fad28194740 10 client.16866 fetch_fsmap learned FSMap version 27 2024-04-02T14:42:00.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.289+0000 7fad28194740 1 -- 172.21.15.90:0/4206132322 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55bef1ab5bf0 con 0x55bef1a35d10 2024-04-02T14:42:00.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.289+0000 7facffe00640 1 -- 172.21.15.90:0/4206132322 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fad0c009cc0 con 0x55bef1a35d10 2024-04-02T14:42:00.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.289+0000 7facffe00640 1 -- 172.21.15.90:0/4206132322 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fad1000b040 con 0x55bef1a35d10 2024-04-02T14:42:00.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.289+0000 7fad28194740 10 client.16866 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:42:00.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.289+0000 7fad28194740 10 client.16866 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:42:00.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.289+0000 7fad28194740 20 client.16866 populate_metadata read hostname 'smithi090' 2024-04-02T14:42:00.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.289+0000 7fad28194740 1 --2- 172.21.15.90:0/4206132322 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55bef1a4f530 0x55bef1ab7ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:00.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.289+0000 7fad28194740 4 client.16866 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-02T14:42:00.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.289+0000 7fad28194740 1 -- 172.21.15.90:0/4206132322 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55bef19a7c30 con 0x55bef1a4f530 2024-04-02T14:42:00.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.290+0000 7fad1de00640 1 --2- 172.21.15.90:0/4206132322 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55bef1a4f530 0x55bef1ab7ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:00.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.291+0000 7facffe00640 10 client.16866 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:42:00.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.291+0000 7fad1de00640 1 --2- 172.21.15.90:0/4206132322 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55bef1a4f530 0x55bef1ab7ee0 crc :-1 s=READY pgs=195 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3fdff79e12747423 server_cookie=ba5408c95ccc7310 in_seq=0 out_seq=0 2024-04-02T14:42:00.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.291+0000 7facffe00640 1 -- 172.21.15.90:0/4206132322 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55bef19a7c30 con 0x55bef1a4f530 2024-04-02T14:42:00.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.291+0000 7facffe00640 10 client.16866 handle_command_reply: tid=0 2024-04-02T14:42:00.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.291+0000 7fad28194740 1 client.16866 shutdown 2024-04-02T14:42:00.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.291+0000 7fad28194740 1 -- 172.21.15.90:0/4206132322 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bef1a35d10 msgr2=0x55bef1a47530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:00.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.291+0000 7fad28194740 1 --2- 172.21.15.90:0/4206132322 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bef1a35d10 0x55bef1a47530 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fad100ce2f0 tx=0x7fad10004570 comp rx=0 tx=0).stop 2024-04-02T14:42:00.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.292+0000 7fad28194740 1 -- 172.21.15.90:0/4206132322 shutdown_connections 2024-04-02T14:42:00.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.292+0000 7fad28194740 1 --2- 172.21.15.90:0/4206132322 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55bef1a4f530 0x55bef1ab7ee0 crc :-1 s=READY pgs=195 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:00.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.292+0000 7fad28194740 1 --2- 172.21.15.90:0/4206132322 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bef1a3f080 0x55bef1a3c080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:00.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.292+0000 7fad28194740 1 --2- 172.21.15.90:0/4206132322 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55bef1a369f0 0x55bef1a3b940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:00.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.292+0000 7fad28194740 1 --2- 172.21.15.90:0/4206132322 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bef1a35d10 0x55bef1a47530 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:00.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.292+0000 7fad28194740 1 -- 172.21.15.90:0/4206132322 >> 172.21.15.90:0/4206132322 conn(0x55bef19fca50 msgr2=0x55bef1a32320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:42:00.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.292+0000 7facffe00640 0 client.16866 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:42:00.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.292+0000 7fad28194740 1 -- 172.21.15.90:0/4206132322 shutdown_connections 2024-04-02T14:42:00.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.292+0000 7fad28194740 1 -- 172.21.15.90:0/4206132322 wait complete. 2024-04-02T14:42:00.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.292+0000 7fad28194740 20 client.16866 trim_cache size 0 max 16384 2024-04-02T14:42:00.293 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-02T14:42:00.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.293+0000 7fad25400640 1 -- 172.21.15.90:0/186631147 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fad040a3da0 msgr2=0x7fad040a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:00.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.293+0000 7fad25400640 1 --2- 172.21.15.90:0/186631147 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fad040a3da0 0x7fad040a6260 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fad18009990 tx=0x7fad18009920 comp rx=0 tx=0).stop 2024-04-02T14:42:00.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.293+0000 7fad25400640 1 -- 172.21.15.90:0/186631147 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fad20100380 msgr2=0x7fad201070a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:00.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.293+0000 7fad25400640 1 --2- 172.21.15.90:0/186631147 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fad20100380 0x7fad201070a0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fad1000fdc0 tx=0x7fad1000d7b0 comp rx=0 tx=0).stop 2024-04-02T14:42:00.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.293+0000 7fad25400640 1 -- 172.21.15.90:0/186631147 shutdown_connections 2024-04-02T14:42:00.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.293+0000 7fad25400640 1 --2- 172.21.15.90:0/186631147 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fad040a3da0 0x7fad040a6260 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:00.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.293+0000 7fad25400640 1 --2- 172.21.15.90:0/186631147 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fad20107b20 0x7fad201aabc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:00.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.293+0000 7fad25400640 1 --2- 172.21.15.90:0/186631147 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fad20105450 0x7fad201075e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:00.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.293+0000 7fad25400640 1 --2- 172.21.15.90:0/186631147 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fad20100380 0x7fad201070a0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:00.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.293+0000 7fad25400640 1 -- 172.21.15.90:0/186631147 >> 172.21.15.90:0/186631147 conn(0x7fad20076f20 msgr2=0x7fad201033b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:42:00.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.293+0000 7fad25400640 1 -- 172.21.15.90:0/186631147 shutdown_connections 2024-04-02T14:42:00.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:00.293+0000 7fad25400640 1 -- 172.21.15.90:0/186631147 wait complete. 2024-04-02T14:42:01.317 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-02T14:42:01.317 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T14:42:01.318 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-02T14:42:01.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.537+0000 7f10d1800640 1 -- 172.21.15.90:0/3687710356 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f10cc10bff0 msgr2=0x7f10cc10e490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:01.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.537+0000 7f10d1800640 1 --2- 172.21.15.90:0/3687710356 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f10cc10bff0 0x7f10cc10e490 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f10c4009ca0 tx=0x7f10c402c190 comp rx=0 tx=0).stop 2024-04-02T14:42:01.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.537+0000 7f10d1800640 1 -- 172.21.15.90:0/3687710356 shutdown_connections 2024-04-02T14:42:01.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.537+0000 7f10d1800640 1 --2- 172.21.15.90:0/3687710356 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f10cc10bff0 0x7f10cc10e490 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:01.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.537+0000 7f10d1800640 1 --2- 172.21.15.90:0/3687710356 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f10cc1034f0 0x7f10cc10b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:01.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.537+0000 7f10d1800640 1 --2- 172.21.15.90:0/3687710356 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f10cc102bd0 0x7f10cc102fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:01.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.537+0000 7f10d1800640 1 -- 172.21.15.90:0/3687710356 >> 172.21.15.90:0/3687710356 conn(0x7f10cc0fc410 msgr2=0x7f10cc0fe830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:42:01.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.537+0000 7f10d1800640 1 -- 172.21.15.90:0/3687710356 shutdown_connections 2024-04-02T14:42:01.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.537+0000 7f10d1800640 1 -- 172.21.15.90:0/3687710356 wait complete. 2024-04-02T14:42:01.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.538+0000 7f10d1800640 1 Processor -- start 2024-04-02T14:42:01.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.539+0000 7f10d1800640 1 -- start start 2024-04-02T14:42:01.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.539+0000 7f10d1800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f10cc102bd0 0x7f10cc19a450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:01.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.539+0000 7f10d1800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f10cc1034f0 0x7f10cc19a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:01.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.539+0000 7f10d1800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f10cc10bff0 0x7f10cc19ed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:01.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.539+0000 7f10d1800640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f10cc111df0 con 0x7f10cc1034f0 2024-04-02T14:42:01.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.539+0000 7f10d1800640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f10cc111c70 con 0x7f10cc102bd0 2024-04-02T14:42:01.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.539+0000 7f10d1800640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f10cc111f70 con 0x7f10cc10bff0 2024-04-02T14:42:01.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.539+0000 7f10cb400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f10cc1034f0 0x7f10cc19a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:01.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.539+0000 7f10d0e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f10cc10bff0 0x7f10cc19ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:01.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.539+0000 7f10cb400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f10cc1034f0 0x7f10cc19a990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38304/0 (socket says 172.21.15.90:38304) 2024-04-02T14:42:01.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.539+0000 7f10cb400640 1 -- 172.21.15.90:0/2949055224 learned_addr learned my addr 172.21.15.90:0/2949055224 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:42:01.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.539+0000 7f10d0e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f10cc10bff0 0x7f10cc19ed10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:56670/0 (socket says 172.21.15.90:56670) 2024-04-02T14:42:01.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.539+0000 7f10cbe00640 1 --2- 172.21.15.90:0/2949055224 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f10cc102bd0 0x7f10cc19a450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:01.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.540+0000 7f10cbe00640 1 -- 172.21.15.90:0/2949055224 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f10cc10bff0 msgr2=0x7f10cc19ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:01.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.540+0000 7f10cbe00640 1 --2- 172.21.15.90:0/2949055224 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f10cc10bff0 0x7f10cc19ed10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:01.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.540+0000 7f10cbe00640 1 -- 172.21.15.90:0/2949055224 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f10cc1034f0 msgr2=0x7f10cc19a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:01.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.540+0000 7f10cbe00640 1 --2- 172.21.15.90:0/2949055224 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f10cc1034f0 0x7f10cc19a990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:01.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.540+0000 7f10cbe00640 1 -- 172.21.15.90:0/2949055224 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10cc19f440 con 0x7f10cc102bd0 2024-04-02T14:42:01.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.540+0000 7f10d0e00640 1 --2- 172.21.15.90:0/2949055224 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f10cc10bff0 0x7f10cc19ed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:42:01.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.540+0000 7f10cb400640 1 --2- 172.21.15.90:0/2949055224 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f10cc1034f0 0x7f10cc19a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:42:01.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.540+0000 7f10cbe00640 1 --2- 172.21.15.90:0/2949055224 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f10cc102bd0 0x7f10cc19a450 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f10bc00e9f0 tx=0x7f10bc00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:42:01.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.540+0000 7f10c8c00640 1 -- 172.21.15.90:0/2949055224 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10bc00cd80 con 0x7f10cc102bd0 2024-04-02T14:42:01.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.540+0000 7f10d1800640 1 -- 172.21.15.90:0/2949055224 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10cc19f730 con 0x7f10cc102bd0 2024-04-02T14:42:01.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.541+0000 7f10d1800640 1 -- 172.21.15.90:0/2949055224 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10cc1a6fc0 con 0x7f10cc102bd0 2024-04-02T14:42:01.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.541+0000 7f10c8c00640 1 -- 172.21.15.90:0/2949055224 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f10bc0035f0 con 0x7f10cc102bd0 2024-04-02T14:42:01.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.541+0000 7f10c8c00640 1 -- 172.21.15.90:0/2949055224 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10bc010670 con 0x7f10cc102bd0 2024-04-02T14:42:01.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.544+0000 7f10d5be3740 1 Processor -- start 2024-04-02T14:42:01.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.544+0000 7f10d5be3740 1 -- start start 2024-04-02T14:42:01.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.544+0000 7f10c8c00640 1 -- 172.21.15.90:0/2949055224 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f10bc010810 con 0x7f10cc102bd0 2024-04-02T14:42:01.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.544+0000 7f10d5be3740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5603d3f7d1f0 0x5603d3f7bf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:01.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.544+0000 7f10d5be3740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5603d3f7c510 0x5603d3f7c990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:01.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.545+0000 7f10d5be3740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5603d3f85880 0x5603d3f87c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:01.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.545+0000 7f10d5be3740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5603d3ee0310 con 0x5603d3f7d1f0 2024-04-02T14:42:01.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.545+0000 7f10d5be3740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x5603d3e88690 con 0x5603d3f7c510 2024-04-02T14:42:01.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.545+0000 7f10d5be3740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5603d3edc2c0 con 0x5603d3f85880 2024-04-02T14:42:01.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.545+0000 7f10cbe00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5603d3f7c510 0x5603d3f7c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:01.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.545+0000 7f10d0e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5603d3f7d1f0 0x5603d3f7bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:01.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.545+0000 7f10cbe00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5603d3f7c510 0x5603d3f7c990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:36828/0 (socket says 172.21.15.90:36828) 2024-04-02T14:42:01.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.545+0000 7f10cbe00640 1 -- 172.21.15.90:0/153384139 learned_addr learned my addr 172.21.15.90:0/153384139 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:42:01.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.545+0000 7f10cb400640 1 --2- 172.21.15.90:0/153384139 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5603d3f85880 0x5603d3f87c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:01.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.545+0000 7f10cbe00640 1 -- 172.21.15.90:0/153384139 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5603d3f85880 msgr2=0x5603d3f87c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:42:01.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.545+0000 7f10cbe00640 1 --2- 172.21.15.90:0/153384139 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5603d3f85880 0x5603d3f87c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:01.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.545+0000 7f10cbe00640 1 -- 172.21.15.90:0/153384139 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5603d3f7d1f0 msgr2=0x5603d3f7bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:42:01.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.545+0000 7f10cbe00640 1 --2- 172.21.15.90:0/153384139 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5603d3f7d1f0 0x5603d3f7bf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:01.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.545+0000 7f10cbe00640 1 -- 172.21.15.90:0/153384139 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5603d3cbc7e0 con 0x5603d3f7c510 2024-04-02T14:42:01.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.546+0000 7f10cbe00640 1 --2- 172.21.15.90:0/153384139 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5603d3f7c510 0x5603d3f7c990 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f10bc0ce030 tx=0x7f10bc0d6f10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4cf2c069f6a3322b server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:42:01.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.546+0000 7f10c1600640 1 -- 172.21.15.90:0/153384139 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10bc0df070 con 0x5603d3f7c510 2024-04-02T14:42:01.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.546+0000 7f10c1600640 1 -- 172.21.15.90:0/153384139 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f10bc0d8de0 con 0x5603d3f7c510 2024-04-02T14:42:01.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.546+0000 7f10c1600640 1 -- 172.21.15.90:0/153384139 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10bc0da8e0 con 0x5603d3f7c510 2024-04-02T14:42:01.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.546+0000 7f10c8c00640 1 --2- 172.21.15.90:0/2949055224 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f10b00a3da0 0x7f10b00a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:01.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.546+0000 7f10c8c00640 1 -- 172.21.15.90:0/2949055224 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f10bc0d0960 con 0x7f10cc102bd0 2024-04-02T14:42:01.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.546+0000 7f10d0e00640 1 --2- 172.21.15.90:0/2949055224 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f10b00a3da0 0x7f10b00a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:01.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.547+0000 7f10d5be3740 1 -- 172.21.15.90:0/153384139 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5603d3f7c510 msgr2=0x5603d3f7c990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:01.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.547+0000 7f10d5be3740 1 --2- 172.21.15.90:0/153384139 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5603d3f7c510 0x5603d3f7c990 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f10bc0ce030 tx=0x7f10bc0d6f10 comp rx=0 tx=0).stop 2024-04-02T14:42:01.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.547+0000 7f10d5be3740 1 -- 172.21.15.90:0/153384139 shutdown_connections 2024-04-02T14:42:01.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.547+0000 7f10d5be3740 1 --2- 172.21.15.90:0/153384139 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5603d3f85880 0x5603d3f87c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:01.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.547+0000 7f10d5be3740 1 --2- 172.21.15.90:0/153384139 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5603d3f7c510 0x5603d3f7c990 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:01.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.547+0000 7f10d5be3740 1 --2- 172.21.15.90:0/153384139 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5603d3f7d1f0 0x5603d3f7bf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:01.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.547+0000 7f10d5be3740 1 -- 172.21.15.90:0/153384139 >> 172.21.15.90:0/153384139 conn(0x5603d3f10270 msgr2=0x5603d3f76f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:42:01.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.547+0000 7f10d5be3740 1 -- 172.21.15.90:0/153384139 shutdown_connections 2024-04-02T14:42:01.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.547+0000 7f10d0e00640 1 --2- 172.21.15.90:0/2949055224 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f10b00a3da0 0x7f10b00a6260 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f10c4004af0 tx=0x7f10c4005c50 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:42:01.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.547+0000 7f10d5be3740 1 -- 172.21.15.90:0/153384139 wait complete. 2024-04-02T14:42:01.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.548+0000 7f10d5be3740 1 -- start start 2024-04-02T14:42:01.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.548+0000 7f10d5be3740 1 Processor -- start 2024-04-02T14:42:01.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.548+0000 7f10d5be3740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5603d3f7c510 0x5603d3f827c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:01.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.548+0000 7f10d5be3740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5603d3f7d1f0 0x5603d3f82d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:01.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.548+0000 7f10d5be3740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5603d3f85880 0x5603d3f83240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:01.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.548+0000 7f10d5be3740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5603d3f886f0 con 0x5603d3f85880 2024-04-02T14:42:01.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.548+0000 7f10d5be3740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x5603d3e9ba60 con 0x5603d3f7c510 2024-04-02T14:42:01.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.548+0000 7f10d5be3740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5603d3f88870 con 0x5603d3f7d1f0 2024-04-02T14:42:01.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.548+0000 7f10d5be3740 10 client.0 _ll_register_callbacks cb 0x5603d3ea1030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:42:01.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.548+0000 7f10cbe00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5603d3f7c510 0x5603d3f827c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:01.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.548+0000 7f10cbe00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5603d3f7c510 0x5603d3f827c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:36842/0 (socket says 172.21.15.90:36842) 2024-04-02T14:42:01.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.548+0000 7f10cbe00640 1 -- 172.21.15.90:0/4149667890 learned_addr learned my addr 172.21.15.90:0/4149667890 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:42:01.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.548+0000 7f10d0e00640 1 --2- 172.21.15.90:0/4149667890 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5603d3f85880 0x5603d3f83240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:01.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.549+0000 7f10cb400640 1 --2- 172.21.15.90:0/4149667890 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5603d3f7d1f0 0x5603d3f82d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:01.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.549+0000 7f10cbe00640 1 -- 172.21.15.90:0/4149667890 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5603d3f7d1f0 msgr2=0x5603d3f82d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:42:01.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.549+0000 7f10cbe00640 1 --2- 172.21.15.90:0/4149667890 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5603d3f7d1f0 0x5603d3f82d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:01.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.549+0000 7f10cbe00640 1 -- 172.21.15.90:0/4149667890 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5603d3f85880 msgr2=0x5603d3f83240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:42:01.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.549+0000 7f10cbe00640 1 --2- 172.21.15.90:0/4149667890 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5603d3f85880 0x5603d3f83240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:01.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.549+0000 7f10cbe00640 1 -- 172.21.15.90:0/4149667890 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5603d3d1cf70 con 0x5603d3f7c510 2024-04-02T14:42:01.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.549+0000 7f10cbe00640 1 --2- 172.21.15.90:0/4149667890 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5603d3f7c510 0x5603d3f827c0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f10bc0d61b0 tx=0x7f10bc0d6790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=38e3855cdf03e189 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:42:01.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.549+0000 7f10b6000640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:42:01.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.549+0000 7f10b6000640 1 -- 172.21.15.90:0/4149667890 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10bc0df070 con 0x5603d3f7c510 2024-04-02T14:42:01.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.549+0000 7f10b6000640 1 -- 172.21.15.90:0/4149667890 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f10bc0ce340 con 0x5603d3f7c510 2024-04-02T14:42:01.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.549+0000 7f10d5be3740 1 -- 172.21.15.90:0/4149667890 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f10bc0d8de0 con 0x5603d3f7c510 2024-04-02T14:42:01.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.549+0000 7f10b6000640 1 -- 172.21.15.90:0/4149667890 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f10bc098bb0 con 0x5603d3f7c510 2024-04-02T14:42:01.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.549+0000 7f10d5be3740 10 client.27608 fetch_fsmap learned FSMap version 27 2024-04-02T14:42:01.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.549+0000 7f10d5be3740 1 -- 172.21.15.90:0/4149667890 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5603d3bb4170 con 0x5603d3f7c510 2024-04-02T14:42:01.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.550+0000 7f10b6000640 1 -- 172.21.15.90:0/4149667890 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f10bc0d8de0 con 0x5603d3f7c510 2024-04-02T14:42:01.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.550+0000 7f10b6000640 1 -- 172.21.15.90:0/4149667890 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f10bc098f80 con 0x5603d3f7c510 2024-04-02T14:42:01.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.550+0000 7f10d5be3740 10 client.27608 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:42:01.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.550+0000 7f10d5be3740 10 client.27608 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:42:01.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.550+0000 7f10d5be3740 20 client.27608 populate_metadata read hostname 'smithi090' 2024-04-02T14:42:01.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.550+0000 7f10d5be3740 1 --2- 172.21.15.90:0/4149667890 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5603d3f95b80 0x5603d3f97f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:01.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.550+0000 7f10d5be3740 4 client.27608 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:42:01.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.550+0000 7f10d5be3740 1 -- 172.21.15.90:0/4149667890 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5603d3eed850 con 0x5603d3f95b80 2024-04-02T14:42:01.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.550+0000 7f10cb400640 1 --2- 172.21.15.90:0/4149667890 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5603d3f95b80 0x5603d3f97f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:01.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.551+0000 7f10cb400640 1 --2- 172.21.15.90:0/4149667890 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5603d3f95b80 0x5603d3f97f70 crc :-1 s=READY pgs=196 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=943b45e2ade77e90 server_cookie=9991e2d10b3eec7b in_seq=0 out_seq=0 2024-04-02T14:42:01.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.551+0000 7f10b6000640 10 client.27608 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:42:01.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.551+0000 7f10b6000640 1 -- 172.21.15.90:0/4149667890 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5603d3eed850 con 0x5603d3f95b80 2024-04-02T14:42:01.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.551+0000 7f10b6000640 10 client.27608 handle_command_reply: tid=0 2024-04-02T14:42:01.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.552+0000 7f10d5be3740 1 client.27608 shutdown 2024-04-02T14:42:01.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.552+0000 7f10d5be3740 1 -- 172.21.15.90:0/4149667890 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5603d3f7c510 msgr2=0x5603d3f827c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:01.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.552+0000 7f10d5be3740 1 --2- 172.21.15.90:0/4149667890 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5603d3f7c510 0x5603d3f827c0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f10bc0d61b0 tx=0x7f10bc0d6790 comp rx=0 tx=0).stop 2024-04-02T14:42:01.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.552+0000 7f10d5be3740 1 -- 172.21.15.90:0/4149667890 shutdown_connections 2024-04-02T14:42:01.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.552+0000 7f10d5be3740 1 --2- 172.21.15.90:0/4149667890 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5603d3f95b80 0x5603d3f97f70 crc :-1 s=READY pgs=196 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:01.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.552+0000 7f10d5be3740 1 --2- 172.21.15.90:0/4149667890 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5603d3f85880 0x5603d3f83240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:01.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.552+0000 7f10d5be3740 1 --2- 172.21.15.90:0/4149667890 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5603d3f7d1f0 0x5603d3f82d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:01.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.552+0000 7f10d5be3740 1 --2- 172.21.15.90:0/4149667890 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5603d3f7c510 0x5603d3f827c0 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:01.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.552+0000 7f10b6000640 0 client.27608 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:42:01.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.552+0000 7f10d5be3740 1 -- 172.21.15.90:0/4149667890 >> 172.21.15.90:0/4149667890 conn(0x5603d3f10270 msgr2=0x5603d3f78cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:42:01.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.552+0000 7f10d5be3740 1 -- 172.21.15.90:0/4149667890 shutdown_connections 2024-04-02T14:42:01.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.552+0000 7f10d5be3740 1 -- 172.21.15.90:0/4149667890 wait complete. 2024-04-02T14:42:01.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.552+0000 7f10d5be3740 20 client.27608 trim_cache size 0 max 16384 2024-04-02T14:42:01.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.573+0000 7f10d5be3740 1 Processor -- start 2024-04-02T14:42:01.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.573+0000 7f10d5be3740 1 -- start start 2024-04-02T14:42:01.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.573+0000 7f10d5be3740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5603d3f7c510 0x5603d3f84ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:01.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.573+0000 7f10d5be3740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5603d3f7d1f0 0x5603d3f9af90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:01.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.573+0000 7f10d5be3740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5603d3f85880 0x5603d3f9b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:01.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.573+0000 7f10d5be3740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f10ac00e750 con 0x5603d3f7d1f0 2024-04-02T14:42:01.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.573+0000 7f10d5be3740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f10ac010290 con 0x5603d3f85880 2024-04-02T14:42:01.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.573+0000 7f10d5be3740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f10ac00ee20 con 0x5603d3f7c510 2024-04-02T14:42:01.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.574+0000 7f10cbe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5603d3f7c510 0x5603d3f84ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:01.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.574+0000 7f10cb400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5603d3f7d1f0 0x5603d3f9af90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:01.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.574+0000 7f10d0e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5603d3f85880 0x5603d3f9b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:01.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.574+0000 7f10cbe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5603d3f7c510 0x5603d3f84ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:56690/0 (socket says 172.21.15.90:56690) 2024-04-02T14:42:01.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.574+0000 7f10cbe00640 1 -- 172.21.15.90:0/1581828826 learned_addr learned my addr 172.21.15.90:0/1581828826 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:42:01.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.574+0000 7f10cbe00640 1 -- 172.21.15.90:0/1581828826 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5603d3f85880 msgr2=0x5603d3f9b4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:42:01.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.574+0000 7f10cbe00640 1 --2- 172.21.15.90:0/1581828826 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5603d3f85880 0x5603d3f9b4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:01.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.574+0000 7f10cbe00640 1 -- 172.21.15.90:0/1581828826 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5603d3f7d1f0 msgr2=0x5603d3f9af90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:42:01.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.574+0000 7f10cbe00640 1 --2- 172.21.15.90:0/1581828826 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5603d3f7d1f0 0x5603d3f9af90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:01.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.574+0000 7f10cbe00640 1 -- 172.21.15.90:0/1581828826 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5603d3f7b540 con 0x5603d3f7c510 2024-04-02T14:42:01.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.574+0000 7f10cbe00640 1 --2- 172.21.15.90:0/1581828826 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5603d3f7c510 0x5603d3f84ef0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f10bc0da410 tx=0x7f10bc0ce0e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=39234a5b93410edb server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:42:01.576 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.575+0000 7f10c0c00640 1 -- 172.21.15.90:0/1581828826 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10bc0df070 con 0x5603d3f7c510 2024-04-02T14:42:01.576 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.575+0000 7f10c0c00640 1 -- 172.21.15.90:0/1581828826 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f10bc085e50 con 0x5603d3f7c510 2024-04-02T14:42:01.576 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.575+0000 7f10c0c00640 1 -- 172.21.15.90:0/1581828826 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10bc09c270 con 0x5603d3f7c510 2024-04-02T14:42:01.576 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.575+0000 7f10d5be3740 1 -- 172.21.15.90:0/1581828826 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5603d3f7c510 msgr2=0x5603d3f84ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:01.576 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.575+0000 7f10d5be3740 1 --2- 172.21.15.90:0/1581828826 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5603d3f7c510 0x5603d3f84ef0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f10bc0da410 tx=0x7f10bc0ce0e0 comp rx=0 tx=0).stop 2024-04-02T14:42:01.576 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.575+0000 7f10d5be3740 1 -- 172.21.15.90:0/1581828826 shutdown_connections 2024-04-02T14:42:01.576 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.575+0000 7f10d5be3740 1 --2- 172.21.15.90:0/1581828826 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5603d3f85880 0x5603d3f9b4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:01.576 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.575+0000 7f10d5be3740 1 --2- 172.21.15.90:0/1581828826 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5603d3f7d1f0 0x5603d3f9af90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:01.576 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.575+0000 7f10d5be3740 1 --2- 172.21.15.90:0/1581828826 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5603d3f7c510 0x5603d3f84ef0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:01.576 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.575+0000 7f10d5be3740 1 -- 172.21.15.90:0/1581828826 >> 172.21.15.90:0/1581828826 conn(0x5603d3f10270 msgr2=0x5603d3f75f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:42:01.576 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.576+0000 7f10d5be3740 1 -- 172.21.15.90:0/1581828826 shutdown_connections 2024-04-02T14:42:01.576 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.576+0000 7f10d5be3740 1 -- 172.21.15.90:0/1581828826 wait complete. 2024-04-02T14:42:01.576 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.576+0000 7f10d5be3740 1 -- start start 2024-04-02T14:42:01.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.576+0000 7f10d5be3740 1 Processor -- start 2024-04-02T14:42:01.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.577+0000 7f10d5be3740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5603d3f7c510 0x5603d3f9b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:01.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.577+0000 7f10d5be3740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5603d3f7d1f0 0x5603d3f99490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:01.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.577+0000 7f10d5be3740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5603d3f85880 0x5603d3f999d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:01.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.577+0000 7f10d5be3740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f10ac00fbc0 con 0x5603d3f7d1f0 2024-04-02T14:42:01.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.577+0000 7f10d5be3740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f10ac00f4f0 con 0x5603d3f7c510 2024-04-02T14:42:01.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.577+0000 7f10d5be3740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f10ac00c730 con 0x5603d3f85880 2024-04-02T14:42:01.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.577+0000 7f10d5be3740 10 client.0 _ll_register_callbacks cb 0x7f10bc005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:42:01.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.577+0000 7f10cbe00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5603d3f7c510 0x5603d3f9b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:01.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.577+0000 7f10d0e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5603d3f85880 0x5603d3f999d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:01.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.577+0000 7f10cbe00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5603d3f7c510 0x5603d3f9b0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:36870/0 (socket says 172.21.15.90:36870) 2024-04-02T14:42:01.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.577+0000 7f10cbe00640 1 -- 172.21.15.90:0/2562998597 learned_addr learned my addr 172.21.15.90:0/2562998597 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:42:01.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.577+0000 7f10cb400640 1 --2- 172.21.15.90:0/2562998597 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5603d3f7d1f0 0x5603d3f99490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:01.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.578+0000 7f10cbe00640 1 -- 172.21.15.90:0/2562998597 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5603d3f85880 msgr2=0x5603d3f999d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:42:01.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.578+0000 7f10cbe00640 1 --2- 172.21.15.90:0/2562998597 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5603d3f85880 0x5603d3f999d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:01.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.578+0000 7f10cbe00640 1 -- 172.21.15.90:0/2562998597 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5603d3f7d1f0 msgr2=0x5603d3f99490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:42:01.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.578+0000 7f10cbe00640 1 --2- 172.21.15.90:0/2562998597 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5603d3f7d1f0 0x5603d3f99490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:01.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.578+0000 7f10cbe00640 1 -- 172.21.15.90:0/2562998597 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5603d3f99f90 con 0x5603d3f7c510 2024-04-02T14:42:01.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.578+0000 7f10cb400640 1 --2- 172.21.15.90:0/2562998597 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5603d3f7d1f0 0x5603d3f99490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:42:01.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.578+0000 7f10cbe00640 1 --2- 172.21.15.90:0/2562998597 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5603d3f7c510 0x5603d3f9b0f0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f10bc0992e0 tx=0x7f10bc009450 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d992a4d26ebecc03 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:42:01.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.578+0000 7f10b6000640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:42:01.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.578+0000 7f10b6000640 1 -- 172.21.15.90:0/2562998597 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10bc0df070 con 0x5603d3f7c510 2024-04-02T14:42:01.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.578+0000 7f10b6000640 1 -- 172.21.15.90:0/2562998597 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f10bc0a52f0 con 0x5603d3f7c510 2024-04-02T14:42:01.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.578+0000 7f10d5be3740 1 -- 172.21.15.90:0/2562998597 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f10bc085e50 con 0x5603d3f7c510 2024-04-02T14:42:01.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.578+0000 7f10b6000640 1 -- 172.21.15.90:0/2562998597 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f10bc09c450 con 0x5603d3f7c510 2024-04-02T14:42:01.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.578+0000 7f10d5be3740 10 client.27614 fetch_fsmap learned FSMap version 27 2024-04-02T14:42:01.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.578+0000 7f10d5be3740 1 -- 172.21.15.90:0/2562998597 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5603d3f99f90 con 0x5603d3f7c510 2024-04-02T14:42:01.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.578+0000 7f10b6000640 1 -- 172.21.15.90:0/2562998597 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f10bc085e50 con 0x5603d3f7c510 2024-04-02T14:42:01.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.579+0000 7f10b6000640 1 -- 172.21.15.90:0/2562998597 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f10bc0d8be0 con 0x5603d3f7c510 2024-04-02T14:42:01.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.579+0000 7f10d5be3740 10 client.27614 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:42:01.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.579+0000 7f10d5be3740 10 client.27614 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:42:01.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.579+0000 7f10d5be3740 20 client.27614 populate_metadata read hostname 'smithi090' 2024-04-02T14:42:01.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.579+0000 7f10d5be3740 1 --2- 172.21.15.90:0/2562998597 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5603d3ffce30 0x5603d3fff220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:01.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.579+0000 7f10d5be3740 4 client.27614 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-02T14:42:01.582 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.579+0000 7f10d5be3740 1 -- 172.21.15.90:0/2562998597 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5603d3ee7ba0 con 0x5603d3ffce30 2024-04-02T14:42:01.582 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.579+0000 7f10cb400640 1 --2- 172.21.15.90:0/2562998597 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5603d3ffce30 0x5603d3fff220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:01.582 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.579+0000 7f10cb400640 1 --2- 172.21.15.90:0/2562998597 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5603d3ffce30 0x5603d3fff220 crc :-1 s=READY pgs=197 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f3ad5f1727ca0dbc server_cookie=7b01542fbc78ac4b in_seq=0 out_seq=0 2024-04-02T14:42:01.582 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.579+0000 7f10b6000640 10 client.27614 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:42:01.583 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.580+0000 7f10b6000640 1 -- 172.21.15.90:0/2562998597 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5603d3ee7ba0 con 0x5603d3ffce30 2024-04-02T14:42:01.583 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.580+0000 7f10b6000640 10 client.27614 handle_command_reply: tid=0 2024-04-02T14:42:01.583 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.580+0000 7f10d5be3740 1 client.27614 shutdown 2024-04-02T14:42:01.583 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.581+0000 7f10d5be3740 1 -- 172.21.15.90:0/2562998597 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5603d3f7c510 msgr2=0x5603d3f9b0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:01.583 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.581+0000 7f10d5be3740 1 --2- 172.21.15.90:0/2562998597 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5603d3f7c510 0x5603d3f9b0f0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f10bc0992e0 tx=0x7f10bc009450 comp rx=0 tx=0).stop 2024-04-02T14:42:01.583 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.581+0000 7f10d5be3740 1 -- 172.21.15.90:0/2562998597 shutdown_connections 2024-04-02T14:42:01.583 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.581+0000 7f10d5be3740 1 --2- 172.21.15.90:0/2562998597 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5603d3ffce30 0x5603d3fff220 crc :-1 s=READY pgs=197 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:01.583 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.581+0000 7f10d5be3740 1 --2- 172.21.15.90:0/2562998597 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5603d3f85880 0x5603d3f999d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:01.583 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.581+0000 7f10d5be3740 1 --2- 172.21.15.90:0/2562998597 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5603d3f7d1f0 0x5603d3f99490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:01.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.581+0000 7f10d5be3740 1 --2- 172.21.15.90:0/2562998597 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5603d3f7c510 0x5603d3f9b0f0 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:01.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.581+0000 7f10d5be3740 1 -- 172.21.15.90:0/2562998597 >> 172.21.15.90:0/2562998597 conn(0x5603d3f10270 msgr2=0x5603d3f78310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:42:01.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.581+0000 7f10b6000640 0 client.27614 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:42:01.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.581+0000 7f10d5be3740 1 -- 172.21.15.90:0/2562998597 shutdown_connections 2024-04-02T14:42:01.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.581+0000 7f10d5be3740 1 -- 172.21.15.90:0/2562998597 wait complete. 2024-04-02T14:42:01.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.581+0000 7f10d5be3740 20 client.27614 trim_cache size 0 max 16384 2024-04-02T14:42:01.584 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-02T14:42:01.584 INFO:teuthology.orchestra.run.smithi090.stdout: "return_code": 0, 2024-04-02T14:42:01.584 INFO:teuthology.orchestra.run.smithi090.stdout: "scrub_tag": "9a300704-3b61-4929-813d-4cd4f5e9cc50", 2024-04-02T14:42:01.584 INFO:teuthology.orchestra.run.smithi090.stdout: "mode": "asynchronous" 2024-04-02T14:42:01.584 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-02T14:42:01.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.582+0000 7f10d1800640 1 -- 172.21.15.90:0/2949055224 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f10b00a3da0 msgr2=0x7f10b00a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:01.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.582+0000 7f10d1800640 1 --2- 172.21.15.90:0/2949055224 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f10b00a3da0 0x7f10b00a6260 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f10c4004af0 tx=0x7f10c4005c50 comp rx=0 tx=0).stop 2024-04-02T14:42:01.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.582+0000 7f10d1800640 1 -- 172.21.15.90:0/2949055224 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f10cc102bd0 msgr2=0x7f10cc19a450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:01.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.582+0000 7f10d1800640 1 --2- 172.21.15.90:0/2949055224 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f10cc102bd0 0x7f10cc19a450 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f10bc00e9f0 tx=0x7f10bc00eeb0 comp rx=0 tx=0).stop 2024-04-02T14:42:01.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.582+0000 7f10d1800640 1 -- 172.21.15.90:0/2949055224 shutdown_connections 2024-04-02T14:42:01.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.582+0000 7f10d1800640 1 --2- 172.21.15.90:0/2949055224 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f10b00a3da0 0x7f10b00a6260 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:01.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.582+0000 7f10d1800640 1 --2- 172.21.15.90:0/2949055224 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f10cc10bff0 0x7f10cc19ed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:01.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.582+0000 7f10d1800640 1 --2- 172.21.15.90:0/2949055224 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f10cc1034f0 0x7f10cc19a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:01.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.582+0000 7f10d1800640 1 --2- 172.21.15.90:0/2949055224 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f10cc102bd0 0x7f10cc19a450 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:01.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.582+0000 7f10d1800640 1 -- 172.21.15.90:0/2949055224 >> 172.21.15.90:0/2949055224 conn(0x7f10cc0fc410 msgr2=0x7f10cc100750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:42:01.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.582+0000 7f10d1800640 1 -- 172.21.15.90:0/2949055224 shutdown_connections 2024-04-02T14:42:01.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.582+0000 7f10d1800640 1 -- 172.21.15.90:0/2949055224 wait complete. 2024-04-02T14:42:01.598 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-02T14:42:01.598 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-02T14:42:01.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.813+0000 7f232be00640 1 -- 172.21.15.90:0/32713372 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f232c1082b0 msgr2=0x7f232c108690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:01.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.813+0000 7f232be00640 1 --2- 172.21.15.90:0/32713372 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f232c1082b0 0x7f232c108690 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f231c009860 tx=0x7f231c02bb70 comp rx=0 tx=0).stop 2024-04-02T14:42:01.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.813+0000 7f232be00640 1 -- 172.21.15.90:0/32713372 shutdown_connections 2024-04-02T14:42:01.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.813+0000 7f232be00640 1 --2- 172.21.15.90:0/32713372 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f232c10d300 0x7f232c114ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:01.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.813+0000 7f232be00640 1 --2- 172.21.15.90:0/32713372 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f232c108c60 0x7f232c10cc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:01.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.813+0000 7f232be00640 1 --2- 172.21.15.90:0/32713372 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f232c1082b0 0x7f232c108690 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:01.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.813+0000 7f232be00640 1 -- 172.21.15.90:0/32713372 >> 172.21.15.90:0/32713372 conn(0x7f232c077920 msgr2=0x7f232c106920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:42:01.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.813+0000 7f232be00640 1 -- 172.21.15.90:0/32713372 shutdown_connections 2024-04-02T14:42:01.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.813+0000 7f232be00640 1 -- 172.21.15.90:0/32713372 wait complete. 2024-04-02T14:42:01.815 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.814+0000 7f232be00640 1 Processor -- start 2024-04-02T14:42:01.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.814+0000 7f232be00640 1 -- start start 2024-04-02T14:42:01.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.815+0000 7f232be00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f232c1082b0 0x7f232c1a45e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:01.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.815+0000 7f232be00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f232c10d300 0x7f232c1a4b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:01.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.815+0000 7f232be00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f232c19e6f0 0x7f232c19eb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:01.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.815+0000 7f232be00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f232c1183d0 con 0x7f232c1082b0 2024-04-02T14:42:01.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.815+0000 7f232be00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f232c118250 con 0x7f232c19e6f0 2024-04-02T14:42:01.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.815+0000 7f232be00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f232c118550 con 0x7f232c10d300 2024-04-02T14:42:01.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.815+0000 7f232aa00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f232c1082b0 0x7f232c1a45e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:01.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.815+0000 7f232a000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f232c10d300 0x7f232c1a4b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:01.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.815+0000 7f232b400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f232c19e6f0 0x7f232c19eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:01.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.815+0000 7f232a000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f232c10d300 0x7f232c1a4b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:56714/0 (socket says 172.21.15.90:56714) 2024-04-02T14:42:01.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.815+0000 7f232a000640 1 -- 172.21.15.90:0/503672238 learned_addr learned my addr 172.21.15.90:0/503672238 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:42:01.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.815+0000 7f232b400640 1 -- 172.21.15.90:0/503672238 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f232c10d300 msgr2=0x7f232c1a4b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:01.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.815+0000 7f232b400640 1 --2- 172.21.15.90:0/503672238 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f232c10d300 0x7f232c1a4b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:01.817 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.815+0000 7f232b400640 1 -- 172.21.15.90:0/503672238 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f232c1082b0 msgr2=0x7f232c1a45e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:01.817 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.815+0000 7f232b400640 1 --2- 172.21.15.90:0/503672238 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f232c1082b0 0x7f232c1a45e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:01.817 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.815+0000 7f232b400640 1 -- 172.21.15.90:0/503672238 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f232c19f2f0 con 0x7f232c19e6f0 2024-04-02T14:42:01.817 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.815+0000 7f232aa00640 1 --2- 172.21.15.90:0/503672238 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f232c1082b0 0x7f232c1a45e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:42:01.817 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.816+0000 7f232b400640 1 --2- 172.21.15.90:0/503672238 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f232c19e6f0 0x7f232c19eb70 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f2324005fd0 tx=0x7f232400eda0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:42:01.817 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.816+0000 7f2323400640 1 -- 172.21.15.90:0/503672238 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f232400cd80 con 0x7f232c19e6f0 2024-04-02T14:42:01.817 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.816+0000 7f232be00640 1 -- 172.21.15.90:0/503672238 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f232c1ab380 con 0x7f232c19e6f0 2024-04-02T14:42:01.817 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.816+0000 7f2323400640 1 -- 172.21.15.90:0/503672238 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2324005020 con 0x7f232c19e6f0 2024-04-02T14:42:01.817 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.816+0000 7f2323400640 1 -- 172.21.15.90:0/503672238 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2324010690 con 0x7f232c19e6f0 2024-04-02T14:42:01.817 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.816+0000 7f232be00640 1 -- 172.21.15.90:0/503672238 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f232c1ab780 con 0x7f232c19e6f0 2024-04-02T14:42:01.818 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.819+0000 7f23347d6740 1 Processor -- start 2024-04-02T14:42:01.818 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.819+0000 7f2323400640 1 -- 172.21.15.90:0/503672238 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f2324005540 con 0x7f232c19e6f0 2024-04-02T14:42:01.820 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.819+0000 7f23347d6740 1 -- start start 2024-04-02T14:42:01.820 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.820+0000 7f23347d6740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55902bfaa960 0x55902bfaad40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:01.820 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.820+0000 7f23347d6740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55902bfab280 0x55902bfb3920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:01.820 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.820+0000 7f23347d6740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55902bfb3e60 0x55902bfb6250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:01.820 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.820+0000 7f23347d6740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55902bf0e770 con 0x55902bfaa960 2024-04-02T14:42:01.820 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.820+0000 7f23347d6740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55902beb6700 con 0x55902bfab280 2024-04-02T14:42:01.820 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.820+0000 7f23347d6740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55902bf0a720 con 0x55902bfb3e60 2024-04-02T14:42:01.820 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.820+0000 7f232a000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55902bfaa960 0x55902bfaad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:01.820 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.820+0000 7f232aa00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55902bfab280 0x55902bfb3920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:01.820 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.820+0000 7f232a000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55902bfaa960 0x55902bfaad40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38356/0 (socket says 172.21.15.90:38356) 2024-04-02T14:42:01.820 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.820+0000 7f232a000640 1 -- 172.21.15.90:0/2942335632 learned_addr learned my addr 172.21.15.90:0/2942335632 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:42:01.820 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.820+0000 7f232a000640 1 --2- 172.21.15.90:0/2942335632 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55902bfb3e60 0x55902bfb6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:01.821 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.820+0000 7f232aa00640 1 -- 172.21.15.90:0/2942335632 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55902bfb3e60 msgr2=0x55902bfb6250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:42:01.821 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.820+0000 7f232aa00640 1 --2- 172.21.15.90:0/2942335632 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55902bfb3e60 0x55902bfb6250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:01.821 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.820+0000 7f232aa00640 1 -- 172.21.15.90:0/2942335632 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55902bfaa960 msgr2=0x55902bfaad40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:42:01.821 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.820+0000 7f232aa00640 1 --2- 172.21.15.90:0/2942335632 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55902bfaa960 0x55902bfaad40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:01.821 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.820+0000 7f232aa00640 1 -- 172.21.15.90:0/2942335632 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55902bcec7d0 con 0x55902bfab280 2024-04-02T14:42:01.821 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.821+0000 7f232aa00640 1 --2- 172.21.15.90:0/2942335632 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55902bfab280 0x55902bfb3920 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f231c037040 tx=0x7f231c0056d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9ec69bd007272a2e server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:42:01.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.821+0000 7f231be00640 1 -- 172.21.15.90:0/2942335632 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f231c044020 con 0x55902bfab280 2024-04-02T14:42:01.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.821+0000 7f231be00640 1 -- 172.21.15.90:0/2942335632 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f231c034440 con 0x55902bfab280 2024-04-02T14:42:01.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.821+0000 7f231be00640 1 -- 172.21.15.90:0/2942335632 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f231c039040 con 0x55902bfab280 2024-04-02T14:42:01.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.821+0000 7f2323400640 1 --2- 172.21.15.90:0/503672238 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f23100a3d20 0x7f23100a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:01.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.821+0000 7f2323400640 1 -- 172.21.15.90:0/503672238 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f23240d0df0 con 0x7f232c19e6f0 2024-04-02T14:42:01.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.821+0000 7f232a000640 1 --2- 172.21.15.90:0/503672238 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f23100a3d20 0x7f23100a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:01.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.822+0000 7f23347d6740 1 -- 172.21.15.90:0/2942335632 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55902bfab280 msgr2=0x55902bfb3920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:01.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.822+0000 7f23347d6740 1 --2- 172.21.15.90:0/2942335632 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55902bfab280 0x55902bfb3920 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f231c037040 tx=0x7f231c0056d0 comp rx=0 tx=0).stop 2024-04-02T14:42:01.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.822+0000 7f23347d6740 1 -- 172.21.15.90:0/2942335632 shutdown_connections 2024-04-02T14:42:01.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.822+0000 7f23347d6740 1 --2- 172.21.15.90:0/2942335632 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55902bfb3e60 0x55902bfb6250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:01.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.822+0000 7f23347d6740 1 --2- 172.21.15.90:0/2942335632 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55902bfab280 0x55902bfb3920 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:01.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.822+0000 7f23347d6740 1 --2- 172.21.15.90:0/2942335632 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55902bfaa960 0x55902bfaad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:01.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.822+0000 7f23347d6740 1 -- 172.21.15.90:0/2942335632 >> 172.21.15.90:0/2942335632 conn(0x55902bfa5090 msgr2=0x55902bfa5470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:42:01.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.822+0000 7f23347d6740 1 -- 172.21.15.90:0/2942335632 shutdown_connections 2024-04-02T14:42:01.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.822+0000 7f23347d6740 1 -- 172.21.15.90:0/2942335632 wait complete. 2024-04-02T14:42:01.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.822+0000 7f232a000640 1 --2- 172.21.15.90:0/503672238 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f23100a3d20 0x7f23100a61e0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x55902bfa6220 tx=0x7f2314015040 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:42:01.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.822+0000 7f23347d6740 1 -- start start 2024-04-02T14:42:01.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.822+0000 7f23347d6740 1 Processor -- start 2024-04-02T14:42:01.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.823+0000 7f23347d6740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55902bfaa960 0x55902bfb0e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:01.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.823+0000 7f23347d6740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55902bfab280 0x55902bfb1350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:01.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.823+0000 7f23347d6740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55902bfb3e60 0x55902bfb1890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:01.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.823+0000 7f23347d6740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55902bfb6bf0 con 0x55902bfab280 2024-04-02T14:42:01.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.823+0000 7f23347d6740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55902bec9ac0 con 0x55902bfaa960 2024-04-02T14:42:01.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.823+0000 7f23347d6740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55902bfb6d70 con 0x55902bfb3e60 2024-04-02T14:42:01.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.823+0000 7f23347d6740 10 client.0 _ll_register_callbacks cb 0x55902becc180 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:42:01.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.823+0000 7f232aa00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55902bfaa960 0x55902bfb0e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:01.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.823+0000 7f232aa00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55902bfaa960 0x55902bfb0e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:36912/0 (socket says 172.21.15.90:36912) 2024-04-02T14:42:01.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.823+0000 7f232aa00640 1 -- 172.21.15.90:0/4105190256 learned_addr learned my addr 172.21.15.90:0/4105190256 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:42:01.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.823+0000 7f232b400640 1 --2- 172.21.15.90:0/4105190256 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55902bfb3e60 0x55902bfb1890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:01.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.824+0000 7f232a000640 1 --2- 172.21.15.90:0/4105190256 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55902bfab280 0x55902bfb1350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:01.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.824+0000 7f232aa00640 1 -- 172.21.15.90:0/4105190256 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55902bfb3e60 msgr2=0x55902bfb1890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:42:01.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.824+0000 7f232aa00640 1 --2- 172.21.15.90:0/4105190256 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55902bfb3e60 0x55902bfb1890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:01.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.824+0000 7f232aa00640 1 -- 172.21.15.90:0/4105190256 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55902bfab280 msgr2=0x55902bfb1350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:42:01.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.824+0000 7f232aa00640 1 --2- 172.21.15.90:0/4105190256 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55902bfab280 0x55902bfb1350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:01.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.824+0000 7f232aa00640 1 -- 172.21.15.90:0/4105190256 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55902bc223f0 con 0x55902bfaa960 2024-04-02T14:42:01.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.824+0000 7f232a000640 1 --2- 172.21.15.90:0/4105190256 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55902bfab280 0x55902bfb1350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:42:01.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.824+0000 7f232aa00640 1 --2- 172.21.15.90:0/4105190256 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55902bfaa960 0x55902bfb0e10 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f231c0095f0 tx=0x7f231c005890 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6598386542ddaca6 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:42:01.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.824+0000 7f2318c00640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:42:01.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.824+0000 7f2318c00640 1 -- 172.21.15.90:0/4105190256 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f231c050070 con 0x55902bfaa960 2024-04-02T14:42:01.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.824+0000 7f2318c00640 1 -- 172.21.15.90:0/4105190256 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f231c02c700 con 0x55902bfaa960 2024-04-02T14:42:01.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.824+0000 7f23347d6740 1 -- 172.21.15.90:0/4105190256 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f231c034440 con 0x55902bfaa960 2024-04-02T14:42:01.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.825+0000 7f2318c00640 1 -- 172.21.15.90:0/4105190256 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f231c05c050 con 0x55902bfaa960 2024-04-02T14:42:01.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.825+0000 7f23347d6740 10 client.27626 fetch_fsmap learned FSMap version 27 2024-04-02T14:42:01.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.825+0000 7f23347d6740 1 -- 172.21.15.90:0/4105190256 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55902bbe20f0 con 0x55902bfaa960 2024-04-02T14:42:01.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.825+0000 7f2318c00640 1 -- 172.21.15.90:0/4105190256 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f231c034440 con 0x55902bfaa960 2024-04-02T14:42:01.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.825+0000 7f2318c00640 1 -- 172.21.15.90:0/4105190256 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f231c0035f0 con 0x55902bfaa960 2024-04-02T14:42:01.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.825+0000 7f23347d6740 10 client.27626 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:42:01.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.825+0000 7f23347d6740 10 client.27626 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:42:01.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.825+0000 7f23347d6740 20 client.27626 populate_metadata read hostname 'smithi090' 2024-04-02T14:42:01.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.825+0000 7f23347d6740 1 --2- 172.21.15.90:0/4105190256 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55902bfc3c00 0x55902bfc5ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:01.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.825+0000 7f23347d6740 4 client.27626 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:42:01.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.825+0000 7f23347d6740 1 -- 172.21.15.90:0/4105190256 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55902beaf630 con 0x55902bfc3c00 2024-04-02T14:42:01.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.826+0000 7f232a000640 1 --2- 172.21.15.90:0/4105190256 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55902bfc3c00 0x55902bfc5ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:01.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.826+0000 7f232a000640 1 --2- 172.21.15.90:0/4105190256 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55902bfc3c00 0x55902bfc5ff0 crc :-1 s=READY pgs=198 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5782a781277d104c server_cookie=f30fd6b3dd75f77d in_seq=0 out_seq=0 2024-04-02T14:42:01.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.826+0000 7f2318c00640 10 client.27626 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:42:01.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.827+0000 7f2318c00640 1 -- 172.21.15.90:0/4105190256 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55902beaf630 con 0x55902bfc3c00 2024-04-02T14:42:01.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.827+0000 7f2318c00640 10 client.27626 handle_command_reply: tid=0 2024-04-02T14:42:01.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.827+0000 7f23347d6740 1 client.27626 shutdown 2024-04-02T14:42:01.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.827+0000 7f23347d6740 1 -- 172.21.15.90:0/4105190256 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55902bfaa960 msgr2=0x55902bfb0e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:01.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.827+0000 7f23347d6740 1 --2- 172.21.15.90:0/4105190256 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55902bfaa960 0x55902bfb0e10 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f231c0095f0 tx=0x7f231c005890 comp rx=0 tx=0).stop 2024-04-02T14:42:01.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.827+0000 7f23347d6740 1 -- 172.21.15.90:0/4105190256 shutdown_connections 2024-04-02T14:42:01.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.827+0000 7f23347d6740 1 --2- 172.21.15.90:0/4105190256 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55902bfc3c00 0x55902bfc5ff0 crc :-1 s=READY pgs=198 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:01.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.827+0000 7f23347d6740 1 --2- 172.21.15.90:0/4105190256 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55902bfb3e60 0x55902bfb1890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:01.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.827+0000 7f2318c00640 0 client.27626 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:42:01.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.827+0000 7f23347d6740 1 --2- 172.21.15.90:0/4105190256 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55902bfab280 0x55902bfb1350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:01.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.827+0000 7f23347d6740 1 --2- 172.21.15.90:0/4105190256 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55902bfaa960 0x55902bfb0e10 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:01.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.827+0000 7f23347d6740 1 -- 172.21.15.90:0/4105190256 >> 172.21.15.90:0/4105190256 conn(0x55902bfa5090 msgr2=0x55902bfa4310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:42:01.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.827+0000 7f23347d6740 1 -- 172.21.15.90:0/4105190256 shutdown_connections 2024-04-02T14:42:01.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.827+0000 7f23347d6740 1 -- 172.21.15.90:0/4105190256 wait complete. 2024-04-02T14:42:01.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.828+0000 7f23347d6740 20 client.27626 trim_cache size 0 max 16384 2024-04-02T14:42:01.831 INFO:teuthology.orchestra.run.smithi090.stderr:progress: 390.0 s, 2051.8 tps, lat 46.085 ms stddev 70.063 2024-04-02T14:42:01.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.846+0000 7f23347d6740 1 Processor -- start 2024-04-02T14:42:01.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.847+0000 7f23347d6740 1 -- start start 2024-04-02T14:42:01.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.847+0000 7f23347d6740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55902bfaa960 0x55902bfb3540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:01.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.847+0000 7f23347d6740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55902bfab280 0x55902bfbfab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:01.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.847+0000 7f23347d6740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55902bfb3e60 0x55902bfa5470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:01.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.847+0000 7f23347d6740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f232c1b3ba0 con 0x55902bfb3e60 2024-04-02T14:42:01.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.847+0000 7f23347d6740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f232c1b56e0 con 0x55902bfaa960 2024-04-02T14:42:01.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.847+0000 7f23347d6740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f232c1b4270 con 0x55902bfab280 2024-04-02T14:42:01.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.847+0000 7f232aa00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55902bfaa960 0x55902bfb3540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:01.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.847+0000 7f232a000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55902bfab280 0x55902bfbfab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:01.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.847+0000 7f232aa00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55902bfaa960 0x55902bfb3540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:36928/0 (socket says 172.21.15.90:36928) 2024-04-02T14:42:01.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.847+0000 7f232aa00640 1 -- 172.21.15.90:0/3174660363 learned_addr learned my addr 172.21.15.90:0/3174660363 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:42:01.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.847+0000 7f232b400640 1 --2- 172.21.15.90:0/3174660363 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55902bfb3e60 0x55902bfa5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:01.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.847+0000 7f232aa00640 1 -- 172.21.15.90:0/3174660363 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55902bfab280 msgr2=0x55902bfbfab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:42:01.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.847+0000 7f232aa00640 1 --2- 172.21.15.90:0/3174660363 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55902bfab280 0x55902bfbfab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:01.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.848+0000 7f232aa00640 1 -- 172.21.15.90:0/3174660363 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55902bfb3e60 msgr2=0x55902bfa5470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:42:01.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.848+0000 7f232aa00640 1 --2- 172.21.15.90:0/3174660363 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55902bfb3e60 0x55902bfa5470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:01.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.848+0000 7f232aa00640 1 -- 172.21.15.90:0/3174660363 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55902bfab0b0 con 0x55902bfaa960 2024-04-02T14:42:01.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.848+0000 7f232b400640 1 --2- 172.21.15.90:0/3174660363 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55902bfb3e60 0x55902bfa5470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:42:01.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.848+0000 7f232aa00640 1 --2- 172.21.15.90:0/3174660363 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55902bfaa960 0x55902bfb3540 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f231c02c0b0 tx=0x7f231c0056d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f99ff0f4008c54b8 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:42:01.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.848+0000 7f231b400640 1 -- 172.21.15.90:0/3174660363 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f231c050040 con 0x55902bfaa960 2024-04-02T14:42:01.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.848+0000 7f231b400640 1 -- 172.21.15.90:0/3174660363 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f231c02cab0 con 0x55902bfaa960 2024-04-02T14:42:01.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.848+0000 7f231b400640 1 -- 172.21.15.90:0/3174660363 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f231c044040 con 0x55902bfaa960 2024-04-02T14:42:01.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.849+0000 7f23347d6740 1 -- 172.21.15.90:0/3174660363 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55902bfaa960 msgr2=0x55902bfb3540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:01.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.849+0000 7f23347d6740 1 --2- 172.21.15.90:0/3174660363 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55902bfaa960 0x55902bfb3540 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f231c02c0b0 tx=0x7f231c0056d0 comp rx=0 tx=0).stop 2024-04-02T14:42:01.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.849+0000 7f23347d6740 1 -- 172.21.15.90:0/3174660363 shutdown_connections 2024-04-02T14:42:01.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.849+0000 7f23347d6740 1 --2- 172.21.15.90:0/3174660363 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55902bfb3e60 0x55902bfa5470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:01.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.849+0000 7f23347d6740 1 --2- 172.21.15.90:0/3174660363 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55902bfab280 0x55902bfbfab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:01.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.849+0000 7f23347d6740 1 --2- 172.21.15.90:0/3174660363 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55902bfaa960 0x55902bfb3540 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:01.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.849+0000 7f23347d6740 1 -- 172.21.15.90:0/3174660363 >> 172.21.15.90:0/3174660363 conn(0x55902bfa5090 msgr2=0x55902bfa3a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:42:01.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.849+0000 7f23347d6740 1 -- 172.21.15.90:0/3174660363 shutdown_connections 2024-04-02T14:42:01.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.849+0000 7f23347d6740 1 -- 172.21.15.90:0/3174660363 wait complete. 2024-04-02T14:42:01.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.849+0000 7f23347d6740 1 -- start start 2024-04-02T14:42:01.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.850+0000 7f23347d6740 1 Processor -- start 2024-04-02T14:42:01.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.850+0000 7f23347d6740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55902bfaa960 0x55902bfbc3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:01.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.850+0000 7f23347d6740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55902bfab280 0x55902bfad840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:01.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.850+0000 7f23347d6740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55902bfb3e60 0x55902bfade00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:01.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.850+0000 7f23347d6740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f232c1b5010 con 0x55902bfab280 2024-04-02T14:42:01.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.850+0000 7f23347d6740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f232c1b4940 con 0x55902bfaa960 2024-04-02T14:42:01.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.850+0000 7f23347d6740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f232c1b1b80 con 0x55902bfb3e60 2024-04-02T14:42:01.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.850+0000 7f23347d6740 10 client.0 _ll_register_callbacks cb 0x7f2314009b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:42:01.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.850+0000 7f232aa00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55902bfaa960 0x55902bfbc3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:01.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.850+0000 7f232aa00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55902bfaa960 0x55902bfbc3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:36944/0 (socket says 172.21.15.90:36944) 2024-04-02T14:42:01.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.850+0000 7f232aa00640 1 -- 172.21.15.90:0/2384429081 learned_addr learned my addr 172.21.15.90:0/2384429081 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:42:01.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.850+0000 7f232a000640 1 --2- 172.21.15.90:0/2384429081 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55902bfab280 0x55902bfad840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:01.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.850+0000 7f232b400640 1 --2- 172.21.15.90:0/2384429081 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55902bfb3e60 0x55902bfade00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:01.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.850+0000 7f232aa00640 1 -- 172.21.15.90:0/2384429081 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55902bfb3e60 msgr2=0x55902bfade00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:42:01.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.851+0000 7f232aa00640 1 --2- 172.21.15.90:0/2384429081 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55902bfb3e60 0x55902bfade00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:01.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.851+0000 7f232aa00640 1 -- 172.21.15.90:0/2384429081 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55902bfab280 msgr2=0x55902bfad840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:42:01.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.851+0000 7f232aa00640 1 --2- 172.21.15.90:0/2384429081 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55902bfab280 0x55902bfad840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:01.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.851+0000 7f232aa00640 1 -- 172.21.15.90:0/2384429081 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55902bfae680 con 0x55902bfaa960 2024-04-02T14:42:01.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.851+0000 7f232aa00640 1 --2- 172.21.15.90:0/2384429081 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55902bfaa960 0x55902bfbc3e0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f231c003a10 tx=0x7f231c02bb70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1eaa14cbe5cec813 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:42:01.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.851+0000 7f2318c00640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:42:01.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.851+0000 7f2318c00640 1 -- 172.21.15.90:0/2384429081 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f231c044070 con 0x55902bfaa960 2024-04-02T14:42:01.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.851+0000 7f2318c00640 1 -- 172.21.15.90:0/2384429081 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8506+0+0 (secure 0 0 0) 0x7f231c003e00 con 0x55902bfaa960 2024-04-02T14:42:01.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.851+0000 7f23347d6740 1 -- 172.21.15.90:0/2384429081 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f231c02cab0 con 0x55902bfaa960 2024-04-02T14:42:01.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.851+0000 7f2318c00640 1 -- 172.21.15.90:0/2384429081 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f231c050040 con 0x55902bfaa960 2024-04-02T14:42:01.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.851+0000 7f23347d6740 10 client.27632 fetch_fsmap learned FSMap version 27 2024-04-02T14:42:01.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.852+0000 7f23347d6740 1 -- 172.21.15.90:0/2384429081 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55902c029fd0 con 0x55902bfaa960 2024-04-02T14:42:01.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.852+0000 7f2318c00640 1 -- 172.21.15.90:0/2384429081 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f231c02cab0 con 0x55902bfaa960 2024-04-02T14:42:01.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.852+0000 7f2318c00640 1 -- 172.21.15.90:0/2384429081 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f231c032990 con 0x55902bfaa960 2024-04-02T14:42:01.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.853+0000 7f23347d6740 10 client.27632 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:42:01.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.853+0000 7f23347d6740 10 client.27632 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:42:01.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.853+0000 7f23347d6740 20 client.27632 populate_metadata read hostname 'smithi090' 2024-04-02T14:42:01.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.853+0000 7f23347d6740 1 --2- 172.21.15.90:0/2384429081 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55902bfc3c00 0x55902c02c590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:01.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.853+0000 7f23347d6740 4 client.27632 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-02T14:42:01.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.853+0000 7f23347d6740 1 -- 172.21.15.90:0/2384429081 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55902bf1bcb0 con 0x55902bfc3c00 2024-04-02T14:42:01.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.853+0000 7f232a000640 1 --2- 172.21.15.90:0/2384429081 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55902bfc3c00 0x55902c02c590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:01.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.853+0000 7f2318c00640 10 client.27632 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:42:01.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.853+0000 7f232a000640 1 --2- 172.21.15.90:0/2384429081 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55902bfc3c00 0x55902c02c590 crc :-1 s=READY pgs=199 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ce3786ca8609fa0d server_cookie=b2f4cf2300c58ddf in_seq=0 out_seq=0 2024-04-02T14:42:01.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.854+0000 7f2318c00640 1 -- 172.21.15.90:0/2384429081 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55902bf1bcb0 con 0x55902bfc3c00 2024-04-02T14:42:01.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.854+0000 7f2318c00640 10 client.27632 handle_command_reply: tid=0 2024-04-02T14:42:01.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.854+0000 7f23347d6740 1 client.27632 shutdown 2024-04-02T14:42:01.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.854+0000 7f23347d6740 1 -- 172.21.15.90:0/2384429081 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55902bfaa960 msgr2=0x55902bfbc3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:01.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.854+0000 7f23347d6740 1 --2- 172.21.15.90:0/2384429081 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55902bfaa960 0x55902bfbc3e0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f231c003a10 tx=0x7f231c02bb70 comp rx=0 tx=0).stop 2024-04-02T14:42:01.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.854+0000 7f23347d6740 1 -- 172.21.15.90:0/2384429081 shutdown_connections 2024-04-02T14:42:01.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.854+0000 7f23347d6740 1 --2- 172.21.15.90:0/2384429081 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55902bfc3c00 0x55902c02c590 crc :-1 s=READY pgs=199 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:01.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.854+0000 7f23347d6740 1 --2- 172.21.15.90:0/2384429081 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55902bfb3e60 0x55902bfade00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:01.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.854+0000 7f23347d6740 1 --2- 172.21.15.90:0/2384429081 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55902bfab280 0x55902bfad840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:01.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.854+0000 7f2318c00640 0 client.27632 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:42:01.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.854+0000 7f23347d6740 1 --2- 172.21.15.90:0/2384429081 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55902bfaa960 0x55902bfbc3e0 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:01.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.854+0000 7f23347d6740 1 -- 172.21.15.90:0/2384429081 >> 172.21.15.90:0/2384429081 conn(0x55902bfa5090 msgr2=0x55902bfa4310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:42:01.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.854+0000 7f23347d6740 1 -- 172.21.15.90:0/2384429081 shutdown_connections 2024-04-02T14:42:01.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.854+0000 7f23347d6740 1 -- 172.21.15.90:0/2384429081 wait complete. 2024-04-02T14:42:01.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.854+0000 7f23347d6740 20 client.27632 trim_cache size 0 max 16384 2024-04-02T14:42:01.858 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-02T14:42:01.859 INFO:teuthology.orchestra.run.smithi090.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-02T14:42:01.859 INFO:teuthology.orchestra.run.smithi090.stdout: "scrubs": { 2024-04-02T14:42:01.859 INFO:teuthology.orchestra.run.smithi090.stdout: "9a300704-3b61-4929-813d-4cd4f5e9cc50": { 2024-04-02T14:42:01.859 INFO:teuthology.orchestra.run.smithi090.stdout: "path": "/", 2024-04-02T14:42:01.859 INFO:teuthology.orchestra.run.smithi090.stdout: "tag": "9a300704-3b61-4929-813d-4cd4f5e9cc50", 2024-04-02T14:42:01.859 INFO:teuthology.orchestra.run.smithi090.stdout: "options": "recursive,force" 2024-04-02T14:42:01.859 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:42:01.859 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:42:01.859 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-02T14:42:01.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.855+0000 7f232be00640 1 -- 172.21.15.90:0/503672238 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f23100a3d20 msgr2=0x7f23100a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:01.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.855+0000 7f232be00640 1 --2- 172.21.15.90:0/503672238 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f23100a3d20 0x7f23100a61e0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x55902bfa6220 tx=0x7f2314015040 comp rx=0 tx=0).stop 2024-04-02T14:42:01.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.855+0000 7f232be00640 1 -- 172.21.15.90:0/503672238 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f232c19e6f0 msgr2=0x7f232c19eb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:01.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.855+0000 7f232be00640 1 --2- 172.21.15.90:0/503672238 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f232c19e6f0 0x7f232c19eb70 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f2324005fd0 tx=0x7f232400eda0 comp rx=0 tx=0).stop 2024-04-02T14:42:01.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.855+0000 7f232be00640 1 -- 172.21.15.90:0/503672238 shutdown_connections 2024-04-02T14:42:01.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.855+0000 7f232be00640 1 --2- 172.21.15.90:0/503672238 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f23100a3d20 0x7f23100a61e0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:01.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.855+0000 7f232be00640 1 --2- 172.21.15.90:0/503672238 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f232c19e6f0 0x7f232c19eb70 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:01.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.855+0000 7f232be00640 1 --2- 172.21.15.90:0/503672238 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f232c10d300 0x7f232c1a4b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:01.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.855+0000 7f232be00640 1 --2- 172.21.15.90:0/503672238 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f232c1082b0 0x7f232c1a45e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:01.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.855+0000 7f232be00640 1 -- 172.21.15.90:0/503672238 >> 172.21.15.90:0/503672238 conn(0x7f232c077920 msgr2=0x7f232c079570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:42:01.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.855+0000 7f232be00640 1 -- 172.21.15.90:0/503672238 shutdown_connections 2024-04-02T14:42:01.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:01.855+0000 7f232be00640 1 -- 172.21.15.90:0/503672238 wait complete. 2024-04-02T14:42:01.872 INFO:tasks.cephfs.filesystem:scrub status for tag:9a300704-3b61-4929-813d-4cd4f5e9cc50 - {'path': '/', 'tag': '9a300704-3b61-4929-813d-4cd4f5e9cc50', 'options': 'recursive,force'} 2024-04-02T14:42:02.358 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:42:02 smithi092 ceph-mon[36497]: pgmap v217: 97 pgs: 97 active+clean; 4.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 7.9 MiB/s wr, 772 op/s 2024-04-02T14:42:02.358 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:42:02 smithi092 ceph-mon[36497]: scrub queued for path: / 2024-04-02T14:42:02.358 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:42:02 smithi092 ceph-mon[36497]: scrub summary: idle+waiting paths [/] 2024-04-02T14:42:02.358 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:42:02 smithi092 ceph-mon[36497]: scrub summary: active paths [/] 2024-04-02T14:42:02.358 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:42:02 smithi092 ceph-mon[36497]: scrub summary: idle+waiting paths [/] 2024-04-02T14:42:02.434 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:42:02 smithi177 ceph-mon[36431]: pgmap v217: 97 pgs: 97 active+clean; 4.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 7.9 MiB/s wr, 772 op/s 2024-04-02T14:42:02.434 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:42:02 smithi177 ceph-mon[36431]: scrub queued for path: / 2024-04-02T14:42:02.434 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:42:02 smithi177 ceph-mon[36431]: scrub summary: idle+waiting paths [/] 2024-04-02T14:42:02.434 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:42:02 smithi177 ceph-mon[36431]: scrub summary: active paths [/] 2024-04-02T14:42:02.434 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:42:02 smithi177 ceph-mon[36431]: scrub summary: idle+waiting paths [/] 2024-04-02T14:42:02.436 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:42:02 smithi090 ceph-mon[31674]: pgmap v217: 97 pgs: 97 active+clean; 4.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 7.9 MiB/s wr, 772 op/s 2024-04-02T14:42:02.436 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:42:02 smithi090 ceph-mon[31674]: scrub queued for path: / 2024-04-02T14:42:02.436 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:42:02 smithi090 ceph-mon[31674]: scrub summary: idle+waiting paths [/] 2024-04-02T14:42:02.436 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:42:02 smithi090 ceph-mon[31674]: scrub summary: active paths [/] 2024-04-02T14:42:02.436 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:42:02 smithi090 ceph-mon[31674]: scrub summary: idle+waiting paths [/] 2024-04-02T14:42:03.358 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:42:03 smithi092 ceph-mon[36497]: osdmap e92: 12 total, 12 up, 12 in 2024-04-02T14:42:03.434 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:42:03 smithi177 ceph-mon[36431]: osdmap e92: 12 total, 12 up, 12 in 2024-04-02T14:42:03.435 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:42:03 smithi090 ceph-mon[31674]: osdmap e92: 12 total, 12 up, 12 in 2024-04-02T14:42:04.358 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:42:04 smithi092 ceph-mon[36497]: pgmap v219: 97 pgs: 97 active+clean; 4.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 7.5 MiB/s wr, 744 op/s 2024-04-02T14:42:04.434 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:42:04 smithi177 ceph-mon[36431]: pgmap v219: 97 pgs: 97 active+clean; 4.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 7.5 MiB/s wr, 744 op/s 2024-04-02T14:42:04.436 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:42:04 smithi090 ceph-mon[31674]: pgmap v219: 97 pgs: 97 active+clean; 4.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 7.5 MiB/s wr, 744 op/s 2024-04-02T14:42:04.547 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.547+0000 7f8dbbc00640 1 -- 172.21.15.90:0/1902697510 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8db41048f0 msgr2=0x7f8db4108930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:04.547 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.547+0000 7f8dbbc00640 1 --2- 172.21.15.90:0/1902697510 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8db41048f0 0x7f8db4108930 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f8da0009900 tx=0x7f8da002bc30 comp rx=0 tx=0).stop 2024-04-02T14:42:04.547 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.548+0000 7f8dbbc00640 1 -- 172.21.15.90:0/1902697510 shutdown_connections 2024-04-02T14:42:04.548 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.548+0000 7f8dbbc00640 1 --2- 172.21.15.90:0/1902697510 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8db4109010 0x7f8db4110be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:04.548 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.548+0000 7f8dbbc00640 1 --2- 172.21.15.90:0/1902697510 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8db41048f0 0x7f8db4108930 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:04.548 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.548+0000 7f8dbbc00640 1 --2- 172.21.15.90:0/1902697510 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8db4103f40 0x7f8db4104320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:04.548 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.548+0000 7f8dbbc00640 1 -- 172.21.15.90:0/1902697510 >> 172.21.15.90:0/1902697510 conn(0x7f8db40ff7c0 msgr2=0x7f8db4101be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:42:04.548 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.548+0000 7f8dbbc00640 1 -- 172.21.15.90:0/1902697510 shutdown_connections 2024-04-02T14:42:04.548 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.548+0000 7f8dbbc00640 1 -- 172.21.15.90:0/1902697510 wait complete. 2024-04-02T14:42:04.550 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.549+0000 7f8dbbc00640 1 Processor -- start 2024-04-02T14:42:04.550 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.549+0000 7f8dbbc00640 1 -- start start 2024-04-02T14:42:04.550 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.549+0000 7f8dbbc00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8db4103f40 0x7f8db419a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:04.550 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.549+0000 7f8dbbc00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8db4109010 0x7f8db419ab30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:04.550 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.549+0000 7f8dbbc00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8db419ee60 0x7f8db419f310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:04.550 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.549+0000 7f8dbbc00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f8db41140f0 con 0x7f8db419ee60 2024-04-02T14:42:04.550 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.549+0000 7f8dbbc00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f8db4113f70 con 0x7f8db4109010 2024-04-02T14:42:04.550 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.549+0000 7f8dbbc00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f8db4114270 con 0x7f8db4103f40 2024-04-02T14:42:04.551 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.549+0000 7f8db3e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8db4109010 0x7f8db419ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:04.551 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.549+0000 7f8db9200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8db4103f40 0x7f8db419a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:04.551 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.549+0000 7f8db3e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8db4109010 0x7f8db419ab30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:50096/0 (socket says 172.21.15.90:50096) 2024-04-02T14:42:04.551 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.549+0000 7f8db9c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8db419ee60 0x7f8db419f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:04.551 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.549+0000 7f8db3e00640 1 -- 172.21.15.90:0/3327752529 learned_addr learned my addr 172.21.15.90:0/3327752529 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:42:04.551 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.550+0000 7f8db3e00640 1 -- 172.21.15.90:0/3327752529 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8db4103f40 msgr2=0x7f8db419a5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:04.551 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.550+0000 7f8db3e00640 1 --2- 172.21.15.90:0/3327752529 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8db4103f40 0x7f8db419a5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:04.551 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.550+0000 7f8db3e00640 1 -- 172.21.15.90:0/3327752529 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8db419ee60 msgr2=0x7f8db419f310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:04.551 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.550+0000 7f8db3e00640 1 --2- 172.21.15.90:0/3327752529 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8db419ee60 0x7f8db419f310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:04.551 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.550+0000 7f8db3e00640 1 -- 172.21.15.90:0/3327752529 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8db419f950 con 0x7f8db4109010 2024-04-02T14:42:04.551 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.550+0000 7f8db3e00640 1 --2- 172.21.15.90:0/3327752529 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8db4109010 0x7f8db419ab30 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f8da002c240 tx=0x7f8da0004360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:42:04.551 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.550+0000 7f8db1600640 1 -- 172.21.15.90:0/3327752529 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8da0043020 con 0x7f8db4109010 2024-04-02T14:42:04.551 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.550+0000 7f8db1600640 1 -- 172.21.15.90:0/3327752529 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8da00054e0 con 0x7f8db4109010 2024-04-02T14:42:04.551 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.550+0000 7f8dbbc00640 1 -- 172.21.15.90:0/3327752529 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8db419fbe0 con 0x7f8db4109010 2024-04-02T14:42:04.552 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.550+0000 7f8db1600640 1 -- 172.21.15.90:0/3327752529 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8da0003bb0 con 0x7f8db4109010 2024-04-02T14:42:04.552 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.550+0000 7f8dbbc00640 1 -- 172.21.15.90:0/3327752529 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8db41a7400 con 0x7f8db4109010 2024-04-02T14:42:04.555 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.555+0000 7f8db1600640 1 -- 172.21.15.90:0/3327752529 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f8da0046020 con 0x7f8db4109010 2024-04-02T14:42:04.555 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.555+0000 7f8dbe995740 1 Processor -- start 2024-04-02T14:42:04.555 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.555+0000 7f8dbe995740 1 -- start start 2024-04-02T14:42:04.555 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.555+0000 7f8dbe995740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a57ed2aeb0 0x55a57ed2b290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:04.555 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.555+0000 7f8dbe995740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a57ed2b7d0 0x55a57ed33e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:04.555 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.555+0000 7f8dbe995740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a57ed343b0 0x55a57ed367a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:04.555 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.555+0000 7f8dbe995740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55a57ec8fd70 con 0x55a57ed2b7d0 2024-04-02T14:42:04.555 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.555+0000 7f8dbe995740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55a57ec382a0 con 0x55a57ed343b0 2024-04-02T14:42:04.555 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.555+0000 7f8dbe995740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55a57ec8bd20 con 0x55a57ed2aeb0 2024-04-02T14:42:04.557 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.556+0000 7f8db9c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a57ed2aeb0 0x55a57ed2b290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:04.557 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.556+0000 7f8db9c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a57ed2aeb0 0x55a57ed2b290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:59186/0 (socket says 172.21.15.90:59186) 2024-04-02T14:42:04.557 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.556+0000 7f8db9c00640 1 -- 172.21.15.90:0/738449182 learned_addr learned my addr 172.21.15.90:0/738449182 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:42:04.557 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.556+0000 7f8db3e00640 1 --2- 172.21.15.90:0/738449182 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a57ed343b0 0x55a57ed367a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:04.558 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.556+0000 7f8db9200640 1 --2- 172.21.15.90:0/738449182 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a57ed2b7d0 0x55a57ed33e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:04.558 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.556+0000 7f8db9c00640 1 -- 172.21.15.90:0/738449182 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a57ed343b0 msgr2=0x55a57ed367a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:42:04.558 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.556+0000 7f8db9c00640 1 --2- 172.21.15.90:0/738449182 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a57ed343b0 0x55a57ed367a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:04.558 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.556+0000 7f8db9c00640 1 -- 172.21.15.90:0/738449182 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a57ed2b7d0 msgr2=0x55a57ed33e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:42:04.558 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.556+0000 7f8db9c00640 1 --2- 172.21.15.90:0/738449182 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a57ed2b7d0 0x55a57ed33e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:04.558 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.556+0000 7f8db9c00640 1 -- 172.21.15.90:0/738449182 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a57ec4c840 con 0x55a57ed2aeb0 2024-04-02T14:42:04.558 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.556+0000 7f8db3e00640 1 --2- 172.21.15.90:0/738449182 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a57ed343b0 0x55a57ed367a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:42:04.558 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.556+0000 7f8db1600640 1 --2- 172.21.15.90:0/3327752529 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f8d9c0a3dc0 0x7f8d9c0a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:04.558 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.556+0000 7f8db9c00640 1 --2- 172.21.15.90:0/738449182 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a57ed2aeb0 0x55a57ed2b290 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f8dac009f50 tx=0x7f8dac0118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=40e577ceaea4965b server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:42:04.558 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.557+0000 7f8db1600640 1 -- 172.21.15.90:0/3327752529 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8594+0+0 (secure 0 0 0) 0x7f8da00f2730 con 0x7f8db4109010 2024-04-02T14:42:04.558 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.557+0000 7f8da6000640 1 -- 172.21.15.90:0/738449182 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8dac004b20 con 0x55a57ed2aeb0 2024-04-02T14:42:04.558 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.557+0000 7f8da6000640 1 -- 172.21.15.90:0/738449182 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8dac004cc0 con 0x55a57ed2aeb0 2024-04-02T14:42:04.558 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.557+0000 7f8da6000640 1 -- 172.21.15.90:0/738449182 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8dac005020 con 0x55a57ed2aeb0 2024-04-02T14:42:04.559 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.557+0000 7f8db9200640 1 --2- 172.21.15.90:0/3327752529 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f8d9c0a3dc0 0x7f8d9c0a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:04.559 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.558+0000 7f8dbe995740 1 -- 172.21.15.90:0/738449182 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a57ed2aeb0 msgr2=0x55a57ed2b290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:04.559 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.558+0000 7f8db9200640 1 --2- 172.21.15.90:0/3327752529 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f8d9c0a3dc0 0x7f8d9c0a6280 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x55a57ed2a980 tx=0x7f8da8005f90 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:42:04.559 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.558+0000 7f8dbe995740 1 --2- 172.21.15.90:0/738449182 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a57ed2aeb0 0x55a57ed2b290 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f8dac009f50 tx=0x7f8dac0118d0 comp rx=0 tx=0).stop 2024-04-02T14:42:04.559 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.558+0000 7f8dbe995740 1 -- 172.21.15.90:0/738449182 shutdown_connections 2024-04-02T14:42:04.559 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.558+0000 7f8dbe995740 1 --2- 172.21.15.90:0/738449182 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a57ed343b0 0x55a57ed367a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:04.559 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.558+0000 7f8dbe995740 1 --2- 172.21.15.90:0/738449182 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a57ed2b7d0 0x55a57ed33e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:04.559 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.558+0000 7f8dbe995740 1 --2- 172.21.15.90:0/738449182 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a57ed2aeb0 0x55a57ed2b290 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:04.559 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.558+0000 7f8dbe995740 1 -- 172.21.15.90:0/738449182 >> 172.21.15.90:0/738449182 conn(0x55a57ed255e0 msgr2=0x55a57ed259c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:42:04.559 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.558+0000 7f8dbe995740 1 -- 172.21.15.90:0/738449182 shutdown_connections 2024-04-02T14:42:04.559 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.558+0000 7f8dbe995740 1 -- 172.21.15.90:0/738449182 wait complete. 2024-04-02T14:42:04.562 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.559+0000 7f8dbe995740 1 -- start start 2024-04-02T14:42:04.562 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.559+0000 7f8dbe995740 1 Processor -- start 2024-04-02T14:42:04.562 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.559+0000 7f8dbe995740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a57ed2aeb0 0x55a57ed31470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:04.563 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.559+0000 7f8dbe995740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a57ed2b7d0 0x55a57ed319b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:04.563 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.559+0000 7f8dbe995740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a57ed343b0 0x55a57ed31ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:04.563 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.559+0000 7f8dbe995740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55a57ed371a0 con 0x55a57ed2b7d0 2024-04-02T14:42:04.563 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.559+0000 7f8dbe995740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55a57ec4b660 con 0x55a57ed2aeb0 2024-04-02T14:42:04.563 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.559+0000 7f8dbe995740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55a57ed37320 con 0x55a57ed343b0 2024-04-02T14:42:04.563 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.559+0000 7f8dbe995740 10 client.0 _ll_register_callbacks cb 0x55a57ec4b050 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:42:04.563 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.559+0000 7f8db9200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a57ed2aeb0 0x55a57ed31470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:04.563 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.559+0000 7f8db3e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a57ed2b7d0 0x55a57ed319b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:04.563 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.560+0000 7f8db9c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a57ed343b0 0x55a57ed31ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:04.563 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.560+0000 7f8db9200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a57ed2aeb0 0x55a57ed31470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:50114/0 (socket says 172.21.15.90:50114) 2024-04-02T14:42:04.563 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.560+0000 7f8db9200640 1 -- 172.21.15.90:0/3818462561 learned_addr learned my addr 172.21.15.90:0/3818462561 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:42:04.563 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.560+0000 7f8db9c00640 1 -- 172.21.15.90:0/3818462561 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a57ed2aeb0 msgr2=0x55a57ed31470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:42:04.564 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.560+0000 7f8db9c00640 1 --2- 172.21.15.90:0/3818462561 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a57ed2aeb0 0x55a57ed31470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:04.564 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.560+0000 7f8db9c00640 1 -- 172.21.15.90:0/3818462561 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a57ed2b7d0 msgr2=0x55a57ed319b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:42:04.564 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.560+0000 7f8db9c00640 1 --2- 172.21.15.90:0/3818462561 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a57ed2b7d0 0x55a57ed319b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:04.564 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.560+0000 7f8db9c00640 1 -- 172.21.15.90:0/3818462561 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a57ec5ae80 con 0x55a57ed343b0 2024-04-02T14:42:04.564 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.560+0000 7f8db9200640 1 --2- 172.21.15.90:0/3818462561 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a57ed2aeb0 0x55a57ed31470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:42:04.564 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.560+0000 7f8db9c00640 1 --2- 172.21.15.90:0/3818462561 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a57ed343b0 0x55a57ed31ef0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f8dac01b040 tx=0x7f8dac004300 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c665d3c0f4ce54c9 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:42:04.564 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.560+0000 7f8d9aa00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:42:04.564 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.561+0000 7f8d9aa00640 1 -- 172.21.15.90:0/3818462561 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8dac028070 con 0x55a57ed343b0 2024-04-02T14:42:04.564 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.561+0000 7f8d9aa00640 1 -- 172.21.15.90:0/3818462561 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8594+0+0 (secure 0 0 0) 0x7f8dac003a30 con 0x55a57ed343b0 2024-04-02T14:42:04.564 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.561+0000 7f8dbe995740 1 -- 172.21.15.90:0/3818462561 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8dac004cc0 con 0x55a57ed343b0 2024-04-02T14:42:04.564 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.561+0000 7f8d9aa00640 1 -- 172.21.15.90:0/3818462561 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f8dac003c30 con 0x55a57ed343b0 2024-04-02T14:42:04.564 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.561+0000 7f8dbe995740 10 client.27511 fetch_fsmap learned FSMap version 27 2024-04-02T14:42:04.564 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.561+0000 7f8dbe995740 1 -- 172.21.15.90:0/3818462561 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55a57e9a4210 con 0x55a57ed343b0 2024-04-02T14:42:04.564 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.561+0000 7f8d9aa00640 1 -- 172.21.15.90:0/3818462561 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8dac004cc0 con 0x55a57ed343b0 2024-04-02T14:42:04.565 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.561+0000 7f8d9aa00640 1 -- 172.21.15.90:0/3818462561 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f8dac024440 con 0x55a57ed343b0 2024-04-02T14:42:04.565 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.562+0000 7f8dbe995740 10 client.27511 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:42:04.565 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.562+0000 7f8dbe995740 10 client.27511 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-02T14:42:04.565 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.562+0000 7f8dbe995740 20 client.27511 populate_metadata read hostname 'smithi090' 2024-04-02T14:42:04.565 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.562+0000 7f8dbe995740 1 --2- 172.21.15.90:0/3818462561 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55a57ed44770 0x55a57ed46b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:04.565 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.562+0000 7f8dbe995740 4 client.27511 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:42:04.565 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.562+0000 7f8dbe995740 1 -- 172.21.15.90:0/3818462561 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a57ec3fcd0 con 0x55a57ed44770 2024-04-02T14:42:04.565 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.562+0000 7f8db9200640 1 --2- 172.21.15.90:0/3818462561 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55a57ed44770 0x55a57ed46b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:04.565 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.563+0000 7f8db9200640 1 --2- 172.21.15.90:0/3818462561 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55a57ed44770 0x55a57ed46b60 crc :-1 s=READY pgs=200 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bd1ac2915b4f81b9 server_cookie=a8cb3310c5e0ee99 in_seq=0 out_seq=0 2024-04-02T14:42:04.565 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.563+0000 7f8d9aa00640 10 client.27511 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:42:04.565 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.563+0000 7f8d9aa00640 1 -- 172.21.15.90:0/3818462561 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a57ec3fcd0 con 0x55a57ed44770 2024-04-02T14:42:04.565 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.563+0000 7f8d9aa00640 10 client.27511 handle_command_reply: tid=0 2024-04-02T14:42:04.565 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.563+0000 7f8dbe995740 1 client.27511 shutdown 2024-04-02T14:42:04.566 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.563+0000 7f8dbe995740 1 -- 172.21.15.90:0/3818462561 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a57ed343b0 msgr2=0x55a57ed31ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:04.567 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.563+0000 7f8dbe995740 1 --2- 172.21.15.90:0/3818462561 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a57ed343b0 0x55a57ed31ef0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f8dac01b040 tx=0x7f8dac004300 comp rx=0 tx=0).stop 2024-04-02T14:42:04.567 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.564+0000 7f8dbe995740 1 -- 172.21.15.90:0/3818462561 shutdown_connections 2024-04-02T14:42:04.567 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.564+0000 7f8dbe995740 1 --2- 172.21.15.90:0/3818462561 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55a57ed44770 0x55a57ed46b60 crc :-1 s=READY pgs=200 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:04.567 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.564+0000 7f8dbe995740 1 --2- 172.21.15.90:0/3818462561 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a57ed343b0 0x55a57ed31ef0 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:04.567 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.564+0000 7f8dbe995740 1 --2- 172.21.15.90:0/3818462561 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a57ed2b7d0 0x55a57ed319b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:04.567 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.564+0000 7f8dbe995740 1 --2- 172.21.15.90:0/3818462561 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a57ed2aeb0 0x55a57ed31470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:04.567 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.564+0000 7f8d9aa00640 0 client.27511 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:42:04.567 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.564+0000 7f8dbe995740 1 -- 172.21.15.90:0/3818462561 >> 172.21.15.90:0/3818462561 conn(0x55a57ed255e0 msgr2=0x55a57ed27330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:42:04.567 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.564+0000 7f8dbe995740 1 -- 172.21.15.90:0/3818462561 shutdown_connections 2024-04-02T14:42:04.567 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.564+0000 7f8dbe995740 1 -- 172.21.15.90:0/3818462561 wait complete. 2024-04-02T14:42:04.567 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.564+0000 7f8dbe995740 20 client.27511 trim_cache size 0 max 16384 2024-04-02T14:42:04.588 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.588+0000 7f8dbe995740 1 Processor -- start 2024-04-02T14:42:04.588 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.588+0000 7f8dbe995740 1 -- start start 2024-04-02T14:42:04.588 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.588+0000 7f8dbe995740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a57ed2aeb0 0x55a57ed40060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:04.589 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.588+0000 7f8dbe995740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a57ed2b7d0 0x55a57ed31d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:04.589 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.588+0000 7f8dbe995740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a57ed343b0 0x55a57ed4c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:04.589 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.588+0000 7f8dbe995740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f8db41afa40 con 0x55a57ed343b0 2024-04-02T14:42:04.589 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.588+0000 7f8dbe995740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f8db41b1580 con 0x55a57ed2b7d0 2024-04-02T14:42:04.589 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.588+0000 7f8dbe995740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f8db41b0110 con 0x55a57ed2aeb0 2024-04-02T14:42:04.589 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.589+0000 7f8db3e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a57ed2b7d0 0x55a57ed31d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:04.589 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.589+0000 7f8db3e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a57ed2b7d0 0x55a57ed31d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:50122/0 (socket says 172.21.15.90:50122) 2024-04-02T14:42:04.589 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.589+0000 7f8db3e00640 1 -- 172.21.15.90:0/3493554007 learned_addr learned my addr 172.21.15.90:0/3493554007 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:42:04.589 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.589+0000 7f8db9200640 1 --2- 172.21.15.90:0/3493554007 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a57ed2aeb0 0x55a57ed40060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:04.589 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.589+0000 7f8db9c00640 1 --2- 172.21.15.90:0/3493554007 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a57ed343b0 0x55a57ed4c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:04.589 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.589+0000 7f8db9200640 1 -- 172.21.15.90:0/3493554007 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a57ed2b7d0 msgr2=0x55a57ed31d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:42:04.589 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.589+0000 7f8db9200640 1 --2- 172.21.15.90:0/3493554007 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a57ed2b7d0 0x55a57ed31d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:04.589 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.589+0000 7f8db9200640 1 -- 172.21.15.90:0/3493554007 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a57ed343b0 msgr2=0x55a57ed4c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:42:04.589 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.589+0000 7f8db9200640 1 --2- 172.21.15.90:0/3493554007 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a57ed343b0 0x55a57ed4c2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:04.590 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.589+0000 7f8db9200640 1 -- 172.21.15.90:0/3493554007 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a57ea70620 con 0x55a57ed2aeb0 2024-04-02T14:42:04.590 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.589+0000 7f8db9c00640 1 --2- 172.21.15.90:0/3493554007 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a57ed343b0 0x55a57ed4c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:42:04.592 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.589+0000 7f8db9200640 1 --2- 172.21.15.90:0/3493554007 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a57ed2aeb0 0x55a57ed40060 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x55a57ed30cd0 tx=0x7f8da801b000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=19844ed69496e954 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:42:04.592 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.590+0000 7f8da5600640 1 -- 172.21.15.90:0/3493554007 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8da8003e10 con 0x55a57ed2aeb0 2024-04-02T14:42:04.592 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.590+0000 7f8da5600640 1 -- 172.21.15.90:0/3493554007 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8da802b900 con 0x55a57ed2aeb0 2024-04-02T14:42:04.593 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.590+0000 7f8da5600640 1 -- 172.21.15.90:0/3493554007 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8da801f970 con 0x55a57ed2aeb0 2024-04-02T14:42:04.593 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.590+0000 7f8dbe995740 1 -- 172.21.15.90:0/3493554007 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a57ed2aeb0 msgr2=0x55a57ed40060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:04.593 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.590+0000 7f8dbe995740 1 --2- 172.21.15.90:0/3493554007 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a57ed2aeb0 0x55a57ed40060 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x55a57ed30cd0 tx=0x7f8da801b000 comp rx=0 tx=0).stop 2024-04-02T14:42:04.593 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.591+0000 7f8dbe995740 1 -- 172.21.15.90:0/3493554007 shutdown_connections 2024-04-02T14:42:04.593 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.591+0000 7f8dbe995740 1 --2- 172.21.15.90:0/3493554007 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a57ed343b0 0x55a57ed4c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:04.593 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.591+0000 7f8dbe995740 1 --2- 172.21.15.90:0/3493554007 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a57ed2b7d0 0x55a57ed31d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:04.593 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.591+0000 7f8dbe995740 1 --2- 172.21.15.90:0/3493554007 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a57ed2aeb0 0x55a57ed40060 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:04.593 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.591+0000 7f8dbe995740 1 -- 172.21.15.90:0/3493554007 >> 172.21.15.90:0/3493554007 conn(0x55a57ed255e0 msgr2=0x55a57ed27230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:42:04.593 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.591+0000 7f8dbe995740 1 -- 172.21.15.90:0/3493554007 shutdown_connections 2024-04-02T14:42:04.593 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.591+0000 7f8dbe995740 1 -- 172.21.15.90:0/3493554007 wait complete. 2024-04-02T14:42:04.593 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.591+0000 7f8dbe995740 1 -- start start 2024-04-02T14:42:04.593 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.591+0000 7f8dbe995740 1 Processor -- start 2024-04-02T14:42:04.593 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.592+0000 7f8dbe995740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a57ed2aeb0 0x55a57ed3c760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:04.593 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.592+0000 7f8dbe995740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a57ed2b7d0 0x55a57ed2dce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:04.594 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.592+0000 7f8dbe995740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a57ed343b0 0x55a57ed2e220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:04.594 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.592+0000 7f8dbe995740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f8db41b0eb0 con 0x55a57ed2aeb0 2024-04-02T14:42:04.594 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.592+0000 7f8dbe995740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f8db41b07e0 con 0x55a57ed343b0 2024-04-02T14:42:04.594 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.592+0000 7f8dbe995740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f8db41ada40 con 0x55a57ed2b7d0 2024-04-02T14:42:04.594 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.592+0000 7f8dbe995740 10 client.0 _ll_register_callbacks cb 0x7f8dac009e20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:42:04.594 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.592+0000 7f8db3e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a57ed2b7d0 0x55a57ed2dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:04.594 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.592+0000 7f8db9200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a57ed2aeb0 0x55a57ed3c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:04.594 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.592+0000 7f8db3e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a57ed2b7d0 0x55a57ed2dce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:59224/0 (socket says 172.21.15.90:59224) 2024-04-02T14:42:04.594 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.592+0000 7f8db9c00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a57ed343b0 0x55a57ed2e220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:04.594 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.592+0000 7f8db3e00640 1 -- 172.21.15.90:0/3765238547 learned_addr learned my addr 172.21.15.90:0/3765238547 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:42:04.594 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.593+0000 7f8db3e00640 1 -- 172.21.15.90:0/3765238547 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a57ed343b0 msgr2=0x55a57ed2e220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:42:04.594 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.593+0000 7f8db3e00640 1 --2- 172.21.15.90:0/3765238547 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a57ed343b0 0x55a57ed2e220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:04.594 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.593+0000 7f8db3e00640 1 -- 172.21.15.90:0/3765238547 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a57ed2aeb0 msgr2=0x55a57ed3c760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:42:04.594 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.593+0000 7f8db3e00640 1 --2- 172.21.15.90:0/3765238547 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a57ed2aeb0 0x55a57ed3c760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:04.595 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.593+0000 7f8db3e00640 1 -- 172.21.15.90:0/3765238547 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a57ed2e860 con 0x55a57ed2b7d0 2024-04-02T14:42:04.595 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.593+0000 7f8db9200640 1 --2- 172.21.15.90:0/3765238547 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a57ed2aeb0 0x55a57ed3c760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:42:04.595 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.593+0000 7f8db9c00640 1 --2- 172.21.15.90:0/3765238547 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a57ed343b0 0x55a57ed2e220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:42:04.595 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.593+0000 7f8db3e00640 1 --2- 172.21.15.90:0/3765238547 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a57ed2b7d0 0x55a57ed2dce0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f8da00efff0 tx=0x7f8da002bc30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=57bb0d0493bc7163 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:42:04.595 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.593+0000 7f8d9aa00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:42:04.598 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.593+0000 7f8d9aa00640 1 -- 172.21.15.90:0/3765238547 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8da00be560 con 0x55a57ed2b7d0 2024-04-02T14:42:04.598 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.593+0000 7f8d9aa00640 1 -- 172.21.15.90:0/3765238547 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8594+0+0 (secure 0 0 0) 0x7f8da00bae00 con 0x55a57ed2b7d0 2024-04-02T14:42:04.598 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.593+0000 7f8dbe995740 1 -- 172.21.15.90:0/3765238547 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8da802b900 con 0x55a57ed2b7d0 2024-04-02T14:42:04.598 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.594+0000 7f8d9aa00640 1 -- 172.21.15.90:0/3765238547 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f8da00be0f0 con 0x55a57ed2b7d0 2024-04-02T14:42:04.598 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.594+0000 7f8dbe995740 10 client.27517 fetch_fsmap learned FSMap version 27 2024-04-02T14:42:04.599 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.594+0000 7f8dbe995740 1 -- 172.21.15.90:0/3765238547 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55a57edaa990 con 0x55a57ed2b7d0 2024-04-02T14:42:04.599 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.594+0000 7f8d9aa00640 1 -- 172.21.15.90:0/3765238547 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8da802b900 con 0x55a57ed2b7d0 2024-04-02T14:42:04.599 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.594+0000 7f8d9aa00640 1 -- 172.21.15.90:0/3765238547 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f8da002cb90 con 0x55a57ed2b7d0 2024-04-02T14:42:04.599 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.594+0000 7f8dbe995740 10 client.27517 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:42:04.599 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.594+0000 7f8dbe995740 10 client.27517 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-02T14:42:04.599 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.594+0000 7f8dbe995740 20 client.27517 populate_metadata read hostname 'smithi090' 2024-04-02T14:42:04.600 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.594+0000 7f8dbe995740 1 --2- 172.21.15.90:0/3765238547 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55a57edab1e0 0x55a57edad600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:04.600 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.594+0000 7f8dbe995740 4 client.27517 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-02T14:42:04.600 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.594+0000 7f8dbe995740 1 -- 172.21.15.90:0/3765238547 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55a57ec9d2b0 con 0x55a57edab1e0 2024-04-02T14:42:04.600 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.595+0000 7f8db9200640 1 --2- 172.21.15.90:0/3765238547 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55a57edab1e0 0x55a57edad600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:04.600 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.595+0000 7f8db9200640 1 --2- 172.21.15.90:0/3765238547 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55a57edab1e0 0x55a57edad600 crc :-1 s=READY pgs=201 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3d20fd060d043962 server_cookie=dc22f225840973d in_seq=0 out_seq=0 2024-04-02T14:42:04.600 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.595+0000 7f8d9aa00640 10 client.27517 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:42:04.615 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.615+0000 7f8d9aa00640 1 -- 172.21.15.90:0/3765238547 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55a57ec9d2b0 con 0x55a57edab1e0 2024-04-02T14:42:04.616 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.615+0000 7f8d9aa00640 10 client.27517 handle_command_reply: tid=0 2024-04-02T14:42:04.616 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.615+0000 7f8dbe995740 1 client.27517 shutdown 2024-04-02T14:42:04.616 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.616+0000 7f8dbe995740 1 -- 172.21.15.90:0/3765238547 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a57ed2b7d0 msgr2=0x55a57ed2dce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:04.616 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.616+0000 7f8dbe995740 1 --2- 172.21.15.90:0/3765238547 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a57ed2b7d0 0x55a57ed2dce0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f8da00efff0 tx=0x7f8da002bc30 comp rx=0 tx=0).stop 2024-04-02T14:42:04.616 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.616+0000 7f8dbe995740 1 -- 172.21.15.90:0/3765238547 shutdown_connections 2024-04-02T14:42:04.616 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.616+0000 7f8dbe995740 1 --2- 172.21.15.90:0/3765238547 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55a57edab1e0 0x55a57edad600 crc :-1 s=READY pgs=201 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:04.616 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.616+0000 7f8dbe995740 1 --2- 172.21.15.90:0/3765238547 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a57ed343b0 0x55a57ed2e220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:04.616 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.616+0000 7f8dbe995740 1 --2- 172.21.15.90:0/3765238547 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a57ed2b7d0 0x55a57ed2dce0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:04.616 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.616+0000 7f8dbe995740 1 --2- 172.21.15.90:0/3765238547 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a57ed2aeb0 0x55a57ed3c760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:04.616 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.616+0000 7f8d9aa00640 0 client.27517 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:42:04.616 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.616+0000 7f8dbe995740 1 -- 172.21.15.90:0/3765238547 >> 172.21.15.90:0/3765238547 conn(0x55a57ed255e0 msgr2=0x55a57ed23db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:42:04.616 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.616+0000 7f8dbe995740 1 -- 172.21.15.90:0/3765238547 shutdown_connections 2024-04-02T14:42:04.616 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.616+0000 7f8dbe995740 1 -- 172.21.15.90:0/3765238547 wait complete. 2024-04-02T14:42:04.617 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.616+0000 7f8dbe995740 20 client.27517 trim_cache size 0 max 16384 2024-04-02T14:42:04.617 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:{ 2024-04-02T14:42:04.617 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "message": "", 2024-04-02T14:42:04.617 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "return_code": 0 2024-04-02T14:42:04.617 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:} 2024-04-02T14:42:04.618 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.617+0000 7f8dbbc00640 1 -- 172.21.15.90:0/3327752529 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f8d9c0a3dc0 msgr2=0x7f8d9c0a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:04.618 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.617+0000 7f8dbbc00640 1 --2- 172.21.15.90:0/3327752529 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f8d9c0a3dc0 0x7f8d9c0a6280 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x55a57ed2a980 tx=0x7f8da8005f90 comp rx=0 tx=0).stop 2024-04-02T14:42:04.618 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.617+0000 7f8dbbc00640 1 -- 172.21.15.90:0/3327752529 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8db4109010 msgr2=0x7f8db419ab30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:04.618 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.617+0000 7f8dbbc00640 1 --2- 172.21.15.90:0/3327752529 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8db4109010 0x7f8db419ab30 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f8da002c240 tx=0x7f8da0004360 comp rx=0 tx=0).stop 2024-04-02T14:42:04.618 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.617+0000 7f8dbbc00640 1 -- 172.21.15.90:0/3327752529 shutdown_connections 2024-04-02T14:42:04.618 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.617+0000 7f8dbbc00640 1 --2- 172.21.15.90:0/3327752529 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f8d9c0a3dc0 0x7f8d9c0a6280 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:04.618 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.617+0000 7f8dbbc00640 1 --2- 172.21.15.90:0/3327752529 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8db419ee60 0x7f8db419f310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:04.618 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.617+0000 7f8dbbc00640 1 --2- 172.21.15.90:0/3327752529 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8db4109010 0x7f8db419ab30 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:04.618 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.617+0000 7f8dbbc00640 1 --2- 172.21.15.90:0/3327752529 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8db4103f40 0x7f8db419a5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:04.618 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.617+0000 7f8dbbc00640 1 -- 172.21.15.90:0/3327752529 >> 172.21.15.90:0/3327752529 conn(0x7f8db40ff7c0 msgr2=0x7f8db4101a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:42:04.618 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.617+0000 7f8dbbc00640 1 -- 172.21.15.90:0/3327752529 shutdown_connections 2024-04-02T14:42:04.619 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:04.618+0000 7f8dbbc00640 1 -- 172.21.15.90:0/3327752529 wait complete. 2024-04-02T14:42:06.357 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:42:06 smithi092 ceph-mon[36497]: pgmap v220: 97 pgs: 15 active+clean+snaptrim, 52 active+clean, 30 active+clean+snaptrim_wait; 3.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 144 KiB/s rd, 6.1 MiB/s wr, 734 op/s 2024-04-02T14:42:06.433 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:42:06 smithi177 ceph-mon[36431]: pgmap v220: 97 pgs: 15 active+clean+snaptrim, 52 active+clean, 30 active+clean+snaptrim_wait; 3.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 144 KiB/s rd, 6.1 MiB/s wr, 734 op/s 2024-04-02T14:42:06.438 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:42:06 smithi090 ceph-mon[31674]: pgmap v220: 97 pgs: 15 active+clean+snaptrim, 52 active+clean, 30 active+clean+snaptrim_wait; 3.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 144 KiB/s rd, 6.1 MiB/s wr, 734 op/s 2024-04-02T14:42:06.829 INFO:teuthology.orchestra.run.smithi090.stderr:progress: 395.0 s, 71.2 tps, lat 1308.052 ms stddev 1060.621 2024-04-02T14:42:07.358 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:42:07 smithi092 ceph-mon[36497]: scrub summary: idle 2024-04-02T14:42:07.433 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:42:07 smithi177 ceph-mon[36431]: scrub summary: idle 2024-04-02T14:42:07.435 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:42:07 smithi090 ceph-mon[31674]: scrub summary: idle 2024-04-02T14:42:08.358 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:42:08 smithi092 ceph-mon[36497]: pgmap v221: 97 pgs: 15 active+clean+snaptrim, 52 active+clean, 30 active+clean+snaptrim_wait; 3.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 144 KiB/s rd, 6.1 MiB/s wr, 734 op/s 2024-04-02T14:42:08.433 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:42:08 smithi177 ceph-mon[36431]: pgmap v221: 97 pgs: 15 active+clean+snaptrim, 52 active+clean, 30 active+clean+snaptrim_wait; 3.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 144 KiB/s rd, 6.1 MiB/s wr, 734 op/s 2024-04-02T14:42:08.435 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:42:08 smithi090 ceph-mon[31674]: pgmap v221: 97 pgs: 15 active+clean+snaptrim, 52 active+clean, 30 active+clean+snaptrim_wait; 3.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 144 KiB/s rd, 6.1 MiB/s wr, 734 op/s 2024-04-02T14:42:10.358 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:42:10 smithi092 ceph-mon[36497]: pgmap v222: 97 pgs: 6 active+clean+snaptrim, 74 active+clean, 17 active+clean+snaptrim_wait; 3.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 3.5 MiB/s wr, 534 op/s 2024-04-02T14:42:10.433 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:42:10 smithi177 ceph-mon[36431]: pgmap v222: 97 pgs: 6 active+clean+snaptrim, 74 active+clean, 17 active+clean+snaptrim_wait; 3.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 3.5 MiB/s wr, 534 op/s 2024-04-02T14:42:10.435 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:42:10 smithi090 ceph-mon[31674]: pgmap v222: 97 pgs: 6 active+clean+snaptrim, 74 active+clean, 17 active+clean+snaptrim_wait; 3.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 3.5 MiB/s wr, 534 op/s 2024-04-02T14:42:11.829 INFO:teuthology.orchestra.run.smithi090.stderr:progress: 400.0 s, 130.6 tps, lat 796.260 ms stddev 490.119 2024-04-02T14:42:12.358 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:42:12 smithi092 ceph-mon[36497]: pgmap v223: 97 pgs: 97 active+clean; 3.8 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 2.7 MiB/s wr, 444 op/s 2024-04-02T14:42:12.358 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:42:12 smithi092 ceph-mon[36497]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T14:42:12.434 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:42:12 smithi177 ceph-mon[36431]: pgmap v223: 97 pgs: 97 active+clean; 3.8 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 2.7 MiB/s wr, 444 op/s 2024-04-02T14:42:12.434 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:42:12 smithi177 ceph-mon[36431]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T14:42:12.436 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:42:12 smithi090 ceph-mon[31674]: pgmap v223: 97 pgs: 97 active+clean; 3.8 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 2.7 MiB/s wr, 444 op/s 2024-04-02T14:42:12.436 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:42:12 smithi090 ceph-mon[31674]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T14:42:13.358 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:42:13 smithi092 ceph-mon[36497]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:42:13.358 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:42:13 smithi092 ceph-mon[36497]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T14:42:13.358 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:42:13 smithi092 ceph-mon[36497]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' 2024-04-02T14:42:13.433 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:42:13 smithi177 ceph-mon[36431]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:42:13.434 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:42:13 smithi177 ceph-mon[36431]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T14:42:13.434 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:42:13 smithi177 ceph-mon[36431]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' 2024-04-02T14:42:13.436 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:42:13 smithi090 ceph-mon[31674]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:42:13.436 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:42:13 smithi090 ceph-mon[31674]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T14:42:13.436 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:42:13 smithi090 ceph-mon[31674]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' 2024-04-02T14:42:14.358 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:42:14 smithi092 ceph-mon[36497]: pgmap v224: 97 pgs: 97 active+clean; 3.8 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 2.5 MiB/s wr, 425 op/s 2024-04-02T14:42:14.434 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:42:14 smithi177 ceph-mon[36431]: pgmap v224: 97 pgs: 97 active+clean; 3.8 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 2.5 MiB/s wr, 425 op/s 2024-04-02T14:42:14.436 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:42:14 smithi090 ceph-mon[31674]: pgmap v224: 97 pgs: 97 active+clean; 3.8 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 2.5 MiB/s wr, 425 op/s 2024-04-02T14:42:16.357 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:42:16 smithi092 ceph-mon[36497]: pgmap v225: 97 pgs: 97 active+clean; 3.8 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 3.0 MiB/s wr, 497 op/s 2024-04-02T14:42:16.358 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:42:16 smithi092 ceph-mon[36497]: osdmap e93: 12 total, 12 up, 12 in 2024-04-02T14:42:16.433 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:42:16 smithi177 ceph-mon[36431]: pgmap v225: 97 pgs: 97 active+clean; 3.8 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 3.0 MiB/s wr, 497 op/s 2024-04-02T14:42:16.433 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:42:16 smithi177 ceph-mon[36431]: osdmap e93: 12 total, 12 up, 12 in 2024-04-02T14:42:16.435 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:42:16 smithi090 ceph-mon[31674]: pgmap v225: 97 pgs: 97 active+clean; 3.8 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 3.0 MiB/s wr, 497 op/s 2024-04-02T14:42:16.435 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:42:16 smithi090 ceph-mon[31674]: osdmap e93: 12 total, 12 up, 12 in 2024-04-02T14:42:16.829 INFO:teuthology.orchestra.run.smithi090.stderr:progress: 405.0 s, 140.4 tps, lat 699.195 ms stddev 444.260 2024-04-02T14:42:17.763 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.764+0000 7f4051a00640 1 -- 172.21.15.90:0/2294204045 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f404c103f20 msgr2=0x7f404c104300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:17.763 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.764+0000 7f4051a00640 1 --2- 172.21.15.90:0/2294204045 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f404c103f20 0x7f404c104300 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f403c009ca0 tx=0x7f403c02c100 comp rx=0 tx=0).stop 2024-04-02T14:42:17.763 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.764+0000 7f4051a00640 1 -- 172.21.15.90:0/2294204045 shutdown_connections 2024-04-02T14:42:17.764 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.764+0000 7f4051a00640 1 --2- 172.21.15.90:0/2294204045 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f404c108ff0 0x7f404c110bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:17.764 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.764+0000 7f4051a00640 1 --2- 172.21.15.90:0/2294204045 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f404c1048d0 0x7f404c108910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:17.764 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.764+0000 7f4051a00640 1 --2- 172.21.15.90:0/2294204045 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f404c103f20 0x7f404c104300 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:17.764 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.764+0000 7f4051a00640 1 -- 172.21.15.90:0/2294204045 >> 172.21.15.90:0/2294204045 conn(0x7f404c0ff7c0 msgr2=0x7f404c101be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:42:17.764 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.764+0000 7f4051a00640 1 -- 172.21.15.90:0/2294204045 shutdown_connections 2024-04-02T14:42:17.764 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.764+0000 7f4051a00640 1 -- 172.21.15.90:0/2294204045 wait complete. 2024-04-02T14:42:17.765 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.765+0000 7f4051a00640 1 Processor -- start 2024-04-02T14:42:17.765 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.765+0000 7f4051a00640 1 -- start start 2024-04-02T14:42:17.765 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.765+0000 7f4051a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f404c103f20 0x7f404c19a520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:17.765 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.765+0000 7f4051a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f404c108ff0 0x7f404c19aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:17.765 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.765+0000 7f4051a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f404c19ede0 0x7f404c19f290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:17.766 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.765+0000 7f4051a00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f404c1140a0 con 0x7f404c19ede0 2024-04-02T14:42:17.766 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.765+0000 7f4051a00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f404c113f20 con 0x7f404c108ff0 2024-04-02T14:42:17.766 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.765+0000 7f4051a00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f404c114220 con 0x7f404c103f20 2024-04-02T14:42:17.766 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.766+0000 7f4042000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f404c108ff0 0x7f404c19aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:17.766 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.766+0000 7f404aa00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f404c103f20 0x7f404c19a520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:17.766 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.766+0000 7f4042000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f404c108ff0 0x7f404c19aa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:34926/0 (socket says 172.21.15.90:34926) 2024-04-02T14:42:17.766 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.766+0000 7f4042000640 1 -- 172.21.15.90:0/2345401906 learned_addr learned my addr 172.21.15.90:0/2345401906 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:42:17.766 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.766+0000 7f404b400640 1 --2- 172.21.15.90:0/2345401906 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f404c19ede0 0x7f404c19f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:17.766 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.766+0000 7f404aa00640 1 -- 172.21.15.90:0/2345401906 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f404c108ff0 msgr2=0x7f404c19aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:17.766 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.766+0000 7f404aa00640 1 --2- 172.21.15.90:0/2345401906 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f404c108ff0 0x7f404c19aa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:17.766 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.766+0000 7f404aa00640 1 -- 172.21.15.90:0/2345401906 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f404c19ede0 msgr2=0x7f404c19f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:17.766 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.766+0000 7f404aa00640 1 --2- 172.21.15.90:0/2345401906 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f404c19ede0 0x7f404c19f290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:17.766 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.766+0000 7f404aa00640 1 -- 172.21.15.90:0/2345401906 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f404c19f8d0 con 0x7f404c103f20 2024-04-02T14:42:17.767 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.766+0000 7f4042000640 1 --2- 172.21.15.90:0/2345401906 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f404c108ff0 0x7f404c19aa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:42:17.767 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.767+0000 7f404aa00640 1 --2- 172.21.15.90:0/2345401906 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f404c103f20 0x7f404c19a520 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f403c02c640 tx=0x7f403c035d10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:42:17.768 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.767+0000 7f4043e00640 1 -- 172.21.15.90:0/2345401906 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f403c039040 con 0x7f404c103f20 2024-04-02T14:42:17.768 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.767+0000 7f4043e00640 1 -- 172.21.15.90:0/2345401906 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f403c044070 con 0x7f404c103f20 2024-04-02T14:42:17.768 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.767+0000 7f4051a00640 1 -- 172.21.15.90:0/2345401906 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f404c19fb60 con 0x7f404c103f20 2024-04-02T14:42:17.768 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.767+0000 7f4043e00640 1 -- 172.21.15.90:0/2345401906 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f403c0042a0 con 0x7f404c103f20 2024-04-02T14:42:17.768 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.767+0000 7f4051a00640 1 -- 172.21.15.90:0/2345401906 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f404c1a7320 con 0x7f404c103f20 2024-04-02T14:42:17.769 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.770+0000 7f4043e00640 1 -- 172.21.15.90:0/2345401906 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f403c004440 con 0x7f404c103f20 2024-04-02T14:42:17.769 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.771+0000 7f4054768740 1 Processor -- start 2024-04-02T14:42:17.770 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.771+0000 7f4054768740 1 -- start start 2024-04-02T14:42:17.770 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.771+0000 7f4054768740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555820865eb0 0x555820866290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:17.770 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.771+0000 7f4054768740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5558208667d0 0x55582086ee70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:17.770 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.771+0000 7f4054768740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55582086f3b0 0x5558208717a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:17.771 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.771+0000 7f4054768740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5558207cad70 con 0x5558208667d0 2024-04-02T14:42:17.771 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.771+0000 7f4054768740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x5558207732a0 con 0x55582086f3b0 2024-04-02T14:42:17.771 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.771+0000 7f4054768740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5558207c6d20 con 0x555820865eb0 2024-04-02T14:42:17.772 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.772+0000 7f404b400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555820865eb0 0x555820866290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:17.772 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.772+0000 7f404b400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555820865eb0 0x555820866290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:49364/0 (socket says 172.21.15.90:49364) 2024-04-02T14:42:17.772 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.772+0000 7f404b400640 1 -- 172.21.15.90:0/2374240021 learned_addr learned my addr 172.21.15.90:0/2374240021 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:42:17.772 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.772+0000 7f404aa00640 1 --2- 172.21.15.90:0/2374240021 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5558208667d0 0x55582086ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:17.772 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.772+0000 7f4042000640 1 --2- 172.21.15.90:0/2374240021 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55582086f3b0 0x5558208717a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:17.772 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.772+0000 7f404b400640 1 -- 172.21.15.90:0/2374240021 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55582086f3b0 msgr2=0x5558208717a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:42:17.772 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.772+0000 7f404b400640 1 --2- 172.21.15.90:0/2374240021 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55582086f3b0 0x5558208717a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:17.773 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.772+0000 7f404b400640 1 -- 172.21.15.90:0/2374240021 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5558208667d0 msgr2=0x55582086ee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:42:17.773 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.772+0000 7f404b400640 1 --2- 172.21.15.90:0/2374240021 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5558208667d0 0x55582086ee70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:17.773 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.772+0000 7f404b400640 1 -- 172.21.15.90:0/2374240021 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555820787840 con 0x555820865eb0 2024-04-02T14:42:17.773 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.772+0000 7f404b400640 1 --2- 172.21.15.90:0/2374240021 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555820865eb0 0x555820866290 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f4044009c20 tx=0x7f4044011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f6cb12a1413fd678 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:42:17.773 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.773+0000 7f4043e00640 1 --2- 172.21.15.90:0/2345401906 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f40300a3e40 0x7f40300a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:17.773 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.773+0000 7f4043e00640 1 -- 172.21.15.90:0/2345401906 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8550+0+0 (secure 0 0 0) 0x7f403c0f2a10 con 0x7f404c103f20 2024-04-02T14:42:17.773 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.773+0000 7f403be00640 1 -- 172.21.15.90:0/2374240021 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f404401e070 con 0x555820865eb0 2024-04-02T14:42:17.773 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.773+0000 7f404aa00640 1 --2- 172.21.15.90:0/2345401906 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f40300a3e40 0x7f40300a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:17.773 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.773+0000 7f403be00640 1 -- 172.21.15.90:0/2374240021 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4044019b10 con 0x555820865eb0 2024-04-02T14:42:17.773 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.773+0000 7f403be00640 1 -- 172.21.15.90:0/2374240021 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f404400eaa0 con 0x555820865eb0 2024-04-02T14:42:17.776 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.774+0000 7f4054768740 1 -- 172.21.15.90:0/2374240021 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555820865eb0 msgr2=0x555820866290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:17.776 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.774+0000 7f404aa00640 1 --2- 172.21.15.90:0/2345401906 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f40300a3e40 0x7f40300a6300 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x555820865980 tx=0x7f403c02c100 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:42:17.776 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.774+0000 7f4054768740 1 --2- 172.21.15.90:0/2374240021 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555820865eb0 0x555820866290 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f4044009c20 tx=0x7f4044011810 comp rx=0 tx=0).stop 2024-04-02T14:42:17.776 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.774+0000 7f4054768740 1 -- 172.21.15.90:0/2374240021 shutdown_connections 2024-04-02T14:42:17.776 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.774+0000 7f4054768740 1 --2- 172.21.15.90:0/2374240021 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55582086f3b0 0x5558208717a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:17.776 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.774+0000 7f4054768740 1 --2- 172.21.15.90:0/2374240021 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5558208667d0 0x55582086ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:17.776 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.774+0000 7f4054768740 1 --2- 172.21.15.90:0/2374240021 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555820865eb0 0x555820866290 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:17.776 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.774+0000 7f4054768740 1 -- 172.21.15.90:0/2374240021 >> 172.21.15.90:0/2374240021 conn(0x5558208605e0 msgr2=0x5558208609c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:42:17.776 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.774+0000 7f4054768740 1 -- 172.21.15.90:0/2374240021 shutdown_connections 2024-04-02T14:42:17.776 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.774+0000 7f4054768740 1 -- 172.21.15.90:0/2374240021 wait complete. 2024-04-02T14:42:17.776 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.775+0000 7f4054768740 1 -- start start 2024-04-02T14:42:17.777 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.775+0000 7f4054768740 1 Processor -- start 2024-04-02T14:42:17.777 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.775+0000 7f4054768740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555820865eb0 0x55582086c490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:17.777 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.775+0000 7f4054768740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5558208667d0 0x55582086c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:17.777 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.775+0000 7f4054768740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55582086f3b0 0x55582086cf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:17.777 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.775+0000 7f4054768740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5558208721a0 con 0x555820865eb0 2024-04-02T14:42:17.777 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.775+0000 7f4054768740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x555820786660 con 0x55582086f3b0 2024-04-02T14:42:17.777 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.775+0000 7f4054768740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x555820872320 con 0x5558208667d0 2024-04-02T14:42:17.777 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.775+0000 7f4054768740 10 client.0 _ll_register_callbacks cb 0x555820786050 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:42:17.777 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.776+0000 7f4042000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5558208667d0 0x55582086c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:17.777 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.776+0000 7f4042000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5558208667d0 0x55582086c9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:49368/0 (socket says 172.21.15.90:49368) 2024-04-02T14:42:17.777 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.776+0000 7f4042000640 1 -- 172.21.15.90:0/1507464001 learned_addr learned my addr 172.21.15.90:0/1507464001 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:42:17.777 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.776+0000 7f404b400640 1 --2- 172.21.15.90:0/1507464001 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555820865eb0 0x55582086c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:17.777 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.776+0000 7f404b400640 1 --2- 172.21.15.90:0/1507464001 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55582086f3b0 0x55582086cf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:17.777 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.776+0000 7f4042000640 1 -- 172.21.15.90:0/1507464001 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55582086f3b0 msgr2=0x55582086cf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:42:17.777 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.776+0000 7f4042000640 1 --2- 172.21.15.90:0/1507464001 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55582086f3b0 0x55582086cf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:17.778 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.776+0000 7f4042000640 1 -- 172.21.15.90:0/1507464001 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555820865eb0 msgr2=0x55582086c490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:42:17.778 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.776+0000 7f4042000640 1 --2- 172.21.15.90:0/1507464001 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555820865eb0 0x55582086c490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:17.778 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.776+0000 7f4042000640 1 -- 172.21.15.90:0/1507464001 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555820795e80 con 0x5558208667d0 2024-04-02T14:42:17.778 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.776+0000 7f404b400640 1 --2- 172.21.15.90:0/1507464001 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555820865eb0 0x55582086c490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:42:17.778 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.776+0000 7f4042000640 1 --2- 172.21.15.90:0/1507464001 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5558208667d0 0x55582086c9d0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f4034013a10 tx=0x7f4034013ed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=876b7f3e2731d85f server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:42:17.778 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.776+0000 7f4038c00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:42:17.778 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.777+0000 7f4038c00640 1 -- 172.21.15.90:0/1507464001 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4034011db0 con 0x5558208667d0 2024-04-02T14:42:17.778 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.777+0000 7f4038c00640 1 -- 172.21.15.90:0/1507464001 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8550+0+0 (secure 0 0 0) 0x7f4034004030 con 0x5558208667d0 2024-04-02T14:42:17.778 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.777+0000 7f4054768740 1 -- 172.21.15.90:0/1507464001 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4044019b10 con 0x5558208667d0 2024-04-02T14:42:17.780 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.777+0000 7f4038c00640 1 -- 172.21.15.90:0/1507464001 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f4034004230 con 0x5558208667d0 2024-04-02T14:42:17.780 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.777+0000 7f4054768740 10 client.27529 fetch_fsmap learned FSMap version 27 2024-04-02T14:42:17.780 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.777+0000 7f4054768740 1 -- 172.21.15.90:0/1507464001 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5558204df210 con 0x5558208667d0 2024-04-02T14:42:17.780 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.778+0000 7f4038c00640 1 -- 172.21.15.90:0/1507464001 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4044019b10 con 0x5558208667d0 2024-04-02T14:42:17.780 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.778+0000 7f4038c00640 1 -- 172.21.15.90:0/1507464001 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f403401b500 con 0x5558208667d0 2024-04-02T14:42:17.780 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.778+0000 7f4054768740 10 client.27529 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:42:17.780 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.778+0000 7f4054768740 10 client.27529 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-02T14:42:17.781 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.778+0000 7f4054768740 20 client.27529 populate_metadata read hostname 'smithi090' 2024-04-02T14:42:17.781 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.778+0000 7f4054768740 1 --2- 172.21.15.90:0/1507464001 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55582087f850 0x555820881c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:17.781 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.778+0000 7f4054768740 4 client.27529 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:42:17.781 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.778+0000 7f4054768740 1 -- 172.21.15.90:0/1507464001 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55582077acd0 con 0x55582087f850 2024-04-02T14:42:17.781 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.778+0000 7f404b400640 1 --2- 172.21.15.90:0/1507464001 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55582087f850 0x555820881c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:17.781 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.779+0000 7f404b400640 1 --2- 172.21.15.90:0/1507464001 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55582087f850 0x555820881c60 crc :-1 s=READY pgs=202 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f28f9b1919e46b2b server_cookie=3dafda3751803d1e in_seq=0 out_seq=0 2024-04-02T14:42:17.781 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.779+0000 7f4038c00640 10 client.27529 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:42:17.781 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.780+0000 7f4038c00640 1 -- 172.21.15.90:0/1507464001 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55582077acd0 con 0x55582087f850 2024-04-02T14:42:17.781 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.780+0000 7f4038c00640 10 client.27529 handle_command_reply: tid=0 2024-04-02T14:42:17.781 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.780+0000 7f4054768740 1 client.27529 shutdown 2024-04-02T14:42:17.781 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.780+0000 7f4054768740 1 -- 172.21.15.90:0/1507464001 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5558208667d0 msgr2=0x55582086c9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:17.781 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.780+0000 7f4054768740 1 --2- 172.21.15.90:0/1507464001 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5558208667d0 0x55582086c9d0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f4034013a10 tx=0x7f4034013ed0 comp rx=0 tx=0).stop 2024-04-02T14:42:17.781 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.780+0000 7f4054768740 1 -- 172.21.15.90:0/1507464001 shutdown_connections 2024-04-02T14:42:17.781 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.780+0000 7f4054768740 1 --2- 172.21.15.90:0/1507464001 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55582087f850 0x555820881c60 crc :-1 s=READY pgs=202 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:17.781 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.780+0000 7f4054768740 1 --2- 172.21.15.90:0/1507464001 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55582086f3b0 0x55582086cf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:17.782 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.780+0000 7f4054768740 1 --2- 172.21.15.90:0/1507464001 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5558208667d0 0x55582086c9d0 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:17.782 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.780+0000 7f4054768740 1 --2- 172.21.15.90:0/1507464001 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555820865eb0 0x55582086c490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:17.782 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.780+0000 7f4054768740 1 -- 172.21.15.90:0/1507464001 >> 172.21.15.90:0/1507464001 conn(0x5558208605e0 msgr2=0x555820862330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:42:17.782 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.780+0000 7f4038c00640 0 client.27529 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:42:17.782 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.780+0000 7f4054768740 1 -- 172.21.15.90:0/1507464001 shutdown_connections 2024-04-02T14:42:17.782 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.781+0000 7f4054768740 1 -- 172.21.15.90:0/1507464001 wait complete. 2024-04-02T14:42:17.782 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.781+0000 7f4054768740 20 client.27529 trim_cache size 0 max 16384 2024-04-02T14:42:17.798 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.798+0000 7f4054768740 1 Processor -- start 2024-04-02T14:42:17.799 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.799+0000 7f4054768740 1 -- start start 2024-04-02T14:42:17.799 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.799+0000 7f4054768740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555820865eb0 0x55582086ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:17.799 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.799+0000 7f4054768740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5558208667d0 0x55582087b060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:17.799 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.799+0000 7f4054768740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55582086f3b0 0x55582085f4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:17.799 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.799+0000 7f4054768740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f404c1af9f0 con 0x55582086f3b0 2024-04-02T14:42:17.799 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.799+0000 7f4054768740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f404c1b1530 con 0x555820865eb0 2024-04-02T14:42:17.799 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.799+0000 7f4054768740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f404c1b00c0 con 0x5558208667d0 2024-04-02T14:42:17.799 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.799+0000 7f404b400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555820865eb0 0x55582086ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:17.799 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.799+0000 7f404b400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555820865eb0 0x55582086ebc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:34960/0 (socket says 172.21.15.90:34960) 2024-04-02T14:42:17.799 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.799+0000 7f404b400640 1 -- 172.21.15.90:0/3115195384 learned_addr learned my addr 172.21.15.90:0/3115195384 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:42:17.799 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.799+0000 7f4042000640 1 --2- 172.21.15.90:0/3115195384 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5558208667d0 0x55582087b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:17.799 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.799+0000 7f404b400640 1 --2- 172.21.15.90:0/3115195384 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55582086f3b0 0x55582085f4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:17.800 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.800+0000 7f404b400640 1 -- 172.21.15.90:0/3115195384 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5558208667d0 msgr2=0x55582087b060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:42:17.800 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.800+0000 7f404b400640 1 --2- 172.21.15.90:0/3115195384 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5558208667d0 0x55582087b060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:17.800 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.800+0000 7f404b400640 1 -- 172.21.15.90:0/3115195384 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55582086f3b0 msgr2=0x55582085f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:42:17.800 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.800+0000 7f404b400640 1 --2- 172.21.15.90:0/3115195384 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55582086f3b0 0x55582085f4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:17.800 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.800+0000 7f404b400640 1 -- 172.21.15.90:0/3115195384 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5558205ab620 con 0x555820865eb0 2024-04-02T14:42:17.800 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.800+0000 7f4042000640 1 --2- 172.21.15.90:0/3115195384 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5558208667d0 0x55582087b060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:42:17.803 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.800+0000 7f404b400640 1 --2- 172.21.15.90:0/3115195384 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555820865eb0 0x55582086ebc0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f404400e6b0 tx=0x7f404400e640 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b81c901055fce050 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:42:17.803 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.800+0000 7f403b400640 1 -- 172.21.15.90:0/3115195384 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f404401e040 con 0x555820865eb0 2024-04-02T14:42:17.803 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.800+0000 7f403b400640 1 -- 172.21.15.90:0/3115195384 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4044005530 con 0x555820865eb0 2024-04-02T14:42:17.803 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.800+0000 7f403b400640 1 -- 172.21.15.90:0/3115195384 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f404400ea40 con 0x555820865eb0 2024-04-02T14:42:17.803 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.801+0000 7f4054768740 1 -- 172.21.15.90:0/3115195384 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555820865eb0 msgr2=0x55582086ebc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:17.804 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.801+0000 7f4054768740 1 --2- 172.21.15.90:0/3115195384 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555820865eb0 0x55582086ebc0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f404400e6b0 tx=0x7f404400e640 comp rx=0 tx=0).stop 2024-04-02T14:42:17.804 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.801+0000 7f4054768740 1 -- 172.21.15.90:0/3115195384 shutdown_connections 2024-04-02T14:42:17.804 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.801+0000 7f4054768740 1 --2- 172.21.15.90:0/3115195384 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55582086f3b0 0x55582085f4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:17.804 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.801+0000 7f4054768740 1 --2- 172.21.15.90:0/3115195384 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5558208667d0 0x55582087b060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:17.804 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.801+0000 7f4054768740 1 --2- 172.21.15.90:0/3115195384 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555820865eb0 0x55582086ebc0 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:17.804 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.801+0000 7f4054768740 1 -- 172.21.15.90:0/3115195384 >> 172.21.15.90:0/3115195384 conn(0x5558208605e0 msgr2=0x55582085edb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:42:17.804 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.801+0000 7f4054768740 1 -- 172.21.15.90:0/3115195384 shutdown_connections 2024-04-02T14:42:17.804 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.801+0000 7f4054768740 1 -- 172.21.15.90:0/3115195384 wait complete. 2024-04-02T14:42:17.804 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.802+0000 7f4054768740 1 -- start start 2024-04-02T14:42:17.804 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.802+0000 7f4054768740 1 Processor -- start 2024-04-02T14:42:17.804 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.802+0000 7f4054768740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555820865eb0 0x5558208732b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:17.804 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.802+0000 7f4054768740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5558208667d0 0x5558208737f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:17.804 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.802+0000 7f4054768740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55582086f3b0 0x5558208e4de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:17.804 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.802+0000 7f4054768740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f404c1b0e60 con 0x55582086f3b0 2024-04-02T14:42:17.805 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.802+0000 7f4054768740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f404c1b0790 con 0x5558208667d0 2024-04-02T14:42:17.805 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.802+0000 7f4054768740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f404c1ad9f0 con 0x555820865eb0 2024-04-02T14:42:17.805 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.802+0000 7f4054768740 10 client.0 _ll_register_callbacks cb 0x7f4034013650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:42:17.805 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.802+0000 7f4042000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5558208667d0 0x5558208737f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:17.805 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.802+0000 7f404b400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555820865eb0 0x5558208732b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:17.805 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.802+0000 7f4042000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5558208667d0 0x5558208737f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:34962/0 (socket says 172.21.15.90:34962) 2024-04-02T14:42:17.805 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.802+0000 7f4042000640 1 -- 172.21.15.90:0/3780723116 learned_addr learned my addr 172.21.15.90:0/3780723116 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:42:17.805 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.803+0000 7f404b400640 1 --2- 172.21.15.90:0/3780723116 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55582086f3b0 0x5558208e4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:17.805 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.803+0000 7f4042000640 1 -- 172.21.15.90:0/3780723116 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555820865eb0 msgr2=0x5558208732b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:42:17.805 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.803+0000 7f4042000640 1 --2- 172.21.15.90:0/3780723116 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555820865eb0 0x5558208732b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:17.805 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.803+0000 7f4042000640 1 -- 172.21.15.90:0/3780723116 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55582086f3b0 msgr2=0x5558208e4de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:42:17.805 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.803+0000 7f4042000640 1 --2- 172.21.15.90:0/3780723116 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55582086f3b0 0x5558208e4de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:17.805 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.803+0000 7f4042000640 1 -- 172.21.15.90:0/3780723116 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5558208740c0 con 0x5558208667d0 2024-04-02T14:42:17.805 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.803+0000 7f404b400640 1 --2- 172.21.15.90:0/3780723116 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55582086f3b0 0x5558208e4de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:42:17.805 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.803+0000 7f4042000640 1 --2- 172.21.15.90:0/3780723116 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5558208667d0 0x5558208737f0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f4034016020 tx=0x7f4034005d40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9988f67e77bc6519 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:42:17.806 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.803+0000 7f4038c00640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:42:17.806 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.803+0000 7f4038c00640 1 -- 172.21.15.90:0/3780723116 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4034005600 con 0x5558208667d0 2024-04-02T14:42:17.806 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.803+0000 7f4038c00640 1 -- 172.21.15.90:0/3780723116 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8550+0+0 (secure 0 0 0) 0x7f403401b660 con 0x5558208667d0 2024-04-02T14:42:17.806 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.803+0000 7f4054768740 1 -- 172.21.15.90:0/3780723116 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4044005530 con 0x5558208667d0 2024-04-02T14:42:17.806 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.804+0000 7f4038c00640 1 -- 172.21.15.90:0/3780723116 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f403402d420 con 0x5558208667d0 2024-04-02T14:42:17.806 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.804+0000 7f4054768740 10 client.27665 fetch_fsmap learned FSMap version 27 2024-04-02T14:42:17.806 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.804+0000 7f4054768740 1 -- 172.21.15.90:0/3780723116 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5558208e6460 con 0x5558208667d0 2024-04-02T14:42:17.806 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.804+0000 7f4038c00640 1 -- 172.21.15.90:0/3780723116 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4044005530 con 0x5558208667d0 2024-04-02T14:42:17.806 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.804+0000 7f4038c00640 1 -- 172.21.15.90:0/3780723116 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f4034025620 con 0x5558208667d0 2024-04-02T14:42:17.806 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.804+0000 7f4054768740 10 client.27665 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:42:17.806 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.804+0000 7f4054768740 10 client.27665 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-02T14:42:17.806 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.804+0000 7f4054768740 20 client.27665 populate_metadata read hostname 'smithi090' 2024-04-02T14:42:17.806 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.804+0000 7f4054768740 1 --2- 172.21.15.90:0/3780723116 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5558208e6be0 0x5558208e9000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:17.807 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.804+0000 7f4054768740 4 client.27665 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-02T14:42:17.807 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.804+0000 7f4054768740 1 -- 172.21.15.90:0/3780723116 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5558207d82b0 con 0x5558208e6be0 2024-04-02T14:42:17.807 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.804+0000 7f404b400640 1 --2- 172.21.15.90:0/3780723116 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5558208e6be0 0x5558208e9000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:17.807 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.805+0000 7f404b400640 1 --2- 172.21.15.90:0/3780723116 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5558208e6be0 0x5558208e9000 crc :-1 s=READY pgs=203 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6fcd208c48f5090a server_cookie=8dadd84169ec644e in_seq=0 out_seq=0 2024-04-02T14:42:17.807 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.805+0000 7f4038c00640 10 client.27665 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:42:17.822 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.822+0000 7f4038c00640 1 -- 172.21.15.90:0/3780723116 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5558207d82b0 con 0x5558208e6be0 2024-04-02T14:42:17.822 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.822+0000 7f4038c00640 10 client.27665 handle_command_reply: tid=0 2024-04-02T14:42:17.822 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.822+0000 7f4054768740 1 client.27665 shutdown 2024-04-02T14:42:17.822 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.822+0000 7f4054768740 1 -- 172.21.15.90:0/3780723116 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5558208667d0 msgr2=0x5558208737f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:17.822 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.822+0000 7f4054768740 1 --2- 172.21.15.90:0/3780723116 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5558208667d0 0x5558208737f0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f4034016020 tx=0x7f4034005d40 comp rx=0 tx=0).stop 2024-04-02T14:42:17.822 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.822+0000 7f4054768740 1 -- 172.21.15.90:0/3780723116 shutdown_connections 2024-04-02T14:42:17.822 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.822+0000 7f4054768740 1 --2- 172.21.15.90:0/3780723116 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5558208e6be0 0x5558208e9000 crc :-1 s=READY pgs=203 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:17.822 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.822+0000 7f4054768740 1 --2- 172.21.15.90:0/3780723116 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55582086f3b0 0x5558208e4de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:17.822 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.822+0000 7f4054768740 1 --2- 172.21.15.90:0/3780723116 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5558208667d0 0x5558208737f0 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:17.823 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.822+0000 7f4054768740 1 --2- 172.21.15.90:0/3780723116 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555820865eb0 0x5558208732b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:17.823 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.822+0000 7f4038c00640 0 client.27665 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:42:17.823 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.822+0000 7f4054768740 1 -- 172.21.15.90:0/3780723116 >> 172.21.15.90:0/3780723116 conn(0x5558208605e0 msgr2=0x55582085f640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:42:17.823 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.823+0000 7f4054768740 1 -- 172.21.15.90:0/3780723116 shutdown_connections 2024-04-02T14:42:17.823 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.823+0000 7f4054768740 1 -- 172.21.15.90:0/3780723116 wait complete. 2024-04-02T14:42:17.823 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.823+0000 7f4054768740 20 client.27665 trim_cache size 0 max 16384 2024-04-02T14:42:17.823 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:{ 2024-04-02T14:42:17.823 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "message": "", 2024-04-02T14:42:17.823 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "return_code": 0 2024-04-02T14:42:17.823 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:} 2024-04-02T14:42:17.824 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.823+0000 7f4051a00640 1 -- 172.21.15.90:0/2345401906 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f40300a3e40 msgr2=0x7f40300a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:17.824 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.823+0000 7f4051a00640 1 --2- 172.21.15.90:0/2345401906 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f40300a3e40 0x7f40300a6300 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x555820865980 tx=0x7f403c02c100 comp rx=0 tx=0).stop 2024-04-02T14:42:17.824 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.823+0000 7f4051a00640 1 -- 172.21.15.90:0/2345401906 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f404c103f20 msgr2=0x7f404c19a520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:17.824 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.823+0000 7f4051a00640 1 --2- 172.21.15.90:0/2345401906 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f404c103f20 0x7f404c19a520 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f403c02c640 tx=0x7f403c035d10 comp rx=0 tx=0).stop 2024-04-02T14:42:17.824 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.824+0000 7f4051a00640 1 -- 172.21.15.90:0/2345401906 shutdown_connections 2024-04-02T14:42:17.824 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.824+0000 7f4051a00640 1 --2- 172.21.15.90:0/2345401906 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f40300a3e40 0x7f40300a6300 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:17.825 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.824+0000 7f4051a00640 1 --2- 172.21.15.90:0/2345401906 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f404c19ede0 0x7f404c19f290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:17.825 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.824+0000 7f4051a00640 1 --2- 172.21.15.90:0/2345401906 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f404c108ff0 0x7f404c19aa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:17.825 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.824+0000 7f4051a00640 1 --2- 172.21.15.90:0/2345401906 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f404c103f20 0x7f404c19a520 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:17.825 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.824+0000 7f4051a00640 1 -- 172.21.15.90:0/2345401906 >> 172.21.15.90:0/2345401906 conn(0x7f404c0ff7c0 msgr2=0x7f404c101ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:42:17.825 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.824+0000 7f4051a00640 1 -- 172.21.15.90:0/2345401906 shutdown_connections 2024-04-02T14:42:17.825 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:17.824+0000 7f4051a00640 1 -- 172.21.15.90:0/2345401906 wait complete. 2024-04-02T14:42:18.357 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:42:18 smithi092 ceph-mon[36497]: pgmap v227: 97 pgs: 97 active+clean; 3.8 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 1.4 MiB/s wr, 257 op/s 2024-04-02T14:42:18.433 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:42:18 smithi177 ceph-mon[36431]: pgmap v227: 97 pgs: 97 active+clean; 3.8 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 1.4 MiB/s wr, 257 op/s 2024-04-02T14:42:18.435 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:42:18 smithi090 ceph-mon[31674]: pgmap v227: 97 pgs: 97 active+clean; 3.8 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 1.4 MiB/s wr, 257 op/s 2024-04-02T14:42:20.357 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:42:20 smithi092 ceph-mon[36497]: pgmap v228: 97 pgs: 97 active+clean; 3.8 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 1.5 MiB/s wr, 265 op/s 2024-04-02T14:42:20.433 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:42:20 smithi177 ceph-mon[36431]: pgmap v228: 97 pgs: 97 active+clean; 3.8 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 1.5 MiB/s wr, 265 op/s 2024-04-02T14:42:20.435 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:42:20 smithi090 ceph-mon[31674]: pgmap v228: 97 pgs: 97 active+clean; 3.8 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 1.5 MiB/s wr, 265 op/s 2024-04-02T14:42:21.828 INFO:teuthology.orchestra.run.smithi090.stderr:progress: 410.0 s, 136.6 tps, lat 760.182 ms stddev 476.563 2024-04-02T14:42:22.357 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:42:22 smithi092 ceph-mon[36497]: pgmap v229: 97 pgs: 97 active+clean; 3.8 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 1.5 MiB/s wr, 262 op/s 2024-04-02T14:42:22.433 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:42:22 smithi177 ceph-mon[36431]: pgmap v229: 97 pgs: 97 active+clean; 3.8 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 1.5 MiB/s wr, 262 op/s 2024-04-02T14:42:22.436 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:42:22 smithi090 ceph-mon[31674]: pgmap v229: 97 pgs: 97 active+clean; 3.8 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 1.5 MiB/s wr, 262 op/s 2024-04-02T14:42:24.357 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:42:24 smithi092 ceph-mon[36497]: pgmap v230: 97 pgs: 97 active+clean; 3.8 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 1.5 MiB/s wr, 262 op/s 2024-04-02T14:42:24.433 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:42:24 smithi177 ceph-mon[36431]: pgmap v230: 97 pgs: 97 active+clean; 3.8 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 1.5 MiB/s wr, 262 op/s 2024-04-02T14:42:24.436 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:42:24 smithi090 ceph-mon[31674]: pgmap v230: 97 pgs: 97 active+clean; 3.8 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 1.5 MiB/s wr, 262 op/s 2024-04-02T14:42:26.357 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:42:26 smithi092 ceph-mon[36497]: pgmap v231: 97 pgs: 97 active+clean; 3.8 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 3.7 MiB/s wr, 378 op/s 2024-04-02T14:42:26.433 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:42:26 smithi177 ceph-mon[36431]: pgmap v231: 97 pgs: 97 active+clean; 3.8 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 3.7 MiB/s wr, 378 op/s 2024-04-02T14:42:26.435 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:42:26 smithi090 ceph-mon[31674]: pgmap v231: 97 pgs: 97 active+clean; 3.8 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 3.7 MiB/s wr, 378 op/s 2024-04-02T14:42:26.829 INFO:teuthology.orchestra.run.smithi090.stderr:progress: 415.0 s, 71.8 tps, lat 766.136 ms stddev 483.523 2024-04-02T14:42:28.357 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:42:28 smithi092 ceph-mon[36497]: pgmap v232: 97 pgs: 97 active+clean; 3.8 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 3.2 MiB/s wr, 329 op/s 2024-04-02T14:42:28.433 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:42:28 smithi177 ceph-mon[36431]: pgmap v232: 97 pgs: 97 active+clean; 3.8 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 3.2 MiB/s wr, 329 op/s 2024-04-02T14:42:28.435 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:42:28 smithi090 ceph-mon[31674]: pgmap v232: 97 pgs: 97 active+clean; 3.8 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 3.2 MiB/s wr, 329 op/s 2024-04-02T14:42:30.357 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:42:30 smithi092 ceph-mon[36497]: pgmap v233: 97 pgs: 97 active+clean; 3.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 14 MiB/s wr, 740 op/s 2024-04-02T14:42:30.433 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:42:30 smithi177 ceph-mon[36431]: pgmap v233: 97 pgs: 97 active+clean; 3.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 14 MiB/s wr, 740 op/s 2024-04-02T14:42:30.437 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:42:30 smithi090 ceph-mon[31674]: pgmap v233: 97 pgs: 97 active+clean; 3.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 14 MiB/s wr, 740 op/s 2024-04-02T14:42:31.025 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.027+0000 7f896de00640 1 -- 172.21.15.90:0/2658397902 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8968108cf0 msgr2=0x7f896810cd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:31.025 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.027+0000 7f896de00640 1 --2- 172.21.15.90:0/2658397902 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8968108cf0 0x7f896810cd30 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f8950009900 tx=0x7f895002bc30 comp rx=0 tx=0).stop 2024-04-02T14:42:31.025 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.027+0000 7f896de00640 1 -- 172.21.15.90:0/2658397902 shutdown_connections 2024-04-02T14:42:31.026 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.027+0000 7f896de00640 1 --2- 172.21.15.90:0/2658397902 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f896810d410 0x7f8968114fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:31.026 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.027+0000 7f896de00640 1 --2- 172.21.15.90:0/2658397902 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8968108cf0 0x7f896810cd30 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:31.026 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.027+0000 7f896de00640 1 --2- 172.21.15.90:0/2658397902 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8968108340 0x7f8968108720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:31.026 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.027+0000 7f896de00640 1 -- 172.21.15.90:0/2658397902 >> 172.21.15.90:0/2658397902 conn(0x7f8968076070 msgr2=0x7f8968078490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:42:31.026 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.027+0000 7f896de00640 1 -- 172.21.15.90:0/2658397902 shutdown_connections 2024-04-02T14:42:31.026 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.027+0000 7f896de00640 1 -- 172.21.15.90:0/2658397902 wait complete. 2024-04-02T14:42:31.026 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.029+0000 7f896de00640 1 Processor -- start 2024-04-02T14:42:31.027 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.029+0000 7f896de00640 1 -- start start 2024-04-02T14:42:31.027 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.030+0000 7f896de00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8968108340 0x7f8968082060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:31.028 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.030+0000 7f896de00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f896810d410 0x7f89680825a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:31.028 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.030+0000 7f896de00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f896807b1f0 0x7f896807b6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:31.028 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.030+0000 7f896de00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f89681184d0 con 0x7f896807b1f0 2024-04-02T14:42:31.028 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.030+0000 7f896de00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f8968118350 con 0x7f896810d410 2024-04-02T14:42:31.028 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.030+0000 7f896de00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f8968118650 con 0x7f8968108340 2024-04-02T14:42:31.028 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.030+0000 7f8966a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8968108340 0x7f8968082060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:31.029 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.030+0000 7f8966a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8968108340 0x7f8968082060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:57458/0 (socket says 172.21.15.90:57458) 2024-04-02T14:42:31.029 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.030+0000 7f8966a00640 1 -- 172.21.15.90:0/2139955438 learned_addr learned my addr 172.21.15.90:0/2139955438 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:42:31.029 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.030+0000 7f8966000640 1 --2- 172.21.15.90:0/2139955438 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f896810d410 0x7f89680825a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:31.029 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.030+0000 7f8967400640 1 --2- 172.21.15.90:0/2139955438 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f896807b1f0 0x7f896807b6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:31.030 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.031+0000 7f8966a00640 1 -- 172.21.15.90:0/2139955438 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f896810d410 msgr2=0x7f89680825a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:31.030 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.031+0000 7f8966a00640 1 --2- 172.21.15.90:0/2139955438 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f896810d410 0x7f89680825a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:31.030 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.031+0000 7f8966a00640 1 -- 172.21.15.90:0/2139955438 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f896807b1f0 msgr2=0x7f896807b6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:31.030 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.031+0000 7f8966a00640 1 --2- 172.21.15.90:0/2139955438 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f896807b1f0 0x7f896807b6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:31.030 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.031+0000 7f8966a00640 1 -- 172.21.15.90:0/2139955438 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f896807fb80 con 0x7f8968108340 2024-04-02T14:42:31.030 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.031+0000 7f8966a00640 1 --2- 172.21.15.90:0/2139955438 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8968108340 0x7f8968082060 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f895800d950 tx=0x7f895800de10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:42:31.030 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.032+0000 7f896ca00640 1 -- 172.21.15.90:0/2139955438 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8958017070 con 0x7f8968108340 2024-04-02T14:42:31.030 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.032+0000 7f896ca00640 1 -- 172.21.15.90:0/2139955438 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8958009c30 con 0x7f8968108340 2024-04-02T14:42:31.030 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.032+0000 7f896ca00640 1 -- 172.21.15.90:0/2139955438 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f895800f6b0 con 0x7f8968108340 2024-04-02T14:42:31.030 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.032+0000 7f896de00640 1 -- 172.21.15.90:0/2139955438 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f896807fe70 con 0x7f8968108340 2024-04-02T14:42:31.031 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.032+0000 7f896de00640 1 -- 172.21.15.90:0/2139955438 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8968080340 con 0x7f8968108340 2024-04-02T14:42:31.036 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.038+0000 7f896ca00640 1 -- 172.21.15.90:0/2139955438 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f89580109d0 con 0x7f8968108340 2024-04-02T14:42:31.037 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.039+0000 7f8970ba3740 1 Processor -- start 2024-04-02T14:42:31.038 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.040+0000 7f8970ba3740 1 -- start start 2024-04-02T14:42:31.038 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.040+0000 7f8970ba3740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ee8d019eb0 0x55ee8d01a290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:31.038 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.040+0000 7f8970ba3740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ee8d01a7d0 0x55ee8d022e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:31.038 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.040+0000 7f8970ba3740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ee8d0233b0 0x55ee8d0257a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:31.039 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.040+0000 7f8970ba3740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55ee8cf7ed70 con 0x55ee8d019eb0 2024-04-02T14:42:31.039 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.040+0000 7f8970ba3740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55ee8cf272a0 con 0x55ee8d0233b0 2024-04-02T14:42:31.039 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.040+0000 7f8970ba3740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55ee8cf7ad20 con 0x55ee8d01a7d0 2024-04-02T14:42:31.039 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.041+0000 7f8967400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ee8d019eb0 0x55ee8d01a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:31.040 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.041+0000 7f8967400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ee8d019eb0 0x55ee8d01a290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34708/0 (socket says 172.21.15.90:34708) 2024-04-02T14:42:31.040 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.041+0000 7f8967400640 1 -- 172.21.15.90:0/3344938452 learned_addr learned my addr 172.21.15.90:0/3344938452 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:42:31.040 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.041+0000 7f8966000640 1 --2- 172.21.15.90:0/3344938452 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ee8d0233b0 0x55ee8d0257a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:31.040 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.041+0000 7f8966a00640 1 --2- 172.21.15.90:0/3344938452 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ee8d01a7d0 0x55ee8d022e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:31.040 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.041+0000 7f8967400640 1 -- 172.21.15.90:0/3344938452 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ee8d01a7d0 msgr2=0x55ee8d022e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:42:31.040 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.042+0000 7f896ca00640 1 --2- 172.21.15.90:0/2139955438 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f894c0a3f70 0x7f894c0a6430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:31.040 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.042+0000 7f8967400640 1 --2- 172.21.15.90:0/3344938452 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ee8d01a7d0 0x55ee8d022e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:31.041 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.042+0000 7f8967400640 1 -- 172.21.15.90:0/3344938452 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ee8d0233b0 msgr2=0x55ee8d0257a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:42:31.041 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.042+0000 7f8967400640 1 --2- 172.21.15.90:0/3344938452 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ee8d0233b0 0x55ee8d0257a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:31.041 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.042+0000 7f896ca00640 1 -- 172.21.15.90:0/2139955438 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8550+0+0 (secure 0 0 0) 0x7f89580d0370 con 0x7f8968108340 2024-04-02T14:42:31.041 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.042+0000 7f8967400640 1 -- 172.21.15.90:0/3344938452 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ee8cf3b840 con 0x55ee8d019eb0 2024-04-02T14:42:31.041 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.042+0000 7f8966000640 1 --2- 172.21.15.90:0/3344938452 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ee8d0233b0 0x55ee8d0257a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:42:31.041 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.042+0000 7f8966a00640 1 --2- 172.21.15.90:0/3344938452 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ee8d01a7d0 0x55ee8d022e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:42:31.041 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.043+0000 7f8967400640 1 --2- 172.21.15.90:0/2139955438 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f894c0a3f70 0x7f894c0a6430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:31.042 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.043+0000 7f8967400640 1 --2- 172.21.15.90:0/3344938452 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ee8d019eb0 0x55ee8d01a290 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f8960009f20 tx=0x7f8960010860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8b6b6c7e45bf879 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:42:31.042 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.044+0000 7f8957e00640 1 -- 172.21.15.90:0/3344938452 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8960011980 con 0x55ee8d019eb0 2024-04-02T14:42:31.042 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.044+0000 7f8957e00640 1 -- 172.21.15.90:0/3344938452 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8960011ba0 con 0x55ee8d019eb0 2024-04-02T14:42:31.042 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.044+0000 7f8957e00640 1 -- 172.21.15.90:0/3344938452 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f896001a770 con 0x55ee8d019eb0 2024-04-02T14:42:31.045 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.047+0000 7f8967400640 1 --2- 172.21.15.90:0/2139955438 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f894c0a3f70 0x7f894c0a6430 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f8960011010 tx=0x7f896001a9d0 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:42:31.045 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.047+0000 7f8970ba3740 1 -- 172.21.15.90:0/3344938452 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ee8d019eb0 msgr2=0x55ee8d01a290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:31.045 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.047+0000 7f8970ba3740 1 --2- 172.21.15.90:0/3344938452 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ee8d019eb0 0x55ee8d01a290 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f8960009f20 tx=0x7f8960010860 comp rx=0 tx=0).stop 2024-04-02T14:42:31.045 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.047+0000 7f8970ba3740 1 -- 172.21.15.90:0/3344938452 shutdown_connections 2024-04-02T14:42:31.045 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.047+0000 7f8970ba3740 1 --2- 172.21.15.90:0/3344938452 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ee8d0233b0 0x55ee8d0257a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:31.045 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.047+0000 7f8970ba3740 1 --2- 172.21.15.90:0/3344938452 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ee8d01a7d0 0x55ee8d022e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:31.045 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.047+0000 7f8970ba3740 1 --2- 172.21.15.90:0/3344938452 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ee8d019eb0 0x55ee8d01a290 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:31.045 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.047+0000 7f8970ba3740 1 -- 172.21.15.90:0/3344938452 >> 172.21.15.90:0/3344938452 conn(0x55ee8d0145e0 msgr2=0x55ee8d0149c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:42:31.046 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.048+0000 7f8970ba3740 1 -- 172.21.15.90:0/3344938452 shutdown_connections 2024-04-02T14:42:31.046 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.048+0000 7f8970ba3740 1 -- 172.21.15.90:0/3344938452 wait complete. 2024-04-02T14:42:31.047 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.049+0000 7f8970ba3740 1 -- start start 2024-04-02T14:42:31.048 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.050+0000 7f8970ba3740 1 Processor -- start 2024-04-02T14:42:31.049 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.051+0000 7f8970ba3740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ee8d019eb0 0x55ee8d020380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:31.049 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.051+0000 7f8970ba3740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ee8d01a7d0 0x55ee8d0208c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:31.050 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.051+0000 7f8970ba3740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ee8d0233b0 0x55ee8d020e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:31.050 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.052+0000 7f8966000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ee8d01a7d0 0x55ee8d0208c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:31.050 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.052+0000 7f8966000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ee8d01a7d0 0x55ee8d0208c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:57468/0 (socket says 172.21.15.90:57468) 2024-04-02T14:42:31.051 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.052+0000 7f8966a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ee8d019eb0 0x55ee8d020380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:31.051 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.052+0000 7f8970ba3740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55ee8d0261a0 con 0x55ee8d019eb0 2024-04-02T14:42:31.051 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.053+0000 7f8970ba3740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55ee8cf3a660 con 0x55ee8d0233b0 2024-04-02T14:42:31.051 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.053+0000 7f8970ba3740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55ee8d026320 con 0x55ee8d01a7d0 2024-04-02T14:42:31.051 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.053+0000 7f8970ba3740 10 client.0 _ll_register_callbacks cb 0x55ee8cf3a050 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:42:31.051 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.053+0000 7f8966a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ee8d019eb0 0x55ee8d020380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34712/0 (socket says 172.21.15.90:34712) 2024-04-02T14:42:31.051 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.053+0000 7f8966a00640 1 -- 172.21.15.90:0/3080258511 learned_addr learned my addr 172.21.15.90:0/3080258511 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:42:31.051 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.053+0000 7f8967400640 1 --2- 172.21.15.90:0/3080258511 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ee8d0233b0 0x55ee8d020e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:31.052 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.054+0000 7f8966a00640 1 -- 172.21.15.90:0/3080258511 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ee8d01a7d0 msgr2=0x55ee8d0208c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:42:31.052 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.054+0000 7f8966a00640 1 --2- 172.21.15.90:0/3080258511 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ee8d01a7d0 0x55ee8d0208c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:31.052 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.054+0000 7f8966a00640 1 -- 172.21.15.90:0/3080258511 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ee8d0233b0 msgr2=0x55ee8d020e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:42:31.052 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.054+0000 7f8966a00640 1 --2- 172.21.15.90:0/3080258511 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ee8d0233b0 0x55ee8d020e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:31.052 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.054+0000 7f8966a00640 1 -- 172.21.15.90:0/3080258511 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ee8cf49e80 con 0x55ee8d019eb0 2024-04-02T14:42:31.052 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.054+0000 7f8966000640 1 --2- 172.21.15.90:0/3080258511 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ee8d01a7d0 0x55ee8d0208c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:42:31.053 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.055+0000 7f8966a00640 1 --2- 172.21.15.90:0/3080258511 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ee8d019eb0 0x55ee8d020380 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f8958010fd0 tx=0x7f895809bd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92ff77bbb82032ac server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:42:31.053 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.055+0000 7f8954c00640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-02T14:42:31.053 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.055+0000 7f8954c00640 1 -- 172.21.15.90:0/3080258511 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8958096e60 con 0x55ee8d019eb0 2024-04-02T14:42:31.053 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.055+0000 7f8970ba3740 1 -- 172.21.15.90:0/3080258511 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8960011ba0 con 0x55ee8d019eb0 2024-04-02T14:42:31.053 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.055+0000 7f8954c00640 1 -- 172.21.15.90:0/3080258511 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8550+0+0 (secure 0 0 0) 0x7f8958098960 con 0x55ee8d019eb0 2024-04-02T14:42:31.054 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.056+0000 7f8954c00640 1 -- 172.21.15.90:0/3080258511 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f89580a06a0 con 0x55ee8d019eb0 2024-04-02T14:42:31.054 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.056+0000 7f8970ba3740 10 client.16947 fetch_fsmap learned FSMap version 27 2024-04-02T14:42:31.054 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.056+0000 7f8970ba3740 1 -- 172.21.15.90:0/3080258511 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55ee8cc93210 con 0x55ee8d019eb0 2024-04-02T14:42:31.054 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.056+0000 7f8954c00640 1 -- 172.21.15.90:0/3080258511 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8960011ba0 con 0x55ee8d019eb0 2024-04-02T14:42:31.054 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.057+0000 7f8954c00640 1 -- 172.21.15.90:0/3080258511 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f895809b740 con 0x55ee8d019eb0 2024-04-02T14:42:31.055 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.057+0000 7f8970ba3740 10 client.16947 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:42:31.055 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.057+0000 7f8970ba3740 10 client.16947 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-02T14:42:31.055 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.057+0000 7f8970ba3740 20 client.16947 populate_metadata read hostname 'smithi090' 2024-04-02T14:42:31.055 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.057+0000 7f8970ba3740 1 --2- 172.21.15.90:0/3080258511 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55ee8d033480 0x55ee8d035870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:31.055 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.057+0000 7f8970ba3740 4 client.16947 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:42:31.055 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.057+0000 7f8970ba3740 1 -- 172.21.15.90:0/3080258511 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ee8cf2ecd0 con 0x55ee8d033480 2024-04-02T14:42:31.055 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.058+0000 7f8966000640 1 --2- 172.21.15.90:0/3080258511 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55ee8d033480 0x55ee8d035870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:31.057 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.059+0000 7f8966000640 1 --2- 172.21.15.90:0/3080258511 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55ee8d033480 0x55ee8d035870 crc :-1 s=READY pgs=204 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ee1e9dfaffb4e1f1 server_cookie=ca0b5e8838ac4b0c in_seq=0 out_seq=0 2024-04-02T14:42:31.057 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.059+0000 7f8954c00640 10 client.16947 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:42:31.058 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.060+0000 7f8954c00640 1 -- 172.21.15.90:0/3080258511 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ee8cf2ecd0 con 0x55ee8d033480 2024-04-02T14:42:31.058 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.060+0000 7f8954c00640 10 client.16947 handle_command_reply: tid=0 2024-04-02T14:42:31.059 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.061+0000 7f8970ba3740 1 client.16947 shutdown 2024-04-02T14:42:31.059 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.061+0000 7f8970ba3740 1 -- 172.21.15.90:0/3080258511 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ee8d019eb0 msgr2=0x55ee8d020380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:31.059 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.061+0000 7f8970ba3740 1 --2- 172.21.15.90:0/3080258511 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ee8d019eb0 0x55ee8d020380 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f8958010fd0 tx=0x7f895809bd10 comp rx=0 tx=0).stop 2024-04-02T14:42:31.061 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.062+0000 7f8970ba3740 1 -- 172.21.15.90:0/3080258511 shutdown_connections 2024-04-02T14:42:31.061 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.062+0000 7f8970ba3740 1 --2- 172.21.15.90:0/3080258511 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55ee8d033480 0x55ee8d035870 crc :-1 s=READY pgs=204 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:31.061 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.062+0000 7f8970ba3740 1 --2- 172.21.15.90:0/3080258511 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ee8d0233b0 0x55ee8d020e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:31.061 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.062+0000 7f8970ba3740 1 --2- 172.21.15.90:0/3080258511 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ee8d01a7d0 0x55ee8d0208c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:31.061 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.062+0000 7f8970ba3740 1 --2- 172.21.15.90:0/3080258511 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ee8d019eb0 0x55ee8d020380 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:31.061 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.062+0000 7f8970ba3740 1 -- 172.21.15.90:0/3080258511 >> 172.21.15.90:0/3080258511 conn(0x55ee8d0145e0 msgr2=0x55ee8d016330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:42:31.061 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.062+0000 7f8954c00640 0 client.16947 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:42:31.061 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.062+0000 7f8970ba3740 1 -- 172.21.15.90:0/3080258511 shutdown_connections 2024-04-02T14:42:31.061 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.062+0000 7f8970ba3740 1 -- 172.21.15.90:0/3080258511 wait complete. 2024-04-02T14:42:31.061 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.063+0000 7f8970ba3740 20 client.16947 trim_cache size 0 max 16384 2024-04-02T14:42:31.111 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.113+0000 7f8970ba3740 1 Processor -- start 2024-04-02T14:42:31.111 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.113+0000 7f8970ba3740 1 -- start start 2024-04-02T14:42:31.112 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.114+0000 7f8970ba3740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ee8d019eb0 0x55ee8d022ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:31.112 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.114+0000 7f8970ba3740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ee8d01a7d0 0x55ee8d02f060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:31.112 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.114+0000 7f8970ba3740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ee8d0233b0 0x55ee8d03b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:31.112 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.114+0000 7f8970ba3740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f89681bc5a0 con 0x55ee8d0233b0 2024-04-02T14:42:31.112 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.114+0000 7f8970ba3740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f89681be0e0 con 0x55ee8d01a7d0 2024-04-02T14:42:31.112 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.114+0000 7f8970ba3740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f89681bcc70 con 0x55ee8d019eb0 2024-04-02T14:42:31.113 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.114+0000 7f8966a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ee8d019eb0 0x55ee8d022ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:31.113 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.114+0000 7f8966a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ee8d019eb0 0x55ee8d022ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:57474/0 (socket says 172.21.15.90:57474) 2024-04-02T14:42:31.113 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.114+0000 7f8966a00640 1 -- 172.21.15.90:0/2918239250 learned_addr learned my addr 172.21.15.90:0/2918239250 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:42:31.113 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.115+0000 7f8966000640 1 --2- 172.21.15.90:0/2918239250 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ee8d01a7d0 0x55ee8d02f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:31.113 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.115+0000 7f8967400640 1 --2- 172.21.15.90:0/2918239250 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ee8d0233b0 0x55ee8d03b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:31.113 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.115+0000 7f8966a00640 1 -- 172.21.15.90:0/2918239250 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ee8d01a7d0 msgr2=0x55ee8d02f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:42:31.113 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.115+0000 7f8966a00640 1 --2- 172.21.15.90:0/2918239250 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ee8d01a7d0 0x55ee8d02f060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:31.114 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.115+0000 7f8966a00640 1 -- 172.21.15.90:0/2918239250 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ee8d0233b0 msgr2=0x55ee8d03b0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:42:31.114 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.115+0000 7f8966a00640 1 --2- 172.21.15.90:0/2918239250 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ee8d0233b0 0x55ee8d03b0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:31.114 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.115+0000 7f8966a00640 1 -- 172.21.15.90:0/2918239250 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ee8cd5f620 con 0x55ee8d019eb0 2024-04-02T14:42:31.114 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.115+0000 7f8966000640 1 --2- 172.21.15.90:0/2918239250 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ee8d01a7d0 0x55ee8d02f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:42:31.114 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.115+0000 7f8967400640 1 --2- 172.21.15.90:0/2918239250 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ee8d0233b0 0x55ee8d03b0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:42:31.114 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.116+0000 7f8966a00640 1 --2- 172.21.15.90:0/2918239250 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ee8d019eb0 0x55ee8d022ab0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f89580ce0a0 tx=0x7f895809a800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=843b62041c54b456 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:42:31.115 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.116+0000 7f8957400640 1 -- 172.21.15.90:0/2918239250 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8958014030 con 0x55ee8d019eb0 2024-04-02T14:42:31.115 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.116+0000 7f8957400640 1 -- 172.21.15.90:0/2918239250 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f89580a4f60 con 0x55ee8d019eb0 2024-04-02T14:42:31.115 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.116+0000 7f8957400640 1 -- 172.21.15.90:0/2918239250 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8958098250 con 0x55ee8d019eb0 2024-04-02T14:42:31.116 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.117+0000 7f8970ba3740 1 -- 172.21.15.90:0/2918239250 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ee8d019eb0 msgr2=0x55ee8d022ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:31.116 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.117+0000 7f8970ba3740 1 --2- 172.21.15.90:0/2918239250 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ee8d019eb0 0x55ee8d022ab0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f89580ce0a0 tx=0x7f895809a800 comp rx=0 tx=0).stop 2024-04-02T14:42:31.116 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.117+0000 7f8970ba3740 1 -- 172.21.15.90:0/2918239250 shutdown_connections 2024-04-02T14:42:31.116 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.117+0000 7f8970ba3740 1 --2- 172.21.15.90:0/2918239250 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ee8d0233b0 0x55ee8d03b0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:31.116 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.117+0000 7f8970ba3740 1 --2- 172.21.15.90:0/2918239250 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ee8d01a7d0 0x55ee8d02f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:31.116 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.117+0000 7f8970ba3740 1 --2- 172.21.15.90:0/2918239250 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ee8d019eb0 0x55ee8d022ab0 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:31.116 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.117+0000 7f8970ba3740 1 -- 172.21.15.90:0/2918239250 >> 172.21.15.90:0/2918239250 conn(0x55ee8d0145e0 msgr2=0x55ee8d020bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:42:31.116 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.118+0000 7f8970ba3740 1 -- 172.21.15.90:0/2918239250 shutdown_connections 2024-04-02T14:42:31.116 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.118+0000 7f8970ba3740 1 -- 172.21.15.90:0/2918239250 wait complete. 2024-04-02T14:42:31.117 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.119+0000 7f8970ba3740 1 -- start start 2024-04-02T14:42:31.117 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.119+0000 7f8970ba3740 1 Processor -- start 2024-04-02T14:42:31.118 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.120+0000 7f8970ba3740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ee8d019eb0 0x55ee8d01d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:31.118 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.120+0000 7f8970ba3740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ee8d01a7d0 0x55ee8d01dd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:31.118 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.120+0000 7f8970ba3740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ee8d0233b0 0x55ee8d098960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:31.119 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.120+0000 7f8970ba3740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f89681bda10 con 0x55ee8d019eb0 2024-04-02T14:42:31.119 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.120+0000 7f8970ba3740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f89681bd340 con 0x55ee8d0233b0 2024-04-02T14:42:31.119 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.120+0000 7f8970ba3740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f89681ba5a0 con 0x55ee8d01a7d0 2024-04-02T14:42:31.119 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.120+0000 7f8970ba3740 10 client.0 _ll_register_callbacks cb 0x7f895809c1b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:42:31.119 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.121+0000 7f8967400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ee8d0233b0 0x55ee8d098960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:31.119 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.121+0000 7f8967400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ee8d0233b0 0x55ee8d098960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:58264/0 (socket says 172.21.15.90:58264) 2024-04-02T14:42:31.119 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.121+0000 7f8967400640 1 -- 172.21.15.90:0/3685126686 learned_addr learned my addr 172.21.15.90:0/3685126686 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:42:31.119 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.121+0000 7f8966000640 1 --2- 172.21.15.90:0/3685126686 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ee8d01a7d0 0x55ee8d01dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:31.120 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.121+0000 7f8967400640 1 -- 172.21.15.90:0/3685126686 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ee8d01a7d0 msgr2=0x55ee8d01dd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:42:31.120 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.121+0000 7f8967400640 1 --2- 172.21.15.90:0/3685126686 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ee8d01a7d0 0x55ee8d01dd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:31.120 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.121+0000 7f8967400640 1 -- 172.21.15.90:0/3685126686 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ee8d019eb0 msgr2=0x55ee8d01d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:42:31.120 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.122+0000 7f8966a00640 1 --2- 172.21.15.90:0/3685126686 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ee8d019eb0 0x55ee8d01d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:31.120 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.122+0000 7f8967400640 1 --2- 172.21.15.90:0/3685126686 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ee8d019eb0 0x55ee8d01d7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:31.121 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.122+0000 7f8967400640 1 -- 172.21.15.90:0/3685126686 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ee8d098ef0 con 0x55ee8d0233b0 2024-04-02T14:42:31.122 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.123+0000 7f8967400640 1 --2- 172.21.15.90:0/3685126686 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ee8d0233b0 0x55ee8d098960 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f8960009fd0 tx=0x7f8960015000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=316436b50ef72548 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:42:31.122 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.123+0000 7f8954c00640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:42:31.123 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.125+0000 7f8954c00640 1 -- 172.21.15.90:0/3685126686 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8960035070 con 0x55ee8d0233b0 2024-04-02T14:42:31.123 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.125+0000 7f8954c00640 1 -- 172.21.15.90:0/3685126686 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8550+0+0 (secure 0 0 0) 0x7f8960003b70 con 0x55ee8d0233b0 2024-04-02T14:42:31.123 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.126+0000 7f8970ba3740 1 -- 172.21.15.90:0/3685126686 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f89580a4f60 con 0x55ee8d0233b0 2024-04-02T14:42:31.124 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.127+0000 7f8954c00640 1 -- 172.21.15.90:0/3685126686 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f896003e050 con 0x55ee8d0233b0 2024-04-02T14:42:31.124 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.127+0000 7f8970ba3740 10 client.27680 fetch_fsmap learned FSMap version 27 2024-04-02T14:42:31.125 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.127+0000 7f8970ba3740 1 -- 172.21.15.90:0/3685126686 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55ee8d09a150 con 0x55ee8d0233b0 2024-04-02T14:42:31.125 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.128+0000 7f8954c00640 1 -- 172.21.15.90:0/3685126686 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f89580a4f60 con 0x55ee8d0233b0 2024-04-02T14:42:31.127 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.129+0000 7f8954c00640 1 -- 172.21.15.90:0/3685126686 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f8960030070 con 0x55ee8d0233b0 2024-04-02T14:42:31.127 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.129+0000 7f8970ba3740 10 client.27680 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:42:31.127 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.129+0000 7f8970ba3740 10 client.27680 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-02T14:42:31.127 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.129+0000 7f8970ba3740 20 client.27680 populate_metadata read hostname 'smithi090' 2024-04-02T14:42:31.127 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.129+0000 7f8970ba3740 1 --2- 172.21.15.90:0/3685126686 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55ee8d09a670 0x55ee8d09ca90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:31.127 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.129+0000 7f8970ba3740 4 client.27680 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-02T14:42:31.127 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.129+0000 7f8970ba3740 1 -- 172.21.15.90:0/3685126686 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55ee8cf8c2b0 con 0x55ee8d09a670 2024-04-02T14:42:31.128 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.130+0000 7f8966a00640 1 --2- 172.21.15.90:0/3685126686 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55ee8d09a670 0x55ee8d09ca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:31.129 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.131+0000 7f8966a00640 1 --2- 172.21.15.90:0/3685126686 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55ee8d09a670 0x55ee8d09ca90 crc :-1 s=READY pgs=205 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c379eb51e4ac130e server_cookie=72fe52d5e1e9320f in_seq=0 out_seq=0 2024-04-02T14:42:31.129 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.131+0000 7f8954c00640 10 client.27680 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:42:31.199 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.201+0000 7f8954c00640 1 -- 172.21.15.90:0/3685126686 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55ee8cf8c2b0 con 0x55ee8d09a670 2024-04-02T14:42:31.200 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.201+0000 7f8954c00640 10 client.27680 handle_command_reply: tid=0 2024-04-02T14:42:31.200 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.202+0000 7f8970ba3740 1 client.27680 shutdown 2024-04-02T14:42:31.200 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.202+0000 7f8970ba3740 1 -- 172.21.15.90:0/3685126686 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ee8d0233b0 msgr2=0x55ee8d098960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:31.200 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.202+0000 7f8970ba3740 1 --2- 172.21.15.90:0/3685126686 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ee8d0233b0 0x55ee8d098960 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f8960009fd0 tx=0x7f8960015000 comp rx=0 tx=0).stop 2024-04-02T14:42:31.201 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.203+0000 7f8970ba3740 1 -- 172.21.15.90:0/3685126686 shutdown_connections 2024-04-02T14:42:31.201 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.203+0000 7f8970ba3740 1 --2- 172.21.15.90:0/3685126686 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55ee8d09a670 0x55ee8d09ca90 crc :-1 s=READY pgs=205 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:31.201 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.203+0000 7f8954c00640 0 client.27680 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:42:31.201 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.203+0000 7f8970ba3740 1 --2- 172.21.15.90:0/3685126686 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ee8d0233b0 0x55ee8d098960 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:31.201 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.203+0000 7f8970ba3740 1 --2- 172.21.15.90:0/3685126686 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ee8d01a7d0 0x55ee8d01dd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:31.201 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.203+0000 7f8970ba3740 1 --2- 172.21.15.90:0/3685126686 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ee8d019eb0 0x55ee8d01d7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:31.201 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.203+0000 7f8970ba3740 1 -- 172.21.15.90:0/3685126686 >> 172.21.15.90:0/3685126686 conn(0x55ee8d0145e0 msgr2=0x55ee8d0158d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:42:31.202 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.203+0000 7f8970ba3740 1 -- 172.21.15.90:0/3685126686 shutdown_connections 2024-04-02T14:42:31.202 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.204+0000 7f8970ba3740 1 -- 172.21.15.90:0/3685126686 wait complete. 2024-04-02T14:42:31.202 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.204+0000 7f8970ba3740 20 client.27680 trim_cache size 0 max 16384 2024-04-02T14:42:31.203 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:{ 2024-04-02T14:42:31.203 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "message": "", 2024-04-02T14:42:31.203 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "return_code": 0 2024-04-02T14:42:31.203 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:} 2024-04-02T14:42:31.203 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.205+0000 7f896de00640 1 -- 172.21.15.90:0/2139955438 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f894c0a3f70 msgr2=0x7f894c0a6430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:31.203 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.205+0000 7f896de00640 1 --2- 172.21.15.90:0/2139955438 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f894c0a3f70 0x7f894c0a6430 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f8960011010 tx=0x7f896001a9d0 comp rx=0 tx=0).stop 2024-04-02T14:42:31.203 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.205+0000 7f896de00640 1 -- 172.21.15.90:0/2139955438 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8968108340 msgr2=0x7f8968082060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:31.204 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.205+0000 7f896de00640 1 --2- 172.21.15.90:0/2139955438 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8968108340 0x7f8968082060 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f895800d950 tx=0x7f895800de10 comp rx=0 tx=0).stop 2024-04-02T14:42:31.204 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.205+0000 7f896de00640 1 -- 172.21.15.90:0/2139955438 shutdown_connections 2024-04-02T14:42:31.204 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.205+0000 7f896de00640 1 --2- 172.21.15.90:0/2139955438 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f894c0a3f70 0x7f894c0a6430 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:31.204 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.205+0000 7f896de00640 1 --2- 172.21.15.90:0/2139955438 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f896807b1f0 0x7f896807b6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:31.204 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.205+0000 7f896de00640 1 --2- 172.21.15.90:0/2139955438 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f896810d410 0x7f89680825a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:31.204 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.205+0000 7f896de00640 1 --2- 172.21.15.90:0/2139955438 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8968108340 0x7f8968082060 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:31.204 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.205+0000 7f896de00640 1 -- 172.21.15.90:0/2139955438 >> 172.21.15.90:0/2139955438 conn(0x7f8968076070 msgr2=0x7f8968078330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:42:31.204 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.206+0000 7f896de00640 1 -- 172.21.15.90:0/2139955438 shutdown_connections 2024-04-02T14:42:31.205 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:31.206+0000 7f896de00640 1 -- 172.21.15.90:0/2139955438 wait complete. 2024-04-02T14:42:31.828 INFO:teuthology.orchestra.run.smithi090.stderr:progress: 420.0 s, 3042.0 tps, lat 49.901 ms stddev 223.637 2024-04-02T14:42:31.872 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-02T14:42:31.873 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-02T14:42:32.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.185+0000 7f0236800640 1 -- 172.21.15.90:0/200851129 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0230108fe0 msgr2=0x7f0230110bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:32.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.185+0000 7f0236800640 1 --2- 172.21.15.90:0/200851129 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0230108fe0 0x7f0230110bb0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f0228009ca0 tx=0x7f022802c100 comp rx=0 tx=0).stop 2024-04-02T14:42:32.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.185+0000 7f0236800640 1 -- 172.21.15.90:0/200851129 shutdown_connections 2024-04-02T14:42:32.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.185+0000 7f0236800640 1 --2- 172.21.15.90:0/200851129 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0230108fe0 0x7f0230110bb0 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:32.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.185+0000 7f0236800640 1 --2- 172.21.15.90:0/200851129 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f02301048c0 0x7f0230108900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:32.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.185+0000 7f0236800640 1 --2- 172.21.15.90:0/200851129 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0230103f10 0x7f02301042f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:32.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.185+0000 7f0236800640 1 -- 172.21.15.90:0/200851129 >> 172.21.15.90:0/200851129 conn(0x7f02300ff750 msgr2=0x7f0230101b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:42:32.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.187+0000 7f0236800640 1 -- 172.21.15.90:0/200851129 shutdown_connections 2024-04-02T14:42:32.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.187+0000 7f0236800640 1 -- 172.21.15.90:0/200851129 wait complete. 2024-04-02T14:42:32.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.188+0000 7f0236800640 1 Processor -- start 2024-04-02T14:42:32.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.189+0000 7f0236800640 1 -- start start 2024-04-02T14:42:32.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.191+0000 7f0236800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0230103f10 0x7f0230076010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:32.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.191+0000 7f0236800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0230108fe0 0x7f0230076550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:32.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.191+0000 7f0236800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0230076a90 0x7f02301aabc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:32.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.191+0000 7f0236800640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f02301140f0 con 0x7f0230103f10 2024-04-02T14:42:32.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.191+0000 7f0236800640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f0230113f70 con 0x7f0230076a90 2024-04-02T14:42:32.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.191+0000 7f0236800640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f0230114270 con 0x7f0230108fe0 2024-04-02T14:42:32.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.191+0000 7f022fe00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0230076a90 0x7f02301aabc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:32.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.191+0000 7f022ea00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0230108fe0 0x7f0230076550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:32.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.191+0000 7f022fe00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0230076a90 0x7f02301aabc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:58270/0 (socket says 172.21.15.90:58270) 2024-04-02T14:42:32.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.191+0000 7f022fe00640 1 -- 172.21.15.90:0/3141353420 learned_addr learned my addr 172.21.15.90:0/3141353420 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:42:32.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.192+0000 7f022ea00640 1 -- 172.21.15.90:0/3141353420 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0230076a90 msgr2=0x7f02301aabc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:32.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.192+0000 7f022ea00640 1 --2- 172.21.15.90:0/3141353420 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0230076a90 0x7f02301aabc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:32.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.192+0000 7f022ea00640 1 -- 172.21.15.90:0/3141353420 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0230103f10 msgr2=0x7f0230076010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:32.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.192+0000 7f022ea00640 1 --2- 172.21.15.90:0/3141353420 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0230103f10 0x7f0230076010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:32.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.192+0000 7f022ea00640 1 -- 172.21.15.90:0/3141353420 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02301ab100 con 0x7f0230108fe0 2024-04-02T14:42:32.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.192+0000 7f022fe00640 1 --2- 172.21.15.90:0/3141353420 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0230076a90 0x7f02301aabc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:42:32.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.193+0000 7f022ea00640 1 --2- 172.21.15.90:0/3141353420 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0230108fe0 0x7f0230076550 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f021c00b3c0 tx=0x7f021c00b880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:42:32.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.193+0000 7f0227e00640 1 -- 172.21.15.90:0/3141353420 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f021c011830 con 0x7f0230108fe0 2024-04-02T14:42:32.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.193+0000 7f0227e00640 1 -- 172.21.15.90:0/3141353420 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f021c016070 con 0x7f0230108fe0 2024-04-02T14:42:32.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.193+0000 7f0227e00640 1 -- 172.21.15.90:0/3141353420 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f021c010e50 con 0x7f0230108fe0 2024-04-02T14:42:32.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.194+0000 7f0236800640 1 -- 172.21.15.90:0/3141353420 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02301ab3f0 con 0x7f0230108fe0 2024-04-02T14:42:32.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.194+0000 7f0236800640 1 -- 172.21.15.90:0/3141353420 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02301ab8c0 con 0x7f0230108fe0 2024-04-02T14:42:32.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.200+0000 7f023959c740 1 Processor -- start 2024-04-02T14:42:32.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.201+0000 7f023959c740 1 -- start start 2024-04-02T14:42:32.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.202+0000 7f023959c740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564321c6c7b0 0x564321c6cb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:32.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.202+0000 7f023959c740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564321c6d0d0 0x564321c75770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:32.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.202+0000 7f023959c740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564321c75cb0 0x564321c780a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:32.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.202+0000 7f023959c740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x564321bd0770 con 0x564321c6d0d0 2024-04-02T14:42:32.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.202+0000 7f023959c740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x564321b78700 con 0x564321c6c7b0 2024-04-02T14:42:32.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.202+0000 7f023959c740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x564321bcc720 con 0x564321c75cb0 2024-04-02T14:42:32.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.202+0000 7f022fe00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564321c6c7b0 0x564321c6cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:32.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.202+0000 7f022fe00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564321c6c7b0 0x564321c6cb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:58276/0 (socket says 172.21.15.90:58276) 2024-04-02T14:42:32.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.202+0000 7f022fe00640 1 -- 172.21.15.90:0/1346654109 learned_addr learned my addr 172.21.15.90:0/1346654109 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:42:32.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.203+0000 7f022f400640 1 --2- 172.21.15.90:0/1346654109 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564321c6d0d0 0x564321c75770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:32.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.204+0000 7f022fe00640 1 -- 172.21.15.90:0/1346654109 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564321c75cb0 msgr2=0x564321c780a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-02T14:42:32.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.204+0000 7f022fe00640 1 --2- 172.21.15.90:0/1346654109 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564321c75cb0 0x564321c780a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:32.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.204+0000 7f022fe00640 1 -- 172.21.15.90:0/1346654109 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564321c6d0d0 msgr2=0x564321c75770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:42:32.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.204+0000 7f022fe00640 1 --2- 172.21.15.90:0/1346654109 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564321c6d0d0 0x564321c75770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:32.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.204+0000 7f022fe00640 1 -- 172.21.15.90:0/1346654109 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5643219ae7d0 con 0x564321c6c7b0 2024-04-02T14:42:32.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.204+0000 7f022f400640 1 --2- 172.21.15.90:0/1346654109 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564321c6d0d0 0x564321c75770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:42:32.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.204+0000 7f022fe00640 1 --2- 172.21.15.90:0/1346654109 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564321c6c7b0 0x564321c6cb90 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f0228009830 tx=0x7f0228009860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cbf66a1d0d8ae516 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:42:32.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.205+0000 7f0227e00640 1 -- 172.21.15.90:0/3141353420 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f021c01b460 con 0x7f0230108fe0 2024-04-02T14:42:32.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.205+0000 7f0224c00640 1 -- 172.21.15.90:0/1346654109 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0228043070 con 0x564321c6c7b0 2024-04-02T14:42:32.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.205+0000 7f0224c00640 1 -- 172.21.15.90:0/1346654109 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0228003970 con 0x564321c6c7b0 2024-04-02T14:42:32.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.206+0000 7f0224c00640 1 -- 172.21.15.90:0/1346654109 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0228039040 con 0x564321c6c7b0 2024-04-02T14:42:32.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.207+0000 7f0227e00640 1 --2- 172.21.15.90:0/3141353420 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f02140a4000 0x7f02140a64c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:32.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.207+0000 7f0227e00640 1 -- 172.21.15.90:0/3141353420 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8550+0+0 (secure 0 0 0) 0x7f021c0cfc50 con 0x7f0230108fe0 2024-04-02T14:42:32.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.208+0000 7f022f400640 1 --2- 172.21.15.90:0/3141353420 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f02140a4000 0x7f02140a64c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:32.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.210+0000 7f022f400640 1 --2- 172.21.15.90:0/3141353420 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f02140a4000 0x7f02140a64c0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x564321c6c280 tx=0x7f0220009620 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:42:32.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.210+0000 7f023959c740 1 -- 172.21.15.90:0/1346654109 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564321c6c7b0 msgr2=0x564321c6cb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:32.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.210+0000 7f023959c740 1 --2- 172.21.15.90:0/1346654109 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564321c6c7b0 0x564321c6cb90 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f0228009830 tx=0x7f0228009860 comp rx=0 tx=0).stop 2024-04-02T14:42:32.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.211+0000 7f023959c740 1 -- 172.21.15.90:0/1346654109 shutdown_connections 2024-04-02T14:42:32.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.211+0000 7f023959c740 1 --2- 172.21.15.90:0/1346654109 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564321c75cb0 0x564321c780a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:32.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.211+0000 7f023959c740 1 --2- 172.21.15.90:0/1346654109 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564321c6d0d0 0x564321c75770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:32.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.211+0000 7f023959c740 1 --2- 172.21.15.90:0/1346654109 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564321c6c7b0 0x564321c6cb90 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:32.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.211+0000 7f023959c740 1 -- 172.21.15.90:0/1346654109 >> 172.21.15.90:0/1346654109 conn(0x564321c66ee0 msgr2=0x564321c672c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:42:32.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.212+0000 7f023959c740 1 -- 172.21.15.90:0/1346654109 shutdown_connections 2024-04-02T14:42:32.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.212+0000 7f023959c740 1 -- 172.21.15.90:0/1346654109 wait complete. 2024-04-02T14:42:32.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.213+0000 7f023959c740 1 -- start start 2024-04-02T14:42:32.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.213+0000 7f023959c740 1 Processor -- start 2024-04-02T14:42:32.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.215+0000 7f023959c740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564321c6c7b0 0x564321c72df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:32.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.215+0000 7f023959c740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564321c6d0d0 0x564321c73330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:32.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.215+0000 7f023959c740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564321c75cb0 0x564321c73870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:32.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.215+0000 7f023959c740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x564321c78a40 con 0x564321c6c7b0 2024-04-02T14:42:32.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.215+0000 7f023959c740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x564321b8bac0 con 0x564321c6d0d0 2024-04-02T14:42:32.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.215+0000 7f023959c740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x564321c78bc0 con 0x564321c75cb0 2024-04-02T14:42:32.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.215+0000 7f023959c740 10 client.0 _ll_register_callbacks cb 0x564321b8e180 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:42:32.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.215+0000 7f022fe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564321c75cb0 0x564321c73870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:32.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.215+0000 7f022fe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564321c75cb0 0x564321c73870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:57500/0 (socket says 172.21.15.90:57500) 2024-04-02T14:42:32.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.215+0000 7f022fe00640 1 -- 172.21.15.90:0/2969064640 learned_addr learned my addr 172.21.15.90:0/2969064640 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:42:32.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.216+0000 7f022ea00640 1 --2- 172.21.15.90:0/2969064640 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564321c6d0d0 0x564321c73330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:32.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.216+0000 7f022fe00640 1 -- 172.21.15.90:0/2969064640 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564321c6d0d0 msgr2=0x564321c73330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:42:32.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.216+0000 7f022fe00640 1 --2- 172.21.15.90:0/2969064640 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564321c6d0d0 0x564321c73330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:32.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.216+0000 7f022fe00640 1 -- 172.21.15.90:0/2969064640 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564321c6c7b0 msgr2=0x564321c72df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:42:32.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.216+0000 7f022fe00640 1 --2- 172.21.15.90:0/2969064640 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564321c6c7b0 0x564321c72df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:32.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.216+0000 7f022fe00640 1 -- 172.21.15.90:0/2969064640 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5643218e43f0 con 0x564321c75cb0 2024-04-02T14:42:32.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.216+0000 7f022fe00640 1 --2- 172.21.15.90:0/2969064640 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564321c75cb0 0x564321c73870 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f0228009a00 tx=0x7f0228005bf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ef1905ca6b36136d server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:42:32.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.216+0000 7f0219600640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:42:32.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.217+0000 7f0219600640 1 -- 172.21.15.90:0/2969064640 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f022804c070 con 0x564321c75cb0 2024-04-02T14:42:32.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.217+0000 7f023959c740 1 -- 172.21.15.90:0/2969064640 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0228003970 con 0x564321c75cb0 2024-04-02T14:42:32.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.217+0000 7f0219600640 1 -- 172.21.15.90:0/2969064640 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8550+0+0 (secure 0 0 0) 0x7f02280329a0 con 0x564321c75cb0 2024-04-02T14:42:32.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.217+0000 7f0219600640 1 -- 172.21.15.90:0/2969064640 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f0228058050 con 0x564321c75cb0 2024-04-02T14:42:32.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.217+0000 7f023959c740 10 client.27556 fetch_fsmap learned FSMap version 27 2024-04-02T14:42:32.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.217+0000 7f023959c740 1 -- 172.21.15.90:0/2969064640 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5643218a40f0 con 0x564321c75cb0 2024-04-02T14:42:32.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.218+0000 7f0219600640 1 -- 172.21.15.90:0/2969064640 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0228003970 con 0x564321c75cb0 2024-04-02T14:42:32.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.218+0000 7f0219600640 1 -- 172.21.15.90:0/2969064640 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f0228048450 con 0x564321c75cb0 2024-04-02T14:42:32.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.219+0000 7f023959c740 10 client.27556 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:42:32.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.219+0000 7f023959c740 10 client.27556 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:42:32.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.219+0000 7f023959c740 20 client.27556 populate_metadata read hostname 'smithi090' 2024-04-02T14:42:32.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.219+0000 7f023959c740 1 --2- 172.21.15.90:0/2969064640 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x564321c85be0 0x564321c87fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:32.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.219+0000 7f023959c740 4 client.27556 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:42:32.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.219+0000 7f023959c740 1 -- 172.21.15.90:0/2969064640 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564321b71630 con 0x564321c85be0 2024-04-02T14:42:32.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.219+0000 7f022f400640 1 --2- 172.21.15.90:0/2969064640 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x564321c85be0 0x564321c87fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:32.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.220+0000 7f022f400640 1 --2- 172.21.15.90:0/2969064640 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x564321c85be0 0x564321c87fd0 crc :-1 s=READY pgs=206 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7b79ae49f55e064a server_cookie=8d131ba80eaf52d0 in_seq=0 out_seq=0 2024-04-02T14:42:32.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.220+0000 7f0219600640 10 client.27556 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:42:32.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.221+0000 7f0219600640 1 -- 172.21.15.90:0/2969064640 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x564321b71630 con 0x564321c85be0 2024-04-02T14:42:32.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.221+0000 7f0219600640 10 client.27556 handle_command_reply: tid=0 2024-04-02T14:42:32.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.221+0000 7f023959c740 1 client.27556 shutdown 2024-04-02T14:42:32.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.222+0000 7f023959c740 1 -- 172.21.15.90:0/2969064640 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564321c75cb0 msgr2=0x564321c73870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:32.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.222+0000 7f023959c740 1 --2- 172.21.15.90:0/2969064640 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564321c75cb0 0x564321c73870 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f0228009a00 tx=0x7f0228005bf0 comp rx=0 tx=0).stop 2024-04-02T14:42:32.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.222+0000 7f023959c740 1 -- 172.21.15.90:0/2969064640 shutdown_connections 2024-04-02T14:42:32.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.222+0000 7f023959c740 1 --2- 172.21.15.90:0/2969064640 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x564321c85be0 0x564321c87fd0 crc :-1 s=READY pgs=206 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:32.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.222+0000 7f023959c740 1 --2- 172.21.15.90:0/2969064640 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564321c75cb0 0x564321c73870 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:32.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.222+0000 7f023959c740 1 --2- 172.21.15.90:0/2969064640 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564321c6d0d0 0x564321c73330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:32.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.222+0000 7f023959c740 1 --2- 172.21.15.90:0/2969064640 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564321c6c7b0 0x564321c72df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:32.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.222+0000 7f023959c740 1 -- 172.21.15.90:0/2969064640 >> 172.21.15.90:0/2969064640 conn(0x564321c66ee0 msgr2=0x564321c68c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:42:32.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.222+0000 7f0219600640 0 client.27556 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:42:32.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.223+0000 7f023959c740 1 -- 172.21.15.90:0/2969064640 shutdown_connections 2024-04-02T14:42:32.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.223+0000 7f023959c740 1 -- 172.21.15.90:0/2969064640 wait complete. 2024-04-02T14:42:32.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.223+0000 7f023959c740 20 client.27556 trim_cache size 0 max 16384 2024-04-02T14:42:32.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.253+0000 7f023959c740 1 Processor -- start 2024-04-02T14:42:32.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.253+0000 7f023959c740 1 -- start start 2024-04-02T14:42:32.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.254+0000 7f023959c740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564321c6c7b0 0x564321c75520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:32.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.254+0000 7f023959c740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564321c6d0d0 0x564321c726d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:32.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.254+0000 7f023959c740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564321c75cb0 0x564321c65de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:32.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.254+0000 7f023959c740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f02301b3e90 con 0x564321c6c7b0 2024-04-02T14:42:32.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.254+0000 7f023959c740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f02301b59d0 con 0x564321c6d0d0 2024-04-02T14:42:32.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.254+0000 7f023959c740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f02301b4560 con 0x564321c75cb0 2024-04-02T14:42:32.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.254+0000 7f022fe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564321c75cb0 0x564321c65de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:32.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.254+0000 7f022fe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564321c75cb0 0x564321c65de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:57512/0 (socket says 172.21.15.90:57512) 2024-04-02T14:42:32.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.254+0000 7f022fe00640 1 -- 172.21.15.90:0/1413611989 learned_addr learned my addr 172.21.15.90:0/1413611989 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:42:32.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.254+0000 7f022ea00640 1 --2- 172.21.15.90:0/1413611989 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564321c6d0d0 0x564321c726d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:32.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.254+0000 7f022f400640 1 --2- 172.21.15.90:0/1413611989 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564321c6c7b0 0x564321c75520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:32.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.255+0000 7f022fe00640 1 -- 172.21.15.90:0/1413611989 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564321c6d0d0 msgr2=0x564321c726d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:42:32.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.255+0000 7f022fe00640 1 --2- 172.21.15.90:0/1413611989 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564321c6d0d0 0x564321c726d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:32.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.255+0000 7f022fe00640 1 -- 172.21.15.90:0/1413611989 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564321c6c7b0 msgr2=0x564321c75520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:42:32.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.255+0000 7f022fe00640 1 --2- 172.21.15.90:0/1413611989 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564321c6c7b0 0x564321c75520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:32.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.255+0000 7f022fe00640 1 -- 172.21.15.90:0/1413611989 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564321c78b00 con 0x564321c75cb0 2024-04-02T14:42:32.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.255+0000 7f022ea00640 1 --2- 172.21.15.90:0/1413611989 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564321c6d0d0 0x564321c726d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:42:32.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.255+0000 7f022f400640 1 --2- 172.21.15.90:0/1413611989 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564321c6c7b0 0x564321c75520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:42:32.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.255+0000 7f022fe00640 1 --2- 172.21.15.90:0/1413611989 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564321c75cb0 0x564321c65de0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f0228048990 tx=0x7f02280324f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d168b9b11f4ea774 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:42:32.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.256+0000 7f021be00640 1 -- 172.21.15.90:0/1413611989 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0228055030 con 0x564321c75cb0 2024-04-02T14:42:32.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.256+0000 7f021be00640 1 -- 172.21.15.90:0/1413611989 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0228003970 con 0x564321c75cb0 2024-04-02T14:42:32.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.256+0000 7f021be00640 1 -- 172.21.15.90:0/1413611989 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f022804c050 con 0x564321c75cb0 2024-04-02T14:42:32.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.257+0000 7f023959c740 1 -- 172.21.15.90:0/1413611989 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564321c75cb0 msgr2=0x564321c65de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:32.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.257+0000 7f023959c740 1 --2- 172.21.15.90:0/1413611989 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564321c75cb0 0x564321c65de0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f0228048990 tx=0x7f02280324f0 comp rx=0 tx=0).stop 2024-04-02T14:42:32.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.257+0000 7f023959c740 1 -- 172.21.15.90:0/1413611989 shutdown_connections 2024-04-02T14:42:32.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.257+0000 7f023959c740 1 --2- 172.21.15.90:0/1413611989 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564321c75cb0 0x564321c65de0 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:32.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.257+0000 7f023959c740 1 --2- 172.21.15.90:0/1413611989 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564321c6d0d0 0x564321c726d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:32.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.257+0000 7f023959c740 1 --2- 172.21.15.90:0/1413611989 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564321c6c7b0 0x564321c75520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:32.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.257+0000 7f023959c740 1 -- 172.21.15.90:0/1413611989 >> 172.21.15.90:0/1413611989 conn(0x564321c66ee0 msgr2=0x564321c656d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:42:32.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.258+0000 7f023959c740 1 -- 172.21.15.90:0/1413611989 shutdown_connections 2024-04-02T14:42:32.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.258+0000 7f023959c740 1 -- 172.21.15.90:0/1413611989 wait complete. 2024-04-02T14:42:32.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.258+0000 7f023959c740 1 -- start start 2024-04-02T14:42:32.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.258+0000 7f023959c740 1 Processor -- start 2024-04-02T14:42:32.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.259+0000 7f023959c740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564321c6c7b0 0x564321c8b170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:32.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.259+0000 7f023959c740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564321c6d0d0 0x564321c88550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:32.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.259+0000 7f023959c740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564321c75cb0 0x564321c88c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:32.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.259+0000 7f023959c740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f02301b5300 con 0x564321c75cb0 2024-04-02T14:42:32.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.259+0000 7f023959c740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f02301b4c30 con 0x564321c6c7b0 2024-04-02T14:42:32.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.259+0000 7f023959c740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f02301b1e90 con 0x564321c6d0d0 2024-04-02T14:42:32.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.259+0000 7f023959c740 10 client.0 _ll_register_callbacks cb 0x7f0220009570 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:42:32.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.259+0000 7f022ea00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564321c6d0d0 0x564321c88550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:32.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.260+0000 7f022ea00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564321c6d0d0 0x564321c88550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:57526/0 (socket says 172.21.15.90:57526) 2024-04-02T14:42:32.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.260+0000 7f022ea00640 1 -- 172.21.15.90:0/2304307803 learned_addr learned my addr 172.21.15.90:0/2304307803 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:42:32.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.260+0000 7f022f400640 1 --2- 172.21.15.90:0/2304307803 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564321c6c7b0 0x564321c8b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:32.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.260+0000 7f022fe00640 1 --2- 172.21.15.90:0/2304307803 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564321c75cb0 0x564321c88c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:32.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.260+0000 7f022ea00640 1 -- 172.21.15.90:0/2304307803 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564321c6c7b0 msgr2=0x564321c8b170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:42:32.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.260+0000 7f022ea00640 1 --2- 172.21.15.90:0/2304307803 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564321c6c7b0 0x564321c8b170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:32.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.260+0000 7f022ea00640 1 -- 172.21.15.90:0/2304307803 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564321c75cb0 msgr2=0x564321c88c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:42:32.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.260+0000 7f022ea00640 1 --2- 172.21.15.90:0/2304307803 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564321c75cb0 0x564321c88c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:32.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.260+0000 7f022ea00640 1 -- 172.21.15.90:0/2304307803 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564321ceadc0 con 0x564321c6d0d0 2024-04-02T14:42:32.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.261+0000 7f022ea00640 1 --2- 172.21.15.90:0/2304307803 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564321c6d0d0 0x564321c88550 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f021c0980e0 tx=0x7f021c0105f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1ccd6f3273495b4b server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:42:32.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.261+0000 7f0219600640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:42:32.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.261+0000 7f0219600640 1 -- 172.21.15.90:0/2304307803 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f021c0cd7a0 con 0x564321c6d0d0 2024-04-02T14:42:32.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.261+0000 7f0219600640 1 -- 172.21.15.90:0/2304307803 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8550+0+0 (secure 0 0 0) 0x7f021c0a4460 con 0x564321c6d0d0 2024-04-02T14:42:32.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.261+0000 7f023959c740 1 -- 172.21.15.90:0/2304307803 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0228003970 con 0x564321c6d0d0 2024-04-02T14:42:32.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.261+0000 7f0219600640 1 -- 172.21.15.90:0/2304307803 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f021c0a0030 con 0x564321c6d0d0 2024-04-02T14:42:32.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.261+0000 7f023959c740 10 client.27562 fetch_fsmap learned FSMap version 27 2024-04-02T14:42:32.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.261+0000 7f023959c740 1 -- 172.21.15.90:0/2304307803 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x564321cebf40 con 0x564321c6d0d0 2024-04-02T14:42:32.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.261+0000 7f0219600640 1 -- 172.21.15.90:0/2304307803 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0228003970 con 0x564321c6d0d0 2024-04-02T14:42:32.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.262+0000 7f0219600640 1 -- 172.21.15.90:0/2304307803 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f021c09ab40 con 0x564321c6d0d0 2024-04-02T14:42:32.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.262+0000 7f023959c740 10 client.27562 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:42:32.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.262+0000 7f023959c740 10 client.27562 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:42:32.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.262+0000 7f023959c740 20 client.27562 populate_metadata read hostname 'smithi090' 2024-04-02T14:42:32.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.262+0000 7f023959c740 1 --2- 172.21.15.90:0/2304307803 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x564321cec670 0x564321ceea90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:32.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.262+0000 7f023959c740 4 client.27562 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-02T14:42:32.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.262+0000 7f023959c740 1 -- 172.21.15.90:0/2304307803 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "scrub status"}) -- 0x564321bddcb0 con 0x564321cec670 2024-04-02T14:42:32.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.262+0000 7f022f400640 1 --2- 172.21.15.90:0/2304307803 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x564321cec670 0x564321ceea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:32.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.265+0000 7f022f400640 1 --2- 172.21.15.90:0/2304307803 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x564321cec670 0x564321ceea90 crc :-1 s=READY pgs=207 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a4518ba03a07f1fd server_cookie=6dfaf2f034dcc26e in_seq=0 out_seq=0 2024-04-02T14:42:32.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.265+0000 7f0219600640 10 client.27562 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:42:32.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.265+0000 7f0219600640 1 -- 172.21.15.90:0/2304307803 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x564321bddcb0 con 0x564321cec670 2024-04-02T14:42:32.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.265+0000 7f0219600640 10 client.27562 handle_command_reply: tid=0 2024-04-02T14:42:32.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.265+0000 7f023959c740 1 client.27562 shutdown 2024-04-02T14:42:32.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.266+0000 7f023959c740 1 -- 172.21.15.90:0/2304307803 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564321c6d0d0 msgr2=0x564321c88550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:32.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.266+0000 7f023959c740 1 --2- 172.21.15.90:0/2304307803 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564321c6d0d0 0x564321c88550 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f021c0980e0 tx=0x7f021c0105f0 comp rx=0 tx=0).stop 2024-04-02T14:42:32.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.266+0000 7f023959c740 1 -- 172.21.15.90:0/2304307803 shutdown_connections 2024-04-02T14:42:32.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.266+0000 7f023959c740 1 --2- 172.21.15.90:0/2304307803 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x564321cec670 0x564321ceea90 crc :-1 s=READY pgs=207 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:32.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.266+0000 7f023959c740 1 --2- 172.21.15.90:0/2304307803 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564321c75cb0 0x564321c88c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:32.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.266+0000 7f023959c740 1 --2- 172.21.15.90:0/2304307803 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564321c6d0d0 0x564321c88550 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:32.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.266+0000 7f023959c740 1 --2- 172.21.15.90:0/2304307803 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564321c6c7b0 0x564321c8b170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:32.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.266+0000 7f0219600640 0 client.27562 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:42:32.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.266+0000 7f023959c740 1 -- 172.21.15.90:0/2304307803 >> 172.21.15.90:0/2304307803 conn(0x564321c66ee0 msgr2=0x564321c65f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:42:32.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.267+0000 7f023959c740 1 -- 172.21.15.90:0/2304307803 shutdown_connections 2024-04-02T14:42:32.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.268+0000 7f023959c740 1 -- 172.21.15.90:0/2304307803 wait complete. 2024-04-02T14:42:32.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.269+0000 7f023959c740 20 client.27562 trim_cache size 0 max 16384 2024-04-02T14:42:32.267 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-02T14:42:32.267 INFO:teuthology.orchestra.run.smithi090.stdout: "status": "no active scrubs running", 2024-04-02T14:42:32.267 INFO:teuthology.orchestra.run.smithi090.stdout: "scrubs": {} 2024-04-02T14:42:32.267 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-02T14:42:32.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.270+0000 7f0236800640 1 -- 172.21.15.90:0/3141353420 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f02140a4000 msgr2=0x7f02140a64c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:32.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.270+0000 7f0236800640 1 --2- 172.21.15.90:0/3141353420 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f02140a4000 0x7f02140a64c0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x564321c6c280 tx=0x7f0220009620 comp rx=0 tx=0).stop 2024-04-02T14:42:32.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.270+0000 7f0236800640 1 -- 172.21.15.90:0/3141353420 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0230108fe0 msgr2=0x7f0230076550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:32.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.270+0000 7f0236800640 1 --2- 172.21.15.90:0/3141353420 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0230108fe0 0x7f0230076550 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f021c00b3c0 tx=0x7f021c00b880 comp rx=0 tx=0).stop 2024-04-02T14:42:32.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.271+0000 7f0236800640 1 -- 172.21.15.90:0/3141353420 shutdown_connections 2024-04-02T14:42:32.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.271+0000 7f0236800640 1 --2- 172.21.15.90:0/3141353420 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f02140a4000 0x7f02140a64c0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:32.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.271+0000 7f0236800640 1 --2- 172.21.15.90:0/3141353420 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0230076a90 0x7f02301aabc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:32.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.271+0000 7f0236800640 1 --2- 172.21.15.90:0/3141353420 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0230108fe0 0x7f0230076550 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:32.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.271+0000 7f0236800640 1 --2- 172.21.15.90:0/3141353420 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0230103f10 0x7f0230076010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:32.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.271+0000 7f0236800640 1 -- 172.21.15.90:0/3141353420 >> 172.21.15.90:0/3141353420 conn(0x7f02300ff750 msgr2=0x7f0230101b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:42:32.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.271+0000 7f0236800640 1 -- 172.21.15.90:0/3141353420 shutdown_connections 2024-04-02T14:42:32.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.271+0000 7f0236800640 1 -- 172.21.15.90:0/3141353420 wait complete. 2024-04-02T14:42:32.290 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:42:32 smithi090 ceph-mon[31674]: pgmap v234: 97 pgs: 97 active+clean; 4.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 28 MiB/s wr, 1.25k op/s 2024-04-02T14:42:32.295 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-02T14:42:32.296 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T14:42:32.358 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:42:32 smithi092 ceph-mon[36497]: pgmap v234: 97 pgs: 97 active+clean; 4.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 28 MiB/s wr, 1.25k op/s 2024-04-02T14:42:32.433 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:42:32 smithi177 ceph-mon[36431]: pgmap v234: 97 pgs: 97 active+clean; 4.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 28 MiB/s wr, 1.25k op/s 2024-04-02T14:42:32.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.588+0000 7f2cbc800640 1 -- 172.21.15.90:0/2394949253 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2cb4108fd0 msgr2=0x7f2cb4110ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:32.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.588+0000 7f2cbc800640 1 --2- 172.21.15.90:0/2394949253 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2cb4108fd0 0x7f2cb4110ba0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f2cb0009be0 tx=0x7f2cb002bef0 comp rx=0 tx=0).stop 2024-04-02T14:42:32.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.591+0000 7f2cbc800640 1 -- 172.21.15.90:0/2394949253 shutdown_connections 2024-04-02T14:42:32.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.591+0000 7f2cbc800640 1 --2- 172.21.15.90:0/2394949253 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2cb4108fd0 0x7f2cb4110ba0 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:32.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.591+0000 7f2cbc800640 1 --2- 172.21.15.90:0/2394949253 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2cb41048b0 0x7f2cb41088f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:32.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.591+0000 7f2cbc800640 1 --2- 172.21.15.90:0/2394949253 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2cb4103f00 0x7f2cb41042e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:32.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.591+0000 7f2cbc800640 1 -- 172.21.15.90:0/2394949253 >> 172.21.15.90:0/2394949253 conn(0x7f2cb40ff760 msgr2=0x7f2cb4101b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:42:32.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.591+0000 7f2cbc800640 1 -- 172.21.15.90:0/2394949253 shutdown_connections 2024-04-02T14:42:32.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.591+0000 7f2cbc800640 1 -- 172.21.15.90:0/2394949253 wait complete. 2024-04-02T14:42:32.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.592+0000 7f2cbc800640 1 Processor -- start 2024-04-02T14:42:32.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.593+0000 7f2cbc800640 1 -- start start 2024-04-02T14:42:32.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.594+0000 7f2cbc800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2cb4103f00 0x7f2cb419a490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:32.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.594+0000 7f2cbc800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2cb4108fd0 0x7f2cb419a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:32.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.594+0000 7f2cbc800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2cb419ed50 0x7f2cb419f200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:32.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.594+0000 7f2cbc800640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f2cb41140b0 con 0x7f2cb4108fd0 2024-04-02T14:42:32.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.594+0000 7f2cb9e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2cb4103f00 0x7f2cb419a490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:32.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.594+0000 7f2cb9e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2cb4103f00 0x7f2cb419a490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:57540/0 (socket says 172.21.15.90:57540) 2024-04-02T14:42:32.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.594+0000 7f2cb9e00640 1 -- 172.21.15.90:0/2730825616 learned_addr learned my addr 172.21.15.90:0/2730825616 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:42:32.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.595+0000 7f2cbc800640 1 -- 172.21.15.90:0/2730825616 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f2cb4113f30 con 0x7f2cb419ed50 2024-04-02T14:42:32.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.595+0000 7f2cbc800640 1 -- 172.21.15.90:0/2730825616 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f2cb4114230 con 0x7f2cb4103f00 2024-04-02T14:42:32.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.595+0000 7f2cb9400640 1 --2- 172.21.15.90:0/2730825616 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2cb4108fd0 0x7f2cb419a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:32.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.595+0000 7f2cb9e00640 1 -- 172.21.15.90:0/2730825616 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2cb419ed50 msgr2=0x7f2cb419f200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:32.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.595+0000 7f2cb9e00640 1 --2- 172.21.15.90:0/2730825616 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2cb419ed50 0x7f2cb419f200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:32.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.595+0000 7f2cb9e00640 1 -- 172.21.15.90:0/2730825616 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2cb4108fd0 msgr2=0x7f2cb419a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:32.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.595+0000 7f2cb9e00640 1 --2- 172.21.15.90:0/2730825616 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2cb4108fd0 0x7f2cb419a9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:32.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.596+0000 7f2cb9e00640 1 -- 172.21.15.90:0/2730825616 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2cb419f840 con 0x7f2cb4103f00 2024-04-02T14:42:32.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.596+0000 7f2cb9e00640 1 --2- 172.21.15.90:0/2730825616 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2cb4103f00 0x7f2cb419a490 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f2ca800c920 tx=0x7f2ca800cde0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:42:32.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.598+0000 7f2caea00640 1 -- 172.21.15.90:0/2730825616 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ca8007c70 con 0x7f2cb4103f00 2024-04-02T14:42:32.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.598+0000 7f2cbc800640 1 -- 172.21.15.90:0/2730825616 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2cb419fb30 con 0x7f2cb4103f00 2024-04-02T14:42:32.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.598+0000 7f2cbc800640 1 -- 172.21.15.90:0/2730825616 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2cb41a7320 con 0x7f2cb4103f00 2024-04-02T14:42:32.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.598+0000 7f2caea00640 1 -- 172.21.15.90:0/2730825616 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2ca8007e10 con 0x7f2cb4103f00 2024-04-02T14:42:32.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.598+0000 7f2caea00640 1 -- 172.21.15.90:0/2730825616 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ca80188b0 con 0x7f2cb4103f00 2024-04-02T14:42:32.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.602+0000 7f2cbc800640 1 -- 172.21.15.90:0/2730825616 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2c98005290 con 0x7f2cb4103f00 2024-04-02T14:42:32.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.606+0000 7f2caea00640 1 -- 172.21.15.90:0/2730825616 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f2ca8018b70 con 0x7f2cb4103f00 2024-04-02T14:42:32.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.609+0000 7f2caea00640 1 --2- 172.21.15.90:0/2730825616 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f2c9c0a3ef0 0x7f2c9c0a63b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:32.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.609+0000 7f2caea00640 1 -- 172.21.15.90:0/2730825616 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8550+0+0 (secure 0 0 0) 0x7f2ca80d0880 con 0x7f2cb4103f00 2024-04-02T14:42:32.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.610+0000 7f2cb9400640 1 --2- 172.21.15.90:0/2730825616 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f2c9c0a3ef0 0x7f2c9c0a63b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:32.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.611+0000 7f2cb9400640 1 --2- 172.21.15.90:0/2730825616 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f2c9c0a3ef0 0x7f2c9c0a63b0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f2ca40096c0 tx=0x7f2ca4009260 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:42:32.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:32.612+0000 7f2caea00640 1 -- 172.21.15.90:0/2730825616 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7f2ca8014030 con 0x7f2cb4103f00 2024-04-02T14:42:33.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.059+0000 7f2cbc800640 1 -- 172.21.15.90:0/2730825616 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2c98005850 con 0x7f2cb4103f00 2024-04-02T14:42:33.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.060+0000 7f2caea00640 1 -- 172.21.15.90:0/2730825616 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11927 (secure 0 0 0) 0x7f2ca801f300 con 0x7f2cb4103f00 2024-04-02T14:42:33.058 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-02T14:42:33.058 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14412,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6837/2852160309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6836","nonce":2852160309},{"type":"v1","addr":"172.21.15.90:6837","nonce":2852160309}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14478,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6841/1919376858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6840","nonce":1919376858},{"type":"v1","addr":"172.21.15.90:6841","nonce":1919376858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-02T14:32:22.319083+0000","modified":"2024-04-02T14:34:52.266152+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24590,"mds_1":24560,"mds_2":24547,"mds_3":24539,"mds_4":24523},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24454":{"gid":24454,"name":"l","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6833/1564319431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":1564319431},{"type":"v1","addr":"172.21.15.177:6833","nonce":1564319431}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.92:6833/1570337372","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":1570337372},{"type":"v1","addr":"172.21.15.92:6833","nonce":1570337372}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"j","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.90:6835/1277124891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6834","nonce":1277124891},{"type":"v1","addr":"172.21.15.90:6835","nonce":1277124891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.92:6835/860721164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":860721164},{"type":"v1","addr":"172.21.15.92:6835","nonce":860721164}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24511":{"gid":24511,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6835/3177244988","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":3177244988},{"type":"v1","addr":"172.21.15.177:6835","nonce":3177244988}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"c","rank":4,"incarnation":25,"state":"up:active","state_seq":9,"addr":"172.21.15.177:6837/4223414858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":4223414858},{"type":"v1","addr":"172.21.15.177:6837","nonce":4223414858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24539":{"gid":24539,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":8,"addr":"172.21.15.92:6837/1270635939","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":1270635939},{"type":"v1","addr":"172.21.15.92:6837","nonce":1270635939}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24547":{"gid":24547,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":7,"addr":"172.21.15.177:6839/664895663","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":664895663},{"type":"v1","addr":"172.21.15.177:6839","nonce":664895663}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"g","rank":1,"incarnation":19,"state":"up:active","state_seq":5,"addr":"172.21.15.90:6839/1498915870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6838","nonce":1498915870},{"type":"v1","addr":"172.21.15.90:6839","nonce":1498915870}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24590":{"gid":24590,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.92:6839/1054888219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":1054888219},{"type":"v1","addr":"172.21.15.92:6839","nonce":1054888219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24590,"qdb_cluster":[24547,24590,24523,24539,24560]},"id":1}]} 2024-04-02T14:42:33.058 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 27 2024-04-02T14:42:33.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.063+0000 7f2cbc800640 1 -- 172.21.15.90:0/2730825616 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f2c9c0a3ef0 msgr2=0x7f2c9c0a63b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:33.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.063+0000 7f2cbc800640 1 --2- 172.21.15.90:0/2730825616 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f2c9c0a3ef0 0x7f2c9c0a63b0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f2ca40096c0 tx=0x7f2ca4009260 comp rx=0 tx=0).stop 2024-04-02T14:42:33.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.063+0000 7f2cbc800640 1 -- 172.21.15.90:0/2730825616 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2cb4103f00 msgr2=0x7f2cb419a490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:33.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.063+0000 7f2cbc800640 1 --2- 172.21.15.90:0/2730825616 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2cb4103f00 0x7f2cb419a490 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f2ca800c920 tx=0x7f2ca800cde0 comp rx=0 tx=0).stop 2024-04-02T14:42:33.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.064+0000 7f2cbc800640 1 -- 172.21.15.90:0/2730825616 shutdown_connections 2024-04-02T14:42:33.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.064+0000 7f2cbc800640 1 --2- 172.21.15.90:0/2730825616 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f2c9c0a3ef0 0x7f2c9c0a63b0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:33.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.064+0000 7f2cbc800640 1 --2- 172.21.15.90:0/2730825616 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2cb419ed50 0x7f2cb419f200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:33.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.064+0000 7f2cbc800640 1 --2- 172.21.15.90:0/2730825616 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2cb4108fd0 0x7f2cb419a9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:33.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.064+0000 7f2cbc800640 1 --2- 172.21.15.90:0/2730825616 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2cb4103f00 0x7f2cb419a490 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:33.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.064+0000 7f2cbc800640 1 -- 172.21.15.90:0/2730825616 >> 172.21.15.90:0/2730825616 conn(0x7f2cb40ff760 msgr2=0x7f2cb4100260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:42:33.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.064+0000 7f2cbc800640 1 -- 172.21.15.90:0/2730825616 shutdown_connections 2024-04-02T14:42:33.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.066+0000 7f2cbc800640 1 -- 172.21.15.90:0/2730825616 wait complete. 2024-04-02T14:42:33.089 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-04-02T14:42:33.357 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:42:33 smithi092 ceph-mon[36497]: from='client.? 172.21.15.90:0/2730825616' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-02T14:42:33.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.369+0000 7ff82e400640 1 -- 172.21.15.90:0/2120559242 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff82810e1a0 msgr2=0x7ff8281105f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:33.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.369+0000 7ff82e400640 1 --2- 172.21.15.90:0/2120559242 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff82810e1a0 0x7ff8281105f0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7ff820009900 tx=0x7ff82002bcc0 comp rx=0 tx=0).stop 2024-04-02T14:42:33.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.369+0000 7ff82e400640 1 -- 172.21.15.90:0/2120559242 shutdown_connections 2024-04-02T14:42:33.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.369+0000 7ff82e400640 1 --2- 172.21.15.90:0/2120559242 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff82810e1a0 0x7ff8281105f0 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:33.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.369+0000 7ff82e400640 1 --2- 172.21.15.90:0/2120559242 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff828074850 0x7ff82810dc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:33.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.369+0000 7ff82e400640 1 --2- 172.21.15.90:0/2120559242 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff828073f30 0x7ff828074310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:33.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.369+0000 7ff82e400640 1 -- 172.21.15.90:0/2120559242 >> 172.21.15.90:0/2120559242 conn(0x7ff8280fd470 msgr2=0x7ff8280ff890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:42:33.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.369+0000 7ff82e400640 1 -- 172.21.15.90:0/2120559242 shutdown_connections 2024-04-02T14:42:33.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.369+0000 7ff82e400640 1 -- 172.21.15.90:0/2120559242 wait complete. 2024-04-02T14:42:33.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.371+0000 7ff82e400640 1 Processor -- start 2024-04-02T14:42:33.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.372+0000 7ff82e400640 1 -- start start 2024-04-02T14:42:33.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.373+0000 7ff82e400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff828073f30 0x7ff82810c1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:33.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.373+0000 7ff82e400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff828074850 0x7ff82810c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:33.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.373+0000 7ff82e400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff82810e1a0 0x7ff8281aa730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:33.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.374+0000 7ff82d000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff828073f30 0x7ff82810c1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:33.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.374+0000 7ff82d000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff828073f30 0x7ff82810c1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:58320/0 (socket says 172.21.15.90:58320) 2024-04-02T14:42:33.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.374+0000 7ff82d000640 1 -- 172.21.15.90:0/4189709682 learned_addr learned my addr 172.21.15.90:0/4189709682 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:42:33.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.374+0000 7ff81fe00640 1 --2- 172.21.15.90:0/4189709682 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff828074850 0x7ff82810c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:33.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.374+0000 7ff82e400640 1 -- 172.21.15.90:0/4189709682 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7ff828102590 con 0x7ff82810e1a0 2024-04-02T14:42:33.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.374+0000 7ff82e400640 1 -- 172.21.15.90:0/4189709682 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7ff828102410 con 0x7ff828073f30 2024-04-02T14:42:33.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.374+0000 7ff82e400640 1 -- 172.21.15.90:0/4189709682 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7ff828102710 con 0x7ff828074850 2024-04-02T14:42:33.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.374+0000 7ff82da00640 1 --2- 172.21.15.90:0/4189709682 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff82810e1a0 0x7ff8281aa730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:33.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.375+0000 7ff82d000640 1 -- 172.21.15.90:0/4189709682 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff828074850 msgr2=0x7ff82810c6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:33.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.375+0000 7ff82d000640 1 --2- 172.21.15.90:0/4189709682 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff828074850 0x7ff82810c6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:33.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.375+0000 7ff82d000640 1 -- 172.21.15.90:0/4189709682 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff82810e1a0 msgr2=0x7ff8281aa730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:33.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.375+0000 7ff82d000640 1 --2- 172.21.15.90:0/4189709682 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff82810e1a0 0x7ff8281aa730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:33.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.375+0000 7ff82d000640 1 -- 172.21.15.90:0/4189709682 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8281aac70 con 0x7ff828073f30 2024-04-02T14:42:33.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.375+0000 7ff82d000640 1 --2- 172.21.15.90:0/4189709682 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff828073f30 0x7ff82810c1b0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7ff81800e9f0 tx=0x7ff81800edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:42:33.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.375+0000 7ff826000640 1 -- 172.21.15.90:0/4189709682 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff81800cd80 con 0x7ff828073f30 2024-04-02T14:42:33.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.376+0000 7ff82e400640 1 -- 172.21.15.90:0/4189709682 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff8281aae40 con 0x7ff828073f30 2024-04-02T14:42:33.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.376+0000 7ff826000640 1 -- 172.21.15.90:0/4189709682 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff818005500 con 0x7ff828073f30 2024-04-02T14:42:33.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.376+0000 7ff826000640 1 -- 172.21.15.90:0/4189709682 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff818010690 con 0x7ff828073f30 2024-04-02T14:42:33.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.376+0000 7ff82e400640 1 -- 172.21.15.90:0/4189709682 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8281ab300 con 0x7ff828073f30 2024-04-02T14:42:33.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.380+0000 7ff826000640 1 -- 172.21.15.90:0/4189709682 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7ff818010830 con 0x7ff828073f30 2024-04-02T14:42:33.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.381+0000 7ff826000640 1 --2- 172.21.15.90:0/4189709682 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7ff80c0a3de0 0x7ff80c0a62a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:33.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.381+0000 7ff832655740 1 Processor -- start 2024-04-02T14:42:33.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.382+0000 7ff826000640 1 -- 172.21.15.90:0/4189709682 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8550+0+0 (secure 0 0 0) 0x7ff8180d0c60 con 0x7ff828073f30 2024-04-02T14:42:33.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.382+0000 7ff81fe00640 1 --2- 172.21.15.90:0/4189709682 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7ff80c0a3de0 0x7ff80c0a62a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:33.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.383+0000 7ff832655740 1 -- start start 2024-04-02T14:42:33.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.383+0000 7ff832655740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55db1ccb8ba0 0x55db1ccb78f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:33.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.383+0000 7ff832655740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55db1ccb7ec0 0x55db1ccb8340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:33.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.383+0000 7ff832655740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55db1ccc1230 0x55db1ccc3620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:33.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.383+0000 7ff81fe00640 1 --2- 172.21.15.90:0/4189709682 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7ff80c0a3de0 0x7ff80c0a62a0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7ff82810dbf0 tx=0x7ff814009500 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:42:33.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.384+0000 7ff832655740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55db1cbf8400 con 0x55db1ccc1230 2024-04-02T14:42:33.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.384+0000 7ff832655740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55db1cbdcf10 con 0x55db1ccb8ba0 2024-04-02T14:42:33.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.384+0000 7ff832655740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55db1cc1c6f0 con 0x55db1ccb7ec0 2024-04-02T14:42:33.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.384+0000 7ff81fe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55db1ccb7ec0 0x55db1ccb8340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:33.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.384+0000 7ff82d000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55db1ccb8ba0 0x55db1ccb78f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:33.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.384+0000 7ff81fe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55db1ccb7ec0 0x55db1ccb8340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:57572/0 (socket says 172.21.15.90:57572) 2024-04-02T14:42:33.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.384+0000 7ff81fe00640 1 -- 172.21.15.90:0/3167098878 learned_addr learned my addr 172.21.15.90:0/3167098878 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:42:33.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.384+0000 7ff82da00640 1 --2- 172.21.15.90:0/3167098878 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55db1ccc1230 0x55db1ccc3620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:33.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.384+0000 7ff82d000640 1 -- 172.21.15.90:0/3167098878 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55db1ccb7ec0 msgr2=0x55db1ccb8340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:42:33.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.384+0000 7ff82d000640 1 --2- 172.21.15.90:0/3167098878 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55db1ccb7ec0 0x55db1ccb8340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:33.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.384+0000 7ff82d000640 1 -- 172.21.15.90:0/3167098878 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55db1ccc1230 msgr2=0x55db1ccc3620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:42:33.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.384+0000 7ff82d000640 1 --2- 172.21.15.90:0/3167098878 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55db1ccc1230 0x55db1ccc3620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:33.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.384+0000 7ff82d000640 1 -- 172.21.15.90:0/3167098878 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55db1c9303f0 con 0x55db1ccb8ba0 2024-04-02T14:42:33.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.385+0000 7ff82d000640 1 --2- 172.21.15.90:0/3167098878 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55db1ccb8ba0 0x55db1ccb78f0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7ff8180ce390 tx=0x7ff81809c7e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=815f4184a4120e0c server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:42:33.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.385+0000 7ff81e000640 1 -- 172.21.15.90:0/3167098878 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8180b6070 con 0x55db1ccb8ba0 2024-04-02T14:42:33.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.385+0000 7ff81e000640 1 -- 172.21.15.90:0/3167098878 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff8180ce610 con 0x55db1ccb8ba0 2024-04-02T14:42:33.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.385+0000 7ff81e000640 1 -- 172.21.15.90:0/3167098878 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8180ced40 con 0x55db1ccb8ba0 2024-04-02T14:42:33.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.386+0000 7ff832655740 1 -- 172.21.15.90:0/3167098878 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55db1ccb8ba0 msgr2=0x55db1ccb78f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:33.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.386+0000 7ff832655740 1 --2- 172.21.15.90:0/3167098878 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55db1ccb8ba0 0x55db1ccb78f0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7ff8180ce390 tx=0x7ff81809c7e0 comp rx=0 tx=0).stop 2024-04-02T14:42:33.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.386+0000 7ff832655740 1 -- 172.21.15.90:0/3167098878 shutdown_connections 2024-04-02T14:42:33.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.386+0000 7ff832655740 1 --2- 172.21.15.90:0/3167098878 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55db1ccc1230 0x55db1ccc3620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:33.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.386+0000 7ff832655740 1 --2- 172.21.15.90:0/3167098878 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55db1ccb7ec0 0x55db1ccb8340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:33.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.386+0000 7ff832655740 1 --2- 172.21.15.90:0/3167098878 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55db1ccb8ba0 0x55db1ccb78f0 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:33.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.386+0000 7ff832655740 1 -- 172.21.15.90:0/3167098878 >> 172.21.15.90:0/3167098878 conn(0x55db1cc7ea50 msgr2=0x55db1ccb2910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:42:33.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.386+0000 7ff832655740 1 -- 172.21.15.90:0/3167098878 shutdown_connections 2024-04-02T14:42:33.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.386+0000 7ff832655740 1 -- 172.21.15.90:0/3167098878 wait complete. 2024-04-02T14:42:33.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.387+0000 7ff832655740 1 -- start start 2024-04-02T14:42:33.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.388+0000 7ff832655740 1 Processor -- start 2024-04-02T14:42:33.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.388+0000 7ff832655740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55db1ccb7ec0 0x55db1ccbe240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:33.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.388+0000 7ff832655740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55db1ccb8ba0 0x55db1ccbe780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:33.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.388+0000 7ff832655740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55db1ccc1230 0x55db1ccbecc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:33.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.388+0000 7ff832655740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55db1cc186a0 con 0x55db1ccb8ba0 2024-04-02T14:42:33.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.389+0000 7ff832655740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55db1cbc4980 con 0x55db1ccb7ec0 2024-04-02T14:42:33.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.389+0000 7ff832655740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55db1cbd7cb0 con 0x55db1ccc1230 2024-04-02T14:42:33.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.389+0000 7ff832655740 10 client.0 _ll_register_callbacks cb 0x55db1cbd76a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:42:33.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.389+0000 7ff82d000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55db1ccb7ec0 0x55db1ccbe240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:33.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.389+0000 7ff82d000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55db1ccb7ec0 0x55db1ccbe240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:58332/0 (socket says 172.21.15.90:58332) 2024-04-02T14:42:33.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.389+0000 7ff82d000640 1 -- 172.21.15.90:0/2388639508 learned_addr learned my addr 172.21.15.90:0/2388639508 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:42:33.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.389+0000 7ff82da00640 1 --2- 172.21.15.90:0/2388639508 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55db1ccc1230 0x55db1ccbecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:33.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.390+0000 7ff81fe00640 1 --2- 172.21.15.90:0/2388639508 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55db1ccb8ba0 0x55db1ccbe780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:33.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.390+0000 7ff82d000640 1 -- 172.21.15.90:0/2388639508 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55db1ccc1230 msgr2=0x55db1ccbecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:42:33.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.390+0000 7ff82d000640 1 --2- 172.21.15.90:0/2388639508 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55db1ccc1230 0x55db1ccbecc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:33.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.390+0000 7ff82d000640 1 -- 172.21.15.90:0/2388639508 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55db1ccb8ba0 msgr2=0x55db1ccbe780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:42:33.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.390+0000 7ff82d000640 1 --2- 172.21.15.90:0/2388639508 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55db1ccb8ba0 0x55db1ccbe780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:33.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.390+0000 7ff82d000640 1 -- 172.21.15.90:0/2388639508 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55db1c9fa7d0 con 0x55db1ccb7ec0 2024-04-02T14:42:33.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.390+0000 7ff81fe00640 1 --2- 172.21.15.90:0/2388639508 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55db1ccb8ba0 0x55db1ccbe780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:42:33.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.390+0000 7ff82d000640 1 --2- 172.21.15.90:0/2388639508 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55db1ccb7ec0 0x55db1ccbe240 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7ff81800e9c0 tx=0x7ff818010d50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=52b92b37aaa53ffe server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:42:33.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.390+0000 7ff812a00640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:42:33.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.390+0000 7ff812a00640 1 -- 172.21.15.90:0/2388639508 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8180bb020 con 0x55db1ccb7ec0 2024-04-02T14:42:33.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.390+0000 7ff812a00640 1 -- 172.21.15.90:0/2388639508 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8550+0+0 (secure 0 0 0) 0x7ff8180cecf0 con 0x55db1ccb7ec0 2024-04-02T14:42:33.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.390+0000 7ff832655740 1 -- 172.21.15.90:0/2388639508 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff8180ce610 con 0x55db1ccb7ec0 2024-04-02T14:42:33.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.391+0000 7ff812a00640 1 -- 172.21.15.90:0/2388639508 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7ff8180b6040 con 0x55db1ccb7ec0 2024-04-02T14:42:33.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.391+0000 7ff832655740 10 client.27713 fetch_fsmap learned FSMap version 27 2024-04-02T14:42:33.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.391+0000 7ff832655740 1 -- 172.21.15.90:0/2388639508 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55db1cce10f0 con 0x55db1ccb7ec0 2024-04-02T14:42:33.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.391+0000 7ff812a00640 1 -- 172.21.15.90:0/2388639508 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff8180ce610 con 0x55db1ccb7ec0 2024-04-02T14:42:33.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.391+0000 7ff812a00640 1 -- 172.21.15.90:0/2388639508 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7ff8180a23f0 con 0x55db1ccb7ec0 2024-04-02T14:42:33.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.392+0000 7ff832655740 10 client.27713 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:42:33.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.392+0000 7ff832655740 10 client.27713 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.c 2024-04-02T14:42:33.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.392+0000 7ff832655740 20 client.27713 populate_metadata read hostname 'smithi090' 2024-04-02T14:42:33.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.392+0000 7ff832655740 1 --2- 172.21.15.90:0/2388639508 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x55db1cce2440 0x55db1cce4830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:33.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.392+0000 7ff832655740 4 client.27713 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:42:33.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.392+0000 7ff832655740 1 -- 172.21.15.90:0/2388639508 --> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55db1cbdc040 con 0x55db1cce2440 2024-04-02T14:42:33.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.393+0000 7ff81fe00640 1 --2- 172.21.15.90:0/2388639508 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x55db1cce2440 0x55db1cce4830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:33.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.394+0000 7ff81fe00640 1 --2- 172.21.15.90:0/2388639508 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x55db1cce2440 0x55db1cce4830 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=834db84139c2d804 server_cookie=2aa7004349690b6a in_seq=0 out_seq=0 2024-04-02T14:42:33.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.394+0000 7ff812a00640 10 client.27713 ms_handle_connect on v2:172.21.15.177:6836/4223414858 2024-04-02T14:42:33.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.395+0000 7ff812a00640 1 -- 172.21.15.90:0/2388639508 <== mds.4 v2:172.21.15.177:6836/4223414858 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55db1cbdc040 con 0x55db1cce2440 2024-04-02T14:42:33.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.395+0000 7ff812a00640 10 client.27713 handle_command_reply: tid=0 2024-04-02T14:42:33.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.395+0000 7ff832655740 1 client.27713 shutdown 2024-04-02T14:42:33.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.395+0000 7ff832655740 1 -- 172.21.15.90:0/2388639508 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55db1ccb7ec0 msgr2=0x55db1ccbe240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:33.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.395+0000 7ff832655740 1 --2- 172.21.15.90:0/2388639508 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55db1ccb7ec0 0x55db1ccbe240 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7ff81800e9c0 tx=0x7ff818010d50 comp rx=0 tx=0).stop 2024-04-02T14:42:33.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.395+0000 7ff832655740 1 -- 172.21.15.90:0/2388639508 shutdown_connections 2024-04-02T14:42:33.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.395+0000 7ff832655740 1 --2- 172.21.15.90:0/2388639508 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55db1ccc1230 0x55db1ccbecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:33.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.395+0000 7ff832655740 1 --2- 172.21.15.90:0/2388639508 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x55db1cce2440 0x55db1cce4830 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:33.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.395+0000 7ff832655740 1 --2- 172.21.15.90:0/2388639508 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55db1ccb8ba0 0x55db1ccbe780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:33.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.395+0000 7ff832655740 1 --2- 172.21.15.90:0/2388639508 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55db1ccb7ec0 0x55db1ccbe240 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:33.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.395+0000 7ff812a00640 0 client.27713 ms_handle_reset on v2:172.21.15.177:6836/4223414858 2024-04-02T14:42:33.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.395+0000 7ff832655740 1 -- 172.21.15.90:0/2388639508 >> 172.21.15.90:0/2388639508 conn(0x55db1cc7ea50 msgr2=0x55db1ccb4660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:42:33.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.395+0000 7ff832655740 1 -- 172.21.15.90:0/2388639508 shutdown_connections 2024-04-02T14:42:33.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.395+0000 7ff832655740 1 -- 172.21.15.90:0/2388639508 wait complete. 2024-04-02T14:42:33.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.395+0000 7ff832655740 20 client.27713 trim_cache size 0 max 16384 2024-04-02T14:42:33.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.419+0000 7ff832655740 1 Processor -- start 2024-04-02T14:42:33.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.420+0000 7ff832655740 1 -- start start 2024-04-02T14:42:33.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.420+0000 7ff832655740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55db1ccb7ec0 0x55db1ccc0970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:33.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.421+0000 7ff832655740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55db1ccb8ba0 0x55db1cccc8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:33.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.421+0000 7ff832655740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55db1ccc1230 0x55db1ccc2620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:33.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.421+0000 7ff832655740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7ff8281ab590 con 0x55db1ccb7ec0 2024-04-02T14:42:33.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.421+0000 7ff832655740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7ff8281acfe0 con 0x55db1ccb8ba0 2024-04-02T14:42:33.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.421+0000 7ff832655740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7ff8281abb70 con 0x55db1ccc1230 2024-04-02T14:42:33.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.421+0000 7ff81fe00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55db1ccb8ba0 0x55db1cccc8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:33.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.421+0000 7ff82d000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55db1ccb7ec0 0x55db1ccc0970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:33.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.421+0000 7ff81fe00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55db1ccb8ba0 0x55db1cccc8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:58336/0 (socket says 172.21.15.90:58336) 2024-04-02T14:42:33.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.421+0000 7ff81fe00640 1 -- 172.21.15.90:0/1012863117 learned_addr learned my addr 172.21.15.90:0/1012863117 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:42:33.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.421+0000 7ff82da00640 1 --2- 172.21.15.90:0/1012863117 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55db1ccc1230 0x55db1ccc2620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:33.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.421+0000 7ff82d000640 1 -- 172.21.15.90:0/1012863117 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55db1ccc1230 msgr2=0x55db1ccc2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:42:33.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.422+0000 7ff82d000640 1 --2- 172.21.15.90:0/1012863117 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55db1ccc1230 0x55db1ccc2620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:33.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.422+0000 7ff82d000640 1 -- 172.21.15.90:0/1012863117 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55db1ccb8ba0 msgr2=0x55db1cccc8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:42:33.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.422+0000 7ff82d000640 1 --2- 172.21.15.90:0/1012863117 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55db1ccb8ba0 0x55db1cccc8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:33.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.422+0000 7ff82d000640 1 -- 172.21.15.90:0/1012863117 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55db1ccb6ef0 con 0x55db1ccb7ec0 2024-04-02T14:42:33.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.422+0000 7ff82da00640 1 --2- 172.21.15.90:0/1012863117 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55db1ccc1230 0x55db1ccc2620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:42:33.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.422+0000 7ff82d000640 1 --2- 172.21.15.90:0/1012863117 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55db1ccb7ec0 0x55db1ccc0970 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7ff818099850 tx=0x7ff81809c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6eeeab7032b99868 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:42:33.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.422+0000 7ff81d600640 1 -- 172.21.15.90:0/1012863117 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8180bb070 con 0x55db1ccb7ec0 2024-04-02T14:42:33.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.423+0000 7ff81d600640 1 -- 172.21.15.90:0/1012863117 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff81809af20 con 0x55db1ccb7ec0 2024-04-02T14:42:33.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.423+0000 7ff81d600640 1 -- 172.21.15.90:0/1012863117 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8180b1040 con 0x55db1ccb7ec0 2024-04-02T14:42:33.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.423+0000 7ff832655740 1 -- 172.21.15.90:0/1012863117 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55db1ccb7ec0 msgr2=0x55db1ccc0970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:33.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.423+0000 7ff832655740 1 --2- 172.21.15.90:0/1012863117 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55db1ccb7ec0 0x55db1ccc0970 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7ff818099850 tx=0x7ff81809c7e0 comp rx=0 tx=0).stop 2024-04-02T14:42:33.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.424+0000 7ff832655740 1 -- 172.21.15.90:0/1012863117 shutdown_connections 2024-04-02T14:42:33.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.424+0000 7ff832655740 1 --2- 172.21.15.90:0/1012863117 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55db1ccc1230 0x55db1ccc2620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:33.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.425+0000 7ff832655740 1 --2- 172.21.15.90:0/1012863117 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55db1ccb8ba0 0x55db1cccc8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:33.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.425+0000 7ff832655740 1 --2- 172.21.15.90:0/1012863117 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55db1ccb7ec0 0x55db1ccc0970 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:33.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.425+0000 7ff832655740 1 -- 172.21.15.90:0/1012863117 >> 172.21.15.90:0/1012863117 conn(0x55db1cc7ea50 msgr2=0x55db1cccc1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:42:33.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.425+0000 7ff832655740 1 -- 172.21.15.90:0/1012863117 shutdown_connections 2024-04-02T14:42:33.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.426+0000 7ff832655740 1 -- 172.21.15.90:0/1012863117 wait complete. 2024-04-02T14:42:33.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.427+0000 7ff832655740 1 -- start start 2024-04-02T14:42:33.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.428+0000 7ff832655740 1 Processor -- start 2024-04-02T14:42:33.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.429+0000 7ff832655740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55db1ccb7ec0 0x55db1cce7a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:33.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.429+0000 7ff832655740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55db1ccb8ba0 0x55db1cce2e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:33.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.429+0000 7ff832655740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55db1ccc1230 0x55db1cce3360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:33.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.429+0000 7ff832655740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7ff8281ac910 con 0x55db1ccb8ba0 2024-04-02T14:42:33.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.429+0000 7ff832655740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7ff8281ac240 con 0x55db1ccc1230 2024-04-02T14:42:33.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.429+0000 7ff832655740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7ff828071c40 con 0x55db1ccb7ec0 2024-04-02T14:42:33.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.429+0000 7ff832655740 10 client.0 _ll_register_callbacks cb 0x7ff81809cd40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:42:33.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.430+0000 7ff82da00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55db1ccc1230 0x55db1cce3360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:33.429 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.430+0000 7ff82da00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55db1ccc1230 0x55db1cce3360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:58348/0 (socket says 172.21.15.90:58348) 2024-04-02T14:42:33.429 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.430+0000 7ff82da00640 1 -- 172.21.15.90:0/80276054 learned_addr learned my addr 172.21.15.90:0/80276054 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:42:33.429 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.430+0000 7ff81fe00640 1 --2- 172.21.15.90:0/80276054 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55db1ccb8ba0 0x55db1cce2e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:33.429 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.431+0000 7ff82da00640 1 -- 172.21.15.90:0/80276054 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55db1ccb7ec0 msgr2=0x55db1cce7a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:42:33.429 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.431+0000 7ff82da00640 1 --2- 172.21.15.90:0/80276054 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55db1ccb7ec0 0x55db1cce7a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:33.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.431+0000 7ff82da00640 1 -- 172.21.15.90:0/80276054 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55db1ccb8ba0 msgr2=0x55db1cce2e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:42:33.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.431+0000 7ff82da00640 1 --2- 172.21.15.90:0/80276054 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55db1ccb8ba0 0x55db1cce2e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:33.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.431+0000 7ff82da00640 1 -- 172.21.15.90:0/80276054 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55db1cce39a0 con 0x55db1ccc1230 2024-04-02T14:42:33.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.431+0000 7ff81fe00640 1 --2- 172.21.15.90:0/80276054 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55db1ccb8ba0 0x55db1cce2e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:42:33.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.432+0000 7ff812a00640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:42:33.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.432+0000 7ff82da00640 1 --2- 172.21.15.90:0/80276054 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55db1ccc1230 0x55db1cce3360 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7ff820005ab0 tx=0x7ff820004560 comp rx=0 tx=0).ready entity=mon.2 client_cookie=81304f8110a84e59 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:42:33.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.432+0000 7ff812a00640 1 -- 172.21.15.90:0/80276054 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8200046c0 con 0x55db1ccc1230 2024-04-02T14:42:33.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.432+0000 7ff812a00640 1 -- 172.21.15.90:0/80276054 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8550+0+0 (secure 0 0 0) 0x7ff820004860 con 0x55db1ccc1230 2024-04-02T14:42:33.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.432+0000 7ff832655740 1 -- 172.21.15.90:0/80276054 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff81809af20 con 0x55db1ccc1230 2024-04-02T14:42:33.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.433+0000 7ff812a00640 1 -- 172.21.15.90:0/80276054 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7ff820004a60 con 0x55db1ccc1230 2024-04-02T14:42:33.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.433+0000 7ff832655740 10 client.27719 fetch_fsmap learned FSMap version 27 2024-04-02T14:42:33.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.433+0000 7ff832655740 1 -- 172.21.15.90:0/80276054 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55db1cd48440 con 0x55db1ccc1230 2024-04-02T14:42:33.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.434+0000 7ff812a00640 1 -- 172.21.15.90:0/80276054 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff81809af20 con 0x55db1ccc1230 2024-04-02T14:42:33.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.434+0000 7ff812a00640 1 -- 172.21.15.90:0/80276054 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7ff8200038f0 con 0x55db1ccc1230 2024-04-02T14:42:33.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.434+0000 7ff832655740 10 client.27719 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:42:33.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.434+0000 7ff832655740 10 client.27719 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.c 2024-04-02T14:42:33.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.435+0000 7ff832655740 20 client.27719 populate_metadata read hostname 'smithi090' 2024-04-02T14:42:33.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.435+0000 7ff832655740 1 --2- 172.21.15.90:0/80276054 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x55db1cd48bf0 0x55db1cd4afe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:33.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.435+0000 7ff832655740 4 client.27719 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-02T14:42:33.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.435+0000 7ff832655740 1 -- 172.21.15.90:0/80276054 --> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55db1cc29c30 con 0x55db1cd48bf0 2024-04-02T14:42:33.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.435+0000 7ff82d000640 1 --2- 172.21.15.90:0/80276054 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x55db1cd48bf0 0x55db1cd4afe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:33.434 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:42:33 smithi177 ceph-mon[36431]: from='client.? 172.21.15.90:0/2730825616' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-02T14:42:33.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.436+0000 7ff82d000640 1 --2- 172.21.15.90:0/80276054 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x55db1cd48bf0 0x55db1cd4afe0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=362bf1a9a55e525c server_cookie=3a639f80db6bdd7b in_seq=0 out_seq=0 2024-04-02T14:42:33.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.436+0000 7ff812a00640 10 client.27719 ms_handle_connect on v2:172.21.15.177:6836/4223414858 2024-04-02T14:42:33.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.436+0000 7ff812a00640 1 -- 172.21.15.90:0/80276054 <== mds.4 v2:172.21.15.177:6836/4223414858 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55db1cc29c30 con 0x55db1cd48bf0 2024-04-02T14:42:33.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.436+0000 7ff812a00640 10 client.27719 handle_command_reply: tid=0 2024-04-02T14:42:33.437 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:42:33 smithi090 ceph-mon[31674]: from='client.? 172.21.15.90:0/2730825616' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-02T14:42:33.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.436+0000 7ff832655740 1 client.27719 shutdown 2024-04-02T14:42:33.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.443+0000 7ff832655740 1 -- 172.21.15.90:0/80276054 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55db1ccc1230 msgr2=0x55db1cce3360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:33.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.443+0000 7ff832655740 1 --2- 172.21.15.90:0/80276054 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55db1ccc1230 0x55db1cce3360 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7ff820005ab0 tx=0x7ff820004560 comp rx=0 tx=0).stop 2024-04-02T14:42:33.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.444+0000 7ff832655740 1 -- 172.21.15.90:0/80276054 shutdown_connections 2024-04-02T14:42:33.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.444+0000 7ff832655740 1 --2- 172.21.15.90:0/80276054 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55db1ccc1230 0x55db1cce3360 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:33.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.444+0000 7ff832655740 1 --2- 172.21.15.90:0/80276054 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x55db1cd48bf0 0x55db1cd4afe0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:33.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.444+0000 7ff832655740 1 --2- 172.21.15.90:0/80276054 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55db1ccb8ba0 0x55db1cce2e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:33.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.444+0000 7ff832655740 1 --2- 172.21.15.90:0/80276054 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55db1ccb7ec0 0x55db1cce7a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:33.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.444+0000 7ff812a00640 0 client.27719 ms_handle_reset on v2:172.21.15.177:6836/4223414858 2024-04-02T14:42:33.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.444+0000 7ff832655740 1 -- 172.21.15.90:0/80276054 >> 172.21.15.90:0/80276054 conn(0x55db1cc7ea50 msgr2=0x55db1ccb4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:42:33.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.445+0000 7ff832655740 1 -- 172.21.15.90:0/80276054 shutdown_connections 2024-04-02T14:42:33.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.445+0000 7ff832655740 1 -- 172.21.15.90:0/80276054 wait complete. 2024-04-02T14:42:33.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.446+0000 7ff832655740 20 client.27719 trim_cache size 0 max 16384 2024-04-02T14:42:33.444 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-02T14:42:33.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.447+0000 7ff82e400640 1 -- 172.21.15.90:0/4189709682 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7ff80c0a3de0 msgr2=0x7ff80c0a62a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:33.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.447+0000 7ff82e400640 1 --2- 172.21.15.90:0/4189709682 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7ff80c0a3de0 0x7ff80c0a62a0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7ff82810dbf0 tx=0x7ff814009500 comp rx=0 tx=0).stop 2024-04-02T14:42:33.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.447+0000 7ff82e400640 1 -- 172.21.15.90:0/4189709682 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff828073f30 msgr2=0x7ff82810c1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:33.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.447+0000 7ff82e400640 1 --2- 172.21.15.90:0/4189709682 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff828073f30 0x7ff82810c1b0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7ff81800e9f0 tx=0x7ff81800edf0 comp rx=0 tx=0).stop 2024-04-02T14:42:33.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.448+0000 7ff82e400640 1 -- 172.21.15.90:0/4189709682 shutdown_connections 2024-04-02T14:42:33.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.448+0000 7ff82e400640 1 --2- 172.21.15.90:0/4189709682 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7ff80c0a3de0 0x7ff80c0a62a0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:33.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.448+0000 7ff82e400640 1 --2- 172.21.15.90:0/4189709682 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff82810e1a0 0x7ff8281aa730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:33.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.448+0000 7ff82e400640 1 --2- 172.21.15.90:0/4189709682 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff828074850 0x7ff82810c6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:33.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.448+0000 7ff82e400640 1 --2- 172.21.15.90:0/4189709682 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff828073f30 0x7ff82810c1b0 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:33.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.448+0000 7ff82e400640 1 -- 172.21.15.90:0/4189709682 >> 172.21.15.90:0/4189709682 conn(0x7ff8280fd470 msgr2=0x7ff8280fe220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:42:33.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.448+0000 7ff82e400640 1 -- 172.21.15.90:0/4189709682 shutdown_connections 2024-04-02T14:42:33.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.448+0000 7ff82e400640 1 -- 172.21.15.90:0/4189709682 wait complete. 2024-04-02T14:42:33.478 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-04-02T14:42:33.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.741+0000 7f47ec600640 1 -- 172.21.15.90:0/1332133865 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f47e4108fe0 msgr2=0x7f47e4110bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:33.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.741+0000 7f47ec600640 1 --2- 172.21.15.90:0/1332133865 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f47e4108fe0 0x7f47e4110bb0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f47dc009ca0 tx=0x7f47dc02c100 comp rx=0 tx=0).stop 2024-04-02T14:42:33.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.742+0000 7f47ec600640 1 -- 172.21.15.90:0/1332133865 shutdown_connections 2024-04-02T14:42:33.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.742+0000 7f47ec600640 1 --2- 172.21.15.90:0/1332133865 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f47e4108fe0 0x7f47e4110bb0 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:33.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.742+0000 7f47ec600640 1 --2- 172.21.15.90:0/1332133865 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f47e41048c0 0x7f47e4108900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:33.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.742+0000 7f47ec600640 1 --2- 172.21.15.90:0/1332133865 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f47e4103f10 0x7f47e41042f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:33.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.742+0000 7f47ec600640 1 -- 172.21.15.90:0/1332133865 >> 172.21.15.90:0/1332133865 conn(0x7f47e40ff750 msgr2=0x7f47e4101b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:42:33.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.742+0000 7f47ec600640 1 -- 172.21.15.90:0/1332133865 shutdown_connections 2024-04-02T14:42:33.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.742+0000 7f47ec600640 1 -- 172.21.15.90:0/1332133865 wait complete. 2024-04-02T14:42:33.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.743+0000 7f47ec600640 1 Processor -- start 2024-04-02T14:42:33.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.743+0000 7f47ec600640 1 -- start start 2024-04-02T14:42:33.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.743+0000 7f47ec600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f47e4103f10 0x7f47e411a220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:33.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.743+0000 7f47ec600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f47e4108fe0 0x7f47e4110040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:33.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.743+0000 7f47ec600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f47e4110580 0x7f47e4110a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:33.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.743+0000 7f47ec600640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f47e4077c60 con 0x7f47e4108fe0 2024-04-02T14:42:33.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.743+0000 7f47ec600640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f47e4077ae0 con 0x7f47e4110580 2024-04-02T14:42:33.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.743+0000 7f47ec600640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f47e4077de0 con 0x7f47e4103f10 2024-04-02T14:42:33.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.744+0000 7f47e9c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f47e4103f10 0x7f47e411a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:33.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.744+0000 7f47e9200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f47e4108fe0 0x7f47e4110040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:33.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.744+0000 7f47e9c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f47e4103f10 0x7f47e411a220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:57616/0 (socket says 172.21.15.90:57616) 2024-04-02T14:42:33.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.744+0000 7f47e9c00640 1 -- 172.21.15.90:0/4254170793 learned_addr learned my addr 172.21.15.90:0/4254170793 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:42:33.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.744+0000 7f47ea600640 1 --2- 172.21.15.90:0/4254170793 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f47e4110580 0x7f47e4110a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:33.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.744+0000 7f47e9200640 1 -- 172.21.15.90:0/4254170793 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f47e4103f10 msgr2=0x7f47e411a220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:33.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.744+0000 7f47e9200640 1 --2- 172.21.15.90:0/4254170793 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f47e4103f10 0x7f47e411a220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:33.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.744+0000 7f47e9200640 1 -- 172.21.15.90:0/4254170793 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f47e4110580 msgr2=0x7f47e4110a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:33.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.744+0000 7f47e9200640 1 --2- 172.21.15.90:0/4254170793 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f47e4110580 0x7f47e4110a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:33.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.744+0000 7f47e9200640 1 -- 172.21.15.90:0/4254170793 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47e4115d10 con 0x7f47e4108fe0 2024-04-02T14:42:33.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.744+0000 7f47e9c00640 1 --2- 172.21.15.90:0/4254170793 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f47e4103f10 0x7f47e411a220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:42:33.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.744+0000 7f47e9200640 1 --2- 172.21.15.90:0/4254170793 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f47e4108fe0 0x7f47e4110040 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f47d800e9c0 tx=0x7f47d800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:42:33.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.745+0000 7f47e2000640 1 -- 172.21.15.90:0/4254170793 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47d800ccc0 con 0x7f47e4108fe0 2024-04-02T14:42:33.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.745+0000 7f47e2000640 1 -- 172.21.15.90:0/4254170793 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f47d8010910 con 0x7f47e4108fe0 2024-04-02T14:42:33.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.745+0000 7f47e2000640 1 -- 172.21.15.90:0/4254170793 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47d8018800 con 0x7f47e4108fe0 2024-04-02T14:42:33.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.746+0000 7f47ec600640 1 -- 172.21.15.90:0/4254170793 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f47e4116000 con 0x7f47e4108fe0 2024-04-02T14:42:33.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.746+0000 7f47ec600640 1 -- 172.21.15.90:0/4254170793 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47e4116500 con 0x7f47e4108fe0 2024-04-02T14:42:33.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.751+0000 7f47e2000640 1 -- 172.21.15.90:0/4254170793 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f47d8010ac0 con 0x7f47e4108fe0 2024-04-02T14:42:33.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.753+0000 7f47e2000640 1 --2- 172.21.15.90:0/4254170793 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f47cc0a3ea0 0x7f47cc0a6360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:33.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.753+0000 7f47e2000640 1 -- 172.21.15.90:0/4254170793 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8550+0+0 (secure 0 0 0) 0x7f47d80d0a80 con 0x7f47e4108fe0 2024-04-02T14:42:33.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.753+0000 7f47ef23e740 1 Processor -- start 2024-04-02T14:42:33.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.753+0000 7f47e9c00640 1 --2- 172.21.15.90:0/4254170793 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f47cc0a3ea0 0x7f47cc0a6360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:33.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.753+0000 7f47ef23e740 1 -- start start 2024-04-02T14:42:33.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.754+0000 7f47ef23e740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x562e1ca1f9f0 0x562e1ca1e740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:33.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.754+0000 7f47e9c00640 1 --2- 172.21.15.90:0/4254170793 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f47cc0a3ea0 0x7f47cc0a6360 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f47d0005fd0 tx=0x7f47d000b480 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:42:33.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.754+0000 7f47ef23e740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562e1ca1ed10 0x562e1ca1f190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:33.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.754+0000 7f47ef23e740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562e1ca28080 0x562e1ca2a470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:33.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.754+0000 7f47ef23e740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x562e1c95f400 con 0x562e1ca1ed10 2024-04-02T14:42:33.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.754+0000 7f47ef23e740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x562e1c943f10 con 0x562e1ca1f9f0 2024-04-02T14:42:33.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.754+0000 7f47ef23e740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x562e1c9836f0 con 0x562e1ca28080 2024-04-02T14:42:33.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.754+0000 7f47e9200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562e1ca1ed10 0x562e1ca1f190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:33.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.754+0000 7f47e9c00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x562e1ca1f9f0 0x562e1ca1e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:33.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.754+0000 7f47e9200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562e1ca1ed10 0x562e1ca1f190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34884/0 (socket says 172.21.15.90:34884) 2024-04-02T14:42:33.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.754+0000 7f47e9200640 1 -- 172.21.15.90:0/3334025203 learned_addr learned my addr 172.21.15.90:0/3334025203 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:42:33.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.754+0000 7f47e9c00640 1 --2- 172.21.15.90:0/3334025203 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x562e1ca1f9f0 0x562e1ca1e740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:58376/0 (socket says 172.21.15.90:58376) 2024-04-02T14:42:33.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.755+0000 7f47ea600640 1 --2- 172.21.15.90:0/3334025203 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562e1ca28080 0x562e1ca2a470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:33.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.755+0000 7f47e9200640 1 -- 172.21.15.90:0/3334025203 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562e1ca28080 msgr2=0x562e1ca2a470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:42:33.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.755+0000 7f47e9200640 1 --2- 172.21.15.90:0/3334025203 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562e1ca28080 0x562e1ca2a470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:33.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.755+0000 7f47e9200640 1 -- 172.21.15.90:0/3334025203 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x562e1ca1f9f0 msgr2=0x562e1ca1e740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:42:33.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.755+0000 7f47e9200640 1 --2- 172.21.15.90:0/3334025203 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x562e1ca1f9f0 0x562e1ca1e740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:33.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.755+0000 7f47e9200640 1 -- 172.21.15.90:0/3334025203 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562e1c6973f0 con 0x562e1ca1ed10 2024-04-02T14:42:33.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.755+0000 7f47e9200640 1 --2- 172.21.15.90:0/3334025203 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562e1ca1ed10 0x562e1ca1f190 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f47d8099420 tx=0x7f47d80a0d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b18779b324728a08 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:42:33.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.755+0000 7f47d6a00640 1 -- 172.21.15.90:0/3334025203 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47d80ce3d0 con 0x562e1ca1ed10 2024-04-02T14:42:33.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.755+0000 7f47d6a00640 1 -- 172.21.15.90:0/3334025203 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f47d80990c0 con 0x562e1ca1ed10 2024-04-02T14:42:33.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.755+0000 7f47d6a00640 1 -- 172.21.15.90:0/3334025203 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47d809c130 con 0x562e1ca1ed10 2024-04-02T14:42:33.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.757+0000 7f47ef23e740 1 -- 172.21.15.90:0/3334025203 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562e1ca1ed10 msgr2=0x562e1ca1f190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:33.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.757+0000 7f47ef23e740 1 --2- 172.21.15.90:0/3334025203 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562e1ca1ed10 0x562e1ca1f190 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f47d8099420 tx=0x7f47d80a0d50 comp rx=0 tx=0).stop 2024-04-02T14:42:33.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.758+0000 7f47ef23e740 1 -- 172.21.15.90:0/3334025203 shutdown_connections 2024-04-02T14:42:33.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.758+0000 7f47ef23e740 1 --2- 172.21.15.90:0/3334025203 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562e1ca28080 0x562e1ca2a470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:33.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.758+0000 7f47ef23e740 1 --2- 172.21.15.90:0/3334025203 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562e1ca1ed10 0x562e1ca1f190 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:33.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.758+0000 7f47ef23e740 1 --2- 172.21.15.90:0/3334025203 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x562e1ca1f9f0 0x562e1ca1e740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:33.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.758+0000 7f47ef23e740 1 -- 172.21.15.90:0/3334025203 >> 172.21.15.90:0/3334025203 conn(0x562e1c9e5a50 msgr2=0x562e1ca19760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:42:33.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.758+0000 7f47ef23e740 1 -- 172.21.15.90:0/3334025203 shutdown_connections 2024-04-02T14:42:33.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.758+0000 7f47ef23e740 1 -- 172.21.15.90:0/3334025203 wait complete. 2024-04-02T14:42:33.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.759+0000 7f47ef23e740 1 -- start start 2024-04-02T14:42:33.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.759+0000 7f47ef23e740 1 Processor -- start 2024-04-02T14:42:33.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.760+0000 7f47ef23e740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x562e1ca1ed10 0x562e1ca25070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:33.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.760+0000 7f47ef23e740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562e1ca1f9f0 0x562e1ca255b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:33.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.760+0000 7f47ef23e740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562e1ca28080 0x562e1ca25af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:33.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.760+0000 7f47ef23e740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x562e1c97f6a0 con 0x562e1ca28080 2024-04-02T14:42:33.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.760+0000 7f47ef23e740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x562e1c92b980 con 0x562e1ca1ed10 2024-04-02T14:42:33.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.760+0000 7f47ef23e740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x562e1c93ecb0 con 0x562e1ca1f9f0 2024-04-02T14:42:33.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.760+0000 7f47ef23e740 10 client.0 _ll_register_callbacks cb 0x562e1c93e6a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:42:33.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.760+0000 7f47e9c00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x562e1ca1ed10 0x562e1ca25070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:33.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.760+0000 7f47ea600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562e1ca28080 0x562e1ca25af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:33.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.760+0000 7f47e9c00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x562e1ca1ed10 0x562e1ca25070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:58388/0 (socket says 172.21.15.90:58388) 2024-04-02T14:42:33.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.760+0000 7f47e9c00640 1 -- 172.21.15.90:0/826987866 learned_addr learned my addr 172.21.15.90:0/826987866 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:42:33.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.760+0000 7f47e9200640 1 --2- 172.21.15.90:0/826987866 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562e1ca1f9f0 0x562e1ca255b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:33.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.760+0000 7f47e9200640 1 -- 172.21.15.90:0/826987866 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x562e1ca1ed10 msgr2=0x562e1ca25070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:42:33.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.760+0000 7f47e9200640 1 --2- 172.21.15.90:0/826987866 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x562e1ca1ed10 0x562e1ca25070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:33.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.760+0000 7f47e9200640 1 -- 172.21.15.90:0/826987866 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562e1ca28080 msgr2=0x562e1ca25af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:42:33.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.760+0000 7f47e9200640 1 --2- 172.21.15.90:0/826987866 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562e1ca28080 0x562e1ca25af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:33.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.760+0000 7f47e9200640 1 -- 172.21.15.90:0/826987866 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562e1c7617d0 con 0x562e1ca1f9f0 2024-04-02T14:42:33.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.760+0000 7f47e9c00640 1 --2- 172.21.15.90:0/826987866 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x562e1ca1ed10 0x562e1ca25070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:42:33.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.761+0000 7f47e9200640 1 --2- 172.21.15.90:0/826987866 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562e1ca1f9f0 0x562e1ca255b0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f47d8098eb0 tx=0x7f47d80bbe60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=14a6116172e64184 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:42:33.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.762+0000 7f47cb400640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:42:33.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.762+0000 7f47cb400640 1 -- 172.21.15.90:0/826987866 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47d809c120 con 0x562e1ca1f9f0 2024-04-02T14:42:33.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.762+0000 7f47cb400640 1 -- 172.21.15.90:0/826987866 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8550+0+0 (secure 0 0 0) 0x7f47d8010430 con 0x562e1ca1f9f0 2024-04-02T14:42:33.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.763+0000 7f47ef23e740 1 -- 172.21.15.90:0/826987866 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f47d80990c0 con 0x562e1ca1f9f0 2024-04-02T14:42:33.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.763+0000 7f47cb400640 1 -- 172.21.15.90:0/826987866 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f47d8010630 con 0x562e1ca1f9f0 2024-04-02T14:42:33.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.763+0000 7f47ef23e740 10 client.27592 fetch_fsmap learned FSMap version 27 2024-04-02T14:42:33.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.763+0000 7f47ef23e740 1 -- 172.21.15.90:0/826987866 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x562e1ca36ec0 con 0x562e1ca1f9f0 2024-04-02T14:42:33.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.763+0000 7f47cb400640 1 -- 172.21.15.90:0/826987866 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f47d80990c0 con 0x562e1ca1f9f0 2024-04-02T14:42:33.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.764+0000 7f47cb400640 1 -- 172.21.15.90:0/826987866 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f47d80189a0 con 0x562e1ca1f9f0 2024-04-02T14:42:33.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.764+0000 7f47ef23e740 10 client.27592 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:42:33.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.764+0000 7f47ef23e740 10 client.27592 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-04-02T14:42:33.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.764+0000 7f47ef23e740 20 client.27592 populate_metadata read hostname 'smithi090' 2024-04-02T14:42:33.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.764+0000 7f47ef23e740 1 --2- 172.21.15.90:0/826987866 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x562e1ca383c0 0x562e1ca3a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:33.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.764+0000 7f47ef23e740 4 client.27592 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:42:33.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.764+0000 7f47ef23e740 1 -- 172.21.15.90:0/826987866 --> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562e1c943040 con 0x562e1ca383c0 2024-04-02T14:42:33.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.764+0000 7f47e9c00640 1 --2- 172.21.15.90:0/826987866 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x562e1ca383c0 0x562e1ca3a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:33.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.765+0000 7f47cb400640 10 client.27592 ms_handle_connect on v2:172.21.15.92:6836/1270635939 2024-04-02T14:42:33.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.765+0000 7f47e9c00640 1 --2- 172.21.15.90:0/826987866 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x562e1ca383c0 0x562e1ca3a7b0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=285f587ef5d93e6c server_cookie=b3217ec76b91811a in_seq=0 out_seq=0 2024-04-02T14:42:33.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.765+0000 7f47cb400640 1 -- 172.21.15.90:0/826987866 <== mds.3 v2:172.21.15.92:6836/1270635939 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x562e1c943040 con 0x562e1ca383c0 2024-04-02T14:42:33.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.765+0000 7f47cb400640 10 client.27592 handle_command_reply: tid=0 2024-04-02T14:42:33.764 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.766+0000 7f47ef23e740 1 client.27592 shutdown 2024-04-02T14:42:33.764 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.767+0000 7f47ef23e740 1 -- 172.21.15.90:0/826987866 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562e1ca1f9f0 msgr2=0x562e1ca255b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:33.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.767+0000 7f47ef23e740 1 --2- 172.21.15.90:0/826987866 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562e1ca1f9f0 0x562e1ca255b0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f47d8098eb0 tx=0x7f47d80bbe60 comp rx=0 tx=0).stop 2024-04-02T14:42:33.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.767+0000 7f47ef23e740 1 -- 172.21.15.90:0/826987866 shutdown_connections 2024-04-02T14:42:33.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.767+0000 7f47ef23e740 1 --2- 172.21.15.90:0/826987866 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x562e1ca383c0 0x562e1ca3a7b0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:33.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.767+0000 7f47ef23e740 1 --2- 172.21.15.90:0/826987866 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562e1ca28080 0x562e1ca25af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:33.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.767+0000 7f47ef23e740 1 --2- 172.21.15.90:0/826987866 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562e1ca1f9f0 0x562e1ca255b0 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:33.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.767+0000 7f47ef23e740 1 --2- 172.21.15.90:0/826987866 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x562e1ca1ed10 0x562e1ca25070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:33.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.767+0000 7f47ef23e740 1 -- 172.21.15.90:0/826987866 >> 172.21.15.90:0/826987866 conn(0x562e1c9e5a50 msgr2=0x562e1ca1b4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:42:33.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.769+0000 7f47cb400640 0 client.27592 ms_handle_reset on v2:172.21.15.92:6836/1270635939 2024-04-02T14:42:33.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.769+0000 7f47ef23e740 1 -- 172.21.15.90:0/826987866 shutdown_connections 2024-04-02T14:42:33.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.769+0000 7f47ef23e740 1 -- 172.21.15.90:0/826987866 wait complete. 2024-04-02T14:42:33.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.769+0000 7f47ef23e740 20 client.27592 trim_cache size 0 max 16384 2024-04-02T14:42:33.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.789+0000 7f47ef23e740 1 Processor -- start 2024-04-02T14:42:33.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.789+0000 7f47ef23e740 1 -- start start 2024-04-02T14:42:33.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.789+0000 7f47ef23e740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562e1ca1ed10 0x562e1ca27730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:33.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.789+0000 7f47ef23e740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x562e1ca1f9f0 0x562e1ca33a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:33.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.789+0000 7f47ef23e740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562e1ca28080 0x562e1ca40080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:33.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.789+0000 7f47ef23e740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f47e41bc7d0 con 0x562e1ca28080 2024-04-02T14:42:33.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.789+0000 7f47ef23e740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f47e41be310 con 0x562e1ca1f9f0 2024-04-02T14:42:33.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.789+0000 7f47ef23e740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f47e41bcea0 con 0x562e1ca1ed10 2024-04-02T14:42:33.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.790+0000 7f47e9200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x562e1ca1f9f0 0x562e1ca33a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:33.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.790+0000 7f47e9200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x562e1ca1f9f0 0x562e1ca33a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:58390/0 (socket says 172.21.15.90:58390) 2024-04-02T14:42:33.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.790+0000 7f47e9200640 1 -- 172.21.15.90:0/777042176 learned_addr learned my addr 172.21.15.90:0/777042176 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:42:33.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.790+0000 7f47e9c00640 1 --2- 172.21.15.90:0/777042176 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562e1ca1ed10 0x562e1ca27730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:33.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.790+0000 7f47ea600640 1 --2- 172.21.15.90:0/777042176 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562e1ca28080 0x562e1ca40080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:33.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.790+0000 7f47e9200640 1 -- 172.21.15.90:0/777042176 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562e1ca1ed10 msgr2=0x562e1ca27730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:42:33.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.792+0000 7f47e9200640 1 --2- 172.21.15.90:0/777042176 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562e1ca1ed10 0x562e1ca27730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:33.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.792+0000 7f47e9200640 1 -- 172.21.15.90:0/777042176 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562e1ca28080 msgr2=0x562e1ca40080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:42:33.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.792+0000 7f47e9200640 1 --2- 172.21.15.90:0/777042176 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562e1ca28080 0x562e1ca40080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:33.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.792+0000 7f47e9200640 1 -- 172.21.15.90:0/777042176 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562e1ca24e40 con 0x562e1ca1f9f0 2024-04-02T14:42:33.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.792+0000 7f47ea600640 1 --2- 172.21.15.90:0/777042176 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562e1ca28080 0x562e1ca40080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:42:33.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.792+0000 7f47e9c00640 1 --2- 172.21.15.90:0/777042176 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562e1ca1ed10 0x562e1ca27730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:42:33.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.792+0000 7f47e9200640 1 --2- 172.21.15.90:0/777042176 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x562e1ca1f9f0 0x562e1ca33a90 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f47d809ad00 tx=0x7f47d809a2a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=924d9c168151039e server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:42:33.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.792+0000 7f47d6000640 1 -- 172.21.15.90:0/777042176 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47d8010630 con 0x562e1ca1f9f0 2024-04-02T14:42:33.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.793+0000 7f47d6000640 1 -- 172.21.15.90:0/777042176 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f47d80b8070 con 0x562e1ca1f9f0 2024-04-02T14:42:33.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.793+0000 7f47d6000640 1 -- 172.21.15.90:0/777042176 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47d80d0750 con 0x562e1ca1f9f0 2024-04-02T14:42:33.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.793+0000 7f47ef23e740 1 -- 172.21.15.90:0/777042176 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x562e1ca1f9f0 msgr2=0x562e1ca33a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:33.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.793+0000 7f47ef23e740 1 --2- 172.21.15.90:0/777042176 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x562e1ca1f9f0 0x562e1ca33a90 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f47d809ad00 tx=0x7f47d809a2a0 comp rx=0 tx=0).stop 2024-04-02T14:42:33.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.793+0000 7f47ef23e740 1 -- 172.21.15.90:0/777042176 shutdown_connections 2024-04-02T14:42:33.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.794+0000 7f47ef23e740 1 --2- 172.21.15.90:0/777042176 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562e1ca28080 0x562e1ca40080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:33.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.794+0000 7f47ef23e740 1 --2- 172.21.15.90:0/777042176 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x562e1ca1f9f0 0x562e1ca33a90 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:33.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.794+0000 7f47ef23e740 1 --2- 172.21.15.90:0/777042176 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562e1ca1ed10 0x562e1ca27730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:33.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.794+0000 7f47ef23e740 1 -- 172.21.15.90:0/777042176 >> 172.21.15.90:0/777042176 conn(0x562e1c9e5a50 msgr2=0x562e1ca19290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:42:33.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.794+0000 7f47ef23e740 1 -- 172.21.15.90:0/777042176 shutdown_connections 2024-04-02T14:42:33.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.794+0000 7f47ef23e740 1 -- 172.21.15.90:0/777042176 wait complete. 2024-04-02T14:42:33.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.794+0000 7f47ef23e740 1 -- start start 2024-04-02T14:42:33.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.794+0000 7f47ef23e740 1 Processor -- start 2024-04-02T14:42:33.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.795+0000 7f47ef23e740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x562e1ca1ed10 0x562e1ca30230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:33.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.795+0000 7f47ef23e740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562e1ca1f9f0 0x562e1ca295c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:33.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.795+0000 7f47ef23e740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562e1ca28080 0x562e1ca29b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:33.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.795+0000 7f47ef23e740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f47e41bdc40 con 0x562e1ca1f9f0 2024-04-02T14:42:33.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.795+0000 7f47ef23e740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f47e41bd570 con 0x562e1ca1ed10 2024-04-02T14:42:33.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.795+0000 7f47ef23e740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f47e41ba790 con 0x562e1ca28080 2024-04-02T14:42:33.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.795+0000 7f47ef23e740 10 client.0 _ll_register_callbacks cb 0x7f47d809c440 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:42:33.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.795+0000 7f47e9c00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x562e1ca1ed10 0x562e1ca30230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:33.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.796+0000 7f47e9c00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x562e1ca1ed10 0x562e1ca30230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:58396/0 (socket says 172.21.15.90:58396) 2024-04-02T14:42:33.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.796+0000 7f47e9c00640 1 -- 172.21.15.90:0/3262584 learned_addr learned my addr 172.21.15.90:0/3262584 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:42:33.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.796+0000 7f47e9200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562e1ca1f9f0 0x562e1ca295c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:33.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.796+0000 7f47ea600640 1 --2- 172.21.15.90:0/3262584 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562e1ca28080 0x562e1ca29b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:33.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.796+0000 7f47e9c00640 1 -- 172.21.15.90:0/3262584 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562e1ca28080 msgr2=0x562e1ca29b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:42:33.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.796+0000 7f47e9c00640 1 --2- 172.21.15.90:0/3262584 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562e1ca28080 0x562e1ca29b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:33.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.796+0000 7f47e9c00640 1 -- 172.21.15.90:0/3262584 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562e1ca1f9f0 msgr2=0x562e1ca295c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:42:33.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.796+0000 7f47e9c00640 1 --2- 172.21.15.90:0/3262584 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562e1ca1f9f0 0x562e1ca295c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:33.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.796+0000 7f47e9c00640 1 -- 172.21.15.90:0/3262584 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562e1ca28460 con 0x562e1ca1ed10 2024-04-02T14:42:33.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.796+0000 7f47e9200640 1 --2- 172.21.15.90:0/3262584 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562e1ca1f9f0 0x562e1ca295c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:42:33.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.797+0000 7f47e9c00640 1 --2- 172.21.15.90:0/3262584 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x562e1ca1ed10 0x562e1ca30230 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f47d0011fc0 tx=0x7f47d0013070 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d7b128cf011b2692 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:42:33.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.797+0000 7f47cb400640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:42:33.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.797+0000 7f47cb400640 1 -- 172.21.15.90:0/3262584 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47d002a440 con 0x562e1ca1ed10 2024-04-02T14:42:33.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.797+0000 7f47ef23e740 1 -- 172.21.15.90:0/3262584 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f47d80b8070 con 0x562e1ca1ed10 2024-04-02T14:42:33.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.797+0000 7f47cb400640 1 -- 172.21.15.90:0/3262584 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8550+0+0 (secure 0 0 0) 0x7f47d002a5e0 con 0x562e1ca1ed10 2024-04-02T14:42:33.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.797+0000 7f47cb400640 1 -- 172.21.15.90:0/3262584 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f47d0039050 con 0x562e1ca1ed10 2024-04-02T14:42:33.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.797+0000 7f47ef23e740 10 client.27737 fetch_fsmap learned FSMap version 27 2024-04-02T14:42:33.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.797+0000 7f47ef23e740 1 -- 172.21.15.90:0/3262584 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x562e1ca2f1c0 con 0x562e1ca1ed10 2024-04-02T14:42:33.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.798+0000 7f47cb400640 1 -- 172.21.15.90:0/3262584 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f47d80b8070 con 0x562e1ca1ed10 2024-04-02T14:42:33.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.798+0000 7f47cb400640 1 -- 172.21.15.90:0/3262584 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f47d00047a0 con 0x562e1ca1ed10 2024-04-02T14:42:33.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.798+0000 7f47ef23e740 10 client.27737 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:42:33.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.798+0000 7f47ef23e740 10 client.27737 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-04-02T14:42:33.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.798+0000 7f47ef23e740 20 client.27737 populate_metadata read hostname 'smithi090' 2024-04-02T14:42:33.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.798+0000 7f47ef23e740 1 --2- 172.21.15.90:0/3262584 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x562e1ca2f8e0 0x562e1caa0900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:33.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.798+0000 7f47ef23e740 4 client.27737 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-02T14:42:33.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.798+0000 7f47ef23e740 1 -- 172.21.15.90:0/3262584 --> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] -- command(tid 0: {"prefix": "damage ls"}) -- 0x562e1c990c30 con 0x562e1ca2f8e0 2024-04-02T14:42:33.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.798+0000 7f47e9200640 1 --2- 172.21.15.90:0/3262584 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x562e1ca2f8e0 0x562e1caa0900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:33.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.799+0000 7f47e9200640 1 --2- 172.21.15.90:0/3262584 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x562e1ca2f8e0 0x562e1caa0900 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=8434e825ca189e7e server_cookie=1367df60435548f4 in_seq=0 out_seq=0 2024-04-02T14:42:33.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.799+0000 7f47cb400640 10 client.27737 ms_handle_connect on v2:172.21.15.92:6836/1270635939 2024-04-02T14:42:33.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.799+0000 7f47cb400640 1 -- 172.21.15.90:0/3262584 <== mds.3 v2:172.21.15.92:6836/1270635939 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x562e1c990c30 con 0x562e1ca2f8e0 2024-04-02T14:42:33.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.799+0000 7f47cb400640 10 client.27737 handle_command_reply: tid=0 2024-04-02T14:42:33.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.799+0000 7f47ef23e740 1 client.27737 shutdown 2024-04-02T14:42:33.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.799+0000 7f47ef23e740 1 -- 172.21.15.90:0/3262584 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x562e1ca1ed10 msgr2=0x562e1ca30230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:33.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.799+0000 7f47ef23e740 1 --2- 172.21.15.90:0/3262584 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x562e1ca1ed10 0x562e1ca30230 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f47d0011fc0 tx=0x7f47d0013070 comp rx=0 tx=0).stop 2024-04-02T14:42:33.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.799+0000 7f47ef23e740 1 -- 172.21.15.90:0/3262584 shutdown_connections 2024-04-02T14:42:33.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.799+0000 7f47ef23e740 1 --2- 172.21.15.90:0/3262584 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x562e1ca2f8e0 0x562e1caa0900 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:33.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.799+0000 7f47ef23e740 1 --2- 172.21.15.90:0/3262584 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562e1ca28080 0x562e1ca29b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:33.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.799+0000 7f47ef23e740 1 --2- 172.21.15.90:0/3262584 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562e1ca1f9f0 0x562e1ca295c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:33.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.799+0000 7f47ef23e740 1 --2- 172.21.15.90:0/3262584 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x562e1ca1ed10 0x562e1ca30230 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:33.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.799+0000 7f47ef23e740 1 -- 172.21.15.90:0/3262584 >> 172.21.15.90:0/3262584 conn(0x562e1c9e5a50 msgr2=0x562e1ca18010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:42:33.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.799+0000 7f47cb400640 0 client.27737 ms_handle_reset on v2:172.21.15.92:6836/1270635939 2024-04-02T14:42:33.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.800+0000 7f47ef23e740 1 -- 172.21.15.90:0/3262584 shutdown_connections 2024-04-02T14:42:33.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.800+0000 7f47ef23e740 1 -- 172.21.15.90:0/3262584 wait complete. 2024-04-02T14:42:33.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.800+0000 7f47ef23e740 20 client.27737 trim_cache size 0 max 16384 2024-04-02T14:42:33.801 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-02T14:42:33.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.800+0000 7f47ec600640 1 -- 172.21.15.90:0/4254170793 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f47cc0a3ea0 msgr2=0x7f47cc0a6360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:33.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.800+0000 7f47ec600640 1 --2- 172.21.15.90:0/4254170793 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f47cc0a3ea0 0x7f47cc0a6360 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f47d0005fd0 tx=0x7f47d000b480 comp rx=0 tx=0).stop 2024-04-02T14:42:33.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.800+0000 7f47ec600640 1 -- 172.21.15.90:0/4254170793 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f47e4108fe0 msgr2=0x7f47e4110040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:33.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.800+0000 7f47ec600640 1 --2- 172.21.15.90:0/4254170793 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f47e4108fe0 0x7f47e4110040 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f47d800e9c0 tx=0x7f47d800ee80 comp rx=0 tx=0).stop 2024-04-02T14:42:33.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.801+0000 7f47ec600640 1 -- 172.21.15.90:0/4254170793 shutdown_connections 2024-04-02T14:42:33.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.801+0000 7f47ec600640 1 --2- 172.21.15.90:0/4254170793 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f47cc0a3ea0 0x7f47cc0a6360 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:33.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.801+0000 7f47ec600640 1 --2- 172.21.15.90:0/4254170793 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f47e4110580 0x7f47e4110a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:33.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.801+0000 7f47ec600640 1 --2- 172.21.15.90:0/4254170793 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f47e4108fe0 0x7f47e4110040 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:33.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.801+0000 7f47ec600640 1 --2- 172.21.15.90:0/4254170793 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f47e4103f10 0x7f47e411a220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:33.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.801+0000 7f47ec600640 1 -- 172.21.15.90:0/4254170793 >> 172.21.15.90:0/4254170793 conn(0x7f47e40ff750 msgr2=0x7f47e40793b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:42:33.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.801+0000 7f47ec600640 1 -- 172.21.15.90:0/4254170793 shutdown_connections 2024-04-02T14:42:33.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:33.801+0000 7f47ec600640 1 -- 172.21.15.90:0/4254170793 wait complete. 2024-04-02T14:42:33.826 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-02T14:42:34.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.111+0000 7f208d400640 1 -- 172.21.15.90:0/1801567394 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2088107ab0 msgr2=0x7f2088107e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:34.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.111+0000 7f208d400640 1 --2- 172.21.15.90:0/1801567394 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2088107ab0 0x7f2088107e90 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f2078009900 tx=0x7f207802bc30 comp rx=0 tx=0).stop 2024-04-02T14:42:34.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.111+0000 7f208d400640 1 -- 172.21.15.90:0/1801567394 shutdown_connections 2024-04-02T14:42:34.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.111+0000 7f208d400640 1 --2- 172.21.15.90:0/1801567394 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2088102460 0x7f208810e970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:34.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.111+0000 7f208d400640 1 --2- 172.21.15.90:0/1801567394 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2088101aa0 0x7f2088101f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:34.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.111+0000 7f208d400640 1 --2- 172.21.15.90:0/1801567394 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2088107ab0 0x7f2088107e90 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:34.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.111+0000 7f208d400640 1 -- 172.21.15.90:0/1801567394 >> 172.21.15.90:0/1801567394 conn(0x7f20880fd530 msgr2=0x7f20880ff950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:42:34.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.112+0000 7f208d400640 1 -- 172.21.15.90:0/1801567394 shutdown_connections 2024-04-02T14:42:34.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.113+0000 7f208d400640 1 -- 172.21.15.90:0/1801567394 wait complete. 2024-04-02T14:42:34.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.114+0000 7f208d400640 1 Processor -- start 2024-04-02T14:42:34.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.114+0000 7f208d400640 1 -- start start 2024-04-02T14:42:34.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.114+0000 7f208d400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2088101aa0 0x7f208810ca60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:34.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.114+0000 7f208d400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2088102460 0x7f208810cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:34.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.114+0000 7f208d400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2088107ab0 0x7f208819a980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:34.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.115+0000 7f208d400640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f2088111ee0 con 0x7f2088102460 2024-04-02T14:42:34.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.115+0000 7f208d400640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f2088111d60 con 0x7f2088107ab0 2024-04-02T14:42:34.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.115+0000 7f208d400640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f2088112060 con 0x7f2088101aa0 2024-04-02T14:42:34.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.115+0000 7f2086800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2088101aa0 0x7f208810ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:34.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.115+0000 7f2087200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2088107ab0 0x7f208819a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:34.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.115+0000 7f2086800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2088101aa0 0x7f208810ca60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:47332/0 (socket says 172.21.15.90:47332) 2024-04-02T14:42:34.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.115+0000 7f2086800640 1 -- 172.21.15.90:0/2002787078 learned_addr learned my addr 172.21.15.90:0/2002787078 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:42:34.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.115+0000 7f2085e00640 1 --2- 172.21.15.90:0/2002787078 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2088102460 0x7f208810cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:34.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.115+0000 7f2087200640 1 -- 172.21.15.90:0/2002787078 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2088101aa0 msgr2=0x7f208810ca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:34.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.115+0000 7f2087200640 1 --2- 172.21.15.90:0/2002787078 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2088101aa0 0x7f208810ca60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:34.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.115+0000 7f2087200640 1 -- 172.21.15.90:0/2002787078 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2088102460 msgr2=0x7f208810cfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:34.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.115+0000 7f2087200640 1 --2- 172.21.15.90:0/2002787078 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2088102460 0x7f208810cfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:34.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.115+0000 7f2087200640 1 -- 172.21.15.90:0/2002787078 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f208819b0b0 con 0x7f2088107ab0 2024-04-02T14:42:34.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.116+0000 7f2087200640 1 --2- 172.21.15.90:0/2002787078 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2088107ab0 0x7f208819a980 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f2080009fd0 tx=0x7f208000ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:42:34.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.116+0000 7f207f400640 1 -- 172.21.15.90:0/2002787078 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f208000ce20 con 0x7f2088107ab0 2024-04-02T14:42:34.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.116+0000 7f207f400640 1 -- 172.21.15.90:0/2002787078 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2080005020 con 0x7f2088107ab0 2024-04-02T14:42:34.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.116+0000 7f207f400640 1 -- 172.21.15.90:0/2002787078 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2080010720 con 0x7f2088107ab0 2024-04-02T14:42:34.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.116+0000 7f208d400640 1 -- 172.21.15.90:0/2002787078 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f208819b3a0 con 0x7f2088107ab0 2024-04-02T14:42:34.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.117+0000 7f208d400640 1 -- 172.21.15.90:0/2002787078 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20881978b0 con 0x7f2088107ab0 2024-04-02T14:42:34.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.121+0000 7f207f400640 1 -- 172.21.15.90:0/2002787078 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f2080005540 con 0x7f2088107ab0 2024-04-02T14:42:34.120 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.123+0000 7f209022d740 1 Processor -- start 2024-04-02T14:42:34.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.123+0000 7f209022d740 1 -- start start 2024-04-02T14:42:34.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.124+0000 7f209022d740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556a759569f0 0x556a75955740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:34.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.124+0000 7f209022d740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556a75955d10 0x556a75956190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:34.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.124+0000 7f209022d740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556a7595f080 0x556a75961470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:34.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.124+0000 7f209022d740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x556a75896400 con 0x556a75955d10 2024-04-02T14:42:34.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.124+0000 7f209022d740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x556a7587af10 con 0x556a759569f0 2024-04-02T14:42:34.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.124+0000 7f209022d740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x556a758ba6f0 con 0x556a7595f080 2024-04-02T14:42:34.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.124+0000 7f2086800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556a75955d10 0x556a75956190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:34.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.125+0000 7f2085e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556a7595f080 0x556a75961470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:34.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.125+0000 7f2086800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556a75955d10 0x556a75956190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:58802/0 (socket says 172.21.15.90:58802) 2024-04-02T14:42:34.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.125+0000 7f2086800640 1 -- 172.21.15.90:0/3939067299 learned_addr learned my addr 172.21.15.90:0/3939067299 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:42:34.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.125+0000 7f2085e00640 1 --2- 172.21.15.90:0/3939067299 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556a759569f0 0x556a75955740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:34.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.125+0000 7f207f400640 1 --2- 172.21.15.90:0/2002787078 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f206c0a3ef0 0x7f206c0a63b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:34.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.125+0000 7f2085e00640 1 -- 172.21.15.90:0/3939067299 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556a759569f0 msgr2=0x556a75955740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:42:34.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.125+0000 7f2085e00640 1 --2- 172.21.15.90:0/3939067299 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556a759569f0 0x556a75955740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:34.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.125+0000 7f2085e00640 1 -- 172.21.15.90:0/3939067299 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556a75955d10 msgr2=0x556a75956190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:42:34.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.125+0000 7f2085e00640 1 --2- 172.21.15.90:0/3939067299 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556a75955d10 0x556a75956190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:34.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.125+0000 7f2085e00640 1 -- 172.21.15.90:0/3939067299 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556a755ce3f0 con 0x556a7595f080 2024-04-02T14:42:34.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.125+0000 7f2086800640 1 --2- 172.21.15.90:0/3939067299 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556a75955d10 0x556a75956190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:42:34.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.125+0000 7f2087200640 1 --2- 172.21.15.90:0/2002787078 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f206c0a3ef0 0x7f206c0a63b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:34.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.126+0000 7f207f400640 1 -- 172.21.15.90:0/2002787078 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8550+0+0 (secure 0 0 0) 0x7f20800d0ea0 con 0x7f2088107ab0 2024-04-02T14:42:34.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.126+0000 7f2085e00640 1 --2- 172.21.15.90:0/3939067299 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556a7595f080 0x556a75961470 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f20740126c0 tx=0x7f2074009640 comp rx=0 tx=0).ready entity=mon.1 client_cookie=108b553c14e4a9f7 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:42:34.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.127+0000 7f2073400640 1 -- 172.21.15.90:0/3939067299 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2074012c40 con 0x556a7595f080 2024-04-02T14:42:34.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.127+0000 7f2087200640 1 --2- 172.21.15.90:0/2002787078 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f206c0a3ef0 0x7f206c0a63b0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f2080005b40 tx=0x7f2080099060 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:42:34.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.127+0000 7f2073400640 1 -- 172.21.15.90:0/3939067299 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2074012de0 con 0x556a7595f080 2024-04-02T14:42:34.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.127+0000 7f2073400640 1 -- 172.21.15.90:0/3939067299 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f207401ec40 con 0x556a7595f080 2024-04-02T14:42:34.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.128+0000 7f209022d740 1 -- 172.21.15.90:0/3939067299 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556a7595f080 msgr2=0x556a75961470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:34.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.128+0000 7f209022d740 1 --2- 172.21.15.90:0/3939067299 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556a7595f080 0x556a75961470 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f20740126c0 tx=0x7f2074009640 comp rx=0 tx=0).stop 2024-04-02T14:42:34.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.128+0000 7f209022d740 1 -- 172.21.15.90:0/3939067299 shutdown_connections 2024-04-02T14:42:34.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.128+0000 7f209022d740 1 --2- 172.21.15.90:0/3939067299 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556a7595f080 0x556a75961470 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:34.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.128+0000 7f209022d740 1 --2- 172.21.15.90:0/3939067299 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556a75955d10 0x556a75956190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:34.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.128+0000 7f209022d740 1 --2- 172.21.15.90:0/3939067299 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556a759569f0 0x556a75955740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:34.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.128+0000 7f209022d740 1 -- 172.21.15.90:0/3939067299 >> 172.21.15.90:0/3939067299 conn(0x556a7591ca50 msgr2=0x556a75950760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:42:34.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.128+0000 7f209022d740 1 -- 172.21.15.90:0/3939067299 shutdown_connections 2024-04-02T14:42:34.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.128+0000 7f209022d740 1 -- 172.21.15.90:0/3939067299 wait complete. 2024-04-02T14:42:34.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.129+0000 7f209022d740 1 -- start start 2024-04-02T14:42:34.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.129+0000 7f209022d740 1 Processor -- start 2024-04-02T14:42:34.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.130+0000 7f209022d740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556a75955d10 0x556a7595c0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:34.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.130+0000 7f209022d740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556a759569f0 0x556a7595c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:34.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.130+0000 7f209022d740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556a7595f080 0x556a7595cb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:34.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.130+0000 7f209022d740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x556a758b66a0 con 0x556a759569f0 2024-04-02T14:42:34.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.130+0000 7f209022d740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x556a75862980 con 0x556a75955d10 2024-04-02T14:42:34.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.130+0000 7f209022d740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x556a75875cb0 con 0x556a7595f080 2024-04-02T14:42:34.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.130+0000 7f209022d740 10 client.0 _ll_register_callbacks cb 0x556a758756a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:42:34.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.130+0000 7f2085e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556a75955d10 0x556a7595c0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:34.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.130+0000 7f2086800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556a759569f0 0x556a7595c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:34.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.131+0000 7f2086800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556a759569f0 0x556a7595c600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:58806/0 (socket says 172.21.15.90:58806) 2024-04-02T14:42:34.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.131+0000 7f2086800640 1 -- 172.21.15.90:0/2260787174 learned_addr learned my addr 172.21.15.90:0/2260787174 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:42:34.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.131+0000 7f2085e00640 1 --2- 172.21.15.90:0/2260787174 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556a7595f080 0x556a7595cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:34.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.132+0000 7f2086800640 1 -- 172.21.15.90:0/2260787174 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556a7595f080 msgr2=0x556a7595cb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:42:34.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.132+0000 7f2086800640 1 --2- 172.21.15.90:0/2260787174 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556a7595f080 0x556a7595cb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:34.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.132+0000 7f2086800640 1 -- 172.21.15.90:0/2260787174 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556a75955d10 msgr2=0x556a7595c0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:42:34.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.132+0000 7f2086800640 1 --2- 172.21.15.90:0/2260787174 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556a75955d10 0x556a7595c0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:34.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.132+0000 7f2086800640 1 -- 172.21.15.90:0/2260787174 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556a756987d0 con 0x556a759569f0 2024-04-02T14:42:34.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.133+0000 7f2086800640 1 --2- 172.21.15.90:0/2260787174 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556a759569f0 0x556a7595c600 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f207802c800 tx=0x7f207802c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da66d162bee1a022 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:42:34.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.133+0000 7f2067e00640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-02T14:42:34.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.133+0000 7f2067e00640 1 -- 172.21.15.90:0/2260787174 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20780046c0 con 0x556a759569f0 2024-04-02T14:42:34.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.133+0000 7f2067e00640 1 -- 172.21.15.90:0/2260787174 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8550+0+0 (secure 0 0 0) 0x7f2078004860 con 0x556a759569f0 2024-04-02T14:42:34.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.134+0000 7f209022d740 1 -- 172.21.15.90:0/2260787174 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2074012de0 con 0x556a759569f0 2024-04-02T14:42:34.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.134+0000 7f2067e00640 1 -- 172.21.15.90:0/2260787174 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f2078004a60 con 0x556a759569f0 2024-04-02T14:42:34.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.134+0000 7f209022d740 10 client.16998 fetch_fsmap learned FSMap version 27 2024-04-02T14:42:34.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.134+0000 7f209022d740 1 -- 172.21.15.90:0/2260787174 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x556a7597efa0 con 0x556a759569f0 2024-04-02T14:42:34.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.135+0000 7f2067e00640 1 -- 172.21.15.90:0/2260787174 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2074012de0 con 0x556a759569f0 2024-04-02T14:42:34.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.135+0000 7f2067e00640 1 -- 172.21.15.90:0/2260787174 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f2078003a80 con 0x556a759569f0 2024-04-02T14:42:34.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.135+0000 7f209022d740 10 client.16998 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:42:34.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.135+0000 7f209022d740 10 client.16998 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-04-02T14:42:34.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.135+0000 7f209022d740 20 client.16998 populate_metadata read hostname 'smithi090' 2024-04-02T14:42:34.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.135+0000 7f209022d740 1 --2- 172.21.15.90:0/2260787174 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x556a75980460 0x556a75982850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:34.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.135+0000 7f209022d740 4 client.16998 mds_command: new command op to 24547 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:42:34.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.135+0000 7f209022d740 1 -- 172.21.15.90:0/2260787174 --> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556a7587a040 con 0x556a75980460 2024-04-02T14:42:34.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.135+0000 7f2085e00640 1 --2- 172.21.15.90:0/2260787174 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x556a75980460 0x556a75982850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:34.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.136+0000 7f2085e00640 1 --2- 172.21.15.90:0/2260787174 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x556a75980460 0x556a75982850 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=f47a79b0d7b08d13 server_cookie=a9b28f27ece194d2 in_seq=0 out_seq=0 2024-04-02T14:42:34.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.136+0000 7f2067e00640 10 client.16998 ms_handle_connect on v2:172.21.15.177:6838/664895663 2024-04-02T14:42:34.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.137+0000 7f2067e00640 1 -- 172.21.15.90:0/2260787174 <== mds.2 v2:172.21.15.177:6838/664895663 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x556a7587a040 con 0x556a75980460 2024-04-02T14:42:34.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.137+0000 7f2067e00640 10 client.16998 handle_command_reply: tid=0 2024-04-02T14:42:34.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.137+0000 7f209022d740 1 client.16998 shutdown 2024-04-02T14:42:34.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.138+0000 7f209022d740 1 -- 172.21.15.90:0/2260787174 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556a759569f0 msgr2=0x556a7595c600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:34.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.138+0000 7f209022d740 1 --2- 172.21.15.90:0/2260787174 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556a759569f0 0x556a7595c600 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f207802c800 tx=0x7f207802c8b0 comp rx=0 tx=0).stop 2024-04-02T14:42:34.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.138+0000 7f209022d740 1 -- 172.21.15.90:0/2260787174 shutdown_connections 2024-04-02T14:42:34.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.138+0000 7f209022d740 1 --2- 172.21.15.90:0/2260787174 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x556a75980460 0x556a75982850 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:34.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.138+0000 7f209022d740 1 --2- 172.21.15.90:0/2260787174 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556a7595f080 0x556a7595cb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:34.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.138+0000 7f209022d740 1 --2- 172.21.15.90:0/2260787174 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556a759569f0 0x556a7595c600 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:34.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.138+0000 7f2067e00640 0 client.16998 ms_handle_reset on v2:172.21.15.177:6838/664895663 2024-04-02T14:42:34.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.138+0000 7f209022d740 1 --2- 172.21.15.90:0/2260787174 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556a75955d10 0x556a7595c0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:34.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.138+0000 7f209022d740 1 -- 172.21.15.90:0/2260787174 >> 172.21.15.90:0/2260787174 conn(0x556a7591ca50 msgr2=0x556a759524b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:42:34.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.138+0000 7f209022d740 1 -- 172.21.15.90:0/2260787174 shutdown_connections 2024-04-02T14:42:34.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.139+0000 7f209022d740 1 -- 172.21.15.90:0/2260787174 wait complete. 2024-04-02T14:42:34.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.139+0000 7f209022d740 20 client.16998 trim_cache size 0 max 16384 2024-04-02T14:42:34.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.166+0000 7f209022d740 1 Processor -- start 2024-04-02T14:42:34.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.166+0000 7f209022d740 1 -- start start 2024-04-02T14:42:34.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.166+0000 7f209022d740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556a75955d10 0x556a7595e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:34.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.166+0000 7f209022d740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556a7595f080 0x556a7597bb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:34.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.166+0000 7f209022d740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556a7595c790 0x556a75988120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:34.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.166+0000 7f209022d740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f2088072db0 con 0x556a7595f080 2024-04-02T14:42:34.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.166+0000 7f209022d740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f20881b5780 con 0x556a75955d10 2024-04-02T14:42:34.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.166+0000 7f209022d740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f2088073480 con 0x556a7595c790 2024-04-02T14:42:34.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.167+0000 7f2085e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556a75955d10 0x556a7595e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:34.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.167+0000 7f2085e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556a75955d10 0x556a7595e7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:55048/0 (socket says 172.21.15.90:55048) 2024-04-02T14:42:34.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.167+0000 7f2085e00640 1 -- 172.21.15.90:0/1054493325 learned_addr learned my addr 172.21.15.90:0/1054493325 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:42:34.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.167+0000 7f2086800640 1 --2- 172.21.15.90:0/1054493325 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556a7595f080 0x556a7597bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:34.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.167+0000 7f2085e00640 1 --2- 172.21.15.90:0/1054493325 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556a7595c790 0x556a75988120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:34.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.167+0000 7f2085e00640 1 -- 172.21.15.90:0/1054493325 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556a7595c790 msgr2=0x556a75988120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:42:34.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.167+0000 7f2085e00640 1 --2- 172.21.15.90:0/1054493325 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556a7595c790 0x556a75988120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:34.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.167+0000 7f2085e00640 1 -- 172.21.15.90:0/1054493325 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556a7595f080 msgr2=0x556a7597bb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:42:34.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.167+0000 7f2085e00640 1 --2- 172.21.15.90:0/1054493325 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556a7595f080 0x556a7597bb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:34.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.167+0000 7f2085e00640 1 -- 172.21.15.90:0/1054493325 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556a75954d40 con 0x556a75955d10 2024-04-02T14:42:34.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.167+0000 7f2086800640 1 --2- 172.21.15.90:0/1054493325 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556a7595f080 0x556a7597bb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:42:34.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.168+0000 7f2085e00640 1 --2- 172.21.15.90:0/1054493325 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556a75955d10 0x556a7595e7f0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f2074012690 tx=0x7f2074005f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a2edab70377092b0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:42:34.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.168+0000 7f2072a00640 1 -- 172.21.15.90:0/1054493325 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2074024040 con 0x556a75955d10 2024-04-02T14:42:34.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.168+0000 7f2072a00640 1 -- 172.21.15.90:0/1054493325 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f207401e950 con 0x556a75955d10 2024-04-02T14:42:34.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.168+0000 7f2072a00640 1 -- 172.21.15.90:0/1054493325 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f207402c6e0 con 0x556a75955d10 2024-04-02T14:42:34.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.169+0000 7f209022d740 1 -- 172.21.15.90:0/1054493325 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556a75955d10 msgr2=0x556a7595e7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:34.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.169+0000 7f209022d740 1 --2- 172.21.15.90:0/1054493325 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556a75955d10 0x556a7595e7f0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f2074012690 tx=0x7f2074005f90 comp rx=0 tx=0).stop 2024-04-02T14:42:34.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.170+0000 7f209022d740 1 -- 172.21.15.90:0/1054493325 shutdown_connections 2024-04-02T14:42:34.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.170+0000 7f209022d740 1 --2- 172.21.15.90:0/1054493325 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556a7595c790 0x556a75988120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:34.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.170+0000 7f209022d740 1 --2- 172.21.15.90:0/1054493325 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556a7595f080 0x556a7597bb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:34.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.170+0000 7f209022d740 1 --2- 172.21.15.90:0/1054493325 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556a75955d10 0x556a7595e7f0 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:34.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.170+0000 7f209022d740 1 -- 172.21.15.90:0/1054493325 >> 172.21.15.90:0/1054493325 conn(0x556a7591ca50 msgr2=0x556a7595c080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:42:34.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.171+0000 7f209022d740 1 -- 172.21.15.90:0/1054493325 shutdown_connections 2024-04-02T14:42:34.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.171+0000 7f209022d740 1 -- 172.21.15.90:0/1054493325 wait complete. 2024-04-02T14:42:34.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.171+0000 7f209022d740 1 -- start start 2024-04-02T14:42:34.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.171+0000 7f209022d740 1 Processor -- start 2024-04-02T14:42:34.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.172+0000 7f209022d740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556a75955d10 0x556a75978040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:34.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.172+0000 7f209022d740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556a7595c790 0x556a75978580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:34.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.172+0000 7f209022d740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556a7595f080 0x556a75958330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:34.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.172+0000 7f209022d740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f20881b50b0 con 0x556a7595f080 2024-04-02T14:42:34.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.172+0000 7f209022d740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f2088073b50 con 0x556a75955d10 2024-04-02T14:42:34.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.172+0000 7f209022d740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f2088070db0 con 0x556a7595c790 2024-04-02T14:42:34.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.172+0000 7f209022d740 10 client.0 _ll_register_callbacks cb 0x7f2078005d50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:42:34.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.173+0000 7f2085e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556a75955d10 0x556a75978040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:34.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.173+0000 7f2086800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556a7595c790 0x556a75978580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:34.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.173+0000 7f2085e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556a75955d10 0x556a75978040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:55054/0 (socket says 172.21.15.90:55054) 2024-04-02T14:42:34.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.173+0000 7f2085e00640 1 -- 172.21.15.90:0/3658987075 learned_addr learned my addr 172.21.15.90:0/3658987075 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:42:34.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.173+0000 7f2085e00640 1 --2- 172.21.15.90:0/3658987075 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556a7595f080 0x556a75958330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:34.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.173+0000 7f2086800640 1 -- 172.21.15.90:0/3658987075 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556a75955d10 msgr2=0x556a75978040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:42:34.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.173+0000 7f2086800640 1 --2- 172.21.15.90:0/3658987075 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556a75955d10 0x556a75978040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:34.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.173+0000 7f2086800640 1 -- 172.21.15.90:0/3658987075 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556a7595f080 msgr2=0x556a75958330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:42:34.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.173+0000 7f2086800640 1 --2- 172.21.15.90:0/3658987075 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556a7595f080 0x556a75958330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:34.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.173+0000 7f2086800640 1 -- 172.21.15.90:0/3658987075 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556a75958b30 con 0x556a7595c790 2024-04-02T14:42:34.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.173+0000 7f2085e00640 1 --2- 172.21.15.90:0/3658987075 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556a75955d10 0x556a75978040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:42:34.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.174+0000 7f2067e00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:42:34.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.174+0000 7f2086800640 1 --2- 172.21.15.90:0/3658987075 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556a7595c790 0x556a75978580 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x556a7595d180 tx=0x7f2078005c90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=936eeabbe030711e server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:42:34.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.175+0000 7f2067e00640 1 -- 172.21.15.90:0/3658987075 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2078034a80 con 0x556a7595c790 2024-04-02T14:42:34.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.175+0000 7f209022d740 1 -- 172.21.15.90:0/3658987075 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f207401e950 con 0x556a7595c790 2024-04-02T14:42:34.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.175+0000 7f2067e00640 1 -- 172.21.15.90:0/3658987075 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8550+0+0 (secure 0 0 0) 0x7f2078034880 con 0x556a7595c790 2024-04-02T14:42:34.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.176+0000 7f2067e00640 1 -- 172.21.15.90:0/3658987075 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f2078034a80 con 0x556a7595c790 2024-04-02T14:42:34.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.176+0000 7f209022d740 10 client.27613 fetch_fsmap learned FSMap version 27 2024-04-02T14:42:34.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.176+0000 7f209022d740 1 -- 172.21.15.90:0/3658987075 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x556a75959e30 con 0x556a7595c790 2024-04-02T14:42:34.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.176+0000 7f2067e00640 1 -- 172.21.15.90:0/3658987075 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f207401e950 con 0x556a7595c790 2024-04-02T14:42:34.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.176+0000 7f2067e00640 1 -- 172.21.15.90:0/3658987075 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f2078034c20 con 0x556a7595c790 2024-04-02T14:42:34.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.176+0000 7f209022d740 10 client.27613 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:42:34.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.176+0000 7f209022d740 10 client.27613 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-04-02T14:42:34.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.176+0000 7f209022d740 20 client.27613 populate_metadata read hostname 'smithi090' 2024-04-02T14:42:34.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.176+0000 7f209022d740 1 --2- 172.21.15.90:0/3658987075 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x556a759e7ba0 0x556a759e9f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:34.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.176+0000 7f209022d740 4 client.27613 mds_command: new command op to 24547 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-02T14:42:34.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.176+0000 7f209022d740 1 -- 172.21.15.90:0/3658987075 --> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] -- command(tid 0: {"prefix": "damage ls"}) -- 0x556a758c7c30 con 0x556a759e7ba0 2024-04-02T14:42:34.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.177+0000 7f2085e00640 1 --2- 172.21.15.90:0/3658987075 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x556a759e7ba0 0x556a759e9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:34.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.177+0000 7f2085e00640 1 --2- 172.21.15.90:0/3658987075 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x556a759e7ba0 0x556a759e9f90 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=2c7765e973aaa2e8 server_cookie=d5eaf2298fd8a869 in_seq=0 out_seq=0 2024-04-02T14:42:34.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.177+0000 7f2067e00640 10 client.27613 ms_handle_connect on v2:172.21.15.177:6838/664895663 2024-04-02T14:42:34.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.177+0000 7f2067e00640 1 -- 172.21.15.90:0/3658987075 <== mds.2 v2:172.21.15.177:6838/664895663 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x556a758c7c30 con 0x556a759e7ba0 2024-04-02T14:42:34.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.177+0000 7f2067e00640 10 client.27613 handle_command_reply: tid=0 2024-04-02T14:42:34.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.177+0000 7f209022d740 1 client.27613 shutdown 2024-04-02T14:42:34.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.177+0000 7f209022d740 1 -- 172.21.15.90:0/3658987075 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556a7595c790 msgr2=0x556a75978580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:34.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.177+0000 7f209022d740 1 --2- 172.21.15.90:0/3658987075 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556a7595c790 0x556a75978580 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x556a7595d180 tx=0x7f2078005c90 comp rx=0 tx=0).stop 2024-04-02T14:42:34.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.178+0000 7f209022d740 1 -- 172.21.15.90:0/3658987075 shutdown_connections 2024-04-02T14:42:34.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.178+0000 7f209022d740 1 --2- 172.21.15.90:0/3658987075 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x556a759e7ba0 0x556a759e9f90 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:34.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.178+0000 7f209022d740 1 --2- 172.21.15.90:0/3658987075 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556a7595f080 0x556a75958330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:34.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.178+0000 7f209022d740 1 --2- 172.21.15.90:0/3658987075 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556a7595c790 0x556a75978580 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:34.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.178+0000 7f209022d740 1 --2- 172.21.15.90:0/3658987075 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556a75955d10 0x556a75978040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:34.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.178+0000 7f209022d740 1 -- 172.21.15.90:0/3658987075 >> 172.21.15.90:0/3658987075 conn(0x556a7591ca50 msgr2=0x556a7594f010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:42:34.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.178+0000 7f2067e00640 0 client.27613 ms_handle_reset on v2:172.21.15.177:6838/664895663 2024-04-02T14:42:34.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.178+0000 7f209022d740 1 -- 172.21.15.90:0/3658987075 shutdown_connections 2024-04-02T14:42:34.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.178+0000 7f209022d740 1 -- 172.21.15.90:0/3658987075 wait complete. 2024-04-02T14:42:34.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.178+0000 7f209022d740 20 client.27613 trim_cache size 0 max 16384 2024-04-02T14:42:34.178 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-02T14:42:34.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.179+0000 7f208d400640 1 -- 172.21.15.90:0/2002787078 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f206c0a3ef0 msgr2=0x7f206c0a63b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:34.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.179+0000 7f208d400640 1 --2- 172.21.15.90:0/2002787078 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f206c0a3ef0 0x7f206c0a63b0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f2080005b40 tx=0x7f2080099060 comp rx=0 tx=0).stop 2024-04-02T14:42:34.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.180+0000 7f208d400640 1 -- 172.21.15.90:0/2002787078 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2088107ab0 msgr2=0x7f208819a980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:34.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.180+0000 7f208d400640 1 --2- 172.21.15.90:0/2002787078 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2088107ab0 0x7f208819a980 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f2080009fd0 tx=0x7f208000ef10 comp rx=0 tx=0).stop 2024-04-02T14:42:34.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.180+0000 7f208d400640 1 -- 172.21.15.90:0/2002787078 shutdown_connections 2024-04-02T14:42:34.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.180+0000 7f208d400640 1 --2- 172.21.15.90:0/2002787078 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f206c0a3ef0 0x7f206c0a63b0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:34.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.180+0000 7f208d400640 1 --2- 172.21.15.90:0/2002787078 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2088107ab0 0x7f208819a980 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:34.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.180+0000 7f208d400640 1 --2- 172.21.15.90:0/2002787078 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2088102460 0x7f208810cfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:34.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.180+0000 7f208d400640 1 --2- 172.21.15.90:0/2002787078 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2088101aa0 0x7f208810ca60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:34.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.180+0000 7f208d400640 1 -- 172.21.15.90:0/2002787078 >> 172.21.15.90:0/2002787078 conn(0x7f20880fd530 msgr2=0x7f20880ff920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:42:34.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.180+0000 7f208d400640 1 -- 172.21.15.90:0/2002787078 shutdown_connections 2024-04-02T14:42:34.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.180+0000 7f208d400640 1 -- 172.21.15.90:0/2002787078 wait complete. 2024-04-02T14:42:34.203 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-02T14:42:34.357 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:42:34 smithi092 ceph-mon[36497]: pgmap v235: 97 pgs: 97 active+clean; 4.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 28 MiB/s wr, 1.22k op/s 2024-04-02T14:42:34.433 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:42:34 smithi177 ceph-mon[36431]: pgmap v235: 97 pgs: 97 active+clean; 4.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 28 MiB/s wr, 1.22k op/s 2024-04-02T14:42:34.435 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:42:34 smithi090 ceph-mon[31674]: pgmap v235: 97 pgs: 97 active+clean; 4.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 28 MiB/s wr, 1.22k op/s 2024-04-02T14:42:34.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.493+0000 7fd922800640 1 -- 172.21.15.90:0/2473380731 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd91c10ec40 msgr2=0x7fd91c108b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:34.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.493+0000 7fd922800640 1 --2- 172.21.15.90:0/2473380731 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd91c10ec40 0x7fd91c108b00 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7fd904009860 tx=0x7fd90402bb70 comp rx=0 tx=0).stop 2024-04-02T14:42:34.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.493+0000 7fd922800640 1 -- 172.21.15.90:0/2473380731 shutdown_connections 2024-04-02T14:42:34.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.493+0000 7fd922800640 1 --2- 172.21.15.90:0/2473380731 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd91c109040 0x7fd91c1094c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:34.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.493+0000 7fd922800640 1 --2- 172.21.15.90:0/2473380731 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd91c10ec40 0x7fd91c108b00 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:34.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.493+0000 7fd922800640 1 --2- 172.21.15.90:0/2473380731 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd91c10e320 0x7fd91c10e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:34.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.493+0000 7fd922800640 1 -- 172.21.15.90:0/2473380731 >> 172.21.15.90:0/2473380731 conn(0x7fd91c06d040 msgr2=0x7fd91c06d450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:42:34.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.494+0000 7fd922800640 1 -- 172.21.15.90:0/2473380731 shutdown_connections 2024-04-02T14:42:34.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.494+0000 7fd922800640 1 -- 172.21.15.90:0/2473380731 wait complete. 2024-04-02T14:42:34.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.495+0000 7fd922800640 1 Processor -- start 2024-04-02T14:42:34.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.496+0000 7fd922800640 1 -- start start 2024-04-02T14:42:34.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.497+0000 7fd922800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd91c109040 0x7fd91c1a2e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:34.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.497+0000 7fd922800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd91c10e320 0x7fd91c1a3360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:34.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.497+0000 7fd922800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd91c10ec40 0x7fd91c1a76e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:34.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.497+0000 7fd922800640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fd91c112460 con 0x7fd91c109040 2024-04-02T14:42:34.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.497+0000 7fd922800640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fd91c1122e0 con 0x7fd91c10ec40 2024-04-02T14:42:34.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.497+0000 7fd922800640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fd91c11ca80 con 0x7fd91c10e320 2024-04-02T14:42:34.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.497+0000 7fd913e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd91c10e320 0x7fd91c1a3360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:34.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.497+0000 7fd913e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd91c10e320 0x7fd91c1a3360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:47398/0 (socket says 172.21.15.90:47398) 2024-04-02T14:42:34.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.497+0000 7fd913e00640 1 -- 172.21.15.90:0/1709275161 learned_addr learned my addr 172.21.15.90:0/1709275161 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:42:34.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.497+0000 7fd91b400640 1 --2- 172.21.15.90:0/1709275161 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd91c109040 0x7fd91c1a2e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:34.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.498+0000 7fd91be00640 1 --2- 172.21.15.90:0/1709275161 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd91c10ec40 0x7fd91c1a76e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:34.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.498+0000 7fd91b400640 1 -- 172.21.15.90:0/1709275161 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd91c10e320 msgr2=0x7fd91c1a3360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:34.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.498+0000 7fd91b400640 1 --2- 172.21.15.90:0/1709275161 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd91c10e320 0x7fd91c1a3360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:34.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.498+0000 7fd91b400640 1 -- 172.21.15.90:0/1709275161 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd91c10ec40 msgr2=0x7fd91c1a76e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:34.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.498+0000 7fd91b400640 1 --2- 172.21.15.90:0/1709275161 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd91c10ec40 0x7fd91c1a76e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:34.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.498+0000 7fd91b400640 1 -- 172.21.15.90:0/1709275161 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd91c1a7e10 con 0x7fd91c109040 2024-04-02T14:42:34.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.499+0000 7fd913e00640 1 --2- 172.21.15.90:0/1709275161 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd91c10e320 0x7fd91c1a3360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:42:34.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.500+0000 7fd91b400640 1 --2- 172.21.15.90:0/1709275161 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd91c109040 0x7fd91c1a2e20 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fd90c00b3f0 tx=0x7fd90c00b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:42:34.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.500+0000 7fd918c00640 1 -- 172.21.15.90:0/1709275161 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd90c015020 con 0x7fd91c109040 2024-04-02T14:42:34.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.500+0000 7fd918c00640 1 -- 172.21.15.90:0/1709275161 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd90c011d10 con 0x7fd91c109040 2024-04-02T14:42:34.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.500+0000 7fd918c00640 1 -- 172.21.15.90:0/1709275161 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd90c00f2e0 con 0x7fd91c109040 2024-04-02T14:42:34.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.500+0000 7fd922800640 1 -- 172.21.15.90:0/1709275161 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd91c1a8100 con 0x7fd91c109040 2024-04-02T14:42:34.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.500+0000 7fd922800640 1 -- 172.21.15.90:0/1709275161 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd91c1af990 con 0x7fd91c109040 2024-04-02T14:42:34.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.511+0000 7fd925439740 1 Processor -- start 2024-04-02T14:42:34.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.511+0000 7fd925439740 1 -- start start 2024-04-02T14:42:34.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.511+0000 7fd925439740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56526cb899f0 0x56526cb88740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:34.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.511+0000 7fd918c00640 1 -- 172.21.15.90:0/1709275161 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fd90c018020 con 0x7fd91c109040 2024-04-02T14:42:34.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.512+0000 7fd925439740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56526cb88d10 0x56526cb89190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:34.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.512+0000 7fd91b400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56526cb88d10 0x56526cb89190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:34.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.512+0000 7fd91b400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56526cb88d10 0x56526cb89190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:55072/0 (socket says 172.21.15.90:55072) 2024-04-02T14:42:34.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.512+0000 7fd91b400640 1 -- 172.21.15.90:0/3612235636 learned_addr learned my addr 172.21.15.90:0/3612235636 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:42:34.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.513+0000 7fd925439740 1 --2- 172.21.15.90:0/3612235636 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56526cb9a890 0x56526cb9cc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:34.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.513+0000 7fd925439740 1 -- 172.21.15.90:0/3612235636 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x56526cac9400 con 0x56526cb899f0 2024-04-02T14:42:34.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.513+0000 7fd925439740 1 -- 172.21.15.90:0/3612235636 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x56526caadf10 con 0x56526cb88d10 2024-04-02T14:42:34.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.513+0000 7fd925439740 1 -- 172.21.15.90:0/3612235636 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x56526caed6f0 con 0x56526cb9a890 2024-04-02T14:42:34.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.513+0000 7fd913e00640 1 --2- 172.21.15.90:0/3612235636 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56526cb9a890 0x56526cb9cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:34.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.513+0000 7fd91b400640 1 -- 172.21.15.90:0/3612235636 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56526cb9a890 msgr2=0x56526cb9cc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:42:34.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.514+0000 7fd91b400640 1 --2- 172.21.15.90:0/3612235636 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56526cb9a890 0x56526cb9cc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:34.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.514+0000 7fd91b400640 1 -- 172.21.15.90:0/3612235636 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56526cb899f0 msgr2=0x56526cb88740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:42:34.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.514+0000 7fd91b400640 1 --2- 172.21.15.90:0/3612235636 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56526cb899f0 0x56526cb88740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:34.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.514+0000 7fd91b400640 1 -- 172.21.15.90:0/3612235636 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56526c8013f0 con 0x56526cb88d10 2024-04-02T14:42:34.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.514+0000 7fd918c00640 1 --2- 172.21.15.90:0/1709275161 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fd9000a3e20 0x7fd9000a62e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:34.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.514+0000 7fd918c00640 1 -- 172.21.15.90:0/1709275161 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8550+0+0 (secure 0 0 0) 0x7fd90c0d09b0 con 0x7fd91c109040 2024-04-02T14:42:34.514 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.516+0000 7fd91b400640 1 --2- 172.21.15.90:0/3612235636 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56526cb88d10 0x56526cb89190 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7fd90c0ce6b0 tx=0x7fd90c0d8570 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2a09e82eb8ad975c server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:42:34.514 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.517+0000 7fd91be00640 1 --2- 172.21.15.90:0/1709275161 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fd9000a3e20 0x7fd9000a62e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:34.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.518+0000 7fd910c00640 1 -- 172.21.15.90:0/3612235636 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd90c099260 con 0x56526cb88d10 2024-04-02T14:42:34.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.518+0000 7fd910c00640 1 -- 172.21.15.90:0/3612235636 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd90c0bd070 con 0x56526cb88d10 2024-04-02T14:42:34.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.518+0000 7fd910c00640 1 -- 172.21.15.90:0/3612235636 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd90c09c920 con 0x56526cb88d10 2024-04-02T14:42:34.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.520+0000 7fd925439740 1 -- 172.21.15.90:0/3612235636 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56526cb88d10 msgr2=0x56526cb89190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:34.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.520+0000 7fd91be00640 1 --2- 172.21.15.90:0/1709275161 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fd9000a3e20 0x7fd9000a62e0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fd914009980 tx=0x7fd914009910 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:42:34.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.520+0000 7fd925439740 1 --2- 172.21.15.90:0/3612235636 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56526cb88d10 0x56526cb89190 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7fd90c0ce6b0 tx=0x7fd90c0d8570 comp rx=0 tx=0).stop 2024-04-02T14:42:34.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.521+0000 7fd925439740 1 -- 172.21.15.90:0/3612235636 shutdown_connections 2024-04-02T14:42:34.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.521+0000 7fd925439740 1 --2- 172.21.15.90:0/3612235636 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56526cb9a890 0x56526cb9cc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:34.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.521+0000 7fd925439740 1 --2- 172.21.15.90:0/3612235636 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56526cb88d10 0x56526cb89190 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:34.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.521+0000 7fd925439740 1 --2- 172.21.15.90:0/3612235636 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56526cb899f0 0x56526cb88740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:34.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.521+0000 7fd925439740 1 -- 172.21.15.90:0/3612235636 >> 172.21.15.90:0/3612235636 conn(0x56526cb4fa50 msgr2=0x56526cb83760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:42:34.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.521+0000 7fd925439740 1 -- 172.21.15.90:0/3612235636 shutdown_connections 2024-04-02T14:42:34.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.521+0000 7fd925439740 1 -- 172.21.15.90:0/3612235636 wait complete. 2024-04-02T14:42:34.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.523+0000 7fd925439740 1 -- start start 2024-04-02T14:42:34.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.524+0000 7fd925439740 1 Processor -- start 2024-04-02T14:42:34.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.525+0000 7fd925439740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56526cb88d10 0x56526cb978a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:34.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.525+0000 7fd925439740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56526cb899f0 0x56526cb97de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:34.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.525+0000 7fd925439740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56526cb9a890 0x56526cb98320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:34.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.525+0000 7fd925439740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x56526cae96a0 con 0x56526cb9a890 2024-04-02T14:42:34.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.525+0000 7fd925439740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x56526ca95980 con 0x56526cb899f0 2024-04-02T14:42:34.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.525+0000 7fd925439740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x56526caa8cb0 con 0x56526cb88d10 2024-04-02T14:42:34.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.525+0000 7fd925439740 10 client.0 _ll_register_callbacks cb 0x56526caa86a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:42:34.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.526+0000 7fd913e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56526cb899f0 0x56526cb97de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:34.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.526+0000 7fd91b400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56526cb88d10 0x56526cb978a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:34.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.526+0000 7fd91be00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56526cb9a890 0x56526cb98320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:34.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.526+0000 7fd913e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56526cb899f0 0x56526cb97de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:55074/0 (socket says 172.21.15.90:55074) 2024-04-02T14:42:34.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.526+0000 7fd913e00640 1 -- 172.21.15.90:0/681492771 learned_addr learned my addr 172.21.15.90:0/681492771 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:42:34.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.527+0000 7fd91b400640 1 -- 172.21.15.90:0/681492771 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56526cb899f0 msgr2=0x56526cb97de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:42:34.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.527+0000 7fd91b400640 1 --2- 172.21.15.90:0/681492771 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56526cb899f0 0x56526cb97de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:34.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.527+0000 7fd91b400640 1 -- 172.21.15.90:0/681492771 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56526cb9a890 msgr2=0x56526cb98320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:42:34.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.527+0000 7fd91b400640 1 --2- 172.21.15.90:0/681492771 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56526cb9a890 0x56526cb98320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:34.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.527+0000 7fd91b400640 1 -- 172.21.15.90:0/681492771 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56526c8cb7d0 con 0x56526cb88d10 2024-04-02T14:42:34.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.527+0000 7fd91be00640 1 --2- 172.21.15.90:0/681492771 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56526cb9a890 0x56526cb98320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:42:34.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.528+0000 7fd91b400640 1 --2- 172.21.15.90:0/681492771 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56526cb88d10 0x56526cb978a0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fd90c0ce6e0 tx=0x7fd90c010f20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=73c743346eff09ae server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:42:34.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.528+0000 7fd909600640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:42:34.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.528+0000 7fd909600640 1 -- 172.21.15.90:0/681492771 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd90c09cb00 con 0x56526cb88d10 2024-04-02T14:42:34.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.529+0000 7fd909600640 1 -- 172.21.15.90:0/681492771 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8550+0+0 (secure 0 0 0) 0x7fd90c099630 con 0x56526cb88d10 2024-04-02T14:42:34.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.529+0000 7fd925439740 1 -- 172.21.15.90:0/681492771 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd90c0bd070 con 0x56526cb88d10 2024-04-02T14:42:34.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.529+0000 7fd909600640 1 -- 172.21.15.90:0/681492771 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fd90c09cd90 con 0x56526cb88d10 2024-04-02T14:42:34.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.530+0000 7fd925439740 10 client.27619 fetch_fsmap learned FSMap version 27 2024-04-02T14:42:34.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.530+0000 7fd925439740 1 -- 172.21.15.90:0/681492771 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x56526cba95f0 con 0x56526cb88d10 2024-04-02T14:42:34.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.530+0000 7fd909600640 1 -- 172.21.15.90:0/681492771 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd90c0bd070 con 0x56526cb88d10 2024-04-02T14:42:34.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.531+0000 7fd909600640 1 -- 172.21.15.90:0/681492771 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fd90c09f050 con 0x56526cb88d10 2024-04-02T14:42:34.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.531+0000 7fd925439740 10 client.27619 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:42:34.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.531+0000 7fd925439740 10 client.27619 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2024-04-02T14:42:34.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.531+0000 7fd925439740 20 client.27619 populate_metadata read hostname 'smithi090' 2024-04-02T14:42:34.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.531+0000 7fd925439740 1 --2- 172.21.15.90:0/681492771 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x56526cbaa940 0x56526cbacd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:34.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.531+0000 7fd925439740 4 client.27619 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:42:34.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.531+0000 7fd925439740 1 -- 172.21.15.90:0/681492771 --> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56526caad040 con 0x56526cbaa940 2024-04-02T14:42:34.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.533+0000 7fd913e00640 1 --2- 172.21.15.90:0/681492771 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x56526cbaa940 0x56526cbacd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:34.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.534+0000 7fd913e00640 1 --2- 172.21.15.90:0/681492771 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x56526cbaa940 0x56526cbacd30 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=65cfabcf32f44783 server_cookie=4c6b486b7d13468f in_seq=0 out_seq=0 2024-04-02T14:42:34.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.534+0000 7fd909600640 10 client.27619 ms_handle_connect on v2:172.21.15.90:6838/1498915870 2024-04-02T14:42:34.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.535+0000 7fd909600640 1 -- 172.21.15.90:0/681492771 <== mds.1 v2:172.21.15.90:6838/1498915870 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56526caad040 con 0x56526cbaa940 2024-04-02T14:42:34.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.535+0000 7fd909600640 10 client.27619 handle_command_reply: tid=0 2024-04-02T14:42:34.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.535+0000 7fd925439740 1 client.27619 shutdown 2024-04-02T14:42:34.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.536+0000 7fd925439740 1 -- 172.21.15.90:0/681492771 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56526cb88d10 msgr2=0x56526cb978a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:34.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.536+0000 7fd925439740 1 --2- 172.21.15.90:0/681492771 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56526cb88d10 0x56526cb978a0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fd90c0ce6e0 tx=0x7fd90c010f20 comp rx=0 tx=0).stop 2024-04-02T14:42:34.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.536+0000 7fd925439740 1 -- 172.21.15.90:0/681492771 shutdown_connections 2024-04-02T14:42:34.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.536+0000 7fd925439740 1 --2- 172.21.15.90:0/681492771 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x56526cbaa940 0x56526cbacd30 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:34.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.536+0000 7fd925439740 1 --2- 172.21.15.90:0/681492771 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56526cb9a890 0x56526cb98320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:34.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.536+0000 7fd909600640 0 client.27619 ms_handle_reset on v2:172.21.15.90:6838/1498915870 2024-04-02T14:42:34.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.536+0000 7fd925439740 1 --2- 172.21.15.90:0/681492771 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56526cb899f0 0x56526cb97de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:34.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.536+0000 7fd925439740 1 --2- 172.21.15.90:0/681492771 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56526cb88d10 0x56526cb978a0 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:34.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.536+0000 7fd925439740 1 -- 172.21.15.90:0/681492771 >> 172.21.15.90:0/681492771 conn(0x56526cb4fa50 msgr2=0x56526cb854b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:42:34.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.536+0000 7fd925439740 1 -- 172.21.15.90:0/681492771 shutdown_connections 2024-04-02T14:42:34.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.537+0000 7fd925439740 1 -- 172.21.15.90:0/681492771 wait complete. 2024-04-02T14:42:34.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.537+0000 7fd925439740 20 client.27619 trim_cache size 0 max 16384 2024-04-02T14:42:34.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.562+0000 7fd925439740 1 Processor -- start 2024-04-02T14:42:34.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.563+0000 7fd925439740 1 -- start start 2024-04-02T14:42:34.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.563+0000 7fd925439740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56526cb88d10 0x56526cb99fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:34.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.563+0000 7fd925439740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56526cb899f0 0x56526cb96a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:34.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.563+0000 7fd925439740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56526cb9a890 0x56526cb82720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:34.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.563+0000 7fd925439740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fd91c1b3fa0 con 0x56526cb899f0 2024-04-02T14:42:34.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.563+0000 7fd925439740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fd91c1b5ae0 con 0x56526cb88d10 2024-04-02T14:42:34.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.563+0000 7fd925439740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fd91c1b4670 con 0x56526cb9a890 2024-04-02T14:42:34.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.564+0000 7fd91be00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56526cb9a890 0x56526cb82720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:34.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.564+0000 7fd91be00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56526cb9a890 0x56526cb82720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:47414/0 (socket says 172.21.15.90:47414) 2024-04-02T14:42:34.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.564+0000 7fd91be00640 1 -- 172.21.15.90:0/1351666156 learned_addr learned my addr 172.21.15.90:0/1351666156 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:42:34.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.564+0000 7fd91b400640 1 --2- 172.21.15.90:0/1351666156 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56526cb88d10 0x56526cb99fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:34.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.564+0000 7fd913e00640 1 --2- 172.21.15.90:0/1351666156 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56526cb899f0 0x56526cb96a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:34.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.565+0000 7fd91b400640 1 -- 172.21.15.90:0/1351666156 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56526cb9a890 msgr2=0x56526cb82720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:42:34.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.565+0000 7fd91b400640 1 --2- 172.21.15.90:0/1351666156 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56526cb9a890 0x56526cb82720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:34.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.565+0000 7fd91b400640 1 -- 172.21.15.90:0/1351666156 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56526cb899f0 msgr2=0x56526cb96a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:42:34.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.565+0000 7fd91b400640 1 --2- 172.21.15.90:0/1351666156 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56526cb899f0 0x56526cb96a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:34.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.565+0000 7fd91b400640 1 -- 172.21.15.90:0/1351666156 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56526cb87d40 con 0x56526cb88d10 2024-04-02T14:42:34.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.565+0000 7fd91b400640 1 --2- 172.21.15.90:0/1351666156 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56526cb88d10 0x56526cb99fd0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7fd90c09cf50 tx=0x7fd90c0d8570 comp rx=0 tx=0).ready entity=mon.2 client_cookie=49ddac41493a6027 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:42:34.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.565+0000 7fd90be00640 1 -- 172.21.15.90:0/1351666156 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd90c099260 con 0x56526cb88d10 2024-04-02T14:42:34.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.566+0000 7fd90be00640 1 -- 172.21.15.90:0/1351666156 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd90c0bd050 con 0x56526cb88d10 2024-04-02T14:42:34.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.566+0000 7fd90be00640 1 -- 172.21.15.90:0/1351666156 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd90c09c920 con 0x56526cb88d10 2024-04-02T14:42:34.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.567+0000 7fd925439740 1 -- 172.21.15.90:0/1351666156 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56526cb88d10 msgr2=0x56526cb99fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:34.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.567+0000 7fd925439740 1 --2- 172.21.15.90:0/1351666156 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56526cb88d10 0x56526cb99fd0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7fd90c09cf50 tx=0x7fd90c0d8570 comp rx=0 tx=0).stop 2024-04-02T14:42:34.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.567+0000 7fd925439740 1 -- 172.21.15.90:0/1351666156 shutdown_connections 2024-04-02T14:42:34.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.567+0000 7fd925439740 1 --2- 172.21.15.90:0/1351666156 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56526cb9a890 0x56526cb82720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:34.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.567+0000 7fd925439740 1 --2- 172.21.15.90:0/1351666156 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56526cb899f0 0x56526cb96a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:34.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.567+0000 7fd925439740 1 --2- 172.21.15.90:0/1351666156 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56526cb88d10 0x56526cb99fd0 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:34.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.567+0000 7fd925439740 1 -- 172.21.15.90:0/1351666156 >> 172.21.15.90:0/1351666156 conn(0x56526cb4fa50 msgr2=0x56526cb82010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:42:34.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.567+0000 7fd925439740 1 -- 172.21.15.90:0/1351666156 shutdown_connections 2024-04-02T14:42:34.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.567+0000 7fd925439740 1 -- 172.21.15.90:0/1351666156 wait complete. 2024-04-02T14:42:34.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.568+0000 7fd925439740 1 -- start start 2024-04-02T14:42:34.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.568+0000 7fd925439740 1 Processor -- start 2024-04-02T14:42:34.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.568+0000 7fd925439740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56526cb88d10 0x56526cb944c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:34.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.568+0000 7fd925439740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56526cb899f0 0x56526cb94a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:34.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.568+0000 7fd925439740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56526cb9a890 0x56526cb94f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:34.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.569+0000 7fd925439740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fd91c1b5410 con 0x56526cb88d10 2024-04-02T14:42:34.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.569+0000 7fd925439740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fd91c1b4d40 con 0x56526cb9a890 2024-04-02T14:42:34.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.569+0000 7fd925439740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fd91c1b1fc0 con 0x56526cb899f0 2024-04-02T14:42:34.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.569+0000 7fd925439740 10 client.0 _ll_register_callbacks cb 0x7fd90c0a5c30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:42:34.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.569+0000 7fd913e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56526cb899f0 0x56526cb94a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:34.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.569+0000 7fd91b400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56526cb88d10 0x56526cb944c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:34.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.569+0000 7fd913e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56526cb899f0 0x56526cb94a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:47424/0 (socket says 172.21.15.90:47424) 2024-04-02T14:42:34.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.569+0000 7fd913e00640 1 -- 172.21.15.90:0/228224904 learned_addr learned my addr 172.21.15.90:0/228224904 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:42:34.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.569+0000 7fd91be00640 1 --2- 172.21.15.90:0/228224904 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56526cb9a890 0x56526cb94f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:34.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.570+0000 7fd91be00640 1 -- 172.21.15.90:0/228224904 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56526cb899f0 msgr2=0x56526cb94a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:42:34.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.570+0000 7fd91be00640 1 --2- 172.21.15.90:0/228224904 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56526cb899f0 0x56526cb94a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:34.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.570+0000 7fd91be00640 1 -- 172.21.15.90:0/228224904 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56526cb88d10 msgr2=0x56526cb944c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:42:34.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.570+0000 7fd91be00640 1 --2- 172.21.15.90:0/228224904 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56526cb88d10 0x56526cb944c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:34.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.570+0000 7fd91be00640 1 -- 172.21.15.90:0/228224904 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56526cc0fb10 con 0x56526cb9a890 2024-04-02T14:42:34.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.570+0000 7fd91b400640 1 --2- 172.21.15.90:0/228224904 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56526cb88d10 0x56526cb944c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:42:34.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.570+0000 7fd91be00640 1 --2- 172.21.15.90:0/228224904 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56526cb9a890 0x56526cb94f40 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x56526cb84880 tx=0x7fd914009690 comp rx=0 tx=0).ready entity=mon.2 client_cookie=770665982dd829eb server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:42:34.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.570+0000 7fd909600640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:42:34.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.571+0000 7fd909600640 1 -- 172.21.15.90:0/228224904 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd91400e600 con 0x56526cb9a890 2024-04-02T14:42:34.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.571+0000 7fd909600640 1 -- 172.21.15.90:0/228224904 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8550+0+0 (secure 0 0 0) 0x7fd91400e7a0 con 0x56526cb9a890 2024-04-02T14:42:34.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.571+0000 7fd925439740 1 -- 172.21.15.90:0/228224904 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd90c0bd050 con 0x56526cb9a890 2024-04-02T14:42:34.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.571+0000 7fd909600640 1 -- 172.21.15.90:0/228224904 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fd914037050 con 0x56526cb9a890 2024-04-02T14:42:34.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.571+0000 7fd925439740 10 client.27773 fetch_fsmap learned FSMap version 27 2024-04-02T14:42:34.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.571+0000 7fd925439740 1 -- 172.21.15.90:0/228224904 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x56526cc10e20 con 0x56526cb9a890 2024-04-02T14:42:34.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.571+0000 7fd909600640 1 -- 172.21.15.90:0/228224904 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd91402e070 con 0x56526cb9a890 2024-04-02T14:42:34.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.571+0000 7fd909600640 1 -- 172.21.15.90:0/228224904 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fd914029b70 con 0x56526cb9a890 2024-04-02T14:42:34.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.572+0000 7fd925439740 10 client.27773 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:42:34.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.572+0000 7fd925439740 10 client.27773 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2024-04-02T14:42:34.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.572+0000 7fd925439740 20 client.27773 populate_metadata read hostname 'smithi090' 2024-04-02T14:42:34.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.572+0000 7fd925439740 1 --2- 172.21.15.90:0/228224904 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x56526cc11620 0x56526cc13a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:34.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.572+0000 7fd925439740 4 client.27773 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-02T14:42:34.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.572+0000 7fd925439740 1 -- 172.21.15.90:0/228224904 --> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7fd90c0bd050 con 0x56526cc11620 2024-04-02T14:42:34.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.572+0000 7fd91b400640 1 --2- 172.21.15.90:0/228224904 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x56526cc11620 0x56526cc13a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:34.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.572+0000 7fd91b400640 1 --2- 172.21.15.90:0/228224904 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x56526cc11620 0x56526cc13a40 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=12e12374aa393231 server_cookie=eb9e95b0427418b2 in_seq=0 out_seq=0 2024-04-02T14:42:34.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.572+0000 7fd909600640 10 client.27773 ms_handle_connect on v2:172.21.15.90:6838/1498915870 2024-04-02T14:42:34.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.573+0000 7fd909600640 1 -- 172.21.15.90:0/228224904 <== mds.1 v2:172.21.15.90:6838/1498915870 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7fd90c0bd050 con 0x56526cc11620 2024-04-02T14:42:34.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.573+0000 7fd909600640 10 client.27773 handle_command_reply: tid=0 2024-04-02T14:42:34.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.573+0000 7fd925439740 1 client.27773 shutdown 2024-04-02T14:42:34.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.573+0000 7fd925439740 1 -- 172.21.15.90:0/228224904 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56526cb9a890 msgr2=0x56526cb94f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:34.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.573+0000 7fd925439740 1 --2- 172.21.15.90:0/228224904 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56526cb9a890 0x56526cb94f40 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x56526cb84880 tx=0x7fd914009690 comp rx=0 tx=0).stop 2024-04-02T14:42:34.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.573+0000 7fd925439740 1 -- 172.21.15.90:0/228224904 shutdown_connections 2024-04-02T14:42:34.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.573+0000 7fd925439740 1 --2- 172.21.15.90:0/228224904 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x56526cc11620 0x56526cc13a40 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:34.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.573+0000 7fd925439740 1 --2- 172.21.15.90:0/228224904 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56526cb9a890 0x56526cb94f40 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:34.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.573+0000 7fd925439740 1 --2- 172.21.15.90:0/228224904 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56526cb899f0 0x56526cb94a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:34.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.573+0000 7fd925439740 1 --2- 172.21.15.90:0/228224904 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56526cb88d10 0x56526cb944c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:34.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.573+0000 7fd925439740 1 -- 172.21.15.90:0/228224904 >> 172.21.15.90:0/228224904 conn(0x56526cb4fa50 msgr2=0x56526cb96880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:42:34.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.573+0000 7fd909600640 0 client.27773 ms_handle_reset on v2:172.21.15.90:6838/1498915870 2024-04-02T14:42:34.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.573+0000 7fd925439740 1 -- 172.21.15.90:0/228224904 shutdown_connections 2024-04-02T14:42:34.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.573+0000 7fd925439740 1 -- 172.21.15.90:0/228224904 wait complete. 2024-04-02T14:42:34.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.573+0000 7fd925439740 20 client.27773 trim_cache size 0 max 16384 2024-04-02T14:42:34.574 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-02T14:42:34.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.574+0000 7fd922800640 1 -- 172.21.15.90:0/1709275161 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fd9000a3e20 msgr2=0x7fd9000a62e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:34.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.575+0000 7fd922800640 1 --2- 172.21.15.90:0/1709275161 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fd9000a3e20 0x7fd9000a62e0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fd914009980 tx=0x7fd914009910 comp rx=0 tx=0).stop 2024-04-02T14:42:34.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.575+0000 7fd922800640 1 -- 172.21.15.90:0/1709275161 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd91c109040 msgr2=0x7fd91c1a2e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:34.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.575+0000 7fd922800640 1 --2- 172.21.15.90:0/1709275161 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd91c109040 0x7fd91c1a2e20 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fd90c00b3f0 tx=0x7fd90c00b8b0 comp rx=0 tx=0).stop 2024-04-02T14:42:34.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.575+0000 7fd922800640 1 -- 172.21.15.90:0/1709275161 shutdown_connections 2024-04-02T14:42:34.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.575+0000 7fd922800640 1 --2- 172.21.15.90:0/1709275161 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fd9000a3e20 0x7fd9000a62e0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:34.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.575+0000 7fd922800640 1 --2- 172.21.15.90:0/1709275161 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd91c10ec40 0x7fd91c1a76e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:34.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.575+0000 7fd922800640 1 --2- 172.21.15.90:0/1709275161 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd91c10e320 0x7fd91c1a3360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:34.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.575+0000 7fd922800640 1 --2- 172.21.15.90:0/1709275161 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd91c109040 0x7fd91c1a2e20 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:34.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.575+0000 7fd922800640 1 -- 172.21.15.90:0/1709275161 >> 172.21.15.90:0/1709275161 conn(0x7fd91c06d040 msgr2=0x7fd91c10c0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:42:34.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.576+0000 7fd922800640 1 -- 172.21.15.90:0/1709275161 shutdown_connections 2024-04-02T14:42:34.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.576+0000 7fd922800640 1 -- 172.21.15.90:0/1709275161 wait complete. 2024-04-02T14:42:34.597 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-02T14:42:34.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.861+0000 7fe141200640 1 -- 172.21.15.90:0/3655200547 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe13c108fe0 msgr2=0x7fe13c110bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:34.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.861+0000 7fe141200640 1 --2- 172.21.15.90:0/3655200547 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe13c108fe0 0x7fe13c110bb0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fe134009be0 tx=0x7fe13402bef0 comp rx=0 tx=0).stop 2024-04-02T14:42:34.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.861+0000 7fe141200640 1 -- 172.21.15.90:0/3655200547 shutdown_connections 2024-04-02T14:42:34.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.861+0000 7fe141200640 1 --2- 172.21.15.90:0/3655200547 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe13c108fe0 0x7fe13c110bb0 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:34.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.861+0000 7fe141200640 1 --2- 172.21.15.90:0/3655200547 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe13c1048c0 0x7fe13c108900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:34.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.861+0000 7fe141200640 1 --2- 172.21.15.90:0/3655200547 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe13c103f10 0x7fe13c1042f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:34.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.861+0000 7fe141200640 1 -- 172.21.15.90:0/3655200547 >> 172.21.15.90:0/3655200547 conn(0x7fe13c0ff770 msgr2=0x7fe13c101b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:42:34.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.861+0000 7fe141200640 1 -- 172.21.15.90:0/3655200547 shutdown_connections 2024-04-02T14:42:34.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.862+0000 7fe141200640 1 -- 172.21.15.90:0/3655200547 wait complete. 2024-04-02T14:42:34.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.863+0000 7fe141200640 1 Processor -- start 2024-04-02T14:42:34.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.863+0000 7fe141200640 1 -- start start 2024-04-02T14:42:34.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.863+0000 7fe141200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe13c103f10 0x7fe13c19a570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:34.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.863+0000 7fe141200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe13c108fe0 0x7fe13c19aab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:34.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.863+0000 7fe141200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe13c19ee30 0x7fe13c19f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:34.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.863+0000 7fe141200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fe13c114090 con 0x7fe13c103f10 2024-04-02T14:42:34.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.863+0000 7fe141200640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fe13c113f10 con 0x7fe13c108fe0 2024-04-02T14:42:34.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.863+0000 7fe141200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fe13c114210 con 0x7fe13c19ee30 2024-04-02T14:42:34.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.864+0000 7fe139400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe13c108fe0 0x7fe13c19aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:34.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.865+0000 7fe13a800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe13c19ee30 0x7fe13c19f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:34.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.865+0000 7fe13a800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe13c19ee30 0x7fe13c19f2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:47454/0 (socket says 172.21.15.90:47454) 2024-04-02T14:42:34.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.865+0000 7fe13a800640 1 -- 172.21.15.90:0/2058479297 learned_addr learned my addr 172.21.15.90:0/2058479297 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:42:34.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.865+0000 7fe139e00640 1 --2- 172.21.15.90:0/2058479297 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe13c103f10 0x7fe13c19a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:34.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.865+0000 7fe13a800640 1 -- 172.21.15.90:0/2058479297 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe13c108fe0 msgr2=0x7fe13c19aab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:34.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.865+0000 7fe13a800640 1 --2- 172.21.15.90:0/2058479297 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe13c108fe0 0x7fe13c19aab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:34.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.865+0000 7fe13a800640 1 -- 172.21.15.90:0/2058479297 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe13c103f10 msgr2=0x7fe13c19a570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:34.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.865+0000 7fe13a800640 1 --2- 172.21.15.90:0/2058479297 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe13c103f10 0x7fe13c19a570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:34.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.865+0000 7fe13a800640 1 -- 172.21.15.90:0/2058479297 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe13c19f920 con 0x7fe13c19ee30 2024-04-02T14:42:34.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.866+0000 7fe13a800640 1 --2- 172.21.15.90:0/2058479297 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe13c19ee30 0x7fe13c19f2e0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7fe1340096f0 tx=0x7fe1340336b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:42:34.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.866+0000 7fe132a00640 1 -- 172.21.15.90:0/2058479297 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe134043020 con 0x7fe13c19ee30 2024-04-02T14:42:34.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.866+0000 7fe141200640 1 -- 172.21.15.90:0/2058479297 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe13c19fbb0 con 0x7fe13c19ee30 2024-04-02T14:42:34.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.866+0000 7fe141200640 1 -- 172.21.15.90:0/2058479297 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe13c076260 con 0x7fe13c19ee30 2024-04-02T14:42:34.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.867+0000 7fe132a00640 1 -- 172.21.15.90:0/2058479297 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe1340338a0 con 0x7fe13c19ee30 2024-04-02T14:42:34.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.867+0000 7fe132a00640 1 -- 172.21.15.90:0/2058479297 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe13403e6b0 con 0x7fe13c19ee30 2024-04-02T14:42:34.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.872+0000 7fe132a00640 1 -- 172.21.15.90:0/2058479297 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fe134046020 con 0x7fe13c19ee30 2024-04-02T14:42:34.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.874+0000 7fe143f9c740 1 Processor -- start 2024-04-02T14:42:34.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.876+0000 7fe132a00640 1 --2- 172.21.15.90:0/2058479297 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fe1200a3e20 0x7fe1200a62e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:34.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.876+0000 7fe132a00640 1 -- 172.21.15.90:0/2058479297 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8550+0+0 (secure 0 0 0) 0x7fe1340f2e40 con 0x7fe13c19ee30 2024-04-02T14:42:34.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.877+0000 7fe143f9c740 1 -- start start 2024-04-02T14:42:34.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.877+0000 7fe139400640 1 --2- 172.21.15.90:0/2058479297 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fe1200a3e20 0x7fe1200a62e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:34.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.877+0000 7fe143f9c740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bae22409f0 0x55bae223f740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:34.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.877+0000 7fe143f9c740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bae223fd10 0x55bae2240190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:34.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.877+0000 7fe143f9c740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55bae2249080 0x55bae224b470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:34.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.877+0000 7fe143f9c740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55bae2180400 con 0x55bae223fd10 2024-04-02T14:42:34.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.877+0000 7fe143f9c740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55bae2164f10 con 0x55bae2249080 2024-04-02T14:42:34.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.877+0000 7fe143f9c740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55bae21a46f0 con 0x55bae22409f0 2024-04-02T14:42:34.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.878+0000 7fe13a800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55bae2249080 0x55bae224b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:34.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.878+0000 7fe13a800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55bae2249080 0x55bae224b470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:55132/0 (socket says 172.21.15.90:55132) 2024-04-02T14:42:34.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.878+0000 7fe13a800640 1 -- 172.21.15.90:0/2473539123 learned_addr learned my addr 172.21.15.90:0/2473539123 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:42:34.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.878+0000 7fe139400640 1 --2- 172.21.15.90:0/2058479297 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fe1200a3e20 0x7fe1200a62e0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fe13c102f20 tx=0x7fe128005f90 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:42:34.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.878+0000 7fe13a800640 1 -- 172.21.15.90:0/2473539123 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bae22409f0 msgr2=0x55bae223f740 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-02T14:42:34.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.878+0000 7fe13a800640 1 --2- 172.21.15.90:0/2473539123 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bae22409f0 0x55bae223f740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:34.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.879+0000 7fe13a800640 1 -- 172.21.15.90:0/2473539123 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bae223fd10 msgr2=0x55bae2240190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:42:34.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.879+0000 7fe13a800640 1 --2- 172.21.15.90:0/2473539123 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bae223fd10 0x55bae2240190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:34.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.879+0000 7fe13a800640 1 -- 172.21.15.90:0/2473539123 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bae1eb83f0 con 0x55bae2249080 2024-04-02T14:42:34.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.879+0000 7fe13a800640 1 --2- 172.21.15.90:0/2473539123 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55bae2249080 0x55bae224b470 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7fe1340c75b0 tx=0x7fe1340be7d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e2469bb38a1996a0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:42:34.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.883+0000 7fe127400640 1 -- 172.21.15.90:0/2473539123 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1340bee70 con 0x55bae2249080 2024-04-02T14:42:34.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.883+0000 7fe127400640 1 -- 172.21.15.90:0/2473539123 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe1340c3220 con 0x55bae2249080 2024-04-02T14:42:34.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.884+0000 7fe143f9c740 1 -- 172.21.15.90:0/2473539123 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55bae2249080 msgr2=0x55bae224b470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:34.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.884+0000 7fe143f9c740 1 --2- 172.21.15.90:0/2473539123 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55bae2249080 0x55bae224b470 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7fe1340c75b0 tx=0x7fe1340be7d0 comp rx=0 tx=0).stop 2024-04-02T14:42:34.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.884+0000 7fe143f9c740 1 -- 172.21.15.90:0/2473539123 shutdown_connections 2024-04-02T14:42:34.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.884+0000 7fe143f9c740 1 --2- 172.21.15.90:0/2473539123 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55bae2249080 0x55bae224b470 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:34.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.885+0000 7fe143f9c740 1 --2- 172.21.15.90:0/2473539123 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bae223fd10 0x55bae2240190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:34.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.885+0000 7fe143f9c740 1 --2- 172.21.15.90:0/2473539123 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bae22409f0 0x55bae223f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:34.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.885+0000 7fe143f9c740 1 -- 172.21.15.90:0/2473539123 >> 172.21.15.90:0/2473539123 conn(0x55bae2206a50 msgr2=0x55bae223a760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:42:34.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.888+0000 7fe143f9c740 1 -- 172.21.15.90:0/2473539123 shutdown_connections 2024-04-02T14:42:34.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.888+0000 7fe143f9c740 1 -- 172.21.15.90:0/2473539123 wait complete. 2024-04-02T14:42:34.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.889+0000 7fe143f9c740 1 -- start start 2024-04-02T14:42:34.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.889+0000 7fe143f9c740 1 Processor -- start 2024-04-02T14:42:34.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.893+0000 7fe143f9c740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bae223fd10 0x55bae2245f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:34.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.893+0000 7fe143f9c740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bae22409f0 0x55bae22464b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:34.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.893+0000 7fe143f9c740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55bae2249080 0x55bae22469f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:34.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.893+0000 7fe143f9c740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55bae21a06a0 con 0x55bae223fd10 2024-04-02T14:42:34.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.893+0000 7fe143f9c740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55bae214c980 con 0x55bae2249080 2024-04-02T14:42:34.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.893+0000 7fe143f9c740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55bae215fcb0 con 0x55bae22409f0 2024-04-02T14:42:34.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.894+0000 7fe143f9c740 10 client.0 _ll_register_callbacks cb 0x55bae215f6a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:42:34.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.894+0000 7fe13a800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55bae2249080 0x55bae22469f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:34.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.894+0000 7fe13a800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55bae2249080 0x55bae22469f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:55138/0 (socket says 172.21.15.90:55138) 2024-04-02T14:42:34.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.894+0000 7fe13a800640 1 -- 172.21.15.90:0/3093196157 learned_addr learned my addr 172.21.15.90:0/3093196157 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:42:34.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.894+0000 7fe139400640 1 --2- 172.21.15.90:0/3093196157 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bae22409f0 0x55bae22464b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:34.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.895+0000 7fe13a800640 1 -- 172.21.15.90:0/3093196157 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bae22409f0 msgr2=0x55bae22464b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:42:34.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.895+0000 7fe13a800640 1 --2- 172.21.15.90:0/3093196157 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bae22409f0 0x55bae22464b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:34.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.895+0000 7fe13a800640 1 -- 172.21.15.90:0/3093196157 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bae223fd10 msgr2=0x55bae2245f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:42:34.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.895+0000 7fe13a800640 1 --2- 172.21.15.90:0/3093196157 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bae223fd10 0x55bae2245f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:34.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.895+0000 7fe13a800640 1 -- 172.21.15.90:0/3093196157 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bae1f827d0 con 0x55bae2249080 2024-04-02T14:42:34.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.896+0000 7fe13a800640 1 --2- 172.21.15.90:0/3093196157 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55bae2249080 0x55bae22469f0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7fe1340bb840 tx=0x7fe134009520 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3ba6c2a813d13eb2 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:42:34.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.896+0000 7fe11be00640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:42:34.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.897+0000 7fe11be00640 1 -- 172.21.15.90:0/3093196157 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1340dc070 con 0x55bae2249080 2024-04-02T14:42:34.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.897+0000 7fe143f9c740 1 -- 172.21.15.90:0/3093196157 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe1340c3220 con 0x55bae2249080 2024-04-02T14:42:34.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.897+0000 7fe11be00640 1 -- 172.21.15.90:0/3093196157 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8550+0+0 (secure 0 0 0) 0x7fe134003b80 con 0x55bae2249080 2024-04-02T14:42:34.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.898+0000 7fe11be00640 1 -- 172.21.15.90:0/3093196157 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fe1340f09a0 con 0x55bae2249080 2024-04-02T14:42:34.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.898+0000 7fe143f9c740 10 client.27785 fetch_fsmap learned FSMap version 27 2024-04-02T14:42:34.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.898+0000 7fe143f9c740 1 -- 172.21.15.90:0/3093196157 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55bae2257ba0 con 0x55bae2249080 2024-04-02T14:42:34.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.899+0000 7fe11be00640 1 -- 172.21.15.90:0/3093196157 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe1340c3220 con 0x55bae2249080 2024-04-02T14:42:34.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.899+0000 7fe11be00640 1 -- 172.21.15.90:0/3093196157 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fe1340be2c0 con 0x55bae2249080 2024-04-02T14:42:34.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.899+0000 7fe143f9c740 10 client.27785 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:42:34.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.899+0000 7fe143f9c740 10 client.27785 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:42:34.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.899+0000 7fe143f9c740 20 client.27785 populate_metadata read hostname 'smithi090' 2024-04-02T14:42:34.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.899+0000 7fe143f9c740 1 --2- 172.21.15.90:0/3093196157 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55bae2258d20 0x55bae225b110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:34.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.899+0000 7fe143f9c740 4 client.27785 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:42:34.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.899+0000 7fe143f9c740 1 -- 172.21.15.90:0/3093196157 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bae2164040 con 0x55bae2258d20 2024-04-02T14:42:34.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.900+0000 7fe139e00640 1 --2- 172.21.15.90:0/3093196157 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55bae2258d20 0x55bae225b110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:34.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.902+0000 7fe139e00640 1 --2- 172.21.15.90:0/3093196157 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55bae2258d20 0x55bae225b110 crc :-1 s=READY pgs=208 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b832eef74f04842d server_cookie=c3666509e1506a2a in_seq=0 out_seq=0 2024-04-02T14:42:34.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.902+0000 7fe11be00640 10 client.27785 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:42:34.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.903+0000 7fe11be00640 1 -- 172.21.15.90:0/3093196157 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55bae2164040 con 0x55bae2258d20 2024-04-02T14:42:34.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.903+0000 7fe11be00640 10 client.27785 handle_command_reply: tid=0 2024-04-02T14:42:34.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.903+0000 7fe143f9c740 1 client.27785 shutdown 2024-04-02T14:42:34.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.903+0000 7fe143f9c740 1 -- 172.21.15.90:0/3093196157 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55bae2249080 msgr2=0x55bae22469f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:34.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.903+0000 7fe143f9c740 1 --2- 172.21.15.90:0/3093196157 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55bae2249080 0x55bae22469f0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7fe1340bb840 tx=0x7fe134009520 comp rx=0 tx=0).stop 2024-04-02T14:42:34.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.904+0000 7fe143f9c740 1 -- 172.21.15.90:0/3093196157 shutdown_connections 2024-04-02T14:42:34.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.904+0000 7fe143f9c740 1 --2- 172.21.15.90:0/3093196157 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55bae2258d20 0x55bae225b110 crc :-1 s=READY pgs=208 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:34.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.904+0000 7fe143f9c740 1 --2- 172.21.15.90:0/3093196157 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55bae2249080 0x55bae22469f0 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:34.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.904+0000 7fe143f9c740 1 --2- 172.21.15.90:0/3093196157 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bae22409f0 0x55bae22464b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:34.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.904+0000 7fe143f9c740 1 --2- 172.21.15.90:0/3093196157 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bae223fd10 0x55bae2245f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:34.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.904+0000 7fe143f9c740 1 -- 172.21.15.90:0/3093196157 >> 172.21.15.90:0/3093196157 conn(0x55bae2206a50 msgr2=0x55bae223c4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:42:34.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.904+0000 7fe11be00640 0 client.27785 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:42:34.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.904+0000 7fe143f9c740 1 -- 172.21.15.90:0/3093196157 shutdown_connections 2024-04-02T14:42:34.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.906+0000 7fe143f9c740 1 -- 172.21.15.90:0/3093196157 wait complete. 2024-04-02T14:42:34.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.906+0000 7fe143f9c740 20 client.27785 trim_cache size 0 max 16384 2024-04-02T14:42:34.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.936+0000 7fe143f9c740 1 Processor -- start 2024-04-02T14:42:34.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.937+0000 7fe143f9c740 1 -- start start 2024-04-02T14:42:34.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.937+0000 7fe143f9c740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55bae223fd10 0x55bae22486a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:34.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.937+0000 7fe143f9c740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bae22409f0 0x55bae22548c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:34.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.937+0000 7fe143f9c740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bae2249080 0x55bae2246640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:34.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.937+0000 7fe143f9c740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fe13c1ab5a0 con 0x55bae22409f0 2024-04-02T14:42:34.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.937+0000 7fe143f9c740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fe13c1ad0e0 con 0x55bae223fd10 2024-04-02T14:42:34.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.937+0000 7fe143f9c740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fe13c1abc70 con 0x55bae2249080 2024-04-02T14:42:34.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.938+0000 7fe139400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bae22409f0 0x55bae22548c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:34.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.938+0000 7fe139e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55bae223fd10 0x55bae22486a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:34.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.938+0000 7fe139e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55bae223fd10 0x55bae22486a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:55150/0 (socket says 172.21.15.90:55150) 2024-04-02T14:42:34.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.938+0000 7fe13a800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bae2249080 0x55bae2246640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:34.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.938+0000 7fe139e00640 1 -- 172.21.15.90:0/1145008922 learned_addr learned my addr 172.21.15.90:0/1145008922 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:42:34.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.938+0000 7fe13a800640 1 -- 172.21.15.90:0/1145008922 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55bae223fd10 msgr2=0x55bae22486a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:42:34.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.938+0000 7fe13a800640 1 --2- 172.21.15.90:0/1145008922 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55bae223fd10 0x55bae22486a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:34.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.938+0000 7fe13a800640 1 -- 172.21.15.90:0/1145008922 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bae22409f0 msgr2=0x55bae22548c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:42:34.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.938+0000 7fe13a800640 1 --2- 172.21.15.90:0/1145008922 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bae22409f0 0x55bae22548c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:34.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.938+0000 7fe13a800640 1 -- 172.21.15.90:0/1145008922 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bae223ed40 con 0x55bae2249080 2024-04-02T14:42:34.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.938+0000 7fe139400640 1 --2- 172.21.15.90:0/1145008922 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bae22409f0 0x55bae22548c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:42:34.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.939+0000 7fe13a800640 1 --2- 172.21.15.90:0/1145008922 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bae2249080 0x55bae2246640 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fe1340bb2d0 tx=0x7fe1340be7d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c31517c12f43f962 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:42:34.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.939+0000 7fe126a00640 1 -- 172.21.15.90:0/1145008922 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1340dc070 con 0x55bae2249080 2024-04-02T14:42:34.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.940+0000 7fe126a00640 1 -- 172.21.15.90:0/1145008922 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe1340c3220 con 0x55bae2249080 2024-04-02T14:42:34.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.940+0000 7fe126a00640 1 -- 172.21.15.90:0/1145008922 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1340be0d0 con 0x55bae2249080 2024-04-02T14:42:34.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.941+0000 7fe143f9c740 1 -- 172.21.15.90:0/1145008922 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bae2249080 msgr2=0x55bae2246640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:34.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.941+0000 7fe143f9c740 1 --2- 172.21.15.90:0/1145008922 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bae2249080 0x55bae2246640 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fe1340bb2d0 tx=0x7fe1340be7d0 comp rx=0 tx=0).stop 2024-04-02T14:42:34.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.941+0000 7fe143f9c740 1 -- 172.21.15.90:0/1145008922 shutdown_connections 2024-04-02T14:42:34.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.941+0000 7fe143f9c740 1 --2- 172.21.15.90:0/1145008922 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bae2249080 0x55bae2246640 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:34.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.941+0000 7fe143f9c740 1 --2- 172.21.15.90:0/1145008922 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bae22409f0 0x55bae22548c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:34.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.941+0000 7fe143f9c740 1 --2- 172.21.15.90:0/1145008922 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55bae223fd10 0x55bae22486a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:34.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.941+0000 7fe143f9c740 1 -- 172.21.15.90:0/1145008922 >> 172.21.15.90:0/1145008922 conn(0x55bae2206a50 msgr2=0x55bae2245f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:42:34.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.942+0000 7fe143f9c740 1 -- 172.21.15.90:0/1145008922 shutdown_connections 2024-04-02T14:42:34.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.942+0000 7fe143f9c740 1 -- 172.21.15.90:0/1145008922 wait complete. 2024-04-02T14:42:34.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.943+0000 7fe143f9c740 1 -- start start 2024-04-02T14:42:34.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.944+0000 7fe143f9c740 1 Processor -- start 2024-04-02T14:42:34.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.945+0000 7fe143f9c740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55bae223fd10 0x55bae2250fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:34.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.945+0000 7fe143f9c740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bae22409f0 0x55bae22431f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:34.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.945+0000 7fe143f9c740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bae2249080 0x55bae2243730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:34.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.945+0000 7fe143f9c740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fe13c1aca10 con 0x55bae22409f0 2024-04-02T14:42:34.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.945+0000 7fe143f9c740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fe13c1ac340 con 0x55bae223fd10 2024-04-02T14:42:34.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.945+0000 7fe143f9c740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fe13c1a95a0 con 0x55bae2249080 2024-04-02T14:42:34.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.945+0000 7fe143f9c740 10 client.0 _ll_register_callbacks cb 0x7fe1340c7ae0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:42:34.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.945+0000 7fe13a800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bae2249080 0x55bae2243730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:34.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.945+0000 7fe13a800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bae2249080 0x55bae2243730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:47478/0 (socket says 172.21.15.90:47478) 2024-04-02T14:42:34.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.945+0000 7fe13a800640 1 -- 172.21.15.90:0/1262557697 learned_addr learned my addr 172.21.15.90:0/1262557697 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:42:34.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.946+0000 7fe13a800640 1 -- 172.21.15.90:0/1262557697 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55bae223fd10 msgr2=0x55bae2250fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:42:34.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.946+0000 7fe13a800640 1 --2- 172.21.15.90:0/1262557697 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55bae223fd10 0x55bae2250fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:34.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.946+0000 7fe13a800640 1 -- 172.21.15.90:0/1262557697 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bae22409f0 msgr2=0x55bae22431f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-02T14:42:34.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.946+0000 7fe13a800640 1 --2- 172.21.15.90:0/1262557697 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bae22409f0 0x55bae22431f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:34.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.946+0000 7fe13a800640 1 -- 172.21.15.90:0/1262557697 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bae2243c70 con 0x55bae2249080 2024-04-02T14:42:34.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.947+0000 7fe13a800640 1 --2- 172.21.15.90:0/1262557697 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bae2249080 0x55bae2243730 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fe134004af0 tx=0x7fe1340be0a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c0143e4b61bdb831 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:42:34.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.947+0000 7fe11be00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:42:34.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.947+0000 7fe11be00640 1 -- 172.21.15.90:0/1262557697 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1340dc070 con 0x55bae2249080 2024-04-02T14:42:34.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.947+0000 7fe143f9c740 1 -- 172.21.15.90:0/1262557697 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe1340c3220 con 0x55bae2249080 2024-04-02T14:42:34.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.947+0000 7fe11be00640 1 -- 172.21.15.90:0/1262557697 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8550+0+0 (secure 0 0 0) 0x7fe1340a8100 con 0x55bae2249080 2024-04-02T14:42:34.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.948+0000 7fe11be00640 1 -- 172.21.15.90:0/1262557697 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fe134003d90 con 0x55bae2249080 2024-04-02T14:42:34.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.948+0000 7fe143f9c740 10 client.27640 fetch_fsmap learned FSMap version 27 2024-04-02T14:42:34.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.948+0000 7fe143f9c740 1 -- 172.21.15.90:0/1262557697 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55bae2243c70 con 0x55bae2249080 2024-04-02T14:42:34.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.948+0000 7fe11be00640 1 -- 172.21.15.90:0/1262557697 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe1340c3220 con 0x55bae2249080 2024-04-02T14:42:34.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.949+0000 7fe11be00640 1 -- 172.21.15.90:0/1262557697 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fe1340c43b0 con 0x55bae2249080 2024-04-02T14:42:34.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.949+0000 7fe143f9c740 10 client.27640 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:42:34.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.949+0000 7fe143f9c740 10 client.27640 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:42:34.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.949+0000 7fe143f9c740 20 client.27640 populate_metadata read hostname 'smithi090' 2024-04-02T14:42:34.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.949+0000 7fe143f9c740 1 --2- 172.21.15.90:0/1262557697 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55bae22bf200 0x55bae22c15f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:34.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.950+0000 7fe143f9c740 4 client.27640 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-02T14:42:34.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.950+0000 7fe143f9c740 1 -- 172.21.15.90:0/1262557697 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55bae21b1c30 con 0x55bae22bf200 2024-04-02T14:42:34.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.950+0000 7fe139e00640 1 --2- 172.21.15.90:0/1262557697 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55bae22bf200 0x55bae22c15f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:34.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.951+0000 7fe139e00640 1 --2- 172.21.15.90:0/1262557697 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55bae22bf200 0x55bae22c15f0 crc :-1 s=READY pgs=209 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8b86e0f85a1e8bff server_cookie=aeb41752a800b5bd in_seq=0 out_seq=0 2024-04-02T14:42:34.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.951+0000 7fe11be00640 10 client.27640 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:42:34.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.951+0000 7fe11be00640 1 -- 172.21.15.90:0/1262557697 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55bae21b1c30 con 0x55bae22bf200 2024-04-02T14:42:34.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.951+0000 7fe11be00640 10 client.27640 handle_command_reply: tid=0 2024-04-02T14:42:34.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.952+0000 7fe143f9c740 1 client.27640 shutdown 2024-04-02T14:42:34.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.952+0000 7fe143f9c740 1 -- 172.21.15.90:0/1262557697 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bae2249080 msgr2=0x55bae2243730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:34.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.952+0000 7fe143f9c740 1 --2- 172.21.15.90:0/1262557697 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bae2249080 0x55bae2243730 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fe134004af0 tx=0x7fe1340be0a0 comp rx=0 tx=0).stop 2024-04-02T14:42:34.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.953+0000 7fe143f9c740 1 -- 172.21.15.90:0/1262557697 shutdown_connections 2024-04-02T14:42:34.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.953+0000 7fe143f9c740 1 --2- 172.21.15.90:0/1262557697 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55bae22bf200 0x55bae22c15f0 crc :-1 s=READY pgs=209 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:34.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.953+0000 7fe143f9c740 1 --2- 172.21.15.90:0/1262557697 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bae2249080 0x55bae2243730 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:34.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.953+0000 7fe143f9c740 1 --2- 172.21.15.90:0/1262557697 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bae22409f0 0x55bae22431f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:34.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.953+0000 7fe11be00640 0 client.27640 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:42:34.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.953+0000 7fe143f9c740 1 --2- 172.21.15.90:0/1262557697 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55bae223fd10 0x55bae2250fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:34.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.953+0000 7fe143f9c740 1 -- 172.21.15.90:0/1262557697 >> 172.21.15.90:0/1262557697 conn(0x55bae2206a50 msgr2=0x55bae22398a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:42:34.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.953+0000 7fe143f9c740 1 -- 172.21.15.90:0/1262557697 shutdown_connections 2024-04-02T14:42:34.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.953+0000 7fe143f9c740 1 -- 172.21.15.90:0/1262557697 wait complete. 2024-04-02T14:42:34.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.953+0000 7fe143f9c740 20 client.27640 trim_cache size 0 max 16384 2024-04-02T14:42:34.953 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-02T14:42:34.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.956+0000 7fe141200640 1 -- 172.21.15.90:0/2058479297 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fe1200a3e20 msgr2=0x7fe1200a62e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:34.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.956+0000 7fe141200640 1 --2- 172.21.15.90:0/2058479297 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fe1200a3e20 0x7fe1200a62e0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fe13c102f20 tx=0x7fe128005f90 comp rx=0 tx=0).stop 2024-04-02T14:42:34.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.956+0000 7fe141200640 1 -- 172.21.15.90:0/2058479297 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe13c19ee30 msgr2=0x7fe13c19f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:34.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.956+0000 7fe141200640 1 --2- 172.21.15.90:0/2058479297 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe13c19ee30 0x7fe13c19f2e0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7fe1340096f0 tx=0x7fe1340336b0 comp rx=0 tx=0).stop 2024-04-02T14:42:34.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.957+0000 7fe141200640 1 -- 172.21.15.90:0/2058479297 shutdown_connections 2024-04-02T14:42:34.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.957+0000 7fe141200640 1 --2- 172.21.15.90:0/2058479297 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fe1200a3e20 0x7fe1200a62e0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:34.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.957+0000 7fe141200640 1 --2- 172.21.15.90:0/2058479297 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe13c19ee30 0x7fe13c19f2e0 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:34.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.957+0000 7fe141200640 1 --2- 172.21.15.90:0/2058479297 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe13c108fe0 0x7fe13c19aab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:34.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.957+0000 7fe141200640 1 --2- 172.21.15.90:0/2058479297 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe13c103f10 0x7fe13c19a570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:34.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.957+0000 7fe141200640 1 -- 172.21.15.90:0/2058479297 >> 172.21.15.90:0/2058479297 conn(0x7fe13c0ff770 msgr2=0x7fe13c100350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:42:34.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.958+0000 7fe141200640 1 -- 172.21.15.90:0/2058479297 shutdown_connections 2024-04-02T14:42:34.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:34.958+0000 7fe141200640 1 -- 172.21.15.90:0/2058479297 wait complete. 2024-04-02T14:42:35.981 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-02T14:42:35.982 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T14:42:35.982 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-02T14:42:36.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.246+0000 7fa809400640 1 -- 172.21.15.90:0/133687571 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa804107a80 msgr2=0x7fa804107e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:36.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.246+0000 7fa809400640 1 --2- 172.21.15.90:0/133687571 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa804107a80 0x7fa804107e60 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7fa7f4009900 tx=0x7fa7f402bc40 comp rx=0 tx=0).stop 2024-04-02T14:42:36.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.246+0000 7fa809400640 1 -- 172.21.15.90:0/133687571 shutdown_connections 2024-04-02T14:42:36.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.246+0000 7fa809400640 1 --2- 172.21.15.90:0/133687571 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa8040696a0 0x7fa80410e940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:36.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.246+0000 7fa809400640 1 --2- 172.21.15.90:0/133687571 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa804068ce0 0x7fa804069160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:36.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.246+0000 7fa809400640 1 --2- 172.21.15.90:0/133687571 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa804107a80 0x7fa804107e60 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:36.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.246+0000 7fa809400640 1 -- 172.21.15.90:0/133687571 >> 172.21.15.90:0/133687571 conn(0x7fa804077b00 msgr2=0x7fa804077f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:42:36.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.246+0000 7fa809400640 1 -- 172.21.15.90:0/133687571 shutdown_connections 2024-04-02T14:42:36.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.246+0000 7fa809400640 1 -- 172.21.15.90:0/133687571 wait complete. 2024-04-02T14:42:36.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.249+0000 7fa809400640 1 Processor -- start 2024-04-02T14:42:36.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.249+0000 7fa809400640 1 -- start start 2024-04-02T14:42:36.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.250+0000 7fa809400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa804068ce0 0x7fa8041a35c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:36.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.250+0000 7fa809400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa8040696a0 0x7fa8041a3b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:36.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.250+0000 7fa809400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa804107a80 0x7fa80410c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:36.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.250+0000 7fa809400640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fa804111e60 con 0x7fa804068ce0 2024-04-02T14:42:36.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.250+0000 7fa809400640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fa804111ce0 con 0x7fa8040696a0 2024-04-02T14:42:36.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.250+0000 7fa809400640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fa804111fe0 con 0x7fa804107a80 2024-04-02T14:42:36.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.250+0000 7fa803200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa804107a80 0x7fa80410c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:36.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.250+0000 7fa803200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa804107a80 0x7fa80410c9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:47494/0 (socket says 172.21.15.90:47494) 2024-04-02T14:42:36.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.250+0000 7fa803200640 1 -- 172.21.15.90:0/4191972145 learned_addr learned my addr 172.21.15.90:0/4191972145 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:42:36.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.250+0000 7fa801e00640 1 --2- 172.21.15.90:0/4191972145 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa8040696a0 0x7fa8041a3b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:36.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.251+0000 7fa802800640 1 --2- 172.21.15.90:0/4191972145 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa804068ce0 0x7fa8041a35c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:36.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.251+0000 7fa803200640 1 -- 172.21.15.90:0/4191972145 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa8040696a0 msgr2=0x7fa8041a3b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:36.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.251+0000 7fa803200640 1 --2- 172.21.15.90:0/4191972145 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa8040696a0 0x7fa8041a3b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:36.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.251+0000 7fa803200640 1 -- 172.21.15.90:0/4191972145 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa804068ce0 msgr2=0x7fa8041a35c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:36.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.251+0000 7fa803200640 1 --2- 172.21.15.90:0/4191972145 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa804068ce0 0x7fa8041a35c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:36.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.251+0000 7fa803200640 1 -- 172.21.15.90:0/4191972145 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa80410d120 con 0x7fa804107a80 2024-04-02T14:42:36.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.251+0000 7fa803200640 1 --2- 172.21.15.90:0/4191972145 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa804107a80 0x7fa80410c9d0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fa7fc00eaa0 tx=0x7fa7fc00ef60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:42:36.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.252+0000 7fa7fb400640 1 -- 172.21.15.90:0/4191972145 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7fc00cdb0 con 0x7fa804107a80 2024-04-02T14:42:36.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.252+0000 7fa7fb400640 1 -- 172.21.15.90:0/4191972145 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa7fc005020 con 0x7fa804107a80 2024-04-02T14:42:36.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.252+0000 7fa7fb400640 1 -- 172.21.15.90:0/4191972145 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7fc0116b0 con 0x7fa804107a80 2024-04-02T14:42:36.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.252+0000 7fa809400640 1 -- 172.21.15.90:0/4191972145 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa80410d410 con 0x7fa804107a80 2024-04-02T14:42:36.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.252+0000 7fa809400640 1 -- 172.21.15.90:0/4191972145 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8041ab4e0 con 0x7fa804107a80 2024-04-02T14:42:36.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.256+0000 7fa7fb400640 1 -- 172.21.15.90:0/4191972145 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fa7fc004b20 con 0x7fa804107a80 2024-04-02T14:42:36.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.257+0000 7fa80c1ce740 1 Processor -- start 2024-04-02T14:42:36.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.257+0000 7fa80c1ce740 1 -- start start 2024-04-02T14:42:36.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.257+0000 7fa7fb400640 1 --2- 172.21.15.90:0/4191972145 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fa7e80a3da0 0x7fa7e80a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:36.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.257+0000 7fa80c1ce740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5563190b3030 0x5563190b1d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:36.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.257+0000 7fa80c1ce740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5563190b2350 0x5563190b27d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:36.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.257+0000 7fa80c1ce740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5563190bb6c0 0x5563190bdab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:36.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.257+0000 7fa7fb400640 1 -- 172.21.15.90:0/4191972145 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8550+0+0 (secure 0 0 0) 0x7fa7fc0d0aa0 con 0x7fa804107a80 2024-04-02T14:42:36.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.257+0000 7fa80c1ce740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x556319016310 con 0x5563190b2350 2024-04-02T14:42:36.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.257+0000 7fa80c1ce740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x556318fbe690 con 0x5563190b3030 2024-04-02T14:42:36.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.257+0000 7fa80c1ce740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5563190122c0 con 0x5563190bb6c0 2024-04-02T14:42:36.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.257+0000 7fa801e00640 1 --2- 172.21.15.90:0/4191972145 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fa7e80a3da0 0x7fa7e80a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:36.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.257+0000 7fa802800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5563190b3030 0x5563190b1d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:36.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.258+0000 7fa802800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5563190b3030 0x5563190b1d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:55174/0 (socket says 172.21.15.90:55174) 2024-04-02T14:42:36.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.258+0000 7fa802800640 1 -- 172.21.15.90:0/3915454073 learned_addr learned my addr 172.21.15.90:0/3915454073 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:42:36.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.258+0000 7fa803200640 1 --2- 172.21.15.90:0/3915454073 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5563190bb6c0 0x5563190bdab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:36.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.258+0000 7fa801e00640 1 --2- 172.21.15.90:0/3915454073 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5563190b2350 0x5563190b27d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:36.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.258+0000 7fa803200640 1 -- 172.21.15.90:0/3915454073 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5563190b3030 msgr2=0x5563190b1d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:42:36.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.258+0000 7fa803200640 1 --2- 172.21.15.90:0/3915454073 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5563190b3030 0x5563190b1d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:36.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.258+0000 7fa803200640 1 -- 172.21.15.90:0/3915454073 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5563190b2350 msgr2=0x5563190b27d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:42:36.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.258+0000 7fa803200640 1 --2- 172.21.15.90:0/3915454073 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5563190b2350 0x5563190b27d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:36.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.259+0000 7fa803200640 1 -- 172.21.15.90:0/3915454073 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556318df27e0 con 0x5563190bb6c0 2024-04-02T14:42:36.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.262+0000 7fa801e00640 1 --2- 172.21.15.90:0/4191972145 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fa7e80a3da0 0x7fa7e80a6260 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x5563190b3580 tx=0x7fa7f000b570 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:42:36.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.262+0000 7fa803200640 1 --2- 172.21.15.90:0/3915454073 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5563190bb6c0 0x5563190bdab0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fa7fc0ce170 tx=0x7fa7fc09cb00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=654b1f7b38d4bab3 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:42:36.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.262+0000 7fa7ef400640 1 -- 172.21.15.90:0/3915454073 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7fc01f040 con 0x5563190bb6c0 2024-04-02T14:42:36.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.263+0000 7fa7ef400640 1 -- 172.21.15.90:0/3915454073 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa7fc09acc0 con 0x5563190bb6c0 2024-04-02T14:42:36.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.263+0000 7fa7ef400640 1 -- 172.21.15.90:0/3915454073 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7fc0a5400 con 0x5563190bb6c0 2024-04-02T14:42:36.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.263+0000 7fa80c1ce740 1 -- 172.21.15.90:0/3915454073 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5563190bb6c0 msgr2=0x5563190bdab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:36.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.263+0000 7fa80c1ce740 1 --2- 172.21.15.90:0/3915454073 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5563190bb6c0 0x5563190bdab0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fa7fc0ce170 tx=0x7fa7fc09cb00 comp rx=0 tx=0).stop 2024-04-02T14:42:36.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.264+0000 7fa80c1ce740 1 -- 172.21.15.90:0/3915454073 shutdown_connections 2024-04-02T14:42:36.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.264+0000 7fa80c1ce740 1 --2- 172.21.15.90:0/3915454073 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5563190bb6c0 0x5563190bdab0 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:36.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.264+0000 7fa80c1ce740 1 --2- 172.21.15.90:0/3915454073 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5563190b2350 0x5563190b27d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:36.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.264+0000 7fa80c1ce740 1 --2- 172.21.15.90:0/3915454073 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5563190b3030 0x5563190b1d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:36.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.264+0000 7fa80c1ce740 1 -- 172.21.15.90:0/3915454073 >> 172.21.15.90:0/3915454073 conn(0x556319046270 msgr2=0x5563190acda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:42:36.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.264+0000 7fa80c1ce740 1 -- 172.21.15.90:0/3915454073 shutdown_connections 2024-04-02T14:42:36.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.264+0000 7fa80c1ce740 1 -- 172.21.15.90:0/3915454073 wait complete. 2024-04-02T14:42:36.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.265+0000 7fa80c1ce740 1 -- start start 2024-04-02T14:42:36.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.265+0000 7fa80c1ce740 1 Processor -- start 2024-04-02T14:42:36.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.265+0000 7fa80c1ce740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5563190b2350 0x5563190b86f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:36.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.266+0000 7fa80c1ce740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5563190b3030 0x5563190b8c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:36.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.266+0000 7fa80c1ce740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5563190bb6c0 0x5563190b9170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:36.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.266+0000 7fa80c1ce740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5563190be530 con 0x5563190bb6c0 2024-04-02T14:42:36.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.266+0000 7fa80c1ce740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x556318fd1a60 con 0x5563190b3030 2024-04-02T14:42:36.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.266+0000 7fa80c1ce740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5563190be6b0 con 0x5563190b2350 2024-04-02T14:42:36.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.266+0000 7fa80c1ce740 10 client.0 _ll_register_callbacks cb 0x556318fd7030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:42:36.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.266+0000 7fa801e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5563190b3030 0x5563190b8c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:36.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.266+0000 7fa802800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5563190b2350 0x5563190b86f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:36.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.266+0000 7fa801e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5563190b3030 0x5563190b8c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:55178/0 (socket says 172.21.15.90:55178) 2024-04-02T14:42:36.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.266+0000 7fa802800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5563190b2350 0x5563190b86f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:47504/0 (socket says 172.21.15.90:47504) 2024-04-02T14:42:36.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.266+0000 7fa803200640 1 --2- 172.21.15.90:0/2222517417 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5563190bb6c0 0x5563190b9170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:36.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.266+0000 7fa801e00640 1 -- 172.21.15.90:0/2222517417 learned_addr learned my addr 172.21.15.90:0/2222517417 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:42:36.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.266+0000 7fa803200640 1 -- 172.21.15.90:0/2222517417 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5563190b2350 msgr2=0x5563190b86f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:42:36.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.266+0000 7fa803200640 1 --2- 172.21.15.90:0/2222517417 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5563190b2350 0x5563190b86f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:36.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.266+0000 7fa803200640 1 -- 172.21.15.90:0/2222517417 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5563190b3030 msgr2=0x5563190b8c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:42:36.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.266+0000 7fa803200640 1 --2- 172.21.15.90:0/2222517417 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5563190b3030 0x5563190b8c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:36.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.266+0000 7fa803200640 1 -- 172.21.15.90:0/2222517417 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556318e52f70 con 0x5563190bb6c0 2024-04-02T14:42:36.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.267+0000 7fa801e00640 1 --2- 172.21.15.90:0/2222517417 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5563190b3030 0x5563190b8c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:42:36.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.267+0000 7fa803200640 1 --2- 172.21.15.90:0/2222517417 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5563190bb6c0 0x5563190b9170 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fa7fc0ce170 tx=0x7fa7fc009270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85aa11aa2a356dca server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:42:36.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.267+0000 7fa7e3e00640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-02T14:42:36.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.267+0000 7fa7e3e00640 1 -- 172.21.15.90:0/2222517417 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7fc01f070 con 0x5563190bb6c0 2024-04-02T14:42:36.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.267+0000 7fa7e3e00640 1 -- 172.21.15.90:0/2222517417 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8550+0+0 (secure 0 0 0) 0x7fa7fc0bbb80 con 0x5563190bb6c0 2024-04-02T14:42:36.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.267+0000 7fa80c1ce740 1 -- 172.21.15.90:0/2222517417 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa7fc09acc0 con 0x5563190bb6c0 2024-04-02T14:42:36.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.267+0000 7fa7e3e00640 1 -- 172.21.15.90:0/2222517417 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fa7fc099180 con 0x5563190bb6c0 2024-04-02T14:42:36.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.267+0000 7fa80c1ce740 10 client.17037 fetch_fsmap learned FSMap version 27 2024-04-02T14:42:36.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.267+0000 7fa80c1ce740 1 -- 172.21.15.90:0/2222517417 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x556318cea170 con 0x5563190bb6c0 2024-04-02T14:42:36.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.268+0000 7fa7e3e00640 1 -- 172.21.15.90:0/2222517417 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa7fc09acc0 con 0x5563190bb6c0 2024-04-02T14:42:36.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.268+0000 7fa7e3e00640 1 -- 172.21.15.90:0/2222517417 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fa7fc00f040 con 0x5563190bb6c0 2024-04-02T14:42:36.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.268+0000 7fa80c1ce740 10 client.17037 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:42:36.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.268+0000 7fa80c1ce740 10 client.17037 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:42:36.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.268+0000 7fa80c1ce740 20 client.17037 populate_metadata read hostname 'smithi090' 2024-04-02T14:42:36.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.268+0000 7fa80c1ce740 1 --2- 172.21.15.90:0/2222517417 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5563190cba20 0x5563190cde10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:36.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.268+0000 7fa80c1ce740 4 client.17037 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:42:36.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.268+0000 7fa80c1ce740 1 -- 172.21.15.90:0/2222517417 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556319023850 con 0x5563190cba20 2024-04-02T14:42:36.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.268+0000 7fa802800640 1 --2- 172.21.15.90:0/2222517417 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5563190cba20 0x5563190cde10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:36.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.270+0000 7fa7e3e00640 10 client.17037 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:42:36.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.270+0000 7fa802800640 1 --2- 172.21.15.90:0/2222517417 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5563190cba20 0x5563190cde10 crc :-1 s=READY pgs=210 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f34977edeb9f407c server_cookie=392d293623258b34 in_seq=0 out_seq=0 2024-04-02T14:42:36.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.271+0000 7fa7e3e00640 1 -- 172.21.15.90:0/2222517417 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x556319023850 con 0x5563190cba20 2024-04-02T14:42:36.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.271+0000 7fa7e3e00640 10 client.17037 handle_command_reply: tid=0 2024-04-02T14:42:36.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.271+0000 7fa80c1ce740 1 client.17037 shutdown 2024-04-02T14:42:36.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.272+0000 7fa80c1ce740 1 -- 172.21.15.90:0/2222517417 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5563190bb6c0 msgr2=0x5563190b9170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:36.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.272+0000 7fa80c1ce740 1 --2- 172.21.15.90:0/2222517417 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5563190bb6c0 0x5563190b9170 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fa7fc0ce170 tx=0x7fa7fc009270 comp rx=0 tx=0).stop 2024-04-02T14:42:36.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.272+0000 7fa80c1ce740 1 -- 172.21.15.90:0/2222517417 shutdown_connections 2024-04-02T14:42:36.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.272+0000 7fa80c1ce740 1 --2- 172.21.15.90:0/2222517417 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5563190cba20 0x5563190cde10 crc :-1 s=READY pgs=210 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:36.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.272+0000 7fa80c1ce740 1 --2- 172.21.15.90:0/2222517417 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5563190bb6c0 0x5563190b9170 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:36.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.272+0000 7fa80c1ce740 1 --2- 172.21.15.90:0/2222517417 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5563190b3030 0x5563190b8c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:36.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.272+0000 7fa80c1ce740 1 --2- 172.21.15.90:0/2222517417 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5563190b2350 0x5563190b86f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:36.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.272+0000 7fa80c1ce740 1 -- 172.21.15.90:0/2222517417 >> 172.21.15.90:0/2222517417 conn(0x556319046270 msgr2=0x5563190aeaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:42:36.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.272+0000 7fa7e3e00640 0 client.17037 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:42:36.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.272+0000 7fa80c1ce740 1 -- 172.21.15.90:0/2222517417 shutdown_connections 2024-04-02T14:42:36.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.272+0000 7fa80c1ce740 1 -- 172.21.15.90:0/2222517417 wait complete. 2024-04-02T14:42:36.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.272+0000 7fa80c1ce740 20 client.17037 trim_cache size 0 max 16384 2024-04-02T14:42:36.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.297+0000 7fa80c1ce740 1 Processor -- start 2024-04-02T14:42:36.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.297+0000 7fa80c1ce740 1 -- start start 2024-04-02T14:42:36.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.297+0000 7fa80c1ce740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5563190b2350 0x5563190badb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:36.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.297+0000 7fa80c1ce740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5563190b3030 0x5563190b8f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:36.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.297+0000 7fa80c1ce740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5563190bb6c0 0x5563190bcab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:36.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.297+0000 7fa80c1ce740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fa8041b3f40 con 0x5563190b2350 2024-04-02T14:42:36.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.297+0000 7fa80c1ce740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fa8041b5a80 con 0x5563190b3030 2024-04-02T14:42:36.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.297+0000 7fa80c1ce740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fa8041b4610 con 0x5563190bb6c0 2024-04-02T14:42:36.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.298+0000 7fa802800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5563190b2350 0x5563190badb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:36.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.298+0000 7fa801e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5563190b3030 0x5563190b8f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:36.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.298+0000 7fa802800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5563190b2350 0x5563190badb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:59010/0 (socket says 172.21.15.90:59010) 2024-04-02T14:42:36.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.298+0000 7fa802800640 1 -- 172.21.15.90:0/1384481584 learned_addr learned my addr 172.21.15.90:0/1384481584 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:42:36.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.298+0000 7fa803200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5563190bb6c0 0x5563190bcab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:36.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.299+0000 7fa802800640 1 -- 172.21.15.90:0/1384481584 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5563190bb6c0 msgr2=0x5563190bcab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:42:36.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.299+0000 7fa802800640 1 --2- 172.21.15.90:0/1384481584 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5563190bb6c0 0x5563190bcab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:36.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.299+0000 7fa802800640 1 -- 172.21.15.90:0/1384481584 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5563190b3030 msgr2=0x5563190b8f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:42:36.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.299+0000 7fa802800640 1 --2- 172.21.15.90:0/1384481584 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5563190b3030 0x5563190b8f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:36.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.299+0000 7fa802800640 1 -- 172.21.15.90:0/1384481584 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5563190b1380 con 0x5563190b2350 2024-04-02T14:42:36.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.299+0000 7fa801e00640 1 --2- 172.21.15.90:0/1384481584 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5563190b3030 0x5563190b8f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:42:36.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.300+0000 7fa802800640 1 --2- 172.21.15.90:0/1384481584 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5563190b2350 0x5563190badb0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fa7f4000c00 tx=0x7fa7f402e040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b716c41b12f149cc server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:42:36.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.300+0000 7fa7eea00640 1 -- 172.21.15.90:0/1384481584 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7f404b020 con 0x5563190b2350 2024-04-02T14:42:36.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.300+0000 7fa7eea00640 1 -- 172.21.15.90:0/1384481584 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa7f4047dc0 con 0x5563190b2350 2024-04-02T14:42:36.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.300+0000 7fa7eea00640 1 -- 172.21.15.90:0/1384481584 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7f40477d0 con 0x5563190b2350 2024-04-02T14:42:36.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.301+0000 7fa80c1ce740 1 -- 172.21.15.90:0/1384481584 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5563190b2350 msgr2=0x5563190badb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:36.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.301+0000 7fa80c1ce740 1 --2- 172.21.15.90:0/1384481584 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5563190b2350 0x5563190badb0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fa7f4000c00 tx=0x7fa7f402e040 comp rx=0 tx=0).stop 2024-04-02T14:42:36.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.301+0000 7fa80c1ce740 1 -- 172.21.15.90:0/1384481584 shutdown_connections 2024-04-02T14:42:36.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.301+0000 7fa80c1ce740 1 --2- 172.21.15.90:0/1384481584 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5563190bb6c0 0x5563190bcab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:36.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.301+0000 7fa80c1ce740 1 --2- 172.21.15.90:0/1384481584 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5563190b3030 0x5563190b8f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:36.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.301+0000 7fa80c1ce740 1 --2- 172.21.15.90:0/1384481584 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5563190b2350 0x5563190badb0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:36.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.301+0000 7fa80c1ce740 1 -- 172.21.15.90:0/1384481584 >> 172.21.15.90:0/1384481584 conn(0x556319046270 msgr2=0x5563190abe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:42:36.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.302+0000 7fa80c1ce740 1 -- 172.21.15.90:0/1384481584 shutdown_connections 2024-04-02T14:42:36.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.302+0000 7fa80c1ce740 1 -- 172.21.15.90:0/1384481584 wait complete. 2024-04-02T14:42:36.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.302+0000 7fa80c1ce740 1 -- start start 2024-04-02T14:42:36.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.303+0000 7fa80c1ce740 1 Processor -- start 2024-04-02T14:42:36.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.303+0000 7fa80c1ce740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5563190b2350 0x5563190d0e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:36.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.303+0000 7fa80c1ce740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5563190b3030 0x5563190d1380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:36.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.303+0000 7fa80c1ce740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5563190bb6c0 0x5563190cde10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:36.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.303+0000 7fa80c1ce740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fa8041b53b0 con 0x5563190b2350 2024-04-02T14:42:36.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.303+0000 7fa80c1ce740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fa8041b4ce0 con 0x5563190bb6c0 2024-04-02T14:42:36.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.303+0000 7fa80c1ce740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fa8041b1f40 con 0x5563190b3030 2024-04-02T14:42:36.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.303+0000 7fa80c1ce740 10 client.0 _ll_register_callbacks cb 0x7fa7fc0a0070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:42:36.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.303+0000 7fa802800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5563190b2350 0x5563190d0e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:36.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.304+0000 7fa802800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5563190b2350 0x5563190d0e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:59020/0 (socket says 172.21.15.90:59020) 2024-04-02T14:42:36.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.304+0000 7fa802800640 1 -- 172.21.15.90:0/3348143216 learned_addr learned my addr 172.21.15.90:0/3348143216 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:42:36.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.304+0000 7fa803200640 1 --2- 172.21.15.90:0/3348143216 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5563190bb6c0 0x5563190cde10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:36.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.304+0000 7fa801e00640 1 --2- 172.21.15.90:0/3348143216 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5563190b3030 0x5563190d1380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:36.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.304+0000 7fa802800640 1 -- 172.21.15.90:0/3348143216 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5563190b3030 msgr2=0x5563190d1380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:42:36.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.304+0000 7fa802800640 1 --2- 172.21.15.90:0/3348143216 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5563190b3030 0x5563190d1380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:36.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.304+0000 7fa802800640 1 -- 172.21.15.90:0/3348143216 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5563190bb6c0 msgr2=0x5563190cde10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:42:36.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.304+0000 7fa802800640 1 --2- 172.21.15.90:0/3348143216 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5563190bb6c0 0x5563190cde10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:36.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.304+0000 7fa802800640 1 -- 172.21.15.90:0/3348143216 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5563190ce3a0 con 0x5563190b2350 2024-04-02T14:42:36.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.304+0000 7fa801e00640 1 --2- 172.21.15.90:0/3348143216 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5563190b3030 0x5563190d1380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:42:36.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.304+0000 7fa802800640 1 --2- 172.21.15.90:0/3348143216 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5563190b2350 0x5563190d0e40 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fa7f4005c40 tx=0x7fa7f40043a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4e73ef40012cb77 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:42:36.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.304+0000 7fa7e3e00640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-02T14:42:36.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.305+0000 7fa7e3e00640 1 -- 172.21.15.90:0/3348143216 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7f402ccd0 con 0x5563190b2350 2024-04-02T14:42:36.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.305+0000 7fa7e3e00640 1 -- 172.21.15.90:0/3348143216 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8550+0+0 (secure 0 0 0) 0x7fa7f4005040 con 0x5563190b2350 2024-04-02T14:42:36.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.305+0000 7fa80c1ce740 1 -- 172.21.15.90:0/3348143216 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa7f4047dc0 con 0x5563190b2350 2024-04-02T14:42:36.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.305+0000 7fa7e3e00640 1 -- 172.21.15.90:0/3348143216 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fa7f4005260 con 0x5563190b2350 2024-04-02T14:42:36.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.305+0000 7fa80c1ce740 10 client.17043 fetch_fsmap learned FSMap version 27 2024-04-02T14:42:36.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.305+0000 7fa80c1ce740 1 -- 172.21.15.90:0/3348143216 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5563191324c0 con 0x5563190b2350 2024-04-02T14:42:36.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.305+0000 7fa7e3e00640 1 -- 172.21.15.90:0/3348143216 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa7f4047dc0 con 0x5563190b2350 2024-04-02T14:42:36.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.305+0000 7fa7e3e00640 1 -- 172.21.15.90:0/3348143216 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fa7f4047450 con 0x5563190b2350 2024-04-02T14:42:36.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.305+0000 7fa80c1ce740 10 client.17043 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:42:36.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.305+0000 7fa80c1ce740 10 client.17043 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:42:36.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.306+0000 7fa80c1ce740 20 client.17043 populate_metadata read hostname 'smithi090' 2024-04-02T14:42:36.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.306+0000 7fa80c1ce740 1 --2- 172.21.15.90:0/3348143216 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x556319132b30 0x556319134f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:36.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.306+0000 7fa80c1ce740 4 client.17043 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-02T14:42:36.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.306+0000 7fa80c1ce740 1 -- 172.21.15.90:0/3348143216 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55631901dba0 con 0x556319132b30 2024-04-02T14:42:36.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.306+0000 7fa801e00640 1 --2- 172.21.15.90:0/3348143216 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x556319132b30 0x556319134f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:36.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.307+0000 7fa801e00640 1 --2- 172.21.15.90:0/3348143216 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x556319132b30 0x556319134f50 crc :-1 s=READY pgs=211 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=76860e4f5c14934d server_cookie=d9a4bb9e35d339 in_seq=0 out_seq=0 2024-04-02T14:42:36.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.307+0000 7fa7e3e00640 10 client.17043 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:42:36.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.308+0000 7fa7e3e00640 1 -- 172.21.15.90:0/3348143216 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55631901dba0 con 0x556319132b30 2024-04-02T14:42:36.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.308+0000 7fa7e3e00640 10 client.17043 handle_command_reply: tid=0 2024-04-02T14:42:36.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.308+0000 7fa80c1ce740 1 client.17043 shutdown 2024-04-02T14:42:36.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.309+0000 7fa80c1ce740 1 -- 172.21.15.90:0/3348143216 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5563190b2350 msgr2=0x5563190d0e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:36.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.309+0000 7fa80c1ce740 1 --2- 172.21.15.90:0/3348143216 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5563190b2350 0x5563190d0e40 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fa7f4005c40 tx=0x7fa7f40043a0 comp rx=0 tx=0).stop 2024-04-02T14:42:36.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.309+0000 7fa80c1ce740 1 -- 172.21.15.90:0/3348143216 shutdown_connections 2024-04-02T14:42:36.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.309+0000 7fa80c1ce740 1 --2- 172.21.15.90:0/3348143216 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x556319132b30 0x556319134f50 crc :-1 s=READY pgs=211 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:36.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.309+0000 7fa80c1ce740 1 --2- 172.21.15.90:0/3348143216 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5563190bb6c0 0x5563190cde10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:36.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.309+0000 7fa80c1ce740 1 --2- 172.21.15.90:0/3348143216 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5563190b3030 0x5563190d1380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:36.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.309+0000 7fa80c1ce740 1 --2- 172.21.15.90:0/3348143216 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5563190b2350 0x5563190d0e40 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:36.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.309+0000 7fa7e3e00640 0 client.17043 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:42:36.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.309+0000 7fa80c1ce740 1 -- 172.21.15.90:0/3348143216 >> 172.21.15.90:0/3348143216 conn(0x556319046270 msgr2=0x5563190c6940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:42:36.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.310+0000 7fa80c1ce740 1 -- 172.21.15.90:0/3348143216 shutdown_connections 2024-04-02T14:42:36.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.310+0000 7fa80c1ce740 1 -- 172.21.15.90:0/3348143216 wait complete. 2024-04-02T14:42:36.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.310+0000 7fa80c1ce740 20 client.17043 trim_cache size 0 max 16384 2024-04-02T14:42:36.311 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-02T14:42:36.311 INFO:teuthology.orchestra.run.smithi090.stdout: "return_code": 0, 2024-04-02T14:42:36.311 INFO:teuthology.orchestra.run.smithi090.stdout: "scrub_tag": "1971236c-32a9-4664-9575-9f38d93618cb", 2024-04-02T14:42:36.311 INFO:teuthology.orchestra.run.smithi090.stdout: "mode": "asynchronous" 2024-04-02T14:42:36.311 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-02T14:42:36.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.311+0000 7fa809400640 1 -- 172.21.15.90:0/4191972145 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fa7e80a3da0 msgr2=0x7fa7e80a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:36.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.311+0000 7fa809400640 1 --2- 172.21.15.90:0/4191972145 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fa7e80a3da0 0x7fa7e80a6260 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x5563190b3580 tx=0x7fa7f000b570 comp rx=0 tx=0).stop 2024-04-02T14:42:36.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.311+0000 7fa809400640 1 -- 172.21.15.90:0/4191972145 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa804107a80 msgr2=0x7fa80410c9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:36.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.311+0000 7fa809400640 1 --2- 172.21.15.90:0/4191972145 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa804107a80 0x7fa80410c9d0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fa7fc00eaa0 tx=0x7fa7fc00ef60 comp rx=0 tx=0).stop 2024-04-02T14:42:36.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.312+0000 7fa809400640 1 -- 172.21.15.90:0/4191972145 shutdown_connections 2024-04-02T14:42:36.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.312+0000 7fa809400640 1 --2- 172.21.15.90:0/4191972145 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fa7e80a3da0 0x7fa7e80a6260 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:36.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.312+0000 7fa809400640 1 --2- 172.21.15.90:0/4191972145 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa804107a80 0x7fa80410c9d0 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:36.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.312+0000 7fa809400640 1 --2- 172.21.15.90:0/4191972145 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa8040696a0 0x7fa8041a3b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:36.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.312+0000 7fa809400640 1 --2- 172.21.15.90:0/4191972145 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa804068ce0 0x7fa8041a35c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:36.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.312+0000 7fa809400640 1 -- 172.21.15.90:0/4191972145 >> 172.21.15.90:0/4191972145 conn(0x7fa804077b00 msgr2=0x7fa804105580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:42:36.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.312+0000 7fa809400640 1 -- 172.21.15.90:0/4191972145 shutdown_connections 2024-04-02T14:42:36.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.312+0000 7fa809400640 1 -- 172.21.15.90:0/4191972145 wait complete. 2024-04-02T14:42:36.338 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:42:36 smithi090 ceph-mon[31674]: pgmap v236: 97 pgs: 97 active+clean; 4.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 31 MiB/s wr, 1.53k op/s 2024-04-02T14:42:36.339 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-02T14:42:36.340 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-02T14:42:36.357 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:42:36 smithi092 ceph-mon[36497]: pgmap v236: 97 pgs: 97 active+clean; 4.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 31 MiB/s wr, 1.53k op/s 2024-04-02T14:42:36.433 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:42:36 smithi177 ceph-mon[36431]: pgmap v236: 97 pgs: 97 active+clean; 4.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 31 MiB/s wr, 1.53k op/s 2024-04-02T14:42:36.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.627+0000 7f0de8e00640 1 -- 172.21.15.90:0/433992538 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0de4106220 msgr2=0x7f0de4106600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:36.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.627+0000 7f0de8e00640 1 --2- 172.21.15.90:0/433992538 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0de4106220 0x7f0de4106600 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f0dd4009900 tx=0x7f0dd402bc30 comp rx=0 tx=0).stop 2024-04-02T14:42:36.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.627+0000 7f0de8e00640 1 -- 172.21.15.90:0/433992538 shutdown_connections 2024-04-02T14:42:36.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.627+0000 7f0de8e00640 1 --2- 172.21.15.90:0/433992538 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0de4100e20 0x7f0de410cfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:36.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.627+0000 7f0de8e00640 1 --2- 172.21.15.90:0/433992538 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0de4100480 0x7f0de41008e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:36.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.627+0000 7f0de8e00640 1 --2- 172.21.15.90:0/433992538 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0de4106220 0x7f0de4106600 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:36.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.627+0000 7f0de8e00640 1 -- 172.21.15.90:0/433992538 >> 172.21.15.90:0/433992538 conn(0x7f0de4076b50 msgr2=0x7f0de40fdee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:42:36.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.627+0000 7f0de8e00640 1 -- 172.21.15.90:0/433992538 shutdown_connections 2024-04-02T14:42:36.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.627+0000 7f0de8e00640 1 -- 172.21.15.90:0/433992538 wait complete. 2024-04-02T14:42:36.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.628+0000 7f0de8e00640 1 Processor -- start 2024-04-02T14:42:36.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.628+0000 7f0de8e00640 1 -- start start 2024-04-02T14:42:36.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.629+0000 7f0de8e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0de4100480 0x7f0de4101df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:36.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.629+0000 7f0de8e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0de4100e20 0x7f0de4102330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:36.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.629+0000 7f0de8e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0de4106220 0x7f0de4102870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:36.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.629+0000 7f0de8e00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f0de4110df0 con 0x7f0de4100480 2024-04-02T14:42:36.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.629+0000 7f0de8e00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f0de4110c70 con 0x7f0de4106220 2024-04-02T14:42:36.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.629+0000 7f0de8e00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f0de4110f70 con 0x7f0de4100e20 2024-04-02T14:42:36.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.629+0000 7f0de2800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0de4106220 0x7f0de4102870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:36.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.629+0000 7f0de1400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0de4100e20 0x7f0de4102330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:36.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.629+0000 7f0de2800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0de4106220 0x7f0de4102870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:55208/0 (socket says 172.21.15.90:55208) 2024-04-02T14:42:36.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.629+0000 7f0de2800640 1 -- 172.21.15.90:0/3758364561 learned_addr learned my addr 172.21.15.90:0/3758364561 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:42:36.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.629+0000 7f0de1e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0de4100480 0x7f0de4101df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:36.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.629+0000 7f0de1400640 1 -- 172.21.15.90:0/3758364561 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0de4106220 msgr2=0x7f0de4102870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:36.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.629+0000 7f0de1400640 1 --2- 172.21.15.90:0/3758364561 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0de4106220 0x7f0de4102870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:36.630 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.629+0000 7f0de1400640 1 -- 172.21.15.90:0/3758364561 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0de4100480 msgr2=0x7f0de4101df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:36.630 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.629+0000 7f0de1400640 1 --2- 172.21.15.90:0/3758364561 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0de4100480 0x7f0de4101df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:36.630 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.629+0000 7f0de1400640 1 -- 172.21.15.90:0/3758364561 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0de419c520 con 0x7f0de4100e20 2024-04-02T14:42:36.630 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.630+0000 7f0de2800640 1 --2- 172.21.15.90:0/3758364561 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0de4106220 0x7f0de4102870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:42:36.630 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.630+0000 7f0de1400640 1 --2- 172.21.15.90:0/3758364561 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0de4100e20 0x7f0de4102330 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f0dd0005fd0 tx=0x7f0dd000dde0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:42:36.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.631+0000 7f0ddaa00640 1 -- 172.21.15.90:0/3758364561 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dd0009c30 con 0x7f0de4100e20 2024-04-02T14:42:36.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.631+0000 7f0ddaa00640 1 -- 172.21.15.90:0/3758364561 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0dd0009dd0 con 0x7f0de4100e20 2024-04-02T14:42:36.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.631+0000 7f0ddaa00640 1 -- 172.21.15.90:0/3758364561 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dd0010460 con 0x7f0de4100e20 2024-04-02T14:42:36.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.632+0000 7f0de8e00640 1 -- 172.21.15.90:0/3758364561 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0de419c810 con 0x7f0de4100e20 2024-04-02T14:42:36.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.632+0000 7f0de8e00640 1 -- 172.21.15.90:0/3758364561 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0de419cd10 con 0x7f0de4100e20 2024-04-02T14:42:36.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.637+0000 7f0debbe0740 1 Processor -- start 2024-04-02T14:42:36.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.637+0000 7f0debbe0740 1 -- start start 2024-04-02T14:42:36.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.637+0000 7f0ddaa00640 1 -- 172.21.15.90:0/3758364561 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f0dd0003ef0 con 0x7f0de4100e20 2024-04-02T14:42:36.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.638+0000 7f0debbe0740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d9ecb237b0 0x55d9ecb23b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:36.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.638+0000 7f0debbe0740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d9ecb240d0 0x55d9ecb2c770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:36.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.638+0000 7f0debbe0740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d9ecb2ccb0 0x55d9ecb2f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:36.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.638+0000 7f0debbe0740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55d9eca87770 con 0x55d9ecb237b0 2024-04-02T14:42:36.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.638+0000 7f0debbe0740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55d9eca2f700 con 0x55d9ecb2ccb0 2024-04-02T14:42:36.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.638+0000 7f0debbe0740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55d9eca83720 con 0x55d9ecb240d0 2024-04-02T14:42:36.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.638+0000 7f0de1e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d9ecb240d0 0x55d9ecb2c770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:36.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.638+0000 7f0de1400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d9ecb2ccb0 0x55d9ecb2f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:36.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.638+0000 7f0de2800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d9ecb237b0 0x55d9ecb23b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:36.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.638+0000 7f0de1400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d9ecb2ccb0 0x55d9ecb2f0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:55212/0 (socket says 172.21.15.90:55212) 2024-04-02T14:42:36.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.638+0000 7f0de1400640 1 -- 172.21.15.90:0/3220326063 learned_addr learned my addr 172.21.15.90:0/3220326063 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:42:36.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.638+0000 7f0de1e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d9ecb240d0 0x55d9ecb2c770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:47562/0 (socket says 172.21.15.90:47562) 2024-04-02T14:42:36.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.639+0000 7f0de2800640 1 -- 172.21.15.90:0/3220326063 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d9ecb240d0 msgr2=0x55d9ecb2c770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:42:36.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.639+0000 7f0de2800640 1 --2- 172.21.15.90:0/3220326063 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d9ecb240d0 0x55d9ecb2c770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:36.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.639+0000 7f0de2800640 1 -- 172.21.15.90:0/3220326063 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d9ecb2ccb0 msgr2=0x55d9ecb2f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:42:36.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.639+0000 7f0de2800640 1 --2- 172.21.15.90:0/3220326063 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d9ecb2ccb0 0x55d9ecb2f0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:36.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.639+0000 7f0de2800640 1 -- 172.21.15.90:0/3220326063 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d9ec8657d0 con 0x55d9ecb237b0 2024-04-02T14:42:36.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.639+0000 7f0de1400640 1 --2- 172.21.15.90:0/3220326063 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d9ecb2ccb0 0x55d9ecb2f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:42:36.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.639+0000 7f0de2800640 1 --2- 172.21.15.90:0/3220326063 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d9ecb237b0 0x55d9ecb23b90 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f0ddc011760 tx=0x7f0ddc011c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d63e318941571a7f server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:42:36.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.639+0000 7f0ddaa00640 1 --2- 172.21.15.90:0/3758364561 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f0dc80a3e20 0x7f0dc80a62e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:36.640 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.639+0000 7f0ddaa00640 1 -- 172.21.15.90:0/3758364561 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8550+0+0 (secure 0 0 0) 0x7f0dd00cff70 con 0x7f0de4100e20 2024-04-02T14:42:36.640 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.640+0000 7f0dcf400640 1 -- 172.21.15.90:0/3220326063 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ddc004d20 con 0x55d9ecb237b0 2024-04-02T14:42:36.640 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.640+0000 7f0de1e00640 1 --2- 172.21.15.90:0/3758364561 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f0dc80a3e20 0x7f0dc80a62e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:36.640 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.640+0000 7f0dcf400640 1 -- 172.21.15.90:0/3220326063 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0ddc005020 con 0x55d9ecb237b0 2024-04-02T14:42:36.640 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.640+0000 7f0dcf400640 1 -- 172.21.15.90:0/3220326063 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ddc019650 con 0x55d9ecb237b0 2024-04-02T14:42:36.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.641+0000 7f0de1e00640 1 --2- 172.21.15.90:0/3758364561 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f0dc80a3e20 0x7f0dc80a62e0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f0dd4000c00 tx=0x7f0dd402df90 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:42:36.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.641+0000 7f0debbe0740 1 -- 172.21.15.90:0/3220326063 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d9ecb237b0 msgr2=0x55d9ecb23b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:36.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.641+0000 7f0debbe0740 1 --2- 172.21.15.90:0/3220326063 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d9ecb237b0 0x55d9ecb23b90 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f0ddc011760 tx=0x7f0ddc011c20 comp rx=0 tx=0).stop 2024-04-02T14:42:36.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.642+0000 7f0debbe0740 1 -- 172.21.15.90:0/3220326063 shutdown_connections 2024-04-02T14:42:36.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.642+0000 7f0debbe0740 1 --2- 172.21.15.90:0/3220326063 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d9ecb2ccb0 0x55d9ecb2f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:36.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.642+0000 7f0debbe0740 1 --2- 172.21.15.90:0/3220326063 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d9ecb240d0 0x55d9ecb2c770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:36.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.642+0000 7f0debbe0740 1 --2- 172.21.15.90:0/3220326063 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d9ecb237b0 0x55d9ecb23b90 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:36.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.642+0000 7f0debbe0740 1 -- 172.21.15.90:0/3220326063 >> 172.21.15.90:0/3220326063 conn(0x55d9ecb1dee0 msgr2=0x55d9ecb1e2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:42:36.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.642+0000 7f0debbe0740 1 -- 172.21.15.90:0/3220326063 shutdown_connections 2024-04-02T14:42:36.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.643+0000 7f0debbe0740 1 -- 172.21.15.90:0/3220326063 wait complete. 2024-04-02T14:42:36.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.643+0000 7f0debbe0740 1 -- start start 2024-04-02T14:42:36.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.643+0000 7f0debbe0740 1 Processor -- start 2024-04-02T14:42:36.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.644+0000 7f0debbe0740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d9ecb237b0 0x55d9ecb29d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:36.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.644+0000 7f0debbe0740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d9ecb240d0 0x55d9ecb2a250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:36.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.644+0000 7f0debbe0740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d9ecb2ccb0 0x55d9ecb2a790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:36.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.644+0000 7f0debbe0740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55d9ecb2faa0 con 0x55d9ecb2ccb0 2024-04-02T14:42:36.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.644+0000 7f0debbe0740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55d9eca42ac0 con 0x55d9ecb240d0 2024-04-02T14:42:36.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.644+0000 7f0debbe0740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55d9ecb2fc20 con 0x55d9ecb237b0 2024-04-02T14:42:36.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.644+0000 7f0debbe0740 10 client.0 _ll_register_callbacks cb 0x55d9eca45180 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:42:36.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.644+0000 7f0de2800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d9ecb2ccb0 0x55d9ecb2a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:36.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.644+0000 7f0de1400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d9ecb240d0 0x55d9ecb2a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:36.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.645+0000 7f0de2800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d9ecb2ccb0 0x55d9ecb2a790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:59056/0 (socket says 172.21.15.90:59056) 2024-04-02T14:42:36.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.645+0000 7f0de2800640 1 -- 172.21.15.90:0/2827146593 learned_addr learned my addr 172.21.15.90:0/2827146593 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:42:36.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.645+0000 7f0de1e00640 1 --2- 172.21.15.90:0/2827146593 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d9ecb237b0 0x55d9ecb29d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:36.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.645+0000 7f0de2800640 1 -- 172.21.15.90:0/2827146593 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d9ecb237b0 msgr2=0x55d9ecb29d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:42:36.646 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.645+0000 7f0de2800640 1 --2- 172.21.15.90:0/2827146593 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d9ecb237b0 0x55d9ecb29d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:36.646 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.645+0000 7f0de2800640 1 -- 172.21.15.90:0/2827146593 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d9ecb240d0 msgr2=0x55d9ecb2a250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:42:36.646 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.645+0000 7f0de2800640 1 --2- 172.21.15.90:0/2827146593 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d9ecb240d0 0x55d9ecb2a250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:36.646 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.645+0000 7f0de2800640 1 -- 172.21.15.90:0/2827146593 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d9ec79b3f0 con 0x55d9ecb2ccb0 2024-04-02T14:42:36.646 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.646+0000 7f0de2800640 1 --2- 172.21.15.90:0/2827146593 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d9ecb2ccb0 0x55d9ecb2a790 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f0ddc0049d0 tx=0x7f0ddc004c70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=865cd329d6e59bdc server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:42:36.646 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.646+0000 7f0dc3e00640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-02T14:42:36.646 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.646+0000 7f0dc3e00640 1 -- 172.21.15.90:0/2827146593 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ddc028070 con 0x55d9ecb2ccb0 2024-04-02T14:42:36.646 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.646+0000 7f0dc3e00640 1 -- 172.21.15.90:0/2827146593 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8550+0+0 (secure 0 0 0) 0x7f0ddc00fa60 con 0x55d9ecb2ccb0 2024-04-02T14:42:36.646 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.646+0000 7f0debbe0740 1 -- 172.21.15.90:0/2827146593 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0ddc005020 con 0x55d9ecb2ccb0 2024-04-02T14:42:36.646 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.646+0000 7f0dc3e00640 1 -- 172.21.15.90:0/2827146593 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f0ddc034050 con 0x55d9ecb2ccb0 2024-04-02T14:42:36.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.647+0000 7f0debbe0740 10 client.17055 fetch_fsmap learned FSMap version 27 2024-04-02T14:42:36.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.647+0000 7f0debbe0740 1 -- 172.21.15.90:0/2827146593 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55d9ec75b0f0 con 0x55d9ecb2ccb0 2024-04-02T14:42:36.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.647+0000 7f0dc3e00640 1 -- 172.21.15.90:0/2827146593 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0ddc005020 con 0x55d9ecb2ccb0 2024-04-02T14:42:36.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.648+0000 7f0dc3e00640 1 -- 172.21.15.90:0/2827146593 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f0ddc0105e0 con 0x55d9ecb2ccb0 2024-04-02T14:42:36.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.648+0000 7f0debbe0740 10 client.17055 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:42:36.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.648+0000 7f0debbe0740 10 client.17055 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:42:36.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.648+0000 7f0debbe0740 20 client.17055 populate_metadata read hostname 'smithi090' 2024-04-02T14:42:36.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.648+0000 7f0debbe0740 1 --2- 172.21.15.90:0/2827146593 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55d9ecb3cc20 0x55d9ecb3f010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:36.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.648+0000 7f0debbe0740 4 client.17055 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:42:36.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.648+0000 7f0debbe0740 1 -- 172.21.15.90:0/2827146593 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d9eca28630 con 0x55d9ecb3cc20 2024-04-02T14:42:36.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.648+0000 7f0de1e00640 1 --2- 172.21.15.90:0/2827146593 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55d9ecb3cc20 0x55d9ecb3f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:36.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.649+0000 7f0de1e00640 1 --2- 172.21.15.90:0/2827146593 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55d9ecb3cc20 0x55d9ecb3f010 crc :-1 s=READY pgs=212 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c28563e8fcb2745f server_cookie=76336734f8adb5cf in_seq=0 out_seq=0 2024-04-02T14:42:36.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.649+0000 7f0dc3e00640 10 client.17055 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:42:36.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.650+0000 7f0dc3e00640 1 -- 172.21.15.90:0/2827146593 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d9eca28630 con 0x55d9ecb3cc20 2024-04-02T14:42:36.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.650+0000 7f0dc3e00640 10 client.17055 handle_command_reply: tid=0 2024-04-02T14:42:36.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.650+0000 7f0debbe0740 1 client.17055 shutdown 2024-04-02T14:42:36.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.650+0000 7f0debbe0740 1 -- 172.21.15.90:0/2827146593 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d9ecb2ccb0 msgr2=0x55d9ecb2a790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:36.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.650+0000 7f0debbe0740 1 --2- 172.21.15.90:0/2827146593 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d9ecb2ccb0 0x55d9ecb2a790 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f0ddc0049d0 tx=0x7f0ddc004c70 comp rx=0 tx=0).stop 2024-04-02T14:42:36.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.650+0000 7f0debbe0740 1 -- 172.21.15.90:0/2827146593 shutdown_connections 2024-04-02T14:42:36.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.650+0000 7f0debbe0740 1 --2- 172.21.15.90:0/2827146593 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55d9ecb3cc20 0x55d9ecb3f010 crc :-1 s=READY pgs=212 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:36.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.650+0000 7f0debbe0740 1 --2- 172.21.15.90:0/2827146593 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d9ecb2ccb0 0x55d9ecb2a790 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:36.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.650+0000 7f0debbe0740 1 --2- 172.21.15.90:0/2827146593 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d9ecb240d0 0x55d9ecb2a250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:36.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.650+0000 7f0debbe0740 1 --2- 172.21.15.90:0/2827146593 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d9ecb237b0 0x55d9ecb29d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:36.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.650+0000 7f0debbe0740 1 -- 172.21.15.90:0/2827146593 >> 172.21.15.90:0/2827146593 conn(0x55d9ecb1dee0 msgr2=0x55d9ecb1fc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:42:36.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.650+0000 7f0dc3e00640 0 client.17055 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:42:36.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.650+0000 7f0debbe0740 1 -- 172.21.15.90:0/2827146593 shutdown_connections 2024-04-02T14:42:36.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.650+0000 7f0debbe0740 1 -- 172.21.15.90:0/2827146593 wait complete. 2024-04-02T14:42:36.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.650+0000 7f0debbe0740 20 client.17055 trim_cache size 0 max 16384 2024-04-02T14:42:36.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.688+0000 7f0debbe0740 1 Processor -- start 2024-04-02T14:42:36.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.689+0000 7f0debbe0740 1 -- start start 2024-04-02T14:42:36.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.689+0000 7f0debbe0740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d9ecb237b0 0x55d9ecb38960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:36.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.689+0000 7f0debbe0740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d9ecb240d0 0x55d9ecb294e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:36.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.689+0000 7f0debbe0740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d9ecb2ccb0 0x55d9ecb1f940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:36.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.689+0000 7f0debbe0740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f0de41bc6f0 con 0x55d9ecb2ccb0 2024-04-02T14:42:36.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.689+0000 7f0debbe0740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f0de41be230 con 0x55d9ecb240d0 2024-04-02T14:42:36.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.689+0000 7f0debbe0740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f0de41bcdc0 con 0x55d9ecb237b0 2024-04-02T14:42:36.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.690+0000 7f0de1e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d9ecb237b0 0x55d9ecb38960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:36.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.690+0000 7f0de1e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d9ecb237b0 0x55d9ecb38960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:47582/0 (socket says 172.21.15.90:47582) 2024-04-02T14:42:36.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.690+0000 7f0de1e00640 1 -- 172.21.15.90:0/2066471200 learned_addr learned my addr 172.21.15.90:0/2066471200 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:42:36.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.690+0000 7f0de1e00640 1 -- 172.21.15.90:0/2066471200 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d9ecb240d0 msgr2=0x55d9ecb294e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:42:36.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.690+0000 7f0de1e00640 1 --2- 172.21.15.90:0/2066471200 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d9ecb240d0 0x55d9ecb294e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:36.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.690+0000 7f0de1e00640 1 -- 172.21.15.90:0/2066471200 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d9ecb2ccb0 msgr2=0x55d9ecb1f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:42:36.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.690+0000 7f0de1e00640 1 --2- 172.21.15.90:0/2066471200 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d9ecb2ccb0 0x55d9ecb1f940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:36.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.690+0000 7f0de1e00640 1 -- 172.21.15.90:0/2066471200 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d9ecb2abd0 con 0x55d9ecb237b0 2024-04-02T14:42:36.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.691+0000 7f0de1e00640 1 --2- 172.21.15.90:0/2066471200 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d9ecb237b0 0x55d9ecb38960 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f0dd4005ec0 tx=0x7f0dd4042000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=997a568377a4b57c server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:42:36.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.691+0000 7f0dcea00640 1 -- 172.21.15.90:0/2066471200 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dd405e070 con 0x55d9ecb237b0 2024-04-02T14:42:36.690 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.691+0000 7f0dcea00640 1 -- 172.21.15.90:0/2066471200 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0dd40046c0 con 0x55d9ecb237b0 2024-04-02T14:42:36.690 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.691+0000 7f0dcea00640 1 -- 172.21.15.90:0/2066471200 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dd4046070 con 0x55d9ecb237b0 2024-04-02T14:42:36.690 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.692+0000 7f0debbe0740 1 -- 172.21.15.90:0/2066471200 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d9ecb237b0 msgr2=0x55d9ecb38960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:36.691 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.692+0000 7f0debbe0740 1 --2- 172.21.15.90:0/2066471200 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d9ecb237b0 0x55d9ecb38960 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f0dd4005ec0 tx=0x7f0dd4042000 comp rx=0 tx=0).stop 2024-04-02T14:42:36.691 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.692+0000 7f0debbe0740 1 -- 172.21.15.90:0/2066471200 shutdown_connections 2024-04-02T14:42:36.691 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.692+0000 7f0debbe0740 1 --2- 172.21.15.90:0/2066471200 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d9ecb2ccb0 0x55d9ecb1f940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:36.691 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.692+0000 7f0debbe0740 1 --2- 172.21.15.90:0/2066471200 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d9ecb240d0 0x55d9ecb294e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:36.691 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.692+0000 7f0debbe0740 1 --2- 172.21.15.90:0/2066471200 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d9ecb237b0 0x55d9ecb38960 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:36.691 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.692+0000 7f0debbe0740 1 -- 172.21.15.90:0/2066471200 >> 172.21.15.90:0/2066471200 conn(0x55d9ecb1dee0 msgr2=0x55d9ecb1f230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:42:36.691 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.692+0000 7f0debbe0740 1 -- 172.21.15.90:0/2066471200 shutdown_connections 2024-04-02T14:42:36.691 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.692+0000 7f0debbe0740 1 -- 172.21.15.90:0/2066471200 wait complete. 2024-04-02T14:42:36.691 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.693+0000 7f0debbe0740 1 -- start start 2024-04-02T14:42:36.691 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.693+0000 7f0debbe0740 1 Processor -- start 2024-04-02T14:42:36.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.694+0000 7f0debbe0740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d9ecb237b0 0x55d9ecb9e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:36.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.694+0000 7f0debbe0740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d9ecb240d0 0x55d9ecb34e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:36.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.694+0000 7f0debbe0740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d9ecb2ccb0 0x55d9ecb353b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:36.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.694+0000 7f0debbe0740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f0de41bdb60 con 0x55d9ecb237b0 2024-04-02T14:42:36.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.694+0000 7f0debbe0740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f0de41bd490 con 0x55d9ecb240d0 2024-04-02T14:42:36.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.694+0000 7f0debbe0740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f0de4068d40 con 0x55d9ecb2ccb0 2024-04-02T14:42:36.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.694+0000 7f0debbe0740 10 client.0 _ll_register_callbacks cb 0x7f0ddc011660 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:42:36.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.694+0000 7f0de1400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d9ecb240d0 0x55d9ecb34e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:36.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.694+0000 7f0de1400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d9ecb240d0 0x55d9ecb34e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:55244/0 (socket says 172.21.15.90:55244) 2024-04-02T14:42:36.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.694+0000 7f0de1400640 1 -- 172.21.15.90:0/749740267 learned_addr learned my addr 172.21.15.90:0/749740267 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:42:36.694 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.694+0000 7f0de2800640 1 --2- 172.21.15.90:0/749740267 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d9ecb2ccb0 0x55d9ecb353b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:36.694 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.694+0000 7f0de1400640 1 -- 172.21.15.90:0/749740267 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d9ecb2ccb0 msgr2=0x55d9ecb353b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:42:36.694 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.694+0000 7f0de1400640 1 --2- 172.21.15.90:0/749740267 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d9ecb2ccb0 0x55d9ecb353b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:36.694 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.694+0000 7f0de1400640 1 -- 172.21.15.90:0/749740267 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d9ecb237b0 msgr2=0x55d9ecb9e5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:42:36.694 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.694+0000 7f0de1400640 1 --2- 172.21.15.90:0/749740267 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d9ecb237b0 0x55d9ecb9e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:36.694 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.695+0000 7f0de1400640 1 -- 172.21.15.90:0/749740267 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d9ecb26960 con 0x55d9ecb240d0 2024-04-02T14:42:36.695 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.695+0000 7f0de2800640 1 --2- 172.21.15.90:0/749740267 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d9ecb2ccb0 0x55d9ecb353b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:42:36.695 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.695+0000 7f0de1400640 1 --2- 172.21.15.90:0/749740267 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d9ecb240d0 0x55d9ecb34e70 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f0dd0098910 tx=0x7f0dd00d34d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f2d38b40f16c4f01 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:42:36.695 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.695+0000 7f0dc3e00640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:42:36.695 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.695+0000 7f0dc3e00640 1 -- 172.21.15.90:0/749740267 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dd0020040 con 0x55d9ecb240d0 2024-04-02T14:42:36.695 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.695+0000 7f0dc3e00640 1 -- 172.21.15.90:0/749740267 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8550+0+0 (secure 0 0 0) 0x7f0dd00b52d0 con 0x55d9ecb240d0 2024-04-02T14:42:36.695 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.695+0000 7f0debbe0740 1 -- 172.21.15.90:0/749740267 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0dd40046c0 con 0x55d9ecb240d0 2024-04-02T14:42:36.695 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.696+0000 7f0dc3e00640 1 -- 172.21.15.90:0/749740267 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f0dd009ba80 con 0x55d9ecb240d0 2024-04-02T14:42:36.695 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.696+0000 7f0debbe0740 10 client.27821 fetch_fsmap learned FSMap version 27 2024-04-02T14:42:36.695 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.696+0000 7f0debbe0740 1 -- 172.21.15.90:0/749740267 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55d9ecbb4340 con 0x55d9ecb240d0 2024-04-02T14:42:36.695 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.696+0000 7f0dc3e00640 1 -- 172.21.15.90:0/749740267 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0dd40046c0 con 0x55d9ecb240d0 2024-04-02T14:42:36.695 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.696+0000 7f0dc3e00640 1 -- 172.21.15.90:0/749740267 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f0dd0023060 con 0x55d9ecb240d0 2024-04-02T14:42:36.695 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.696+0000 7f0debbe0740 10 client.27821 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:42:36.695 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.696+0000 7f0debbe0740 10 client.27821 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:42:36.695 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.696+0000 7f0debbe0740 20 client.27821 populate_metadata read hostname 'smithi090' 2024-04-02T14:42:36.695 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.696+0000 7f0debbe0740 1 --2- 172.21.15.90:0/749740267 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55d9ecbb4af0 0x55d9ecbb6f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:36.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.696+0000 7f0debbe0740 4 client.27821 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-02T14:42:36.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.696+0000 7f0debbe0740 1 -- 172.21.15.90:0/749740267 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55d9eca94cb0 con 0x55d9ecbb4af0 2024-04-02T14:42:36.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.697+0000 7f0de1e00640 1 --2- 172.21.15.90:0/749740267 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55d9ecbb4af0 0x55d9ecbb6f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:36.697 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.697+0000 7f0de1e00640 1 --2- 172.21.15.90:0/749740267 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55d9ecbb4af0 0x55d9ecbb6f10 crc :-1 s=READY pgs=213 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=794f097a02b3eff1 server_cookie=437bcea5dc89b828 in_seq=0 out_seq=0 2024-04-02T14:42:36.697 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.697+0000 7f0dc3e00640 10 client.27821 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:42:36.697 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.697+0000 7f0dc3e00640 1 -- 172.21.15.90:0/749740267 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55d9eca94cb0 con 0x55d9ecbb4af0 2024-04-02T14:42:36.697 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.697+0000 7f0dc3e00640 10 client.27821 handle_command_reply: tid=0 2024-04-02T14:42:36.697 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.697+0000 7f0debbe0740 1 client.27821 shutdown 2024-04-02T14:42:36.697 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.697+0000 7f0debbe0740 1 -- 172.21.15.90:0/749740267 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d9ecb240d0 msgr2=0x55d9ecb34e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:36.697 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.697+0000 7f0debbe0740 1 --2- 172.21.15.90:0/749740267 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d9ecb240d0 0x55d9ecb34e70 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f0dd0098910 tx=0x7f0dd00d34d0 comp rx=0 tx=0).stop 2024-04-02T14:42:36.697 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.698+0000 7f0debbe0740 1 -- 172.21.15.90:0/749740267 shutdown_connections 2024-04-02T14:42:36.697 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.698+0000 7f0debbe0740 1 --2- 172.21.15.90:0/749740267 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55d9ecbb4af0 0x55d9ecbb6f10 crc :-1 s=READY pgs=213 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:36.697 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.698+0000 7f0debbe0740 1 --2- 172.21.15.90:0/749740267 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d9ecb2ccb0 0x55d9ecb353b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:36.697 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.698+0000 7f0debbe0740 1 --2- 172.21.15.90:0/749740267 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d9ecb240d0 0x55d9ecb34e70 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:36.697 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.698+0000 7f0debbe0740 1 --2- 172.21.15.90:0/749740267 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d9ecb237b0 0x55d9ecb9e5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:36.697 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.698+0000 7f0dc3e00640 0 client.27821 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:42:36.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.698+0000 7f0debbe0740 1 -- 172.21.15.90:0/749740267 >> 172.21.15.90:0/749740267 conn(0x55d9ecb1dee0 msgr2=0x55d9ecb1fac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:42:36.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.699+0000 7f0debbe0740 1 -- 172.21.15.90:0/749740267 shutdown_connections 2024-04-02T14:42:36.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.699+0000 7f0debbe0740 1 -- 172.21.15.90:0/749740267 wait complete. 2024-04-02T14:42:36.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.699+0000 7f0debbe0740 20 client.27821 trim_cache size 0 max 16384 2024-04-02T14:42:36.698 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-02T14:42:36.698 INFO:teuthology.orchestra.run.smithi090.stdout: "status": "no active scrubs running", 2024-04-02T14:42:36.699 INFO:teuthology.orchestra.run.smithi090.stdout: "scrubs": {} 2024-04-02T14:42:36.699 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-02T14:42:36.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.700+0000 7f0de8e00640 1 -- 172.21.15.90:0/3758364561 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f0dc80a3e20 msgr2=0x7f0dc80a62e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:36.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.700+0000 7f0de8e00640 1 --2- 172.21.15.90:0/3758364561 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f0dc80a3e20 0x7f0dc80a62e0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f0dd4000c00 tx=0x7f0dd402df90 comp rx=0 tx=0).stop 2024-04-02T14:42:36.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.700+0000 7f0de8e00640 1 -- 172.21.15.90:0/3758364561 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0de4100e20 msgr2=0x7f0de4102330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:36.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.700+0000 7f0de8e00640 1 --2- 172.21.15.90:0/3758364561 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0de4100e20 0x7f0de4102330 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f0dd0005fd0 tx=0x7f0dd000dde0 comp rx=0 tx=0).stop 2024-04-02T14:42:36.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.701+0000 7f0de8e00640 1 -- 172.21.15.90:0/3758364561 shutdown_connections 2024-04-02T14:42:36.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.701+0000 7f0de8e00640 1 --2- 172.21.15.90:0/3758364561 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f0dc80a3e20 0x7f0dc80a62e0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:36.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.701+0000 7f0de8e00640 1 --2- 172.21.15.90:0/3758364561 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0de4106220 0x7f0de4102870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:36.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.701+0000 7f0de8e00640 1 --2- 172.21.15.90:0/3758364561 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0de4100e20 0x7f0de4102330 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:36.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.701+0000 7f0de8e00640 1 --2- 172.21.15.90:0/3758364561 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0de4100480 0x7f0de4101df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:36.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.701+0000 7f0de8e00640 1 -- 172.21.15.90:0/3758364561 >> 172.21.15.90:0/3758364561 conn(0x7f0de4076b50 msgr2=0x7f0de40fe8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:42:36.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.701+0000 7f0de8e00640 1 -- 172.21.15.90:0/3758364561 shutdown_connections 2024-04-02T14:42:36.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.701+0000 7f0de8e00640 1 -- 172.21.15.90:0/3758364561 wait complete. 2024-04-02T14:42:36.722 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-02T14:42:36.722 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T14:42:36.828 INFO:teuthology.orchestra.run.smithi090.stderr:progress: 425.0 s, 2141.1 tps, lat 45.984 ms stddev 51.340 2024-04-02T14:42:36.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.994+0000 7fbdd1200640 1 -- 172.21.15.90:0/3168681161 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbdcc112930 msgr2=0x7fbdcc114d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:36.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.994+0000 7fbdd1200640 1 --2- 172.21.15.90:0/3168681161 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbdcc112930 0x7fbdcc114d20 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7fbdc4009cd0 tx=0x7fbdc402c0e0 comp rx=0 tx=0).stop 2024-04-02T14:42:36.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.997+0000 7fbdd1200640 1 -- 172.21.15.90:0/3168681161 shutdown_connections 2024-04-02T14:42:36.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.997+0000 7fbdd1200640 1 --2- 172.21.15.90:0/3168681161 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbdcc112930 0x7fbdcc114d20 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:36.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.997+0000 7fbdd1200640 1 --2- 172.21.15.90:0/3168681161 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbdcc076950 0x7fbdcc1123f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:36.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.997+0000 7fbdd1200640 1 --2- 172.21.15.90:0/3168681161 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbdcc076030 0x7fbdcc076410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:36.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.997+0000 7fbdd1200640 1 -- 172.21.15.90:0/3168681161 >> 172.21.15.90:0/3168681161 conn(0x7fbdcc0ff760 msgr2=0x7fbdcc101b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:42:36.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.998+0000 7fbdd1200640 1 -- 172.21.15.90:0/3168681161 shutdown_connections 2024-04-02T14:42:36.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.998+0000 7fbdd1200640 1 -- 172.21.15.90:0/3168681161 wait complete. 2024-04-02T14:42:36.997 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.999+0000 7fbdd1200640 1 Processor -- start 2024-04-02T14:42:36.997 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:36.999+0000 7fbdd1200640 1 -- start start 2024-04-02T14:42:36.997 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.000+0000 7fbdd1200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbdcc076030 0x7fbdcc1a4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:36.998 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.000+0000 7fbdd1200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbdcc076950 0x7fbdcc1a4cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:36.998 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.000+0000 7fbdd1200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbdcc112930 0x7fbdcc19e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:36.998 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.000+0000 7fbdd1200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fbdcc1184d0 con 0x7fbdcc112930 2024-04-02T14:42:36.998 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.000+0000 7fbdd1200640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fbdcc118350 con 0x7fbdcc076950 2024-04-02T14:42:36.998 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.000+0000 7fbdd1200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fbdcc118650 con 0x7fbdcc076030 2024-04-02T14:42:36.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.000+0000 7fbdc9e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbdcc076030 0x7fbdcc1a4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:37.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.000+0000 7fbdca800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbdcc112930 0x7fbdcc19e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:37.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.000+0000 7fbdc9e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbdcc076030 0x7fbdcc1a4770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:47614/0 (socket says 172.21.15.90:47614) 2024-04-02T14:42:37.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.000+0000 7fbdc9e00640 1 -- 172.21.15.90:0/1220448605 learned_addr learned my addr 172.21.15.90:0/1220448605 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:42:37.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.001+0000 7fbdca800640 1 -- 172.21.15.90:0/1220448605 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbdcc076030 msgr2=0x7fbdcc1a4770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:37.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.001+0000 7fbdca800640 1 --2- 172.21.15.90:0/1220448605 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbdcc076030 0x7fbdcc1a4770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:37.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.001+0000 7fbdca800640 1 -- 172.21.15.90:0/1220448605 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbdcc076950 msgr2=0x7fbdcc1a4cb0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-02T14:42:37.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.001+0000 7fbdca800640 1 --2- 172.21.15.90:0/1220448605 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbdcc076950 0x7fbdcc1a4cb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:37.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.001+0000 7fbdca800640 1 -- 172.21.15.90:0/1220448605 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbdcc19f0a0 con 0x7fbdcc112930 2024-04-02T14:42:37.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.001+0000 7fbdca800640 1 --2- 172.21.15.90:0/1220448605 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbdcc112930 0x7fbdcc19e920 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fbdc402c6f0 tx=0x7fbdc4035d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:42:37.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.001+0000 7fbdc2a00640 1 -- 172.21.15.90:0/1220448605 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdc4039040 con 0x7fbdcc112930 2024-04-02T14:42:37.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.002+0000 7fbdc2a00640 1 -- 172.21.15.90:0/1220448605 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbdc4044070 con 0x7fbdcc112930 2024-04-02T14:42:37.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.002+0000 7fbdc2a00640 1 -- 172.21.15.90:0/1220448605 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdc40043c0 con 0x7fbdcc112930 2024-04-02T14:42:37.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.002+0000 7fbdd1200640 1 -- 172.21.15.90:0/1220448605 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbdcc19f330 con 0x7fbdcc112930 2024-04-02T14:42:37.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.002+0000 7fbdd1200640 1 -- 172.21.15.90:0/1220448605 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbdcc19f6e0 con 0x7fbdcc112930 2024-04-02T14:42:37.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.004+0000 7fbdd1200640 1 -- 172.21.15.90:0/1220448605 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbdcc068d40 con 0x7fbdcc112930 2024-04-02T14:42:37.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.006+0000 7fbdc2a00640 1 -- 172.21.15.90:0/1220448605 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fbdc4004560 con 0x7fbdcc112930 2024-04-02T14:42:37.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.007+0000 7fbdc2a00640 1 --2- 172.21.15.90:0/1220448605 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fbdb00a3da0 0x7fbdb00a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:37.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.008+0000 7fbdc2a00640 1 -- 172.21.15.90:0/1220448605 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8550+0+0 (secure 0 0 0) 0x7fbdc40f2c10 con 0x7fbdcc112930 2024-04-02T14:42:37.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.008+0000 7fbdc9e00640 1 --2- 172.21.15.90:0/1220448605 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fbdb00a3da0 0x7fbdb00a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:37.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.009+0000 7fbdc9e00640 1 --2- 172.21.15.90:0/1220448605 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fbdb00a3da0 0x7fbdb00a6260 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fbdcc19fa60 tx=0x7fbdbc005f90 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:42:37.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.018+0000 7fbdc2a00640 1 -- 172.21.15.90:0/1220448605 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7fbdc403e3a0 con 0x7fbdcc112930 2024-04-02T14:42:37.357 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:42:37 smithi092 ceph-mon[36497]: scrub queued for path: / 2024-04-02T14:42:37.358 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:42:37 smithi092 ceph-mon[36497]: scrub summary: idle+waiting paths [/] 2024-04-02T14:42:37.358 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:42:37 smithi092 ceph-mon[36497]: scrub summary: active paths [/] 2024-04-02T14:42:37.358 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:42:37 smithi092 ceph-mon[36497]: scrub summary: idle+waiting paths [/] 2024-04-02T14:42:37.358 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:42:37 smithi092 ceph-mon[36497]: scrub summary: idle 2024-04-02T14:42:37.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.405+0000 7fbdd1200640 1 -- 172.21.15.90:0/1220448605 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbdcc02d3a0 con 0x7fbdcc112930 2024-04-02T14:42:37.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.406+0000 7fbdc2a00640 1 -- 172.21.15.90:0/1220448605 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11927 (secure 0 0 0) 0x7fbdc40bf3a0 con 0x7fbdcc112930 2024-04-02T14:42:37.404 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-02T14:42:37.404 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14412,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6837/2852160309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6836","nonce":2852160309},{"type":"v1","addr":"172.21.15.90:6837","nonce":2852160309}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14478,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6841/1919376858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6840","nonce":1919376858},{"type":"v1","addr":"172.21.15.90:6841","nonce":1919376858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-02T14:32:22.319083+0000","modified":"2024-04-02T14:34:52.266152+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24590,"mds_1":24560,"mds_2":24547,"mds_3":24539,"mds_4":24523},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24454":{"gid":24454,"name":"l","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6833/1564319431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":1564319431},{"type":"v1","addr":"172.21.15.177:6833","nonce":1564319431}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.92:6833/1570337372","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":1570337372},{"type":"v1","addr":"172.21.15.92:6833","nonce":1570337372}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"j","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.90:6835/1277124891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6834","nonce":1277124891},{"type":"v1","addr":"172.21.15.90:6835","nonce":1277124891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.92:6835/860721164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":860721164},{"type":"v1","addr":"172.21.15.92:6835","nonce":860721164}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24511":{"gid":24511,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6835/3177244988","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":3177244988},{"type":"v1","addr":"172.21.15.177:6835","nonce":3177244988}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"c","rank":4,"incarnation":25,"state":"up:active","state_seq":9,"addr":"172.21.15.177:6837/4223414858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":4223414858},{"type":"v1","addr":"172.21.15.177:6837","nonce":4223414858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24539":{"gid":24539,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":8,"addr":"172.21.15.92:6837/1270635939","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":1270635939},{"type":"v1","addr":"172.21.15.92:6837","nonce":1270635939}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24547":{"gid":24547,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":7,"addr":"172.21.15.177:6839/664895663","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":664895663},{"type":"v1","addr":"172.21.15.177:6839","nonce":664895663}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"g","rank":1,"incarnation":19,"state":"up:active","state_seq":5,"addr":"172.21.15.90:6839/1498915870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6838","nonce":1498915870},{"type":"v1","addr":"172.21.15.90:6839","nonce":1498915870}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24590":{"gid":24590,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.92:6839/1054888219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":1054888219},{"type":"v1","addr":"172.21.15.92:6839","nonce":1054888219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24590,"qdb_cluster":[24547,24590,24523,24539,24560]},"id":1}]} 2024-04-02T14:42:37.404 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 27 2024-04-02T14:42:37.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.409+0000 7fbdd1200640 1 -- 172.21.15.90:0/1220448605 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fbdb00a3da0 msgr2=0x7fbdb00a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:37.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.409+0000 7fbdd1200640 1 --2- 172.21.15.90:0/1220448605 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fbdb00a3da0 0x7fbdb00a6260 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fbdcc19fa60 tx=0x7fbdbc005f90 comp rx=0 tx=0).stop 2024-04-02T14:42:37.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.409+0000 7fbdd1200640 1 -- 172.21.15.90:0/1220448605 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbdcc112930 msgr2=0x7fbdcc19e920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:37.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.409+0000 7fbdd1200640 1 --2- 172.21.15.90:0/1220448605 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbdcc112930 0x7fbdcc19e920 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fbdc402c6f0 tx=0x7fbdc4035d10 comp rx=0 tx=0).stop 2024-04-02T14:42:37.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.409+0000 7fbdd1200640 1 -- 172.21.15.90:0/1220448605 shutdown_connections 2024-04-02T14:42:37.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.409+0000 7fbdd1200640 1 --2- 172.21.15.90:0/1220448605 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fbdb00a3da0 0x7fbdb00a6260 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:37.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.409+0000 7fbdd1200640 1 --2- 172.21.15.90:0/1220448605 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbdcc112930 0x7fbdcc19e920 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:37.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.409+0000 7fbdd1200640 1 --2- 172.21.15.90:0/1220448605 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbdcc076950 0x7fbdcc1a4cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:37.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.409+0000 7fbdd1200640 1 --2- 172.21.15.90:0/1220448605 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbdcc076030 0x7fbdcc1a4770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:37.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.409+0000 7fbdd1200640 1 -- 172.21.15.90:0/1220448605 >> 172.21.15.90:0/1220448605 conn(0x7fbdcc0ff760 msgr2=0x7fbdcc1140c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:42:37.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.409+0000 7fbdd1200640 1 -- 172.21.15.90:0/1220448605 shutdown_connections 2024-04-02T14:42:37.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.409+0000 7fbdd1200640 1 -- 172.21.15.90:0/1220448605 wait complete. 2024-04-02T14:42:37.425 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:42:37 smithi090 ceph-mon[31674]: scrub queued for path: / 2024-04-02T14:42:37.426 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:42:37 smithi090 ceph-mon[31674]: scrub summary: idle+waiting paths [/] 2024-04-02T14:42:37.426 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:42:37 smithi090 ceph-mon[31674]: scrub summary: active paths [/] 2024-04-02T14:42:37.426 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:42:37 smithi090 ceph-mon[31674]: scrub summary: idle+waiting paths [/] 2024-04-02T14:42:37.426 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:42:37 smithi090 ceph-mon[31674]: scrub summary: idle 2024-04-02T14:42:37.428 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-04-02T14:42:37.433 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:42:37 smithi177 ceph-mon[36431]: scrub queued for path: / 2024-04-02T14:42:37.434 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:42:37 smithi177 ceph-mon[36431]: scrub summary: idle+waiting paths [/] 2024-04-02T14:42:37.434 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:42:37 smithi177 ceph-mon[36431]: scrub summary: active paths [/] 2024-04-02T14:42:37.434 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:42:37 smithi177 ceph-mon[36431]: scrub summary: idle+waiting paths [/] 2024-04-02T14:42:37.434 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:42:37 smithi177 ceph-mon[36431]: scrub summary: idle 2024-04-02T14:42:37.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.677+0000 7f8b95c00640 1 -- 172.21.15.90:0/2085498800 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8b90073ed0 msgr2=0x7f8b900742b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:37.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.677+0000 7f8b95c00640 1 --2- 172.21.15.90:0/2085498800 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8b90073ed0 0x7f8b900742b0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f8b80009900 tx=0x7f8b8002bcd0 comp rx=0 tx=0).stop 2024-04-02T14:42:37.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.678+0000 7f8b95c00640 1 -- 172.21.15.90:0/2085498800 shutdown_connections 2024-04-02T14:42:37.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.678+0000 7f8b95c00640 1 --2- 172.21.15.90:0/2085498800 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8b901059a0 0x7f8b90114b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:37.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.678+0000 7f8b95c00640 1 --2- 172.21.15.90:0/2085498800 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8b900747f0 0x7f8b901125e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:37.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.678+0000 7f8b95c00640 1 --2- 172.21.15.90:0/2085498800 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8b90073ed0 0x7f8b900742b0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:37.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.678+0000 7f8b95c00640 1 -- 172.21.15.90:0/2085498800 >> 172.21.15.90:0/2085498800 conn(0x7f8b90078f00 msgr2=0x7f8b90079310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:42:37.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.678+0000 7f8b95c00640 1 -- 172.21.15.90:0/2085498800 shutdown_connections 2024-04-02T14:42:37.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.679+0000 7f8b95c00640 1 -- 172.21.15.90:0/2085498800 wait complete. 2024-04-02T14:42:37.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.680+0000 7f8b95c00640 1 Processor -- start 2024-04-02T14:42:37.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.680+0000 7f8b95c00640 1 -- start start 2024-04-02T14:42:37.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.680+0000 7f8b95c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8b90073ed0 0x7f8b90073270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:37.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.680+0000 7f8b95c00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8b900747f0 0x7f8b900737b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:37.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.681+0000 7f8b95c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8b9006c400 0x7f8b9006c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:37.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.681+0000 7f8b95c00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f8b90118500 con 0x7f8b90073ed0 2024-04-02T14:42:37.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.681+0000 7f8b95c00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f8b90118380 con 0x7f8b900747f0 2024-04-02T14:42:37.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.681+0000 7f8b95c00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f8b90118680 con 0x7f8b9006c400 2024-04-02T14:42:37.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.681+0000 7f8b8ea00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8b90073ed0 0x7f8b90073270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:37.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.681+0000 7f8b8e000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8b900747f0 0x7f8b900737b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:37.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.681+0000 7f8b8ea00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8b90073ed0 0x7f8b90073270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:59120/0 (socket says 172.21.15.90:59120) 2024-04-02T14:42:37.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.681+0000 7f8b8ea00640 1 -- 172.21.15.90:0/2957676260 learned_addr learned my addr 172.21.15.90:0/2957676260 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:42:37.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.682+0000 7f8b8f400640 1 --2- 172.21.15.90:0/2957676260 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8b9006c400 0x7f8b9006c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:37.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.682+0000 7f8b8ea00640 1 -- 172.21.15.90:0/2957676260 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8b9006c400 msgr2=0x7f8b9006c8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:37.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.682+0000 7f8b8ea00640 1 --2- 172.21.15.90:0/2957676260 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8b9006c400 0x7f8b9006c8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:37.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.682+0000 7f8b8ea00640 1 -- 172.21.15.90:0/2957676260 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8b900747f0 msgr2=0x7f8b900737b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:37.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.682+0000 7f8b8ea00640 1 --2- 172.21.15.90:0/2957676260 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8b900747f0 0x7f8b900737b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:37.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.682+0000 7f8b8ea00640 1 -- 172.21.15.90:0/2957676260 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b90070d90 con 0x7f8b90073ed0 2024-04-02T14:42:37.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.682+0000 7f8b8f400640 1 --2- 172.21.15.90:0/2957676260 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8b9006c400 0x7f8b9006c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:42:37.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.682+0000 7f8b8e000640 1 --2- 172.21.15.90:0/2957676260 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8b900747f0 0x7f8b900737b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:42:37.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.682+0000 7f8b8ea00640 1 --2- 172.21.15.90:0/2957676260 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8b90073ed0 0x7f8b90073270 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f8b80000c00 tx=0x7f8b80004430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:42:37.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.682+0000 7f8b87400640 1 -- 172.21.15.90:0/2957676260 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b80042020 con 0x7f8b90073ed0 2024-04-02T14:42:37.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.683+0000 7f8b87400640 1 -- 172.21.15.90:0/2957676260 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8b800054e0 con 0x7f8b90073ed0 2024-04-02T14:42:37.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.683+0000 7f8b87400640 1 -- 172.21.15.90:0/2957676260 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b80003cc0 con 0x7f8b90073ed0 2024-04-02T14:42:37.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.683+0000 7f8b95c00640 1 -- 172.21.15.90:0/2957676260 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b90071020 con 0x7f8b90073ed0 2024-04-02T14:42:37.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.683+0000 7f8b95c00640 1 -- 172.21.15.90:0/2957676260 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b90071520 con 0x7f8b90073ed0 2024-04-02T14:42:37.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.686+0000 7f8b87400640 1 -- 172.21.15.90:0/2957676260 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f8b80045020 con 0x7f8b90073ed0 2024-04-02T14:42:37.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.690+0000 7f8b87400640 1 --2- 172.21.15.90:0/2957676260 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f8b740a3da0 0x7f8b740a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:37.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.690+0000 7f8b87400640 1 -- 172.21.15.90:0/2957676260 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8550+0+0 (secure 0 0 0) 0x7f8b800f5170 con 0x7f8b90073ed0 2024-04-02T14:42:37.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.691+0000 7f8b8e000640 1 --2- 172.21.15.90:0/2957676260 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f8b740a3da0 0x7f8b740a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:37.691 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.691+0000 7f8b98a0a740 1 Processor -- start 2024-04-02T14:42:37.691 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.691+0000 7f8b8e000640 1 --2- 172.21.15.90:0/2957676260 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f8b740a3da0 0x7f8b740a6260 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f8b9006dab0 tx=0x7f8b7c00b410 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:42:37.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.692+0000 7f8b98a0a740 1 -- start start 2024-04-02T14:42:37.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.692+0000 7f8b98a0a740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b547b069f0 0x55b547b05740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:37.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.692+0000 7f8b98a0a740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b547b05d10 0x55b547b06190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:37.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.692+0000 7f8b98a0a740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b547b0f080 0x55b547b11470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:37.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.692+0000 7f8b98a0a740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55b547a46400 con 0x55b547b05d10 2024-04-02T14:42:37.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.692+0000 7f8b98a0a740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55b547a2af10 con 0x55b547b069f0 2024-04-02T14:42:37.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.692+0000 7f8b98a0a740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55b547a6a6f0 con 0x55b547b0f080 2024-04-02T14:42:37.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.692+0000 7f8b8f400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b547b0f080 0x55b547b11470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:37.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.692+0000 7f8b8ea00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b547b069f0 0x55b547b05740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:37.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.692+0000 7f8b8f400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b547b0f080 0x55b547b11470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:47646/0 (socket says 172.21.15.90:47646) 2024-04-02T14:42:37.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.692+0000 7f8b8f400640 1 -- 172.21.15.90:0/2568823720 learned_addr learned my addr 172.21.15.90:0/2568823720 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:42:37.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.693+0000 7f8b8e000640 1 --2- 172.21.15.90:0/2568823720 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b547b05d10 0x55b547b06190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:37.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.693+0000 7f8b8ea00640 1 -- 172.21.15.90:0/2568823720 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b547b0f080 msgr2=0x55b547b11470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:42:37.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.693+0000 7f8b8ea00640 1 --2- 172.21.15.90:0/2568823720 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b547b0f080 0x55b547b11470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:37.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.693+0000 7f8b8ea00640 1 -- 172.21.15.90:0/2568823720 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b547b05d10 msgr2=0x55b547b06190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:42:37.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.693+0000 7f8b8ea00640 1 --2- 172.21.15.90:0/2568823720 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b547b05d10 0x55b547b06190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:37.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.693+0000 7f8b8ea00640 1 -- 172.21.15.90:0/2568823720 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b54777e3f0 con 0x55b547b069f0 2024-04-02T14:42:37.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.693+0000 7f8b8e000640 1 --2- 172.21.15.90:0/2568823720 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b547b05d10 0x55b547b06190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:42:37.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.693+0000 7f8b8ea00640 1 --2- 172.21.15.90:0/2568823720 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b547b069f0 0x55b547b05740 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f8b800eefa0 tx=0x7f8b80009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a32af8ef27d0f6a5 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:42:37.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.693+0000 7f8b7be00640 1 -- 172.21.15.90:0/2568823720 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b800f6060 con 0x55b547b069f0 2024-04-02T14:42:37.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.693+0000 7f8b7be00640 1 -- 172.21.15.90:0/2568823720 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8b80005690 con 0x55b547b069f0 2024-04-02T14:42:37.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.693+0000 7f8b7be00640 1 -- 172.21.15.90:0/2568823720 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b800c1ba0 con 0x55b547b069f0 2024-04-02T14:42:37.697 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.694+0000 7f8b98a0a740 1 -- 172.21.15.90:0/2568823720 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b547b069f0 msgr2=0x55b547b05740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:37.697 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.694+0000 7f8b98a0a740 1 --2- 172.21.15.90:0/2568823720 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b547b069f0 0x55b547b05740 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f8b800eefa0 tx=0x7f8b80009260 comp rx=0 tx=0).stop 2024-04-02T14:42:37.697 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.695+0000 7f8b98a0a740 1 -- 172.21.15.90:0/2568823720 shutdown_connections 2024-04-02T14:42:37.697 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.695+0000 7f8b98a0a740 1 --2- 172.21.15.90:0/2568823720 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b547b0f080 0x55b547b11470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:37.697 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.695+0000 7f8b98a0a740 1 --2- 172.21.15.90:0/2568823720 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b547b05d10 0x55b547b06190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:37.697 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.695+0000 7f8b98a0a740 1 --2- 172.21.15.90:0/2568823720 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b547b069f0 0x55b547b05740 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:37.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.695+0000 7f8b98a0a740 1 -- 172.21.15.90:0/2568823720 >> 172.21.15.90:0/2568823720 conn(0x55b547acca50 msgr2=0x55b547b00760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:42:37.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.695+0000 7f8b98a0a740 1 -- 172.21.15.90:0/2568823720 shutdown_connections 2024-04-02T14:42:37.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.695+0000 7f8b98a0a740 1 -- 172.21.15.90:0/2568823720 wait complete. 2024-04-02T14:42:37.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.696+0000 7f8b98a0a740 1 -- start start 2024-04-02T14:42:37.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.696+0000 7f8b98a0a740 1 Processor -- start 2024-04-02T14:42:37.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.697+0000 7f8b98a0a740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b547b05d10 0x55b547b0c010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:37.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.697+0000 7f8b98a0a740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b547b069f0 0x55b547b0c550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:37.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.697+0000 7f8b98a0a740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b547b0f080 0x55b547b0ca90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:37.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.697+0000 7f8b98a0a740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55b547a666a0 con 0x55b547b05d10 2024-04-02T14:42:37.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.697+0000 7f8b98a0a740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55b547a12980 con 0x55b547b0f080 2024-04-02T14:42:37.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.697+0000 7f8b98a0a740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55b547a25cb0 con 0x55b547b069f0 2024-04-02T14:42:37.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.697+0000 7f8b98a0a740 10 client.0 _ll_register_callbacks cb 0x55b547a256a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:42:37.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.697+0000 7f8b8f400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b547b0f080 0x55b547b0ca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:37.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.697+0000 7f8b8f400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b547b0f080 0x55b547b0ca90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:55294/0 (socket says 172.21.15.90:55294) 2024-04-02T14:42:37.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.697+0000 7f8b8f400640 1 -- 172.21.15.90:0/2746124754 learned_addr learned my addr 172.21.15.90:0/2746124754 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:42:37.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.697+0000 7f8b8ea00640 1 --2- 172.21.15.90:0/2746124754 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b547b05d10 0x55b547b0c010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:37.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.697+0000 7f8b8e000640 1 --2- 172.21.15.90:0/2746124754 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b547b069f0 0x55b547b0c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:37.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.697+0000 7f8b8f400640 1 -- 172.21.15.90:0/2746124754 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b547b069f0 msgr2=0x55b547b0c550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:42:37.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.697+0000 7f8b8f400640 1 --2- 172.21.15.90:0/2746124754 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b547b069f0 0x55b547b0c550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:37.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.697+0000 7f8b8f400640 1 -- 172.21.15.90:0/2746124754 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b547b05d10 msgr2=0x55b547b0c010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:42:37.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.697+0000 7f8b8f400640 1 --2- 172.21.15.90:0/2746124754 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b547b05d10 0x55b547b0c010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:37.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.698+0000 7f8b8f400640 1 -- 172.21.15.90:0/2746124754 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b5478487d0 con 0x55b547b0f080 2024-04-02T14:42:37.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.698+0000 7f8b8e000640 1 --2- 172.21.15.90:0/2746124754 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b547b069f0 0x55b547b0c550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:42:37.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.698+0000 7f8b8f400640 1 --2- 172.21.15.90:0/2746124754 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b547b0f080 0x55b547b0ca90 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f8b88008340 tx=0x7f8b88008f80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a170c6ec6066f701 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:42:37.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.698+0000 7f8b78c00640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:42:37.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.698+0000 7f8b78c00640 1 -- 172.21.15.90:0/2746124754 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b8801d060 con 0x55b547b0f080 2024-04-02T14:42:37.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.698+0000 7f8b78c00640 1 -- 172.21.15.90:0/2746124754 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8550+0+0 (secure 0 0 0) 0x7f8b8800ed50 con 0x55b547b0f080 2024-04-02T14:42:37.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.698+0000 7f8b98a0a740 1 -- 172.21.15.90:0/2746124754 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55b547a2a040 con 0x55b547b0f080 2024-04-02T14:42:37.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.698+0000 7f8b78c00640 1 -- 172.21.15.90:0/2746124754 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f8b88021430 con 0x55b547b0f080 2024-04-02T14:42:37.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.698+0000 7f8b98a0a740 10 client.27836 fetch_fsmap learned FSMap version 27 2024-04-02T14:42:37.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.698+0000 7f8b98a0a740 1 -- 172.21.15.90:0/2746124754 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55b547b2eb90 con 0x55b547b0f080 2024-04-02T14:42:37.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.699+0000 7f8b78c00640 1 -- 172.21.15.90:0/2746124754 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55b547a2a040 con 0x55b547b0f080 2024-04-02T14:42:37.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.699+0000 7f8b78c00640 1 -- 172.21.15.90:0/2746124754 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f8b88020590 con 0x55b547b0f080 2024-04-02T14:42:37.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.699+0000 7f8b98a0a740 10 client.27836 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:42:37.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.699+0000 7f8b98a0a740 10 client.27836 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.c 2024-04-02T14:42:37.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.699+0000 7f8b98a0a740 20 client.27836 populate_metadata read hostname 'smithi090' 2024-04-02T14:42:37.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.699+0000 7f8b98a0a740 1 --2- 172.21.15.90:0/2746124754 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x55b547b300f0 0x55b547b32530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:37.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.699+0000 7f8b98a0a740 4 client.27836 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:42:37.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.699+0000 7f8b98a0a740 1 -- 172.21.15.90:0/2746124754 --> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b547a77c30 con 0x55b547b300f0 2024-04-02T14:42:37.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.699+0000 7f8b8ea00640 1 --2- 172.21.15.90:0/2746124754 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x55b547b300f0 0x55b547b32530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:37.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.700+0000 7f8b8ea00640 1 --2- 172.21.15.90:0/2746124754 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x55b547b300f0 0x55b547b32530 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=698762f948c9bb77 server_cookie=1faa0e19ac5fc3a6 in_seq=0 out_seq=0 2024-04-02T14:42:37.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.700+0000 7f8b78c00640 10 client.27836 ms_handle_connect on v2:172.21.15.177:6836/4223414858 2024-04-02T14:42:37.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.701+0000 7f8b78c00640 1 -- 172.21.15.90:0/2746124754 <== mds.4 v2:172.21.15.177:6836/4223414858 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b547a77c30 con 0x55b547b300f0 2024-04-02T14:42:37.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.701+0000 7f8b78c00640 10 client.27836 handle_command_reply: tid=0 2024-04-02T14:42:37.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.701+0000 7f8b98a0a740 1 client.27836 shutdown 2024-04-02T14:42:37.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.701+0000 7f8b98a0a740 1 -- 172.21.15.90:0/2746124754 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b547b0f080 msgr2=0x55b547b0ca90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:37.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.701+0000 7f8b98a0a740 1 --2- 172.21.15.90:0/2746124754 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b547b0f080 0x55b547b0ca90 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f8b88008340 tx=0x7f8b88008f80 comp rx=0 tx=0).stop 2024-04-02T14:42:37.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.702+0000 7f8b98a0a740 1 -- 172.21.15.90:0/2746124754 shutdown_connections 2024-04-02T14:42:37.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.702+0000 7f8b98a0a740 1 --2- 172.21.15.90:0/2746124754 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b547b0f080 0x55b547b0ca90 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:37.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.702+0000 7f8b98a0a740 1 --2- 172.21.15.90:0/2746124754 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b547b069f0 0x55b547b0c550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:37.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.702+0000 7f8b98a0a740 1 --2- 172.21.15.90:0/2746124754 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x55b547b300f0 0x55b547b32530 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:37.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.702+0000 7f8b98a0a740 1 --2- 172.21.15.90:0/2746124754 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b547b05d10 0x55b547b0c010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:37.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.702+0000 7f8b98a0a740 1 -- 172.21.15.90:0/2746124754 >> 172.21.15.90:0/2746124754 conn(0x55b547acca50 msgr2=0x55b547b024b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:42:37.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.702+0000 7f8b78c00640 0 client.27836 ms_handle_reset on v2:172.21.15.177:6836/4223414858 2024-04-02T14:42:37.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.702+0000 7f8b98a0a740 1 -- 172.21.15.90:0/2746124754 shutdown_connections 2024-04-02T14:42:37.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.702+0000 7f8b98a0a740 1 -- 172.21.15.90:0/2746124754 wait complete. 2024-04-02T14:42:37.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.703+0000 7f8b98a0a740 20 client.27836 trim_cache size 0 max 16384 2024-04-02T14:42:37.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.723+0000 7f8b98a0a740 1 Processor -- start 2024-04-02T14:42:37.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.723+0000 7f8b98a0a740 1 -- start start 2024-04-02T14:42:37.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.723+0000 7f8b98a0a740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b547b05d10 0x55b547b0e740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:37.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.723+0000 7f8b98a0a740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b547b069f0 0x55b547b2b630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:37.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.723+0000 7f8b98a0a740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b547b0f080 0x55b547b022c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:37.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.723+0000 7f8b98a0a740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f8b90111ac0 con 0x55b547b0f080 2024-04-02T14:42:37.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.723+0000 7f8b98a0a740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f8b901b5a30 con 0x55b547b069f0 2024-04-02T14:42:37.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.723+0000 7f8b98a0a740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f8b90112190 con 0x55b547b05d10 2024-04-02T14:42:37.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.724+0000 7f8b8f400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b547b0f080 0x55b547b022c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:37.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.724+0000 7f8b8f400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b547b0f080 0x55b547b022c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:59150/0 (socket says 172.21.15.90:59150) 2024-04-02T14:42:37.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.724+0000 7f8b8f400640 1 -- 172.21.15.90:0/2532851928 learned_addr learned my addr 172.21.15.90:0/2532851928 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:42:37.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.724+0000 7f8b8f400640 1 -- 172.21.15.90:0/2532851928 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b547b05d10 msgr2=0x55b547b0e740 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-02T14:42:37.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.724+0000 7f8b8f400640 1 --2- 172.21.15.90:0/2532851928 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b547b05d10 0x55b547b0e740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:37.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.724+0000 7f8b8f400640 1 -- 172.21.15.90:0/2532851928 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b547b069f0 msgr2=0x55b547b2b630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:42:37.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.724+0000 7f8b8e000640 1 --2- 172.21.15.90:0/2532851928 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b547b069f0 0x55b547b2b630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:37.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.725+0000 7f8b8f400640 1 --2- 172.21.15.90:0/2532851928 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b547b069f0 0x55b547b2b630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:37.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.725+0000 7f8b8f400640 1 -- 172.21.15.90:0/2532851928 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b547b01900 con 0x55b547b0f080 2024-04-02T14:42:37.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.725+0000 7f8b8f400640 1 --2- 172.21.15.90:0/2532851928 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b547b0f080 0x55b547b022c0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f8b88010890 tx=0x7f8b880083d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=367d1c3cc59e1bd3 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:42:37.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.725+0000 7f8b8e000640 1 --2- 172.21.15.90:0/2532851928 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b547b069f0 0x55b547b2b630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:42:37.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.725+0000 7f8b7b400640 1 -- 172.21.15.90:0/2532851928 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b88020c20 con 0x55b547b0f080 2024-04-02T14:42:37.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.725+0000 7f8b7b400640 1 -- 172.21.15.90:0/2532851928 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8b88020dc0 con 0x55b547b0f080 2024-04-02T14:42:37.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.725+0000 7f8b7b400640 1 -- 172.21.15.90:0/2532851928 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b88021a00 con 0x55b547b0f080 2024-04-02T14:42:37.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.726+0000 7f8b98a0a740 1 -- 172.21.15.90:0/2532851928 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b547b0f080 msgr2=0x55b547b022c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:37.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.726+0000 7f8b98a0a740 1 --2- 172.21.15.90:0/2532851928 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b547b0f080 0x55b547b022c0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f8b88010890 tx=0x7f8b880083d0 comp rx=0 tx=0).stop 2024-04-02T14:42:37.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.726+0000 7f8b98a0a740 1 -- 172.21.15.90:0/2532851928 shutdown_connections 2024-04-02T14:42:37.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.726+0000 7f8b98a0a740 1 --2- 172.21.15.90:0/2532851928 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b547b0f080 0x55b547b022c0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:37.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.726+0000 7f8b98a0a740 1 --2- 172.21.15.90:0/2532851928 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b547b069f0 0x55b547b2b630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:37.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.726+0000 7f8b98a0a740 1 --2- 172.21.15.90:0/2532851928 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b547b05d10 0x55b547b0e740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:37.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.726+0000 7f8b98a0a740 1 -- 172.21.15.90:0/2532851928 >> 172.21.15.90:0/2532851928 conn(0x55b547acca50 msgr2=0x55b547b01bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:42:37.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.726+0000 7f8b98a0a740 1 -- 172.21.15.90:0/2532851928 shutdown_connections 2024-04-02T14:42:37.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.726+0000 7f8b98a0a740 1 -- 172.21.15.90:0/2532851928 wait complete. 2024-04-02T14:42:37.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.727+0000 7f8b98a0a740 1 -- start start 2024-04-02T14:42:37.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.727+0000 7f8b98a0a740 1 Processor -- start 2024-04-02T14:42:37.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.727+0000 7f8b98a0a740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b547b05d10 0x55b547b09340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:37.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.727+0000 7f8b98a0a740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b547b069f0 0x55b547b09880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:37.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.727+0000 7f8b98a0a740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b547b0f080 0x55b547b956b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:37.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.727+0000 7f8b98a0a740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f8b901b5360 con 0x55b547b0f080 2024-04-02T14:42:37.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.727+0000 7f8b98a0a740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f8b901b4c90 con 0x55b547b05d10 2024-04-02T14:42:37.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.727+0000 7f8b98a0a740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f8b9010fac0 con 0x55b547b069f0 2024-04-02T14:42:37.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.727+0000 7f8b98a0a740 10 client.0 _ll_register_callbacks cb 0x7f8b88010780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:42:37.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.727+0000 7f8b8ea00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b547b05d10 0x55b547b09340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:37.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.727+0000 7f8b8ea00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b547b05d10 0x55b547b09340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:55308/0 (socket says 172.21.15.90:55308) 2024-04-02T14:42:37.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.727+0000 7f8b8ea00640 1 -- 172.21.15.90:0/1229859345 learned_addr learned my addr 172.21.15.90:0/1229859345 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:42:37.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.727+0000 7f8b8f400640 1 --2- 172.21.15.90:0/1229859345 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b547b0f080 0x55b547b956b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:37.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.727+0000 7f8b8e000640 1 --2- 172.21.15.90:0/1229859345 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b547b069f0 0x55b547b09880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:37.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.728+0000 7f8b8ea00640 1 -- 172.21.15.90:0/1229859345 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b547b069f0 msgr2=0x55b547b09880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:42:37.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.728+0000 7f8b8ea00640 1 --2- 172.21.15.90:0/1229859345 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b547b069f0 0x55b547b09880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:37.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.728+0000 7f8b8ea00640 1 -- 172.21.15.90:0/1229859345 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b547b0f080 msgr2=0x55b547b956b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:42:37.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.728+0000 7f8b8ea00640 1 --2- 172.21.15.90:0/1229859345 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b547b0f080 0x55b547b956b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:37.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.728+0000 7f8b8ea00640 1 -- 172.21.15.90:0/1229859345 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b547b04d40 con 0x55b547b05d10 2024-04-02T14:42:37.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.728+0000 7f8b8f400640 1 --2- 172.21.15.90:0/1229859345 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b547b0f080 0x55b547b956b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:42:37.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.729+0000 7f8b8ea00640 1 --2- 172.21.15.90:0/1229859345 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b547b05d10 0x55b547b09340 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f8b80006010 tx=0x7f8b80005b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=670fc3d85bc7d665 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:42:37.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.729+0000 7f8b78c00640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:42:37.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.730+0000 7f8b78c00640 1 -- 172.21.15.90:0/1229859345 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b800da020 con 0x55b547b05d10 2024-04-02T14:42:37.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.730+0000 7f8b98a0a740 1 -- 172.21.15.90:0/1229859345 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8b88020dc0 con 0x55b547b05d10 2024-04-02T14:42:37.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.731+0000 7f8b78c00640 1 -- 172.21.15.90:0/1229859345 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8550+0+0 (secure 0 0 0) 0x7f8b800a80b0 con 0x55b547b05d10 2024-04-02T14:42:37.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.731+0000 7f8b78c00640 1 -- 172.21.15.90:0/1229859345 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f8b800da020 con 0x55b547b05d10 2024-04-02T14:42:37.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.731+0000 7f8b98a0a740 10 client.27839 fetch_fsmap learned FSMap version 27 2024-04-02T14:42:37.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.731+0000 7f8b98a0a740 1 -- 172.21.15.90:0/1229859345 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55b547b96c50 con 0x55b547b05d10 2024-04-02T14:42:37.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.732+0000 7f8b78c00640 1 -- 172.21.15.90:0/1229859345 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8b88020dc0 con 0x55b547b05d10 2024-04-02T14:42:37.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.732+0000 7f8b78c00640 1 -- 172.21.15.90:0/1229859345 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f8b800ef110 con 0x55b547b05d10 2024-04-02T14:42:37.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.732+0000 7f8b98a0a740 10 client.27839 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:42:37.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.732+0000 7f8b98a0a740 10 client.27839 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.c 2024-04-02T14:42:37.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.732+0000 7f8b98a0a740 20 client.27839 populate_metadata read hostname 'smithi090' 2024-04-02T14:42:37.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.732+0000 7f8b98a0a740 1 --2- 172.21.15.90:0/1229859345 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x55b547b97280 0x55b547b99670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:37.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.732+0000 7f8b98a0a740 4 client.27839 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-02T14:42:37.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.732+0000 7f8b98a0a740 1 -- 172.21.15.90:0/1229859345 --> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55b547a71f80 con 0x55b547b97280 2024-04-02T14:42:37.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.733+0000 7f8b8e000640 1 --2- 172.21.15.90:0/1229859345 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x55b547b97280 0x55b547b99670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:37.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.733+0000 7f8b78c00640 10 client.27839 ms_handle_connect on v2:172.21.15.177:6836/4223414858 2024-04-02T14:42:37.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.733+0000 7f8b8e000640 1 --2- 172.21.15.90:0/1229859345 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x55b547b97280 0x55b547b99670 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=e395eba700f04edb server_cookie=357e3ceb8faad8be in_seq=0 out_seq=0 2024-04-02T14:42:37.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.733+0000 7f8b78c00640 1 -- 172.21.15.90:0/1229859345 <== mds.4 v2:172.21.15.177:6836/4223414858 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55b547a71f80 con 0x55b547b97280 2024-04-02T14:42:37.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.733+0000 7f8b78c00640 10 client.27839 handle_command_reply: tid=0 2024-04-02T14:42:37.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.733+0000 7f8b98a0a740 1 client.27839 shutdown 2024-04-02T14:42:37.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.734+0000 7f8b98a0a740 1 -- 172.21.15.90:0/1229859345 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b547b05d10 msgr2=0x55b547b09340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:37.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.734+0000 7f8b98a0a740 1 --2- 172.21.15.90:0/1229859345 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b547b05d10 0x55b547b09340 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f8b80006010 tx=0x7f8b80005b50 comp rx=0 tx=0).stop 2024-04-02T14:42:37.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.734+0000 7f8b98a0a740 1 -- 172.21.15.90:0/1229859345 shutdown_connections 2024-04-02T14:42:37.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.734+0000 7f8b98a0a740 1 --2- 172.21.15.90:0/1229859345 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x55b547b97280 0x55b547b99670 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:37.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.734+0000 7f8b98a0a740 1 --2- 172.21.15.90:0/1229859345 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b547b0f080 0x55b547b956b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:37.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.734+0000 7f8b98a0a740 1 --2- 172.21.15.90:0/1229859345 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b547b069f0 0x55b547b09880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:37.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.734+0000 7f8b98a0a740 1 --2- 172.21.15.90:0/1229859345 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b547b05d10 0x55b547b09340 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:37.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.734+0000 7f8b78c00640 0 client.27839 ms_handle_reset on v2:172.21.15.177:6836/4223414858 2024-04-02T14:42:37.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.734+0000 7f8b98a0a740 1 -- 172.21.15.90:0/1229859345 >> 172.21.15.90:0/1229859345 conn(0x55b547acca50 msgr2=0x55b547b0c840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:42:37.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.734+0000 7f8b98a0a740 1 -- 172.21.15.90:0/1229859345 shutdown_connections 2024-04-02T14:42:37.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.734+0000 7f8b98a0a740 1 -- 172.21.15.90:0/1229859345 wait complete. 2024-04-02T14:42:37.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.735+0000 7f8b98a0a740 20 client.27839 trim_cache size 0 max 16384 2024-04-02T14:42:37.736 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-02T14:42:37.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.735+0000 7f8b95c00640 1 -- 172.21.15.90:0/2957676260 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f8b740a3da0 msgr2=0x7f8b740a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:37.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.735+0000 7f8b95c00640 1 --2- 172.21.15.90:0/2957676260 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f8b740a3da0 0x7f8b740a6260 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f8b9006dab0 tx=0x7f8b7c00b410 comp rx=0 tx=0).stop 2024-04-02T14:42:37.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.735+0000 7f8b95c00640 1 -- 172.21.15.90:0/2957676260 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8b90073ed0 msgr2=0x7f8b90073270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:37.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.735+0000 7f8b95c00640 1 --2- 172.21.15.90:0/2957676260 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8b90073ed0 0x7f8b90073270 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f8b80000c00 tx=0x7f8b80004430 comp rx=0 tx=0).stop 2024-04-02T14:42:37.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.735+0000 7f8b95c00640 1 -- 172.21.15.90:0/2957676260 shutdown_connections 2024-04-02T14:42:37.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.735+0000 7f8b95c00640 1 --2- 172.21.15.90:0/2957676260 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f8b740a3da0 0x7f8b740a6260 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:37.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.735+0000 7f8b95c00640 1 --2- 172.21.15.90:0/2957676260 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8b9006c400 0x7f8b9006c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:37.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.735+0000 7f8b95c00640 1 --2- 172.21.15.90:0/2957676260 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8b900747f0 0x7f8b900737b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:37.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.735+0000 7f8b95c00640 1 --2- 172.21.15.90:0/2957676260 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8b90073ed0 0x7f8b90073270 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:37.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.735+0000 7f8b95c00640 1 -- 172.21.15.90:0/2957676260 >> 172.21.15.90:0/2957676260 conn(0x7f8b90078f00 msgr2=0x7f8b90105d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:42:37.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.736+0000 7f8b95c00640 1 -- 172.21.15.90:0/2957676260 shutdown_connections 2024-04-02T14:42:37.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:37.736+0000 7f8b95c00640 1 -- 172.21.15.90:0/2957676260 wait complete. 2024-04-02T14:42:37.755 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-04-02T14:42:38.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.056+0000 7f8b16c00640 1 -- 172.21.15.90:0/2275103269 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8b100fff50 msgr2=0x7f8b10100330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:38.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.056+0000 7f8b16c00640 1 --2- 172.21.15.90:0/2275103269 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8b100fff50 0x7f8b10100330 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f8af8009900 tx=0x7f8af802bc30 comp rx=0 tx=0).stop 2024-04-02T14:42:38.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.057+0000 7f8b16c00640 1 -- 172.21.15.90:0/2275103269 shutdown_connections 2024-04-02T14:42:38.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.057+0000 7f8b16c00640 1 --2- 172.21.15.90:0/2275103269 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8b10105020 0x7f8b1010cbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:38.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.057+0000 7f8b16c00640 1 --2- 172.21.15.90:0/2275103269 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8b10100900 0x7f8b10104940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:38.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.057+0000 7f8b16c00640 1 --2- 172.21.15.90:0/2275103269 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8b100fff50 0x7f8b10100330 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:38.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.057+0000 7f8b16c00640 1 -- 172.21.15.90:0/2275103269 >> 172.21.15.90:0/2275103269 conn(0x7f8b10076b50 msgr2=0x7f8b100fdee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:42:38.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.057+0000 7f8b16c00640 1 -- 172.21.15.90:0/2275103269 shutdown_connections 2024-04-02T14:42:38.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.057+0000 7f8b16c00640 1 -- 172.21.15.90:0/2275103269 wait complete. 2024-04-02T14:42:38.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.059+0000 7f8b16c00640 1 Processor -- start 2024-04-02T14:42:38.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.059+0000 7f8b16c00640 1 -- start start 2024-04-02T14:42:38.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.059+0000 7f8b16c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8b100fff50 0x7f8b101095d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:38.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.059+0000 7f8b16c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8b10105020 0x7f8b10109b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:38.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.060+0000 7f8b16c00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8b101a4630 0x7f8b101a4ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:38.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.060+0000 7f8b16c00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f8b1010fd50 con 0x7f8b100fff50 2024-04-02T14:42:38.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.060+0000 7f8b16c00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f8b1010fbd0 con 0x7f8b101a4630 2024-04-02T14:42:38.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.060+0000 7f8b16c00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f8b1010fed0 con 0x7f8b10105020 2024-04-02T14:42:38.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.060+0000 7f8b0fe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8b100fff50 0x7f8b101095d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:38.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.060+0000 7f8b14c00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8b101a4630 0x7f8b101a4ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:38.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.060+0000 7f8b14c00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8b101a4630 0x7f8b101a4ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:55320/0 (socket says 172.21.15.90:55320) 2024-04-02T14:42:38.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.060+0000 7f8b14c00640 1 -- 172.21.15.90:0/2862386076 learned_addr learned my addr 172.21.15.90:0/2862386076 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:42:38.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.060+0000 7f8b0f400640 1 --2- 172.21.15.90:0/2862386076 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8b10105020 0x7f8b10109b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:38.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.060+0000 7f8b14c00640 1 -- 172.21.15.90:0/2862386076 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8b10105020 msgr2=0x7f8b10109b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:38.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.060+0000 7f8b14c00640 1 --2- 172.21.15.90:0/2862386076 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8b10105020 0x7f8b10109b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:38.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.060+0000 7f8b14c00640 1 -- 172.21.15.90:0/2862386076 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8b100fff50 msgr2=0x7f8b101095d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:38.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.060+0000 7f8b14c00640 1 --2- 172.21.15.90:0/2862386076 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8b100fff50 0x7f8b101095d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:38.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.060+0000 7f8b14c00640 1 -- 172.21.15.90:0/2862386076 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b101a5070 con 0x7f8b101a4630 2024-04-02T14:42:38.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.061+0000 7f8b14c00640 1 --2- 172.21.15.90:0/2862386076 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8b101a4630 0x7f8b101a4ab0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f8b08009fd0 tx=0x7f8b0800b260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:42:38.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.062+0000 7f8b0cc00640 1 -- 172.21.15.90:0/2862386076 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b08017070 con 0x7f8b101a4630 2024-04-02T14:42:38.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.062+0000 7f8b0cc00640 1 -- 172.21.15.90:0/2862386076 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8b08003e50 con 0x7f8b101a4630 2024-04-02T14:42:38.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.062+0000 7f8b0cc00640 1 -- 172.21.15.90:0/2862386076 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b08004820 con 0x7f8b101a4630 2024-04-02T14:42:38.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.062+0000 7f8b16c00640 1 -- 172.21.15.90:0/2862386076 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b101a5360 con 0x7f8b101a4630 2024-04-02T14:42:38.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.062+0000 7f8b16c00640 1 -- 172.21.15.90:0/2862386076 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b101ab920 con 0x7f8b101a4630 2024-04-02T14:42:38.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.067+0000 7f8b19832740 1 Processor -- start 2024-04-02T14:42:38.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.069+0000 7f8b0cc00640 1 -- 172.21.15.90:0/2862386076 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f8b080040b0 con 0x7f8b101a4630 2024-04-02T14:42:38.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.069+0000 7f8b19832740 1 -- start start 2024-04-02T14:42:38.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.070+0000 7f8b19832740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ba041869f0 0x55ba04185740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:38.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.070+0000 7f8b19832740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ba04185d10 0x55ba04186190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:38.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.070+0000 7f8b19832740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ba0418f080 0x55ba04191470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:38.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.070+0000 7f8b19832740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55ba040c6400 con 0x55ba041869f0 2024-04-02T14:42:38.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.070+0000 7f8b19832740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55ba040aaf10 con 0x55ba0418f080 2024-04-02T14:42:38.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.070+0000 7f8b19832740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55ba040ea6f0 con 0x55ba04185d10 2024-04-02T14:42:38.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.070+0000 7f8b0f400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ba041869f0 0x55ba04185740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:38.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.071+0000 7f8b0f400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ba041869f0 0x55ba04185740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:59186/0 (socket says 172.21.15.90:59186) 2024-04-02T14:42:38.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.071+0000 7f8b0f400640 1 -- 172.21.15.90:0/2808707338 learned_addr learned my addr 172.21.15.90:0/2808707338 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:42:38.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.071+0000 7f8b0f400640 1 --2- 172.21.15.90:0/2808707338 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ba0418f080 0x55ba04191470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:38.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.071+0000 7f8b0cc00640 1 --2- 172.21.15.90:0/2862386076 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f8af40a3ea0 0x7f8af40a6360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:38.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.071+0000 7f8b0cc00640 1 -- 172.21.15.90:0/2862386076 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8550+0+0 (secure 0 0 0) 0x7f8b080d0420 con 0x7f8b101a4630 2024-04-02T14:42:38.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.072+0000 7f8b0fe00640 1 --2- 172.21.15.90:0/2808707338 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ba04185d10 0x55ba04186190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:38.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.072+0000 7f8b0f400640 1 -- 172.21.15.90:0/2808707338 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ba04185d10 msgr2=0x55ba04186190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:42:38.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.072+0000 7f8b0f400640 1 --2- 172.21.15.90:0/2808707338 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ba04185d10 0x55ba04186190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:38.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.072+0000 7f8b0f400640 1 -- 172.21.15.90:0/2808707338 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ba0418f080 msgr2=0x55ba04191470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:42:38.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.072+0000 7f8b0f400640 1 --2- 172.21.15.90:0/2808707338 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ba0418f080 0x55ba04191470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:38.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.072+0000 7f8b0f400640 1 -- 172.21.15.90:0/2808707338 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ba03dfe3f0 con 0x55ba041869f0 2024-04-02T14:42:38.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.073+0000 7f8b14c00640 1 --2- 172.21.15.90:0/2862386076 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f8af40a3ea0 0x7f8af40a6360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:38.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.073+0000 7f8b0f400640 1 --2- 172.21.15.90:0/2808707338 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ba041869f0 0x55ba04185740 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f8b00005bc0 tx=0x7f8b000097a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79d38b7f778a5690 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:42:38.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.074+0000 7f8b05200640 1 -- 172.21.15.90:0/2808707338 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b00004030 con 0x55ba041869f0 2024-04-02T14:42:38.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.074+0000 7f8b05200640 1 -- 172.21.15.90:0/2808707338 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8b0000fca0 con 0x55ba041869f0 2024-04-02T14:42:38.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.074+0000 7f8b05200640 1 -- 172.21.15.90:0/2808707338 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b00003970 con 0x55ba041869f0 2024-04-02T14:42:38.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.075+0000 7f8b19832740 1 -- 172.21.15.90:0/2808707338 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ba041869f0 msgr2=0x55ba04185740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:38.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.075+0000 7f8b19832740 1 --2- 172.21.15.90:0/2808707338 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ba041869f0 0x55ba04185740 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f8b00005bc0 tx=0x7f8b000097a0 comp rx=0 tx=0).stop 2024-04-02T14:42:38.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.075+0000 7f8b14c00640 1 --2- 172.21.15.90:0/2862386076 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f8af40a3ea0 0x7f8af40a6360 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f8b08098800 tx=0x7f8b080092c0 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:42:38.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.075+0000 7f8b19832740 1 -- 172.21.15.90:0/2808707338 shutdown_connections 2024-04-02T14:42:38.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.075+0000 7f8b19832740 1 --2- 172.21.15.90:0/2808707338 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ba0418f080 0x55ba04191470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:38.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.075+0000 7f8b19832740 1 --2- 172.21.15.90:0/2808707338 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ba04185d10 0x55ba04186190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:38.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.075+0000 7f8b19832740 1 --2- 172.21.15.90:0/2808707338 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ba041869f0 0x55ba04185740 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:38.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.075+0000 7f8b19832740 1 -- 172.21.15.90:0/2808707338 >> 172.21.15.90:0/2808707338 conn(0x55ba0414ca50 msgr2=0x55ba04180760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:42:38.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.075+0000 7f8b19832740 1 -- 172.21.15.90:0/2808707338 shutdown_connections 2024-04-02T14:42:38.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.075+0000 7f8b19832740 1 -- 172.21.15.90:0/2808707338 wait complete. 2024-04-02T14:42:38.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.076+0000 7f8b19832740 1 -- start start 2024-04-02T14:42:38.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.077+0000 7f8b19832740 1 Processor -- start 2024-04-02T14:42:38.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.077+0000 7f8b19832740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ba04185d10 0x55ba0418c070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:38.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.077+0000 7f8b19832740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ba041869f0 0x55ba0418c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:38.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.078+0000 7f8b19832740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ba0418f080 0x55ba0418caf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:38.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.078+0000 7f8b19832740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55ba040e66a0 con 0x55ba04185d10 2024-04-02T14:42:38.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.078+0000 7f8b19832740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55ba04092980 con 0x55ba0418f080 2024-04-02T14:42:38.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.078+0000 7f8b19832740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55ba040a5cb0 con 0x55ba041869f0 2024-04-02T14:42:38.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.078+0000 7f8b19832740 10 client.0 _ll_register_callbacks cb 0x55ba040a56a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:42:38.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.078+0000 7f8b0f400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ba0418f080 0x55ba0418caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:38.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.078+0000 7f8b0f400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ba0418f080 0x55ba0418caf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:55342/0 (socket says 172.21.15.90:55342) 2024-04-02T14:42:38.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.078+0000 7f8b0f400640 1 -- 172.21.15.90:0/1604093534 learned_addr learned my addr 172.21.15.90:0/1604093534 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:42:38.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.078+0000 7f8b0f400640 1 --2- 172.21.15.90:0/1604093534 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ba04185d10 0x55ba0418c070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:38.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.078+0000 7f8b0fe00640 1 --2- 172.21.15.90:0/1604093534 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ba041869f0 0x55ba0418c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:38.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.079+0000 7f8b0f400640 1 -- 172.21.15.90:0/1604093534 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ba041869f0 msgr2=0x55ba0418c5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:42:38.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.079+0000 7f8b0f400640 1 --2- 172.21.15.90:0/1604093534 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ba041869f0 0x55ba0418c5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:38.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.079+0000 7f8b0f400640 1 -- 172.21.15.90:0/1604093534 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ba04185d10 msgr2=0x55ba0418c070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:42:38.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.079+0000 7f8b0f400640 1 --2- 172.21.15.90:0/1604093534 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ba04185d10 0x55ba0418c070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:38.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.079+0000 7f8b0f400640 1 -- 172.21.15.90:0/1604093534 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ba03ec87d0 con 0x55ba0418f080 2024-04-02T14:42:38.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.079+0000 7f8b0fe00640 1 --2- 172.21.15.90:0/1604093534 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ba041869f0 0x55ba0418c5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:42:38.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.079+0000 7f8afd600640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:42:38.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.079+0000 7f8b0f400640 1 --2- 172.21.15.90:0/1604093534 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ba0418f080 0x55ba0418caf0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f8b00009940 tx=0x7f8b00004610 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a421f7535363ac8a server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:42:38.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.080+0000 7f8afd600640 1 -- 172.21.15.90:0/1604093534 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b00028040 con 0x55ba0418f080 2024-04-02T14:42:38.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.080+0000 7f8afd600640 1 -- 172.21.15.90:0/1604093534 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8550+0+0 (secure 0 0 0) 0x7f8b0002f630 con 0x55ba0418f080 2024-04-02T14:42:38.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.080+0000 7f8b19832740 1 -- 172.21.15.90:0/1604093534 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8b0000fca0 con 0x55ba0418f080 2024-04-02T14:42:38.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.080+0000 7f8afd600640 1 -- 172.21.15.90:0/1604093534 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f8b0000fe50 con 0x55ba0418f080 2024-04-02T14:42:38.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.080+0000 7f8b19832740 10 client.27851 fetch_fsmap learned FSMap version 27 2024-04-02T14:42:38.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.080+0000 7f8b19832740 1 -- 172.21.15.90:0/1604093534 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55ba0419dec0 con 0x55ba0418f080 2024-04-02T14:42:38.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.081+0000 7f8afd600640 1 -- 172.21.15.90:0/1604093534 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8b0000fca0 con 0x55ba0418f080 2024-04-02T14:42:38.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.081+0000 7f8afd600640 1 -- 172.21.15.90:0/1604093534 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f8b0001f830 con 0x55ba0418f080 2024-04-02T14:42:38.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.081+0000 7f8b19832740 10 client.27851 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:42:38.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.081+0000 7f8b19832740 10 client.27851 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-04-02T14:42:38.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.082+0000 7f8b19832740 20 client.27851 populate_metadata read hostname 'smithi090' 2024-04-02T14:42:38.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.082+0000 7f8b19832740 1 --2- 172.21.15.90:0/1604093534 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x55ba0419f210 0x55ba041a1600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:38.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.082+0000 7f8b19832740 4 client.27851 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:42:38.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.082+0000 7f8b19832740 1 -- 172.21.15.90:0/1604093534 --> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ba040aa040 con 0x55ba0419f210 2024-04-02T14:42:38.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.082+0000 7f8b0fe00640 1 --2- 172.21.15.90:0/1604093534 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x55ba0419f210 0x55ba041a1600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:38.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.083+0000 7f8b0fe00640 1 --2- 172.21.15.90:0/1604093534 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x55ba0419f210 0x55ba041a1600 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=357d8439b0f2a0ef server_cookie=9a224573177455e3 in_seq=0 out_seq=0 2024-04-02T14:42:38.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.083+0000 7f8afd600640 10 client.27851 ms_handle_connect on v2:172.21.15.92:6836/1270635939 2024-04-02T14:42:38.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.084+0000 7f8afd600640 1 -- 172.21.15.90:0/1604093534 <== mds.3 v2:172.21.15.92:6836/1270635939 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ba040aa040 con 0x55ba0419f210 2024-04-02T14:42:38.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.084+0000 7f8afd600640 10 client.27851 handle_command_reply: tid=0 2024-04-02T14:42:38.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.084+0000 7f8b19832740 1 client.27851 shutdown 2024-04-02T14:42:38.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.085+0000 7f8b19832740 1 -- 172.21.15.90:0/1604093534 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ba0418f080 msgr2=0x55ba0418caf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:38.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.085+0000 7f8b19832740 1 --2- 172.21.15.90:0/1604093534 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ba0418f080 0x55ba0418caf0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f8b00009940 tx=0x7f8b00004610 comp rx=0 tx=0).stop 2024-04-02T14:42:38.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.085+0000 7f8b19832740 1 -- 172.21.15.90:0/1604093534 shutdown_connections 2024-04-02T14:42:38.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.085+0000 7f8b19832740 1 --2- 172.21.15.90:0/1604093534 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x55ba0419f210 0x55ba041a1600 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:38.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.086+0000 7f8b19832740 1 --2- 172.21.15.90:0/1604093534 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ba0418f080 0x55ba0418caf0 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:38.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.086+0000 7f8b19832740 1 --2- 172.21.15.90:0/1604093534 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ba041869f0 0x55ba0418c5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:38.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.086+0000 7f8b19832740 1 --2- 172.21.15.90:0/1604093534 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ba04185d10 0x55ba0418c070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:38.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.086+0000 7f8b19832740 1 -- 172.21.15.90:0/1604093534 >> 172.21.15.90:0/1604093534 conn(0x55ba0414ca50 msgr2=0x55ba041824b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:42:38.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.086+0000 7f8afd600640 0 client.27851 ms_handle_reset on v2:172.21.15.92:6836/1270635939 2024-04-02T14:42:38.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.086+0000 7f8b19832740 1 -- 172.21.15.90:0/1604093534 shutdown_connections 2024-04-02T14:42:38.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.086+0000 7f8b19832740 1 -- 172.21.15.90:0/1604093534 wait complete. 2024-04-02T14:42:38.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.086+0000 7f8b19832740 20 client.27851 trim_cache size 0 max 16384 2024-04-02T14:42:38.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.111+0000 7f8b19832740 1 Processor -- start 2024-04-02T14:42:38.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.111+0000 7f8b19832740 1 -- start start 2024-04-02T14:42:38.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.112+0000 7f8b19832740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ba04185d10 0x55ba0419aa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:38.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.112+0000 7f8b19832740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ba041869f0 0x55ba04180290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:38.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.112+0000 7f8b19832740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ba0418f080 0x55ba0418b820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:38.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.112+0000 7f8b19832740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f8b101b3fd0 con 0x55ba04185d10 2024-04-02T14:42:38.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.112+0000 7f8b19832740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f8b101b5b10 con 0x55ba0418f080 2024-04-02T14:42:38.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.112+0000 7f8b19832740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f8b101b46a0 con 0x55ba041869f0 2024-04-02T14:42:38.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.112+0000 7f8b0fe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ba041869f0 0x55ba04180290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:38.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.112+0000 7f8b0fe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ba041869f0 0x55ba04180290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:47718/0 (socket says 172.21.15.90:47718) 2024-04-02T14:42:38.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.112+0000 7f8b0fe00640 1 -- 172.21.15.90:0/2314619823 learned_addr learned my addr 172.21.15.90:0/2314619823 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:42:38.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.112+0000 7f8b0f400640 1 --2- 172.21.15.90:0/2314619823 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ba04185d10 0x55ba0419aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:38.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.113+0000 7f8b0f400640 1 --2- 172.21.15.90:0/2314619823 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ba0418f080 0x55ba0418b820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:38.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.113+0000 7f8b0fe00640 1 -- 172.21.15.90:0/2314619823 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ba0418f080 msgr2=0x55ba0418b820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:42:38.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.113+0000 7f8b0fe00640 1 --2- 172.21.15.90:0/2314619823 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ba0418f080 0x55ba0418b820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:38.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.113+0000 7f8b0fe00640 1 -- 172.21.15.90:0/2314619823 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ba04185d10 msgr2=0x55ba0419aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:42:38.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.113+0000 7f8b0fe00640 1 --2- 172.21.15.90:0/2314619823 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ba04185d10 0x55ba0419aa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:38.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.113+0000 7f8b0fe00640 1 -- 172.21.15.90:0/2314619823 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ba04180970 con 0x55ba041869f0 2024-04-02T14:42:38.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.113+0000 7f8b0f400640 1 --2- 172.21.15.90:0/2314619823 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ba0418f080 0x55ba0418b820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:42:38.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.114+0000 7f8b0fe00640 1 --2- 172.21.15.90:0/2314619823 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ba041869f0 0x55ba04180290 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f8af8009230 tx=0x7f8af80058b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=219ced4a84ca5cd0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:42:38.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.114+0000 7f8affe00640 1 -- 172.21.15.90:0/2314619823 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8af8004380 con 0x55ba041869f0 2024-04-02T14:42:38.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.114+0000 7f8affe00640 1 -- 172.21.15.90:0/2314619823 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8af8005040 con 0x55ba041869f0 2024-04-02T14:42:38.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.114+0000 7f8affe00640 1 -- 172.21.15.90:0/2314619823 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8af8004990 con 0x55ba041869f0 2024-04-02T14:42:38.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.115+0000 7f8b19832740 1 -- 172.21.15.90:0/2314619823 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ba041869f0 msgr2=0x55ba04180290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:38.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.115+0000 7f8b19832740 1 --2- 172.21.15.90:0/2314619823 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ba041869f0 0x55ba04180290 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f8af8009230 tx=0x7f8af80058b0 comp rx=0 tx=0).stop 2024-04-02T14:42:38.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.115+0000 7f8b19832740 1 -- 172.21.15.90:0/2314619823 shutdown_connections 2024-04-02T14:42:38.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.115+0000 7f8b19832740 1 --2- 172.21.15.90:0/2314619823 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ba0418f080 0x55ba0418b820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:38.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.115+0000 7f8b19832740 1 --2- 172.21.15.90:0/2314619823 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ba041869f0 0x55ba04180290 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:38.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.115+0000 7f8b19832740 1 --2- 172.21.15.90:0/2314619823 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ba04185d10 0x55ba0419aa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:38.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.115+0000 7f8b19832740 1 -- 172.21.15.90:0/2314619823 >> 172.21.15.90:0/2314619823 conn(0x55ba0414ca50 msgr2=0x55ba041823d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:42:38.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.116+0000 7f8b19832740 1 -- 172.21.15.90:0/2314619823 shutdown_connections 2024-04-02T14:42:38.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.116+0000 7f8b19832740 1 -- 172.21.15.90:0/2314619823 wait complete. 2024-04-02T14:42:38.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.116+0000 7f8b19832740 1 -- start start 2024-04-02T14:42:38.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.117+0000 7f8b19832740 1 Processor -- start 2024-04-02T14:42:38.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.118+0000 7f8b19832740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ba04185d10 0x55ba04192150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:38.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.118+0000 7f8b19832740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ba041869f0 0x55ba04192690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:38.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.118+0000 7f8b19832740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ba0418f080 0x55ba04203580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:38.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.118+0000 7f8b19832740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f8b101b5440 con 0x55ba04185d10 2024-04-02T14:42:38.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.118+0000 7f8b19832740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f8b101b4d70 con 0x55ba0418f080 2024-04-02T14:42:38.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.118+0000 7f8b19832740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f8b101b1fd0 con 0x55ba041869f0 2024-04-02T14:42:38.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.118+0000 7f8b19832740 10 client.0 _ll_register_callbacks cb 0x7f8b0000faa0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:42:38.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.119+0000 7f8b0f400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ba04185d10 0x55ba04192150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:38.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.119+0000 7f8b0fe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ba041869f0 0x55ba04192690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:38.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.119+0000 7f8b0f400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ba04185d10 0x55ba04192150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:59200/0 (socket says 172.21.15.90:59200) 2024-04-02T14:42:38.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.119+0000 7f8b0f400640 1 -- 172.21.15.90:0/1204464652 learned_addr learned my addr 172.21.15.90:0/1204464652 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:42:38.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.119+0000 7f8b0f400640 1 --2- 172.21.15.90:0/1204464652 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ba0418f080 0x55ba04203580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:38.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.119+0000 7f8b0fe00640 1 -- 172.21.15.90:0/1204464652 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ba0418f080 msgr2=0x55ba04203580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:42:38.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.119+0000 7f8b0fe00640 1 --2- 172.21.15.90:0/1204464652 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ba0418f080 0x55ba04203580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:38.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.119+0000 7f8b0fe00640 1 -- 172.21.15.90:0/1204464652 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ba04185d10 msgr2=0x55ba04192150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:42:38.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.119+0000 7f8b0fe00640 1 --2- 172.21.15.90:0/1204464652 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ba04185d10 0x55ba04192150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:38.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.119+0000 7f8b0fe00640 1 -- 172.21.15.90:0/1204464652 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ba04203af0 con 0x55ba041869f0 2024-04-02T14:42:38.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.120+0000 7f8b0fe00640 1 --2- 172.21.15.90:0/1204464652 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ba041869f0 0x55ba04192690 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x55ba040c6420 tx=0x7f8af80042e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3dccceb1aaf99825 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:42:38.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.120+0000 7f8afd600640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:42:38.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.120+0000 7f8afd600640 1 -- 172.21.15.90:0/1204464652 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8af8054070 con 0x55ba041869f0 2024-04-02T14:42:38.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.120+0000 7f8b19832740 1 -- 172.21.15.90:0/1204464652 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8af8005040 con 0x55ba041869f0 2024-04-02T14:42:38.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.121+0000 7f8afd600640 1 -- 172.21.15.90:0/1204464652 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8550+0+0 (secure 0 0 0) 0x7f8af802ca80 con 0x55ba041869f0 2024-04-02T14:42:38.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.121+0000 7f8afd600640 1 -- 172.21.15.90:0/1204464652 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f8af802cc80 con 0x55ba041869f0 2024-04-02T14:42:38.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.122+0000 7f8b19832740 10 client.27697 fetch_fsmap learned FSMap version 27 2024-04-02T14:42:38.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.122+0000 7f8b19832740 1 -- 172.21.15.90:0/1204464652 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55ba04204db0 con 0x55ba041869f0 2024-04-02T14:42:38.120 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.122+0000 7f8afd600640 1 -- 172.21.15.90:0/1204464652 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8af8005040 con 0x55ba041869f0 2024-04-02T14:42:38.120 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.122+0000 7f8afd600640 1 -- 172.21.15.90:0/1204464652 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f8af8004cb0 con 0x55ba041869f0 2024-04-02T14:42:38.120 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.122+0000 7f8b19832740 10 client.27697 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:42:38.120 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.123+0000 7f8b19832740 10 client.27697 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-04-02T14:42:38.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.123+0000 7f8b19832740 20 client.27697 populate_metadata read hostname 'smithi090' 2024-04-02T14:42:38.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.123+0000 7f8b19832740 1 --2- 172.21.15.90:0/1204464652 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x55ba04205430 0x55ba04207850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:38.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.123+0000 7f8b19832740 4 client.27697 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-02T14:42:38.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.123+0000 7f8b19832740 1 -- 172.21.15.90:0/1204464652 --> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55ba040f7c30 con 0x55ba04205430 2024-04-02T14:42:38.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.123+0000 7f8b0f400640 1 --2- 172.21.15.90:0/1204464652 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x55ba04205430 0x55ba04207850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:38.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.124+0000 7f8b0f400640 1 --2- 172.21.15.90:0/1204464652 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x55ba04205430 0x55ba04207850 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=ef5dadea21170d0 server_cookie=f9e4e0a75b227962 in_seq=0 out_seq=0 2024-04-02T14:42:38.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.124+0000 7f8afd600640 10 client.27697 ms_handle_connect on v2:172.21.15.92:6836/1270635939 2024-04-02T14:42:38.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.124+0000 7f8afd600640 1 -- 172.21.15.90:0/1204464652 <== mds.3 v2:172.21.15.92:6836/1270635939 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55ba040f7c30 con 0x55ba04205430 2024-04-02T14:42:38.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.124+0000 7f8afd600640 10 client.27697 handle_command_reply: tid=0 2024-04-02T14:42:38.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.124+0000 7f8b19832740 1 client.27697 shutdown 2024-04-02T14:42:38.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.124+0000 7f8b19832740 1 -- 172.21.15.90:0/1204464652 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ba041869f0 msgr2=0x55ba04192690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:38.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.124+0000 7f8b19832740 1 --2- 172.21.15.90:0/1204464652 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ba041869f0 0x55ba04192690 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x55ba040c6420 tx=0x7f8af80042e0 comp rx=0 tx=0).stop 2024-04-02T14:42:38.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.125+0000 7f8b19832740 1 -- 172.21.15.90:0/1204464652 shutdown_connections 2024-04-02T14:42:38.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.125+0000 7f8b19832740 1 --2- 172.21.15.90:0/1204464652 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x55ba04205430 0x55ba04207850 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:38.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.125+0000 7f8b19832740 1 --2- 172.21.15.90:0/1204464652 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ba0418f080 0x55ba04203580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:38.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.125+0000 7f8b19832740 1 --2- 172.21.15.90:0/1204464652 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ba041869f0 0x55ba04192690 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:38.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.125+0000 7f8b19832740 1 --2- 172.21.15.90:0/1204464652 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ba04185d10 0x55ba04192150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:38.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.125+0000 7f8b19832740 1 -- 172.21.15.90:0/1204464652 >> 172.21.15.90:0/1204464652 conn(0x55ba0414ca50 msgr2=0x55ba0417f010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:42:38.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.125+0000 7f8afd600640 0 client.27697 ms_handle_reset on v2:172.21.15.92:6836/1270635939 2024-04-02T14:42:38.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.125+0000 7f8b19832740 1 -- 172.21.15.90:0/1204464652 shutdown_connections 2024-04-02T14:42:38.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.125+0000 7f8b19832740 1 -- 172.21.15.90:0/1204464652 wait complete. 2024-04-02T14:42:38.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.125+0000 7f8b19832740 20 client.27697 trim_cache size 0 max 16384 2024-04-02T14:42:38.124 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-02T14:42:38.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.126+0000 7f8b16c00640 1 -- 172.21.15.90:0/2862386076 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f8af40a3ea0 msgr2=0x7f8af40a6360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:38.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.126+0000 7f8b16c00640 1 --2- 172.21.15.90:0/2862386076 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f8af40a3ea0 0x7f8af40a6360 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f8b08098800 tx=0x7f8b080092c0 comp rx=0 tx=0).stop 2024-04-02T14:42:38.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.126+0000 7f8b16c00640 1 -- 172.21.15.90:0/2862386076 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8b101a4630 msgr2=0x7f8b101a4ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:38.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.126+0000 7f8b16c00640 1 --2- 172.21.15.90:0/2862386076 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8b101a4630 0x7f8b101a4ab0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f8b08009fd0 tx=0x7f8b0800b260 comp rx=0 tx=0).stop 2024-04-02T14:42:38.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.126+0000 7f8b16c00640 1 -- 172.21.15.90:0/2862386076 shutdown_connections 2024-04-02T14:42:38.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.127+0000 7f8b16c00640 1 --2- 172.21.15.90:0/2862386076 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f8af40a3ea0 0x7f8af40a6360 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:38.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.127+0000 7f8b16c00640 1 --2- 172.21.15.90:0/2862386076 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8b101a4630 0x7f8b101a4ab0 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:38.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.127+0000 7f8b16c00640 1 --2- 172.21.15.90:0/2862386076 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8b10105020 0x7f8b10109b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:38.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.127+0000 7f8b16c00640 1 --2- 172.21.15.90:0/2862386076 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8b100fff50 0x7f8b101095d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:38.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.127+0000 7f8b16c00640 1 -- 172.21.15.90:0/2862386076 >> 172.21.15.90:0/2862386076 conn(0x7f8b10076b50 msgr2=0x7f8b101032b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:42:38.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.127+0000 7f8b16c00640 1 -- 172.21.15.90:0/2862386076 shutdown_connections 2024-04-02T14:42:38.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.128+0000 7f8b16c00640 1 -- 172.21.15.90:0/2862386076 wait complete. 2024-04-02T14:42:38.150 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-02T14:42:38.357 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:42:38 smithi092 ceph-mon[36497]: pgmap v237: 97 pgs: 97 active+clean; 4.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 29 MiB/s wr, 1.31k op/s 2024-04-02T14:42:38.358 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:42:38 smithi092 ceph-mon[36497]: from='client.? 172.21.15.90:0/1220448605' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-02T14:42:38.433 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:42:38 smithi177 ceph-mon[36431]: pgmap v237: 97 pgs: 97 active+clean; 4.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 29 MiB/s wr, 1.31k op/s 2024-04-02T14:42:38.433 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:42:38 smithi177 ceph-mon[36431]: from='client.? 172.21.15.90:0/1220448605' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-02T14:42:38.435 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:42:38 smithi090 ceph-mon[31674]: pgmap v237: 97 pgs: 97 active+clean; 4.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 29 MiB/s wr, 1.31k op/s 2024-04-02T14:42:38.435 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:42:38 smithi090 ceph-mon[31674]: from='client.? 172.21.15.90:0/1220448605' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-02T14:42:38.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.462+0000 7f2903200640 1 -- 172.21.15.90:0/1026202825 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f28fc101a90 msgr2=0x7f28fc101f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:38.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.462+0000 7f2903200640 1 --2- 172.21.15.90:0/1026202825 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f28fc101a90 0x7f28fc101f10 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f28e8009900 tx=0x7f28e802bc40 comp rx=0 tx=0).stop 2024-04-02T14:42:38.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.462+0000 7f2903200640 1 -- 172.21.15.90:0/1026202825 shutdown_connections 2024-04-02T14:42:38.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.462+0000 7f2903200640 1 --2- 172.21.15.90:0/1026202825 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f28fc102450 0x7f28fc10e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:38.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.462+0000 7f2903200640 1 --2- 172.21.15.90:0/1026202825 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f28fc101a90 0x7f28fc101f10 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:38.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.462+0000 7f2903200640 1 --2- 172.21.15.90:0/1026202825 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f28fc107aa0 0x7f28fc107e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:38.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.462+0000 7f2903200640 1 -- 172.21.15.90:0/1026202825 >> 172.21.15.90:0/1026202825 conn(0x7f28fc078f00 msgr2=0x7f28fc079310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:42:38.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.462+0000 7f2903200640 1 -- 172.21.15.90:0/1026202825 shutdown_connections 2024-04-02T14:42:38.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.463+0000 7f2903200640 1 -- 172.21.15.90:0/1026202825 wait complete. 2024-04-02T14:42:38.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.464+0000 7f2903200640 1 Processor -- start 2024-04-02T14:42:38.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.464+0000 7f2903200640 1 -- start start 2024-04-02T14:42:38.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.464+0000 7f2903200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f28fc101a90 0x7f28fc1a4680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:38.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.464+0000 7f2903200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f28fc102450 0x7f28fc1a4bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:38.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.464+0000 7f2903200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f28fc107aa0 0x7f28fc10ca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:38.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.465+0000 7f2903200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f28fc111f00 con 0x7f28fc101a90 2024-04-02T14:42:38.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.465+0000 7f2903200640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f28fc111d80 con 0x7f28fc107aa0 2024-04-02T14:42:38.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.465+0000 7f2903200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f28fc112080 con 0x7f28fc102450 2024-04-02T14:42:38.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.465+0000 7f28fb400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f28fc102450 0x7f28fc1a4bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:38.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.465+0000 7f2901200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f28fc107aa0 0x7f28fc10ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:38.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.465+0000 7f28fb400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f28fc102450 0x7f28fc1a4bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:47758/0 (socket says 172.21.15.90:47758) 2024-04-02T14:42:38.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.465+0000 7f28fb400640 1 -- 172.21.15.90:0/3051854303 learned_addr learned my addr 172.21.15.90:0/3051854303 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:42:38.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.465+0000 7f28fbe00640 1 --2- 172.21.15.90:0/3051854303 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f28fc101a90 0x7f28fc1a4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:38.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.465+0000 7f2901200640 1 -- 172.21.15.90:0/3051854303 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f28fc102450 msgr2=0x7f28fc1a4bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:38.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.466+0000 7f2901200640 1 --2- 172.21.15.90:0/3051854303 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f28fc102450 0x7f28fc1a4bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:38.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.466+0000 7f2901200640 1 -- 172.21.15.90:0/3051854303 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f28fc101a90 msgr2=0x7f28fc1a4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:38.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.466+0000 7f2901200640 1 --2- 172.21.15.90:0/3051854303 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f28fc101a90 0x7f28fc1a4680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:38.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.466+0000 7f2901200640 1 -- 172.21.15.90:0/3051854303 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28fc10d220 con 0x7f28fc107aa0 2024-04-02T14:42:38.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.466+0000 7f28fb400640 1 --2- 172.21.15.90:0/3051854303 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f28fc102450 0x7f28fc1a4bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:42:38.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.466+0000 7f2901200640 1 --2- 172.21.15.90:0/3051854303 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f28fc107aa0 0x7f28fc10ca50 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f28f4009fd0 tx=0x7f28f400ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:42:38.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.466+0000 7f28f8c00640 1 -- 172.21.15.90:0/3051854303 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28f400ce20 con 0x7f28fc107aa0 2024-04-02T14:42:38.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.467+0000 7f28f8c00640 1 -- 172.21.15.90:0/3051854303 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f28f4005020 con 0x7f28fc107aa0 2024-04-02T14:42:38.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.467+0000 7f2903200640 1 -- 172.21.15.90:0/3051854303 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28fc10d510 con 0x7f28fc107aa0 2024-04-02T14:42:38.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.467+0000 7f28f8c00640 1 -- 172.21.15.90:0/3051854303 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28f4010720 con 0x7f28fc107aa0 2024-04-02T14:42:38.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.467+0000 7f2903200640 1 -- 172.21.15.90:0/3051854303 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28fc1ab4d0 con 0x7f28fc107aa0 2024-04-02T14:42:38.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.472+0000 7f28f8c00640 1 -- 172.21.15.90:0/3051854303 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f28f4004b20 con 0x7f28fc107aa0 2024-04-02T14:42:38.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.472+0000 7f2905fc6740 1 Processor -- start 2024-04-02T14:42:38.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.473+0000 7f2905fc6740 1 -- start start 2024-04-02T14:42:38.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.473+0000 7f28f8c00640 1 --2- 172.21.15.90:0/3051854303 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f28e00a3e20 0x7f28e00a62e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:38.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.473+0000 7f2905fc6740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d5b4a149f0 0x55d5b4a13740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:38.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.473+0000 7f2905fc6740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d5b4a13d10 0x55d5b4a14190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:38.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.473+0000 7f2905fc6740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d5b4a1d080 0x55d5b4a1f470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:38.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.474+0000 7f2905fc6740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55d5b4954400 con 0x55d5b4a13d10 2024-04-02T14:42:38.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.474+0000 7f2905fc6740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55d5b4938f10 con 0x55d5b4a149f0 2024-04-02T14:42:38.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.474+0000 7f2905fc6740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55d5b49786f0 con 0x55d5b4a1d080 2024-04-02T14:42:38.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.474+0000 7f28f8c00640 1 -- 172.21.15.90:0/3051854303 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8550+0+0 (secure 0 0 0) 0x7f28f4043060 con 0x7f28fc107aa0 2024-04-02T14:42:38.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.474+0000 7f28fb400640 1 --2- 172.21.15.90:0/3051854303 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f28e00a3e20 0x7f28e00a62e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:38.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.474+0000 7f28fb400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d5b4a13d10 0x55d5b4a14190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:38.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.474+0000 7f28fb400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d5b4a13d10 0x55d5b4a14190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:59234/0 (socket says 172.21.15.90:59234) 2024-04-02T14:42:38.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.474+0000 7f28fb400640 1 -- 172.21.15.90:0/1323929916 learned_addr learned my addr 172.21.15.90:0/1323929916 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:42:38.473 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.475+0000 7f2901200640 1 --2- 172.21.15.90:0/1323929916 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d5b4a1d080 0x55d5b4a1f470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:38.473 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.475+0000 7f28fbe00640 1 --2- 172.21.15.90:0/1323929916 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d5b4a149f0 0x55d5b4a13740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:38.473 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.475+0000 7f28fb400640 1 -- 172.21.15.90:0/1323929916 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d5b4a1d080 msgr2=0x55d5b4a1f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:42:38.473 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.475+0000 7f28fb400640 1 --2- 172.21.15.90:0/1323929916 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d5b4a1d080 0x55d5b4a1f470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:38.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.475+0000 7f28fb400640 1 -- 172.21.15.90:0/1323929916 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d5b4a149f0 msgr2=0x55d5b4a13740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:42:38.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.475+0000 7f28fb400640 1 --2- 172.21.15.90:0/1323929916 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d5b4a149f0 0x55d5b4a13740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:38.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.475+0000 7f28fb400640 1 -- 172.21.15.90:0/1323929916 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d5b468c3f0 con 0x55d5b4a13d10 2024-04-02T14:42:38.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.475+0000 7f28fb400640 1 --2- 172.21.15.90:0/3051854303 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f28e00a3e20 0x7f28e00a62e0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f28e802c860 tx=0x7f28e802c750 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:42:38.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.475+0000 7f28fb400640 1 --2- 172.21.15.90:0/1323929916 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d5b4a13d10 0x55d5b4a14190 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f28e8005ab0 tx=0x7f28e8048000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e20238bd77b0d798 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:42:38.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.476+0000 7f28f1600640 1 -- 172.21.15.90:0/1323929916 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28e802cac0 con 0x55d5b4a13d10 2024-04-02T14:42:38.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.476+0000 7f28f1600640 1 -- 172.21.15.90:0/1323929916 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f28e8004490 con 0x55d5b4a13d10 2024-04-02T14:42:38.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.476+0000 7f28f1600640 1 -- 172.21.15.90:0/1323929916 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28e8003a90 con 0x55d5b4a13d10 2024-04-02T14:42:38.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.477+0000 7f2905fc6740 1 -- 172.21.15.90:0/1323929916 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d5b4a13d10 msgr2=0x55d5b4a14190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:38.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.477+0000 7f2905fc6740 1 --2- 172.21.15.90:0/1323929916 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d5b4a13d10 0x55d5b4a14190 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f28e8005ab0 tx=0x7f28e8048000 comp rx=0 tx=0).stop 2024-04-02T14:42:38.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.478+0000 7f2905fc6740 1 -- 172.21.15.90:0/1323929916 shutdown_connections 2024-04-02T14:42:38.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.478+0000 7f2905fc6740 1 --2- 172.21.15.90:0/1323929916 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d5b4a1d080 0x55d5b4a1f470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:38.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.478+0000 7f2905fc6740 1 --2- 172.21.15.90:0/1323929916 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d5b4a13d10 0x55d5b4a14190 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:38.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.478+0000 7f2905fc6740 1 --2- 172.21.15.90:0/1323929916 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d5b4a149f0 0x55d5b4a13740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:38.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.478+0000 7f2905fc6740 1 -- 172.21.15.90:0/1323929916 >> 172.21.15.90:0/1323929916 conn(0x55d5b49daa50 msgr2=0x55d5b4a0e760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:42:38.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.480+0000 7f2905fc6740 1 -- 172.21.15.90:0/1323929916 shutdown_connections 2024-04-02T14:42:38.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.480+0000 7f2905fc6740 1 -- 172.21.15.90:0/1323929916 wait complete. 2024-04-02T14:42:38.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.481+0000 7f2905fc6740 1 -- start start 2024-04-02T14:42:38.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.482+0000 7f2905fc6740 1 Processor -- start 2024-04-02T14:42:38.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.483+0000 7f2905fc6740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d5b4a13d10 0x55d5b4a1a000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:38.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.484+0000 7f2905fc6740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d5b4a149f0 0x55d5b4a1a540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:38.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.484+0000 7f2905fc6740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d5b4a1d080 0x55d5b4a1aa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:38.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.484+0000 7f28fbe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d5b4a13d10 0x55d5b4a1a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:38.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.484+0000 7f28fb400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d5b4a149f0 0x55d5b4a1a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:38.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.484+0000 7f28fbe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d5b4a13d10 0x55d5b4a1a000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:47786/0 (socket says 172.21.15.90:47786) 2024-04-02T14:42:38.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.484+0000 7f28fbe00640 1 -- 172.21.15.90:0/401085651 learned_addr learned my addr 172.21.15.90:0/401085651 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:42:38.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.484+0000 7f2905fc6740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55d5b49746a0 con 0x55d5b4a1d080 2024-04-02T14:42:38.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.485+0000 7f2905fc6740 1 -- 172.21.15.90:0/401085651 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55d5b4920980 con 0x55d5b4a149f0 2024-04-02T14:42:38.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.485+0000 7f2905fc6740 1 -- 172.21.15.90:0/401085651 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55d5b4933cb0 con 0x55d5b4a13d10 2024-04-02T14:42:38.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.485+0000 7f2905fc6740 10 client.0 _ll_register_callbacks cb 0x55d5b49336a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:42:38.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.485+0000 7f28fb400640 1 -- 172.21.15.90:0/401085651 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d5b4a13d10 msgr2=0x55d5b4a1a000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:42:38.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.485+0000 7f28fb400640 1 --2- 172.21.15.90:0/401085651 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d5b4a13d10 0x55d5b4a1a000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:38.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.485+0000 7f28fb400640 1 -- 172.21.15.90:0/401085651 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d5b4a1d080 msgr2=0x55d5b4a1aa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:42:38.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.485+0000 7f2901200640 1 --2- 172.21.15.90:0/401085651 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d5b4a1d080 0x55d5b4a1aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:38.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.485+0000 7f28fb400640 1 --2- 172.21.15.90:0/401085651 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d5b4a1d080 0x55d5b4a1aa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:38.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.485+0000 7f28fb400640 1 -- 172.21.15.90:0/401085651 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d5b47567d0 con 0x55d5b4a149f0 2024-04-02T14:42:38.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.486+0000 7f2901200640 1 --2- 172.21.15.90:0/401085651 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d5b4a1d080 0x55d5b4a1aa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:42:38.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.486+0000 7f28fb400640 1 --2- 172.21.15.90:0/401085651 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d5b4a149f0 0x55d5b4a1a540 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f28e8000c00 tx=0x7f28e80035f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=889d8f389dd7a782 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:42:38.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.486+0000 7f28e6000640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:42:38.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.487+0000 7f28e6000640 1 -- 172.21.15.90:0/401085651 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28e8059070 con 0x55d5b4a149f0 2024-04-02T14:42:38.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.487+0000 7f2905fc6740 1 -- 172.21.15.90:0/401085651 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f28e8004490 con 0x55d5b4a149f0 2024-04-02T14:42:38.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.487+0000 7f28e6000640 1 -- 172.21.15.90:0/401085651 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8550+0+0 (secure 0 0 0) 0x7f28e8004ee0 con 0x55d5b4a149f0 2024-04-02T14:42:38.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.488+0000 7f28e6000640 1 -- 172.21.15.90:0/401085651 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f28e8063050 con 0x55d5b4a149f0 2024-04-02T14:42:38.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.488+0000 7f2905fc6740 10 client.27863 fetch_fsmap learned FSMap version 27 2024-04-02T14:42:38.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.489+0000 7f2905fc6740 1 -- 172.21.15.90:0/401085651 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55d5b4a2bf00 con 0x55d5b4a149f0 2024-04-02T14:42:38.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.489+0000 7f28e6000640 1 -- 172.21.15.90:0/401085651 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f28e8004490 con 0x55d5b4a149f0 2024-04-02T14:42:38.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.489+0000 7f28e6000640 1 -- 172.21.15.90:0/401085651 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f28e805eb10 con 0x55d5b4a149f0 2024-04-02T14:42:38.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.490+0000 7f2905fc6740 10 client.27863 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:42:38.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.490+0000 7f2905fc6740 10 client.27863 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-04-02T14:42:38.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.490+0000 7f2905fc6740 20 client.27863 populate_metadata read hostname 'smithi090' 2024-04-02T14:42:38.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.490+0000 7f2905fc6740 1 --2- 172.21.15.90:0/401085651 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x55d5b4a2d250 0x55d5b4a2f640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:38.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.490+0000 7f2905fc6740 4 client.27863 mds_command: new command op to 24547 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:42:38.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.490+0000 7f2905fc6740 1 -- 172.21.15.90:0/401085651 --> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d5b4938040 con 0x55d5b4a2d250 2024-04-02T14:42:38.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.490+0000 7f28fbe00640 1 --2- 172.21.15.90:0/401085651 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x55d5b4a2d250 0x55d5b4a2f640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:38.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.491+0000 7f28fbe00640 1 --2- 172.21.15.90:0/401085651 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x55d5b4a2d250 0x55d5b4a2f640 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=5fd38526f048db2a server_cookie=5fbf33d004ca1495 in_seq=0 out_seq=0 2024-04-02T14:42:38.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.491+0000 7f28e6000640 10 client.27863 ms_handle_connect on v2:172.21.15.177:6838/664895663 2024-04-02T14:42:38.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.492+0000 7f28e6000640 1 -- 172.21.15.90:0/401085651 <== mds.2 v2:172.21.15.177:6838/664895663 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d5b4938040 con 0x55d5b4a2d250 2024-04-02T14:42:38.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.492+0000 7f28e6000640 10 client.27863 handle_command_reply: tid=0 2024-04-02T14:42:38.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.492+0000 7f2905fc6740 1 client.27863 shutdown 2024-04-02T14:42:38.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.493+0000 7f2905fc6740 1 -- 172.21.15.90:0/401085651 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d5b4a149f0 msgr2=0x55d5b4a1a540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:38.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.493+0000 7f2905fc6740 1 --2- 172.21.15.90:0/401085651 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d5b4a149f0 0x55d5b4a1a540 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f28e8000c00 tx=0x7f28e80035f0 comp rx=0 tx=0).stop 2024-04-02T14:42:38.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.494+0000 7f2905fc6740 1 -- 172.21.15.90:0/401085651 shutdown_connections 2024-04-02T14:42:38.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.494+0000 7f2905fc6740 1 --2- 172.21.15.90:0/401085651 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x55d5b4a2d250 0x55d5b4a2f640 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:38.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.494+0000 7f2905fc6740 1 --2- 172.21.15.90:0/401085651 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d5b4a1d080 0x55d5b4a1aa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:38.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.494+0000 7f2905fc6740 1 --2- 172.21.15.90:0/401085651 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d5b4a149f0 0x55d5b4a1a540 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:38.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.494+0000 7f2905fc6740 1 --2- 172.21.15.90:0/401085651 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d5b4a13d10 0x55d5b4a1a000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:38.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.494+0000 7f2905fc6740 1 -- 172.21.15.90:0/401085651 >> 172.21.15.90:0/401085651 conn(0x55d5b49daa50 msgr2=0x55d5b4a104b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:42:38.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.494+0000 7f28e6000640 0 client.27863 ms_handle_reset on v2:172.21.15.177:6838/664895663 2024-04-02T14:42:38.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.494+0000 7f2905fc6740 1 -- 172.21.15.90:0/401085651 shutdown_connections 2024-04-02T14:42:38.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.495+0000 7f2905fc6740 1 -- 172.21.15.90:0/401085651 wait complete. 2024-04-02T14:42:38.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.495+0000 7f2905fc6740 20 client.27863 trim_cache size 0 max 16384 2024-04-02T14:42:38.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.520+0000 7f2905fc6740 1 Processor -- start 2024-04-02T14:42:38.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.521+0000 7f2905fc6740 1 -- start start 2024-04-02T14:42:38.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.522+0000 7f2905fc6740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d5b4a13d10 0x55d5b4a1c730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:38.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.522+0000 7f2905fc6740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d5b4a149f0 0x55d5b4a28760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:38.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.522+0000 7f28fbe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d5b4a13d10 0x55d5b4a1c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:38.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.522+0000 7f2905fc6740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d5b4a1d080 0x55d5b4a1e470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:38.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.522+0000 7f2905fc6740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f28fc1b3ed0 con 0x55d5b4a149f0 2024-04-02T14:42:38.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.522+0000 7f2905fc6740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f28fc1b5a10 con 0x55d5b4a1d080 2024-04-02T14:42:38.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.522+0000 7f2905fc6740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f28fc1b45a0 con 0x55d5b4a13d10 2024-04-02T14:42:38.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.522+0000 7f28fbe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d5b4a13d10 0x55d5b4a1c730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:47788/0 (socket says 172.21.15.90:47788) 2024-04-02T14:42:38.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.522+0000 7f28fbe00640 1 -- 172.21.15.90:0/2089899189 learned_addr learned my addr 172.21.15.90:0/2089899189 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:42:38.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.522+0000 7f28fb400640 1 --2- 172.21.15.90:0/2089899189 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d5b4a149f0 0x55d5b4a28760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:38.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.523+0000 7f2901200640 1 --2- 172.21.15.90:0/2089899189 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d5b4a1d080 0x55d5b4a1e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:38.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.523+0000 7f28fbe00640 1 -- 172.21.15.90:0/2089899189 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d5b4a1d080 msgr2=0x55d5b4a1e470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:42:38.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.523+0000 7f28fbe00640 1 --2- 172.21.15.90:0/2089899189 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d5b4a1d080 0x55d5b4a1e470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:38.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.523+0000 7f28fbe00640 1 -- 172.21.15.90:0/2089899189 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d5b4a149f0 msgr2=0x55d5b4a28760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:42:38.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.523+0000 7f28fbe00640 1 --2- 172.21.15.90:0/2089899189 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d5b4a149f0 0x55d5b4a28760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:38.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.523+0000 7f28fbe00640 1 -- 172.21.15.90:0/2089899189 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d5b4a12d40 con 0x55d5b4a13d10 2024-04-02T14:42:38.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.524+0000 7f28fbe00640 1 --2- 172.21.15.90:0/2089899189 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d5b4a13d10 0x55d5b4a1c730 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f28ec004810 tx=0x7f28ec00cd80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ec9658d5d2eef4ad server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:42:38.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.524+0000 7f28f0c00640 1 -- 172.21.15.90:0/2089899189 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28ec00f040 con 0x55d5b4a13d10 2024-04-02T14:42:38.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.524+0000 7f28f0c00640 1 -- 172.21.15.90:0/2089899189 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f28ec01e030 con 0x55d5b4a13d10 2024-04-02T14:42:38.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.524+0000 7f28f0c00640 1 -- 172.21.15.90:0/2089899189 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28ec0156b0 con 0x55d5b4a13d10 2024-04-02T14:42:38.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.525+0000 7f2905fc6740 1 -- 172.21.15.90:0/2089899189 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d5b4a13d10 msgr2=0x55d5b4a1c730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:38.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.526+0000 7f2905fc6740 1 --2- 172.21.15.90:0/2089899189 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d5b4a13d10 0x55d5b4a1c730 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f28ec004810 tx=0x7f28ec00cd80 comp rx=0 tx=0).stop 2024-04-02T14:42:38.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.527+0000 7f2905fc6740 1 -- 172.21.15.90:0/2089899189 shutdown_connections 2024-04-02T14:42:38.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.527+0000 7f2905fc6740 1 --2- 172.21.15.90:0/2089899189 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d5b4a1d080 0x55d5b4a1e470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:38.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.527+0000 7f2905fc6740 1 --2- 172.21.15.90:0/2089899189 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d5b4a149f0 0x55d5b4a28760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:38.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.527+0000 7f2905fc6740 1 --2- 172.21.15.90:0/2089899189 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d5b4a13d10 0x55d5b4a1c730 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:38.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.527+0000 7f2905fc6740 1 -- 172.21.15.90:0/2089899189 >> 172.21.15.90:0/2089899189 conn(0x55d5b49daa50 msgr2=0x55d5b4a28050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:42:38.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.527+0000 7f2905fc6740 1 -- 172.21.15.90:0/2089899189 shutdown_connections 2024-04-02T14:42:38.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.527+0000 7f2905fc6740 1 -- 172.21.15.90:0/2089899189 wait complete. 2024-04-02T14:42:38.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.528+0000 7f2905fc6740 1 -- start start 2024-04-02T14:42:38.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.528+0000 7f2905fc6740 1 Processor -- start 2024-04-02T14:42:38.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.530+0000 7f2905fc6740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d5b4a13d10 0x55d5b4a32290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:38.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.530+0000 7f2905fc6740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d5b4a149f0 0x55d5b4a327d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:38.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.530+0000 7f2905fc6740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d5b4a1d080 0x55d5b4a165e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:38.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.530+0000 7f2905fc6740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f28fc1b5340 con 0x55d5b4a149f0 2024-04-02T14:42:38.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.530+0000 7f2905fc6740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f28fc1b4c70 con 0x55d5b4a1d080 2024-04-02T14:42:38.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.530+0000 7f2905fc6740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f28fc1b1ed0 con 0x55d5b4a13d10 2024-04-02T14:42:38.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.530+0000 7f2905fc6740 10 client.0 _ll_register_callbacks cb 0x7f28e802cca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:42:38.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.531+0000 7f2901200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d5b4a1d080 0x55d5b4a165e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:38.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.531+0000 7f2901200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d5b4a1d080 0x55d5b4a165e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:55404/0 (socket says 172.21.15.90:55404) 2024-04-02T14:42:38.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.531+0000 7f2901200640 1 -- 172.21.15.90:0/3705276470 learned_addr learned my addr 172.21.15.90:0/3705276470 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:42:38.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.531+0000 7f28fbe00640 1 --2- 172.21.15.90:0/3705276470 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d5b4a13d10 0x55d5b4a32290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:38.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.531+0000 7f28fb400640 1 --2- 172.21.15.90:0/3705276470 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d5b4a149f0 0x55d5b4a327d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:38.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.532+0000 7f28fbe00640 1 -- 172.21.15.90:0/3705276470 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d5b4a1d080 msgr2=0x55d5b4a165e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:42:38.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.532+0000 7f28fbe00640 1 --2- 172.21.15.90:0/3705276470 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d5b4a1d080 0x55d5b4a165e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:38.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.532+0000 7f28fbe00640 1 -- 172.21.15.90:0/3705276470 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d5b4a149f0 msgr2=0x55d5b4a327d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:42:38.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.532+0000 7f28fbe00640 1 --2- 172.21.15.90:0/3705276470 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d5b4a149f0 0x55d5b4a327d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:38.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.532+0000 7f28fbe00640 1 -- 172.21.15.90:0/3705276470 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d5b4a16e10 con 0x55d5b4a13d10 2024-04-02T14:42:38.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.533+0000 7f2901200640 1 --2- 172.21.15.90:0/3705276470 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d5b4a1d080 0x55d5b4a165e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:42:38.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.533+0000 7f28fb400640 1 --2- 172.21.15.90:0/3705276470 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d5b4a149f0 0x55d5b4a327d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:42:38.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.533+0000 7f28fbe00640 1 --2- 172.21.15.90:0/3705276470 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d5b4a13d10 0x55d5b4a32290 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f28ec006cd0 tx=0x7f28ec00c2a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e812e55340d3832d server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:42:38.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.533+0000 7f28e6000640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:42:38.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.534+0000 7f28e6000640 1 -- 172.21.15.90:0/3705276470 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28ec02a020 con 0x55d5b4a13d10 2024-04-02T14:42:38.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.534+0000 7f28e6000640 1 -- 172.21.15.90:0/3705276470 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8550+0+0 (secure 0 0 0) 0x7f28ec00f070 con 0x55d5b4a13d10 2024-04-02T14:42:38.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.534+0000 7f2905fc6740 1 -- 172.21.15.90:0/3705276470 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f28ec01e030 con 0x55d5b4a13d10 2024-04-02T14:42:38.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.535+0000 7f28e6000640 1 -- 172.21.15.90:0/3705276470 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f28ec02a020 con 0x55d5b4a13d10 2024-04-02T14:42:38.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.535+0000 7f2905fc6740 10 client.27715 fetch_fsmap learned FSMap version 27 2024-04-02T14:42:38.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.535+0000 7f2905fc6740 1 -- 172.21.15.90:0/3705276470 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55d5b4a92f00 con 0x55d5b4a13d10 2024-04-02T14:42:38.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.535+0000 7f28e6000640 1 -- 172.21.15.90:0/3705276470 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f28ec01e030 con 0x55d5b4a13d10 2024-04-02T14:42:38.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.537+0000 7f28e6000640 1 -- 172.21.15.90:0/3705276470 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f28ec0236f0 con 0x55d5b4a13d10 2024-04-02T14:42:38.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.537+0000 7f2905fc6740 10 client.27715 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:42:38.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.537+0000 7f2905fc6740 10 client.27715 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-04-02T14:42:38.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.537+0000 7f2905fc6740 20 client.27715 populate_metadata read hostname 'smithi090' 2024-04-02T14:42:38.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.537+0000 7f2905fc6740 1 --2- 172.21.15.90:0/3705276470 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x55d5b4a93510 0x55d5b4a95900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:38.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.537+0000 7f2905fc6740 4 client.27715 mds_command: new command op to 24547 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-02T14:42:38.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.537+0000 7f2905fc6740 1 -- 172.21.15.90:0/3705276470 --> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55d5b4985c30 con 0x55d5b4a93510 2024-04-02T14:42:38.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.538+0000 7f28fb400640 1 --2- 172.21.15.90:0/3705276470 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x55d5b4a93510 0x55d5b4a95900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:38.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.539+0000 7f28fb400640 1 --2- 172.21.15.90:0/3705276470 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x55d5b4a93510 0x55d5b4a95900 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=b062c714886fcb8b server_cookie=d5710e5d0d0b294f in_seq=0 out_seq=0 2024-04-02T14:42:38.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.539+0000 7f28e6000640 10 client.27715 ms_handle_connect on v2:172.21.15.177:6838/664895663 2024-04-02T14:42:38.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.540+0000 7f28e6000640 1 -- 172.21.15.90:0/3705276470 <== mds.2 v2:172.21.15.177:6838/664895663 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55d5b4985c30 con 0x55d5b4a93510 2024-04-02T14:42:38.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.540+0000 7f28e6000640 10 client.27715 handle_command_reply: tid=0 2024-04-02T14:42:38.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.540+0000 7f2905fc6740 1 client.27715 shutdown 2024-04-02T14:42:38.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.542+0000 7f2905fc6740 1 -- 172.21.15.90:0/3705276470 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d5b4a13d10 msgr2=0x55d5b4a32290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:38.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.542+0000 7f2905fc6740 1 --2- 172.21.15.90:0/3705276470 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d5b4a13d10 0x55d5b4a32290 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f28ec006cd0 tx=0x7f28ec00c2a0 comp rx=0 tx=0).stop 2024-04-02T14:42:38.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.543+0000 7f2905fc6740 1 -- 172.21.15.90:0/3705276470 shutdown_connections 2024-04-02T14:42:38.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.543+0000 7f2905fc6740 1 --2- 172.21.15.90:0/3705276470 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x55d5b4a93510 0x55d5b4a95900 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:38.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.543+0000 7f2905fc6740 1 --2- 172.21.15.90:0/3705276470 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d5b4a1d080 0x55d5b4a165e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:38.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.543+0000 7f2905fc6740 1 --2- 172.21.15.90:0/3705276470 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d5b4a149f0 0x55d5b4a327d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:38.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.543+0000 7f2905fc6740 1 --2- 172.21.15.90:0/3705276470 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d5b4a13d10 0x55d5b4a32290 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:38.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.543+0000 7f2905fc6740 1 -- 172.21.15.90:0/3705276470 >> 172.21.15.90:0/3705276470 conn(0x55d5b49daa50 msgr2=0x55d5b4a0d8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:42:38.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.543+0000 7f28e6000640 0 client.27715 ms_handle_reset on v2:172.21.15.177:6838/664895663 2024-04-02T14:42:38.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.544+0000 7f2905fc6740 1 -- 172.21.15.90:0/3705276470 shutdown_connections 2024-04-02T14:42:38.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.544+0000 7f2905fc6740 1 -- 172.21.15.90:0/3705276470 wait complete. 2024-04-02T14:42:38.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.544+0000 7f2905fc6740 20 client.27715 trim_cache size 0 max 16384 2024-04-02T14:42:38.542 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-02T14:42:38.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.545+0000 7f2903200640 1 -- 172.21.15.90:0/3051854303 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f28e00a3e20 msgr2=0x7f28e00a62e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:38.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.545+0000 7f2903200640 1 --2- 172.21.15.90:0/3051854303 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f28e00a3e20 0x7f28e00a62e0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f28e802c860 tx=0x7f28e802c750 comp rx=0 tx=0).stop 2024-04-02T14:42:38.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.545+0000 7f2903200640 1 -- 172.21.15.90:0/3051854303 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f28fc107aa0 msgr2=0x7f28fc10ca50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:38.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.545+0000 7f2903200640 1 --2- 172.21.15.90:0/3051854303 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f28fc107aa0 0x7f28fc10ca50 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f28f4009fd0 tx=0x7f28f400ef10 comp rx=0 tx=0).stop 2024-04-02T14:42:38.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.545+0000 7f2903200640 1 -- 172.21.15.90:0/3051854303 shutdown_connections 2024-04-02T14:42:38.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.545+0000 7f2903200640 1 --2- 172.21.15.90:0/3051854303 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f28e00a3e20 0x7f28e00a62e0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:38.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.545+0000 7f2903200640 1 --2- 172.21.15.90:0/3051854303 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f28fc107aa0 0x7f28fc10ca50 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:38.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.545+0000 7f2903200640 1 --2- 172.21.15.90:0/3051854303 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f28fc102450 0x7f28fc1a4bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:38.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.545+0000 7f2903200640 1 --2- 172.21.15.90:0/3051854303 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f28fc101a90 0x7f28fc1a4680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:38.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.545+0000 7f2903200640 1 -- 172.21.15.90:0/3051854303 >> 172.21.15.90:0/3051854303 conn(0x7f28fc078f00 msgr2=0x7f28fc073e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:42:38.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.546+0000 7f2903200640 1 -- 172.21.15.90:0/3051854303 shutdown_connections 2024-04-02T14:42:38.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.546+0000 7f2903200640 1 -- 172.21.15.90:0/3051854303 wait complete. 2024-04-02T14:42:38.570 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-02T14:42:38.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.891+0000 7fd0a0e00640 1 -- 172.21.15.90:0/1743257225 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd09c101a90 msgr2=0x7fd09c101f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:38.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.891+0000 7fd0a0e00640 1 --2- 172.21.15.90:0/1743257225 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd09c101a90 0x7fd09c101f10 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7fd088009900 tx=0x7fd08802bc30 comp rx=0 tx=0).stop 2024-04-02T14:42:38.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.892+0000 7fd0a0e00640 1 -- 172.21.15.90:0/1743257225 shutdown_connections 2024-04-02T14:42:38.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.892+0000 7fd0a0e00640 1 --2- 172.21.15.90:0/1743257225 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd09c102450 0x7fd09c10e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:38.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.892+0000 7fd0a0e00640 1 --2- 172.21.15.90:0/1743257225 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd09c101a90 0x7fd09c101f10 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:38.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.892+0000 7fd0a0e00640 1 --2- 172.21.15.90:0/1743257225 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd09c107aa0 0x7fd09c107e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:38.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.892+0000 7fd0a0e00640 1 -- 172.21.15.90:0/1743257225 >> 172.21.15.90:0/1743257225 conn(0x7fd09c078f00 msgr2=0x7fd09c079310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:42:38.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.892+0000 7fd0a0e00640 1 -- 172.21.15.90:0/1743257225 shutdown_connections 2024-04-02T14:42:38.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.892+0000 7fd0a0e00640 1 -- 172.21.15.90:0/1743257225 wait complete. 2024-04-02T14:42:38.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.893+0000 7fd0a0e00640 1 Processor -- start 2024-04-02T14:42:38.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.894+0000 7fd0a0e00640 1 -- start start 2024-04-02T14:42:38.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.894+0000 7fd0a0e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd09c101a90 0x7fd09c19a560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:38.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.894+0000 7fd0a0e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd09c102450 0x7fd09c19aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:38.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.894+0000 7fd0a0e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd09c107aa0 0x7fd09c19ee20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:38.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.894+0000 7fd0a0e00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fd09c075510 con 0x7fd09c102450 2024-04-02T14:42:38.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.894+0000 7fd0a0e00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fd09c075390 con 0x7fd09c107aa0 2024-04-02T14:42:38.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.894+0000 7fd0a0e00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fd09c075690 con 0x7fd09c101a90 2024-04-02T14:42:38.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.895+0000 7fd09a800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd09c107aa0 0x7fd09c19ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:38.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.895+0000 7fd099e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd09c101a90 0x7fd09c19a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:38.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.895+0000 7fd099e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd09c101a90 0x7fd09c19a560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:47812/0 (socket says 172.21.15.90:47812) 2024-04-02T14:42:38.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.895+0000 7fd099e00640 1 -- 172.21.15.90:0/2432419593 learned_addr learned my addr 172.21.15.90:0/2432419593 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:42:38.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.895+0000 7fd09a800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd09c107aa0 0x7fd09c19ee20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:55414/0 (socket says 172.21.15.90:55414) 2024-04-02T14:42:38.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.895+0000 7fd09a800640 1 -- 172.21.15.90:0/2432419593 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd09c101a90 msgr2=0x7fd09c19a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:38.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.895+0000 7fd09a800640 1 --2- 172.21.15.90:0/2432419593 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd09c101a90 0x7fd09c19a560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:38.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.895+0000 7fd09a800640 1 -- 172.21.15.90:0/2432419593 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd09c102450 msgr2=0x7fd09c19aaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:38.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.895+0000 7fd09a800640 1 --2- 172.21.15.90:0/2432419593 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd09c102450 0x7fd09c19aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:38.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.895+0000 7fd09a800640 1 -- 172.21.15.90:0/2432419593 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd09c19f550 con 0x7fd09c107aa0 2024-04-02T14:42:38.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.896+0000 7fd09a800640 1 --2- 172.21.15.90:0/2432419593 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd09c107aa0 0x7fd09c19ee20 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7fd094009fd0 tx=0x7fd09400ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:42:38.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.896+0000 7fd092a00640 1 -- 172.21.15.90:0/2432419593 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd09400ce20 con 0x7fd09c107aa0 2024-04-02T14:42:38.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.896+0000 7fd092a00640 1 -- 172.21.15.90:0/2432419593 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd094005020 con 0x7fd09c107aa0 2024-04-02T14:42:38.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.896+0000 7fd092a00640 1 -- 172.21.15.90:0/2432419593 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd094010720 con 0x7fd09c107aa0 2024-04-02T14:42:38.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.896+0000 7fd0a0e00640 1 -- 172.21.15.90:0/2432419593 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd09c19f840 con 0x7fd09c107aa0 2024-04-02T14:42:38.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.896+0000 7fd0a0e00640 1 -- 172.21.15.90:0/2432419593 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd09c1a70d0 con 0x7fd09c107aa0 2024-04-02T14:42:38.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.900+0000 7fd0a3b81740 1 Processor -- start 2024-04-02T14:42:38.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.903+0000 7fd092a00640 1 -- 172.21.15.90:0/2432419593 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fd0940108c0 con 0x7fd09c107aa0 2024-04-02T14:42:38.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.904+0000 7fd0a3b81740 1 -- start start 2024-04-02T14:42:38.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.904+0000 7fd0a3b81740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5576596169f0 0x557659615740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:38.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.904+0000 7fd0a3b81740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557659615d10 0x557659616190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:38.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.904+0000 7fd0a3b81740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557659627890 0x557659629c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:38.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.904+0000 7fd0a3b81740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x557659556400 con 0x557659615d10 2024-04-02T14:42:38.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.904+0000 7fd0a3b81740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55765953af10 con 0x5576596169f0 2024-04-02T14:42:38.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.904+0000 7fd0a3b81740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55765957a6f0 con 0x557659627890 2024-04-02T14:42:38.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.905+0000 7fd099e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557659615d10 0x557659616190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:38.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.905+0000 7fd099e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557659615d10 0x557659616190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:59292/0 (socket says 172.21.15.90:59292) 2024-04-02T14:42:38.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.905+0000 7fd099e00640 1 -- 172.21.15.90:0/3096959586 learned_addr learned my addr 172.21.15.90:0/3096959586 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:42:38.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.906+0000 7fd099400640 1 --2- 172.21.15.90:0/3096959586 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5576596169f0 0x557659615740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:38.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.906+0000 7fd092a00640 1 --2- 172.21.15.90:0/2432419593 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fd0800a3de0 0x7fd0800a62a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:38.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.906+0000 7fd099400640 1 --2- 172.21.15.90:0/3096959586 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557659627890 0x557659629c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:38.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.906+0000 7fd092a00640 1 -- 172.21.15.90:0/2432419593 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8550+0+0 (secure 0 0 0) 0x7fd0940d0fb0 con 0x7fd09c107aa0 2024-04-02T14:42:38.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.906+0000 7fd099400640 1 -- 172.21.15.90:0/3096959586 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5576596169f0 msgr2=0x557659615740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:42:38.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.906+0000 7fd099400640 1 --2- 172.21.15.90:0/3096959586 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5576596169f0 0x557659615740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:38.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.906+0000 7fd099400640 1 -- 172.21.15.90:0/3096959586 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557659615d10 msgr2=0x557659616190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:42:38.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.906+0000 7fd099400640 1 --2- 172.21.15.90:0/3096959586 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557659615d10 0x557659616190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:38.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.906+0000 7fd099400640 1 -- 172.21.15.90:0/3096959586 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55765928e3f0 con 0x557659627890 2024-04-02T14:42:38.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.906+0000 7fd09a800640 1 --2- 172.21.15.90:0/2432419593 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fd0800a3de0 0x7fd0800a62a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:38.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.907+0000 7fd099400640 1 --2- 172.21.15.90:0/3096959586 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557659627890 0x557659629c80 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fd088036040 tx=0x7fd088005fd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=41cb7885237e2f88 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:42:38.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.907+0000 7fd087400640 1 -- 172.21.15.90:0/3096959586 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0880043d0 con 0x557659627890 2024-04-02T14:42:38.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.907+0000 7fd087400640 1 -- 172.21.15.90:0/3096959586 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd088047070 con 0x557659627890 2024-04-02T14:42:38.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.907+0000 7fd087400640 1 -- 172.21.15.90:0/3096959586 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd088003b80 con 0x557659627890 2024-04-02T14:42:38.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.908+0000 7fd0a3b81740 1 -- 172.21.15.90:0/3096959586 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557659627890 msgr2=0x557659629c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:38.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.908+0000 7fd0a3b81740 1 --2- 172.21.15.90:0/3096959586 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557659627890 0x557659629c80 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fd088036040 tx=0x7fd088005fd0 comp rx=0 tx=0).stop 2024-04-02T14:42:38.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.908+0000 7fd09a800640 1 --2- 172.21.15.90:0/2432419593 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fd0800a3de0 0x7fd0800a62a0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fd0940a5610 tx=0x7fd094005b20 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:42:38.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.908+0000 7fd0a3b81740 1 -- 172.21.15.90:0/3096959586 shutdown_connections 2024-04-02T14:42:38.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.908+0000 7fd0a3b81740 1 --2- 172.21.15.90:0/3096959586 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557659627890 0x557659629c80 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:38.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.908+0000 7fd0a3b81740 1 --2- 172.21.15.90:0/3096959586 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557659615d10 0x557659616190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:38.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.908+0000 7fd0a3b81740 1 --2- 172.21.15.90:0/3096959586 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5576596169f0 0x557659615740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:38.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.908+0000 7fd0a3b81740 1 -- 172.21.15.90:0/3096959586 >> 172.21.15.90:0/3096959586 conn(0x5576595dca50 msgr2=0x557659610760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:42:38.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.908+0000 7fd0a3b81740 1 -- 172.21.15.90:0/3096959586 shutdown_connections 2024-04-02T14:42:38.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.908+0000 7fd0a3b81740 1 -- 172.21.15.90:0/3096959586 wait complete. 2024-04-02T14:42:38.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.909+0000 7fd0a3b81740 1 -- start start 2024-04-02T14:42:38.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.909+0000 7fd0a3b81740 1 Processor -- start 2024-04-02T14:42:38.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.910+0000 7fd0a3b81740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557659615d10 0x5576596248a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:38.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.910+0000 7fd0a3b81740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5576596169f0 0x557659624de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:38.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.910+0000 7fd0a3b81740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557659627890 0x557659625320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:38.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.910+0000 7fd0a3b81740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5576595766a0 con 0x5576596169f0 2024-04-02T14:42:38.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.910+0000 7fd0a3b81740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x557659522980 con 0x557659627890 2024-04-02T14:42:38.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.910+0000 7fd0a3b81740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x557659535cb0 con 0x557659615d10 2024-04-02T14:42:38.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.910+0000 7fd0a3b81740 10 client.0 _ll_register_callbacks cb 0x5576595356a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:42:38.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.910+0000 7fd099400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557659615d10 0x5576596248a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:38.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.910+0000 7fd099400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557659615d10 0x5576596248a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:47836/0 (socket says 172.21.15.90:47836) 2024-04-02T14:42:38.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.910+0000 7fd099400640 1 -- 172.21.15.90:0/1106575689 learned_addr learned my addr 172.21.15.90:0/1106575689 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:42:38.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.910+0000 7fd099400640 1 --2- 172.21.15.90:0/1106575689 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557659627890 0x557659625320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:38.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.910+0000 7fd099e00640 1 --2- 172.21.15.90:0/1106575689 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5576596169f0 0x557659624de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:38.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.911+0000 7fd099400640 1 -- 172.21.15.90:0/1106575689 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557659627890 msgr2=0x557659625320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:42:38.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.911+0000 7fd099400640 1 --2- 172.21.15.90:0/1106575689 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557659627890 0x557659625320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:38.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.911+0000 7fd099400640 1 -- 172.21.15.90:0/1106575689 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5576596169f0 msgr2=0x557659624de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:42:38.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.911+0000 7fd099400640 1 --2- 172.21.15.90:0/1106575689 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5576596169f0 0x557659624de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:38.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.911+0000 7fd099400640 1 -- 172.21.15.90:0/1106575689 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5576593587d0 con 0x557659615d10 2024-04-02T14:42:38.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.911+0000 7fd099e00640 1 --2- 172.21.15.90:0/1106575689 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5576596169f0 0x557659624de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:42:38.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.911+0000 7fd099400640 1 --2- 172.21.15.90:0/1106575689 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557659615d10 0x5576596248a0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fd0880098d0 tx=0x7fd08802c330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=23809e1d141839c server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:42:38.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.911+0000 7fd07be00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:42:38.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.911+0000 7fd07be00640 1 -- 172.21.15.90:0/1106575689 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd088003b00 con 0x557659615d10 2024-04-02T14:42:38.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.911+0000 7fd07be00640 1 -- 172.21.15.90:0/1106575689 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8550+0+0 (secure 0 0 0) 0x7fd088003ca0 con 0x557659615d10 2024-04-02T14:42:38.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.911+0000 7fd0a3b81740 1 -- 172.21.15.90:0/1106575689 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd088047070 con 0x557659615d10 2024-04-02T14:42:38.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.912+0000 7fd07be00640 1 -- 172.21.15.90:0/1106575689 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fd088060050 con 0x557659615d10 2024-04-02T14:42:38.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.912+0000 7fd0a3b81740 10 client.27727 fetch_fsmap learned FSMap version 27 2024-04-02T14:42:38.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.912+0000 7fd0a3b81740 1 -- 172.21.15.90:0/1106575689 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5576596366e0 con 0x557659615d10 2024-04-02T14:42:38.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.912+0000 7fd07be00640 1 -- 172.21.15.90:0/1106575689 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd088047070 con 0x557659615d10 2024-04-02T14:42:38.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.912+0000 7fd07be00640 1 -- 172.21.15.90:0/1106575689 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fd088034440 con 0x557659615d10 2024-04-02T14:42:38.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.912+0000 7fd0a3b81740 10 client.27727 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:42:38.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.912+0000 7fd0a3b81740 10 client.27727 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2024-04-02T14:42:38.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.912+0000 7fd0a3b81740 20 client.27727 populate_metadata read hostname 'smithi090' 2024-04-02T14:42:38.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.912+0000 7fd0a3b81740 1 --2- 172.21.15.90:0/1106575689 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x557659637a30 0x557659639e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:38.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.912+0000 7fd0a3b81740 4 client.27727 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:42:38.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.912+0000 7fd0a3b81740 1 -- 172.21.15.90:0/1106575689 --> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55765953a040 con 0x557659637a30 2024-04-02T14:42:38.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.913+0000 7fd099e00640 1 --2- 172.21.15.90:0/1106575689 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x557659637a30 0x557659639e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:38.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.914+0000 7fd099e00640 1 --2- 172.21.15.90:0/1106575689 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x557659637a30 0x557659639e20 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=5efb3d738513878f server_cookie=8cca640543b1d467 in_seq=0 out_seq=0 2024-04-02T14:42:38.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.914+0000 7fd07be00640 10 client.27727 ms_handle_connect on v2:172.21.15.90:6838/1498915870 2024-04-02T14:42:38.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.916+0000 7fd07be00640 1 -- 172.21.15.90:0/1106575689 <== mds.1 v2:172.21.15.90:6838/1498915870 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55765953a040 con 0x557659637a30 2024-04-02T14:42:38.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.916+0000 7fd07be00640 10 client.27727 handle_command_reply: tid=0 2024-04-02T14:42:38.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.916+0000 7fd0a3b81740 1 client.27727 shutdown 2024-04-02T14:42:38.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.916+0000 7fd0a3b81740 1 -- 172.21.15.90:0/1106575689 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557659615d10 msgr2=0x5576596248a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:38.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.916+0000 7fd0a3b81740 1 --2- 172.21.15.90:0/1106575689 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557659615d10 0x5576596248a0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fd0880098d0 tx=0x7fd08802c330 comp rx=0 tx=0).stop 2024-04-02T14:42:38.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.916+0000 7fd0a3b81740 1 -- 172.21.15.90:0/1106575689 shutdown_connections 2024-04-02T14:42:38.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.916+0000 7fd0a3b81740 1 --2- 172.21.15.90:0/1106575689 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x557659637a30 0x557659639e20 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:38.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.916+0000 7fd0a3b81740 1 --2- 172.21.15.90:0/1106575689 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557659627890 0x557659625320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:38.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.916+0000 7fd0a3b81740 1 --2- 172.21.15.90:0/1106575689 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5576596169f0 0x557659624de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:38.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.916+0000 7fd0a3b81740 1 --2- 172.21.15.90:0/1106575689 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557659615d10 0x5576596248a0 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:38.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.916+0000 7fd0a3b81740 1 -- 172.21.15.90:0/1106575689 >> 172.21.15.90:0/1106575689 conn(0x5576595dca50 msgr2=0x5576596124b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:42:38.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.916+0000 7fd07be00640 0 client.27727 ms_handle_reset on v2:172.21.15.90:6838/1498915870 2024-04-02T14:42:38.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.917+0000 7fd0a3b81740 1 -- 172.21.15.90:0/1106575689 shutdown_connections 2024-04-02T14:42:38.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.917+0000 7fd0a3b81740 1 -- 172.21.15.90:0/1106575689 wait complete. 2024-04-02T14:42:38.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.918+0000 7fd0a3b81740 20 client.27727 trim_cache size 0 max 16384 2024-04-02T14:42:38.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.940+0000 7fd0a3b81740 1 Processor -- start 2024-04-02T14:42:38.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.940+0000 7fd0a3b81740 1 -- start start 2024-04-02T14:42:38.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.941+0000 7fd0a3b81740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557659615d10 0x557659626fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:38.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.941+0000 7fd0a3b81740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5576596169f0 0x557659628c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:38.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.941+0000 7fd0a3b81740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557659627890 0x5576596291c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:38.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.941+0000 7fd0a3b81740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fd09c1ab6f0 con 0x557659627890 2024-04-02T14:42:38.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.941+0000 7fd0a3b81740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fd09c1ad230 con 0x5576596169f0 2024-04-02T14:42:38.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.941+0000 7fd0a3b81740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fd09c1abdc0 con 0x557659615d10 2024-04-02T14:42:38.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.942+0000 7fd099400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557659615d10 0x557659626fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:38.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.942+0000 7fd099400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557659615d10 0x557659626fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:47844/0 (socket says 172.21.15.90:47844) 2024-04-02T14:42:38.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.942+0000 7fd099e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5576596169f0 0x557659628c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:38.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.942+0000 7fd099e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5576596169f0 0x557659628c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:55444/0 (socket says 172.21.15.90:55444) 2024-04-02T14:42:38.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.942+0000 7fd099e00640 1 -- 172.21.15.90:0/3236679461 learned_addr learned my addr 172.21.15.90:0/3236679461 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:42:38.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.942+0000 7fd099400640 1 --2- 172.21.15.90:0/3236679461 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557659627890 0x5576596291c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:38.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.943+0000 7fd099e00640 1 -- 172.21.15.90:0/3236679461 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557659615d10 msgr2=0x557659626fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:42:38.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.943+0000 7fd099e00640 1 --2- 172.21.15.90:0/3236679461 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557659615d10 0x557659626fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:38.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.943+0000 7fd099e00640 1 -- 172.21.15.90:0/3236679461 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557659627890 msgr2=0x5576596291c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:42:38.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.943+0000 7fd099e00640 1 --2- 172.21.15.90:0/3236679461 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557659627890 0x5576596291c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:38.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.943+0000 7fd099e00640 1 -- 172.21.15.90:0/3236679461 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557659614d40 con 0x5576596169f0 2024-04-02T14:42:38.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.944+0000 7fd099e00640 1 --2- 172.21.15.90:0/3236679461 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5576596169f0 0x557659628c80 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x557659624100 tx=0x7fd08c0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=942e8bbca6f83d87 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:42:38.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.944+0000 7fd086a00640 1 -- 172.21.15.90:0/3236679461 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd08c01adc0 con 0x5576596169f0 2024-04-02T14:42:38.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.944+0000 7fd086a00640 1 -- 172.21.15.90:0/3236679461 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd08c01f070 con 0x5576596169f0 2024-04-02T14:42:38.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.944+0000 7fd086a00640 1 -- 172.21.15.90:0/3236679461 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd08c0035f0 con 0x5576596169f0 2024-04-02T14:42:38.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.945+0000 7fd0a3b81740 1 -- 172.21.15.90:0/3236679461 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5576596169f0 msgr2=0x557659628c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:38.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.945+0000 7fd0a3b81740 1 --2- 172.21.15.90:0/3236679461 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5576596169f0 0x557659628c80 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x557659624100 tx=0x7fd08c0046c0 comp rx=0 tx=0).stop 2024-04-02T14:42:38.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.945+0000 7fd0a3b81740 1 -- 172.21.15.90:0/3236679461 shutdown_connections 2024-04-02T14:42:38.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.945+0000 7fd0a3b81740 1 --2- 172.21.15.90:0/3236679461 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557659627890 0x5576596291c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:38.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.945+0000 7fd0a3b81740 1 --2- 172.21.15.90:0/3236679461 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5576596169f0 0x557659628c80 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:38.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.945+0000 7fd0a3b81740 1 --2- 172.21.15.90:0/3236679461 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557659615d10 0x557659626fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:38.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.945+0000 7fd0a3b81740 1 -- 172.21.15.90:0/3236679461 >> 172.21.15.90:0/3236679461 conn(0x5576595dca50 msgr2=0x557659610290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:42:38.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.946+0000 7fd0a3b81740 1 -- 172.21.15.90:0/3236679461 shutdown_connections 2024-04-02T14:42:38.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.946+0000 7fd0a3b81740 1 -- 172.21.15.90:0/3236679461 wait complete. 2024-04-02T14:42:38.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.946+0000 7fd0a3b81740 1 -- start start 2024-04-02T14:42:38.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.947+0000 7fd0a3b81740 1 Processor -- start 2024-04-02T14:42:38.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.947+0000 7fd0a3b81740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557659615d10 0x557659699150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:38.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.947+0000 7fd0a3b81740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5576596169f0 0x557659699690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:38.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.947+0000 7fd0a3b81740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557659627890 0x55765963cc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:38.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.947+0000 7fd0a3b81740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fd09c1acb60 con 0x557659627890 2024-04-02T14:42:38.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.947+0000 7fd0a3b81740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fd09c1ac490 con 0x5576596169f0 2024-04-02T14:42:38.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.947+0000 7fd0a3b81740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fd09c1a9710 con 0x557659615d10 2024-04-02T14:42:38.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.947+0000 7fd0a3b81740 10 client.0 _ll_register_callbacks cb 0x7fd0880035f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:42:38.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.947+0000 7fd099400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557659615d10 0x557659699150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:38.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.948+0000 7fd099400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557659615d10 0x557659699150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:47846/0 (socket says 172.21.15.90:47846) 2024-04-02T14:42:38.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.948+0000 7fd099400640 1 -- 172.21.15.90:0/987285938 learned_addr learned my addr 172.21.15.90:0/987285938 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:42:38.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.948+0000 7fd099e00640 1 --2- 172.21.15.90:0/987285938 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5576596169f0 0x557659699690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:38.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.948+0000 7fd099400640 1 -- 172.21.15.90:0/987285938 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5576596169f0 msgr2=0x557659699690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:42:38.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.948+0000 7fd099400640 1 --2- 172.21.15.90:0/987285938 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5576596169f0 0x557659699690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:38.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.949+0000 7fd099400640 1 -- 172.21.15.90:0/987285938 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557659627890 msgr2=0x55765963cc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:42:38.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.949+0000 7fd099400640 1 --2- 172.21.15.90:0/987285938 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557659627890 0x55765963cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:38.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.949+0000 7fd099400640 1 -- 172.21.15.90:0/987285938 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557659697bd0 con 0x557659615d10 2024-04-02T14:42:38.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.949+0000 7fd099400640 1 --2- 172.21.15.90:0/987285938 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557659615d10 0x557659699150 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fd0880098d0 tx=0x7fd088005a20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a7fa8d1fc4a6a319 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:42:38.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.949+0000 7fd07be00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:42:38.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.950+0000 7fd07be00640 1 -- 172.21.15.90:0/987285938 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd088034640 con 0x557659615d10 2024-04-02T14:42:38.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.950+0000 7fd07be00640 1 -- 172.21.15.90:0/987285938 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8550+0+0 (secure 0 0 0) 0x7fd088005030 con 0x557659615d10 2024-04-02T14:42:38.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.950+0000 7fd0a3b81740 1 -- 172.21.15.90:0/987285938 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd08c01f070 con 0x557659615d10 2024-04-02T14:42:38.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.950+0000 7fd07be00640 1 -- 172.21.15.90:0/987285938 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fd088009520 con 0x557659615d10 2024-04-02T14:42:38.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.951+0000 7fd0a3b81740 10 client.27733 fetch_fsmap learned FSMap version 27 2024-04-02T14:42:38.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.951+0000 7fd0a3b81740 1 -- 172.21.15.90:0/987285938 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55765961b130 con 0x557659615d10 2024-04-02T14:42:38.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.951+0000 7fd07be00640 1 -- 172.21.15.90:0/987285938 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd08c01f070 con 0x557659615d10 2024-04-02T14:42:38.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.951+0000 7fd07be00640 1 -- 172.21.15.90:0/987285938 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fd088004cc0 con 0x557659615d10 2024-04-02T14:42:38.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.951+0000 7fd0a3b81740 10 client.27733 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:42:38.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.951+0000 7fd0a3b81740 10 client.27733 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2024-04-02T14:42:38.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.952+0000 7fd0a3b81740 20 client.27733 populate_metadata read hostname 'smithi090' 2024-04-02T14:42:38.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.952+0000 7fd0a3b81740 1 --2- 172.21.15.90:0/987285938 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x55765961b880 0x55765961dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:38.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.952+0000 7fd0a3b81740 4 client.27733 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-02T14:42:38.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.952+0000 7fd0a3b81740 1 -- 172.21.15.90:0/987285938 --> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] -- command(tid 0: {"prefix": "damage ls"}) -- 0x557659587c30 con 0x55765961b880 2024-04-02T14:42:38.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.952+0000 7fd099e00640 1 --2- 172.21.15.90:0/987285938 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x55765961b880 0x55765961dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:38.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.953+0000 7fd099e00640 1 --2- 172.21.15.90:0/987285938 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x55765961b880 0x55765961dca0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=4e50cc04fbc2df60 server_cookie=af39d3b03e0d01cd in_seq=0 out_seq=0 2024-04-02T14:42:38.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.953+0000 7fd07be00640 10 client.27733 ms_handle_connect on v2:172.21.15.90:6838/1498915870 2024-04-02T14:42:38.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.953+0000 7fd07be00640 1 -- 172.21.15.90:0/987285938 <== mds.1 v2:172.21.15.90:6838/1498915870 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x557659587c30 con 0x55765961b880 2024-04-02T14:42:38.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.953+0000 7fd07be00640 10 client.27733 handle_command_reply: tid=0 2024-04-02T14:42:38.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.953+0000 7fd0a3b81740 1 client.27733 shutdown 2024-04-02T14:42:38.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.954+0000 7fd0a3b81740 1 -- 172.21.15.90:0/987285938 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557659615d10 msgr2=0x557659699150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:38.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.954+0000 7fd0a3b81740 1 --2- 172.21.15.90:0/987285938 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557659615d10 0x557659699150 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fd0880098d0 tx=0x7fd088005a20 comp rx=0 tx=0).stop 2024-04-02T14:42:38.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.954+0000 7fd0a3b81740 1 -- 172.21.15.90:0/987285938 shutdown_connections 2024-04-02T14:42:38.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.954+0000 7fd0a3b81740 1 --2- 172.21.15.90:0/987285938 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x55765961b880 0x55765961dca0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:38.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.954+0000 7fd0a3b81740 1 --2- 172.21.15.90:0/987285938 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557659627890 0x55765963cc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:38.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.954+0000 7fd0a3b81740 1 --2- 172.21.15.90:0/987285938 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5576596169f0 0x557659699690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:38.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.954+0000 7fd0a3b81740 1 --2- 172.21.15.90:0/987285938 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557659615d10 0x557659699150 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:38.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.954+0000 7fd0a3b81740 1 -- 172.21.15.90:0/987285938 >> 172.21.15.90:0/987285938 conn(0x5576595dca50 msgr2=0x5576596327b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:42:38.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.954+0000 7fd07be00640 0 client.27733 ms_handle_reset on v2:172.21.15.90:6838/1498915870 2024-04-02T14:42:38.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.954+0000 7fd0a3b81740 1 -- 172.21.15.90:0/987285938 shutdown_connections 2024-04-02T14:42:38.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.954+0000 7fd0a3b81740 1 -- 172.21.15.90:0/987285938 wait complete. 2024-04-02T14:42:38.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.954+0000 7fd0a3b81740 20 client.27733 trim_cache size 0 max 16384 2024-04-02T14:42:38.957 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-02T14:42:38.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.955+0000 7fd0a0e00640 1 -- 172.21.15.90:0/2432419593 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fd0800a3de0 msgr2=0x7fd0800a62a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:38.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.955+0000 7fd0a0e00640 1 --2- 172.21.15.90:0/2432419593 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fd0800a3de0 0x7fd0800a62a0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fd0940a5610 tx=0x7fd094005b20 comp rx=0 tx=0).stop 2024-04-02T14:42:38.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.955+0000 7fd0a0e00640 1 -- 172.21.15.90:0/2432419593 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd09c107aa0 msgr2=0x7fd09c19ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:38.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.955+0000 7fd0a0e00640 1 --2- 172.21.15.90:0/2432419593 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd09c107aa0 0x7fd09c19ee20 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7fd094009fd0 tx=0x7fd09400ef10 comp rx=0 tx=0).stop 2024-04-02T14:42:38.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.956+0000 7fd0a0e00640 1 -- 172.21.15.90:0/2432419593 shutdown_connections 2024-04-02T14:42:38.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.956+0000 7fd0a0e00640 1 --2- 172.21.15.90:0/2432419593 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fd0800a3de0 0x7fd0800a62a0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:38.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.956+0000 7fd0a0e00640 1 --2- 172.21.15.90:0/2432419593 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd09c107aa0 0x7fd09c19ee20 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:38.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.956+0000 7fd0a0e00640 1 --2- 172.21.15.90:0/2432419593 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd09c102450 0x7fd09c19aaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:38.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.956+0000 7fd0a0e00640 1 --2- 172.21.15.90:0/2432419593 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd09c101a90 0x7fd09c19a560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:38.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.956+0000 7fd0a0e00640 1 -- 172.21.15.90:0/2432419593 >> 172.21.15.90:0/2432419593 conn(0x7fd09c078f00 msgr2=0x7fd09c103e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:42:38.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.956+0000 7fd0a0e00640 1 -- 172.21.15.90:0/2432419593 shutdown_connections 2024-04-02T14:42:38.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:38.956+0000 7fd0a0e00640 1 -- 172.21.15.90:0/2432419593 wait complete. 2024-04-02T14:42:38.981 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-02T14:42:39.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.277+0000 7f2477e00640 1 -- 172.21.15.90:0/3332010962 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2470107aa0 msgr2=0x7f2470107e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:39.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.277+0000 7f2477e00640 1 --2- 172.21.15.90:0/3332010962 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2470107aa0 0x7f2470107e80 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f245c009930 tx=0x7f245c02bc30 comp rx=0 tx=0).stop 2024-04-02T14:42:39.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.277+0000 7f2477e00640 1 -- 172.21.15.90:0/3332010962 shutdown_connections 2024-04-02T14:42:39.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.277+0000 7f2477e00640 1 --2- 172.21.15.90:0/3332010962 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2470102450 0x7f247010e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:39.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.277+0000 7f2477e00640 1 --2- 172.21.15.90:0/3332010962 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2470101a90 0x7f2470101f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:39.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.277+0000 7f2477e00640 1 --2- 172.21.15.90:0/3332010962 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2470107aa0 0x7f2470107e80 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:39.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.277+0000 7f2477e00640 1 -- 172.21.15.90:0/3332010962 >> 172.21.15.90:0/3332010962 conn(0x7f2470078f00 msgr2=0x7f2470079310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:42:39.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.277+0000 7f2477e00640 1 -- 172.21.15.90:0/3332010962 shutdown_connections 2024-04-02T14:42:39.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.277+0000 7f2477e00640 1 -- 172.21.15.90:0/3332010962 wait complete. 2024-04-02T14:42:39.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.279+0000 7f2477e00640 1 Processor -- start 2024-04-02T14:42:39.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.281+0000 7f2477e00640 1 -- start start 2024-04-02T14:42:39.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.281+0000 7f2477e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2470101a90 0x7f2470198310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:39.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.281+0000 7f2477e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2470102450 0x7f2470198850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:39.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.281+0000 7f2477e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2470107aa0 0x7f247019cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:39.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.281+0000 7f2477e00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f2470111ed0 con 0x7f2470107aa0 2024-04-02T14:42:39.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.281+0000 7f2477e00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f2470111d50 con 0x7f2470101a90 2024-04-02T14:42:39.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.281+0000 7f2477e00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f2470112050 con 0x7f2470102450 2024-04-02T14:42:39.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.282+0000 7f2474a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2470102450 0x7f2470198850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:39.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.282+0000 7f2474a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2470102450 0x7f2470198850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:47868/0 (socket says 172.21.15.90:47868) 2024-04-02T14:42:39.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.282+0000 7f2474a00640 1 -- 172.21.15.90:0/451305204 learned_addr learned my addr 172.21.15.90:0/451305204 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:42:39.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.282+0000 7f2475400640 1 --2- 172.21.15.90:0/451305204 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2470101a90 0x7f2470198310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:39.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.282+0000 7f2474a00640 1 -- 172.21.15.90:0/451305204 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2470101a90 msgr2=0x7f2470198310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:39.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.282+0000 7f2474a00640 1 --2- 172.21.15.90:0/451305204 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2470101a90 0x7f2470198310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:39.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.282+0000 7f2474a00640 1 -- 172.21.15.90:0/451305204 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2470107aa0 msgr2=0x7f247019cbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:39.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.282+0000 7f2474a00640 1 --2- 172.21.15.90:0/451305204 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2470107aa0 0x7f247019cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:39.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.282+0000 7f2474a00640 1 -- 172.21.15.90:0/451305204 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f247019d300 con 0x7f2470102450 2024-04-02T14:42:39.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.282+0000 7f2475400640 1 --2- 172.21.15.90:0/451305204 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2470101a90 0x7f2470198310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:42:39.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.283+0000 7f2474a00640 1 --2- 172.21.15.90:0/451305204 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2470102450 0x7f2470198850 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f246400bea0 tx=0x7f246400bf90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:42:39.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.283+0000 7f246e000640 1 -- 172.21.15.90:0/451305204 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2464018070 con 0x7f2470102450 2024-04-02T14:42:39.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.283+0000 7f246e000640 1 -- 172.21.15.90:0/451305204 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2464012c40 con 0x7f2470102450 2024-04-02T14:42:39.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.283+0000 7f246e000640 1 -- 172.21.15.90:0/451305204 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2464018070 con 0x7f2470102450 2024-04-02T14:42:39.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.283+0000 7f2477e00640 1 -- 172.21.15.90:0/451305204 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f247019d5f0 con 0x7f2470102450 2024-04-02T14:42:39.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.283+0000 7f2477e00640 1 -- 172.21.15.90:0/451305204 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2470073d30 con 0x7f2470102450 2024-04-02T14:42:39.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.288+0000 7f247aa16740 1 Processor -- start 2024-04-02T14:42:39.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.288+0000 7f247aa16740 1 -- start start 2024-04-02T14:42:39.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.289+0000 7f247aa16740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559ce3bc99f0 0x559ce3bc8740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:39.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.289+0000 7f247aa16740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559ce3bc8d10 0x559ce3bc9190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:39.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.289+0000 7f247aa16740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559ce3bd2080 0x559ce3bd4470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:39.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.289+0000 7f247aa16740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x559ce3b09400 con 0x559ce3bd2080 2024-04-02T14:42:39.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.289+0000 7f247aa16740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x559ce3aedf10 con 0x559ce3bc8d10 2024-04-02T14:42:39.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.289+0000 7f247aa16740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x559ce3b2d6f0 con 0x559ce3bc99f0 2024-04-02T14:42:39.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.289+0000 7f2475400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559ce3bc8d10 0x559ce3bc9190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:39.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.290+0000 7f2475400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559ce3bc8d10 0x559ce3bc9190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:55464/0 (socket says 172.21.15.90:55464) 2024-04-02T14:42:39.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.290+0000 7f2475400640 1 -- 172.21.15.90:0/1082074428 learned_addr learned my addr 172.21.15.90:0/1082074428 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:42:39.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.290+0000 7f246e000640 1 -- 172.21.15.90:0/451305204 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f2464012760 con 0x7f2470102450 2024-04-02T14:42:39.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.290+0000 7f2475e00640 1 --2- 172.21.15.90:0/1082074428 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559ce3bc99f0 0x559ce3bc8740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:39.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.290+0000 7f2475400640 1 -- 172.21.15.90:0/1082074428 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559ce3bc99f0 msgr2=0x559ce3bc8740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:42:39.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.290+0000 7f2475400640 1 --2- 172.21.15.90:0/1082074428 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559ce3bc99f0 0x559ce3bc8740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:39.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.290+0000 7f2475400640 1 -- 172.21.15.90:0/1082074428 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559ce3bd2080 msgr2=0x559ce3bd4470 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-02T14:42:39.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.290+0000 7f2475400640 1 --2- 172.21.15.90:0/1082074428 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559ce3bd2080 0x559ce3bd4470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:39.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.291+0000 7f2475400640 1 -- 172.21.15.90:0/1082074428 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559ce38413f0 con 0x559ce3bc8d10 2024-04-02T14:42:39.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.291+0000 7f2475400640 1 --2- 172.21.15.90:0/1082074428 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559ce3bc8d10 0x559ce3bc9190 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f245c006010 tx=0x7f245c02c650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=be255494111540a9 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:42:39.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.292+0000 7f2462600640 1 -- 172.21.15.90:0/1082074428 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f245c043020 con 0x559ce3bc8d10 2024-04-02T14:42:39.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.292+0000 7f2462600640 1 -- 172.21.15.90:0/1082074428 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f245c02c880 con 0x559ce3bc8d10 2024-04-02T14:42:39.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.292+0000 7f2462600640 1 -- 172.21.15.90:0/1082074428 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f245c0046c0 con 0x559ce3bc8d10 2024-04-02T14:42:39.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.292+0000 7f246e000640 1 --2- 172.21.15.90:0/451305204 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f24580a3fa0 0x7f24580a6460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:39.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.293+0000 7f246e000640 1 -- 172.21.15.90:0/451305204 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8550+0+0 (secure 0 0 0) 0x7f24640d4200 con 0x7f2470102450 2024-04-02T14:42:39.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.293+0000 7f2475e00640 1 --2- 172.21.15.90:0/451305204 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f24580a3fa0 0x7f24580a6460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:39.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.294+0000 7f247aa16740 1 -- 172.21.15.90:0/1082074428 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559ce3bc8d10 msgr2=0x559ce3bc9190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:39.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.294+0000 7f247aa16740 1 --2- 172.21.15.90:0/1082074428 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559ce3bc8d10 0x559ce3bc9190 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f245c006010 tx=0x7f245c02c650 comp rx=0 tx=0).stop 2024-04-02T14:42:39.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.294+0000 7f2475e00640 1 --2- 172.21.15.90:0/451305204 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f24580a3fa0 0x7f24580a6460 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f2468005cc0 tx=0x7f246800a330 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:42:39.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.294+0000 7f247aa16740 1 -- 172.21.15.90:0/1082074428 shutdown_connections 2024-04-02T14:42:39.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.294+0000 7f247aa16740 1 --2- 172.21.15.90:0/1082074428 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559ce3bd2080 0x559ce3bd4470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:39.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.294+0000 7f247aa16740 1 --2- 172.21.15.90:0/1082074428 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559ce3bc8d10 0x559ce3bc9190 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:39.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.294+0000 7f247aa16740 1 --2- 172.21.15.90:0/1082074428 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559ce3bc99f0 0x559ce3bc8740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:39.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.294+0000 7f247aa16740 1 -- 172.21.15.90:0/1082074428 >> 172.21.15.90:0/1082074428 conn(0x559ce3b8fa50 msgr2=0x559ce3bc3760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:42:39.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.294+0000 7f247aa16740 1 -- 172.21.15.90:0/1082074428 shutdown_connections 2024-04-02T14:42:39.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.294+0000 7f247aa16740 1 -- 172.21.15.90:0/1082074428 wait complete. 2024-04-02T14:42:39.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.295+0000 7f247aa16740 1 -- start start 2024-04-02T14:42:39.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.295+0000 7f247aa16740 1 Processor -- start 2024-04-02T14:42:39.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.296+0000 7f247aa16740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559ce3bc8d10 0x559ce3bcf070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:39.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.296+0000 7f247aa16740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559ce3bc99f0 0x559ce3bcf5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:39.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.296+0000 7f247aa16740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559ce3bd2080 0x559ce3bcfaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:39.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.296+0000 7f247aa16740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x559ce3b296a0 con 0x559ce3bd2080 2024-04-02T14:42:39.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.296+0000 7f247aa16740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x559ce3ad5980 con 0x559ce3bc8d10 2024-04-02T14:42:39.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.296+0000 7f247aa16740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x559ce3ae8cb0 con 0x559ce3bc99f0 2024-04-02T14:42:39.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.296+0000 7f247aa16740 10 client.0 _ll_register_callbacks cb 0x559ce3ae86a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:42:39.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.296+0000 7f2474a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559ce3bc99f0 0x559ce3bcf5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:39.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.296+0000 7f2474a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559ce3bc99f0 0x559ce3bcf5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:47900/0 (socket says 172.21.15.90:47900) 2024-04-02T14:42:39.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.296+0000 7f2474a00640 1 -- 172.21.15.90:0/2622446669 learned_addr learned my addr 172.21.15.90:0/2622446669 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:42:39.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.296+0000 7f2475400640 1 --2- 172.21.15.90:0/2622446669 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559ce3bc8d10 0x559ce3bcf070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:39.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.296+0000 7f2475e00640 1 --2- 172.21.15.90:0/2622446669 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559ce3bd2080 0x559ce3bcfaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:39.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.297+0000 7f2474a00640 1 -- 172.21.15.90:0/2622446669 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559ce3bc8d10 msgr2=0x559ce3bcf070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:42:39.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.297+0000 7f2474a00640 1 --2- 172.21.15.90:0/2622446669 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559ce3bc8d10 0x559ce3bcf070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:39.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.297+0000 7f2474a00640 1 -- 172.21.15.90:0/2622446669 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559ce3bd2080 msgr2=0x559ce3bcfaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:42:39.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.297+0000 7f2474a00640 1 --2- 172.21.15.90:0/2622446669 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559ce3bd2080 0x559ce3bcfaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:39.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.297+0000 7f2474a00640 1 -- 172.21.15.90:0/2622446669 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559ce390b7d0 con 0x559ce3bc99f0 2024-04-02T14:42:39.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.298+0000 7f2474a00640 1 --2- 172.21.15.90:0/2622446669 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559ce3bc99f0 0x559ce3bcf5b0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f24640058c0 tx=0x7f2464019370 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dff92c00bf8eb185 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:42:39.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.298+0000 7f2456a00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:42:39.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.298+0000 7f2456a00640 1 -- 172.21.15.90:0/2622446669 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24640a4640 con 0x559ce3bc99f0 2024-04-02T14:42:39.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.298+0000 7f247aa16740 1 -- 172.21.15.90:0/2622446669 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f245c02c880 con 0x559ce3bc99f0 2024-04-02T14:42:39.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.298+0000 7f2456a00640 1 -- 172.21.15.90:0/2622446669 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8550+0+0 (secure 0 0 0) 0x7f2464009cc0 con 0x559ce3bc99f0 2024-04-02T14:42:39.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.299+0000 7f2456a00640 1 -- 172.21.15.90:0/2622446669 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f24640a8f00 con 0x559ce3bc99f0 2024-04-02T14:42:39.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.299+0000 7f247aa16740 10 client.27745 fetch_fsmap learned FSMap version 27 2024-04-02T14:42:39.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.299+0000 7f247aa16740 1 -- 172.21.15.90:0/2622446669 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x559ce3be97e0 con 0x559ce3bc99f0 2024-04-02T14:42:39.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.299+0000 7f2456a00640 1 -- 172.21.15.90:0/2622446669 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f245c02c880 con 0x559ce3bc99f0 2024-04-02T14:42:39.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.300+0000 7f2456a00640 1 -- 172.21.15.90:0/2622446669 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f24640a8a70 con 0x559ce3bc99f0 2024-04-02T14:42:39.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.300+0000 7f247aa16740 10 client.27745 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:42:39.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.300+0000 7f247aa16740 10 client.27745 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:42:39.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.300+0000 7f247aa16740 20 client.27745 populate_metadata read hostname 'smithi090' 2024-04-02T14:42:39.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.300+0000 7f247aa16740 1 --2- 172.21.15.90:0/2622446669 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x559ce3beabf0 0x559ce3becfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:39.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.300+0000 7f247aa16740 4 client.27745 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:42:39.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.300+0000 7f247aa16740 1 -- 172.21.15.90:0/2622446669 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559ce3aed040 con 0x559ce3beabf0 2024-04-02T14:42:39.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.300+0000 7f2475400640 1 --2- 172.21.15.90:0/2622446669 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x559ce3beabf0 0x559ce3becfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:39.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.301+0000 7f2475400640 1 --2- 172.21.15.90:0/2622446669 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x559ce3beabf0 0x559ce3becfe0 crc :-1 s=READY pgs=214 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6a694d63250bbd03 server_cookie=5f84e7189eae00ce in_seq=0 out_seq=0 2024-04-02T14:42:39.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.301+0000 7f2456a00640 10 client.27745 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:42:39.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.303+0000 7f2456a00640 1 -- 172.21.15.90:0/2622446669 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x559ce3aed040 con 0x559ce3beabf0 2024-04-02T14:42:39.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.303+0000 7f2456a00640 10 client.27745 handle_command_reply: tid=0 2024-04-02T14:42:39.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.303+0000 7f247aa16740 1 client.27745 shutdown 2024-04-02T14:42:39.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.303+0000 7f247aa16740 1 -- 172.21.15.90:0/2622446669 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559ce3bc99f0 msgr2=0x559ce3bcf5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:39.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.303+0000 7f247aa16740 1 --2- 172.21.15.90:0/2622446669 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559ce3bc99f0 0x559ce3bcf5b0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f24640058c0 tx=0x7f2464019370 comp rx=0 tx=0).stop 2024-04-02T14:42:39.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.303+0000 7f247aa16740 1 -- 172.21.15.90:0/2622446669 shutdown_connections 2024-04-02T14:42:39.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.303+0000 7f247aa16740 1 --2- 172.21.15.90:0/2622446669 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x559ce3beabf0 0x559ce3becfe0 crc :-1 s=READY pgs=214 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:39.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.303+0000 7f247aa16740 1 --2- 172.21.15.90:0/2622446669 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559ce3bd2080 0x559ce3bcfaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:39.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.303+0000 7f247aa16740 1 --2- 172.21.15.90:0/2622446669 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559ce3bc99f0 0x559ce3bcf5b0 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:39.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.303+0000 7f247aa16740 1 --2- 172.21.15.90:0/2622446669 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559ce3bc8d10 0x559ce3bcf070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:39.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.303+0000 7f247aa16740 1 -- 172.21.15.90:0/2622446669 >> 172.21.15.90:0/2622446669 conn(0x559ce3b8fa50 msgr2=0x559ce3bc54b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:42:39.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.303+0000 7f2456a00640 0 client.27745 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:42:39.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.304+0000 7f247aa16740 1 -- 172.21.15.90:0/2622446669 shutdown_connections 2024-04-02T14:42:39.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.304+0000 7f247aa16740 1 -- 172.21.15.90:0/2622446669 wait complete. 2024-04-02T14:42:39.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.304+0000 7f247aa16740 20 client.27745 trim_cache size 0 max 16384 2024-04-02T14:42:39.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.332+0000 7f247aa16740 1 Processor -- start 2024-04-02T14:42:39.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.332+0000 7f247aa16740 1 -- start start 2024-04-02T14:42:39.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.333+0000 7f247aa16740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559ce3bc8d10 0x559ce3bddaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:39.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.333+0000 7f247aa16740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559ce3bc99f0 0x559ce3bcf910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:39.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.333+0000 7f247aa16740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559ce3bd2080 0x559ce3bc52c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:39.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.333+0000 7f247aa16740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f24701ab670 con 0x559ce3bd2080 2024-04-02T14:42:39.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.333+0000 7f247aa16740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f24701ad1b0 con 0x559ce3bc8d10 2024-04-02T14:42:39.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.333+0000 7f247aa16740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f24701abd40 con 0x559ce3bc99f0 2024-04-02T14:42:39.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.333+0000 7f2475400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559ce3bc8d10 0x559ce3bddaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:39.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.333+0000 7f2475400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559ce3bc8d10 0x559ce3bddaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:55488/0 (socket says 172.21.15.90:55488) 2024-04-02T14:42:39.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.333+0000 7f2475400640 1 -- 172.21.15.90:0/3630454575 learned_addr learned my addr 172.21.15.90:0/3630454575 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:42:39.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.334+0000 7f2475e00640 1 --2- 172.21.15.90:0/3630454575 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559ce3bd2080 0x559ce3bc52c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:39.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.334+0000 7f2475400640 1 -- 172.21.15.90:0/3630454575 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559ce3bc99f0 msgr2=0x559ce3bcf910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:42:39.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.334+0000 7f2474a00640 1 --2- 172.21.15.90:0/3630454575 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559ce3bc99f0 0x559ce3bcf910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:39.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.334+0000 7f2475400640 1 --2- 172.21.15.90:0/3630454575 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559ce3bc99f0 0x559ce3bcf910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:39.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.334+0000 7f2475400640 1 -- 172.21.15.90:0/3630454575 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559ce3bd2080 msgr2=0x559ce3bc52c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:42:39.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.334+0000 7f2475400640 1 --2- 172.21.15.90:0/3630454575 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559ce3bd2080 0x559ce3bc52c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:39.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.334+0000 7f2475400640 1 -- 172.21.15.90:0/3630454575 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559ce3bc3970 con 0x559ce3bc8d10 2024-04-02T14:42:39.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.335+0000 7f2475400640 1 --2- 172.21.15.90:0/3630454575 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559ce3bc8d10 0x559ce3bddaf0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f245c009230 tx=0x7f245c02bc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4fdaa208f61eca1c server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:42:39.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.335+0000 7f2461c00640 1 -- 172.21.15.90:0/3630454575 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f245c043040 con 0x559ce3bc8d10 2024-04-02T14:42:39.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.335+0000 7f2461c00640 1 -- 172.21.15.90:0/3630454575 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f245c02c880 con 0x559ce3bc8d10 2024-04-02T14:42:39.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.335+0000 7f2461c00640 1 -- 172.21.15.90:0/3630454575 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f245c053740 con 0x559ce3bc8d10 2024-04-02T14:42:39.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.336+0000 7f247aa16740 1 -- 172.21.15.90:0/3630454575 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559ce3bc8d10 msgr2=0x559ce3bddaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:39.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.336+0000 7f247aa16740 1 --2- 172.21.15.90:0/3630454575 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559ce3bc8d10 0x559ce3bddaf0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f245c009230 tx=0x7f245c02bc30 comp rx=0 tx=0).stop 2024-04-02T14:42:39.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.336+0000 7f247aa16740 1 -- 172.21.15.90:0/3630454575 shutdown_connections 2024-04-02T14:42:39.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.336+0000 7f247aa16740 1 --2- 172.21.15.90:0/3630454575 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559ce3bd2080 0x559ce3bc52c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:39.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.336+0000 7f247aa16740 1 --2- 172.21.15.90:0/3630454575 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559ce3bc99f0 0x559ce3bcf910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:39.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.336+0000 7f247aa16740 1 --2- 172.21.15.90:0/3630454575 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559ce3bc8d10 0x559ce3bddaf0 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:39.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.336+0000 7f247aa16740 1 -- 172.21.15.90:0/3630454575 >> 172.21.15.90:0/3630454575 conn(0x559ce3b8fa50 msgr2=0x559ce3bc4bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:42:39.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.336+0000 7f247aa16740 1 -- 172.21.15.90:0/3630454575 shutdown_connections 2024-04-02T14:42:39.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.337+0000 7f247aa16740 1 -- 172.21.15.90:0/3630454575 wait complete. 2024-04-02T14:42:39.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.337+0000 7f247aa16740 1 -- start start 2024-04-02T14:42:39.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.337+0000 7f247aa16740 1 Processor -- start 2024-04-02T14:42:39.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.338+0000 7f247aa16740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559ce3bc8d10 0x559ce3bda500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:39.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.338+0000 7f247aa16740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559ce3bc99f0 0x559ce3bcc4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:39.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.338+0000 7f2475400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559ce3bc8d10 0x559ce3bda500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:39.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.338+0000 7f2475400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559ce3bc8d10 0x559ce3bda500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:47920/0 (socket says 172.21.15.90:47920) 2024-04-02T14:42:39.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.338+0000 7f2475400640 1 -- 172.21.15.90:0/1648456271 learned_addr learned my addr 172.21.15.90:0/1648456271 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:42:39.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.338+0000 7f247aa16740 1 --2- 172.21.15.90:0/1648456271 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559ce3bd2080 0x559ce3bccbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:39.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.339+0000 7f247aa16740 1 -- 172.21.15.90:0/1648456271 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f24701acae0 con 0x559ce3bd2080 2024-04-02T14:42:39.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.339+0000 7f247aa16740 1 -- 172.21.15.90:0/1648456271 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f24701ac410 con 0x559ce3bc99f0 2024-04-02T14:42:39.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.339+0000 7f247aa16740 1 -- 172.21.15.90:0/1648456271 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f24701a9670 con 0x559ce3bc8d10 2024-04-02T14:42:39.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.339+0000 7f247aa16740 10 client.0 _ll_register_callbacks cb 0x7f246409fbf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:42:39.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.339+0000 7f2475e00640 1 --2- 172.21.15.90:0/1648456271 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559ce3bd2080 0x559ce3bccbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:39.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.339+0000 7f2474a00640 1 --2- 172.21.15.90:0/1648456271 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559ce3bc99f0 0x559ce3bcc4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:39.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.339+0000 7f2475400640 1 -- 172.21.15.90:0/1648456271 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559ce3bc99f0 msgr2=0x559ce3bcc4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:42:39.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.339+0000 7f2475400640 1 --2- 172.21.15.90:0/1648456271 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559ce3bc99f0 0x559ce3bcc4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:39.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.339+0000 7f2475400640 1 -- 172.21.15.90:0/1648456271 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559ce3bd2080 msgr2=0x559ce3bccbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:42:39.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.339+0000 7f2475400640 1 --2- 172.21.15.90:0/1648456271 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559ce3bd2080 0x559ce3bccbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:39.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.339+0000 7f2475400640 1 -- 172.21.15.90:0/1648456271 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559ce3bd2c30 con 0x559ce3bc8d10 2024-04-02T14:42:39.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.339+0000 7f2474a00640 1 --2- 172.21.15.90:0/1648456271 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559ce3bc99f0 0x559ce3bcc4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:42:39.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.339+0000 7f2475400640 1 --2- 172.21.15.90:0/1648456271 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559ce3bc8d10 0x559ce3bda500 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f245c053e40 tx=0x7f245c000bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e28a10c0227ed202 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:42:39.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.339+0000 7f2456a00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:42:39.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.340+0000 7f2456a00640 1 -- 172.21.15.90:0/1648456271 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f245c043040 con 0x559ce3bc8d10 2024-04-02T14:42:39.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.340+0000 7f2456a00640 1 -- 172.21.15.90:0/1648456271 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8550+0+0 (secure 0 0 0) 0x7f245c004240 con 0x559ce3bc8d10 2024-04-02T14:42:39.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.340+0000 7f247aa16740 1 -- 172.21.15.90:0/1648456271 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f245c02c880 con 0x559ce3bc8d10 2024-04-02T14:42:39.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.340+0000 7f2456a00640 1 -- 172.21.15.90:0/1648456271 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f245c004440 con 0x559ce3bc8d10 2024-04-02T14:42:39.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.340+0000 7f247aa16740 10 client.27748 fetch_fsmap learned FSMap version 27 2024-04-02T14:42:39.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.340+0000 7f247aa16740 1 -- 172.21.15.90:0/1648456271 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x559ce3c512f0 con 0x559ce3bc8d10 2024-04-02T14:42:39.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.342+0000 7f2456a00640 1 -- 172.21.15.90:0/1648456271 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f245c02c880 con 0x559ce3bc8d10 2024-04-02T14:42:39.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.342+0000 7f2456a00640 1 -- 172.21.15.90:0/1648456271 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f245c034440 con 0x559ce3bc8d10 2024-04-02T14:42:39.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.342+0000 7f247aa16740 10 client.27748 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:42:39.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.342+0000 7f247aa16740 10 client.27748 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:42:39.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.342+0000 7f247aa16740 20 client.27748 populate_metadata read hostname 'smithi090' 2024-04-02T14:42:39.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.342+0000 7f247aa16740 1 --2- 172.21.15.90:0/1648456271 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x559ce3c518f0 0x559ce3c53d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:39.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.342+0000 7f247aa16740 4 client.27748 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-02T14:42:39.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.342+0000 7f247aa16740 1 -- 172.21.15.90:0/1648456271 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "damage ls"}) -- 0x559ce3b3ac30 con 0x559ce3c518f0 2024-04-02T14:42:39.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.343+0000 7f2474a00640 1 --2- 172.21.15.90:0/1648456271 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x559ce3c518f0 0x559ce3c53d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:39.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.344+0000 7f2474a00640 1 --2- 172.21.15.90:0/1648456271 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x559ce3c518f0 0x559ce3c53d10 crc :-1 s=READY pgs=215 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=65509ec80a8fedfb server_cookie=5dc05d4b095a0117 in_seq=0 out_seq=0 2024-04-02T14:42:39.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.344+0000 7f2456a00640 10 client.27748 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:42:39.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.344+0000 7f2456a00640 1 -- 172.21.15.90:0/1648456271 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x559ce3b3ac30 con 0x559ce3c518f0 2024-04-02T14:42:39.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.344+0000 7f2456a00640 10 client.27748 handle_command_reply: tid=0 2024-04-02T14:42:39.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.344+0000 7f247aa16740 1 client.27748 shutdown 2024-04-02T14:42:39.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.344+0000 7f247aa16740 1 -- 172.21.15.90:0/1648456271 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559ce3bc8d10 msgr2=0x559ce3bda500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:39.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.344+0000 7f247aa16740 1 --2- 172.21.15.90:0/1648456271 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559ce3bc8d10 0x559ce3bda500 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f245c053e40 tx=0x7f245c000bc0 comp rx=0 tx=0).stop 2024-04-02T14:42:39.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.345+0000 7f247aa16740 1 -- 172.21.15.90:0/1648456271 shutdown_connections 2024-04-02T14:42:39.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.345+0000 7f247aa16740 1 --2- 172.21.15.90:0/1648456271 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x559ce3c518f0 0x559ce3c53d10 crc :-1 s=READY pgs=215 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:39.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.345+0000 7f247aa16740 1 --2- 172.21.15.90:0/1648456271 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559ce3bd2080 0x559ce3bccbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:39.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.345+0000 7f247aa16740 1 --2- 172.21.15.90:0/1648456271 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559ce3bc99f0 0x559ce3bcc4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:39.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.345+0000 7f247aa16740 1 --2- 172.21.15.90:0/1648456271 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559ce3bc8d10 0x559ce3bda500 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:39.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.345+0000 7f247aa16740 1 -- 172.21.15.90:0/1648456271 >> 172.21.15.90:0/1648456271 conn(0x559ce3b8fa50 msgr2=0x559ce3bc2010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:42:39.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.345+0000 7f2456a00640 0 client.27748 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:42:39.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.346+0000 7f247aa16740 1 -- 172.21.15.90:0/1648456271 shutdown_connections 2024-04-02T14:42:39.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.346+0000 7f247aa16740 1 -- 172.21.15.90:0/1648456271 wait complete. 2024-04-02T14:42:39.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.346+0000 7f247aa16740 20 client.27748 trim_cache size 0 max 16384 2024-04-02T14:42:39.345 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-02T14:42:39.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.347+0000 7f2477e00640 1 -- 172.21.15.90:0/451305204 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f24580a3fa0 msgr2=0x7f24580a6460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:39.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.347+0000 7f2477e00640 1 --2- 172.21.15.90:0/451305204 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f24580a3fa0 0x7f24580a6460 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f2468005cc0 tx=0x7f246800a330 comp rx=0 tx=0).stop 2024-04-02T14:42:39.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.347+0000 7f2477e00640 1 -- 172.21.15.90:0/451305204 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2470102450 msgr2=0x7f2470198850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:39.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.347+0000 7f2477e00640 1 --2- 172.21.15.90:0/451305204 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2470102450 0x7f2470198850 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f246400bea0 tx=0x7f246400bf90 comp rx=0 tx=0).stop 2024-04-02T14:42:39.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.348+0000 7f2477e00640 1 -- 172.21.15.90:0/451305204 shutdown_connections 2024-04-02T14:42:39.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.348+0000 7f2477e00640 1 --2- 172.21.15.90:0/451305204 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f24580a3fa0 0x7f24580a6460 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:39.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.348+0000 7f2477e00640 1 --2- 172.21.15.90:0/451305204 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2470107aa0 0x7f247019cbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:39.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.348+0000 7f2477e00640 1 --2- 172.21.15.90:0/451305204 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2470102450 0x7f2470198850 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:39.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.348+0000 7f2477e00640 1 --2- 172.21.15.90:0/451305204 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2470101a90 0x7f2470198310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:39.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.348+0000 7f2477e00640 1 -- 172.21.15.90:0/451305204 >> 172.21.15.90:0/451305204 conn(0x7f2470078f00 msgr2=0x7f2470104340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:42:39.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.349+0000 7f2477e00640 1 -- 172.21.15.90:0/451305204 shutdown_connections 2024-04-02T14:42:39.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:39.349+0000 7f2477e00640 1 -- 172.21.15.90:0/451305204 wait complete. 2024-04-02T14:42:40.372 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-02T14:42:40.372 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T14:42:40.372 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-02T14:42:40.433 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:42:40 smithi177 ceph-mon[36431]: pgmap v238: 97 pgs: 97 active+clean; 4.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 30 MiB/s wr, 1.50k op/s 2024-04-02T14:42:40.435 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:42:40 smithi090 ceph-mon[31674]: pgmap v238: 97 pgs: 97 active+clean; 4.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 30 MiB/s wr, 1.50k op/s 2024-04-02T14:42:40.607 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:42:40 smithi092 ceph-mon[36497]: pgmap v238: 97 pgs: 97 active+clean; 4.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 30 MiB/s wr, 1.50k op/s 2024-04-02T14:42:40.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.656+0000 7ffab6a00640 1 -- 172.21.15.90:0/3603423734 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ffab010c110 msgr2=0x7ffab010e540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:40.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.657+0000 7ffab6a00640 1 --2- 172.21.15.90:0/3603423734 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ffab010c110 0x7ffab010e540 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7ffaa8009be0 tx=0x7ffaa802bf80 comp rx=0 tx=0).stop 2024-04-02T14:42:40.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.657+0000 7ffab6a00640 1 -- 172.21.15.90:0/3603423734 shutdown_connections 2024-04-02T14:42:40.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.657+0000 7ffab6a00640 1 --2- 172.21.15.90:0/3603423734 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ffab010c110 0x7ffab010e540 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:40.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.657+0000 7ffab6a00640 1 --2- 172.21.15.90:0/3603423734 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffab0103660 0x7ffab010bb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:40.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.657+0000 7ffab6a00640 1 --2- 172.21.15.90:0/3603423734 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffab0102d40 0x7ffab0103120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:40.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.657+0000 7ffab6a00640 1 -- 172.21.15.90:0/3603423734 >> 172.21.15.90:0/3603423734 conn(0x7ffab0076d80 msgr2=0x7ffab0077190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:42:40.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.657+0000 7ffab6a00640 1 -- 172.21.15.90:0/3603423734 shutdown_connections 2024-04-02T14:42:40.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.657+0000 7ffab6a00640 1 -- 172.21.15.90:0/3603423734 wait complete. 2024-04-02T14:42:40.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.659+0000 7ffab6a00640 1 Processor -- start 2024-04-02T14:42:40.657 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.659+0000 7ffab6a00640 1 -- start start 2024-04-02T14:42:40.657 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.660+0000 7ffab6a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffab0102d40 0x7ffab010f150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:40.657 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.660+0000 7ffab6a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffab0103660 0x7ffab010f690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:40.658 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.660+0000 7ffab6a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ffab010c110 0x7ffab010fbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:40.658 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.660+0000 7ffab6a00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7ffab01140e0 con 0x7ffab0103660 2024-04-02T14:42:40.658 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.660+0000 7ffab6a00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7ffab0113f60 con 0x7ffab010c110 2024-04-02T14:42:40.658 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.660+0000 7ffab6a00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7ffab0114260 con 0x7ffab0102d40 2024-04-02T14:42:40.658 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.660+0000 7ffaafe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffab0102d40 0x7ffab010f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:40.658 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.660+0000 7ffaafe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffab0102d40 0x7ffab010f150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:47948/0 (socket says 172.21.15.90:47948) 2024-04-02T14:42:40.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.660+0000 7ffaafe00640 1 -- 172.21.15.90:0/3246734184 learned_addr learned my addr 172.21.15.90:0/3246734184 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:42:40.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.660+0000 7ffab4a00640 1 --2- 172.21.15.90:0/3246734184 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ffab010c110 0x7ffab010fbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:40.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.661+0000 7ffaaf400640 1 --2- 172.21.15.90:0/3246734184 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffab0103660 0x7ffab010f690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:40.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.661+0000 7ffab4a00640 1 -- 172.21.15.90:0/3246734184 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffab0102d40 msgr2=0x7ffab010f150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:40.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.661+0000 7ffab4a00640 1 --2- 172.21.15.90:0/3246734184 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffab0102d40 0x7ffab010f150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:40.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.661+0000 7ffab4a00640 1 -- 172.21.15.90:0/3246734184 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffab0103660 msgr2=0x7ffab010f690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:40.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.661+0000 7ffab4a00640 1 --2- 172.21.15.90:0/3246734184 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffab0103660 0x7ffab010f690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:40.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.661+0000 7ffab4a00640 1 -- 172.21.15.90:0/3246734184 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffab01aabc0 con 0x7ffab010c110 2024-04-02T14:42:40.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.661+0000 7ffab4a00640 1 --2- 172.21.15.90:0/3246734184 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ffab010c110 0x7ffab010fbd0 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7ffaa802c590 tx=0x7ffaa8004270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:42:40.660 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.662+0000 7ffaacc00640 1 -- 172.21.15.90:0/3246734184 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffaa8043020 con 0x7ffab010c110 2024-04-02T14:42:40.660 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.662+0000 7ffaacc00640 1 -- 172.21.15.90:0/3246734184 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffaa8004630 con 0x7ffab010c110 2024-04-02T14:42:40.660 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.662+0000 7ffaacc00640 1 -- 172.21.15.90:0/3246734184 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffaa803e740 con 0x7ffab010c110 2024-04-02T14:42:40.660 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.662+0000 7ffab6a00640 1 -- 172.21.15.90:0/3246734184 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffab01aae50 con 0x7ffab010c110 2024-04-02T14:42:40.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.663+0000 7ffab6a00640 1 -- 172.21.15.90:0/3246734184 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffab01ab2c0 con 0x7ffab010c110 2024-04-02T14:42:40.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.667+0000 7ffaacc00640 1 -- 172.21.15.90:0/3246734184 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7ffaa8046020 con 0x7ffab010c110 2024-04-02T14:42:40.666 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.669+0000 7ffab9795740 1 Processor -- start 2024-04-02T14:42:40.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.669+0000 7ffaacc00640 1 --2- 172.21.15.90:0/3246734184 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7ffa940a3ea0 0x7ffa940a6360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:40.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.669+0000 7ffab9795740 1 -- start start 2024-04-02T14:42:40.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.670+0000 7ffaacc00640 1 -- 172.21.15.90:0/3246734184 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8550+0+0 (secure 0 0 0) 0x7ffaa80f2ad0 con 0x7ffab010c110 2024-04-02T14:42:40.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.671+0000 7ffab9795740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558fe1f05030 0x558fe1f03d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:40.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.671+0000 7ffab9795740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558fe1f04350 0x558fe1f047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:40.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.671+0000 7ffaaf400640 1 --2- 172.21.15.90:0/3246734184 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7ffa940a3ea0 0x7ffa940a6360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:40.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.672+0000 7ffaafe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558fe1f05030 0x558fe1f03d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:40.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.672+0000 7ffab9795740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558fe1f0d6c0 0x558fe1f0fab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:40.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.672+0000 7ffaafe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558fe1f05030 0x558fe1f03d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:47958/0 (socket says 172.21.15.90:47958) 2024-04-02T14:42:40.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.672+0000 7ffaaf400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558fe1f04350 0x558fe1f047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:40.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.672+0000 7ffab4a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558fe1f0d6c0 0x558fe1f0fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:40.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.673+0000 7ffaafe00640 1 -- 172.21.15.90:0/4161605964 learned_addr learned my addr 172.21.15.90:0/4161605964 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:42:40.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.673+0000 7ffab9795740 1 -- 172.21.15.90:0/4161605964 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x558fe1e68310 con 0x558fe1f0d6c0 2024-04-02T14:42:40.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.673+0000 7ffab9795740 1 -- 172.21.15.90:0/4161605964 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x558fe1e10690 con 0x558fe1f04350 2024-04-02T14:42:40.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.673+0000 7ffab9795740 1 -- 172.21.15.90:0/4161605964 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x558fe1e642c0 con 0x558fe1f05030 2024-04-02T14:42:40.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.673+0000 7ffab4a00640 1 -- 172.21.15.90:0/4161605964 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558fe1f05030 msgr2=0x558fe1f03d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:42:40.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.673+0000 7ffab4a00640 1 --2- 172.21.15.90:0/4161605964 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558fe1f05030 0x558fe1f03d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:40.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.673+0000 7ffab4a00640 1 -- 172.21.15.90:0/4161605964 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558fe1f04350 msgr2=0x558fe1f047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:42:40.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.673+0000 7ffab4a00640 1 --2- 172.21.15.90:0/4161605964 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558fe1f04350 0x558fe1f047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:40.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.673+0000 7ffab4a00640 1 -- 172.21.15.90:0/4161605964 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558fe1c447e0 con 0x558fe1f0d6c0 2024-04-02T14:42:40.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.673+0000 7ffaaf400640 1 --2- 172.21.15.90:0/3246734184 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7ffa940a3ea0 0x7ffa940a6360 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7ffab01108c0 tx=0x7ffa9c0127c0 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:42:40.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.674+0000 7ffab4a00640 1 --2- 172.21.15.90:0/4161605964 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558fe1f0d6c0 0x558fe1f0fab0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7ffaa80f0310 tx=0x7ffaa80be4c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c80953ad188714af server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:42:40.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.675+0000 7ffaa5200640 1 -- 172.21.15.90:0/4161605964 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffaa80beb60 con 0x558fe1f0d6c0 2024-04-02T14:42:40.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.675+0000 7ffaa5200640 1 -- 172.21.15.90:0/4161605964 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffaa80c2f10 con 0x558fe1f0d6c0 2024-04-02T14:42:40.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.676+0000 7ffab9795740 1 -- 172.21.15.90:0/4161605964 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558fe1f0d6c0 msgr2=0x558fe1f0fab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:40.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.676+0000 7ffab9795740 1 --2- 172.21.15.90:0/4161605964 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558fe1f0d6c0 0x558fe1f0fab0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7ffaa80f0310 tx=0x7ffaa80be4c0 comp rx=0 tx=0).stop 2024-04-02T14:42:40.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.677+0000 7ffab9795740 1 -- 172.21.15.90:0/4161605964 shutdown_connections 2024-04-02T14:42:40.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.677+0000 7ffab9795740 1 --2- 172.21.15.90:0/4161605964 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558fe1f0d6c0 0x558fe1f0fab0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:40.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.677+0000 7ffab9795740 1 --2- 172.21.15.90:0/4161605964 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558fe1f04350 0x558fe1f047d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:40.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.678+0000 7ffab9795740 1 --2- 172.21.15.90:0/4161605964 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558fe1f05030 0x558fe1f03d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:40.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.678+0000 7ffab9795740 1 -- 172.21.15.90:0/4161605964 >> 172.21.15.90:0/4161605964 conn(0x558fe1e98270 msgr2=0x558fe1efeda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:42:40.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.678+0000 7ffab9795740 1 -- 172.21.15.90:0/4161605964 shutdown_connections 2024-04-02T14:42:40.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.679+0000 7ffab9795740 1 -- 172.21.15.90:0/4161605964 wait complete. 2024-04-02T14:42:40.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.679+0000 7ffab9795740 1 -- start start 2024-04-02T14:42:40.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.680+0000 7ffab9795740 1 Processor -- start 2024-04-02T14:42:40.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.680+0000 7ffab9795740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558fe1f04350 0x558fe1f0a570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:40.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.680+0000 7ffab9795740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558fe1f05030 0x558fe1f0aab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:40.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.680+0000 7ffab9795740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558fe1f0d6c0 0x558fe1f0aff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:40.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.680+0000 7ffab9795740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x558fe1e68310 con 0x558fe1f04350 2024-04-02T14:42:40.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.680+0000 7ffab9795740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x558fe1e23a60 con 0x558fe1f05030 2024-04-02T14:42:40.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.680+0000 7ffab9795740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x558fe1f102f0 con 0x558fe1f0d6c0 2024-04-02T14:42:40.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.680+0000 7ffab9795740 10 client.0 _ll_register_callbacks cb 0x558fe1e29030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:42:40.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.681+0000 7ffaaf400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558fe1f05030 0x558fe1f0aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:40.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.681+0000 7ffaaf400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558fe1f05030 0x558fe1f0aab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:55540/0 (socket says 172.21.15.90:55540) 2024-04-02T14:42:40.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.681+0000 7ffaaf400640 1 -- 172.21.15.90:0/21852551 learned_addr learned my addr 172.21.15.90:0/21852551 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:42:40.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.681+0000 7ffaaf400640 1 -- 172.21.15.90:0/21852551 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558fe1f0d6c0 msgr2=0x558fe1f0aff0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-02T14:42:40.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.681+0000 7ffaaf400640 1 --2- 172.21.15.90:0/21852551 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558fe1f0d6c0 0x558fe1f0aff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:40.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.681+0000 7ffaaf400640 1 -- 172.21.15.90:0/21852551 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558fe1f04350 msgr2=0x558fe1f0a570 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-02T14:42:40.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.681+0000 7ffaafe00640 1 --2- 172.21.15.90:0/21852551 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558fe1f04350 0x558fe1f0a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:40.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.682+0000 7ffaaf400640 1 --2- 172.21.15.90:0/21852551 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558fe1f04350 0x558fe1f0a570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:40.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.682+0000 7ffaaf400640 1 -- 172.21.15.90:0/21852551 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558fe1ca4f70 con 0x558fe1f05030 2024-04-02T14:42:40.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.682+0000 7ffaafe00640 1 --2- 172.21.15.90:0/21852551 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558fe1f04350 0x558fe1f0a570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:42:40.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.683+0000 7ffaaf400640 1 --2- 172.21.15.90:0/21852551 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558fe1f05030 0x558fe1f0aab0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7ffa9c00ec40 tx=0x7ffa9c004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=df65c219a5c0df28 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:42:40.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.683+0000 7ffa99600640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:42:40.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.683+0000 7ffa99600640 1 -- 172.21.15.90:0/21852551 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa9c01f650 con 0x558fe1f05030 2024-04-02T14:42:40.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.684+0000 7ffa99600640 1 -- 172.21.15.90:0/21852551 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8550+0+0 (secure 0 0 0) 0x7ffa9c01f7f0 con 0x558fe1f05030 2024-04-02T14:42:40.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.684+0000 7ffab9795740 1 -- 172.21.15.90:0/21852551 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffaa80c2f10 con 0x558fe1f05030 2024-04-02T14:42:40.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.684+0000 7ffa99600640 1 -- 172.21.15.90:0/21852551 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7ffa9c01f9f0 con 0x558fe1f05030 2024-04-02T14:42:40.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.684+0000 7ffab9795740 10 client.27911 fetch_fsmap learned FSMap version 27 2024-04-02T14:42:40.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.684+0000 7ffab9795740 1 -- 172.21.15.90:0/21852551 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x558fe1b3c170 con 0x558fe1f05030 2024-04-02T14:42:40.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.685+0000 7ffa99600640 1 -- 172.21.15.90:0/21852551 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffaa80c2f10 con 0x558fe1f05030 2024-04-02T14:42:40.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.686+0000 7ffa99600640 1 -- 172.21.15.90:0/21852551 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7ffa9c0278f0 con 0x558fe1f05030 2024-04-02T14:42:40.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.686+0000 7ffab9795740 10 client.27911 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:42:40.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.686+0000 7ffab9795740 10 client.27911 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:42:40.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.686+0000 7ffab9795740 20 client.27911 populate_metadata read hostname 'smithi090' 2024-04-02T14:42:40.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.686+0000 7ffab9795740 1 --2- 172.21.15.90:0/21852551 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x558fe1f1d750 0x558fe1f1fb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:40.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.686+0000 7ffab9795740 4 client.27911 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:42:40.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.687+0000 7ffab9795740 1 -- 172.21.15.90:0/21852551 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558fe1e75850 con 0x558fe1f1d750 2024-04-02T14:42:40.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.687+0000 7ffaafe00640 1 --2- 172.21.15.90:0/21852551 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x558fe1f1d750 0x558fe1f1fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:40.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.687+0000 7ffaafe00640 1 --2- 172.21.15.90:0/21852551 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x558fe1f1d750 0x558fe1f1fb40 crc :-1 s=READY pgs=216 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c1a881a935124cfc server_cookie=4229e3760f826c93 in_seq=0 out_seq=0 2024-04-02T14:42:40.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.687+0000 7ffa99600640 10 client.27911 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:42:40.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.688+0000 7ffa99600640 1 -- 172.21.15.90:0/21852551 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x558fe1e75850 con 0x558fe1f1d750 2024-04-02T14:42:40.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.688+0000 7ffa99600640 10 client.27911 handle_command_reply: tid=0 2024-04-02T14:42:40.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.688+0000 7ffab9795740 1 client.27911 shutdown 2024-04-02T14:42:40.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.689+0000 7ffab9795740 1 -- 172.21.15.90:0/21852551 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558fe1f05030 msgr2=0x558fe1f0aab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:40.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.689+0000 7ffab9795740 1 --2- 172.21.15.90:0/21852551 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558fe1f05030 0x558fe1f0aab0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7ffa9c00ec40 tx=0x7ffa9c004730 comp rx=0 tx=0).stop 2024-04-02T14:42:40.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.689+0000 7ffab9795740 1 -- 172.21.15.90:0/21852551 shutdown_connections 2024-04-02T14:42:40.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.689+0000 7ffab9795740 1 --2- 172.21.15.90:0/21852551 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x558fe1f1d750 0x558fe1f1fb40 crc :-1 s=READY pgs=216 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:40.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.689+0000 7ffab9795740 1 --2- 172.21.15.90:0/21852551 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558fe1f0d6c0 0x558fe1f0aff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:40.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.689+0000 7ffab9795740 1 --2- 172.21.15.90:0/21852551 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558fe1f05030 0x558fe1f0aab0 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:40.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.689+0000 7ffab9795740 1 --2- 172.21.15.90:0/21852551 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558fe1f04350 0x558fe1f0a570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:40.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.689+0000 7ffab9795740 1 -- 172.21.15.90:0/21852551 >> 172.21.15.90:0/21852551 conn(0x558fe1e98270 msgr2=0x558fe1f00af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:42:40.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.689+0000 7ffa99600640 0 client.27911 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:42:40.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.690+0000 7ffab9795740 1 -- 172.21.15.90:0/21852551 shutdown_connections 2024-04-02T14:42:40.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.691+0000 7ffab9795740 1 -- 172.21.15.90:0/21852551 wait complete. 2024-04-02T14:42:40.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.691+0000 7ffab9795740 20 client.27911 trim_cache size 0 max 16384 2024-04-02T14:42:40.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.730+0000 7ffab9795740 1 Processor -- start 2024-04-02T14:42:40.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.730+0000 7ffab9795740 1 -- start start 2024-04-02T14:42:40.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.730+0000 7ffab9795740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558fe1f04350 0x558fe1f0cca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:40.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.730+0000 7ffab9795740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558fe1f05030 0x558fe1f190d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:40.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.730+0000 7ffab9795740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558fe1f0d6c0 0x558fe1efdca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:40.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.730+0000 7ffab9795740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7ffab01b3d80 con 0x558fe1f04350 2024-04-02T14:42:40.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.730+0000 7ffab9795740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7ffab01b58c0 con 0x558fe1f0d6c0 2024-04-02T14:42:40.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.730+0000 7ffab9795740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7ffab01b4450 con 0x558fe1f05030 2024-04-02T14:42:40.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.731+0000 7ffab4a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558fe1f0d6c0 0x558fe1efdca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:40.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.731+0000 7ffab4a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558fe1f0d6c0 0x558fe1efdca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:55550/0 (socket says 172.21.15.90:55550) 2024-04-02T14:42:40.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.731+0000 7ffab4a00640 1 -- 172.21.15.90:0/2230570865 learned_addr learned my addr 172.21.15.90:0/2230570865 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:42:40.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.732+0000 7ffab4a00640 1 -- 172.21.15.90:0/2230570865 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558fe1f05030 msgr2=0x558fe1f190d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-02T14:42:40.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.732+0000 7ffab4a00640 1 --2- 172.21.15.90:0/2230570865 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558fe1f05030 0x558fe1f190d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:40.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.732+0000 7ffab4a00640 1 -- 172.21.15.90:0/2230570865 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558fe1f04350 msgr2=0x558fe1f0cca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:42:40.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.732+0000 7ffab4a00640 1 --2- 172.21.15.90:0/2230570865 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558fe1f04350 0x558fe1f0cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:40.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.732+0000 7ffab4a00640 1 -- 172.21.15.90:0/2230570865 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558fe1f1feb0 con 0x558fe1f0d6c0 2024-04-02T14:42:40.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.733+0000 7ffab4a00640 1 --2- 172.21.15.90:0/2230570865 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558fe1f0d6c0 0x558fe1efdca0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7ffaa80f03c0 tx=0x7ffaa80be5f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3f388124306bc633 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:42:40.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.734+0000 7ffa9be00640 1 -- 172.21.15.90:0/2230570865 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffaa80c2f10 con 0x558fe1f0d6c0 2024-04-02T14:42:40.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.734+0000 7ffa9be00640 1 -- 172.21.15.90:0/2230570865 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffaa80c77d0 con 0x558fe1f0d6c0 2024-04-02T14:42:40.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.734+0000 7ffa9be00640 1 -- 172.21.15.90:0/2230570865 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffaa80be1f0 con 0x558fe1f0d6c0 2024-04-02T14:42:40.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.735+0000 7ffab9795740 1 -- 172.21.15.90:0/2230570865 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558fe1f0d6c0 msgr2=0x558fe1efdca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:40.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.735+0000 7ffab9795740 1 --2- 172.21.15.90:0/2230570865 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558fe1f0d6c0 0x558fe1efdca0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7ffaa80f03c0 tx=0x7ffaa80be5f0 comp rx=0 tx=0).stop 2024-04-02T14:42:40.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.738+0000 7ffab9795740 1 -- 172.21.15.90:0/2230570865 shutdown_connections 2024-04-02T14:42:40.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.738+0000 7ffab9795740 1 --2- 172.21.15.90:0/2230570865 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558fe1f0d6c0 0x558fe1efdca0 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:40.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.738+0000 7ffab9795740 1 --2- 172.21.15.90:0/2230570865 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558fe1f05030 0x558fe1f190d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:40.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.738+0000 7ffab9795740 1 --2- 172.21.15.90:0/2230570865 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558fe1f04350 0x558fe1f0cca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:40.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.738+0000 7ffab9795740 1 -- 172.21.15.90:0/2230570865 >> 172.21.15.90:0/2230570865 conn(0x558fe1e98270 msgr2=0x558fe1efd590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:42:40.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.739+0000 7ffab9795740 1 -- 172.21.15.90:0/2230570865 shutdown_connections 2024-04-02T14:42:40.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.739+0000 7ffab9795740 1 -- 172.21.15.90:0/2230570865 wait complete. 2024-04-02T14:42:40.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.740+0000 7ffab9795740 1 -- start start 2024-04-02T14:42:40.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.740+0000 7ffab9795740 1 Processor -- start 2024-04-02T14:42:40.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.741+0000 7ffab9795740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558fe1f04350 0x558fe1f155e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:40.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.742+0000 7ffab9795740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558fe1f05030 0x558fe1f15b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:40.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.742+0000 7ffab9795740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558fe1f0d6c0 0x558fe1f0e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:40.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.742+0000 7ffaafe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558fe1f04350 0x558fe1f155e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:40.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.742+0000 7ffab9795740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7ffab01b51f0 con 0x558fe1f04350 2024-04-02T14:42:40.741 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.742+0000 7ffaaf400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558fe1f05030 0x558fe1f15b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:40.741 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.742+0000 7ffaaf400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558fe1f05030 0x558fe1f15b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:47966/0 (socket says 172.21.15.90:47966) 2024-04-02T14:42:40.741 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.742+0000 7ffaaf400640 1 -- 172.21.15.90:0/1584162288 learned_addr learned my addr 172.21.15.90:0/1584162288 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:42:40.741 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.743+0000 7ffab4a00640 1 --2- 172.21.15.90:0/1584162288 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558fe1f0d6c0 0x558fe1f0e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:40.741 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.743+0000 7ffab9795740 1 -- 172.21.15.90:0/1584162288 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7ffab01b4b20 con 0x558fe1f0d6c0 2024-04-02T14:42:40.741 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.743+0000 7ffab9795740 1 -- 172.21.15.90:0/1584162288 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7ffab01b1d80 con 0x558fe1f05030 2024-04-02T14:42:40.741 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.743+0000 7ffab9795740 10 client.0 _ll_register_callbacks cb 0x7ffa9c00e480 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:42:40.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.744+0000 7ffaaf400640 1 -- 172.21.15.90:0/1584162288 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558fe1f0d6c0 msgr2=0x558fe1f0e6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:42:40.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.744+0000 7ffaaf400640 1 --2- 172.21.15.90:0/1584162288 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558fe1f0d6c0 0x558fe1f0e6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:40.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.744+0000 7ffaaf400640 1 -- 172.21.15.90:0/1584162288 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558fe1f04350 msgr2=0x558fe1f155e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:42:40.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.744+0000 7ffaaf400640 1 --2- 172.21.15.90:0/1584162288 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558fe1f04350 0x558fe1f155e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:40.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.744+0000 7ffaaf400640 1 -- 172.21.15.90:0/1584162288 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558fe1f0ec10 con 0x558fe1f05030 2024-04-02T14:42:40.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.744+0000 7ffaafe00640 1 --2- 172.21.15.90:0/1584162288 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558fe1f04350 0x558fe1f155e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:42:40.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.745+0000 7ffaaf400640 1 --2- 172.21.15.90:0/1584162288 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558fe1f05030 0x558fe1f15b20 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7ffa9c009530 tx=0x7ffa9c014050 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1072474a3b83cbe9 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:42:40.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.745+0000 7ffa99600640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:42:40.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.746+0000 7ffa99600640 1 -- 172.21.15.90:0/1584162288 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa9c03e030 con 0x558fe1f05030 2024-04-02T14:42:40.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.746+0000 7ffa99600640 1 -- 172.21.15.90:0/1584162288 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8550+0+0 (secure 0 0 0) 0x7ffa9c036070 con 0x558fe1f05030 2024-04-02T14:42:40.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.746+0000 7ffab9795740 1 -- 172.21.15.90:0/1584162288 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffaa80c77d0 con 0x558fe1f05030 2024-04-02T14:42:40.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.747+0000 7ffa99600640 1 -- 172.21.15.90:0/1584162288 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7ffa9c048050 con 0x558fe1f05030 2024-04-02T14:42:40.745 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.747+0000 7ffab9795740 10 client.27760 fetch_fsmap learned FSMap version 27 2024-04-02T14:42:40.745 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.747+0000 7ffab9795740 1 -- 172.21.15.90:0/1584162288 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x558fe1f844d0 con 0x558fe1f05030 2024-04-02T14:42:40.746 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.747+0000 7ffa99600640 1 -- 172.21.15.90:0/1584162288 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffaa80c77d0 con 0x558fe1f05030 2024-04-02T14:42:40.746 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.747+0000 7ffa99600640 1 -- 172.21.15.90:0/1584162288 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7ffa9c005040 con 0x558fe1f05030 2024-04-02T14:42:40.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.748+0000 7ffab9795740 10 client.27760 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:42:40.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.748+0000 7ffab9795740 10 client.27760 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:42:40.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.748+0000 7ffab9795740 20 client.27760 populate_metadata read hostname 'smithi090' 2024-04-02T14:42:40.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.748+0000 7ffab9795740 1 --2- 172.21.15.90:0/1584162288 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x558fe1f84c80 0x558fe1f870a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:40.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.748+0000 7ffab9795740 4 client.27760 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-02T14:42:40.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.748+0000 7ffab9795740 1 -- 172.21.15.90:0/1584162288 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x558fe1e6fba0 con 0x558fe1f84c80 2024-04-02T14:42:40.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.748+0000 7ffaafe00640 1 --2- 172.21.15.90:0/1584162288 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x558fe1f84c80 0x558fe1f870a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:40.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.749+0000 7ffaafe00640 1 --2- 172.21.15.90:0/1584162288 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x558fe1f84c80 0x558fe1f870a0 crc :-1 s=READY pgs=217 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a4ad1a28ae6eff50 server_cookie=93f9535ddbbd9680 in_seq=0 out_seq=0 2024-04-02T14:42:40.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.749+0000 7ffa99600640 10 client.27760 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:42:40.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.751+0000 7ffa99600640 1 -- 172.21.15.90:0/1584162288 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x558fe1e6fba0 con 0x558fe1f84c80 2024-04-02T14:42:40.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.751+0000 7ffa99600640 10 client.27760 handle_command_reply: tid=0 2024-04-02T14:42:40.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.751+0000 7ffab9795740 1 client.27760 shutdown 2024-04-02T14:42:40.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.751+0000 7ffab9795740 1 -- 172.21.15.90:0/1584162288 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558fe1f05030 msgr2=0x558fe1f15b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:40.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.751+0000 7ffab9795740 1 --2- 172.21.15.90:0/1584162288 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558fe1f05030 0x558fe1f15b20 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7ffa9c009530 tx=0x7ffa9c014050 comp rx=0 tx=0).stop 2024-04-02T14:42:40.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.752+0000 7ffab9795740 1 -- 172.21.15.90:0/1584162288 shutdown_connections 2024-04-02T14:42:40.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.752+0000 7ffab9795740 1 --2- 172.21.15.90:0/1584162288 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x558fe1f84c80 0x558fe1f870a0 crc :-1 s=READY pgs=217 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:40.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.752+0000 7ffab9795740 1 --2- 172.21.15.90:0/1584162288 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558fe1f0d6c0 0x558fe1f0e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:40.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.752+0000 7ffa99600640 0 client.27760 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:42:40.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.752+0000 7ffab9795740 1 --2- 172.21.15.90:0/1584162288 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558fe1f05030 0x558fe1f15b20 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:40.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.752+0000 7ffab9795740 1 --2- 172.21.15.90:0/1584162288 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558fe1f04350 0x558fe1f155e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:40.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.752+0000 7ffab9795740 1 -- 172.21.15.90:0/1584162288 >> 172.21.15.90:0/1584162288 conn(0x558fe1e98270 msgr2=0x558fe1efde20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:42:40.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.753+0000 7ffab9795740 1 -- 172.21.15.90:0/1584162288 shutdown_connections 2024-04-02T14:42:40.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.753+0000 7ffab9795740 1 -- 172.21.15.90:0/1584162288 wait complete. 2024-04-02T14:42:40.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.753+0000 7ffab9795740 20 client.27760 trim_cache size 0 max 16384 2024-04-02T14:42:40.751 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-02T14:42:40.752 INFO:teuthology.orchestra.run.smithi090.stdout: "return_code": 0, 2024-04-02T14:42:40.752 INFO:teuthology.orchestra.run.smithi090.stdout: "scrub_tag": "dfde56b1-ee62-4b22-a196-daf9f60b6d07", 2024-04-02T14:42:40.752 INFO:teuthology.orchestra.run.smithi090.stdout: "mode": "asynchronous" 2024-04-02T14:42:40.752 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-02T14:42:40.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.755+0000 7ffab6a00640 1 -- 172.21.15.90:0/3246734184 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7ffa940a3ea0 msgr2=0x7ffa940a6360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:40.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.755+0000 7ffab6a00640 1 --2- 172.21.15.90:0/3246734184 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7ffa940a3ea0 0x7ffa940a6360 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7ffab01108c0 tx=0x7ffa9c0127c0 comp rx=0 tx=0).stop 2024-04-02T14:42:40.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.755+0000 7ffab6a00640 1 -- 172.21.15.90:0/3246734184 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ffab010c110 msgr2=0x7ffab010fbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:40.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.755+0000 7ffab6a00640 1 --2- 172.21.15.90:0/3246734184 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ffab010c110 0x7ffab010fbd0 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7ffaa802c590 tx=0x7ffaa8004270 comp rx=0 tx=0).stop 2024-04-02T14:42:40.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.756+0000 7ffab6a00640 1 -- 172.21.15.90:0/3246734184 shutdown_connections 2024-04-02T14:42:40.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.756+0000 7ffab6a00640 1 --2- 172.21.15.90:0/3246734184 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7ffa940a3ea0 0x7ffa940a6360 secure :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7ffab01108c0 tx=0x7ffa9c0127c0 comp rx=0 tx=0).stop 2024-04-02T14:42:40.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.756+0000 7ffab6a00640 1 --2- 172.21.15.90:0/3246734184 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ffab010c110 0x7ffab010fbd0 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:40.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.756+0000 7ffab6a00640 1 --2- 172.21.15.90:0/3246734184 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffab0103660 0x7ffab010f690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:40.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.756+0000 7ffab6a00640 1 --2- 172.21.15.90:0/3246734184 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffab0102d40 0x7ffab010f150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:40.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.756+0000 7ffab6a00640 1 -- 172.21.15.90:0/3246734184 >> 172.21.15.90:0/3246734184 conn(0x7ffab0076d80 msgr2=0x7ffab00fd8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:42:40.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.756+0000 7ffab6a00640 1 -- 172.21.15.90:0/3246734184 shutdown_connections 2024-04-02T14:42:40.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:40.756+0000 7ffab6a00640 1 -- 172.21.15.90:0/3246734184 wait complete. 2024-04-02T14:42:40.786 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-02T14:42:40.787 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-02T14:42:41.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.085+0000 7f61d4e00640 1 -- 172.21.15.90:0/1223737847 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f61d0106870 msgr2=0x7f61d0106c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:41.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.085+0000 7f61d4e00640 1 --2- 172.21.15.90:0/1223737847 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f61d0106870 0x7f61d0106c50 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f61c0009930 tx=0x7f61c002bc30 comp rx=0 tx=0).stop 2024-04-02T14:42:41.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.086+0000 7f61d4e00640 1 -- 172.21.15.90:0/1223737847 shutdown_connections 2024-04-02T14:42:41.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.086+0000 7f61d4e00640 1 --2- 172.21.15.90:0/1223737847 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f61d0101220 0x7f61d010d730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:41.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.086+0000 7f61d4e00640 1 --2- 172.21.15.90:0/1223737847 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f61d0100860 0x7f61d0100ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:41.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.086+0000 7f61d4e00640 1 --2- 172.21.15.90:0/1223737847 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f61d0106870 0x7f61d0106c50 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:41.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.086+0000 7f61d4e00640 1 -- 172.21.15.90:0/1223737847 >> 172.21.15.90:0/1223737847 conn(0x7f61d00fc350 msgr2=0x7f61d00fe770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:42:41.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.086+0000 7f61d4e00640 1 -- 172.21.15.90:0/1223737847 shutdown_connections 2024-04-02T14:42:41.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.086+0000 7f61d4e00640 1 -- 172.21.15.90:0/1223737847 wait complete. 2024-04-02T14:42:41.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.087+0000 7f61d4e00640 1 Processor -- start 2024-04-02T14:42:41.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.087+0000 7f61d4e00640 1 -- start start 2024-04-02T14:42:41.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.088+0000 7f61d4e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f61d0100860 0x7f61d019a4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:41.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.088+0000 7f61d4e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f61d0101220 0x7f61d019a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:41.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.088+0000 7f61d4e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f61d0106870 0x7f61d019ed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:41.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.088+0000 7f61d4e00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f61d0113fa0 con 0x7f61d0101220 2024-04-02T14:42:41.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.088+0000 7f61d4e00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f61d0113e20 con 0x7f61d0100860 2024-04-02T14:42:41.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.088+0000 7f61d4e00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f61d0114120 con 0x7f61d0106870 2024-04-02T14:42:41.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.088+0000 7f61cf400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f61d0100860 0x7f61d019a4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:41.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.088+0000 7f61cf400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f61d0100860 0x7f61d019a4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:55592/0 (socket says 172.21.15.90:55592) 2024-04-02T14:42:41.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.088+0000 7f61cf400640 1 -- 172.21.15.90:0/3931208149 learned_addr learned my addr 172.21.15.90:0/3931208149 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:42:41.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.089+0000 7f61cfe00640 1 --2- 172.21.15.90:0/3931208149 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f61d0106870 0x7f61d019ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:41.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.089+0000 7f61cea00640 1 --2- 172.21.15.90:0/3931208149 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f61d0101220 0x7f61d019a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:41.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.089+0000 7f61cf400640 1 -- 172.21.15.90:0/3931208149 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f61d0106870 msgr2=0x7f61d019ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:41.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.089+0000 7f61cf400640 1 --2- 172.21.15.90:0/3931208149 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f61d0106870 0x7f61d019ed10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:41.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.089+0000 7f61cf400640 1 -- 172.21.15.90:0/3931208149 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f61d0101220 msgr2=0x7f61d019a9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:41.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.089+0000 7f61cf400640 1 --2- 172.21.15.90:0/3931208149 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f61d0101220 0x7f61d019a9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:41.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.089+0000 7f61cf400640 1 -- 172.21.15.90:0/3931208149 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61d019f440 con 0x7f61d0100860 2024-04-02T14:42:41.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.089+0000 7f61cf400640 1 --2- 172.21.15.90:0/3931208149 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f61d0100860 0x7f61d019a4a0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f61c0009900 tx=0x7f61c0004360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:42:41.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.090+0000 7f61c7e00640 1 -- 172.21.15.90:0/3931208149 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61c0043020 con 0x7f61d0100860 2024-04-02T14:42:41.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.090+0000 7f61c7e00640 1 -- 172.21.15.90:0/3931208149 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f61c00054e0 con 0x7f61d0100860 2024-04-02T14:42:41.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.090+0000 7f61c7e00640 1 -- 172.21.15.90:0/3931208149 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61c0003bb0 con 0x7f61d0100860 2024-04-02T14:42:41.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.090+0000 7f61d4e00640 1 -- 172.21.15.90:0/3931208149 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61d019f6d0 con 0x7f61d0100860 2024-04-02T14:42:41.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.090+0000 7f61d4e00640 1 -- 172.21.15.90:0/3931208149 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61d01a6fc0 con 0x7f61d0100860 2024-04-02T14:42:41.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.094+0000 7f61c7e00640 1 -- 172.21.15.90:0/3931208149 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f61c0046020 con 0x7f61d0100860 2024-04-02T14:42:41.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.095+0000 7f61d921f740 1 Processor -- start 2024-04-02T14:42:41.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.095+0000 7f61d921f740 1 -- start start 2024-04-02T14:42:41.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.096+0000 7f61d921f740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a4dee69960 0x55a4dee69d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:41.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.096+0000 7f61d921f740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a4dee6a280 0x55a4dee72920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:41.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.096+0000 7f61d921f740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a4dee72e60 0x55a4dee75250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:41.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.096+0000 7f61d921f740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55a4dedcd770 con 0x55a4dee69960 2024-04-02T14:42:41.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.096+0000 7f61d921f740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55a4ded75700 con 0x55a4dee72e60 2024-04-02T14:42:41.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.096+0000 7f61d921f740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55a4dedc9720 con 0x55a4dee6a280 2024-04-02T14:42:41.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.096+0000 7f61cea00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a4dee72e60 0x55a4dee75250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:41.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.096+0000 7f61cea00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a4dee72e60 0x55a4dee75250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:55594/0 (socket says 172.21.15.90:55594) 2024-04-02T14:42:41.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.096+0000 7f61cea00640 1 -- 172.21.15.90:0/445186584 learned_addr learned my addr 172.21.15.90:0/445186584 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:42:41.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.096+0000 7f61cfe00640 1 --2- 172.21.15.90:0/445186584 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a4dee69960 0x55a4dee69d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:41.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.096+0000 7f61c7e00640 1 --2- 172.21.15.90:0/3931208149 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f61b40a3e10 0x7f61b40a62d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:41.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.097+0000 7f61c7e00640 1 -- 172.21.15.90:0/3931208149 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8550+0+0 (secure 0 0 0) 0x7f61c00f2890 con 0x7f61d0100860 2024-04-02T14:42:41.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.097+0000 7f61cea00640 1 -- 172.21.15.90:0/445186584 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a4dee6a280 msgr2=0x55a4dee72920 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-02T14:42:41.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.097+0000 7f61cea00640 1 --2- 172.21.15.90:0/445186584 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a4dee6a280 0x55a4dee72920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:41.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.097+0000 7f61cea00640 1 -- 172.21.15.90:0/445186584 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a4dee69960 msgr2=0x55a4dee69d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:42:41.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.097+0000 7f61cea00640 1 --2- 172.21.15.90:0/445186584 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a4dee69960 0x55a4dee69d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:41.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.097+0000 7f61cea00640 1 -- 172.21.15.90:0/445186584 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a4debab7d0 con 0x55a4dee72e60 2024-04-02T14:42:41.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.098+0000 7f61cfe00640 1 --2- 172.21.15.90:0/3931208149 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f61b40a3e10 0x7f61b40a62d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:41.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.098+0000 7f61cea00640 1 --2- 172.21.15.90:0/445186584 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a4dee72e60 0x55a4dee75250 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f61bc009c10 tx=0x7f61bc0117b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9dbe230764e56404 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:42:41.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.098+0000 7f61c4c00640 1 -- 172.21.15.90:0/445186584 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61bc012450 con 0x55a4dee72e60 2024-04-02T14:42:41.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.098+0000 7f61c4c00640 1 -- 172.21.15.90:0/445186584 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f61bc012ad0 con 0x55a4dee72e60 2024-04-02T14:42:41.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.098+0000 7f61c4c00640 1 -- 172.21.15.90:0/445186584 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61bc00fbd0 con 0x55a4dee72e60 2024-04-02T14:42:41.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.099+0000 7f61d921f740 1 -- 172.21.15.90:0/445186584 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a4dee72e60 msgr2=0x55a4dee75250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:41.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.099+0000 7f61d921f740 1 --2- 172.21.15.90:0/445186584 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a4dee72e60 0x55a4dee75250 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f61bc009c10 tx=0x7f61bc0117b0 comp rx=0 tx=0).stop 2024-04-02T14:42:41.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.099+0000 7f61cfe00640 1 --2- 172.21.15.90:0/3931208149 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f61b40a3e10 0x7f61b40a62d0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f61c8005fd0 tx=0x7f61c8004ab0 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:42:41.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.100+0000 7f61d921f740 1 -- 172.21.15.90:0/445186584 shutdown_connections 2024-04-02T14:42:41.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.100+0000 7f61d921f740 1 --2- 172.21.15.90:0/445186584 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a4dee72e60 0x55a4dee75250 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:41.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.100+0000 7f61d921f740 1 --2- 172.21.15.90:0/445186584 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a4dee6a280 0x55a4dee72920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:41.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.100+0000 7f61d921f740 1 --2- 172.21.15.90:0/445186584 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a4dee69960 0x55a4dee69d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:41.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.100+0000 7f61d921f740 1 -- 172.21.15.90:0/445186584 >> 172.21.15.90:0/445186584 conn(0x55a4dee64090 msgr2=0x55a4dee64470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:42:41.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.100+0000 7f61d921f740 1 -- 172.21.15.90:0/445186584 shutdown_connections 2024-04-02T14:42:41.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.100+0000 7f61d921f740 1 -- 172.21.15.90:0/445186584 wait complete. 2024-04-02T14:42:41.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.100+0000 7f61d921f740 1 -- start start 2024-04-02T14:42:41.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.101+0000 7f61d921f740 1 Processor -- start 2024-04-02T14:42:41.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.101+0000 7f61d921f740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a4dee69960 0x55a4dee6ff50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:41.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.101+0000 7f61d921f740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a4dee6a280 0x55a4dee70490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:41.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.101+0000 7f61d921f740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a4dee72e60 0x55a4dee709d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:41.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.101+0000 7f61d921f740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55a4dee75c50 con 0x55a4dee72e60 2024-04-02T14:42:41.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.101+0000 7f61d921f740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55a4ded88ac0 con 0x55a4dee6a280 2024-04-02T14:42:41.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.101+0000 7f61d921f740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55a4dee75dd0 con 0x55a4dee69960 2024-04-02T14:42:41.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.101+0000 7f61d921f740 10 client.0 _ll_register_callbacks cb 0x55a4ded8b180 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:42:41.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.101+0000 7f61cea00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a4dee6a280 0x55a4dee70490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:41.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.101+0000 7f61cea00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a4dee6a280 0x55a4dee70490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:55610/0 (socket says 172.21.15.90:55610) 2024-04-02T14:42:41.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.101+0000 7f61cea00640 1 -- 172.21.15.90:0/1945934567 learned_addr learned my addr 172.21.15.90:0/1945934567 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:42:41.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.101+0000 7f61cfe00640 1 --2- 172.21.15.90:0/1945934567 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a4dee72e60 0x55a4dee709d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:41.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.101+0000 7f61cf400640 1 --2- 172.21.15.90:0/1945934567 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a4dee69960 0x55a4dee6ff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:41.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.102+0000 7f61cea00640 1 -- 172.21.15.90:0/1945934567 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a4dee69960 msgr2=0x55a4dee6ff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:42:41.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.102+0000 7f61cea00640 1 --2- 172.21.15.90:0/1945934567 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a4dee69960 0x55a4dee6ff50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:41.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.102+0000 7f61cea00640 1 -- 172.21.15.90:0/1945934567 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a4dee72e60 msgr2=0x55a4dee709d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:42:41.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.102+0000 7f61cea00640 1 --2- 172.21.15.90:0/1945934567 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a4dee72e60 0x55a4dee709d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:41.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.102+0000 7f61cea00640 1 -- 172.21.15.90:0/1945934567 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a4deae13f0 con 0x55a4dee6a280 2024-04-02T14:42:41.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.102+0000 7f61cea00640 1 --2- 172.21.15.90:0/1945934567 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a4dee6a280 0x55a4dee70490 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f61bc00a010 tx=0x7f61bc012eb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=27cf29f004589b82 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:42:41.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.102+0000 7f61b9600640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:42:41.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.102+0000 7f61b9600640 1 -- 172.21.15.90:0/1945934567 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61bc028070 con 0x55a4dee6a280 2024-04-02T14:42:41.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.102+0000 7f61d921f740 1 -- 172.21.15.90:0/1945934567 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f61bc012ad0 con 0x55a4dee6a280 2024-04-02T14:42:41.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.102+0000 7f61b9600640 1 -- 172.21.15.90:0/1945934567 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8550+0+0 (secure 0 0 0) 0x7f61bc012860 con 0x55a4dee6a280 2024-04-02T14:42:41.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.103+0000 7f61b9600640 1 -- 172.21.15.90:0/1945934567 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f61bc00f950 con 0x55a4dee6a280 2024-04-02T14:42:41.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.103+0000 7f61d921f740 10 client.27929 fetch_fsmap learned FSMap version 27 2024-04-02T14:42:41.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.103+0000 7f61d921f740 1 -- 172.21.15.90:0/1945934567 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55a4deaa10f0 con 0x55a4dee6a280 2024-04-02T14:42:41.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.103+0000 7f61b9600640 1 -- 172.21.15.90:0/1945934567 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f61bc012ad0 con 0x55a4dee6a280 2024-04-02T14:42:41.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.103+0000 7f61b9600640 1 -- 172.21.15.90:0/1945934567 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f61bc0046c0 con 0x55a4dee6a280 2024-04-02T14:42:41.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.103+0000 7f61d921f740 10 client.27929 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:42:41.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.103+0000 7f61d921f740 10 client.27929 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:42:41.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.103+0000 7f61d921f740 20 client.27929 populate_metadata read hostname 'smithi090' 2024-04-02T14:42:41.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.103+0000 7f61d921f740 1 --2- 172.21.15.90:0/1945934567 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55a4dee82e90 0x55a4dee85280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:41.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.103+0000 7f61d921f740 4 client.27929 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:42:41.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.103+0000 7f61d921f740 1 -- 172.21.15.90:0/1945934567 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a4ded6e630 con 0x55a4dee82e90 2024-04-02T14:42:41.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.103+0000 7f61cf400640 1 --2- 172.21.15.90:0/1945934567 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55a4dee82e90 0x55a4dee85280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:41.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.104+0000 7f61cf400640 1 --2- 172.21.15.90:0/1945934567 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55a4dee82e90 0x55a4dee85280 crc :-1 s=READY pgs=218 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=55ea61d745d2ed85 server_cookie=ccdf046fd7bffb03 in_seq=0 out_seq=0 2024-04-02T14:42:41.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.104+0000 7f61b9600640 10 client.27929 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:42:41.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.104+0000 7f61b9600640 1 -- 172.21.15.90:0/1945934567 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a4ded6e630 con 0x55a4dee82e90 2024-04-02T14:42:41.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.104+0000 7f61b9600640 10 client.27929 handle_command_reply: tid=0 2024-04-02T14:42:41.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.105+0000 7f61d921f740 1 client.27929 shutdown 2024-04-02T14:42:41.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.105+0000 7f61d921f740 1 -- 172.21.15.90:0/1945934567 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a4dee6a280 msgr2=0x55a4dee70490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:41.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.105+0000 7f61d921f740 1 --2- 172.21.15.90:0/1945934567 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a4dee6a280 0x55a4dee70490 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f61bc00a010 tx=0x7f61bc012eb0 comp rx=0 tx=0).stop 2024-04-02T14:42:41.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.105+0000 7f61d921f740 1 -- 172.21.15.90:0/1945934567 shutdown_connections 2024-04-02T14:42:41.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.105+0000 7f61d921f740 1 --2- 172.21.15.90:0/1945934567 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55a4dee82e90 0x55a4dee85280 crc :-1 s=READY pgs=218 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:41.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.105+0000 7f61d921f740 1 --2- 172.21.15.90:0/1945934567 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a4dee72e60 0x55a4dee709d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:41.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.105+0000 7f61d921f740 1 --2- 172.21.15.90:0/1945934567 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a4dee6a280 0x55a4dee70490 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:41.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.105+0000 7f61b9600640 0 client.27929 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:42:41.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.105+0000 7f61d921f740 1 --2- 172.21.15.90:0/1945934567 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a4dee69960 0x55a4dee6ff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:41.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.105+0000 7f61d921f740 1 -- 172.21.15.90:0/1945934567 >> 172.21.15.90:0/1945934567 conn(0x55a4dee64090 msgr2=0x55a4dee65de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:42:41.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.105+0000 7f61d921f740 1 -- 172.21.15.90:0/1945934567 shutdown_connections 2024-04-02T14:42:41.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.105+0000 7f61d921f740 1 -- 172.21.15.90:0/1945934567 wait complete. 2024-04-02T14:42:41.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.105+0000 7f61d921f740 20 client.27929 trim_cache size 0 max 16384 2024-04-02T14:42:41.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.139+0000 7f61d921f740 1 Processor -- start 2024-04-02T14:42:41.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.139+0000 7f61d921f740 1 -- start start 2024-04-02T14:42:41.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.140+0000 7f61d921f740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a4dee69960 0x55a4dee72680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:41.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.140+0000 7f61d921f740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a4dee6a280 0x55a4dee7eb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:41.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.140+0000 7f61d921f740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a4dee72e60 0x55a4dee8a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:41.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.140+0000 7f61d921f740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f61d01a7190 con 0x55a4dee69960 2024-04-02T14:42:41.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.140+0000 7f61d921f740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f61d01a8ca0 con 0x55a4dee6a280 2024-04-02T14:42:41.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.140+0000 7f61d921f740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f61d01a7830 con 0x55a4dee72e60 2024-04-02T14:42:41.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.140+0000 7f61cfe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a4dee72e60 0x55a4dee8a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:41.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.140+0000 7f61cfe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a4dee72e60 0x55a4dee8a8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:47994/0 (socket says 172.21.15.90:47994) 2024-04-02T14:42:41.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.140+0000 7f61cfe00640 1 -- 172.21.15.90:0/4106921290 learned_addr learned my addr 172.21.15.90:0/4106921290 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:42:41.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.141+0000 7f61cea00640 1 --2- 172.21.15.90:0/4106921290 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a4dee6a280 0x55a4dee7eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:41.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.141+0000 7f61cf400640 1 --2- 172.21.15.90:0/4106921290 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a4dee69960 0x55a4dee72680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:41.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.141+0000 7f61cea00640 1 -- 172.21.15.90:0/4106921290 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a4dee72e60 msgr2=0x55a4dee8a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:42:41.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.141+0000 7f61cea00640 1 --2- 172.21.15.90:0/4106921290 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a4dee72e60 0x55a4dee8a8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:41.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.141+0000 7f61cea00640 1 -- 172.21.15.90:0/4106921290 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a4dee69960 msgr2=0x55a4dee72680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:42:41.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.141+0000 7f61cea00640 1 --2- 172.21.15.90:0/4106921290 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a4dee69960 0x55a4dee72680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:41.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.141+0000 7f61cea00640 1 -- 172.21.15.90:0/4106921290 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a4dee6e500 con 0x55a4dee6a280 2024-04-02T14:42:41.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.142+0000 7f61cea00640 1 --2- 172.21.15.90:0/4106921290 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a4dee6a280 0x55a4dee7eb40 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f61bc004a60 tx=0x7f61bc0117b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e3ed712ee3913edf server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:42:41.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.143+0000 7f61bbe00640 1 -- 172.21.15.90:0/4106921290 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61bc028040 con 0x55a4dee6a280 2024-04-02T14:42:41.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.143+0000 7f61bbe00640 1 -- 172.21.15.90:0/4106921290 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f61bc0096c0 con 0x55a4dee6a280 2024-04-02T14:42:41.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.143+0000 7f61bbe00640 1 -- 172.21.15.90:0/4106921290 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61bc028040 con 0x55a4dee6a280 2024-04-02T14:42:41.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.144+0000 7f61d921f740 1 -- 172.21.15.90:0/4106921290 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a4dee6a280 msgr2=0x55a4dee7eb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:41.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.144+0000 7f61d921f740 1 --2- 172.21.15.90:0/4106921290 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a4dee6a280 0x55a4dee7eb40 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f61bc004a60 tx=0x7f61bc0117b0 comp rx=0 tx=0).stop 2024-04-02T14:42:41.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.144+0000 7f61d921f740 1 -- 172.21.15.90:0/4106921290 shutdown_connections 2024-04-02T14:42:41.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.144+0000 7f61d921f740 1 --2- 172.21.15.90:0/4106921290 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a4dee72e60 0x55a4dee8a8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:41.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.144+0000 7f61d921f740 1 --2- 172.21.15.90:0/4106921290 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a4dee6a280 0x55a4dee7eb40 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:41.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.144+0000 7f61d921f740 1 --2- 172.21.15.90:0/4106921290 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a4dee69960 0x55a4dee72680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:41.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.144+0000 7f61d921f740 1 -- 172.21.15.90:0/4106921290 >> 172.21.15.90:0/4106921290 conn(0x55a4dee64090 msgr2=0x55a4dee63310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:42:41.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.144+0000 7f61d921f740 1 -- 172.21.15.90:0/4106921290 shutdown_connections 2024-04-02T14:42:41.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.145+0000 7f61d921f740 1 -- 172.21.15.90:0/4106921290 wait complete. 2024-04-02T14:42:41.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.145+0000 7f61d921f740 1 -- start start 2024-04-02T14:42:41.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.145+0000 7f61d921f740 1 Processor -- start 2024-04-02T14:42:41.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.147+0000 7f61d921f740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a4dee69960 0x55a4dee6cb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:41.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.147+0000 7f61d921f740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a4dee6a280 0x55a4dee6d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:41.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.147+0000 7f61d921f740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a4dee72e60 0x55a4dee6d610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:41.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.147+0000 7f61cf400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a4dee69960 0x55a4dee6cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:41.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.148+0000 7f61cf400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a4dee69960 0x55a4dee6cb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:59464/0 (socket says 172.21.15.90:59464) 2024-04-02T14:42:41.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.148+0000 7f61cfe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a4dee72e60 0x55a4dee6d610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:41.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.148+0000 7f61cfe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a4dee72e60 0x55a4dee6d610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:47996/0 (socket says 172.21.15.90:47996) 2024-04-02T14:42:41.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.148+0000 7f61d921f740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f61d01a85d0 con 0x55a4dee69960 2024-04-02T14:42:41.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.148+0000 7f61cf400640 1 -- 172.21.15.90:0/176271187 learned_addr learned my addr 172.21.15.90:0/176271187 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:42:41.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.148+0000 7f61cea00640 1 --2- 172.21.15.90:0/176271187 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a4dee6a280 0x55a4dee6d0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:41.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.148+0000 7f61d921f740 1 -- 172.21.15.90:0/176271187 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f61d01a7f00 con 0x55a4dee6a280 2024-04-02T14:42:41.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.148+0000 7f61d921f740 1 -- 172.21.15.90:0/176271187 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f61d0071cf0 con 0x55a4dee72e60 2024-04-02T14:42:41.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.148+0000 7f61d921f740 10 client.0 _ll_register_callbacks cb 0x7f61bc009a90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:42:41.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.149+0000 7f61cfe00640 1 -- 172.21.15.90:0/176271187 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a4dee6a280 msgr2=0x55a4dee6d0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:42:41.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.149+0000 7f61cfe00640 1 --2- 172.21.15.90:0/176271187 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a4dee6a280 0x55a4dee6d0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:41.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.149+0000 7f61cfe00640 1 -- 172.21.15.90:0/176271187 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a4dee69960 msgr2=0x55a4dee6cb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:42:41.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.149+0000 7f61cfe00640 1 --2- 172.21.15.90:0/176271187 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a4dee69960 0x55a4dee6cb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:41.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.149+0000 7f61cfe00640 1 -- 172.21.15.90:0/176271187 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a4deef0960 con 0x55a4dee72e60 2024-04-02T14:42:41.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.149+0000 7f61cea00640 1 --2- 172.21.15.90:0/176271187 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a4dee6a280 0x55a4dee6d0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:42:41.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.149+0000 7f61b9600640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:42:41.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.149+0000 7f61cfe00640 1 --2- 172.21.15.90:0/176271187 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a4dee72e60 0x55a4dee6d610 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f61c8004410 tx=0x7f61c8004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d1833f6e39690731 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:42:41.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.153+0000 7f61b9600640 1 -- 172.21.15.90:0/176271187 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61c8018070 con 0x55a4dee72e60 2024-04-02T14:42:41.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.153+0000 7f61b9600640 1 -- 172.21.15.90:0/176271187 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8550+0+0 (secure 0 0 0) 0x7f61c8027070 con 0x55a4dee72e60 2024-04-02T14:42:41.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.153+0000 7f61d921f740 1 -- 172.21.15.90:0/176271187 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f61bc0096c0 con 0x55a4dee72e60 2024-04-02T14:42:41.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.155+0000 7f61b9600640 1 -- 172.21.15.90:0/176271187 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f61c8018070 con 0x55a4dee72e60 2024-04-02T14:42:41.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.155+0000 7f61d921f740 10 client.27775 fetch_fsmap learned FSMap version 27 2024-04-02T14:42:41.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.155+0000 7f61d921f740 1 -- 172.21.15.90:0/176271187 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55a4deef1c70 con 0x55a4dee72e60 2024-04-02T14:42:41.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.156+0000 7f61b9600640 1 -- 172.21.15.90:0/176271187 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f61bc0096c0 con 0x55a4dee72e60 2024-04-02T14:42:41.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.156+0000 7f61b9600640 1 -- 172.21.15.90:0/176271187 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f61c80339e0 con 0x55a4dee72e60 2024-04-02T14:42:41.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.157+0000 7f61d921f740 10 client.27775 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:42:41.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.157+0000 7f61d921f740 10 client.27775 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:42:41.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.157+0000 7f61d921f740 20 client.27775 populate_metadata read hostname 'smithi090' 2024-04-02T14:42:41.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.157+0000 7f61d921f740 1 --2- 172.21.15.90:0/176271187 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55a4deef23f0 0x55a4deef4810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:41.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.157+0000 7f61d921f740 4 client.27775 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-02T14:42:41.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.157+0000 7f61cf400640 1 --2- 172.21.15.90:0/176271187 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55a4deef23f0 0x55a4deef4810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:41.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.157+0000 7f61d921f740 1 -- 172.21.15.90:0/176271187 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55a4deddacb0 con 0x55a4deef23f0 2024-04-02T14:42:41.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.158+0000 7f61cf400640 1 --2- 172.21.15.90:0/176271187 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55a4deef23f0 0x55a4deef4810 crc :-1 s=READY pgs=219 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9e5888249344b7aa server_cookie=2a79a20405119006 in_seq=0 out_seq=0 2024-04-02T14:42:41.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.158+0000 7f61b9600640 10 client.27775 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:42:41.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.158+0000 7f61b9600640 1 -- 172.21.15.90:0/176271187 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55a4deddacb0 con 0x55a4deef23f0 2024-04-02T14:42:41.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.158+0000 7f61b9600640 10 client.27775 handle_command_reply: tid=0 2024-04-02T14:42:41.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.158+0000 7f61d921f740 1 client.27775 shutdown 2024-04-02T14:42:41.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.159+0000 7f61d921f740 1 -- 172.21.15.90:0/176271187 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a4dee72e60 msgr2=0x55a4dee6d610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:41.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.159+0000 7f61d921f740 1 --2- 172.21.15.90:0/176271187 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a4dee72e60 0x55a4dee6d610 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f61c8004410 tx=0x7f61c8004a40 comp rx=0 tx=0).stop 2024-04-02T14:42:41.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.159+0000 7f61d921f740 1 -- 172.21.15.90:0/176271187 shutdown_connections 2024-04-02T14:42:41.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.159+0000 7f61d921f740 1 --2- 172.21.15.90:0/176271187 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55a4deef23f0 0x55a4deef4810 crc :-1 s=READY pgs=219 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:41.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.159+0000 7f61d921f740 1 --2- 172.21.15.90:0/176271187 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a4dee72e60 0x55a4dee6d610 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:41.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.159+0000 7f61d921f740 1 --2- 172.21.15.90:0/176271187 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a4dee6a280 0x55a4dee6d0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:41.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.159+0000 7f61d921f740 1 --2- 172.21.15.90:0/176271187 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a4dee69960 0x55a4dee6cb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:41.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.159+0000 7f61d921f740 1 -- 172.21.15.90:0/176271187 >> 172.21.15.90:0/176271187 conn(0x55a4dee64090 msgr2=0x55a4dee65430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:42:41.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.159+0000 7f61b9600640 0 client.27775 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:42:41.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.160+0000 7f61d921f740 1 -- 172.21.15.90:0/176271187 shutdown_connections 2024-04-02T14:42:41.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.160+0000 7f61d921f740 1 -- 172.21.15.90:0/176271187 wait complete. 2024-04-02T14:42:41.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.160+0000 7f61d921f740 20 client.27775 trim_cache size 0 max 16384 2024-04-02T14:42:41.159 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-02T14:42:41.159 INFO:teuthology.orchestra.run.smithi090.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-02T14:42:41.159 INFO:teuthology.orchestra.run.smithi090.stdout: "scrubs": { 2024-04-02T14:42:41.159 INFO:teuthology.orchestra.run.smithi090.stdout: "dfde56b1-ee62-4b22-a196-daf9f60b6d07": { 2024-04-02T14:42:41.159 INFO:teuthology.orchestra.run.smithi090.stdout: "path": "/", 2024-04-02T14:42:41.159 INFO:teuthology.orchestra.run.smithi090.stdout: "tag": "dfde56b1-ee62-4b22-a196-daf9f60b6d07", 2024-04-02T14:42:41.159 INFO:teuthology.orchestra.run.smithi090.stdout: "options": "recursive,force" 2024-04-02T14:42:41.160 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:42:41.160 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:42:41.160 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-02T14:42:41.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.161+0000 7f61d4e00640 1 -- 172.21.15.90:0/3931208149 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f61b40a3e10 msgr2=0x7f61b40a62d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:41.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.161+0000 7f61d4e00640 1 --2- 172.21.15.90:0/3931208149 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f61b40a3e10 0x7f61b40a62d0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f61c8005fd0 tx=0x7f61c8004ab0 comp rx=0 tx=0).stop 2024-04-02T14:42:41.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.161+0000 7f61d4e00640 1 -- 172.21.15.90:0/3931208149 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f61d0100860 msgr2=0x7f61d019a4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:41.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.161+0000 7f61d4e00640 1 --2- 172.21.15.90:0/3931208149 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f61d0100860 0x7f61d019a4a0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f61c0009900 tx=0x7f61c0004360 comp rx=0 tx=0).stop 2024-04-02T14:42:41.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.161+0000 7f61d4e00640 1 -- 172.21.15.90:0/3931208149 shutdown_connections 2024-04-02T14:42:41.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.161+0000 7f61d4e00640 1 --2- 172.21.15.90:0/3931208149 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f61b40a3e10 0x7f61b40a62d0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:41.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.161+0000 7f61d4e00640 1 --2- 172.21.15.90:0/3931208149 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f61d0106870 0x7f61d019ed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:41.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.161+0000 7f61d4e00640 1 --2- 172.21.15.90:0/3931208149 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f61d0101220 0x7f61d019a9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:41.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.161+0000 7f61d4e00640 1 --2- 172.21.15.90:0/3931208149 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f61d0100860 0x7f61d019a4a0 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:41.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.161+0000 7f61d4e00640 1 -- 172.21.15.90:0/3931208149 >> 172.21.15.90:0/3931208149 conn(0x7f61d00fc350 msgr2=0x7f61d00fcba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:42:41.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.161+0000 7f61d4e00640 1 -- 172.21.15.90:0/3931208149 shutdown_connections 2024-04-02T14:42:41.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:42:41.162+0000 7f61d4e00640 1 -- 172.21.15.90:0/3931208149 wait complete. 2024-04-02T14:42:41.187 INFO:tasks.cephfs.filesystem:scrub status for tag:dfde56b1-ee62-4b22-a196-daf9f60b6d07 - {'path': '/', 'tag': 'dfde56b1-ee62-4b22-a196-daf9f60b6d07', 'options': 'recursive,force'} 2024-04-02T14:42:41.433 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:42:41 smithi177 ceph-mon[36431]: scrub queued for path: / 2024-04-02T14:42:41.433 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:42:41 smithi177 ceph-mon[36431]: scrub summary: idle+waiting paths [/] 2024-04-02T14:42:41.433 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:42:41 smithi177 ceph-mon[36431]: scrub summary: active paths [/] 2024-04-02T14:42:41.435 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:42:41 smithi090 ceph-mon[31674]: scrub queued for path: / 2024-04-02T14:42:41.435 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:42:41 smithi090 ceph-mon[31674]: scrub summary: idle+waiting paths [/] 2024-04-02T14:42:41.435 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:42:41 smithi090 ceph-mon[31674]: scrub summary: active paths [/] 2024-04-02T14:42:41.607 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:42:41 smithi092 ceph-mon[36497]: scrub queued for path: / 2024-04-02T14:42:41.607 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:42:41 smithi092 ceph-mon[36497]: scrub summary: idle+waiting paths [/] 2024-04-02T14:42:41.607 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:42:41 smithi092 ceph-mon[36497]: scrub summary: active paths [/] 2024-04-02T14:42:41.828 INFO:teuthology.orchestra.run.smithi090.stderr:progress: 430.0 s, 2281.7 tps, lat 43.932 ms stddev 55.195 2024-04-02T14:42:42.433 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:42:42 smithi177 ceph-mon[36431]: pgmap v239: 97 pgs: 97 active+clean; 4.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 20 MiB/s wr, 1.24k op/s 2024-04-02T14:42:42.434 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:42:42 smithi177 ceph-mon[36431]: scrub summary: idle+waiting paths [/] 2024-04-02T14:42:42.434 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:42:42 smithi177 ceph-mon[36431]: scrub summary: idle 2024-04-02T14:42:42.435 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:42:42 smithi090 ceph-mon[31674]: pgmap v239: 97 pgs: 97 active+clean; 4.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 20 MiB/s wr, 1.24k op/s 2024-04-02T14:42:42.435 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:42:42 smithi090 ceph-mon[31674]: scrub summary: idle+waiting paths [/] 2024-04-02T14:42:42.435 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:42:42 smithi090 ceph-mon[31674]: scrub summary: idle 2024-04-02T14:42:42.607 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:42:42 smithi092 ceph-mon[36497]: pgmap v239: 97 pgs: 97 active+clean; 4.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 20 MiB/s wr, 1.24k op/s 2024-04-02T14:42:42.607 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:42:42 smithi092 ceph-mon[36497]: scrub summary: idle+waiting paths [/] 2024-04-02T14:42:42.607 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:42:42 smithi092 ceph-mon[36497]: scrub summary: idle 2024-04-02T14:42:44.396 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.398+0000 7f142e000640 1 -- 172.21.15.90:0/3225443842 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1428101ab0 msgr2=0x7f1428101f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:44.396 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.398+0000 7f142e000640 1 --2- 172.21.15.90:0/3225443842 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1428101ab0 0x7f1428101f30 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f1410009900 tx=0x7f141002bc30 comp rx=0 tx=0).stop 2024-04-02T14:42:44.396 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.398+0000 7f142e000640 1 -- 172.21.15.90:0/3225443842 shutdown_connections 2024-04-02T14:42:44.396 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.398+0000 7f142e000640 1 --2- 172.21.15.90:0/3225443842 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1428102470 0x7f142810e980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:44.397 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.398+0000 7f142e000640 1 --2- 172.21.15.90:0/3225443842 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1428101ab0 0x7f1428101f30 secure :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f1410009900 tx=0x7f141002bc30 comp rx=0 tx=0).stop 2024-04-02T14:42:44.397 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.398+0000 7f142e000640 1 --2- 172.21.15.90:0/3225443842 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1428107ac0 0x7f1428107ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:44.397 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.398+0000 7f142e000640 1 -- 172.21.15.90:0/3225443842 >> 172.21.15.90:0/3225443842 conn(0x7f14280fd5c0 msgr2=0x7f14280ff9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:42:44.397 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.399+0000 7f142e000640 1 -- 172.21.15.90:0/3225443842 shutdown_connections 2024-04-02T14:42:44.397 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.399+0000 7f142e000640 1 -- 172.21.15.90:0/3225443842 wait complete. 2024-04-02T14:42:44.398 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.400+0000 7f142e000640 1 Processor -- start 2024-04-02T14:42:44.398 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.401+0000 7f142e000640 1 -- start start 2024-04-02T14:42:44.398 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.401+0000 7f142e000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1428102470 0x7f142819a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:44.399 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.401+0000 7f142e000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1428107ac0 0x7f142819ac00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:44.399 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.401+0000 7f142e000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f142819ef80 0x7f142819f430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:44.399 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.401+0000 7f142e000640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f1428111ed0 con 0x7f1428102470 2024-04-02T14:42:44.399 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.401+0000 7f142e000640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f1428111d50 con 0x7f1428107ac0 2024-04-02T14:42:44.399 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.401+0000 7f142e000640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f1428112050 con 0x7f142819ef80 2024-04-02T14:42:44.400 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.402+0000 7f1427e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f142819ef80 0x7f142819f430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:44.400 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.402+0000 7f1427e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f142819ef80 0x7f142819f430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:36336/0 (socket says 172.21.15.90:36336) 2024-04-02T14:42:44.400 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.402+0000 7f1427e00640 1 -- 172.21.15.90:0/2458304637 learned_addr learned my addr 172.21.15.90:0/2458304637 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:42:44.400 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.402+0000 7f1427400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1428102470 0x7f142819a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:44.400 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.402+0000 7f1426a00640 1 --2- 172.21.15.90:0/2458304637 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1428107ac0 0x7f142819ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:44.400 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.402+0000 7f1427e00640 1 -- 172.21.15.90:0/2458304637 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1428107ac0 msgr2=0x7f142819ac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:44.400 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.402+0000 7f1427e00640 1 --2- 172.21.15.90:0/2458304637 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1428107ac0 0x7f142819ac00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:44.401 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.402+0000 7f1427e00640 1 -- 172.21.15.90:0/2458304637 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1428102470 msgr2=0x7f142819a6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:44.401 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.402+0000 7f1427e00640 1 --2- 172.21.15.90:0/2458304637 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1428102470 0x7f142819a6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:44.401 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.402+0000 7f1427e00640 1 -- 172.21.15.90:0/2458304637 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f142819fa70 con 0x7f142819ef80 2024-04-02T14:42:44.401 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.403+0000 7f1427e00640 1 --2- 172.21.15.90:0/2458304637 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f142819ef80 0x7f142819f430 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f142000eb10 tx=0x7f142000c2d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:42:44.401 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.403+0000 7f141fe00640 1 -- 172.21.15.90:0/2458304637 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1420004b20 con 0x7f142819ef80 2024-04-02T14:42:44.401 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.403+0000 7f141fe00640 1 -- 172.21.15.90:0/2458304637 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1420004cc0 con 0x7f142819ef80 2024-04-02T14:42:44.401 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.403+0000 7f141fe00640 1 -- 172.21.15.90:0/2458304637 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1420010760 con 0x7f142819ef80 2024-04-02T14:42:44.402 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.403+0000 7f142e000640 1 -- 172.21.15.90:0/2458304637 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f142819fd60 con 0x7f142819ef80 2024-04-02T14:42:44.402 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.403+0000 7f142e000640 1 -- 172.21.15.90:0/2458304637 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14281a7520 con 0x7f142819ef80 2024-04-02T14:42:44.404 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.407+0000 7f141fe00640 1 -- 172.21.15.90:0/2458304637 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f1420005020 con 0x7f142819ef80 2024-04-02T14:42:44.406 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.409+0000 7f141fe00640 1 --2- 172.21.15.90:0/2458304637 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f140c0a3f20 0x7f140c0a63e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:44.407 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.409+0000 7f141fe00640 1 -- 172.21.15.90:0/2458304637 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8550+0+0 (secure 0 0 0) 0x7f14200d0ca0 con 0x7f142819ef80 2024-04-02T14:42:44.407 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.409+0000 7f1427400640 1 --2- 172.21.15.90:0/2458304637 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f140c0a3f20 0x7f140c0a63e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:44.408 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.410+0000 7f1427400640 1 --2- 172.21.15.90:0/2458304637 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f140c0a3f20 0x7f140c0a63e0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f1418009b50 tx=0x7f1418005f90 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:42:44.408 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.411+0000 7f1430d6b740 1 Processor -- start 2024-04-02T14:42:44.409 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.411+0000 7f1430d6b740 1 -- start start 2024-04-02T14:42:44.409 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.412+0000 7f1430d6b740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a3966adeb0 0x55a3966ae290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:44.409 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.412+0000 7f1430d6b740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a3966ae7d0 0x55a3966b6e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:44.410 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.412+0000 7f1430d6b740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a3966b73b0 0x55a3966b97a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:44.410 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.412+0000 7f1430d6b740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55a396612d70 con 0x55a3966adeb0 2024-04-02T14:42:44.410 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.412+0000 7f1430d6b740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55a3965bb2a0 con 0x55a3966ae7d0 2024-04-02T14:42:44.410 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.412+0000 7f1430d6b740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55a39660ed20 con 0x55a3966b73b0 2024-04-02T14:42:44.410 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.412+0000 7f1427e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a3966b73b0 0x55a3966b97a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:44.411 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.413+0000 7f1427e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a3966b73b0 0x55a3966b97a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:36344/0 (socket says 172.21.15.90:36344) 2024-04-02T14:42:44.411 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.413+0000 7f1427e00640 1 -- 172.21.15.90:0/2395231723 learned_addr learned my addr 172.21.15.90:0/2395231723 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:42:44.411 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.413+0000 7f1426a00640 1 --2- 172.21.15.90:0/2395231723 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a3966ae7d0 0x55a3966b6e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:44.411 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.413+0000 7f1427400640 1 --2- 172.21.15.90:0/2395231723 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a3966adeb0 0x55a3966ae290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:44.411 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.413+0000 7f1426a00640 1 -- 172.21.15.90:0/2395231723 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a3966b73b0 msgr2=0x55a3966b97a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:42:44.411 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.413+0000 7f1426a00640 1 --2- 172.21.15.90:0/2395231723 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a3966b73b0 0x55a3966b97a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:44.411 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.413+0000 7f1426a00640 1 -- 172.21.15.90:0/2395231723 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a3966adeb0 msgr2=0x55a3966ae290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:42:44.411 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.413+0000 7f1426a00640 1 --2- 172.21.15.90:0/2395231723 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a3966adeb0 0x55a3966ae290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:44.411 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.413+0000 7f1426a00640 1 -- 172.21.15.90:0/2395231723 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a3965cf840 con 0x55a3966ae7d0 2024-04-02T14:42:44.412 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.413+0000 7f1427400640 1 --2- 172.21.15.90:0/2395231723 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a3966adeb0 0x55a3966ae290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:42:44.412 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.414+0000 7f1426a00640 1 --2- 172.21.15.90:0/2395231723 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a3966ae7d0 0x55a3966b6e70 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f1410008000 tx=0x7f1410005fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7527e0f745a76e76 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:42:44.412 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.414+0000 7f1417e00640 1 -- 172.21.15.90:0/2395231723 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1410042070 con 0x55a3966ae7d0 2024-04-02T14:42:44.413 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.415+0000 7f1417e00640 1 -- 172.21.15.90:0/2395231723 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f141002c8b0 con 0x55a3966ae7d0 2024-04-02T14:42:44.413 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.415+0000 7f1417e00640 1 -- 172.21.15.90:0/2395231723 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1410004c20 con 0x55a3966ae7d0 2024-04-02T14:42:44.414 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.416+0000 7f1430d6b740 1 -- 172.21.15.90:0/2395231723 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a3966ae7d0 msgr2=0x55a3966b6e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:44.414 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.416+0000 7f1430d6b740 1 --2- 172.21.15.90:0/2395231723 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a3966ae7d0 0x55a3966b6e70 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f1410008000 tx=0x7f1410005fd0 comp rx=0 tx=0).stop 2024-04-02T14:42:44.415 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.417+0000 7f1430d6b740 1 -- 172.21.15.90:0/2395231723 shutdown_connections 2024-04-02T14:42:44.415 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.417+0000 7f1430d6b740 1 --2- 172.21.15.90:0/2395231723 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a3966b73b0 0x55a3966b97a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:44.415 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.417+0000 7f1430d6b740 1 --2- 172.21.15.90:0/2395231723 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a3966ae7d0 0x55a3966b6e70 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:44.415 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.417+0000 7f1430d6b740 1 --2- 172.21.15.90:0/2395231723 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a3966adeb0 0x55a3966ae290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:44.415 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.417+0000 7f1430d6b740 1 -- 172.21.15.90:0/2395231723 >> 172.21.15.90:0/2395231723 conn(0x55a3966a85e0 msgr2=0x55a3966a89c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:42:44.415 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.417+0000 7f1430d6b740 1 -- 172.21.15.90:0/2395231723 shutdown_connections 2024-04-02T14:42:44.415 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.417+0000 7f1430d6b740 1 -- 172.21.15.90:0/2395231723 wait complete. 2024-04-02T14:42:44.415 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.418+0000 7f1430d6b740 1 -- start start 2024-04-02T14:42:44.416 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.418+0000 7f1430d6b740 1 Processor -- start 2024-04-02T14:42:44.416 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.419+0000 7f1430d6b740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a3966adeb0 0x55a3966b43b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:44.417 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.419+0000 7f1430d6b740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a3966ae7d0 0x55a3966b48f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:44.417 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.419+0000 7f1430d6b740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a3966b73b0 0x55a3966b4e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:44.417 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.419+0000 7f1430d6b740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55a3966ba140 con 0x55a3966adeb0 2024-04-02T14:42:44.417 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.419+0000 7f1430d6b740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55a3965ce660 con 0x55a3966b73b0 2024-04-02T14:42:44.417 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.419+0000 7f1430d6b740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55a3966ba2c0 con 0x55a3966ae7d0 2024-04-02T14:42:44.417 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.419+0000 7f1430d6b740 10 client.0 _ll_register_callbacks cb 0x55a3965ce050 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:42:44.418 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.419+0000 7f1427e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a3966b73b0 0x55a3966b4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:44.418 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.420+0000 7f1427e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a3966b73b0 0x55a3966b4e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:57758/0 (socket says 172.21.15.90:57758) 2024-04-02T14:42:44.418 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.420+0000 7f1427e00640 1 -- 172.21.15.90:0/596933747 learned_addr learned my addr 172.21.15.90:0/596933747 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:42:44.418 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.420+0000 7f1427400640 1 --2- 172.21.15.90:0/596933747 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a3966adeb0 0x55a3966b43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:44.418 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.420+0000 7f1426a00640 1 --2- 172.21.15.90:0/596933747 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a3966ae7d0 0x55a3966b48f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:44.418 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.420+0000 7f1427e00640 1 -- 172.21.15.90:0/596933747 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a3966ae7d0 msgr2=0x55a3966b48f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:42:44.418 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.420+0000 7f1427e00640 1 --2- 172.21.15.90:0/596933747 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a3966ae7d0 0x55a3966b48f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:44.418 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.420+0000 7f1427e00640 1 -- 172.21.15.90:0/596933747 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a3966adeb0 msgr2=0x55a3966b43b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:42:44.418 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.420+0000 7f1427e00640 1 --2- 172.21.15.90:0/596933747 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a3966adeb0 0x55a3966b43b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:44.419 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.420+0000 7f1427e00640 1 -- 172.21.15.90:0/596933747 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a3965dde80 con 0x55a3966b73b0 2024-04-02T14:42:44.419 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.421+0000 7f1427e00640 1 --2- 172.21.15.90:0/596933747 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a3966b73b0 0x55a3966b4e30 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f1420004af0 tx=0x7f142009c640 comp rx=0 tx=0).ready entity=mon.2 client_cookie=78287e0d74371be6 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:42:44.419 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.421+0000 7f1414c00640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:42:44.419 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.421+0000 7f1414c00640 1 -- 172.21.15.90:0/596933747 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f142009d080 con 0x55a3966b73b0 2024-04-02T14:42:44.420 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.421+0000 7f1414c00640 1 -- 172.21.15.90:0/596933747 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8550+0+0 (secure 0 0 0) 0x7f14200a5400 con 0x55a3966b73b0 2024-04-02T14:42:44.420 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.421+0000 7f1430d6b740 1 -- 172.21.15.90:0/596933747 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f141002c8b0 con 0x55a3966b73b0 2024-04-02T14:42:44.420 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.421+0000 7f1414c00640 1 -- 172.21.15.90:0/596933747 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f14200b6170 con 0x55a3966b73b0 2024-04-02T14:42:44.420 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.422+0000 7f1430d6b740 10 client.27944 fetch_fsmap learned FSMap version 27 2024-04-02T14:42:44.420 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.422+0000 7f1430d6b740 1 -- 172.21.15.90:0/596933747 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55a396327210 con 0x55a3966b73b0 2024-04-02T14:42:44.420 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.422+0000 7f1414c00640 1 -- 172.21.15.90:0/596933747 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f141002c8b0 con 0x55a3966b73b0 2024-04-02T14:42:44.420 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.422+0000 7f1414c00640 1 -- 172.21.15.90:0/596933747 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f1420005310 con 0x55a3966b73b0 2024-04-02T14:42:44.420 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.422+0000 7f1430d6b740 10 client.27944 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:42:44.421 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.422+0000 7f1430d6b740 10 client.27944 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-02T14:42:44.421 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.422+0000 7f1430d6b740 20 client.27944 populate_metadata read hostname 'smithi090' 2024-04-02T14:42:44.421 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.422+0000 7f1430d6b740 1 --2- 172.21.15.90:0/596933747 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55a3966c79d0 0x55a3966c9dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:44.421 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.422+0000 7f1430d6b740 4 client.27944 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:42:44.421 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.422+0000 7f1430d6b740 1 -- 172.21.15.90:0/596933747 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a3965c2cd0 con 0x55a3966c79d0 2024-04-02T14:42:44.421 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.423+0000 7f1427400640 1 --2- 172.21.15.90:0/596933747 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55a3966c79d0 0x55a3966c9dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:44.423 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.425+0000 7f1427400640 1 --2- 172.21.15.90:0/596933747 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55a3966c79d0 0x55a3966c9dc0 crc :-1 s=READY pgs=220 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a1b4455d7d9d9896 server_cookie=4ba0deffae9e8887 in_seq=0 out_seq=0 2024-04-02T14:42:44.423 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.425+0000 7f1414c00640 10 client.27944 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:42:44.424 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.426+0000 7f1414c00640 1 -- 172.21.15.90:0/596933747 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a3965c2cd0 con 0x55a3966c79d0 2024-04-02T14:42:44.424 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.426+0000 7f1414c00640 10 client.27944 handle_command_reply: tid=0 2024-04-02T14:42:44.424 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.426+0000 7f1430d6b740 1 client.27944 shutdown 2024-04-02T14:42:44.424 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.426+0000 7f1430d6b740 1 -- 172.21.15.90:0/596933747 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a3966b73b0 msgr2=0x55a3966b4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:44.425 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.426+0000 7f1430d6b740 1 --2- 172.21.15.90:0/596933747 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a3966b73b0 0x55a3966b4e30 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f1420004af0 tx=0x7f142009c640 comp rx=0 tx=0).stop 2024-04-02T14:42:44.425 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.427+0000 7f1430d6b740 1 -- 172.21.15.90:0/596933747 shutdown_connections 2024-04-02T14:42:44.425 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.427+0000 7f1430d6b740 1 --2- 172.21.15.90:0/596933747 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55a3966c79d0 0x55a3966c9dc0 crc :-1 s=READY pgs=220 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:44.425 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.427+0000 7f1430d6b740 1 --2- 172.21.15.90:0/596933747 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a3966b73b0 0x55a3966b4e30 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:44.425 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.427+0000 7f1430d6b740 1 --2- 172.21.15.90:0/596933747 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a3966ae7d0 0x55a3966b48f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:44.425 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.427+0000 7f1430d6b740 1 --2- 172.21.15.90:0/596933747 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a3966adeb0 0x55a3966b43b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:44.425 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.427+0000 7f1430d6b740 1 -- 172.21.15.90:0/596933747 >> 172.21.15.90:0/596933747 conn(0x55a3966a85e0 msgr2=0x55a3966aa330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:42:44.425 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.427+0000 7f1414c00640 0 client.27944 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:42:44.425 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.427+0000 7f1430d6b740 1 -- 172.21.15.90:0/596933747 shutdown_connections 2024-04-02T14:42:44.425 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.427+0000 7f1430d6b740 1 -- 172.21.15.90:0/596933747 wait complete. 2024-04-02T14:42:44.425 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.427+0000 7f1430d6b740 20 client.27944 trim_cache size 0 max 16384 2024-04-02T14:42:44.433 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:42:44 smithi177 ceph-mon[36431]: pgmap v240: 97 pgs: 97 active+clean; 4.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 5.5 MiB/s wr, 669 op/s 2024-04-02T14:42:44.435 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:42:44 smithi090 ceph-mon[31674]: pgmap v240: 97 pgs: 97 active+clean; 4.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 5.5 MiB/s wr, 669 op/s 2024-04-02T14:42:44.468 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.470+0000 7f1430d6b740 1 Processor -- start 2024-04-02T14:42:44.471 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.473+0000 7f1430d6b740 1 -- start start 2024-04-02T14:42:44.473 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.474+0000 7f1430d6b740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a3966adeb0 0x55a3966b6ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:44.473 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.474+0000 7f1430d6b740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a3966ae7d0 0x55a3966c3000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:44.473 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.474+0000 7f1430d6b740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a3966b73b0 0x55a3966b4a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:44.473 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.474+0000 7f1430d6b740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f14281afe60 con 0x55a3966b73b0 2024-04-02T14:42:44.474 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.474+0000 7f1430d6b740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f14281b19a0 con 0x55a3966ae7d0 2024-04-02T14:42:44.474 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.474+0000 7f1430d6b740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f14281b0530 con 0x55a3966adeb0 2024-04-02T14:42:44.474 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.475+0000 7f1426a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a3966ae7d0 0x55a3966c3000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:44.474 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.475+0000 7f1426a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a3966ae7d0 0x55a3966c3000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:57764/0 (socket says 172.21.15.90:57764) 2024-04-02T14:42:44.474 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.475+0000 7f1426a00640 1 -- 172.21.15.90:0/237686351 learned_addr learned my addr 172.21.15.90:0/237686351 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:42:44.474 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.475+0000 7f1427400640 1 --2- 172.21.15.90:0/237686351 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a3966adeb0 0x55a3966b6ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:44.474 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.476+0000 7f1426a00640 1 -- 172.21.15.90:0/237686351 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a3966adeb0 msgr2=0x55a3966b6ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:42:44.475 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.476+0000 7f1426a00640 1 --2- 172.21.15.90:0/237686351 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a3966adeb0 0x55a3966b6ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:44.475 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.476+0000 7f1426a00640 1 -- 172.21.15.90:0/237686351 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a3966b73b0 msgr2=0x55a3966b4a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:42:44.475 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.476+0000 7f1426a00640 1 --2- 172.21.15.90:0/237686351 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a3966b73b0 0x55a3966b4a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:44.475 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.476+0000 7f1426a00640 1 -- 172.21.15.90:0/237686351 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a3963f3620 con 0x55a3966ae7d0 2024-04-02T14:42:44.475 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.476+0000 7f1427400640 1 --2- 172.21.15.90:0/237686351 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a3966adeb0 0x55a3966b6ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:42:44.477 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.480+0000 7f1426a00640 1 --2- 172.21.15.90:0/237686351 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a3966ae7d0 0x55a3966c3000 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f14100047d0 tx=0x7f1410004800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ffccd2f3412b6d69 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:42:44.481 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.483+0000 7f1417400640 1 -- 172.21.15.90:0/237686351 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f141004f070 con 0x55a3966ae7d0 2024-04-02T14:42:44.481 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.483+0000 7f1417400640 1 -- 172.21.15.90:0/237686351 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f141002cab0 con 0x55a3966ae7d0 2024-04-02T14:42:44.481 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.483+0000 7f1417400640 1 -- 172.21.15.90:0/237686351 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1410042040 con 0x55a3966ae7d0 2024-04-02T14:42:44.483 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.485+0000 7f1430d6b740 1 -- 172.21.15.90:0/237686351 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a3966ae7d0 msgr2=0x55a3966c3000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:44.483 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.485+0000 7f1430d6b740 1 --2- 172.21.15.90:0/237686351 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a3966ae7d0 0x55a3966c3000 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f14100047d0 tx=0x7f1410004800 comp rx=0 tx=0).stop 2024-04-02T14:42:44.483 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.485+0000 7f1430d6b740 1 -- 172.21.15.90:0/237686351 shutdown_connections 2024-04-02T14:42:44.483 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.485+0000 7f1430d6b740 1 --2- 172.21.15.90:0/237686351 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a3966b73b0 0x55a3966b4a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:44.483 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.485+0000 7f1430d6b740 1 --2- 172.21.15.90:0/237686351 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a3966ae7d0 0x55a3966c3000 secure :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f14100047d0 tx=0x7f1410004800 comp rx=0 tx=0).stop 2024-04-02T14:42:44.483 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.485+0000 7f1430d6b740 1 --2- 172.21.15.90:0/237686351 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a3966adeb0 0x55a3966b6ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:44.483 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.485+0000 7f1430d6b740 1 -- 172.21.15.90:0/237686351 >> 172.21.15.90:0/237686351 conn(0x55a3966a85e0 msgr2=0x55a3966b4370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:42:44.485 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.488+0000 7f1430d6b740 1 -- 172.21.15.90:0/237686351 shutdown_connections 2024-04-02T14:42:44.486 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.488+0000 7f1430d6b740 1 -- 172.21.15.90:0/237686351 wait complete. 2024-04-02T14:42:44.486 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.489+0000 7f1430d6b740 1 -- start start 2024-04-02T14:42:44.487 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.489+0000 7f1430d6b740 1 Processor -- start 2024-04-02T14:42:44.488 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.490+0000 7f1430d6b740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a3966adeb0 0x55a3966b12d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:44.488 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.490+0000 7f1430d6b740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a3966b73b0 0x55a3966b1810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:44.488 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.490+0000 7f1430d6b740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a3966b1e10 0x55a39672cf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:44.488 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.490+0000 7f1430d6b740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f14281b12d0 con 0x55a3966b1e10 2024-04-02T14:42:44.488 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.490+0000 7f1430d6b740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f14281b0c00 con 0x55a3966b73b0 2024-04-02T14:42:44.488 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.490+0000 7f1430d6b740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f14281ade60 con 0x55a3966adeb0 2024-04-02T14:42:44.488 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.490+0000 7f1430d6b740 10 client.0 _ll_register_callbacks cb 0x7f142009ca60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:42:44.489 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.491+0000 7f1427e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a3966b1e10 0x55a39672cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:44.489 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.491+0000 7f1427e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a3966b1e10 0x55a39672cf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:60268/0 (socket says 172.21.15.90:60268) 2024-04-02T14:42:44.489 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.491+0000 7f1427e00640 1 -- 172.21.15.90:0/2276508650 learned_addr learned my addr 172.21.15.90:0/2276508650 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:42:44.489 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.491+0000 7f1426a00640 1 --2- 172.21.15.90:0/2276508650 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a3966b73b0 0x55a3966b1810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:44.490 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.492+0000 7f1427e00640 1 -- 172.21.15.90:0/2276508650 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a3966adeb0 msgr2=0x55a3966b12d0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-02T14:42:44.490 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.492+0000 7f1427e00640 1 --2- 172.21.15.90:0/2276508650 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a3966adeb0 0x55a3966b12d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:44.490 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.492+0000 7f1427e00640 1 -- 172.21.15.90:0/2276508650 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a3966b73b0 msgr2=0x55a3966b1810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:42:44.490 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.492+0000 7f1427e00640 1 --2- 172.21.15.90:0/2276508650 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a3966b73b0 0x55a3966b1810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:44.490 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.492+0000 7f1427e00640 1 -- 172.21.15.90:0/2276508650 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a39672d600 con 0x55a3966b1e10 2024-04-02T14:42:44.490 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.492+0000 7f1427e00640 1 --2- 172.21.15.90:0/2276508650 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a3966b1e10 0x55a39672cf20 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f14200b6310 tx=0x7f1420098ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa7e6e9efb2a1309 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:42:44.490 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.492+0000 7f1414c00640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-02T14:42:44.491 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.494+0000 7f1414c00640 1 -- 172.21.15.90:0/2276508650 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f142009c760 con 0x55a3966b1e10 2024-04-02T14:42:44.492 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.494+0000 7f1414c00640 1 -- 172.21.15.90:0/2276508650 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8550+0+0 (secure 0 0 0) 0x7f14200a5400 con 0x55a3966b1e10 2024-04-02T14:42:44.492 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.494+0000 7f1430d6b740 1 -- 172.21.15.90:0/2276508650 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f141002cab0 con 0x55a3966b1e10 2024-04-02T14:42:44.492 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.494+0000 7f1414c00640 1 -- 172.21.15.90:0/2276508650 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f14200a9030 con 0x55a3966b1e10 2024-04-02T14:42:44.492 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.494+0000 7f1430d6b740 10 client.17166 fetch_fsmap learned FSMap version 27 2024-04-02T14:42:44.492 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.494+0000 7f1430d6b740 1 -- 172.21.15.90:0/2276508650 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55a39672e860 con 0x55a3966b1e10 2024-04-02T14:42:44.492 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.495+0000 7f1414c00640 1 -- 172.21.15.90:0/2276508650 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f141002cab0 con 0x55a3966b1e10 2024-04-02T14:42:44.495 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.495+0000 7f1414c00640 1 -- 172.21.15.90:0/2276508650 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f1420098a40 con 0x55a3966b1e10 2024-04-02T14:42:44.495 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.495+0000 7f1430d6b740 10 client.17166 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:42:44.495 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.495+0000 7f1430d6b740 10 client.17166 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-02T14:42:44.495 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.495+0000 7f1430d6b740 20 client.17166 populate_metadata read hostname 'smithi090' 2024-04-02T14:42:44.495 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.495+0000 7f1430d6b740 1 --2- 172.21.15.90:0/2276508650 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55a39672ef80 0x55a396731370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:44.495 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.495+0000 7f1430d6b740 4 client.17166 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-02T14:42:44.495 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.495+0000 7f1430d6b740 1 -- 172.21.15.90:0/2276508650 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55a3966202b0 con 0x55a39672ef80 2024-04-02T14:42:44.495 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.496+0000 7f1427400640 1 --2- 172.21.15.90:0/2276508650 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55a39672ef80 0x55a396731370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:44.496 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.496+0000 7f1427400640 1 --2- 172.21.15.90:0/2276508650 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55a39672ef80 0x55a396731370 crc :-1 s=READY pgs=221 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fef401ae0f689f99 server_cookie=e469b79f1f892d64 in_seq=0 out_seq=0 2024-04-02T14:42:44.496 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.497+0000 7f1414c00640 10 client.17166 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:42:44.540 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.542+0000 7f1414c00640 1 -- 172.21.15.90:0/2276508650 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55a3966202b0 con 0x55a39672ef80 2024-04-02T14:42:44.540 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.542+0000 7f1414c00640 10 client.17166 handle_command_reply: tid=0 2024-04-02T14:42:44.540 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.542+0000 7f1430d6b740 1 client.17166 shutdown 2024-04-02T14:42:44.540 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.542+0000 7f1430d6b740 1 -- 172.21.15.90:0/2276508650 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a3966b1e10 msgr2=0x55a39672cf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:44.540 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.542+0000 7f1430d6b740 1 --2- 172.21.15.90:0/2276508650 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a3966b1e10 0x55a39672cf20 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f14200b6310 tx=0x7f1420098ec0 comp rx=0 tx=0).stop 2024-04-02T14:42:44.541 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.543+0000 7f1430d6b740 1 -- 172.21.15.90:0/2276508650 shutdown_connections 2024-04-02T14:42:44.541 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.543+0000 7f1430d6b740 1 --2- 172.21.15.90:0/2276508650 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55a39672ef80 0x55a396731370 crc :-1 s=READY pgs=221 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:44.541 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.543+0000 7f1430d6b740 1 --2- 172.21.15.90:0/2276508650 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a3966b1e10 0x55a39672cf20 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:44.541 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.543+0000 7f1430d6b740 1 --2- 172.21.15.90:0/2276508650 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a3966b73b0 0x55a3966b1810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:44.541 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.543+0000 7f1430d6b740 1 --2- 172.21.15.90:0/2276508650 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a3966adeb0 0x55a3966b12d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:44.541 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.543+0000 7f1430d6b740 1 -- 172.21.15.90:0/2276508650 >> 172.21.15.90:0/2276508650 conn(0x55a3966a85e0 msgr2=0x55a3966b4c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:42:44.541 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.543+0000 7f1414c00640 0 client.17166 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:42:44.541 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.543+0000 7f1430d6b740 1 -- 172.21.15.90:0/2276508650 shutdown_connections 2024-04-02T14:42:44.542 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.543+0000 7f1430d6b740 1 -- 172.21.15.90:0/2276508650 wait complete. 2024-04-02T14:42:44.542 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.544+0000 7f1430d6b740 20 client.17166 trim_cache size 0 max 16384 2024-04-02T14:42:44.543 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:{ 2024-04-02T14:42:44.543 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "message": "", 2024-04-02T14:42:44.543 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "return_code": 0 2024-04-02T14:42:44.543 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:} 2024-04-02T14:42:44.544 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.545+0000 7f142e000640 1 -- 172.21.15.90:0/2458304637 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f140c0a3f20 msgr2=0x7f140c0a63e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:44.544 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.545+0000 7f142e000640 1 --2- 172.21.15.90:0/2458304637 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f140c0a3f20 0x7f140c0a63e0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f1418009b50 tx=0x7f1418005f90 comp rx=0 tx=0).stop 2024-04-02T14:42:44.544 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.545+0000 7f142e000640 1 -- 172.21.15.90:0/2458304637 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f142819ef80 msgr2=0x7f142819f430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:44.544 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.545+0000 7f142e000640 1 --2- 172.21.15.90:0/2458304637 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f142819ef80 0x7f142819f430 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f142000eb10 tx=0x7f142000c2d0 comp rx=0 tx=0).stop 2024-04-02T14:42:44.544 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.545+0000 7f142e000640 1 -- 172.21.15.90:0/2458304637 shutdown_connections 2024-04-02T14:42:44.544 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.545+0000 7f142e000640 1 --2- 172.21.15.90:0/2458304637 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f140c0a3f20 0x7f140c0a63e0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:44.544 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.545+0000 7f142e000640 1 --2- 172.21.15.90:0/2458304637 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f142819ef80 0x7f142819f430 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:44.544 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.545+0000 7f142e000640 1 --2- 172.21.15.90:0/2458304637 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1428107ac0 0x7f142819ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:44.544 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.545+0000 7f142e000640 1 --2- 172.21.15.90:0/2458304637 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1428102470 0x7f142819a6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:44.544 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.545+0000 7f142e000640 1 -- 172.21.15.90:0/2458304637 >> 172.21.15.90:0/2458304637 conn(0x7f14280fd5c0 msgr2=0x7f142810d260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:42:44.544 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.546+0000 7f142e000640 1 -- 172.21.15.90:0/2458304637 shutdown_connections 2024-04-02T14:42:44.544 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:44.546+0000 7f142e000640 1 -- 172.21.15.90:0/2458304637 wait complete. 2024-04-02T14:42:44.607 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:42:44 smithi092 ceph-mon[36497]: pgmap v240: 97 pgs: 97 active+clean; 4.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 5.5 MiB/s wr, 669 op/s 2024-04-02T14:42:46.434 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:42:46 smithi177 ceph-mon[36431]: pgmap v241: 97 pgs: 97 active+clean; 4.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 222 KiB/s rd, 11 MiB/s wr, 1.09k op/s 2024-04-02T14:42:46.436 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:42:46 smithi090 ceph-mon[31674]: pgmap v241: 97 pgs: 97 active+clean; 4.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 222 KiB/s rd, 11 MiB/s wr, 1.09k op/s 2024-04-02T14:42:46.608 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:42:46 smithi092 ceph-mon[36497]: pgmap v241: 97 pgs: 97 active+clean; 4.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 222 KiB/s rd, 11 MiB/s wr, 1.09k op/s 2024-04-02T14:42:46.828 INFO:teuthology.orchestra.run.smithi090.stderr:progress: 435.0 s, 1745.0 tps, lat 57.737 ms stddev 64.411 2024-04-02T14:42:48.433 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:42:48 smithi177 ceph-mon[36431]: pgmap v242: 97 pgs: 97 active+clean; 4.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 199 KiB/s rd, 8.0 MiB/s wr, 779 op/s 2024-04-02T14:42:48.435 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:42:48 smithi090 ceph-mon[31674]: pgmap v242: 97 pgs: 97 active+clean; 4.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 199 KiB/s rd, 8.0 MiB/s wr, 779 op/s 2024-04-02T14:42:48.611 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:42:48 smithi092 ceph-mon[36497]: pgmap v242: 97 pgs: 97 active+clean; 4.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 199 KiB/s rd, 8.0 MiB/s wr, 779 op/s 2024-04-02T14:42:50.433 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:42:50 smithi177 ceph-mon[36431]: pgmap v243: 97 pgs: 97 active+clean; 4.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 216 KiB/s rd, 12 MiB/s wr, 932 op/s 2024-04-02T14:42:50.435 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:42:50 smithi090 ceph-mon[31674]: pgmap v243: 97 pgs: 97 active+clean; 4.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 216 KiB/s rd, 12 MiB/s wr, 932 op/s 2024-04-02T14:42:50.607 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:42:50 smithi092 ceph-mon[36497]: pgmap v243: 97 pgs: 97 active+clean; 4.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 216 KiB/s rd, 12 MiB/s wr, 932 op/s 2024-04-02T14:42:51.828 INFO:teuthology.orchestra.run.smithi090.stderr:progress: 440.0 s, 2023.4 tps, lat 49.587 ms stddev 56.530 2024-04-02T14:42:52.433 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:42:52 smithi177 ceph-mon[36431]: pgmap v244: 97 pgs: 97 active+clean; 4.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 14 MiB/s wr, 911 op/s 2024-04-02T14:42:52.434 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:42:52 smithi090 ceph-mon[31674]: pgmap v244: 97 pgs: 97 active+clean; 4.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 14 MiB/s wr, 911 op/s 2024-04-02T14:42:52.607 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:42:52 smithi092 ceph-mon[36497]: pgmap v244: 97 pgs: 97 active+clean; 4.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 14 MiB/s wr, 911 op/s 2024-04-02T14:42:54.433 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:42:54 smithi177 ceph-mon[36431]: pgmap v245: 97 pgs: 97 active+clean; 4.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 13 MiB/s wr, 748 op/s 2024-04-02T14:42:54.434 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:42:54 smithi090 ceph-mon[31674]: pgmap v245: 97 pgs: 97 active+clean; 4.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 13 MiB/s wr, 748 op/s 2024-04-02T14:42:54.607 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:42:54 smithi092 ceph-mon[36497]: pgmap v245: 97 pgs: 97 active+clean; 4.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 13 MiB/s wr, 748 op/s 2024-04-02T14:42:56.433 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:42:56 smithi177 ceph-mon[36431]: pgmap v246: 97 pgs: 97 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 23 MiB/s wr, 1.18k op/s 2024-04-02T14:42:56.434 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:42:56 smithi090 ceph-mon[31674]: pgmap v246: 97 pgs: 97 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 23 MiB/s wr, 1.18k op/s 2024-04-02T14:42:56.607 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:42:56 smithi092 ceph-mon[36497]: pgmap v246: 97 pgs: 97 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 23 MiB/s wr, 1.18k op/s 2024-04-02T14:42:56.828 INFO:teuthology.orchestra.run.smithi090.stderr:progress: 445.0 s, 2207.2 tps, lat 44.864 ms stddev 49.629 2024-04-02T14:42:57.704 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.705+0000 7fd07b200640 1 -- 172.21.15.90:0/3928524683 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd074102470 msgr2=0x7fd07410e980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:57.704 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.705+0000 7fd07b200640 1 --2- 172.21.15.90:0/3928524683 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd074102470 0x7fd07410e980 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fd06c009930 tx=0x7fd06c02bc30 comp rx=0 tx=0).stop 2024-04-02T14:42:57.706 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.707+0000 7fd07b200640 1 -- 172.21.15.90:0/3928524683 shutdown_connections 2024-04-02T14:42:57.706 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.707+0000 7fd07b200640 1 --2- 172.21.15.90:0/3928524683 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd074102470 0x7fd07410e980 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:57.706 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.707+0000 7fd07b200640 1 --2- 172.21.15.90:0/3928524683 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd074101ab0 0x7fd074101f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:57.706 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.707+0000 7fd07b200640 1 --2- 172.21.15.90:0/3928524683 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd074107ac0 0x7fd074107ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:57.706 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.707+0000 7fd07b200640 1 -- 172.21.15.90:0/3928524683 >> 172.21.15.90:0/3928524683 conn(0x7fd0740fd5c0 msgr2=0x7fd0740ff9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:42:57.706 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.707+0000 7fd07b200640 1 -- 172.21.15.90:0/3928524683 shutdown_connections 2024-04-02T14:42:57.707 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.708+0000 7fd07b200640 1 -- 172.21.15.90:0/3928524683 wait complete. 2024-04-02T14:42:57.707 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.709+0000 7fd07b200640 1 Processor -- start 2024-04-02T14:42:57.708 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.709+0000 7fd07b200640 1 -- start start 2024-04-02T14:42:57.709 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.710+0000 7fd07b200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd074101ab0 0x7fd074108a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:57.709 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.710+0000 7fd07b200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd074102470 0x7fd074108f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:57.709 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.710+0000 7fd07b200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd074107ac0 0x7fd074109480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:57.709 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.710+0000 7fd07b200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fd074111f20 con 0x7fd074101ab0 2024-04-02T14:42:57.709 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.710+0000 7fd07b200640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fd074111da0 con 0x7fd074107ac0 2024-04-02T14:42:57.709 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.710+0000 7fd07b200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fd0741120a0 con 0x7fd074102470 2024-04-02T14:42:57.709 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.711+0000 7fd073e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd074101ab0 0x7fd074108a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:57.710 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.712+0000 7fd079200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd074107ac0 0x7fd074109480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:57.710 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.712+0000 7fd079200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd074107ac0 0x7fd074109480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:57558/0 (socket says 172.21.15.90:57558) 2024-04-02T14:42:57.710 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.712+0000 7fd079200640 1 -- 172.21.15.90:0/3778863884 learned_addr learned my addr 172.21.15.90:0/3778863884 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:42:57.712 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.713+0000 7fd079200640 1 -- 172.21.15.90:0/3778863884 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd074102470 msgr2=0x7fd074108f40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T14:42:57.712 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.714+0000 7fd079200640 1 --2- 172.21.15.90:0/3778863884 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd074102470 0x7fd074108f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:57.712 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.714+0000 7fd079200640 1 -- 172.21.15.90:0/3778863884 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd074101ab0 msgr2=0x7fd074108a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:57.712 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.714+0000 7fd079200640 1 --2- 172.21.15.90:0/3778863884 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd074101ab0 0x7fd074108a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:57.713 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.714+0000 7fd079200640 1 -- 172.21.15.90:0/3778863884 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd07419a290 con 0x7fd074107ac0 2024-04-02T14:42:57.714 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.714+0000 7fd079200640 1 --2- 172.21.15.90:0/3778863884 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd074107ac0 0x7fd074109480 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7fd06c036040 tx=0x7fd06c004270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:42:57.715 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.717+0000 7fd071600640 1 -- 172.21.15.90:0/3778863884 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd06c043020 con 0x7fd074107ac0 2024-04-02T14:42:57.715 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.717+0000 7fd071600640 1 -- 172.21.15.90:0/3778863884 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd06c005000 con 0x7fd074107ac0 2024-04-02T14:42:57.715 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.717+0000 7fd071600640 1 -- 172.21.15.90:0/3778863884 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd06c003a90 con 0x7fd074107ac0 2024-04-02T14:42:57.715 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.717+0000 7fd07b200640 1 -- 172.21.15.90:0/3778863884 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd07419a520 con 0x7fd074107ac0 2024-04-02T14:42:57.716 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.717+0000 7fd07b200640 1 -- 172.21.15.90:0/3778863884 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd07419aa20 con 0x7fd074107ac0 2024-04-02T14:42:57.725 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.725+0000 7fd071600640 1 -- 172.21.15.90:0/3778863884 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fd06c046020 con 0x7fd074107ac0 2024-04-02T14:42:57.729 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.731+0000 7fd071600640 1 --2- 172.21.15.90:0/3778863884 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fd0580a3ec0 0x7fd0580a6380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:57.731 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.732+0000 7fd073e00640 1 --2- 172.21.15.90:0/3778863884 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fd0580a3ec0 0x7fd0580a6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:57.731 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.732+0000 7fd071600640 1 -- 172.21.15.90:0/3778863884 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8550+0+0 (secure 0 0 0) 0x7fd06c0f26e0 con 0x7fd074107ac0 2024-04-02T14:42:57.733 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.735+0000 7fd073e00640 1 --2- 172.21.15.90:0/3778863884 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fd0580a3ec0 0x7fd0580a6380 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fd064005fd0 tx=0x7fd064009500 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:42:57.733 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.735+0000 7fd07df64740 1 Processor -- start 2024-04-02T14:42:57.734 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.735+0000 7fd07df64740 1 -- start start 2024-04-02T14:42:57.734 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.736+0000 7fd07df64740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55dd525a9eb0 0x55dd525aa290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:57.735 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.736+0000 7fd07df64740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dd525aa7d0 0x55dd525b2e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:57.735 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.736+0000 7fd07df64740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dd525b33b0 0x55dd525b57a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:57.735 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.736+0000 7fd07df64740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55dd5250ed70 con 0x55dd525a9eb0 2024-04-02T14:42:57.735 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.736+0000 7fd07df64740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55dd524b72a0 con 0x55dd525aa7d0 2024-04-02T14:42:57.735 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.736+0000 7fd07df64740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55dd5250ad20 con 0x55dd525b33b0 2024-04-02T14:42:57.735 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.736+0000 7fd06b400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dd525aa7d0 0x55dd525b2e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:57.735 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.736+0000 7fd06b400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dd525aa7d0 0x55dd525b2e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:57562/0 (socket says 172.21.15.90:57562) 2024-04-02T14:42:57.735 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.736+0000 7fd06b400640 1 -- 172.21.15.90:0/4172734154 learned_addr learned my addr 172.21.15.90:0/4172734154 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:42:57.735 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.736+0000 7fd073e00640 1 --2- 172.21.15.90:0/4172734154 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55dd525a9eb0 0x55dd525aa290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:57.736 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.737+0000 7fd06b400640 1 -- 172.21.15.90:0/4172734154 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dd525b33b0 msgr2=0x55dd525b57a0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-02T14:42:57.736 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.737+0000 7fd06b400640 1 --2- 172.21.15.90:0/4172734154 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dd525b33b0 0x55dd525b57a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:57.736 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.737+0000 7fd06b400640 1 -- 172.21.15.90:0/4172734154 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55dd525a9eb0 msgr2=0x55dd525aa290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:42:57.736 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.737+0000 7fd06b400640 1 --2- 172.21.15.90:0/4172734154 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55dd525a9eb0 0x55dd525aa290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:57.737 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.737+0000 7fd06b400640 1 -- 172.21.15.90:0/4172734154 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dd524cb840 con 0x55dd525aa7d0 2024-04-02T14:42:57.737 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.737+0000 7fd073e00640 1 --2- 172.21.15.90:0/4172734154 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55dd525a9eb0 0x55dd525aa290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:42:57.737 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.737+0000 7fd06b400640 1 --2- 172.21.15.90:0/4172734154 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dd525aa7d0 0x55dd525b2e70 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7fd05c009bb0 tx=0x7fd05c00e720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5d463e08e80defe0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:42:57.741 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.741+0000 7fd069600640 1 -- 172.21.15.90:0/4172734154 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd05c005020 con 0x55dd525aa7d0 2024-04-02T14:42:57.741 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.741+0000 7fd069600640 1 -- 172.21.15.90:0/4172734154 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd05c0047a0 con 0x55dd525aa7d0 2024-04-02T14:42:57.741 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.741+0000 7fd069600640 1 -- 172.21.15.90:0/4172734154 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd05c010690 con 0x55dd525aa7d0 2024-04-02T14:42:57.741 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.742+0000 7fd07df64740 1 -- 172.21.15.90:0/4172734154 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dd525aa7d0 msgr2=0x55dd525b2e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:57.741 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.742+0000 7fd07df64740 1 --2- 172.21.15.90:0/4172734154 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dd525aa7d0 0x55dd525b2e70 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7fd05c009bb0 tx=0x7fd05c00e720 comp rx=0 tx=0).stop 2024-04-02T14:42:57.742 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.743+0000 7fd07df64740 1 -- 172.21.15.90:0/4172734154 shutdown_connections 2024-04-02T14:42:57.742 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.743+0000 7fd07df64740 1 --2- 172.21.15.90:0/4172734154 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dd525b33b0 0x55dd525b57a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:57.742 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.743+0000 7fd07df64740 1 --2- 172.21.15.90:0/4172734154 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dd525aa7d0 0x55dd525b2e70 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:57.742 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.743+0000 7fd07df64740 1 --2- 172.21.15.90:0/4172734154 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55dd525a9eb0 0x55dd525aa290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:57.742 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.743+0000 7fd07df64740 1 -- 172.21.15.90:0/4172734154 >> 172.21.15.90:0/4172734154 conn(0x55dd525a45e0 msgr2=0x55dd525a49c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:42:57.742 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.743+0000 7fd07df64740 1 -- 172.21.15.90:0/4172734154 shutdown_connections 2024-04-02T14:42:57.742 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.743+0000 7fd07df64740 1 -- 172.21.15.90:0/4172734154 wait complete. 2024-04-02T14:42:57.743 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.744+0000 7fd07df64740 1 -- start start 2024-04-02T14:42:57.743 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.745+0000 7fd07df64740 1 Processor -- start 2024-04-02T14:42:57.746 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.746+0000 7fd07df64740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55dd525a9eb0 0x55dd525b0480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:57.746 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.746+0000 7fd07df64740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dd525aa7d0 0x55dd525b09c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:57.746 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.746+0000 7fd07df64740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dd525b33b0 0x55dd525b0f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:57.746 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.746+0000 7fd07df64740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55dd525b61a0 con 0x55dd525a9eb0 2024-04-02T14:42:57.746 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.746+0000 7fd07df64740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55dd524ca660 con 0x55dd525b33b0 2024-04-02T14:42:57.746 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.746+0000 7fd07df64740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55dd525b6320 con 0x55dd525aa7d0 2024-04-02T14:42:57.746 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.746+0000 7fd07df64740 10 client.0 _ll_register_callbacks cb 0x55dd524ca050 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:42:57.747 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.746+0000 7fd06b400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dd525aa7d0 0x55dd525b09c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:57.747 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.746+0000 7fd06b400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dd525aa7d0 0x55dd525b09c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:33076/0 (socket says 172.21.15.90:33076) 2024-04-02T14:42:57.747 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.746+0000 7fd06b400640 1 -- 172.21.15.90:0/3437205278 learned_addr learned my addr 172.21.15.90:0/3437205278 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:42:57.747 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.746+0000 7fd073e00640 1 --2- 172.21.15.90:0/3437205278 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55dd525a9eb0 0x55dd525b0480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:57.747 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.747+0000 7fd073e00640 1 -- 172.21.15.90:0/3437205278 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dd525aa7d0 msgr2=0x55dd525b09c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:42:57.747 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.747+0000 7fd073e00640 1 --2- 172.21.15.90:0/3437205278 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dd525aa7d0 0x55dd525b09c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:57.747 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.747+0000 7fd073e00640 1 -- 172.21.15.90:0/3437205278 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dd525b33b0 msgr2=0x55dd525b0f00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-02T14:42:57.748 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.748+0000 7fd073e00640 1 --2- 172.21.15.90:0/3437205278 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dd525b33b0 0x55dd525b0f00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:57.748 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.748+0000 7fd073e00640 1 -- 172.21.15.90:0/3437205278 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dd524d9e80 con 0x55dd525a9eb0 2024-04-02T14:42:57.748 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.748+0000 7fd073e00640 1 --2- 172.21.15.90:0/3437205278 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55dd525a9eb0 0x55dd525b0480 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fd064003f60 tx=0x7fd064004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3eb2b0520fe1f2f6 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:42:57.748 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.748+0000 7fd062000640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-02T14:42:57.748 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.749+0000 7fd062000640 1 -- 172.21.15.90:0/3437205278 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd06401db40 con 0x55dd525a9eb0 2024-04-02T14:42:57.748 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.749+0000 7fd062000640 1 -- 172.21.15.90:0/3437205278 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8550+0+0 (secure 0 0 0) 0x7fd06401dce0 con 0x55dd525a9eb0 2024-04-02T14:42:57.748 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.749+0000 7fd07df64740 1 -- 172.21.15.90:0/3437205278 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd05c0047a0 con 0x55dd525a9eb0 2024-04-02T14:42:57.749 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.751+0000 7fd062000640 1 -- 172.21.15.90:0/3437205278 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fd064040430 con 0x55dd525a9eb0 2024-04-02T14:42:57.749 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.751+0000 7fd07df64740 10 client.17175 fetch_fsmap learned FSMap version 27 2024-04-02T14:42:57.749 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.751+0000 7fd07df64740 1 -- 172.21.15.90:0/3437205278 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55dd52223210 con 0x55dd525a9eb0 2024-04-02T14:42:57.750 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.751+0000 7fd062000640 1 -- 172.21.15.90:0/3437205278 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd05c0047a0 con 0x55dd525a9eb0 2024-04-02T14:42:57.751 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.753+0000 7fd062000640 1 -- 172.21.15.90:0/3437205278 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fd06401b760 con 0x55dd525a9eb0 2024-04-02T14:42:57.751 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.753+0000 7fd07df64740 10 client.17175 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:42:57.751 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.753+0000 7fd07df64740 10 client.17175 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-02T14:42:57.751 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.753+0000 7fd07df64740 20 client.17175 populate_metadata read hostname 'smithi090' 2024-04-02T14:42:57.752 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.753+0000 7fd07df64740 1 --2- 172.21.15.90:0/3437205278 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55dd525c3810 0x55dd525c5c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:57.752 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.753+0000 7fd07df64740 4 client.17175 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:42:57.752 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.753+0000 7fd07df64740 1 -- 172.21.15.90:0/3437205278 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55dd524becd0 con 0x55dd525c3810 2024-04-02T14:42:57.752 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.754+0000 7fd06b400640 1 --2- 172.21.15.90:0/3437205278 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55dd525c3810 0x55dd525c5c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:57.754 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.756+0000 7fd062000640 10 client.17175 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:42:57.755 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.756+0000 7fd06b400640 1 --2- 172.21.15.90:0/3437205278 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55dd525c3810 0x55dd525c5c00 crc :-1 s=READY pgs=222 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6c6432df9b00ca88 server_cookie=882dbdbec553abcb in_seq=0 out_seq=0 2024-04-02T14:42:57.756 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.757+0000 7fd062000640 1 -- 172.21.15.90:0/3437205278 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55dd524becd0 con 0x55dd525c3810 2024-04-02T14:42:57.756 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.757+0000 7fd062000640 10 client.17175 handle_command_reply: tid=0 2024-04-02T14:42:57.756 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.757+0000 7fd07df64740 1 client.17175 shutdown 2024-04-02T14:42:57.756 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.758+0000 7fd07df64740 1 -- 172.21.15.90:0/3437205278 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55dd525a9eb0 msgr2=0x55dd525b0480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:57.756 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.758+0000 7fd07df64740 1 --2- 172.21.15.90:0/3437205278 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55dd525a9eb0 0x55dd525b0480 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fd064003f60 tx=0x7fd064004ab0 comp rx=0 tx=0).stop 2024-04-02T14:42:57.756 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.758+0000 7fd07df64740 1 -- 172.21.15.90:0/3437205278 shutdown_connections 2024-04-02T14:42:57.757 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.758+0000 7fd07df64740 1 --2- 172.21.15.90:0/3437205278 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55dd525c3810 0x55dd525c5c00 crc :-1 s=READY pgs=222 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:57.757 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.758+0000 7fd07df64740 1 --2- 172.21.15.90:0/3437205278 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dd525b33b0 0x55dd525b0f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:57.757 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.758+0000 7fd07df64740 1 --2- 172.21.15.90:0/3437205278 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dd525aa7d0 0x55dd525b09c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:57.757 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.758+0000 7fd07df64740 1 --2- 172.21.15.90:0/3437205278 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55dd525a9eb0 0x55dd525b0480 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:57.757 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.758+0000 7fd062000640 0 client.17175 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:42:57.757 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.758+0000 7fd07df64740 1 -- 172.21.15.90:0/3437205278 >> 172.21.15.90:0/3437205278 conn(0x55dd525a45e0 msgr2=0x55dd525a6330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:42:57.757 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.759+0000 7fd07df64740 1 -- 172.21.15.90:0/3437205278 shutdown_connections 2024-04-02T14:42:57.758 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.759+0000 7fd07df64740 1 -- 172.21.15.90:0/3437205278 wait complete. 2024-04-02T14:42:57.758 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.759+0000 7fd07df64740 20 client.17175 trim_cache size 0 max 16384 2024-04-02T14:42:57.802 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.803+0000 7fd07df64740 1 Processor -- start 2024-04-02T14:42:57.802 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.804+0000 7fd07df64740 1 -- start start 2024-04-02T14:42:57.803 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.804+0000 7fd07df64740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dd525a9eb0 0x55dd525b2bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:57.804 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.804+0000 7fd07df64740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55dd525aa7d0 0x55dd525bf090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:57.804 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.804+0000 7fd07df64740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dd525b33b0 0x55dd525a62c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:57.804 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.804+0000 7fd07df64740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fd0741ab5e0 con 0x55dd525aa7d0 2024-04-02T14:42:57.804 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.804+0000 7fd07df64740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fd0741ad120 con 0x55dd525a9eb0 2024-04-02T14:42:57.804 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.804+0000 7fd07df64740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fd0741abcb0 con 0x55dd525b33b0 2024-04-02T14:42:57.804 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.805+0000 7fd073e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dd525a9eb0 0x55dd525b2bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:57.804 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.806+0000 7fd079200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dd525b33b0 0x55dd525a62c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:57.804 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.806+0000 7fd079200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dd525b33b0 0x55dd525a62c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:33084/0 (socket says 172.21.15.90:33084) 2024-04-02T14:42:57.805 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.806+0000 7fd079200640 1 -- 172.21.15.90:0/3149059043 learned_addr learned my addr 172.21.15.90:0/3149059043 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:42:57.805 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.807+0000 7fd079200640 1 -- 172.21.15.90:0/3149059043 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dd525a9eb0 msgr2=0x55dd525b2bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:42:57.806 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.807+0000 7fd079200640 1 --2- 172.21.15.90:0/3149059043 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dd525a9eb0 0x55dd525b2bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:57.806 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.807+0000 7fd079200640 1 -- 172.21.15.90:0/3149059043 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55dd525aa7d0 msgr2=0x55dd525bf090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:42:57.806 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.807+0000 7fd079200640 1 --2- 172.21.15.90:0/3149059043 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55dd525aa7d0 0x55dd525bf090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:57.806 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.807+0000 7fd079200640 1 -- 172.21.15.90:0/3149059043 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dd525af430 con 0x55dd525b33b0 2024-04-02T14:42:57.806 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.808+0000 7fd079200640 1 --2- 172.21.15.90:0/3149059043 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dd525b33b0 0x55dd525a62c0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fd06c02c170 tx=0x7fd06c0bab90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=88e0dc2d21ad03f4 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:42:57.807 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.808+0000 7fd068c00640 1 -- 172.21.15.90:0/3149059043 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd06c0bef30 con 0x55dd525b33b0 2024-04-02T14:42:57.807 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.808+0000 7fd068c00640 1 -- 172.21.15.90:0/3149059043 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd06c0be830 con 0x55dd525b33b0 2024-04-02T14:42:57.807 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.809+0000 7fd068c00640 1 -- 172.21.15.90:0/3149059043 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd06c0f0030 con 0x55dd525b33b0 2024-04-02T14:42:57.807 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.809+0000 7fd07df64740 1 -- 172.21.15.90:0/3149059043 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dd525b33b0 msgr2=0x55dd525a62c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:57.807 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.809+0000 7fd07df64740 1 --2- 172.21.15.90:0/3149059043 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dd525b33b0 0x55dd525a62c0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fd06c02c170 tx=0x7fd06c0bab90 comp rx=0 tx=0).stop 2024-04-02T14:42:57.809 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.810+0000 7fd07df64740 1 -- 172.21.15.90:0/3149059043 shutdown_connections 2024-04-02T14:42:57.809 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.810+0000 7fd07df64740 1 --2- 172.21.15.90:0/3149059043 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dd525b33b0 0x55dd525a62c0 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:57.809 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.810+0000 7fd07df64740 1 --2- 172.21.15.90:0/3149059043 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55dd525aa7d0 0x55dd525bf090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:57.809 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.810+0000 7fd07df64740 1 --2- 172.21.15.90:0/3149059043 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dd525a9eb0 0x55dd525b2bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:57.809 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.810+0000 7fd07df64740 1 -- 172.21.15.90:0/3149059043 >> 172.21.15.90:0/3149059043 conn(0x55dd525a45e0 msgr2=0x55dd525a49c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:42:57.809 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.810+0000 7fd07df64740 1 -- 172.21.15.90:0/3149059043 shutdown_connections 2024-04-02T14:42:57.809 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.810+0000 7fd07df64740 1 -- 172.21.15.90:0/3149059043 wait complete. 2024-04-02T14:42:57.810 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.811+0000 7fd07df64740 1 -- start start 2024-04-02T14:42:57.810 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.811+0000 7fd07df64740 1 Processor -- start 2024-04-02T14:42:57.811 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.812+0000 7fd07df64740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55dd525a9eb0 0x55dd525b76d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:57.811 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.812+0000 7fd07df64740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dd525aa7d0 0x55dd525b7c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:57.811 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.812+0000 7fd07df64740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dd525b33b0 0x55dd52628d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:57.811 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.812+0000 7fd07df64740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fd0741aca50 con 0x55dd525a9eb0 2024-04-02T14:42:57.811 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.812+0000 7fd07df64740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fd0741ac380 con 0x55dd525b33b0 2024-04-02T14:42:57.812 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.812+0000 7fd07df64740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fd07404f860 con 0x55dd525aa7d0 2024-04-02T14:42:57.812 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.812+0000 7fd07df64740 10 client.0 _ll_register_callbacks cb 0x7fd064004050 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:42:57.812 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.812+0000 7fd06b400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dd525aa7d0 0x55dd525b7c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:57.812 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.812+0000 7fd06b400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dd525aa7d0 0x55dd525b7c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:33092/0 (socket says 172.21.15.90:33092) 2024-04-02T14:42:57.812 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.812+0000 7fd06b400640 1 -- 172.21.15.90:0/2761636805 learned_addr learned my addr 172.21.15.90:0/2761636805 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:42:57.812 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.812+0000 7fd079200640 1 --2- 172.21.15.90:0/2761636805 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dd525b33b0 0x55dd52628d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:57.812 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.812+0000 7fd073e00640 1 --2- 172.21.15.90:0/2761636805 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55dd525a9eb0 0x55dd525b76d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:57.812 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.813+0000 7fd079200640 1 -- 172.21.15.90:0/2761636805 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dd525aa7d0 msgr2=0x55dd525b7c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:42:57.812 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.813+0000 7fd079200640 1 --2- 172.21.15.90:0/2761636805 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dd525aa7d0 0x55dd525b7c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:57.812 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.813+0000 7fd079200640 1 -- 172.21.15.90:0/2761636805 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55dd525a9eb0 msgr2=0x55dd525b76d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:42:57.812 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.813+0000 7fd079200640 1 --2- 172.21.15.90:0/2761636805 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55dd525a9eb0 0x55dd525b76d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:57.812 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.813+0000 7fd079200640 1 -- 172.21.15.90:0/2761636805 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dd522ef620 con 0x55dd525b33b0 2024-04-02T14:42:57.814 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.813+0000 7fd06b400640 1 --2- 172.21.15.90:0/2761636805 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dd525aa7d0 0x55dd525b7c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:42:57.814 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.813+0000 7fd073e00640 1 --2- 172.21.15.90:0/2761636805 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55dd525a9eb0 0x55dd525b76d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:42:57.814 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.813+0000 7fd079200640 1 --2- 172.21.15.90:0/2761636805 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dd525b33b0 0x55dd52628d80 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7fd06c0bb200 tx=0x7fd06c009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d626ad82b1fb9d06 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:42:57.814 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.814+0000 7fd062000640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:42:57.814 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.814+0000 7fd062000640 1 -- 172.21.15.90:0/2761636805 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd06c0c2be0 con 0x55dd525b33b0 2024-04-02T14:42:57.814 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.814+0000 7fd062000640 1 -- 172.21.15.90:0/2761636805 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8550+0+0 (secure 0 0 0) 0x7fd06c0f0030 con 0x55dd525b33b0 2024-04-02T14:42:57.814 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.814+0000 7fd07df64740 1 -- 172.21.15.90:0/2761636805 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd06c0be830 con 0x55dd525b33b0 2024-04-02T14:42:57.814 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.815+0000 7fd062000640 1 -- 172.21.15.90:0/2761636805 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fd06c0f0230 con 0x55dd525b33b0 2024-04-02T14:42:57.814 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.815+0000 7fd07df64740 10 client.27965 fetch_fsmap learned FSMap version 27 2024-04-02T14:42:57.814 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.815+0000 7fd07df64740 1 -- 172.21.15.90:0/2761636805 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55dd5262a3d0 con 0x55dd525b33b0 2024-04-02T14:42:57.815 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.815+0000 7fd062000640 1 -- 172.21.15.90:0/2761636805 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd06c0be830 con 0x55dd525b33b0 2024-04-02T14:42:57.815 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.816+0000 7fd062000640 1 -- 172.21.15.90:0/2761636805 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fd06c0c7010 con 0x55dd525b33b0 2024-04-02T14:42:57.815 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.816+0000 7fd07df64740 10 client.27965 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:42:57.815 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.816+0000 7fd07df64740 10 client.27965 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-02T14:42:57.815 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.816+0000 7fd07df64740 20 client.27965 populate_metadata read hostname 'smithi090' 2024-04-02T14:42:57.815 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.816+0000 7fd07df64740 1 --2- 172.21.15.90:0/2761636805 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55dd5262aac0 0x55dd5262cee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:42:57.816 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.816+0000 7fd07df64740 4 client.27965 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-02T14:42:57.816 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.816+0000 7fd07df64740 1 -- 172.21.15.90:0/2761636805 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55dd5251c2b0 con 0x55dd5262aac0 2024-04-02T14:42:57.816 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.816+0000 7fd073e00640 1 --2- 172.21.15.90:0/2761636805 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55dd5262aac0 0x55dd5262cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:42:57.816 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.817+0000 7fd073e00640 1 --2- 172.21.15.90:0/2761636805 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55dd5262aac0 0x55dd5262cee0 crc :-1 s=READY pgs=223 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ff000d213cbfc7e3 server_cookie=b6d7e90856ab7eb3 in_seq=0 out_seq=0 2024-04-02T14:42:57.816 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.817+0000 7fd062000640 10 client.27965 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:42:57.838 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.840+0000 7fd062000640 1 -- 172.21.15.90:0/2761636805 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55dd5251c2b0 con 0x55dd5262aac0 2024-04-02T14:42:57.838 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.840+0000 7fd062000640 10 client.27965 handle_command_reply: tid=0 2024-04-02T14:42:57.839 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.840+0000 7fd07df64740 1 client.27965 shutdown 2024-04-02T14:42:57.839 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.840+0000 7fd07df64740 1 -- 172.21.15.90:0/2761636805 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dd525b33b0 msgr2=0x55dd52628d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:57.839 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.840+0000 7fd07df64740 1 --2- 172.21.15.90:0/2761636805 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dd525b33b0 0x55dd52628d80 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7fd06c0bb200 tx=0x7fd06c009260 comp rx=0 tx=0).stop 2024-04-02T14:42:57.839 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.841+0000 7fd07df64740 1 -- 172.21.15.90:0/2761636805 shutdown_connections 2024-04-02T14:42:57.840 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.841+0000 7fd07df64740 1 --2- 172.21.15.90:0/2761636805 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55dd5262aac0 0x55dd5262cee0 crc :-1 s=READY pgs=223 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:57.840 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.841+0000 7fd07df64740 1 --2- 172.21.15.90:0/2761636805 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dd525b33b0 0x55dd52628d80 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:57.840 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.841+0000 7fd07df64740 1 --2- 172.21.15.90:0/2761636805 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dd525aa7d0 0x55dd525b7c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:57.840 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.841+0000 7fd062000640 0 client.27965 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:42:57.840 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.841+0000 7fd07df64740 1 --2- 172.21.15.90:0/2761636805 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55dd525a9eb0 0x55dd525b76d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:57.840 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.841+0000 7fd07df64740 1 -- 172.21.15.90:0/2761636805 >> 172.21.15.90:0/2761636805 conn(0x55dd525a45e0 msgr2=0x55dd525b0cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:42:57.840 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.841+0000 7fd07df64740 1 -- 172.21.15.90:0/2761636805 shutdown_connections 2024-04-02T14:42:57.841 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.842+0000 7fd07df64740 1 -- 172.21.15.90:0/2761636805 wait complete. 2024-04-02T14:42:57.841 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.842+0000 7fd07df64740 20 client.27965 trim_cache size 0 max 16384 2024-04-02T14:42:57.841 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:{ 2024-04-02T14:42:57.841 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "message": "", 2024-04-02T14:42:57.841 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "return_code": 0 2024-04-02T14:42:57.841 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:} 2024-04-02T14:42:57.842 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.843+0000 7fd07b200640 1 -- 172.21.15.90:0/3778863884 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fd0580a3ec0 msgr2=0x7fd0580a6380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:57.842 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.843+0000 7fd07b200640 1 --2- 172.21.15.90:0/3778863884 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fd0580a3ec0 0x7fd0580a6380 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fd064005fd0 tx=0x7fd064009500 comp rx=0 tx=0).stop 2024-04-02T14:42:57.842 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.843+0000 7fd07b200640 1 -- 172.21.15.90:0/3778863884 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd074107ac0 msgr2=0x7fd074109480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:42:57.842 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.843+0000 7fd07b200640 1 --2- 172.21.15.90:0/3778863884 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd074107ac0 0x7fd074109480 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7fd06c036040 tx=0x7fd06c004270 comp rx=0 tx=0).stop 2024-04-02T14:42:57.843 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.844+0000 7fd07b200640 1 -- 172.21.15.90:0/3778863884 shutdown_connections 2024-04-02T14:42:57.843 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.844+0000 7fd07b200640 1 --2- 172.21.15.90:0/3778863884 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fd0580a3ec0 0x7fd0580a6380 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:57.843 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.844+0000 7fd07b200640 1 --2- 172.21.15.90:0/3778863884 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd074107ac0 0x7fd074109480 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:57.843 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.844+0000 7fd07b200640 1 --2- 172.21.15.90:0/3778863884 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd074102470 0x7fd074108f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:57.843 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.844+0000 7fd07b200640 1 --2- 172.21.15.90:0/3778863884 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd074101ab0 0x7fd074108a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:42:57.843 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.844+0000 7fd07b200640 1 -- 172.21.15.90:0/3778863884 >> 172.21.15.90:0/3778863884 conn(0x7fd0740fd5c0 msgr2=0x7fd07410d220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:42:57.844 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.845+0000 7fd07b200640 1 -- 172.21.15.90:0/3778863884 shutdown_connections 2024-04-02T14:42:57.844 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:42:57.845+0000 7fd07b200640 1 -- 172.21.15.90:0/3778863884 wait complete. 2024-04-02T14:42:58.434 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:42:58 smithi177 ceph-mon[36431]: pgmap v247: 97 pgs: 97 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 17 MiB/s wr, 760 op/s 2024-04-02T14:42:58.434 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:42:58 smithi090 ceph-mon[31674]: pgmap v247: 97 pgs: 97 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 17 MiB/s wr, 760 op/s 2024-04-02T14:42:58.607 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:42:58 smithi092 ceph-mon[36497]: pgmap v247: 97 pgs: 97 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 17 MiB/s wr, 760 op/s 2024-04-02T14:43:00.433 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:43:00 smithi177 ceph-mon[36431]: pgmap v248: 97 pgs: 97 active+clean; 4.4 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 23 MiB/s wr, 953 op/s 2024-04-02T14:43:00.434 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:43:00 smithi090 ceph-mon[31674]: pgmap v248: 97 pgs: 97 active+clean; 4.4 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 23 MiB/s wr, 953 op/s 2024-04-02T14:43:00.607 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:43:00 smithi092 ceph-mon[36497]: pgmap v248: 97 pgs: 97 active+clean; 4.4 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 23 MiB/s wr, 953 op/s 2024-04-02T14:43:01.828 INFO:teuthology.orchestra.run.smithi090.stderr:progress: 450.0 s, 1861.7 tps, lat 51.263 ms stddev 63.211 2024-04-02T14:43:02.607 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:43:02 smithi092 ceph-mon[36497]: pgmap v249: 97 pgs: 97 active+clean; 4.4 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 20 MiB/s wr, 894 op/s 2024-04-02T14:43:02.683 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:43:02 smithi177 ceph-mon[36431]: pgmap v249: 97 pgs: 97 active+clean; 4.4 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 20 MiB/s wr, 894 op/s 2024-04-02T14:43:02.685 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:43:02 smithi090 ceph-mon[31674]: pgmap v249: 97 pgs: 97 active+clean; 4.4 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 20 MiB/s wr, 894 op/s 2024-04-02T14:43:04.607 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:43:04 smithi092 ceph-mon[36497]: pgmap v250: 97 pgs: 97 active+clean; 4.4 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 16 MiB/s wr, 720 op/s 2024-04-02T14:43:04.682 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:43:04 smithi177 ceph-mon[36431]: pgmap v250: 97 pgs: 97 active+clean; 4.4 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 16 MiB/s wr, 720 op/s 2024-04-02T14:43:04.684 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:43:04 smithi090 ceph-mon[31674]: pgmap v250: 97 pgs: 97 active+clean; 4.4 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 16 MiB/s wr, 720 op/s 2024-04-02T14:43:06.607 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:43:06 smithi092 ceph-mon[36497]: pgmap v251: 97 pgs: 97 active+clean; 4.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 21 MiB/s wr, 938 op/s 2024-04-02T14:43:06.683 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:43:06 smithi177 ceph-mon[36431]: pgmap v251: 97 pgs: 97 active+clean; 4.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 21 MiB/s wr, 938 op/s 2024-04-02T14:43:06.685 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:43:06 smithi090 ceph-mon[31674]: pgmap v251: 97 pgs: 97 active+clean; 4.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 21 MiB/s wr, 938 op/s 2024-04-02T14:43:06.829 INFO:teuthology.orchestra.run.smithi090.stderr:progress: 455.0 s, 172.2 tps, lat 574.555 ms stddev 331.067 2024-04-02T14:43:07.607 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:43:07 smithi092 ceph-mon[36497]: pgmap v252: 97 pgs: 97 active+clean; 4.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 12 MiB/s wr, 504 op/s 2024-04-02T14:43:07.607 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:43:07 smithi092 ceph-mon[36497]: osdmap e94: 12 total, 12 up, 12 in 2024-04-02T14:43:07.682 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:43:07 smithi177 ceph-mon[36431]: pgmap v252: 97 pgs: 97 active+clean; 4.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 12 MiB/s wr, 504 op/s 2024-04-02T14:43:07.683 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:43:07 smithi177 ceph-mon[36431]: osdmap e94: 12 total, 12 up, 12 in 2024-04-02T14:43:07.684 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:43:07 smithi090 ceph-mon[31674]: pgmap v252: 97 pgs: 97 active+clean; 4.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 12 MiB/s wr, 504 op/s 2024-04-02T14:43:07.685 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:43:07 smithi090 ceph-mon[31674]: osdmap e94: 12 total, 12 up, 12 in 2024-04-02T14:43:09.857 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:43:09 smithi092 ceph-mon[36497]: pgmap v254: 97 pgs: 23 active+clean+snaptrim_wait, 14 active+clean+snaptrim, 60 active+clean; 4.4 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 7.5 MiB/s wr, 414 op/s 2024-04-02T14:43:09.932 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:43:09 smithi177 ceph-mon[36431]: pgmap v254: 97 pgs: 23 active+clean+snaptrim_wait, 14 active+clean+snaptrim, 60 active+clean; 4.4 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 7.5 MiB/s wr, 414 op/s 2024-04-02T14:43:09.934 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:43:09 smithi090 ceph-mon[31674]: pgmap v254: 97 pgs: 23 active+clean+snaptrim_wait, 14 active+clean+snaptrim, 60 active+clean; 4.4 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 7.5 MiB/s wr, 414 op/s 2024-04-02T14:43:10.996 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:10.997+0000 7f15f7200640 1 -- 172.21.15.90:0/360939227 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f15f81044b0 msgr2=0x7f15f81109c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:10.996 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:10.997+0000 7f15f7200640 1 --2- 172.21.15.90:0/360939227 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f15f81044b0 0x7f15f81109c0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f15f0009ca0 tx=0x7f15f002c0f0 comp rx=0 tx=0).stop 2024-04-02T14:43:10.996 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:10.998+0000 7f15f7200640 1 -- 172.21.15.90:0/360939227 shutdown_connections 2024-04-02T14:43:10.996 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:10.998+0000 7f15f7200640 1 --2- 172.21.15.90:0/360939227 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f15f81044b0 0x7f15f81109c0 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:10.997 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:10.998+0000 7f15f7200640 1 --2- 172.21.15.90:0/360939227 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f15f8103af0 0x7f15f8103f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:10.997 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:10.998+0000 7f15f7200640 1 --2- 172.21.15.90:0/360939227 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f15f8109b90 0x7f15f8109f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:10.997 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:10.998+0000 7f15f7200640 1 -- 172.21.15.90:0/360939227 >> 172.21.15.90:0/360939227 conn(0x7f15f80ff5e0 msgr2=0x7f15f8101a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:43:10.997 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:10.998+0000 7f15f7200640 1 -- 172.21.15.90:0/360939227 shutdown_connections 2024-04-02T14:43:10.997 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:10.998+0000 7f15f7200640 1 -- 172.21.15.90:0/360939227 wait complete. 2024-04-02T14:43:10.997 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:10.999+0000 7f15f7200640 1 Processor -- start 2024-04-02T14:43:10.997 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:10.999+0000 7f15f7200640 1 -- start start 2024-04-02T14:43:10.998 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.000+0000 7f15f7200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f15f8103af0 0x7f15f819a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:10.998 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.000+0000 7f15f7200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f15f81044b0 0x7f15f819a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:10.998 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.000+0000 7f15f7200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f15f8109b90 0x7f15f819ec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:10.998 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.000+0000 7f15f7200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f15f8113ee0 con 0x7f15f8103af0 2024-04-02T14:43:10.998 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.000+0000 7f15f7200640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f15f8113d60 con 0x7f15f81044b0 2024-04-02T14:43:10.998 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.000+0000 7f15f7200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f15f8114060 con 0x7f15f8109b90 2024-04-02T14:43:10.998 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.000+0000 7f15f5400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f15f81044b0 0x7f15f819a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:10.999 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.000+0000 7f15f5e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f15f8103af0 0x7f15f819a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:10.999 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.000+0000 7f15f5400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f15f81044b0 0x7f15f819a920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:33336/0 (socket says 172.21.15.90:33336) 2024-04-02T14:43:10.999 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.000+0000 7f15f5400640 1 -- 172.21.15.90:0/2108902454 learned_addr learned my addr 172.21.15.90:0/2108902454 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:43:10.999 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.000+0000 7f15f6800640 1 --2- 172.21.15.90:0/2108902454 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f15f8109b90 0x7f15f819ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:11.000 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.000+0000 7f15f5400640 1 -- 172.21.15.90:0/2108902454 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f15f8109b90 msgr2=0x7f15f819ec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:11.000 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.001+0000 7f15f5400640 1 --2- 172.21.15.90:0/2108902454 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f15f8109b90 0x7f15f819ec50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:11.000 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.001+0000 7f15f5400640 1 -- 172.21.15.90:0/2108902454 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f15f8103af0 msgr2=0x7f15f819a3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:11.000 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.001+0000 7f15f5400640 1 --2- 172.21.15.90:0/2108902454 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f15f8103af0 0x7f15f819a3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:11.000 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.001+0000 7f15f5400640 1 -- 172.21.15.90:0/2108902454 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15f819f380 con 0x7f15f81044b0 2024-04-02T14:43:11.001 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.001+0000 7f15f5e00640 1 --2- 172.21.15.90:0/2108902454 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f15f8103af0 0x7f15f819a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:43:11.001 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.001+0000 7f15f6800640 1 --2- 172.21.15.90:0/2108902454 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f15f8109b90 0x7f15f819ec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:43:11.001 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.001+0000 7f15f5400640 1 --2- 172.21.15.90:0/2108902454 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f15f81044b0 0x7f15f819a920 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f15e400e9f0 tx=0x7f15e400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:43:11.001 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.001+0000 7f15eea00640 1 -- 172.21.15.90:0/2108902454 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15e400cd80 con 0x7f15f81044b0 2024-04-02T14:43:11.002 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.001+0000 7f15eea00640 1 -- 172.21.15.90:0/2108902454 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f15e4010910 con 0x7f15f81044b0 2024-04-02T14:43:11.002 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.001+0000 7f15f7200640 1 -- 172.21.15.90:0/2108902454 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f15f819f670 con 0x7f15f81044b0 2024-04-02T14:43:11.002 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.002+0000 7f15eea00640 1 -- 172.21.15.90:0/2108902454 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15e4018980 con 0x7f15f81044b0 2024-04-02T14:43:11.002 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.002+0000 7f15f7200640 1 -- 172.21.15.90:0/2108902454 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15f8075f40 con 0x7f15f81044b0 2024-04-02T14:43:11.004 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.005+0000 7f15eea00640 1 -- 172.21.15.90:0/2108902454 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f15e4010430 con 0x7f15f81044b0 2024-04-02T14:43:11.004 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.006+0000 7f15ffdb1740 1 Processor -- start 2024-04-02T14:43:11.004 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.006+0000 7f15ffdb1740 1 -- start start 2024-04-02T14:43:11.005 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.006+0000 7f15ffdb1740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dfd7ee50e0 0x55dfd7ee54c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:11.005 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.006+0000 7f15ffdb1740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dfd7ee5a00 0x55dfd7eee0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:11.005 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.006+0000 7f15ffdb1740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55dfd7eee5e0 0x55dfd7ef09d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:11.005 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.006+0000 7f15ffdb1740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55dfd7e49d70 con 0x55dfd7eee5e0 2024-04-02T14:43:11.005 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.006+0000 7f15ffdb1740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55dfd7df22a0 con 0x55dfd7ee50e0 2024-04-02T14:43:11.005 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.007+0000 7f15ffdb1740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55dfd7e45d20 con 0x55dfd7ee5a00 2024-04-02T14:43:11.005 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.007+0000 7f15f5e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dfd7ee5a00 0x55dfd7eee0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:11.006 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.007+0000 7f15f5e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dfd7ee5a00 0x55dfd7eee0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:43246/0 (socket says 172.21.15.90:43246) 2024-04-02T14:43:11.006 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.007+0000 7f15f6800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dfd7ee50e0 0x55dfd7ee54c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:11.006 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.007+0000 7f15f5e00640 1 -- 172.21.15.90:0/612633472 learned_addr learned my addr 172.21.15.90:0/612633472 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:43:11.006 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.007+0000 7f15f6800640 1 -- 172.21.15.90:0/612633472 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dfd7ee5a00 msgr2=0x55dfd7eee0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:43:11.006 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.007+0000 7f15f6800640 1 --2- 172.21.15.90:0/612633472 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dfd7ee5a00 0x55dfd7eee0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:11.006 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.007+0000 7f15f6800640 1 -- 172.21.15.90:0/612633472 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55dfd7eee5e0 msgr2=0x55dfd7ef09d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-02T14:43:11.006 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.007+0000 7f15f6800640 1 --2- 172.21.15.90:0/612633472 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55dfd7eee5e0 0x55dfd7ef09d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:11.006 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.007+0000 7f15f6800640 1 -- 172.21.15.90:0/612633472 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dfd7e06840 con 0x55dfd7ee50e0 2024-04-02T14:43:11.007 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.007+0000 7f15f6800640 1 --2- 172.21.15.90:0/612633472 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dfd7ee50e0 0x55dfd7ee54c0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f15f0009bf0 tx=0x7f15f00323d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d9b3ce9970b5b4ab server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:43:11.007 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.008+0000 7f15eea00640 1 --2- 172.21.15.90:0/2108902454 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f15dc0ac620 0x7f15dc0aeae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:11.007 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.008+0000 7f15e3400640 1 -- 172.21.15.90:0/612633472 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15f0044020 con 0x55dfd7ee50e0 2024-04-02T14:43:11.007 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.008+0000 7f15e3400640 1 -- 172.21.15.90:0/612633472 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f15f0003970 con 0x55dfd7ee50e0 2024-04-02T14:43:11.007 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.008+0000 7f15e3400640 1 -- 172.21.15.90:0/612633472 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15f0039040 con 0x55dfd7ee50e0 2024-04-02T14:43:11.007 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.008+0000 7f15eea00640 1 -- 172.21.15.90:0/2108902454 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8594+0+0 (secure 0 0 0) 0x7f15e40d0ae0 con 0x7f15f81044b0 2024-04-02T14:43:11.007 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.008+0000 7f15f5e00640 1 --2- 172.21.15.90:0/2108902454 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f15dc0ac620 0x7f15dc0aeae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:11.008 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.009+0000 7f15ffdb1740 1 -- 172.21.15.90:0/612633472 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dfd7ee50e0 msgr2=0x55dfd7ee54c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:11.008 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.009+0000 7f15f5e00640 1 --2- 172.21.15.90:0/2108902454 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f15dc0ac620 0x7f15dc0aeae0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x55dfd7ee4bb0 tx=0x7f15e800b410 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:43:11.008 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.009+0000 7f15ffdb1740 1 --2- 172.21.15.90:0/612633472 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dfd7ee50e0 0x55dfd7ee54c0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f15f0009bf0 tx=0x7f15f00323d0 comp rx=0 tx=0).stop 2024-04-02T14:43:11.008 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.009+0000 7f15ffdb1740 1 -- 172.21.15.90:0/612633472 shutdown_connections 2024-04-02T14:43:11.008 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.009+0000 7f15ffdb1740 1 --2- 172.21.15.90:0/612633472 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55dfd7eee5e0 0x55dfd7ef09d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:11.009 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.009+0000 7f15ffdb1740 1 --2- 172.21.15.90:0/612633472 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dfd7ee5a00 0x55dfd7eee0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:11.009 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.009+0000 7f15ffdb1740 1 --2- 172.21.15.90:0/612633472 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dfd7ee50e0 0x55dfd7ee54c0 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:11.009 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.009+0000 7f15ffdb1740 1 -- 172.21.15.90:0/612633472 >> 172.21.15.90:0/612633472 conn(0x55dfd7edf810 msgr2=0x55dfd7edfbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:43:11.009 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.010+0000 7f15ffdb1740 1 -- 172.21.15.90:0/612633472 shutdown_connections 2024-04-02T14:43:11.009 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.010+0000 7f15ffdb1740 1 -- 172.21.15.90:0/612633472 wait complete. 2024-04-02T14:43:11.009 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.010+0000 7f15ffdb1740 1 -- start start 2024-04-02T14:43:11.010 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.010+0000 7f15ffdb1740 1 Processor -- start 2024-04-02T14:43:11.010 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.011+0000 7f15ffdb1740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dfd7ee50e0 0x55dfd7eeb700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:11.010 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.011+0000 7f15ffdb1740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dfd7ee5a00 0x55dfd7eebc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:11.010 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.011+0000 7f15ffdb1740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55dfd7eee5e0 0x55dfd7eec180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:11.010 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.011+0000 7f15ffdb1740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55dfd7ef1370 con 0x55dfd7eee5e0 2024-04-02T14:43:11.010 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.011+0000 7f15ffdb1740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55dfd7e05660 con 0x55dfd7ee50e0 2024-04-02T14:43:11.010 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.011+0000 7f15ffdb1740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55dfd7ef14f0 con 0x55dfd7ee5a00 2024-04-02T14:43:11.010 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.011+0000 7f15ffdb1740 10 client.0 _ll_register_callbacks cb 0x55dfd7e05050 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:43:11.010 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.011+0000 7f15f5e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dfd7ee50e0 0x55dfd7eeb700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:11.010 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.011+0000 7f15f5400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dfd7ee5a00 0x55dfd7eebc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:11.011 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.011+0000 7f15f5e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dfd7ee50e0 0x55dfd7eeb700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:33356/0 (socket says 172.21.15.90:33356) 2024-04-02T14:43:11.011 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.011+0000 7f15f5e00640 1 -- 172.21.15.90:0/1957997227 learned_addr learned my addr 172.21.15.90:0/1957997227 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:43:11.011 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.012+0000 7f15f5e00640 1 -- 172.21.15.90:0/1957997227 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dfd7ee5a00 msgr2=0x55dfd7eebc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:43:11.011 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.012+0000 7f15f5e00640 1 --2- 172.21.15.90:0/1957997227 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dfd7ee5a00 0x55dfd7eebc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:11.011 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.012+0000 7f15f5e00640 1 -- 172.21.15.90:0/1957997227 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55dfd7eee5e0 msgr2=0x55dfd7eec180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:43:11.011 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.012+0000 7f15f5e00640 1 --2- 172.21.15.90:0/1957997227 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55dfd7eee5e0 0x55dfd7eec180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:11.011 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.012+0000 7f15f5e00640 1 -- 172.21.15.90:0/1957997227 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dfd7e14e80 con 0x55dfd7ee50e0 2024-04-02T14:43:11.012 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.012+0000 7f15f5e00640 1 --2- 172.21.15.90:0/1957997227 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dfd7ee50e0 0x55dfd7eeb700 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f15e80108b0 tx=0x7f15e80046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fbad736482339cd8 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:43:11.012 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.012+0000 7f15d7e00640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:43:11.012 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.012+0000 7f15d7e00640 1 -- 172.21.15.90:0/1957997227 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15e802c060 con 0x55dfd7ee50e0 2024-04-02T14:43:11.012 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.012+0000 7f15d7e00640 1 -- 172.21.15.90:0/1957997227 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8594+0+0 (secure 0 0 0) 0x7f15e8017030 con 0x55dfd7ee50e0 2024-04-02T14:43:11.013 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.012+0000 7f15ffdb1740 1 -- 172.21.15.90:0/1957997227 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f15f0003970 con 0x55dfd7ee50e0 2024-04-02T14:43:11.013 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.013+0000 7f15d7e00640 1 -- 172.21.15.90:0/1957997227 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f15e8010e90 con 0x55dfd7ee50e0 2024-04-02T14:43:11.013 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.013+0000 7f15ffdb1740 10 client.27977 fetch_fsmap learned FSMap version 27 2024-04-02T14:43:11.013 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.013+0000 7f15ffdb1740 1 -- 172.21.15.90:0/1957997227 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55dfd7b5e210 con 0x55dfd7ee50e0 2024-04-02T14:43:11.013 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.013+0000 7f15d7e00640 1 -- 172.21.15.90:0/1957997227 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f15f0003970 con 0x55dfd7ee50e0 2024-04-02T14:43:11.013 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.013+0000 7f15d7e00640 1 -- 172.21.15.90:0/1957997227 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f15e801b390 con 0x55dfd7ee50e0 2024-04-02T14:43:11.013 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.013+0000 7f15ffdb1740 10 client.27977 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:43:11.013 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.013+0000 7f15ffdb1740 10 client.27977 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-02T14:43:11.013 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.013+0000 7f15ffdb1740 20 client.27977 populate_metadata read hostname 'smithi090' 2024-04-02T14:43:11.013 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.013+0000 7f15ffdb1740 1 --2- 172.21.15.90:0/1957997227 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55dfd7efeab0 0x55dfd7f00f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:11.013 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.013+0000 7f15ffdb1740 4 client.27977 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:43:11.013 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.013+0000 7f15ffdb1740 1 -- 172.21.15.90:0/1957997227 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55dfd7df9cd0 con 0x55dfd7efeab0 2024-04-02T14:43:11.013 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.014+0000 7f15f5400640 1 --2- 172.21.15.90:0/1957997227 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55dfd7efeab0 0x55dfd7f00f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:11.013 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.014+0000 7f15d7e00640 10 client.27977 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:43:11.014 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.014+0000 7f15f5400640 1 --2- 172.21.15.90:0/1957997227 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55dfd7efeab0 0x55dfd7f00f20 crc :-1 s=READY pgs=224 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6fc46aeb517f2577 server_cookie=400a806fe2c37704 in_seq=0 out_seq=0 2024-04-02T14:43:11.015 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.015+0000 7f15d7e00640 1 -- 172.21.15.90:0/1957997227 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55dfd7df9cd0 con 0x55dfd7efeab0 2024-04-02T14:43:11.015 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.015+0000 7f15d7e00640 10 client.27977 handle_command_reply: tid=0 2024-04-02T14:43:11.015 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.015+0000 7f15ffdb1740 1 client.27977 shutdown 2024-04-02T14:43:11.015 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.015+0000 7f15ffdb1740 1 -- 172.21.15.90:0/1957997227 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dfd7ee50e0 msgr2=0x55dfd7eeb700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:11.015 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.015+0000 7f15ffdb1740 1 --2- 172.21.15.90:0/1957997227 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dfd7ee50e0 0x55dfd7eeb700 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f15e80108b0 tx=0x7f15e80046c0 comp rx=0 tx=0).stop 2024-04-02T14:43:11.015 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.015+0000 7f15ffdb1740 1 -- 172.21.15.90:0/1957997227 shutdown_connections 2024-04-02T14:43:11.015 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.015+0000 7f15ffdb1740 1 --2- 172.21.15.90:0/1957997227 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55dfd7efeab0 0x55dfd7f00f20 crc :-1 s=READY pgs=224 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:11.015 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.015+0000 7f15ffdb1740 1 --2- 172.21.15.90:0/1957997227 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55dfd7eee5e0 0x55dfd7eec180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:11.015 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.015+0000 7f15ffdb1740 1 --2- 172.21.15.90:0/1957997227 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dfd7ee5a00 0x55dfd7eebc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:11.015 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.015+0000 7f15ffdb1740 1 --2- 172.21.15.90:0/1957997227 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dfd7ee50e0 0x55dfd7eeb700 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:11.015 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.015+0000 7f15ffdb1740 1 -- 172.21.15.90:0/1957997227 >> 172.21.15.90:0/1957997227 conn(0x55dfd7edf810 msgr2=0x55dfd7ee1560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:43:11.015 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.015+0000 7f15d7e00640 0 client.27977 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:43:11.016 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.016+0000 7f15ffdb1740 1 -- 172.21.15.90:0/1957997227 shutdown_connections 2024-04-02T14:43:11.016 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.016+0000 7f15ffdb1740 1 -- 172.21.15.90:0/1957997227 wait complete. 2024-04-02T14:43:11.016 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.016+0000 7f15ffdb1740 20 client.27977 trim_cache size 0 max 16384 2024-04-02T14:43:11.033 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.034+0000 7f15ffdb1740 1 Processor -- start 2024-04-02T14:43:11.033 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.034+0000 7f15ffdb1740 1 -- start start 2024-04-02T14:43:11.033 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.034+0000 7f15ffdb1740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55dfd7ee50e0 0x55dfd7eede30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:11.033 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.034+0000 7f15ffdb1740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dfd7ee5a00 0x55dfd7efa290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:11.034 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.034+0000 7f15ffdb1740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dfd7eee5e0 0x55dfd7f067f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:11.034 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.034+0000 7f15ffdb1740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f15d800e870 con 0x55dfd7ee50e0 2024-04-02T14:43:11.034 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.034+0000 7f15ffdb1740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f15d80103b0 con 0x55dfd7eee5e0 2024-04-02T14:43:11.034 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.034+0000 7f15ffdb1740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f15d800ef40 con 0x55dfd7ee5a00 2024-04-02T14:43:11.034 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.035+0000 7f15f6800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dfd7eee5e0 0x55dfd7f067f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:11.034 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.035+0000 7f15f5400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dfd7ee5a00 0x55dfd7efa290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:11.034 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.035+0000 7f15f5e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55dfd7ee50e0 0x55dfd7eede30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:11.034 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.035+0000 7f15f6800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dfd7eee5e0 0x55dfd7f067f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:33362/0 (socket says 172.21.15.90:33362) 2024-04-02T14:43:11.034 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.035+0000 7f15f5400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dfd7ee5a00 0x55dfd7efa290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:43256/0 (socket says 172.21.15.90:43256) 2024-04-02T14:43:11.034 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.035+0000 7f15f6800640 1 -- 172.21.15.90:0/347474927 learned_addr learned my addr 172.21.15.90:0/347474927 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:43:11.034 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.035+0000 7f15f6800640 1 -- 172.21.15.90:0/347474927 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dfd7ee5a00 msgr2=0x55dfd7efa290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:43:11.034 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.035+0000 7f15f6800640 1 --2- 172.21.15.90:0/347474927 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dfd7ee5a00 0x55dfd7efa290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:11.034 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.035+0000 7f15f6800640 1 -- 172.21.15.90:0/347474927 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55dfd7ee50e0 msgr2=0x55dfd7eede30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:43:11.035 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.035+0000 7f15f6800640 1 --2- 172.21.15.90:0/347474927 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55dfd7ee50e0 0x55dfd7eede30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:11.035 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.035+0000 7f15f6800640 1 -- 172.21.15.90:0/347474927 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dfd7c2a620 con 0x55dfd7eee5e0 2024-04-02T14:43:11.035 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.035+0000 7f15f5400640 1 --2- 172.21.15.90:0/347474927 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dfd7ee5a00 0x55dfd7efa290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:43:11.035 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.035+0000 7f15f5e00640 1 --2- 172.21.15.90:0/347474927 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55dfd7ee50e0 0x55dfd7eede30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:43:11.038 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.035+0000 7f15f6800640 1 --2- 172.21.15.90:0/347474927 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dfd7eee5e0 0x55dfd7f067f0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f15f002c630 tx=0x7f15f0009a10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cfce6288ae17ba9a server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:43:11.038 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.036+0000 7f15e2a00640 1 -- 172.21.15.90:0/347474927 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15f004d020 con 0x55dfd7eee5e0 2024-04-02T14:43:11.038 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.036+0000 7f15e2a00640 1 -- 172.21.15.90:0/347474927 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f15f0003970 con 0x55dfd7eee5e0 2024-04-02T14:43:11.039 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.036+0000 7f15e2a00640 1 -- 172.21.15.90:0/347474927 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15f0039050 con 0x55dfd7eee5e0 2024-04-02T14:43:11.039 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.037+0000 7f15ffdb1740 1 -- 172.21.15.90:0/347474927 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dfd7eee5e0 msgr2=0x55dfd7f067f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:11.039 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.037+0000 7f15ffdb1740 1 --2- 172.21.15.90:0/347474927 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dfd7eee5e0 0x55dfd7f067f0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f15f002c630 tx=0x7f15f0009a10 comp rx=0 tx=0).stop 2024-04-02T14:43:11.039 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.037+0000 7f15ffdb1740 1 -- 172.21.15.90:0/347474927 shutdown_connections 2024-04-02T14:43:11.039 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.037+0000 7f15ffdb1740 1 --2- 172.21.15.90:0/347474927 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dfd7eee5e0 0x55dfd7f067f0 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:11.039 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.037+0000 7f15ffdb1740 1 --2- 172.21.15.90:0/347474927 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dfd7ee5a00 0x55dfd7efa290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:11.039 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.037+0000 7f15ffdb1740 1 --2- 172.21.15.90:0/347474927 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55dfd7ee50e0 0x55dfd7eede30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:11.039 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.037+0000 7f15ffdb1740 1 -- 172.21.15.90:0/347474927 >> 172.21.15.90:0/347474927 conn(0x55dfd7edf810 msgr2=0x55dfd7ee14d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:43:11.039 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.037+0000 7f15ffdb1740 1 -- 172.21.15.90:0/347474927 shutdown_connections 2024-04-02T14:43:11.039 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.037+0000 7f15ffdb1740 1 -- 172.21.15.90:0/347474927 wait complete. 2024-04-02T14:43:11.039 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.037+0000 7f15ffdb1740 1 -- start start 2024-04-02T14:43:11.039 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.037+0000 7f15ffdb1740 1 Processor -- start 2024-04-02T14:43:11.040 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.038+0000 7f15ffdb1740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dfd7ee50e0 0x55dfd7ef6a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:11.040 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.038+0000 7f15ffdb1740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dfd7ee5a00 0x55dfd7ee7f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:11.040 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.038+0000 7f15ffdb1740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55dfd7eee5e0 0x55dfd7ee84b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:11.040 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.038+0000 7f15ffdb1740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f15d800fce0 con 0x55dfd7eee5e0 2024-04-02T14:43:11.040 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.038+0000 7f15ffdb1740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f15d800f610 con 0x55dfd7ee5a00 2024-04-02T14:43:11.040 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.038+0000 7f15ffdb1740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f15d800c870 con 0x55dfd7ee50e0 2024-04-02T14:43:11.040 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.038+0000 7f15ffdb1740 10 client.0 _ll_register_callbacks cb 0x7f15e8010770 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:43:11.040 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.038+0000 7f15f5400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dfd7ee5a00 0x55dfd7ee7f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:11.040 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.038+0000 7f15f5e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dfd7ee50e0 0x55dfd7ef6a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:11.040 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.038+0000 7f15f5400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dfd7ee5a00 0x55dfd7ee7f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:33368/0 (socket says 172.21.15.90:33368) 2024-04-02T14:43:11.040 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.038+0000 7f15f5400640 1 -- 172.21.15.90:0/746597317 learned_addr learned my addr 172.21.15.90:0/746597317 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:43:11.040 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.038+0000 7f15f6800640 1 --2- 172.21.15.90:0/746597317 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55dfd7eee5e0 0x55dfd7ee84b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:11.040 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.038+0000 7f15f5400640 1 -- 172.21.15.90:0/746597317 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dfd7ee50e0 msgr2=0x55dfd7ef6a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:43:11.041 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.038+0000 7f15f5400640 1 --2- 172.21.15.90:0/746597317 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dfd7ee50e0 0x55dfd7ef6a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:11.041 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.038+0000 7f15f5400640 1 -- 172.21.15.90:0/746597317 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55dfd7eee5e0 msgr2=0x55dfd7ee84b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:43:11.041 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.038+0000 7f15f5400640 1 --2- 172.21.15.90:0/746597317 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55dfd7eee5e0 0x55dfd7ee84b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:11.041 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.038+0000 7f15f5400640 1 -- 172.21.15.90:0/746597317 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dfd7ee8b20 con 0x55dfd7ee5a00 2024-04-02T14:43:11.041 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.038+0000 7f15f5e00640 1 --2- 172.21.15.90:0/746597317 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dfd7ee50e0 0x55dfd7ef6a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:43:11.041 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.039+0000 7f15f5400640 1 --2- 172.21.15.90:0/746597317 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dfd7ee5a00 0x55dfd7ee7f70 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f15e40993a0 tx=0x7f15e4087040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3045239bd535d3e8 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:43:11.041 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.039+0000 7f15d7e00640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:43:11.041 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.039+0000 7f15d7e00640 1 -- 172.21.15.90:0/746597317 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15e40a5640 con 0x55dfd7ee5a00 2024-04-02T14:43:11.041 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.039+0000 7f15d7e00640 1 -- 172.21.15.90:0/746597317 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8594+0+0 (secure 0 0 0) 0x7f15e40ce400 con 0x55dfd7ee5a00 2024-04-02T14:43:11.041 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.039+0000 7f15ffdb1740 1 -- 172.21.15.90:0/746597317 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f15f0003970 con 0x55dfd7ee5a00 2024-04-02T14:43:11.041 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.039+0000 7f15d7e00640 1 -- 172.21.15.90:0/746597317 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f15e40ce600 con 0x55dfd7ee5a00 2024-04-02T14:43:11.041 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.039+0000 7f15ffdb1740 10 client.27983 fetch_fsmap learned FSMap version 27 2024-04-02T14:43:11.041 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.039+0000 7f15ffdb1740 1 -- 172.21.15.90:0/746597317 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55dfd7f64f40 con 0x55dfd7ee5a00 2024-04-02T14:43:11.041 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.039+0000 7f15d7e00640 1 -- 172.21.15.90:0/746597317 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f15f0003970 con 0x55dfd7ee5a00 2024-04-02T14:43:11.041 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.039+0000 7f15d7e00640 1 -- 172.21.15.90:0/746597317 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f15e40870b0 con 0x55dfd7ee5a00 2024-04-02T14:43:11.042 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.039+0000 7f15ffdb1740 10 client.27983 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:43:11.042 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.039+0000 7f15ffdb1740 10 client.27983 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-02T14:43:11.042 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.039+0000 7f15ffdb1740 20 client.27983 populate_metadata read hostname 'smithi090' 2024-04-02T14:43:11.042 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.040+0000 7f15ffdb1740 1 --2- 172.21.15.90:0/746597317 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55dfd7f656c0 0x55dfd7f67ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:11.042 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.040+0000 7f15ffdb1740 4 client.27983 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-02T14:43:11.042 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.040+0000 7f15ffdb1740 1 -- 172.21.15.90:0/746597317 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55dfd7e572b0 con 0x55dfd7f656c0 2024-04-02T14:43:11.042 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.040+0000 7f15f5e00640 1 --2- 172.21.15.90:0/746597317 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55dfd7f656c0 0x55dfd7f67ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:11.042 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.040+0000 7f15f5e00640 1 --2- 172.21.15.90:0/746597317 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55dfd7f656c0 0x55dfd7f67ae0 crc :-1 s=READY pgs=225 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d555fc0f2ada364d server_cookie=e8b9680225d4d76e in_seq=0 out_seq=0 2024-04-02T14:43:11.042 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.040+0000 7f15d7e00640 10 client.27983 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:43:11.065 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.066+0000 7f15d7e00640 1 -- 172.21.15.90:0/746597317 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55dfd7e572b0 con 0x55dfd7f656c0 2024-04-02T14:43:11.065 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.066+0000 7f15d7e00640 10 client.27983 handle_command_reply: tid=0 2024-04-02T14:43:11.065 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.066+0000 7f15ffdb1740 1 client.27983 shutdown 2024-04-02T14:43:11.065 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.066+0000 7f15ffdb1740 1 -- 172.21.15.90:0/746597317 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dfd7ee5a00 msgr2=0x55dfd7ee7f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:11.065 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.066+0000 7f15ffdb1740 1 --2- 172.21.15.90:0/746597317 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dfd7ee5a00 0x55dfd7ee7f70 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f15e40993a0 tx=0x7f15e4087040 comp rx=0 tx=0).stop 2024-04-02T14:43:11.065 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.066+0000 7f15ffdb1740 1 -- 172.21.15.90:0/746597317 shutdown_connections 2024-04-02T14:43:11.065 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.066+0000 7f15ffdb1740 1 --2- 172.21.15.90:0/746597317 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55dfd7f656c0 0x55dfd7f67ae0 crc :-1 s=READY pgs=225 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:11.065 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.066+0000 7f15ffdb1740 1 --2- 172.21.15.90:0/746597317 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55dfd7eee5e0 0x55dfd7ee84b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:11.065 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.066+0000 7f15ffdb1740 1 --2- 172.21.15.90:0/746597317 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dfd7ee5a00 0x55dfd7ee7f70 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:11.065 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.066+0000 7f15ffdb1740 1 --2- 172.21.15.90:0/746597317 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dfd7ee50e0 0x55dfd7ef6a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:11.066 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.066+0000 7f15d7e00640 0 client.27983 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:43:11.066 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.066+0000 7f15ffdb1740 1 -- 172.21.15.90:0/746597317 >> 172.21.15.90:0/746597317 conn(0x55dfd7edf810 msgr2=0x55dfd7ee1490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:43:11.066 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.066+0000 7f15ffdb1740 1 -- 172.21.15.90:0/746597317 shutdown_connections 2024-04-02T14:43:11.066 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.066+0000 7f15ffdb1740 1 -- 172.21.15.90:0/746597317 wait complete. 2024-04-02T14:43:11.066 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.066+0000 7f15ffdb1740 20 client.27983 trim_cache size 0 max 16384 2024-04-02T14:43:11.066 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:{ 2024-04-02T14:43:11.066 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "message": "", 2024-04-02T14:43:11.066 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "return_code": 0 2024-04-02T14:43:11.066 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:} 2024-04-02T14:43:11.067 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.067+0000 7f15f7200640 1 -- 172.21.15.90:0/2108902454 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f15dc0ac620 msgr2=0x7f15dc0aeae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:11.067 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.067+0000 7f15f7200640 1 --2- 172.21.15.90:0/2108902454 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f15dc0ac620 0x7f15dc0aeae0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x55dfd7ee4bb0 tx=0x7f15e800b410 comp rx=0 tx=0).stop 2024-04-02T14:43:11.067 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.067+0000 7f15f7200640 1 -- 172.21.15.90:0/2108902454 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f15f81044b0 msgr2=0x7f15f819a920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:11.067 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.067+0000 7f15f7200640 1 --2- 172.21.15.90:0/2108902454 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f15f81044b0 0x7f15f819a920 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f15e400e9f0 tx=0x7f15e400eeb0 comp rx=0 tx=0).stop 2024-04-02T14:43:11.067 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.067+0000 7f15f7200640 1 -- 172.21.15.90:0/2108902454 shutdown_connections 2024-04-02T14:43:11.067 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.067+0000 7f15f7200640 1 --2- 172.21.15.90:0/2108902454 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f15dc0ac620 0x7f15dc0aeae0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:11.067 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.067+0000 7f15f7200640 1 --2- 172.21.15.90:0/2108902454 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f15f8109b90 0x7f15f819ec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:11.068 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.067+0000 7f15f7200640 1 --2- 172.21.15.90:0/2108902454 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f15f81044b0 0x7f15f819a920 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:11.068 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.067+0000 7f15f7200640 1 --2- 172.21.15.90:0/2108902454 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f15f8103af0 0x7f15f819a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:11.068 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.067+0000 7f15f7200640 1 -- 172.21.15.90:0/2108902454 >> 172.21.15.90:0/2108902454 conn(0x7f15f80ff5e0 msgr2=0x7f15f80ffcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:43:11.068 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.068+0000 7f15f7200640 1 -- 172.21.15.90:0/2108902454 shutdown_connections 2024-04-02T14:43:11.068 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:11.068+0000 7f15f7200640 1 -- 172.21.15.90:0/2108902454 wait complete. 2024-04-02T14:43:11.188 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-02T14:43:11.189 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-02T14:43:11.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.406+0000 7f15b1a00640 1 -- 172.21.15.90:0/2827741283 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f15ac076030 msgr2=0x7f15ac076410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:11.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.406+0000 7f15b1a00640 1 --2- 172.21.15.90:0/2827741283 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f15ac076030 0x7f15ac076410 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f159c009930 tx=0x7f159c02bc30 comp rx=0 tx=0).stop 2024-04-02T14:43:11.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.406+0000 7f15b1a00640 1 -- 172.21.15.90:0/2827741283 shutdown_connections 2024-04-02T14:43:11.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.406+0000 7f15b1a00640 1 --2- 172.21.15.90:0/2827741283 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f15ac112940 0x7f15ac114d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:11.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.406+0000 7f15b1a00640 1 --2- 172.21.15.90:0/2827741283 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f15ac076950 0x7f15ac112400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:11.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.406+0000 7f15b1a00640 1 --2- 172.21.15.90:0/2827741283 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f15ac076030 0x7f15ac076410 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:11.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.406+0000 7f15b1a00640 1 -- 172.21.15.90:0/2827741283 >> 172.21.15.90:0/2827741283 conn(0x7f15ac0ff7b0 msgr2=0x7f15ac101bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:43:11.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.406+0000 7f15b1a00640 1 -- 172.21.15.90:0/2827741283 shutdown_connections 2024-04-02T14:43:11.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.406+0000 7f15b1a00640 1 -- 172.21.15.90:0/2827741283 wait complete. 2024-04-02T14:43:11.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.407+0000 7f15b1a00640 1 Processor -- start 2024-04-02T14:43:11.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.407+0000 7f15b1a00640 1 -- start start 2024-04-02T14:43:11.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.408+0000 7f15b1a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f15ac076030 0x7f15ac1a46e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:11.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.408+0000 7f15b1a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f15ac076950 0x7f15ac1a4c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:11.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.408+0000 7f15b1a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f15ac112940 0x7f15ac19e890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:11.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.408+0000 7f15b1a00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f15ac118490 con 0x7f15ac076030 2024-04-02T14:43:11.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.408+0000 7f15b1a00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f15ac118310 con 0x7f15ac112940 2024-04-02T14:43:11.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.408+0000 7f15b1a00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f15ac118610 con 0x7f15ac076950 2024-04-02T14:43:11.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.408+0000 7f15aa000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f15ac076950 0x7f15ac1a4c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:11.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.408+0000 7f15aaa00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f15ac076030 0x7f15ac1a46e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:11.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.408+0000 7f15ab400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f15ac112940 0x7f15ac19e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:11.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.408+0000 7f15aa000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f15ac076950 0x7f15ac1a4c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:43290/0 (socket says 172.21.15.90:43290) 2024-04-02T14:43:11.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.408+0000 7f15aa000640 1 -- 172.21.15.90:0/2010419183 learned_addr learned my addr 172.21.15.90:0/2010419183 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:43:11.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.408+0000 7f15aa000640 1 -- 172.21.15.90:0/2010419183 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f15ac112940 msgr2=0x7f15ac19e890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:11.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.408+0000 7f15aa000640 1 --2- 172.21.15.90:0/2010419183 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f15ac112940 0x7f15ac19e890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:11.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.408+0000 7f15aa000640 1 -- 172.21.15.90:0/2010419183 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f15ac076030 msgr2=0x7f15ac1a46e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:11.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.408+0000 7f15aa000640 1 --2- 172.21.15.90:0/2010419183 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f15ac076030 0x7f15ac1a46e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:11.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.408+0000 7f15aa000640 1 -- 172.21.15.90:0/2010419183 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15ac19f010 con 0x7f15ac076950 2024-04-02T14:43:11.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.409+0000 7f15ab400640 1 --2- 172.21.15.90:0/2010419183 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f15ac112940 0x7f15ac19e890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:43:11.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.409+0000 7f15aa000640 1 --2- 172.21.15.90:0/2010419183 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f15ac076950 0x7f15ac1a4c20 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f159800e9f0 tx=0x7f159800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:43:11.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.409+0000 7f15a3400640 1 -- 172.21.15.90:0/2010419183 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f159800cd80 con 0x7f15ac076950 2024-04-02T14:43:11.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.409+0000 7f15a3400640 1 -- 172.21.15.90:0/2010419183 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1598010910 con 0x7f15ac076950 2024-04-02T14:43:11.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.409+0000 7f15b1a00640 1 -- 172.21.15.90:0/2010419183 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f15ac19f300 con 0x7f15ac076950 2024-04-02T14:43:11.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.409+0000 7f15a3400640 1 -- 172.21.15.90:0/2010419183 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1598018990 con 0x7f15ac076950 2024-04-02T14:43:11.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.409+0000 7f15b1a00640 1 -- 172.21.15.90:0/2010419183 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15ac1ab5a0 con 0x7f15ac076950 2024-04-02T14:43:11.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.414+0000 7f15b4773740 1 Processor -- start 2024-04-02T14:43:11.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.414+0000 7f15b4773740 1 -- start start 2024-04-02T14:43:11.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.414+0000 7f15b4773740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ddf1b6d7b0 0x55ddf1b6db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:11.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.414+0000 7f15a3400640 1 -- 172.21.15.90:0/2010419183 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f1598010ac0 con 0x7f15ac076950 2024-04-02T14:43:11.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.415+0000 7f15b4773740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ddf1b6e0d0 0x55ddf1b76770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:11.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.415+0000 7f15b4773740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ddf1b76cb0 0x55ddf1b790a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:11.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.415+0000 7f15b4773740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55ddf1ad1770 con 0x55ddf1b6e0d0 2024-04-02T14:43:11.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.415+0000 7f15b4773740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55ddf1a79700 con 0x55ddf1b6d7b0 2024-04-02T14:43:11.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.415+0000 7f15b4773740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55ddf1acd720 con 0x55ddf1b76cb0 2024-04-02T14:43:11.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.415+0000 7f15aa000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ddf1b76cb0 0x55ddf1b790a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:11.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.415+0000 7f15aa000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ddf1b76cb0 0x55ddf1b790a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:43294/0 (socket says 172.21.15.90:43294) 2024-04-02T14:43:11.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.415+0000 7f15aa000640 1 -- 172.21.15.90:0/1591373225 learned_addr learned my addr 172.21.15.90:0/1591373225 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:43:11.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.415+0000 7f15ab400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ddf1b6d7b0 0x55ddf1b6db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:11.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.415+0000 7f15aaa00640 1 --2- 172.21.15.90:0/1591373225 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ddf1b6e0d0 0x55ddf1b76770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:11.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.415+0000 7f15ab400640 1 -- 172.21.15.90:0/1591373225 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ddf1b76cb0 msgr2=0x55ddf1b790a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:43:11.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.415+0000 7f15ab400640 1 --2- 172.21.15.90:0/1591373225 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ddf1b76cb0 0x55ddf1b790a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:11.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.415+0000 7f15ab400640 1 -- 172.21.15.90:0/1591373225 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ddf1b6e0d0 msgr2=0x55ddf1b76770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:43:11.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.415+0000 7f15ab400640 1 --2- 172.21.15.90:0/1591373225 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ddf1b6e0d0 0x55ddf1b76770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:11.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.415+0000 7f15ab400640 1 -- 172.21.15.90:0/1591373225 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ddf18af7d0 con 0x55ddf1b6d7b0 2024-04-02T14:43:11.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.416+0000 7f15ab400640 1 --2- 172.21.15.90:0/1591373225 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ddf1b6d7b0 0x55ddf1b6db90 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f15ac06af90 tx=0x7f15a4011bf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fc15b6de7ade8c87 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:43:11.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.416+0000 7f1597e00640 1 -- 172.21.15.90:0/1591373225 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15a4004d20 con 0x55ddf1b6d7b0 2024-04-02T14:43:11.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.416+0000 7f1597e00640 1 -- 172.21.15.90:0/1591373225 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f15a400f970 con 0x55ddf1b6d7b0 2024-04-02T14:43:11.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.416+0000 7f1597e00640 1 -- 172.21.15.90:0/1591373225 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15a40045b0 con 0x55ddf1b6d7b0 2024-04-02T14:43:11.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.417+0000 7f15a3400640 1 --2- 172.21.15.90:0/2010419183 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f15900a3da0 0x7f15900a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:11.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.417+0000 7f15a3400640 1 -- 172.21.15.90:0/2010419183 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8594+0+0 (secure 0 0 0) 0x7f15980d0ad0 con 0x7f15ac076950 2024-04-02T14:43:11.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.417+0000 7f15b4773740 1 -- 172.21.15.90:0/1591373225 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ddf1b6d7b0 msgr2=0x55ddf1b6db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:11.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.417+0000 7f15b4773740 1 --2- 172.21.15.90:0/1591373225 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ddf1b6d7b0 0x55ddf1b6db90 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f15ac06af90 tx=0x7f15a4011bf0 comp rx=0 tx=0).stop 2024-04-02T14:43:11.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.417+0000 7f15aaa00640 1 --2- 172.21.15.90:0/2010419183 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f15900a3da0 0x7f15900a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:11.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.417+0000 7f15b4773740 1 -- 172.21.15.90:0/1591373225 shutdown_connections 2024-04-02T14:43:11.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.417+0000 7f15b4773740 1 --2- 172.21.15.90:0/1591373225 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ddf1b76cb0 0x55ddf1b790a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:11.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.417+0000 7f15b4773740 1 --2- 172.21.15.90:0/1591373225 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ddf1b6e0d0 0x55ddf1b76770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:11.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.417+0000 7f15b4773740 1 --2- 172.21.15.90:0/1591373225 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ddf1b6d7b0 0x55ddf1b6db90 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:11.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.417+0000 7f15b4773740 1 -- 172.21.15.90:0/1591373225 >> 172.21.15.90:0/1591373225 conn(0x55ddf1b67ee0 msgr2=0x55ddf1b682c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:43:11.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.417+0000 7f15b4773740 1 -- 172.21.15.90:0/1591373225 shutdown_connections 2024-04-02T14:43:11.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.417+0000 7f15b4773740 1 -- 172.21.15.90:0/1591373225 wait complete. 2024-04-02T14:43:11.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.418+0000 7f15aaa00640 1 --2- 172.21.15.90:0/2010419183 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f15900a3da0 0x7f15900a6260 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f159c005ab0 tx=0x7f159c005fd0 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:43:11.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.418+0000 7f15b4773740 1 -- start start 2024-04-02T14:43:11.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.418+0000 7f15b4773740 1 Processor -- start 2024-04-02T14:43:11.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.419+0000 7f15b4773740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ddf1b6d7b0 0x55ddf1b73d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:11.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.419+0000 7f15b4773740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ddf1b6e0d0 0x55ddf1b742c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:11.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.419+0000 7f15b4773740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ddf1b76cb0 0x55ddf1b74800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:11.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.419+0000 7f15b4773740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55ddf1b79aa0 con 0x55ddf1b76cb0 2024-04-02T14:43:11.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.419+0000 7f15b4773740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55ddf1a8cac0 con 0x55ddf1b6d7b0 2024-04-02T14:43:11.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.419+0000 7f15b4773740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55ddf1b79c20 con 0x55ddf1b6e0d0 2024-04-02T14:43:11.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.419+0000 7f15aaa00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ddf1b6d7b0 0x55ddf1b73d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:11.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.419+0000 7f15b4773740 10 client.0 _ll_register_callbacks cb 0x55ddf1a8f180 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:43:11.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.419+0000 7f15aaa00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ddf1b6d7b0 0x55ddf1b73d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:33410/0 (socket says 172.21.15.90:33410) 2024-04-02T14:43:11.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.419+0000 7f15aaa00640 1 -- 172.21.15.90:0/256488684 learned_addr learned my addr 172.21.15.90:0/256488684 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:43:11.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.419+0000 7f15aa000640 1 --2- 172.21.15.90:0/256488684 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ddf1b6e0d0 0x55ddf1b742c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:11.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.419+0000 7f15ab400640 1 --2- 172.21.15.90:0/256488684 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ddf1b76cb0 0x55ddf1b74800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:11.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.420+0000 7f15aaa00640 1 -- 172.21.15.90:0/256488684 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ddf1b6e0d0 msgr2=0x55ddf1b742c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:43:11.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.420+0000 7f15aaa00640 1 --2- 172.21.15.90:0/256488684 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ddf1b6e0d0 0x55ddf1b742c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:11.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.420+0000 7f15aaa00640 1 -- 172.21.15.90:0/256488684 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ddf1b76cb0 msgr2=0x55ddf1b74800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:43:11.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.420+0000 7f15aaa00640 1 --2- 172.21.15.90:0/256488684 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ddf1b76cb0 0x55ddf1b74800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:11.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.420+0000 7f15aaa00640 1 -- 172.21.15.90:0/256488684 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ddf17e53f0 con 0x55ddf1b6d7b0 2024-04-02T14:43:11.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.420+0000 7f15aa000640 1 --2- 172.21.15.90:0/256488684 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ddf1b6e0d0 0x55ddf1b742c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:43:11.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.420+0000 7f15aaa00640 1 --2- 172.21.15.90:0/256488684 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ddf1b6d7b0 0x55ddf1b73d80 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f159c035000 tx=0x7f159c009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=414e7fcf0c19286c server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:43:11.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.420+0000 7f1594c00640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:43:11.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.420+0000 7f1594c00640 1 -- 172.21.15.90:0/256488684 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f159c04f070 con 0x55ddf1b6d7b0 2024-04-02T14:43:11.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.420+0000 7f1594c00640 1 -- 172.21.15.90:0/256488684 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8594+0+0 (secure 0 0 0) 0x7f159c004f50 con 0x55ddf1b6d7b0 2024-04-02T14:43:11.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.420+0000 7f15b4773740 1 -- 172.21.15.90:0/256488684 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f15a400f970 con 0x55ddf1b6d7b0 2024-04-02T14:43:11.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.421+0000 7f1594c00640 1 -- 172.21.15.90:0/256488684 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f159c005150 con 0x55ddf1b6d7b0 2024-04-02T14:43:11.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.421+0000 7f15b4773740 10 client.27995 fetch_fsmap learned FSMap version 27 2024-04-02T14:43:11.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.421+0000 7f15b4773740 1 -- 172.21.15.90:0/256488684 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55ddf17a50f0 con 0x55ddf1b6d7b0 2024-04-02T14:43:11.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.421+0000 7f1594c00640 1 -- 172.21.15.90:0/256488684 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f15a400f970 con 0x55ddf1b6d7b0 2024-04-02T14:43:11.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.421+0000 7f1594c00640 1 -- 172.21.15.90:0/256488684 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f159c003ce0 con 0x55ddf1b6d7b0 2024-04-02T14:43:11.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.421+0000 7f15b4773740 10 client.27995 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:43:11.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.421+0000 7f15b4773740 10 client.27995 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:43:11.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.421+0000 7f15b4773740 20 client.27995 populate_metadata read hostname 'smithi090' 2024-04-02T14:43:11.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.421+0000 7f15b4773740 1 --2- 172.21.15.90:0/256488684 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55ddf1b86cd0 0x55ddf1b89140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:11.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.421+0000 7f15b4773740 4 client.27995 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:43:11.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.421+0000 7f15b4773740 1 -- 172.21.15.90:0/256488684 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ddf1a72630 con 0x55ddf1b86cd0 2024-04-02T14:43:11.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.422+0000 7f15aa000640 1 --2- 172.21.15.90:0/256488684 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55ddf1b86cd0 0x55ddf1b89140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:11.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.422+0000 7f15aa000640 1 --2- 172.21.15.90:0/256488684 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55ddf1b86cd0 0x55ddf1b89140 crc :-1 s=READY pgs=226 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=47519209db5062ff server_cookie=b443b9b3439d4e7c in_seq=0 out_seq=0 2024-04-02T14:43:11.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.422+0000 7f1594c00640 10 client.27995 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:43:11.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.423+0000 7f1594c00640 1 -- 172.21.15.90:0/256488684 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ddf1a72630 con 0x55ddf1b86cd0 2024-04-02T14:43:11.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.423+0000 7f1594c00640 10 client.27995 handle_command_reply: tid=0 2024-04-02T14:43:11.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.423+0000 7f15b4773740 1 client.27995 shutdown 2024-04-02T14:43:11.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.423+0000 7f15b4773740 1 -- 172.21.15.90:0/256488684 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ddf1b6d7b0 msgr2=0x55ddf1b73d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:11.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.423+0000 7f15b4773740 1 --2- 172.21.15.90:0/256488684 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ddf1b6d7b0 0x55ddf1b73d80 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f159c035000 tx=0x7f159c009260 comp rx=0 tx=0).stop 2024-04-02T14:43:11.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.423+0000 7f15b4773740 1 -- 172.21.15.90:0/256488684 shutdown_connections 2024-04-02T14:43:11.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.423+0000 7f15b4773740 1 --2- 172.21.15.90:0/256488684 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55ddf1b86cd0 0x55ddf1b89140 crc :-1 s=READY pgs=226 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:11.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.423+0000 7f15b4773740 1 --2- 172.21.15.90:0/256488684 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ddf1b76cb0 0x55ddf1b74800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:11.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.423+0000 7f15b4773740 1 --2- 172.21.15.90:0/256488684 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ddf1b6e0d0 0x55ddf1b742c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:11.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.423+0000 7f15b4773740 1 --2- 172.21.15.90:0/256488684 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ddf1b6d7b0 0x55ddf1b73d80 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:11.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.423+0000 7f1594c00640 0 client.27995 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:43:11.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.423+0000 7f15b4773740 1 -- 172.21.15.90:0/256488684 >> 172.21.15.90:0/256488684 conn(0x55ddf1b67ee0 msgr2=0x55ddf1b69c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:43:11.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.423+0000 7f15b4773740 1 -- 172.21.15.90:0/256488684 shutdown_connections 2024-04-02T14:43:11.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.423+0000 7f15b4773740 1 -- 172.21.15.90:0/256488684 wait complete. 2024-04-02T14:43:11.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.424+0000 7f15b4773740 20 client.27995 trim_cache size 0 max 16384 2024-04-02T14:43:11.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.446+0000 7f15b4773740 1 Processor -- start 2024-04-02T14:43:11.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.446+0000 7f15b4773740 1 -- start start 2024-04-02T14:43:11.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.446+0000 7f15b4773740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ddf1b6d7b0 0x55ddf1b764b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:11.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.447+0000 7f15b4773740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ddf1b6e0d0 0x55ddf1b82960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:11.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.447+0000 7f15b4773740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ddf1b76cb0 0x55ddf1b69a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:11.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.447+0000 7f15b4773740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f15ac1b4020 con 0x55ddf1b6e0d0 2024-04-02T14:43:11.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.447+0000 7f15b4773740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f15ac1b5b60 con 0x55ddf1b6d7b0 2024-04-02T14:43:11.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.447+0000 7f15b4773740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f15ac1b46f0 con 0x55ddf1b76cb0 2024-04-02T14:43:11.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.447+0000 7f15aaa00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ddf1b6d7b0 0x55ddf1b764b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:11.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.447+0000 7f15aaa00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ddf1b6d7b0 0x55ddf1b764b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:33412/0 (socket says 172.21.15.90:33412) 2024-04-02T14:43:11.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.447+0000 7f15aaa00640 1 -- 172.21.15.90:0/2056419062 learned_addr learned my addr 172.21.15.90:0/2056419062 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:43:11.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.447+0000 7f15aa000640 1 --2- 172.21.15.90:0/2056419062 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ddf1b6e0d0 0x55ddf1b82960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:11.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.447+0000 7f15ab400640 1 --2- 172.21.15.90:0/2056419062 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ddf1b76cb0 0x55ddf1b69a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:11.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.447+0000 7f15aaa00640 1 -- 172.21.15.90:0/2056419062 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ddf1b76cb0 msgr2=0x55ddf1b69a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:43:11.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.447+0000 7f15aaa00640 1 --2- 172.21.15.90:0/2056419062 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ddf1b76cb0 0x55ddf1b69a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:11.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.447+0000 7f15aaa00640 1 -- 172.21.15.90:0/2056419062 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ddf1b6e0d0 msgr2=0x55ddf1b82960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:43:11.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.447+0000 7f15aaa00640 1 --2- 172.21.15.90:0/2056419062 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ddf1b6e0d0 0x55ddf1b82960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:11.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.447+0000 7f15aaa00640 1 -- 172.21.15.90:0/2056419062 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ddf1b82ea0 con 0x55ddf1b6d7b0 2024-04-02T14:43:11.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.448+0000 7f15aaa00640 1 --2- 172.21.15.90:0/2056419062 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ddf1b6d7b0 0x55ddf1b764b0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f159c02c170 tx=0x7f159c02cdd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c1129fe5d737ff5f server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:43:11.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.448+0000 7f1597400640 1 -- 172.21.15.90:0/2056419062 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f159c063070 con 0x55ddf1b6d7b0 2024-04-02T14:43:11.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.448+0000 7f1597400640 1 -- 172.21.15.90:0/2056419062 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f159c004a90 con 0x55ddf1b6d7b0 2024-04-02T14:43:11.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.448+0000 7f1597400640 1 -- 172.21.15.90:0/2056419062 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f159c04f040 con 0x55ddf1b6d7b0 2024-04-02T14:43:11.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.449+0000 7f15b4773740 1 -- 172.21.15.90:0/2056419062 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ddf1b6d7b0 msgr2=0x55ddf1b764b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:11.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.449+0000 7f15b4773740 1 --2- 172.21.15.90:0/2056419062 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ddf1b6d7b0 0x55ddf1b764b0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f159c02c170 tx=0x7f159c02cdd0 comp rx=0 tx=0).stop 2024-04-02T14:43:11.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.449+0000 7f15b4773740 1 -- 172.21.15.90:0/2056419062 shutdown_connections 2024-04-02T14:43:11.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.449+0000 7f15b4773740 1 --2- 172.21.15.90:0/2056419062 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ddf1b76cb0 0x55ddf1b69a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:11.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.449+0000 7f15b4773740 1 --2- 172.21.15.90:0/2056419062 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ddf1b6e0d0 0x55ddf1b82960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:11.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.449+0000 7f15b4773740 1 --2- 172.21.15.90:0/2056419062 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ddf1b6d7b0 0x55ddf1b764b0 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:11.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.449+0000 7f15b4773740 1 -- 172.21.15.90:0/2056419062 >> 172.21.15.90:0/2056419062 conn(0x55ddf1b67ee0 msgr2=0x55ddf1b69310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:43:11.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.449+0000 7f15b4773740 1 -- 172.21.15.90:0/2056419062 shutdown_connections 2024-04-02T14:43:11.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.449+0000 7f15b4773740 1 -- 172.21.15.90:0/2056419062 wait complete. 2024-04-02T14:43:11.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.450+0000 7f15b4773740 1 -- start start 2024-04-02T14:43:11.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.450+0000 7f15b4773740 1 Processor -- start 2024-04-02T14:43:11.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.451+0000 7f15b4773740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ddf1b6d7b0 0x55ddf1b7f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:11.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.451+0000 7f15b4773740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ddf1b6e0d0 0x55ddf1b70650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:11.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.451+0000 7f15b4773740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ddf1b76cb0 0x55ddf1b70b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:11.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.451+0000 7f15b4773740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f15ac1b5490 con 0x55ddf1b76cb0 2024-04-02T14:43:11.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.451+0000 7f15b4773740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f15ac1b4dc0 con 0x55ddf1b6d7b0 2024-04-02T14:43:11.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.451+0000 7f15b4773740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f15ac1b2020 con 0x55ddf1b6e0d0 2024-04-02T14:43:11.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.451+0000 7f15b4773740 10 client.0 _ll_register_callbacks cb 0x7f1598005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:43:11.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.451+0000 7f15aaa00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ddf1b6d7b0 0x55ddf1b7f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:11.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.451+0000 7f15aa000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ddf1b6e0d0 0x55ddf1b70650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:11.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.451+0000 7f15aaa00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ddf1b6d7b0 0x55ddf1b7f1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:33428/0 (socket says 172.21.15.90:33428) 2024-04-02T14:43:11.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.451+0000 7f15aaa00640 1 -- 172.21.15.90:0/713057999 learned_addr learned my addr 172.21.15.90:0/713057999 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:43:11.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.451+0000 7f15ab400640 1 --2- 172.21.15.90:0/713057999 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ddf1b76cb0 0x55ddf1b70b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:11.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.452+0000 7f15aaa00640 1 -- 172.21.15.90:0/713057999 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ddf1b6e0d0 msgr2=0x55ddf1b70650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:43:11.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.452+0000 7f15aaa00640 1 --2- 172.21.15.90:0/713057999 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ddf1b6e0d0 0x55ddf1b70650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:11.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.452+0000 7f15aaa00640 1 -- 172.21.15.90:0/713057999 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ddf1b76cb0 msgr2=0x55ddf1b70b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:43:11.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.452+0000 7f15aaa00640 1 --2- 172.21.15.90:0/713057999 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ddf1b76cb0 0x55ddf1b70b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:11.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.452+0000 7f15aaa00640 1 -- 172.21.15.90:0/713057999 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ddf1b71340 con 0x55ddf1b6d7b0 2024-04-02T14:43:11.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.452+0000 7f15ab400640 1 --2- 172.21.15.90:0/713057999 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ddf1b76cb0 0x55ddf1b70b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:43:11.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.452+0000 7f15aa000640 1 --2- 172.21.15.90:0/713057999 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ddf1b6e0d0 0x55ddf1b70650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:43:11.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.452+0000 7f15aaa00640 1 --2- 172.21.15.90:0/713057999 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ddf1b6d7b0 0x55ddf1b7f1b0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f159c04b4a0 tx=0x7f159c04b690 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c2800a0a3e295ce3 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:43:11.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.452+0000 7f1594c00640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:43:11.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.452+0000 7f1594c00640 1 -- 172.21.15.90:0/713057999 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f159c04f070 con 0x55ddf1b6d7b0 2024-04-02T14:43:11.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.452+0000 7f1594c00640 1 -- 172.21.15.90:0/713057999 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8594+0+0 (secure 0 0 0) 0x7f159c005500 con 0x55ddf1b6d7b0 2024-04-02T14:43:11.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.452+0000 7f15b4773740 1 -- 172.21.15.90:0/713057999 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f159c004a90 con 0x55ddf1b6d7b0 2024-04-02T14:43:11.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.453+0000 7f1594c00640 1 -- 172.21.15.90:0/713057999 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f159c076050 con 0x55ddf1b6d7b0 2024-04-02T14:43:11.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.453+0000 7f15b4773740 10 client.28001 fetch_fsmap learned FSMap version 27 2024-04-02T14:43:11.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.453+0000 7f15b4773740 1 -- 172.21.15.90:0/713057999 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55ddf1bed210 con 0x55ddf1b6d7b0 2024-04-02T14:43:11.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.453+0000 7f1594c00640 1 -- 172.21.15.90:0/713057999 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f159c004a90 con 0x55ddf1b6d7b0 2024-04-02T14:43:11.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.453+0000 7f1594c00640 1 -- 172.21.15.90:0/713057999 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f159c05f580 con 0x55ddf1b6d7b0 2024-04-02T14:43:11.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.453+0000 7f15b4773740 10 client.28001 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:43:11.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.453+0000 7f15b4773740 10 client.28001 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:43:11.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.453+0000 7f15b4773740 20 client.28001 populate_metadata read hostname 'smithi090' 2024-04-02T14:43:11.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.453+0000 7f15b4773740 1 --2- 172.21.15.90:0/713057999 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55ddf1bed860 0x55ddf1befc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:11.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.453+0000 7f15b4773740 4 client.28001 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-02T14:43:11.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.453+0000 7f15b4773740 1 -- 172.21.15.90:0/713057999 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55ddf1adecb0 con 0x55ddf1bed860 2024-04-02T14:43:11.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.453+0000 7f15aa000640 1 --2- 172.21.15.90:0/713057999 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55ddf1bed860 0x55ddf1befc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:11.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.454+0000 7f15aa000640 1 --2- 172.21.15.90:0/713057999 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55ddf1bed860 0x55ddf1befc80 crc :-1 s=READY pgs=227 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a2fc5bc23fe98073 server_cookie=a891ae114631be03 in_seq=0 out_seq=0 2024-04-02T14:43:11.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.454+0000 7f1594c00640 10 client.28001 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:43:11.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.454+0000 7f1594c00640 1 -- 172.21.15.90:0/713057999 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55ddf1adecb0 con 0x55ddf1bed860 2024-04-02T14:43:11.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.454+0000 7f1594c00640 10 client.28001 handle_command_reply: tid=0 2024-04-02T14:43:11.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.454+0000 7f15b4773740 1 client.28001 shutdown 2024-04-02T14:43:11.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.454+0000 7f15b4773740 1 -- 172.21.15.90:0/713057999 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ddf1b6d7b0 msgr2=0x55ddf1b7f1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:11.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.454+0000 7f15b4773740 1 --2- 172.21.15.90:0/713057999 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ddf1b6d7b0 0x55ddf1b7f1b0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f159c04b4a0 tx=0x7f159c04b690 comp rx=0 tx=0).stop 2024-04-02T14:43:11.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.454+0000 7f15b4773740 1 -- 172.21.15.90:0/713057999 shutdown_connections 2024-04-02T14:43:11.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.454+0000 7f15b4773740 1 --2- 172.21.15.90:0/713057999 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55ddf1bed860 0x55ddf1befc80 crc :-1 s=READY pgs=227 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:11.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.454+0000 7f15b4773740 1 --2- 172.21.15.90:0/713057999 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ddf1b76cb0 0x55ddf1b70b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:11.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.454+0000 7f15b4773740 1 --2- 172.21.15.90:0/713057999 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ddf1b6e0d0 0x55ddf1b70650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:11.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.454+0000 7f1594c00640 0 client.28001 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:43:11.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.454+0000 7f15b4773740 1 --2- 172.21.15.90:0/713057999 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ddf1b6d7b0 0x55ddf1b7f1b0 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:11.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.454+0000 7f15b4773740 1 -- 172.21.15.90:0/713057999 >> 172.21.15.90:0/713057999 conn(0x55ddf1b67ee0 msgr2=0x55ddf1b66f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:43:11.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.455+0000 7f15b4773740 1 -- 172.21.15.90:0/713057999 shutdown_connections 2024-04-02T14:43:11.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.455+0000 7f15b4773740 1 -- 172.21.15.90:0/713057999 wait complete. 2024-04-02T14:43:11.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.455+0000 7f15b4773740 20 client.28001 trim_cache size 0 max 16384 2024-04-02T14:43:11.457 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-02T14:43:11.457 INFO:teuthology.orchestra.run.smithi090.stdout: "status": "no active scrubs running", 2024-04-02T14:43:11.457 INFO:teuthology.orchestra.run.smithi090.stdout: "scrubs": {} 2024-04-02T14:43:11.457 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-02T14:43:11.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.456+0000 7f15b1a00640 1 -- 172.21.15.90:0/2010419183 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f15900a3da0 msgr2=0x7f15900a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:11.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.456+0000 7f15b1a00640 1 --2- 172.21.15.90:0/2010419183 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f15900a3da0 0x7f15900a6260 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f159c005ab0 tx=0x7f159c005fd0 comp rx=0 tx=0).stop 2024-04-02T14:43:11.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.456+0000 7f15b1a00640 1 -- 172.21.15.90:0/2010419183 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f15ac076950 msgr2=0x7f15ac1a4c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:11.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.456+0000 7f15b1a00640 1 --2- 172.21.15.90:0/2010419183 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f15ac076950 0x7f15ac1a4c20 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f159800e9f0 tx=0x7f159800eeb0 comp rx=0 tx=0).stop 2024-04-02T14:43:11.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.456+0000 7f15b1a00640 1 -- 172.21.15.90:0/2010419183 shutdown_connections 2024-04-02T14:43:11.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.456+0000 7f15b1a00640 1 --2- 172.21.15.90:0/2010419183 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f15900a3da0 0x7f15900a6260 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:11.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.456+0000 7f15b1a00640 1 --2- 172.21.15.90:0/2010419183 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f15ac112940 0x7f15ac19e890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:11.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.456+0000 7f15b1a00640 1 --2- 172.21.15.90:0/2010419183 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f15ac076950 0x7f15ac1a4c20 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:11.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.456+0000 7f15b1a00640 1 --2- 172.21.15.90:0/2010419183 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f15ac076030 0x7f15ac1a46e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:11.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.456+0000 7f15b1a00640 1 -- 172.21.15.90:0/2010419183 >> 172.21.15.90:0/2010419183 conn(0x7f15ac0ff7b0 msgr2=0x7f15ac101ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:43:11.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.456+0000 7f15b1a00640 1 -- 172.21.15.90:0/2010419183 shutdown_connections 2024-04-02T14:43:11.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.456+0000 7f15b1a00640 1 -- 172.21.15.90:0/2010419183 wait complete. 2024-04-02T14:43:11.476 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-02T14:43:11.476 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T14:43:11.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.698+0000 7f9328200640 1 -- 172.21.15.90:0/573965903 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9320102d70 msgr2=0x7f9320103150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:11.697 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.698+0000 7f9328200640 1 --2- 172.21.15.90:0/573965903 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9320102d70 0x7f9320103150 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f9314009860 tx=0x7f931402bb70 comp rx=0 tx=0).stop 2024-04-02T14:43:11.697 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.698+0000 7f9328200640 1 -- 172.21.15.90:0/573965903 shutdown_connections 2024-04-02T14:43:11.697 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.698+0000 7f9328200640 1 --2- 172.21.15.90:0/573965903 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f932010c140 0x7f932010e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:11.697 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.698+0000 7f9328200640 1 --2- 172.21.15.90:0/573965903 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9320103690 0x7f932010bb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:11.697 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.698+0000 7f9328200640 1 --2- 172.21.15.90:0/573965903 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9320102d70 0x7f9320103150 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:11.697 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.698+0000 7f9328200640 1 -- 172.21.15.90:0/573965903 >> 172.21.15.90:0/573965903 conn(0x7f9320077b00 msgr2=0x7f9320077f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:43:11.697 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.698+0000 7f9328200640 1 -- 172.21.15.90:0/573965903 shutdown_connections 2024-04-02T14:43:11.697 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.698+0000 7f9328200640 1 -- 172.21.15.90:0/573965903 wait complete. 2024-04-02T14:43:11.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.699+0000 7f9328200640 1 Processor -- start 2024-04-02T14:43:11.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.699+0000 7f9328200640 1 -- start start 2024-04-02T14:43:11.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.699+0000 7f9328200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9320102d70 0x7f932019a640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:11.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.700+0000 7f9328200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9320103690 0x7f932019ab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:11.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.700+0000 7f9328200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f932010c140 0x7f932019ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:11.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.700+0000 7f9328200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f9320111f40 con 0x7f932010c140 2024-04-02T14:43:11.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.700+0000 7f9328200640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f9320111dc0 con 0x7f9320103690 2024-04-02T14:43:11.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.700+0000 7f9328200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f93201120c0 con 0x7f9320102d70 2024-04-02T14:43:11.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.700+0000 7f9325800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9320102d70 0x7f932019a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:11.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.700+0000 7f9325800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9320102d70 0x7f932019a640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:43344/0 (socket says 172.21.15.90:43344) 2024-04-02T14:43:11.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.700+0000 7f9325800640 1 -- 172.21.15.90:0/3289185302 learned_addr learned my addr 172.21.15.90:0/3289185302 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:43:11.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.700+0000 7f9324e00640 1 --2- 172.21.15.90:0/3289185302 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9320103690 0x7f932019ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:11.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.700+0000 7f9326200640 1 --2- 172.21.15.90:0/3289185302 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f932010c140 0x7f932019ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:11.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.700+0000 7f9325800640 1 -- 172.21.15.90:0/3289185302 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9320103690 msgr2=0x7f932019ab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:11.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.700+0000 7f9325800640 1 --2- 172.21.15.90:0/3289185302 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9320103690 0x7f932019ab80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:11.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.700+0000 7f9325800640 1 -- 172.21.15.90:0/3289185302 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f932010c140 msgr2=0x7f932019ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:11.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.700+0000 7f9325800640 1 --2- 172.21.15.90:0/3289185302 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f932010c140 0x7f932019ef00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:11.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.700+0000 7f9325800640 1 -- 172.21.15.90:0/3289185302 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f932019f630 con 0x7f9320102d70 2024-04-02T14:43:11.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.701+0000 7f9325800640 1 --2- 172.21.15.90:0/3289185302 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9320102d70 0x7f932019a640 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f9314037040 tx=0x7f93140045b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:43:11.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.701+0000 7f931e000640 1 -- 172.21.15.90:0/3289185302 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9314041020 con 0x7f9320102d70 2024-04-02T14:43:11.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.701+0000 7f931e000640 1 -- 172.21.15.90:0/3289185302 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f931402c9f0 con 0x7f9320102d70 2024-04-02T14:43:11.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.701+0000 7f931e000640 1 -- 172.21.15.90:0/3289185302 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9314039040 con 0x7f9320102d70 2024-04-02T14:43:11.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.701+0000 7f9328200640 1 -- 172.21.15.90:0/3289185302 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f932019f8c0 con 0x7f9320102d70 2024-04-02T14:43:11.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.701+0000 7f9328200640 1 -- 172.21.15.90:0/3289185302 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f93201a71b0 con 0x7f9320102d70 2024-04-02T14:43:11.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.702+0000 7f9328200640 1 -- 172.21.15.90:0/3289185302 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f93200688f0 con 0x7f9320102d70 2024-04-02T14:43:11.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.704+0000 7f931e000640 1 -- 172.21.15.90:0/3289185302 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f9314044020 con 0x7f9320102d70 2024-04-02T14:43:11.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.706+0000 7f931e000640 1 --2- 172.21.15.90:0/3289185302 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f93080a3da0 0x7f93080a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:11.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.706+0000 7f931e000640 1 -- 172.21.15.90:0/3289185302 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8594+0+0 (secure 0 0 0) 0x7f93140f1ec0 con 0x7f9320102d70 2024-04-02T14:43:11.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.706+0000 7f9324e00640 1 --2- 172.21.15.90:0/3289185302 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f93080a3da0 0x7f93080a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:11.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.707+0000 7f9324e00640 1 --2- 172.21.15.90:0/3289185302 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f93080a3da0 0x7f93080a6260 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f932010b270 tx=0x7f930c009260 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:43:11.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:11.710+0000 7f931e000640 1 -- 172.21.15.90:0/3289185302 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7f931404b020 con 0x7f9320102d70 2024-04-02T14:43:11.828 INFO:teuthology.orchestra.run.smithi090.stderr:progress: 460.0 s, 63.4 tps, lat 1568.525 ms stddev 1823.786 2024-04-02T14:43:12.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.004+0000 7f9328200640 1 -- 172.21.15.90:0/3289185302 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f93201a73e0 con 0x7f9320102d70 2024-04-02T14:43:12.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.006+0000 7f931e000640 1 -- 172.21.15.90:0/3289185302 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11927 (secure 0 0 0) 0x7f93140be5c0 con 0x7f9320102d70 2024-04-02T14:43:12.005 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-02T14:43:12.005 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14412,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6837/2852160309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6836","nonce":2852160309},{"type":"v1","addr":"172.21.15.90:6837","nonce":2852160309}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14478,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6841/1919376858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6840","nonce":1919376858},{"type":"v1","addr":"172.21.15.90:6841","nonce":1919376858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-02T14:32:22.319083+0000","modified":"2024-04-02T14:34:52.266152+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24590,"mds_1":24560,"mds_2":24547,"mds_3":24539,"mds_4":24523},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24454":{"gid":24454,"name":"l","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6833/1564319431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":1564319431},{"type":"v1","addr":"172.21.15.177:6833","nonce":1564319431}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.92:6833/1570337372","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":1570337372},{"type":"v1","addr":"172.21.15.92:6833","nonce":1570337372}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"j","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.90:6835/1277124891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6834","nonce":1277124891},{"type":"v1","addr":"172.21.15.90:6835","nonce":1277124891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.92:6835/860721164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":860721164},{"type":"v1","addr":"172.21.15.92:6835","nonce":860721164}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24511":{"gid":24511,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6835/3177244988","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":3177244988},{"type":"v1","addr":"172.21.15.177:6835","nonce":3177244988}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"c","rank":4,"incarnation":25,"state":"up:active","state_seq":9,"addr":"172.21.15.177:6837/4223414858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":4223414858},{"type":"v1","addr":"172.21.15.177:6837","nonce":4223414858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24539":{"gid":24539,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":8,"addr":"172.21.15.92:6837/1270635939","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":1270635939},{"type":"v1","addr":"172.21.15.92:6837","nonce":1270635939}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24547":{"gid":24547,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":7,"addr":"172.21.15.177:6839/664895663","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":664895663},{"type":"v1","addr":"172.21.15.177:6839","nonce":664895663}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"g","rank":1,"incarnation":19,"state":"up:active","state_seq":5,"addr":"172.21.15.90:6839/1498915870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6838","nonce":1498915870},{"type":"v1","addr":"172.21.15.90:6839","nonce":1498915870}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24590":{"gid":24590,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.92:6839/1054888219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":1054888219},{"type":"v1","addr":"172.21.15.92:6839","nonce":1054888219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24590,"qdb_cluster":[24547,24590,24523,24539,24560]},"id":1}]} 2024-04-02T14:43:12.005 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 27 2024-04-02T14:43:12.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.009+0000 7f9328200640 1 -- 172.21.15.90:0/3289185302 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f93080a3da0 msgr2=0x7f93080a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:12.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.009+0000 7f9328200640 1 --2- 172.21.15.90:0/3289185302 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f93080a3da0 0x7f93080a6260 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f932010b270 tx=0x7f930c009260 comp rx=0 tx=0).stop 2024-04-02T14:43:12.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.010+0000 7f9328200640 1 -- 172.21.15.90:0/3289185302 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9320102d70 msgr2=0x7f932019a640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:12.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.010+0000 7f9328200640 1 --2- 172.21.15.90:0/3289185302 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9320102d70 0x7f932019a640 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f9314037040 tx=0x7f93140045b0 comp rx=0 tx=0).stop 2024-04-02T14:43:12.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.010+0000 7f9328200640 1 -- 172.21.15.90:0/3289185302 shutdown_connections 2024-04-02T14:43:12.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.010+0000 7f9328200640 1 --2- 172.21.15.90:0/3289185302 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f93080a3da0 0x7f93080a6260 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:12.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.010+0000 7f9328200640 1 --2- 172.21.15.90:0/3289185302 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f932010c140 0x7f932019ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:12.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.010+0000 7f9328200640 1 --2- 172.21.15.90:0/3289185302 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9320103690 0x7f932019ab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:12.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.010+0000 7f9328200640 1 --2- 172.21.15.90:0/3289185302 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9320102d70 0x7f932019a640 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:12.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.010+0000 7f9328200640 1 -- 172.21.15.90:0/3289185302 >> 172.21.15.90:0/3289185302 conn(0x7f9320077b00 msgr2=0x7f9320107f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:43:12.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.010+0000 7f9328200640 1 -- 172.21.15.90:0/3289185302 shutdown_connections 2024-04-02T14:43:12.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.010+0000 7f9328200640 1 -- 172.21.15.90:0/3289185302 wait complete. 2024-04-02T14:43:12.027 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-04-02T14:43:12.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.250+0000 7f3174200640 1 -- 172.21.15.90:0/4284837299 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f316c074850 msgr2=0x7f316c10dd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:12.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.250+0000 7f3174200640 1 --2- 172.21.15.90:0/4284837299 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f316c074850 0x7f316c10dd90 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f3158009900 tx=0x7f315802bcd0 comp rx=0 tx=0).stop 2024-04-02T14:43:12.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.250+0000 7f3174200640 1 -- 172.21.15.90:0/4284837299 shutdown_connections 2024-04-02T14:43:12.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.250+0000 7f3174200640 1 --2- 172.21.15.90:0/4284837299 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f316c10e2d0 0x7f316c110720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:12.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.250+0000 7f3174200640 1 --2- 172.21.15.90:0/4284837299 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f316c074850 0x7f316c10dd90 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:12.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.250+0000 7f3174200640 1 --2- 172.21.15.90:0/4284837299 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f316c073f30 0x7f316c074310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:12.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.250+0000 7f3174200640 1 -- 172.21.15.90:0/4284837299 >> 172.21.15.90:0/4284837299 conn(0x7f316c0fd540 msgr2=0x7f316c0ff960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:43:12.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.250+0000 7f3174200640 1 -- 172.21.15.90:0/4284837299 shutdown_connections 2024-04-02T14:43:12.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.250+0000 7f3174200640 1 -- 172.21.15.90:0/4284837299 wait complete. 2024-04-02T14:43:12.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.251+0000 7f3174200640 1 Processor -- start 2024-04-02T14:43:12.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.251+0000 7f3174200640 1 -- start start 2024-04-02T14:43:12.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.252+0000 7f3174200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f316c073f30 0x7f316c19a590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:12.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.252+0000 7f3174200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f316c074850 0x7f316c19aad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:12.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.252+0000 7f3174200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f316c10e2d0 0x7f316c19ee50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:12.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.252+0000 7f3174200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f316c114070 con 0x7f316c074850 2024-04-02T14:43:12.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.252+0000 7f3174200640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f316c113ef0 con 0x7f316c10e2d0 2024-04-02T14:43:12.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.252+0000 7f3174200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f316c1141f0 con 0x7f316c073f30 2024-04-02T14:43:12.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.252+0000 7f3171800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f316c073f30 0x7f316c19a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:12.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.252+0000 7f3172200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f316c10e2d0 0x7f316c19ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:12.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.252+0000 7f3172200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f316c10e2d0 0x7f316c19ee50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:33452/0 (socket says 172.21.15.90:33452) 2024-04-02T14:43:12.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.252+0000 7f3172200640 1 -- 172.21.15.90:0/3285888551 learned_addr learned my addr 172.21.15.90:0/3285888551 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:43:12.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.252+0000 7f3170e00640 1 --2- 172.21.15.90:0/3285888551 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f316c074850 0x7f316c19aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:12.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.252+0000 7f3171800640 1 -- 172.21.15.90:0/3285888551 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f316c10e2d0 msgr2=0x7f316c19ee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:12.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.252+0000 7f3171800640 1 --2- 172.21.15.90:0/3285888551 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f316c10e2d0 0x7f316c19ee50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:12.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.252+0000 7f3171800640 1 -- 172.21.15.90:0/3285888551 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f316c074850 msgr2=0x7f316c19aad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:12.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.252+0000 7f3171800640 1 --2- 172.21.15.90:0/3285888551 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f316c074850 0x7f316c19aad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:12.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.252+0000 7f3171800640 1 -- 172.21.15.90:0/3285888551 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f316c19f580 con 0x7f316c073f30 2024-04-02T14:43:12.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.252+0000 7f3172200640 1 --2- 172.21.15.90:0/3285888551 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f316c10e2d0 0x7f316c19ee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:43:12.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.253+0000 7f3171800640 1 --2- 172.21.15.90:0/3285888551 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f316c073f30 0x7f316c19a590 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f316000e9f0 tx=0x7f316000eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:43:12.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.253+0000 7f316a000640 1 -- 172.21.15.90:0/3285888551 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f316000ccc0 con 0x7f316c073f30 2024-04-02T14:43:12.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.253+0000 7f316a000640 1 -- 172.21.15.90:0/3285888551 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3160010910 con 0x7f316c073f30 2024-04-02T14:43:12.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.253+0000 7f3174200640 1 -- 172.21.15.90:0/3285888551 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f316c19f870 con 0x7f316c073f30 2024-04-02T14:43:12.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.253+0000 7f316a000640 1 -- 172.21.15.90:0/3285888551 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3160018960 con 0x7f316c073f30 2024-04-02T14:43:12.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.253+0000 7f3174200640 1 -- 172.21.15.90:0/3285888551 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f316c1a7100 con 0x7f316c073f30 2024-04-02T14:43:12.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.256+0000 7f3176f6c740 1 Processor -- start 2024-04-02T14:43:12.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.256+0000 7f3176f6c740 1 -- start start 2024-04-02T14:43:12.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.257+0000 7f316a000640 1 -- 172.21.15.90:0/3285888551 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f3160010ac0 con 0x7f316c073f30 2024-04-02T14:43:12.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.257+0000 7f3176f6c740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5641c38f99f0 0x5641c38f8740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:12.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.257+0000 7f3176f6c740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5641c38f8d10 0x5641c38f9190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:12.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.257+0000 7f3176f6c740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5641c3902080 0x5641c3904470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:12.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.257+0000 7f3176f6c740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5641c3839400 con 0x5641c38f99f0 2024-04-02T14:43:12.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.257+0000 7f3176f6c740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x5641c381df10 con 0x5641c38f8d10 2024-04-02T14:43:12.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.257+0000 7f3176f6c740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5641c385d6f0 con 0x5641c3902080 2024-04-02T14:43:12.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.257+0000 7f3171800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5641c38f8d10 0x5641c38f9190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:12.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.257+0000 7f3171800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5641c38f8d10 0x5641c38f9190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:33464/0 (socket says 172.21.15.90:33464) 2024-04-02T14:43:12.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.257+0000 7f3172200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5641c38f99f0 0x5641c38f8740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:12.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.257+0000 7f3171800640 1 -- 172.21.15.90:0/748297779 learned_addr learned my addr 172.21.15.90:0/748297779 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:43:12.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.257+0000 7f3170e00640 1 --2- 172.21.15.90:0/748297779 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5641c3902080 0x5641c3904470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:12.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.258+0000 7f3171800640 1 -- 172.21.15.90:0/748297779 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5641c3902080 msgr2=0x5641c3904470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:43:12.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.258+0000 7f3171800640 1 --2- 172.21.15.90:0/748297779 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5641c3902080 0x5641c3904470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:12.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.258+0000 7f3171800640 1 -- 172.21.15.90:0/748297779 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5641c38f99f0 msgr2=0x5641c38f8740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:43:12.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.258+0000 7f3171800640 1 --2- 172.21.15.90:0/748297779 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5641c38f99f0 0x5641c38f8740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:12.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.258+0000 7f3171800640 1 -- 172.21.15.90:0/748297779 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5641c35713f0 con 0x5641c38f8d10 2024-04-02T14:43:12.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.258+0000 7f3171800640 1 --2- 172.21.15.90:0/748297779 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5641c38f8d10 0x5641c38f9190 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f31600ce1c0 tx=0x7f31600d96a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=645441b91dfe59a2 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:43:12.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.258+0000 7f315ea00640 1 -- 172.21.15.90:0/748297779 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31600e0020 con 0x5641c38f8d10 2024-04-02T14:43:12.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.258+0000 7f315ea00640 1 -- 172.21.15.90:0/748297779 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f31600dcb30 con 0x5641c38f8d10 2024-04-02T14:43:12.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.258+0000 7f315ea00640 1 -- 172.21.15.90:0/748297779 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31600dbb30 con 0x5641c38f8d10 2024-04-02T14:43:12.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.258+0000 7f316a000640 1 --2- 172.21.15.90:0/3285888551 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f31540a3d20 0x7f31540a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:12.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.259+0000 7f316a000640 1 -- 172.21.15.90:0/3285888551 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8594+0+0 (secure 0 0 0) 0x7f31600d0be0 con 0x7f316c073f30 2024-04-02T14:43:12.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.259+0000 7f3172200640 1 --2- 172.21.15.90:0/3285888551 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f31540a3d20 0x7f31540a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:12.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.259+0000 7f3176f6c740 1 -- 172.21.15.90:0/748297779 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5641c38f8d10 msgr2=0x5641c38f9190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:12.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.259+0000 7f3176f6c740 1 --2- 172.21.15.90:0/748297779 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5641c38f8d10 0x5641c38f9190 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f31600ce1c0 tx=0x7f31600d96a0 comp rx=0 tx=0).stop 2024-04-02T14:43:12.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.260+0000 7f3176f6c740 1 -- 172.21.15.90:0/748297779 shutdown_connections 2024-04-02T14:43:12.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.260+0000 7f3176f6c740 1 --2- 172.21.15.90:0/748297779 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5641c3902080 0x5641c3904470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:12.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.260+0000 7f3176f6c740 1 --2- 172.21.15.90:0/748297779 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5641c38f8d10 0x5641c38f9190 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:12.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.260+0000 7f3176f6c740 1 --2- 172.21.15.90:0/748297779 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5641c38f99f0 0x5641c38f8740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:12.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.260+0000 7f3176f6c740 1 -- 172.21.15.90:0/748297779 >> 172.21.15.90:0/748297779 conn(0x5641c38bfa50 msgr2=0x5641c38f3760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:43:12.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.260+0000 7f3172200640 1 --2- 172.21.15.90:0/3285888551 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f31540a3d20 0x7f31540a61e0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f31640049d0 tx=0x7f3164009500 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:43:12.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.260+0000 7f3176f6c740 1 -- 172.21.15.90:0/748297779 shutdown_connections 2024-04-02T14:43:12.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.260+0000 7f3176f6c740 1 -- 172.21.15.90:0/748297779 wait complete. 2024-04-02T14:43:12.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.260+0000 7f3176f6c740 1 -- start start 2024-04-02T14:43:12.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.261+0000 7f3176f6c740 1 Processor -- start 2024-04-02T14:43:12.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.261+0000 7f3176f6c740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5641c38f8d10 0x5641c38ff060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:12.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.261+0000 7f3176f6c740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5641c38f99f0 0x5641c38ff5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:12.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.261+0000 7f3176f6c740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5641c3902080 0x5641c38ffae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:12.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.261+0000 7f3176f6c740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5641c38596a0 con 0x5641c38f99f0 2024-04-02T14:43:12.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.261+0000 7f3176f6c740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x5641c3805980 con 0x5641c3902080 2024-04-02T14:43:12.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.261+0000 7f3176f6c740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5641c3818cb0 con 0x5641c38f8d10 2024-04-02T14:43:12.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.261+0000 7f3176f6c740 10 client.0 _ll_register_callbacks cb 0x5641c38186a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:43:12.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.261+0000 7f3170e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5641c38f99f0 0x5641c38ff5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:12.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.262+0000 7f3171800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5641c38f8d10 0x5641c38ff060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:12.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.262+0000 7f3170e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5641c38f99f0 0x5641c38ff5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:37382/0 (socket says 172.21.15.90:37382) 2024-04-02T14:43:12.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.262+0000 7f3170e00640 1 -- 172.21.15.90:0/1237495154 learned_addr learned my addr 172.21.15.90:0/1237495154 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:43:12.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.262+0000 7f3172200640 1 --2- 172.21.15.90:0/1237495154 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5641c3902080 0x5641c38ffae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:12.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.262+0000 7f3171800640 1 -- 172.21.15.90:0/1237495154 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5641c3902080 msgr2=0x5641c38ffae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:43:12.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.262+0000 7f3171800640 1 --2- 172.21.15.90:0/1237495154 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5641c3902080 0x5641c38ffae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:12.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.262+0000 7f3171800640 1 -- 172.21.15.90:0/1237495154 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5641c38f99f0 msgr2=0x5641c38ff5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:43:12.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.262+0000 7f3171800640 1 --2- 172.21.15.90:0/1237495154 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5641c38f99f0 0x5641c38ff5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:12.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.262+0000 7f3171800640 1 -- 172.21.15.90:0/1237495154 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5641c363b7d0 con 0x5641c38f8d10 2024-04-02T14:43:12.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.262+0000 7f3171800640 1 --2- 172.21.15.90:0/1237495154 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5641c38f8d10 0x5641c38ff060 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f31600d0fd0 tx=0x7f31600d4f60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=429d34c1b373de4 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:43:12.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.262+0000 7f3153400640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:43:12.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.263+0000 7f3153400640 1 -- 172.21.15.90:0/1237495154 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31600e0030 con 0x5641c38f8d10 2024-04-02T14:43:12.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.263+0000 7f3153400640 1 -- 172.21.15.90:0/1237495154 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8594+0+0 (secure 0 0 0) 0x7f31600bd0f0 con 0x5641c38f8d10 2024-04-02T14:43:12.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.263+0000 7f3176f6c740 1 -- 172.21.15.90:0/1237495154 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f31600dcb30 con 0x5641c38f8d10 2024-04-02T14:43:12.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.263+0000 7f3153400640 1 -- 172.21.15.90:0/1237495154 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f3160098e20 con 0x5641c38f8d10 2024-04-02T14:43:12.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.263+0000 7f3176f6c740 10 client.27853 fetch_fsmap learned FSMap version 27 2024-04-02T14:43:12.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.263+0000 7f3176f6c740 1 -- 172.21.15.90:0/1237495154 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5641c3910f20 con 0x5641c38f8d10 2024-04-02T14:43:12.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.263+0000 7f3153400640 1 -- 172.21.15.90:0/1237495154 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f31600dcb30 con 0x5641c38f8d10 2024-04-02T14:43:12.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.263+0000 7f3153400640 1 -- 172.21.15.90:0/1237495154 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f31600dcce0 con 0x5641c38f8d10 2024-04-02T14:43:12.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.263+0000 7f3176f6c740 10 client.27853 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:43:12.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.263+0000 7f3176f6c740 10 client.27853 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.c 2024-04-02T14:43:12.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.264+0000 7f3176f6c740 20 client.27853 populate_metadata read hostname 'smithi090' 2024-04-02T14:43:12.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.264+0000 7f3176f6c740 1 --2- 172.21.15.90:0/1237495154 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x5641c3912270 0x5641c3914660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:12.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.264+0000 7f3176f6c740 4 client.27853 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:43:12.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.264+0000 7f3176f6c740 1 -- 172.21.15.90:0/1237495154 --> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5641c381d040 con 0x5641c3912270 2024-04-02T14:43:12.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.264+0000 7f3170e00640 1 --2- 172.21.15.90:0/1237495154 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x5641c3912270 0x5641c3914660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:12.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.264+0000 7f3153400640 10 client.27853 ms_handle_connect on v2:172.21.15.177:6836/4223414858 2024-04-02T14:43:12.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.264+0000 7f3170e00640 1 --2- 172.21.15.90:0/1237495154 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x5641c3912270 0x5641c3914660 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=28a8421c5a781d18 server_cookie=ce6b7adb2af2f887 in_seq=0 out_seq=0 2024-04-02T14:43:12.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.265+0000 7f3153400640 1 -- 172.21.15.90:0/1237495154 <== mds.4 v2:172.21.15.177:6836/4223414858 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5641c381d040 con 0x5641c3912270 2024-04-02T14:43:12.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.265+0000 7f3153400640 10 client.27853 handle_command_reply: tid=0 2024-04-02T14:43:12.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.265+0000 7f3176f6c740 1 client.27853 shutdown 2024-04-02T14:43:12.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.265+0000 7f3176f6c740 1 -- 172.21.15.90:0/1237495154 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5641c38f8d10 msgr2=0x5641c38ff060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:12.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.265+0000 7f3176f6c740 1 --2- 172.21.15.90:0/1237495154 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5641c38f8d10 0x5641c38ff060 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f31600d0fd0 tx=0x7f31600d4f60 comp rx=0 tx=0).stop 2024-04-02T14:43:12.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.266+0000 7f3176f6c740 1 -- 172.21.15.90:0/1237495154 shutdown_connections 2024-04-02T14:43:12.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.266+0000 7f3176f6c740 1 --2- 172.21.15.90:0/1237495154 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5641c3902080 0x5641c38ffae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:12.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.266+0000 7f3176f6c740 1 --2- 172.21.15.90:0/1237495154 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x5641c3912270 0x5641c3914660 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:12.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.266+0000 7f3176f6c740 1 --2- 172.21.15.90:0/1237495154 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5641c38f99f0 0x5641c38ff5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:12.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.266+0000 7f3176f6c740 1 --2- 172.21.15.90:0/1237495154 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5641c38f8d10 0x5641c38ff060 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:12.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.266+0000 7f3153400640 0 client.27853 ms_handle_reset on v2:172.21.15.177:6836/4223414858 2024-04-02T14:43:12.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.266+0000 7f3176f6c740 1 -- 172.21.15.90:0/1237495154 >> 172.21.15.90:0/1237495154 conn(0x5641c38bfa50 msgr2=0x5641c38f54b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:43:12.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.266+0000 7f3176f6c740 1 -- 172.21.15.90:0/1237495154 shutdown_connections 2024-04-02T14:43:12.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.266+0000 7f3176f6c740 1 -- 172.21.15.90:0/1237495154 wait complete. 2024-04-02T14:43:12.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.266+0000 7f3176f6c740 20 client.27853 trim_cache size 0 max 16384 2024-04-02T14:43:12.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.282+0000 7f3176f6c740 1 Processor -- start 2024-04-02T14:43:12.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.282+0000 7f3176f6c740 1 -- start start 2024-04-02T14:43:12.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.282+0000 7f3176f6c740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5641c38f8d10 0x5641c3901790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:12.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.282+0000 7f3176f6c740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5641c38f99f0 0x5641c38fe960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:12.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.282+0000 7f3176f6c740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5641c3902080 0x5641c38f4bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:12.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.282+0000 7f3176f6c740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f316c1afbe0 con 0x5641c38f99f0 2024-04-02T14:43:12.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.282+0000 7f3176f6c740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f316c1b1720 con 0x5641c3902080 2024-04-02T14:43:12.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.282+0000 7f3176f6c740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f316c1b02b0 con 0x5641c38f8d10 2024-04-02T14:43:12.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.283+0000 7f3172200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5641c3902080 0x5641c38f4bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:12.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.283+0000 7f3171800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5641c38f8d10 0x5641c3901790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:12.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.283+0000 7f3170e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5641c38f99f0 0x5641c38fe960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:12.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.283+0000 7f3172200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5641c3902080 0x5641c38f4bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:33488/0 (socket says 172.21.15.90:33488) 2024-04-02T14:43:12.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.283+0000 7f3172200640 1 -- 172.21.15.90:0/2092696516 learned_addr learned my addr 172.21.15.90:0/2092696516 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:43:12.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.283+0000 7f3171800640 1 -- 172.21.15.90:0/2092696516 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5641c3902080 msgr2=0x5641c38f4bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:43:12.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.283+0000 7f3171800640 1 --2- 172.21.15.90:0/2092696516 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5641c3902080 0x5641c38f4bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:12.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.283+0000 7f3171800640 1 -- 172.21.15.90:0/2092696516 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5641c38f99f0 msgr2=0x5641c38fe960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:43:12.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.283+0000 7f3171800640 1 --2- 172.21.15.90:0/2092696516 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5641c38f99f0 0x5641c38fe960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:12.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.283+0000 7f3171800640 1 -- 172.21.15.90:0/2092696516 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5641c38f5a60 con 0x5641c38f8d10 2024-04-02T14:43:12.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.283+0000 7f3172200640 1 --2- 172.21.15.90:0/2092696516 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5641c3902080 0x5641c38f4bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:43:12.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.283+0000 7f3171800640 1 --2- 172.21.15.90:0/2092696516 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5641c38f8d10 0x5641c3901790 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f3160099550 tx=0x7f316009b090 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3150980eff1c5738 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:43:12.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.284+0000 7f315e000640 1 -- 172.21.15.90:0/2092696516 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31600e0070 con 0x5641c38f8d10 2024-04-02T14:43:12.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.284+0000 7f315e000640 1 -- 172.21.15.90:0/2092696516 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f31600a0e80 con 0x5641c38f8d10 2024-04-02T14:43:12.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.284+0000 7f315e000640 1 -- 172.21.15.90:0/2092696516 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f316009c4e0 con 0x5641c38f8d10 2024-04-02T14:43:12.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.284+0000 7f3176f6c740 1 -- 172.21.15.90:0/2092696516 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5641c38f8d10 msgr2=0x5641c3901790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:12.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.284+0000 7f3176f6c740 1 --2- 172.21.15.90:0/2092696516 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5641c38f8d10 0x5641c3901790 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f3160099550 tx=0x7f316009b090 comp rx=0 tx=0).stop 2024-04-02T14:43:12.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.285+0000 7f3176f6c740 1 -- 172.21.15.90:0/2092696516 shutdown_connections 2024-04-02T14:43:12.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.285+0000 7f3176f6c740 1 --2- 172.21.15.90:0/2092696516 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5641c3902080 0x5641c38f4bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:12.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.285+0000 7f3176f6c740 1 --2- 172.21.15.90:0/2092696516 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5641c38f99f0 0x5641c38fe960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:12.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.285+0000 7f3176f6c740 1 --2- 172.21.15.90:0/2092696516 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5641c38f8d10 0x5641c3901790 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:12.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.285+0000 7f3176f6c740 1 -- 172.21.15.90:0/2092696516 >> 172.21.15.90:0/2092696516 conn(0x5641c38bfa50 msgr2=0x5641c390d890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:43:12.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.285+0000 7f3176f6c740 1 -- 172.21.15.90:0/2092696516 shutdown_connections 2024-04-02T14:43:12.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.285+0000 7f3176f6c740 1 -- 172.21.15.90:0/2092696516 wait complete. 2024-04-02T14:43:12.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.285+0000 7f3176f6c740 1 -- start start 2024-04-02T14:43:12.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.285+0000 7f3176f6c740 1 Processor -- start 2024-04-02T14:43:12.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.286+0000 7f3176f6c740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5641c38f8d10 0x5641c39175b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:12.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.286+0000 7f3176f6c740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5641c38f99f0 0x5641c3917af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:12.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.286+0000 7f3176f6c740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5641c3902080 0x5641c3912aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:12.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.286+0000 7f3176f6c740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f316c1b1050 con 0x5641c38f8d10 2024-04-02T14:43:12.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.286+0000 7f3176f6c740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f316c1b0980 con 0x5641c3902080 2024-04-02T14:43:12.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.286+0000 7f3176f6c740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f316c1adbc0 con 0x5641c38f99f0 2024-04-02T14:43:12.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.286+0000 7f3176f6c740 10 client.0 _ll_register_callbacks cb 0x7f316000ce60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:43:12.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.286+0000 7f3172200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5641c3902080 0x5641c3912aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:12.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.286+0000 7f3171800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5641c38f8d10 0x5641c39175b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:12.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.286+0000 7f3170e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5641c38f99f0 0x5641c3917af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:12.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.286+0000 7f3172200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5641c3902080 0x5641c3912aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:33496/0 (socket says 172.21.15.90:33496) 2024-04-02T14:43:12.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.286+0000 7f3172200640 1 -- 172.21.15.90:0/4164757314 learned_addr learned my addr 172.21.15.90:0/4164757314 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:43:12.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.286+0000 7f3172200640 1 -- 172.21.15.90:0/4164757314 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5641c38f99f0 msgr2=0x5641c3917af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:43:12.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.286+0000 7f3172200640 1 --2- 172.21.15.90:0/4164757314 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5641c38f99f0 0x5641c3917af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:12.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.286+0000 7f3172200640 1 -- 172.21.15.90:0/4164757314 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5641c38f8d10 msgr2=0x5641c39175b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:43:12.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.286+0000 7f3172200640 1 --2- 172.21.15.90:0/4164757314 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5641c38f8d10 0x5641c39175b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:12.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.286+0000 7f3172200640 1 -- 172.21.15.90:0/4164757314 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5641c38f3970 con 0x5641c3902080 2024-04-02T14:43:12.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.286+0000 7f3170e00640 1 --2- 172.21.15.90:0/4164757314 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5641c38f99f0 0x5641c3917af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:43:12.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.286+0000 7f3171800640 1 --2- 172.21.15.90:0/4164757314 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5641c38f8d10 0x5641c39175b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:43:12.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.287+0000 7f3172200640 1 --2- 172.21.15.90:0/4164757314 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5641c3902080 0x5641c3912aa0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f316400ff20 tx=0x7f3164004ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6c28ebeb348f2a1e server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:43:12.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.287+0000 7f3153400640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:43:12.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.287+0000 7f3153400640 1 -- 172.21.15.90:0/4164757314 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3164004b20 con 0x5641c3902080 2024-04-02T14:43:12.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.287+0000 7f3176f6c740 1 -- 172.21.15.90:0/4164757314 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f31600a0e80 con 0x5641c3902080 2024-04-02T14:43:12.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.287+0000 7f3153400640 1 -- 172.21.15.90:0/4164757314 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8594+0+0 (secure 0 0 0) 0x7f3164004cc0 con 0x5641c3902080 2024-04-02T14:43:12.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.287+0000 7f3153400640 1 -- 172.21.15.90:0/4164757314 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f3164005020 con 0x5641c3902080 2024-04-02T14:43:12.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.287+0000 7f3176f6c740 10 client.28025 fetch_fsmap learned FSMap version 27 2024-04-02T14:43:12.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.287+0000 7f3176f6c740 1 -- 172.21.15.90:0/4164757314 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5641c3977e40 con 0x5641c3902080 2024-04-02T14:43:12.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.287+0000 7f3153400640 1 -- 172.21.15.90:0/4164757314 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f31600a0e80 con 0x5641c3902080 2024-04-02T14:43:12.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.288+0000 7f3153400640 1 -- 172.21.15.90:0/4164757314 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f3164029070 con 0x5641c3902080 2024-04-02T14:43:12.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.288+0000 7f3176f6c740 10 client.28025 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:43:12.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.288+0000 7f3176f6c740 10 client.28025 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.c 2024-04-02T14:43:12.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.288+0000 7f3176f6c740 20 client.28025 populate_metadata read hostname 'smithi090' 2024-04-02T14:43:12.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.288+0000 7f3176f6c740 1 --2- 172.21.15.90:0/4164757314 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x5641c3978640 0x5641c397aa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:12.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.288+0000 7f3176f6c740 4 client.28025 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-02T14:43:12.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.288+0000 7f3176f6c740 1 -- 172.21.15.90:0/4164757314 --> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5641c386ac30 con 0x5641c3978640 2024-04-02T14:43:12.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.288+0000 7f3171800640 1 --2- 172.21.15.90:0/4164757314 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x5641c3978640 0x5641c397aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:12.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.288+0000 7f3171800640 1 --2- 172.21.15.90:0/4164757314 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x5641c3978640 0x5641c397aa60 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=83a0b1d0c7736a54 server_cookie=da43cc15490b6f17 in_seq=0 out_seq=0 2024-04-02T14:43:12.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.288+0000 7f3153400640 10 client.28025 ms_handle_connect on v2:172.21.15.177:6836/4223414858 2024-04-02T14:43:12.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.289+0000 7f3153400640 1 -- 172.21.15.90:0/4164757314 <== mds.4 v2:172.21.15.177:6836/4223414858 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5641c386ac30 con 0x5641c3978640 2024-04-02T14:43:12.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.289+0000 7f3153400640 10 client.28025 handle_command_reply: tid=0 2024-04-02T14:43:12.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.289+0000 7f3176f6c740 1 client.28025 shutdown 2024-04-02T14:43:12.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.289+0000 7f3176f6c740 1 -- 172.21.15.90:0/4164757314 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5641c3902080 msgr2=0x5641c3912aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:12.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.289+0000 7f3176f6c740 1 --2- 172.21.15.90:0/4164757314 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5641c3902080 0x5641c3912aa0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f316400ff20 tx=0x7f3164004ab0 comp rx=0 tx=0).stop 2024-04-02T14:43:12.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.289+0000 7f3176f6c740 1 -- 172.21.15.90:0/4164757314 shutdown_connections 2024-04-02T14:43:12.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.289+0000 7f3176f6c740 1 --2- 172.21.15.90:0/4164757314 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5641c3902080 0x5641c3912aa0 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:12.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.289+0000 7f3176f6c740 1 --2- 172.21.15.90:0/4164757314 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5641c38f99f0 0x5641c3917af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:12.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.289+0000 7f3176f6c740 1 --2- 172.21.15.90:0/4164757314 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x5641c3978640 0x5641c397aa60 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:12.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.289+0000 7f3176f6c740 1 --2- 172.21.15.90:0/4164757314 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5641c38f8d10 0x5641c39175b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:12.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.289+0000 7f3176f6c740 1 -- 172.21.15.90:0/4164757314 >> 172.21.15.90:0/4164757314 conn(0x5641c38bfa50 msgr2=0x5641c38ff960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:43:12.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.289+0000 7f3153400640 0 client.28025 ms_handle_reset on v2:172.21.15.177:6836/4223414858 2024-04-02T14:43:12.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.289+0000 7f3176f6c740 1 -- 172.21.15.90:0/4164757314 shutdown_connections 2024-04-02T14:43:12.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.289+0000 7f3176f6c740 1 -- 172.21.15.90:0/4164757314 wait complete. 2024-04-02T14:43:12.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.289+0000 7f3176f6c740 20 client.28025 trim_cache size 0 max 16384 2024-04-02T14:43:12.293 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-02T14:43:12.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.290+0000 7f3174200640 1 -- 172.21.15.90:0/3285888551 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f31540a3d20 msgr2=0x7f31540a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:12.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.290+0000 7f3174200640 1 --2- 172.21.15.90:0/3285888551 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f31540a3d20 0x7f31540a61e0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f31640049d0 tx=0x7f3164009500 comp rx=0 tx=0).stop 2024-04-02T14:43:12.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.290+0000 7f3174200640 1 -- 172.21.15.90:0/3285888551 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f316c073f30 msgr2=0x7f316c19a590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:12.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.290+0000 7f3174200640 1 --2- 172.21.15.90:0/3285888551 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f316c073f30 0x7f316c19a590 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f316000e9f0 tx=0x7f316000eeb0 comp rx=0 tx=0).stop 2024-04-02T14:43:12.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.290+0000 7f3174200640 1 -- 172.21.15.90:0/3285888551 shutdown_connections 2024-04-02T14:43:12.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.290+0000 7f3174200640 1 --2- 172.21.15.90:0/3285888551 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f31540a3d20 0x7f31540a61e0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:12.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.290+0000 7f3174200640 1 --2- 172.21.15.90:0/3285888551 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f316c10e2d0 0x7f316c19ee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:12.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.290+0000 7f3174200640 1 --2- 172.21.15.90:0/3285888551 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f316c074850 0x7f316c19aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:12.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.290+0000 7f3174200640 1 --2- 172.21.15.90:0/3285888551 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f316c073f30 0x7f316c19a590 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:12.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.290+0000 7f3174200640 1 -- 172.21.15.90:0/3285888551 >> 172.21.15.90:0/3285888551 conn(0x7f316c0fd540 msgr2=0x7f316c0fe2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:43:12.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.290+0000 7f3174200640 1 -- 172.21.15.90:0/3285888551 shutdown_connections 2024-04-02T14:43:12.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.290+0000 7f3174200640 1 -- 172.21.15.90:0/3285888551 wait complete. 2024-04-02T14:43:12.302 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:43:12 smithi090 ceph-mon[31674]: pgmap v255: 97 pgs: 40 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 35 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 6.6 MiB/s wr, 342 op/s 2024-04-02T14:43:12.302 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:43:12 smithi090 ceph-mon[31674]: from='client.? 172.21.15.90:0/3289185302' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-02T14:43:12.306 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-04-02T14:43:12.357 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:43:12 smithi092 ceph-mon[36497]: pgmap v255: 97 pgs: 40 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 35 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 6.6 MiB/s wr, 342 op/s 2024-04-02T14:43:12.357 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:43:12 smithi092 ceph-mon[36497]: from='client.? 172.21.15.90:0/3289185302' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-02T14:43:12.432 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:43:12 smithi177 ceph-mon[36431]: pgmap v255: 97 pgs: 40 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 35 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 6.6 MiB/s wr, 342 op/s 2024-04-02T14:43:12.433 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:43:12 smithi177 ceph-mon[36431]: from='client.? 172.21.15.90:0/3289185302' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-02T14:43:12.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.522+0000 7fb389e00640 1 -- 172.21.15.90:0/2347611228 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb3841058a0 msgr2=0x7fb384105c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:12.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.522+0000 7fb389e00640 1 --2- 172.21.15.90:0/2347611228 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb3841058a0 0x7fb384105c80 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7fb374009900 tx=0x7fb37402bcd0 comp rx=0 tx=0).stop 2024-04-02T14:43:12.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.522+0000 7fb389e00640 1 -- 172.21.15.90:0/2347611228 shutdown_connections 2024-04-02T14:43:12.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.522+0000 7fb389e00640 1 --2- 172.21.15.90:0/2347611228 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb384101360 0x7fb38410d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:12.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.522+0000 7fb389e00640 1 --2- 172.21.15.90:0/2347611228 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb3841009c0 0x7fb384100e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:12.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.522+0000 7fb389e00640 1 --2- 172.21.15.90:0/2347611228 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb3841058a0 0x7fb384105c80 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:12.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.523+0000 7fb389e00640 1 -- 172.21.15.90:0/2347611228 >> 172.21.15.90:0/2347611228 conn(0x7fb384077a90 msgr2=0x7fb384077ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:43:12.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.523+0000 7fb389e00640 1 -- 172.21.15.90:0/2347611228 shutdown_connections 2024-04-02T14:43:12.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.523+0000 7fb389e00640 1 -- 172.21.15.90:0/2347611228 wait complete. 2024-04-02T14:43:12.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.524+0000 7fb389e00640 1 Processor -- start 2024-04-02T14:43:12.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.524+0000 7fb389e00640 1 -- start start 2024-04-02T14:43:12.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.524+0000 7fb389e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb3841009c0 0x7fb38419a5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:12.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.524+0000 7fb389e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb384101360 0x7fb38419ab10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:12.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.524+0000 7fb389e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb3841058a0 0x7fb38419ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:12.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.524+0000 7fb389e00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fb384110de0 con 0x7fb384101360 2024-04-02T14:43:12.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.524+0000 7fb389e00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fb384110c60 con 0x7fb3841058a0 2024-04-02T14:43:12.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.524+0000 7fb389e00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fb384110f60 con 0x7fb3841009c0 2024-04-02T14:43:12.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.524+0000 7fb382a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb3841009c0 0x7fb38419a5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:12.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.524+0000 7fb382000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb384101360 0x7fb38419ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:12.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.524+0000 7fb382a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb3841009c0 0x7fb38419a5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:43420/0 (socket says 172.21.15.90:43420) 2024-04-02T14:43:12.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.524+0000 7fb382a00640 1 -- 172.21.15.90:0/2120721234 learned_addr learned my addr 172.21.15.90:0/2120721234 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:43:12.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.524+0000 7fb383400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb3841058a0 0x7fb38419ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:12.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.525+0000 7fb382a00640 1 -- 172.21.15.90:0/2120721234 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb3841058a0 msgr2=0x7fb38419ee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:12.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.525+0000 7fb382a00640 1 --2- 172.21.15.90:0/2120721234 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb3841058a0 0x7fb38419ee40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:12.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.525+0000 7fb382a00640 1 -- 172.21.15.90:0/2120721234 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb384101360 msgr2=0x7fb38419ab10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:12.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.525+0000 7fb382a00640 1 --2- 172.21.15.90:0/2120721234 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb384101360 0x7fb38419ab10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:12.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.525+0000 7fb382a00640 1 -- 172.21.15.90:0/2120721234 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb38419f570 con 0x7fb3841009c0 2024-04-02T14:43:12.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.525+0000 7fb383400640 1 --2- 172.21.15.90:0/2120721234 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb3841058a0 0x7fb38419ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:43:12.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.525+0000 7fb382a00640 1 --2- 172.21.15.90:0/2120721234 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb3841009c0 0x7fb38419a5d0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fb374000c00 tx=0x7fb374004360 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:43:12.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.525+0000 7fb388a00640 1 -- 172.21.15.90:0/2120721234 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb374042020 con 0x7fb3841009c0 2024-04-02T14:43:12.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.525+0000 7fb388a00640 1 -- 172.21.15.90:0/2120721234 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb3740054e0 con 0x7fb3841009c0 2024-04-02T14:43:12.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.525+0000 7fb388a00640 1 -- 172.21.15.90:0/2120721234 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb374003b20 con 0x7fb3841009c0 2024-04-02T14:43:12.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.525+0000 7fb389e00640 1 -- 172.21.15.90:0/2120721234 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb38419f800 con 0x7fb3841009c0 2024-04-02T14:43:12.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.526+0000 7fb389e00640 1 -- 172.21.15.90:0/2120721234 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3841a70f0 con 0x7fb3841009c0 2024-04-02T14:43:12.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.529+0000 7fb38cbb8740 1 Processor -- start 2024-04-02T14:43:12.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.529+0000 7fb38cbb8740 1 -- start start 2024-04-02T14:43:12.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.529+0000 7fb38cbb8740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564d131899f0 0x564d13188740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:12.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.529+0000 7fb38cbb8740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564d13188d10 0x564d13189190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:12.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.529+0000 7fb38cbb8740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564d13192080 0x564d13194470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:12.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.529+0000 7fb38cbb8740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x564d130c9400 con 0x564d131899f0 2024-04-02T14:43:12.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.529+0000 7fb38cbb8740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x564d130adf10 con 0x564d13188d10 2024-04-02T14:43:12.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.529+0000 7fb38cbb8740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x564d130ed6f0 con 0x564d13192080 2024-04-02T14:43:12.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.530+0000 7fb383400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564d131899f0 0x564d13188740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:12.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.530+0000 7fb383400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564d131899f0 0x564d13188740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:37416/0 (socket says 172.21.15.90:37416) 2024-04-02T14:43:12.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.530+0000 7fb383400640 1 -- 172.21.15.90:0/3111825835 learned_addr learned my addr 172.21.15.90:0/3111825835 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:43:12.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.530+0000 7fb382000640 1 --2- 172.21.15.90:0/3111825835 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564d13192080 0x564d13194470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:12.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.530+0000 7fb388a00640 1 -- 172.21.15.90:0/2120721234 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fb374045020 con 0x7fb3841009c0 2024-04-02T14:43:12.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.530+0000 7fb382a00640 1 --2- 172.21.15.90:0/3111825835 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564d13188d10 0x564d13189190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:12.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.530+0000 7fb383400640 1 -- 172.21.15.90:0/3111825835 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564d13192080 msgr2=0x564d13194470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:43:12.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.530+0000 7fb383400640 1 --2- 172.21.15.90:0/3111825835 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564d13192080 0x564d13194470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:12.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.530+0000 7fb383400640 1 -- 172.21.15.90:0/3111825835 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564d13188d10 msgr2=0x564d13189190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:43:12.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.530+0000 7fb383400640 1 --2- 172.21.15.90:0/3111825835 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564d13188d10 0x564d13189190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:12.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.530+0000 7fb383400640 1 -- 172.21.15.90:0/3111825835 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564d12e013f0 con 0x564d131899f0 2024-04-02T14:43:12.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.530+0000 7fb382000640 1 --2- 172.21.15.90:0/3111825835 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564d13192080 0x564d13194470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:43:12.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.530+0000 7fb382a00640 1 --2- 172.21.15.90:0/3111825835 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564d13188d10 0x564d13189190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:43:12.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.531+0000 7fb383400640 1 --2- 172.21.15.90:0/3111825835 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564d131899f0 0x564d13188740 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fb37c011790 tx=0x7fb37c011c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c463b9164c8d1c4b server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:43:12.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.531+0000 7fb36fe00640 1 -- 172.21.15.90:0/3111825835 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb37c003c30 con 0x564d131899f0 2024-04-02T14:43:12.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.531+0000 7fb36fe00640 1 -- 172.21.15.90:0/3111825835 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb37c00fe80 con 0x564d131899f0 2024-04-02T14:43:12.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.531+0000 7fb36fe00640 1 -- 172.21.15.90:0/3111825835 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb37c0193e0 con 0x564d131899f0 2024-04-02T14:43:12.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.531+0000 7fb388a00640 1 --2- 172.21.15.90:0/2120721234 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fb3680a3da0 0x7fb3680a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:12.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.532+0000 7fb38cbb8740 1 -- 172.21.15.90:0/3111825835 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564d131899f0 msgr2=0x564d13188740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:12.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.532+0000 7fb38cbb8740 1 --2- 172.21.15.90:0/3111825835 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564d131899f0 0x564d13188740 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fb37c011790 tx=0x7fb37c011c50 comp rx=0 tx=0).stop 2024-04-02T14:43:12.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.532+0000 7fb382a00640 1 --2- 172.21.15.90:0/2120721234 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fb3680a3da0 0x7fb3680a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:12.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.532+0000 7fb388a00640 1 -- 172.21.15.90:0/2120721234 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8594+0+0 (secure 0 0 0) 0x7fb3740f2360 con 0x7fb3841009c0 2024-04-02T14:43:12.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.532+0000 7fb38cbb8740 1 -- 172.21.15.90:0/3111825835 shutdown_connections 2024-04-02T14:43:12.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.532+0000 7fb38cbb8740 1 --2- 172.21.15.90:0/3111825835 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564d13192080 0x564d13194470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:12.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.532+0000 7fb38cbb8740 1 --2- 172.21.15.90:0/3111825835 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564d13188d10 0x564d13189190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:12.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.532+0000 7fb38cbb8740 1 --2- 172.21.15.90:0/3111825835 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564d131899f0 0x564d13188740 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:12.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.532+0000 7fb38cbb8740 1 -- 172.21.15.90:0/3111825835 >> 172.21.15.90:0/3111825835 conn(0x564d1314fa50 msgr2=0x564d13183760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:43:12.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.532+0000 7fb38cbb8740 1 -- 172.21.15.90:0/3111825835 shutdown_connections 2024-04-02T14:43:12.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.532+0000 7fb38cbb8740 1 -- 172.21.15.90:0/3111825835 wait complete. 2024-04-02T14:43:12.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.532+0000 7fb382a00640 1 --2- 172.21.15.90:0/2120721234 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fb3680a3da0 0x7fb3680a6260 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x564d13189f40 tx=0x7fb374049000 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:43:12.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.533+0000 7fb38cbb8740 1 -- start start 2024-04-02T14:43:12.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.533+0000 7fb38cbb8740 1 Processor -- start 2024-04-02T14:43:12.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.533+0000 7fb38cbb8740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564d13188d10 0x564d1318f020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:12.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.534+0000 7fb38cbb8740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564d131899f0 0x564d1318f560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:12.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.534+0000 7fb38cbb8740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564d13192080 0x564d1318faa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:12.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.534+0000 7fb38cbb8740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x564d130e96a0 con 0x564d13188d10 2024-04-02T14:43:12.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.534+0000 7fb38cbb8740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x564d13095980 con 0x564d131899f0 2024-04-02T14:43:12.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.534+0000 7fb38cbb8740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x564d130a8cb0 con 0x564d13192080 2024-04-02T14:43:12.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.534+0000 7fb38cbb8740 10 client.0 _ll_register_callbacks cb 0x564d130a86a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:43:12.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.534+0000 7fb383400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564d13188d10 0x564d1318f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:12.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.534+0000 7fb383400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564d13188d10 0x564d1318f020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:37432/0 (socket says 172.21.15.90:37432) 2024-04-02T14:43:12.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.534+0000 7fb383400640 1 -- 172.21.15.90:0/2344040250 learned_addr learned my addr 172.21.15.90:0/2344040250 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:43:12.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.534+0000 7fb382000640 1 --2- 172.21.15.90:0/2344040250 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564d131899f0 0x564d1318f560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:12.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.534+0000 7fb383400640 1 --2- 172.21.15.90:0/2344040250 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564d13192080 0x564d1318faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:12.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.534+0000 7fb382000640 1 -- 172.21.15.90:0/2344040250 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564d13192080 msgr2=0x564d1318faa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:43:12.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.534+0000 7fb382000640 1 --2- 172.21.15.90:0/2344040250 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564d13192080 0x564d1318faa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:12.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.534+0000 7fb382000640 1 -- 172.21.15.90:0/2344040250 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564d13188d10 msgr2=0x564d1318f020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:43:12.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.534+0000 7fb382000640 1 --2- 172.21.15.90:0/2344040250 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564d13188d10 0x564d1318f020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:12.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.534+0000 7fb382000640 1 -- 172.21.15.90:0/2344040250 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564d12ecb7d0 con 0x564d131899f0 2024-04-02T14:43:12.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.534+0000 7fb383400640 1 --2- 172.21.15.90:0/2344040250 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564d13188d10 0x564d1318f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:43:12.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.535+0000 7fb382000640 1 --2- 172.21.15.90:0/2344040250 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564d131899f0 0x564d1318f560 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7fb3700126d0 tx=0x7fb370012b90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a2e7264b5f6caa5 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:43:12.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.535+0000 7fb36cc00640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:43:12.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.535+0000 7fb36cc00640 1 -- 172.21.15.90:0/2344040250 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb370004030 con 0x564d131899f0 2024-04-02T14:43:12.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.535+0000 7fb36cc00640 1 -- 172.21.15.90:0/2344040250 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8594+0+0 (secure 0 0 0) 0x7fb3700041d0 con 0x564d131899f0 2024-04-02T14:43:12.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.535+0000 7fb38cbb8740 1 -- 172.21.15.90:0/2344040250 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x564d130ad040 con 0x564d131899f0 2024-04-02T14:43:12.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.535+0000 7fb36cc00640 1 -- 172.21.15.90:0/2344040250 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fb3700043d0 con 0x564d131899f0 2024-04-02T14:43:12.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.535+0000 7fb38cbb8740 10 client.28034 fetch_fsmap learned FSMap version 27 2024-04-02T14:43:12.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.535+0000 7fb38cbb8740 1 -- 172.21.15.90:0/2344040250 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x564d131a0ed0 con 0x564d131899f0 2024-04-02T14:43:12.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.536+0000 7fb36cc00640 1 -- 172.21.15.90:0/2344040250 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x564d130ad040 con 0x564d131899f0 2024-04-02T14:43:12.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.536+0000 7fb36cc00640 1 -- 172.21.15.90:0/2344040250 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fb370003bf0 con 0x564d131899f0 2024-04-02T14:43:12.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.536+0000 7fb38cbb8740 10 client.28034 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:43:12.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.536+0000 7fb38cbb8740 10 client.28034 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-04-02T14:43:12.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.536+0000 7fb38cbb8740 20 client.28034 populate_metadata read hostname 'smithi090' 2024-04-02T14:43:12.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.536+0000 7fb38cbb8740 1 --2- 172.21.15.90:0/2344040250 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x564d131a22e0 0x564d131a46d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:12.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.536+0000 7fb38cbb8740 4 client.28034 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:43:12.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.536+0000 7fb38cbb8740 1 -- 172.21.15.90:0/2344040250 --> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564d130fac30 con 0x564d131a22e0 2024-04-02T14:43:12.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.536+0000 7fb383400640 1 --2- 172.21.15.90:0/2344040250 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x564d131a22e0 0x564d131a46d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:12.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.537+0000 7fb36cc00640 10 client.28034 ms_handle_connect on v2:172.21.15.92:6836/1270635939 2024-04-02T14:43:12.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.537+0000 7fb383400640 1 --2- 172.21.15.90:0/2344040250 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x564d131a22e0 0x564d131a46d0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=f2befeede7af94d server_cookie=9273edd028368fe4 in_seq=0 out_seq=0 2024-04-02T14:43:12.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.537+0000 7fb36cc00640 1 -- 172.21.15.90:0/2344040250 <== mds.3 v2:172.21.15.92:6836/1270635939 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x564d130fac30 con 0x564d131a22e0 2024-04-02T14:43:12.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.537+0000 7fb36cc00640 10 client.28034 handle_command_reply: tid=0 2024-04-02T14:43:12.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.537+0000 7fb38cbb8740 1 client.28034 shutdown 2024-04-02T14:43:12.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.538+0000 7fb38cbb8740 1 -- 172.21.15.90:0/2344040250 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564d131899f0 msgr2=0x564d1318f560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:12.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.538+0000 7fb38cbb8740 1 --2- 172.21.15.90:0/2344040250 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564d131899f0 0x564d1318f560 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7fb3700126d0 tx=0x7fb370012b90 comp rx=0 tx=0).stop 2024-04-02T14:43:12.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.538+0000 7fb38cbb8740 1 -- 172.21.15.90:0/2344040250 shutdown_connections 2024-04-02T14:43:12.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.538+0000 7fb38cbb8740 1 --2- 172.21.15.90:0/2344040250 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x564d131a22e0 0x564d131a46d0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:12.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.538+0000 7fb38cbb8740 1 --2- 172.21.15.90:0/2344040250 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564d13192080 0x564d1318faa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:12.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.538+0000 7fb38cbb8740 1 --2- 172.21.15.90:0/2344040250 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564d131899f0 0x564d1318f560 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:12.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.538+0000 7fb38cbb8740 1 --2- 172.21.15.90:0/2344040250 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564d13188d10 0x564d1318f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:12.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.538+0000 7fb38cbb8740 1 -- 172.21.15.90:0/2344040250 >> 172.21.15.90:0/2344040250 conn(0x564d1314fa50 msgr2=0x564d131854b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:43:12.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.538+0000 7fb36cc00640 0 client.28034 ms_handle_reset on v2:172.21.15.92:6836/1270635939 2024-04-02T14:43:12.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.538+0000 7fb38cbb8740 1 -- 172.21.15.90:0/2344040250 shutdown_connections 2024-04-02T14:43:12.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.538+0000 7fb38cbb8740 1 -- 172.21.15.90:0/2344040250 wait complete. 2024-04-02T14:43:12.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.538+0000 7fb38cbb8740 20 client.28034 trim_cache size 0 max 16384 2024-04-02T14:43:12.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.553+0000 7fb38cbb8740 1 Processor -- start 2024-04-02T14:43:12.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.554+0000 7fb38cbb8740 1 -- start start 2024-04-02T14:43:12.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.554+0000 7fb38cbb8740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564d13188d10 0x564d13191750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:12.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.554+0000 7fb38cbb8740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564d131899f0 0x564d1319db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:12.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.554+0000 7fb38cbb8740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564d13192080 0x564d131a9fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:12.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.554+0000 7fb38cbb8740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fb3841afad0 con 0x564d13192080 2024-04-02T14:43:12.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.554+0000 7fb38cbb8740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fb3841b1610 con 0x564d131899f0 2024-04-02T14:43:12.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.554+0000 7fb38cbb8740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fb3841b01a0 con 0x564d13188d10 2024-04-02T14:43:12.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.554+0000 7fb383400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564d13188d10 0x564d13191750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:12.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.554+0000 7fb382000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564d131899f0 0x564d1319db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:12.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.554+0000 7fb383400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564d13188d10 0x564d13191750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:43440/0 (socket says 172.21.15.90:43440) 2024-04-02T14:43:12.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.554+0000 7fb383400640 1 -- 172.21.15.90:0/1634024011 learned_addr learned my addr 172.21.15.90:0/1634024011 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:43:12.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.554+0000 7fb382000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564d131899f0 0x564d1319db20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:33552/0 (socket says 172.21.15.90:33552) 2024-04-02T14:43:12.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.554+0000 7fb383400640 1 --2- 172.21.15.90:0/1634024011 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564d13192080 0x564d131a9fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:12.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.555+0000 7fb382000640 1 -- 172.21.15.90:0/1634024011 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564d13188d10 msgr2=0x564d13191750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:43:12.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.555+0000 7fb382000640 1 --2- 172.21.15.90:0/1634024011 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564d13188d10 0x564d13191750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:12.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.555+0000 7fb382000640 1 -- 172.21.15.90:0/1634024011 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564d13192080 msgr2=0x564d131a9fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:43:12.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.555+0000 7fb382000640 1 --2- 172.21.15.90:0/1634024011 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564d13192080 0x564d131a9fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:12.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.555+0000 7fb382000640 1 -- 172.21.15.90:0/1634024011 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564d13187d40 con 0x564d131899f0 2024-04-02T14:43:12.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.555+0000 7fb382000640 1 --2- 172.21.15.90:0/1634024011 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564d131899f0 0x564d1319db20 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7fb370010480 tx=0x7fb370009fb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=be4c157cf0fffd85 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:43:12.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.555+0000 7fb36f400640 1 -- 172.21.15.90:0/1634024011 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3700056c0 con 0x564d131899f0 2024-04-02T14:43:12.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.556+0000 7fb36f400640 1 -- 172.21.15.90:0/1634024011 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb37000fe00 con 0x564d131899f0 2024-04-02T14:43:12.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.556+0000 7fb36f400640 1 -- 172.21.15.90:0/1634024011 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb370003b50 con 0x564d131899f0 2024-04-02T14:43:12.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.557+0000 7fb38cbb8740 1 -- 172.21.15.90:0/1634024011 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564d131899f0 msgr2=0x564d1319db20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:12.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.557+0000 7fb38cbb8740 1 --2- 172.21.15.90:0/1634024011 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564d131899f0 0x564d1319db20 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7fb370010480 tx=0x7fb370009fb0 comp rx=0 tx=0).stop 2024-04-02T14:43:12.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.557+0000 7fb38cbb8740 1 -- 172.21.15.90:0/1634024011 shutdown_connections 2024-04-02T14:43:12.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.557+0000 7fb38cbb8740 1 --2- 172.21.15.90:0/1634024011 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564d13192080 0x564d131a9fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:12.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.557+0000 7fb38cbb8740 1 --2- 172.21.15.90:0/1634024011 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564d131899f0 0x564d1319db20 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:12.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.557+0000 7fb38cbb8740 1 --2- 172.21.15.90:0/1634024011 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564d13188d10 0x564d13191750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:12.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.557+0000 7fb38cbb8740 1 -- 172.21.15.90:0/1634024011 >> 172.21.15.90:0/1634024011 conn(0x564d1314fa50 msgr2=0x564d13183290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:43:12.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.557+0000 7fb38cbb8740 1 -- 172.21.15.90:0/1634024011 shutdown_connections 2024-04-02T14:43:12.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.557+0000 7fb38cbb8740 1 -- 172.21.15.90:0/1634024011 wait complete. 2024-04-02T14:43:12.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.557+0000 7fb38cbb8740 1 -- start start 2024-04-02T14:43:12.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.558+0000 7fb38cbb8740 1 Processor -- start 2024-04-02T14:43:12.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.558+0000 7fb38cbb8740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564d13188d10 0x564d13195940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:12.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.558+0000 7fb38cbb8740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564d131899f0 0x564d13195e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:12.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.558+0000 7fb38cbb8740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564d13192080 0x564d131963c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:12.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.558+0000 7fb38cbb8740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fb3841b0f40 con 0x564d13192080 2024-04-02T14:43:12.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.558+0000 7fb38cbb8740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fb3841b0870 con 0x564d13188d10 2024-04-02T14:43:12.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.558+0000 7fb38cbb8740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fb3841adad0 con 0x564d131899f0 2024-04-02T14:43:12.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.558+0000 7fb38cbb8740 10 client.0 _ll_register_callbacks cb 0x7fb370009ec0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:43:12.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.558+0000 7fb382000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564d131899f0 0x564d13195e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:12.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.558+0000 7fb383400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564d13188d10 0x564d13195940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:12.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.558+0000 7fb382000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564d131899f0 0x564d13195e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:43452/0 (socket says 172.21.15.90:43452) 2024-04-02T14:43:12.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.558+0000 7fb382000640 1 -- 172.21.15.90:0/2448319348 learned_addr learned my addr 172.21.15.90:0/2448319348 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:43:12.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.558+0000 7fb383400640 1 --2- 172.21.15.90:0/2448319348 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564d13192080 0x564d131963c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:12.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.559+0000 7fb383400640 1 -- 172.21.15.90:0/2448319348 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564d131899f0 msgr2=0x564d13195e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:43:12.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.559+0000 7fb383400640 1 --2- 172.21.15.90:0/2448319348 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564d131899f0 0x564d13195e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:12.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.559+0000 7fb383400640 1 -- 172.21.15.90:0/2448319348 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564d13192080 msgr2=0x564d131963c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:43:12.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.559+0000 7fb383400640 1 --2- 172.21.15.90:0/2448319348 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564d13192080 0x564d131963c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:12.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.559+0000 7fb383400640 1 -- 172.21.15.90:0/2448319348 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564d13207830 con 0x564d13188d10 2024-04-02T14:43:12.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.559+0000 7fb382000640 1 --2- 172.21.15.90:0/2448319348 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564d131899f0 0x564d13195e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:43:12.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.559+0000 7fb383400640 1 --2- 172.21.15.90:0/2448319348 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564d13188d10 0x564d13195940 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7fb37c0046a0 tx=0x7fb37c011c50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a5f18e3d139f0c0a server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:43:12.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.559+0000 7fb36cc00640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:43:12.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.559+0000 7fb36cc00640 1 -- 172.21.15.90:0/2448319348 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb37c01f040 con 0x564d13188d10 2024-04-02T14:43:12.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.559+0000 7fb36cc00640 1 -- 172.21.15.90:0/2448319348 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8594+0+0 (secure 0 0 0) 0x7fb37c003b70 con 0x564d13188d10 2024-04-02T14:43:12.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.559+0000 7fb38cbb8740 1 -- 172.21.15.90:0/2448319348 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb37000fe00 con 0x564d13188d10 2024-04-02T14:43:12.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.560+0000 7fb36cc00640 1 -- 172.21.15.90:0/2448319348 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fb37c003d70 con 0x564d13188d10 2024-04-02T14:43:12.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.560+0000 7fb38cbb8740 10 client.28040 fetch_fsmap learned FSMap version 27 2024-04-02T14:43:12.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.560+0000 7fb38cbb8740 1 -- 172.21.15.90:0/2448319348 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x564d13208a10 con 0x564d13188d10 2024-04-02T14:43:12.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.560+0000 7fb36cc00640 1 -- 172.21.15.90:0/2448319348 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb37000fe00 con 0x564d13188d10 2024-04-02T14:43:12.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.560+0000 7fb36cc00640 1 -- 172.21.15.90:0/2448319348 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fb37c018070 con 0x564d13188d10 2024-04-02T14:43:12.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.560+0000 7fb38cbb8740 10 client.28040 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:43:12.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.560+0000 7fb38cbb8740 10 client.28040 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-04-02T14:43:12.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.560+0000 7fb38cbb8740 20 client.28040 populate_metadata read hostname 'smithi090' 2024-04-02T14:43:12.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.560+0000 7fb38cbb8740 1 --2- 172.21.15.90:0/2448319348 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x564d13208ee0 0x564d1320b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:12.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.560+0000 7fb38cbb8740 4 client.28040 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-02T14:43:12.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.560+0000 7fb38cbb8740 1 -- 172.21.15.90:0/2448319348 --> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] -- command(tid 0: {"prefix": "damage ls"}) -- 0x564d130f4f80 con 0x564d13208ee0 2024-04-02T14:43:12.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.561+0000 7fb382000640 1 --2- 172.21.15.90:0/2448319348 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x564d13208ee0 0x564d1320b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:12.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.561+0000 7fb382000640 1 --2- 172.21.15.90:0/2448319348 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x564d13208ee0 0x564d1320b2d0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=7c717c4d0b6b1d26 server_cookie=3f62ee95f2b07dad in_seq=0 out_seq=0 2024-04-02T14:43:12.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.561+0000 7fb36cc00640 10 client.28040 ms_handle_connect on v2:172.21.15.92:6836/1270635939 2024-04-02T14:43:12.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.561+0000 7fb36cc00640 1 -- 172.21.15.90:0/2448319348 <== mds.3 v2:172.21.15.92:6836/1270635939 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x564d130f4f80 con 0x564d13208ee0 2024-04-02T14:43:12.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.561+0000 7fb36cc00640 10 client.28040 handle_command_reply: tid=0 2024-04-02T14:43:12.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.561+0000 7fb38cbb8740 1 client.28040 shutdown 2024-04-02T14:43:12.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.561+0000 7fb38cbb8740 1 -- 172.21.15.90:0/2448319348 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564d13188d10 msgr2=0x564d13195940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:12.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.562+0000 7fb38cbb8740 1 --2- 172.21.15.90:0/2448319348 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564d13188d10 0x564d13195940 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7fb37c0046a0 tx=0x7fb37c011c50 comp rx=0 tx=0).stop 2024-04-02T14:43:12.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.562+0000 7fb38cbb8740 1 -- 172.21.15.90:0/2448319348 shutdown_connections 2024-04-02T14:43:12.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.562+0000 7fb38cbb8740 1 --2- 172.21.15.90:0/2448319348 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x564d13208ee0 0x564d1320b2d0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:12.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.562+0000 7fb38cbb8740 1 --2- 172.21.15.90:0/2448319348 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564d13192080 0x564d131963c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:12.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.562+0000 7fb38cbb8740 1 --2- 172.21.15.90:0/2448319348 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564d131899f0 0x564d13195e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:12.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.562+0000 7fb38cbb8740 1 --2- 172.21.15.90:0/2448319348 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564d13188d10 0x564d13195940 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:12.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.562+0000 7fb36cc00640 0 client.28040 ms_handle_reset on v2:172.21.15.92:6836/1270635939 2024-04-02T14:43:12.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.562+0000 7fb38cbb8740 1 -- 172.21.15.90:0/2448319348 >> 172.21.15.90:0/2448319348 conn(0x564d1314fa50 msgr2=0x564d13182010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:43:12.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.562+0000 7fb38cbb8740 1 -- 172.21.15.90:0/2448319348 shutdown_connections 2024-04-02T14:43:12.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.562+0000 7fb38cbb8740 1 -- 172.21.15.90:0/2448319348 wait complete. 2024-04-02T14:43:12.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.562+0000 7fb38cbb8740 20 client.28040 trim_cache size 0 max 16384 2024-04-02T14:43:12.567 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-02T14:43:12.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.563+0000 7fb389e00640 1 -- 172.21.15.90:0/2120721234 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fb3680a3da0 msgr2=0x7fb3680a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:12.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.563+0000 7fb389e00640 1 --2- 172.21.15.90:0/2120721234 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fb3680a3da0 0x7fb3680a6260 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x564d13189f40 tx=0x7fb374049000 comp rx=0 tx=0).stop 2024-04-02T14:43:12.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.563+0000 7fb389e00640 1 -- 172.21.15.90:0/2120721234 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb3841009c0 msgr2=0x7fb38419a5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:12.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.563+0000 7fb389e00640 1 --2- 172.21.15.90:0/2120721234 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb3841009c0 0x7fb38419a5d0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fb374000c00 tx=0x7fb374004360 comp rx=0 tx=0).stop 2024-04-02T14:43:12.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.563+0000 7fb389e00640 1 -- 172.21.15.90:0/2120721234 shutdown_connections 2024-04-02T14:43:12.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.563+0000 7fb389e00640 1 --2- 172.21.15.90:0/2120721234 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fb3680a3da0 0x7fb3680a6260 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:12.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.563+0000 7fb389e00640 1 --2- 172.21.15.90:0/2120721234 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb3841058a0 0x7fb38419ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:12.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.563+0000 7fb389e00640 1 --2- 172.21.15.90:0/2120721234 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb384101360 0x7fb38419ab10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:12.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.563+0000 7fb389e00640 1 --2- 172.21.15.90:0/2120721234 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb3841009c0 0x7fb38419a5d0 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:12.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.563+0000 7fb389e00640 1 -- 172.21.15.90:0/2120721234 >> 172.21.15.90:0/2120721234 conn(0x7fb384077a90 msgr2=0x7fb384069f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:43:12.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.563+0000 7fb389e00640 1 -- 172.21.15.90:0/2120721234 shutdown_connections 2024-04-02T14:43:12.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.563+0000 7fb389e00640 1 -- 172.21.15.90:0/2120721234 wait complete. 2024-04-02T14:43:12.579 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-02T14:43:12.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.793+0000 7f153ea00640 1 -- 172.21.15.90:0/2307737014 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1538103df0 msgr2=0x7f15381041d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:12.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.793+0000 7f153ea00640 1 --2- 172.21.15.90:0/2307737014 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1538103df0 0x7f15381041d0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f152c009900 tx=0x7f152c02bc40 comp rx=0 tx=0).stop 2024-04-02T14:43:12.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.793+0000 7f153ea00640 1 -- 172.21.15.90:0/2307737014 shutdown_connections 2024-04-02T14:43:12.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.793+0000 7f153ea00640 1 --2- 172.21.15.90:0/2307737014 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1538108ec0 0x7f1538110a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:12.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.793+0000 7f153ea00640 1 --2- 172.21.15.90:0/2307737014 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f15381047a0 0x7f15381087e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:12.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.793+0000 7f153ea00640 1 --2- 172.21.15.90:0/2307737014 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1538103df0 0x7f15381041d0 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:12.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.793+0000 7f153ea00640 1 -- 172.21.15.90:0/2307737014 >> 172.21.15.90:0/2307737014 conn(0x7f15380ff610 msgr2=0x7f1538101a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:43:12.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.793+0000 7f153ea00640 1 -- 172.21.15.90:0/2307737014 shutdown_connections 2024-04-02T14:43:12.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.793+0000 7f153ea00640 1 -- 172.21.15.90:0/2307737014 wait complete. 2024-04-02T14:43:12.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.794+0000 7f153ea00640 1 Processor -- start 2024-04-02T14:43:12.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.794+0000 7f153ea00640 1 -- start start 2024-04-02T14:43:12.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.794+0000 7f153ea00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1538103df0 0x7f153819a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:12.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.794+0000 7f153ea00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1538108ec0 0x7f153819aa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:12.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.795+0000 7f153ea00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f153819eda0 0x7f153819f250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:12.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.795+0000 7f153ea00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f1538113fc0 con 0x7f1538108ec0 2024-04-02T14:43:12.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.795+0000 7f153ea00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f1538113e40 con 0x7f1538103df0 2024-04-02T14:43:12.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.795+0000 7f153ea00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f1538114140 con 0x7f153819eda0 2024-04-02T14:43:12.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.795+0000 7f153d600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1538103df0 0x7f153819a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:12.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.795+0000 7f153e000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f153819eda0 0x7f153819f250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:12.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.795+0000 7f153d600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1538103df0 0x7f153819a4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:33580/0 (socket says 172.21.15.90:33580) 2024-04-02T14:43:12.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.795+0000 7f153cc00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1538108ec0 0x7f153819aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:12.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.795+0000 7f153d600640 1 -- 172.21.15.90:0/2199639104 learned_addr learned my addr 172.21.15.90:0/2199639104 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:43:12.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.795+0000 7f153d600640 1 -- 172.21.15.90:0/2199639104 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f153819eda0 msgr2=0x7f153819f250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:12.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.795+0000 7f153d600640 1 --2- 172.21.15.90:0/2199639104 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f153819eda0 0x7f153819f250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:12.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.795+0000 7f153d600640 1 -- 172.21.15.90:0/2199639104 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1538108ec0 msgr2=0x7f153819aa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:12.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.795+0000 7f153d600640 1 --2- 172.21.15.90:0/2199639104 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1538108ec0 0x7f153819aa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:12.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.795+0000 7f153d600640 1 -- 172.21.15.90:0/2199639104 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f153819f890 con 0x7f1538103df0 2024-04-02T14:43:12.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.795+0000 7f153e000640 1 --2- 172.21.15.90:0/2199639104 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f153819eda0 0x7f153819f250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:43:12.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.796+0000 7f153d600640 1 --2- 172.21.15.90:0/2199639104 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1538103df0 0x7f153819a4e0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f152c005ab0 tx=0x7f152c0042f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:43:12.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.796+0000 7f1536000640 1 -- 172.21.15.90:0/2199639104 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f152c041070 con 0x7f1538103df0 2024-04-02T14:43:12.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.796+0000 7f1536000640 1 -- 172.21.15.90:0/2199639104 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f152c02cca0 con 0x7f1538103df0 2024-04-02T14:43:12.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.796+0000 7f1536000640 1 -- 172.21.15.90:0/2199639104 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f152c003d90 con 0x7f1538103df0 2024-04-02T14:43:12.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.796+0000 7f153ea00640 1 -- 172.21.15.90:0/2199639104 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f153819fb20 con 0x7f1538103df0 2024-04-02T14:43:12.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.796+0000 7f153ea00640 1 -- 172.21.15.90:0/2199639104 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15381a72e0 con 0x7f1538103df0 2024-04-02T14:43:12.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.799+0000 7f1536000640 1 -- 172.21.15.90:0/2199639104 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f152c044070 con 0x7f1538103df0 2024-04-02T14:43:12.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.799+0000 7f1542e22740 1 Processor -- start 2024-04-02T14:43:12.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.800+0000 7f1542e22740 1 -- start start 2024-04-02T14:43:12.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.800+0000 7f1542e22740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e9f502eba0 0x55e9f502d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:12.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.800+0000 7f1542e22740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e9f502dec0 0x55e9f502e340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:12.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.800+0000 7f1542e22740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e9f5037230 0x55e9f5039620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:12.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.800+0000 7f1542e22740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55e9f4f6e400 con 0x55e9f502eba0 2024-04-02T14:43:12.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.800+0000 7f1542e22740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55e9f4f52f10 con 0x55e9f5037230 2024-04-02T14:43:12.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.800+0000 7f1542e22740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55e9f4f926f0 con 0x55e9f502dec0 2024-04-02T14:43:12.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.800+0000 7f153d600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e9f502dec0 0x55e9f502e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:12.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.800+0000 7f153e000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e9f502eba0 0x55e9f502d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:12.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.800+0000 7f153cc00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e9f5037230 0x55e9f5039620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:12.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.800+0000 7f153d600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e9f502dec0 0x55e9f502e340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:43476/0 (socket says 172.21.15.90:43476) 2024-04-02T14:43:12.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.800+0000 7f153d600640 1 -- 172.21.15.90:0/2911960840 learned_addr learned my addr 172.21.15.90:0/2911960840 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:43:12.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.800+0000 7f1536000640 1 --2- 172.21.15.90:0/2199639104 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f15200a3d20 0x7f15200a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:12.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.801+0000 7f153cc00640 1 -- 172.21.15.90:0/2911960840 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e9f502dec0 msgr2=0x55e9f502e340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:43:12.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.801+0000 7f153cc00640 1 --2- 172.21.15.90:0/2911960840 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e9f502dec0 0x55e9f502e340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:12.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.801+0000 7f153cc00640 1 -- 172.21.15.90:0/2911960840 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e9f502eba0 msgr2=0x55e9f502d8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:43:12.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.801+0000 7f153cc00640 1 --2- 172.21.15.90:0/2911960840 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e9f502eba0 0x55e9f502d8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:12.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.801+0000 7f153cc00640 1 -- 172.21.15.90:0/2911960840 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e9f4ca63f0 con 0x55e9f5037230 2024-04-02T14:43:12.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.801+0000 7f1536000640 1 -- 172.21.15.90:0/2199639104 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8594+0+0 (secure 0 0 0) 0x7f152c0f2130 con 0x7f1538103df0 2024-04-02T14:43:12.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.801+0000 7f153e000640 1 --2- 172.21.15.90:0/2199639104 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f15200a3d20 0x7f15200a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:12.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.801+0000 7f153cc00640 1 --2- 172.21.15.90:0/2911960840 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e9f5037230 0x55e9f5039620 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f1538069830 tx=0x7f1524011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2403119dfae799e9 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:43:12.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.801+0000 7f152a600640 1 -- 172.21.15.90:0/2911960840 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1524004230 con 0x55e9f5037230 2024-04-02T14:43:12.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.801+0000 7f152a600640 1 -- 172.21.15.90:0/2911960840 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1524009e10 con 0x55e9f5037230 2024-04-02T14:43:12.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.801+0000 7f152a600640 1 -- 172.21.15.90:0/2911960840 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1524004d70 con 0x55e9f5037230 2024-04-02T14:43:12.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.802+0000 7f1542e22740 1 -- 172.21.15.90:0/2911960840 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e9f5037230 msgr2=0x55e9f5039620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:12.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.802+0000 7f153e000640 1 --2- 172.21.15.90:0/2199639104 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f15200a3d20 0x7f15200a61e0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f15300094d0 tx=0x7f1530014040 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:43:12.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.802+0000 7f1542e22740 1 --2- 172.21.15.90:0/2911960840 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e9f5037230 0x55e9f5039620 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f1538069830 tx=0x7f1524011810 comp rx=0 tx=0).stop 2024-04-02T14:43:12.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.802+0000 7f1542e22740 1 -- 172.21.15.90:0/2911960840 shutdown_connections 2024-04-02T14:43:12.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.802+0000 7f1542e22740 1 --2- 172.21.15.90:0/2911960840 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e9f5037230 0x55e9f5039620 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:12.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.802+0000 7f1542e22740 1 --2- 172.21.15.90:0/2911960840 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e9f502dec0 0x55e9f502e340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:12.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.802+0000 7f1542e22740 1 --2- 172.21.15.90:0/2911960840 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e9f502eba0 0x55e9f502d8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:12.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.802+0000 7f1542e22740 1 -- 172.21.15.90:0/2911960840 >> 172.21.15.90:0/2911960840 conn(0x55e9f4ff4a50 msgr2=0x55e9f5028910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:43:12.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.802+0000 7f1542e22740 1 -- 172.21.15.90:0/2911960840 shutdown_connections 2024-04-02T14:43:12.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.802+0000 7f1542e22740 1 -- 172.21.15.90:0/2911960840 wait complete. 2024-04-02T14:43:12.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.803+0000 7f1542e22740 1 -- start start 2024-04-02T14:43:12.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.803+0000 7f1542e22740 1 Processor -- start 2024-04-02T14:43:12.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.803+0000 7f1542e22740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e9f502dec0 0x55e9f50341a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:12.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.804+0000 7f1542e22740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e9f502eba0 0x55e9f50346e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:12.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.804+0000 7f1542e22740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e9f5037230 0x55e9f5034c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:12.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.804+0000 7f1542e22740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55e9f4f8e6a0 con 0x55e9f502eba0 2024-04-02T14:43:12.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.804+0000 7f1542e22740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55e9f4f3a980 con 0x55e9f5037230 2024-04-02T14:43:12.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.804+0000 7f1542e22740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55e9f4f4dcb0 con 0x55e9f502dec0 2024-04-02T14:43:12.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.804+0000 7f1542e22740 10 client.0 _ll_register_callbacks cb 0x55e9f4f4d6a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:43:12.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.804+0000 7f153d600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e9f502dec0 0x55e9f50341a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:12.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.804+0000 7f153e000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e9f5037230 0x55e9f5034c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:12.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.804+0000 7f153d600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e9f502dec0 0x55e9f50341a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:43482/0 (socket says 172.21.15.90:43482) 2024-04-02T14:43:12.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.804+0000 7f153d600640 1 -- 172.21.15.90:0/2612003595 learned_addr learned my addr 172.21.15.90:0/2612003595 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:43:12.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.804+0000 7f153cc00640 1 --2- 172.21.15.90:0/2612003595 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e9f502eba0 0x55e9f50346e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:12.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.804+0000 7f153e000640 1 -- 172.21.15.90:0/2612003595 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e9f502dec0 msgr2=0x55e9f50341a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:43:12.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.804+0000 7f153e000640 1 --2- 172.21.15.90:0/2612003595 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e9f502dec0 0x55e9f50341a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:12.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.804+0000 7f153e000640 1 -- 172.21.15.90:0/2612003595 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e9f502eba0 msgr2=0x55e9f50346e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:43:12.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.804+0000 7f153e000640 1 --2- 172.21.15.90:0/2612003595 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e9f502eba0 0x55e9f50346e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:12.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.804+0000 7f153e000640 1 -- 172.21.15.90:0/2612003595 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e9f4d707d0 con 0x55e9f5037230 2024-04-02T14:43:12.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.804+0000 7f153d600640 1 --2- 172.21.15.90:0/2612003595 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e9f502dec0 0x55e9f50341a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:43:12.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.805+0000 7f153e000640 1 --2- 172.21.15.90:0/2612003595 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e9f5037230 0x55e9f5034c20 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f1530012810 tx=0x7f1530009500 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6b770db3fc6ec882 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:43:12.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.805+0000 7f151ea00640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:43:12.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.805+0000 7f151ea00640 1 -- 172.21.15.90:0/2612003595 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f153001c890 con 0x55e9f5037230 2024-04-02T14:43:12.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.805+0000 7f151ea00640 1 -- 172.21.15.90:0/2612003595 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8594+0+0 (secure 0 0 0) 0x7f153001ca30 con 0x55e9f5037230 2024-04-02T14:43:12.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.805+0000 7f1542e22740 1 -- 172.21.15.90:0/2612003595 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1524009e10 con 0x55e9f5037230 2024-04-02T14:43:12.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.805+0000 7f151ea00640 1 -- 172.21.15.90:0/2612003595 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f153001cc30 con 0x55e9f5037230 2024-04-02T14:43:12.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.805+0000 7f1542e22740 10 client.28052 fetch_fsmap learned FSMap version 27 2024-04-02T14:43:12.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.805+0000 7f1542e22740 1 -- 172.21.15.90:0/2612003595 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55e9f50460e0 con 0x55e9f5037230 2024-04-02T14:43:12.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.805+0000 7f151ea00640 1 -- 172.21.15.90:0/2612003595 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1524009e10 con 0x55e9f5037230 2024-04-02T14:43:12.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.806+0000 7f151ea00640 1 -- 172.21.15.90:0/2612003595 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f1530025020 con 0x55e9f5037230 2024-04-02T14:43:12.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.806+0000 7f1542e22740 10 client.28052 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:43:12.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.806+0000 7f1542e22740 10 client.28052 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-04-02T14:43:12.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.806+0000 7f1542e22740 20 client.28052 populate_metadata read hostname 'smithi090' 2024-04-02T14:43:12.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.806+0000 7f1542e22740 1 --2- 172.21.15.90:0/2612003595 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x55e9f5047590 0x55e9f50499f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:12.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.806+0000 7f1542e22740 4 client.28052 mds_command: new command op to 24547 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:43:12.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.806+0000 7f1542e22740 1 -- 172.21.15.90:0/2612003595 --> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e9f4f52040 con 0x55e9f5047590 2024-04-02T14:43:12.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.806+0000 7f153d600640 1 --2- 172.21.15.90:0/2612003595 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x55e9f5047590 0x55e9f50499f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:12.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.807+0000 7f153d600640 1 --2- 172.21.15.90:0/2612003595 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x55e9f5047590 0x55e9f50499f0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=502aada803348364 server_cookie=4bd41bb6757fbaa2 in_seq=0 out_seq=0 2024-04-02T14:43:12.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.807+0000 7f151ea00640 10 client.28052 ms_handle_connect on v2:172.21.15.177:6838/664895663 2024-04-02T14:43:12.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.807+0000 7f151ea00640 1 -- 172.21.15.90:0/2612003595 <== mds.2 v2:172.21.15.177:6838/664895663 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e9f4f52040 con 0x55e9f5047590 2024-04-02T14:43:12.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.807+0000 7f151ea00640 10 client.28052 handle_command_reply: tid=0 2024-04-02T14:43:12.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.807+0000 7f1542e22740 1 client.28052 shutdown 2024-04-02T14:43:12.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.808+0000 7f1542e22740 1 -- 172.21.15.90:0/2612003595 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e9f5037230 msgr2=0x55e9f5034c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:12.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.808+0000 7f1542e22740 1 --2- 172.21.15.90:0/2612003595 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e9f5037230 0x55e9f5034c20 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f1530012810 tx=0x7f1530009500 comp rx=0 tx=0).stop 2024-04-02T14:43:12.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.808+0000 7f1542e22740 1 -- 172.21.15.90:0/2612003595 shutdown_connections 2024-04-02T14:43:12.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.808+0000 7f1542e22740 1 --2- 172.21.15.90:0/2612003595 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x55e9f5047590 0x55e9f50499f0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:12.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.808+0000 7f1542e22740 1 --2- 172.21.15.90:0/2612003595 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e9f5037230 0x55e9f5034c20 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:12.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.808+0000 7f1542e22740 1 --2- 172.21.15.90:0/2612003595 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e9f502eba0 0x55e9f50346e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:12.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.808+0000 7f151ea00640 0 client.28052 ms_handle_reset on v2:172.21.15.177:6838/664895663 2024-04-02T14:43:12.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.808+0000 7f1542e22740 1 --2- 172.21.15.90:0/2612003595 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e9f502dec0 0x55e9f50341a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:12.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.808+0000 7f1542e22740 1 -- 172.21.15.90:0/2612003595 >> 172.21.15.90:0/2612003595 conn(0x55e9f4ff4a50 msgr2=0x55e9f502a660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:43:12.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.808+0000 7f1542e22740 1 -- 172.21.15.90:0/2612003595 shutdown_connections 2024-04-02T14:43:12.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.808+0000 7f1542e22740 1 -- 172.21.15.90:0/2612003595 wait complete. 2024-04-02T14:43:12.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.808+0000 7f1542e22740 20 client.28052 trim_cache size 0 max 16384 2024-04-02T14:43:12.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.824+0000 7f1542e22740 1 Processor -- start 2024-04-02T14:43:12.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.824+0000 7f1542e22740 1 -- start start 2024-04-02T14:43:12.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.825+0000 7f1542e22740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e9f502dec0 0x55e9f5042cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:12.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.825+0000 7f1542e22740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e9f502eba0 0x55e9f504f2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:12.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.825+0000 7f1542e22740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e9f5037230 0x55e9f504f800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:12.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.825+0000 7f1542e22740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f15381af990 con 0x55e9f502dec0 2024-04-02T14:43:12.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.825+0000 7f1542e22740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f15381b14d0 con 0x55e9f502eba0 2024-04-02T14:43:12.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.825+0000 7f1542e22740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f15381b0060 con 0x55e9f5037230 2024-04-02T14:43:12.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.825+0000 7f153e000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e9f5037230 0x55e9f504f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:12.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.825+0000 7f153cc00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e9f502eba0 0x55e9f504f2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:12.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.825+0000 7f153d600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e9f502dec0 0x55e9f5042cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:12.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.825+0000 7f153e000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e9f5037230 0x55e9f504f800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:43488/0 (socket says 172.21.15.90:43488) 2024-04-02T14:43:12.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.825+0000 7f153e000640 1 -- 172.21.15.90:0/1519868922 learned_addr learned my addr 172.21.15.90:0/1519868922 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:43:12.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.825+0000 7f153cc00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e9f502eba0 0x55e9f504f2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:33600/0 (socket says 172.21.15.90:33600) 2024-04-02T14:43:12.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.825+0000 7f153e000640 1 -- 172.21.15.90:0/1519868922 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e9f502eba0 msgr2=0x55e9f504f2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:43:12.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.825+0000 7f153e000640 1 --2- 172.21.15.90:0/1519868922 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e9f502eba0 0x55e9f504f2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:12.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.825+0000 7f153e000640 1 -- 172.21.15.90:0/1519868922 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e9f502dec0 msgr2=0x55e9f5042cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:43:12.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.825+0000 7f153e000640 1 --2- 172.21.15.90:0/1519868922 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e9f502dec0 0x55e9f5042cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:12.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.825+0000 7f153e000640 1 -- 172.21.15.90:0/1519868922 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e9f5032710 con 0x55e9f5037230 2024-04-02T14:43:12.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.826+0000 7f153e000640 1 --2- 172.21.15.90:0/1519868922 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e9f5037230 0x55e9f504f800 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f153001cf90 tx=0x7f1530005b70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=367eebd5b35f690f server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:43:12.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.826+0000 7f1529c00640 1 -- 172.21.15.90:0/1519868922 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1530004970 con 0x55e9f5037230 2024-04-02T14:43:12.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.826+0000 7f1529c00640 1 -- 172.21.15.90:0/1519868922 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f153001d7c0 con 0x55e9f5037230 2024-04-02T14:43:12.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.826+0000 7f1529c00640 1 -- 172.21.15.90:0/1519868922 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15300278e0 con 0x55e9f5037230 2024-04-02T14:43:12.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.827+0000 7f1542e22740 1 -- 172.21.15.90:0/1519868922 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e9f5037230 msgr2=0x55e9f504f800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:12.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.827+0000 7f1542e22740 1 --2- 172.21.15.90:0/1519868922 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e9f5037230 0x55e9f504f800 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f153001cf90 tx=0x7f1530005b70 comp rx=0 tx=0).stop 2024-04-02T14:43:12.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.827+0000 7f1542e22740 1 -- 172.21.15.90:0/1519868922 shutdown_connections 2024-04-02T14:43:12.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.827+0000 7f1542e22740 1 --2- 172.21.15.90:0/1519868922 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e9f5037230 0x55e9f504f800 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:12.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.827+0000 7f1542e22740 1 --2- 172.21.15.90:0/1519868922 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e9f502eba0 0x55e9f504f2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:12.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.827+0000 7f1542e22740 1 --2- 172.21.15.90:0/1519868922 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e9f502dec0 0x55e9f5042cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:12.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.827+0000 7f1542e22740 1 -- 172.21.15.90:0/1519868922 >> 172.21.15.90:0/1519868922 conn(0x55e9f4ff4a50 msgr2=0x55e9f5028440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:43:12.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.827+0000 7f1542e22740 1 -- 172.21.15.90:0/1519868922 shutdown_connections 2024-04-02T14:43:12.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.827+0000 7f1542e22740 1 -- 172.21.15.90:0/1519868922 wait complete. 2024-04-02T14:43:12.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.828+0000 7f1542e22740 1 -- start start 2024-04-02T14:43:12.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.828+0000 7f1542e22740 1 Processor -- start 2024-04-02T14:43:12.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.828+0000 7f1542e22740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e9f502dec0 0x55e9f503f6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:12.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.828+0000 7f1542e22740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e9f502eba0 0x55e9f5030c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:12.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.828+0000 7f1542e22740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e9f5037230 0x55e9f50311d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:12.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.828+0000 7f1542e22740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f15381b0e00 con 0x55e9f5037230 2024-04-02T14:43:12.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.828+0000 7f1542e22740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f15381b0730 con 0x55e9f502dec0 2024-04-02T14:43:12.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.828+0000 7f1542e22740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f15381ad990 con 0x55e9f502eba0 2024-04-02T14:43:12.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.828+0000 7f1542e22740 10 client.0 _ll_register_callbacks cb 0x7f1530012510 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:43:12.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.829+0000 7f153d600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e9f502dec0 0x55e9f503f6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:12.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.829+0000 7f153cc00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e9f502eba0 0x55e9f5030c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:12.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.829+0000 7f153d600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e9f502dec0 0x55e9f503f6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:33610/0 (socket says 172.21.15.90:33610) 2024-04-02T14:43:12.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.829+0000 7f153d600640 1 -- 172.21.15.90:0/408814923 learned_addr learned my addr 172.21.15.90:0/408814923 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:43:12.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.829+0000 7f153e000640 1 --2- 172.21.15.90:0/408814923 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e9f5037230 0x55e9f50311d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:12.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.829+0000 7f153d600640 1 -- 172.21.15.90:0/408814923 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e9f502eba0 msgr2=0x55e9f5030c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:43:12.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.829+0000 7f153d600640 1 --2- 172.21.15.90:0/408814923 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e9f502eba0 0x55e9f5030c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:12.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.829+0000 7f153d600640 1 -- 172.21.15.90:0/408814923 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e9f5037230 msgr2=0x55e9f50311d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:43:12.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.829+0000 7f153d600640 1 --2- 172.21.15.90:0/408814923 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e9f5037230 0x55e9f50311d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:12.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.829+0000 7f153d600640 1 -- 172.21.15.90:0/408814923 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e9f5031a00 con 0x55e9f502dec0 2024-04-02T14:43:12.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.829+0000 7f153d600640 1 --2- 172.21.15.90:0/408814923 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e9f502dec0 0x55e9f503f6a0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f152c0babb0 tx=0x7f152c0f4040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=be5e3cafc04e6b56 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:43:12.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.829+0000 7f151ea00640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:43:12.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.829+0000 7f151ea00640 1 -- 172.21.15.90:0/408814923 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f152c04b040 con 0x55e9f502dec0 2024-04-02T14:43:12.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.829+0000 7f151ea00640 1 -- 172.21.15.90:0/408814923 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8594+0+0 (secure 0 0 0) 0x7f152c0ba850 con 0x55e9f502dec0 2024-04-02T14:43:12.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.829+0000 7f1542e22740 1 -- 172.21.15.90:0/408814923 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f153001d7c0 con 0x55e9f502dec0 2024-04-02T14:43:12.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.830+0000 7f151ea00640 1 -- 172.21.15.90:0/408814923 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f152c0bdd20 con 0x55e9f502dec0 2024-04-02T14:43:12.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.830+0000 7f1542e22740 10 client.28058 fetch_fsmap learned FSMap version 27 2024-04-02T14:43:12.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.830+0000 7f1542e22740 1 -- 172.21.15.90:0/408814923 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55e9f50b6410 con 0x55e9f502dec0 2024-04-02T14:43:12.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.830+0000 7f151ea00640 1 -- 172.21.15.90:0/408814923 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f153001d7c0 con 0x55e9f502dec0 2024-04-02T14:43:12.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.830+0000 7f151ea00640 1 -- 172.21.15.90:0/408814923 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f152c0de040 con 0x55e9f502dec0 2024-04-02T14:43:12.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.830+0000 7f1542e22740 10 client.28058 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:43:12.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.830+0000 7f1542e22740 10 client.28058 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-04-02T14:43:12.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.830+0000 7f1542e22740 20 client.28058 populate_metadata read hostname 'smithi090' 2024-04-02T14:43:12.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.830+0000 7f1542e22740 1 --2- 172.21.15.90:0/408814923 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x55e9f50b6b80 0x55e9f50b8f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:12.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.830+0000 7f1542e22740 4 client.28058 mds_command: new command op to 24547 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-02T14:43:12.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.830+0000 7f1542e22740 1 -- 172.21.15.90:0/408814923 --> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55e9f4f9fc30 con 0x55e9f50b6b80 2024-04-02T14:43:12.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.831+0000 7f153cc00640 1 --2- 172.21.15.90:0/408814923 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x55e9f50b6b80 0x55e9f50b8f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:12.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.832+0000 7f153cc00640 1 --2- 172.21.15.90:0/408814923 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x55e9f50b6b80 0x55e9f50b8f70 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=b96d488e05ba6e70 server_cookie=c39ad9ce483ab8ad in_seq=0 out_seq=0 2024-04-02T14:43:12.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.832+0000 7f151ea00640 10 client.28058 ms_handle_connect on v2:172.21.15.177:6838/664895663 2024-04-02T14:43:12.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.832+0000 7f151ea00640 1 -- 172.21.15.90:0/408814923 <== mds.2 v2:172.21.15.177:6838/664895663 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55e9f4f9fc30 con 0x55e9f50b6b80 2024-04-02T14:43:12.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.832+0000 7f151ea00640 10 client.28058 handle_command_reply: tid=0 2024-04-02T14:43:12.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.832+0000 7f1542e22740 1 client.28058 shutdown 2024-04-02T14:43:12.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.832+0000 7f1542e22740 1 -- 172.21.15.90:0/408814923 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e9f502dec0 msgr2=0x55e9f503f6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:12.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.832+0000 7f1542e22740 1 --2- 172.21.15.90:0/408814923 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e9f502dec0 0x55e9f503f6a0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f152c0babb0 tx=0x7f152c0f4040 comp rx=0 tx=0).stop 2024-04-02T14:43:12.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.832+0000 7f1542e22740 1 -- 172.21.15.90:0/408814923 shutdown_connections 2024-04-02T14:43:12.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.832+0000 7f1542e22740 1 --2- 172.21.15.90:0/408814923 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x55e9f50b6b80 0x55e9f50b8f70 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:12.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.832+0000 7f1542e22740 1 --2- 172.21.15.90:0/408814923 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e9f5037230 0x55e9f50311d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:12.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.832+0000 7f151ea00640 0 client.28058 ms_handle_reset on v2:172.21.15.177:6838/664895663 2024-04-02T14:43:12.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.832+0000 7f1542e22740 1 --2- 172.21.15.90:0/408814923 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e9f502eba0 0x55e9f5030c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:12.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.832+0000 7f1542e22740 1 --2- 172.21.15.90:0/408814923 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e9f502dec0 0x55e9f503f6a0 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:12.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.832+0000 7f1542e22740 1 -- 172.21.15.90:0/408814923 >> 172.21.15.90:0/408814923 conn(0x55e9f4ff4a50 msgr2=0x55e9f5027260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:43:12.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.833+0000 7f1542e22740 1 -- 172.21.15.90:0/408814923 shutdown_connections 2024-04-02T14:43:12.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.833+0000 7f1542e22740 1 -- 172.21.15.90:0/408814923 wait complete. 2024-04-02T14:43:12.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.833+0000 7f1542e22740 20 client.28058 trim_cache size 0 max 16384 2024-04-02T14:43:12.834 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-02T14:43:12.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.833+0000 7f153ea00640 1 -- 172.21.15.90:0/2199639104 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f15200a3d20 msgr2=0x7f15200a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:12.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.833+0000 7f153ea00640 1 --2- 172.21.15.90:0/2199639104 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f15200a3d20 0x7f15200a61e0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f15300094d0 tx=0x7f1530014040 comp rx=0 tx=0).stop 2024-04-02T14:43:12.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.833+0000 7f153ea00640 1 -- 172.21.15.90:0/2199639104 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1538103df0 msgr2=0x7f153819a4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:12.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.833+0000 7f153ea00640 1 --2- 172.21.15.90:0/2199639104 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1538103df0 0x7f153819a4e0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f152c005ab0 tx=0x7f152c0042f0 comp rx=0 tx=0).stop 2024-04-02T14:43:12.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.834+0000 7f153ea00640 1 -- 172.21.15.90:0/2199639104 shutdown_connections 2024-04-02T14:43:12.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.834+0000 7f153ea00640 1 --2- 172.21.15.90:0/2199639104 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f15200a3d20 0x7f15200a61e0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:12.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.834+0000 7f153ea00640 1 --2- 172.21.15.90:0/2199639104 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f153819eda0 0x7f153819f250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:12.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.834+0000 7f153ea00640 1 --2- 172.21.15.90:0/2199639104 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1538108ec0 0x7f153819aa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:12.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.834+0000 7f153ea00640 1 --2- 172.21.15.90:0/2199639104 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1538103df0 0x7f153819a4e0 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:12.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.834+0000 7f153ea00640 1 -- 172.21.15.90:0/2199639104 >> 172.21.15.90:0/2199639104 conn(0x7f15380ff610 msgr2=0x7f1538101960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:43:12.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.834+0000 7f153ea00640 1 -- 172.21.15.90:0/2199639104 shutdown_connections 2024-04-02T14:43:12.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:12.834+0000 7f153ea00640 1 -- 172.21.15.90:0/2199639104 wait complete. 2024-04-02T14:43:12.851 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-02T14:43:13.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.079+0000 7f21b5a00640 1 -- 172.21.15.90:0/2218493412 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f21b0112940 msgr2=0x7f21b0114d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:13.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.079+0000 7f21b5a00640 1 --2- 172.21.15.90:0/2218493412 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f21b0112940 0x7f21b0114d30 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f21a8009cd0 tx=0x7f21a802c0e0 comp rx=0 tx=0).stop 2024-04-02T14:43:13.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.080+0000 7f21b5a00640 1 -- 172.21.15.90:0/2218493412 shutdown_connections 2024-04-02T14:43:13.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.080+0000 7f21b5a00640 1 --2- 172.21.15.90:0/2218493412 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f21b0112940 0x7f21b0114d30 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:13.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.080+0000 7f21b5a00640 1 --2- 172.21.15.90:0/2218493412 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f21b0076950 0x7f21b0112400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:13.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.080+0000 7f21b5a00640 1 --2- 172.21.15.90:0/2218493412 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f21b0076030 0x7f21b0076410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:13.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.080+0000 7f21b5a00640 1 -- 172.21.15.90:0/2218493412 >> 172.21.15.90:0/2218493412 conn(0x7f21b00ff7b0 msgr2=0x7f21b0101bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:43:13.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.081+0000 7f21b5a00640 1 -- 172.21.15.90:0/2218493412 shutdown_connections 2024-04-02T14:43:13.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.081+0000 7f21b5a00640 1 -- 172.21.15.90:0/2218493412 wait complete. 2024-04-02T14:43:13.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.082+0000 7f21b5a00640 1 Processor -- start 2024-04-02T14:43:13.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.082+0000 7f21b5a00640 1 -- start start 2024-04-02T14:43:13.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.083+0000 7f21b5a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f21b0076030 0x7f21b01a46b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:13.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.083+0000 7f21b5a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f21b0076950 0x7f21b01a4bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:13.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.083+0000 7f21b5a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f21b0112940 0x7f21b019e820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:13.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.083+0000 7f21b5a00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f21b0118460 con 0x7f21b0076030 2024-04-02T14:43:13.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.083+0000 7f21b5a00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f21b01182e0 con 0x7f21b0076950 2024-04-02T14:43:13.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.083+0000 7f21b5a00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f21b01185e0 con 0x7f21b0112940 2024-04-02T14:43:13.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.083+0000 7f21a7e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f21b0076950 0x7f21b01a4bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:13.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.083+0000 7f21a7e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f21b0076950 0x7f21b01a4bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:33628/0 (socket says 172.21.15.90:33628) 2024-04-02T14:43:13.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.083+0000 7f21a7e00640 1 -- 172.21.15.90:0/1730367966 learned_addr learned my addr 172.21.15.90:0/1730367966 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:43:13.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.083+0000 7f21aea00640 1 --2- 172.21.15.90:0/1730367966 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f21b0076030 0x7f21b01a46b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:13.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.083+0000 7f21af400640 1 --2- 172.21.15.90:0/1730367966 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f21b0112940 0x7f21b019e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:13.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.083+0000 7f21a7e00640 1 -- 172.21.15.90:0/1730367966 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f21b0112940 msgr2=0x7f21b019e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:13.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.083+0000 7f21a7e00640 1 --2- 172.21.15.90:0/1730367966 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f21b0112940 0x7f21b019e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:13.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.083+0000 7f21a7e00640 1 -- 172.21.15.90:0/1730367966 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f21b0076030 msgr2=0x7f21b01a46b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:13.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.083+0000 7f21a7e00640 1 --2- 172.21.15.90:0/1730367966 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f21b0076030 0x7f21b01a46b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:13.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.083+0000 7f21a7e00640 1 -- 172.21.15.90:0/1730367966 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21b019eff0 con 0x7f21b0076950 2024-04-02T14:43:13.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.084+0000 7f21a7e00640 1 --2- 172.21.15.90:0/1730367966 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f21b0076950 0x7f21b01a4bf0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f219800e9f0 tx=0x7f219800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:43:13.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.084+0000 7f21a7400640 1 -- 172.21.15.90:0/1730367966 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f219800cd80 con 0x7f21b0076950 2024-04-02T14:43:13.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.084+0000 7f21a7400640 1 -- 172.21.15.90:0/1730367966 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2198010910 con 0x7f21b0076950 2024-04-02T14:43:13.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.084+0000 7f21a7400640 1 -- 172.21.15.90:0/1730367966 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2198018990 con 0x7f21b0076950 2024-04-02T14:43:13.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.084+0000 7f21b5a00640 1 -- 172.21.15.90:0/1730367966 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f21b019f2e0 con 0x7f21b0076950 2024-04-02T14:43:13.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.084+0000 7f21b5a00640 1 -- 172.21.15.90:0/1730367966 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21b01ab500 con 0x7f21b0076950 2024-04-02T14:43:13.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.089+0000 7f21b877b740 1 Processor -- start 2024-04-02T14:43:13.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.089+0000 7f21b877b740 1 -- start start 2024-04-02T14:43:13.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.089+0000 7f21b877b740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556dd88579f0 0x556dd8856740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:13.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.089+0000 7f21b877b740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556dd8856d10 0x556dd8857190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:13.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.089+0000 7f21b877b740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556dd8860080 0x556dd8862470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:13.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.089+0000 7f21b877b740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x556dd8797400 con 0x556dd88579f0 2024-04-02T14:43:13.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.089+0000 7f21b877b740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x556dd877bf10 con 0x556dd8860080 2024-04-02T14:43:13.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.089+0000 7f21b877b740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x556dd87bb6f0 con 0x556dd8856d10 2024-04-02T14:43:13.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.090+0000 7f21a7400640 1 -- 172.21.15.90:0/1730367966 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f2198010ac0 con 0x7f21b0076950 2024-04-02T14:43:13.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.090+0000 7f21aea00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556dd8856d10 0x556dd8857190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:13.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.090+0000 7f21aea00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556dd8856d10 0x556dd8857190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:43518/0 (socket says 172.21.15.90:43518) 2024-04-02T14:43:13.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.090+0000 7f21aea00640 1 -- 172.21.15.90:0/3874987561 learned_addr learned my addr 172.21.15.90:0/3874987561 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:43:13.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.090+0000 7f21af400640 1 --2- 172.21.15.90:0/3874987561 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556dd88579f0 0x556dd8856740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:13.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.090+0000 7f21a7e00640 1 --2- 172.21.15.90:0/3874987561 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556dd8860080 0x556dd8862470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:13.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.090+0000 7f21aea00640 1 -- 172.21.15.90:0/3874987561 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556dd8860080 msgr2=0x556dd8862470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:43:13.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.090+0000 7f21aea00640 1 --2- 172.21.15.90:0/3874987561 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556dd8860080 0x556dd8862470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:13.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.090+0000 7f21aea00640 1 -- 172.21.15.90:0/3874987561 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556dd88579f0 msgr2=0x556dd8856740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:43:13.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.090+0000 7f21aea00640 1 --2- 172.21.15.90:0/3874987561 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556dd88579f0 0x556dd8856740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:13.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.090+0000 7f21aea00640 1 -- 172.21.15.90:0/3874987561 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556dd84cf3f0 con 0x556dd8856d10 2024-04-02T14:43:13.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.091+0000 7f21aea00640 1 --2- 172.21.15.90:0/3874987561 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556dd8856d10 0x556dd8857190 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f21a000bd10 tx=0x7f21a00107a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5fdf056056395855 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:43:13.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.091+0000 7f219fe00640 1 -- 172.21.15.90:0/3874987561 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21a0004030 con 0x556dd8856d10 2024-04-02T14:43:13.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.091+0000 7f219fe00640 1 -- 172.21.15.90:0/3874987561 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f21a00041d0 con 0x556dd8856d10 2024-04-02T14:43:13.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.091+0000 7f219fe00640 1 -- 172.21.15.90:0/3874987561 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21a00117e0 con 0x556dd8856d10 2024-04-02T14:43:13.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.092+0000 7f21a7400640 1 --2- 172.21.15.90:0/1730367966 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f21940a3da0 0x7f21940a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:13.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.092+0000 7f21a7400640 1 -- 172.21.15.90:0/1730367966 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8594+0+0 (secure 0 0 0) 0x7f21980d0a40 con 0x7f21b0076950 2024-04-02T14:43:13.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.092+0000 7f21b877b740 1 -- 172.21.15.90:0/3874987561 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556dd8856d10 msgr2=0x556dd8857190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:13.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.092+0000 7f21af400640 1 --2- 172.21.15.90:0/1730367966 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f21940a3da0 0x7f21940a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:13.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.092+0000 7f21b877b740 1 --2- 172.21.15.90:0/3874987561 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556dd8856d10 0x556dd8857190 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f21a000bd10 tx=0x7f21a00107a0 comp rx=0 tx=0).stop 2024-04-02T14:43:13.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.093+0000 7f21b877b740 1 -- 172.21.15.90:0/3874987561 shutdown_connections 2024-04-02T14:43:13.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.093+0000 7f21b877b740 1 --2- 172.21.15.90:0/3874987561 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556dd8860080 0x556dd8862470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:13.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.093+0000 7f21b877b740 1 --2- 172.21.15.90:0/3874987561 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556dd8856d10 0x556dd8857190 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:13.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.093+0000 7f21b877b740 1 --2- 172.21.15.90:0/3874987561 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556dd88579f0 0x556dd8856740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:13.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.093+0000 7f21b877b740 1 -- 172.21.15.90:0/3874987561 >> 172.21.15.90:0/3874987561 conn(0x556dd881da50 msgr2=0x556dd8851760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:43:13.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.093+0000 7f21b877b740 1 -- 172.21.15.90:0/3874987561 shutdown_connections 2024-04-02T14:43:13.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.093+0000 7f21b877b740 1 -- 172.21.15.90:0/3874987561 wait complete. 2024-04-02T14:43:13.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.093+0000 7f21af400640 1 --2- 172.21.15.90:0/1730367966 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f21940a3da0 0x7f21940a6260 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f21a8009bf0 tx=0x7f21a8005d80 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:43:13.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.093+0000 7f21b877b740 1 -- start start 2024-04-02T14:43:13.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.094+0000 7f21b877b740 1 Processor -- start 2024-04-02T14:43:13.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.094+0000 7f21b877b740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556dd8856d10 0x556dd885cff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:13.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.094+0000 7f21b877b740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556dd88579f0 0x556dd885d530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:13.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.094+0000 7f21b877b740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556dd8860080 0x556dd885da70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:13.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.094+0000 7f21b877b740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x556dd87b76a0 con 0x556dd88579f0 2024-04-02T14:43:13.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.095+0000 7f21b877b740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x556dd8763980 con 0x556dd8860080 2024-04-02T14:43:13.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.095+0000 7f21b877b740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x556dd8776cb0 con 0x556dd8856d10 2024-04-02T14:43:13.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.095+0000 7f21b877b740 10 client.0 _ll_register_callbacks cb 0x556dd87766a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:43:13.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.095+0000 7f21af400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556dd8860080 0x556dd885da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:13.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.095+0000 7f21aea00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556dd8856d10 0x556dd885cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:13.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.095+0000 7f21af400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556dd8860080 0x556dd885da70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:33656/0 (socket says 172.21.15.90:33656) 2024-04-02T14:43:13.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.095+0000 7f21af400640 1 -- 172.21.15.90:0/2618840052 learned_addr learned my addr 172.21.15.90:0/2618840052 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:43:13.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.095+0000 7f21a7e00640 1 --2- 172.21.15.90:0/2618840052 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556dd88579f0 0x556dd885d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:13.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.095+0000 7f21aea00640 1 -- 172.21.15.90:0/2618840052 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556dd8860080 msgr2=0x556dd885da70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:43:13.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.095+0000 7f21aea00640 1 --2- 172.21.15.90:0/2618840052 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556dd8860080 0x556dd885da70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:13.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.095+0000 7f21aea00640 1 -- 172.21.15.90:0/2618840052 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556dd88579f0 msgr2=0x556dd885d530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:43:13.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.095+0000 7f21aea00640 1 --2- 172.21.15.90:0/2618840052 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556dd88579f0 0x556dd885d530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:13.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.095+0000 7f21aea00640 1 -- 172.21.15.90:0/2618840052 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556dd85997d0 con 0x556dd8856d10 2024-04-02T14:43:13.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.095+0000 7f21af400640 1 --2- 172.21.15.90:0/2618840052 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556dd8860080 0x556dd885da70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:43:13.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.096+0000 7f21aea00640 1 --2- 172.21.15.90:0/2618840052 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556dd8856d10 0x556dd885cff0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f21a000bd10 tx=0x7f21a000b8e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4cd1ebb6a04ddba6 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:43:13.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.096+0000 7f219cc00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:43:13.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.096+0000 7f219cc00640 1 -- 172.21.15.90:0/2618840052 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21a0004030 con 0x556dd8856d10 2024-04-02T14:43:13.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.096+0000 7f21b877b740 1 -- 172.21.15.90:0/2618840052 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f21a00041d0 con 0x556dd8856d10 2024-04-02T14:43:13.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.096+0000 7f219cc00640 1 -- 172.21.15.90:0/2618840052 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8594+0+0 (secure 0 0 0) 0x7f21a00037f0 con 0x556dd8856d10 2024-04-02T14:43:13.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.096+0000 7f219cc00640 1 -- 172.21.15.90:0/2618840052 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f21a00039f0 con 0x556dd8856d10 2024-04-02T14:43:13.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.096+0000 7f21b877b740 10 client.27907 fetch_fsmap learned FSMap version 27 2024-04-02T14:43:13.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.096+0000 7f21b877b740 1 -- 172.21.15.90:0/2618840052 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x556dd886eef0 con 0x556dd8856d10 2024-04-02T14:43:13.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.097+0000 7f219cc00640 1 -- 172.21.15.90:0/2618840052 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f21a00041d0 con 0x556dd8856d10 2024-04-02T14:43:13.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.097+0000 7f219cc00640 1 -- 172.21.15.90:0/2618840052 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f21a0023070 con 0x556dd8856d10 2024-04-02T14:43:13.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.097+0000 7f21b877b740 10 client.27907 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:43:13.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.097+0000 7f21b877b740 10 client.27907 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2024-04-02T14:43:13.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.097+0000 7f21b877b740 20 client.27907 populate_metadata read hostname 'smithi090' 2024-04-02T14:43:13.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.097+0000 7f21b877b740 1 --2- 172.21.15.90:0/2618840052 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x556dd88702a0 0x556dd8872690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:13.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.097+0000 7f21b877b740 4 client.27907 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:43:13.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.097+0000 7f21b877b740 1 -- 172.21.15.90:0/2618840052 --> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556dd877b040 con 0x556dd88702a0 2024-04-02T14:43:13.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.097+0000 7f21a7e00640 1 --2- 172.21.15.90:0/2618840052 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x556dd88702a0 0x556dd8872690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:13.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.098+0000 7f21a7e00640 1 --2- 172.21.15.90:0/2618840052 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x556dd88702a0 0x556dd8872690 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=41e5d303276ad20f server_cookie=bc5433a206c5a132 in_seq=0 out_seq=0 2024-04-02T14:43:13.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.098+0000 7f219cc00640 10 client.27907 ms_handle_connect on v2:172.21.15.90:6838/1498915870 2024-04-02T14:43:13.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.099+0000 7f219cc00640 1 -- 172.21.15.90:0/2618840052 <== mds.1 v2:172.21.15.90:6838/1498915870 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x556dd877b040 con 0x556dd88702a0 2024-04-02T14:43:13.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.099+0000 7f219cc00640 10 client.27907 handle_command_reply: tid=0 2024-04-02T14:43:13.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.099+0000 7f21b877b740 1 client.27907 shutdown 2024-04-02T14:43:13.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.100+0000 7f21b877b740 1 -- 172.21.15.90:0/2618840052 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556dd8856d10 msgr2=0x556dd885cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:13.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.100+0000 7f21b877b740 1 --2- 172.21.15.90:0/2618840052 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556dd8856d10 0x556dd885cff0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f21a000bd10 tx=0x7f21a000b8e0 comp rx=0 tx=0).stop 2024-04-02T14:43:13.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.100+0000 7f21b877b740 1 -- 172.21.15.90:0/2618840052 shutdown_connections 2024-04-02T14:43:13.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.100+0000 7f21b877b740 1 --2- 172.21.15.90:0/2618840052 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x556dd88702a0 0x556dd8872690 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:13.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.100+0000 7f21b877b740 1 --2- 172.21.15.90:0/2618840052 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556dd8860080 0x556dd885da70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:13.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.100+0000 7f21b877b740 1 --2- 172.21.15.90:0/2618840052 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556dd88579f0 0x556dd885d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:13.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.100+0000 7f21b877b740 1 --2- 172.21.15.90:0/2618840052 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556dd8856d10 0x556dd885cff0 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:13.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.100+0000 7f21b877b740 1 -- 172.21.15.90:0/2618840052 >> 172.21.15.90:0/2618840052 conn(0x556dd881da50 msgr2=0x556dd88534b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:43:13.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.100+0000 7f219cc00640 0 client.27907 ms_handle_reset on v2:172.21.15.90:6838/1498915870 2024-04-02T14:43:13.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.100+0000 7f21b877b740 1 -- 172.21.15.90:0/2618840052 shutdown_connections 2024-04-02T14:43:13.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.100+0000 7f21b877b740 1 -- 172.21.15.90:0/2618840052 wait complete. 2024-04-02T14:43:13.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.100+0000 7f21b877b740 20 client.27907 trim_cache size 0 max 16384 2024-04-02T14:43:13.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.117+0000 7f21b877b740 1 Processor -- start 2024-04-02T14:43:13.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.117+0000 7f21b877b740 1 -- start start 2024-04-02T14:43:13.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.118+0000 7f21b877b740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556dd8856d10 0x556dd8853680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:13.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.118+0000 7f21b877b740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556dd88579f0 0x556dd886bb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:13.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.118+0000 7f21b877b740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556dd8860080 0x556dd885c550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:13.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.118+0000 7f21b877b740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f21b01b40c0 con 0x556dd8856d10 2024-04-02T14:43:13.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.118+0000 7f21b877b740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f21b01b5c00 con 0x556dd8860080 2024-04-02T14:43:13.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.118+0000 7f21b877b740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f21b01b4790 con 0x556dd88579f0 2024-04-02T14:43:13.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.118+0000 7f21aea00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556dd8856d10 0x556dd8853680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:13.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.118+0000 7f21a7e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556dd88579f0 0x556dd886bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:13.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.118+0000 7f21aea00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556dd8856d10 0x556dd8853680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:37536/0 (socket says 172.21.15.90:37536) 2024-04-02T14:43:13.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.118+0000 7f21aea00640 1 -- 172.21.15.90:0/1812803859 learned_addr learned my addr 172.21.15.90:0/1812803859 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:43:13.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.118+0000 7f21a7e00640 1 --2- 172.21.15.90:0/1812803859 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556dd88579f0 0x556dd886bb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:43538/0 (socket says 172.21.15.90:43538) 2024-04-02T14:43:13.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.118+0000 7f21af400640 1 --2- 172.21.15.90:0/1812803859 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556dd8860080 0x556dd885c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:13.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.119+0000 7f21a7e00640 1 -- 172.21.15.90:0/1812803859 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556dd8860080 msgr2=0x556dd885c550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:43:13.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.119+0000 7f21a7e00640 1 --2- 172.21.15.90:0/1812803859 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556dd8860080 0x556dd885c550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:13.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.119+0000 7f21a7e00640 1 -- 172.21.15.90:0/1812803859 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556dd8856d10 msgr2=0x556dd8853680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:43:13.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.119+0000 7f21a7e00640 1 --2- 172.21.15.90:0/1812803859 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556dd8856d10 0x556dd8853680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:13.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.119+0000 7f21a7e00640 1 -- 172.21.15.90:0/1812803859 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556dd8872c20 con 0x556dd88579f0 2024-04-02T14:43:13.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.119+0000 7f21a7e00640 1 --2- 172.21.15.90:0/1812803859 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556dd88579f0 0x556dd886bb50 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f21980ce140 tx=0x7f21980d2fc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bdcaeb73b3273b67 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:43:13.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.119+0000 7f219f400640 1 -- 172.21.15.90:0/1812803859 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21980d7070 con 0x556dd88579f0 2024-04-02T14:43:13.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.119+0000 7f219f400640 1 -- 172.21.15.90:0/1812803859 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f219809ca40 con 0x556dd88579f0 2024-04-02T14:43:13.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.119+0000 7f219f400640 1 -- 172.21.15.90:0/1812803859 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21980bbb50 con 0x556dd88579f0 2024-04-02T14:43:13.120 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.120+0000 7f21b877b740 1 -- 172.21.15.90:0/1812803859 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556dd88579f0 msgr2=0x556dd886bb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:13.120 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.120+0000 7f21b877b740 1 --2- 172.21.15.90:0/1812803859 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556dd88579f0 0x556dd886bb50 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f21980ce140 tx=0x7f21980d2fc0 comp rx=0 tx=0).stop 2024-04-02T14:43:13.120 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.120+0000 7f21b877b740 1 -- 172.21.15.90:0/1812803859 shutdown_connections 2024-04-02T14:43:13.120 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.120+0000 7f21b877b740 1 --2- 172.21.15.90:0/1812803859 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556dd8860080 0x556dd885c550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:13.120 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.120+0000 7f21b877b740 1 --2- 172.21.15.90:0/1812803859 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556dd88579f0 0x556dd886bb50 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:13.120 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.120+0000 7f21b877b740 1 --2- 172.21.15.90:0/1812803859 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556dd8856d10 0x556dd8853680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:13.120 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.121+0000 7f21b877b740 1 -- 172.21.15.90:0/1812803859 >> 172.21.15.90:0/1812803859 conn(0x556dd881da50 msgr2=0x556dd88508a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:43:13.120 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.121+0000 7f21b877b740 1 -- 172.21.15.90:0/1812803859 shutdown_connections 2024-04-02T14:43:13.120 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.121+0000 7f21b877b740 1 -- 172.21.15.90:0/1812803859 wait complete. 2024-04-02T14:43:13.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.121+0000 7f21b877b740 1 -- start start 2024-04-02T14:43:13.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.121+0000 7f21b877b740 1 Processor -- start 2024-04-02T14:43:13.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.122+0000 7f21b877b740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556dd8856d10 0x556dd8868340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:13.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.122+0000 7f21b877b740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556dd88579f0 0x556dd8859880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:13.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.122+0000 7f21b877b740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556dd8860080 0x556dd8859dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:13.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.122+0000 7f21b877b740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f21b01b5530 con 0x556dd88579f0 2024-04-02T14:43:13.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.122+0000 7f21b877b740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f21b01b4e60 con 0x556dd8856d10 2024-04-02T14:43:13.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.122+0000 7f21b877b740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f21b01b20c0 con 0x556dd8860080 2024-04-02T14:43:13.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.122+0000 7f21b877b740 10 client.0 _ll_register_callbacks cb 0x7f21a0003d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:43:13.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.122+0000 7f21aea00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556dd8856d10 0x556dd8868340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:13.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.122+0000 7f21aea00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556dd8856d10 0x556dd8868340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:33676/0 (socket says 172.21.15.90:33676) 2024-04-02T14:43:13.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.122+0000 7f21aea00640 1 -- 172.21.15.90:0/2577364509 learned_addr learned my addr 172.21.15.90:0/2577364509 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:43:13.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.122+0000 7f21af400640 1 --2- 172.21.15.90:0/2577364509 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556dd8860080 0x556dd8859dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:13.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.122+0000 7f21a7e00640 1 --2- 172.21.15.90:0/2577364509 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556dd88579f0 0x556dd8859880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:13.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.123+0000 7f21aea00640 1 -- 172.21.15.90:0/2577364509 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556dd8860080 msgr2=0x556dd8859dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:43:13.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.123+0000 7f21aea00640 1 --2- 172.21.15.90:0/2577364509 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556dd8860080 0x556dd8859dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:13.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.123+0000 7f21aea00640 1 -- 172.21.15.90:0/2577364509 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556dd88579f0 msgr2=0x556dd8859880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:43:13.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.123+0000 7f21aea00640 1 --2- 172.21.15.90:0/2577364509 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556dd88579f0 0x556dd8859880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:13.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.123+0000 7f21aea00640 1 -- 172.21.15.90:0/2577364509 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556dd8855d40 con 0x556dd8856d10 2024-04-02T14:43:13.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.123+0000 7f21af400640 1 --2- 172.21.15.90:0/2577364509 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556dd8860080 0x556dd8859dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:43:13.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.123+0000 7f21aea00640 1 --2- 172.21.15.90:0/2577364509 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556dd8856d10 0x556dd8868340 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f21a0004c30 tx=0x7f21a00116e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f06a6d6a74c4e907 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:43:13.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.123+0000 7f219cc00640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:43:13.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.123+0000 7f219cc00640 1 -- 172.21.15.90:0/2577364509 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21a0003b90 con 0x556dd8856d10 2024-04-02T14:43:13.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.123+0000 7f219cc00640 1 -- 172.21.15.90:0/2577364509 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8594+0+0 (secure 0 0 0) 0x7f21a0004790 con 0x556dd8856d10 2024-04-02T14:43:13.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.123+0000 7f21b877b740 1 -- 172.21.15.90:0/2577364509 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f219809ca40 con 0x556dd8856d10 2024-04-02T14:43:13.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.124+0000 7f219cc00640 1 -- 172.21.15.90:0/2577364509 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f21a0004990 con 0x556dd8856d10 2024-04-02T14:43:13.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.124+0000 7f21b877b740 10 client.28076 fetch_fsmap learned FSMap version 27 2024-04-02T14:43:13.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.124+0000 7f21b877b740 1 -- 172.21.15.90:0/2577364509 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x556dd8867ea0 con 0x556dd8856d10 2024-04-02T14:43:13.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.124+0000 7f219cc00640 1 -- 172.21.15.90:0/2577364509 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f219809ca40 con 0x556dd8856d10 2024-04-02T14:43:13.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.125+0000 7f219cc00640 1 -- 172.21.15.90:0/2577364509 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f21a0023040 con 0x556dd8856d10 2024-04-02T14:43:13.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.125+0000 7f21b877b740 10 client.28076 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:43:13.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.125+0000 7f21b877b740 10 client.28076 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2024-04-02T14:43:13.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.125+0000 7f21b877b740 20 client.28076 populate_metadata read hostname 'smithi090' 2024-04-02T14:43:13.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.125+0000 7f21b877b740 1 --2- 172.21.15.90:0/2577364509 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x556dd88d6a80 0x556dd88d8e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:13.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.125+0000 7f21b877b740 4 client.28076 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-02T14:43:13.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.125+0000 7f21b877b740 1 -- 172.21.15.90:0/2577364509 --> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] -- command(tid 0: {"prefix": "damage ls"}) -- 0x556dd87c8c30 con 0x556dd88d6a80 2024-04-02T14:43:13.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.126+0000 7f21a7e00640 1 --2- 172.21.15.90:0/2577364509 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x556dd88d6a80 0x556dd88d8e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:13.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.127+0000 7f21a7e00640 1 --2- 172.21.15.90:0/2577364509 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x556dd88d6a80 0x556dd88d8e70 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=216ab32ac2872f5c server_cookie=300ab5245dcabea1 in_seq=0 out_seq=0 2024-04-02T14:43:13.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.127+0000 7f219cc00640 10 client.28076 ms_handle_connect on v2:172.21.15.90:6838/1498915870 2024-04-02T14:43:13.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.127+0000 7f219cc00640 1 -- 172.21.15.90:0/2577364509 <== mds.1 v2:172.21.15.90:6838/1498915870 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x556dd87c8c30 con 0x556dd88d6a80 2024-04-02T14:43:13.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.127+0000 7f219cc00640 10 client.28076 handle_command_reply: tid=0 2024-04-02T14:43:13.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.127+0000 7f21b877b740 1 client.28076 shutdown 2024-04-02T14:43:13.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.127+0000 7f21b877b740 1 -- 172.21.15.90:0/2577364509 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556dd8856d10 msgr2=0x556dd8868340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:13.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.127+0000 7f21b877b740 1 --2- 172.21.15.90:0/2577364509 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556dd8856d10 0x556dd8868340 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f21a0004c30 tx=0x7f21a00116e0 comp rx=0 tx=0).stop 2024-04-02T14:43:13.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.127+0000 7f21b877b740 1 -- 172.21.15.90:0/2577364509 shutdown_connections 2024-04-02T14:43:13.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.127+0000 7f21b877b740 1 --2- 172.21.15.90:0/2577364509 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x556dd88d6a80 0x556dd88d8e70 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:13.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.127+0000 7f21b877b740 1 --2- 172.21.15.90:0/2577364509 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556dd8860080 0x556dd8859dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:13.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.127+0000 7f21b877b740 1 --2- 172.21.15.90:0/2577364509 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556dd88579f0 0x556dd8859880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:13.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.127+0000 7f21b877b740 1 --2- 172.21.15.90:0/2577364509 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556dd8856d10 0x556dd8868340 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:13.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.127+0000 7f219cc00640 0 client.28076 ms_handle_reset on v2:172.21.15.90:6838/1498915870 2024-04-02T14:43:13.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.127+0000 7f21b877b740 1 -- 172.21.15.90:0/2577364509 >> 172.21.15.90:0/2577364509 conn(0x556dd881da50 msgr2=0x556dd885d550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:43:13.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.128+0000 7f21b877b740 1 -- 172.21.15.90:0/2577364509 shutdown_connections 2024-04-02T14:43:13.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.128+0000 7f21b877b740 1 -- 172.21.15.90:0/2577364509 wait complete. 2024-04-02T14:43:13.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.128+0000 7f21b877b740 20 client.28076 trim_cache size 0 max 16384 2024-04-02T14:43:13.128 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-02T14:43:13.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.128+0000 7f21b5a00640 1 -- 172.21.15.90:0/1730367966 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f21940a3da0 msgr2=0x7f21940a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:13.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.128+0000 7f21b5a00640 1 --2- 172.21.15.90:0/1730367966 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f21940a3da0 0x7f21940a6260 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f21a8009bf0 tx=0x7f21a8005d80 comp rx=0 tx=0).stop 2024-04-02T14:43:13.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.128+0000 7f21b5a00640 1 -- 172.21.15.90:0/1730367966 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f21b0076950 msgr2=0x7f21b01a4bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:13.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.128+0000 7f21b5a00640 1 --2- 172.21.15.90:0/1730367966 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f21b0076950 0x7f21b01a4bf0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f219800e9f0 tx=0x7f219800eeb0 comp rx=0 tx=0).stop 2024-04-02T14:43:13.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.129+0000 7f21b5a00640 1 -- 172.21.15.90:0/1730367966 shutdown_connections 2024-04-02T14:43:13.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.129+0000 7f21b5a00640 1 --2- 172.21.15.90:0/1730367966 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f21940a3da0 0x7f21940a6260 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:13.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.129+0000 7f21b5a00640 1 --2- 172.21.15.90:0/1730367966 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f21b0112940 0x7f21b019e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:13.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.129+0000 7f21b5a00640 1 --2- 172.21.15.90:0/1730367966 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f21b0076950 0x7f21b01a4bf0 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:13.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.129+0000 7f21b5a00640 1 --2- 172.21.15.90:0/1730367966 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f21b0076030 0x7f21b01a46b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:13.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.129+0000 7f21b5a00640 1 -- 172.21.15.90:0/1730367966 >> 172.21.15.90:0/1730367966 conn(0x7f21b00ff7b0 msgr2=0x7f21b0101ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:43:13.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.129+0000 7f21b5a00640 1 -- 172.21.15.90:0/1730367966 shutdown_connections 2024-04-02T14:43:13.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.129+0000 7f21b5a00640 1 -- 172.21.15.90:0/1730367966 wait complete. 2024-04-02T14:43:13.143 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:43:13 smithi090 ceph-mon[31674]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T14:43:13.145 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-02T14:43:13.356 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:43:13 smithi092 ceph-mon[36497]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T14:43:13.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.359+0000 7f3b4a800640 1 -- 172.21.15.90:0/3517724962 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3b44103f10 msgr2=0x7f3b441042f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:13.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.359+0000 7f3b4a800640 1 --2- 172.21.15.90:0/3517724962 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3b44103f10 0x7f3b441042f0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f3b34009900 tx=0x7f3b3402bc50 comp rx=0 tx=0).stop 2024-04-02T14:43:13.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.359+0000 7f3b4a800640 1 -- 172.21.15.90:0/3517724962 shutdown_connections 2024-04-02T14:43:13.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.359+0000 7f3b4a800640 1 --2- 172.21.15.90:0/3517724962 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3b44108fe0 0x7f3b44110bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:13.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.359+0000 7f3b4a800640 1 --2- 172.21.15.90:0/3517724962 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3b441048c0 0x7f3b44108900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:13.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.359+0000 7f3b4a800640 1 --2- 172.21.15.90:0/3517724962 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3b44103f10 0x7f3b441042f0 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:13.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.359+0000 7f3b4a800640 1 -- 172.21.15.90:0/3517724962 >> 172.21.15.90:0/3517724962 conn(0x7f3b440ff7b0 msgr2=0x7f3b44101bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:43:13.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.360+0000 7f3b4a800640 1 -- 172.21.15.90:0/3517724962 shutdown_connections 2024-04-02T14:43:13.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.360+0000 7f3b4a800640 1 -- 172.21.15.90:0/3517724962 wait complete. 2024-04-02T14:43:13.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.361+0000 7f3b4a800640 1 Processor -- start 2024-04-02T14:43:13.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.361+0000 7f3b4a800640 1 -- start start 2024-04-02T14:43:13.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.361+0000 7f3b4a800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3b44103f10 0x7f3b4419a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:13.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.361+0000 7f3b4a800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3b44108fe0 0x7f3b4419ab30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:13.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.361+0000 7f3b4a800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3b4419eeb0 0x7f3b4419f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:13.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.361+0000 7f3b4a800640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f3b441140d0 con 0x7f3b44108fe0 2024-04-02T14:43:13.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.361+0000 7f3b4a800640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f3b44113f50 con 0x7f3b44103f10 2024-04-02T14:43:13.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.361+0000 7f3b4a800640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f3b44114250 con 0x7f3b4419eeb0 2024-04-02T14:43:13.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.362+0000 7f3b3be00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3b44108fe0 0x7f3b4419ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:13.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.362+0000 7f3b43400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3b44103f10 0x7f3b4419a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:13.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.362+0000 7f3b43e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3b4419eeb0 0x7f3b4419f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:13.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.362+0000 7f3b43400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3b44103f10 0x7f3b4419a5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:33700/0 (socket says 172.21.15.90:33700) 2024-04-02T14:43:13.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.362+0000 7f3b43400640 1 -- 172.21.15.90:0/3805102841 learned_addr learned my addr 172.21.15.90:0/3805102841 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:43:13.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.362+0000 7f3b43400640 1 -- 172.21.15.90:0/3805102841 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3b4419eeb0 msgr2=0x7f3b4419f360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:13.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.362+0000 7f3b43400640 1 --2- 172.21.15.90:0/3805102841 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3b4419eeb0 0x7f3b4419f360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:13.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.362+0000 7f3b43400640 1 -- 172.21.15.90:0/3805102841 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3b44108fe0 msgr2=0x7f3b4419ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:13.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.362+0000 7f3b43400640 1 --2- 172.21.15.90:0/3805102841 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3b44108fe0 0x7f3b4419ab30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:13.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.362+0000 7f3b43400640 1 -- 172.21.15.90:0/3805102841 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b4419f9a0 con 0x7f3b44103f10 2024-04-02T14:43:13.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.362+0000 7f3b43400640 1 --2- 172.21.15.90:0/3805102841 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3b44103f10 0x7f3b4419a5f0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f3b34005ab0 tx=0x7f3b340042d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:43:13.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.363+0000 7f3b40c00640 1 -- 172.21.15.90:0/3805102841 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b34041070 con 0x7f3b44103f10 2024-04-02T14:43:13.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.363+0000 7f3b40c00640 1 -- 172.21.15.90:0/3805102841 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3b340044c0 con 0x7f3b44103f10 2024-04-02T14:43:13.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.363+0000 7f3b4a800640 1 -- 172.21.15.90:0/3805102841 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b4419fc30 con 0x7f3b44103f10 2024-04-02T14:43:13.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.363+0000 7f3b40c00640 1 -- 172.21.15.90:0/3805102841 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b34003d30 con 0x7f3b44103f10 2024-04-02T14:43:13.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.363+0000 7f3b4a800640 1 -- 172.21.15.90:0/3805102841 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b441a7450 con 0x7f3b44103f10 2024-04-02T14:43:13.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.366+0000 7f3b40c00640 1 -- 172.21.15.90:0/3805102841 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f3b34044070 con 0x7f3b44103f10 2024-04-02T14:43:13.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.366+0000 7f3b4d464740 1 Processor -- start 2024-04-02T14:43:13.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.367+0000 7f3b4d464740 1 -- start start 2024-04-02T14:43:13.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.367+0000 7f3b4d464740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555d8649b9f0 0x555d8649a740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:13.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.367+0000 7f3b4d464740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555d8649ad10 0x555d8649b190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:13.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.367+0000 7f3b4d464740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555d864a4080 0x555d864a6470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:13.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.367+0000 7f3b4d464740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x555d863db400 con 0x555d8649b9f0 2024-04-02T14:43:13.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.367+0000 7f3b4d464740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x555d863bff10 con 0x555d864a4080 2024-04-02T14:43:13.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.367+0000 7f3b4d464740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x555d863ff6f0 con 0x555d8649ad10 2024-04-02T14:43:13.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.367+0000 7f3b43400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555d8649ad10 0x555d8649b190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:13.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.368+0000 7f3b43400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555d8649ad10 0x555d8649b190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:43590/0 (socket says 172.21.15.90:43590) 2024-04-02T14:43:13.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.368+0000 7f3b43400640 1 -- 172.21.15.90:0/2596348401 learned_addr learned my addr 172.21.15.90:0/2596348401 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:43:13.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.368+0000 7f3b43e00640 1 --2- 172.21.15.90:0/2596348401 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555d8649b9f0 0x555d8649a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:13.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.368+0000 7f3b3be00640 1 --2- 172.21.15.90:0/2596348401 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555d864a4080 0x555d864a6470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:13.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.368+0000 7f3b43400640 1 -- 172.21.15.90:0/2596348401 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555d864a4080 msgr2=0x555d864a6470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:43:13.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.368+0000 7f3b43400640 1 --2- 172.21.15.90:0/2596348401 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555d864a4080 0x555d864a6470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:13.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.368+0000 7f3b43400640 1 -- 172.21.15.90:0/2596348401 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555d8649b9f0 msgr2=0x555d8649a740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:43:13.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.368+0000 7f3b43400640 1 --2- 172.21.15.90:0/2596348401 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555d8649b9f0 0x555d8649a740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:13.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.368+0000 7f3b43400640 1 -- 172.21.15.90:0/2596348401 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555d861133f0 con 0x555d8649ad10 2024-04-02T14:43:13.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.368+0000 7f3b43400640 1 --2- 172.21.15.90:0/2596348401 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555d8649ad10 0x555d8649b190 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f3b340ef780 tx=0x7f3b340fa1a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9cc85697563afaa8 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:43:13.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.368+0000 7f3b40c00640 1 --2- 172.21.15.90:0/3805102841 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f3b280a3d20 0x7f3b280a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:13.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.369+0000 7f3b38c00640 1 -- 172.21.15.90:0/2596348401 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b340ba330 con 0x555d8649ad10 2024-04-02T14:43:13.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.369+0000 7f3b38c00640 1 -- 172.21.15.90:0/2596348401 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3b340ff030 con 0x555d8649ad10 2024-04-02T14:43:13.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.369+0000 7f3b38c00640 1 -- 172.21.15.90:0/2596348401 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b340bd4e0 con 0x555d8649ad10 2024-04-02T14:43:13.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.369+0000 7f3b40c00640 1 -- 172.21.15.90:0/3805102841 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8594+0+0 (secure 0 0 0) 0x7f3b340f2090 con 0x7f3b44103f10 2024-04-02T14:43:13.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.369+0000 7f3b43e00640 1 --2- 172.21.15.90:0/3805102841 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f3b280a3d20 0x7f3b280a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:13.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.370+0000 7f3b4d464740 1 -- 172.21.15.90:0/2596348401 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555d8649ad10 msgr2=0x555d8649b190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:13.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.370+0000 7f3b4d464740 1 --2- 172.21.15.90:0/2596348401 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555d8649ad10 0x555d8649b190 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f3b340ef780 tx=0x7f3b340fa1a0 comp rx=0 tx=0).stop 2024-04-02T14:43:13.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.370+0000 7f3b43e00640 1 --2- 172.21.15.90:0/3805102841 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f3b280a3d20 0x7f3b280a61e0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f3b3c005fd0 tx=0x7f3b3c009500 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:43:13.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.370+0000 7f3b4d464740 1 -- 172.21.15.90:0/2596348401 shutdown_connections 2024-04-02T14:43:13.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.370+0000 7f3b4d464740 1 --2- 172.21.15.90:0/2596348401 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555d864a4080 0x555d864a6470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:13.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.370+0000 7f3b4d464740 1 --2- 172.21.15.90:0/2596348401 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555d8649ad10 0x555d8649b190 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:13.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.370+0000 7f3b4d464740 1 --2- 172.21.15.90:0/2596348401 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555d8649b9f0 0x555d8649a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:13.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.370+0000 7f3b4d464740 1 -- 172.21.15.90:0/2596348401 >> 172.21.15.90:0/2596348401 conn(0x555d86461a50 msgr2=0x555d86495760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:43:13.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.370+0000 7f3b4d464740 1 -- 172.21.15.90:0/2596348401 shutdown_connections 2024-04-02T14:43:13.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.370+0000 7f3b4d464740 1 -- 172.21.15.90:0/2596348401 wait complete. 2024-04-02T14:43:13.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.371+0000 7f3b4d464740 1 -- start start 2024-04-02T14:43:13.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.371+0000 7f3b4d464740 1 Processor -- start 2024-04-02T14:43:13.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.371+0000 7f3b4d464740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555d8649ad10 0x555d864a0ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:13.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.371+0000 7f3b4d464740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555d8649b9f0 0x555d864a1530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:13.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.371+0000 7f3b4d464740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555d864a4080 0x555d864a1a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:13.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.371+0000 7f3b4d464740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x555d863fb6a0 con 0x555d8649ad10 2024-04-02T14:43:13.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.371+0000 7f3b4d464740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x555d863a7980 con 0x555d864a4080 2024-04-02T14:43:13.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.371+0000 7f3b4d464740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x555d863bacb0 con 0x555d8649b9f0 2024-04-02T14:43:13.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.371+0000 7f3b4d464740 10 client.0 _ll_register_callbacks cb 0x555d863ba6a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:43:13.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.372+0000 7f3b3be00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555d8649b9f0 0x555d864a1530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:13.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.372+0000 7f3b3be00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555d8649b9f0 0x555d864a1530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:43594/0 (socket says 172.21.15.90:43594) 2024-04-02T14:43:13.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.372+0000 7f3b3be00640 1 -- 172.21.15.90:0/1672733155 learned_addr learned my addr 172.21.15.90:0/1672733155 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:43:13.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.372+0000 7f3b43e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555d864a4080 0x555d864a1a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:13.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.372+0000 7f3b43400640 1 --2- 172.21.15.90:0/1672733155 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555d8649ad10 0x555d864a0ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:13.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.372+0000 7f3b3be00640 1 -- 172.21.15.90:0/1672733155 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555d864a4080 msgr2=0x555d864a1a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:43:13.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.372+0000 7f3b3be00640 1 --2- 172.21.15.90:0/1672733155 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555d864a4080 0x555d864a1a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:13.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.372+0000 7f3b3be00640 1 -- 172.21.15.90:0/1672733155 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555d8649ad10 msgr2=0x555d864a0ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:43:13.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.372+0000 7f3b3be00640 1 --2- 172.21.15.90:0/1672733155 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555d8649ad10 0x555d864a0ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:13.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.372+0000 7f3b3be00640 1 -- 172.21.15.90:0/1672733155 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555d861dd7d0 con 0x555d8649b9f0 2024-04-02T14:43:13.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.372+0000 7f3b3be00640 1 --2- 172.21.15.90:0/1672733155 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555d8649b9f0 0x555d864a1530 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f3b2c012700 tx=0x7f3b2c012bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=48681b7934282cd8 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:43:13.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.372+0000 7f3b31600640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:43:13.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.373+0000 7f3b31600640 1 -- 172.21.15.90:0/1672733155 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b2c004050 con 0x555d8649b9f0 2024-04-02T14:43:13.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.373+0000 7f3b31600640 1 -- 172.21.15.90:0/1672733155 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8594+0+0 (secure 0 0 0) 0x7f3b2c0041f0 con 0x555d8649b9f0 2024-04-02T14:43:13.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.373+0000 7f3b4d464740 1 -- 172.21.15.90:0/1672733155 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3b340ff030 con 0x555d8649b9f0 2024-04-02T14:43:13.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.373+0000 7f3b31600640 1 -- 172.21.15.90:0/1672733155 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f3b2c0043f0 con 0x555d8649b9f0 2024-04-02T14:43:13.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.373+0000 7f3b4d464740 10 client.27925 fetch_fsmap learned FSMap version 27 2024-04-02T14:43:13.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.373+0000 7f3b4d464740 1 -- 172.21.15.90:0/1672733155 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x555d864b2bf0 con 0x555d8649b9f0 2024-04-02T14:43:13.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.373+0000 7f3b31600640 1 -- 172.21.15.90:0/1672733155 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3b340ff030 con 0x555d8649b9f0 2024-04-02T14:43:13.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.374+0000 7f3b31600640 1 -- 172.21.15.90:0/1672733155 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f3b2c003bc0 con 0x555d8649b9f0 2024-04-02T14:43:13.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.374+0000 7f3b4d464740 10 client.27925 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:43:13.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.374+0000 7f3b4d464740 10 client.27925 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:43:13.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.374+0000 7f3b4d464740 20 client.27925 populate_metadata read hostname 'smithi090' 2024-04-02T14:43:13.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.374+0000 7f3b4d464740 1 --2- 172.21.15.90:0/1672733155 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x555d864b40b0 0x555d864b6540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:13.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.374+0000 7f3b4d464740 4 client.27925 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:43:13.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.374+0000 7f3b4d464740 1 -- 172.21.15.90:0/1672733155 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555d863bf040 con 0x555d864b40b0 2024-04-02T14:43:13.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.374+0000 7f3b43400640 1 --2- 172.21.15.90:0/1672733155 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x555d864b40b0 0x555d864b6540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:13.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.375+0000 7f3b43400640 1 --2- 172.21.15.90:0/1672733155 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x555d864b40b0 0x555d864b6540 crc :-1 s=READY pgs=228 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ff45c8737fc2e3de server_cookie=94d4d2416d1eafa4 in_seq=0 out_seq=0 2024-04-02T14:43:13.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.375+0000 7f3b31600640 10 client.27925 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:43:13.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.375+0000 7f3b31600640 1 -- 172.21.15.90:0/1672733155 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x555d863bf040 con 0x555d864b40b0 2024-04-02T14:43:13.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.375+0000 7f3b31600640 10 client.27925 handle_command_reply: tid=0 2024-04-02T14:43:13.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.376+0000 7f3b4d464740 1 client.27925 shutdown 2024-04-02T14:43:13.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.376+0000 7f3b4d464740 1 -- 172.21.15.90:0/1672733155 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555d8649b9f0 msgr2=0x555d864a1530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:13.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.376+0000 7f3b4d464740 1 --2- 172.21.15.90:0/1672733155 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555d8649b9f0 0x555d864a1530 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f3b2c012700 tx=0x7f3b2c012bc0 comp rx=0 tx=0).stop 2024-04-02T14:43:13.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.376+0000 7f3b4d464740 1 -- 172.21.15.90:0/1672733155 shutdown_connections 2024-04-02T14:43:13.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.376+0000 7f3b4d464740 1 --2- 172.21.15.90:0/1672733155 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x555d864b40b0 0x555d864b6540 crc :-1 s=READY pgs=228 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:13.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.376+0000 7f3b4d464740 1 --2- 172.21.15.90:0/1672733155 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555d864a4080 0x555d864a1a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:13.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.376+0000 7f3b4d464740 1 --2- 172.21.15.90:0/1672733155 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555d8649b9f0 0x555d864a1530 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:13.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.376+0000 7f3b4d464740 1 --2- 172.21.15.90:0/1672733155 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555d8649ad10 0x555d864a0ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:13.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.376+0000 7f3b4d464740 1 -- 172.21.15.90:0/1672733155 >> 172.21.15.90:0/1672733155 conn(0x555d86461a50 msgr2=0x555d864974b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:43:13.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.376+0000 7f3b31600640 0 client.27925 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:43:13.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.376+0000 7f3b4d464740 1 -- 172.21.15.90:0/1672733155 shutdown_connections 2024-04-02T14:43:13.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.376+0000 7f3b4d464740 1 -- 172.21.15.90:0/1672733155 wait complete. 2024-04-02T14:43:13.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.376+0000 7f3b4d464740 20 client.27925 trim_cache size 0 max 16384 2024-04-02T14:43:13.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.393+0000 7f3b4d464740 1 Processor -- start 2024-04-02T14:43:13.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.393+0000 7f3b4d464740 1 -- start start 2024-04-02T14:43:13.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.394+0000 7f3b4d464740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555d8649ad10 0x555d864a3720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:13.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.394+0000 7f3b4d464740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555d8649b9f0 0x555d864af870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:13.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.394+0000 7f3b4d464740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555d864a4080 0x555d864bbe50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:13.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.394+0000 7f3b4d464740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f3b441afc00 con 0x555d8649b9f0 2024-04-02T14:43:13.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.394+0000 7f3b4d464740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f3b441b1740 con 0x555d8649ad10 2024-04-02T14:43:13.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.394+0000 7f3b4d464740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f3b441b02d0 con 0x555d864a4080 2024-04-02T14:43:13.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.394+0000 7f3b43400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555d8649ad10 0x555d864a3720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:13.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.394+0000 7f3b3be00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555d8649b9f0 0x555d864af870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:13.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.394+0000 7f3b43400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555d8649ad10 0x555d864a3720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:33720/0 (socket says 172.21.15.90:33720) 2024-04-02T14:43:13.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.394+0000 7f3b43400640 1 -- 172.21.15.90:0/3810675661 learned_addr learned my addr 172.21.15.90:0/3810675661 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:43:13.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.394+0000 7f3b43e00640 1 --2- 172.21.15.90:0/3810675661 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555d864a4080 0x555d864bbe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:13.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.394+0000 7f3b43400640 1 -- 172.21.15.90:0/3810675661 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555d864a4080 msgr2=0x555d864bbe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:43:13.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.394+0000 7f3b43400640 1 --2- 172.21.15.90:0/3810675661 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555d864a4080 0x555d864bbe50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:13.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.394+0000 7f3b43400640 1 -- 172.21.15.90:0/3810675661 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555d8649b9f0 msgr2=0x555d864af870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:43:13.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.395+0000 7f3b43400640 1 --2- 172.21.15.90:0/3810675661 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555d8649b9f0 0x555d864af870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:13.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.395+0000 7f3b43400640 1 -- 172.21.15.90:0/3810675661 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555d8649fff0 con 0x555d8649ad10 2024-04-02T14:43:13.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.395+0000 7f3b43400640 1 --2- 172.21.15.90:0/3810675661 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555d8649ad10 0x555d864a3720 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f3b340bdd90 tx=0x7f3b340037d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=41f697f7c10cde4d server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:43:13.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.395+0000 7f3b33e00640 1 -- 172.21.15.90:0/3810675661 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b3404b070 con 0x555d8649ad10 2024-04-02T14:43:13.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.395+0000 7f3b33e00640 1 -- 172.21.15.90:0/3810675661 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3b340ff050 con 0x555d8649ad10 2024-04-02T14:43:13.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.395+0000 7f3b33e00640 1 -- 172.21.15.90:0/3810675661 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b340bd810 con 0x555d8649ad10 2024-04-02T14:43:13.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.396+0000 7f3b4d464740 1 -- 172.21.15.90:0/3810675661 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555d8649ad10 msgr2=0x555d864a3720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:13.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.396+0000 7f3b4d464740 1 --2- 172.21.15.90:0/3810675661 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555d8649ad10 0x555d864a3720 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f3b340bdd90 tx=0x7f3b340037d0 comp rx=0 tx=0).stop 2024-04-02T14:43:13.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.396+0000 7f3b4d464740 1 -- 172.21.15.90:0/3810675661 shutdown_connections 2024-04-02T14:43:13.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.396+0000 7f3b4d464740 1 --2- 172.21.15.90:0/3810675661 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555d864a4080 0x555d864bbe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:13.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.396+0000 7f3b4d464740 1 --2- 172.21.15.90:0/3810675661 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555d8649b9f0 0x555d864af870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:13.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.396+0000 7f3b4d464740 1 --2- 172.21.15.90:0/3810675661 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555d8649ad10 0x555d864a3720 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:13.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.396+0000 7f3b4d464740 1 -- 172.21.15.90:0/3810675661 >> 172.21.15.90:0/3810675661 conn(0x555d86461a50 msgr2=0x555d8649f8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:43:13.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.396+0000 7f3b4d464740 1 -- 172.21.15.90:0/3810675661 shutdown_connections 2024-04-02T14:43:13.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.396+0000 7f3b4d464740 1 -- 172.21.15.90:0/3810675661 wait complete. 2024-04-02T14:43:13.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.397+0000 7f3b4d464740 1 -- start start 2024-04-02T14:43:13.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.397+0000 7f3b4d464740 1 Processor -- start 2024-04-02T14:43:13.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.397+0000 7f3b4d464740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555d8649ad10 0x555d864ac1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:13.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.397+0000 7f3b4d464740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555d8649b9f0 0x555d8649e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:13.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.397+0000 7f3b4d464740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555d864a4080 0x555d8649ea60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:13.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.397+0000 7f3b4d464740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f3b441b1070 con 0x555d864a4080 2024-04-02T14:43:13.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.397+0000 7f3b4d464740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f3b441b09a0 con 0x555d8649ad10 2024-04-02T14:43:13.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.397+0000 7f3b4d464740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f3b441adc00 con 0x555d8649b9f0 2024-04-02T14:43:13.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.397+0000 7f3b4d464740 10 client.0 _ll_register_callbacks cb 0x7f3b2c009d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:43:13.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.397+0000 7f3b43400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555d8649ad10 0x555d864ac1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:13.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.397+0000 7f3b43400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555d8649ad10 0x555d864ac1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:33732/0 (socket says 172.21.15.90:33732) 2024-04-02T14:43:13.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.397+0000 7f3b43400640 1 -- 172.21.15.90:0/2885917915 learned_addr learned my addr 172.21.15.90:0/2885917915 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:43:13.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.397+0000 7f3b3be00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555d8649b9f0 0x555d8649e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:13.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.397+0000 7f3b43e00640 1 --2- 172.21.15.90:0/2885917915 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555d864a4080 0x555d8649ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:13.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.398+0000 7f3b43400640 1 -- 172.21.15.90:0/2885917915 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555d8649b9f0 msgr2=0x555d8649e4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:43:13.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.398+0000 7f3b43400640 1 --2- 172.21.15.90:0/2885917915 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555d8649b9f0 0x555d8649e4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:13.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.398+0000 7f3b43400640 1 -- 172.21.15.90:0/2885917915 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555d864a4080 msgr2=0x555d8649ea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:43:13.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.398+0000 7f3b43400640 1 --2- 172.21.15.90:0/2885917915 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555d864a4080 0x555d8649ea60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:13.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.398+0000 7f3b43400640 1 -- 172.21.15.90:0/2885917915 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555d864a4d60 con 0x555d8649ad10 2024-04-02T14:43:13.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.398+0000 7f3b43e00640 1 --2- 172.21.15.90:0/2885917915 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555d864a4080 0x555d8649ea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:43:13.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.398+0000 7f3b43400640 1 --2- 172.21.15.90:0/2885917915 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555d8649ad10 0x555d864ac1d0 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f3b340f5710 tx=0x7f3b34003f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2e5bab774f3e69e5 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:43:13.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.398+0000 7f3b31600640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:43:13.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.398+0000 7f3b31600640 1 -- 172.21.15.90:0/2885917915 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b3404b050 con 0x555d8649ad10 2024-04-02T14:43:13.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.398+0000 7f3b31600640 1 -- 172.21.15.90:0/2885917915 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8594+0+0 (secure 0 0 0) 0x7f3b340bc130 con 0x555d8649ad10 2024-04-02T14:43:13.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.398+0000 7f3b4d464740 1 -- 172.21.15.90:0/2885917915 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3b340ff050 con 0x555d8649ad10 2024-04-02T14:43:13.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.398+0000 7f3b31600640 1 -- 172.21.15.90:0/2885917915 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f3b340bd9f0 con 0x555d8649ad10 2024-04-02T14:43:13.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.398+0000 7f3b4d464740 10 client.28094 fetch_fsmap learned FSMap version 27 2024-04-02T14:43:13.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.398+0000 7f3b4d464740 1 -- 172.21.15.90:0/2885917915 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x555d8651aa00 con 0x555d8649ad10 2024-04-02T14:43:13.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.399+0000 7f3b31600640 1 -- 172.21.15.90:0/2885917915 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3b340ff050 con 0x555d8649ad10 2024-04-02T14:43:13.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.399+0000 7f3b31600640 1 -- 172.21.15.90:0/2885917915 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f3b3407e040 con 0x555d8649ad10 2024-04-02T14:43:13.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.399+0000 7f3b4d464740 10 client.28094 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:43:13.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.399+0000 7f3b4d464740 10 client.28094 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:43:13.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.399+0000 7f3b4d464740 20 client.28094 populate_metadata read hostname 'smithi090' 2024-04-02T14:43:13.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.399+0000 7f3b4d464740 1 --2- 172.21.15.90:0/2885917915 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x555d8651ad40 0x555d8651d130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:13.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.399+0000 7f3b4d464740 4 client.28094 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-02T14:43:13.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.399+0000 7f3b4d464740 1 -- 172.21.15.90:0/2885917915 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "damage ls"}) -- 0x555d8640cc30 con 0x555d8651ad40 2024-04-02T14:43:13.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.399+0000 7f3b3be00640 1 --2- 172.21.15.90:0/2885917915 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x555d8651ad40 0x555d8651d130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:13.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.400+0000 7f3b3be00640 1 --2- 172.21.15.90:0/2885917915 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x555d8651ad40 0x555d8651d130 crc :-1 s=READY pgs=229 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=693b490541825ea1 server_cookie=a08d2075a06a3457 in_seq=0 out_seq=0 2024-04-02T14:43:13.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.400+0000 7f3b31600640 10 client.28094 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:43:13.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.400+0000 7f3b31600640 1 -- 172.21.15.90:0/2885917915 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x555d8640cc30 con 0x555d8651ad40 2024-04-02T14:43:13.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.400+0000 7f3b31600640 10 client.28094 handle_command_reply: tid=0 2024-04-02T14:43:13.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.400+0000 7f3b4d464740 1 client.28094 shutdown 2024-04-02T14:43:13.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.400+0000 7f3b4d464740 1 -- 172.21.15.90:0/2885917915 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555d8649ad10 msgr2=0x555d864ac1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:13.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.400+0000 7f3b4d464740 1 --2- 172.21.15.90:0/2885917915 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555d8649ad10 0x555d864ac1d0 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f3b340f5710 tx=0x7f3b34003f90 comp rx=0 tx=0).stop 2024-04-02T14:43:13.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.400+0000 7f3b4d464740 1 -- 172.21.15.90:0/2885917915 shutdown_connections 2024-04-02T14:43:13.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.400+0000 7f3b4d464740 1 --2- 172.21.15.90:0/2885917915 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x555d8651ad40 0x555d8651d130 crc :-1 s=READY pgs=229 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:13.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.400+0000 7f3b4d464740 1 --2- 172.21.15.90:0/2885917915 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555d864a4080 0x555d8649ea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:13.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.400+0000 7f3b4d464740 1 --2- 172.21.15.90:0/2885917915 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555d8649b9f0 0x555d8649e4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:13.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.400+0000 7f3b4d464740 1 --2- 172.21.15.90:0/2885917915 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555d8649ad10 0x555d864ac1d0 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:13.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.400+0000 7f3b4d464740 1 -- 172.21.15.90:0/2885917915 >> 172.21.15.90:0/2885917915 conn(0x555d86461a50 msgr2=0x555d864a0f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:43:13.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.400+0000 7f3b31600640 0 client.28094 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:43:13.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.401+0000 7f3b4d464740 1 -- 172.21.15.90:0/2885917915 shutdown_connections 2024-04-02T14:43:13.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.401+0000 7f3b4d464740 1 -- 172.21.15.90:0/2885917915 wait complete. 2024-04-02T14:43:13.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.401+0000 7f3b4d464740 20 client.28094 trim_cache size 0 max 16384 2024-04-02T14:43:13.403 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-02T14:43:13.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.401+0000 7f3b4a800640 1 -- 172.21.15.90:0/3805102841 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f3b280a3d20 msgr2=0x7f3b280a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:13.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.401+0000 7f3b4a800640 1 --2- 172.21.15.90:0/3805102841 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f3b280a3d20 0x7f3b280a61e0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f3b3c005fd0 tx=0x7f3b3c009500 comp rx=0 tx=0).stop 2024-04-02T14:43:13.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.401+0000 7f3b4a800640 1 -- 172.21.15.90:0/3805102841 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3b44103f10 msgr2=0x7f3b4419a5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:13.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.401+0000 7f3b4a800640 1 --2- 172.21.15.90:0/3805102841 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3b44103f10 0x7f3b4419a5f0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f3b34005ab0 tx=0x7f3b340042d0 comp rx=0 tx=0).stop 2024-04-02T14:43:13.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.402+0000 7f3b4a800640 1 -- 172.21.15.90:0/3805102841 shutdown_connections 2024-04-02T14:43:13.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.402+0000 7f3b4a800640 1 --2- 172.21.15.90:0/3805102841 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f3b280a3d20 0x7f3b280a61e0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:13.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.402+0000 7f3b4a800640 1 --2- 172.21.15.90:0/3805102841 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3b4419eeb0 0x7f3b4419f360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:13.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.402+0000 7f3b4a800640 1 --2- 172.21.15.90:0/3805102841 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3b44108fe0 0x7f3b4419ab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:13.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.402+0000 7f3b4a800640 1 --2- 172.21.15.90:0/3805102841 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3b44103f10 0x7f3b4419a5f0 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:13.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.402+0000 7f3b4a800640 1 -- 172.21.15.90:0/3805102841 >> 172.21.15.90:0/3805102841 conn(0x7f3b440ff7b0 msgr2=0x7f3b44101ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:43:13.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.402+0000 7f3b4a800640 1 -- 172.21.15.90:0/3805102841 shutdown_connections 2024-04-02T14:43:13.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:13.402+0000 7f3b4a800640 1 -- 172.21.15.90:0/3805102841 wait complete. 2024-04-02T14:43:13.432 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:43:13 smithi177 ceph-mon[36431]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T14:43:14.357 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:43:14 smithi092 ceph-mon[36497]: pgmap v256: 97 pgs: 40 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 35 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 6.6 MiB/s wr, 342 op/s 2024-04-02T14:43:14.418 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-02T14:43:14.418 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T14:43:14.419 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-02T14:43:14.432 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:43:14 smithi177 ceph-mon[36431]: pgmap v256: 97 pgs: 40 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 35 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 6.6 MiB/s wr, 342 op/s 2024-04-02T14:43:14.434 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:43:14 smithi090 ceph-mon[31674]: pgmap v256: 97 pgs: 40 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 35 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 6.6 MiB/s wr, 342 op/s 2024-04-02T14:43:14.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.636+0000 7f97c1c00640 1 -- 172.21.15.90:0/844993663 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f97bc108310 msgr2=0x7f97bc1086f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:14.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.636+0000 7f97c1c00640 1 --2- 172.21.15.90:0/844993663 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f97bc108310 0x7f97bc1086f0 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f97ac009900 tx=0x7f97ac02bc40 comp rx=0 tx=0).stop 2024-04-02T14:43:14.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.636+0000 7f97c1c00640 1 -- 172.21.15.90:0/844993663 shutdown_connections 2024-04-02T14:43:14.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.636+0000 7f97c1c00640 1 --2- 172.21.15.90:0/844993663 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f97bc10d3e0 0x7f97bc114fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:14.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.636+0000 7f97c1c00640 1 --2- 172.21.15.90:0/844993663 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f97bc108cc0 0x7f97bc10cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:14.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.636+0000 7f97c1c00640 1 --2- 172.21.15.90:0/844993663 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f97bc108310 0x7f97bc1086f0 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:14.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.636+0000 7f97c1c00640 1 -- 172.21.15.90:0/844993663 >> 172.21.15.90:0/844993663 conn(0x7f97bc076040 msgr2=0x7f97bc078460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:43:14.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.636+0000 7f97c1c00640 1 -- 172.21.15.90:0/844993663 shutdown_connections 2024-04-02T14:43:14.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.636+0000 7f97c1c00640 1 -- 172.21.15.90:0/844993663 wait complete. 2024-04-02T14:43:14.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.637+0000 7f97c1c00640 1 Processor -- start 2024-04-02T14:43:14.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.637+0000 7f97c1c00640 1 -- start start 2024-04-02T14:43:14.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.638+0000 7f97c1c00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f97bc108310 0x7f97bc1a4680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:14.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.638+0000 7f97c1c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f97bc10d3e0 0x7f97bc1a4bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:14.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.638+0000 7f97c1c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f97bc19e7f0 0x7f97bc19eca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:14.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.638+0000 7f97c1c00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f97bc118470 con 0x7f97bc19e7f0 2024-04-02T14:43:14.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.638+0000 7f97c1c00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f97bc1182f0 con 0x7f97bc108310 2024-04-02T14:43:14.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.638+0000 7f97c1c00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f97bc1185f0 con 0x7f97bc10d3e0 2024-04-02T14:43:14.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.638+0000 7f97baa00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f97bc108310 0x7f97bc1a4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:14.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.638+0000 7f97baa00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f97bc108310 0x7f97bc1a4680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:36044/0 (socket says 172.21.15.90:36044) 2024-04-02T14:43:14.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.638+0000 7f97ba000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f97bc10d3e0 0x7f97bc1a4bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:14.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.638+0000 7f97baa00640 1 -- 172.21.15.90:0/3114376126 learned_addr learned my addr 172.21.15.90:0/3114376126 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:43:14.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.638+0000 7f97bb400640 1 --2- 172.21.15.90:0/3114376126 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f97bc19e7f0 0x7f97bc19eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:14.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.638+0000 7f97baa00640 1 -- 172.21.15.90:0/3114376126 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f97bc10d3e0 msgr2=0x7f97bc1a4bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:14.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.638+0000 7f97baa00640 1 --2- 172.21.15.90:0/3114376126 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f97bc10d3e0 0x7f97bc1a4bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:14.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.638+0000 7f97baa00640 1 -- 172.21.15.90:0/3114376126 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f97bc19e7f0 msgr2=0x7f97bc19eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:14.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.639+0000 7f97baa00640 1 --2- 172.21.15.90:0/3114376126 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f97bc19e7f0 0x7f97bc19eca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:14.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.639+0000 7f97baa00640 1 -- 172.21.15.90:0/3114376126 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97bc19f380 con 0x7f97bc108310 2024-04-02T14:43:14.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.639+0000 7f97bb400640 1 --2- 172.21.15.90:0/3114376126 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f97bc19e7f0 0x7f97bc19eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:43:14.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.639+0000 7f97baa00640 1 --2- 172.21.15.90:0/3114376126 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f97bc108310 0x7f97bc1a4680 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f97ac006010 tx=0x7f97ac02cb40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:43:14.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.640+0000 7f97b3400640 1 -- 172.21.15.90:0/3114376126 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97ac041070 con 0x7f97bc108310 2024-04-02T14:43:14.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.640+0000 7f97b3400640 1 -- 172.21.15.90:0/3114376126 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f97ac004450 con 0x7f97bc108310 2024-04-02T14:43:14.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.640+0000 7f97b3400640 1 -- 172.21.15.90:0/3114376126 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97ac003e30 con 0x7f97bc108310 2024-04-02T14:43:14.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.640+0000 7f97c1c00640 1 -- 172.21.15.90:0/3114376126 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97bc19f610 con 0x7f97bc108310 2024-04-02T14:43:14.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.640+0000 7f97c1c00640 1 -- 172.21.15.90:0/3114376126 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97bc1ab7c0 con 0x7f97bc108310 2024-04-02T14:43:14.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.645+0000 7f97b3400640 1 -- 172.21.15.90:0/3114376126 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f97ac044070 con 0x7f97bc108310 2024-04-02T14:43:14.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.647+0000 7f97b3400640 1 --2- 172.21.15.90:0/3114376126 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f97a00a3e70 0x7f97a00a6330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:14.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.647+0000 7f97b3400640 1 -- 172.21.15.90:0/3114376126 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8594+0+0 (secure 0 0 0) 0x7f97ac0f2450 con 0x7f97bc108310 2024-04-02T14:43:14.646 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.648+0000 7f97ba000640 1 --2- 172.21.15.90:0/3114376126 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f97a00a3e70 0x7f97a00a6330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:14.646 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.648+0000 7f97c4818740 1 Processor -- start 2024-04-02T14:43:14.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.649+0000 7f97c4818740 1 -- start start 2024-04-02T14:43:14.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.649+0000 7f97ba000640 1 --2- 172.21.15.90:0/3114376126 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f97a00a3e70 0x7f97a00a6330 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f97bc107440 tx=0x7f97a8005f90 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:43:14.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.649+0000 7f97c4818740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564fb9665030 0x564fb9663d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:14.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.649+0000 7f97c4818740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564fb9664350 0x564fb96647d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:14.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.649+0000 7f97c4818740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564fb966d6c0 0x564fb966fab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:14.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.649+0000 7f97c4818740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x564fb95c8310 con 0x564fb9664350 2024-04-02T14:43:14.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.649+0000 7f97c4818740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x564fb9570690 con 0x564fb966d6c0 2024-04-02T14:43:14.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.649+0000 7f97c4818740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x564fb95c42c0 con 0x564fb9665030 2024-04-02T14:43:14.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.649+0000 7f97ba000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564fb9664350 0x564fb96647d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:14.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.650+0000 7f97ba000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564fb9664350 0x564fb96647d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:60412/0 (socket says 172.21.15.90:60412) 2024-04-02T14:43:14.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.650+0000 7f97ba000640 1 -- 172.21.15.90:0/3812885553 learned_addr learned my addr 172.21.15.90:0/3812885553 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:43:14.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.650+0000 7f97bb400640 1 --2- 172.21.15.90:0/3812885553 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564fb966d6c0 0x564fb966fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:14.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.650+0000 7f97baa00640 1 --2- 172.21.15.90:0/3812885553 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564fb9665030 0x564fb9663d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:14.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.650+0000 7f97ba000640 1 -- 172.21.15.90:0/3812885553 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564fb9665030 msgr2=0x564fb9663d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:43:14.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.650+0000 7f97ba000640 1 --2- 172.21.15.90:0/3812885553 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564fb9665030 0x564fb9663d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:14.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.650+0000 7f97ba000640 1 -- 172.21.15.90:0/3812885553 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564fb966d6c0 msgr2=0x564fb966fab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:43:14.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.650+0000 7f97ba000640 1 --2- 172.21.15.90:0/3812885553 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564fb966d6c0 0x564fb966fab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:14.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.650+0000 7f97ba000640 1 -- 172.21.15.90:0/3812885553 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564fb93a47e0 con 0x564fb9664350 2024-04-02T14:43:14.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.650+0000 7f97bb400640 1 --2- 172.21.15.90:0/3812885553 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564fb966d6c0 0x564fb966fab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:43:14.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.651+0000 7f97ba000640 1 --2- 172.21.15.90:0/3812885553 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564fb9664350 0x564fb96647d0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f97a800e600 tx=0x7f97a8014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f11770332f34512d server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:43:14.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.651+0000 7f97a7e00640 1 -- 172.21.15.90:0/3812885553 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97a8004020 con 0x564fb9664350 2024-04-02T14:43:14.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.651+0000 7f97a7e00640 1 -- 172.21.15.90:0/3812885553 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f97a80041c0 con 0x564fb9664350 2024-04-02T14:43:14.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.651+0000 7f97a7e00640 1 -- 172.21.15.90:0/3812885553 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97a801c760 con 0x564fb9664350 2024-04-02T14:43:14.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.652+0000 7f97c4818740 1 -- 172.21.15.90:0/3812885553 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564fb9664350 msgr2=0x564fb96647d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:14.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.652+0000 7f97c4818740 1 --2- 172.21.15.90:0/3812885553 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564fb9664350 0x564fb96647d0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f97a800e600 tx=0x7f97a8014000 comp rx=0 tx=0).stop 2024-04-02T14:43:14.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.652+0000 7f97c4818740 1 -- 172.21.15.90:0/3812885553 shutdown_connections 2024-04-02T14:43:14.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.652+0000 7f97c4818740 1 --2- 172.21.15.90:0/3812885553 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564fb966d6c0 0x564fb966fab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:14.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.652+0000 7f97c4818740 1 --2- 172.21.15.90:0/3812885553 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564fb9664350 0x564fb96647d0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:14.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.652+0000 7f97c4818740 1 --2- 172.21.15.90:0/3812885553 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564fb9665030 0x564fb9663d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:14.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.652+0000 7f97c4818740 1 -- 172.21.15.90:0/3812885553 >> 172.21.15.90:0/3812885553 conn(0x564fb95f8270 msgr2=0x564fb965eda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:43:14.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.652+0000 7f97c4818740 1 -- 172.21.15.90:0/3812885553 shutdown_connections 2024-04-02T14:43:14.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.652+0000 7f97c4818740 1 -- 172.21.15.90:0/3812885553 wait complete. 2024-04-02T14:43:14.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.653+0000 7f97c4818740 1 -- start start 2024-04-02T14:43:14.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.653+0000 7f97c4818740 1 Processor -- start 2024-04-02T14:43:14.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.654+0000 7f97c4818740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564fb9665030 0x564fb966a750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:14.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.654+0000 7f97c4818740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564fb966d6c0 0x564fb966ac90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:14.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.654+0000 7f97c4818740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564fb966b1d0 0x564fb9679a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:14.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.654+0000 7f97c4818740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x564fb9670530 con 0x564fb9665030 2024-04-02T14:43:14.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.654+0000 7f97c4818740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x564fb9583a60 con 0x564fb966d6c0 2024-04-02T14:43:14.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.654+0000 7f97c4818740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x564fb96706b0 con 0x564fb966b1d0 2024-04-02T14:43:14.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.654+0000 7f97bb400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564fb966b1d0 0x564fb9679a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:14.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.654+0000 7f97ba000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564fb966d6c0 0x564fb966ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:14.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.654+0000 7f97ba000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564fb966d6c0 0x564fb966ac90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:36058/0 (socket says 172.21.15.90:36058) 2024-04-02T14:43:14.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.654+0000 7f97bb400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564fb966b1d0 0x564fb9679a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:58460/0 (socket says 172.21.15.90:58460) 2024-04-02T14:43:14.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.654+0000 7f97ba000640 1 -- 172.21.15.90:0/1074893895 learned_addr learned my addr 172.21.15.90:0/1074893895 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:43:14.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.654+0000 7f97c4818740 10 client.0 _ll_register_callbacks cb 0x564fb9589030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:43:14.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.654+0000 7f97baa00640 1 --2- 172.21.15.90:0/1074893895 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564fb9665030 0x564fb966a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:14.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.655+0000 7f97ba000640 1 -- 172.21.15.90:0/1074893895 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564fb966b1d0 msgr2=0x564fb9679a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:43:14.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.655+0000 7f97ba000640 1 --2- 172.21.15.90:0/1074893895 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564fb966b1d0 0x564fb9679a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:14.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.655+0000 7f97ba000640 1 -- 172.21.15.90:0/1074893895 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564fb9665030 msgr2=0x564fb966a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:43:14.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.655+0000 7f97ba000640 1 --2- 172.21.15.90:0/1074893895 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564fb9665030 0x564fb966a750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:14.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.655+0000 7f97ba000640 1 -- 172.21.15.90:0/1074893895 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564fb9404f70 con 0x564fb966d6c0 2024-04-02T14:43:14.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.656+0000 7f97ba000640 1 --2- 172.21.15.90:0/1074893895 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564fb966d6c0 0x564fb966ac90 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x564fb9665580 tx=0x7f97a8003d30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2ad85b751957fe11 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:43:14.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.656+0000 7f97a4c00640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:43:14.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.656+0000 7f97a4c00640 1 -- 172.21.15.90:0/1074893895 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97a801c4a0 con 0x564fb966d6c0 2024-04-02T14:43:14.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.656+0000 7f97c4818740 1 -- 172.21.15.90:0/1074893895 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f97a80041c0 con 0x564fb966d6c0 2024-04-02T14:43:14.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.656+0000 7f97a4c00640 1 -- 172.21.15.90:0/1074893895 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8594+0+0 (secure 0 0 0) 0x7f97a8004370 con 0x564fb966d6c0 2024-04-02T14:43:14.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.657+0000 7f97a4c00640 1 -- 172.21.15.90:0/1074893895 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f97a804d050 con 0x564fb966d6c0 2024-04-02T14:43:14.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.657+0000 7f97c4818740 10 client.28106 fetch_fsmap learned FSMap version 27 2024-04-02T14:43:14.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.657+0000 7f97c4818740 1 -- 172.21.15.90:0/1074893895 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x564fb929c170 con 0x564fb966d6c0 2024-04-02T14:43:14.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.657+0000 7f97a4c00640 1 -- 172.21.15.90:0/1074893895 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f97a80041c0 con 0x564fb966d6c0 2024-04-02T14:43:14.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.657+0000 7f97a4c00640 1 -- 172.21.15.90:0/1074893895 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f97a8048030 con 0x564fb966d6c0 2024-04-02T14:43:14.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.657+0000 7f97c4818740 10 client.28106 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:43:14.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.657+0000 7f97c4818740 10 client.28106 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:43:14.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.657+0000 7f97c4818740 20 client.28106 populate_metadata read hostname 'smithi090' 2024-04-02T14:43:14.657 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.657+0000 7f97c4818740 1 --2- 172.21.15.90:0/1074893895 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x564fb967df70 0x564fb9680360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:14.657 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.657+0000 7f97c4818740 4 client.28106 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:43:14.657 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.657+0000 7f97c4818740 1 -- 172.21.15.90:0/1074893895 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564fb95d5850 con 0x564fb967df70 2024-04-02T14:43:14.657 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.658+0000 7f97baa00640 1 --2- 172.21.15.90:0/1074893895 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x564fb967df70 0x564fb9680360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:14.657 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.658+0000 7f97baa00640 1 --2- 172.21.15.90:0/1074893895 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x564fb967df70 0x564fb9680360 crc :-1 s=READY pgs=230 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5c72d37d7b6bc034 server_cookie=e99ce3e4cffb8f50 in_seq=0 out_seq=0 2024-04-02T14:43:14.657 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.658+0000 7f97a4c00640 10 client.28106 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:43:14.658 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.659+0000 7f97a4c00640 1 -- 172.21.15.90:0/1074893895 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x564fb95d5850 con 0x564fb967df70 2024-04-02T14:43:14.658 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.659+0000 7f97a4c00640 10 client.28106 handle_command_reply: tid=0 2024-04-02T14:43:14.658 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.659+0000 7f97c4818740 1 client.28106 shutdown 2024-04-02T14:43:14.658 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.659+0000 7f97c4818740 1 -- 172.21.15.90:0/1074893895 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564fb966d6c0 msgr2=0x564fb966ac90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:14.658 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.659+0000 7f97c4818740 1 --2- 172.21.15.90:0/1074893895 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564fb966d6c0 0x564fb966ac90 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x564fb9665580 tx=0x7f97a8003d30 comp rx=0 tx=0).stop 2024-04-02T14:43:14.658 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.660+0000 7f97c4818740 1 -- 172.21.15.90:0/1074893895 shutdown_connections 2024-04-02T14:43:14.658 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.660+0000 7f97c4818740 1 --2- 172.21.15.90:0/1074893895 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x564fb967df70 0x564fb9680360 crc :-1 s=READY pgs=230 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:14.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.660+0000 7f97c4818740 1 --2- 172.21.15.90:0/1074893895 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564fb966b1d0 0x564fb9679a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:14.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.660+0000 7f97a4c00640 0 client.28106 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:43:14.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.660+0000 7f97c4818740 1 --2- 172.21.15.90:0/1074893895 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564fb966d6c0 0x564fb966ac90 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:14.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.660+0000 7f97c4818740 1 --2- 172.21.15.90:0/1074893895 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564fb9665030 0x564fb966a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:14.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.660+0000 7f97c4818740 1 -- 172.21.15.90:0/1074893895 >> 172.21.15.90:0/1074893895 conn(0x564fb95f8270 msgr2=0x564fb9660af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:43:14.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.660+0000 7f97c4818740 1 -- 172.21.15.90:0/1074893895 shutdown_connections 2024-04-02T14:43:14.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.660+0000 7f97c4818740 1 -- 172.21.15.90:0/1074893895 wait complete. 2024-04-02T14:43:14.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.660+0000 7f97c4818740 20 client.28106 trim_cache size 0 max 16384 2024-04-02T14:43:14.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.688+0000 7f97c4818740 1 Processor -- start 2024-04-02T14:43:14.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.688+0000 7f97c4818740 1 -- start start 2024-04-02T14:43:14.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.688+0000 7f97c4818740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564fb9665030 0x564fb96696b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:14.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.688+0000 7f97c4818740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564fb966b1d0 0x564fb9685900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:14.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.688+0000 7f97c4818740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564fb966d6c0 0x564fb9685e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:14.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.688+0000 7f97c4818740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f97bc082800 con 0x564fb966d6c0 2024-04-02T14:43:14.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.688+0000 7f97c4818740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f97bc1ad220 con 0x564fb9665030 2024-04-02T14:43:14.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.688+0000 7f97c4818740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f97bc1abdb0 con 0x564fb966b1d0 2024-04-02T14:43:14.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.689+0000 7f97ba000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564fb966b1d0 0x564fb9685900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:14.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.689+0000 7f97baa00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564fb9665030 0x564fb96696b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:14.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.689+0000 7f97ba000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564fb966b1d0 0x564fb9685900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:58466/0 (socket says 172.21.15.90:58466) 2024-04-02T14:43:14.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.689+0000 7f97ba000640 1 -- 172.21.15.90:0/3414785155 learned_addr learned my addr 172.21.15.90:0/3414785155 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:43:14.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.689+0000 7f97bb400640 1 --2- 172.21.15.90:0/3414785155 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564fb966d6c0 0x564fb9685e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:14.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.689+0000 7f97ba000640 1 -- 172.21.15.90:0/3414785155 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564fb9665030 msgr2=0x564fb96696b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:43:14.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.689+0000 7f97ba000640 1 --2- 172.21.15.90:0/3414785155 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564fb9665030 0x564fb96696b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:14.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.689+0000 7f97ba000640 1 -- 172.21.15.90:0/3414785155 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564fb966d6c0 msgr2=0x564fb9685e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:43:14.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.689+0000 7f97ba000640 1 --2- 172.21.15.90:0/3414785155 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564fb966d6c0 0x564fb9685e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:14.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.689+0000 7f97ba000640 1 -- 172.21.15.90:0/3414785155 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564fb9663380 con 0x564fb966b1d0 2024-04-02T14:43:14.690 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.690+0000 7f97ba000640 1 --2- 172.21.15.90:0/3414785155 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564fb966b1d0 0x564fb9685900 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f97a8000c00 tx=0x7f97a801c430 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1e25662ada17ea01 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:43:14.690 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.690+0000 7f97a7400640 1 -- 172.21.15.90:0/3414785155 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97a801c4a0 con 0x564fb966b1d0 2024-04-02T14:43:14.690 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.690+0000 7f97a7400640 1 -- 172.21.15.90:0/3414785155 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f97a8026480 con 0x564fb966b1d0 2024-04-02T14:43:14.690 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.690+0000 7f97a7400640 1 -- 172.21.15.90:0/3414785155 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97a80052d0 con 0x564fb966b1d0 2024-04-02T14:43:14.690 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.691+0000 7f97c4818740 1 -- 172.21.15.90:0/3414785155 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564fb966b1d0 msgr2=0x564fb9685900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:14.690 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.691+0000 7f97c4818740 1 --2- 172.21.15.90:0/3414785155 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564fb966b1d0 0x564fb9685900 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f97a8000c00 tx=0x7f97a801c430 comp rx=0 tx=0).stop 2024-04-02T14:43:14.690 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.691+0000 7f97c4818740 1 -- 172.21.15.90:0/3414785155 shutdown_connections 2024-04-02T14:43:14.690 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.691+0000 7f97c4818740 1 --2- 172.21.15.90:0/3414785155 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564fb966d6c0 0x564fb9685e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:14.690 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.691+0000 7f97c4818740 1 --2- 172.21.15.90:0/3414785155 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564fb966b1d0 0x564fb9685900 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:14.690 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.691+0000 7f97c4818740 1 --2- 172.21.15.90:0/3414785155 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564fb9665030 0x564fb96696b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:14.690 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.691+0000 7f97c4818740 1 -- 172.21.15.90:0/3414785155 >> 172.21.15.90:0/3414785155 conn(0x564fb95f8270 msgr2=0x564fb9660a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:43:14.691 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.691+0000 7f97c4818740 1 -- 172.21.15.90:0/3414785155 shutdown_connections 2024-04-02T14:43:14.691 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.691+0000 7f97c4818740 1 -- 172.21.15.90:0/3414785155 wait complete. 2024-04-02T14:43:14.691 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.691+0000 7f97c4818740 1 -- start start 2024-04-02T14:43:14.691 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.692+0000 7f97c4818740 1 Processor -- start 2024-04-02T14:43:14.694 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.692+0000 7f97c4818740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564fb9665030 0x564fb9675ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:14.694 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.692+0000 7f97c4818740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564fb966b1d0 0x564fb96669e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:14.694 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.692+0000 7f97c4818740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564fb966d6c0 0x564fb9666f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:14.694 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.692+0000 7f97c4818740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f97bc1acb50 con 0x564fb966b1d0 2024-04-02T14:43:14.694 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.692+0000 7f97c4818740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f97bc1ac480 con 0x564fb9665030 2024-04-02T14:43:14.694 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.692+0000 7f97c4818740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f97bc080800 con 0x564fb966d6c0 2024-04-02T14:43:14.694 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.692+0000 7f97c4818740 10 client.0 _ll_register_callbacks cb 0x7f97b40099c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:43:14.694 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.692+0000 7f97baa00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564fb9665030 0x564fb9675ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:14.694 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.692+0000 7f97bb400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564fb966d6c0 0x564fb9666f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:14.694 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.692+0000 7f97baa00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564fb9665030 0x564fb9675ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:36084/0 (socket says 172.21.15.90:36084) 2024-04-02T14:43:14.694 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.692+0000 7f97baa00640 1 -- 172.21.15.90:0/2107442153 learned_addr learned my addr 172.21.15.90:0/2107442153 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:43:14.695 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.692+0000 7f97ba000640 1 --2- 172.21.15.90:0/2107442153 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564fb966b1d0 0x564fb96669e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:14.695 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.693+0000 7f97baa00640 1 -- 172.21.15.90:0/2107442153 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564fb966d6c0 msgr2=0x564fb9666f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:43:14.695 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.693+0000 7f97baa00640 1 --2- 172.21.15.90:0/2107442153 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564fb966d6c0 0x564fb9666f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:14.695 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.693+0000 7f97baa00640 1 -- 172.21.15.90:0/2107442153 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564fb966b1d0 msgr2=0x564fb96669e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:43:14.695 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.693+0000 7f97baa00640 1 --2- 172.21.15.90:0/2107442153 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564fb966b1d0 0x564fb96669e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:14.695 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.693+0000 7f97baa00640 1 -- 172.21.15.90:0/2107442153 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564fb96675e0 con 0x564fb9665030 2024-04-02T14:43:14.695 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.693+0000 7f97baa00640 1 --2- 172.21.15.90:0/2107442153 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564fb9665030 0x564fb9675ca0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f97ac0efd10 tx=0x7f97ac0095e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1cb50d17f2097009 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:43:14.695 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.693+0000 7f97a4c00640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:43:14.695 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.693+0000 7f97a4c00640 1 -- 172.21.15.90:0/2107442153 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97ac04b040 con 0x564fb9665030 2024-04-02T14:43:14.695 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.693+0000 7f97a4c00640 1 -- 172.21.15.90:0/2107442153 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8594+0+0 (secure 0 0 0) 0x7f97ac0bc7b0 con 0x564fb9665030 2024-04-02T14:43:14.695 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.693+0000 7f97c4818740 1 -- 172.21.15.90:0/2107442153 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f97a8026480 con 0x564fb9665030 2024-04-02T14:43:14.695 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.694+0000 7f97a4c00640 1 -- 172.21.15.90:0/2107442153 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f97ac0be4b0 con 0x564fb9665030 2024-04-02T14:43:14.695 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.694+0000 7f97c4818740 10 client.28112 fetch_fsmap learned FSMap version 27 2024-04-02T14:43:14.695 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.694+0000 7f97c4818740 1 -- 172.21.15.90:0/2107442153 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x564fb96e5180 con 0x564fb9665030 2024-04-02T14:43:14.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.694+0000 7f97a4c00640 1 -- 172.21.15.90:0/2107442153 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f97a8026480 con 0x564fb9665030 2024-04-02T14:43:14.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.694+0000 7f97a4c00640 1 -- 172.21.15.90:0/2107442153 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f97ac0bab20 con 0x564fb9665030 2024-04-02T14:43:14.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.694+0000 7f97c4818740 10 client.28112 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:43:14.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.694+0000 7f97c4818740 10 client.28112 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:43:14.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.694+0000 7f97c4818740 20 client.28112 populate_metadata read hostname 'smithi090' 2024-04-02T14:43:14.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.694+0000 7f97c4818740 1 --2- 172.21.15.90:0/2107442153 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x564fb96e5790 0x564fb96e7b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:14.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.694+0000 7f97c4818740 4 client.28112 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-02T14:43:14.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.694+0000 7f97c4818740 1 -- 172.21.15.90:0/2107442153 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x564fb95cfba0 con 0x564fb96e5790 2024-04-02T14:43:14.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.694+0000 7f97ba000640 1 --2- 172.21.15.90:0/2107442153 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x564fb96e5790 0x564fb96e7b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:14.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.695+0000 7f97ba000640 1 --2- 172.21.15.90:0/2107442153 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x564fb96e5790 0x564fb96e7b80 crc :-1 s=READY pgs=231 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ef63cef8c5c706ab server_cookie=cd818e3cb33eb16 in_seq=0 out_seq=0 2024-04-02T14:43:14.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.695+0000 7f97a4c00640 10 client.28112 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:43:14.697 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.696+0000 7f97a4c00640 1 -- 172.21.15.90:0/2107442153 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x564fb95cfba0 con 0x564fb96e5790 2024-04-02T14:43:14.697 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.696+0000 7f97a4c00640 10 client.28112 handle_command_reply: tid=0 2024-04-02T14:43:14.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.696+0000 7f97c4818740 1 client.28112 shutdown 2024-04-02T14:43:14.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.696+0000 7f97c4818740 1 -- 172.21.15.90:0/2107442153 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564fb9665030 msgr2=0x564fb9675ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:14.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.696+0000 7f97c4818740 1 --2- 172.21.15.90:0/2107442153 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564fb9665030 0x564fb9675ca0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f97ac0efd10 tx=0x7f97ac0095e0 comp rx=0 tx=0).stop 2024-04-02T14:43:14.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.697+0000 7f97c4818740 1 -- 172.21.15.90:0/2107442153 shutdown_connections 2024-04-02T14:43:14.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.697+0000 7f97c4818740 1 --2- 172.21.15.90:0/2107442153 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x564fb96e5790 0x564fb96e7b80 crc :-1 s=READY pgs=231 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:14.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.697+0000 7f97c4818740 1 --2- 172.21.15.90:0/2107442153 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564fb966d6c0 0x564fb9666f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:14.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.697+0000 7f97c4818740 1 --2- 172.21.15.90:0/2107442153 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564fb966b1d0 0x564fb96669e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:14.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.697+0000 7f97c4818740 1 --2- 172.21.15.90:0/2107442153 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564fb9665030 0x564fb9675ca0 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:14.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.697+0000 7f97c4818740 1 -- 172.21.15.90:0/2107442153 >> 172.21.15.90:0/2107442153 conn(0x564fb95f8270 msgr2=0x564fb965d590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:43:14.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.697+0000 7f97a4c00640 0 client.28112 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:43:14.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.697+0000 7f97c4818740 1 -- 172.21.15.90:0/2107442153 shutdown_connections 2024-04-02T14:43:14.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.697+0000 7f97c4818740 1 -- 172.21.15.90:0/2107442153 wait complete. 2024-04-02T14:43:14.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.697+0000 7f97c4818740 20 client.28112 trim_cache size 0 max 16384 2024-04-02T14:43:14.699 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-02T14:43:14.699 INFO:teuthology.orchestra.run.smithi090.stdout: "return_code": 0, 2024-04-02T14:43:14.699 INFO:teuthology.orchestra.run.smithi090.stdout: "scrub_tag": "8784c6ef-15a6-4892-a821-d63c9edd4f88", 2024-04-02T14:43:14.699 INFO:teuthology.orchestra.run.smithi090.stdout: "mode": "asynchronous" 2024-04-02T14:43:14.699 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-02T14:43:14.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.698+0000 7f97c1c00640 1 -- 172.21.15.90:0/3114376126 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f97a00a3e70 msgr2=0x7f97a00a6330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:14.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.698+0000 7f97c1c00640 1 --2- 172.21.15.90:0/3114376126 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f97a00a3e70 0x7f97a00a6330 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f97bc107440 tx=0x7f97a8005f90 comp rx=0 tx=0).stop 2024-04-02T14:43:14.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.698+0000 7f97c1c00640 1 -- 172.21.15.90:0/3114376126 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f97bc108310 msgr2=0x7f97bc1a4680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:14.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.698+0000 7f97c1c00640 1 --2- 172.21.15.90:0/3114376126 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f97bc108310 0x7f97bc1a4680 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f97ac006010 tx=0x7f97ac02cb40 comp rx=0 tx=0).stop 2024-04-02T14:43:14.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.698+0000 7f97c1c00640 1 -- 172.21.15.90:0/3114376126 shutdown_connections 2024-04-02T14:43:14.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.698+0000 7f97c1c00640 1 --2- 172.21.15.90:0/3114376126 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f97a00a3e70 0x7f97a00a6330 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:14.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.698+0000 7f97c1c00640 1 --2- 172.21.15.90:0/3114376126 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f97bc19e7f0 0x7f97bc19eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:14.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.698+0000 7f97c1c00640 1 --2- 172.21.15.90:0/3114376126 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f97bc10d3e0 0x7f97bc1a4bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:14.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.698+0000 7f97c1c00640 1 --2- 172.21.15.90:0/3114376126 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f97bc108310 0x7f97bc1a4680 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:14.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.698+0000 7f97c1c00640 1 -- 172.21.15.90:0/3114376126 >> 172.21.15.90:0/3114376126 conn(0x7f97bc076040 msgr2=0x7f97bc077390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:43:14.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.699+0000 7f97c1c00640 1 -- 172.21.15.90:0/3114376126 shutdown_connections 2024-04-02T14:43:14.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.699+0000 7f97c1c00640 1 -- 172.21.15.90:0/3114376126 wait complete. 2024-04-02T14:43:14.715 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-02T14:43:14.715 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-02T14:43:14.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.976+0000 7f1dfe600640 1 -- 172.21.15.90:0/1149087526 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1df81036b0 msgr2=0x7f1df810bb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:14.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.976+0000 7f1dfe600640 1 --2- 172.21.15.90:0/1149087526 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1df81036b0 0x7f1df810bb90 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f1de4009900 tx=0x7f1de402bcd0 comp rx=0 tx=0).stop 2024-04-02T14:43:14.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.976+0000 7f1dfe600640 1 -- 172.21.15.90:0/1149087526 shutdown_connections 2024-04-02T14:43:14.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.976+0000 7f1dfe600640 1 --2- 172.21.15.90:0/1149087526 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1df810c0d0 0x7f1df810e540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:14.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.976+0000 7f1dfe600640 1 --2- 172.21.15.90:0/1149087526 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1df81036b0 0x7f1df810bb90 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:14.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.976+0000 7f1dfe600640 1 --2- 172.21.15.90:0/1149087526 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1df8102d90 0x7f1df8103170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:14.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.976+0000 7f1dfe600640 1 -- 172.21.15.90:0/1149087526 >> 172.21.15.90:0/1149087526 conn(0x7f1df8077b60 msgr2=0x7f1df8077f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:43:14.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.976+0000 7f1dfe600640 1 -- 172.21.15.90:0/1149087526 shutdown_connections 2024-04-02T14:43:14.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.976+0000 7f1dfe600640 1 -- 172.21.15.90:0/1149087526 wait complete. 2024-04-02T14:43:14.977 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.977+0000 7f1dfe600640 1 Processor -- start 2024-04-02T14:43:14.977 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.977+0000 7f1dfe600640 1 -- start start 2024-04-02T14:43:14.977 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.978+0000 7f1dfe600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1df8102d90 0x7f1df81a45d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:14.977 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.978+0000 7f1dfe600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1df81036b0 0x7f1df81a4b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:14.977 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.978+0000 7f1dfe600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1df810c0d0 0x7f1df819e690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:14.977 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.978+0000 7f1dfe600640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f1df8118510 con 0x7f1df8102d90 2024-04-02T14:43:14.977 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.978+0000 7f1dfe600640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f1df8118390 con 0x7f1df81036b0 2024-04-02T14:43:14.977 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.978+0000 7f1dfe600640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f1df8118690 con 0x7f1df810c0d0 2024-04-02T14:43:14.978 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.978+0000 7f1df7e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1df810c0d0 0x7f1df819e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:14.978 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.978+0000 7f1df7400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1df8102d90 0x7f1df81a45d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:14.978 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.978+0000 7f1df7e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1df810c0d0 0x7f1df819e690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:58506/0 (socket says 172.21.15.90:58506) 2024-04-02T14:43:14.978 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.978+0000 7f1df7e00640 1 -- 172.21.15.90:0/3870437646 learned_addr learned my addr 172.21.15.90:0/3870437646 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:43:14.978 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.979+0000 7f1df6a00640 1 --2- 172.21.15.90:0/3870437646 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1df81036b0 0x7f1df81a4b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:14.979 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.979+0000 7f1df7e00640 1 -- 172.21.15.90:0/3870437646 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1df81036b0 msgr2=0x7f1df81a4b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:14.979 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.979+0000 7f1df7e00640 1 --2- 172.21.15.90:0/3870437646 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1df81036b0 0x7f1df81a4b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:14.980 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.979+0000 7f1df7e00640 1 -- 172.21.15.90:0/3870437646 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1df8102d90 msgr2=0x7f1df81a45d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:14.980 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.979+0000 7f1df7e00640 1 --2- 172.21.15.90:0/3870437646 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1df8102d90 0x7f1df81a45d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:14.980 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.979+0000 7f1df7e00640 1 -- 172.21.15.90:0/3870437646 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1df819ef50 con 0x7f1df810c0d0 2024-04-02T14:43:14.980 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.979+0000 7f1df6a00640 1 --2- 172.21.15.90:0/3870437646 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1df81036b0 0x7f1df81a4b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:43:14.980 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.980+0000 7f1df7e00640 1 --2- 172.21.15.90:0/3870437646 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1df810c0d0 0x7f1df819e690 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f1df000cab0 tx=0x7f1df000ceb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:43:14.980 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.980+0000 7f1defe00640 1 -- 172.21.15.90:0/3870437646 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1df000ad60 con 0x7f1df810c0d0 2024-04-02T14:43:14.980 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.980+0000 7f1dfe600640 1 -- 172.21.15.90:0/3870437646 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1df819f240 con 0x7f1df810c0d0 2024-04-02T14:43:14.980 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.980+0000 7f1defe00640 1 -- 172.21.15.90:0/3870437646 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1df000f930 con 0x7f1df810c0d0 2024-04-02T14:43:14.980 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.980+0000 7f1defe00640 1 -- 172.21.15.90:0/3870437646 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1df0018a60 con 0x7f1df810c0d0 2024-04-02T14:43:14.980 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.980+0000 7f1dfe600640 1 -- 172.21.15.90:0/3870437646 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1df81ab550 con 0x7f1df810c0d0 2024-04-02T14:43:14.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.985+0000 7f1defe00640 1 -- 172.21.15.90:0/3870437646 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f1df000fae0 con 0x7f1df810c0d0 2024-04-02T14:43:14.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.986+0000 7f1e0123a740 1 Processor -- start 2024-04-02T14:43:14.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.987+0000 7f1e0123a740 1 -- start start 2024-04-02T14:43:14.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.987+0000 7f1e0123a740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5574b97b97b0 0x5574b97b9b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:14.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.987+0000 7f1e0123a740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5574b97ba0d0 0x5574b97c2770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:14.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.987+0000 7f1e0123a740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5574b97c2cb0 0x5574b97c50a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:14.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.987+0000 7f1e0123a740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5574b971d770 con 0x5574b97b97b0 2024-04-02T14:43:14.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.987+0000 7f1e0123a740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x5574b96c5700 con 0x5574b97c2cb0 2024-04-02T14:43:14.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.987+0000 7f1e0123a740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5574b9719720 con 0x5574b97ba0d0 2024-04-02T14:43:14.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.987+0000 7f1df7400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5574b97ba0d0 0x5574b97c2770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:14.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.987+0000 7f1df6a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5574b97b97b0 0x5574b97b9b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:14.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.987+0000 7f1df7400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5574b97ba0d0 0x5574b97c2770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:58522/0 (socket says 172.21.15.90:58522) 2024-04-02T14:43:14.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.987+0000 7f1df7400640 1 -- 172.21.15.90:0/2992269831 learned_addr learned my addr 172.21.15.90:0/2992269831 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:43:14.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.987+0000 7f1defe00640 1 --2- 172.21.15.90:0/3870437646 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f1ddc0a3d80 0x7f1ddc0a6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:14.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.987+0000 7f1df6a00640 1 --2- 172.21.15.90:0/2992269831 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5574b97c2cb0 0x5574b97c50a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:14.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.988+0000 7f1df7400640 1 -- 172.21.15.90:0/2992269831 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5574b97c2cb0 msgr2=0x5574b97c50a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:43:14.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.988+0000 7f1df7400640 1 --2- 172.21.15.90:0/2992269831 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5574b97c2cb0 0x5574b97c50a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:14.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.988+0000 7f1df7400640 1 -- 172.21.15.90:0/2992269831 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5574b97b97b0 msgr2=0x5574b97b9b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:43:14.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.988+0000 7f1df7400640 1 --2- 172.21.15.90:0/2992269831 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5574b97b97b0 0x5574b97b9b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:14.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.988+0000 7f1df7400640 1 -- 172.21.15.90:0/2992269831 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5574b94fb7d0 con 0x5574b97ba0d0 2024-04-02T14:43:14.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.988+0000 7f1defe00640 1 -- 172.21.15.90:0/3870437646 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8594+0+0 (secure 0 0 0) 0x7f1df00d0f30 con 0x7f1df810c0d0 2024-04-02T14:43:14.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.988+0000 7f1df7400640 1 --2- 172.21.15.90:0/2992269831 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5574b97ba0d0 0x5574b97c2770 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f1de800ad00 tx=0x7f1de800f790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=21d558ad740ccf02 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:43:14.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.988+0000 7f1df7e00640 1 --2- 172.21.15.90:0/3870437646 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f1ddc0a3d80 0x7f1ddc0a6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:14.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.988+0000 7f1decc00640 1 -- 172.21.15.90:0/2992269831 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1de801d070 con 0x5574b97ba0d0 2024-04-02T14:43:14.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.988+0000 7f1decc00640 1 -- 172.21.15.90:0/2992269831 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1de8018b20 con 0x5574b97ba0d0 2024-04-02T14:43:14.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.988+0000 7f1decc00640 1 -- 172.21.15.90:0/2992269831 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1de8010bc0 con 0x5574b97ba0d0 2024-04-02T14:43:14.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.990+0000 7f1e0123a740 1 -- 172.21.15.90:0/2992269831 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5574b97ba0d0 msgr2=0x5574b97c2770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:14.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.990+0000 7f1e0123a740 1 --2- 172.21.15.90:0/2992269831 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5574b97ba0d0 0x5574b97c2770 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f1de800ad00 tx=0x7f1de800f790 comp rx=0 tx=0).stop 2024-04-02T14:43:14.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.990+0000 7f1df7e00640 1 --2- 172.21.15.90:0/3870437646 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f1ddc0a3d80 0x7f1ddc0a6240 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f1df009cb80 tx=0x7f1df000a7d0 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:43:14.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.990+0000 7f1e0123a740 1 -- 172.21.15.90:0/2992269831 shutdown_connections 2024-04-02T14:43:14.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.990+0000 7f1e0123a740 1 --2- 172.21.15.90:0/2992269831 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5574b97c2cb0 0x5574b97c50a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:14.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.990+0000 7f1e0123a740 1 --2- 172.21.15.90:0/2992269831 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5574b97ba0d0 0x5574b97c2770 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:14.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.990+0000 7f1e0123a740 1 --2- 172.21.15.90:0/2992269831 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5574b97b97b0 0x5574b97b9b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:14.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.990+0000 7f1e0123a740 1 -- 172.21.15.90:0/2992269831 >> 172.21.15.90:0/2992269831 conn(0x5574b97b3ee0 msgr2=0x5574b97b42c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:43:14.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.990+0000 7f1e0123a740 1 -- 172.21.15.90:0/2992269831 shutdown_connections 2024-04-02T14:43:14.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.990+0000 7f1e0123a740 1 -- 172.21.15.90:0/2992269831 wait complete. 2024-04-02T14:43:14.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.990+0000 7f1e0123a740 1 -- start start 2024-04-02T14:43:14.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.990+0000 7f1e0123a740 1 Processor -- start 2024-04-02T14:43:14.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.991+0000 7f1e0123a740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5574b97b97b0 0x5574b97bfd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:14.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.991+0000 7f1e0123a740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5574b97ba0d0 0x5574b97c02d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:14.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.991+0000 7f1e0123a740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5574b97c2cb0 0x5574b97c0810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:14.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.991+0000 7f1e0123a740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5574b97c5aa0 con 0x5574b97ba0d0 2024-04-02T14:43:14.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.991+0000 7f1e0123a740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x5574b96d8ac0 con 0x5574b97b97b0 2024-04-02T14:43:14.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.991+0000 7f1e0123a740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5574b97c5c20 con 0x5574b97c2cb0 2024-04-02T14:43:14.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.991+0000 7f1e0123a740 10 client.0 _ll_register_callbacks cb 0x5574b96db180 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:43:14.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.992+0000 7f1df6a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5574b97b97b0 0x5574b97bfd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:14.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.992+0000 7f1df6a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5574b97b97b0 0x5574b97bfd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:36128/0 (socket says 172.21.15.90:36128) 2024-04-02T14:43:14.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.992+0000 7f1df6a00640 1 -- 172.21.15.90:0/2509888950 learned_addr learned my addr 172.21.15.90:0/2509888950 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:43:14.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.992+0000 7f1df6a00640 1 --2- 172.21.15.90:0/2509888950 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5574b97c2cb0 0x5574b97c0810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:14.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.992+0000 7f1df7400640 1 --2- 172.21.15.90:0/2509888950 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5574b97ba0d0 0x5574b97c02d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:14.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.992+0000 7f1df6a00640 1 -- 172.21.15.90:0/2509888950 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5574b97c2cb0 msgr2=0x5574b97c0810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:43:14.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.992+0000 7f1df6a00640 1 --2- 172.21.15.90:0/2509888950 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5574b97c2cb0 0x5574b97c0810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:14.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.992+0000 7f1df6a00640 1 -- 172.21.15.90:0/2509888950 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5574b97ba0d0 msgr2=0x5574b97c02d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:43:14.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.992+0000 7f1df6a00640 1 --2- 172.21.15.90:0/2509888950 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5574b97ba0d0 0x5574b97c02d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:14.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.992+0000 7f1df6a00640 1 -- 172.21.15.90:0/2509888950 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5574b94313f0 con 0x5574b97b97b0 2024-04-02T14:43:14.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.992+0000 7f1df7400640 1 --2- 172.21.15.90:0/2509888950 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5574b97ba0d0 0x5574b97c02d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:43:14.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.992+0000 7f1df6a00640 1 --2- 172.21.15.90:0/2509888950 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5574b97b97b0 0x5574b97bfd90 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f1de4009230 tx=0x7f1de4000bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e5d8c42bdd0eb4f9 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:43:14.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.993+0000 7f1de1600640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:43:14.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.993+0000 7f1de1600640 1 -- 172.21.15.90:0/2509888950 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1de402cb20 con 0x5574b97b97b0 2024-04-02T14:43:14.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.993+0000 7f1e0123a740 1 -- 172.21.15.90:0/2509888950 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1de8018b20 con 0x5574b97b97b0 2024-04-02T14:43:14.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.993+0000 7f1de1600640 1 -- 172.21.15.90:0/2509888950 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8594+0+0 (secure 0 0 0) 0x7f1de402ccc0 con 0x5574b97b97b0 2024-04-02T14:43:14.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.993+0000 7f1de1600640 1 -- 172.21.15.90:0/2509888950 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f1de4055050 con 0x5574b97b97b0 2024-04-02T14:43:14.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.993+0000 7f1e0123a740 10 client.28121 fetch_fsmap learned FSMap version 27 2024-04-02T14:43:14.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.993+0000 7f1e0123a740 1 -- 172.21.15.90:0/2509888950 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5574b93f10f0 con 0x5574b97b97b0 2024-04-02T14:43:14.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.994+0000 7f1de1600640 1 -- 172.21.15.90:0/2509888950 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1de8018b20 con 0x5574b97b97b0 2024-04-02T14:43:14.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.994+0000 7f1de1600640 1 -- 172.21.15.90:0/2509888950 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f1de402e450 con 0x5574b97b97b0 2024-04-02T14:43:14.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.994+0000 7f1e0123a740 10 client.28121 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:43:14.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.994+0000 7f1e0123a740 10 client.28121 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:43:14.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.994+0000 7f1e0123a740 20 client.28121 populate_metadata read hostname 'smithi090' 2024-04-02T14:43:14.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.994+0000 7f1e0123a740 1 --2- 172.21.15.90:0/2509888950 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5574b97d2d00 0x5574b97d50f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:14.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.994+0000 7f1e0123a740 4 client.28121 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:43:14.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.994+0000 7f1e0123a740 1 -- 172.21.15.90:0/2509888950 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5574b96be630 con 0x5574b97d2d00 2024-04-02T14:43:14.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.995+0000 7f1df7400640 1 --2- 172.21.15.90:0/2509888950 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5574b97d2d00 0x5574b97d50f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:14.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.996+0000 7f1df7400640 1 --2- 172.21.15.90:0/2509888950 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5574b97d2d00 0x5574b97d50f0 crc :-1 s=READY pgs=232 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fe43c69bae3d140b server_cookie=a896ca3565128e16 in_seq=0 out_seq=0 2024-04-02T14:43:14.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.996+0000 7f1de1600640 10 client.28121 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:43:14.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.997+0000 7f1de1600640 1 -- 172.21.15.90:0/2509888950 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5574b96be630 con 0x5574b97d2d00 2024-04-02T14:43:14.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.997+0000 7f1de1600640 10 client.28121 handle_command_reply: tid=0 2024-04-02T14:43:14.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.997+0000 7f1e0123a740 1 client.28121 shutdown 2024-04-02T14:43:14.997 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.997+0000 7f1e0123a740 1 -- 172.21.15.90:0/2509888950 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5574b97b97b0 msgr2=0x5574b97bfd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:14.997 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.997+0000 7f1e0123a740 1 --2- 172.21.15.90:0/2509888950 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5574b97b97b0 0x5574b97bfd90 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f1de4009230 tx=0x7f1de4000bc0 comp rx=0 tx=0).stop 2024-04-02T14:43:14.997 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.998+0000 7f1e0123a740 1 -- 172.21.15.90:0/2509888950 shutdown_connections 2024-04-02T14:43:14.997 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.998+0000 7f1e0123a740 1 --2- 172.21.15.90:0/2509888950 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5574b97d2d00 0x5574b97d50f0 crc :-1 s=READY pgs=232 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:14.998 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.998+0000 7f1e0123a740 1 --2- 172.21.15.90:0/2509888950 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5574b97c2cb0 0x5574b97c0810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:14.998 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.998+0000 7f1e0123a740 1 --2- 172.21.15.90:0/2509888950 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5574b97ba0d0 0x5574b97c02d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:14.998 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.998+0000 7f1de1600640 0 client.28121 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:43:14.998 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.998+0000 7f1e0123a740 1 --2- 172.21.15.90:0/2509888950 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5574b97b97b0 0x5574b97bfd90 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:14.998 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.998+0000 7f1e0123a740 1 -- 172.21.15.90:0/2509888950 >> 172.21.15.90:0/2509888950 conn(0x5574b97b3ee0 msgr2=0x5574b97b5c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:43:14.998 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.998+0000 7f1e0123a740 1 -- 172.21.15.90:0/2509888950 shutdown_connections 2024-04-02T14:43:14.998 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.998+0000 7f1e0123a740 1 -- 172.21.15.90:0/2509888950 wait complete. 2024-04-02T14:43:14.998 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:14.998+0000 7f1e0123a740 20 client.28121 trim_cache size 0 max 16384 2024-04-02T14:43:15.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:15.023+0000 7f1e0123a740 1 Processor -- start 2024-04-02T14:43:15.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:15.023+0000 7f1e0123a740 1 -- start start 2024-04-02T14:43:15.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:15.023+0000 7f1e0123a740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5574b97b97b0 0x5574b97c24c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:15.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:15.023+0000 7f1e0123a740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5574b97ba0d0 0x5574b97becd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:15.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:15.023+0000 7f1e0123a740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5574b97c2cb0 0x5574b97ce960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:15.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:15.023+0000 7f1e0123a740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f1df81b3d90 con 0x5574b97c2cb0 2024-04-02T14:43:15.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:15.023+0000 7f1e0123a740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f1df81b58d0 con 0x5574b97b97b0 2024-04-02T14:43:15.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:15.023+0000 7f1e0123a740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f1df81b4460 con 0x5574b97ba0d0 2024-04-02T14:43:15.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:15.024+0000 7f1df6a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5574b97b97b0 0x5574b97c24c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:15.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:15.024+0000 7f1df7400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5574b97ba0d0 0x5574b97becd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:15.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:15.024+0000 7f1df7400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5574b97ba0d0 0x5574b97becd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:58530/0 (socket says 172.21.15.90:58530) 2024-04-02T14:43:15.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:15.024+0000 7f1df7400640 1 -- 172.21.15.90:0/239061246 learned_addr learned my addr 172.21.15.90:0/239061246 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:43:15.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:15.024+0000 7f1df6a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5574b97b97b0 0x5574b97c24c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:36130/0 (socket says 172.21.15.90:36130) 2024-04-02T14:43:15.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:15.024+0000 7f1df6a00640 1 --2- 172.21.15.90:0/239061246 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5574b97c2cb0 0x5574b97ce960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:15.025 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:15.024+0000 7f1df7400640 1 -- 172.21.15.90:0/239061246 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5574b97b97b0 msgr2=0x5574b97c24c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:43:15.025 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:15.024+0000 7f1df7400640 1 --2- 172.21.15.90:0/239061246 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5574b97b97b0 0x5574b97c24c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:15.025 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:15.024+0000 7f1df7400640 1 -- 172.21.15.90:0/239061246 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5574b97c2cb0 msgr2=0x5574b97ce960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:43:15.025 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:15.024+0000 7f1df7400640 1 --2- 172.21.15.90:0/239061246 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5574b97c2cb0 0x5574b97ce960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:15.025 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:15.024+0000 7f1df7400640 1 -- 172.21.15.90:0/239061246 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5574b97d5720 con 0x5574b97ba0d0 2024-04-02T14:43:15.025 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:15.025+0000 7f1df7400640 1 --2- 172.21.15.90:0/239061246 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5574b97ba0d0 0x5574b97becd0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x5574b97b9280 tx=0x7f1de8010430 comp rx=0 tx=0).ready entity=mon.1 client_cookie=454ec44b57386c5e server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:43:15.025 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:15.025+0000 7f1de3e00640 1 -- 172.21.15.90:0/239061246 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1de801d040 con 0x5574b97ba0d0 2024-04-02T14:43:15.025 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:15.025+0000 7f1de3e00640 1 -- 172.21.15.90:0/239061246 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1de802d070 con 0x5574b97ba0d0 2024-04-02T14:43:15.025 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:15.025+0000 7f1de3e00640 1 -- 172.21.15.90:0/239061246 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1de8005500 con 0x5574b97ba0d0 2024-04-02T14:43:15.025 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:15.026+0000 7f1e0123a740 1 -- 172.21.15.90:0/239061246 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5574b97ba0d0 msgr2=0x5574b97becd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:15.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:15.026+0000 7f1e0123a740 1 --2- 172.21.15.90:0/239061246 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5574b97ba0d0 0x5574b97becd0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x5574b97b9280 tx=0x7f1de8010430 comp rx=0 tx=0).stop 2024-04-02T14:43:15.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:15.026+0000 7f1e0123a740 1 -- 172.21.15.90:0/239061246 shutdown_connections 2024-04-02T14:43:15.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:15.026+0000 7f1e0123a740 1 --2- 172.21.15.90:0/239061246 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5574b97c2cb0 0x5574b97ce960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:15.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:15.026+0000 7f1e0123a740 1 --2- 172.21.15.90:0/239061246 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5574b97ba0d0 0x5574b97becd0 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:15.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:15.026+0000 7f1e0123a740 1 --2- 172.21.15.90:0/239061246 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5574b97b97b0 0x5574b97c24c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:15.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:15.026+0000 7f1e0123a740 1 -- 172.21.15.90:0/239061246 >> 172.21.15.90:0/239061246 conn(0x5574b97b3ee0 msgr2=0x5574b97b2f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:43:15.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:15.026+0000 7f1e0123a740 1 -- 172.21.15.90:0/239061246 shutdown_connections 2024-04-02T14:43:15.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:15.026+0000 7f1e0123a740 1 -- 172.21.15.90:0/239061246 wait complete. 2024-04-02T14:43:15.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:15.027+0000 7f1e0123a740 1 -- start start 2024-04-02T14:43:15.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:15.027+0000 7f1e0123a740 1 Processor -- start 2024-04-02T14:43:15.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:15.027+0000 7f1e0123a740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5574b97b97b0 0x5574b97c6b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:15.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:15.028+0000 7f1e0123a740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5574b97ba0d0 0x5574b97c70a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:15.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:15.028+0000 7f1e0123a740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5574b97c2cb0 0x5574b97c7600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:15.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:15.028+0000 7f1e0123a740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f1df81b5200 con 0x5574b97ba0d0 2024-04-02T14:43:15.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:15.028+0000 7f1e0123a740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f1df81b4b30 con 0x5574b97b97b0 2024-04-02T14:43:15.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:15.028+0000 7f1e0123a740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f1df81b1d90 con 0x5574b97c2cb0 2024-04-02T14:43:15.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:15.028+0000 7f1e0123a740 10 client.0 _ll_register_callbacks cb 0x7f1de402ea80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:43:15.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:15.028+0000 7f1df7400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5574b97ba0d0 0x5574b97c70a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:15.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:15.028+0000 7f1df6a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5574b97b97b0 0x5574b97c6b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:15.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:15.028+0000 7f1df6a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5574b97b97b0 0x5574b97c6b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:36132/0 (socket says 172.21.15.90:36132) 2024-04-02T14:43:15.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:15.028+0000 7f1df6a00640 1 -- 172.21.15.90:0/2458846586 learned_addr learned my addr 172.21.15.90:0/2458846586 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:43:15.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:15.028+0000 7f1df6a00640 1 --2- 172.21.15.90:0/2458846586 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5574b97c2cb0 0x5574b97c7600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:15.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:15.028+0000 7f1df6a00640 1 -- 172.21.15.90:0/2458846586 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5574b97c2cb0 msgr2=0x5574b97c7600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:43:15.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:15.028+0000 7f1df6a00640 1 --2- 172.21.15.90:0/2458846586 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5574b97c2cb0 0x5574b97c7600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:15.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:15.028+0000 7f1df6a00640 1 -- 172.21.15.90:0/2458846586 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5574b97ba0d0 msgr2=0x5574b97c70a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:43:15.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:15.028+0000 7f1df6a00640 1 --2- 172.21.15.90:0/2458846586 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5574b97ba0d0 0x5574b97c70a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:15.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:15.028+0000 7f1df6a00640 1 -- 172.21.15.90:0/2458846586 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5574b9838490 con 0x5574b97b97b0 2024-04-02T14:43:15.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:15.028+0000 7f1df7400640 1 --2- 172.21.15.90:0/2458846586 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5574b97ba0d0 0x5574b97c70a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:43:15.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:15.029+0000 7f1df6a00640 1 --2- 172.21.15.90:0/2458846586 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5574b97b97b0 0x5574b97c6b60 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f1de4005150 tx=0x7f1de4036040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dee94892cf95a8a2 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:43:15.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:15.029+0000 7f1de1600640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:43:15.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:15.029+0000 7f1de1600640 1 -- 172.21.15.90:0/2458846586 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1de402c670 con 0x5574b97b97b0 2024-04-02T14:43:15.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:15.029+0000 7f1e0123a740 1 -- 172.21.15.90:0/2458846586 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1de802d070 con 0x5574b97b97b0 2024-04-02T14:43:15.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:15.029+0000 7f1de1600640 1 -- 172.21.15.90:0/2458846586 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8594+0+0 (secure 0 0 0) 0x7f1de402c920 con 0x5574b97b97b0 2024-04-02T14:43:15.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:15.029+0000 7f1de1600640 1 -- 172.21.15.90:0/2458846586 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f1de402cb20 con 0x5574b97b97b0 2024-04-02T14:43:15.031 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:15.029+0000 7f1e0123a740 10 client.28127 fetch_fsmap learned FSMap version 27 2024-04-02T14:43:15.031 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:15.029+0000 7f1e0123a740 1 -- 172.21.15.90:0/2458846586 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5574b98397a0 con 0x5574b97b97b0 2024-04-02T14:43:15.031 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:15.029+0000 7f1de1600640 1 -- 172.21.15.90:0/2458846586 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1de802d070 con 0x5574b97b97b0 2024-04-02T14:43:15.031 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:15.030+0000 7f1de1600640 1 -- 172.21.15.90:0/2458846586 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f1de4061020 con 0x5574b97b97b0 2024-04-02T14:43:15.031 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:15.030+0000 7f1e0123a740 10 client.28127 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:43:15.031 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:15.030+0000 7f1e0123a740 10 client.28127 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:43:15.031 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:15.030+0000 7f1e0123a740 20 client.28127 populate_metadata read hostname 'smithi090' 2024-04-02T14:43:15.031 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:15.030+0000 7f1e0123a740 1 --2- 172.21.15.90:0/2458846586 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5574b9839da0 0x5574b983c190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:15.031 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:15.030+0000 7f1e0123a740 4 client.28127 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-02T14:43:15.031 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:15.030+0000 7f1e0123a740 1 -- 172.21.15.90:0/2458846586 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5574b972acb0 con 0x5574b9839da0 2024-04-02T14:43:15.031 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:15.031+0000 7f1df7400640 1 --2- 172.21.15.90:0/2458846586 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5574b9839da0 0x5574b983c190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:15.033 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:15.031+0000 7f1df7400640 1 --2- 172.21.15.90:0/2458846586 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5574b9839da0 0x5574b983c190 crc :-1 s=READY pgs=233 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cdaddf39ed80983b server_cookie=962d8fa239ae1210 in_seq=0 out_seq=0 2024-04-02T14:43:15.033 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:15.031+0000 7f1de1600640 10 client.28127 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:43:15.033 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:15.031+0000 7f1de1600640 1 -- 172.21.15.90:0/2458846586 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x5574b972acb0 con 0x5574b9839da0 2024-04-02T14:43:15.033 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:15.031+0000 7f1de1600640 10 client.28127 handle_command_reply: tid=0 2024-04-02T14:43:15.034 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:15.031+0000 7f1e0123a740 1 client.28127 shutdown 2024-04-02T14:43:15.034 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:15.032+0000 7f1e0123a740 1 -- 172.21.15.90:0/2458846586 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5574b97b97b0 msgr2=0x5574b97c6b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:15.034 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:15.032+0000 7f1e0123a740 1 --2- 172.21.15.90:0/2458846586 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5574b97b97b0 0x5574b97c6b60 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f1de4005150 tx=0x7f1de4036040 comp rx=0 tx=0).stop 2024-04-02T14:43:15.034 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:15.032+0000 7f1e0123a740 1 -- 172.21.15.90:0/2458846586 shutdown_connections 2024-04-02T14:43:15.034 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:15.032+0000 7f1e0123a740 1 --2- 172.21.15.90:0/2458846586 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5574b9839da0 0x5574b983c190 crc :-1 s=READY pgs=233 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:15.034 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:15.032+0000 7f1e0123a740 1 --2- 172.21.15.90:0/2458846586 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5574b97c2cb0 0x5574b97c7600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:15.034 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:15.032+0000 7f1e0123a740 1 --2- 172.21.15.90:0/2458846586 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5574b97ba0d0 0x5574b97c70a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:15.034 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:15.032+0000 7f1e0123a740 1 --2- 172.21.15.90:0/2458846586 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5574b97b97b0 0x5574b97c6b60 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:15.034 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:15.032+0000 7f1e0123a740 1 -- 172.21.15.90:0/2458846586 >> 172.21.15.90:0/2458846586 conn(0x5574b97b3ee0 msgr2=0x5574b97bfd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:43:15.034 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:15.032+0000 7f1de1600640 0 client.28127 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:43:15.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:15.032+0000 7f1e0123a740 1 -- 172.21.15.90:0/2458846586 shutdown_connections 2024-04-02T14:43:15.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:15.032+0000 7f1e0123a740 1 -- 172.21.15.90:0/2458846586 wait complete. 2024-04-02T14:43:15.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:15.032+0000 7f1e0123a740 20 client.28127 trim_cache size 0 max 16384 2024-04-02T14:43:15.035 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-02T14:43:15.035 INFO:teuthology.orchestra.run.smithi090.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-02T14:43:15.035 INFO:teuthology.orchestra.run.smithi090.stdout: "scrubs": { 2024-04-02T14:43:15.035 INFO:teuthology.orchestra.run.smithi090.stdout: "8784c6ef-15a6-4892-a821-d63c9edd4f88": { 2024-04-02T14:43:15.035 INFO:teuthology.orchestra.run.smithi090.stdout: "path": "/", 2024-04-02T14:43:15.036 INFO:teuthology.orchestra.run.smithi090.stdout: "tag": "8784c6ef-15a6-4892-a821-d63c9edd4f88", 2024-04-02T14:43:15.036 INFO:teuthology.orchestra.run.smithi090.stdout: "options": "recursive,force" 2024-04-02T14:43:15.036 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:43:15.036 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:43:15.036 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-02T14:43:15.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:15.033+0000 7f1dfe600640 1 -- 172.21.15.90:0/3870437646 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f1ddc0a3d80 msgr2=0x7f1ddc0a6240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:15.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:15.033+0000 7f1dfe600640 1 --2- 172.21.15.90:0/3870437646 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f1ddc0a3d80 0x7f1ddc0a6240 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f1df009cb80 tx=0x7f1df000a7d0 comp rx=0 tx=0).stop 2024-04-02T14:43:15.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:15.033+0000 7f1dfe600640 1 -- 172.21.15.90:0/3870437646 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1df810c0d0 msgr2=0x7f1df819e690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:15.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:15.033+0000 7f1dfe600640 1 --2- 172.21.15.90:0/3870437646 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1df810c0d0 0x7f1df819e690 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f1df000cab0 tx=0x7f1df000ceb0 comp rx=0 tx=0).stop 2024-04-02T14:43:15.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:15.033+0000 7f1dfe600640 1 -- 172.21.15.90:0/3870437646 shutdown_connections 2024-04-02T14:43:15.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:15.033+0000 7f1dfe600640 1 --2- 172.21.15.90:0/3870437646 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f1ddc0a3d80 0x7f1ddc0a6240 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:15.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:15.033+0000 7f1dfe600640 1 --2- 172.21.15.90:0/3870437646 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1df810c0d0 0x7f1df819e690 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:15.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:15.033+0000 7f1dfe600640 1 --2- 172.21.15.90:0/3870437646 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1df81036b0 0x7f1df81a4b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:15.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:15.033+0000 7f1dfe600640 1 --2- 172.21.15.90:0/3870437646 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1df8102d90 0x7f1df81a45d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:15.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:15.033+0000 7f1dfe600640 1 -- 172.21.15.90:0/3870437646 >> 172.21.15.90:0/3870437646 conn(0x7f1df8077b60 msgr2=0x7f1df80fd9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:43:15.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:15.033+0000 7f1dfe600640 1 -- 172.21.15.90:0/3870437646 shutdown_connections 2024-04-02T14:43:15.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:15.033+0000 7f1dfe600640 1 -- 172.21.15.90:0/3870437646 wait complete. 2024-04-02T14:43:15.051 INFO:tasks.cephfs.filesystem:scrub status for tag:8784c6ef-15a6-4892-a821-d63c9edd4f88 - {'path': '/', 'tag': '8784c6ef-15a6-4892-a821-d63c9edd4f88', 'options': 'recursive,force'} 2024-04-02T14:43:15.185 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:43:15 smithi090 ceph-mon[31674]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:43:15.185 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:43:15 smithi090 ceph-mon[31674]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T14:43:15.185 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:43:15 smithi090 ceph-mon[31674]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' 2024-04-02T14:43:15.185 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:43:15 smithi090 ceph-mon[31674]: scrub queued for path: / 2024-04-02T14:43:15.186 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:43:15 smithi090 ceph-mon[31674]: scrub summary: idle+waiting paths [/] 2024-04-02T14:43:15.186 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:43:15 smithi090 ceph-mon[31674]: scrub summary: active paths [/] 2024-04-02T14:43:15.357 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:43:15 smithi092 ceph-mon[36497]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:43:15.357 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:43:15 smithi092 ceph-mon[36497]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T14:43:15.357 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:43:15 smithi092 ceph-mon[36497]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' 2024-04-02T14:43:15.357 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:43:15 smithi092 ceph-mon[36497]: scrub queued for path: / 2024-04-02T14:43:15.358 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:43:15 smithi092 ceph-mon[36497]: scrub summary: idle+waiting paths [/] 2024-04-02T14:43:15.358 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:43:15 smithi092 ceph-mon[36497]: scrub summary: active paths [/] 2024-04-02T14:43:15.433 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:43:15 smithi177 ceph-mon[36431]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:43:15.433 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:43:15 smithi177 ceph-mon[36431]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T14:43:15.433 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:43:15 smithi177 ceph-mon[36431]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' 2024-04-02T14:43:15.433 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:43:15 smithi177 ceph-mon[36431]: scrub queued for path: / 2024-04-02T14:43:15.433 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:43:15 smithi177 ceph-mon[36431]: scrub summary: idle+waiting paths [/] 2024-04-02T14:43:15.433 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:43:15 smithi177 ceph-mon[36431]: scrub summary: active paths [/] 2024-04-02T14:43:16.357 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:43:16 smithi092 ceph-mon[36497]: pgmap v257: 97 pgs: 97 active+clean; 4.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 1.3 MiB/s wr, 171 op/s 2024-04-02T14:43:16.357 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:43:16 smithi092 ceph-mon[36497]: scrub summary: idle+waiting paths [/] 2024-04-02T14:43:16.432 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:43:16 smithi177 ceph-mon[36431]: pgmap v257: 97 pgs: 97 active+clean; 4.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 1.3 MiB/s wr, 171 op/s 2024-04-02T14:43:16.433 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:43:16 smithi177 ceph-mon[36431]: scrub summary: idle+waiting paths [/] 2024-04-02T14:43:16.434 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:43:16 smithi090 ceph-mon[31674]: pgmap v257: 97 pgs: 97 active+clean; 4.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 1.3 MiB/s wr, 171 op/s 2024-04-02T14:43:16.434 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:43:16 smithi090 ceph-mon[31674]: scrub summary: idle+waiting paths [/] 2024-04-02T14:43:16.828 INFO:teuthology.orchestra.run.smithi090.stderr:progress: 465.0 s, 206.0 tps, lat 464.117 ms stddev 258.406 2024-04-02T14:43:17.357 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:43:17 smithi092 ceph-mon[36497]: scrub summary: idle 2024-04-02T14:43:17.433 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:43:17 smithi177 ceph-mon[36431]: scrub summary: idle 2024-04-02T14:43:17.434 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:43:17 smithi090 ceph-mon[31674]: scrub summary: idle 2024-04-02T14:43:18.362 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:43:18 smithi092 ceph-mon[36497]: pgmap v258: 97 pgs: 97 active+clean; 4.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 1.3 MiB/s wr, 171 op/s 2024-04-02T14:43:18.433 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:43:18 smithi177 ceph-mon[36431]: pgmap v258: 97 pgs: 97 active+clean; 4.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 1.3 MiB/s wr, 171 op/s 2024-04-02T14:43:18.435 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:43:18 smithi090 ceph-mon[31674]: pgmap v258: 97 pgs: 97 active+clean; 4.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 1.3 MiB/s wr, 171 op/s 2024-04-02T14:43:20.357 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:43:20 smithi092 ceph-mon[36497]: pgmap v259: 97 pgs: 97 active+clean; 4.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 1.3 MiB/s wr, 260 op/s 2024-04-02T14:43:20.357 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:43:20 smithi092 ceph-mon[36497]: osdmap e95: 12 total, 12 up, 12 in 2024-04-02T14:43:20.432 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:43:20 smithi177 ceph-mon[36431]: pgmap v259: 97 pgs: 97 active+clean; 4.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 1.3 MiB/s wr, 260 op/s 2024-04-02T14:43:20.433 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:43:20 smithi177 ceph-mon[36431]: osdmap e95: 12 total, 12 up, 12 in 2024-04-02T14:43:20.434 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:43:20 smithi090 ceph-mon[31674]: pgmap v259: 97 pgs: 97 active+clean; 4.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 1.3 MiB/s wr, 260 op/s 2024-04-02T14:43:20.434 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:43:20 smithi090 ceph-mon[31674]: osdmap e95: 12 total, 12 up, 12 in 2024-04-02T14:43:21.828 INFO:teuthology.orchestra.run.smithi090.stderr:progress: 470.0 s, 673.4 tps, lat 162.626 ms stddev 324.496 2024-04-02T14:43:22.356 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:43:22 smithi092 ceph-mon[36497]: pgmap v261: 97 pgs: 97 active+clean; 4.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 143 KiB/s rd, 1.6 MiB/s wr, 314 op/s 2024-04-02T14:43:22.433 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:43:22 smithi177 ceph-mon[36431]: pgmap v261: 97 pgs: 97 active+clean; 4.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 143 KiB/s rd, 1.6 MiB/s wr, 314 op/s 2024-04-02T14:43:22.434 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:43:22 smithi090 ceph-mon[31674]: pgmap v261: 97 pgs: 97 active+clean; 4.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 143 KiB/s rd, 1.6 MiB/s wr, 314 op/s 2024-04-02T14:43:24.234 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.236+0000 7fee11e00640 1 -- 172.21.15.90:0/2124017774 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fee0c100c50 msgr2=0x7fee0c1010d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:24.234 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.236+0000 7fee11e00640 1 --2- 172.21.15.90:0/2124017774 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fee0c100c50 0x7fee0c1010d0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fedf4009860 tx=0x7fedf402bb70 comp rx=0 tx=0).stop 2024-04-02T14:43:24.234 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.236+0000 7fee11e00640 1 -- 172.21.15.90:0/2124017774 shutdown_connections 2024-04-02T14:43:24.235 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.236+0000 7fee11e00640 1 --2- 172.21.15.90:0/2124017774 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fee0c101610 0x7fee0c10d830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:24.235 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.236+0000 7fee11e00640 1 --2- 172.21.15.90:0/2124017774 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fee0c100c50 0x7fee0c1010d0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:24.235 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.236+0000 7fee11e00640 1 --2- 172.21.15.90:0/2124017774 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fee0c1059a0 0x7fee0c105d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:24.235 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.236+0000 7fee11e00640 1 -- 172.21.15.90:0/2124017774 >> 172.21.15.90:0/2124017774 conn(0x7fee0c077850 msgr2=0x7fee0c077c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:43:24.235 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.237+0000 7fee11e00640 1 -- 172.21.15.90:0/2124017774 shutdown_connections 2024-04-02T14:43:24.235 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.238+0000 7fee11e00640 1 -- 172.21.15.90:0/2124017774 wait complete. 2024-04-02T14:43:24.237 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.239+0000 7fee11e00640 1 Processor -- start 2024-04-02T14:43:24.237 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.239+0000 7fee11e00640 1 -- start start 2024-04-02T14:43:24.238 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.240+0000 7fee11e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fee0c100c50 0x7fee0c19a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:24.240 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.241+0000 7fee11e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fee0c101610 0x7fee0c19aae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:24.240 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.241+0000 7fee11e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fee0c1059a0 0x7fee0c19ee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:24.240 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.241+0000 7fee11e00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fee0c110e80 con 0x7fee0c100c50 2024-04-02T14:43:24.240 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.241+0000 7fee11e00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fee0c110d00 con 0x7fee0c101610 2024-04-02T14:43:24.240 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.241+0000 7fee11e00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fee0c111000 con 0x7fee0c1059a0 2024-04-02T14:43:24.240 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.241+0000 7fee03e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fee0c101610 0x7fee0c19aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:24.240 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.241+0000 7fee03e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fee0c101610 0x7fee0c19aae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:34832/0 (socket says 172.21.15.90:34832) 2024-04-02T14:43:24.240 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.241+0000 7fee03e00640 1 -- 172.21.15.90:0/1453452649 learned_addr learned my addr 172.21.15.90:0/1453452649 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:43:24.240 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.241+0000 7fee0b400640 1 --2- 172.21.15.90:0/1453452649 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fee0c1059a0 0x7fee0c19ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:24.240 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.241+0000 7fee0aa00640 1 --2- 172.21.15.90:0/1453452649 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fee0c100c50 0x7fee0c19a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:24.240 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.242+0000 7fee03e00640 1 -- 172.21.15.90:0/1453452649 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fee0c1059a0 msgr2=0x7fee0c19ee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:24.240 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.242+0000 7fee03e00640 1 --2- 172.21.15.90:0/1453452649 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fee0c1059a0 0x7fee0c19ee60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:24.241 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.242+0000 7fee03e00640 1 -- 172.21.15.90:0/1453452649 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fee0c100c50 msgr2=0x7fee0c19a5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:24.241 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.242+0000 7fee03e00640 1 --2- 172.21.15.90:0/1453452649 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fee0c100c50 0x7fee0c19a5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:24.241 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.242+0000 7fee03e00640 1 -- 172.21.15.90:0/1453452649 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee0c19f590 con 0x7fee0c101610 2024-04-02T14:43:24.242 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.242+0000 7fee03e00640 1 --2- 172.21.15.90:0/1453452649 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fee0c101610 0x7fee0c19aae0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7fedf4037040 tx=0x7fedf4032480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:43:24.242 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.243+0000 7fee10a00640 1 -- 172.21.15.90:0/1453452649 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedf4041020 con 0x7fee0c101610 2024-04-02T14:43:24.242 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.243+0000 7fee10a00640 1 -- 172.21.15.90:0/1453452649 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fedf402c9f0 con 0x7fee0c101610 2024-04-02T14:43:24.242 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.243+0000 7fee10a00640 1 -- 172.21.15.90:0/1453452649 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedf4039040 con 0x7fee0c101610 2024-04-02T14:43:24.242 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.243+0000 7fee11e00640 1 -- 172.21.15.90:0/1453452649 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee0c19f820 con 0x7fee0c101610 2024-04-02T14:43:24.243 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.243+0000 7fee11e00640 1 -- 172.21.15.90:0/1453452649 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee0c1a7110 con 0x7fee0c101610 2024-04-02T14:43:24.244 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.246+0000 7fee14baa740 1 Processor -- start 2024-04-02T14:43:24.245 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.247+0000 7fee14baa740 1 -- start start 2024-04-02T14:43:24.245 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.247+0000 7fee14baa740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b358b71cc0 0x55b358b720a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:24.245 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.247+0000 7fee10a00640 1 -- 172.21.15.90:0/1453452649 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fedf4044020 con 0x7fee0c101610 2024-04-02T14:43:24.245 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.247+0000 7fee14baa740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b358b72670 0x55b358b83680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:24.245 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.247+0000 7fee14baa740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b358b83bc0 0x55b358b85fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:24.246 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.247+0000 7fee14baa740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55b358ad6d70 con 0x55b358b71cc0 2024-04-02T14:43:24.246 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.247+0000 7fee14baa740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55b358a7f2a0 con 0x55b358b83bc0 2024-04-02T14:43:24.246 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.247+0000 7fee14baa740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55b358ad2d20 con 0x55b358b72670 2024-04-02T14:43:24.247 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.247+0000 7fee0aa00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b358b72670 0x55b358b83680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:24.247 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.248+0000 7fee0aa00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b358b72670 0x55b358b83680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:53274/0 (socket says 172.21.15.90:53274) 2024-04-02T14:43:24.248 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.248+0000 7fee0aa00640 1 -- 172.21.15.90:0/3317083847 learned_addr learned my addr 172.21.15.90:0/3317083847 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:43:24.248 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.248+0000 7fee03e00640 1 --2- 172.21.15.90:0/3317083847 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b358b83bc0 0x55b358b85fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:24.248 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.248+0000 7fee0b400640 1 --2- 172.21.15.90:0/3317083847 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b358b71cc0 0x55b358b720a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:24.248 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.248+0000 7fee03e00640 1 -- 172.21.15.90:0/3317083847 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b358b72670 msgr2=0x55b358b83680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:43:24.248 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.248+0000 7fee03e00640 1 --2- 172.21.15.90:0/3317083847 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b358b72670 0x55b358b83680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:24.248 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.248+0000 7fee03e00640 1 -- 172.21.15.90:0/3317083847 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b358b71cc0 msgr2=0x55b358b720a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:43:24.248 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.248+0000 7fee03e00640 1 --2- 172.21.15.90:0/3317083847 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b358b71cc0 0x55b358b720a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:24.249 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.248+0000 7fee03e00640 1 -- 172.21.15.90:0/3317083847 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b358a93840 con 0x55b358b83bc0 2024-04-02T14:43:24.249 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.249+0000 7fee0aa00640 1 --2- 172.21.15.90:0/3317083847 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b358b72670 0x55b358b83680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:43:24.249 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.249+0000 7fee0b400640 1 --2- 172.21.15.90:0/3317083847 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b358b71cc0 0x55b358b720a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:43:24.249 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.249+0000 7fee03e00640 1 --2- 172.21.15.90:0/3317083847 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b358b83bc0 0x55b358b85fb0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7fedf40ef210 tx=0x7fedf40f5a90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bbb7a9c0bef7a288 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:43:24.249 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.249+0000 7fedfbe00640 1 -- 172.21.15.90:0/3317083847 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedf4105070 con 0x55b358b83bc0 2024-04-02T14:43:24.249 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.249+0000 7fedfbe00640 1 -- 172.21.15.90:0/3317083847 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fedf40f1e50 con 0x55b358b83bc0 2024-04-02T14:43:24.249 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.249+0000 7fedfbe00640 1 -- 172.21.15.90:0/3317083847 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedf4100d20 con 0x55b358b83bc0 2024-04-02T14:43:24.249 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.249+0000 7fee10a00640 1 --2- 172.21.15.90:0/1453452649 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fedf00a3e40 0x7fedf00a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:24.249 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.250+0000 7fee10a00640 1 -- 172.21.15.90:0/1453452649 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8550+0+0 (secure 0 0 0) 0x7fedf40f1a60 con 0x7fee0c101610 2024-04-02T14:43:24.254 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.250+0000 7fee0b400640 1 --2- 172.21.15.90:0/1453452649 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fedf00a3e40 0x7fedf00a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:24.254 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.250+0000 7fee14baa740 1 -- 172.21.15.90:0/3317083847 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b358b83bc0 msgr2=0x55b358b85fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:24.254 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.250+0000 7fee14baa740 1 --2- 172.21.15.90:0/3317083847 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b358b83bc0 0x55b358b85fb0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7fedf40ef210 tx=0x7fedf40f5a90 comp rx=0 tx=0).stop 2024-04-02T14:43:24.254 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.250+0000 7fee14baa740 1 -- 172.21.15.90:0/3317083847 shutdown_connections 2024-04-02T14:43:24.254 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.250+0000 7fee14baa740 1 --2- 172.21.15.90:0/3317083847 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b358b83bc0 0x55b358b85fb0 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:24.254 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.250+0000 7fee14baa740 1 --2- 172.21.15.90:0/3317083847 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b358b72670 0x55b358b83680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:24.254 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.250+0000 7fee14baa740 1 --2- 172.21.15.90:0/3317083847 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b358b71cc0 0x55b358b720a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:24.254 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.250+0000 7fee14baa740 1 -- 172.21.15.90:0/3317083847 >> 172.21.15.90:0/3317083847 conn(0x55b358b6c5e0 msgr2=0x55b358b6c9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:43:24.254 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.251+0000 7fee14baa740 1 -- 172.21.15.90:0/3317083847 shutdown_connections 2024-04-02T14:43:24.254 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.251+0000 7fee0b400640 1 --2- 172.21.15.90:0/1453452649 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fedf00a3e40 0x7fedf00a6300 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fee04005fd0 tx=0x7fee04011040 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:43:24.255 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.251+0000 7fee14baa740 1 -- 172.21.15.90:0/3317083847 wait complete. 2024-04-02T14:43:24.255 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.251+0000 7fee14baa740 1 -- start start 2024-04-02T14:43:24.255 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.251+0000 7fee14baa740 1 Processor -- start 2024-04-02T14:43:24.255 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.252+0000 7fee14baa740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b358b71cc0 0x55b358b80c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:24.255 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.252+0000 7fee14baa740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b358b72670 0x55b358b81140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:24.255 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.252+0000 7fee14baa740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b358b83bc0 0x55b358b81680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:24.255 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.252+0000 7fee14baa740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55b358a7f2a0 con 0x55b358b72670 2024-04-02T14:43:24.255 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.252+0000 7fee14baa740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55b358a92660 con 0x55b358b71cc0 2024-04-02T14:43:24.255 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.252+0000 7fee14baa740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55b358b86740 con 0x55b358b83bc0 2024-04-02T14:43:24.255 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.252+0000 7fee14baa740 10 client.0 _ll_register_callbacks cb 0x55b358a92050 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:43:24.255 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.252+0000 7fee0aa00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b358b71cc0 0x55b358b80c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:24.255 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.252+0000 7fee0aa00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b358b71cc0 0x55b358b80c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:34856/0 (socket says 172.21.15.90:34856) 2024-04-02T14:43:24.255 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.252+0000 7fee0aa00640 1 -- 172.21.15.90:0/3935089606 learned_addr learned my addr 172.21.15.90:0/3935089606 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:43:24.255 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.252+0000 7fee0b400640 1 --2- 172.21.15.90:0/3935089606 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b358b83bc0 0x55b358b81680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:24.256 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.252+0000 7fee0aa00640 1 -- 172.21.15.90:0/3935089606 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b358b83bc0 msgr2=0x55b358b81680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:43:24.256 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.252+0000 7fee0aa00640 1 --2- 172.21.15.90:0/3935089606 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b358b83bc0 0x55b358b81680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:24.256 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.252+0000 7fee0aa00640 1 -- 172.21.15.90:0/3935089606 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b358b72670 msgr2=0x55b358b81140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:43:24.256 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.252+0000 7fee0aa00640 1 --2- 172.21.15.90:0/3935089606 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b358b72670 0x55b358b81140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:24.256 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.252+0000 7fee0aa00640 1 -- 172.21.15.90:0/3935089606 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b358aa1e80 con 0x55b358b71cc0 2024-04-02T14:43:24.256 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.253+0000 7fee0aa00640 1 --2- 172.21.15.90:0/3935089606 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b358b71cc0 0x55b358b80c00 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7fedfc00bac0 tx=0x7fedfc004600 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8a7650d51192b624 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:43:24.256 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.253+0000 7fedf8c00640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:43:24.256 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.253+0000 7fedf8c00640 1 -- 172.21.15.90:0/3935089606 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedfc011440 con 0x55b358b71cc0 2024-04-02T14:43:24.256 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.253+0000 7fedf8c00640 1 -- 172.21.15.90:0/3935089606 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8550+0+0 (secure 0 0 0) 0x7fedfc0115e0 con 0x55b358b71cc0 2024-04-02T14:43:24.256 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.253+0000 7fee14baa740 1 -- 172.21.15.90:0/3935089606 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fedf40f1e50 con 0x55b358b71cc0 2024-04-02T14:43:24.256 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.253+0000 7fedf8c00640 1 -- 172.21.15.90:0/3935089606 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fedfc0117e0 con 0x55b358b71cc0 2024-04-02T14:43:24.256 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.253+0000 7fee14baa740 10 client.28139 fetch_fsmap learned FSMap version 27 2024-04-02T14:43:24.256 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.253+0000 7fee14baa740 1 -- 172.21.15.90:0/3935089606 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55b3587eb210 con 0x55b358b71cc0 2024-04-02T14:43:24.256 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.254+0000 7fedf8c00640 1 -- 172.21.15.90:0/3935089606 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fedf40f1e50 con 0x55b358b71cc0 2024-04-02T14:43:24.256 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.254+0000 7fedf8c00640 1 -- 172.21.15.90:0/3935089606 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fedfc023a60 con 0x55b358b71cc0 2024-04-02T14:43:24.257 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.254+0000 7fee14baa740 10 client.28139 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:43:24.257 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.254+0000 7fee14baa740 10 client.28139 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-02T14:43:24.257 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.254+0000 7fee14baa740 20 client.28139 populate_metadata read hostname 'smithi090' 2024-04-02T14:43:24.257 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.254+0000 7fee14baa740 1 --2- 172.21.15.90:0/3935089606 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55b358b93ce0 0x55b358b96180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:24.257 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.254+0000 7fee14baa740 4 client.28139 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:43:24.257 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.254+0000 7fee14baa740 1 -- 172.21.15.90:0/3935089606 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b358a86cd0 con 0x55b358b93ce0 2024-04-02T14:43:24.257 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.254+0000 7fee03e00640 1 --2- 172.21.15.90:0/3935089606 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55b358b93ce0 0x55b358b96180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:24.258 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.256+0000 7fee03e00640 1 --2- 172.21.15.90:0/3935089606 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55b358b93ce0 0x55b358b96180 crc :-1 s=READY pgs=234 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ce69615d174c2019 server_cookie=356df029b3ada43f in_seq=0 out_seq=0 2024-04-02T14:43:24.258 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.256+0000 7fedf8c00640 10 client.28139 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:43:24.258 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.257+0000 7fedf8c00640 1 -- 172.21.15.90:0/3935089606 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b358a86cd0 con 0x55b358b93ce0 2024-04-02T14:43:24.258 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.257+0000 7fedf8c00640 10 client.28139 handle_command_reply: tid=0 2024-04-02T14:43:24.259 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.257+0000 7fee14baa740 1 client.28139 shutdown 2024-04-02T14:43:24.259 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.258+0000 7fee14baa740 1 -- 172.21.15.90:0/3935089606 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b358b71cc0 msgr2=0x55b358b80c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:24.259 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.258+0000 7fee14baa740 1 --2- 172.21.15.90:0/3935089606 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b358b71cc0 0x55b358b80c00 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7fedfc00bac0 tx=0x7fedfc004600 comp rx=0 tx=0).stop 2024-04-02T14:43:24.259 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.258+0000 7fee14baa740 1 -- 172.21.15.90:0/3935089606 shutdown_connections 2024-04-02T14:43:24.259 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.258+0000 7fee14baa740 1 --2- 172.21.15.90:0/3935089606 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55b358b93ce0 0x55b358b96180 crc :-1 s=READY pgs=234 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:24.259 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.258+0000 7fee14baa740 1 --2- 172.21.15.90:0/3935089606 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b358b83bc0 0x55b358b81680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:24.259 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.258+0000 7fee14baa740 1 --2- 172.21.15.90:0/3935089606 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b358b72670 0x55b358b81140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:24.259 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.258+0000 7fee14baa740 1 --2- 172.21.15.90:0/3935089606 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b358b71cc0 0x55b358b80c00 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:24.259 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.258+0000 7fedf8c00640 0 client.28139 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:43:24.259 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.258+0000 7fee14baa740 1 -- 172.21.15.90:0/3935089606 >> 172.21.15.90:0/3935089606 conn(0x55b358b6c5e0 msgr2=0x55b358b6e330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:43:24.259 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.258+0000 7fee14baa740 1 -- 172.21.15.90:0/3935089606 shutdown_connections 2024-04-02T14:43:24.259 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.258+0000 7fee14baa740 1 -- 172.21.15.90:0/3935089606 wait complete. 2024-04-02T14:43:24.259 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.258+0000 7fee14baa740 20 client.28139 trim_cache size 0 max 16384 2024-04-02T14:43:24.280 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.281+0000 7fee14baa740 1 Processor -- start 2024-04-02T14:43:24.280 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.281+0000 7fee14baa740 1 -- start start 2024-04-02T14:43:24.280 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.281+0000 7fee14baa740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b358b71cc0 0x55b358b83330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:24.280 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.281+0000 7fee14baa740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b358b72670 0x55b358b8f480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:24.280 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.281+0000 7fee14baa740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b358b83bc0 0x55b358b6e140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:24.280 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.281+0000 7fee14baa740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fee0c1ab5b0 con 0x55b358b83bc0 2024-04-02T14:43:24.280 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.281+0000 7fee14baa740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fee0c1ad0f0 con 0x55b358b71cc0 2024-04-02T14:43:24.280 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.281+0000 7fee14baa740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fee0c1abc80 con 0x55b358b72670 2024-04-02T14:43:24.281 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.282+0000 7fee0b400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b358b83bc0 0x55b358b6e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:24.281 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.282+0000 7fee0aa00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b358b71cc0 0x55b358b83330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:24.281 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.282+0000 7fee0aa00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b358b71cc0 0x55b358b83330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:34868/0 (socket says 172.21.15.90:34868) 2024-04-02T14:43:24.281 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.282+0000 7fee0aa00640 1 -- 172.21.15.90:0/1257426989 learned_addr learned my addr 172.21.15.90:0/1257426989 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:43:24.281 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.282+0000 7fee0b400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b358b83bc0 0x55b358b6e140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:48180/0 (socket says 172.21.15.90:48180) 2024-04-02T14:43:24.281 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.282+0000 7fee03e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b358b72670 0x55b358b8f480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:24.281 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.282+0000 7fee0aa00640 1 -- 172.21.15.90:0/1257426989 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b358b72670 msgr2=0x55b358b8f480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:43:24.281 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.282+0000 7fee0aa00640 1 --2- 172.21.15.90:0/1257426989 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b358b72670 0x55b358b8f480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:24.281 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.282+0000 7fee0aa00640 1 -- 172.21.15.90:0/1257426989 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b358b83bc0 msgr2=0x55b358b6e140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:43:24.281 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.282+0000 7fee0aa00640 1 --2- 172.21.15.90:0/1257426989 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b358b83bc0 0x55b358b6e140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:24.281 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.282+0000 7fee0aa00640 1 -- 172.21.15.90:0/1257426989 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b358b72410 con 0x55b358b71cc0 2024-04-02T14:43:24.284 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.282+0000 7fee0aa00640 1 --2- 172.21.15.90:0/1257426989 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b358b71cc0 0x55b358b83330 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7fedfc00c000 tx=0x7fedfc004050 comp rx=0 tx=0).ready entity=mon.2 client_cookie=deabd2024a25711e server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:43:24.284 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.282+0000 7fedfb400640 1 -- 172.21.15.90:0/1257426989 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedfc011500 con 0x55b358b71cc0 2024-04-02T14:43:24.284 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.283+0000 7fedfb400640 1 -- 172.21.15.90:0/1257426989 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fedfc01b4a0 con 0x55b358b71cc0 2024-04-02T14:43:24.284 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.283+0000 7fedfb400640 1 -- 172.21.15.90:0/1257426989 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedfc005070 con 0x55b358b71cc0 2024-04-02T14:43:24.284 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.283+0000 7fee14baa740 1 -- 172.21.15.90:0/1257426989 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b358b71cc0 msgr2=0x55b358b83330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:24.284 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.283+0000 7fee14baa740 1 --2- 172.21.15.90:0/1257426989 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b358b71cc0 0x55b358b83330 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7fedfc00c000 tx=0x7fedfc004050 comp rx=0 tx=0).stop 2024-04-02T14:43:24.285 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.283+0000 7fee14baa740 1 -- 172.21.15.90:0/1257426989 shutdown_connections 2024-04-02T14:43:24.285 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.283+0000 7fee14baa740 1 --2- 172.21.15.90:0/1257426989 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b358b83bc0 0x55b358b6e140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:24.285 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.283+0000 7fee14baa740 1 --2- 172.21.15.90:0/1257426989 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b358b72670 0x55b358b8f480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:24.285 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.283+0000 7fee14baa740 1 --2- 172.21.15.90:0/1257426989 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b358b71cc0 0x55b358b83330 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:24.285 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.283+0000 7fee14baa740 1 -- 172.21.15.90:0/1257426989 >> 172.21.15.90:0/1257426989 conn(0x55b358b6c5e0 msgr2=0x55b358b6da30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:43:24.285 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.283+0000 7fee14baa740 1 -- 172.21.15.90:0/1257426989 shutdown_connections 2024-04-02T14:43:24.285 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.283+0000 7fee14baa740 1 -- 172.21.15.90:0/1257426989 wait complete. 2024-04-02T14:43:24.285 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.284+0000 7fee14baa740 1 -- start start 2024-04-02T14:43:24.285 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.284+0000 7fee14baa740 1 Processor -- start 2024-04-02T14:43:24.285 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.284+0000 7fee14baa740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b358b71cc0 0x55b358b87a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:24.285 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.284+0000 7fee14baa740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b358b72670 0x55b358b87fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:24.285 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.284+0000 7fee14baa740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b358b83bc0 0x55b358bf92c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:24.285 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.284+0000 7fee14baa740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fee0c1aca20 con 0x55b358b83bc0 2024-04-02T14:43:24.285 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.284+0000 7fee14baa740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fee0c1ac350 con 0x55b358b71cc0 2024-04-02T14:43:24.286 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.284+0000 7fee14baa740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fee0c1a9610 con 0x55b358b72670 2024-04-02T14:43:24.286 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.284+0000 7fee14baa740 10 client.0 _ll_register_callbacks cb 0x7fedfc0044c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:43:24.286 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.285+0000 7fee0aa00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b358b71cc0 0x55b358b87a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:24.286 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.285+0000 7fee0aa00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b358b71cc0 0x55b358b87a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:34872/0 (socket says 172.21.15.90:34872) 2024-04-02T14:43:24.286 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.285+0000 7fee0aa00640 1 -- 172.21.15.90:0/3249563475 learned_addr learned my addr 172.21.15.90:0/3249563475 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:43:24.286 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.285+0000 7fee03e00640 1 --2- 172.21.15.90:0/3249563475 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b358b72670 0x55b358b87fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:24.286 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.285+0000 7fee0b400640 1 --2- 172.21.15.90:0/3249563475 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b358b83bc0 0x55b358bf92c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:24.286 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.285+0000 7fee0aa00640 1 -- 172.21.15.90:0/3249563475 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b358b72670 msgr2=0x55b358b87fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:43:24.286 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.285+0000 7fee0aa00640 1 --2- 172.21.15.90:0/3249563475 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b358b72670 0x55b358b87fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:24.286 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.285+0000 7fee0aa00640 1 -- 172.21.15.90:0/3249563475 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b358b83bc0 msgr2=0x55b358bf92c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:43:24.286 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.285+0000 7fee0aa00640 1 --2- 172.21.15.90:0/3249563475 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b358b83bc0 0x55b358bf92c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:24.286 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.285+0000 7fee0aa00640 1 -- 172.21.15.90:0/3249563475 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b3588b7620 con 0x55b358b71cc0 2024-04-02T14:43:24.286 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.285+0000 7fee03e00640 1 --2- 172.21.15.90:0/3249563475 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b358b72670 0x55b358b87fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:43:24.286 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.285+0000 7fee0aa00640 1 --2- 172.21.15.90:0/3249563475 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b358b71cc0 0x55b358b87a90 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7fedfc005d00 tx=0x7fedfc003f00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=35bc08d540e088a2 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:43:24.287 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.286+0000 7fedf8c00640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:43:24.287 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.286+0000 7fedf8c00640 1 -- 172.21.15.90:0/3249563475 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedfc034060 con 0x55b358b71cc0 2024-04-02T14:43:24.287 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.286+0000 7fedf8c00640 1 -- 172.21.15.90:0/3249563475 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8550+0+0 (secure 0 0 0) 0x7fedfc011980 con 0x55b358b71cc0 2024-04-02T14:43:24.287 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.286+0000 7fee14baa740 1 -- 172.21.15.90:0/3249563475 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fedfc01b4a0 con 0x55b358b71cc0 2024-04-02T14:43:24.287 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.287+0000 7fedf8c00640 1 -- 172.21.15.90:0/3249563475 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fedfc011b80 con 0x55b358b71cc0 2024-04-02T14:43:24.288 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.288+0000 7fee14baa740 10 client.28145 fetch_fsmap learned FSMap version 27 2024-04-02T14:43:24.288 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.288+0000 7fee14baa740 1 -- 172.21.15.90:0/3249563475 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55b358bfa8b0 con 0x55b358b71cc0 2024-04-02T14:43:24.288 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.288+0000 7fedf8c00640 1 -- 172.21.15.90:0/3249563475 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fedfc01b4a0 con 0x55b358b71cc0 2024-04-02T14:43:24.288 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.290+0000 7fedf8c00640 1 -- 172.21.15.90:0/3249563475 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fedfc02b030 con 0x55b358b71cc0 2024-04-02T14:43:24.288 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.290+0000 7fee14baa740 10 client.28145 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:43:24.289 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.290+0000 7fee14baa740 10 client.28145 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-02T14:43:24.289 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.291+0000 7fee14baa740 20 client.28145 populate_metadata read hostname 'smithi090' 2024-04-02T14:43:24.289 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.291+0000 7fee14baa740 1 --2- 172.21.15.90:0/3249563475 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55b358bfaeb0 0x55b358bfd2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:24.289 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.291+0000 7fee14baa740 4 client.28145 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-02T14:43:24.289 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.291+0000 7fee14baa740 1 -- 172.21.15.90:0/3249563475 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55b358ae42b0 con 0x55b358bfaeb0 2024-04-02T14:43:24.290 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.292+0000 7fee03e00640 1 --2- 172.21.15.90:0/3249563475 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55b358bfaeb0 0x55b358bfd2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:24.290 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.292+0000 7fee03e00640 1 --2- 172.21.15.90:0/3249563475 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55b358bfaeb0 0x55b358bfd2a0 crc :-1 s=READY pgs=235 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cff8c9a06d1acfb6 server_cookie=dea997e025c13d48 in_seq=0 out_seq=0 2024-04-02T14:43:24.290 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:24.292+0000 7fedf8c00640 10 client.28145 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:43:24.356 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:43:24 smithi092 ceph-mon[36497]: pgmap v262: 97 pgs: 97 active+clean; 4.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 143 KiB/s rd, 1.6 MiB/s wr, 314 op/s 2024-04-02T14:43:24.432 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:43:24 smithi177 ceph-mon[36431]: pgmap v262: 97 pgs: 97 active+clean; 4.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 143 KiB/s rd, 1.6 MiB/s wr, 314 op/s 2024-04-02T14:43:24.434 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:43:24 smithi090 ceph-mon[31674]: pgmap v262: 97 pgs: 97 active+clean; 4.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 143 KiB/s rd, 1.6 MiB/s wr, 314 op/s 2024-04-02T14:43:26.037 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:26.039+0000 7fedf8c00640 1 -- 172.21.15.90:0/3249563475 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55b358ae42b0 con 0x55b358bfaeb0 2024-04-02T14:43:26.037 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:26.039+0000 7fedf8c00640 10 client.28145 handle_command_reply: tid=0 2024-04-02T14:43:26.037 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:26.039+0000 7fee14baa740 1 client.28145 shutdown 2024-04-02T14:43:26.038 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:26.039+0000 7fee14baa740 1 -- 172.21.15.90:0/3249563475 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b358b71cc0 msgr2=0x55b358b87a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:26.038 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:26.040+0000 7fee14baa740 1 --2- 172.21.15.90:0/3249563475 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b358b71cc0 0x55b358b87a90 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7fedfc005d00 tx=0x7fedfc003f00 comp rx=0 tx=0).stop 2024-04-02T14:43:26.038 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:26.040+0000 7fee14baa740 1 -- 172.21.15.90:0/3249563475 shutdown_connections 2024-04-02T14:43:26.038 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:26.040+0000 7fee14baa740 1 --2- 172.21.15.90:0/3249563475 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55b358bfaeb0 0x55b358bfd2a0 crc :-1 s=READY pgs=235 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:26.038 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:26.040+0000 7fee14baa740 1 --2- 172.21.15.90:0/3249563475 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b358b83bc0 0x55b358bf92c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:26.039 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:26.040+0000 7fee14baa740 1 --2- 172.21.15.90:0/3249563475 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b358b72670 0x55b358b87fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:26.039 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:26.040+0000 7fedf8c00640 0 client.28145 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:43:26.039 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:26.040+0000 7fee14baa740 1 --2- 172.21.15.90:0/3249563475 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b358b71cc0 0x55b358b87a90 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:26.039 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:26.040+0000 7fee14baa740 1 -- 172.21.15.90:0/3249563475 >> 172.21.15.90:0/3249563475 conn(0x55b358b6c5e0 msgr2=0x55b358b6b640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:43:26.039 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:26.041+0000 7fee14baa740 1 -- 172.21.15.90:0/3249563475 shutdown_connections 2024-04-02T14:43:26.039 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:26.041+0000 7fee14baa740 1 -- 172.21.15.90:0/3249563475 wait complete. 2024-04-02T14:43:26.040 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:26.041+0000 7fee14baa740 20 client.28145 trim_cache size 0 max 16384 2024-04-02T14:43:26.041 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:{ 2024-04-02T14:43:26.041 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "message": "", 2024-04-02T14:43:26.041 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "return_code": 0 2024-04-02T14:43:26.041 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:} 2024-04-02T14:43:26.041 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:26.043+0000 7fee11e00640 1 -- 172.21.15.90:0/1453452649 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fedf00a3e40 msgr2=0x7fedf00a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:26.041 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:26.043+0000 7fee11e00640 1 --2- 172.21.15.90:0/1453452649 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fedf00a3e40 0x7fedf00a6300 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fee04005fd0 tx=0x7fee04011040 comp rx=0 tx=0).stop 2024-04-02T14:43:26.041 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:26.043+0000 7fee11e00640 1 -- 172.21.15.90:0/1453452649 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fee0c101610 msgr2=0x7fee0c19aae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:26.041 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:26.043+0000 7fee11e00640 1 --2- 172.21.15.90:0/1453452649 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fee0c101610 0x7fee0c19aae0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7fedf4037040 tx=0x7fedf4032480 comp rx=0 tx=0).stop 2024-04-02T14:43:26.042 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:26.044+0000 7fee11e00640 1 -- 172.21.15.90:0/1453452649 shutdown_connections 2024-04-02T14:43:26.042 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:26.044+0000 7fee11e00640 1 --2- 172.21.15.90:0/1453452649 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fedf00a3e40 0x7fedf00a6300 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:26.042 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:26.044+0000 7fee11e00640 1 --2- 172.21.15.90:0/1453452649 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fee0c1059a0 0x7fee0c19ee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:26.042 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:26.044+0000 7fee11e00640 1 --2- 172.21.15.90:0/1453452649 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fee0c101610 0x7fee0c19aae0 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:26.042 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:26.044+0000 7fee11e00640 1 --2- 172.21.15.90:0/1453452649 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fee0c100c50 0x7fee0c19a5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:26.042 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:26.044+0000 7fee11e00640 1 -- 172.21.15.90:0/1453452649 >> 172.21.15.90:0/1453452649 conn(0x7fee0c077850 msgr2=0x7fee0c069180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:43:26.043 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:26.044+0000 7fee11e00640 1 -- 172.21.15.90:0/1453452649 shutdown_connections 2024-04-02T14:43:26.043 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:26.044+0000 7fee11e00640 1 -- 172.21.15.90:0/1453452649 wait complete. 2024-04-02T14:43:26.356 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:43:26 smithi092 ceph-mon[36497]: pgmap v263: 97 pgs: 97 active+clean; 4.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 28 MiB/s wr, 1.87k op/s 2024-04-02T14:43:26.432 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:43:26 smithi177 ceph-mon[36431]: pgmap v263: 97 pgs: 97 active+clean; 4.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 28 MiB/s wr, 1.87k op/s 2024-04-02T14:43:26.434 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:43:26 smithi090 ceph-mon[31674]: pgmap v263: 97 pgs: 97 active+clean; 4.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 28 MiB/s wr, 1.87k op/s 2024-04-02T14:43:26.827 INFO:teuthology.orchestra.run.smithi090.stderr:progress: 475.0 s, 1655.4 tps, lat 61.213 ms stddev 76.369 2024-04-02T14:43:28.356 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:43:28 smithi092 ceph-mon[36497]: pgmap v264: 97 pgs: 97 active+clean; 4.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 28 MiB/s wr, 1.87k op/s 2024-04-02T14:43:28.432 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:43:28 smithi177 ceph-mon[36431]: pgmap v264: 97 pgs: 97 active+clean; 4.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 28 MiB/s wr, 1.87k op/s 2024-04-02T14:43:28.435 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:43:28 smithi090 ceph-mon[31674]: pgmap v264: 97 pgs: 97 active+clean; 4.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 28 MiB/s wr, 1.87k op/s 2024-04-02T14:43:30.356 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:43:30 smithi092 ceph-mon[36497]: pgmap v265: 97 pgs: 97 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 31 MiB/s wr, 1.95k op/s 2024-04-02T14:43:30.432 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:43:30 smithi177 ceph-mon[36431]: pgmap v265: 97 pgs: 97 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 31 MiB/s wr, 1.95k op/s 2024-04-02T14:43:30.434 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:43:30 smithi090 ceph-mon[31674]: pgmap v265: 97 pgs: 97 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 31 MiB/s wr, 1.95k op/s 2024-04-02T14:43:31.827 INFO:teuthology.orchestra.run.smithi090.stderr:progress: 480.0 s, 2079.4 tps, lat 47.900 ms stddev 57.479 2024-04-02T14:43:32.432 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:43:32 smithi177 ceph-mon[36431]: pgmap v266: 97 pgs: 97 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 31 MiB/s wr, 1.90k op/s 2024-04-02T14:43:32.434 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:43:32 smithi090 ceph-mon[31674]: pgmap v266: 97 pgs: 97 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 31 MiB/s wr, 1.90k op/s 2024-04-02T14:43:32.606 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:43:32 smithi092 ceph-mon[36497]: pgmap v266: 97 pgs: 97 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 31 MiB/s wr, 1.90k op/s 2024-04-02T14:43:34.408 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:43:34 smithi092 ceph-mon[36497]: pgmap v267: 97 pgs: 97 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 27 MiB/s wr, 1.66k op/s 2024-04-02T14:43:34.432 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:43:34 smithi177 ceph-mon[36431]: pgmap v267: 97 pgs: 97 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 27 MiB/s wr, 1.66k op/s 2024-04-02T14:43:34.434 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:43:34 smithi090 ceph-mon[31674]: pgmap v267: 97 pgs: 97 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 27 MiB/s wr, 1.66k op/s 2024-04-02T14:43:36.432 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:43:36 smithi177 ceph-mon[36431]: pgmap v268: 97 pgs: 97 active+clean; 4.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 45 MiB/s wr, 2.57k op/s 2024-04-02T14:43:36.434 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:43:36 smithi090 ceph-mon[31674]: pgmap v268: 97 pgs: 97 active+clean; 4.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 45 MiB/s wr, 2.57k op/s 2024-04-02T14:43:36.606 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:43:36 smithi092 ceph-mon[36497]: pgmap v268: 97 pgs: 97 active+clean; 4.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 45 MiB/s wr, 2.57k op/s 2024-04-02T14:43:36.827 INFO:teuthology.orchestra.run.smithi090.stderr:progress: 485.0 s, 1709.0 tps, lat 57.276 ms stddev 110.591 2024-04-02T14:43:38.432 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:43:38 smithi177 ceph-mon[36431]: pgmap v269: 97 pgs: 97 active+clean; 4.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 23 MiB/s wr, 1.20k op/s 2024-04-02T14:43:38.434 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:43:38 smithi090 ceph-mon[31674]: pgmap v269: 97 pgs: 97 active+clean; 4.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 23 MiB/s wr, 1.20k op/s 2024-04-02T14:43:38.606 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:43:38 smithi092 ceph-mon[36497]: pgmap v269: 97 pgs: 97 active+clean; 4.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 23 MiB/s wr, 1.20k op/s 2024-04-02T14:43:39.250 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.253+0000 7f22f2000640 1 -- 172.21.15.90:0/3201071612 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f22ec0747f0 msgr2=0x7f22ec10de10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:39.250 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.253+0000 7f22f2000640 1 --2- 172.21.15.90:0/3201071612 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f22ec0747f0 0x7f22ec10de10 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f22d8009900 tx=0x7f22d802bcd0 comp rx=0 tx=0).stop 2024-04-02T14:43:39.251 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.253+0000 7f22f2000640 1 -- 172.21.15.90:0/3201071612 shutdown_connections 2024-04-02T14:43:39.251 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.253+0000 7f22f2000640 1 --2- 172.21.15.90:0/3201071612 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f22ec10e350 0x7f22ec110760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:39.251 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.253+0000 7f22f2000640 1 --2- 172.21.15.90:0/3201071612 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f22ec0747f0 0x7f22ec10de10 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:39.251 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.253+0000 7f22f2000640 1 --2- 172.21.15.90:0/3201071612 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f22ec073ed0 0x7f22ec0742b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:39.251 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.253+0000 7f22f2000640 1 -- 172.21.15.90:0/3201071612 >> 172.21.15.90:0/3201071612 conn(0x7f22ec0fd5c0 msgr2=0x7f22ec0ff9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:43:39.251 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.253+0000 7f22f2000640 1 -- 172.21.15.90:0/3201071612 shutdown_connections 2024-04-02T14:43:39.251 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.253+0000 7f22f2000640 1 -- 172.21.15.90:0/3201071612 wait complete. 2024-04-02T14:43:39.251 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.255+0000 7f22f2000640 1 Processor -- start 2024-04-02T14:43:39.252 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.256+0000 7f22f2000640 1 -- start start 2024-04-02T14:43:39.253 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.256+0000 7f22f2000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f22ec073ed0 0x7f22ec10af50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:39.253 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.256+0000 7f22f2000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f22ec10e350 0x7f22ec10b490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:39.253 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.256+0000 7f22f2000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f22ec10b9d0 0x7f22ec1aae40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:39.254 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.257+0000 7f22f2000640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f22ec114100 con 0x7f22ec10e350 2024-04-02T14:43:39.254 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.257+0000 7f22f2000640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f22ec113f80 con 0x7f22ec073ed0 2024-04-02T14:43:39.254 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.257+0000 7f22eb400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f22ec073ed0 0x7f22ec10af50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:39.254 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.257+0000 7f22f2000640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f22ec114280 con 0x7f22ec10b9d0 2024-04-02T14:43:39.254 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.257+0000 7f22eb400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f22ec073ed0 0x7f22ec10af50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:53326/0 (socket says 172.21.15.90:53326) 2024-04-02T14:43:39.254 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.257+0000 7f22ebe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f22ec10b9d0 0x7f22ec1aae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:39.254 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.257+0000 7f22ebe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f22ec10b9d0 0x7f22ec1aae40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:41850/0 (socket says 172.21.15.90:41850) 2024-04-02T14:43:39.254 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.257+0000 7f22eb400640 1 -- 172.21.15.90:0/1057100275 learned_addr learned my addr 172.21.15.90:0/1057100275 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:43:39.255 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.258+0000 7f22eaa00640 1 --2- 172.21.15.90:0/1057100275 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f22ec10e350 0x7f22ec10b490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:39.255 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.258+0000 7f22ebe00640 1 -- 172.21.15.90:0/1057100275 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f22ec073ed0 msgr2=0x7f22ec10af50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:39.255 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.258+0000 7f22ebe00640 1 --2- 172.21.15.90:0/1057100275 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f22ec073ed0 0x7f22ec10af50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:39.255 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.258+0000 7f22ebe00640 1 -- 172.21.15.90:0/1057100275 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f22ec10e350 msgr2=0x7f22ec10b490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:39.255 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.258+0000 7f22ebe00640 1 --2- 172.21.15.90:0/1057100275 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f22ec10e350 0x7f22ec10b490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:39.255 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.258+0000 7f22ebe00640 1 -- 172.21.15.90:0/1057100275 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22ec1ab380 con 0x7f22ec10b9d0 2024-04-02T14:43:39.256 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.259+0000 7f22ebe00640 1 --2- 172.21.15.90:0/1057100275 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f22ec10b9d0 0x7f22ec1aae40 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f22ec074e20 tx=0x7f22e400dea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:43:39.256 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.259+0000 7f22e3e00640 1 -- 172.21.15.90:0/1057100275 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22e400bd40 con 0x7f22ec10b9d0 2024-04-02T14:43:39.256 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.260+0000 7f22e3e00640 1 -- 172.21.15.90:0/1057100275 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f22e4003e50 con 0x7f22ec10b9d0 2024-04-02T14:43:39.256 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.260+0000 7f22e3e00640 1 -- 172.21.15.90:0/1057100275 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22e40106c0 con 0x7f22ec10b9d0 2024-04-02T14:43:39.257 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.260+0000 7f22f2000640 1 -- 172.21.15.90:0/1057100275 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22ec1ab550 con 0x7f22ec10b9d0 2024-04-02T14:43:39.258 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.260+0000 7f22f2000640 1 -- 172.21.15.90:0/1057100275 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22ec1aba20 con 0x7f22ec10b9d0 2024-04-02T14:43:39.261 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.265+0000 7f22f4dab740 1 Processor -- start 2024-04-02T14:43:39.263 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.266+0000 7f22e3e00640 1 -- 172.21.15.90:0/1057100275 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f22e4003970 con 0x7f22ec10b9d0 2024-04-02T14:43:39.265 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.267+0000 7f22f4dab740 1 -- start start 2024-04-02T14:43:39.265 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.267+0000 7f22f4dab740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556ca588aeb0 0x556ca588b290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:39.265 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.267+0000 7f22f4dab740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556ca588b7d0 0x556ca5893e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:39.265 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.267+0000 7f22f4dab740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556ca58943b0 0x556ca58967a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:39.265 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.267+0000 7f22f4dab740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x556ca57efd70 con 0x556ca588b7d0 2024-04-02T14:43:39.265 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.267+0000 7f22f4dab740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x556ca57982a0 con 0x556ca58943b0 2024-04-02T14:43:39.265 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.267+0000 7f22f4dab740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x556ca57ebd20 con 0x556ca588aeb0 2024-04-02T14:43:39.265 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.268+0000 7f22eaa00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556ca588aeb0 0x556ca588b290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:39.265 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.268+0000 7f22eaa00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556ca588aeb0 0x556ca588b290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:41856/0 (socket says 172.21.15.90:41856) 2024-04-02T14:43:39.265 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.268+0000 7f22eaa00640 1 -- 172.21.15.90:0/160635525 learned_addr learned my addr 172.21.15.90:0/160635525 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:43:39.265 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.268+0000 7f22eb400640 1 --2- 172.21.15.90:0/160635525 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556ca588b7d0 0x556ca5893e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:39.265 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.268+0000 7f22eaa00640 1 --2- 172.21.15.90:0/160635525 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556ca58943b0 0x556ca58967a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:39.266 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.268+0000 7f22eb400640 1 -- 172.21.15.90:0/160635525 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556ca588aeb0 msgr2=0x556ca588b290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:43:39.266 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.268+0000 7f22eb400640 1 --2- 172.21.15.90:0/160635525 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556ca588aeb0 0x556ca588b290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:39.266 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.268+0000 7f22eb400640 1 -- 172.21.15.90:0/160635525 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556ca58943b0 msgr2=0x556ca58967a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:43:39.266 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.268+0000 7f22eb400640 1 --2- 172.21.15.90:0/160635525 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556ca58943b0 0x556ca58967a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:39.266 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.268+0000 7f22eb400640 1 -- 172.21.15.90:0/160635525 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556ca57ac840 con 0x556ca588b7d0 2024-04-02T14:43:39.267 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.268+0000 7f22eaa00640 1 --2- 172.21.15.90:0/160635525 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556ca58943b0 0x556ca58967a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:43:39.267 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.269+0000 7f22e3e00640 1 --2- 172.21.15.90:0/1057100275 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f22d00a3f20 0x7f22d00a63e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:39.267 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.269+0000 7f22e3e00640 1 -- 172.21.15.90:0/1057100275 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8550+0+0 (secure 0 0 0) 0x7f22e40d09e0 con 0x7f22ec10b9d0 2024-04-02T14:43:39.267 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.269+0000 7f22eaa00640 1 --2- 172.21.15.90:0/1057100275 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f22d00a3f20 0x7f22d00a63e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:39.267 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.269+0000 7f22eb400640 1 --2- 172.21.15.90:0/160635525 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556ca588b7d0 0x556ca5893e70 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f22dc005fd0 tx=0x7f22dc00f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18a68f0bdb58d868 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:43:39.267 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.270+0000 7f22d7e00640 1 -- 172.21.15.90:0/160635525 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22dc01d070 con 0x556ca588b7d0 2024-04-02T14:43:39.267 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.271+0000 7f22eaa00640 1 --2- 172.21.15.90:0/1057100275 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f22d00a3f20 0x7f22d00a63e0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f22d802c2e0 tx=0x7f22d802bcd0 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:43:39.268 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.271+0000 7f22d7e00640 1 -- 172.21.15.90:0/160635525 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f22dc018b20 con 0x556ca588b7d0 2024-04-02T14:43:39.268 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.271+0000 7f22d7e00640 1 -- 172.21.15.90:0/160635525 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22dc01d070 con 0x556ca588b7d0 2024-04-02T14:43:39.269 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.272+0000 7f22f4dab740 1 -- 172.21.15.90:0/160635525 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556ca588b7d0 msgr2=0x556ca5893e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:39.269 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.272+0000 7f22f4dab740 1 --2- 172.21.15.90:0/160635525 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556ca588b7d0 0x556ca5893e70 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f22dc005fd0 tx=0x7f22dc00f790 comp rx=0 tx=0).stop 2024-04-02T14:43:39.269 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.272+0000 7f22f4dab740 1 -- 172.21.15.90:0/160635525 shutdown_connections 2024-04-02T14:43:39.269 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.272+0000 7f22f4dab740 1 --2- 172.21.15.90:0/160635525 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556ca58943b0 0x556ca58967a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:39.269 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.272+0000 7f22f4dab740 1 --2- 172.21.15.90:0/160635525 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556ca588b7d0 0x556ca5893e70 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:39.269 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.272+0000 7f22f4dab740 1 --2- 172.21.15.90:0/160635525 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556ca588aeb0 0x556ca588b290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:39.269 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.272+0000 7f22f4dab740 1 -- 172.21.15.90:0/160635525 >> 172.21.15.90:0/160635525 conn(0x556ca58855e0 msgr2=0x556ca58859c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:43:39.270 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.272+0000 7f22f4dab740 1 -- 172.21.15.90:0/160635525 shutdown_connections 2024-04-02T14:43:39.270 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.273+0000 7f22f4dab740 1 -- 172.21.15.90:0/160635525 wait complete. 2024-04-02T14:43:39.270 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.273+0000 7f22f4dab740 1 -- start start 2024-04-02T14:43:39.270 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.274+0000 7f22f4dab740 1 Processor -- start 2024-04-02T14:43:39.271 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.274+0000 7f22f4dab740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556ca588aeb0 0x556ca5891510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:39.271 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.274+0000 7f22f4dab740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556ca58943b0 0x556ca5891a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:39.271 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.275+0000 7f22f4dab740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556ca5891f90 0x556ca58b1620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:39.272 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.275+0000 7f22f4dab740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x556ca58a81b0 con 0x556ca58943b0 2024-04-02T14:43:39.272 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.275+0000 7f22f4dab740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x556ca57ab660 con 0x556ca588aeb0 2024-04-02T14:43:39.272 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.275+0000 7f22f4dab740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x556ca58a8330 con 0x556ca5891f90 2024-04-02T14:43:39.272 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.275+0000 7f22f4dab740 10 client.0 _ll_register_callbacks cb 0x556ca57ab050 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:43:39.273 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.275+0000 7f22eb400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556ca588aeb0 0x556ca5891510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:39.273 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.275+0000 7f22eb400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556ca588aeb0 0x556ca5891510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:53336/0 (socket says 172.21.15.90:53336) 2024-04-02T14:43:39.273 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.275+0000 7f22eb400640 1 -- 172.21.15.90:0/145041259 learned_addr learned my addr 172.21.15.90:0/145041259 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:43:39.273 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.275+0000 7f22ebe00640 1 --2- 172.21.15.90:0/145041259 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556ca5891f90 0x556ca58b1620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:39.273 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.275+0000 7f22eaa00640 1 --2- 172.21.15.90:0/145041259 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556ca58943b0 0x556ca5891a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:39.273 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.275+0000 7f22eb400640 1 -- 172.21.15.90:0/145041259 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556ca5891f90 msgr2=0x556ca58b1620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:43:39.273 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.275+0000 7f22eb400640 1 --2- 172.21.15.90:0/145041259 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556ca5891f90 0x556ca58b1620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:39.273 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.275+0000 7f22eb400640 1 -- 172.21.15.90:0/145041259 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556ca58943b0 msgr2=0x556ca5891a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:43:39.273 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.276+0000 7f22eb400640 1 --2- 172.21.15.90:0/145041259 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556ca58943b0 0x556ca5891a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:39.274 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.276+0000 7f22eb400640 1 -- 172.21.15.90:0/145041259 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556ca57bae80 con 0x556ca588aeb0 2024-04-02T14:43:39.274 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.276+0000 7f22eaa00640 1 --2- 172.21.15.90:0/145041259 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556ca58943b0 0x556ca5891a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:43:39.274 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.277+0000 7f22eb400640 1 --2- 172.21.15.90:0/145041259 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556ca588aeb0 0x556ca5891510 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f22dc00fc90 tx=0x7f22dc00afd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=55a127fcb80d6491 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:43:39.274 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.277+0000 7f22d4c00640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:43:39.274 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.277+0000 7f22d4c00640 1 -- 172.21.15.90:0/145041259 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22dc027020 con 0x556ca588aeb0 2024-04-02T14:43:39.274 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.277+0000 7f22d4c00640 1 -- 172.21.15.90:0/145041259 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8550+0+0 (secure 0 0 0) 0x7f22dc005040 con 0x556ca588aeb0 2024-04-02T14:43:39.274 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.278+0000 7f22f4dab740 1 -- 172.21.15.90:0/145041259 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f22dc018b20 con 0x556ca588aeb0 2024-04-02T14:43:39.275 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.278+0000 7f22d4c00640 1 -- 172.21.15.90:0/145041259 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f22dc01d040 con 0x556ca588aeb0 2024-04-02T14:43:39.275 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.278+0000 7f22f4dab740 10 client.28157 fetch_fsmap learned FSMap version 27 2024-04-02T14:43:39.275 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.278+0000 7f22f4dab740 1 -- 172.21.15.90:0/145041259 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x556ca5504210 con 0x556ca588aeb0 2024-04-02T14:43:39.275 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.278+0000 7f22d4c00640 1 -- 172.21.15.90:0/145041259 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f22dc018b20 con 0x556ca588aeb0 2024-04-02T14:43:39.276 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.279+0000 7f22d4c00640 1 -- 172.21.15.90:0/145041259 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f22dc0046c0 con 0x556ca588aeb0 2024-04-02T14:43:39.276 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.279+0000 7f22f4dab740 10 client.28157 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:43:39.276 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.279+0000 7f22f4dab740 10 client.28157 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-02T14:43:39.276 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.279+0000 7f22f4dab740 20 client.28157 populate_metadata read hostname 'smithi090' 2024-04-02T14:43:39.276 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.279+0000 7f22f4dab740 1 --2- 172.21.15.90:0/145041259 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x556ca58b5c30 0x556ca58b8020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:39.276 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.279+0000 7f22f4dab740 4 client.28157 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:43:39.277 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.279+0000 7f22f4dab740 1 -- 172.21.15.90:0/145041259 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556ca579fcd0 con 0x556ca58b5c30 2024-04-02T14:43:39.277 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.280+0000 7f22eaa00640 1 --2- 172.21.15.90:0/145041259 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x556ca58b5c30 0x556ca58b8020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:39.277 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.281+0000 7f22eaa00640 1 --2- 172.21.15.90:0/145041259 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x556ca58b5c30 0x556ca58b8020 crc :-1 s=READY pgs=236 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d8e29fd3294edc99 server_cookie=d4245e23eeb2894 in_seq=0 out_seq=0 2024-04-02T14:43:39.278 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.281+0000 7f22d4c00640 10 client.28157 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:43:39.278 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.281+0000 7f22d4c00640 1 -- 172.21.15.90:0/145041259 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x556ca579fcd0 con 0x556ca58b5c30 2024-04-02T14:43:39.278 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.281+0000 7f22d4c00640 10 client.28157 handle_command_reply: tid=0 2024-04-02T14:43:39.279 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.283+0000 7f22f4dab740 1 client.28157 shutdown 2024-04-02T14:43:39.280 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.283+0000 7f22f4dab740 1 -- 172.21.15.90:0/145041259 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556ca588aeb0 msgr2=0x556ca5891510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:39.280 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.283+0000 7f22f4dab740 1 --2- 172.21.15.90:0/145041259 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556ca588aeb0 0x556ca5891510 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f22dc00fc90 tx=0x7f22dc00afd0 comp rx=0 tx=0).stop 2024-04-02T14:43:39.280 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.284+0000 7f22f4dab740 1 -- 172.21.15.90:0/145041259 shutdown_connections 2024-04-02T14:43:39.281 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.284+0000 7f22f4dab740 1 --2- 172.21.15.90:0/145041259 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x556ca58b5c30 0x556ca58b8020 crc :-1 s=READY pgs=236 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:39.281 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.284+0000 7f22f4dab740 1 --2- 172.21.15.90:0/145041259 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556ca5891f90 0x556ca58b1620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:39.281 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.284+0000 7f22f4dab740 1 --2- 172.21.15.90:0/145041259 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556ca58943b0 0x556ca5891a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:39.281 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.284+0000 7f22f4dab740 1 --2- 172.21.15.90:0/145041259 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556ca588aeb0 0x556ca5891510 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:39.281 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.284+0000 7f22f4dab740 1 -- 172.21.15.90:0/145041259 >> 172.21.15.90:0/145041259 conn(0x556ca58855e0 msgr2=0x556ca5887330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:43:39.281 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.284+0000 7f22d4c00640 0 client.28157 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:43:39.281 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.284+0000 7f22f4dab740 1 -- 172.21.15.90:0/145041259 shutdown_connections 2024-04-02T14:43:39.282 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.284+0000 7f22f4dab740 1 -- 172.21.15.90:0/145041259 wait complete. 2024-04-02T14:43:39.282 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.284+0000 7f22f4dab740 20 client.28157 trim_cache size 0 max 16384 2024-04-02T14:43:39.308 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.311+0000 7f22f4dab740 1 Processor -- start 2024-04-02T14:43:39.308 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.311+0000 7f22f4dab740 1 -- start start 2024-04-02T14:43:39.309 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.311+0000 7f22f4dab740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556ca588aeb0 0x556ca58b1590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:39.309 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.311+0000 7f22f4dab740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556ca5891f90 0x556ca58844c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:39.309 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.311+0000 7f22f4dab740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556ca58943b0 0x556ca58bb040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:39.309 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.312+0000 7f22f4dab740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f22ec1abd10 con 0x556ca5891f90 2024-04-02T14:43:39.309 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.312+0000 7f22f4dab740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f22ec1ad730 con 0x556ca588aeb0 2024-04-02T14:43:39.309 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.312+0000 7f22f4dab740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f22ec1ac2c0 con 0x556ca58943b0 2024-04-02T14:43:39.310 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.312+0000 7f22eb400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556ca588aeb0 0x556ca58b1590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:39.310 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.312+0000 7f22eb400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556ca588aeb0 0x556ca58b1590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:53346/0 (socket says 172.21.15.90:53346) 2024-04-02T14:43:39.310 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.312+0000 7f22eb400640 1 -- 172.21.15.90:0/1416896303 learned_addr learned my addr 172.21.15.90:0/1416896303 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:43:39.310 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.312+0000 7f22ebe00640 1 --2- 172.21.15.90:0/1416896303 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556ca58943b0 0x556ca58bb040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:39.310 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.312+0000 7f22eaa00640 1 --2- 172.21.15.90:0/1416896303 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556ca5891f90 0x556ca58844c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:39.310 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.312+0000 7f22eb400640 1 -- 172.21.15.90:0/1416896303 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556ca58943b0 msgr2=0x556ca58bb040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:43:39.310 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.312+0000 7f22eb400640 1 --2- 172.21.15.90:0/1416896303 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556ca58943b0 0x556ca58bb040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:39.310 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.312+0000 7f22eb400640 1 -- 172.21.15.90:0/1416896303 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556ca5891f90 msgr2=0x556ca58844c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:43:39.310 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.312+0000 7f22eb400640 1 --2- 172.21.15.90:0/1416896303 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556ca5891f90 0x556ca58844c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:39.310 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.312+0000 7f22eb400640 1 -- 172.21.15.90:0/1416896303 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556ca55d0620 con 0x556ca588aeb0 2024-04-02T14:43:39.310 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.312+0000 7f22ebe00640 1 --2- 172.21.15.90:0/1416896303 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556ca58943b0 0x556ca58bb040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:43:39.311 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.313+0000 7f22eb400640 1 --2- 172.21.15.90:0/1416896303 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556ca588aeb0 0x556ca58b1590 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f22dc00ab80 tx=0x7f22dc00a640 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8b03be1b195fe3f server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:43:39.311 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.313+0000 7f22d7400640 1 -- 172.21.15.90:0/1416896303 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22dc03a070 con 0x556ca588aeb0 2024-04-02T14:43:39.311 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.314+0000 7f22d7400640 1 -- 172.21.15.90:0/1416896303 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f22dc00a7f0 con 0x556ca588aeb0 2024-04-02T14:43:39.311 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.314+0000 7f22d7400640 1 -- 172.21.15.90:0/1416896303 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22dc03a070 con 0x556ca588aeb0 2024-04-02T14:43:39.312 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.315+0000 7f22f4dab740 1 -- 172.21.15.90:0/1416896303 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556ca588aeb0 msgr2=0x556ca58b1590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:39.312 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.315+0000 7f22f4dab740 1 --2- 172.21.15.90:0/1416896303 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556ca588aeb0 0x556ca58b1590 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f22dc00ab80 tx=0x7f22dc00a640 comp rx=0 tx=0).stop 2024-04-02T14:43:39.312 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.315+0000 7f22f4dab740 1 -- 172.21.15.90:0/1416896303 shutdown_connections 2024-04-02T14:43:39.312 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.315+0000 7f22f4dab740 1 --2- 172.21.15.90:0/1416896303 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556ca58943b0 0x556ca58bb040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:39.312 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.315+0000 7f22f4dab740 1 --2- 172.21.15.90:0/1416896303 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556ca5891f90 0x556ca58844c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:39.313 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.315+0000 7f22f4dab740 1 --2- 172.21.15.90:0/1416896303 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556ca588aeb0 0x556ca58b1590 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:39.313 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.315+0000 7f22f4dab740 1 -- 172.21.15.90:0/1416896303 >> 172.21.15.90:0/1416896303 conn(0x556ca58855e0 msgr2=0x556ca5883db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:43:39.313 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.316+0000 7f22f4dab740 1 -- 172.21.15.90:0/1416896303 shutdown_connections 2024-04-02T14:43:39.313 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.316+0000 7f22f4dab740 1 -- 172.21.15.90:0/1416896303 wait complete. 2024-04-02T14:43:39.313 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.316+0000 7f22f4dab740 1 -- start start 2024-04-02T14:43:39.313 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.317+0000 7f22f4dab740 1 Processor -- start 2024-04-02T14:43:39.315 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.317+0000 7f22f4dab740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556ca588aeb0 0x556ca58bb370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:39.315 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.317+0000 7f22f4dab740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556ca5891f90 0x556ca58b8700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:39.315 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.318+0000 7f22f4dab740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556ca58943b0 0x556ca58b8c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:39.316 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.318+0000 7f22f4dab740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f22ec1ad060 con 0x556ca5891f90 2024-04-02T14:43:39.316 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.318+0000 7f22f4dab740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f22ec1ac990 con 0x556ca588aeb0 2024-04-02T14:43:39.316 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.318+0000 7f22f4dab740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f22ec071c10 con 0x556ca58943b0 2024-04-02T14:43:39.316 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.318+0000 7f22f4dab740 10 client.0 _ll_register_callbacks cb 0x7f22d8009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:43:39.316 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.318+0000 7f22ebe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556ca58943b0 0x556ca58b8c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:39.316 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.318+0000 7f22eaa00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556ca5891f90 0x556ca58b8700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:39.316 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.318+0000 7f22ebe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556ca58943b0 0x556ca58b8c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:41900/0 (socket says 172.21.15.90:41900) 2024-04-02T14:43:39.316 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.318+0000 7f22ebe00640 1 -- 172.21.15.90:0/2124319179 learned_addr learned my addr 172.21.15.90:0/2124319179 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:43:39.316 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.318+0000 7f22eb400640 1 --2- 172.21.15.90:0/2124319179 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556ca588aeb0 0x556ca58bb370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:39.316 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.318+0000 7f22ebe00640 1 -- 172.21.15.90:0/2124319179 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556ca588aeb0 msgr2=0x556ca58bb370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:43:39.316 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.318+0000 7f22ebe00640 1 --2- 172.21.15.90:0/2124319179 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556ca588aeb0 0x556ca58bb370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:39.316 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.318+0000 7f22ebe00640 1 -- 172.21.15.90:0/2124319179 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556ca5891f90 msgr2=0x556ca58b8700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:43:39.316 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.318+0000 7f22ebe00640 1 --2- 172.21.15.90:0/2124319179 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556ca5891f90 0x556ca58b8700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:39.316 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.318+0000 7f22ebe00640 1 -- 172.21.15.90:0/2124319179 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556ca58b9440 con 0x556ca58943b0 2024-04-02T14:43:39.317 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.319+0000 7f22ebe00640 1 --2- 172.21.15.90:0/2124319179 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556ca58943b0 0x556ca58b8c40 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f22e40ce170 tx=0x7f22e409c5b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=829420480b587e4c server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:43:39.317 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.319+0000 7f22d4c00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:43:39.317 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.319+0000 7f22d4c00640 1 -- 172.21.15.90:0/2124319179 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22e40a0d10 con 0x556ca58943b0 2024-04-02T14:43:39.317 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.319+0000 7f22d4c00640 1 -- 172.21.15.90:0/2124319179 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8550+0+0 (secure 0 0 0) 0x7f22e40ce390 con 0x556ca58943b0 2024-04-02T14:43:39.317 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.319+0000 7f22f4dab740 1 -- 172.21.15.90:0/2124319179 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f22dc00a7f0 con 0x556ca58943b0 2024-04-02T14:43:39.317 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.320+0000 7f22d4c00640 1 -- 172.21.15.90:0/2124319179 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f22e40ac050 con 0x556ca58943b0 2024-04-02T14:43:39.317 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.320+0000 7f22f4dab740 10 client.28000 fetch_fsmap learned FSMap version 27 2024-04-02T14:43:39.318 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.321+0000 7f22f4dab740 1 -- 172.21.15.90:0/2124319179 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x556ca591be80 con 0x556ca58943b0 2024-04-02T14:43:39.318 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.321+0000 7f22d4c00640 1 -- 172.21.15.90:0/2124319179 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f22dc00a7f0 con 0x556ca58943b0 2024-04-02T14:43:39.319 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.322+0000 7f22d4c00640 1 -- 172.21.15.90:0/2124319179 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f22e4024060 con 0x556ca58943b0 2024-04-02T14:43:39.319 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.322+0000 7f22f4dab740 10 client.28000 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:43:39.319 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.322+0000 7f22f4dab740 10 client.28000 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-02T14:43:39.319 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.322+0000 7f22f4dab740 20 client.28000 populate_metadata read hostname 'smithi090' 2024-04-02T14:43:39.319 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.322+0000 7f22f4dab740 1 --2- 172.21.15.90:0/2124319179 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x556ca591c4d0 0x556ca591e970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:39.319 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.322+0000 7f22f4dab740 4 client.28000 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-02T14:43:39.319 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.322+0000 7f22f4dab740 1 -- 172.21.15.90:0/2124319179 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "flush journal"}) -- 0x556ca57fd2b0 con 0x556ca591c4d0 2024-04-02T14:43:39.319 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.322+0000 7f22eb400640 1 --2- 172.21.15.90:0/2124319179 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x556ca591c4d0 0x556ca591e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:39.320 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.322+0000 7f22eb400640 1 --2- 172.21.15.90:0/2124319179 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x556ca591c4d0 0x556ca591e970 crc :-1 s=READY pgs=237 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1abd3f2c120823cc server_cookie=730fa60c44802362 in_seq=0 out_seq=0 2024-04-02T14:43:39.320 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.322+0000 7f22d4c00640 10 client.28000 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:43:39.386 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.389+0000 7f22d4c00640 1 -- 172.21.15.90:0/2124319179 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x556ca57fd2b0 con 0x556ca591c4d0 2024-04-02T14:43:39.387 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.389+0000 7f22d4c00640 10 client.28000 handle_command_reply: tid=0 2024-04-02T14:43:39.387 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.390+0000 7f22f4dab740 1 client.28000 shutdown 2024-04-02T14:43:39.387 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.390+0000 7f22f4dab740 1 -- 172.21.15.90:0/2124319179 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556ca58943b0 msgr2=0x556ca58b8c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:39.387 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.390+0000 7f22f4dab740 1 --2- 172.21.15.90:0/2124319179 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556ca58943b0 0x556ca58b8c40 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f22e40ce170 tx=0x7f22e409c5b0 comp rx=0 tx=0).stop 2024-04-02T14:43:39.388 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.390+0000 7f22f4dab740 1 -- 172.21.15.90:0/2124319179 shutdown_connections 2024-04-02T14:43:39.388 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.390+0000 7f22f4dab740 1 --2- 172.21.15.90:0/2124319179 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x556ca591c4d0 0x556ca591e970 crc :-1 s=READY pgs=237 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:39.388 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.390+0000 7f22f4dab740 1 --2- 172.21.15.90:0/2124319179 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556ca58943b0 0x556ca58b8c40 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:39.388 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.390+0000 7f22f4dab740 1 --2- 172.21.15.90:0/2124319179 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556ca5891f90 0x556ca58b8700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:39.388 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.390+0000 7f22f4dab740 1 --2- 172.21.15.90:0/2124319179 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556ca588aeb0 0x556ca58bb370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:39.388 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.390+0000 7f22f4dab740 1 -- 172.21.15.90:0/2124319179 >> 172.21.15.90:0/2124319179 conn(0x556ca58855e0 msgr2=0x556ca5884640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:43:39.388 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.390+0000 7f22d4c00640 0 client.28000 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:43:39.388 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.391+0000 7f22f4dab740 1 -- 172.21.15.90:0/2124319179 shutdown_connections 2024-04-02T14:43:39.389 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.391+0000 7f22f4dab740 1 -- 172.21.15.90:0/2124319179 wait complete. 2024-04-02T14:43:39.389 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.391+0000 7f22f4dab740 20 client.28000 trim_cache size 0 max 16384 2024-04-02T14:43:39.389 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:{ 2024-04-02T14:43:39.389 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "message": "", 2024-04-02T14:43:39.389 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "return_code": 0 2024-04-02T14:43:39.389 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:} 2024-04-02T14:43:39.390 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.392+0000 7f22f2000640 1 -- 172.21.15.90:0/1057100275 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f22d00a3f20 msgr2=0x7f22d00a63e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:39.390 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.392+0000 7f22f2000640 1 --2- 172.21.15.90:0/1057100275 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f22d00a3f20 0x7f22d00a63e0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f22d802c2e0 tx=0x7f22d802bcd0 comp rx=0 tx=0).stop 2024-04-02T14:43:39.390 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.392+0000 7f22f2000640 1 -- 172.21.15.90:0/1057100275 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f22ec10b9d0 msgr2=0x7f22ec1aae40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:39.390 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.392+0000 7f22f2000640 1 --2- 172.21.15.90:0/1057100275 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f22ec10b9d0 0x7f22ec1aae40 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f22ec074e20 tx=0x7f22e400dea0 comp rx=0 tx=0).stop 2024-04-02T14:43:39.390 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.392+0000 7f22f2000640 1 -- 172.21.15.90:0/1057100275 shutdown_connections 2024-04-02T14:43:39.390 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.392+0000 7f22f2000640 1 --2- 172.21.15.90:0/1057100275 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f22d00a3f20 0x7f22d00a63e0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:39.390 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.392+0000 7f22f2000640 1 --2- 172.21.15.90:0/1057100275 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f22ec10b9d0 0x7f22ec1aae40 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:39.390 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.392+0000 7f22f2000640 1 --2- 172.21.15.90:0/1057100275 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f22ec10e350 0x7f22ec10b490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:39.391 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.392+0000 7f22f2000640 1 --2- 172.21.15.90:0/1057100275 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f22ec073ed0 0x7f22ec10af50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:39.391 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.392+0000 7f22f2000640 1 -- 172.21.15.90:0/1057100275 >> 172.21.15.90:0/1057100275 conn(0x7f22ec0fd5c0 msgr2=0x7f22ec0fee60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:43:39.391 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.392+0000 7f22f2000640 1 -- 172.21.15.90:0/1057100275 shutdown_connections 2024-04-02T14:43:39.391 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:39.392+0000 7f22f2000640 1 -- 172.21.15.90:0/1057100275 wait complete. 2024-04-02T14:43:40.432 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:43:40 smithi177 ceph-mon[36431]: pgmap v270: 97 pgs: 97 active+clean; 4.6 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 34 MiB/s wr, 1.87k op/s 2024-04-02T14:43:40.434 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:43:40 smithi090 ceph-mon[31674]: pgmap v270: 97 pgs: 97 active+clean; 4.6 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 34 MiB/s wr, 1.87k op/s 2024-04-02T14:43:40.606 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:43:40 smithi092 ceph-mon[36497]: pgmap v270: 97 pgs: 97 active+clean; 4.6 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 34 MiB/s wr, 1.87k op/s 2024-04-02T14:43:41.827 INFO:teuthology.orchestra.run.smithi090.stderr:progress: 490.0 s, 2020.4 tps, lat 50.643 ms stddev 72.420 2024-04-02T14:43:42.432 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:43:42 smithi177 ceph-mon[36431]: pgmap v271: 97 pgs: 97 active+clean; 4.6 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 184 KiB/s rd, 36 MiB/s wr, 2.01k op/s 2024-04-02T14:43:42.434 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:43:42 smithi090 ceph-mon[31674]: pgmap v271: 97 pgs: 97 active+clean; 4.6 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 184 KiB/s rd, 36 MiB/s wr, 2.01k op/s 2024-04-02T14:43:42.606 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:43:42 smithi092 ceph-mon[36497]: pgmap v271: 97 pgs: 97 active+clean; 4.6 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 184 KiB/s rd, 36 MiB/s wr, 2.01k op/s 2024-04-02T14:43:44.432 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:43:44 smithi177 ceph-mon[36431]: pgmap v272: 97 pgs: 97 active+clean; 4.6 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 183 KiB/s rd, 34 MiB/s wr, 1.90k op/s 2024-04-02T14:43:44.434 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:43:44 smithi090 ceph-mon[31674]: pgmap v272: 97 pgs: 97 active+clean; 4.6 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 183 KiB/s rd, 34 MiB/s wr, 1.90k op/s 2024-04-02T14:43:44.606 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:43:44 smithi092 ceph-mon[36497]: pgmap v272: 97 pgs: 97 active+clean; 4.6 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 183 KiB/s rd, 34 MiB/s wr, 1.90k op/s 2024-04-02T14:43:45.052 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-02T14:43:45.052 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-02T14:43:45.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.307+0000 7f9b32000640 1 -- 172.21.15.90:0/3351187263 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9b2c108310 msgr2=0x7f9b2c1086f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:45.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.307+0000 7f9b32000640 1 --2- 172.21.15.90:0/3351187263 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9b2c108310 0x7f9b2c1086f0 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f9b1c009900 tx=0x7f9b1c02bc30 comp rx=0 tx=0).stop 2024-04-02T14:43:45.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.308+0000 7f9b32000640 1 -- 172.21.15.90:0/3351187263 shutdown_connections 2024-04-02T14:43:45.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.308+0000 7f9b32000640 1 --2- 172.21.15.90:0/3351187263 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9b2c10d3e0 0x7f9b2c114fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:45.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.308+0000 7f9b32000640 1 --2- 172.21.15.90:0/3351187263 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9b2c108cc0 0x7f9b2c10cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:45.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.308+0000 7f9b32000640 1 --2- 172.21.15.90:0/3351187263 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9b2c108310 0x7f9b2c1086f0 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:45.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.308+0000 7f9b32000640 1 -- 172.21.15.90:0/3351187263 >> 172.21.15.90:0/3351187263 conn(0x7f9b2c076040 msgr2=0x7f9b2c078460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:43:45.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.308+0000 7f9b32000640 1 -- 172.21.15.90:0/3351187263 shutdown_connections 2024-04-02T14:43:45.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.308+0000 7f9b32000640 1 -- 172.21.15.90:0/3351187263 wait complete. 2024-04-02T14:43:45.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.310+0000 7f9b32000640 1 Processor -- start 2024-04-02T14:43:45.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.310+0000 7f9b32000640 1 -- start start 2024-04-02T14:43:45.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.310+0000 7f9b32000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9b2c108310 0x7f9b2c1a46d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:45.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.310+0000 7f9b32000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9b2c10d3e0 0x7f9b2c1a4c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:45.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.310+0000 7f9b32000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9b2c19e880 0x7f9b2c19ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:45.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.310+0000 7f9b32000640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f9b2c118560 con 0x7f9b2c10d3e0 2024-04-02T14:43:45.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.310+0000 7f9b32000640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f9b2c1183e0 con 0x7f9b2c108310 2024-04-02T14:43:45.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.310+0000 7f9b32000640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f9b2c1186e0 con 0x7f9b2c19e880 2024-04-02T14:43:45.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.311+0000 7f9b2be00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9b2c19e880 0x7f9b2c19ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:45.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.311+0000 7f9b2b400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9b2c108310 0x7f9b2c1a46d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:45.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.311+0000 7f9b2be00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9b2c19e880 0x7f9b2c19ed30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:52388/0 (socket says 172.21.15.90:52388) 2024-04-02T14:43:45.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.311+0000 7f9b2be00640 1 -- 172.21.15.90:0/83651650 learned_addr learned my addr 172.21.15.90:0/83651650 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:43:45.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.311+0000 7f9b23e00640 1 --2- 172.21.15.90:0/83651650 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9b2c10d3e0 0x7f9b2c1a4c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:45.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.311+0000 7f9b2be00640 1 -- 172.21.15.90:0/83651650 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9b2c108310 msgr2=0x7f9b2c1a46d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:45.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.311+0000 7f9b2be00640 1 --2- 172.21.15.90:0/83651650 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9b2c108310 0x7f9b2c1a46d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:45.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.311+0000 7f9b2be00640 1 -- 172.21.15.90:0/83651650 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9b2c10d3e0 msgr2=0x7f9b2c1a4c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:45.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.311+0000 7f9b2be00640 1 --2- 172.21.15.90:0/83651650 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9b2c10d3e0 0x7f9b2c1a4c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:45.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.311+0000 7f9b2be00640 1 -- 172.21.15.90:0/83651650 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b2c19f3a0 con 0x7f9b2c19e880 2024-04-02T14:43:45.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.311+0000 7f9b2b400640 1 --2- 172.21.15.90:0/83651650 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9b2c108310 0x7f9b2c1a46d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:43:45.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.311+0000 7f9b2be00640 1 --2- 172.21.15.90:0/83651650 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9b2c19e880 0x7f9b2c19ed30 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f9b2400b970 tx=0x7f9b2400be30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:43:45.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.312+0000 7f9b28c00640 1 -- 172.21.15.90:0/83651650 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b24019070 con 0x7f9b2c19e880 2024-04-02T14:43:45.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.312+0000 7f9b28c00640 1 -- 172.21.15.90:0/83651650 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9b2400cb40 con 0x7f9b2c19e880 2024-04-02T14:43:45.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.312+0000 7f9b28c00640 1 -- 172.21.15.90:0/83651650 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b24007650 con 0x7f9b2c19e880 2024-04-02T14:43:45.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.312+0000 7f9b32000640 1 -- 172.21.15.90:0/83651650 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9b2c19f690 con 0x7f9b2c19e880 2024-04-02T14:43:45.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.312+0000 7f9b32000640 1 -- 172.21.15.90:0/83651650 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9b2c1ab890 con 0x7f9b2c19e880 2024-04-02T14:43:45.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.316+0000 7f9b34dc6740 1 Processor -- start 2024-04-02T14:43:45.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.317+0000 7f9b28c00640 1 -- 172.21.15.90:0/83651650 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f9b2401c070 con 0x7f9b2c19e880 2024-04-02T14:43:45.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.318+0000 7f9b34dc6740 1 -- start start 2024-04-02T14:43:45.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.318+0000 7f9b34dc6740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564ecc8757b0 0x564ecc875b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:45.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.318+0000 7f9b34dc6740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564ecc8760d0 0x564ecc87e770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:45.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.318+0000 7f9b34dc6740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564ecc87ecb0 0x564ecc8810a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:45.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.318+0000 7f9b34dc6740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x564ecc7d9770 con 0x564ecc8757b0 2024-04-02T14:43:45.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.318+0000 7f9b34dc6740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x564ecc781700 con 0x564ecc87ecb0 2024-04-02T14:43:45.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.318+0000 7f9b34dc6740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x564ecc7d5720 con 0x564ecc8760d0 2024-04-02T14:43:45.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.319+0000 7f9b23e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564ecc8757b0 0x564ecc875b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:45.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.319+0000 7f9b2b400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564ecc8760d0 0x564ecc87e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:45.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.319+0000 7f9b23e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564ecc8757b0 0x564ecc875b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33500/0 (socket says 172.21.15.90:33500) 2024-04-02T14:43:45.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.319+0000 7f9b23e00640 1 -- 172.21.15.90:0/1781004032 learned_addr learned my addr 172.21.15.90:0/1781004032 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:43:45.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.319+0000 7f9b23e00640 1 --2- 172.21.15.90:0/1781004032 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564ecc87ecb0 0x564ecc8810a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:45.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.320+0000 7f9b23e00640 1 -- 172.21.15.90:0/1781004032 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564ecc8760d0 msgr2=0x564ecc87e770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:43:45.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.320+0000 7f9b23e00640 1 --2- 172.21.15.90:0/1781004032 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564ecc8760d0 0x564ecc87e770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:45.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.320+0000 7f9b23e00640 1 -- 172.21.15.90:0/1781004032 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564ecc87ecb0 msgr2=0x564ecc8810a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:43:45.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.320+0000 7f9b23e00640 1 --2- 172.21.15.90:0/1781004032 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564ecc87ecb0 0x564ecc8810a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:45.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.320+0000 7f9b23e00640 1 -- 172.21.15.90:0/1781004032 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564ecc5b77d0 con 0x564ecc8757b0 2024-04-02T14:43:45.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.320+0000 7f9b28c00640 1 --2- 172.21.15.90:0/83651650 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f9b100a3de0 0x7f9b100a62a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:45.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.320+0000 7f9b28c00640 1 -- 172.21.15.90:0/83651650 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8550+0+0 (secure 0 0 0) 0x7f9b240cfc50 con 0x7f9b2c19e880 2024-04-02T14:43:45.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.320+0000 7f9b2b400640 1 --2- 172.21.15.90:0/83651650 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f9b100a3de0 0x7f9b100a62a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:45.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.320+0000 7f9b23e00640 1 --2- 172.21.15.90:0/1781004032 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564ecc8757b0 0x564ecc875b90 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f9b14014920 tx=0x7f9b1400b410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e70ededee65c489b server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:43:45.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.321+0000 7f9b1be00640 1 -- 172.21.15.90:0/1781004032 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b1400d6e0 con 0x564ecc8757b0 2024-04-02T14:43:45.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.321+0000 7f9b1be00640 1 -- 172.21.15.90:0/1781004032 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9b1400dd60 con 0x564ecc8757b0 2024-04-02T14:43:45.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.321+0000 7f9b1be00640 1 -- 172.21.15.90:0/1781004032 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b14011ad0 con 0x564ecc8757b0 2024-04-02T14:43:45.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.322+0000 7f9b34dc6740 1 -- 172.21.15.90:0/1781004032 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564ecc8757b0 msgr2=0x564ecc875b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:45.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.322+0000 7f9b34dc6740 1 --2- 172.21.15.90:0/1781004032 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564ecc8757b0 0x564ecc875b90 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f9b14014920 tx=0x7f9b1400b410 comp rx=0 tx=0).stop 2024-04-02T14:43:45.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.322+0000 7f9b2b400640 1 --2- 172.21.15.90:0/83651650 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f9b100a3de0 0x7f9b100a62a0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f9b1c000c00 tx=0x7f9b1c02df90 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:43:45.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.323+0000 7f9b34dc6740 1 -- 172.21.15.90:0/1781004032 shutdown_connections 2024-04-02T14:43:45.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.323+0000 7f9b34dc6740 1 --2- 172.21.15.90:0/1781004032 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564ecc87ecb0 0x564ecc8810a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:45.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.323+0000 7f9b34dc6740 1 --2- 172.21.15.90:0/1781004032 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564ecc8760d0 0x564ecc87e770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:45.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.323+0000 7f9b34dc6740 1 --2- 172.21.15.90:0/1781004032 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564ecc8757b0 0x564ecc875b90 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:45.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.323+0000 7f9b34dc6740 1 -- 172.21.15.90:0/1781004032 >> 172.21.15.90:0/1781004032 conn(0x564ecc86fee0 msgr2=0x564ecc8702c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:43:45.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.323+0000 7f9b34dc6740 1 -- 172.21.15.90:0/1781004032 shutdown_connections 2024-04-02T14:43:45.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.323+0000 7f9b34dc6740 1 -- 172.21.15.90:0/1781004032 wait complete. 2024-04-02T14:43:45.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.324+0000 7f9b34dc6740 1 -- start start 2024-04-02T14:43:45.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.324+0000 7f9b34dc6740 1 Processor -- start 2024-04-02T14:43:45.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.325+0000 7f9b34dc6740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564ecc8757b0 0x564ecc87bce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:45.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.325+0000 7f9b34dc6740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564ecc8760d0 0x564ecc87c220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:45.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.325+0000 7f9b34dc6740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564ecc87ecb0 0x564ecc87c760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:45.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.325+0000 7f9b34dc6740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x564ecc881a40 con 0x564ecc8760d0 2024-04-02T14:43:45.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.325+0000 7f9b34dc6740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x564ecc794ac0 con 0x564ecc8757b0 2024-04-02T14:43:45.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.325+0000 7f9b34dc6740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x564ecc881bc0 con 0x564ecc87ecb0 2024-04-02T14:43:45.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.325+0000 7f9b34dc6740 10 client.0 _ll_register_callbacks cb 0x564ecc797180 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:43:45.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.325+0000 7f9b23e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564ecc8760d0 0x564ecc87c220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:45.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.325+0000 7f9b2b400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564ecc8757b0 0x564ecc87bce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:45.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.325+0000 7f9b2be00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564ecc87ecb0 0x564ecc87c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:45.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.325+0000 7f9b2b400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564ecc8757b0 0x564ecc87bce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:36172/0 (socket says 172.21.15.90:36172) 2024-04-02T14:43:45.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.325+0000 7f9b2b400640 1 -- 172.21.15.90:0/1973113623 learned_addr learned my addr 172.21.15.90:0/1973113623 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:43:45.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.326+0000 7f9b2b400640 1 -- 172.21.15.90:0/1973113623 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564ecc87ecb0 msgr2=0x564ecc87c760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:43:45.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.326+0000 7f9b2b400640 1 --2- 172.21.15.90:0/1973113623 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564ecc87ecb0 0x564ecc87c760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:45.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.326+0000 7f9b2b400640 1 -- 172.21.15.90:0/1973113623 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564ecc8760d0 msgr2=0x564ecc87c220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:43:45.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.326+0000 7f9b2b400640 1 --2- 172.21.15.90:0/1973113623 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564ecc8760d0 0x564ecc87c220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:45.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.326+0000 7f9b2b400640 1 -- 172.21.15.90:0/1973113623 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564ecc4ed3f0 con 0x564ecc8757b0 2024-04-02T14:43:45.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.326+0000 7f9b2b400640 1 --2- 172.21.15.90:0/1973113623 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564ecc8757b0 0x564ecc87bce0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f9b1c035000 tx=0x7f9b1c009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5f140d183e80bab5 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:43:45.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.326+0000 7f9b18c00640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:43:45.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.327+0000 7f9b18c00640 1 -- 172.21.15.90:0/1973113623 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b1c046070 con 0x564ecc8757b0 2024-04-02T14:43:45.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.327+0000 7f9b18c00640 1 -- 172.21.15.90:0/1973113623 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8550+0+0 (secure 0 0 0) 0x7f9b1c02c650 con 0x564ecc8757b0 2024-04-02T14:43:45.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.327+0000 7f9b34dc6740 1 -- 172.21.15.90:0/1973113623 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9b1400dd60 con 0x564ecc8757b0 2024-04-02T14:43:45.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.327+0000 7f9b18c00640 1 -- 172.21.15.90:0/1973113623 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f9b1c003e30 con 0x564ecc8757b0 2024-04-02T14:43:45.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.327+0000 7f9b34dc6740 10 client.28175 fetch_fsmap learned FSMap version 27 2024-04-02T14:43:45.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.327+0000 7f9b34dc6740 1 -- 172.21.15.90:0/1973113623 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x564ecc4ad0f0 con 0x564ecc8757b0 2024-04-02T14:43:45.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.327+0000 7f9b18c00640 1 -- 172.21.15.90:0/1973113623 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9b1400dd60 con 0x564ecc8757b0 2024-04-02T14:43:45.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.327+0000 7f9b18c00640 1 -- 172.21.15.90:0/1973113623 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f9b1c0346b0 con 0x564ecc8757b0 2024-04-02T14:43:45.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.327+0000 7f9b34dc6740 10 client.28175 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:43:45.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.327+0000 7f9b34dc6740 10 client.28175 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:43:45.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.327+0000 7f9b34dc6740 20 client.28175 populate_metadata read hostname 'smithi090' 2024-04-02T14:43:45.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.327+0000 7f9b34dc6740 1 --2- 172.21.15.90:0/1973113623 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x564ecc88e980 0x564ecc890e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:45.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.327+0000 7f9b34dc6740 4 client.28175 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:43:45.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.327+0000 7f9b34dc6740 1 -- 172.21.15.90:0/1973113623 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564ecc77a630 con 0x564ecc88e980 2024-04-02T14:43:45.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.328+0000 7f9b23e00640 1 --2- 172.21.15.90:0/1973113623 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x564ecc88e980 0x564ecc890e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:45.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.329+0000 7f9b23e00640 1 --2- 172.21.15.90:0/1973113623 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x564ecc88e980 0x564ecc890e20 crc :-1 s=READY pgs=238 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=575f57e48b8ab5c5 server_cookie=f77faefae176d590 in_seq=0 out_seq=0 2024-04-02T14:43:45.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.329+0000 7f9b18c00640 10 client.28175 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:43:45.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.330+0000 7f9b18c00640 1 -- 172.21.15.90:0/1973113623 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x564ecc77a630 con 0x564ecc88e980 2024-04-02T14:43:45.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.330+0000 7f9b18c00640 10 client.28175 handle_command_reply: tid=0 2024-04-02T14:43:45.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.331+0000 7f9b34dc6740 1 client.28175 shutdown 2024-04-02T14:43:45.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.331+0000 7f9b34dc6740 1 -- 172.21.15.90:0/1973113623 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564ecc8757b0 msgr2=0x564ecc87bce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:45.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.331+0000 7f9b34dc6740 1 --2- 172.21.15.90:0/1973113623 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564ecc8757b0 0x564ecc87bce0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f9b1c035000 tx=0x7f9b1c009260 comp rx=0 tx=0).stop 2024-04-02T14:43:45.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.331+0000 7f9b34dc6740 1 -- 172.21.15.90:0/1973113623 shutdown_connections 2024-04-02T14:43:45.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.331+0000 7f9b34dc6740 1 --2- 172.21.15.90:0/1973113623 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x564ecc88e980 0x564ecc890e20 crc :-1 s=READY pgs=238 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:45.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.331+0000 7f9b34dc6740 1 --2- 172.21.15.90:0/1973113623 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564ecc87ecb0 0x564ecc87c760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:45.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.331+0000 7f9b34dc6740 1 --2- 172.21.15.90:0/1973113623 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564ecc8760d0 0x564ecc87c220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:45.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.331+0000 7f9b34dc6740 1 --2- 172.21.15.90:0/1973113623 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564ecc8757b0 0x564ecc87bce0 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:45.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.331+0000 7f9b18c00640 0 client.28175 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:43:45.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.331+0000 7f9b34dc6740 1 -- 172.21.15.90:0/1973113623 >> 172.21.15.90:0/1973113623 conn(0x564ecc86fee0 msgr2=0x564ecc871c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:43:45.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.331+0000 7f9b34dc6740 1 -- 172.21.15.90:0/1973113623 shutdown_connections 2024-04-02T14:43:45.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.331+0000 7f9b34dc6740 1 -- 172.21.15.90:0/1973113623 wait complete. 2024-04-02T14:43:45.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.331+0000 7f9b34dc6740 20 client.28175 trim_cache size 0 max 16384 2024-04-02T14:43:45.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.355+0000 7f9b34dc6740 1 Processor -- start 2024-04-02T14:43:45.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.355+0000 7f9b34dc6740 1 -- start start 2024-04-02T14:43:45.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.356+0000 7f9b34dc6740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564ecc86fee0 0x564ecc87e410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:45.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.356+0000 7f9b34dc6740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564ecc8757b0 0x564ecc88a900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:45.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.356+0000 7f9b34dc6740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564ecc8760d0 0x564ecc8966f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:45.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.356+0000 7f9b34dc6740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f9b2c1b3f10 con 0x564ecc8760d0 2024-04-02T14:43:45.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.356+0000 7f9b34dc6740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f9b2c1b5a50 con 0x564ecc86fee0 2024-04-02T14:43:45.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.356+0000 7f9b34dc6740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f9b2c1b45e0 con 0x564ecc8757b0 2024-04-02T14:43:45.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.356+0000 7f9b2b400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564ecc86fee0 0x564ecc87e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:45.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.356+0000 7f9b2be00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564ecc8760d0 0x564ecc8966f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:45.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.356+0000 7f9b2b400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564ecc86fee0 0x564ecc87e410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:36186/0 (socket says 172.21.15.90:36186) 2024-04-02T14:43:45.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.356+0000 7f9b2b400640 1 -- 172.21.15.90:0/1218235329 learned_addr learned my addr 172.21.15.90:0/1218235329 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:43:45.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.356+0000 7f9b23e00640 1 --2- 172.21.15.90:0/1218235329 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564ecc8757b0 0x564ecc88a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:45.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.356+0000 7f9b2b400640 1 -- 172.21.15.90:0/1218235329 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564ecc8757b0 msgr2=0x564ecc88a900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:43:45.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.356+0000 7f9b2b400640 1 --2- 172.21.15.90:0/1218235329 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564ecc8757b0 0x564ecc88a900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:45.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.356+0000 7f9b2b400640 1 -- 172.21.15.90:0/1218235329 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564ecc8760d0 msgr2=0x564ecc8966f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:43:45.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.356+0000 7f9b2b400640 1 --2- 172.21.15.90:0/1218235329 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564ecc8760d0 0x564ecc8966f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:45.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.356+0000 7f9b2b400640 1 -- 172.21.15.90:0/1218235329 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564ecc8748a0 con 0x564ecc86fee0 2024-04-02T14:43:45.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.356+0000 7f9b23e00640 1 --2- 172.21.15.90:0/1218235329 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564ecc8757b0 0x564ecc88a900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:43:45.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.357+0000 7f9b2b400640 1 --2- 172.21.15.90:0/1218235329 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564ecc86fee0 0x564ecc87e410 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f9b1c034ad0 tx=0x7f9b1c02c2b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f894d4030386a2fc server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:43:45.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.357+0000 7f9b1b400640 1 -- 172.21.15.90:0/1218235329 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b1c02cb90 con 0x564ecc86fee0 2024-04-02T14:43:45.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.357+0000 7f9b1b400640 1 -- 172.21.15.90:0/1218235329 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9b1c034d40 con 0x564ecc86fee0 2024-04-02T14:43:45.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.357+0000 7f9b1b400640 1 -- 172.21.15.90:0/1218235329 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b1c04ccb0 con 0x564ecc86fee0 2024-04-02T14:43:45.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.358+0000 7f9b34dc6740 1 -- 172.21.15.90:0/1218235329 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564ecc86fee0 msgr2=0x564ecc87e410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:45.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.358+0000 7f9b34dc6740 1 --2- 172.21.15.90:0/1218235329 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564ecc86fee0 0x564ecc87e410 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f9b1c034ad0 tx=0x7f9b1c02c2b0 comp rx=0 tx=0).stop 2024-04-02T14:43:45.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.358+0000 7f9b34dc6740 1 -- 172.21.15.90:0/1218235329 shutdown_connections 2024-04-02T14:43:45.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.358+0000 7f9b34dc6740 1 --2- 172.21.15.90:0/1218235329 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564ecc8760d0 0x564ecc8966f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:45.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.358+0000 7f9b34dc6740 1 --2- 172.21.15.90:0/1218235329 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564ecc8757b0 0x564ecc88a900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:45.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.358+0000 7f9b34dc6740 1 --2- 172.21.15.90:0/1218235329 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564ecc86fee0 0x564ecc87e410 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:45.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.358+0000 7f9b34dc6740 1 -- 172.21.15.90:0/1218235329 >> 172.21.15.90:0/1218235329 conn(0x564ecc87a920 msgr2=0x564ecc871ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:43:45.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.358+0000 7f9b34dc6740 1 -- 172.21.15.90:0/1218235329 shutdown_connections 2024-04-02T14:43:45.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.358+0000 7f9b34dc6740 1 -- 172.21.15.90:0/1218235329 wait complete. 2024-04-02T14:43:45.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.358+0000 7f9b34dc6740 1 -- start start 2024-04-02T14:43:45.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.359+0000 7f9b34dc6740 1 Processor -- start 2024-04-02T14:43:45.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.360+0000 7f9b34dc6740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564ecc86fee0 0x564ecc887220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:45.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.360+0000 7f9b34dc6740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564ecc8757b0 0x564ecc8835a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:45.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.360+0000 7f9b34dc6740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564ecc8760d0 0x564ecc883ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:45.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.360+0000 7f9b34dc6740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f9b2c1b5380 con 0x564ecc8757b0 2024-04-02T14:43:45.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.360+0000 7f9b34dc6740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f9b2c1b4cb0 con 0x564ecc8760d0 2024-04-02T14:43:45.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.360+0000 7f9b34dc6740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f9b2c1b1f10 con 0x564ecc86fee0 2024-04-02T14:43:45.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.360+0000 7f9b34dc6740 10 client.0 _ll_register_callbacks cb 0x7f9b1c0044c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:43:45.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.360+0000 7f9b2be00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564ecc8760d0 0x564ecc883ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:45.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.360+0000 7f9b2be00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564ecc8760d0 0x564ecc883ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:36188/0 (socket says 172.21.15.90:36188) 2024-04-02T14:43:45.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.360+0000 7f9b2be00640 1 -- 172.21.15.90:0/3145394880 learned_addr learned my addr 172.21.15.90:0/3145394880 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:43:45.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.361+0000 7f9b23e00640 1 --2- 172.21.15.90:0/3145394880 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564ecc8757b0 0x564ecc8835a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:45.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.361+0000 7f9b2b400640 1 --2- 172.21.15.90:0/3145394880 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564ecc86fee0 0x564ecc887220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:45.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.361+0000 7f9b2be00640 1 -- 172.21.15.90:0/3145394880 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564ecc86fee0 msgr2=0x564ecc887220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:43:45.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.361+0000 7f9b2be00640 1 --2- 172.21.15.90:0/3145394880 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564ecc86fee0 0x564ecc887220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:45.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.361+0000 7f9b2be00640 1 -- 172.21.15.90:0/3145394880 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564ecc8757b0 msgr2=0x564ecc8835a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:43:45.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.361+0000 7f9b2be00640 1 --2- 172.21.15.90:0/3145394880 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564ecc8757b0 0x564ecc8835a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:45.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.361+0000 7f9b2be00640 1 -- 172.21.15.90:0/3145394880 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564ecc884310 con 0x564ecc8760d0 2024-04-02T14:43:45.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.362+0000 7f9b2b400640 1 --2- 172.21.15.90:0/3145394880 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564ecc86fee0 0x564ecc887220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:43:45.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.362+0000 7f9b23e00640 1 --2- 172.21.15.90:0/3145394880 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564ecc8757b0 0x564ecc8835a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:43:45.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.362+0000 7f9b2be00640 1 --2- 172.21.15.90:0/3145394880 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564ecc8760d0 0x564ecc883ae0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f9b240cd4f0 tx=0x7f9b24014d00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d6c3954da9f37f55 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:43:45.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.362+0000 7f9b18c00640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:43:45.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.362+0000 7f9b18c00640 1 -- 172.21.15.90:0/3145394880 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b240cd780 con 0x564ecc8760d0 2024-04-02T14:43:45.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.362+0000 7f9b18c00640 1 -- 172.21.15.90:0/3145394880 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8550+0+0 (secure 0 0 0) 0x7f9b240a44c0 con 0x564ecc8760d0 2024-04-02T14:43:45.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.362+0000 7f9b34dc6740 1 -- 172.21.15.90:0/3145394880 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9b1c034d40 con 0x564ecc8760d0 2024-04-02T14:43:45.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.362+0000 7f9b18c00640 1 -- 172.21.15.90:0/3145394880 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f9b240a46c0 con 0x564ecc8760d0 2024-04-02T14:43:45.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.362+0000 7f9b34dc6740 10 client.28181 fetch_fsmap learned FSMap version 27 2024-04-02T14:43:45.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.362+0000 7f9b34dc6740 1 -- 172.21.15.90:0/3145394880 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x564ecc906ff0 con 0x564ecc8760d0 2024-04-02T14:43:45.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.362+0000 7f9b18c00640 1 -- 172.21.15.90:0/3145394880 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9b1c034d40 con 0x564ecc8760d0 2024-04-02T14:43:45.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.363+0000 7f9b18c00640 1 -- 172.21.15.90:0/3145394880 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f9b2409b1e0 con 0x564ecc8760d0 2024-04-02T14:43:45.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.363+0000 7f9b34dc6740 10 client.28181 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:43:45.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.363+0000 7f9b34dc6740 10 client.28181 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:43:45.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.363+0000 7f9b34dc6740 20 client.28181 populate_metadata read hostname 'smithi090' 2024-04-02T14:43:45.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.363+0000 7f9b34dc6740 1 --2- 172.21.15.90:0/3145394880 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x564ecc87ecb0 0x564ecc9097b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:45.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.363+0000 7f9b34dc6740 4 client.28181 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-02T14:43:45.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.363+0000 7f9b34dc6740 1 -- 172.21.15.90:0/3145394880 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "scrub status"}) -- 0x564ecc7e6cb0 con 0x564ecc87ecb0 2024-04-02T14:43:45.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.363+0000 7f9b2b400640 1 --2- 172.21.15.90:0/3145394880 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x564ecc87ecb0 0x564ecc9097b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:45.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.363+0000 7f9b18c00640 10 client.28181 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:43:45.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.363+0000 7f9b2b400640 1 --2- 172.21.15.90:0/3145394880 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x564ecc87ecb0 0x564ecc9097b0 crc :-1 s=READY pgs=239 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b28544ab4cac6062 server_cookie=992e176cdd51f52d in_seq=0 out_seq=0 2024-04-02T14:43:45.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.364+0000 7f9b18c00640 1 -- 172.21.15.90:0/3145394880 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x564ecc7e6cb0 con 0x564ecc87ecb0 2024-04-02T14:43:45.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.364+0000 7f9b18c00640 10 client.28181 handle_command_reply: tid=0 2024-04-02T14:43:45.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.364+0000 7f9b34dc6740 1 client.28181 shutdown 2024-04-02T14:43:45.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.364+0000 7f9b34dc6740 1 -- 172.21.15.90:0/3145394880 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564ecc8760d0 msgr2=0x564ecc883ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:45.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.364+0000 7f9b34dc6740 1 --2- 172.21.15.90:0/3145394880 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564ecc8760d0 0x564ecc883ae0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f9b240cd4f0 tx=0x7f9b24014d00 comp rx=0 tx=0).stop 2024-04-02T14:43:45.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.364+0000 7f9b34dc6740 1 -- 172.21.15.90:0/3145394880 shutdown_connections 2024-04-02T14:43:45.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.364+0000 7f9b34dc6740 1 --2- 172.21.15.90:0/3145394880 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x564ecc87ecb0 0x564ecc9097b0 crc :-1 s=READY pgs=239 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:45.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.364+0000 7f9b34dc6740 1 --2- 172.21.15.90:0/3145394880 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564ecc8760d0 0x564ecc883ae0 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:45.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.364+0000 7f9b34dc6740 1 --2- 172.21.15.90:0/3145394880 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564ecc8757b0 0x564ecc8835a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:45.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.364+0000 7f9b34dc6740 1 --2- 172.21.15.90:0/3145394880 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564ecc86fee0 0x564ecc887220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:45.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.364+0000 7f9b34dc6740 1 -- 172.21.15.90:0/3145394880 >> 172.21.15.90:0/3145394880 conn(0x564ecc87a920 msgr2=0x564ecc86e6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:43:45.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.364+0000 7f9b18c00640 0 client.28181 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:43:45.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.364+0000 7f9b34dc6740 1 -- 172.21.15.90:0/3145394880 shutdown_connections 2024-04-02T14:43:45.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.364+0000 7f9b34dc6740 1 -- 172.21.15.90:0/3145394880 wait complete. 2024-04-02T14:43:45.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.364+0000 7f9b34dc6740 20 client.28181 trim_cache size 0 max 16384 2024-04-02T14:43:45.364 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-02T14:43:45.365 INFO:teuthology.orchestra.run.smithi090.stdout: "status": "no active scrubs running", 2024-04-02T14:43:45.365 INFO:teuthology.orchestra.run.smithi090.stdout: "scrubs": {} 2024-04-02T14:43:45.365 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-02T14:43:45.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.365+0000 7f9b32000640 1 -- 172.21.15.90:0/83651650 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f9b100a3de0 msgr2=0x7f9b100a62a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:45.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.365+0000 7f9b32000640 1 --2- 172.21.15.90:0/83651650 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f9b100a3de0 0x7f9b100a62a0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f9b1c000c00 tx=0x7f9b1c02df90 comp rx=0 tx=0).stop 2024-04-02T14:43:45.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.365+0000 7f9b32000640 1 -- 172.21.15.90:0/83651650 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9b2c19e880 msgr2=0x7f9b2c19ed30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:45.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.365+0000 7f9b32000640 1 --2- 172.21.15.90:0/83651650 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9b2c19e880 0x7f9b2c19ed30 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f9b2400b970 tx=0x7f9b2400be30 comp rx=0 tx=0).stop 2024-04-02T14:43:45.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.366+0000 7f9b32000640 1 -- 172.21.15.90:0/83651650 shutdown_connections 2024-04-02T14:43:45.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.366+0000 7f9b32000640 1 --2- 172.21.15.90:0/83651650 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f9b100a3de0 0x7f9b100a62a0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:45.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.366+0000 7f9b32000640 1 --2- 172.21.15.90:0/83651650 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9b2c19e880 0x7f9b2c19ed30 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:45.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.366+0000 7f9b32000640 1 --2- 172.21.15.90:0/83651650 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9b2c10d3e0 0x7f9b2c1a4c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:45.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.366+0000 7f9b32000640 1 --2- 172.21.15.90:0/83651650 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9b2c108310 0x7f9b2c1a46d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:45.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.366+0000 7f9b32000640 1 -- 172.21.15.90:0/83651650 >> 172.21.15.90:0/83651650 conn(0x7f9b2c076040 msgr2=0x7f9b2c078380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:43:45.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.366+0000 7f9b32000640 1 -- 172.21.15.90:0/83651650 shutdown_connections 2024-04-02T14:43:45.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.367+0000 7f9b32000640 1 -- 172.21.15.90:0/83651650 wait complete. 2024-04-02T14:43:45.384 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-02T14:43:45.384 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T14:43:45.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.646+0000 7ffb16800640 1 -- 172.21.15.90:0/565617502 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ffb101022b0 msgr2=0x7ffb1010e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:45.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.646+0000 7ffb16800640 1 --2- 172.21.15.90:0/565617502 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ffb101022b0 0x7ffb1010e7c0 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7ffafc009860 tx=0x7ffafc02bb70 comp rx=0 tx=0).stop 2024-04-02T14:43:45.646 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.649+0000 7ffb16800640 1 -- 172.21.15.90:0/565617502 shutdown_connections 2024-04-02T14:43:45.646 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.649+0000 7ffb16800640 1 --2- 172.21.15.90:0/565617502 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ffb101022b0 0x7ffb1010e7c0 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:45.646 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.649+0000 7ffb16800640 1 --2- 172.21.15.90:0/565617502 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffb101018f0 0x7ffb10101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:45.646 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.649+0000 7ffb16800640 1 --2- 172.21.15.90:0/565617502 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffb10107900 0x7ffb10107ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:45.646 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.649+0000 7ffb16800640 1 -- 172.21.15.90:0/565617502 >> 172.21.15.90:0/565617502 conn(0x7ffb100fd3e0 msgr2=0x7ffb100ff800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:43:45.646 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.649+0000 7ffb16800640 1 -- 172.21.15.90:0/565617502 shutdown_connections 2024-04-02T14:43:45.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.650+0000 7ffb16800640 1 -- 172.21.15.90:0/565617502 wait complete. 2024-04-02T14:43:45.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.651+0000 7ffb16800640 1 Processor -- start 2024-04-02T14:43:45.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.651+0000 7ffb16800640 1 -- start start 2024-04-02T14:43:45.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.652+0000 7ffb16800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffb101018f0 0x7ffb1019a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:45.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.652+0000 7ffb16800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ffb101022b0 0x7ffb1019a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:45.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.652+0000 7ffb16800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffb10107900 0x7ffb1019ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:45.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.652+0000 7ffb16800640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7ffb10111d10 con 0x7ffb101018f0 2024-04-02T14:43:45.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.652+0000 7ffb16800640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7ffb10111b90 con 0x7ffb101022b0 2024-04-02T14:43:45.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.652+0000 7ffb16800640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7ffb10111e90 con 0x7ffb10107900 2024-04-02T14:43:45.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.652+0000 7ffb14a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ffb101022b0 0x7ffb1019a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:45.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.652+0000 7ffb14a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ffb101022b0 0x7ffb1019a940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:36206/0 (socket says 172.21.15.90:36206) 2024-04-02T14:43:45.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.652+0000 7ffb15e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffb10107900 0x7ffb1019ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:45.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.652+0000 7ffb14a00640 1 -- 172.21.15.90:0/2848957490 learned_addr learned my addr 172.21.15.90:0/2848957490 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:43:45.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.652+0000 7ffb15400640 1 --2- 172.21.15.90:0/2848957490 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffb101018f0 0x7ffb1019a400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:45.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.653+0000 7ffb14a00640 1 -- 172.21.15.90:0/2848957490 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffb10107900 msgr2=0x7ffb1019ecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:45.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.653+0000 7ffb14a00640 1 --2- 172.21.15.90:0/2848957490 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffb10107900 0x7ffb1019ecc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:45.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.653+0000 7ffb14a00640 1 -- 172.21.15.90:0/2848957490 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffb101018f0 msgr2=0x7ffb1019a400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:45.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.653+0000 7ffb14a00640 1 --2- 172.21.15.90:0/2848957490 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffb101018f0 0x7ffb1019a400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:45.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.653+0000 7ffb14a00640 1 -- 172.21.15.90:0/2848957490 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb1019f3f0 con 0x7ffb101022b0 2024-04-02T14:43:45.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.653+0000 7ffb15e00640 1 --2- 172.21.15.90:0/2848957490 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffb10107900 0x7ffb1019ecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:43:45.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.656+0000 7ffb14a00640 1 --2- 172.21.15.90:0/2848957490 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ffb101022b0 0x7ffb1019a940 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7ffb0400d950 tx=0x7ffb0400de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:43:45.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.656+0000 7ffb0e000640 1 -- 172.21.15.90:0/2848957490 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb04017070 con 0x7ffb101022b0 2024-04-02T14:43:45.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.656+0000 7ffb16800640 1 -- 172.21.15.90:0/2848957490 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffb1019f6e0 con 0x7ffb101022b0 2024-04-02T14:43:45.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.656+0000 7ffb0e000640 1 -- 172.21.15.90:0/2848957490 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffb04009c30 con 0x7ffb101022b0 2024-04-02T14:43:45.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.656+0000 7ffb0e000640 1 -- 172.21.15.90:0/2848957490 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb0400f6b0 con 0x7ffb101022b0 2024-04-02T14:43:45.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.656+0000 7ffb16800640 1 -- 172.21.15.90:0/2848957490 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffb101a6f70 con 0x7ffb101022b0 2024-04-02T14:43:45.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.662+0000 7ffb03400640 1 -- 172.21.15.90:0/2848957490 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffb10103030 con 0x7ffb101022b0 2024-04-02T14:43:45.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.664+0000 7ffb0e000640 1 -- 172.21.15.90:0/2848957490 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7ffb040109d0 con 0x7ffb101022b0 2024-04-02T14:43:45.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.666+0000 7ffb0e000640 1 --2- 172.21.15.90:0/2848957490 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7ffaf80a3e20 0x7ffaf80a62e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:45.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.666+0000 7ffb0e000640 1 -- 172.21.15.90:0/2848957490 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8550+0+0 (secure 0 0 0) 0x7ffb040cff70 con 0x7ffb101022b0 2024-04-02T14:43:45.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.667+0000 7ffb15400640 1 --2- 172.21.15.90:0/2848957490 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7ffaf80a3e20 0x7ffaf80a62e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:45.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.667+0000 7ffb15400640 1 --2- 172.21.15.90:0/2848957490 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7ffaf80a3e20 0x7ffaf80a62e0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7ffb0800a400 tx=0x7ffb0800d040 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:43:45.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:45.673+0000 7ffb0e000640 1 -- 172.21.15.90:0/2848957490 <== mon.2 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7ffb04009de0 con 0x7ffb101022b0 2024-04-02T14:43:46.034 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.037+0000 7ffb03400640 1 -- 172.21.15.90:0/2848957490 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ffb10068810 con 0x7ffb101022b0 2024-04-02T14:43:46.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.039+0000 7ffb0e000640 1 -- 172.21.15.90:0/2848957490 <== mon.2 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11927 (secure 0 0 0) 0x7ffb0409bf40 con 0x7ffb101022b0 2024-04-02T14:43:46.036 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-02T14:43:46.036 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14412,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6837/2852160309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6836","nonce":2852160309},{"type":"v1","addr":"172.21.15.90:6837","nonce":2852160309}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14478,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6841/1919376858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6840","nonce":1919376858},{"type":"v1","addr":"172.21.15.90:6841","nonce":1919376858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-02T14:32:22.319083+0000","modified":"2024-04-02T14:34:52.266152+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24590,"mds_1":24560,"mds_2":24547,"mds_3":24539,"mds_4":24523},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24454":{"gid":24454,"name":"l","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6833/1564319431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":1564319431},{"type":"v1","addr":"172.21.15.177:6833","nonce":1564319431}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.92:6833/1570337372","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":1570337372},{"type":"v1","addr":"172.21.15.92:6833","nonce":1570337372}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"j","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.90:6835/1277124891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6834","nonce":1277124891},{"type":"v1","addr":"172.21.15.90:6835","nonce":1277124891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.92:6835/860721164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":860721164},{"type":"v1","addr":"172.21.15.92:6835","nonce":860721164}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24511":{"gid":24511,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6835/3177244988","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":3177244988},{"type":"v1","addr":"172.21.15.177:6835","nonce":3177244988}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"c","rank":4,"incarnation":25,"state":"up:active","state_seq":9,"addr":"172.21.15.177:6837/4223414858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":4223414858},{"type":"v1","addr":"172.21.15.177:6837","nonce":4223414858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24539":{"gid":24539,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":8,"addr":"172.21.15.92:6837/1270635939","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":1270635939},{"type":"v1","addr":"172.21.15.92:6837","nonce":1270635939}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24547":{"gid":24547,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":7,"addr":"172.21.15.177:6839/664895663","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":664895663},{"type":"v1","addr":"172.21.15.177:6839","nonce":664895663}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"g","rank":1,"incarnation":19,"state":"up:active","state_seq":5,"addr":"172.21.15.90:6839/1498915870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6838","nonce":1498915870},{"type":"v1","addr":"172.21.15.90:6839","nonce":1498915870}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24590":{"gid":24590,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.92:6839/1054888219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":1054888219},{"type":"v1","addr":"172.21.15.92:6839","nonce":1054888219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24590,"qdb_cluster":[24547,24590,24523,24539,24560]},"id":1}]} 2024-04-02T14:43:46.037 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 27 2024-04-02T14:43:46.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.044+0000 7ffb03400640 1 -- 172.21.15.90:0/2848957490 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7ffaf80a3e20 msgr2=0x7ffaf80a62e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:46.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.044+0000 7ffb03400640 1 --2- 172.21.15.90:0/2848957490 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7ffaf80a3e20 0x7ffaf80a62e0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7ffb0800a400 tx=0x7ffb0800d040 comp rx=0 tx=0).stop 2024-04-02T14:43:46.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.044+0000 7ffb03400640 1 -- 172.21.15.90:0/2848957490 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ffb101022b0 msgr2=0x7ffb1019a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:46.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.044+0000 7ffb03400640 1 --2- 172.21.15.90:0/2848957490 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ffb101022b0 0x7ffb1019a940 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7ffb0400d950 tx=0x7ffb0400de10 comp rx=0 tx=0).stop 2024-04-02T14:43:46.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.044+0000 7ffb03400640 1 -- 172.21.15.90:0/2848957490 shutdown_connections 2024-04-02T14:43:46.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.044+0000 7ffb03400640 1 --2- 172.21.15.90:0/2848957490 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7ffaf80a3e20 0x7ffaf80a62e0 secure :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7ffb0800a400 tx=0x7ffb0800d040 comp rx=0 tx=0).stop 2024-04-02T14:43:46.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.044+0000 7ffb03400640 1 --2- 172.21.15.90:0/2848957490 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffb10107900 0x7ffb1019ecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:46.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.044+0000 7ffb03400640 1 --2- 172.21.15.90:0/2848957490 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ffb101022b0 0x7ffb1019a940 secure :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7ffb0400d950 tx=0x7ffb0400de10 comp rx=0 tx=0).stop 2024-04-02T14:43:46.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.044+0000 7ffb03400640 1 --2- 172.21.15.90:0/2848957490 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffb101018f0 0x7ffb1019a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:46.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.044+0000 7ffb03400640 1 -- 172.21.15.90:0/2848957490 >> 172.21.15.90:0/2848957490 conn(0x7ffb100fd3e0 msgr2=0x7ffb1010d450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:43:46.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.045+0000 7ffb03400640 1 -- 172.21.15.90:0/2848957490 shutdown_connections 2024-04-02T14:43:46.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.046+0000 7ffb03400640 1 -- 172.21.15.90:0/2848957490 wait complete. 2024-04-02T14:43:46.073 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-04-02T14:43:46.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.320+0000 7fc7d9a00640 1 -- 172.21.15.90:0/1384196087 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc7d4102450 msgr2=0x7fc7d410e960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:46.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.320+0000 7fc7d9a00640 1 --2- 172.21.15.90:0/1384196087 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc7d4102450 0x7fc7d410e960 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7fc7cc009be0 tx=0x7fc7cc02bef0 comp rx=0 tx=0).stop 2024-04-02T14:43:46.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.320+0000 7fc7d9a00640 1 -- 172.21.15.90:0/1384196087 shutdown_connections 2024-04-02T14:43:46.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.320+0000 7fc7d9a00640 1 --2- 172.21.15.90:0/1384196087 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc7d4102450 0x7fc7d410e960 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:46.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.320+0000 7fc7d9a00640 1 --2- 172.21.15.90:0/1384196087 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc7d4101a90 0x7fc7d4101f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:46.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.320+0000 7fc7d9a00640 1 --2- 172.21.15.90:0/1384196087 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc7d4107aa0 0x7fc7d4107e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:46.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.320+0000 7fc7d9a00640 1 -- 172.21.15.90:0/1384196087 >> 172.21.15.90:0/1384196087 conn(0x7fc7d4078f00 msgr2=0x7fc7d4079310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:43:46.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.322+0000 7fc7d9a00640 1 -- 172.21.15.90:0/1384196087 shutdown_connections 2024-04-02T14:43:46.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.322+0000 7fc7d9a00640 1 -- 172.21.15.90:0/1384196087 wait complete. 2024-04-02T14:43:46.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.324+0000 7fc7d9a00640 1 Processor -- start 2024-04-02T14:43:46.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.324+0000 7fc7d9a00640 1 -- start start 2024-04-02T14:43:46.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.324+0000 7fc7d9a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc7d4101a90 0x7fc7d419a520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:46.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.324+0000 7fc7d9a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc7d4102450 0x7fc7d419aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:46.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.324+0000 7fc7d9a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc7d4107aa0 0x7fc7d419ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:46.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.324+0000 7fc7d9a00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fc7d4111ed0 con 0x7fc7d4101a90 2024-04-02T14:43:46.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.324+0000 7fc7d9a00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fc7d4111d50 con 0x7fc7d4107aa0 2024-04-02T14:43:46.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.324+0000 7fc7d9a00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fc7d4112050 con 0x7fc7d4102450 2024-04-02T14:43:46.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.325+0000 7fc7d2a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc7d4101a90 0x7fc7d419a520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:46.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.325+0000 7fc7d3400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc7d4107aa0 0x7fc7d419ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:46.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.325+0000 7fc7d3400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc7d4107aa0 0x7fc7d419ede0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:36222/0 (socket says 172.21.15.90:36222) 2024-04-02T14:43:46.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.325+0000 7fc7d3400640 1 -- 172.21.15.90:0/1070072528 learned_addr learned my addr 172.21.15.90:0/1070072528 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:43:46.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.325+0000 7fc7d2000640 1 --2- 172.21.15.90:0/1070072528 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc7d4102450 0x7fc7d419aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:46.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.325+0000 7fc7d3400640 1 -- 172.21.15.90:0/1070072528 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc7d4102450 msgr2=0x7fc7d419aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:46.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.325+0000 7fc7d3400640 1 --2- 172.21.15.90:0/1070072528 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc7d4102450 0x7fc7d419aa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:46.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.325+0000 7fc7d3400640 1 -- 172.21.15.90:0/1070072528 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc7d4101a90 msgr2=0x7fc7d419a520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:46.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.325+0000 7fc7d3400640 1 --2- 172.21.15.90:0/1070072528 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc7d4101a90 0x7fc7d419a520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:46.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.325+0000 7fc7d3400640 1 -- 172.21.15.90:0/1070072528 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7d419f510 con 0x7fc7d4107aa0 2024-04-02T14:43:46.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.326+0000 7fc7d3400640 1 --2- 172.21.15.90:0/1070072528 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc7d4107aa0 0x7fc7d419ede0 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7fc7cc033420 tx=0x7fc7cc033720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:43:46.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.326+0000 7fc7cb400640 1 -- 172.21.15.90:0/1070072528 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7cc043020 con 0x7fc7d4107aa0 2024-04-02T14:43:46.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.326+0000 7fc7cb400640 1 -- 172.21.15.90:0/1070072528 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc7cc003e50 con 0x7fc7d4107aa0 2024-04-02T14:43:46.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.326+0000 7fc7cb400640 1 -- 172.21.15.90:0/1070072528 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7cc03e6b0 con 0x7fc7d4107aa0 2024-04-02T14:43:46.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.326+0000 7fc7d9a00640 1 -- 172.21.15.90:0/1070072528 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc7d419f7a0 con 0x7fc7d4107aa0 2024-04-02T14:43:46.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.326+0000 7fc7d9a00640 1 -- 172.21.15.90:0/1070072528 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7d41a7090 con 0x7fc7d4107aa0 2024-04-02T14:43:46.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.332+0000 7fc7dc76d740 1 Processor -- start 2024-04-02T14:43:46.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.336+0000 7fc7cb400640 1 -- 172.21.15.90:0/1070072528 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fc7cc046020 con 0x7fc7d4107aa0 2024-04-02T14:43:46.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.336+0000 7fc7dc76d740 1 -- start start 2024-04-02T14:43:46.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.336+0000 7fc7dc76d740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55734dd6e200 0x55734dd6cf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:46.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.336+0000 7fc7dc76d740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55734dd6d520 0x55734dd6d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:46.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.336+0000 7fc7dc76d740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55734dd76890 0x55734dd78c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:46.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.336+0000 7fc7dc76d740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55734dca5400 con 0x55734dd76890 2024-04-02T14:43:46.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.336+0000 7fc7dc76d740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55734dc89f10 con 0x55734dd6e200 2024-04-02T14:43:46.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.337+0000 7fc7dc76d740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55734dcc96f0 con 0x55734dd6d520 2024-04-02T14:43:46.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.337+0000 7fc7d2a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55734dd6d520 0x55734dd6d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:46.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.337+0000 7fc7d2000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55734dd6e200 0x55734dd6cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:46.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.337+0000 7fc7d2a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55734dd6d520 0x55734dd6d9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:52476/0 (socket says 172.21.15.90:52476) 2024-04-02T14:43:46.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.337+0000 7fc7d2a00640 1 -- 172.21.15.90:0/1323629419 learned_addr learned my addr 172.21.15.90:0/1323629419 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:43:46.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.337+0000 7fc7d2000640 1 --2- 172.21.15.90:0/1323629419 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55734dd76890 0x55734dd78c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:46.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.337+0000 7fc7d2a00640 1 -- 172.21.15.90:0/1323629419 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55734dd6e200 msgr2=0x55734dd6cf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:43:46.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.337+0000 7fc7d2a00640 1 --2- 172.21.15.90:0/1323629419 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55734dd6e200 0x55734dd6cf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:46.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.337+0000 7fc7d2a00640 1 -- 172.21.15.90:0/1323629419 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55734dd76890 msgr2=0x55734dd78c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:43:46.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.337+0000 7fc7d2a00640 1 --2- 172.21.15.90:0/1323629419 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55734dd76890 0x55734dd78c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:46.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.337+0000 7fc7d2a00640 1 -- 172.21.15.90:0/1323629419 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55734d9dd3f0 con 0x55734dd6d520 2024-04-02T14:43:46.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.338+0000 7fc7d2a00640 1 --2- 172.21.15.90:0/1323629419 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55734dd6d520 0x55734dd6d9a0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fc7c4009c50 tx=0x7fc7c4011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=231990a7ba6a2e38 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:43:46.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.338+0000 7fc7c3e00640 1 -- 172.21.15.90:0/1323629419 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7c401e070 con 0x55734dd6d520 2024-04-02T14:43:46.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.338+0000 7fc7c3e00640 1 -- 172.21.15.90:0/1323629419 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc7c4019b10 con 0x55734dd6d520 2024-04-02T14:43:46.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.338+0000 7fc7c3e00640 1 -- 172.21.15.90:0/1323629419 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7c400eb10 con 0x55734dd6d520 2024-04-02T14:43:46.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.338+0000 7fc7cb400640 1 --2- 172.21.15.90:0/1070072528 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fc7b80a3da0 0x7fc7b80a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:46.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.339+0000 7fc7cb400640 1 -- 172.21.15.90:0/1070072528 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8550+0+0 (secure 0 0 0) 0x7fc7cc0f2d60 con 0x7fc7d4107aa0 2024-04-02T14:43:46.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.339+0000 7fc7d2000640 1 --2- 172.21.15.90:0/1070072528 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fc7b80a3da0 0x7fc7b80a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:46.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.339+0000 7fc7dc76d740 1 -- 172.21.15.90:0/1323629419 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55734dd6d520 msgr2=0x55734dd6d9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:46.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.339+0000 7fc7dc76d740 1 --2- 172.21.15.90:0/1323629419 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55734dd6d520 0x55734dd6d9a0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fc7c4009c50 tx=0x7fc7c4011810 comp rx=0 tx=0).stop 2024-04-02T14:43:46.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.339+0000 7fc7dc76d740 1 -- 172.21.15.90:0/1323629419 shutdown_connections 2024-04-02T14:43:46.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.339+0000 7fc7dc76d740 1 --2- 172.21.15.90:0/1323629419 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55734dd76890 0x55734dd78c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:46.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.339+0000 7fc7dc76d740 1 --2- 172.21.15.90:0/1323629419 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55734dd6d520 0x55734dd6d9a0 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:46.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.339+0000 7fc7dc76d740 1 --2- 172.21.15.90:0/1323629419 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55734dd6e200 0x55734dd6cf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:46.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.339+0000 7fc7dc76d740 1 -- 172.21.15.90:0/1323629419 >> 172.21.15.90:0/1323629419 conn(0x55734dd2ba50 msgr2=0x55734dd67f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:43:46.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.340+0000 7fc7dc76d740 1 -- 172.21.15.90:0/1323629419 shutdown_connections 2024-04-02T14:43:46.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.340+0000 7fc7d2000640 1 --2- 172.21.15.90:0/1070072528 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fc7b80a3da0 0x7fc7b80a6260 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fc7bc005d00 tx=0x7fc7bc00b390 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:43:46.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.340+0000 7fc7dc76d740 1 -- 172.21.15.90:0/1323629419 wait complete. 2024-04-02T14:43:46.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.341+0000 7fc7dc76d740 1 -- start start 2024-04-02T14:43:46.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.341+0000 7fc7dc76d740 1 Processor -- start 2024-04-02T14:43:46.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.342+0000 7fc7dc76d740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55734dd6d520 0x55734dd739a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:46.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.342+0000 7fc7dc76d740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55734dd6e200 0x55734dd73ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:46.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.342+0000 7fc7dc76d740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55734dd76890 0x55734dd74420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:46.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.343+0000 7fc7d2a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55734dd6d520 0x55734dd739a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:46.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.343+0000 7fc7dc76d740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55734dcc56a0 con 0x55734dd6e200 2024-04-02T14:43:46.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.343+0000 7fc7dc76d740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55734dc71980 con 0x55734dd6d520 2024-04-02T14:43:46.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.343+0000 7fc7dc76d740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55734dc84cb0 con 0x55734dd76890 2024-04-02T14:43:46.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.343+0000 7fc7dc76d740 10 client.0 _ll_register_callbacks cb 0x55734dc846a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:43:46.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.343+0000 7fc7d3400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55734dd76890 0x55734dd74420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:46.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.343+0000 7fc7d3400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55734dd76890 0x55734dd74420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:52478/0 (socket says 172.21.15.90:52478) 2024-04-02T14:43:46.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.343+0000 7fc7d3400640 1 -- 172.21.15.90:0/164685592 learned_addr learned my addr 172.21.15.90:0/164685592 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:43:46.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.343+0000 7fc7d2000640 1 --2- 172.21.15.90:0/164685592 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55734dd6e200 0x55734dd73ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:46.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.344+0000 7fc7d3400640 1 -- 172.21.15.90:0/164685592 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55734dd6d520 msgr2=0x55734dd739a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:43:46.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.344+0000 7fc7d3400640 1 --2- 172.21.15.90:0/164685592 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55734dd6d520 0x55734dd739a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:46.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.344+0000 7fc7d3400640 1 -- 172.21.15.90:0/164685592 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55734dd6e200 msgr2=0x55734dd73ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:43:46.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.344+0000 7fc7d3400640 1 --2- 172.21.15.90:0/164685592 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55734dd6e200 0x55734dd73ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:46.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.344+0000 7fc7d3400640 1 -- 172.21.15.90:0/164685592 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55734daa77d0 con 0x55734dd76890 2024-04-02T14:43:46.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.344+0000 7fc7d2000640 1 --2- 172.21.15.90:0/164685592 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55734dd6e200 0x55734dd73ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:43:46.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.344+0000 7fc7d2a00640 1 --2- 172.21.15.90:0/164685592 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55734dd6d520 0x55734dd739a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:43:46.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.344+0000 7fc7d3400640 1 --2- 172.21.15.90:0/164685592 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55734dd76890 0x55734dd74420 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7fc7cc0c74d0 tx=0x7fc7cc0be8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=867a8c5915407236 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:43:46.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.344+0000 7fc7c0c00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:43:46.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.345+0000 7fc7c0c00640 1 -- 172.21.15.90:0/164685592 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7cc0bed90 con 0x55734dd76890 2024-04-02T14:43:46.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.345+0000 7fc7c0c00640 1 -- 172.21.15.90:0/164685592 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8550+0+0 (secure 0 0 0) 0x7fc7cc0f06b0 con 0x55734dd76890 2024-04-02T14:43:46.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.345+0000 7fc7dc76d740 1 -- 172.21.15.90:0/164685592 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc7c4019b10 con 0x55734dd76890 2024-04-02T14:43:46.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.346+0000 7fc7c0c00640 1 -- 172.21.15.90:0/164685592 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fc7cc0f08b0 con 0x55734dd76890 2024-04-02T14:43:46.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.346+0000 7fc7dc76d740 10 client.28033 fetch_fsmap learned FSMap version 27 2024-04-02T14:43:46.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.346+0000 7fc7dc76d740 1 -- 172.21.15.90:0/164685592 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55734dd8df60 con 0x55734dd76890 2024-04-02T14:43:46.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.347+0000 7fc7c0c00640 1 -- 172.21.15.90:0/164685592 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc7c4019b10 con 0x55734dd76890 2024-04-02T14:43:46.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.347+0000 7fc7c0c00640 1 -- 172.21.15.90:0/164685592 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fc7cc0f0b80 con 0x55734dd76890 2024-04-02T14:43:46.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.347+0000 7fc7dc76d740 10 client.28033 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:43:46.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.347+0000 7fc7dc76d740 10 client.28033 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.c 2024-04-02T14:43:46.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.347+0000 7fc7dc76d740 20 client.28033 populate_metadata read hostname 'smithi090' 2024-04-02T14:43:46.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.347+0000 7fc7dc76d740 1 --2- 172.21.15.90:0/164685592 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x55734dd8f3c0 0x55734dd91810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:46.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.347+0000 7fc7dc76d740 4 client.28033 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:43:46.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.347+0000 7fc7dc76d740 1 -- 172.21.15.90:0/164685592 --> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55734dc89040 con 0x55734dd8f3c0 2024-04-02T14:43:46.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.348+0000 7fc7d2a00640 1 --2- 172.21.15.90:0/164685592 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x55734dd8f3c0 0x55734dd91810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:46.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.348+0000 7fc7c0c00640 10 client.28033 ms_handle_connect on v2:172.21.15.177:6836/4223414858 2024-04-02T14:43:46.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.348+0000 7fc7d2a00640 1 --2- 172.21.15.90:0/164685592 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x55734dd8f3c0 0x55734dd91810 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=8acaf9757c559660 server_cookie=dcafe23bcb9f8c34 in_seq=0 out_seq=0 2024-04-02T14:43:46.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.349+0000 7fc7c0c00640 1 -- 172.21.15.90:0/164685592 <== mds.4 v2:172.21.15.177:6836/4223414858 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55734dc89040 con 0x55734dd8f3c0 2024-04-02T14:43:46.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.349+0000 7fc7c0c00640 10 client.28033 handle_command_reply: tid=0 2024-04-02T14:43:46.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.350+0000 7fc7dc76d740 1 client.28033 shutdown 2024-04-02T14:43:46.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.350+0000 7fc7dc76d740 1 -- 172.21.15.90:0/164685592 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55734dd76890 msgr2=0x55734dd74420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:46.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.350+0000 7fc7dc76d740 1 --2- 172.21.15.90:0/164685592 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55734dd76890 0x55734dd74420 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7fc7cc0c74d0 tx=0x7fc7cc0be8d0 comp rx=0 tx=0).stop 2024-04-02T14:43:46.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.351+0000 7fc7dc76d740 1 -- 172.21.15.90:0/164685592 shutdown_connections 2024-04-02T14:43:46.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.351+0000 7fc7dc76d740 1 --2- 172.21.15.90:0/164685592 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55734dd76890 0x55734dd74420 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:46.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.352+0000 7fc7dc76d740 1 --2- 172.21.15.90:0/164685592 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x55734dd8f3c0 0x55734dd91810 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:46.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.352+0000 7fc7dc76d740 1 --2- 172.21.15.90:0/164685592 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55734dd6e200 0x55734dd73ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:46.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.352+0000 7fc7dc76d740 1 --2- 172.21.15.90:0/164685592 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55734dd6d520 0x55734dd739a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:46.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.352+0000 7fc7dc76d740 1 -- 172.21.15.90:0/164685592 >> 172.21.15.90:0/164685592 conn(0x55734dd2ba50 msgr2=0x55734dd69cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:43:46.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.352+0000 7fc7c0c00640 0 client.28033 ms_handle_reset on v2:172.21.15.177:6836/4223414858 2024-04-02T14:43:46.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.353+0000 7fc7dc76d740 1 -- 172.21.15.90:0/164685592 shutdown_connections 2024-04-02T14:43:46.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.353+0000 7fc7dc76d740 1 -- 172.21.15.90:0/164685592 wait complete. 2024-04-02T14:43:46.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.353+0000 7fc7dc76d740 20 client.28033 trim_cache size 0 max 16384 2024-04-02T14:43:46.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.378+0000 7fc7dc76d740 1 Processor -- start 2024-04-02T14:43:46.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.378+0000 7fc7dc76d740 1 -- start start 2024-04-02T14:43:46.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.378+0000 7fc7dc76d740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55734dd6d520 0x55734dd760d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:46.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.379+0000 7fc7dc76d740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55734dd6e200 0x55734dd72b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:46.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.379+0000 7fc7dc76d740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55734dd76890 0x55734dd5e720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:46.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.379+0000 7fc7dc76d740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fc7d41ab630 con 0x55734dd6e200 2024-04-02T14:43:46.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.379+0000 7fc7dc76d740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fc7d41ad170 con 0x55734dd76890 2024-04-02T14:43:46.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.379+0000 7fc7dc76d740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fc7d41abd00 con 0x55734dd6d520 2024-04-02T14:43:46.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.379+0000 7fc7d2a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55734dd6d520 0x55734dd760d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:46.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.379+0000 7fc7d2000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55734dd6e200 0x55734dd72b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:46.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.379+0000 7fc7d2a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55734dd6d520 0x55734dd760d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:52494/0 (socket says 172.21.15.90:52494) 2024-04-02T14:43:46.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.379+0000 7fc7d2a00640 1 -- 172.21.15.90:0/580696453 learned_addr learned my addr 172.21.15.90:0/580696453 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:43:46.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.379+0000 7fc7d3400640 1 --2- 172.21.15.90:0/580696453 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55734dd76890 0x55734dd5e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:46.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.379+0000 7fc7d2000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55734dd6e200 0x55734dd72b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33606/0 (socket says 172.21.15.90:33606) 2024-04-02T14:43:46.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.379+0000 7fc7d2a00640 1 -- 172.21.15.90:0/580696453 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55734dd76890 msgr2=0x55734dd5e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:43:46.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.379+0000 7fc7d2a00640 1 --2- 172.21.15.90:0/580696453 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55734dd76890 0x55734dd5e720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:46.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.379+0000 7fc7d2a00640 1 -- 172.21.15.90:0/580696453 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55734dd6e200 msgr2=0x55734dd72b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:43:46.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.379+0000 7fc7d2a00640 1 --2- 172.21.15.90:0/580696453 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55734dd6e200 0x55734dd72b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:46.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.379+0000 7fc7d2a00640 1 -- 172.21.15.90:0/580696453 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55734dd5adb0 con 0x55734dd6d520 2024-04-02T14:43:46.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.380+0000 7fc7d2a00640 1 --2- 172.21.15.90:0/580696453 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55734dd6d520 0x55734dd760d0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fc7c4011d10 tx=0x7fc7c4009f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3a3dc5d7721935d5 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:43:46.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.380+0000 7fc7c3400640 1 -- 172.21.15.90:0/580696453 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7c401e040 con 0x55734dd6d520 2024-04-02T14:43:46.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.380+0000 7fc7c3400640 1 -- 172.21.15.90:0/580696453 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc7c4005520 con 0x55734dd6d520 2024-04-02T14:43:46.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.380+0000 7fc7c3400640 1 -- 172.21.15.90:0/580696453 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7c4004970 con 0x55734dd6d520 2024-04-02T14:43:46.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.381+0000 7fc7dc76d740 1 -- 172.21.15.90:0/580696453 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55734dd6d520 msgr2=0x55734dd760d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:46.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.381+0000 7fc7dc76d740 1 --2- 172.21.15.90:0/580696453 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55734dd6d520 0x55734dd760d0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fc7c4011d10 tx=0x7fc7c4009f90 comp rx=0 tx=0).stop 2024-04-02T14:43:46.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.381+0000 7fc7dc76d740 1 -- 172.21.15.90:0/580696453 shutdown_connections 2024-04-02T14:43:46.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.381+0000 7fc7dc76d740 1 --2- 172.21.15.90:0/580696453 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55734dd76890 0x55734dd5e720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:46.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.381+0000 7fc7dc76d740 1 --2- 172.21.15.90:0/580696453 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55734dd6e200 0x55734dd72b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:46.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.381+0000 7fc7dc76d740 1 --2- 172.21.15.90:0/580696453 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55734dd6d520 0x55734dd760d0 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:46.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.381+0000 7fc7dc76d740 1 -- 172.21.15.90:0/580696453 >> 172.21.15.90:0/580696453 conn(0x55734dd2ba50 msgr2=0x55734dd5e010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:43:46.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.382+0000 7fc7dc76d740 1 -- 172.21.15.90:0/580696453 shutdown_connections 2024-04-02T14:43:46.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.382+0000 7fc7dc76d740 1 -- 172.21.15.90:0/580696453 wait complete. 2024-04-02T14:43:46.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.383+0000 7fc7dc76d740 1 -- start start 2024-04-02T14:43:46.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.383+0000 7fc7dc76d740 1 Processor -- start 2024-04-02T14:43:46.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.384+0000 7fc7dc76d740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55734dd6d520 0x55734dd6fc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:46.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.384+0000 7fc7dc76d740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55734dd6e200 0x55734dd70150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:46.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.384+0000 7fc7dc76d740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55734dd76890 0x55734dd70690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:46.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.384+0000 7fc7dc76d740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fc7d41acaa0 con 0x55734dd6e200 2024-04-02T14:43:46.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.384+0000 7fc7dc76d740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fc7d41ac3d0 con 0x55734dd76890 2024-04-02T14:43:46.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.384+0000 7fc7dc76d740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fc7d41a9690 con 0x55734dd6d520 2024-04-02T14:43:46.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.384+0000 7fc7dc76d740 10 client.0 _ll_register_callbacks cb 0x7fc7cc0bb030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:43:46.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.384+0000 7fc7d2a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55734dd6d520 0x55734dd6fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:46.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.384+0000 7fc7d2000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55734dd6e200 0x55734dd70150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:46.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.384+0000 7fc7d2a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55734dd6d520 0x55734dd6fc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:52496/0 (socket says 172.21.15.90:52496) 2024-04-02T14:43:46.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.385+0000 7fc7d3400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55734dd76890 0x55734dd70690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:46.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.385+0000 7fc7d2000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55734dd6e200 0x55734dd70150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33618/0 (socket says 172.21.15.90:33618) 2024-04-02T14:43:46.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.385+0000 7fc7d3400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55734dd76890 0x55734dd70690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:36250/0 (socket says 172.21.15.90:36250) 2024-04-02T14:43:46.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.385+0000 7fc7d2a00640 1 -- 172.21.15.90:0/4102820659 learned_addr learned my addr 172.21.15.90:0/4102820659 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:43:46.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.385+0000 7fc7d3400640 1 -- 172.21.15.90:0/4102820659 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55734dd6d520 msgr2=0x55734dd6fc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:43:46.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.385+0000 7fc7d3400640 1 --2- 172.21.15.90:0/4102820659 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55734dd6d520 0x55734dd6fc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:46.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.385+0000 7fc7d3400640 1 -- 172.21.15.90:0/4102820659 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55734dd6e200 msgr2=0x55734dd70150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:43:46.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.385+0000 7fc7d3400640 1 --2- 172.21.15.90:0/4102820659 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55734dd6e200 0x55734dd70150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:46.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.385+0000 7fc7d3400640 1 -- 172.21.15.90:0/4102820659 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55734dd655b0 con 0x55734dd76890 2024-04-02T14:43:46.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.385+0000 7fc7d2a00640 1 --2- 172.21.15.90:0/4102820659 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55734dd6d520 0x55734dd6fc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:43:46.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.385+0000 7fc7d3400640 1 --2- 172.21.15.90:0/4102820659 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55734dd76890 0x55734dd70690 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x55734dd767d0 tx=0x7fc7cc004630 comp rx=0 tx=0).ready entity=mon.2 client_cookie=141fea08a5fc5640 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:43:46.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.385+0000 7fc7c0c00640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:43:46.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.385+0000 7fc7c0c00640 1 -- 172.21.15.90:0/4102820659 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7cc0c7570 con 0x55734dd76890 2024-04-02T14:43:46.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.386+0000 7fc7c0c00640 1 -- 172.21.15.90:0/4102820659 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8550+0+0 (secure 0 0 0) 0x7fc7cc0d8200 con 0x55734dd76890 2024-04-02T14:43:46.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.386+0000 7fc7dc76d740 1 -- 172.21.15.90:0/4102820659 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc7c4005520 con 0x55734dd76890 2024-04-02T14:43:46.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.386+0000 7fc7c0c00640 1 -- 172.21.15.90:0/4102820659 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fc7cc0c3140 con 0x55734dd76890 2024-04-02T14:43:46.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.386+0000 7fc7dc76d740 10 client.28205 fetch_fsmap learned FSMap version 27 2024-04-02T14:43:46.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.386+0000 7fc7dc76d740 1 -- 172.21.15.90:0/4102820659 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55734ddf5be0 con 0x55734dd76890 2024-04-02T14:43:46.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.387+0000 7fc7c0c00640 1 -- 172.21.15.90:0/4102820659 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc7c4005520 con 0x55734dd76890 2024-04-02T14:43:46.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.387+0000 7fc7c0c00640 1 -- 172.21.15.90:0/4102820659 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fc7cc004b40 con 0x55734dd76890 2024-04-02T14:43:46.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.388+0000 7fc7dc76d740 10 client.28205 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:43:46.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.388+0000 7fc7dc76d740 10 client.28205 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.c 2024-04-02T14:43:46.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.388+0000 7fc7dc76d740 20 client.28205 populate_metadata read hostname 'smithi090' 2024-04-02T14:43:46.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.388+0000 7fc7dc76d740 1 --2- 172.21.15.90:0/4102820659 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x55734ddf6160 0x55734ddf8580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:46.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.388+0000 7fc7dc76d740 4 client.28205 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-02T14:43:46.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.388+0000 7fc7dc76d740 1 -- 172.21.15.90:0/4102820659 --> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55734dcd6c30 con 0x55734ddf6160 2024-04-02T14:43:46.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.388+0000 7fc7d2a00640 1 --2- 172.21.15.90:0/4102820659 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x55734ddf6160 0x55734ddf8580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:46.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.389+0000 7fc7d2a00640 1 --2- 172.21.15.90:0/4102820659 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x55734ddf6160 0x55734ddf8580 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=92732147bf51d894 server_cookie=c0520c3c93fdd4c0 in_seq=0 out_seq=0 2024-04-02T14:43:46.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.389+0000 7fc7c0c00640 10 client.28205 ms_handle_connect on v2:172.21.15.177:6836/4223414858 2024-04-02T14:43:46.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.389+0000 7fc7c0c00640 1 -- 172.21.15.90:0/4102820659 <== mds.4 v2:172.21.15.177:6836/4223414858 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55734dcd6c30 con 0x55734ddf6160 2024-04-02T14:43:46.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.389+0000 7fc7c0c00640 10 client.28205 handle_command_reply: tid=0 2024-04-02T14:43:46.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.389+0000 7fc7dc76d740 1 client.28205 shutdown 2024-04-02T14:43:46.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.390+0000 7fc7dc76d740 1 -- 172.21.15.90:0/4102820659 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55734dd76890 msgr2=0x55734dd70690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:46.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.390+0000 7fc7dc76d740 1 --2- 172.21.15.90:0/4102820659 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55734dd76890 0x55734dd70690 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x55734dd767d0 tx=0x7fc7cc004630 comp rx=0 tx=0).stop 2024-04-02T14:43:46.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.390+0000 7fc7dc76d740 1 -- 172.21.15.90:0/4102820659 shutdown_connections 2024-04-02T14:43:46.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.390+0000 7fc7dc76d740 1 --2- 172.21.15.90:0/4102820659 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55734dd76890 0x55734dd70690 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:46.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.390+0000 7fc7dc76d740 1 --2- 172.21.15.90:0/4102820659 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x55734ddf6160 0x55734ddf8580 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:46.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.390+0000 7fc7dc76d740 1 --2- 172.21.15.90:0/4102820659 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55734dd6e200 0x55734dd70150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:46.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.390+0000 7fc7dc76d740 1 --2- 172.21.15.90:0/4102820659 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55734dd6d520 0x55734dd6fc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:46.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.390+0000 7fc7c0c00640 0 client.28205 ms_handle_reset on v2:172.21.15.177:6836/4223414858 2024-04-02T14:43:46.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.390+0000 7fc7dc76d740 1 -- 172.21.15.90:0/4102820659 >> 172.21.15.90:0/4102820659 conn(0x55734dd2ba50 msgr2=0x55734dd741f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:43:46.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.390+0000 7fc7dc76d740 1 -- 172.21.15.90:0/4102820659 shutdown_connections 2024-04-02T14:43:46.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.390+0000 7fc7dc76d740 1 -- 172.21.15.90:0/4102820659 wait complete. 2024-04-02T14:43:46.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.390+0000 7fc7dc76d740 20 client.28205 trim_cache size 0 max 16384 2024-04-02T14:43:46.388 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-02T14:43:46.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.391+0000 7fc7d9a00640 1 -- 172.21.15.90:0/1070072528 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fc7b80a3da0 msgr2=0x7fc7b80a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:46.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.391+0000 7fc7d9a00640 1 --2- 172.21.15.90:0/1070072528 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fc7b80a3da0 0x7fc7b80a6260 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fc7bc005d00 tx=0x7fc7bc00b390 comp rx=0 tx=0).stop 2024-04-02T14:43:46.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.391+0000 7fc7d9a00640 1 -- 172.21.15.90:0/1070072528 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc7d4107aa0 msgr2=0x7fc7d419ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:46.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.391+0000 7fc7d9a00640 1 --2- 172.21.15.90:0/1070072528 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc7d4107aa0 0x7fc7d419ede0 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7fc7cc033420 tx=0x7fc7cc033720 comp rx=0 tx=0).stop 2024-04-02T14:43:46.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.391+0000 7fc7d9a00640 1 -- 172.21.15.90:0/1070072528 shutdown_connections 2024-04-02T14:43:46.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.391+0000 7fc7d9a00640 1 --2- 172.21.15.90:0/1070072528 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fc7b80a3da0 0x7fc7b80a6260 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:46.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.391+0000 7fc7d9a00640 1 --2- 172.21.15.90:0/1070072528 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc7d4107aa0 0x7fc7d419ede0 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:46.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.391+0000 7fc7d9a00640 1 --2- 172.21.15.90:0/1070072528 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc7d4102450 0x7fc7d419aa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:46.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.391+0000 7fc7d9a00640 1 --2- 172.21.15.90:0/1070072528 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc7d4101a90 0x7fc7d419a520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:46.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.391+0000 7fc7d9a00640 1 -- 172.21.15.90:0/1070072528 >> 172.21.15.90:0/1070072528 conn(0x7fc7d4078f00 msgr2=0x7fc7d410d220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:43:46.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.391+0000 7fc7d9a00640 1 -- 172.21.15.90:0/1070072528 shutdown_connections 2024-04-02T14:43:46.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.391+0000 7fc7d9a00640 1 -- 172.21.15.90:0/1070072528 wait complete. 2024-04-02T14:43:46.406 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:43:46 smithi090 ceph-mon[31674]: pgmap v273: 97 pgs: 97 active+clean; 4.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 205 KiB/s rd, 37 MiB/s wr, 2.19k op/s 2024-04-02T14:43:46.407 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:43:46 smithi090 ceph-mon[31674]: from='client.? 172.21.15.90:0/2848957490' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-02T14:43:46.410 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-04-02T14:43:46.432 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:43:46 smithi177 ceph-mon[36431]: pgmap v273: 97 pgs: 97 active+clean; 4.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 205 KiB/s rd, 37 MiB/s wr, 2.19k op/s 2024-04-02T14:43:46.432 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:43:46 smithi177 ceph-mon[36431]: from='client.? 172.21.15.90:0/2848957490' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-02T14:43:46.606 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:43:46 smithi092 ceph-mon[36497]: pgmap v273: 97 pgs: 97 active+clean; 4.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 205 KiB/s rd, 37 MiB/s wr, 2.19k op/s 2024-04-02T14:43:46.606 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:43:46 smithi092 ceph-mon[36497]: from='client.? 172.21.15.90:0/2848957490' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-02T14:43:46.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.708+0000 7fa94dc00640 1 -- 172.21.15.90:0/3531667657 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa948101a90 msgr2=0x7fa948101f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:46.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.708+0000 7fa94dc00640 1 --2- 172.21.15.90:0/3531667657 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa948101a90 0x7fa948101f10 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fa934009890 tx=0x7fa93402bb90 comp rx=0 tx=0).stop 2024-04-02T14:43:46.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.709+0000 7fa94dc00640 1 -- 172.21.15.90:0/3531667657 shutdown_connections 2024-04-02T14:43:46.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.709+0000 7fa94dc00640 1 --2- 172.21.15.90:0/3531667657 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa948102450 0x7fa94810e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:46.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.709+0000 7fa94dc00640 1 --2- 172.21.15.90:0/3531667657 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa948101a90 0x7fa948101f10 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:46.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.709+0000 7fa94dc00640 1 --2- 172.21.15.90:0/3531667657 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa948107aa0 0x7fa948107e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:46.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.709+0000 7fa94dc00640 1 -- 172.21.15.90:0/3531667657 >> 172.21.15.90:0/3531667657 conn(0x7fa948078f00 msgr2=0x7fa948079310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:43:46.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.709+0000 7fa94dc00640 1 -- 172.21.15.90:0/3531667657 shutdown_connections 2024-04-02T14:43:46.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.709+0000 7fa94dc00640 1 -- 172.21.15.90:0/3531667657 wait complete. 2024-04-02T14:43:46.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.710+0000 7fa94dc00640 1 Processor -- start 2024-04-02T14:43:46.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.711+0000 7fa94dc00640 1 -- start start 2024-04-02T14:43:46.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.711+0000 7fa94dc00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa948101a90 0x7fa94810dd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:46.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.711+0000 7fa94dc00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa948102450 0x7fa94810e270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:46.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.711+0000 7fa94dc00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa948107aa0 0x7fa9481a32c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:46.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.711+0000 7fa94dc00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fa948111eb0 con 0x7fa948101a90 2024-04-02T14:43:46.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.711+0000 7fa94dc00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fa948111d30 con 0x7fa948102450 2024-04-02T14:43:46.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.711+0000 7fa94dc00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fa948112030 con 0x7fa948107aa0 2024-04-02T14:43:46.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.713+0000 7fa947400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa948107aa0 0x7fa9481a32c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:46.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.713+0000 7fa947400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa948107aa0 0x7fa9481a32c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:52512/0 (socket says 172.21.15.90:52512) 2024-04-02T14:43:46.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.713+0000 7fa947400640 1 -- 172.21.15.90:0/3899101268 learned_addr learned my addr 172.21.15.90:0/3899101268 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:43:46.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.713+0000 7fa946a00640 1 --2- 172.21.15.90:0/3899101268 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa948101a90 0x7fa94810dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:46.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.714+0000 7fa946000640 1 --2- 172.21.15.90:0/3899101268 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa948102450 0x7fa94810e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:46.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.714+0000 7fa947400640 1 -- 172.21.15.90:0/3899101268 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa948102450 msgr2=0x7fa94810e270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:46.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.714+0000 7fa947400640 1 --2- 172.21.15.90:0/3899101268 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa948102450 0x7fa94810e270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:46.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.714+0000 7fa947400640 1 -- 172.21.15.90:0/3899101268 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa948101a90 msgr2=0x7fa94810dd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:46.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.714+0000 7fa947400640 1 --2- 172.21.15.90:0/3899101268 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa948101a90 0x7fa94810dd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:46.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.714+0000 7fa947400640 1 -- 172.21.15.90:0/3899101268 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9481a39f0 con 0x7fa948107aa0 2024-04-02T14:43:46.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.715+0000 7fa946a00640 1 --2- 172.21.15.90:0/3899101268 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa948101a90 0x7fa94810dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:43:46.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.717+0000 7fa947400640 1 --2- 172.21.15.90:0/3899101268 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa948107aa0 0x7fa9481a32c0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fa93800d750 tx=0x7fa93800dc10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:43:46.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.717+0000 7fa93f400640 1 -- 172.21.15.90:0/3899101268 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa938017070 con 0x7fa948107aa0 2024-04-02T14:43:46.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.717+0000 7fa93f400640 1 -- 172.21.15.90:0/3899101268 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa938005500 con 0x7fa948107aa0 2024-04-02T14:43:46.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.717+0000 7fa93f400640 1 -- 172.21.15.90:0/3899101268 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa938004da0 con 0x7fa948107aa0 2024-04-02T14:43:46.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.717+0000 7fa94dc00640 1 -- 172.21.15.90:0/3899101268 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9481a3ce0 con 0x7fa948107aa0 2024-04-02T14:43:46.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.718+0000 7fa94dc00640 1 -- 172.21.15.90:0/3899101268 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9481ab570 con 0x7fa948107aa0 2024-04-02T14:43:46.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.724+0000 7fa93f400640 1 -- 172.21.15.90:0/3899101268 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fa938004f40 con 0x7fa948107aa0 2024-04-02T14:43:46.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.725+0000 7fa95087b740 1 Processor -- start 2024-04-02T14:43:46.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.726+0000 7fa95087b740 1 -- start start 2024-04-02T14:43:46.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.726+0000 7fa95087b740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55733a37e9f0 0x55733a37d740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:46.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.726+0000 7fa95087b740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55733a37dd10 0x55733a37e190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:46.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.726+0000 7fa93f400640 1 --2- 172.21.15.90:0/3899101268 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fa92c0a3e20 0x7fa92c0a62e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:46.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.727+0000 7fa946000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55733a37e9f0 0x55733a37d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:46.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.727+0000 7fa93f400640 1 -- 172.21.15.90:0/3899101268 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8550+0+0 (secure 0 0 0) 0x7fa9380d0130 con 0x7fa948107aa0 2024-04-02T14:43:46.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.727+0000 7fa946000640 1 --2- 172.21.15.90:0/3899101268 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fa92c0a3e20 0x7fa92c0a62e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:46.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.727+0000 7fa946000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55733a37e9f0 0x55733a37d740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33652/0 (socket says 172.21.15.90:33652) 2024-04-02T14:43:46.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.727+0000 7fa946000640 1 -- 172.21.15.90:0/2957560768 learned_addr learned my addr 172.21.15.90:0/2957560768 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:43:46.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.727+0000 7fa95087b740 1 --2- 172.21.15.90:0/2957560768 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55733a387080 0x55733a389470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:46.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.727+0000 7fa95087b740 1 -- 172.21.15.90:0/2957560768 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55733a2be400 con 0x55733a37e9f0 2024-04-02T14:43:46.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.727+0000 7fa95087b740 1 -- 172.21.15.90:0/2957560768 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55733a2a2f10 con 0x55733a37dd10 2024-04-02T14:43:46.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.727+0000 7fa95087b740 1 -- 172.21.15.90:0/2957560768 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55733a2e26f0 con 0x55733a387080 2024-04-02T14:43:46.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.727+0000 7fa947400640 1 --2- 172.21.15.90:0/2957560768 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55733a387080 0x55733a389470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:46.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.728+0000 7fa947400640 1 -- 172.21.15.90:0/2957560768 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55733a37dd10 msgr2=0x55733a37e190 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-02T14:43:46.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.729+0000 7fa947400640 1 --2- 172.21.15.90:0/2957560768 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55733a37dd10 0x55733a37e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:46.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.729+0000 7fa947400640 1 -- 172.21.15.90:0/2957560768 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55733a37e9f0 msgr2=0x55733a37d740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:43:46.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.729+0000 7fa947400640 1 --2- 172.21.15.90:0/2957560768 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55733a37e9f0 0x55733a37d740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:46.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.729+0000 7fa947400640 1 -- 172.21.15.90:0/2957560768 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557339ff63f0 con 0x55733a387080 2024-04-02T14:43:46.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.730+0000 7fa946000640 1 --2- 172.21.15.90:0/2957560768 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55733a37e9f0 0x55733a37d740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:43:46.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.731+0000 7fa946000640 1 --2- 172.21.15.90:0/3899101268 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fa92c0a3e20 0x7fa92c0a62e0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fa9340095f0 tx=0x7fa934009540 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:43:46.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.731+0000 7fa947400640 1 --2- 172.21.15.90:0/2957560768 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55733a387080 0x55733a389470 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fa9380cd800 tx=0x7fa93800df50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e3cb2101ca9f0aba server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:43:46.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.731+0000 7fa933e00640 1 -- 172.21.15.90:0/2957560768 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa938020040 con 0x55733a387080 2024-04-02T14:43:46.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.731+0000 7fa933e00640 1 -- 172.21.15.90:0/2957560768 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa9380a4cc0 con 0x55733a387080 2024-04-02T14:43:46.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.731+0000 7fa933e00640 1 -- 172.21.15.90:0/2957560768 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9380cda80 con 0x55733a387080 2024-04-02T14:43:46.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.732+0000 7fa95087b740 1 -- 172.21.15.90:0/2957560768 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55733a387080 msgr2=0x55733a389470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:46.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.732+0000 7fa95087b740 1 --2- 172.21.15.90:0/2957560768 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55733a387080 0x55733a389470 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fa9380cd800 tx=0x7fa93800df50 comp rx=0 tx=0).stop 2024-04-02T14:43:46.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.733+0000 7fa95087b740 1 -- 172.21.15.90:0/2957560768 shutdown_connections 2024-04-02T14:43:46.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.733+0000 7fa95087b740 1 --2- 172.21.15.90:0/2957560768 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55733a387080 0x55733a389470 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:46.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.733+0000 7fa95087b740 1 --2- 172.21.15.90:0/2957560768 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55733a37dd10 0x55733a37e190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:46.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.733+0000 7fa95087b740 1 --2- 172.21.15.90:0/2957560768 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55733a37e9f0 0x55733a37d740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:46.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.733+0000 7fa95087b740 1 -- 172.21.15.90:0/2957560768 >> 172.21.15.90:0/2957560768 conn(0x55733a344a50 msgr2=0x55733a378760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:43:46.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.733+0000 7fa95087b740 1 -- 172.21.15.90:0/2957560768 shutdown_connections 2024-04-02T14:43:46.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.733+0000 7fa95087b740 1 -- 172.21.15.90:0/2957560768 wait complete. 2024-04-02T14:43:46.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.734+0000 7fa95087b740 1 -- start start 2024-04-02T14:43:46.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.734+0000 7fa95087b740 1 Processor -- start 2024-04-02T14:43:46.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.734+0000 7fa95087b740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55733a37dd10 0x55733a384090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:46.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.734+0000 7fa95087b740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55733a37e9f0 0x55733a3845d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:46.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.735+0000 7fa95087b740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55733a387080 0x55733a384b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:46.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.735+0000 7fa95087b740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55733a2de6a0 con 0x55733a387080 2024-04-02T14:43:46.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.735+0000 7fa95087b740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55733a28a980 con 0x55733a37e9f0 2024-04-02T14:43:46.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.735+0000 7fa95087b740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55733a29dcb0 con 0x55733a37dd10 2024-04-02T14:43:46.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.735+0000 7fa95087b740 10 client.0 _ll_register_callbacks cb 0x55733a29d6a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:43:46.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.735+0000 7fa946a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55733a37dd10 0x55733a384090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:46.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.735+0000 7fa946a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55733a37dd10 0x55733a384090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:52530/0 (socket says 172.21.15.90:52530) 2024-04-02T14:43:46.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.735+0000 7fa946a00640 1 -- 172.21.15.90:0/1084091146 learned_addr learned my addr 172.21.15.90:0/1084091146 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:43:46.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.735+0000 7fa946000640 1 --2- 172.21.15.90:0/1084091146 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55733a37e9f0 0x55733a3845d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:46.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.735+0000 7fa947400640 1 --2- 172.21.15.90:0/1084091146 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55733a387080 0x55733a384b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:46.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.735+0000 7fa946a00640 1 -- 172.21.15.90:0/1084091146 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55733a37e9f0 msgr2=0x55733a3845d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:43:46.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.735+0000 7fa946a00640 1 --2- 172.21.15.90:0/1084091146 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55733a37e9f0 0x55733a3845d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:46.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.736+0000 7fa946a00640 1 -- 172.21.15.90:0/1084091146 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55733a387080 msgr2=0x55733a384b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:43:46.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.736+0000 7fa946a00640 1 --2- 172.21.15.90:0/1084091146 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55733a387080 0x55733a384b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:46.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.736+0000 7fa946a00640 1 -- 172.21.15.90:0/1084091146 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55733a0c07d0 con 0x55733a37dd10 2024-04-02T14:43:46.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.736+0000 7fa946000640 1 --2- 172.21.15.90:0/1084091146 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55733a37e9f0 0x55733a3845d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:43:46.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.737+0000 7fa946a00640 1 --2- 172.21.15.90:0/1084091146 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55733a37dd10 0x55733a384090 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7fa940009bb0 tx=0x7fa94000d870 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9bfb883ccd6f042d server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:43:46.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.737+0000 7fa930c00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:43:46.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.737+0000 7fa930c00640 1 -- 172.21.15.90:0/1084091146 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa94001b020 con 0x55733a37dd10 2024-04-02T14:43:46.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.737+0000 7fa930c00640 1 -- 172.21.15.90:0/1084091146 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8550+0+0 (secure 0 0 0) 0x7fa940003b70 con 0x55733a37dd10 2024-04-02T14:43:46.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.737+0000 7fa95087b740 1 -- 172.21.15.90:0/1084091146 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa9380a4cc0 con 0x55733a37dd10 2024-04-02T14:43:46.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.737+0000 7fa930c00640 1 -- 172.21.15.90:0/1084091146 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fa940003d70 con 0x55733a37dd10 2024-04-02T14:43:46.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.738+0000 7fa95087b740 10 client.28051 fetch_fsmap learned FSMap version 27 2024-04-02T14:43:46.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.738+0000 7fa95087b740 1 -- 172.21.15.90:0/1084091146 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55733a395f80 con 0x55733a37dd10 2024-04-02T14:43:46.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.738+0000 7fa930c00640 1 -- 172.21.15.90:0/1084091146 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa9380a4cc0 con 0x55733a37dd10 2024-04-02T14:43:46.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.738+0000 7fa930c00640 1 -- 172.21.15.90:0/1084091146 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fa94000b040 con 0x55733a37dd10 2024-04-02T14:43:46.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.738+0000 7fa95087b740 10 client.28051 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:43:46.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.738+0000 7fa95087b740 10 client.28051 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-04-02T14:43:46.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.738+0000 7fa95087b740 20 client.28051 populate_metadata read hostname 'smithi090' 2024-04-02T14:43:46.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.738+0000 7fa95087b740 1 --2- 172.21.15.90:0/1084091146 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x55733a397390 0x55733a399800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:46.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.738+0000 7fa95087b740 4 client.28051 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:43:46.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.738+0000 7fa95087b740 1 -- 172.21.15.90:0/1084091146 --> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55733a2a2040 con 0x55733a397390 2024-04-02T14:43:46.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.739+0000 7fa946000640 1 --2- 172.21.15.90:0/1084091146 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x55733a397390 0x55733a399800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:46.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.739+0000 7fa946000640 1 --2- 172.21.15.90:0/1084091146 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x55733a397390 0x55733a399800 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=46deadc96f4da80a server_cookie=10ffd9f9568001b6 in_seq=0 out_seq=0 2024-04-02T14:43:46.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.739+0000 7fa930c00640 10 client.28051 ms_handle_connect on v2:172.21.15.92:6836/1270635939 2024-04-02T14:43:46.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.740+0000 7fa930c00640 1 -- 172.21.15.90:0/1084091146 <== mds.3 v2:172.21.15.92:6836/1270635939 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55733a2a2040 con 0x55733a397390 2024-04-02T14:43:46.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.740+0000 7fa930c00640 10 client.28051 handle_command_reply: tid=0 2024-04-02T14:43:46.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.740+0000 7fa95087b740 1 client.28051 shutdown 2024-04-02T14:43:46.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.740+0000 7fa95087b740 1 -- 172.21.15.90:0/1084091146 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55733a37dd10 msgr2=0x55733a384090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:46.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.740+0000 7fa95087b740 1 --2- 172.21.15.90:0/1084091146 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55733a37dd10 0x55733a384090 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7fa940009bb0 tx=0x7fa94000d870 comp rx=0 tx=0).stop 2024-04-02T14:43:46.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.740+0000 7fa95087b740 1 -- 172.21.15.90:0/1084091146 shutdown_connections 2024-04-02T14:43:46.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.740+0000 7fa95087b740 1 --2- 172.21.15.90:0/1084091146 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x55733a397390 0x55733a399800 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:46.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.740+0000 7fa95087b740 1 --2- 172.21.15.90:0/1084091146 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55733a387080 0x55733a384b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:46.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.740+0000 7fa95087b740 1 --2- 172.21.15.90:0/1084091146 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55733a37e9f0 0x55733a3845d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:46.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.740+0000 7fa95087b740 1 --2- 172.21.15.90:0/1084091146 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55733a37dd10 0x55733a384090 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:46.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.740+0000 7fa95087b740 1 -- 172.21.15.90:0/1084091146 >> 172.21.15.90:0/1084091146 conn(0x55733a344a50 msgr2=0x55733a37a4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:43:46.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.740+0000 7fa930c00640 0 client.28051 ms_handle_reset on v2:172.21.15.92:6836/1270635939 2024-04-02T14:43:46.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.741+0000 7fa95087b740 1 -- 172.21.15.90:0/1084091146 shutdown_connections 2024-04-02T14:43:46.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.741+0000 7fa95087b740 1 -- 172.21.15.90:0/1084091146 wait complete. 2024-04-02T14:43:46.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.741+0000 7fa95087b740 20 client.28051 trim_cache size 0 max 16384 2024-04-02T14:43:46.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.766+0000 7fa95087b740 1 Processor -- start 2024-04-02T14:43:46.764 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.766+0000 7fa95087b740 1 -- start start 2024-04-02T14:43:46.764 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.766+0000 7fa95087b740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55733a37dd10 0x55733a3867c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:46.764 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.766+0000 7fa95087b740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55733a37e9f0 0x55733a392a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:46.764 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.766+0000 7fa95087b740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55733a387080 0x55733a39f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:46.764 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.766+0000 7fa95087b740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fa9481b3f20 con 0x55733a37e9f0 2024-04-02T14:43:46.764 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.766+0000 7fa95087b740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fa9481b5a60 con 0x55733a37dd10 2024-04-02T14:43:46.764 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.766+0000 7fa95087b740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fa9481b45f0 con 0x55733a387080 2024-04-02T14:43:46.764 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.766+0000 7fa946a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55733a37dd10 0x55733a3867c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:46.764 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.766+0000 7fa946000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55733a37e9f0 0x55733a392a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:46.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.766+0000 7fa946a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55733a37dd10 0x55733a3867c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:36302/0 (socket says 172.21.15.90:36302) 2024-04-02T14:43:46.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.766+0000 7fa946a00640 1 -- 172.21.15.90:0/1802261960 learned_addr learned my addr 172.21.15.90:0/1802261960 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:43:46.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.767+0000 7fa947400640 1 --2- 172.21.15.90:0/1802261960 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55733a387080 0x55733a39f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:46.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.767+0000 7fa946a00640 1 -- 172.21.15.90:0/1802261960 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55733a387080 msgr2=0x55733a39f0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:43:46.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.767+0000 7fa946a00640 1 --2- 172.21.15.90:0/1802261960 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55733a387080 0x55733a39f0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:46.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.767+0000 7fa946a00640 1 -- 172.21.15.90:0/1802261960 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55733a37e9f0 msgr2=0x55733a392a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:43:46.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.767+0000 7fa946a00640 1 --2- 172.21.15.90:0/1802261960 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55733a37e9f0 0x55733a392a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:46.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.767+0000 7fa946a00640 1 -- 172.21.15.90:0/1802261960 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55733a37fc30 con 0x55733a37dd10 2024-04-02T14:43:46.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.767+0000 7fa946a00640 1 --2- 172.21.15.90:0/1802261960 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55733a37dd10 0x55733a3867c0 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7fa940000f80 tx=0x7fa940005de0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7033bf956a009d0d server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:43:46.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.767+0000 7fa933400640 1 -- 172.21.15.90:0/1802261960 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa94002f070 con 0x55733a37dd10 2024-04-02T14:43:46.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.768+0000 7fa933400640 1 -- 172.21.15.90:0/1802261960 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa940009540 con 0x55733a37dd10 2024-04-02T14:43:46.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.768+0000 7fa933400640 1 -- 172.21.15.90:0/1802261960 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa94001b040 con 0x55733a37dd10 2024-04-02T14:43:46.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.768+0000 7fa95087b740 1 -- 172.21.15.90:0/1802261960 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55733a37dd10 msgr2=0x55733a3867c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:46.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.768+0000 7fa95087b740 1 --2- 172.21.15.90:0/1802261960 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55733a37dd10 0x55733a3867c0 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7fa940000f80 tx=0x7fa940005de0 comp rx=0 tx=0).stop 2024-04-02T14:43:46.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.768+0000 7fa95087b740 1 -- 172.21.15.90:0/1802261960 shutdown_connections 2024-04-02T14:43:46.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.768+0000 7fa95087b740 1 --2- 172.21.15.90:0/1802261960 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55733a387080 0x55733a39f0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:46.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.768+0000 7fa95087b740 1 --2- 172.21.15.90:0/1802261960 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55733a37e9f0 0x55733a392a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:46.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.768+0000 7fa95087b740 1 --2- 172.21.15.90:0/1802261960 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55733a37dd10 0x55733a3867c0 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:46.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.768+0000 7fa95087b740 1 -- 172.21.15.90:0/1802261960 >> 172.21.15.90:0/1802261960 conn(0x55733a344a50 msgr2=0x55733a378290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:43:46.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.769+0000 7fa95087b740 1 -- 172.21.15.90:0/1802261960 shutdown_connections 2024-04-02T14:43:46.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.770+0000 7fa95087b740 1 -- 172.21.15.90:0/1802261960 wait complete. 2024-04-02T14:43:46.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.770+0000 7fa95087b740 1 -- start start 2024-04-02T14:43:46.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.771+0000 7fa95087b740 1 Processor -- start 2024-04-02T14:43:46.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.771+0000 7fa95087b740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55733a37dd10 0x55733a38f430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:46.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.771+0000 7fa95087b740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55733a37e9f0 0x55733a3814d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:46.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.772+0000 7fa95087b740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55733a387080 0x55733a381ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:46.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.772+0000 7fa95087b740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fa9481b5390 con 0x55733a37e9f0 2024-04-02T14:43:46.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.772+0000 7fa95087b740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fa9481b4cc0 con 0x55733a37dd10 2024-04-02T14:43:46.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.772+0000 7fa95087b740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fa9481b1f20 con 0x55733a387080 2024-04-02T14:43:46.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.772+0000 7fa95087b740 10 client.0 _ll_register_callbacks cb 0x7fa94000ebf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:43:46.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.772+0000 7fa946a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55733a37dd10 0x55733a38f430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:46.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.772+0000 7fa946a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55733a37dd10 0x55733a38f430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:36314/0 (socket says 172.21.15.90:36314) 2024-04-02T14:43:46.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.772+0000 7fa946a00640 1 -- 172.21.15.90:0/831457518 learned_addr learned my addr 172.21.15.90:0/831457518 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:43:46.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.772+0000 7fa947400640 1 --2- 172.21.15.90:0/831457518 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55733a387080 0x55733a381ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:46.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.772+0000 7fa946000640 1 --2- 172.21.15.90:0/831457518 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55733a37e9f0 0x55733a3814d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:46.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.773+0000 7fa946a00640 1 -- 172.21.15.90:0/831457518 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55733a387080 msgr2=0x55733a381ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:43:46.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.773+0000 7fa946a00640 1 --2- 172.21.15.90:0/831457518 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55733a387080 0x55733a381ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:46.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.773+0000 7fa946a00640 1 -- 172.21.15.90:0/831457518 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55733a37e9f0 msgr2=0x55733a3814d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:43:46.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.773+0000 7fa946a00640 1 --2- 172.21.15.90:0/831457518 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55733a37e9f0 0x55733a3814d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:46.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.773+0000 7fa946a00640 1 -- 172.21.15.90:0/831457518 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55733a387c70 con 0x55733a37dd10 2024-04-02T14:43:46.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.773+0000 7fa946000640 1 --2- 172.21.15.90:0/831457518 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55733a37e9f0 0x55733a3814d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:43:46.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.773+0000 7fa946a00640 1 --2- 172.21.15.90:0/831457518 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55733a37dd10 0x55733a38f430 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7fa940009730 tx=0x7fa94000d870 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2fb672dca784bd7a server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:43:46.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.773+0000 7fa930c00640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:43:46.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.773+0000 7fa930c00640 1 -- 172.21.15.90:0/831457518 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa94002f040 con 0x55733a37dd10 2024-04-02T14:43:46.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.774+0000 7fa930c00640 1 -- 172.21.15.90:0/831457518 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8550+0+0 (secure 0 0 0) 0x7fa940004110 con 0x55733a37dd10 2024-04-02T14:43:46.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.774+0000 7fa95087b740 1 -- 172.21.15.90:0/831457518 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa940009540 con 0x55733a37dd10 2024-04-02T14:43:46.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.774+0000 7fa930c00640 1 -- 172.21.15.90:0/831457518 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fa94001b040 con 0x55733a37dd10 2024-04-02T14:43:46.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.774+0000 7fa95087b740 10 client.28217 fetch_fsmap learned FSMap version 27 2024-04-02T14:43:46.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.774+0000 7fa95087b740 1 -- 172.21.15.90:0/831457518 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55733a3fdc10 con 0x55733a37dd10 2024-04-02T14:43:46.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.775+0000 7fa930c00640 1 -- 172.21.15.90:0/831457518 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa940009540 con 0x55733a37dd10 2024-04-02T14:43:46.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.775+0000 7fa930c00640 1 -- 172.21.15.90:0/831457518 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fa94000b070 con 0x55733a37dd10 2024-04-02T14:43:46.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.775+0000 7fa95087b740 10 client.28217 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:43:46.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.775+0000 7fa95087b740 10 client.28217 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-04-02T14:43:46.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.775+0000 7fa95087b740 20 client.28217 populate_metadata read hostname 'smithi090' 2024-04-02T14:43:46.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.775+0000 7fa95087b740 1 --2- 172.21.15.90:0/831457518 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x55733a3fe280 0x55733a4006a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:46.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.775+0000 7fa95087b740 4 client.28217 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-02T14:43:46.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.775+0000 7fa95087b740 1 -- 172.21.15.90:0/831457518 --> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55733a2efc30 con 0x55733a3fe280 2024-04-02T14:43:46.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.776+0000 7fa946000640 1 --2- 172.21.15.90:0/831457518 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x55733a3fe280 0x55733a4006a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:46.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.776+0000 7fa946000640 1 --2- 172.21.15.90:0/831457518 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x55733a3fe280 0x55733a4006a0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=30d014e096d8f54 server_cookie=6de205206bf6d5da in_seq=0 out_seq=0 2024-04-02T14:43:46.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.776+0000 7fa930c00640 10 client.28217 ms_handle_connect on v2:172.21.15.92:6836/1270635939 2024-04-02T14:43:46.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.776+0000 7fa930c00640 1 -- 172.21.15.90:0/831457518 <== mds.3 v2:172.21.15.92:6836/1270635939 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55733a2efc30 con 0x55733a3fe280 2024-04-02T14:43:46.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.776+0000 7fa930c00640 10 client.28217 handle_command_reply: tid=0 2024-04-02T14:43:46.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.776+0000 7fa95087b740 1 client.28217 shutdown 2024-04-02T14:43:46.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.776+0000 7fa95087b740 1 -- 172.21.15.90:0/831457518 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55733a37dd10 msgr2=0x55733a38f430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:46.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.776+0000 7fa95087b740 1 --2- 172.21.15.90:0/831457518 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55733a37dd10 0x55733a38f430 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7fa940009730 tx=0x7fa94000d870 comp rx=0 tx=0).stop 2024-04-02T14:43:46.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.776+0000 7fa95087b740 1 -- 172.21.15.90:0/831457518 shutdown_connections 2024-04-02T14:43:46.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.776+0000 7fa95087b740 1 --2- 172.21.15.90:0/831457518 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x55733a3fe280 0x55733a4006a0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:46.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.776+0000 7fa95087b740 1 --2- 172.21.15.90:0/831457518 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55733a387080 0x55733a381ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:46.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.776+0000 7fa95087b740 1 --2- 172.21.15.90:0/831457518 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55733a37e9f0 0x55733a3814d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:46.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.776+0000 7fa95087b740 1 --2- 172.21.15.90:0/831457518 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55733a37dd10 0x55733a38f430 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:46.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.776+0000 7fa930c00640 0 client.28217 ms_handle_reset on v2:172.21.15.92:6836/1270635939 2024-04-02T14:43:46.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.777+0000 7fa95087b740 1 -- 172.21.15.90:0/831457518 >> 172.21.15.90:0/831457518 conn(0x55733a344a50 msgr2=0x55733a377010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:43:46.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.777+0000 7fa95087b740 1 -- 172.21.15.90:0/831457518 shutdown_connections 2024-04-02T14:43:46.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.777+0000 7fa95087b740 1 -- 172.21.15.90:0/831457518 wait complete. 2024-04-02T14:43:46.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.777+0000 7fa95087b740 20 client.28217 trim_cache size 0 max 16384 2024-04-02T14:43:46.780 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-02T14:43:46.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.778+0000 7fa94dc00640 1 -- 172.21.15.90:0/3899101268 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fa92c0a3e20 msgr2=0x7fa92c0a62e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:46.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.778+0000 7fa94dc00640 1 --2- 172.21.15.90:0/3899101268 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fa92c0a3e20 0x7fa92c0a62e0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fa9340095f0 tx=0x7fa934009540 comp rx=0 tx=0).stop 2024-04-02T14:43:46.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.778+0000 7fa94dc00640 1 -- 172.21.15.90:0/3899101268 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa948107aa0 msgr2=0x7fa9481a32c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:46.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.778+0000 7fa94dc00640 1 --2- 172.21.15.90:0/3899101268 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa948107aa0 0x7fa9481a32c0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fa93800d750 tx=0x7fa93800dc10 comp rx=0 tx=0).stop 2024-04-02T14:43:46.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.778+0000 7fa94dc00640 1 -- 172.21.15.90:0/3899101268 shutdown_connections 2024-04-02T14:43:46.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.778+0000 7fa94dc00640 1 --2- 172.21.15.90:0/3899101268 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fa92c0a3e20 0x7fa92c0a62e0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:46.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.778+0000 7fa94dc00640 1 --2- 172.21.15.90:0/3899101268 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa948107aa0 0x7fa9481a32c0 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:46.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.778+0000 7fa94dc00640 1 --2- 172.21.15.90:0/3899101268 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa948102450 0x7fa94810e270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:46.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.778+0000 7fa94dc00640 1 --2- 172.21.15.90:0/3899101268 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa948101a90 0x7fa94810dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:46.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.778+0000 7fa94dc00640 1 -- 172.21.15.90:0/3899101268 >> 172.21.15.90:0/3899101268 conn(0x7fa948078f00 msgr2=0x7fa948104210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:43:46.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.778+0000 7fa94dc00640 1 -- 172.21.15.90:0/3899101268 shutdown_connections 2024-04-02T14:43:46.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:46.778+0000 7fa94dc00640 1 -- 172.21.15.90:0/3899101268 wait complete. 2024-04-02T14:43:46.797 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-02T14:43:46.827 INFO:teuthology.orchestra.run.smithi090.stderr:progress: 495.0 s, 1672.3 tps, lat 59.905 ms stddev 75.063 2024-04-02T14:43:47.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.074+0000 7f1afda00640 1 -- 172.21.15.90:0/3168847594 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1af8108fe0 msgr2=0x7f1af8110bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:47.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.074+0000 7f1afda00640 1 --2- 172.21.15.90:0/3168847594 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1af8108fe0 0x7f1af8110bb0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f1ae4009860 tx=0x7f1ae402bb70 comp rx=0 tx=0).stop 2024-04-02T14:43:47.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.075+0000 7f1afda00640 1 -- 172.21.15.90:0/3168847594 shutdown_connections 2024-04-02T14:43:47.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.075+0000 7f1afda00640 1 --2- 172.21.15.90:0/3168847594 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1af8108fe0 0x7f1af8110bb0 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:47.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.075+0000 7f1afda00640 1 --2- 172.21.15.90:0/3168847594 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1af81048c0 0x7f1af8108900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:47.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.075+0000 7f1afda00640 1 --2- 172.21.15.90:0/3168847594 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1af8103f10 0x7f1af81042f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:47.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.075+0000 7f1afda00640 1 -- 172.21.15.90:0/3168847594 >> 172.21.15.90:0/3168847594 conn(0x7f1af80ff7b0 msgr2=0x7f1af8101bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:43:47.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.076+0000 7f1afda00640 1 -- 172.21.15.90:0/3168847594 shutdown_connections 2024-04-02T14:43:47.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.076+0000 7f1afda00640 1 -- 172.21.15.90:0/3168847594 wait complete. 2024-04-02T14:43:47.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.077+0000 7f1afda00640 1 Processor -- start 2024-04-02T14:43:47.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.077+0000 7f1afda00640 1 -- start start 2024-04-02T14:43:47.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.078+0000 7f1afda00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1af8103f10 0x7f1af819a530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:47.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.078+0000 7f1afda00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1af8108fe0 0x7f1af819aa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:47.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.078+0000 7f1afda00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1af819edf0 0x7f1af819f2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:47.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.078+0000 7f1afda00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f1af8114090 con 0x7f1af8108fe0 2024-04-02T14:43:47.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.078+0000 7f1afda00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f1af8113f10 con 0x7f1af8103f10 2024-04-02T14:43:47.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.078+0000 7f1afda00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f1af8114210 con 0x7f1af819edf0 2024-04-02T14:43:47.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.078+0000 7f1af6a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1af8103f10 0x7f1af819a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:47.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.078+0000 7f1af6a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1af8103f10 0x7f1af819a530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:36328/0 (socket says 172.21.15.90:36328) 2024-04-02T14:43:47.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.078+0000 7f1af6a00640 1 -- 172.21.15.90:0/2494604386 learned_addr learned my addr 172.21.15.90:0/2494604386 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:43:47.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.078+0000 7f1af6000640 1 --2- 172.21.15.90:0/2494604386 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1af8108fe0 0x7f1af819aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:47.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.079+0000 7f1af6000640 1 -- 172.21.15.90:0/2494604386 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1af819edf0 msgr2=0x7f1af819f2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:47.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.079+0000 7f1af6000640 1 --2- 172.21.15.90:0/2494604386 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1af819edf0 0x7f1af819f2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:47.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.079+0000 7f1af6000640 1 -- 172.21.15.90:0/2494604386 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1af8103f10 msgr2=0x7f1af819a530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:47.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.079+0000 7f1af6000640 1 --2- 172.21.15.90:0/2494604386 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1af8103f10 0x7f1af819a530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:47.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.079+0000 7f1af6000640 1 -- 172.21.15.90:0/2494604386 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1af819f8e0 con 0x7f1af8108fe0 2024-04-02T14:43:47.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.082+0000 7f1af6000640 1 --2- 172.21.15.90:0/2494604386 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1af8108fe0 0x7f1af819aa70 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f1af000ccb0 tx=0x7f1af000a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:43:47.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.084+0000 7f1aef400640 1 -- 172.21.15.90:0/2494604386 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1af000f450 con 0x7f1af8108fe0 2024-04-02T14:43:47.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.085+0000 7f1aef400640 1 -- 172.21.15.90:0/2494604386 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1af000fad0 con 0x7f1af8108fe0 2024-04-02T14:43:47.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.085+0000 7f1aef400640 1 -- 172.21.15.90:0/2494604386 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1af0018bb0 con 0x7f1af8108fe0 2024-04-02T14:43:47.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.085+0000 7f1afda00640 1 -- 172.21.15.90:0/2494604386 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1af819fbd0 con 0x7f1af8108fe0 2024-04-02T14:43:47.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.085+0000 7f1afda00640 1 -- 172.21.15.90:0/2494604386 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1af81a73c0 con 0x7f1af8108fe0 2024-04-02T14:43:47.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.092+0000 7f1aef400640 1 -- 172.21.15.90:0/2494604386 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f1af000f5f0 con 0x7f1af8108fe0 2024-04-02T14:43:47.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.092+0000 7f1b005f3740 1 Processor -- start 2024-04-02T14:43:47.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.093+0000 7f1b005f3740 1 -- start start 2024-04-02T14:43:47.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.095+0000 7f1b005f3740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555fb9ec09f0 0x555fb9ebf740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:47.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.095+0000 7f1b005f3740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555fb9ebfd10 0x555fb9ec0190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:47.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.095+0000 7f1b005f3740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555fb9ec9080 0x555fb9ecb470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:47.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.095+0000 7f1b005f3740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x555fb9e00400 con 0x555fb9ec9080 2024-04-02T14:43:47.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.095+0000 7f1b005f3740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x555fb9de4f10 con 0x555fb9ebfd10 2024-04-02T14:43:47.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.095+0000 7f1b005f3740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x555fb9e246f0 con 0x555fb9ec09f0 2024-04-02T14:43:47.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.095+0000 7f1af7400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555fb9ec09f0 0x555fb9ebf740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:47.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.095+0000 7f1af6a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555fb9ebfd10 0x555fb9ec0190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:47.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.095+0000 7f1af6000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555fb9ec9080 0x555fb9ecb470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:47.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.095+0000 7f1af7400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555fb9ec09f0 0x555fb9ebf740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:52602/0 (socket says 172.21.15.90:52602) 2024-04-02T14:43:47.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.095+0000 7f1af7400640 1 -- 172.21.15.90:0/3046570174 learned_addr learned my addr 172.21.15.90:0/3046570174 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:43:47.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.096+0000 7f1aef400640 1 --2- 172.21.15.90:0/2494604386 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f1adc0a3ef0 0x7f1adc0a63b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:47.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.096+0000 7f1af6a00640 1 -- 172.21.15.90:0/3046570174 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555fb9ec09f0 msgr2=0x555fb9ebf740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:43:47.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.096+0000 7f1af6a00640 1 --2- 172.21.15.90:0/3046570174 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555fb9ec09f0 0x555fb9ebf740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:47.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.096+0000 7f1af6a00640 1 -- 172.21.15.90:0/3046570174 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555fb9ec9080 msgr2=0x555fb9ecb470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:43:47.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.096+0000 7f1af6a00640 1 --2- 172.21.15.90:0/3046570174 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555fb9ec9080 0x555fb9ecb470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:47.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.096+0000 7f1af6a00640 1 -- 172.21.15.90:0/3046570174 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555fb9b383f0 con 0x555fb9ebfd10 2024-04-02T14:43:47.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.097+0000 7f1af7400640 1 --2- 172.21.15.90:0/2494604386 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f1adc0a3ef0 0x7f1adc0a63b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:47.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.097+0000 7f1aef400640 1 -- 172.21.15.90:0/2494604386 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8550+0+0 (secure 0 0 0) 0x7f1af00d0b90 con 0x7f1af8108fe0 2024-04-02T14:43:47.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.098+0000 7f1af6a00640 1 --2- 172.21.15.90:0/3046570174 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555fb9ebfd10 0x555fb9ec0190 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f1ae80058c0 tx=0x7f1ae800bc70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d3e88b0ec0644931 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:43:47.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.099+0000 7f1ae3e00640 1 -- 172.21.15.90:0/3046570174 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ae8003dd0 con 0x555fb9ebfd10 2024-04-02T14:43:47.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.099+0000 7f1ae3e00640 1 -- 172.21.15.90:0/3046570174 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1ae8015070 con 0x555fb9ebfd10 2024-04-02T14:43:47.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.099+0000 7f1ae3e00640 1 -- 172.21.15.90:0/3046570174 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ae8011730 con 0x555fb9ebfd10 2024-04-02T14:43:47.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.100+0000 7f1b005f3740 1 -- 172.21.15.90:0/3046570174 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555fb9ebfd10 msgr2=0x555fb9ec0190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:47.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.100+0000 7f1b005f3740 1 --2- 172.21.15.90:0/3046570174 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555fb9ebfd10 0x555fb9ec0190 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f1ae80058c0 tx=0x7f1ae800bc70 comp rx=0 tx=0).stop 2024-04-02T14:43:47.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.101+0000 7f1af7400640 1 --2- 172.21.15.90:0/2494604386 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f1adc0a3ef0 0x7f1adc0a63b0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f1ae40095f0 tx=0x7f1ae402e040 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:43:47.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.101+0000 7f1b005f3740 1 -- 172.21.15.90:0/3046570174 shutdown_connections 2024-04-02T14:43:47.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.101+0000 7f1b005f3740 1 --2- 172.21.15.90:0/3046570174 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555fb9ec9080 0x555fb9ecb470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:47.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.101+0000 7f1b005f3740 1 --2- 172.21.15.90:0/3046570174 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555fb9ebfd10 0x555fb9ec0190 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:47.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.101+0000 7f1b005f3740 1 --2- 172.21.15.90:0/3046570174 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555fb9ec09f0 0x555fb9ebf740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:47.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.101+0000 7f1b005f3740 1 -- 172.21.15.90:0/3046570174 >> 172.21.15.90:0/3046570174 conn(0x555fb9e86a50 msgr2=0x555fb9eba760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:43:47.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.101+0000 7f1b005f3740 1 -- 172.21.15.90:0/3046570174 shutdown_connections 2024-04-02T14:43:47.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.101+0000 7f1b005f3740 1 -- 172.21.15.90:0/3046570174 wait complete. 2024-04-02T14:43:47.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.102+0000 7f1b005f3740 1 -- start start 2024-04-02T14:43:47.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.104+0000 7f1b005f3740 1 Processor -- start 2024-04-02T14:43:47.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.105+0000 7f1b005f3740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555fb9ebfd10 0x555fb9ec5f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:47.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.108+0000 7f1b005f3740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555fb9ec09f0 0x555fb9ec64c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:47.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.108+0000 7f1b005f3740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555fb9ec9080 0x555fb9ec6a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:47.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.108+0000 7f1b005f3740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x555fb9e206a0 con 0x555fb9ebfd10 2024-04-02T14:43:47.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.108+0000 7f1b005f3740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x555fb9dcc980 con 0x555fb9ec09f0 2024-04-02T14:43:47.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.108+0000 7f1b005f3740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x555fb9ddfcb0 con 0x555fb9ec9080 2024-04-02T14:43:47.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.108+0000 7f1b005f3740 10 client.0 _ll_register_callbacks cb 0x555fb9ddf6a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:43:47.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.109+0000 7f1af6000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555fb9ec09f0 0x555fb9ec64c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:47.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.109+0000 7f1af7400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555fb9ec9080 0x555fb9ec6a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:47.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.109+0000 7f1af7400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555fb9ec9080 0x555fb9ec6a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:52614/0 (socket says 172.21.15.90:52614) 2024-04-02T14:43:47.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.109+0000 7f1af7400640 1 -- 172.21.15.90:0/1868752868 learned_addr learned my addr 172.21.15.90:0/1868752868 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:43:47.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.110+0000 7f1af6a00640 1 --2- 172.21.15.90:0/1868752868 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555fb9ebfd10 0x555fb9ec5f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:47.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.110+0000 7f1af6000640 1 -- 172.21.15.90:0/1868752868 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555fb9ec9080 msgr2=0x555fb9ec6a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:43:47.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.110+0000 7f1af6000640 1 --2- 172.21.15.90:0/1868752868 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555fb9ec9080 0x555fb9ec6a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:47.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.110+0000 7f1af6000640 1 -- 172.21.15.90:0/1868752868 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555fb9ebfd10 msgr2=0x555fb9ec5f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:43:47.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.110+0000 7f1af6000640 1 --2- 172.21.15.90:0/1868752868 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555fb9ebfd10 0x555fb9ec5f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:47.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.110+0000 7f1af6000640 1 -- 172.21.15.90:0/1868752868 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555fb9c027d0 con 0x555fb9ec09f0 2024-04-02T14:43:47.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.111+0000 7f1af6000640 1 --2- 172.21.15.90:0/1868752868 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555fb9ec09f0 0x555fb9ec64c0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f1af00ce260 tx=0x7f1af0018610 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d7159a673a450588 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:43:47.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.111+0000 7f1ae0c00640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:43:47.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.112+0000 7f1ae0c00640 1 -- 172.21.15.90:0/1868752868 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1af009cab0 con 0x555fb9ec09f0 2024-04-02T14:43:47.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.112+0000 7f1ae0c00640 1 -- 172.21.15.90:0/1868752868 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8550+0+0 (secure 0 0 0) 0x7f1af000fc80 con 0x555fb9ec09f0 2024-04-02T14:43:47.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.112+0000 7f1b005f3740 1 -- 172.21.15.90:0/1868752868 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1ae8015070 con 0x555fb9ec09f0 2024-04-02T14:43:47.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.114+0000 7f1ae0c00640 1 -- 172.21.15.90:0/1868752868 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f1af00a0e60 con 0x555fb9ec09f0 2024-04-02T14:43:47.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.114+0000 7f1b005f3740 10 client.28229 fetch_fsmap learned FSMap version 27 2024-04-02T14:43:47.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.114+0000 7f1b005f3740 1 -- 172.21.15.90:0/1868752868 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x555fb9ef16d0 con 0x555fb9ec09f0 2024-04-02T14:43:47.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.116+0000 7f1ae0c00640 1 -- 172.21.15.90:0/1868752868 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1ae8015070 con 0x555fb9ec09f0 2024-04-02T14:43:47.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.116+0000 7f1ae0c00640 1 -- 172.21.15.90:0/1868752868 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f1af0020030 con 0x555fb9ec09f0 2024-04-02T14:43:47.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.116+0000 7f1b005f3740 10 client.28229 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:43:47.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.116+0000 7f1b005f3740 10 client.28229 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-04-02T14:43:47.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.116+0000 7f1b005f3740 20 client.28229 populate_metadata read hostname 'smithi090' 2024-04-02T14:43:47.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.116+0000 7f1b005f3740 1 --2- 172.21.15.90:0/1868752868 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x555fb9ef2b90 0x555fb9ef4f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:47.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.117+0000 7f1b005f3740 4 client.28229 mds_command: new command op to 24547 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:43:47.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.117+0000 7f1b005f3740 1 -- 172.21.15.90:0/1868752868 --> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555fb9de4040 con 0x555fb9ef2b90 2024-04-02T14:43:47.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.117+0000 7f1af6a00640 1 --2- 172.21.15.90:0/1868752868 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x555fb9ef2b90 0x555fb9ef4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:47.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.118+0000 7f1af6a00640 1 --2- 172.21.15.90:0/1868752868 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x555fb9ef2b90 0x555fb9ef4f80 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=28f9d0e1ae1a7303 server_cookie=823e3abcfe0e02c6 in_seq=0 out_seq=0 2024-04-02T14:43:47.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.118+0000 7f1ae0c00640 10 client.28229 ms_handle_connect on v2:172.21.15.177:6838/664895663 2024-04-02T14:43:47.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.118+0000 7f1ae0c00640 1 -- 172.21.15.90:0/1868752868 <== mds.2 v2:172.21.15.177:6838/664895663 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x555fb9de4040 con 0x555fb9ef2b90 2024-04-02T14:43:47.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.118+0000 7f1ae0c00640 10 client.28229 handle_command_reply: tid=0 2024-04-02T14:43:47.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.118+0000 7f1b005f3740 1 client.28229 shutdown 2024-04-02T14:43:47.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.120+0000 7f1b005f3740 1 -- 172.21.15.90:0/1868752868 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555fb9ec09f0 msgr2=0x555fb9ec64c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:47.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.120+0000 7f1b005f3740 1 --2- 172.21.15.90:0/1868752868 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555fb9ec09f0 0x555fb9ec64c0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f1af00ce260 tx=0x7f1af0018610 comp rx=0 tx=0).stop 2024-04-02T14:43:47.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.120+0000 7f1b005f3740 1 -- 172.21.15.90:0/1868752868 shutdown_connections 2024-04-02T14:43:47.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.120+0000 7f1b005f3740 1 --2- 172.21.15.90:0/1868752868 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x555fb9ef2b90 0x555fb9ef4f80 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:47.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.120+0000 7f1b005f3740 1 --2- 172.21.15.90:0/1868752868 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555fb9ec9080 0x555fb9ec6a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:47.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.120+0000 7f1b005f3740 1 --2- 172.21.15.90:0/1868752868 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555fb9ec09f0 0x555fb9ec64c0 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:47.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.120+0000 7f1b005f3740 1 --2- 172.21.15.90:0/1868752868 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555fb9ebfd10 0x555fb9ec5f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:47.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.120+0000 7f1ae0c00640 0 client.28229 ms_handle_reset on v2:172.21.15.177:6838/664895663 2024-04-02T14:43:47.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.120+0000 7f1b005f3740 1 -- 172.21.15.90:0/1868752868 >> 172.21.15.90:0/1868752868 conn(0x555fb9e86a50 msgr2=0x555fb9ebc4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:43:47.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.120+0000 7f1b005f3740 1 -- 172.21.15.90:0/1868752868 shutdown_connections 2024-04-02T14:43:47.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.120+0000 7f1b005f3740 1 -- 172.21.15.90:0/1868752868 wait complete. 2024-04-02T14:43:47.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.121+0000 7f1b005f3740 20 client.28229 trim_cache size 0 max 16384 2024-04-02T14:43:47.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.152+0000 7f1b005f3740 1 Processor -- start 2024-04-02T14:43:47.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.153+0000 7f1b005f3740 1 -- start start 2024-04-02T14:43:47.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.153+0000 7f1b005f3740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555fb9ebfd10 0x555fb9ec86b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:47.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.153+0000 7f1b005f3740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555fb9ec09f0 0x555fb9ec6650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:47.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.153+0000 7f1b005f3740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555fb9ec9080 0x555fb9efa810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:47.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.153+0000 7f1b005f3740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f1af806f560 con 0x555fb9ebfd10 2024-04-02T14:43:47.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.153+0000 7f1b005f3740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f1af80710a0 con 0x555fb9ec09f0 2024-04-02T14:43:47.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.153+0000 7f1b005f3740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f1af806fc30 con 0x555fb9ec9080 2024-04-02T14:43:47.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.154+0000 7f1af6a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555fb9ebfd10 0x555fb9ec86b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:47.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.154+0000 7f1af6a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555fb9ebfd10 0x555fb9ec86b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33714/0 (socket says 172.21.15.90:33714) 2024-04-02T14:43:47.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.154+0000 7f1af6000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555fb9ec09f0 0x555fb9ec6650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:47.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.154+0000 7f1af6a00640 1 -- 172.21.15.90:0/1371802125 learned_addr learned my addr 172.21.15.90:0/1371802125 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:43:47.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.155+0000 7f1af7400640 1 --2- 172.21.15.90:0/1371802125 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555fb9ec9080 0x555fb9efa810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:47.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.155+0000 7f1af6000640 1 -- 172.21.15.90:0/1371802125 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555fb9ec9080 msgr2=0x555fb9efa810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:43:47.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.155+0000 7f1af6000640 1 --2- 172.21.15.90:0/1371802125 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555fb9ec9080 0x555fb9efa810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:47.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.155+0000 7f1af6000640 1 -- 172.21.15.90:0/1371802125 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555fb9ebfd10 msgr2=0x555fb9ec86b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:43:47.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.155+0000 7f1af6000640 1 --2- 172.21.15.90:0/1371802125 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555fb9ebfd10 0x555fb9ec86b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:47.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.155+0000 7f1af6000640 1 -- 172.21.15.90:0/1371802125 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555fb9eba970 con 0x555fb9ec09f0 2024-04-02T14:43:47.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.155+0000 7f1af7400640 1 --2- 172.21.15.90:0/1371802125 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555fb9ec9080 0x555fb9efa810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:43:47.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.155+0000 7f1af6a00640 1 --2- 172.21.15.90:0/1371802125 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555fb9ebfd10 0x555fb9ec86b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:43:47.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.156+0000 7f1af6000640 1 --2- 172.21.15.90:0/1371802125 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555fb9ec09f0 0x555fb9ec6650 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f1af00078a0 tx=0x7f1af0000f40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=51eee956f07852ca server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:43:47.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.156+0000 7f1ae3400640 1 -- 172.21.15.90:0/1371802125 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1af009c410 con 0x555fb9ec09f0 2024-04-02T14:43:47.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.156+0000 7f1ae3400640 1 -- 172.21.15.90:0/1371802125 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1af00d8040 con 0x555fb9ec09f0 2024-04-02T14:43:47.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.157+0000 7f1ae3400640 1 -- 172.21.15.90:0/1371802125 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1af000f8e0 con 0x555fb9ec09f0 2024-04-02T14:43:47.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.158+0000 7f1b005f3740 1 -- 172.21.15.90:0/1371802125 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555fb9ec09f0 msgr2=0x555fb9ec6650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:47.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.158+0000 7f1b005f3740 1 --2- 172.21.15.90:0/1371802125 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555fb9ec09f0 0x555fb9ec6650 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f1af00078a0 tx=0x7f1af0000f40 comp rx=0 tx=0).stop 2024-04-02T14:43:47.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.159+0000 7f1b005f3740 1 -- 172.21.15.90:0/1371802125 shutdown_connections 2024-04-02T14:43:47.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.159+0000 7f1b005f3740 1 --2- 172.21.15.90:0/1371802125 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555fb9ec9080 0x555fb9efa810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:47.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.159+0000 7f1b005f3740 1 --2- 172.21.15.90:0/1371802125 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555fb9ec09f0 0x555fb9ec6650 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:47.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.159+0000 7f1b005f3740 1 --2- 172.21.15.90:0/1371802125 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555fb9ebfd10 0x555fb9ec86b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:47.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.159+0000 7f1b005f3740 1 -- 172.21.15.90:0/1371802125 >> 172.21.15.90:0/1371802125 conn(0x555fb9e86a50 msgr2=0x555fb9ec5f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:43:47.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.159+0000 7f1b005f3740 1 -- 172.21.15.90:0/1371802125 shutdown_connections 2024-04-02T14:43:47.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.160+0000 7f1b005f3740 1 -- 172.21.15.90:0/1371802125 wait complete. 2024-04-02T14:43:47.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.160+0000 7f1b005f3740 1 -- start start 2024-04-02T14:43:47.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.160+0000 7f1b005f3740 1 Processor -- start 2024-04-02T14:43:47.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.161+0000 7f1b005f3740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555fb9ebfd10 0x555fb9eead60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:47.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.161+0000 7f1b005f3740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555fb9ec09f0 0x555fb9ec2a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:47.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.161+0000 7f1b005f3740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555fb9ec9080 0x555fb9ec3090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:47.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.161+0000 7f1b005f3740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f1af80709d0 con 0x555fb9ec9080 2024-04-02T14:43:47.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.161+0000 7f1b005f3740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f1af8070300 con 0x555fb9ec09f0 2024-04-02T14:43:47.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.161+0000 7f1b005f3740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f1af806d560 con 0x555fb9ebfd10 2024-04-02T14:43:47.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.161+0000 7f1b005f3740 10 client.0 _ll_register_callbacks cb 0x555fb9ddf6a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:43:47.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.161+0000 7f1af6a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555fb9ebfd10 0x555fb9eead60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:47.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.162+0000 7f1af6000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555fb9ec09f0 0x555fb9ec2a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:47.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.162+0000 7f1af6a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555fb9ebfd10 0x555fb9eead60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:52636/0 (socket says 172.21.15.90:52636) 2024-04-02T14:43:47.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.162+0000 7f1af6a00640 1 -- 172.21.15.90:0/4064866420 learned_addr learned my addr 172.21.15.90:0/4064866420 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:43:47.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.162+0000 7f1af7400640 1 --2- 172.21.15.90:0/4064866420 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555fb9ec9080 0x555fb9ec3090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:47.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.162+0000 7f1af6a00640 1 -- 172.21.15.90:0/4064866420 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555fb9ec09f0 msgr2=0x555fb9ec2a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:43:47.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.162+0000 7f1af6a00640 1 --2- 172.21.15.90:0/4064866420 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555fb9ec09f0 0x555fb9ec2a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:47.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.162+0000 7f1af6a00640 1 -- 172.21.15.90:0/4064866420 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555fb9ec9080 msgr2=0x555fb9ec3090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:43:47.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.162+0000 7f1af6a00640 1 --2- 172.21.15.90:0/4064866420 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555fb9ec9080 0x555fb9ec3090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:47.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.162+0000 7f1af6a00640 1 -- 172.21.15.90:0/4064866420 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555fb9ec38c0 con 0x555fb9ebfd10 2024-04-02T14:43:47.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.162+0000 7f1af7400640 1 --2- 172.21.15.90:0/4064866420 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555fb9ec9080 0x555fb9ec3090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:43:47.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.162+0000 7f1af6a00640 1 --2- 172.21.15.90:0/4064866420 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555fb9ebfd10 0x555fb9eead60 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f1ae8005f30 tx=0x7f1ae8013040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=46a35548ffdc5128 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:43:47.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.162+0000 7f1ae0c00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:43:47.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.163+0000 7f1ae0c00640 1 -- 172.21.15.90:0/4064866420 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ae80283c0 con 0x555fb9ebfd10 2024-04-02T14:43:47.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.163+0000 7f1ae0c00640 1 -- 172.21.15.90:0/4064866420 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8550+0+0 (secure 0 0 0) 0x7f1ae8028560 con 0x555fb9ebfd10 2024-04-02T14:43:47.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.163+0000 7f1b005f3740 1 -- 172.21.15.90:0/4064866420 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1af00d8040 con 0x555fb9ebfd10 2024-04-02T14:43:47.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.163+0000 7f1ae0c00640 1 -- 172.21.15.90:0/4064866420 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f1ae8028760 con 0x555fb9ebfd10 2024-04-02T14:43:47.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.163+0000 7f1b005f3740 10 client.28069 fetch_fsmap learned FSMap version 27 2024-04-02T14:43:47.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.163+0000 7f1b005f3740 1 -- 172.21.15.90:0/4064866420 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x555fb9f59150 con 0x555fb9ebfd10 2024-04-02T14:43:47.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.163+0000 7f1ae0c00640 1 -- 172.21.15.90:0/4064866420 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1af00d8040 con 0x555fb9ebfd10 2024-04-02T14:43:47.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.164+0000 7f1ae0c00640 1 -- 172.21.15.90:0/4064866420 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f1ae80117a0 con 0x555fb9ebfd10 2024-04-02T14:43:47.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.164+0000 7f1b005f3740 10 client.28069 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:43:47.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.164+0000 7f1b005f3740 10 client.28069 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-04-02T14:43:47.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.164+0000 7f1b005f3740 20 client.28069 populate_metadata read hostname 'smithi090' 2024-04-02T14:43:47.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.164+0000 7f1b005f3740 1 --2- 172.21.15.90:0/4064866420 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x555fb9f59600 0x555fb9f5ba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:47.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.164+0000 7f1b005f3740 4 client.28069 mds_command: new command op to 24547 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-02T14:43:47.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.164+0000 7f1b005f3740 1 -- 172.21.15.90:0/4064866420 --> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] -- command(tid 0: {"prefix": "damage ls"}) -- 0x555fb9e31c30 con 0x555fb9f59600 2024-04-02T14:43:47.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.165+0000 7f1af6000640 1 --2- 172.21.15.90:0/4064866420 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x555fb9f59600 0x555fb9f5ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:47.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.167+0000 7f1ae0c00640 10 client.28069 ms_handle_connect on v2:172.21.15.177:6838/664895663 2024-04-02T14:43:47.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.167+0000 7f1af6000640 1 --2- 172.21.15.90:0/4064866420 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x555fb9f59600 0x555fb9f5ba20 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=bb62bdcbb1ad0e5f server_cookie=c83e340fb6a258de in_seq=0 out_seq=0 2024-04-02T14:43:47.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.168+0000 7f1ae0c00640 1 -- 172.21.15.90:0/4064866420 <== mds.2 v2:172.21.15.177:6838/664895663 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x555fb9e31c30 con 0x555fb9f59600 2024-04-02T14:43:47.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.168+0000 7f1ae0c00640 10 client.28069 handle_command_reply: tid=0 2024-04-02T14:43:47.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.168+0000 7f1b005f3740 1 client.28069 shutdown 2024-04-02T14:43:47.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.168+0000 7f1b005f3740 1 -- 172.21.15.90:0/4064866420 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555fb9ebfd10 msgr2=0x555fb9eead60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:47.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.168+0000 7f1b005f3740 1 --2- 172.21.15.90:0/4064866420 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555fb9ebfd10 0x555fb9eead60 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f1ae8005f30 tx=0x7f1ae8013040 comp rx=0 tx=0).stop 2024-04-02T14:43:47.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.168+0000 7f1b005f3740 1 -- 172.21.15.90:0/4064866420 shutdown_connections 2024-04-02T14:43:47.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.168+0000 7f1b005f3740 1 --2- 172.21.15.90:0/4064866420 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x555fb9f59600 0x555fb9f5ba20 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:47.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.168+0000 7f1b005f3740 1 --2- 172.21.15.90:0/4064866420 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555fb9ec9080 0x555fb9ec3090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:47.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.168+0000 7f1b005f3740 1 --2- 172.21.15.90:0/4064866420 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555fb9ec09f0 0x555fb9ec2a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:47.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.168+0000 7f1ae0c00640 0 client.28069 ms_handle_reset on v2:172.21.15.177:6838/664895663 2024-04-02T14:43:47.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.168+0000 7f1b005f3740 1 --2- 172.21.15.90:0/4064866420 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555fb9ebfd10 0x555fb9eead60 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:47.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.168+0000 7f1b005f3740 1 -- 172.21.15.90:0/4064866420 >> 172.21.15.90:0/4064866420 conn(0x555fb9e86a50 msgr2=0x555fb9eb9010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:43:47.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.169+0000 7f1b005f3740 1 -- 172.21.15.90:0/4064866420 shutdown_connections 2024-04-02T14:43:47.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.169+0000 7f1b005f3740 1 -- 172.21.15.90:0/4064866420 wait complete. 2024-04-02T14:43:47.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.170+0000 7f1b005f3740 20 client.28069 trim_cache size 0 max 16384 2024-04-02T14:43:47.167 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-02T14:43:47.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.171+0000 7f1afda00640 1 -- 172.21.15.90:0/2494604386 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f1adc0a3ef0 msgr2=0x7f1adc0a63b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:47.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.171+0000 7f1afda00640 1 --2- 172.21.15.90:0/2494604386 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f1adc0a3ef0 0x7f1adc0a63b0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f1ae40095f0 tx=0x7f1ae402e040 comp rx=0 tx=0).stop 2024-04-02T14:43:47.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.171+0000 7f1afda00640 1 -- 172.21.15.90:0/2494604386 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1af8108fe0 msgr2=0x7f1af819aa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:47.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.171+0000 7f1afda00640 1 --2- 172.21.15.90:0/2494604386 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1af8108fe0 0x7f1af819aa70 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f1af000ccb0 tx=0x7f1af000a620 comp rx=0 tx=0).stop 2024-04-02T14:43:47.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.172+0000 7f1afda00640 1 -- 172.21.15.90:0/2494604386 shutdown_connections 2024-04-02T14:43:47.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.172+0000 7f1afda00640 1 --2- 172.21.15.90:0/2494604386 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f1adc0a3ef0 0x7f1adc0a63b0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:47.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.172+0000 7f1afda00640 1 --2- 172.21.15.90:0/2494604386 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1af819edf0 0x7f1af819f2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:47.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.172+0000 7f1afda00640 1 --2- 172.21.15.90:0/2494604386 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1af8108fe0 0x7f1af819aa70 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:47.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.172+0000 7f1afda00640 1 --2- 172.21.15.90:0/2494604386 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1af8103f10 0x7f1af819a530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:47.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.172+0000 7f1afda00640 1 -- 172.21.15.90:0/2494604386 >> 172.21.15.90:0/2494604386 conn(0x7f1af80ff7b0 msgr2=0x7f1af81002f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:43:47.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.172+0000 7f1afda00640 1 -- 172.21.15.90:0/2494604386 shutdown_connections 2024-04-02T14:43:47.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.172+0000 7f1afda00640 1 -- 172.21.15.90:0/2494604386 wait complete. 2024-04-02T14:43:47.195 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-02T14:43:47.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.548+0000 7fd070000640 1 -- 172.21.15.90:0/1063061451 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd068103f10 msgr2=0x7fd0681042f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:47.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.548+0000 7fd070000640 1 --2- 172.21.15.90:0/1063061451 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd068103f10 0x7fd0681042f0 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7fd05c01a910 tx=0x7fd05c03cc60 comp rx=0 tx=0).stop 2024-04-02T14:43:47.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.548+0000 7fd070000640 1 -- 172.21.15.90:0/1063061451 shutdown_connections 2024-04-02T14:43:47.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.548+0000 7fd070000640 1 --2- 172.21.15.90:0/1063061451 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd068108fe0 0x7fd068110bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:47.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.548+0000 7fd070000640 1 --2- 172.21.15.90:0/1063061451 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd0681048c0 0x7fd068108900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:47.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.548+0000 7fd070000640 1 --2- 172.21.15.90:0/1063061451 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd068103f10 0x7fd0681042f0 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:47.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.548+0000 7fd070000640 1 -- 172.21.15.90:0/1063061451 >> 172.21.15.90:0/1063061451 conn(0x7fd0680ff7b0 msgr2=0x7fd068101bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:43:47.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.549+0000 7fd070000640 1 -- 172.21.15.90:0/1063061451 shutdown_connections 2024-04-02T14:43:47.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.549+0000 7fd070000640 1 -- 172.21.15.90:0/1063061451 wait complete. 2024-04-02T14:43:47.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.551+0000 7fd070000640 1 Processor -- start 2024-04-02T14:43:47.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.551+0000 7fd070000640 1 -- start start 2024-04-02T14:43:47.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.552+0000 7fd070000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd068103f10 0x7fd0681107b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:47.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.552+0000 7fd070000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd068108fe0 0x7fd06810c840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:47.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.553+0000 7fd070000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd06810cd80 0x7fd06810d230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:47.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.553+0000 7fd070000640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fd06806df00 con 0x7fd068103f10 2024-04-02T14:43:47.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.553+0000 7fd070000640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fd06806dd80 con 0x7fd068108fe0 2024-04-02T14:43:47.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.553+0000 7fd070000640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fd06806e080 con 0x7fd06810cd80 2024-04-02T14:43:47.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.553+0000 7fd06e000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd06810cd80 0x7fd06810d230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:47.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.553+0000 7fd06e000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd06810cd80 0x7fd06810d230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:52666/0 (socket says 172.21.15.90:52666) 2024-04-02T14:43:47.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.553+0000 7fd06e000640 1 -- 172.21.15.90:0/3269904659 learned_addr learned my addr 172.21.15.90:0/3269904659 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:43:47.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.553+0000 7fd06d600640 1 --2- 172.21.15.90:0/3269904659 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd068103f10 0x7fd0681107b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:47.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.553+0000 7fd06cc00640 1 --2- 172.21.15.90:0/3269904659 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd068108fe0 0x7fd06810c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:47.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.554+0000 7fd06e000640 1 -- 172.21.15.90:0/3269904659 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd068108fe0 msgr2=0x7fd06810c840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:47.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.554+0000 7fd06e000640 1 --2- 172.21.15.90:0/3269904659 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd068108fe0 0x7fd06810c840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:47.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.554+0000 7fd06e000640 1 -- 172.21.15.90:0/3269904659 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd068103f10 msgr2=0x7fd0681107b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:47.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.554+0000 7fd06e000640 1 --2- 172.21.15.90:0/3269904659 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd068103f10 0x7fd0681107b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:47.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.554+0000 7fd06e000640 1 -- 172.21.15.90:0/3269904659 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd06819e460 con 0x7fd06810cd80 2024-04-02T14:43:47.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.554+0000 7fd06e000640 1 --2- 172.21.15.90:0/3269904659 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd06810cd80 0x7fd06810d230 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7fd06000cce0 tx=0x7fd06000a620 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:43:47.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.555+0000 7fd066000640 1 -- 172.21.15.90:0/3269904659 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd06000f450 con 0x7fd06810cd80 2024-04-02T14:43:47.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.555+0000 7fd066000640 1 -- 172.21.15.90:0/3269904659 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd06000fad0 con 0x7fd06810cd80 2024-04-02T14:43:47.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.555+0000 7fd066000640 1 -- 172.21.15.90:0/3269904659 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd060018c10 con 0x7fd06810cd80 2024-04-02T14:43:47.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.555+0000 7fd070000640 1 -- 172.21.15.90:0/3269904659 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd06819e6f0 con 0x7fd06810cd80 2024-04-02T14:43:47.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.555+0000 7fd070000640 1 -- 172.21.15.90:0/3269904659 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd06819eb60 con 0x7fd06810cd80 2024-04-02T14:43:47.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.560+0000 7fd066000640 1 -- 172.21.15.90:0/3269904659 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fd06000f5f0 con 0x7fd06810cd80 2024-04-02T14:43:47.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.561+0000 7fd072dfc740 1 Processor -- start 2024-04-02T14:43:47.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.562+0000 7fd072dfc740 1 -- start start 2024-04-02T14:43:47.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.563+0000 7fd072dfc740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d5d5a9b200 0x55d5d5a99f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:47.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.563+0000 7fd072dfc740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d5d5a9a520 0x55d5d5a9a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:47.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.563+0000 7fd072dfc740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d5d5a94190 0x55d5d5aae7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:47.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.563+0000 7fd072dfc740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55d5d59d2400 con 0x55d5d5a9b200 2024-04-02T14:43:47.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.563+0000 7fd072dfc740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55d5d59b6f10 con 0x55d5d5a94190 2024-04-02T14:43:47.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.563+0000 7fd072dfc740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55d5d59f66f0 con 0x55d5d5a9a520 2024-04-02T14:43:47.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.564+0000 7fd06d600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d5d5a9a520 0x55d5d5a9a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:47.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.564+0000 7fd06d600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d5d5a9a520 0x55d5d5a9a9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:52674/0 (socket says 172.21.15.90:52674) 2024-04-02T14:43:47.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.564+0000 7fd06d600640 1 -- 172.21.15.90:0/547559010 learned_addr learned my addr 172.21.15.90:0/547559010 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:43:47.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.565+0000 7fd066000640 1 --2- 172.21.15.90:0/3269904659 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fd0500a3da0 0x7fd0500a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:47.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.565+0000 7fd06cc00640 1 --2- 172.21.15.90:0/547559010 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d5d5a94190 0x55d5d5aae7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:47.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.565+0000 7fd06cc00640 1 --2- 172.21.15.90:0/547559010 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d5d5a9b200 0x55d5d5a99f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:47.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.566+0000 7fd06d600640 1 -- 172.21.15.90:0/547559010 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d5d5a94190 msgr2=0x55d5d5aae7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:43:47.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.566+0000 7fd06d600640 1 --2- 172.21.15.90:0/547559010 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d5d5a94190 0x55d5d5aae7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:47.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.566+0000 7fd06d600640 1 -- 172.21.15.90:0/547559010 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d5d5a9b200 msgr2=0x55d5d5a99f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:43:47.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.566+0000 7fd06d600640 1 --2- 172.21.15.90:0/547559010 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d5d5a9b200 0x55d5d5a99f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:47.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.566+0000 7fd066000640 1 -- 172.21.15.90:0/3269904659 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8550+0+0 (secure 0 0 0) 0x7fd0600d0cf0 con 0x7fd06810cd80 2024-04-02T14:43:47.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.566+0000 7fd06d600640 1 -- 172.21.15.90:0/547559010 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d5d570a3f0 con 0x55d5d5a9a520 2024-04-02T14:43:47.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.567+0000 7fd06cc00640 1 --2- 172.21.15.90:0/547559010 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d5d5a9b200 0x55d5d5a99f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:43:47.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.567+0000 7fd06d600640 1 --2- 172.21.15.90:0/547559010 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d5d5a9a520 0x55d5d5a9a9a0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7fd05c047040 tx=0x7fd05c004810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b1d4d9eb530d85e server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:43:47.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.567+0000 7fd06e000640 1 --2- 172.21.15.90:0/3269904659 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fd0500a3da0 0x7fd0500a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:47.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.568+0000 7fd05a600640 1 -- 172.21.15.90:0/547559010 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd05c004200 con 0x55d5d5a9a520 2024-04-02T14:43:47.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.568+0000 7fd05a600640 1 -- 172.21.15.90:0/547559010 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd05c0043a0 con 0x55d5d5a9a520 2024-04-02T14:43:47.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.568+0000 7fd05a600640 1 -- 172.21.15.90:0/547559010 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd05c004200 con 0x55d5d5a9a520 2024-04-02T14:43:47.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.569+0000 7fd072dfc740 1 -- 172.21.15.90:0/547559010 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d5d5a9a520 msgr2=0x55d5d5a9a9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:47.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.569+0000 7fd072dfc740 1 --2- 172.21.15.90:0/547559010 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d5d5a9a520 0x55d5d5a9a9a0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7fd05c047040 tx=0x7fd05c004810 comp rx=0 tx=0).stop 2024-04-02T14:43:47.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.569+0000 7fd072dfc740 1 -- 172.21.15.90:0/547559010 shutdown_connections 2024-04-02T14:43:47.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.569+0000 7fd072dfc740 1 --2- 172.21.15.90:0/547559010 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d5d5a94190 0x55d5d5aae7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:47.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.569+0000 7fd072dfc740 1 --2- 172.21.15.90:0/547559010 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d5d5a9a520 0x55d5d5a9a9a0 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:47.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.569+0000 7fd072dfc740 1 --2- 172.21.15.90:0/547559010 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d5d5a9b200 0x55d5d5a99f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:47.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.569+0000 7fd072dfc740 1 -- 172.21.15.90:0/547559010 >> 172.21.15.90:0/547559010 conn(0x55d5d5a58a50 msgr2=0x55d5d5a94f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:43:47.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.570+0000 7fd06e000640 1 --2- 172.21.15.90:0/3269904659 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fd0500a3da0 0x7fd0500a6260 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fd06009cc00 tx=0x7fd06000ff60 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:43:47.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.570+0000 7fd072dfc740 1 -- 172.21.15.90:0/547559010 shutdown_connections 2024-04-02T14:43:47.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.570+0000 7fd072dfc740 1 -- 172.21.15.90:0/547559010 wait complete. 2024-04-02T14:43:47.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.570+0000 7fd072dfc740 1 -- start start 2024-04-02T14:43:47.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.571+0000 7fd072dfc740 1 Processor -- start 2024-04-02T14:43:47.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.572+0000 7fd072dfc740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d5d5a94190 0x55d5d5a91180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:47.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.572+0000 7fd072dfc740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d5d5a9a520 0x55d5d5a916c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:47.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.572+0000 7fd072dfc740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d5d5a9b200 0x55d5d5a91c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:47.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.572+0000 7fd072dfc740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55d5d59f26a0 con 0x55d5d5a9a520 2024-04-02T14:43:47.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.572+0000 7fd072dfc740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55d5d599e980 con 0x55d5d5a94190 2024-04-02T14:43:47.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.572+0000 7fd072dfc740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55d5d59b1cb0 con 0x55d5d5a9b200 2024-04-02T14:43:47.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.572+0000 7fd072dfc740 10 client.0 _ll_register_callbacks cb 0x55d5d59b16a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:43:47.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.572+0000 7fd06cc00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d5d5a94190 0x55d5d5a91180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:47.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.572+0000 7fd06cc00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d5d5a94190 0x55d5d5a91180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:36412/0 (socket says 172.21.15.90:36412) 2024-04-02T14:43:47.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.572+0000 7fd06cc00640 1 -- 172.21.15.90:0/2993198809 learned_addr learned my addr 172.21.15.90:0/2993198809 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:43:47.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.573+0000 7fd06cc00640 1 --2- 172.21.15.90:0/2993198809 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d5d5a9b200 0x55d5d5a91c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:47.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.573+0000 7fd06cc00640 1 -- 172.21.15.90:0/2993198809 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d5d5a9b200 msgr2=0x55d5d5a91c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:43:47.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.573+0000 7fd06cc00640 1 --2- 172.21.15.90:0/2993198809 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d5d5a9b200 0x55d5d5a91c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:47.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.573+0000 7fd06cc00640 1 -- 172.21.15.90:0/2993198809 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d5d5a9a520 msgr2=0x55d5d5a916c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:43:47.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.573+0000 7fd06cc00640 1 --2- 172.21.15.90:0/2993198809 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d5d5a9a520 0x55d5d5a916c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:47.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.573+0000 7fd06cc00640 1 -- 172.21.15.90:0/2993198809 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d5d57d47d0 con 0x55d5d5a94190 2024-04-02T14:43:47.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.574+0000 7fd06cc00640 1 --2- 172.21.15.90:0/2993198809 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d5d5a94190 0x55d5d5a91180 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7fd05400f490 tx=0x7fd054004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=af4b8de6d576fac6 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:43:47.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.574+0000 7fd04ea00640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:43:47.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.574+0000 7fd04ea00640 1 -- 172.21.15.90:0/2993198809 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd054016070 con 0x55d5d5a94190 2024-04-02T14:43:47.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.574+0000 7fd04ea00640 1 -- 172.21.15.90:0/2993198809 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8550+0+0 (secure 0 0 0) 0x7fd054004020 con 0x55d5d5a94190 2024-04-02T14:43:47.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.574+0000 7fd072dfc740 1 -- 172.21.15.90:0/2993198809 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd05c0043a0 con 0x55d5d5a94190 2024-04-02T14:43:47.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.574+0000 7fd04ea00640 1 -- 172.21.15.90:0/2993198809 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fd054024050 con 0x55d5d5a94190 2024-04-02T14:43:47.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.574+0000 7fd072dfc740 10 client.28247 fetch_fsmap learned FSMap version 27 2024-04-02T14:43:47.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.574+0000 7fd072dfc740 1 -- 172.21.15.90:0/2993198809 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55d5d5abad10 con 0x55d5d5a94190 2024-04-02T14:43:47.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.575+0000 7fd04ea00640 1 -- 172.21.15.90:0/2993198809 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd05c0043a0 con 0x55d5d5a94190 2024-04-02T14:43:47.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.576+0000 7fd04ea00640 1 -- 172.21.15.90:0/2993198809 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fd05401f440 con 0x55d5d5a94190 2024-04-02T14:43:47.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.577+0000 7fd072dfc740 10 client.28247 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:43:47.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.577+0000 7fd072dfc740 10 client.28247 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2024-04-02T14:43:47.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.577+0000 7fd072dfc740 20 client.28247 populate_metadata read hostname 'smithi090' 2024-04-02T14:43:47.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.577+0000 7fd072dfc740 1 --2- 172.21.15.90:0/2993198809 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x55d5d5abc1d0 0x55d5d5abe600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:47.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.577+0000 7fd072dfc740 4 client.28247 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:43:47.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.577+0000 7fd072dfc740 1 -- 172.21.15.90:0/2993198809 --> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d5d59b6040 con 0x55d5d5abc1d0 2024-04-02T14:43:47.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.578+0000 7fd06d600640 1 --2- 172.21.15.90:0/2993198809 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x55d5d5abc1d0 0x55d5d5abe600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:47.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.579+0000 7fd06d600640 1 --2- 172.21.15.90:0/2993198809 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x55d5d5abc1d0 0x55d5d5abe600 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=c10c8cdf39ba2bf8 server_cookie=dce436cda206b49e in_seq=0 out_seq=0 2024-04-02T14:43:47.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.579+0000 7fd04ea00640 10 client.28247 ms_handle_connect on v2:172.21.15.90:6838/1498915870 2024-04-02T14:43:47.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.580+0000 7fd04ea00640 1 -- 172.21.15.90:0/2993198809 <== mds.1 v2:172.21.15.90:6838/1498915870 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d5d59b6040 con 0x55d5d5abc1d0 2024-04-02T14:43:47.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.580+0000 7fd04ea00640 10 client.28247 handle_command_reply: tid=0 2024-04-02T14:43:47.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.580+0000 7fd072dfc740 1 client.28247 shutdown 2024-04-02T14:43:47.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.581+0000 7fd072dfc740 1 -- 172.21.15.90:0/2993198809 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d5d5a94190 msgr2=0x55d5d5a91180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:47.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.581+0000 7fd072dfc740 1 --2- 172.21.15.90:0/2993198809 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d5d5a94190 0x55d5d5a91180 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7fd05400f490 tx=0x7fd054004730 comp rx=0 tx=0).stop 2024-04-02T14:43:47.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.581+0000 7fd072dfc740 1 -- 172.21.15.90:0/2993198809 shutdown_connections 2024-04-02T14:43:47.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.581+0000 7fd072dfc740 1 --2- 172.21.15.90:0/2993198809 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x55d5d5abc1d0 0x55d5d5abe600 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:47.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.581+0000 7fd072dfc740 1 --2- 172.21.15.90:0/2993198809 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d5d5a9b200 0x55d5d5a91c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:47.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.581+0000 7fd04ea00640 0 client.28247 ms_handle_reset on v2:172.21.15.90:6838/1498915870 2024-04-02T14:43:47.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.581+0000 7fd072dfc740 1 --2- 172.21.15.90:0/2993198809 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d5d5a9a520 0x55d5d5a916c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:47.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.581+0000 7fd072dfc740 1 --2- 172.21.15.90:0/2993198809 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d5d5a94190 0x55d5d5a91180 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:47.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.581+0000 7fd072dfc740 1 -- 172.21.15.90:0/2993198809 >> 172.21.15.90:0/2993198809 conn(0x55d5d5a58a50 msgr2=0x55d5d5a96cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:43:47.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.582+0000 7fd072dfc740 1 -- 172.21.15.90:0/2993198809 shutdown_connections 2024-04-02T14:43:47.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.582+0000 7fd072dfc740 1 -- 172.21.15.90:0/2993198809 wait complete. 2024-04-02T14:43:47.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.582+0000 7fd072dfc740 20 client.28247 trim_cache size 0 max 16384 2024-04-02T14:43:47.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.603+0000 7fd072dfc740 1 Processor -- start 2024-04-02T14:43:47.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.604+0000 7fd072dfc740 1 -- start start 2024-04-02T14:43:47.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.604+0000 7fd072dfc740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d5d5a94190 0x55d5d5ab8650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:47.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.604+0000 7fd072dfc740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d5d5a9a520 0x55d5d5ab7ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:47.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.604+0000 7fd072dfc740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d5d5a9b200 0x55d5d5a90a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:47.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.604+0000 7fd072dfc740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fd0681a2f00 con 0x55d5d5a94190 2024-04-02T14:43:47.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.604+0000 7fd072dfc740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fd0681a4a40 con 0x55d5d5a9a520 2024-04-02T14:43:47.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.604+0000 7fd072dfc740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fd0681a35d0 con 0x55d5d5a9b200 2024-04-02T14:43:47.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.604+0000 7fd06cc00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d5d5a94190 0x55d5d5ab8650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:47.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.604+0000 7fd06cc00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d5d5a94190 0x55d5d5ab8650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33772/0 (socket says 172.21.15.90:33772) 2024-04-02T14:43:47.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.604+0000 7fd06cc00640 1 -- 172.21.15.90:0/432543491 learned_addr learned my addr 172.21.15.90:0/432543491 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:43:47.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.605+0000 7fd06d600640 1 --2- 172.21.15.90:0/432543491 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d5d5a9a520 0x55d5d5ab7ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:47.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.605+0000 7fd06cc00640 1 --2- 172.21.15.90:0/432543491 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d5d5a9b200 0x55d5d5a90a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:47.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.605+0000 7fd06cc00640 1 -- 172.21.15.90:0/432543491 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d5d5a9b200 msgr2=0x55d5d5a90a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:43:47.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.605+0000 7fd06cc00640 1 --2- 172.21.15.90:0/432543491 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d5d5a9b200 0x55d5d5a90a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:47.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.605+0000 7fd06cc00640 1 -- 172.21.15.90:0/432543491 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d5d5a9a520 msgr2=0x55d5d5ab7ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:43:47.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.605+0000 7fd06cc00640 1 --2- 172.21.15.90:0/432543491 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d5d5a9a520 0x55d5d5ab7ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:47.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.605+0000 7fd06cc00640 1 -- 172.21.15.90:0/432543491 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d5d5a99550 con 0x55d5d5a94190 2024-04-02T14:43:47.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.605+0000 7fd06cc00640 1 --2- 172.21.15.90:0/432543491 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d5d5a94190 0x55d5d5ab8650 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fd0540109e0 tx=0x7fd054005dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b12b805a79442c95 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:43:47.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.606+0000 7fd059c00640 1 -- 172.21.15.90:0/432543491 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd054032070 con 0x55d5d5a94190 2024-04-02T14:43:47.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.606+0000 7fd059c00640 1 -- 172.21.15.90:0/432543491 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd05400ab50 con 0x55d5d5a94190 2024-04-02T14:43:47.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.606+0000 7fd059c00640 1 -- 172.21.15.90:0/432543491 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd054016040 con 0x55d5d5a94190 2024-04-02T14:43:47.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.607+0000 7fd072dfc740 1 -- 172.21.15.90:0/432543491 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d5d5a94190 msgr2=0x55d5d5ab8650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:47.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.607+0000 7fd072dfc740 1 --2- 172.21.15.90:0/432543491 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d5d5a94190 0x55d5d5ab8650 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fd0540109e0 tx=0x7fd054005dc0 comp rx=0 tx=0).stop 2024-04-02T14:43:47.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.608+0000 7fd072dfc740 1 -- 172.21.15.90:0/432543491 shutdown_connections 2024-04-02T14:43:47.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.608+0000 7fd072dfc740 1 --2- 172.21.15.90:0/432543491 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d5d5a9b200 0x55d5d5a90a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:47.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.608+0000 7fd072dfc740 1 --2- 172.21.15.90:0/432543491 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d5d5a9a520 0x55d5d5ab7ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:47.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.608+0000 7fd072dfc740 1 --2- 172.21.15.90:0/432543491 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d5d5a94190 0x55d5d5ab8650 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:47.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.608+0000 7fd072dfc740 1 -- 172.21.15.90:0/432543491 >> 172.21.15.90:0/432543491 conn(0x55d5d5a58a50 msgr2=0x55d5d5a96bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:43:47.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.608+0000 7fd072dfc740 1 -- 172.21.15.90:0/432543491 shutdown_connections 2024-04-02T14:43:47.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.608+0000 7fd072dfc740 1 -- 172.21.15.90:0/432543491 wait complete. 2024-04-02T14:43:47.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.608+0000 7fd072dfc740 1 -- start start 2024-04-02T14:43:47.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.609+0000 7fd072dfc740 1 Processor -- start 2024-04-02T14:43:47.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.609+0000 7fd072dfc740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d5d5a94190 0x55d5d5a8ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:47.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.609+0000 7fd072dfc740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d5d5a9a520 0x55d5d5a8e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:47.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.609+0000 7fd072dfc740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d5d5a9b200 0x55d5d5a8e830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:47.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.609+0000 7fd072dfc740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fd0681a4370 con 0x55d5d5a9a520 2024-04-02T14:43:47.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.609+0000 7fd072dfc740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fd0681a3ca0 con 0x55d5d5a94190 2024-04-02T14:43:47.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.609+0000 7fd072dfc740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fd0681a0f00 con 0x55d5d5a9b200 2024-04-02T14:43:47.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.609+0000 7fd072dfc740 10 client.0 _ll_register_callbacks cb 0x7fd054010cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:43:47.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.609+0000 7fd06cc00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d5d5a94190 0x55d5d5a8ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:47.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.609+0000 7fd06d600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d5d5a9a520 0x55d5d5a8e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:47.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.609+0000 7fd06d600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d5d5a9a520 0x55d5d5a8e2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33774/0 (socket says 172.21.15.90:33774) 2024-04-02T14:43:47.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.610+0000 7fd06d600640 1 -- 172.21.15.90:0/2073997509 learned_addr learned my addr 172.21.15.90:0/2073997509 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:43:47.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.609+0000 7fd06cc00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d5d5a94190 0x55d5d5a8ddb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:36424/0 (socket says 172.21.15.90:36424) 2024-04-02T14:43:47.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.610+0000 7fd06cc00640 1 --2- 172.21.15.90:0/2073997509 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d5d5a9b200 0x55d5d5a8e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:47.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.610+0000 7fd06d600640 1 -- 172.21.15.90:0/2073997509 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d5d5a9b200 msgr2=0x55d5d5a8e830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:43:47.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.610+0000 7fd06d600640 1 --2- 172.21.15.90:0/2073997509 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d5d5a9b200 0x55d5d5a8e830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:47.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.610+0000 7fd06d600640 1 -- 172.21.15.90:0/2073997509 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d5d5a94190 msgr2=0x55d5d5a8ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:43:47.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.610+0000 7fd06d600640 1 --2- 172.21.15.90:0/2073997509 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d5d5a94190 0x55d5d5a8ddb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:47.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.610+0000 7fd06d600640 1 -- 172.21.15.90:0/2073997509 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d5d5b21740 con 0x55d5d5a9a520 2024-04-02T14:43:47.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.610+0000 7fd06cc00640 1 --2- 172.21.15.90:0/2073997509 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d5d5a94190 0x55d5d5a8ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:43:47.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.610+0000 7fd06d600640 1 --2- 172.21.15.90:0/2073997509 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d5d5a9a520 0x55d5d5a8e2f0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fd05c01a6d0 tx=0x7fd05c03da40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f57afa1ae38beabe server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:43:47.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.610+0000 7fd04ea00640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-02T14:43:47.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.611+0000 7fd04ea00640 1 -- 172.21.15.90:0/2073997509 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd05c0043a0 con 0x55d5d5a9a520 2024-04-02T14:43:47.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.611+0000 7fd04ea00640 1 -- 172.21.15.90:0/2073997509 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8550+0+0 (secure 0 0 0) 0x7fd05c03ddc0 con 0x55d5d5a9a520 2024-04-02T14:43:47.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.611+0000 7fd072dfc740 1 -- 172.21.15.90:0/2073997509 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd05400ab50 con 0x55d5d5a9a520 2024-04-02T14:43:47.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.612+0000 7fd04ea00640 1 -- 172.21.15.90:0/2073997509 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fd05c005850 con 0x55d5d5a9a520 2024-04-02T14:43:47.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.612+0000 7fd072dfc740 10 client.17436 fetch_fsmap learned FSMap version 27 2024-04-02T14:43:47.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.612+0000 7fd072dfc740 1 -- 172.21.15.90:0/2073997509 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55d5d5b22940 con 0x55d5d5a9a520 2024-04-02T14:43:47.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.613+0000 7fd04ea00640 1 -- 172.21.15.90:0/2073997509 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd05400ab50 con 0x55d5d5a9a520 2024-04-02T14:43:47.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.613+0000 7fd04ea00640 1 -- 172.21.15.90:0/2073997509 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fd05c003b10 con 0x55d5d5a9a520 2024-04-02T14:43:47.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.613+0000 7fd072dfc740 10 client.17436 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:43:47.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.613+0000 7fd072dfc740 10 client.17436 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2024-04-02T14:43:47.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.613+0000 7fd072dfc740 20 client.17436 populate_metadata read hostname 'smithi090' 2024-04-02T14:43:47.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.613+0000 7fd072dfc740 1 --2- 172.21.15.90:0/2073997509 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x55d5d5b22cd0 0x55d5d5b250f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:47.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.613+0000 7fd072dfc740 4 client.17436 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-02T14:43:47.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.613+0000 7fd072dfc740 1 -- 172.21.15.90:0/2073997509 --> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55d5d5a03c30 con 0x55d5d5b22cd0 2024-04-02T14:43:47.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.614+0000 7fd06cc00640 1 --2- 172.21.15.90:0/2073997509 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x55d5d5b22cd0 0x55d5d5b250f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:47.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.615+0000 7fd06cc00640 1 --2- 172.21.15.90:0/2073997509 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x55d5d5b22cd0 0x55d5d5b250f0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=289d41efb6334ab0 server_cookie=a055e89a4edfc2d in_seq=0 out_seq=0 2024-04-02T14:43:47.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.615+0000 7fd04ea00640 10 client.17436 ms_handle_connect on v2:172.21.15.90:6838/1498915870 2024-04-02T14:43:47.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.615+0000 7fd04ea00640 1 -- 172.21.15.90:0/2073997509 <== mds.1 v2:172.21.15.90:6838/1498915870 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55d5d5a03c30 con 0x55d5d5b22cd0 2024-04-02T14:43:47.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.615+0000 7fd04ea00640 10 client.17436 handle_command_reply: tid=0 2024-04-02T14:43:47.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.615+0000 7fd072dfc740 1 client.17436 shutdown 2024-04-02T14:43:47.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.615+0000 7fd072dfc740 1 -- 172.21.15.90:0/2073997509 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d5d5a9a520 msgr2=0x55d5d5a8e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:47.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.615+0000 7fd072dfc740 1 --2- 172.21.15.90:0/2073997509 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d5d5a9a520 0x55d5d5a8e2f0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fd05c01a6d0 tx=0x7fd05c03da40 comp rx=0 tx=0).stop 2024-04-02T14:43:47.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.616+0000 7fd072dfc740 1 -- 172.21.15.90:0/2073997509 shutdown_connections 2024-04-02T14:43:47.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.616+0000 7fd072dfc740 1 --2- 172.21.15.90:0/2073997509 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x55d5d5b22cd0 0x55d5d5b250f0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:47.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.616+0000 7fd072dfc740 1 --2- 172.21.15.90:0/2073997509 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d5d5a9b200 0x55d5d5a8e830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:47.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.616+0000 7fd072dfc740 1 --2- 172.21.15.90:0/2073997509 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d5d5a9a520 0x55d5d5a8e2f0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:47.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.616+0000 7fd072dfc740 1 --2- 172.21.15.90:0/2073997509 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d5d5a94190 0x55d5d5a8ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:47.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.616+0000 7fd04ea00640 0 client.17436 ms_handle_reset on v2:172.21.15.90:6838/1498915870 2024-04-02T14:43:47.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.616+0000 7fd072dfc740 1 -- 172.21.15.90:0/2073997509 >> 172.21.15.90:0/2073997509 conn(0x55d5d5a58a50 msgr2=0x55d5d5a961c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:43:47.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.617+0000 7fd072dfc740 1 -- 172.21.15.90:0/2073997509 shutdown_connections 2024-04-02T14:43:47.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.617+0000 7fd072dfc740 1 -- 172.21.15.90:0/2073997509 wait complete. 2024-04-02T14:43:47.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.617+0000 7fd072dfc740 20 client.17436 trim_cache size 0 max 16384 2024-04-02T14:43:47.617 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-02T14:43:47.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.618+0000 7fd070000640 1 -- 172.21.15.90:0/3269904659 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fd0500a3da0 msgr2=0x7fd0500a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:47.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.618+0000 7fd070000640 1 --2- 172.21.15.90:0/3269904659 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fd0500a3da0 0x7fd0500a6260 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fd06009cc00 tx=0x7fd06000ff60 comp rx=0 tx=0).stop 2024-04-02T14:43:47.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.618+0000 7fd070000640 1 -- 172.21.15.90:0/3269904659 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd06810cd80 msgr2=0x7fd06810d230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:47.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.618+0000 7fd070000640 1 --2- 172.21.15.90:0/3269904659 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd06810cd80 0x7fd06810d230 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7fd06000cce0 tx=0x7fd06000a620 comp rx=0 tx=0).stop 2024-04-02T14:43:47.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.618+0000 7fd070000640 1 -- 172.21.15.90:0/3269904659 shutdown_connections 2024-04-02T14:43:47.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.618+0000 7fd070000640 1 --2- 172.21.15.90:0/3269904659 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fd0500a3da0 0x7fd0500a6260 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:47.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.618+0000 7fd070000640 1 --2- 172.21.15.90:0/3269904659 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd06810cd80 0x7fd06810d230 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:47.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.618+0000 7fd070000640 1 --2- 172.21.15.90:0/3269904659 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd068108fe0 0x7fd06810c840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:47.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.618+0000 7fd070000640 1 --2- 172.21.15.90:0/3269904659 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd068103f10 0x7fd0681107b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:47.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.618+0000 7fd070000640 1 -- 172.21.15.90:0/3269904659 >> 172.21.15.90:0/3269904659 conn(0x7fd0680ff7b0 msgr2=0x7fd068101ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:43:47.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.618+0000 7fd070000640 1 -- 172.21.15.90:0/3269904659 shutdown_connections 2024-04-02T14:43:47.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.618+0000 7fd070000640 1 -- 172.21.15.90:0/3269904659 wait complete. 2024-04-02T14:43:47.641 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-02T14:43:47.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.935+0000 7f5d9e800640 1 -- 172.21.15.90:0/3203922147 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5d981048c0 msgr2=0x7f5d98108900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:47.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.935+0000 7f5d9e800640 1 --2- 172.21.15.90:0/3203922147 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5d981048c0 0x7f5d98108900 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f5d88009900 tx=0x7f5d8802bc30 comp rx=0 tx=0).stop 2024-04-02T14:43:47.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.936+0000 7f5d9e800640 1 -- 172.21.15.90:0/3203922147 shutdown_connections 2024-04-02T14:43:47.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.936+0000 7f5d9e800640 1 --2- 172.21.15.90:0/3203922147 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5d98108fe0 0x7f5d98110bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:47.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.936+0000 7f5d9e800640 1 --2- 172.21.15.90:0/3203922147 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5d981048c0 0x7f5d98108900 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:47.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.936+0000 7f5d9e800640 1 --2- 172.21.15.90:0/3203922147 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5d98103f10 0x7f5d981042f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:47.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.936+0000 7f5d9e800640 1 -- 172.21.15.90:0/3203922147 >> 172.21.15.90:0/3203922147 conn(0x7f5d980ff770 msgr2=0x7f5d98101b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:43:47.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.936+0000 7f5d9e800640 1 -- 172.21.15.90:0/3203922147 shutdown_connections 2024-04-02T14:43:47.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.937+0000 7f5d9e800640 1 -- 172.21.15.90:0/3203922147 wait complete. 2024-04-02T14:43:47.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.938+0000 7f5d9e800640 1 Processor -- start 2024-04-02T14:43:47.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.938+0000 7f5d9e800640 1 -- start start 2024-04-02T14:43:47.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.939+0000 7f5d9e800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5d98103f10 0x7f5d98076010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:47.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.939+0000 7f5d9e800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5d98108fe0 0x7f5d98076550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:47.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.939+0000 7f5d9e800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5d98076a90 0x7f5d981aac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:47.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.939+0000 7f5d9e800640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f5d98114140 con 0x7f5d98108fe0 2024-04-02T14:43:47.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.939+0000 7f5d9e800640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f5d98113fc0 con 0x7f5d98076a90 2024-04-02T14:43:47.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.939+0000 7f5d9e800640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f5d981142c0 con 0x7f5d98103f10 2024-04-02T14:43:47.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.939+0000 7f5d97400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5d98103f10 0x7f5d98076010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:47.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.939+0000 7f5d96a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5d98108fe0 0x7f5d98076550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:47.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.939+0000 7f5d97400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5d98103f10 0x7f5d98076010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:52710/0 (socket says 172.21.15.90:52710) 2024-04-02T14:43:47.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.939+0000 7f5d97400640 1 -- 172.21.15.90:0/881220932 learned_addr learned my addr 172.21.15.90:0/881220932 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:43:47.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.940+0000 7f5d97400640 1 -- 172.21.15.90:0/881220932 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5d98076a90 msgr2=0x7f5d981aac30 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-02T14:43:47.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.940+0000 7f5d97400640 1 --2- 172.21.15.90:0/881220932 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5d98076a90 0x7f5d981aac30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:47.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.940+0000 7f5d97400640 1 -- 172.21.15.90:0/881220932 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5d98108fe0 msgr2=0x7f5d98076550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:47.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.940+0000 7f5d97400640 1 --2- 172.21.15.90:0/881220932 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5d98108fe0 0x7f5d98076550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:47.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.940+0000 7f5d97400640 1 -- 172.21.15.90:0/881220932 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d9819a3d0 con 0x7f5d98103f10 2024-04-02T14:43:47.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.940+0000 7f5d97400640 1 --2- 172.21.15.90:0/881220932 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5d98103f10 0x7f5d98076010 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f5d9000ba50 tx=0x7f5d9000bf10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:43:47.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.941+0000 7f5d8fe00640 1 -- 172.21.15.90:0/881220932 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d90019070 con 0x7f5d98103f10 2024-04-02T14:43:47.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.941+0000 7f5d8fe00640 1 -- 172.21.15.90:0/881220932 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5d9000cbe0 con 0x7f5d98103f10 2024-04-02T14:43:47.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.941+0000 7f5d8fe00640 1 -- 172.21.15.90:0/881220932 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d900076e0 con 0x7f5d98103f10 2024-04-02T14:43:47.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.941+0000 7f5d9e800640 1 -- 172.21.15.90:0/881220932 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d9819a660 con 0x7f5d98103f10 2024-04-02T14:43:47.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.941+0000 7f5d9e800640 1 -- 172.21.15.90:0/881220932 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d9819ab80 con 0x7f5d98103f10 2024-04-02T14:43:47.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.949+0000 7f5d8fe00640 1 -- 172.21.15.90:0/881220932 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f5d9001c070 con 0x7f5d98103f10 2024-04-02T14:43:47.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.950+0000 7f5da1417740 1 Processor -- start 2024-04-02T14:43:47.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.952+0000 7f5da1417740 1 -- start start 2024-04-02T14:43:47.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.952+0000 7f5da1417740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x562663a529f0 0x562663a51740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:47.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.952+0000 7f5da1417740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562663a51d10 0x562663a52190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:47.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.952+0000 7f5da1417740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562663a5b080 0x562663a5d470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:47.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.952+0000 7f5da1417740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x562663992400 con 0x562663a51d10 2024-04-02T14:43:47.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.952+0000 7f5da1417740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x562663976f10 con 0x562663a529f0 2024-04-02T14:43:47.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.952+0000 7f5da1417740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5626639b66f0 con 0x562663a5b080 2024-04-02T14:43:47.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.953+0000 7f5d97e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x562663a529f0 0x562663a51740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:47.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.953+0000 7f5d97e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x562663a529f0 0x562663a51740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:36460/0 (socket says 172.21.15.90:36460) 2024-04-02T14:43:47.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.953+0000 7f5d97e00640 1 -- 172.21.15.90:0/610145188 learned_addr learned my addr 172.21.15.90:0/610145188 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:43:47.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.953+0000 7f5d96a00640 1 --2- 172.21.15.90:0/610145188 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562663a5b080 0x562663a5d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:47.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.953+0000 7f5d97e00640 1 -- 172.21.15.90:0/610145188 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562663a5b080 msgr2=0x562663a5d470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:43:47.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.954+0000 7f5d97e00640 1 --2- 172.21.15.90:0/610145188 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562663a5b080 0x562663a5d470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:47.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.954+0000 7f5d97e00640 1 -- 172.21.15.90:0/610145188 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562663a51d10 msgr2=0x562663a52190 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-02T14:43:47.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.954+0000 7f5d97e00640 1 --2- 172.21.15.90:0/610145188 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562663a51d10 0x562663a52190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:47.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.954+0000 7f5d97e00640 1 -- 172.21.15.90:0/610145188 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5626636ca3f0 con 0x562663a529f0 2024-04-02T14:43:47.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.954+0000 7f5d97e00640 1 --2- 172.21.15.90:0/610145188 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x562663a529f0 0x562663a51740 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f5d84009bd0 tx=0x7f5d8400e720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c1eecb8a4131cb4e server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:43:47.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.955+0000 7f5d8cc00640 1 -- 172.21.15.90:0/610145188 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d84005020 con 0x562663a529f0 2024-04-02T14:43:47.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.955+0000 7f5d8cc00640 1 -- 172.21.15.90:0/610145188 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5d840047a0 con 0x562663a529f0 2024-04-02T14:43:47.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.956+0000 7f5d8fe00640 1 --2- 172.21.15.90:0/881220932 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f5d7c0a3f20 0x7f5d7c0a63e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:47.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.956+0000 7f5da1417740 1 -- 172.21.15.90:0/610145188 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x562663a529f0 msgr2=0x562663a51740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:47.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.956+0000 7f5da1417740 1 --2- 172.21.15.90:0/610145188 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x562663a529f0 0x562663a51740 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f5d84009bd0 tx=0x7f5d8400e720 comp rx=0 tx=0).stop 2024-04-02T14:43:47.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.956+0000 7f5da1417740 1 -- 172.21.15.90:0/610145188 shutdown_connections 2024-04-02T14:43:47.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.956+0000 7f5da1417740 1 --2- 172.21.15.90:0/610145188 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562663a5b080 0x562663a5d470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:47.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.956+0000 7f5da1417740 1 --2- 172.21.15.90:0/610145188 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562663a51d10 0x562663a52190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:47.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.956+0000 7f5da1417740 1 --2- 172.21.15.90:0/610145188 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x562663a529f0 0x562663a51740 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:47.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.956+0000 7f5da1417740 1 -- 172.21.15.90:0/610145188 >> 172.21.15.90:0/610145188 conn(0x562663a18a50 msgr2=0x562663a4c760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:43:47.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.956+0000 7f5d8fe00640 1 -- 172.21.15.90:0/881220932 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8550+0+0 (secure 0 0 0) 0x7f5d900d0200 con 0x7f5d98103f10 2024-04-02T14:43:47.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.956+0000 7f5d97400640 1 --2- 172.21.15.90:0/881220932 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f5d7c0a3f20 0x7f5d7c0a63e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:47.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.957+0000 7f5da1417740 1 -- 172.21.15.90:0/610145188 shutdown_connections 2024-04-02T14:43:47.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.957+0000 7f5da1417740 1 -- 172.21.15.90:0/610145188 wait complete. 2024-04-02T14:43:47.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.957+0000 7f5d97400640 1 --2- 172.21.15.90:0/881220932 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f5d7c0a3f20 0x7f5d7c0a63e0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x562663a52f40 tx=0x7f5d9000e040 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:43:47.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.958+0000 7f5da1417740 1 -- start start 2024-04-02T14:43:47.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.958+0000 7f5da1417740 1 Processor -- start 2024-04-02T14:43:47.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.959+0000 7f5da1417740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x562663a51d10 0x562663a58000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:47.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.959+0000 7f5da1417740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562663a529f0 0x562663a58540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:47.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.959+0000 7f5da1417740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562663a5b080 0x562663a58a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:47.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.959+0000 7f5da1417740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5626639b26a0 con 0x562663a5b080 2024-04-02T14:43:47.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.959+0000 7f5da1417740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x56266395e980 con 0x562663a51d10 2024-04-02T14:43:47.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.959+0000 7f5da1417740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x562663971cb0 con 0x562663a529f0 2024-04-02T14:43:47.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.959+0000 7f5da1417740 10 client.0 _ll_register_callbacks cb 0x5626639716a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:43:47.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.959+0000 7f5d97e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x562663a51d10 0x562663a58000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:47.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.959+0000 7f5d97e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x562663a51d10 0x562663a58000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:36476/0 (socket says 172.21.15.90:36476) 2024-04-02T14:43:47.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.959+0000 7f5d97e00640 1 -- 172.21.15.90:0/1163119961 learned_addr learned my addr 172.21.15.90:0/1163119961 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:43:47.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.959+0000 7f5d96a00640 1 --2- 172.21.15.90:0/1163119961 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562663a529f0 0x562663a58540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:47.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.960+0000 7f5d97e00640 1 --2- 172.21.15.90:0/1163119961 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562663a5b080 0x562663a58a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:47.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.960+0000 7f5d96a00640 1 -- 172.21.15.90:0/1163119961 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x562663a51d10 msgr2=0x562663a58000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:43:47.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.960+0000 7f5d96a00640 1 --2- 172.21.15.90:0/1163119961 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x562663a51d10 0x562663a58000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:47.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.960+0000 7f5d96a00640 1 -- 172.21.15.90:0/1163119961 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562663a5b080 msgr2=0x562663a58a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:43:47.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.960+0000 7f5d96a00640 1 --2- 172.21.15.90:0/1163119961 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562663a5b080 0x562663a58a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:47.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.960+0000 7f5d96a00640 1 -- 172.21.15.90:0/1163119961 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5626637947d0 con 0x562663a529f0 2024-04-02T14:43:47.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.960+0000 7f5d97e00640 1 --2- 172.21.15.90:0/1163119961 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562663a5b080 0x562663a58a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:43:47.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.961+0000 7f5d96a00640 1 --2- 172.21.15.90:0/1163119961 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562663a529f0 0x562663a58540 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f5d88009230 tx=0x7f5d8802c7e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c44bfa039ee9695a server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:43:47.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.961+0000 7f5d81600640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:43:47.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.962+0000 7f5d81600640 1 -- 172.21.15.90:0/1163119961 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d88005030 con 0x562663a529f0 2024-04-02T14:43:47.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.962+0000 7f5d81600640 1 -- 172.21.15.90:0/1163119961 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8550+0+0 (secure 0 0 0) 0x7f5d880051d0 con 0x562663a529f0 2024-04-02T14:43:47.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.962+0000 7f5da1417740 1 -- 172.21.15.90:0/1163119961 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5d840047a0 con 0x562663a529f0 2024-04-02T14:43:47.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.963+0000 7f5d81600640 1 -- 172.21.15.90:0/1163119961 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f5d880053d0 con 0x562663a529f0 2024-04-02T14:43:47.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.963+0000 7f5da1417740 10 client.28096 fetch_fsmap learned FSMap version 27 2024-04-02T14:43:47.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.963+0000 7f5da1417740 1 -- 172.21.15.90:0/1163119961 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x562663a69fd0 con 0x562663a529f0 2024-04-02T14:43:47.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.963+0000 7f5d81600640 1 -- 172.21.15.90:0/1163119961 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5d840047a0 con 0x562663a529f0 2024-04-02T14:43:47.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.963+0000 7f5d81600640 1 -- 172.21.15.90:0/1163119961 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f5d880036a0 con 0x562663a529f0 2024-04-02T14:43:47.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.964+0000 7f5da1417740 10 client.28096 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:43:47.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.964+0000 7f5da1417740 10 client.28096 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:43:47.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.964+0000 7f5da1417740 20 client.28096 populate_metadata read hostname 'smithi090' 2024-04-02T14:43:47.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.964+0000 7f5da1417740 1 --2- 172.21.15.90:0/1163119961 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x562663a6b4e0 0x562663a6d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:47.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.964+0000 7f5da1417740 4 client.28096 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:43:47.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.964+0000 7f5d97e00640 1 --2- 172.21.15.90:0/1163119961 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x562663a6b4e0 0x562663a6d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:47.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.964+0000 7f5da1417740 1 -- 172.21.15.90:0/1163119961 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562663976040 con 0x562663a6b4e0 2024-04-02T14:43:47.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.965+0000 7f5d97e00640 1 --2- 172.21.15.90:0/1163119961 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x562663a6b4e0 0x562663a6d8d0 crc :-1 s=READY pgs=240 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2fb76ce9a7c1ac3d server_cookie=b4fb54ab6cfd02e8 in_seq=0 out_seq=0 2024-04-02T14:43:47.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.965+0000 7f5d81600640 10 client.28096 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:43:47.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.966+0000 7f5d81600640 1 -- 172.21.15.90:0/1163119961 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x562663976040 con 0x562663a6b4e0 2024-04-02T14:43:47.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.966+0000 7f5d81600640 10 client.28096 handle_command_reply: tid=0 2024-04-02T14:43:47.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.966+0000 7f5da1417740 1 client.28096 shutdown 2024-04-02T14:43:47.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.967+0000 7f5da1417740 1 -- 172.21.15.90:0/1163119961 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562663a529f0 msgr2=0x562663a58540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:47.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.967+0000 7f5da1417740 1 --2- 172.21.15.90:0/1163119961 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562663a529f0 0x562663a58540 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f5d88009230 tx=0x7f5d8802c7e0 comp rx=0 tx=0).stop 2024-04-02T14:43:47.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.967+0000 7f5da1417740 1 -- 172.21.15.90:0/1163119961 shutdown_connections 2024-04-02T14:43:47.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.967+0000 7f5da1417740 1 --2- 172.21.15.90:0/1163119961 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x562663a6b4e0 0x562663a6d8d0 crc :-1 s=READY pgs=240 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:47.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.967+0000 7f5da1417740 1 --2- 172.21.15.90:0/1163119961 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562663a5b080 0x562663a58a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:47.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.967+0000 7f5da1417740 1 --2- 172.21.15.90:0/1163119961 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562663a529f0 0x562663a58540 secure :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f5d88009230 tx=0x7f5d8802c7e0 comp rx=0 tx=0).stop 2024-04-02T14:43:47.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.967+0000 7f5da1417740 1 --2- 172.21.15.90:0/1163119961 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x562663a51d10 0x562663a58000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:47.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.967+0000 7f5da1417740 1 -- 172.21.15.90:0/1163119961 >> 172.21.15.90:0/1163119961 conn(0x562663a18a50 msgr2=0x562663a4e4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:43:47.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.967+0000 7f5d81600640 0 client.28096 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:43:47.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.967+0000 7f5da1417740 1 -- 172.21.15.90:0/1163119961 shutdown_connections 2024-04-02T14:43:47.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.967+0000 7f5da1417740 1 -- 172.21.15.90:0/1163119961 wait complete. 2024-04-02T14:43:47.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.968+0000 7f5da1417740 20 client.28096 trim_cache size 0 max 16384 2024-04-02T14:43:47.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.989+0000 7f5da1417740 1 Processor -- start 2024-04-02T14:43:47.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.989+0000 7f5da1417740 1 -- start start 2024-04-02T14:43:47.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.990+0000 7f5da1417740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x562663a51d10 0x562663a66c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:47.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.990+0000 7f5da1417740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562663a5b080 0x562663a588a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:47.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.990+0000 7f5da1417740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562663a4c6f0 0x562663a59e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:47.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.990+0000 7f5da1417740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f5d981bc860 con 0x562663a5b080 2024-04-02T14:43:47.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.990+0000 7f5da1417740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f5d981be3a0 con 0x562663a51d10 2024-04-02T14:43:47.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.990+0000 7f5da1417740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f5d981bcf30 con 0x562663a4c6f0 2024-04-02T14:43:47.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.990+0000 7f5d97e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x562663a51d10 0x562663a66c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:47.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.990+0000 7f5d96a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562663a5b080 0x562663a588a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:47.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.991+0000 7f5d96a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562663a5b080 0x562663a588a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33806/0 (socket says 172.21.15.90:33806) 2024-04-02T14:43:47.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.991+0000 7f5d96a00640 1 -- 172.21.15.90:0/128871133 learned_addr learned my addr 172.21.15.90:0/128871133 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:43:47.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.991+0000 7f5d97e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x562663a51d10 0x562663a66c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:36486/0 (socket says 172.21.15.90:36486) 2024-04-02T14:43:47.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.991+0000 7f5d97e00640 1 --2- 172.21.15.90:0/128871133 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562663a4c6f0 0x562663a59e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:47.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.991+0000 7f5d96a00640 1 -- 172.21.15.90:0/128871133 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562663a4c6f0 msgr2=0x562663a59e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:43:47.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.991+0000 7f5d96a00640 1 --2- 172.21.15.90:0/128871133 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562663a4c6f0 0x562663a59e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:47.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.991+0000 7f5d96a00640 1 -- 172.21.15.90:0/128871133 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x562663a51d10 msgr2=0x562663a66c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:43:47.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.991+0000 7f5d96a00640 1 --2- 172.21.15.90:0/128871133 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x562663a51d10 0x562663a66c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:47.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.991+0000 7f5d96a00640 1 -- 172.21.15.90:0/128871133 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562663a58de0 con 0x562663a5b080 2024-04-02T14:43:47.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.992+0000 7f5d96a00640 1 --2- 172.21.15.90:0/128871133 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562663a5b080 0x562663a588a0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f5d88003c30 tx=0x7f5d880058b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=374b569b891b231 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:43:47.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.992+0000 7f5d83e00640 1 -- 172.21.15.90:0/128871133 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d88000e60 con 0x562663a5b080 2024-04-02T14:43:47.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.992+0000 7f5d83e00640 1 -- 172.21.15.90:0/128871133 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5d88057020 con 0x562663a5b080 2024-04-02T14:43:47.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.992+0000 7f5d83e00640 1 -- 172.21.15.90:0/128871133 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d880035f0 con 0x562663a5b080 2024-04-02T14:43:47.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.993+0000 7f5da1417740 1 -- 172.21.15.90:0/128871133 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562663a5b080 msgr2=0x562663a588a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:47.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.993+0000 7f5da1417740 1 --2- 172.21.15.90:0/128871133 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562663a5b080 0x562663a588a0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f5d88003c30 tx=0x7f5d880058b0 comp rx=0 tx=0).stop 2024-04-02T14:43:47.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.993+0000 7f5da1417740 1 -- 172.21.15.90:0/128871133 shutdown_connections 2024-04-02T14:43:47.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.993+0000 7f5da1417740 1 --2- 172.21.15.90:0/128871133 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562663a4c6f0 0x562663a59e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:47.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.993+0000 7f5da1417740 1 --2- 172.21.15.90:0/128871133 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562663a5b080 0x562663a588a0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:47.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.993+0000 7f5da1417740 1 --2- 172.21.15.90:0/128871133 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x562663a51d10 0x562663a66c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:47.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.993+0000 7f5da1417740 1 -- 172.21.15.90:0/128871133 >> 172.21.15.90:0/128871133 conn(0x562663a18a50 msgr2=0x562663a59720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:43:47.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.993+0000 7f5da1417740 1 -- 172.21.15.90:0/128871133 shutdown_connections 2024-04-02T14:43:47.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.994+0000 7f5da1417740 1 -- 172.21.15.90:0/128871133 wait complete. 2024-04-02T14:43:47.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.994+0000 7f5da1417740 1 -- start start 2024-04-02T14:43:47.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.994+0000 7f5da1417740 1 Processor -- start 2024-04-02T14:43:47.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.995+0000 7f5da1417740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x562663a4c6f0 0x562663a5c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:47.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.995+0000 7f5da1417740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562663a51d10 0x562663a5c800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:47.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.995+0000 7f5da1417740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562663a5b080 0x562663a61730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:47.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.995+0000 7f5da1417740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f5d981bdcd0 con 0x562663a51d10 2024-04-02T14:43:47.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.995+0000 7f5da1417740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f5d981bd600 con 0x562663a4c6f0 2024-04-02T14:43:47.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.995+0000 7f5da1417740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f5d981a12c0 con 0x562663a5b080 2024-04-02T14:43:47.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.995+0000 7f5da1417740 10 client.0 _ll_register_callbacks cb 0x7f5d88004520 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:43:47.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.995+0000 7f5d97e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x562663a4c6f0 0x562663a5c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:47.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.995+0000 7f5d97e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x562663a4c6f0 0x562663a5c2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:36492/0 (socket says 172.21.15.90:36492) 2024-04-02T14:43:47.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.995+0000 7f5d97e00640 1 -- 172.21.15.90:0/3040945860 learned_addr learned my addr 172.21.15.90:0/3040945860 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:43:47.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.995+0000 7f5d97e00640 1 --2- 172.21.15.90:0/3040945860 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562663a5b080 0x562663a61730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:47.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.995+0000 7f5d96a00640 1 --2- 172.21.15.90:0/3040945860 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562663a51d10 0x562663a5c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:47.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.995+0000 7f5d97e00640 1 -- 172.21.15.90:0/3040945860 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562663a5b080 msgr2=0x562663a61730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:43:47.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.995+0000 7f5d97e00640 1 --2- 172.21.15.90:0/3040945860 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562663a5b080 0x562663a61730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:47.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.995+0000 7f5d97e00640 1 -- 172.21.15.90:0/3040945860 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562663a51d10 msgr2=0x562663a5c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:43:47.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.995+0000 7f5d97e00640 1 --2- 172.21.15.90:0/3040945860 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562663a51d10 0x562663a5c800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:47.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.995+0000 7f5d97e00640 1 -- 172.21.15.90:0/3040945860 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562663a61c70 con 0x562663a4c6f0 2024-04-02T14:43:47.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.995+0000 7f5d97e00640 1 --2- 172.21.15.90:0/3040945860 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x562663a4c6f0 0x562663a5c2c0 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f5d84009bd0 tx=0x7f5d84005f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=51c55fa0a2fe205e server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:43:47.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.996+0000 7f5d81600640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:43:47.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.996+0000 7f5d81600640 1 -- 172.21.15.90:0/3040945860 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d8401d040 con 0x562663a4c6f0 2024-04-02T14:43:47.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.996+0000 7f5d81600640 1 -- 172.21.15.90:0/3040945860 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8550+0+0 (secure 0 0 0) 0x7f5d84011ce0 con 0x562663a4c6f0 2024-04-02T14:43:47.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.996+0000 7f5da1417740 1 -- 172.21.15.90:0/3040945860 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5d88057020 con 0x562663a4c6f0 2024-04-02T14:43:47.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.996+0000 7f5d81600640 1 -- 172.21.15.90:0/3040945860 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f5d84004fb0 con 0x562663a4c6f0 2024-04-02T14:43:47.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.996+0000 7f5da1417740 10 client.28268 fetch_fsmap learned FSMap version 27 2024-04-02T14:43:47.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.996+0000 7f5da1417740 1 -- 172.21.15.90:0/3040945860 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x562663ad2100 con 0x562663a4c6f0 2024-04-02T14:43:47.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.998+0000 7f5d81600640 1 -- 172.21.15.90:0/3040945860 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5d88057020 con 0x562663a4c6f0 2024-04-02T14:43:47.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.998+0000 7f5d81600640 1 -- 172.21.15.90:0/3040945860 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f5d840039e0 con 0x562663a4c6f0 2024-04-02T14:43:47.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.998+0000 7f5da1417740 10 client.28268 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:43:47.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.998+0000 7f5da1417740 10 client.28268 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:43:47.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.998+0000 7f5da1417740 20 client.28268 populate_metadata read hostname 'smithi090' 2024-04-02T14:43:47.997 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.998+0000 7f5da1417740 1 --2- 172.21.15.90:0/3040945860 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x562663ad26a0 0x562663ad4ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:47.997 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.998+0000 7f5da1417740 4 client.28268 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-02T14:43:47.997 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.998+0000 7f5da1417740 1 -- 172.21.15.90:0/3040945860 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5626639c3c30 con 0x562663ad26a0 2024-04-02T14:43:47.997 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:47.999+0000 7f5d96a00640 1 --2- 172.21.15.90:0/3040945860 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x562663ad26a0 0x562663ad4ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:47.998 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:48.000+0000 7f5d96a00640 1 --2- 172.21.15.90:0/3040945860 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x562663ad26a0 0x562663ad4ac0 crc :-1 s=READY pgs=241 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=52143dd3957e746d server_cookie=e82565f8be016397 in_seq=0 out_seq=0 2024-04-02T14:43:47.998 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:48.000+0000 7f5d81600640 10 client.28268 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:43:47.998 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:48.000+0000 7f5d81600640 1 -- 172.21.15.90:0/3040945860 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5626639c3c30 con 0x562663ad26a0 2024-04-02T14:43:47.998 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:48.000+0000 7f5d81600640 10 client.28268 handle_command_reply: tid=0 2024-04-02T14:43:47.998 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:48.000+0000 7f5da1417740 1 client.28268 shutdown 2024-04-02T14:43:47.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:48.001+0000 7f5da1417740 1 -- 172.21.15.90:0/3040945860 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x562663a4c6f0 msgr2=0x562663a5c2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:47.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:48.001+0000 7f5da1417740 1 --2- 172.21.15.90:0/3040945860 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x562663a4c6f0 0x562663a5c2c0 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f5d84009bd0 tx=0x7f5d84005f90 comp rx=0 tx=0).stop 2024-04-02T14:43:47.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:48.001+0000 7f5da1417740 1 -- 172.21.15.90:0/3040945860 shutdown_connections 2024-04-02T14:43:47.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:48.001+0000 7f5da1417740 1 --2- 172.21.15.90:0/3040945860 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x562663ad26a0 0x562663ad4ac0 crc :-1 s=READY pgs=241 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:47.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:48.001+0000 7f5da1417740 1 --2- 172.21.15.90:0/3040945860 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562663a5b080 0x562663a61730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:47.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:48.001+0000 7f5d81600640 0 client.28268 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:43:47.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:48.001+0000 7f5da1417740 1 --2- 172.21.15.90:0/3040945860 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562663a51d10 0x562663a5c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:47.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:48.001+0000 7f5da1417740 1 --2- 172.21.15.90:0/3040945860 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x562663a4c6f0 0x562663a5c2c0 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:47.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:48.001+0000 7f5da1417740 1 -- 172.21.15.90:0/3040945860 >> 172.21.15.90:0/3040945860 conn(0x562663a18a50 msgr2=0x562663a4b8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:43:47.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:48.001+0000 7f5da1417740 1 -- 172.21.15.90:0/3040945860 shutdown_connections 2024-04-02T14:43:47.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:48.001+0000 7f5da1417740 1 -- 172.21.15.90:0/3040945860 wait complete. 2024-04-02T14:43:47.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:48.001+0000 7f5da1417740 20 client.28268 trim_cache size 0 max 16384 2024-04-02T14:43:48.000 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-02T14:43:48.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:48.002+0000 7f5d9e800640 1 -- 172.21.15.90:0/881220932 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f5d7c0a3f20 msgr2=0x7f5d7c0a63e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:48.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:48.002+0000 7f5d9e800640 1 --2- 172.21.15.90:0/881220932 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f5d7c0a3f20 0x7f5d7c0a63e0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x562663a52f40 tx=0x7f5d9000e040 comp rx=0 tx=0).stop 2024-04-02T14:43:48.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:48.002+0000 7f5d9e800640 1 -- 172.21.15.90:0/881220932 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5d98103f10 msgr2=0x7f5d98076010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:48.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:48.002+0000 7f5d9e800640 1 --2- 172.21.15.90:0/881220932 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5d98103f10 0x7f5d98076010 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f5d9000ba50 tx=0x7f5d9000bf10 comp rx=0 tx=0).stop 2024-04-02T14:43:48.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:48.003+0000 7f5d9e800640 1 -- 172.21.15.90:0/881220932 shutdown_connections 2024-04-02T14:43:48.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:48.003+0000 7f5d9e800640 1 --2- 172.21.15.90:0/881220932 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f5d7c0a3f20 0x7f5d7c0a63e0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:48.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:48.003+0000 7f5d9e800640 1 --2- 172.21.15.90:0/881220932 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5d98076a90 0x7f5d981aac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:48.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:48.003+0000 7f5d9e800640 1 --2- 172.21.15.90:0/881220932 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5d98108fe0 0x7f5d98076550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:48.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:48.003+0000 7f5d9e800640 1 --2- 172.21.15.90:0/881220932 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5d98103f10 0x7f5d98076010 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:48.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:48.003+0000 7f5d9e800640 1 -- 172.21.15.90:0/881220932 >> 172.21.15.90:0/881220932 conn(0x7f5d980ff770 msgr2=0x7f5d98101aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:43:48.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:48.003+0000 7f5d9e800640 1 -- 172.21.15.90:0/881220932 shutdown_connections 2024-04-02T14:43:48.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:48.003+0000 7f5d9e800640 1 -- 172.21.15.90:0/881220932 wait complete. 2024-04-02T14:43:48.434 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:43:48 smithi090 ceph-mon[31674]: pgmap v274: 97 pgs: 97 active+clean; 4.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 185 KiB/s rd, 20 MiB/s wr, 1.27k op/s 2024-04-02T14:43:48.606 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:43:48 smithi092 ceph-mon[36497]: pgmap v274: 97 pgs: 97 active+clean; 4.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 185 KiB/s rd, 20 MiB/s wr, 1.27k op/s 2024-04-02T14:43:48.682 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:43:48 smithi177 ceph-mon[36431]: pgmap v274: 97 pgs: 97 active+clean; 4.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 185 KiB/s rd, 20 MiB/s wr, 1.27k op/s 2024-04-02T14:43:49.023 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-02T14:43:49.024 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T14:43:49.024 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-02T14:43:49.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.294+0000 7fc883800640 1 -- 172.21.15.90:0/3754374170 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc87c10d3e0 msgr2=0x7fc87c114fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:49.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.294+0000 7fc883800640 1 --2- 172.21.15.90:0/3754374170 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc87c10d3e0 0x7fc87c114fb0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fc874009ca0 tx=0x7fc87402c100 comp rx=0 tx=0).stop 2024-04-02T14:43:49.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.294+0000 7fc883800640 1 -- 172.21.15.90:0/3754374170 shutdown_connections 2024-04-02T14:43:49.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.294+0000 7fc883800640 1 --2- 172.21.15.90:0/3754374170 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc87c10d3e0 0x7fc87c114fb0 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:49.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.294+0000 7fc883800640 1 --2- 172.21.15.90:0/3754374170 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc87c108cc0 0x7fc87c10cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:49.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.294+0000 7fc883800640 1 --2- 172.21.15.90:0/3754374170 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc87c108310 0x7fc87c1086f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:49.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.294+0000 7fc883800640 1 -- 172.21.15.90:0/3754374170 >> 172.21.15.90:0/3754374170 conn(0x7fc87c076040 msgr2=0x7fc87c078460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:43:49.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.294+0000 7fc883800640 1 -- 172.21.15.90:0/3754374170 shutdown_connections 2024-04-02T14:43:49.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.294+0000 7fc883800640 1 -- 172.21.15.90:0/3754374170 wait complete. 2024-04-02T14:43:49.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.296+0000 7fc883800640 1 Processor -- start 2024-04-02T14:43:49.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.296+0000 7fc883800640 1 -- start start 2024-04-02T14:43:49.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.296+0000 7fc883800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc87c108310 0x7fc87c1a46b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:49.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.296+0000 7fc883800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc87c10d3e0 0x7fc87c1a4bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:49.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.296+0000 7fc883800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc87c19e820 0x7fc87c19ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:49.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.296+0000 7fc883800640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fc87c1184a0 con 0x7fc87c19e820 2024-04-02T14:43:49.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.296+0000 7fc883800640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fc87c118320 con 0x7fc87c108310 2024-04-02T14:43:49.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.296+0000 7fc883800640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fc87c118620 con 0x7fc87c10d3e0 2024-04-02T14:43:49.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.297+0000 7fc881800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc87c19e820 0x7fc87c19ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:49.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.297+0000 7fc87be00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc87c10d3e0 0x7fc87c1a4bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:49.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.297+0000 7fc880e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc87c108310 0x7fc87c1a46b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:49.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.297+0000 7fc881800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc87c19e820 0x7fc87c19ecd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33842/0 (socket says 172.21.15.90:33842) 2024-04-02T14:43:49.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.297+0000 7fc87be00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc87c10d3e0 0x7fc87c1a4bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:52754/0 (socket says 172.21.15.90:52754) 2024-04-02T14:43:49.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.297+0000 7fc881800640 1 -- 172.21.15.90:0/2307091508 learned_addr learned my addr 172.21.15.90:0/2307091508 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:43:49.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.297+0000 7fc880e00640 1 -- 172.21.15.90:0/2307091508 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc87c10d3e0 msgr2=0x7fc87c1a4bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:49.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.297+0000 7fc880e00640 1 --2- 172.21.15.90:0/2307091508 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc87c10d3e0 0x7fc87c1a4bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:49.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.297+0000 7fc880e00640 1 -- 172.21.15.90:0/2307091508 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc87c19e820 msgr2=0x7fc87c19ecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:49.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.297+0000 7fc880e00640 1 --2- 172.21.15.90:0/2307091508 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc87c19e820 0x7fc87c19ecd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:49.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.297+0000 7fc880e00640 1 -- 172.21.15.90:0/2307091508 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc87c19f3b0 con 0x7fc87c108310 2024-04-02T14:43:49.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.298+0000 7fc881800640 1 --2- 172.21.15.90:0/2307091508 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc87c19e820 0x7fc87c19ecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:43:49.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.298+0000 7fc880e00640 1 --2- 172.21.15.90:0/2307091508 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc87c108310 0x7fc87c1a46b0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7fc87000e9f0 tx=0x7fc87000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:43:49.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.298+0000 7fc879600640 1 -- 172.21.15.90:0/2307091508 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc87000ccc0 con 0x7fc87c108310 2024-04-02T14:43:49.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.298+0000 7fc879600640 1 -- 172.21.15.90:0/2307091508 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc870005020 con 0x7fc87c108310 2024-04-02T14:43:49.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.298+0000 7fc879600640 1 -- 172.21.15.90:0/2307091508 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc870004a00 con 0x7fc87c108310 2024-04-02T14:43:49.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.298+0000 7fc883800640 1 -- 172.21.15.90:0/2307091508 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc87c19f6a0 con 0x7fc87c108310 2024-04-02T14:43:49.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.298+0000 7fc883800640 1 -- 172.21.15.90:0/2307091508 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc87c1ab830 con 0x7fc87c108310 2024-04-02T14:43:49.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.304+0000 7fc886426740 1 Processor -- start 2024-04-02T14:43:49.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.304+0000 7fc886426740 1 -- start start 2024-04-02T14:43:49.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.305+0000 7fc886426740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559e4bc11030 0x559e4bc0fd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:49.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.305+0000 7fc886426740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559e4bc10350 0x559e4bc107d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:49.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.305+0000 7fc886426740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559e4bc196c0 0x559e4bc1bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:49.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.305+0000 7fc886426740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x559e4bb74310 con 0x559e4bc11030 2024-04-02T14:43:49.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.305+0000 7fc886426740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x559e4bb1c690 con 0x559e4bc196c0 2024-04-02T14:43:49.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.305+0000 7fc886426740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x559e4bb702c0 con 0x559e4bc10350 2024-04-02T14:43:49.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.305+0000 7fc87be00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559e4bc196c0 0x559e4bc1bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:49.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.305+0000 7fc87be00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559e4bc196c0 0x559e4bc1bab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:36526/0 (socket says 172.21.15.90:36526) 2024-04-02T14:43:49.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.305+0000 7fc87be00640 1 -- 172.21.15.90:0/3102964367 learned_addr learned my addr 172.21.15.90:0/3102964367 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:43:49.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.306+0000 7fc87be00640 1 -- 172.21.15.90:0/3102964367 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559e4bc10350 msgr2=0x559e4bc107d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-02T14:43:49.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.306+0000 7fc87be00640 1 --2- 172.21.15.90:0/3102964367 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559e4bc10350 0x559e4bc107d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:49.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.306+0000 7fc87be00640 1 -- 172.21.15.90:0/3102964367 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559e4bc11030 msgr2=0x559e4bc0fd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:43:49.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.306+0000 7fc87be00640 1 --2- 172.21.15.90:0/3102964367 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559e4bc11030 0x559e4bc0fd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:49.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.306+0000 7fc87be00640 1 -- 172.21.15.90:0/3102964367 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559e4b9507e0 con 0x559e4bc196c0 2024-04-02T14:43:49.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.306+0000 7fc879600640 1 -- 172.21.15.90:0/2307091508 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fc870005560 con 0x7fc87c108310 2024-04-02T14:43:49.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.306+0000 7fc87be00640 1 --2- 172.21.15.90:0/3102964367 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559e4bc196c0 0x559e4bc1bab0 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7fc86800bba0 tx=0x7fc8680107a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ad995fbe2a56a62e server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:43:49.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.306+0000 7fc86e000640 1 -- 172.21.15.90:0/3102964367 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc86801c480 con 0x559e4bc196c0 2024-04-02T14:43:49.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.306+0000 7fc86e000640 1 -- 172.21.15.90:0/3102964367 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc868011920 con 0x559e4bc196c0 2024-04-02T14:43:49.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.306+0000 7fc86e000640 1 -- 172.21.15.90:0/3102964367 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc86801a8c0 con 0x559e4bc196c0 2024-04-02T14:43:49.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.307+0000 7fc886426740 1 -- 172.21.15.90:0/3102964367 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559e4bc196c0 msgr2=0x559e4bc1bab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:49.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.307+0000 7fc886426740 1 --2- 172.21.15.90:0/3102964367 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559e4bc196c0 0x559e4bc1bab0 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7fc86800bba0 tx=0x7fc8680107a0 comp rx=0 tx=0).stop 2024-04-02T14:43:49.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.307+0000 7fc886426740 1 -- 172.21.15.90:0/3102964367 shutdown_connections 2024-04-02T14:43:49.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.307+0000 7fc886426740 1 --2- 172.21.15.90:0/3102964367 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559e4bc196c0 0x559e4bc1bab0 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:49.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.307+0000 7fc886426740 1 --2- 172.21.15.90:0/3102964367 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559e4bc10350 0x559e4bc107d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:49.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.307+0000 7fc886426740 1 --2- 172.21.15.90:0/3102964367 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559e4bc11030 0x559e4bc0fd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:49.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.307+0000 7fc886426740 1 -- 172.21.15.90:0/3102964367 >> 172.21.15.90:0/3102964367 conn(0x559e4bba4270 msgr2=0x559e4bc0ada0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:43:49.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.308+0000 7fc886426740 1 -- 172.21.15.90:0/3102964367 shutdown_connections 2024-04-02T14:43:49.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.308+0000 7fc886426740 1 -- 172.21.15.90:0/3102964367 wait complete. 2024-04-02T14:43:49.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.308+0000 7fc879600640 1 --2- 172.21.15.90:0/2307091508 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fc8640a3d70 0x7fc8640a6230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:49.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.308+0000 7fc879600640 1 -- 172.21.15.90:0/2307091508 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8550+0+0 (secure 0 0 0) 0x7fc8700d0780 con 0x7fc87c108310 2024-04-02T14:43:49.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.308+0000 7fc886426740 1 -- start start 2024-04-02T14:43:49.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.309+0000 7fc87be00640 1 --2- 172.21.15.90:0/2307091508 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fc8640a3d70 0x7fc8640a6230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:49.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.309+0000 7fc886426740 1 Processor -- start 2024-04-02T14:43:49.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.310+0000 7fc87be00640 1 --2- 172.21.15.90:0/2307091508 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fc8640a3d70 0x7fc8640a6230 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fc86800b960 tx=0x7fc868013040 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:43:49.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.310+0000 7fc886426740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559e4bc10350 0x559e4bc166a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:49.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.310+0000 7fc886426740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559e4bc11030 0x559e4bc16be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:49.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.310+0000 7fc886426740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559e4bc196c0 0x559e4bc17120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:49.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.310+0000 7fc886426740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x559e4bc1c530 con 0x559e4bc10350 2024-04-02T14:43:49.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.310+0000 7fc886426740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x559e4bb2fa60 con 0x559e4bc196c0 2024-04-02T14:43:49.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.310+0000 7fc886426740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x559e4bc1c6b0 con 0x559e4bc11030 2024-04-02T14:43:49.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.310+0000 7fc886426740 10 client.0 _ll_register_callbacks cb 0x559e4bb35030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:43:49.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.310+0000 7fc87be00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559e4bc11030 0x559e4bc16be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:49.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.310+0000 7fc87be00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559e4bc11030 0x559e4bc16be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:52770/0 (socket says 172.21.15.90:52770) 2024-04-02T14:43:49.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.310+0000 7fc87be00640 1 -- 172.21.15.90:0/519475772 learned_addr learned my addr 172.21.15.90:0/519475772 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:43:49.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.310+0000 7fc881800640 1 --2- 172.21.15.90:0/519475772 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559e4bc196c0 0x559e4bc17120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:49.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.310+0000 7fc880e00640 1 --2- 172.21.15.90:0/519475772 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559e4bc10350 0x559e4bc166a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:49.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.310+0000 7fc87be00640 1 -- 172.21.15.90:0/519475772 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559e4bc196c0 msgr2=0x559e4bc17120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:43:49.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.310+0000 7fc87be00640 1 --2- 172.21.15.90:0/519475772 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559e4bc196c0 0x559e4bc17120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:49.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.310+0000 7fc87be00640 1 -- 172.21.15.90:0/519475772 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559e4bc10350 msgr2=0x559e4bc166a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:43:49.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.310+0000 7fc87be00640 1 --2- 172.21.15.90:0/519475772 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559e4bc10350 0x559e4bc166a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:49.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.310+0000 7fc87be00640 1 -- 172.21.15.90:0/519475772 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559e4b9b0f70 con 0x559e4bc11030 2024-04-02T14:43:49.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.311+0000 7fc87be00640 1 --2- 172.21.15.90:0/519475772 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559e4bc11030 0x559e4bc16be0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fc868011dd0 tx=0x7fc86801a380 comp rx=0 tx=0).ready entity=mon.1 client_cookie=17ecdc6b1f42d21f server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:43:49.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.311+0000 7fc862a00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:43:49.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.311+0000 7fc862a00640 1 -- 172.21.15.90:0/519475772 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc868037060 con 0x559e4bc11030 2024-04-02T14:43:49.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.311+0000 7fc886426740 1 -- 172.21.15.90:0/519475772 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc868011920 con 0x559e4bc11030 2024-04-02T14:43:49.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.311+0000 7fc862a00640 1 -- 172.21.15.90:0/519475772 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8550+0+0 (secure 0 0 0) 0x7fc86801a910 con 0x559e4bc11030 2024-04-02T14:43:49.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.311+0000 7fc862a00640 1 -- 172.21.15.90:0/519475772 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fc86801ab10 con 0x559e4bc11030 2024-04-02T14:43:49.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.311+0000 7fc886426740 10 client.28111 fetch_fsmap learned FSMap version 27 2024-04-02T14:43:49.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.311+0000 7fc886426740 1 -- 172.21.15.90:0/519475772 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x559e4b848170 con 0x559e4bc11030 2024-04-02T14:43:49.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.312+0000 7fc862a00640 1 -- 172.21.15.90:0/519475772 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc868011920 con 0x559e4bc11030 2024-04-02T14:43:49.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.312+0000 7fc862a00640 1 -- 172.21.15.90:0/519475772 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fc868027410 con 0x559e4bc11030 2024-04-02T14:43:49.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.312+0000 7fc886426740 10 client.28111 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:43:49.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.312+0000 7fc886426740 10 client.28111 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:43:49.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.312+0000 7fc886426740 20 client.28111 populate_metadata read hostname 'smithi090' 2024-04-02T14:43:49.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.312+0000 7fc886426740 1 --2- 172.21.15.90:0/519475772 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x559e4bc29aa0 0x559e4bc2be90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:49.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.312+0000 7fc886426740 4 client.28111 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:43:49.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.312+0000 7fc886426740 1 -- 172.21.15.90:0/519475772 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559e4bb81850 con 0x559e4bc29aa0 2024-04-02T14:43:49.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.312+0000 7fc880e00640 1 --2- 172.21.15.90:0/519475772 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x559e4bc29aa0 0x559e4bc2be90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:49.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.313+0000 7fc880e00640 1 --2- 172.21.15.90:0/519475772 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x559e4bc29aa0 0x559e4bc2be90 crc :-1 s=READY pgs=242 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cc17b64b21267484 server_cookie=3547c1531ec2a7 in_seq=0 out_seq=0 2024-04-02T14:43:49.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.313+0000 7fc862a00640 10 client.28111 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:43:49.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.313+0000 7fc862a00640 1 -- 172.21.15.90:0/519475772 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x559e4bb81850 con 0x559e4bc29aa0 2024-04-02T14:43:49.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.313+0000 7fc862a00640 10 client.28111 handle_command_reply: tid=0 2024-04-02T14:43:49.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.314+0000 7fc886426740 1 client.28111 shutdown 2024-04-02T14:43:49.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.314+0000 7fc886426740 1 -- 172.21.15.90:0/519475772 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559e4bc11030 msgr2=0x559e4bc16be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:49.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.314+0000 7fc886426740 1 --2- 172.21.15.90:0/519475772 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559e4bc11030 0x559e4bc16be0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fc868011dd0 tx=0x7fc86801a380 comp rx=0 tx=0).stop 2024-04-02T14:43:49.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.314+0000 7fc886426740 1 -- 172.21.15.90:0/519475772 shutdown_connections 2024-04-02T14:43:49.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.314+0000 7fc886426740 1 --2- 172.21.15.90:0/519475772 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x559e4bc29aa0 0x559e4bc2be90 crc :-1 s=READY pgs=242 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:49.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.314+0000 7fc886426740 1 --2- 172.21.15.90:0/519475772 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559e4bc196c0 0x559e4bc17120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:49.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.314+0000 7fc886426740 1 --2- 172.21.15.90:0/519475772 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559e4bc11030 0x559e4bc16be0 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:49.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.314+0000 7fc886426740 1 --2- 172.21.15.90:0/519475772 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559e4bc10350 0x559e4bc166a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:49.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.314+0000 7fc862a00640 0 client.28111 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:43:49.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.314+0000 7fc886426740 1 -- 172.21.15.90:0/519475772 >> 172.21.15.90:0/519475772 conn(0x559e4bba4270 msgr2=0x559e4bc0caf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:43:49.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.315+0000 7fc886426740 1 -- 172.21.15.90:0/519475772 shutdown_connections 2024-04-02T14:43:49.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.315+0000 7fc886426740 1 -- 172.21.15.90:0/519475772 wait complete. 2024-04-02T14:43:49.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.316+0000 7fc886426740 20 client.28111 trim_cache size 0 max 16384 2024-04-02T14:43:49.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.350+0000 7fc886426740 1 Processor -- start 2024-04-02T14:43:49.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.350+0000 7fc886426740 1 -- start start 2024-04-02T14:43:49.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.351+0000 7fc886426740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559e4bc10350 0x559e4bc254c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:49.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.351+0000 7fc886426740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559e4bc11030 0x559e4bc31490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:49.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.351+0000 7fc886426740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559e4bc196c0 0x559e4bc319d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:49.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.351+0000 7fc886426740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fc87c1b3f80 con 0x559e4bc11030 2024-04-02T14:43:49.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.351+0000 7fc886426740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fc87c1b5ac0 con 0x559e4bc10350 2024-04-02T14:43:49.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.351+0000 7fc886426740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fc87c1b4650 con 0x559e4bc196c0 2024-04-02T14:43:49.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.352+0000 7fc880e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559e4bc10350 0x559e4bc254c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:49.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.352+0000 7fc881800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559e4bc196c0 0x559e4bc319d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:49.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.352+0000 7fc881800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559e4bc196c0 0x559e4bc319d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:52780/0 (socket says 172.21.15.90:52780) 2024-04-02T14:43:49.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.352+0000 7fc881800640 1 -- 172.21.15.90:0/540531158 learned_addr learned my addr 172.21.15.90:0/540531158 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:43:49.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.352+0000 7fc880e00640 1 -- 172.21.15.90:0/540531158 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559e4bc196c0 msgr2=0x559e4bc319d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:43:49.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.352+0000 7fc880e00640 1 --2- 172.21.15.90:0/540531158 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559e4bc196c0 0x559e4bc319d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:49.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.352+0000 7fc880e00640 1 -- 172.21.15.90:0/540531158 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559e4bc11030 msgr2=0x559e4bc31490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:43:49.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.352+0000 7fc880e00640 1 --2- 172.21.15.90:0/540531158 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559e4bc11030 0x559e4bc31490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:49.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.352+0000 7fc880e00640 1 -- 172.21.15.90:0/540531158 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559e4bc04fc0 con 0x559e4bc10350 2024-04-02T14:43:49.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.353+0000 7fc880e00640 1 --2- 172.21.15.90:0/540531158 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559e4bc10350 0x559e4bc254c0 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7fc8700cde50 tx=0x7fc8700054d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d656ac4ecf3da82f server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:43:49.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.355+0000 7fc86d600640 1 -- 172.21.15.90:0/540531158 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc87001c040 con 0x559e4bc10350 2024-04-02T14:43:49.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.355+0000 7fc86d600640 1 -- 172.21.15.90:0/540531158 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc87009a9a0 con 0x559e4bc10350 2024-04-02T14:43:49.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.355+0000 7fc86d600640 1 -- 172.21.15.90:0/540531158 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc87009c900 con 0x559e4bc10350 2024-04-02T14:43:49.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.357+0000 7fc886426740 1 -- 172.21.15.90:0/540531158 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559e4bc10350 msgr2=0x559e4bc254c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:49.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.357+0000 7fc886426740 1 --2- 172.21.15.90:0/540531158 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559e4bc10350 0x559e4bc254c0 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7fc8700cde50 tx=0x7fc8700054d0 comp rx=0 tx=0).stop 2024-04-02T14:43:49.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.357+0000 7fc886426740 1 -- 172.21.15.90:0/540531158 shutdown_connections 2024-04-02T14:43:49.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.357+0000 7fc886426740 1 --2- 172.21.15.90:0/540531158 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559e4bc196c0 0x559e4bc319d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:49.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.357+0000 7fc886426740 1 --2- 172.21.15.90:0/540531158 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559e4bc11030 0x559e4bc31490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:49.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.357+0000 7fc886426740 1 --2- 172.21.15.90:0/540531158 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559e4bc10350 0x559e4bc254c0 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:49.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.357+0000 7fc886426740 1 -- 172.21.15.90:0/540531158 >> 172.21.15.90:0/540531158 conn(0x559e4bba4270 msgr2=0x559e4bc09e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:43:49.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.358+0000 7fc886426740 1 -- 172.21.15.90:0/540531158 shutdown_connections 2024-04-02T14:43:49.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.358+0000 7fc886426740 1 -- 172.21.15.90:0/540531158 wait complete. 2024-04-02T14:43:49.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.358+0000 7fc886426740 1 -- start start 2024-04-02T14:43:49.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.359+0000 7fc886426740 1 Processor -- start 2024-04-02T14:43:49.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.359+0000 7fc886426740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559e4bc10350 0x559e4bc21bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:49.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.359+0000 7fc886426740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559e4bc11030 0x559e4bc1e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:49.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.359+0000 7fc886426740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559e4bc196c0 0x559e4bc1eee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:49.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.359+0000 7fc886426740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fc87c1b53f0 con 0x559e4bc10350 2024-04-02T14:43:49.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.359+0000 7fc886426740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fc87c1b4d20 con 0x559e4bc196c0 2024-04-02T14:43:49.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.359+0000 7fc886426740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fc87c1b1f80 con 0x559e4bc11030 2024-04-02T14:43:49.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.359+0000 7fc886426740 10 client.0 _ll_register_callbacks cb 0x7fc86800ba20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:43:49.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.360+0000 7fc87be00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559e4bc11030 0x559e4bc1e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:49.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.360+0000 7fc87be00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559e4bc11030 0x559e4bc1e9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:52790/0 (socket says 172.21.15.90:52790) 2024-04-02T14:43:49.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.360+0000 7fc87be00640 1 -- 172.21.15.90:0/1090507579 learned_addr learned my addr 172.21.15.90:0/1090507579 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:43:49.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.361+0000 7fc87be00640 1 -- 172.21.15.90:0/1090507579 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559e4bc196c0 msgr2=0x559e4bc1eee0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-02T14:43:49.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.361+0000 7fc87be00640 1 --2- 172.21.15.90:0/1090507579 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559e4bc196c0 0x559e4bc1eee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:49.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.361+0000 7fc87be00640 1 -- 172.21.15.90:0/1090507579 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559e4bc10350 msgr2=0x559e4bc21bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:43:49.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.361+0000 7fc87be00640 1 --2- 172.21.15.90:0/1090507579 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559e4bc10350 0x559e4bc21bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:49.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.362+0000 7fc87be00640 1 -- 172.21.15.90:0/1090507579 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559e4bc1f420 con 0x559e4bc11030 2024-04-02T14:43:49.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.362+0000 7fc87be00640 1 --2- 172.21.15.90:0/1090507579 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559e4bc11030 0x559e4bc1e9a0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7fc868005d00 tx=0x7fc868010de0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1681d731f2a72331 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:43:49.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.362+0000 7fc862a00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:43:49.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.365+0000 7fc862a00640 1 -- 172.21.15.90:0/1090507579 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc868037070 con 0x559e4bc11030 2024-04-02T14:43:49.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.365+0000 7fc862a00640 1 -- 172.21.15.90:0/1090507579 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8550+0+0 (secure 0 0 0) 0x7fc86801a910 con 0x559e4bc11030 2024-04-02T14:43:49.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.365+0000 7fc886426740 1 -- 172.21.15.90:0/1090507579 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc87009a9a0 con 0x559e4bc11030 2024-04-02T14:43:49.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.365+0000 7fc862a00640 1 -- 172.21.15.90:0/1090507579 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fc86804a050 con 0x559e4bc11030 2024-04-02T14:43:49.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.365+0000 7fc886426740 10 client.28117 fetch_fsmap learned FSMap version 27 2024-04-02T14:43:49.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.366+0000 7fc886426740 1 -- 172.21.15.90:0/1090507579 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x559e4bca1a40 con 0x559e4bc11030 2024-04-02T14:43:49.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.366+0000 7fc862a00640 1 -- 172.21.15.90:0/1090507579 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc87009a9a0 con 0x559e4bc11030 2024-04-02T14:43:49.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.366+0000 7fc862a00640 1 -- 172.21.15.90:0/1090507579 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fc868027a50 con 0x559e4bc11030 2024-04-02T14:43:49.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.367+0000 7fc886426740 10 client.28117 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:43:49.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.367+0000 7fc886426740 10 client.28117 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:43:49.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.367+0000 7fc886426740 20 client.28117 populate_metadata read hostname 'smithi090' 2024-04-02T14:43:49.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.367+0000 7fc886426740 1 --2- 172.21.15.90:0/1090507579 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x559e4bca2160 0x559e4bca45b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:49.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.367+0000 7fc886426740 4 client.28117 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-02T14:43:49.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.367+0000 7fc886426740 1 -- 172.21.15.90:0/1090507579 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x559e4bb7bba0 con 0x559e4bca2160 2024-04-02T14:43:49.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.368+0000 7fc880e00640 1 --2- 172.21.15.90:0/1090507579 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x559e4bca2160 0x559e4bca45b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:49.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.369+0000 7fc880e00640 1 --2- 172.21.15.90:0/1090507579 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x559e4bca2160 0x559e4bca45b0 crc :-1 s=READY pgs=243 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7d4115cb02f8f667 server_cookie=54da9748629b3218 in_seq=0 out_seq=0 2024-04-02T14:43:49.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.369+0000 7fc862a00640 10 client.28117 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:43:49.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.374+0000 7fc862a00640 1 -- 172.21.15.90:0/1090507579 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x559e4bb7bba0 con 0x559e4bca2160 2024-04-02T14:43:49.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.375+0000 7fc862a00640 10 client.28117 handle_command_reply: tid=0 2024-04-02T14:43:49.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.375+0000 7fc886426740 1 client.28117 shutdown 2024-04-02T14:43:49.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.375+0000 7fc886426740 1 -- 172.21.15.90:0/1090507579 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559e4bc11030 msgr2=0x559e4bc1e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:49.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.375+0000 7fc886426740 1 --2- 172.21.15.90:0/1090507579 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559e4bc11030 0x559e4bc1e9a0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7fc868005d00 tx=0x7fc868010de0 comp rx=0 tx=0).stop 2024-04-02T14:43:49.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.376+0000 7fc886426740 1 -- 172.21.15.90:0/1090507579 shutdown_connections 2024-04-02T14:43:49.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.376+0000 7fc886426740 1 --2- 172.21.15.90:0/1090507579 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x559e4bca2160 0x559e4bca45b0 crc :-1 s=READY pgs=243 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:49.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.376+0000 7fc886426740 1 --2- 172.21.15.90:0/1090507579 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559e4bc196c0 0x559e4bc1eee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:49.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.376+0000 7fc886426740 1 --2- 172.21.15.90:0/1090507579 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559e4bc11030 0x559e4bc1e9a0 secure :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7fc868005d00 tx=0x7fc868010de0 comp rx=0 tx=0).stop 2024-04-02T14:43:49.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.376+0000 7fc886426740 1 --2- 172.21.15.90:0/1090507579 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559e4bc10350 0x559e4bc21bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:49.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.376+0000 7fc862a00640 0 client.28117 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:43:49.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.376+0000 7fc886426740 1 -- 172.21.15.90:0/1090507579 >> 172.21.15.90:0/1090507579 conn(0x559e4bba4270 msgr2=0x559e4bc0c070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:43:49.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.376+0000 7fc886426740 1 -- 172.21.15.90:0/1090507579 shutdown_connections 2024-04-02T14:43:49.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.376+0000 7fc886426740 1 -- 172.21.15.90:0/1090507579 wait complete. 2024-04-02T14:43:49.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.377+0000 7fc886426740 20 client.28117 trim_cache size 0 max 16384 2024-04-02T14:43:49.375 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-02T14:43:49.375 INFO:teuthology.orchestra.run.smithi090.stdout: "return_code": 0, 2024-04-02T14:43:49.375 INFO:teuthology.orchestra.run.smithi090.stdout: "scrub_tag": "4c95946c-3531-4cd0-993f-dcbcdb2ebc91", 2024-04-02T14:43:49.376 INFO:teuthology.orchestra.run.smithi090.stdout: "mode": "asynchronous" 2024-04-02T14:43:49.376 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-02T14:43:49.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.379+0000 7fc883800640 1 -- 172.21.15.90:0/2307091508 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fc8640a3d70 msgr2=0x7fc8640a6230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:49.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.379+0000 7fc883800640 1 --2- 172.21.15.90:0/2307091508 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fc8640a3d70 0x7fc8640a6230 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fc86800b960 tx=0x7fc868013040 comp rx=0 tx=0).stop 2024-04-02T14:43:49.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.379+0000 7fc883800640 1 -- 172.21.15.90:0/2307091508 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc87c108310 msgr2=0x7fc87c1a46b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:49.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.379+0000 7fc883800640 1 --2- 172.21.15.90:0/2307091508 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc87c108310 0x7fc87c1a46b0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7fc87000e9f0 tx=0x7fc87000eeb0 comp rx=0 tx=0).stop 2024-04-02T14:43:49.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.380+0000 7fc883800640 1 -- 172.21.15.90:0/2307091508 shutdown_connections 2024-04-02T14:43:49.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.380+0000 7fc883800640 1 --2- 172.21.15.90:0/2307091508 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fc8640a3d70 0x7fc8640a6230 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:49.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.380+0000 7fc883800640 1 --2- 172.21.15.90:0/2307091508 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc87c19e820 0x7fc87c19ecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:49.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.380+0000 7fc883800640 1 --2- 172.21.15.90:0/2307091508 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc87c10d3e0 0x7fc87c1a4bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:49.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.380+0000 7fc883800640 1 --2- 172.21.15.90:0/2307091508 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc87c108310 0x7fc87c1a46b0 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:49.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.380+0000 7fc883800640 1 -- 172.21.15.90:0/2307091508 >> 172.21.15.90:0/2307091508 conn(0x7fc87c076040 msgr2=0x7fc87c078370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:43:49.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.380+0000 7fc883800640 1 -- 172.21.15.90:0/2307091508 shutdown_connections 2024-04-02T14:43:49.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.381+0000 7fc883800640 1 -- 172.21.15.90:0/2307091508 wait complete. 2024-04-02T14:43:49.400 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-02T14:43:49.401 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-02T14:43:49.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.666+0000 7fa76be00640 1 -- 172.21.15.90:0/1697012343 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa76c102b30 msgr2=0x7fa76c102f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:49.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.666+0000 7fa76be00640 1 --2- 172.21.15.90:0/1697012343 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa76c102b30 0x7fa76c102f10 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fa75c009860 tx=0x7fa75c02bc10 comp rx=0 tx=0).stop 2024-04-02T14:43:49.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.666+0000 7fa76be00640 1 -- 172.21.15.90:0/1697012343 shutdown_connections 2024-04-02T14:43:49.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.666+0000 7fa76be00640 1 --2- 172.21.15.90:0/1697012343 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa76c10bf50 0x7fa76c10e3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:49.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.666+0000 7fa76be00640 1 --2- 172.21.15.90:0/1697012343 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa76c103450 0x7fa76c10b930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:49.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.666+0000 7fa76be00640 1 --2- 172.21.15.90:0/1697012343 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa76c102b30 0x7fa76c102f10 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:49.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.666+0000 7fa76be00640 1 -- 172.21.15.90:0/1697012343 >> 172.21.15.90:0/1697012343 conn(0x7fa76c0fc350 msgr2=0x7fa76c0fe770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:43:49.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.667+0000 7fa76be00640 1 -- 172.21.15.90:0/1697012343 shutdown_connections 2024-04-02T14:43:49.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.667+0000 7fa76be00640 1 -- 172.21.15.90:0/1697012343 wait complete. 2024-04-02T14:43:49.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.668+0000 7fa76be00640 1 Processor -- start 2024-04-02T14:43:49.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.668+0000 7fa76be00640 1 -- start start 2024-04-02T14:43:49.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.668+0000 7fa76be00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa76c103450 0x7fa76c19a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:49.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.668+0000 7fa76be00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa76c10bf50 0x7fa76c19abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:49.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.668+0000 7fa76be00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa76c19ef30 0x7fa76c19f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:49.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.668+0000 7fa76be00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fa76c111d50 con 0x7fa76c19ef30 2024-04-02T14:43:49.666 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.668+0000 7fa76be00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fa76c111bd0 con 0x7fa76c10bf50 2024-04-02T14:43:49.666 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.668+0000 7fa76be00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fa76c111ed0 con 0x7fa76c103450 2024-04-02T14:43:49.666 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.668+0000 7fa76aa00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa76c103450 0x7fa76c19a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:49.666 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.668+0000 7fa76a000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa76c10bf50 0x7fa76c19abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:49.666 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.668+0000 7fa76b400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa76c19ef30 0x7fa76c19f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:49.666 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.668+0000 7fa76aa00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa76c103450 0x7fa76c19a670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:52808/0 (socket says 172.21.15.90:52808) 2024-04-02T14:43:49.666 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.668+0000 7fa76aa00640 1 -- 172.21.15.90:0/3485341765 learned_addr learned my addr 172.21.15.90:0/3485341765 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:43:49.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.669+0000 7fa76a000640 1 -- 172.21.15.90:0/3485341765 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa76c103450 msgr2=0x7fa76c19a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:49.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.669+0000 7fa76a000640 1 --2- 172.21.15.90:0/3485341765 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa76c103450 0x7fa76c19a670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:49.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.669+0000 7fa76a000640 1 -- 172.21.15.90:0/3485341765 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa76c19ef30 msgr2=0x7fa76c19f3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:49.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.669+0000 7fa76a000640 1 --2- 172.21.15.90:0/3485341765 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa76c19ef30 0x7fa76c19f3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:49.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.670+0000 7fa76a000640 1 -- 172.21.15.90:0/3485341765 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa76c19fa20 con 0x7fa76c10bf50 2024-04-02T14:43:49.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.670+0000 7fa76aa00640 1 --2- 172.21.15.90:0/3485341765 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa76c103450 0x7fa76c19a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:43:49.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.670+0000 7fa76b400640 1 --2- 172.21.15.90:0/3485341765 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa76c19ef30 0x7fa76c19f3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:43:49.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.671+0000 7fa76a000640 1 --2- 172.21.15.90:0/3485341765 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa76c10bf50 0x7fa76c19abb0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7fa75800e9f0 tx=0x7fa75800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:43:49.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.672+0000 7fa763400640 1 -- 172.21.15.90:0/3485341765 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa75800ccc0 con 0x7fa76c10bf50 2024-04-02T14:43:49.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.672+0000 7fa76be00640 1 -- 172.21.15.90:0/3485341765 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa76c19fd10 con 0x7fa76c10bf50 2024-04-02T14:43:49.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.672+0000 7fa763400640 1 -- 172.21.15.90:0/3485341765 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa7580035f0 con 0x7fa76c10bf50 2024-04-02T14:43:49.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.672+0000 7fa763400640 1 -- 172.21.15.90:0/3485341765 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa75800ce60 con 0x7fa76c10bf50 2024-04-02T14:43:49.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.672+0000 7fa76be00640 1 -- 172.21.15.90:0/3485341765 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa76c1a7550 con 0x7fa76c10bf50 2024-04-02T14:43:49.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.677+0000 7fa77439b740 1 Processor -- start 2024-04-02T14:43:49.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.678+0000 7fa77439b740 1 -- start start 2024-04-02T14:43:49.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.678+0000 7fa763400640 1 -- 172.21.15.90:0/3485341765 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fa758010670 con 0x7fa76c10bf50 2024-04-02T14:43:49.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.678+0000 7fa77439b740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a7be870960 0x55a7be870d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:49.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.678+0000 7fa77439b740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a7be871280 0x55a7be879920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:49.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.678+0000 7fa77439b740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a7be879e60 0x55a7be87c250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:49.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.678+0000 7fa77439b740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55a7be7d4770 con 0x55a7be871280 2024-04-02T14:43:49.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.678+0000 7fa77439b740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55a7be77c700 con 0x55a7be879e60 2024-04-02T14:43:49.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.678+0000 7fa77439b740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55a7be7d0720 con 0x55a7be870960 2024-04-02T14:43:49.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.679+0000 7fa76b400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a7be870960 0x55a7be870d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:49.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.679+0000 7fa76b400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a7be870960 0x55a7be870d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:52818/0 (socket says 172.21.15.90:52818) 2024-04-02T14:43:49.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.679+0000 7fa76b400640 1 -- 172.21.15.90:0/3068080784 learned_addr learned my addr 172.21.15.90:0/3068080784 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:43:49.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.679+0000 7fa76aa00640 1 --2- 172.21.15.90:0/3068080784 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a7be871280 0x55a7be879920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:49.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.679+0000 7fa76a000640 1 --2- 172.21.15.90:0/3068080784 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a7be879e60 0x55a7be87c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:49.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.679+0000 7fa76b400640 1 -- 172.21.15.90:0/3068080784 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a7be879e60 msgr2=0x55a7be87c250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:43:49.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.679+0000 7fa76b400640 1 --2- 172.21.15.90:0/3068080784 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a7be879e60 0x55a7be87c250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:49.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.679+0000 7fa76b400640 1 -- 172.21.15.90:0/3068080784 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a7be871280 msgr2=0x55a7be879920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:43:49.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.679+0000 7fa76b400640 1 --2- 172.21.15.90:0/3068080784 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a7be871280 0x55a7be879920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:49.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.679+0000 7fa76b400640 1 -- 172.21.15.90:0/3068080784 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a7be5b27d0 con 0x55a7be870960 2024-04-02T14:43:49.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.680+0000 7fa76b400640 1 --2- 172.21.15.90:0/3068080784 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a7be870960 0x55a7be870d40 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fa764009fc0 tx=0x7fa764018cf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6d11a1a48cd19f2c server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:43:49.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.680+0000 7fa763400640 1 --2- 172.21.15.90:0/3485341765 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fa7500a3da0 0x7fa7500a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:49.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.681+0000 7fa763400640 1 -- 172.21.15.90:0/3485341765 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8550+0+0 (secure 0 0 0) 0x7fa7580d0b20 con 0x7fa76c10bf50 2024-04-02T14:43:49.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.681+0000 7fa757400640 1 -- 172.21.15.90:0/3068080784 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa764026070 con 0x55a7be870960 2024-04-02T14:43:49.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.681+0000 7fa757400640 1 -- 172.21.15.90:0/3068080784 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa764003e50 con 0x55a7be870960 2024-04-02T14:43:49.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.681+0000 7fa757400640 1 -- 172.21.15.90:0/3068080784 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa764026070 con 0x55a7be870960 2024-04-02T14:43:49.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.681+0000 7fa76aa00640 1 --2- 172.21.15.90:0/3485341765 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fa7500a3da0 0x7fa7500a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:49.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.682+0000 7fa77439b740 1 -- 172.21.15.90:0/3068080784 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a7be870960 msgr2=0x55a7be870d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:49.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.682+0000 7fa77439b740 1 --2- 172.21.15.90:0/3068080784 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a7be870960 0x55a7be870d40 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fa764009fc0 tx=0x7fa764018cf0 comp rx=0 tx=0).stop 2024-04-02T14:43:49.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.682+0000 7fa77439b740 1 -- 172.21.15.90:0/3068080784 shutdown_connections 2024-04-02T14:43:49.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.682+0000 7fa77439b740 1 --2- 172.21.15.90:0/3068080784 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a7be879e60 0x55a7be87c250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:49.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.682+0000 7fa77439b740 1 --2- 172.21.15.90:0/3068080784 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a7be871280 0x55a7be879920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:49.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.682+0000 7fa77439b740 1 --2- 172.21.15.90:0/3068080784 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a7be870960 0x55a7be870d40 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:49.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.682+0000 7fa77439b740 1 -- 172.21.15.90:0/3068080784 >> 172.21.15.90:0/3068080784 conn(0x55a7be86b090 msgr2=0x55a7be86b470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:43:49.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.682+0000 7fa76aa00640 1 --2- 172.21.15.90:0/3485341765 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fa7500a3da0 0x7fa7500a6260 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fa75c02c220 tx=0x7fa75c02e040 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:43:49.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.682+0000 7fa77439b740 1 -- 172.21.15.90:0/3068080784 shutdown_connections 2024-04-02T14:43:49.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.682+0000 7fa77439b740 1 -- 172.21.15.90:0/3068080784 wait complete. 2024-04-02T14:43:49.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.683+0000 7fa77439b740 1 -- start start 2024-04-02T14:43:49.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.683+0000 7fa77439b740 1 Processor -- start 2024-04-02T14:43:49.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.683+0000 7fa77439b740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a7be870960 0x55a7be876f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:49.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.684+0000 7fa77439b740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a7be871280 0x55a7be877460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:49.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.684+0000 7fa77439b740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a7be879e60 0x55a7be8779a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:49.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.684+0000 7fa76aa00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a7be870960 0x55a7be876f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:49.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.684+0000 7fa77439b740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55a7be87cc50 con 0x55a7be871280 2024-04-02T14:43:49.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.684+0000 7fa77439b740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55a7be78fac0 con 0x55a7be879e60 2024-04-02T14:43:49.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.684+0000 7fa77439b740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55a7be87cdd0 con 0x55a7be870960 2024-04-02T14:43:49.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.684+0000 7fa76aa00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a7be870960 0x55a7be876f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:52824/0 (socket says 172.21.15.90:52824) 2024-04-02T14:43:49.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.684+0000 7fa76aa00640 1 -- 172.21.15.90:0/102851965 learned_addr learned my addr 172.21.15.90:0/102851965 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:43:49.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.684+0000 7fa77439b740 10 client.0 _ll_register_callbacks cb 0x55a7be792180 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:43:49.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.684+0000 7fa76b400640 1 --2- 172.21.15.90:0/102851965 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a7be879e60 0x55a7be8779a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:49.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.684+0000 7fa76a000640 1 --2- 172.21.15.90:0/102851965 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a7be871280 0x55a7be877460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:49.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.685+0000 7fa76aa00640 1 -- 172.21.15.90:0/102851965 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a7be879e60 msgr2=0x55a7be8779a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:43:49.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.685+0000 7fa76aa00640 1 --2- 172.21.15.90:0/102851965 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a7be879e60 0x55a7be8779a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:49.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.685+0000 7fa76aa00640 1 -- 172.21.15.90:0/102851965 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a7be871280 msgr2=0x55a7be877460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:43:49.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.685+0000 7fa76aa00640 1 --2- 172.21.15.90:0/102851965 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a7be871280 0x55a7be877460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:49.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.685+0000 7fa76aa00640 1 -- 172.21.15.90:0/102851965 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a7be4e83f0 con 0x55a7be870960 2024-04-02T14:43:49.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.685+0000 7fa76a000640 1 --2- 172.21.15.90:0/102851965 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a7be871280 0x55a7be877460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:43:49.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.685+0000 7fa76aa00640 1 --2- 172.21.15.90:0/102851965 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a7be870960 0x55a7be876f20 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fa75c009230 tx=0x7fa75c02bc10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fda7c18c2a8cae70 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:43:49.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.685+0000 7fa74be00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:43:49.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.686+0000 7fa74be00640 1 -- 172.21.15.90:0/102851965 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa75c051020 con 0x55a7be870960 2024-04-02T14:43:49.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.686+0000 7fa74be00640 1 -- 172.21.15.90:0/102851965 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8550+0+0 (secure 0 0 0) 0x7fa75c02c7b0 con 0x55a7be870960 2024-04-02T14:43:49.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.686+0000 7fa77439b740 1 -- 172.21.15.90:0/102851965 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa764003e50 con 0x55a7be870960 2024-04-02T14:43:49.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.686+0000 7fa74be00640 1 -- 172.21.15.90:0/102851965 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fa75c039040 con 0x55a7be870960 2024-04-02T14:43:49.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.686+0000 7fa77439b740 10 client.28129 fetch_fsmap learned FSMap version 27 2024-04-02T14:43:49.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.686+0000 7fa77439b740 1 -- 172.21.15.90:0/102851965 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55a7be4a80f0 con 0x55a7be870960 2024-04-02T14:43:49.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.686+0000 7fa74be00640 1 -- 172.21.15.90:0/102851965 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa75c046040 con 0x55a7be870960 2024-04-02T14:43:49.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.687+0000 7fa74be00640 1 -- 172.21.15.90:0/102851965 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fa75c04c480 con 0x55a7be870960 2024-04-02T14:43:49.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.687+0000 7fa77439b740 10 client.28129 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:43:49.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.687+0000 7fa77439b740 10 client.28129 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:43:49.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.687+0000 7fa77439b740 20 client.28129 populate_metadata read hostname 'smithi090' 2024-04-02T14:43:49.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.687+0000 7fa77439b740 1 --2- 172.21.15.90:0/102851965 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55a7be889ee0 0x55a7be88c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:49.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.687+0000 7fa77439b740 4 client.28129 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:43:49.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.687+0000 7fa77439b740 1 -- 172.21.15.90:0/102851965 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fa764003e50 con 0x55a7be889ee0 2024-04-02T14:43:49.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.687+0000 7fa76a000640 1 --2- 172.21.15.90:0/102851965 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55a7be889ee0 0x55a7be88c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:49.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.688+0000 7fa76a000640 1 --2- 172.21.15.90:0/102851965 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55a7be889ee0 0x55a7be88c2d0 crc :-1 s=READY pgs=244 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f2c0a13a3e33558d server_cookie=614f80b5cfd9c106 in_seq=0 out_seq=0 2024-04-02T14:43:49.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.688+0000 7fa74be00640 10 client.28129 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:43:49.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.689+0000 7fa74be00640 1 -- 172.21.15.90:0/102851965 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x7fa764003e50 con 0x55a7be889ee0 2024-04-02T14:43:49.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.689+0000 7fa74be00640 10 client.28129 handle_command_reply: tid=0 2024-04-02T14:43:49.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.689+0000 7fa77439b740 1 client.28129 shutdown 2024-04-02T14:43:49.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.690+0000 7fa77439b740 1 -- 172.21.15.90:0/102851965 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a7be870960 msgr2=0x55a7be876f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:49.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.690+0000 7fa77439b740 1 --2- 172.21.15.90:0/102851965 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a7be870960 0x55a7be876f20 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fa75c009230 tx=0x7fa75c02bc10 comp rx=0 tx=0).stop 2024-04-02T14:43:49.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.690+0000 7fa77439b740 1 -- 172.21.15.90:0/102851965 shutdown_connections 2024-04-02T14:43:49.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.690+0000 7fa77439b740 1 --2- 172.21.15.90:0/102851965 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55a7be889ee0 0x55a7be88c2d0 crc :-1 s=READY pgs=244 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:49.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.690+0000 7fa77439b740 1 --2- 172.21.15.90:0/102851965 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a7be879e60 0x55a7be8779a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:49.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.690+0000 7fa77439b740 1 --2- 172.21.15.90:0/102851965 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a7be871280 0x55a7be877460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:49.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.690+0000 7fa77439b740 1 --2- 172.21.15.90:0/102851965 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a7be870960 0x55a7be876f20 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:49.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.690+0000 7fa74be00640 0 client.28129 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:43:49.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.690+0000 7fa77439b740 1 -- 172.21.15.90:0/102851965 >> 172.21.15.90:0/102851965 conn(0x55a7be86b090 msgr2=0x55a7be86cde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:43:49.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.690+0000 7fa77439b740 1 -- 172.21.15.90:0/102851965 shutdown_connections 2024-04-02T14:43:49.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.690+0000 7fa77439b740 1 -- 172.21.15.90:0/102851965 wait complete. 2024-04-02T14:43:49.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.690+0000 7fa77439b740 20 client.28129 trim_cache size 0 max 16384 2024-04-02T14:43:49.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.720+0000 7fa77439b740 1 Processor -- start 2024-04-02T14:43:49.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.720+0000 7fa77439b740 1 -- start start 2024-04-02T14:43:49.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.720+0000 7fa77439b740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a7be870960 0x55a7be885b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:49.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.720+0000 7fa77439b740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a7be871280 0x55a7be8777c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:49.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.721+0000 7fa77439b740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a7be879e60 0x55a7be891ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:49.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.721+0000 7fa77439b740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fa76c1abab0 con 0x55a7be870960 2024-04-02T14:43:49.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.721+0000 7fa77439b740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fa76c1ad5f0 con 0x55a7be871280 2024-04-02T14:43:49.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.721+0000 7fa77439b740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fa76c1ac180 con 0x55a7be879e60 2024-04-02T14:43:49.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.721+0000 7fa76aa00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a7be870960 0x55a7be885b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:49.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.721+0000 7fa76a000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a7be871280 0x55a7be8777c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:49.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.721+0000 7fa76aa00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a7be870960 0x55a7be885b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33936/0 (socket says 172.21.15.90:33936) 2024-04-02T14:43:49.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.721+0000 7fa76aa00640 1 -- 172.21.15.90:0/2999201344 learned_addr learned my addr 172.21.15.90:0/2999201344 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:43:49.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.721+0000 7fa76b400640 1 --2- 172.21.15.90:0/2999201344 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a7be879e60 0x55a7be891ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:49.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.721+0000 7fa76aa00640 1 -- 172.21.15.90:0/2999201344 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a7be879e60 msgr2=0x55a7be891ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:43:49.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.721+0000 7fa76aa00640 1 --2- 172.21.15.90:0/2999201344 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a7be879e60 0x55a7be891ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:49.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.721+0000 7fa76aa00640 1 -- 172.21.15.90:0/2999201344 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a7be871280 msgr2=0x55a7be8777c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:43:49.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.721+0000 7fa76aa00640 1 --2- 172.21.15.90:0/2999201344 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a7be871280 0x55a7be8777c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:49.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.721+0000 7fa76aa00640 1 -- 172.21.15.90:0/2999201344 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a7be87c5c0 con 0x55a7be870960 2024-04-02T14:43:49.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.721+0000 7fa76a000640 1 --2- 172.21.15.90:0/2999201344 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a7be871280 0x55a7be8777c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:43:49.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.722+0000 7fa76aa00640 1 --2- 172.21.15.90:0/2999201344 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a7be870960 0x55a7be885b10 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fa75c04c970 tx=0x7fa75c04cfb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7edbd5cfa9f7087f server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:43:49.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.723+0000 7fa756a00640 1 -- 172.21.15.90:0/2999201344 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa75c066020 con 0x55a7be870960 2024-04-02T14:43:49.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.723+0000 7fa756a00640 1 -- 172.21.15.90:0/2999201344 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa75c02ca90 con 0x55a7be870960 2024-04-02T14:43:49.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.723+0000 7fa756a00640 1 -- 172.21.15.90:0/2999201344 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa75c062780 con 0x55a7be870960 2024-04-02T14:43:49.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.724+0000 7fa77439b740 1 -- 172.21.15.90:0/2999201344 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a7be870960 msgr2=0x55a7be885b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:49.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.724+0000 7fa77439b740 1 --2- 172.21.15.90:0/2999201344 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a7be870960 0x55a7be885b10 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fa75c04c970 tx=0x7fa75c04cfb0 comp rx=0 tx=0).stop 2024-04-02T14:43:49.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.724+0000 7fa77439b740 1 -- 172.21.15.90:0/2999201344 shutdown_connections 2024-04-02T14:43:49.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.724+0000 7fa77439b740 1 --2- 172.21.15.90:0/2999201344 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a7be879e60 0x55a7be891ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:49.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.724+0000 7fa77439b740 1 --2- 172.21.15.90:0/2999201344 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a7be871280 0x55a7be8777c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:49.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.724+0000 7fa77439b740 1 --2- 172.21.15.90:0/2999201344 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a7be870960 0x55a7be885b10 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:49.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.724+0000 7fa77439b740 1 -- 172.21.15.90:0/2999201344 >> 172.21.15.90:0/2999201344 conn(0x55a7be86b090 msgr2=0x55a7be86a310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:43:49.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.725+0000 7fa77439b740 1 -- 172.21.15.90:0/2999201344 shutdown_connections 2024-04-02T14:43:49.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.725+0000 7fa77439b740 1 -- 172.21.15.90:0/2999201344 wait complete. 2024-04-02T14:43:49.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.726+0000 7fa77439b740 1 -- start start 2024-04-02T14:43:49.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.727+0000 7fa77439b740 1 Processor -- start 2024-04-02T14:43:49.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.727+0000 7fa77439b740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a7be870960 0x55a7be873ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:49.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.727+0000 7fa77439b740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a7be871280 0x55a7be874400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:49.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.727+0000 7fa77439b740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a7be879e60 0x55a7be874960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:49.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.728+0000 7fa77439b740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fa76c1acf20 con 0x55a7be871280 2024-04-02T14:43:49.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.728+0000 7fa77439b740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fa76c1ac850 con 0x55a7be879e60 2024-04-02T14:43:49.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.728+0000 7fa77439b740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fa76c1a9af0 con 0x55a7be870960 2024-04-02T14:43:49.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.728+0000 7fa77439b740 10 client.0 _ll_register_callbacks cb 0x7fa764009460 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:43:49.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.728+0000 7fa76a000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a7be871280 0x55a7be874400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:49.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.728+0000 7fa76b400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a7be879e60 0x55a7be874960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:49.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.728+0000 7fa76b400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a7be879e60 0x55a7be874960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:36576/0 (socket says 172.21.15.90:36576) 2024-04-02T14:43:49.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.728+0000 7fa76b400640 1 -- 172.21.15.90:0/3049878899 learned_addr learned my addr 172.21.15.90:0/3049878899 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:43:49.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.729+0000 7fa76b400640 1 -- 172.21.15.90:0/3049878899 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a7be870960 msgr2=0x55a7be873ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:43:49.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.729+0000 7fa76b400640 1 --2- 172.21.15.90:0/3049878899 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a7be870960 0x55a7be873ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:49.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.729+0000 7fa76b400640 1 -- 172.21.15.90:0/3049878899 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a7be871280 msgr2=0x55a7be874400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:43:49.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.729+0000 7fa76b400640 1 --2- 172.21.15.90:0/3049878899 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a7be871280 0x55a7be874400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:49.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.729+0000 7fa76b400640 1 -- 172.21.15.90:0/3049878899 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a7be8ef5f0 con 0x55a7be879e60 2024-04-02T14:43:49.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.731+0000 7fa76b400640 1 --2- 172.21.15.90:0/3049878899 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a7be879e60 0x55a7be874960 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7fa764009f90 tx=0x7fa764004b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b113e1b2dc409992 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:43:49.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.731+0000 7fa74be00640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:43:49.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.733+0000 7fa74be00640 1 -- 172.21.15.90:0/3049878899 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa764026040 con 0x55a7be879e60 2024-04-02T14:43:49.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.734+0000 7fa74be00640 1 -- 172.21.15.90:0/3049878899 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8550+0+0 (secure 0 0 0) 0x7fa764005240 con 0x55a7be879e60 2024-04-02T14:43:49.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.735+0000 7fa77439b740 1 -- 172.21.15.90:0/3049878899 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa75c02ca90 con 0x55a7be879e60 2024-04-02T14:43:49.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.742+0000 7fa74be00640 1 -- 172.21.15.90:0/3049878899 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fa764004e50 con 0x55a7be879e60 2024-04-02T14:43:49.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.742+0000 7fa77439b740 10 client.28301 fetch_fsmap learned FSMap version 27 2024-04-02T14:43:49.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.742+0000 7fa77439b740 1 -- 172.21.15.90:0/3049878899 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55a7be8f0900 con 0x55a7be879e60 2024-04-02T14:43:49.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.743+0000 7fa74be00640 1 -- 172.21.15.90:0/3049878899 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa75c02ca90 con 0x55a7be879e60 2024-04-02T14:43:49.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.744+0000 7fa74be00640 1 -- 172.21.15.90:0/3049878899 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fa764020070 con 0x55a7be879e60 2024-04-02T14:43:49.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.744+0000 7fa77439b740 10 client.28301 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:43:49.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.744+0000 7fa77439b740 10 client.28301 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:43:49.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.744+0000 7fa77439b740 20 client.28301 populate_metadata read hostname 'smithi090' 2024-04-02T14:43:49.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.744+0000 7fa77439b740 1 --2- 172.21.15.90:0/3049878899 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55a7be8f0e40 0x55a7be8f3230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:49.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.744+0000 7fa77439b740 4 client.28301 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-02T14:43:49.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.744+0000 7fa77439b740 1 -- 172.21.15.90:0/3049878899 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55a7be775630 con 0x55a7be8f0e40 2024-04-02T14:43:49.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.746+0000 7fa76aa00640 1 --2- 172.21.15.90:0/3049878899 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55a7be8f0e40 0x55a7be8f3230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:49.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.746+0000 7fa76aa00640 1 --2- 172.21.15.90:0/3049878899 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55a7be8f0e40 0x55a7be8f3230 crc :-1 s=READY pgs=245 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e556176058dc9f20 server_cookie=33eeff92cfded210 in_seq=0 out_seq=0 2024-04-02T14:43:49.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.746+0000 7fa74be00640 10 client.28301 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:43:49.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.747+0000 7fa74be00640 1 -- 172.21.15.90:0/3049878899 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55a7be775630 con 0x55a7be8f0e40 2024-04-02T14:43:49.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.747+0000 7fa74be00640 10 client.28301 handle_command_reply: tid=0 2024-04-02T14:43:49.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.747+0000 7fa77439b740 1 client.28301 shutdown 2024-04-02T14:43:49.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.747+0000 7fa77439b740 1 -- 172.21.15.90:0/3049878899 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a7be879e60 msgr2=0x55a7be874960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:49.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.747+0000 7fa77439b740 1 --2- 172.21.15.90:0/3049878899 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a7be879e60 0x55a7be874960 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7fa764009f90 tx=0x7fa764004b50 comp rx=0 tx=0).stop 2024-04-02T14:43:49.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.748+0000 7fa77439b740 1 -- 172.21.15.90:0/3049878899 shutdown_connections 2024-04-02T14:43:49.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.748+0000 7fa77439b740 1 --2- 172.21.15.90:0/3049878899 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55a7be8f0e40 0x55a7be8f3230 crc :-1 s=READY pgs=245 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:49.745 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.748+0000 7fa77439b740 1 --2- 172.21.15.90:0/3049878899 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a7be879e60 0x55a7be874960 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:49.745 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.748+0000 7fa77439b740 1 --2- 172.21.15.90:0/3049878899 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a7be871280 0x55a7be874400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:49.745 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.748+0000 7fa77439b740 1 --2- 172.21.15.90:0/3049878899 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a7be870960 0x55a7be873ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:49.745 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.748+0000 7fa77439b740 1 -- 172.21.15.90:0/3049878899 >> 172.21.15.90:0/3049878899 conn(0x55a7be86b090 msgr2=0x55a7be86c4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:43:49.745 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.748+0000 7fa74be00640 0 client.28301 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:43:49.745 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.748+0000 7fa77439b740 1 -- 172.21.15.90:0/3049878899 shutdown_connections 2024-04-02T14:43:49.745 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.749+0000 7fa77439b740 1 -- 172.21.15.90:0/3049878899 wait complete. 2024-04-02T14:43:49.747 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-02T14:43:49.747 INFO:teuthology.orchestra.run.smithi090.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-02T14:43:49.747 INFO:teuthology.orchestra.run.smithi090.stdout: "scrubs": { 2024-04-02T14:43:49.747 INFO:teuthology.orchestra.run.smithi090.stdout: "4c95946c-3531-4cd0-993f-dcbcdb2ebc91": { 2024-04-02T14:43:49.747 INFO:teuthology.orchestra.run.smithi090.stdout: "path": "/", 2024-04-02T14:43:49.747 INFO:teuthology.orchestra.run.smithi090.stdout: "tag": "4c95946c-3531-4cd0-993f-dcbcdb2ebc91", 2024-04-02T14:43:49.748 INFO:teuthology.orchestra.run.smithi090.stdout: "options": "recursive,force" 2024-04-02T14:43:49.748 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:43:49.748 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:43:49.748 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-02T14:43:49.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.750+0000 7fa77439b740 20 client.28301 trim_cache size 0 max 16384 2024-04-02T14:43:49.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.751+0000 7fa76be00640 1 -- 172.21.15.90:0/3485341765 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fa7500a3da0 msgr2=0x7fa7500a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:49.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.751+0000 7fa76be00640 1 --2- 172.21.15.90:0/3485341765 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fa7500a3da0 0x7fa7500a6260 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fa75c02c220 tx=0x7fa75c02e040 comp rx=0 tx=0).stop 2024-04-02T14:43:49.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.751+0000 7fa76be00640 1 -- 172.21.15.90:0/3485341765 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa76c10bf50 msgr2=0x7fa76c19abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:49.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.751+0000 7fa76be00640 1 --2- 172.21.15.90:0/3485341765 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa76c10bf50 0x7fa76c19abb0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7fa75800e9f0 tx=0x7fa75800eeb0 comp rx=0 tx=0).stop 2024-04-02T14:43:49.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.751+0000 7fa76be00640 1 -- 172.21.15.90:0/3485341765 shutdown_connections 2024-04-02T14:43:49.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.751+0000 7fa76be00640 1 --2- 172.21.15.90:0/3485341765 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fa7500a3da0 0x7fa7500a6260 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:49.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.751+0000 7fa76be00640 1 --2- 172.21.15.90:0/3485341765 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa76c19ef30 0x7fa76c19f3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:49.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.751+0000 7fa76be00640 1 --2- 172.21.15.90:0/3485341765 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa76c10bf50 0x7fa76c19abb0 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:49.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.751+0000 7fa76be00640 1 --2- 172.21.15.90:0/3485341765 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa76c103450 0x7fa76c19a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:49.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.751+0000 7fa76be00640 1 -- 172.21.15.90:0/3485341765 >> 172.21.15.90:0/3485341765 conn(0x7fa76c0fc350 msgr2=0x7fa76c0fe740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:43:49.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.751+0000 7fa76be00640 1 -- 172.21.15.90:0/3485341765 shutdown_connections 2024-04-02T14:43:49.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:43:49.751+0000 7fa76be00640 1 -- 172.21.15.90:0/3485341765 wait complete. 2024-04-02T14:43:49.783 INFO:tasks.cephfs.filesystem:scrub status for tag:4c95946c-3531-4cd0-993f-dcbcdb2ebc91 - {'path': '/', 'tag': '4c95946c-3531-4cd0-993f-dcbcdb2ebc91', 'options': 'recursive,force'} 2024-04-02T14:43:50.606 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:43:50 smithi092 ceph-mon[36497]: pgmap v275: 97 pgs: 97 active+clean; 4.7 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 250 KiB/s rd, 22 MiB/s wr, 1.43k op/s 2024-04-02T14:43:50.607 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:43:50 smithi092 ceph-mon[36497]: scrub queued for path: / 2024-04-02T14:43:50.607 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:43:50 smithi092 ceph-mon[36497]: scrub summary: idle+waiting paths [/] 2024-04-02T14:43:50.607 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:43:50 smithi092 ceph-mon[36497]: scrub summary: active paths [/] 2024-04-02T14:43:50.607 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:43:50 smithi092 ceph-mon[36497]: scrub summary: idle+waiting paths [/] 2024-04-02T14:43:50.682 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:43:50 smithi177 ceph-mon[36431]: pgmap v275: 97 pgs: 97 active+clean; 4.7 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 250 KiB/s rd, 22 MiB/s wr, 1.43k op/s 2024-04-02T14:43:50.682 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:43:50 smithi177 ceph-mon[36431]: scrub queued for path: / 2024-04-02T14:43:50.682 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:43:50 smithi177 ceph-mon[36431]: scrub summary: idle+waiting paths [/] 2024-04-02T14:43:50.682 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:43:50 smithi177 ceph-mon[36431]: scrub summary: active paths [/] 2024-04-02T14:43:50.683 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:43:50 smithi177 ceph-mon[36431]: scrub summary: idle+waiting paths [/] 2024-04-02T14:43:50.684 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:43:50 smithi090 ceph-mon[31674]: pgmap v275: 97 pgs: 97 active+clean; 4.7 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 250 KiB/s rd, 22 MiB/s wr, 1.43k op/s 2024-04-02T14:43:50.684 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:43:50 smithi090 ceph-mon[31674]: scrub queued for path: / 2024-04-02T14:43:50.684 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:43:50 smithi090 ceph-mon[31674]: scrub summary: idle+waiting paths [/] 2024-04-02T14:43:50.684 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:43:50 smithi090 ceph-mon[31674]: scrub summary: active paths [/] 2024-04-02T14:43:50.684 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:43:50 smithi090 ceph-mon[31674]: scrub summary: idle+waiting paths [/] 2024-04-02T14:43:51.827 INFO:teuthology.orchestra.run.smithi090.stderr:progress: 500.0 s, 1708.9 tps, lat 57.666 ms stddev 69.165 2024-04-02T14:43:52.551 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.554+0000 7f719ba00640 1 -- 172.21.15.90:0/2125880837 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7194112ac0 msgr2=0x7f7194114eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:52.552 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.554+0000 7f719ba00640 1 --2- 172.21.15.90:0/2125880837 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7194112ac0 0x7f7194114eb0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f718c009ca0 tx=0x7f718c02c180 comp rx=0 tx=0).stop 2024-04-02T14:43:52.552 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.555+0000 7f719ba00640 1 -- 172.21.15.90:0/2125880837 shutdown_connections 2024-04-02T14:43:52.552 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.555+0000 7f719ba00640 1 --2- 172.21.15.90:0/2125880837 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7194112ac0 0x7f7194114eb0 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:52.552 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.555+0000 7f719ba00640 1 --2- 172.21.15.90:0/2125880837 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f71940747c0 0x7f7194112580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:52.552 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.555+0000 7f719ba00640 1 --2- 172.21.15.90:0/2125880837 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7194073ea0 0x7f7194074280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:52.552 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.555+0000 7f719ba00640 1 -- 172.21.15.90:0/2125880837 >> 172.21.15.90:0/2125880837 conn(0x7f71940fd580 msgr2=0x7f71940ff9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:43:52.552 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.555+0000 7f719ba00640 1 -- 172.21.15.90:0/2125880837 shutdown_connections 2024-04-02T14:43:52.552 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.555+0000 7f719ba00640 1 -- 172.21.15.90:0/2125880837 wait complete. 2024-04-02T14:43:52.553 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.556+0000 7f719ba00640 1 Processor -- start 2024-04-02T14:43:52.553 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.556+0000 7f719ba00640 1 -- start start 2024-04-02T14:43:52.553 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.556+0000 7f719ba00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7194073ea0 0x7f71941a4630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:52.553 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.556+0000 7f719ba00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f71940747c0 0x7f71941a4b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:52.553 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.556+0000 7f719ba00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7194112ac0 0x7f719419e740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:52.553 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.556+0000 7f719ba00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f7194118460 con 0x7f7194073ea0 2024-04-02T14:43:52.553 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.556+0000 7f719ba00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f71941182e0 con 0x7f7194112ac0 2024-04-02T14:43:52.553 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.556+0000 7f719ba00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f71941185e0 con 0x7f71940747c0 2024-04-02T14:43:52.554 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.557+0000 7f7199a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7194112ac0 0x7f719419e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:52.554 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.557+0000 7f7193e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f71940747c0 0x7f71941a4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:52.554 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.557+0000 7f7193e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f71940747c0 0x7f71941a4b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:52872/0 (socket says 172.21.15.90:52872) 2024-04-02T14:43:52.554 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.557+0000 7f7193e00640 1 -- 172.21.15.90:0/597905137 learned_addr learned my addr 172.21.15.90:0/597905137 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:43:52.555 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.557+0000 7f7199a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7194112ac0 0x7f719419e740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:36592/0 (socket says 172.21.15.90:36592) 2024-04-02T14:43:52.555 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.557+0000 7f7199000640 1 --2- 172.21.15.90:0/597905137 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7194073ea0 0x7f71941a4630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:52.555 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.557+0000 7f7199a00640 1 -- 172.21.15.90:0/597905137 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f71940747c0 msgr2=0x7f71941a4b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:52.555 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.557+0000 7f7199a00640 1 --2- 172.21.15.90:0/597905137 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f71940747c0 0x7f71941a4b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:52.555 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.557+0000 7f7199a00640 1 -- 172.21.15.90:0/597905137 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7194073ea0 msgr2=0x7f71941a4630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:52.555 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.557+0000 7f7199a00640 1 --2- 172.21.15.90:0/597905137 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7194073ea0 0x7f71941a4630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:52.555 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.557+0000 7f7199a00640 1 -- 172.21.15.90:0/597905137 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f719419f000 con 0x7f7194112ac0 2024-04-02T14:43:52.555 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.557+0000 7f7199a00640 1 --2- 172.21.15.90:0/597905137 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7194112ac0 0x7f719419e740 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f718c037040 tx=0x7f718c035e50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:43:52.555 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.558+0000 7f7191600640 1 -- 172.21.15.90:0/597905137 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f718c039040 con 0x7f7194112ac0 2024-04-02T14:43:52.555 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.558+0000 7f719ba00640 1 -- 172.21.15.90:0/597905137 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f719419f290 con 0x7f7194112ac0 2024-04-02T14:43:52.555 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.558+0000 7f7191600640 1 -- 172.21.15.90:0/597905137 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f718c044070 con 0x7f7194112ac0 2024-04-02T14:43:52.555 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.558+0000 7f7191600640 1 -- 172.21.15.90:0/597905137 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f718c0328d0 con 0x7f7194112ac0 2024-04-02T14:43:52.555 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.558+0000 7f719ba00640 1 -- 172.21.15.90:0/597905137 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f71941ab3d0 con 0x7f7194112ac0 2024-04-02T14:43:52.558 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.561+0000 7f719e7c0740 1 Processor -- start 2024-04-02T14:43:52.559 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.563+0000 7f7191600640 1 -- 172.21.15.90:0/597905137 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f718c032a70 con 0x7f7194112ac0 2024-04-02T14:43:52.560 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.563+0000 7f719e7c0740 1 -- start start 2024-04-02T14:43:52.561 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.564+0000 7f719e7c0740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562081260eb0 0x562081261290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:52.561 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.564+0000 7f719e7c0740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5620812617d0 0x562081269e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:52.561 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.564+0000 7f719e7c0740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56208126a3b0 0x56208126c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:52.561 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.564+0000 7f719e7c0740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5620811c5d70 con 0x562081260eb0 2024-04-02T14:43:52.561 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.564+0000 7f719e7c0740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x56208116e2a0 con 0x56208126a3b0 2024-04-02T14:43:52.561 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.564+0000 7f719e7c0740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5620811c1d20 con 0x5620812617d0 2024-04-02T14:43:52.562 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.565+0000 7f7199000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5620812617d0 0x562081269e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:52.562 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.565+0000 7f7199000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5620812617d0 0x562081269e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:52882/0 (socket says 172.21.15.90:52882) 2024-04-02T14:43:52.562 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.565+0000 7f7199000640 1 -- 172.21.15.90:0/2531583090 learned_addr learned my addr 172.21.15.90:0/2531583090 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:43:52.562 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.565+0000 7f7193e00640 1 --2- 172.21.15.90:0/2531583090 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56208126a3b0 0x56208126c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:52.562 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.565+0000 7f7193e00640 1 --2- 172.21.15.90:0/2531583090 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562081260eb0 0x562081261290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:52.562 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.565+0000 7f7199000640 1 -- 172.21.15.90:0/2531583090 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56208126a3b0 msgr2=0x56208126c7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:43:52.562 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.565+0000 7f7199000640 1 --2- 172.21.15.90:0/2531583090 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56208126a3b0 0x56208126c7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:52.562 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.565+0000 7f7199000640 1 -- 172.21.15.90:0/2531583090 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562081260eb0 msgr2=0x562081261290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:43:52.563 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.565+0000 7f7199000640 1 --2- 172.21.15.90:0/2531583090 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562081260eb0 0x562081261290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:52.563 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.565+0000 7f7191600640 1 --2- 172.21.15.90:0/597905137 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f717c0a3e40 0x7f717c0a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:52.563 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.565+0000 7f7199000640 1 -- 172.21.15.90:0/2531583090 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562081182840 con 0x5620812617d0 2024-04-02T14:43:52.563 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.565+0000 7f7193e00640 1 --2- 172.21.15.90:0/2531583090 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562081260eb0 0x562081261290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:43:52.563 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.566+0000 7f7191600640 1 -- 172.21.15.90:0/597905137 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8550+0+0 (secure 0 0 0) 0x7f718c0f2b60 con 0x7f7194112ac0 2024-04-02T14:43:52.563 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.566+0000 7f7193e00640 1 --2- 172.21.15.90:0/597905137 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f717c0a3e40 0x7f717c0a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:52.564 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.566+0000 7f7199000640 1 --2- 172.21.15.90:0/2531583090 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5620812617d0 0x562081269e70 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f718800bd10 tx=0x7f71880107a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=96aaaaa962ac7796 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:43:52.564 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.566+0000 7f7186000640 1 -- 172.21.15.90:0/2531583090 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7188004030 con 0x5620812617d0 2024-04-02T14:43:52.564 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.566+0000 7f7186000640 1 -- 172.21.15.90:0/2531583090 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f71880041d0 con 0x5620812617d0 2024-04-02T14:43:52.564 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.566+0000 7f7186000640 1 -- 172.21.15.90:0/2531583090 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7188011840 con 0x5620812617d0 2024-04-02T14:43:52.565 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.567+0000 7f7193e00640 1 --2- 172.21.15.90:0/597905137 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f717c0a3e40 0x7f717c0a6300 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x56208125c770 tx=0x7f7180015040 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:43:52.565 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.567+0000 7f719e7c0740 1 -- 172.21.15.90:0/2531583090 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5620812617d0 msgr2=0x562081269e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:52.565 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.567+0000 7f719e7c0740 1 --2- 172.21.15.90:0/2531583090 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5620812617d0 0x562081269e70 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f718800bd10 tx=0x7f71880107a0 comp rx=0 tx=0).stop 2024-04-02T14:43:52.565 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.568+0000 7f719e7c0740 1 -- 172.21.15.90:0/2531583090 shutdown_connections 2024-04-02T14:43:52.565 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.568+0000 7f719e7c0740 1 --2- 172.21.15.90:0/2531583090 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56208126a3b0 0x56208126c7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:52.565 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.568+0000 7f719e7c0740 1 --2- 172.21.15.90:0/2531583090 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5620812617d0 0x562081269e70 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:52.565 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.568+0000 7f719e7c0740 1 --2- 172.21.15.90:0/2531583090 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562081260eb0 0x562081261290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:52.565 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.568+0000 7f719e7c0740 1 -- 172.21.15.90:0/2531583090 >> 172.21.15.90:0/2531583090 conn(0x56208125b5e0 msgr2=0x56208125b9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:43:52.565 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.568+0000 7f719e7c0740 1 -- 172.21.15.90:0/2531583090 shutdown_connections 2024-04-02T14:43:52.566 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.568+0000 7f719e7c0740 1 -- 172.21.15.90:0/2531583090 wait complete. 2024-04-02T14:43:52.566 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.568+0000 7f719e7c0740 1 -- start start 2024-04-02T14:43:52.566 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.568+0000 7f719e7c0740 1 Processor -- start 2024-04-02T14:43:52.568 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.569+0000 7f719e7c0740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562081260eb0 0x562081267410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:52.569 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.569+0000 7f719e7c0740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5620812617d0 0x562081267950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:52.569 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.569+0000 7f719e7c0740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56208126a3b0 0x562081267e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:52.569 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.569+0000 7f719e7c0740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x56208126d1a0 con 0x562081260eb0 2024-04-02T14:43:52.569 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.569+0000 7f719e7c0740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x562081181660 con 0x56208126a3b0 2024-04-02T14:43:52.569 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.569+0000 7f719e7c0740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x56208126d320 con 0x5620812617d0 2024-04-02T14:43:52.569 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.569+0000 7f719e7c0740 10 client.0 _ll_register_callbacks cb 0x562081181050 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:43:52.569 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.569+0000 7f7193e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5620812617d0 0x562081267950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:52.569 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.569+0000 7f7193e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5620812617d0 0x562081267950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:52888/0 (socket says 172.21.15.90:52888) 2024-04-02T14:43:52.569 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.569+0000 7f7193e00640 1 -- 172.21.15.90:0/2396874874 learned_addr learned my addr 172.21.15.90:0/2396874874 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:43:52.569 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.569+0000 7f7199000640 1 --2- 172.21.15.90:0/2396874874 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562081260eb0 0x562081267410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:52.569 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.569+0000 7f7199a00640 1 --2- 172.21.15.90:0/2396874874 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56208126a3b0 0x562081267e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:52.569 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.570+0000 7f7193e00640 1 -- 172.21.15.90:0/2396874874 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56208126a3b0 msgr2=0x562081267e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:43:52.569 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.570+0000 7f7193e00640 1 --2- 172.21.15.90:0/2396874874 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56208126a3b0 0x562081267e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:52.569 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.570+0000 7f7193e00640 1 -- 172.21.15.90:0/2396874874 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562081260eb0 msgr2=0x562081267410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:43:52.570 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.570+0000 7f7193e00640 1 --2- 172.21.15.90:0/2396874874 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562081260eb0 0x562081267410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:52.570 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.570+0000 7f7193e00640 1 -- 172.21.15.90:0/2396874874 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562081190e80 con 0x5620812617d0 2024-04-02T14:43:52.570 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.570+0000 7f7193e00640 1 --2- 172.21.15.90:0/2396874874 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5620812617d0 0x562081267950 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f718001f9a0 tx=0x7f71800046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dc649f05b2797c4b server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:43:52.570 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.570+0000 7f717aa00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:43:52.570 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.570+0000 7f717aa00640 1 -- 172.21.15.90:0/2396874874 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f718001f320 con 0x5620812617d0 2024-04-02T14:43:52.570 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.570+0000 7f717aa00640 1 -- 172.21.15.90:0/2396874874 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8550+0+0 (secure 0 0 0) 0x7f7180011720 con 0x5620812617d0 2024-04-02T14:43:52.570 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.570+0000 7f719e7c0740 1 -- 172.21.15.90:0/2396874874 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f71880041d0 con 0x5620812617d0 2024-04-02T14:43:52.570 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.570+0000 7f717aa00640 1 -- 172.21.15.90:0/2396874874 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f718002f050 con 0x5620812617d0 2024-04-02T14:43:52.570 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.570+0000 7f719e7c0740 10 client.28144 fetch_fsmap learned FSMap version 27 2024-04-02T14:43:52.570 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.570+0000 7f719e7c0740 1 -- 172.21.15.90:0/2396874874 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x562080eda210 con 0x5620812617d0 2024-04-02T14:43:52.570 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.571+0000 7f717aa00640 1 -- 172.21.15.90:0/2396874874 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f71880041d0 con 0x5620812617d0 2024-04-02T14:43:52.570 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.571+0000 7f717aa00640 1 -- 172.21.15.90:0/2396874874 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f718000f040 con 0x5620812617d0 2024-04-02T14:43:52.570 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.571+0000 7f719e7c0740 10 client.28144 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:43:52.571 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.571+0000 7f719e7c0740 10 client.28144 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-02T14:43:52.571 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.571+0000 7f719e7c0740 20 client.28144 populate_metadata read hostname 'smithi090' 2024-04-02T14:43:52.571 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.571+0000 7f719e7c0740 1 --2- 172.21.15.90:0/2396874874 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x56208127a710 0x56208127cb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:52.571 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.571+0000 7f719e7c0740 4 client.28144 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:43:52.571 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.571+0000 7f719e7c0740 1 -- 172.21.15.90:0/2396874874 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562081175cd0 con 0x56208127a710 2024-04-02T14:43:52.571 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.571+0000 7f7199000640 1 --2- 172.21.15.90:0/2396874874 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x56208127a710 0x56208127cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:52.571 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.572+0000 7f7199000640 1 --2- 172.21.15.90:0/2396874874 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x56208127a710 0x56208127cb80 crc :-1 s=READY pgs=246 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d7b80ffd8b66c58a server_cookie=f0fb8e06626a37f in_seq=0 out_seq=0 2024-04-02T14:43:52.571 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.572+0000 7f717aa00640 10 client.28144 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:43:52.572 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.573+0000 7f717aa00640 1 -- 172.21.15.90:0/2396874874 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x562081175cd0 con 0x56208127a710 2024-04-02T14:43:52.572 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.573+0000 7f717aa00640 10 client.28144 handle_command_reply: tid=0 2024-04-02T14:43:52.572 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.573+0000 7f719e7c0740 1 client.28144 shutdown 2024-04-02T14:43:52.572 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.574+0000 7f719e7c0740 1 -- 172.21.15.90:0/2396874874 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5620812617d0 msgr2=0x562081267950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:52.572 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.574+0000 7f719e7c0740 1 --2- 172.21.15.90:0/2396874874 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5620812617d0 0x562081267950 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f718001f9a0 tx=0x7f71800046c0 comp rx=0 tx=0).stop 2024-04-02T14:43:52.572 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.574+0000 7f719e7c0740 1 -- 172.21.15.90:0/2396874874 shutdown_connections 2024-04-02T14:43:52.572 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.574+0000 7f719e7c0740 1 --2- 172.21.15.90:0/2396874874 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x56208127a710 0x56208127cb80 crc :-1 s=READY pgs=246 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:52.573 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.574+0000 7f719e7c0740 1 --2- 172.21.15.90:0/2396874874 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56208126a3b0 0x562081267e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:52.573 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.574+0000 7f719e7c0740 1 --2- 172.21.15.90:0/2396874874 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5620812617d0 0x562081267950 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:52.573 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.574+0000 7f717aa00640 0 client.28144 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:43:52.573 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.574+0000 7f719e7c0740 1 --2- 172.21.15.90:0/2396874874 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562081260eb0 0x562081267410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:52.573 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.574+0000 7f719e7c0740 1 -- 172.21.15.90:0/2396874874 >> 172.21.15.90:0/2396874874 conn(0x56208125b5e0 msgr2=0x56208125d330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:43:52.573 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.575+0000 7f719e7c0740 1 -- 172.21.15.90:0/2396874874 shutdown_connections 2024-04-02T14:43:52.573 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.575+0000 7f719e7c0740 1 -- 172.21.15.90:0/2396874874 wait complete. 2024-04-02T14:43:52.573 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.575+0000 7f719e7c0740 20 client.28144 trim_cache size 0 max 16384 2024-04-02T14:43:52.606 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:43:52 smithi092 ceph-mon[36497]: pgmap v276: 97 pgs: 97 active+clean; 4.7 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 411 KiB/s rd, 12 MiB/s wr, 869 op/s 2024-04-02T14:43:52.606 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:43:52 smithi092 ceph-mon[36497]: scrub summary: idle 2024-04-02T14:43:52.611 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.614+0000 7f719e7c0740 1 Processor -- start 2024-04-02T14:43:52.611 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.614+0000 7f719e7c0740 1 -- start start 2024-04-02T14:43:52.611 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.615+0000 7f719e7c0740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562081260eb0 0x562081269b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:52.611 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.615+0000 7f719e7c0740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5620812617d0 0x562081276090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:52.611 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.615+0000 7f719e7c0740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56208126a3b0 0x562081282490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:52.612 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.615+0000 7f719e7c0740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f71941b3db0 con 0x562081260eb0 2024-04-02T14:43:52.612 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.615+0000 7f719e7c0740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f71941b58f0 con 0x56208126a3b0 2024-04-02T14:43:52.612 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.615+0000 7f719e7c0740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f71941b4480 con 0x5620812617d0 2024-04-02T14:43:52.612 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.615+0000 7f7193e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5620812617d0 0x562081276090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:52.612 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.615+0000 7f7193e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5620812617d0 0x562081276090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:52900/0 (socket says 172.21.15.90:52900) 2024-04-02T14:43:52.612 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.615+0000 7f7193e00640 1 -- 172.21.15.90:0/1828528990 learned_addr learned my addr 172.21.15.90:0/1828528990 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:43:52.613 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.616+0000 7f7199000640 1 --2- 172.21.15.90:0/1828528990 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562081260eb0 0x562081269b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:52.613 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.616+0000 7f7193e00640 1 -- 172.21.15.90:0/1828528990 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56208126a3b0 msgr2=0x562081282490 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-02T14:43:52.613 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.616+0000 7f7193e00640 1 --2- 172.21.15.90:0/1828528990 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56208126a3b0 0x562081282490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:52.613 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.616+0000 7f7193e00640 1 -- 172.21.15.90:0/1828528990 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562081260eb0 msgr2=0x562081269b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:43:52.613 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.616+0000 7f7193e00640 1 --2- 172.21.15.90:0/1828528990 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562081260eb0 0x562081269b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:52.613 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.616+0000 7f7193e00640 1 -- 172.21.15.90:0/1828528990 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562080fa6620 con 0x5620812617d0 2024-04-02T14:43:52.613 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.616+0000 7f7193e00640 1 --2- 172.21.15.90:0/1828528990 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5620812617d0 0x562081276090 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f718001cd10 tx=0x7f7180005740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fc8f6ec667296e14 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:43:52.613 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.616+0000 7f7185600640 1 -- 172.21.15.90:0/1828528990 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f718001f320 con 0x5620812617d0 2024-04-02T14:43:52.614 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.617+0000 7f7185600640 1 -- 172.21.15.90:0/1828528990 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7180038070 con 0x5620812617d0 2024-04-02T14:43:52.614 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.617+0000 7f7185600640 1 -- 172.21.15.90:0/1828528990 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7180011d20 con 0x5620812617d0 2024-04-02T14:43:52.614 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.618+0000 7f719e7c0740 1 -- 172.21.15.90:0/1828528990 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5620812617d0 msgr2=0x562081276090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:52.614 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.618+0000 7f719e7c0740 1 --2- 172.21.15.90:0/1828528990 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5620812617d0 0x562081276090 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f718001cd10 tx=0x7f7180005740 comp rx=0 tx=0).stop 2024-04-02T14:43:52.615 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.618+0000 7f719e7c0740 1 -- 172.21.15.90:0/1828528990 shutdown_connections 2024-04-02T14:43:52.615 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.618+0000 7f719e7c0740 1 --2- 172.21.15.90:0/1828528990 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56208126a3b0 0x562081282490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:52.615 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.618+0000 7f719e7c0740 1 --2- 172.21.15.90:0/1828528990 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5620812617d0 0x562081276090 secure :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f718001cd10 tx=0x7f7180005740 comp rx=0 tx=0).stop 2024-04-02T14:43:52.615 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.618+0000 7f719e7c0740 1 --2- 172.21.15.90:0/1828528990 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562081260eb0 0x562081269b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:52.615 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.618+0000 7f719e7c0740 1 -- 172.21.15.90:0/1828528990 >> 172.21.15.90:0/1828528990 conn(0x56208125b5e0 msgr2=0x56208125b9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:43:52.616 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.618+0000 7f719e7c0740 1 -- 172.21.15.90:0/1828528990 shutdown_connections 2024-04-02T14:43:52.616 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.618+0000 7f719e7c0740 1 -- 172.21.15.90:0/1828528990 wait complete. 2024-04-02T14:43:52.616 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.619+0000 7f719e7c0740 1 -- start start 2024-04-02T14:43:52.616 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.619+0000 7f719e7c0740 1 Processor -- start 2024-04-02T14:43:52.617 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.620+0000 7f719e7c0740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x562081260eb0 0x56208126e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:52.617 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.620+0000 7f719e7c0740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56208126a3b0 0x56208126ec20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:52.617 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.620+0000 7f719e7c0740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56208127a710 0x5620812dfd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:52.617 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.620+0000 7f719e7c0740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f71941b5220 con 0x56208126a3b0 2024-04-02T14:43:52.617 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.620+0000 7f719e7c0740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f71941b4b50 con 0x562081260eb0 2024-04-02T14:43:52.617 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.620+0000 7f719e7c0740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f71941b1db0 con 0x56208127a710 2024-04-02T14:43:52.617 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.620+0000 7f719e7c0740 10 client.0 _ll_register_callbacks cb 0x7f718001f620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:43:52.618 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.620+0000 7f7199000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x562081260eb0 0x56208126e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:52.618 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.621+0000 7f7199000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x562081260eb0 0x56208126e6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:36628/0 (socket says 172.21.15.90:36628) 2024-04-02T14:43:52.618 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.621+0000 7f7199000640 1 -- 172.21.15.90:0/162898391 learned_addr learned my addr 172.21.15.90:0/162898391 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:43:52.618 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.621+0000 7f7199a00640 1 --2- 172.21.15.90:0/162898391 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56208127a710 0x5620812dfd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:52.618 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.621+0000 7f7199a00640 1 -- 172.21.15.90:0/162898391 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x562081260eb0 msgr2=0x56208126e6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:43:52.618 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.621+0000 7f7199a00640 1 --2- 172.21.15.90:0/162898391 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x562081260eb0 0x56208126e6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:52.618 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.621+0000 7f7199a00640 1 -- 172.21.15.90:0/162898391 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56208126a3b0 msgr2=0x56208126ec20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-02T14:43:52.618 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.621+0000 7f7193e00640 1 --2- 172.21.15.90:0/162898391 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56208126a3b0 0x56208126ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:52.619 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.621+0000 7f7199a00640 1 --2- 172.21.15.90:0/162898391 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56208126a3b0 0x56208126ec20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:52.620 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.621+0000 7f7199a00640 1 -- 172.21.15.90:0/162898391 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5620812f1300 con 0x56208127a710 2024-04-02T14:43:52.620 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.622+0000 7f7199a00640 1 --2- 172.21.15.90:0/162898391 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56208127a710 0x5620812dfd00 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f718c0f0310 tx=0x7f718c0be790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ab8edae6b151f74b server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:43:52.620 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.622+0000 7f717aa00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:43:52.620 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.623+0000 7f717aa00640 1 -- 172.21.15.90:0/162898391 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f718c0beb90 con 0x56208127a710 2024-04-02T14:43:52.620 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.623+0000 7f717aa00640 1 -- 172.21.15.90:0/162898391 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8550+0+0 (secure 0 0 0) 0x7f718c0df0b0 con 0x56208127a710 2024-04-02T14:43:52.620 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.623+0000 7f719e7c0740 1 -- 172.21.15.90:0/162898391 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7180038070 con 0x56208127a710 2024-04-02T14:43:52.621 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.624+0000 7f717aa00640 1 -- 172.21.15.90:0/162898391 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f718c0c2f40 con 0x56208127a710 2024-04-02T14:43:52.622 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.625+0000 7f719e7c0740 10 client.28150 fetch_fsmap learned FSMap version 27 2024-04-02T14:43:52.622 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.625+0000 7f719e7c0740 1 -- 172.21.15.90:0/162898391 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5620812f2650 con 0x56208127a710 2024-04-02T14:43:52.623 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.626+0000 7f717aa00640 1 -- 172.21.15.90:0/162898391 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7180038070 con 0x56208127a710 2024-04-02T14:43:52.623 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.626+0000 7f717aa00640 1 -- 172.21.15.90:0/162898391 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f718c03c030 con 0x56208127a710 2024-04-02T14:43:52.623 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.626+0000 7f719e7c0740 10 client.28150 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:43:52.623 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.626+0000 7f719e7c0740 10 client.28150 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-02T14:43:52.623 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.626+0000 7f719e7c0740 20 client.28150 populate_metadata read hostname 'smithi090' 2024-04-02T14:43:52.623 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.626+0000 7f719e7c0740 1 --2- 172.21.15.90:0/162898391 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5620812f2da0 0x5620812f5190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:43:52.624 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.626+0000 7f719e7c0740 4 client.28150 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-02T14:43:52.624 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.626+0000 7f719e7c0740 1 -- 172.21.15.90:0/162898391 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5620811d32b0 con 0x5620812f2da0 2024-04-02T14:43:52.627 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.630+0000 7f7199000640 1 --2- 172.21.15.90:0/162898391 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5620812f2da0 0x5620812f5190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:43:52.627 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.631+0000 7f7199000640 1 --2- 172.21.15.90:0/162898391 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5620812f2da0 0x5620812f5190 crc :-1 s=READY pgs=247 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ca90588eeb0e269b server_cookie=a8c34a2db76b73f3 in_seq=0 out_seq=0 2024-04-02T14:43:52.627 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.631+0000 7f717aa00640 10 client.28150 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:43:52.682 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:43:52 smithi177 ceph-mon[36431]: pgmap v276: 97 pgs: 97 active+clean; 4.7 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 411 KiB/s rd, 12 MiB/s wr, 869 op/s 2024-04-02T14:43:52.683 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:43:52 smithi177 ceph-mon[36431]: scrub summary: idle 2024-04-02T14:43:52.684 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:43:52 smithi090 ceph-mon[31674]: pgmap v276: 97 pgs: 97 active+clean; 4.7 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 411 KiB/s rd, 12 MiB/s wr, 869 op/s 2024-04-02T14:43:52.684 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:43:52 smithi090 ceph-mon[31674]: scrub summary: idle 2024-04-02T14:43:52.691 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.693+0000 7f717aa00640 1 -- 172.21.15.90:0/162898391 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5620811d32b0 con 0x5620812f2da0 2024-04-02T14:43:52.691 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.693+0000 7f717aa00640 10 client.28150 handle_command_reply: tid=0 2024-04-02T14:43:52.692 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.693+0000 7f719e7c0740 1 client.28150 shutdown 2024-04-02T14:43:52.692 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.694+0000 7f719e7c0740 1 -- 172.21.15.90:0/162898391 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56208127a710 msgr2=0x5620812dfd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:52.692 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.694+0000 7f719e7c0740 1 --2- 172.21.15.90:0/162898391 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56208127a710 0x5620812dfd00 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f718c0f0310 tx=0x7f718c0be790 comp rx=0 tx=0).stop 2024-04-02T14:43:52.692 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.694+0000 7f719e7c0740 1 -- 172.21.15.90:0/162898391 shutdown_connections 2024-04-02T14:43:52.692 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.694+0000 7f719e7c0740 1 --2- 172.21.15.90:0/162898391 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5620812f2da0 0x5620812f5190 crc :-1 s=READY pgs=247 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:52.692 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.694+0000 7f719e7c0740 1 --2- 172.21.15.90:0/162898391 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56208127a710 0x5620812dfd00 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:52.692 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.694+0000 7f719e7c0740 1 --2- 172.21.15.90:0/162898391 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56208126a3b0 0x56208126ec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:52.692 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.694+0000 7f719e7c0740 1 --2- 172.21.15.90:0/162898391 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x562081260eb0 0x56208126e6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:52.693 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.694+0000 7f719e7c0740 1 -- 172.21.15.90:0/162898391 >> 172.21.15.90:0/162898391 conn(0x56208125b5e0 msgr2=0x562081259db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:43:52.693 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.694+0000 7f717aa00640 0 client.28150 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:43:52.693 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.694+0000 7f719e7c0740 1 -- 172.21.15.90:0/162898391 shutdown_connections 2024-04-02T14:43:52.693 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.694+0000 7f719e7c0740 1 -- 172.21.15.90:0/162898391 wait complete. 2024-04-02T14:43:52.693 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.695+0000 7f719e7c0740 20 client.28150 trim_cache size 0 max 16384 2024-04-02T14:43:52.694 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:{ 2024-04-02T14:43:52.694 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "message": "", 2024-04-02T14:43:52.694 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "return_code": 0 2024-04-02T14:43:52.694 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:} 2024-04-02T14:43:52.695 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.697+0000 7f719ba00640 1 -- 172.21.15.90:0/597905137 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f717c0a3e40 msgr2=0x7f717c0a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:52.695 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.697+0000 7f719ba00640 1 --2- 172.21.15.90:0/597905137 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f717c0a3e40 0x7f717c0a6300 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x56208125c770 tx=0x7f7180015040 comp rx=0 tx=0).stop 2024-04-02T14:43:52.696 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.697+0000 7f719ba00640 1 -- 172.21.15.90:0/597905137 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7194112ac0 msgr2=0x7f719419e740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:43:52.696 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.697+0000 7f719ba00640 1 --2- 172.21.15.90:0/597905137 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7194112ac0 0x7f719419e740 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f718c037040 tx=0x7f718c035e50 comp rx=0 tx=0).stop 2024-04-02T14:43:52.696 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.697+0000 7f719ba00640 1 -- 172.21.15.90:0/597905137 shutdown_connections 2024-04-02T14:43:52.696 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.697+0000 7f719ba00640 1 --2- 172.21.15.90:0/597905137 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f717c0a3e40 0x7f717c0a6300 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:52.696 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.697+0000 7f719ba00640 1 --2- 172.21.15.90:0/597905137 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7194112ac0 0x7f719419e740 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:52.696 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.697+0000 7f719ba00640 1 --2- 172.21.15.90:0/597905137 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f71940747c0 0x7f71941a4b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:52.697 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.697+0000 7f719ba00640 1 --2- 172.21.15.90:0/597905137 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7194073ea0 0x7f71941a4630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:43:52.697 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.697+0000 7f719ba00640 1 -- 172.21.15.90:0/597905137 >> 172.21.15.90:0/597905137 conn(0x7f71940fd580 msgr2=0x7f71940fec70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:43:52.697 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.698+0000 7f719ba00640 1 -- 172.21.15.90:0/597905137 shutdown_connections 2024-04-02T14:43:52.697 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:43:52.698+0000 7f719ba00640 1 -- 172.21.15.90:0/597905137 wait complete. 2024-04-02T14:43:54.607 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:43:54 smithi092 ceph-mon[36497]: pgmap v277: 97 pgs: 97 active+clean; 4.7 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 295 KiB/s rd, 7.5 MiB/s wr, 555 op/s 2024-04-02T14:43:54.682 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:43:54 smithi177 ceph-mon[36431]: pgmap v277: 97 pgs: 97 active+clean; 4.7 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 295 KiB/s rd, 7.5 MiB/s wr, 555 op/s 2024-04-02T14:43:54.684 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:43:54 smithi090 ceph-mon[31674]: pgmap v277: 97 pgs: 97 active+clean; 4.7 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 295 KiB/s rd, 7.5 MiB/s wr, 555 op/s 2024-04-02T14:43:55.606 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:43:55 smithi092 ceph-mon[36497]: pgmap v278: 97 pgs: 97 active+clean; 4.7 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 441 KiB/s rd, 12 MiB/s wr, 1.03k op/s 2024-04-02T14:43:55.682 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:43:55 smithi177 ceph-mon[36431]: pgmap v278: 97 pgs: 97 active+clean; 4.7 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 441 KiB/s rd, 12 MiB/s wr, 1.03k op/s 2024-04-02T14:43:55.684 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:43:55 smithi090 ceph-mon[31674]: pgmap v278: 97 pgs: 97 active+clean; 4.7 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 441 KiB/s rd, 12 MiB/s wr, 1.03k op/s 2024-04-02T14:43:56.827 INFO:teuthology.orchestra.run.smithi090.stderr:progress: 505.0 s, 1679.0 tps, lat 60.203 ms stddev 74.681 2024-04-02T14:43:57.856 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:43:57 smithi092 ceph-mon[36497]: pgmap v279: 97 pgs: 97 active+clean; 4.7 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 419 KiB/s rd, 7.9 MiB/s wr, 744 op/s 2024-04-02T14:43:57.932 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:43:57 smithi177 ceph-mon[36431]: pgmap v279: 97 pgs: 97 active+clean; 4.7 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 419 KiB/s rd, 7.9 MiB/s wr, 744 op/s 2024-04-02T14:43:57.935 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:43:57 smithi090 ceph-mon[31674]: pgmap v279: 97 pgs: 97 active+clean; 4.7 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 419 KiB/s rd, 7.9 MiB/s wr, 744 op/s 2024-04-02T14:43:59.856 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:43:59 smithi092 ceph-mon[36497]: pgmap v280: 97 pgs: 97 active+clean; 4.7 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 428 KiB/s rd, 9.6 MiB/s wr, 897 op/s 2024-04-02T14:43:59.933 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:43:59 smithi177 ceph-mon[36431]: pgmap v280: 97 pgs: 97 active+clean; 4.7 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 428 KiB/s rd, 9.6 MiB/s wr, 897 op/s 2024-04-02T14:43:59.934 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:43:59 smithi090 ceph-mon[31674]: pgmap v280: 97 pgs: 97 active+clean; 4.7 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 428 KiB/s rd, 9.6 MiB/s wr, 897 op/s 2024-04-02T14:44:01.827 INFO:teuthology.orchestra.run.smithi090.stderr:progress: 510.0 s, 1452.8 tps, lat 66.753 ms stddev 93.744 2024-04-02T14:44:01.856 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:44:01 smithi092 ceph-mon[36497]: pgmap v281: 97 pgs: 97 active+clean; 4.7 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 363 KiB/s rd, 9.1 MiB/s wr, 849 op/s 2024-04-02T14:44:01.932 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:44:01 smithi177 ceph-mon[36431]: pgmap v281: 97 pgs: 97 active+clean; 4.7 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 363 KiB/s rd, 9.1 MiB/s wr, 849 op/s 2024-04-02T14:44:01.934 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:44:01 smithi090 ceph-mon[31674]: pgmap v281: 97 pgs: 97 active+clean; 4.7 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 363 KiB/s rd, 9.1 MiB/s wr, 849 op/s 2024-04-02T14:44:03.856 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:44:03 smithi092 ceph-mon[36497]: pgmap v282: 97 pgs: 97 active+clean; 4.7 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 156 KiB/s rd, 7.4 MiB/s wr, 740 op/s 2024-04-02T14:44:03.932 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:44:03 smithi177 ceph-mon[36431]: pgmap v282: 97 pgs: 97 active+clean; 4.7 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 156 KiB/s rd, 7.4 MiB/s wr, 740 op/s 2024-04-02T14:44:03.934 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:44:03 smithi090 ceph-mon[31674]: pgmap v282: 97 pgs: 97 active+clean; 4.7 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 156 KiB/s rd, 7.4 MiB/s wr, 740 op/s 2024-04-02T14:44:05.856 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:44:05 smithi092 ceph-mon[36497]: pgmap v283: 97 pgs: 97 active+clean; 4.7 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 188 KiB/s rd, 11 MiB/s wr, 1.10k op/s 2024-04-02T14:44:05.914 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:05.916+0000 7fb703200640 1 -- 172.21.15.90:0/782019190 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb6fc100c50 msgr2=0x7fb6fc1010d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:05.914 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:05.916+0000 7fb703200640 1 --2- 172.21.15.90:0/782019190 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb6fc100c50 0x7fb6fc1010d0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fb6e8009890 tx=0x7fb6e802bb70 comp rx=0 tx=0).stop 2024-04-02T14:44:05.914 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:05.916+0000 7fb703200640 1 -- 172.21.15.90:0/782019190 shutdown_connections 2024-04-02T14:44:05.914 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:05.916+0000 7fb703200640 1 --2- 172.21.15.90:0/782019190 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb6fc101610 0x7fb6fc10d830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:05.914 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:05.916+0000 7fb703200640 1 --2- 172.21.15.90:0/782019190 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb6fc100c50 0x7fb6fc1010d0 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:05.914 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:05.916+0000 7fb703200640 1 --2- 172.21.15.90:0/782019190 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb6fc1059a0 0x7fb6fc105d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:05.914 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:05.916+0000 7fb703200640 1 -- 172.21.15.90:0/782019190 >> 172.21.15.90:0/782019190 conn(0x7fb6fc077850 msgr2=0x7fb6fc077c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:44:05.915 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:05.916+0000 7fb703200640 1 -- 172.21.15.90:0/782019190 shutdown_connections 2024-04-02T14:44:05.915 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:05.916+0000 7fb703200640 1 -- 172.21.15.90:0/782019190 wait complete. 2024-04-02T14:44:05.915 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:05.918+0000 7fb703200640 1 Processor -- start 2024-04-02T14:44:05.915 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:05.918+0000 7fb703200640 1 -- start start 2024-04-02T14:44:05.916 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:05.919+0000 7fb703200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb6fc100c50 0x7fb6fc10a270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:05.917 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:05.919+0000 7fb703200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb6fc101610 0x7fb6fc10a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:05.917 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:05.919+0000 7fb703200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb6fc1059a0 0x7fb6fc1a4520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:05.917 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:05.919+0000 7fb703200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fb6fc10f340 con 0x7fb6fc101610 2024-04-02T14:44:05.918 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:05.919+0000 7fb703200640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fb6fc10f1c0 con 0x7fb6fc1059a0 2024-04-02T14:44:05.918 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:05.919+0000 7fb703200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fb6fc10f4c0 con 0x7fb6fc100c50 2024-04-02T14:44:05.918 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:05.919+0000 7fb701200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb6fc1059a0 0x7fb6fc1a4520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:05.918 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:05.919+0000 7fb701200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb6fc1059a0 0x7fb6fc1a4520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:54400/0 (socket says 172.21.15.90:54400) 2024-04-02T14:44:05.918 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:05.919+0000 7fb701200640 1 -- 172.21.15.90:0/3197639612 learned_addr learned my addr 172.21.15.90:0/3197639612 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:44:05.918 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:05.920+0000 7fb6fbe00640 1 --2- 172.21.15.90:0/3197639612 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb6fc100c50 0x7fb6fc10a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:05.919 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:05.920+0000 7fb6fb400640 1 --2- 172.21.15.90:0/3197639612 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb6fc101610 0x7fb6fc10a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:05.919 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:05.920+0000 7fb701200640 1 -- 172.21.15.90:0/3197639612 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb6fc100c50 msgr2=0x7fb6fc10a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:05.919 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:05.920+0000 7fb701200640 1 --2- 172.21.15.90:0/3197639612 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb6fc100c50 0x7fb6fc10a270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:05.919 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:05.920+0000 7fb701200640 1 -- 172.21.15.90:0/3197639612 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb6fc101610 msgr2=0x7fb6fc10a7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:05.919 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:05.920+0000 7fb701200640 1 --2- 172.21.15.90:0/3197639612 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb6fc101610 0x7fb6fc10a7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:05.919 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:05.920+0000 7fb701200640 1 -- 172.21.15.90:0/3197639612 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6fc1a4c50 con 0x7fb6fc1059a0 2024-04-02T14:44:05.920 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:05.921+0000 7fb701200640 1 --2- 172.21.15.90:0/3197639612 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb6fc1059a0 0x7fb6fc1a4520 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7fb6f400da40 tx=0x7fb6f400df00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:44:05.920 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:05.921+0000 7fb6f8c00640 1 -- 172.21.15.90:0/3197639612 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6f4017070 con 0x7fb6fc1059a0 2024-04-02T14:44:05.920 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:05.921+0000 7fb6f8c00640 1 -- 172.21.15.90:0/3197639612 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb6f4003e50 con 0x7fb6fc1059a0 2024-04-02T14:44:05.920 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:05.921+0000 7fb6f8c00640 1 -- 172.21.15.90:0/3197639612 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6f4010460 con 0x7fb6fc1059a0 2024-04-02T14:44:05.921 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:05.922+0000 7fb703200640 1 -- 172.21.15.90:0/3197639612 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6fc1a4f40 con 0x7fb6fc1059a0 2024-04-02T14:44:05.921 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:05.922+0000 7fb703200640 1 -- 172.21.15.90:0/3197639612 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6fc1ab5d0 con 0x7fb6fc1059a0 2024-04-02T14:44:05.925 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:05.928+0000 7fb6f8c00640 1 -- 172.21.15.90:0/3197639612 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fb6f4003970 con 0x7fb6fc1059a0 2024-04-02T14:44:05.926 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:05.929+0000 7fb6f8c00640 1 --2- 172.21.15.90:0/3197639612 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fb6e00a3e40 0x7fb6e00a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:05.926 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:05.929+0000 7fb70600f740 1 Processor -- start 2024-04-02T14:44:05.928 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:05.929+0000 7fb6f8c00640 1 -- 172.21.15.90:0/3197639612 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8550+0+0 (secure 0 0 0) 0x7fb6f40d04b0 con 0x7fb6fc1059a0 2024-04-02T14:44:05.928 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:05.929+0000 7fb70600f740 1 -- start start 2024-04-02T14:44:05.928 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:05.929+0000 7fb70600f740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559d1a73deb0 0x559d1a73e290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:05.928 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:05.929+0000 7fb70600f740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559d1a73e7d0 0x559d1a746e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:05.929 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:05.929+0000 7fb70600f740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559d1a7473b0 0x559d1a7497a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:05.929 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:05.929+0000 7fb70600f740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x559d1a6a2d70 con 0x559d1a73deb0 2024-04-02T14:44:05.929 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:05.929+0000 7fb70600f740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x559d1a64b2a0 con 0x559d1a7473b0 2024-04-02T14:44:05.929 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:05.929+0000 7fb70600f740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x559d1a69ed20 con 0x559d1a73e7d0 2024-04-02T14:44:05.929 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:05.930+0000 7fb6fb400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559d1a73e7d0 0x559d1a746e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:05.929 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:05.930+0000 7fb6fb400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559d1a73e7d0 0x559d1a746e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:42318/0 (socket says 172.21.15.90:42318) 2024-04-02T14:44:05.929 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:05.930+0000 7fb6fb400640 1 -- 172.21.15.90:0/1085121204 learned_addr learned my addr 172.21.15.90:0/1085121204 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:44:05.929 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:05.930+0000 7fb701200640 1 --2- 172.21.15.90:0/1085121204 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559d1a7473b0 0x559d1a7497a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:05.930 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:05.930+0000 7fb6fbe00640 1 --2- 172.21.15.90:0/3197639612 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fb6e00a3e40 0x7fb6e00a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:05.930 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:05.930+0000 7fb6fbe00640 1 --2- 172.21.15.90:0/1085121204 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559d1a73deb0 0x559d1a73e290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:05.930 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:05.930+0000 7fb701200640 1 -- 172.21.15.90:0/1085121204 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559d1a73e7d0 msgr2=0x559d1a746e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:44:05.930 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:05.930+0000 7fb701200640 1 --2- 172.21.15.90:0/1085121204 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559d1a73e7d0 0x559d1a746e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:05.930 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:05.930+0000 7fb701200640 1 -- 172.21.15.90:0/1085121204 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559d1a73deb0 msgr2=0x559d1a73e290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:44:05.930 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:05.930+0000 7fb701200640 1 --2- 172.21.15.90:0/1085121204 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559d1a73deb0 0x559d1a73e290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:05.930 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:05.930+0000 7fb701200640 1 -- 172.21.15.90:0/1085121204 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559d1a65f840 con 0x559d1a7473b0 2024-04-02T14:44:05.930 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:05.930+0000 7fb6fbe00640 1 --2- 172.21.15.90:0/1085121204 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559d1a73deb0 0x559d1a73e290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:44:05.931 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:05.930+0000 7fb701200640 1 --2- 172.21.15.90:0/1085121204 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559d1a7473b0 0x559d1a7497a0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7fb6f40cdbe0 tx=0x7fb6f409bd90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=612cf342177ac300 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:44:05.931 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:05.931+0000 7fb6f1600640 1 -- 172.21.15.90:0/1085121204 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6f4020070 con 0x559d1a7473b0 2024-04-02T14:44:05.931 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:05.931+0000 7fb6f1600640 1 -- 172.21.15.90:0/1085121204 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb6f409c430 con 0x559d1a7473b0 2024-04-02T14:44:05.931 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:05.931+0000 7fb6f1600640 1 -- 172.21.15.90:0/1085121204 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6f4098aa0 con 0x559d1a7473b0 2024-04-02T14:44:05.933 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:05.932+0000 7fb70600f740 1 -- 172.21.15.90:0/1085121204 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559d1a7473b0 msgr2=0x559d1a7497a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:05.933 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:05.932+0000 7fb6fbe00640 1 --2- 172.21.15.90:0/3197639612 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fb6e00a3e40 0x7fb6e00a6300 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fb6ec0096c0 tx=0x7fb6ec005a50 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:44:05.933 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:05.932+0000 7fb70600f740 1 --2- 172.21.15.90:0/1085121204 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559d1a7473b0 0x559d1a7497a0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7fb6f40cdbe0 tx=0x7fb6f409bd90 comp rx=0 tx=0).stop 2024-04-02T14:44:05.933 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:05.932+0000 7fb70600f740 1 -- 172.21.15.90:0/1085121204 shutdown_connections 2024-04-02T14:44:05.934 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:05.932+0000 7fb70600f740 1 --2- 172.21.15.90:0/1085121204 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559d1a7473b0 0x559d1a7497a0 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:05.934 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:05.932+0000 7fb70600f740 1 --2- 172.21.15.90:0/1085121204 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559d1a73e7d0 0x559d1a746e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:05.934 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:05.932+0000 7fb70600f740 1 --2- 172.21.15.90:0/1085121204 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559d1a73deb0 0x559d1a73e290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:05.934 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:05.932+0000 7fb70600f740 1 -- 172.21.15.90:0/1085121204 >> 172.21.15.90:0/1085121204 conn(0x559d1a7385e0 msgr2=0x559d1a7389c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:44:05.934 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:05.932+0000 7fb70600f740 1 -- 172.21.15.90:0/1085121204 shutdown_connections 2024-04-02T14:44:05.934 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:05.932+0000 7fb70600f740 1 -- 172.21.15.90:0/1085121204 wait complete. 2024-04-02T14:44:05.934 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:05.933+0000 7fb70600f740 1 -- start start 2024-04-02T14:44:05.934 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:05.933+0000 7fb70600f740 1 Processor -- start 2024-04-02T14:44:05.934 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:05.934+0000 7fb70600f740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559d1a73deb0 0x559d1a744350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:05.934 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:05.934+0000 7fb70600f740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559d1a73e7d0 0x559d1a744890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:05.934 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:05.934+0000 7fb70600f740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559d1a7473b0 0x559d1a744dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:05.934 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:05.934+0000 7fb70600f740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x559d1a75b1b0 con 0x559d1a73deb0 2024-04-02T14:44:05.935 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:05.935+0000 7fb70600f740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x559d1a65e660 con 0x559d1a7473b0 2024-04-02T14:44:05.935 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:05.935+0000 7fb70600f740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x559d1a75b330 con 0x559d1a73e7d0 2024-04-02T14:44:05.935 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:05.935+0000 7fb70600f740 10 client.0 _ll_register_callbacks cb 0x559d1a65e050 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:44:05.935 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:05.935+0000 7fb6fbe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559d1a73deb0 0x559d1a744350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:05.935 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:05.935+0000 7fb6fb400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559d1a73e7d0 0x559d1a744890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:05.935 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:05.935+0000 7fb6fb400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559d1a73e7d0 0x559d1a744890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:42328/0 (socket says 172.21.15.90:42328) 2024-04-02T14:44:05.935 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:05.935+0000 7fb6fb400640 1 -- 172.21.15.90:0/935122716 learned_addr learned my addr 172.21.15.90:0/935122716 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:44:05.935 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:05.935+0000 7fb701200640 1 --2- 172.21.15.90:0/935122716 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559d1a7473b0 0x559d1a744dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:05.935 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:05.936+0000 7fb6fb400640 1 -- 172.21.15.90:0/935122716 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559d1a7473b0 msgr2=0x559d1a744dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:44:05.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:44:05 smithi177 ceph-mon[36431]: pgmap v283: 97 pgs: 97 active+clean; 4.7 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 188 KiB/s rd, 11 MiB/s wr, 1.10k op/s 2024-04-02T14:44:05.936 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:44:05 smithi090 ceph-mon[31674]: pgmap v283: 97 pgs: 97 active+clean; 4.7 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 188 KiB/s rd, 11 MiB/s wr, 1.10k op/s 2024-04-02T14:44:05.938 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:05.936+0000 7fb6fb400640 1 --2- 172.21.15.90:0/935122716 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559d1a7473b0 0x559d1a744dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:05.938 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:05.936+0000 7fb6fb400640 1 -- 172.21.15.90:0/935122716 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559d1a73deb0 msgr2=0x559d1a744350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:44:05.938 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:05.936+0000 7fb6fb400640 1 --2- 172.21.15.90:0/935122716 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559d1a73deb0 0x559d1a744350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:05.938 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:05.936+0000 7fb6fb400640 1 -- 172.21.15.90:0/935122716 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559d1a66de80 con 0x559d1a73e7d0 2024-04-02T14:44:05.938 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:05.936+0000 7fb701200640 1 --2- 172.21.15.90:0/935122716 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559d1a7473b0 0x559d1a744dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:44:05.938 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:05.937+0000 7fb6fb400640 1 --2- 172.21.15.90:0/935122716 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559d1a73e7d0 0x559d1a744890 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fb6e802c180 tx=0x7fb6e8005d00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e72e667b11de4fd3 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:44:05.938 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:05.937+0000 7fb6e6000640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:44:05.939 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:05.937+0000 7fb6e6000640 1 -- 172.21.15.90:0/935122716 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6e8039040 con 0x559d1a73e7d0 2024-04-02T14:44:05.939 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:05.937+0000 7fb6e6000640 1 -- 172.21.15.90:0/935122716 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8550+0+0 (secure 0 0 0) 0x7fb6e8034440 con 0x559d1a73e7d0 2024-04-02T14:44:05.939 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:05.938+0000 7fb70600f740 1 -- 172.21.15.90:0/935122716 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb6f409c430 con 0x559d1a73e7d0 2024-04-02T14:44:05.939 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:05.938+0000 7fb6e6000640 1 -- 172.21.15.90:0/935122716 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fb6e8034640 con 0x559d1a73e7d0 2024-04-02T14:44:05.939 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:05.939+0000 7fb70600f740 10 client.28162 fetch_fsmap learned FSMap version 27 2024-04-02T14:44:05.939 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:05.939+0000 7fb70600f740 1 -- 172.21.15.90:0/935122716 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x559d1a3b7210 con 0x559d1a73e7d0 2024-04-02T14:44:05.939 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:05.939+0000 7fb6e6000640 1 -- 172.21.15.90:0/935122716 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb6f409c430 con 0x559d1a73e7d0 2024-04-02T14:44:05.939 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:05.939+0000 7fb6e6000640 1 -- 172.21.15.90:0/935122716 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fb6e8004ac0 con 0x559d1a73e7d0 2024-04-02T14:44:05.939 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:05.940+0000 7fb70600f740 10 client.28162 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:44:05.939 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:05.940+0000 7fb70600f740 10 client.28162 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-02T14:44:05.940 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:05.940+0000 7fb70600f740 20 client.28162 populate_metadata read hostname 'smithi090' 2024-04-02T14:44:05.940 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:05.940+0000 7fb70600f740 1 --2- 172.21.15.90:0/935122716 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x559d1a768610 0x559d1a76aa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:05.940 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:05.940+0000 7fb70600f740 4 client.28162 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:44:05.940 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:05.940+0000 7fb70600f740 1 -- 172.21.15.90:0/935122716 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559d1a652cd0 con 0x559d1a768610 2024-04-02T14:44:05.940 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:05.940+0000 7fb6fbe00640 1 --2- 172.21.15.90:0/935122716 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x559d1a768610 0x559d1a76aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:05.941 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:05.941+0000 7fb6fbe00640 1 --2- 172.21.15.90:0/935122716 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x559d1a768610 0x559d1a76aa60 crc :-1 s=READY pgs=248 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ef5200bea4197a2f server_cookie=5c4a48b130ad3d13 in_seq=0 out_seq=0 2024-04-02T14:44:05.941 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:05.941+0000 7fb6e6000640 10 client.28162 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:44:05.941 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:05.942+0000 7fb6e6000640 1 -- 172.21.15.90:0/935122716 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x559d1a652cd0 con 0x559d1a768610 2024-04-02T14:44:05.942 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:05.942+0000 7fb6e6000640 10 client.28162 handle_command_reply: tid=0 2024-04-02T14:44:05.942 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:05.942+0000 7fb70600f740 1 client.28162 shutdown 2024-04-02T14:44:05.942 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:05.942+0000 7fb70600f740 1 -- 172.21.15.90:0/935122716 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559d1a73e7d0 msgr2=0x559d1a744890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:05.942 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:05.942+0000 7fb70600f740 1 --2- 172.21.15.90:0/935122716 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559d1a73e7d0 0x559d1a744890 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fb6e802c180 tx=0x7fb6e8005d00 comp rx=0 tx=0).stop 2024-04-02T14:44:05.942 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:05.943+0000 7fb70600f740 1 -- 172.21.15.90:0/935122716 shutdown_connections 2024-04-02T14:44:05.942 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:05.943+0000 7fb70600f740 1 --2- 172.21.15.90:0/935122716 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x559d1a768610 0x559d1a76aa60 crc :-1 s=READY pgs=248 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:05.942 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:05.943+0000 7fb70600f740 1 --2- 172.21.15.90:0/935122716 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559d1a7473b0 0x559d1a744dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:05.942 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:05.943+0000 7fb70600f740 1 --2- 172.21.15.90:0/935122716 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559d1a73e7d0 0x559d1a744890 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:05.942 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:05.943+0000 7fb70600f740 1 --2- 172.21.15.90:0/935122716 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559d1a73deb0 0x559d1a744350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:05.942 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:05.943+0000 7fb70600f740 1 -- 172.21.15.90:0/935122716 >> 172.21.15.90:0/935122716 conn(0x559d1a7385e0 msgr2=0x559d1a73a330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:44:05.943 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:05.943+0000 7fb6e6000640 0 client.28162 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:44:05.943 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:05.943+0000 7fb70600f740 1 -- 172.21.15.90:0/935122716 shutdown_connections 2024-04-02T14:44:05.943 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:05.943+0000 7fb70600f740 1 -- 172.21.15.90:0/935122716 wait complete. 2024-04-02T14:44:05.943 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:05.944+0000 7fb70600f740 20 client.28162 trim_cache size 0 max 16384 2024-04-02T14:44:05.981 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:05.984+0000 7fb70600f740 1 Processor -- start 2024-04-02T14:44:05.982 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:05.984+0000 7fb70600f740 1 -- start start 2024-04-02T14:44:05.982 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:05.984+0000 7fb70600f740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559d1a73deb0 0x559d1a746a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:05.982 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:05.984+0000 7fb70600f740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559d1a73e7d0 0x559d1a764070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:05.982 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:05.984+0000 7fb70600f740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559d1a7473b0 0x559d1a770370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:05.982 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:05.984+0000 7fb70600f740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fb6fc1ab7a0 con 0x559d1a73deb0 2024-04-02T14:44:05.982 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:05.984+0000 7fb70600f740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fb6fc1ad180 con 0x559d1a7473b0 2024-04-02T14:44:05.982 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:05.984+0000 7fb70600f740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fb6fc1abd10 con 0x559d1a73e7d0 2024-04-02T14:44:05.983 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:05.985+0000 7fb701200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559d1a7473b0 0x559d1a770370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:05.983 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:05.985+0000 7fb701200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559d1a7473b0 0x559d1a770370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:54428/0 (socket says 172.21.15.90:54428) 2024-04-02T14:44:05.983 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:05.985+0000 7fb701200640 1 -- 172.21.15.90:0/3396122179 learned_addr learned my addr 172.21.15.90:0/3396122179 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:44:05.983 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:05.985+0000 7fb701200640 1 -- 172.21.15.90:0/3396122179 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559d1a73e7d0 msgr2=0x559d1a764070 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-02T14:44:05.983 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:05.985+0000 7fb701200640 1 --2- 172.21.15.90:0/3396122179 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559d1a73e7d0 0x559d1a764070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:05.983 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:05.985+0000 7fb701200640 1 -- 172.21.15.90:0/3396122179 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559d1a73deb0 msgr2=0x559d1a746a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:44:05.983 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:05.985+0000 7fb701200640 1 --2- 172.21.15.90:0/3396122179 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559d1a73deb0 0x559d1a746a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:05.983 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:05.985+0000 7fb701200640 1 -- 172.21.15.90:0/3396122179 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559d1a73f180 con 0x559d1a7473b0 2024-04-02T14:44:05.984 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:05.986+0000 7fb701200640 1 --2- 172.21.15.90:0/3396122179 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559d1a7473b0 0x559d1a770370 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7fb6f40ce1e0 tx=0x7fb6f4004250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9e759112773ad4ee server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:44:05.984 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:05.986+0000 7fb6f0c00640 1 -- 172.21.15.90:0/3396122179 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6f40b5070 con 0x559d1a7473b0 2024-04-02T14:44:05.986 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:05.987+0000 7fb6f0c00640 1 -- 172.21.15.90:0/3396122179 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb6f4003c60 con 0x559d1a7473b0 2024-04-02T14:44:05.986 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:05.987+0000 7fb6f0c00640 1 -- 172.21.15.90:0/3396122179 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6f4020040 con 0x559d1a7473b0 2024-04-02T14:44:05.986 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:05.988+0000 7fb70600f740 1 -- 172.21.15.90:0/3396122179 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559d1a7473b0 msgr2=0x559d1a770370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:05.986 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:05.988+0000 7fb70600f740 1 --2- 172.21.15.90:0/3396122179 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559d1a7473b0 0x559d1a770370 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7fb6f40ce1e0 tx=0x7fb6f4004250 comp rx=0 tx=0).stop 2024-04-02T14:44:05.987 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:05.989+0000 7fb70600f740 1 -- 172.21.15.90:0/3396122179 shutdown_connections 2024-04-02T14:44:05.987 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:05.989+0000 7fb70600f740 1 --2- 172.21.15.90:0/3396122179 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559d1a7473b0 0x559d1a770370 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:05.987 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:05.989+0000 7fb70600f740 1 --2- 172.21.15.90:0/3396122179 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559d1a73e7d0 0x559d1a764070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:05.987 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:05.989+0000 7fb70600f740 1 --2- 172.21.15.90:0/3396122179 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559d1a73deb0 0x559d1a746a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:05.987 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:05.990+0000 7fb70600f740 1 -- 172.21.15.90:0/3396122179 >> 172.21.15.90:0/3396122179 conn(0x559d1a7385e0 msgr2=0x559d1a7389c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:44:05.988 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:05.990+0000 7fb70600f740 1 -- 172.21.15.90:0/3396122179 shutdown_connections 2024-04-02T14:44:05.988 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:05.991+0000 7fb70600f740 1 -- 172.21.15.90:0/3396122179 wait complete. 2024-04-02T14:44:05.989 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:05.991+0000 7fb70600f740 1 -- start start 2024-04-02T14:44:05.989 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:05.992+0000 7fb70600f740 1 Processor -- start 2024-04-02T14:44:05.992 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:05.995+0000 7fb70600f740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559d1a73deb0 0x559d1a7412b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:05.993 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:05.995+0000 7fb70600f740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559d1a73e7d0 0x559d1a7417f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:05.993 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:05.996+0000 7fb70600f740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559d1a7473b0 0x559d1a741d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:05.994 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:05.996+0000 7fb701200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559d1a7473b0 0x559d1a741d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:05.994 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:05.996+0000 7fb701200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559d1a7473b0 0x559d1a741d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:54434/0 (socket says 172.21.15.90:54434) 2024-04-02T14:44:05.994 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:05.996+0000 7fb701200640 1 -- 172.21.15.90:0/3782331023 learned_addr learned my addr 172.21.15.90:0/3782331023 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:44:05.994 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:05.996+0000 7fb70600f740 1 -- 172.21.15.90:0/3782331023 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fb6fc1acab0 con 0x559d1a73e7d0 2024-04-02T14:44:05.995 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:05.997+0000 7fb6fbe00640 1 --2- 172.21.15.90:0/3782331023 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559d1a73deb0 0x559d1a7412b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:05.995 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:05.997+0000 7fb70600f740 1 -- 172.21.15.90:0/3782331023 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fb6fc1ac3e0 con 0x559d1a7473b0 2024-04-02T14:44:05.995 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:05.997+0000 7fb70600f740 1 -- 172.21.15.90:0/3782331023 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fb6fc10b5c0 con 0x559d1a73deb0 2024-04-02T14:44:05.995 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:05.997+0000 7fb70600f740 10 client.0 _ll_register_callbacks cb 0x7fb6e80096b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:44:05.995 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:05.998+0000 7fb701200640 1 -- 172.21.15.90:0/3782331023 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559d1a73deb0 msgr2=0x559d1a7412b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:44:05.995 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:05.998+0000 7fb701200640 1 --2- 172.21.15.90:0/3782331023 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559d1a73deb0 0x559d1a7412b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:05.995 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:05.998+0000 7fb701200640 1 -- 172.21.15.90:0/3782331023 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559d1a73e7d0 msgr2=0x559d1a7417f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:44:05.995 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:05.998+0000 7fb701200640 1 --2- 172.21.15.90:0/3782331023 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559d1a73e7d0 0x559d1a7417f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:05.996 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:05.998+0000 7fb701200640 1 -- 172.21.15.90:0/3782331023 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559d1a483620 con 0x559d1a7473b0 2024-04-02T14:44:05.996 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:05.998+0000 7fb701200640 1 --2- 172.21.15.90:0/3782331023 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559d1a7473b0 0x559d1a741d30 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7fb6f409c3c0 tx=0x7fb6f400bed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=330042025fdb13d6 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:44:05.996 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:05.998+0000 7fb6e6000640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:44:05.997 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:05.999+0000 7fb6e6000640 1 -- 172.21.15.90:0/3782331023 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6f40ba060 con 0x559d1a7473b0 2024-04-02T14:44:05.997 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:05.999+0000 7fb6e6000640 1 -- 172.21.15.90:0/3782331023 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8550+0+0 (secure 0 0 0) 0x7fb6f40b5bb0 con 0x559d1a7473b0 2024-04-02T14:44:05.997 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:05.999+0000 7fb70600f740 1 -- 172.21.15.90:0/3782331023 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x559d1a6b02b0 con 0x559d1a7473b0 2024-04-02T14:44:05.997 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:06.000+0000 7fb6e6000640 1 -- 172.21.15.90:0/3782331023 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fb6f4020070 con 0x559d1a7473b0 2024-04-02T14:44:05.998 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:06.000+0000 7fb70600f740 10 client.28334 fetch_fsmap learned FSMap version 27 2024-04-02T14:44:05.998 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:06.000+0000 7fb70600f740 1 -- 172.21.15.90:0/3782331023 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x559d1a7cef10 con 0x559d1a7473b0 2024-04-02T14:44:05.998 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:06.000+0000 7fb6e6000640 1 -- 172.21.15.90:0/3782331023 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x559d1a6b02b0 con 0x559d1a7473b0 2024-04-02T14:44:06.001 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:06.002+0000 7fb6e6000640 1 -- 172.21.15.90:0/3782331023 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fb6f409b2f0 con 0x559d1a7473b0 2024-04-02T14:44:06.001 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:06.002+0000 7fb70600f740 10 client.28334 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:44:06.001 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:06.002+0000 7fb70600f740 10 client.28334 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-02T14:44:06.001 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:06.002+0000 7fb70600f740 20 client.28334 populate_metadata read hostname 'smithi090' 2024-04-02T14:44:06.001 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:06.002+0000 7fb70600f740 1 --2- 172.21.15.90:0/3782331023 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x559d1a7cef10 0x559d1a7d1300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:06.001 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:06.002+0000 7fb70600f740 4 client.28334 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-02T14:44:06.001 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:06.002+0000 7fb70600f740 1 -- 172.21.15.90:0/3782331023 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "flush journal"}) -- 0x559d1a6aa600 con 0x559d1a7cef10 2024-04-02T14:44:06.001 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:06.003+0000 7fb6fbe00640 1 --2- 172.21.15.90:0/3782331023 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x559d1a7cef10 0x559d1a7d1300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:06.002 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:06.004+0000 7fb6fbe00640 1 --2- 172.21.15.90:0/3782331023 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x559d1a7cef10 0x559d1a7d1300 crc :-1 s=READY pgs=249 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b5b6422b4a8e2aa8 server_cookie=31dcdf8170389661 in_seq=0 out_seq=0 2024-04-02T14:44:06.002 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:06.004+0000 7fb6e6000640 10 client.28334 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:44:06.057 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:06.059+0000 7fb6e6000640 1 -- 172.21.15.90:0/3782331023 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x559d1a6aa600 con 0x559d1a7cef10 2024-04-02T14:44:06.058 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:06.059+0000 7fb6e6000640 10 client.28334 handle_command_reply: tid=0 2024-04-02T14:44:06.058 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:06.059+0000 7fb70600f740 1 client.28334 shutdown 2024-04-02T14:44:06.058 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:06.060+0000 7fb70600f740 1 -- 172.21.15.90:0/3782331023 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559d1a7473b0 msgr2=0x559d1a741d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:06.058 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:06.060+0000 7fb70600f740 1 --2- 172.21.15.90:0/3782331023 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559d1a7473b0 0x559d1a741d30 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7fb6f409c3c0 tx=0x7fb6f400bed0 comp rx=0 tx=0).stop 2024-04-02T14:44:06.059 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:06.060+0000 7fb70600f740 1 -- 172.21.15.90:0/3782331023 shutdown_connections 2024-04-02T14:44:06.059 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:06.060+0000 7fb70600f740 1 --2- 172.21.15.90:0/3782331023 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x559d1a7cef10 0x559d1a7d1300 crc :-1 s=READY pgs=249 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:06.059 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:06.060+0000 7fb70600f740 1 --2- 172.21.15.90:0/3782331023 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559d1a7473b0 0x559d1a741d30 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:06.059 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:06.060+0000 7fb70600f740 1 --2- 172.21.15.90:0/3782331023 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559d1a73e7d0 0x559d1a7417f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:06.059 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:06.060+0000 7fb70600f740 1 --2- 172.21.15.90:0/3782331023 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559d1a73deb0 0x559d1a7412b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:06.059 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:06.060+0000 7fb70600f740 1 -- 172.21.15.90:0/3782331023 >> 172.21.15.90:0/3782331023 conn(0x559d1a7385e0 msgr2=0x559d1a7399a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:44:06.059 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:06.060+0000 7fb6e6000640 0 client.28334 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:44:06.059 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:06.061+0000 7fb70600f740 1 -- 172.21.15.90:0/3782331023 shutdown_connections 2024-04-02T14:44:06.059 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:06.061+0000 7fb70600f740 1 -- 172.21.15.90:0/3782331023 wait complete. 2024-04-02T14:44:06.059 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:06.061+0000 7fb70600f740 20 client.28334 trim_cache size 0 max 16384 2024-04-02T14:44:06.060 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:{ 2024-04-02T14:44:06.060 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "message": "", 2024-04-02T14:44:06.060 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "return_code": 0 2024-04-02T14:44:06.060 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:} 2024-04-02T14:44:06.060 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:06.062+0000 7fb703200640 1 -- 172.21.15.90:0/3197639612 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fb6e00a3e40 msgr2=0x7fb6e00a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:06.060 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:06.062+0000 7fb703200640 1 --2- 172.21.15.90:0/3197639612 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fb6e00a3e40 0x7fb6e00a6300 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fb6ec0096c0 tx=0x7fb6ec005a50 comp rx=0 tx=0).stop 2024-04-02T14:44:06.060 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:06.062+0000 7fb703200640 1 -- 172.21.15.90:0/3197639612 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb6fc1059a0 msgr2=0x7fb6fc1a4520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:06.061 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:06.062+0000 7fb703200640 1 --2- 172.21.15.90:0/3197639612 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb6fc1059a0 0x7fb6fc1a4520 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7fb6f400da40 tx=0x7fb6f400df00 comp rx=0 tx=0).stop 2024-04-02T14:44:06.061 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:06.062+0000 7fb703200640 1 -- 172.21.15.90:0/3197639612 shutdown_connections 2024-04-02T14:44:06.061 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:06.062+0000 7fb703200640 1 --2- 172.21.15.90:0/3197639612 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fb6e00a3e40 0x7fb6e00a6300 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:06.061 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:06.062+0000 7fb703200640 1 --2- 172.21.15.90:0/3197639612 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb6fc1059a0 0x7fb6fc1a4520 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:06.061 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:06.062+0000 7fb703200640 1 --2- 172.21.15.90:0/3197639612 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb6fc101610 0x7fb6fc10a7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:06.061 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:06.062+0000 7fb703200640 1 --2- 172.21.15.90:0/3197639612 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb6fc100c50 0x7fb6fc10a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:06.061 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:06.062+0000 7fb703200640 1 -- 172.21.15.90:0/3197639612 >> 172.21.15.90:0/3197639612 conn(0x7fb6fc077850 msgr2=0x7fb6fc1045b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:44:06.061 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:06.062+0000 7fb703200640 1 -- 172.21.15.90:0/3197639612 shutdown_connections 2024-04-02T14:44:06.061 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:06.062+0000 7fb703200640 1 -- 172.21.15.90:0/3197639612 wait complete. 2024-04-02T14:44:06.827 INFO:teuthology.orchestra.run.smithi090.stderr:progress: 515.0 s, 804.4 tps, lat 127.091 ms stddev 327.374 2024-04-02T14:44:07.932 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:44:07 smithi177 ceph-mon[36431]: pgmap v284: 97 pgs: 97 active+clean; 4.7 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 7.0 MiB/s wr, 625 op/s 2024-04-02T14:44:07.933 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:44:07 smithi090 ceph-mon[31674]: pgmap v284: 97 pgs: 97 active+clean; 4.7 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 7.0 MiB/s wr, 625 op/s 2024-04-02T14:44:08.106 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:44:07 smithi092 ceph-mon[36497]: pgmap v284: 97 pgs: 97 active+clean; 4.7 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 7.0 MiB/s wr, 625 op/s 2024-04-02T14:44:08.932 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:44:08 smithi177 ceph-mon[36431]: osdmap e96: 12 total, 12 up, 12 in 2024-04-02T14:44:08.933 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:44:08 smithi090 ceph-mon[31674]: osdmap e96: 12 total, 12 up, 12 in 2024-04-02T14:44:09.106 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:44:08 smithi092 ceph-mon[36497]: osdmap e96: 12 total, 12 up, 12 in 2024-04-02T14:44:09.932 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:44:09 smithi177 ceph-mon[36431]: pgmap v286: 97 pgs: 12 active+clean+snaptrim, 17 active+clean+snaptrim_wait, 68 active+clean; 4.7 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 14 MiB/s wr, 1.18k op/s 2024-04-02T14:44:09.933 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:44:09 smithi090 ceph-mon[31674]: pgmap v286: 97 pgs: 12 active+clean+snaptrim, 17 active+clean+snaptrim_wait, 68 active+clean; 4.7 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 14 MiB/s wr, 1.18k op/s 2024-04-02T14:44:10.106 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:44:09 smithi092 ceph-mon[36497]: pgmap v286: 97 pgs: 12 active+clean+snaptrim, 17 active+clean+snaptrim_wait, 68 active+clean; 4.7 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 14 MiB/s wr, 1.18k op/s 2024-04-02T14:44:11.827 INFO:teuthology.orchestra.run.smithi090.stderr:progress: 520.0 s, 483.4 tps, lat 207.918 ms stddev 524.896 2024-04-02T14:44:11.932 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:44:11 smithi177 ceph-mon[36431]: pgmap v287: 97 pgs: 22 active+clean+snaptrim, 40 active+clean+snaptrim_wait, 35 active+clean; 4.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 20 MiB/s wr, 1.68k op/s 2024-04-02T14:44:11.933 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:44:11 smithi090 ceph-mon[31674]: pgmap v287: 97 pgs: 22 active+clean+snaptrim, 40 active+clean+snaptrim_wait, 35 active+clean; 4.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 20 MiB/s wr, 1.68k op/s 2024-04-02T14:44:12.106 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:44:11 smithi092 ceph-mon[36497]: pgmap v287: 97 pgs: 22 active+clean+snaptrim, 40 active+clean+snaptrim_wait, 35 active+clean; 4.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 20 MiB/s wr, 1.68k op/s 2024-04-02T14:44:13.932 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:44:13 smithi177 ceph-mon[36431]: pgmap v288: 97 pgs: 22 active+clean+snaptrim, 40 active+clean+snaptrim_wait, 35 active+clean; 4.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 20 MiB/s wr, 1.68k op/s 2024-04-02T14:44:13.935 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:44:13 smithi090 ceph-mon[31674]: pgmap v288: 97 pgs: 22 active+clean+snaptrim, 40 active+clean+snaptrim_wait, 35 active+clean; 4.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 20 MiB/s wr, 1.68k op/s 2024-04-02T14:44:14.106 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:44:13 smithi092 ceph-mon[36497]: pgmap v288: 97 pgs: 22 active+clean+snaptrim, 40 active+clean+snaptrim_wait, 35 active+clean; 4.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 20 MiB/s wr, 1.68k op/s 2024-04-02T14:44:14.932 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:44:14 smithi177 ceph-mon[36431]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T14:44:14.934 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:44:14 smithi090 ceph-mon[31674]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T14:44:15.106 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:44:14 smithi092 ceph-mon[36497]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T14:44:16.356 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:44:16 smithi092 ceph-mon[36497]: pgmap v289: 97 pgs: 97 active+clean; 4.4 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 18 MiB/s wr, 1.49k op/s 2024-04-02T14:44:16.356 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:44:16 smithi092 ceph-mon[36497]: osdmap e97: 12 total, 12 up, 12 in 2024-04-02T14:44:16.432 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:44:16 smithi177 ceph-mon[36431]: pgmap v289: 97 pgs: 97 active+clean; 4.4 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 18 MiB/s wr, 1.49k op/s 2024-04-02T14:44:16.432 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:44:16 smithi177 ceph-mon[36431]: osdmap e97: 12 total, 12 up, 12 in 2024-04-02T14:44:16.433 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:44:16 smithi090 ceph-mon[31674]: pgmap v289: 97 pgs: 97 active+clean; 4.4 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 18 MiB/s wr, 1.49k op/s 2024-04-02T14:44:16.434 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:44:16 smithi090 ceph-mon[31674]: osdmap e97: 12 total, 12 up, 12 in 2024-04-02T14:44:16.827 INFO:teuthology.orchestra.run.smithi090.stderr:progress: 525.0 s, 2149.4 tps, lat 46.336 ms stddev 57.779 2024-04-02T14:44:17.356 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:44:17 smithi092 ceph-mon[36497]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:44:17.356 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:44:17 smithi092 ceph-mon[36497]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T14:44:17.356 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:44:17 smithi092 ceph-mon[36497]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' 2024-04-02T14:44:17.432 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:44:17 smithi177 ceph-mon[36431]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:44:17.432 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:44:17 smithi177 ceph-mon[36431]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T14:44:17.432 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:44:17 smithi177 ceph-mon[36431]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' 2024-04-02T14:44:17.433 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:44:17 smithi090 ceph-mon[31674]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:44:17.433 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:44:17 smithi090 ceph-mon[31674]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T14:44:17.434 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:44:17 smithi090 ceph-mon[31674]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' 2024-04-02T14:44:18.432 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:44:18 smithi177 ceph-mon[36431]: pgmap v291: 97 pgs: 97 active+clean; 4.4 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 20 MiB/s wr, 1.67k op/s 2024-04-02T14:44:18.433 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:44:18 smithi090 ceph-mon[31674]: pgmap v291: 97 pgs: 97 active+clean; 4.4 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 20 MiB/s wr, 1.67k op/s 2024-04-02T14:44:18.606 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:44:18 smithi092 ceph-mon[36497]: pgmap v291: 97 pgs: 97 active+clean; 4.4 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 20 MiB/s wr, 1.67k op/s 2024-04-02T14:44:19.250 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.251+0000 7ff1d6a00640 1 -- 172.21.15.90:0/3582773358 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff1d0074850 msgr2=0x7ff1d010ddd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:19.250 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.251+0000 7ff1d6a00640 1 --2- 172.21.15.90:0/3582773358 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff1d0074850 0x7ff1d010ddd0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7ff1c0009860 tx=0x7ff1c002bc50 comp rx=0 tx=0).stop 2024-04-02T14:44:19.250 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.252+0000 7ff1d6a00640 1 -- 172.21.15.90:0/3582773358 shutdown_connections 2024-04-02T14:44:19.250 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.252+0000 7ff1d6a00640 1 --2- 172.21.15.90:0/3582773358 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff1d010e310 0x7ff1d0110760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:19.250 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.252+0000 7ff1d6a00640 1 --2- 172.21.15.90:0/3582773358 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff1d0074850 0x7ff1d010ddd0 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:19.250 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.252+0000 7ff1d6a00640 1 --2- 172.21.15.90:0/3582773358 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff1d0073f30 0x7ff1d0074310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:19.250 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.252+0000 7ff1d6a00640 1 -- 172.21.15.90:0/3582773358 >> 172.21.15.90:0/3582773358 conn(0x7ff1d00fd5c0 msgr2=0x7ff1d00ff9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:44:19.251 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.252+0000 7ff1d6a00640 1 -- 172.21.15.90:0/3582773358 shutdown_connections 2024-04-02T14:44:19.251 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.252+0000 7ff1d6a00640 1 -- 172.21.15.90:0/3582773358 wait complete. 2024-04-02T14:44:19.251 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.253+0000 7ff1d6a00640 1 Processor -- start 2024-04-02T14:44:19.251 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.254+0000 7ff1d6a00640 1 -- start start 2024-04-02T14:44:19.252 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.254+0000 7ff1d6a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff1d0073f30 0x7ff1d0107440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:19.252 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.255+0000 7ff1d6a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff1d0074850 0x7ff1d0107980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:19.252 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.255+0000 7ff1d6a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff1d010e310 0x7ff1d0102550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:19.252 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.255+0000 7ff1d6a00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7ff1d01140f0 con 0x7ff1d0074850 2024-04-02T14:44:19.253 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.255+0000 7ff1d6a00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7ff1d0113f70 con 0x7ff1d0073f30 2024-04-02T14:44:19.253 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.255+0000 7ff1d6a00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7ff1d0114270 con 0x7ff1d010e310 2024-04-02T14:44:19.253 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.255+0000 7ff1cfe00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff1d0073f30 0x7ff1d0107440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:19.253 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.255+0000 7ff1cfe00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff1d0073f30 0x7ff1d0107440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:38768/0 (socket says 172.21.15.90:38768) 2024-04-02T14:44:19.253 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.255+0000 7ff1cfe00640 1 -- 172.21.15.90:0/2602132970 learned_addr learned my addr 172.21.15.90:0/2602132970 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:44:19.253 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.255+0000 7ff1d4a00640 1 --2- 172.21.15.90:0/2602132970 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff1d010e310 0x7ff1d0102550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:19.253 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.255+0000 7ff1cf400640 1 --2- 172.21.15.90:0/2602132970 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff1d0074850 0x7ff1d0107980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:19.254 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.256+0000 7ff1cfe00640 1 -- 172.21.15.90:0/2602132970 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff1d010e310 msgr2=0x7ff1d0102550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:19.254 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.256+0000 7ff1cfe00640 1 --2- 172.21.15.90:0/2602132970 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff1d010e310 0x7ff1d0102550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:19.254 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.256+0000 7ff1cfe00640 1 -- 172.21.15.90:0/2602132970 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff1d0074850 msgr2=0x7ff1d0107980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:19.254 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.256+0000 7ff1cfe00640 1 --2- 172.21.15.90:0/2602132970 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff1d0074850 0x7ff1d0107980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:19.254 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.256+0000 7ff1cfe00640 1 -- 172.21.15.90:0/2602132970 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1d0102e10 con 0x7ff1d0073f30 2024-04-02T14:44:19.255 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.257+0000 7ff1cfe00640 1 --2- 172.21.15.90:0/2602132970 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff1d0073f30 0x7ff1d0107440 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7ff1b800b4a0 tx=0x7ff1b800b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:44:19.255 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.257+0000 7ff1ccc00640 1 -- 172.21.15.90:0/2602132970 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1b8014070 con 0x7ff1d0073f30 2024-04-02T14:44:19.255 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.257+0000 7ff1d6a00640 1 -- 172.21.15.90:0/2602132970 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1d0103100 con 0x7ff1d0073f30 2024-04-02T14:44:19.255 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.257+0000 7ff1d6a00640 1 -- 172.21.15.90:0/2602132970 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1d010d330 con 0x7ff1d0073f30 2024-04-02T14:44:19.255 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.258+0000 7ff1ccc00640 1 -- 172.21.15.90:0/2602132970 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff1b8010af0 con 0x7ff1d0073f30 2024-04-02T14:44:19.255 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.258+0000 7ff1ccc00640 1 -- 172.21.15.90:0/2602132970 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1b8014070 con 0x7ff1d0073f30 2024-04-02T14:44:19.260 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.263+0000 7ff1d95fa740 1 Processor -- start 2024-04-02T14:44:19.261 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.263+0000 7ff1d95fa740 1 -- start start 2024-04-02T14:44:19.261 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.264+0000 7ff1d95fa740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x565059aeceb0 0x565059aed290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:19.262 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.264+0000 7ff1d95fa740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x565059aed7d0 0x565059af5e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:19.262 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.264+0000 7ff1d95fa740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x565059af63b0 0x565059af87a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:19.262 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.264+0000 7ff1d95fa740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x565059a51d70 con 0x565059af63b0 2024-04-02T14:44:19.262 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.264+0000 7ff1d95fa740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x5650599fa2a0 con 0x565059aed7d0 2024-04-02T14:44:19.262 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.264+0000 7ff1d95fa740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x565059a4dd20 con 0x565059aeceb0 2024-04-02T14:44:19.262 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.264+0000 7ff1d4a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x565059aeceb0 0x565059aed290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:19.262 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.264+0000 7ff1d4a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x565059aeceb0 0x565059aed290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:39164/0 (socket says 172.21.15.90:39164) 2024-04-02T14:44:19.262 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.264+0000 7ff1d4a00640 1 -- 172.21.15.90:0/1043439567 learned_addr learned my addr 172.21.15.90:0/1043439567 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:44:19.263 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.265+0000 7ff1d4a00640 1 -- 172.21.15.90:0/1043439567 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x565059aed7d0 msgr2=0x565059af5e70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-02T14:44:19.263 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.265+0000 7ff1d4a00640 1 --2- 172.21.15.90:0/1043439567 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x565059aed7d0 0x565059af5e70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:19.263 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.265+0000 7ff1d4a00640 1 -- 172.21.15.90:0/1043439567 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x565059af63b0 msgr2=0x565059af87a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:44:19.263 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.265+0000 7ff1d4a00640 1 --2- 172.21.15.90:0/1043439567 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x565059af63b0 0x565059af87a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:19.263 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.265+0000 7ff1d4a00640 1 -- 172.21.15.90:0/1043439567 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x565059a0e840 con 0x565059aeceb0 2024-04-02T14:44:19.263 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.266+0000 7ff1d4a00640 1 --2- 172.21.15.90:0/1043439567 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x565059aeceb0 0x565059aed290 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7ff1c800e4d0 tx=0x7ff1c80095c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dbe9352a67a18fd9 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:44:19.264 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.266+0000 7ff1c5200640 1 -- 172.21.15.90:0/1043439567 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1c8003970 con 0x565059aeceb0 2024-04-02T14:44:19.264 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.266+0000 7ff1c5200640 1 -- 172.21.15.90:0/1043439567 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff1c800d770 con 0x565059aeceb0 2024-04-02T14:44:19.264 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.266+0000 7ff1c5200640 1 -- 172.21.15.90:0/1043439567 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1c8004560 con 0x565059aeceb0 2024-04-02T14:44:19.264 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.266+0000 7ff1ccc00640 1 -- 172.21.15.90:0/2602132970 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7ff1b8017070 con 0x7ff1d0073f30 2024-04-02T14:44:19.264 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.267+0000 7ff1d95fa740 1 -- 172.21.15.90:0/1043439567 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x565059aeceb0 msgr2=0x565059aed290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:19.265 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.267+0000 7ff1d95fa740 1 --2- 172.21.15.90:0/1043439567 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x565059aeceb0 0x565059aed290 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7ff1c800e4d0 tx=0x7ff1c80095c0 comp rx=0 tx=0).stop 2024-04-02T14:44:19.265 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.268+0000 7ff1d95fa740 1 -- 172.21.15.90:0/1043439567 shutdown_connections 2024-04-02T14:44:19.265 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.268+0000 7ff1d95fa740 1 --2- 172.21.15.90:0/1043439567 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x565059af63b0 0x565059af87a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:19.265 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.268+0000 7ff1d95fa740 1 --2- 172.21.15.90:0/1043439567 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x565059aed7d0 0x565059af5e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:19.265 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.268+0000 7ff1d95fa740 1 --2- 172.21.15.90:0/1043439567 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x565059aeceb0 0x565059aed290 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:19.265 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.268+0000 7ff1d95fa740 1 -- 172.21.15.90:0/1043439567 >> 172.21.15.90:0/1043439567 conn(0x565059ae75e0 msgr2=0x565059ae79c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:44:19.267 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.270+0000 7ff1ccc00640 1 --2- 172.21.15.90:0/2602132970 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7ff1b40a40c0 0x7ff1b40a6580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:19.267 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.270+0000 7ff1ccc00640 1 -- 172.21.15.90:0/2602132970 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8550+0+0 (secure 0 0 0) 0x7ff1b80cf640 con 0x7ff1d0073f30 2024-04-02T14:44:19.268 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.271+0000 7ff1cf400640 1 --2- 172.21.15.90:0/2602132970 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7ff1b40a40c0 0x7ff1b40a6580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:19.269 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.272+0000 7ff1cf400640 1 --2- 172.21.15.90:0/2602132970 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7ff1b40a40c0 0x7ff1b40a6580 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7ff1c0000c00 tx=0x7ff1c0005f10 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:44:19.272 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.274+0000 7ff1d95fa740 1 -- 172.21.15.90:0/1043439567 shutdown_connections 2024-04-02T14:44:19.272 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.275+0000 7ff1d95fa740 1 -- 172.21.15.90:0/1043439567 wait complete. 2024-04-02T14:44:19.272 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.275+0000 7ff1d95fa740 1 -- start start 2024-04-02T14:44:19.273 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.276+0000 7ff1d95fa740 1 Processor -- start 2024-04-02T14:44:19.275 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.277+0000 7ff1d95fa740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x565059aeceb0 0x565059af3380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:19.275 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.277+0000 7ff1d95fa740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x565059aed7d0 0x565059af38c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:19.275 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.277+0000 7ff1d95fa740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x565059af63b0 0x565059af3e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:19.275 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.277+0000 7ff1d95fa740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x565059af91a0 con 0x565059aed7d0 2024-04-02T14:44:19.275 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.277+0000 7ff1d95fa740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x565059a0d660 con 0x565059aeceb0 2024-04-02T14:44:19.275 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.277+0000 7ff1d95fa740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x565059af9320 con 0x565059af63b0 2024-04-02T14:44:19.275 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.277+0000 7ff1d95fa740 10 client.0 _ll_register_callbacks cb 0x565059a0d050 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:44:19.275 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.277+0000 7ff1cfe00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x565059aeceb0 0x565059af3380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:19.275 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.277+0000 7ff1cfe00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x565059aeceb0 0x565059af3380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:38770/0 (socket says 172.21.15.90:38770) 2024-04-02T14:44:19.275 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.277+0000 7ff1cfe00640 1 -- 172.21.15.90:0/4085421044 learned_addr learned my addr 172.21.15.90:0/4085421044 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:44:19.276 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.278+0000 7ff1cfe00640 1 -- 172.21.15.90:0/4085421044 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x565059af63b0 msgr2=0x565059af3e00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-02T14:44:19.276 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.278+0000 7ff1cfe00640 1 --2- 172.21.15.90:0/4085421044 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x565059af63b0 0x565059af3e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:19.276 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.278+0000 7ff1cfe00640 1 -- 172.21.15.90:0/4085421044 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x565059aed7d0 msgr2=0x565059af38c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:44:19.276 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.278+0000 7ff1cf400640 1 --2- 172.21.15.90:0/4085421044 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x565059aed7d0 0x565059af38c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:19.277 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.278+0000 7ff1cfe00640 1 --2- 172.21.15.90:0/4085421044 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x565059aed7d0 0x565059af38c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:19.277 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.278+0000 7ff1cfe00640 1 -- 172.21.15.90:0/4085421044 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x565059a1ce80 con 0x565059aeceb0 2024-04-02T14:44:19.277 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.278+0000 7ff1cf400640 1 --2- 172.21.15.90:0/4085421044 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x565059aed7d0 0x565059af38c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:44:19.277 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.279+0000 7ff1cfe00640 1 --2- 172.21.15.90:0/4085421044 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x565059aeceb0 0x565059af3380 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7ff1b80cce80 tx=0x7ff1b80174f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3606a82f85fda6f2 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:44:19.277 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.279+0000 7ff1bd600640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:44:19.277 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.279+0000 7ff1bd600640 1 -- 172.21.15.90:0/4085421044 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1b801e070 con 0x565059aeceb0 2024-04-02T14:44:19.277 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.279+0000 7ff1bd600640 1 -- 172.21.15.90:0/4085421044 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8550+0+0 (secure 0 0 0) 0x7ff1b80ccf90 con 0x565059aeceb0 2024-04-02T14:44:19.277 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.280+0000 7ff1d95fa740 1 -- 172.21.15.90:0/4085421044 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff1c800d770 con 0x565059aeceb0 2024-04-02T14:44:19.278 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.281+0000 7ff1bd600640 1 -- 172.21.15.90:0/4085421044 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7ff1b80cd190 con 0x565059aeceb0 2024-04-02T14:44:19.278 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.281+0000 7ff1d95fa740 10 client.28343 fetch_fsmap learned FSMap version 27 2024-04-02T14:44:19.278 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.281+0000 7ff1d95fa740 1 -- 172.21.15.90:0/4085421044 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x565059766210 con 0x565059aeceb0 2024-04-02T14:44:19.279 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.281+0000 7ff1bd600640 1 -- 172.21.15.90:0/4085421044 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff1b80cd190 con 0x565059aeceb0 2024-04-02T14:44:19.280 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.283+0000 7ff1bd600640 1 -- 172.21.15.90:0/4085421044 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7ff1b809aac0 con 0x565059aeceb0 2024-04-02T14:44:19.280 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.283+0000 7ff1d95fa740 10 client.28343 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:44:19.280 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.283+0000 7ff1d95fa740 10 client.28343 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-02T14:44:19.280 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.283+0000 7ff1d95fa740 20 client.28343 populate_metadata read hostname 'smithi090' 2024-04-02T14:44:19.281 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.283+0000 7ff1d95fa740 1 --2- 172.21.15.90:0/4085421044 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x565059b066e0 0x565059b08ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:19.281 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.283+0000 7ff1d95fa740 4 client.28343 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:44:19.281 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.283+0000 7ff1d95fa740 1 -- 172.21.15.90:0/4085421044 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x565059a01cd0 con 0x565059b066e0 2024-04-02T14:44:19.283 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.285+0000 7ff1cf400640 1 --2- 172.21.15.90:0/4085421044 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x565059b066e0 0x565059b08ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:19.283 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.285+0000 7ff1cf400640 1 --2- 172.21.15.90:0/4085421044 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x565059b066e0 0x565059b08ad0 crc :-1 s=READY pgs=250 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8b929e455f61ab4d server_cookie=65bd85d56446a0e in_seq=0 out_seq=0 2024-04-02T14:44:19.283 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.286+0000 7ff1bd600640 10 client.28343 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:44:19.284 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.286+0000 7ff1bd600640 1 -- 172.21.15.90:0/4085421044 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x565059a01cd0 con 0x565059b066e0 2024-04-02T14:44:19.284 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.287+0000 7ff1bd600640 10 client.28343 handle_command_reply: tid=0 2024-04-02T14:44:19.284 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.287+0000 7ff1d95fa740 1 client.28343 shutdown 2024-04-02T14:44:19.284 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.287+0000 7ff1d95fa740 1 -- 172.21.15.90:0/4085421044 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x565059aeceb0 msgr2=0x565059af3380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:19.284 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.287+0000 7ff1d95fa740 1 --2- 172.21.15.90:0/4085421044 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x565059aeceb0 0x565059af3380 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7ff1b80cce80 tx=0x7ff1b80174f0 comp rx=0 tx=0).stop 2024-04-02T14:44:19.285 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.288+0000 7ff1d95fa740 1 -- 172.21.15.90:0/4085421044 shutdown_connections 2024-04-02T14:44:19.285 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.288+0000 7ff1d95fa740 1 --2- 172.21.15.90:0/4085421044 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x565059b066e0 0x565059b08ad0 crc :-1 s=READY pgs=250 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:19.285 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.288+0000 7ff1d95fa740 1 --2- 172.21.15.90:0/4085421044 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x565059af63b0 0x565059af3e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:19.285 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.288+0000 7ff1d95fa740 1 --2- 172.21.15.90:0/4085421044 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x565059aed7d0 0x565059af38c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:19.285 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.288+0000 7ff1d95fa740 1 --2- 172.21.15.90:0/4085421044 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x565059aeceb0 0x565059af3380 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:19.285 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.288+0000 7ff1bd600640 0 client.28343 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:44:19.285 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.288+0000 7ff1d95fa740 1 -- 172.21.15.90:0/4085421044 >> 172.21.15.90:0/4085421044 conn(0x565059ae75e0 msgr2=0x565059ae9330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:44:19.286 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.288+0000 7ff1d95fa740 1 -- 172.21.15.90:0/4085421044 shutdown_connections 2024-04-02T14:44:19.286 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.289+0000 7ff1d95fa740 1 -- 172.21.15.90:0/4085421044 wait complete. 2024-04-02T14:44:19.286 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.289+0000 7ff1d95fa740 20 client.28343 trim_cache size 0 max 16384 2024-04-02T14:44:19.324 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.327+0000 7ff1d95fa740 1 Processor -- start 2024-04-02T14:44:19.325 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.327+0000 7ff1d95fa740 1 -- start start 2024-04-02T14:44:19.325 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.327+0000 7ff1d95fa740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x565059aeceb0 0x565059af5ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:19.325 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.327+0000 7ff1d95fa740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x565059aed7d0 0x565059b020c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:19.325 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.327+0000 7ff1d95fa740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x565059af63b0 0x565059ae6640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:19.325 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.327+0000 7ff1d95fa740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7ff1ac00e890 con 0x565059aeceb0 2024-04-02T14:44:19.325 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.327+0000 7ff1d95fa740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7ff1ac0103d0 con 0x565059af63b0 2024-04-02T14:44:19.325 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.327+0000 7ff1d95fa740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7ff1ac00ef60 con 0x565059aed7d0 2024-04-02T14:44:19.326 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.328+0000 7ff1cf400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x565059aed7d0 0x565059b020c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:19.326 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.328+0000 7ff1d4a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x565059af63b0 0x565059ae6640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:19.326 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.328+0000 7ff1cf400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x565059aed7d0 0x565059b020c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:39176/0 (socket says 172.21.15.90:39176) 2024-04-02T14:44:19.326 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.328+0000 7ff1cf400640 1 -- 172.21.15.90:0/352839870 learned_addr learned my addr 172.21.15.90:0/352839870 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:44:19.326 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.329+0000 7ff1cfe00640 1 --2- 172.21.15.90:0/352839870 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x565059aeceb0 0x565059af5ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:19.326 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.329+0000 7ff1d4a00640 1 -- 172.21.15.90:0/352839870 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x565059aed7d0 msgr2=0x565059b020c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:44:19.327 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.329+0000 7ff1d4a00640 1 --2- 172.21.15.90:0/352839870 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x565059aed7d0 0x565059b020c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:19.327 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.329+0000 7ff1d4a00640 1 -- 172.21.15.90:0/352839870 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x565059aeceb0 msgr2=0x565059af5ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:44:19.327 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.329+0000 7ff1d4a00640 1 --2- 172.21.15.90:0/352839870 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x565059aeceb0 0x565059af5ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:19.327 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.329+0000 7ff1d4a00640 1 -- 172.21.15.90:0/352839870 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x565059af2330 con 0x565059af63b0 2024-04-02T14:44:19.327 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.330+0000 7ff1d4a00640 1 --2- 172.21.15.90:0/352839870 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x565059af63b0 0x565059ae6640 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7ff1c800ed60 tx=0x7ff1c8005c80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=86bfca8998173e02 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:44:19.328 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.330+0000 7ff1bfe00640 1 -- 172.21.15.90:0/352839870 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1c8026020 con 0x565059af63b0 2024-04-02T14:44:19.328 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.330+0000 7ff1bfe00640 1 -- 172.21.15.90:0/352839870 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff1c8003970 con 0x565059af63b0 2024-04-02T14:44:19.328 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.330+0000 7ff1bfe00640 1 -- 172.21.15.90:0/352839870 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1c8026020 con 0x565059af63b0 2024-04-02T14:44:19.329 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.331+0000 7ff1d95fa740 1 -- 172.21.15.90:0/352839870 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x565059af63b0 msgr2=0x565059ae6640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:19.329 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.331+0000 7ff1d95fa740 1 --2- 172.21.15.90:0/352839870 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x565059af63b0 0x565059ae6640 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7ff1c800ed60 tx=0x7ff1c8005c80 comp rx=0 tx=0).stop 2024-04-02T14:44:19.329 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.331+0000 7ff1d95fa740 1 -- 172.21.15.90:0/352839870 shutdown_connections 2024-04-02T14:44:19.329 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.331+0000 7ff1d95fa740 1 --2- 172.21.15.90:0/352839870 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x565059af63b0 0x565059ae6640 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:19.329 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.331+0000 7ff1d95fa740 1 --2- 172.21.15.90:0/352839870 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x565059aed7d0 0x565059b020c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:19.329 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.331+0000 7ff1d95fa740 1 --2- 172.21.15.90:0/352839870 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x565059aeceb0 0x565059af5ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:19.329 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.331+0000 7ff1d95fa740 1 -- 172.21.15.90:0/352839870 >> 172.21.15.90:0/352839870 conn(0x565059ae75e0 msgr2=0x565059ae79c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:44:19.329 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.331+0000 7ff1d95fa740 1 -- 172.21.15.90:0/352839870 shutdown_connections 2024-04-02T14:44:19.329 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.331+0000 7ff1d95fa740 1 -- 172.21.15.90:0/352839870 wait complete. 2024-04-02T14:44:19.329 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.332+0000 7ff1d95fa740 1 -- start start 2024-04-02T14:44:19.330 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.332+0000 7ff1d95fa740 1 Processor -- start 2024-04-02T14:44:19.331 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.332+0000 7ff1d95fa740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x565059aeceb0 0x565059af0180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:19.331 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.332+0000 7ff1d95fa740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x565059aed7d0 0x565059af06c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:19.332 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.332+0000 7ff1d95fa740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x565059af63b0 0x565059af0c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:19.332 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.332+0000 7ff1d95fa740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7ff1ac00fd00 con 0x565059aed7d0 2024-04-02T14:44:19.332 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.332+0000 7ff1d95fa740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7ff1ac00f630 con 0x565059af63b0 2024-04-02T14:44:19.332 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.332+0000 7ff1d95fa740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7ff1ac00c850 con 0x565059aeceb0 2024-04-02T14:44:19.332 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.332+0000 7ff1d95fa740 10 client.0 _ll_register_callbacks cb 0x7ff1b809b1b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:44:19.332 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.333+0000 7ff1cfe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x565059aeceb0 0x565059af0180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:19.332 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.333+0000 7ff1d4a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x565059af63b0 0x565059af0c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:19.332 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.333+0000 7ff1cfe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x565059aeceb0 0x565059af0180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:39192/0 (socket says 172.21.15.90:39192) 2024-04-02T14:44:19.332 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.333+0000 7ff1cfe00640 1 -- 172.21.15.90:0/211635410 learned_addr learned my addr 172.21.15.90:0/211635410 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:44:19.332 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.333+0000 7ff1cf400640 1 --2- 172.21.15.90:0/211635410 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x565059aed7d0 0x565059af06c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:19.332 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.333+0000 7ff1d4a00640 1 -- 172.21.15.90:0/211635410 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x565059aeceb0 msgr2=0x565059af0180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:44:19.332 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.333+0000 7ff1d4a00640 1 --2- 172.21.15.90:0/211635410 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x565059aeceb0 0x565059af0180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:19.332 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.333+0000 7ff1d4a00640 1 -- 172.21.15.90:0/211635410 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x565059aed7d0 msgr2=0x565059af06c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:44:19.333 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.333+0000 7ff1d4a00640 1 --2- 172.21.15.90:0/211635410 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x565059aed7d0 0x565059af06c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:19.333 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.333+0000 7ff1d4a00640 1 -- 172.21.15.90:0/211635410 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x565059832620 con 0x565059af63b0 2024-04-02T14:44:19.333 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.333+0000 7ff1cf400640 1 --2- 172.21.15.90:0/211635410 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x565059aed7d0 0x565059af06c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:44:19.333 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.333+0000 7ff1d4a00640 1 --2- 172.21.15.90:0/211635410 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x565059af63b0 0x565059af0c00 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7ff1c800ed60 tx=0x7ff1c8003d50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=21217bc4b603f995 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:44:19.333 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.333+0000 7ff1bd600640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:44:19.333 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.334+0000 7ff1bd600640 1 -- 172.21.15.90:0/211635410 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1c8035060 con 0x565059af63b0 2024-04-02T14:44:19.333 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.334+0000 7ff1bd600640 1 -- 172.21.15.90:0/211635410 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8550+0+0 (secure 0 0 0) 0x7ff1c800db30 con 0x565059af63b0 2024-04-02T14:44:19.333 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.334+0000 7ff1d95fa740 1 -- 172.21.15.90:0/211635410 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff1c8003970 con 0x565059af63b0 2024-04-02T14:44:19.333 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.334+0000 7ff1bd600640 1 -- 172.21.15.90:0/211635410 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7ff1c8026020 con 0x565059af63b0 2024-04-02T14:44:19.333 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.334+0000 7ff1d95fa740 10 client.28349 fetch_fsmap learned FSMap version 27 2024-04-02T14:44:19.333 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.334+0000 7ff1d95fa740 1 -- 172.21.15.90:0/211635410 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x565059b6ce70 con 0x565059af63b0 2024-04-02T14:44:19.333 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.334+0000 7ff1bd600640 1 -- 172.21.15.90:0/211635410 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff1c8003970 con 0x565059af63b0 2024-04-02T14:44:19.334 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.335+0000 7ff1bd600640 1 -- 172.21.15.90:0/211635410 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7ff1c800b040 con 0x565059af63b0 2024-04-02T14:44:19.334 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.335+0000 7ff1d95fa740 10 client.28349 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:44:19.334 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.335+0000 7ff1d95fa740 10 client.28349 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-02T14:44:19.334 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.335+0000 7ff1d95fa740 20 client.28349 populate_metadata read hostname 'smithi090' 2024-04-02T14:44:19.334 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.335+0000 7ff1d95fa740 1 --2- 172.21.15.90:0/211635410 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x565059b6d360 0x565059b6f780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:19.335 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.335+0000 7ff1d95fa740 4 client.28349 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-02T14:44:19.335 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.335+0000 7ff1d95fa740 1 -- 172.21.15.90:0/211635410 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "flush journal"}) -- 0x565059a5f2b0 con 0x565059b6d360 2024-04-02T14:44:19.335 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.335+0000 7ff1cfe00640 1 --2- 172.21.15.90:0/211635410 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x565059b6d360 0x565059b6f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:19.335 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.336+0000 7ff1cfe00640 1 --2- 172.21.15.90:0/211635410 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x565059b6d360 0x565059b6f780 crc :-1 s=READY pgs=251 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ab640c513d3bbf68 server_cookie=18ddbb6c27010e06 in_seq=0 out_seq=0 2024-04-02T14:44:19.335 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.336+0000 7ff1bd600640 10 client.28349 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:44:19.366 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.368+0000 7ff1bd600640 1 -- 172.21.15.90:0/211635410 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x565059a5f2b0 con 0x565059b6d360 2024-04-02T14:44:19.366 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.368+0000 7ff1bd600640 10 client.28349 handle_command_reply: tid=0 2024-04-02T14:44:19.366 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.368+0000 7ff1d95fa740 1 client.28349 shutdown 2024-04-02T14:44:19.366 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.368+0000 7ff1d95fa740 1 -- 172.21.15.90:0/211635410 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x565059af63b0 msgr2=0x565059af0c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:19.366 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.368+0000 7ff1d95fa740 1 --2- 172.21.15.90:0/211635410 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x565059af63b0 0x565059af0c00 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7ff1c800ed60 tx=0x7ff1c8003d50 comp rx=0 tx=0).stop 2024-04-02T14:44:19.367 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.368+0000 7ff1d95fa740 1 -- 172.21.15.90:0/211635410 shutdown_connections 2024-04-02T14:44:19.367 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.368+0000 7ff1d95fa740 1 --2- 172.21.15.90:0/211635410 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x565059b6d360 0x565059b6f780 crc :-1 s=READY pgs=251 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:19.367 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.369+0000 7ff1d95fa740 1 --2- 172.21.15.90:0/211635410 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x565059af63b0 0x565059af0c00 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:19.367 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.369+0000 7ff1d95fa740 1 --2- 172.21.15.90:0/211635410 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x565059aed7d0 0x565059af06c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:19.367 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.369+0000 7ff1d95fa740 1 --2- 172.21.15.90:0/211635410 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x565059aeceb0 0x565059af0180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:19.367 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.369+0000 7ff1d95fa740 1 -- 172.21.15.90:0/211635410 >> 172.21.15.90:0/211635410 conn(0x565059ae75e0 msgr2=0x565059af3bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:44:19.367 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.369+0000 7ff1bd600640 0 client.28349 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:44:19.367 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.369+0000 7ff1d95fa740 1 -- 172.21.15.90:0/211635410 shutdown_connections 2024-04-02T14:44:19.367 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.369+0000 7ff1d95fa740 1 -- 172.21.15.90:0/211635410 wait complete. 2024-04-02T14:44:19.367 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.369+0000 7ff1d95fa740 20 client.28349 trim_cache size 0 max 16384 2024-04-02T14:44:19.367 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:{ 2024-04-02T14:44:19.368 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "message": "", 2024-04-02T14:44:19.368 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "return_code": 0 2024-04-02T14:44:19.368 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:} 2024-04-02T14:44:19.368 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.369+0000 7ff1d6a00640 1 -- 172.21.15.90:0/2602132970 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7ff1b40a40c0 msgr2=0x7ff1b40a6580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:19.368 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.370+0000 7ff1d6a00640 1 --2- 172.21.15.90:0/2602132970 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7ff1b40a40c0 0x7ff1b40a6580 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7ff1c0000c00 tx=0x7ff1c0005f10 comp rx=0 tx=0).stop 2024-04-02T14:44:19.369 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.370+0000 7ff1d6a00640 1 -- 172.21.15.90:0/2602132970 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff1d0073f30 msgr2=0x7ff1d0107440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:19.369 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.370+0000 7ff1d6a00640 1 --2- 172.21.15.90:0/2602132970 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff1d0073f30 0x7ff1d0107440 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7ff1b800b4a0 tx=0x7ff1b800b960 comp rx=0 tx=0).stop 2024-04-02T14:44:19.369 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.370+0000 7ff1d6a00640 1 -- 172.21.15.90:0/2602132970 shutdown_connections 2024-04-02T14:44:19.369 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.370+0000 7ff1d6a00640 1 --2- 172.21.15.90:0/2602132970 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7ff1b40a40c0 0x7ff1b40a6580 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:19.369 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.370+0000 7ff1d6a00640 1 --2- 172.21.15.90:0/2602132970 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff1d010e310 0x7ff1d0102550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:19.369 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.370+0000 7ff1d6a00640 1 --2- 172.21.15.90:0/2602132970 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff1d0074850 0x7ff1d0107980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:19.369 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.370+0000 7ff1d6a00640 1 --2- 172.21.15.90:0/2602132970 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff1d0073f30 0x7ff1d0107440 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:19.369 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.370+0000 7ff1d6a00640 1 -- 172.21.15.90:0/2602132970 >> 172.21.15.90:0/2602132970 conn(0x7ff1d00fd5c0 msgr2=0x7ff1d00fe290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:44:19.369 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.370+0000 7ff1d6a00640 1 -- 172.21.15.90:0/2602132970 shutdown_connections 2024-04-02T14:44:19.369 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:19.370+0000 7ff1d6a00640 1 -- 172.21.15.90:0/2602132970 wait complete. 2024-04-02T14:44:19.784 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-02T14:44:19.785 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-02T14:44:20.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.045+0000 7fdc18a00640 1 -- 172.21.15.90:0/3917214342 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdc140747c0 msgr2=0x7fdc14112430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:20.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.045+0000 7fdc18a00640 1 --2- 172.21.15.90:0/3917214342 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdc140747c0 0x7fdc14112430 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fdc00009860 tx=0x7fdc0002bc50 comp rx=0 tx=0).stop 2024-04-02T14:44:20.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.046+0000 7fdc18a00640 1 -- 172.21.15.90:0/3917214342 shutdown_connections 2024-04-02T14:44:20.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.046+0000 7fdc18a00640 1 --2- 172.21.15.90:0/3917214342 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdc14112970 0x7fdc14114d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:20.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.046+0000 7fdc18a00640 1 --2- 172.21.15.90:0/3917214342 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdc140747c0 0x7fdc14112430 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:20.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.046+0000 7fdc18a00640 1 --2- 172.21.15.90:0/3917214342 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdc14073ea0 0x7fdc14074280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:20.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.046+0000 7fdc18a00640 1 -- 172.21.15.90:0/3917214342 >> 172.21.15.90:0/3917214342 conn(0x7fdc140fd470 msgr2=0x7fdc140ff890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:44:20.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.047+0000 7fdc18a00640 1 -- 172.21.15.90:0/3917214342 shutdown_connections 2024-04-02T14:44:20.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.047+0000 7fdc18a00640 1 -- 172.21.15.90:0/3917214342 wait complete. 2024-04-02T14:44:20.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.048+0000 7fdc18a00640 1 Processor -- start 2024-04-02T14:44:20.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.051+0000 7fdc18a00640 1 -- start start 2024-04-02T14:44:20.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.052+0000 7fdc18a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdc14073ea0 0x7fdc141a4590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:20.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.052+0000 7fdc18a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdc140747c0 0x7fdc141a4ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:20.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.052+0000 7fdc18a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdc14112970 0x7fdc1419e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:20.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.052+0000 7fdc18a00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fdc14118340 con 0x7fdc140747c0 2024-04-02T14:44:20.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.052+0000 7fdc18a00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fdc141181c0 con 0x7fdc14112970 2024-04-02T14:44:20.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.052+0000 7fdc18a00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fdc141184c0 con 0x7fdc14073ea0 2024-04-02T14:44:20.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.053+0000 7fdc11e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdc140747c0 0x7fdc141a4ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:20.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.053+0000 7fdc13200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdc14112970 0x7fdc1419e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:20.052 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.053+0000 7fdc13200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdc14112970 0x7fdc1419e6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:38818/0 (socket says 172.21.15.90:38818) 2024-04-02T14:44:20.052 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.053+0000 7fdc13200640 1 -- 172.21.15.90:0/3660767760 learned_addr learned my addr 172.21.15.90:0/3660767760 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:44:20.052 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.055+0000 7fdc12800640 1 --2- 172.21.15.90:0/3660767760 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdc14073ea0 0x7fdc141a4590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:20.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.055+0000 7fdc12800640 1 -- 172.21.15.90:0/3660767760 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdc14112970 msgr2=0x7fdc1419e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:20.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.055+0000 7fdc12800640 1 --2- 172.21.15.90:0/3660767760 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdc14112970 0x7fdc1419e6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:20.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.055+0000 7fdc12800640 1 -- 172.21.15.90:0/3660767760 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdc140747c0 msgr2=0x7fdc141a4ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:20.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.055+0000 7fdc12800640 1 --2- 172.21.15.90:0/3660767760 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdc140747c0 0x7fdc141a4ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:20.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.055+0000 7fdc12800640 1 -- 172.21.15.90:0/3660767760 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc1419ef60 con 0x7fdc14073ea0 2024-04-02T14:44:20.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.056+0000 7fdc12800640 1 --2- 172.21.15.90:0/3660767760 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdc14073ea0 0x7fdc141a4590 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7fdc0400c970 tx=0x7fdc0400ce30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:44:20.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.057+0000 7fdc0b400640 1 -- 172.21.15.90:0/3660767760 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc04007bb0 con 0x7fdc14073ea0 2024-04-02T14:44:20.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.057+0000 7fdc0b400640 1 -- 172.21.15.90:0/3660767760 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdc04007d50 con 0x7fdc14073ea0 2024-04-02T14:44:20.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.057+0000 7fdc18a00640 1 -- 172.21.15.90:0/3660767760 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc1419f250 con 0x7fdc14073ea0 2024-04-02T14:44:20.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.057+0000 7fdc0b400640 1 -- 172.21.15.90:0/3660767760 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc04018820 con 0x7fdc14073ea0 2024-04-02T14:44:20.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.057+0000 7fdc18a00640 1 -- 172.21.15.90:0/3660767760 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc141ab510 con 0x7fdc14073ea0 2024-04-02T14:44:20.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.066+0000 7fdc1c22c740 1 Processor -- start 2024-04-02T14:44:20.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.068+0000 7fdc0b400640 1 -- 172.21.15.90:0/3660767760 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fdc040189c0 con 0x7fdc14073ea0 2024-04-02T14:44:20.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.070+0000 7fdc1c22c740 1 -- start start 2024-04-02T14:44:20.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.070+0000 7fdc1c22c740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559fd90bb960 0x559fd90bbd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:20.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.070+0000 7fdc1c22c740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559fd90bc280 0x559fd90c4920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:20.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.070+0000 7fdc1c22c740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559fd90c4e60 0x559fd90c7250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:20.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.070+0000 7fdc1c22c740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x559fd901f770 con 0x559fd90c4e60 2024-04-02T14:44:20.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.070+0000 7fdc1c22c740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x559fd8fc7700 con 0x559fd90bc280 2024-04-02T14:44:20.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.070+0000 7fdc1c22c740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x559fd901b720 con 0x559fd90bb960 2024-04-02T14:44:20.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.071+0000 7fdc11e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559fd90c4e60 0x559fd90c7250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:20.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.071+0000 7fdc11e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559fd90c4e60 0x559fd90c7250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:44830/0 (socket says 172.21.15.90:44830) 2024-04-02T14:44:20.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.071+0000 7fdc11e00640 1 -- 172.21.15.90:0/779863473 learned_addr learned my addr 172.21.15.90:0/779863473 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:44:20.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.072+0000 7fdc11e00640 1 -- 172.21.15.90:0/779863473 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559fd90bb960 msgr2=0x559fd90bbd40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-02T14:44:20.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.072+0000 7fdc11e00640 1 --2- 172.21.15.90:0/779863473 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559fd90bb960 0x559fd90bbd40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:20.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.072+0000 7fdc11e00640 1 -- 172.21.15.90:0/779863473 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559fd90bc280 msgr2=0x559fd90c4920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:44:20.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.072+0000 7fdc11e00640 1 --2- 172.21.15.90:0/779863473 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559fd90bc280 0x559fd90c4920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:20.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.072+0000 7fdc11e00640 1 -- 172.21.15.90:0/779863473 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559fd8dfd7d0 con 0x559fd90c4e60 2024-04-02T14:44:20.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.073+0000 7fdc11e00640 1 --2- 172.21.15.90:0/779863473 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559fd90c4e60 0x559fd90c7250 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fdc00005e00 tx=0x7fdc000095a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c960dda8c78363a server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:44:20.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.073+0000 7fdbff400640 1 -- 172.21.15.90:0/779863473 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc00039050 con 0x559fd90c4e60 2024-04-02T14:44:20.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.074+0000 7fdbff400640 1 -- 172.21.15.90:0/779863473 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdc0003a040 con 0x559fd90c4e60 2024-04-02T14:44:20.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.074+0000 7fdbff400640 1 -- 172.21.15.90:0/779863473 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc000052c0 con 0x559fd90c4e60 2024-04-02T14:44:20.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.074+0000 7fdc0b400640 1 --2- 172.21.15.90:0/3660767760 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fdbf80a3d20 0x7fdbf80a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:20.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.074+0000 7fdc0b400640 1 -- 172.21.15.90:0/3660767760 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8550+0+0 (secure 0 0 0) 0x7fdc040d0800 con 0x7fdc14073ea0 2024-04-02T14:44:20.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.075+0000 7fdc1c22c740 1 -- 172.21.15.90:0/779863473 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559fd90c4e60 msgr2=0x559fd90c7250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:20.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.075+0000 7fdc1c22c740 1 --2- 172.21.15.90:0/779863473 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559fd90c4e60 0x559fd90c7250 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fdc00005e00 tx=0x7fdc000095a0 comp rx=0 tx=0).stop 2024-04-02T14:44:20.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.075+0000 7fdc13200640 1 --2- 172.21.15.90:0/3660767760 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fdbf80a3d20 0x7fdbf80a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:20.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.077+0000 7fdc13200640 1 --2- 172.21.15.90:0/3660767760 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fdbf80a3d20 0x7fdbf80a61e0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fdc0c005c40 tx=0x7fdc0c009720 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:44:20.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.079+0000 7fdc1c22c740 1 -- 172.21.15.90:0/779863473 shutdown_connections 2024-04-02T14:44:20.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.079+0000 7fdc1c22c740 1 --2- 172.21.15.90:0/779863473 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559fd90c4e60 0x559fd90c7250 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:20.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.079+0000 7fdc1c22c740 1 --2- 172.21.15.90:0/779863473 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559fd90bc280 0x559fd90c4920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:20.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.079+0000 7fdc1c22c740 1 --2- 172.21.15.90:0/779863473 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559fd90bb960 0x559fd90bbd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:20.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.079+0000 7fdc1c22c740 1 -- 172.21.15.90:0/779863473 >> 172.21.15.90:0/779863473 conn(0x559fd90b6090 msgr2=0x559fd90b6470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:44:20.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.079+0000 7fdc1c22c740 1 -- 172.21.15.90:0/779863473 shutdown_connections 2024-04-02T14:44:20.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.079+0000 7fdc1c22c740 1 -- 172.21.15.90:0/779863473 wait complete. 2024-04-02T14:44:20.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.080+0000 7fdc1c22c740 1 -- start start 2024-04-02T14:44:20.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.080+0000 7fdc1c22c740 1 Processor -- start 2024-04-02T14:44:20.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.080+0000 7fdc1c22c740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559fd90bb960 0x559fd90c1e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:20.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.081+0000 7fdc1c22c740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559fd90bc280 0x559fd90c23a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:20.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.081+0000 7fdc1c22c740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559fd90c4e60 0x559fd90c28e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:20.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.081+0000 7fdc12800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559fd90bb960 0x559fd90c1e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:20.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.081+0000 7fdc12800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559fd90bb960 0x559fd90c1e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:38832/0 (socket says 172.21.15.90:38832) 2024-04-02T14:44:20.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.081+0000 7fdc11e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559fd90bc280 0x559fd90c23a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:20.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.081+0000 7fdc12800640 1 -- 172.21.15.90:0/2313615702 learned_addr learned my addr 172.21.15.90:0/2313615702 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:44:20.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.081+0000 7fdc1c22c740 1 -- 172.21.15.90:0/2313615702 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x559fd90c7bf0 con 0x559fd90c4e60 2024-04-02T14:44:20.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.081+0000 7fdc1c22c740 1 -- 172.21.15.90:0/2313615702 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x559fd8fdaac0 con 0x559fd90bb960 2024-04-02T14:44:20.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.081+0000 7fdc1c22c740 1 -- 172.21.15.90:0/2313615702 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x559fd90c7d70 con 0x559fd90bc280 2024-04-02T14:44:20.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.081+0000 7fdc1c22c740 10 client.0 _ll_register_callbacks cb 0x559fd8fdd180 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:44:20.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.082+0000 7fdc13200640 1 --2- 172.21.15.90:0/2313615702 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559fd90c4e60 0x559fd90c28e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:20.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.083+0000 7fdc12800640 1 -- 172.21.15.90:0/2313615702 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559fd90bc280 msgr2=0x559fd90c23a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:44:20.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.083+0000 7fdc12800640 1 --2- 172.21.15.90:0/2313615702 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559fd90bc280 0x559fd90c23a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:20.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.083+0000 7fdc12800640 1 -- 172.21.15.90:0/2313615702 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559fd90c4e60 msgr2=0x559fd90c28e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:44:20.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.083+0000 7fdc12800640 1 --2- 172.21.15.90:0/2313615702 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559fd90c4e60 0x559fd90c28e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:20.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.083+0000 7fdc12800640 1 -- 172.21.15.90:0/2313615702 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559fd8d333f0 con 0x559fd90bb960 2024-04-02T14:44:20.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.083+0000 7fdc12800640 1 --2- 172.21.15.90:0/2313615702 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559fd90bb960 0x559fd90c1e60 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7fdc040cdfe0 tx=0x7fdc04018cb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ad34774a6a3859e4 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:44:20.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.083+0000 7fdbf3e00640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:44:20.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.084+0000 7fdbf3e00640 1 -- 172.21.15.90:0/2313615702 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc040a0b30 con 0x559fd90bb960 2024-04-02T14:44:20.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.084+0000 7fdbf3e00640 1 -- 172.21.15.90:0/2313615702 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8550+0+0 (secure 0 0 0) 0x7fdc040a4f60 con 0x559fd90bb960 2024-04-02T14:44:20.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.084+0000 7fdc1c22c740 1 -- 172.21.15.90:0/2313615702 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdc0003a040 con 0x559fd90bb960 2024-04-02T14:44:20.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.085+0000 7fdbf3e00640 1 -- 172.21.15.90:0/2313615702 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fdc040a53f0 con 0x559fd90bb960 2024-04-02T14:44:20.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.085+0000 7fdc1c22c740 10 client.28358 fetch_fsmap learned FSMap version 27 2024-04-02T14:44:20.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.085+0000 7fdc1c22c740 1 -- 172.21.15.90:0/2313615702 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x559fd8cf30f0 con 0x559fd90bb960 2024-04-02T14:44:20.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.086+0000 7fdbf3e00640 1 -- 172.21.15.90:0/2313615702 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdc0003a040 con 0x559fd90bb960 2024-04-02T14:44:20.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.087+0000 7fdbf3e00640 1 -- 172.21.15.90:0/2313615702 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fdc04098040 con 0x559fd90bb960 2024-04-02T14:44:20.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.087+0000 7fdc1c22c740 10 client.28358 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:44:20.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.087+0000 7fdc1c22c740 10 client.28358 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:44:20.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.087+0000 7fdc1c22c740 20 client.28358 populate_metadata read hostname 'smithi090' 2024-04-02T14:44:20.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.087+0000 7fdc1c22c740 1 --2- 172.21.15.90:0/2313615702 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x559fd90d4de0 0x559fd90d7240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:20.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.087+0000 7fdc1c22c740 4 client.28358 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:44:20.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.087+0000 7fdc1c22c740 1 -- 172.21.15.90:0/2313615702 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559fd8fc0630 con 0x559fd90d4de0 2024-04-02T14:44:20.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.088+0000 7fdc11e00640 1 --2- 172.21.15.90:0/2313615702 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x559fd90d4de0 0x559fd90d7240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:20.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.089+0000 7fdc11e00640 1 --2- 172.21.15.90:0/2313615702 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x559fd90d4de0 0x559fd90d7240 crc :-1 s=READY pgs=252 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6b16cbd99ee8ef76 server_cookie=bb96168786eca746 in_seq=0 out_seq=0 2024-04-02T14:44:20.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.089+0000 7fdbf3e00640 10 client.28358 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:44:20.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.090+0000 7fdbf3e00640 1 -- 172.21.15.90:0/2313615702 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x559fd8fc0630 con 0x559fd90d4de0 2024-04-02T14:44:20.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.090+0000 7fdbf3e00640 10 client.28358 handle_command_reply: tid=0 2024-04-02T14:44:20.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.090+0000 7fdc1c22c740 1 client.28358 shutdown 2024-04-02T14:44:20.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.091+0000 7fdc1c22c740 1 -- 172.21.15.90:0/2313615702 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559fd90bb960 msgr2=0x559fd90c1e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:20.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.091+0000 7fdc1c22c740 1 --2- 172.21.15.90:0/2313615702 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559fd90bb960 0x559fd90c1e60 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7fdc040cdfe0 tx=0x7fdc04018cb0 comp rx=0 tx=0).stop 2024-04-02T14:44:20.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.091+0000 7fdc1c22c740 1 -- 172.21.15.90:0/2313615702 shutdown_connections 2024-04-02T14:44:20.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.091+0000 7fdc1c22c740 1 --2- 172.21.15.90:0/2313615702 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x559fd90d4de0 0x559fd90d7240 crc :-1 s=READY pgs=252 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:20.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.091+0000 7fdc1c22c740 1 --2- 172.21.15.90:0/2313615702 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559fd90c4e60 0x559fd90c28e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:20.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.091+0000 7fdc1c22c740 1 --2- 172.21.15.90:0/2313615702 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559fd90bc280 0x559fd90c23a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:20.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.091+0000 7fdc1c22c740 1 --2- 172.21.15.90:0/2313615702 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559fd90bb960 0x559fd90c1e60 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:20.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.091+0000 7fdbf3e00640 0 client.28358 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:44:20.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.091+0000 7fdc1c22c740 1 -- 172.21.15.90:0/2313615702 >> 172.21.15.90:0/2313615702 conn(0x559fd90b6090 msgr2=0x559fd90b7de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:44:20.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.091+0000 7fdc1c22c740 1 -- 172.21.15.90:0/2313615702 shutdown_connections 2024-04-02T14:44:20.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.091+0000 7fdc1c22c740 1 -- 172.21.15.90:0/2313615702 wait complete. 2024-04-02T14:44:20.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.091+0000 7fdc1c22c740 20 client.28358 trim_cache size 0 max 16384 2024-04-02T14:44:20.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.131+0000 7fdc1c22c740 1 Processor -- start 2024-04-02T14:44:20.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.131+0000 7fdc1c22c740 1 -- start start 2024-04-02T14:44:20.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.131+0000 7fdc1c22c740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559fd90bb960 0x559fd90d0b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:20.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.131+0000 7fdc1c22c740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559fd90bc280 0x559fd90c0da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:20.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.131+0000 7fdc1c22c740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559fd90c4e60 0x559fd90c2700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:20.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.131+0000 7fdc1c22c740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fdbf400e580 con 0x559fd90bc280 2024-04-02T14:44:20.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.131+0000 7fdc1c22c740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fdbf40100c0 con 0x559fd90bb960 2024-04-02T14:44:20.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.131+0000 7fdc1c22c740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fdbf400ec50 con 0x559fd90c4e60 2024-04-02T14:44:20.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.132+0000 7fdc13200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559fd90c4e60 0x559fd90c2700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:20.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.132+0000 7fdc13200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559fd90c4e60 0x559fd90c2700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:39232/0 (socket says 172.21.15.90:39232) 2024-04-02T14:44:20.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.132+0000 7fdc13200640 1 -- 172.21.15.90:0/1312366236 learned_addr learned my addr 172.21.15.90:0/1312366236 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:44:20.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.132+0000 7fdc12800640 1 --2- 172.21.15.90:0/1312366236 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559fd90bb960 0x559fd90d0b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:20.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.132+0000 7fdc13200640 1 -- 172.21.15.90:0/1312366236 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559fd90bb960 msgr2=0x559fd90d0b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:44:20.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.132+0000 7fdc13200640 1 --2- 172.21.15.90:0/1312366236 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559fd90bb960 0x559fd90d0b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:20.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.132+0000 7fdc13200640 1 -- 172.21.15.90:0/1312366236 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559fd90bc280 msgr2=0x559fd90c0da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:44:20.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.132+0000 7fdc13200640 1 --2- 172.21.15.90:0/1312366236 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559fd90bc280 0x559fd90c0da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:20.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.132+0000 7fdc13200640 1 -- 172.21.15.90:0/1312366236 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559fd90c03b0 con 0x559fd90c4e60 2024-04-02T14:44:20.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.133+0000 7fdc13200640 1 --2- 172.21.15.90:0/1312366236 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559fd90c4e60 0x559fd90c2700 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fdc0c01d7a0 tx=0x7fdc0c014000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=19e04a9bc103c14a server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:44:20.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.134+0000 7fdbfea00640 1 -- 172.21.15.90:0/1312366236 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc0c02b070 con 0x559fd90c4e60 2024-04-02T14:44:20.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.134+0000 7fdbfea00640 1 -- 172.21.15.90:0/1312366236 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdc0c01bdf0 con 0x559fd90c4e60 2024-04-02T14:44:20.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.134+0000 7fdbfea00640 1 -- 172.21.15.90:0/1312366236 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc0c026db0 con 0x559fd90c4e60 2024-04-02T14:44:20.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.135+0000 7fdc1c22c740 1 -- 172.21.15.90:0/1312366236 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559fd90c4e60 msgr2=0x559fd90c2700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:20.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.135+0000 7fdc1c22c740 1 --2- 172.21.15.90:0/1312366236 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559fd90c4e60 0x559fd90c2700 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fdc0c01d7a0 tx=0x7fdc0c014000 comp rx=0 tx=0).stop 2024-04-02T14:44:20.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.135+0000 7fdc1c22c740 1 -- 172.21.15.90:0/1312366236 shutdown_connections 2024-04-02T14:44:20.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.135+0000 7fdc1c22c740 1 --2- 172.21.15.90:0/1312366236 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559fd90c4e60 0x559fd90c2700 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:20.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.135+0000 7fdc1c22c740 1 --2- 172.21.15.90:0/1312366236 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559fd90bc280 0x559fd90c0da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:20.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.135+0000 7fdc1c22c740 1 --2- 172.21.15.90:0/1312366236 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559fd90bb960 0x559fd90d0b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:20.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.135+0000 7fdc1c22c740 1 -- 172.21.15.90:0/1312366236 >> 172.21.15.90:0/1312366236 conn(0x559fd90b6090 msgr2=0x559fd90b5310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:44:20.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.135+0000 7fdc1c22c740 1 -- 172.21.15.90:0/1312366236 shutdown_connections 2024-04-02T14:44:20.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.136+0000 7fdc1c22c740 1 -- 172.21.15.90:0/1312366236 wait complete. 2024-04-02T14:44:20.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.136+0000 7fdc1c22c740 1 -- start start 2024-04-02T14:44:20.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.136+0000 7fdc1c22c740 1 Processor -- start 2024-04-02T14:44:20.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.137+0000 7fdc1c22c740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559fd90bb960 0x559fd90c8e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:20.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.137+0000 7fdc1c22c740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559fd90bc280 0x559fd90c93a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:20.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.137+0000 7fdc1c22c740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559fd90c4e60 0x559fd913a380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:20.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.137+0000 7fdc1c22c740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fdbf400f9f0 con 0x559fd90c4e60 2024-04-02T14:44:20.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.137+0000 7fdc1c22c740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fdbf400f320 con 0x559fd90bb960 2024-04-02T14:44:20.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.137+0000 7fdc1c22c740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fdbf400c610 con 0x559fd90bc280 2024-04-02T14:44:20.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.137+0000 7fdc1c22c740 10 client.0 _ll_register_callbacks cb 0x7fdc04007f00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:44:20.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.137+0000 7fdc11e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559fd90bc280 0x559fd90c93a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:20.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.137+0000 7fdc11e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559fd90bc280 0x559fd90c93a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:39246/0 (socket says 172.21.15.90:39246) 2024-04-02T14:44:20.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.137+0000 7fdc11e00640 1 -- 172.21.15.90:0/647306890 learned_addr learned my addr 172.21.15.90:0/647306890 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:44:20.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.138+0000 7fdc12800640 1 --2- 172.21.15.90:0/647306890 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559fd90bb960 0x559fd90c8e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:20.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.138+0000 7fdc13200640 1 --2- 172.21.15.90:0/647306890 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559fd90c4e60 0x559fd913a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:20.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.138+0000 7fdc11e00640 1 -- 172.21.15.90:0/647306890 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559fd90bb960 msgr2=0x559fd90c8e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:44:20.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.138+0000 7fdc11e00640 1 --2- 172.21.15.90:0/647306890 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559fd90bb960 0x559fd90c8e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:20.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.138+0000 7fdc11e00640 1 -- 172.21.15.90:0/647306890 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559fd90c4e60 msgr2=0x559fd913a380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:44:20.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.138+0000 7fdc11e00640 1 --2- 172.21.15.90:0/647306890 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559fd90c4e60 0x559fd913a380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:20.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.138+0000 7fdc11e00640 1 -- 172.21.15.90:0/647306890 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559fd914b8d0 con 0x559fd90bc280 2024-04-02T14:44:20.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.138+0000 7fdc13200640 1 --2- 172.21.15.90:0/647306890 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559fd90c4e60 0x559fd913a380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:44:20.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.138+0000 7fdc11e00640 1 --2- 172.21.15.90:0/647306890 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559fd90bc280 0x559fd90c93a0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7fdc00005b60 tx=0x7fdc000093a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8203548fc93e30e2 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:44:20.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.138+0000 7fdbf3e00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:44:20.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.139+0000 7fdbf3e00640 1 -- 172.21.15.90:0/647306890 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc0004b070 con 0x559fd90bc280 2024-04-02T14:44:20.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.139+0000 7fdbf3e00640 1 -- 172.21.15.90:0/647306890 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8550+0+0 (secure 0 0 0) 0x7fdc00004030 con 0x559fd90bc280 2024-04-02T14:44:20.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.139+0000 7fdc1c22c740 1 -- 172.21.15.90:0/647306890 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdc0c01bdf0 con 0x559fd90bc280 2024-04-02T14:44:20.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.139+0000 7fdbf3e00640 1 -- 172.21.15.90:0/647306890 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fdc00039040 con 0x559fd90bc280 2024-04-02T14:44:20.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.140+0000 7fdc1c22c740 10 client.28192 fetch_fsmap learned FSMap version 27 2024-04-02T14:44:20.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.140+0000 7fdc1c22c740 1 -- 172.21.15.90:0/647306890 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x559fd914ca80 con 0x559fd90bc280 2024-04-02T14:44:20.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.140+0000 7fdbf3e00640 1 -- 172.21.15.90:0/647306890 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdc0c01bdf0 con 0x559fd90bc280 2024-04-02T14:44:20.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.140+0000 7fdbf3e00640 1 -- 172.21.15.90:0/647306890 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fdc0004c380 con 0x559fd90bc280 2024-04-02T14:44:20.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.141+0000 7fdc1c22c740 10 client.28192 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:44:20.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.141+0000 7fdc1c22c740 10 client.28192 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:44:20.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.141+0000 7fdc1c22c740 20 client.28192 populate_metadata read hostname 'smithi090' 2024-04-02T14:44:20.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.141+0000 7fdc1c22c740 1 --2- 172.21.15.90:0/647306890 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x559fd914cd60 0x559fd914f180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:20.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.141+0000 7fdc1c22c740 4 client.28192 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-02T14:44:20.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.141+0000 7fdc1c22c740 1 -- 172.21.15.90:0/647306890 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "scrub status"}) -- 0x559fd902ccb0 con 0x559fd914cd60 2024-04-02T14:44:20.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.141+0000 7fdc12800640 1 --2- 172.21.15.90:0/647306890 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x559fd914cd60 0x559fd914f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:20.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.142+0000 7fdc12800640 1 --2- 172.21.15.90:0/647306890 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x559fd914cd60 0x559fd914f180 crc :-1 s=READY pgs=253 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ace74c3456dd61a server_cookie=14d5150c8f5dd222 in_seq=0 out_seq=0 2024-04-02T14:44:20.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.142+0000 7fdbf3e00640 10 client.28192 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:44:20.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.142+0000 7fdbf3e00640 1 -- 172.21.15.90:0/647306890 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x559fd902ccb0 con 0x559fd914cd60 2024-04-02T14:44:20.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.142+0000 7fdbf3e00640 10 client.28192 handle_command_reply: tid=0 2024-04-02T14:44:20.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.142+0000 7fdc1c22c740 1 client.28192 shutdown 2024-04-02T14:44:20.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.143+0000 7fdc1c22c740 1 -- 172.21.15.90:0/647306890 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559fd90bc280 msgr2=0x559fd90c93a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:20.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.143+0000 7fdc1c22c740 1 --2- 172.21.15.90:0/647306890 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559fd90bc280 0x559fd90c93a0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7fdc00005b60 tx=0x7fdc000093a0 comp rx=0 tx=0).stop 2024-04-02T14:44:20.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.143+0000 7fdc1c22c740 1 -- 172.21.15.90:0/647306890 shutdown_connections 2024-04-02T14:44:20.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.143+0000 7fdc1c22c740 1 --2- 172.21.15.90:0/647306890 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x559fd914cd60 0x559fd914f180 crc :-1 s=READY pgs=253 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:20.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.143+0000 7fdbf3e00640 0 client.28192 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:44:20.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.143+0000 7fdc1c22c740 1 --2- 172.21.15.90:0/647306890 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559fd90c4e60 0x559fd913a380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:20.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.143+0000 7fdc1c22c740 1 --2- 172.21.15.90:0/647306890 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559fd90bc280 0x559fd90c93a0 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:20.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.143+0000 7fdc1c22c740 1 --2- 172.21.15.90:0/647306890 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559fd90bb960 0x559fd90c8e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:20.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.143+0000 7fdc1c22c740 1 -- 172.21.15.90:0/647306890 >> 172.21.15.90:0/647306890 conn(0x559fd90b6090 msgr2=0x559fd90b5310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:44:20.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.144+0000 7fdc1c22c740 1 -- 172.21.15.90:0/647306890 shutdown_connections 2024-04-02T14:44:20.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.144+0000 7fdc1c22c740 1 -- 172.21.15.90:0/647306890 wait complete. 2024-04-02T14:44:20.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.144+0000 7fdc1c22c740 20 client.28192 trim_cache size 0 max 16384 2024-04-02T14:44:20.142 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-02T14:44:20.143 INFO:teuthology.orchestra.run.smithi090.stdout: "status": "no active scrubs running", 2024-04-02T14:44:20.143 INFO:teuthology.orchestra.run.smithi090.stdout: "scrubs": {} 2024-04-02T14:44:20.143 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-02T14:44:20.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.145+0000 7fdc18a00640 1 -- 172.21.15.90:0/3660767760 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fdbf80a3d20 msgr2=0x7fdbf80a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:20.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.145+0000 7fdc18a00640 1 --2- 172.21.15.90:0/3660767760 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fdbf80a3d20 0x7fdbf80a61e0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fdc0c005c40 tx=0x7fdc0c009720 comp rx=0 tx=0).stop 2024-04-02T14:44:20.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.145+0000 7fdc18a00640 1 -- 172.21.15.90:0/3660767760 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdc14073ea0 msgr2=0x7fdc141a4590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:20.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.145+0000 7fdc18a00640 1 --2- 172.21.15.90:0/3660767760 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdc14073ea0 0x7fdc141a4590 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7fdc0400c970 tx=0x7fdc0400ce30 comp rx=0 tx=0).stop 2024-04-02T14:44:20.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.146+0000 7fdc18a00640 1 -- 172.21.15.90:0/3660767760 shutdown_connections 2024-04-02T14:44:20.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.146+0000 7fdc18a00640 1 --2- 172.21.15.90:0/3660767760 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fdbf80a3d20 0x7fdbf80a61e0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:20.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.146+0000 7fdc18a00640 1 --2- 172.21.15.90:0/3660767760 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdc14112970 0x7fdc1419e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:20.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.146+0000 7fdc18a00640 1 --2- 172.21.15.90:0/3660767760 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdc140747c0 0x7fdc141a4ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:20.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.146+0000 7fdc18a00640 1 --2- 172.21.15.90:0/3660767760 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdc14073ea0 0x7fdc141a4590 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:20.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.146+0000 7fdc18a00640 1 -- 172.21.15.90:0/3660767760 >> 172.21.15.90:0/3660767760 conn(0x7fdc140fd470 msgr2=0x7fdc140fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:44:20.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.146+0000 7fdc18a00640 1 -- 172.21.15.90:0/3660767760 shutdown_connections 2024-04-02T14:44:20.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.146+0000 7fdc18a00640 1 -- 172.21.15.90:0/3660767760 wait complete. 2024-04-02T14:44:20.169 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-02T14:44:20.170 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T14:44:20.432 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:44:20 smithi177 ceph-mon[36431]: pgmap v292: 97 pgs: 97 active+clean; 4.4 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 13 MiB/s wr, 1.06k op/s 2024-04-02T14:44:20.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.436+0000 7f4a41200640 1 -- 172.21.15.90:0/2430818503 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4a3c1048b0 msgr2=0x7f4a3c1088f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:20.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.436+0000 7f4a41200640 1 --2- 172.21.15.90:0/2430818503 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4a3c1048b0 0x7f4a3c1088f0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f4a24009900 tx=0x7f4a2402bc40 comp rx=0 tx=0).stop 2024-04-02T14:44:20.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.436+0000 7f4a41200640 1 -- 172.21.15.90:0/2430818503 shutdown_connections 2024-04-02T14:44:20.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.436+0000 7f4a41200640 1 --2- 172.21.15.90:0/2430818503 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4a3c108fd0 0x7f4a3c110ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:20.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.436+0000 7f4a41200640 1 --2- 172.21.15.90:0/2430818503 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4a3c1048b0 0x7f4a3c1088f0 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:20.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.436+0000 7f4a41200640 1 --2- 172.21.15.90:0/2430818503 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4a3c103f00 0x7f4a3c1042e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:20.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.436+0000 7f4a41200640 1 -- 172.21.15.90:0/2430818503 >> 172.21.15.90:0/2430818503 conn(0x7f4a3c0ff760 msgr2=0x7f4a3c101b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:44:20.434 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:44:20 smithi090 ceph-mon[31674]: pgmap v292: 97 pgs: 97 active+clean; 4.4 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 13 MiB/s wr, 1.06k op/s 2024-04-02T14:44:20.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.436+0000 7f4a41200640 1 -- 172.21.15.90:0/2430818503 shutdown_connections 2024-04-02T14:44:20.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.436+0000 7f4a41200640 1 -- 172.21.15.90:0/2430818503 wait complete. 2024-04-02T14:44:20.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.437+0000 7f4a41200640 1 Processor -- start 2024-04-02T14:44:20.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.438+0000 7f4a41200640 1 -- start start 2024-04-02T14:44:20.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.439+0000 7f4a41200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4a3c103f00 0x7f4a3c19a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:20.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.439+0000 7f4a41200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4a3c108fd0 0x7f4a3c19abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:20.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.439+0000 7f4a41200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4a3c19ef10 0x7f4a3c19f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:20.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.439+0000 7f4a41200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f4a3c1140c0 con 0x7f4a3c103f00 2024-04-02T14:44:20.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.439+0000 7f4a41200640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f4a3c113f40 con 0x7f4a3c19ef10 2024-04-02T14:44:20.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.439+0000 7f4a41200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f4a3c114240 con 0x7f4a3c108fd0 2024-04-02T14:44:20.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.440+0000 7f4a33e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4a3c108fd0 0x7f4a3c19abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:20.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.440+0000 7f4a33e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4a3c108fd0 0x7f4a3c19abe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:39258/0 (socket says 172.21.15.90:39258) 2024-04-02T14:44:20.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.440+0000 7f4a33e00640 1 -- 172.21.15.90:0/1815278239 learned_addr learned my addr 172.21.15.90:0/1815278239 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:44:20.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.440+0000 7f4a3a800640 1 --2- 172.21.15.90:0/1815278239 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4a3c19ef10 0x7f4a3c19f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:20.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.440+0000 7f4a39e00640 1 --2- 172.21.15.90:0/1815278239 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4a3c103f00 0x7f4a3c19a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:20.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.440+0000 7f4a33e00640 1 -- 172.21.15.90:0/1815278239 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4a3c19ef10 msgr2=0x7f4a3c19f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:20.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.440+0000 7f4a33e00640 1 --2- 172.21.15.90:0/1815278239 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4a3c19ef10 0x7f4a3c19f3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:20.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.440+0000 7f4a33e00640 1 -- 172.21.15.90:0/1815278239 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4a3c103f00 msgr2=0x7f4a3c19a6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:20.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.440+0000 7f4a33e00640 1 --2- 172.21.15.90:0/1815278239 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4a3c103f00 0x7f4a3c19a6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:20.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.440+0000 7f4a33e00640 1 -- 172.21.15.90:0/1815278239 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a3c19fa00 con 0x7f4a3c108fd0 2024-04-02T14:44:20.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.441+0000 7f4a33e00640 1 --2- 172.21.15.90:0/1815278239 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4a3c108fd0 0x7f4a3c19abe0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f4a24036040 tx=0x7f4a2402cc00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:44:20.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.441+0000 7f4a32a00640 1 -- 172.21.15.90:0/1815278239 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a24041070 con 0x7f4a3c108fd0 2024-04-02T14:44:20.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.441+0000 7f4a32a00640 1 -- 172.21.15.90:0/1815278239 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4a240054e0 con 0x7f4a3c108fd0 2024-04-02T14:44:20.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.441+0000 7f4a41200640 1 -- 172.21.15.90:0/1815278239 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a3c19fc90 con 0x7f4a3c108fd0 2024-04-02T14:44:20.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.441+0000 7f4a32a00640 1 -- 172.21.15.90:0/1815278239 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a24003e30 con 0x7f4a3c108fd0 2024-04-02T14:44:20.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.441+0000 7f4a41200640 1 -- 172.21.15.90:0/1815278239 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a3c0761d0 con 0x7f4a3c108fd0 2024-04-02T14:44:20.440 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.442+0000 7f4a41200640 1 -- 172.21.15.90:0/1815278239 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4a3c068d40 con 0x7f4a3c108fd0 2024-04-02T14:44:20.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.445+0000 7f4a32a00640 1 -- 172.21.15.90:0/1815278239 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f4a24044070 con 0x7f4a3c108fd0 2024-04-02T14:44:20.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.447+0000 7f4a32a00640 1 --2- 172.21.15.90:0/1815278239 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f4a200a3d20 0x7f4a200a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:20.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.448+0000 7f4a32a00640 1 -- 172.21.15.90:0/1815278239 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8550+0+0 (secure 0 0 0) 0x7f4a240f25e0 con 0x7f4a3c108fd0 2024-04-02T14:44:20.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.448+0000 7f4a39e00640 1 --2- 172.21.15.90:0/1815278239 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f4a200a3d20 0x7f4a200a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:20.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.450+0000 7f4a39e00640 1 --2- 172.21.15.90:0/1815278239 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f4a200a3d20 0x7f4a200a61e0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f4a2c0095f0 tx=0x7f4a2c005f90 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:44:20.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.452+0000 7f4a32a00640 1 -- 172.21.15.90:0/1815278239 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7f4a240be720 con 0x7f4a3c108fd0 2024-04-02T14:44:20.606 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:44:20 smithi092 ceph-mon[36497]: pgmap v292: 97 pgs: 97 active+clean; 4.4 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 13 MiB/s wr, 1.06k op/s 2024-04-02T14:44:20.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.848+0000 7f4a41200640 1 -- 172.21.15.90:0/1815278239 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4a3c1048b0 con 0x7f4a3c108fd0 2024-04-02T14:44:20.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.849+0000 7f4a32a00640 1 -- 172.21.15.90:0/1815278239 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11927 (secure 0 0 0) 0x7f4a240c2a20 con 0x7f4a3c108fd0 2024-04-02T14:44:20.847 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-02T14:44:20.847 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14412,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6837/2852160309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6836","nonce":2852160309},{"type":"v1","addr":"172.21.15.90:6837","nonce":2852160309}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14478,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6841/1919376858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6840","nonce":1919376858},{"type":"v1","addr":"172.21.15.90:6841","nonce":1919376858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-02T14:32:22.319083+0000","modified":"2024-04-02T14:34:52.266152+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24590,"mds_1":24560,"mds_2":24547,"mds_3":24539,"mds_4":24523},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24454":{"gid":24454,"name":"l","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6833/1564319431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":1564319431},{"type":"v1","addr":"172.21.15.177:6833","nonce":1564319431}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.92:6833/1570337372","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":1570337372},{"type":"v1","addr":"172.21.15.92:6833","nonce":1570337372}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"j","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.90:6835/1277124891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6834","nonce":1277124891},{"type":"v1","addr":"172.21.15.90:6835","nonce":1277124891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.92:6835/860721164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":860721164},{"type":"v1","addr":"172.21.15.92:6835","nonce":860721164}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24511":{"gid":24511,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6835/3177244988","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":3177244988},{"type":"v1","addr":"172.21.15.177:6835","nonce":3177244988}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"c","rank":4,"incarnation":25,"state":"up:active","state_seq":9,"addr":"172.21.15.177:6837/4223414858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":4223414858},{"type":"v1","addr":"172.21.15.177:6837","nonce":4223414858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24539":{"gid":24539,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":8,"addr":"172.21.15.92:6837/1270635939","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":1270635939},{"type":"v1","addr":"172.21.15.92:6837","nonce":1270635939}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24547":{"gid":24547,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":7,"addr":"172.21.15.177:6839/664895663","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":664895663},{"type":"v1","addr":"172.21.15.177:6839","nonce":664895663}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"g","rank":1,"incarnation":19,"state":"up:active","state_seq":5,"addr":"172.21.15.90:6839/1498915870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6838","nonce":1498915870},{"type":"v1","addr":"172.21.15.90:6839","nonce":1498915870}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24590":{"gid":24590,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.92:6839/1054888219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":1054888219},{"type":"v1","addr":"172.21.15.92:6839","nonce":1054888219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24590,"qdb_cluster":[24547,24590,24523,24539,24560]},"id":1}]} 2024-04-02T14:44:20.848 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 27 2024-04-02T14:44:20.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.852+0000 7f4a41200640 1 -- 172.21.15.90:0/1815278239 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f4a200a3d20 msgr2=0x7f4a200a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:20.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.852+0000 7f4a41200640 1 --2- 172.21.15.90:0/1815278239 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f4a200a3d20 0x7f4a200a61e0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f4a2c0095f0 tx=0x7f4a2c005f90 comp rx=0 tx=0).stop 2024-04-02T14:44:20.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.852+0000 7f4a41200640 1 -- 172.21.15.90:0/1815278239 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4a3c108fd0 msgr2=0x7f4a3c19abe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:20.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.852+0000 7f4a41200640 1 --2- 172.21.15.90:0/1815278239 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4a3c108fd0 0x7f4a3c19abe0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f4a24036040 tx=0x7f4a2402cc00 comp rx=0 tx=0).stop 2024-04-02T14:44:20.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.852+0000 7f4a41200640 1 -- 172.21.15.90:0/1815278239 shutdown_connections 2024-04-02T14:44:20.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.852+0000 7f4a41200640 1 --2- 172.21.15.90:0/1815278239 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f4a200a3d20 0x7f4a200a61e0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:20.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.852+0000 7f4a41200640 1 --2- 172.21.15.90:0/1815278239 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4a3c19ef10 0x7f4a3c19f3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:20.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.852+0000 7f4a41200640 1 --2- 172.21.15.90:0/1815278239 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4a3c108fd0 0x7f4a3c19abe0 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:20.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.853+0000 7f4a41200640 1 --2- 172.21.15.90:0/1815278239 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4a3c103f00 0x7f4a3c19a6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:20.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.853+0000 7f4a41200640 1 -- 172.21.15.90:0/1815278239 >> 172.21.15.90:0/1815278239 conn(0x7f4a3c0ff760 msgr2=0x7f4a3c100a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:44:20.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.853+0000 7f4a41200640 1 -- 172.21.15.90:0/1815278239 shutdown_connections 2024-04-02T14:44:20.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:20.853+0000 7f4a41200640 1 -- 172.21.15.90:0/1815278239 wait complete. 2024-04-02T14:44:20.879 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-04-02T14:44:21.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.130+0000 7fd272c00640 1 -- 172.21.15.90:0/418645140 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd2640099e0 msgr2=0x7fd2640a6650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:21.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.130+0000 7fd272c00640 1 --2- 172.21.15.90:0/418645140 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd2640099e0 0x7fd2640a6650 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7fd26c067910 tx=0x7fd26c097a90 comp rx=0 tx=0).stop 2024-04-02T14:44:21.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.131+0000 7fd272c00640 1 -- 172.21.15.90:0/418645140 shutdown_connections 2024-04-02T14:44:21.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.131+0000 7fd272c00640 1 --2- 172.21.15.90:0/418645140 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd2640a6b90 0x7fd2640a8f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:21.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.131+0000 7fd272c00640 1 --2- 172.21.15.90:0/418645140 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd2640099e0 0x7fd2640a6650 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:21.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.131+0000 7fd272c00640 1 --2- 172.21.15.90:0/418645140 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd2640090c0 0x7fd2640094a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:21.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.131+0000 7fd272c00640 1 -- 172.21.15.90:0/418645140 >> 172.21.15.90:0/418645140 conn(0x7fd2640939c0 msgr2=0x7fd264095de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:44:21.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.131+0000 7fd272c00640 1 -- 172.21.15.90:0/418645140 shutdown_connections 2024-04-02T14:44:21.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.131+0000 7fd272c00640 1 -- 172.21.15.90:0/418645140 wait complete. 2024-04-02T14:44:21.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.133+0000 7fd272c00640 1 Processor -- start 2024-04-02T14:44:21.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.133+0000 7fd272c00640 1 -- start start 2024-04-02T14:44:21.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.134+0000 7fd272c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd2640090c0 0x7fd2641388f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:21.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.134+0000 7fd272c00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd2640099e0 0x7fd264138e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:21.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.134+0000 7fd272c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd2640a6b90 0x7fd264132aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:21.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.134+0000 7fd272c00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fd2640ac700 con 0x7fd2640090c0 2024-04-02T14:44:21.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.134+0000 7fd272c00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fd2640ac580 con 0x7fd2640099e0 2024-04-02T14:44:21.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.134+0000 7fd272c00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fd2640ac880 con 0x7fd2640a6b90 2024-04-02T14:44:21.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.134+0000 7fd272200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd2640a6b90 0x7fd264132aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:21.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.134+0000 7fd272200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd2640a6b90 0x7fd264132aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:39272/0 (socket says 172.21.15.90:39272) 2024-04-02T14:44:21.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.134+0000 7fd272200640 1 -- 172.21.15.90:0/3663900746 learned_addr learned my addr 172.21.15.90:0/3663900746 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:44:21.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.134+0000 7fd270e00640 1 --2- 172.21.15.90:0/3663900746 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd2640099e0 0x7fd264138e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:21.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.135+0000 7fd270e00640 1 -- 172.21.15.90:0/3663900746 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd2640a6b90 msgr2=0x7fd264132aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:21.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.135+0000 7fd270e00640 1 --2- 172.21.15.90:0/3663900746 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd2640a6b90 0x7fd264132aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:21.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.135+0000 7fd270e00640 1 -- 172.21.15.90:0/3663900746 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd2640090c0 msgr2=0x7fd2641388f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:21.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.135+0000 7fd270e00640 1 --2- 172.21.15.90:0/3663900746 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd2640090c0 0x7fd2641388f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:21.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.135+0000 7fd270e00640 1 -- 172.21.15.90:0/3663900746 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd264133220 con 0x7fd2640099e0 2024-04-02T14:44:21.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.135+0000 7fd272200640 1 --2- 172.21.15.90:0/3663900746 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd2640a6b90 0x7fd264132aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:44:21.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.135+0000 7fd270e00640 1 --2- 172.21.15.90:0/3663900746 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd2640099e0 0x7fd264138e30 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7fd26c0673c0 tx=0x7fd26c09a740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:44:21.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.136+0000 7fd26a000640 1 -- 172.21.15.90:0/3663900746 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd26c09e070 con 0x7fd2640099e0 2024-04-02T14:44:21.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.136+0000 7fd272c00640 1 -- 172.21.15.90:0/3663900746 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2641334b0 con 0x7fd2640099e0 2024-04-02T14:44:21.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.136+0000 7fd272c00640 1 -- 172.21.15.90:0/3663900746 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd26413f7b0 con 0x7fd2640099e0 2024-04-02T14:44:21.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.137+0000 7fd26a000640 1 -- 172.21.15.90:0/3663900746 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd26c099a60 con 0x7fd2640099e0 2024-04-02T14:44:21.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.137+0000 7fd26a000640 1 -- 172.21.15.90:0/3663900746 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd26c0a26e0 con 0x7fd2640099e0 2024-04-02T14:44:21.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.143+0000 7fd26a000640 1 -- 172.21.15.90:0/3663900746 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fd26c0a1020 con 0x7fd2640099e0 2024-04-02T14:44:21.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.143+0000 7fd276fe7740 1 Processor -- start 2024-04-02T14:44:21.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.144+0000 7fd276fe7740 1 -- start start 2024-04-02T14:44:21.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.145+0000 7fd276fe7740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56461564bba0 0x56461564a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:21.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.145+0000 7fd276fe7740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56461564aec0 0x56461564b340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:21.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.145+0000 7fd276fe7740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564615654230 0x564615656620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:21.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.145+0000 7fd276fe7740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x56461558b400 con 0x564615654230 2024-04-02T14:44:21.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.145+0000 7fd272200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56461564bba0 0x56461564a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:21.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.145+0000 7fd276fe7740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x56461556ff10 con 0x56461564aec0 2024-04-02T14:44:21.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.145+0000 7fd272200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56461564bba0 0x56461564a8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:39276/0 (socket says 172.21.15.90:39276) 2024-04-02T14:44:21.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.145+0000 7fd272200640 1 -- 172.21.15.90:0/2712361733 learned_addr learned my addr 172.21.15.90:0/2712361733 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:44:21.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.145+0000 7fd276fe7740 1 -- 172.21.15.90:0/2712361733 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5646155af6f0 con 0x56461564bba0 2024-04-02T14:44:21.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.145+0000 7fd26a000640 1 --2- 172.21.15.90:0/3663900746 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fd2540a3e20 0x7fd2540a62e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:21.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.146+0000 7fd26a000640 1 -- 172.21.15.90:0/3663900746 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8550+0+0 (secure 0 0 0) 0x7fd26c15c870 con 0x7fd2640099e0 2024-04-02T14:44:21.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.147+0000 7fd271800640 1 --2- 172.21.15.90:0/2712361733 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56461564aec0 0x56461564b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:21.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.147+0000 7fd272200640 1 --2- 172.21.15.90:0/3663900746 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fd2540a3e20 0x7fd2540a62e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:21.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.147+0000 7fd270e00640 1 --2- 172.21.15.90:0/2712361733 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564615654230 0x564615656620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:21.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.148+0000 7fd271800640 1 -- 172.21.15.90:0/2712361733 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56461564bba0 msgr2=0x56461564a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:44:21.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.148+0000 7fd271800640 1 --2- 172.21.15.90:0/2712361733 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56461564bba0 0x56461564a8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:21.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.148+0000 7fd271800640 1 -- 172.21.15.90:0/2712361733 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564615654230 msgr2=0x564615656620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:44:21.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.148+0000 7fd271800640 1 --2- 172.21.15.90:0/2712361733 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564615654230 0x564615656620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:21.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.148+0000 7fd271800640 1 -- 172.21.15.90:0/2712361733 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5646152c33f0 con 0x56461564aec0 2024-04-02T14:44:21.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.150+0000 7fd271800640 1 --2- 172.21.15.90:0/2712361733 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56461564aec0 0x56461564b340 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7fd26000a900 tx=0x7fd26000adc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ca25249bf07bc8ef server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:44:21.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.150+0000 7fd25ea00640 1 -- 172.21.15.90:0/2712361733 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd26000cd80 con 0x56461564aec0 2024-04-02T14:44:21.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.151+0000 7fd272200640 1 --2- 172.21.15.90:0/3663900746 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fd2540a3e20 0x7fd2540a62e0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fd258015740 tx=0x7fd2580046c0 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:44:21.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.151+0000 7fd25ea00640 1 -- 172.21.15.90:0/2712361733 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd26000f930 con 0x56461564aec0 2024-04-02T14:44:21.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.151+0000 7fd25ea00640 1 -- 172.21.15.90:0/2712361733 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd260018940 con 0x56461564aec0 2024-04-02T14:44:21.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.151+0000 7fd276fe7740 1 -- 172.21.15.90:0/2712361733 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56461564aec0 msgr2=0x56461564b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:21.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.151+0000 7fd276fe7740 1 --2- 172.21.15.90:0/2712361733 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56461564aec0 0x56461564b340 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7fd26000a900 tx=0x7fd26000adc0 comp rx=0 tx=0).stop 2024-04-02T14:44:21.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.152+0000 7fd276fe7740 1 -- 172.21.15.90:0/2712361733 shutdown_connections 2024-04-02T14:44:21.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.152+0000 7fd276fe7740 1 --2- 172.21.15.90:0/2712361733 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564615654230 0x564615656620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:21.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.152+0000 7fd276fe7740 1 --2- 172.21.15.90:0/2712361733 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56461564aec0 0x56461564b340 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:21.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.152+0000 7fd276fe7740 1 --2- 172.21.15.90:0/2712361733 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56461564bba0 0x56461564a8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:21.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.152+0000 7fd276fe7740 1 -- 172.21.15.90:0/2712361733 >> 172.21.15.90:0/2712361733 conn(0x564615611a50 msgr2=0x564615645910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:44:21.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.152+0000 7fd276fe7740 1 -- 172.21.15.90:0/2712361733 shutdown_connections 2024-04-02T14:44:21.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.152+0000 7fd276fe7740 1 -- 172.21.15.90:0/2712361733 wait complete. 2024-04-02T14:44:21.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.153+0000 7fd276fe7740 1 -- start start 2024-04-02T14:44:21.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.153+0000 7fd276fe7740 1 Processor -- start 2024-04-02T14:44:21.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.154+0000 7fd276fe7740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56461564aec0 0x5646156511d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:21.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.154+0000 7fd276fe7740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56461564bba0 0x564615651710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:21.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.154+0000 7fd276fe7740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564615654230 0x564615651c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:21.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.154+0000 7fd276fe7740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5646155ab6a0 con 0x56461564bba0 2024-04-02T14:44:21.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.154+0000 7fd271800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56461564aec0 0x5646156511d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:21.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.155+0000 7fd271800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56461564aec0 0x5646156511d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:38888/0 (socket says 172.21.15.90:38888) 2024-04-02T14:44:21.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.155+0000 7fd271800640 1 -- 172.21.15.90:0/54162485 learned_addr learned my addr 172.21.15.90:0/54162485 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:44:21.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.155+0000 7fd276fe7740 1 -- 172.21.15.90:0/54162485 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x564615557980 con 0x56461564aec0 2024-04-02T14:44:21.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.155+0000 7fd276fe7740 1 -- 172.21.15.90:0/54162485 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x56461556acb0 con 0x564615654230 2024-04-02T14:44:21.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.155+0000 7fd276fe7740 10 client.0 _ll_register_callbacks cb 0x56461556a6a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:44:21.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.155+0000 7fd272200640 1 --2- 172.21.15.90:0/54162485 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564615654230 0x564615651c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:21.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.155+0000 7fd271800640 1 -- 172.21.15.90:0/54162485 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564615654230 msgr2=0x564615651c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:44:21.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.155+0000 7fd271800640 1 --2- 172.21.15.90:0/54162485 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564615654230 0x564615651c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:21.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.155+0000 7fd271800640 1 -- 172.21.15.90:0/54162485 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56461564bba0 msgr2=0x564615651710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:44:21.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.155+0000 7fd271800640 1 --2- 172.21.15.90:0/54162485 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56461564bba0 0x564615651710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:21.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.155+0000 7fd271800640 1 -- 172.21.15.90:0/54162485 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56461538d7d0 con 0x56461564aec0 2024-04-02T14:44:21.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.156+0000 7fd271800640 1 --2- 172.21.15.90:0/54162485 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56461564aec0 0x5646156511d0 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7fd260005a70 tx=0x7fd26000afd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f23eab0517a4c910 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:44:21.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.156+0000 7fd253400640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:44:21.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.157+0000 7fd253400640 1 -- 172.21.15.90:0/54162485 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd26000fae0 con 0x56461564aec0 2024-04-02T14:44:21.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.157+0000 7fd253400640 1 -- 172.21.15.90:0/54162485 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8550+0+0 (secure 0 0 0) 0x7fd26000fc80 con 0x56461564aec0 2024-04-02T14:44:21.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.157+0000 7fd276fe7740 1 -- 172.21.15.90:0/54162485 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd26000f930 con 0x56461564aec0 2024-04-02T14:44:21.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.158+0000 7fd253400640 1 -- 172.21.15.90:0/54162485 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fd2600186c0 con 0x56461564aec0 2024-04-02T14:44:21.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.158+0000 7fd276fe7740 10 client.28379 fetch_fsmap learned FSMap version 27 2024-04-02T14:44:21.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.158+0000 7fd276fe7740 1 -- 172.21.15.90:0/54162485 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x564615663110 con 0x56461564aec0 2024-04-02T14:44:21.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.160+0000 7fd253400640 1 -- 172.21.15.90:0/54162485 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd26000f930 con 0x56461564aec0 2024-04-02T14:44:21.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.160+0000 7fd253400640 1 -- 172.21.15.90:0/54162485 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fd260023b60 con 0x56461564aec0 2024-04-02T14:44:21.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.160+0000 7fd276fe7740 10 client.28379 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:44:21.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.160+0000 7fd276fe7740 10 client.28379 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.c 2024-04-02T14:44:21.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.160+0000 7fd276fe7740 20 client.28379 populate_metadata read hostname 'smithi090' 2024-04-02T14:44:21.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.160+0000 7fd276fe7740 1 --2- 172.21.15.90:0/54162485 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x564615664510 0x564615666900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:21.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.160+0000 7fd276fe7740 4 client.28379 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:44:21.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.160+0000 7fd276fe7740 1 -- 172.21.15.90:0/54162485 --> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56461556f040 con 0x564615664510 2024-04-02T14:44:21.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.161+0000 7fd270e00640 1 --2- 172.21.15.90:0/54162485 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x564615664510 0x564615666900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:21.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.161+0000 7fd270e00640 1 --2- 172.21.15.90:0/54162485 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x564615664510 0x564615666900 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=b7f101efffab293e server_cookie=49ec67b75769f252 in_seq=0 out_seq=0 2024-04-02T14:44:21.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.161+0000 7fd253400640 10 client.28379 ms_handle_connect on v2:172.21.15.177:6836/4223414858 2024-04-02T14:44:21.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.162+0000 7fd253400640 1 -- 172.21.15.90:0/54162485 <== mds.4 v2:172.21.15.177:6836/4223414858 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56461556f040 con 0x564615664510 2024-04-02T14:44:21.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.162+0000 7fd253400640 10 client.28379 handle_command_reply: tid=0 2024-04-02T14:44:21.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.162+0000 7fd276fe7740 1 client.28379 shutdown 2024-04-02T14:44:21.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.163+0000 7fd276fe7740 1 -- 172.21.15.90:0/54162485 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56461564aec0 msgr2=0x5646156511d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:21.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.163+0000 7fd276fe7740 1 --2- 172.21.15.90:0/54162485 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56461564aec0 0x5646156511d0 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7fd260005a70 tx=0x7fd26000afd0 comp rx=0 tx=0).stop 2024-04-02T14:44:21.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.163+0000 7fd276fe7740 1 -- 172.21.15.90:0/54162485 shutdown_connections 2024-04-02T14:44:21.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.163+0000 7fd276fe7740 1 --2- 172.21.15.90:0/54162485 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564615654230 0x564615651c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:21.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.163+0000 7fd276fe7740 1 --2- 172.21.15.90:0/54162485 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x564615664510 0x564615666900 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:21.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.163+0000 7fd276fe7740 1 --2- 172.21.15.90:0/54162485 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56461564bba0 0x564615651710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:21.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.163+0000 7fd276fe7740 1 --2- 172.21.15.90:0/54162485 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56461564aec0 0x5646156511d0 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:21.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.163+0000 7fd276fe7740 1 -- 172.21.15.90:0/54162485 >> 172.21.15.90:0/54162485 conn(0x564615611a50 msgr2=0x564615647660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:44:21.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.163+0000 7fd253400640 0 client.28379 ms_handle_reset on v2:172.21.15.177:6836/4223414858 2024-04-02T14:44:21.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.164+0000 7fd276fe7740 1 -- 172.21.15.90:0/54162485 shutdown_connections 2024-04-02T14:44:21.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.164+0000 7fd276fe7740 1 -- 172.21.15.90:0/54162485 wait complete. 2024-04-02T14:44:21.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.165+0000 7fd276fe7740 20 client.28379 trim_cache size 0 max 16384 2024-04-02T14:44:21.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.189+0000 7fd276fe7740 1 Processor -- start 2024-04-02T14:44:21.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.190+0000 7fd276fe7740 1 -- start start 2024-04-02T14:44:21.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.191+0000 7fd276fe7740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56461564aec0 0x564615653900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:21.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.191+0000 7fd276fe7740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56461564bba0 0x56461565fd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:21.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.192+0000 7fd276fe7740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564615654230 0x564615644970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:21.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.192+0000 7fd276fe7740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fd24c00e8f0 con 0x56461564bba0 2024-04-02T14:44:21.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.192+0000 7fd276fe7740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fd24c010430 con 0x564615654230 2024-04-02T14:44:21.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.192+0000 7fd276fe7740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fd24c00efc0 con 0x56461564aec0 2024-04-02T14:44:21.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.192+0000 7fd271800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56461564aec0 0x564615653900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:21.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.192+0000 7fd271800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56461564aec0 0x564615653900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:39294/0 (socket says 172.21.15.90:39294) 2024-04-02T14:44:21.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.192+0000 7fd271800640 1 -- 172.21.15.90:0/276726372 learned_addr learned my addr 172.21.15.90:0/276726372 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:44:21.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.192+0000 7fd272200640 1 --2- 172.21.15.90:0/276726372 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564615654230 0x564615644970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:21.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.193+0000 7fd270e00640 1 --2- 172.21.15.90:0/276726372 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56461564bba0 0x56461565fd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:21.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.193+0000 7fd271800640 1 -- 172.21.15.90:0/276726372 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564615654230 msgr2=0x564615644970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:44:21.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.193+0000 7fd271800640 1 --2- 172.21.15.90:0/276726372 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564615654230 0x564615644970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:21.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.193+0000 7fd271800640 1 -- 172.21.15.90:0/276726372 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56461564bba0 msgr2=0x56461565fd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:44:21.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.193+0000 7fd271800640 1 --2- 172.21.15.90:0/276726372 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56461564bba0 0x56461565fd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:21.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.193+0000 7fd271800640 1 -- 172.21.15.90:0/276726372 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564615649ef0 con 0x56461564aec0 2024-04-02T14:44:21.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.194+0000 7fd271800640 1 --2- 172.21.15.90:0/276726372 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56461564aec0 0x564615653900 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fd26000a330 tx=0x7fd260018400 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7be63f98231b32eb server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:44:21.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.194+0000 7fd25e000640 1 -- 172.21.15.90:0/276726372 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd26003a070 con 0x56461564aec0 2024-04-02T14:44:21.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.194+0000 7fd25e000640 1 -- 172.21.15.90:0/276726372 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd260028070 con 0x56461564aec0 2024-04-02T14:44:21.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.194+0000 7fd25e000640 1 -- 172.21.15.90:0/276726372 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd26000f6e0 con 0x56461564aec0 2024-04-02T14:44:21.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.196+0000 7fd276fe7740 1 -- 172.21.15.90:0/276726372 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56461564aec0 msgr2=0x564615653900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:21.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.196+0000 7fd276fe7740 1 --2- 172.21.15.90:0/276726372 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56461564aec0 0x564615653900 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fd26000a330 tx=0x7fd260018400 comp rx=0 tx=0).stop 2024-04-02T14:44:21.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.196+0000 7fd276fe7740 1 -- 172.21.15.90:0/276726372 shutdown_connections 2024-04-02T14:44:21.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.196+0000 7fd276fe7740 1 --2- 172.21.15.90:0/276726372 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564615654230 0x564615644970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:21.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.196+0000 7fd276fe7740 1 --2- 172.21.15.90:0/276726372 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56461564bba0 0x56461565fd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:21.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.196+0000 7fd276fe7740 1 --2- 172.21.15.90:0/276726372 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56461564aec0 0x564615653900 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:21.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.196+0000 7fd276fe7740 1 -- 172.21.15.90:0/276726372 >> 172.21.15.90:0/276726372 conn(0x564615611a50 msgr2=0x564615644260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:44:21.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.196+0000 7fd276fe7740 1 -- 172.21.15.90:0/276726372 shutdown_connections 2024-04-02T14:44:21.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.196+0000 7fd276fe7740 1 -- 172.21.15.90:0/276726372 wait complete. 2024-04-02T14:44:21.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.197+0000 7fd276fe7740 1 -- start start 2024-04-02T14:44:21.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.198+0000 7fd276fe7740 1 Processor -- start 2024-04-02T14:44:21.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.199+0000 7fd276fe7740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56461564aec0 0x56461564e790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:21.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.199+0000 7fd276fe7740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56461564bba0 0x56461564ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:21.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.199+0000 7fd276fe7740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564615654230 0x564615654ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:21.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.199+0000 7fd276fe7740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fd24c00fd60 con 0x56461564aec0 2024-04-02T14:44:21.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.199+0000 7fd276fe7740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fd24c00f690 con 0x56461564bba0 2024-04-02T14:44:21.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.199+0000 7fd276fe7740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fd24c00c8f0 con 0x564615654230 2024-04-02T14:44:21.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.199+0000 7fd276fe7740 10 client.0 _ll_register_callbacks cb 0x7fd26000ceb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:44:21.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.200+0000 7fd270e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56461564bba0 0x56461564ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:21.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.200+0000 7fd271800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56461564aec0 0x56461564e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:21.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.200+0000 7fd271800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56461564aec0 0x56461564e790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:44962/0 (socket says 172.21.15.90:44962) 2024-04-02T14:44:21.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.200+0000 7fd271800640 1 -- 172.21.15.90:0/3595907096 learned_addr learned my addr 172.21.15.90:0/3595907096 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:44:21.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.200+0000 7fd270e00640 1 -- 172.21.15.90:0/3595907096 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564615654230 msgr2=0x564615654ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:44:21.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.200+0000 7fd270e00640 1 --2- 172.21.15.90:0/3595907096 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564615654230 0x564615654ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:21.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.200+0000 7fd270e00640 1 -- 172.21.15.90:0/3595907096 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56461564aec0 msgr2=0x56461564e790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:44:21.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.200+0000 7fd270e00640 1 --2- 172.21.15.90:0/3595907096 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56461564aec0 0x56461564e790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:21.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.200+0000 7fd270e00640 1 -- 172.21.15.90:0/3595907096 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56461565ca60 con 0x56461564bba0 2024-04-02T14:44:21.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.200+0000 7fd270e00640 1 --2- 172.21.15.90:0/3595907096 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56461564bba0 0x56461564ecd0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7fd26c159f40 tx=0x7fd26c0501d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=efc32927ac7ce3fd server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:44:21.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.200+0000 7fd253400640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:44:21.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.201+0000 7fd253400640 1 -- 172.21.15.90:0/3595907096 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd26c128810 con 0x56461564bba0 2024-04-02T14:44:21.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.201+0000 7fd253400640 1 -- 172.21.15.90:0/3595907096 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8550+0+0 (secure 0 0 0) 0x7fd26c15a1c0 con 0x56461564bba0 2024-04-02T14:44:21.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.201+0000 7fd276fe7740 1 -- 172.21.15.90:0/3595907096 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd260028070 con 0x56461564bba0 2024-04-02T14:44:21.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.201+0000 7fd253400640 1 -- 172.21.15.90:0/3595907096 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fd26c15a3c0 con 0x56461564bba0 2024-04-02T14:44:21.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.201+0000 7fd276fe7740 10 client.28385 fetch_fsmap learned FSMap version 27 2024-04-02T14:44:21.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.201+0000 7fd276fe7740 1 -- 172.21.15.90:0/3595907096 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5646156dbef0 con 0x56461564bba0 2024-04-02T14:44:21.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.201+0000 7fd253400640 1 -- 172.21.15.90:0/3595907096 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd260028070 con 0x56461564bba0 2024-04-02T14:44:21.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.201+0000 7fd253400640 1 -- 172.21.15.90:0/3595907096 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fd26c127c60 con 0x56461564bba0 2024-04-02T14:44:21.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.201+0000 7fd276fe7740 10 client.28385 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:44:21.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.201+0000 7fd276fe7740 10 client.28385 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.c 2024-04-02T14:44:21.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.201+0000 7fd276fe7740 20 client.28385 populate_metadata read hostname 'smithi090' 2024-04-02T14:44:21.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.201+0000 7fd276fe7740 1 --2- 172.21.15.90:0/3595907096 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x5646156dc620 0x5646156dea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:21.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.201+0000 7fd276fe7740 4 client.28385 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-02T14:44:21.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.201+0000 7fd276fe7740 1 -- 172.21.15.90:0/3595907096 --> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5646155bcc30 con 0x5646156dc620 2024-04-02T14:44:21.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.202+0000 7fd271800640 1 --2- 172.21.15.90:0/3595907096 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x5646156dc620 0x5646156dea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:21.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.202+0000 7fd253400640 10 client.28385 ms_handle_connect on v2:172.21.15.177:6836/4223414858 2024-04-02T14:44:21.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.202+0000 7fd271800640 1 --2- 172.21.15.90:0/3595907096 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x5646156dc620 0x5646156dea40 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=d272f51a28aa8fdf server_cookie=6dfd3883df1f9b39 in_seq=0 out_seq=0 2024-04-02T14:44:21.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.203+0000 7fd253400640 1 -- 172.21.15.90:0/3595907096 <== mds.4 v2:172.21.15.177:6836/4223414858 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5646155bcc30 con 0x5646156dc620 2024-04-02T14:44:21.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.203+0000 7fd253400640 10 client.28385 handle_command_reply: tid=0 2024-04-02T14:44:21.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.203+0000 7fd276fe7740 1 client.28385 shutdown 2024-04-02T14:44:21.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.203+0000 7fd276fe7740 1 -- 172.21.15.90:0/3595907096 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56461564bba0 msgr2=0x56461564ecd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:21.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.203+0000 7fd276fe7740 1 --2- 172.21.15.90:0/3595907096 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56461564bba0 0x56461564ecd0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7fd26c159f40 tx=0x7fd26c0501d0 comp rx=0 tx=0).stop 2024-04-02T14:44:21.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.203+0000 7fd276fe7740 1 -- 172.21.15.90:0/3595907096 shutdown_connections 2024-04-02T14:44:21.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.203+0000 7fd276fe7740 1 --2- 172.21.15.90:0/3595907096 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564615654230 0x564615654ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:21.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.203+0000 7fd276fe7740 1 --2- 172.21.15.90:0/3595907096 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56461564bba0 0x56461564ecd0 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:21.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.203+0000 7fd276fe7740 1 --2- 172.21.15.90:0/3595907096 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x5646156dc620 0x5646156dea40 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:21.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.203+0000 7fd276fe7740 1 --2- 172.21.15.90:0/3595907096 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56461564aec0 0x56461564e790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:21.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.203+0000 7fd276fe7740 1 -- 172.21.15.90:0/3595907096 >> 172.21.15.90:0/3595907096 conn(0x564615611a50 msgr2=0x564615644af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:44:21.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.203+0000 7fd253400640 0 client.28385 ms_handle_reset on v2:172.21.15.177:6836/4223414858 2024-04-02T14:44:21.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.204+0000 7fd276fe7740 1 -- 172.21.15.90:0/3595907096 shutdown_connections 2024-04-02T14:44:21.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.204+0000 7fd276fe7740 1 -- 172.21.15.90:0/3595907096 wait complete. 2024-04-02T14:44:21.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.204+0000 7fd276fe7740 20 client.28385 trim_cache size 0 max 16384 2024-04-02T14:44:21.204 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-02T14:44:21.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.206+0000 7fd272c00640 1 -- 172.21.15.90:0/3663900746 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fd2540a3e20 msgr2=0x7fd2540a62e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:21.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.206+0000 7fd272c00640 1 --2- 172.21.15.90:0/3663900746 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fd2540a3e20 0x7fd2540a62e0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fd258015740 tx=0x7fd2580046c0 comp rx=0 tx=0).stop 2024-04-02T14:44:21.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.206+0000 7fd272c00640 1 -- 172.21.15.90:0/3663900746 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd2640099e0 msgr2=0x7fd264138e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:21.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.206+0000 7fd272c00640 1 --2- 172.21.15.90:0/3663900746 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd2640099e0 0x7fd264138e30 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7fd26c0673c0 tx=0x7fd26c09a740 comp rx=0 tx=0).stop 2024-04-02T14:44:21.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.206+0000 7fd272c00640 1 -- 172.21.15.90:0/3663900746 shutdown_connections 2024-04-02T14:44:21.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.206+0000 7fd272c00640 1 --2- 172.21.15.90:0/3663900746 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fd2540a3e20 0x7fd2540a62e0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:21.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.206+0000 7fd272c00640 1 --2- 172.21.15.90:0/3663900746 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd2640a6b90 0x7fd264132aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:21.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.206+0000 7fd272c00640 1 --2- 172.21.15.90:0/3663900746 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd2640099e0 0x7fd264138e30 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:21.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.206+0000 7fd272c00640 1 --2- 172.21.15.90:0/3663900746 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd2640090c0 0x7fd2641388f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:21.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.206+0000 7fd272c00640 1 -- 172.21.15.90:0/3663900746 >> 172.21.15.90:0/3663900746 conn(0x7fd2640939c0 msgr2=0x7fd264095db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:44:21.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.206+0000 7fd272c00640 1 -- 172.21.15.90:0/3663900746 shutdown_connections 2024-04-02T14:44:21.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.206+0000 7fd272c00640 1 -- 172.21.15.90:0/3663900746 wait complete. 2024-04-02T14:44:21.224 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-04-02T14:44:21.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.462+0000 7fe611400640 1 -- 172.21.15.90:0/2857190231 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe60c107b50 msgr2=0x7fe60c107f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:21.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.462+0000 7fe611400640 1 --2- 172.21.15.90:0/2857190231 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe60c107b50 0x7fe60c107f30 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fe5fc009930 tx=0x7fe5fc02bc30 comp rx=0 tx=0).stop 2024-04-02T14:44:21.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.462+0000 7fe611400640 1 -- 172.21.15.90:0/2857190231 shutdown_connections 2024-04-02T14:44:21.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.462+0000 7fe611400640 1 --2- 172.21.15.90:0/2857190231 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe60c0696c0 0x7fe60c10e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:21.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.462+0000 7fe611400640 1 --2- 172.21.15.90:0/2857190231 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe60c068d00 0x7fe60c069180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:21.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.462+0000 7fe611400640 1 --2- 172.21.15.90:0/2857190231 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe60c107b50 0x7fe60c107f30 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:21.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.462+0000 7fe611400640 1 -- 172.21.15.90:0/2857190231 >> 172.21.15.90:0/2857190231 conn(0x7fe60c077b00 msgr2=0x7fe60c077f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:44:21.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.462+0000 7fe611400640 1 -- 172.21.15.90:0/2857190231 shutdown_connections 2024-04-02T14:44:21.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.462+0000 7fe611400640 1 -- 172.21.15.90:0/2857190231 wait complete. 2024-04-02T14:44:21.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.463+0000 7fe611400640 1 Processor -- start 2024-04-02T14:44:21.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.463+0000 7fe611400640 1 -- start start 2024-04-02T14:44:21.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.464+0000 7fe611400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe60c068d00 0x7fe60c1009c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:21.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.464+0000 7fe611400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe60c0696c0 0x7fe60c100f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:21.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.464+0000 7fe611400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe60c107b50 0x7fe60c101440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:21.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.464+0000 7fe611400640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fe60c111ef0 con 0x7fe60c107b50 2024-04-02T14:44:21.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.464+0000 7fe611400640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fe60c111d70 con 0x7fe60c0696c0 2024-04-02T14:44:21.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.464+0000 7fe611400640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fe60c112070 con 0x7fe60c068d00 2024-04-02T14:44:21.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.464+0000 7fe60b200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe60c107b50 0x7fe60c101440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:21.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.464+0000 7fe60a800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe60c068d00 0x7fe60c1009c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:21.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.464+0000 7fe603e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe60c0696c0 0x7fe60c100f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:21.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.464+0000 7fe60b200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe60c107b50 0x7fe60c101440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:44990/0 (socket says 172.21.15.90:44990) 2024-04-02T14:44:21.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.464+0000 7fe60a800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe60c068d00 0x7fe60c1009c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:39312/0 (socket says 172.21.15.90:39312) 2024-04-02T14:44:21.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.464+0000 7fe603e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe60c0696c0 0x7fe60c100f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:38916/0 (socket says 172.21.15.90:38916) 2024-04-02T14:44:21.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.464+0000 7fe60b200640 1 -- 172.21.15.90:0/2287871412 learned_addr learned my addr 172.21.15.90:0/2287871412 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:44:21.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.465+0000 7fe60a800640 1 -- 172.21.15.90:0/2287871412 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe60c0696c0 msgr2=0x7fe60c100f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:21.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.465+0000 7fe60a800640 1 --2- 172.21.15.90:0/2287871412 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe60c0696c0 0x7fe60c100f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:21.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.465+0000 7fe60a800640 1 -- 172.21.15.90:0/2287871412 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe60c107b50 msgr2=0x7fe60c101440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:21.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.465+0000 7fe60a800640 1 --2- 172.21.15.90:0/2287871412 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe60c107b50 0x7fe60c101440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:21.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.465+0000 7fe60a800640 1 -- 172.21.15.90:0/2287871412 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe60c1a68b0 con 0x7fe60c068d00 2024-04-02T14:44:21.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.465+0000 7fe603e00640 1 --2- 172.21.15.90:0/2287871412 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe60c0696c0 0x7fe60c100f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:44:21.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.465+0000 7fe60a800640 1 --2- 172.21.15.90:0/2287871412 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe60c068d00 0x7fe60c1009c0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7fe5fc000c00 tx=0x7fe5fc004970 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:44:21.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.465+0000 7fe603400640 1 -- 172.21.15.90:0/2287871412 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5fc043020 con 0x7fe60c068d00 2024-04-02T14:44:21.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.465+0000 7fe611400640 1 -- 172.21.15.90:0/2287871412 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe60c1a6b40 con 0x7fe60c068d00 2024-04-02T14:44:21.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.465+0000 7fe603400640 1 -- 172.21.15.90:0/2287871412 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe5fc0054e0 con 0x7fe60c068d00 2024-04-02T14:44:21.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.465+0000 7fe603400640 1 -- 172.21.15.90:0/2287871412 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5fc003a00 con 0x7fe60c068d00 2024-04-02T14:44:21.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.467+0000 7fe611400640 1 -- 172.21.15.90:0/2287871412 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe60c1a70b0 con 0x7fe60c068d00 2024-04-02T14:44:21.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.470+0000 7fe603400640 1 -- 172.21.15.90:0/2287871412 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fe5fc046020 con 0x7fe60c068d00 2024-04-02T14:44:21.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.479+0000 7fe6141e6740 1 Processor -- start 2024-04-02T14:44:21.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.479+0000 7fe6141e6740 1 -- start start 2024-04-02T14:44:21.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.479+0000 7fe603400640 1 --2- 172.21.15.90:0/2287871412 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fe5f00a3d20 0x7fe5f00a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:21.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.480+0000 7fe603400640 1 -- 172.21.15.90:0/2287871412 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8550+0+0 (secure 0 0 0) 0x7fe5fc0f5170 con 0x7fe60c068d00 2024-04-02T14:44:21.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.480+0000 7fe60b200640 1 --2- 172.21.15.90:0/2287871412 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fe5f00a3d20 0x7fe5f00a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:21.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.480+0000 7fe6141e6740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ef820999f0 0x55ef82098740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:21.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.481+0000 7fe6141e6740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ef82098d10 0x55ef82099190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:21.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.481+0000 7fe60b200640 1 --2- 172.21.15.90:0/2287871412 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fe5f00a3d20 0x7fe5f00a61e0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fe6040099a0 tx=0x7fe604009500 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:44:21.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.481+0000 7fe6141e6740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ef820a2080 0x55ef820a4470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:21.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.481+0000 7fe6141e6740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55ef81fd9400 con 0x55ef820a2080 2024-04-02T14:44:21.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.481+0000 7fe6141e6740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55ef81fbdf10 con 0x55ef82098d10 2024-04-02T14:44:21.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.481+0000 7fe6141e6740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55ef81ffd6f0 con 0x55ef820999f0 2024-04-02T14:44:21.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.483+0000 7fe60a800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ef820999f0 0x55ef82098740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:21.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.483+0000 7fe603e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ef82098d10 0x55ef82099190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:21.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.483+0000 7fe60a800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ef820999f0 0x55ef82098740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:39328/0 (socket says 172.21.15.90:39328) 2024-04-02T14:44:21.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.483+0000 7fe60a800640 1 -- 172.21.15.90:0/3993466540 learned_addr learned my addr 172.21.15.90:0/3993466540 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:44:21.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.484+0000 7fe60a800640 1 -- 172.21.15.90:0/3993466540 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ef82098d10 msgr2=0x55ef82099190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:44:21.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.484+0000 7fe60a800640 1 --2- 172.21.15.90:0/3993466540 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ef82098d10 0x55ef82099190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:21.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.484+0000 7fe60a800640 1 -- 172.21.15.90:0/3993466540 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ef820a2080 msgr2=0x55ef820a4470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:44:21.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.484+0000 7fe60a800640 1 --2- 172.21.15.90:0/3993466540 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ef820a2080 0x55ef820a4470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:21.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.484+0000 7fe60a800640 1 -- 172.21.15.90:0/3993466540 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ef81d113f0 con 0x55ef820999f0 2024-04-02T14:44:21.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.484+0000 7fe60a800640 1 --2- 172.21.15.90:0/3993466540 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ef820999f0 0x55ef82098740 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7fe5fc02c170 tx=0x7fe5fc004580 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2578e26130c9bcc6 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:44:21.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.484+0000 7fe5fb400640 1 -- 172.21.15.90:0/3993466540 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5fc0f6060 con 0x55ef820999f0 2024-04-02T14:44:21.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.484+0000 7fe5fb400640 1 -- 172.21.15.90:0/3993466540 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe5fc0bdfe0 con 0x55ef820999f0 2024-04-02T14:44:21.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.484+0000 7fe5fb400640 1 -- 172.21.15.90:0/3993466540 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5fc0ef200 con 0x55ef820999f0 2024-04-02T14:44:21.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.485+0000 7fe6141e6740 1 -- 172.21.15.90:0/3993466540 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ef820999f0 msgr2=0x55ef82098740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:21.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.485+0000 7fe6141e6740 1 --2- 172.21.15.90:0/3993466540 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ef820999f0 0x55ef82098740 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7fe5fc02c170 tx=0x7fe5fc004580 comp rx=0 tx=0).stop 2024-04-02T14:44:21.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.488+0000 7fe6141e6740 1 -- 172.21.15.90:0/3993466540 shutdown_connections 2024-04-02T14:44:21.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.488+0000 7fe6141e6740 1 --2- 172.21.15.90:0/3993466540 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ef820a2080 0x55ef820a4470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:21.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.489+0000 7fe6141e6740 1 --2- 172.21.15.90:0/3993466540 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ef82098d10 0x55ef82099190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:21.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.489+0000 7fe6141e6740 1 --2- 172.21.15.90:0/3993466540 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ef820999f0 0x55ef82098740 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:21.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.489+0000 7fe6141e6740 1 -- 172.21.15.90:0/3993466540 >> 172.21.15.90:0/3993466540 conn(0x55ef8205fa50 msgr2=0x55ef82093760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:44:21.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.489+0000 7fe6141e6740 1 -- 172.21.15.90:0/3993466540 shutdown_connections 2024-04-02T14:44:21.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.490+0000 7fe6141e6740 1 -- 172.21.15.90:0/3993466540 wait complete. 2024-04-02T14:44:21.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.492+0000 7fe6141e6740 1 -- start start 2024-04-02T14:44:21.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.493+0000 7fe6141e6740 1 Processor -- start 2024-04-02T14:44:21.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.494+0000 7fe6141e6740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ef82098d10 0x55ef8209f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:21.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.495+0000 7fe6141e6740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ef820999f0 0x55ef8209f630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:21.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.495+0000 7fe6141e6740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ef820a2080 0x55ef8209fb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:21.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.495+0000 7fe6141e6740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55ef81ff96a0 con 0x55ef820999f0 2024-04-02T14:44:21.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.495+0000 7fe6141e6740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55ef81fa5980 con 0x55ef82098d10 2024-04-02T14:44:21.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.495+0000 7fe6141e6740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55ef81fb8cb0 con 0x55ef820a2080 2024-04-02T14:44:21.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.495+0000 7fe6141e6740 10 client.0 _ll_register_callbacks cb 0x55ef81fb86a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:44:21.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.495+0000 7fe60a800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ef82098d10 0x55ef8209f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:21.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.495+0000 7fe60a800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ef82098d10 0x55ef8209f0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:38928/0 (socket says 172.21.15.90:38928) 2024-04-02T14:44:21.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.495+0000 7fe60a800640 1 -- 172.21.15.90:0/210781134 learned_addr learned my addr 172.21.15.90:0/210781134 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:44:21.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.495+0000 7fe603e00640 1 --2- 172.21.15.90:0/210781134 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ef820999f0 0x55ef8209f630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:21.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.495+0000 7fe60b200640 1 --2- 172.21.15.90:0/210781134 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ef820a2080 0x55ef8209fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:21.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.496+0000 7fe603e00640 1 -- 172.21.15.90:0/210781134 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ef820a2080 msgr2=0x55ef8209fb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:44:21.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.496+0000 7fe603e00640 1 --2- 172.21.15.90:0/210781134 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ef820a2080 0x55ef8209fb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:21.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.496+0000 7fe603e00640 1 -- 172.21.15.90:0/210781134 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ef82098d10 msgr2=0x55ef8209f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:44:21.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.496+0000 7fe603e00640 1 --2- 172.21.15.90:0/210781134 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ef82098d10 0x55ef8209f0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:21.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.496+0000 7fe603e00640 1 -- 172.21.15.90:0/210781134 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ef81ddb7d0 con 0x55ef820999f0 2024-04-02T14:44:21.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.499+0000 7fe603e00640 1 --2- 172.21.15.90:0/210781134 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ef820999f0 0x55ef8209f630 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fe60c102450 tx=0x7fe5f400f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2584dad35461e3c6 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:44:21.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.499+0000 7fe5efe00640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-02T14:44:21.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.503+0000 7fe5efe00640 1 -- 172.21.15.90:0/210781134 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5f4012110 con 0x55ef820999f0 2024-04-02T14:44:21.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.503+0000 7fe5efe00640 1 -- 172.21.15.90:0/210781134 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8550+0+0 (secure 0 0 0) 0x7fe5f4004030 con 0x55ef820999f0 2024-04-02T14:44:21.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.503+0000 7fe6141e6740 1 -- 172.21.15.90:0/210781134 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe5fc0bdfe0 con 0x55ef820999f0 2024-04-02T14:44:21.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.505+0000 7fe5efe00640 1 -- 172.21.15.90:0/210781134 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fe5f40122b0 con 0x55ef820999f0 2024-04-02T14:44:21.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.505+0000 7fe6141e6740 10 client.17523 fetch_fsmap learned FSMap version 27 2024-04-02T14:44:21.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.505+0000 7fe6141e6740 1 -- 172.21.15.90:0/210781134 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55ef820ade40 con 0x55ef820999f0 2024-04-02T14:44:21.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.506+0000 7fe5efe00640 1 -- 172.21.15.90:0/210781134 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe5fc0bdfe0 con 0x55ef820999f0 2024-04-02T14:44:21.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.507+0000 7fe5efe00640 1 -- 172.21.15.90:0/210781134 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fe5f4010430 con 0x55ef820999f0 2024-04-02T14:44:21.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.507+0000 7fe6141e6740 10 client.17523 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:44:21.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.507+0000 7fe6141e6740 10 client.17523 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-04-02T14:44:21.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.507+0000 7fe6141e6740 20 client.17523 populate_metadata read hostname 'smithi090' 2024-04-02T14:44:21.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.507+0000 7fe6141e6740 1 --2- 172.21.15.90:0/210781134 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x55ef820af350 0x55ef820b1770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:21.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.507+0000 7fe6141e6740 4 client.17523 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:44:21.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.507+0000 7fe6141e6740 1 -- 172.21.15.90:0/210781134 --> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ef81fbd040 con 0x55ef820af350 2024-04-02T14:44:21.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.507+0000 7fe60a800640 1 --2- 172.21.15.90:0/210781134 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x55ef820af350 0x55ef820b1770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:21.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.509+0000 7fe60a800640 1 --2- 172.21.15.90:0/210781134 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x55ef820af350 0x55ef820b1770 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=6bf02c461a165532 server_cookie=df1610044696641f in_seq=0 out_seq=0 2024-04-02T14:44:21.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.509+0000 7fe5efe00640 10 client.17523 ms_handle_connect on v2:172.21.15.92:6836/1270635939 2024-04-02T14:44:21.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.510+0000 7fe5efe00640 1 -- 172.21.15.90:0/210781134 <== mds.3 v2:172.21.15.92:6836/1270635939 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ef81fbd040 con 0x55ef820af350 2024-04-02T14:44:21.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.510+0000 7fe5efe00640 10 client.17523 handle_command_reply: tid=0 2024-04-02T14:44:21.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.510+0000 7fe6141e6740 1 client.17523 shutdown 2024-04-02T14:44:21.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.511+0000 7fe6141e6740 1 -- 172.21.15.90:0/210781134 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ef820999f0 msgr2=0x55ef8209f630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:21.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.511+0000 7fe6141e6740 1 --2- 172.21.15.90:0/210781134 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ef820999f0 0x55ef8209f630 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fe60c102450 tx=0x7fe5f400f790 comp rx=0 tx=0).stop 2024-04-02T14:44:21.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.515+0000 7fe6141e6740 1 -- 172.21.15.90:0/210781134 shutdown_connections 2024-04-02T14:44:21.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.515+0000 7fe6141e6740 1 --2- 172.21.15.90:0/210781134 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x55ef820af350 0x55ef820b1770 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:21.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.515+0000 7fe6141e6740 1 --2- 172.21.15.90:0/210781134 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ef820a2080 0x55ef8209fb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:21.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.515+0000 7fe6141e6740 1 --2- 172.21.15.90:0/210781134 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ef820999f0 0x55ef8209f630 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:21.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.515+0000 7fe6141e6740 1 --2- 172.21.15.90:0/210781134 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ef82098d10 0x55ef8209f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:21.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.515+0000 7fe6141e6740 1 -- 172.21.15.90:0/210781134 >> 172.21.15.90:0/210781134 conn(0x55ef8205fa50 msgr2=0x55ef820954b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:44:21.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.515+0000 7fe5efe00640 0 client.17523 ms_handle_reset on v2:172.21.15.92:6836/1270635939 2024-04-02T14:44:21.514 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.516+0000 7fe6141e6740 1 -- 172.21.15.90:0/210781134 shutdown_connections 2024-04-02T14:44:21.514 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.516+0000 7fe6141e6740 1 -- 172.21.15.90:0/210781134 wait complete. 2024-04-02T14:44:21.514 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.516+0000 7fe6141e6740 20 client.17523 trim_cache size 0 max 16384 2024-04-02T14:44:21.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.545+0000 7fe6141e6740 1 Processor -- start 2024-04-02T14:44:21.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.547+0000 7fe6141e6740 1 -- start start 2024-04-02T14:44:21.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.547+0000 7fe6141e6740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ef82098d10 0x55ef820a1820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:21.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.548+0000 7fe6141e6740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ef820999f0 0x55ef820ad5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:21.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.548+0000 7fe6141e6740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ef820a2080 0x55ef820be600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:21.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.548+0000 7fe6141e6740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fe60c1a7470 con 0x55ef820999f0 2024-04-02T14:44:21.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.548+0000 7fe6141e6740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fe60c1a8fb0 con 0x55ef82098d10 2024-04-02T14:44:21.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.548+0000 7fe6141e6740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fe60c1a7b40 con 0x55ef820a2080 2024-04-02T14:44:21.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.548+0000 7fe60a800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ef82098d10 0x55ef820a1820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:21.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.549+0000 7fe60a800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ef82098d10 0x55ef820a1820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:38938/0 (socket says 172.21.15.90:38938) 2024-04-02T14:44:21.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.549+0000 7fe60a800640 1 -- 172.21.15.90:0/4138237534 learned_addr learned my addr 172.21.15.90:0/4138237534 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:44:21.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.549+0000 7fe603e00640 1 --2- 172.21.15.90:0/4138237534 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ef820999f0 0x55ef820ad5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:21.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.549+0000 7fe60b200640 1 --2- 172.21.15.90:0/4138237534 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ef820a2080 0x55ef820be600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:21.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.549+0000 7fe60a800640 1 -- 172.21.15.90:0/4138237534 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ef820a2080 msgr2=0x55ef820be600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:44:21.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.549+0000 7fe60a800640 1 --2- 172.21.15.90:0/4138237534 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ef820a2080 0x55ef820be600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:21.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.549+0000 7fe60a800640 1 -- 172.21.15.90:0/4138237534 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ef820999f0 msgr2=0x55ef820ad5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:44:21.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.550+0000 7fe60a800640 1 --2- 172.21.15.90:0/4138237534 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ef820999f0 0x55ef820ad5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:21.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.550+0000 7fe60a800640 1 -- 172.21.15.90:0/4138237534 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ef82093970 con 0x55ef82098d10 2024-04-02T14:44:21.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.550+0000 7fe603e00640 1 --2- 172.21.15.90:0/4138237534 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ef820999f0 0x55ef820ad5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:44:21.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.550+0000 7fe60a800640 1 --2- 172.21.15.90:0/4138237534 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ef82098d10 0x55ef820a1820 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7fe5fc0096c0 tx=0x7fe5fc02bc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8afb7e47de53b2eb server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:44:21.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.550+0000 7fe5faa00640 1 -- 172.21.15.90:0/4138237534 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5fc0bd8e0 con 0x55ef82098d10 2024-04-02T14:44:21.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.550+0000 7fe5faa00640 1 -- 172.21.15.90:0/4138237534 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe5fc0c1c90 con 0x55ef82098d10 2024-04-02T14:44:21.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.550+0000 7fe5faa00640 1 -- 172.21.15.90:0/4138237534 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5fc0a80b0 con 0x55ef82098d10 2024-04-02T14:44:21.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.552+0000 7fe6141e6740 1 -- 172.21.15.90:0/4138237534 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ef82098d10 msgr2=0x55ef820a1820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:21.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.552+0000 7fe6141e6740 1 --2- 172.21.15.90:0/4138237534 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ef82098d10 0x55ef820a1820 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7fe5fc0096c0 tx=0x7fe5fc02bc30 comp rx=0 tx=0).stop 2024-04-02T14:44:21.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.552+0000 7fe6141e6740 1 -- 172.21.15.90:0/4138237534 shutdown_connections 2024-04-02T14:44:21.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.552+0000 7fe6141e6740 1 --2- 172.21.15.90:0/4138237534 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ef820a2080 0x55ef820be600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:21.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.552+0000 7fe6141e6740 1 --2- 172.21.15.90:0/4138237534 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ef820999f0 0x55ef820ad5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:21.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.552+0000 7fe6141e6740 1 --2- 172.21.15.90:0/4138237534 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ef82098d10 0x55ef820a1820 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:21.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.552+0000 7fe6141e6740 1 -- 172.21.15.90:0/4138237534 >> 172.21.15.90:0/4138237534 conn(0x55ef8205fa50 msgr2=0x55ef820bf350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:44:21.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.554+0000 7fe6141e6740 1 -- 172.21.15.90:0/4138237534 shutdown_connections 2024-04-02T14:44:21.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.554+0000 7fe6141e6740 1 -- 172.21.15.90:0/4138237534 wait complete. 2024-04-02T14:44:21.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.555+0000 7fe6141e6740 1 -- start start 2024-04-02T14:44:21.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.555+0000 7fe6141e6740 1 Processor -- start 2024-04-02T14:44:21.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.556+0000 7fe6141e6740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ef82098d10 0x55ef820a9ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:21.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.556+0000 7fe6141e6740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ef820999f0 0x55ef820aa010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:21.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.556+0000 7fe6141e6740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ef820a2080 0x55ef820a3810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:21.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.556+0000 7fe6141e6740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fe60c1a88e0 con 0x55ef82098d10 2024-04-02T14:44:21.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.556+0000 7fe6141e6740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fe60c1a8210 con 0x55ef820a2080 2024-04-02T14:44:21.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.556+0000 7fe6141e6740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fe60c071ee0 con 0x55ef820999f0 2024-04-02T14:44:21.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.556+0000 7fe6141e6740 10 client.0 _ll_register_callbacks cb 0x7fe5f4012930 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:44:21.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.558+0000 7fe60b200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ef820a2080 0x55ef820a3810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:21.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.558+0000 7fe60b200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ef820a2080 0x55ef820a3810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:38948/0 (socket says 172.21.15.90:38948) 2024-04-02T14:44:21.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.558+0000 7fe60b200640 1 -- 172.21.15.90:0/376300310 learned_addr learned my addr 172.21.15.90:0/376300310 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:44:21.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.558+0000 7fe60a800640 1 --2- 172.21.15.90:0/376300310 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ef82098d10 0x55ef820a9ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:21.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.558+0000 7fe603e00640 1 --2- 172.21.15.90:0/376300310 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ef820999f0 0x55ef820aa010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:21.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.559+0000 7fe60b200640 1 -- 172.21.15.90:0/376300310 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ef820999f0 msgr2=0x55ef820aa010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:44:21.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.559+0000 7fe60b200640 1 --2- 172.21.15.90:0/376300310 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ef820999f0 0x55ef820aa010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:21.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.559+0000 7fe60b200640 1 -- 172.21.15.90:0/376300310 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ef82098d10 msgr2=0x55ef820a9ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:44:21.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.559+0000 7fe60b200640 1 --2- 172.21.15.90:0/376300310 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ef82098d10 0x55ef820a9ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:21.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.559+0000 7fe60b200640 1 -- 172.21.15.90:0/376300310 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ef820a4010 con 0x55ef820a2080 2024-04-02T14:44:21.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.559+0000 7fe60b200640 1 --2- 172.21.15.90:0/376300310 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ef820a2080 0x55ef820a3810 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7fe60403eef0 tx=0x7fe604004ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5115382bcbe21a7d server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:44:21.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.560+0000 7fe5efe00640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:44:21.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.560+0000 7fe5efe00640 1 -- 172.21.15.90:0/376300310 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe60401ae30 con 0x55ef820a2080 2024-04-02T14:44:21.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.560+0000 7fe5efe00640 1 -- 172.21.15.90:0/376300310 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8550+0+0 (secure 0 0 0) 0x7fe6040485d0 con 0x55ef820a2080 2024-04-02T14:44:21.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.560+0000 7fe6141e6740 1 -- 172.21.15.90:0/376300310 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe5fc0c1c90 con 0x55ef820a2080 2024-04-02T14:44:21.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.560+0000 7fe5efe00640 1 -- 172.21.15.90:0/376300310 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fe6040487d0 con 0x55ef820a2080 2024-04-02T14:44:21.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.561+0000 7fe6141e6740 10 client.28403 fetch_fsmap learned FSMap version 27 2024-04-02T14:44:21.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.561+0000 7fe6141e6740 1 -- 172.21.15.90:0/376300310 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55ef821206f0 con 0x55ef820a2080 2024-04-02T14:44:21.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.562+0000 7fe5efe00640 1 -- 172.21.15.90:0/376300310 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe5fc0c1c90 con 0x55ef820a2080 2024-04-02T14:44:21.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.562+0000 7fe5efe00640 1 -- 172.21.15.90:0/376300310 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fe604018040 con 0x55ef820a2080 2024-04-02T14:44:21.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.562+0000 7fe6141e6740 10 client.28403 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:44:21.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.562+0000 7fe6141e6740 10 client.28403 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-04-02T14:44:21.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.562+0000 7fe6141e6740 20 client.28403 populate_metadata read hostname 'smithi090' 2024-04-02T14:44:21.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.562+0000 7fe6141e6740 1 --2- 172.21.15.90:0/376300310 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x55ef82120ef0 0x55ef82123310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:21.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.563+0000 7fe6141e6740 4 client.28403 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-02T14:44:21.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.563+0000 7fe6141e6740 1 -- 172.21.15.90:0/376300310 --> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55ef8200ac30 con 0x55ef82120ef0 2024-04-02T14:44:21.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.563+0000 7fe60a800640 1 --2- 172.21.15.90:0/376300310 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x55ef82120ef0 0x55ef82123310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:21.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.564+0000 7fe5efe00640 10 client.28403 ms_handle_connect on v2:172.21.15.92:6836/1270635939 2024-04-02T14:44:21.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.564+0000 7fe60a800640 1 --2- 172.21.15.90:0/376300310 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x55ef82120ef0 0x55ef82123310 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=c90555eed6874120 server_cookie=9bd524db9d7ba6fc in_seq=0 out_seq=0 2024-04-02T14:44:21.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.564+0000 7fe5efe00640 1 -- 172.21.15.90:0/376300310 <== mds.3 v2:172.21.15.92:6836/1270635939 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55ef8200ac30 con 0x55ef82120ef0 2024-04-02T14:44:21.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.564+0000 7fe5efe00640 10 client.28403 handle_command_reply: tid=0 2024-04-02T14:44:21.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.564+0000 7fe6141e6740 1 client.28403 shutdown 2024-04-02T14:44:21.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.565+0000 7fe6141e6740 1 -- 172.21.15.90:0/376300310 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ef820a2080 msgr2=0x55ef820a3810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:21.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.565+0000 7fe6141e6740 1 --2- 172.21.15.90:0/376300310 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ef820a2080 0x55ef820a3810 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7fe60403eef0 tx=0x7fe604004ab0 comp rx=0 tx=0).stop 2024-04-02T14:44:21.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.565+0000 7fe6141e6740 1 -- 172.21.15.90:0/376300310 shutdown_connections 2024-04-02T14:44:21.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.565+0000 7fe6141e6740 1 --2- 172.21.15.90:0/376300310 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x55ef82120ef0 0x55ef82123310 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:21.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.565+0000 7fe6141e6740 1 --2- 172.21.15.90:0/376300310 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ef820a2080 0x55ef820a3810 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:21.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.565+0000 7fe6141e6740 1 --2- 172.21.15.90:0/376300310 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ef820999f0 0x55ef820aa010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:21.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.565+0000 7fe6141e6740 1 --2- 172.21.15.90:0/376300310 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ef82098d10 0x55ef820a9ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:21.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.565+0000 7fe6141e6740 1 -- 172.21.15.90:0/376300310 >> 172.21.15.90:0/376300310 conn(0x55ef8205fa50 msgr2=0x55ef82092010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:44:21.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.565+0000 7fe5efe00640 0 client.28403 ms_handle_reset on v2:172.21.15.92:6836/1270635939 2024-04-02T14:44:21.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.565+0000 7fe6141e6740 1 -- 172.21.15.90:0/376300310 shutdown_connections 2024-04-02T14:44:21.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.565+0000 7fe6141e6740 1 -- 172.21.15.90:0/376300310 wait complete. 2024-04-02T14:44:21.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.566+0000 7fe6141e6740 20 client.28403 trim_cache size 0 max 16384 2024-04-02T14:44:21.564 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-02T14:44:21.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.567+0000 7fe611400640 1 -- 172.21.15.90:0/2287871412 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fe5f00a3d20 msgr2=0x7fe5f00a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:21.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.567+0000 7fe611400640 1 --2- 172.21.15.90:0/2287871412 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fe5f00a3d20 0x7fe5f00a61e0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fe6040099a0 tx=0x7fe604009500 comp rx=0 tx=0).stop 2024-04-02T14:44:21.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.567+0000 7fe611400640 1 -- 172.21.15.90:0/2287871412 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe60c068d00 msgr2=0x7fe60c1009c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:21.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.567+0000 7fe611400640 1 --2- 172.21.15.90:0/2287871412 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe60c068d00 0x7fe60c1009c0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7fe5fc000c00 tx=0x7fe5fc004970 comp rx=0 tx=0).stop 2024-04-02T14:44:21.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.567+0000 7fe611400640 1 -- 172.21.15.90:0/2287871412 shutdown_connections 2024-04-02T14:44:21.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.567+0000 7fe611400640 1 --2- 172.21.15.90:0/2287871412 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fe5f00a3d20 0x7fe5f00a61e0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:21.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.567+0000 7fe611400640 1 --2- 172.21.15.90:0/2287871412 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe60c107b50 0x7fe60c101440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:21.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.567+0000 7fe611400640 1 --2- 172.21.15.90:0/2287871412 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe60c0696c0 0x7fe60c100f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:21.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.567+0000 7fe611400640 1 --2- 172.21.15.90:0/2287871412 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe60c068d00 0x7fe60c1009c0 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:21.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.567+0000 7fe611400640 1 -- 172.21.15.90:0/2287871412 >> 172.21.15.90:0/2287871412 conn(0x7fe60c077b00 msgr2=0x7fe60c0fe510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:44:21.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.567+0000 7fe611400640 1 -- 172.21.15.90:0/2287871412 shutdown_connections 2024-04-02T14:44:21.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.567+0000 7fe611400640 1 -- 172.21.15.90:0/2287871412 wait complete. 2024-04-02T14:44:21.581 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:44:21 smithi090 ceph-mon[31674]: from='client.? 172.21.15.90:0/1815278239' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-02T14:44:21.584 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-02T14:44:21.605 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:44:21 smithi092 ceph-mon[36497]: from='client.? 172.21.15.90:0/1815278239' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-02T14:44:21.682 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:44:21 smithi177 ceph-mon[36431]: from='client.? 172.21.15.90:0/1815278239' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-02T14:44:21.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.818+0000 7fb174600640 1 -- 172.21.15.90:0/2049031877 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb16c073f30 msgr2=0x7fb16c074310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:21.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.818+0000 7fb174600640 1 --2- 172.21.15.90:0/2049031877 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb16c073f30 0x7fb16c074310 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7fb160009900 tx=0x7fb16002bcd0 comp rx=0 tx=0).stop 2024-04-02T14:44:21.817 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.818+0000 7fb174600640 1 -- 172.21.15.90:0/2049031877 shutdown_connections 2024-04-02T14:44:21.817 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.818+0000 7fb174600640 1 --2- 172.21.15.90:0/2049031877 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb16c10e2c0 0x7fb16c110710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:21.817 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.818+0000 7fb174600640 1 --2- 172.21.15.90:0/2049031877 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb16c074850 0x7fb16c10dd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:21.817 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.818+0000 7fb174600640 1 --2- 172.21.15.90:0/2049031877 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb16c073f30 0x7fb16c074310 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:21.817 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.818+0000 7fb174600640 1 -- 172.21.15.90:0/2049031877 >> 172.21.15.90:0/2049031877 conn(0x7fb16c0fd590 msgr2=0x7fb16c0ff9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:44:21.817 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.819+0000 7fb174600640 1 -- 172.21.15.90:0/2049031877 shutdown_connections 2024-04-02T14:44:21.817 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.819+0000 7fb174600640 1 -- 172.21.15.90:0/2049031877 wait complete. 2024-04-02T14:44:21.817 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.820+0000 7fb174600640 1 Processor -- start 2024-04-02T14:44:21.818 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.820+0000 7fb174600640 1 -- start start 2024-04-02T14:44:21.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.821+0000 7fb174600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb16c073f30 0x7fb16c1019e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:21.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.822+0000 7fb174600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb16c074850 0x7fb16c101f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:21.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.822+0000 7fb174600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb16c10e2c0 0x7fb16c102460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:21.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.822+0000 7fb174600640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fb16c114080 con 0x7fb16c073f30 2024-04-02T14:44:21.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.822+0000 7fb174600640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fb16c113f00 con 0x7fb16c10e2c0 2024-04-02T14:44:21.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.822+0000 7fb174600640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fb16c114200 con 0x7fb16c074850 2024-04-02T14:44:21.820 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.822+0000 7fb171200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb16c074850 0x7fb16c101f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:21.821 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.822+0000 7fb171c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb16c073f30 0x7fb16c1019e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:21.821 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.822+0000 7fb171200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb16c074850 0x7fb16c101f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:39380/0 (socket says 172.21.15.90:39380) 2024-04-02T14:44:21.821 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.822+0000 7fb172600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb16c10e2c0 0x7fb16c102460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:21.821 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.822+0000 7fb171200640 1 -- 172.21.15.90:0/208881555 learned_addr learned my addr 172.21.15.90:0/208881555 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:44:21.821 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.823+0000 7fb172600640 1 -- 172.21.15.90:0/208881555 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb16c074850 msgr2=0x7fb16c101f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:21.821 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.823+0000 7fb172600640 1 --2- 172.21.15.90:0/208881555 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb16c074850 0x7fb16c101f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:21.821 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.823+0000 7fb172600640 1 -- 172.21.15.90:0/208881555 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb16c073f30 msgr2=0x7fb16c1019e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:21.821 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.823+0000 7fb172600640 1 --2- 172.21.15.90:0/208881555 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb16c073f30 0x7fb16c1019e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:21.821 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.823+0000 7fb172600640 1 -- 172.21.15.90:0/208881555 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb16c1aac80 con 0x7fb16c10e2c0 2024-04-02T14:44:21.821 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.823+0000 7fb172600640 1 --2- 172.21.15.90:0/208881555 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb16c10e2c0 0x7fb16c102460 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7fb16400ea80 tx=0x7fb16400ef40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:44:21.821 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.823+0000 7fb16a000640 1 -- 172.21.15.90:0/208881555 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb16400cea0 con 0x7fb16c10e2c0 2024-04-02T14:44:21.821 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.823+0000 7fb174600640 1 -- 172.21.15.90:0/208881555 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb16c1aaf70 con 0x7fb16c10e2c0 2024-04-02T14:44:21.822 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.823+0000 7fb16a000640 1 -- 172.21.15.90:0/208881555 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb164005020 con 0x7fb16c10e2c0 2024-04-02T14:44:21.822 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.823+0000 7fb16a000640 1 -- 172.21.15.90:0/208881555 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1640106d0 con 0x7fb16c10e2c0 2024-04-02T14:44:21.822 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.823+0000 7fb174600640 1 -- 172.21.15.90:0/208881555 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb16c1ab490 con 0x7fb16c10e2c0 2024-04-02T14:44:21.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.827+0000 7fb17739d740 1 Processor -- start 2024-04-02T14:44:21.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.827+0000 7fb16a000640 1 -- 172.21.15.90:0/208881555 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fb164005540 con 0x7fb16c10e2c0 2024-04-02T14:44:21.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.827+0000 7fb17739d740 1 -- start start 2024-04-02T14:44:21.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.827+0000 7fb17739d740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55be9e8b89f0 0x55be9e8b7740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:21.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.827+0000 7fb17739d740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55be9e8b7d10 0x55be9e8b8190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:21.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.827+0000 7fb17739d740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55be9e8c1080 0x55be9e8c3470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:21.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.827+0000 7fb17739d740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55be9e7f8400 con 0x55be9e8b89f0 2024-04-02T14:44:21.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.827+0000 7fb17739d740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55be9e7dcf10 con 0x55be9e8c1080 2024-04-02T14:44:21.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.827+0000 7fb17739d740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55be9e81c6f0 con 0x55be9e8b7d10 2024-04-02T14:44:21.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.828+0000 7fb171200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55be9e8b89f0 0x55be9e8b7740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:21.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.828+0000 7fb171200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55be9e8b89f0 0x55be9e8b7740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:45070/0 (socket says 172.21.15.90:45070) 2024-04-02T14:44:21.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.828+0000 7fb171200640 1 -- 172.21.15.90:0/1914381251 learned_addr learned my addr 172.21.15.90:0/1914381251 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:44:21.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.828+0000 7fb171c00640 1 --2- 172.21.15.90:0/1914381251 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55be9e8b7d10 0x55be9e8b8190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:21.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.828+0000 7fb171200640 1 --2- 172.21.15.90:0/1914381251 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55be9e8c1080 0x55be9e8c3470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:21.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.828+0000 7fb171c00640 1 -- 172.21.15.90:0/1914381251 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55be9e8c1080 msgr2=0x55be9e8c3470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:44:21.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.828+0000 7fb171c00640 1 --2- 172.21.15.90:0/1914381251 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55be9e8c1080 0x55be9e8c3470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:21.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.828+0000 7fb171c00640 1 -- 172.21.15.90:0/1914381251 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55be9e8b89f0 msgr2=0x55be9e8b7740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:44:21.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.828+0000 7fb171c00640 1 --2- 172.21.15.90:0/1914381251 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55be9e8b89f0 0x55be9e8b7740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:21.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.828+0000 7fb171c00640 1 -- 172.21.15.90:0/1914381251 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55be9e5303f0 con 0x55be9e8b7d10 2024-04-02T14:44:21.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.828+0000 7fb171200640 1 --2- 172.21.15.90:0/1914381251 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55be9e8b89f0 0x55be9e8b7740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:44:21.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.828+0000 7fb171c00640 1 --2- 172.21.15.90:0/1914381251 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55be9e8b7d10 0x55be9e8b8190 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7fb16002c210 tx=0x7fb160009560 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e039f9a8fb871c57 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:44:21.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.829+0000 7fb15ea00640 1 -- 172.21.15.90:0/1914381251 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb160043070 con 0x55be9e8b7d10 2024-04-02T14:44:21.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.829+0000 7fb15ea00640 1 -- 172.21.15.90:0/1914381251 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb16002c6f0 con 0x55be9e8b7d10 2024-04-02T14:44:21.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.829+0000 7fb15ea00640 1 -- 172.21.15.90:0/1914381251 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb160004a40 con 0x55be9e8b7d10 2024-04-02T14:44:21.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.829+0000 7fb16a000640 1 --2- 172.21.15.90:0/208881555 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fb1540a3da0 0x7fb1540a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:21.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.829+0000 7fb16a000640 1 -- 172.21.15.90:0/208881555 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8550+0+0 (secure 0 0 0) 0x7fb1640d0e40 con 0x7fb16c10e2c0 2024-04-02T14:44:21.828 INFO:teuthology.orchestra.run.smithi090.stderr:progress: 530.0 s, 1692.0 tps, lat 59.436 ms stddev 71.247 2024-04-02T14:44:21.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.830+0000 7fb17739d740 1 -- 172.21.15.90:0/1914381251 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55be9e8b7d10 msgr2=0x55be9e8b8190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:21.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.830+0000 7fb171200640 1 --2- 172.21.15.90:0/208881555 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fb1540a3da0 0x7fb1540a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:21.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.830+0000 7fb17739d740 1 --2- 172.21.15.90:0/1914381251 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55be9e8b7d10 0x55be9e8b8190 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7fb16002c210 tx=0x7fb160009560 comp rx=0 tx=0).stop 2024-04-02T14:44:21.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.830+0000 7fb17739d740 1 -- 172.21.15.90:0/1914381251 shutdown_connections 2024-04-02T14:44:21.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.830+0000 7fb17739d740 1 --2- 172.21.15.90:0/1914381251 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55be9e8c1080 0x55be9e8c3470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:21.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.830+0000 7fb17739d740 1 --2- 172.21.15.90:0/1914381251 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55be9e8b7d10 0x55be9e8b8190 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:21.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.830+0000 7fb17739d740 1 --2- 172.21.15.90:0/1914381251 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55be9e8b89f0 0x55be9e8b7740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:21.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.830+0000 7fb17739d740 1 -- 172.21.15.90:0/1914381251 >> 172.21.15.90:0/1914381251 conn(0x55be9e87ea50 msgr2=0x55be9e8b2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:44:21.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.831+0000 7fb171200640 1 --2- 172.21.15.90:0/208881555 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fb1540a3da0 0x7fb1540a6260 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fb1580097f0 tx=0x7fb158009910 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:44:21.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.831+0000 7fb17739d740 1 -- 172.21.15.90:0/1914381251 shutdown_connections 2024-04-02T14:44:21.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.831+0000 7fb17739d740 1 -- 172.21.15.90:0/1914381251 wait complete. 2024-04-02T14:44:21.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.832+0000 7fb17739d740 1 -- start start 2024-04-02T14:44:21.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.832+0000 7fb17739d740 1 Processor -- start 2024-04-02T14:44:21.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.833+0000 7fb17739d740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55be9e8b7d10 0x55be9e8bdfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:21.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.833+0000 7fb17739d740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55be9e8b89f0 0x55be9e8be510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:21.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.833+0000 7fb17739d740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55be9e8c1080 0x55be9e8bea50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:21.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.833+0000 7fb17739d740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55be9e8186a0 con 0x55be9e8b89f0 2024-04-02T14:44:21.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.833+0000 7fb17739d740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55be9e7c4980 con 0x55be9e8c1080 2024-04-02T14:44:21.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.833+0000 7fb17739d740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55be9e7d7cb0 con 0x55be9e8b7d10 2024-04-02T14:44:21.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.833+0000 7fb17739d740 10 client.0 _ll_register_callbacks cb 0x55be9e7d76a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:44:21.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.833+0000 7fb171c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55be9e8b7d10 0x55be9e8bdfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:21.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.833+0000 7fb171c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55be9e8b7d10 0x55be9e8bdfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:39402/0 (socket says 172.21.15.90:39402) 2024-04-02T14:44:21.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.833+0000 7fb171c00640 1 -- 172.21.15.90:0/3692709496 learned_addr learned my addr 172.21.15.90:0/3692709496 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:44:21.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.833+0000 7fb171200640 1 --2- 172.21.15.90:0/3692709496 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55be9e8b89f0 0x55be9e8be510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:21.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.834+0000 7fb172600640 1 --2- 172.21.15.90:0/3692709496 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55be9e8c1080 0x55be9e8bea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:21.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.834+0000 7fb171c00640 1 -- 172.21.15.90:0/3692709496 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55be9e8c1080 msgr2=0x55be9e8bea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:44:21.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.834+0000 7fb171c00640 1 --2- 172.21.15.90:0/3692709496 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55be9e8c1080 0x55be9e8bea50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:21.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.834+0000 7fb171c00640 1 -- 172.21.15.90:0/3692709496 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55be9e8b89f0 msgr2=0x55be9e8be510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:44:21.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.834+0000 7fb171c00640 1 --2- 172.21.15.90:0/3692709496 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55be9e8b89f0 0x55be9e8be510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:21.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.834+0000 7fb171c00640 1 -- 172.21.15.90:0/3692709496 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55be9e5fa7d0 con 0x55be9e8b7d10 2024-04-02T14:44:21.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.834+0000 7fb172600640 1 --2- 172.21.15.90:0/3692709496 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55be9e8c1080 0x55be9e8bea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:44:21.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.834+0000 7fb171200640 1 --2- 172.21.15.90:0/3692709496 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55be9e8b89f0 0x55be9e8be510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:44:21.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.834+0000 7fb171c00640 1 --2- 172.21.15.90:0/3692709496 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55be9e8b7d10 0x55be9e8bdfd0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7fb160005ab0 tx=0x7fb1600049f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1f15ec86345f93eb server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:44:21.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.834+0000 7fb153400640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:44:21.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.834+0000 7fb153400640 1 -- 172.21.15.90:0/3692709496 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb16004c070 con 0x55be9e8b7d10 2024-04-02T14:44:21.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.834+0000 7fb153400640 1 -- 172.21.15.90:0/3692709496 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8550+0+0 (secure 0 0 0) 0x7fb16002cb20 con 0x55be9e8b7d10 2024-04-02T14:44:21.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.834+0000 7fb17739d740 1 -- 172.21.15.90:0/3692709496 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb16002c6f0 con 0x55be9e8b7d10 2024-04-02T14:44:21.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.834+0000 7fb153400640 1 -- 172.21.15.90:0/3692709496 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fb160058050 con 0x55be9e8b7d10 2024-04-02T14:44:21.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.834+0000 7fb17739d740 10 client.28243 fetch_fsmap learned FSMap version 27 2024-04-02T14:44:21.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.835+0000 7fb17739d740 1 -- 172.21.15.90:0/3692709496 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55be9e8cfea0 con 0x55be9e8b7d10 2024-04-02T14:44:21.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.835+0000 7fb153400640 1 -- 172.21.15.90:0/3692709496 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb16002c6f0 con 0x55be9e8b7d10 2024-04-02T14:44:21.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.835+0000 7fb153400640 1 -- 172.21.15.90:0/3692709496 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fb160003990 con 0x55be9e8b7d10 2024-04-02T14:44:21.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.835+0000 7fb17739d740 10 client.28243 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:44:21.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.835+0000 7fb17739d740 10 client.28243 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-04-02T14:44:21.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.835+0000 7fb17739d740 20 client.28243 populate_metadata read hostname 'smithi090' 2024-04-02T14:44:21.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.835+0000 7fb17739d740 1 --2- 172.21.15.90:0/3692709496 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x55be9e8d11f0 0x55be9e8d35e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:21.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.835+0000 7fb17739d740 4 client.28243 mds_command: new command op to 24547 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:44:21.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.835+0000 7fb17739d740 1 -- 172.21.15.90:0/3692709496 --> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55be9e7dc040 con 0x55be9e8d11f0 2024-04-02T14:44:21.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.836+0000 7fb171200640 1 --2- 172.21.15.90:0/3692709496 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x55be9e8d11f0 0x55be9e8d35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:21.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.836+0000 7fb171200640 1 --2- 172.21.15.90:0/3692709496 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x55be9e8d11f0 0x55be9e8d35e0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=3526d8e2be379470 server_cookie=9ad69c6094130d0 in_seq=0 out_seq=0 2024-04-02T14:44:21.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.836+0000 7fb153400640 10 client.28243 ms_handle_connect on v2:172.21.15.177:6838/664895663 2024-04-02T14:44:21.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.837+0000 7fb153400640 1 -- 172.21.15.90:0/3692709496 <== mds.2 v2:172.21.15.177:6838/664895663 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55be9e7dc040 con 0x55be9e8d11f0 2024-04-02T14:44:21.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.837+0000 7fb153400640 10 client.28243 handle_command_reply: tid=0 2024-04-02T14:44:21.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.837+0000 7fb17739d740 1 client.28243 shutdown 2024-04-02T14:44:21.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.837+0000 7fb17739d740 1 -- 172.21.15.90:0/3692709496 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55be9e8b7d10 msgr2=0x55be9e8bdfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:21.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.837+0000 7fb17739d740 1 --2- 172.21.15.90:0/3692709496 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55be9e8b7d10 0x55be9e8bdfd0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7fb160005ab0 tx=0x7fb1600049f0 comp rx=0 tx=0).stop 2024-04-02T14:44:21.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.837+0000 7fb17739d740 1 -- 172.21.15.90:0/3692709496 shutdown_connections 2024-04-02T14:44:21.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.837+0000 7fb17739d740 1 --2- 172.21.15.90:0/3692709496 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x55be9e8d11f0 0x55be9e8d35e0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:21.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.837+0000 7fb17739d740 1 --2- 172.21.15.90:0/3692709496 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55be9e8c1080 0x55be9e8bea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:21.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.837+0000 7fb17739d740 1 --2- 172.21.15.90:0/3692709496 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55be9e8b89f0 0x55be9e8be510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:21.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.837+0000 7fb17739d740 1 --2- 172.21.15.90:0/3692709496 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55be9e8b7d10 0x55be9e8bdfd0 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:21.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.837+0000 7fb153400640 0 client.28243 ms_handle_reset on v2:172.21.15.177:6838/664895663 2024-04-02T14:44:21.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.837+0000 7fb17739d740 1 -- 172.21.15.90:0/3692709496 >> 172.21.15.90:0/3692709496 conn(0x55be9e87ea50 msgr2=0x55be9e8b44b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:44:21.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.837+0000 7fb17739d740 1 -- 172.21.15.90:0/3692709496 shutdown_connections 2024-04-02T14:44:21.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.837+0000 7fb17739d740 1 -- 172.21.15.90:0/3692709496 wait complete. 2024-04-02T14:44:21.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.837+0000 7fb17739d740 20 client.28243 trim_cache size 0 max 16384 2024-04-02T14:44:21.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.853+0000 7fb17739d740 1 Processor -- start 2024-04-02T14:44:21.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.854+0000 7fb17739d740 1 -- start start 2024-04-02T14:44:21.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.854+0000 7fb17739d740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55be9e8b7d10 0x55be9e8c0700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:21.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.854+0000 7fb17739d740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55be9e8b89f0 0x55be9e8b42c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:21.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.854+0000 7fb17739d740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55be9e8c1080 0x55be9e8c2470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:21.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.854+0000 7fb17739d740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fb16c1b3e10 con 0x55be9e8b7d10 2024-04-02T14:44:21.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.854+0000 7fb17739d740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fb16c1b5950 con 0x55be9e8c1080 2024-04-02T14:44:21.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.854+0000 7fb17739d740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fb16c1b44e0 con 0x55be9e8b89f0 2024-04-02T14:44:21.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.855+0000 7fb171c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55be9e8b7d10 0x55be9e8c0700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:21.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.855+0000 7fb172600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55be9e8c1080 0x55be9e8c2470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:21.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.855+0000 7fb172600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55be9e8c1080 0x55be9e8c2470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:38986/0 (socket says 172.21.15.90:38986) 2024-04-02T14:44:21.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.855+0000 7fb172600640 1 -- 172.21.15.90:0/399604236 learned_addr learned my addr 172.21.15.90:0/399604236 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:44:21.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.856+0000 7fb171200640 1 --2- 172.21.15.90:0/399604236 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55be9e8b89f0 0x55be9e8b42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:21.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.856+0000 7fb172600640 1 -- 172.21.15.90:0/399604236 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55be9e8b89f0 msgr2=0x55be9e8b42c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:44:21.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.856+0000 7fb172600640 1 --2- 172.21.15.90:0/399604236 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55be9e8b89f0 0x55be9e8b42c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:21.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.856+0000 7fb172600640 1 -- 172.21.15.90:0/399604236 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55be9e8b7d10 msgr2=0x55be9e8c0700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:44:21.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.856+0000 7fb172600640 1 --2- 172.21.15.90:0/399604236 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55be9e8b7d10 0x55be9e8c0700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:21.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.856+0000 7fb172600640 1 -- 172.21.15.90:0/399604236 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55be9e8b6d40 con 0x55be9e8c1080 2024-04-02T14:44:21.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.856+0000 7fb171c00640 1 --2- 172.21.15.90:0/399604236 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55be9e8b7d10 0x55be9e8c0700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:44:21.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.856+0000 7fb172600640 1 --2- 172.21.15.90:0/399604236 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55be9e8c1080 0x55be9e8c2470 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7fb16409ccf0 tx=0x7fb16409d5a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6e326f0b29687375 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:44:21.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.857+0000 7fb15e000640 1 -- 172.21.15.90:0/399604236 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1640a59d0 con 0x55be9e8c1080 2024-04-02T14:44:21.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.857+0000 7fb15e000640 1 -- 172.21.15.90:0/399604236 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb1640b6270 con 0x55be9e8c1080 2024-04-02T14:44:21.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.857+0000 7fb15e000640 1 -- 172.21.15.90:0/399604236 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb16409cfc0 con 0x55be9e8c1080 2024-04-02T14:44:21.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.857+0000 7fb17739d740 1 -- 172.21.15.90:0/399604236 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55be9e8c1080 msgr2=0x55be9e8c2470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:21.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.857+0000 7fb17739d740 1 --2- 172.21.15.90:0/399604236 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55be9e8c1080 0x55be9e8c2470 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7fb16409ccf0 tx=0x7fb16409d5a0 comp rx=0 tx=0).stop 2024-04-02T14:44:21.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.857+0000 7fb17739d740 1 -- 172.21.15.90:0/399604236 shutdown_connections 2024-04-02T14:44:21.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.857+0000 7fb17739d740 1 --2- 172.21.15.90:0/399604236 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55be9e8c1080 0x55be9e8c2470 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:21.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.857+0000 7fb17739d740 1 --2- 172.21.15.90:0/399604236 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55be9e8b89f0 0x55be9e8b42c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:21.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.857+0000 7fb17739d740 1 --2- 172.21.15.90:0/399604236 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55be9e8b7d10 0x55be9e8c0700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:21.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.857+0000 7fb17739d740 1 -- 172.21.15.90:0/399604236 >> 172.21.15.90:0/399604236 conn(0x55be9e87ea50 msgr2=0x55be9e8b3bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:44:21.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.858+0000 7fb17739d740 1 -- 172.21.15.90:0/399604236 shutdown_connections 2024-04-02T14:44:21.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.858+0000 7fb17739d740 1 -- 172.21.15.90:0/399604236 wait complete. 2024-04-02T14:44:21.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.858+0000 7fb17739d740 1 -- start start 2024-04-02T14:44:21.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.858+0000 7fb17739d740 1 Processor -- start 2024-04-02T14:44:21.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.858+0000 7fb17739d740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55be9e8b7d10 0x55be9e8d21c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:21.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.858+0000 7fb17739d740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55be9e8b89f0 0x55be9e8d2700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:21.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.859+0000 7fb17739d740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55be9e8c1080 0x55be9e8d2c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:21.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.859+0000 7fb17739d740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fb16c1b5280 con 0x55be9e8c1080 2024-04-02T14:44:21.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.859+0000 7fb17739d740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fb16c1b4bb0 con 0x55be9e8b89f0 2024-04-02T14:44:21.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.859+0000 7fb17739d740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fb16c1b1e10 con 0x55be9e8b7d10 2024-04-02T14:44:21.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.859+0000 7fb17739d740 10 client.0 _ll_register_callbacks cb 0x7fb160004370 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:44:21.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.859+0000 7fb171c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55be9e8b7d10 0x55be9e8d21c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:21.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.859+0000 7fb171c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55be9e8b7d10 0x55be9e8d21c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:39416/0 (socket says 172.21.15.90:39416) 2024-04-02T14:44:21.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.859+0000 7fb171c00640 1 -- 172.21.15.90:0/24655023 learned_addr learned my addr 172.21.15.90:0/24655023 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:44:21.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.859+0000 7fb172600640 1 --2- 172.21.15.90:0/24655023 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55be9e8c1080 0x55be9e8d2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:21.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.859+0000 7fb171200640 1 --2- 172.21.15.90:0/24655023 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55be9e8b89f0 0x55be9e8d2700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:21.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.859+0000 7fb171c00640 1 -- 172.21.15.90:0/24655023 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55be9e8b89f0 msgr2=0x55be9e8d2700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:44:21.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.859+0000 7fb171c00640 1 --2- 172.21.15.90:0/24655023 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55be9e8b89f0 0x55be9e8d2700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:21.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.859+0000 7fb171c00640 1 -- 172.21.15.90:0/24655023 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55be9e8c1080 msgr2=0x55be9e8d2c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:44:21.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.859+0000 7fb171c00640 1 --2- 172.21.15.90:0/24655023 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55be9e8c1080 0x55be9e8d2c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:21.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.859+0000 7fb171c00640 1 -- 172.21.15.90:0/24655023 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55be9e936590 con 0x55be9e8b7d10 2024-04-02T14:44:21.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.859+0000 7fb171c00640 1 --2- 172.21.15.90:0/24655023 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55be9e8b7d10 0x55be9e8d21c0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7fb160005a80 tx=0x7fb16002efe0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ca8435583186d3a8 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:44:21.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.859+0000 7fb153400640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:44:21.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.860+0000 7fb153400640 1 -- 172.21.15.90:0/24655023 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb16004c040 con 0x55be9e8b7d10 2024-04-02T14:44:21.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.860+0000 7fb153400640 1 -- 172.21.15.90:0/24655023 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8550+0+0 (secure 0 0 0) 0x7fb16002c6f0 con 0x55be9e8b7d10 2024-04-02T14:44:21.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.860+0000 7fb17739d740 1 -- 172.21.15.90:0/24655023 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb1640b6270 con 0x55be9e8b7d10 2024-04-02T14:44:21.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.860+0000 7fb153400640 1 -- 172.21.15.90:0/24655023 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fb160043070 con 0x55be9e8b7d10 2024-04-02T14:44:21.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.860+0000 7fb17739d740 10 client.28249 fetch_fsmap learned FSMap version 27 2024-04-02T14:44:21.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.860+0000 7fb17739d740 1 -- 172.21.15.90:0/24655023 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55be9e937850 con 0x55be9e8b7d10 2024-04-02T14:44:21.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.860+0000 7fb153400640 1 -- 172.21.15.90:0/24655023 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb1640b6270 con 0x55be9e8b7d10 2024-04-02T14:44:21.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.861+0000 7fb153400640 1 -- 172.21.15.90:0/24655023 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fb160048450 con 0x55be9e8b7d10 2024-04-02T14:44:21.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.861+0000 7fb17739d740 10 client.28249 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:44:21.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.861+0000 7fb17739d740 10 client.28249 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-04-02T14:44:21.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.861+0000 7fb17739d740 20 client.28249 populate_metadata read hostname 'smithi090' 2024-04-02T14:44:21.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.861+0000 7fb17739d740 1 --2- 172.21.15.90:0/24655023 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x55be9e937e00 0x55be9e93a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:21.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.861+0000 7fb17739d740 4 client.28249 mds_command: new command op to 24547 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-02T14:44:21.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.861+0000 7fb17739d740 1 -- 172.21.15.90:0/24655023 --> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55be9e829c30 con 0x55be9e937e00 2024-04-02T14:44:21.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.863+0000 7fb171200640 1 --2- 172.21.15.90:0/24655023 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x55be9e937e00 0x55be9e93a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:21.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.863+0000 7fb171200640 1 --2- 172.21.15.90:0/24655023 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x55be9e937e00 0x55be9e93a1f0 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=ec06f23647f769d9 server_cookie=d35266fd4088e708 in_seq=0 out_seq=0 2024-04-02T14:44:21.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.863+0000 7fb153400640 10 client.28249 ms_handle_connect on v2:172.21.15.177:6838/664895663 2024-04-02T14:44:21.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.863+0000 7fb153400640 1 -- 172.21.15.90:0/24655023 <== mds.2 v2:172.21.15.177:6838/664895663 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55be9e829c30 con 0x55be9e937e00 2024-04-02T14:44:21.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.863+0000 7fb153400640 10 client.28249 handle_command_reply: tid=0 2024-04-02T14:44:21.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.863+0000 7fb17739d740 1 client.28249 shutdown 2024-04-02T14:44:21.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.864+0000 7fb17739d740 1 -- 172.21.15.90:0/24655023 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55be9e8b7d10 msgr2=0x55be9e8d21c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:21.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.864+0000 7fb17739d740 1 --2- 172.21.15.90:0/24655023 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55be9e8b7d10 0x55be9e8d21c0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7fb160005a80 tx=0x7fb16002efe0 comp rx=0 tx=0).stop 2024-04-02T14:44:21.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.864+0000 7fb17739d740 1 -- 172.21.15.90:0/24655023 shutdown_connections 2024-04-02T14:44:21.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.864+0000 7fb17739d740 1 --2- 172.21.15.90:0/24655023 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x55be9e937e00 0x55be9e93a1f0 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:21.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.864+0000 7fb17739d740 1 --2- 172.21.15.90:0/24655023 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55be9e8c1080 0x55be9e8d2c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:21.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.864+0000 7fb17739d740 1 --2- 172.21.15.90:0/24655023 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55be9e8b89f0 0x55be9e8d2700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:21.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.864+0000 7fb17739d740 1 --2- 172.21.15.90:0/24655023 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55be9e8b7d10 0x55be9e8d21c0 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:21.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.864+0000 7fb153400640 0 client.28249 ms_handle_reset on v2:172.21.15.177:6838/664895663 2024-04-02T14:44:21.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.864+0000 7fb17739d740 1 -- 172.21.15.90:0/24655023 >> 172.21.15.90:0/24655023 conn(0x55be9e87ea50 msgr2=0x55be9e8b18a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:44:21.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.864+0000 7fb17739d740 1 -- 172.21.15.90:0/24655023 shutdown_connections 2024-04-02T14:44:21.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.864+0000 7fb17739d740 1 -- 172.21.15.90:0/24655023 wait complete. 2024-04-02T14:44:21.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.864+0000 7fb17739d740 20 client.28249 trim_cache size 0 max 16384 2024-04-02T14:44:21.867 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-02T14:44:21.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.865+0000 7fb174600640 1 -- 172.21.15.90:0/208881555 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fb1540a3da0 msgr2=0x7fb1540a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:21.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.865+0000 7fb174600640 1 --2- 172.21.15.90:0/208881555 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fb1540a3da0 0x7fb1540a6260 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fb1580097f0 tx=0x7fb158009910 comp rx=0 tx=0).stop 2024-04-02T14:44:21.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.865+0000 7fb174600640 1 -- 172.21.15.90:0/208881555 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb16c10e2c0 msgr2=0x7fb16c102460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:21.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.865+0000 7fb174600640 1 --2- 172.21.15.90:0/208881555 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb16c10e2c0 0x7fb16c102460 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7fb16400ea80 tx=0x7fb16400ef40 comp rx=0 tx=0).stop 2024-04-02T14:44:21.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.865+0000 7fb174600640 1 -- 172.21.15.90:0/208881555 shutdown_connections 2024-04-02T14:44:21.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.865+0000 7fb174600640 1 --2- 172.21.15.90:0/208881555 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fb1540a3da0 0x7fb1540a6260 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:21.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.865+0000 7fb174600640 1 --2- 172.21.15.90:0/208881555 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb16c10e2c0 0x7fb16c102460 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:21.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.865+0000 7fb174600640 1 --2- 172.21.15.90:0/208881555 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb16c074850 0x7fb16c101f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:21.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.865+0000 7fb174600640 1 --2- 172.21.15.90:0/208881555 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb16c073f30 0x7fb16c1019e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:21.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.865+0000 7fb174600640 1 -- 172.21.15.90:0/208881555 >> 172.21.15.90:0/208881555 conn(0x7fb16c0fd590 msgr2=0x7fb16c0fe340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:44:21.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.865+0000 7fb174600640 1 -- 172.21.15.90:0/208881555 shutdown_connections 2024-04-02T14:44:21.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:21.865+0000 7fb174600640 1 -- 172.21.15.90:0/208881555 wait complete. 2024-04-02T14:44:21.891 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-02T14:44:22.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.173+0000 7fe1c1400640 1 -- 172.21.15.90:0/1162929706 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe1bc10e350 msgr2=0x7fe1bc10e7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:22.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.173+0000 7fe1c1400640 1 --2- 172.21.15.90:0/1162929706 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe1bc10e350 0x7fe1bc10e7d0 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7fe1a8009900 tx=0x7fe1a802bc30 comp rx=0 tx=0).stop 2024-04-02T14:44:22.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.174+0000 7fe1c1400640 1 -- 172.21.15.90:0/1162929706 shutdown_connections 2024-04-02T14:44:22.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.174+0000 7fe1c1400640 1 --2- 172.21.15.90:0/1162929706 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe1bc11b2e0 0x7fe1bc11d6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:22.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.174+0000 7fe1c1400640 1 --2- 172.21.15.90:0/1162929706 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe1bc10e350 0x7fe1bc10e7d0 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:22.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.174+0000 7fe1c1400640 1 --2- 172.21.15.90:0/1162929706 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe1bc110320 0x7fe1bc10de10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:22.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.174+0000 7fe1c1400640 1 -- 172.21.15.90:0/1162929706 >> 172.21.15.90:0/1162929706 conn(0x7fe1bc06cbe0 msgr2=0x7fe1bc06cff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:44:22.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.174+0000 7fe1c1400640 1 -- 172.21.15.90:0/1162929706 shutdown_connections 2024-04-02T14:44:22.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.174+0000 7fe1c1400640 1 -- 172.21.15.90:0/1162929706 wait complete. 2024-04-02T14:44:22.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.176+0000 7fe1c1400640 1 Processor -- start 2024-04-02T14:44:22.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.176+0000 7fe1c1400640 1 -- start start 2024-04-02T14:44:22.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.176+0000 7fe1c1400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe1bc10e350 0x7fe1bc1a7190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:22.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.176+0000 7fe1c1400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe1bc110320 0x7fe1bc1a76d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:22.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.176+0000 7fe1c1400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe1bc11b2e0 0x7fe1bc1aba50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:22.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.176+0000 7fe1c1400640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fe1bc120cd0 con 0x7fe1bc11b2e0 2024-04-02T14:44:22.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.176+0000 7fe1c1400640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fe1bc120b50 con 0x7fe1bc110320 2024-04-02T14:44:22.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.176+0000 7fe1c1400640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fe1bc120e50 con 0x7fe1bc10e350 2024-04-02T14:44:22.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.177+0000 7fe1b9e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe1bc110320 0x7fe1bc1a76d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:22.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.177+0000 7fe1b9e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe1bc110320 0x7fe1bc1a76d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:39014/0 (socket says 172.21.15.90:39014) 2024-04-02T14:44:22.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.177+0000 7fe1b9e00640 1 -- 172.21.15.90:0/3513824508 learned_addr learned my addr 172.21.15.90:0/3513824508 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:44:22.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.177+0000 7fe1ba800640 1 --2- 172.21.15.90:0/3513824508 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe1bc10e350 0x7fe1bc1a7190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:22.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.177+0000 7fe1b9e00640 1 -- 172.21.15.90:0/3513824508 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe1bc10e350 msgr2=0x7fe1bc1a7190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:22.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.177+0000 7fe1bb200640 1 --2- 172.21.15.90:0/3513824508 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe1bc11b2e0 0x7fe1bc1aba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:22.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.177+0000 7fe1b9e00640 1 --2- 172.21.15.90:0/3513824508 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe1bc10e350 0x7fe1bc1a7190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:22.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.177+0000 7fe1b9e00640 1 -- 172.21.15.90:0/3513824508 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe1bc11b2e0 msgr2=0x7fe1bc1aba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:22.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.177+0000 7fe1b9e00640 1 --2- 172.21.15.90:0/3513824508 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe1bc11b2e0 0x7fe1bc1aba50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:22.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.177+0000 7fe1b9e00640 1 -- 172.21.15.90:0/3513824508 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1bc1ac180 con 0x7fe1bc110320 2024-04-02T14:44:22.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.178+0000 7fe1b9e00640 1 --2- 172.21.15.90:0/3513824508 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe1bc110320 0x7fe1bc1a76d0 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7fe1a802c240 tx=0x7fe1a80042f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:44:22.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.178+0000 7fe1b3400640 1 -- 172.21.15.90:0/3513824508 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1a8043020 con 0x7fe1bc110320 2024-04-02T14:44:22.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.178+0000 7fe1c1400640 1 -- 172.21.15.90:0/3513824508 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe1bc1ac410 con 0x7fe1bc110320 2024-04-02T14:44:22.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.178+0000 7fe1c1400640 1 -- 172.21.15.90:0/3513824508 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1bc1b3d00 con 0x7fe1bc110320 2024-04-02T14:44:22.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.178+0000 7fe1b3400640 1 -- 172.21.15.90:0/3513824508 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe1a80054e0 con 0x7fe1bc110320 2024-04-02T14:44:22.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.178+0000 7fe1b3400640 1 -- 172.21.15.90:0/3513824508 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1a80038d0 con 0x7fe1bc110320 2024-04-02T14:44:22.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.185+0000 7fe1c4165740 1 Processor -- start 2024-04-02T14:44:22.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.185+0000 7fe1c4165740 1 -- start start 2024-04-02T14:44:22.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.186+0000 7fe1c4165740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559a1b7b19f0 0x559a1b7b0740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:22.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.186+0000 7fe1c4165740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559a1b7b0d10 0x559a1b7b1190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:22.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.186+0000 7fe1c4165740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559a1b7ba080 0x559a1b7bc470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:22.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.186+0000 7fe1c4165740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x559a1b6f1400 con 0x559a1b7ba080 2024-04-02T14:44:22.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.186+0000 7fe1c4165740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x559a1b6d5f10 con 0x559a1b7b0d10 2024-04-02T14:44:22.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.186+0000 7fe1c4165740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x559a1b7156f0 con 0x559a1b7b19f0 2024-04-02T14:44:22.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.186+0000 7fe1b3400640 1 -- 172.21.15.90:0/3513824508 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fe1a8046020 con 0x7fe1bc110320 2024-04-02T14:44:22.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.186+0000 7fe1bb200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559a1b7b19f0 0x559a1b7b0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:22.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.186+0000 7fe1bb200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559a1b7b19f0 0x559a1b7b0740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:39446/0 (socket says 172.21.15.90:39446) 2024-04-02T14:44:22.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.186+0000 7fe1bb200640 1 -- 172.21.15.90:0/1277194493 learned_addr learned my addr 172.21.15.90:0/1277194493 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:44:22.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.187+0000 7fe1ba800640 1 --2- 172.21.15.90:0/1277194493 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559a1b7b0d10 0x559a1b7b1190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:22.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.187+0000 7fe1bb200640 1 -- 172.21.15.90:0/1277194493 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559a1b7b0d10 msgr2=0x559a1b7b1190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:44:22.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.187+0000 7fe1bb200640 1 --2- 172.21.15.90:0/1277194493 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559a1b7b0d10 0x559a1b7b1190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:22.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.187+0000 7fe1bb200640 1 -- 172.21.15.90:0/1277194493 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559a1b7ba080 msgr2=0x559a1b7bc470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:44:22.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.187+0000 7fe1bb200640 1 --2- 172.21.15.90:0/1277194493 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559a1b7ba080 0x559a1b7bc470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:22.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.187+0000 7fe1bb200640 1 -- 172.21.15.90:0/1277194493 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559a1b4293f0 con 0x559a1b7b19f0 2024-04-02T14:44:22.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.187+0000 7fe1ba800640 1 --2- 172.21.15.90:0/1277194493 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559a1b7b0d10 0x559a1b7b1190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:44:22.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.187+0000 7fe1bb200640 1 --2- 172.21.15.90:0/1277194493 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559a1b7b19f0 0x559a1b7b0740 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fe1b4009e30 tx=0x7fe1b400f730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=db8593294eee3829 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:44:22.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.188+0000 7fe1a7400640 1 -- 172.21.15.90:0/1277194493 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1b4011640 con 0x559a1b7b19f0 2024-04-02T14:44:22.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.188+0000 7fe1a7400640 1 -- 172.21.15.90:0/1277194493 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe1b4011cc0 con 0x559a1b7b19f0 2024-04-02T14:44:22.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.188+0000 7fe1a7400640 1 -- 172.21.15.90:0/1277194493 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1b401bd00 con 0x559a1b7b19f0 2024-04-02T14:44:22.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.188+0000 7fe1b3400640 1 --2- 172.21.15.90:0/3513824508 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fe1a00a3d20 0x7fe1a00a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:22.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.188+0000 7fe1c4165740 1 -- 172.21.15.90:0/1277194493 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559a1b7b19f0 msgr2=0x559a1b7b0740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:22.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.188+0000 7fe1c4165740 1 --2- 172.21.15.90:0/1277194493 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559a1b7b19f0 0x559a1b7b0740 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fe1b4009e30 tx=0x7fe1b400f730 comp rx=0 tx=0).stop 2024-04-02T14:44:22.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.189+0000 7fe1c4165740 1 -- 172.21.15.90:0/1277194493 shutdown_connections 2024-04-02T14:44:22.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.189+0000 7fe1c4165740 1 --2- 172.21.15.90:0/1277194493 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559a1b7ba080 0x559a1b7bc470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:22.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.189+0000 7fe1c4165740 1 --2- 172.21.15.90:0/1277194493 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559a1b7b0d10 0x559a1b7b1190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:22.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.189+0000 7fe1c4165740 1 --2- 172.21.15.90:0/1277194493 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559a1b7b19f0 0x559a1b7b0740 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:22.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.189+0000 7fe1c4165740 1 -- 172.21.15.90:0/1277194493 >> 172.21.15.90:0/1277194493 conn(0x559a1b777a50 msgr2=0x559a1b7ab760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:44:22.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.189+0000 7fe1b3400640 1 -- 172.21.15.90:0/3513824508 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8550+0+0 (secure 0 0 0) 0x7fe1a80f22d0 con 0x7fe1bc110320 2024-04-02T14:44:22.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.189+0000 7fe1c4165740 1 -- 172.21.15.90:0/1277194493 shutdown_connections 2024-04-02T14:44:22.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.189+0000 7fe1ba800640 1 --2- 172.21.15.90:0/3513824508 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fe1a00a3d20 0x7fe1a00a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:22.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.189+0000 7fe1c4165740 1 -- 172.21.15.90:0/1277194493 wait complete. 2024-04-02T14:44:22.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.190+0000 7fe1c4165740 1 -- start start 2024-04-02T14:44:22.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.190+0000 7fe1c4165740 1 Processor -- start 2024-04-02T14:44:22.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.191+0000 7fe1c4165740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559a1b7b0d10 0x559a1b7b88d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:22.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.191+0000 7fe1c4165740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559a1b7b19f0 0x559a1b7b8e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:22.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.191+0000 7fe1c4165740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559a1b7ba080 0x559a1b7b9350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:22.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.191+0000 7fe1c4165740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x559a1b7116a0 con 0x559a1b7ba080 2024-04-02T14:44:22.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.191+0000 7fe1c4165740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x559a1b6bd980 con 0x559a1b7b19f0 2024-04-02T14:44:22.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.191+0000 7fe1c4165740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x559a1b6d0cb0 con 0x559a1b7b0d10 2024-04-02T14:44:22.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.192+0000 7fe1c4165740 10 client.0 _ll_register_callbacks cb 0x559a1b6d06a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:44:22.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.192+0000 7fe1ba800640 1 --2- 172.21.15.90:0/3513824508 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fe1a00a3d20 0x7fe1a00a61e0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x559a1b7b1f40 tx=0x7fe1ac005f90 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:44:22.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.192+0000 7fe1b9e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559a1b7b19f0 0x559a1b7b8e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:22.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.192+0000 7fe1ba800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559a1b7b0d10 0x559a1b7b88d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:22.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.192+0000 7fe1b9e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559a1b7b19f0 0x559a1b7b8e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:39038/0 (socket says 172.21.15.90:39038) 2024-04-02T14:44:22.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.192+0000 7fe1b9e00640 1 -- 172.21.15.90:0/1836861268 learned_addr learned my addr 172.21.15.90:0/1836861268 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:44:22.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.192+0000 7fe1bb200640 1 --2- 172.21.15.90:0/1836861268 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559a1b7ba080 0x559a1b7b9350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:22.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.193+0000 7fe1ba800640 1 -- 172.21.15.90:0/1836861268 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559a1b7b19f0 msgr2=0x559a1b7b8e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:44:22.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.193+0000 7fe1ba800640 1 --2- 172.21.15.90:0/1836861268 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559a1b7b19f0 0x559a1b7b8e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:22.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.193+0000 7fe1ba800640 1 -- 172.21.15.90:0/1836861268 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559a1b7ba080 msgr2=0x559a1b7b9350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:44:22.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.193+0000 7fe1ba800640 1 --2- 172.21.15.90:0/1836861268 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559a1b7ba080 0x559a1b7b9350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:22.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.193+0000 7fe1ba800640 1 -- 172.21.15.90:0/1836861268 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559a1b4f37d0 con 0x559a1b7b0d10 2024-04-02T14:44:22.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.193+0000 7fe1ba800640 1 --2- 172.21.15.90:0/1836861268 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559a1b7b0d10 0x559a1b7b88d0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fe1ac0138f0 tx=0x7fe1ac0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f8f9ffbf6626bd61 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:44:22.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.193+0000 7fe19be00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:44:22.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.193+0000 7fe19be00640 1 -- 172.21.15.90:0/1836861268 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1ac011ea0 con 0x559a1b7b0d10 2024-04-02T14:44:22.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.193+0000 7fe19be00640 1 -- 172.21.15.90:0/1836861268 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8550+0+0 (secure 0 0 0) 0x7fe1ac01c660 con 0x559a1b7b0d10 2024-04-02T14:44:22.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.194+0000 7fe1c4165740 1 -- 172.21.15.90:0/1836861268 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe1b4011cc0 con 0x559a1b7b0d10 2024-04-02T14:44:22.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.195+0000 7fe19be00640 1 -- 172.21.15.90:0/1836861268 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fe1ac01c860 con 0x559a1b7b0d10 2024-04-02T14:44:22.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.195+0000 7fe1c4165740 10 client.28261 fetch_fsmap learned FSMap version 27 2024-04-02T14:44:22.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.195+0000 7fe1c4165740 1 -- 172.21.15.90:0/1836861268 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x559a1b7c9080 con 0x559a1b7b0d10 2024-04-02T14:44:22.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.195+0000 7fe19be00640 1 -- 172.21.15.90:0/1836861268 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe1b4011cc0 con 0x559a1b7b0d10 2024-04-02T14:44:22.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.196+0000 7fe19be00640 1 -- 172.21.15.90:0/1836861268 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fe1ac025070 con 0x559a1b7b0d10 2024-04-02T14:44:22.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.196+0000 7fe1c4165740 10 client.28261 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:44:22.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.196+0000 7fe1c4165740 10 client.28261 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2024-04-02T14:44:22.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.196+0000 7fe1c4165740 20 client.28261 populate_metadata read hostname 'smithi090' 2024-04-02T14:44:22.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.196+0000 7fe1c4165740 1 --2- 172.21.15.90:0/1836861268 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x559a1b7ca530 0x559a1b7cc9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:22.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.196+0000 7fe1c4165740 4 client.28261 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:44:22.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.196+0000 7fe1c4165740 1 -- 172.21.15.90:0/1836861268 --> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559a1b6d5040 con 0x559a1b7ca530 2024-04-02T14:44:22.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.196+0000 7fe1b9e00640 1 --2- 172.21.15.90:0/1836861268 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x559a1b7ca530 0x559a1b7cc9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:22.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.197+0000 7fe1b9e00640 1 --2- 172.21.15.90:0/1836861268 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x559a1b7ca530 0x559a1b7cc9c0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=3e9a40e207cb6c7d server_cookie=764a431331cc1fb8 in_seq=0 out_seq=0 2024-04-02T14:44:22.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.197+0000 7fe19be00640 10 client.28261 ms_handle_connect on v2:172.21.15.90:6838/1498915870 2024-04-02T14:44:22.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.197+0000 7fe19be00640 1 -- 172.21.15.90:0/1836861268 <== mds.1 v2:172.21.15.90:6838/1498915870 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x559a1b6d5040 con 0x559a1b7ca530 2024-04-02T14:44:22.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.197+0000 7fe19be00640 10 client.28261 handle_command_reply: tid=0 2024-04-02T14:44:22.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.197+0000 7fe1c4165740 1 client.28261 shutdown 2024-04-02T14:44:22.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.198+0000 7fe1c4165740 1 -- 172.21.15.90:0/1836861268 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559a1b7b0d10 msgr2=0x559a1b7b88d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:22.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.198+0000 7fe1c4165740 1 --2- 172.21.15.90:0/1836861268 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559a1b7b0d10 0x559a1b7b88d0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fe1ac0138f0 tx=0x7fe1ac0046c0 comp rx=0 tx=0).stop 2024-04-02T14:44:22.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.198+0000 7fe1c4165740 1 -- 172.21.15.90:0/1836861268 shutdown_connections 2024-04-02T14:44:22.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.198+0000 7fe1c4165740 1 --2- 172.21.15.90:0/1836861268 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x559a1b7ca530 0x559a1b7cc9c0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:22.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.198+0000 7fe1c4165740 1 --2- 172.21.15.90:0/1836861268 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559a1b7ba080 0x559a1b7b9350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:22.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.198+0000 7fe1c4165740 1 --2- 172.21.15.90:0/1836861268 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559a1b7b19f0 0x559a1b7b8e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:22.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.198+0000 7fe1c4165740 1 --2- 172.21.15.90:0/1836861268 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559a1b7b0d10 0x559a1b7b88d0 secure :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fe1ac0138f0 tx=0x7fe1ac0046c0 comp rx=0 tx=0).stop 2024-04-02T14:44:22.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.198+0000 7fe1c4165740 1 -- 172.21.15.90:0/1836861268 >> 172.21.15.90:0/1836861268 conn(0x559a1b777a50 msgr2=0x559a1b7acc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:44:22.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.199+0000 7fe19be00640 0 client.28261 ms_handle_reset on v2:172.21.15.90:6838/1498915870 2024-04-02T14:44:22.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.199+0000 7fe1c4165740 1 -- 172.21.15.90:0/1836861268 shutdown_connections 2024-04-02T14:44:22.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.199+0000 7fe1c4165740 1 -- 172.21.15.90:0/1836861268 wait complete. 2024-04-02T14:44:22.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.199+0000 7fe1c4165740 20 client.28261 trim_cache size 0 max 16384 2024-04-02T14:44:22.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.217+0000 7fe1c4165740 1 Processor -- start 2024-04-02T14:44:22.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.217+0000 7fe1c4165740 1 -- start start 2024-04-02T14:44:22.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.217+0000 7fe1c4165740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559a1b7b19f0 0x559a1b7b9790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:22.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.217+0000 7fe1c4165740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559a1b7ba080 0x559a1b7b7a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:22.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.217+0000 7fe1c4165740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559a1b7ab9a0 0x559a1b7d2290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:22.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.217+0000 7fe1c4165740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fe19c00e7d0 con 0x559a1b7b19f0 2024-04-02T14:44:22.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.217+0000 7fe1c4165740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fe19c010310 con 0x559a1b7ba080 2024-04-02T14:44:22.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.217+0000 7fe1c4165740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fe19c00eea0 con 0x559a1b7ab9a0 2024-04-02T14:44:22.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.218+0000 7fe1bb200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559a1b7ab9a0 0x559a1b7d2290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:22.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.217+0000 7fe1ba800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559a1b7b19f0 0x559a1b7b9790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:22.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.218+0000 7fe1bb200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559a1b7ab9a0 0x559a1b7d2290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:39456/0 (socket says 172.21.15.90:39456) 2024-04-02T14:44:22.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.218+0000 7fe1bb200640 1 -- 172.21.15.90:0/3871983606 learned_addr learned my addr 172.21.15.90:0/3871983606 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:44:22.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.218+0000 7fe1ba800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559a1b7b19f0 0x559a1b7b9790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:45144/0 (socket says 172.21.15.90:45144) 2024-04-02T14:44:22.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.218+0000 7fe1b9e00640 1 --2- 172.21.15.90:0/3871983606 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559a1b7ba080 0x559a1b7b7a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:22.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.218+0000 7fe1bb200640 1 -- 172.21.15.90:0/3871983606 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559a1b7ba080 msgr2=0x559a1b7b7a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:44:22.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.218+0000 7fe1bb200640 1 --2- 172.21.15.90:0/3871983606 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559a1b7ba080 0x559a1b7b7a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:22.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.218+0000 7fe1bb200640 1 -- 172.21.15.90:0/3871983606 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559a1b7b19f0 msgr2=0x559a1b7b9790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:44:22.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.218+0000 7fe1bb200640 1 --2- 172.21.15.90:0/3871983606 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559a1b7b19f0 0x559a1b7b9790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:22.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.218+0000 7fe1bb200640 1 -- 172.21.15.90:0/3871983606 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559a1b7c6ad0 con 0x559a1b7ab9a0 2024-04-02T14:44:22.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.218+0000 7fe1b9e00640 1 --2- 172.21.15.90:0/3871983606 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559a1b7ba080 0x559a1b7b7a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:44:22.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.218+0000 7fe1ba800640 1 --2- 172.21.15.90:0/3871983606 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559a1b7b19f0 0x559a1b7b9790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:44:22.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.218+0000 7fe1bb200640 1 --2- 172.21.15.90:0/3871983606 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559a1b7ab9a0 0x559a1b7d2290 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fe1b401d090 tx=0x7fe1b4009c20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5d53b7a0d12bddde server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:44:22.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.219+0000 7fe1a6a00640 1 -- 172.21.15.90:0/3871983606 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1b4003a70 con 0x559a1b7ab9a0 2024-04-02T14:44:22.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.219+0000 7fe1a6a00640 1 -- 172.21.15.90:0/3871983606 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe1b4003c10 con 0x559a1b7ab9a0 2024-04-02T14:44:22.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.219+0000 7fe1a6a00640 1 -- 172.21.15.90:0/3871983606 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1b40116a0 con 0x559a1b7ab9a0 2024-04-02T14:44:22.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.220+0000 7fe1c4165740 1 -- 172.21.15.90:0/3871983606 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559a1b7ab9a0 msgr2=0x559a1b7d2290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:22.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.220+0000 7fe1c4165740 1 --2- 172.21.15.90:0/3871983606 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559a1b7ab9a0 0x559a1b7d2290 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fe1b401d090 tx=0x7fe1b4009c20 comp rx=0 tx=0).stop 2024-04-02T14:44:22.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.220+0000 7fe1c4165740 1 -- 172.21.15.90:0/3871983606 shutdown_connections 2024-04-02T14:44:22.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.220+0000 7fe1c4165740 1 --2- 172.21.15.90:0/3871983606 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559a1b7ab9a0 0x559a1b7d2290 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:22.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.220+0000 7fe1c4165740 1 --2- 172.21.15.90:0/3871983606 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559a1b7ba080 0x559a1b7b7a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:22.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.220+0000 7fe1c4165740 1 --2- 172.21.15.90:0/3871983606 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559a1b7b19f0 0x559a1b7b9790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:22.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.220+0000 7fe1c4165740 1 -- 172.21.15.90:0/3871983606 >> 172.21.15.90:0/3871983606 conn(0x559a1b777a50 msgr2=0x559a1b7ab290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:44:22.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.221+0000 7fe1c4165740 1 -- 172.21.15.90:0/3871983606 shutdown_connections 2024-04-02T14:44:22.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.221+0000 7fe1c4165740 1 -- 172.21.15.90:0/3871983606 wait complete. 2024-04-02T14:44:22.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.221+0000 7fe1c4165740 1 -- start start 2024-04-02T14:44:22.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.222+0000 7fe1c4165740 1 Processor -- start 2024-04-02T14:44:22.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.222+0000 7fe1c4165740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559a1b7ab9a0 0x559a1b7c2090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:22.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.222+0000 7fe1c4165740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559a1b7b19f0 0x559a1b7c25d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:22.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.222+0000 7fe1c4165740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559a1b7ba080 0x559a1b7b37c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:22.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.222+0000 7fe1c4165740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fe19c00fc40 con 0x559a1b7ab9a0 2024-04-02T14:44:22.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.222+0000 7fe1c4165740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fe19c00f570 con 0x559a1b7b19f0 2024-04-02T14:44:22.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.222+0000 7fe1c4165740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fe19c00c7d0 con 0x559a1b7ba080 2024-04-02T14:44:22.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.222+0000 7fe1c4165740 10 client.0 _ll_register_callbacks cb 0x7fe1ac011940 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:44:22.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.223+0000 7fe1bb200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559a1b7ba080 0x559a1b7b37c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:22.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.223+0000 7fe1bb200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559a1b7ba080 0x559a1b7b37c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:39464/0 (socket says 172.21.15.90:39464) 2024-04-02T14:44:22.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.223+0000 7fe1bb200640 1 -- 172.21.15.90:0/2024561987 learned_addr learned my addr 172.21.15.90:0/2024561987 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:44:22.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.223+0000 7fe1ba800640 1 --2- 172.21.15.90:0/2024561987 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559a1b7ab9a0 0x559a1b7c2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:22.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.223+0000 7fe1b9e00640 1 --2- 172.21.15.90:0/2024561987 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559a1b7b19f0 0x559a1b7c25d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:22.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.223+0000 7fe1bb200640 1 -- 172.21.15.90:0/2024561987 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559a1b7b19f0 msgr2=0x559a1b7c25d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:44:22.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.223+0000 7fe1bb200640 1 --2- 172.21.15.90:0/2024561987 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559a1b7b19f0 0x559a1b7c25d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:22.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.223+0000 7fe1bb200640 1 -- 172.21.15.90:0/2024561987 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559a1b7ab9a0 msgr2=0x559a1b7c2090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:44:22.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.223+0000 7fe1bb200640 1 --2- 172.21.15.90:0/2024561987 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559a1b7ab9a0 0x559a1b7c2090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:22.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.223+0000 7fe1bb200640 1 -- 172.21.15.90:0/2024561987 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559a1b7b3fa0 con 0x559a1b7ba080 2024-04-02T14:44:22.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.223+0000 7fe1bb200640 1 --2- 172.21.15.90:0/2024561987 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559a1b7ba080 0x559a1b7b37c0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fe1b4005a70 tx=0x7fe1b400f730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aa0994ef8dfe5756 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:44:22.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.223+0000 7fe19be00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:44:22.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.224+0000 7fe19be00640 1 -- 172.21.15.90:0/2024561987 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1b4003dc0 con 0x559a1b7ba080 2024-04-02T14:44:22.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.224+0000 7fe1c4165740 1 -- 172.21.15.90:0/2024561987 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe1b4003c10 con 0x559a1b7ba080 2024-04-02T14:44:22.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.224+0000 7fe19be00640 1 -- 172.21.15.90:0/2024561987 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8550+0+0 (secure 0 0 0) 0x7fe1b40052d0 con 0x559a1b7ba080 2024-04-02T14:44:22.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.224+0000 7fe19be00640 1 -- 172.21.15.90:0/2024561987 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fe1b404c050 con 0x559a1b7ba080 2024-04-02T14:44:22.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.224+0000 7fe1c4165740 10 client.28267 fetch_fsmap learned FSMap version 27 2024-04-02T14:44:22.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.224+0000 7fe1c4165740 1 -- 172.21.15.90:0/2024561987 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x559a1b8305e0 con 0x559a1b7ba080 2024-04-02T14:44:22.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.224+0000 7fe19be00640 1 -- 172.21.15.90:0/2024561987 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe1b4003c10 con 0x559a1b7ba080 2024-04-02T14:44:22.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.224+0000 7fe19be00640 1 -- 172.21.15.90:0/2024561987 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fe1b4011440 con 0x559a1b7ba080 2024-04-02T14:44:22.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.224+0000 7fe1c4165740 10 client.28267 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:44:22.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.224+0000 7fe1c4165740 10 client.28267 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2024-04-02T14:44:22.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.224+0000 7fe1c4165740 20 client.28267 populate_metadata read hostname 'smithi090' 2024-04-02T14:44:22.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.224+0000 7fe1c4165740 1 --2- 172.21.15.90:0/2024561987 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x559a1b830a50 0x559a1b832e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:22.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.224+0000 7fe1c4165740 4 client.28267 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-02T14:44:22.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.224+0000 7fe1c4165740 1 -- 172.21.15.90:0/2024561987 --> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] -- command(tid 0: {"prefix": "damage ls"}) -- 0x559a1b722c30 con 0x559a1b830a50 2024-04-02T14:44:22.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.225+0000 7fe1ba800640 1 --2- 172.21.15.90:0/2024561987 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x559a1b830a50 0x559a1b832e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:22.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.225+0000 7fe1ba800640 1 --2- 172.21.15.90:0/2024561987 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x559a1b830a50 0x559a1b832e40 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=93b67ca6148b5d0b server_cookie=f82115d7a7c1a918 in_seq=0 out_seq=0 2024-04-02T14:44:22.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.225+0000 7fe19be00640 10 client.28267 ms_handle_connect on v2:172.21.15.90:6838/1498915870 2024-04-02T14:44:22.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.226+0000 7fe19be00640 1 -- 172.21.15.90:0/2024561987 <== mds.1 v2:172.21.15.90:6838/1498915870 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x559a1b722c30 con 0x559a1b830a50 2024-04-02T14:44:22.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.226+0000 7fe19be00640 10 client.28267 handle_command_reply: tid=0 2024-04-02T14:44:22.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.226+0000 7fe1c4165740 1 client.28267 shutdown 2024-04-02T14:44:22.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.227+0000 7fe1c4165740 1 -- 172.21.15.90:0/2024561987 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559a1b7ba080 msgr2=0x559a1b7b37c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:22.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.227+0000 7fe1c4165740 1 --2- 172.21.15.90:0/2024561987 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559a1b7ba080 0x559a1b7b37c0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fe1b4005a70 tx=0x7fe1b400f730 comp rx=0 tx=0).stop 2024-04-02T14:44:22.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.227+0000 7fe1c4165740 1 -- 172.21.15.90:0/2024561987 shutdown_connections 2024-04-02T14:44:22.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.227+0000 7fe1c4165740 1 --2- 172.21.15.90:0/2024561987 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x559a1b830a50 0x559a1b832e40 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:22.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.227+0000 7fe1c4165740 1 --2- 172.21.15.90:0/2024561987 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559a1b7ba080 0x559a1b7b37c0 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:22.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.227+0000 7fe1c4165740 1 --2- 172.21.15.90:0/2024561987 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559a1b7b19f0 0x559a1b7c25d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:22.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.227+0000 7fe1c4165740 1 --2- 172.21.15.90:0/2024561987 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559a1b7ab9a0 0x559a1b7c2090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:22.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.227+0000 7fe1c4165740 1 -- 172.21.15.90:0/2024561987 >> 172.21.15.90:0/2024561987 conn(0x559a1b777a50 msgr2=0x559a1b7aa010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:44:22.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.227+0000 7fe19be00640 0 client.28267 ms_handle_reset on v2:172.21.15.90:6838/1498915870 2024-04-02T14:44:22.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.228+0000 7fe1c4165740 1 -- 172.21.15.90:0/2024561987 shutdown_connections 2024-04-02T14:44:22.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.228+0000 7fe1c4165740 1 -- 172.21.15.90:0/2024561987 wait complete. 2024-04-02T14:44:22.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.228+0000 7fe1c4165740 20 client.28267 trim_cache size 0 max 16384 2024-04-02T14:44:22.227 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-02T14:44:22.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.229+0000 7fe1c1400640 1 -- 172.21.15.90:0/3513824508 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fe1a00a3d20 msgr2=0x7fe1a00a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:22.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.229+0000 7fe1c1400640 1 --2- 172.21.15.90:0/3513824508 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fe1a00a3d20 0x7fe1a00a61e0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x559a1b7b1f40 tx=0x7fe1ac005f90 comp rx=0 tx=0).stop 2024-04-02T14:44:22.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.229+0000 7fe1c1400640 1 -- 172.21.15.90:0/3513824508 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe1bc110320 msgr2=0x7fe1bc1a76d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:22.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.229+0000 7fe1c1400640 1 --2- 172.21.15.90:0/3513824508 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe1bc110320 0x7fe1bc1a76d0 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7fe1a802c240 tx=0x7fe1a80042f0 comp rx=0 tx=0).stop 2024-04-02T14:44:22.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.229+0000 7fe1c1400640 1 -- 172.21.15.90:0/3513824508 shutdown_connections 2024-04-02T14:44:22.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.229+0000 7fe1c1400640 1 --2- 172.21.15.90:0/3513824508 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fe1a00a3d20 0x7fe1a00a61e0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:22.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.229+0000 7fe1c1400640 1 --2- 172.21.15.90:0/3513824508 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe1bc11b2e0 0x7fe1bc1aba50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:22.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.229+0000 7fe1c1400640 1 --2- 172.21.15.90:0/3513824508 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe1bc110320 0x7fe1bc1a76d0 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:22.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.229+0000 7fe1c1400640 1 --2- 172.21.15.90:0/3513824508 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe1bc10e350 0x7fe1bc1a7190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:22.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.230+0000 7fe1c1400640 1 -- 172.21.15.90:0/3513824508 >> 172.21.15.90:0/3513824508 conn(0x7fe1bc06cbe0 msgr2=0x7fe1bc0725d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:44:22.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.230+0000 7fe1c1400640 1 -- 172.21.15.90:0/3513824508 shutdown_connections 2024-04-02T14:44:22.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.230+0000 7fe1c1400640 1 -- 172.21.15.90:0/3513824508 wait complete. 2024-04-02T14:44:22.246 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-02T14:44:22.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.481+0000 7f032f800640 1 -- 172.21.15.90:0/3980211030 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0328108fe0 msgr2=0x7f0328110bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:22.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.481+0000 7f032f800640 1 --2- 172.21.15.90:0/3980211030 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0328108fe0 0x7f0328110bb0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f0320009ca0 tx=0x7f032002c0f0 comp rx=0 tx=0).stop 2024-04-02T14:44:22.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.481+0000 7f032f800640 1 -- 172.21.15.90:0/3980211030 shutdown_connections 2024-04-02T14:44:22.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.481+0000 7f032f800640 1 --2- 172.21.15.90:0/3980211030 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0328108fe0 0x7f0328110bb0 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:22.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.481+0000 7f032f800640 1 --2- 172.21.15.90:0/3980211030 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f03281048c0 0x7f0328108900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:22.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.481+0000 7f032f800640 1 --2- 172.21.15.90:0/3980211030 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0328103f10 0x7f03281042f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:22.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.481+0000 7f032f800640 1 -- 172.21.15.90:0/3980211030 >> 172.21.15.90:0/3980211030 conn(0x7f03280ff7b0 msgr2=0x7f0328101bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:44:22.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.481+0000 7f032f800640 1 -- 172.21.15.90:0/3980211030 shutdown_connections 2024-04-02T14:44:22.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.482+0000 7f032f800640 1 -- 172.21.15.90:0/3980211030 wait complete. 2024-04-02T14:44:22.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.483+0000 7f032f800640 1 Processor -- start 2024-04-02T14:44:22.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.483+0000 7f032f800640 1 -- start start 2024-04-02T14:44:22.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.484+0000 7f032f800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0328103f10 0x7f03280732d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:22.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.484+0000 7f032f800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0328108fe0 0x7f032806c3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:22.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.484+0000 7f032f800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0328073920 0x7f032806c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:22.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.484+0000 7f032f800640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f0328114090 con 0x7f0328103f10 2024-04-02T14:44:22.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.484+0000 7f032f800640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f0328113f10 con 0x7f0328073920 2024-04-02T14:44:22.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.484+0000 7f032f800640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f0328114210 con 0x7f0328108fe0 2024-04-02T14:44:22.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.484+0000 7f0327e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0328108fe0 0x7f032806c3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:22.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.484+0000 7f032ce00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0328103f10 0x7f03280732d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:22.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.484+0000 7f0327e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0328108fe0 0x7f032806c3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:39478/0 (socket says 172.21.15.90:39478) 2024-04-02T14:44:22.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.484+0000 7f0327e00640 1 -- 172.21.15.90:0/3040228085 learned_addr learned my addr 172.21.15.90:0/3040228085 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:44:22.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.484+0000 7f032d800640 1 --2- 172.21.15.90:0/3040228085 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0328073920 0x7f032806c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:22.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.485+0000 7f032d800640 1 -- 172.21.15.90:0/3040228085 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0328108fe0 msgr2=0x7f032806c3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:22.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.485+0000 7f032d800640 1 --2- 172.21.15.90:0/3040228085 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0328108fe0 0x7f032806c3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:22.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.485+0000 7f032d800640 1 -- 172.21.15.90:0/3040228085 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0328103f10 msgr2=0x7f03280732d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:22.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.485+0000 7f032d800640 1 --2- 172.21.15.90:0/3040228085 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0328103f10 0x7f03280732d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:22.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.485+0000 7f032d800640 1 -- 172.21.15.90:0/3040228085 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0328070df0 con 0x7f0328073920 2024-04-02T14:44:22.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.485+0000 7f032ce00640 1 --2- 172.21.15.90:0/3040228085 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0328103f10 0x7f03280732d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:44:22.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.485+0000 7f032d800640 1 --2- 172.21.15.90:0/3040228085 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0328073920 0x7f032806c9d0 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f0320009bf0 tx=0x7f0320032ea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:44:22.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.485+0000 7f0325600640 1 -- 172.21.15.90:0/3040228085 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0320041020 con 0x7f0328073920 2024-04-02T14:44:22.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.486+0000 7f0325600640 1 -- 172.21.15.90:0/3040228085 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0320034cd0 con 0x7f0328073920 2024-04-02T14:44:22.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.486+0000 7f032f800640 1 -- 172.21.15.90:0/3040228085 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0328071080 con 0x7f0328073920 2024-04-02T14:44:22.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.486+0000 7f0325600640 1 -- 172.21.15.90:0/3040228085 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0320039040 con 0x7f0328073920 2024-04-02T14:44:22.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.486+0000 7f032f800640 1 -- 172.21.15.90:0/3040228085 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0328071540 con 0x7f0328073920 2024-04-02T14:44:22.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.490+0000 7f0325600640 1 -- 172.21.15.90:0/3040228085 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f0320044020 con 0x7f0328073920 2024-04-02T14:44:22.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.492+0000 7f0325600640 1 --2- 172.21.15.90:0/3040228085 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f03100a3d20 0x7f03100a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:22.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.492+0000 7f0332472740 1 Processor -- start 2024-04-02T14:44:22.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.492+0000 7f0325600640 1 -- 172.21.15.90:0/3040228085 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8550+0+0 (secure 0 0 0) 0x7f03200f2350 con 0x7f0328073920 2024-04-02T14:44:22.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.492+0000 7f0332472740 1 -- start start 2024-04-02T14:44:22.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.492+0000 7f032ce00640 1 --2- 172.21.15.90:0/3040228085 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f03100a3d20 0x7f03100a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:22.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.492+0000 7f0332472740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e4f6f0d9f0 0x55e4f6f0c740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:22.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.492+0000 7f0332472740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e4f6f0cd10 0x55e4f6f0d190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:22.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.492+0000 7f0332472740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e4f6f1e890 0x55e4f6f20c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:22.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.493+0000 7f0332472740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55e4f6e4d400 con 0x55e4f6f0cd10 2024-04-02T14:44:22.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.493+0000 7f0332472740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55e4f6e31f10 con 0x55e4f6f0d9f0 2024-04-02T14:44:22.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.493+0000 7f0332472740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55e4f6e716f0 con 0x55e4f6f1e890 2024-04-02T14:44:22.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.493+0000 7f0327e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e4f6f0cd10 0x55e4f6f0d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:22.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.493+0000 7f032d800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e4f6f1e890 0x55e4f6f20c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:22.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.493+0000 7f0327e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e4f6f0cd10 0x55e4f6f0d190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:45196/0 (socket says 172.21.15.90:45196) 2024-04-02T14:44:22.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.493+0000 7f0327e00640 1 -- 172.21.15.90:0/3744612545 learned_addr learned my addr 172.21.15.90:0/3744612545 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:44:22.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.493+0000 7f032ce00640 1 --2- 172.21.15.90:0/3040228085 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f03100a3d20 0x7f03100a61e0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f031c009fc0 tx=0x7f031c005f90 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:44:22.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.493+0000 7f032ce00640 1 --2- 172.21.15.90:0/3744612545 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e4f6f0d9f0 0x55e4f6f0c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:22.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.493+0000 7f032d800640 1 -- 172.21.15.90:0/3744612545 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e4f6f0d9f0 msgr2=0x55e4f6f0c740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:44:22.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.493+0000 7f032d800640 1 --2- 172.21.15.90:0/3744612545 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e4f6f0d9f0 0x55e4f6f0c740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:22.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.493+0000 7f032d800640 1 -- 172.21.15.90:0/3744612545 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e4f6f0cd10 msgr2=0x55e4f6f0d190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:44:22.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.493+0000 7f032d800640 1 --2- 172.21.15.90:0/3744612545 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e4f6f0cd10 0x55e4f6f0d190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:22.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.493+0000 7f032d800640 1 -- 172.21.15.90:0/3744612545 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e4f6b853f0 con 0x55e4f6f1e890 2024-04-02T14:44:22.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.493+0000 7f0327e00640 1 --2- 172.21.15.90:0/3744612545 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e4f6f0cd10 0x55e4f6f0d190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:44:22.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.494+0000 7f032d800640 1 --2- 172.21.15.90:0/3744612545 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e4f6f1e890 0x55e4f6f20c80 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f03200efb30 tx=0x7f0320009170 comp rx=0 tx=0).ready entity=mon.1 client_cookie=553513e54a878ed7 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:44:22.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.495+0000 7f031a000640 1 -- 172.21.15.90:0/3744612545 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03200f1070 con 0x55e4f6f1e890 2024-04-02T14:44:22.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.495+0000 7f031a000640 1 -- 172.21.15.90:0/3744612545 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0320003e30 con 0x55e4f6f1e890 2024-04-02T14:44:22.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.495+0000 7f031a000640 1 -- 172.21.15.90:0/3744612545 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03200ba9f0 con 0x55e4f6f1e890 2024-04-02T14:44:22.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.496+0000 7f0332472740 1 -- 172.21.15.90:0/3744612545 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e4f6f1e890 msgr2=0x55e4f6f20c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:22.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.496+0000 7f0332472740 1 --2- 172.21.15.90:0/3744612545 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e4f6f1e890 0x55e4f6f20c80 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f03200efb30 tx=0x7f0320009170 comp rx=0 tx=0).stop 2024-04-02T14:44:22.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.496+0000 7f0332472740 1 -- 172.21.15.90:0/3744612545 shutdown_connections 2024-04-02T14:44:22.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.496+0000 7f0332472740 1 --2- 172.21.15.90:0/3744612545 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e4f6f1e890 0x55e4f6f20c80 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:22.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.496+0000 7f0332472740 1 --2- 172.21.15.90:0/3744612545 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e4f6f0cd10 0x55e4f6f0d190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:22.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.496+0000 7f0332472740 1 --2- 172.21.15.90:0/3744612545 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e4f6f0d9f0 0x55e4f6f0c740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:22.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.496+0000 7f0332472740 1 -- 172.21.15.90:0/3744612545 >> 172.21.15.90:0/3744612545 conn(0x55e4f6ed3a50 msgr2=0x55e4f6f07760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:44:22.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.496+0000 7f0332472740 1 -- 172.21.15.90:0/3744612545 shutdown_connections 2024-04-02T14:44:22.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.496+0000 7f0332472740 1 -- 172.21.15.90:0/3744612545 wait complete. 2024-04-02T14:44:22.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.497+0000 7f0332472740 1 -- start start 2024-04-02T14:44:22.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.497+0000 7f0332472740 1 Processor -- start 2024-04-02T14:44:22.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.497+0000 7f0332472740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e4f6f0cd10 0x55e4f6f1b900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:22.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.497+0000 7f0332472740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e4f6f0d9f0 0x55e4f6f1be40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:22.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.497+0000 7f0332472740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e4f6f1e890 0x55e4f6f1c380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:22.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.497+0000 7f0332472740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55e4f6e6d6a0 con 0x55e4f6f1e890 2024-04-02T14:44:22.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.498+0000 7f0332472740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55e4f6e19980 con 0x55e4f6f0cd10 2024-04-02T14:44:22.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.498+0000 7f0332472740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55e4f6e2ccb0 con 0x55e4f6f0d9f0 2024-04-02T14:44:22.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.498+0000 7f0332472740 10 client.0 _ll_register_callbacks cb 0x55e4f6e2c6a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:44:22.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.498+0000 7f032ce00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e4f6f0cd10 0x55e4f6f1b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:22.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.498+0000 7f032ce00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e4f6f0cd10 0x55e4f6f1b900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:39074/0 (socket says 172.21.15.90:39074) 2024-04-02T14:44:22.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.498+0000 7f032ce00640 1 -- 172.21.15.90:0/1155060740 learned_addr learned my addr 172.21.15.90:0/1155060740 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:44:22.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.498+0000 7f0327e00640 1 --2- 172.21.15.90:0/1155060740 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e4f6f0d9f0 0x55e4f6f1be40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:22.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.498+0000 7f032d800640 1 --2- 172.21.15.90:0/1155060740 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e4f6f1e890 0x55e4f6f1c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:22.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.498+0000 7f032ce00640 1 -- 172.21.15.90:0/1155060740 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e4f6f0d9f0 msgr2=0x55e4f6f1be40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:44:22.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.498+0000 7f032ce00640 1 --2- 172.21.15.90:0/1155060740 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e4f6f0d9f0 0x55e4f6f1be40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:22.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.498+0000 7f032ce00640 1 -- 172.21.15.90:0/1155060740 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e4f6f1e890 msgr2=0x55e4f6f1c380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:44:22.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.498+0000 7f032ce00640 1 --2- 172.21.15.90:0/1155060740 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e4f6f1e890 0x55e4f6f1c380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:22.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.498+0000 7f032ce00640 1 -- 172.21.15.90:0/1155060740 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e4f6c4f7d0 con 0x55e4f6f0cd10 2024-04-02T14:44:22.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.498+0000 7f032ce00640 1 --2- 172.21.15.90:0/1155060740 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e4f6f0cd10 0x55e4f6f1b900 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f031c003760 tx=0x7f031c004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=61f64126d487355b server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:44:22.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.498+0000 7f030ea00640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:44:22.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.499+0000 7f030ea00640 1 -- 172.21.15.90:0/1155060740 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f031c00fad0 con 0x55e4f6f0cd10 2024-04-02T14:44:22.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.499+0000 7f0332472740 1 -- 172.21.15.90:0/1155060740 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0320003e30 con 0x55e4f6f0cd10 2024-04-02T14:44:22.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.499+0000 7f030ea00640 1 -- 172.21.15.90:0/1155060740 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8550+0+0 (secure 0 0 0) 0x7f031c00fc70 con 0x55e4f6f0cd10 2024-04-02T14:44:22.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.499+0000 7f030ea00640 1 -- 172.21.15.90:0/1155060740 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f031c00fe70 con 0x55e4f6f0cd10 2024-04-02T14:44:22.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.499+0000 7f0332472740 10 client.28448 fetch_fsmap learned FSMap version 27 2024-04-02T14:44:22.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.499+0000 7f0332472740 1 -- 172.21.15.90:0/1155060740 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55e4f6f2d520 con 0x55e4f6f0cd10 2024-04-02T14:44:22.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.499+0000 7f030ea00640 1 -- 172.21.15.90:0/1155060740 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0320003e30 con 0x55e4f6f0cd10 2024-04-02T14:44:22.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.499+0000 7f030ea00640 1 -- 172.21.15.90:0/1155060740 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f031c025070 con 0x55e4f6f0cd10 2024-04-02T14:44:22.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.500+0000 7f0332472740 10 client.28448 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:44:22.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.500+0000 7f0332472740 10 client.28448 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:44:22.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.500+0000 7f0332472740 20 client.28448 populate_metadata read hostname 'smithi090' 2024-04-02T14:44:22.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.500+0000 7f0332472740 1 --2- 172.21.15.90:0/1155060740 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55e4f6f2e9d0 0x55e4f6f30de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:22.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.500+0000 7f0332472740 4 client.28448 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:44:22.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.500+0000 7f0332472740 1 -- 172.21.15.90:0/1155060740 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e4f6e31040 con 0x55e4f6f2e9d0 2024-04-02T14:44:22.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.500+0000 7f0327e00640 1 --2- 172.21.15.90:0/1155060740 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55e4f6f2e9d0 0x55e4f6f30de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:22.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.500+0000 7f030ea00640 10 client.28448 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:44:22.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.500+0000 7f0327e00640 1 --2- 172.21.15.90:0/1155060740 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55e4f6f2e9d0 0x55e4f6f30de0 crc :-1 s=READY pgs=254 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=41558d5faa1405b8 server_cookie=4ecc4891d03107a1 in_seq=0 out_seq=0 2024-04-02T14:44:22.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.501+0000 7f030ea00640 1 -- 172.21.15.90:0/1155060740 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e4f6e31040 con 0x55e4f6f2e9d0 2024-04-02T14:44:22.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.501+0000 7f030ea00640 10 client.28448 handle_command_reply: tid=0 2024-04-02T14:44:22.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.502+0000 7f0332472740 1 client.28448 shutdown 2024-04-02T14:44:22.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.503+0000 7f0332472740 1 -- 172.21.15.90:0/1155060740 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e4f6f0cd10 msgr2=0x55e4f6f1b900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:22.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.503+0000 7f0332472740 1 --2- 172.21.15.90:0/1155060740 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e4f6f0cd10 0x55e4f6f1b900 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f031c003760 tx=0x7f031c004730 comp rx=0 tx=0).stop 2024-04-02T14:44:22.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.503+0000 7f0332472740 1 -- 172.21.15.90:0/1155060740 shutdown_connections 2024-04-02T14:44:22.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.503+0000 7f0332472740 1 --2- 172.21.15.90:0/1155060740 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55e4f6f2e9d0 0x55e4f6f30de0 crc :-1 s=READY pgs=254 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:22.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.503+0000 7f0332472740 1 --2- 172.21.15.90:0/1155060740 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e4f6f1e890 0x55e4f6f1c380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:22.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.503+0000 7f0332472740 1 --2- 172.21.15.90:0/1155060740 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e4f6f0d9f0 0x55e4f6f1be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:22.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.503+0000 7f0332472740 1 --2- 172.21.15.90:0/1155060740 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e4f6f0cd10 0x55e4f6f1b900 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:22.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.503+0000 7f0332472740 1 -- 172.21.15.90:0/1155060740 >> 172.21.15.90:0/1155060740 conn(0x55e4f6ed3a50 msgr2=0x55e4f6f094b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:44:22.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.503+0000 7f030ea00640 0 client.28448 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:44:22.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.504+0000 7f0332472740 1 -- 172.21.15.90:0/1155060740 shutdown_connections 2024-04-02T14:44:22.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.504+0000 7f0332472740 1 -- 172.21.15.90:0/1155060740 wait complete. 2024-04-02T14:44:22.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.504+0000 7f0332472740 20 client.28448 trim_cache size 0 max 16384 2024-04-02T14:44:22.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.530+0000 7f0332472740 1 Processor -- start 2024-04-02T14:44:22.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.530+0000 7f0332472740 1 -- start start 2024-04-02T14:44:22.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.531+0000 7f0332472740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e4f6f0cd10 0x55e4f6f1e030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:22.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.531+0000 7f0332472740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e4f6f0d9f0 0x55e4f6f29f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:22.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.531+0000 7f0332472740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e4f6f1e890 0x55e4f6f1bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:22.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.531+0000 7f0332472740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f03281b3e20 con 0x55e4f6f1e890 2024-04-02T14:44:22.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.531+0000 7f0332472740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f03281b5960 con 0x55e4f6f0d9f0 2024-04-02T14:44:22.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.531+0000 7f0332472740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f03281b44f0 con 0x55e4f6f0cd10 2024-04-02T14:44:22.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.531+0000 7f032ce00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e4f6f0cd10 0x55e4f6f1e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:22.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.531+0000 7f032ce00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e4f6f0cd10 0x55e4f6f1e030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:39516/0 (socket says 172.21.15.90:39516) 2024-04-02T14:44:22.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.531+0000 7f032ce00640 1 -- 172.21.15.90:0/3065578856 learned_addr learned my addr 172.21.15.90:0/3065578856 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:44:22.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.531+0000 7f0327e00640 1 --2- 172.21.15.90:0/3065578856 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e4f6f0d9f0 0x55e4f6f29f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:22.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.532+0000 7f032d800640 1 --2- 172.21.15.90:0/3065578856 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e4f6f1e890 0x55e4f6f1bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:22.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.532+0000 7f0327e00640 1 -- 172.21.15.90:0/3065578856 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e4f6f0cd10 msgr2=0x55e4f6f1e030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:44:22.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.532+0000 7f0327e00640 1 --2- 172.21.15.90:0/3065578856 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e4f6f0cd10 0x55e4f6f1e030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:22.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.532+0000 7f0327e00640 1 -- 172.21.15.90:0/3065578856 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e4f6f1e890 msgr2=0x55e4f6f1bfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:44:22.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.532+0000 7f0327e00640 1 --2- 172.21.15.90:0/3065578856 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e4f6f1e890 0x55e4f6f1bfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:22.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.532+0000 7f0327e00640 1 -- 172.21.15.90:0/3065578856 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e4f6f19e50 con 0x55e4f6f0d9f0 2024-04-02T14:44:22.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.534+0000 7f0327e00640 1 --2- 172.21.15.90:0/3065578856 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e4f6f0d9f0 0x55e4f6f29f90 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x55e4f6f31650 tx=0x7f0314005f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ba6c975df4dba796 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:44:22.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.535+0000 7f0319600640 1 -- 172.21.15.90:0/3065578856 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0314004310 con 0x55e4f6f0d9f0 2024-04-02T14:44:22.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.535+0000 7f0319600640 1 -- 172.21.15.90:0/3065578856 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f03140044b0 con 0x55e4f6f0d9f0 2024-04-02T14:44:22.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.535+0000 7f0319600640 1 -- 172.21.15.90:0/3065578856 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f031401e390 con 0x55e4f6f0d9f0 2024-04-02T14:44:22.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.539+0000 7f0332472740 1 -- 172.21.15.90:0/3065578856 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e4f6f0d9f0 msgr2=0x55e4f6f29f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:22.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.539+0000 7f0332472740 1 --2- 172.21.15.90:0/3065578856 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e4f6f0d9f0 0x55e4f6f29f90 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x55e4f6f31650 tx=0x7f0314005f90 comp rx=0 tx=0).stop 2024-04-02T14:44:22.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.539+0000 7f0332472740 1 -- 172.21.15.90:0/3065578856 shutdown_connections 2024-04-02T14:44:22.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.539+0000 7f0332472740 1 --2- 172.21.15.90:0/3065578856 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e4f6f1e890 0x55e4f6f1bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:22.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.539+0000 7f0332472740 1 --2- 172.21.15.90:0/3065578856 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e4f6f0d9f0 0x55e4f6f29f90 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:22.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.539+0000 7f0332472740 1 --2- 172.21.15.90:0/3065578856 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e4f6f0cd10 0x55e4f6f1e030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:22.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.540+0000 7f0332472740 1 -- 172.21.15.90:0/3065578856 >> 172.21.15.90:0/3065578856 conn(0x55e4f6ed3a50 msgr2=0x55e4f6f1b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:44:22.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.543+0000 7f0332472740 1 -- 172.21.15.90:0/3065578856 shutdown_connections 2024-04-02T14:44:22.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.543+0000 7f0332472740 1 -- 172.21.15.90:0/3065578856 wait complete. 2024-04-02T14:44:22.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.543+0000 7f0332472740 1 -- start start 2024-04-02T14:44:22.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.544+0000 7f0332472740 1 Processor -- start 2024-04-02T14:44:22.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.546+0000 7f0332472740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e4f6f0cd10 0x55e4f6f1f700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:22.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.546+0000 7f0332472740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e4f6f0d9f0 0x55e4f6f1fc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:22.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.546+0000 7f0332472740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e4f6f1e890 0x55e4f6f201c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:22.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.547+0000 7f0332472740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f03281b5290 con 0x55e4f6f0cd10 2024-04-02T14:44:22.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.547+0000 7f0332472740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f03281b4bc0 con 0x55e4f6f1e890 2024-04-02T14:44:22.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.547+0000 7f0332472740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f032807a0b0 con 0x55e4f6f0d9f0 2024-04-02T14:44:22.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.547+0000 7f0332472740 10 client.0 _ll_register_callbacks cb 0x7f031c0035f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:44:22.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.547+0000 7f032d800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e4f6f1e890 0x55e4f6f201c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:22.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.547+0000 7f032d800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e4f6f1e890 0x55e4f6f201c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:39096/0 (socket says 172.21.15.90:39096) 2024-04-02T14:44:22.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.547+0000 7f032d800640 1 -- 172.21.15.90:0/4011674561 learned_addr learned my addr 172.21.15.90:0/4011674561 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:44:22.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.547+0000 7f0327e00640 1 --2- 172.21.15.90:0/4011674561 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e4f6f0d9f0 0x55e4f6f1fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:22.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.548+0000 7f032ce00640 1 --2- 172.21.15.90:0/4011674561 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e4f6f0cd10 0x55e4f6f1f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:22.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.548+0000 7f032d800640 1 -- 172.21.15.90:0/4011674561 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e4f6f0d9f0 msgr2=0x55e4f6f1fc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:44:22.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.548+0000 7f032d800640 1 --2- 172.21.15.90:0/4011674561 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e4f6f0d9f0 0x55e4f6f1fc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:22.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.548+0000 7f032d800640 1 -- 172.21.15.90:0/4011674561 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e4f6f0cd10 msgr2=0x55e4f6f1f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:44:22.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.548+0000 7f0327e00640 1 --2- 172.21.15.90:0/4011674561 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e4f6f0d9f0 0x55e4f6f1fc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:44:22.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.548+0000 7f032d800640 1 --2- 172.21.15.90:0/4011674561 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e4f6f0cd10 0x55e4f6f1f700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:22.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.548+0000 7f032d800640 1 -- 172.21.15.90:0/4011674561 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e4f6f24bc0 con 0x55e4f6f1e890 2024-04-02T14:44:22.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.549+0000 7f032ce00640 1 --2- 172.21.15.90:0/4011674561 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e4f6f0cd10 0x55e4f6f1f700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:44:22.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.549+0000 7f032d800640 1 --2- 172.21.15.90:0/4011674561 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e4f6f1e890 0x55e4f6f201c0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f03200bad50 tx=0x7f03200b9fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e07fdc5052b6578a server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:44:22.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.549+0000 7f030ea00640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:44:22.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.550+0000 7f030ea00640 1 -- 172.21.15.90:0/4011674561 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03200f1070 con 0x55e4f6f1e890 2024-04-02T14:44:22.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.551+0000 7f030ea00640 1 -- 172.21.15.90:0/4011674561 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8550+0+0 (secure 0 0 0) 0x7f03200c6b60 con 0x55e4f6f1e890 2024-04-02T14:44:22.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.551+0000 7f0332472740 1 -- 172.21.15.90:0/4011674561 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f03140044b0 con 0x55e4f6f1e890 2024-04-02T14:44:22.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.551+0000 7f030ea00640 1 -- 172.21.15.90:0/4011674561 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f03200dd320 con 0x55e4f6f1e890 2024-04-02T14:44:22.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.551+0000 7f0332472740 10 client.28454 fetch_fsmap learned FSMap version 27 2024-04-02T14:44:22.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.551+0000 7f0332472740 1 -- 172.21.15.90:0/4011674561 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55e4f6f95510 con 0x55e4f6f1e890 2024-04-02T14:44:22.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.555+0000 7f030ea00640 1 -- 172.21.15.90:0/4011674561 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f03140044b0 con 0x55e4f6f1e890 2024-04-02T14:44:22.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.557+0000 7f030ea00640 1 -- 172.21.15.90:0/4011674561 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f032003c030 con 0x55e4f6f1e890 2024-04-02T14:44:22.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.557+0000 7f0332472740 10 client.28454 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:44:22.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.557+0000 7f0332472740 10 client.28454 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:44:22.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.557+0000 7f0332472740 20 client.28454 populate_metadata read hostname 'smithi090' 2024-04-02T14:44:22.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.557+0000 7f0332472740 1 --2- 172.21.15.90:0/4011674561 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55e4f6f95990 0x55e4f6f97db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:22.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.558+0000 7f0332472740 4 client.28454 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-02T14:44:22.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.558+0000 7f0332472740 1 -- 172.21.15.90:0/4011674561 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55e4f6e7ec30 con 0x55e4f6f95990 2024-04-02T14:44:22.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.559+0000 7f032ce00640 1 --2- 172.21.15.90:0/4011674561 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55e4f6f95990 0x55e4f6f97db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:22.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.560+0000 7f032ce00640 1 --2- 172.21.15.90:0/4011674561 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55e4f6f95990 0x55e4f6f97db0 crc :-1 s=READY pgs=255 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7bd90af6dd5b57ae server_cookie=e9433e545658c1af in_seq=0 out_seq=0 2024-04-02T14:44:22.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.560+0000 7f030ea00640 10 client.28454 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:44:22.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.560+0000 7f030ea00640 1 -- 172.21.15.90:0/4011674561 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55e4f6e7ec30 con 0x55e4f6f95990 2024-04-02T14:44:22.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.560+0000 7f030ea00640 10 client.28454 handle_command_reply: tid=0 2024-04-02T14:44:22.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.560+0000 7f0332472740 1 client.28454 shutdown 2024-04-02T14:44:22.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.561+0000 7f0332472740 1 -- 172.21.15.90:0/4011674561 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e4f6f1e890 msgr2=0x55e4f6f201c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:22.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.561+0000 7f0332472740 1 --2- 172.21.15.90:0/4011674561 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e4f6f1e890 0x55e4f6f201c0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f03200bad50 tx=0x7f03200b9fd0 comp rx=0 tx=0).stop 2024-04-02T14:44:22.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.561+0000 7f0332472740 1 -- 172.21.15.90:0/4011674561 shutdown_connections 2024-04-02T14:44:22.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.561+0000 7f0332472740 1 --2- 172.21.15.90:0/4011674561 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55e4f6f95990 0x55e4f6f97db0 crc :-1 s=READY pgs=255 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:22.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.561+0000 7f0332472740 1 --2- 172.21.15.90:0/4011674561 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e4f6f1e890 0x55e4f6f201c0 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:22.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.561+0000 7f0332472740 1 --2- 172.21.15.90:0/4011674561 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e4f6f0d9f0 0x55e4f6f1fc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:22.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.561+0000 7f030ea00640 0 client.28454 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:44:22.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.561+0000 7f0332472740 1 --2- 172.21.15.90:0/4011674561 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e4f6f0cd10 0x55e4f6f1f700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:22.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.561+0000 7f0332472740 1 -- 172.21.15.90:0/4011674561 >> 172.21.15.90:0/4011674561 conn(0x55e4f6ed3a50 msgr2=0x55e4f6f068a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:44:22.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.561+0000 7f0332472740 1 -- 172.21.15.90:0/4011674561 shutdown_connections 2024-04-02T14:44:22.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.562+0000 7f0332472740 1 -- 172.21.15.90:0/4011674561 wait complete. 2024-04-02T14:44:22.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.563+0000 7f0332472740 20 client.28454 trim_cache size 0 max 16384 2024-04-02T14:44:22.561 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-02T14:44:22.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.564+0000 7f032f800640 1 -- 172.21.15.90:0/3040228085 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f03100a3d20 msgr2=0x7f03100a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:22.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.564+0000 7f032f800640 1 --2- 172.21.15.90:0/3040228085 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f03100a3d20 0x7f03100a61e0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f031c009fc0 tx=0x7f031c005f90 comp rx=0 tx=0).stop 2024-04-02T14:44:22.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.564+0000 7f032f800640 1 -- 172.21.15.90:0/3040228085 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0328073920 msgr2=0x7f032806c9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:22.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.564+0000 7f032f800640 1 --2- 172.21.15.90:0/3040228085 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0328073920 0x7f032806c9d0 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f0320009bf0 tx=0x7f0320032ea0 comp rx=0 tx=0).stop 2024-04-02T14:44:22.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.564+0000 7f032f800640 1 -- 172.21.15.90:0/3040228085 shutdown_connections 2024-04-02T14:44:22.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.564+0000 7f032f800640 1 --2- 172.21.15.90:0/3040228085 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f03100a3d20 0x7f03100a61e0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:22.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.564+0000 7f032f800640 1 --2- 172.21.15.90:0/3040228085 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0328073920 0x7f032806c9d0 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:22.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.564+0000 7f032f800640 1 --2- 172.21.15.90:0/3040228085 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0328108fe0 0x7f032806c3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:22.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.564+0000 7f032f800640 1 --2- 172.21.15.90:0/3040228085 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0328103f10 0x7f03280732d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:22.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.564+0000 7f032f800640 1 -- 172.21.15.90:0/3040228085 >> 172.21.15.90:0/3040228085 conn(0x7f03280ff7b0 msgr2=0x7f0328101a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:44:22.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.567+0000 7f032f800640 1 -- 172.21.15.90:0/3040228085 shutdown_connections 2024-04-02T14:44:22.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:22.567+0000 7f032f800640 1 -- 172.21.15.90:0/3040228085 wait complete. 2024-04-02T14:44:22.589 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:44:22 smithi090 ceph-mon[31674]: pgmap v293: 97 pgs: 97 active+clean; 4.4 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 6.4 MiB/s wr, 580 op/s 2024-04-02T14:44:22.605 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:44:22 smithi092 ceph-mon[36497]: pgmap v293: 97 pgs: 97 active+clean; 4.4 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 6.4 MiB/s wr, 580 op/s 2024-04-02T14:44:22.682 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:44:22 smithi177 ceph-mon[36431]: pgmap v293: 97 pgs: 97 active+clean; 4.4 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 6.4 MiB/s wr, 580 op/s 2024-04-02T14:44:23.592 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-02T14:44:23.592 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T14:44:23.592 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-02T14:44:23.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.856+0000 7fc33be00640 1 -- 172.21.15.90:0/604083058 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc33c101960 msgr2=0x7fc33c101de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:23.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.856+0000 7fc33be00640 1 --2- 172.21.15.90:0/604083058 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc33c101960 0x7fc33c101de0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7fc328009900 tx=0x7fc32802bc60 comp rx=0 tx=0).stop 2024-04-02T14:44:23.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.856+0000 7fc33be00640 1 -- 172.21.15.90:0/604083058 shutdown_connections 2024-04-02T14:44:23.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.856+0000 7fc33be00640 1 --2- 172.21.15.90:0/604083058 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc33c102320 0x7fc33c10e830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:23.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.856+0000 7fc33be00640 1 --2- 172.21.15.90:0/604083058 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc33c101960 0x7fc33c101de0 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:23.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.856+0000 7fc33be00640 1 --2- 172.21.15.90:0/604083058 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc33c107970 0x7fc33c107d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:23.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.856+0000 7fc33be00640 1 -- 172.21.15.90:0/604083058 >> 172.21.15.90:0/604083058 conn(0x7fc33c0fd470 msgr2=0x7fc33c0ff890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:44:23.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.856+0000 7fc33be00640 1 -- 172.21.15.90:0/604083058 shutdown_connections 2024-04-02T14:44:23.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.856+0000 7fc33be00640 1 -- 172.21.15.90:0/604083058 wait complete. 2024-04-02T14:44:23.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.857+0000 7fc33be00640 1 Processor -- start 2024-04-02T14:44:23.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.857+0000 7fc33be00640 1 -- start start 2024-04-02T14:44:23.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.858+0000 7fc33be00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc33c101960 0x7fc33c19a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:23.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.858+0000 7fc33be00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc33c102320 0x7fc33c19a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:23.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.858+0000 7fc33be00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc33c107970 0x7fc33c19ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:23.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.858+0000 7fc33be00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fc33c111da0 con 0x7fc33c102320 2024-04-02T14:44:23.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.858+0000 7fc33be00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fc33c111c20 con 0x7fc33c101960 2024-04-02T14:44:23.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.858+0000 7fc33be00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fc33c111f20 con 0x7fc33c107970 2024-04-02T14:44:23.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.858+0000 7fc33aa00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc33c101960 0x7fc33c19a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:23.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.858+0000 7fc33aa00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc33c101960 0x7fc33c19a3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:36270/0 (socket says 172.21.15.90:36270) 2024-04-02T14:44:23.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.858+0000 7fc33aa00640 1 -- 172.21.15.90:0/4116824136 learned_addr learned my addr 172.21.15.90:0/4116824136 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:44:23.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.858+0000 7fc33a000640 1 --2- 172.21.15.90:0/4116824136 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc33c102320 0x7fc33c19a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:23.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.858+0000 7fc33b400640 1 --2- 172.21.15.90:0/4116824136 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc33c107970 0x7fc33c19ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:23.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.858+0000 7fc33aa00640 1 -- 172.21.15.90:0/4116824136 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc33c107970 msgr2=0x7fc33c19ec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:23.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.858+0000 7fc33aa00640 1 --2- 172.21.15.90:0/4116824136 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc33c107970 0x7fc33c19ec70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:23.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.858+0000 7fc33aa00640 1 -- 172.21.15.90:0/4116824136 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc33c102320 msgr2=0x7fc33c19a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:23.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.859+0000 7fc33aa00640 1 --2- 172.21.15.90:0/4116824136 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc33c102320 0x7fc33c19a8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:23.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.859+0000 7fc33aa00640 1 -- 172.21.15.90:0/4116824136 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc33c19f3a0 con 0x7fc33c101960 2024-04-02T14:44:23.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.859+0000 7fc33a000640 1 --2- 172.21.15.90:0/4116824136 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc33c102320 0x7fc33c19a8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:44:23.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.860+0000 7fc33aa00640 1 --2- 172.21.15.90:0/4116824136 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc33c101960 0x7fc33c19a3b0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7fc32c00e9f0 tx=0x7fc32c00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:44:23.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.861+0000 7fc333400640 1 -- 172.21.15.90:0/4116824136 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc32c00cd80 con 0x7fc33c101960 2024-04-02T14:44:23.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.861+0000 7fc333400640 1 -- 172.21.15.90:0/4116824136 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc32c010910 con 0x7fc33c101960 2024-04-02T14:44:23.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.861+0000 7fc333400640 1 -- 172.21.15.90:0/4116824136 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc32c018990 con 0x7fc33c101960 2024-04-02T14:44:23.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.861+0000 7fc33be00640 1 -- 172.21.15.90:0/4116824136 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc33c19f690 con 0x7fc33c101960 2024-04-02T14:44:23.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.861+0000 7fc33be00640 1 -- 172.21.15.90:0/4116824136 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc33c1a6f20 con 0x7fc33c101960 2024-04-02T14:44:23.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.864+0000 7fc344244740 1 Processor -- start 2024-04-02T14:44:23.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.864+0000 7fc344244740 1 -- start start 2024-04-02T14:44:23.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.865+0000 7fc344244740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ba5b49fa00 0x55ba5b49e750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:23.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.865+0000 7fc344244740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ba5b49ed20 0x55ba5b49f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:23.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.865+0000 7fc344244740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ba5b4a8090 0x55ba5b4aa480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:23.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.865+0000 7fc344244740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55ba5b3fa310 con 0x55ba5b49fa00 2024-04-02T14:44:23.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.865+0000 7fc344244740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55ba5b3a2690 con 0x55ba5b49ed20 2024-04-02T14:44:23.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.865+0000 7fc344244740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55ba5b3f62c0 con 0x55ba5b4a8090 2024-04-02T14:44:23.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.865+0000 7fc33b400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ba5b49fa00 0x55ba5b49e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:23.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.865+0000 7fc33b400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ba5b49fa00 0x55ba5b49e750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41500/0 (socket says 172.21.15.90:41500) 2024-04-02T14:44:23.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.865+0000 7fc33b400640 1 -- 172.21.15.90:0/2586833049 learned_addr learned my addr 172.21.15.90:0/2586833049 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:44:23.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.865+0000 7fc333400640 1 -- 172.21.15.90:0/4116824136 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fc32c018b30 con 0x7fc33c101960 2024-04-02T14:44:23.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.865+0000 7fc33a000640 1 --2- 172.21.15.90:0/2586833049 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ba5b4a8090 0x55ba5b4aa480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:23.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.866+0000 7fc33b400640 1 -- 172.21.15.90:0/2586833049 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ba5b4a8090 msgr2=0x55ba5b4aa480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:44:23.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.866+0000 7fc33b400640 1 --2- 172.21.15.90:0/2586833049 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ba5b4a8090 0x55ba5b4aa480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:23.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.866+0000 7fc33b400640 1 -- 172.21.15.90:0/2586833049 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ba5b49ed20 msgr2=0x55ba5b49f1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:44:23.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.866+0000 7fc33aa00640 1 --2- 172.21.15.90:0/2586833049 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ba5b49ed20 0x55ba5b49f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:23.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.866+0000 7fc33b400640 1 --2- 172.21.15.90:0/2586833049 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ba5b49ed20 0x55ba5b49f1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:23.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.866+0000 7fc33b400640 1 -- 172.21.15.90:0/2586833049 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ba5b1d67e0 con 0x55ba5b49fa00 2024-04-02T14:44:23.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.866+0000 7fc33aa00640 1 --2- 172.21.15.90:0/2586833049 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ba5b49ed20 0x55ba5b49f1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:44:23.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.866+0000 7fc33b400640 1 --2- 172.21.15.90:0/2586833049 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ba5b49fa00 0x55ba5b49e750 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fc334007e50 tx=0x7fc33400e720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e9d7a3e6f15bfcf server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:44:23.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.868+0000 7fc333400640 1 --2- 172.21.15.90:0/4116824136 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fc3200a3ef0 0x7fc3200a63b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:23.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.868+0000 7fc327400640 1 -- 172.21.15.90:0/2586833049 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc334014070 con 0x55ba5b49fa00 2024-04-02T14:44:23.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.868+0000 7fc327400640 1 -- 172.21.15.90:0/2586833049 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc33400ee90 con 0x55ba5b49fa00 2024-04-02T14:44:23.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.869+0000 7fc333400640 1 -- 172.21.15.90:0/4116824136 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8550+0+0 (secure 0 0 0) 0x7fc32c0d0b60 con 0x7fc33c101960 2024-04-02T14:44:23.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.869+0000 7fc327400640 1 -- 172.21.15.90:0/2586833049 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc334014070 con 0x55ba5b49fa00 2024-04-02T14:44:23.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.869+0000 7fc33aa00640 1 --2- 172.21.15.90:0/4116824136 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fc3200a3ef0 0x7fc3200a63b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:23.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.869+0000 7fc344244740 1 -- 172.21.15.90:0/2586833049 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ba5b49fa00 msgr2=0x55ba5b49e750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:23.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.869+0000 7fc344244740 1 --2- 172.21.15.90:0/2586833049 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ba5b49fa00 0x55ba5b49e750 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fc334007e50 tx=0x7fc33400e720 comp rx=0 tx=0).stop 2024-04-02T14:44:23.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.870+0000 7fc344244740 1 -- 172.21.15.90:0/2586833049 shutdown_connections 2024-04-02T14:44:23.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.870+0000 7fc344244740 1 --2- 172.21.15.90:0/2586833049 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ba5b4a8090 0x55ba5b4aa480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:23.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.870+0000 7fc344244740 1 --2- 172.21.15.90:0/2586833049 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ba5b49ed20 0x55ba5b49f1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:23.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.870+0000 7fc344244740 1 --2- 172.21.15.90:0/2586833049 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ba5b49fa00 0x55ba5b49e750 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:23.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.870+0000 7fc344244740 1 -- 172.21.15.90:0/2586833049 >> 172.21.15.90:0/2586833049 conn(0x55ba5b42a270 msgr2=0x55ba5b499770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:44:23.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.870+0000 7fc344244740 1 -- 172.21.15.90:0/2586833049 shutdown_connections 2024-04-02T14:44:23.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.870+0000 7fc344244740 1 -- 172.21.15.90:0/2586833049 wait complete. 2024-04-02T14:44:23.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.870+0000 7fc33aa00640 1 --2- 172.21.15.90:0/4116824136 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fc3200a3ef0 0x7fc3200a63b0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x55ba5b49ff50 tx=0x7fc32c000bc0 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:44:23.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.871+0000 7fc344244740 1 -- start start 2024-04-02T14:44:23.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.871+0000 7fc344244740 1 Processor -- start 2024-04-02T14:44:23.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.871+0000 7fc344244740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ba5b49ed20 0x55ba5b4a4f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:23.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.871+0000 7fc344244740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ba5b49fa00 0x55ba5b4a54b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:23.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.871+0000 7fc344244740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ba5b4a8090 0x55ba5b4a59f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:23.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.871+0000 7fc344244740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55ba5b4aafa0 con 0x55ba5b49fa00 2024-04-02T14:44:23.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.871+0000 7fc344244740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55ba5b3b5a60 con 0x55ba5b49ed20 2024-04-02T14:44:23.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.871+0000 7fc344244740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55ba5b4ab120 con 0x55ba5b4a8090 2024-04-02T14:44:23.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.871+0000 7fc344244740 10 client.0 _ll_register_callbacks cb 0x55ba5b3bb030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:44:23.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.872+0000 7fc33b400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ba5b49ed20 0x55ba5b4a4f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:23.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.872+0000 7fc33a000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ba5b49fa00 0x55ba5b4a54b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:23.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.872+0000 7fc33b400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ba5b49ed20 0x55ba5b4a4f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:36280/0 (socket says 172.21.15.90:36280) 2024-04-02T14:44:23.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.872+0000 7fc33b400640 1 -- 172.21.15.90:0/3027012645 learned_addr learned my addr 172.21.15.90:0/3027012645 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:44:23.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.872+0000 7fc33b400640 1 --2- 172.21.15.90:0/3027012645 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ba5b4a8090 0x55ba5b4a59f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:23.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.872+0000 7fc33a000640 1 -- 172.21.15.90:0/3027012645 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ba5b4a8090 msgr2=0x55ba5b4a59f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:44:23.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.872+0000 7fc33a000640 1 --2- 172.21.15.90:0/3027012645 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ba5b4a8090 0x55ba5b4a59f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:23.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.872+0000 7fc33a000640 1 -- 172.21.15.90:0/3027012645 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ba5b49ed20 msgr2=0x55ba5b4a4f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:44:23.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.872+0000 7fc33a000640 1 --2- 172.21.15.90:0/3027012645 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ba5b49ed20 0x55ba5b4a4f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:23.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.872+0000 7fc33a000640 1 -- 172.21.15.90:0/3027012645 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ba5b236f70 con 0x55ba5b49fa00 2024-04-02T14:44:23.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.872+0000 7fc33b400640 1 --2- 172.21.15.90:0/3027012645 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ba5b4a8090 0x55ba5b4a59f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:44:23.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.872+0000 7fc33a000640 1 --2- 172.21.15.90:0/3027012645 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ba5b49fa00 0x55ba5b4a54b0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fc32802c270 tx=0x7fc328004340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6bc10610ebe82ab0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:44:23.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.872+0000 7fc31be00640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-02T14:44:23.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.873+0000 7fc31be00640 1 -- 172.21.15.90:0/3027012645 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc32802c810 con 0x55ba5b49fa00 2024-04-02T14:44:23.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.873+0000 7fc31be00640 1 -- 172.21.15.90:0/3027012645 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8550+0+0 (secure 0 0 0) 0x7fc32802c9b0 con 0x55ba5b49fa00 2024-04-02T14:44:23.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.873+0000 7fc344244740 1 -- 172.21.15.90:0/3027012645 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc33400ee90 con 0x55ba5b49fa00 2024-04-02T14:44:23.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.874+0000 7fc31be00640 1 -- 172.21.15.90:0/3027012645 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fc32802cbb0 con 0x55ba5b49fa00 2024-04-02T14:44:23.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.874+0000 7fc344244740 10 client.17577 fetch_fsmap learned FSMap version 27 2024-04-02T14:44:23.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.874+0000 7fc344244740 1 -- 172.21.15.90:0/3027012645 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55ba5b0ce170 con 0x55ba5b49fa00 2024-04-02T14:44:23.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.875+0000 7fc31be00640 1 -- 172.21.15.90:0/3027012645 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc33400ee90 con 0x55ba5b49fa00 2024-04-02T14:44:23.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.875+0000 7fc31be00640 1 -- 172.21.15.90:0/3027012645 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fc328004c80 con 0x55ba5b49fa00 2024-04-02T14:44:23.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.876+0000 7fc344244740 10 client.17577 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:44:23.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.876+0000 7fc344244740 10 client.17577 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:44:23.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.876+0000 7fc344244740 20 client.17577 populate_metadata read hostname 'smithi090' 2024-04-02T14:44:23.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.876+0000 7fc344244740 1 --2- 172.21.15.90:0/3027012645 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55ba5b4b84c0 0x55ba5b4ba8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:23.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.876+0000 7fc344244740 4 client.17577 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:44:23.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.876+0000 7fc344244740 1 -- 172.21.15.90:0/3027012645 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ba5b407850 con 0x55ba5b4b84c0 2024-04-02T14:44:23.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.876+0000 7fc33b400640 1 --2- 172.21.15.90:0/3027012645 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55ba5b4b84c0 0x55ba5b4ba8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:23.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.877+0000 7fc31be00640 10 client.17577 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:44:23.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.877+0000 7fc33b400640 1 --2- 172.21.15.90:0/3027012645 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55ba5b4b84c0 0x55ba5b4ba8b0 crc :-1 s=READY pgs=256 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bd6aa8c6d5356e02 server_cookie=fa1fa3bb7df21966 in_seq=0 out_seq=0 2024-04-02T14:44:23.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.877+0000 7fc31be00640 1 -- 172.21.15.90:0/3027012645 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ba5b407850 con 0x55ba5b4b84c0 2024-04-02T14:44:23.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.877+0000 7fc31be00640 10 client.17577 handle_command_reply: tid=0 2024-04-02T14:44:23.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.878+0000 7fc344244740 1 client.17577 shutdown 2024-04-02T14:44:23.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.878+0000 7fc344244740 1 -- 172.21.15.90:0/3027012645 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ba5b49fa00 msgr2=0x55ba5b4a54b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:23.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.878+0000 7fc344244740 1 --2- 172.21.15.90:0/3027012645 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ba5b49fa00 0x55ba5b4a54b0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fc32802c270 tx=0x7fc328004340 comp rx=0 tx=0).stop 2024-04-02T14:44:23.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.878+0000 7fc344244740 1 -- 172.21.15.90:0/3027012645 shutdown_connections 2024-04-02T14:44:23.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.878+0000 7fc344244740 1 --2- 172.21.15.90:0/3027012645 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55ba5b4b84c0 0x55ba5b4ba8b0 crc :-1 s=READY pgs=256 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:23.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.878+0000 7fc344244740 1 --2- 172.21.15.90:0/3027012645 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ba5b4a8090 0x55ba5b4a59f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:23.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.878+0000 7fc344244740 1 --2- 172.21.15.90:0/3027012645 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ba5b49fa00 0x55ba5b4a54b0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:23.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.878+0000 7fc344244740 1 --2- 172.21.15.90:0/3027012645 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ba5b49ed20 0x55ba5b4a4f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:23.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.878+0000 7fc344244740 1 -- 172.21.15.90:0/3027012645 >> 172.21.15.90:0/3027012645 conn(0x55ba5b42a270 msgr2=0x55ba5b49b4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:44:23.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.878+0000 7fc31be00640 0 client.17577 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:44:23.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.878+0000 7fc344244740 1 -- 172.21.15.90:0/3027012645 shutdown_connections 2024-04-02T14:44:23.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.878+0000 7fc344244740 1 -- 172.21.15.90:0/3027012645 wait complete. 2024-04-02T14:44:23.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.878+0000 7fc344244740 20 client.17577 trim_cache size 0 max 16384 2024-04-02T14:44:23.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.906+0000 7fc344244740 1 Processor -- start 2024-04-02T14:44:23.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.907+0000 7fc344244740 1 -- start start 2024-04-02T14:44:23.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.907+0000 7fc344244740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ba5b49ed20 0x55ba5b4a76a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:23.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.907+0000 7fc344244740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ba5b49fa00 0x55ba5b4b3f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:23.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.907+0000 7fc344244740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ba5b4a8090 0x55ba5b4c0150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:23.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.907+0000 7fc344244740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fc33c1ab580 con 0x55ba5b49ed20 2024-04-02T14:44:23.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.907+0000 7fc344244740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fc33c1ad0c0 con 0x55ba5b49fa00 2024-04-02T14:44:23.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.907+0000 7fc344244740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fc33c1abc50 con 0x55ba5b4a8090 2024-04-02T14:44:23.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.907+0000 7fc33b400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ba5b49ed20 0x55ba5b4a76a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:23.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.907+0000 7fc33b400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ba5b49ed20 0x55ba5b4a76a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41522/0 (socket says 172.21.15.90:41522) 2024-04-02T14:44:23.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.907+0000 7fc33b400640 1 -- 172.21.15.90:0/3916966330 learned_addr learned my addr 172.21.15.90:0/3916966330 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:44:23.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.907+0000 7fc33a000640 1 --2- 172.21.15.90:0/3916966330 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ba5b49fa00 0x55ba5b4b3f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:23.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.907+0000 7fc33b400640 1 --2- 172.21.15.90:0/3916966330 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ba5b4a8090 0x55ba5b4c0150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:23.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.908+0000 7fc33b400640 1 -- 172.21.15.90:0/3916966330 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ba5b4a8090 msgr2=0x55ba5b4c0150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:44:23.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.908+0000 7fc33b400640 1 --2- 172.21.15.90:0/3916966330 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ba5b4a8090 0x55ba5b4c0150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:23.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.908+0000 7fc33b400640 1 -- 172.21.15.90:0/3916966330 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ba5b49fa00 msgr2=0x55ba5b4b3f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:44:23.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.908+0000 7fc33b400640 1 --2- 172.21.15.90:0/3916966330 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ba5b49fa00 0x55ba5b4b3f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:23.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.908+0000 7fc33b400640 1 -- 172.21.15.90:0/3916966330 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ba5b4a3ef0 con 0x55ba5b49ed20 2024-04-02T14:44:23.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.908+0000 7fc33a000640 1 --2- 172.21.15.90:0/3916966330 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ba5b49fa00 0x55ba5b4b3f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:44:23.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.908+0000 7fc33b400640 1 --2- 172.21.15.90:0/3916966330 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ba5b49ed20 0x55ba5b4a76a0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fc334011fd0 tx=0x7fc334010f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9343f9f25a8fba33 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:44:23.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.910+0000 7fc326a00640 1 -- 172.21.15.90:0/3916966330 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc334014040 con 0x55ba5b49ed20 2024-04-02T14:44:23.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.910+0000 7fc326a00640 1 -- 172.21.15.90:0/3916966330 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc334003e50 con 0x55ba5b49ed20 2024-04-02T14:44:23.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.910+0000 7fc326a00640 1 -- 172.21.15.90:0/3916966330 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc334005600 con 0x55ba5b49ed20 2024-04-02T14:44:23.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.911+0000 7fc344244740 1 -- 172.21.15.90:0/3916966330 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ba5b49ed20 msgr2=0x55ba5b4a76a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:23.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.911+0000 7fc344244740 1 --2- 172.21.15.90:0/3916966330 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ba5b49ed20 0x55ba5b4a76a0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fc334011fd0 tx=0x7fc334010f20 comp rx=0 tx=0).stop 2024-04-02T14:44:23.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.911+0000 7fc344244740 1 -- 172.21.15.90:0/3916966330 shutdown_connections 2024-04-02T14:44:23.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.911+0000 7fc344244740 1 --2- 172.21.15.90:0/3916966330 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ba5b4a8090 0x55ba5b4c0150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:23.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.911+0000 7fc344244740 1 --2- 172.21.15.90:0/3916966330 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ba5b49fa00 0x55ba5b4b3f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:23.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.911+0000 7fc344244740 1 --2- 172.21.15.90:0/3916966330 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ba5b49ed20 0x55ba5b4a76a0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:23.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.911+0000 7fc344244740 1 -- 172.21.15.90:0/3916966330 >> 172.21.15.90:0/3916966330 conn(0x55ba5b42a270 msgr2=0x55ba5b49b450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:44:23.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.911+0000 7fc344244740 1 -- 172.21.15.90:0/3916966330 shutdown_connections 2024-04-02T14:44:23.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.911+0000 7fc344244740 1 -- 172.21.15.90:0/3916966330 wait complete. 2024-04-02T14:44:23.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.912+0000 7fc344244740 1 -- start start 2024-04-02T14:44:23.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.912+0000 7fc344244740 1 Processor -- start 2024-04-02T14:44:23.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.912+0000 7fc344244740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ba5b49ed20 0x55ba5b4b06c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:23.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.912+0000 7fc344244740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ba5b49fa00 0x55ba5b4ad430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:23.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.912+0000 7fc344244740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ba5b4a8090 0x55ba5b4ad970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:23.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.913+0000 7fc344244740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fc33c1ac9f0 con 0x55ba5b49ed20 2024-04-02T14:44:23.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.913+0000 7fc344244740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fc33c1ac320 con 0x55ba5b49fa00 2024-04-02T14:44:23.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.913+0000 7fc344244740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fc33c1a95e0 con 0x55ba5b4a8090 2024-04-02T14:44:23.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.913+0000 7fc344244740 10 client.0 _ll_register_callbacks cb 0x7fc328009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:44:23.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.913+0000 7fc33b400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ba5b49ed20 0x55ba5b4b06c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:23.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.913+0000 7fc33a000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ba5b49fa00 0x55ba5b4ad430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:23.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.913+0000 7fc33b400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ba5b49ed20 0x55ba5b4b06c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41532/0 (socket says 172.21.15.90:41532) 2024-04-02T14:44:23.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.913+0000 7fc33b400640 1 -- 172.21.15.90:0/538360463 learned_addr learned my addr 172.21.15.90:0/538360463 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:44:23.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.913+0000 7fc33b400640 1 --2- 172.21.15.90:0/538360463 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ba5b4a8090 0x55ba5b4ad970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:23.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.913+0000 7fc33a000640 1 -- 172.21.15.90:0/538360463 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ba5b4a8090 msgr2=0x55ba5b4ad970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:44:23.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.913+0000 7fc33a000640 1 --2- 172.21.15.90:0/538360463 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ba5b4a8090 0x55ba5b4ad970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:23.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.913+0000 7fc33a000640 1 -- 172.21.15.90:0/538360463 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ba5b49ed20 msgr2=0x55ba5b4b06c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:44:23.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.913+0000 7fc33a000640 1 --2- 172.21.15.90:0/538360463 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ba5b49ed20 0x55ba5b4b06c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:23.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.913+0000 7fc33a000640 1 -- 172.21.15.90:0/538360463 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ba5b4adfe0 con 0x55ba5b49fa00 2024-04-02T14:44:23.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.913+0000 7fc33a000640 1 --2- 172.21.15.90:0/538360463 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ba5b49fa00 0x55ba5b4ad430 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7fc328009900 tx=0x7fc32802ca40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b71989e6ff00a1 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:44:23.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.913+0000 7fc31be00640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:44:23.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.914+0000 7fc31be00640 1 -- 172.21.15.90:0/538360463 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc328053070 con 0x55ba5b49fa00 2024-04-02T14:44:23.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.914+0000 7fc31be00640 1 -- 172.21.15.90:0/538360463 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8550+0+0 (secure 0 0 0) 0x7fc3280035f0 con 0x55ba5b49fa00 2024-04-02T14:44:23.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.914+0000 7fc344244740 1 -- 172.21.15.90:0/538360463 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc334003e50 con 0x55ba5b49fa00 2024-04-02T14:44:23.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.914+0000 7fc31be00640 1 -- 172.21.15.90:0/538360463 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fc32805f050 con 0x55ba5b49fa00 2024-04-02T14:44:23.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.914+0000 7fc344244740 10 client.28469 fetch_fsmap learned FSMap version 27 2024-04-02T14:44:23.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.914+0000 7fc344244740 1 -- 172.21.15.90:0/538360463 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55ba5b51f880 con 0x55ba5b49fa00 2024-04-02T14:44:23.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.914+0000 7fc31be00640 1 -- 172.21.15.90:0/538360463 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc334003e50 con 0x55ba5b49fa00 2024-04-02T14:44:23.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.915+0000 7fc31be00640 1 -- 172.21.15.90:0/538360463 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fc328058440 con 0x55ba5b49fa00 2024-04-02T14:44:23.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.915+0000 7fc344244740 10 client.28469 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:44:23.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.915+0000 7fc344244740 10 client.28469 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:44:23.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.915+0000 7fc344244740 20 client.28469 populate_metadata read hostname 'smithi090' 2024-04-02T14:44:23.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.915+0000 7fc344244740 1 --2- 172.21.15.90:0/538360463 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55ba5b520030 0x55ba5b522450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:23.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.915+0000 7fc344244740 4 client.28469 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-02T14:44:23.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.915+0000 7fc344244740 1 -- 172.21.15.90:0/538360463 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55ba5b401ba0 con 0x55ba5b520030 2024-04-02T14:44:23.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.915+0000 7fc33b400640 1 --2- 172.21.15.90:0/538360463 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55ba5b520030 0x55ba5b522450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:23.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.916+0000 7fc33b400640 1 --2- 172.21.15.90:0/538360463 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55ba5b520030 0x55ba5b522450 crc :-1 s=READY pgs=257 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=93f985bef593c98a server_cookie=1c1090e196ede5c5 in_seq=0 out_seq=0 2024-04-02T14:44:23.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.916+0000 7fc31be00640 10 client.28469 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:44:23.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.917+0000 7fc31be00640 1 -- 172.21.15.90:0/538360463 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55ba5b401ba0 con 0x55ba5b520030 2024-04-02T14:44:23.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.917+0000 7fc31be00640 10 client.28469 handle_command_reply: tid=0 2024-04-02T14:44:23.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.917+0000 7fc344244740 1 client.28469 shutdown 2024-04-02T14:44:23.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.917+0000 7fc344244740 1 -- 172.21.15.90:0/538360463 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ba5b49fa00 msgr2=0x55ba5b4ad430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:23.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.917+0000 7fc344244740 1 --2- 172.21.15.90:0/538360463 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ba5b49fa00 0x55ba5b4ad430 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7fc328009900 tx=0x7fc32802ca40 comp rx=0 tx=0).stop 2024-04-02T14:44:23.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.917+0000 7fc344244740 1 -- 172.21.15.90:0/538360463 shutdown_connections 2024-04-02T14:44:23.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.917+0000 7fc344244740 1 --2- 172.21.15.90:0/538360463 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55ba5b520030 0x55ba5b522450 crc :-1 s=READY pgs=257 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:23.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.917+0000 7fc344244740 1 --2- 172.21.15.90:0/538360463 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ba5b4a8090 0x55ba5b4ad970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:23.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.917+0000 7fc344244740 1 --2- 172.21.15.90:0/538360463 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ba5b49fa00 0x55ba5b4ad430 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:23.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.917+0000 7fc344244740 1 --2- 172.21.15.90:0/538360463 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ba5b49ed20 0x55ba5b4b06c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:23.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.917+0000 7fc31be00640 0 client.28469 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:44:23.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.917+0000 7fc344244740 1 -- 172.21.15.90:0/538360463 >> 172.21.15.90:0/538360463 conn(0x55ba5b42a270 msgr2=0x55ba5b48f670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:44:23.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.918+0000 7fc344244740 1 -- 172.21.15.90:0/538360463 shutdown_connections 2024-04-02T14:44:23.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.918+0000 7fc344244740 1 -- 172.21.15.90:0/538360463 wait complete. 2024-04-02T14:44:23.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.918+0000 7fc344244740 20 client.28469 trim_cache size 0 max 16384 2024-04-02T14:44:23.919 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-02T14:44:23.919 INFO:teuthology.orchestra.run.smithi090.stdout: "return_code": 0, 2024-04-02T14:44:23.919 INFO:teuthology.orchestra.run.smithi090.stdout: "scrub_tag": "e9c18b75-a6e7-4f2f-9899-f5f892b96fc2", 2024-04-02T14:44:23.919 INFO:teuthology.orchestra.run.smithi090.stdout: "mode": "asynchronous" 2024-04-02T14:44:23.919 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-02T14:44:23.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.918+0000 7fc33be00640 1 -- 172.21.15.90:0/4116824136 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fc3200a3ef0 msgr2=0x7fc3200a63b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:23.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.918+0000 7fc33be00640 1 --2- 172.21.15.90:0/4116824136 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fc3200a3ef0 0x7fc3200a63b0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x55ba5b49ff50 tx=0x7fc32c000bc0 comp rx=0 tx=0).stop 2024-04-02T14:44:23.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.918+0000 7fc33be00640 1 -- 172.21.15.90:0/4116824136 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc33c101960 msgr2=0x7fc33c19a3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:23.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.918+0000 7fc33be00640 1 --2- 172.21.15.90:0/4116824136 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc33c101960 0x7fc33c19a3b0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7fc32c00e9f0 tx=0x7fc32c00eeb0 comp rx=0 tx=0).stop 2024-04-02T14:44:23.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.919+0000 7fc33be00640 1 -- 172.21.15.90:0/4116824136 shutdown_connections 2024-04-02T14:44:23.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.919+0000 7fc33be00640 1 --2- 172.21.15.90:0/4116824136 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fc3200a3ef0 0x7fc3200a63b0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:23.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.919+0000 7fc33be00640 1 --2- 172.21.15.90:0/4116824136 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc33c107970 0x7fc33c19ec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:23.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.919+0000 7fc33be00640 1 --2- 172.21.15.90:0/4116824136 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc33c102320 0x7fc33c19a8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:23.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.919+0000 7fc33be00640 1 --2- 172.21.15.90:0/4116824136 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc33c101960 0x7fc33c19a3b0 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:23.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.919+0000 7fc33be00640 1 -- 172.21.15.90:0/4116824136 >> 172.21.15.90:0/4116824136 conn(0x7fc33c0fd470 msgr2=0x7fc33c0fd850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:44:23.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.919+0000 7fc33be00640 1 -- 172.21.15.90:0/4116824136 shutdown_connections 2024-04-02T14:44:23.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:23.919+0000 7fc33be00640 1 -- 172.21.15.90:0/4116824136 wait complete. 2024-04-02T14:44:23.938 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-02T14:44:23.938 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-02T14:44:24.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.234+0000 7f15b5c00640 1 -- 172.21.15.90:0/3475895739 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f15b010e240 msgr2=0x7f15b010e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:24.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.234+0000 7f15b5c00640 1 --2- 172.21.15.90:0/3475895739 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f15b010e240 0x7f15b010e620 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f15a8009c10 tx=0x7f15a802bef0 comp rx=0 tx=0).stop 2024-04-02T14:44:24.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.234+0000 7f15b5c00640 1 -- 172.21.15.90:0/3475895739 shutdown_connections 2024-04-02T14:44:24.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.234+0000 7f15b5c00640 1 --2- 172.21.15.90:0/3475895739 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f15b0109300 0x7f15b0115000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:24.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.234+0000 7f15b5c00640 1 --2- 172.21.15.90:0/3475895739 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f15b0108940 0x7f15b0108dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:24.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.234+0000 7f15b5c00640 1 --2- 172.21.15.90:0/3475895739 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f15b010e240 0x7f15b010e620 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:24.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.234+0000 7f15b5c00640 1 -- 172.21.15.90:0/3475895739 >> 172.21.15.90:0/3475895739 conn(0x7f15b0077860 msgr2=0x7f15b0079c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:44:24.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.234+0000 7f15b5c00640 1 -- 172.21.15.90:0/3475895739 shutdown_connections 2024-04-02T14:44:24.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.234+0000 7f15b5c00640 1 -- 172.21.15.90:0/3475895739 wait complete. 2024-04-02T14:44:24.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.235+0000 7f15b5c00640 1 Processor -- start 2024-04-02T14:44:24.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.236+0000 7f15b5c00640 1 -- start start 2024-04-02T14:44:24.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.236+0000 7f15b5c00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f15b0108940 0x7f15b01a4790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:24.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.236+0000 7f15b5c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f15b0109300 0x7f15b01a4cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:24.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.236+0000 7f15b5c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f15b010e240 0x7f15b019e990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:24.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.236+0000 7f15b5c00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f15b0118590 con 0x7f15b0109300 2024-04-02T14:44:24.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.236+0000 7f15b5c00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f15b0118410 con 0x7f15b0108940 2024-04-02T14:44:24.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.236+0000 7f15b5c00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f15b0118710 con 0x7f15b010e240 2024-04-02T14:44:24.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.237+0000 7f15af400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f15b010e240 0x7f15b019e990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:24.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.237+0000 7f15af400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f15b010e240 0x7f15b019e990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:43966/0 (socket says 172.21.15.90:43966) 2024-04-02T14:44:24.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.237+0000 7f15ae000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f15b0109300 0x7f15b01a4cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:24.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.237+0000 7f15af400640 1 -- 172.21.15.90:0/3781754865 learned_addr learned my addr 172.21.15.90:0/3781754865 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:44:24.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.237+0000 7f15aea00640 1 --2- 172.21.15.90:0/3781754865 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f15b0108940 0x7f15b01a4790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:24.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.237+0000 7f15aea00640 1 -- 172.21.15.90:0/3781754865 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f15b010e240 msgr2=0x7f15b019e990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:24.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.237+0000 7f15aea00640 1 --2- 172.21.15.90:0/3781754865 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f15b010e240 0x7f15b019e990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:24.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.237+0000 7f15aea00640 1 -- 172.21.15.90:0/3781754865 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f15b0109300 msgr2=0x7f15b01a4cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:24.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.237+0000 7f15aea00640 1 --2- 172.21.15.90:0/3781754865 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f15b0109300 0x7f15b01a4cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:24.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.237+0000 7f15aea00640 1 -- 172.21.15.90:0/3781754865 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15b019f110 con 0x7f15b0108940 2024-04-02T14:44:24.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.238+0000 7f15aea00640 1 --2- 172.21.15.90:0/3781754865 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f15b0108940 0x7f15b01a4790 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f15a8009b30 tx=0x7f15a80336b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:44:24.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.238+0000 7f15a7400640 1 -- 172.21.15.90:0/3781754865 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15a8043020 con 0x7f15b0108940 2024-04-02T14:44:24.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.239+0000 7f15a7400640 1 -- 172.21.15.90:0/3781754865 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f15a8003b80 con 0x7f15b0108940 2024-04-02T14:44:24.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.239+0000 7f15a7400640 1 -- 172.21.15.90:0/3781754865 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15a803e740 con 0x7f15b0108940 2024-04-02T14:44:24.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.239+0000 7f15b5c00640 1 -- 172.21.15.90:0/3781754865 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f15b019f3a0 con 0x7f15b0108940 2024-04-02T14:44:24.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.240+0000 7f15b5c00640 1 -- 172.21.15.90:0/3781754865 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15b01ab650 con 0x7f15b0108940 2024-04-02T14:44:24.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.244+0000 7f15b8860740 1 Processor -- start 2024-04-02T14:44:24.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.245+0000 7f15b8860740 1 -- start start 2024-04-02T14:44:24.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.245+0000 7f15b8860740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561252bc97b0 0x561252bc9b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:24.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.245+0000 7f15b8860740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561252bca0d0 0x561252bd2770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:24.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.245+0000 7f15b8860740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561252bd2cb0 0x561252bd50a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:24.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.245+0000 7f15b8860740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x561252b2d770 con 0x561252bd2cb0 2024-04-02T14:44:24.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.245+0000 7f15b8860740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x561252ad5700 con 0x561252bca0d0 2024-04-02T14:44:24.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.245+0000 7f15b8860740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x561252b29720 con 0x561252bc97b0 2024-04-02T14:44:24.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.246+0000 7f15af400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561252bc97b0 0x561252bc9b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:24.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.246+0000 7f15a7400640 1 -- 172.21.15.90:0/3781754865 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f15a8046020 con 0x7f15b0108940 2024-04-02T14:44:24.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.246+0000 7f15ae000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561252bd2cb0 0x561252bd50a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:24.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.246+0000 7f15ae000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561252bd2cb0 0x561252bd50a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41554/0 (socket says 172.21.15.90:41554) 2024-04-02T14:44:24.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.246+0000 7f15ae000640 1 -- 172.21.15.90:0/409687560 learned_addr learned my addr 172.21.15.90:0/409687560 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:44:24.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.246+0000 7f15ae000640 1 -- 172.21.15.90:0/409687560 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561252bc97b0 msgr2=0x561252bc9b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:44:24.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.246+0000 7f15ae000640 1 --2- 172.21.15.90:0/409687560 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561252bc97b0 0x561252bc9b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:24.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.246+0000 7f15ae000640 1 -- 172.21.15.90:0/409687560 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561252bca0d0 msgr2=0x561252bd2770 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-02T14:44:24.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.246+0000 7f15ae000640 1 --2- 172.21.15.90:0/409687560 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561252bca0d0 0x561252bd2770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:24.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.246+0000 7f15ae000640 1 -- 172.21.15.90:0/409687560 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56125290b7d0 con 0x561252bd2cb0 2024-04-02T14:44:24.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.247+0000 7f15ae000640 1 --2- 172.21.15.90:0/409687560 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561252bd2cb0 0x561252bd50a0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f159800b710 tx=0x7f15980045f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe7865c42ae4cf76 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:44:24.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.247+0000 7f159fe00640 1 -- 172.21.15.90:0/409687560 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1598003bd0 con 0x561252bd2cb0 2024-04-02T14:44:24.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.247+0000 7f159fe00640 1 -- 172.21.15.90:0/409687560 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1598015070 con 0x561252bd2cb0 2024-04-02T14:44:24.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.247+0000 7f159fe00640 1 -- 172.21.15.90:0/409687560 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15980106a0 con 0x561252bd2cb0 2024-04-02T14:44:24.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.248+0000 7f15a7400640 1 --2- 172.21.15.90:0/3781754865 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f15940a3f20 0x7f15940a63e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:24.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.248+0000 7f15a7400640 1 -- 172.21.15.90:0/3781754865 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8550+0+0 (secure 0 0 0) 0x7f15a80f2d60 con 0x7f15b0108940 2024-04-02T14:44:24.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.248+0000 7f15b8860740 1 -- 172.21.15.90:0/409687560 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561252bd2cb0 msgr2=0x561252bd50a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:24.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.248+0000 7f15b8860740 1 --2- 172.21.15.90:0/409687560 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561252bd2cb0 0x561252bd50a0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f159800b710 tx=0x7f15980045f0 comp rx=0 tx=0).stop 2024-04-02T14:44:24.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.249+0000 7f15b8860740 1 -- 172.21.15.90:0/409687560 shutdown_connections 2024-04-02T14:44:24.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.249+0000 7f15b8860740 1 --2- 172.21.15.90:0/409687560 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561252bd2cb0 0x561252bd50a0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:24.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.249+0000 7f15b8860740 1 --2- 172.21.15.90:0/409687560 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561252bca0d0 0x561252bd2770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:24.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.249+0000 7f15b8860740 1 --2- 172.21.15.90:0/409687560 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561252bc97b0 0x561252bc9b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:24.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.249+0000 7f15b8860740 1 -- 172.21.15.90:0/409687560 >> 172.21.15.90:0/409687560 conn(0x561252bc3ee0 msgr2=0x561252bc42c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:44:24.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.249+0000 7f15af400640 1 --2- 172.21.15.90:0/3781754865 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f15940a3f20 0x7f15940a63e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:24.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.249+0000 7f15b8860740 1 -- 172.21.15.90:0/409687560 shutdown_connections 2024-04-02T14:44:24.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.249+0000 7f15b8860740 1 -- 172.21.15.90:0/409687560 wait complete. 2024-04-02T14:44:24.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.250+0000 7f15b8860740 1 -- start start 2024-04-02T14:44:24.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.250+0000 7f15af400640 1 --2- 172.21.15.90:0/3781754865 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f15940a3f20 0x7f15940a63e0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f15a000bb80 tx=0x7f15a000b390 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:44:24.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.251+0000 7f15b8860740 1 Processor -- start 2024-04-02T14:44:24.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.252+0000 7f15b8860740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561252bc97b0 0x561252bcfcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:24.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.252+0000 7f15b8860740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561252bca0d0 0x561252bd0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:24.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.252+0000 7f15b8860740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561252bd2cb0 0x561252bd0740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:24.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.252+0000 7f15b8860740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x561252bd5aa0 con 0x561252bca0d0 2024-04-02T14:44:24.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.252+0000 7f15b8860740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x561252ae8ac0 con 0x561252bd2cb0 2024-04-02T14:44:24.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.252+0000 7f15b8860740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x561252bd5c20 con 0x561252bc97b0 2024-04-02T14:44:24.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.252+0000 7f15b8860740 10 client.0 _ll_register_callbacks cb 0x561252aeb180 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:44:24.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.252+0000 7f15af400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561252bd2cb0 0x561252bd0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:24.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.253+0000 7f15af400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561252bd2cb0 0x561252bd0740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:36334/0 (socket says 172.21.15.90:36334) 2024-04-02T14:44:24.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.253+0000 7f15af400640 1 -- 172.21.15.90:0/37563109 learned_addr learned my addr 172.21.15.90:0/37563109 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:44:24.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.253+0000 7f15ae000640 1 --2- 172.21.15.90:0/37563109 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561252bca0d0 0x561252bd0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:24.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.253+0000 7f15af400640 1 -- 172.21.15.90:0/37563109 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561252bc97b0 msgr2=0x561252bcfcc0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-02T14:44:24.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.253+0000 7f15af400640 1 --2- 172.21.15.90:0/37563109 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561252bc97b0 0x561252bcfcc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:24.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.253+0000 7f15af400640 1 -- 172.21.15.90:0/37563109 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561252bca0d0 msgr2=0x561252bd0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:44:24.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.253+0000 7f15af400640 1 --2- 172.21.15.90:0/37563109 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561252bca0d0 0x561252bd0200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:24.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.253+0000 7f15af400640 1 -- 172.21.15.90:0/37563109 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5612528413f0 con 0x561252bd2cb0 2024-04-02T14:44:24.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.254+0000 7f15af400640 1 --2- 172.21.15.90:0/37563109 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561252bd2cb0 0x561252bd0740 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f15a00126d0 tx=0x7f15a00046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7d54177842533b22 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:44:24.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.254+0000 7f159cc00640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:44:24.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.254+0000 7f159cc00640 1 -- 172.21.15.90:0/37563109 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15a001f8f0 con 0x561252bd2cb0 2024-04-02T14:44:24.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.254+0000 7f159cc00640 1 -- 172.21.15.90:0/37563109 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8550+0+0 (secure 0 0 0) 0x7f15a001fa90 con 0x561252bd2cb0 2024-04-02T14:44:24.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.254+0000 7f15b8860740 1 -- 172.21.15.90:0/37563109 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1598015070 con 0x561252bd2cb0 2024-04-02T14:44:24.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.254+0000 7f159cc00640 1 -- 172.21.15.90:0/37563109 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f15a001fc90 con 0x561252bd2cb0 2024-04-02T14:44:24.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.255+0000 7f15b8860740 10 client.28478 fetch_fsmap learned FSMap version 27 2024-04-02T14:44:24.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.255+0000 7f15b8860740 1 -- 172.21.15.90:0/37563109 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5612528010f0 con 0x561252bd2cb0 2024-04-02T14:44:24.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.255+0000 7f159cc00640 1 -- 172.21.15.90:0/37563109 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1598015070 con 0x561252bd2cb0 2024-04-02T14:44:24.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.255+0000 7f159cc00640 1 -- 172.21.15.90:0/37563109 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f15a0027400 con 0x561252bd2cb0 2024-04-02T14:44:24.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.255+0000 7f15b8860740 10 client.28478 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:44:24.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.255+0000 7f15b8860740 10 client.28478 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:44:24.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.255+0000 7f15b8860740 20 client.28478 populate_metadata read hostname 'smithi090' 2024-04-02T14:44:24.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.255+0000 7f15b8860740 1 --2- 172.21.15.90:0/37563109 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x561252bf3e30 0x561252bf62b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:24.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.255+0000 7f15b8860740 4 client.28478 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:44:24.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.256+0000 7f15b8860740 1 -- 172.21.15.90:0/37563109 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561252ace630 con 0x561252bf3e30 2024-04-02T14:44:24.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.256+0000 7f15aea00640 1 --2- 172.21.15.90:0/37563109 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x561252bf3e30 0x561252bf62b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:24.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.256+0000 7f15aea00640 1 --2- 172.21.15.90:0/37563109 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x561252bf3e30 0x561252bf62b0 crc :-1 s=READY pgs=258 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8dac946f43da11f3 server_cookie=cc49b00ac265a492 in_seq=0 out_seq=0 2024-04-02T14:44:24.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.256+0000 7f159cc00640 10 client.28478 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:44:24.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.257+0000 7f159cc00640 1 -- 172.21.15.90:0/37563109 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x561252ace630 con 0x561252bf3e30 2024-04-02T14:44:24.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.257+0000 7f159cc00640 10 client.28478 handle_command_reply: tid=0 2024-04-02T14:44:24.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.259+0000 7f15b8860740 1 client.28478 shutdown 2024-04-02T14:44:24.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.260+0000 7f15b8860740 1 -- 172.21.15.90:0/37563109 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561252bd2cb0 msgr2=0x561252bd0740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:24.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.260+0000 7f15b8860740 1 --2- 172.21.15.90:0/37563109 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561252bd2cb0 0x561252bd0740 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f15a00126d0 tx=0x7f15a00046c0 comp rx=0 tx=0).stop 2024-04-02T14:44:24.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.260+0000 7f15b8860740 1 -- 172.21.15.90:0/37563109 shutdown_connections 2024-04-02T14:44:24.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.260+0000 7f15b8860740 1 --2- 172.21.15.90:0/37563109 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x561252bf3e30 0x561252bf62b0 crc :-1 s=READY pgs=258 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:24.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.260+0000 7f15b8860740 1 --2- 172.21.15.90:0/37563109 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561252bd2cb0 0x561252bd0740 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:24.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.260+0000 7f15b8860740 1 --2- 172.21.15.90:0/37563109 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561252bca0d0 0x561252bd0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:24.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.260+0000 7f15b8860740 1 --2- 172.21.15.90:0/37563109 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561252bc97b0 0x561252bcfcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:24.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.260+0000 7f15b8860740 1 -- 172.21.15.90:0/37563109 >> 172.21.15.90:0/37563109 conn(0x561252bc3ee0 msgr2=0x561252bc5c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:44:24.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.260+0000 7f159cc00640 0 client.28478 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:44:24.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.261+0000 7f15b8860740 1 -- 172.21.15.90:0/37563109 shutdown_connections 2024-04-02T14:44:24.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.261+0000 7f15b8860740 1 -- 172.21.15.90:0/37563109 wait complete. 2024-04-02T14:44:24.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.261+0000 7f15b8860740 20 client.28478 trim_cache size 0 max 16384 2024-04-02T14:44:24.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.290+0000 7f15b8860740 1 Processor -- start 2024-04-02T14:44:24.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.290+0000 7f15b8860740 1 -- start start 2024-04-02T14:44:24.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.290+0000 7f15b8860740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561252bc97b0 0x561252bd23f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:24.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.291+0000 7f15b8860740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561252bca0d0 0x561252bcec00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:24.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.291+0000 7f15b8860740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561252bd2cb0 0x561252bde990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:24.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.291+0000 7f15b8860740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f15b0073b40 con 0x561252bd2cb0 2024-04-02T14:44:24.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.291+0000 7f15b8860740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f15b01ad170 con 0x561252bca0d0 2024-04-02T14:44:24.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.291+0000 7f15b8860740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f15b01abd00 con 0x561252bc97b0 2024-04-02T14:44:24.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.291+0000 7f15aea00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561252bc97b0 0x561252bd23f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:24.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.291+0000 7f15af400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561252bd2cb0 0x561252bde990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:24.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.291+0000 7f15aea00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561252bc97b0 0x561252bd23f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:43974/0 (socket says 172.21.15.90:43974) 2024-04-02T14:44:24.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.291+0000 7f15aea00640 1 -- 172.21.15.90:0/3747375535 learned_addr learned my addr 172.21.15.90:0/3747375535 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:44:24.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.291+0000 7f15ae000640 1 --2- 172.21.15.90:0/3747375535 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561252bca0d0 0x561252bcec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:24.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.291+0000 7f15aea00640 1 -- 172.21.15.90:0/3747375535 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561252bca0d0 msgr2=0x561252bcec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:44:24.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.291+0000 7f15aea00640 1 --2- 172.21.15.90:0/3747375535 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561252bca0d0 0x561252bcec00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:24.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.291+0000 7f15aea00640 1 -- 172.21.15.90:0/3747375535 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561252bd2cb0 msgr2=0x561252bde990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:44:24.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.291+0000 7f15aea00640 1 --2- 172.21.15.90:0/3747375535 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561252bd2cb0 0x561252bde990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:24.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.291+0000 7f15aea00640 1 -- 172.21.15.90:0/3747375535 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561252bc9180 con 0x561252bc97b0 2024-04-02T14:44:24.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.292+0000 7f15aea00640 1 --2- 172.21.15.90:0/3747375535 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561252bc97b0 0x561252bd23f0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f15a80bb760 tx=0x7f15a802bef0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c4f1d1252befbe33 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:44:24.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.292+0000 7f159f400640 1 -- 172.21.15.90:0/3747375535 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15a80be6f0 con 0x561252bc97b0 2024-04-02T14:44:24.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.292+0000 7f159f400640 1 -- 172.21.15.90:0/3747375535 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f15a80bf490 con 0x561252bc97b0 2024-04-02T14:44:24.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.292+0000 7f159f400640 1 -- 172.21.15.90:0/3747375535 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15a80d8270 con 0x561252bc97b0 2024-04-02T14:44:24.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.293+0000 7f15b8860740 1 -- 172.21.15.90:0/3747375535 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561252bc97b0 msgr2=0x561252bd23f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:24.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.293+0000 7f15b8860740 1 --2- 172.21.15.90:0/3747375535 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561252bc97b0 0x561252bd23f0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f15a80bb760 tx=0x7f15a802bef0 comp rx=0 tx=0).stop 2024-04-02T14:44:24.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.293+0000 7f15b8860740 1 -- 172.21.15.90:0/3747375535 shutdown_connections 2024-04-02T14:44:24.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.293+0000 7f15b8860740 1 --2- 172.21.15.90:0/3747375535 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561252bd2cb0 0x561252bde990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:24.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.293+0000 7f15b8860740 1 --2- 172.21.15.90:0/3747375535 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561252bca0d0 0x561252bcec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:24.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.293+0000 7f15b8860740 1 --2- 172.21.15.90:0/3747375535 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561252bc97b0 0x561252bd23f0 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:24.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.293+0000 7f15b8860740 1 -- 172.21.15.90:0/3747375535 >> 172.21.15.90:0/3747375535 conn(0x561252bc3ee0 msgr2=0x561252bd0510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:44:24.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.293+0000 7f15b8860740 1 -- 172.21.15.90:0/3747375535 shutdown_connections 2024-04-02T14:44:24.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.293+0000 7f15b8860740 1 -- 172.21.15.90:0/3747375535 wait complete. 2024-04-02T14:44:24.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.293+0000 7f15b8860740 1 -- start start 2024-04-02T14:44:24.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.293+0000 7f15b8860740 1 Processor -- start 2024-04-02T14:44:24.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.294+0000 7f15b8860740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561252bc97b0 0x561252bccbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:24.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.294+0000 7f15b8860740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561252bca0d0 0x561252bcd110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:24.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.294+0000 7f15b8860740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561252bd2cb0 0x561252bcd650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:24.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.294+0000 7f15b8860740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f15b01acaa0 con 0x561252bca0d0 2024-04-02T14:44:24.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.294+0000 7f15b8860740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f15b01ac3d0 con 0x561252bd2cb0 2024-04-02T14:44:24.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.294+0000 7f15b8860740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f15b0071b40 con 0x561252bc97b0 2024-04-02T14:44:24.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.294+0000 7f15b8860740 10 client.0 _ll_register_callbacks cb 0x7f15a000b870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:44:24.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.294+0000 7f15ae000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561252bca0d0 0x561252bcd110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:24.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.294+0000 7f15ae000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561252bca0d0 0x561252bcd110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41578/0 (socket says 172.21.15.90:41578) 2024-04-02T14:44:24.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.294+0000 7f15ae000640 1 -- 172.21.15.90:0/2710908581 learned_addr learned my addr 172.21.15.90:0/2710908581 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:44:24.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.294+0000 7f15aea00640 1 --2- 172.21.15.90:0/2710908581 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561252bc97b0 0x561252bccbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:24.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.294+0000 7f15af400640 1 --2- 172.21.15.90:0/2710908581 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561252bd2cb0 0x561252bcd650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:24.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.294+0000 7f15ae000640 1 -- 172.21.15.90:0/2710908581 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561252bc97b0 msgr2=0x561252bccbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:44:24.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.295+0000 7f15ae000640 1 --2- 172.21.15.90:0/2710908581 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561252bc97b0 0x561252bccbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:24.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.295+0000 7f15ae000640 1 -- 172.21.15.90:0/2710908581 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561252bd2cb0 msgr2=0x561252bcd650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:44:24.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.295+0000 7f15ae000640 1 --2- 172.21.15.90:0/2710908581 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561252bd2cb0 0x561252bcd650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:24.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.295+0000 7f15ae000640 1 -- 172.21.15.90:0/2710908581 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561252c59510 con 0x561252bca0d0 2024-04-02T14:44:24.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.296+0000 7f15aea00640 1 --2- 172.21.15.90:0/2710908581 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561252bc97b0 0x561252bccbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:44:24.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.296+0000 7f15af400640 1 --2- 172.21.15.90:0/2710908581 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561252bd2cb0 0x561252bcd650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:44:24.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.297+0000 7f15ae000640 1 --2- 172.21.15.90:0/2710908581 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561252bca0d0 0x561252bcd110 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f159800b710 tx=0x7f1598011f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92d69b1943694409 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:44:24.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.297+0000 7f159cc00640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-02T14:44:24.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.298+0000 7f159cc00640 1 -- 172.21.15.90:0/2710908581 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1598028020 con 0x561252bca0d0 2024-04-02T14:44:24.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.298+0000 7f15b8860740 1 -- 172.21.15.90:0/2710908581 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f15a80bf490 con 0x561252bca0d0 2024-04-02T14:44:24.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.298+0000 7f159cc00640 1 -- 172.21.15.90:0/2710908581 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8550+0+0 (secure 0 0 0) 0x7f1598011440 con 0x561252bca0d0 2024-04-02T14:44:24.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.299+0000 7f159cc00640 1 -- 172.21.15.90:0/2710908581 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f1598033050 con 0x561252bca0d0 2024-04-02T14:44:24.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.299+0000 7f15b8860740 10 client.17601 fetch_fsmap learned FSMap version 27 2024-04-02T14:44:24.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.299+0000 7f15b8860740 1 -- 172.21.15.90:0/2710908581 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x561252c5a820 con 0x561252bca0d0 2024-04-02T14:44:24.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.299+0000 7f159cc00640 1 -- 172.21.15.90:0/2710908581 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f15a80bf490 con 0x561252bca0d0 2024-04-02T14:44:24.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.299+0000 7f159cc00640 1 -- 172.21.15.90:0/2710908581 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f1598004770 con 0x561252bca0d0 2024-04-02T14:44:24.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.300+0000 7f15b8860740 10 client.17601 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:44:24.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.300+0000 7f15b8860740 10 client.17601 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:44:24.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.300+0000 7f15b8860740 20 client.17601 populate_metadata read hostname 'smithi090' 2024-04-02T14:44:24.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.300+0000 7f15b8860740 1 --2- 172.21.15.90:0/2710908581 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x561252c5ac70 0x561252c5d060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:24.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.300+0000 7f15b8860740 4 client.17601 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-02T14:44:24.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.300+0000 7f15b8860740 1 -- 172.21.15.90:0/2710908581 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "scrub status"}) -- 0x561252b3acb0 con 0x561252c5ac70 2024-04-02T14:44:24.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.300+0000 7f15aea00640 1 --2- 172.21.15.90:0/2710908581 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x561252c5ac70 0x561252c5d060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:24.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.301+0000 7f159cc00640 10 client.17601 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:44:24.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.301+0000 7f15aea00640 1 --2- 172.21.15.90:0/2710908581 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x561252c5ac70 0x561252c5d060 crc :-1 s=READY pgs=259 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e48ce22e287aa7de server_cookie=32319065df38a645 in_seq=0 out_seq=0 2024-04-02T14:44:24.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.301+0000 7f159cc00640 1 -- 172.21.15.90:0/2710908581 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x561252b3acb0 con 0x561252c5ac70 2024-04-02T14:44:24.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.301+0000 7f159cc00640 10 client.17601 handle_command_reply: tid=0 2024-04-02T14:44:24.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.301+0000 7f15b8860740 1 client.17601 shutdown 2024-04-02T14:44:24.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.301+0000 7f15b8860740 1 -- 172.21.15.90:0/2710908581 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561252bca0d0 msgr2=0x561252bcd110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:24.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.301+0000 7f15b8860740 1 --2- 172.21.15.90:0/2710908581 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561252bca0d0 0x561252bcd110 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f159800b710 tx=0x7f1598011f00 comp rx=0 tx=0).stop 2024-04-02T14:44:24.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.301+0000 7f15b8860740 1 -- 172.21.15.90:0/2710908581 shutdown_connections 2024-04-02T14:44:24.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.301+0000 7f15b8860740 1 --2- 172.21.15.90:0/2710908581 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x561252c5ac70 0x561252c5d060 crc :-1 s=READY pgs=259 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:24.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.301+0000 7f15b8860740 1 --2- 172.21.15.90:0/2710908581 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561252bd2cb0 0x561252bcd650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:24.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.301+0000 7f15b8860740 1 --2- 172.21.15.90:0/2710908581 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561252bca0d0 0x561252bcd110 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:24.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.301+0000 7f15b8860740 1 --2- 172.21.15.90:0/2710908581 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561252bc97b0 0x561252bccbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:24.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.301+0000 7f15b8860740 1 -- 172.21.15.90:0/2710908581 >> 172.21.15.90:0/2710908581 conn(0x561252bc3ee0 msgr2=0x561252bc5230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:44:24.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.301+0000 7f159cc00640 0 client.17601 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:44:24.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.302+0000 7f15b8860740 1 -- 172.21.15.90:0/2710908581 shutdown_connections 2024-04-02T14:44:24.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.302+0000 7f15b8860740 1 -- 172.21.15.90:0/2710908581 wait complete. 2024-04-02T14:44:24.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.302+0000 7f15b8860740 20 client.17601 trim_cache size 0 max 16384 2024-04-02T14:44:24.301 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-02T14:44:24.301 INFO:teuthology.orchestra.run.smithi090.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-02T14:44:24.301 INFO:teuthology.orchestra.run.smithi090.stdout: "scrubs": { 2024-04-02T14:44:24.301 INFO:teuthology.orchestra.run.smithi090.stdout: "e9c18b75-a6e7-4f2f-9899-f5f892b96fc2": { 2024-04-02T14:44:24.301 INFO:teuthology.orchestra.run.smithi090.stdout: "path": "/", 2024-04-02T14:44:24.301 INFO:teuthology.orchestra.run.smithi090.stdout: "tag": "e9c18b75-a6e7-4f2f-9899-f5f892b96fc2", 2024-04-02T14:44:24.301 INFO:teuthology.orchestra.run.smithi090.stdout: "options": "recursive,force" 2024-04-02T14:44:24.301 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:44:24.301 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:44:24.301 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-02T14:44:24.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.303+0000 7f15b5c00640 1 -- 172.21.15.90:0/3781754865 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f15940a3f20 msgr2=0x7f15940a63e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:24.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.303+0000 7f15b5c00640 1 --2- 172.21.15.90:0/3781754865 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f15940a3f20 0x7f15940a63e0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f15a000bb80 tx=0x7f15a000b390 comp rx=0 tx=0).stop 2024-04-02T14:44:24.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.303+0000 7f15b5c00640 1 -- 172.21.15.90:0/3781754865 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f15b0108940 msgr2=0x7f15b01a4790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:24.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.303+0000 7f15b5c00640 1 --2- 172.21.15.90:0/3781754865 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f15b0108940 0x7f15b01a4790 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f15a8009b30 tx=0x7f15a80336b0 comp rx=0 tx=0).stop 2024-04-02T14:44:24.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.303+0000 7f15b5c00640 1 -- 172.21.15.90:0/3781754865 shutdown_connections 2024-04-02T14:44:24.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.303+0000 7f15b5c00640 1 --2- 172.21.15.90:0/3781754865 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f15940a3f20 0x7f15940a63e0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:24.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.303+0000 7f15b5c00640 1 --2- 172.21.15.90:0/3781754865 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f15b010e240 0x7f15b019e990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:24.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.303+0000 7f15b5c00640 1 --2- 172.21.15.90:0/3781754865 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f15b0109300 0x7f15b01a4cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:24.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.303+0000 7f15b5c00640 1 --2- 172.21.15.90:0/3781754865 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f15b0108940 0x7f15b01a4790 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:24.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.303+0000 7f15b5c00640 1 -- 172.21.15.90:0/3781754865 >> 172.21.15.90:0/3781754865 conn(0x7f15b0077860 msgr2=0x7f15b010b080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:44:24.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.304+0000 7f15b5c00640 1 -- 172.21.15.90:0/3781754865 shutdown_connections 2024-04-02T14:44:24.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:24.304+0000 7f15b5c00640 1 -- 172.21.15.90:0/3781754865 wait complete. 2024-04-02T14:44:24.322 INFO:tasks.cephfs.filesystem:scrub status for tag:e9c18b75-a6e7-4f2f-9899-f5f892b96fc2 - {'path': '/', 'tag': 'e9c18b75-a6e7-4f2f-9899-f5f892b96fc2', 'options': 'recursive,force'} 2024-04-02T14:44:24.606 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:44:24 smithi092 ceph-mon[36497]: pgmap v294: 97 pgs: 97 active+clean; 4.4 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 6.4 MiB/s wr, 580 op/s 2024-04-02T14:44:24.681 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:44:24 smithi177 ceph-mon[36431]: pgmap v294: 97 pgs: 97 active+clean; 4.4 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 6.4 MiB/s wr, 580 op/s 2024-04-02T14:44:24.683 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:44:24 smithi090 ceph-mon[31674]: pgmap v294: 97 pgs: 97 active+clean; 4.4 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 6.4 MiB/s wr, 580 op/s 2024-04-02T14:44:25.606 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:44:25 smithi092 ceph-mon[36497]: scrub queued for path: / 2024-04-02T14:44:25.606 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:44:25 smithi092 ceph-mon[36497]: scrub summary: idle+waiting paths [/] 2024-04-02T14:44:25.606 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:44:25 smithi092 ceph-mon[36497]: scrub summary: active paths [/] 2024-04-02T14:44:25.606 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:44:25 smithi092 ceph-mon[36497]: scrub summary: idle+waiting paths [/] 2024-04-02T14:44:25.682 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:44:25 smithi177 ceph-mon[36431]: scrub queued for path: / 2024-04-02T14:44:25.682 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:44:25 smithi177 ceph-mon[36431]: scrub summary: idle+waiting paths [/] 2024-04-02T14:44:25.682 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:44:25 smithi177 ceph-mon[36431]: scrub summary: active paths [/] 2024-04-02T14:44:25.682 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:44:25 smithi177 ceph-mon[36431]: scrub summary: idle+waiting paths [/] 2024-04-02T14:44:25.683 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:44:25 smithi090 ceph-mon[31674]: scrub queued for path: / 2024-04-02T14:44:25.683 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:44:25 smithi090 ceph-mon[31674]: scrub summary: idle+waiting paths [/] 2024-04-02T14:44:25.683 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:44:25 smithi090 ceph-mon[31674]: scrub summary: active paths [/] 2024-04-02T14:44:25.683 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:44:25 smithi090 ceph-mon[31674]: scrub summary: idle+waiting paths [/] 2024-04-02T14:44:26.606 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:44:26 smithi092 ceph-mon[36497]: pgmap v295: 97 pgs: 97 active+clean; 4.4 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 7.2 MiB/s wr, 707 op/s 2024-04-02T14:44:26.681 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:44:26 smithi177 ceph-mon[36431]: pgmap v295: 97 pgs: 97 active+clean; 4.4 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 7.2 MiB/s wr, 707 op/s 2024-04-02T14:44:26.683 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:44:26 smithi090 ceph-mon[31674]: pgmap v295: 97 pgs: 97 active+clean; 4.4 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 7.2 MiB/s wr, 707 op/s 2024-04-02T14:44:26.827 INFO:teuthology.orchestra.run.smithi090.stderr:progress: 535.0 s, 1469.4 tps, lat 68.317 ms stddev 79.436 2024-04-02T14:44:27.582 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:44:27 smithi177 ceph-mon[36431]: scrub summary: idle 2024-04-02T14:44:27.606 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:44:27 smithi092 ceph-mon[36497]: scrub summary: idle 2024-04-02T14:44:27.686 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:44:27 smithi090 ceph-mon[31674]: scrub summary: idle 2024-04-02T14:44:28.606 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:44:28 smithi092 ceph-mon[36497]: pgmap v296: 97 pgs: 97 active+clean; 4.4 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 6.3 MiB/s wr, 617 op/s 2024-04-02T14:44:28.682 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:44:28 smithi177 ceph-mon[36431]: pgmap v296: 97 pgs: 97 active+clean; 4.4 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 6.3 MiB/s wr, 617 op/s 2024-04-02T14:44:28.683 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:44:28 smithi090 ceph-mon[31674]: pgmap v296: 97 pgs: 97 active+clean; 4.4 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 6.3 MiB/s wr, 617 op/s 2024-04-02T14:44:30.606 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:44:30 smithi092 ceph-mon[36497]: pgmap v297: 97 pgs: 97 active+clean; 4.4 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 152 KiB/s rd, 7.0 MiB/s wr, 752 op/s 2024-04-02T14:44:30.682 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:44:30 smithi177 ceph-mon[36431]: pgmap v297: 97 pgs: 97 active+clean; 4.4 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 152 KiB/s rd, 7.0 MiB/s wr, 752 op/s 2024-04-02T14:44:30.683 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:44:30 smithi090 ceph-mon[31674]: pgmap v297: 97 pgs: 97 active+clean; 4.4 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 152 KiB/s rd, 7.0 MiB/s wr, 752 op/s 2024-04-02T14:44:31.826 INFO:teuthology.orchestra.run.smithi090.stderr:progress: 540.0 s, 2179.8 tps, lat 45.560 ms stddev 52.612 2024-04-02T14:44:32.565 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.567+0000 7f695c400640 1 -- 172.21.15.90:0/438233620 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6954101570 msgr2=0x7f69541019f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:32.565 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.567+0000 7f695c400640 1 --2- 172.21.15.90:0/438233620 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6954101570 0x7f69541019f0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f6940009900 tx=0x7f694002bc30 comp rx=0 tx=0).stop 2024-04-02T14:44:32.565 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.567+0000 7f695c400640 1 -- 172.21.15.90:0/438233620 shutdown_connections 2024-04-02T14:44:32.565 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.567+0000 7f695c400640 1 --2- 172.21.15.90:0/438233620 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6954101f30 0x7f695410d890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:32.565 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.567+0000 7f695c400640 1 --2- 172.21.15.90:0/438233620 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6954101570 0x7f69541019f0 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:32.565 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.567+0000 7f695c400640 1 --2- 172.21.15.90:0/438233620 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6954105a00 0x7f6954105de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:32.565 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.567+0000 7f695c400640 1 -- 172.21.15.90:0/438233620 >> 172.21.15.90:0/438233620 conn(0x7f6954076b90 msgr2=0x7f69540fdee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:44:32.565 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.568+0000 7f695c400640 1 -- 172.21.15.90:0/438233620 shutdown_connections 2024-04-02T14:44:32.565 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.568+0000 7f695c400640 1 -- 172.21.15.90:0/438233620 wait complete. 2024-04-02T14:44:32.566 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.569+0000 7f695c400640 1 Processor -- start 2024-04-02T14:44:32.567 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.570+0000 7f695c400640 1 -- start start 2024-04-02T14:44:32.567 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.570+0000 7f695c400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6954101570 0x7f69541a00f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:32.567 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.570+0000 7f695c400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6954101f30 0x7f69541a0630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:32.567 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.570+0000 7f695c400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6954105a00 0x7f695419a700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:32.568 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.571+0000 7f695c400640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f6954110e80 con 0x7f6954101570 2024-04-02T14:44:32.568 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.571+0000 7f695c400640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f6954110d00 con 0x7f6954101f30 2024-04-02T14:44:32.568 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.571+0000 7f695c400640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f6954111000 con 0x7f6954105a00 2024-04-02T14:44:32.568 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.571+0000 7f6959a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6954101570 0x7f69541a00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:32.568 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.571+0000 7f6959000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6954101f30 0x7f69541a0630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:32.568 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.571+0000 7f6959000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6954101f30 0x7f69541a0630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:36372/0 (socket says 172.21.15.90:36372) 2024-04-02T14:44:32.568 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.571+0000 7f6959000640 1 -- 172.21.15.90:0/746376200 learned_addr learned my addr 172.21.15.90:0/746376200 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:44:32.568 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.571+0000 7f695a400640 1 --2- 172.21.15.90:0/746376200 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6954105a00 0x7f695419a700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:32.569 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.571+0000 7f6959000640 1 -- 172.21.15.90:0/746376200 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6954105a00 msgr2=0x7f695419a700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:32.569 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.571+0000 7f6959000640 1 --2- 172.21.15.90:0/746376200 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6954105a00 0x7f695419a700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:32.569 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.571+0000 7f6959000640 1 -- 172.21.15.90:0/746376200 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6954101570 msgr2=0x7f69541a00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:32.569 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.571+0000 7f6959000640 1 --2- 172.21.15.90:0/746376200 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6954101570 0x7f69541a00f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:32.569 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.571+0000 7f6959000640 1 -- 172.21.15.90:0/746376200 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f695419aed0 con 0x7f6954101f30 2024-04-02T14:44:32.569 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.572+0000 7f6959000640 1 --2- 172.21.15.90:0/746376200 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6954101f30 0x7f69541a0630 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f694002efd0 tx=0x7f6940004430 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:44:32.570 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.572+0000 7f6952000640 1 -- 172.21.15.90:0/746376200 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6940041070 con 0x7f6954101f30 2024-04-02T14:44:32.570 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.573+0000 7f6952000640 1 -- 172.21.15.90:0/746376200 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f69400054e0 con 0x7f6954101f30 2024-04-02T14:44:32.570 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.573+0000 7f695c400640 1 -- 172.21.15.90:0/746376200 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f695419b160 con 0x7f6954101f30 2024-04-02T14:44:32.570 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.573+0000 7f6952000640 1 -- 172.21.15.90:0/746376200 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6940003d40 con 0x7f6954101f30 2024-04-02T14:44:32.570 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.573+0000 7f695c400640 1 -- 172.21.15.90:0/746376200 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69541a6f40 con 0x7f6954101f30 2024-04-02T14:44:32.575 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.578+0000 7f6952000640 1 -- 172.21.15.90:0/746376200 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f6940044070 con 0x7f6954101f30 2024-04-02T14:44:32.577 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.579+0000 7f6952000640 1 --2- 172.21.15.90:0/746376200 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f693c0a3f20 0x7f693c0a63e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:32.577 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.580+0000 7f6952000640 1 -- 172.21.15.90:0/746376200 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8550+0+0 (secure 0 0 0) 0x7f69400f2160 con 0x7f6954101f30 2024-04-02T14:44:32.577 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.580+0000 7f6959a00640 1 --2- 172.21.15.90:0/746376200 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f693c0a3f20 0x7f693c0a63e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:32.578 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.581+0000 7f695f18a740 1 Processor -- start 2024-04-02T14:44:32.578 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.581+0000 7f695f18a740 1 -- start start 2024-04-02T14:44:32.579 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.582+0000 7f695f18a740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560f1222ecc0 0x560f1222f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:32.579 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.582+0000 7f695f18a740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560f1222f670 0x560f12240680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:32.579 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.582+0000 7f695f18a740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560f12240bc0 0x560f12242fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:32.580 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.582+0000 7f695f18a740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x560f12193d70 con 0x560f1222ecc0 2024-04-02T14:44:32.580 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.582+0000 7f695f18a740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x560f1213c2a0 con 0x560f1222f670 2024-04-02T14:44:32.580 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.582+0000 7f695f18a740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x560f1218fd20 con 0x560f12240bc0 2024-04-02T14:44:32.580 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.582+0000 7f6959a00640 1 --2- 172.21.15.90:0/746376200 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f693c0a3f20 0x7f693c0a63e0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f6948009bd0 tx=0x7f6948005f90 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:44:32.580 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.582+0000 7f695a400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560f12240bc0 0x560f12242fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:32.580 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.582+0000 7f695a400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560f12240bc0 0x560f12242fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:44016/0 (socket says 172.21.15.90:44016) 2024-04-02T14:44:32.580 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.582+0000 7f695a400640 1 -- 172.21.15.90:0/1177144699 learned_addr learned my addr 172.21.15.90:0/1177144699 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:44:32.580 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.582+0000 7f6959000640 1 --2- 172.21.15.90:0/1177144699 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560f1222f670 0x560f12240680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:32.581 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.583+0000 7f6959a00640 1 --2- 172.21.15.90:0/1177144699 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560f1222ecc0 0x560f1222f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:32.581 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.583+0000 7f6959000640 1 -- 172.21.15.90:0/1177144699 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560f12240bc0 msgr2=0x560f12242fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:44:32.581 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.583+0000 7f6959000640 1 --2- 172.21.15.90:0/1177144699 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560f12240bc0 0x560f12242fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:32.581 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.583+0000 7f6959000640 1 -- 172.21.15.90:0/1177144699 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560f1222ecc0 msgr2=0x560f1222f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:44:32.581 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.583+0000 7f6959000640 1 --2- 172.21.15.90:0/1177144699 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560f1222ecc0 0x560f1222f0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:32.581 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.583+0000 7f6959000640 1 -- 172.21.15.90:0/1177144699 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560f12150840 con 0x560f1222f670 2024-04-02T14:44:32.581 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.584+0000 7f6959000640 1 --2- 172.21.15.90:0/1177144699 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560f1222f670 0x560f12240680 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f69400ef910 tx=0x7f694002c7e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5e3d9864c2cba6d server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:44:32.581 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.584+0000 7f6946a00640 1 -- 172.21.15.90:0/1177144699 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f694004b040 con 0x560f1222f670 2024-04-02T14:44:32.582 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.584+0000 7f6946a00640 1 -- 172.21.15.90:0/1177144699 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6940004c80 con 0x560f1222f670 2024-04-02T14:44:32.582 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.584+0000 7f6946a00640 1 -- 172.21.15.90:0/1177144699 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69400ba8b0 con 0x560f1222f670 2024-04-02T14:44:32.583 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.586+0000 7f695f18a740 1 -- 172.21.15.90:0/1177144699 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560f1222f670 msgr2=0x560f12240680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:32.584 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.586+0000 7f695f18a740 1 --2- 172.21.15.90:0/1177144699 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560f1222f670 0x560f12240680 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f69400ef910 tx=0x7f694002c7e0 comp rx=0 tx=0).stop 2024-04-02T14:44:32.584 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.587+0000 7f695f18a740 1 -- 172.21.15.90:0/1177144699 shutdown_connections 2024-04-02T14:44:32.584 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.587+0000 7f695f18a740 1 --2- 172.21.15.90:0/1177144699 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560f12240bc0 0x560f12242fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:32.584 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.587+0000 7f695f18a740 1 --2- 172.21.15.90:0/1177144699 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560f1222f670 0x560f12240680 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:32.584 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.587+0000 7f695f18a740 1 --2- 172.21.15.90:0/1177144699 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560f1222ecc0 0x560f1222f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:32.584 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.587+0000 7f695f18a740 1 -- 172.21.15.90:0/1177144699 >> 172.21.15.90:0/1177144699 conn(0x560f122295e0 msgr2=0x560f122299c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:44:32.585 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.587+0000 7f695f18a740 1 -- 172.21.15.90:0/1177144699 shutdown_connections 2024-04-02T14:44:32.585 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.587+0000 7f695f18a740 1 -- 172.21.15.90:0/1177144699 wait complete. 2024-04-02T14:44:32.585 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.588+0000 7f695f18a740 1 -- start start 2024-04-02T14:44:32.585 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.588+0000 7f695f18a740 1 Processor -- start 2024-04-02T14:44:32.587 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.590+0000 7f695f18a740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560f1222ecc0 0x560f1223dce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:32.587 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.590+0000 7f695f18a740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560f1222f670 0x560f1223e220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:32.587 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.590+0000 7f695f18a740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560f12240bc0 0x560f1223e760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:32.587 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.590+0000 7f695f18a740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x560f12243740 con 0x560f12240bc0 2024-04-02T14:44:32.587 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.590+0000 7f695f18a740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x560f1214f660 con 0x560f1222ecc0 2024-04-02T14:44:32.587 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.590+0000 7f695f18a740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x560f122438c0 con 0x560f1222f670 2024-04-02T14:44:32.588 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.590+0000 7f695f18a740 10 client.0 _ll_register_callbacks cb 0x560f1214f050 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:44:32.588 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.590+0000 7f6959a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560f1222ecc0 0x560f1223dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:32.588 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.591+0000 7f6959000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560f1222f670 0x560f1223e220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:32.588 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.591+0000 7f6959a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560f1222ecc0 0x560f1223dce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:36402/0 (socket says 172.21.15.90:36402) 2024-04-02T14:44:32.589 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.591+0000 7f6959a00640 1 -- 172.21.15.90:0/2772947802 learned_addr learned my addr 172.21.15.90:0/2772947802 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:44:32.589 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.591+0000 7f695a400640 1 --2- 172.21.15.90:0/2772947802 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560f12240bc0 0x560f1223e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:32.589 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.591+0000 7f6959000640 1 -- 172.21.15.90:0/2772947802 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560f1222ecc0 msgr2=0x560f1223dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:44:32.589 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.591+0000 7f6959000640 1 --2- 172.21.15.90:0/2772947802 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560f1222ecc0 0x560f1223dce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:32.589 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.591+0000 7f6959000640 1 -- 172.21.15.90:0/2772947802 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560f12240bc0 msgr2=0x560f1223e760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:44:32.589 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.591+0000 7f6959000640 1 --2- 172.21.15.90:0/2772947802 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560f12240bc0 0x560f1223e760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:32.589 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.591+0000 7f6959000640 1 -- 172.21.15.90:0/2772947802 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560f1215ee80 con 0x560f1222f670 2024-04-02T14:44:32.589 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.591+0000 7f695a400640 1 --2- 172.21.15.90:0/2772947802 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560f12240bc0 0x560f1223e760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:44:32.590 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.592+0000 7f6959000640 1 --2- 172.21.15.90:0/2772947802 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560f1222f670 0x560f1223e220 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x560f122435d0 tx=0x7f6940049000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2307ae310e5c1465 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:44:32.590 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.592+0000 7f693b400640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:44:32.590 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.592+0000 7f693b400640 1 -- 172.21.15.90:0/2772947802 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69400db060 con 0x560f1222f670 2024-04-02T14:44:32.590 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.592+0000 7f693b400640 1 -- 172.21.15.90:0/2772947802 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8550+0+0 (secure 0 0 0) 0x7f69400c6970 con 0x560f1222f670 2024-04-02T14:44:32.590 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.592+0000 7f695f18a740 1 -- 172.21.15.90:0/2772947802 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6940004c80 con 0x560f1222f670 2024-04-02T14:44:32.590 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.593+0000 7f693b400640 1 -- 172.21.15.90:0/2772947802 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f694004b040 con 0x560f1222f670 2024-04-02T14:44:32.591 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.593+0000 7f695f18a740 10 client.28324 fetch_fsmap learned FSMap version 27 2024-04-02T14:44:32.591 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.593+0000 7f695f18a740 1 -- 172.21.15.90:0/2772947802 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x560f11ea8210 con 0x560f1222f670 2024-04-02T14:44:32.591 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.593+0000 7f693b400640 1 -- 172.21.15.90:0/2772947802 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f69400cb070 con 0x560f1222f670 2024-04-02T14:44:32.591 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.594+0000 7f693b400640 1 -- 172.21.15.90:0/2772947802 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f69400bd690 con 0x560f1222f670 2024-04-02T14:44:32.591 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.594+0000 7f695f18a740 10 client.28324 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:44:32.591 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.594+0000 7f695f18a740 10 client.28324 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-02T14:44:32.591 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.594+0000 7f695f18a740 20 client.28324 populate_metadata read hostname 'smithi090' 2024-04-02T14:44:32.591 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.594+0000 7f695f18a740 1 --2- 172.21.15.90:0/2772947802 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x560f12250bd0 0x560f12252fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:32.592 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.594+0000 7f695f18a740 4 client.28324 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:44:32.592 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.594+0000 7f695f18a740 1 -- 172.21.15.90:0/2772947802 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560f12143cd0 con 0x560f12250bd0 2024-04-02T14:44:32.592 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.594+0000 7f6959a00640 1 --2- 172.21.15.90:0/2772947802 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x560f12250bd0 0x560f12252fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:32.592 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.595+0000 7f6959a00640 1 --2- 172.21.15.90:0/2772947802 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x560f12250bd0 0x560f12252fc0 crc :-1 s=READY pgs=260 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2de71a288730fb97 server_cookie=d896ce39c722b96c in_seq=0 out_seq=0 2024-04-02T14:44:32.592 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.595+0000 7f693b400640 10 client.28324 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:44:32.594 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.596+0000 7f693b400640 1 -- 172.21.15.90:0/2772947802 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x560f12143cd0 con 0x560f12250bd0 2024-04-02T14:44:32.594 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.596+0000 7f693b400640 10 client.28324 handle_command_reply: tid=0 2024-04-02T14:44:32.594 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.597+0000 7f695f18a740 1 client.28324 shutdown 2024-04-02T14:44:32.598 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.601+0000 7f695f18a740 1 -- 172.21.15.90:0/2772947802 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560f1222f670 msgr2=0x560f1223e220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:32.598 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.601+0000 7f695f18a740 1 --2- 172.21.15.90:0/2772947802 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560f1222f670 0x560f1223e220 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x560f122435d0 tx=0x7f6940049000 comp rx=0 tx=0).stop 2024-04-02T14:44:32.599 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.601+0000 7f695f18a740 1 -- 172.21.15.90:0/2772947802 shutdown_connections 2024-04-02T14:44:32.599 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.601+0000 7f695f18a740 1 --2- 172.21.15.90:0/2772947802 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x560f12250bd0 0x560f12252fc0 crc :-1 s=READY pgs=260 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:32.599 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.601+0000 7f695f18a740 1 --2- 172.21.15.90:0/2772947802 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560f12240bc0 0x560f1223e760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:32.599 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.601+0000 7f695f18a740 1 --2- 172.21.15.90:0/2772947802 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560f1222f670 0x560f1223e220 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:32.599 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.601+0000 7f695f18a740 1 --2- 172.21.15.90:0/2772947802 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560f1222ecc0 0x560f1223dce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:32.599 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.601+0000 7f693b400640 0 client.28324 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:44:32.599 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.601+0000 7f695f18a740 1 -- 172.21.15.90:0/2772947802 >> 172.21.15.90:0/2772947802 conn(0x560f122295e0 msgr2=0x560f1222b330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:44:32.599 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.602+0000 7f695f18a740 1 -- 172.21.15.90:0/2772947802 shutdown_connections 2024-04-02T14:44:32.600 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.602+0000 7f695f18a740 1 -- 172.21.15.90:0/2772947802 wait complete. 2024-04-02T14:44:32.600 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.603+0000 7f695f18a740 20 client.28324 trim_cache size 0 max 16384 2024-04-02T14:44:32.605 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:44:32 smithi092 ceph-mon[36497]: pgmap v298: 97 pgs: 97 active+clean; 4.5 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 159 KiB/s rd, 7.9 MiB/s wr, 809 op/s 2024-04-02T14:44:32.647 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.648+0000 7f695f18a740 1 Processor -- start 2024-04-02T14:44:32.647 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.648+0000 7f695f18a740 1 -- start start 2024-04-02T14:44:32.648 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.649+0000 7f695f18a740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560f1222ecc0 0x560f12240410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:32.648 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.649+0000 7f695f18a740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560f1222f670 0x560f1222b140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:32.648 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.649+0000 7f695f18a740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560f12240bc0 0x560f122586b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:32.648 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.649+0000 7f695f18a740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f69541ab570 con 0x560f12240bc0 2024-04-02T14:44:32.648 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.649+0000 7f695f18a740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f69541ad0b0 con 0x560f1222f670 2024-04-02T14:44:32.648 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.649+0000 7f695f18a740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f69541abc40 con 0x560f1222ecc0 2024-04-02T14:44:32.648 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.649+0000 7f6959000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560f1222f670 0x560f1222b140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:32.648 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.649+0000 7f6959000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560f1222f670 0x560f1222b140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:36410/0 (socket says 172.21.15.90:36410) 2024-04-02T14:44:32.648 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.649+0000 7f6959000640 1 -- 172.21.15.90:0/1775082650 learned_addr learned my addr 172.21.15.90:0/1775082650 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:44:32.648 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.649+0000 7f695a400640 1 --2- 172.21.15.90:0/1775082650 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560f12240bc0 0x560f122586b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:32.648 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.649+0000 7f6959a00640 1 --2- 172.21.15.90:0/1775082650 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560f1222ecc0 0x560f12240410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:32.649 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.650+0000 7f6959000640 1 -- 172.21.15.90:0/1775082650 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560f1222ecc0 msgr2=0x560f12240410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:44:32.650 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.650+0000 7f6959000640 1 --2- 172.21.15.90:0/1775082650 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560f1222ecc0 0x560f12240410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:32.650 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.650+0000 7f6959000640 1 -- 172.21.15.90:0/1775082650 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560f12240bc0 msgr2=0x560f122586b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:44:32.650 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.650+0000 7f6959000640 1 --2- 172.21.15.90:0/1775082650 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560f12240bc0 0x560f122586b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:32.650 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.650+0000 7f6959000640 1 -- 172.21.15.90:0/1775082650 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560f11f74620 con 0x560f1222f670 2024-04-02T14:44:32.650 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.650+0000 7f695a400640 1 --2- 172.21.15.90:0/1775082650 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560f12240bc0 0x560f122586b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:44:32.650 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.650+0000 7f6959a00640 1 --2- 172.21.15.90:0/1775082650 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560f1222ecc0 0x560f12240410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:44:32.650 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.651+0000 7f6959000640 1 --2- 172.21.15.90:0/1775082650 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560f1222f670 0x560f1222b140 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f69400ba6a0 tx=0x7f6940005fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bd406f19ad7d3ad5 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:44:32.650 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.652+0000 7f6946000640 1 -- 172.21.15.90:0/1775082650 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69400db030 con 0x560f1222f670 2024-04-02T14:44:32.650 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.652+0000 7f6946000640 1 -- 172.21.15.90:0/1775082650 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f69400cb070 con 0x560f1222f670 2024-04-02T14:44:32.650 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.652+0000 7f6946000640 1 -- 172.21.15.90:0/1775082650 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69400be240 con 0x560f1222f670 2024-04-02T14:44:32.651 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.653+0000 7f695f18a740 1 -- 172.21.15.90:0/1775082650 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560f1222f670 msgr2=0x560f1222b140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:32.651 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.653+0000 7f695f18a740 1 --2- 172.21.15.90:0/1775082650 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560f1222f670 0x560f1222b140 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f69400ba6a0 tx=0x7f6940005fd0 comp rx=0 tx=0).stop 2024-04-02T14:44:32.651 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.654+0000 7f695f18a740 1 -- 172.21.15.90:0/1775082650 shutdown_connections 2024-04-02T14:44:32.651 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.654+0000 7f695f18a740 1 --2- 172.21.15.90:0/1775082650 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560f12240bc0 0x560f122586b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:32.651 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.654+0000 7f695f18a740 1 --2- 172.21.15.90:0/1775082650 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560f1222f670 0x560f1222b140 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:32.651 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.654+0000 7f695f18a740 1 --2- 172.21.15.90:0/1775082650 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560f1222ecc0 0x560f12240410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:32.651 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.654+0000 7f695f18a740 1 -- 172.21.15.90:0/1775082650 >> 172.21.15.90:0/1775082650 conn(0x560f122295e0 msgr2=0x560f1222aa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:44:32.652 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.655+0000 7f695f18a740 1 -- 172.21.15.90:0/1775082650 shutdown_connections 2024-04-02T14:44:32.652 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.655+0000 7f695f18a740 1 -- 172.21.15.90:0/1775082650 wait complete. 2024-04-02T14:44:32.653 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.655+0000 7f695f18a740 1 -- start start 2024-04-02T14:44:32.653 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.656+0000 7f695f18a740 1 Processor -- start 2024-04-02T14:44:32.654 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.656+0000 7f695f18a740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560f1222ecc0 0x560f12248b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:32.654 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.656+0000 7f695f18a740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560f1222f670 0x560f12249050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:32.654 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.656+0000 7f695f18a740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560f12240bc0 0x560f12237ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:32.654 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.656+0000 7f695f18a740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f69541ac9e0 con 0x560f1222f670 2024-04-02T14:44:32.654 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.656+0000 7f695f18a740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f69541ac310 con 0x560f1222ecc0 2024-04-02T14:44:32.654 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.656+0000 7f695f18a740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f69541a95b0 con 0x560f12240bc0 2024-04-02T14:44:32.654 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.656+0000 7f695f18a740 10 client.0 _ll_register_callbacks cb 0x7f6940009610 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:44:32.654 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.657+0000 7f6959a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560f1222ecc0 0x560f12248b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:32.655 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.657+0000 7f6959a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560f1222ecc0 0x560f12248b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:36420/0 (socket says 172.21.15.90:36420) 2024-04-02T14:44:32.655 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.657+0000 7f6959a00640 1 -- 172.21.15.90:0/3054084802 learned_addr learned my addr 172.21.15.90:0/3054084802 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:44:32.655 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.657+0000 7f695a400640 1 --2- 172.21.15.90:0/3054084802 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560f12240bc0 0x560f12237ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:32.656 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.657+0000 7f6959a00640 1 -- 172.21.15.90:0/3054084802 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560f12240bc0 msgr2=0x560f12237ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:44:32.656 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.657+0000 7f6959a00640 1 --2- 172.21.15.90:0/3054084802 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560f12240bc0 0x560f12237ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:32.656 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.657+0000 7f6959a00640 1 -- 172.21.15.90:0/3054084802 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560f1222f670 msgr2=0x560f12249050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:44:32.656 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.657+0000 7f6959a00640 1 --2- 172.21.15.90:0/3054084802 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560f1222f670 0x560f12249050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:32.656 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.657+0000 7f6959a00640 1 -- 172.21.15.90:0/3054084802 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560f12245ae0 con 0x560f1222ecc0 2024-04-02T14:44:32.656 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.658+0000 7f6959a00640 1 --2- 172.21.15.90:0/3054084802 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560f1222ecc0 0x560f12248b10 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x560f122535f0 tx=0x7f6948004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=96dc729a6a609673 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:44:32.656 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.658+0000 7f693b400640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:44:32.656 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.658+0000 7f693b400640 1 -- 172.21.15.90:0/3054084802 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f694802c4c0 con 0x560f1222ecc0 2024-04-02T14:44:32.656 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.658+0000 7f693b400640 1 -- 172.21.15.90:0/3054084802 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8550+0+0 (secure 0 0 0) 0x7f694802c660 con 0x560f1222ecc0 2024-04-02T14:44:32.656 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.659+0000 7f695f18a740 1 -- 172.21.15.90:0/3054084802 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f69400cb070 con 0x560f1222ecc0 2024-04-02T14:44:32.657 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.659+0000 7f693b400640 1 -- 172.21.15.90:0/3054084802 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f694802c860 con 0x560f1222ecc0 2024-04-02T14:44:32.657 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.659+0000 7f695f18a740 10 client.28502 fetch_fsmap learned FSMap version 27 2024-04-02T14:44:32.657 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.659+0000 7f695f18a740 1 -- 172.21.15.90:0/3054084802 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x560f122c0190 con 0x560f1222ecc0 2024-04-02T14:44:32.657 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.659+0000 7f693b400640 1 -- 172.21.15.90:0/3054084802 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f69400cb070 con 0x560f1222ecc0 2024-04-02T14:44:32.657 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.659+0000 7f693b400640 1 -- 172.21.15.90:0/3054084802 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f694803f4b0 con 0x560f1222ecc0 2024-04-02T14:44:32.657 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.660+0000 7f695f18a740 10 client.28502 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:44:32.657 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.660+0000 7f695f18a740 10 client.28502 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-02T14:44:32.658 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.660+0000 7f695f18a740 20 client.28502 populate_metadata read hostname 'smithi090' 2024-04-02T14:44:32.658 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.660+0000 7f695f18a740 1 --2- 172.21.15.90:0/3054084802 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x560f122c09e0 0x560f122c2e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:32.658 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.660+0000 7f695f18a740 4 client.28502 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-02T14:44:32.658 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.660+0000 7f695f18a740 1 -- 172.21.15.90:0/3054084802 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "flush journal"}) -- 0x560f121a12b0 con 0x560f122c09e0 2024-04-02T14:44:32.658 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.660+0000 7f6959000640 1 --2- 172.21.15.90:0/3054084802 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x560f122c09e0 0x560f122c2e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:32.659 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.661+0000 7f6959000640 1 --2- 172.21.15.90:0/3054084802 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x560f122c09e0 0x560f122c2e00 crc :-1 s=READY pgs=261 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b3b83d6e3ab412fb server_cookie=1d180e2b3834ba92 in_seq=0 out_seq=0 2024-04-02T14:44:32.659 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.662+0000 7f693b400640 10 client.28502 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:44:32.681 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:44:32 smithi177 ceph-mon[36431]: pgmap v298: 97 pgs: 97 active+clean; 4.5 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 159 KiB/s rd, 7.9 MiB/s wr, 809 op/s 2024-04-02T14:44:32.683 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:44:32 smithi090 ceph-mon[31674]: pgmap v298: 97 pgs: 97 active+clean; 4.5 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 159 KiB/s rd, 7.9 MiB/s wr, 809 op/s 2024-04-02T14:44:32.684 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.687+0000 7f693b400640 1 -- 172.21.15.90:0/3054084802 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x560f121a12b0 con 0x560f122c09e0 2024-04-02T14:44:32.684 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.687+0000 7f693b400640 10 client.28502 handle_command_reply: tid=0 2024-04-02T14:44:32.685 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.687+0000 7f695f18a740 1 client.28502 shutdown 2024-04-02T14:44:32.685 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.688+0000 7f695f18a740 1 -- 172.21.15.90:0/3054084802 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560f1222ecc0 msgr2=0x560f12248b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:32.685 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.688+0000 7f695f18a740 1 --2- 172.21.15.90:0/3054084802 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560f1222ecc0 0x560f12248b10 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x560f122535f0 tx=0x7f6948004730 comp rx=0 tx=0).stop 2024-04-02T14:44:32.685 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.688+0000 7f695f18a740 1 -- 172.21.15.90:0/3054084802 shutdown_connections 2024-04-02T14:44:32.685 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.688+0000 7f695f18a740 1 --2- 172.21.15.90:0/3054084802 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x560f122c09e0 0x560f122c2e00 crc :-1 s=READY pgs=261 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:32.686 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.688+0000 7f695f18a740 1 --2- 172.21.15.90:0/3054084802 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560f12240bc0 0x560f12237ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:32.686 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.688+0000 7f695f18a740 1 --2- 172.21.15.90:0/3054084802 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560f1222f670 0x560f12249050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:32.686 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.688+0000 7f695f18a740 1 --2- 172.21.15.90:0/3054084802 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560f1222ecc0 0x560f12248b10 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:32.686 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.688+0000 7f695f18a740 1 -- 172.21.15.90:0/3054084802 >> 172.21.15.90:0/3054084802 conn(0x560f122295e0 msgr2=0x560f1223dc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:44:32.686 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.688+0000 7f693b400640 0 client.28502 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:44:32.687 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.688+0000 7f695f18a740 1 -- 172.21.15.90:0/3054084802 shutdown_connections 2024-04-02T14:44:32.687 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.688+0000 7f695f18a740 1 -- 172.21.15.90:0/3054084802 wait complete. 2024-04-02T14:44:32.687 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.689+0000 7f695f18a740 20 client.28502 trim_cache size 0 max 16384 2024-04-02T14:44:32.687 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:{ 2024-04-02T14:44:32.687 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "message": "", 2024-04-02T14:44:32.688 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "return_code": 0 2024-04-02T14:44:32.688 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:} 2024-04-02T14:44:32.688 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.689+0000 7f695c400640 1 -- 172.21.15.90:0/746376200 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f693c0a3f20 msgr2=0x7f693c0a63e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:32.688 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.689+0000 7f695c400640 1 --2- 172.21.15.90:0/746376200 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f693c0a3f20 0x7f693c0a63e0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f6948009bd0 tx=0x7f6948005f90 comp rx=0 tx=0).stop 2024-04-02T14:44:32.689 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.689+0000 7f695c400640 1 -- 172.21.15.90:0/746376200 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6954101f30 msgr2=0x7f69541a0630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:32.689 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.689+0000 7f695c400640 1 --2- 172.21.15.90:0/746376200 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6954101f30 0x7f69541a0630 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f694002efd0 tx=0x7f6940004430 comp rx=0 tx=0).stop 2024-04-02T14:44:32.689 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.689+0000 7f695c400640 1 -- 172.21.15.90:0/746376200 shutdown_connections 2024-04-02T14:44:32.689 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.689+0000 7f695c400640 1 --2- 172.21.15.90:0/746376200 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f693c0a3f20 0x7f693c0a63e0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:32.689 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.689+0000 7f695c400640 1 --2- 172.21.15.90:0/746376200 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6954105a00 0x7f695419a700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:32.689 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.689+0000 7f695c400640 1 --2- 172.21.15.90:0/746376200 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6954101f30 0x7f69541a0630 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:32.689 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.689+0000 7f695c400640 1 --2- 172.21.15.90:0/746376200 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6954101570 0x7f69541a00f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:32.689 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.689+0000 7f695c400640 1 -- 172.21.15.90:0/746376200 >> 172.21.15.90:0/746376200 conn(0x7f6954076b90 msgr2=0x7f6954102620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:44:32.689 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.689+0000 7f695c400640 1 -- 172.21.15.90:0/746376200 shutdown_connections 2024-04-02T14:44:32.689 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:32.689+0000 7f695c400640 1 -- 172.21.15.90:0/746376200 wait complete. 2024-04-02T14:44:33.607 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:44:33 smithi092 ceph-mon[36497]: pgmap v299: 97 pgs: 97 active+clean; 4.5 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 158 KiB/s rd, 6.8 MiB/s wr, 689 op/s 2024-04-02T14:44:34.190 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:44:33 smithi090 ceph-mon[31674]: pgmap v299: 97 pgs: 97 active+clean; 4.5 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 158 KiB/s rd, 6.8 MiB/s wr, 689 op/s 2024-04-02T14:44:34.190 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:44:33 smithi177 ceph-mon[36431]: pgmap v299: 97 pgs: 97 active+clean; 4.5 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 158 KiB/s rd, 6.8 MiB/s wr, 689 op/s 2024-04-02T14:44:35.856 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:44:35 smithi092 ceph-mon[36497]: pgmap v300: 97 pgs: 97 active+clean; 4.5 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 174 KiB/s rd, 9.8 MiB/s wr, 975 op/s 2024-04-02T14:44:35.931 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:44:35 smithi177 ceph-mon[36431]: pgmap v300: 97 pgs: 97 active+clean; 4.5 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 174 KiB/s rd, 9.8 MiB/s wr, 975 op/s 2024-04-02T14:44:35.935 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:44:35 smithi090 ceph-mon[31674]: pgmap v300: 97 pgs: 97 active+clean; 4.5 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 174 KiB/s rd, 9.8 MiB/s wr, 975 op/s 2024-04-02T14:44:36.827 INFO:teuthology.orchestra.run.smithi090.stderr:progress: 545.0 s, 2759.4 tps, lat 35.625 ms stddev 41.541 2024-04-02T14:44:37.855 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:44:37 smithi092 ceph-mon[36497]: pgmap v301: 97 pgs: 97 active+clean; 4.5 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 7.0 MiB/s wr, 665 op/s 2024-04-02T14:44:37.931 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:44:37 smithi177 ceph-mon[36431]: pgmap v301: 97 pgs: 97 active+clean; 4.5 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 7.0 MiB/s wr, 665 op/s 2024-04-02T14:44:37.933 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:44:37 smithi090 ceph-mon[31674]: pgmap v301: 97 pgs: 97 active+clean; 4.5 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 7.0 MiB/s wr, 665 op/s 2024-04-02T14:44:39.855 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:44:39 smithi092 ceph-mon[36497]: pgmap v302: 97 pgs: 97 active+clean; 4.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 16 MiB/s wr, 1.30k op/s 2024-04-02T14:44:39.931 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:44:39 smithi177 ceph-mon[36431]: pgmap v302: 97 pgs: 97 active+clean; 4.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 16 MiB/s wr, 1.30k op/s 2024-04-02T14:44:39.933 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:44:39 smithi090 ceph-mon[31674]: pgmap v302: 97 pgs: 97 active+clean; 4.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 16 MiB/s wr, 1.30k op/s 2024-04-02T14:44:41.826 INFO:teuthology.orchestra.run.smithi090.stderr:progress: 550.0 s, 1524.9 tps, lat 66.439 ms stddev 149.924 2024-04-02T14:44:41.855 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:44:41 smithi092 ceph-mon[36497]: pgmap v303: 97 pgs: 97 active+clean; 4.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 26 MiB/s wr, 1.90k op/s 2024-04-02T14:44:41.931 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:44:41 smithi177 ceph-mon[36431]: pgmap v303: 97 pgs: 97 active+clean; 4.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 26 MiB/s wr, 1.90k op/s 2024-04-02T14:44:41.933 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:44:41 smithi090 ceph-mon[31674]: pgmap v303: 97 pgs: 97 active+clean; 4.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 26 MiB/s wr, 1.90k op/s 2024-04-02T14:44:43.931 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:44:43 smithi177 ceph-mon[36431]: pgmap v304: 97 pgs: 97 active+clean; 4.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 23 MiB/s wr, 1.69k op/s 2024-04-02T14:44:43.933 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:44:43 smithi090 ceph-mon[31674]: pgmap v304: 97 pgs: 97 active+clean; 4.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 23 MiB/s wr, 1.69k op/s 2024-04-02T14:44:44.105 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:44:43 smithi092 ceph-mon[36497]: pgmap v304: 97 pgs: 97 active+clean; 4.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 23 MiB/s wr, 1.69k op/s 2024-04-02T14:44:45.876 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.880+0000 7f7089a00640 1 -- 172.21.15.90:0/3162720362 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f70841048f0 msgr2=0x7f7084108930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:45.877 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.880+0000 7f7089a00640 1 --2- 172.21.15.90:0/3162720362 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f70841048f0 0x7f7084108930 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f7070009900 tx=0x7f707002bc30 comp rx=0 tx=0).stop 2024-04-02T14:44:45.877 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.880+0000 7f7089a00640 1 -- 172.21.15.90:0/3162720362 shutdown_connections 2024-04-02T14:44:45.877 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.880+0000 7f7089a00640 1 --2- 172.21.15.90:0/3162720362 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7084109010 0x7f7084110be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:45.878 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.880+0000 7f7089a00640 1 --2- 172.21.15.90:0/3162720362 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f70841048f0 0x7f7084108930 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:45.878 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.880+0000 7f7089a00640 1 --2- 172.21.15.90:0/3162720362 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7084103f40 0x7f7084104320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:45.878 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.880+0000 7f7089a00640 1 -- 172.21.15.90:0/3162720362 >> 172.21.15.90:0/3162720362 conn(0x7f70840ff7c0 msgr2=0x7f7084101be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:44:45.878 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.881+0000 7f7089a00640 1 -- 172.21.15.90:0/3162720362 shutdown_connections 2024-04-02T14:44:45.878 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.881+0000 7f7089a00640 1 -- 172.21.15.90:0/3162720362 wait complete. 2024-04-02T14:44:45.878 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.883+0000 7f7089a00640 1 Processor -- start 2024-04-02T14:44:45.879 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.883+0000 7f7089a00640 1 -- start start 2024-04-02T14:44:45.880 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.883+0000 7f7089a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7084103f40 0x7f708410fd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:45.880 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.883+0000 7f7089a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7084109010 0x7f7084110250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:45.880 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.883+0000 7f7089a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f70841b0dc0 0x7f70841b31b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:45.880 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.884+0000 7f7089a00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f70840777a0 con 0x7f70841b0dc0 2024-04-02T14:44:45.880 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.884+0000 7f7089a00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f7084077620 con 0x7f7084103f40 2024-04-02T14:44:45.880 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.884+0000 7f7089a00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f7084077920 con 0x7f7084109010 2024-04-02T14:44:45.881 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.884+0000 7f7082a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7084103f40 0x7f708410fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:45.881 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.884+0000 7f7082000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7084109010 0x7f7084110250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:45.881 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.884+0000 7f7082a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7084103f40 0x7f708410fd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:43716/0 (socket says 172.21.15.90:43716) 2024-04-02T14:44:45.881 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.884+0000 7f7082a00640 1 -- 172.21.15.90:0/3453519199 learned_addr learned my addr 172.21.15.90:0/3453519199 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:44:45.881 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.884+0000 7f7083400640 1 --2- 172.21.15.90:0/3453519199 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f70841b0dc0 0x7f70841b31b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:45.882 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.884+0000 7f7082a00640 1 -- 172.21.15.90:0/3453519199 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7084109010 msgr2=0x7f7084110250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:45.882 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.884+0000 7f7082a00640 1 --2- 172.21.15.90:0/3453519199 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7084109010 0x7f7084110250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:45.882 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.884+0000 7f7082a00640 1 -- 172.21.15.90:0/3453519199 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f70841b0dc0 msgr2=0x7f70841b31b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:45.882 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.884+0000 7f7082a00640 1 --2- 172.21.15.90:0/3453519199 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f70841b0dc0 0x7f70841b31b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:45.882 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.884+0000 7f7082a00640 1 -- 172.21.15.90:0/3453519199 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70841b37f0 con 0x7f7084103f40 2024-04-02T14:44:45.882 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.884+0000 7f7083400640 1 --2- 172.21.15.90:0/3453519199 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f70841b0dc0 0x7f70841b31b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:44:45.882 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.885+0000 7f7082a00640 1 --2- 172.21.15.90:0/3453519199 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7084103f40 0x7f708410fd10 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f707400e9f0 tx=0x7f707400edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:44:45.882 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.885+0000 7f707b400640 1 -- 172.21.15.90:0/3453519199 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f707400cd80 con 0x7f7084103f40 2024-04-02T14:44:45.882 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.885+0000 7f707b400640 1 -- 172.21.15.90:0/3453519199 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7074005500 con 0x7f7084103f40 2024-04-02T14:44:45.883 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.885+0000 7f707b400640 1 -- 172.21.15.90:0/3453519199 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7074010690 con 0x7f7084103f40 2024-04-02T14:44:45.883 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.885+0000 7f7089a00640 1 -- 172.21.15.90:0/3453519199 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70841b3ae0 con 0x7f7084103f40 2024-04-02T14:44:45.883 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.886+0000 7f7089a00640 1 -- 172.21.15.90:0/3453519199 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70841b3fb0 con 0x7f7084103f40 2024-04-02T14:44:45.886 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.890+0000 7f707b400640 1 -- 172.21.15.90:0/3453519199 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f70740046c0 con 0x7f7084103f40 2024-04-02T14:44:45.887 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.891+0000 7f708c684740 1 Processor -- start 2024-04-02T14:44:45.888 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.893+0000 7f708c684740 1 -- start start 2024-04-02T14:44:45.890 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.893+0000 7f707b400640 1 --2- 172.21.15.90:0/3453519199 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f70680a3fa0 0x7f70680a6460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:45.890 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.893+0000 7f708c684740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f95f681eb0 0x55f95f682290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:45.890 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.893+0000 7f708c684740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f95f6827d0 0x55f95f68ae70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:45.890 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.893+0000 7f708c684740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f95f68b3b0 0x55f95f68d7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:45.890 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.893+0000 7f708c684740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55f95f5e6d70 con 0x55f95f681eb0 2024-04-02T14:44:45.890 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.893+0000 7f708c684740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55f95f58f2a0 con 0x55f95f6827d0 2024-04-02T14:44:45.891 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.893+0000 7f708c684740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55f95f5e2d20 con 0x55f95f68b3b0 2024-04-02T14:44:45.891 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.894+0000 7f7082000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f95f6827d0 0x55f95f68ae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:45.891 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.894+0000 7f7082000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f95f6827d0 0x55f95f68ae70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:43728/0 (socket says 172.21.15.90:43728) 2024-04-02T14:44:45.891 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.894+0000 7f7082000640 1 -- 172.21.15.90:0/2079212203 learned_addr learned my addr 172.21.15.90:0/2079212203 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:44:45.891 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.894+0000 7f707b400640 1 -- 172.21.15.90:0/3453519199 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8550+0+0 (secure 0 0 0) 0x7f70740d0ba0 con 0x7f7084103f40 2024-04-02T14:44:45.891 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.894+0000 7f7082a00640 1 --2- 172.21.15.90:0/2079212203 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f95f681eb0 0x55f95f682290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:45.891 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.894+0000 7f7083400640 1 --2- 172.21.15.90:0/3453519199 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f70680a3fa0 0x7f70680a6460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:45.891 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.894+0000 7f7082000640 1 -- 172.21.15.90:0/2079212203 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f95f68b3b0 msgr2=0x55f95f68d7a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-02T14:44:45.891 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.894+0000 7f7082000640 1 --2- 172.21.15.90:0/2079212203 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f95f68b3b0 0x55f95f68d7a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:45.891 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.894+0000 7f7082000640 1 -- 172.21.15.90:0/2079212203 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f95f681eb0 msgr2=0x55f95f682290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:44:45.891 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.894+0000 7f7082000640 1 --2- 172.21.15.90:0/2079212203 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f95f681eb0 0x55f95f682290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:45.891 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.894+0000 7f7082000640 1 -- 172.21.15.90:0/2079212203 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f95f5a3840 con 0x55f95f6827d0 2024-04-02T14:44:45.892 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.894+0000 7f7082000640 1 --2- 172.21.15.90:0/2079212203 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f95f6827d0 0x55f95f68ae70 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f7070000c00 tx=0x7f7070009560 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e3dac950d2461ea0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:44:45.892 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.895+0000 7f706fe00640 1 -- 172.21.15.90:0/2079212203 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7070045070 con 0x55f95f6827d0 2024-04-02T14:44:45.892 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.895+0000 7f706fe00640 1 -- 172.21.15.90:0/2079212203 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f707002c650 con 0x55f95f6827d0 2024-04-02T14:44:45.892 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.895+0000 7f706fe00640 1 -- 172.21.15.90:0/2079212203 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7070004a40 con 0x55f95f6827d0 2024-04-02T14:44:45.893 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.897+0000 7f708c684740 1 -- 172.21.15.90:0/2079212203 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f95f6827d0 msgr2=0x55f95f68ae70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:45.893 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.897+0000 7f7083400640 1 --2- 172.21.15.90:0/3453519199 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f70680a3fa0 0x7f70680a6460 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f708410a5b0 tx=0x7f707c009500 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:44:45.893 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.897+0000 7f708c684740 1 --2- 172.21.15.90:0/2079212203 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f95f6827d0 0x55f95f68ae70 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f7070000c00 tx=0x7f7070009560 comp rx=0 tx=0).stop 2024-04-02T14:44:45.894 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.898+0000 7f708c684740 1 -- 172.21.15.90:0/2079212203 shutdown_connections 2024-04-02T14:44:45.894 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.898+0000 7f708c684740 1 --2- 172.21.15.90:0/2079212203 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f95f68b3b0 0x55f95f68d7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:45.894 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.898+0000 7f708c684740 1 --2- 172.21.15.90:0/2079212203 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f95f6827d0 0x55f95f68ae70 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:45.894 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.898+0000 7f708c684740 1 --2- 172.21.15.90:0/2079212203 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f95f681eb0 0x55f95f682290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:45.894 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.898+0000 7f708c684740 1 -- 172.21.15.90:0/2079212203 >> 172.21.15.90:0/2079212203 conn(0x55f95f67c5e0 msgr2=0x55f95f67c9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:44:45.895 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.898+0000 7f708c684740 1 -- 172.21.15.90:0/2079212203 shutdown_connections 2024-04-02T14:44:45.895 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.898+0000 7f708c684740 1 -- 172.21.15.90:0/2079212203 wait complete. 2024-04-02T14:44:45.895 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.899+0000 7f708c684740 1 -- start start 2024-04-02T14:44:45.895 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.899+0000 7f708c684740 1 Processor -- start 2024-04-02T14:44:45.895 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.899+0000 7f708c684740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f95f681eb0 0x55f95f6884d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:45.895 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.899+0000 7f708c684740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f95f6827d0 0x55f95f688a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:45.896 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.899+0000 7f708c684740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f95f68b3b0 0x55f95f688f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:45.896 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.899+0000 7f708c684740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55f95f68e140 con 0x55f95f68b3b0 2024-04-02T14:44:45.896 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.899+0000 7f708c684740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55f95f5a2660 con 0x55f95f6827d0 2024-04-02T14:44:45.896 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.899+0000 7f708c684740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55f95f68e2c0 con 0x55f95f681eb0 2024-04-02T14:44:45.896 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.900+0000 7f708c684740 10 client.0 _ll_register_callbacks cb 0x55f95f5a2050 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:44:45.897 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.900+0000 7f7082000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f95f6827d0 0x55f95f688a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:45.897 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.900+0000 7f7082000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f95f6827d0 0x55f95f688a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:43740/0 (socket says 172.21.15.90:43740) 2024-04-02T14:44:45.897 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.900+0000 7f7082000640 1 -- 172.21.15.90:0/55894493 learned_addr learned my addr 172.21.15.90:0/55894493 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:44:45.897 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.900+0000 7f7082000640 1 -- 172.21.15.90:0/55894493 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f95f681eb0 msgr2=0x55f95f6884d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:44:45.897 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.900+0000 7f7082a00640 1 --2- 172.21.15.90:0/55894493 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f95f681eb0 0x55f95f6884d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:45.897 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.901+0000 7f7082000640 1 --2- 172.21.15.90:0/55894493 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f95f681eb0 0x55f95f6884d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:45.897 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.901+0000 7f7082000640 1 -- 172.21.15.90:0/55894493 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f95f68b3b0 msgr2=0x55f95f688f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:44:45.897 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.901+0000 7f7082000640 1 --2- 172.21.15.90:0/55894493 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f95f68b3b0 0x55f95f688f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:45.897 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.901+0000 7f7082000640 1 -- 172.21.15.90:0/55894493 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f95f5b1e80 con 0x55f95f6827d0 2024-04-02T14:44:45.898 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.902+0000 7f7082000640 1 --2- 172.21.15.90:0/55894493 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f95f6827d0 0x55f95f688a10 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f7070036040 tx=0x7f70700049f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9cc47fcbde34c151 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:44:45.898 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.902+0000 7f706cc00640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:44:45.899 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.903+0000 7f706cc00640 1 -- 172.21.15.90:0/55894493 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7070045040 con 0x55f95f6827d0 2024-04-02T14:44:45.899 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.903+0000 7f706cc00640 1 -- 172.21.15.90:0/55894493 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8550+0+0 (secure 0 0 0) 0x7f70700050e0 con 0x55f95f6827d0 2024-04-02T14:44:45.899 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.903+0000 7f708c684740 1 -- 172.21.15.90:0/55894493 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f707002c650 con 0x55f95f6827d0 2024-04-02T14:44:45.899 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.903+0000 7f706cc00640 1 -- 172.21.15.90:0/55894493 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f707005a050 con 0x55f95f6827d0 2024-04-02T14:44:45.899 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.903+0000 7f708c684740 10 client.28514 fetch_fsmap learned FSMap version 27 2024-04-02T14:44:45.899 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.903+0000 7f708c684740 1 -- 172.21.15.90:0/55894493 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55f95f2fb210 con 0x55f95f6827d0 2024-04-02T14:44:45.899 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.903+0000 7f706cc00640 1 -- 172.21.15.90:0/55894493 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f707002c650 con 0x55f95f6827d0 2024-04-02T14:44:45.901 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.904+0000 7f706cc00640 1 -- 172.21.15.90:0/55894493 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f707004abd0 con 0x55f95f6827d0 2024-04-02T14:44:45.901 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.904+0000 7f708c684740 10 client.28514 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:44:45.901 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.904+0000 7f708c684740 10 client.28514 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-02T14:44:45.901 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.904+0000 7f708c684740 20 client.28514 populate_metadata read hostname 'smithi090' 2024-04-02T14:44:45.901 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.904+0000 7f708c684740 1 --2- 172.21.15.90:0/55894493 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55f95f69b650 0x55f95f69da40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:45.901 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.904+0000 7f708c684740 4 client.28514 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:44:45.901 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.904+0000 7f708c684740 1 -- 172.21.15.90:0/55894493 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f95f596cd0 con 0x55f95f69b650 2024-04-02T14:44:45.901 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.904+0000 7f7082a00640 1 --2- 172.21.15.90:0/55894493 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55f95f69b650 0x55f95f69da40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:45.901 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.905+0000 7f7082a00640 1 --2- 172.21.15.90:0/55894493 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55f95f69b650 0x55f95f69da40 crc :-1 s=READY pgs=262 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dc8e7636a7afc3a0 server_cookie=ee44bac38b73cd8d in_seq=0 out_seq=0 2024-04-02T14:44:45.901 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.905+0000 7f706cc00640 10 client.28514 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:44:45.902 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.905+0000 7f706cc00640 1 -- 172.21.15.90:0/55894493 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f95f596cd0 con 0x55f95f69b650 2024-04-02T14:44:45.902 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.905+0000 7f706cc00640 10 client.28514 handle_command_reply: tid=0 2024-04-02T14:44:45.902 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.906+0000 7f708c684740 1 client.28514 shutdown 2024-04-02T14:44:45.902 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.906+0000 7f708c684740 1 -- 172.21.15.90:0/55894493 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f95f6827d0 msgr2=0x55f95f688a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:45.903 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.906+0000 7f708c684740 1 --2- 172.21.15.90:0/55894493 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f95f6827d0 0x55f95f688a10 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f7070036040 tx=0x7f70700049f0 comp rx=0 tx=0).stop 2024-04-02T14:44:45.903 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.906+0000 7f708c684740 1 -- 172.21.15.90:0/55894493 shutdown_connections 2024-04-02T14:44:45.903 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.906+0000 7f708c684740 1 --2- 172.21.15.90:0/55894493 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55f95f69b650 0x55f95f69da40 crc :-1 s=READY pgs=262 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:45.903 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.907+0000 7f708c684740 1 --2- 172.21.15.90:0/55894493 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f95f68b3b0 0x55f95f688f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:45.903 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.907+0000 7f708c684740 1 --2- 172.21.15.90:0/55894493 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f95f6827d0 0x55f95f688a10 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:45.903 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.907+0000 7f708c684740 1 --2- 172.21.15.90:0/55894493 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f95f681eb0 0x55f95f6884d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:45.903 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.907+0000 7f708c684740 1 -- 172.21.15.90:0/55894493 >> 172.21.15.90:0/55894493 conn(0x55f95f67c5e0 msgr2=0x55f95f67e330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:44:45.903 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.907+0000 7f706cc00640 0 client.28514 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:44:45.905 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.910+0000 7f708c684740 1 -- 172.21.15.90:0/55894493 shutdown_connections 2024-04-02T14:44:45.906 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.911+0000 7f708c684740 1 -- 172.21.15.90:0/55894493 wait complete. 2024-04-02T14:44:45.907 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.911+0000 7f708c684740 20 client.28514 trim_cache size 0 max 16384 2024-04-02T14:44:45.931 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:44:45 smithi177 ceph-mon[36431]: pgmap v305: 97 pgs: 97 active+clean; 4.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 25 MiB/s wr, 1.96k op/s 2024-04-02T14:44:45.933 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:44:45 smithi090 ceph-mon[31674]: pgmap v305: 97 pgs: 97 active+clean; 4.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 25 MiB/s wr, 1.96k op/s 2024-04-02T14:44:45.949 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.953+0000 7f708c684740 1 Processor -- start 2024-04-02T14:44:45.949 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.953+0000 7f708c684740 1 -- start start 2024-04-02T14:44:45.950 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.954+0000 7f708c684740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f95f681eb0 0x55f95f68ac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:45.950 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.954+0000 7f708c684740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f95f6827d0 0x55f95f697000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:45.950 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.954+0000 7f708c684740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f95f68b3b0 0x55f95f6a30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:45.950 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.954+0000 7f708c684740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f70841bc510 con 0x55f95f68b3b0 2024-04-02T14:44:45.950 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.954+0000 7f708c684740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f70841be050 con 0x55f95f6827d0 2024-04-02T14:44:45.951 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.954+0000 7f708c684740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f70841bcbe0 con 0x55f95f681eb0 2024-04-02T14:44:45.951 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.955+0000 7f7083400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f95f68b3b0 0x55f95f6a30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:45.951 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.955+0000 7f7082000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f95f6827d0 0x55f95f697000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:45.951 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.955+0000 7f7082000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f95f6827d0 0x55f95f697000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:43752/0 (socket says 172.21.15.90:43752) 2024-04-02T14:44:45.951 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.955+0000 7f7083400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f95f68b3b0 0x55f95f6a30a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:37172/0 (socket says 172.21.15.90:37172) 2024-04-02T14:44:45.952 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.955+0000 7f7083400640 1 -- 172.21.15.90:0/3991229569 learned_addr learned my addr 172.21.15.90:0/3991229569 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:44:45.952 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.956+0000 7f7082000640 1 -- 172.21.15.90:0/3991229569 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f95f681eb0 msgr2=0x55f95f68ac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:44:45.953 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.956+0000 7f7082000640 1 --2- 172.21.15.90:0/3991229569 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f95f681eb0 0x55f95f68ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:45.953 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.956+0000 7f7082000640 1 -- 172.21.15.90:0/3991229569 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f95f68b3b0 msgr2=0x55f95f6a30a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:44:45.953 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.956+0000 7f7082000640 1 --2- 172.21.15.90:0/3991229569 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f95f68b3b0 0x55f95f6a30a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:45.953 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.956+0000 7f7082000640 1 -- 172.21.15.90:0/3991229569 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f95f697f80 con 0x55f95f6827d0 2024-04-02T14:44:45.953 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.956+0000 7f7083400640 1 --2- 172.21.15.90:0/3991229569 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f95f68b3b0 0x55f95f6a30a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:44:45.953 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.957+0000 7f7082000640 1 --2- 172.21.15.90:0/3991229569 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f95f6827d0 0x55f95f697000 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f7070005ec0 tx=0x7f707002cdb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e77f25037ab53d1b server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:44:45.954 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.957+0000 7f706f400640 1 -- 172.21.15.90:0/3991229569 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7070045030 con 0x55f95f6827d0 2024-04-02T14:44:45.954 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.957+0000 7f706f400640 1 -- 172.21.15.90:0/3991229569 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7070009520 con 0x55f95f6827d0 2024-04-02T14:44:45.954 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.957+0000 7f706f400640 1 -- 172.21.15.90:0/3991229569 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7070005380 con 0x55f95f6827d0 2024-04-02T14:44:45.957 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.959+0000 7f708c684740 1 -- 172.21.15.90:0/3991229569 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f95f6827d0 msgr2=0x55f95f697000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:45.957 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.959+0000 7f708c684740 1 --2- 172.21.15.90:0/3991229569 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f95f6827d0 0x55f95f697000 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f7070005ec0 tx=0x7f707002cdb0 comp rx=0 tx=0).stop 2024-04-02T14:44:45.957 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.959+0000 7f708c684740 1 -- 172.21.15.90:0/3991229569 shutdown_connections 2024-04-02T14:44:45.957 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.959+0000 7f708c684740 1 --2- 172.21.15.90:0/3991229569 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f95f68b3b0 0x55f95f6a30a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:45.957 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.959+0000 7f708c684740 1 --2- 172.21.15.90:0/3991229569 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f95f6827d0 0x55f95f697000 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:45.957 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.959+0000 7f708c684740 1 --2- 172.21.15.90:0/3991229569 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f95f681eb0 0x55f95f68ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:45.957 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.959+0000 7f708c684740 1 -- 172.21.15.90:0/3991229569 >> 172.21.15.90:0/3991229569 conn(0x55f95f67c5e0 msgr2=0x55f95f67c9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:44:45.957 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.960+0000 7f708c684740 1 -- 172.21.15.90:0/3991229569 shutdown_connections 2024-04-02T14:44:45.958 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.962+0000 7f708c684740 1 -- 172.21.15.90:0/3991229569 wait complete. 2024-04-02T14:44:45.958 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.962+0000 7f708c684740 1 -- start start 2024-04-02T14:44:45.958 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.963+0000 7f708c684740 1 Processor -- start 2024-04-02T14:44:45.960 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.964+0000 7f708c684740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f95f681eb0 0x55f95f693780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:45.960 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.964+0000 7f708c684740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f95f6827d0 0x55f95f685000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:45.960 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.964+0000 7f7082a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f95f681eb0 0x55f95f693780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:45.960 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.964+0000 7f708c684740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f95f68b3b0 0x55f95f685540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:45.960 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.964+0000 7f708c684740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f70841bd980 con 0x55f95f6827d0 2024-04-02T14:44:45.961 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.964+0000 7f7082a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f95f681eb0 0x55f95f693780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:54542/0 (socket says 172.21.15.90:54542) 2024-04-02T14:44:45.961 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.964+0000 7f708c684740 1 -- 172.21.15.90:0/2993046800 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f70841bd2b0 con 0x55f95f68b3b0 2024-04-02T14:44:45.961 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.964+0000 7f7082a00640 1 -- 172.21.15.90:0/2993046800 learned_addr learned my addr 172.21.15.90:0/2993046800 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:44:45.961 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.964+0000 7f708c684740 1 -- 172.21.15.90:0/2993046800 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f70841ba510 con 0x55f95f681eb0 2024-04-02T14:44:45.961 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.965+0000 7f708c684740 10 client.0 _ll_register_callbacks cb 0x7f707002cc80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:44:45.961 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.965+0000 7f7083400640 1 --2- 172.21.15.90:0/2993046800 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f95f68b3b0 0x55f95f685540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:45.961 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.965+0000 7f7082a00640 1 -- 172.21.15.90:0/2993046800 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f95f68b3b0 msgr2=0x55f95f685540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:44:45.962 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.966+0000 7f7082a00640 1 --2- 172.21.15.90:0/2993046800 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f95f68b3b0 0x55f95f685540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:45.962 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.966+0000 7f7082a00640 1 -- 172.21.15.90:0/2993046800 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f95f6827d0 msgr2=0x55f95f685000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:44:45.962 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.966+0000 7f7082a00640 1 --2- 172.21.15.90:0/2993046800 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f95f6827d0 0x55f95f685000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:45.962 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.966+0000 7f7082a00640 1 -- 172.21.15.90:0/2993046800 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f95f3c7620 con 0x55f95f681eb0 2024-04-02T14:44:45.963 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.967+0000 7f7082a00640 1 --2- 172.21.15.90:0/2993046800 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f95f681eb0 0x55f95f693780 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f707400c6a0 tx=0x7f707409c9c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a5ae706127f0d62d server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:44:45.963 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.967+0000 7f706cc00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:44:45.966 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.970+0000 7f706cc00640 1 -- 172.21.15.90:0/2993046800 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70740a0e70 con 0x55f95f681eb0 2024-04-02T14:44:45.966 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.970+0000 7f706cc00640 1 -- 172.21.15.90:0/2993046800 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8550+0+0 (secure 0 0 0) 0x7f707409ab60 con 0x55f95f681eb0 2024-04-02T14:44:45.966 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.970+0000 7f708c684740 1 -- 172.21.15.90:0/2993046800 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7070009520 con 0x55f95f681eb0 2024-04-02T14:44:45.967 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.971+0000 7f706cc00640 1 -- 172.21.15.90:0/2993046800 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f70740a0e70 con 0x55f95f681eb0 2024-04-02T14:44:45.967 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.971+0000 7f708c684740 10 client.28339 fetch_fsmap learned FSMap version 27 2024-04-02T14:44:45.967 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.971+0000 7f708c684740 1 -- 172.21.15.90:0/2993046800 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55f95f701690 con 0x55f95f681eb0 2024-04-02T14:44:45.968 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.972+0000 7f706cc00640 1 -- 172.21.15.90:0/2993046800 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f70740b5070 con 0x55f95f681eb0 2024-04-02T14:44:45.969 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.973+0000 7f706cc00640 1 -- 172.21.15.90:0/2993046800 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f70740a52a0 con 0x55f95f681eb0 2024-04-02T14:44:45.969 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.973+0000 7f708c684740 10 client.28339 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:44:45.969 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.973+0000 7f708c684740 10 client.28339 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-02T14:44:45.969 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.973+0000 7f708c684740 20 client.28339 populate_metadata read hostname 'smithi090' 2024-04-02T14:44:45.969 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.973+0000 7f708c684740 1 --2- 172.21.15.90:0/2993046800 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55f95f701e10 0x55f95f704230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:45.970 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.973+0000 7f708c684740 4 client.28339 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-02T14:44:45.970 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.973+0000 7f708c684740 1 -- 172.21.15.90:0/2993046800 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7f7070009520 con 0x55f95f701e10 2024-04-02T14:44:45.970 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.974+0000 7f7082000640 1 --2- 172.21.15.90:0/2993046800 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55f95f701e10 0x55f95f704230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:45.970 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.975+0000 7f7082000640 1 --2- 172.21.15.90:0/2993046800 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55f95f701e10 0x55f95f704230 crc :-1 s=READY pgs=263 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b002fcc5e83924f8 server_cookie=43d698336beb63ec in_seq=0 out_seq=0 2024-04-02T14:44:45.971 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:45.975+0000 7f706cc00640 10 client.28339 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:44:46.014 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:46.017+0000 7f706cc00640 1 -- 172.21.15.90:0/2993046800 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7f7070009520 con 0x55f95f701e10 2024-04-02T14:44:46.014 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:46.017+0000 7f706cc00640 10 client.28339 handle_command_reply: tid=0 2024-04-02T14:44:46.014 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:46.017+0000 7f708c684740 1 client.28339 shutdown 2024-04-02T14:44:46.014 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:46.018+0000 7f708c684740 1 -- 172.21.15.90:0/2993046800 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f95f681eb0 msgr2=0x55f95f693780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:46.015 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:46.018+0000 7f708c684740 1 --2- 172.21.15.90:0/2993046800 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f95f681eb0 0x55f95f693780 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f707400c6a0 tx=0x7f707409c9c0 comp rx=0 tx=0).stop 2024-04-02T14:44:46.016 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:46.019+0000 7f708c684740 1 -- 172.21.15.90:0/2993046800 shutdown_connections 2024-04-02T14:44:46.016 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:46.019+0000 7f708c684740 1 --2- 172.21.15.90:0/2993046800 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55f95f701e10 0x55f95f704230 crc :-1 s=READY pgs=263 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:46.016 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:46.019+0000 7f708c684740 1 --2- 172.21.15.90:0/2993046800 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f95f68b3b0 0x55f95f685540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:46.016 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:46.019+0000 7f708c684740 1 --2- 172.21.15.90:0/2993046800 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f95f6827d0 0x55f95f685000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:46.017 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:46.019+0000 7f708c684740 1 --2- 172.21.15.90:0/2993046800 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f95f681eb0 0x55f95f693780 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:46.017 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:46.019+0000 7f708c684740 1 -- 172.21.15.90:0/2993046800 >> 172.21.15.90:0/2993046800 conn(0x55f95f67c5e0 msgr2=0x55f95f688490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:44:46.017 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:46.019+0000 7f706cc00640 0 client.28339 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:44:46.018 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:{ 2024-04-02T14:44:46.018 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "message": "", 2024-04-02T14:44:46.018 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "return_code": 0 2024-04-02T14:44:46.018 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:} 2024-04-02T14:44:46.020 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:46.020+0000 7f708c684740 1 -- 172.21.15.90:0/2993046800 shutdown_connections 2024-04-02T14:44:46.021 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:46.020+0000 7f708c684740 1 -- 172.21.15.90:0/2993046800 wait complete. 2024-04-02T14:44:46.021 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:46.020+0000 7f708c684740 20 client.28339 trim_cache size 0 max 16384 2024-04-02T14:44:46.021 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:46.021+0000 7f7089a00640 1 -- 172.21.15.90:0/3453519199 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f70680a3fa0 msgr2=0x7f70680a6460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:46.021 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:46.021+0000 7f7089a00640 1 --2- 172.21.15.90:0/3453519199 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f70680a3fa0 0x7f70680a6460 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f708410a5b0 tx=0x7f707c009500 comp rx=0 tx=0).stop 2024-04-02T14:44:46.021 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:46.021+0000 7f7089a00640 1 -- 172.21.15.90:0/3453519199 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7084103f40 msgr2=0x7f708410fd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:46.022 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:46.021+0000 7f7089a00640 1 --2- 172.21.15.90:0/3453519199 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7084103f40 0x7f708410fd10 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f707400e9f0 tx=0x7f707400edf0 comp rx=0 tx=0).stop 2024-04-02T14:44:46.022 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:46.022+0000 7f7089a00640 1 -- 172.21.15.90:0/3453519199 shutdown_connections 2024-04-02T14:44:46.022 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:46.022+0000 7f7089a00640 1 --2- 172.21.15.90:0/3453519199 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f70680a3fa0 0x7f70680a6460 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:46.022 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:46.022+0000 7f7089a00640 1 --2- 172.21.15.90:0/3453519199 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f70841b0dc0 0x7f70841b31b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:46.022 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:46.022+0000 7f7089a00640 1 --2- 172.21.15.90:0/3453519199 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7084109010 0x7f7084110250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:46.023 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:46.022+0000 7f7089a00640 1 --2- 172.21.15.90:0/3453519199 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7084103f40 0x7f708410fd10 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:46.023 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:46.022+0000 7f7089a00640 1 -- 172.21.15.90:0/3453519199 >> 172.21.15.90:0/3453519199 conn(0x7f70840ff7c0 msgr2=0x7f7084101bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:44:46.023 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:46.022+0000 7f7089a00640 1 -- 172.21.15.90:0/3453519199 shutdown_connections 2024-04-02T14:44:46.023 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:46.022+0000 7f7089a00640 1 -- 172.21.15.90:0/3453519199 wait complete. 2024-04-02T14:44:46.105 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:44:45 smithi092 ceph-mon[36497]: pgmap v305: 97 pgs: 97 active+clean; 4.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 25 MiB/s wr, 1.96k op/s 2024-04-02T14:44:46.826 INFO:teuthology.orchestra.run.smithi090.stderr:progress: 555.0 s, 2284.2 tps, lat 44.125 ms stddev 54.913 2024-04-02T14:44:47.931 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:44:47 smithi177 ceph-mon[36431]: pgmap v306: 97 pgs: 97 active+clean; 4.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 22 MiB/s wr, 1.68k op/s 2024-04-02T14:44:47.933 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:44:47 smithi090 ceph-mon[31674]: pgmap v306: 97 pgs: 97 active+clean; 4.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 22 MiB/s wr, 1.68k op/s 2024-04-02T14:44:48.105 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:44:47 smithi092 ceph-mon[36497]: pgmap v306: 97 pgs: 97 active+clean; 4.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 22 MiB/s wr, 1.68k op/s 2024-04-02T14:44:49.931 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:44:49 smithi177 ceph-mon[36431]: pgmap v307: 97 pgs: 97 active+clean; 4.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 24 MiB/s wr, 1.82k op/s 2024-04-02T14:44:49.936 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:44:49 smithi090 ceph-mon[31674]: pgmap v307: 97 pgs: 97 active+clean; 4.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 24 MiB/s wr, 1.82k op/s 2024-04-02T14:44:50.105 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:44:49 smithi092 ceph-mon[36497]: pgmap v307: 97 pgs: 97 active+clean; 4.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 24 MiB/s wr, 1.82k op/s 2024-04-02T14:44:51.827 INFO:teuthology.orchestra.run.smithi090.stderr:progress: 560.0 s, 2502.2 tps, lat 39.550 ms stddev 48.315 2024-04-02T14:44:51.931 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:44:51 smithi177 ceph-mon[36431]: pgmap v308: 97 pgs: 97 active+clean; 4.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 17 MiB/s wr, 1.35k op/s 2024-04-02T14:44:51.933 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:44:51 smithi090 ceph-mon[31674]: pgmap v308: 97 pgs: 97 active+clean; 4.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 17 MiB/s wr, 1.35k op/s 2024-04-02T14:44:52.105 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:44:51 smithi092 ceph-mon[36497]: pgmap v308: 97 pgs: 97 active+clean; 4.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 17 MiB/s wr, 1.35k op/s 2024-04-02T14:44:53.931 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:44:53 smithi177 ceph-mon[36431]: pgmap v309: 97 pgs: 97 active+clean; 4.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 6.2 MiB/s wr, 583 op/s 2024-04-02T14:44:53.933 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:44:53 smithi090 ceph-mon[31674]: pgmap v309: 97 pgs: 97 active+clean; 4.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 6.2 MiB/s wr, 583 op/s 2024-04-02T14:44:54.105 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:44:53 smithi092 ceph-mon[36497]: pgmap v309: 97 pgs: 97 active+clean; 4.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 6.2 MiB/s wr, 583 op/s 2024-04-02T14:44:54.323 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-02T14:44:54.323 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-02T14:44:54.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.619+0000 7feb80400640 1 -- 172.21.15.90:0/4173818980 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7feb7806a150 msgr2=0x7feb7806a5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:54.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.619+0000 7feb80400640 1 --2- 172.21.15.90:0/4173818980 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7feb7806a150 0x7feb7806a5b0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7feb64009930 tx=0x7feb6402bc30 comp rx=0 tx=0).stop 2024-04-02T14:44:54.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.619+0000 7feb80400640 1 -- 172.21.15.90:0/4173818980 shutdown_connections 2024-04-02T14:44:54.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.619+0000 7feb80400640 1 --2- 172.21.15.90:0/4173818980 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7feb7806aaf0 0x7feb78117180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:54.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.619+0000 7feb80400640 1 --2- 172.21.15.90:0/4173818980 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7feb7806a150 0x7feb7806a5b0 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:54.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.619+0000 7feb80400640 1 --2- 172.21.15.90:0/4173818980 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7feb78110400 0x7feb781107e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:54.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.619+0000 7feb80400640 1 -- 172.21.15.90:0/4173818980 >> 172.21.15.90:0/4173818980 conn(0x7feb7806dd50 msgr2=0x7feb7806e160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:44:54.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.620+0000 7feb80400640 1 -- 172.21.15.90:0/4173818980 shutdown_connections 2024-04-02T14:44:54.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.620+0000 7feb80400640 1 -- 172.21.15.90:0/4173818980 wait complete. 2024-04-02T14:44:54.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.622+0000 7feb80400640 1 Processor -- start 2024-04-02T14:44:54.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.624+0000 7feb80400640 1 -- start start 2024-04-02T14:44:54.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.625+0000 7feb80400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7feb7806a150 0x7feb781a2f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:54.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.625+0000 7feb80400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7feb7806aaf0 0x7feb781a3440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:54.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.625+0000 7feb80400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7feb78110400 0x7feb781a77c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:54.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.625+0000 7feb7da00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7feb7806a150 0x7feb781a2f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:54.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.625+0000 7feb7da00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7feb7806a150 0x7feb781a2f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:40658/0 (socket says 172.21.15.90:40658) 2024-04-02T14:44:54.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.625+0000 7feb80400640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7feb7810b870 con 0x7feb7806aaf0 2024-04-02T14:44:54.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.625+0000 7feb80400640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7feb7810b6f0 con 0x7feb7806a150 2024-04-02T14:44:54.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.625+0000 7feb80400640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7feb7810b9f0 con 0x7feb78110400 2024-04-02T14:44:54.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.625+0000 7feb7da00640 1 -- 172.21.15.90:0/213419243 learned_addr learned my addr 172.21.15.90:0/213419243 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:44:54.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.626+0000 7feb7e400640 1 --2- 172.21.15.90:0/213419243 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7feb78110400 0x7feb781a77c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:54.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.626+0000 7feb7da00640 1 -- 172.21.15.90:0/213419243 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7feb78110400 msgr2=0x7feb781a77c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:54.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.626+0000 7feb7da00640 1 --2- 172.21.15.90:0/213419243 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7feb78110400 0x7feb781a77c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:54.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.626+0000 7feb7da00640 1 -- 172.21.15.90:0/213419243 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7feb7806aaf0 msgr2=0x7feb781a3440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:54.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.626+0000 7feb7da00640 1 --2- 172.21.15.90:0/213419243 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7feb7806aaf0 0x7feb781a3440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:54.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.626+0000 7feb7da00640 1 -- 172.21.15.90:0/213419243 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb781a7ef0 con 0x7feb7806a150 2024-04-02T14:44:54.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.627+0000 7feb7da00640 1 --2- 172.21.15.90:0/213419243 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7feb7806a150 0x7feb781a2f00 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7feb6c00b4a0 tx=0x7feb6c00b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:44:54.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.627+0000 7feb76000640 1 -- 172.21.15.90:0/213419243 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb6c014070 con 0x7feb7806a150 2024-04-02T14:44:54.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.627+0000 7feb76000640 1 -- 172.21.15.90:0/213419243 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feb6c010af0 con 0x7feb7806a150 2024-04-02T14:44:54.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.627+0000 7feb76000640 1 -- 172.21.15.90:0/213419243 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb6c017d10 con 0x7feb7806a150 2024-04-02T14:44:54.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.628+0000 7feb80400640 1 -- 172.21.15.90:0/213419243 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feb781a81e0 con 0x7feb7806a150 2024-04-02T14:44:54.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.628+0000 7feb80400640 1 -- 172.21.15.90:0/213419243 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feb781afa70 con 0x7feb7806a150 2024-04-02T14:44:54.630 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.634+0000 7feb83169740 1 Processor -- start 2024-04-02T14:44:54.630 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.634+0000 7feb83169740 1 -- start start 2024-04-02T14:44:54.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.635+0000 7feb83169740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c910886fc0 0x55c9108873a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:54.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.635+0000 7feb83169740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c9108878e0 0x55c91088ff80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:54.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.635+0000 7feb83169740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c9108904c0 0x55c9108928b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:54.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.635+0000 7feb83169740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55c9107e2770 con 0x55c910886fc0 2024-04-02T14:44:54.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.635+0000 7feb83169740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55c91078a700 con 0x55c9108878e0 2024-04-02T14:44:54.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.635+0000 7feb83169740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55c9107de720 con 0x55c9108904c0 2024-04-02T14:44:54.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.639+0000 7feb7d000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c9108904c0 0x55c9108928b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:54.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.639+0000 7feb7d000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c9108904c0 0x55c9108928b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:60282/0 (socket says 172.21.15.90:60282) 2024-04-02T14:44:54.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.639+0000 7feb7d000640 1 -- 172.21.15.90:0/3701639681 learned_addr learned my addr 172.21.15.90:0/3701639681 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:44:54.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.639+0000 7feb76000640 1 -- 172.21.15.90:0/213419243 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7feb6c017070 con 0x7feb7806a150 2024-04-02T14:44:54.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.639+0000 7feb7e400640 1 --2- 172.21.15.90:0/3701639681 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c910886fc0 0x55c9108873a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:54.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.640+0000 7feb7d000640 1 -- 172.21.15.90:0/3701639681 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c9108878e0 msgr2=0x55c91088ff80 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-02T14:44:54.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.640+0000 7feb7d000640 1 --2- 172.21.15.90:0/3701639681 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c9108878e0 0x55c91088ff80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:54.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.640+0000 7feb7d000640 1 -- 172.21.15.90:0/3701639681 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c910886fc0 msgr2=0x55c9108873a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:44:54.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.640+0000 7feb7d000640 1 --2- 172.21.15.90:0/3701639681 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c910886fc0 0x55c9108873a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:54.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.640+0000 7feb7d000640 1 -- 172.21.15.90:0/3701639681 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c9105c07d0 con 0x55c9108904c0 2024-04-02T14:44:54.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.640+0000 7feb7d000640 1 --2- 172.21.15.90:0/3701639681 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c9108904c0 0x55c9108928b0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7feb6402c240 tx=0x7feb64009610 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6999ddac42eacc48 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:44:54.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.641+0000 7feb7e400640 1 --2- 172.21.15.90:0/3701639681 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c910886fc0 0x55c9108873a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:44:54.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.642+0000 7feb76000640 1 --2- 172.21.15.90:0/213419243 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7feb600a3e20 0x7feb600a62e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:54.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.641+0000 7feb6aa00640 1 -- 172.21.15.90:0/3701639681 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb64041020 con 0x55c9108904c0 2024-04-02T14:44:54.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.642+0000 7feb76000640 1 -- 172.21.15.90:0/213419243 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8550+0+0 (secure 0 0 0) 0x7feb6c0cfb20 con 0x7feb7806a150 2024-04-02T14:44:54.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.642+0000 7feb6aa00640 1 -- 172.21.15.90:0/3701639681 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feb6402c880 con 0x55c9108904c0 2024-04-02T14:44:54.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.642+0000 7feb6aa00640 1 -- 172.21.15.90:0/3701639681 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb64005600 con 0x55c9108904c0 2024-04-02T14:44:54.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.643+0000 7feb83169740 1 -- 172.21.15.90:0/3701639681 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c9108904c0 msgr2=0x55c9108928b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:54.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.643+0000 7feb83169740 1 --2- 172.21.15.90:0/3701639681 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c9108904c0 0x55c9108928b0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7feb6402c240 tx=0x7feb64009610 comp rx=0 tx=0).stop 2024-04-02T14:44:54.640 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.643+0000 7feb83169740 1 -- 172.21.15.90:0/3701639681 shutdown_connections 2024-04-02T14:44:54.640 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.643+0000 7feb83169740 1 --2- 172.21.15.90:0/3701639681 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c9108904c0 0x55c9108928b0 secure :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7feb6402c240 tx=0x7feb64009610 comp rx=0 tx=0).stop 2024-04-02T14:44:54.640 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.643+0000 7feb83169740 1 --2- 172.21.15.90:0/3701639681 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c9108878e0 0x55c91088ff80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:54.640 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.643+0000 7feb83169740 1 --2- 172.21.15.90:0/3701639681 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c910886fc0 0x55c9108873a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:54.640 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.643+0000 7feb83169740 1 -- 172.21.15.90:0/3701639681 >> 172.21.15.90:0/3701639681 conn(0x55c9108816f0 msgr2=0x55c910881ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:44:54.640 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.644+0000 7feb7e400640 1 --2- 172.21.15.90:0/213419243 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7feb600a3e20 0x7feb600a62e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:54.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.644+0000 7feb83169740 1 -- 172.21.15.90:0/3701639681 shutdown_connections 2024-04-02T14:44:54.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.645+0000 7feb83169740 1 -- 172.21.15.90:0/3701639681 wait complete. 2024-04-02T14:44:54.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.647+0000 7feb83169740 1 -- start start 2024-04-02T14:44:54.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.647+0000 7feb83169740 1 Processor -- start 2024-04-02T14:44:54.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.648+0000 7feb7e400640 1 --2- 172.21.15.90:0/213419243 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7feb600a3e20 0x7feb600a62e0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7feb70005fd0 tx=0x7feb70009500 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:44:54.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.648+0000 7feb83169740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c910886fc0 0x55c91088d620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:54.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.648+0000 7feb83169740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c9108878e0 0x55c91088db60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:54.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.648+0000 7feb83169740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c91088e0a0 0x55c91089c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:54.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.648+0000 7feb83169740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55c910893250 con 0x55c91088e0a0 2024-04-02T14:44:54.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.648+0000 7feb83169740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55c91079dac0 con 0x55c910886fc0 2024-04-02T14:44:54.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.648+0000 7feb83169740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55c9108933d0 con 0x55c9108878e0 2024-04-02T14:44:54.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.648+0000 7feb83169740 10 client.0 _ll_register_callbacks cb 0x55c9107a0180 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:44:54.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.648+0000 7feb7da00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c910886fc0 0x55c91088d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:54.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.648+0000 7feb7da00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c910886fc0 0x55c91088d620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:40678/0 (socket says 172.21.15.90:40678) 2024-04-02T14:44:54.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.648+0000 7feb7da00640 1 -- 172.21.15.90:0/1599147645 learned_addr learned my addr 172.21.15.90:0/1599147645 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:44:54.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.649+0000 7feb7d000640 1 --2- 172.21.15.90:0/1599147645 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c9108878e0 0x55c91088db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:54.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.649+0000 7feb7da00640 1 -- 172.21.15.90:0/1599147645 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c9108878e0 msgr2=0x55c91088db60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:44:54.646 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.649+0000 7feb7da00640 1 --2- 172.21.15.90:0/1599147645 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c9108878e0 0x55c91088db60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:54.646 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.649+0000 7feb7da00640 1 -- 172.21.15.90:0/1599147645 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c91088e0a0 msgr2=0x55c91089c7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:44:54.646 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.649+0000 7feb7da00640 1 --2- 172.21.15.90:0/1599147645 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c91088e0a0 0x55c91089c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:54.646 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.649+0000 7feb7da00640 1 -- 172.21.15.90:0/1599147645 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c9104f63f0 con 0x55c910886fc0 2024-04-02T14:44:54.646 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.649+0000 7feb7d000640 1 --2- 172.21.15.90:0/1599147645 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c9108878e0 0x55c91088db60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:44:54.646 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.649+0000 7feb7da00640 1 --2- 172.21.15.90:0/1599147645 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c910886fc0 0x55c91088d620 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7feb6c00b470 tx=0x7feb6c09b780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=318e66a4b1b08f2e server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:44:54.646 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.649+0000 7feb5f400640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:44:54.646 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.650+0000 7feb5f400640 1 -- 172.21.15.90:0/1599147645 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb6c01e070 con 0x55c910886fc0 2024-04-02T14:44:54.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.650+0000 7feb5f400640 1 -- 172.21.15.90:0/1599147645 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8550+0+0 (secure 0 0 0) 0x7feb6c0981c0 con 0x55c910886fc0 2024-04-02T14:44:54.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.650+0000 7feb83169740 1 -- 172.21.15.90:0/1599147645 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7feb6402c880 con 0x55c910886fc0 2024-04-02T14:44:54.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.653+0000 7feb5f400640 1 -- 172.21.15.90:0/1599147645 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7feb6c09ff00 con 0x55c910886fc0 2024-04-02T14:44:54.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.654+0000 7feb83169740 10 client.28529 fetch_fsmap learned FSMap version 27 2024-04-02T14:44:54.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.654+0000 7feb83169740 1 -- 172.21.15.90:0/1599147645 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55c9104b60f0 con 0x55c910886fc0 2024-04-02T14:44:54.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.654+0000 7feb5f400640 1 -- 172.21.15.90:0/1599147645 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feb6402c880 con 0x55c910886fc0 2024-04-02T14:44:54.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.654+0000 7feb5f400640 1 -- 172.21.15.90:0/1599147645 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7feb6c0cd500 con 0x55c910886fc0 2024-04-02T14:44:54.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.655+0000 7feb83169740 10 client.28529 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:44:54.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.655+0000 7feb83169740 10 client.28529 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:44:54.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.655+0000 7feb83169740 20 client.28529 populate_metadata read hostname 'smithi090' 2024-04-02T14:44:54.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.655+0000 7feb83169740 1 --2- 172.21.15.90:0/1599147645 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55c9108a0a40 0x55c9108a2e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:54.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.655+0000 7feb83169740 4 client.28529 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:44:54.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.655+0000 7feb83169740 1 -- 172.21.15.90:0/1599147645 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c910783630 con 0x55c9108a0a40 2024-04-02T14:44:54.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.655+0000 7feb7d000640 1 --2- 172.21.15.90:0/1599147645 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55c9108a0a40 0x55c9108a2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:54.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.657+0000 7feb7d000640 1 --2- 172.21.15.90:0/1599147645 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55c9108a0a40 0x55c9108a2e90 crc :-1 s=READY pgs=264 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1be695c20f49ee59 server_cookie=504feb7b9c9b9d2d in_seq=0 out_seq=0 2024-04-02T14:44:54.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.657+0000 7feb5f400640 10 client.28529 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:44:54.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.658+0000 7feb5f400640 1 -- 172.21.15.90:0/1599147645 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c910783630 con 0x55c9108a0a40 2024-04-02T14:44:54.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.658+0000 7feb5f400640 10 client.28529 handle_command_reply: tid=0 2024-04-02T14:44:54.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.658+0000 7feb83169740 1 client.28529 shutdown 2024-04-02T14:44:54.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.658+0000 7feb83169740 1 -- 172.21.15.90:0/1599147645 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c910886fc0 msgr2=0x55c91088d620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:54.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.658+0000 7feb83169740 1 --2- 172.21.15.90:0/1599147645 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c910886fc0 0x55c91088d620 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7feb6c00b470 tx=0x7feb6c09b780 comp rx=0 tx=0).stop 2024-04-02T14:44:54.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.658+0000 7feb83169740 1 -- 172.21.15.90:0/1599147645 shutdown_connections 2024-04-02T14:44:54.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.658+0000 7feb83169740 1 --2- 172.21.15.90:0/1599147645 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55c9108a0a40 0x55c9108a2e90 crc :-1 s=READY pgs=264 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:54.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.658+0000 7feb83169740 1 --2- 172.21.15.90:0/1599147645 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c91088e0a0 0x55c91089c7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:54.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.658+0000 7feb83169740 1 --2- 172.21.15.90:0/1599147645 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c9108878e0 0x55c91088db60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:54.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.658+0000 7feb83169740 1 --2- 172.21.15.90:0/1599147645 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c910886fc0 0x55c91088d620 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:54.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.658+0000 7feb5f400640 0 client.28529 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:44:54.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.658+0000 7feb83169740 1 -- 172.21.15.90:0/1599147645 >> 172.21.15.90:0/1599147645 conn(0x55c9108816f0 msgr2=0x55c910883440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:44:54.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.660+0000 7feb83169740 1 -- 172.21.15.90:0/1599147645 shutdown_connections 2024-04-02T14:44:54.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.660+0000 7feb83169740 1 -- 172.21.15.90:0/1599147645 wait complete. 2024-04-02T14:44:54.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.660+0000 7feb83169740 20 client.28529 trim_cache size 0 max 16384 2024-04-02T14:44:54.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.702+0000 7feb83169740 1 Processor -- start 2024-04-02T14:44:54.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.702+0000 7feb83169740 1 -- start start 2024-04-02T14:44:54.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.703+0000 7feb83169740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c910886fc0 0x55c910877de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:54.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.703+0000 7feb83169740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c9108878e0 0x55c9108a8760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:54.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.703+0000 7feb83169740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c91088e0a0 0x55c9108a8ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:54.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.703+0000 7feb83169740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7feb781b3f80 con 0x55c9108878e0 2024-04-02T14:44:54.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.703+0000 7feb83169740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7feb781b5ac0 con 0x55c910886fc0 2024-04-02T14:44:54.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.703+0000 7feb83169740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7feb781b4650 con 0x55c91088e0a0 2024-04-02T14:44:54.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.703+0000 7feb7da00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c910886fc0 0x55c910877de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:54.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.703+0000 7feb7da00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c910886fc0 0x55c910877de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:40690/0 (socket says 172.21.15.90:40690) 2024-04-02T14:44:54.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.703+0000 7feb7da00640 1 -- 172.21.15.90:0/2156120147 learned_addr learned my addr 172.21.15.90:0/2156120147 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:44:54.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.704+0000 7feb7e400640 1 --2- 172.21.15.90:0/2156120147 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c91088e0a0 0x55c9108a8ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:54.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.704+0000 7feb7d000640 1 --2- 172.21.15.90:0/2156120147 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c9108878e0 0x55c9108a8760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:54.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.704+0000 7feb7e400640 1 -- 172.21.15.90:0/2156120147 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c910886fc0 msgr2=0x55c910877de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:44:54.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.704+0000 7feb7e400640 1 --2- 172.21.15.90:0/2156120147 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c910886fc0 0x55c910877de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:54.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.704+0000 7feb7e400640 1 -- 172.21.15.90:0/2156120147 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c9108878e0 msgr2=0x55c9108a8760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:44:54.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.704+0000 7feb7e400640 1 --2- 172.21.15.90:0/2156120147 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c9108878e0 0x55c9108a8760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:54.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.704+0000 7feb7e400640 1 -- 172.21.15.90:0/2156120147 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c9108733b0 con 0x55c91088e0a0 2024-04-02T14:44:54.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.705+0000 7feb7da00640 1 --2- 172.21.15.90:0/2156120147 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c910886fc0 0x55c910877de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:44:54.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.708+0000 7feb7e400640 1 --2- 172.21.15.90:0/2156120147 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c91088e0a0 0x55c9108a8ca0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7feb700099a0 tx=0x7feb70014070 comp rx=0 tx=0).ready entity=mon.1 client_cookie=56cecd8b6af72c77 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:44:54.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.709+0000 7feb6a000640 1 -- 172.21.15.90:0/2156120147 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb7001f6e0 con 0x55c91088e0a0 2024-04-02T14:44:54.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.709+0000 7feb6a000640 1 -- 172.21.15.90:0/2156120147 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feb7001fd60 con 0x55c91088e0a0 2024-04-02T14:44:54.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.709+0000 7feb6a000640 1 -- 172.21.15.90:0/2156120147 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb7001e3b0 con 0x55c91088e0a0 2024-04-02T14:44:54.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.710+0000 7feb83169740 1 -- 172.21.15.90:0/2156120147 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c91088e0a0 msgr2=0x55c9108a8ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:54.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.710+0000 7feb83169740 1 --2- 172.21.15.90:0/2156120147 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c91088e0a0 0x55c9108a8ca0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7feb700099a0 tx=0x7feb70014070 comp rx=0 tx=0).stop 2024-04-02T14:44:54.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.711+0000 7feb83169740 1 -- 172.21.15.90:0/2156120147 shutdown_connections 2024-04-02T14:44:54.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.711+0000 7feb83169740 1 --2- 172.21.15.90:0/2156120147 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c91088e0a0 0x55c9108a8ca0 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:54.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.711+0000 7feb83169740 1 --2- 172.21.15.90:0/2156120147 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c9108878e0 0x55c9108a8760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:54.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.711+0000 7feb83169740 1 --2- 172.21.15.90:0/2156120147 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c910886fc0 0x55c910877de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:54.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.711+0000 7feb83169740 1 -- 172.21.15.90:0/2156120147 >> 172.21.15.90:0/2156120147 conn(0x55c9108816f0 msgr2=0x55c9108776d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:44:54.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.711+0000 7feb83169740 1 -- 172.21.15.90:0/2156120147 shutdown_connections 2024-04-02T14:44:54.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.711+0000 7feb83169740 1 -- 172.21.15.90:0/2156120147 wait complete. 2024-04-02T14:44:54.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.711+0000 7feb83169740 1 -- start start 2024-04-02T14:44:54.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.712+0000 7feb83169740 1 Processor -- start 2024-04-02T14:44:54.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.716+0000 7feb83169740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c910886fc0 0x55c910898890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:54.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.716+0000 7feb83169740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c9108878e0 0x55c910894c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:54.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.716+0000 7feb83169740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c91088e0a0 0x55c910895150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:54.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.717+0000 7feb83169740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7feb781b53f0 con 0x55c910886fc0 2024-04-02T14:44:54.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.717+0000 7feb83169740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7feb781b4d20 con 0x55c91088e0a0 2024-04-02T14:44:54.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.717+0000 7feb83169740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7feb781b1f80 con 0x55c9108878e0 2024-04-02T14:44:54.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.717+0000 7feb83169740 10 client.0 _ll_register_callbacks cb 0x7feb6c017630 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:44:54.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.717+0000 7feb7d000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c9108878e0 0x55c910894c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:54.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.717+0000 7feb7d000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c9108878e0 0x55c910894c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:60306/0 (socket says 172.21.15.90:60306) 2024-04-02T14:44:54.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.718+0000 7feb7d000640 1 -- 172.21.15.90:0/3981277412 learned_addr learned my addr 172.21.15.90:0/3981277412 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:44:54.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.719+0000 7feb7d000640 1 -- 172.21.15.90:0/3981277412 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c91088e0a0 msgr2=0x55c910895150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:44:54.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.719+0000 7feb7d000640 1 --2- 172.21.15.90:0/3981277412 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c91088e0a0 0x55c910895150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:54.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.719+0000 7feb7d000640 1 -- 172.21.15.90:0/3981277412 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c910886fc0 msgr2=0x55c910898890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:44:54.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.719+0000 7feb7d000640 1 --2- 172.21.15.90:0/3981277412 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c910886fc0 0x55c910898890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:54.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.719+0000 7feb7d000640 1 -- 172.21.15.90:0/3981277412 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c910895710 con 0x55c9108878e0 2024-04-02T14:44:54.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.722+0000 7feb7d000640 1 --2- 172.21.15.90:0/3981277412 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c9108878e0 0x55c910894c10 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x55c910882880 tx=0x7feb64005e50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b6579030841b47bf server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:44:54.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.722+0000 7feb5f400640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:44:54.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.722+0000 7feb5f400640 1 -- 172.21.15.90:0/3981277412 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb64041070 con 0x55c9108878e0 2024-04-02T14:44:54.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.723+0000 7feb5f400640 1 -- 172.21.15.90:0/3981277412 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8550+0+0 (secure 0 0 0) 0x7feb64005440 con 0x55c9108878e0 2024-04-02T14:44:54.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.723+0000 7feb83169740 1 -- 172.21.15.90:0/3981277412 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7feb7001fd60 con 0x55c9108878e0 2024-04-02T14:44:54.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.723+0000 7feb5f400640 1 -- 172.21.15.90:0/3981277412 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7feb64005640 con 0x55c9108878e0 2024-04-02T14:44:54.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.724+0000 7feb83169740 10 client.28357 fetch_fsmap learned FSMap version 27 2024-04-02T14:44:54.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.724+0000 7feb83169740 1 -- 172.21.15.90:0/3981277412 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55c910907df0 con 0x55c9108878e0 2024-04-02T14:44:54.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.725+0000 7feb5f400640 1 -- 172.21.15.90:0/3981277412 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feb7001fd60 con 0x55c9108878e0 2024-04-02T14:44:54.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.725+0000 7feb5f400640 1 -- 172.21.15.90:0/3981277412 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7feb640035f0 con 0x55c9108878e0 2024-04-02T14:44:54.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.725+0000 7feb83169740 10 client.28357 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:44:54.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.725+0000 7feb83169740 10 client.28357 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:44:54.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.725+0000 7feb83169740 20 client.28357 populate_metadata read hostname 'smithi090' 2024-04-02T14:44:54.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.725+0000 7feb83169740 1 --2- 172.21.15.90:0/3981277412 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55c910908130 0x55c91090a570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:54.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.725+0000 7feb83169740 4 client.28357 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-02T14:44:54.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.725+0000 7feb83169740 1 -- 172.21.15.90:0/3981277412 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55c91088def0 con 0x55c910908130 2024-04-02T14:44:54.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.725+0000 7feb7da00640 1 --2- 172.21.15.90:0/3981277412 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55c910908130 0x55c91090a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:54.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.726+0000 7feb7da00640 1 --2- 172.21.15.90:0/3981277412 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55c910908130 0x55c91090a570 crc :-1 s=READY pgs=265 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c763ee1ac5bdc02c server_cookie=a6490aa7f193c452 in_seq=0 out_seq=0 2024-04-02T14:44:54.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.726+0000 7feb5f400640 10 client.28357 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:44:54.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.726+0000 7feb5f400640 1 -- 172.21.15.90:0/3981277412 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55c91088def0 con 0x55c910908130 2024-04-02T14:44:54.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.726+0000 7feb5f400640 10 client.28357 handle_command_reply: tid=0 2024-04-02T14:44:54.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.727+0000 7feb83169740 1 client.28357 shutdown 2024-04-02T14:44:54.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.727+0000 7feb83169740 1 -- 172.21.15.90:0/3981277412 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c9108878e0 msgr2=0x55c910894c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:54.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.727+0000 7feb83169740 1 --2- 172.21.15.90:0/3981277412 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c9108878e0 0x55c910894c10 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x55c910882880 tx=0x7feb64005e50 comp rx=0 tx=0).stop 2024-04-02T14:44:54.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.728+0000 7feb83169740 1 -- 172.21.15.90:0/3981277412 shutdown_connections 2024-04-02T14:44:54.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.728+0000 7feb83169740 1 --2- 172.21.15.90:0/3981277412 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55c910908130 0x55c91090a570 crc :-1 s=READY pgs=265 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:54.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.728+0000 7feb83169740 1 --2- 172.21.15.90:0/3981277412 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c91088e0a0 0x55c910895150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:54.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.728+0000 7feb83169740 1 --2- 172.21.15.90:0/3981277412 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c9108878e0 0x55c910894c10 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:54.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.728+0000 7feb83169740 1 --2- 172.21.15.90:0/3981277412 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c910886fc0 0x55c910898890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:54.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.728+0000 7feb83169740 1 -- 172.21.15.90:0/3981277412 >> 172.21.15.90:0/3981277412 conn(0x55c9108816f0 msgr2=0x55c910877f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:44:54.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.728+0000 7feb5f400640 0 client.28357 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:44:54.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.728+0000 7feb83169740 1 -- 172.21.15.90:0/3981277412 shutdown_connections 2024-04-02T14:44:54.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.728+0000 7feb83169740 1 -- 172.21.15.90:0/3981277412 wait complete. 2024-04-02T14:44:54.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.728+0000 7feb83169740 20 client.28357 trim_cache size 0 max 16384 2024-04-02T14:44:54.725 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-02T14:44:54.725 INFO:teuthology.orchestra.run.smithi090.stdout: "status": "no active scrubs running", 2024-04-02T14:44:54.725 INFO:teuthology.orchestra.run.smithi090.stdout: "scrubs": {} 2024-04-02T14:44:54.725 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-02T14:44:54.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.729+0000 7feb80400640 1 -- 172.21.15.90:0/213419243 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7feb600a3e20 msgr2=0x7feb600a62e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:54.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.729+0000 7feb80400640 1 --2- 172.21.15.90:0/213419243 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7feb600a3e20 0x7feb600a62e0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7feb70005fd0 tx=0x7feb70009500 comp rx=0 tx=0).stop 2024-04-02T14:44:54.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.729+0000 7feb80400640 1 -- 172.21.15.90:0/213419243 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7feb7806a150 msgr2=0x7feb781a2f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:54.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.729+0000 7feb80400640 1 --2- 172.21.15.90:0/213419243 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7feb7806a150 0x7feb781a2f00 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7feb6c00b4a0 tx=0x7feb6c00b960 comp rx=0 tx=0).stop 2024-04-02T14:44:54.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.729+0000 7feb80400640 1 -- 172.21.15.90:0/213419243 shutdown_connections 2024-04-02T14:44:54.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.729+0000 7feb80400640 1 --2- 172.21.15.90:0/213419243 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7feb600a3e20 0x7feb600a62e0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:54.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.729+0000 7feb80400640 1 --2- 172.21.15.90:0/213419243 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7feb78110400 0x7feb781a77c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:54.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.729+0000 7feb80400640 1 --2- 172.21.15.90:0/213419243 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7feb7806aaf0 0x7feb781a3440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:54.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.729+0000 7feb80400640 1 --2- 172.21.15.90:0/213419243 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7feb7806a150 0x7feb781a2f00 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:54.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.729+0000 7feb80400640 1 -- 172.21.15.90:0/213419243 >> 172.21.15.90:0/213419243 conn(0x7feb7806dd50 msgr2=0x7feb78072400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:44:54.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.730+0000 7feb80400640 1 -- 172.21.15.90:0/213419243 shutdown_connections 2024-04-02T14:44:54.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:54.730+0000 7feb80400640 1 -- 172.21.15.90:0/213419243 wait complete. 2024-04-02T14:44:54.757 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-02T14:44:54.757 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T14:44:55.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.050+0000 7f534f800640 1 -- 172.21.15.90:0/1457759931 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5348103f00 msgr2=0x7f53481042e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:55.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.050+0000 7f534f800640 1 --2- 172.21.15.90:0/1457759931 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5348103f00 0x7f53481042e0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f533c009900 tx=0x7f533c02bc40 comp rx=0 tx=0).stop 2024-04-02T14:44:55.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.051+0000 7f534f800640 1 -- 172.21.15.90:0/1457759931 shutdown_connections 2024-04-02T14:44:55.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.051+0000 7f534f800640 1 --2- 172.21.15.90:0/1457759931 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5348108fd0 0x7f5348110ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:55.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.051+0000 7f534f800640 1 --2- 172.21.15.90:0/1457759931 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f53481048b0 0x7f53481088f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:55.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.051+0000 7f534f800640 1 --2- 172.21.15.90:0/1457759931 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5348103f00 0x7f53481042e0 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:55.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.051+0000 7f534f800640 1 -- 172.21.15.90:0/1457759931 >> 172.21.15.90:0/1457759931 conn(0x7f53480ff760 msgr2=0x7f5348101b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:44:55.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.051+0000 7f534f800640 1 -- 172.21.15.90:0/1457759931 shutdown_connections 2024-04-02T14:44:55.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.051+0000 7f534f800640 1 -- 172.21.15.90:0/1457759931 wait complete. 2024-04-02T14:44:55.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.052+0000 7f534f800640 1 Processor -- start 2024-04-02T14:44:55.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.052+0000 7f534f800640 1 -- start start 2024-04-02T14:44:55.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.053+0000 7f534f800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5348103f00 0x7f534819a580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:55.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.053+0000 7f534f800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5348108fd0 0x7f534819aac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:55.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.053+0000 7f534f800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f534819edf0 0x7f534819f2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:55.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.053+0000 7f534f800640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f53481140c0 con 0x7f534819edf0 2024-04-02T14:44:55.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.053+0000 7f534f800640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f5348113f40 con 0x7f5348108fd0 2024-04-02T14:44:55.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.053+0000 7f534f800640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f5348114240 con 0x7f5348103f00 2024-04-02T14:44:55.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.054+0000 7f5347e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5348108fd0 0x7f534819aac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:55.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.054+0000 7f534ce00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5348103f00 0x7f534819a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:55.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.054+0000 7f534ce00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5348103f00 0x7f534819a580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:60312/0 (socket says 172.21.15.90:60312) 2024-04-02T14:44:55.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.054+0000 7f534ce00640 1 -- 172.21.15.90:0/3844449256 learned_addr learned my addr 172.21.15.90:0/3844449256 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:44:55.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.054+0000 7f534d800640 1 --2- 172.21.15.90:0/3844449256 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f534819edf0 0x7f534819f2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:55.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.054+0000 7f534ce00640 1 -- 172.21.15.90:0/3844449256 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5348108fd0 msgr2=0x7f534819aac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:55.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.054+0000 7f534ce00640 1 --2- 172.21.15.90:0/3844449256 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5348108fd0 0x7f534819aac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:55.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.054+0000 7f534ce00640 1 -- 172.21.15.90:0/3844449256 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f534819edf0 msgr2=0x7f534819f2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:55.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.054+0000 7f534ce00640 1 --2- 172.21.15.90:0/3844449256 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f534819edf0 0x7f534819f2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:55.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.054+0000 7f534ce00640 1 -- 172.21.15.90:0/3844449256 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f534819f8e0 con 0x7f5348103f00 2024-04-02T14:44:55.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.055+0000 7f534ce00640 1 --2- 172.21.15.90:0/3844449256 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5348103f00 0x7f534819a580 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f533c005ab0 tx=0x7f533c0042f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:44:55.052 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.055+0000 7f5345600640 1 -- 172.21.15.90:0/3844449256 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f533c041070 con 0x7f5348103f00 2024-04-02T14:44:55.052 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.056+0000 7f534f800640 1 -- 172.21.15.90:0/3844449256 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f534819fb70 con 0x7f5348103f00 2024-04-02T14:44:55.052 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.056+0000 7f5345600640 1 -- 172.21.15.90:0/3844449256 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f533c02cca0 con 0x7f5348103f00 2024-04-02T14:44:55.052 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.056+0000 7f534f800640 1 -- 172.21.15.90:0/3844449256 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53481a7390 con 0x7f5348103f00 2024-04-02T14:44:55.052 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.056+0000 7f5345600640 1 -- 172.21.15.90:0/3844449256 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f533c041070 con 0x7f5348103f00 2024-04-02T14:44:55.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.060+0000 7f534f800640 1 -- 172.21.15.90:0/3844449256 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5328005290 con 0x7f5348103f00 2024-04-02T14:44:55.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.061+0000 7f5345600640 1 -- 172.21.15.90:0/3844449256 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f533c044070 con 0x7f5348103f00 2024-04-02T14:44:55.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.064+0000 7f5345600640 1 --2- 172.21.15.90:0/3844449256 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f53300a41a0 0x7f53300a6660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:55.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.064+0000 7f5345600640 1 -- 172.21.15.90:0/3844449256 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8550+0+0 (secure 0 0 0) 0x7f533c02caa0 con 0x7f5348103f00 2024-04-02T14:44:55.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.065+0000 7f5347e00640 1 --2- 172.21.15.90:0/3844449256 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f53300a41a0 0x7f53300a6660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:55.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.068+0000 7f5345600640 1 -- 172.21.15.90:0/3844449256 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7f533c0bf380 con 0x7f5348103f00 2024-04-02T14:44:55.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.069+0000 7f5347e00640 1 --2- 172.21.15.90:0/3844449256 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f53300a41a0 0x7f53300a6660 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f5348102e50 tx=0x7f5334005f90 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:44:55.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.533+0000 7f534f800640 1 -- 172.21.15.90:0/3844449256 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5328005850 con 0x7f5348103f00 2024-04-02T14:44:55.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.535+0000 7f5345600640 1 -- 172.21.15.90:0/3844449256 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11927 (secure 0 0 0) 0x7f533c0bec80 con 0x7f5348103f00 2024-04-02T14:44:55.531 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-02T14:44:55.532 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14412,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6837/2852160309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6836","nonce":2852160309},{"type":"v1","addr":"172.21.15.90:6837","nonce":2852160309}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14478,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6841/1919376858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6840","nonce":1919376858},{"type":"v1","addr":"172.21.15.90:6841","nonce":1919376858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-02T14:32:22.319083+0000","modified":"2024-04-02T14:34:52.266152+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24590,"mds_1":24560,"mds_2":24547,"mds_3":24539,"mds_4":24523},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24454":{"gid":24454,"name":"l","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6833/1564319431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":1564319431},{"type":"v1","addr":"172.21.15.177:6833","nonce":1564319431}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.92:6833/1570337372","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":1570337372},{"type":"v1","addr":"172.21.15.92:6833","nonce":1570337372}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"j","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.90:6835/1277124891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6834","nonce":1277124891},{"type":"v1","addr":"172.21.15.90:6835","nonce":1277124891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.92:6835/860721164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":860721164},{"type":"v1","addr":"172.21.15.92:6835","nonce":860721164}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24511":{"gid":24511,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6835/3177244988","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":3177244988},{"type":"v1","addr":"172.21.15.177:6835","nonce":3177244988}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"c","rank":4,"incarnation":25,"state":"up:active","state_seq":9,"addr":"172.21.15.177:6837/4223414858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":4223414858},{"type":"v1","addr":"172.21.15.177:6837","nonce":4223414858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24539":{"gid":24539,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":8,"addr":"172.21.15.92:6837/1270635939","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":1270635939},{"type":"v1","addr":"172.21.15.92:6837","nonce":1270635939}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24547":{"gid":24547,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":7,"addr":"172.21.15.177:6839/664895663","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":664895663},{"type":"v1","addr":"172.21.15.177:6839","nonce":664895663}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"g","rank":1,"incarnation":19,"state":"up:active","state_seq":5,"addr":"172.21.15.90:6839/1498915870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6838","nonce":1498915870},{"type":"v1","addr":"172.21.15.90:6839","nonce":1498915870}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24590":{"gid":24590,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.92:6839/1054888219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":1054888219},{"type":"v1","addr":"172.21.15.92:6839","nonce":1054888219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24590,"qdb_cluster":[24547,24590,24523,24539,24560]},"id":1}]} 2024-04-02T14:44:55.532 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 27 2024-04-02T14:44:55.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.541+0000 7f534f800640 1 -- 172.21.15.90:0/3844449256 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f53300a41a0 msgr2=0x7f53300a6660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:55.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.541+0000 7f534f800640 1 --2- 172.21.15.90:0/3844449256 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f53300a41a0 0x7f53300a6660 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f5348102e50 tx=0x7f5334005f90 comp rx=0 tx=0).stop 2024-04-02T14:44:55.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.541+0000 7f534f800640 1 -- 172.21.15.90:0/3844449256 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5348103f00 msgr2=0x7f534819a580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:55.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.541+0000 7f534f800640 1 --2- 172.21.15.90:0/3844449256 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5348103f00 0x7f534819a580 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f533c005ab0 tx=0x7f533c0042f0 comp rx=0 tx=0).stop 2024-04-02T14:44:55.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.542+0000 7f534f800640 1 -- 172.21.15.90:0/3844449256 shutdown_connections 2024-04-02T14:44:55.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.542+0000 7f534f800640 1 --2- 172.21.15.90:0/3844449256 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f53300a41a0 0x7f53300a6660 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:55.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.542+0000 7f534f800640 1 --2- 172.21.15.90:0/3844449256 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f534819edf0 0x7f534819f2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:55.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.542+0000 7f534f800640 1 --2- 172.21.15.90:0/3844449256 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5348108fd0 0x7f534819aac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:55.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.542+0000 7f534f800640 1 --2- 172.21.15.90:0/3844449256 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5348103f00 0x7f534819a580 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:55.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.542+0000 7f534f800640 1 -- 172.21.15.90:0/3844449256 >> 172.21.15.90:0/3844449256 conn(0x7f53480ff760 msgr2=0x7f5348101a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:44:55.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.542+0000 7f534f800640 1 -- 172.21.15.90:0/3844449256 shutdown_connections 2024-04-02T14:44:55.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.542+0000 7f534f800640 1 -- 172.21.15.90:0/3844449256 wait complete. 2024-04-02T14:44:55.570 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-04-02T14:44:55.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.860+0000 7f2439200640 1 -- 172.21.15.90:0/2539827230 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f24341045c0 msgr2=0x7f2434104a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:55.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.860+0000 7f2439200640 1 --2- 172.21.15.90:0/2539827230 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f24341045c0 0x7f2434104a40 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f242c009cd0 tx=0x7f242c02c100 comp rx=0 tx=0).stop 2024-04-02T14:44:55.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.860+0000 7f2439200640 1 -- 172.21.15.90:0/2539827230 shutdown_connections 2024-04-02T14:44:55.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.860+0000 7f2439200640 1 --2- 172.21.15.90:0/2539827230 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f24341045c0 0x7f2434104a40 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:55.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.860+0000 7f2439200640 1 --2- 172.21.15.90:0/2539827230 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2434069240 0x7f24340696a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:55.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.860+0000 7f2439200640 1 --2- 172.21.15.90:0/2539827230 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2434106a50 0x7f2434068d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:55.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.860+0000 7f2439200640 1 -- 172.21.15.90:0/2539827230 >> 172.21.15.90:0/2539827230 conn(0x7f2434077b00 msgr2=0x7f2434077f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:44:55.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.860+0000 7f2439200640 1 -- 172.21.15.90:0/2539827230 shutdown_connections 2024-04-02T14:44:55.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.860+0000 7f2439200640 1 -- 172.21.15.90:0/2539827230 wait complete. 2024-04-02T14:44:55.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.861+0000 7f2439200640 1 Processor -- start 2024-04-02T14:44:55.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.861+0000 7f2439200640 1 -- start start 2024-04-02T14:44:55.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.862+0000 7f2439200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2434069240 0x7f24341a4640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:55.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.862+0000 7f2439200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f24341045c0 0x7f24341a4b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:55.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.862+0000 7f2439200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2434106a50 0x7f243419e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:55.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.862+0000 7f2439200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f24341184b0 con 0x7f2434106a50 2024-04-02T14:44:55.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.862+0000 7f2439200640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f2434118330 con 0x7f24341045c0 2024-04-02T14:44:55.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.862+0000 7f2439200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f2434118630 con 0x7f2434069240 2024-04-02T14:44:55.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.862+0000 7f2431e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2434069240 0x7f24341a4640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:55.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.862+0000 7f2431400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f24341045c0 0x7f24341a4b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:55.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.862+0000 7f2431e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2434069240 0x7f24341a4640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:60334/0 (socket says 172.21.15.90:60334) 2024-04-02T14:44:55.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.862+0000 7f2431e00640 1 -- 172.21.15.90:0/380217982 learned_addr learned my addr 172.21.15.90:0/380217982 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:44:55.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.862+0000 7f2432800640 1 --2- 172.21.15.90:0/380217982 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2434106a50 0x7f243419e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:55.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.863+0000 7f2431e00640 1 -- 172.21.15.90:0/380217982 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f24341045c0 msgr2=0x7f24341a4b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:55.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.863+0000 7f2431e00640 1 --2- 172.21.15.90:0/380217982 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f24341045c0 0x7f24341a4b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:55.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.863+0000 7f2431e00640 1 -- 172.21.15.90:0/380217982 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2434106a50 msgr2=0x7f243419e7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:55.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.863+0000 7f2431e00640 1 --2- 172.21.15.90:0/380217982 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2434106a50 0x7f243419e7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:55.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.863+0000 7f2431e00640 1 -- 172.21.15.90:0/380217982 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f243419ef80 con 0x7f2434069240 2024-04-02T14:44:55.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.864+0000 7f2431e00640 1 --2- 172.21.15.90:0/380217982 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2434069240 0x7f24341a4640 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f242400b4a0 tx=0x7f242400b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:44:55.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.864+0000 7f242aa00640 1 -- 172.21.15.90:0/380217982 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2424014070 con 0x7f2434069240 2024-04-02T14:44:55.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.865+0000 7f242aa00640 1 -- 172.21.15.90:0/380217982 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2424010af0 con 0x7f2434069240 2024-04-02T14:44:55.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.865+0000 7f242aa00640 1 -- 172.21.15.90:0/380217982 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2424017d10 con 0x7f2434069240 2024-04-02T14:44:55.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.865+0000 7f2439200640 1 -- 172.21.15.90:0/380217982 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f243419f270 con 0x7f2434069240 2024-04-02T14:44:55.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.865+0000 7f2439200640 1 -- 172.21.15.90:0/380217982 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24341ab490 con 0x7f2434069240 2024-04-02T14:44:55.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.870+0000 7f242aa00640 1 -- 172.21.15.90:0/380217982 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f2424017070 con 0x7f2434069240 2024-04-02T14:44:55.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.870+0000 7f243bf76740 1 Processor -- start 2024-04-02T14:44:55.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.870+0000 7f243bf76740 1 -- start start 2024-04-02T14:44:55.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.871+0000 7f243bf76740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5612e23339f0 0x5612e2332740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:55.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.871+0000 7f243bf76740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5612e2332d10 0x5612e2333190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:55.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.871+0000 7f243bf76740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5612e233c080 0x5612e233e470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:55.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.871+0000 7f243bf76740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5612e2273400 con 0x5612e23339f0 2024-04-02T14:44:55.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.871+0000 7f243bf76740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x5612e2257f10 con 0x5612e2332d10 2024-04-02T14:44:55.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.871+0000 7f243bf76740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5612e22976f0 con 0x5612e233c080 2024-04-02T14:44:55.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.871+0000 7f2431e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5612e2332d10 0x5612e2333190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:55.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.871+0000 7f2432800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5612e23339f0 0x5612e2332740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:55.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.871+0000 7f2431400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5612e233c080 0x5612e233e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:55.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.871+0000 7f2431e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5612e2332d10 0x5612e2333190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:40732/0 (socket says 172.21.15.90:40732) 2024-04-02T14:44:55.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.871+0000 7f2431e00640 1 -- 172.21.15.90:0/3627449369 learned_addr learned my addr 172.21.15.90:0/3627449369 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:44:55.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.871+0000 7f2431400640 1 --2- 172.21.15.90:0/3627449369 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5612e233c080 0x5612e233e470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:60350/0 (socket says 172.21.15.90:60350) 2024-04-02T14:44:55.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.872+0000 7f2431400640 1 -- 172.21.15.90:0/3627449369 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5612e2332d10 msgr2=0x5612e2333190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:44:55.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.872+0000 7f2431400640 1 --2- 172.21.15.90:0/3627449369 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5612e2332d10 0x5612e2333190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:55.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.872+0000 7f2431400640 1 -- 172.21.15.90:0/3627449369 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5612e23339f0 msgr2=0x5612e2332740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:44:55.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.872+0000 7f2431400640 1 --2- 172.21.15.90:0/3627449369 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5612e23339f0 0x5612e2332740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:55.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.872+0000 7f2431400640 1 -- 172.21.15.90:0/3627449369 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5612e1fab3f0 con 0x5612e233c080 2024-04-02T14:44:55.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.872+0000 7f2431400640 1 --2- 172.21.15.90:0/3627449369 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5612e233c080 0x5612e233e470 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f243419fb00 tx=0x7f24200107a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f58c8f30e7eb452c server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:44:55.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.872+0000 7f242aa00640 1 --2- 172.21.15.90:0/380217982 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f24180a3da0 0x7f24180a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:55.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.872+0000 7f242aa00640 1 -- 172.21.15.90:0/380217982 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8550+0+0 (secure 0 0 0) 0x7f24240cf970 con 0x7f2434069240 2024-04-02T14:44:55.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.873+0000 7f241f400640 1 -- 172.21.15.90:0/3627449369 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f242001c680 con 0x5612e233c080 2024-04-02T14:44:55.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.873+0000 7f241f400640 1 -- 172.21.15.90:0/3627449369 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f242000be20 con 0x5612e233c080 2024-04-02T14:44:55.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.873+0000 7f241f400640 1 -- 172.21.15.90:0/3627449369 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2420011c70 con 0x5612e233c080 2024-04-02T14:44:55.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.873+0000 7f243bf76740 1 -- 172.21.15.90:0/3627449369 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5612e233c080 msgr2=0x5612e233e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:55.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.873+0000 7f243bf76740 1 --2- 172.21.15.90:0/3627449369 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5612e233c080 0x5612e233e470 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f243419fb00 tx=0x7f24200107a0 comp rx=0 tx=0).stop 2024-04-02T14:44:55.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.874+0000 7f2432800640 1 --2- 172.21.15.90:0/380217982 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f24180a3da0 0x7f24180a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:55.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.874+0000 7f243bf76740 1 -- 172.21.15.90:0/3627449369 shutdown_connections 2024-04-02T14:44:55.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.874+0000 7f243bf76740 1 --2- 172.21.15.90:0/3627449369 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5612e233c080 0x5612e233e470 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:55.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.874+0000 7f243bf76740 1 --2- 172.21.15.90:0/3627449369 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5612e2332d10 0x5612e2333190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:55.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.874+0000 7f243bf76740 1 --2- 172.21.15.90:0/3627449369 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5612e23339f0 0x5612e2332740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:55.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.874+0000 7f243bf76740 1 -- 172.21.15.90:0/3627449369 >> 172.21.15.90:0/3627449369 conn(0x5612e22f9a50 msgr2=0x5612e232d760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:44:55.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.875+0000 7f243bf76740 1 -- 172.21.15.90:0/3627449369 shutdown_connections 2024-04-02T14:44:55.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.875+0000 7f2432800640 1 --2- 172.21.15.90:0/380217982 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f24180a3da0 0x7f24180a6260 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f242c037040 tx=0x7f242c02e040 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:44:55.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.875+0000 7f243bf76740 1 -- 172.21.15.90:0/3627449369 wait complete. 2024-04-02T14:44:55.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.875+0000 7f243bf76740 1 -- start start 2024-04-02T14:44:55.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.876+0000 7f243bf76740 1 Processor -- start 2024-04-02T14:44:55.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.876+0000 7f243bf76740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5612e2332d10 0x5612e2339070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:55.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.876+0000 7f243bf76740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5612e23339f0 0x5612e23395b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:55.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.876+0000 7f243bf76740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5612e233c080 0x5612e2339af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:55.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.876+0000 7f243bf76740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5612e22936a0 con 0x5612e2332d10 2024-04-02T14:44:55.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.876+0000 7f243bf76740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x5612e223f980 con 0x5612e233c080 2024-04-02T14:44:55.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.876+0000 7f243bf76740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5612e2252cb0 con 0x5612e23339f0 2024-04-02T14:44:55.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.876+0000 7f243bf76740 10 client.0 _ll_register_callbacks cb 0x5612e22526a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:44:55.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.877+0000 7f2432800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5612e233c080 0x5612e2339af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:55.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.877+0000 7f2432800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5612e233c080 0x5612e2339af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:40738/0 (socket says 172.21.15.90:40738) 2024-04-02T14:44:55.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.877+0000 7f2432800640 1 -- 172.21.15.90:0/1451233045 learned_addr learned my addr 172.21.15.90:0/1451233045 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:44:55.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.877+0000 7f2431400640 1 --2- 172.21.15.90:0/1451233045 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5612e23339f0 0x5612e23395b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:55.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.877+0000 7f2431e00640 1 --2- 172.21.15.90:0/1451233045 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5612e2332d10 0x5612e2339070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:55.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.877+0000 7f2432800640 1 -- 172.21.15.90:0/1451233045 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5612e23339f0 msgr2=0x5612e23395b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:44:55.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.877+0000 7f2432800640 1 --2- 172.21.15.90:0/1451233045 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5612e23339f0 0x5612e23395b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:55.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.877+0000 7f2432800640 1 -- 172.21.15.90:0/1451233045 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5612e2332d10 msgr2=0x5612e2339070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:44:55.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.877+0000 7f2432800640 1 --2- 172.21.15.90:0/1451233045 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5612e2332d10 0x5612e2339070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:55.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.877+0000 7f2432800640 1 -- 172.21.15.90:0/1451233045 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5612e20757d0 con 0x5612e233c080 2024-04-02T14:44:55.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.877+0000 7f2431e00640 1 --2- 172.21.15.90:0/1451233045 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5612e2332d10 0x5612e2339070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:44:55.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.878+0000 7f2432800640 1 --2- 172.21.15.90:0/1451233045 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5612e233c080 0x5612e2339af0 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f242c005e30 tx=0x7f242c02c100 comp rx=0 tx=0).ready entity=mon.2 client_cookie=186292cafd9caa58 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:44:55.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.878+0000 7f2413e00640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:44:55.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.878+0000 7f2413e00640 1 -- 172.21.15.90:0/1451233045 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f242c039040 con 0x5612e233c080 2024-04-02T14:44:55.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.878+0000 7f243bf76740 1 -- 172.21.15.90:0/1451233045 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f242000be20 con 0x5612e233c080 2024-04-02T14:44:55.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.878+0000 7f2413e00640 1 -- 172.21.15.90:0/1451233045 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8550+0+0 (secure 0 0 0) 0x7f242c03fdb0 con 0x5612e233c080 2024-04-02T14:44:55.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.878+0000 7f2413e00640 1 -- 172.21.15.90:0/1451233045 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f242c05a050 con 0x5612e233c080 2024-04-02T14:44:55.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.878+0000 7f243bf76740 10 client.28550 fetch_fsmap learned FSMap version 27 2024-04-02T14:44:55.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.879+0000 7f243bf76740 1 -- 172.21.15.90:0/1451233045 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5612e234af20 con 0x5612e233c080 2024-04-02T14:44:55.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.879+0000 7f2413e00640 1 -- 172.21.15.90:0/1451233045 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f242000be20 con 0x5612e233c080 2024-04-02T14:44:55.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.879+0000 7f2413e00640 1 -- 172.21.15.90:0/1451233045 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f242c04c480 con 0x5612e233c080 2024-04-02T14:44:55.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.879+0000 7f243bf76740 10 client.28550 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:44:55.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.880+0000 7f243bf76740 10 client.28550 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.c 2024-04-02T14:44:55.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.880+0000 7f243bf76740 20 client.28550 populate_metadata read hostname 'smithi090' 2024-04-02T14:44:55.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.880+0000 7f243bf76740 1 --2- 172.21.15.90:0/1451233045 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x5612e234c480 0x5612e234e900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:55.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.880+0000 7f243bf76740 4 client.28550 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:44:55.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.880+0000 7f243bf76740 1 -- 172.21.15.90:0/1451233045 --> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5612e2257040 con 0x5612e234c480 2024-04-02T14:44:55.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.881+0000 7f2431e00640 1 --2- 172.21.15.90:0/1451233045 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x5612e234c480 0x5612e234e900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:55.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.883+0000 7f2413e00640 10 client.28550 ms_handle_connect on v2:172.21.15.177:6836/4223414858 2024-04-02T14:44:55.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.883+0000 7f2431e00640 1 --2- 172.21.15.90:0/1451233045 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x5612e234c480 0x5612e234e900 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=afb27e1ea024d1ba server_cookie=88e086f0f5f6ac9c in_seq=0 out_seq=0 2024-04-02T14:44:55.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.883+0000 7f2413e00640 1 -- 172.21.15.90:0/1451233045 <== mds.4 v2:172.21.15.177:6836/4223414858 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5612e2257040 con 0x5612e234c480 2024-04-02T14:44:55.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.883+0000 7f2413e00640 10 client.28550 handle_command_reply: tid=0 2024-04-02T14:44:55.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.884+0000 7f243bf76740 1 client.28550 shutdown 2024-04-02T14:44:55.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.885+0000 7f243bf76740 1 -- 172.21.15.90:0/1451233045 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5612e233c080 msgr2=0x5612e2339af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:55.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.885+0000 7f243bf76740 1 --2- 172.21.15.90:0/1451233045 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5612e233c080 0x5612e2339af0 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f242c005e30 tx=0x7f242c02c100 comp rx=0 tx=0).stop 2024-04-02T14:44:55.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.885+0000 7f243bf76740 1 -- 172.21.15.90:0/1451233045 shutdown_connections 2024-04-02T14:44:55.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.885+0000 7f243bf76740 1 --2- 172.21.15.90:0/1451233045 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5612e233c080 0x5612e2339af0 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:55.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.885+0000 7f243bf76740 1 --2- 172.21.15.90:0/1451233045 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5612e23339f0 0x5612e23395b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:55.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.885+0000 7f243bf76740 1 --2- 172.21.15.90:0/1451233045 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x5612e234c480 0x5612e234e900 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:55.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.885+0000 7f243bf76740 1 --2- 172.21.15.90:0/1451233045 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5612e2332d10 0x5612e2339070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:55.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.885+0000 7f2413e00640 0 client.28550 ms_handle_reset on v2:172.21.15.177:6836/4223414858 2024-04-02T14:44:55.882 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.885+0000 7f243bf76740 1 -- 172.21.15.90:0/1451233045 >> 172.21.15.90:0/1451233045 conn(0x5612e22f9a50 msgr2=0x5612e232f4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:44:55.882 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.885+0000 7f243bf76740 1 -- 172.21.15.90:0/1451233045 shutdown_connections 2024-04-02T14:44:55.882 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.886+0000 7f243bf76740 1 -- 172.21.15.90:0/1451233045 wait complete. 2024-04-02T14:44:55.882 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.886+0000 7f243bf76740 20 client.28550 trim_cache size 0 max 16384 2024-04-02T14:44:55.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.916+0000 7f243bf76740 1 Processor -- start 2024-04-02T14:44:55.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.916+0000 7f243bf76740 1 -- start start 2024-04-02T14:44:55.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.916+0000 7f243bf76740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5612e2332d10 0x5612e233b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:55.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.916+0000 7f243bf76740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5612e23339f0 0x5612e2347af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:55.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.916+0000 7f243bf76740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5612e233c080 0x5612e23381d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:55.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.917+0000 7f2431400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5612e23339f0 0x5612e2347af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:55.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.917+0000 7f2431400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5612e23339f0 0x5612e2347af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:40744/0 (socket says 172.21.15.90:40744) 2024-04-02T14:44:55.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.917+0000 7f2431400640 1 -- 172.21.15.90:0/4228029471 learned_addr learned my addr 172.21.15.90:0/4228029471 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:44:55.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.917+0000 7f2431e00640 1 --2- 172.21.15.90:0/4228029471 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5612e2332d10 0x5612e233b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:55.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.917+0000 7f243bf76740 1 -- 172.21.15.90:0/4228029471 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f24341b3ed0 con 0x5612e233c080 2024-04-02T14:44:55.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.917+0000 7f243bf76740 1 -- 172.21.15.90:0/4228029471 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f24341b5a10 con 0x5612e23339f0 2024-04-02T14:44:55.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.917+0000 7f243bf76740 1 -- 172.21.15.90:0/4228029471 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f24341b45a0 con 0x5612e2332d10 2024-04-02T14:44:55.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.917+0000 7f2432800640 1 --2- 172.21.15.90:0/4228029471 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5612e233c080 0x5612e23381d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:55.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.917+0000 7f2431400640 1 -- 172.21.15.90:0/4228029471 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5612e2332d10 msgr2=0x5612e233b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:44:55.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.917+0000 7f2431400640 1 --2- 172.21.15.90:0/4228029471 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5612e2332d10 0x5612e233b7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:55.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.917+0000 7f2431400640 1 -- 172.21.15.90:0/4228029471 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5612e233c080 msgr2=0x5612e23381d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:44:55.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.917+0000 7f2431400640 1 --2- 172.21.15.90:0/4228029471 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5612e233c080 0x5612e23381d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:55.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.917+0000 7f2431400640 1 -- 172.21.15.90:0/4228029471 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5612e23375c0 con 0x5612e23339f0 2024-04-02T14:44:55.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.917+0000 7f2432800640 1 --2- 172.21.15.90:0/4228029471 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5612e233c080 0x5612e23381d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:44:55.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.919+0000 7f2431400640 1 --2- 172.21.15.90:0/4228029471 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5612e23339f0 0x5612e2347af0 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f2420005d00 tx=0x7f242000bc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4ecf312f4e10ddab server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:44:55.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.920+0000 7f241ea00640 1 -- 172.21.15.90:0/4228029471 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2420004020 con 0x5612e23339f0 2024-04-02T14:44:55.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.920+0000 7f241ea00640 1 -- 172.21.15.90:0/4228029471 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2420005520 con 0x5612e23339f0 2024-04-02T14:44:55.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.920+0000 7f241ea00640 1 -- 172.21.15.90:0/4228029471 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2420004aa0 con 0x5612e23339f0 2024-04-02T14:44:55.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.921+0000 7f243bf76740 1 -- 172.21.15.90:0/4228029471 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5612e23339f0 msgr2=0x5612e2347af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:55.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.921+0000 7f243bf76740 1 --2- 172.21.15.90:0/4228029471 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5612e23339f0 0x5612e2347af0 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f2420005d00 tx=0x7f242000bc30 comp rx=0 tx=0).stop 2024-04-02T14:44:55.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.922+0000 7f243bf76740 1 -- 172.21.15.90:0/4228029471 shutdown_connections 2024-04-02T14:44:55.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.922+0000 7f243bf76740 1 --2- 172.21.15.90:0/4228029471 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5612e233c080 0x5612e23381d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:55.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.922+0000 7f243bf76740 1 --2- 172.21.15.90:0/4228029471 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5612e23339f0 0x5612e2347af0 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:55.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.922+0000 7f243bf76740 1 --2- 172.21.15.90:0/4228029471 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5612e2332d10 0x5612e233b7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:55.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.922+0000 7f243bf76740 1 -- 172.21.15.90:0/4228029471 >> 172.21.15.90:0/4228029471 conn(0x5612e22f9a50 msgr2=0x5612e232c010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:44:55.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.922+0000 7f243bf76740 1 -- 172.21.15.90:0/4228029471 shutdown_connections 2024-04-02T14:44:55.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.923+0000 7f243bf76740 1 -- 172.21.15.90:0/4228029471 wait complete. 2024-04-02T14:44:55.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.923+0000 7f243bf76740 1 -- start start 2024-04-02T14:44:55.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.924+0000 7f243bf76740 1 Processor -- start 2024-04-02T14:44:55.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.925+0000 7f243bf76740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5612e2332d10 0x5612e233c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:55.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.925+0000 7f243bf76740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5612e23339f0 0x5612e233cb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:55.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.925+0000 7f243bf76740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5612e233c080 0x5612e2343000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:55.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.925+0000 7f243bf76740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f24341b5340 con 0x5612e23339f0 2024-04-02T14:44:55.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.925+0000 7f243bf76740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f24341b4c70 con 0x5612e233c080 2024-04-02T14:44:55.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.925+0000 7f243bf76740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f24341b1ed0 con 0x5612e2332d10 2024-04-02T14:44:55.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.925+0000 7f243bf76740 10 client.0 _ll_register_callbacks cb 0x7f242c032960 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:44:55.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.926+0000 7f2432800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5612e233c080 0x5612e2343000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:55.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.926+0000 7f2432800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5612e233c080 0x5612e2343000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:40746/0 (socket says 172.21.15.90:40746) 2024-04-02T14:44:55.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.926+0000 7f2432800640 1 -- 172.21.15.90:0/3522320552 learned_addr learned my addr 172.21.15.90:0/3522320552 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:44:55.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.926+0000 7f2431e00640 1 --2- 172.21.15.90:0/3522320552 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5612e2332d10 0x5612e233c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:55.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.926+0000 7f2432800640 1 -- 172.21.15.90:0/3522320552 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5612e2332d10 msgr2=0x5612e233c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:44:55.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.926+0000 7f2432800640 1 --2- 172.21.15.90:0/3522320552 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5612e2332d10 0x5612e233c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:55.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.926+0000 7f2432800640 1 -- 172.21.15.90:0/3522320552 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5612e23339f0 msgr2=0x5612e233cb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:44:55.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.926+0000 7f2432800640 1 --2- 172.21.15.90:0/3522320552 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5612e23339f0 0x5612e233cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:55.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.927+0000 7f2432800640 1 -- 172.21.15.90:0/3522320552 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5612e2343540 con 0x5612e233c080 2024-04-02T14:44:55.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.927+0000 7f2432800640 1 --2- 172.21.15.90:0/3522320552 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5612e233c080 0x5612e2343000 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f242c02e010 tx=0x7f242c04cdf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a68e2a46e4bd03e9 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:44:55.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.927+0000 7f2413e00640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:44:55.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.927+0000 7f2431e00640 1 --2- 172.21.15.90:0/3522320552 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5612e2332d10 0x5612e233c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:44:55.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.928+0000 7f2413e00640 1 -- 172.21.15.90:0/3522320552 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f242c062020 con 0x5612e233c080 2024-04-02T14:44:55.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.928+0000 7f2413e00640 1 -- 172.21.15.90:0/3522320552 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8550+0+0 (secure 0 0 0) 0x7f242c03fbb0 con 0x5612e233c080 2024-04-02T14:44:55.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.928+0000 7f243bf76740 1 -- 172.21.15.90:0/3522320552 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2420005520 con 0x5612e233c080 2024-04-02T14:44:55.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.929+0000 7f2413e00640 1 -- 172.21.15.90:0/3522320552 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f242c039040 con 0x5612e233c080 2024-04-02T14:44:55.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.929+0000 7f243bf76740 10 client.28556 fetch_fsmap learned FSMap version 27 2024-04-02T14:44:55.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.929+0000 7f243bf76740 1 -- 172.21.15.90:0/3522320552 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5612e23447e0 con 0x5612e233c080 2024-04-02T14:44:55.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.929+0000 7f2413e00640 1 -- 172.21.15.90:0/3522320552 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2420005520 con 0x5612e233c080 2024-04-02T14:44:55.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.929+0000 7f2413e00640 1 -- 172.21.15.90:0/3522320552 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f242c05e430 con 0x5612e233c080 2024-04-02T14:44:55.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.930+0000 7f243bf76740 10 client.28556 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:44:55.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.930+0000 7f243bf76740 10 client.28556 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.c 2024-04-02T14:44:55.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.930+0000 7f243bf76740 20 client.28556 populate_metadata read hostname 'smithi090' 2024-04-02T14:44:55.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.930+0000 7f243bf76740 1 --2- 172.21.15.90:0/3522320552 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x5612e23b2dc0 0x5612e23b5220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:55.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.930+0000 7f243bf76740 4 client.28556 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-02T14:44:55.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.930+0000 7f243bf76740 1 -- 172.21.15.90:0/3522320552 --> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5612e22a4c30 con 0x5612e23b2dc0 2024-04-02T14:44:55.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.931+0000 7f2431e00640 1 --2- 172.21.15.90:0/3522320552 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x5612e23b2dc0 0x5612e23b5220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:55.929 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.933+0000 7f2431e00640 1 --2- 172.21.15.90:0/3522320552 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x5612e23b2dc0 0x5612e23b5220 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=7cab2c84214473ed server_cookie=b132fa49ced2fa58 in_seq=0 out_seq=0 2024-04-02T14:44:55.929 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.933+0000 7f2413e00640 10 client.28556 ms_handle_connect on v2:172.21.15.177:6836/4223414858 2024-04-02T14:44:55.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.933+0000 7f2413e00640 1 -- 172.21.15.90:0/3522320552 <== mds.4 v2:172.21.15.177:6836/4223414858 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5612e22a4c30 con 0x5612e23b2dc0 2024-04-02T14:44:55.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.933+0000 7f2413e00640 10 client.28556 handle_command_reply: tid=0 2024-04-02T14:44:55.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.934+0000 7f243bf76740 1 client.28556 shutdown 2024-04-02T14:44:55.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.934+0000 7f243bf76740 1 -- 172.21.15.90:0/3522320552 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5612e233c080 msgr2=0x5612e2343000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:55.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.934+0000 7f243bf76740 1 --2- 172.21.15.90:0/3522320552 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5612e233c080 0x5612e2343000 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f242c02e010 tx=0x7f242c04cdf0 comp rx=0 tx=0).stop 2024-04-02T14:44:55.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.935+0000 7f243bf76740 1 -- 172.21.15.90:0/3522320552 shutdown_connections 2024-04-02T14:44:55.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.935+0000 7f243bf76740 1 --2- 172.21.15.90:0/3522320552 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5612e233c080 0x5612e2343000 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:55.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.935+0000 7f243bf76740 1 --2- 172.21.15.90:0/3522320552 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x5612e23b2dc0 0x5612e23b5220 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:55.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.935+0000 7f243bf76740 1 --2- 172.21.15.90:0/3522320552 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5612e23339f0 0x5612e233cb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:55.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.935+0000 7f243bf76740 1 --2- 172.21.15.90:0/3522320552 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5612e2332d10 0x5612e233c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:55.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.935+0000 7f243bf76740 1 -- 172.21.15.90:0/3522320552 >> 172.21.15.90:0/3522320552 conn(0x5612e22f9a50 msgr2=0x5612e232c010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:44:55.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.935+0000 7f2413e00640 0 client.28556 ms_handle_reset on v2:172.21.15.177:6836/4223414858 2024-04-02T14:44:55.932 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:44:55 smithi177 ceph-mon[36431]: pgmap v310: 97 pgs: 97 active+clean; 4.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 8.8 MiB/s wr, 851 op/s 2024-04-02T14:44:55.932 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:44:55 smithi177 ceph-mon[36431]: from='client.? 172.21.15.90:0/3844449256' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-02T14:44:55.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.936+0000 7f243bf76740 1 -- 172.21.15.90:0/3522320552 shutdown_connections 2024-04-02T14:44:55.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.936+0000 7f243bf76740 1 -- 172.21.15.90:0/3522320552 wait complete. 2024-04-02T14:44:55.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.936+0000 7f243bf76740 20 client.28556 trim_cache size 0 max 16384 2024-04-02T14:44:55.933 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-02T14:44:55.933 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:44:55 smithi090 ceph-mon[31674]: pgmap v310: 97 pgs: 97 active+clean; 4.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 8.8 MiB/s wr, 851 op/s 2024-04-02T14:44:55.933 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:44:55 smithi090 ceph-mon[31674]: from='client.? 172.21.15.90:0/3844449256' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-02T14:44:55.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.938+0000 7f2439200640 1 -- 172.21.15.90:0/380217982 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f24180a3da0 msgr2=0x7f24180a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:55.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.938+0000 7f2439200640 1 --2- 172.21.15.90:0/380217982 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f24180a3da0 0x7f24180a6260 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f242c037040 tx=0x7f242c02e040 comp rx=0 tx=0).stop 2024-04-02T14:44:55.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.938+0000 7f2439200640 1 -- 172.21.15.90:0/380217982 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2434069240 msgr2=0x7f24341a4640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:55.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.938+0000 7f2439200640 1 --2- 172.21.15.90:0/380217982 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2434069240 0x7f24341a4640 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f242400b4a0 tx=0x7f242400b960 comp rx=0 tx=0).stop 2024-04-02T14:44:55.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.938+0000 7f2439200640 1 -- 172.21.15.90:0/380217982 shutdown_connections 2024-04-02T14:44:55.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.938+0000 7f2439200640 1 --2- 172.21.15.90:0/380217982 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f24180a3da0 0x7f24180a6260 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:55.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.938+0000 7f2439200640 1 --2- 172.21.15.90:0/380217982 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2434106a50 0x7f243419e7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:55.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.938+0000 7f2439200640 1 --2- 172.21.15.90:0/380217982 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f24341045c0 0x7f24341a4b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:55.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.938+0000 7f2439200640 1 --2- 172.21.15.90:0/380217982 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2434069240 0x7f24341a4640 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:55.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.938+0000 7f2439200640 1 -- 172.21.15.90:0/380217982 >> 172.21.15.90:0/380217982 conn(0x7f2434077b00 msgr2=0x7f24341019c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:44:55.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.938+0000 7f2439200640 1 -- 172.21.15.90:0/380217982 shutdown_connections 2024-04-02T14:44:55.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:55.938+0000 7f2439200640 1 -- 172.21.15.90:0/380217982 wait complete. 2024-04-02T14:44:55.964 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-04-02T14:44:56.105 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:44:55 smithi092 ceph-mon[36497]: pgmap v310: 97 pgs: 97 active+clean; 4.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 8.8 MiB/s wr, 851 op/s 2024-04-02T14:44:56.105 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:44:55 smithi092 ceph-mon[36497]: from='client.? 172.21.15.90:0/3844449256' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-02T14:44:56.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.264+0000 7fcc43e00640 1 -- 172.21.15.90:0/1736410932 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcc44103450 msgr2=0x7fcc4410b930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:56.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.264+0000 7fcc43e00640 1 --2- 172.21.15.90:0/1736410932 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcc44103450 0x7fcc4410b930 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fcc30009900 tx=0x7fcc3002bcd0 comp rx=0 tx=0).stop 2024-04-02T14:44:56.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.264+0000 7fcc43e00640 1 -- 172.21.15.90:0/1736410932 shutdown_connections 2024-04-02T14:44:56.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.264+0000 7fcc43e00640 1 --2- 172.21.15.90:0/1736410932 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcc4410bf50 0x7fcc4410e3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:56.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.264+0000 7fcc43e00640 1 --2- 172.21.15.90:0/1736410932 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcc44103450 0x7fcc4410b930 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:56.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.264+0000 7fcc43e00640 1 --2- 172.21.15.90:0/1736410932 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcc44102b30 0x7fcc44102f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:56.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.264+0000 7fcc43e00640 1 -- 172.21.15.90:0/1736410932 >> 172.21.15.90:0/1736410932 conn(0x7fcc440fc350 msgr2=0x7fcc440fe770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:44:56.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.264+0000 7fcc43e00640 1 -- 172.21.15.90:0/1736410932 shutdown_connections 2024-04-02T14:44:56.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.264+0000 7fcc43e00640 1 -- 172.21.15.90:0/1736410932 wait complete. 2024-04-02T14:44:56.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.266+0000 7fcc43e00640 1 Processor -- start 2024-04-02T14:44:56.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.267+0000 7fcc43e00640 1 -- start start 2024-04-02T14:44:56.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.267+0000 7fcc43e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcc44102b30 0x7fcc441a4600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:56.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.267+0000 7fcc43e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcc44103450 0x7fcc441a4b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:56.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.267+0000 7fcc43e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcc4410bf50 0x7fcc4410ab00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:56.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.267+0000 7fcc43e00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fcc44111d50 con 0x7fcc44102b30 2024-04-02T14:44:56.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.267+0000 7fcc43e00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fcc44111bd0 con 0x7fcc44103450 2024-04-02T14:44:56.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.267+0000 7fcc43e00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fcc44111ed0 con 0x7fcc4410bf50 2024-04-02T14:44:56.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.268+0000 7fcc42000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcc44103450 0x7fcc441a4b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:56.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.268+0000 7fcc42000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcc44103450 0x7fcc441a4b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:40758/0 (socket says 172.21.15.90:40758) 2024-04-02T14:44:56.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.268+0000 7fcc42000640 1 -- 172.21.15.90:0/3041120331 learned_addr learned my addr 172.21.15.90:0/3041120331 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:44:56.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.268+0000 7fcc42a00640 1 --2- 172.21.15.90:0/3041120331 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcc44102b30 0x7fcc441a4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:56.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.268+0000 7fcc43400640 1 --2- 172.21.15.90:0/3041120331 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcc4410bf50 0x7fcc4410ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:56.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.269+0000 7fcc42000640 1 -- 172.21.15.90:0/3041120331 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcc4410bf50 msgr2=0x7fcc4410ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:56.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.269+0000 7fcc42000640 1 --2- 172.21.15.90:0/3041120331 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcc4410bf50 0x7fcc4410ab00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:56.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.269+0000 7fcc42000640 1 -- 172.21.15.90:0/3041120331 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcc44102b30 msgr2=0x7fcc441a4600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:56.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.269+0000 7fcc42000640 1 --2- 172.21.15.90:0/3041120331 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcc44102b30 0x7fcc441a4600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:56.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.269+0000 7fcc42000640 1 -- 172.21.15.90:0/3041120331 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc4410b370 con 0x7fcc44103450 2024-04-02T14:44:56.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.269+0000 7fcc43400640 1 --2- 172.21.15.90:0/3041120331 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcc4410bf50 0x7fcc4410ab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:44:56.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.269+0000 7fcc42000640 1 --2- 172.21.15.90:0/3041120331 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcc44103450 0x7fcc441a4b40 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7fcc30005940 tx=0x7fcc3002cbd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:44:56.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.269+0000 7fcc3b400640 1 -- 172.21.15.90:0/3041120331 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc30041070 con 0x7fcc44103450 2024-04-02T14:44:56.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.269+0000 7fcc3b400640 1 -- 172.21.15.90:0/3041120331 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcc30004450 con 0x7fcc44103450 2024-04-02T14:44:56.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.270+0000 7fcc43e00640 1 -- 172.21.15.90:0/3041120331 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc4410b600 con 0x7fcc44103450 2024-04-02T14:44:56.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.269+0000 7fcc3b400640 1 -- 172.21.15.90:0/3041120331 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc30003d80 con 0x7fcc44103450 2024-04-02T14:44:56.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.270+0000 7fcc43e00640 1 -- 172.21.15.90:0/3041120331 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc441ab330 con 0x7fcc44103450 2024-04-02T14:44:56.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.276+0000 7fcc4c205740 1 Processor -- start 2024-04-02T14:44:56.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.277+0000 7fcc4c205740 1 -- start start 2024-04-02T14:44:56.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.278+0000 7fcc4c205740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5599f10e3ba0 0x5599f10e28f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:56.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.278+0000 7fcc3b400640 1 -- 172.21.15.90:0/3041120331 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fcc30044070 con 0x7fcc44103450 2024-04-02T14:44:56.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.278+0000 7fcc4c205740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5599f10e2ec0 0x5599f10e3340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:56.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.278+0000 7fcc43400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5599f10e3ba0 0x5599f10e28f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:56.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.278+0000 7fcc43400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5599f10e3ba0 0x5599f10e28f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:60396/0 (socket says 172.21.15.90:60396) 2024-04-02T14:44:56.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.278+0000 7fcc43400640 1 -- 172.21.15.90:0/2042827896 learned_addr learned my addr 172.21.15.90:0/2042827896 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:44:56.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.278+0000 7fcc4c205740 1 --2- 172.21.15.90:0/2042827896 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5599f10ec230 0x5599f10ee620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:56.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.279+0000 7fcc4c205740 1 -- 172.21.15.90:0/2042827896 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5599f1023400 con 0x5599f10e2ec0 2024-04-02T14:44:56.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.279+0000 7fcc4c205740 1 -- 172.21.15.90:0/2042827896 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x5599f1007f10 con 0x5599f10ec230 2024-04-02T14:44:56.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.279+0000 7fcc4c205740 1 -- 172.21.15.90:0/2042827896 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5599f10476f0 con 0x5599f10e3ba0 2024-04-02T14:44:56.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.279+0000 7fcc42a00640 1 --2- 172.21.15.90:0/2042827896 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5599f10e2ec0 0x5599f10e3340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:56.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.279+0000 7fcc42000640 1 --2- 172.21.15.90:0/2042827896 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5599f10ec230 0x5599f10ee620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:56.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.279+0000 7fcc42000640 1 -- 172.21.15.90:0/2042827896 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5599f10e3ba0 msgr2=0x5599f10e28f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:44:56.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.280+0000 7fcc42000640 1 --2- 172.21.15.90:0/2042827896 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5599f10e3ba0 0x5599f10e28f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:56.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.280+0000 7fcc42000640 1 -- 172.21.15.90:0/2042827896 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5599f10e2ec0 msgr2=0x5599f10e3340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:44:56.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.280+0000 7fcc42000640 1 --2- 172.21.15.90:0/2042827896 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5599f10e2ec0 0x5599f10e3340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:56.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.280+0000 7fcc42000640 1 -- 172.21.15.90:0/2042827896 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5599f0d5b3f0 con 0x5599f10ec230 2024-04-02T14:44:56.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.280+0000 7fcc43400640 1 --2- 172.21.15.90:0/2042827896 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5599f10e3ba0 0x5599f10e28f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:44:56.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.280+0000 7fcc42a00640 1 --2- 172.21.15.90:0/2042827896 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5599f10e2ec0 0x5599f10e3340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:44:56.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.280+0000 7fcc3b400640 1 --2- 172.21.15.90:0/3041120331 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fcc280a3da0 0x7fcc280a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:56.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.280+0000 7fcc42000640 1 --2- 172.21.15.90:0/2042827896 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5599f10ec230 0x5599f10ee620 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7fcc300efb40 tx=0x7fcc300fa450 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dd56c054df828b19 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:44:56.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.281+0000 7fcc43400640 1 --2- 172.21.15.90:0/3041120331 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fcc280a3da0 0x7fcc280a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:56.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.281+0000 7fcc2f400640 1 -- 172.21.15.90:0/2042827896 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc300ba630 con 0x5599f10ec230 2024-04-02T14:44:56.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.281+0000 7fcc2f400640 1 -- 172.21.15.90:0/2042827896 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcc30004c80 con 0x5599f10ec230 2024-04-02T14:44:56.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.281+0000 7fcc2f400640 1 -- 172.21.15.90:0/2042827896 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc300bdcf0 con 0x5599f10ec230 2024-04-02T14:44:56.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.281+0000 7fcc3b400640 1 -- 172.21.15.90:0/3041120331 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8550+0+0 (secure 0 0 0) 0x7fcc300f2360 con 0x7fcc44103450 2024-04-02T14:44:56.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.282+0000 7fcc4c205740 1 -- 172.21.15.90:0/2042827896 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5599f10ec230 msgr2=0x5599f10ee620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:56.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.283+0000 7fcc4c205740 1 --2- 172.21.15.90:0/2042827896 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5599f10ec230 0x5599f10ee620 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7fcc300efb40 tx=0x7fcc300fa450 comp rx=0 tx=0).stop 2024-04-02T14:44:56.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.283+0000 7fcc4c205740 1 -- 172.21.15.90:0/2042827896 shutdown_connections 2024-04-02T14:44:56.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.283+0000 7fcc4c205740 1 --2- 172.21.15.90:0/2042827896 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5599f10ec230 0x5599f10ee620 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:56.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.283+0000 7fcc4c205740 1 --2- 172.21.15.90:0/2042827896 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5599f10e2ec0 0x5599f10e3340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:56.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.283+0000 7fcc4c205740 1 --2- 172.21.15.90:0/2042827896 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5599f10e3ba0 0x5599f10e28f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:56.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.283+0000 7fcc4c205740 1 -- 172.21.15.90:0/2042827896 >> 172.21.15.90:0/2042827896 conn(0x5599f10a9a50 msgr2=0x5599f10dd910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:44:56.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.283+0000 7fcc43400640 1 --2- 172.21.15.90:0/3041120331 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fcc280a3da0 0x7fcc280a6260 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fcc3c009fd0 tx=0x7fcc3c0098a0 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:44:56.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.283+0000 7fcc4c205740 1 -- 172.21.15.90:0/2042827896 shutdown_connections 2024-04-02T14:44:56.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.284+0000 7fcc4c205740 1 -- 172.21.15.90:0/2042827896 wait complete. 2024-04-02T14:44:56.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.284+0000 7fcc4c205740 1 -- start start 2024-04-02T14:44:56.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.285+0000 7fcc4c205740 1 Processor -- start 2024-04-02T14:44:56.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.286+0000 7fcc4c205740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5599f10e2ec0 0x5599f10e91e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:56.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.286+0000 7fcc4c205740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5599f10e3ba0 0x5599f10e9720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:56.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.286+0000 7fcc4c205740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5599f10ec230 0x5599f10e9c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:56.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.286+0000 7fcc4c205740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5599f10436a0 con 0x5599f10e2ec0 2024-04-02T14:44:56.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.286+0000 7fcc4c205740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x5599f0fef980 con 0x5599f10ec230 2024-04-02T14:44:56.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.286+0000 7fcc4c205740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5599f1002cb0 con 0x5599f10e3ba0 2024-04-02T14:44:56.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.286+0000 7fcc4c205740 10 client.0 _ll_register_callbacks cb 0x5599f10026a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:44:56.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.286+0000 7fcc42000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5599f10e3ba0 0x5599f10e9720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:56.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.286+0000 7fcc42000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5599f10e3ba0 0x5599f10e9720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:60402/0 (socket says 172.21.15.90:60402) 2024-04-02T14:44:56.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.286+0000 7fcc42000640 1 -- 172.21.15.90:0/599718886 learned_addr learned my addr 172.21.15.90:0/599718886 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:44:56.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.286+0000 7fcc43400640 1 --2- 172.21.15.90:0/599718886 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5599f10ec230 0x5599f10e9c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:56.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.286+0000 7fcc42a00640 1 --2- 172.21.15.90:0/599718886 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5599f10e2ec0 0x5599f10e91e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:56.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.287+0000 7fcc42000640 1 -- 172.21.15.90:0/599718886 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5599f10ec230 msgr2=0x5599f10e9c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:44:56.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.287+0000 7fcc42000640 1 --2- 172.21.15.90:0/599718886 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5599f10ec230 0x5599f10e9c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:56.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.287+0000 7fcc42000640 1 -- 172.21.15.90:0/599718886 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5599f10e2ec0 msgr2=0x5599f10e91e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:44:56.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.287+0000 7fcc42000640 1 --2- 172.21.15.90:0/599718886 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5599f10e2ec0 0x5599f10e91e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:56.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.287+0000 7fcc42000640 1 -- 172.21.15.90:0/599718886 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5599f0e257d0 con 0x5599f10e3ba0 2024-04-02T14:44:56.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.287+0000 7fcc42000640 1 --2- 172.21.15.90:0/599718886 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5599f10e3ba0 0x5599f10e9720 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7fcc3002c8c0 tx=0x7fcc3002cf40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5625bec3b788cf9f server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:44:56.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.287+0000 7fcc23e00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:44:56.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.287+0000 7fcc23e00640 1 -- 172.21.15.90:0/599718886 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc3004b070 con 0x5599f10e3ba0 2024-04-02T14:44:56.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.288+0000 7fcc23e00640 1 -- 172.21.15.90:0/599718886 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8550+0+0 (secure 0 0 0) 0x7fcc300efcb0 con 0x5599f10e3ba0 2024-04-02T14:44:56.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.288+0000 7fcc4c205740 1 -- 172.21.15.90:0/599718886 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcc30004c80 con 0x5599f10e3ba0 2024-04-02T14:44:56.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.288+0000 7fcc23e00640 1 -- 172.21.15.90:0/599718886 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fcc300bded0 con 0x5599f10e3ba0 2024-04-02T14:44:56.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.288+0000 7fcc4c205740 10 client.28390 fetch_fsmap learned FSMap version 27 2024-04-02T14:44:56.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.288+0000 7fcc4c205740 1 -- 172.21.15.90:0/599718886 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5599f10fac10 con 0x5599f10e3ba0 2024-04-02T14:44:56.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.288+0000 7fcc23e00640 1 -- 172.21.15.90:0/599718886 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcc30004c80 con 0x5599f10e3ba0 2024-04-02T14:44:56.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.288+0000 7fcc23e00640 1 -- 172.21.15.90:0/599718886 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fcc300be5f0 con 0x5599f10e3ba0 2024-04-02T14:44:56.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.289+0000 7fcc4c205740 10 client.28390 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:44:56.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.289+0000 7fcc4c205740 10 client.28390 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-04-02T14:44:56.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.289+0000 7fcc4c205740 20 client.28390 populate_metadata read hostname 'smithi090' 2024-04-02T14:44:56.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.289+0000 7fcc4c205740 1 --2- 172.21.15.90:0/599718886 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x5599f10fbd90 0x5599f10fe180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:56.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.289+0000 7fcc4c205740 4 client.28390 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:44:56.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.289+0000 7fcc4c205740 1 -- 172.21.15.90:0/599718886 --> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5599f1007040 con 0x5599f10fbd90 2024-04-02T14:44:56.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.289+0000 7fcc42a00640 1 --2- 172.21.15.90:0/599718886 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x5599f10fbd90 0x5599f10fe180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:56.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.289+0000 7fcc23e00640 10 client.28390 ms_handle_connect on v2:172.21.15.92:6836/1270635939 2024-04-02T14:44:56.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.291+0000 7fcc42a00640 1 --2- 172.21.15.90:0/599718886 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x5599f10fbd90 0x5599f10fe180 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=9c5c3994e4225000 server_cookie=adf6643fde887a22 in_seq=0 out_seq=0 2024-04-02T14:44:56.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.292+0000 7fcc23e00640 1 -- 172.21.15.90:0/599718886 <== mds.3 v2:172.21.15.92:6836/1270635939 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5599f1007040 con 0x5599f10fbd90 2024-04-02T14:44:56.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.292+0000 7fcc23e00640 10 client.28390 handle_command_reply: tid=0 2024-04-02T14:44:56.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.292+0000 7fcc4c205740 1 client.28390 shutdown 2024-04-02T14:44:56.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.293+0000 7fcc4c205740 1 -- 172.21.15.90:0/599718886 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5599f10e3ba0 msgr2=0x5599f10e9720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:56.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.293+0000 7fcc4c205740 1 --2- 172.21.15.90:0/599718886 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5599f10e3ba0 0x5599f10e9720 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7fcc3002c8c0 tx=0x7fcc3002cf40 comp rx=0 tx=0).stop 2024-04-02T14:44:56.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.293+0000 7fcc4c205740 1 -- 172.21.15.90:0/599718886 shutdown_connections 2024-04-02T14:44:56.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.293+0000 7fcc4c205740 1 --2- 172.21.15.90:0/599718886 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x5599f10fbd90 0x5599f10fe180 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:56.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.293+0000 7fcc4c205740 1 --2- 172.21.15.90:0/599718886 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5599f10ec230 0x5599f10e9c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:56.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.293+0000 7fcc4c205740 1 --2- 172.21.15.90:0/599718886 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5599f10e3ba0 0x5599f10e9720 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:56.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.293+0000 7fcc4c205740 1 --2- 172.21.15.90:0/599718886 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5599f10e2ec0 0x5599f10e91e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:56.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.293+0000 7fcc4c205740 1 -- 172.21.15.90:0/599718886 >> 172.21.15.90:0/599718886 conn(0x5599f10a9a50 msgr2=0x5599f10df660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:44:56.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.293+0000 7fcc23e00640 0 client.28390 ms_handle_reset on v2:172.21.15.92:6836/1270635939 2024-04-02T14:44:56.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.293+0000 7fcc4c205740 1 -- 172.21.15.90:0/599718886 shutdown_connections 2024-04-02T14:44:56.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.294+0000 7fcc4c205740 1 -- 172.21.15.90:0/599718886 wait complete. 2024-04-02T14:44:56.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.294+0000 7fcc4c205740 20 client.28390 trim_cache size 0 max 16384 2024-04-02T14:44:56.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.321+0000 7fcc4c205740 1 Processor -- start 2024-04-02T14:44:56.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.322+0000 7fcc4c205740 1 -- start start 2024-04-02T14:44:56.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.322+0000 7fcc4c205740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5599f10e2ec0 0x5599f10f7a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:56.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.322+0000 7fcc4c205740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5599f10e3ba0 0x5599f10e9a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:56.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.322+0000 7fcc4c205740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5599f10ec230 0x5599f10df470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:56.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.322+0000 7fcc4c205740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fcc441b3b90 con 0x5599f10e2ec0 2024-04-02T14:44:56.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.322+0000 7fcc4c205740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fcc441b56d0 con 0x5599f10ec230 2024-04-02T14:44:56.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.322+0000 7fcc4c205740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fcc441b4260 con 0x5599f10e3ba0 2024-04-02T14:44:56.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.323+0000 7fcc42a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5599f10e2ec0 0x5599f10f7a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:56.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.323+0000 7fcc42a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5599f10e2ec0 0x5599f10f7a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38810/0 (socket says 172.21.15.90:38810) 2024-04-02T14:44:56.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.323+0000 7fcc42a00640 1 -- 172.21.15.90:0/2560121703 learned_addr learned my addr 172.21.15.90:0/2560121703 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:44:56.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.323+0000 7fcc42000640 1 --2- 172.21.15.90:0/2560121703 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5599f10e3ba0 0x5599f10e9a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:56.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.323+0000 7fcc43400640 1 --2- 172.21.15.90:0/2560121703 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5599f10ec230 0x5599f10df470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:56.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.324+0000 7fcc42a00640 1 -- 172.21.15.90:0/2560121703 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5599f10e3ba0 msgr2=0x5599f10e9a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:44:56.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.324+0000 7fcc42a00640 1 --2- 172.21.15.90:0/2560121703 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5599f10e3ba0 0x5599f10e9a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:56.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.324+0000 7fcc42a00640 1 -- 172.21.15.90:0/2560121703 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5599f10ec230 msgr2=0x5599f10df470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:44:56.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.324+0000 7fcc42a00640 1 --2- 172.21.15.90:0/2560121703 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5599f10ec230 0x5599f10df470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:56.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.324+0000 7fcc42a00640 1 -- 172.21.15.90:0/2560121703 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5599f10e1ef0 con 0x5599f10e2ec0 2024-04-02T14:44:56.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.326+0000 7fcc42a00640 1 --2- 172.21.15.90:0/2560121703 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5599f10e2ec0 0x5599f10f7a20 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x5599f10fe7e0 tx=0x7fcc34005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dae8684ab48ff7b7 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:44:56.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.327+0000 7fcc2ea00640 1 -- 172.21.15.90:0/2560121703 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc34004250 con 0x5599f10e2ec0 2024-04-02T14:44:56.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.327+0000 7fcc2ea00640 1 -- 172.21.15.90:0/2560121703 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcc34027070 con 0x5599f10e2ec0 2024-04-02T14:44:56.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.327+0000 7fcc2ea00640 1 -- 172.21.15.90:0/2560121703 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc34003b50 con 0x5599f10e2ec0 2024-04-02T14:44:56.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.328+0000 7fcc4c205740 1 -- 172.21.15.90:0/2560121703 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5599f10e2ec0 msgr2=0x5599f10f7a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:56.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.328+0000 7fcc4c205740 1 --2- 172.21.15.90:0/2560121703 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5599f10e2ec0 0x5599f10f7a20 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x5599f10fe7e0 tx=0x7fcc34005f90 comp rx=0 tx=0).stop 2024-04-02T14:44:56.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.329+0000 7fcc4c205740 1 -- 172.21.15.90:0/2560121703 shutdown_connections 2024-04-02T14:44:56.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.329+0000 7fcc4c205740 1 --2- 172.21.15.90:0/2560121703 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5599f10ec230 0x5599f10df470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:56.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.329+0000 7fcc4c205740 1 --2- 172.21.15.90:0/2560121703 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5599f10e3ba0 0x5599f10e9a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:56.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.329+0000 7fcc4c205740 1 --2- 172.21.15.90:0/2560121703 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5599f10e2ec0 0x5599f10f7a20 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:56.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.329+0000 7fcc4c205740 1 -- 172.21.15.90:0/2560121703 >> 172.21.15.90:0/2560121703 conn(0x5599f10a9a50 msgr2=0x5599f10ded60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:44:56.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.329+0000 7fcc4c205740 1 -- 172.21.15.90:0/2560121703 shutdown_connections 2024-04-02T14:44:56.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.329+0000 7fcc4c205740 1 -- 172.21.15.90:0/2560121703 wait complete. 2024-04-02T14:44:56.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.330+0000 7fcc4c205740 1 -- start start 2024-04-02T14:44:56.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.330+0000 7fcc4c205740 1 Processor -- start 2024-04-02T14:44:56.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.331+0000 7fcc4c205740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5599f10e3ba0 0x5599f10f4410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:56.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.331+0000 7fcc4c205740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5599f10ec230 0x5599f10e6520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:56.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.331+0000 7fcc4c205740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5599f10e6ab0 0x5599f10e6f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:56.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.331+0000 7fcc4c205740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fcc441b5000 con 0x5599f10ec230 2024-04-02T14:44:56.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.331+0000 7fcc4c205740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fcc441b4930 con 0x5599f10e3ba0 2024-04-02T14:44:56.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.331+0000 7fcc4c205740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fcc441b1b90 con 0x5599f10e6ab0 2024-04-02T14:44:56.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.331+0000 7fcc4c205740 10 client.0 _ll_register_callbacks cb 0x7fcc300fa210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:44:56.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.331+0000 7fcc42a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5599f10e3ba0 0x5599f10f4410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:56.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.332+0000 7fcc42a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5599f10e3ba0 0x5599f10f4410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:40790/0 (socket says 172.21.15.90:40790) 2024-04-02T14:44:56.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.332+0000 7fcc42a00640 1 -- 172.21.15.90:0/1975110142 learned_addr learned my addr 172.21.15.90:0/1975110142 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:44:56.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.332+0000 7fcc42000640 1 --2- 172.21.15.90:0/1975110142 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5599f10ec230 0x5599f10e6520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:56.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.332+0000 7fcc43400640 1 --2- 172.21.15.90:0/1975110142 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5599f10e6ab0 0x5599f10e6f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:56.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.333+0000 7fcc42a00640 1 -- 172.21.15.90:0/1975110142 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5599f10e6ab0 msgr2=0x5599f10e6f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:44:56.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.333+0000 7fcc42a00640 1 --2- 172.21.15.90:0/1975110142 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5599f10e6ab0 0x5599f10e6f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:56.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.333+0000 7fcc42a00640 1 -- 172.21.15.90:0/1975110142 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5599f10ec230 msgr2=0x5599f10e6520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:44:56.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.334+0000 7fcc42a00640 1 --2- 172.21.15.90:0/1975110142 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5599f10ec230 0x5599f10e6520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:56.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.334+0000 7fcc42a00640 1 -- 172.21.15.90:0/1975110142 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5599f10ecf00 con 0x5599f10e3ba0 2024-04-02T14:44:56.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.335+0000 7fcc42a00640 1 --2- 172.21.15.90:0/1975110142 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5599f10e3ba0 0x5599f10f4410 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7fcc340036d0 tx=0x7fcc34009be0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=626e2a55538368c7 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:44:56.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.335+0000 7fcc23e00640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:44:56.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.335+0000 7fcc23e00640 1 -- 172.21.15.90:0/1975110142 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc340037d0 con 0x5599f10e3ba0 2024-04-02T14:44:56.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.335+0000 7fcc23e00640 1 -- 172.21.15.90:0/1975110142 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8550+0+0 (secure 0 0 0) 0x7fcc34003970 con 0x5599f10e3ba0 2024-04-02T14:44:56.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.335+0000 7fcc4c205740 1 -- 172.21.15.90:0/1975110142 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcc34027070 con 0x5599f10e3ba0 2024-04-02T14:44:56.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.337+0000 7fcc23e00640 1 -- 172.21.15.90:0/1975110142 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fcc3403a050 con 0x5599f10e3ba0 2024-04-02T14:44:56.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.337+0000 7fcc4c205740 10 client.28574 fetch_fsmap learned FSMap version 27 2024-04-02T14:44:56.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.337+0000 7fcc4c205740 1 -- 172.21.15.90:0/1975110142 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5599f11626a0 con 0x5599f10e3ba0 2024-04-02T14:44:56.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.337+0000 7fcc23e00640 1 -- 172.21.15.90:0/1975110142 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcc34027070 con 0x5599f10e3ba0 2024-04-02T14:44:56.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.337+0000 7fcc23e00640 1 -- 172.21.15.90:0/1975110142 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fcc34035410 con 0x5599f10e3ba0 2024-04-02T14:44:56.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.338+0000 7fcc4c205740 10 client.28574 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:44:56.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.338+0000 7fcc4c205740 10 client.28574 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-04-02T14:44:56.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.338+0000 7fcc4c205740 20 client.28574 populate_metadata read hostname 'smithi090' 2024-04-02T14:44:56.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.338+0000 7fcc4c205740 1 --2- 172.21.15.90:0/1975110142 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x5599f1162df0 0x5599f1165210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:56.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.338+0000 7fcc4c205740 4 client.28574 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-02T14:44:56.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.338+0000 7fcc4c205740 1 -- 172.21.15.90:0/1975110142 --> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5599f1054c30 con 0x5599f1162df0 2024-04-02T14:44:56.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.338+0000 7fcc42000640 1 --2- 172.21.15.90:0/1975110142 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x5599f1162df0 0x5599f1165210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:56.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.339+0000 7fcc42000640 1 --2- 172.21.15.90:0/1975110142 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x5599f1162df0 0x5599f1165210 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=2cdab360e106423 server_cookie=d67dfd7f7926f1dc in_seq=0 out_seq=0 2024-04-02T14:44:56.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.339+0000 7fcc23e00640 10 client.28574 ms_handle_connect on v2:172.21.15.92:6836/1270635939 2024-04-02T14:44:56.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.339+0000 7fcc23e00640 1 -- 172.21.15.90:0/1975110142 <== mds.3 v2:172.21.15.92:6836/1270635939 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5599f1054c30 con 0x5599f1162df0 2024-04-02T14:44:56.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.339+0000 7fcc23e00640 10 client.28574 handle_command_reply: tid=0 2024-04-02T14:44:56.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.339+0000 7fcc4c205740 1 client.28574 shutdown 2024-04-02T14:44:56.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.340+0000 7fcc4c205740 1 -- 172.21.15.90:0/1975110142 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5599f10e3ba0 msgr2=0x5599f10f4410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:56.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.340+0000 7fcc4c205740 1 --2- 172.21.15.90:0/1975110142 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5599f10e3ba0 0x5599f10f4410 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7fcc340036d0 tx=0x7fcc34009be0 comp rx=0 tx=0).stop 2024-04-02T14:44:56.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.340+0000 7fcc4c205740 1 -- 172.21.15.90:0/1975110142 shutdown_connections 2024-04-02T14:44:56.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.340+0000 7fcc4c205740 1 --2- 172.21.15.90:0/1975110142 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x5599f1162df0 0x5599f1165210 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:56.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.340+0000 7fcc4c205740 1 --2- 172.21.15.90:0/1975110142 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5599f10e6ab0 0x5599f10e6f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:56.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.340+0000 7fcc4c205740 1 --2- 172.21.15.90:0/1975110142 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5599f10ec230 0x5599f10e6520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:56.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.340+0000 7fcc4c205740 1 --2- 172.21.15.90:0/1975110142 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5599f10e3ba0 0x5599f10f4410 secure :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7fcc340036d0 tx=0x7fcc34009be0 comp rx=0 tx=0).stop 2024-04-02T14:44:56.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.340+0000 7fcc4c205740 1 -- 172.21.15.90:0/1975110142 >> 172.21.15.90:0/1975110142 conn(0x5599f10a9a50 msgr2=0x5599f10dcaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:44:56.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.340+0000 7fcc23e00640 0 client.28574 ms_handle_reset on v2:172.21.15.92:6836/1270635939 2024-04-02T14:44:56.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.340+0000 7fcc4c205740 1 -- 172.21.15.90:0/1975110142 shutdown_connections 2024-04-02T14:44:56.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.341+0000 7fcc4c205740 1 -- 172.21.15.90:0/1975110142 wait complete. 2024-04-02T14:44:56.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.341+0000 7fcc4c205740 20 client.28574 trim_cache size 0 max 16384 2024-04-02T14:44:56.338 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-02T14:44:56.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.343+0000 7fcc43e00640 1 -- 172.21.15.90:0/3041120331 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fcc280a3da0 msgr2=0x7fcc280a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:56.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.343+0000 7fcc43e00640 1 --2- 172.21.15.90:0/3041120331 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fcc280a3da0 0x7fcc280a6260 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fcc3c009fd0 tx=0x7fcc3c0098a0 comp rx=0 tx=0).stop 2024-04-02T14:44:56.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.343+0000 7fcc43e00640 1 -- 172.21.15.90:0/3041120331 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcc44103450 msgr2=0x7fcc441a4b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:56.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.343+0000 7fcc43e00640 1 --2- 172.21.15.90:0/3041120331 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcc44103450 0x7fcc441a4b40 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7fcc30005940 tx=0x7fcc3002cbd0 comp rx=0 tx=0).stop 2024-04-02T14:44:56.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.343+0000 7fcc43e00640 1 -- 172.21.15.90:0/3041120331 shutdown_connections 2024-04-02T14:44:56.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.343+0000 7fcc43e00640 1 --2- 172.21.15.90:0/3041120331 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fcc280a3da0 0x7fcc280a6260 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:56.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.343+0000 7fcc43e00640 1 --2- 172.21.15.90:0/3041120331 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcc4410bf50 0x7fcc4410ab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:56.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.343+0000 7fcc43e00640 1 --2- 172.21.15.90:0/3041120331 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcc44103450 0x7fcc441a4b40 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:56.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.343+0000 7fcc43e00640 1 --2- 172.21.15.90:0/3041120331 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcc44102b30 0x7fcc441a4600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:56.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.343+0000 7fcc43e00640 1 -- 172.21.15.90:0/3041120331 >> 172.21.15.90:0/3041120331 conn(0x7fcc440fc350 msgr2=0x7fcc440fcfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:44:56.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.344+0000 7fcc43e00640 1 -- 172.21.15.90:0/3041120331 shutdown_connections 2024-04-02T14:44:56.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.344+0000 7fcc43e00640 1 -- 172.21.15.90:0/3041120331 wait complete. 2024-04-02T14:44:56.367 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-02T14:44:56.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.672+0000 7f8d62800640 1 -- 172.21.15.90:0/3462695424 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8d5c0696c0 msgr2=0x7f8d5c10e960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:56.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.672+0000 7f8d62800640 1 --2- 172.21.15.90:0/3462695424 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8d5c0696c0 0x7f8d5c10e960 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f8d4c009890 tx=0x7f8d4c02bb70 comp rx=0 tx=0).stop 2024-04-02T14:44:56.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.673+0000 7f8d62800640 1 -- 172.21.15.90:0/3462695424 shutdown_connections 2024-04-02T14:44:56.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.673+0000 7f8d62800640 1 --2- 172.21.15.90:0/3462695424 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8d5c0696c0 0x7f8d5c10e960 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:56.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.673+0000 7f8d62800640 1 --2- 172.21.15.90:0/3462695424 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8d5c068d00 0x7f8d5c069180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:56.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.673+0000 7f8d62800640 1 --2- 172.21.15.90:0/3462695424 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8d5c107b50 0x7f8d5c107f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:56.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.673+0000 7f8d62800640 1 -- 172.21.15.90:0/3462695424 >> 172.21.15.90:0/3462695424 conn(0x7f8d5c077b00 msgr2=0x7f8d5c077f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:44:56.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.673+0000 7f8d62800640 1 -- 172.21.15.90:0/3462695424 shutdown_connections 2024-04-02T14:44:56.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.673+0000 7f8d62800640 1 -- 172.21.15.90:0/3462695424 wait complete. 2024-04-02T14:44:56.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.674+0000 7f8d62800640 1 Processor -- start 2024-04-02T14:44:56.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.677+0000 7f8d62800640 1 -- start start 2024-04-02T14:44:56.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.677+0000 7f8d62800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8d5c068d00 0x7f8d5c19a380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:56.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.677+0000 7f8d62800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8d5c0696c0 0x7f8d5c19a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:56.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.677+0000 7f8d62800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8d5c107b50 0x7f8d5c19ec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:56.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.678+0000 7f8d62800640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f8d5c111ec0 con 0x7f8d5c107b50 2024-04-02T14:44:56.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.678+0000 7f8d62800640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f8d5c111d40 con 0x7f8d5c068d00 2024-04-02T14:44:56.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.678+0000 7f8d62800640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f8d5c112040 con 0x7f8d5c0696c0 2024-04-02T14:44:56.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.678+0000 7f8d5aa00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8d5c0696c0 0x7f8d5c19a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:56.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.678+0000 7f8d5aa00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8d5c0696c0 0x7f8d5c19a8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:60432/0 (socket says 172.21.15.90:60432) 2024-04-02T14:44:56.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.678+0000 7f8d5aa00640 1 -- 172.21.15.90:0/961049032 learned_addr learned my addr 172.21.15.90:0/961049032 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:44:56.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.678+0000 7f8d5b400640 1 --2- 172.21.15.90:0/961049032 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8d5c068d00 0x7f8d5c19a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:56.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.679+0000 7f8d5aa00640 1 -- 172.21.15.90:0/961049032 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8d5c068d00 msgr2=0x7f8d5c19a380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:56.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.680+0000 7f8d5aa00640 1 --2- 172.21.15.90:0/961049032 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8d5c068d00 0x7f8d5c19a380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:56.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.680+0000 7f8d5aa00640 1 -- 172.21.15.90:0/961049032 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8d5c107b50 msgr2=0x7f8d5c19ec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:56.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.680+0000 7f8d5aa00640 1 --2- 172.21.15.90:0/961049032 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8d5c107b50 0x7f8d5c19ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:56.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.680+0000 7f8d5aa00640 1 -- 172.21.15.90:0/961049032 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d5c19f370 con 0x7f8d5c0696c0 2024-04-02T14:44:56.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.680+0000 7f8d5b400640 1 --2- 172.21.15.90:0/961049032 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8d5c068d00 0x7f8d5c19a380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:44:56.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.680+0000 7f8d5aa00640 1 --2- 172.21.15.90:0/961049032 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8d5c0696c0 0x7f8d5c19a8c0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f8d4400b4a0 tx=0x7f8d4400b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:44:56.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.681+0000 7f8d53e00640 1 -- 172.21.15.90:0/961049032 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d44014070 con 0x7f8d5c0696c0 2024-04-02T14:44:56.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.681+0000 7f8d53e00640 1 -- 172.21.15.90:0/961049032 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8d44010af0 con 0x7f8d5c0696c0 2024-04-02T14:44:56.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.681+0000 7f8d53e00640 1 -- 172.21.15.90:0/961049032 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d44017d10 con 0x7f8d5c0696c0 2024-04-02T14:44:56.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.681+0000 7f8d62800640 1 -- 172.21.15.90:0/961049032 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d5c19f660 con 0x7f8d5c0696c0 2024-04-02T14:44:56.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.681+0000 7f8d62800640 1 -- 172.21.15.90:0/961049032 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d5c1a6ef0 con 0x7f8d5c0696c0 2024-04-02T14:44:56.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.686+0000 7f8d53e00640 1 -- 172.21.15.90:0/961049032 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f8d44017070 con 0x7f8d5c0696c0 2024-04-02T14:44:56.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.688+0000 7f8d53e00640 1 --2- 172.21.15.90:0/961049032 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f8d400a3da0 0x7f8d400a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:56.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.688+0000 7f8d53e00640 1 -- 172.21.15.90:0/961049032 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8550+0+0 (secure 0 0 0) 0x7f8d440cfa90 con 0x7f8d5c0696c0 2024-04-02T14:44:56.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.692+0000 7f8d65418740 1 Processor -- start 2024-04-02T14:44:56.690 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.694+0000 7f8d5b400640 1 --2- 172.21.15.90:0/961049032 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f8d400a3da0 0x7f8d400a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:56.690 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.694+0000 7f8d65418740 1 -- start start 2024-04-02T14:44:56.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.696+0000 7f8d65418740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5590d8d3f9f0 0x5590d8d3e740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:56.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.697+0000 7f8d65418740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5590d8d3ed10 0x5590d8d3f190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:56.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.697+0000 7f8d65418740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5590d8d48080 0x5590d8d4a470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:56.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.697+0000 7f8d65418740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5590d8c7f400 con 0x5590d8d3f9f0 2024-04-02T14:44:56.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.697+0000 7f8d65418740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x5590d8c63f10 con 0x5590d8d48080 2024-04-02T14:44:56.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.697+0000 7f8d65418740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5590d8ca36f0 con 0x5590d8d3ed10 2024-04-02T14:44:56.694 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.697+0000 7f8d5aa00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5590d8d3ed10 0x5590d8d3f190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:56.694 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.697+0000 7f8d5be00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5590d8d48080 0x5590d8d4a470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:56.694 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.698+0000 7f8d5aa00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5590d8d3ed10 0x5590d8d3f190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:60440/0 (socket says 172.21.15.90:60440) 2024-04-02T14:44:56.694 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.698+0000 7f8d5be00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5590d8d48080 0x5590d8d4a470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:40832/0 (socket says 172.21.15.90:40832) 2024-04-02T14:44:56.694 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.698+0000 7f8d5aa00640 1 -- 172.21.15.90:0/469587924 learned_addr learned my addr 172.21.15.90:0/469587924 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:44:56.694 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.698+0000 7f8d5b400640 1 --2- 172.21.15.90:0/961049032 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f8d400a3da0 0x7f8d400a6260 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f8d54005fd0 tx=0x7f8d540095c0 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:44:56.694 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.698+0000 7f8d5b400640 1 --2- 172.21.15.90:0/469587924 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5590d8d3f9f0 0x5590d8d3e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:56.695 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.699+0000 7f8d5aa00640 1 -- 172.21.15.90:0/469587924 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5590d8d48080 msgr2=0x5590d8d4a470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:44:56.695 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.699+0000 7f8d5aa00640 1 --2- 172.21.15.90:0/469587924 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5590d8d48080 0x5590d8d4a470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:56.695 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.699+0000 7f8d5aa00640 1 -- 172.21.15.90:0/469587924 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5590d8d3f9f0 msgr2=0x5590d8d3e740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:44:56.695 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.699+0000 7f8d5aa00640 1 --2- 172.21.15.90:0/469587924 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5590d8d3f9f0 0x5590d8d3e740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:56.695 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.699+0000 7f8d5aa00640 1 -- 172.21.15.90:0/469587924 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5590d89b73f0 con 0x5590d8d3ed10 2024-04-02T14:44:56.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.700+0000 7f8d5aa00640 1 --2- 172.21.15.90:0/469587924 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5590d8d3ed10 0x5590d8d3f190 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f8d4409bab0 tx=0x7f8d440047d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=addd4ce200529edc server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:44:56.697 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.701+0000 7f8d50c00640 1 -- 172.21.15.90:0/469587924 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d4401e050 con 0x5590d8d3ed10 2024-04-02T14:44:56.697 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.701+0000 7f8d50c00640 1 -- 172.21.15.90:0/469587924 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8d4409ce50 con 0x5590d8d3ed10 2024-04-02T14:44:56.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.701+0000 7f8d50c00640 1 -- 172.21.15.90:0/469587924 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d440cd440 con 0x5590d8d3ed10 2024-04-02T14:44:56.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.702+0000 7f8d65418740 1 -- 172.21.15.90:0/469587924 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5590d8d3ed10 msgr2=0x5590d8d3f190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:56.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.702+0000 7f8d65418740 1 --2- 172.21.15.90:0/469587924 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5590d8d3ed10 0x5590d8d3f190 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f8d4409bab0 tx=0x7f8d440047d0 comp rx=0 tx=0).stop 2024-04-02T14:44:56.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.703+0000 7f8d65418740 1 -- 172.21.15.90:0/469587924 shutdown_connections 2024-04-02T14:44:56.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.703+0000 7f8d65418740 1 --2- 172.21.15.90:0/469587924 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5590d8d48080 0x5590d8d4a470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:56.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.703+0000 7f8d65418740 1 --2- 172.21.15.90:0/469587924 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5590d8d3ed10 0x5590d8d3f190 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:56.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.703+0000 7f8d65418740 1 --2- 172.21.15.90:0/469587924 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5590d8d3f9f0 0x5590d8d3e740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:56.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.703+0000 7f8d65418740 1 -- 172.21.15.90:0/469587924 >> 172.21.15.90:0/469587924 conn(0x5590d8d05a50 msgr2=0x5590d8d39760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:44:56.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.705+0000 7f8d65418740 1 -- 172.21.15.90:0/469587924 shutdown_connections 2024-04-02T14:44:56.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.705+0000 7f8d65418740 1 -- 172.21.15.90:0/469587924 wait complete. 2024-04-02T14:44:56.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.706+0000 7f8d65418740 1 -- start start 2024-04-02T14:44:56.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.707+0000 7f8d65418740 1 Processor -- start 2024-04-02T14:44:56.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.707+0000 7f8d65418740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5590d8d3f9f0 0x5590d8d479a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:56.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.707+0000 7f8d65418740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5590d8d48080 0x5590d8d44c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:56.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.707+0000 7f8d65418740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5590d8d451d0 0x5590d8d45630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:56.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.708+0000 7f8d5b400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5590d8d3f9f0 0x5590d8d479a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:56.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.708+0000 7f8d5b400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5590d8d3f9f0 0x5590d8d479a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38844/0 (socket says 172.21.15.90:38844) 2024-04-02T14:44:56.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.708+0000 7f8d65418740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5590d8c9f6a0 con 0x5590d8d3f9f0 2024-04-02T14:44:56.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.708+0000 7f8d65418740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x5590d8c4b980 con 0x5590d8d48080 2024-04-02T14:44:56.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.708+0000 7f8d5aa00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5590d8d48080 0x5590d8d44c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:56.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.708+0000 7f8d5aa00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5590d8d48080 0x5590d8d44c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:40840/0 (socket says 172.21.15.90:40840) 2024-04-02T14:44:56.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.708+0000 7f8d5aa00640 1 -- 172.21.15.90:0/218412270 learned_addr learned my addr 172.21.15.90:0/218412270 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:44:56.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.709+0000 7f8d5be00640 1 --2- 172.21.15.90:0/218412270 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5590d8d451d0 0x5590d8d45630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:56.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.709+0000 7f8d65418740 1 -- 172.21.15.90:0/218412270 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5590d8c5ecb0 con 0x5590d8d451d0 2024-04-02T14:44:56.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.709+0000 7f8d65418740 10 client.0 _ll_register_callbacks cb 0x5590d8c5e6a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:44:56.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.710+0000 7f8d5be00640 1 -- 172.21.15.90:0/218412270 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5590d8d48080 msgr2=0x5590d8d44c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:44:56.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.710+0000 7f8d5be00640 1 --2- 172.21.15.90:0/218412270 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5590d8d48080 0x5590d8d44c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:56.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.710+0000 7f8d5be00640 1 -- 172.21.15.90:0/218412270 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5590d8d3f9f0 msgr2=0x5590d8d479a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:44:56.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.710+0000 7f8d5be00640 1 --2- 172.21.15.90:0/218412270 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5590d8d3f9f0 0x5590d8d479a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:56.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.710+0000 7f8d5be00640 1 -- 172.21.15.90:0/218412270 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5590d8a817d0 con 0x5590d8d451d0 2024-04-02T14:44:56.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.711+0000 7f8d5be00640 1 --2- 172.21.15.90:0/218412270 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5590d8d451d0 0x5590d8d45630 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f8d4c0058c0 tx=0x7f8d4c034cb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4231b4b391d760e2 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:44:56.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.712+0000 7f8d49600640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:44:56.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.712+0000 7f8d49600640 1 -- 172.21.15.90:0/218412270 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d4c043070 con 0x5590d8d451d0 2024-04-02T14:44:56.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.712+0000 7f8d49600640 1 -- 172.21.15.90:0/218412270 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8550+0+0 (secure 0 0 0) 0x7f8d4c004450 con 0x5590d8d451d0 2024-04-02T14:44:56.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.712+0000 7f8d65418740 1 -- 172.21.15.90:0/218412270 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8d4409ce50 con 0x5590d8d451d0 2024-04-02T14:44:56.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.713+0000 7f8d49600640 1 -- 172.21.15.90:0/218412270 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f8d4c04d050 con 0x5590d8d451d0 2024-04-02T14:44:56.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.713+0000 7f8d65418740 10 client.28408 fetch_fsmap learned FSMap version 27 2024-04-02T14:44:56.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.714+0000 7f8d65418740 1 -- 172.21.15.90:0/218412270 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5590d8d57040 con 0x5590d8d451d0 2024-04-02T14:44:56.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.715+0000 7f8d49600640 1 -- 172.21.15.90:0/218412270 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8d4409ce50 con 0x5590d8d451d0 2024-04-02T14:44:56.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.715+0000 7f8d49600640 1 -- 172.21.15.90:0/218412270 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f8d4c02c900 con 0x5590d8d451d0 2024-04-02T14:44:56.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.715+0000 7f8d65418740 10 client.28408 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:44:56.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.715+0000 7f8d65418740 10 client.28408 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-04-02T14:44:56.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.715+0000 7f8d65418740 20 client.28408 populate_metadata read hostname 'smithi090' 2024-04-02T14:44:56.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.715+0000 7f8d65418740 1 --2- 172.21.15.90:0/218412270 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x5590d8d58500 0x5590d8d5a910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:56.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.715+0000 7f8d65418740 4 client.28408 mds_command: new command op to 24547 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:44:56.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.715+0000 7f8d65418740 1 -- 172.21.15.90:0/218412270 --> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5590d8c63040 con 0x5590d8d58500 2024-04-02T14:44:56.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.716+0000 7f8d5b400640 1 --2- 172.21.15.90:0/218412270 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x5590d8d58500 0x5590d8d5a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:56.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.718+0000 7f8d5b400640 1 --2- 172.21.15.90:0/218412270 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x5590d8d58500 0x5590d8d5a910 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=68d0b8be1ae054ae server_cookie=498b73fe9eac151a in_seq=0 out_seq=0 2024-04-02T14:44:56.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.718+0000 7f8d49600640 10 client.28408 ms_handle_connect on v2:172.21.15.177:6838/664895663 2024-04-02T14:44:56.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.719+0000 7f8d49600640 1 -- 172.21.15.90:0/218412270 <== mds.2 v2:172.21.15.177:6838/664895663 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5590d8c63040 con 0x5590d8d58500 2024-04-02T14:44:56.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.719+0000 7f8d49600640 10 client.28408 handle_command_reply: tid=0 2024-04-02T14:44:56.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.719+0000 7f8d65418740 1 client.28408 shutdown 2024-04-02T14:44:56.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.720+0000 7f8d65418740 1 -- 172.21.15.90:0/218412270 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5590d8d451d0 msgr2=0x5590d8d45630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:56.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.720+0000 7f8d65418740 1 --2- 172.21.15.90:0/218412270 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5590d8d451d0 0x5590d8d45630 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f8d4c0058c0 tx=0x7f8d4c034cb0 comp rx=0 tx=0).stop 2024-04-02T14:44:56.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.721+0000 7f8d65418740 1 -- 172.21.15.90:0/218412270 shutdown_connections 2024-04-02T14:44:56.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.721+0000 7f8d65418740 1 --2- 172.21.15.90:0/218412270 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x5590d8d58500 0x5590d8d5a910 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:56.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.721+0000 7f8d65418740 1 --2- 172.21.15.90:0/218412270 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5590d8d451d0 0x5590d8d45630 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:56.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.721+0000 7f8d65418740 1 --2- 172.21.15.90:0/218412270 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5590d8d48080 0x5590d8d44c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:56.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.721+0000 7f8d65418740 1 --2- 172.21.15.90:0/218412270 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5590d8d3f9f0 0x5590d8d479a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:56.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.721+0000 7f8d65418740 1 -- 172.21.15.90:0/218412270 >> 172.21.15.90:0/218412270 conn(0x5590d8d05a50 msgr2=0x5590d8d3b4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:44:56.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.721+0000 7f8d49600640 0 client.28408 ms_handle_reset on v2:172.21.15.177:6838/664895663 2024-04-02T14:44:56.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.721+0000 7f8d65418740 1 -- 172.21.15.90:0/218412270 shutdown_connections 2024-04-02T14:44:56.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.721+0000 7f8d65418740 1 -- 172.21.15.90:0/218412270 wait complete. 2024-04-02T14:44:56.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.721+0000 7f8d65418740 20 client.28408 trim_cache size 0 max 16384 2024-04-02T14:44:56.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.755+0000 7f8d65418740 1 Processor -- start 2024-04-02T14:44:56.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.755+0000 7f8d65418740 1 -- start start 2024-04-02T14:44:56.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.755+0000 7f8d65418740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5590d8d3f9f0 0x5590d8d53b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:56.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.755+0000 7f8d65418740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5590d8d451d0 0x5590d8d43b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:56.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.755+0000 7f8d65418740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5590d8d48080 0x5590d8d49470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:56.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.755+0000 7f8d65418740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f8d5c1afa70 con 0x5590d8d48080 2024-04-02T14:44:56.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.755+0000 7f8d65418740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f8d5c1b15b0 con 0x5590d8d451d0 2024-04-02T14:44:56.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.755+0000 7f8d65418740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f8d5c1b0140 con 0x5590d8d3f9f0 2024-04-02T14:44:56.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.756+0000 7f8d5b400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5590d8d3f9f0 0x5590d8d53b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:56.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.756+0000 7f8d5b400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5590d8d3f9f0 0x5590d8d53b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:60466/0 (socket says 172.21.15.90:60466) 2024-04-02T14:44:56.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.756+0000 7f8d5b400640 1 -- 172.21.15.90:0/667048707 learned_addr learned my addr 172.21.15.90:0/667048707 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:44:56.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.756+0000 7f8d5aa00640 1 --2- 172.21.15.90:0/667048707 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5590d8d451d0 0x5590d8d43b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:56.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.756+0000 7f8d5be00640 1 --2- 172.21.15.90:0/667048707 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5590d8d48080 0x5590d8d49470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:56.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.757+0000 7f8d5b400640 1 -- 172.21.15.90:0/667048707 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5590d8d451d0 msgr2=0x5590d8d43b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:44:56.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.757+0000 7f8d5b400640 1 --2- 172.21.15.90:0/667048707 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5590d8d451d0 0x5590d8d43b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:56.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.757+0000 7f8d5b400640 1 -- 172.21.15.90:0/667048707 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5590d8d48080 msgr2=0x5590d8d49470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:44:56.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.757+0000 7f8d5b400640 1 --2- 172.21.15.90:0/667048707 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5590d8d48080 0x5590d8d49470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:56.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.757+0000 7f8d5b400640 1 -- 172.21.15.90:0/667048707 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5590d8d39ca0 con 0x5590d8d3f9f0 2024-04-02T14:44:56.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.758+0000 7f8d5b400640 1 --2- 172.21.15.90:0/667048707 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5590d8d3f9f0 0x5590d8d53b30 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x5590d8d5b150 tx=0x7f8d54013000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8bf05d35732fad0d server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:44:56.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.759+0000 7f8d4be00640 1 -- 172.21.15.90:0/667048707 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d54019b10 con 0x5590d8d3f9f0 2024-04-02T14:44:56.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.759+0000 7f8d4be00640 1 -- 172.21.15.90:0/667048707 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8d54019cb0 con 0x5590d8d3f9f0 2024-04-02T14:44:56.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.759+0000 7f8d4be00640 1 -- 172.21.15.90:0/667048707 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d54027760 con 0x5590d8d3f9f0 2024-04-02T14:44:56.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.760+0000 7f8d65418740 1 -- 172.21.15.90:0/667048707 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5590d8d3f9f0 msgr2=0x5590d8d53b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:56.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.760+0000 7f8d65418740 1 --2- 172.21.15.90:0/667048707 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5590d8d3f9f0 0x5590d8d53b30 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x5590d8d5b150 tx=0x7f8d54013000 comp rx=0 tx=0).stop 2024-04-02T14:44:56.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.761+0000 7f8d65418740 1 -- 172.21.15.90:0/667048707 shutdown_connections 2024-04-02T14:44:56.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.761+0000 7f8d65418740 1 --2- 172.21.15.90:0/667048707 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5590d8d48080 0x5590d8d49470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:56.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.761+0000 7f8d65418740 1 --2- 172.21.15.90:0/667048707 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5590d8d451d0 0x5590d8d43b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:56.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.761+0000 7f8d65418740 1 --2- 172.21.15.90:0/667048707 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5590d8d3f9f0 0x5590d8d53b30 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:56.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.761+0000 7f8d65418740 1 -- 172.21.15.90:0/667048707 >> 172.21.15.90:0/667048707 conn(0x5590d8d05a50 msgr2=0x5590d8d43430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:44:56.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.761+0000 7f8d65418740 1 -- 172.21.15.90:0/667048707 shutdown_connections 2024-04-02T14:44:56.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.761+0000 7f8d65418740 1 -- 172.21.15.90:0/667048707 wait complete. 2024-04-02T14:44:56.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.762+0000 7f8d65418740 1 -- start start 2024-04-02T14:44:56.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.762+0000 7f8d65418740 1 Processor -- start 2024-04-02T14:44:56.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.763+0000 7f8d65418740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5590d8d3f9f0 0x5590d8d50500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:56.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.763+0000 7f8d65418740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5590d8d451d0 0x5590d8d42210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:56.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.763+0000 7f8d65418740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5590d8d48080 0x5590d8d42840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:56.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.763+0000 7f8d65418740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f8d5c1b0ee0 con 0x5590d8d3f9f0 2024-04-02T14:44:56.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.763+0000 7f8d65418740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f8d5c1b0810 con 0x5590d8d451d0 2024-04-02T14:44:56.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.763+0000 7f8d5aa00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5590d8d451d0 0x5590d8d42210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:56.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.763+0000 7f8d5aa00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5590d8d451d0 0x5590d8d42210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:40862/0 (socket says 172.21.15.90:40862) 2024-04-02T14:44:56.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.763+0000 7f8d5aa00640 1 -- 172.21.15.90:0/1371991620 learned_addr learned my addr 172.21.15.90:0/1371991620 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:44:56.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.763+0000 7f8d65418740 1 -- 172.21.15.90:0/1371991620 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f8d5c1ada70 con 0x5590d8d48080 2024-04-02T14:44:56.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.763+0000 7f8d5b400640 1 --2- 172.21.15.90:0/1371991620 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5590d8d3f9f0 0x5590d8d50500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:56.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.763+0000 7f8d5be00640 1 --2- 172.21.15.90:0/1371991620 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5590d8d48080 0x5590d8d42840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:56.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.763+0000 7f8d65418740 10 client.0 _ll_register_callbacks cb 0x7f8d4c0348b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:44:56.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.763+0000 7f8d5be00640 1 -- 172.21.15.90:0/1371991620 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5590d8d451d0 msgr2=0x5590d8d42210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:44:56.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.763+0000 7f8d5be00640 1 --2- 172.21.15.90:0/1371991620 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5590d8d451d0 0x5590d8d42210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:56.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.763+0000 7f8d5be00640 1 -- 172.21.15.90:0/1371991620 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5590d8d3f9f0 msgr2=0x5590d8d50500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:44:56.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.763+0000 7f8d5be00640 1 --2- 172.21.15.90:0/1371991620 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5590d8d3f9f0 0x5590d8d50500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:56.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.763+0000 7f8d5be00640 1 -- 172.21.15.90:0/1371991620 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5590d8d4a4e0 con 0x5590d8d48080 2024-04-02T14:44:56.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.764+0000 7f8d65418740 1 -- 172.21.15.90:0/1371991620 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8d54019cb0 con 0x5590d8d48080 2024-04-02T14:44:56.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.764+0000 7f8d5be00640 1 --2- 172.21.15.90:0/1371991620 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5590d8d48080 0x5590d8d42840 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f8d4c02c0b0 tx=0x7f8d4c005270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6f4ac662ef23bc05 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:44:56.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.764+0000 7f8d49600640 10 client.28414 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:44:56.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.764+0000 7f8d49600640 1 -- 172.21.15.90:0/1371991620 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d4c043050 con 0x5590d8d48080 2024-04-02T14:44:56.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.764+0000 7f8d49600640 1 -- 172.21.15.90:0/1371991620 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8550+0+0 (secure 0 0 0) 0x7f8d4c02c510 con 0x5590d8d48080 2024-04-02T14:44:56.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.765+0000 7f8d49600640 1 -- 172.21.15.90:0/1371991620 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f8d4c039040 con 0x5590d8d48080 2024-04-02T14:44:56.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.765+0000 7f8d65418740 10 client.28414 fetch_fsmap learned FSMap version 27 2024-04-02T14:44:56.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.765+0000 7f8d65418740 1 -- 172.21.15.90:0/1371991620 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({fsmap=27}) -- 0x5590d8dc7380 con 0x5590d8d48080 2024-04-02T14:44:56.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.765+0000 7f8d49600640 1 -- 172.21.15.90:0/1371991620 <== mon.1 v2:172.21.15.177:3300/0 4 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f8d4c02c790 con 0x5590d8d48080 2024-04-02T14:44:56.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.765+0000 7f8d65418740 10 client.28414 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:44:56.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.765+0000 7f8d65418740 10 client.28414 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-04-02T14:44:56.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.765+0000 7f8d65418740 20 client.28414 populate_metadata read hostname 'smithi090' 2024-04-02T14:44:56.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.765+0000 7f8d65418740 1 --2- 172.21.15.90:0/1371991620 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x5590d8dc7930 0x5590d8dc9d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:56.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.765+0000 7f8d65418740 4 client.28414 mds_command: new command op to 24547 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-02T14:44:56.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.765+0000 7f8d65418740 1 -- 172.21.15.90:0/1371991620 --> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5590d8cb0c30 con 0x5590d8dc7930 2024-04-02T14:44:56.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.766+0000 7f8d5b400640 1 --2- 172.21.15.90:0/1371991620 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x5590d8dc7930 0x5590d8dc9d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:56.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.766+0000 7f8d5b400640 1 --2- 172.21.15.90:0/1371991620 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x5590d8dc7930 0x5590d8dc9d70 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=cd25715a75d77694 server_cookie=9d134a184680613f in_seq=0 out_seq=0 2024-04-02T14:44:56.764 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.766+0000 7f8d49600640 10 client.28414 ms_handle_connect on v2:172.21.15.177:6838/664895663 2024-04-02T14:44:56.764 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.766+0000 7f8d49600640 1 -- 172.21.15.90:0/1371991620 <== mds.2 v2:172.21.15.177:6838/664895663 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5590d8cb0c30 con 0x5590d8dc7930 2024-04-02T14:44:56.764 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.766+0000 7f8d49600640 10 client.28414 handle_command_reply: tid=0 2024-04-02T14:44:56.764 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.766+0000 7f8d65418740 1 client.28414 shutdown 2024-04-02T14:44:56.764 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.767+0000 7f8d65418740 1 -- 172.21.15.90:0/1371991620 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5590d8d48080 msgr2=0x5590d8d42840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:56.764 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.767+0000 7f8d65418740 1 --2- 172.21.15.90:0/1371991620 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5590d8d48080 0x5590d8d42840 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f8d4c02c0b0 tx=0x7f8d4c005270 comp rx=0 tx=0).stop 2024-04-02T14:44:56.764 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.767+0000 7f8d65418740 1 -- 172.21.15.90:0/1371991620 shutdown_connections 2024-04-02T14:44:56.764 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.767+0000 7f8d65418740 1 --2- 172.21.15.90:0/1371991620 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x5590d8dc7930 0x5590d8dc9d70 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:56.764 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.767+0000 7f8d65418740 1 --2- 172.21.15.90:0/1371991620 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5590d8d48080 0x5590d8d42840 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:56.764 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.767+0000 7f8d65418740 1 --2- 172.21.15.90:0/1371991620 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5590d8d451d0 0x5590d8d42210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:56.764 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.767+0000 7f8d65418740 1 --2- 172.21.15.90:0/1371991620 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5590d8d3f9f0 0x5590d8d50500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:56.764 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.767+0000 7f8d65418740 1 -- 172.21.15.90:0/1371991620 >> 172.21.15.90:0/1371991620 conn(0x5590d8d05a50 msgr2=0x5590d8d388a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:44:56.764 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.767+0000 7f8d49600640 0 client.28414 ms_handle_reset on v2:172.21.15.177:6838/664895663 2024-04-02T14:44:56.764 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.767+0000 7f8d65418740 1 -- 172.21.15.90:0/1371991620 shutdown_connections 2024-04-02T14:44:56.764 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.767+0000 7f8d65418740 1 -- 172.21.15.90:0/1371991620 wait complete. 2024-04-02T14:44:56.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.767+0000 7f8d65418740 20 client.28414 trim_cache size 0 max 16384 2024-04-02T14:44:56.765 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-02T14:44:56.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.769+0000 7f8d62800640 1 -- 172.21.15.90:0/961049032 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f8d400a3da0 msgr2=0x7f8d400a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:56.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.769+0000 7f8d62800640 1 --2- 172.21.15.90:0/961049032 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f8d400a3da0 0x7f8d400a6260 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f8d54005fd0 tx=0x7f8d540095c0 comp rx=0 tx=0).stop 2024-04-02T14:44:56.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.769+0000 7f8d62800640 1 -- 172.21.15.90:0/961049032 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8d5c0696c0 msgr2=0x7f8d5c19a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:56.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.769+0000 7f8d62800640 1 --2- 172.21.15.90:0/961049032 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8d5c0696c0 0x7f8d5c19a8c0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f8d4400b4a0 tx=0x7f8d4400b960 comp rx=0 tx=0).stop 2024-04-02T14:44:56.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.769+0000 7f8d62800640 1 -- 172.21.15.90:0/961049032 shutdown_connections 2024-04-02T14:44:56.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.769+0000 7f8d62800640 1 --2- 172.21.15.90:0/961049032 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f8d400a3da0 0x7f8d400a6260 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:56.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.769+0000 7f8d62800640 1 --2- 172.21.15.90:0/961049032 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8d5c107b50 0x7f8d5c19ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:56.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.769+0000 7f8d62800640 1 --2- 172.21.15.90:0/961049032 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8d5c0696c0 0x7f8d5c19a8c0 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:56.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.769+0000 7f8d62800640 1 --2- 172.21.15.90:0/961049032 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8d5c068d00 0x7f8d5c19a380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:56.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.769+0000 7f8d62800640 1 -- 172.21.15.90:0/961049032 >> 172.21.15.90:0/961049032 conn(0x7f8d5c077b00 msgr2=0x7f8d5c0fe590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:44:56.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.769+0000 7f8d62800640 1 -- 172.21.15.90:0/961049032 shutdown_connections 2024-04-02T14:44:56.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:56.769+0000 7f8d62800640 1 -- 172.21.15.90:0/961049032 wait complete. 2024-04-02T14:44:56.787 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-02T14:44:56.826 INFO:teuthology.orchestra.run.smithi090.stderr:progress: 565.0 s, 2111.8 tps, lat 47.794 ms stddev 68.481 2024-04-02T14:44:57.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.068+0000 7f13d4c00640 1 -- 172.21.15.90:0/3957500164 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f13d01094c0 msgr2=0x7f13d01153b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:57.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.068+0000 7f13d4c00640 1 --2- 172.21.15.90:0/3957500164 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f13d01094c0 0x7f13d01153b0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f13c800b710 tx=0x7f13c802c760 comp rx=0 tx=0).stop 2024-04-02T14:44:57.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.068+0000 7f13d4c00640 1 -- 172.21.15.90:0/3957500164 shutdown_connections 2024-04-02T14:44:57.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.068+0000 7f13d4c00640 1 --2- 172.21.15.90:0/3957500164 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f13d01094c0 0x7f13d01153b0 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:57.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.069+0000 7f13d4c00640 1 --2- 172.21.15.90:0/3957500164 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f13d0108b00 0x7f13d0108f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:57.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.069+0000 7f13d4c00640 1 --2- 172.21.15.90:0/3957500164 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f13d010e630 0x7f13d010ea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:57.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.069+0000 7f13d4c00640 1 -- 172.21.15.90:0/3957500164 >> 172.21.15.90:0/3957500164 conn(0x7f13d006cbe0 msgr2=0x7f13d006cff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:44:57.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.070+0000 7f13d4c00640 1 -- 172.21.15.90:0/3957500164 shutdown_connections 2024-04-02T14:44:57.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.071+0000 7f13d4c00640 1 -- 172.21.15.90:0/3957500164 wait complete. 2024-04-02T14:44:57.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.072+0000 7f13d4c00640 1 Processor -- start 2024-04-02T14:44:57.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.072+0000 7f13d4c00640 1 -- start start 2024-04-02T14:44:57.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.073+0000 7f13d4c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f13d0108b00 0x7f13d01a5690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:57.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.073+0000 7f13d4c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f13d01094c0 0x7f13d01a5bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:57.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.073+0000 7f13d4c00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f13d010e630 0x7f13d019e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:57.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.073+0000 7f13d4c00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f13d01184a0 con 0x7f13d0108b00 2024-04-02T14:44:57.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.073+0000 7f13d4c00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f13d0118320 con 0x7f13d010e630 2024-04-02T14:44:57.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.073+0000 7f13d4c00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f13d0118620 con 0x7f13d01094c0 2024-04-02T14:44:57.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.074+0000 7f13cde00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f13d0108b00 0x7f13d01a5690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:57.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.074+0000 7f13cde00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f13d0108b00 0x7f13d01a5690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38902/0 (socket says 172.21.15.90:38902) 2024-04-02T14:44:57.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.074+0000 7f13cde00640 1 -- 172.21.15.90:0/1509930049 learned_addr learned my addr 172.21.15.90:0/1509930049 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:44:57.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.074+0000 7f13cd400640 1 --2- 172.21.15.90:0/1509930049 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f13d01094c0 0x7f13d01a5bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:57.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.075+0000 7f13cd400640 1 -- 172.21.15.90:0/1509930049 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f13d010e630 msgr2=0x7f13d019e8f0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-02T14:44:57.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.075+0000 7f13cd400640 1 --2- 172.21.15.90:0/1509930049 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f13d010e630 0x7f13d019e8f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:57.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.075+0000 7f13cd400640 1 -- 172.21.15.90:0/1509930049 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f13d0108b00 msgr2=0x7f13d01a5690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:57.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.075+0000 7f13cd400640 1 --2- 172.21.15.90:0/1509930049 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f13d0108b00 0x7f13d01a5690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:57.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.075+0000 7f13cd400640 1 -- 172.21.15.90:0/1509930049 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13d019f1b0 con 0x7f13d01094c0 2024-04-02T14:44:57.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.075+0000 7f13cd400640 1 --2- 172.21.15.90:0/1509930049 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f13d01094c0 0x7f13d01a5bd0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f13b8009fd0 tx=0x7f13b800ec20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:44:57.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.076+0000 7f13c6a00640 1 -- 172.21.15.90:0/1509930049 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13b800ccc0 con 0x7f13d01094c0 2024-04-02T14:44:57.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.076+0000 7f13d4c00640 1 -- 172.21.15.90:0/1509930049 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13d019f4a0 con 0x7f13d01094c0 2024-04-02T14:44:57.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.076+0000 7f13c6a00640 1 -- 172.21.15.90:0/1509930049 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f13b800ee70 con 0x7f13d01094c0 2024-04-02T14:44:57.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.076+0000 7f13c6a00640 1 -- 172.21.15.90:0/1509930049 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13b8004a00 con 0x7f13d01094c0 2024-04-02T14:44:57.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.076+0000 7f13d4c00640 1 -- 172.21.15.90:0/1509930049 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13d01a3440 con 0x7f13d01094c0 2024-04-02T14:44:57.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.082+0000 7f13c6a00640 1 -- 172.21.15.90:0/1509930049 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f13b8004ba0 con 0x7f13d01094c0 2024-04-02T14:44:57.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.084+0000 7f13d786d740 1 Processor -- start 2024-04-02T14:44:57.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.085+0000 7f13d786d740 1 -- start start 2024-04-02T14:44:57.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.085+0000 7f13c6a00640 1 --2- 172.21.15.90:0/1509930049 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f13b40a3f20 0x7f13b40a63e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:57.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.086+0000 7f13c6a00640 1 -- 172.21.15.90:0/1509930049 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8550+0+0 (secure 0 0 0) 0x7f13b80047a0 con 0x7f13d01094c0 2024-04-02T14:44:57.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.086+0000 7f13ce800640 1 --2- 172.21.15.90:0/1509930049 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f13b40a3f20 0x7f13b40a63e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:57.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.087+0000 7f13d786d740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fd682bd9f0 0x55fd682bc740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:57.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.087+0000 7f13d786d740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fd682bcd10 0x55fd682bd190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:57.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.087+0000 7f13d786d740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fd682c6080 0x55fd682c8470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:57.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.087+0000 7f13d786d740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55fd681fd400 con 0x55fd682c6080 2024-04-02T14:44:57.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.087+0000 7f13d786d740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55fd681e1f10 con 0x55fd682bd9f0 2024-04-02T14:44:57.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.087+0000 7f13d786d740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55fd682216f0 con 0x55fd682bcd10 2024-04-02T14:44:57.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.087+0000 7f13cd400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fd682bcd10 0x55fd682bd190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:57.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.087+0000 7f13cd400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fd682bcd10 0x55fd682bd190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:60508/0 (socket says 172.21.15.90:60508) 2024-04-02T14:44:57.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.087+0000 7f13cd400640 1 -- 172.21.15.90:0/3122281161 learned_addr learned my addr 172.21.15.90:0/3122281161 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:44:57.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.088+0000 7f13ce800640 1 --2- 172.21.15.90:0/3122281161 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fd682c6080 0x55fd682c8470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:57.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.088+0000 7f13cd400640 1 -- 172.21.15.90:0/3122281161 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fd682bd9f0 msgr2=0x55fd682bc740 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-02T14:44:57.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.088+0000 7f13cd400640 1 --2- 172.21.15.90:0/3122281161 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fd682bd9f0 0x55fd682bc740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:57.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.088+0000 7f13cd400640 1 -- 172.21.15.90:0/3122281161 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fd682c6080 msgr2=0x55fd682c8470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:44:57.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.088+0000 7f13cd400640 1 --2- 172.21.15.90:0/3122281161 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fd682c6080 0x55fd682c8470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:57.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.088+0000 7f13cd400640 1 -- 172.21.15.90:0/3122281161 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fd67f353f0 con 0x55fd682bcd10 2024-04-02T14:44:57.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.089+0000 7f13ce800640 1 --2- 172.21.15.90:0/3122281161 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fd682c6080 0x55fd682c8470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:44:57.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.089+0000 7f13ce800640 1 --2- 172.21.15.90:0/1509930049 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f13b40a3f20 0x7f13b40a63e0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f13c8005f00 tx=0x7f13c8005fd0 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:44:57.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.089+0000 7f13cd400640 1 --2- 172.21.15.90:0/3122281161 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fd682bcd10 0x55fd682bd190 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f13b80989f0 tx=0x7f13b809c0d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=da9f3036a7b385a4 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:44:57.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.090+0000 7f13bf000640 1 -- 172.21.15.90:0/3122281161 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13b801c070 con 0x55fd682bcd10 2024-04-02T14:44:57.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.090+0000 7f13bf000640 1 -- 172.21.15.90:0/3122281161 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f13b809cc90 con 0x55fd682bcd10 2024-04-02T14:44:57.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.090+0000 7f13bf000640 1 -- 172.21.15.90:0/3122281161 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13b8098c00 con 0x55fd682bcd10 2024-04-02T14:44:57.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.090+0000 7f13d786d740 1 -- 172.21.15.90:0/3122281161 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fd682bcd10 msgr2=0x55fd682bd190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:57.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.090+0000 7f13d786d740 1 --2- 172.21.15.90:0/3122281161 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fd682bcd10 0x55fd682bd190 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f13b80989f0 tx=0x7f13b809c0d0 comp rx=0 tx=0).stop 2024-04-02T14:44:57.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.091+0000 7f13d786d740 1 -- 172.21.15.90:0/3122281161 shutdown_connections 2024-04-02T14:44:57.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.091+0000 7f13d786d740 1 --2- 172.21.15.90:0/3122281161 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fd682c6080 0x55fd682c8470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:57.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.091+0000 7f13d786d740 1 --2- 172.21.15.90:0/3122281161 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fd682bcd10 0x55fd682bd190 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:57.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.091+0000 7f13d786d740 1 --2- 172.21.15.90:0/3122281161 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fd682bd9f0 0x55fd682bc740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:57.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.091+0000 7f13d786d740 1 -- 172.21.15.90:0/3122281161 >> 172.21.15.90:0/3122281161 conn(0x55fd68283a50 msgr2=0x55fd682b7760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:44:57.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.091+0000 7f13d786d740 1 -- 172.21.15.90:0/3122281161 shutdown_connections 2024-04-02T14:44:57.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.091+0000 7f13d786d740 1 -- 172.21.15.90:0/3122281161 wait complete. 2024-04-02T14:44:57.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.091+0000 7f13d786d740 1 -- start start 2024-04-02T14:44:57.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.092+0000 7f13d786d740 1 Processor -- start 2024-04-02T14:44:57.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.092+0000 7f13d786d740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fd682bcd10 0x55fd682c3140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:57.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.092+0000 7f13d786d740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fd682bd9f0 0x55fd682c3680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:57.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.092+0000 7f13d786d740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fd682c6080 0x55fd682c3bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:57.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.093+0000 7f13cde00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fd682bcd10 0x55fd682c3140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:57.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.093+0000 7f13cde00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fd682bcd10 0x55fd682c3140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:60522/0 (socket says 172.21.15.90:60522) 2024-04-02T14:44:57.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.093+0000 7f13d786d740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55fd6821d6a0 con 0x55fd682c6080 2024-04-02T14:44:57.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.093+0000 7f13d786d740 1 -- 172.21.15.90:0/2902646617 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55fd681c9980 con 0x55fd682bd9f0 2024-04-02T14:44:57.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.093+0000 7f13d786d740 1 -- 172.21.15.90:0/2902646617 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55fd681dccb0 con 0x55fd682bcd10 2024-04-02T14:44:57.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.093+0000 7f13d786d740 10 client.0 _ll_register_callbacks cb 0x55fd681dc6a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:44:57.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.093+0000 7f13cde00640 1 -- 172.21.15.90:0/2902646617 learned_addr learned my addr 172.21.15.90:0/2902646617 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:44:57.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.093+0000 7f13cd400640 1 --2- 172.21.15.90:0/2902646617 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fd682bd9f0 0x55fd682c3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:57.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.093+0000 7f13cde00640 1 -- 172.21.15.90:0/2902646617 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fd682bd9f0 msgr2=0x55fd682c3680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:44:57.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.093+0000 7f13cde00640 1 --2- 172.21.15.90:0/2902646617 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fd682bd9f0 0x55fd682c3680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:57.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.093+0000 7f13cde00640 1 -- 172.21.15.90:0/2902646617 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fd682c6080 msgr2=0x55fd682c3bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:44:57.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.093+0000 7f13cde00640 1 --2- 172.21.15.90:0/2902646617 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fd682c6080 0x55fd682c3bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:57.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.094+0000 7f13cde00640 1 -- 172.21.15.90:0/2902646617 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fd67fff7d0 con 0x55fd682bcd10 2024-04-02T14:44:57.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.096+0000 7f13cde00640 1 --2- 172.21.15.90:0/2902646617 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fd682bcd10 0x55fd682c3140 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f13c0004700 tx=0x7f13c000fc10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b381ea435e710755 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:44:57.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.096+0000 7f13b3400640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:44:57.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.097+0000 7f13b3400640 1 -- 172.21.15.90:0/2902646617 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13c000db20 con 0x55fd682bcd10 2024-04-02T14:44:57.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.097+0000 7f13b3400640 1 -- 172.21.15.90:0/2902646617 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8550+0+0 (secure 0 0 0) 0x7f13c000dcc0 con 0x55fd682bcd10 2024-04-02T14:44:57.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.097+0000 7f13d786d740 1 -- 172.21.15.90:0/2902646617 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f13b809cc90 con 0x55fd682bcd10 2024-04-02T14:44:57.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.098+0000 7f13b3400640 1 -- 172.21.15.90:0/2902646617 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f13c0005020 con 0x55fd682bcd10 2024-04-02T14:44:57.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.098+0000 7f13d786d740 10 client.28426 fetch_fsmap learned FSMap version 27 2024-04-02T14:44:57.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.098+0000 7f13d786d740 1 -- 172.21.15.90:0/2902646617 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55fd682d5050 con 0x55fd682bcd10 2024-04-02T14:44:57.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.098+0000 7f13b3400640 1 -- 172.21.15.90:0/2902646617 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f13c00056a0 con 0x55fd682bcd10 2024-04-02T14:44:57.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.099+0000 7f13b3400640 1 -- 172.21.15.90:0/2902646617 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f13c0004930 con 0x55fd682bcd10 2024-04-02T14:44:57.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.100+0000 7f13d786d740 10 client.28426 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:44:57.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.100+0000 7f13d786d740 10 client.28426 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2024-04-02T14:44:57.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.100+0000 7f13d786d740 20 client.28426 populate_metadata read hostname 'smithi090' 2024-04-02T14:44:57.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.100+0000 7f13d786d740 1 --2- 172.21.15.90:0/2902646617 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x55fd682d6560 0x55fd682d8970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:57.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.100+0000 7f13d786d740 4 client.28426 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:44:57.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.100+0000 7f13d786d740 1 -- 172.21.15.90:0/2902646617 --> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f13b809cc90 con 0x55fd682d6560 2024-04-02T14:44:57.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.100+0000 7f13cd400640 1 --2- 172.21.15.90:0/2902646617 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x55fd682d6560 0x55fd682d8970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:57.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.102+0000 7f13cd400640 1 --2- 172.21.15.90:0/2902646617 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x55fd682d6560 0x55fd682d8970 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=656d4dd6c06ef3c9 server_cookie=7b8fb983c37bf325 in_seq=0 out_seq=0 2024-04-02T14:44:57.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.102+0000 7f13b3400640 10 client.28426 ms_handle_connect on v2:172.21.15.90:6838/1498915870 2024-04-02T14:44:57.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.105+0000 7f13b3400640 1 -- 172.21.15.90:0/2902646617 <== mds.1 v2:172.21.15.90:6838/1498915870 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x7f13b809cc90 con 0x55fd682d6560 2024-04-02T14:44:57.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.105+0000 7f13b3400640 10 client.28426 handle_command_reply: tid=0 2024-04-02T14:44:57.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.105+0000 7f13d786d740 1 client.28426 shutdown 2024-04-02T14:44:57.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.105+0000 7f13d786d740 1 -- 172.21.15.90:0/2902646617 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fd682bcd10 msgr2=0x55fd682c3140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:57.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.105+0000 7f13d786d740 1 --2- 172.21.15.90:0/2902646617 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fd682bcd10 0x55fd682c3140 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f13c0004700 tx=0x7f13c000fc10 comp rx=0 tx=0).stop 2024-04-02T14:44:57.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.106+0000 7f13d786d740 1 -- 172.21.15.90:0/2902646617 shutdown_connections 2024-04-02T14:44:57.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.106+0000 7f13d786d740 1 --2- 172.21.15.90:0/2902646617 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x55fd682d6560 0x55fd682d8970 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:57.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.106+0000 7f13d786d740 1 --2- 172.21.15.90:0/2902646617 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fd682c6080 0x55fd682c3bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:57.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.106+0000 7f13d786d740 1 --2- 172.21.15.90:0/2902646617 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fd682bd9f0 0x55fd682c3680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:57.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.106+0000 7f13d786d740 1 --2- 172.21.15.90:0/2902646617 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fd682bcd10 0x55fd682c3140 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:57.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.106+0000 7f13d786d740 1 -- 172.21.15.90:0/2902646617 >> 172.21.15.90:0/2902646617 conn(0x55fd68283a50 msgr2=0x55fd682b94b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:44:57.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.106+0000 7f13b3400640 0 client.28426 ms_handle_reset on v2:172.21.15.90:6838/1498915870 2024-04-02T14:44:57.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.106+0000 7f13d786d740 1 -- 172.21.15.90:0/2902646617 shutdown_connections 2024-04-02T14:44:57.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.106+0000 7f13d786d740 1 -- 172.21.15.90:0/2902646617 wait complete. 2024-04-02T14:44:57.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.106+0000 7f13d786d740 20 client.28426 trim_cache size 0 max 16384 2024-04-02T14:44:57.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.143+0000 7f13d786d740 1 Processor -- start 2024-04-02T14:44:57.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.144+0000 7f13d786d740 1 -- start start 2024-04-02T14:44:57.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.144+0000 7f13d786d740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fd682bcd10 0x55fd682d1a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:57.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.144+0000 7f13d786d740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fd682bd9f0 0x55fd682c3bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:57.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.144+0000 7f13d786d740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fd682c6080 0x55fd682c2a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:57.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.144+0000 7f13d786d740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f13d01c4fc0 con 0x55fd682bd9f0 2024-04-02T14:44:57.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.144+0000 7f13d786d740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f13d01c6b00 con 0x55fd682c6080 2024-04-02T14:44:57.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.144+0000 7f13d786d740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f13d01c5690 con 0x55fd682bcd10 2024-04-02T14:44:57.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.145+0000 7f13ce800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fd682c6080 0x55fd682c2a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:57.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.145+0000 7f13ce800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fd682c6080 0x55fd682c2a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:40912/0 (socket says 172.21.15.90:40912) 2024-04-02T14:44:57.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.145+0000 7f13ce800640 1 -- 172.21.15.90:0/2077413322 learned_addr learned my addr 172.21.15.90:0/2077413322 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:44:57.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.145+0000 7f13cde00640 1 --2- 172.21.15.90:0/2077413322 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fd682bcd10 0x55fd682d1a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:57.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.146+0000 7f13cde00640 1 -- 172.21.15.90:0/2077413322 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fd682c6080 msgr2=0x55fd682c2a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:44:57.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.146+0000 7f13cde00640 1 --2- 172.21.15.90:0/2077413322 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fd682c6080 0x55fd682c2a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:57.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.146+0000 7f13cde00640 1 -- 172.21.15.90:0/2077413322 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fd682bd9f0 msgr2=0x55fd682c3bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:44:57.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.146+0000 7f13cde00640 1 --2- 172.21.15.90:0/2077413322 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fd682bd9f0 0x55fd682c3bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:57.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.146+0000 7f13cde00640 1 -- 172.21.15.90:0/2077413322 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fd682c8820 con 0x55fd682bcd10 2024-04-02T14:44:57.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.146+0000 7f13cde00640 1 --2- 172.21.15.90:0/2077413322 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fd682bcd10 0x55fd682d1a90 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f13c0001eb0 tx=0x7f13c00058d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7cf4f446f7d0b9f1 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:44:57.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.146+0000 7f13be600640 1 -- 172.21.15.90:0/2077413322 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13c002f070 con 0x55fd682bcd10 2024-04-02T14:44:57.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.146+0000 7f13be600640 1 -- 172.21.15.90:0/2077413322 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f13c0004510 con 0x55fd682bcd10 2024-04-02T14:44:57.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.146+0000 7f13be600640 1 -- 172.21.15.90:0/2077413322 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13c0005020 con 0x55fd682bcd10 2024-04-02T14:44:57.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.147+0000 7f13d786d740 1 -- 172.21.15.90:0/2077413322 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fd682bcd10 msgr2=0x55fd682d1a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:57.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.148+0000 7f13d786d740 1 --2- 172.21.15.90:0/2077413322 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fd682bcd10 0x55fd682d1a90 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f13c0001eb0 tx=0x7f13c00058d0 comp rx=0 tx=0).stop 2024-04-02T14:44:57.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.148+0000 7f13d786d740 1 -- 172.21.15.90:0/2077413322 shutdown_connections 2024-04-02T14:44:57.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.148+0000 7f13d786d740 1 --2- 172.21.15.90:0/2077413322 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fd682c6080 0x55fd682c2a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:57.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.148+0000 7f13d786d740 1 --2- 172.21.15.90:0/2077413322 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fd682bd9f0 0x55fd682c3bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:57.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.148+0000 7f13d786d740 1 --2- 172.21.15.90:0/2077413322 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fd682bcd10 0x55fd682d1a90 secure :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f13c0001eb0 tx=0x7f13c00058d0 comp rx=0 tx=0).stop 2024-04-02T14:44:57.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.148+0000 7f13d786d740 1 -- 172.21.15.90:0/2077413322 >> 172.21.15.90:0/2077413322 conn(0x55fd68283a50 msgr2=0x55fd682b68a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:44:57.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.148+0000 7f13d786d740 1 -- 172.21.15.90:0/2077413322 shutdown_connections 2024-04-02T14:44:57.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.148+0000 7f13d786d740 1 -- 172.21.15.90:0/2077413322 wait complete. 2024-04-02T14:44:57.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.149+0000 7f13d786d740 1 -- start start 2024-04-02T14:44:57.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.149+0000 7f13d786d740 1 Processor -- start 2024-04-02T14:44:57.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.150+0000 7f13d786d740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fd682bd9f0 0x55fd682ce4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:57.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.150+0000 7f13d786d740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fd682c6080 0x55fd682c6bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:57.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.150+0000 7f13d786d740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fd682c7270 0x55fd682cc030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:57.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.150+0000 7f13d786d740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f13d01c6430 con 0x55fd682c6080 2024-04-02T14:44:57.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.150+0000 7f13d786d740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f13d01c5d60 con 0x55fd682c7270 2024-04-02T14:44:57.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.150+0000 7f13d786d740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f13d01c2f80 con 0x55fd682bd9f0 2024-04-02T14:44:57.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.150+0000 7f13d786d740 10 client.0 _ll_register_callbacks cb 0x7f13c000f3c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:44:57.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.150+0000 7f13cde00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fd682bd9f0 0x55fd682ce4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:57.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.151+0000 7f13cde00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fd682bd9f0 0x55fd682ce4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:60532/0 (socket says 172.21.15.90:60532) 2024-04-02T14:44:57.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.151+0000 7f13cde00640 1 -- 172.21.15.90:0/1983393993 learned_addr learned my addr 172.21.15.90:0/1983393993 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:44:57.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.151+0000 7f13ce800640 1 --2- 172.21.15.90:0/1983393993 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fd682c7270 0x55fd682cc030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:57.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.151+0000 7f13cd400640 1 --2- 172.21.15.90:0/1983393993 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fd682c6080 0x55fd682c6bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:57.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.151+0000 7f13cde00640 1 -- 172.21.15.90:0/1983393993 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fd682c7270 msgr2=0x55fd682cc030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:44:57.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.151+0000 7f13cde00640 1 --2- 172.21.15.90:0/1983393993 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fd682c7270 0x55fd682cc030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:57.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.151+0000 7f13cde00640 1 -- 172.21.15.90:0/1983393993 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fd682c6080 msgr2=0x55fd682c6bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:44:57.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.152+0000 7f13cde00640 1 --2- 172.21.15.90:0/1983393993 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fd682c6080 0x55fd682c6bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:57.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.152+0000 7f13cde00640 1 -- 172.21.15.90:0/1983393993 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fd682c79e0 con 0x55fd682bd9f0 2024-04-02T14:44:57.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.152+0000 7f13cd400640 1 --2- 172.21.15.90:0/1983393993 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fd682c6080 0x55fd682c6bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:44:57.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.152+0000 7f13cde00640 1 --2- 172.21.15.90:0/1983393993 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fd682bd9f0 0x55fd682ce4f0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f13c000fc50 tx=0x7f13c0004480 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6334a28a1c6547e5 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:44:57.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.153+0000 7f13b3400640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:44:57.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.154+0000 7f13b3400640 1 -- 172.21.15.90:0/1983393993 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13c002f040 con 0x55fd682bd9f0 2024-04-02T14:44:57.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.154+0000 7f13b3400640 1 -- 172.21.15.90:0/1983393993 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8550+0+0 (secure 0 0 0) 0x7f13c0005e10 con 0x55fd682bd9f0 2024-04-02T14:44:57.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.154+0000 7f13d786d740 1 -- 172.21.15.90:0/1983393993 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f13c0004510 con 0x55fd682bd9f0 2024-04-02T14:44:57.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.155+0000 7f13b3400640 1 -- 172.21.15.90:0/1983393993 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f13c0004850 con 0x55fd682bd9f0 2024-04-02T14:44:57.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.155+0000 7f13d786d740 10 client.28432 fetch_fsmap learned FSMap version 27 2024-04-02T14:44:57.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.155+0000 7f13d786d740 1 -- 172.21.15.90:0/1983393993 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55fd6833cf90 con 0x55fd682bd9f0 2024-04-02T14:44:57.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.155+0000 7f13b3400640 1 -- 172.21.15.90:0/1983393993 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f13c0004510 con 0x55fd682bd9f0 2024-04-02T14:44:57.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.156+0000 7f13b3400640 1 -- 172.21.15.90:0/1983393993 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f13c0032070 con 0x55fd682bd9f0 2024-04-02T14:44:57.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.156+0000 7f13d786d740 10 client.28432 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:44:57.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.156+0000 7f13d786d740 10 client.28432 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2024-04-02T14:44:57.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.156+0000 7f13d786d740 20 client.28432 populate_metadata read hostname 'smithi090' 2024-04-02T14:44:57.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.156+0000 7f13d786d740 1 --2- 172.21.15.90:0/1983393993 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x55fd6833d570 0x55fd6833f990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:57.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.156+0000 7f13d786d740 4 client.28432 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-02T14:44:57.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.156+0000 7f13d786d740 1 -- 172.21.15.90:0/1983393993 --> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55fd681e1040 con 0x55fd6833d570 2024-04-02T14:44:57.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.156+0000 7f13cd400640 1 --2- 172.21.15.90:0/1983393993 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x55fd6833d570 0x55fd6833f990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:57.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.158+0000 7f13b3400640 10 client.28432 ms_handle_connect on v2:172.21.15.90:6838/1498915870 2024-04-02T14:44:57.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.158+0000 7f13cd400640 1 --2- 172.21.15.90:0/1983393993 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x55fd6833d570 0x55fd6833f990 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=fcb74e88da30a3f6 server_cookie=f9fd3a7658215c51 in_seq=0 out_seq=0 2024-04-02T14:44:57.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.159+0000 7f13b3400640 1 -- 172.21.15.90:0/1983393993 <== mds.1 v2:172.21.15.90:6838/1498915870 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55fd681e1040 con 0x55fd6833d570 2024-04-02T14:44:57.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.159+0000 7f13b3400640 10 client.28432 handle_command_reply: tid=0 2024-04-02T14:44:57.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.159+0000 7f13d786d740 1 client.28432 shutdown 2024-04-02T14:44:57.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.160+0000 7f13d786d740 1 -- 172.21.15.90:0/1983393993 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fd682bd9f0 msgr2=0x55fd682ce4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:57.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.160+0000 7f13d786d740 1 --2- 172.21.15.90:0/1983393993 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fd682bd9f0 0x55fd682ce4f0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f13c000fc50 tx=0x7f13c0004480 comp rx=0 tx=0).stop 2024-04-02T14:44:57.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.160+0000 7f13d786d740 1 -- 172.21.15.90:0/1983393993 shutdown_connections 2024-04-02T14:44:57.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.160+0000 7f13d786d740 1 --2- 172.21.15.90:0/1983393993 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x55fd6833d570 0x55fd6833f990 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:57.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.160+0000 7f13d786d740 1 --2- 172.21.15.90:0/1983393993 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fd682c7270 0x55fd682cc030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:57.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.160+0000 7f13d786d740 1 --2- 172.21.15.90:0/1983393993 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fd682c6080 0x55fd682c6bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:57.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.160+0000 7f13d786d740 1 --2- 172.21.15.90:0/1983393993 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fd682bd9f0 0x55fd682ce4f0 secure :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f13c000fc50 tx=0x7f13c0004480 comp rx=0 tx=0).stop 2024-04-02T14:44:57.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.160+0000 7f13d786d740 1 -- 172.21.15.90:0/1983393993 >> 172.21.15.90:0/1983393993 conn(0x55fd68283a50 msgr2=0x55fd682b8760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:44:57.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.160+0000 7f13b3400640 0 client.28432 ms_handle_reset on v2:172.21.15.90:6838/1498915870 2024-04-02T14:44:57.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.160+0000 7f13d786d740 1 -- 172.21.15.90:0/1983393993 shutdown_connections 2024-04-02T14:44:57.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.161+0000 7f13d786d740 1 -- 172.21.15.90:0/1983393993 wait complete. 2024-04-02T14:44:57.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.161+0000 7f13d786d740 20 client.28432 trim_cache size 0 max 16384 2024-04-02T14:44:57.157 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-02T14:44:57.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.162+0000 7f13d4c00640 1 -- 172.21.15.90:0/1509930049 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f13b40a3f20 msgr2=0x7f13b40a63e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:57.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.162+0000 7f13d4c00640 1 --2- 172.21.15.90:0/1509930049 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f13b40a3f20 0x7f13b40a63e0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f13c8005f00 tx=0x7f13c8005fd0 comp rx=0 tx=0).stop 2024-04-02T14:44:57.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.162+0000 7f13d4c00640 1 -- 172.21.15.90:0/1509930049 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f13d01094c0 msgr2=0x7f13d01a5bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:57.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.162+0000 7f13d4c00640 1 --2- 172.21.15.90:0/1509930049 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f13d01094c0 0x7f13d01a5bd0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f13b8009fd0 tx=0x7f13b800ec20 comp rx=0 tx=0).stop 2024-04-02T14:44:57.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.163+0000 7f13d4c00640 1 -- 172.21.15.90:0/1509930049 shutdown_connections 2024-04-02T14:44:57.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.163+0000 7f13d4c00640 1 --2- 172.21.15.90:0/1509930049 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f13b40a3f20 0x7f13b40a63e0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:57.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.163+0000 7f13d4c00640 1 --2- 172.21.15.90:0/1509930049 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f13d010e630 0x7f13d019e8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:57.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.163+0000 7f13d4c00640 1 --2- 172.21.15.90:0/1509930049 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f13d01094c0 0x7f13d01a5bd0 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:57.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.163+0000 7f13d4c00640 1 --2- 172.21.15.90:0/1509930049 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f13d0108b00 0x7f13d01a5690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:57.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.163+0000 7f13d4c00640 1 -- 172.21.15.90:0/1509930049 >> 172.21.15.90:0/1509930049 conn(0x7f13d006cbe0 msgr2=0x7f13d0071fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:44:57.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.163+0000 7f13d4c00640 1 -- 172.21.15.90:0/1509930049 shutdown_connections 2024-04-02T14:44:57.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.163+0000 7f13d4c00640 1 -- 172.21.15.90:0/1509930049 wait complete. 2024-04-02T14:44:57.186 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-02T14:44:57.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.518+0000 7f0e33000640 1 -- 172.21.15.90:0/2983146710 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0e2c10af10 msgr2=0x7f0e2c10ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:57.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.518+0000 7f0e33000640 1 --2- 172.21.15.90:0/2983146710 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0e2c10af10 0x7f0e2c10ef00 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f0e1c009900 tx=0x7f0e1c02bc30 comp rx=0 tx=0).stop 2024-04-02T14:44:57.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.519+0000 7f0e33000640 1 -- 172.21.15.90:0/2983146710 shutdown_connections 2024-04-02T14:44:57.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.519+0000 7f0e33000640 1 --2- 172.21.15.90:0/2983146710 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0e2c10f5e0 0x7f0e2c1171b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:57.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.519+0000 7f0e33000640 1 --2- 172.21.15.90:0/2983146710 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0e2c10af10 0x7f0e2c10ef00 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:57.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.519+0000 7f0e33000640 1 --2- 172.21.15.90:0/2983146710 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0e2c10a560 0x7f0e2c10a940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:57.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.519+0000 7f0e33000640 1 -- 172.21.15.90:0/2983146710 >> 172.21.15.90:0/2983146710 conn(0x7f0e2c073090 msgr2=0x7f0e2c0734a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:44:57.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.519+0000 7f0e33000640 1 -- 172.21.15.90:0/2983146710 shutdown_connections 2024-04-02T14:44:57.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.519+0000 7f0e33000640 1 -- 172.21.15.90:0/2983146710 wait complete. 2024-04-02T14:44:57.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.523+0000 7f0e33000640 1 Processor -- start 2024-04-02T14:44:57.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.524+0000 7f0e33000640 1 -- start start 2024-04-02T14:44:57.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.525+0000 7f0e33000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0e2c10a560 0x7f0e2c1a6950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:57.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.525+0000 7f0e33000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0e2c10f5e0 0x7f0e2c1a6e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:57.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.526+0000 7f0e33000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0e2c1a0b00 0x7f0e2c1a0fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:57.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.526+0000 7f0e33000640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f0e2c11a7a0 con 0x7f0e2c1a0b00 2024-04-02T14:44:57.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.526+0000 7f0e33000640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f0e2c11a620 con 0x7f0e2c10f5e0 2024-04-02T14:44:57.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.526+0000 7f0e33000640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f0e2c11a920 con 0x7f0e2c10a560 2024-04-02T14:44:57.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.526+0000 7f0e2b400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0e2c10f5e0 0x7f0e2c1a6e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:57.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.526+0000 7f0e2b400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0e2c10f5e0 0x7f0e2c1a6e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:40938/0 (socket says 172.21.15.90:40938) 2024-04-02T14:44:57.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.526+0000 7f0e2b400640 1 -- 172.21.15.90:0/3645970646 learned_addr learned my addr 172.21.15.90:0/3645970646 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:44:57.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.527+0000 7f0e31000640 1 --2- 172.21.15.90:0/3645970646 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0e2c1a0b00 0x7f0e2c1a0fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:57.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.526+0000 7f0e2be00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0e2c10a560 0x7f0e2c1a6950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:57.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.528+0000 7f0e2b400640 1 -- 172.21.15.90:0/3645970646 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0e2c10a560 msgr2=0x7f0e2c1a6950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:57.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.528+0000 7f0e2b400640 1 --2- 172.21.15.90:0/3645970646 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0e2c10a560 0x7f0e2c1a6950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:57.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.528+0000 7f0e2b400640 1 -- 172.21.15.90:0/3645970646 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0e2c1a0b00 msgr2=0x7f0e2c1a0fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:57.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.529+0000 7f0e2b400640 1 --2- 172.21.15.90:0/3645970646 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0e2c1a0b00 0x7f0e2c1a0fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:57.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.529+0000 7f0e2b400640 1 -- 172.21.15.90:0/3645970646 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e2c1a1620 con 0x7f0e2c10f5e0 2024-04-02T14:44:57.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.529+0000 7f0e31000640 1 --2- 172.21.15.90:0/3645970646 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0e2c1a0b00 0x7f0e2c1a0fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:44:57.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.529+0000 7f0e2b400640 1 --2- 172.21.15.90:0/3645970646 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0e2c10f5e0 0x7f0e2c1a6e90 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f0e1c000c00 tx=0x7f0e1c0045e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:44:57.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.530+0000 7f0e28c00640 1 -- 172.21.15.90:0/3645970646 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e1c043020 con 0x7f0e2c10f5e0 2024-04-02T14:44:57.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.530+0000 7f0e28c00640 1 -- 172.21.15.90:0/3645970646 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0e1c0054e0 con 0x7f0e2c10f5e0 2024-04-02T14:44:57.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.530+0000 7f0e28c00640 1 -- 172.21.15.90:0/3645970646 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e1c003d40 con 0x7f0e2c10f5e0 2024-04-02T14:44:57.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.531+0000 7f0e33000640 1 -- 172.21.15.90:0/3645970646 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0e2c1a18b0 con 0x7f0e2c10f5e0 2024-04-02T14:44:57.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.531+0000 7f0e33000640 1 -- 172.21.15.90:0/3645970646 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0e2c06b4c0 con 0x7f0e2c10f5e0 2024-04-02T14:44:57.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.543+0000 7f0e28c00640 1 -- 172.21.15.90:0/3645970646 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f0e1c046020 con 0x7f0e2c10f5e0 2024-04-02T14:44:57.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.545+0000 7f0e35c0c740 1 Processor -- start 2024-04-02T14:44:57.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.546+0000 7f0e35c0c740 1 -- start start 2024-04-02T14:44:57.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.547+0000 7f0e35c0c740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560350e50200 0x560350e4ef50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:57.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.547+0000 7f0e28c00640 1 --2- 172.21.15.90:0/3645970646 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f0e100a3e60 0x7f0e100a6320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:57.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.547+0000 7f0e35c0c740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560350e4f520 0x560350e4f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:57.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.547+0000 7f0e35c0c740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560350e58890 0x560350e5ac80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:57.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.547+0000 7f0e35c0c740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x560350d87400 con 0x560350e50200 2024-04-02T14:44:57.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.547+0000 7f0e35c0c740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x560350d6bf10 con 0x560350e58890 2024-04-02T14:44:57.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.547+0000 7f0e35c0c740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x560350dab6f0 con 0x560350e4f520 2024-04-02T14:44:57.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.547+0000 7f0e28c00640 1 -- 172.21.15.90:0/3645970646 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8550+0+0 (secure 0 0 0) 0x7f0e1c0f2780 con 0x7f0e2c10f5e0 2024-04-02T14:44:57.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.548+0000 7f0e2be00640 1 --2- 172.21.15.90:0/3645970646 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f0e100a3e60 0x7f0e100a6320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:57.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.548+0000 7f0e31000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560350e58890 0x560350e5ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:57.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.548+0000 7f0e31000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560350e50200 0x560350e4ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:57.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.548+0000 7f0e31000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560350e50200 0x560350e4ef50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38972/0 (socket says 172.21.15.90:38972) 2024-04-02T14:44:57.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.548+0000 7f0e31000640 1 -- 172.21.15.90:0/1204101239 learned_addr learned my addr 172.21.15.90:0/1204101239 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:44:57.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.548+0000 7f0e2b400640 1 --2- 172.21.15.90:0/1204101239 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560350e4f520 0x560350e4f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:57.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.549+0000 7f0e2b400640 1 -- 172.21.15.90:0/1204101239 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560350e58890 msgr2=0x560350e5ac80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:44:57.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.549+0000 7f0e2b400640 1 --2- 172.21.15.90:0/1204101239 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560350e58890 0x560350e5ac80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:57.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.549+0000 7f0e2b400640 1 -- 172.21.15.90:0/1204101239 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560350e50200 msgr2=0x560350e4ef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:44:57.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.549+0000 7f0e2b400640 1 --2- 172.21.15.90:0/1204101239 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560350e50200 0x560350e4ef50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:57.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.549+0000 7f0e2b400640 1 -- 172.21.15.90:0/1204101239 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560350abf3f0 con 0x560350e4f520 2024-04-02T14:44:57.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.549+0000 7f0e31000640 1 --2- 172.21.15.90:0/1204101239 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560350e58890 0x560350e5ac80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:44:57.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.549+0000 7f0e2b400640 1 --2- 172.21.15.90:0/1204101239 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560350e4f520 0x560350e4f9a0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f0e1c0eff60 tx=0x7f0e1c003840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=951059f1f4c33a66 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:44:57.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.549+0000 7f0e21600640 1 -- 172.21.15.90:0/1204101239 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e1c02cd30 con 0x560350e4f520 2024-04-02T14:44:57.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.550+0000 7f0e21600640 1 -- 172.21.15.90:0/1204101239 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0e1c004c80 con 0x560350e4f520 2024-04-02T14:44:57.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.550+0000 7f0e21600640 1 -- 172.21.15.90:0/1204101239 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e1c0bcab0 con 0x560350e4f520 2024-04-02T14:44:57.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.552+0000 7f0e35c0c740 1 -- 172.21.15.90:0/1204101239 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560350e4f520 msgr2=0x560350e4f9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:57.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.552+0000 7f0e35c0c740 1 --2- 172.21.15.90:0/1204101239 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560350e4f520 0x560350e4f9a0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f0e1c0eff60 tx=0x7f0e1c003840 comp rx=0 tx=0).stop 2024-04-02T14:44:57.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.552+0000 7f0e2be00640 1 --2- 172.21.15.90:0/3645970646 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f0e100a3e60 0x7f0e100a6320 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f0e1400de30 tx=0x7f0e1400f040 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:44:57.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.552+0000 7f0e35c0c740 1 -- 172.21.15.90:0/1204101239 shutdown_connections 2024-04-02T14:44:57.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.553+0000 7f0e35c0c740 1 --2- 172.21.15.90:0/1204101239 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560350e58890 0x560350e5ac80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:57.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.553+0000 7f0e35c0c740 1 --2- 172.21.15.90:0/1204101239 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560350e4f520 0x560350e4f9a0 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:57.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.553+0000 7f0e35c0c740 1 --2- 172.21.15.90:0/1204101239 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560350e50200 0x560350e4ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:57.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.553+0000 7f0e35c0c740 1 -- 172.21.15.90:0/1204101239 >> 172.21.15.90:0/1204101239 conn(0x560350e0da50 msgr2=0x560350e49f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:44:57.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.554+0000 7f0e35c0c740 1 -- 172.21.15.90:0/1204101239 shutdown_connections 2024-04-02T14:44:57.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.554+0000 7f0e35c0c740 1 -- 172.21.15.90:0/1204101239 wait complete. 2024-04-02T14:44:57.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.555+0000 7f0e35c0c740 1 -- start start 2024-04-02T14:44:57.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.556+0000 7f0e35c0c740 1 Processor -- start 2024-04-02T14:44:57.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.557+0000 7f0e35c0c740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560350e4f520 0x560350e55950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:57.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.557+0000 7f0e35c0c740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560350e50200 0x560350e55e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:57.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.557+0000 7f0e35c0c740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560350e58890 0x560350e563d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:57.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.557+0000 7f0e35c0c740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x560350da76a0 con 0x560350e4f520 2024-04-02T14:44:57.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.557+0000 7f0e35c0c740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x560350d53980 con 0x560350e50200 2024-04-02T14:44:57.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.557+0000 7f0e35c0c740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x560350d66cb0 con 0x560350e58890 2024-04-02T14:44:57.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.557+0000 7f0e35c0c740 10 client.0 _ll_register_callbacks cb 0x560350d666a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:44:57.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.557+0000 7f0e2b400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560350e50200 0x560350e55e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:57.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.557+0000 7f0e2b400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560350e50200 0x560350e55e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:40960/0 (socket says 172.21.15.90:40960) 2024-04-02T14:44:57.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.557+0000 7f0e2b400640 1 -- 172.21.15.90:0/2349305215 learned_addr learned my addr 172.21.15.90:0/2349305215 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:44:57.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.557+0000 7f0e31000640 1 --2- 172.21.15.90:0/2349305215 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560350e58890 0x560350e563d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:57.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.558+0000 7f0e2be00640 1 --2- 172.21.15.90:0/2349305215 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560350e4f520 0x560350e55950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:57.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.558+0000 7f0e31000640 1 -- 172.21.15.90:0/2349305215 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560350e50200 msgr2=0x560350e55e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:44:57.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.558+0000 7f0e31000640 1 --2- 172.21.15.90:0/2349305215 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560350e50200 0x560350e55e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:57.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.558+0000 7f0e31000640 1 -- 172.21.15.90:0/2349305215 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560350e4f520 msgr2=0x560350e55950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:44:57.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.558+0000 7f0e31000640 1 --2- 172.21.15.90:0/2349305215 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560350e4f520 0x560350e55950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:57.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.558+0000 7f0e31000640 1 -- 172.21.15.90:0/2349305215 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560350b897d0 con 0x560350e58890 2024-04-02T14:44:57.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.558+0000 7f0e2be00640 1 --2- 172.21.15.90:0/2349305215 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560350e4f520 0x560350e55950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:44:57.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.558+0000 7f0e31000640 1 --2- 172.21.15.90:0/2349305215 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560350e58890 0x560350e563d0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f0e2400bf30 tx=0x7f0e2400b400 comp rx=0 tx=0).ready entity=mon.1 client_cookie=792bff2f0a4e951f server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:44:57.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.558+0000 7f0e1a000640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:44:57.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.558+0000 7f0e1a000640 1 -- 172.21.15.90:0/2349305215 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e2401f060 con 0x560350e58890 2024-04-02T14:44:57.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.559+0000 7f0e1a000640 1 -- 172.21.15.90:0/2349305215 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8550+0+0 (secure 0 0 0) 0x7f0e24008740 con 0x560350e58890 2024-04-02T14:44:57.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.559+0000 7f0e35c0c740 1 -- 172.21.15.90:0/2349305215 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0e1c004c80 con 0x560350e58890 2024-04-02T14:44:57.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.559+0000 7f0e1a000640 1 -- 172.21.15.90:0/2349305215 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f0e24014040 con 0x560350e58890 2024-04-02T14:44:57.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.559+0000 7f0e35c0c740 10 client.28441 fetch_fsmap learned FSMap version 27 2024-04-02T14:44:57.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.559+0000 7f0e35c0c740 1 -- 172.21.15.90:0/2349305215 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x560350e67510 con 0x560350e58890 2024-04-02T14:44:57.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.560+0000 7f0e1a000640 1 -- 172.21.15.90:0/2349305215 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0e1c004c80 con 0x560350e58890 2024-04-02T14:44:57.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.560+0000 7f0e1a000640 1 -- 172.21.15.90:0/2349305215 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f0e24023ac0 con 0x560350e58890 2024-04-02T14:44:57.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.561+0000 7f0e35c0c740 10 client.28441 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:44:57.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.561+0000 7f0e35c0c740 10 client.28441 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:44:57.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.561+0000 7f0e35c0c740 20 client.28441 populate_metadata read hostname 'smithi090' 2024-04-02T14:44:57.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.561+0000 7f0e35c0c740 1 --2- 172.21.15.90:0/2349305215 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x560350e68970 0x560350e6ad60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:57.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.561+0000 7f0e35c0c740 4 client.28441 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:44:57.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.561+0000 7f0e35c0c740 1 -- 172.21.15.90:0/2349305215 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560350d6b040 con 0x560350e68970 2024-04-02T14:44:57.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.562+0000 7f0e2be00640 1 --2- 172.21.15.90:0/2349305215 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x560350e68970 0x560350e6ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:57.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.563+0000 7f0e2be00640 1 --2- 172.21.15.90:0/2349305215 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x560350e68970 0x560350e6ad60 crc :-1 s=READY pgs=266 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d8a8aefcf2517adc server_cookie=65af00c3f1fc3534 in_seq=0 out_seq=0 2024-04-02T14:44:57.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.563+0000 7f0e1a000640 10 client.28441 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:44:57.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.563+0000 7f0e1a000640 1 -- 172.21.15.90:0/2349305215 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x560350d6b040 con 0x560350e68970 2024-04-02T14:44:57.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.563+0000 7f0e1a000640 10 client.28441 handle_command_reply: tid=0 2024-04-02T14:44:57.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.564+0000 7f0e35c0c740 1 client.28441 shutdown 2024-04-02T14:44:57.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.564+0000 7f0e35c0c740 1 -- 172.21.15.90:0/2349305215 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560350e58890 msgr2=0x560350e563d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:57.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.564+0000 7f0e35c0c740 1 --2- 172.21.15.90:0/2349305215 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560350e58890 0x560350e563d0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f0e2400bf30 tx=0x7f0e2400b400 comp rx=0 tx=0).stop 2024-04-02T14:44:57.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.565+0000 7f0e35c0c740 1 -- 172.21.15.90:0/2349305215 shutdown_connections 2024-04-02T14:44:57.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.565+0000 7f0e35c0c740 1 --2- 172.21.15.90:0/2349305215 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x560350e68970 0x560350e6ad60 crc :-1 s=READY pgs=266 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:57.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.565+0000 7f0e35c0c740 1 --2- 172.21.15.90:0/2349305215 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560350e58890 0x560350e563d0 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:57.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.565+0000 7f0e35c0c740 1 --2- 172.21.15.90:0/2349305215 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560350e50200 0x560350e55e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:57.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.565+0000 7f0e35c0c740 1 --2- 172.21.15.90:0/2349305215 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560350e4f520 0x560350e55950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:57.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.565+0000 7f0e35c0c740 1 -- 172.21.15.90:0/2349305215 >> 172.21.15.90:0/2349305215 conn(0x560350e0da50 msgr2=0x560350e4bcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:44:57.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.565+0000 7f0e1a000640 0 client.28441 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:44:57.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.565+0000 7f0e35c0c740 1 -- 172.21.15.90:0/2349305215 shutdown_connections 2024-04-02T14:44:57.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.565+0000 7f0e35c0c740 1 -- 172.21.15.90:0/2349305215 wait complete. 2024-04-02T14:44:57.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.565+0000 7f0e35c0c740 20 client.28441 trim_cache size 0 max 16384 2024-04-02T14:44:57.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.584+0000 7f0e35c0c740 1 Processor -- start 2024-04-02T14:44:57.582 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.584+0000 7f0e35c0c740 1 -- start start 2024-04-02T14:44:57.582 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.584+0000 7f0e35c0c740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560350e4f520 0x560350e58080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:57.582 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.584+0000 7f0e35c0c740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560350e50200 0x560350e55230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:57.582 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.584+0000 7f0e35c0c740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560350e58890 0x560350e70630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:57.582 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.584+0000 7f0e35c0c740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f0e2c1b3f70 con 0x560350e58890 2024-04-02T14:44:57.582 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.584+0000 7f0e35c0c740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f0e2c1b5ab0 con 0x560350e4f520 2024-04-02T14:44:57.582 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.584+0000 7f0e35c0c740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f0e2c1b4640 con 0x560350e50200 2024-04-02T14:44:57.582 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.585+0000 7f0e2b400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560350e50200 0x560350e55230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:57.582 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.585+0000 7f0e31000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560350e58890 0x560350e70630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:57.582 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.585+0000 7f0e2be00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560350e4f520 0x560350e58080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:57.583 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.585+0000 7f0e2b400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560350e50200 0x560350e55230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:60580/0 (socket says 172.21.15.90:60580) 2024-04-02T14:44:57.583 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.585+0000 7f0e2b400640 1 -- 172.21.15.90:0/769859127 learned_addr learned my addr 172.21.15.90:0/769859127 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:44:57.583 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.585+0000 7f0e2be00640 1 -- 172.21.15.90:0/769859127 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560350e50200 msgr2=0x560350e55230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:44:57.583 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.585+0000 7f0e2be00640 1 --2- 172.21.15.90:0/769859127 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560350e50200 0x560350e55230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:57.583 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.585+0000 7f0e2be00640 1 -- 172.21.15.90:0/769859127 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560350e58890 msgr2=0x560350e70630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:44:57.583 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.585+0000 7f0e2be00640 1 --2- 172.21.15.90:0/769859127 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560350e58890 0x560350e70630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:57.583 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.585+0000 7f0e2be00640 1 -- 172.21.15.90:0/769859127 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560350e53ea0 con 0x560350e4f520 2024-04-02T14:44:57.583 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.585+0000 7f0e2b400640 1 --2- 172.21.15.90:0/769859127 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560350e50200 0x560350e55230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:44:57.583 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.585+0000 7f0e2be00640 1 --2- 172.21.15.90:0/769859127 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560350e4f520 0x560350e58080 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f0e1400dc10 tx=0x7f0e14011000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b163ba3fcf2559a6 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:44:57.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.586+0000 7f0e20c00640 1 -- 172.21.15.90:0/769859127 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e14029070 con 0x560350e4f520 2024-04-02T14:44:57.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.586+0000 7f0e20c00640 1 -- 172.21.15.90:0/769859127 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0e1400be10 con 0x560350e4f520 2024-04-02T14:44:57.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.586+0000 7f0e20c00640 1 -- 172.21.15.90:0/769859127 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e14014070 con 0x560350e4f520 2024-04-02T14:44:57.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.586+0000 7f0e35c0c740 1 -- 172.21.15.90:0/769859127 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560350e4f520 msgr2=0x560350e58080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:57.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.586+0000 7f0e35c0c740 1 --2- 172.21.15.90:0/769859127 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560350e4f520 0x560350e58080 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f0e1400dc10 tx=0x7f0e14011000 comp rx=0 tx=0).stop 2024-04-02T14:44:57.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.587+0000 7f0e35c0c740 1 -- 172.21.15.90:0/769859127 shutdown_connections 2024-04-02T14:44:57.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.587+0000 7f0e35c0c740 1 --2- 172.21.15.90:0/769859127 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560350e58890 0x560350e70630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:57.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.587+0000 7f0e35c0c740 1 --2- 172.21.15.90:0/769859127 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560350e50200 0x560350e55230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:57.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.587+0000 7f0e35c0c740 1 --2- 172.21.15.90:0/769859127 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560350e4f520 0x560350e58080 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:57.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.587+0000 7f0e35c0c740 1 -- 172.21.15.90:0/769859127 >> 172.21.15.90:0/769859127 conn(0x560350e0da50 msgr2=0x560350e408a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:44:57.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.587+0000 7f0e35c0c740 1 -- 172.21.15.90:0/769859127 shutdown_connections 2024-04-02T14:44:57.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.587+0000 7f0e35c0c740 1 -- 172.21.15.90:0/769859127 wait complete. 2024-04-02T14:44:57.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.587+0000 7f0e35c0c740 1 -- start start 2024-04-02T14:44:57.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.587+0000 7f0e35c0c740 1 Processor -- start 2024-04-02T14:44:57.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.588+0000 7f0e35c0c740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560350e4f520 0x560350e58c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:57.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.588+0000 7f0e35c0c740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560350e50200 0x560350e591b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:57.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.588+0000 7f0e35c0c740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560350e58890 0x560350e596f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:57.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.588+0000 7f0e35c0c740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f0e2c1b53e0 con 0x560350e50200 2024-04-02T14:44:57.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.588+0000 7f0e35c0c740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f0e2c1b4d10 con 0x560350e58890 2024-04-02T14:44:57.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.588+0000 7f0e35c0c740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f0e2c1b1f90 con 0x560350e4f520 2024-04-02T14:44:57.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.588+0000 7f0e35c0c740 10 client.0 _ll_register_callbacks cb 0x7f0e2400b350 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:44:57.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.588+0000 7f0e2be00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560350e4f520 0x560350e58c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:57.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.588+0000 7f0e2b400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560350e50200 0x560350e591b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:57.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.588+0000 7f0e31000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560350e58890 0x560350e596f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:57.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.588+0000 7f0e2be00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560350e4f520 0x560350e58c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:60584/0 (socket says 172.21.15.90:60584) 2024-04-02T14:44:57.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.588+0000 7f0e2be00640 1 -- 172.21.15.90:0/2411940877 learned_addr learned my addr 172.21.15.90:0/2411940877 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:44:57.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.588+0000 7f0e2be00640 1 -- 172.21.15.90:0/2411940877 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560350e58890 msgr2=0x560350e596f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:44:57.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.588+0000 7f0e2be00640 1 --2- 172.21.15.90:0/2411940877 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560350e58890 0x560350e596f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:57.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.588+0000 7f0e2be00640 1 -- 172.21.15.90:0/2411940877 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560350e50200 msgr2=0x560350e591b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:44:57.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.588+0000 7f0e2be00640 1 --2- 172.21.15.90:0/2411940877 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560350e50200 0x560350e591b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:57.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.588+0000 7f0e2be00640 1 -- 172.21.15.90:0/2411940877 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560350e5f6c0 con 0x560350e4f520 2024-04-02T14:44:57.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.590+0000 7f0e2be00640 1 --2- 172.21.15.90:0/2411940877 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560350e4f520 0x560350e58c70 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f0e14027040 tx=0x7f0e1400d520 comp rx=0 tx=0).ready entity=mon.1 client_cookie=31bf9ee16fb20803 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:44:57.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.591+0000 7f0e1a000640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:44:57.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.591+0000 7f0e1a000640 1 -- 172.21.15.90:0/2411940877 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e14029030 con 0x560350e4f520 2024-04-02T14:44:57.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.591+0000 7f0e1a000640 1 -- 172.21.15.90:0/2411940877 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8550+0+0 (secure 0 0 0) 0x7f0e14022040 con 0x560350e4f520 2024-04-02T14:44:57.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.591+0000 7f0e35c0c740 1 -- 172.21.15.90:0/2411940877 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0e1400be10 con 0x560350e4f520 2024-04-02T14:44:57.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.591+0000 7f0e1a000640 1 -- 172.21.15.90:0/2411940877 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f0e1401aa10 con 0x560350e4f520 2024-04-02T14:44:57.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.591+0000 7f0e35c0c740 10 client.28447 fetch_fsmap learned FSMap version 27 2024-04-02T14:44:57.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.591+0000 7f0e35c0c740 1 -- 172.21.15.90:0/2411940877 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x560350e609c0 con 0x560350e4f520 2024-04-02T14:44:57.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.592+0000 7f0e1a000640 1 -- 172.21.15.90:0/2411940877 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0e1400be10 con 0x560350e4f520 2024-04-02T14:44:57.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.592+0000 7f0e1a000640 1 -- 172.21.15.90:0/2411940877 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f0e14023630 con 0x560350e4f520 2024-04-02T14:44:57.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.592+0000 7f0e35c0c740 10 client.28447 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:44:57.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.592+0000 7f0e35c0c740 10 client.28447 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:44:57.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.592+0000 7f0e35c0c740 20 client.28447 populate_metadata read hostname 'smithi090' 2024-04-02T14:44:57.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.592+0000 7f0e35c0c740 1 --2- 172.21.15.90:0/2411940877 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x560350ecef60 0x560350ed1350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:57.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.592+0000 7f0e35c0c740 4 client.28447 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-02T14:44:57.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.592+0000 7f0e35c0c740 1 -- 172.21.15.90:0/2411940877 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "damage ls"}) -- 0x560350db8c30 con 0x560350ecef60 2024-04-02T14:44:57.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.592+0000 7f0e2b400640 1 --2- 172.21.15.90:0/2411940877 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x560350ecef60 0x560350ed1350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:57.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.593+0000 7f0e1a000640 10 client.28447 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:44:57.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.593+0000 7f0e2b400640 1 --2- 172.21.15.90:0/2411940877 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x560350ecef60 0x560350ed1350 crc :-1 s=READY pgs=267 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=feda84f8fd39b287 server_cookie=52974179c63bb695 in_seq=0 out_seq=0 2024-04-02T14:44:57.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.593+0000 7f0e1a000640 1 -- 172.21.15.90:0/2411940877 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x560350db8c30 con 0x560350ecef60 2024-04-02T14:44:57.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.593+0000 7f0e1a000640 10 client.28447 handle_command_reply: tid=0 2024-04-02T14:44:57.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.593+0000 7f0e35c0c740 1 client.28447 shutdown 2024-04-02T14:44:57.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.594+0000 7f0e35c0c740 1 -- 172.21.15.90:0/2411940877 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560350e4f520 msgr2=0x560350e58c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:57.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.594+0000 7f0e35c0c740 1 --2- 172.21.15.90:0/2411940877 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560350e4f520 0x560350e58c70 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f0e14027040 tx=0x7f0e1400d520 comp rx=0 tx=0).stop 2024-04-02T14:44:57.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.594+0000 7f0e35c0c740 1 -- 172.21.15.90:0/2411940877 shutdown_connections 2024-04-02T14:44:57.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.594+0000 7f0e35c0c740 1 --2- 172.21.15.90:0/2411940877 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x560350ecef60 0x560350ed1350 crc :-1 s=READY pgs=267 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:57.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.594+0000 7f0e35c0c740 1 --2- 172.21.15.90:0/2411940877 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560350e58890 0x560350e596f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:57.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.594+0000 7f0e35c0c740 1 --2- 172.21.15.90:0/2411940877 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560350e50200 0x560350e591b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:57.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.594+0000 7f0e35c0c740 1 --2- 172.21.15.90:0/2411940877 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560350e4f520 0x560350e58c70 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:57.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.594+0000 7f0e35c0c740 1 -- 172.21.15.90:0/2411940877 >> 172.21.15.90:0/2411940877 conn(0x560350e0da50 msgr2=0x560350e55910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:44:57.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.594+0000 7f0e1a000640 0 client.28447 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:44:57.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.594+0000 7f0e35c0c740 1 -- 172.21.15.90:0/2411940877 shutdown_connections 2024-04-02T14:44:57.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.595+0000 7f0e35c0c740 1 -- 172.21.15.90:0/2411940877 wait complete. 2024-04-02T14:44:57.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.595+0000 7f0e35c0c740 20 client.28447 trim_cache size 0 max 16384 2024-04-02T14:44:57.595 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-02T14:44:57.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.595+0000 7f0e33000640 1 -- 172.21.15.90:0/3645970646 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f0e100a3e60 msgr2=0x7f0e100a6320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:57.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.595+0000 7f0e33000640 1 --2- 172.21.15.90:0/3645970646 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f0e100a3e60 0x7f0e100a6320 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f0e1400de30 tx=0x7f0e1400f040 comp rx=0 tx=0).stop 2024-04-02T14:44:57.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.595+0000 7f0e33000640 1 -- 172.21.15.90:0/3645970646 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0e2c10f5e0 msgr2=0x7f0e2c1a6e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:57.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.595+0000 7f0e33000640 1 --2- 172.21.15.90:0/3645970646 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0e2c10f5e0 0x7f0e2c1a6e90 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f0e1c000c00 tx=0x7f0e1c0045e0 comp rx=0 tx=0).stop 2024-04-02T14:44:57.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.596+0000 7f0e33000640 1 -- 172.21.15.90:0/3645970646 shutdown_connections 2024-04-02T14:44:57.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.596+0000 7f0e33000640 1 --2- 172.21.15.90:0/3645970646 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f0e100a3e60 0x7f0e100a6320 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:57.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.596+0000 7f0e33000640 1 --2- 172.21.15.90:0/3645970646 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0e2c1a0b00 0x7f0e2c1a0fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:57.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.596+0000 7f0e33000640 1 --2- 172.21.15.90:0/3645970646 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0e2c10f5e0 0x7f0e2c1a6e90 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:57.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.596+0000 7f0e33000640 1 --2- 172.21.15.90:0/3645970646 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0e2c10a560 0x7f0e2c1a6950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:57.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.596+0000 7f0e33000640 1 -- 172.21.15.90:0/3645970646 >> 172.21.15.90:0/3645970646 conn(0x7f0e2c073090 msgr2=0x7f0e2c06e7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:44:57.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.596+0000 7f0e33000640 1 -- 172.21.15.90:0/3645970646 shutdown_connections 2024-04-02T14:44:57.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:57.596+0000 7f0e33000640 1 -- 172.21.15.90:0/3645970646 wait complete. 2024-04-02T14:44:57.931 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:44:57 smithi177 ceph-mon[36431]: pgmap v311: 97 pgs: 97 active+clean; 4.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 6.0 MiB/s wr, 575 op/s 2024-04-02T14:44:57.933 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:44:57 smithi090 ceph-mon[31674]: pgmap v311: 97 pgs: 97 active+clean; 4.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 6.0 MiB/s wr, 575 op/s 2024-04-02T14:44:58.105 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:44:57 smithi092 ceph-mon[36497]: pgmap v311: 97 pgs: 97 active+clean; 4.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 6.0 MiB/s wr, 575 op/s 2024-04-02T14:44:58.614 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-02T14:44:58.614 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T14:44:58.614 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-02T14:44:58.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:58.889+0000 7f49f9000640 1 -- 172.21.15.90:0/803416617 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f49f4069600 msgr2=0x7f49f410df60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:58.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:58.889+0000 7f49f9000640 1 --2- 172.21.15.90:0/803416617 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f49f4069600 0x7f49f410df60 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f49e0009900 tx=0x7f49e002bc30 comp rx=0 tx=0).stop 2024-04-02T14:44:58.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:58.890+0000 7f49f9000640 1 -- 172.21.15.90:0/803416617 shutdown_connections 2024-04-02T14:44:58.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:58.890+0000 7f49f9000640 1 --2- 172.21.15.90:0/803416617 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f49f410e4a0 0x7f49f4110890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:58.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:58.890+0000 7f49f9000640 1 --2- 172.21.15.90:0/803416617 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f49f4069600 0x7f49f410df60 secure :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f49e0009900 tx=0x7f49e002bc30 comp rx=0 tx=0).stop 2024-04-02T14:44:58.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:58.890+0000 7f49f9000640 1 --2- 172.21.15.90:0/803416617 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f49f4068ce0 0x7f49f40690c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:58.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:58.890+0000 7f49f9000640 1 -- 172.21.15.90:0/803416617 >> 172.21.15.90:0/803416617 conn(0x7f49f4077b00 msgr2=0x7f49f4077f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:44:58.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:58.893+0000 7f49f9000640 1 -- 172.21.15.90:0/803416617 shutdown_connections 2024-04-02T14:44:58.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:58.893+0000 7f49f9000640 1 -- 172.21.15.90:0/803416617 wait complete. 2024-04-02T14:44:58.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:58.897+0000 7f49f9000640 1 Processor -- start 2024-04-02T14:44:58.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:58.897+0000 7f49f9000640 1 -- start start 2024-04-02T14:44:58.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:58.897+0000 7f49f9000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f49f4068ce0 0x7f49f41a05a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:58.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:58.897+0000 7f49f9000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f49f410e4a0 0x7f49f41a0ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:58.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:58.897+0000 7f49f9000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f49f419a660 0x7f49f419aae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:58.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:58.897+0000 7f49f9000640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f49f4114130 con 0x7f49f419a660 2024-04-02T14:44:58.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:58.897+0000 7f49f9000640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f49f4113fb0 con 0x7f49f410e4a0 2024-04-02T14:44:58.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:58.897+0000 7f49f9000640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f49f41142b0 con 0x7f49f4068ce0 2024-04-02T14:44:58.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:58.898+0000 7f49f1400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f49f410e4a0 0x7f49f41a0ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:58.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:58.898+0000 7f49f1400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f49f410e4a0 0x7f49f41a0ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:40992/0 (socket says 172.21.15.90:40992) 2024-04-02T14:44:58.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:58.898+0000 7f49f1400640 1 -- 172.21.15.90:0/434754759 learned_addr learned my addr 172.21.15.90:0/434754759 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:44:58.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:58.898+0000 7f49f1400640 1 -- 172.21.15.90:0/434754759 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f49f4068ce0 msgr2=0x7f49f41a05a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T14:44:58.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:58.898+0000 7f49f1400640 1 --2- 172.21.15.90:0/434754759 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f49f4068ce0 0x7f49f41a05a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:58.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:58.898+0000 7f49f1400640 1 -- 172.21.15.90:0/434754759 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f49f419a660 msgr2=0x7f49f419aae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:58.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:58.898+0000 7f49f1400640 1 --2- 172.21.15.90:0/434754759 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f49f419a660 0x7f49f419aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:58.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:58.899+0000 7f49f1400640 1 -- 172.21.15.90:0/434754759 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49f419b260 con 0x7f49f410e4a0 2024-04-02T14:44:58.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:58.899+0000 7f49f1400640 1 --2- 172.21.15.90:0/434754759 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f49f410e4a0 0x7f49f41a0ae0 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f49e002c240 tx=0x7f49e0004430 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:44:58.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:58.900+0000 7f49eaa00640 1 -- 172.21.15.90:0/434754759 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49e0043020 con 0x7f49f410e4a0 2024-04-02T14:44:58.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:58.900+0000 7f49eaa00640 1 -- 172.21.15.90:0/434754759 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f49e00054e0 con 0x7f49f410e4a0 2024-04-02T14:44:58.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:58.900+0000 7f49eaa00640 1 -- 172.21.15.90:0/434754759 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49e0003bd0 con 0x7f49f410e4a0 2024-04-02T14:44:58.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:58.900+0000 7f49f9000640 1 -- 172.21.15.90:0/434754759 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49f419b4f0 con 0x7f49f410e4a0 2024-04-02T14:44:58.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:58.902+0000 7f49f9000640 1 -- 172.21.15.90:0/434754759 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49f41a7640 con 0x7f49f410e4a0 2024-04-02T14:44:58.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:58.908+0000 7f49eaa00640 1 -- 172.21.15.90:0/434754759 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f49e0046020 con 0x7f49f410e4a0 2024-04-02T14:44:58.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:58.915+0000 7f49eaa00640 1 --2- 172.21.15.90:0/434754759 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f49d80a3e70 0x7f49d80a6330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:58.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:58.915+0000 7f49eaa00640 1 -- 172.21.15.90:0/434754759 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8550+0+0 (secure 0 0 0) 0x7f49e00f26c0 con 0x7f49f410e4a0 2024-04-02T14:44:58.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:58.916+0000 7f49fbc5a740 1 Processor -- start 2024-04-02T14:44:58.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:58.917+0000 7f49fbc5a740 1 -- start start 2024-04-02T14:44:58.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:58.918+0000 7f49fbc5a740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564b09210030 0x564b0920ed80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:58.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:58.918+0000 7f49fbc5a740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564b0920f350 0x564b0920f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:58.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:58.918+0000 7f49fbc5a740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564b092186c0 0x564b0921aab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:58.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:58.918+0000 7f49fbc5a740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x564b09173310 con 0x564b0920f350 2024-04-02T14:44:58.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:58.918+0000 7f49fbc5a740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x564b0911b690 con 0x564b092186c0 2024-04-02T14:44:58.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:58.918+0000 7f49fbc5a740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x564b0916f2c0 con 0x564b09210030 2024-04-02T14:44:58.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:58.918+0000 7f49f1e00640 1 --2- 172.21.15.90:0/434754759 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f49d80a3e70 0x7f49d80a6330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:58.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:58.919+0000 7f49f2800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564b092186c0 0x564b0921aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:58.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:58.919+0000 7f49f2800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564b092186c0 0x564b0921aab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:41008/0 (socket says 172.21.15.90:41008) 2024-04-02T14:44:58.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:58.919+0000 7f49f2800640 1 -- 172.21.15.90:0/876182082 learned_addr learned my addr 172.21.15.90:0/876182082 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:44:58.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:58.919+0000 7f49f1400640 1 --2- 172.21.15.90:0/876182082 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564b0920f350 0x564b0920f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:58.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:58.920+0000 7f49f1e00640 1 --2- 172.21.15.90:0/876182082 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564b09210030 0x564b0920ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:58.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:58.920+0000 7f49f1400640 1 -- 172.21.15.90:0/876182082 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564b09210030 msgr2=0x564b0920ed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:44:58.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:58.921+0000 7f49f1400640 1 --2- 172.21.15.90:0/876182082 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564b09210030 0x564b0920ed80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:58.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:58.921+0000 7f49f1400640 1 -- 172.21.15.90:0/876182082 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564b092186c0 msgr2=0x564b0921aab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:44:58.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:58.921+0000 7f49f1400640 1 --2- 172.21.15.90:0/876182082 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564b092186c0 0x564b0921aab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:58.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:58.921+0000 7f49f1400640 1 -- 172.21.15.90:0/876182082 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564b08f4f7e0 con 0x564b0920f350 2024-04-02T14:44:58.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:58.921+0000 7f49f2800640 1 --2- 172.21.15.90:0/876182082 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564b092186c0 0x564b0921aab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:44:58.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:58.921+0000 7f49f1e00640 1 --2- 172.21.15.90:0/876182082 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564b09210030 0x564b0920ed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:44:58.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:58.923+0000 7f49f1e00640 1 --2- 172.21.15.90:0/434754759 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f49d80a3e70 0x7f49d80a6330 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f49e400bbb0 tx=0x7f49e400b3a0 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:44:58.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:58.923+0000 7f49f1400640 1 --2- 172.21.15.90:0/876182082 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564b0920f350 0x564b0920f7d0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f49e00eff00 tx=0x7f49e00ba3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95872d04651eb5b9 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:44:58.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:58.924+0000 7f49df400640 1 -- 172.21.15.90:0/876182082 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49e0004c80 con 0x564b0920f350 2024-04-02T14:44:58.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:58.924+0000 7f49df400640 1 -- 172.21.15.90:0/876182082 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f49e0005690 con 0x564b0920f350 2024-04-02T14:44:58.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:58.924+0000 7f49df400640 1 -- 172.21.15.90:0/876182082 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49e002cc20 con 0x564b0920f350 2024-04-02T14:44:58.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:58.925+0000 7f49fbc5a740 1 -- 172.21.15.90:0/876182082 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564b0920f350 msgr2=0x564b0920f7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:58.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:58.925+0000 7f49fbc5a740 1 --2- 172.21.15.90:0/876182082 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564b0920f350 0x564b0920f7d0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f49e00eff00 tx=0x7f49e00ba3f0 comp rx=0 tx=0).stop 2024-04-02T14:44:58.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:58.927+0000 7f49fbc5a740 1 -- 172.21.15.90:0/876182082 shutdown_connections 2024-04-02T14:44:58.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:58.928+0000 7f49fbc5a740 1 --2- 172.21.15.90:0/876182082 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564b092186c0 0x564b0921aab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:58.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:58.928+0000 7f49fbc5a740 1 --2- 172.21.15.90:0/876182082 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564b0920f350 0x564b0920f7d0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:58.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:58.928+0000 7f49fbc5a740 1 --2- 172.21.15.90:0/876182082 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564b09210030 0x564b0920ed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:58.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:58.928+0000 7f49fbc5a740 1 -- 172.21.15.90:0/876182082 >> 172.21.15.90:0/876182082 conn(0x564b091a3270 msgr2=0x564b09209da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:44:58.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:58.931+0000 7f49fbc5a740 1 -- 172.21.15.90:0/876182082 shutdown_connections 2024-04-02T14:44:58.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:58.931+0000 7f49fbc5a740 1 -- 172.21.15.90:0/876182082 wait complete. 2024-04-02T14:44:58.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:58.933+0000 7f49fbc5a740 1 -- start start 2024-04-02T14:44:58.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:58.933+0000 7f49fbc5a740 1 Processor -- start 2024-04-02T14:44:58.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:58.934+0000 7f49fbc5a740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564b0920f350 0x564b092156e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:58.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:58.934+0000 7f49fbc5a740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564b09210030 0x564b09215c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:58.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:58.934+0000 7f49fbc5a740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564b092186c0 0x564b09216160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:58.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:58.934+0000 7f49fbc5a740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x564b0921b2f0 con 0x564b09210030 2024-04-02T14:44:58.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:58.934+0000 7f49fbc5a740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x564b0912ea60 con 0x564b092186c0 2024-04-02T14:44:58.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:58.934+0000 7f49fbc5a740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x564b0921b470 con 0x564b0920f350 2024-04-02T14:44:58.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:58.934+0000 7f49fbc5a740 10 client.0 _ll_register_callbacks cb 0x564b09134030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:44:58.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:58.934+0000 7f49f1e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564b0920f350 0x564b092156e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:58.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:58.934+0000 7f49f1e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564b0920f350 0x564b092156e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:60602/0 (socket says 172.21.15.90:60602) 2024-04-02T14:44:58.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:58.934+0000 7f49f1e00640 1 -- 172.21.15.90:0/3324854097 learned_addr learned my addr 172.21.15.90:0/3324854097 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:44:58.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:58.935+0000 7f49f1e00640 1 -- 172.21.15.90:0/3324854097 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564b092186c0 msgr2=0x564b09216160 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-02T14:44:58.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:58.935+0000 7f49f1e00640 1 --2- 172.21.15.90:0/3324854097 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564b092186c0 0x564b09216160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:58.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:58.935+0000 7f49f1e00640 1 -- 172.21.15.90:0/3324854097 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564b09210030 msgr2=0x564b09215c20 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-02T14:44:58.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:58.935+0000 7f49f1400640 1 --2- 172.21.15.90:0/3324854097 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564b09210030 0x564b09215c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:58.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:58.936+0000 7f49f1e00640 1 --2- 172.21.15.90:0/3324854097 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564b09210030 0x564b09215c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:58.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:58.936+0000 7f49f1e00640 1 -- 172.21.15.90:0/3324854097 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564b08faff70 con 0x564b0920f350 2024-04-02T14:44:58.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:58.936+0000 7f49f1400640 1 --2- 172.21.15.90:0/3324854097 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564b09210030 0x564b09215c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:44:58.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:58.937+0000 7f49f1e00640 1 --2- 172.21.15.90:0/3324854097 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564b0920f350 0x564b092156e0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f49e400dfe0 tx=0x7f49e4004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6fabc8822f44365e server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:44:58.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:58.937+0000 7f49d3e00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:44:58.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:58.938+0000 7f49d3e00640 1 -- 172.21.15.90:0/3324854097 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49e4022020 con 0x564b0920f350 2024-04-02T14:44:58.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:58.938+0000 7f49fbc5a740 1 -- 172.21.15.90:0/3324854097 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x564b09180850 con 0x564b0920f350 2024-04-02T14:44:58.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:58.938+0000 7f49d3e00640 1 -- 172.21.15.90:0/3324854097 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8550+0+0 (secure 0 0 0) 0x7f49e401e640 con 0x564b0920f350 2024-04-02T14:44:58.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:58.939+0000 7f49d3e00640 1 -- 172.21.15.90:0/3324854097 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f49e4011e30 con 0x564b0920f350 2024-04-02T14:44:58.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:58.939+0000 7f49fbc5a740 10 client.28453 fetch_fsmap learned FSMap version 27 2024-04-02T14:44:58.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:58.939+0000 7f49fbc5a740 1 -- 172.21.15.90:0/3324854097 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x564b08e47170 con 0x564b0920f350 2024-04-02T14:44:58.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:58.940+0000 7f49d3e00640 1 -- 172.21.15.90:0/3324854097 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x564b09180850 con 0x564b0920f350 2024-04-02T14:44:58.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:58.940+0000 7f49d3e00640 1 -- 172.21.15.90:0/3324854097 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f49e402f020 con 0x564b0920f350 2024-04-02T14:44:58.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:58.940+0000 7f49fbc5a740 10 client.28453 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:44:58.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:58.940+0000 7f49fbc5a740 10 client.28453 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:44:58.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:58.940+0000 7f49fbc5a740 20 client.28453 populate_metadata read hostname 'smithi090' 2024-04-02T14:44:58.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:58.940+0000 7f49fbc5a740 1 --2- 172.21.15.90:0/3324854097 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x564b09242070 0x564b09244460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:58.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:58.940+0000 7f49fbc5a740 4 client.28453 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:44:58.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:58.940+0000 7f49fbc5a740 1 -- 172.21.15.90:0/3324854097 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564b0917aba0 con 0x564b09242070 2024-04-02T14:44:58.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:58.941+0000 7f49f1400640 1 --2- 172.21.15.90:0/3324854097 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x564b09242070 0x564b09244460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:58.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:58.942+0000 7f49f1400640 1 --2- 172.21.15.90:0/3324854097 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x564b09242070 0x564b09244460 crc :-1 s=READY pgs=268 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b38a7eb6c4e30b41 server_cookie=7041c6fd1bd8b786 in_seq=0 out_seq=0 2024-04-02T14:44:58.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:58.942+0000 7f49d3e00640 10 client.28453 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:44:58.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:58.943+0000 7f49d3e00640 1 -- 172.21.15.90:0/3324854097 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x564b0917aba0 con 0x564b09242070 2024-04-02T14:44:58.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:58.943+0000 7f49d3e00640 10 client.28453 handle_command_reply: tid=0 2024-04-02T14:44:58.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:58.943+0000 7f49fbc5a740 1 client.28453 shutdown 2024-04-02T14:44:58.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:58.944+0000 7f49fbc5a740 1 -- 172.21.15.90:0/3324854097 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564b0920f350 msgr2=0x564b092156e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:58.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:58.944+0000 7f49fbc5a740 1 --2- 172.21.15.90:0/3324854097 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564b0920f350 0x564b092156e0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f49e400dfe0 tx=0x7f49e4004730 comp rx=0 tx=0).stop 2024-04-02T14:44:58.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:58.944+0000 7f49fbc5a740 1 -- 172.21.15.90:0/3324854097 shutdown_connections 2024-04-02T14:44:58.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:58.944+0000 7f49fbc5a740 1 --2- 172.21.15.90:0/3324854097 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x564b09242070 0x564b09244460 crc :-1 s=READY pgs=268 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:58.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:58.944+0000 7f49d3e00640 0 client.28453 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:44:58.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:58.944+0000 7f49fbc5a740 1 --2- 172.21.15.90:0/3324854097 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564b092186c0 0x564b09216160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:58.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:58.945+0000 7f49fbc5a740 1 --2- 172.21.15.90:0/3324854097 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564b09210030 0x564b09215c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:58.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:58.945+0000 7f49fbc5a740 1 --2- 172.21.15.90:0/3324854097 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564b0920f350 0x564b092156e0 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:58.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:58.945+0000 7f49fbc5a740 1 -- 172.21.15.90:0/3324854097 >> 172.21.15.90:0/3324854097 conn(0x564b091a3270 msgr2=0x564b0920baf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:44:58.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:58.946+0000 7f49fbc5a740 1 -- 172.21.15.90:0/3324854097 shutdown_connections 2024-04-02T14:44:58.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:58.946+0000 7f49fbc5a740 1 -- 172.21.15.90:0/3324854097 wait complete. 2024-04-02T14:44:58.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:58.946+0000 7f49fbc5a740 20 client.28453 trim_cache size 0 max 16384 2024-04-02T14:44:58.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:58.988+0000 7f49fbc5a740 1 Processor -- start 2024-04-02T14:44:58.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:58.988+0000 7f49fbc5a740 1 -- start start 2024-04-02T14:44:58.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:58.989+0000 7f49fbc5a740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564b0920f350 0x564b09217e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:58.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:58.989+0000 7f49fbc5a740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564b09210030 0x564b09249cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:58.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:58.989+0000 7f49fbc5a740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564b092186c0 0x564b0924a200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:58.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:58.989+0000 7f49fbc5a740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f49f406fa40 con 0x564b0920f350 2024-04-02T14:44:58.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:58.989+0000 7f49fbc5a740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f49f4071580 con 0x564b09210030 2024-04-02T14:44:58.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:58.989+0000 7f49fbc5a740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f49f4070110 con 0x564b092186c0 2024-04-02T14:44:58.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:58.989+0000 7f49f2800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564b092186c0 0x564b0924a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:58.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:58.989+0000 7f49f2800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564b092186c0 0x564b0924a200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:60610/0 (socket says 172.21.15.90:60610) 2024-04-02T14:44:58.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:58.989+0000 7f49f1400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564b09210030 0x564b09249cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:58.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:58.989+0000 7f49f1400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564b09210030 0x564b09249cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:41024/0 (socket says 172.21.15.90:41024) 2024-04-02T14:44:58.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:58.989+0000 7f49f1400640 1 -- 172.21.15.90:0/712208343 learned_addr learned my addr 172.21.15.90:0/712208343 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:44:58.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:58.989+0000 7f49f1e00640 1 --2- 172.21.15.90:0/712208343 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564b0920f350 0x564b09217e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:58.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:58.991+0000 7f49f1400640 1 -- 172.21.15.90:0/712208343 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564b092186c0 msgr2=0x564b0924a200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:44:58.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:58.991+0000 7f49f1400640 1 --2- 172.21.15.90:0/712208343 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564b092186c0 0x564b0924a200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:58.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:58.991+0000 7f49f1400640 1 -- 172.21.15.90:0/712208343 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564b0920f350 msgr2=0x564b09217e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:44:58.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:58.991+0000 7f49f1400640 1 --2- 172.21.15.90:0/712208343 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564b0920f350 0x564b09217e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:58.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:58.991+0000 7f49f1400640 1 -- 172.21.15.90:0/712208343 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564b09244a90 con 0x564b09210030 2024-04-02T14:44:58.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:58.991+0000 7f49f1e00640 1 --2- 172.21.15.90:0/712208343 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564b0920f350 0x564b09217e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:44:58.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:58.992+0000 7f49f1400640 1 --2- 172.21.15.90:0/712208343 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564b09210030 0x564b09249cc0 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f49e0004e00 tx=0x7f49e0004ee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2f3fdb8b55ce8c97 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:44:58.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:58.992+0000 7f49dea00640 1 -- 172.21.15.90:0/712208343 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49e00be0b0 con 0x564b09210030 2024-04-02T14:44:58.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:58.993+0000 7f49dea00640 1 -- 172.21.15.90:0/712208343 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f49e00be750 con 0x564b09210030 2024-04-02T14:44:58.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:58.994+0000 7f49fbc5a740 1 -- 172.21.15.90:0/712208343 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564b09210030 msgr2=0x564b09249cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:58.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:58.994+0000 7f49fbc5a740 1 --2- 172.21.15.90:0/712208343 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564b09210030 0x564b09249cc0 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f49e0004e00 tx=0x7f49e0004ee0 comp rx=0 tx=0).stop 2024-04-02T14:44:58.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:58.994+0000 7f49fbc5a740 1 -- 172.21.15.90:0/712208343 shutdown_connections 2024-04-02T14:44:58.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:58.994+0000 7f49fbc5a740 1 --2- 172.21.15.90:0/712208343 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564b092186c0 0x564b0924a200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:58.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:58.994+0000 7f49fbc5a740 1 --2- 172.21.15.90:0/712208343 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564b09210030 0x564b09249cc0 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:58.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:58.994+0000 7f49fbc5a740 1 --2- 172.21.15.90:0/712208343 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564b0920f350 0x564b09217e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:58.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:58.994+0000 7f49fbc5a740 1 -- 172.21.15.90:0/712208343 >> 172.21.15.90:0/712208343 conn(0x564b091a3270 msgr2=0x564b09208e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:44:58.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:58.995+0000 7f49fbc5a740 1 -- 172.21.15.90:0/712208343 shutdown_connections 2024-04-02T14:44:58.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:58.995+0000 7f49fbc5a740 1 -- 172.21.15.90:0/712208343 wait complete. 2024-04-02T14:44:58.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:58.996+0000 7f49fbc5a740 1 -- start start 2024-04-02T14:44:58.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:58.996+0000 7f49fbc5a740 1 Processor -- start 2024-04-02T14:44:58.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:58.999+0000 7f49fbc5a740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564b0920f350 0x564b092a5520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:58.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.000+0000 7f49fbc5a740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564b09210030 0x564b092a5a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:58.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.000+0000 7f49fbc5a740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564b092186c0 0x564b09211970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:58.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.000+0000 7f49fbc5a740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f49f4070eb0 con 0x564b092186c0 2024-04-02T14:44:58.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.000+0000 7f49fbc5a740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f49f40707e0 con 0x564b0920f350 2024-04-02T14:44:58.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.000+0000 7f49fbc5a740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f49f406d9c0 con 0x564b09210030 2024-04-02T14:44:58.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.000+0000 7f49fbc5a740 10 client.0 _ll_register_callbacks cb 0x7f49e400dc90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:44:58.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.000+0000 7f49f1400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564b09210030 0x564b092a5a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:58.997 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.000+0000 7f49f1e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564b0920f350 0x564b092a5520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:58.997 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.000+0000 7f49f1400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564b09210030 0x564b092a5a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:60620/0 (socket says 172.21.15.90:60620) 2024-04-02T14:44:58.997 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.000+0000 7f49f1400640 1 -- 172.21.15.90:0/4185174636 learned_addr learned my addr 172.21.15.90:0/4185174636 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:44:58.998 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.000+0000 7f49f2800640 1 --2- 172.21.15.90:0/4185174636 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564b092186c0 0x564b09211970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:58.998 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.001+0000 7f49f1400640 1 -- 172.21.15.90:0/4185174636 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564b0920f350 msgr2=0x564b092a5520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:44:58.998 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.001+0000 7f49f1400640 1 --2- 172.21.15.90:0/4185174636 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564b0920f350 0x564b092a5520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:58.998 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.001+0000 7f49f1400640 1 -- 172.21.15.90:0/4185174636 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564b092186c0 msgr2=0x564b09211970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:44:58.998 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.001+0000 7f49f1400640 1 --2- 172.21.15.90:0/4185174636 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564b092186c0 0x564b09211970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:58.998 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.001+0000 7f49f1400640 1 -- 172.21.15.90:0/4185174636 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564b09211eb0 con 0x564b09210030 2024-04-02T14:44:58.998 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.002+0000 7f49f1400640 1 --2- 172.21.15.90:0/4185174636 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564b09210030 0x564b092a5a60 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f49e00bca00 tx=0x7f49e00035f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=587caa88df4ec7f7 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:44:58.998 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.002+0000 7f49d3e00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:44:58.998 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.002+0000 7f49d3e00640 1 -- 172.21.15.90:0/4185174636 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49e00baf60 con 0x564b09210030 2024-04-02T14:44:58.998 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.002+0000 7f49fbc5a740 1 -- 172.21.15.90:0/4185174636 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f49e00be750 con 0x564b09210030 2024-04-02T14:44:58.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.002+0000 7f49d3e00640 1 -- 172.21.15.90:0/4185174636 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8550+0+0 (secure 0 0 0) 0x7f49e00be9b0 con 0x564b09210030 2024-04-02T14:44:58.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.003+0000 7f49d3e00640 1 -- 172.21.15.90:0/4185174636 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f49e00be0b0 con 0x564b09210030 2024-04-02T14:44:59.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.003+0000 7f49fbc5a740 10 client.28459 fetch_fsmap learned FSMap version 27 2024-04-02T14:44:59.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.003+0000 7f49fbc5a740 1 -- 172.21.15.90:0/4185174636 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x564b092131f0 con 0x564b09210030 2024-04-02T14:44:59.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.004+0000 7f49d3e00640 1 -- 172.21.15.90:0/4185174636 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f49e00be750 con 0x564b09210030 2024-04-02T14:44:59.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.004+0000 7f49d3e00640 1 -- 172.21.15.90:0/4185174636 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f49e00c6f50 con 0x564b09210030 2024-04-02T14:44:59.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.004+0000 7f49fbc5a740 10 client.28459 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:44:59.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.004+0000 7f49fbc5a740 10 client.28459 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:44:59.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.004+0000 7f49fbc5a740 20 client.28459 populate_metadata read hostname 'smithi090' 2024-04-02T14:44:59.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.004+0000 7f49fbc5a740 1 --2- 172.21.15.90:0/4185174636 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x564b09213350 0x564b092266f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:59.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.004+0000 7f49fbc5a740 4 client.28459 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-02T14:44:59.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.004+0000 7f49fbc5a740 1 -- 172.21.15.90:0/4185174636 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x564b090f8aa0 con 0x564b09213350 2024-04-02T14:44:59.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.005+0000 7f49f1e00640 1 --2- 172.21.15.90:0/4185174636 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x564b09213350 0x564b092266f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:59.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.006+0000 7f49f1e00640 1 --2- 172.21.15.90:0/4185174636 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x564b09213350 0x564b092266f0 crc :-1 s=READY pgs=269 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=50840b4ad7f3fdde server_cookie=375dbe13f95de9df in_seq=0 out_seq=0 2024-04-02T14:44:59.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.006+0000 7f49d3e00640 10 client.28459 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:44:59.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.007+0000 7f49d3e00640 1 -- 172.21.15.90:0/4185174636 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x564b090f8aa0 con 0x564b09213350 2024-04-02T14:44:59.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.007+0000 7f49d3e00640 10 client.28459 handle_command_reply: tid=0 2024-04-02T14:44:59.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.008+0000 7f49fbc5a740 1 client.28459 shutdown 2024-04-02T14:44:59.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.009+0000 7f49fbc5a740 1 -- 172.21.15.90:0/4185174636 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564b09210030 msgr2=0x564b092a5a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:59.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.009+0000 7f49fbc5a740 1 --2- 172.21.15.90:0/4185174636 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564b09210030 0x564b092a5a60 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f49e00bca00 tx=0x7f49e00035f0 comp rx=0 tx=0).stop 2024-04-02T14:44:59.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.009+0000 7f49fbc5a740 1 -- 172.21.15.90:0/4185174636 shutdown_connections 2024-04-02T14:44:59.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.009+0000 7f49fbc5a740 1 --2- 172.21.15.90:0/4185174636 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x564b09213350 0x564b092266f0 crc :-1 s=READY pgs=269 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:59.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.009+0000 7f49fbc5a740 1 --2- 172.21.15.90:0/4185174636 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564b092186c0 0x564b09211970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:59.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.009+0000 7f49fbc5a740 1 --2- 172.21.15.90:0/4185174636 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564b09210030 0x564b092a5a60 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:59.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.009+0000 7f49fbc5a740 1 --2- 172.21.15.90:0/4185174636 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564b0920f350 0x564b092a5520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:59.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.009+0000 7f49d3e00640 0 client.28459 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:44:59.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.009+0000 7f49fbc5a740 1 -- 172.21.15.90:0/4185174636 >> 172.21.15.90:0/4185174636 conn(0x564b091a3270 msgr2=0x564b0920afd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:44:59.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.010+0000 7f49fbc5a740 1 -- 172.21.15.90:0/4185174636 shutdown_connections 2024-04-02T14:44:59.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.010+0000 7f49fbc5a740 1 -- 172.21.15.90:0/4185174636 wait complete. 2024-04-02T14:44:59.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.010+0000 7f49fbc5a740 20 client.28459 trim_cache size 0 max 16384 2024-04-02T14:44:59.006 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-02T14:44:59.006 INFO:teuthology.orchestra.run.smithi090.stdout: "return_code": 0, 2024-04-02T14:44:59.006 INFO:teuthology.orchestra.run.smithi090.stdout: "scrub_tag": "0f2eb92d-5324-4420-99e2-479e4213fae2", 2024-04-02T14:44:59.006 INFO:teuthology.orchestra.run.smithi090.stdout: "mode": "asynchronous" 2024-04-02T14:44:59.007 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-02T14:44:59.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.011+0000 7f49f9000640 1 -- 172.21.15.90:0/434754759 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f49d80a3e70 msgr2=0x7f49d80a6330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:59.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.011+0000 7f49f9000640 1 --2- 172.21.15.90:0/434754759 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f49d80a3e70 0x7f49d80a6330 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f49e400bbb0 tx=0x7f49e400b3a0 comp rx=0 tx=0).stop 2024-04-02T14:44:59.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.011+0000 7f49f9000640 1 -- 172.21.15.90:0/434754759 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f49f410e4a0 msgr2=0x7f49f41a0ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:59.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.011+0000 7f49f9000640 1 --2- 172.21.15.90:0/434754759 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f49f410e4a0 0x7f49f41a0ae0 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f49e002c240 tx=0x7f49e0004430 comp rx=0 tx=0).stop 2024-04-02T14:44:59.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.011+0000 7f49f9000640 1 -- 172.21.15.90:0/434754759 shutdown_connections 2024-04-02T14:44:59.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.011+0000 7f49f9000640 1 --2- 172.21.15.90:0/434754759 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f49d80a3e70 0x7f49d80a6330 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:59.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.011+0000 7f49f9000640 1 --2- 172.21.15.90:0/434754759 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f49f419a660 0x7f49f419aae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:59.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.011+0000 7f49f9000640 1 --2- 172.21.15.90:0/434754759 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f49f410e4a0 0x7f49f41a0ae0 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:59.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.011+0000 7f49f9000640 1 --2- 172.21.15.90:0/434754759 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f49f4068ce0 0x7f49f41a05a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:59.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.011+0000 7f49f9000640 1 -- 172.21.15.90:0/434754759 >> 172.21.15.90:0/434754759 conn(0x7f49f4077b00 msgr2=0x7f49f410fb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:44:59.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.011+0000 7f49f9000640 1 -- 172.21.15.90:0/434754759 shutdown_connections 2024-04-02T14:44:59.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.011+0000 7f49f9000640 1 -- 172.21.15.90:0/434754759 wait complete. 2024-04-02T14:44:59.029 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-02T14:44:59.029 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-02T14:44:59.224 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.228+0000 7fdbfbc00640 1 -- 172.21.15.90:0/1745640238 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdbf4103f40 msgr2=0x7fdbf4104320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:59.224 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.228+0000 7fdbfbc00640 1 --2- 172.21.15.90:0/1745640238 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdbf4103f40 0x7fdbf4104320 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7fdbe8009900 tx=0x7fdbe802bc30 comp rx=0 tx=0).stop 2024-04-02T14:44:59.224 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.228+0000 7fdbfbc00640 1 -- 172.21.15.90:0/1745640238 shutdown_connections 2024-04-02T14:44:59.225 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.228+0000 7fdbfbc00640 1 --2- 172.21.15.90:0/1745640238 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdbf4109010 0x7fdbf4110be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:59.225 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.228+0000 7fdbfbc00640 1 --2- 172.21.15.90:0/1745640238 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdbf41048f0 0x7fdbf4108930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:59.225 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.228+0000 7fdbfbc00640 1 --2- 172.21.15.90:0/1745640238 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdbf4103f40 0x7fdbf4104320 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:59.225 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.228+0000 7fdbfbc00640 1 -- 172.21.15.90:0/1745640238 >> 172.21.15.90:0/1745640238 conn(0x7fdbf40ff7c0 msgr2=0x7fdbf4101be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:44:59.225 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.228+0000 7fdbfbc00640 1 -- 172.21.15.90:0/1745640238 shutdown_connections 2024-04-02T14:44:59.225 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.229+0000 7fdbfbc00640 1 -- 172.21.15.90:0/1745640238 wait complete. 2024-04-02T14:44:59.225 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.230+0000 7fdbfbc00640 1 Processor -- start 2024-04-02T14:44:59.226 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.230+0000 7fdbfbc00640 1 -- start start 2024-04-02T14:44:59.227 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.231+0000 7fdbfbc00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdbf4109010 0x7fdbf41106e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:59.227 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.231+0000 7fdbfbc00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdbf4110c20 0x7fdbf4073580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:59.227 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.232+0000 7fdbfbc00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdbf410e830 0x7fdbf410ec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:59.227 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.232+0000 7fdbfbc00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fdbf40762d0 con 0x7fdbf4109010 2024-04-02T14:44:59.228 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.232+0000 7fdbf3e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdbf4110c20 0x7fdbf4073580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:59.228 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.232+0000 7fdbf3e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdbf4110c20 0x7fdbf4073580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:60630/0 (socket says 172.21.15.90:60630) 2024-04-02T14:44:59.228 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.232+0000 7fdbf3e00640 1 -- 172.21.15.90:0/293184469 learned_addr learned my addr 172.21.15.90:0/293184469 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:44:59.228 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.232+0000 7fdbfbc00640 1 -- 172.21.15.90:0/293184469 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fdbf4076150 con 0x7fdbf410e830 2024-04-02T14:44:59.228 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.232+0000 7fdbfbc00640 1 -- 172.21.15.90:0/293184469 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fdbf4076450 con 0x7fdbf4110c20 2024-04-02T14:44:59.228 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.232+0000 7fdbf9c00640 1 --2- 172.21.15.90:0/293184469 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdbf410e830 0x7fdbf410ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:59.229 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.233+0000 7fdbf9200640 1 --2- 172.21.15.90:0/293184469 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdbf4109010 0x7fdbf41106e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:59.229 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.233+0000 7fdbf3e00640 1 -- 172.21.15.90:0/293184469 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdbf410e830 msgr2=0x7fdbf410ec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:59.229 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.233+0000 7fdbf3e00640 1 --2- 172.21.15.90:0/293184469 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdbf410e830 0x7fdbf410ec90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:59.229 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.233+0000 7fdbf3e00640 1 -- 172.21.15.90:0/293184469 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdbf4109010 msgr2=0x7fdbf41106e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:59.229 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.233+0000 7fdbf3e00640 1 --2- 172.21.15.90:0/293184469 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdbf4109010 0x7fdbf41106e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:59.230 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.233+0000 7fdbf3e00640 1 -- 172.21.15.90:0/293184469 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdbf410f330 con 0x7fdbf4110c20 2024-04-02T14:44:59.230 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.233+0000 7fdbf9c00640 1 --2- 172.21.15.90:0/293184469 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdbf410e830 0x7fdbf410ec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:44:59.230 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.233+0000 7fdbf9200640 1 --2- 172.21.15.90:0/293184469 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdbf4109010 0x7fdbf41106e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:44:59.230 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.234+0000 7fdbf3e00640 1 --2- 172.21.15.90:0/293184469 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdbf4110c20 0x7fdbf4073580 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7fdbe000e9a0 tx=0x7fdbe000ee60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:44:59.230 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.235+0000 7fdbf1600640 1 -- 172.21.15.90:0/293184469 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbe000cd80 con 0x7fdbf4110c20 2024-04-02T14:44:59.231 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.235+0000 7fdbf1600640 1 -- 172.21.15.90:0/293184469 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdbe00035f0 con 0x7fdbf4110c20 2024-04-02T14:44:59.231 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.235+0000 7fdbf1600640 1 -- 172.21.15.90:0/293184469 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbe0010670 con 0x7fdbf4110c20 2024-04-02T14:44:59.231 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.235+0000 7fdbfbc00640 1 -- 172.21.15.90:0/293184469 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdbf410f620 con 0x7fdbf4110c20 2024-04-02T14:44:59.233 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.237+0000 7fdbfbc00640 1 -- 172.21.15.90:0/293184469 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdbf41b45a0 con 0x7fdbf4110c20 2024-04-02T14:44:59.239 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.243+0000 7fdbfe9d4740 1 Processor -- start 2024-04-02T14:44:59.239 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.243+0000 7fdbf1600640 1 -- 172.21.15.90:0/293184469 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fdbe00037a0 con 0x7fdbf4110c20 2024-04-02T14:44:59.239 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.244+0000 7fdbfe9d4740 1 -- start start 2024-04-02T14:44:59.241 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.244+0000 7fdbfe9d4740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561fee0c0eb0 0x561fee0c1290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:59.241 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.245+0000 7fdbfe9d4740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561fee0c17d0 0x561fee0c9e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:59.241 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.245+0000 7fdbfe9d4740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561fee0ca3b0 0x561fee0cc7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:59.241 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.245+0000 7fdbfe9d4740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x561fee025d70 con 0x561fee0c17d0 2024-04-02T14:44:59.241 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.245+0000 7fdbfe9d4740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x561fedfce2a0 con 0x561fee0c0eb0 2024-04-02T14:44:59.242 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.245+0000 7fdbfe9d4740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x561fee021d20 con 0x561fee0ca3b0 2024-04-02T14:44:59.242 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.245+0000 7fdbf3e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561fee0ca3b0 0x561fee0cc7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:59.242 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.245+0000 7fdbf3e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561fee0ca3b0 0x561fee0cc7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:60634/0 (socket says 172.21.15.90:60634) 2024-04-02T14:44:59.242 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.245+0000 7fdbf3e00640 1 -- 172.21.15.90:0/683039665 learned_addr learned my addr 172.21.15.90:0/683039665 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:44:59.242 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.245+0000 7fdbf9c00640 1 --2- 172.21.15.90:0/683039665 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561fee0c0eb0 0x561fee0c1290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:59.242 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.245+0000 7fdbf9200640 1 --2- 172.21.15.90:0/683039665 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561fee0c17d0 0x561fee0c9e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:59.244 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.246+0000 7fdbf3e00640 1 -- 172.21.15.90:0/683039665 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561fee0c0eb0 msgr2=0x561fee0c1290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:44:59.244 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.246+0000 7fdbf3e00640 1 --2- 172.21.15.90:0/683039665 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561fee0c0eb0 0x561fee0c1290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:59.244 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.246+0000 7fdbf3e00640 1 -- 172.21.15.90:0/683039665 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561fee0c17d0 msgr2=0x561fee0c9e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:44:59.244 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.246+0000 7fdbf3e00640 1 --2- 172.21.15.90:0/683039665 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561fee0c17d0 0x561fee0c9e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:59.245 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.246+0000 7fdbf3e00640 1 -- 172.21.15.90:0/683039665 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561fedfe2840 con 0x561fee0ca3b0 2024-04-02T14:44:59.245 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.246+0000 7fdbf1600640 1 --2- 172.21.15.90:0/293184469 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fdbdc0a3f70 0x7fdbdc0a6430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:59.245 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.246+0000 7fdbf1600640 1 -- 172.21.15.90:0/293184469 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8550+0+0 (secure 0 0 0) 0x7fdbe00d09f0 con 0x7fdbf4110c20 2024-04-02T14:44:59.245 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.246+0000 7fdbf3e00640 1 --2- 172.21.15.90:0/683039665 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561fee0ca3b0 0x561fee0cc7a0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7fdbe00ce090 tx=0x7fdbe00d4e80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2e9bef9dc4b18f22 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:44:59.246 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.247+0000 7fdbf9c00640 1 --2- 172.21.15.90:0/293184469 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fdbdc0a3f70 0x7fdbdc0a6430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:59.246 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.247+0000 7fdbe6000640 1 -- 172.21.15.90:0/683039665 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbe0098c10 con 0x561fee0ca3b0 2024-04-02T14:44:59.246 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.247+0000 7fdbe6000640 1 -- 172.21.15.90:0/683039665 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdbe009d070 con 0x561fee0ca3b0 2024-04-02T14:44:59.246 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.247+0000 7fdbe6000640 1 -- 172.21.15.90:0/683039665 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbe009c2d0 con 0x561fee0ca3b0 2024-04-02T14:44:59.246 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.248+0000 7fdbfe9d4740 1 -- 172.21.15.90:0/683039665 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561fee0ca3b0 msgr2=0x561fee0cc7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:59.246 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.248+0000 7fdbfe9d4740 1 --2- 172.21.15.90:0/683039665 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561fee0ca3b0 0x561fee0cc7a0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7fdbe00ce090 tx=0x7fdbe00d4e80 comp rx=0 tx=0).stop 2024-04-02T14:44:59.248 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.249+0000 7fdbf9c00640 1 --2- 172.21.15.90:0/293184469 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fdbdc0a3f70 0x7fdbdc0a6430 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fdbec009fd0 tx=0x7fdbec009500 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:44:59.248 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.249+0000 7fdbfe9d4740 1 -- 172.21.15.90:0/683039665 shutdown_connections 2024-04-02T14:44:59.248 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.249+0000 7fdbfe9d4740 1 --2- 172.21.15.90:0/683039665 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561fee0ca3b0 0x561fee0cc7a0 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:59.248 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.249+0000 7fdbfe9d4740 1 --2- 172.21.15.90:0/683039665 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561fee0c17d0 0x561fee0c9e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:59.248 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.249+0000 7fdbfe9d4740 1 --2- 172.21.15.90:0/683039665 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561fee0c0eb0 0x561fee0c1290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:59.248 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.249+0000 7fdbfe9d4740 1 -- 172.21.15.90:0/683039665 >> 172.21.15.90:0/683039665 conn(0x561fee0bb5e0 msgr2=0x561fee0bb9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:44:59.248 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.249+0000 7fdbfe9d4740 1 -- 172.21.15.90:0/683039665 shutdown_connections 2024-04-02T14:44:59.248 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.249+0000 7fdbfe9d4740 1 -- 172.21.15.90:0/683039665 wait complete. 2024-04-02T14:44:59.248 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.249+0000 7fdbfe9d4740 1 -- start start 2024-04-02T14:44:59.249 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.250+0000 7fdbfe9d4740 1 Processor -- start 2024-04-02T14:44:59.249 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.250+0000 7fdbfe9d4740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561fee0c0eb0 0x561fee0c74d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:59.249 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.250+0000 7fdbfe9d4740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561fee0c17d0 0x561fee0c7a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:59.249 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.250+0000 7fdbfe9d4740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561fee0ca3b0 0x561fee0c7f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:59.249 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.250+0000 7fdbfe9d4740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x561fee0cd1a0 con 0x561fee0c17d0 2024-04-02T14:44:59.249 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.250+0000 7fdbfe9d4740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x561fedfe1660 con 0x561fee0c0eb0 2024-04-02T14:44:59.249 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.250+0000 7fdbfe9d4740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x561fee0cd320 con 0x561fee0ca3b0 2024-04-02T14:44:59.249 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.250+0000 7fdbfe9d4740 10 client.0 _ll_register_callbacks cb 0x561fedfe1050 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:44:59.249 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.251+0000 7fdbf9200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561fee0c0eb0 0x561fee0c74d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:59.249 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.251+0000 7fdbf9200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561fee0c0eb0 0x561fee0c74d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:41070/0 (socket says 172.21.15.90:41070) 2024-04-02T14:44:59.249 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.251+0000 7fdbf9200640 1 -- 172.21.15.90:0/2934076745 learned_addr learned my addr 172.21.15.90:0/2934076745 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:44:59.249 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.251+0000 7fdbf9c00640 1 --2- 172.21.15.90:0/2934076745 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561fee0ca3b0 0x561fee0c7f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:59.249 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.251+0000 7fdbf3e00640 1 --2- 172.21.15.90:0/2934076745 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561fee0c17d0 0x561fee0c7a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:59.250 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.252+0000 7fdbf9c00640 1 -- 172.21.15.90:0/2934076745 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561fee0c0eb0 msgr2=0x561fee0c74d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:44:59.250 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.252+0000 7fdbf9c00640 1 --2- 172.21.15.90:0/2934076745 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561fee0c0eb0 0x561fee0c74d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:59.250 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.252+0000 7fdbf9c00640 1 -- 172.21.15.90:0/2934076745 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561fee0c17d0 msgr2=0x561fee0c7a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:44:59.250 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.252+0000 7fdbf9c00640 1 --2- 172.21.15.90:0/2934076745 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561fee0c17d0 0x561fee0c7a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:59.250 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.252+0000 7fdbf9c00640 1 -- 172.21.15.90:0/2934076745 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561fedff0e80 con 0x561fee0ca3b0 2024-04-02T14:44:59.250 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.252+0000 7fdbf3e00640 1 --2- 172.21.15.90:0/2934076745 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561fee0c17d0 0x561fee0c7a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:44:59.251 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.252+0000 7fdbf9c00640 1 --2- 172.21.15.90:0/2934076745 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561fee0ca3b0 0x561fee0c7f50 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7fdbec003f20 tx=0x7fdbec004ab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d3524b04b230660d server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:44:59.251 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.253+0000 7fdbdaa00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:44:59.251 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.253+0000 7fdbdaa00640 1 -- 172.21.15.90:0/2934076745 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbec00fe60 con 0x561fee0ca3b0 2024-04-02T14:44:59.251 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.253+0000 7fdbdaa00640 1 -- 172.21.15.90:0/2934076745 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8550+0+0 (secure 0 0 0) 0x7fdbec0047d0 con 0x561fee0ca3b0 2024-04-02T14:44:59.252 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.253+0000 7fdbfe9d4740 1 -- 172.21.15.90:0/2934076745 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdbe009d070 con 0x561fee0ca3b0 2024-04-02T14:44:59.252 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.253+0000 7fdbdaa00640 1 -- 172.21.15.90:0/2934076745 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fdbec012020 con 0x561fee0ca3b0 2024-04-02T14:44:59.252 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.254+0000 7fdbfe9d4740 10 client.28471 fetch_fsmap learned FSMap version 27 2024-04-02T14:44:59.252 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.254+0000 7fdbfe9d4740 1 -- 172.21.15.90:0/2934076745 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x561fedd3a210 con 0x561fee0ca3b0 2024-04-02T14:44:59.252 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.254+0000 7fdbdaa00640 1 -- 172.21.15.90:0/2934076745 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdbe009d070 con 0x561fee0ca3b0 2024-04-02T14:44:59.252 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.254+0000 7fdbdaa00640 1 -- 172.21.15.90:0/2934076745 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fdbec027070 con 0x561fee0ca3b0 2024-04-02T14:44:59.252 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.254+0000 7fdbfe9d4740 10 client.28471 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:44:59.252 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.254+0000 7fdbfe9d4740 10 client.28471 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-02T14:44:59.252 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.254+0000 7fdbfe9d4740 20 client.28471 populate_metadata read hostname 'smithi090' 2024-04-02T14:44:59.252 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.254+0000 7fdbfe9d4740 1 --2- 172.21.15.90:0/2934076745 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x561fee0da830 0x561fee0dcc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:59.252 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.255+0000 7fdbfe9d4740 4 client.28471 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:44:59.252 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.255+0000 7fdbfe9d4740 1 -- 172.21.15.90:0/2934076745 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561fedfd5cd0 con 0x561fee0da830 2024-04-02T14:44:59.252 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.255+0000 7fdbf9200640 1 --2- 172.21.15.90:0/2934076745 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x561fee0da830 0x561fee0dcc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:59.253 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.257+0000 7fdbf9200640 1 --2- 172.21.15.90:0/2934076745 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x561fee0da830 0x561fee0dcc20 crc :-1 s=READY pgs=270 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1317cd06b2cbec9d server_cookie=5504fd2420fbf636 in_seq=0 out_seq=0 2024-04-02T14:44:59.253 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.257+0000 7fdbdaa00640 10 client.28471 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:44:59.253 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.258+0000 7fdbdaa00640 1 -- 172.21.15.90:0/2934076745 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x561fedfd5cd0 con 0x561fee0da830 2024-04-02T14:44:59.254 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.258+0000 7fdbdaa00640 10 client.28471 handle_command_reply: tid=0 2024-04-02T14:44:59.254 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.258+0000 7fdbfe9d4740 1 client.28471 shutdown 2024-04-02T14:44:59.254 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.259+0000 7fdbfe9d4740 1 -- 172.21.15.90:0/2934076745 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561fee0ca3b0 msgr2=0x561fee0c7f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:59.254 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.259+0000 7fdbfe9d4740 1 --2- 172.21.15.90:0/2934076745 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561fee0ca3b0 0x561fee0c7f50 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7fdbec003f20 tx=0x7fdbec004ab0 comp rx=0 tx=0).stop 2024-04-02T14:44:59.255 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.259+0000 7fdbfe9d4740 1 -- 172.21.15.90:0/2934076745 shutdown_connections 2024-04-02T14:44:59.255 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.259+0000 7fdbfe9d4740 1 --2- 172.21.15.90:0/2934076745 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x561fee0da830 0x561fee0dcc20 crc :-1 s=READY pgs=270 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:59.255 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.259+0000 7fdbfe9d4740 1 --2- 172.21.15.90:0/2934076745 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561fee0ca3b0 0x561fee0c7f50 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:59.255 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.259+0000 7fdbfe9d4740 1 --2- 172.21.15.90:0/2934076745 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561fee0c17d0 0x561fee0c7a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:59.255 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.259+0000 7fdbfe9d4740 1 --2- 172.21.15.90:0/2934076745 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561fee0c0eb0 0x561fee0c74d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:59.255 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.259+0000 7fdbfe9d4740 1 -- 172.21.15.90:0/2934076745 >> 172.21.15.90:0/2934076745 conn(0x561fee0bb5e0 msgr2=0x561fee0bd330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:44:59.255 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.259+0000 7fdbdaa00640 0 client.28471 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:44:59.255 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.259+0000 7fdbfe9d4740 1 -- 172.21.15.90:0/2934076745 shutdown_connections 2024-04-02T14:44:59.255 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.259+0000 7fdbfe9d4740 1 -- 172.21.15.90:0/2934076745 wait complete. 2024-04-02T14:44:59.256 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.259+0000 7fdbfe9d4740 20 client.28471 trim_cache size 0 max 16384 2024-04-02T14:44:59.289 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.293+0000 7fdbfe9d4740 1 Processor -- start 2024-04-02T14:44:59.289 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.293+0000 7fdbfe9d4740 1 -- start start 2024-04-02T14:44:59.290 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.293+0000 7fdbfe9d4740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561fee0c0eb0 0x561fee0d6060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:59.290 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.293+0000 7fdbfe9d4740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561fee0c17d0 0x561fee0c7d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:59.290 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.294+0000 7fdbfe9d4740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561fee0ca3b0 0x561fee0e2510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:59.290 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.294+0000 7fdbfe9d4740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fdbf41bcd90 con 0x561fee0c0eb0 2024-04-02T14:44:59.290 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.294+0000 7fdbfe9d4740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fdbf41be8d0 con 0x561fee0c17d0 2024-04-02T14:44:59.290 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.294+0000 7fdbfe9d4740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fdbf41bd460 con 0x561fee0ca3b0 2024-04-02T14:44:59.291 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.294+0000 7fdbf9c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561fee0ca3b0 0x561fee0e2510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:59.291 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.294+0000 7fdbf9c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561fee0ca3b0 0x561fee0e2510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:60640/0 (socket says 172.21.15.90:60640) 2024-04-02T14:44:59.291 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.294+0000 7fdbf9c00640 1 -- 172.21.15.90:0/2541365405 learned_addr learned my addr 172.21.15.90:0/2541365405 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:44:59.291 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.295+0000 7fdbf9200640 1 --2- 172.21.15.90:0/2541365405 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561fee0c0eb0 0x561fee0d6060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:59.291 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.295+0000 7fdbf3e00640 1 --2- 172.21.15.90:0/2541365405 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561fee0c17d0 0x561fee0c7d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:59.293 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.295+0000 7fdbf9c00640 1 -- 172.21.15.90:0/2541365405 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561fee0c17d0 msgr2=0x561fee0c7d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:44:59.293 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.295+0000 7fdbf9c00640 1 --2- 172.21.15.90:0/2541365405 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561fee0c17d0 0x561fee0c7d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:59.294 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.295+0000 7fdbf9c00640 1 -- 172.21.15.90:0/2541365405 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561fee0c0eb0 msgr2=0x561fee0d6060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:44:59.294 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.295+0000 7fdbf9c00640 1 --2- 172.21.15.90:0/2541365405 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561fee0c0eb0 0x561fee0d6060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:59.294 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.295+0000 7fdbf9c00640 1 -- 172.21.15.90:0/2541365405 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561fede06620 con 0x561fee0ca3b0 2024-04-02T14:44:59.294 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.296+0000 7fdbf9c00640 1 --2- 172.21.15.90:0/2541365405 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561fee0ca3b0 0x561fee0e2510 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7fdbec004410 tx=0x7fdbec005bd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f03eb50a88c87d1a server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:44:59.294 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.298+0000 7fdbe5600640 1 -- 172.21.15.90:0/2541365405 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbec037070 con 0x561fee0ca3b0 2024-04-02T14:44:59.294 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.298+0000 7fdbe5600640 1 -- 172.21.15.90:0/2541365405 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdbec0121c0 con 0x561fee0ca3b0 2024-04-02T14:44:59.294 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.298+0000 7fdbe5600640 1 -- 172.21.15.90:0/2541365405 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbec012020 con 0x561fee0ca3b0 2024-04-02T14:44:59.294 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.299+0000 7fdbfe9d4740 1 -- 172.21.15.90:0/2541365405 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561fee0ca3b0 msgr2=0x561fee0e2510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:59.294 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.299+0000 7fdbfe9d4740 1 --2- 172.21.15.90:0/2541365405 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561fee0ca3b0 0x561fee0e2510 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7fdbec004410 tx=0x7fdbec005bd0 comp rx=0 tx=0).stop 2024-04-02T14:44:59.295 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.299+0000 7fdbfe9d4740 1 -- 172.21.15.90:0/2541365405 shutdown_connections 2024-04-02T14:44:59.295 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.299+0000 7fdbfe9d4740 1 --2- 172.21.15.90:0/2541365405 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561fee0ca3b0 0x561fee0e2510 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:59.295 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.299+0000 7fdbfe9d4740 1 --2- 172.21.15.90:0/2541365405 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561fee0c17d0 0x561fee0c7d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:59.296 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.299+0000 7fdbfe9d4740 1 --2- 172.21.15.90:0/2541365405 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561fee0c0eb0 0x561fee0d6060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:59.296 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.299+0000 7fdbfe9d4740 1 -- 172.21.15.90:0/2541365405 >> 172.21.15.90:0/2541365405 conn(0x561fee0bb5e0 msgr2=0x561fee0b9db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:44:59.296 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.300+0000 7fdbfe9d4740 1 -- 172.21.15.90:0/2541365405 shutdown_connections 2024-04-02T14:44:59.296 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.300+0000 7fdbfe9d4740 1 -- 172.21.15.90:0/2541365405 wait complete. 2024-04-02T14:44:59.296 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.300+0000 7fdbfe9d4740 1 -- start start 2024-04-02T14:44:59.296 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.301+0000 7fdbfe9d4740 1 Processor -- start 2024-04-02T14:44:59.299 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.304+0000 7fdbfe9d4740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561fee0c0eb0 0x561fee0d2840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:59.300 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.304+0000 7fdbfe9d4740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561fee0c17d0 0x561fee0cf540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:59.300 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.304+0000 7fdbfe9d4740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561fee0ca3b0 0x561fee0cfa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:59.303 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.304+0000 7fdbfe9d4740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fdbf41be200 con 0x561fee0c0eb0 2024-04-02T14:44:59.303 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.304+0000 7fdbfe9d4740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fdbf41bdb30 con 0x561fee0c17d0 2024-04-02T14:44:59.303 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.304+0000 7fdbfe9d4740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fdbf41bad50 con 0x561fee0ca3b0 2024-04-02T14:44:59.303 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.304+0000 7fdbfe9d4740 10 client.0 _ll_register_callbacks cb 0x7fdbec003de0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:44:59.303 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.304+0000 7fdbf9c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561fee0ca3b0 0x561fee0cfa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:59.303 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.304+0000 7fdbf9200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561fee0c0eb0 0x561fee0d2840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:59.303 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.304+0000 7fdbf9c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561fee0ca3b0 0x561fee0cfa80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:60656/0 (socket says 172.21.15.90:60656) 2024-04-02T14:44:59.303 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.304+0000 7fdbf9c00640 1 -- 172.21.15.90:0/175796452 learned_addr learned my addr 172.21.15.90:0/175796452 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:44:59.303 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.304+0000 7fdbf3e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561fee0c17d0 0x561fee0cf540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:59.304 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.305+0000 7fdbf9c00640 1 -- 172.21.15.90:0/175796452 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561fee0c17d0 msgr2=0x561fee0cf540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:44:59.304 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.305+0000 7fdbf9c00640 1 --2- 172.21.15.90:0/175796452 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561fee0c17d0 0x561fee0cf540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:59.304 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.305+0000 7fdbf9c00640 1 -- 172.21.15.90:0/175796452 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561fee0c0eb0 msgr2=0x561fee0d2840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:44:59.304 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.305+0000 7fdbf9c00640 1 --2- 172.21.15.90:0/175796452 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561fee0c0eb0 0x561fee0d2840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:59.304 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.305+0000 7fdbf9c00640 1 -- 172.21.15.90:0/175796452 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561fee0d01c0 con 0x561fee0ca3b0 2024-04-02T14:44:59.304 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.307+0000 7fdbf9c00640 1 --2- 172.21.15.90:0/175796452 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561fee0ca3b0 0x561fee0cfa80 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7fdbec00f880 tx=0x7fdbec004440 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ccfd3a72561edf5b server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:44:59.304 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.307+0000 7fdbdaa00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:44:59.305 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.307+0000 7fdbdaa00640 1 -- 172.21.15.90:0/175796452 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbec037040 con 0x561fee0ca3b0 2024-04-02T14:44:59.305 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.308+0000 7fdbdaa00640 1 -- 172.21.15.90:0/175796452 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8550+0+0 (secure 0 0 0) 0x7fdbec0283f0 con 0x561fee0ca3b0 2024-04-02T14:44:59.305 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.308+0000 7fdbfe9d4740 1 -- 172.21.15.90:0/175796452 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdbec0121c0 con 0x561fee0ca3b0 2024-04-02T14:44:59.306 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.309+0000 7fdbdaa00640 1 -- 172.21.15.90:0/175796452 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fdbec048050 con 0x561fee0ca3b0 2024-04-02T14:44:59.306 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.309+0000 7fdbfe9d4740 10 client.28477 fetch_fsmap learned FSMap version 27 2024-04-02T14:44:59.306 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.309+0000 7fdbfe9d4740 1 -- 172.21.15.90:0/175796452 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x561fee141cd0 con 0x561fee0ca3b0 2024-04-02T14:44:59.306 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.309+0000 7fdbdaa00640 1 -- 172.21.15.90:0/175796452 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdbec0121c0 con 0x561fee0ca3b0 2024-04-02T14:44:59.306 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.309+0000 7fdbdaa00640 1 -- 172.21.15.90:0/175796452 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fdbec043030 con 0x561fee0ca3b0 2024-04-02T14:44:59.306 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.309+0000 7fdbfe9d4740 10 client.28477 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:44:59.306 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.309+0000 7fdbfe9d4740 10 client.28477 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-02T14:44:59.307 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.309+0000 7fdbfe9d4740 20 client.28477 populate_metadata read hostname 'smithi090' 2024-04-02T14:44:59.307 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.309+0000 7fdbfe9d4740 1 --2- 172.21.15.90:0/175796452 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x561fee1422d0 0x561fee1446c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:59.307 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.309+0000 7fdbfe9d4740 4 client.28477 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-02T14:44:59.307 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.309+0000 7fdbfe9d4740 1 -- 172.21.15.90:0/175796452 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "flush journal"}) -- 0x561fee0332b0 con 0x561fee1422d0 2024-04-02T14:44:59.307 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.310+0000 7fdbf9200640 1 --2- 172.21.15.90:0/175796452 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x561fee1422d0 0x561fee1446c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:59.308 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.312+0000 7fdbf9200640 1 --2- 172.21.15.90:0/175796452 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x561fee1422d0 0x561fee1446c0 crc :-1 s=READY pgs=271 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=88b15ed646cf439d server_cookie=f710638d4db1fef7 in_seq=0 out_seq=0 2024-04-02T14:44:59.308 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.312+0000 7fdbdaa00640 10 client.28477 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:44:59.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.351+0000 7f1d83000640 1 -- 172.21.15.90:0/3980277690 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1d7c106a50 msgr2=0x7f1d7c068d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:59.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.351+0000 7f1d83000640 1 --2- 172.21.15.90:0/3980277690 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1d7c106a50 0x7f1d7c068d00 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f1d74009ca0 tx=0x7f1d7402c120 comp rx=0 tx=0).stop 2024-04-02T14:44:59.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.352+0000 7f1d83000640 1 -- 172.21.15.90:0/3980277690 shutdown_connections 2024-04-02T14:44:59.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.352+0000 7f1d83000640 1 --2- 172.21.15.90:0/3980277690 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1d7c1045c0 0x7f1d7c104a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:59.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.352+0000 7f1d83000640 1 --2- 172.21.15.90:0/3980277690 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1d7c069240 0x7f1d7c0696a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:59.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.352+0000 7f1d83000640 1 --2- 172.21.15.90:0/3980277690 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1d7c106a50 0x7f1d7c068d00 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:59.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.352+0000 7f1d83000640 1 -- 172.21.15.90:0/3980277690 >> 172.21.15.90:0/3980277690 conn(0x7f1d7c077b00 msgr2=0x7f1d7c077f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:44:59.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.352+0000 7f1d83000640 1 -- 172.21.15.90:0/3980277690 shutdown_connections 2024-04-02T14:44:59.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.352+0000 7f1d83000640 1 -- 172.21.15.90:0/3980277690 wait complete. 2024-04-02T14:44:59.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.353+0000 7f1d83000640 1 Processor -- start 2024-04-02T14:44:59.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.356+0000 7f1d83000640 1 -- start start 2024-04-02T14:44:59.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.357+0000 7f1d83000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1d7c069240 0x7f1d7c1a46e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:59.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.357+0000 7f1d83000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1d7c1045c0 0x7f1d7c1a4c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:59.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.357+0000 7f1d83000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1d7c106a50 0x7f1d7c19e850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:59.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.357+0000 7f1d83000640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f1d7c118550 con 0x7f1d7c1045c0 2024-04-02T14:44:59.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.357+0000 7f1d83000640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f1d7c1183d0 con 0x7f1d7c106a50 2024-04-02T14:44:59.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.357+0000 7f1d83000640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f1d7c1186d0 con 0x7f1d7c069240 2024-04-02T14:44:59.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.357+0000 7f1d7b400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1d7c1045c0 0x7f1d7c1a4c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:59.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.357+0000 7f1d81000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1d7c106a50 0x7f1d7c19e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:59.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.358+0000 7f1d81000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1d7c106a50 0x7f1d7c19e850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:41106/0 (socket says 172.21.15.90:41106) 2024-04-02T14:44:59.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.358+0000 7f1d81000640 1 -- 172.21.15.90:0/190944222 learned_addr learned my addr 172.21.15.90:0/190944222 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:44:59.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.358+0000 7f1d81000640 1 -- 172.21.15.90:0/190944222 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1d7c069240 msgr2=0x7f1d7c1a46e0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-02T14:44:59.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.358+0000 7f1d81000640 1 --2- 172.21.15.90:0/190944222 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1d7c069240 0x7f1d7c1a46e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:59.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.358+0000 7f1d81000640 1 -- 172.21.15.90:0/190944222 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1d7c1045c0 msgr2=0x7f1d7c1a4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:59.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.358+0000 7f1d81000640 1 --2- 172.21.15.90:0/190944222 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1d7c1045c0 0x7f1d7c1a4c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:59.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.358+0000 7f1d81000640 1 -- 172.21.15.90:0/190944222 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d7c19f020 con 0x7f1d7c106a50 2024-04-02T14:44:59.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.360+0000 7f1d81000640 1 --2- 172.21.15.90:0/190944222 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1d7c106a50 0x7f1d7c19e850 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f1d6c00b3f0 tx=0x7f1d6c00b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:44:59.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.360+0000 7f1d78c00640 1 -- 172.21.15.90:0/190944222 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d6c015020 con 0x7f1d7c106a50 2024-04-02T14:44:59.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.360+0000 7f1d78c00640 1 -- 172.21.15.90:0/190944222 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1d6c011d10 con 0x7f1d7c106a50 2024-04-02T14:44:59.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.360+0000 7f1d78c00640 1 -- 172.21.15.90:0/190944222 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d6c00f2e0 con 0x7f1d7c106a50 2024-04-02T14:44:59.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.360+0000 7f1d83000640 1 -- 172.21.15.90:0/190944222 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1d7c19f310 con 0x7f1d7c106a50 2024-04-02T14:44:59.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.361+0000 7f1d83000640 1 -- 172.21.15.90:0/190944222 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1d7c1ab530 con 0x7f1d7c106a50 2024-04-02T14:44:59.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.366+0000 7f1d85dcb740 1 Processor -- start 2024-04-02T14:44:59.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.366+0000 7f1d78c00640 1 -- 172.21.15.90:0/190944222 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f1d6c018020 con 0x7f1d7c106a50 2024-04-02T14:44:59.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.366+0000 7f1d85dcb740 1 -- start start 2024-04-02T14:44:59.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.367+0000 7f1d85dcb740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b7a70df7b0 0x55b7a70dfb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:59.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.367+0000 7f1d85dcb740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b7a70e00d0 0x55b7a70e8770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:59.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.367+0000 7f1d85dcb740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b7a70e8cb0 0x55b7a70eb0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:59.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.367+0000 7f1d85dcb740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55b7a7043770 con 0x55b7a70e8cb0 2024-04-02T14:44:59.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.367+0000 7f1d85dcb740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55b7a6feb700 con 0x55b7a70df7b0 2024-04-02T14:44:59.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.367+0000 7f1d85dcb740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55b7a703f720 con 0x55b7a70e00d0 2024-04-02T14:44:59.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.367+0000 7f1d7be00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b7a70e00d0 0x55b7a70e8770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:59.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.367+0000 7f1d7be00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b7a70e00d0 0x55b7a70e8770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:60692/0 (socket says 172.21.15.90:60692) 2024-04-02T14:44:59.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.367+0000 7f1d7be00640 1 -- 172.21.15.90:0/1131993165 learned_addr learned my addr 172.21.15.90:0/1131993165 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:44:59.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.368+0000 7f1d7be00640 1 -- 172.21.15.90:0/1131993165 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b7a70df7b0 msgr2=0x55b7a70dfb90 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-02T14:44:59.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.368+0000 7f1d7be00640 1 --2- 172.21.15.90:0/1131993165 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b7a70df7b0 0x55b7a70dfb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:59.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.368+0000 7f1d7be00640 1 -- 172.21.15.90:0/1131993165 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b7a70e8cb0 msgr2=0x55b7a70eb0a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-02T14:44:59.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.368+0000 7f1d7b400640 1 --2- 172.21.15.90:0/1131993165 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b7a70e8cb0 0x55b7a70eb0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:59.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.368+0000 7f1d7be00640 1 --2- 172.21.15.90:0/1131993165 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b7a70e8cb0 0x55b7a70eb0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:59.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.369+0000 7f1d7be00640 1 -- 172.21.15.90:0/1131993165 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b7a6e217d0 con 0x55b7a70e00d0 2024-04-02T14:44:59.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.369+0000 7f1d7be00640 1 --2- 172.21.15.90:0/1131993165 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b7a70e00d0 0x55b7a70e8770 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f1d74037040 tx=0x7f1d74009890 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5433b2f74b87bf03 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:44:59.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.369+0000 7f1d71600640 1 -- 172.21.15.90:0/1131993165 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d74039050 con 0x55b7a70e00d0 2024-04-02T14:44:59.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.369+0000 7f1d71600640 1 -- 172.21.15.90:0/1131993165 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1d74045070 con 0x55b7a70e00d0 2024-04-02T14:44:59.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.369+0000 7f1d71600640 1 -- 172.21.15.90:0/1131993165 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d740042a0 con 0x55b7a70e00d0 2024-04-02T14:44:59.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.370+0000 7f1d85dcb740 1 -- 172.21.15.90:0/1131993165 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b7a70e00d0 msgr2=0x55b7a70e8770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:59.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.370+0000 7f1d85dcb740 1 --2- 172.21.15.90:0/1131993165 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b7a70e00d0 0x55b7a70e8770 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f1d74037040 tx=0x7f1d74009890 comp rx=0 tx=0).stop 2024-04-02T14:44:59.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.371+0000 7f1d85dcb740 1 -- 172.21.15.90:0/1131993165 shutdown_connections 2024-04-02T14:44:59.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.371+0000 7f1d85dcb740 1 --2- 172.21.15.90:0/1131993165 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b7a70e8cb0 0x55b7a70eb0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:59.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.371+0000 7f1d85dcb740 1 --2- 172.21.15.90:0/1131993165 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b7a70e00d0 0x55b7a70e8770 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:59.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.371+0000 7f1d85dcb740 1 --2- 172.21.15.90:0/1131993165 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b7a70df7b0 0x55b7a70dfb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:59.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.371+0000 7f1d85dcb740 1 -- 172.21.15.90:0/1131993165 >> 172.21.15.90:0/1131993165 conn(0x55b7a70d9ee0 msgr2=0x55b7a70da2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:44:59.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.372+0000 7f1d78c00640 1 --2- 172.21.15.90:0/190944222 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f1d600a3cb0 0x7f1d600a6170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:59.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.373+0000 7f1d78c00640 1 -- 172.21.15.90:0/190944222 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8550+0+0 (secure 0 0 0) 0x7f1d6c0cff80 con 0x7f1d7c106a50 2024-04-02T14:44:59.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.373+0000 7f1d7be00640 1 --2- 172.21.15.90:0/190944222 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f1d600a3cb0 0x7f1d600a6170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:59.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.372+0000 7f1d85dcb740 1 -- 172.21.15.90:0/1131993165 shutdown_connections 2024-04-02T14:44:59.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.373+0000 7f1d85dcb740 1 -- 172.21.15.90:0/1131993165 wait complete. 2024-04-02T14:44:59.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.374+0000 7f1d7be00640 1 --2- 172.21.15.90:0/190944222 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f1d600a3cb0 0x7f1d600a6170 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f1d740325a0 tx=0x7f1d74000f40 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:44:59.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.374+0000 7f1d85dcb740 1 -- start start 2024-04-02T14:44:59.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.374+0000 7f1d85dcb740 1 Processor -- start 2024-04-02T14:44:59.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.375+0000 7f1d85dcb740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b7a70df7b0 0x55b7a70e5d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:59.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.375+0000 7f1d85dcb740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b7a70e00d0 0x55b7a70e62a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:59.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.375+0000 7f1d85dcb740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b7a70e8cb0 0x55b7a70e67e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:59.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.375+0000 7f1d85dcb740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55b7a70eba40 con 0x55b7a70df7b0 2024-04-02T14:44:59.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.375+0000 7f1d85dcb740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55b7a6ffeac0 con 0x55b7a70e00d0 2024-04-02T14:44:59.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.376+0000 7f1d7be00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b7a70df7b0 0x55b7a70e5d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:59.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.376+0000 7f1d7be00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b7a70df7b0 0x55b7a70e5d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:39144/0 (socket says 172.21.15.90:39144) 2024-04-02T14:44:59.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.376+0000 7f1d7be00640 1 -- 172.21.15.90:0/2876621396 learned_addr learned my addr 172.21.15.90:0/2876621396 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:44:59.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.376+0000 7f1d81000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b7a70e8cb0 0x55b7a70e67e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:59.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.376+0000 7f1d7b400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b7a70e00d0 0x55b7a70e62a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:59.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.378+0000 7f1d85dcb740 1 -- 172.21.15.90:0/2876621396 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55b7a70ebbc0 con 0x55b7a70e8cb0 2024-04-02T14:44:59.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.379+0000 7f1d7b400640 1 -- 172.21.15.90:0/2876621396 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b7a70e8cb0 msgr2=0x55b7a70e67e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:44:59.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.379+0000 7f1d7b400640 1 --2- 172.21.15.90:0/2876621396 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b7a70e8cb0 0x55b7a70e67e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:59.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.379+0000 7f1d7b400640 1 -- 172.21.15.90:0/2876621396 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b7a70df7b0 msgr2=0x55b7a70e5d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:44:59.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.379+0000 7f1d7b400640 1 --2- 172.21.15.90:0/2876621396 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b7a70df7b0 0x55b7a70e5d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:59.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.379+0000 7f1d7b400640 1 -- 172.21.15.90:0/2876621396 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b7a6d573f0 con 0x55b7a70e00d0 2024-04-02T14:44:59.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.379+0000 7f1d85dcb740 10 client.0 _ll_register_callbacks cb 0x55b7a7001180 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:44:59.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.380+0000 7f1d7b400640 1 --2- 172.21.15.90:0/2876621396 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b7a70e00d0 0x55b7a70e62a0 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f1d7c0782c0 tx=0x7f1d6800e540 comp rx=0 tx=0).ready entity=mon.2 client_cookie=45b8c1093230b5b7 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:44:59.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.380+0000 7f1d66000640 10 client.28652 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:44:59.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.381+0000 7f1d85dcb740 1 -- 172.21.15.90:0/2876621396 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1d74045070 con 0x55b7a70e00d0 2024-04-02T14:44:59.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.381+0000 7f1d66000640 1 -- 172.21.15.90:0/2876621396 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d68017020 con 0x55b7a70e00d0 2024-04-02T14:44:59.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.381+0000 7f1d66000640 1 -- 172.21.15.90:0/2876621396 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8550+0+0 (secure 0 0 0) 0x7f1d68013410 con 0x55b7a70e00d0 2024-04-02T14:44:59.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.381+0000 7f1d66000640 1 -- 172.21.15.90:0/2876621396 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f1d68022050 con 0x55b7a70e00d0 2024-04-02T14:44:59.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.382+0000 7f1d85dcb740 10 client.28652 fetch_fsmap learned FSMap version 27 2024-04-02T14:44:59.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.382+0000 7f1d85dcb740 1 -- 172.21.15.90:0/2876621396 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({fsmap=27}) -- 0x55b7a6d170f0 con 0x55b7a70e00d0 2024-04-02T14:44:59.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.382+0000 7f1d66000640 1 -- 172.21.15.90:0/2876621396 <== mon.2 v2:172.21.15.92:3300/0 4 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f1d68012050 con 0x55b7a70e00d0 2024-04-02T14:44:59.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.383+0000 7f1d85dcb740 10 client.28652 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:44:59.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.383+0000 7f1d85dcb740 10 client.28652 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:44:59.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.383+0000 7f1d85dcb740 20 client.28652 populate_metadata read hostname 'smithi090' 2024-04-02T14:44:59.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.383+0000 7f1d85dcb740 1 --2- 172.21.15.90:0/2876621396 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55b7a7109bb0 0x55b7a710bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:59.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.383+0000 7f1d85dcb740 4 client.28652 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:44:59.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.383+0000 7f1d85dcb740 1 -- 172.21.15.90:0/2876621396 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b7a6fe4630 con 0x55b7a7109bb0 2024-04-02T14:44:59.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.386+0000 7f1d7be00640 1 --2- 172.21.15.90:0/2876621396 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55b7a7109bb0 0x55b7a710bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:59.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.388+0000 7f1d7be00640 1 --2- 172.21.15.90:0/2876621396 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55b7a7109bb0 0x55b7a710bfa0 crc :-1 s=READY pgs=272 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=39eaaa9d5324a8bc server_cookie=d1d38753e567fec in_seq=0 out_seq=0 2024-04-02T14:44:59.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.389+0000 7f1d66000640 10 client.28652 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:44:59.393 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.397+0000 7fdbdaa00640 1 -- 172.21.15.90:0/175796452 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x561fee0332b0 con 0x561fee1422d0 2024-04-02T14:44:59.393 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.397+0000 7fdbdaa00640 10 client.28477 handle_command_reply: tid=0 2024-04-02T14:44:59.393 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.397+0000 7fdbfe9d4740 1 client.28477 shutdown 2024-04-02T14:44:59.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.397+0000 7f1d66000640 1 -- 172.21.15.90:0/2876621396 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b7a6fe4630 con 0x55b7a7109bb0 2024-04-02T14:44:59.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.397+0000 7f1d66000640 10 client.28652 handle_command_reply: tid=0 2024-04-02T14:44:59.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.398+0000 7f1d85dcb740 1 client.28652 shutdown 2024-04-02T14:44:59.394 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.398+0000 7fdbfe9d4740 1 -- 172.21.15.90:0/175796452 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561fee0ca3b0 msgr2=0x561fee0cfa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:59.394 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.398+0000 7fdbfe9d4740 1 --2- 172.21.15.90:0/175796452 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561fee0ca3b0 0x561fee0cfa80 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7fdbec00f880 tx=0x7fdbec004440 comp rx=0 tx=0).stop 2024-04-02T14:44:59.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.399+0000 7f1d85dcb740 1 -- 172.21.15.90:0/2876621396 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b7a70e00d0 msgr2=0x55b7a70e62a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:59.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.399+0000 7f1d85dcb740 1 --2- 172.21.15.90:0/2876621396 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b7a70e00d0 0x55b7a70e62a0 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f1d7c0782c0 tx=0x7f1d6800e540 comp rx=0 tx=0).stop 2024-04-02T14:44:59.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.399+0000 7f1d85dcb740 1 -- 172.21.15.90:0/2876621396 shutdown_connections 2024-04-02T14:44:59.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.399+0000 7f1d85dcb740 1 --2- 172.21.15.90:0/2876621396 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55b7a7109bb0 0x55b7a710bfa0 crc :-1 s=READY pgs=272 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:59.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.399+0000 7f1d85dcb740 1 --2- 172.21.15.90:0/2876621396 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b7a70e8cb0 0x55b7a70e67e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:59.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.399+0000 7f1d85dcb740 1 --2- 172.21.15.90:0/2876621396 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b7a70e00d0 0x55b7a70e62a0 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:59.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.399+0000 7f1d85dcb740 1 --2- 172.21.15.90:0/2876621396 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b7a70df7b0 0x55b7a70e5d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:59.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.399+0000 7f1d85dcb740 1 -- 172.21.15.90:0/2876621396 >> 172.21.15.90:0/2876621396 conn(0x55b7a70d9ee0 msgr2=0x55b7a70dbc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:44:59.397 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.399+0000 7fdbfe9d4740 1 -- 172.21.15.90:0/175796452 shutdown_connections 2024-04-02T14:44:59.397 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.399+0000 7fdbfe9d4740 1 --2- 172.21.15.90:0/175796452 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x561fee1422d0 0x561fee1446c0 crc :-1 s=READY pgs=271 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:59.397 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.399+0000 7fdbfe9d4740 1 --2- 172.21.15.90:0/175796452 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561fee0ca3b0 0x561fee0cfa80 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:59.397 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.399+0000 7fdbfe9d4740 1 --2- 172.21.15.90:0/175796452 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561fee0c17d0 0x561fee0cf540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:59.397 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.399+0000 7fdbfe9d4740 1 --2- 172.21.15.90:0/175796452 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561fee0c0eb0 0x561fee0d2840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:59.397 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.399+0000 7fdbfe9d4740 1 -- 172.21.15.90:0/175796452 >> 172.21.15.90:0/175796452 conn(0x561fee0bb5e0 msgr2=0x561fee0bca10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:44:59.397 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.399+0000 7fdbdaa00640 0 client.28477 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:44:59.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.400+0000 7f1d66000640 0 client.28652 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:44:59.398 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.400+0000 7fdbfe9d4740 1 -- 172.21.15.90:0/175796452 shutdown_connections 2024-04-02T14:44:59.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.400+0000 7f1d85dcb740 1 -- 172.21.15.90:0/2876621396 shutdown_connections 2024-04-02T14:44:59.399 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.400+0000 7fdbfe9d4740 1 -- 172.21.15.90:0/175796452 wait complete. 2024-04-02T14:44:59.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.400+0000 7f1d85dcb740 1 -- 172.21.15.90:0/2876621396 wait complete. 2024-04-02T14:44:59.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.401+0000 7f1d85dcb740 20 client.28652 trim_cache size 0 max 16384 2024-04-02T14:44:59.400 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.401+0000 7fdbfe9d4740 20 client.28477 trim_cache size 0 max 16384 2024-04-02T14:44:59.400 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:{ 2024-04-02T14:44:59.400 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "message": "", 2024-04-02T14:44:59.400 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "return_code": 0 2024-04-02T14:44:59.401 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:} 2024-04-02T14:44:59.401 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.402+0000 7fdbfbc00640 1 -- 172.21.15.90:0/293184469 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fdbdc0a3f70 msgr2=0x7fdbdc0a6430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:59.401 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.402+0000 7fdbfbc00640 1 --2- 172.21.15.90:0/293184469 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fdbdc0a3f70 0x7fdbdc0a6430 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fdbec009fd0 tx=0x7fdbec009500 comp rx=0 tx=0).stop 2024-04-02T14:44:59.401 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.403+0000 7fdbfbc00640 1 -- 172.21.15.90:0/293184469 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdbf4110c20 msgr2=0x7fdbf4073580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:59.401 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.403+0000 7fdbfbc00640 1 --2- 172.21.15.90:0/293184469 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdbf4110c20 0x7fdbf4073580 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7fdbe000e9a0 tx=0x7fdbe000ee60 comp rx=0 tx=0).stop 2024-04-02T14:44:59.402 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.403+0000 7fdbfbc00640 1 -- 172.21.15.90:0/293184469 shutdown_connections 2024-04-02T14:44:59.402 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.403+0000 7fdbfbc00640 1 --2- 172.21.15.90:0/293184469 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fdbdc0a3f70 0x7fdbdc0a6430 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:59.402 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.403+0000 7fdbfbc00640 1 --2- 172.21.15.90:0/293184469 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdbf410e830 0x7fdbf410ec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:59.402 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.403+0000 7fdbfbc00640 1 --2- 172.21.15.90:0/293184469 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdbf4110c20 0x7fdbf4073580 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:59.402 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.403+0000 7fdbfbc00640 1 --2- 172.21.15.90:0/293184469 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdbf4109010 0x7fdbf41106e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:59.402 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.403+0000 7fdbfbc00640 1 -- 172.21.15.90:0/293184469 >> 172.21.15.90:0/293184469 conn(0x7fdbf40ff7c0 msgr2=0x7fdbf4101b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:44:59.402 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.404+0000 7fdbfbc00640 1 -- 172.21.15.90:0/293184469 shutdown_connections 2024-04-02T14:44:59.402 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:44:59.404+0000 7fdbfbc00640 1 -- 172.21.15.90:0/293184469 wait complete. 2024-04-02T14:44:59.429 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.434+0000 7f1d85dcb740 1 Processor -- start 2024-04-02T14:44:59.429 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.434+0000 7f1d85dcb740 1 -- start start 2024-04-02T14:44:59.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.434+0000 7f1d85dcb740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b7a70d9ee0 0x55b7a70e8490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:59.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.434+0000 7f1d85dcb740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b7a70df7b0 0x55b7a70e4ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:59.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.434+0000 7f1d85dcb740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b7a70e00d0 0x55b7a70f4930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:59.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.434+0000 7f1d85dcb740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f1d7c1ab7f0 con 0x55b7a70df7b0 2024-04-02T14:44:59.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.434+0000 7f1d85dcb740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f1d7c1ad290 con 0x55b7a70e00d0 2024-04-02T14:44:59.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.434+0000 7f1d85dcb740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f1d7c1abe20 con 0x55b7a70d9ee0 2024-04-02T14:44:59.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.435+0000 7f1d81000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b7a70e00d0 0x55b7a70f4930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:59.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.435+0000 7f1d7be00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b7a70d9ee0 0x55b7a70e8490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:59.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.435+0000 7f1d81000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b7a70e00d0 0x55b7a70f4930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:41130/0 (socket says 172.21.15.90:41130) 2024-04-02T14:44:59.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.435+0000 7f1d7be00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b7a70d9ee0 0x55b7a70e8490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:60700/0 (socket says 172.21.15.90:60700) 2024-04-02T14:44:59.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.435+0000 7f1d81000640 1 -- 172.21.15.90:0/3066747252 learned_addr learned my addr 172.21.15.90:0/3066747252 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:44:59.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.435+0000 7f1d7b400640 1 --2- 172.21.15.90:0/3066747252 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b7a70df7b0 0x55b7a70e4ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:59.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.435+0000 7f1d7be00640 1 -- 172.21.15.90:0/3066747252 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b7a70e00d0 msgr2=0x55b7a70f4930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:44:59.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.435+0000 7f1d7be00640 1 --2- 172.21.15.90:0/3066747252 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b7a70e00d0 0x55b7a70f4930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:59.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.435+0000 7f1d7be00640 1 -- 172.21.15.90:0/3066747252 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b7a70df7b0 msgr2=0x55b7a70e4ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:44:59.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.435+0000 7f1d7be00640 1 --2- 172.21.15.90:0/3066747252 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b7a70df7b0 0x55b7a70e4ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:59.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.435+0000 7f1d7be00640 1 -- 172.21.15.90:0/3066747252 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b7a70e0b40 con 0x55b7a70d9ee0 2024-04-02T14:44:59.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.435+0000 7f1d7be00640 1 --2- 172.21.15.90:0/3066747252 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b7a70d9ee0 0x55b7a70e8490 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x55b7a710c770 tx=0x7f1d74003d00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3705b42352d4f880 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:44:59.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.435+0000 7f1d70c00640 1 -- 172.21.15.90:0/3066747252 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d7405cdc0 con 0x55b7a70d9ee0 2024-04-02T14:44:59.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.435+0000 7f1d70c00640 1 -- 172.21.15.90:0/3066747252 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1d7403a040 con 0x55b7a70d9ee0 2024-04-02T14:44:59.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.436+0000 7f1d70c00640 1 -- 172.21.15.90:0/3066747252 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d740346a0 con 0x55b7a70d9ee0 2024-04-02T14:44:59.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.436+0000 7f1d85dcb740 1 -- 172.21.15.90:0/3066747252 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b7a70d9ee0 msgr2=0x55b7a70e8490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:59.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.436+0000 7f1d85dcb740 1 --2- 172.21.15.90:0/3066747252 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b7a70d9ee0 0x55b7a70e8490 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x55b7a710c770 tx=0x7f1d74003d00 comp rx=0 tx=0).stop 2024-04-02T14:44:59.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.436+0000 7f1d85dcb740 1 -- 172.21.15.90:0/3066747252 shutdown_connections 2024-04-02T14:44:59.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.436+0000 7f1d85dcb740 1 --2- 172.21.15.90:0/3066747252 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b7a70e00d0 0x55b7a70f4930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:59.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.436+0000 7f1d85dcb740 1 --2- 172.21.15.90:0/3066747252 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b7a70df7b0 0x55b7a70e4ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:59.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.436+0000 7f1d85dcb740 1 --2- 172.21.15.90:0/3066747252 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b7a70d9ee0 0x55b7a70e8490 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:59.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.436+0000 7f1d85dcb740 1 -- 172.21.15.90:0/3066747252 >> 172.21.15.90:0/3066747252 conn(0x55b7a70e1730 msgr2=0x55b7a70d8f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:44:59.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.436+0000 7f1d85dcb740 1 -- 172.21.15.90:0/3066747252 shutdown_connections 2024-04-02T14:44:59.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.437+0000 7f1d85dcb740 1 -- 172.21.15.90:0/3066747252 wait complete. 2024-04-02T14:44:59.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.437+0000 7f1d85dcb740 1 -- start start 2024-04-02T14:44:59.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.437+0000 7f1d85dcb740 1 Processor -- start 2024-04-02T14:44:59.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.438+0000 7f1d85dcb740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b7a70d9ee0 0x55b7a70ecbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:59.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.438+0000 7f1d85dcb740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b7a70df7b0 0x55b7a70ed130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:59.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.438+0000 7f1d85dcb740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b7a70e00d0 0x55b7a70ed6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:59.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.438+0000 7f1d85dcb740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f1d7c1acbc0 con 0x55b7a70d9ee0 2024-04-02T14:44:59.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.438+0000 7f1d85dcb740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f1d7c1ac4f0 con 0x55b7a70e00d0 2024-04-02T14:44:59.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.438+0000 7f1d85dcb740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f1d7c101000 con 0x55b7a70df7b0 2024-04-02T14:44:59.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.438+0000 7f1d85dcb740 10 client.0 _ll_register_callbacks cb 0x7f1d68006f30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:44:59.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.438+0000 7f1d81000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b7a70e00d0 0x55b7a70ed6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:59.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.438+0000 7f1d81000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b7a70e00d0 0x55b7a70ed6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:41146/0 (socket says 172.21.15.90:41146) 2024-04-02T14:44:59.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.438+0000 7f1d81000640 1 -- 172.21.15.90:0/2895898039 learned_addr learned my addr 172.21.15.90:0/2895898039 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:44:59.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.438+0000 7f1d7be00640 1 --2- 172.21.15.90:0/2895898039 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b7a70d9ee0 0x55b7a70ecbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:59.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.438+0000 7f1d7b400640 1 --2- 172.21.15.90:0/2895898039 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b7a70df7b0 0x55b7a70ed130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:59.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.439+0000 7f1d7b400640 1 -- 172.21.15.90:0/2895898039 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b7a70e00d0 msgr2=0x55b7a70ed6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:44:59.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.439+0000 7f1d7b400640 1 --2- 172.21.15.90:0/2895898039 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b7a70e00d0 0x55b7a70ed6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:59.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.439+0000 7f1d7b400640 1 -- 172.21.15.90:0/2895898039 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b7a70d9ee0 msgr2=0x55b7a70ecbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:44:59.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.439+0000 7f1d7b400640 1 --2- 172.21.15.90:0/2895898039 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b7a70d9ee0 0x55b7a70ecbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:59.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.439+0000 7f1d7b400640 1 -- 172.21.15.90:0/2895898039 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b7a716f330 con 0x55b7a70df7b0 2024-04-02T14:44:59.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.440+0000 7f1d7be00640 1 --2- 172.21.15.90:0/2895898039 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b7a70d9ee0 0x55b7a70ecbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:44:59.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.440+0000 7f1d81000640 1 --2- 172.21.15.90:0/2895898039 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b7a70e00d0 0x55b7a70ed6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:44:59.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.440+0000 7f1d7b400640 1 --2- 172.21.15.90:0/2895898039 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b7a70df7b0 0x55b7a70ed130 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f1d68000c00 tx=0x7f1d68011000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a5002f48ca427ae5 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:44:59.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.440+0000 7f1d66000640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:44:59.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.441+0000 7f1d66000640 1 -- 172.21.15.90:0/2895898039 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d6802d020 con 0x55b7a70df7b0 2024-04-02T14:44:59.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.442+0000 7f1d66000640 1 -- 172.21.15.90:0/2895898039 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8550+0+0 (secure 0 0 0) 0x7f1d680139c0 con 0x55b7a70df7b0 2024-04-02T14:44:59.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.442+0000 7f1d85dcb740 1 -- 172.21.15.90:0/2895898039 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1d7403a040 con 0x55b7a70df7b0 2024-04-02T14:44:59.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.442+0000 7f1d66000640 1 -- 172.21.15.90:0/2895898039 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f1d68017040 con 0x55b7a70df7b0 2024-04-02T14:44:59.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.442+0000 7f1d85dcb740 10 client.28492 fetch_fsmap learned FSMap version 27 2024-04-02T14:44:59.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.442+0000 7f1d85dcb740 1 -- 172.21.15.90:0/2895898039 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55b7a7170640 con 0x55b7a70df7b0 2024-04-02T14:44:59.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.443+0000 7f1d66000640 1 -- 172.21.15.90:0/2895898039 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1d7403a040 con 0x55b7a70df7b0 2024-04-02T14:44:59.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.443+0000 7f1d66000640 1 -- 172.21.15.90:0/2895898039 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f1d68012040 con 0x55b7a70df7b0 2024-04-02T14:44:59.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.443+0000 7f1d85dcb740 10 client.28492 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:44:59.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.443+0000 7f1d85dcb740 10 client.28492 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:44:59.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.443+0000 7f1d85dcb740 20 client.28492 populate_metadata read hostname 'smithi090' 2024-04-02T14:44:59.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.443+0000 7f1d85dcb740 1 --2- 172.21.15.90:0/2895898039 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55b7a70e8cb0 0x55b7a7172d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:44:59.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.443+0000 7f1d85dcb740 4 client.28492 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-02T14:44:59.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.443+0000 7f1d85dcb740 1 -- 172.21.15.90:0/2895898039 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55b7a7050cb0 con 0x55b7a70e8cb0 2024-04-02T14:44:59.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.443+0000 7f1d7be00640 1 --2- 172.21.15.90:0/2895898039 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55b7a70e8cb0 0x55b7a7172d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:44:59.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.444+0000 7f1d7be00640 1 --2- 172.21.15.90:0/2895898039 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55b7a70e8cb0 0x55b7a7172d70 crc :-1 s=READY pgs=273 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=75d78f995ecccffe server_cookie=b88efd5484fa426d in_seq=0 out_seq=0 2024-04-02T14:44:59.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.444+0000 7f1d66000640 10 client.28492 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:44:59.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.444+0000 7f1d66000640 1 -- 172.21.15.90:0/2895898039 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55b7a7050cb0 con 0x55b7a70e8cb0 2024-04-02T14:44:59.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.444+0000 7f1d66000640 10 client.28492 handle_command_reply: tid=0 2024-04-02T14:44:59.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.444+0000 7f1d85dcb740 1 client.28492 shutdown 2024-04-02T14:44:59.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.445+0000 7f1d85dcb740 1 -- 172.21.15.90:0/2895898039 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b7a70df7b0 msgr2=0x55b7a70ed130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:59.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.445+0000 7f1d85dcb740 1 --2- 172.21.15.90:0/2895898039 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b7a70df7b0 0x55b7a70ed130 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f1d68000c00 tx=0x7f1d68011000 comp rx=0 tx=0).stop 2024-04-02T14:44:59.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.445+0000 7f1d85dcb740 1 -- 172.21.15.90:0/2895898039 shutdown_connections 2024-04-02T14:44:59.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.445+0000 7f1d85dcb740 1 --2- 172.21.15.90:0/2895898039 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55b7a70e8cb0 0x55b7a7172d70 crc :-1 s=READY pgs=273 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:59.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.445+0000 7f1d85dcb740 1 --2- 172.21.15.90:0/2895898039 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b7a70e00d0 0x55b7a70ed6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:59.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.445+0000 7f1d85dcb740 1 --2- 172.21.15.90:0/2895898039 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b7a70df7b0 0x55b7a70ed130 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:59.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.445+0000 7f1d85dcb740 1 --2- 172.21.15.90:0/2895898039 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b7a70d9ee0 0x55b7a70ecbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:59.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.445+0000 7f1d85dcb740 1 -- 172.21.15.90:0/2895898039 >> 172.21.15.90:0/2895898039 conn(0x55b7a70e1730 msgr2=0x55b7a70e5b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:44:59.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.445+0000 7f1d66000640 0 client.28492 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:44:59.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.445+0000 7f1d85dcb740 1 -- 172.21.15.90:0/2895898039 shutdown_connections 2024-04-02T14:44:59.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.445+0000 7f1d85dcb740 1 -- 172.21.15.90:0/2895898039 wait complete. 2024-04-02T14:44:59.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.446+0000 7f1d85dcb740 20 client.28492 trim_cache size 0 max 16384 2024-04-02T14:44:59.445 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-02T14:44:59.445 INFO:teuthology.orchestra.run.smithi090.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-02T14:44:59.445 INFO:teuthology.orchestra.run.smithi090.stdout: "scrubs": { 2024-04-02T14:44:59.445 INFO:teuthology.orchestra.run.smithi090.stdout: "0f2eb92d-5324-4420-99e2-479e4213fae2": { 2024-04-02T14:44:59.445 INFO:teuthology.orchestra.run.smithi090.stdout: "path": "/", 2024-04-02T14:44:59.445 INFO:teuthology.orchestra.run.smithi090.stdout: "tag": "0f2eb92d-5324-4420-99e2-479e4213fae2", 2024-04-02T14:44:59.445 INFO:teuthology.orchestra.run.smithi090.stdout: "options": "recursive,force" 2024-04-02T14:44:59.445 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:44:59.445 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:44:59.446 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-02T14:44:59.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.446+0000 7f1d83000640 1 -- 172.21.15.90:0/190944222 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f1d600a3cb0 msgr2=0x7f1d600a6170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:59.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.446+0000 7f1d83000640 1 --2- 172.21.15.90:0/190944222 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f1d600a3cb0 0x7f1d600a6170 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f1d740325a0 tx=0x7f1d74000f40 comp rx=0 tx=0).stop 2024-04-02T14:44:59.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.447+0000 7f1d83000640 1 -- 172.21.15.90:0/190944222 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1d7c106a50 msgr2=0x7f1d7c19e850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:44:59.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.447+0000 7f1d83000640 1 --2- 172.21.15.90:0/190944222 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1d7c106a50 0x7f1d7c19e850 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f1d6c00b3f0 tx=0x7f1d6c00b8b0 comp rx=0 tx=0).stop 2024-04-02T14:44:59.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.447+0000 7f1d83000640 1 -- 172.21.15.90:0/190944222 shutdown_connections 2024-04-02T14:44:59.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.447+0000 7f1d83000640 1 --2- 172.21.15.90:0/190944222 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f1d600a3cb0 0x7f1d600a6170 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:59.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.447+0000 7f1d83000640 1 --2- 172.21.15.90:0/190944222 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1d7c106a50 0x7f1d7c19e850 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:59.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.447+0000 7f1d83000640 1 --2- 172.21.15.90:0/190944222 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1d7c1045c0 0x7f1d7c1a4c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:59.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.447+0000 7f1d83000640 1 --2- 172.21.15.90:0/190944222 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1d7c069240 0x7f1d7c1a46e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:44:59.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.447+0000 7f1d83000640 1 -- 172.21.15.90:0/190944222 >> 172.21.15.90:0/190944222 conn(0x7f1d7c077b00 msgr2=0x7f1d7c101920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:44:59.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.447+0000 7f1d83000640 1 -- 172.21.15.90:0/190944222 shutdown_connections 2024-04-02T14:44:59.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:44:59.447+0000 7f1d83000640 1 -- 172.21.15.90:0/190944222 wait complete. 2024-04-02T14:44:59.466 INFO:tasks.cephfs.filesystem:scrub status for tag:0f2eb92d-5324-4420-99e2-479e4213fae2 - {'path': '/', 'tag': '0f2eb92d-5324-4420-99e2-479e4213fae2', 'options': 'recursive,force'} 2024-04-02T14:44:59.931 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:44:59 smithi177 ceph-mon[36431]: pgmap v312: 97 pgs: 97 active+clean; 4.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 6.9 MiB/s wr, 690 op/s 2024-04-02T14:44:59.933 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:44:59 smithi090 ceph-mon[31674]: pgmap v312: 97 pgs: 97 active+clean; 4.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 6.9 MiB/s wr, 690 op/s 2024-04-02T14:45:00.105 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:44:59 smithi092 ceph-mon[36497]: pgmap v312: 97 pgs: 97 active+clean; 4.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 6.9 MiB/s wr, 690 op/s 2024-04-02T14:45:00.931 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:45:00 smithi177 ceph-mon[36431]: scrub queued for path: / 2024-04-02T14:45:00.931 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:45:00 smithi177 ceph-mon[36431]: scrub summary: idle+waiting paths [/] 2024-04-02T14:45:00.931 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:45:00 smithi177 ceph-mon[36431]: scrub summary: active paths [/] 2024-04-02T14:45:00.931 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:45:00 smithi177 ceph-mon[36431]: scrub summary: idle+waiting paths [/] 2024-04-02T14:45:00.933 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:45:00 smithi090 ceph-mon[31674]: scrub queued for path: / 2024-04-02T14:45:00.933 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:45:00 smithi090 ceph-mon[31674]: scrub summary: idle+waiting paths [/] 2024-04-02T14:45:00.933 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:45:00 smithi090 ceph-mon[31674]: scrub summary: active paths [/] 2024-04-02T14:45:00.933 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:45:00 smithi090 ceph-mon[31674]: scrub summary: idle+waiting paths [/] 2024-04-02T14:45:01.105 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:45:00 smithi092 ceph-mon[36497]: scrub queued for path: / 2024-04-02T14:45:01.105 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:45:00 smithi092 ceph-mon[36497]: scrub summary: idle+waiting paths [/] 2024-04-02T14:45:01.106 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:45:00 smithi092 ceph-mon[36497]: scrub summary: active paths [/] 2024-04-02T14:45:01.106 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:45:00 smithi092 ceph-mon[36497]: scrub summary: idle+waiting paths [/] 2024-04-02T14:45:01.826 INFO:teuthology.orchestra.run.smithi090.stderr:progress: 570.0 s, 1674.8 tps, lat 56.808 ms stddev 89.751 2024-04-02T14:45:02.105 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:45:01 smithi092 ceph-mon[36497]: pgmap v313: 97 pgs: 97 active+clean; 4.8 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 7.9 MiB/s wr, 795 op/s 2024-04-02T14:45:02.181 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:45:01 smithi177 ceph-mon[36431]: pgmap v313: 97 pgs: 97 active+clean; 4.8 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 7.9 MiB/s wr, 795 op/s 2024-04-02T14:45:02.183 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:45:01 smithi090 ceph-mon[31674]: pgmap v313: 97 pgs: 97 active+clean; 4.8 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 7.9 MiB/s wr, 795 op/s 2024-04-02T14:45:03.105 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:45:02 smithi092 ceph-mon[36497]: scrub summary: idle 2024-04-02T14:45:03.181 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:45:02 smithi177 ceph-mon[36431]: scrub summary: idle 2024-04-02T14:45:03.184 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:45:02 smithi090 ceph-mon[31674]: scrub summary: idle 2024-04-02T14:45:04.105 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:45:03 smithi092 ceph-mon[36497]: pgmap v314: 97 pgs: 97 active+clean; 4.8 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 6.0 MiB/s wr, 628 op/s 2024-04-02T14:45:04.181 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:45:03 smithi177 ceph-mon[36431]: pgmap v314: 97 pgs: 97 active+clean; 4.8 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 6.0 MiB/s wr, 628 op/s 2024-04-02T14:45:04.183 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:45:03 smithi090 ceph-mon[31674]: pgmap v314: 97 pgs: 97 active+clean; 4.8 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 6.0 MiB/s wr, 628 op/s 2024-04-02T14:45:06.105 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:45:05 smithi092 ceph-mon[36497]: pgmap v315: 97 pgs: 97 active+clean; 4.8 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 16 MiB/s wr, 1.46k op/s 2024-04-02T14:45:06.181 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:45:05 smithi177 ceph-mon[36431]: pgmap v315: 97 pgs: 97 active+clean; 4.8 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 16 MiB/s wr, 1.46k op/s 2024-04-02T14:45:06.184 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:45:05 smithi090 ceph-mon[31674]: pgmap v315: 97 pgs: 97 active+clean; 4.8 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 16 MiB/s wr, 1.46k op/s 2024-04-02T14:45:06.826 INFO:teuthology.orchestra.run.smithi090.stderr:progress: 575.0 s, 611.2 tps, lat 168.102 ms stddev 513.166 2024-04-02T14:45:08.105 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:45:07 smithi092 ceph-mon[36497]: pgmap v316: 97 pgs: 97 active+clean; 4.8 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 13 MiB/s wr, 1.19k op/s 2024-04-02T14:45:08.105 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:45:07 smithi092 ceph-mon[36497]: osdmap e98: 12 total, 12 up, 12 in 2024-04-02T14:45:08.181 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:45:07 smithi177 ceph-mon[36431]: pgmap v316: 97 pgs: 97 active+clean; 4.8 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 13 MiB/s wr, 1.19k op/s 2024-04-02T14:45:08.181 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:45:07 smithi177 ceph-mon[36431]: osdmap e98: 12 total, 12 up, 12 in 2024-04-02T14:45:08.183 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:45:07 smithi090 ceph-mon[31674]: pgmap v316: 97 pgs: 97 active+clean; 4.8 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 13 MiB/s wr, 1.19k op/s 2024-04-02T14:45:08.183 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:45:07 smithi090 ceph-mon[31674]: osdmap e98: 12 total, 12 up, 12 in 2024-04-02T14:45:10.105 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:45:09 smithi092 ceph-mon[36497]: pgmap v318: 97 pgs: 11 active+clean+snaptrim_wait, 8 active+clean+snaptrim, 78 active+clean; 4.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 152 KiB/s rd, 18 MiB/s wr, 1.58k op/s 2024-04-02T14:45:10.181 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:45:09 smithi177 ceph-mon[36431]: pgmap v318: 97 pgs: 11 active+clean+snaptrim_wait, 8 active+clean+snaptrim, 78 active+clean; 4.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 152 KiB/s rd, 18 MiB/s wr, 1.58k op/s 2024-04-02T14:45:10.183 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:45:09 smithi090 ceph-mon[31674]: pgmap v318: 97 pgs: 11 active+clean+snaptrim_wait, 8 active+clean+snaptrim, 78 active+clean; 4.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 152 KiB/s rd, 18 MiB/s wr, 1.58k op/s 2024-04-02T14:45:11.826 INFO:teuthology.orchestra.run.smithi090.stderr:progress: 580.0 s, 355.2 tps, lat 288.266 ms stddev 759.085 2024-04-02T14:45:12.105 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:45:11 smithi092 ceph-mon[36497]: pgmap v319: 97 pgs: 38 active+clean+snaptrim_wait, 21 active+clean+snaptrim, 38 active+clean; 4.6 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 22 MiB/s wr, 1.83k op/s 2024-04-02T14:45:12.181 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:45:11 smithi177 ceph-mon[36431]: pgmap v319: 97 pgs: 38 active+clean+snaptrim_wait, 21 active+clean+snaptrim, 38 active+clean; 4.6 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 22 MiB/s wr, 1.83k op/s 2024-04-02T14:45:12.183 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:45:11 smithi090 ceph-mon[31674]: pgmap v319: 97 pgs: 38 active+clean+snaptrim_wait, 21 active+clean+snaptrim, 38 active+clean; 4.6 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 22 MiB/s wr, 1.83k op/s 2024-04-02T14:45:12.600 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.603+0000 7f0c7f400640 1 -- 172.21.15.90:0/3969491063 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0c78112ac0 msgr2=0x7f0c78114eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:45:12.600 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.603+0000 7f0c7f400640 1 --2- 172.21.15.90:0/3969491063 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0c78112ac0 0x7f0c78114eb0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f0c70009900 tx=0x7f0c7002bcd0 comp rx=0 tx=0).stop 2024-04-02T14:45:12.600 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.603+0000 7f0c7f400640 1 -- 172.21.15.90:0/3969491063 shutdown_connections 2024-04-02T14:45:12.601 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.603+0000 7f0c7f400640 1 --2- 172.21.15.90:0/3969491063 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0c78112ac0 0x7f0c78114eb0 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:12.601 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.603+0000 7f0c7f400640 1 --2- 172.21.15.90:0/3969491063 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0c780747c0 0x7f0c78112580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:12.601 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.603+0000 7f0c7f400640 1 --2- 172.21.15.90:0/3969491063 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0c78073ea0 0x7f0c78074280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:12.601 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.603+0000 7f0c7f400640 1 -- 172.21.15.90:0/3969491063 >> 172.21.15.90:0/3969491063 conn(0x7f0c780fd580 msgr2=0x7f0c780ff9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:45:12.601 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.603+0000 7f0c7f400640 1 -- 172.21.15.90:0/3969491063 shutdown_connections 2024-04-02T14:45:12.601 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.604+0000 7f0c7f400640 1 -- 172.21.15.90:0/3969491063 wait complete. 2024-04-02T14:45:12.601 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.604+0000 7f0c7f400640 1 Processor -- start 2024-04-02T14:45:12.602 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.605+0000 7f0c7f400640 1 -- start start 2024-04-02T14:45:12.602 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.606+0000 7f0c7f400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0c78073ea0 0x7f0c781a4650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:12.603 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.606+0000 7f0c7f400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0c780747c0 0x7f0c781a4b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:12.603 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.606+0000 7f0c7f400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0c78112ac0 0x7f0c7819e710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:12.603 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.606+0000 7f0c7f400640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f0c781184b0 con 0x7f0c78112ac0 2024-04-02T14:45:12.603 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.606+0000 7f0c7f400640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f0c78118330 con 0x7f0c780747c0 2024-04-02T14:45:12.604 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.606+0000 7f0c7f400640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f0c78118630 con 0x7f0c78073ea0 2024-04-02T14:45:12.604 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.606+0000 7f0c7ca00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0c78073ea0 0x7f0c781a4650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:12.604 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.606+0000 7f0c7ca00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0c78073ea0 0x7f0c781a4650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:51062/0 (socket says 172.21.15.90:51062) 2024-04-02T14:45:12.604 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.606+0000 7f0c7ca00640 1 -- 172.21.15.90:0/1113974471 learned_addr learned my addr 172.21.15.90:0/1113974471 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:45:12.604 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.606+0000 7f0c77e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0c780747c0 0x7f0c781a4b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:12.604 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.607+0000 7f0c7ca00640 1 -- 172.21.15.90:0/1113974471 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0c780747c0 msgr2=0x7f0c781a4b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:45:12.604 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.607+0000 7f0c7ca00640 1 --2- 172.21.15.90:0/1113974471 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0c780747c0 0x7f0c781a4b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:12.604 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.607+0000 7f0c7ca00640 1 -- 172.21.15.90:0/1113974471 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0c78112ac0 msgr2=0x7f0c7819e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:45:12.604 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.607+0000 7f0c7d400640 1 --2- 172.21.15.90:0/1113974471 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0c78112ac0 0x7f0c7819e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:12.604 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.607+0000 7f0c7ca00640 1 --2- 172.21.15.90:0/1113974471 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0c78112ac0 0x7f0c7819e710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:12.604 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.607+0000 7f0c7ca00640 1 -- 172.21.15.90:0/1113974471 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c7819efd0 con 0x7f0c78073ea0 2024-04-02T14:45:12.605 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.607+0000 7f0c7d400640 1 --2- 172.21.15.90:0/1113974471 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0c78112ac0 0x7f0c7819e710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:45:12.605 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.607+0000 7f0c7ca00640 1 --2- 172.21.15.90:0/1113974471 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0c78073ea0 0x7f0c781a4650 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f0c6c00ebc0 tx=0x7f0c6c00c6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:45:12.605 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.608+0000 7f0c75600640 1 -- 172.21.15.90:0/1113974471 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c6c010630 con 0x7f0c78073ea0 2024-04-02T14:45:12.605 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.608+0000 7f0c75600640 1 -- 172.21.15.90:0/1113974471 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0c6c010cb0 con 0x7f0c78073ea0 2024-04-02T14:45:12.605 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.608+0000 7f0c7f400640 1 -- 172.21.15.90:0/1113974471 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0c7819f2c0 con 0x7f0c78073ea0 2024-04-02T14:45:12.605 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.608+0000 7f0c75600640 1 -- 172.21.15.90:0/1113974471 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c6c018c00 con 0x7f0c78073ea0 2024-04-02T14:45:12.605 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.608+0000 7f0c7f400640 1 -- 172.21.15.90:0/1113974471 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0c781ab5d0 con 0x7f0c78073ea0 2024-04-02T14:45:12.609 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.613+0000 7f0c82186740 1 Processor -- start 2024-04-02T14:45:12.610 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.613+0000 7f0c82186740 1 -- start start 2024-04-02T14:45:12.611 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.613+0000 7f0c82186740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5624c0fefeb0 0x5624c0ff0290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:12.611 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.613+0000 7f0c82186740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5624c0ff07d0 0x5624c0ff8e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:12.611 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.613+0000 7f0c82186740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5624c0ff93b0 0x5624c0ffb7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:12.611 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.614+0000 7f0c82186740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5624c0f54d70 con 0x5624c0fefeb0 2024-04-02T14:45:12.611 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.614+0000 7f0c82186740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x5624c0efd2a0 con 0x5624c0ff07d0 2024-04-02T14:45:12.611 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.614+0000 7f0c82186740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5624c0f50d20 con 0x5624c0ff93b0 2024-04-02T14:45:12.611 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.614+0000 7f0c75600640 1 -- 172.21.15.90:0/1113974471 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f0c6c0107d0 con 0x7f0c78073ea0 2024-04-02T14:45:12.611 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.614+0000 7f0c77e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5624c0ff93b0 0x5624c0ffb7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:12.611 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.614+0000 7f0c77e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5624c0ff93b0 0x5624c0ffb7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:51070/0 (socket says 172.21.15.90:51070) 2024-04-02T14:45:12.611 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.614+0000 7f0c77e00640 1 -- 172.21.15.90:0/954815937 learned_addr learned my addr 172.21.15.90:0/954815937 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:45:12.611 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.614+0000 7f0c7d400640 1 --2- 172.21.15.90:0/954815937 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5624c0fefeb0 0x5624c0ff0290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:12.612 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.614+0000 7f0c77e00640 1 -- 172.21.15.90:0/954815937 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5624c0ff07d0 msgr2=0x5624c0ff8e70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-02T14:45:12.613 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.615+0000 7f0c77e00640 1 --2- 172.21.15.90:0/954815937 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5624c0ff07d0 0x5624c0ff8e70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:12.613 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.615+0000 7f0c77e00640 1 -- 172.21.15.90:0/954815937 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5624c0fefeb0 msgr2=0x5624c0ff0290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:45:12.613 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.615+0000 7f0c77e00640 1 --2- 172.21.15.90:0/954815937 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5624c0fefeb0 0x5624c0ff0290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:12.613 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.615+0000 7f0c77e00640 1 -- 172.21.15.90:0/954815937 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5624c0f11840 con 0x5624c0ff93b0 2024-04-02T14:45:12.613 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.615+0000 7f0c7d400640 1 --2- 172.21.15.90:0/954815937 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5624c0fefeb0 0x5624c0ff0290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:45:12.613 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.615+0000 7f0c77e00640 1 --2- 172.21.15.90:0/954815937 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5624c0ff93b0 0x5624c0ffb7a0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f0c64009c10 tx=0x7f0c64011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ba95fefa30d8b07e server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:45:12.613 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.616+0000 7f0c69c00640 1 -- 172.21.15.90:0/954815937 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c64004230 con 0x5624c0ff93b0 2024-04-02T14:45:12.613 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.616+0000 7f0c69c00640 1 -- 172.21.15.90:0/954815937 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0c640043d0 con 0x5624c0ff93b0 2024-04-02T14:45:12.613 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.616+0000 7f0c69c00640 1 -- 172.21.15.90:0/954815937 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c640197a0 con 0x5624c0ff93b0 2024-04-02T14:45:12.614 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.617+0000 7f0c82186740 1 -- 172.21.15.90:0/954815937 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5624c0ff93b0 msgr2=0x5624c0ffb7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:45:12.614 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.617+0000 7f0c82186740 1 --2- 172.21.15.90:0/954815937 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5624c0ff93b0 0x5624c0ffb7a0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f0c64009c10 tx=0x7f0c64011810 comp rx=0 tx=0).stop 2024-04-02T14:45:12.615 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.617+0000 7f0c75600640 1 --2- 172.21.15.90:0/1113974471 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f0c600a3fa0 0x7f0c600a6460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:12.615 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.618+0000 7f0c75600640 1 -- 172.21.15.90:0/1113974471 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 8594+0+0 (secure 0 0 0) 0x7f0c6c0d1030 con 0x7f0c78073ea0 2024-04-02T14:45:12.615 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.618+0000 7f0c7d400640 1 --2- 172.21.15.90:0/1113974471 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f0c600a3fa0 0x7f0c600a6460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:12.616 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.619+0000 7f0c82186740 1 -- 172.21.15.90:0/954815937 shutdown_connections 2024-04-02T14:45:12.616 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.619+0000 7f0c82186740 1 --2- 172.21.15.90:0/954815937 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5624c0ff93b0 0x5624c0ffb7a0 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:12.617 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.620+0000 7f0c7d400640 1 --2- 172.21.15.90:0/1113974471 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f0c600a3fa0 0x7f0c600a6460 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f0c7002c210 tx=0x7f0c70005b50 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:45:12.617 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.620+0000 7f0c82186740 1 --2- 172.21.15.90:0/954815937 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5624c0ff07d0 0x5624c0ff8e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:12.617 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.620+0000 7f0c82186740 1 --2- 172.21.15.90:0/954815937 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5624c0fefeb0 0x5624c0ff0290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:12.617 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.620+0000 7f0c82186740 1 -- 172.21.15.90:0/954815937 >> 172.21.15.90:0/954815937 conn(0x5624c0fea5e0 msgr2=0x5624c0fea9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:45:12.617 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.620+0000 7f0c82186740 1 -- 172.21.15.90:0/954815937 shutdown_connections 2024-04-02T14:45:12.617 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.620+0000 7f0c82186740 1 -- 172.21.15.90:0/954815937 wait complete. 2024-04-02T14:45:12.618 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.621+0000 7f0c82186740 1 -- start start 2024-04-02T14:45:12.618 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.621+0000 7f0c82186740 1 Processor -- start 2024-04-02T14:45:12.619 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.622+0000 7f0c82186740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5624c0fefeb0 0x5624c0ff64a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:12.619 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.622+0000 7f0c82186740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5624c0ff07d0 0x5624c0ff69e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:12.620 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.622+0000 7f0c82186740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5624c0ff93b0 0x5624c0ff6f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:12.620 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.622+0000 7f0c82186740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5624c0ffc1a0 con 0x5624c0ff93b0 2024-04-02T14:45:12.620 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.622+0000 7f0c82186740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x5624c0f10660 con 0x5624c0ff07d0 2024-04-02T14:45:12.620 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.622+0000 7f0c82186740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5624c0ffc320 con 0x5624c0fefeb0 2024-04-02T14:45:12.620 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.622+0000 7f0c82186740 10 client.0 _ll_register_callbacks cb 0x5624c0f10050 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:45:12.620 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.622+0000 7f0c7ca00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5624c0fefeb0 0x5624c0ff64a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:12.620 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.622+0000 7f0c7ca00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5624c0fefeb0 0x5624c0ff64a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:51072/0 (socket says 172.21.15.90:51072) 2024-04-02T14:45:12.620 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.622+0000 7f0c7ca00640 1 -- 172.21.15.90:0/4108972982 learned_addr learned my addr 172.21.15.90:0/4108972982 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:45:12.620 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.622+0000 7f0c7d400640 1 --2- 172.21.15.90:0/4108972982 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5624c0ff93b0 0x5624c0ff6f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:12.620 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.622+0000 7f0c77e00640 1 --2- 172.21.15.90:0/4108972982 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5624c0ff07d0 0x5624c0ff69e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:12.622 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.623+0000 7f0c7ca00640 1 -- 172.21.15.90:0/4108972982 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5624c0ff07d0 msgr2=0x5624c0ff69e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:45:12.622 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.623+0000 7f0c7ca00640 1 --2- 172.21.15.90:0/4108972982 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5624c0ff07d0 0x5624c0ff69e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:12.623 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.623+0000 7f0c7ca00640 1 -- 172.21.15.90:0/4108972982 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5624c0ff93b0 msgr2=0x5624c0ff6f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:45:12.623 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.623+0000 7f0c7ca00640 1 --2- 172.21.15.90:0/4108972982 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5624c0ff93b0 0x5624c0ff6f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:12.623 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.623+0000 7f0c7ca00640 1 -- 172.21.15.90:0/4108972982 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5624c0f1fe80 con 0x5624c0fefeb0 2024-04-02T14:45:12.623 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.623+0000 7f0c7d400640 1 --2- 172.21.15.90:0/4108972982 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5624c0ff93b0 0x5624c0ff6f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:45:12.623 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.623+0000 7f0c7ca00640 1 --2- 172.21.15.90:0/4108972982 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5624c0fefeb0 0x5624c0ff64a0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f0c6c09cc20 tx=0x7f0c6c09cdb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c48ce71e6b0282c5 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:45:12.623 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.624+0000 7f0c5e000640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:45:12.623 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.624+0000 7f0c5e000640 1 -- 172.21.15.90:0/4108972982 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c6c0b6060 con 0x5624c0fefeb0 2024-04-02T14:45:12.623 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.624+0000 7f0c5e000640 1 -- 172.21.15.90:0/4108972982 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8594+0+0 (secure 0 0 0) 0x7f0c6c0b6400 con 0x5624c0fefeb0 2024-04-02T14:45:12.623 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.624+0000 7f0c82186740 1 -- 172.21.15.90:0/4108972982 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0c640043d0 con 0x5624c0fefeb0 2024-04-02T14:45:12.623 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.624+0000 7f0c5e000640 1 -- 172.21.15.90:0/4108972982 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f0c6c0bb050 con 0x5624c0fefeb0 2024-04-02T14:45:12.623 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.624+0000 7f0c82186740 10 client.28504 fetch_fsmap learned FSMap version 27 2024-04-02T14:45:12.623 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.624+0000 7f0c82186740 1 -- 172.21.15.90:0/4108972982 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5624c0c69210 con 0x5624c0fefeb0 2024-04-02T14:45:12.623 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.625+0000 7f0c5e000640 1 -- 172.21.15.90:0/4108972982 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0c640043d0 con 0x5624c0fefeb0 2024-04-02T14:45:12.623 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.625+0000 7f0c5e000640 1 -- 172.21.15.90:0/4108972982 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f0c6c0a0030 con 0x5624c0fefeb0 2024-04-02T14:45:12.624 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.625+0000 7f0c82186740 10 client.28504 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:45:12.624 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.625+0000 7f0c82186740 10 client.28504 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-02T14:45:12.624 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.625+0000 7f0c82186740 20 client.28504 populate_metadata read hostname 'smithi090' 2024-04-02T14:45:12.624 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.625+0000 7f0c82186740 1 --2- 172.21.15.90:0/4108972982 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5624c10097f0 0x5624c100bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:12.624 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.625+0000 7f0c82186740 4 client.28504 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:45:12.624 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.625+0000 7f0c82186740 1 -- 172.21.15.90:0/4108972982 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5624c0f04cd0 con 0x5624c10097f0 2024-04-02T14:45:12.624 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.626+0000 7f0c77e00640 1 --2- 172.21.15.90:0/4108972982 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5624c10097f0 0x5624c100bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:12.625 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.627+0000 7f0c77e00640 1 --2- 172.21.15.90:0/4108972982 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5624c10097f0 0x5624c100bbe0 crc :-1 s=READY pgs=274 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3925bd8d54e8f152 server_cookie=1923c9ae948cf361 in_seq=0 out_seq=0 2024-04-02T14:45:12.625 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.627+0000 7f0c5e000640 10 client.28504 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:45:12.625 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.628+0000 7f0c5e000640 1 -- 172.21.15.90:0/4108972982 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5624c0f04cd0 con 0x5624c10097f0 2024-04-02T14:45:12.625 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.628+0000 7f0c5e000640 10 client.28504 handle_command_reply: tid=0 2024-04-02T14:45:12.625 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.628+0000 7f0c82186740 1 client.28504 shutdown 2024-04-02T14:45:12.625 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.628+0000 7f0c82186740 1 -- 172.21.15.90:0/4108972982 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5624c0fefeb0 msgr2=0x5624c0ff64a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:45:12.625 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.628+0000 7f0c82186740 1 --2- 172.21.15.90:0/4108972982 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5624c0fefeb0 0x5624c0ff64a0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f0c6c09cc20 tx=0x7f0c6c09cdb0 comp rx=0 tx=0).stop 2024-04-02T14:45:12.626 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.628+0000 7f0c82186740 1 -- 172.21.15.90:0/4108972982 shutdown_connections 2024-04-02T14:45:12.626 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.628+0000 7f0c82186740 1 --2- 172.21.15.90:0/4108972982 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5624c10097f0 0x5624c100bbe0 crc :-1 s=READY pgs=274 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:12.626 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.628+0000 7f0c82186740 1 --2- 172.21.15.90:0/4108972982 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5624c0ff93b0 0x5624c0ff6f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:12.626 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.628+0000 7f0c82186740 1 --2- 172.21.15.90:0/4108972982 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5624c0ff07d0 0x5624c0ff69e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:12.626 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.628+0000 7f0c82186740 1 --2- 172.21.15.90:0/4108972982 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5624c0fefeb0 0x5624c0ff64a0 secure :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f0c6c09cc20 tx=0x7f0c6c09cdb0 comp rx=0 tx=0).stop 2024-04-02T14:45:12.626 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.628+0000 7f0c82186740 1 -- 172.21.15.90:0/4108972982 >> 172.21.15.90:0/4108972982 conn(0x5624c0fea5e0 msgr2=0x5624c0fec330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:45:12.626 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.629+0000 7f0c5e000640 0 client.28504 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:45:12.626 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.629+0000 7f0c82186740 1 -- 172.21.15.90:0/4108972982 shutdown_connections 2024-04-02T14:45:12.626 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.629+0000 7f0c82186740 1 -- 172.21.15.90:0/4108972982 wait complete. 2024-04-02T14:45:12.626 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.629+0000 7f0c82186740 20 client.28504 trim_cache size 0 max 16384 2024-04-02T14:45:12.670 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.673+0000 7f0c82186740 1 Processor -- start 2024-04-02T14:45:12.670 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.674+0000 7f0c82186740 1 -- start start 2024-04-02T14:45:12.671 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.674+0000 7f0c82186740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5624c0ff07d0 0x5624c0ff8bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:12.672 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.674+0000 7f0c82186740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5624c0ff93b0 0x5624c1005060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:12.672 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.674+0000 7f0c82186740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5624c0fe9640 0x5624c1011390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:12.672 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.674+0000 7f0c82186740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f0c781b3f00 con 0x5624c0ff07d0 2024-04-02T14:45:12.672 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.674+0000 7f0c82186740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f0c781b5a40 con 0x5624c0fe9640 2024-04-02T14:45:12.672 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.674+0000 7f0c82186740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f0c781b45d0 con 0x5624c0ff93b0 2024-04-02T14:45:12.672 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.675+0000 7f0c7d400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5624c0fe9640 0x5624c1011390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:12.672 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.675+0000 7f0c7d400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5624c0fe9640 0x5624c1011390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:34340/0 (socket says 172.21.15.90:34340) 2024-04-02T14:45:12.672 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.675+0000 7f0c7d400640 1 -- 172.21.15.90:0/3624910525 learned_addr learned my addr 172.21.15.90:0/3624910525 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:45:12.673 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.676+0000 7f0c7d400640 1 -- 172.21.15.90:0/3624910525 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5624c0ff93b0 msgr2=0x5624c1005060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:45:12.673 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.676+0000 7f0c7d400640 1 --2- 172.21.15.90:0/3624910525 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5624c0ff93b0 0x5624c1005060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:12.673 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.677+0000 7f0c7d400640 1 -- 172.21.15.90:0/3624910525 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5624c0ff07d0 msgr2=0x5624c0ff8bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:45:12.674 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.677+0000 7f0c7d400640 1 --2- 172.21.15.90:0/3624910525 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5624c0ff07d0 0x5624c0ff8bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:12.674 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.677+0000 7f0c7d400640 1 -- 172.21.15.90:0/3624910525 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5624c0ff5450 con 0x5624c0fe9640 2024-04-02T14:45:12.675 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.678+0000 7f0c7d400640 1 --2- 172.21.15.90:0/3624910525 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5624c0fe9640 0x5624c1011390 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f0c7806b150 tx=0x7f0c7003e000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=48e923b7e0accc93 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:45:12.675 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.679+0000 7f0c69200640 1 -- 172.21.15.90:0/3624910525 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c70047050 con 0x5624c0fe9640 2024-04-02T14:45:12.676 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.679+0000 7f0c69200640 1 -- 172.21.15.90:0/3624910525 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0c70051070 con 0x5624c0fe9640 2024-04-02T14:45:12.676 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.679+0000 7f0c69200640 1 -- 172.21.15.90:0/3624910525 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c70004d70 con 0x5624c0fe9640 2024-04-02T14:45:12.678 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.682+0000 7f0c82186740 1 -- 172.21.15.90:0/3624910525 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5624c0fe9640 msgr2=0x5624c1011390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:45:12.678 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.682+0000 7f0c82186740 1 --2- 172.21.15.90:0/3624910525 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5624c0fe9640 0x5624c1011390 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f0c7806b150 tx=0x7f0c7003e000 comp rx=0 tx=0).stop 2024-04-02T14:45:12.678 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.682+0000 7f0c82186740 1 -- 172.21.15.90:0/3624910525 shutdown_connections 2024-04-02T14:45:12.679 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.682+0000 7f0c82186740 1 --2- 172.21.15.90:0/3624910525 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5624c0fe9640 0x5624c1011390 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:12.679 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.682+0000 7f0c82186740 1 --2- 172.21.15.90:0/3624910525 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5624c0ff93b0 0x5624c1005060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:12.679 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.682+0000 7f0c82186740 1 --2- 172.21.15.90:0/3624910525 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5624c0ff07d0 0x5624c0ff8bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:12.679 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.682+0000 7f0c82186740 1 -- 172.21.15.90:0/3624910525 >> 172.21.15.90:0/3624910525 conn(0x5624c0fea5e0 msgr2=0x5624c0fea9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:45:12.680 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.684+0000 7f0c82186740 1 -- 172.21.15.90:0/3624910525 shutdown_connections 2024-04-02T14:45:12.682 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.684+0000 7f0c82186740 1 -- 172.21.15.90:0/3624910525 wait complete. 2024-04-02T14:45:12.682 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.684+0000 7f0c82186740 1 -- start start 2024-04-02T14:45:12.682 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.685+0000 7f0c82186740 1 Processor -- start 2024-04-02T14:45:12.683 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.687+0000 7f0c82186740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5624c0fe9640 0x5624c1001760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:12.684 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.687+0000 7f0c82186740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5624c0ff07d0 0x5624c0ffe540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:12.684 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.688+0000 7f0c82186740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5624c0ff93b0 0x5624c0ffea80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:12.686 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.688+0000 7f0c77e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5624c0ff07d0 0x5624c0ffe540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:12.686 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.689+0000 7f0c7d400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5624c0ff93b0 0x5624c0ffea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:12.686 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.689+0000 7f0c77e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5624c0ff07d0 0x5624c0ffe540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:34344/0 (socket says 172.21.15.90:34344) 2024-04-02T14:45:12.686 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.689+0000 7f0c7d400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5624c0ff93b0 0x5624c0ffea80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:51088/0 (socket says 172.21.15.90:51088) 2024-04-02T14:45:12.686 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.689+0000 7f0c82186740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f0c781b5370 con 0x5624c0fe9640 2024-04-02T14:45:12.686 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.689+0000 7f0c77e00640 1 -- 172.21.15.90:0/270474430 learned_addr learned my addr 172.21.15.90:0/270474430 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:45:12.686 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.689+0000 7f0c82186740 1 -- 172.21.15.90:0/270474430 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f0c781b4ca0 con 0x5624c0ff07d0 2024-04-02T14:45:12.686 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.689+0000 7f0c82186740 1 -- 172.21.15.90:0/270474430 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f0c781b1f00 con 0x5624c0ff93b0 2024-04-02T14:45:12.686 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.689+0000 7f0c82186740 10 client.0 _ll_register_callbacks cb 0x7f0c6c098aa0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:45:12.687 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.690+0000 7f0c7d400640 1 -- 172.21.15.90:0/270474430 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5624c0ff07d0 msgr2=0x5624c0ffe540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:45:12.687 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.690+0000 7f0c7d400640 1 --2- 172.21.15.90:0/270474430 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5624c0ff07d0 0x5624c0ffe540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:12.687 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.690+0000 7f0c7d400640 1 -- 172.21.15.90:0/270474430 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5624c0fe9640 msgr2=0x5624c1001760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:45:12.687 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.690+0000 7f0c7d400640 1 --2- 172.21.15.90:0/270474430 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5624c0fe9640 0x5624c1001760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:12.687 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.690+0000 7f0c7d400640 1 -- 172.21.15.90:0/270474430 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5624c0d35620 con 0x5624c0ff93b0 2024-04-02T14:45:12.688 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.691+0000 7f0c7d400640 1 --2- 172.21.15.90:0/270474430 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5624c0ff93b0 0x5624c0ffea80 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f0c70006010 tx=0x7f0c70003cf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ec7417514180f517 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:45:12.689 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.691+0000 7f0c5e000640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:45:12.689 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.692+0000 7f0c5e000640 1 -- 172.21.15.90:0/270474430 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c7005a070 con 0x5624c0ff93b0 2024-04-02T14:45:12.689 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.692+0000 7f0c5e000640 1 -- 172.21.15.90:0/270474430 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8594+0+0 (secure 0 0 0) 0x7f0c70041680 con 0x5624c0ff93b0 2024-04-02T14:45:12.689 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.692+0000 7f0c82186740 1 -- 172.21.15.90:0/270474430 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0c70051070 con 0x5624c0ff93b0 2024-04-02T14:45:12.689 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.693+0000 7f0c5e000640 1 -- 172.21.15.90:0/270474430 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f0c70047020 con 0x5624c0ff93b0 2024-04-02T14:45:12.689 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.693+0000 7f0c82186740 10 client.28507 fetch_fsmap learned FSMap version 27 2024-04-02T14:45:12.690 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.693+0000 7f0c82186740 1 -- 172.21.15.90:0/270474430 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5624c10707a0 con 0x5624c0ff93b0 2024-04-02T14:45:12.690 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.694+0000 7f0c5e000640 1 -- 172.21.15.90:0/270474430 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0c70051070 con 0x5624c0ff93b0 2024-04-02T14:45:12.692 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.696+0000 7f0c5e000640 1 -- 172.21.15.90:0/270474430 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f0c7004c640 con 0x5624c0ff93b0 2024-04-02T14:45:12.693 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.696+0000 7f0c82186740 10 client.28507 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:45:12.693 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.696+0000 7f0c82186740 10 client.28507 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-02T14:45:12.693 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.696+0000 7f0c82186740 20 client.28507 populate_metadata read hostname 'smithi090' 2024-04-02T14:45:12.693 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.696+0000 7f0c82186740 1 --2- 172.21.15.90:0/270474430 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5624c1070dc0 0x5624c10731e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:12.693 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.696+0000 7f0c82186740 4 client.28507 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-02T14:45:12.693 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.696+0000 7f0c82186740 1 -- 172.21.15.90:0/270474430 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5624c0f622b0 con 0x5624c1070dc0 2024-04-02T14:45:12.693 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.697+0000 7f0c7ca00640 1 --2- 172.21.15.90:0/270474430 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5624c1070dc0 0x5624c10731e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:12.694 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.697+0000 7f0c7ca00640 1 --2- 172.21.15.90:0/270474430 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5624c1070dc0 0x5624c10731e0 crc :-1 s=READY pgs=275 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7b3197477cc117c7 server_cookie=79ef73b3c76e535a in_seq=0 out_seq=0 2024-04-02T14:45:12.694 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.697+0000 7f0c5e000640 10 client.28507 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:45:12.720 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.723+0000 7f0c5e000640 1 -- 172.21.15.90:0/270474430 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5624c0f622b0 con 0x5624c1070dc0 2024-04-02T14:45:12.720 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.723+0000 7f0c5e000640 10 client.28507 handle_command_reply: tid=0 2024-04-02T14:45:12.720 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.723+0000 7f0c82186740 1 client.28507 shutdown 2024-04-02T14:45:12.720 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.723+0000 7f0c82186740 1 -- 172.21.15.90:0/270474430 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5624c0ff93b0 msgr2=0x5624c0ffea80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:45:12.720 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.723+0000 7f0c82186740 1 --2- 172.21.15.90:0/270474430 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5624c0ff93b0 0x5624c0ffea80 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f0c70006010 tx=0x7f0c70003cf0 comp rx=0 tx=0).stop 2024-04-02T14:45:12.721 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.724+0000 7f0c82186740 1 -- 172.21.15.90:0/270474430 shutdown_connections 2024-04-02T14:45:12.721 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.724+0000 7f0c82186740 1 --2- 172.21.15.90:0/270474430 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5624c1070dc0 0x5624c10731e0 crc :-1 s=READY pgs=275 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:12.721 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.724+0000 7f0c82186740 1 --2- 172.21.15.90:0/270474430 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5624c0ff93b0 0x5624c0ffea80 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:12.721 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.724+0000 7f0c82186740 1 --2- 172.21.15.90:0/270474430 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5624c0ff07d0 0x5624c0ffe540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:12.721 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.724+0000 7f0c5e000640 0 client.28507 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:45:12.721 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.724+0000 7f0c82186740 1 --2- 172.21.15.90:0/270474430 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5624c0fe9640 0x5624c1001760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:12.721 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.724+0000 7f0c82186740 1 -- 172.21.15.90:0/270474430 >> 172.21.15.90:0/270474430 conn(0x5624c0fea5e0 msgr2=0x5624c0feb930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:45:12.721 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.724+0000 7f0c82186740 1 -- 172.21.15.90:0/270474430 shutdown_connections 2024-04-02T14:45:12.722 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.724+0000 7f0c82186740 1 -- 172.21.15.90:0/270474430 wait complete. 2024-04-02T14:45:12.722 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.724+0000 7f0c82186740 20 client.28507 trim_cache size 0 max 16384 2024-04-02T14:45:12.722 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:{ 2024-04-02T14:45:12.722 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "message": "", 2024-04-02T14:45:12.722 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "return_code": 0 2024-04-02T14:45:12.722 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:} 2024-04-02T14:45:12.723 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.726+0000 7f0c7f400640 1 -- 172.21.15.90:0/1113974471 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f0c600a3fa0 msgr2=0x7f0c600a6460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:45:12.723 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.726+0000 7f0c7f400640 1 --2- 172.21.15.90:0/1113974471 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f0c600a3fa0 0x7f0c600a6460 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f0c7002c210 tx=0x7f0c70005b50 comp rx=0 tx=0).stop 2024-04-02T14:45:12.723 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.726+0000 7f0c7f400640 1 -- 172.21.15.90:0/1113974471 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0c78073ea0 msgr2=0x7f0c781a4650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:45:12.723 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.726+0000 7f0c7f400640 1 --2- 172.21.15.90:0/1113974471 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0c78073ea0 0x7f0c781a4650 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f0c6c00ebc0 tx=0x7f0c6c00c6a0 comp rx=0 tx=0).stop 2024-04-02T14:45:12.723 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.726+0000 7f0c7f400640 1 -- 172.21.15.90:0/1113974471 shutdown_connections 2024-04-02T14:45:12.723 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.726+0000 7f0c7f400640 1 --2- 172.21.15.90:0/1113974471 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f0c600a3fa0 0x7f0c600a6460 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:12.723 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.726+0000 7f0c7f400640 1 --2- 172.21.15.90:0/1113974471 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0c78112ac0 0x7f0c7819e710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:12.723 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.726+0000 7f0c7f400640 1 --2- 172.21.15.90:0/1113974471 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0c780747c0 0x7f0c781a4b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:12.723 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.726+0000 7f0c7f400640 1 --2- 172.21.15.90:0/1113974471 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0c78073ea0 0x7f0c781a4650 secure :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f0c6c00ebc0 tx=0x7f0c6c00c6a0 comp rx=0 tx=0).stop 2024-04-02T14:45:12.723 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.726+0000 7f0c7f400640 1 -- 172.21.15.90:0/1113974471 >> 172.21.15.90:0/1113974471 conn(0x7f0c780fd580 msgr2=0x7f0c780fe1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:45:12.724 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.728+0000 7f0c7f400640 1 -- 172.21.15.90:0/1113974471 shutdown_connections 2024-04-02T14:45:12.724 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:12.728+0000 7f0c7f400640 1 -- 172.21.15.90:0/1113974471 wait complete. 2024-04-02T14:45:13.986 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:45:13 smithi090 ceph-mon[31674]: pgmap v320: 97 pgs: 38 active+clean+snaptrim_wait, 21 active+clean+snaptrim, 38 active+clean; 4.6 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 22 MiB/s wr, 1.83k op/s 2024-04-02T14:45:14.105 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:45:13 smithi092 ceph-mon[36497]: pgmap v320: 97 pgs: 38 active+clean+snaptrim_wait, 21 active+clean+snaptrim, 38 active+clean; 4.6 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 22 MiB/s wr, 1.83k op/s 2024-04-02T14:45:14.181 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:45:13 smithi177 ceph-mon[36431]: pgmap v320: 97 pgs: 38 active+clean+snaptrim_wait, 21 active+clean+snaptrim, 38 active+clean; 4.6 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 22 MiB/s wr, 1.83k op/s 2024-04-02T14:45:16.355 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:45:16 smithi092 ceph-mon[36497]: pgmap v321: 97 pgs: 97 active+clean; 4.3 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 13 MiB/s wr, 1.03k op/s 2024-04-02T14:45:16.355 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:45:16 smithi092 ceph-mon[36497]: osdmap e99: 12 total, 12 up, 12 in 2024-04-02T14:45:16.431 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:45:16 smithi177 ceph-mon[36431]: pgmap v321: 97 pgs: 97 active+clean; 4.3 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 13 MiB/s wr, 1.03k op/s 2024-04-02T14:45:16.431 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:45:16 smithi177 ceph-mon[36431]: osdmap e99: 12 total, 12 up, 12 in 2024-04-02T14:45:16.433 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:45:16 smithi090 ceph-mon[31674]: pgmap v321: 97 pgs: 97 active+clean; 4.3 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 13 MiB/s wr, 1.03k op/s 2024-04-02T14:45:16.433 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:45:16 smithi090 ceph-mon[31674]: osdmap e99: 12 total, 12 up, 12 in 2024-04-02T14:45:16.826 INFO:teuthology.orchestra.run.smithi090.stderr:progress: 585.0 s, 3005.2 tps, lat 33.266 ms stddev 39.035 2024-04-02T14:45:17.355 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:45:17 smithi092 ceph-mon[36497]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T14:45:17.431 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:45:17 smithi177 ceph-mon[36431]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T14:45:17.433 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:45:17 smithi090 ceph-mon[31674]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T14:45:18.363 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:45:18 smithi092 ceph-mon[36497]: pgmap v323: 97 pgs: 97 active+clean; 4.3 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 13 MiB/s wr, 1.05k op/s 2024-04-02T14:45:18.364 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:45:18 smithi092 ceph-mon[36497]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' 2024-04-02T14:45:18.364 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:45:18 smithi092 ceph-mon[36497]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' 2024-04-02T14:45:18.431 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:45:18 smithi177 ceph-mon[36431]: pgmap v323: 97 pgs: 97 active+clean; 4.3 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 13 MiB/s wr, 1.05k op/s 2024-04-02T14:45:18.431 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:45:18 smithi177 ceph-mon[36431]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' 2024-04-02T14:45:18.431 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:45:18 smithi177 ceph-mon[36431]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' 2024-04-02T14:45:18.433 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:45:18 smithi090 ceph-mon[31674]: pgmap v323: 97 pgs: 97 active+clean; 4.3 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 13 MiB/s wr, 1.05k op/s 2024-04-02T14:45:18.434 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:45:18 smithi090 ceph-mon[31674]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' 2024-04-02T14:45:18.434 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:45:18 smithi090 ceph-mon[31674]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' 2024-04-02T14:45:19.855 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:45:19 smithi092 ceph-mon[36497]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' 2024-04-02T14:45:19.856 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:45:19 smithi092 ceph-mon[36497]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' 2024-04-02T14:45:19.856 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:45:19 smithi092 ceph-mon[36497]: pgmap v324: 97 pgs: 97 active+clean; 4.3 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 11 MiB/s wr, 924 op/s 2024-04-02T14:45:19.856 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:45:19 smithi092 ceph-mon[36497]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' 2024-04-02T14:45:19.856 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:45:19 smithi092 ceph-mon[36497]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' 2024-04-02T14:45:19.856 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:45:19 smithi092 ceph-mon[36497]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-02T14:45:19.856 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:45:19 smithi092 ceph-mon[36497]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-02T14:45:19.856 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:45:19 smithi092 ceph-mon[36497]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-02T14:45:19.856 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:45:19 smithi092 ceph-mon[36497]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-02T14:45:19.856 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:45:19 smithi092 ceph-mon[36497]: Adjusting osd_memory_target on smithi092 to 174.1M 2024-04-02T14:45:19.856 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:45:19 smithi092 ceph-mon[36497]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-02T14:45:19.856 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:45:19 smithi092 ceph-mon[36497]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-02T14:45:19.856 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:45:19 smithi092 ceph-mon[36497]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-02T14:45:19.856 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:45:19 smithi092 ceph-mon[36497]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-02T14:45:19.932 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:45:19 smithi177 ceph-mon[36431]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' 2024-04-02T14:45:19.932 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:45:19 smithi177 ceph-mon[36431]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' 2024-04-02T14:45:19.932 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:45:19 smithi177 ceph-mon[36431]: pgmap v324: 97 pgs: 97 active+clean; 4.3 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 11 MiB/s wr, 924 op/s 2024-04-02T14:45:19.932 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:45:19 smithi177 ceph-mon[36431]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' 2024-04-02T14:45:19.932 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:45:19 smithi177 ceph-mon[36431]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' 2024-04-02T14:45:19.932 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:45:19 smithi177 ceph-mon[36431]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-02T14:45:19.933 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:45:19 smithi177 ceph-mon[36431]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-02T14:45:19.933 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:45:19 smithi177 ceph-mon[36431]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-02T14:45:19.933 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:45:19 smithi177 ceph-mon[36431]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-02T14:45:19.933 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:45:19 smithi177 ceph-mon[36431]: Adjusting osd_memory_target on smithi092 to 174.1M 2024-04-02T14:45:19.933 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:45:19 smithi177 ceph-mon[36431]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-02T14:45:19.933 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:45:19 smithi177 ceph-mon[36431]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-02T14:45:19.933 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:45:19 smithi177 ceph-mon[36431]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-02T14:45:19.933 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:45:19 smithi177 ceph-mon[36431]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-02T14:45:19.934 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:45:19 smithi090 ceph-mon[31674]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' 2024-04-02T14:45:19.934 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:45:19 smithi090 ceph-mon[31674]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' 2024-04-02T14:45:19.934 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:45:19 smithi090 ceph-mon[31674]: pgmap v324: 97 pgs: 97 active+clean; 4.3 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 11 MiB/s wr, 924 op/s 2024-04-02T14:45:19.934 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:45:19 smithi090 ceph-mon[31674]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' 2024-04-02T14:45:19.934 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:45:19 smithi090 ceph-mon[31674]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' 2024-04-02T14:45:19.934 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:45:19 smithi090 ceph-mon[31674]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-02T14:45:19.934 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:45:19 smithi090 ceph-mon[31674]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-02T14:45:19.934 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:45:19 smithi090 ceph-mon[31674]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-02T14:45:19.934 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:45:19 smithi090 ceph-mon[31674]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-02T14:45:19.935 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:45:19 smithi090 ceph-mon[31674]: Adjusting osd_memory_target on smithi092 to 174.1M 2024-04-02T14:45:19.935 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:45:19 smithi090 ceph-mon[31674]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-02T14:45:19.935 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:45:19 smithi090 ceph-mon[31674]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-02T14:45:19.935 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:45:19 smithi090 ceph-mon[31674]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-02T14:45:19.935 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:45:19 smithi090 ceph-mon[31674]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-02T14:45:21.355 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:45:21 smithi092 ceph-mon[36497]: Unable to set osd_memory_target on smithi092 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-02T14:45:21.355 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:45:21 smithi092 ceph-mon[36497]: Adjusting osd_memory_target on smithi177 to 174.1M 2024-04-02T14:45:21.355 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:45:21 smithi092 ceph-mon[36497]: Unable to set osd_memory_target on smithi177 to 182576128: error parsing value: Value '182576128' is below minimum 939524096 2024-04-02T14:45:21.431 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:45:21 smithi177 ceph-mon[36431]: Unable to set osd_memory_target on smithi092 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-02T14:45:21.431 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:45:21 smithi177 ceph-mon[36431]: Adjusting osd_memory_target on smithi177 to 174.1M 2024-04-02T14:45:21.431 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:45:21 smithi177 ceph-mon[36431]: Unable to set osd_memory_target on smithi177 to 182576128: error parsing value: Value '182576128' is below minimum 939524096 2024-04-02T14:45:21.433 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:45:21 smithi090 ceph-mon[31674]: Unable to set osd_memory_target on smithi092 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-02T14:45:21.433 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:45:21 smithi090 ceph-mon[31674]: Adjusting osd_memory_target on smithi177 to 174.1M 2024-04-02T14:45:21.433 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:45:21 smithi090 ceph-mon[31674]: Unable to set osd_memory_target on smithi177 to 182576128: error parsing value: Value '182576128' is below minimum 939524096 2024-04-02T14:45:21.826 INFO:teuthology.orchestra.run.smithi090.stderr:progress: 590.0 s, 1888.4 tps, lat 52.816 ms stddev 63.820 2024-04-02T14:45:22.431 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:45:22 smithi177 ceph-mon[36431]: pgmap v325: 97 pgs: 97 active+clean; 4.3 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 5.9 MiB/s wr, 548 op/s 2024-04-02T14:45:22.432 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:45:22 smithi177 ceph-mon[36431]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' 2024-04-02T14:45:22.432 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:45:22 smithi177 ceph-mon[36431]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' 2024-04-02T14:45:22.432 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:45:22 smithi090 ceph-mon[31674]: pgmap v325: 97 pgs: 97 active+clean; 4.3 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 5.9 MiB/s wr, 548 op/s 2024-04-02T14:45:22.433 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:45:22 smithi090 ceph-mon[31674]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' 2024-04-02T14:45:22.433 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:45:22 smithi090 ceph-mon[31674]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' 2024-04-02T14:45:22.605 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:45:22 smithi092 ceph-mon[36497]: pgmap v325: 97 pgs: 97 active+clean; 4.3 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 5.9 MiB/s wr, 548 op/s 2024-04-02T14:45:22.605 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:45:22 smithi092 ceph-mon[36497]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' 2024-04-02T14:45:22.605 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:45:22 smithi092 ceph-mon[36497]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' 2024-04-02T14:45:23.434 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:45:23 smithi090 ceph-mon[31674]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-02T14:45:23.434 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:45:23 smithi090 ceph-mon[31674]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-02T14:45:23.434 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:45:23 smithi090 ceph-mon[31674]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-02T14:45:23.435 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:45:23 smithi090 ceph-mon[31674]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-02T14:45:23.435 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:45:23 smithi090 ceph-mon[31674]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:45:23.435 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:45:23 smithi090 ceph-mon[31674]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T14:45:23.435 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:45:23 smithi090 ceph-mon[31674]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' 2024-04-02T14:45:23.435 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:45:23 smithi177 ceph-mon[36431]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-02T14:45:23.435 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:45:23 smithi177 ceph-mon[36431]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-02T14:45:23.435 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:45:23 smithi177 ceph-mon[36431]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-02T14:45:23.435 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:45:23 smithi177 ceph-mon[36431]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-02T14:45:23.435 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:45:23 smithi177 ceph-mon[36431]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:45:23.435 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:45:23 smithi177 ceph-mon[36431]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T14:45:23.435 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:45:23 smithi177 ceph-mon[36431]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' 2024-04-02T14:45:23.605 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:45:23 smithi092 ceph-mon[36497]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-02T14:45:23.605 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:45:23 smithi092 ceph-mon[36497]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-02T14:45:23.605 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:45:23 smithi092 ceph-mon[36497]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-02T14:45:23.605 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:45:23 smithi092 ceph-mon[36497]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-02T14:45:23.606 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:45:23 smithi092 ceph-mon[36497]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:45:23.606 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:45:23 smithi092 ceph-mon[36497]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T14:45:23.606 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:45:23 smithi092 ceph-mon[36497]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' 2024-04-02T14:45:24.431 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:45:24 smithi177 ceph-mon[36431]: pgmap v326: 97 pgs: 97 active+clean; 4.3 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 5.9 MiB/s wr, 548 op/s 2024-04-02T14:45:24.431 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:45:24 smithi177 ceph-mon[36431]: Adjusting osd_memory_target on smithi090 to 176.7M 2024-04-02T14:45:24.431 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:45:24 smithi177 ceph-mon[36431]: Unable to set osd_memory_target on smithi090 to 185366630: error parsing value: Value '185366630' is below minimum 939524096 2024-04-02T14:45:24.433 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:45:24 smithi090 ceph-mon[31674]: pgmap v326: 97 pgs: 97 active+clean; 4.3 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 5.9 MiB/s wr, 548 op/s 2024-04-02T14:45:24.433 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:45:24 smithi090 ceph-mon[31674]: Adjusting osd_memory_target on smithi090 to 176.7M 2024-04-02T14:45:24.433 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:45:24 smithi090 ceph-mon[31674]: Unable to set osd_memory_target on smithi090 to 185366630: error parsing value: Value '185366630' is below minimum 939524096 2024-04-02T14:45:24.605 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:45:24 smithi092 ceph-mon[36497]: pgmap v326: 97 pgs: 97 active+clean; 4.3 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 5.9 MiB/s wr, 548 op/s 2024-04-02T14:45:24.605 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:45:24 smithi092 ceph-mon[36497]: Adjusting osd_memory_target on smithi090 to 176.7M 2024-04-02T14:45:24.605 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:45:24 smithi092 ceph-mon[36497]: Unable to set osd_memory_target on smithi090 to 185366630: error parsing value: Value '185366630' is below minimum 939524096 2024-04-02T14:45:25.915 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.918+0000 7f606a800640 1 -- 172.21.15.90:0/2995003777 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6064109020 msgr2=0x7f6064110bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:45:25.915 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.918+0000 7f606a800640 1 --2- 172.21.15.90:0/2995003777 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6064109020 0x7f6064110bf0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f605c009be0 tx=0x7f605c02bef0 comp rx=0 tx=0).stop 2024-04-02T14:45:25.916 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.918+0000 7f606a800640 1 -- 172.21.15.90:0/2995003777 shutdown_connections 2024-04-02T14:45:25.916 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.918+0000 7f606a800640 1 --2- 172.21.15.90:0/2995003777 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6064109020 0x7f6064110bf0 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:25.916 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.918+0000 7f606a800640 1 --2- 172.21.15.90:0/2995003777 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6064104900 0x7f6064108940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:25.916 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.918+0000 7f606a800640 1 --2- 172.21.15.90:0/2995003777 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6064103f50 0x7f6064104330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:25.916 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.918+0000 7f606a800640 1 -- 172.21.15.90:0/2995003777 >> 172.21.15.90:0/2995003777 conn(0x7f60640ff7b0 msgr2=0x7f6064101bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:45:25.916 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.919+0000 7f606a800640 1 -- 172.21.15.90:0/2995003777 shutdown_connections 2024-04-02T14:45:25.916 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.919+0000 7f606a800640 1 -- 172.21.15.90:0/2995003777 wait complete. 2024-04-02T14:45:25.917 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.920+0000 7f606a800640 1 Processor -- start 2024-04-02T14:45:25.917 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.920+0000 7f606a800640 1 -- start start 2024-04-02T14:45:25.917 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.920+0000 7f606a800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6064103f50 0x7f606419a4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:25.918 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.920+0000 7f606a800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6064109020 0x7f606419aa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:25.918 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.920+0000 7f606a800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f606419ed90 0x7f606419f240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:25.918 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.920+0000 7f606a800640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f6064114110 con 0x7f6064103f50 2024-04-02T14:45:25.918 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.920+0000 7f6062a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6064109020 0x7f606419aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:25.918 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.920+0000 7f6063400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6064103f50 0x7f606419a4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:25.918 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.920+0000 7f6062a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6064109020 0x7f606419aa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:39410/0 (socket says 172.21.15.90:39410) 2024-04-02T14:45:25.918 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.920+0000 7f6062a00640 1 -- 172.21.15.90:0/2381060169 learned_addr learned my addr 172.21.15.90:0/2381060169 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:45:25.918 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.920+0000 7f606a800640 1 -- 172.21.15.90:0/2381060169 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f6064113f90 con 0x7f6064109020 2024-04-02T14:45:25.918 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.920+0000 7f606a800640 1 -- 172.21.15.90:0/2381060169 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f6064114290 con 0x7f606419ed90 2024-04-02T14:45:25.918 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.921+0000 7f6063e00640 1 --2- 172.21.15.90:0/2381060169 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f606419ed90 0x7f606419f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:25.918 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.921+0000 7f6062a00640 1 -- 172.21.15.90:0/2381060169 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f606419ed90 msgr2=0x7f606419f240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:45:25.918 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.921+0000 7f6062a00640 1 --2- 172.21.15.90:0/2381060169 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f606419ed90 0x7f606419f240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:25.918 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.921+0000 7f6062a00640 1 -- 172.21.15.90:0/2381060169 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6064103f50 msgr2=0x7f606419a4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:45:25.919 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.921+0000 7f6062a00640 1 --2- 172.21.15.90:0/2381060169 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6064103f50 0x7f606419a4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:25.919 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.921+0000 7f6062a00640 1 -- 172.21.15.90:0/2381060169 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f606419f880 con 0x7f6064109020 2024-04-02T14:45:25.919 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.921+0000 7f6062a00640 1 --2- 172.21.15.90:0/2381060169 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6064109020 0x7f606419aa10 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f605000e9c0 tx=0x7f605000edc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:45:25.919 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.921+0000 7f605be00640 1 -- 172.21.15.90:0/2381060169 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f605000ccc0 con 0x7f6064109020 2024-04-02T14:45:25.919 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.921+0000 7f605be00640 1 -- 172.21.15.90:0/2381060169 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6050005500 con 0x7f6064109020 2024-04-02T14:45:25.919 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.921+0000 7f606a800640 1 -- 172.21.15.90:0/2381060169 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f606419fb70 con 0x7f6064109020 2024-04-02T14:45:25.919 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.922+0000 7f606a800640 1 -- 172.21.15.90:0/2381060169 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60641a73b0 con 0x7f6064109020 2024-04-02T14:45:25.920 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.922+0000 7f605be00640 1 -- 172.21.15.90:0/2381060169 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6050010430 con 0x7f6064109020 2024-04-02T14:45:25.925 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.928+0000 7f606d57e740 1 Processor -- start 2024-04-02T14:45:25.925 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.928+0000 7f605be00640 1 -- 172.21.15.90:0/2381060169 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f6050005020 con 0x7f6064109020 2024-04-02T14:45:25.925 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.928+0000 7f606d57e740 1 -- start start 2024-04-02T14:45:25.925 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.928+0000 7f606d57e740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f8f7fa4cc0 0x55f8f7fa50a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:25.925 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.928+0000 7f606d57e740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f8f7fa5670 0x55f8f7fb6680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:25.925 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.929+0000 7f606d57e740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f8f7fb6bc0 0x55f8f7fb8fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:25.927 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.929+0000 7f6063e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f8f7fa4cc0 0x55f8f7fa50a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:25.927 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.929+0000 7f6063400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f8f7fa5670 0x55f8f7fb6680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:25.927 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.929+0000 7f606d57e740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55f8f7f09d70 con 0x55f8f7fa4cc0 2024-04-02T14:45:25.927 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.929+0000 7f6063400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f8f7fa5670 0x55f8f7fb6680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:39422/0 (socket says 172.21.15.90:39422) 2024-04-02T14:45:25.927 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.929+0000 7f6063400640 1 -- 172.21.15.90:0/499464052 learned_addr learned my addr 172.21.15.90:0/499464052 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:45:25.927 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.929+0000 7f606d57e740 1 -- 172.21.15.90:0/499464052 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55f8f7eb22a0 con 0x55f8f7fa5670 2024-04-02T14:45:25.927 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.929+0000 7f606d57e740 1 -- 172.21.15.90:0/499464052 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55f8f7f05d20 con 0x55f8f7fb6bc0 2024-04-02T14:45:25.927 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.929+0000 7f6062a00640 1 --2- 172.21.15.90:0/499464052 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f8f7fb6bc0 0x55f8f7fb8fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:25.927 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.929+0000 7f6063400640 1 -- 172.21.15.90:0/499464052 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f8f7fb6bc0 msgr2=0x55f8f7fb8fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:45:25.927 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.929+0000 7f6063400640 1 --2- 172.21.15.90:0/499464052 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f8f7fb6bc0 0x55f8f7fb8fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:25.927 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.929+0000 7f6063400640 1 -- 172.21.15.90:0/499464052 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f8f7fa4cc0 msgr2=0x55f8f7fa50a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:45:25.928 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.929+0000 7f6063400640 1 --2- 172.21.15.90:0/499464052 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f8f7fa4cc0 0x55f8f7fa50a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:25.928 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.929+0000 7f6063400640 1 -- 172.21.15.90:0/499464052 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f8f7ec6840 con 0x55f8f7fa5670 2024-04-02T14:45:25.928 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.929+0000 7f6062a00640 1 --2- 172.21.15.90:0/499464052 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f8f7fb6bc0 0x55f8f7fb8fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:45:25.928 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.930+0000 7f6063400640 1 --2- 172.21.15.90:0/499464052 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f8f7fa5670 0x55f8f7fb6680 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f6054009d00 tx=0x7f6054011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a2c6cddc48d1c9c0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:45:25.928 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.930+0000 7f605be00640 1 --2- 172.21.15.90:0/2381060169 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f60480a3e40 0x7f60480a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:25.928 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.930+0000 7f6058c00640 1 -- 172.21.15.90:0/499464052 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f605401e070 con 0x55f8f7fa5670 2024-04-02T14:45:25.928 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.930+0000 7f605be00640 1 -- 172.21.15.90:0/2381060169 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 8550+0+0 (secure 0 0 0) 0x7f60500d0a40 con 0x7f6064109020 2024-04-02T14:45:25.928 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.930+0000 7f6058c00640 1 -- 172.21.15.90:0/499464052 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6054019b10 con 0x55f8f7fa5670 2024-04-02T14:45:25.928 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.930+0000 7f6058c00640 1 -- 172.21.15.90:0/499464052 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f605400eab0 con 0x55f8f7fa5670 2024-04-02T14:45:25.928 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.930+0000 7f6063e00640 1 --2- 172.21.15.90:0/2381060169 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f60480a3e40 0x7f60480a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:25.929 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.932+0000 7f6063e00640 1 --2- 172.21.15.90:0/2381060169 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f60480a3e40 0x7f60480a6300 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f605c033420 tx=0x7f605c004a40 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:45:25.929 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.932+0000 7f606d57e740 1 -- 172.21.15.90:0/499464052 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f8f7fa5670 msgr2=0x55f8f7fb6680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:45:25.929 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.932+0000 7f606d57e740 1 --2- 172.21.15.90:0/499464052 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f8f7fa5670 0x55f8f7fb6680 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f6054009d00 tx=0x7f6054011810 comp rx=0 tx=0).stop 2024-04-02T14:45:25.930 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.933+0000 7f606d57e740 1 -- 172.21.15.90:0/499464052 shutdown_connections 2024-04-02T14:45:25.930 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.933+0000 7f606d57e740 1 --2- 172.21.15.90:0/499464052 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f8f7fb6bc0 0x55f8f7fb8fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:25.930 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.933+0000 7f606d57e740 1 --2- 172.21.15.90:0/499464052 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f8f7fa5670 0x55f8f7fb6680 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:25.930 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.933+0000 7f606d57e740 1 --2- 172.21.15.90:0/499464052 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f8f7fa4cc0 0x55f8f7fa50a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:25.930 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.933+0000 7f606d57e740 1 -- 172.21.15.90:0/499464052 >> 172.21.15.90:0/499464052 conn(0x55f8f7f9f5e0 msgr2=0x55f8f7f9f9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:45:25.930 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.933+0000 7f606d57e740 1 -- 172.21.15.90:0/499464052 shutdown_connections 2024-04-02T14:45:25.930 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.933+0000 7f606d57e740 1 -- 172.21.15.90:0/499464052 wait complete. 2024-04-02T14:45:25.931 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.934+0000 7f606d57e740 1 -- start start 2024-04-02T14:45:25.931 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.934+0000 7f606d57e740 1 Processor -- start 2024-04-02T14:45:25.931 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.935+0000 7f606d57e740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f8f7fa4cc0 0x55f8f7fb3be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:25.932 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.935+0000 7f606d57e740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f8f7fa5670 0x55f8f7fb4120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:25.932 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.935+0000 7f606d57e740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f8f7fb6bc0 0x55f8f7fb4660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:25.932 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.935+0000 7f606d57e740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55f8f7fb9980 con 0x55f8f7fa4cc0 2024-04-02T14:45:25.932 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.935+0000 7f606d57e740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55f8f7ec5660 con 0x55f8f7fa5670 2024-04-02T14:45:25.932 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.935+0000 7f606d57e740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55f8f7fb9b00 con 0x55f8f7fb6bc0 2024-04-02T14:45:25.933 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.935+0000 7f6062a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f8f7fa5670 0x55f8f7fb4120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:25.933 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.935+0000 7f606d57e740 10 client.0 _ll_register_callbacks cb 0x55f8f7ec5050 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:45:25.933 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.936+0000 7f6062a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f8f7fa5670 0x55f8f7fb4120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:39430/0 (socket says 172.21.15.90:39430) 2024-04-02T14:45:25.933 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.936+0000 7f6062a00640 1 -- 172.21.15.90:0/3060349290 learned_addr learned my addr 172.21.15.90:0/3060349290 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:45:25.933 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.936+0000 7f6063e00640 1 --2- 172.21.15.90:0/3060349290 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f8f7fb6bc0 0x55f8f7fb4660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:25.933 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.936+0000 7f6063400640 1 --2- 172.21.15.90:0/3060349290 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f8f7fa4cc0 0x55f8f7fb3be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:25.934 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.936+0000 7f6062a00640 1 -- 172.21.15.90:0/3060349290 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f8f7fb6bc0 msgr2=0x55f8f7fb4660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:45:25.934 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.936+0000 7f6062a00640 1 --2- 172.21.15.90:0/3060349290 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f8f7fb6bc0 0x55f8f7fb4660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:25.934 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.936+0000 7f6062a00640 1 -- 172.21.15.90:0/3060349290 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f8f7fa4cc0 msgr2=0x55f8f7fb3be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:45:25.934 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.936+0000 7f6062a00640 1 --2- 172.21.15.90:0/3060349290 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f8f7fa4cc0 0x55f8f7fb3be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:25.934 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.936+0000 7f6062a00640 1 -- 172.21.15.90:0/3060349290 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f8f7ed4e80 con 0x55f8f7fa5670 2024-04-02T14:45:25.934 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.936+0000 7f6063e00640 1 --2- 172.21.15.90:0/3060349290 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f8f7fb6bc0 0x55f8f7fb4660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:45:25.934 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.936+0000 7f6062a00640 1 --2- 172.21.15.90:0/3060349290 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f8f7fa5670 0x55f8f7fb4120 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f605009c950 tx=0x7f6050098d90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b70101756a0c555f server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:45:25.934 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.936+0000 7f604d600640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:45:25.934 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.937+0000 7f604d600640 1 -- 172.21.15.90:0/3060349290 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60500d7070 con 0x55f8f7fa5670 2024-04-02T14:45:25.934 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.937+0000 7f604d600640 1 -- 172.21.15.90:0/3060349290 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8550+0+0 (secure 0 0 0) 0x7f6050016330 con 0x55f8f7fa5670 2024-04-02T14:45:25.934 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.937+0000 7f606d57e740 1 -- 172.21.15.90:0/3060349290 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6054019b10 con 0x55f8f7fa5670 2024-04-02T14:45:25.935 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.937+0000 7f604d600640 1 -- 172.21.15.90:0/3060349290 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f6050016530 con 0x55f8f7fa5670 2024-04-02T14:45:25.935 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.937+0000 7f606d57e740 10 client.28685 fetch_fsmap learned FSMap version 27 2024-04-02T14:45:25.935 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.937+0000 7f606d57e740 1 -- 172.21.15.90:0/3060349290 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55f8f7c1e210 con 0x55f8f7fa5670 2024-04-02T14:45:25.935 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.937+0000 7f604d600640 1 -- 172.21.15.90:0/3060349290 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6054019b10 con 0x55f8f7fa5670 2024-04-02T14:45:25.935 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.938+0000 7f604d600640 1 -- 172.21.15.90:0/3060349290 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f60500a51a0 con 0x55f8f7fa5670 2024-04-02T14:45:25.935 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.938+0000 7f606d57e740 10 client.28685 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:45:25.935 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.938+0000 7f606d57e740 10 client.28685 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-02T14:45:25.935 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.938+0000 7f606d57e740 20 client.28685 populate_metadata read hostname 'smithi090' 2024-04-02T14:45:25.936 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.938+0000 7f606d57e740 1 --2- 172.21.15.90:0/3060349290 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55f8f7fc7010 0x55f8f7fc9400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:25.936 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.938+0000 7f606d57e740 4 client.28685 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:45:25.936 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.938+0000 7f606d57e740 1 -- 172.21.15.90:0/3060349290 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f8f7eb9cd0 con 0x55f8f7fc7010 2024-04-02T14:45:25.936 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.939+0000 7f6063400640 1 --2- 172.21.15.90:0/3060349290 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55f8f7fc7010 0x55f8f7fc9400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:25.936 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.940+0000 7f604d600640 10 client.28685 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:45:25.936 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.940+0000 7f6063400640 1 --2- 172.21.15.90:0/3060349290 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55f8f7fc7010 0x55f8f7fc9400 crc :-1 s=READY pgs=276 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=84b88293c8298cd1 server_cookie=b8354a1b84564897 in_seq=0 out_seq=0 2024-04-02T14:45:25.937 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.941+0000 7f604d600640 1 -- 172.21.15.90:0/3060349290 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f8f7eb9cd0 con 0x55f8f7fc7010 2024-04-02T14:45:25.937 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.941+0000 7f604d600640 10 client.28685 handle_command_reply: tid=0 2024-04-02T14:45:25.937 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.941+0000 7f606d57e740 1 client.28685 shutdown 2024-04-02T14:45:25.938 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.941+0000 7f606d57e740 1 -- 172.21.15.90:0/3060349290 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f8f7fa5670 msgr2=0x55f8f7fb4120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:45:25.938 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.941+0000 7f606d57e740 1 --2- 172.21.15.90:0/3060349290 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f8f7fa5670 0x55f8f7fb4120 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f605009c950 tx=0x7f6050098d90 comp rx=0 tx=0).stop 2024-04-02T14:45:25.938 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.941+0000 7f606d57e740 1 -- 172.21.15.90:0/3060349290 shutdown_connections 2024-04-02T14:45:25.938 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.941+0000 7f606d57e740 1 --2- 172.21.15.90:0/3060349290 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55f8f7fc7010 0x55f8f7fc9400 crc :-1 s=READY pgs=276 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:25.938 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.941+0000 7f606d57e740 1 --2- 172.21.15.90:0/3060349290 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f8f7fb6bc0 0x55f8f7fb4660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:25.938 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.941+0000 7f606d57e740 1 --2- 172.21.15.90:0/3060349290 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f8f7fa5670 0x55f8f7fb4120 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:25.938 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.941+0000 7f606d57e740 1 --2- 172.21.15.90:0/3060349290 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f8f7fa4cc0 0x55f8f7fb3be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:25.939 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.941+0000 7f606d57e740 1 -- 172.21.15.90:0/3060349290 >> 172.21.15.90:0/3060349290 conn(0x55f8f7f9f5e0 msgr2=0x55f8f7fa1330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:45:25.939 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.941+0000 7f604d600640 0 client.28685 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:45:25.939 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.942+0000 7f606d57e740 1 -- 172.21.15.90:0/3060349290 shutdown_connections 2024-04-02T14:45:25.939 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.942+0000 7f606d57e740 1 -- 172.21.15.90:0/3060349290 wait complete. 2024-04-02T14:45:25.939 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.943+0000 7f606d57e740 20 client.28685 trim_cache size 0 max 16384 2024-04-02T14:45:25.963 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.966+0000 7f606d57e740 1 Processor -- start 2024-04-02T14:45:25.963 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.966+0000 7f606d57e740 1 -- start start 2024-04-02T14:45:25.963 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.966+0000 7f606d57e740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f8f7fa4cc0 0x55f8f7fb6310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:25.964 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.966+0000 7f606d57e740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f8f7fa5670 0x55f8f7fc28b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:25.964 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.966+0000 7f606d57e740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f8f7fb6bc0 0x55f8f7fcecc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:25.964 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.966+0000 7f606d57e740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f604400e780 con 0x55f8f7fa4cc0 2024-04-02T14:45:25.964 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.967+0000 7f606d57e740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f60440102c0 con 0x55f8f7fa5670 2024-04-02T14:45:25.964 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.967+0000 7f606d57e740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f604400ee50 con 0x55f8f7fb6bc0 2024-04-02T14:45:25.964 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.967+0000 7f6062a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f8f7fa5670 0x55f8f7fc28b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:25.965 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.967+0000 7f6062a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f8f7fa5670 0x55f8f7fc28b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:39444/0 (socket says 172.21.15.90:39444) 2024-04-02T14:45:25.965 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.967+0000 7f6062a00640 1 -- 172.21.15.90:0/4269482331 learned_addr learned my addr 172.21.15.90:0/4269482331 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:45:25.965 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.967+0000 7f6063e00640 1 --2- 172.21.15.90:0/4269482331 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f8f7fb6bc0 0x55f8f7fcecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:25.965 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.968+0000 7f6062a00640 1 -- 172.21.15.90:0/4269482331 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f8f7fb6bc0 msgr2=0x55f8f7fcecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:45:25.965 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.968+0000 7f6063400640 1 --2- 172.21.15.90:0/4269482331 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f8f7fa4cc0 0x55f8f7fb6310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:25.965 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.968+0000 7f6062a00640 1 --2- 172.21.15.90:0/4269482331 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f8f7fb6bc0 0x55f8f7fcecc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:25.966 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.968+0000 7f6062a00640 1 -- 172.21.15.90:0/4269482331 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f8f7fa4cc0 msgr2=0x55f8f7fb6310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:45:25.966 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.968+0000 7f6062a00640 1 --2- 172.21.15.90:0/4269482331 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f8f7fa4cc0 0x55f8f7fb6310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:25.966 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.968+0000 7f6062a00640 1 -- 172.21.15.90:0/4269482331 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f8f7fc9770 con 0x55f8f7fa5670 2024-04-02T14:45:25.966 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.968+0000 7f6063e00640 1 --2- 172.21.15.90:0/4269482331 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f8f7fb6bc0 0x55f8f7fcecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:45:25.966 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.968+0000 7f6063400640 1 --2- 172.21.15.90:0/4269482331 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f8f7fa4cc0 0x55f8f7fb6310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:45:25.966 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.969+0000 7f6062a00640 1 --2- 172.21.15.90:0/4269482331 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f8f7fa5670 0x55f8f7fc28b0 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f605009c6c0 tx=0x7f6050098c60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4ef5a3140575b75f server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:45:25.966 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.969+0000 7f604fe00640 1 -- 172.21.15.90:0/4269482331 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f605009c500 con 0x55f8f7fa5670 2024-04-02T14:45:25.966 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.969+0000 7f604fe00640 1 -- 172.21.15.90:0/4269482331 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f605009d0c0 con 0x55f8f7fa5670 2024-04-02T14:45:25.966 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.969+0000 7f604fe00640 1 -- 172.21.15.90:0/4269482331 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6050016330 con 0x55f8f7fa5670 2024-04-02T14:45:25.967 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.970+0000 7f606d57e740 1 -- 172.21.15.90:0/4269482331 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f8f7fa5670 msgr2=0x55f8f7fc28b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:45:25.967 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.970+0000 7f606d57e740 1 --2- 172.21.15.90:0/4269482331 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f8f7fa5670 0x55f8f7fc28b0 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f605009c6c0 tx=0x7f6050098c60 comp rx=0 tx=0).stop 2024-04-02T14:45:25.967 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.970+0000 7f606d57e740 1 -- 172.21.15.90:0/4269482331 shutdown_connections 2024-04-02T14:45:25.967 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.970+0000 7f606d57e740 1 --2- 172.21.15.90:0/4269482331 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f8f7fb6bc0 0x55f8f7fcecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:25.967 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.970+0000 7f606d57e740 1 --2- 172.21.15.90:0/4269482331 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f8f7fa5670 0x55f8f7fc28b0 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:25.967 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.970+0000 7f606d57e740 1 --2- 172.21.15.90:0/4269482331 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f8f7fa4cc0 0x55f8f7fb6310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:25.967 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.970+0000 7f606d57e740 1 -- 172.21.15.90:0/4269482331 >> 172.21.15.90:0/4269482331 conn(0x55f8f7f9f5e0 msgr2=0x55f8f7f9f9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:45:25.967 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.970+0000 7f606d57e740 1 -- 172.21.15.90:0/4269482331 shutdown_connections 2024-04-02T14:45:25.967 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.971+0000 7f606d57e740 1 -- 172.21.15.90:0/4269482331 wait complete. 2024-04-02T14:45:25.968 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.971+0000 7f606d57e740 1 -- start start 2024-04-02T14:45:25.969 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.973+0000 7f606d57e740 1 Processor -- start 2024-04-02T14:45:25.970 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.973+0000 7f606d57e740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f8f7fa4cc0 0x55f8f7fb0a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:25.970 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.973+0000 7f606d57e740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f8f7fa5670 0x55f8f7fb0fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:25.970 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.973+0000 7f606d57e740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f8f7fb6bc0 0x55f8f7fb14f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:25.970 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.973+0000 7f606d57e740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f604400fbf0 con 0x55f8f7fa5670 2024-04-02T14:45:25.971 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.973+0000 7f606d57e740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f604400f520 con 0x55f8f7fa4cc0 2024-04-02T14:45:25.971 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.973+0000 7f606d57e740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f604400c780 con 0x55f8f7fb6bc0 2024-04-02T14:45:25.971 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.973+0000 7f606d57e740 10 client.0 _ll_register_callbacks cb 0x7f60500bd0c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:45:25.971 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.974+0000 7f6063400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f8f7fa4cc0 0x55f8f7fb0a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:25.971 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.974+0000 7f6063400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f8f7fa4cc0 0x55f8f7fb0a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:39446/0 (socket says 172.21.15.90:39446) 2024-04-02T14:45:25.971 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.974+0000 7f6063400640 1 -- 172.21.15.90:0/3879427150 learned_addr learned my addr 172.21.15.90:0/3879427150 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:45:25.971 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.974+0000 7f6063e00640 1 --2- 172.21.15.90:0/3879427150 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f8f7fb6bc0 0x55f8f7fb14f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:25.971 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.974+0000 7f6062a00640 1 --2- 172.21.15.90:0/3879427150 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f8f7fa5670 0x55f8f7fb0fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:25.971 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.974+0000 7f6063400640 1 -- 172.21.15.90:0/3879427150 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f8f7fb6bc0 msgr2=0x55f8f7fb14f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:45:25.971 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.974+0000 7f6063400640 1 --2- 172.21.15.90:0/3879427150 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f8f7fb6bc0 0x55f8f7fb14f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:25.971 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.974+0000 7f6063400640 1 -- 172.21.15.90:0/3879427150 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f8f7fa5670 msgr2=0x55f8f7fb0fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:45:25.971 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.974+0000 7f6063400640 1 --2- 172.21.15.90:0/3879427150 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f8f7fa5670 0x55f8f7fb0fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:25.971 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.974+0000 7f6063400640 1 -- 172.21.15.90:0/3879427150 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f8f7cea620 con 0x55f8f7fa4cc0 2024-04-02T14:45:25.972 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.975+0000 7f6063400640 1 --2- 172.21.15.90:0/3879427150 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f8f7fa4cc0 0x55f8f7fb0a70 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f605401c040 tx=0x7f605400e3b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fff4140162ce659d server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:45:25.972 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.975+0000 7f604d600640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:45:25.972 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.975+0000 7f604d600640 1 -- 172.21.15.90:0/3879427150 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f605401e040 con 0x55f8f7fa4cc0 2024-04-02T14:45:25.972 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.975+0000 7f604d600640 1 -- 172.21.15.90:0/3879427150 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8550+0+0 (secure 0 0 0) 0x7f60540050b0 con 0x55f8f7fa4cc0 2024-04-02T14:45:25.972 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.976+0000 7f606d57e740 1 -- 172.21.15.90:0/3879427150 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f605009d0c0 con 0x55f8f7fa4cc0 2024-04-02T14:45:25.973 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.976+0000 7f604d600640 1 -- 172.21.15.90:0/3879427150 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f6054009260 con 0x55f8f7fa4cc0 2024-04-02T14:45:25.973 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.976+0000 7f606d57e740 10 client.28691 fetch_fsmap learned FSMap version 27 2024-04-02T14:45:25.973 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.976+0000 7f606d57e740 1 -- 172.21.15.90:0/3879427150 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55f8f802d7a0 con 0x55f8f7fa4cc0 2024-04-02T14:45:25.974 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.977+0000 7f604d600640 1 -- 172.21.15.90:0/3879427150 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f605009d0c0 con 0x55f8f7fa4cc0 2024-04-02T14:45:25.974 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.977+0000 7f604d600640 1 -- 172.21.15.90:0/3879427150 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f6054024040 con 0x55f8f7fa4cc0 2024-04-02T14:45:25.975 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.977+0000 7f606d57e740 10 client.28691 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:45:25.975 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.977+0000 7f606d57e740 10 client.28691 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-02T14:45:25.975 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.977+0000 7f606d57e740 20 client.28691 populate_metadata read hostname 'smithi090' 2024-04-02T14:45:25.975 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.977+0000 7f606d57e740 1 --2- 172.21.15.90:0/3879427150 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55f8f802dc90 0x55f8f8030080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:25.975 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.977+0000 7f606d57e740 4 client.28691 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-02T14:45:25.975 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.977+0000 7f606d57e740 1 -- 172.21.15.90:0/3879427150 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55f8f7f172b0 con 0x55f8f802dc90 2024-04-02T14:45:25.975 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.978+0000 7f6062a00640 1 --2- 172.21.15.90:0/3879427150 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55f8f802dc90 0x55f8f8030080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:25.975 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.978+0000 7f6062a00640 1 --2- 172.21.15.90:0/3879427150 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55f8f802dc90 0x55f8f8030080 crc :-1 s=READY pgs=277 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e998b504aac9628c server_cookie=aadcb3b853c7e4e in_seq=0 out_seq=0 2024-04-02T14:45:25.975 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:25.978+0000 7f604d600640 10 client.28691 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:45:26.045 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:26.048+0000 7f604d600640 1 -- 172.21.15.90:0/3879427150 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55f8f7f172b0 con 0x55f8f802dc90 2024-04-02T14:45:26.046 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:26.048+0000 7f604d600640 10 client.28691 handle_command_reply: tid=0 2024-04-02T14:45:26.046 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:26.049+0000 7f606d57e740 1 client.28691 shutdown 2024-04-02T14:45:26.046 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:26.049+0000 7f606d57e740 1 -- 172.21.15.90:0/3879427150 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f8f7fa4cc0 msgr2=0x55f8f7fb0a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:45:26.046 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:26.049+0000 7f606d57e740 1 --2- 172.21.15.90:0/3879427150 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f8f7fa4cc0 0x55f8f7fb0a70 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f605401c040 tx=0x7f605400e3b0 comp rx=0 tx=0).stop 2024-04-02T14:45:26.046 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:26.049+0000 7f606d57e740 1 -- 172.21.15.90:0/3879427150 shutdown_connections 2024-04-02T14:45:26.046 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:26.049+0000 7f606d57e740 1 --2- 172.21.15.90:0/3879427150 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55f8f802dc90 0x55f8f8030080 crc :-1 s=READY pgs=277 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:26.046 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:26.049+0000 7f606d57e740 1 --2- 172.21.15.90:0/3879427150 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f8f7fb6bc0 0x55f8f7fb14f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:26.046 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:26.049+0000 7f606d57e740 1 --2- 172.21.15.90:0/3879427150 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f8f7fa5670 0x55f8f7fb0fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:26.046 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:26.049+0000 7f606d57e740 1 --2- 172.21.15.90:0/3879427150 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f8f7fa4cc0 0x55f8f7fb0a70 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:26.046 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:26.049+0000 7f606d57e740 1 -- 172.21.15.90:0/3879427150 >> 172.21.15.90:0/3879427150 conn(0x55f8f7f9f5e0 msgr2=0x55f8f7f9ddb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:45:26.046 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:26.049+0000 7f604d600640 0 client.28691 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:45:26.047 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:26.049+0000 7f606d57e740 1 -- 172.21.15.90:0/3879427150 shutdown_connections 2024-04-02T14:45:26.047 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:26.049+0000 7f606d57e740 1 -- 172.21.15.90:0/3879427150 wait complete. 2024-04-02T14:45:26.047 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:26.050+0000 7f606d57e740 20 client.28691 trim_cache size 0 max 16384 2024-04-02T14:45:26.047 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:{ 2024-04-02T14:45:26.047 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "message": "", 2024-04-02T14:45:26.047 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "return_code": 0 2024-04-02T14:45:26.048 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:} 2024-04-02T14:45:26.048 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:26.051+0000 7f606a800640 1 -- 172.21.15.90:0/2381060169 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f60480a3e40 msgr2=0x7f60480a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:45:26.048 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:26.051+0000 7f606a800640 1 --2- 172.21.15.90:0/2381060169 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f60480a3e40 0x7f60480a6300 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f605c033420 tx=0x7f605c004a40 comp rx=0 tx=0).stop 2024-04-02T14:45:26.048 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:26.051+0000 7f606a800640 1 -- 172.21.15.90:0/2381060169 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6064109020 msgr2=0x7f606419aa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:45:26.048 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:26.051+0000 7f606a800640 1 --2- 172.21.15.90:0/2381060169 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6064109020 0x7f606419aa10 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f605000e9c0 tx=0x7f605000edc0 comp rx=0 tx=0).stop 2024-04-02T14:45:26.048 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:26.052+0000 7f606a800640 1 -- 172.21.15.90:0/2381060169 shutdown_connections 2024-04-02T14:45:26.048 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:26.052+0000 7f606a800640 1 --2- 172.21.15.90:0/2381060169 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f60480a3e40 0x7f60480a6300 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:26.048 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:26.052+0000 7f606a800640 1 --2- 172.21.15.90:0/2381060169 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f606419ed90 0x7f606419f240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:26.049 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:26.052+0000 7f606a800640 1 --2- 172.21.15.90:0/2381060169 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6064109020 0x7f606419aa10 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:26.049 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:26.052+0000 7f606a800640 1 --2- 172.21.15.90:0/2381060169 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6064103f50 0x7f606419a4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:26.049 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:26.052+0000 7f606a800640 1 -- 172.21.15.90:0/2381060169 >> 172.21.15.90:0/2381060169 conn(0x7f60640ff7b0 msgr2=0x7f6064100160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:45:26.049 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:26.052+0000 7f606a800640 1 -- 172.21.15.90:0/2381060169 shutdown_connections 2024-04-02T14:45:26.049 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:26.052+0000 7f606a800640 1 -- 172.21.15.90:0/2381060169 wait complete. 2024-04-02T14:45:26.431 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:45:26 smithi177 ceph-mon[36431]: pgmap v327: 97 pgs: 97 active+clean; 4.4 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 6.8 MiB/s wr, 676 op/s 2024-04-02T14:45:26.433 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:45:26 smithi090 ceph-mon[31674]: pgmap v327: 97 pgs: 97 active+clean; 4.4 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 6.8 MiB/s wr, 676 op/s 2024-04-02T14:45:26.605 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:45:26 smithi092 ceph-mon[36497]: pgmap v327: 97 pgs: 97 active+clean; 4.4 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 6.8 MiB/s wr, 676 op/s 2024-04-02T14:45:26.827 INFO:teuthology.orchestra.run.smithi090.stderr:progress: 595.0 s, 2280.0 tps, lat 43.921 ms stddev 56.795 2024-04-02T14:45:28.431 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:45:28 smithi177 ceph-mon[36431]: pgmap v328: 97 pgs: 97 active+clean; 4.4 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 6.0 MiB/s wr, 589 op/s 2024-04-02T14:45:28.432 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:45:28 smithi090 ceph-mon[31674]: pgmap v328: 97 pgs: 97 active+clean; 4.4 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 6.0 MiB/s wr, 589 op/s 2024-04-02T14:45:28.605 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:45:28 smithi092 ceph-mon[36497]: pgmap v328: 97 pgs: 97 active+clean; 4.4 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 6.0 MiB/s wr, 589 op/s 2024-04-02T14:45:29.466 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-02T14:45:29.467 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-02T14:45:29.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.759+0000 7fc4b4c00640 1 -- 172.21.15.90:0/3523092105 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc4b0101330 msgr2=0x7fc4b010d840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:45:29.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.759+0000 7fc4b4c00640 1 --2- 172.21.15.90:0/3523092105 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc4b0101330 0x7fc4b010d840 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7fc4a8009be0 tx=0x7fc4a802bef0 comp rx=0 tx=0).stop 2024-04-02T14:45:29.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.760+0000 7fc4b4c00640 1 -- 172.21.15.90:0/3523092105 shutdown_connections 2024-04-02T14:45:29.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.760+0000 7fc4b4c00640 1 --2- 172.21.15.90:0/3523092105 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc4b0101330 0x7fc4b010d840 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:29.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.760+0000 7fc4b4c00640 1 --2- 172.21.15.90:0/3523092105 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc4b0100970 0x7fc4b0100df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:29.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.760+0000 7fc4b4c00640 1 --2- 172.21.15.90:0/3523092105 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc4b0106980 0x7fc4b0106d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:29.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.760+0000 7fc4b4c00640 1 -- 172.21.15.90:0/3523092105 >> 172.21.15.90:0/3523092105 conn(0x7fc4b0077b00 msgr2=0x7fc4b0077f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:45:29.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.760+0000 7fc4b4c00640 1 -- 172.21.15.90:0/3523092105 shutdown_connections 2024-04-02T14:45:29.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.760+0000 7fc4b4c00640 1 -- 172.21.15.90:0/3523092105 wait complete. 2024-04-02T14:45:29.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.761+0000 7fc4b4c00640 1 Processor -- start 2024-04-02T14:45:29.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.761+0000 7fc4b4c00640 1 -- start start 2024-04-02T14:45:29.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.762+0000 7fc4b4c00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc4b0100970 0x7fc4b010dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:29.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.762+0000 7fc4b4c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc4b0101330 0x7fc4b010e510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:29.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.762+0000 7fc4b4c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc4b0106980 0x7fc4b010ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:29.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.762+0000 7fc4b4c00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fc4b0111ec0 con 0x7fc4b0101330 2024-04-02T14:45:29.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.762+0000 7fc4b4c00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fc4b0111d40 con 0x7fc4b0100970 2024-04-02T14:45:29.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.762+0000 7fc4b4c00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fc4b0112040 con 0x7fc4b0106980 2024-04-02T14:45:29.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.762+0000 7fc4ae800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc4b0106980 0x7fc4b010ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:29.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.762+0000 7fc4ae800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc4b0106980 0x7fc4b010ea50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:55478/0 (socket says 172.21.15.90:55478) 2024-04-02T14:45:29.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.762+0000 7fc4ae800640 1 -- 172.21.15.90:0/3580577046 learned_addr learned my addr 172.21.15.90:0/3580577046 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:45:29.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.763+0000 7fc4ade00640 1 --2- 172.21.15.90:0/3580577046 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc4b0100970 0x7fc4b010dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:29.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.763+0000 7fc4ae800640 1 -- 172.21.15.90:0/3580577046 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc4b0100970 msgr2=0x7fc4b010dfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:45:29.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.763+0000 7fc4ae800640 1 --2- 172.21.15.90:0/3580577046 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc4b0100970 0x7fc4b010dfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:29.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.763+0000 7fc4ae800640 1 -- 172.21.15.90:0/3580577046 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc4b0101330 msgr2=0x7fc4b010e510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:45:29.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.763+0000 7fc4ae800640 1 --2- 172.21.15.90:0/3580577046 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc4b0101330 0x7fc4b010e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:29.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.763+0000 7fc4ae800640 1 -- 172.21.15.90:0/3580577046 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4b019c4a0 con 0x7fc4b0106980 2024-04-02T14:45:29.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.764+0000 7fc4ae800640 1 --2- 172.21.15.90:0/3580577046 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc4b0106980 0x7fc4b010ea50 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7fc4a8004af0 tx=0x7fc4a802cbc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:45:29.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.764+0000 7fc4a6a00640 1 -- 172.21.15.90:0/3580577046 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4a8043020 con 0x7fc4b0106980 2024-04-02T14:45:29.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.764+0000 7fc4a6a00640 1 -- 172.21.15.90:0/3580577046 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc4a8003e50 con 0x7fc4b0106980 2024-04-02T14:45:29.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.764+0000 7fc4a6a00640 1 -- 172.21.15.90:0/3580577046 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4a803e6b0 con 0x7fc4b0106980 2024-04-02T14:45:29.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.765+0000 7fc4b4c00640 1 -- 172.21.15.90:0/3580577046 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc4b019c730 con 0x7fc4b0106980 2024-04-02T14:45:29.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.765+0000 7fc4b4c00640 1 -- 172.21.15.90:0/3580577046 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4b019cc30 con 0x7fc4b0106980 2024-04-02T14:45:29.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.768+0000 7fc4b787e740 1 Processor -- start 2024-04-02T14:45:29.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.771+0000 7fc4a6a00640 1 -- 172.21.15.90:0/3580577046 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fc4a8046020 con 0x7fc4b0106980 2024-04-02T14:45:29.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.772+0000 7fc4b787e740 1 -- start start 2024-04-02T14:45:29.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.773+0000 7fc4b787e740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5599ccdf67b0 0x5599ccdf6b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:29.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.773+0000 7fc4b787e740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5599ccdf70d0 0x5599ccdff770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:29.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.773+0000 7fc4b787e740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5599ccdffcb0 0x5599cce020a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:29.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.773+0000 7fc4b787e740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5599ccd5a770 con 0x5599ccdf67b0 2024-04-02T14:45:29.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.773+0000 7fc4b787e740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x5599ccd02700 con 0x5599ccdffcb0 2024-04-02T14:45:29.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.773+0000 7fc4b787e740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5599ccd56720 con 0x5599ccdf70d0 2024-04-02T14:45:29.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.773+0000 7fc4ade00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5599ccdf70d0 0x5599ccdff770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:29.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.773+0000 7fc4ade00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5599ccdf70d0 0x5599ccdff770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:55488/0 (socket says 172.21.15.90:55488) 2024-04-02T14:45:29.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.773+0000 7fc4ade00640 1 -- 172.21.15.90:0/909006905 learned_addr learned my addr 172.21.15.90:0/909006905 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:45:29.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.773+0000 7fc4ade00640 1 -- 172.21.15.90:0/909006905 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5599ccdffcb0 msgr2=0x5599cce020a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-02T14:45:29.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.773+0000 7fc4ade00640 1 --2- 172.21.15.90:0/909006905 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5599ccdffcb0 0x5599cce020a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:29.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.773+0000 7fc4ade00640 1 -- 172.21.15.90:0/909006905 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5599ccdf67b0 msgr2=0x5599ccdf6b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:45:29.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.774+0000 7fc4a6a00640 1 --2- 172.21.15.90:0/3580577046 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fc4940a3da0 0x7fc4940a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:29.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.774+0000 7fc4ad400640 1 --2- 172.21.15.90:0/909006905 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5599ccdf67b0 0x5599ccdf6b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:29.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.774+0000 7fc4a6a00640 1 -- 172.21.15.90:0/3580577046 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 8550+0+0 (secure 0 0 0) 0x7fc4a80f2e20 con 0x7fc4b0106980 2024-04-02T14:45:29.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.774+0000 7fc4ade00640 1 --2- 172.21.15.90:0/909006905 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5599ccdf67b0 0x5599ccdf6b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:29.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.774+0000 7fc4ade00640 1 -- 172.21.15.90:0/909006905 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5599ccb387d0 con 0x5599ccdf70d0 2024-04-02T14:45:29.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.775+0000 7fc4ad400640 1 --2- 172.21.15.90:0/909006905 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5599ccdf67b0 0x5599ccdf6b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:45:29.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.775+0000 7fc4ade00640 1 --2- 172.21.15.90:0/909006905 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5599ccdf70d0 0x5599ccdff770 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7fc4a0009bd0 tx=0x7fc4a0010740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1f90fcf6d15be66e server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:45:29.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.775+0000 7fc49b000640 1 -- 172.21.15.90:0/909006905 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4a0004020 con 0x5599ccdf70d0 2024-04-02T14:45:29.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.775+0000 7fc49b000640 1 -- 172.21.15.90:0/909006905 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc4a00041c0 con 0x5599ccdf70d0 2024-04-02T14:45:29.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.775+0000 7fc49b000640 1 -- 172.21.15.90:0/909006905 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4a00126b0 con 0x5599ccdf70d0 2024-04-02T14:45:29.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.776+0000 7fc4ad400640 1 --2- 172.21.15.90:0/3580577046 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fc4940a3da0 0x7fc4940a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:29.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.777+0000 7fc4b787e740 1 -- 172.21.15.90:0/909006905 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5599ccdf70d0 msgr2=0x5599ccdff770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:45:29.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.777+0000 7fc4b787e740 1 --2- 172.21.15.90:0/909006905 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5599ccdf70d0 0x5599ccdff770 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7fc4a0009bd0 tx=0x7fc4a0010740 comp rx=0 tx=0).stop 2024-04-02T14:45:29.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.777+0000 7fc4b787e740 1 -- 172.21.15.90:0/909006905 shutdown_connections 2024-04-02T14:45:29.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.777+0000 7fc4b787e740 1 --2- 172.21.15.90:0/909006905 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5599ccdffcb0 0x5599cce020a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:29.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.777+0000 7fc4b787e740 1 --2- 172.21.15.90:0/909006905 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5599ccdf70d0 0x5599ccdff770 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:29.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.777+0000 7fc4b787e740 1 --2- 172.21.15.90:0/909006905 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5599ccdf67b0 0x5599ccdf6b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:29.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.777+0000 7fc4b787e740 1 -- 172.21.15.90:0/909006905 >> 172.21.15.90:0/909006905 conn(0x5599ccdf0ee0 msgr2=0x5599ccdf12c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:45:29.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.777+0000 7fc4b787e740 1 -- 172.21.15.90:0/909006905 shutdown_connections 2024-04-02T14:45:29.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.778+0000 7fc4b787e740 1 -- 172.21.15.90:0/909006905 wait complete. 2024-04-02T14:45:29.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.778+0000 7fc4ad400640 1 --2- 172.21.15.90:0/3580577046 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fc4940a3da0 0x7fc4940a6260 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x5599ccdf2070 tx=0x7fc49c005d50 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:45:29.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.778+0000 7fc4b787e740 1 -- start start 2024-04-02T14:45:29.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.778+0000 7fc4b787e740 1 Processor -- start 2024-04-02T14:45:29.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.779+0000 7fc4b787e740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5599ccdf67b0 0x5599ccdfcc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:29.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.779+0000 7fc4b787e740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5599ccdf70d0 0x5599ccdfd1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:29.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.779+0000 7fc4b787e740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5599ccdffcb0 0x5599ccdfd6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:29.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.779+0000 7fc4b787e740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5599cce02aa0 con 0x5599ccdf67b0 2024-04-02T14:45:29.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.779+0000 7fc4b787e740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x5599ccd15ac0 con 0x5599ccdffcb0 2024-04-02T14:45:29.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.779+0000 7fc4b787e740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5599cce02c20 con 0x5599ccdf70d0 2024-04-02T14:45:29.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.779+0000 7fc4b787e740 10 client.0 _ll_register_callbacks cb 0x5599ccd18180 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:45:29.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.779+0000 7fc4ad400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5599ccdf70d0 0x5599ccdfd1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:29.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.779+0000 7fc4ade00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5599ccdf67b0 0x5599ccdfcc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:29.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.779+0000 7fc4ad400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5599ccdf70d0 0x5599ccdfd1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:55490/0 (socket says 172.21.15.90:55490) 2024-04-02T14:45:29.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.779+0000 7fc4ad400640 1 -- 172.21.15.90:0/1662113129 learned_addr learned my addr 172.21.15.90:0/1662113129 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:45:29.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.780+0000 7fc4ad400640 1 -- 172.21.15.90:0/1662113129 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5599ccdffcb0 msgr2=0x5599ccdfd6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:45:29.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.780+0000 7fc4ae800640 1 --2- 172.21.15.90:0/1662113129 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5599ccdffcb0 0x5599ccdfd6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:29.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.781+0000 7fc4ad400640 1 --2- 172.21.15.90:0/1662113129 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5599ccdffcb0 0x5599ccdfd6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:29.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.781+0000 7fc4ad400640 1 -- 172.21.15.90:0/1662113129 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5599ccdf67b0 msgr2=0x5599ccdfcc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:45:29.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.781+0000 7fc4ad400640 1 --2- 172.21.15.90:0/1662113129 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5599ccdf67b0 0x5599ccdfcc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:29.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.781+0000 7fc4ad400640 1 -- 172.21.15.90:0/1662113129 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5599cca6e3f0 con 0x5599ccdf70d0 2024-04-02T14:45:29.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.781+0000 7fc4ade00640 1 --2- 172.21.15.90:0/1662113129 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5599ccdf67b0 0x5599ccdfcc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:45:29.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.781+0000 7fc4ae800640 1 --2- 172.21.15.90:0/1662113129 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5599ccdffcb0 0x5599ccdfd6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:45:29.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.781+0000 7fc4ad400640 1 --2- 172.21.15.90:0/1662113129 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5599ccdf70d0 0x5599ccdfd1b0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7fc4b019c470 tx=0x7fc49c005a00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5aa8ef4f274c3f13 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:45:29.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.781+0000 7fc48f400640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:45:29.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.782+0000 7fc48f400640 1 -- 172.21.15.90:0/1662113129 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc49c019830 con 0x5599ccdf70d0 2024-04-02T14:45:29.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.782+0000 7fc48f400640 1 -- 172.21.15.90:0/1662113129 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8550+0+0 (secure 0 0 0) 0x7fc49c0199d0 con 0x5599ccdf70d0 2024-04-02T14:45:29.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.782+0000 7fc4b787e740 1 -- 172.21.15.90:0/1662113129 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc4a00041c0 con 0x5599ccdf70d0 2024-04-02T14:45:29.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.782+0000 7fc48f400640 1 -- 172.21.15.90:0/1662113129 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fc49c02d050 con 0x5599ccdf70d0 2024-04-02T14:45:29.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.782+0000 7fc4b787e740 10 client.28534 fetch_fsmap learned FSMap version 27 2024-04-02T14:45:29.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.782+0000 7fc4b787e740 1 -- 172.21.15.90:0/1662113129 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5599cca2e0f0 con 0x5599ccdf70d0 2024-04-02T14:45:29.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.783+0000 7fc48f400640 1 -- 172.21.15.90:0/1662113129 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc4a00041c0 con 0x5599ccdf70d0 2024-04-02T14:45:29.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.783+0000 7fc48f400640 1 -- 172.21.15.90:0/1662113129 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fc49c016030 con 0x5599ccdf70d0 2024-04-02T14:45:29.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.783+0000 7fc4b787e740 10 client.28534 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:45:29.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.783+0000 7fc4b787e740 10 client.28534 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:45:29.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.783+0000 7fc4b787e740 20 client.28534 populate_metadata read hostname 'smithi090' 2024-04-02T14:45:29.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.783+0000 7fc4b787e740 1 --2- 172.21.15.90:0/1662113129 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5599cce0fcb0 0x5599cce120f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:29.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.783+0000 7fc4b787e740 4 client.28534 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:45:29.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.783+0000 7fc4b787e740 1 -- 172.21.15.90:0/1662113129 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5599cccfb630 con 0x5599cce0fcb0 2024-04-02T14:45:29.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.784+0000 7fc4ade00640 1 --2- 172.21.15.90:0/1662113129 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5599cce0fcb0 0x5599cce120f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:29.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.785+0000 7fc48f400640 10 client.28534 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:45:29.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.785+0000 7fc4ade00640 1 --2- 172.21.15.90:0/1662113129 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5599cce0fcb0 0x5599cce120f0 crc :-1 s=READY pgs=278 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e766cc6afe55976d server_cookie=61d3412a616b00ba in_seq=0 out_seq=0 2024-04-02T14:45:29.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.786+0000 7fc48f400640 1 -- 172.21.15.90:0/1662113129 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5599cccfb630 con 0x5599cce0fcb0 2024-04-02T14:45:29.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.786+0000 7fc48f400640 10 client.28534 handle_command_reply: tid=0 2024-04-02T14:45:29.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.787+0000 7fc4b787e740 1 client.28534 shutdown 2024-04-02T14:45:29.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.787+0000 7fc4b787e740 1 -- 172.21.15.90:0/1662113129 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5599ccdf70d0 msgr2=0x5599ccdfd1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:45:29.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.787+0000 7fc4b787e740 1 --2- 172.21.15.90:0/1662113129 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5599ccdf70d0 0x5599ccdfd1b0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7fc4b019c470 tx=0x7fc49c005a00 comp rx=0 tx=0).stop 2024-04-02T14:45:29.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.787+0000 7fc4b787e740 1 -- 172.21.15.90:0/1662113129 shutdown_connections 2024-04-02T14:45:29.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.787+0000 7fc4b787e740 1 --2- 172.21.15.90:0/1662113129 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5599cce0fcb0 0x5599cce120f0 crc :-1 s=READY pgs=278 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:29.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.787+0000 7fc48f400640 0 client.28534 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:45:29.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.787+0000 7fc4b787e740 1 --2- 172.21.15.90:0/1662113129 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5599ccdffcb0 0x5599ccdfd6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:29.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.787+0000 7fc4b787e740 1 --2- 172.21.15.90:0/1662113129 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5599ccdf70d0 0x5599ccdfd1b0 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:29.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.787+0000 7fc4b787e740 1 --2- 172.21.15.90:0/1662113129 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5599ccdf67b0 0x5599ccdfcc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:29.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.788+0000 7fc4b787e740 1 -- 172.21.15.90:0/1662113129 >> 172.21.15.90:0/1662113129 conn(0x5599ccdf0ee0 msgr2=0x5599ccdf2c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:45:29.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.788+0000 7fc4b787e740 1 -- 172.21.15.90:0/1662113129 shutdown_connections 2024-04-02T14:45:29.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.788+0000 7fc4b787e740 1 -- 172.21.15.90:0/1662113129 wait complete. 2024-04-02T14:45:29.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.788+0000 7fc4b787e740 20 client.28534 trim_cache size 0 max 16384 2024-04-02T14:45:29.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.834+0000 7fc4b787e740 1 Processor -- start 2024-04-02T14:45:29.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.834+0000 7fc4b787e740 1 -- start start 2024-04-02T14:45:29.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.835+0000 7fc4b787e740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5599ccdf67b0 0x5599ccdff3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:29.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.835+0000 7fc4b787e740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5599ccdf70d0 0x5599cce0b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:29.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.835+0000 7fc4b787e740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5599ccdffcb0 0x5599ccdf12c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:29.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.835+0000 7fc4b787e740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fc4b01bc780 con 0x5599ccdf70d0 2024-04-02T14:45:29.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.835+0000 7fc4b787e740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fc4b01be2c0 con 0x5599ccdf67b0 2024-04-02T14:45:29.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.835+0000 7fc4b787e740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fc4b01bce50 con 0x5599ccdffcb0 2024-04-02T14:45:29.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.836+0000 7fc4ade00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5599ccdf67b0 0x5599ccdff3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:29.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.836+0000 7fc4ade00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5599ccdf67b0 0x5599ccdff3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:39478/0 (socket says 172.21.15.90:39478) 2024-04-02T14:45:29.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.836+0000 7fc4ade00640 1 -- 172.21.15.90:0/205780179 learned_addr learned my addr 172.21.15.90:0/205780179 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:45:29.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.836+0000 7fc4ae800640 1 --2- 172.21.15.90:0/205780179 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5599ccdffcb0 0x5599ccdf12c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:29.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.836+0000 7fc4ad400640 1 --2- 172.21.15.90:0/205780179 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5599ccdf70d0 0x5599cce0b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:29.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.837+0000 7fc4ade00640 1 -- 172.21.15.90:0/205780179 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5599ccdffcb0 msgr2=0x5599ccdf12c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:45:29.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.837+0000 7fc4ade00640 1 --2- 172.21.15.90:0/205780179 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5599ccdffcb0 0x5599ccdf12c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:29.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.837+0000 7fc4ade00640 1 -- 172.21.15.90:0/205780179 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5599ccdf70d0 msgr2=0x5599cce0b9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:45:29.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.837+0000 7fc4ade00640 1 --2- 172.21.15.90:0/205780179 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5599ccdf70d0 0x5599cce0b9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:29.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.837+0000 7fc4ade00640 1 -- 172.21.15.90:0/205780179 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5599ccdfb1c0 con 0x5599ccdf67b0 2024-04-02T14:45:29.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.837+0000 7fc4ae800640 1 --2- 172.21.15.90:0/205780179 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5599ccdffcb0 0x5599ccdf12c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:45:29.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.837+0000 7fc4ad400640 1 --2- 172.21.15.90:0/205780179 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5599ccdf70d0 0x5599cce0b9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:45:29.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.838+0000 7fc4ade00640 1 --2- 172.21.15.90:0/205780179 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5599ccdf67b0 0x5599ccdff3a0 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7fc4a0009ba0 tx=0x7fc4a0003dd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fde86d6e55330406 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:45:29.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.838+0000 7fc49a600640 1 -- 172.21.15.90:0/205780179 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4a002e070 con 0x5599ccdf67b0 2024-04-02T14:45:29.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.838+0000 7fc49a600640 1 -- 172.21.15.90:0/205780179 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc4a0009780 con 0x5599ccdf67b0 2024-04-02T14:45:29.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.839+0000 7fc49a600640 1 -- 172.21.15.90:0/205780179 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4a002e070 con 0x5599ccdf67b0 2024-04-02T14:45:29.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.840+0000 7fc4b787e740 1 -- 172.21.15.90:0/205780179 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5599ccdf67b0 msgr2=0x5599ccdff3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:45:29.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.840+0000 7fc4b787e740 1 --2- 172.21.15.90:0/205780179 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5599ccdf67b0 0x5599ccdff3a0 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7fc4a0009ba0 tx=0x7fc4a0003dd0 comp rx=0 tx=0).stop 2024-04-02T14:45:29.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.840+0000 7fc4b787e740 1 -- 172.21.15.90:0/205780179 shutdown_connections 2024-04-02T14:45:29.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.840+0000 7fc4b787e740 1 --2- 172.21.15.90:0/205780179 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5599ccdffcb0 0x5599ccdf12c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:29.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.840+0000 7fc4b787e740 1 --2- 172.21.15.90:0/205780179 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5599ccdf70d0 0x5599cce0b9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:29.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.840+0000 7fc4b787e740 1 --2- 172.21.15.90:0/205780179 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5599ccdf67b0 0x5599ccdff3a0 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:29.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.840+0000 7fc4b787e740 1 -- 172.21.15.90:0/205780179 >> 172.21.15.90:0/205780179 conn(0x5599ccdf0ee0 msgr2=0x5599ccdf2310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:45:29.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.840+0000 7fc4b787e740 1 -- 172.21.15.90:0/205780179 shutdown_connections 2024-04-02T14:45:29.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.841+0000 7fc4b787e740 1 -- 172.21.15.90:0/205780179 wait complete. 2024-04-02T14:45:29.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.841+0000 7fc4b787e740 1 -- start start 2024-04-02T14:45:29.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.841+0000 7fc4b787e740 1 Processor -- start 2024-04-02T14:45:29.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.842+0000 7fc4b787e740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5599ccdf67b0 0x5599cce03910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:29.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.842+0000 7fc4b787e740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5599ccdf70d0 0x5599cce03e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:29.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.842+0000 7fc4b787e740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5599ccdffcb0 0x5599cce04390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:29.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.842+0000 7fc4b787e740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fc4b01bdbf0 con 0x5599ccdf67b0 2024-04-02T14:45:29.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.842+0000 7fc4b787e740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fc4b01bd520 con 0x5599ccdf70d0 2024-04-02T14:45:29.840 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.842+0000 7fc4b787e740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fc4b01a3740 con 0x5599ccdffcb0 2024-04-02T14:45:29.840 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.842+0000 7fc4b787e740 10 client.0 _ll_register_callbacks cb 0x7fc49c00cf50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:45:29.840 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.843+0000 7fc4ae800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5599ccdffcb0 0x5599cce04390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:29.840 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.843+0000 7fc4ae800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5599ccdffcb0 0x5599cce04390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:55510/0 (socket says 172.21.15.90:55510) 2024-04-02T14:45:29.840 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.843+0000 7fc4ae800640 1 -- 172.21.15.90:0/2409909835 learned_addr learned my addr 172.21.15.90:0/2409909835 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:45:29.841 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.843+0000 7fc4ad400640 1 --2- 172.21.15.90:0/2409909835 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5599ccdf70d0 0x5599cce03e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:29.841 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.843+0000 7fc4ade00640 1 --2- 172.21.15.90:0/2409909835 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5599ccdf67b0 0x5599cce03910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:29.841 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.844+0000 7fc4ae800640 1 -- 172.21.15.90:0/2409909835 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5599ccdf70d0 msgr2=0x5599cce03e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:45:29.841 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.844+0000 7fc4ae800640 1 --2- 172.21.15.90:0/2409909835 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5599ccdf70d0 0x5599cce03e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:29.841 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.844+0000 7fc4ae800640 1 -- 172.21.15.90:0/2409909835 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5599ccdf67b0 msgr2=0x5599cce03910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:45:29.841 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.844+0000 7fc4ae800640 1 --2- 172.21.15.90:0/2409909835 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5599ccdf67b0 0x5599cce03910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:29.841 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.844+0000 7fc4ae800640 1 -- 172.21.15.90:0/2409909835 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5599cce75280 con 0x5599ccdffcb0 2024-04-02T14:45:29.841 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.844+0000 7fc4ade00640 1 --2- 172.21.15.90:0/2409909835 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5599ccdf67b0 0x5599cce03910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:45:29.841 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.844+0000 7fc4ae800640 1 --2- 172.21.15.90:0/2409909835 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5599ccdffcb0 0x5599cce04390 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7fc4a8033420 tx=0x7fc4a80bf690 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d9aa7820a6c30c4d server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:45:29.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.844+0000 7fc48f400640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:45:29.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.845+0000 7fc48f400640 1 -- 172.21.15.90:0/2409909835 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4a80d7020 con 0x5599ccdffcb0 2024-04-02T14:45:29.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.845+0000 7fc48f400640 1 -- 172.21.15.90:0/2409909835 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8550+0+0 (secure 0 0 0) 0x7fc4a803cda0 con 0x5599ccdffcb0 2024-04-02T14:45:29.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.849+0000 7fc4b787e740 1 -- 172.21.15.90:0/2409909835 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc4a0009780 con 0x5599ccdffcb0 2024-04-02T14:45:29.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.849+0000 7fc48f400640 1 -- 172.21.15.90:0/2409909835 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fc4a80bd150 con 0x5599ccdffcb0 2024-04-02T14:45:29.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.849+0000 7fc4b787e740 10 client.28540 fetch_fsmap learned FSMap version 27 2024-04-02T14:45:29.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.849+0000 7fc4b787e740 1 -- 172.21.15.90:0/2409909835 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5599cce765d0 con 0x5599ccdffcb0 2024-04-02T14:45:29.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.850+0000 7fc48f400640 1 -- 172.21.15.90:0/2409909835 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc4a0009780 con 0x5599ccdffcb0 2024-04-02T14:45:29.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.850+0000 7fc48f400640 1 -- 172.21.15.90:0/2409909835 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fc4a80c7730 con 0x5599ccdffcb0 2024-04-02T14:45:29.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.850+0000 7fc4b787e740 10 client.28540 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:45:29.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.850+0000 7fc4b787e740 10 client.28540 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:45:29.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.850+0000 7fc4b787e740 20 client.28540 populate_metadata read hostname 'smithi090' 2024-04-02T14:45:29.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.850+0000 7fc4b787e740 1 --2- 172.21.15.90:0/2409909835 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5599cce0fcb0 0x5599cce78d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:29.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.850+0000 7fc4b787e740 4 client.28540 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-02T14:45:29.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.850+0000 7fc4b787e740 1 -- 172.21.15.90:0/2409909835 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5599ccd67cb0 con 0x5599cce0fcb0 2024-04-02T14:45:29.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.851+0000 7fc4ade00640 1 --2- 172.21.15.90:0/2409909835 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5599cce0fcb0 0x5599cce78d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:29.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.851+0000 7fc4ade00640 1 --2- 172.21.15.90:0/2409909835 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5599cce0fcb0 0x5599cce78d60 crc :-1 s=READY pgs=279 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=61556294d4222882 server_cookie=2c9675b9e1bce6cd in_seq=0 out_seq=0 2024-04-02T14:45:29.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.851+0000 7fc48f400640 10 client.28540 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:45:29.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.852+0000 7fc48f400640 1 -- 172.21.15.90:0/2409909835 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5599ccd67cb0 con 0x5599cce0fcb0 2024-04-02T14:45:29.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.852+0000 7fc48f400640 10 client.28540 handle_command_reply: tid=0 2024-04-02T14:45:29.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.852+0000 7fc4b787e740 1 client.28540 shutdown 2024-04-02T14:45:29.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.852+0000 7fc4b787e740 1 -- 172.21.15.90:0/2409909835 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5599ccdffcb0 msgr2=0x5599cce04390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:45:29.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.852+0000 7fc4b787e740 1 --2- 172.21.15.90:0/2409909835 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5599ccdffcb0 0x5599cce04390 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7fc4a8033420 tx=0x7fc4a80bf690 comp rx=0 tx=0).stop 2024-04-02T14:45:29.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.852+0000 7fc4b787e740 1 -- 172.21.15.90:0/2409909835 shutdown_connections 2024-04-02T14:45:29.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.852+0000 7fc4b787e740 1 --2- 172.21.15.90:0/2409909835 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5599cce0fcb0 0x5599cce78d60 crc :-1 s=READY pgs=279 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:29.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.852+0000 7fc4b787e740 1 --2- 172.21.15.90:0/2409909835 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5599ccdffcb0 0x5599cce04390 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:29.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.852+0000 7fc4b787e740 1 --2- 172.21.15.90:0/2409909835 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5599ccdf70d0 0x5599cce03e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:29.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.852+0000 7fc4b787e740 1 --2- 172.21.15.90:0/2409909835 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5599ccdf67b0 0x5599cce03910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:29.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.852+0000 7fc4b787e740 1 -- 172.21.15.90:0/2409909835 >> 172.21.15.90:0/2409909835 conn(0x5599ccdf0ee0 msgr2=0x5599ccdf2ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:45:29.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.852+0000 7fc48f400640 0 client.28540 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:45:29.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.853+0000 7fc4b787e740 1 -- 172.21.15.90:0/2409909835 shutdown_connections 2024-04-02T14:45:29.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.853+0000 7fc4b787e740 1 -- 172.21.15.90:0/2409909835 wait complete. 2024-04-02T14:45:29.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.853+0000 7fc4b787e740 20 client.28540 trim_cache size 0 max 16384 2024-04-02T14:45:29.851 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-02T14:45:29.851 INFO:teuthology.orchestra.run.smithi090.stdout: "status": "no active scrubs running", 2024-04-02T14:45:29.851 INFO:teuthology.orchestra.run.smithi090.stdout: "scrubs": {} 2024-04-02T14:45:29.852 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-02T14:45:29.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.854+0000 7fc4b4c00640 1 -- 172.21.15.90:0/3580577046 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fc4940a3da0 msgr2=0x7fc4940a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:45:29.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.854+0000 7fc4b4c00640 1 --2- 172.21.15.90:0/3580577046 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fc4940a3da0 0x7fc4940a6260 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x5599ccdf2070 tx=0x7fc49c005d50 comp rx=0 tx=0).stop 2024-04-02T14:45:29.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.854+0000 7fc4b4c00640 1 -- 172.21.15.90:0/3580577046 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc4b0106980 msgr2=0x7fc4b010ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:45:29.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.854+0000 7fc4b4c00640 1 --2- 172.21.15.90:0/3580577046 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc4b0106980 0x7fc4b010ea50 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7fc4a8004af0 tx=0x7fc4a802cbc0 comp rx=0 tx=0).stop 2024-04-02T14:45:29.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.854+0000 7fc4b4c00640 1 -- 172.21.15.90:0/3580577046 shutdown_connections 2024-04-02T14:45:29.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.854+0000 7fc4b4c00640 1 --2- 172.21.15.90:0/3580577046 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fc4940a3da0 0x7fc4940a6260 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:29.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.854+0000 7fc4b4c00640 1 --2- 172.21.15.90:0/3580577046 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc4b0106980 0x7fc4b010ea50 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:29.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.854+0000 7fc4b4c00640 1 --2- 172.21.15.90:0/3580577046 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc4b0101330 0x7fc4b010e510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:29.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.854+0000 7fc4b4c00640 1 --2- 172.21.15.90:0/3580577046 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc4b0100970 0x7fc4b010dfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:29.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.854+0000 7fc4b4c00640 1 -- 172.21.15.90:0/3580577046 >> 172.21.15.90:0/3580577046 conn(0x7fc4b0077b00 msgr2=0x7fc4b0103030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:45:29.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.857+0000 7fc4b4c00640 1 -- 172.21.15.90:0/3580577046 shutdown_connections 2024-04-02T14:45:29.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:29.857+0000 7fc4b4c00640 1 -- 172.21.15.90:0/3580577046 wait complete. 2024-04-02T14:45:29.884 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-02T14:45:29.884 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T14:45:30.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:30.158+0000 7fcecbe00640 1 -- 172.21.15.90:0/1949036366 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcecc1078d0 msgr2=0x7fcecc107cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:45:30.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:30.158+0000 7fcecbe00640 1 --2- 172.21.15.90:0/1949036366 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcecc1078d0 0x7fcecc107cb0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7fcebc009890 tx=0x7fcebc02bb70 comp rx=0 tx=0).stop 2024-04-02T14:45:30.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:30.158+0000 7fcecbe00640 1 -- 172.21.15.90:0/1949036366 shutdown_connections 2024-04-02T14:45:30.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:30.158+0000 7fcecbe00640 1 --2- 172.21.15.90:0/1949036366 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcecc102280 0x7fcecc10e790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:30.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:30.158+0000 7fcecbe00640 1 --2- 172.21.15.90:0/1949036366 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcecc1018c0 0x7fcecc101d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:30.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:30.158+0000 7fcecbe00640 1 --2- 172.21.15.90:0/1949036366 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcecc1078d0 0x7fcecc107cb0 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:30.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:30.158+0000 7fcecbe00640 1 -- 172.21.15.90:0/1949036366 >> 172.21.15.90:0/1949036366 conn(0x7fcecc0fd3d0 msgr2=0x7fcecc0ff7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:45:30.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:30.159+0000 7fcecbe00640 1 -- 172.21.15.90:0/1949036366 shutdown_connections 2024-04-02T14:45:30.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:30.159+0000 7fcecbe00640 1 -- 172.21.15.90:0/1949036366 wait complete. 2024-04-02T14:45:30.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:30.160+0000 7fcecbe00640 1 Processor -- start 2024-04-02T14:45:30.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:30.161+0000 7fcecbe00640 1 -- start start 2024-04-02T14:45:30.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:30.161+0000 7fcecbe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcecc1018c0 0x7fcecc10dc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:30.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:30.161+0000 7fcecbe00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcecc102280 0x7fcecc10e150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:30.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:30.161+0000 7fcecbe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcecc1078d0 0x7fcecc1a3140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:30.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:30.161+0000 7fcecbe00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fcecc111d00 con 0x7fcecc1078d0 2024-04-02T14:45:30.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:30.161+0000 7fcecbe00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fcecc111b80 con 0x7fcecc102280 2024-04-02T14:45:30.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:30.161+0000 7fcecbe00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fcecc111e80 con 0x7fcecc1018c0 2024-04-02T14:45:30.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:30.162+0000 7fceca000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcecc102280 0x7fcecc10e150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:30.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:30.162+0000 7fceca000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcecc102280 0x7fcecc10e150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:39498/0 (socket says 172.21.15.90:39498) 2024-04-02T14:45:30.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:30.162+0000 7fceca000640 1 -- 172.21.15.90:0/3747817417 learned_addr learned my addr 172.21.15.90:0/3747817417 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:45:30.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:30.162+0000 7fcecb400640 1 --2- 172.21.15.90:0/3747817417 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcecc1078d0 0x7fcecc1a3140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:30.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:30.163+0000 7fcecb400640 1 -- 172.21.15.90:0/3747817417 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcecc1018c0 msgr2=0x7fcecc10dc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:45:30.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:30.163+0000 7fcecb400640 1 --2- 172.21.15.90:0/3747817417 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcecc1018c0 0x7fcecc10dc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:30.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:30.163+0000 7fcecb400640 1 -- 172.21.15.90:0/3747817417 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcecc102280 msgr2=0x7fcecc10e150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:45:30.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:30.163+0000 7fcecb400640 1 --2- 172.21.15.90:0/3747817417 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcecc102280 0x7fcecc10e150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:30.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:30.163+0000 7fcecb400640 1 -- 172.21.15.90:0/3747817417 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcecc1a3840 con 0x7fcecc1078d0 2024-04-02T14:45:30.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:30.164+0000 7fcecb400640 1 --2- 172.21.15.90:0/3747817417 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcecc1078d0 0x7fcecc1a3140 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fcec400da10 tx=0x7fcec400ded0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:45:30.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:30.164+0000 7fcec3400640 1 -- 172.21.15.90:0/3747817417 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcec400bd70 con 0x7fcecc1078d0 2024-04-02T14:45:30.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:30.164+0000 7fcec3400640 1 -- 172.21.15.90:0/3747817417 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcec4003e50 con 0x7fcecc1078d0 2024-04-02T14:45:30.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:30.164+0000 7fcec3400640 1 -- 172.21.15.90:0/3747817417 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcec40106a0 con 0x7fcecc1078d0 2024-04-02T14:45:30.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:30.165+0000 7fcecbe00640 1 -- 172.21.15.90:0/3747817417 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcecc1a3b30 con 0x7fcecc1078d0 2024-04-02T14:45:30.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:30.166+0000 7fcecbe00640 1 -- 172.21.15.90:0/3747817417 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcecc197190 con 0x7fcecc1078d0 2024-04-02T14:45:30.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:30.168+0000 7fcecbe00640 1 -- 172.21.15.90:0/3747817417 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcecc103000 con 0x7fcecc1078d0 2024-04-02T14:45:30.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:30.171+0000 7fcec3400640 1 -- 172.21.15.90:0/3747817417 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fcec4003970 con 0x7fcecc1078d0 2024-04-02T14:45:30.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:30.173+0000 7fcec3400640 1 --2- 172.21.15.90:0/3747817417 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fceb00a3ea0 0x7fceb00a6360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:30.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:30.174+0000 7fcec3400640 1 -- 172.21.15.90:0/3747817417 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 8550+0+0 (secure 0 0 0) 0x7fcec40d0310 con 0x7fcecc1078d0 2024-04-02T14:45:30.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:30.174+0000 7fcecaa00640 1 --2- 172.21.15.90:0/3747817417 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fceb00a3ea0 0x7fceb00a6360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:30.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:30.176+0000 7fcecaa00640 1 --2- 172.21.15.90:0/3747817417 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fceb00a3ea0 0x7fceb00a6360 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fcebc009890 tx=0x7fcebc005b20 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:45:30.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:30.184+0000 7fcec3400640 1 -- 172.21.15.90:0/3747817417 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7fcec409ca10 con 0x7fcecc1078d0 2024-04-02T14:45:30.605 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:45:30 smithi092 ceph-mon[36497]: pgmap v329: 97 pgs: 97 active+clean; 4.4 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 6.8 MiB/s wr, 661 op/s 2024-04-02T14:45:30.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:30.647+0000 7fcecbe00640 1 -- 172.21.15.90:0/3747817417 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcecc197480 con 0x7fcecc1078d0 2024-04-02T14:45:30.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:30.653+0000 7fcec3400640 1 -- 172.21.15.90:0/3747817417 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11927 (secure 0 0 0) 0x7fcec409c310 con 0x7fcecc1078d0 2024-04-02T14:45:30.650 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-02T14:45:30.650 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14412,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6837/2852160309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6836","nonce":2852160309},{"type":"v1","addr":"172.21.15.90:6837","nonce":2852160309}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14478,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6841/1919376858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6840","nonce":1919376858},{"type":"v1","addr":"172.21.15.90:6841","nonce":1919376858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-02T14:32:22.319083+0000","modified":"2024-04-02T14:34:52.266152+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24590,"mds_1":24560,"mds_2":24547,"mds_3":24539,"mds_4":24523},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24454":{"gid":24454,"name":"l","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6833/1564319431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":1564319431},{"type":"v1","addr":"172.21.15.177:6833","nonce":1564319431}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.92:6833/1570337372","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":1570337372},{"type":"v1","addr":"172.21.15.92:6833","nonce":1570337372}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"j","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.90:6835/1277124891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6834","nonce":1277124891},{"type":"v1","addr":"172.21.15.90:6835","nonce":1277124891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.92:6835/860721164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":860721164},{"type":"v1","addr":"172.21.15.92:6835","nonce":860721164}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24511":{"gid":24511,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6835/3177244988","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":3177244988},{"type":"v1","addr":"172.21.15.177:6835","nonce":3177244988}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"c","rank":4,"incarnation":25,"state":"up:active","state_seq":9,"addr":"172.21.15.177:6837/4223414858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":4223414858},{"type":"v1","addr":"172.21.15.177:6837","nonce":4223414858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24539":{"gid":24539,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":8,"addr":"172.21.15.92:6837/1270635939","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":1270635939},{"type":"v1","addr":"172.21.15.92:6837","nonce":1270635939}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24547":{"gid":24547,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":7,"addr":"172.21.15.177:6839/664895663","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":664895663},{"type":"v1","addr":"172.21.15.177:6839","nonce":664895663}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"g","rank":1,"incarnation":19,"state":"up:active","state_seq":5,"addr":"172.21.15.90:6839/1498915870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6838","nonce":1498915870},{"type":"v1","addr":"172.21.15.90:6839","nonce":1498915870}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24590":{"gid":24590,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.92:6839/1054888219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":1054888219},{"type":"v1","addr":"172.21.15.92:6839","nonce":1054888219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24590,"qdb_cluster":[24547,24590,24523,24539,24560]},"id":1}]} 2024-04-02T14:45:30.650 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 27 2024-04-02T14:45:30.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:30.657+0000 7fcecbe00640 1 -- 172.21.15.90:0/3747817417 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fceb00a3ea0 msgr2=0x7fceb00a6360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:45:30.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:30.657+0000 7fcecbe00640 1 --2- 172.21.15.90:0/3747817417 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fceb00a3ea0 0x7fceb00a6360 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fcebc009890 tx=0x7fcebc005b20 comp rx=0 tx=0).stop 2024-04-02T14:45:30.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:30.657+0000 7fcecbe00640 1 -- 172.21.15.90:0/3747817417 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcecc1078d0 msgr2=0x7fcecc1a3140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:45:30.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:30.657+0000 7fcecbe00640 1 --2- 172.21.15.90:0/3747817417 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcecc1078d0 0x7fcecc1a3140 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fcec400da10 tx=0x7fcec400ded0 comp rx=0 tx=0).stop 2024-04-02T14:45:30.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:30.657+0000 7fcecbe00640 1 -- 172.21.15.90:0/3747817417 shutdown_connections 2024-04-02T14:45:30.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:30.657+0000 7fcecbe00640 1 --2- 172.21.15.90:0/3747817417 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fceb00a3ea0 0x7fceb00a6360 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:30.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:30.657+0000 7fcecbe00640 1 --2- 172.21.15.90:0/3747817417 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcecc1078d0 0x7fcecc1a3140 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:30.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:30.657+0000 7fcecbe00640 1 --2- 172.21.15.90:0/3747817417 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcecc102280 0x7fcecc10e150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:30.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:30.657+0000 7fcecbe00640 1 --2- 172.21.15.90:0/3747817417 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcecc1018c0 0x7fcecc10dc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:30.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:30.657+0000 7fcecbe00640 1 -- 172.21.15.90:0/3747817417 >> 172.21.15.90:0/3747817417 conn(0x7fcecc0fd3d0 msgr2=0x7fcecc0fda10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:45:30.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:30.657+0000 7fcecbe00640 1 -- 172.21.15.90:0/3747817417 shutdown_connections 2024-04-02T14:45:30.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:30.657+0000 7fcecbe00640 1 -- 172.21.15.90:0/3747817417 wait complete. 2024-04-02T14:45:30.679 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:45:30 smithi090 ceph-mon[31674]: pgmap v329: 97 pgs: 97 active+clean; 4.4 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 6.8 MiB/s wr, 661 op/s 2024-04-02T14:45:30.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:45:30 smithi177 ceph-mon[36431]: pgmap v329: 97 pgs: 97 active+clean; 4.4 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 6.8 MiB/s wr, 661 op/s 2024-04-02T14:45:30.685 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-04-02T14:45:30.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:30.963+0000 7f27d8800640 1 -- 172.21.15.90:0/3691924066 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f27d0069600 msgr2=0x7f27d010df60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:45:30.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:30.963+0000 7f27d8800640 1 --2- 172.21.15.90:0/3691924066 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f27d0069600 0x7f27d010df60 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f27c0009900 tx=0x7f27c002bc40 comp rx=0 tx=0).stop 2024-04-02T14:45:30.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:30.963+0000 7f27d8800640 1 -- 172.21.15.90:0/3691924066 shutdown_connections 2024-04-02T14:45:30.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:30.963+0000 7f27d8800640 1 --2- 172.21.15.90:0/3691924066 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f27d010e4a0 0x7f27d0110890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:30.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:30.963+0000 7f27d8800640 1 --2- 172.21.15.90:0/3691924066 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f27d0069600 0x7f27d010df60 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:30.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:30.963+0000 7f27d8800640 1 --2- 172.21.15.90:0/3691924066 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f27d0068ce0 0x7f27d00690c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:30.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:30.963+0000 7f27d8800640 1 -- 172.21.15.90:0/3691924066 >> 172.21.15.90:0/3691924066 conn(0x7f27d0077b00 msgr2=0x7f27d0077f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:45:30.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:30.964+0000 7f27d8800640 1 -- 172.21.15.90:0/3691924066 shutdown_connections 2024-04-02T14:45:30.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:30.964+0000 7f27d8800640 1 -- 172.21.15.90:0/3691924066 wait complete. 2024-04-02T14:45:30.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:30.965+0000 7f27d8800640 1 Processor -- start 2024-04-02T14:45:30.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:30.966+0000 7f27d8800640 1 -- start start 2024-04-02T14:45:30.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:30.966+0000 7f27d8800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f27d0068ce0 0x7f27d0102bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:30.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:30.966+0000 7f27d8800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f27d0069600 0x7f27d0103110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:30.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:30.967+0000 7f27d5e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f27d0068ce0 0x7f27d0102bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:30.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:30.967+0000 7f27d8800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f27d010e4a0 0x7f27d0103650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:30.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:30.967+0000 7f27d5e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f27d0068ce0 0x7f27d0102bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:39522/0 (socket says 172.21.15.90:39522) 2024-04-02T14:45:30.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:30.967+0000 7f27d6800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f27d010e4a0 0x7f27d0103650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:30.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:30.967+0000 7f27d6800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f27d010e4a0 0x7f27d0103650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:55538/0 (socket says 172.21.15.90:55538) 2024-04-02T14:45:30.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:30.967+0000 7f27d8800640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f27d0114090 con 0x7f27d0069600 2024-04-02T14:45:30.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:30.967+0000 7f27d8800640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f27d0113f10 con 0x7f27d0068ce0 2024-04-02T14:45:30.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:30.967+0000 7f27d8800640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f27d0114210 con 0x7f27d010e4a0 2024-04-02T14:45:30.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:30.967+0000 7f27d5e00640 1 -- 172.21.15.90:0/1880970598 learned_addr learned my addr 172.21.15.90:0/1880970598 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:45:30.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:30.968+0000 7f27d5400640 1 --2- 172.21.15.90:0/1880970598 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f27d0069600 0x7f27d0103110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:30.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:30.968+0000 7f27d6800640 1 -- 172.21.15.90:0/1880970598 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f27d0068ce0 msgr2=0x7f27d0102bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:45:30.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:30.968+0000 7f27d6800640 1 --2- 172.21.15.90:0/1880970598 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f27d0068ce0 0x7f27d0102bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:30.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:30.968+0000 7f27d6800640 1 -- 172.21.15.90:0/1880970598 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f27d0069600 msgr2=0x7f27d0103110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:45:30.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:30.968+0000 7f27d6800640 1 --2- 172.21.15.90:0/1880970598 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f27d0069600 0x7f27d0103110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:30.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:30.968+0000 7f27d6800640 1 -- 172.21.15.90:0/1880970598 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27d01aac60 con 0x7f27d010e4a0 2024-04-02T14:45:30.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:30.969+0000 7f27d6800640 1 --2- 172.21.15.90:0/1880970598 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f27d010e4a0 0x7f27d0103650 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f27cc005fd0 tx=0x7f27cc00edd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:45:30.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:30.969+0000 7f27caa00640 1 -- 172.21.15.90:0/1880970598 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27cc004d20 con 0x7f27d010e4a0 2024-04-02T14:45:30.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:30.969+0000 7f27caa00640 1 -- 172.21.15.90:0/1880970598 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f27cc005020 con 0x7f27d010e4a0 2024-04-02T14:45:30.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:30.969+0000 7f27caa00640 1 -- 172.21.15.90:0/1880970598 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27cc0116b0 con 0x7f27d010e4a0 2024-04-02T14:45:30.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:30.969+0000 7f27d8800640 1 -- 172.21.15.90:0/1880970598 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27d01aaf50 con 0x7f27d010e4a0 2024-04-02T14:45:30.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:30.970+0000 7f27d8800640 1 -- 172.21.15.90:0/1880970598 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27d01ab3c0 con 0x7f27d010e4a0 2024-04-02T14:45:30.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:30.974+0000 7f27db568740 1 Processor -- start 2024-04-02T14:45:30.972 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:30.975+0000 7f27caa00640 1 -- 172.21.15.90:0/1880970598 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f27cc011910 con 0x7f27d010e4a0 2024-04-02T14:45:30.972 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:30.976+0000 7f27db568740 1 -- start start 2024-04-02T14:45:30.972 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:30.976+0000 7f27db568740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e18fbe89f0 0x55e18fbe7740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:30.972 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:30.976+0000 7f27db568740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e18fbe7d10 0x55e18fbe8190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:30.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:30.976+0000 7f27db568740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e18fbf1080 0x55e18fbf3470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:30.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:30.976+0000 7f27db568740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55e18fb28400 con 0x55e18fbf1080 2024-04-02T14:45:30.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:30.976+0000 7f27db568740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55e18fb0cf10 con 0x55e18fbe89f0 2024-04-02T14:45:30.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:30.976+0000 7f27db568740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55e18fb4c6f0 con 0x55e18fbe7d10 2024-04-02T14:45:30.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:30.977+0000 7f27d5400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e18fbf1080 0x55e18fbf3470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:30.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:30.977+0000 7f27d5400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e18fbe89f0 0x55e18fbe7740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:30.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:30.977+0000 7f27d5400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e18fbe89f0 0x55e18fbe7740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:39524/0 (socket says 172.21.15.90:39524) 2024-04-02T14:45:30.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:30.977+0000 7f27d5400640 1 -- 172.21.15.90:0/2628105576 learned_addr learned my addr 172.21.15.90:0/2628105576 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:45:30.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:30.977+0000 7f27caa00640 1 --2- 172.21.15.90:0/1880970598 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f27b80a3ef0 0x7f27b80a63b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:30.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:30.978+0000 7f27caa00640 1 -- 172.21.15.90:0/1880970598 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 8550+0+0 (secure 0 0 0) 0x7f27cc0d07e0 con 0x7f27d010e4a0 2024-04-02T14:45:30.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:30.978+0000 7f27d5e00640 1 --2- 172.21.15.90:0/2628105576 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e18fbe7d10 0x55e18fbe8190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:30.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:30.978+0000 7f27d6800640 1 --2- 172.21.15.90:0/1880970598 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f27b80a3ef0 0x7f27b80a63b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:30.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:30.978+0000 7f27d5400640 1 -- 172.21.15.90:0/2628105576 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e18fbe7d10 msgr2=0x55e18fbe8190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:45:30.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:30.978+0000 7f27d5400640 1 --2- 172.21.15.90:0/2628105576 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e18fbe7d10 0x55e18fbe8190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:30.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:30.978+0000 7f27d5400640 1 -- 172.21.15.90:0/2628105576 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e18fbe89f0 msgr2=0x55e18fbe7740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:45:30.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:30.978+0000 7f27d5400640 1 --2- 172.21.15.90:0/2628105576 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e18fbe89f0 0x55e18fbe7740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:30.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:30.979+0000 7f27d5400640 1 -- 172.21.15.90:0/2628105576 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e18f8603f0 con 0x55e18fbf1080 2024-04-02T14:45:30.976 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:30.979+0000 7f27d5e00640 1 --2- 172.21.15.90:0/2628105576 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e18fbe7d10 0x55e18fbe8190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:45:30.976 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:30.979+0000 7f27d6800640 1 --2- 172.21.15.90:0/1880970598 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f27b80a3ef0 0x7f27b80a63b0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f27cc09c400 tx=0x7f27cc0054b0 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:45:30.976 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:30.979+0000 7f27d5400640 1 --2- 172.21.15.90:0/2628105576 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e18fbf1080 0x55e18fbf3470 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f27c002efd0 tx=0x7f27c0005a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dead842fa8d03ac7 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:45:30.976 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:30.980+0000 7f27bf000640 1 -- 172.21.15.90:0/2628105576 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27c00044a0 con 0x55e18fbf1080 2024-04-02T14:45:30.976 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:30.980+0000 7f27bf000640 1 -- 172.21.15.90:0/2628105576 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f27c0047070 con 0x55e18fbf1080 2024-04-02T14:45:30.977 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:30.980+0000 7f27bf000640 1 -- 172.21.15.90:0/2628105576 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27c0004920 con 0x55e18fbf1080 2024-04-02T14:45:30.977 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:30.981+0000 7f27db568740 1 -- 172.21.15.90:0/2628105576 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e18fbf1080 msgr2=0x55e18fbf3470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:45:30.977 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:30.981+0000 7f27db568740 1 --2- 172.21.15.90:0/2628105576 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e18fbf1080 0x55e18fbf3470 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f27c002efd0 tx=0x7f27c0005a20 comp rx=0 tx=0).stop 2024-04-02T14:45:30.978 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:30.981+0000 7f27db568740 1 -- 172.21.15.90:0/2628105576 shutdown_connections 2024-04-02T14:45:30.978 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:30.981+0000 7f27db568740 1 --2- 172.21.15.90:0/2628105576 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e18fbf1080 0x55e18fbf3470 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:30.978 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:30.981+0000 7f27db568740 1 --2- 172.21.15.90:0/2628105576 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e18fbe7d10 0x55e18fbe8190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:30.978 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:30.981+0000 7f27db568740 1 --2- 172.21.15.90:0/2628105576 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e18fbe89f0 0x55e18fbe7740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:30.978 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:30.981+0000 7f27db568740 1 -- 172.21.15.90:0/2628105576 >> 172.21.15.90:0/2628105576 conn(0x55e18fbaea50 msgr2=0x55e18fbe2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:45:30.978 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:30.982+0000 7f27db568740 1 -- 172.21.15.90:0/2628105576 shutdown_connections 2024-04-02T14:45:30.979 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:30.982+0000 7f27db568740 1 -- 172.21.15.90:0/2628105576 wait complete. 2024-04-02T14:45:30.979 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:30.983+0000 7f27db568740 1 -- start start 2024-04-02T14:45:30.979 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:30.983+0000 7f27db568740 1 Processor -- start 2024-04-02T14:45:30.980 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:30.984+0000 7f27db568740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e18fbe7d10 0x55e18fbee060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:30.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:30.984+0000 7f27db568740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e18fbe89f0 0x55e18fbee5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:30.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:30.984+0000 7f27db568740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e18fbf1080 0x55e18fbeeae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:30.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:30.984+0000 7f27db568740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55e18fb486a0 con 0x55e18fbe7d10 2024-04-02T14:45:30.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:30.984+0000 7f27db568740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55e18faf4980 con 0x55e18fbe89f0 2024-04-02T14:45:30.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:30.984+0000 7f27db568740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55e18fb07cb0 con 0x55e18fbf1080 2024-04-02T14:45:30.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:30.984+0000 7f27db568740 10 client.0 _ll_register_callbacks cb 0x55e18fb076a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:45:30.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:30.985+0000 7f27d5400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e18fbe7d10 0x55e18fbee060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:30.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:30.985+0000 7f27d5400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e18fbe7d10 0x55e18fbee060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:42752/0 (socket says 172.21.15.90:42752) 2024-04-02T14:45:30.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:30.985+0000 7f27d5400640 1 -- 172.21.15.90:0/535211804 learned_addr learned my addr 172.21.15.90:0/535211804 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:45:30.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:30.985+0000 7f27d5400640 1 --2- 172.21.15.90:0/535211804 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e18fbf1080 0x55e18fbeeae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:30.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:30.985+0000 7f27d5e00640 1 --2- 172.21.15.90:0/535211804 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e18fbe89f0 0x55e18fbee5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:30.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:30.985+0000 7f27d5e00640 1 -- 172.21.15.90:0/535211804 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e18fbf1080 msgr2=0x55e18fbeeae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:45:30.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:30.985+0000 7f27d5e00640 1 --2- 172.21.15.90:0/535211804 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e18fbf1080 0x55e18fbeeae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:30.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:30.985+0000 7f27d5e00640 1 -- 172.21.15.90:0/535211804 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e18fbe7d10 msgr2=0x55e18fbee060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:45:30.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:30.986+0000 7f27d5e00640 1 --2- 172.21.15.90:0/535211804 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e18fbe7d10 0x55e18fbee060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:30.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:30.986+0000 7f27d5e00640 1 -- 172.21.15.90:0/535211804 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e18f92a7d0 con 0x55e18fbe89f0 2024-04-02T14:45:30.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:30.986+0000 7f27d5400640 1 --2- 172.21.15.90:0/535211804 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e18fbe7d10 0x55e18fbee060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:45:30.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:30.987+0000 7f27d5e00640 1 --2- 172.21.15.90:0/535211804 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e18fbe89f0 0x55e18fbee5a0 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f27c400ba10 tx=0x7f27c400bed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c29beab70a30d602 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:45:30.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:30.987+0000 7f27b3400640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:45:30.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:30.988+0000 7f27b3400640 1 -- 172.21.15.90:0/535211804 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27c4011440 con 0x55e18fbe89f0 2024-04-02T14:45:30.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:30.988+0000 7f27b3400640 1 -- 172.21.15.90:0/535211804 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8550+0+0 (secure 0 0 0) 0x7f27c40115e0 con 0x55e18fbe89f0 2024-04-02T14:45:30.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:30.988+0000 7f27db568740 1 -- 172.21.15.90:0/535211804 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f27c0047070 con 0x55e18fbe89f0 2024-04-02T14:45:30.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:30.989+0000 7f27b3400640 1 -- 172.21.15.90:0/535211804 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f27c40117e0 con 0x55e18fbe89f0 2024-04-02T14:45:30.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:30.989+0000 7f27db568740 10 client.28718 fetch_fsmap learned FSMap version 27 2024-04-02T14:45:30.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:30.989+0000 7f27db568740 1 -- 172.21.15.90:0/535211804 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55e18fbfff70 con 0x55e18fbe89f0 2024-04-02T14:45:30.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:30.989+0000 7f27b3400640 1 -- 172.21.15.90:0/535211804 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f27c0047070 con 0x55e18fbe89f0 2024-04-02T14:45:30.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:30.989+0000 7f27b3400640 1 -- 172.21.15.90:0/535211804 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f27c401bbe0 con 0x55e18fbe89f0 2024-04-02T14:45:30.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:30.989+0000 7f27db568740 10 client.28718 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:45:30.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:30.989+0000 7f27db568740 10 client.28718 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.c 2024-04-02T14:45:30.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:30.989+0000 7f27db568740 20 client.28718 populate_metadata read hostname 'smithi090' 2024-04-02T14:45:30.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:30.990+0000 7f27db568740 1 --2- 172.21.15.90:0/535211804 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x55e18fc01430 0x55e18fc03820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:30.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:30.990+0000 7f27db568740 4 client.28718 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:45:30.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:30.990+0000 7f27db568740 1 -- 172.21.15.90:0/535211804 --> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e18fb0c040 con 0x55e18fc01430 2024-04-02T14:45:30.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:30.990+0000 7f27d5400640 1 --2- 172.21.15.90:0/535211804 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x55e18fc01430 0x55e18fc03820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:30.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:30.991+0000 7f27d5400640 1 --2- 172.21.15.90:0/535211804 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x55e18fc01430 0x55e18fc03820 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=2c7aa2ce1cc32a78 server_cookie=6809160138e4923f in_seq=0 out_seq=0 2024-04-02T14:45:30.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:30.991+0000 7f27b3400640 10 client.28718 ms_handle_connect on v2:172.21.15.177:6836/4223414858 2024-04-02T14:45:30.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:30.991+0000 7f27b3400640 1 -- 172.21.15.90:0/535211804 <== mds.4 v2:172.21.15.177:6836/4223414858 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e18fb0c040 con 0x55e18fc01430 2024-04-02T14:45:30.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:30.991+0000 7f27b3400640 10 client.28718 handle_command_reply: tid=0 2024-04-02T14:45:30.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:30.992+0000 7f27db568740 1 client.28718 shutdown 2024-04-02T14:45:30.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:30.992+0000 7f27db568740 1 -- 172.21.15.90:0/535211804 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e18fbe89f0 msgr2=0x55e18fbee5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:45:30.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:30.992+0000 7f27db568740 1 --2- 172.21.15.90:0/535211804 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e18fbe89f0 0x55e18fbee5a0 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f27c400ba10 tx=0x7f27c400bed0 comp rx=0 tx=0).stop 2024-04-02T14:45:30.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:30.992+0000 7f27db568740 1 -- 172.21.15.90:0/535211804 shutdown_connections 2024-04-02T14:45:30.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:30.992+0000 7f27db568740 1 --2- 172.21.15.90:0/535211804 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e18fbf1080 0x55e18fbeeae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:30.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:30.992+0000 7f27db568740 1 --2- 172.21.15.90:0/535211804 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e18fbe89f0 0x55e18fbee5a0 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:30.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:30.992+0000 7f27db568740 1 --2- 172.21.15.90:0/535211804 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x55e18fc01430 0x55e18fc03820 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:30.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:30.992+0000 7f27db568740 1 --2- 172.21.15.90:0/535211804 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e18fbe7d10 0x55e18fbee060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:30.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:30.992+0000 7f27db568740 1 -- 172.21.15.90:0/535211804 >> 172.21.15.90:0/535211804 conn(0x55e18fbaea50 msgr2=0x55e18fbe44b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:45:30.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:30.992+0000 7f27b3400640 0 client.28718 ms_handle_reset on v2:172.21.15.177:6836/4223414858 2024-04-02T14:45:30.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:30.993+0000 7f27db568740 1 -- 172.21.15.90:0/535211804 shutdown_connections 2024-04-02T14:45:30.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:30.993+0000 7f27db568740 1 -- 172.21.15.90:0/535211804 wait complete. 2024-04-02T14:45:30.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:30.993+0000 7f27db568740 20 client.28718 trim_cache size 0 max 16384 2024-04-02T14:45:31.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.011+0000 7f27db568740 1 Processor -- start 2024-04-02T14:45:31.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.011+0000 7f27db568740 1 -- start start 2024-04-02T14:45:31.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.012+0000 7f27db568740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e18fbe7d10 0x55e18fbf0790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:31.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.012+0000 7f27db568740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e18fbe89f0 0x55e18fbfca90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:31.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.012+0000 7f27db568740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e18fbf1080 0x55e18fbe1720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:31.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.012+0000 7f27db568740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f27d01b3f40 con 0x55e18fbe89f0 2024-04-02T14:45:31.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.012+0000 7f27db568740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f27d01b5a80 con 0x55e18fbf1080 2024-04-02T14:45:31.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.012+0000 7f27db568740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f27d01b4610 con 0x55e18fbe7d10 2024-04-02T14:45:31.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.012+0000 7f27d5400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e18fbf1080 0x55e18fbe1720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:31.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.012+0000 7f27d5400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e18fbf1080 0x55e18fbe1720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:39546/0 (socket says 172.21.15.90:39546) 2024-04-02T14:45:31.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.012+0000 7f27d5400640 1 -- 172.21.15.90:0/3969408026 learned_addr learned my addr 172.21.15.90:0/3969408026 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:45:31.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.012+0000 7f27d5e00640 1 --2- 172.21.15.90:0/3969408026 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e18fbe89f0 0x55e18fbfca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:31.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.013+0000 7f27d5400640 1 --2- 172.21.15.90:0/3969408026 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e18fbe7d10 0x55e18fbf0790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:31.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.013+0000 7f27d5e00640 1 -- 172.21.15.90:0/3969408026 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e18fbe7d10 msgr2=0x55e18fbf0790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:45:31.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.013+0000 7f27d5e00640 1 --2- 172.21.15.90:0/3969408026 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e18fbe7d10 0x55e18fbf0790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:31.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.013+0000 7f27d5e00640 1 -- 172.21.15.90:0/3969408026 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e18fbf1080 msgr2=0x55e18fbe1720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:45:31.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.013+0000 7f27d5e00640 1 --2- 172.21.15.90:0/3969408026 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e18fbf1080 0x55e18fbe1720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:31.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.013+0000 7f27d5e00640 1 -- 172.21.15.90:0/3969408026 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e18fc03b50 con 0x55e18fbe89f0 2024-04-02T14:45:31.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.013+0000 7f27d5400640 1 --2- 172.21.15.90:0/3969408026 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e18fbf1080 0x55e18fbe1720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:45:31.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.014+0000 7f27d5e00640 1 --2- 172.21.15.90:0/3969408026 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e18fbe89f0 0x55e18fbfca90 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f27c40105f0 tx=0x7f27c4010900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ab80227c24c2928 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:45:31.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.014+0000 7f27be600640 1 -- 172.21.15.90:0/3969408026 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27c4027040 con 0x55e18fbe89f0 2024-04-02T14:45:31.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.014+0000 7f27be600640 1 -- 172.21.15.90:0/3969408026 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f27c400b5b0 con 0x55e18fbe89f0 2024-04-02T14:45:31.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.014+0000 7f27be600640 1 -- 172.21.15.90:0/3969408026 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27c4027040 con 0x55e18fbe89f0 2024-04-02T14:45:31.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.015+0000 7f27db568740 1 -- 172.21.15.90:0/3969408026 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e18fbe89f0 msgr2=0x55e18fbfca90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:45:31.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.015+0000 7f27db568740 1 --2- 172.21.15.90:0/3969408026 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e18fbe89f0 0x55e18fbfca90 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f27c40105f0 tx=0x7f27c4010900 comp rx=0 tx=0).stop 2024-04-02T14:45:31.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.016+0000 7f27db568740 1 -- 172.21.15.90:0/3969408026 shutdown_connections 2024-04-02T14:45:31.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.016+0000 7f27db568740 1 --2- 172.21.15.90:0/3969408026 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e18fbf1080 0x55e18fbe1720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:31.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.016+0000 7f27db568740 1 --2- 172.21.15.90:0/3969408026 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e18fbe89f0 0x55e18fbfca90 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:31.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.016+0000 7f27db568740 1 --2- 172.21.15.90:0/3969408026 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e18fbe7d10 0x55e18fbf0790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:31.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.016+0000 7f27db568740 1 -- 172.21.15.90:0/3969408026 >> 172.21.15.90:0/3969408026 conn(0x55e18fbaea50 msgr2=0x55e18fbe1010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:45:31.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.016+0000 7f27db568740 1 -- 172.21.15.90:0/3969408026 shutdown_connections 2024-04-02T14:45:31.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.016+0000 7f27db568740 1 -- 172.21.15.90:0/3969408026 wait complete. 2024-04-02T14:45:31.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.017+0000 7f27db568740 1 -- start start 2024-04-02T14:45:31.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.017+0000 7f27db568740 1 Processor -- start 2024-04-02T14:45:31.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.018+0000 7f27db568740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e18fbe7d10 0x55e18fbf1d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:31.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.018+0000 7f27db568740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e18fbe89f0 0x55e18fbf22d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:31.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.018+0000 7f27db568740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e18fbf1080 0x55e18fbf2810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:31.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.018+0000 7f27db568740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f27d01b53b0 con 0x55e18fbe89f0 2024-04-02T14:45:31.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.018+0000 7f27db568740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f27d01b4ce0 con 0x55e18fbf1080 2024-04-02T14:45:31.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.018+0000 7f27db568740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f27d01b1f40 con 0x55e18fbe7d10 2024-04-02T14:45:31.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.018+0000 7f27db568740 10 client.0 _ll_register_callbacks cb 0x7f27c40044f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:45:31.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.018+0000 7f27d5400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e18fbe7d10 0x55e18fbf1d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:31.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.018+0000 7f27d5e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e18fbe89f0 0x55e18fbf22d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:31.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.018+0000 7f27d5e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e18fbe89f0 0x55e18fbf22d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:42766/0 (socket says 172.21.15.90:42766) 2024-04-02T14:45:31.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.018+0000 7f27d5e00640 1 -- 172.21.15.90:0/2379793060 learned_addr learned my addr 172.21.15.90:0/2379793060 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:45:31.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.018+0000 7f27d5400640 1 --2- 172.21.15.90:0/2379793060 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e18fbf1080 0x55e18fbf2810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:31.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.019+0000 7f27d5e00640 1 -- 172.21.15.90:0/2379793060 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e18fbe7d10 msgr2=0x55e18fbf1d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:45:31.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.019+0000 7f27d5e00640 1 --2- 172.21.15.90:0/2379793060 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e18fbe7d10 0x55e18fbf1d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:31.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.019+0000 7f27d5e00640 1 -- 172.21.15.90:0/2379793060 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e18fbf1080 msgr2=0x55e18fbf2810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:45:31.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.019+0000 7f27d5e00640 1 --2- 172.21.15.90:0/2379793060 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e18fbf1080 0x55e18fbf2810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:31.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.019+0000 7f27d5e00640 1 -- 172.21.15.90:0/2379793060 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e18fbf72d0 con 0x55e18fbe89f0 2024-04-02T14:45:31.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.019+0000 7f27d5e00640 1 --2- 172.21.15.90:0/2379793060 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e18fbe89f0 0x55e18fbf22d0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f27c4010480 tx=0x7f27c4005690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad7e6f24d96eced9 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:45:31.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.019+0000 7f27b3400640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-02T14:45:31.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.019+0000 7f27b3400640 1 -- 172.21.15.90:0/2379793060 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27c4027040 con 0x55e18fbe89f0 2024-04-02T14:45:31.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.020+0000 7f27b3400640 1 -- 172.21.15.90:0/2379793060 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8550+0+0 (secure 0 0 0) 0x7f27c4011b90 con 0x55e18fbe89f0 2024-04-02T14:45:31.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.020+0000 7f27db568740 1 -- 172.21.15.90:0/2379793060 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f27c400b5b0 con 0x55e18fbe89f0 2024-04-02T14:45:31.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.020+0000 7f27b3400640 1 -- 172.21.15.90:0/2379793060 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f27c4011d90 con 0x55e18fbe89f0 2024-04-02T14:45:31.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.020+0000 7f27d5400640 1 --2- 172.21.15.90:0/2379793060 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e18fbf1080 0x55e18fbf2810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:45:31.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.020+0000 7f27db568740 10 client.17775 fetch_fsmap learned FSMap version 27 2024-04-02T14:45:31.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.020+0000 7f27db568740 1 -- 172.21.15.90:0/2379793060 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55e18fc67cd0 con 0x55e18fbe89f0 2024-04-02T14:45:31.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.021+0000 7f27b3400640 1 -- 172.21.15.90:0/2379793060 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f27c400b5b0 con 0x55e18fbe89f0 2024-04-02T14:45:31.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.021+0000 7f27b3400640 1 -- 172.21.15.90:0/2379793060 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f27c402a070 con 0x55e18fbe89f0 2024-04-02T14:45:31.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.021+0000 7f27db568740 10 client.17775 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:45:31.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.021+0000 7f27db568740 10 client.17775 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.c 2024-04-02T14:45:31.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.021+0000 7f27db568740 20 client.17775 populate_metadata read hostname 'smithi090' 2024-04-02T14:45:31.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.021+0000 7f27db568740 1 --2- 172.21.15.90:0/2379793060 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x55e18fc683b0 0x55e18fc6a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:31.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.021+0000 7f27db568740 4 client.17775 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-02T14:45:31.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.021+0000 7f27db568740 1 -- 172.21.15.90:0/2379793060 --> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55e18fb59c30 con 0x55e18fc683b0 2024-04-02T14:45:31.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.022+0000 7f27d5400640 1 --2- 172.21.15.90:0/2379793060 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x55e18fc683b0 0x55e18fc6a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:31.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.023+0000 7f27d5400640 1 --2- 172.21.15.90:0/2379793060 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x55e18fc683b0 0x55e18fc6a7d0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=1350213f162d2eac server_cookie=e59a9ced7d02a2ce in_seq=0 out_seq=0 2024-04-02T14:45:31.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.024+0000 7f27b3400640 10 client.17775 ms_handle_connect on v2:172.21.15.177:6836/4223414858 2024-04-02T14:45:31.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.024+0000 7f27b3400640 1 -- 172.21.15.90:0/2379793060 <== mds.4 v2:172.21.15.177:6836/4223414858 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55e18fb59c30 con 0x55e18fc683b0 2024-04-02T14:45:31.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.024+0000 7f27b3400640 10 client.17775 handle_command_reply: tid=0 2024-04-02T14:45:31.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.024+0000 7f27db568740 1 client.17775 shutdown 2024-04-02T14:45:31.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.025+0000 7f27db568740 1 -- 172.21.15.90:0/2379793060 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e18fbe89f0 msgr2=0x55e18fbf22d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:45:31.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.025+0000 7f27db568740 1 --2- 172.21.15.90:0/2379793060 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e18fbe89f0 0x55e18fbf22d0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f27c4010480 tx=0x7f27c4005690 comp rx=0 tx=0).stop 2024-04-02T14:45:31.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.025+0000 7f27db568740 1 -- 172.21.15.90:0/2379793060 shutdown_connections 2024-04-02T14:45:31.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.025+0000 7f27db568740 1 --2- 172.21.15.90:0/2379793060 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e18fbf1080 0x55e18fbf2810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:31.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.025+0000 7f27db568740 1 --2- 172.21.15.90:0/2379793060 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x55e18fc683b0 0x55e18fc6a7d0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:31.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.025+0000 7f27db568740 1 --2- 172.21.15.90:0/2379793060 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e18fbe89f0 0x55e18fbf22d0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:31.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.025+0000 7f27db568740 1 --2- 172.21.15.90:0/2379793060 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e18fbe7d10 0x55e18fbf1d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:31.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.025+0000 7f27db568740 1 -- 172.21.15.90:0/2379793060 >> 172.21.15.90:0/2379793060 conn(0x55e18fbaea50 msgr2=0x55e18fbe18a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:45:31.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.025+0000 7f27b3400640 0 client.17775 ms_handle_reset on v2:172.21.15.177:6836/4223414858 2024-04-02T14:45:31.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.025+0000 7f27db568740 1 -- 172.21.15.90:0/2379793060 shutdown_connections 2024-04-02T14:45:31.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.025+0000 7f27db568740 1 -- 172.21.15.90:0/2379793060 wait complete. 2024-04-02T14:45:31.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.026+0000 7f27db568740 20 client.17775 trim_cache size 0 max 16384 2024-04-02T14:45:31.023 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-02T14:45:31.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.027+0000 7f27d8800640 1 -- 172.21.15.90:0/1880970598 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f27b80a3ef0 msgr2=0x7f27b80a63b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:45:31.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.027+0000 7f27d8800640 1 --2- 172.21.15.90:0/1880970598 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f27b80a3ef0 0x7f27b80a63b0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f27cc09c400 tx=0x7f27cc0054b0 comp rx=0 tx=0).stop 2024-04-02T14:45:31.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.027+0000 7f27d8800640 1 -- 172.21.15.90:0/1880970598 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f27d010e4a0 msgr2=0x7f27d0103650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:45:31.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.027+0000 7f27d8800640 1 --2- 172.21.15.90:0/1880970598 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f27d010e4a0 0x7f27d0103650 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f27cc005fd0 tx=0x7f27cc00edd0 comp rx=0 tx=0).stop 2024-04-02T14:45:31.025 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.027+0000 7f27d8800640 1 -- 172.21.15.90:0/1880970598 shutdown_connections 2024-04-02T14:45:31.025 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.027+0000 7f27d8800640 1 --2- 172.21.15.90:0/1880970598 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f27b80a3ef0 0x7f27b80a63b0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:31.025 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.027+0000 7f27d8800640 1 --2- 172.21.15.90:0/1880970598 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f27d010e4a0 0x7f27d0103650 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:31.025 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.027+0000 7f27d8800640 1 --2- 172.21.15.90:0/1880970598 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f27d0069600 0x7f27d0103110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:31.025 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.027+0000 7f27d8800640 1 --2- 172.21.15.90:0/1880970598 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f27d0068ce0 0x7f27d0102bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:31.025 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.027+0000 7f27d8800640 1 -- 172.21.15.90:0/1880970598 >> 172.21.15.90:0/1880970598 conn(0x7f27d0077b00 msgr2=0x7f27d010fa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:45:31.025 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.028+0000 7f27d8800640 1 -- 172.21.15.90:0/1880970598 shutdown_connections 2024-04-02T14:45:31.025 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.028+0000 7f27d8800640 1 -- 172.21.15.90:0/1880970598 wait complete. 2024-04-02T14:45:31.054 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-04-02T14:45:31.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.361+0000 7f929fa00640 1 -- 172.21.15.90:0/1546234834 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9298100ce0 msgr2=0x7f9298101160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:45:31.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.361+0000 7f929fa00640 1 --2- 172.21.15.90:0/1546234834 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9298100ce0 0x7f9298101160 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f9288009900 tx=0x7f928802bc40 comp rx=0 tx=0).stop 2024-04-02T14:45:31.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.362+0000 7f929fa00640 1 -- 172.21.15.90:0/1546234834 shutdown_connections 2024-04-02T14:45:31.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.362+0000 7f929fa00640 1 --2- 172.21.15.90:0/1546234834 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f92981016a0 0x7f929806a070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:31.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.362+0000 7f929fa00640 1 --2- 172.21.15.90:0/1546234834 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9298100ce0 0x7f9298101160 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:31.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.362+0000 7f929fa00640 1 --2- 172.21.15.90:0/1546234834 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9298106a30 0x7f9298106e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:31.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.362+0000 7f929fa00640 1 -- 172.21.15.90:0/1546234834 >> 172.21.15.90:0/1546234834 conn(0x7f92980779e0 msgr2=0x7f9298077df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:45:31.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.362+0000 7f929fa00640 1 -- 172.21.15.90:0/1546234834 shutdown_connections 2024-04-02T14:45:31.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.362+0000 7f929fa00640 1 -- 172.21.15.90:0/1546234834 wait complete. 2024-04-02T14:45:31.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.364+0000 7f929fa00640 1 Processor -- start 2024-04-02T14:45:31.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.365+0000 7f929fa00640 1 -- start start 2024-04-02T14:45:31.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.365+0000 7f929fa00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9298100ce0 0x7f929819ae50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:31.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.365+0000 7f929fa00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f92981016a0 0x7f9298105090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:31.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.365+0000 7f929fa00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9298106a30 0x7f92981055d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:31.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.365+0000 7f929fa00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f9298111ed0 con 0x7f9298106a30 2024-04-02T14:45:31.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.365+0000 7f929fa00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f9298111d50 con 0x7f92981016a0 2024-04-02T14:45:31.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.365+0000 7f929fa00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f9298112050 con 0x7f9298100ce0 2024-04-02T14:45:31.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.365+0000 7f9297e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f92981016a0 0x7f9298105090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:31.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.365+0000 7f9297e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f92981016a0 0x7f9298105090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:39562/0 (socket says 172.21.15.90:39562) 2024-04-02T14:45:31.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.365+0000 7f9297e00640 1 -- 172.21.15.90:0/907404353 learned_addr learned my addr 172.21.15.90:0/907404353 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:45:31.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.365+0000 7f929d000640 1 --2- 172.21.15.90:0/907404353 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9298100ce0 0x7f929819ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:31.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.366+0000 7f9297e00640 1 -- 172.21.15.90:0/907404353 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9298100ce0 msgr2=0x7f929819ae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:45:31.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.366+0000 7f9297e00640 1 --2- 172.21.15.90:0/907404353 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9298100ce0 0x7f929819ae50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:31.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.366+0000 7f9297e00640 1 -- 172.21.15.90:0/907404353 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9298106a30 msgr2=0x7f92981055d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:45:31.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.366+0000 7f9297e00640 1 --2- 172.21.15.90:0/907404353 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9298106a30 0x7f92981055d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:31.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.366+0000 7f9297e00640 1 -- 172.21.15.90:0/907404353 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9298105cb0 con 0x7f92981016a0 2024-04-02T14:45:31.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.366+0000 7f929d000640 1 --2- 172.21.15.90:0/907404353 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9298100ce0 0x7f929819ae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:45:31.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.366+0000 7f9297e00640 1 --2- 172.21.15.90:0/907404353 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f92981016a0 0x7f9298105090 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f928802efd0 tx=0x7f9288004390 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:45:31.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.366+0000 7f9295600640 1 -- 172.21.15.90:0/907404353 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9288041070 con 0x7f92981016a0 2024-04-02T14:45:31.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.367+0000 7f929fa00640 1 -- 172.21.15.90:0/907404353 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f92981aafa0 con 0x7f92981016a0 2024-04-02T14:45:31.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.367+0000 7f9295600640 1 -- 172.21.15.90:0/907404353 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f928802cca0 con 0x7f92981016a0 2024-04-02T14:45:31.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.367+0000 7f9295600640 1 -- 172.21.15.90:0/907404353 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9288003d60 con 0x7f92981016a0 2024-04-02T14:45:31.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.367+0000 7f929fa00640 1 -- 172.21.15.90:0/907404353 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f92981ab350 con 0x7f92981016a0 2024-04-02T14:45:31.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.371+0000 7f9295600640 1 -- 172.21.15.90:0/907404353 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f9288044070 con 0x7f92981016a0 2024-04-02T14:45:31.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.372+0000 7f92a267e740 1 Processor -- start 2024-04-02T14:45:31.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.372+0000 7f92a267e740 1 -- start start 2024-04-02T14:45:31.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.372+0000 7f92a267e740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e25d4d09f0 0x55e25d4cf740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:31.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.373+0000 7f92a267e740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e25d4cfd10 0x55e25d4d0190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:31.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.373+0000 7f92a267e740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e25d4d9080 0x55e25d4db470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:31.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.373+0000 7f92a267e740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55e25d410400 con 0x55e25d4d09f0 2024-04-02T14:45:31.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.373+0000 7f92a267e740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55e25d3f4f10 con 0x55e25d4d9080 2024-04-02T14:45:31.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.373+0000 7f92a267e740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55e25d4346f0 con 0x55e25d4cfd10 2024-04-02T14:45:31.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.373+0000 7f9297e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e25d4d9080 0x55e25d4db470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:31.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.373+0000 7f9297e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e25d4d9080 0x55e25d4db470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:39576/0 (socket says 172.21.15.90:39576) 2024-04-02T14:45:31.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.373+0000 7f9297e00640 1 -- 172.21.15.90:0/174096120 learned_addr learned my addr 172.21.15.90:0/174096120 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:45:31.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.374+0000 7f929da00640 1 --2- 172.21.15.90:0/174096120 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e25d4d09f0 0x55e25d4cf740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:31.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.375+0000 7f9297e00640 1 -- 172.21.15.90:0/174096120 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e25d4cfd10 msgr2=0x55e25d4d0190 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-02T14:45:31.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.375+0000 7f9297e00640 1 --2- 172.21.15.90:0/174096120 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e25d4cfd10 0x55e25d4d0190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:31.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.375+0000 7f9297e00640 1 -- 172.21.15.90:0/174096120 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e25d4d09f0 msgr2=0x55e25d4cf740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:45:31.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.375+0000 7f9297e00640 1 --2- 172.21.15.90:0/174096120 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e25d4d09f0 0x55e25d4cf740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:31.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.375+0000 7f9297e00640 1 -- 172.21.15.90:0/174096120 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e25d1483f0 con 0x55e25d4d9080 2024-04-02T14:45:31.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.376+0000 7f9297e00640 1 --2- 172.21.15.90:0/174096120 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e25d4d9080 0x55e25d4db470 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f92880efa10 tx=0x7f92880fa3d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=28d367327aeb5946 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:45:31.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.376+0000 7f9295600640 1 --2- 172.21.15.90:0/907404353 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f92800a3de0 0x7f92800a62a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:31.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.377+0000 7f9295600640 1 -- 172.21.15.90:0/907404353 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 8550+0+0 (secure 0 0 0) 0x7f92880f2230 con 0x7f92981016a0 2024-04-02T14:45:31.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.377+0000 7f929da00640 1 --2- 172.21.15.90:0/907404353 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f92800a3de0 0x7f92800a62a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:31.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.377+0000 7f9286000640 1 -- 172.21.15.90:0/174096120 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92880f42d0 con 0x55e25d4d9080 2024-04-02T14:45:31.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.377+0000 7f9286000640 1 -- 172.21.15.90:0/174096120 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f92880ff030 con 0x55e25d4d9080 2024-04-02T14:45:31.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.378+0000 7f9286000640 1 -- 172.21.15.90:0/174096120 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92880f42d0 con 0x55e25d4d9080 2024-04-02T14:45:31.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.378+0000 7f92a267e740 1 -- 172.21.15.90:0/174096120 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e25d4d9080 msgr2=0x55e25d4db470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:45:31.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.378+0000 7f92a267e740 1 --2- 172.21.15.90:0/174096120 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e25d4d9080 0x55e25d4db470 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f92880efa10 tx=0x7f92880fa3d0 comp rx=0 tx=0).stop 2024-04-02T14:45:31.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.379+0000 7f929da00640 1 --2- 172.21.15.90:0/907404353 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f92800a3de0 0x7f92800a62a0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f9290009270 tx=0x7f92900095e0 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:45:31.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.379+0000 7f92a267e740 1 -- 172.21.15.90:0/174096120 shutdown_connections 2024-04-02T14:45:31.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.379+0000 7f92a267e740 1 --2- 172.21.15.90:0/174096120 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e25d4d9080 0x55e25d4db470 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:31.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.379+0000 7f92a267e740 1 --2- 172.21.15.90:0/174096120 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e25d4cfd10 0x55e25d4d0190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:31.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.379+0000 7f92a267e740 1 --2- 172.21.15.90:0/174096120 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e25d4d09f0 0x55e25d4cf740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:31.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.379+0000 7f92a267e740 1 -- 172.21.15.90:0/174096120 >> 172.21.15.90:0/174096120 conn(0x55e25d496a50 msgr2=0x55e25d4ca760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:45:31.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.379+0000 7f92a267e740 1 -- 172.21.15.90:0/174096120 shutdown_connections 2024-04-02T14:45:31.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.379+0000 7f92a267e740 1 -- 172.21.15.90:0/174096120 wait complete. 2024-04-02T14:45:31.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.380+0000 7f92a267e740 1 -- start start 2024-04-02T14:45:31.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.380+0000 7f92a267e740 1 Processor -- start 2024-04-02T14:45:31.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.380+0000 7f92a267e740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e25d4cfd10 0x55e25d4d5fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:31.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.381+0000 7f92a267e740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e25d4d09f0 0x55e25d4d64f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:31.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.381+0000 7f92a267e740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e25d4d9080 0x55e25d4d6a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:31.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.381+0000 7f92a267e740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55e25d4306a0 con 0x55e25d4cfd10 2024-04-02T14:45:31.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.381+0000 7f92a267e740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55e25d3dc980 con 0x55e25d4d09f0 2024-04-02T14:45:31.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.381+0000 7f92a267e740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55e25d3efcb0 con 0x55e25d4d9080 2024-04-02T14:45:31.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.381+0000 7f92a267e740 10 client.0 _ll_register_callbacks cb 0x55e25d3ef6a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:45:31.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.381+0000 7f9297e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e25d4d09f0 0x55e25d4d64f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:31.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.381+0000 7f929da00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e25d4d9080 0x55e25d4d6a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:31.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.381+0000 7f9297e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e25d4d09f0 0x55e25d4d64f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:39590/0 (socket says 172.21.15.90:39590) 2024-04-02T14:45:31.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.381+0000 7f9297e00640 1 -- 172.21.15.90:0/1784651318 learned_addr learned my addr 172.21.15.90:0/1784651318 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:45:31.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.381+0000 7f929d000640 1 --2- 172.21.15.90:0/1784651318 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e25d4cfd10 0x55e25d4d5fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:31.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.382+0000 7f929da00640 1 -- 172.21.15.90:0/1784651318 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e25d4d09f0 msgr2=0x55e25d4d64f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:45:31.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.382+0000 7f929da00640 1 --2- 172.21.15.90:0/1784651318 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e25d4d09f0 0x55e25d4d64f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:31.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.382+0000 7f929da00640 1 -- 172.21.15.90:0/1784651318 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e25d4cfd10 msgr2=0x55e25d4d5fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:45:31.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.382+0000 7f929da00640 1 --2- 172.21.15.90:0/1784651318 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e25d4cfd10 0x55e25d4d5fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:31.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.382+0000 7f929da00640 1 -- 172.21.15.90:0/1784651318 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e25d2127d0 con 0x55e25d4d9080 2024-04-02T14:45:31.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.382+0000 7f929da00640 1 --2- 172.21.15.90:0/1784651318 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e25d4d9080 0x55e25d4d6a30 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f9290012780 tx=0x7f9290004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=20ab5f9c23771be7 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:45:31.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.382+0000 7f927aa00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:45:31.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.382+0000 7f927aa00640 1 -- 172.21.15.90:0/1784651318 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9290003b70 con 0x55e25d4d9080 2024-04-02T14:45:31.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.382+0000 7f927aa00640 1 -- 172.21.15.90:0/1784651318 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8550+0+0 (secure 0 0 0) 0x7f9290003d10 con 0x55e25d4d9080 2024-04-02T14:45:31.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.383+0000 7f92a267e740 1 -- 172.21.15.90:0/1784651318 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f92880ff030 con 0x55e25d4d9080 2024-04-02T14:45:31.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.383+0000 7f927aa00640 1 -- 172.21.15.90:0/1784651318 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f929001d8c0 con 0x55e25d4d9080 2024-04-02T14:45:31.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.383+0000 7f92a267e740 10 client.28570 fetch_fsmap learned FSMap version 27 2024-04-02T14:45:31.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.383+0000 7f92a267e740 1 -- 172.21.15.90:0/1784651318 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55e25d4e7c10 con 0x55e25d4d9080 2024-04-02T14:45:31.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.384+0000 7f927aa00640 1 -- 172.21.15.90:0/1784651318 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f92880ff030 con 0x55e25d4d9080 2024-04-02T14:45:31.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.384+0000 7f927aa00640 1 -- 172.21.15.90:0/1784651318 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f929000f040 con 0x55e25d4d9080 2024-04-02T14:45:31.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.384+0000 7f92a267e740 10 client.28570 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:45:31.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.384+0000 7f92a267e740 10 client.28570 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-04-02T14:45:31.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.384+0000 7f92a267e740 20 client.28570 populate_metadata read hostname 'smithi090' 2024-04-02T14:45:31.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.384+0000 7f92a267e740 1 --2- 172.21.15.90:0/1784651318 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x55e25d4e9170 0x55e25d4eb560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:31.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.384+0000 7f92a267e740 4 client.28570 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:45:31.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.384+0000 7f92a267e740 1 -- 172.21.15.90:0/1784651318 --> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e25d3f4040 con 0x55e25d4e9170 2024-04-02T14:45:31.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.385+0000 7f929d000640 1 --2- 172.21.15.90:0/1784651318 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x55e25d4e9170 0x55e25d4eb560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:31.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.386+0000 7f929d000640 1 --2- 172.21.15.90:0/1784651318 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x55e25d4e9170 0x55e25d4eb560 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=593832718c100b6c server_cookie=2eab33a718ec168 in_seq=0 out_seq=0 2024-04-02T14:45:31.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.386+0000 7f927aa00640 10 client.28570 ms_handle_connect on v2:172.21.15.92:6836/1270635939 2024-04-02T14:45:31.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.387+0000 7f927aa00640 1 -- 172.21.15.90:0/1784651318 <== mds.3 v2:172.21.15.92:6836/1270635939 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e25d3f4040 con 0x55e25d4e9170 2024-04-02T14:45:31.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.387+0000 7f927aa00640 10 client.28570 handle_command_reply: tid=0 2024-04-02T14:45:31.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.387+0000 7f92a267e740 1 client.28570 shutdown 2024-04-02T14:45:31.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.387+0000 7f92a267e740 1 -- 172.21.15.90:0/1784651318 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e25d4d9080 msgr2=0x55e25d4d6a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:45:31.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.387+0000 7f92a267e740 1 --2- 172.21.15.90:0/1784651318 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e25d4d9080 0x55e25d4d6a30 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f9290012780 tx=0x7f9290004a40 comp rx=0 tx=0).stop 2024-04-02T14:45:31.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.388+0000 7f92a267e740 1 -- 172.21.15.90:0/1784651318 shutdown_connections 2024-04-02T14:45:31.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.388+0000 7f92a267e740 1 --2- 172.21.15.90:0/1784651318 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x55e25d4e9170 0x55e25d4eb560 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:31.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.388+0000 7f927aa00640 0 client.28570 ms_handle_reset on v2:172.21.15.92:6836/1270635939 2024-04-02T14:45:31.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.388+0000 7f92a267e740 1 --2- 172.21.15.90:0/1784651318 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e25d4d9080 0x55e25d4d6a30 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:31.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.388+0000 7f92a267e740 1 --2- 172.21.15.90:0/1784651318 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e25d4d09f0 0x55e25d4d64f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:31.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.388+0000 7f92a267e740 1 --2- 172.21.15.90:0/1784651318 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e25d4cfd10 0x55e25d4d5fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:31.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.388+0000 7f92a267e740 1 -- 172.21.15.90:0/1784651318 >> 172.21.15.90:0/1784651318 conn(0x55e25d496a50 msgr2=0x55e25d4cc4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:45:31.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.388+0000 7f92a267e740 1 -- 172.21.15.90:0/1784651318 shutdown_connections 2024-04-02T14:45:31.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.388+0000 7f92a267e740 1 -- 172.21.15.90:0/1784651318 wait complete. 2024-04-02T14:45:31.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.389+0000 7f92a267e740 20 client.28570 trim_cache size 0 max 16384 2024-04-02T14:45:31.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.412+0000 7f92a267e740 1 Processor -- start 2024-04-02T14:45:31.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.413+0000 7f92a267e740 1 -- start start 2024-04-02T14:45:31.410 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.413+0000 7f92a267e740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e25d4cfd10 0x55e25d4e4870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:31.410 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.413+0000 7f92a267e740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e25d4d09f0 0x55e25d4d5110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:31.410 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.413+0000 7f92a267e740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e25d4d9080 0x55e25d4d6850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:31.410 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.413+0000 7f92a267e740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f92981b3e60 con 0x55e25d4d09f0 2024-04-02T14:45:31.410 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.413+0000 7f92a267e740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f92981b59a0 con 0x55e25d4cfd10 2024-04-02T14:45:31.410 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.413+0000 7f92a267e740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f92981b4530 con 0x55e25d4d9080 2024-04-02T14:45:31.410 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.413+0000 7f929d000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e25d4cfd10 0x55e25d4e4870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:31.410 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.413+0000 7f929da00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e25d4d9080 0x55e25d4d6850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:31.410 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.413+0000 7f929d000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e25d4cfd10 0x55e25d4e4870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:39602/0 (socket says 172.21.15.90:39602) 2024-04-02T14:45:31.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.413+0000 7f929d000640 1 -- 172.21.15.90:0/1833829601 learned_addr learned my addr 172.21.15.90:0/1833829601 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:45:31.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.413+0000 7f9297e00640 1 --2- 172.21.15.90:0/1833829601 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e25d4d09f0 0x55e25d4d5110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:31.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.415+0000 7f929da00640 1 -- 172.21.15.90:0/1833829601 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e25d4cfd10 msgr2=0x55e25d4e4870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:45:31.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.415+0000 7f929da00640 1 --2- 172.21.15.90:0/1833829601 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e25d4cfd10 0x55e25d4e4870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:31.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.415+0000 7f929da00640 1 -- 172.21.15.90:0/1833829601 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e25d4d09f0 msgr2=0x55e25d4d5110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:45:31.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.415+0000 7f929da00640 1 --2- 172.21.15.90:0/1833829601 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e25d4d09f0 0x55e25d4d5110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:31.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.415+0000 7f929da00640 1 -- 172.21.15.90:0/1833829601 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e25d4ced40 con 0x55e25d4d9080 2024-04-02T14:45:31.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.415+0000 7f929d000640 1 --2- 172.21.15.90:0/1833829601 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e25d4cfd10 0x55e25d4e4870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:45:31.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.415+0000 7f9297e00640 1 --2- 172.21.15.90:0/1833829601 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e25d4d09f0 0x55e25d4d5110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:45:31.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.415+0000 7f929da00640 1 --2- 172.21.15.90:0/1833829601 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e25d4d9080 0x55e25d4d6850 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f9290031470 tx=0x7f9290005bf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e623a16ac1e9e67c server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:45:31.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.415+0000 7f9285600640 1 -- 172.21.15.90:0/1833829601 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9290036070 con 0x55e25d4d9080 2024-04-02T14:45:31.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.416+0000 7f9285600640 1 -- 172.21.15.90:0/1833829601 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9290004d20 con 0x55e25d4d9080 2024-04-02T14:45:31.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.416+0000 7f9285600640 1 -- 172.21.15.90:0/1833829601 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9290031da0 con 0x55e25d4d9080 2024-04-02T14:45:31.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.417+0000 7f92a267e740 1 -- 172.21.15.90:0/1833829601 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e25d4d9080 msgr2=0x55e25d4d6850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:45:31.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.417+0000 7f92a267e740 1 --2- 172.21.15.90:0/1833829601 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e25d4d9080 0x55e25d4d6850 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f9290031470 tx=0x7f9290005bf0 comp rx=0 tx=0).stop 2024-04-02T14:45:31.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.417+0000 7f92a267e740 1 -- 172.21.15.90:0/1833829601 shutdown_connections 2024-04-02T14:45:31.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.417+0000 7f92a267e740 1 --2- 172.21.15.90:0/1833829601 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e25d4d9080 0x55e25d4d6850 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:31.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.417+0000 7f92a267e740 1 --2- 172.21.15.90:0/1833829601 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e25d4d09f0 0x55e25d4d5110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:31.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.417+0000 7f92a267e740 1 --2- 172.21.15.90:0/1833829601 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e25d4cfd10 0x55e25d4e4870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:31.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.417+0000 7f92a267e740 1 -- 172.21.15.90:0/1833829601 >> 172.21.15.90:0/1833829601 conn(0x55e25d496a50 msgr2=0x55e25d4c9010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:45:31.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.417+0000 7f92a267e740 1 -- 172.21.15.90:0/1833829601 shutdown_connections 2024-04-02T14:45:31.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.417+0000 7f92a267e740 1 -- 172.21.15.90:0/1833829601 wait complete. 2024-04-02T14:45:31.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.418+0000 7f92a267e740 1 -- start start 2024-04-02T14:45:31.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.418+0000 7f92a267e740 1 Processor -- start 2024-04-02T14:45:31.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.418+0000 7f92a267e740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e25d4cfd10 0x55e25d4e0bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:31.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.418+0000 7f92a267e740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e25d4d09f0 0x55e25d4e1100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:31.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.418+0000 7f92a267e740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e25d4d9080 0x55e25d4dc090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:31.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.418+0000 7f92a267e740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f92981b52d0 con 0x55e25d4d09f0 2024-04-02T14:45:31.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.418+0000 7f92a267e740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f92981b4c00 con 0x55e25d4cfd10 2024-04-02T14:45:31.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.418+0000 7f92a267e740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f92981b1e60 con 0x55e25d4d9080 2024-04-02T14:45:31.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.418+0000 7f92a267e740 10 client.0 _ll_register_callbacks cb 0x7f9290012450 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:45:31.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.419+0000 7f9297e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e25d4d09f0 0x55e25d4e1100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:31.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.419+0000 7f929d000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e25d4cfd10 0x55e25d4e0bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:31.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.419+0000 7f9297e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e25d4d09f0 0x55e25d4e1100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:42830/0 (socket says 172.21.15.90:42830) 2024-04-02T14:45:31.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.419+0000 7f9297e00640 1 -- 172.21.15.90:0/2041589582 learned_addr learned my addr 172.21.15.90:0/2041589582 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:45:31.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.419+0000 7f929da00640 1 --2- 172.21.15.90:0/2041589582 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e25d4d9080 0x55e25d4dc090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:31.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.419+0000 7f9297e00640 1 -- 172.21.15.90:0/2041589582 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e25d4d9080 msgr2=0x55e25d4dc090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:45:31.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.419+0000 7f9297e00640 1 --2- 172.21.15.90:0/2041589582 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e25d4d9080 0x55e25d4dc090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:31.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.419+0000 7f9297e00640 1 -- 172.21.15.90:0/2041589582 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e25d4cfd10 msgr2=0x55e25d4e0bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:45:31.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.419+0000 7f9297e00640 1 --2- 172.21.15.90:0/2041589582 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e25d4cfd10 0x55e25d4e0bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:31.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.419+0000 7f9297e00640 1 -- 172.21.15.90:0/2041589582 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e25d4dc890 con 0x55e25d4d09f0 2024-04-02T14:45:31.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.420+0000 7f9297e00640 1 --2- 172.21.15.90:0/2041589582 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e25d4d09f0 0x55e25d4e1100 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f9288000c00 tx=0x7f928802bc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3387f758f8e0c2e2 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:45:31.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.420+0000 7f927aa00640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-02T14:45:31.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.421+0000 7f927aa00640 1 -- 172.21.15.90:0/2041589582 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f928804b040 con 0x55e25d4d09f0 2024-04-02T14:45:31.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.421+0000 7f927aa00640 1 -- 172.21.15.90:0/2041589582 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8550+0+0 (secure 0 0 0) 0x7f92880bc5c0 con 0x55e25d4d09f0 2024-04-02T14:45:31.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.421+0000 7f92a267e740 1 -- 172.21.15.90:0/2041589582 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9290004d20 con 0x55e25d4d09f0 2024-04-02T14:45:31.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.422+0000 7f927aa00640 1 -- 172.21.15.90:0/2041589582 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f92880ba560 con 0x55e25d4d09f0 2024-04-02T14:45:31.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.422+0000 7f92a267e740 10 client.17787 fetch_fsmap learned FSMap version 27 2024-04-02T14:45:31.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.422+0000 7f92a267e740 1 -- 172.21.15.90:0/2041589582 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55e25d54f1b0 con 0x55e25d4d09f0 2024-04-02T14:45:31.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.422+0000 7f927aa00640 1 -- 172.21.15.90:0/2041589582 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9290004d20 con 0x55e25d4d09f0 2024-04-02T14:45:31.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.422+0000 7f927aa00640 1 -- 172.21.15.90:0/2041589582 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f92880efbe0 con 0x55e25d4d09f0 2024-04-02T14:45:31.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.423+0000 7f92a267e740 10 client.17787 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:45:31.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.423+0000 7f92a267e740 10 client.17787 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-04-02T14:45:31.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.423+0000 7f92a267e740 20 client.17787 populate_metadata read hostname 'smithi090' 2024-04-02T14:45:31.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.423+0000 7f92a267e740 1 --2- 172.21.15.90:0/2041589582 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x55e25d4e9170 0x55e25d551600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:31.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.423+0000 7f92a267e740 4 client.17787 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-02T14:45:31.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.423+0000 7f92a267e740 1 -- 172.21.15.90:0/2041589582 --> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55e25d441c30 con 0x55e25d4e9170 2024-04-02T14:45:31.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.423+0000 7f929d000640 1 --2- 172.21.15.90:0/2041589582 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x55e25d4e9170 0x55e25d551600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:31.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.423+0000 7f927aa00640 10 client.17787 ms_handle_connect on v2:172.21.15.92:6836/1270635939 2024-04-02T14:45:31.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.423+0000 7f929d000640 1 --2- 172.21.15.90:0/2041589582 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x55e25d4e9170 0x55e25d551600 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=d8362c3ab84e6853 server_cookie=fa62f07781aa7c03 in_seq=0 out_seq=0 2024-04-02T14:45:31.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.424+0000 7f927aa00640 1 -- 172.21.15.90:0/2041589582 <== mds.3 v2:172.21.15.92:6836/1270635939 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55e25d441c30 con 0x55e25d4e9170 2024-04-02T14:45:31.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.424+0000 7f927aa00640 10 client.17787 handle_command_reply: tid=0 2024-04-02T14:45:31.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.424+0000 7f92a267e740 1 client.17787 shutdown 2024-04-02T14:45:31.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.424+0000 7f92a267e740 1 -- 172.21.15.90:0/2041589582 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e25d4d09f0 msgr2=0x55e25d4e1100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:45:31.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.424+0000 7f92a267e740 1 --2- 172.21.15.90:0/2041589582 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e25d4d09f0 0x55e25d4e1100 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f9288000c00 tx=0x7f928802bc40 comp rx=0 tx=0).stop 2024-04-02T14:45:31.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.424+0000 7f92a267e740 1 -- 172.21.15.90:0/2041589582 shutdown_connections 2024-04-02T14:45:31.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.424+0000 7f92a267e740 1 --2- 172.21.15.90:0/2041589582 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x55e25d4e9170 0x55e25d551600 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:31.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.425+0000 7f92a267e740 1 --2- 172.21.15.90:0/2041589582 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e25d4d9080 0x55e25d4dc090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:31.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.425+0000 7f92a267e740 1 --2- 172.21.15.90:0/2041589582 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e25d4d09f0 0x55e25d4e1100 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:31.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.425+0000 7f92a267e740 1 --2- 172.21.15.90:0/2041589582 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e25d4cfd10 0x55e25d4e0bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:31.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.425+0000 7f92a267e740 1 -- 172.21.15.90:0/2041589582 >> 172.21.15.90:0/2041589582 conn(0x55e25d496a50 msgr2=0x55e25d4d6890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:45:31.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.425+0000 7f927aa00640 0 client.17787 ms_handle_reset on v2:172.21.15.92:6836/1270635939 2024-04-02T14:45:31.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.425+0000 7f92a267e740 1 -- 172.21.15.90:0/2041589582 shutdown_connections 2024-04-02T14:45:31.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.425+0000 7f92a267e740 1 -- 172.21.15.90:0/2041589582 wait complete. 2024-04-02T14:45:31.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.426+0000 7f92a267e740 20 client.17787 trim_cache size 0 max 16384 2024-04-02T14:45:31.424 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-02T14:45:31.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.427+0000 7f929fa00640 1 -- 172.21.15.90:0/907404353 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f92800a3de0 msgr2=0x7f92800a62a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:45:31.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.427+0000 7f929fa00640 1 --2- 172.21.15.90:0/907404353 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f92800a3de0 0x7f92800a62a0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f9290009270 tx=0x7f92900095e0 comp rx=0 tx=0).stop 2024-04-02T14:45:31.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.428+0000 7f929fa00640 1 -- 172.21.15.90:0/907404353 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f92981016a0 msgr2=0x7f9298105090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:45:31.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.428+0000 7f929fa00640 1 --2- 172.21.15.90:0/907404353 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f92981016a0 0x7f9298105090 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f928802efd0 tx=0x7f9288004390 comp rx=0 tx=0).stop 2024-04-02T14:45:31.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.428+0000 7f929fa00640 1 -- 172.21.15.90:0/907404353 shutdown_connections 2024-04-02T14:45:31.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.428+0000 7f929fa00640 1 --2- 172.21.15.90:0/907404353 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f92800a3de0 0x7f92800a62a0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:31.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.428+0000 7f929fa00640 1 --2- 172.21.15.90:0/907404353 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9298106a30 0x7f92981055d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:31.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.428+0000 7f929fa00640 1 --2- 172.21.15.90:0/907404353 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f92981016a0 0x7f9298105090 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:31.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.428+0000 7f929fa00640 1 --2- 172.21.15.90:0/907404353 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9298100ce0 0x7f929819ae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:31.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.428+0000 7f929fa00640 1 -- 172.21.15.90:0/907404353 >> 172.21.15.90:0/907404353 conn(0x7f92980779e0 msgr2=0x7f9298100560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:45:31.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.428+0000 7f929fa00640 1 -- 172.21.15.90:0/907404353 shutdown_connections 2024-04-02T14:45:31.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.429+0000 7f929fa00640 1 -- 172.21.15.90:0/907404353 wait complete. 2024-04-02T14:45:31.432 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:45:31 smithi090 ceph-mon[31674]: from='client.? 172.21.15.90:0/3747817417' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-02T14:45:31.452 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-02T14:45:31.605 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:45:31 smithi092 ceph-mon[36497]: from='client.? 172.21.15.90:0/3747817417' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-02T14:45:31.681 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:45:31 smithi177 ceph-mon[36431]: from='client.? 172.21.15.90:0/3747817417' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-02T14:45:31.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.746+0000 7f1933a00640 1 -- 172.21.15.90:0/1216351340 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f192c107a90 msgr2=0x7f192c107e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:45:31.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.746+0000 7f1933a00640 1 --2- 172.21.15.90:0/1216351340 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f192c107a90 0x7f192c107e70 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f1920009860 tx=0x7f192002bb90 comp rx=0 tx=0).stop 2024-04-02T14:45:31.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.748+0000 7f1933a00640 1 -- 172.21.15.90:0/1216351340 shutdown_connections 2024-04-02T14:45:31.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.748+0000 7f1933a00640 1 --2- 172.21.15.90:0/1216351340 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f192c102440 0x7f192c10e950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:31.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.748+0000 7f1933a00640 1 --2- 172.21.15.90:0/1216351340 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f192c101a80 0x7f192c101f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:31.745 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.748+0000 7f1933a00640 1 --2- 172.21.15.90:0/1216351340 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f192c107a90 0x7f192c107e70 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:31.745 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.748+0000 7f1933a00640 1 -- 172.21.15.90:0/1216351340 >> 172.21.15.90:0/1216351340 conn(0x7f192c0fd590 msgr2=0x7f192c0ff9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:45:31.745 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.748+0000 7f1933a00640 1 -- 172.21.15.90:0/1216351340 shutdown_connections 2024-04-02T14:45:31.745 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.748+0000 7f1933a00640 1 -- 172.21.15.90:0/1216351340 wait complete. 2024-04-02T14:45:31.745 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.749+0000 7f1933a00640 1 Processor -- start 2024-04-02T14:45:31.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.751+0000 7f1933a00640 1 -- start start 2024-04-02T14:45:31.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.752+0000 7f1933a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f192c101a80 0x7f192c19f260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:31.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.752+0000 7f1933a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f192c102440 0x7f192c19f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:31.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.752+0000 7f1933a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f192c107a90 0x7f192c19a700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:31.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.752+0000 7f1933a00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f192c0745f0 con 0x7f192c101a80 2024-04-02T14:45:31.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.752+0000 7f1933a00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f192c074470 con 0x7f192c102440 2024-04-02T14:45:31.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.752+0000 7f1933a00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f192c074770 con 0x7f192c107a90 2024-04-02T14:45:31.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.752+0000 7f1931a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f192c107a90 0x7f192c19a700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:31.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.752+0000 7f1931a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f192c107a90 0x7f192c19a700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:55628/0 (socket says 172.21.15.90:55628) 2024-04-02T14:45:31.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.752+0000 7f1931a00640 1 -- 172.21.15.90:0/875003135 learned_addr learned my addr 172.21.15.90:0/875003135 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:45:31.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.752+0000 7f192be00640 1 --2- 172.21.15.90:0/875003135 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f192c102440 0x7f192c19f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:31.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.753+0000 7f1931a00640 1 -- 172.21.15.90:0/875003135 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f192c102440 msgr2=0x7f192c19f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:45:31.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.754+0000 7f1931000640 1 --2- 172.21.15.90:0/875003135 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f192c101a80 0x7f192c19f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:31.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.755+0000 7f1931a00640 1 --2- 172.21.15.90:0/875003135 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f192c102440 0x7f192c19f7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:31.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.755+0000 7f1931a00640 1 -- 172.21.15.90:0/875003135 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f192c101a80 msgr2=0x7f192c19f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:45:31.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.755+0000 7f1931a00640 1 --2- 172.21.15.90:0/875003135 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f192c101a80 0x7f192c19f260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:31.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.755+0000 7f1931a00640 1 -- 172.21.15.90:0/875003135 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f192c19ae80 con 0x7f192c107a90 2024-04-02T14:45:31.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.755+0000 7f1931000640 1 --2- 172.21.15.90:0/875003135 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f192c101a80 0x7f192c19f260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:45:31.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.755+0000 7f192be00640 1 --2- 172.21.15.90:0/875003135 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f192c102440 0x7f192c19f7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:45:31.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.755+0000 7f1931a00640 1 --2- 172.21.15.90:0/875003135 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f192c107a90 0x7f192c19a700 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f1924009fd0 tx=0x7f192400c800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:45:31.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.755+0000 7f1929600640 1 -- 172.21.15.90:0/875003135 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1924004b20 con 0x7f192c107a90 2024-04-02T14:45:31.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.756+0000 7f1929600640 1 -- 172.21.15.90:0/875003135 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1924003e50 con 0x7f192c107a90 2024-04-02T14:45:31.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.756+0000 7f1929600640 1 -- 172.21.15.90:0/875003135 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f192400b630 con 0x7f192c107a90 2024-04-02T14:45:31.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.756+0000 7f1933a00640 1 -- 172.21.15.90:0/875003135 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f192c19b170 con 0x7f192c107a90 2024-04-02T14:45:31.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.756+0000 7f1933a00640 1 -- 172.21.15.90:0/875003135 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f192c1a71d0 con 0x7f192c107a90 2024-04-02T14:45:31.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.760+0000 7f1936826740 1 Processor -- start 2024-04-02T14:45:31.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.766+0000 7f1929600640 1 -- 172.21.15.90:0/875003135 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f192401d020 con 0x7f192c107a90 2024-04-02T14:45:31.764 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.768+0000 7f1929600640 1 --2- 172.21.15.90:0/875003135 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f19140a3ef0 0x7f19140a63b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:31.764 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.768+0000 7f1929600640 1 -- 172.21.15.90:0/875003135 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 8550+0+0 (secure 0 0 0) 0x7f19240d0870 con 0x7f192c107a90 2024-04-02T14:45:31.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.768+0000 7f192be00640 1 --2- 172.21.15.90:0/875003135 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f19140a3ef0 0x7f19140a63b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:31.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.769+0000 7f1936826740 1 -- start start 2024-04-02T14:45:31.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.769+0000 7f1936826740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fe71c4d200 0x55fe71c4bf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:31.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.769+0000 7f1936826740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fe71c4c520 0x55fe71c4c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:31.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.769+0000 7f1936826740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fe71c55890 0x55fe71c57c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:31.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.769+0000 7f1936826740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55fe71b84400 con 0x55fe71c4c520 2024-04-02T14:45:31.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.769+0000 7f1936826740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55fe71b68f10 con 0x55fe71c55890 2024-04-02T14:45:31.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.769+0000 7f1936826740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55fe71ba86f0 con 0x55fe71c4d200 2024-04-02T14:45:31.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.770+0000 7f1931a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fe71c55890 0x55fe71c57c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:31.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.770+0000 7f1931a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fe71c55890 0x55fe71c57c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:39634/0 (socket says 172.21.15.90:39634) 2024-04-02T14:45:31.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.770+0000 7f1931a00640 1 -- 172.21.15.90:0/966240392 learned_addr learned my addr 172.21.15.90:0/966240392 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:45:31.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.770+0000 7f1931000640 1 --2- 172.21.15.90:0/966240392 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fe71c4d200 0x55fe71c4bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:31.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.770+0000 7f1931a00640 1 -- 172.21.15.90:0/966240392 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fe71c4d200 msgr2=0x55fe71c4bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:45:31.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.770+0000 7f1931a00640 1 --2- 172.21.15.90:0/966240392 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fe71c4d200 0x55fe71c4bf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:31.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.770+0000 7f1931a00640 1 -- 172.21.15.90:0/966240392 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fe71c4c520 msgr2=0x55fe71c4c9a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-02T14:45:31.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.770+0000 7f192be00640 1 --2- 172.21.15.90:0/966240392 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fe71c4c520 0x55fe71c4c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:31.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.770+0000 7f1931a00640 1 --2- 172.21.15.90:0/966240392 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fe71c4c520 0x55fe71c4c9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:31.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.770+0000 7f1931a00640 1 -- 172.21.15.90:0/966240392 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fe718bc3f0 con 0x55fe71c55890 2024-04-02T14:45:31.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.772+0000 7f192be00640 1 --2- 172.21.15.90:0/875003135 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f19140a3ef0 0x7f19140a63b0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f1918006fd0 tx=0x7f1918008040 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:45:31.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.773+0000 7f1931a00640 1 --2- 172.21.15.90:0/966240392 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fe71c55890 0x55fe71c57c80 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f1924099160 tx=0x7f192409c630 comp rx=0 tx=0).ready entity=mon.2 client_cookie=84f40e94d2048858 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:45:31.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.773+0000 7f191e000640 1 -- 172.21.15.90:0/966240392 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f192409ce90 con 0x55fe71c55890 2024-04-02T14:45:31.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.773+0000 7f191e000640 1 -- 172.21.15.90:0/966240392 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f192409c790 con 0x55fe71c55890 2024-04-02T14:45:31.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.773+0000 7f191e000640 1 -- 172.21.15.90:0/966240392 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19240a4f70 con 0x55fe71c55890 2024-04-02T14:45:31.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.775+0000 7f1936826740 1 -- 172.21.15.90:0/966240392 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fe71c55890 msgr2=0x55fe71c57c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:45:31.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.775+0000 7f1936826740 1 --2- 172.21.15.90:0/966240392 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fe71c55890 0x55fe71c57c80 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f1924099160 tx=0x7f192409c630 comp rx=0 tx=0).stop 2024-04-02T14:45:31.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.776+0000 7f1936826740 1 -- 172.21.15.90:0/966240392 shutdown_connections 2024-04-02T14:45:31.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.776+0000 7f1936826740 1 --2- 172.21.15.90:0/966240392 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fe71c55890 0x55fe71c57c80 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:31.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.776+0000 7f1936826740 1 --2- 172.21.15.90:0/966240392 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fe71c4c520 0x55fe71c4c9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:31.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.776+0000 7f1936826740 1 --2- 172.21.15.90:0/966240392 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fe71c4d200 0x55fe71c4bf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:31.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.776+0000 7f1936826740 1 -- 172.21.15.90:0/966240392 >> 172.21.15.90:0/966240392 conn(0x55fe71c0aa50 msgr2=0x55fe71c46f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:45:31.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.778+0000 7f1936826740 1 -- 172.21.15.90:0/966240392 shutdown_connections 2024-04-02T14:45:31.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.779+0000 7f1936826740 1 -- 172.21.15.90:0/966240392 wait complete. 2024-04-02T14:45:31.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.779+0000 7f1936826740 1 -- start start 2024-04-02T14:45:31.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.779+0000 7f1936826740 1 Processor -- start 2024-04-02T14:45:31.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.781+0000 7f1936826740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fe71c4c520 0x55fe71c528b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:31.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.781+0000 7f1936826740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fe71c4d200 0x55fe71c52df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:31.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.781+0000 7f1936826740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fe71c55890 0x55fe71c53330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:31.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.781+0000 7f1936826740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55fe71ba46a0 con 0x55fe71c55890 2024-04-02T14:45:31.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.781+0000 7f1936826740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55fe71b50980 con 0x55fe71c4c520 2024-04-02T14:45:31.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.781+0000 7f1936826740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55fe71b63cb0 con 0x55fe71c4d200 2024-04-02T14:45:31.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.781+0000 7f1936826740 10 client.0 _ll_register_callbacks cb 0x55fe71b636a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:45:31.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.781+0000 7f1931000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fe71c4c520 0x55fe71c528b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:31.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.782+0000 7f192be00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fe71c4d200 0x55fe71c52df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:31.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.782+0000 7f1931000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fe71c4c520 0x55fe71c528b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:39648/0 (socket says 172.21.15.90:39648) 2024-04-02T14:45:31.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.782+0000 7f1931000640 1 -- 172.21.15.90:0/3138443946 learned_addr learned my addr 172.21.15.90:0/3138443946 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:45:31.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.782+0000 7f1931a00640 1 --2- 172.21.15.90:0/3138443946 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fe71c55890 0x55fe71c53330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:31.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.782+0000 7f1931000640 1 -- 172.21.15.90:0/3138443946 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fe71c4d200 msgr2=0x55fe71c52df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:45:31.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.782+0000 7f1931000640 1 --2- 172.21.15.90:0/3138443946 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fe71c4d200 0x55fe71c52df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:31.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.782+0000 7f1931000640 1 -- 172.21.15.90:0/3138443946 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fe71c55890 msgr2=0x55fe71c53330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:45:31.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.782+0000 7f1931000640 1 --2- 172.21.15.90:0/3138443946 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fe71c55890 0x55fe71c53330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:31.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.782+0000 7f1931000640 1 -- 172.21.15.90:0/3138443946 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fe719867d0 con 0x55fe71c4c520 2024-04-02T14:45:31.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.782+0000 7f1931a00640 1 --2- 172.21.15.90:0/3138443946 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fe71c55890 0x55fe71c53330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:45:31.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.783+0000 7f1931000640 1 --2- 172.21.15.90:0/3138443946 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fe71c4c520 0x55fe71c528b0 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f192c19bcf0 tx=0x7f19200042f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9d6612bbbb1a0b57 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:45:31.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.783+0000 7f1912a00640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:45:31.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.783+0000 7f1912a00640 1 -- 172.21.15.90:0/3138443946 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1920039030 con 0x55fe71c4c520 2024-04-02T14:45:31.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.783+0000 7f1912a00640 1 -- 172.21.15.90:0/3138443946 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8550+0+0 (secure 0 0 0) 0x7f1920034850 con 0x55fe71c4c520 2024-04-02T14:45:31.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.783+0000 7f1936826740 1 -- 172.21.15.90:0/3138443946 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f192409c790 con 0x55fe71c4c520 2024-04-02T14:45:31.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.783+0000 7f1912a00640 1 -- 172.21.15.90:0/3138443946 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f192004d050 con 0x55fe71c4c520 2024-04-02T14:45:31.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.784+0000 7f1936826740 10 client.28751 fetch_fsmap learned FSMap version 27 2024-04-02T14:45:31.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.784+0000 7f1936826740 1 -- 172.21.15.90:0/3138443946 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55fe71c3fc70 con 0x55fe71c4c520 2024-04-02T14:45:31.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.784+0000 7f1912a00640 1 -- 172.21.15.90:0/3138443946 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f192409c790 con 0x55fe71c4c520 2024-04-02T14:45:31.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.784+0000 7f1912a00640 1 -- 172.21.15.90:0/3138443946 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f19200049e0 con 0x55fe71c4c520 2024-04-02T14:45:31.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.785+0000 7f1936826740 10 client.28751 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:45:31.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.785+0000 7f1936826740 10 client.28751 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-04-02T14:45:31.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.785+0000 7f1936826740 20 client.28751 populate_metadata read hostname 'smithi090' 2024-04-02T14:45:31.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.785+0000 7f1936826740 1 --2- 172.21.15.90:0/3138443946 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x55fe71c411d0 0x55fe71c43670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:31.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.785+0000 7f1936826740 4 client.28751 mds_command: new command op to 24547 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:45:31.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.785+0000 7f1936826740 1 -- 172.21.15.90:0/3138443946 --> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fe71b68040 con 0x55fe71c411d0 2024-04-02T14:45:31.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.785+0000 7f192be00640 1 --2- 172.21.15.90:0/3138443946 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x55fe71c411d0 0x55fe71c43670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:31.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.786+0000 7f1912a00640 10 client.28751 ms_handle_connect on v2:172.21.15.177:6838/664895663 2024-04-02T14:45:31.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.786+0000 7f192be00640 1 --2- 172.21.15.90:0/3138443946 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x55fe71c411d0 0x55fe71c43670 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=46f8b311c78706f4 server_cookie=bd87f01b592c8bc in_seq=0 out_seq=0 2024-04-02T14:45:31.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.787+0000 7f1912a00640 1 -- 172.21.15.90:0/3138443946 <== mds.2 v2:172.21.15.177:6838/664895663 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55fe71b68040 con 0x55fe71c411d0 2024-04-02T14:45:31.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.787+0000 7f1912a00640 10 client.28751 handle_command_reply: tid=0 2024-04-02T14:45:31.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.787+0000 7f1936826740 1 client.28751 shutdown 2024-04-02T14:45:31.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.788+0000 7f1936826740 1 -- 172.21.15.90:0/3138443946 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fe71c4c520 msgr2=0x55fe71c528b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:45:31.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.788+0000 7f1936826740 1 --2- 172.21.15.90:0/3138443946 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fe71c4c520 0x55fe71c528b0 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f192c19bcf0 tx=0x7f19200042f0 comp rx=0 tx=0).stop 2024-04-02T14:45:31.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.788+0000 7f1936826740 1 -- 172.21.15.90:0/3138443946 shutdown_connections 2024-04-02T14:45:31.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.788+0000 7f1936826740 1 --2- 172.21.15.90:0/3138443946 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x55fe71c411d0 0x55fe71c43670 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:31.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.788+0000 7f1936826740 1 --2- 172.21.15.90:0/3138443946 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fe71c55890 0x55fe71c53330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:31.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.788+0000 7f1936826740 1 --2- 172.21.15.90:0/3138443946 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fe71c4d200 0x55fe71c52df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:31.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.788+0000 7f1936826740 1 --2- 172.21.15.90:0/3138443946 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fe71c4c520 0x55fe71c528b0 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:31.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.788+0000 7f1912a00640 0 client.28751 ms_handle_reset on v2:172.21.15.177:6838/664895663 2024-04-02T14:45:31.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.788+0000 7f1936826740 1 -- 172.21.15.90:0/3138443946 >> 172.21.15.90:0/3138443946 conn(0x55fe71c0aa50 msgr2=0x55fe71c48cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:45:31.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.788+0000 7f1936826740 1 -- 172.21.15.90:0/3138443946 shutdown_connections 2024-04-02T14:45:31.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.789+0000 7f1936826740 1 -- 172.21.15.90:0/3138443946 wait complete. 2024-04-02T14:45:31.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.789+0000 7f1936826740 20 client.28751 trim_cache size 0 max 16384 2024-04-02T14:45:31.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.816+0000 7f1936826740 1 Processor -- start 2024-04-02T14:45:31.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.816+0000 7f1936826740 1 -- start start 2024-04-02T14:45:31.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.816+0000 7f1936826740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fe71c4c520 0x55fe71c54fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:31.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.816+0000 7f1936826740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fe71c4d200 0x55fe71c61300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:31.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.816+0000 7f1936826740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fe71c55890 0x55fe71c78b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:31.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.816+0000 7f1936826740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f192c1ab700 con 0x55fe71c4d200 2024-04-02T14:45:31.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.816+0000 7f1936826740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f192c1ad240 con 0x55fe71c55890 2024-04-02T14:45:31.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.816+0000 7f1936826740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f192c1abdd0 con 0x55fe71c4c520 2024-04-02T14:45:31.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.817+0000 7f1931a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fe71c55890 0x55fe71c78b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:31.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.817+0000 7f1931a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fe71c55890 0x55fe71c78b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:39662/0 (socket says 172.21.15.90:39662) 2024-04-02T14:45:31.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.817+0000 7f1931a00640 1 -- 172.21.15.90:0/2672711284 learned_addr learned my addr 172.21.15.90:0/2672711284 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:45:31.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.817+0000 7f1931000640 1 --2- 172.21.15.90:0/2672711284 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fe71c4c520 0x55fe71c54fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:31.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.817+0000 7f192be00640 1 --2- 172.21.15.90:0/2672711284 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fe71c4d200 0x55fe71c61300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:31.815 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.818+0000 7f1931a00640 1 -- 172.21.15.90:0/2672711284 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fe71c4c520 msgr2=0x55fe71c54fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:45:31.815 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.818+0000 7f1931a00640 1 --2- 172.21.15.90:0/2672711284 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fe71c4c520 0x55fe71c54fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:31.815 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.818+0000 7f1931a00640 1 -- 172.21.15.90:0/2672711284 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fe71c4d200 msgr2=0x55fe71c61300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:45:31.815 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.818+0000 7f1931a00640 1 --2- 172.21.15.90:0/2672711284 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fe71c4d200 0x55fe71c61300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:31.815 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.818+0000 7f1931a00640 1 -- 172.21.15.90:0/2672711284 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fe71c57ff0 con 0x55fe71c55890 2024-04-02T14:45:31.815 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.818+0000 7f192be00640 1 --2- 172.21.15.90:0/2672711284 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fe71c4d200 0x55fe71c61300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:45:31.815 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.819+0000 7f1931a00640 1 --2- 172.21.15.90:0/2672711284 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fe71c55890 0x55fe71c78b70 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f19240a4ed0 tx=0x7f1924004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8b4806701e1f86c9 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:45:31.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.819+0000 7f191d600640 1 -- 172.21.15.90:0/2672711284 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f192401a040 con 0x55fe71c55890 2024-04-02T14:45:31.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.819+0000 7f191d600640 1 -- 172.21.15.90:0/2672711284 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f19240b9030 con 0x55fe71c55890 2024-04-02T14:45:31.817 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.820+0000 7f191d600640 1 -- 172.21.15.90:0/2672711284 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19240a4f70 con 0x55fe71c55890 2024-04-02T14:45:31.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.823+0000 7f1936826740 1 -- 172.21.15.90:0/2672711284 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fe71c55890 msgr2=0x55fe71c78b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:45:31.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.823+0000 7f1936826740 1 --2- 172.21.15.90:0/2672711284 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fe71c55890 0x55fe71c78b70 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f19240a4ed0 tx=0x7f1924004a40 comp rx=0 tx=0).stop 2024-04-02T14:45:31.820 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.823+0000 7f1936826740 1 -- 172.21.15.90:0/2672711284 shutdown_connections 2024-04-02T14:45:31.820 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.823+0000 7f1936826740 1 --2- 172.21.15.90:0/2672711284 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fe71c55890 0x55fe71c78b70 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:31.820 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.823+0000 7f1936826740 1 --2- 172.21.15.90:0/2672711284 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fe71c4d200 0x55fe71c61300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:31.820 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.823+0000 7f1936826740 1 --2- 172.21.15.90:0/2672711284 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fe71c4c520 0x55fe71c54fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:31.820 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.823+0000 7f1936826740 1 -- 172.21.15.90:0/2672711284 >> 172.21.15.90:0/2672711284 conn(0x55fe71c0aa50 msgr2=0x55fe71c3d8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:45:31.820 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.823+0000 7f1936826740 1 -- 172.21.15.90:0/2672711284 shutdown_connections 2024-04-02T14:45:31.820 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.823+0000 7f1936826740 1 -- 172.21.15.90:0/2672711284 wait complete. 2024-04-02T14:45:31.822 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.824+0000 7f1936826740 1 -- start start 2024-04-02T14:45:31.822 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.824+0000 7f1936826740 1 Processor -- start 2024-04-02T14:45:31.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.826+0000 7f1936826740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fe71c4c520 0x55fe71c57480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:31.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.826+0000 7f1936826740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fe71c4d200 0x55fe71c55c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:31.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.826+0000 7f1936826740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fe71c55890 0x55fe71c563b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:31.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.827+0000 7f192be00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fe71c4d200 0x55fe71c55c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:31.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.827+0000 7f1931a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fe71c55890 0x55fe71c563b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:31.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.827+0000 7f1931a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fe71c55890 0x55fe71c563b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:55660/0 (socket says 172.21.15.90:55660) 2024-04-02T14:45:31.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.827+0000 7f1931a00640 1 -- 172.21.15.90:0/1319209927 learned_addr learned my addr 172.21.15.90:0/1319209927 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:45:31.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.827+0000 7f1936826740 1 -- 172.21.15.90:0/1319209927 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f192c1acb70 con 0x55fe71c4c520 2024-04-02T14:45:31.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.827+0000 7f1936826740 1 -- 172.21.15.90:0/1319209927 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f192c1ac4a0 con 0x55fe71c4d200 2024-04-02T14:45:31.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.827+0000 7f1936826740 1 -- 172.21.15.90:0/1319209927 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f192c1a9720 con 0x55fe71c55890 2024-04-02T14:45:31.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.828+0000 7f1931000640 1 --2- 172.21.15.90:0/1319209927 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fe71c4c520 0x55fe71c57480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:31.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.828+0000 7f1936826740 10 client.0 _ll_register_callbacks cb 0x7f19200096b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:45:31.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.828+0000 7f1931a00640 1 -- 172.21.15.90:0/1319209927 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fe71c4d200 msgr2=0x55fe71c55c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:45:31.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.828+0000 7f1931a00640 1 --2- 172.21.15.90:0/1319209927 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fe71c4d200 0x55fe71c55c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:31.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.828+0000 7f1931a00640 1 -- 172.21.15.90:0/1319209927 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fe71c4c520 msgr2=0x55fe71c57480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:45:31.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.828+0000 7f1931a00640 1 --2- 172.21.15.90:0/1319209927 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fe71c4c520 0x55fe71c57480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:31.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.828+0000 7f1931a00640 1 -- 172.21.15.90:0/1319209927 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fe71c5c810 con 0x55fe71c55890 2024-04-02T14:45:31.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.828+0000 7f1931000640 1 --2- 172.21.15.90:0/1319209927 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fe71c4c520 0x55fe71c57480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:45:31.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.828+0000 7f192be00640 1 --2- 172.21.15.90:0/1319209927 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fe71c4d200 0x55fe71c55c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:45:31.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.828+0000 7f1936826740 1 -- 172.21.15.90:0/1319209927 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f19240b9030 con 0x55fe71c55890 2024-04-02T14:45:31.826 INFO:teuthology.orchestra.run.smithi090.stderr:progress: 600.0 s, 2600.8 tps, lat 37.513 ms stddev 47.664 2024-04-02T14:45:31.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.829+0000 7f1931a00640 1 --2- 172.21.15.90:0/1319209927 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fe71c55890 0x55fe71c563b0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f192400cd00 tx=0x7f192409c0f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ccf79c0f74726db7 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:45:31.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.829+0000 7f1912a00640 10 client.28594 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:45:31.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.830+0000 7f1912a00640 1 -- 172.21.15.90:0/1319209927 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f192401a040 con 0x55fe71c55890 2024-04-02T14:45:31.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.830+0000 7f1912a00640 1 -- 172.21.15.90:0/1319209927 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8550+0+0 (secure 0 0 0) 0x7f19240a5170 con 0x55fe71c55890 2024-04-02T14:45:31.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.830+0000 7f1912a00640 1 -- 172.21.15.90:0/1319209927 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f1924098fa0 con 0x55fe71c55890 2024-04-02T14:45:31.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.830+0000 7f1936826740 10 client.28594 fetch_fsmap learned FSMap version 27 2024-04-02T14:45:31.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.830+0000 7f1936826740 1 -- 172.21.15.90:0/1319209927 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({fsmap=27}) -- 0x55fe71c5d920 con 0x55fe71c55890 2024-04-02T14:45:31.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.831+0000 7f1912a00640 1 -- 172.21.15.90:0/1319209927 <== mon.1 v2:172.21.15.177:3300/0 4 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f192400b040 con 0x55fe71c55890 2024-04-02T14:45:31.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.831+0000 7f1936826740 10 client.28594 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:45:31.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.831+0000 7f1936826740 10 client.28594 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-04-02T14:45:31.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.831+0000 7f1936826740 20 client.28594 populate_metadata read hostname 'smithi090' 2024-04-02T14:45:31.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.831+0000 7f1936826740 1 --2- 172.21.15.90:0/1319209927 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x55fe71c5d920 0x55fe71c5dd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:31.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.831+0000 7f1936826740 4 client.28594 mds_command: new command op to 24547 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-02T14:45:31.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.831+0000 7f1936826740 1 -- 172.21.15.90:0/1319209927 --> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55fe71bb5c30 con 0x55fe71c5d920 2024-04-02T14:45:31.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.832+0000 7f1931000640 1 --2- 172.21.15.90:0/1319209927 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x55fe71c5d920 0x55fe71c5dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:31.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.832+0000 7f1931000640 1 --2- 172.21.15.90:0/1319209927 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x55fe71c5d920 0x55fe71c5dd00 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=e8a97a7f7ab06825 server_cookie=133a5f776dba146b in_seq=0 out_seq=0 2024-04-02T14:45:31.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.833+0000 7f1912a00640 10 client.28594 ms_handle_connect on v2:172.21.15.177:6838/664895663 2024-04-02T14:45:31.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.833+0000 7f1912a00640 1 -- 172.21.15.90:0/1319209927 <== mds.2 v2:172.21.15.177:6838/664895663 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55fe71bb5c30 con 0x55fe71c5d920 2024-04-02T14:45:31.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.833+0000 7f1912a00640 10 client.28594 handle_command_reply: tid=0 2024-04-02T14:45:31.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.833+0000 7f1936826740 1 client.28594 shutdown 2024-04-02T14:45:31.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.834+0000 7f1936826740 1 -- 172.21.15.90:0/1319209927 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fe71c55890 msgr2=0x55fe71c563b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:45:31.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.834+0000 7f1936826740 1 --2- 172.21.15.90:0/1319209927 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fe71c55890 0x55fe71c563b0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f192400cd00 tx=0x7f192409c0f0 comp rx=0 tx=0).stop 2024-04-02T14:45:31.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.834+0000 7f1936826740 1 -- 172.21.15.90:0/1319209927 shutdown_connections 2024-04-02T14:45:31.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.834+0000 7f1936826740 1 --2- 172.21.15.90:0/1319209927 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x55fe71c5d920 0x55fe71c5dd00 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:31.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.834+0000 7f1936826740 1 --2- 172.21.15.90:0/1319209927 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fe71c55890 0x55fe71c563b0 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:31.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.834+0000 7f1936826740 1 --2- 172.21.15.90:0/1319209927 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55fe71c4d200 0x55fe71c55c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:31.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.834+0000 7f1912a00640 0 client.28594 ms_handle_reset on v2:172.21.15.177:6838/664895663 2024-04-02T14:45:31.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.834+0000 7f1936826740 1 --2- 172.21.15.90:0/1319209927 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55fe71c4c520 0x55fe71c57480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:31.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.834+0000 7f1936826740 1 -- 172.21.15.90:0/1319209927 >> 172.21.15.90:0/1319209927 conn(0x55fe71c0aa50 msgr2=0x55fe71c51050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:45:31.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.834+0000 7f1936826740 1 -- 172.21.15.90:0/1319209927 shutdown_connections 2024-04-02T14:45:31.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.834+0000 7f1936826740 1 -- 172.21.15.90:0/1319209927 wait complete. 2024-04-02T14:45:31.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.834+0000 7f1936826740 20 client.28594 trim_cache size 0 max 16384 2024-04-02T14:45:31.833 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-02T14:45:31.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.835+0000 7f1933a00640 1 -- 172.21.15.90:0/875003135 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f19140a3ef0 msgr2=0x7f19140a63b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:45:31.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.835+0000 7f1933a00640 1 --2- 172.21.15.90:0/875003135 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f19140a3ef0 0x7f19140a63b0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f1918006fd0 tx=0x7f1918008040 comp rx=0 tx=0).stop 2024-04-02T14:45:31.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.836+0000 7f1933a00640 1 -- 172.21.15.90:0/875003135 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f192c107a90 msgr2=0x7f192c19a700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:45:31.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.836+0000 7f1933a00640 1 --2- 172.21.15.90:0/875003135 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f192c107a90 0x7f192c19a700 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f1924009fd0 tx=0x7f192400c800 comp rx=0 tx=0).stop 2024-04-02T14:45:31.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.836+0000 7f1933a00640 1 -- 172.21.15.90:0/875003135 shutdown_connections 2024-04-02T14:45:31.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.836+0000 7f1933a00640 1 --2- 172.21.15.90:0/875003135 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f19140a3ef0 0x7f19140a63b0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:31.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.836+0000 7f1933a00640 1 --2- 172.21.15.90:0/875003135 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f192c107a90 0x7f192c19a700 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:31.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.836+0000 7f1933a00640 1 --2- 172.21.15.90:0/875003135 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f192c102440 0x7f192c19f7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:31.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.836+0000 7f1933a00640 1 --2- 172.21.15.90:0/875003135 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f192c101a80 0x7f192c19f260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:31.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.836+0000 7f1933a00640 1 -- 172.21.15.90:0/875003135 >> 172.21.15.90:0/875003135 conn(0x7f192c0fd590 msgr2=0x7f192c0fec40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:45:31.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.836+0000 7f1933a00640 1 -- 172.21.15.90:0/875003135 shutdown_connections 2024-04-02T14:45:31.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:31.836+0000 7f1933a00640 1 -- 172.21.15.90:0/875003135 wait complete. 2024-04-02T14:45:31.856 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-02T14:45:32.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.134+0000 7f6598c00640 1 -- 172.21.15.90:0/3857444509 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6594069620 msgr2=0x7f659410df60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:45:32.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.134+0000 7f6598c00640 1 --2- 172.21.15.90:0/3857444509 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6594069620 0x7f659410df60 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f6580009900 tx=0x7f658002bcd0 comp rx=0 tx=0).stop 2024-04-02T14:45:32.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.134+0000 7f6598c00640 1 -- 172.21.15.90:0/3857444509 shutdown_connections 2024-04-02T14:45:32.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.134+0000 7f6598c00640 1 --2- 172.21.15.90:0/3857444509 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f659410e4a0 0x7f6594110890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:32.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.134+0000 7f6598c00640 1 --2- 172.21.15.90:0/3857444509 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6594069620 0x7f659410df60 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:32.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.134+0000 7f6598c00640 1 --2- 172.21.15.90:0/3857444509 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6594068d00 0x7f65940690e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:32.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.134+0000 7f6598c00640 1 -- 172.21.15.90:0/3857444509 >> 172.21.15.90:0/3857444509 conn(0x7f6594077b00 msgr2=0x7f6594077f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:45:32.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.134+0000 7f6598c00640 1 -- 172.21.15.90:0/3857444509 shutdown_connections 2024-04-02T14:45:32.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.134+0000 7f6598c00640 1 -- 172.21.15.90:0/3857444509 wait complete. 2024-04-02T14:45:32.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.135+0000 7f6598c00640 1 Processor -- start 2024-04-02T14:45:32.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.135+0000 7f6598c00640 1 -- start start 2024-04-02T14:45:32.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.136+0000 7f6598c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6594068d00 0x7f659419a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:32.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.136+0000 7f6598c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6594069620 0x7f659419aae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:32.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.136+0000 7f6598c00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f659410e4a0 0x7f659419ee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:32.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.136+0000 7f6598c00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f6594114080 con 0x7f6594069620 2024-04-02T14:45:32.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.136+0000 7f6598c00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f6594113f00 con 0x7f659410e4a0 2024-04-02T14:45:32.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.136+0000 7f6598c00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f6594114200 con 0x7f6594068d00 2024-04-02T14:45:32.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.137+0000 7f6591e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6594068d00 0x7f659419a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:32.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.137+0000 7f6591e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6594068d00 0x7f659419a5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:55688/0 (socket says 172.21.15.90:55688) 2024-04-02T14:45:32.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.137+0000 7f6591e00640 1 -- 172.21.15.90:0/2254826613 learned_addr learned my addr 172.21.15.90:0/2254826613 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:45:32.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.137+0000 7f6592800640 1 --2- 172.21.15.90:0/2254826613 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f659410e4a0 0x7f659419ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:32.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.137+0000 7f6591400640 1 --2- 172.21.15.90:0/2254826613 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6594069620 0x7f659419aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:32.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.138+0000 7f6591e00640 1 -- 172.21.15.90:0/2254826613 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f659410e4a0 msgr2=0x7f659419ee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:45:32.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.138+0000 7f6591e00640 1 --2- 172.21.15.90:0/2254826613 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f659410e4a0 0x7f659419ee60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:32.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.138+0000 7f6591e00640 1 -- 172.21.15.90:0/2254826613 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6594069620 msgr2=0x7f659419aae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:45:32.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.138+0000 7f6591e00640 1 --2- 172.21.15.90:0/2254826613 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6594069620 0x7f659419aae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:32.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.138+0000 7f6591e00640 1 -- 172.21.15.90:0/2254826613 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f659419f590 con 0x7f6594068d00 2024-04-02T14:45:32.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.138+0000 7f6591400640 1 --2- 172.21.15.90:0/2254826613 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6594069620 0x7f659419aae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:45:32.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.140+0000 7f6591e00640 1 --2- 172.21.15.90:0/2254826613 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6594068d00 0x7f659419a5a0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f658400ea60 tx=0x7f658400ef20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:45:32.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.142+0000 7f658aa00640 1 -- 172.21.15.90:0/2254826613 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f658400cde0 con 0x7f6594068d00 2024-04-02T14:45:32.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.142+0000 7f658aa00640 1 -- 172.21.15.90:0/2254826613 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f65840035f0 con 0x7f6594068d00 2024-04-02T14:45:32.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.142+0000 7f658aa00640 1 -- 172.21.15.90:0/2254826613 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6584010670 con 0x7f6594068d00 2024-04-02T14:45:32.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.142+0000 7f6598c00640 1 -- 172.21.15.90:0/2254826613 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f659419f880 con 0x7f6594068d00 2024-04-02T14:45:32.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.142+0000 7f6598c00640 1 -- 172.21.15.90:0/2254826613 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65941a7110 con 0x7f6594068d00 2024-04-02T14:45:32.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.147+0000 7f659b969740 1 Processor -- start 2024-04-02T14:45:32.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.150+0000 7f659b969740 1 -- start start 2024-04-02T14:45:32.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.151+0000 7f659b969740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a698378200 0x55a698376f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:32.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.151+0000 7f659b969740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a698377520 0x55a6983779a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:32.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.151+0000 7f659b969740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a698380890 0x55a698382c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:32.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.151+0000 7f659b969740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55a6982af400 con 0x55a698377520 2024-04-02T14:45:32.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.151+0000 7f659b969740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55a698293f10 con 0x55a698380890 2024-04-02T14:45:32.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.151+0000 7f659b969740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55a6982d36f0 con 0x55a698378200 2024-04-02T14:45:32.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.152+0000 7f6592800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a698378200 0x55a698376f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:32.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.152+0000 7f6592800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a698378200 0x55a698376f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:55692/0 (socket says 172.21.15.90:55692) 2024-04-02T14:45:32.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.152+0000 7f6592800640 1 -- 172.21.15.90:0/2020006614 learned_addr learned my addr 172.21.15.90:0/2020006614 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:45:32.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.153+0000 7f6591400640 1 --2- 172.21.15.90:0/2020006614 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a698380890 0x55a698382c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:32.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.154+0000 7f6592800640 1 -- 172.21.15.90:0/2020006614 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a698380890 msgr2=0x55a698382c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:45:32.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.154+0000 7f6592800640 1 --2- 172.21.15.90:0/2020006614 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a698380890 0x55a698382c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:32.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.154+0000 7f6592800640 1 -- 172.21.15.90:0/2020006614 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a698377520 msgr2=0x55a6983779a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-02T14:45:32.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.154+0000 7f6592800640 1 --2- 172.21.15.90:0/2020006614 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a698377520 0x55a6983779a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:32.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.154+0000 7f6592800640 1 -- 172.21.15.90:0/2020006614 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a697fe73f0 con 0x55a698378200 2024-04-02T14:45:32.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.155+0000 7f6592800640 1 --2- 172.21.15.90:0/2020006614 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a698378200 0x55a698376f50 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f658c009fd0 tx=0x7f658c0118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=af219a1a5194dbfe server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:45:32.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.155+0000 7f657f000640 1 -- 172.21.15.90:0/2020006614 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f658c004d20 con 0x55a698378200 2024-04-02T14:45:32.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.156+0000 7f657f000640 1 -- 172.21.15.90:0/2020006614 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f658c004050 con 0x55a698378200 2024-04-02T14:45:32.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.156+0000 7f657f000640 1 -- 172.21.15.90:0/2020006614 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f658c005280 con 0x55a698378200 2024-04-02T14:45:32.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.157+0000 7f659b969740 1 -- 172.21.15.90:0/2020006614 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a698378200 msgr2=0x55a698376f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:45:32.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.157+0000 7f659b969740 1 --2- 172.21.15.90:0/2020006614 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a698378200 0x55a698376f50 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f658c009fd0 tx=0x7f658c0118d0 comp rx=0 tx=0).stop 2024-04-02T14:45:32.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.158+0000 7f658aa00640 1 -- 172.21.15.90:0/2254826613 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f65840037a0 con 0x7f6594068d00 2024-04-02T14:45:32.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.160+0000 7f658aa00640 1 --2- 172.21.15.90:0/2254826613 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f65780a3f20 0x7f65780a63e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:32.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.162+0000 7f6592800640 1 --2- 172.21.15.90:0/2254826613 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f65780a3f20 0x7f65780a63e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:32.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.163+0000 7f658aa00640 1 -- 172.21.15.90:0/2254826613 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 8550+0+0 (secure 0 0 0) 0x7f65840d0aa0 con 0x7f6594068d00 2024-04-02T14:45:32.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.164+0000 7f6592800640 1 --2- 172.21.15.90:0/2254826613 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f65780a3f20 0x7f65780a63e0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f658c01b040 tx=0x7f658c004f90 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:45:32.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.166+0000 7f659b969740 1 -- 172.21.15.90:0/2020006614 shutdown_connections 2024-04-02T14:45:32.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.166+0000 7f659b969740 1 --2- 172.21.15.90:0/2020006614 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a698380890 0x55a698382c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:32.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.166+0000 7f659b969740 1 --2- 172.21.15.90:0/2020006614 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a698377520 0x55a6983779a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:32.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.166+0000 7f659b969740 1 --2- 172.21.15.90:0/2020006614 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a698378200 0x55a698376f50 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:32.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.166+0000 7f659b969740 1 -- 172.21.15.90:0/2020006614 >> 172.21.15.90:0/2020006614 conn(0x55a698335a50 msgr2=0x55a698371f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:45:32.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.166+0000 7f659b969740 1 -- 172.21.15.90:0/2020006614 shutdown_connections 2024-04-02T14:45:32.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.167+0000 7f659b969740 1 -- 172.21.15.90:0/2020006614 wait complete. 2024-04-02T14:45:32.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.167+0000 7f659b969740 1 -- start start 2024-04-02T14:45:32.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.168+0000 7f659b969740 1 Processor -- start 2024-04-02T14:45:32.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.169+0000 7f659b969740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a698377520 0x55a69837d880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:32.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.169+0000 7f659b969740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a698378200 0x55a69837ddc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:32.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.169+0000 7f659b969740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a698380890 0x55a69837e300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:32.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.169+0000 7f659b969740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55a6982cf6a0 con 0x55a698377520 2024-04-02T14:45:32.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.169+0000 7f6591400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a698378200 0x55a69837ddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:32.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.169+0000 7f6591400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a698378200 0x55a69837ddc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:55702/0 (socket says 172.21.15.90:55702) 2024-04-02T14:45:32.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.170+0000 7f6591400640 1 -- 172.21.15.90:0/3499413744 learned_addr learned my addr 172.21.15.90:0/3499413744 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:45:32.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.170+0000 7f6592800640 1 --2- 172.21.15.90:0/3499413744 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a698380890 0x55a69837e300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:32.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.170+0000 7f659b969740 1 -- 172.21.15.90:0/3499413744 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55a69827b980 con 0x55a698380890 2024-04-02T14:45:32.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.170+0000 7f659b969740 1 -- 172.21.15.90:0/3499413744 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55a69828ecb0 con 0x55a698378200 2024-04-02T14:45:32.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.170+0000 7f659b969740 10 client.0 _ll_register_callbacks cb 0x55a69828e6a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:45:32.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.171+0000 7f6592800640 1 -- 172.21.15.90:0/3499413744 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a698378200 msgr2=0x55a69837ddc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:45:32.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.171+0000 7f6592800640 1 --2- 172.21.15.90:0/3499413744 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a698378200 0x55a69837ddc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:32.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.171+0000 7f6592800640 1 -- 172.21.15.90:0/3499413744 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a698377520 msgr2=0x55a69837d880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:45:32.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.171+0000 7f6592800640 1 --2- 172.21.15.90:0/3499413744 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a698377520 0x55a69837d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:32.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.171+0000 7f6592800640 1 -- 172.21.15.90:0/3499413744 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a6980b17d0 con 0x55a698380890 2024-04-02T14:45:32.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.172+0000 7f6592800640 1 --2- 172.21.15.90:0/3499413744 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a698380890 0x55a69837e300 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f658c011dd0 tx=0x7f658c0118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c88e1b6eb120d7a6 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:45:32.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.172+0000 7f6573400640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:45:32.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.172+0000 7f6573400640 1 -- 172.21.15.90:0/3499413744 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f658c034070 con 0x55a698380890 2024-04-02T14:45:32.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.172+0000 7f6573400640 1 -- 172.21.15.90:0/3499413744 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8550+0+0 (secure 0 0 0) 0x7f658c003a30 con 0x55a698380890 2024-04-02T14:45:32.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.173+0000 7f659b969740 1 -- 172.21.15.90:0/3499413744 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f658c004050 con 0x55a698380890 2024-04-02T14:45:32.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.173+0000 7f6573400640 1 -- 172.21.15.90:0/3499413744 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f658c03e050 con 0x55a698380890 2024-04-02T14:45:32.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.173+0000 7f659b969740 10 client.28769 fetch_fsmap learned FSMap version 27 2024-04-02T14:45:32.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.173+0000 7f659b969740 1 -- 172.21.15.90:0/3499413744 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55a6983a0770 con 0x55a698380890 2024-04-02T14:45:32.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.174+0000 7f6573400640 1 -- 172.21.15.90:0/3499413744 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f658c004050 con 0x55a698380890 2024-04-02T14:45:32.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.174+0000 7f6573400640 1 -- 172.21.15.90:0/3499413744 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f658c02f070 con 0x55a698380890 2024-04-02T14:45:32.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.174+0000 7f659b969740 10 client.28769 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:45:32.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.174+0000 7f659b969740 10 client.28769 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2024-04-02T14:45:32.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.174+0000 7f659b969740 20 client.28769 populate_metadata read hostname 'smithi090' 2024-04-02T14:45:32.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.174+0000 7f659b969740 1 --2- 172.21.15.90:0/3499413744 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x55a698369030 0x55a69836b4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:32.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.174+0000 7f659b969740 4 client.28769 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:45:32.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.174+0000 7f659b969740 1 -- 172.21.15.90:0/3499413744 --> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a698293040 con 0x55a698369030 2024-04-02T14:45:32.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.175+0000 7f6591e00640 1 --2- 172.21.15.90:0/3499413744 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x55a698369030 0x55a69836b4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:32.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.178+0000 7f6591e00640 1 --2- 172.21.15.90:0/3499413744 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x55a698369030 0x55a69836b4b0 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=51e64f1b070f0969 server_cookie=1e51f76035484426 in_seq=0 out_seq=0 2024-04-02T14:45:32.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.178+0000 7f6573400640 10 client.28769 ms_handle_connect on v2:172.21.15.90:6838/1498915870 2024-04-02T14:45:32.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.180+0000 7f6573400640 1 -- 172.21.15.90:0/3499413744 <== mds.1 v2:172.21.15.90:6838/1498915870 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a698293040 con 0x55a698369030 2024-04-02T14:45:32.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.180+0000 7f6573400640 10 client.28769 handle_command_reply: tid=0 2024-04-02T14:45:32.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.180+0000 7f659b969740 1 client.28769 shutdown 2024-04-02T14:45:32.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.181+0000 7f659b969740 1 -- 172.21.15.90:0/3499413744 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a698380890 msgr2=0x55a69837e300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:45:32.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.181+0000 7f659b969740 1 --2- 172.21.15.90:0/3499413744 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a698380890 0x55a69837e300 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f658c011dd0 tx=0x7f658c0118d0 comp rx=0 tx=0).stop 2024-04-02T14:45:32.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.182+0000 7f659b969740 1 -- 172.21.15.90:0/3499413744 shutdown_connections 2024-04-02T14:45:32.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.182+0000 7f659b969740 1 --2- 172.21.15.90:0/3499413744 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x55a698369030 0x55a69836b4b0 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:32.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.182+0000 7f659b969740 1 --2- 172.21.15.90:0/3499413744 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a698380890 0x55a69837e300 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:32.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.182+0000 7f659b969740 1 --2- 172.21.15.90:0/3499413744 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a698378200 0x55a69837ddc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:32.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.182+0000 7f659b969740 1 --2- 172.21.15.90:0/3499413744 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a698377520 0x55a69837d880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:32.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.182+0000 7f659b969740 1 -- 172.21.15.90:0/3499413744 >> 172.21.15.90:0/3499413744 conn(0x55a698335a50 msgr2=0x55a698373cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:45:32.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.182+0000 7f6573400640 0 client.28769 ms_handle_reset on v2:172.21.15.90:6838/1498915870 2024-04-02T14:45:32.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.183+0000 7f659b969740 1 -- 172.21.15.90:0/3499413744 shutdown_connections 2024-04-02T14:45:32.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.185+0000 7f659b969740 1 -- 172.21.15.90:0/3499413744 wait complete. 2024-04-02T14:45:32.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.185+0000 7f659b969740 20 client.28769 trim_cache size 0 max 16384 2024-04-02T14:45:32.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.223+0000 7f659b969740 1 Processor -- start 2024-04-02T14:45:32.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.223+0000 7f659b969740 1 -- start start 2024-04-02T14:45:32.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.223+0000 7f659b969740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a698377520 0x55a69837ffb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:32.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.223+0000 7f659b969740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a698378200 0x55a69837e120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:32.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.223+0000 7f659b969740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a698380890 0x55a698368720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:32.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.223+0000 7f659b969740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f659406f980 con 0x55a698380890 2024-04-02T14:45:32.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.223+0000 7f659b969740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f65940714c0 con 0x55a698378200 2024-04-02T14:45:32.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.223+0000 7f659b969740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f6594070050 con 0x55a698377520 2024-04-02T14:45:32.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.223+0000 7f6591400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a698378200 0x55a69837e120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:32.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.224+0000 7f6591400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a698378200 0x55a69837e120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:39724/0 (socket says 172.21.15.90:39724) 2024-04-02T14:45:32.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.224+0000 7f6591400640 1 -- 172.21.15.90:0/1039392712 learned_addr learned my addr 172.21.15.90:0/1039392712 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:45:32.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.224+0000 7f6591e00640 1 --2- 172.21.15.90:0/1039392712 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a698377520 0x55a69837ffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:32.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.224+0000 7f6591400640 1 -- 172.21.15.90:0/1039392712 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a698377520 msgr2=0x55a69837ffb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:45:32.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.224+0000 7f6592800640 1 --2- 172.21.15.90:0/1039392712 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a698380890 0x55a698368720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:32.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.224+0000 7f6591400640 1 --2- 172.21.15.90:0/1039392712 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a698377520 0x55a69837ffb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:32.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.224+0000 7f6591400640 1 -- 172.21.15.90:0/1039392712 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a698380890 msgr2=0x55a698368720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:45:32.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.224+0000 7f6591400640 1 --2- 172.21.15.90:0/1039392712 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a698380890 0x55a698368720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:32.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.225+0000 7f6591400640 1 -- 172.21.15.90:0/1039392712 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a69836ba40 con 0x55a698378200 2024-04-02T14:45:32.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.225+0000 7f6591e00640 1 --2- 172.21.15.90:0/1039392712 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a698377520 0x55a69837ffb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:45:32.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.225+0000 7f6592800640 1 --2- 172.21.15.90:0/1039392712 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a698380890 0x55a698368720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:45:32.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.226+0000 7f6591400640 1 --2- 172.21.15.90:0/1039392712 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a698378200 0x55a69837e120 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f6580036040 tx=0x7f658002c770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d276c7bf9e91efd server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:45:32.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.227+0000 7f657e600640 1 -- 172.21.15.90:0/1039392712 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6580004a70 con 0x55a698378200 2024-04-02T14:45:32.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.227+0000 7f657e600640 1 -- 172.21.15.90:0/1039392712 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6580004c10 con 0x55a698378200 2024-04-02T14:45:32.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.227+0000 7f657e600640 1 -- 172.21.15.90:0/1039392712 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65800051a0 con 0x55a698378200 2024-04-02T14:45:32.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.229+0000 7f659b969740 1 -- 172.21.15.90:0/1039392712 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a698378200 msgr2=0x55a69837e120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:45:32.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.229+0000 7f659b969740 1 --2- 172.21.15.90:0/1039392712 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a698378200 0x55a69837e120 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f6580036040 tx=0x7f658002c770 comp rx=0 tx=0).stop 2024-04-02T14:45:32.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.232+0000 7f659b969740 1 -- 172.21.15.90:0/1039392712 shutdown_connections 2024-04-02T14:45:32.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.232+0000 7f659b969740 1 --2- 172.21.15.90:0/1039392712 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a698380890 0x55a698368720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:32.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.232+0000 7f659b969740 1 --2- 172.21.15.90:0/1039392712 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a698378200 0x55a69837e120 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:32.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.232+0000 7f659b969740 1 --2- 172.21.15.90:0/1039392712 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a698377520 0x55a69837ffb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:32.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.232+0000 7f659b969740 1 -- 172.21.15.90:0/1039392712 >> 172.21.15.90:0/1039392712 conn(0x55a698335a50 msgr2=0x55a698368010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:45:32.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.233+0000 7f659b969740 1 -- 172.21.15.90:0/1039392712 shutdown_connections 2024-04-02T14:45:32.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.233+0000 7f659b969740 1 -- 172.21.15.90:0/1039392712 wait complete. 2024-04-02T14:45:32.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.234+0000 7f659b969740 1 -- start start 2024-04-02T14:45:32.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.234+0000 7f659b969740 1 Processor -- start 2024-04-02T14:45:32.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.236+0000 7f659b969740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a698377520 0x55a69836c1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:32.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.236+0000 7f659b969740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a698378200 0x55a69836c6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:32.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.236+0000 7f6591e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a698377520 0x55a69836c1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:32.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.237+0000 7f659b969740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a698380890 0x55a69840e990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:32.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.237+0000 7f6591e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a698377520 0x55a69836c1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:55710/0 (socket says 172.21.15.90:55710) 2024-04-02T14:45:32.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.237+0000 7f659b969740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f6594070df0 con 0x55a698380890 2024-04-02T14:45:32.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.237+0000 7f659b969740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f6594070720 con 0x55a698378200 2024-04-02T14:45:32.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.237+0000 7f659b969740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f659406d9a0 con 0x55a698377520 2024-04-02T14:45:32.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.237+0000 7f659b969740 10 client.0 _ll_register_callbacks cb 0x7f658c0046f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:45:32.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.237+0000 7f6591400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a698378200 0x55a69836c6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:32.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.237+0000 7f6591400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a698378200 0x55a69836c6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:39728/0 (socket says 172.21.15.90:39728) 2024-04-02T14:45:32.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.237+0000 7f6591400640 1 -- 172.21.15.90:0/558136959 learned_addr learned my addr 172.21.15.90:0/558136959 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:45:32.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.237+0000 7f6592800640 1 --2- 172.21.15.90:0/558136959 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a698380890 0x55a69840e990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:32.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.238+0000 7f6591400640 1 -- 172.21.15.90:0/558136959 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a698377520 msgr2=0x55a69836c1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:45:32.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.238+0000 7f6591400640 1 --2- 172.21.15.90:0/558136959 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a698377520 0x55a69836c1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:32.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.238+0000 7f6591400640 1 -- 172.21.15.90:0/558136959 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a698380890 msgr2=0x55a69840e990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:45:32.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.238+0000 7f6591400640 1 --2- 172.21.15.90:0/558136959 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a698380890 0x55a69840e990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:32.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.238+0000 7f6591400640 1 -- 172.21.15.90:0/558136959 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a69840ef00 con 0x55a698378200 2024-04-02T14:45:32.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.238+0000 7f6591e00640 1 --2- 172.21.15.90:0/558136959 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a698377520 0x55a69836c1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:45:32.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.238+0000 7f6573400640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:45:32.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.238+0000 7f6591400640 1 --2- 172.21.15.90:0/558136959 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a698378200 0x55a69836c6e0 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f658002ce30 tx=0x7f6580005870 comp rx=0 tx=0).ready entity=mon.2 client_cookie=173a8a78abc84b75 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:45:32.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.239+0000 7f6573400640 1 -- 172.21.15.90:0/558136959 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6580055060 con 0x55a698378200 2024-04-02T14:45:32.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.240+0000 7f659b969740 1 -- 172.21.15.90:0/558136959 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6580004c10 con 0x55a698378200 2024-04-02T14:45:32.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.240+0000 7f6573400640 1 -- 172.21.15.90:0/558136959 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8550+0+0 (secure 0 0 0) 0x7f6580004040 con 0x55a698378200 2024-04-02T14:45:32.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.241+0000 7f6573400640 1 -- 172.21.15.90:0/558136959 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f6580055060 con 0x55a698378200 2024-04-02T14:45:32.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.241+0000 7f659b969740 10 client.28775 fetch_fsmap learned FSMap version 27 2024-04-02T14:45:32.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.242+0000 7f659b969740 1 -- 172.21.15.90:0/558136959 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55a698410170 con 0x55a698378200 2024-04-02T14:45:32.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.243+0000 7f6573400640 1 -- 172.21.15.90:0/558136959 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6580004c10 con 0x55a698378200 2024-04-02T14:45:32.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.243+0000 7f6573400640 1 -- 172.21.15.90:0/558136959 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f65800035f0 con 0x55a698378200 2024-04-02T14:45:32.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.243+0000 7f659b969740 10 client.28775 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:45:32.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.243+0000 7f659b969740 10 client.28775 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2024-04-02T14:45:32.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.243+0000 7f659b969740 20 client.28775 populate_metadata read hostname 'smithi090' 2024-04-02T14:45:32.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.243+0000 7f659b969740 1 --2- 172.21.15.90:0/558136959 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x55a6984108b0 0x55a698412cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:32.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.243+0000 7f659b969740 4 client.28775 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-02T14:45:32.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.243+0000 7f659b969740 1 -- 172.21.15.90:0/558136959 --> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55a6982e0c30 con 0x55a6984108b0 2024-04-02T14:45:32.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.244+0000 7f6591e00640 1 --2- 172.21.15.90:0/558136959 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x55a6984108b0 0x55a698412cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:32.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.248+0000 7f6591e00640 1 --2- 172.21.15.90:0/558136959 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x55a6984108b0 0x55a698412cd0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=55d4cd5934998703 server_cookie=2b2e48852a7db509 in_seq=0 out_seq=0 2024-04-02T14:45:32.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.250+0000 7f6573400640 10 client.28775 ms_handle_connect on v2:172.21.15.90:6838/1498915870 2024-04-02T14:45:32.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.250+0000 7f6573400640 1 -- 172.21.15.90:0/558136959 <== mds.1 v2:172.21.15.90:6838/1498915870 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55a6982e0c30 con 0x55a6984108b0 2024-04-02T14:45:32.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.250+0000 7f6573400640 10 client.28775 handle_command_reply: tid=0 2024-04-02T14:45:32.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.252+0000 7f659b969740 1 client.28775 shutdown 2024-04-02T14:45:32.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.252+0000 7f659b969740 1 -- 172.21.15.90:0/558136959 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a698378200 msgr2=0x55a69836c6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:45:32.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.252+0000 7f659b969740 1 --2- 172.21.15.90:0/558136959 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a698378200 0x55a69836c6e0 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f658002ce30 tx=0x7f6580005870 comp rx=0 tx=0).stop 2024-04-02T14:45:32.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.252+0000 7f659b969740 1 -- 172.21.15.90:0/558136959 shutdown_connections 2024-04-02T14:45:32.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.252+0000 7f659b969740 1 --2- 172.21.15.90:0/558136959 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x55a6984108b0 0x55a698412cd0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:32.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.252+0000 7f659b969740 1 --2- 172.21.15.90:0/558136959 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a698380890 0x55a69840e990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:32.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.252+0000 7f6573400640 0 client.28775 ms_handle_reset on v2:172.21.15.90:6838/1498915870 2024-04-02T14:45:32.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.252+0000 7f659b969740 1 --2- 172.21.15.90:0/558136959 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a698378200 0x55a69836c6e0 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:32.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.253+0000 7f659b969740 1 --2- 172.21.15.90:0/558136959 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a698377520 0x55a69836c1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:32.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.253+0000 7f659b969740 1 -- 172.21.15.90:0/558136959 >> 172.21.15.90:0/558136959 conn(0x55a698335a50 msgr2=0x55a69839c850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:45:32.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.253+0000 7f659b969740 1 -- 172.21.15.90:0/558136959 shutdown_connections 2024-04-02T14:45:32.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.253+0000 7f659b969740 1 -- 172.21.15.90:0/558136959 wait complete. 2024-04-02T14:45:32.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.253+0000 7f659b969740 20 client.28775 trim_cache size 0 max 16384 2024-04-02T14:45:32.251 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-02T14:45:32.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.254+0000 7f6598c00640 1 -- 172.21.15.90:0/2254826613 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f65780a3f20 msgr2=0x7f65780a63e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:45:32.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.254+0000 7f6598c00640 1 --2- 172.21.15.90:0/2254826613 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f65780a3f20 0x7f65780a63e0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f658c01b040 tx=0x7f658c004f90 comp rx=0 tx=0).stop 2024-04-02T14:45:32.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.254+0000 7f6598c00640 1 -- 172.21.15.90:0/2254826613 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6594068d00 msgr2=0x7f659419a5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:45:32.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.254+0000 7f6598c00640 1 --2- 172.21.15.90:0/2254826613 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6594068d00 0x7f659419a5a0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f658400ea60 tx=0x7f658400ef20 comp rx=0 tx=0).stop 2024-04-02T14:45:32.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.255+0000 7f6598c00640 1 -- 172.21.15.90:0/2254826613 shutdown_connections 2024-04-02T14:45:32.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.255+0000 7f6598c00640 1 --2- 172.21.15.90:0/2254826613 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f65780a3f20 0x7f65780a63e0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:32.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.255+0000 7f6598c00640 1 --2- 172.21.15.90:0/2254826613 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f659410e4a0 0x7f659419ee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:32.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.255+0000 7f6598c00640 1 --2- 172.21.15.90:0/2254826613 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6594069620 0x7f659419aae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:32.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.255+0000 7f6598c00640 1 --2- 172.21.15.90:0/2254826613 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6594068d00 0x7f659419a5a0 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:32.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.255+0000 7f6598c00640 1 -- 172.21.15.90:0/2254826613 >> 172.21.15.90:0/2254826613 conn(0x7f6594077b00 msgr2=0x7f659410fa40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:45:32.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.255+0000 7f6598c00640 1 -- 172.21.15.90:0/2254826613 shutdown_connections 2024-04-02T14:45:32.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.255+0000 7f6598c00640 1 -- 172.21.15.90:0/2254826613 wait complete. 2024-04-02T14:45:32.277 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-02T14:45:32.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.548+0000 7fb29be00640 1 -- 172.21.15.90:0/1326821830 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb294109040 msgr2=0x7fb2941094a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:45:32.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.548+0000 7fb29be00640 1 --2- 172.21.15.90:0/1326821830 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb294109040 0x7fb2941094a0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7fb280009900 tx=0x7fb28002bcd0 comp rx=0 tx=0).stop 2024-04-02T14:45:32.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.548+0000 7fb29be00640 1 -- 172.21.15.90:0/1326821830 shutdown_connections 2024-04-02T14:45:32.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.548+0000 7fb29be00640 1 --2- 172.21.15.90:0/1326821830 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb294116b00 0x7fb294118f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:32.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.548+0000 7fb29be00640 1 --2- 172.21.15.90:0/1326821830 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb294109040 0x7fb2941094a0 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:32.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.548+0000 7fb29be00640 1 --2- 172.21.15.90:0/1326821830 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb29410c220 0x7fb294108b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:32.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.548+0000 7fb29be00640 1 -- 172.21.15.90:0/1326821830 >> 172.21.15.90:0/1326821830 conn(0x7fb29406d040 msgr2=0x7fb29406d450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:45:32.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.549+0000 7fb29be00640 1 -- 172.21.15.90:0/1326821830 shutdown_connections 2024-04-02T14:45:32.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.549+0000 7fb29be00640 1 -- 172.21.15.90:0/1326821830 wait complete. 2024-04-02T14:45:32.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.550+0000 7fb29be00640 1 Processor -- start 2024-04-02T14:45:32.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.551+0000 7fb29be00640 1 -- start start 2024-04-02T14:45:32.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.551+0000 7fb29be00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb294109040 0x7fb29410c780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:32.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.551+0000 7fb29be00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb29410c220 0x7fb29410ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:32.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.551+0000 7fb29be00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb294116b00 0x7fb29410d200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:32.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.551+0000 7fb29be00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fb29411c8b0 con 0x7fb294116b00 2024-04-02T14:45:32.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.551+0000 7fb29be00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fb29411c730 con 0x7fb29410c220 2024-04-02T14:45:32.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.551+0000 7fb29be00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fb29411ca30 con 0x7fb294109040 2024-04-02T14:45:32.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.552+0000 7fb298a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb29410c220 0x7fb29410ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:32.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.552+0000 7fb298a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb29410c220 0x7fb29410ccc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:39746/0 (socket says 172.21.15.90:39746) 2024-04-02T14:45:32.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.552+0000 7fb298a00640 1 -- 172.21.15.90:0/3955072938 learned_addr learned my addr 172.21.15.90:0/3955072938 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:45:32.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.552+0000 7fb299400640 1 --2- 172.21.15.90:0/3955072938 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb294109040 0x7fb29410c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:32.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.552+0000 7fb299e00640 1 --2- 172.21.15.90:0/3955072938 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb294116b00 0x7fb29410d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:32.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.552+0000 7fb298a00640 1 -- 172.21.15.90:0/3955072938 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb294109040 msgr2=0x7fb29410c780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:45:32.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.552+0000 7fb298a00640 1 --2- 172.21.15.90:0/3955072938 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb294109040 0x7fb29410c780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:32.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.552+0000 7fb298a00640 1 -- 172.21.15.90:0/3955072938 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb294116b00 msgr2=0x7fb29410d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:45:32.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.552+0000 7fb298a00640 1 --2- 172.21.15.90:0/3955072938 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb294116b00 0x7fb29410d200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:32.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.552+0000 7fb298a00640 1 -- 172.21.15.90:0/3955072938 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2941b3550 con 0x7fb29410c220 2024-04-02T14:45:32.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.553+0000 7fb298a00640 1 --2- 172.21.15.90:0/3955072938 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb29410c220 0x7fb29410ccc0 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7fb28002c2e0 tx=0x7fb2800042e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:45:32.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.553+0000 7fb292000640 1 -- 172.21.15.90:0/3955072938 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb280041020 con 0x7fb29410c220 2024-04-02T14:45:32.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.553+0000 7fb292000640 1 -- 172.21.15.90:0/3955072938 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb2800054e0 con 0x7fb29410c220 2024-04-02T14:45:32.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.553+0000 7fb29be00640 1 -- 172.21.15.90:0/3955072938 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2941b37e0 con 0x7fb29410c220 2024-04-02T14:45:32.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.553+0000 7fb292000640 1 -- 172.21.15.90:0/3955072938 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb280003a10 con 0x7fb29410c220 2024-04-02T14:45:32.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.554+0000 7fb29be00640 1 -- 172.21.15.90:0/3955072938 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2941b3d20 con 0x7fb29410c220 2024-04-02T14:45:32.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.557+0000 7fb292000640 1 -- 172.21.15.90:0/3955072938 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fb280044020 con 0x7fb29410c220 2024-04-02T14:45:32.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.559+0000 7fb29eb7f740 1 Processor -- start 2024-04-02T14:45:32.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.560+0000 7fb29eb7f740 1 -- start start 2024-04-02T14:45:32.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.560+0000 7fb29eb7f740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a6b13669f0 0x55a6b1365740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:32.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.561+0000 7fb29eb7f740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a6b1365d10 0x55a6b1366190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:32.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.561+0000 7fb29eb7f740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a6b136f080 0x55a6b1371470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:32.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.561+0000 7fb29eb7f740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55a6b12a6400 con 0x55a6b136f080 2024-04-02T14:45:32.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.561+0000 7fb29eb7f740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55a6b128af10 con 0x55a6b13669f0 2024-04-02T14:45:32.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.561+0000 7fb29eb7f740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55a6b12ca6f0 con 0x55a6b1365d10 2024-04-02T14:45:32.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.561+0000 7fb299e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a6b13669f0 0x55a6b1365740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:32.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.561+0000 7fb299e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a6b13669f0 0x55a6b1365740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:39754/0 (socket says 172.21.15.90:39754) 2024-04-02T14:45:32.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.561+0000 7fb299e00640 1 -- 172.21.15.90:0/1115599214 learned_addr learned my addr 172.21.15.90:0/1115599214 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:45:32.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.561+0000 7fb298a00640 1 --2- 172.21.15.90:0/1115599214 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a6b136f080 0x55a6b1371470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:32.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.562+0000 7fb298a00640 1 -- 172.21.15.90:0/1115599214 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a6b1365d10 msgr2=0x55a6b1366190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:45:32.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.562+0000 7fb298a00640 1 --2- 172.21.15.90:0/1115599214 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a6b1365d10 0x55a6b1366190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:32.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.562+0000 7fb298a00640 1 -- 172.21.15.90:0/1115599214 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a6b13669f0 msgr2=0x55a6b1365740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:45:32.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.562+0000 7fb298a00640 1 --2- 172.21.15.90:0/1115599214 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a6b13669f0 0x55a6b1365740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:32.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.562+0000 7fb298a00640 1 -- 172.21.15.90:0/1115599214 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a6b0fde3f0 con 0x55a6b136f080 2024-04-02T14:45:32.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.563+0000 7fb292000640 1 --2- 172.21.15.90:0/3955072938 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fb27c0a3de0 0x7fb27c0a62a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:32.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.563+0000 7fb292000640 1 -- 172.21.15.90:0/3955072938 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 8550+0+0 (secure 0 0 0) 0x7fb2800f1fc0 con 0x7fb29410c220 2024-04-02T14:45:32.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.567+0000 7fb298a00640 1 --2- 172.21.15.90:0/1115599214 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a6b136f080 0x55a6b1371470 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fb2800ef800 tx=0x7fb2800fa1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de05f1ecb887a9bd server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:45:32.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.568+0000 7fb286600640 1 -- 172.21.15.90:0/1115599214 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2800ba2f0 con 0x55a6b136f080 2024-04-02T14:45:32.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.568+0000 7fb286600640 1 -- 172.21.15.90:0/1115599214 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb280005690 con 0x55a6b136f080 2024-04-02T14:45:32.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.568+0000 7fb286600640 1 -- 172.21.15.90:0/1115599214 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2800bd9b0 con 0x55a6b136f080 2024-04-02T14:45:32.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.569+0000 7fb29eb7f740 1 -- 172.21.15.90:0/1115599214 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a6b136f080 msgr2=0x55a6b1371470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:45:32.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.569+0000 7fb29eb7f740 1 --2- 172.21.15.90:0/1115599214 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a6b136f080 0x55a6b1371470 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fb2800ef800 tx=0x7fb2800fa1a0 comp rx=0 tx=0).stop 2024-04-02T14:45:32.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.569+0000 7fb29eb7f740 1 -- 172.21.15.90:0/1115599214 shutdown_connections 2024-04-02T14:45:32.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.569+0000 7fb29eb7f740 1 --2- 172.21.15.90:0/1115599214 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a6b136f080 0x55a6b1371470 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:32.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.569+0000 7fb29eb7f740 1 --2- 172.21.15.90:0/1115599214 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a6b1365d10 0x55a6b1366190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:32.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.569+0000 7fb29eb7f740 1 --2- 172.21.15.90:0/1115599214 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a6b13669f0 0x55a6b1365740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:32.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.569+0000 7fb29eb7f740 1 -- 172.21.15.90:0/1115599214 >> 172.21.15.90:0/1115599214 conn(0x55a6b132ca50 msgr2=0x55a6b1360760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:45:32.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.570+0000 7fb299e00640 1 --2- 172.21.15.90:0/3955072938 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fb27c0a3de0 0x7fb27c0a62a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:32.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.570+0000 7fb29eb7f740 1 -- 172.21.15.90:0/1115599214 shutdown_connections 2024-04-02T14:45:32.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.571+0000 7fb29eb7f740 1 -- 172.21.15.90:0/1115599214 wait complete. 2024-04-02T14:45:32.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.571+0000 7fb299e00640 1 --2- 172.21.15.90:0/3955072938 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fb27c0a3de0 0x7fb27c0a62a0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fb28c003b60 tx=0x7fb28c009500 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:45:32.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.575+0000 7fb29eb7f740 1 -- start start 2024-04-02T14:45:32.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.575+0000 7fb29eb7f740 1 Processor -- start 2024-04-02T14:45:32.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.575+0000 7fb29eb7f740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a6b1365d10 0x55a6b136e940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:32.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.576+0000 7fb29eb7f740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a6b13669f0 0x55a6b136bc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:32.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.576+0000 7fb29eb7f740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a6b136c170 0x55a6b136c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:32.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.576+0000 7fb29eb7f740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55a6b12c66a0 con 0x55a6b13669f0 2024-04-02T14:45:32.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.576+0000 7fb29eb7f740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55a6b1272980 con 0x55a6b1365d10 2024-04-02T14:45:32.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.576+0000 7fb29eb7f740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55a6b1285cb0 con 0x55a6b136c170 2024-04-02T14:45:32.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.576+0000 7fb29eb7f740 10 client.0 _ll_register_callbacks cb 0x55a6b12856a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:45:32.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.576+0000 7fb299e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a6b136c170 0x55a6b136c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:32.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.576+0000 7fb299400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a6b1365d10 0x55a6b136e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:32.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.576+0000 7fb299e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a6b136c170 0x55a6b136c5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:55736/0 (socket says 172.21.15.90:55736) 2024-04-02T14:45:32.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.576+0000 7fb299e00640 1 -- 172.21.15.90:0/1577429664 learned_addr learned my addr 172.21.15.90:0/1577429664 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:45:32.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.577+0000 7fb298a00640 1 --2- 172.21.15.90:0/1577429664 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a6b13669f0 0x55a6b136bc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:32.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.577+0000 7fb299400640 1 -- 172.21.15.90:0/1577429664 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a6b136c170 msgr2=0x55a6b136c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:45:32.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.577+0000 7fb299400640 1 --2- 172.21.15.90:0/1577429664 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a6b136c170 0x55a6b136c5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:32.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.577+0000 7fb299400640 1 -- 172.21.15.90:0/1577429664 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a6b13669f0 msgr2=0x55a6b136bc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:45:32.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.577+0000 7fb299400640 1 --2- 172.21.15.90:0/1577429664 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a6b13669f0 0x55a6b136bc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:32.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.577+0000 7fb299400640 1 -- 172.21.15.90:0/1577429664 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a6b10a87d0 con 0x55a6b1365d10 2024-04-02T14:45:32.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.579+0000 7fb299400640 1 --2- 172.21.15.90:0/1577429664 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a6b1365d10 0x55a6b136e940 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7fb2880103c0 tx=0x7fb288010880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eb6c078bd51a9617 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:45:32.576 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.580+0000 7fb27aa00640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:45:32.576 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.580+0000 7fb27aa00640 1 -- 172.21.15.90:0/1577429664 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb288005220 con 0x55a6b1365d10 2024-04-02T14:45:32.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.580+0000 7fb27aa00640 1 -- 172.21.15.90:0/1577429664 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8550+0+0 (secure 0 0 0) 0x7fb2880053c0 con 0x55a6b1365d10 2024-04-02T14:45:32.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.581+0000 7fb29eb7f740 1 -- 172.21.15.90:0/1577429664 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55a6b128a040 con 0x55a6b1365d10 2024-04-02T14:45:32.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.581+0000 7fb27aa00640 1 -- 172.21.15.90:0/1577429664 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fb2880055c0 con 0x55a6b1365d10 2024-04-02T14:45:32.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.581+0000 7fb29eb7f740 10 client.28787 fetch_fsmap learned FSMap version 27 2024-04-02T14:45:32.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.581+0000 7fb29eb7f740 1 -- 172.21.15.90:0/1577429664 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55a6b137ddd0 con 0x55a6b1365d10 2024-04-02T14:45:32.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.584+0000 7fb27aa00640 1 -- 172.21.15.90:0/1577429664 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55a6b128a040 con 0x55a6b1365d10 2024-04-02T14:45:32.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.584+0000 7fb27aa00640 1 -- 172.21.15.90:0/1577429664 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fb28800f040 con 0x55a6b1365d10 2024-04-02T14:45:32.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.584+0000 7fb29eb7f740 10 client.28787 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:45:32.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.584+0000 7fb29eb7f740 10 client.28787 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:45:32.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.584+0000 7fb29eb7f740 20 client.28787 populate_metadata read hostname 'smithi090' 2024-04-02T14:45:32.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.584+0000 7fb29eb7f740 1 --2- 172.21.15.90:0/1577429664 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55a6b137f1e0 0x55a6b13815d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:32.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.584+0000 7fb29eb7f740 4 client.28787 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:45:32.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.584+0000 7fb29eb7f740 1 -- 172.21.15.90:0/1577429664 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a6b12d7c30 con 0x55a6b137f1e0 2024-04-02T14:45:32.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.585+0000 7fb298a00640 1 --2- 172.21.15.90:0/1577429664 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55a6b137f1e0 0x55a6b13815d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:32.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.585+0000 7fb27aa00640 10 client.28787 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:45:32.582 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.586+0000 7fb298a00640 1 --2- 172.21.15.90:0/1577429664 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55a6b137f1e0 0x55a6b13815d0 crc :-1 s=READY pgs=280 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8d59bbe4c7189a15 server_cookie=cd01431ae7b79994 in_seq=0 out_seq=0 2024-04-02T14:45:32.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.587+0000 7fb27aa00640 1 -- 172.21.15.90:0/1577429664 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a6b12d7c30 con 0x55a6b137f1e0 2024-04-02T14:45:32.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.588+0000 7fb27aa00640 10 client.28787 handle_command_reply: tid=0 2024-04-02T14:45:32.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.589+0000 7fb29eb7f740 1 client.28787 shutdown 2024-04-02T14:45:32.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.590+0000 7fb29eb7f740 1 -- 172.21.15.90:0/1577429664 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a6b1365d10 msgr2=0x55a6b136e940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:45:32.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.590+0000 7fb29eb7f740 1 --2- 172.21.15.90:0/1577429664 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a6b1365d10 0x55a6b136e940 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7fb2880103c0 tx=0x7fb288010880 comp rx=0 tx=0).stop 2024-04-02T14:45:32.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.590+0000 7fb29eb7f740 1 -- 172.21.15.90:0/1577429664 shutdown_connections 2024-04-02T14:45:32.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.591+0000 7fb29eb7f740 1 --2- 172.21.15.90:0/1577429664 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55a6b137f1e0 0x55a6b13815d0 crc :-1 s=READY pgs=280 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:32.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.591+0000 7fb29eb7f740 1 --2- 172.21.15.90:0/1577429664 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a6b136c170 0x55a6b136c5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:32.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.591+0000 7fb29eb7f740 1 --2- 172.21.15.90:0/1577429664 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a6b13669f0 0x55a6b136bc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:32.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.591+0000 7fb27aa00640 0 client.28787 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:45:32.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.591+0000 7fb29eb7f740 1 --2- 172.21.15.90:0/1577429664 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a6b1365d10 0x55a6b136e940 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:32.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.591+0000 7fb29eb7f740 1 -- 172.21.15.90:0/1577429664 >> 172.21.15.90:0/1577429664 conn(0x55a6b132ca50 msgr2=0x55a6b13624b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:45:32.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.592+0000 7fb29eb7f740 1 -- 172.21.15.90:0/1577429664 shutdown_connections 2024-04-02T14:45:32.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.592+0000 7fb29eb7f740 1 -- 172.21.15.90:0/1577429664 wait complete. 2024-04-02T14:45:32.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.592+0000 7fb29eb7f740 20 client.28787 trim_cache size 0 max 16384 2024-04-02T14:45:32.605 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:45:32 smithi092 ceph-mon[36497]: pgmap v330: 97 pgs: 97 active+clean; 4.4 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 8.6 MiB/s wr, 802 op/s 2024-04-02T14:45:32.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.627+0000 7fb29eb7f740 1 Processor -- start 2024-04-02T14:45:32.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.627+0000 7fb29eb7f740 1 -- start start 2024-04-02T14:45:32.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.628+0000 7fb29eb7f740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a6b1365d10 0x55a6b137a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:32.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.628+0000 7fb29eb7f740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a6b13669f0 0x55a6b136e470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:32.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.628+0000 7fb29eb7f740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a6b136c170 0x55a6b1386ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:32.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.628+0000 7fb29eb7f740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fb2941bc7b0 con 0x55a6b13669f0 2024-04-02T14:45:32.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.628+0000 7fb29eb7f740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fb2941be2f0 con 0x55a6b1365d10 2024-04-02T14:45:32.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.628+0000 7fb29eb7f740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fb2941bce80 con 0x55a6b136c170 2024-04-02T14:45:32.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.628+0000 7fb298a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a6b13669f0 0x55a6b136e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:32.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.628+0000 7fb298a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a6b13669f0 0x55a6b136e470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:42962/0 (socket says 172.21.15.90:42962) 2024-04-02T14:45:32.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.628+0000 7fb298a00640 1 -- 172.21.15.90:0/946711106 learned_addr learned my addr 172.21.15.90:0/946711106 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:45:32.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.629+0000 7fb299400640 1 --2- 172.21.15.90:0/946711106 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a6b1365d10 0x55a6b137a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:32.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.629+0000 7fb299e00640 1 --2- 172.21.15.90:0/946711106 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a6b136c170 0x55a6b1386ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:32.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.629+0000 7fb298a00640 1 -- 172.21.15.90:0/946711106 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a6b136c170 msgr2=0x55a6b1386ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:45:32.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.629+0000 7fb298a00640 1 --2- 172.21.15.90:0/946711106 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a6b136c170 0x55a6b1386ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:32.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.629+0000 7fb298a00640 1 -- 172.21.15.90:0/946711106 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a6b1365d10 msgr2=0x55a6b137a8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:45:32.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.629+0000 7fb298a00640 1 --2- 172.21.15.90:0/946711106 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a6b1365d10 0x55a6b137a8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:32.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.629+0000 7fb298a00640 1 -- 172.21.15.90:0/946711106 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a6b1364d40 con 0x55a6b13669f0 2024-04-02T14:45:32.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.629+0000 7fb299e00640 1 --2- 172.21.15.90:0/946711106 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a6b136c170 0x55a6b1386ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:45:32.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.629+0000 7fb299400640 1 --2- 172.21.15.90:0/946711106 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a6b1365d10 0x55a6b137a8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:45:32.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.629+0000 7fb298a00640 1 --2- 172.21.15.90:0/946711106 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a6b13669f0 0x55a6b136e470 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fb28002bd10 tx=0x7fb280009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e819025de38e6fe server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:45:32.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.630+0000 7fb285c00640 1 -- 172.21.15.90:0/946711106 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2800bdb90 con 0x55a6b13669f0 2024-04-02T14:45:32.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.630+0000 7fb285c00640 1 -- 172.21.15.90:0/946711106 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb2800be750 con 0x55a6b13669f0 2024-04-02T14:45:32.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.630+0000 7fb285c00640 1 -- 172.21.15.90:0/946711106 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2800bde20 con 0x55a6b13669f0 2024-04-02T14:45:32.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.631+0000 7fb29eb7f740 1 -- 172.21.15.90:0/946711106 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a6b13669f0 msgr2=0x55a6b136e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:45:32.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.631+0000 7fb29eb7f740 1 --2- 172.21.15.90:0/946711106 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a6b13669f0 0x55a6b136e470 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fb28002bd10 tx=0x7fb280009260 comp rx=0 tx=0).stop 2024-04-02T14:45:32.630 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.633+0000 7fb29eb7f740 1 -- 172.21.15.90:0/946711106 shutdown_connections 2024-04-02T14:45:32.630 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.633+0000 7fb29eb7f740 1 --2- 172.21.15.90:0/946711106 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a6b136c170 0x55a6b1386ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:32.630 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.633+0000 7fb29eb7f740 1 --2- 172.21.15.90:0/946711106 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a6b13669f0 0x55a6b136e470 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:32.630 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.633+0000 7fb29eb7f740 1 --2- 172.21.15.90:0/946711106 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a6b1365d10 0x55a6b137a8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:32.630 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.633+0000 7fb29eb7f740 1 -- 172.21.15.90:0/946711106 >> 172.21.15.90:0/946711106 conn(0x55a6b132ca50 msgr2=0x55a6b136a500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:45:32.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.634+0000 7fb29eb7f740 1 -- 172.21.15.90:0/946711106 shutdown_connections 2024-04-02T14:45:32.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.634+0000 7fb29eb7f740 1 -- 172.21.15.90:0/946711106 wait complete. 2024-04-02T14:45:32.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.635+0000 7fb29eb7f740 1 -- start start 2024-04-02T14:45:32.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.635+0000 7fb29eb7f740 1 Processor -- start 2024-04-02T14:45:32.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.637+0000 7fb29eb7f740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a6b1365d10 0x55a6b1377030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:32.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.637+0000 7fb29eb7f740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a6b13669f0 0x55a6b1368eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:32.634 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.638+0000 7fb299400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a6b1365d10 0x55a6b1377030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:32.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.638+0000 7fb29eb7f740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a6b136c170 0x55a6b13693f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:32.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.638+0000 7fb29eb7f740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fb2941bdc20 con 0x55a6b13669f0 2024-04-02T14:45:32.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.638+0000 7fb29eb7f740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fb2941bd550 con 0x55a6b1365d10 2024-04-02T14:45:32.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.638+0000 7fb29eb7f740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fb2941ba7b0 con 0x55a6b136c170 2024-04-02T14:45:32.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.638+0000 7fb29eb7f740 10 client.0 _ll_register_callbacks cb 0x7fb288009c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:45:32.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.638+0000 7fb298a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a6b13669f0 0x55a6b1368eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:32.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.638+0000 7fb298a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a6b13669f0 0x55a6b1368eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:42976/0 (socket says 172.21.15.90:42976) 2024-04-02T14:45:32.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.638+0000 7fb298a00640 1 -- 172.21.15.90:0/1639680251 learned_addr learned my addr 172.21.15.90:0/1639680251 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:45:32.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.639+0000 7fb299e00640 1 --2- 172.21.15.90:0/1639680251 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a6b136c170 0x55a6b13693f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:32.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.639+0000 7fb299e00640 1 -- 172.21.15.90:0/1639680251 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a6b1365d10 msgr2=0x55a6b1377030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:45:32.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.639+0000 7fb299e00640 1 --2- 172.21.15.90:0/1639680251 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a6b1365d10 0x55a6b1377030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:32.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.639+0000 7fb299e00640 1 -- 172.21.15.90:0/1639680251 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a6b13669f0 msgr2=0x55a6b1368eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:45:32.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.639+0000 7fb299e00640 1 --2- 172.21.15.90:0/1639680251 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a6b13669f0 0x55a6b1368eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:32.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.639+0000 7fb299e00640 1 -- 172.21.15.90:0/1639680251 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a6b1369a90 con 0x55a6b136c170 2024-04-02T14:45:32.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.639+0000 7fb298a00640 1 --2- 172.21.15.90:0/1639680251 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a6b13669f0 0x55a6b1368eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:45:32.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.640+0000 7fb299e00640 1 --2- 172.21.15.90:0/1639680251 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a6b136c170 0x55a6b13693f0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x55a6b137ae90 tx=0x7fb28c004ab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=10d4d257c528285b server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:45:32.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.640+0000 7fb27aa00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:45:32.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.641+0000 7fb27aa00640 1 -- 172.21.15.90:0/1639680251 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb28c01d8e0 con 0x55a6b136c170 2024-04-02T14:45:32.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.641+0000 7fb27aa00640 1 -- 172.21.15.90:0/1639680251 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8550+0+0 (secure 0 0 0) 0x7fb28c017030 con 0x55a6b136c170 2024-04-02T14:45:32.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.641+0000 7fb29eb7f740 1 -- 172.21.15.90:0/1639680251 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb2800be750 con 0x55a6b136c170 2024-04-02T14:45:32.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.642+0000 7fb27aa00640 1 -- 172.21.15.90:0/1639680251 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fb28c01d8e0 con 0x55a6b136c170 2024-04-02T14:45:32.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.642+0000 7fb29eb7f740 10 client.28621 fetch_fsmap learned FSMap version 27 2024-04-02T14:45:32.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.642+0000 7fb29eb7f740 1 -- 172.21.15.90:0/1639680251 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55a6b13e5760 con 0x55a6b136c170 2024-04-02T14:45:32.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.642+0000 7fb27aa00640 1 -- 172.21.15.90:0/1639680251 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb28c02d070 con 0x55a6b136c170 2024-04-02T14:45:32.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.642+0000 7fb27aa00640 1 -- 172.21.15.90:0/1639680251 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fb28c032cd0 con 0x55a6b136c170 2024-04-02T14:45:32.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.643+0000 7fb29eb7f740 10 client.28621 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:45:32.640 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.643+0000 7fb29eb7f740 10 client.28621 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:45:32.640 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.643+0000 7fb29eb7f740 20 client.28621 populate_metadata read hostname 'smithi090' 2024-04-02T14:45:32.640 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.643+0000 7fb29eb7f740 1 --2- 172.21.15.90:0/1639680251 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55a6b13e5dc0 0x55a6b13e81e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:32.640 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.643+0000 7fb29eb7f740 4 client.28621 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-02T14:45:32.640 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.643+0000 7fb29eb7f740 1 -- 172.21.15.90:0/1639680251 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7fb2800be750 con 0x55a6b13e5dc0 2024-04-02T14:45:32.640 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.643+0000 7fb299400640 1 --2- 172.21.15.90:0/1639680251 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55a6b13e5dc0 0x55a6b13e81e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:32.640 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.644+0000 7fb299400640 1 --2- 172.21.15.90:0/1639680251 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55a6b13e5dc0 0x55a6b13e81e0 crc :-1 s=READY pgs=281 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8e8d845a9fa4ef6d server_cookie=7f8424d57b0db15 in_seq=0 out_seq=0 2024-04-02T14:45:32.640 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.644+0000 7fb27aa00640 10 client.28621 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:45:32.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.644+0000 7fb27aa00640 1 -- 172.21.15.90:0/1639680251 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7fb2800be750 con 0x55a6b13e5dc0 2024-04-02T14:45:32.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.644+0000 7fb27aa00640 10 client.28621 handle_command_reply: tid=0 2024-04-02T14:45:32.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.644+0000 7fb29eb7f740 1 client.28621 shutdown 2024-04-02T14:45:32.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.645+0000 7fb29eb7f740 1 -- 172.21.15.90:0/1639680251 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a6b136c170 msgr2=0x55a6b13693f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:45:32.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.645+0000 7fb29eb7f740 1 --2- 172.21.15.90:0/1639680251 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a6b136c170 0x55a6b13693f0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x55a6b137ae90 tx=0x7fb28c004ab0 comp rx=0 tx=0).stop 2024-04-02T14:45:32.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.645+0000 7fb29eb7f740 1 -- 172.21.15.90:0/1639680251 shutdown_connections 2024-04-02T14:45:32.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.645+0000 7fb29eb7f740 1 --2- 172.21.15.90:0/1639680251 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55a6b13e5dc0 0x55a6b13e81e0 crc :-1 s=READY pgs=281 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:32.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.645+0000 7fb29eb7f740 1 --2- 172.21.15.90:0/1639680251 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a6b136c170 0x55a6b13693f0 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:32.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.645+0000 7fb29eb7f740 1 --2- 172.21.15.90:0/1639680251 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a6b13669f0 0x55a6b1368eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:32.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.645+0000 7fb29eb7f740 1 --2- 172.21.15.90:0/1639680251 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a6b1365d10 0x55a6b1377030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:32.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.645+0000 7fb29eb7f740 1 -- 172.21.15.90:0/1639680251 >> 172.21.15.90:0/1639680251 conn(0x55a6b132ca50 msgr2=0x55a6b136a0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:45:32.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.645+0000 7fb27aa00640 0 client.28621 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:45:32.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.646+0000 7fb29eb7f740 1 -- 172.21.15.90:0/1639680251 shutdown_connections 2024-04-02T14:45:32.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.646+0000 7fb29eb7f740 1 -- 172.21.15.90:0/1639680251 wait complete. 2024-04-02T14:45:32.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.646+0000 7fb29eb7f740 20 client.28621 trim_cache size 0 max 16384 2024-04-02T14:45:32.644 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-02T14:45:32.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.648+0000 7fb29be00640 1 -- 172.21.15.90:0/3955072938 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fb27c0a3de0 msgr2=0x7fb27c0a62a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:45:32.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.648+0000 7fb29be00640 1 --2- 172.21.15.90:0/3955072938 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fb27c0a3de0 0x7fb27c0a62a0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fb28c003b60 tx=0x7fb28c009500 comp rx=0 tx=0).stop 2024-04-02T14:45:32.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.648+0000 7fb29be00640 1 -- 172.21.15.90:0/3955072938 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb29410c220 msgr2=0x7fb29410ccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:45:32.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.648+0000 7fb29be00640 1 --2- 172.21.15.90:0/3955072938 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb29410c220 0x7fb29410ccc0 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7fb28002c2e0 tx=0x7fb2800042e0 comp rx=0 tx=0).stop 2024-04-02T14:45:32.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.649+0000 7fb29be00640 1 -- 172.21.15.90:0/3955072938 shutdown_connections 2024-04-02T14:45:32.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.649+0000 7fb29be00640 1 --2- 172.21.15.90:0/3955072938 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fb27c0a3de0 0x7fb27c0a62a0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:32.646 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.649+0000 7fb29be00640 1 --2- 172.21.15.90:0/3955072938 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb294116b00 0x7fb29410d200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:32.646 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.649+0000 7fb29be00640 1 --2- 172.21.15.90:0/3955072938 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb29410c220 0x7fb29410ccc0 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:32.646 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.649+0000 7fb29be00640 1 --2- 172.21.15.90:0/3955072938 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb294109040 0x7fb29410c780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:32.646 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.649+0000 7fb29be00640 1 -- 172.21.15.90:0/3955072938 >> 172.21.15.90:0/3955072938 conn(0x7fb29406d040 msgr2=0x7fb294118280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:45:32.646 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.649+0000 7fb29be00640 1 -- 172.21.15.90:0/3955072938 shutdown_connections 2024-04-02T14:45:32.646 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:32.649+0000 7fb29be00640 1 -- 172.21.15.90:0/3955072938 wait complete. 2024-04-02T14:45:32.674 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:45:32 smithi090 ceph-mon[31674]: pgmap v330: 97 pgs: 97 active+clean; 4.4 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 8.6 MiB/s wr, 802 op/s 2024-04-02T14:45:32.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:45:32 smithi177 ceph-mon[36431]: pgmap v330: 97 pgs: 97 active+clean; 4.4 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 8.6 MiB/s wr, 802 op/s 2024-04-02T14:45:33.679 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-02T14:45:33.680 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T14:45:33.680 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-02T14:45:33.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:33.962+0000 7f304a800640 1 -- 172.21.15.90:0/1755396913 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3044101f30 msgr2=0x7f304410d890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:45:33.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:33.962+0000 7f304a800640 1 --2- 172.21.15.90:0/1755396913 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3044101f30 0x7f304410d890 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f303c009ca0 tx=0x7f303c02c100 comp rx=0 tx=0).stop 2024-04-02T14:45:33.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:33.962+0000 7f304a800640 1 -- 172.21.15.90:0/1755396913 shutdown_connections 2024-04-02T14:45:33.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:33.962+0000 7f304a800640 1 --2- 172.21.15.90:0/1755396913 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3044101f30 0x7f304410d890 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:33.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:33.962+0000 7f304a800640 1 --2- 172.21.15.90:0/1755396913 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3044101570 0x7f30441019f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:33.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:33.962+0000 7f304a800640 1 --2- 172.21.15.90:0/1755396913 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3044105a00 0x7f3044105de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:33.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:33.962+0000 7f304a800640 1 -- 172.21.15.90:0/1755396913 >> 172.21.15.90:0/1755396913 conn(0x7f3044076b50 msgr2=0x7f30440fdee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:45:33.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:33.963+0000 7f304a800640 1 -- 172.21.15.90:0/1755396913 shutdown_connections 2024-04-02T14:45:33.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:33.963+0000 7f304a800640 1 -- 172.21.15.90:0/1755396913 wait complete. 2024-04-02T14:45:33.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:33.964+0000 7f304a800640 1 Processor -- start 2024-04-02T14:45:33.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:33.964+0000 7f304a800640 1 -- start start 2024-04-02T14:45:33.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:33.964+0000 7f304a800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3044101570 0x7f30441a0080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:33.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:33.965+0000 7f304a800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3044101f30 0x7f30441a05c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:33.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:33.965+0000 7f304a800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3044105a00 0x7f304419a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:33.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:33.965+0000 7f304a800640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f304410fd30 con 0x7f3044101570 2024-04-02T14:45:33.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:33.965+0000 7f304a800640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f304410fbb0 con 0x7f3044101f30 2024-04-02T14:45:33.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:33.965+0000 7f304a800640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f304410feb0 con 0x7f3044105a00 2024-04-02T14:45:33.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:33.965+0000 7f3042a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3044101f30 0x7f30441a05c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:33.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:33.965+0000 7f3043e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3044105a00 0x7f304419a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:33.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:33.965+0000 7f3043e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3044105a00 0x7f304419a4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:36156/0 (socket says 172.21.15.90:36156) 2024-04-02T14:45:33.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:33.965+0000 7f3043e00640 1 -- 172.21.15.90:0/535808124 learned_addr learned my addr 172.21.15.90:0/535808124 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:45:33.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:33.965+0000 7f3043400640 1 --2- 172.21.15.90:0/535808124 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3044101570 0x7f30441a0080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:33.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:33.965+0000 7f3043e00640 1 -- 172.21.15.90:0/535808124 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3044101f30 msgr2=0x7f30441a05c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:45:33.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:33.965+0000 7f3043e00640 1 --2- 172.21.15.90:0/535808124 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3044101f30 0x7f30441a05c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:33.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:33.965+0000 7f3043e00640 1 -- 172.21.15.90:0/535808124 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3044101570 msgr2=0x7f30441a0080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:45:33.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:33.965+0000 7f3043e00640 1 --2- 172.21.15.90:0/535808124 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3044101570 0x7f30441a0080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:33.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:33.965+0000 7f3043e00640 1 -- 172.21.15.90:0/535808124 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f304419acb0 con 0x7f3044105a00 2024-04-02T14:45:33.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:33.966+0000 7f3043e00640 1 --2- 172.21.15.90:0/535808124 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3044105a00 0x7f304419a4e0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f303c004af0 tx=0x7f303c034670 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:45:33.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:33.966+0000 7f303be00640 1 -- 172.21.15.90:0/535808124 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f303c039050 con 0x7f3044105a00 2024-04-02T14:45:33.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:33.966+0000 7f303be00640 1 -- 172.21.15.90:0/535808124 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f303c045070 con 0x7f3044105a00 2024-04-02T14:45:33.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:33.966+0000 7f303be00640 1 -- 172.21.15.90:0/535808124 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f303c004700 con 0x7f3044105a00 2024-04-02T14:45:33.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:33.966+0000 7f304a800640 1 -- 172.21.15.90:0/535808124 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f304419af40 con 0x7f3044105a00 2024-04-02T14:45:33.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:33.966+0000 7f304a800640 1 -- 172.21.15.90:0/535808124 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30441a6e60 con 0x7f3044105a00 2024-04-02T14:45:33.969 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:33.973+0000 7f304d40e740 1 Processor -- start 2024-04-02T14:45:33.969 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:33.973+0000 7f303be00640 1 -- 172.21.15.90:0/535808124 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f303c003970 con 0x7f3044105a00 2024-04-02T14:45:33.969 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:33.973+0000 7f304d40e740 1 -- start start 2024-04-02T14:45:33.970 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:33.973+0000 7f304d40e740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bf19a90030 0x55bf19a8ed80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:33.970 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:33.973+0000 7f304d40e740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bf19a8f350 0x55bf19a8f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:33.970 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:33.973+0000 7f304d40e740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55bf19a986c0 0x55bf19a9aab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:33.970 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:33.973+0000 7f304d40e740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55bf199f3310 con 0x55bf19a8f350 2024-04-02T14:45:33.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:33.974+0000 7f304d40e740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55bf1999b690 con 0x55bf19a986c0 2024-04-02T14:45:33.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:33.974+0000 7f304d40e740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55bf199ef2c0 con 0x55bf19a90030 2024-04-02T14:45:33.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:33.974+0000 7f3042a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bf19a90030 0x55bf19a8ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:33.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:33.974+0000 7f3042a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bf19a90030 0x55bf19a8ed80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:36164/0 (socket says 172.21.15.90:36164) 2024-04-02T14:45:33.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:33.974+0000 7f3042a00640 1 -- 172.21.15.90:0/4055671091 learned_addr learned my addr 172.21.15.90:0/4055671091 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:45:33.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:33.974+0000 7f3042a00640 1 --2- 172.21.15.90:0/4055671091 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55bf19a986c0 0x55bf19a9aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:33.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:33.974+0000 7f3043400640 1 --2- 172.21.15.90:0/4055671091 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bf19a8f350 0x55bf19a8f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:33.972 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:33.975+0000 7f3042a00640 1 -- 172.21.15.90:0/4055671091 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55bf19a986c0 msgr2=0x55bf19a9aab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:45:33.972 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:33.975+0000 7f3042a00640 1 --2- 172.21.15.90:0/4055671091 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55bf19a986c0 0x55bf19a9aab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:33.972 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:33.975+0000 7f3042a00640 1 -- 172.21.15.90:0/4055671091 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bf19a8f350 msgr2=0x55bf19a8f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:45:33.972 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:33.975+0000 7f3042a00640 1 --2- 172.21.15.90:0/4055671091 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bf19a8f350 0x55bf19a8f7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:33.972 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:33.975+0000 7f3042a00640 1 -- 172.21.15.90:0/4055671091 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bf197cf7e0 con 0x55bf19a90030 2024-04-02T14:45:33.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:33.975+0000 7f3043400640 1 --2- 172.21.15.90:0/4055671091 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bf19a8f350 0x55bf19a8f7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:45:33.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:33.976+0000 7f303be00640 1 --2- 172.21.15.90:0/535808124 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f30280a3d20 0x7f30280a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:33.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:33.976+0000 7f303be00640 1 -- 172.21.15.90:0/535808124 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 8550+0+0 (secure 0 0 0) 0x7f303c0f3080 con 0x7f3044105a00 2024-04-02T14:45:33.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:33.976+0000 7f3042a00640 1 --2- 172.21.15.90:0/4055671091 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bf19a90030 0x55bf19a8ed80 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f3030012760 tx=0x7f30300096c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3ff6493b04ab629c server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:45:33.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:33.977+0000 7f3043400640 1 --2- 172.21.15.90:0/535808124 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f30280a3d20 0x7f30280a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:33.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:33.977+0000 7f3038c00640 1 -- 172.21.15.90:0/4055671091 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3030012ce0 con 0x55bf19a90030 2024-04-02T14:45:33.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:33.977+0000 7f3038c00640 1 -- 172.21.15.90:0/4055671091 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f30300107d0 con 0x55bf19a90030 2024-04-02T14:45:33.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:33.977+0000 7f3038c00640 1 -- 172.21.15.90:0/4055671091 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3030012ce0 con 0x55bf19a90030 2024-04-02T14:45:33.976 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:33.979+0000 7f304d40e740 1 -- 172.21.15.90:0/4055671091 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bf19a90030 msgr2=0x55bf19a8ed80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:45:33.976 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:33.979+0000 7f304d40e740 1 --2- 172.21.15.90:0/4055671091 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bf19a90030 0x55bf19a8ed80 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f3030012760 tx=0x7f30300096c0 comp rx=0 tx=0).stop 2024-04-02T14:45:33.976 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:33.979+0000 7f3043400640 1 --2- 172.21.15.90:0/535808124 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f30280a3d20 0x7f30280a61e0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x55bf19a90580 tx=0x7f3034005f90 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:45:33.976 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:33.980+0000 7f304d40e740 1 -- 172.21.15.90:0/4055671091 shutdown_connections 2024-04-02T14:45:33.976 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:33.980+0000 7f304d40e740 1 --2- 172.21.15.90:0/4055671091 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55bf19a986c0 0x55bf19a9aab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:33.976 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:33.980+0000 7f304d40e740 1 --2- 172.21.15.90:0/4055671091 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bf19a8f350 0x55bf19a8f7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:33.976 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:33.980+0000 7f304d40e740 1 --2- 172.21.15.90:0/4055671091 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bf19a90030 0x55bf19a8ed80 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:33.976 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:33.980+0000 7f304d40e740 1 -- 172.21.15.90:0/4055671091 >> 172.21.15.90:0/4055671091 conn(0x55bf19a23270 msgr2=0x55bf19a89da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:45:33.977 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:33.980+0000 7f304d40e740 1 -- 172.21.15.90:0/4055671091 shutdown_connections 2024-04-02T14:45:33.977 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:33.980+0000 7f304d40e740 1 -- 172.21.15.90:0/4055671091 wait complete. 2024-04-02T14:45:33.977 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:33.981+0000 7f304d40e740 1 -- start start 2024-04-02T14:45:33.978 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:33.981+0000 7f304d40e740 1 Processor -- start 2024-04-02T14:45:33.978 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:33.982+0000 7f304d40e740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bf19a8f350 0x55bf19a95700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:33.978 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:33.982+0000 7f304d40e740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bf19a90030 0x55bf19a95c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:33.978 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:33.982+0000 7f304d40e740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55bf19a986c0 0x55bf19a96180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:33.979 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:33.982+0000 7f304d40e740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55bf19a9b530 con 0x55bf19a90030 2024-04-02T14:45:33.979 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:33.982+0000 7f304d40e740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55bf199aea60 con 0x55bf19a986c0 2024-04-02T14:45:33.979 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:33.982+0000 7f304d40e740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55bf19a9b6b0 con 0x55bf19a8f350 2024-04-02T14:45:33.980 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:33.983+0000 7f304d40e740 10 client.0 _ll_register_callbacks cb 0x55bf199b4030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:45:33.980 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:33.983+0000 7f3043e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55bf19a986c0 0x55bf19a96180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:33.980 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:33.983+0000 7f3043400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bf19a8f350 0x55bf19a95700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:33.980 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:33.983+0000 7f3043e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55bf19a986c0 0x55bf19a96180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:56526/0 (socket says 172.21.15.90:56526) 2024-04-02T14:45:33.980 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:33.983+0000 7f3043400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bf19a8f350 0x55bf19a95700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:36180/0 (socket says 172.21.15.90:36180) 2024-04-02T14:45:33.980 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:33.983+0000 7f3043e00640 1 -- 172.21.15.90:0/306874710 learned_addr learned my addr 172.21.15.90:0/306874710 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:45:33.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:33.983+0000 7f3043e00640 1 -- 172.21.15.90:0/306874710 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bf19a8f350 msgr2=0x55bf19a95700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:45:33.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:33.983+0000 7f3043e00640 1 --2- 172.21.15.90:0/306874710 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bf19a8f350 0x55bf19a95700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:33.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:33.983+0000 7f3043e00640 1 -- 172.21.15.90:0/306874710 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bf19a90030 msgr2=0x55bf19a95c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:45:33.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:33.983+0000 7f3042a00640 1 --2- 172.21.15.90:0/306874710 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bf19a90030 0x55bf19a95c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:33.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:33.984+0000 7f3043e00640 1 --2- 172.21.15.90:0/306874710 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bf19a90030 0x55bf19a95c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:33.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:33.984+0000 7f3043e00640 1 -- 172.21.15.90:0/306874710 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bf1982ff70 con 0x55bf19a986c0 2024-04-02T14:45:33.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:33.984+0000 7f3042a00640 1 --2- 172.21.15.90:0/306874710 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bf19a90030 0x55bf19a95c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:45:33.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:33.985+0000 7f3043e00640 1 --2- 172.21.15.90:0/306874710 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55bf19a986c0 0x55bf19a96180 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f303c0c77f0 tx=0x7f303c0349e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2c6e58184efab175 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:45:33.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:33.985+0000 7f302d600640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:45:33.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:33.985+0000 7f302d600640 1 -- 172.21.15.90:0/306874710 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f303c0bf7b0 con 0x55bf19a986c0 2024-04-02T14:45:33.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:33.985+0000 7f302d600640 1 -- 172.21.15.90:0/306874710 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8550+0+0 (secure 0 0 0) 0x7f303c0c30c0 con 0x55bf19a986c0 2024-04-02T14:45:33.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:33.986+0000 7f304d40e740 1 -- 172.21.15.90:0/306874710 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f30300107d0 con 0x55bf19a986c0 2024-04-02T14:45:33.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:33.986+0000 7f302d600640 1 -- 172.21.15.90:0/306874710 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f303c0bf0b0 con 0x55bf19a986c0 2024-04-02T14:45:33.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:33.986+0000 7f304d40e740 10 client.28805 fetch_fsmap learned FSMap version 27 2024-04-02T14:45:33.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:33.986+0000 7f304d40e740 1 -- 172.21.15.90:0/306874710 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55bf196c7170 con 0x55bf19a986c0 2024-04-02T14:45:33.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:33.987+0000 7f302d600640 1 -- 172.21.15.90:0/306874710 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f30300107d0 con 0x55bf19a986c0 2024-04-02T14:45:33.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:33.987+0000 7f302d600640 1 -- 172.21.15.90:0/306874710 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f303c0333a0 con 0x55bf19a986c0 2024-04-02T14:45:33.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:33.987+0000 7f304d40e740 10 client.28805 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:45:33.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:33.987+0000 7f304d40e740 10 client.28805 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:45:33.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:33.987+0000 7f304d40e740 20 client.28805 populate_metadata read hostname 'smithi090' 2024-04-02T14:45:33.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:33.987+0000 7f304d40e740 1 --2- 172.21.15.90:0/306874710 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55bf19aa8bb0 0x55bf19aab010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:33.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:33.987+0000 7f304d40e740 4 client.28805 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:45:33.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:33.987+0000 7f304d40e740 1 -- 172.21.15.90:0/306874710 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bf19a00850 con 0x55bf19aa8bb0 2024-04-02T14:45:33.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:33.988+0000 7f3043400640 1 --2- 172.21.15.90:0/306874710 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55bf19aa8bb0 0x55bf19aab010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:33.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:33.991+0000 7f3043400640 1 --2- 172.21.15.90:0/306874710 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55bf19aa8bb0 0x55bf19aab010 crc :-1 s=READY pgs=282 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=41140e4f131fd930 server_cookie=968671d24f1f0af in_seq=0 out_seq=0 2024-04-02T14:45:33.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:33.991+0000 7f302d600640 10 client.28805 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:45:33.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:33.992+0000 7f302d600640 1 -- 172.21.15.90:0/306874710 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55bf19a00850 con 0x55bf19aa8bb0 2024-04-02T14:45:33.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:33.993+0000 7f302d600640 10 client.28805 handle_command_reply: tid=0 2024-04-02T14:45:33.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:33.993+0000 7f304d40e740 1 client.28805 shutdown 2024-04-02T14:45:33.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:33.993+0000 7f304d40e740 1 -- 172.21.15.90:0/306874710 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55bf19a986c0 msgr2=0x55bf19a96180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:45:33.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:33.993+0000 7f304d40e740 1 --2- 172.21.15.90:0/306874710 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55bf19a986c0 0x55bf19a96180 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f303c0c77f0 tx=0x7f303c0349e0 comp rx=0 tx=0).stop 2024-04-02T14:45:33.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:33.993+0000 7f304d40e740 1 -- 172.21.15.90:0/306874710 shutdown_connections 2024-04-02T14:45:33.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:33.993+0000 7f304d40e740 1 --2- 172.21.15.90:0/306874710 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55bf19aa8bb0 0x55bf19aab010 crc :-1 s=READY pgs=282 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:33.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:33.993+0000 7f304d40e740 1 --2- 172.21.15.90:0/306874710 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55bf19a986c0 0x55bf19a96180 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:33.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:33.993+0000 7f304d40e740 1 --2- 172.21.15.90:0/306874710 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bf19a90030 0x55bf19a95c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:33.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:33.993+0000 7f304d40e740 1 --2- 172.21.15.90:0/306874710 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bf19a8f350 0x55bf19a95700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:33.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:33.994+0000 7f304d40e740 1 -- 172.21.15.90:0/306874710 >> 172.21.15.90:0/306874710 conn(0x55bf19a23270 msgr2=0x55bf19a8b260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:45:33.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:33.994+0000 7f302d600640 0 client.28805 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:45:33.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:33.994+0000 7f304d40e740 1 -- 172.21.15.90:0/306874710 shutdown_connections 2024-04-02T14:45:33.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:33.994+0000 7f304d40e740 1 -- 172.21.15.90:0/306874710 wait complete. 2024-04-02T14:45:33.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:33.994+0000 7f304d40e740 20 client.28805 trim_cache size 0 max 16384 2024-04-02T14:45:34.032 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.036+0000 7f304d40e740 1 Processor -- start 2024-04-02T14:45:34.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.037+0000 7f304d40e740 1 -- start start 2024-04-02T14:45:34.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.041+0000 7f304d40e740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bf19a8f350 0x55bf19aa4430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:34.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.041+0000 7f304d40e740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bf19a90030 0x55bf19a88ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:34.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.041+0000 7f304d40e740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55bf19a986c0 0x55bf19ab0920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:34.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.041+0000 7f304d40e740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f30441af760 con 0x55bf19a90030 2024-04-02T14:45:34.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.041+0000 7f304d40e740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f30441b12a0 con 0x55bf19a986c0 2024-04-02T14:45:34.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.041+0000 7f304d40e740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f30441afe30 con 0x55bf19a8f350 2024-04-02T14:45:34.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.042+0000 7f3043400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bf19a8f350 0x55bf19aa4430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:34.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.042+0000 7f3043400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bf19a8f350 0x55bf19aa4430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:36190/0 (socket says 172.21.15.90:36190) 2024-04-02T14:45:34.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.042+0000 7f3043400640 1 -- 172.21.15.90:0/3365115821 learned_addr learned my addr 172.21.15.90:0/3365115821 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:45:34.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.042+0000 7f3043e00640 1 --2- 172.21.15.90:0/3365115821 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55bf19a986c0 0x55bf19ab0920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:34.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.042+0000 7f3043400640 1 -- 172.21.15.90:0/3365115821 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55bf19a986c0 msgr2=0x55bf19ab0920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:45:34.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.042+0000 7f3043400640 1 --2- 172.21.15.90:0/3365115821 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55bf19a986c0 0x55bf19ab0920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:34.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.042+0000 7f3043400640 1 -- 172.21.15.90:0/3365115821 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bf19a90030 msgr2=0x55bf19a88ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:45:34.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.042+0000 7f3043400640 1 --2- 172.21.15.90:0/3365115821 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bf19a90030 0x55bf19a88ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:34.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.042+0000 7f3043400640 1 -- 172.21.15.90:0/3365115821 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bf19aab690 con 0x55bf19a8f350 2024-04-02T14:45:34.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.043+0000 7f3043400640 1 --2- 172.21.15.90:0/3365115821 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bf19a8f350 0x55bf19aa4430 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x55bf19aab8c0 tx=0x7f3034016000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2c02089542fb6a82 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:45:34.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.043+0000 7f302fe00640 1 -- 172.21.15.90:0/3365115821 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3034003e70 con 0x55bf19a8f350 2024-04-02T14:45:34.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.045+0000 7f302fe00640 1 -- 172.21.15.90:0/3365115821 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f303402a810 con 0x55bf19a8f350 2024-04-02T14:45:34.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.045+0000 7f302fe00640 1 -- 172.21.15.90:0/3365115821 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3034003e70 con 0x55bf19a8f350 2024-04-02T14:45:34.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.046+0000 7f304d40e740 1 -- 172.21.15.90:0/3365115821 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bf19a8f350 msgr2=0x55bf19aa4430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:45:34.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.046+0000 7f304d40e740 1 --2- 172.21.15.90:0/3365115821 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bf19a8f350 0x55bf19aa4430 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x55bf19aab8c0 tx=0x7f3034016000 comp rx=0 tx=0).stop 2024-04-02T14:45:34.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.046+0000 7f304d40e740 1 -- 172.21.15.90:0/3365115821 shutdown_connections 2024-04-02T14:45:34.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.046+0000 7f304d40e740 1 --2- 172.21.15.90:0/3365115821 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55bf19a986c0 0x55bf19ab0920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:34.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.046+0000 7f304d40e740 1 --2- 172.21.15.90:0/3365115821 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bf19a90030 0x55bf19a88ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:34.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.046+0000 7f304d40e740 1 --2- 172.21.15.90:0/3365115821 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bf19a8f350 0x55bf19aa4430 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:34.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.046+0000 7f304d40e740 1 -- 172.21.15.90:0/3365115821 >> 172.21.15.90:0/3365115821 conn(0x55bf19a23270 msgr2=0x55bf19a88590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:45:34.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.046+0000 7f304d40e740 1 -- 172.21.15.90:0/3365115821 shutdown_connections 2024-04-02T14:45:34.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.046+0000 7f304d40e740 1 -- 172.21.15.90:0/3365115821 wait complete. 2024-04-02T14:45:34.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.047+0000 7f304d40e740 1 -- start start 2024-04-02T14:45:34.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.047+0000 7f304d40e740 1 Processor -- start 2024-04-02T14:45:34.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.047+0000 7f304d40e740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bf19a8f350 0x55bf19aa0c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:34.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.047+0000 7f304d40e740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55bf19a90030 0x55bf19a92a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:34.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.048+0000 7f304d40e740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bf19a986c0 0x55bf19a92fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:34.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.048+0000 7f3042a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55bf19a90030 0x55bf19a92a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:34.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.048+0000 7f3042a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55bf19a90030 0x55bf19a92a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:56536/0 (socket says 172.21.15.90:56536) 2024-04-02T14:45:34.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.048+0000 7f3042a00640 1 -- 172.21.15.90:0/2855187580 learned_addr learned my addr 172.21.15.90:0/2855187580 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:45:34.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.048+0000 7f3043400640 1 --2- 172.21.15.90:0/2855187580 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bf19a8f350 0x55bf19aa0c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:34.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.048+0000 7f304d40e740 1 -- 172.21.15.90:0/2855187580 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f30441b0bd0 con 0x55bf19a8f350 2024-04-02T14:45:34.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.048+0000 7f304d40e740 1 -- 172.21.15.90:0/2855187580 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f30441b0500 con 0x55bf19a90030 2024-04-02T14:45:34.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.048+0000 7f3043e00640 1 --2- 172.21.15.90:0/2855187580 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bf19a986c0 0x55bf19a92fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:34.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.048+0000 7f304d40e740 1 -- 172.21.15.90:0/2855187580 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f30441ad760 con 0x55bf19a986c0 2024-04-02T14:45:34.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.049+0000 7f304d40e740 10 client.0 _ll_register_callbacks cb 0x7f303c005ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:45:34.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.049+0000 7f3043e00640 1 -- 172.21.15.90:0/2855187580 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55bf19a90030 msgr2=0x55bf19a92a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:45:34.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.049+0000 7f3043e00640 1 --2- 172.21.15.90:0/2855187580 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55bf19a90030 0x55bf19a92a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:34.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.049+0000 7f3043e00640 1 -- 172.21.15.90:0/2855187580 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bf19a8f350 msgr2=0x55bf19aa0c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:45:34.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.049+0000 7f3043e00640 1 --2- 172.21.15.90:0/2855187580 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bf19a8f350 0x55bf19aa0c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:34.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.049+0000 7f3043e00640 1 -- 172.21.15.90:0/2855187580 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bf19a936e0 con 0x55bf19a986c0 2024-04-02T14:45:34.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.049+0000 7f3042a00640 1 --2- 172.21.15.90:0/2855187580 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55bf19a90030 0x55bf19a92a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:45:34.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.050+0000 7f3043e00640 1 --2- 172.21.15.90:0/2855187580 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bf19a986c0 0x55bf19a92fa0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f303c0bd360 tx=0x7f303c005da0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d18e0f4844e5ebe6 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:45:34.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.050+0000 7f302d600640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:45:34.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.054+0000 7f302d600640 1 -- 172.21.15.90:0/2855187580 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f303c0bf7b0 con 0x55bf19a986c0 2024-04-02T14:45:34.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.054+0000 7f302d600640 1 -- 172.21.15.90:0/2855187580 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8550+0+0 (secure 0 0 0) 0x7f303c0d3b50 con 0x55bf19a986c0 2024-04-02T14:45:34.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.054+0000 7f304d40e740 1 -- 172.21.15.90:0/2855187580 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f303402a810 con 0x55bf19a986c0 2024-04-02T14:45:34.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.055+0000 7f302d600640 1 -- 172.21.15.90:0/2855187580 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f303c0c3460 con 0x55bf19a986c0 2024-04-02T14:45:34.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.055+0000 7f304d40e740 10 client.28639 fetch_fsmap learned FSMap version 27 2024-04-02T14:45:34.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.055+0000 7f304d40e740 1 -- 172.21.15.90:0/2855187580 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55bf19b18a70 con 0x55bf19a986c0 2024-04-02T14:45:34.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.058+0000 7f302d600640 1 -- 172.21.15.90:0/2855187580 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f303402a810 con 0x55bf19a986c0 2024-04-02T14:45:34.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.058+0000 7f302d600640 1 -- 172.21.15.90:0/2855187580 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f303c004b40 con 0x55bf19a986c0 2024-04-02T14:45:34.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.058+0000 7f304d40e740 10 client.28639 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:45:34.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.058+0000 7f304d40e740 10 client.28639 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:45:34.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.058+0000 7f304d40e740 20 client.28639 populate_metadata read hostname 'smithi090' 2024-04-02T14:45:34.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.058+0000 7f304d40e740 1 --2- 172.21.15.90:0/2855187580 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55bf19b19000 0x55bf19b1b420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:34.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.058+0000 7f304d40e740 4 client.28639 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-02T14:45:34.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.058+0000 7f304d40e740 1 -- 172.21.15.90:0/2855187580 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55bf199faba0 con 0x55bf19b19000 2024-04-02T14:45:34.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.058+0000 7f3043400640 1 --2- 172.21.15.90:0/2855187580 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55bf19b19000 0x55bf19b1b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:34.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.059+0000 7f3043400640 1 --2- 172.21.15.90:0/2855187580 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55bf19b19000 0x55bf19b1b420 crc :-1 s=READY pgs=283 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=27f9352e14409599 server_cookie=6bf9874be65eb6f8 in_seq=0 out_seq=0 2024-04-02T14:45:34.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.059+0000 7f302d600640 10 client.28639 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:45:34.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.060+0000 7f302d600640 1 -- 172.21.15.90:0/2855187580 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55bf199faba0 con 0x55bf19b19000 2024-04-02T14:45:34.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.060+0000 7f302d600640 10 client.28639 handle_command_reply: tid=0 2024-04-02T14:45:34.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.062+0000 7f304d40e740 1 client.28639 shutdown 2024-04-02T14:45:34.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.062+0000 7f304d40e740 1 -- 172.21.15.90:0/2855187580 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bf19a986c0 msgr2=0x55bf19a92fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:45:34.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.062+0000 7f304d40e740 1 --2- 172.21.15.90:0/2855187580 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bf19a986c0 0x55bf19a92fa0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f303c0bd360 tx=0x7f303c005da0 comp rx=0 tx=0).stop 2024-04-02T14:45:34.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.062+0000 7f304d40e740 1 -- 172.21.15.90:0/2855187580 shutdown_connections 2024-04-02T14:45:34.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.062+0000 7f304d40e740 1 --2- 172.21.15.90:0/2855187580 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55bf19b19000 0x55bf19b1b420 crc :-1 s=READY pgs=283 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:34.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.062+0000 7f304d40e740 1 --2- 172.21.15.90:0/2855187580 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bf19a986c0 0x55bf19a92fa0 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:34.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.062+0000 7f304d40e740 1 --2- 172.21.15.90:0/2855187580 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55bf19a90030 0x55bf19a92a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:34.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.062+0000 7f304d40e740 1 --2- 172.21.15.90:0/2855187580 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bf19a8f350 0x55bf19aa0c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:34.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.062+0000 7f304d40e740 1 -- 172.21.15.90:0/2855187580 >> 172.21.15.90:0/2855187580 conn(0x55bf19a23270 msgr2=0x55bf19a95f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:45:34.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.062+0000 7f302d600640 0 client.28639 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:45:34.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.063+0000 7f304d40e740 1 -- 172.21.15.90:0/2855187580 shutdown_connections 2024-04-02T14:45:34.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.063+0000 7f304d40e740 1 -- 172.21.15.90:0/2855187580 wait complete. 2024-04-02T14:45:34.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.063+0000 7f304d40e740 20 client.28639 trim_cache size 0 max 16384 2024-04-02T14:45:34.061 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-02T14:45:34.061 INFO:teuthology.orchestra.run.smithi090.stdout: "return_code": 0, 2024-04-02T14:45:34.061 INFO:teuthology.orchestra.run.smithi090.stdout: "scrub_tag": "e42a8350-709a-4a19-b3aa-b70abd5741dd", 2024-04-02T14:45:34.061 INFO:teuthology.orchestra.run.smithi090.stdout: "mode": "asynchronous" 2024-04-02T14:45:34.061 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-02T14:45:34.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.064+0000 7f304a800640 1 -- 172.21.15.90:0/535808124 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f30280a3d20 msgr2=0x7f30280a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:45:34.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.064+0000 7f304a800640 1 --2- 172.21.15.90:0/535808124 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f30280a3d20 0x7f30280a61e0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x55bf19a90580 tx=0x7f3034005f90 comp rx=0 tx=0).stop 2024-04-02T14:45:34.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.064+0000 7f304a800640 1 -- 172.21.15.90:0/535808124 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3044105a00 msgr2=0x7f304419a4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:45:34.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.064+0000 7f304a800640 1 --2- 172.21.15.90:0/535808124 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3044105a00 0x7f304419a4e0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f303c004af0 tx=0x7f303c034670 comp rx=0 tx=0).stop 2024-04-02T14:45:34.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.064+0000 7f304a800640 1 -- 172.21.15.90:0/535808124 shutdown_connections 2024-04-02T14:45:34.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.064+0000 7f304a800640 1 --2- 172.21.15.90:0/535808124 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f30280a3d20 0x7f30280a61e0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:34.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.064+0000 7f304a800640 1 --2- 172.21.15.90:0/535808124 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3044105a00 0x7f304419a4e0 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:34.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.064+0000 7f304a800640 1 --2- 172.21.15.90:0/535808124 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3044101f30 0x7f30441a05c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:34.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.064+0000 7f304a800640 1 --2- 172.21.15.90:0/535808124 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3044101570 0x7f30441a0080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:34.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.064+0000 7f304a800640 1 -- 172.21.15.90:0/535808124 >> 172.21.15.90:0/535808124 conn(0x7f3044076b50 msgr2=0x7f3044103800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:45:34.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.065+0000 7f304a800640 1 -- 172.21.15.90:0/535808124 shutdown_connections 2024-04-02T14:45:34.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.065+0000 7f304a800640 1 -- 172.21.15.90:0/535808124 wait complete. 2024-04-02T14:45:34.095 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-02T14:45:34.095 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-02T14:45:34.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.381+0000 7f7d80600640 1 -- 172.21.15.90:0/1017555179 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7d78103f10 msgr2=0x7f7d781042f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:45:34.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.381+0000 7f7d80600640 1 --2- 172.21.15.90:0/1017555179 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7d78103f10 0x7f7d781042f0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f7d68009900 tx=0x7f7d6802bc50 comp rx=0 tx=0).stop 2024-04-02T14:45:34.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.381+0000 7f7d80600640 1 -- 172.21.15.90:0/1017555179 shutdown_connections 2024-04-02T14:45:34.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.381+0000 7f7d80600640 1 --2- 172.21.15.90:0/1017555179 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7d78108fe0 0x7f7d78110bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:34.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.381+0000 7f7d80600640 1 --2- 172.21.15.90:0/1017555179 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7d781048c0 0x7f7d78108900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:34.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.381+0000 7f7d80600640 1 --2- 172.21.15.90:0/1017555179 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7d78103f10 0x7f7d781042f0 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:34.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.381+0000 7f7d80600640 1 -- 172.21.15.90:0/1017555179 >> 172.21.15.90:0/1017555179 conn(0x7f7d780ff750 msgr2=0x7f7d78101b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:45:34.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.381+0000 7f7d80600640 1 -- 172.21.15.90:0/1017555179 shutdown_connections 2024-04-02T14:45:34.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.381+0000 7f7d80600640 1 -- 172.21.15.90:0/1017555179 wait complete. 2024-04-02T14:45:34.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.383+0000 7f7d80600640 1 Processor -- start 2024-04-02T14:45:34.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.383+0000 7f7d80600640 1 -- start start 2024-04-02T14:45:34.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.383+0000 7f7d80600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7d78103f10 0x7f7d78110a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:34.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.383+0000 7f7d80600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7d78108fe0 0x7f7d7810db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:34.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.383+0000 7f7d80600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7d7810e070 0x7f7d7810e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:34.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.383+0000 7f7d80600640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f7d781140d0 con 0x7f7d78108fe0 2024-04-02T14:45:34.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.383+0000 7f7d80600640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f7d78113f50 con 0x7f7d7810e070 2024-04-02T14:45:34.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.383+0000 7f7d80600640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f7d78114250 con 0x7f7d78103f10 2024-04-02T14:45:34.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.384+0000 7f7d7dc00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7d78103f10 0x7f7d78110a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:34.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.384+0000 7f7d7e600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7d7810e070 0x7f7d7810e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:34.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.384+0000 7f7d7dc00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7d78103f10 0x7f7d78110a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:36214/0 (socket says 172.21.15.90:36214) 2024-04-02T14:45:34.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.384+0000 7f7d7dc00640 1 -- 172.21.15.90:0/1371741238 learned_addr learned my addr 172.21.15.90:0/1371741238 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:45:34.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.384+0000 7f7d7dc00640 1 -- 172.21.15.90:0/1371741238 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7d7810e070 msgr2=0x7f7d7810e500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:45:34.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.384+0000 7f7d7dc00640 1 --2- 172.21.15.90:0/1371741238 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7d7810e070 0x7f7d7810e500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:34.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.384+0000 7f7d7dc00640 1 -- 172.21.15.90:0/1371741238 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7d78108fe0 msgr2=0x7f7d7810db30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T14:45:34.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.384+0000 7f7d7d200640 1 --2- 172.21.15.90:0/1371741238 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7d78108fe0 0x7f7d7810db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:34.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.384+0000 7f7d7dc00640 1 --2- 172.21.15.90:0/1371741238 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7d78108fe0 0x7f7d7810db30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:34.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.384+0000 7f7d7dc00640 1 -- 172.21.15.90:0/1371741238 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d781ab050 con 0x7f7d78103f10 2024-04-02T14:45:34.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.385+0000 7f7d7dc00640 1 --2- 172.21.15.90:0/1371741238 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7d78103f10 0x7f7d78110a90 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f7d68036040 tx=0x7f7d6802cbd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:45:34.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.385+0000 7f7d7d200640 1 --2- 172.21.15.90:0/1371741238 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7d78108fe0 0x7f7d7810db30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:45:34.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.385+0000 7f7d76000640 1 -- 172.21.15.90:0/1371741238 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d68043020 con 0x7f7d78103f10 2024-04-02T14:45:34.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.385+0000 7f7d76000640 1 -- 172.21.15.90:0/1371741238 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7d680054e0 con 0x7f7d78103f10 2024-04-02T14:45:34.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.385+0000 7f7d76000640 1 -- 172.21.15.90:0/1371741238 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d68043020 con 0x7f7d78103f10 2024-04-02T14:45:34.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.385+0000 7f7d80600640 1 -- 172.21.15.90:0/1371741238 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d781ab2e0 con 0x7f7d78103f10 2024-04-02T14:45:34.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.386+0000 7f7d80600640 1 -- 172.21.15.90:0/1371741238 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d781ab720 con 0x7f7d78103f10 2024-04-02T14:45:34.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.390+0000 7f7d76000640 1 -- 172.21.15.90:0/1371741238 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f7d68046020 con 0x7f7d78103f10 2024-04-02T14:45:34.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.392+0000 7f7d83226740 1 Processor -- start 2024-04-02T14:45:34.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.392+0000 7f7d83226740 1 -- start start 2024-04-02T14:45:34.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.392+0000 7f7d76000640 1 --2- 172.21.15.90:0/1371741238 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f7d600a3fa0 0x7f7d600a6460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:34.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.393+0000 7f7d76000640 1 -- 172.21.15.90:0/1371741238 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 8550+0+0 (secure 0 0 0) 0x7f7d680f2360 con 0x7f7d78103f10 2024-04-02T14:45:34.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.393+0000 7f7d83226740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5588d14537b0 0x5588d1453b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:34.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.393+0000 7f7d83226740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5588d14540d0 0x5588d145c770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:34.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.393+0000 7f7d83226740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5588d145ccb0 0x5588d145f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:34.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.393+0000 7f7d83226740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5588d13b7770 con 0x5588d14540d0 2024-04-02T14:45:34.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.393+0000 7f7d83226740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x5588d135f700 con 0x5588d14537b0 2024-04-02T14:45:34.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.393+0000 7f7d83226740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5588d13b3720 con 0x5588d145ccb0 2024-04-02T14:45:34.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.393+0000 7f7d7dc00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5588d14537b0 0x5588d1453b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:34.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.393+0000 7f7d7dc00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5588d14537b0 0x5588d1453b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:56554/0 (socket says 172.21.15.90:56554) 2024-04-02T14:45:34.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.393+0000 7f7d7dc00640 1 -- 172.21.15.90:0/3555887815 learned_addr learned my addr 172.21.15.90:0/3555887815 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:45:34.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.394+0000 7f7d7d200640 1 --2- 172.21.15.90:0/3555887815 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5588d14540d0 0x5588d145c770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:34.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.394+0000 7f7d7dc00640 1 -- 172.21.15.90:0/3555887815 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5588d145ccb0 msgr2=0x5588d145f0a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-02T14:45:34.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.394+0000 7f7d7dc00640 1 --2- 172.21.15.90:0/3555887815 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5588d145ccb0 0x5588d145f0a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:34.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.394+0000 7f7d7e600640 1 --2- 172.21.15.90:0/1371741238 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f7d600a3fa0 0x7f7d600a6460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:34.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.394+0000 7f7d7dc00640 1 -- 172.21.15.90:0/3555887815 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5588d14540d0 msgr2=0x5588d145c770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:45:34.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.394+0000 7f7d7dc00640 1 --2- 172.21.15.90:0/3555887815 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5588d14540d0 0x5588d145c770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:34.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.394+0000 7f7d7dc00640 1 -- 172.21.15.90:0/3555887815 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5588d11957d0 con 0x5588d14537b0 2024-04-02T14:45:34.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.394+0000 7f7d7d200640 1 --2- 172.21.15.90:0/3555887815 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5588d14540d0 0x5588d145c770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:45:34.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.395+0000 7f7d7dc00640 1 --2- 172.21.15.90:0/3555887815 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5588d14537b0 0x5588d1453b90 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f7d680ba970 tx=0x7f7d68004ee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=65c50d50c0c7bb67 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:45:34.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.395+0000 7f7d66a00640 1 -- 172.21.15.90:0/3555887815 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d680c28c0 con 0x5588d14537b0 2024-04-02T14:45:34.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.395+0000 7f7d66a00640 1 -- 172.21.15.90:0/3555887815 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7d680c7180 con 0x5588d14537b0 2024-04-02T14:45:34.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.395+0000 7f7d66a00640 1 -- 172.21.15.90:0/3555887815 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d680efea0 con 0x5588d14537b0 2024-04-02T14:45:34.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.396+0000 7f7d83226740 1 -- 172.21.15.90:0/3555887815 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5588d14537b0 msgr2=0x5588d1453b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:45:34.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.396+0000 7f7d83226740 1 --2- 172.21.15.90:0/3555887815 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5588d14537b0 0x5588d1453b90 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f7d680ba970 tx=0x7f7d68004ee0 comp rx=0 tx=0).stop 2024-04-02T14:45:34.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.396+0000 7f7d7e600640 1 --2- 172.21.15.90:0/1371741238 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f7d600a3fa0 0x7f7d600a6460 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x5588d144f070 tx=0x7f7d70009500 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:45:34.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.396+0000 7f7d83226740 1 -- 172.21.15.90:0/3555887815 shutdown_connections 2024-04-02T14:45:34.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.396+0000 7f7d83226740 1 --2- 172.21.15.90:0/3555887815 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5588d145ccb0 0x5588d145f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:34.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.396+0000 7f7d83226740 1 --2- 172.21.15.90:0/3555887815 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5588d14540d0 0x5588d145c770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:34.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.396+0000 7f7d83226740 1 --2- 172.21.15.90:0/3555887815 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5588d14537b0 0x5588d1453b90 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:34.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.397+0000 7f7d83226740 1 -- 172.21.15.90:0/3555887815 >> 172.21.15.90:0/3555887815 conn(0x5588d144dee0 msgr2=0x5588d144e2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:45:34.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.397+0000 7f7d83226740 1 -- 172.21.15.90:0/3555887815 shutdown_connections 2024-04-02T14:45:34.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.397+0000 7f7d83226740 1 -- 172.21.15.90:0/3555887815 wait complete. 2024-04-02T14:45:34.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.397+0000 7f7d83226740 1 -- start start 2024-04-02T14:45:34.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.398+0000 7f7d83226740 1 Processor -- start 2024-04-02T14:45:34.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.398+0000 7f7d83226740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5588d14537b0 0x5588d1459e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:34.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.398+0000 7f7d83226740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5588d14540d0 0x5588d145a340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:34.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.398+0000 7f7d83226740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5588d145ccb0 0x5588d145a880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:34.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.398+0000 7f7d83226740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5588d145f860 con 0x5588d14540d0 2024-04-02T14:45:34.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.398+0000 7f7d83226740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x5588d1372ac0 con 0x5588d14537b0 2024-04-02T14:45:34.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.398+0000 7f7d83226740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5588d145f9e0 con 0x5588d145ccb0 2024-04-02T14:45:34.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.398+0000 7f7d83226740 10 client.0 _ll_register_callbacks cb 0x5588d1375180 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:45:34.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.399+0000 7f7d7dc00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5588d14537b0 0x5588d1459e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:34.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.399+0000 7f7d7dc00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5588d14537b0 0x5588d1459e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:56564/0 (socket says 172.21.15.90:56564) 2024-04-02T14:45:34.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.399+0000 7f7d7dc00640 1 -- 172.21.15.90:0/1211166178 learned_addr learned my addr 172.21.15.90:0/1211166178 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:45:34.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.399+0000 7f7d7d200640 1 --2- 172.21.15.90:0/1211166178 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5588d14540d0 0x5588d145a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:34.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.399+0000 7f7d7e600640 1 --2- 172.21.15.90:0/1211166178 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5588d145ccb0 0x5588d145a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:34.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.399+0000 7f7d7dc00640 1 -- 172.21.15.90:0/1211166178 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5588d145ccb0 msgr2=0x5588d145a880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:45:34.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.399+0000 7f7d7dc00640 1 --2- 172.21.15.90:0/1211166178 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5588d145ccb0 0x5588d145a880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:34.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.399+0000 7f7d7dc00640 1 -- 172.21.15.90:0/1211166178 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5588d14540d0 msgr2=0x5588d145a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:45:34.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.399+0000 7f7d7dc00640 1 --2- 172.21.15.90:0/1211166178 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5588d14540d0 0x5588d145a340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:34.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.399+0000 7f7d7dc00640 1 -- 172.21.15.90:0/1211166178 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5588d10cb3f0 con 0x5588d14537b0 2024-04-02T14:45:34.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.399+0000 7f7d7e600640 1 --2- 172.21.15.90:0/1211166178 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5588d145ccb0 0x5588d145a880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:45:34.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.399+0000 7f7d7dc00640 1 --2- 172.21.15.90:0/1211166178 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5588d14537b0 0x5588d1459e00 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f7d68003630 tx=0x7f7d6802bc50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8985a4ed24094c1d server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:45:34.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.399+0000 7f7d5b400640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:45:34.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.400+0000 7f7d5b400640 1 -- 172.21.15.90:0/1211166178 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d680dc070 con 0x5588d14537b0 2024-04-02T14:45:34.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.400+0000 7f7d5b400640 1 -- 172.21.15.90:0/1211166178 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8550+0+0 (secure 0 0 0) 0x7f7d680bc810 con 0x5588d14537b0 2024-04-02T14:45:34.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.400+0000 7f7d83226740 1 -- 172.21.15.90:0/1211166178 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7d680c7180 con 0x5588d14537b0 2024-04-02T14:45:34.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.400+0000 7f7d5b400640 1 -- 172.21.15.90:0/1211166178 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f7d680d6030 con 0x5588d14537b0 2024-04-02T14:45:34.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.400+0000 7f7d83226740 10 client.28823 fetch_fsmap learned FSMap version 27 2024-04-02T14:45:34.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.400+0000 7f7d83226740 1 -- 172.21.15.90:0/1211166178 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5588d108b0f0 con 0x5588d14537b0 2024-04-02T14:45:34.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.400+0000 7f7d5b400640 1 -- 172.21.15.90:0/1211166178 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7d680c7180 con 0x5588d14537b0 2024-04-02T14:45:34.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.400+0000 7f7d5b400640 1 -- 172.21.15.90:0/1211166178 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f7d680be800 con 0x5588d14537b0 2024-04-02T14:45:34.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.400+0000 7f7d83226740 10 client.28823 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:45:34.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.400+0000 7f7d83226740 10 client.28823 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:45:34.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.400+0000 7f7d83226740 20 client.28823 populate_metadata read hostname 'smithi090' 2024-04-02T14:45:34.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.401+0000 7f7d83226740 1 --2- 172.21.15.90:0/1211166178 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5588d146c900 0x5588d146ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:34.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.401+0000 7f7d83226740 4 client.28823 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:45:34.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.401+0000 7f7d83226740 1 -- 172.21.15.90:0/1211166178 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5588d1358630 con 0x5588d146c900 2024-04-02T14:45:34.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.401+0000 7f7d7d200640 1 --2- 172.21.15.90:0/1211166178 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5588d146c900 0x5588d146ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:34.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.402+0000 7f7d7d200640 1 --2- 172.21.15.90:0/1211166178 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5588d146c900 0x5588d146ecf0 crc :-1 s=READY pgs=284 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eaac480c7eaefbf7 server_cookie=282005927c1c404e in_seq=0 out_seq=0 2024-04-02T14:45:34.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.403+0000 7f7d5b400640 10 client.28823 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:45:34.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.403+0000 7f7d5b400640 1 -- 172.21.15.90:0/1211166178 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5588d1358630 con 0x5588d146c900 2024-04-02T14:45:34.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.403+0000 7f7d5b400640 10 client.28823 handle_command_reply: tid=0 2024-04-02T14:45:34.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.403+0000 7f7d83226740 1 client.28823 shutdown 2024-04-02T14:45:34.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.403+0000 7f7d83226740 1 -- 172.21.15.90:0/1211166178 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5588d14537b0 msgr2=0x5588d1459e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:45:34.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.403+0000 7f7d83226740 1 --2- 172.21.15.90:0/1211166178 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5588d14537b0 0x5588d1459e00 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f7d68003630 tx=0x7f7d6802bc50 comp rx=0 tx=0).stop 2024-04-02T14:45:34.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.404+0000 7f7d83226740 1 -- 172.21.15.90:0/1211166178 shutdown_connections 2024-04-02T14:45:34.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.404+0000 7f7d83226740 1 --2- 172.21.15.90:0/1211166178 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5588d146c900 0x5588d146ecf0 crc :-1 s=READY pgs=284 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:34.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.404+0000 7f7d83226740 1 --2- 172.21.15.90:0/1211166178 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5588d145ccb0 0x5588d145a880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:34.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.404+0000 7f7d83226740 1 --2- 172.21.15.90:0/1211166178 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5588d14540d0 0x5588d145a340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:34.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.404+0000 7f7d83226740 1 --2- 172.21.15.90:0/1211166178 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5588d14537b0 0x5588d1459e00 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:34.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.404+0000 7f7d83226740 1 -- 172.21.15.90:0/1211166178 >> 172.21.15.90:0/1211166178 conn(0x5588d144dee0 msgr2=0x5588d144fc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:45:34.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.404+0000 7f7d5b400640 0 client.28823 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:45:34.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.404+0000 7f7d83226740 1 -- 172.21.15.90:0/1211166178 shutdown_connections 2024-04-02T14:45:34.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.404+0000 7f7d83226740 1 -- 172.21.15.90:0/1211166178 wait complete. 2024-04-02T14:45:34.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.405+0000 7f7d83226740 20 client.28823 trim_cache size 0 max 16384 2024-04-02T14:45:34.436 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:45:34 smithi090 ceph-mon[31674]: pgmap v331: 97 pgs: 97 active+clean; 4.4 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 7.4 MiB/s wr, 655 op/s 2024-04-02T14:45:34.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.442+0000 7f7d83226740 1 Processor -- start 2024-04-02T14:45:34.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.442+0000 7f7d83226740 1 -- start start 2024-04-02T14:45:34.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.443+0000 7f7d83226740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5588d14537b0 0x5588d1468690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:34.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.443+0000 7f7d83226740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5588d14540d0 0x5588d145c530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:34.440 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.443+0000 7f7d83226740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5588d145ccb0 0x5588d1458fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:34.440 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.443+0000 7f7d83226740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f7d781b3f00 con 0x5588d145ccb0 2024-04-02T14:45:34.440 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.443+0000 7f7d83226740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f7d781b5a40 con 0x5588d14540d0 2024-04-02T14:45:34.440 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.443+0000 7f7d83226740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f7d781b45d0 con 0x5588d14537b0 2024-04-02T14:45:34.440 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.443+0000 7f7d7dc00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5588d14537b0 0x5588d1468690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:34.440 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.443+0000 7f7d7dc00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5588d14537b0 0x5588d1468690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:36232/0 (socket says 172.21.15.90:36232) 2024-04-02T14:45:34.440 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.443+0000 7f7d7dc00640 1 -- 172.21.15.90:0/3395048077 learned_addr learned my addr 172.21.15.90:0/3395048077 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:45:34.440 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.443+0000 7f7d7e600640 1 --2- 172.21.15.90:0/3395048077 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5588d145ccb0 0x5588d1458fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:34.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.444+0000 7f7d7e600640 1 -- 172.21.15.90:0/3395048077 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5588d14537b0 msgr2=0x5588d1468690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:45:34.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.444+0000 7f7d7e600640 1 --2- 172.21.15.90:0/3395048077 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5588d14537b0 0x5588d1468690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:34.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.444+0000 7f7d7e600640 1 -- 172.21.15.90:0/3395048077 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5588d14540d0 msgr2=0x5588d145c530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:45:34.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.444+0000 7f7d7e600640 1 --2- 172.21.15.90:0/3395048077 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5588d14540d0 0x5588d145c530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:34.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.444+0000 7f7d7e600640 1 -- 172.21.15.90:0/3395048077 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5588d144e820 con 0x5588d145ccb0 2024-04-02T14:45:34.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.445+0000 7f7d7e600640 1 --2- 172.21.15.90:0/3395048077 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5588d145ccb0 0x5588d1458fa0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f7d70003d10 tx=0x7f7d70014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9facef4fa6758c75 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:45:34.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.446+0000 7f7d66000640 1 -- 172.21.15.90:0/3395048077 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d70004b20 con 0x5588d145ccb0 2024-04-02T14:45:34.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.446+0000 7f7d66000640 1 -- 172.21.15.90:0/3395048077 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7d70004cc0 con 0x5588d145ccb0 2024-04-02T14:45:34.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.446+0000 7f7d66000640 1 -- 172.21.15.90:0/3395048077 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d70004b20 con 0x5588d145ccb0 2024-04-02T14:45:34.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.447+0000 7f7d83226740 1 -- 172.21.15.90:0/3395048077 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5588d145ccb0 msgr2=0x5588d1458fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:45:34.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.447+0000 7f7d83226740 1 --2- 172.21.15.90:0/3395048077 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5588d145ccb0 0x5588d1458fa0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f7d70003d10 tx=0x7f7d70014000 comp rx=0 tx=0).stop 2024-04-02T14:45:34.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.451+0000 7f7d83226740 1 -- 172.21.15.90:0/3395048077 shutdown_connections 2024-04-02T14:45:34.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.451+0000 7f7d83226740 1 --2- 172.21.15.90:0/3395048077 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5588d145ccb0 0x5588d1458fa0 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:34.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.451+0000 7f7d83226740 1 --2- 172.21.15.90:0/3395048077 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5588d14540d0 0x5588d145c530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:34.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.451+0000 7f7d83226740 1 --2- 172.21.15.90:0/3395048077 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5588d14537b0 0x5588d1468690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:34.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.451+0000 7f7d83226740 1 -- 172.21.15.90:0/3395048077 >> 172.21.15.90:0/3395048077 conn(0x5588d144dee0 msgr2=0x5588d144c6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:45:34.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.451+0000 7f7d83226740 1 -- 172.21.15.90:0/3395048077 shutdown_connections 2024-04-02T14:45:34.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.451+0000 7f7d83226740 1 -- 172.21.15.90:0/3395048077 wait complete. 2024-04-02T14:45:34.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.451+0000 7f7d83226740 1 -- start start 2024-04-02T14:45:34.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.452+0000 7f7d83226740 1 Processor -- start 2024-04-02T14:45:34.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.454+0000 7f7d83226740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5588d14537b0 0x5588d1465060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:34.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.456+0000 7f7d83226740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5588d14540d0 0x5588d1456860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:34.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.456+0000 7f7d7dc00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5588d14537b0 0x5588d1465060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:34.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.456+0000 7f7d7dc00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5588d14537b0 0x5588d1465060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:36240/0 (socket says 172.21.15.90:36240) 2024-04-02T14:45:34.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.456+0000 7f7d83226740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5588d145ccb0 0x5588d1456e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:34.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.457+0000 7f7d7dc00640 1 -- 172.21.15.90:0/3817269079 learned_addr learned my addr 172.21.15.90:0/3817269079 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:45:34.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.457+0000 7f7d83226740 1 -- 172.21.15.90:0/3817269079 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f7d781b5370 con 0x5588d14540d0 2024-04-02T14:45:34.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.457+0000 7f7d83226740 1 -- 172.21.15.90:0/3817269079 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f7d781b4ca0 con 0x5588d145ccb0 2024-04-02T14:45:34.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.457+0000 7f7d83226740 1 -- 172.21.15.90:0/3817269079 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f7d781b1f00 con 0x5588d14537b0 2024-04-02T14:45:34.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.457+0000 7f7d83226740 10 client.0 _ll_register_callbacks cb 0x7f7d680be050 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:45:34.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.457+0000 7f7d7e600640 1 --2- 172.21.15.90:0/3817269079 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5588d145ccb0 0x5588d1456e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:34.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.458+0000 7f7d7e600640 1 -- 172.21.15.90:0/3817269079 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5588d14537b0 msgr2=0x5588d1465060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:45:34.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.458+0000 7f7d7e600640 1 --2- 172.21.15.90:0/3817269079 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5588d14537b0 0x5588d1465060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:34.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.459+0000 7f7d7e600640 1 -- 172.21.15.90:0/3817269079 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5588d14540d0 msgr2=0x5588d1456860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:45:34.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.459+0000 7f7d7e600640 1 --2- 172.21.15.90:0/3817269079 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5588d14540d0 0x5588d1456860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:34.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.459+0000 7f7d7e600640 1 -- 172.21.15.90:0/3817269079 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5588d1457630 con 0x5588d145ccb0 2024-04-02T14:45:34.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.459+0000 7f7d7dc00640 1 --2- 172.21.15.90:0/3817269079 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5588d14537b0 0x5588d1465060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:45:34.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.459+0000 7f7d7e600640 1 --2- 172.21.15.90:0/3817269079 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5588d145ccb0 0x5588d1456e00 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f7d70003d10 tx=0x7f7d70005550 comp rx=0 tx=0).ready entity=mon.2 client_cookie=23d3759c904fbcc9 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:45:34.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.459+0000 7f7d5b400640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:45:34.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.460+0000 7f7d5b400640 1 -- 172.21.15.90:0/3817269079 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d70004a40 con 0x5588d145ccb0 2024-04-02T14:45:34.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.460+0000 7f7d5b400640 1 -- 172.21.15.90:0/3817269079 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8550+0+0 (secure 0 0 0) 0x7f7d70021040 con 0x5588d145ccb0 2024-04-02T14:45:34.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.460+0000 7f7d83226740 1 -- 172.21.15.90:0/3817269079 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7d70004cc0 con 0x5588d145ccb0 2024-04-02T14:45:34.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.462+0000 7f7d5b400640 1 -- 172.21.15.90:0/3817269079 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f7d7001c460 con 0x5588d145ccb0 2024-04-02T14:45:34.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.462+0000 7f7d83226740 10 client.28826 fetch_fsmap learned FSMap version 27 2024-04-02T14:45:34.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.462+0000 7f7d83226740 1 -- 172.21.15.90:0/3817269079 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5588d14d2e10 con 0x5588d145ccb0 2024-04-02T14:45:34.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.464+0000 7f7d5b400640 1 -- 172.21.15.90:0/3817269079 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7d70004cc0 con 0x5588d145ccb0 2024-04-02T14:45:34.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.464+0000 7f7d5b400640 1 -- 172.21.15.90:0/3817269079 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f7d70018040 con 0x5588d145ccb0 2024-04-02T14:45:34.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.464+0000 7f7d83226740 10 client.28826 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:45:34.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.464+0000 7f7d83226740 10 client.28826 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:45:34.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.464+0000 7f7d83226740 20 client.28826 populate_metadata read hostname 'smithi090' 2024-04-02T14:45:34.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.464+0000 7f7d83226740 1 --2- 172.21.15.90:0/3817269079 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5588d14d3380 0x5588d14d5790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:34.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.464+0000 7f7d83226740 4 client.28826 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-02T14:45:34.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.464+0000 7f7d83226740 1 -- 172.21.15.90:0/3817269079 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5588d13c4cb0 con 0x5588d14d3380 2024-04-02T14:45:34.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.465+0000 7f7d7dc00640 1 --2- 172.21.15.90:0/3817269079 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5588d14d3380 0x5588d14d5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:34.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.466+0000 7f7d5b400640 10 client.28826 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:45:34.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.466+0000 7f7d7dc00640 1 --2- 172.21.15.90:0/3817269079 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5588d14d3380 0x5588d14d5790 crc :-1 s=READY pgs=285 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dc12978c20f02658 server_cookie=f1a5ec25c1d17efe in_seq=0 out_seq=0 2024-04-02T14:45:34.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.466+0000 7f7d5b400640 1 -- 172.21.15.90:0/3817269079 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x5588d13c4cb0 con 0x5588d14d3380 2024-04-02T14:45:34.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.466+0000 7f7d5b400640 10 client.28826 handle_command_reply: tid=0 2024-04-02T14:45:34.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.466+0000 7f7d83226740 1 client.28826 shutdown 2024-04-02T14:45:34.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.467+0000 7f7d83226740 1 -- 172.21.15.90:0/3817269079 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5588d145ccb0 msgr2=0x5588d1456e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:45:34.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.467+0000 7f7d83226740 1 --2- 172.21.15.90:0/3817269079 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5588d145ccb0 0x5588d1456e00 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f7d70003d10 tx=0x7f7d70005550 comp rx=0 tx=0).stop 2024-04-02T14:45:34.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.467+0000 7f7d83226740 1 -- 172.21.15.90:0/3817269079 shutdown_connections 2024-04-02T14:45:34.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.467+0000 7f7d83226740 1 --2- 172.21.15.90:0/3817269079 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5588d14d3380 0x5588d14d5790 crc :-1 s=READY pgs=285 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:34.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.467+0000 7f7d83226740 1 --2- 172.21.15.90:0/3817269079 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5588d145ccb0 0x5588d1456e00 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:34.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.467+0000 7f7d83226740 1 --2- 172.21.15.90:0/3817269079 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5588d14540d0 0x5588d1456860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:34.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.467+0000 7f7d83226740 1 --2- 172.21.15.90:0/3817269079 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5588d14537b0 0x5588d1465060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:34.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.467+0000 7f7d5b400640 0 client.28826 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:45:34.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.467+0000 7f7d83226740 1 -- 172.21.15.90:0/3817269079 >> 172.21.15.90:0/3817269079 conn(0x5588d144dee0 msgr2=0x5588d144cf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:45:34.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.468+0000 7f7d83226740 1 -- 172.21.15.90:0/3817269079 shutdown_connections 2024-04-02T14:45:34.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.469+0000 7f7d83226740 1 -- 172.21.15.90:0/3817269079 wait complete. 2024-04-02T14:45:34.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.469+0000 7f7d83226740 20 client.28826 trim_cache size 0 max 16384 2024-04-02T14:45:34.465 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-02T14:45:34.465 INFO:teuthology.orchestra.run.smithi090.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-02T14:45:34.466 INFO:teuthology.orchestra.run.smithi090.stdout: "scrubs": { 2024-04-02T14:45:34.466 INFO:teuthology.orchestra.run.smithi090.stdout: "e42a8350-709a-4a19-b3aa-b70abd5741dd": { 2024-04-02T14:45:34.466 INFO:teuthology.orchestra.run.smithi090.stdout: "path": "/", 2024-04-02T14:45:34.466 INFO:teuthology.orchestra.run.smithi090.stdout: "tag": "e42a8350-709a-4a19-b3aa-b70abd5741dd", 2024-04-02T14:45:34.466 INFO:teuthology.orchestra.run.smithi090.stdout: "options": "recursive,force" 2024-04-02T14:45:34.466 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:45:34.466 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:45:34.466 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-02T14:45:34.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.470+0000 7f7d80600640 1 -- 172.21.15.90:0/1371741238 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f7d600a3fa0 msgr2=0x7f7d600a6460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:45:34.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.470+0000 7f7d80600640 1 --2- 172.21.15.90:0/1371741238 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f7d600a3fa0 0x7f7d600a6460 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x5588d144f070 tx=0x7f7d70009500 comp rx=0 tx=0).stop 2024-04-02T14:45:34.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.470+0000 7f7d80600640 1 -- 172.21.15.90:0/1371741238 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7d78103f10 msgr2=0x7f7d78110a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:45:34.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.470+0000 7f7d80600640 1 --2- 172.21.15.90:0/1371741238 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7d78103f10 0x7f7d78110a90 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f7d68036040 tx=0x7f7d6802cbd0 comp rx=0 tx=0).stop 2024-04-02T14:45:34.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.470+0000 7f7d80600640 1 -- 172.21.15.90:0/1371741238 shutdown_connections 2024-04-02T14:45:34.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.470+0000 7f7d80600640 1 --2- 172.21.15.90:0/1371741238 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f7d600a3fa0 0x7f7d600a6460 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:34.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.470+0000 7f7d80600640 1 --2- 172.21.15.90:0/1371741238 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7d7810e070 0x7f7d7810e500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:34.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.470+0000 7f7d80600640 1 --2- 172.21.15.90:0/1371741238 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7d78108fe0 0x7f7d7810db30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:34.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.470+0000 7f7d80600640 1 --2- 172.21.15.90:0/1371741238 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7d78103f10 0x7f7d78110a90 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:34.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.471+0000 7f7d80600640 1 -- 172.21.15.90:0/1371741238 >> 172.21.15.90:0/1371741238 conn(0x7f7d780ff750 msgr2=0x7f7d78101aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:45:34.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.472+0000 7f7d80600640 1 -- 172.21.15.90:0/1371741238 shutdown_connections 2024-04-02T14:45:34.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:45:34.472+0000 7f7d80600640 1 -- 172.21.15.90:0/1371741238 wait complete. 2024-04-02T14:45:34.499 INFO:tasks.cephfs.filesystem:scrub status for tag:e42a8350-709a-4a19-b3aa-b70abd5741dd - {'path': '/', 'tag': 'e42a8350-709a-4a19-b3aa-b70abd5741dd', 'options': 'recursive,force'} 2024-04-02T14:45:34.605 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:45:34 smithi092 ceph-mon[36497]: pgmap v331: 97 pgs: 97 active+clean; 4.4 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 7.4 MiB/s wr, 655 op/s 2024-04-02T14:45:34.681 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:45:34 smithi177 ceph-mon[36431]: pgmap v331: 97 pgs: 97 active+clean; 4.4 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 7.4 MiB/s wr, 655 op/s 2024-04-02T14:45:35.506 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:45:35 smithi092 ceph-mon[36497]: scrub queued for path: / 2024-04-02T14:45:35.506 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:45:35 smithi092 ceph-mon[36497]: scrub summary: idle+waiting paths [/] 2024-04-02T14:45:35.506 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:45:35 smithi092 ceph-mon[36497]: scrub summary: active paths [/] 2024-04-02T14:45:35.506 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:45:35 smithi092 ceph-mon[36497]: scrub summary: idle+waiting paths [/] 2024-04-02T14:45:35.682 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:45:35 smithi177 ceph-mon[36431]: scrub queued for path: / 2024-04-02T14:45:35.682 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:45:35 smithi177 ceph-mon[36431]: scrub summary: idle+waiting paths [/] 2024-04-02T14:45:35.682 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:45:35 smithi177 ceph-mon[36431]: scrub summary: active paths [/] 2024-04-02T14:45:35.682 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:45:35 smithi177 ceph-mon[36431]: scrub summary: idle+waiting paths [/] 2024-04-02T14:45:35.682 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:45:35 smithi090 ceph-mon[31674]: scrub queued for path: / 2024-04-02T14:45:35.682 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:45:35 smithi090 ceph-mon[31674]: scrub summary: idle+waiting paths [/] 2024-04-02T14:45:35.683 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:45:35 smithi090 ceph-mon[31674]: scrub summary: active paths [/] 2024-04-02T14:45:35.683 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:45:35 smithi090 ceph-mon[31674]: scrub summary: idle+waiting paths [/] 2024-04-02T14:45:36.521 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:45:36 smithi092 ceph-mon[36497]: pgmap v332: 97 pgs: 97 active+clean; 4.4 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 10 MiB/s wr, 946 op/s 2024-04-02T14:45:36.681 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:45:36 smithi177 ceph-mon[36431]: pgmap v332: 97 pgs: 97 active+clean; 4.4 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 10 MiB/s wr, 946 op/s 2024-04-02T14:45:36.682 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:45:36 smithi090 ceph-mon[31674]: pgmap v332: 97 pgs: 97 active+clean; 4.4 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 10 MiB/s wr, 946 op/s 2024-04-02T14:45:36.826 INFO:teuthology.orchestra.run.smithi090.stderr:progress: 605.0 s, 1902.9 tps, lat 52.343 ms stddev 62.487 2024-04-02T14:45:37.605 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:45:37 smithi092 ceph-mon[36497]: scrub summary: idle 2024-04-02T14:45:37.681 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:45:37 smithi177 ceph-mon[36431]: scrub summary: idle 2024-04-02T14:45:37.684 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:45:37 smithi090 ceph-mon[31674]: scrub summary: idle 2024-04-02T14:45:38.605 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:45:38 smithi092 ceph-mon[36497]: pgmap v333: 97 pgs: 97 active+clean; 4.4 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 7.5 MiB/s wr, 677 op/s 2024-04-02T14:45:38.681 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:45:38 smithi177 ceph-mon[36431]: pgmap v333: 97 pgs: 97 active+clean; 4.4 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 7.5 MiB/s wr, 677 op/s 2024-04-02T14:45:38.682 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:45:38 smithi090 ceph-mon[31674]: pgmap v333: 97 pgs: 97 active+clean; 4.4 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 7.5 MiB/s wr, 677 op/s 2024-04-02T14:45:39.233 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.236+0000 7fbe70e00640 1 -- 172.21.15.90:0/1075801458 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbe6c112970 msgr2=0x7fbe6c114d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:45:39.233 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.236+0000 7fbe70e00640 1 --2- 172.21.15.90:0/1075801458 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbe6c112970 0x7fbe6c114d60 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7fbe64009c10 tx=0x7fbe6402bef0 comp rx=0 tx=0).stop 2024-04-02T14:45:39.233 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.236+0000 7fbe70e00640 1 -- 172.21.15.90:0/1075801458 shutdown_connections 2024-04-02T14:45:39.233 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.236+0000 7fbe70e00640 1 --2- 172.21.15.90:0/1075801458 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbe6c112970 0x7fbe6c114d60 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:39.234 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.236+0000 7fbe70e00640 1 --2- 172.21.15.90:0/1075801458 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbe6c076950 0x7fbe6c112430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:39.234 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.236+0000 7fbe70e00640 1 --2- 172.21.15.90:0/1075801458 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbe6c076030 0x7fbe6c076410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:39.234 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.236+0000 7fbe70e00640 1 -- 172.21.15.90:0/1075801458 >> 172.21.15.90:0/1075801458 conn(0x7fbe6c0ff7c0 msgr2=0x7fbe6c101be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:45:39.234 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.237+0000 7fbe70e00640 1 -- 172.21.15.90:0/1075801458 shutdown_connections 2024-04-02T14:45:39.234 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.237+0000 7fbe70e00640 1 -- 172.21.15.90:0/1075801458 wait complete. 2024-04-02T14:45:39.236 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.238+0000 7fbe70e00640 1 Processor -- start 2024-04-02T14:45:39.236 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.238+0000 7fbe70e00640 1 -- start start 2024-04-02T14:45:39.237 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.238+0000 7fbe70e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbe6c076030 0x7fbe6c1a4620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:39.237 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.238+0000 7fbe70e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbe6c076950 0x7fbe6c1a4b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:39.237 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.238+0000 7fbe70e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbe6c112970 0x7fbe6c19e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:39.237 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.238+0000 7fbe70e00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fbe6c1184e0 con 0x7fbe6c112970 2024-04-02T14:45:39.237 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.238+0000 7fbe70e00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fbe6c118360 con 0x7fbe6c076950 2024-04-02T14:45:39.237 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.238+0000 7fbe70e00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fbe6c118660 con 0x7fbe6c076030 2024-04-02T14:45:39.237 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.239+0000 7fbe69e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbe6c076030 0x7fbe6c1a4620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:39.237 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.239+0000 7fbe69e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbe6c076030 0x7fbe6c1a4620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:36250/0 (socket says 172.21.15.90:36250) 2024-04-02T14:45:39.237 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.239+0000 7fbe69e00640 1 -- 172.21.15.90:0/1099404043 learned_addr learned my addr 172.21.15.90:0/1099404043 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:45:39.237 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.239+0000 7fbe69400640 1 --2- 172.21.15.90:0/1099404043 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbe6c076950 0x7fbe6c1a4b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:39.237 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.239+0000 7fbe6a800640 1 --2- 172.21.15.90:0/1099404043 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbe6c112970 0x7fbe6c19e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:39.237 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.239+0000 7fbe69400640 1 -- 172.21.15.90:0/1099404043 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbe6c076030 msgr2=0x7fbe6c1a4620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:45:39.237 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.239+0000 7fbe69400640 1 --2- 172.21.15.90:0/1099404043 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbe6c076030 0x7fbe6c1a4620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:39.237 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.239+0000 7fbe69400640 1 -- 172.21.15.90:0/1099404043 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbe6c112970 msgr2=0x7fbe6c19e6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:45:39.238 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.239+0000 7fbe69400640 1 --2- 172.21.15.90:0/1099404043 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbe6c112970 0x7fbe6c19e6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:39.238 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.239+0000 7fbe69400640 1 -- 172.21.15.90:0/1099404043 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe6c19efa0 con 0x7fbe6c076950 2024-04-02T14:45:39.238 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.239+0000 7fbe6a800640 1 --2- 172.21.15.90:0/1099404043 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbe6c112970 0x7fbe6c19e6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:45:39.238 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.240+0000 7fbe69400640 1 --2- 172.21.15.90:0/1099404043 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbe6c076950 0x7fbe6c1a4b60 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7fbe5800e790 tx=0x7fbe5800ec50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:45:39.238 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.240+0000 7fbe62a00640 1 -- 172.21.15.90:0/1099404043 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe5800eea0 con 0x7fbe6c076950 2024-04-02T14:45:39.238 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.240+0000 7fbe62a00640 1 -- 172.21.15.90:0/1099404043 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbe5800ccc0 con 0x7fbe6c076950 2024-04-02T14:45:39.238 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.240+0000 7fbe62a00640 1 -- 172.21.15.90:0/1099404043 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe58018780 con 0x7fbe6c076950 2024-04-02T14:45:39.238 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.240+0000 7fbe70e00640 1 -- 172.21.15.90:0/1099404043 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe6c19f290 con 0x7fbe6c076950 2024-04-02T14:45:39.238 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.240+0000 7fbe70e00640 1 -- 172.21.15.90:0/1099404043 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe6c1ab5a0 con 0x7fbe6c076950 2024-04-02T14:45:39.241 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.245+0000 7fbe73a05740 1 Processor -- start 2024-04-02T14:45:39.241 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.245+0000 7fbe73a05740 1 -- start start 2024-04-02T14:45:39.242 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.246+0000 7fbe73a05740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562857c83eb0 0x562857c84290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:39.242 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.246+0000 7fbe62a00640 1 -- 172.21.15.90:0/1099404043 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fbe58018920 con 0x7fbe6c076950 2024-04-02T14:45:39.242 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.246+0000 7fbe73a05740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x562857c847d0 0x562857c8ce70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:39.242 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.246+0000 7fbe73a05740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562857c8d3b0 0x562857c8f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:39.243 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.246+0000 7fbe73a05740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x562857be8d70 con 0x562857c8d3b0 2024-04-02T14:45:39.243 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.246+0000 7fbe73a05740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x562857b912a0 con 0x562857c847d0 2024-04-02T14:45:39.243 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.246+0000 7fbe73a05740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x562857be4d20 con 0x562857c83eb0 2024-04-02T14:45:39.244 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.246+0000 7fbe6a800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562857c83eb0 0x562857c84290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:39.244 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.246+0000 7fbe6a800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562857c83eb0 0x562857c84290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:36266/0 (socket says 172.21.15.90:36266) 2024-04-02T14:45:39.244 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.246+0000 7fbe6a800640 1 -- 172.21.15.90:0/45982719 learned_addr learned my addr 172.21.15.90:0/45982719 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:45:39.244 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.246+0000 7fbe69e00640 1 --2- 172.21.15.90:0/45982719 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x562857c847d0 0x562857c8ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:39.244 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.246+0000 7fbe69400640 1 --2- 172.21.15.90:0/45982719 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562857c8d3b0 0x562857c8f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:39.244 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.247+0000 7fbe69e00640 1 -- 172.21.15.90:0/45982719 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562857c83eb0 msgr2=0x562857c84290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:45:39.244 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.247+0000 7fbe69e00640 1 --2- 172.21.15.90:0/45982719 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562857c83eb0 0x562857c84290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:39.244 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.247+0000 7fbe69e00640 1 -- 172.21.15.90:0/45982719 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562857c8d3b0 msgr2=0x562857c8f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:45:39.244 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.247+0000 7fbe69e00640 1 --2- 172.21.15.90:0/45982719 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562857c8d3b0 0x562857c8f7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:39.244 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.247+0000 7fbe69e00640 1 -- 172.21.15.90:0/45982719 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562857ba5840 con 0x562857c847d0 2024-04-02T14:45:39.244 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.247+0000 7fbe69400640 1 --2- 172.21.15.90:0/45982719 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562857c8d3b0 0x562857c8f7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:45:39.244 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.247+0000 7fbe6a800640 1 --2- 172.21.15.90:0/45982719 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562857c83eb0 0x562857c84290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:45:39.244 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.247+0000 7fbe69e00640 1 --2- 172.21.15.90:0/45982719 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x562857c847d0 0x562857c8ce70 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7fbe5c009c80 tx=0x7fbe5c011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fb900d096e53756c server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:45:39.245 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.247+0000 7fbe57400640 1 -- 172.21.15.90:0/45982719 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe5c004230 con 0x562857c847d0 2024-04-02T14:45:39.246 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.247+0000 7fbe57400640 1 -- 172.21.15.90:0/45982719 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbe5c0043d0 con 0x562857c847d0 2024-04-02T14:45:39.246 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.247+0000 7fbe57400640 1 -- 172.21.15.90:0/45982719 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe5c004df0 con 0x562857c847d0 2024-04-02T14:45:39.246 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.247+0000 7fbe62a00640 1 --2- 172.21.15.90:0/1099404043 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fbe500a3e40 0x7fbe500a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:39.246 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.248+0000 7fbe62a00640 1 -- 172.21.15.90:0/1099404043 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 8550+0+0 (secure 0 0 0) 0x7fbe580d0970 con 0x7fbe6c076950 2024-04-02T14:45:39.246 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.248+0000 7fbe73a05740 1 -- 172.21.15.90:0/45982719 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x562857c847d0 msgr2=0x562857c8ce70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:45:39.246 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.248+0000 7fbe73a05740 1 --2- 172.21.15.90:0/45982719 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x562857c847d0 0x562857c8ce70 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7fbe5c009c80 tx=0x7fbe5c011810 comp rx=0 tx=0).stop 2024-04-02T14:45:39.246 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.248+0000 7fbe6a800640 1 --2- 172.21.15.90:0/1099404043 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fbe500a3e40 0x7fbe500a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:39.246 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.249+0000 7fbe73a05740 1 -- 172.21.15.90:0/45982719 shutdown_connections 2024-04-02T14:45:39.246 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.249+0000 7fbe73a05740 1 --2- 172.21.15.90:0/45982719 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562857c8d3b0 0x562857c8f7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:39.246 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.249+0000 7fbe73a05740 1 --2- 172.21.15.90:0/45982719 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x562857c847d0 0x562857c8ce70 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:39.247 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.249+0000 7fbe73a05740 1 --2- 172.21.15.90:0/45982719 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562857c83eb0 0x562857c84290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:39.247 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.249+0000 7fbe73a05740 1 -- 172.21.15.90:0/45982719 >> 172.21.15.90:0/45982719 conn(0x562857c7e5e0 msgr2=0x562857c7e9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:45:39.247 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.249+0000 7fbe73a05740 1 -- 172.21.15.90:0/45982719 shutdown_connections 2024-04-02T14:45:39.247 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.249+0000 7fbe6a800640 1 --2- 172.21.15.90:0/1099404043 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fbe500a3e40 0x7fbe500a6300 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fbe64009370 tx=0x7fbe64009960 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:45:39.248 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.249+0000 7fbe73a05740 1 -- 172.21.15.90:0/45982719 wait complete. 2024-04-02T14:45:39.249 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.250+0000 7fbe73a05740 1 -- start start 2024-04-02T14:45:39.249 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.250+0000 7fbe73a05740 1 Processor -- start 2024-04-02T14:45:39.249 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.251+0000 7fbe73a05740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562857c83eb0 0x562857c8a400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:39.249 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.251+0000 7fbe73a05740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562857c847d0 0x562857c8a940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:39.249 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.251+0000 7fbe73a05740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x562857c8d3b0 0x562857c8ae80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:39.249 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.251+0000 7fbe73a05740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x562857c901a0 con 0x562857c83eb0 2024-04-02T14:45:39.249 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.251+0000 7fbe73a05740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x562857ba4660 con 0x562857c8d3b0 2024-04-02T14:45:39.249 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.251+0000 7fbe73a05740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x562857c90320 con 0x562857c847d0 2024-04-02T14:45:39.249 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.251+0000 7fbe73a05740 10 client.0 _ll_register_callbacks cb 0x562857ba4050 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:45:39.249 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.251+0000 7fbe69400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562857c847d0 0x562857c8a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:39.249 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.251+0000 7fbe6a800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x562857c8d3b0 0x562857c8ae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:39.249 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.251+0000 7fbe69e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562857c83eb0 0x562857c8a400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:39.249 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.251+0000 7fbe6a800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x562857c8d3b0 0x562857c8ae80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:56630/0 (socket says 172.21.15.90:56630) 2024-04-02T14:45:39.250 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.251+0000 7fbe6a800640 1 -- 172.21.15.90:0/3065258179 learned_addr learned my addr 172.21.15.90:0/3065258179 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:45:39.250 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.251+0000 7fbe69400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562857c847d0 0x562857c8a940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:36272/0 (socket says 172.21.15.90:36272) 2024-04-02T14:45:39.250 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.252+0000 7fbe6a800640 1 -- 172.21.15.90:0/3065258179 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562857c847d0 msgr2=0x562857c8a940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:45:39.250 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.252+0000 7fbe6a800640 1 --2- 172.21.15.90:0/3065258179 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562857c847d0 0x562857c8a940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:39.250 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.252+0000 7fbe6a800640 1 -- 172.21.15.90:0/3065258179 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562857c83eb0 msgr2=0x562857c8a400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:45:39.250 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.252+0000 7fbe6a800640 1 --2- 172.21.15.90:0/3065258179 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562857c83eb0 0x562857c8a400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:39.250 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.252+0000 7fbe6a800640 1 -- 172.21.15.90:0/3065258179 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562857bb3e80 con 0x562857c8d3b0 2024-04-02T14:45:39.250 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.252+0000 7fbe6a800640 1 --2- 172.21.15.90:0/3065258179 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x562857c8d3b0 0x562857c8ae80 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7fbe64009c10 tx=0x7fbe64009500 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8950ccde8a336c36 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:45:39.250 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.252+0000 7fbe4be00640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:45:39.250 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.252+0000 7fbe4be00640 1 -- 172.21.15.90:0/3065258179 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe64033c60 con 0x562857c8d3b0 2024-04-02T14:45:39.250 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.252+0000 7fbe4be00640 1 -- 172.21.15.90:0/3065258179 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8550+0+0 (secure 0 0 0) 0x7fbe6404c440 con 0x562857c8d3b0 2024-04-02T14:45:39.250 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.252+0000 7fbe73a05740 1 -- 172.21.15.90:0/3065258179 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbe5c0043d0 con 0x562857c8d3b0 2024-04-02T14:45:39.250 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.252+0000 7fbe4be00640 1 -- 172.21.15.90:0/3065258179 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fbe6405a050 con 0x562857c8d3b0 2024-04-02T14:45:39.251 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.252+0000 7fbe73a05740 10 client.28838 fetch_fsmap learned FSMap version 27 2024-04-02T14:45:39.251 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.252+0000 7fbe73a05740 1 -- 172.21.15.90:0/3065258179 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5628578fd210 con 0x562857c8d3b0 2024-04-02T14:45:39.251 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.253+0000 7fbe4be00640 1 -- 172.21.15.90:0/3065258179 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbe5c0043d0 con 0x562857c8d3b0 2024-04-02T14:45:39.252 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.253+0000 7fbe4be00640 1 -- 172.21.15.90:0/3065258179 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fbe6403f6a0 con 0x562857c8d3b0 2024-04-02T14:45:39.252 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.253+0000 7fbe73a05740 10 client.28838 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:45:39.252 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.253+0000 7fbe73a05740 10 client.28838 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-02T14:45:39.252 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.253+0000 7fbe73a05740 20 client.28838 populate_metadata read hostname 'smithi090' 2024-04-02T14:45:39.252 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.253+0000 7fbe73a05740 1 --2- 172.21.15.90:0/3065258179 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x562857c9d710 0x562857c9fb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:39.252 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.253+0000 7fbe73a05740 4 client.28838 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:45:39.252 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.253+0000 7fbe73a05740 1 -- 172.21.15.90:0/3065258179 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562857b98cd0 con 0x562857c9d710 2024-04-02T14:45:39.252 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.254+0000 7fbe69e00640 1 --2- 172.21.15.90:0/3065258179 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x562857c9d710 0x562857c9fb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:39.252 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.255+0000 7fbe69e00640 1 --2- 172.21.15.90:0/3065258179 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x562857c9d710 0x562857c9fb00 crc :-1 s=READY pgs=286 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4c94680194202a15 server_cookie=6a021d86b9ba115c in_seq=0 out_seq=0 2024-04-02T14:45:39.252 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.255+0000 7fbe4be00640 10 client.28838 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:45:39.253 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.256+0000 7fbe4be00640 1 -- 172.21.15.90:0/3065258179 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x562857b98cd0 con 0x562857c9d710 2024-04-02T14:45:39.253 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.256+0000 7fbe4be00640 10 client.28838 handle_command_reply: tid=0 2024-04-02T14:45:39.253 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.256+0000 7fbe73a05740 1 client.28838 shutdown 2024-04-02T14:45:39.254 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.257+0000 7fbe73a05740 1 -- 172.21.15.90:0/3065258179 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x562857c8d3b0 msgr2=0x562857c8ae80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:45:39.254 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.257+0000 7fbe73a05740 1 --2- 172.21.15.90:0/3065258179 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x562857c8d3b0 0x562857c8ae80 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7fbe64009c10 tx=0x7fbe64009500 comp rx=0 tx=0).stop 2024-04-02T14:45:39.254 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.257+0000 7fbe73a05740 1 -- 172.21.15.90:0/3065258179 shutdown_connections 2024-04-02T14:45:39.254 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.257+0000 7fbe73a05740 1 --2- 172.21.15.90:0/3065258179 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x562857c9d710 0x562857c9fb00 crc :-1 s=READY pgs=286 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:39.254 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.257+0000 7fbe73a05740 1 --2- 172.21.15.90:0/3065258179 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x562857c8d3b0 0x562857c8ae80 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:39.254 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.257+0000 7fbe73a05740 1 --2- 172.21.15.90:0/3065258179 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562857c847d0 0x562857c8a940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:39.254 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.257+0000 7fbe4be00640 0 client.28838 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:45:39.254 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.257+0000 7fbe73a05740 1 --2- 172.21.15.90:0/3065258179 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562857c83eb0 0x562857c8a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:39.254 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.257+0000 7fbe73a05740 1 -- 172.21.15.90:0/3065258179 >> 172.21.15.90:0/3065258179 conn(0x562857c7e5e0 msgr2=0x562857c80330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:45:39.254 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.257+0000 7fbe73a05740 1 -- 172.21.15.90:0/3065258179 shutdown_connections 2024-04-02T14:45:39.254 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.257+0000 7fbe73a05740 1 -- 172.21.15.90:0/3065258179 wait complete. 2024-04-02T14:45:39.254 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.257+0000 7fbe73a05740 20 client.28838 trim_cache size 0 max 16384 2024-04-02T14:45:39.283 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.285+0000 7fbe73a05740 1 Processor -- start 2024-04-02T14:45:39.283 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.285+0000 7fbe73a05740 1 -- start start 2024-04-02T14:45:39.283 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.286+0000 7fbe73a05740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562857c83eb0 0x562857c99060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:39.283 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.286+0000 7fbe73a05740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562857c847d0 0x562857ca5410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:39.283 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.286+0000 7fbe73a05740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x562857c8d3b0 0x562857ca5950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:39.283 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.286+0000 7fbe73a05740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fbe6c1b3e70 con 0x562857c83eb0 2024-04-02T14:45:39.283 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.286+0000 7fbe73a05740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fbe6c1b59b0 con 0x562857c8d3b0 2024-04-02T14:45:39.283 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.286+0000 7fbe73a05740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fbe6c1b4540 con 0x562857c847d0 2024-04-02T14:45:39.284 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.286+0000 7fbe69e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562857c83eb0 0x562857c99060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:39.284 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.286+0000 7fbe69400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562857c847d0 0x562857ca5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:39.284 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.286+0000 7fbe69400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562857c847d0 0x562857ca5410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:36280/0 (socket says 172.21.15.90:36280) 2024-04-02T14:45:39.284 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.286+0000 7fbe69e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562857c83eb0 0x562857c99060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34942/0 (socket says 172.21.15.90:34942) 2024-04-02T14:45:39.284 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.286+0000 7fbe69400640 1 -- 172.21.15.90:0/2406141372 learned_addr learned my addr 172.21.15.90:0/2406141372 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:45:39.284 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.286+0000 7fbe6a800640 1 --2- 172.21.15.90:0/2406141372 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x562857c8d3b0 0x562857ca5950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:39.284 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.286+0000 7fbe69400640 1 -- 172.21.15.90:0/2406141372 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x562857c8d3b0 msgr2=0x562857ca5950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:45:39.284 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.286+0000 7fbe69400640 1 --2- 172.21.15.90:0/2406141372 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x562857c8d3b0 0x562857ca5950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:39.284 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.286+0000 7fbe69400640 1 -- 172.21.15.90:0/2406141372 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562857c83eb0 msgr2=0x562857c99060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:45:39.284 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.286+0000 7fbe69400640 1 --2- 172.21.15.90:0/2406141372 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562857c83eb0 0x562857c99060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:39.284 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.286+0000 7fbe69400640 1 -- 172.21.15.90:0/2406141372 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5628579c9620 con 0x562857c847d0 2024-04-02T14:45:39.284 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.287+0000 7fbe69e00640 1 --2- 172.21.15.90:0/2406141372 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562857c83eb0 0x562857c99060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:45:39.284 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.286+0000 7fbe6a800640 1 --2- 172.21.15.90:0/2406141372 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x562857c8d3b0 0x562857ca5950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:45:39.284 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.287+0000 7fbe69400640 1 --2- 172.21.15.90:0/2406141372 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562857c847d0 0x562857ca5410 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7fbe580058c0 tx=0x7fbe580bd040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4dab299789c3e28f server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:45:39.285 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.287+0000 7fbe56a00640 1 -- 172.21.15.90:0/2406141372 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe580a0ca0 con 0x562857c847d0 2024-04-02T14:45:39.285 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.287+0000 7fbe56a00640 1 -- 172.21.15.90:0/2406141372 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbe580a5560 con 0x562857c847d0 2024-04-02T14:45:39.285 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.287+0000 7fbe56a00640 1 -- 172.21.15.90:0/2406141372 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe580a5330 con 0x562857c847d0 2024-04-02T14:45:39.285 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.289+0000 7fbe73a05740 1 -- 172.21.15.90:0/2406141372 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562857c847d0 msgr2=0x562857ca5410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:45:39.286 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.289+0000 7fbe73a05740 1 --2- 172.21.15.90:0/2406141372 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562857c847d0 0x562857ca5410 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7fbe580058c0 tx=0x7fbe580bd040 comp rx=0 tx=0).stop 2024-04-02T14:45:39.286 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.289+0000 7fbe73a05740 1 -- 172.21.15.90:0/2406141372 shutdown_connections 2024-04-02T14:45:39.286 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.289+0000 7fbe73a05740 1 --2- 172.21.15.90:0/2406141372 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x562857c8d3b0 0x562857ca5950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:39.286 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.289+0000 7fbe73a05740 1 --2- 172.21.15.90:0/2406141372 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562857c847d0 0x562857ca5410 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:39.286 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.289+0000 7fbe73a05740 1 --2- 172.21.15.90:0/2406141372 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562857c83eb0 0x562857c99060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:39.286 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.289+0000 7fbe73a05740 1 -- 172.21.15.90:0/2406141372 >> 172.21.15.90:0/2406141372 conn(0x562857c7e5e0 msgr2=0x562857c7cdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:45:39.286 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.290+0000 7fbe73a05740 1 -- 172.21.15.90:0/2406141372 shutdown_connections 2024-04-02T14:45:39.287 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.290+0000 7fbe73a05740 1 -- 172.21.15.90:0/2406141372 wait complete. 2024-04-02T14:45:39.287 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.290+0000 7fbe73a05740 1 -- start start 2024-04-02T14:45:39.287 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.291+0000 7fbe73a05740 1 Processor -- start 2024-04-02T14:45:39.288 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.291+0000 7fbe73a05740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562857c83eb0 0x562857c95570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:39.288 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.291+0000 7fbe73a05740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x562857c847d0 0x562857c95ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:39.288 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.291+0000 7fbe73a05740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562857c8d3b0 0x562857c92540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:39.288 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.292+0000 7fbe69e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562857c83eb0 0x562857c95570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:39.288 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.292+0000 7fbe69e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562857c83eb0 0x562857c95570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:36286/0 (socket says 172.21.15.90:36286) 2024-04-02T14:45:39.288 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.292+0000 7fbe73a05740 1 -- 172.21.15.90:0/159396511 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fbe6c1b52e0 con 0x562857c8d3b0 2024-04-02T14:45:39.288 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.292+0000 7fbe69e00640 1 -- 172.21.15.90:0/159396511 learned_addr learned my addr 172.21.15.90:0/159396511 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:45:39.289 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.292+0000 7fbe73a05740 1 -- 172.21.15.90:0/159396511 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fbe6c1b4c10 con 0x562857c847d0 2024-04-02T14:45:39.289 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.292+0000 7fbe73a05740 1 -- 172.21.15.90:0/159396511 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fbe6c1b1e70 con 0x562857c83eb0 2024-04-02T14:45:39.289 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.292+0000 7fbe73a05740 10 client.0 _ll_register_callbacks cb 0x7fbe64033840 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:45:39.289 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.292+0000 7fbe69400640 1 --2- 172.21.15.90:0/159396511 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x562857c847d0 0x562857c95ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:39.289 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.292+0000 7fbe6a800640 1 --2- 172.21.15.90:0/159396511 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562857c8d3b0 0x562857c92540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:39.289 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.292+0000 7fbe69e00640 1 -- 172.21.15.90:0/159396511 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x562857c847d0 msgr2=0x562857c95ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:45:39.289 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.292+0000 7fbe69e00640 1 --2- 172.21.15.90:0/159396511 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x562857c847d0 0x562857c95ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:39.290 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.292+0000 7fbe69e00640 1 -- 172.21.15.90:0/159396511 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562857c8d3b0 msgr2=0x562857c92540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:45:39.290 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.292+0000 7fbe69e00640 1 --2- 172.21.15.90:0/159396511 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562857c8d3b0 0x562857c92540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:39.290 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.292+0000 7fbe69e00640 1 -- 172.21.15.90:0/159396511 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562857c92b80 con 0x562857c83eb0 2024-04-02T14:45:39.290 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.292+0000 7fbe6a800640 1 --2- 172.21.15.90:0/159396511 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562857c8d3b0 0x562857c92540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:45:39.290 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.292+0000 7fbe69e00640 1 --2- 172.21.15.90:0/159396511 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562857c83eb0 0x562857c95570 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x562857c83980 tx=0x7fbe5c004850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8f0b2aae1013be91 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:45:39.290 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.292+0000 7fbe4be00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:45:39.290 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.293+0000 7fbe4be00640 1 -- 172.21.15.90:0/159396511 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe5c031060 con 0x562857c83eb0 2024-04-02T14:45:39.290 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.294+0000 7fbe4be00640 1 -- 172.21.15.90:0/159396511 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8550+0+0 (secure 0 0 0) 0x7fbe5c004240 con 0x562857c83eb0 2024-04-02T14:45:39.291 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.294+0000 7fbe73a05740 1 -- 172.21.15.90:0/159396511 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbe580a5560 con 0x562857c83eb0 2024-04-02T14:45:39.295 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.299+0000 7fbe4be00640 1 -- 172.21.15.90:0/159396511 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fbe5c01e040 con 0x562857c83eb0 2024-04-02T14:45:39.299 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.303+0000 7fbe73a05740 10 client.28666 fetch_fsmap learned FSMap version 27 2024-04-02T14:45:39.299 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.303+0000 7fbe73a05740 1 -- 172.21.15.90:0/159396511 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x562857d045c0 con 0x562857c83eb0 2024-04-02T14:45:39.300 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.304+0000 7fbe4be00640 1 -- 172.21.15.90:0/159396511 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbe580a5560 con 0x562857c83eb0 2024-04-02T14:45:39.304 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.304+0000 7fbe4be00640 1 -- 172.21.15.90:0/159396511 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fbe5c024040 con 0x562857c83eb0 2024-04-02T14:45:39.305 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.304+0000 7fbe73a05740 10 client.28666 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:45:39.305 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.304+0000 7fbe73a05740 10 client.28666 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-02T14:45:39.305 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.304+0000 7fbe73a05740 20 client.28666 populate_metadata read hostname 'smithi090' 2024-04-02T14:45:39.305 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.304+0000 7fbe73a05740 1 --2- 172.21.15.90:0/159396511 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x562857d04b60 0x562857d06f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:39.305 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.304+0000 7fbe73a05740 4 client.28666 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-02T14:45:39.305 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.304+0000 7fbe73a05740 1 -- 172.21.15.90:0/159396511 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "flush journal"}) -- 0x562857bf62b0 con 0x562857d04b60 2024-04-02T14:45:39.305 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.304+0000 7fbe69400640 1 --2- 172.21.15.90:0/159396511 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x562857d04b60 0x562857d06f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:39.307 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.310+0000 7fbe69400640 1 --2- 172.21.15.90:0/159396511 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x562857d04b60 0x562857d06f80 crc :-1 s=READY pgs=287 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c3bfd0c63d4b0483 server_cookie=54abc97261bf300c in_seq=0 out_seq=0 2024-04-02T14:45:39.307 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.311+0000 7fbe4be00640 10 client.28666 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:45:39.350 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.353+0000 7fbe4be00640 1 -- 172.21.15.90:0/159396511 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x562857bf62b0 con 0x562857d04b60 2024-04-02T14:45:39.350 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.353+0000 7fbe4be00640 10 client.28666 handle_command_reply: tid=0 2024-04-02T14:45:39.350 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.353+0000 7fbe73a05740 1 client.28666 shutdown 2024-04-02T14:45:39.350 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.353+0000 7fbe73a05740 1 -- 172.21.15.90:0/159396511 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562857c83eb0 msgr2=0x562857c95570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:45:39.350 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.353+0000 7fbe73a05740 1 --2- 172.21.15.90:0/159396511 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562857c83eb0 0x562857c95570 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x562857c83980 tx=0x7fbe5c004850 comp rx=0 tx=0).stop 2024-04-02T14:45:39.350 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.353+0000 7fbe73a05740 1 -- 172.21.15.90:0/159396511 shutdown_connections 2024-04-02T14:45:39.350 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.353+0000 7fbe73a05740 1 --2- 172.21.15.90:0/159396511 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x562857d04b60 0x562857d06f80 crc :-1 s=READY pgs=287 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:39.350 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.353+0000 7fbe73a05740 1 --2- 172.21.15.90:0/159396511 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562857c8d3b0 0x562857c92540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:39.350 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.353+0000 7fbe73a05740 1 --2- 172.21.15.90:0/159396511 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x562857c847d0 0x562857c95ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:39.351 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.353+0000 7fbe73a05740 1 --2- 172.21.15.90:0/159396511 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562857c83eb0 0x562857c95570 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:39.351 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.353+0000 7fbe73a05740 1 -- 172.21.15.90:0/159396511 >> 172.21.15.90:0/159396511 conn(0x562857c7e5e0 msgr2=0x562857c7cdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:45:39.351 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.353+0000 7fbe4be00640 0 client.28666 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:45:39.351 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.353+0000 7fbe73a05740 1 -- 172.21.15.90:0/159396511 shutdown_connections 2024-04-02T14:45:39.351 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.353+0000 7fbe73a05740 1 -- 172.21.15.90:0/159396511 wait complete. 2024-04-02T14:45:39.351 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.353+0000 7fbe73a05740 20 client.28666 trim_cache size 0 max 16384 2024-04-02T14:45:39.351 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:{ 2024-04-02T14:45:39.351 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "message": "", 2024-04-02T14:45:39.351 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "return_code": 0 2024-04-02T14:45:39.351 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:} 2024-04-02T14:45:39.352 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.354+0000 7fbe70e00640 1 -- 172.21.15.90:0/1099404043 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fbe500a3e40 msgr2=0x7fbe500a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:45:39.352 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.354+0000 7fbe70e00640 1 --2- 172.21.15.90:0/1099404043 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fbe500a3e40 0x7fbe500a6300 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fbe64009370 tx=0x7fbe64009960 comp rx=0 tx=0).stop 2024-04-02T14:45:39.352 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.354+0000 7fbe70e00640 1 -- 172.21.15.90:0/1099404043 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbe6c076950 msgr2=0x7fbe6c1a4b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:45:39.352 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.354+0000 7fbe70e00640 1 --2- 172.21.15.90:0/1099404043 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbe6c076950 0x7fbe6c1a4b60 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7fbe5800e790 tx=0x7fbe5800ec50 comp rx=0 tx=0).stop 2024-04-02T14:45:39.352 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.355+0000 7fbe70e00640 1 -- 172.21.15.90:0/1099404043 shutdown_connections 2024-04-02T14:45:39.352 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.355+0000 7fbe70e00640 1 --2- 172.21.15.90:0/1099404043 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fbe500a3e40 0x7fbe500a6300 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:39.352 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.355+0000 7fbe70e00640 1 --2- 172.21.15.90:0/1099404043 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbe6c112970 0x7fbe6c19e6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:39.353 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.355+0000 7fbe70e00640 1 --2- 172.21.15.90:0/1099404043 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbe6c076950 0x7fbe6c1a4b60 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:39.353 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.355+0000 7fbe70e00640 1 --2- 172.21.15.90:0/1099404043 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbe6c076030 0x7fbe6c1a4620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:39.353 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.355+0000 7fbe70e00640 1 -- 172.21.15.90:0/1099404043 >> 172.21.15.90:0/1099404043 conn(0x7fbe6c0ff7c0 msgr2=0x7fbe6c101bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:45:39.353 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.355+0000 7fbe70e00640 1 -- 172.21.15.90:0/1099404043 shutdown_connections 2024-04-02T14:45:39.353 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:39.355+0000 7fbe70e00640 1 -- 172.21.15.90:0/1099404043 wait complete. 2024-04-02T14:45:39.605 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:45:39 smithi092 ceph-mon[36497]: pgmap v334: 97 pgs: 97 active+clean; 4.5 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 10 MiB/s wr, 881 op/s 2024-04-02T14:45:39.681 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:45:39 smithi177 ceph-mon[36431]: pgmap v334: 97 pgs: 97 active+clean; 4.5 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 10 MiB/s wr, 881 op/s 2024-04-02T14:45:39.682 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:45:39 smithi090 ceph-mon[31674]: pgmap v334: 97 pgs: 97 active+clean; 4.5 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 10 MiB/s wr, 881 op/s 2024-04-02T14:45:41.827 INFO:teuthology.orchestra.run.smithi090.stderr:progress: 610.0 s, 1302.5 tps, lat 78.043 ms stddev 164.103 2024-04-02T14:45:41.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:45:41 smithi177 ceph-mon[36431]: pgmap v335: 97 pgs: 97 active+clean; 4.6 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 27 MiB/s wr, 2.03k op/s 2024-04-02T14:45:41.932 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:45:41 smithi090 ceph-mon[31674]: pgmap v335: 97 pgs: 97 active+clean; 4.6 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 27 MiB/s wr, 2.03k op/s 2024-04-02T14:45:42.105 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:45:41 smithi092 ceph-mon[36497]: pgmap v335: 97 pgs: 97 active+clean; 4.6 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 27 MiB/s wr, 2.03k op/s 2024-04-02T14:45:43.931 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:45:43 smithi177 ceph-mon[36431]: pgmap v336: 97 pgs: 97 active+clean; 4.6 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 23 MiB/s wr, 1.74k op/s 2024-04-02T14:45:43.932 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:45:43 smithi090 ceph-mon[31674]: pgmap v336: 97 pgs: 97 active+clean; 4.6 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 23 MiB/s wr, 1.74k op/s 2024-04-02T14:45:44.105 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:45:43 smithi092 ceph-mon[36497]: pgmap v336: 97 pgs: 97 active+clean; 4.6 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 23 MiB/s wr, 1.74k op/s 2024-04-02T14:45:45.918 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:45:45 smithi090 ceph-mon[31674]: pgmap v337: 97 pgs: 97 active+clean; 4.6 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 26 MiB/s wr, 2.01k op/s 2024-04-02T14:45:45.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:45:45 smithi177 ceph-mon[36431]: pgmap v337: 97 pgs: 97 active+clean; 4.6 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 26 MiB/s wr, 2.01k op/s 2024-04-02T14:45:46.105 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:45:45 smithi092 ceph-mon[36497]: pgmap v337: 97 pgs: 97 active+clean; 4.6 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 26 MiB/s wr, 2.01k op/s 2024-04-02T14:45:46.826 INFO:teuthology.orchestra.run.smithi090.stderr:progress: 615.0 s, 1412.5 tps, lat 70.350 ms stddev 85.088 2024-04-02T14:45:47.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:45:47 smithi177 ceph-mon[36431]: pgmap v338: 97 pgs: 97 active+clean; 4.6 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 23 MiB/s wr, 1.72k op/s 2024-04-02T14:45:47.932 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:45:47 smithi090 ceph-mon[31674]: pgmap v338: 97 pgs: 97 active+clean; 4.6 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 23 MiB/s wr, 1.72k op/s 2024-04-02T14:45:48.105 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:45:47 smithi092 ceph-mon[36497]: pgmap v338: 97 pgs: 97 active+clean; 4.6 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 23 MiB/s wr, 1.72k op/s 2024-04-02T14:45:49.931 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:45:49 smithi177 ceph-mon[36431]: pgmap v339: 97 pgs: 97 active+clean; 4.6 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 23 MiB/s wr, 1.75k op/s 2024-04-02T14:45:49.932 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:45:49 smithi090 ceph-mon[31674]: pgmap v339: 97 pgs: 97 active+clean; 4.6 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 23 MiB/s wr, 1.75k op/s 2024-04-02T14:45:50.105 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:45:49 smithi092 ceph-mon[36497]: pgmap v339: 97 pgs: 97 active+clean; 4.6 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 23 MiB/s wr, 1.75k op/s 2024-04-02T14:45:51.826 INFO:teuthology.orchestra.run.smithi090.stderr:progress: 620.0 s, 1392.4 tps, lat 72.541 ms stddev 85.843 2024-04-02T14:45:51.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:45:51 smithi177 ceph-mon[36431]: pgmap v340: 97 pgs: 97 active+clean; 4.7 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 23 MiB/s wr, 1.77k op/s 2024-04-02T14:45:51.932 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:45:51 smithi090 ceph-mon[31674]: pgmap v340: 97 pgs: 97 active+clean; 4.7 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 23 MiB/s wr, 1.77k op/s 2024-04-02T14:45:52.104 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:45:51 smithi092 ceph-mon[36497]: pgmap v340: 97 pgs: 97 active+clean; 4.7 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 23 MiB/s wr, 1.77k op/s 2024-04-02T14:45:52.522 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.526+0000 7fefe0800640 1 -- 172.21.15.90:0/3890097653 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fefd8102470 msgr2=0x7fefd810e980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:45:52.522 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.526+0000 7fefe0800640 1 --2- 172.21.15.90:0/3890097653 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fefd8102470 0x7fefd810e980 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7fefd4009c10 tx=0x7fefd402bee0 comp rx=0 tx=0).stop 2024-04-02T14:45:52.522 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.526+0000 7fefe0800640 1 -- 172.21.15.90:0/3890097653 shutdown_connections 2024-04-02T14:45:52.522 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.526+0000 7fefe0800640 1 --2- 172.21.15.90:0/3890097653 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fefd8102470 0x7fefd810e980 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:52.522 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.526+0000 7fefe0800640 1 --2- 172.21.15.90:0/3890097653 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fefd8101ab0 0x7fefd8101f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:52.522 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.526+0000 7fefe0800640 1 --2- 172.21.15.90:0/3890097653 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fefd8107ac0 0x7fefd8107ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:52.522 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.526+0000 7fefe0800640 1 -- 172.21.15.90:0/3890097653 >> 172.21.15.90:0/3890097653 conn(0x7fefd80fd5c0 msgr2=0x7fefd80ff9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:45:52.522 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.526+0000 7fefe0800640 1 -- 172.21.15.90:0/3890097653 shutdown_connections 2024-04-02T14:45:52.522 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.526+0000 7fefe0800640 1 -- 172.21.15.90:0/3890097653 wait complete. 2024-04-02T14:45:52.523 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.527+0000 7fefe0800640 1 Processor -- start 2024-04-02T14:45:52.524 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.527+0000 7fefe0800640 1 -- start start 2024-04-02T14:45:52.524 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.528+0000 7fefe0800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fefd8101ab0 0x7fefd819a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:52.524 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.528+0000 7fefe0800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fefd8102470 0x7fefd819a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:52.524 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.528+0000 7fefe0800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fefd8107ac0 0x7fefd819ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:52.524 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.528+0000 7fefe0800640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fefd8111ef0 con 0x7fefd8107ac0 2024-04-02T14:45:52.524 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.528+0000 7fefe0800640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fefd8111d70 con 0x7fefd8101ab0 2024-04-02T14:45:52.524 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.528+0000 7fefe0800640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fefd8112070 con 0x7fefd8102470 2024-04-02T14:45:52.524 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.528+0000 7fefdde00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fefd8101ab0 0x7fefd819a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:52.524 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.528+0000 7fefde800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fefd8107ac0 0x7fefd819ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:52.524 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.528+0000 7fefdd400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fefd8102470 0x7fefd819a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:52.524 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.528+0000 7fefdde00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fefd8101ab0 0x7fefd819a4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:49662/0 (socket says 172.21.15.90:49662) 2024-04-02T14:45:52.524 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.528+0000 7fefdd400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fefd8102470 0x7fefd819a9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:58600/0 (socket says 172.21.15.90:58600) 2024-04-02T14:45:52.524 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.528+0000 7fefdde00640 1 -- 172.21.15.90:0/2525377012 learned_addr learned my addr 172.21.15.90:0/2525377012 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:45:52.525 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.529+0000 7fefdde00640 1 -- 172.21.15.90:0/2525377012 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fefd8102470 msgr2=0x7fefd819a9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:45:52.525 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.529+0000 7fefdde00640 1 --2- 172.21.15.90:0/2525377012 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fefd8102470 0x7fefd819a9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:52.525 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.529+0000 7fefdde00640 1 -- 172.21.15.90:0/2525377012 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fefd8107ac0 msgr2=0x7fefd819ed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:45:52.525 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.529+0000 7fefdde00640 1 --2- 172.21.15.90:0/2525377012 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fefd8107ac0 0x7fefd819ed70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:52.525 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.529+0000 7fefdde00640 1 -- 172.21.15.90:0/2525377012 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fefd819f4a0 con 0x7fefd8101ab0 2024-04-02T14:45:52.526 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.529+0000 7fefdde00640 1 --2- 172.21.15.90:0/2525377012 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fefd8101ab0 0x7fefd819a4b0 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7fefcc00e9c0 tx=0x7fefcc00ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:45:52.526 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.529+0000 7fefd2a00640 1 -- 172.21.15.90:0/2525377012 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefcc00ccc0 con 0x7fefd8101ab0 2024-04-02T14:45:52.526 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.529+0000 7fefd2a00640 1 -- 172.21.15.90:0/2525377012 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fefcc005500 con 0x7fefd8101ab0 2024-04-02T14:45:52.526 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.529+0000 7fefe0800640 1 -- 172.21.15.90:0/2525377012 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fefd819f790 con 0x7fefd8101ab0 2024-04-02T14:45:52.526 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.529+0000 7fefd2a00640 1 -- 172.21.15.90:0/2525377012 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefcc010430 con 0x7fefd8101ab0 2024-04-02T14:45:52.526 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.529+0000 7fefe0800640 1 -- 172.21.15.90:0/2525377012 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fefd81a7020 con 0x7fefd8101ab0 2024-04-02T14:45:52.530 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.534+0000 7fefd2a00640 1 -- 172.21.15.90:0/2525377012 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fefcc0046c0 con 0x7fefd8101ab0 2024-04-02T14:45:52.530 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.534+0000 7fefe35bb740 1 Processor -- start 2024-04-02T14:45:52.530 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.534+0000 7fefe35bb740 1 -- start start 2024-04-02T14:45:52.531 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.535+0000 7fefe35bb740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559520973eb0 0x559520974290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:52.531 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.535+0000 7fefe35bb740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5595209747d0 0x55952097ce70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:52.531 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.535+0000 7fefe35bb740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55952097d3b0 0x55952097f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:52.531 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.535+0000 7fefe35bb740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5595208d8d70 con 0x559520973eb0 2024-04-02T14:45:52.531 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.535+0000 7fefe35bb740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x5595208812a0 con 0x55952097d3b0 2024-04-02T14:45:52.531 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.535+0000 7fefe35bb740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5595208d4d20 con 0x5595209747d0 2024-04-02T14:45:52.533 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.535+0000 7fefde800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559520973eb0 0x559520974290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:52.533 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.535+0000 7fefdd400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55952097d3b0 0x55952097f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:52.533 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.535+0000 7fefdd400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55952097d3b0 0x55952097f7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:49672/0 (socket says 172.21.15.90:49672) 2024-04-02T14:45:52.533 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.535+0000 7fefdd400640 1 -- 172.21.15.90:0/3925168799 learned_addr learned my addr 172.21.15.90:0/3925168799 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:45:52.533 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.535+0000 7fefdde00640 1 --2- 172.21.15.90:0/3925168799 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5595209747d0 0x55952097ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:52.533 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.536+0000 7fefdd400640 1 -- 172.21.15.90:0/3925168799 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5595209747d0 msgr2=0x55952097ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:45:52.533 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.536+0000 7fefdd400640 1 --2- 172.21.15.90:0/3925168799 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5595209747d0 0x55952097ce70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:52.533 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.536+0000 7fefdd400640 1 -- 172.21.15.90:0/3925168799 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559520973eb0 msgr2=0x559520974290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:45:52.533 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.536+0000 7fefdd400640 1 --2- 172.21.15.90:0/3925168799 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559520973eb0 0x559520974290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:52.533 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.536+0000 7fefdd400640 1 -- 172.21.15.90:0/3925168799 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559520895840 con 0x55952097d3b0 2024-04-02T14:45:52.533 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.536+0000 7fefde800640 1 --2- 172.21.15.90:0/3925168799 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559520973eb0 0x559520974290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:45:52.534 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.536+0000 7fefd2a00640 1 --2- 172.21.15.90:0/2525377012 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fefc00a3dc0 0x7fefc00a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:52.534 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.536+0000 7fefdd400640 1 --2- 172.21.15.90:0/3925168799 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55952097d3b0 0x55952097f7a0 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7fefc8009c90 tx=0x7fefc8011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ad0320901c93d848 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:45:52.534 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.536+0000 7fefd2a00640 1 -- 172.21.15.90:0/2525377012 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 8550+0+0 (secure 0 0 0) 0x7fefcc0d0a80 con 0x7fefd8101ab0 2024-04-02T14:45:52.534 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.536+0000 7fefc7000640 1 -- 172.21.15.90:0/3925168799 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefc801e070 con 0x55952097d3b0 2024-04-02T14:45:52.534 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.536+0000 7fefc7000640 1 -- 172.21.15.90:0/3925168799 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fefc8019b10 con 0x55952097d3b0 2024-04-02T14:45:52.534 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.536+0000 7fefc7000640 1 -- 172.21.15.90:0/3925168799 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefc800eac0 con 0x55952097d3b0 2024-04-02T14:45:52.534 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.536+0000 7fefde800640 1 --2- 172.21.15.90:0/2525377012 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fefc00a3dc0 0x7fefc00a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:52.534 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.537+0000 7fefde800640 1 --2- 172.21.15.90:0/2525377012 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fefc00a3dc0 0x7fefc00a6280 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fefd4004af0 tx=0x7fefd4005bd0 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:45:52.534 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.537+0000 7fefe35bb740 1 -- 172.21.15.90:0/3925168799 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55952097d3b0 msgr2=0x55952097f7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:45:52.534 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.537+0000 7fefe35bb740 1 --2- 172.21.15.90:0/3925168799 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55952097d3b0 0x55952097f7a0 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7fefc8009c90 tx=0x7fefc8011810 comp rx=0 tx=0).stop 2024-04-02T14:45:52.534 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.538+0000 7fefe35bb740 1 -- 172.21.15.90:0/3925168799 shutdown_connections 2024-04-02T14:45:52.534 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.538+0000 7fefe35bb740 1 --2- 172.21.15.90:0/3925168799 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55952097d3b0 0x55952097f7a0 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:52.534 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.538+0000 7fefe35bb740 1 --2- 172.21.15.90:0/3925168799 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5595209747d0 0x55952097ce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:52.534 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.538+0000 7fefe35bb740 1 --2- 172.21.15.90:0/3925168799 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559520973eb0 0x559520974290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:52.534 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.538+0000 7fefe35bb740 1 -- 172.21.15.90:0/3925168799 >> 172.21.15.90:0/3925168799 conn(0x55952096e5e0 msgr2=0x55952096e9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:45:52.535 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.538+0000 7fefe35bb740 1 -- 172.21.15.90:0/3925168799 shutdown_connections 2024-04-02T14:45:52.535 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.539+0000 7fefe35bb740 1 -- 172.21.15.90:0/3925168799 wait complete. 2024-04-02T14:45:52.535 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.540+0000 7fefe35bb740 1 -- start start 2024-04-02T14:45:52.536 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.541+0000 7fefe35bb740 1 Processor -- start 2024-04-02T14:45:52.539 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.541+0000 7fefe35bb740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559520973eb0 0x55952097a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:52.539 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.541+0000 7fefe35bb740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5595209747d0 0x55952097a920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:52.539 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.542+0000 7fefe35bb740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55952097d3b0 0x55952097ae60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:52.539 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.542+0000 7fefe35bb740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5595209801a0 con 0x5595209747d0 2024-04-02T14:45:52.539 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.542+0000 7fefe35bb740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x559520894660 con 0x55952097d3b0 2024-04-02T14:45:52.539 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.542+0000 7fefe35bb740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x559520980320 con 0x559520973eb0 2024-04-02T14:45:52.539 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.542+0000 7fefe35bb740 10 client.0 _ll_register_callbacks cb 0x559520894050 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:45:52.539 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.542+0000 7fefdde00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559520973eb0 0x55952097a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:52.539 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.542+0000 7fefdde00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559520973eb0 0x55952097a3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:58620/0 (socket says 172.21.15.90:58620) 2024-04-02T14:45:52.539 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.542+0000 7fefdde00640 1 -- 172.21.15.90:0/2085255486 learned_addr learned my addr 172.21.15.90:0/2085255486 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:45:52.539 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.542+0000 7fefdd400640 1 --2- 172.21.15.90:0/2085255486 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5595209747d0 0x55952097a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:52.539 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.542+0000 7fefde800640 1 --2- 172.21.15.90:0/2085255486 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55952097d3b0 0x55952097ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:52.539 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.542+0000 7fefdde00640 1 -- 172.21.15.90:0/2085255486 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55952097d3b0 msgr2=0x55952097ae60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:45:52.540 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.542+0000 7fefdde00640 1 --2- 172.21.15.90:0/2085255486 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55952097d3b0 0x55952097ae60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:52.540 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.542+0000 7fefdde00640 1 -- 172.21.15.90:0/2085255486 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5595209747d0 msgr2=0x55952097a920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:45:52.540 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.542+0000 7fefdde00640 1 --2- 172.21.15.90:0/2085255486 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5595209747d0 0x55952097a920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:52.540 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.542+0000 7fefdde00640 1 -- 172.21.15.90:0/2085255486 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5595208a3e80 con 0x559520973eb0 2024-04-02T14:45:52.540 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.542+0000 7fefdd400640 1 --2- 172.21.15.90:0/2085255486 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5595209747d0 0x55952097a920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:45:52.541 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.542+0000 7fefdde00640 1 --2- 172.21.15.90:0/2085255486 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559520973eb0 0x55952097a3e0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7fefcc00c6a0 tx=0x7fefcc009f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=709fda3d5ae44a0c server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:45:52.541 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.542+0000 7fefbb400640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:45:52.541 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.543+0000 7fefbb400640 1 -- 172.21.15.90:0/2085255486 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefcc09c7d0 con 0x559520973eb0 2024-04-02T14:45:52.541 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.543+0000 7fefbb400640 1 -- 172.21.15.90:0/2085255486 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8550+0+0 (secure 0 0 0) 0x7fefcc0ce430 con 0x559520973eb0 2024-04-02T14:45:52.541 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.543+0000 7fefe35bb740 1 -- 172.21.15.90:0/2085255486 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fefc8019b10 con 0x559520973eb0 2024-04-02T14:45:52.541 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.543+0000 7fefbb400640 1 -- 172.21.15.90:0/2085255486 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fefcc09c360 con 0x559520973eb0 2024-04-02T14:45:52.541 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.543+0000 7fefe35bb740 10 client.28678 fetch_fsmap learned FSMap version 27 2024-04-02T14:45:52.541 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.543+0000 7fefe35bb740 1 -- 172.21.15.90:0/2085255486 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5595205ed210 con 0x559520973eb0 2024-04-02T14:45:52.541 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.543+0000 7fefbb400640 1 -- 172.21.15.90:0/2085255486 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fefc8019b10 con 0x559520973eb0 2024-04-02T14:45:52.541 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.543+0000 7fefbb400640 1 -- 172.21.15.90:0/2085255486 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fefcc0a51e0 con 0x559520973eb0 2024-04-02T14:45:52.541 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.543+0000 7fefe35bb740 10 client.28678 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:45:52.541 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.543+0000 7fefe35bb740 10 client.28678 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-02T14:45:52.541 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.543+0000 7fefe35bb740 20 client.28678 populate_metadata read hostname 'smithi090' 2024-04-02T14:45:52.541 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.543+0000 7fefe35bb740 1 --2- 172.21.15.90:0/2085255486 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55952098d7c0 0x55952098fc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:52.542 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.543+0000 7fefe35bb740 4 client.28678 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:45:52.542 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.543+0000 7fefe35bb740 1 -- 172.21.15.90:0/2085255486 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559520888cd0 con 0x55952098d7c0 2024-04-02T14:45:52.542 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.544+0000 7fefdd400640 1 --2- 172.21.15.90:0/2085255486 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55952098d7c0 0x55952098fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:52.543 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.544+0000 7fefdd400640 1 --2- 172.21.15.90:0/2085255486 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55952098d7c0 0x55952098fc10 crc :-1 s=READY pgs=288 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ce10a9f797ce2be7 server_cookie=a339e0c3ccc1b7a1 in_seq=0 out_seq=0 2024-04-02T14:45:52.543 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.544+0000 7fefbb400640 10 client.28678 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:45:52.543 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.545+0000 7fefbb400640 1 -- 172.21.15.90:0/2085255486 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x559520888cd0 con 0x55952098d7c0 2024-04-02T14:45:52.544 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.545+0000 7fefbb400640 10 client.28678 handle_command_reply: tid=0 2024-04-02T14:45:52.544 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.545+0000 7fefe35bb740 1 client.28678 shutdown 2024-04-02T14:45:52.544 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.545+0000 7fefe35bb740 1 -- 172.21.15.90:0/2085255486 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559520973eb0 msgr2=0x55952097a3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:45:52.544 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.545+0000 7fefe35bb740 1 --2- 172.21.15.90:0/2085255486 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559520973eb0 0x55952097a3e0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7fefcc00c6a0 tx=0x7fefcc009f90 comp rx=0 tx=0).stop 2024-04-02T14:45:52.544 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.545+0000 7fefe35bb740 1 -- 172.21.15.90:0/2085255486 shutdown_connections 2024-04-02T14:45:52.544 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.545+0000 7fefe35bb740 1 --2- 172.21.15.90:0/2085255486 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55952098d7c0 0x55952098fc10 crc :-1 s=READY pgs=288 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:52.544 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.546+0000 7fefe35bb740 1 --2- 172.21.15.90:0/2085255486 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55952097d3b0 0x55952097ae60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:52.544 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.546+0000 7fefe35bb740 1 --2- 172.21.15.90:0/2085255486 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5595209747d0 0x55952097a920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:52.544 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.546+0000 7fefe35bb740 1 --2- 172.21.15.90:0/2085255486 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559520973eb0 0x55952097a3e0 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:52.544 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.546+0000 7fefe35bb740 1 -- 172.21.15.90:0/2085255486 >> 172.21.15.90:0/2085255486 conn(0x55952096e5e0 msgr2=0x559520970330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:45:52.544 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.546+0000 7fefbb400640 0 client.28678 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:45:52.544 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.546+0000 7fefe35bb740 1 -- 172.21.15.90:0/2085255486 shutdown_connections 2024-04-02T14:45:52.544 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.546+0000 7fefe35bb740 1 -- 172.21.15.90:0/2085255486 wait complete. 2024-04-02T14:45:52.544 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.547+0000 7fefe35bb740 20 client.28678 trim_cache size 0 max 16384 2024-04-02T14:45:52.567 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.571+0000 7fefe35bb740 1 Processor -- start 2024-04-02T14:45:52.567 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.571+0000 7fefe35bb740 1 -- start start 2024-04-02T14:45:52.567 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.572+0000 7fefe35bb740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559520973eb0 0x55952097cb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:52.567 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.572+0000 7fefe35bb740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5595209747d0 0x559520989090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:52.567 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.572+0000 7fefe35bb740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55952097d3b0 0x559520995520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:52.567 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.572+0000 7fefe35bb740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fefd81af990 con 0x5595209747d0 2024-04-02T14:45:52.567 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.572+0000 7fefe35bb740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fefd81b14d0 con 0x559520973eb0 2024-04-02T14:45:52.567 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.572+0000 7fefe35bb740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fefd81b0060 con 0x55952097d3b0 2024-04-02T14:45:52.568 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.572+0000 7fefdd400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5595209747d0 0x559520989090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:52.568 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.572+0000 7fefde800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55952097d3b0 0x559520995520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:52.568 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.572+0000 7fefdd400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5595209747d0 0x559520989090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:39098/0 (socket says 172.21.15.90:39098) 2024-04-02T14:45:52.569 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.572+0000 7fefdd400640 1 -- 172.21.15.90:0/1722147410 learned_addr learned my addr 172.21.15.90:0/1722147410 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:45:52.569 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.572+0000 7fefdde00640 1 --2- 172.21.15.90:0/1722147410 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559520973eb0 0x55952097cb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:52.569 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.573+0000 7fefdd400640 1 -- 172.21.15.90:0/1722147410 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55952097d3b0 msgr2=0x559520995520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:45:52.569 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.573+0000 7fefdd400640 1 --2- 172.21.15.90:0/1722147410 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55952097d3b0 0x559520995520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:52.569 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.573+0000 7fefdd400640 1 -- 172.21.15.90:0/1722147410 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559520973eb0 msgr2=0x55952097cb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:45:52.569 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.573+0000 7fefdd400640 1 --2- 172.21.15.90:0/1722147410 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559520973eb0 0x55952097cb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:52.569 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.573+0000 7fefdd400640 1 -- 172.21.15.90:0/1722147410 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5595206b9620 con 0x5595209747d0 2024-04-02T14:45:52.570 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.574+0000 7fefdd400640 1 --2- 172.21.15.90:0/1722147410 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5595209747d0 0x559520989090 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fefc80098e0 tx=0x7fefc8000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74cf806e72b56b81 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:45:52.570 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.574+0000 7fefc6600640 1 -- 172.21.15.90:0/1722147410 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefc801e040 con 0x5595209747d0 2024-04-02T14:45:52.570 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.574+0000 7fefc6600640 1 -- 172.21.15.90:0/1722147410 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fefc801f040 con 0x5595209747d0 2024-04-02T14:45:52.570 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.574+0000 7fefc6600640 1 -- 172.21.15.90:0/1722147410 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefc8019a60 con 0x5595209747d0 2024-04-02T14:45:52.570 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.574+0000 7fefdde00640 1 --2- 172.21.15.90:0/1722147410 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559520973eb0 0x55952097cb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:45:52.571 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.575+0000 7fefe35bb740 1 -- 172.21.15.90:0/1722147410 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5595209747d0 msgr2=0x559520989090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:45:52.571 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.575+0000 7fefe35bb740 1 --2- 172.21.15.90:0/1722147410 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5595209747d0 0x559520989090 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fefc80098e0 tx=0x7fefc8000bc0 comp rx=0 tx=0).stop 2024-04-02T14:45:52.571 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.575+0000 7fefe35bb740 1 -- 172.21.15.90:0/1722147410 shutdown_connections 2024-04-02T14:45:52.571 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.575+0000 7fefe35bb740 1 --2- 172.21.15.90:0/1722147410 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55952097d3b0 0x559520995520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:52.571 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.576+0000 7fefe35bb740 1 --2- 172.21.15.90:0/1722147410 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5595209747d0 0x559520989090 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:52.571 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.576+0000 7fefe35bb740 1 --2- 172.21.15.90:0/1722147410 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559520973eb0 0x55952097cb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:52.571 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.576+0000 7fefe35bb740 1 -- 172.21.15.90:0/1722147410 >> 172.21.15.90:0/1722147410 conn(0x55952096e5e0 msgr2=0x55952096e9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:45:52.571 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.576+0000 7fefe35bb740 1 -- 172.21.15.90:0/1722147410 shutdown_connections 2024-04-02T14:45:52.571 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.576+0000 7fefe35bb740 1 -- 172.21.15.90:0/1722147410 wait complete. 2024-04-02T14:45:52.571 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.576+0000 7fefe35bb740 1 -- start start 2024-04-02T14:45:52.575 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.576+0000 7fefe35bb740 1 Processor -- start 2024-04-02T14:45:52.575 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.577+0000 7fefe35bb740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559520973eb0 0x559520977270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:52.575 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.577+0000 7fefe35bb740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5595209747d0 0x5595209777b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:52.575 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.577+0000 7fefe35bb740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55952097d3b0 0x559520977cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:52.575 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.577+0000 7fefe35bb740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fefd81b0e00 con 0x559520973eb0 2024-04-02T14:45:52.575 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.577+0000 7fefe35bb740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fefd81b0730 con 0x5595209747d0 2024-04-02T14:45:52.575 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.577+0000 7fefe35bb740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fefd81ad990 con 0x55952097d3b0 2024-04-02T14:45:52.575 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.577+0000 7fefe35bb740 10 client.0 _ll_register_callbacks cb 0x7fefcc0056b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:45:52.575 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.577+0000 7fefdde00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559520973eb0 0x559520977270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:52.576 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.577+0000 7fefdd400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5595209747d0 0x5595209777b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:52.576 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.577+0000 7fefde800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55952097d3b0 0x559520977cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:52.576 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.577+0000 7fefdde00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559520973eb0 0x559520977270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:39102/0 (socket says 172.21.15.90:39102) 2024-04-02T14:45:52.576 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.577+0000 7fefdd400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5595209747d0 0x5595209777b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:49710/0 (socket says 172.21.15.90:49710) 2024-04-02T14:45:52.576 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.577+0000 7fefde800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55952097d3b0 0x559520977cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:58636/0 (socket says 172.21.15.90:58636) 2024-04-02T14:45:52.576 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.577+0000 7fefdde00640 1 -- 172.21.15.90:0/668199712 learned_addr learned my addr 172.21.15.90:0/668199712 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:45:52.576 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.577+0000 7fefdde00640 1 -- 172.21.15.90:0/668199712 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55952097d3b0 msgr2=0x559520977cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:45:52.577 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.577+0000 7fefdde00640 1 --2- 172.21.15.90:0/668199712 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55952097d3b0 0x559520977cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:52.577 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.577+0000 7fefdde00640 1 -- 172.21.15.90:0/668199712 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5595209747d0 msgr2=0x5595209777b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:45:52.577 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.577+0000 7fefdde00640 1 --2- 172.21.15.90:0/668199712 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5595209747d0 0x5595209777b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:52.577 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.577+0000 7fefdde00640 1 -- 172.21.15.90:0/668199712 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5595209f2f90 con 0x559520973eb0 2024-04-02T14:45:52.577 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.577+0000 7fefdd400640 1 --2- 172.21.15.90:0/668199712 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5595209747d0 0x5595209777b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:45:52.577 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.578+0000 7fefdde00640 1 --2- 172.21.15.90:0/668199712 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559520973eb0 0x559520977270 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fefcc000c00 tx=0x7fefcc004b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b9a20301907e68d server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:45:52.577 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.578+0000 7fefbb400640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-02T14:45:52.577 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.578+0000 7fefbb400640 1 -- 172.21.15.90:0/668199712 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefcc09c540 con 0x559520973eb0 2024-04-02T14:45:52.577 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.578+0000 7fefbb400640 1 -- 172.21.15.90:0/668199712 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8550+0+0 (secure 0 0 0) 0x7fefcc0870b0 con 0x559520973eb0 2024-04-02T14:45:52.577 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.578+0000 7fefe35bb740 1 -- 172.21.15.90:0/668199712 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fefc801f040 con 0x559520973eb0 2024-04-02T14:45:52.577 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.578+0000 7fefbb400640 1 -- 172.21.15.90:0/668199712 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fefcc080050 con 0x559520973eb0 2024-04-02T14:45:52.577 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.578+0000 7fefe35bb740 10 client.17865 fetch_fsmap learned FSMap version 27 2024-04-02T14:45:52.578 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.578+0000 7fefe35bb740 1 -- 172.21.15.90:0/668199712 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5595209f4240 con 0x559520973eb0 2024-04-02T14:45:52.578 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.578+0000 7fefbb400640 1 -- 172.21.15.90:0/668199712 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fefc801f040 con 0x559520973eb0 2024-04-02T14:45:52.578 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.579+0000 7fefbb400640 1 -- 172.21.15.90:0/668199712 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fefcc098750 con 0x559520973eb0 2024-04-02T14:45:52.578 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.579+0000 7fefe35bb740 10 client.17865 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:45:52.578 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.579+0000 7fefe35bb740 10 client.17865 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-02T14:45:52.578 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.579+0000 7fefe35bb740 20 client.17865 populate_metadata read hostname 'smithi090' 2024-04-02T14:45:52.578 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.579+0000 7fefe35bb740 1 --2- 172.21.15.90:0/668199712 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5595209f4900 0x5595209f6cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:45:52.578 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.579+0000 7fefe35bb740 4 client.17865 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-02T14:45:52.578 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.579+0000 7fefe35bb740 1 -- 172.21.15.90:0/668199712 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5595208e62b0 con 0x5595209f4900 2024-04-02T14:45:52.578 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.579+0000 7fefdd400640 1 --2- 172.21.15.90:0/668199712 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5595209f4900 0x5595209f6cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:45:52.579 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.580+0000 7fefdd400640 1 --2- 172.21.15.90:0/668199712 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5595209f4900 0x5595209f6cf0 crc :-1 s=READY pgs=289 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5bc6d7c1c7e408d1 server_cookie=de38675d19362df6 in_seq=0 out_seq=0 2024-04-02T14:45:52.579 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.581+0000 7fefbb400640 10 client.17865 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:45:52.595 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.598+0000 7fefbb400640 1 -- 172.21.15.90:0/668199712 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5595208e62b0 con 0x5595209f4900 2024-04-02T14:45:52.595 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.598+0000 7fefbb400640 10 client.17865 handle_command_reply: tid=0 2024-04-02T14:45:52.595 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.598+0000 7fefe35bb740 1 client.17865 shutdown 2024-04-02T14:45:52.595 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.599+0000 7fefe35bb740 1 -- 172.21.15.90:0/668199712 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559520973eb0 msgr2=0x559520977270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:45:52.595 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.599+0000 7fefe35bb740 1 --2- 172.21.15.90:0/668199712 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559520973eb0 0x559520977270 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fefcc000c00 tx=0x7fefcc004b00 comp rx=0 tx=0).stop 2024-04-02T14:45:52.595 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.599+0000 7fefe35bb740 1 -- 172.21.15.90:0/668199712 shutdown_connections 2024-04-02T14:45:52.595 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.599+0000 7fefe35bb740 1 --2- 172.21.15.90:0/668199712 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5595209f4900 0x5595209f6cf0 crc :-1 s=READY pgs=289 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:52.595 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.599+0000 7fefe35bb740 1 --2- 172.21.15.90:0/668199712 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55952097d3b0 0x559520977cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:52.595 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.599+0000 7fefe35bb740 1 --2- 172.21.15.90:0/668199712 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5595209747d0 0x5595209777b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:52.595 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.599+0000 7fefe35bb740 1 --2- 172.21.15.90:0/668199712 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559520973eb0 0x559520977270 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:52.595 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.599+0000 7fefbb400640 0 client.17865 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:45:52.596 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.599+0000 7fefe35bb740 1 -- 172.21.15.90:0/668199712 >> 172.21.15.90:0/668199712 conn(0x55952096e5e0 msgr2=0x55952096cdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:45:52.596 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.600+0000 7fefe35bb740 1 -- 172.21.15.90:0/668199712 shutdown_connections 2024-04-02T14:45:52.596 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.600+0000 7fefe35bb740 1 -- 172.21.15.90:0/668199712 wait complete. 2024-04-02T14:45:52.596 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.600+0000 7fefe35bb740 20 client.17865 trim_cache size 0 max 16384 2024-04-02T14:45:52.597 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:{ 2024-04-02T14:45:52.597 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "message": "", 2024-04-02T14:45:52.597 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "return_code": 0 2024-04-02T14:45:52.597 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:} 2024-04-02T14:45:52.598 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.601+0000 7fefe0800640 1 -- 172.21.15.90:0/2525377012 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fefc00a3dc0 msgr2=0x7fefc00a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:45:52.598 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.601+0000 7fefe0800640 1 --2- 172.21.15.90:0/2525377012 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fefc00a3dc0 0x7fefc00a6280 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fefd4004af0 tx=0x7fefd4005bd0 comp rx=0 tx=0).stop 2024-04-02T14:45:52.598 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.601+0000 7fefe0800640 1 -- 172.21.15.90:0/2525377012 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fefd8101ab0 msgr2=0x7fefd819a4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:45:52.598 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.601+0000 7fefe0800640 1 --2- 172.21.15.90:0/2525377012 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fefd8101ab0 0x7fefd819a4b0 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7fefcc00e9c0 tx=0x7fefcc00ee80 comp rx=0 tx=0).stop 2024-04-02T14:45:52.598 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.601+0000 7fefe0800640 1 -- 172.21.15.90:0/2525377012 shutdown_connections 2024-04-02T14:45:52.598 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.601+0000 7fefe0800640 1 --2- 172.21.15.90:0/2525377012 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fefc00a3dc0 0x7fefc00a6280 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:52.599 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.601+0000 7fefe0800640 1 --2- 172.21.15.90:0/2525377012 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fefd8107ac0 0x7fefd819ed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:52.599 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.601+0000 7fefe0800640 1 --2- 172.21.15.90:0/2525377012 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fefd8102470 0x7fefd819a9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:52.599 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.601+0000 7fefe0800640 1 --2- 172.21.15.90:0/2525377012 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fefd8101ab0 0x7fefd819a4b0 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:45:52.599 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.601+0000 7fefe0800640 1 -- 172.21.15.90:0/2525377012 >> 172.21.15.90:0/2525377012 conn(0x7fefd80fd5c0 msgr2=0x7fefd80fd9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:45:52.599 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.601+0000 7fefe0800640 1 -- 172.21.15.90:0/2525377012 shutdown_connections 2024-04-02T14:45:52.599 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:45:52.601+0000 7fefe0800640 1 -- 172.21.15.90:0/2525377012 wait complete. 2024-04-02T14:45:53.931 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:45:53 smithi177 ceph-mon[36431]: pgmap v341: 97 pgs: 97 active+clean; 4.7 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 4.9 MiB/s wr, 521 op/s 2024-04-02T14:45:53.935 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:45:53 smithi090 ceph-mon[31674]: pgmap v341: 97 pgs: 97 active+clean; 4.7 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 4.9 MiB/s wr, 521 op/s 2024-04-02T14:45:54.105 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:45:53 smithi092 ceph-mon[36497]: pgmap v341: 97 pgs: 97 active+clean; 4.7 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 4.9 MiB/s wr, 521 op/s 2024-04-02T14:45:55.931 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:45:55 smithi177 ceph-mon[36431]: pgmap v342: 97 pgs: 97 active+clean; 4.7 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 7.2 MiB/s wr, 784 op/s 2024-04-02T14:45:55.932 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:45:55 smithi090 ceph-mon[31674]: pgmap v342: 97 pgs: 97 active+clean; 4.7 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 7.2 MiB/s wr, 784 op/s 2024-04-02T14:45:56.105 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:45:55 smithi092 ceph-mon[36497]: pgmap v342: 97 pgs: 97 active+clean; 4.7 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 7.2 MiB/s wr, 784 op/s 2024-04-02T14:45:56.825 INFO:teuthology.orchestra.run.smithi090.stderr:progress: 625.0 s, 1380.6 tps, lat 72.271 ms stddev 91.416 2024-04-02T14:45:57.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:45:57 smithi177 ceph-mon[36431]: pgmap v343: 97 pgs: 97 active+clean; 4.7 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 4.8 MiB/s wr, 517 op/s 2024-04-02T14:45:57.932 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:45:57 smithi090 ceph-mon[31674]: pgmap v343: 97 pgs: 97 active+clean; 4.7 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 4.8 MiB/s wr, 517 op/s 2024-04-02T14:45:58.104 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:45:57 smithi092 ceph-mon[36497]: pgmap v343: 97 pgs: 97 active+clean; 4.7 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 4.8 MiB/s wr, 517 op/s 2024-04-02T14:45:59.931 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:45:59 smithi177 ceph-mon[36431]: pgmap v344: 97 pgs: 97 active+clean; 4.7 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 5.1 MiB/s wr, 555 op/s 2024-04-02T14:45:59.932 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:45:59 smithi090 ceph-mon[31674]: pgmap v344: 97 pgs: 97 active+clean; 4.7 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 5.1 MiB/s wr, 555 op/s 2024-04-02T14:46:00.104 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:45:59 smithi092 ceph-mon[36497]: pgmap v344: 97 pgs: 97 active+clean; 4.7 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 5.1 MiB/s wr, 555 op/s 2024-04-02T14:46:01.825 INFO:teuthology.orchestra.run.smithi090.stderr:progress: 630.0 s, 1454.0 tps, lat 67.040 ms stddev 86.595 2024-04-02T14:46:01.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:46:01 smithi177 ceph-mon[36431]: pgmap v345: 97 pgs: 97 active+clean; 4.7 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 7.5 MiB/s wr, 811 op/s 2024-04-02T14:46:01.932 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:46:01 smithi090 ceph-mon[31674]: pgmap v345: 97 pgs: 97 active+clean; 4.7 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 7.5 MiB/s wr, 811 op/s 2024-04-02T14:46:02.104 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:46:01 smithi092 ceph-mon[36497]: pgmap v345: 97 pgs: 97 active+clean; 4.7 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 7.5 MiB/s wr, 811 op/s 2024-04-02T14:46:04.105 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:46:03 smithi092 ceph-mon[36497]: pgmap v346: 97 pgs: 97 active+clean; 4.7 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 5.2 MiB/s wr, 585 op/s 2024-04-02T14:46:04.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:46:03 smithi177 ceph-mon[36431]: pgmap v346: 97 pgs: 97 active+clean; 4.7 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 5.2 MiB/s wr, 585 op/s 2024-04-02T14:46:04.182 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:46:03 smithi090 ceph-mon[31674]: pgmap v346: 97 pgs: 97 active+clean; 4.7 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 5.2 MiB/s wr, 585 op/s 2024-04-02T14:46:04.500 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-02T14:46:04.500 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-02T14:46:04.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.702+0000 7f2bfa800640 1 -- 172.21.15.90:0/3357281279 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2bf4073ed0 msgr2=0x7f2bf40742b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:04.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.702+0000 7f2bfa800640 1 --2- 172.21.15.90:0/3357281279 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2bf4073ed0 0x7f2bf40742b0 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f2be8009ca0 tx=0x7f2be802c180 comp rx=0 tx=0).stop 2024-04-02T14:46:04.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.702+0000 7f2bfa800640 1 -- 172.21.15.90:0/3357281279 shutdown_connections 2024-04-02T14:46:04.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.702+0000 7f2bfa800640 1 --2- 172.21.15.90:0/3357281279 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2bf410e210 0x7f2bf4110620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:04.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.702+0000 7f2bfa800640 1 --2- 172.21.15.90:0/3357281279 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2bf40747f0 0x7f2bf410dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:04.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.702+0000 7f2bfa800640 1 --2- 172.21.15.90:0/3357281279 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2bf4073ed0 0x7f2bf40742b0 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:04.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.702+0000 7f2bfa800640 1 -- 172.21.15.90:0/3357281279 >> 172.21.15.90:0/3357281279 conn(0x7f2bf40fd4a0 msgr2=0x7f2bf40ff8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:46:04.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.703+0000 7f2bfa800640 1 -- 172.21.15.90:0/3357281279 shutdown_connections 2024-04-02T14:46:04.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.703+0000 7f2bfa800640 1 -- 172.21.15.90:0/3357281279 wait complete. 2024-04-02T14:46:04.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.704+0000 7f2bfa800640 1 Processor -- start 2024-04-02T14:46:04.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.704+0000 7f2bfa800640 1 -- start start 2024-04-02T14:46:04.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.704+0000 7f2bfa800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2bf4073ed0 0x7f2bf419a4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:04.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.704+0000 7f2bfa800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2bf40747f0 0x7f2bf419aa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:04.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.704+0000 7f2bfa800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2bf410e210 0x7f2bf419ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:04.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.704+0000 7f2bfa800640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f2bf4113f70 con 0x7f2bf410e210 2024-04-02T14:46:04.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.704+0000 7f2bfa800640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f2bf4113df0 con 0x7f2bf4073ed0 2024-04-02T14:46:04.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.704+0000 7f2bfa800640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f2bf41140f0 con 0x7f2bf40747f0 2024-04-02T14:46:04.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.704+0000 7f2bf9400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2bf4073ed0 0x7f2bf419a4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:04.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.704+0000 7f2bf9400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2bf4073ed0 0x7f2bf419a4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:52316/0 (socket says 172.21.15.90:52316) 2024-04-02T14:46:04.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.704+0000 7f2bf8a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2bf40747f0 0x7f2bf419aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:04.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.704+0000 7f2bf9400640 1 -- 172.21.15.90:0/2687215602 learned_addr learned my addr 172.21.15.90:0/2687215602 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:46:04.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.705+0000 7f2bf9e00640 1 --2- 172.21.15.90:0/2687215602 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2bf410e210 0x7f2bf419ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:04.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.705+0000 7f2bf9400640 1 -- 172.21.15.90:0/2687215602 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2bf40747f0 msgr2=0x7f2bf419aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:04.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.705+0000 7f2bf9400640 1 --2- 172.21.15.90:0/2687215602 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2bf40747f0 0x7f2bf419aa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:04.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.705+0000 7f2bf9400640 1 -- 172.21.15.90:0/2687215602 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2bf410e210 msgr2=0x7f2bf419ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:04.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.705+0000 7f2bf9400640 1 --2- 172.21.15.90:0/2687215602 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2bf410e210 0x7f2bf419ed40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:04.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.705+0000 7f2bf9400640 1 -- 172.21.15.90:0/2687215602 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2bf419f470 con 0x7f2bf4073ed0 2024-04-02T14:46:04.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.705+0000 7f2bf9e00640 1 --2- 172.21.15.90:0/2687215602 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2bf410e210 0x7f2bf419ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:46:04.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.705+0000 7f2bf8a00640 1 --2- 172.21.15.90:0/2687215602 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2bf40747f0 0x7f2bf419aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:46:04.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.705+0000 7f2bf9400640 1 --2- 172.21.15.90:0/2687215602 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2bf4073ed0 0x7f2bf419a4d0 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f2be8005e30 tx=0x7f2be8035d90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:46:04.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.705+0000 7f2bf2000640 1 -- 172.21.15.90:0/2687215602 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2be8039040 con 0x7f2bf4073ed0 2024-04-02T14:46:04.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.705+0000 7f2bf2000640 1 -- 172.21.15.90:0/2687215602 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2be8044070 con 0x7f2bf4073ed0 2024-04-02T14:46:04.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.705+0000 7f2bf2000640 1 -- 172.21.15.90:0/2687215602 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2be80328d0 con 0x7f2bf4073ed0 2024-04-02T14:46:04.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.705+0000 7f2bfa800640 1 -- 172.21.15.90:0/2687215602 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2bf419f700 con 0x7f2bf4073ed0 2024-04-02T14:46:04.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.706+0000 7f2bfa800640 1 -- 172.21.15.90:0/2687215602 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2bf41a6ff0 con 0x7f2bf4073ed0 2024-04-02T14:46:04.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.710+0000 7f2bfebbd740 1 Processor -- start 2024-04-02T14:46:04.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.710+0000 7f2bfebbd740 1 -- start start 2024-04-02T14:46:04.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.710+0000 7f2bf2000640 1 -- 172.21.15.90:0/2687215602 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f2be8032a70 con 0x7f2bf4073ed0 2024-04-02T14:46:04.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.710+0000 7f2bfebbd740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561941ebf960 0x561941ebfd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:04.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.710+0000 7f2bfebbd740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561941ec0280 0x561941ec8920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:04.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.710+0000 7f2bfebbd740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561941ec8e60 0x561941ecb250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:04.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.710+0000 7f2bfebbd740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x561941e23770 con 0x561941ebf960 2024-04-02T14:46:04.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.710+0000 7f2bfebbd740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x561941dcb700 con 0x561941ec0280 2024-04-02T14:46:04.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.710+0000 7f2bfebbd740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x561941e1f720 con 0x561941ec8e60 2024-04-02T14:46:04.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.710+0000 7f2bf9400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561941ec0280 0x561941ec8920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:04.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.711+0000 7f2bf8a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561941ec8e60 0x561941ecb250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:04.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.711+0000 7f2bf9e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561941ebf960 0x561941ebfd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:04.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.711+0000 7f2bf8a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561941ec8e60 0x561941ecb250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:58030/0 (socket says 172.21.15.90:58030) 2024-04-02T14:46:04.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.711+0000 7f2bf9400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561941ec0280 0x561941ec8920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:52328/0 (socket says 172.21.15.90:52328) 2024-04-02T14:46:04.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.711+0000 7f2bf8a00640 1 -- 172.21.15.90:0/4108637681 learned_addr learned my addr 172.21.15.90:0/4108637681 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:46:04.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.711+0000 7f2bf8a00640 1 -- 172.21.15.90:0/4108637681 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561941ec0280 msgr2=0x561941ec8920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:46:04.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.711+0000 7f2bf8a00640 1 --2- 172.21.15.90:0/4108637681 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561941ec0280 0x561941ec8920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:04.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.711+0000 7f2bf8a00640 1 -- 172.21.15.90:0/4108637681 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561941ebf960 msgr2=0x561941ebfd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:46:04.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.711+0000 7f2bf8a00640 1 --2- 172.21.15.90:0/4108637681 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561941ebf960 0x561941ebfd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:04.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.711+0000 7f2bf8a00640 1 -- 172.21.15.90:0/4108637681 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561941c017d0 con 0x561941ec8e60 2024-04-02T14:46:04.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.711+0000 7f2bf8a00640 1 --2- 172.21.15.90:0/4108637681 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561941ec8e60 0x561941ecb250 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f2be00114f0 tx=0x7f2be00119b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d77623c9bc7a23e4 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:46:04.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.712+0000 7f2be6600640 1 -- 172.21.15.90:0/4108637681 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2be001e070 con 0x561941ec8e60 2024-04-02T14:46:04.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.712+0000 7f2be6600640 1 -- 172.21.15.90:0/4108637681 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2be000faa0 con 0x561941ec8e60 2024-04-02T14:46:04.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.712+0000 7f2be6600640 1 -- 172.21.15.90:0/4108637681 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2be0019c40 con 0x561941ec8e60 2024-04-02T14:46:04.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.712+0000 7f2bf2000640 1 --2- 172.21.15.90:0/2687215602 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f2bdc0a3d20 0x7f2bdc0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:04.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.712+0000 7f2bf2000640 1 -- 172.21.15.90:0/2687215602 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 8550+0+0 (secure 0 0 0) 0x7f2be80f2c30 con 0x7f2bf4073ed0 2024-04-02T14:46:04.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.712+0000 7f2bf9e00640 1 --2- 172.21.15.90:0/2687215602 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f2bdc0a3d20 0x7f2bdc0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:04.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.712+0000 7f2bfebbd740 1 -- 172.21.15.90:0/4108637681 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561941ec8e60 msgr2=0x561941ecb250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:04.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.712+0000 7f2bfebbd740 1 --2- 172.21.15.90:0/4108637681 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561941ec8e60 0x561941ecb250 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f2be00114f0 tx=0x7f2be00119b0 comp rx=0 tx=0).stop 2024-04-02T14:46:04.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.713+0000 7f2bfebbd740 1 -- 172.21.15.90:0/4108637681 shutdown_connections 2024-04-02T14:46:04.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.713+0000 7f2bfebbd740 1 --2- 172.21.15.90:0/4108637681 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561941ec8e60 0x561941ecb250 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:04.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.713+0000 7f2bfebbd740 1 --2- 172.21.15.90:0/4108637681 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561941ec0280 0x561941ec8920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:04.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.713+0000 7f2bfebbd740 1 --2- 172.21.15.90:0/4108637681 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561941ebf960 0x561941ebfd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:04.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.713+0000 7f2bfebbd740 1 -- 172.21.15.90:0/4108637681 >> 172.21.15.90:0/4108637681 conn(0x561941eba090 msgr2=0x561941eba470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:46:04.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.713+0000 7f2bfebbd740 1 -- 172.21.15.90:0/4108637681 shutdown_connections 2024-04-02T14:46:04.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.713+0000 7f2bf9e00640 1 --2- 172.21.15.90:0/2687215602 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f2bdc0a3d20 0x7f2bdc0a61e0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f2bf406b030 tx=0x7f2bec005f90 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:46:04.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.713+0000 7f2bfebbd740 1 -- 172.21.15.90:0/4108637681 wait complete. 2024-04-02T14:46:04.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.713+0000 7f2bfebbd740 1 -- start start 2024-04-02T14:46:04.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.713+0000 7f2bfebbd740 1 Processor -- start 2024-04-02T14:46:04.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.714+0000 7f2bfebbd740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561941ebf960 0x561941ec5e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:04.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.714+0000 7f2bfebbd740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561941ec0280 0x561941ec6390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:04.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.714+0000 7f2bfebbd740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561941ec8e60 0x561941ec68d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:04.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.714+0000 7f2bfebbd740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x561941ecbc50 con 0x561941ebf960 2024-04-02T14:46:04.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.714+0000 7f2bfebbd740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x561941ddeac0 con 0x561941ec0280 2024-04-02T14:46:04.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.714+0000 7f2bfebbd740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x561941ecbdd0 con 0x561941ec8e60 2024-04-02T14:46:04.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.714+0000 7f2bfebbd740 10 client.0 _ll_register_callbacks cb 0x561941de1180 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:46:04.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.714+0000 7f2bf9400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561941ebf960 0x561941ec5e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:04.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.714+0000 7f2bf9e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561941ec8e60 0x561941ec68d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:04.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.714+0000 7f2bf8a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561941ec0280 0x561941ec6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:04.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.714+0000 7f2bf9e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561941ec8e60 0x561941ec68d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:58042/0 (socket says 172.21.15.90:58042) 2024-04-02T14:46:04.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.714+0000 7f2bf9400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561941ebf960 0x561941ec5e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:60010/0 (socket says 172.21.15.90:60010) 2024-04-02T14:46:04.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.714+0000 7f2bf9e00640 1 -- 172.21.15.90:0/3628853459 learned_addr learned my addr 172.21.15.90:0/3628853459 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:46:04.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.714+0000 7f2bf9e00640 1 -- 172.21.15.90:0/3628853459 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561941ec0280 msgr2=0x561941ec6390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:46:04.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.714+0000 7f2bf9e00640 1 --2- 172.21.15.90:0/3628853459 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561941ec0280 0x561941ec6390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:04.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.714+0000 7f2bf9e00640 1 -- 172.21.15.90:0/3628853459 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561941ebf960 msgr2=0x561941ec5e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:46:04.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.714+0000 7f2bf9e00640 1 --2- 172.21.15.90:0/3628853459 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561941ebf960 0x561941ec5e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:04.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.714+0000 7f2bf9e00640 1 -- 172.21.15.90:0/3628853459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561941b373f0 con 0x561941ec8e60 2024-04-02T14:46:04.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.714+0000 7f2bf8a00640 1 --2- 172.21.15.90:0/3628853459 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561941ec0280 0x561941ec6390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:46:04.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.715+0000 7f2bf9e00640 1 --2- 172.21.15.90:0/3628853459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561941ec8e60 0x561941ec68d0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f2bec020680 tx=0x7f2bec004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=102b5a925be17816 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:46:04.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.715+0000 7f2bdaa00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:46:04.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.715+0000 7f2bdaa00640 1 -- 172.21.15.90:0/3628853459 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bec02d060 con 0x561941ec8e60 2024-04-02T14:46:04.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.715+0000 7f2bdaa00640 1 -- 172.21.15.90:0/3628853459 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8550+0+0 (secure 0 0 0) 0x7f2bec018030 con 0x561941ec8e60 2024-04-02T14:46:04.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.715+0000 7f2bfebbd740 1 -- 172.21.15.90:0/3628853459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2be000faa0 con 0x561941ec8e60 2024-04-02T14:46:04.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.715+0000 7f2bdaa00640 1 -- 172.21.15.90:0/3628853459 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f2bec011710 con 0x561941ec8e60 2024-04-02T14:46:04.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.715+0000 7f2bfebbd740 10 client.28696 fetch_fsmap learned FSMap version 27 2024-04-02T14:46:04.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.715+0000 7f2bfebbd740 1 -- 172.21.15.90:0/3628853459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x561941af70f0 con 0x561941ec8e60 2024-04-02T14:46:04.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.715+0000 7f2bdaa00640 1 -- 172.21.15.90:0/3628853459 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2be000faa0 con 0x561941ec8e60 2024-04-02T14:46:04.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.715+0000 7f2bdaa00640 1 -- 172.21.15.90:0/3628853459 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f2bec028b70 con 0x561941ec8e60 2024-04-02T14:46:04.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.716+0000 7f2bfebbd740 10 client.28696 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:46:04.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.716+0000 7f2bfebbd740 10 client.28696 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:46:04.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.716+0000 7f2bfebbd740 20 client.28696 populate_metadata read hostname 'smithi090' 2024-04-02T14:46:04.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.716+0000 7f2bfebbd740 1 --2- 172.21.15.90:0/3628853459 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x561941ed8e60 0x561941edb2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:04.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.716+0000 7f2bfebbd740 4 client.28696 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:46:04.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.716+0000 7f2bfebbd740 1 -- 172.21.15.90:0/3628853459 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561941dc4630 con 0x561941ed8e60 2024-04-02T14:46:04.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.716+0000 7f2bf9400640 1 --2- 172.21.15.90:0/3628853459 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x561941ed8e60 0x561941edb2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:04.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.716+0000 7f2bf9400640 1 --2- 172.21.15.90:0/3628853459 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x561941ed8e60 0x561941edb2b0 crc :-1 s=READY pgs=290 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5c5d1520d4b963e1 server_cookie=c2987ebbc8f1eccb in_seq=0 out_seq=0 2024-04-02T14:46:04.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.716+0000 7f2bdaa00640 10 client.28696 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:46:04.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.717+0000 7f2bdaa00640 1 -- 172.21.15.90:0/3628853459 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x561941dc4630 con 0x561941ed8e60 2024-04-02T14:46:04.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.717+0000 7f2bdaa00640 10 client.28696 handle_command_reply: tid=0 2024-04-02T14:46:04.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.717+0000 7f2bfebbd740 1 client.28696 shutdown 2024-04-02T14:46:04.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.717+0000 7f2bfebbd740 1 -- 172.21.15.90:0/3628853459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561941ec8e60 msgr2=0x561941ec68d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:04.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.717+0000 7f2bfebbd740 1 --2- 172.21.15.90:0/3628853459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561941ec8e60 0x561941ec68d0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f2bec020680 tx=0x7f2bec004730 comp rx=0 tx=0).stop 2024-04-02T14:46:04.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.718+0000 7f2bfebbd740 1 -- 172.21.15.90:0/3628853459 shutdown_connections 2024-04-02T14:46:04.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.718+0000 7f2bfebbd740 1 --2- 172.21.15.90:0/3628853459 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x561941ed8e60 0x561941edb2b0 crc :-1 s=READY pgs=290 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:04.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.718+0000 7f2bdaa00640 0 client.28696 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:46:04.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.718+0000 7f2bfebbd740 1 --2- 172.21.15.90:0/3628853459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561941ec8e60 0x561941ec68d0 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:04.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.718+0000 7f2bfebbd740 1 --2- 172.21.15.90:0/3628853459 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561941ec0280 0x561941ec6390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:04.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.718+0000 7f2bfebbd740 1 --2- 172.21.15.90:0/3628853459 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561941ebf960 0x561941ec5e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:04.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.718+0000 7f2bfebbd740 1 -- 172.21.15.90:0/3628853459 >> 172.21.15.90:0/3628853459 conn(0x561941eba090 msgr2=0x561941eb9310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:46:04.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.718+0000 7f2bfebbd740 1 -- 172.21.15.90:0/3628853459 shutdown_connections 2024-04-02T14:46:04.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.718+0000 7f2bfebbd740 1 -- 172.21.15.90:0/3628853459 wait complete. 2024-04-02T14:46:04.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.718+0000 7f2bfebbd740 20 client.28696 trim_cache size 0 max 16384 2024-04-02T14:46:04.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.736+0000 7f2bfebbd740 1 Processor -- start 2024-04-02T14:46:04.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.737+0000 7f2bfebbd740 1 -- start start 2024-04-02T14:46:04.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.737+0000 7f2bfebbd740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561941ebf960 0x561941ec8580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:04.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.737+0000 7f2bfebbd740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561941ec0280 0x561941ed4b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:04.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.737+0000 7f2bfebbd740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561941ec8e60 0x561941ee0b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:04.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.737+0000 7f2bfebbd740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f2bf41af8b0 con 0x561941ebf960 2024-04-02T14:46:04.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.737+0000 7f2bfebbd740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f2bf41b13f0 con 0x561941ec0280 2024-04-02T14:46:04.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.737+0000 7f2bfebbd740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f2bf41aff80 con 0x561941ec8e60 2024-04-02T14:46:04.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.737+0000 7f2bf8a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561941ec0280 0x561941ed4b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:04.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.737+0000 7f2bf8a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561941ec0280 0x561941ed4b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:52354/0 (socket says 172.21.15.90:52354) 2024-04-02T14:46:04.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.737+0000 7f2bf8a00640 1 -- 172.21.15.90:0/1711584294 learned_addr learned my addr 172.21.15.90:0/1711584294 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:46:04.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.737+0000 7f2bf9400640 1 --2- 172.21.15.90:0/1711584294 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561941ebf960 0x561941ec8580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:04.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.737+0000 7f2bf9e00640 1 --2- 172.21.15.90:0/1711584294 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561941ec8e60 0x561941ee0b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:04.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.737+0000 7f2bf8a00640 1 -- 172.21.15.90:0/1711584294 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561941ec8e60 msgr2=0x561941ee0b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:46:04.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.737+0000 7f2bf8a00640 1 --2- 172.21.15.90:0/1711584294 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561941ec8e60 0x561941ee0b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:04.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.737+0000 7f2bf8a00640 1 -- 172.21.15.90:0/1711584294 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561941ebf960 msgr2=0x561941ec8580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:46:04.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.737+0000 7f2bf8a00640 1 --2- 172.21.15.90:0/1711584294 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561941ebf960 0x561941ec8580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:04.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.737+0000 7f2bf8a00640 1 -- 172.21.15.90:0/1711584294 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561941ec1a30 con 0x561941ec0280 2024-04-02T14:46:04.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.738+0000 7f2bf8a00640 1 --2- 172.21.15.90:0/1711584294 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561941ec0280 0x561941ed4b10 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f2be0014040 tx=0x7f2be0019840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6e581a0ffacf720e server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:46:04.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.738+0000 7f2be5c00640 1 -- 172.21.15.90:0/1711584294 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2be001e040 con 0x561941ec0280 2024-04-02T14:46:04.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.738+0000 7f2be5c00640 1 -- 172.21.15.90:0/1711584294 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2be000faa0 con 0x561941ec0280 2024-04-02T14:46:04.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.738+0000 7f2be5c00640 1 -- 172.21.15.90:0/1711584294 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2be0005310 con 0x561941ec0280 2024-04-02T14:46:04.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.739+0000 7f2bfebbd740 1 -- 172.21.15.90:0/1711584294 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561941ec0280 msgr2=0x561941ed4b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:04.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.739+0000 7f2bfebbd740 1 --2- 172.21.15.90:0/1711584294 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561941ec0280 0x561941ed4b10 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f2be0014040 tx=0x7f2be0019840 comp rx=0 tx=0).stop 2024-04-02T14:46:04.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.739+0000 7f2bfebbd740 1 -- 172.21.15.90:0/1711584294 shutdown_connections 2024-04-02T14:46:04.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.739+0000 7f2bfebbd740 1 --2- 172.21.15.90:0/1711584294 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561941ec8e60 0x561941ee0b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:04.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.739+0000 7f2bfebbd740 1 --2- 172.21.15.90:0/1711584294 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561941ec0280 0x561941ed4b10 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:04.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.739+0000 7f2bfebbd740 1 --2- 172.21.15.90:0/1711584294 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561941ebf960 0x561941ec8580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:04.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.739+0000 7f2bfebbd740 1 -- 172.21.15.90:0/1711584294 >> 172.21.15.90:0/1711584294 conn(0x561941eba090 msgr2=0x561941eb9310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:46:04.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.739+0000 7f2bfebbd740 1 -- 172.21.15.90:0/1711584294 shutdown_connections 2024-04-02T14:46:04.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.739+0000 7f2bfebbd740 1 -- 172.21.15.90:0/1711584294 wait complete. 2024-04-02T14:46:04.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.739+0000 7f2bfebbd740 1 -- start start 2024-04-02T14:46:04.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.739+0000 7f2bfebbd740 1 Processor -- start 2024-04-02T14:46:04.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.740+0000 7f2bfebbd740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561941ebf960 0x561941ed1020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:04.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.740+0000 7f2bfebbd740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561941ec0280 0x561941ed1560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:04.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.740+0000 7f2bfebbd740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561941ec8e60 0x561941ecdff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:04.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.740+0000 7f2bfebbd740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f2bf41b0d20 con 0x561941ebf960 2024-04-02T14:46:04.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.740+0000 7f2bfebbd740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f2bf41b0650 con 0x561941ec0280 2024-04-02T14:46:04.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.740+0000 7f2bfebbd740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f2bf41ad8b0 con 0x561941ec8e60 2024-04-02T14:46:04.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.740+0000 7f2bfebbd740 10 client.0 _ll_register_callbacks cb 0x7f2bec009900 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:46:04.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.740+0000 7f2bf9400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561941ebf960 0x561941ed1020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:04.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.740+0000 7f2bf8a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561941ec0280 0x561941ed1560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:04.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.740+0000 7f2bf9400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561941ebf960 0x561941ed1020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:60034/0 (socket says 172.21.15.90:60034) 2024-04-02T14:46:04.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.740+0000 7f2bf9400640 1 -- 172.21.15.90:0/330286996 learned_addr learned my addr 172.21.15.90:0/330286996 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:46:04.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.740+0000 7f2bf9e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561941ec8e60 0x561941ecdff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:04.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.741+0000 7f2bf8a00640 1 -- 172.21.15.90:0/330286996 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561941ec8e60 msgr2=0x561941ecdff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:46:04.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.741+0000 7f2bf8a00640 1 --2- 172.21.15.90:0/330286996 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561941ec8e60 0x561941ecdff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:04.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.741+0000 7f2bf8a00640 1 -- 172.21.15.90:0/330286996 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561941ebf960 msgr2=0x561941ed1020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:46:04.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.741+0000 7f2bf8a00640 1 --2- 172.21.15.90:0/330286996 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561941ebf960 0x561941ed1020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:04.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.741+0000 7f2bf8a00640 1 -- 172.21.15.90:0/330286996 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561941ece530 con 0x561941ec0280 2024-04-02T14:46:04.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.741+0000 7f2bf8a00640 1 --2- 172.21.15.90:0/330286996 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561941ec0280 0x561941ed1560 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x561941ecbd60 tx=0x7f2be0009f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b5a04dde8832254f server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:46:04.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.741+0000 7f2bdaa00640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:46:04.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.741+0000 7f2bdaa00640 1 -- 172.21.15.90:0/330286996 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2be0031060 con 0x561941ec0280 2024-04-02T14:46:04.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.741+0000 7f2bdaa00640 1 -- 172.21.15.90:0/330286996 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8550+0+0 (secure 0 0 0) 0x7f2be000f7d0 con 0x561941ec0280 2024-04-02T14:46:04.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.741+0000 7f2bfebbd740 1 -- 172.21.15.90:0/330286996 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2be000faa0 con 0x561941ec0280 2024-04-02T14:46:04.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.741+0000 7f2bdaa00640 1 -- 172.21.15.90:0/330286996 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f2be001e030 con 0x561941ec0280 2024-04-02T14:46:04.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.742+0000 7f2bfebbd740 10 client.28880 fetch_fsmap learned FSMap version 27 2024-04-02T14:46:04.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.742+0000 7f2bfebbd740 1 -- 172.21.15.90:0/330286996 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x561941f3fbf0 con 0x561941ec0280 2024-04-02T14:46:04.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.742+0000 7f2bdaa00640 1 -- 172.21.15.90:0/330286996 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2be000faa0 con 0x561941ec0280 2024-04-02T14:46:04.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.742+0000 7f2bdaa00640 1 -- 172.21.15.90:0/330286996 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f2be0019cc0 con 0x561941ec0280 2024-04-02T14:46:04.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.742+0000 7f2bfebbd740 10 client.28880 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:46:04.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.742+0000 7f2bfebbd740 10 client.28880 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:46:04.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.742+0000 7f2bfebbd740 20 client.28880 populate_metadata read hostname 'smithi090' 2024-04-02T14:46:04.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.742+0000 7f2bfebbd740 1 --2- 172.21.15.90:0/330286996 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x561941ed8e60 0x561941f42020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:04.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.742+0000 7f2bfebbd740 4 client.28880 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-02T14:46:04.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.742+0000 7f2bfebbd740 1 -- 172.21.15.90:0/330286996 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "scrub status"}) -- 0x561941e30cb0 con 0x561941ed8e60 2024-04-02T14:46:04.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.742+0000 7f2bf9400640 1 --2- 172.21.15.90:0/330286996 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x561941ed8e60 0x561941f42020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:04.741 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.743+0000 7f2bf9400640 1 --2- 172.21.15.90:0/330286996 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x561941ed8e60 0x561941f42020 crc :-1 s=READY pgs=291 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=833f44f75096a691 server_cookie=4711e307b259c62a in_seq=0 out_seq=0 2024-04-02T14:46:04.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.743+0000 7f2bdaa00640 10 client.28880 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:46:04.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.743+0000 7f2bdaa00640 1 -- 172.21.15.90:0/330286996 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x561941e30cb0 con 0x561941ed8e60 2024-04-02T14:46:04.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.743+0000 7f2bdaa00640 10 client.28880 handle_command_reply: tid=0 2024-04-02T14:46:04.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.743+0000 7f2bfebbd740 1 client.28880 shutdown 2024-04-02T14:46:04.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.743+0000 7f2bfebbd740 1 -- 172.21.15.90:0/330286996 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561941ec0280 msgr2=0x561941ed1560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:04.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.743+0000 7f2bfebbd740 1 --2- 172.21.15.90:0/330286996 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561941ec0280 0x561941ed1560 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x561941ecbd60 tx=0x7f2be0009f90 comp rx=0 tx=0).stop 2024-04-02T14:46:04.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.743+0000 7f2bfebbd740 1 -- 172.21.15.90:0/330286996 shutdown_connections 2024-04-02T14:46:04.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.743+0000 7f2bfebbd740 1 --2- 172.21.15.90:0/330286996 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x561941ed8e60 0x561941f42020 crc :-1 s=READY pgs=291 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:04.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.743+0000 7f2bfebbd740 1 --2- 172.21.15.90:0/330286996 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561941ec8e60 0x561941ecdff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:04.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.743+0000 7f2bdaa00640 0 client.28880 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:46:04.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.743+0000 7f2bfebbd740 1 --2- 172.21.15.90:0/330286996 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561941ec0280 0x561941ed1560 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:04.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.743+0000 7f2bfebbd740 1 --2- 172.21.15.90:0/330286996 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561941ebf960 0x561941ed1020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:04.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.743+0000 7f2bfebbd740 1 -- 172.21.15.90:0/330286996 >> 172.21.15.90:0/330286996 conn(0x561941eba090 msgr2=0x561941ec5e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:46:04.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.743+0000 7f2bfebbd740 1 -- 172.21.15.90:0/330286996 shutdown_connections 2024-04-02T14:46:04.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.743+0000 7f2bfebbd740 1 -- 172.21.15.90:0/330286996 wait complete. 2024-04-02T14:46:04.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.743+0000 7f2bfebbd740 20 client.28880 trim_cache size 0 max 16384 2024-04-02T14:46:04.743 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-02T14:46:04.743 INFO:teuthology.orchestra.run.smithi090.stdout: "status": "no active scrubs running", 2024-04-02T14:46:04.743 INFO:teuthology.orchestra.run.smithi090.stdout: "scrubs": {} 2024-04-02T14:46:04.743 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-02T14:46:04.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.744+0000 7f2bfa800640 1 -- 172.21.15.90:0/2687215602 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f2bdc0a3d20 msgr2=0x7f2bdc0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:04.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.744+0000 7f2bfa800640 1 --2- 172.21.15.90:0/2687215602 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f2bdc0a3d20 0x7f2bdc0a61e0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f2bf406b030 tx=0x7f2bec005f90 comp rx=0 tx=0).stop 2024-04-02T14:46:04.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.744+0000 7f2bfa800640 1 -- 172.21.15.90:0/2687215602 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2bf4073ed0 msgr2=0x7f2bf419a4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:04.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.744+0000 7f2bfa800640 1 --2- 172.21.15.90:0/2687215602 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2bf4073ed0 0x7f2bf419a4d0 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f2be8005e30 tx=0x7f2be8035d90 comp rx=0 tx=0).stop 2024-04-02T14:46:04.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.744+0000 7f2bfa800640 1 -- 172.21.15.90:0/2687215602 shutdown_connections 2024-04-02T14:46:04.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.744+0000 7f2bfa800640 1 --2- 172.21.15.90:0/2687215602 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f2bdc0a3d20 0x7f2bdc0a61e0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:04.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.744+0000 7f2bfa800640 1 --2- 172.21.15.90:0/2687215602 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2bf410e210 0x7f2bf419ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:04.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.744+0000 7f2bfa800640 1 --2- 172.21.15.90:0/2687215602 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2bf40747f0 0x7f2bf419aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:04.745 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.744+0000 7f2bfa800640 1 --2- 172.21.15.90:0/2687215602 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2bf4073ed0 0x7f2bf419a4d0 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:04.745 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.744+0000 7f2bfa800640 1 -- 172.21.15.90:0/2687215602 >> 172.21.15.90:0/2687215602 conn(0x7f2bf40fd4a0 msgr2=0x7f2bf40fecd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:46:04.745 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.744+0000 7f2bfa800640 1 -- 172.21.15.90:0/2687215602 shutdown_connections 2024-04-02T14:46:04.745 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.744+0000 7f2bfa800640 1 -- 172.21.15.90:0/2687215602 wait complete. 2024-04-02T14:46:04.755 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-02T14:46:04.755 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T14:46:04.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.950+0000 7ff468c00640 1 -- 172.21.15.90:0/525579647 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff464108ae0 msgr2=0x7ff46410cb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:04.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.950+0000 7ff468c00640 1 --2- 172.21.15.90:0/525579647 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff464108ae0 0x7ff46410cb20 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7ff44c009900 tx=0x7ff44c02bc40 comp rx=0 tx=0).stop 2024-04-02T14:46:04.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.950+0000 7ff468c00640 1 -- 172.21.15.90:0/525579647 shutdown_connections 2024-04-02T14:46:04.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.950+0000 7ff468c00640 1 --2- 172.21.15.90:0/525579647 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff46410d200 0x7ff464114dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:04.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.950+0000 7ff468c00640 1 --2- 172.21.15.90:0/525579647 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff464108ae0 0x7ff46410cb20 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:04.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.950+0000 7ff468c00640 1 --2- 172.21.15.90:0/525579647 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff464108130 0x7ff464108510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:04.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.950+0000 7ff468c00640 1 -- 172.21.15.90:0/525579647 >> 172.21.15.90:0/525579647 conn(0x7ff464077620 msgr2=0x7ff464079a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:46:04.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.950+0000 7ff468c00640 1 -- 172.21.15.90:0/525579647 shutdown_connections 2024-04-02T14:46:04.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.950+0000 7ff468c00640 1 -- 172.21.15.90:0/525579647 wait complete. 2024-04-02T14:46:04.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.951+0000 7ff468c00640 1 Processor -- start 2024-04-02T14:46:04.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.951+0000 7ff468c00640 1 -- start start 2024-04-02T14:46:04.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.951+0000 7ff468c00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff464108130 0x7ff4641a4450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:04.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.951+0000 7ff468c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff46410d200 0x7ff4641a4990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:04.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.951+0000 7ff468c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff46419e650 0x7ff46419eb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:04.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.951+0000 7ff468c00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7ff464118270 con 0x7ff46410d200 2024-04-02T14:46:04.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.951+0000 7ff468c00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7ff4641180f0 con 0x7ff464108130 2024-04-02T14:46:04.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.951+0000 7ff468c00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7ff4641183f0 con 0x7ff46419e650 2024-04-02T14:46:04.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.952+0000 7ff463400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff464108130 0x7ff4641a4450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:04.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.952+0000 7ff463400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff464108130 0x7ff4641a4450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:52376/0 (socket says 172.21.15.90:52376) 2024-04-02T14:46:04.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.952+0000 7ff463400640 1 -- 172.21.15.90:0/1476862397 learned_addr learned my addr 172.21.15.90:0/1476862397 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:46:04.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.952+0000 7ff463e00640 1 --2- 172.21.15.90:0/1476862397 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff46419e650 0x7ff46419eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:04.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.952+0000 7ff45be00640 1 --2- 172.21.15.90:0/1476862397 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff46410d200 0x7ff4641a4990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:04.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.952+0000 7ff463400640 1 -- 172.21.15.90:0/1476862397 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff46419e650 msgr2=0x7ff46419eb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:04.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.952+0000 7ff463400640 1 --2- 172.21.15.90:0/1476862397 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff46419e650 0x7ff46419eb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:04.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.952+0000 7ff463400640 1 -- 172.21.15.90:0/1476862397 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff46410d200 msgr2=0x7ff4641a4990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:04.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.952+0000 7ff463400640 1 --2- 172.21.15.90:0/1476862397 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff46410d200 0x7ff4641a4990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:04.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.952+0000 7ff463400640 1 -- 172.21.15.90:0/1476862397 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff46419f170 con 0x7ff464108130 2024-04-02T14:46:04.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.952+0000 7ff463e00640 1 --2- 172.21.15.90:0/1476862397 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff46419e650 0x7ff46419eb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:46:04.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.952+0000 7ff463400640 1 --2- 172.21.15.90:0/1476862397 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff464108130 0x7ff4641a4450 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7ff45400e9f0 tx=0x7ff45400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:46:04.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.952+0000 7ff460c00640 1 -- 172.21.15.90:0/1476862397 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff45400cd80 con 0x7ff464108130 2024-04-02T14:46:04.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.953+0000 7ff460c00640 1 -- 172.21.15.90:0/1476862397 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff454010910 con 0x7ff464108130 2024-04-02T14:46:04.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.953+0000 7ff468c00640 1 -- 172.21.15.90:0/1476862397 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff46419f460 con 0x7ff464108130 2024-04-02T14:46:04.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.953+0000 7ff460c00640 1 -- 172.21.15.90:0/1476862397 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff454018990 con 0x7ff464108130 2024-04-02T14:46:04.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.953+0000 7ff468c00640 1 -- 172.21.15.90:0/1476862397 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4641ab6b0 con 0x7ff464108130 2024-04-02T14:46:04.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.954+0000 7ff468c00640 1 -- 172.21.15.90:0/1476862397 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff464068c60 con 0x7ff464108130 2024-04-02T14:46:04.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.956+0000 7ff460c00640 1 -- 172.21.15.90:0/1476862397 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7ff454010ac0 con 0x7ff464108130 2024-04-02T14:46:04.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.957+0000 7ff460c00640 1 --2- 172.21.15.90:0/1476862397 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7ff4480a3da0 0x7ff4480a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:04.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.957+0000 7ff460c00640 1 -- 172.21.15.90:0/1476862397 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 8550+0+0 (secure 0 0 0) 0x7ff4540d0c00 con 0x7ff464108130 2024-04-02T14:46:04.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.957+0000 7ff45be00640 1 --2- 172.21.15.90:0/1476862397 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7ff4480a3da0 0x7ff4480a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:04.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.958+0000 7ff45be00640 1 --2- 172.21.15.90:0/1476862397 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7ff4480a3da0 0x7ff4480a6260 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7ff44c006010 tx=0x7ff44c000bc0 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:46:04.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:04.961+0000 7ff460c00640 1 -- 172.21.15.90:0/1476862397 <== mon.2 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7ff4540d7080 con 0x7ff464108130 2024-04-02T14:46:05.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.224+0000 7ff468c00640 1 -- 172.21.15.90:0/1476862397 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff464108ae0 con 0x7ff464108130 2024-04-02T14:46:05.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.226+0000 7ff460c00640 1 -- 172.21.15.90:0/1476862397 <== mon.2 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11927 (secure 0 0 0) 0x7ff45409d280 con 0x7ff464108130 2024-04-02T14:46:05.221 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-02T14:46:05.221 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14412,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6837/2852160309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6836","nonce":2852160309},{"type":"v1","addr":"172.21.15.90:6837","nonce":2852160309}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14478,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6841/1919376858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6840","nonce":1919376858},{"type":"v1","addr":"172.21.15.90:6841","nonce":1919376858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-02T14:32:22.319083+0000","modified":"2024-04-02T14:34:52.266152+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24590,"mds_1":24560,"mds_2":24547,"mds_3":24539,"mds_4":24523},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24454":{"gid":24454,"name":"l","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6833/1564319431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":1564319431},{"type":"v1","addr":"172.21.15.177:6833","nonce":1564319431}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.92:6833/1570337372","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":1570337372},{"type":"v1","addr":"172.21.15.92:6833","nonce":1570337372}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"j","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.90:6835/1277124891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6834","nonce":1277124891},{"type":"v1","addr":"172.21.15.90:6835","nonce":1277124891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.92:6835/860721164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":860721164},{"type":"v1","addr":"172.21.15.92:6835","nonce":860721164}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24511":{"gid":24511,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6835/3177244988","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":3177244988},{"type":"v1","addr":"172.21.15.177:6835","nonce":3177244988}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"c","rank":4,"incarnation":25,"state":"up:active","state_seq":9,"addr":"172.21.15.177:6837/4223414858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":4223414858},{"type":"v1","addr":"172.21.15.177:6837","nonce":4223414858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24539":{"gid":24539,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":8,"addr":"172.21.15.92:6837/1270635939","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":1270635939},{"type":"v1","addr":"172.21.15.92:6837","nonce":1270635939}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24547":{"gid":24547,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":7,"addr":"172.21.15.177:6839/664895663","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":664895663},{"type":"v1","addr":"172.21.15.177:6839","nonce":664895663}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"g","rank":1,"incarnation":19,"state":"up:active","state_seq":5,"addr":"172.21.15.90:6839/1498915870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6838","nonce":1498915870},{"type":"v1","addr":"172.21.15.90:6839","nonce":1498915870}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24590":{"gid":24590,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.92:6839/1054888219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":1054888219},{"type":"v1","addr":"172.21.15.92:6839","nonce":1054888219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24590,"qdb_cluster":[24547,24590,24523,24539,24560]},"id":1}]} 2024-04-02T14:46:05.221 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 27 2024-04-02T14:46:05.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.228+0000 7ff468c00640 1 -- 172.21.15.90:0/1476862397 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7ff4480a3da0 msgr2=0x7ff4480a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:05.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.228+0000 7ff468c00640 1 --2- 172.21.15.90:0/1476862397 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7ff4480a3da0 0x7ff4480a6260 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7ff44c006010 tx=0x7ff44c000bc0 comp rx=0 tx=0).stop 2024-04-02T14:46:05.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.228+0000 7ff468c00640 1 -- 172.21.15.90:0/1476862397 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff464108130 msgr2=0x7ff4641a4450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:05.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.228+0000 7ff468c00640 1 --2- 172.21.15.90:0/1476862397 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff464108130 0x7ff4641a4450 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7ff45400e9f0 tx=0x7ff45400eeb0 comp rx=0 tx=0).stop 2024-04-02T14:46:05.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.228+0000 7ff468c00640 1 -- 172.21.15.90:0/1476862397 shutdown_connections 2024-04-02T14:46:05.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.228+0000 7ff468c00640 1 --2- 172.21.15.90:0/1476862397 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7ff4480a3da0 0x7ff4480a6260 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:05.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.228+0000 7ff468c00640 1 --2- 172.21.15.90:0/1476862397 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff46419e650 0x7ff46419eb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:05.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.228+0000 7ff468c00640 1 --2- 172.21.15.90:0/1476862397 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff46410d200 0x7ff4641a4990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:05.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.228+0000 7ff468c00640 1 --2- 172.21.15.90:0/1476862397 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff464108130 0x7ff4641a4450 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:05.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.228+0000 7ff468c00640 1 -- 172.21.15.90:0/1476862397 >> 172.21.15.90:0/1476862397 conn(0x7ff464077620 msgr2=0x7ff464078970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:46:05.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.228+0000 7ff468c00640 1 -- 172.21.15.90:0/1476862397 shutdown_connections 2024-04-02T14:46:05.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.229+0000 7ff468c00640 1 -- 172.21.15.90:0/1476862397 wait complete. 2024-04-02T14:46:05.243 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-04-02T14:46:05.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.513+0000 7f7340c00640 1 -- 172.21.15.90:0/1606563711 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f733c109e20 msgr2=0x7f733c10a200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:05.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.513+0000 7f7340c00640 1 --2- 172.21.15.90:0/1606563711 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f733c109e20 0x7f733c10a200 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f732c009900 tx=0x7f732c02bc30 comp rx=0 tx=0).stop 2024-04-02T14:46:05.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.513+0000 7f7340c00640 1 -- 172.21.15.90:0/1606563711 shutdown_connections 2024-04-02T14:46:05.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.513+0000 7f7340c00640 1 --2- 172.21.15.90:0/1606563711 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f733c100300 0x7f733c110c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:05.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.513+0000 7f7340c00640 1 --2- 172.21.15.90:0/1606563711 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f733c10a7d0 0x7f733c0ffbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:05.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.513+0000 7f7340c00640 1 --2- 172.21.15.90:0/1606563711 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f733c109e20 0x7f733c10a200 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:05.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.513+0000 7f7340c00640 1 -- 172.21.15.90:0/1606563711 >> 172.21.15.90:0/1606563711 conn(0x7f733c076b50 msgr2=0x7f733c0fdee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:46:05.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.513+0000 7f7340c00640 1 -- 172.21.15.90:0/1606563711 shutdown_connections 2024-04-02T14:46:05.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.514+0000 7f7340c00640 1 -- 172.21.15.90:0/1606563711 wait complete. 2024-04-02T14:46:05.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.515+0000 7f7340c00640 1 Processor -- start 2024-04-02T14:46:05.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.515+0000 7f7340c00640 1 -- start start 2024-04-02T14:46:05.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.516+0000 7f7340c00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f733c100300 0x7f733c1a05f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:05.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.516+0000 7f7340c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f733c1a0b30 0x7f733c19a700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:05.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.516+0000 7f7340c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f733c19ac40 0x7f733c19b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:05.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.516+0000 7f7340c00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f733c1140e0 con 0x7f733c19ac40 2024-04-02T14:46:05.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.516+0000 7f7340c00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f733c113f60 con 0x7f733c100300 2024-04-02T14:46:05.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.516+0000 7f7340c00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f733c114260 con 0x7f733c1a0b30 2024-04-02T14:46:05.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.517+0000 7f7339e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f733c100300 0x7f733c1a05f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:05.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.517+0000 7f7339e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f733c100300 0x7f733c1a05f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:52388/0 (socket says 172.21.15.90:52388) 2024-04-02T14:46:05.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.517+0000 7f7339e00640 1 -- 172.21.15.90:0/2074023827 learned_addr learned my addr 172.21.15.90:0/2074023827 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:46:05.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.517+0000 7f733a800640 1 --2- 172.21.15.90:0/2074023827 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f733c19ac40 0x7f733c19b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:05.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.517+0000 7f7339e00640 1 -- 172.21.15.90:0/2074023827 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f733c1a0b30 msgr2=0x7f733c19a700 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T14:46:05.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.518+0000 7f7339400640 1 --2- 172.21.15.90:0/2074023827 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f733c1a0b30 0x7f733c19a700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:05.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.518+0000 7f7339e00640 1 --2- 172.21.15.90:0/2074023827 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f733c1a0b30 0x7f733c19a700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:05.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.518+0000 7f7339e00640 1 -- 172.21.15.90:0/2074023827 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f733c19ac40 msgr2=0x7f733c19b0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:05.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.518+0000 7f7339e00640 1 --2- 172.21.15.90:0/2074023827 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f733c19ac40 0x7f733c19b0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:05.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.518+0000 7f7339e00640 1 -- 172.21.15.90:0/2074023827 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f733c103780 con 0x7f733c100300 2024-04-02T14:46:05.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.518+0000 7f733a800640 1 --2- 172.21.15.90:0/2074023827 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f733c19ac40 0x7f733c19b0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:46:05.514 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.519+0000 7f7339e00640 1 --2- 172.21.15.90:0/2074023827 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f733c100300 0x7f733c1a05f0 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f732c02c240 tx=0x7f732c004430 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:46:05.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.520+0000 7f7332a00640 1 -- 172.21.15.90:0/2074023827 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f732c043020 con 0x7f733c100300 2024-04-02T14:46:05.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.520+0000 7f7332a00640 1 -- 172.21.15.90:0/2074023827 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f732c0054e0 con 0x7f733c100300 2024-04-02T14:46:05.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.520+0000 7f7332a00640 1 -- 172.21.15.90:0/2074023827 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f732c003bc0 con 0x7f733c100300 2024-04-02T14:46:05.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.521+0000 7f7340c00640 1 -- 172.21.15.90:0/2074023827 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f733c103a10 con 0x7f733c100300 2024-04-02T14:46:05.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.521+0000 7f7340c00640 1 -- 172.21.15.90:0/2074023827 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f733c103f10 con 0x7f733c100300 2024-04-02T14:46:05.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.528+0000 7f73437f1740 1 Processor -- start 2024-04-02T14:46:05.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.529+0000 7f73437f1740 1 -- start start 2024-04-02T14:46:05.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.530+0000 7f7332a00640 1 -- 172.21.15.90:0/2074023827 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f732c046020 con 0x7f733c100300 2024-04-02T14:46:05.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.531+0000 7f73437f1740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ab499449f0 0x55ab49943740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:05.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.531+0000 7f73437f1740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ab49943d10 0x55ab49944190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:05.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.531+0000 7f73437f1740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ab4994d080 0x55ab4994f470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:05.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.531+0000 7f73437f1740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55ab49884400 con 0x55ab499449f0 2024-04-02T14:46:05.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.531+0000 7f73437f1740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55ab49868f10 con 0x55ab4994d080 2024-04-02T14:46:05.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.531+0000 7f73437f1740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55ab498a86f0 con 0x55ab49943d10 2024-04-02T14:46:05.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.531+0000 7f733a800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ab499449f0 0x55ab49943740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:05.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.531+0000 7f7339e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ab49943d10 0x55ab49944190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:05.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.531+0000 7f733a800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ab499449f0 0x55ab49943740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:60090/0 (socket says 172.21.15.90:60090) 2024-04-02T14:46:05.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.531+0000 7f733a800640 1 -- 172.21.15.90:0/4104985935 learned_addr learned my addr 172.21.15.90:0/4104985935 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:46:05.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.532+0000 7f7339400640 1 --2- 172.21.15.90:0/4104985935 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ab4994d080 0x55ab4994f470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:05.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.532+0000 7f733a800640 1 -- 172.21.15.90:0/4104985935 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ab49943d10 msgr2=0x55ab49944190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:46:05.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.532+0000 7f733a800640 1 --2- 172.21.15.90:0/4104985935 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ab49943d10 0x55ab49944190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:05.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.532+0000 7f733a800640 1 -- 172.21.15.90:0/4104985935 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ab4994d080 msgr2=0x55ab4994f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:46:05.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.532+0000 7f733a800640 1 --2- 172.21.15.90:0/4104985935 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ab4994d080 0x55ab4994f470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:05.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.532+0000 7f733a800640 1 -- 172.21.15.90:0/4104985935 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ab495bc3f0 con 0x55ab499449f0 2024-04-02T14:46:05.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.532+0000 7f733a800640 1 --2- 172.21.15.90:0/4104985935 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ab499449f0 0x55ab49943740 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f7334009ea0 tx=0x7f7334009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ff55bacead6bb80 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:46:05.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.533+0000 7f7327000640 1 -- 172.21.15.90:0/4104985935 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7334004d20 con 0x55ab499449f0 2024-04-02T14:46:05.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.533+0000 7f7332a00640 1 --2- 172.21.15.90:0/2074023827 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f73200a3e20 0x7f73200a62e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:05.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.533+0000 7f7327000640 1 -- 172.21.15.90:0/4104985935 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7334005500 con 0x55ab499449f0 2024-04-02T14:46:05.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.533+0000 7f7327000640 1 -- 172.21.15.90:0/4104985935 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7334011880 con 0x55ab499449f0 2024-04-02T14:46:05.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.534+0000 7f7339e00640 1 --2- 172.21.15.90:0/2074023827 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f73200a3e20 0x7f73200a62e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:05.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.535+0000 7f73437f1740 1 -- 172.21.15.90:0/4104985935 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ab499449f0 msgr2=0x55ab49943740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:05.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.535+0000 7f73437f1740 1 --2- 172.21.15.90:0/4104985935 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ab499449f0 0x55ab49943740 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f7334009ea0 tx=0x7f7334009f90 comp rx=0 tx=0).stop 2024-04-02T14:46:05.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.535+0000 7f7332a00640 1 -- 172.21.15.90:0/2074023827 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 8550+0+0 (secure 0 0 0) 0x7f732c0f2240 con 0x7f733c100300 2024-04-02T14:46:05.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.536+0000 7f73437f1740 1 -- 172.21.15.90:0/4104985935 shutdown_connections 2024-04-02T14:46:05.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.536+0000 7f73437f1740 1 --2- 172.21.15.90:0/4104985935 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ab4994d080 0x55ab4994f470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:05.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.536+0000 7f73437f1740 1 --2- 172.21.15.90:0/4104985935 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ab49943d10 0x55ab49944190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:05.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.536+0000 7f73437f1740 1 --2- 172.21.15.90:0/4104985935 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ab499449f0 0x55ab49943740 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:05.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.536+0000 7f73437f1740 1 -- 172.21.15.90:0/4104985935 >> 172.21.15.90:0/4104985935 conn(0x55ab4990aa50 msgr2=0x55ab4993e760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:46:05.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.536+0000 7f7339e00640 1 --2- 172.21.15.90:0/2074023827 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f73200a3e20 0x7f73200a62e0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x55ab49944f40 tx=0x7f732c0f7040 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:46:05.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.536+0000 7f73437f1740 1 -- 172.21.15.90:0/4104985935 shutdown_connections 2024-04-02T14:46:05.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.537+0000 7f73437f1740 1 -- 172.21.15.90:0/4104985935 wait complete. 2024-04-02T14:46:05.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.538+0000 7f73437f1740 1 -- start start 2024-04-02T14:46:05.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.538+0000 7f73437f1740 1 Processor -- start 2024-04-02T14:46:05.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.540+0000 7f73437f1740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ab49943d10 0x55ab4994a080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:05.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.541+0000 7f73437f1740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ab499449f0 0x55ab4994a5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:05.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.541+0000 7f733a800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ab49943d10 0x55ab4994a080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:05.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.541+0000 7f733a800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ab49943d10 0x55ab4994a080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:60092/0 (socket says 172.21.15.90:60092) 2024-04-02T14:46:05.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.541+0000 7f73437f1740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ab4994d080 0x55ab4994ab00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:05.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.541+0000 7f73437f1740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55ab498a46a0 con 0x55ab49943d10 2024-04-02T14:46:05.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.541+0000 7f73437f1740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55ab49850980 con 0x55ab4994d080 2024-04-02T14:46:05.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.541+0000 7f73437f1740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55ab49863cb0 con 0x55ab499449f0 2024-04-02T14:46:05.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.541+0000 7f73437f1740 10 client.0 _ll_register_callbacks cb 0x55ab498636a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:46:05.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.542+0000 7f733a800640 1 -- 172.21.15.90:0/1096385064 learned_addr learned my addr 172.21.15.90:0/1096385064 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:46:05.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.542+0000 7f7339400640 1 --2- 172.21.15.90:0/1096385064 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ab499449f0 0x55ab4994a5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:05.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.542+0000 7f733a800640 1 --2- 172.21.15.90:0/1096385064 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ab4994d080 0x55ab4994ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:05.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.543+0000 7f7339400640 1 -- 172.21.15.90:0/1096385064 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ab4994d080 msgr2=0x55ab4994ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:46:05.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.543+0000 7f7339400640 1 --2- 172.21.15.90:0/1096385064 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ab4994d080 0x55ab4994ab00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:05.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.543+0000 7f7339400640 1 -- 172.21.15.90:0/1096385064 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ab49943d10 msgr2=0x55ab4994a080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:46:05.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.543+0000 7f7339400640 1 --2- 172.21.15.90:0/1096385064 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ab49943d10 0x55ab4994a080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:05.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.543+0000 7f7339400640 1 -- 172.21.15.90:0/1096385064 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ab496867d0 con 0x55ab499449f0 2024-04-02T14:46:05.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.544+0000 7f7339400640 1 --2- 172.21.15.90:0/1096385064 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ab499449f0 0x55ab4994a5c0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f7328010370 tx=0x7f7328010830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=24bccd8a3bc1b0df server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:46:05.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.544+0000 7f731b400640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:46:05.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.546+0000 7f731b400640 1 -- 172.21.15.90:0/1096385064 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7328005220 con 0x55ab499449f0 2024-04-02T14:46:05.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.546+0000 7f731b400640 1 -- 172.21.15.90:0/1096385064 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8550+0+0 (secure 0 0 0) 0x7f73280053c0 con 0x55ab499449f0 2024-04-02T14:46:05.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.547+0000 7f73437f1740 1 -- 172.21.15.90:0/1096385064 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7334005500 con 0x55ab499449f0 2024-04-02T14:46:05.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.547+0000 7f731b400640 1 -- 172.21.15.90:0/1096385064 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f73280055c0 con 0x55ab499449f0 2024-04-02T14:46:05.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.549+0000 7f73437f1740 10 client.28717 fetch_fsmap learned FSMap version 27 2024-04-02T14:46:05.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.549+0000 7f73437f1740 1 -- 172.21.15.90:0/1096385064 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55ab4995bf90 con 0x55ab499449f0 2024-04-02T14:46:05.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.550+0000 7f731b400640 1 -- 172.21.15.90:0/1096385064 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7334005500 con 0x55ab499449f0 2024-04-02T14:46:05.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.551+0000 7f731b400640 1 -- 172.21.15.90:0/1096385064 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f732800f040 con 0x55ab499449f0 2024-04-02T14:46:05.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.551+0000 7f73437f1740 10 client.28717 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:46:05.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.551+0000 7f73437f1740 10 client.28717 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.c 2024-04-02T14:46:05.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.551+0000 7f73437f1740 20 client.28717 populate_metadata read hostname 'smithi090' 2024-04-02T14:46:05.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.551+0000 7f73437f1740 1 --2- 172.21.15.90:0/1096385064 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x55ab4995d450 0x55ab4995f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:05.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.551+0000 7f73437f1740 4 client.28717 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:46:05.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.551+0000 7f73437f1740 1 -- 172.21.15.90:0/1096385064 --> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ab49868040 con 0x55ab4995d450 2024-04-02T14:46:05.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.554+0000 7f733a800640 1 --2- 172.21.15.90:0/1096385064 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x55ab4995d450 0x55ab4995f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:05.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.555+0000 7f733a800640 1 --2- 172.21.15.90:0/1096385064 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x55ab4995d450 0x55ab4995f840 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=73c9e7a0c9e6168e server_cookie=c0b4bcc5b0daf7bf in_seq=0 out_seq=0 2024-04-02T14:46:05.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.556+0000 7f731b400640 10 client.28717 ms_handle_connect on v2:172.21.15.177:6836/4223414858 2024-04-02T14:46:05.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.556+0000 7f731b400640 1 -- 172.21.15.90:0/1096385064 <== mds.4 v2:172.21.15.177:6836/4223414858 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ab49868040 con 0x55ab4995d450 2024-04-02T14:46:05.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.556+0000 7f731b400640 10 client.28717 handle_command_reply: tid=0 2024-04-02T14:46:05.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.556+0000 7f73437f1740 1 client.28717 shutdown 2024-04-02T14:46:05.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.557+0000 7f73437f1740 1 -- 172.21.15.90:0/1096385064 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ab499449f0 msgr2=0x55ab4994a5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:05.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.557+0000 7f73437f1740 1 --2- 172.21.15.90:0/1096385064 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ab499449f0 0x55ab4994a5c0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f7328010370 tx=0x7f7328010830 comp rx=0 tx=0).stop 2024-04-02T14:46:05.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.557+0000 7f73437f1740 1 -- 172.21.15.90:0/1096385064 shutdown_connections 2024-04-02T14:46:05.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.557+0000 7f73437f1740 1 --2- 172.21.15.90:0/1096385064 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ab4994d080 0x55ab4994ab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:05.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.558+0000 7f73437f1740 1 --2- 172.21.15.90:0/1096385064 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ab499449f0 0x55ab4994a5c0 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:05.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.558+0000 7f73437f1740 1 --2- 172.21.15.90:0/1096385064 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x55ab4995d450 0x55ab4995f840 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:05.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.558+0000 7f73437f1740 1 --2- 172.21.15.90:0/1096385064 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ab49943d10 0x55ab4994a080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:05.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.558+0000 7f731b400640 0 client.28717 ms_handle_reset on v2:172.21.15.177:6836/4223414858 2024-04-02T14:46:05.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.558+0000 7f73437f1740 1 -- 172.21.15.90:0/1096385064 >> 172.21.15.90:0/1096385064 conn(0x55ab4990aa50 msgr2=0x55ab499404b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:46:05.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.558+0000 7f73437f1740 1 -- 172.21.15.90:0/1096385064 shutdown_connections 2024-04-02T14:46:05.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.558+0000 7f73437f1740 1 -- 172.21.15.90:0/1096385064 wait complete. 2024-04-02T14:46:05.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.558+0000 7f73437f1740 20 client.28717 trim_cache size 0 max 16384 2024-04-02T14:46:05.583 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.588+0000 7f73437f1740 1 Processor -- start 2024-04-02T14:46:05.583 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.588+0000 7f73437f1740 1 -- start start 2024-04-02T14:46:05.583 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.589+0000 7f73437f1740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ab49943d10 0x55ab4994c7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:05.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.589+0000 7f73437f1740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ab499449f0 0x55ab49958af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:05.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.589+0000 7f73437f1740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ab4994d080 0x55ab49940250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:05.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.589+0000 7f73437f1740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f733c1b4560 con 0x55ab4994d080 2024-04-02T14:46:05.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.589+0000 7f73437f1740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f733c1b60a0 con 0x55ab499449f0 2024-04-02T14:46:05.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.589+0000 7f73437f1740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f733c1b4c30 con 0x55ab49943d10 2024-04-02T14:46:05.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.589+0000 7f733a800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ab49943d10 0x55ab4994c7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:05.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.589+0000 7f7339400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ab499449f0 0x55ab49958af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:05.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.589+0000 7f7339400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ab499449f0 0x55ab49958af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:52412/0 (socket says 172.21.15.90:52412) 2024-04-02T14:46:05.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.589+0000 7f7339400640 1 -- 172.21.15.90:0/3818090220 learned_addr learned my addr 172.21.15.90:0/3818090220 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:46:05.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.590+0000 7f7339400640 1 -- 172.21.15.90:0/3818090220 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ab49943d10 msgr2=0x55ab4994c7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:46:05.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.590+0000 7f733a800640 1 --2- 172.21.15.90:0/3818090220 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ab4994d080 0x55ab49940250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:05.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.590+0000 7f7339400640 1 --2- 172.21.15.90:0/3818090220 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ab49943d10 0x55ab4994c7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:05.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.590+0000 7f7339400640 1 -- 172.21.15.90:0/3818090220 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ab4994d080 msgr2=0x55ab49940250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:46:05.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.590+0000 7f7339400640 1 --2- 172.21.15.90:0/3818090220 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ab4994d080 0x55ab49940250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:05.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.590+0000 7f7339400640 1 -- 172.21.15.90:0/3818090220 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ab4995fb70 con 0x55ab499449f0 2024-04-02T14:46:05.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.590+0000 7f733a800640 1 --2- 172.21.15.90:0/3818090220 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ab49943d10 0x55ab4994c7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:46:05.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.592+0000 7f7339400640 1 --2- 172.21.15.90:0/3818090220 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ab499449f0 0x55ab49958af0 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f7328001eb0 tx=0x7f7328026630 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3ed569a7dec24650 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:46:05.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.592+0000 7f7326600640 1 -- 172.21.15.90:0/3818090220 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f732802b070 con 0x55ab499449f0 2024-04-02T14:46:05.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.592+0000 7f7326600640 1 -- 172.21.15.90:0/3818090220 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f732800e410 con 0x55ab499449f0 2024-04-02T14:46:05.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.592+0000 7f7326600640 1 -- 172.21.15.90:0/3818090220 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f732802e610 con 0x55ab499449f0 2024-04-02T14:46:05.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.593+0000 7f73437f1740 1 -- 172.21.15.90:0/3818090220 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ab499449f0 msgr2=0x55ab49958af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:05.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.593+0000 7f73437f1740 1 --2- 172.21.15.90:0/3818090220 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ab499449f0 0x55ab49958af0 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f7328001eb0 tx=0x7f7328026630 comp rx=0 tx=0).stop 2024-04-02T14:46:05.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.594+0000 7f73437f1740 1 -- 172.21.15.90:0/3818090220 shutdown_connections 2024-04-02T14:46:05.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.595+0000 7f73437f1740 1 --2- 172.21.15.90:0/3818090220 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ab4994d080 0x55ab49940250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:05.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.595+0000 7f73437f1740 1 --2- 172.21.15.90:0/3818090220 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ab499449f0 0x55ab49958af0 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:05.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.595+0000 7f73437f1740 1 --2- 172.21.15.90:0/3818090220 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ab49943d10 0x55ab4994c7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:05.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.595+0000 7f73437f1740 1 -- 172.21.15.90:0/3818090220 >> 172.21.15.90:0/3818090220 conn(0x55ab4990aa50 msgr2=0x55ab4993fb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:46:05.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.595+0000 7f73437f1740 1 -- 172.21.15.90:0/3818090220 shutdown_connections 2024-04-02T14:46:05.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.595+0000 7f73437f1740 1 -- 172.21.15.90:0/3818090220 wait complete. 2024-04-02T14:46:05.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.596+0000 7f73437f1740 1 -- start start 2024-04-02T14:46:05.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.596+0000 7f73437f1740 1 Processor -- start 2024-04-02T14:46:05.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.597+0000 7f73437f1740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ab49943d10 0x55ab49947cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:05.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.597+0000 7f73437f1740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ab499449f0 0x55ab49954fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:05.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.597+0000 7f73437f1740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ab4994d080 0x55ab49955550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:05.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.598+0000 7f73437f1740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f733c1b59d0 con 0x55ab49943d10 2024-04-02T14:46:05.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.598+0000 7f73437f1740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f733c1b5300 con 0x55ab499449f0 2024-04-02T14:46:05.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.598+0000 7f73437f1740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f733c1b2560 con 0x55ab4994d080 2024-04-02T14:46:05.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.598+0000 7f73437f1740 10 client.0 _ll_register_callbacks cb 0x7f732800ebc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:46:05.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.598+0000 7f733a800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ab4994d080 0x55ab49955550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:05.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.598+0000 7f733a800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ab4994d080 0x55ab49955550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:58146/0 (socket says 172.21.15.90:58146) 2024-04-02T14:46:05.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.598+0000 7f733a800640 1 -- 172.21.15.90:0/637270240 learned_addr learned my addr 172.21.15.90:0/637270240 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:46:05.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.598+0000 7f7339400640 1 --2- 172.21.15.90:0/637270240 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ab499449f0 0x55ab49954fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:05.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.598+0000 7f733a800640 1 --2- 172.21.15.90:0/637270240 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ab49943d10 0x55ab49947cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:05.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.599+0000 7f733a800640 1 -- 172.21.15.90:0/637270240 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ab499449f0 msgr2=0x55ab49954fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:46:05.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.599+0000 7f733a800640 1 --2- 172.21.15.90:0/637270240 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ab499449f0 0x55ab49954fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:05.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.599+0000 7f733a800640 1 -- 172.21.15.90:0/637270240 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ab49943d10 msgr2=0x55ab49947cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:46:05.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.599+0000 7f733a800640 1 --2- 172.21.15.90:0/637270240 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ab49943d10 0x55ab49947cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:05.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.599+0000 7f733a800640 1 -- 172.21.15.90:0/637270240 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ab4994e250 con 0x55ab4994d080 2024-04-02T14:46:05.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.599+0000 7f733a800640 1 --2- 172.21.15.90:0/637270240 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ab4994d080 0x55ab49955550 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f7334000f80 tx=0x7f733402d040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fb38c7f3acbf7c5c server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:46:05.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.599+0000 7f731b400640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:46:05.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.600+0000 7f731b400640 1 -- 172.21.15.90:0/637270240 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7334039060 con 0x55ab4994d080 2024-04-02T14:46:05.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.600+0000 7f731b400640 1 -- 172.21.15.90:0/637270240 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8550+0+0 (secure 0 0 0) 0x7f733402e340 con 0x55ab4994d080 2024-04-02T14:46:05.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.600+0000 7f73437f1740 1 -- 172.21.15.90:0/637270240 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f732800e410 con 0x55ab4994d080 2024-04-02T14:46:05.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.600+0000 7f731b400640 1 -- 172.21.15.90:0/637270240 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f7334021040 con 0x55ab4994d080 2024-04-02T14:46:05.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.600+0000 7f73437f1740 10 client.28723 fetch_fsmap learned FSMap version 27 2024-04-02T14:46:05.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.600+0000 7f73437f1740 1 -- 172.21.15.90:0/637270240 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55ab499c3f30 con 0x55ab4994d080 2024-04-02T14:46:05.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.601+0000 7f731b400640 1 -- 172.21.15.90:0/637270240 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f73340114b0 con 0x55ab4994d080 2024-04-02T14:46:05.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.601+0000 7f731b400640 1 -- 172.21.15.90:0/637270240 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f7334027040 con 0x55ab4994d080 2024-04-02T14:46:05.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.601+0000 7f73437f1740 10 client.28723 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:46:05.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.601+0000 7f73437f1740 10 client.28723 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.c 2024-04-02T14:46:05.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.601+0000 7f73437f1740 20 client.28723 populate_metadata read hostname 'smithi090' 2024-04-02T14:46:05.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.601+0000 7f73437f1740 1 --2- 172.21.15.90:0/637270240 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x55ab499c44c0 0x55ab499c68b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:05.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.601+0000 7f73437f1740 4 client.28723 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-02T14:46:05.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.601+0000 7f73437f1740 1 -- 172.21.15.90:0/637270240 --> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7f732800e410 con 0x55ab499c44c0 2024-04-02T14:46:05.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.602+0000 7f7339400640 1 --2- 172.21.15.90:0/637270240 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x55ab499c44c0 0x55ab499c68b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:05.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.602+0000 7f7339400640 1 --2- 172.21.15.90:0/637270240 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x55ab499c44c0 0x55ab499c68b0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=cd4a985c038a81a9 server_cookie=fe69a9d71c7835e5 in_seq=0 out_seq=0 2024-04-02T14:46:05.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.602+0000 7f731b400640 10 client.28723 ms_handle_connect on v2:172.21.15.177:6836/4223414858 2024-04-02T14:46:05.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.602+0000 7f731b400640 1 -- 172.21.15.90:0/637270240 <== mds.4 v2:172.21.15.177:6836/4223414858 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7f732800e410 con 0x55ab499c44c0 2024-04-02T14:46:05.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.602+0000 7f731b400640 10 client.28723 handle_command_reply: tid=0 2024-04-02T14:46:05.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.602+0000 7f73437f1740 1 client.28723 shutdown 2024-04-02T14:46:05.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.603+0000 7f73437f1740 1 -- 172.21.15.90:0/637270240 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ab4994d080 msgr2=0x55ab49955550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:05.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.604+0000 7f73437f1740 1 --2- 172.21.15.90:0/637270240 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ab4994d080 0x55ab49955550 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f7334000f80 tx=0x7f733402d040 comp rx=0 tx=0).stop 2024-04-02T14:46:05.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.604+0000 7f73437f1740 1 -- 172.21.15.90:0/637270240 shutdown_connections 2024-04-02T14:46:05.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.604+0000 7f73437f1740 1 --2- 172.21.15.90:0/637270240 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ab4994d080 0x55ab49955550 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:05.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.604+0000 7f73437f1740 1 --2- 172.21.15.90:0/637270240 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ab499449f0 0x55ab49954fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:05.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.604+0000 7f73437f1740 1 --2- 172.21.15.90:0/637270240 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x55ab499c44c0 0x55ab499c68b0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:05.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.604+0000 7f73437f1740 1 --2- 172.21.15.90:0/637270240 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ab49943d10 0x55ab49947cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:05.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.604+0000 7f73437f1740 1 -- 172.21.15.90:0/637270240 >> 172.21.15.90:0/637270240 conn(0x55ab4990aa50 msgr2=0x55ab4994a840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:46:05.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.604+0000 7f731b400640 0 client.28723 ms_handle_reset on v2:172.21.15.177:6836/4223414858 2024-04-02T14:46:05.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.604+0000 7f73437f1740 1 -- 172.21.15.90:0/637270240 shutdown_connections 2024-04-02T14:46:05.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.604+0000 7f73437f1740 1 -- 172.21.15.90:0/637270240 wait complete. 2024-04-02T14:46:05.606 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-02T14:46:05.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.604+0000 7f73437f1740 20 client.28723 trim_cache size 0 max 16384 2024-04-02T14:46:05.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.605+0000 7f7340c00640 1 -- 172.21.15.90:0/2074023827 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f73200a3e20 msgr2=0x7f73200a62e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:05.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.605+0000 7f7340c00640 1 --2- 172.21.15.90:0/2074023827 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f73200a3e20 0x7f73200a62e0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x55ab49944f40 tx=0x7f732c0f7040 comp rx=0 tx=0).stop 2024-04-02T14:46:05.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.605+0000 7f7340c00640 1 -- 172.21.15.90:0/2074023827 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f733c100300 msgr2=0x7f733c1a05f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:05.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.605+0000 7f7340c00640 1 --2- 172.21.15.90:0/2074023827 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f733c100300 0x7f733c1a05f0 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f732c02c240 tx=0x7f732c004430 comp rx=0 tx=0).stop 2024-04-02T14:46:05.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.605+0000 7f7340c00640 1 -- 172.21.15.90:0/2074023827 shutdown_connections 2024-04-02T14:46:05.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.606+0000 7f7340c00640 1 --2- 172.21.15.90:0/2074023827 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f73200a3e20 0x7f73200a62e0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:05.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.606+0000 7f7340c00640 1 --2- 172.21.15.90:0/2074023827 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f733c19ac40 0x7f733c19b0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:05.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.606+0000 7f7340c00640 1 --2- 172.21.15.90:0/2074023827 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f733c1a0b30 0x7f733c19a700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:05.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.606+0000 7f7340c00640 1 --2- 172.21.15.90:0/2074023827 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f733c100300 0x7f733c1a05f0 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:05.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.606+0000 7f7340c00640 1 -- 172.21.15.90:0/2074023827 >> 172.21.15.90:0/2074023827 conn(0x7f733c076b50 msgr2=0x7f733c101be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:46:05.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.607+0000 7f7340c00640 1 -- 172.21.15.90:0/2074023827 shutdown_connections 2024-04-02T14:46:05.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.607+0000 7f7340c00640 1 -- 172.21.15.90:0/2074023827 wait complete. 2024-04-02T14:46:05.636 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-04-02T14:46:05.787 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.791+0000 7f0ff5c00640 1 -- 172.21.15.90:0/1862567901 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0ff0108cf0 msgr2=0x7f0ff010cd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:05.787 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.791+0000 7f0ff5c00640 1 --2- 172.21.15.90:0/1862567901 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0ff0108cf0 0x7f0ff010cd30 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f0fdc009860 tx=0x7f0fdc02bb70 comp rx=0 tx=0).stop 2024-04-02T14:46:05.787 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.791+0000 7f0ff5c00640 1 -- 172.21.15.90:0/1862567901 shutdown_connections 2024-04-02T14:46:05.788 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.791+0000 7f0ff5c00640 1 --2- 172.21.15.90:0/1862567901 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0ff010d410 0x7f0ff0114fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:05.788 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.791+0000 7f0ff5c00640 1 --2- 172.21.15.90:0/1862567901 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0ff0108cf0 0x7f0ff010cd30 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:05.788 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.791+0000 7f0ff5c00640 1 --2- 172.21.15.90:0/1862567901 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0ff0108340 0x7f0ff0108720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:05.788 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.791+0000 7f0ff5c00640 1 -- 172.21.15.90:0/1862567901 >> 172.21.15.90:0/1862567901 conn(0x7f0ff0076070 msgr2=0x7f0ff0078490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:46:05.788 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.792+0000 7f0ff5c00640 1 -- 172.21.15.90:0/1862567901 shutdown_connections 2024-04-02T14:46:05.788 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.792+0000 7f0ff5c00640 1 -- 172.21.15.90:0/1862567901 wait complete. 2024-04-02T14:46:05.788 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.793+0000 7f0ff5c00640 1 Processor -- start 2024-04-02T14:46:05.788 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.793+0000 7f0ff5c00640 1 -- start start 2024-04-02T14:46:05.791 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.796+0000 7f0ff5c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0ff0108340 0x7f0ff00820a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:05.791 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.796+0000 7f0ff5c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0ff010d410 0x7f0ff00825e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:05.792 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.796+0000 7f0ff5c00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0ff007b230 0x7f0ff007b6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:05.792 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.796+0000 7f0feea00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0ff0108340 0x7f0ff00820a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:05.792 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.796+0000 7f0feea00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0ff0108340 0x7f0ff00820a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:58164/0 (socket says 172.21.15.90:58164) 2024-04-02T14:46:05.792 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.796+0000 7f0feea00640 1 -- 172.21.15.90:0/2983238189 learned_addr learned my addr 172.21.15.90:0/2983238189 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:46:05.792 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.797+0000 7f0fee000640 1 --2- 172.21.15.90:0/2983238189 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0ff010d410 0x7f0ff00825e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:05.792 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.797+0000 7f0fef400640 1 --2- 172.21.15.90:0/2983238189 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0ff007b230 0x7f0ff007b6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:05.792 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.797+0000 7f0ff5c00640 1 -- 172.21.15.90:0/2983238189 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f0ff01184f0 con 0x7f0ff010d410 2024-04-02T14:46:05.792 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.797+0000 7f0ff5c00640 1 -- 172.21.15.90:0/2983238189 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f0ff0118370 con 0x7f0ff007b230 2024-04-02T14:46:05.793 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.797+0000 7f0ff5c00640 1 -- 172.21.15.90:0/2983238189 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f0ff0118670 con 0x7f0ff0108340 2024-04-02T14:46:05.793 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.798+0000 7f0fef400640 1 -- 172.21.15.90:0/2983238189 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0ff0108340 msgr2=0x7f0ff00820a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:05.793 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.798+0000 7f0fef400640 1 --2- 172.21.15.90:0/2983238189 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0ff0108340 0x7f0ff00820a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:05.793 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.798+0000 7f0fef400640 1 -- 172.21.15.90:0/2983238189 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0ff010d410 msgr2=0x7f0ff00825e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:05.793 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.798+0000 7f0fef400640 1 --2- 172.21.15.90:0/2983238189 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0ff010d410 0x7f0ff00825e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:05.794 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.798+0000 7f0fef400640 1 -- 172.21.15.90:0/2983238189 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ff007fbc0 con 0x7f0ff007b230 2024-04-02T14:46:05.794 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.798+0000 7f0fef400640 1 --2- 172.21.15.90:0/2983238189 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0ff007b230 0x7f0ff007b6e0 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f0fe0009fd0 tx=0x7f0fe000eba0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:46:05.795 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.800+0000 7f0fe7400640 1 -- 172.21.15.90:0/2983238189 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fe0004220 con 0x7f0ff007b230 2024-04-02T14:46:05.795 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.800+0000 7f0fe7400640 1 -- 172.21.15.90:0/2983238189 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0fe000cd90 con 0x7f0ff007b230 2024-04-02T14:46:05.795 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.800+0000 7f0fe7400640 1 -- 172.21.15.90:0/2983238189 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fe0004a00 con 0x7f0ff007b230 2024-04-02T14:46:05.795 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.800+0000 7f0ff5c00640 1 -- 172.21.15.90:0/2983238189 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ff007feb0 con 0x7f0ff007b230 2024-04-02T14:46:05.795 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.801+0000 7f0ff5c00640 1 -- 172.21.15.90:0/2983238189 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ff0080380 con 0x7f0ff007b230 2024-04-02T14:46:05.807 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.812+0000 7f0ff8998740 1 Processor -- start 2024-04-02T14:46:05.810 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.815+0000 7f0fe7400640 1 -- 172.21.15.90:0/2983238189 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f0fe0004ba0 con 0x7f0ff007b230 2024-04-02T14:46:05.810 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.815+0000 7f0ff8998740 1 -- start start 2024-04-02T14:46:05.812 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.817+0000 7f0fe7400640 1 --2- 172.21.15.90:0/2983238189 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f0fd40a3f70 0x7f0fd40a6430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:05.812 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.818+0000 7f0fe7400640 1 -- 172.21.15.90:0/2983238189 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 8550+0+0 (secure 0 0 0) 0x7f0fe00d0a10 con 0x7f0ff007b230 2024-04-02T14:46:05.813 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.818+0000 7f0fee000640 1 --2- 172.21.15.90:0/2983238189 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f0fd40a3f70 0x7f0fd40a6430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:05.813 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.818+0000 7f0ff8998740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56218645eeb0 0x56218645f290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:05.813 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.818+0000 7f0ff8998740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56218645f7d0 0x562186467e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:05.813 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.818+0000 7f0ff8998740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5621864683b0 0x56218646a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:05.813 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.818+0000 7f0ff8998740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5621863c3d70 con 0x56218645eeb0 2024-04-02T14:46:05.814 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.818+0000 7f0ff8998740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x56218636c2a0 con 0x5621864683b0 2024-04-02T14:46:05.814 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.818+0000 7f0ff8998740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5621863bfd20 con 0x56218645f7d0 2024-04-02T14:46:05.817 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.819+0000 7f0fef400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5621864683b0 0x56218646a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:05.817 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.819+0000 7f0feea00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56218645eeb0 0x56218645f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:05.818 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.819+0000 7f0fee000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56218645f7d0 0x562186467e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:05.818 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.822+0000 7f0fef400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5621864683b0 0x56218646a7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:52450/0 (socket says 172.21.15.90:52450) 2024-04-02T14:46:05.818 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.822+0000 7f0fef400640 1 -- 172.21.15.90:0/949826172 learned_addr learned my addr 172.21.15.90:0/949826172 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:46:05.818 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.823+0000 7f0fee000640 1 --2- 172.21.15.90:0/2983238189 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f0fd40a3f70 0x7f0fd40a6430 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f0fdc009830 tx=0x7f0fdc02c090 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:46:05.819 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.823+0000 7f0feea00640 1 -- 172.21.15.90:0/949826172 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56218645f7d0 msgr2=0x562186467e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:46:05.819 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.823+0000 7f0feea00640 1 --2- 172.21.15.90:0/949826172 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56218645f7d0 0x562186467e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:05.819 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.823+0000 7f0feea00640 1 -- 172.21.15.90:0/949826172 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5621864683b0 msgr2=0x56218646a7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:46:05.819 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.823+0000 7f0feea00640 1 --2- 172.21.15.90:0/949826172 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5621864683b0 0x56218646a7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:05.819 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.823+0000 7f0feea00640 1 -- 172.21.15.90:0/949826172 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562186380840 con 0x56218645eeb0 2024-04-02T14:46:05.820 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.825+0000 7f0feea00640 1 --2- 172.21.15.90:0/949826172 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56218645eeb0 0x56218645f290 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f0fe8007f00 tx=0x7f0fe800f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee861c2c13326f85 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:46:05.821 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.826+0000 7f0fdbe00640 1 -- 172.21.15.90:0/949826172 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fe8010840 con 0x56218645eeb0 2024-04-02T14:46:05.821 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.826+0000 7f0fdbe00640 1 -- 172.21.15.90:0/949826172 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0fe80183f0 con 0x56218645eeb0 2024-04-02T14:46:05.821 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.826+0000 7f0fdbe00640 1 -- 172.21.15.90:0/949826172 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fe8010840 con 0x56218645eeb0 2024-04-02T14:46:05.822 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.827+0000 7f0ff8998740 1 -- 172.21.15.90:0/949826172 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56218645eeb0 msgr2=0x56218645f290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:05.822 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.827+0000 7f0ff8998740 1 --2- 172.21.15.90:0/949826172 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56218645eeb0 0x56218645f290 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f0fe8007f00 tx=0x7f0fe800f790 comp rx=0 tx=0).stop 2024-04-02T14:46:05.822 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.827+0000 7f0ff8998740 1 -- 172.21.15.90:0/949826172 shutdown_connections 2024-04-02T14:46:05.823 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.827+0000 7f0ff8998740 1 --2- 172.21.15.90:0/949826172 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5621864683b0 0x56218646a7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:05.823 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.827+0000 7f0ff8998740 1 --2- 172.21.15.90:0/949826172 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56218645f7d0 0x562186467e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:05.823 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.827+0000 7f0ff8998740 1 --2- 172.21.15.90:0/949826172 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56218645eeb0 0x56218645f290 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:05.823 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.827+0000 7f0ff8998740 1 -- 172.21.15.90:0/949826172 >> 172.21.15.90:0/949826172 conn(0x5621864595e0 msgr2=0x5621864599c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:46:05.825 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.831+0000 7f0ff8998740 1 -- 172.21.15.90:0/949826172 shutdown_connections 2024-04-02T14:46:05.827 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.832+0000 7f0ff8998740 1 -- 172.21.15.90:0/949826172 wait complete. 2024-04-02T14:46:05.828 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.833+0000 7f0ff8998740 1 -- start start 2024-04-02T14:46:05.828 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.833+0000 7f0ff8998740 1 Processor -- start 2024-04-02T14:46:05.829 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.834+0000 7f0ff8998740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56218645eeb0 0x5621864653a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:05.829 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.834+0000 7f0ff8998740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56218645f7d0 0x5621864658e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:05.829 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.834+0000 7f0ff8998740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5621864683b0 0x562186465e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:05.830 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.834+0000 7f0fee000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56218645f7d0 0x5621864658e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:05.830 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.834+0000 7f0fee000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56218645f7d0 0x5621864658e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:52466/0 (socket says 172.21.15.90:52466) 2024-04-02T14:46:05.830 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.834+0000 7f0ff8998740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x56218646b1a0 con 0x56218645eeb0 2024-04-02T14:46:05.830 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.834+0000 7f0fee000640 1 -- 172.21.15.90:0/980367266 learned_addr learned my addr 172.21.15.90:0/980367266 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:46:05.830 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.834+0000 7f0ff8998740 1 -- 172.21.15.90:0/980367266 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x56218637f660 con 0x56218645f7d0 2024-04-02T14:46:05.830 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.834+0000 7f0ff8998740 1 -- 172.21.15.90:0/980367266 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x56218646b320 con 0x5621864683b0 2024-04-02T14:46:05.830 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.834+0000 7f0ff8998740 10 client.0 _ll_register_callbacks cb 0x56218637f050 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:46:05.830 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.835+0000 7f0fef400640 1 --2- 172.21.15.90:0/980367266 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5621864683b0 0x562186465e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:05.830 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.835+0000 7f0feea00640 1 --2- 172.21.15.90:0/980367266 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56218645eeb0 0x5621864653a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:05.831 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.835+0000 7f0fee000640 1 -- 172.21.15.90:0/980367266 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5621864683b0 msgr2=0x562186465e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:46:05.831 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.835+0000 7f0fee000640 1 --2- 172.21.15.90:0/980367266 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5621864683b0 0x562186465e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:05.831 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.835+0000 7f0fee000640 1 -- 172.21.15.90:0/980367266 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56218645eeb0 msgr2=0x5621864653a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:46:05.831 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.835+0000 7f0fee000640 1 --2- 172.21.15.90:0/980367266 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56218645eeb0 0x5621864653a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:05.831 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.835+0000 7f0fee000640 1 -- 172.21.15.90:0/980367266 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56218638ee80 con 0x56218645f7d0 2024-04-02T14:46:05.834 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.836+0000 7f0fee000640 1 --2- 172.21.15.90:0/980367266 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56218645f7d0 0x5621864658e0 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f0ff006b1e0 tx=0x7f0fdc02bb70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2ec633fff1e7bb47 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:46:05.834 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.836+0000 7f0fd8c00640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:46:05.834 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.837+0000 7f0fd8c00640 1 -- 172.21.15.90:0/980367266 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fdc04f070 con 0x56218645f7d0 2024-04-02T14:46:05.834 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.837+0000 7f0fd8c00640 1 -- 172.21.15.90:0/980367266 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8550+0+0 (secure 0 0 0) 0x7f0fdc02c710 con 0x56218645f7d0 2024-04-02T14:46:05.834 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.837+0000 7f0ff8998740 1 -- 172.21.15.90:0/980367266 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0fe80183f0 con 0x56218645f7d0 2024-04-02T14:46:05.835 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.839+0000 7f0fd8c00640 1 -- 172.21.15.90:0/980367266 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f0fdc05b050 con 0x56218645f7d0 2024-04-02T14:46:05.835 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.839+0000 7f0ff8998740 10 client.28916 fetch_fsmap learned FSMap version 27 2024-04-02T14:46:05.835 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.839+0000 7f0ff8998740 1 -- 172.21.15.90:0/980367266 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5621860d8210 con 0x56218645f7d0 2024-04-02T14:46:05.835 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.840+0000 7f0fd8c00640 1 -- 172.21.15.90:0/980367266 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0fe80183f0 con 0x56218645f7d0 2024-04-02T14:46:05.835 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.840+0000 7f0fd8c00640 1 -- 172.21.15.90:0/980367266 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f0fdc032b80 con 0x56218645f7d0 2024-04-02T14:46:05.836 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.841+0000 7f0ff8998740 10 client.28916 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:46:05.836 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.841+0000 7f0ff8998740 10 client.28916 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-02T14:46:05.836 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.841+0000 7f0ff8998740 20 client.28916 populate_metadata read hostname 'smithi090' 2024-04-02T14:46:05.836 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.841+0000 7f0ff8998740 1 --2- 172.21.15.90:0/980367266 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x562186478690 0x56218647aa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:05.836 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.841+0000 7f0ff8998740 4 client.28916 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:46:05.836 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.841+0000 7f0ff8998740 1 -- 172.21.15.90:0/980367266 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562186373cd0 con 0x562186478690 2024-04-02T14:46:05.837 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.842+0000 7f0feea00640 1 --2- 172.21.15.90:0/980367266 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x562186478690 0x56218647aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:05.837 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.842+0000 7f0feea00640 1 --2- 172.21.15.90:0/980367266 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x562186478690 0x56218647aa80 crc :-1 s=READY pgs=292 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=65fb637f9e96b235 server_cookie=9431badb4900b981 in_seq=0 out_seq=0 2024-04-02T14:46:05.839 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.842+0000 7f0fd8c00640 10 client.28916 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:46:05.839 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.844+0000 7f0fd8c00640 1 -- 172.21.15.90:0/980367266 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x562186373cd0 con 0x562186478690 2024-04-02T14:46:05.839 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.844+0000 7f0fd8c00640 10 client.28916 handle_command_reply: tid=0 2024-04-02T14:46:05.839 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.844+0000 7f0ff8998740 1 client.28916 shutdown 2024-04-02T14:46:05.839 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.844+0000 7f0ff8998740 1 -- 172.21.15.90:0/980367266 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56218645f7d0 msgr2=0x5621864658e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:05.840 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.844+0000 7f0ff8998740 1 --2- 172.21.15.90:0/980367266 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56218645f7d0 0x5621864658e0 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f0ff006b1e0 tx=0x7f0fdc02bb70 comp rx=0 tx=0).stop 2024-04-02T14:46:05.840 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.844+0000 7f0ff8998740 1 -- 172.21.15.90:0/980367266 shutdown_connections 2024-04-02T14:46:05.840 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.844+0000 7f0ff8998740 1 --2- 172.21.15.90:0/980367266 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x562186478690 0x56218647aa80 crc :-1 s=READY pgs=292 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:05.840 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.844+0000 7f0ff8998740 1 --2- 172.21.15.90:0/980367266 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5621864683b0 0x562186465e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:05.840 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.844+0000 7f0ff8998740 1 --2- 172.21.15.90:0/980367266 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56218645f7d0 0x5621864658e0 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:05.840 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.844+0000 7f0ff8998740 1 --2- 172.21.15.90:0/980367266 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56218645eeb0 0x5621864653a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:05.840 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.844+0000 7f0fd8c00640 0 client.28916 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:46:05.840 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.844+0000 7f0ff8998740 1 -- 172.21.15.90:0/980367266 >> 172.21.15.90:0/980367266 conn(0x5621864595e0 msgr2=0x56218645b330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:46:05.841 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.845+0000 7f0ff8998740 1 -- 172.21.15.90:0/980367266 shutdown_connections 2024-04-02T14:46:05.841 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.845+0000 7f0ff8998740 1 -- 172.21.15.90:0/980367266 wait complete. 2024-04-02T14:46:05.841 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.845+0000 7f0ff8998740 20 client.28916 trim_cache size 0 max 16384 2024-04-02T14:46:05.870 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.875+0000 7f0ff8998740 1 Processor -- start 2024-04-02T14:46:05.870 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.876+0000 7f0ff8998740 1 -- start start 2024-04-02T14:46:05.871 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.876+0000 7f0ff8998740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56218645eeb0 0x562186474060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:05.871 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.876+0000 7f0ff8998740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56218645f7d0 0x562186480390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:05.871 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.876+0000 7f0ff8998740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5621864683b0 0x5621864808d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:05.872 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.876+0000 7f0ff8998740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f0ff01bc760 con 0x56218645eeb0 2024-04-02T14:46:05.872 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.876+0000 7f0ff8998740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f0ff01be2a0 con 0x56218645f7d0 2024-04-02T14:46:05.872 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.876+0000 7f0ff8998740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f0ff01bce30 con 0x5621864683b0 2024-04-02T14:46:05.872 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.877+0000 7f0fef400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5621864683b0 0x5621864808d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:05.873 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.877+0000 7f0fef400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5621864683b0 0x5621864808d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:58192/0 (socket says 172.21.15.90:58192) 2024-04-02T14:46:05.873 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.877+0000 7f0fef400640 1 -- 172.21.15.90:0/2399527875 learned_addr learned my addr 172.21.15.90:0/2399527875 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:46:05.873 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.877+0000 7f0fee000640 1 --2- 172.21.15.90:0/2399527875 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56218645f7d0 0x562186480390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:05.873 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.877+0000 7f0feea00640 1 --2- 172.21.15.90:0/2399527875 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56218645eeb0 0x562186474060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:05.873 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.877+0000 7f0fee000640 1 -- 172.21.15.90:0/2399527875 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5621864683b0 msgr2=0x5621864808d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:46:05.873 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.877+0000 7f0fee000640 1 --2- 172.21.15.90:0/2399527875 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5621864683b0 0x5621864808d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:05.873 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.877+0000 7f0fee000640 1 -- 172.21.15.90:0/2399527875 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56218645eeb0 msgr2=0x562186474060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:46:05.873 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.877+0000 7f0fee000640 1 --2- 172.21.15.90:0/2399527875 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56218645eeb0 0x562186474060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:05.873 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.877+0000 7f0fee000640 1 -- 172.21.15.90:0/2399527875 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5621861a4620 con 0x56218645f7d0 2024-04-02T14:46:05.873 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.877+0000 7f0feea00640 1 --2- 172.21.15.90:0/2399527875 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56218645eeb0 0x562186474060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:46:05.873 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.878+0000 7f0fef400640 1 --2- 172.21.15.90:0/2399527875 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5621864683b0 0x5621864808d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:46:05.874 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.879+0000 7f0fee000640 1 --2- 172.21.15.90:0/2399527875 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56218645f7d0 0x562186480390 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f0fdc0095f0 tx=0x7f0fdc0056d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c6c7192dd772ec09 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:46:05.875 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.880+0000 7f0fdb400640 1 -- 172.21.15.90:0/2399527875 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fdc04f040 con 0x56218645f7d0 2024-04-02T14:46:05.875 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.880+0000 7f0fdb400640 1 -- 172.21.15.90:0/2399527875 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0fdc02c9f0 con 0x56218645f7d0 2024-04-02T14:46:05.875 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.880+0000 7f0fdb400640 1 -- 172.21.15.90:0/2399527875 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fdc039040 con 0x56218645f7d0 2024-04-02T14:46:05.876 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.881+0000 7f0ff8998740 1 -- 172.21.15.90:0/2399527875 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56218645f7d0 msgr2=0x562186480390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:05.876 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.881+0000 7f0ff8998740 1 --2- 172.21.15.90:0/2399527875 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56218645f7d0 0x562186480390 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f0fdc0095f0 tx=0x7f0fdc0056d0 comp rx=0 tx=0).stop 2024-04-02T14:46:05.878 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.882+0000 7f0ff8998740 1 -- 172.21.15.90:0/2399527875 shutdown_connections 2024-04-02T14:46:05.878 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.882+0000 7f0ff8998740 1 --2- 172.21.15.90:0/2399527875 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5621864683b0 0x5621864808d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:05.878 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.882+0000 7f0ff8998740 1 --2- 172.21.15.90:0/2399527875 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56218645f7d0 0x562186480390 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:05.878 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.882+0000 7f0ff8998740 1 --2- 172.21.15.90:0/2399527875 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56218645eeb0 0x562186474060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:05.878 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.882+0000 7f0ff8998740 1 -- 172.21.15.90:0/2399527875 >> 172.21.15.90:0/2399527875 conn(0x5621864595e0 msgr2=0x562186460730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:46:05.879 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.882+0000 7f0ff8998740 1 -- 172.21.15.90:0/2399527875 shutdown_connections 2024-04-02T14:46:05.879 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.882+0000 7f0ff8998740 1 -- 172.21.15.90:0/2399527875 wait complete. 2024-04-02T14:46:05.879 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.883+0000 7f0ff8998740 1 -- start start 2024-04-02T14:46:05.880 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.883+0000 7f0ff8998740 1 Processor -- start 2024-04-02T14:46:05.882 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.883+0000 7f0ff8998740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56218645eeb0 0x562186462030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:05.882 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.883+0000 7f0ff8998740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56218645f7d0 0x562186462570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:05.882 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.883+0000 7f0ff8998740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5621864683b0 0x562186462ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:05.882 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.884+0000 7f0ff8998740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f0ff01bdbd0 con 0x56218645f7d0 2024-04-02T14:46:05.883 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.884+0000 7f0ff8998740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f0ff01bd500 con 0x56218645eeb0 2024-04-02T14:46:05.883 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.884+0000 7f0ff8998740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f0ff01ba760 con 0x5621864683b0 2024-04-02T14:46:05.883 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.884+0000 7f0ff8998740 10 client.0 _ll_register_callbacks cb 0x7f0fdc0096b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:46:05.883 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.884+0000 7f0fef400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5621864683b0 0x562186462ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:05.883 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.884+0000 7f0fef400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5621864683b0 0x562186462ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:58200/0 (socket says 172.21.15.90:58200) 2024-04-02T14:46:05.883 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.884+0000 7f0fef400640 1 -- 172.21.15.90:0/1656265200 learned_addr learned my addr 172.21.15.90:0/1656265200 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:46:05.883 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.884+0000 7f0feea00640 1 --2- 172.21.15.90:0/1656265200 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56218645eeb0 0x562186462030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:05.883 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.885+0000 7f0fee000640 1 --2- 172.21.15.90:0/1656265200 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56218645f7d0 0x562186462570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:05.883 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.885+0000 7f0fef400640 1 -- 172.21.15.90:0/1656265200 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56218645eeb0 msgr2=0x562186462030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:46:05.883 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.885+0000 7f0fef400640 1 --2- 172.21.15.90:0/1656265200 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56218645eeb0 0x562186462030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:05.883 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.885+0000 7f0fef400640 1 -- 172.21.15.90:0/1656265200 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56218645f7d0 msgr2=0x562186462570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:46:05.883 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.885+0000 7f0fef400640 1 --2- 172.21.15.90:0/1656265200 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56218645f7d0 0x562186462570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:05.884 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.885+0000 7f0fef400640 1 -- 172.21.15.90:0/1656265200 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5621864ddc00 con 0x5621864683b0 2024-04-02T14:46:05.884 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.885+0000 7f0fee000640 1 --2- 172.21.15.90:0/1656265200 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56218645f7d0 0x562186462570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:46:05.884 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.885+0000 7f0fef400640 1 --2- 172.21.15.90:0/1656265200 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5621864683b0 0x562186462ab0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f0fe009c920 tx=0x7f0fe0004f10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e741b54d78c8d040 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:46:05.884 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.886+0000 7f0fd8c00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:46:05.884 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.886+0000 7f0fd8c00640 1 -- 172.21.15.90:0/1656265200 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fe00aa060 con 0x5621864683b0 2024-04-02T14:46:05.884 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.886+0000 7f0fd8c00640 1 -- 172.21.15.90:0/1656265200 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8550+0+0 (secure 0 0 0) 0x7f0fe0099000 con 0x5621864683b0 2024-04-02T14:46:05.884 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.886+0000 7f0ff8998740 1 -- 172.21.15.90:0/1656265200 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0fdc02c9f0 con 0x5621864683b0 2024-04-02T14:46:05.884 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.886+0000 7f0fd8c00640 1 -- 172.21.15.90:0/1656265200 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f0fe00b8070 con 0x5621864683b0 2024-04-02T14:46:05.884 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.886+0000 7f0ff8998740 10 client.28741 fetch_fsmap learned FSMap version 27 2024-04-02T14:46:05.884 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.886+0000 7f0ff8998740 1 -- 172.21.15.90:0/1656265200 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5621864ded80 con 0x5621864683b0 2024-04-02T14:46:05.885 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.887+0000 7f0fd8c00640 1 -- 172.21.15.90:0/1656265200 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0fdc02c9f0 con 0x5621864683b0 2024-04-02T14:46:05.885 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.888+0000 7f0fd8c00640 1 -- 172.21.15.90:0/1656265200 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f0fe001f070 con 0x5621864683b0 2024-04-02T14:46:05.885 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.889+0000 7f0ff8998740 10 client.28741 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:46:05.885 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.889+0000 7f0ff8998740 10 client.28741 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-02T14:46:05.885 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.889+0000 7f0ff8998740 20 client.28741 populate_metadata read hostname 'smithi090' 2024-04-02T14:46:05.885 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.889+0000 7f0ff8998740 1 --2- 172.21.15.90:0/1656265200 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5621864df3e0 0x5621864e1800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:05.886 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.889+0000 7f0ff8998740 4 client.28741 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-02T14:46:05.886 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.889+0000 7f0ff8998740 1 -- 172.21.15.90:0/1656265200 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5621863d12b0 con 0x5621864df3e0 2024-04-02T14:46:05.886 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.890+0000 7f0feea00640 1 --2- 172.21.15.90:0/1656265200 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5621864df3e0 0x5621864e1800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:05.886 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.891+0000 7f0feea00640 1 --2- 172.21.15.90:0/1656265200 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5621864df3e0 0x5621864e1800 crc :-1 s=READY pgs=293 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2282e56bf53cfa45 server_cookie=2d182e64e8bbc9c0 in_seq=0 out_seq=0 2024-04-02T14:46:05.886 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.891+0000 7f0fd8c00640 10 client.28741 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:46:05.908 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.913+0000 7f0fd8c00640 1 -- 172.21.15.90:0/1656265200 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5621863d12b0 con 0x5621864df3e0 2024-04-02T14:46:05.909 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.913+0000 7f0fd8c00640 10 client.28741 handle_command_reply: tid=0 2024-04-02T14:46:05.910 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.914+0000 7f0ff8998740 1 client.28741 shutdown 2024-04-02T14:46:05.910 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.914+0000 7f0ff8998740 1 -- 172.21.15.90:0/1656265200 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5621864683b0 msgr2=0x562186462ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:05.910 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.914+0000 7f0ff8998740 1 --2- 172.21.15.90:0/1656265200 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5621864683b0 0x562186462ab0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f0fe009c920 tx=0x7f0fe0004f10 comp rx=0 tx=0).stop 2024-04-02T14:46:05.910 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.914+0000 7f0ff8998740 1 -- 172.21.15.90:0/1656265200 shutdown_connections 2024-04-02T14:46:05.910 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.914+0000 7f0ff8998740 1 --2- 172.21.15.90:0/1656265200 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5621864df3e0 0x5621864e1800 crc :-1 s=READY pgs=293 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:05.910 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.914+0000 7f0ff8998740 1 --2- 172.21.15.90:0/1656265200 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5621864683b0 0x562186462ab0 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:05.910 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.914+0000 7f0ff8998740 1 --2- 172.21.15.90:0/1656265200 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56218645f7d0 0x562186462570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:05.910 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.914+0000 7f0ff8998740 1 --2- 172.21.15.90:0/1656265200 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56218645eeb0 0x562186462030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:05.910 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.914+0000 7f0ff8998740 1 -- 172.21.15.90:0/1656265200 >> 172.21.15.90:0/1656265200 conn(0x5621864595e0 msgr2=0x562186466a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:46:05.910 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.914+0000 7f0fd8c00640 0 client.28741 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:46:05.910 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.915+0000 7f0ff8998740 1 -- 172.21.15.90:0/1656265200 shutdown_connections 2024-04-02T14:46:05.910 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.915+0000 7f0ff8998740 1 -- 172.21.15.90:0/1656265200 wait complete. 2024-04-02T14:46:05.911 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.915+0000 7f0ff8998740 20 client.28741 trim_cache size 0 max 16384 2024-04-02T14:46:05.911 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:{ 2024-04-02T14:46:05.911 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "message": "", 2024-04-02T14:46:05.911 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "return_code": 0 2024-04-02T14:46:05.911 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:} 2024-04-02T14:46:05.912 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.916+0000 7f0ff5c00640 1 -- 172.21.15.90:0/2983238189 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f0fd40a3f70 msgr2=0x7f0fd40a6430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:05.912 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.916+0000 7f0ff5c00640 1 --2- 172.21.15.90:0/2983238189 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f0fd40a3f70 0x7f0fd40a6430 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f0fdc009830 tx=0x7f0fdc02c090 comp rx=0 tx=0).stop 2024-04-02T14:46:05.912 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.916+0000 7f0ff5c00640 1 -- 172.21.15.90:0/2983238189 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0ff007b230 msgr2=0x7f0ff007b6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:05.912 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.916+0000 7f0ff5c00640 1 --2- 172.21.15.90:0/2983238189 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0ff007b230 0x7f0ff007b6e0 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f0fe0009fd0 tx=0x7f0fe000eba0 comp rx=0 tx=0).stop 2024-04-02T14:46:05.912 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.916+0000 7f0ff5c00640 1 -- 172.21.15.90:0/2983238189 shutdown_connections 2024-04-02T14:46:05.912 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.916+0000 7f0ff5c00640 1 --2- 172.21.15.90:0/2983238189 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f0fd40a3f70 0x7f0fd40a6430 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:05.912 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.916+0000 7f0ff5c00640 1 --2- 172.21.15.90:0/2983238189 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0ff007b230 0x7f0ff007b6e0 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:05.912 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.916+0000 7f0ff5c00640 1 --2- 172.21.15.90:0/2983238189 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0ff010d410 0x7f0ff00825e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:05.912 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.916+0000 7f0ff5c00640 1 --2- 172.21.15.90:0/2983238189 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0ff0108340 0x7f0ff00820a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:05.912 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.916+0000 7f0ff5c00640 1 -- 172.21.15.90:0/2983238189 >> 172.21.15.90:0/2983238189 conn(0x7f0ff0076070 msgr2=0x7f0ff0078350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:46:05.913 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.916+0000 7f0ff5c00640 1 -- 172.21.15.90:0/2983238189 shutdown_connections 2024-04-02T14:46:05.913 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:05.916+0000 7f0ff5c00640 1 -- 172.21.15.90:0/2983238189 wait complete. 2024-04-02T14:46:05.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.932+0000 7fb7c8200640 1 -- 172.21.15.90:0/343807778 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb7c0107aa0 msgr2=0x7fb7c0107e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:05.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.932+0000 7fb7c8200640 1 --2- 172.21.15.90:0/343807778 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb7c0107aa0 0x7fb7c0107e80 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7fb7b4009930 tx=0x7fb7b402bc20 comp rx=0 tx=0).stop 2024-04-02T14:46:05.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.932+0000 7fb7c8200640 1 -- 172.21.15.90:0/343807778 shutdown_connections 2024-04-02T14:46:05.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.932+0000 7fb7c8200640 1 --2- 172.21.15.90:0/343807778 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb7c0102450 0x7fb7c010e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:05.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.932+0000 7fb7c8200640 1 --2- 172.21.15.90:0/343807778 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb7c0101a90 0x7fb7c0101f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:05.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.932+0000 7fb7c8200640 1 --2- 172.21.15.90:0/343807778 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb7c0107aa0 0x7fb7c0107e80 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:05.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.932+0000 7fb7c8200640 1 -- 172.21.15.90:0/343807778 >> 172.21.15.90:0/343807778 conn(0x7fb7c0078f00 msgr2=0x7fb7c0079310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:46:05.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.932+0000 7fb7c8200640 1 -- 172.21.15.90:0/343807778 shutdown_connections 2024-04-02T14:46:05.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.933+0000 7fb7c8200640 1 -- 172.21.15.90:0/343807778 wait complete. 2024-04-02T14:46:05.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.935+0000 7fb7c8200640 1 Processor -- start 2024-04-02T14:46:05.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.935+0000 7fb7c8200640 1 -- start start 2024-04-02T14:46:05.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.935+0000 7fb7c8200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb7c0101a90 0x7fb7c0198360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:05.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.935+0000 7fb7c8200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb7c0102450 0x7fb7c01988a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:05.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.935+0000 7fb7c8200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb7c0107aa0 0x7fb7c019cc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:05.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.936+0000 7fb7c5800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb7c0101a90 0x7fb7c0198360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:05.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.936+0000 7fb7c8200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fb7c0111f00 con 0x7fb7c0107aa0 2024-04-02T14:46:05.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.936+0000 7fb7c8200640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fb7c0111d80 con 0x7fb7c0102450 2024-04-02T14:46:05.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.936+0000 7fb7c8200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fb7c0112080 con 0x7fb7c0101a90 2024-04-02T14:46:05.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.936+0000 7fb7c5800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb7c0101a90 0x7fb7c0198360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:58226/0 (socket says 172.21.15.90:58226) 2024-04-02T14:46:05.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.936+0000 7fb7c5800640 1 -- 172.21.15.90:0/2390671442 learned_addr learned my addr 172.21.15.90:0/2390671442 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:46:05.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.936+0000 7fb7c6200640 1 --2- 172.21.15.90:0/2390671442 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb7c0107aa0 0x7fb7c019cc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:05.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.936+0000 7fb7c4e00640 1 --2- 172.21.15.90:0/2390671442 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb7c0102450 0x7fb7c01988a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:05.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.936+0000 7fb7c5800640 1 -- 172.21.15.90:0/2390671442 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb7c0102450 msgr2=0x7fb7c01988a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:05.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.936+0000 7fb7c5800640 1 --2- 172.21.15.90:0/2390671442 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb7c0102450 0x7fb7c01988a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:05.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.936+0000 7fb7c5800640 1 -- 172.21.15.90:0/2390671442 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb7c0107aa0 msgr2=0x7fb7c019cc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:05.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.936+0000 7fb7c5800640 1 --2- 172.21.15.90:0/2390671442 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb7c0107aa0 0x7fb7c019cc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:05.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.936+0000 7fb7c5800640 1 -- 172.21.15.90:0/2390671442 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7c019d350 con 0x7fb7c0101a90 2024-04-02T14:46:05.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.936+0000 7fb7c6200640 1 --2- 172.21.15.90:0/2390671442 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb7c0107aa0 0x7fb7c019cc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:46:05.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.937+0000 7fb7c5800640 1 --2- 172.21.15.90:0/2390671442 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb7c0101a90 0x7fb7c0198360 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7fb7b4009900 tx=0x7fb7b40045e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:46:05.933 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:46:05 smithi090 ceph-mon[31674]: pgmap v347: 97 pgs: 97 active+clean; 4.7 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 10 MiB/s wr, 1.10k op/s 2024-04-02T14:46:05.933 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:46:05 smithi090 ceph-mon[31674]: from='client.? 172.21.15.90:0/1476862397' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-02T14:46:05.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.937+0000 7fb7be000640 1 -- 172.21.15.90:0/2390671442 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7b4043020 con 0x7fb7c0101a90 2024-04-02T14:46:05.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.937+0000 7fb7be000640 1 -- 172.21.15.90:0/2390671442 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb7b40054e0 con 0x7fb7c0101a90 2024-04-02T14:46:05.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.937+0000 7fb7be000640 1 -- 172.21.15.90:0/2390671442 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7b4003d00 con 0x7fb7c0101a90 2024-04-02T14:46:05.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.938+0000 7fb7c8200640 1 -- 172.21.15.90:0/2390671442 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7c019d5e0 con 0x7fb7c0101a90 2024-04-02T14:46:05.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.938+0000 7fb7c8200640 1 -- 172.21.15.90:0/2390671442 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7c01a4ed0 con 0x7fb7c0101a90 2024-04-02T14:46:05.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.944+0000 7fb7cafa2740 1 Processor -- start 2024-04-02T14:46:05.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.945+0000 7fb7be000640 1 -- 172.21.15.90:0/2390671442 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fb7b4046020 con 0x7fb7c0101a90 2024-04-02T14:46:05.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.945+0000 7fb7cafa2740 1 -- start start 2024-04-02T14:46:05.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.946+0000 7fb7cafa2740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dcdd04f440 0x55dcdd04f820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:05.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.946+0000 7fb7cafa2740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55dcdd03fcb0 0x55dcdd040130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:05.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.946+0000 7fb7cafa2740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dcdd040670 0x55dcdd0570b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:05.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.946+0000 7fb7cafa2740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55dcdcf83400 con 0x55dcdd03fcb0 2024-04-02T14:46:05.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.946+0000 7fb7cafa2740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55dcdcf67f10 con 0x55dcdd040670 2024-04-02T14:46:05.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.946+0000 7fb7cafa2740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55dcdcfa76f0 con 0x55dcdd04f440 2024-04-02T14:46:05.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.946+0000 7fb7c6200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dcdd04f440 0x55dcdd04f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:05.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.946+0000 7fb7c6200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dcdd04f440 0x55dcdd04f820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:58236/0 (socket says 172.21.15.90:58236) 2024-04-02T14:46:05.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.946+0000 7fb7c6200640 1 -- 172.21.15.90:0/2696417447 learned_addr learned my addr 172.21.15.90:0/2696417447 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:46:05.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.946+0000 7fb7c4e00640 1 --2- 172.21.15.90:0/2696417447 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dcdd040670 0x55dcdd0570b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:05.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.946+0000 7fb7c5800640 1 --2- 172.21.15.90:0/2696417447 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55dcdd03fcb0 0x55dcdd040130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:05.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.947+0000 7fb7c4e00640 1 -- 172.21.15.90:0/2696417447 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dcdd04f440 msgr2=0x55dcdd04f820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:46:05.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.947+0000 7fb7c4e00640 1 --2- 172.21.15.90:0/2696417447 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dcdd04f440 0x55dcdd04f820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:05.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.947+0000 7fb7c4e00640 1 -- 172.21.15.90:0/2696417447 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55dcdd03fcb0 msgr2=0x55dcdd040130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:46:05.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.947+0000 7fb7c4e00640 1 --2- 172.21.15.90:0/2696417447 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55dcdd03fcb0 0x55dcdd040130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:05.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.947+0000 7fb7c4e00640 1 -- 172.21.15.90:0/2696417447 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dcdccbb3f0 con 0x55dcdd040670 2024-04-02T14:46:05.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.947+0000 7fb7c4e00640 1 --2- 172.21.15.90:0/2696417447 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dcdd040670 0x55dcdd0570b0 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7fb7ac00abc0 tx=0x7fb7ac00f790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6cd1c14cca8dbd7a server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:46:05.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.947+0000 7fb7be000640 1 --2- 172.21.15.90:0/2390671442 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fb7a80a3ef0 0x7fb7a80a63b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:05.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.947+0000 7fb7b2a00640 1 -- 172.21.15.90:0/2696417447 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7ac005220 con 0x55dcdd040670 2024-04-02T14:46:05.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.947+0000 7fb7be000640 1 -- 172.21.15.90:0/2390671442 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 8550+0+0 (secure 0 0 0) 0x7fb7b40f2200 con 0x7fb7c0101a90 2024-04-02T14:46:05.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.948+0000 7fb7c6200640 1 --2- 172.21.15.90:0/2390671442 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fb7a80a3ef0 0x7fb7a80a63b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:05.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.948+0000 7fb7b2a00640 1 -- 172.21.15.90:0/2696417447 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb7ac004c80 con 0x55dcdd040670 2024-04-02T14:46:05.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.948+0000 7fb7b2a00640 1 -- 172.21.15.90:0/2696417447 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7ac0188d0 con 0x55dcdd040670 2024-04-02T14:46:05.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.949+0000 7fb7c6200640 1 --2- 172.21.15.90:0/2390671442 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fb7a80a3ef0 0x7fb7a80a63b0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fb7b8009fd0 tx=0x7fb7b800d040 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:46:05.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.949+0000 7fb7cafa2740 1 -- 172.21.15.90:0/2696417447 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dcdd040670 msgr2=0x55dcdd0570b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:05.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.949+0000 7fb7cafa2740 1 --2- 172.21.15.90:0/2696417447 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dcdd040670 0x55dcdd0570b0 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7fb7ac00abc0 tx=0x7fb7ac00f790 comp rx=0 tx=0).stop 2024-04-02T14:46:05.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.949+0000 7fb7cafa2740 1 -- 172.21.15.90:0/2696417447 shutdown_connections 2024-04-02T14:46:05.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.949+0000 7fb7cafa2740 1 --2- 172.21.15.90:0/2696417447 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dcdd040670 0x55dcdd0570b0 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:05.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.949+0000 7fb7cafa2740 1 --2- 172.21.15.90:0/2696417447 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55dcdd03fcb0 0x55dcdd040130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:05.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.949+0000 7fb7cafa2740 1 --2- 172.21.15.90:0/2696417447 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dcdd04f440 0x55dcdd04f820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:05.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.949+0000 7fb7cafa2740 1 -- 172.21.15.90:0/2696417447 >> 172.21.15.90:0/2696417447 conn(0x55dcdd009a50 msgr2=0x55dcdd03eba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:46:05.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.950+0000 7fb7cafa2740 1 -- 172.21.15.90:0/2696417447 shutdown_connections 2024-04-02T14:46:05.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.950+0000 7fb7cafa2740 1 -- 172.21.15.90:0/2696417447 wait complete. 2024-04-02T14:46:05.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.950+0000 7fb7cafa2740 1 -- start start 2024-04-02T14:46:05.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.950+0000 7fb7cafa2740 1 Processor -- start 2024-04-02T14:46:05.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.951+0000 7fb7cafa2740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dcdd03fcb0 0x55dcdd0545b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:05.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.951+0000 7fb7cafa2740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dcdd040670 0x55dcdd054af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:05.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.951+0000 7fb7cafa2740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55dcdd04f440 0x55dcdd055030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:05.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.951+0000 7fb7cafa2740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55dcdcfa36a0 con 0x55dcdd04f440 2024-04-02T14:46:05.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.951+0000 7fb7cafa2740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55dcdcf4f980 con 0x55dcdd03fcb0 2024-04-02T14:46:05.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.951+0000 7fb7cafa2740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55dcdcf62cb0 con 0x55dcdd040670 2024-04-02T14:46:05.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.951+0000 7fb7cafa2740 10 client.0 _ll_register_callbacks cb 0x55dcdcf626a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:46:05.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.951+0000 7fb7c5800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dcdd03fcb0 0x55dcdd0545b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:05.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.951+0000 7fb7c5800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dcdd03fcb0 0x55dcdd0545b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:52530/0 (socket says 172.21.15.90:52530) 2024-04-02T14:46:05.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.951+0000 7fb7c5800640 1 -- 172.21.15.90:0/3271705673 learned_addr learned my addr 172.21.15.90:0/3271705673 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:46:05.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.951+0000 7fb7c6200640 1 --2- 172.21.15.90:0/3271705673 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55dcdd04f440 0x55dcdd055030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:05.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.952+0000 7fb7c4e00640 1 --2- 172.21.15.90:0/3271705673 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dcdd040670 0x55dcdd054af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:05.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.952+0000 7fb7c5800640 1 -- 172.21.15.90:0/3271705673 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dcdd040670 msgr2=0x55dcdd054af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:46:05.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.952+0000 7fb7c5800640 1 --2- 172.21.15.90:0/3271705673 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dcdd040670 0x55dcdd054af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:05.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.952+0000 7fb7c5800640 1 -- 172.21.15.90:0/3271705673 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55dcdd04f440 msgr2=0x55dcdd055030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:46:05.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.952+0000 7fb7c5800640 1 --2- 172.21.15.90:0/3271705673 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55dcdd04f440 0x55dcdd055030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:05.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.952+0000 7fb7c5800640 1 -- 172.21.15.90:0/3271705673 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dcdcd857d0 con 0x55dcdd03fcb0 2024-04-02T14:46:05.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.952+0000 7fb7c5800640 1 --2- 172.21.15.90:0/3271705673 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dcdd03fcb0 0x55dcdd0545b0 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7fb7b40bdf30 tx=0x7fb7b4003ea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=61a8dddba459205e server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:46:05.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.952+0000 7fb7a7400640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:46:05.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.952+0000 7fb7a7400640 1 -- 172.21.15.90:0/3271705673 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7b40bdd70 con 0x55dcdd03fcb0 2024-04-02T14:46:05.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.952+0000 7fb7a7400640 1 -- 172.21.15.90:0/3271705673 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8550+0+0 (secure 0 0 0) 0x7fb7b40be490 con 0x55dcdd03fcb0 2024-04-02T14:46:05.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.952+0000 7fb7cafa2740 1 -- 172.21.15.90:0/3271705673 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb7ac004c80 con 0x55dcdd03fcb0 2024-04-02T14:46:05.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.952+0000 7fb7a7400640 1 -- 172.21.15.90:0/3271705673 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fb7b40bdb90 con 0x55dcdd03fcb0 2024-04-02T14:46:05.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.952+0000 7fb7cafa2740 10 client.28931 fetch_fsmap learned FSMap version 27 2024-04-02T14:46:05.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.952+0000 7fb7cafa2740 1 -- 172.21.15.90:0/3271705673 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55dcdd063710 con 0x55dcdd03fcb0 2024-04-02T14:46:05.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.953+0000 7fb7a7400640 1 -- 172.21.15.90:0/3271705673 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb7ac004c80 con 0x55dcdd03fcb0 2024-04-02T14:46:05.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.953+0000 7fb7a7400640 1 -- 172.21.15.90:0/3271705673 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fb7b40c2150 con 0x55dcdd03fcb0 2024-04-02T14:46:05.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.953+0000 7fb7cafa2740 10 client.28931 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:46:05.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.953+0000 7fb7cafa2740 10 client.28931 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-04-02T14:46:05.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.953+0000 7fb7cafa2740 20 client.28931 populate_metadata read hostname 'smithi090' 2024-04-02T14:46:05.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.953+0000 7fb7cafa2740 1 --2- 172.21.15.90:0/3271705673 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x55dcdd064ab0 0x55dcdd066f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:05.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.954+0000 7fb7cafa2740 4 client.28931 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:46:05.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.954+0000 7fb7cafa2740 1 -- 172.21.15.90:0/3271705673 --> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55dcdcf67040 con 0x55dcdd064ab0 2024-04-02T14:46:05.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.954+0000 7fb7c4e00640 1 --2- 172.21.15.90:0/3271705673 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x55dcdd064ab0 0x55dcdd066f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:05.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.955+0000 7fb7c4e00640 1 --2- 172.21.15.90:0/3271705673 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x55dcdd064ab0 0x55dcdd066f20 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=4d6067c532f923e9 server_cookie=bdfe55df917942d4 in_seq=0 out_seq=0 2024-04-02T14:46:05.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.955+0000 7fb7a7400640 10 client.28931 ms_handle_connect on v2:172.21.15.92:6836/1270635939 2024-04-02T14:46:05.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.956+0000 7fb7a7400640 1 -- 172.21.15.90:0/3271705673 <== mds.3 v2:172.21.15.92:6836/1270635939 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55dcdcf67040 con 0x55dcdd064ab0 2024-04-02T14:46:05.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.956+0000 7fb7a7400640 10 client.28931 handle_command_reply: tid=0 2024-04-02T14:46:05.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.958+0000 7fb7cafa2740 1 client.28931 shutdown 2024-04-02T14:46:05.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.967+0000 7fb7cafa2740 1 -- 172.21.15.90:0/3271705673 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dcdd03fcb0 msgr2=0x55dcdd0545b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:05.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.967+0000 7fb7cafa2740 1 --2- 172.21.15.90:0/3271705673 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dcdd03fcb0 0x55dcdd0545b0 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7fb7b40bdf30 tx=0x7fb7b4003ea0 comp rx=0 tx=0).stop 2024-04-02T14:46:05.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.968+0000 7fb7cafa2740 1 -- 172.21.15.90:0/3271705673 shutdown_connections 2024-04-02T14:46:05.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.968+0000 7fb7cafa2740 1 --2- 172.21.15.90:0/3271705673 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x55dcdd064ab0 0x55dcdd066f20 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:05.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.968+0000 7fb7cafa2740 1 --2- 172.21.15.90:0/3271705673 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55dcdd04f440 0x55dcdd055030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:05.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.968+0000 7fb7cafa2740 1 --2- 172.21.15.90:0/3271705673 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dcdd040670 0x55dcdd054af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:05.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.968+0000 7fb7cafa2740 1 --2- 172.21.15.90:0/3271705673 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dcdd03fcb0 0x55dcdd0545b0 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:05.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.968+0000 7fb7cafa2740 1 -- 172.21.15.90:0/3271705673 >> 172.21.15.90:0/3271705673 conn(0x55dcdd009a50 msgr2=0x55dcdd03daa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:46:05.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.970+0000 7fb7a7400640 0 client.28931 ms_handle_reset on v2:172.21.15.92:6836/1270635939 2024-04-02T14:46:05.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.971+0000 7fb7cafa2740 1 -- 172.21.15.90:0/3271705673 shutdown_connections 2024-04-02T14:46:05.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.971+0000 7fb7cafa2740 1 -- 172.21.15.90:0/3271705673 wait complete. 2024-04-02T14:46:05.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:05.972+0000 7fb7cafa2740 20 client.28931 trim_cache size 0 max 16384 2024-04-02T14:46:06.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.019+0000 7fb7cafa2740 1 Processor -- start 2024-04-02T14:46:06.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.019+0000 7fb7cafa2740 1 -- start start 2024-04-02T14:46:06.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.019+0000 7fb7cafa2740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dcdd03fcb0 0x55dcdd056ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:06.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.020+0000 7fb7cafa2740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55dcdd040670 0x55dcdd060390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:06.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.020+0000 7fb7cafa2740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dcdd04f440 0x55dcdd053d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:06.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.020+0000 7fb7cafa2740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fb7c01ab590 con 0x55dcdd040670 2024-04-02T14:46:06.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.020+0000 7fb7cafa2740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fb7c01ad0d0 con 0x55dcdd03fcb0 2024-04-02T14:46:06.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.020+0000 7fb7cafa2740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fb7c01abc60 con 0x55dcdd04f440 2024-04-02T14:46:06.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.020+0000 7fb7c5800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dcdd03fcb0 0x55dcdd056ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:06.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.020+0000 7fb7c5800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dcdd03fcb0 0x55dcdd056ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:52536/0 (socket says 172.21.15.90:52536) 2024-04-02T14:46:06.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.020+0000 7fb7c5800640 1 -- 172.21.15.90:0/3571850235 learned_addr learned my addr 172.21.15.90:0/3571850235 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:46:06.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.020+0000 7fb7c4e00640 1 --2- 172.21.15.90:0/3571850235 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55dcdd040670 0x55dcdd060390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:06.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.020+0000 7fb7c6200640 1 --2- 172.21.15.90:0/3571850235 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dcdd04f440 0x55dcdd053d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:06.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.020+0000 7fb7c5800640 1 -- 172.21.15.90:0/3571850235 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dcdd04f440 msgr2=0x55dcdd053d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:46:06.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.020+0000 7fb7c5800640 1 --2- 172.21.15.90:0/3571850235 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dcdd04f440 0x55dcdd053d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:06.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.020+0000 7fb7c5800640 1 -- 172.21.15.90:0/3571850235 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55dcdd040670 msgr2=0x55dcdd060390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:46:06.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.020+0000 7fb7c5800640 1 --2- 172.21.15.90:0/3571850235 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55dcdd040670 0x55dcdd060390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:06.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.020+0000 7fb7c5800640 1 -- 172.21.15.90:0/3571850235 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dcdd052b00 con 0x55dcdd03fcb0 2024-04-02T14:46:06.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.021+0000 7fb7c6200640 1 --2- 172.21.15.90:0/3571850235 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dcdd04f440 0x55dcdd053d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:46:06.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.021+0000 7fb7c5800640 1 --2- 172.21.15.90:0/3571850235 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dcdd03fcb0 0x55dcdd056ce0 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7fb7b40ba690 tx=0x7fb7b402bc20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e9728cc8192acb68 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:46:06.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.021+0000 7fb7b2000640 1 -- 172.21.15.90:0/3571850235 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7b40bdd70 con 0x55dcdd03fcb0 2024-04-02T14:46:06.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.022+0000 7fb7b2000640 1 -- 172.21.15.90:0/3571850235 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb7b40d4e50 con 0x55dcdd03fcb0 2024-04-02T14:46:06.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.022+0000 7fb7b2000640 1 -- 172.21.15.90:0/3571850235 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7b40c6a10 con 0x55dcdd03fcb0 2024-04-02T14:46:06.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.023+0000 7fb7cafa2740 1 -- 172.21.15.90:0/3571850235 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dcdd03fcb0 msgr2=0x55dcdd056ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:06.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.023+0000 7fb7cafa2740 1 --2- 172.21.15.90:0/3571850235 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dcdd03fcb0 0x55dcdd056ce0 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7fb7b40ba690 tx=0x7fb7b402bc20 comp rx=0 tx=0).stop 2024-04-02T14:46:06.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.024+0000 7fb7cafa2740 1 -- 172.21.15.90:0/3571850235 shutdown_connections 2024-04-02T14:46:06.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.024+0000 7fb7cafa2740 1 --2- 172.21.15.90:0/3571850235 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dcdd04f440 0x55dcdd053d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:06.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.024+0000 7fb7cafa2740 1 --2- 172.21.15.90:0/3571850235 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55dcdd040670 0x55dcdd060390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:06.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.024+0000 7fb7cafa2740 1 --2- 172.21.15.90:0/3571850235 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dcdd03fcb0 0x55dcdd056ce0 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:06.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.024+0000 7fb7cafa2740 1 -- 172.21.15.90:0/3571850235 >> 172.21.15.90:0/3571850235 conn(0x55dcdd009a50 msgr2=0x55dcdd03c8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:46:06.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.024+0000 7fb7cafa2740 1 -- 172.21.15.90:0/3571850235 shutdown_connections 2024-04-02T14:46:06.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.024+0000 7fb7cafa2740 1 -- 172.21.15.90:0/3571850235 wait complete. 2024-04-02T14:46:06.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.024+0000 7fb7cafa2740 1 -- start start 2024-04-02T14:46:06.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.025+0000 7fb7cafa2740 1 Processor -- start 2024-04-02T14:46:06.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.025+0000 7fb7cafa2740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55dcdd03fcb0 0x55dcdd05c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:06.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.025+0000 7fb7cafa2740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dcdd040670 0x55dcdd05cde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:06.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.025+0000 7fb7cafa2740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dcdd04f440 0x55dcdd050a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:06.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.025+0000 7fb7cafa2740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fb7c01aca00 con 0x55dcdd03fcb0 2024-04-02T14:46:06.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.025+0000 7fb7cafa2740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fb7c01ac330 con 0x55dcdd040670 2024-04-02T14:46:06.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.025+0000 7fb7cafa2740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fb7c01a9590 con 0x55dcdd04f440 2024-04-02T14:46:06.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.025+0000 7fb7cafa2740 10 client.0 _ll_register_callbacks cb 0x7fb7b40f5920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:46:06.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.025+0000 7fb7c4e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dcdd040670 0x55dcdd05cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:06.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.025+0000 7fb7c4e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dcdd040670 0x55dcdd05cde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:52544/0 (socket says 172.21.15.90:52544) 2024-04-02T14:46:06.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.025+0000 7fb7c4e00640 1 -- 172.21.15.90:0/2190193820 learned_addr learned my addr 172.21.15.90:0/2190193820 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:46:06.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.025+0000 7fb7c6200640 1 --2- 172.21.15.90:0/2190193820 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dcdd04f440 0x55dcdd050a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:06.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.026+0000 7fb7c5800640 1 --2- 172.21.15.90:0/2190193820 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55dcdd03fcb0 0x55dcdd05c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:06.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.026+0000 7fb7c4e00640 1 -- 172.21.15.90:0/2190193820 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dcdd04f440 msgr2=0x55dcdd050a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:46:06.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.026+0000 7fb7c4e00640 1 --2- 172.21.15.90:0/2190193820 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dcdd04f440 0x55dcdd050a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:06.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.026+0000 7fb7c4e00640 1 -- 172.21.15.90:0/2190193820 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55dcdd03fcb0 msgr2=0x55dcdd05c8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:46:06.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.026+0000 7fb7c4e00640 1 --2- 172.21.15.90:0/2190193820 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55dcdd03fcb0 0x55dcdd05c8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:06.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.026+0000 7fb7c4e00640 1 -- 172.21.15.90:0/2190193820 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dcdd051200 con 0x55dcdd040670 2024-04-02T14:46:06.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.026+0000 7fb7c4e00640 1 --2- 172.21.15.90:0/2190193820 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dcdd040670 0x55dcdd05cde0 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7fb7ac00abc0 tx=0x7fb7ac00a860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4f4bc4ef31ae9205 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:46:06.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.027+0000 7fb7a7400640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:46:06.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.027+0000 7fb7a7400640 1 -- 172.21.15.90:0/2190193820 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7ac005250 con 0x55dcdd040670 2024-04-02T14:46:06.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.027+0000 7fb7a7400640 1 -- 172.21.15.90:0/2190193820 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8550+0+0 (secure 0 0 0) 0x7fb7ac0053f0 con 0x55dcdd040670 2024-04-02T14:46:06.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.027+0000 7fb7cafa2740 1 -- 172.21.15.90:0/2190193820 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb7b40d4e50 con 0x55dcdd040670 2024-04-02T14:46:06.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.028+0000 7fb7a7400640 1 -- 172.21.15.90:0/2190193820 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fb7ac039050 con 0x55dcdd040670 2024-04-02T14:46:06.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.028+0000 7fb7cafa2740 10 client.28937 fetch_fsmap learned FSMap version 27 2024-04-02T14:46:06.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.028+0000 7fb7cafa2740 1 -- 172.21.15.90:0/2190193820 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55dcdd0caaa0 con 0x55dcdd040670 2024-04-02T14:46:06.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.029+0000 7fb7a7400640 1 -- 172.21.15.90:0/2190193820 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb7b40d4e50 con 0x55dcdd040670 2024-04-02T14:46:06.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.029+0000 7fb7a7400640 1 -- 172.21.15.90:0/2190193820 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fb7ac0186e0 con 0x55dcdd040670 2024-04-02T14:46:06.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.029+0000 7fb7cafa2740 10 client.28937 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:46:06.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.029+0000 7fb7cafa2740 10 client.28937 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-04-02T14:46:06.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.029+0000 7fb7cafa2740 20 client.28937 populate_metadata read hostname 'smithi090' 2024-04-02T14:46:06.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.029+0000 7fb7cafa2740 1 --2- 172.21.15.90:0/2190193820 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x55dcdd0caff0 0x55dcdd0cd410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:06.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.029+0000 7fb7cafa2740 4 client.28937 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-02T14:46:06.025 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.029+0000 7fb7cafa2740 1 -- 172.21.15.90:0/2190193820 --> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55dcdcfb4c30 con 0x55dcdd0caff0 2024-04-02T14:46:06.025 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.030+0000 7fb7c5800640 1 --2- 172.21.15.90:0/2190193820 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x55dcdd0caff0 0x55dcdd0cd410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:06.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.030+0000 7fb7c5800640 1 --2- 172.21.15.90:0/2190193820 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x55dcdd0caff0 0x55dcdd0cd410 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=bffed2ca6589602a server_cookie=a3d760d992d4009f in_seq=0 out_seq=0 2024-04-02T14:46:06.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.030+0000 7fb7a7400640 10 client.28937 ms_handle_connect on v2:172.21.15.92:6836/1270635939 2024-04-02T14:46:06.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.031+0000 7fb7a7400640 1 -- 172.21.15.90:0/2190193820 <== mds.3 v2:172.21.15.92:6836/1270635939 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55dcdcfb4c30 con 0x55dcdd0caff0 2024-04-02T14:46:06.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.031+0000 7fb7a7400640 10 client.28937 handle_command_reply: tid=0 2024-04-02T14:46:06.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.031+0000 7fb7cafa2740 1 client.28937 shutdown 2024-04-02T14:46:06.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.031+0000 7fb7cafa2740 1 -- 172.21.15.90:0/2190193820 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dcdd040670 msgr2=0x55dcdd05cde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:06.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.031+0000 7fb7cafa2740 1 --2- 172.21.15.90:0/2190193820 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dcdd040670 0x55dcdd05cde0 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7fb7ac00abc0 tx=0x7fb7ac00a860 comp rx=0 tx=0).stop 2024-04-02T14:46:06.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.031+0000 7fb7cafa2740 1 -- 172.21.15.90:0/2190193820 shutdown_connections 2024-04-02T14:46:06.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.031+0000 7fb7cafa2740 1 --2- 172.21.15.90:0/2190193820 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x55dcdd0caff0 0x55dcdd0cd410 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:06.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.031+0000 7fb7cafa2740 1 --2- 172.21.15.90:0/2190193820 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55dcdd04f440 0x55dcdd050a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:06.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.031+0000 7fb7cafa2740 1 --2- 172.21.15.90:0/2190193820 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55dcdd040670 0x55dcdd05cde0 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:06.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.031+0000 7fb7cafa2740 1 --2- 172.21.15.90:0/2190193820 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55dcdd03fcb0 0x55dcdd05c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:06.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.031+0000 7fb7cafa2740 1 -- 172.21.15.90:0/2190193820 >> 172.21.15.90:0/2190193820 conn(0x55dcdd009a50 msgr2=0x55dcdd03d210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:46:06.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.031+0000 7fb7a7400640 0 client.28937 ms_handle_reset on v2:172.21.15.92:6836/1270635939 2024-04-02T14:46:06.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.031+0000 7fb7cafa2740 1 -- 172.21.15.90:0/2190193820 shutdown_connections 2024-04-02T14:46:06.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.031+0000 7fb7cafa2740 1 -- 172.21.15.90:0/2190193820 wait complete. 2024-04-02T14:46:06.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.031+0000 7fb7cafa2740 20 client.28937 trim_cache size 0 max 16384 2024-04-02T14:46:06.028 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-02T14:46:06.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.032+0000 7fb7c8200640 1 -- 172.21.15.90:0/2390671442 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fb7a80a3ef0 msgr2=0x7fb7a80a63b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:06.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.032+0000 7fb7c8200640 1 --2- 172.21.15.90:0/2390671442 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fb7a80a3ef0 0x7fb7a80a63b0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fb7b8009fd0 tx=0x7fb7b800d040 comp rx=0 tx=0).stop 2024-04-02T14:46:06.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.032+0000 7fb7c8200640 1 -- 172.21.15.90:0/2390671442 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb7c0101a90 msgr2=0x7fb7c0198360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:06.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.032+0000 7fb7c8200640 1 --2- 172.21.15.90:0/2390671442 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb7c0101a90 0x7fb7c0198360 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7fb7b4009900 tx=0x7fb7b40045e0 comp rx=0 tx=0).stop 2024-04-02T14:46:06.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.033+0000 7fb7c8200640 1 -- 172.21.15.90:0/2390671442 shutdown_connections 2024-04-02T14:46:06.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.033+0000 7fb7c8200640 1 --2- 172.21.15.90:0/2390671442 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fb7a80a3ef0 0x7fb7a80a63b0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:06.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.033+0000 7fb7c8200640 1 --2- 172.21.15.90:0/2390671442 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb7c0107aa0 0x7fb7c019cc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:06.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.033+0000 7fb7c8200640 1 --2- 172.21.15.90:0/2390671442 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb7c0102450 0x7fb7c01988a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:06.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.033+0000 7fb7c8200640 1 --2- 172.21.15.90:0/2390671442 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb7c0101a90 0x7fb7c0198360 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:06.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.033+0000 7fb7c8200640 1 -- 172.21.15.90:0/2390671442 >> 172.21.15.90:0/2390671442 conn(0x7fb7c0078f00 msgr2=0x7fb7c01042f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:46:06.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.034+0000 7fb7c8200640 1 -- 172.21.15.90:0/2390671442 shutdown_connections 2024-04-02T14:46:06.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.034+0000 7fb7c8200640 1 -- 172.21.15.90:0/2390671442 wait complete. 2024-04-02T14:46:06.050 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-02T14:46:06.104 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:46:05 smithi092 ceph-mon[36497]: pgmap v347: 97 pgs: 97 active+clean; 4.7 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 10 MiB/s wr, 1.10k op/s 2024-04-02T14:46:06.105 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:46:05 smithi092 ceph-mon[36497]: from='client.? 172.21.15.90:0/1476862397' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-02T14:46:06.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:46:05 smithi177 ceph-mon[36431]: pgmap v347: 97 pgs: 97 active+clean; 4.7 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 10 MiB/s wr, 1.10k op/s 2024-04-02T14:46:06.181 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:46:05 smithi177 ceph-mon[36431]: from='client.? 172.21.15.90:0/1476862397' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-02T14:46:06.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.300+0000 7f7361a00640 1 -- 172.21.15.90:0/2219236172 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f735c112940 msgr2=0x7f735c114d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:06.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.300+0000 7f7361a00640 1 --2- 172.21.15.90:0/2219236172 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f735c112940 0x7f735c114d30 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f7354009cd0 tx=0x7f735402c0e0 comp rx=0 tx=0).stop 2024-04-02T14:46:06.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.300+0000 7f7361a00640 1 -- 172.21.15.90:0/2219236172 shutdown_connections 2024-04-02T14:46:06.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.300+0000 7f7361a00640 1 --2- 172.21.15.90:0/2219236172 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f735c112940 0x7f735c114d30 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:06.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.300+0000 7f7361a00640 1 --2- 172.21.15.90:0/2219236172 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f735c076950 0x7f735c112400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:06.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.300+0000 7f7361a00640 1 --2- 172.21.15.90:0/2219236172 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f735c076030 0x7f735c076410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:06.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.300+0000 7f7361a00640 1 -- 172.21.15.90:0/2219236172 >> 172.21.15.90:0/2219236172 conn(0x7f735c0ff7b0 msgr2=0x7f735c101bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:46:06.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.300+0000 7f7361a00640 1 -- 172.21.15.90:0/2219236172 shutdown_connections 2024-04-02T14:46:06.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.300+0000 7f7361a00640 1 -- 172.21.15.90:0/2219236172 wait complete. 2024-04-02T14:46:06.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.301+0000 7f7361a00640 1 Processor -- start 2024-04-02T14:46:06.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.302+0000 7f7361a00640 1 -- start start 2024-04-02T14:46:06.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.303+0000 7f7361a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f735c076030 0x7f735c1a46b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:06.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.303+0000 7f7361a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f735c076950 0x7f735c1a4bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:06.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.303+0000 7f7361a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f735c112940 0x7f735c19e820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:06.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.303+0000 7f735a000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f735c076950 0x7f735c1a4bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:06.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.303+0000 7f735a000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f735c076950 0x7f735c1a4bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:52566/0 (socket says 172.21.15.90:52566) 2024-04-02T14:46:06.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.303+0000 7f735a000640 1 -- 172.21.15.90:0/1082581323 learned_addr learned my addr 172.21.15.90:0/1082581323 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:46:06.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.303+0000 7f735b400640 1 --2- 172.21.15.90:0/1082581323 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f735c112940 0x7f735c19e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:06.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.304+0000 7f7361a00640 1 -- 172.21.15.90:0/1082581323 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f735c118460 con 0x7f735c076030 2024-04-02T14:46:06.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.304+0000 7f7361a00640 1 -- 172.21.15.90:0/1082581323 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f735c1182e0 con 0x7f735c076950 2024-04-02T14:46:06.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.304+0000 7f7361a00640 1 -- 172.21.15.90:0/1082581323 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f735c1185e0 con 0x7f735c112940 2024-04-02T14:46:06.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.304+0000 7f735a000640 1 -- 172.21.15.90:0/1082581323 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f735c112940 msgr2=0x7f735c19e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:06.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.304+0000 7f735a000640 1 --2- 172.21.15.90:0/1082581323 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f735c112940 0x7f735c19e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:06.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.304+0000 7f735a000640 1 -- 172.21.15.90:0/1082581323 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f735c076030 msgr2=0x7f735c1a46b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:06.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.304+0000 7f735aa00640 1 --2- 172.21.15.90:0/1082581323 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f735c076030 0x7f735c1a46b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:06.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.304+0000 7f735a000640 1 --2- 172.21.15.90:0/1082581323 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f735c076030 0x7f735c1a46b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:06.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.304+0000 7f735a000640 1 -- 172.21.15.90:0/1082581323 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f735c19eff0 con 0x7f735c076950 2024-04-02T14:46:06.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.304+0000 7f735aa00640 1 --2- 172.21.15.90:0/1082581323 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f735c076030 0x7f735c1a46b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:46:06.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.305+0000 7f735a000640 1 --2- 172.21.15.90:0/1082581323 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f735c076950 0x7f735c1a4bf0 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f734800e9f0 tx=0x7f734800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:46:06.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.305+0000 7f7353400640 1 -- 172.21.15.90:0/1082581323 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f734800cd80 con 0x7f735c076950 2024-04-02T14:46:06.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.305+0000 7f7353400640 1 -- 172.21.15.90:0/1082581323 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7348010910 con 0x7f735c076950 2024-04-02T14:46:06.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.305+0000 7f7353400640 1 -- 172.21.15.90:0/1082581323 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73480188f0 con 0x7f735c076950 2024-04-02T14:46:06.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.305+0000 7f7361a00640 1 -- 172.21.15.90:0/1082581323 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f735c19f2e0 con 0x7f735c076950 2024-04-02T14:46:06.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.305+0000 7f7361a00640 1 -- 172.21.15.90:0/1082581323 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f735c1ab500 con 0x7f735c076950 2024-04-02T14:46:06.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.309+0000 7f73647b7740 1 Processor -- start 2024-04-02T14:46:06.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.310+0000 7f73647b7740 1 -- start start 2024-04-02T14:46:06.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.311+0000 7f73647b7740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d89b4d19f0 0x55d89b4d0740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:06.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.311+0000 7f73647b7740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d89b4d0d10 0x55d89b4d1190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:06.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.311+0000 7f735b400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d89b4d19f0 0x55d89b4d0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:06.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.311+0000 7f73647b7740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d89b4da080 0x55d89b4dc470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:06.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.311+0000 7f73647b7740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55d89b411400 con 0x55d89b4d0d10 2024-04-02T14:46:06.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.311+0000 7f73647b7740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55d89b3f5f10 con 0x55d89b4d19f0 2024-04-02T14:46:06.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.311+0000 7f73647b7740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55d89b4356f0 con 0x55d89b4da080 2024-04-02T14:46:06.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.311+0000 7f735b400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d89b4d19f0 0x55d89b4d0740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:52572/0 (socket says 172.21.15.90:52572) 2024-04-02T14:46:06.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.311+0000 7f735b400640 1 -- 172.21.15.90:0/635425302 learned_addr learned my addr 172.21.15.90:0/635425302 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:46:06.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.312+0000 7f735aa00640 1 --2- 172.21.15.90:0/635425302 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d89b4d0d10 0x55d89b4d1190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:06.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.312+0000 7f735b400640 1 -- 172.21.15.90:0/635425302 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d89b4da080 msgr2=0x55d89b4dc470 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-02T14:46:06.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.312+0000 7f735b400640 1 --2- 172.21.15.90:0/635425302 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d89b4da080 0x55d89b4dc470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:06.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.312+0000 7f735b400640 1 -- 172.21.15.90:0/635425302 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d89b4d0d10 msgr2=0x55d89b4d1190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:46:06.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.312+0000 7f735b400640 1 --2- 172.21.15.90:0/635425302 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d89b4d0d10 0x55d89b4d1190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:06.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.312+0000 7f735b400640 1 -- 172.21.15.90:0/635425302 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d89b1493f0 con 0x55d89b4d19f0 2024-04-02T14:46:06.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.312+0000 7f735aa00640 1 --2- 172.21.15.90:0/635425302 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d89b4d0d10 0x55d89b4d1190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:46:06.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.312+0000 7f7353400640 1 -- 172.21.15.90:0/1082581323 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f7348010ac0 con 0x7f735c076950 2024-04-02T14:46:06.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.312+0000 7f735b400640 1 --2- 172.21.15.90:0/635425302 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d89b4d19f0 0x55d89b4d0740 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f7354005cc0 tx=0x7f73540098e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6b72df10575ba2bb server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:46:06.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.313+0000 7f7347e00640 1 -- 172.21.15.90:0/635425302 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7354043070 con 0x55d89b4d19f0 2024-04-02T14:46:06.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.313+0000 7f7347e00640 1 -- 172.21.15.90:0/635425302 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7354032e70 con 0x55d89b4d19f0 2024-04-02T14:46:06.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.313+0000 7f7347e00640 1 -- 172.21.15.90:0/635425302 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7354039040 con 0x55d89b4d19f0 2024-04-02T14:46:06.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.313+0000 7f73647b7740 1 -- 172.21.15.90:0/635425302 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d89b4d19f0 msgr2=0x55d89b4d0740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:06.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.313+0000 7f73647b7740 1 --2- 172.21.15.90:0/635425302 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d89b4d19f0 0x55d89b4d0740 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f7354005cc0 tx=0x7f73540098e0 comp rx=0 tx=0).stop 2024-04-02T14:46:06.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.314+0000 7f7353400640 1 --2- 172.21.15.90:0/1082581323 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f73400a3ef0 0x7f73400a63b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:06.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.314+0000 7f73647b7740 1 -- 172.21.15.90:0/635425302 shutdown_connections 2024-04-02T14:46:06.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.314+0000 7f73647b7740 1 --2- 172.21.15.90:0/635425302 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d89b4da080 0x55d89b4dc470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:06.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.314+0000 7f73647b7740 1 --2- 172.21.15.90:0/635425302 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d89b4d0d10 0x55d89b4d1190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:06.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.314+0000 7f73647b7740 1 --2- 172.21.15.90:0/635425302 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d89b4d19f0 0x55d89b4d0740 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:06.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.314+0000 7f73647b7740 1 -- 172.21.15.90:0/635425302 >> 172.21.15.90:0/635425302 conn(0x55d89b497a50 msgr2=0x55d89b4cb760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:46:06.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.314+0000 7f7353400640 1 -- 172.21.15.90:0/1082581323 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 8550+0+0 (secure 0 0 0) 0x7f73480d0bb0 con 0x7f735c076950 2024-04-02T14:46:06.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.314+0000 7f735b400640 1 --2- 172.21.15.90:0/1082581323 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f73400a3ef0 0x7f73400a63b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:06.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.314+0000 7f73647b7740 1 -- 172.21.15.90:0/635425302 shutdown_connections 2024-04-02T14:46:06.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.314+0000 7f73647b7740 1 -- 172.21.15.90:0/635425302 wait complete. 2024-04-02T14:46:06.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.315+0000 7f735b400640 1 --2- 172.21.15.90:0/1082581323 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f73400a3ef0 0x7f73400a63b0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f735402c6f0 tx=0x7f735402c0e0 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:46:06.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.315+0000 7f73647b7740 1 -- start start 2024-04-02T14:46:06.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.315+0000 7f73647b7740 1 Processor -- start 2024-04-02T14:46:06.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.315+0000 7f73647b7740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d89b4d0d10 0x55d89b4d7080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:06.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.315+0000 7f73647b7740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d89b4d19f0 0x55d89b4d75c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:06.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.315+0000 7f73647b7740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d89b4da080 0x55d89b4d7b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:06.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.315+0000 7f73647b7740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55d89b4316a0 con 0x55d89b4da080 2024-04-02T14:46:06.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.315+0000 7f73647b7740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55d89b3dd980 con 0x55d89b4d19f0 2024-04-02T14:46:06.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.315+0000 7f73647b7740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55d89b3f0cb0 con 0x55d89b4d0d10 2024-04-02T14:46:06.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.315+0000 7f73647b7740 10 client.0 _ll_register_callbacks cb 0x55d89b3f06a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:46:06.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.316+0000 7f735b400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d89b4da080 0x55d89b4d7b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:06.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.316+0000 7f735aa00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d89b4d0d10 0x55d89b4d7080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:06.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.316+0000 7f735b400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d89b4da080 0x55d89b4d7b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:60250/0 (socket says 172.21.15.90:60250) 2024-04-02T14:46:06.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.316+0000 7f735b400640 1 -- 172.21.15.90:0/1057706726 learned_addr learned my addr 172.21.15.90:0/1057706726 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:46:06.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.316+0000 7f735a000640 1 --2- 172.21.15.90:0/1057706726 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d89b4d19f0 0x55d89b4d75c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:06.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.316+0000 7f735b400640 1 -- 172.21.15.90:0/1057706726 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d89b4d0d10 msgr2=0x55d89b4d7080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:46:06.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.316+0000 7f735b400640 1 --2- 172.21.15.90:0/1057706726 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d89b4d0d10 0x55d89b4d7080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:06.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.316+0000 7f735b400640 1 -- 172.21.15.90:0/1057706726 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d89b4d19f0 msgr2=0x55d89b4d75c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:46:06.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.316+0000 7f735b400640 1 --2- 172.21.15.90:0/1057706726 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d89b4d19f0 0x55d89b4d75c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:06.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.316+0000 7f735b400640 1 -- 172.21.15.90:0/1057706726 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d89b2137d0 con 0x55d89b4da080 2024-04-02T14:46:06.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.316+0000 7f735b400640 1 --2- 172.21.15.90:0/1057706726 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d89b4da080 0x55d89b4d7b00 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f7354037040 tx=0x7f73540098e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9ad56a93f764316 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:46:06.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.316+0000 7f7344c00640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-02T14:46:06.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.317+0000 7f7344c00640 1 -- 172.21.15.90:0/1057706726 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7354058070 con 0x55d89b4da080 2024-04-02T14:46:06.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.317+0000 7f7344c00640 1 -- 172.21.15.90:0/1057706726 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8550+0+0 (secure 0 0 0) 0x7f7354004610 con 0x55d89b4da080 2024-04-02T14:46:06.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.317+0000 7f73647b7740 1 -- 172.21.15.90:0/1057706726 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55d89b3f5040 con 0x55d89b4da080 2024-04-02T14:46:06.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.317+0000 7f7344c00640 1 -- 172.21.15.90:0/1057706726 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f7354039040 con 0x55d89b4da080 2024-04-02T14:46:06.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.317+0000 7f73647b7740 10 client.17934 fetch_fsmap learned FSMap version 27 2024-04-02T14:46:06.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.317+0000 7f73647b7740 1 -- 172.21.15.90:0/1057706726 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55d89b4e8f30 con 0x55d89b4da080 2024-04-02T14:46:06.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.317+0000 7f7344c00640 1 -- 172.21.15.90:0/1057706726 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55d89b3f5040 con 0x55d89b4da080 2024-04-02T14:46:06.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.318+0000 7f7344c00640 1 -- 172.21.15.90:0/1057706726 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f7354053070 con 0x55d89b4da080 2024-04-02T14:46:06.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.319+0000 7f73647b7740 10 client.17934 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:46:06.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.319+0000 7f73647b7740 10 client.17934 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-04-02T14:46:06.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.319+0000 7f73647b7740 20 client.17934 populate_metadata read hostname 'smithi090' 2024-04-02T14:46:06.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.319+0000 7f73647b7740 1 --2- 172.21.15.90:0/1057706726 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x55d89b4ea280 0x55d89b4ec670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:06.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.319+0000 7f73647b7740 4 client.17934 mds_command: new command op to 24547 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:46:06.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.319+0000 7f73647b7740 1 -- 172.21.15.90:0/1057706726 --> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d89b442c30 con 0x55d89b4ea280 2024-04-02T14:46:06.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.319+0000 7f735aa00640 1 --2- 172.21.15.90:0/1057706726 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x55d89b4ea280 0x55d89b4ec670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:06.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.320+0000 7f735aa00640 1 --2- 172.21.15.90:0/1057706726 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x55d89b4ea280 0x55d89b4ec670 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=1476dd25bd86b5a1 server_cookie=91eeebb8f1b22fb6 in_seq=0 out_seq=0 2024-04-02T14:46:06.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.320+0000 7f7344c00640 10 client.17934 ms_handle_connect on v2:172.21.15.177:6838/664895663 2024-04-02T14:46:06.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.321+0000 7f7344c00640 1 -- 172.21.15.90:0/1057706726 <== mds.2 v2:172.21.15.177:6838/664895663 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d89b442c30 con 0x55d89b4ea280 2024-04-02T14:46:06.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.321+0000 7f7344c00640 10 client.17934 handle_command_reply: tid=0 2024-04-02T14:46:06.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.321+0000 7f73647b7740 1 client.17934 shutdown 2024-04-02T14:46:06.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.321+0000 7f73647b7740 1 -- 172.21.15.90:0/1057706726 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d89b4da080 msgr2=0x55d89b4d7b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:06.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.321+0000 7f73647b7740 1 --2- 172.21.15.90:0/1057706726 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d89b4da080 0x55d89b4d7b00 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f7354037040 tx=0x7f73540098e0 comp rx=0 tx=0).stop 2024-04-02T14:46:06.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.321+0000 7f73647b7740 1 -- 172.21.15.90:0/1057706726 shutdown_connections 2024-04-02T14:46:06.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.321+0000 7f73647b7740 1 --2- 172.21.15.90:0/1057706726 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x55d89b4ea280 0x55d89b4ec670 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:06.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.321+0000 7f73647b7740 1 --2- 172.21.15.90:0/1057706726 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d89b4da080 0x55d89b4d7b00 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:06.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.321+0000 7f73647b7740 1 --2- 172.21.15.90:0/1057706726 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d89b4d19f0 0x55d89b4d75c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:06.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.321+0000 7f7344c00640 0 client.17934 ms_handle_reset on v2:172.21.15.177:6838/664895663 2024-04-02T14:46:06.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.321+0000 7f73647b7740 1 --2- 172.21.15.90:0/1057706726 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d89b4d0d10 0x55d89b4d7080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:06.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.321+0000 7f73647b7740 1 -- 172.21.15.90:0/1057706726 >> 172.21.15.90:0/1057706726 conn(0x55d89b497a50 msgr2=0x55d89b4cd4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:46:06.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.322+0000 7f73647b7740 1 -- 172.21.15.90:0/1057706726 shutdown_connections 2024-04-02T14:46:06.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.322+0000 7f73647b7740 1 -- 172.21.15.90:0/1057706726 wait complete. 2024-04-02T14:46:06.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.322+0000 7f73647b7740 20 client.17934 trim_cache size 0 max 16384 2024-04-02T14:46:06.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.340+0000 7f73647b7740 1 Processor -- start 2024-04-02T14:46:06.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.340+0000 7f73647b7740 1 -- start start 2024-04-02T14:46:06.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.341+0000 7f73647b7740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d89b4d0d10 0x55d89b4e5af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:06.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.341+0000 7f73647b7740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d89b4d19f0 0x55d89b4cd680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:06.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.341+0000 7f73647b7740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d89b4da080 0x55d89b4d7920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:06.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.341+0000 7f73647b7740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f735c1b3ef0 con 0x55d89b4d0d10 2024-04-02T14:46:06.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.341+0000 7f73647b7740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f735c1b5a30 con 0x55d89b4d19f0 2024-04-02T14:46:06.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.341+0000 7f73647b7740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f735c1b45c0 con 0x55d89b4da080 2024-04-02T14:46:06.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.341+0000 7f735aa00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d89b4d0d10 0x55d89b4e5af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:06.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.341+0000 7f735aa00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d89b4d0d10 0x55d89b4e5af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:60264/0 (socket says 172.21.15.90:60264) 2024-04-02T14:46:06.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.341+0000 7f735aa00640 1 -- 172.21.15.90:0/2028403313 learned_addr learned my addr 172.21.15.90:0/2028403313 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:46:06.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.341+0000 7f735a000640 1 --2- 172.21.15.90:0/2028403313 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d89b4d19f0 0x55d89b4cd680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:06.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.341+0000 7f735b400640 1 --2- 172.21.15.90:0/2028403313 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d89b4da080 0x55d89b4d7920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:06.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.341+0000 7f735aa00640 1 -- 172.21.15.90:0/2028403313 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d89b4da080 msgr2=0x55d89b4d7920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:46:06.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.341+0000 7f735aa00640 1 --2- 172.21.15.90:0/2028403313 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d89b4da080 0x55d89b4d7920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:06.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.341+0000 7f735aa00640 1 -- 172.21.15.90:0/2028403313 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d89b4d19f0 msgr2=0x55d89b4cd680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:46:06.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.342+0000 7f735aa00640 1 --2- 172.21.15.90:0/2028403313 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d89b4d19f0 0x55d89b4cd680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:06.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.342+0000 7f735aa00640 1 -- 172.21.15.90:0/2028403313 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d89b4cc920 con 0x55d89b4d0d10 2024-04-02T14:46:06.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.342+0000 7f735a000640 1 --2- 172.21.15.90:0/2028403313 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d89b4d19f0 0x55d89b4cd680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:46:06.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.342+0000 7f735aa00640 1 --2- 172.21.15.90:0/2028403313 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d89b4d0d10 0x55d89b4e5af0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f734c009bc0 tx=0x7f734c014840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc0ac182fc255aa7 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:46:06.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.342+0000 7f7347400640 1 -- 172.21.15.90:0/2028403313 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f734c0140d0 con 0x55d89b4d0d10 2024-04-02T14:46:06.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.342+0000 7f7347400640 1 -- 172.21.15.90:0/2028403313 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f734c014270 con 0x55d89b4d0d10 2024-04-02T14:46:06.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.342+0000 7f7347400640 1 -- 172.21.15.90:0/2028403313 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f734c01e2d0 con 0x55d89b4d0d10 2024-04-02T14:46:06.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.343+0000 7f73647b7740 1 -- 172.21.15.90:0/2028403313 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d89b4d0d10 msgr2=0x55d89b4e5af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:06.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.343+0000 7f73647b7740 1 --2- 172.21.15.90:0/2028403313 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d89b4d0d10 0x55d89b4e5af0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f734c009bc0 tx=0x7f734c014840 comp rx=0 tx=0).stop 2024-04-02T14:46:06.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.343+0000 7f73647b7740 1 -- 172.21.15.90:0/2028403313 shutdown_connections 2024-04-02T14:46:06.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.343+0000 7f73647b7740 1 --2- 172.21.15.90:0/2028403313 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d89b4da080 0x55d89b4d7920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:06.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.343+0000 7f73647b7740 1 --2- 172.21.15.90:0/2028403313 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d89b4d19f0 0x55d89b4cd680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:06.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.343+0000 7f73647b7740 1 --2- 172.21.15.90:0/2028403313 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d89b4d0d10 0x55d89b4e5af0 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:06.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.343+0000 7f73647b7740 1 -- 172.21.15.90:0/2028403313 >> 172.21.15.90:0/2028403313 conn(0x55d89b497a50 msgr2=0x55d89b4d60b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:46:06.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.343+0000 7f73647b7740 1 -- 172.21.15.90:0/2028403313 shutdown_connections 2024-04-02T14:46:06.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.343+0000 7f73647b7740 1 -- 172.21.15.90:0/2028403313 wait complete. 2024-04-02T14:46:06.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.344+0000 7f73647b7740 1 -- start start 2024-04-02T14:46:06.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.344+0000 7f73647b7740 1 Processor -- start 2024-04-02T14:46:06.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.345+0000 7f73647b7740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d89b4d0d10 0x55d89b4d34d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:06.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.345+0000 7f73647b7740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d89b4d19f0 0x55d89b4d3a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:06.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.346+0000 7f73647b7740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d89b4da080 0x55d89b4d3f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:06.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.346+0000 7f73647b7740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f735c1b5360 con 0x55d89b4d19f0 2024-04-02T14:46:06.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.346+0000 7f73647b7740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f735c1b4c90 con 0x55d89b4d0d10 2024-04-02T14:46:06.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.346+0000 7f73647b7740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f735c1b1ef0 con 0x55d89b4da080 2024-04-02T14:46:06.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.346+0000 7f735b400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d89b4da080 0x55d89b4d3f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:06.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.346+0000 7f73647b7740 10 client.0 _ll_register_callbacks cb 0x7f7354032d90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:46:06.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.346+0000 7f735b400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d89b4da080 0x55d89b4d3f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:58326/0 (socket says 172.21.15.90:58326) 2024-04-02T14:46:06.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.346+0000 7f735b400640 1 -- 172.21.15.90:0/3640695467 learned_addr learned my addr 172.21.15.90:0/3640695467 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:46:06.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.346+0000 7f735aa00640 1 --2- 172.21.15.90:0/3640695467 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d89b4d0d10 0x55d89b4d34d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:06.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.347+0000 7f735a000640 1 --2- 172.21.15.90:0/3640695467 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d89b4d19f0 0x55d89b4d3a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:06.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.347+0000 7f735a000640 1 -- 172.21.15.90:0/3640695467 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d89b4da080 msgr2=0x55d89b4d3f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:46:06.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.348+0000 7f735a000640 1 --2- 172.21.15.90:0/3640695467 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d89b4da080 0x55d89b4d3f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:06.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.348+0000 7f735a000640 1 -- 172.21.15.90:0/3640695467 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d89b4d0d10 msgr2=0x55d89b4d34d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:46:06.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.348+0000 7f735a000640 1 --2- 172.21.15.90:0/3640695467 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d89b4d0d10 0x55d89b4d34d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:06.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.348+0000 7f735a000640 1 -- 172.21.15.90:0/3640695467 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d89b4e12a0 con 0x55d89b4d19f0 2024-04-02T14:46:06.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.348+0000 7f735b400640 1 --2- 172.21.15.90:0/3640695467 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d89b4da080 0x55d89b4d3f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:46:06.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.348+0000 7f735a000640 1 --2- 172.21.15.90:0/3640695467 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d89b4d19f0 0x55d89b4d3a10 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f73480ce280 tx=0x7f73480bd040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9fff03cbbed749d server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:46:06.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.348+0000 7f7344c00640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-02T14:46:06.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.349+0000 7f7344c00640 1 -- 172.21.15.90:0/3640695467 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73480a5740 con 0x55d89b4d19f0 2024-04-02T14:46:06.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.349+0000 7f7344c00640 1 -- 172.21.15.90:0/3640695467 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8550+0+0 (secure 0 0 0) 0x7f73480ce500 con 0x55d89b4d19f0 2024-04-02T14:46:06.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.349+0000 7f73647b7740 1 -- 172.21.15.90:0/3640695467 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f734c014270 con 0x55d89b4d19f0 2024-04-02T14:46:06.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.349+0000 7f7344c00640 1 -- 172.21.15.90:0/3640695467 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f73480ce700 con 0x55d89b4d19f0 2024-04-02T14:46:06.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.349+0000 7f73647b7740 10 client.17940 fetch_fsmap learned FSMap version 27 2024-04-02T14:46:06.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.349+0000 7f73647b7740 1 -- 172.21.15.90:0/3640695467 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55d89b4e2600 con 0x55d89b4d19f0 2024-04-02T14:46:06.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.349+0000 7f7344c00640 1 -- 172.21.15.90:0/3640695467 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f734c014270 con 0x55d89b4d19f0 2024-04-02T14:46:06.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.349+0000 7f7344c00640 1 -- 172.21.15.90:0/3640695467 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f734809b990 con 0x55d89b4d19f0 2024-04-02T14:46:06.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.349+0000 7f73647b7740 10 client.17940 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:46:06.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.349+0000 7f73647b7740 10 client.17940 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-04-02T14:46:06.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.349+0000 7f73647b7740 20 client.17940 populate_metadata read hostname 'smithi090' 2024-04-02T14:46:06.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.349+0000 7f73647b7740 1 --2- 172.21.15.90:0/3640695467 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x55d89b5506c0 0x55d89b552ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:06.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.349+0000 7f73647b7740 4 client.17940 mds_command: new command op to 24547 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-02T14:46:06.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.349+0000 7f73647b7740 1 -- 172.21.15.90:0/3640695467 --> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55d89b43cf80 con 0x55d89b5506c0 2024-04-02T14:46:06.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.350+0000 7f735aa00640 1 --2- 172.21.15.90:0/3640695467 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x55d89b5506c0 0x55d89b552ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:06.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.350+0000 7f7344c00640 10 client.17940 ms_handle_connect on v2:172.21.15.177:6838/664895663 2024-04-02T14:46:06.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.350+0000 7f735aa00640 1 --2- 172.21.15.90:0/3640695467 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x55d89b5506c0 0x55d89b552ae0 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=a71249d32713088e server_cookie=dbec92352137160b in_seq=0 out_seq=0 2024-04-02T14:46:06.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.350+0000 7f7344c00640 1 -- 172.21.15.90:0/3640695467 <== mds.2 v2:172.21.15.177:6838/664895663 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55d89b43cf80 con 0x55d89b5506c0 2024-04-02T14:46:06.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.350+0000 7f7344c00640 10 client.17940 handle_command_reply: tid=0 2024-04-02T14:46:06.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.350+0000 7f73647b7740 1 client.17940 shutdown 2024-04-02T14:46:06.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.350+0000 7f73647b7740 1 -- 172.21.15.90:0/3640695467 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d89b4d19f0 msgr2=0x55d89b4d3a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:06.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.350+0000 7f73647b7740 1 --2- 172.21.15.90:0/3640695467 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d89b4d19f0 0x55d89b4d3a10 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f73480ce280 tx=0x7f73480bd040 comp rx=0 tx=0).stop 2024-04-02T14:46:06.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.350+0000 7f73647b7740 1 -- 172.21.15.90:0/3640695467 shutdown_connections 2024-04-02T14:46:06.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.350+0000 7f73647b7740 1 --2- 172.21.15.90:0/3640695467 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x55d89b5506c0 0x55d89b552ae0 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:06.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.351+0000 7f73647b7740 1 --2- 172.21.15.90:0/3640695467 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d89b4da080 0x55d89b4d3f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:06.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.351+0000 7f73647b7740 1 --2- 172.21.15.90:0/3640695467 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d89b4d19f0 0x55d89b4d3a10 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:06.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.351+0000 7f73647b7740 1 --2- 172.21.15.90:0/3640695467 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d89b4d0d10 0x55d89b4d34d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:06.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.351+0000 7f73647b7740 1 -- 172.21.15.90:0/3640695467 >> 172.21.15.90:0/3640695467 conn(0x55d89b497a50 msgr2=0x55d89b4d7830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:46:06.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.351+0000 7f7344c00640 0 client.17940 ms_handle_reset on v2:172.21.15.177:6838/664895663 2024-04-02T14:46:06.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.351+0000 7f73647b7740 1 -- 172.21.15.90:0/3640695467 shutdown_connections 2024-04-02T14:46:06.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.351+0000 7f73647b7740 1 -- 172.21.15.90:0/3640695467 wait complete. 2024-04-02T14:46:06.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.351+0000 7f73647b7740 20 client.17940 trim_cache size 0 max 16384 2024-04-02T14:46:06.348 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-02T14:46:06.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.352+0000 7f7361a00640 1 -- 172.21.15.90:0/1082581323 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f73400a3ef0 msgr2=0x7f73400a63b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:06.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.352+0000 7f7361a00640 1 --2- 172.21.15.90:0/1082581323 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f73400a3ef0 0x7f73400a63b0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f735402c6f0 tx=0x7f735402c0e0 comp rx=0 tx=0).stop 2024-04-02T14:46:06.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.352+0000 7f7361a00640 1 -- 172.21.15.90:0/1082581323 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f735c076950 msgr2=0x7f735c1a4bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:06.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.352+0000 7f7361a00640 1 --2- 172.21.15.90:0/1082581323 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f735c076950 0x7f735c1a4bf0 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f734800e9f0 tx=0x7f734800eeb0 comp rx=0 tx=0).stop 2024-04-02T14:46:06.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.352+0000 7f7361a00640 1 -- 172.21.15.90:0/1082581323 shutdown_connections 2024-04-02T14:46:06.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.352+0000 7f7361a00640 1 --2- 172.21.15.90:0/1082581323 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f73400a3ef0 0x7f73400a63b0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:06.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.352+0000 7f7361a00640 1 --2- 172.21.15.90:0/1082581323 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f735c112940 0x7f735c19e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:06.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.352+0000 7f7361a00640 1 --2- 172.21.15.90:0/1082581323 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f735c076950 0x7f735c1a4bf0 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:06.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.352+0000 7f7361a00640 1 --2- 172.21.15.90:0/1082581323 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f735c076030 0x7f735c1a46b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:06.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.352+0000 7f7361a00640 1 -- 172.21.15.90:0/1082581323 >> 172.21.15.90:0/1082581323 conn(0x7f735c0ff7b0 msgr2=0x7f735c101ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:46:06.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.353+0000 7f7361a00640 1 -- 172.21.15.90:0/1082581323 shutdown_connections 2024-04-02T14:46:06.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.353+0000 7f7361a00640 1 -- 172.21.15.90:0/1082581323 wait complete. 2024-04-02T14:46:06.371 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-02T14:46:06.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.638+0000 7f3338e00640 1 -- 172.21.15.90:0/830674901 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3334073ed0 msgr2=0x7f33340742b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:06.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.638+0000 7f3338e00640 1 --2- 172.21.15.90:0/830674901 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3334073ed0 0x7f33340742b0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f3324009900 tx=0x7f332402bcc0 comp rx=0 tx=0).stop 2024-04-02T14:46:06.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.638+0000 7f3338e00640 1 -- 172.21.15.90:0/830674901 shutdown_connections 2024-04-02T14:46:06.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.638+0000 7f3338e00640 1 --2- 172.21.15.90:0/830674901 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f333410e210 0x7f3334110620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:06.634 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.638+0000 7f3338e00640 1 --2- 172.21.15.90:0/830674901 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f33340747f0 0x7f333410dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:06.634 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.638+0000 7f3338e00640 1 --2- 172.21.15.90:0/830674901 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3334073ed0 0x7f33340742b0 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:06.634 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.638+0000 7f3338e00640 1 -- 172.21.15.90:0/830674901 >> 172.21.15.90:0/830674901 conn(0x7f33340fd4a0 msgr2=0x7f33340ff8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:46:06.634 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.639+0000 7f3338e00640 1 -- 172.21.15.90:0/830674901 shutdown_connections 2024-04-02T14:46:06.634 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.639+0000 7f3338e00640 1 -- 172.21.15.90:0/830674901 wait complete. 2024-04-02T14:46:06.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.640+0000 7f3338e00640 1 Processor -- start 2024-04-02T14:46:06.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.643+0000 7f3338e00640 1 -- start start 2024-04-02T14:46:06.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.643+0000 7f3338e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3334073ed0 0x7f333410dbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:06.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.643+0000 7f3338e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f33340747f0 0x7f333410ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:06.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.643+0000 7f3338e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f333410e210 0x7f333410b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:06.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.644+0000 7f3333400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3334073ed0 0x7f333410dbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:06.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.644+0000 7f3333400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3334073ed0 0x7f333410dbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:52592/0 (socket says 172.21.15.90:52592) 2024-04-02T14:46:06.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.644+0000 7f3333400640 1 -- 172.21.15.90:0/3502455251 learned_addr learned my addr 172.21.15.90:0/3502455251 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:46:06.640 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.644+0000 7f3338e00640 1 -- 172.21.15.90:0/3502455251 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f3334114060 con 0x7f333410e210 2024-04-02T14:46:06.640 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.644+0000 7f3338e00640 1 -- 172.21.15.90:0/3502455251 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f3334113ee0 con 0x7f3334073ed0 2024-04-02T14:46:06.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.644+0000 7f3338e00640 1 -- 172.21.15.90:0/3502455251 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f33341141e0 con 0x7f33340747f0 2024-04-02T14:46:06.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.644+0000 7f3332a00640 1 --2- 172.21.15.90:0/3502455251 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f33340747f0 0x7f333410ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:06.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.644+0000 7f3333e00640 1 --2- 172.21.15.90:0/3502455251 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f333410e210 0x7f333410b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:06.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.644+0000 7f3332a00640 1 -- 172.21.15.90:0/3502455251 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3334073ed0 msgr2=0x7f333410dbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:06.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.644+0000 7f3332a00640 1 --2- 172.21.15.90:0/3502455251 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3334073ed0 0x7f333410dbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:06.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.644+0000 7f3332a00640 1 -- 172.21.15.90:0/3502455251 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f333410e210 msgr2=0x7f333410b1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:06.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.644+0000 7f3332a00640 1 --2- 172.21.15.90:0/3502455251 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f333410e210 0x7f333410b1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:06.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.645+0000 7f3332a00640 1 -- 172.21.15.90:0/3502455251 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f333410baa0 con 0x7f33340747f0 2024-04-02T14:46:06.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.645+0000 7f3333400640 1 --2- 172.21.15.90:0/3502455251 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3334073ed0 0x7f333410dbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:46:06.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.645+0000 7f3333e00640 1 --2- 172.21.15.90:0/3502455251 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f333410e210 0x7f333410b1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:46:06.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.645+0000 7f3332a00640 1 --2- 172.21.15.90:0/3502455251 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f33340747f0 0x7f333410ac60 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f332000b3f0 tx=0x7f332000b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:46:06.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.646+0000 7f332be00640 1 -- 172.21.15.90:0/3502455251 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3320015020 con 0x7f33340747f0 2024-04-02T14:46:06.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.646+0000 7f332be00640 1 -- 172.21.15.90:0/3502455251 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3320011d10 con 0x7f33340747f0 2024-04-02T14:46:06.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.646+0000 7f332be00640 1 -- 172.21.15.90:0/3502455251 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f332000f2e0 con 0x7f33340747f0 2024-04-02T14:46:06.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.646+0000 7f3338e00640 1 -- 172.21.15.90:0/3502455251 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f33341aaec0 con 0x7f33340747f0 2024-04-02T14:46:06.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.646+0000 7f3338e00640 1 -- 172.21.15.90:0/3502455251 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f33341ab390 con 0x7f33340747f0 2024-04-02T14:46:06.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.659+0000 7f333d052740 1 Processor -- start 2024-04-02T14:46:06.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.659+0000 7f333d052740 1 -- start start 2024-04-02T14:46:06.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.660+0000 7f333d052740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560df7a66ba0 0x560df7a658f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:06.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.661+0000 7f3333e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560df7a66ba0 0x560df7a658f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:06.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.661+0000 7f3333e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560df7a66ba0 0x560df7a658f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:58366/0 (socket says 172.21.15.90:58366) 2024-04-02T14:46:06.657 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.662+0000 7f3333e00640 1 -- 172.21.15.90:0/388799329 learned_addr learned my addr 172.21.15.90:0/388799329 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:46:06.657 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.662+0000 7f333d052740 1 --2- 172.21.15.90:0/388799329 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560df7a65ec0 0x560df7a66340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:06.657 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.662+0000 7f333d052740 1 --2- 172.21.15.90:0/388799329 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560df7a6f230 0x560df7a71620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:06.658 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.663+0000 7f332be00640 1 -- 172.21.15.90:0/3502455251 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f3320018020 con 0x7f33340747f0 2024-04-02T14:46:06.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.663+0000 7f333d052740 1 -- 172.21.15.90:0/388799329 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x560df79a6400 con 0x560df7a65ec0 2024-04-02T14:46:06.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.663+0000 7f333d052740 1 -- 172.21.15.90:0/388799329 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x560df798af10 con 0x560df7a6f230 2024-04-02T14:46:06.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.663+0000 7f333d052740 1 -- 172.21.15.90:0/388799329 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x560df79ca6f0 con 0x560df7a66ba0 2024-04-02T14:46:06.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.663+0000 7f3332a00640 1 --2- 172.21.15.90:0/388799329 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560df7a6f230 0x560df7a71620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:06.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.664+0000 7f3333400640 1 --2- 172.21.15.90:0/388799329 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560df7a65ec0 0x560df7a66340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:06.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.664+0000 7f3332a00640 1 -- 172.21.15.90:0/388799329 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560df7a66ba0 msgr2=0x560df7a658f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:46:06.660 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.664+0000 7f3332a00640 1 --2- 172.21.15.90:0/388799329 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560df7a66ba0 0x560df7a658f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:06.660 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.664+0000 7f3332a00640 1 -- 172.21.15.90:0/388799329 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560df7a65ec0 msgr2=0x560df7a66340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:46:06.660 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.664+0000 7f3332a00640 1 --2- 172.21.15.90:0/388799329 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560df7a65ec0 0x560df7a66340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:06.660 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.664+0000 7f3332a00640 1 -- 172.21.15.90:0/388799329 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560df76de3f0 con 0x560df7a6f230 2024-04-02T14:46:06.660 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.664+0000 7f3332a00640 1 --2- 172.21.15.90:0/388799329 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560df7a6f230 0x560df7a71620 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f33200cd7b0 tx=0x7f33200d86f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5f39a35e85443ebd server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:46:06.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.665+0000 7f332be00640 1 --2- 172.21.15.90:0/3502455251 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f33180a3da0 0x7f33180a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:06.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.665+0000 7f332be00640 1 -- 172.21.15.90:0/3502455251 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 8550+0+0 (secure 0 0 0) 0x7f33200cff70 con 0x7f33340747f0 2024-04-02T14:46:06.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.666+0000 7f3333e00640 1 --2- 172.21.15.90:0/388799329 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560df7a66ba0 0x560df7a658f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:46:06.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.666+0000 7f3328c00640 1 -- 172.21.15.90:0/388799329 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33200e3070 con 0x560df7a6f230 2024-04-02T14:46:06.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.666+0000 7f3328c00640 1 -- 172.21.15.90:0/388799329 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f33200d0b00 con 0x560df7a6f230 2024-04-02T14:46:06.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.666+0000 7f3328c00640 1 -- 172.21.15.90:0/388799329 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33200de910 con 0x560df7a6f230 2024-04-02T14:46:06.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.668+0000 7f333d052740 1 -- 172.21.15.90:0/388799329 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560df7a6f230 msgr2=0x560df7a71620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:06.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.668+0000 7f333d052740 1 --2- 172.21.15.90:0/388799329 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560df7a6f230 0x560df7a71620 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f33200cd7b0 tx=0x7f33200d86f0 comp rx=0 tx=0).stop 2024-04-02T14:46:06.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.668+0000 7f3333e00640 1 --2- 172.21.15.90:0/3502455251 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f33180a3da0 0x7f33180a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:06.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.670+0000 7f333d052740 1 -- 172.21.15.90:0/388799329 shutdown_connections 2024-04-02T14:46:06.666 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.670+0000 7f3333e00640 1 --2- 172.21.15.90:0/3502455251 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f33180a3da0 0x7f33180a6260 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f332c0115d0 tx=0x7f332c009500 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:46:06.666 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.670+0000 7f333d052740 1 --2- 172.21.15.90:0/388799329 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560df7a6f230 0x560df7a71620 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:06.666 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.671+0000 7f333d052740 1 --2- 172.21.15.90:0/388799329 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560df7a65ec0 0x560df7a66340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:06.666 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.671+0000 7f333d052740 1 --2- 172.21.15.90:0/388799329 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560df7a66ba0 0x560df7a658f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:06.666 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.671+0000 7f333d052740 1 -- 172.21.15.90:0/388799329 >> 172.21.15.90:0/388799329 conn(0x560df7a2ca50 msgr2=0x560df7a60910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:46:06.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.671+0000 7f333d052740 1 -- 172.21.15.90:0/388799329 shutdown_connections 2024-04-02T14:46:06.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.671+0000 7f333d052740 1 -- 172.21.15.90:0/388799329 wait complete. 2024-04-02T14:46:06.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.672+0000 7f333d052740 1 -- start start 2024-04-02T14:46:06.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.672+0000 7f333d052740 1 Processor -- start 2024-04-02T14:46:06.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.673+0000 7f333d052740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560df7a65ec0 0x560df7a6c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:06.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.673+0000 7f333d052740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560df7a66ba0 0x560df7a6c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:06.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.673+0000 7f333d052740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560df7a6f230 0x560df7a6cd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:06.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.674+0000 7f3333400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560df7a65ec0 0x560df7a6c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:06.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.674+0000 7f3333400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560df7a65ec0 0x560df7a6c280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:52624/0 (socket says 172.21.15.90:52624) 2024-04-02T14:46:06.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.674+0000 7f3333400640 1 -- 172.21.15.90:0/3082473423 learned_addr learned my addr 172.21.15.90:0/3082473423 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:46:06.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.674+0000 7f3333e00640 1 --2- 172.21.15.90:0/3082473423 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560df7a6f230 0x560df7a6cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:06.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.674+0000 7f333d052740 1 -- 172.21.15.90:0/3082473423 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x560df79c66a0 con 0x560df7a66ba0 2024-04-02T14:46:06.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.674+0000 7f333d052740 1 -- 172.21.15.90:0/3082473423 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x560df7972980 con 0x560df7a65ec0 2024-04-02T14:46:06.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.674+0000 7f3332a00640 1 --2- 172.21.15.90:0/3082473423 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560df7a66ba0 0x560df7a6c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:06.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.674+0000 7f333d052740 1 -- 172.21.15.90:0/3082473423 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x560df7985cb0 con 0x560df7a6f230 2024-04-02T14:46:06.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.674+0000 7f333d052740 10 client.0 _ll_register_callbacks cb 0x560df79856a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:46:06.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.675+0000 7f3333400640 1 -- 172.21.15.90:0/3082473423 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560df7a6f230 msgr2=0x560df7a6cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:46:06.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.675+0000 7f3333400640 1 --2- 172.21.15.90:0/3082473423 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560df7a6f230 0x560df7a6cd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:06.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.675+0000 7f3333400640 1 -- 172.21.15.90:0/3082473423 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560df7a66ba0 msgr2=0x560df7a6c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:46:06.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.675+0000 7f3333400640 1 --2- 172.21.15.90:0/3082473423 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560df7a66ba0 0x560df7a6c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:06.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.675+0000 7f3333400640 1 -- 172.21.15.90:0/3082473423 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560df77a87d0 con 0x560df7a65ec0 2024-04-02T14:46:06.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.676+0000 7f3333400640 1 --2- 172.21.15.90:0/3082473423 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560df7a65ec0 0x560df7a6c280 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f3324004630 tx=0x7f3324005df0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=78ecfc7839b4689e server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:46:06.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.676+0000 7f331d600640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:46:06.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.676+0000 7f331d600640 1 -- 172.21.15.90:0/3082473423 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3324005030 con 0x560df7a65ec0 2024-04-02T14:46:06.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.676+0000 7f331d600640 1 -- 172.21.15.90:0/3082473423 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8550+0+0 (secure 0 0 0) 0x7f33240051d0 con 0x560df7a65ec0 2024-04-02T14:46:06.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.676+0000 7f333d052740 1 -- 172.21.15.90:0/3082473423 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f33200d0b00 con 0x560df7a65ec0 2024-04-02T14:46:06.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.677+0000 7f331d600640 1 -- 172.21.15.90:0/3082473423 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f33240053d0 con 0x560df7a65ec0 2024-04-02T14:46:06.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.677+0000 7f333d052740 10 client.28967 fetch_fsmap learned FSMap version 27 2024-04-02T14:46:06.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.677+0000 7f333d052740 1 -- 172.21.15.90:0/3082473423 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x560df7a7df30 con 0x560df7a65ec0 2024-04-02T14:46:06.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.679+0000 7f331d600640 1 -- 172.21.15.90:0/3082473423 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f33200d0b00 con 0x560df7a65ec0 2024-04-02T14:46:06.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.679+0000 7f331d600640 1 -- 172.21.15.90:0/3082473423 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f33240038a0 con 0x560df7a65ec0 2024-04-02T14:46:06.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.681+0000 7f333d052740 10 client.28967 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:46:06.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.681+0000 7f333d052740 10 client.28967 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2024-04-02T14:46:06.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.681+0000 7f333d052740 20 client.28967 populate_metadata read hostname 'smithi090' 2024-04-02T14:46:06.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.681+0000 7f333d052740 1 --2- 172.21.15.90:0/3082473423 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x560df7a7f390 0x560df7a817a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:06.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.681+0000 7f333d052740 4 client.28967 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:46:06.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.681+0000 7f333d052740 1 -- 172.21.15.90:0/3082473423 --> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560df798a040 con 0x560df7a7f390 2024-04-02T14:46:06.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.681+0000 7f3332a00640 1 --2- 172.21.15.90:0/3082473423 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x560df7a7f390 0x560df7a817a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:06.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.683+0000 7f3332a00640 1 --2- 172.21.15.90:0/3082473423 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x560df7a7f390 0x560df7a817a0 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=3f47f2fa1c23301f server_cookie=eb602e6ec193bb03 in_seq=0 out_seq=0 2024-04-02T14:46:06.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.683+0000 7f331d600640 10 client.28967 ms_handle_connect on v2:172.21.15.90:6838/1498915870 2024-04-02T14:46:06.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.685+0000 7f331d600640 1 -- 172.21.15.90:0/3082473423 <== mds.1 v2:172.21.15.90:6838/1498915870 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x7f332007be50 con 0x560df7a7f390 2024-04-02T14:46:06.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.685+0000 7f331d600640 10 client.28967 handle_command_reply: tid=0 2024-04-02T14:46:06.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.685+0000 7f333d052740 1 client.28967 shutdown 2024-04-02T14:46:06.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.686+0000 7f333d052740 1 -- 172.21.15.90:0/3082473423 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560df7a65ec0 msgr2=0x560df7a6c280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:06.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.686+0000 7f333d052740 1 --2- 172.21.15.90:0/3082473423 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560df7a65ec0 0x560df7a6c280 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f3324004630 tx=0x7f3324005df0 comp rx=0 tx=0).stop 2024-04-02T14:46:06.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.686+0000 7f333d052740 1 -- 172.21.15.90:0/3082473423 shutdown_connections 2024-04-02T14:46:06.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.686+0000 7f333d052740 1 --2- 172.21.15.90:0/3082473423 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x560df7a7f390 0x560df7a817a0 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:06.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.686+0000 7f333d052740 1 --2- 172.21.15.90:0/3082473423 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560df7a6f230 0x560df7a6cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:06.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.686+0000 7f333d052740 1 --2- 172.21.15.90:0/3082473423 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560df7a66ba0 0x560df7a6c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:06.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.686+0000 7f333d052740 1 --2- 172.21.15.90:0/3082473423 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560df7a65ec0 0x560df7a6c280 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:06.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.686+0000 7f331d600640 0 client.28967 ms_handle_reset on v2:172.21.15.90:6838/1498915870 2024-04-02T14:46:06.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.686+0000 7f333d052740 1 -- 172.21.15.90:0/3082473423 >> 172.21.15.90:0/3082473423 conn(0x560df7a2ca50 msgr2=0x560df7a62660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:46:06.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.686+0000 7f333d052740 1 -- 172.21.15.90:0/3082473423 shutdown_connections 2024-04-02T14:46:06.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.686+0000 7f333d052740 1 -- 172.21.15.90:0/3082473423 wait complete. 2024-04-02T14:46:06.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.687+0000 7f333d052740 20 client.28967 trim_cache size 0 max 16384 2024-04-02T14:46:06.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.721+0000 7f333d052740 1 Processor -- start 2024-04-02T14:46:06.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.721+0000 7f333d052740 1 -- start start 2024-04-02T14:46:06.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.722+0000 7f333d052740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560df7a65ec0 0x560df7a6e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:06.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.722+0000 7f333d052740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560df7a66ba0 0x560df7a7ab20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:06.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.722+0000 7f333d052740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560df7a6f230 0x560df7a5f970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:06.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.722+0000 7f333d052740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f33341b3ed0 con 0x560df7a6f230 2024-04-02T14:46:06.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.722+0000 7f333d052740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f33341b5a10 con 0x560df7a66ba0 2024-04-02T14:46:06.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.722+0000 7f333d052740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f33341b45a0 con 0x560df7a65ec0 2024-04-02T14:46:06.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.722+0000 7f3332a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560df7a66ba0 0x560df7a7ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:06.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.723+0000 7f3332a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560df7a66ba0 0x560df7a7ab20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:52634/0 (socket says 172.21.15.90:52634) 2024-04-02T14:46:06.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.723+0000 7f3332a00640 1 -- 172.21.15.90:0/3382149467 learned_addr learned my addr 172.21.15.90:0/3382149467 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:46:06.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.723+0000 7f3333400640 1 --2- 172.21.15.90:0/3382149467 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560df7a65ec0 0x560df7a6e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:06.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.723+0000 7f3333400640 1 -- 172.21.15.90:0/3382149467 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560df7a66ba0 msgr2=0x560df7a7ab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:46:06.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.723+0000 7f3333400640 1 --2- 172.21.15.90:0/3382149467 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560df7a66ba0 0x560df7a7ab20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:06.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.723+0000 7f3333400640 1 -- 172.21.15.90:0/3382149467 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560df7a6f230 msgr2=0x560df7a5f970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:46:06.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.723+0000 7f3333400640 1 --2- 172.21.15.90:0/3382149467 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560df7a6f230 0x560df7a5f970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:06.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.723+0000 7f3333400640 1 -- 172.21.15.90:0/3382149467 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560df7a64ef0 con 0x560df7a65ec0 2024-04-02T14:46:06.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.724+0000 7f3333400640 1 --2- 172.21.15.90:0/3382149467 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560df7a65ec0 0x560df7a6e9b0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f3324000c00 tx=0x7f332402bcc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=41824f1d1e27ff2d server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:46:06.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.724+0000 7f331fe00640 1 -- 172.21.15.90:0/3382149467 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3324054020 con 0x560df7a65ec0 2024-04-02T14:46:06.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.724+0000 7f331fe00640 1 -- 172.21.15.90:0/3382149467 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f332402cca0 con 0x560df7a65ec0 2024-04-02T14:46:06.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.724+0000 7f331fe00640 1 -- 172.21.15.90:0/3382149467 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3324034990 con 0x560df7a65ec0 2024-04-02T14:46:06.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.725+0000 7f333d052740 1 -- 172.21.15.90:0/3382149467 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560df7a65ec0 msgr2=0x560df7a6e9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:06.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.725+0000 7f333d052740 1 --2- 172.21.15.90:0/3382149467 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560df7a65ec0 0x560df7a6e9b0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f3324000c00 tx=0x7f332402bcc0 comp rx=0 tx=0).stop 2024-04-02T14:46:06.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.725+0000 7f333d052740 1 -- 172.21.15.90:0/3382149467 shutdown_connections 2024-04-02T14:46:06.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.725+0000 7f333d052740 1 --2- 172.21.15.90:0/3382149467 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560df7a6f230 0x560df7a5f970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:06.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.725+0000 7f333d052740 1 --2- 172.21.15.90:0/3382149467 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560df7a66ba0 0x560df7a7ab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:06.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.725+0000 7f333d052740 1 --2- 172.21.15.90:0/3382149467 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560df7a65ec0 0x560df7a6e9b0 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:06.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.725+0000 7f333d052740 1 -- 172.21.15.90:0/3382149467 >> 172.21.15.90:0/3382149467 conn(0x560df7a2ca50 msgr2=0x560df7a5f260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:46:06.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.726+0000 7f333d052740 1 -- 172.21.15.90:0/3382149467 shutdown_connections 2024-04-02T14:46:06.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.726+0000 7f333d052740 1 -- 172.21.15.90:0/3382149467 wait complete. 2024-04-02T14:46:06.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.727+0000 7f333d052740 1 -- start start 2024-04-02T14:46:06.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.727+0000 7f333d052740 1 Processor -- start 2024-04-02T14:46:06.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.729+0000 7f333d052740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560df7a65ec0 0x560df7a77500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:06.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.729+0000 7f333d052740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560df7a66ba0 0x560df7a698c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:06.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.729+0000 7f333d052740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560df7a6f230 0x560df7a69f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:06.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.729+0000 7f3332a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560df7a66ba0 0x560df7a698c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:06.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.729+0000 7f3332a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560df7a66ba0 0x560df7a698c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:58394/0 (socket says 172.21.15.90:58394) 2024-04-02T14:46:06.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.730+0000 7f3332a00640 1 -- 172.21.15.90:0/3033613486 learned_addr learned my addr 172.21.15.90:0/3033613486 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:46:06.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.729+0000 7f333d052740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f33341b5340 con 0x560df7a6f230 2024-04-02T14:46:06.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.730+0000 7f333d052740 1 -- 172.21.15.90:0/3033613486 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f33341b4c70 con 0x560df7a65ec0 2024-04-02T14:46:06.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.730+0000 7f3333e00640 1 --2- 172.21.15.90:0/3033613486 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560df7a6f230 0x560df7a69f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:06.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.730+0000 7f3333400640 1 --2- 172.21.15.90:0/3033613486 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560df7a65ec0 0x560df7a77500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:06.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.730+0000 7f333d052740 1 -- 172.21.15.90:0/3033613486 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f33341b1ed0 con 0x560df7a66ba0 2024-04-02T14:46:06.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.730+0000 7f333d052740 10 client.0 _ll_register_callbacks cb 0x7f3324004470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:46:06.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.731+0000 7f3332a00640 1 -- 172.21.15.90:0/3033613486 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560df7a65ec0 msgr2=0x560df7a77500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:46:06.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.731+0000 7f3332a00640 1 --2- 172.21.15.90:0/3033613486 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560df7a65ec0 0x560df7a77500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:06.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.731+0000 7f3332a00640 1 -- 172.21.15.90:0/3033613486 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560df7a6f230 msgr2=0x560df7a69f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:46:06.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.731+0000 7f3332a00640 1 --2- 172.21.15.90:0/3033613486 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560df7a6f230 0x560df7a69f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:06.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.731+0000 7f3332a00640 1 -- 172.21.15.90:0/3033613486 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560df7a70360 con 0x560df7a66ba0 2024-04-02T14:46:06.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.732+0000 7f3332a00640 1 --2- 172.21.15.90:0/3033613486 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560df7a66ba0 0x560df7a698c0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f332000bdb0 tx=0x7f3320000bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=761a9341bd0ce41 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:46:06.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.732+0000 7f331d600640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:46:06.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.733+0000 7f331d600640 1 -- 172.21.15.90:0/3033613486 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33200992d0 con 0x560df7a66ba0 2024-04-02T14:46:06.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.733+0000 7f331d600640 1 -- 172.21.15.90:0/3033613486 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 8594+0+0 (secure 0 0 0) 0x7f332009b0d0 con 0x560df7a66ba0 2024-04-02T14:46:06.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.733+0000 7f333d052740 1 -- 172.21.15.90:0/3033613486 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f332402cca0 con 0x560df7a66ba0 2024-04-02T14:46:06.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.733+0000 7f331d600640 1 -- 172.21.15.90:0/3033613486 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f332009c990 con 0x560df7a66ba0 2024-04-02T14:46:06.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.733+0000 7f333d052740 10 client.28786 fetch_fsmap learned FSMap version 27 2024-04-02T14:46:06.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.733+0000 7f333d052740 1 -- 172.21.15.90:0/3033613486 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x560df7ae5e10 con 0x560df7a66ba0 2024-04-02T14:46:06.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.734+0000 7f331d600640 1 -- 172.21.15.90:0/3033613486 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f332402cca0 con 0x560df7a66ba0 2024-04-02T14:46:06.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.734+0000 7f331d600640 1 -- 172.21.15.90:0/3033613486 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f33200996a0 con 0x560df7a66ba0 2024-04-02T14:46:06.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.734+0000 7f333d052740 10 client.28786 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:46:06.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.734+0000 7f333d052740 10 client.28786 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2024-04-02T14:46:06.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.734+0000 7f333d052740 20 client.28786 populate_metadata read hostname 'smithi090' 2024-04-02T14:46:06.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.734+0000 7f333d052740 1 --2- 172.21.15.90:0/3033613486 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x560df7a7f390 0x560df7ae81c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:06.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.734+0000 7f333d052740 4 client.28786 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-02T14:46:06.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.734+0000 7f333d052740 1 -- 172.21.15.90:0/3033613486 --> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] -- command(tid 0: {"prefix": "damage ls"}) -- 0x560df79d7c30 con 0x560df7a7f390 2024-04-02T14:46:06.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.736+0000 7f3333400640 1 --2- 172.21.15.90:0/3033613486 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x560df7a7f390 0x560df7ae81c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:06.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.736+0000 7f3333400640 1 --2- 172.21.15.90:0/3033613486 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x560df7a7f390 0x560df7ae81c0 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=f2403993640ae9c3 server_cookie=4a833484adce4ed4 in_seq=0 out_seq=0 2024-04-02T14:46:06.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.736+0000 7f331d600640 10 client.28786 ms_handle_connect on v2:172.21.15.90:6838/1498915870 2024-04-02T14:46:06.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.737+0000 7f331d600640 1 -- 172.21.15.90:0/3033613486 <== mds.1 v2:172.21.15.90:6838/1498915870 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x560df79d7c30 con 0x560df7a7f390 2024-04-02T14:46:06.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.737+0000 7f331d600640 10 client.28786 handle_command_reply: tid=0 2024-04-02T14:46:06.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.737+0000 7f333d052740 1 client.28786 shutdown 2024-04-02T14:46:06.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.737+0000 7f333d052740 1 -- 172.21.15.90:0/3033613486 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560df7a66ba0 msgr2=0x560df7a698c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:06.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.737+0000 7f333d052740 1 --2- 172.21.15.90:0/3033613486 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560df7a66ba0 0x560df7a698c0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f332000bdb0 tx=0x7f3320000bc0 comp rx=0 tx=0).stop 2024-04-02T14:46:06.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.738+0000 7f333d052740 1 -- 172.21.15.90:0/3033613486 shutdown_connections 2024-04-02T14:46:06.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.738+0000 7f333d052740 1 --2- 172.21.15.90:0/3033613486 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x560df7a7f390 0x560df7ae81c0 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:06.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.738+0000 7f333d052740 1 --2- 172.21.15.90:0/3033613486 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560df7a6f230 0x560df7a69f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:06.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.738+0000 7f333d052740 1 --2- 172.21.15.90:0/3033613486 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560df7a66ba0 0x560df7a698c0 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:06.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.738+0000 7f333d052740 1 --2- 172.21.15.90:0/3033613486 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560df7a65ec0 0x560df7a77500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:06.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.738+0000 7f331d600640 0 client.28786 ms_handle_reset on v2:172.21.15.90:6838/1498915870 2024-04-02T14:46:06.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.738+0000 7f333d052740 1 -- 172.21.15.90:0/3033613486 >> 172.21.15.90:0/3033613486 conn(0x560df7a2ca50 msgr2=0x560df7a5faf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:46:06.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.738+0000 7f333d052740 1 -- 172.21.15.90:0/3033613486 shutdown_connections 2024-04-02T14:46:06.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.738+0000 7f333d052740 1 -- 172.21.15.90:0/3033613486 wait complete. 2024-04-02T14:46:06.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.739+0000 7f333d052740 20 client.28786 trim_cache size 0 max 16384 2024-04-02T14:46:06.734 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-02T14:46:06.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.740+0000 7f3338e00640 1 -- 172.21.15.90:0/3502455251 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f33180a3da0 msgr2=0x7f33180a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:06.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.740+0000 7f3338e00640 1 --2- 172.21.15.90:0/3502455251 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f33180a3da0 0x7f33180a6260 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f332c0115d0 tx=0x7f332c009500 comp rx=0 tx=0).stop 2024-04-02T14:46:06.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.740+0000 7f3338e00640 1 -- 172.21.15.90:0/3502455251 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f33340747f0 msgr2=0x7f333410ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:06.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.740+0000 7f3338e00640 1 --2- 172.21.15.90:0/3502455251 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f33340747f0 0x7f333410ac60 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f332000b3f0 tx=0x7f332000b8b0 comp rx=0 tx=0).stop 2024-04-02T14:46:06.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.740+0000 7f3338e00640 1 -- 172.21.15.90:0/3502455251 shutdown_connections 2024-04-02T14:46:06.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.740+0000 7f3338e00640 1 --2- 172.21.15.90:0/3502455251 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f33180a3da0 0x7f33180a6260 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:06.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.740+0000 7f3338e00640 1 --2- 172.21.15.90:0/3502455251 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f333410e210 0x7f333410b1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:06.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.740+0000 7f3338e00640 1 --2- 172.21.15.90:0/3502455251 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f33340747f0 0x7f333410ac60 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:06.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.740+0000 7f3338e00640 1 --2- 172.21.15.90:0/3502455251 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3334073ed0 0x7f333410dbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:06.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.740+0000 7f3338e00640 1 -- 172.21.15.90:0/3502455251 >> 172.21.15.90:0/3502455251 conn(0x7f33340fd4a0 msgr2=0x7f33340fe110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:46:06.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.740+0000 7f3338e00640 1 -- 172.21.15.90:0/3502455251 shutdown_connections 2024-04-02T14:46:06.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.740+0000 7f3338e00640 1 -- 172.21.15.90:0/3502455251 wait complete. 2024-04-02T14:46:06.760 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-02T14:46:06.825 INFO:teuthology.orchestra.run.smithi090.stderr:progress: 635.0 s, 533.8 tps, lat 186.569 ms stddev 355.202 2024-04-02T14:46:06.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.968+0000 7f679f200640 1 -- 172.21.15.90:0/3043388085 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f67a0102b30 msgr2=0x7f67a0102f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:06.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.968+0000 7f679f200640 1 --2- 172.21.15.90:0/3043388085 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f67a0102b30 0x7f67a0102f10 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f6790009900 tx=0x7f679002bcd0 comp rx=0 tx=0).stop 2024-04-02T14:46:06.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.968+0000 7f679f200640 1 -- 172.21.15.90:0/3043388085 shutdown_connections 2024-04-02T14:46:06.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.968+0000 7f679f200640 1 --2- 172.21.15.90:0/3043388085 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f67a010bf50 0x7f67a010e3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:06.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.968+0000 7f679f200640 1 --2- 172.21.15.90:0/3043388085 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f67a0103450 0x7f67a010b930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:06.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.968+0000 7f679f200640 1 --2- 172.21.15.90:0/3043388085 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f67a0102b30 0x7f67a0102f10 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:06.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.968+0000 7f679f200640 1 -- 172.21.15.90:0/3043388085 >> 172.21.15.90:0/3043388085 conn(0x7f67a00fc350 msgr2=0x7f67a00fe770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:46:06.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.968+0000 7f679f200640 1 -- 172.21.15.90:0/3043388085 shutdown_connections 2024-04-02T14:46:06.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.968+0000 7f679f200640 1 -- 172.21.15.90:0/3043388085 wait complete. 2024-04-02T14:46:06.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.969+0000 7f679f200640 1 Processor -- start 2024-04-02T14:46:06.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.969+0000 7f679f200640 1 -- start start 2024-04-02T14:46:06.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.970+0000 7f679f200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f67a0102b30 0x7f67a019a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:06.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.970+0000 7f679f200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f67a0103450 0x7f67a019a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:06.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.970+0000 7f679f200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f67a010bf50 0x7f67a019ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:06.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.970+0000 7f679f200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f67a0111da0 con 0x7f67a010bf50 2024-04-02T14:46:06.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.970+0000 7f679f200640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f67a0111c20 con 0x7f67a0102b30 2024-04-02T14:46:06.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.970+0000 7f679f200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f67a0111f20 con 0x7f67a0103450 2024-04-02T14:46:06.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.970+0000 7f679d400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f67a0103450 0x7f67a019a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:06.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.970+0000 7f679d400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f67a0103450 0x7f67a019a9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:58418/0 (socket says 172.21.15.90:58418) 2024-04-02T14:46:06.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.970+0000 7f679d400640 1 -- 172.21.15.90:0/2570259483 learned_addr learned my addr 172.21.15.90:0/2570259483 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:46:06.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.970+0000 7f679e800640 1 --2- 172.21.15.90:0/2570259483 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f67a010bf50 0x7f67a019ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:06.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.970+0000 7f679de00640 1 --2- 172.21.15.90:0/2570259483 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f67a0102b30 0x7f67a019a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:06.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.971+0000 7f679d400640 1 -- 172.21.15.90:0/2570259483 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f67a0102b30 msgr2=0x7f67a019a4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:06.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.971+0000 7f679d400640 1 --2- 172.21.15.90:0/2570259483 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f67a0102b30 0x7f67a019a4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:06.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.971+0000 7f679d400640 1 -- 172.21.15.90:0/2570259483 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f67a010bf50 msgr2=0x7f67a019ed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:06.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.971+0000 7f679d400640 1 --2- 172.21.15.90:0/2570259483 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f67a010bf50 0x7f67a019ed70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:06.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.971+0000 7f679d400640 1 -- 172.21.15.90:0/2570259483 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67a019f4a0 con 0x7f67a0103450 2024-04-02T14:46:06.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.971+0000 7f679d400640 1 --2- 172.21.15.90:0/2570259483 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f67a0103450 0x7f67a019a9f0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f678c00e9f0 tx=0x7f678c00edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:46:06.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.972+0000 7f6796a00640 1 -- 172.21.15.90:0/2570259483 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f678c00cab0 con 0x7f67a0103450 2024-04-02T14:46:06.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.972+0000 7f679f200640 1 -- 172.21.15.90:0/2570259483 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67a019f790 con 0x7f67a0103450 2024-04-02T14:46:06.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.972+0000 7f6796a00640 1 -- 172.21.15.90:0/2570259483 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f678c00cc50 con 0x7f67a0103450 2024-04-02T14:46:06.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.972+0000 7f6796a00640 1 -- 172.21.15.90:0/2570259483 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f678c0187e0 con 0x7f67a0103450 2024-04-02T14:46:06.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.972+0000 7f679f200640 1 -- 172.21.15.90:0/2570259483 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67a01a7020 con 0x7f67a0103450 2024-04-02T14:46:06.970 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.975+0000 7f67a7797740 1 Processor -- start 2024-04-02T14:46:06.970 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.975+0000 7f6796a00640 1 -- 172.21.15.90:0/2570259483 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f678c018b90 con 0x7f67a0103450 2024-04-02T14:46:06.970 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.975+0000 7f67a7797740 1 -- start start 2024-04-02T14:46:06.972 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.975+0000 7f67a7797740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5563d9c27ba0 0x5563d9c268f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:06.972 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.975+0000 7f67a7797740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5563d9c26ec0 0x5563d9c27340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:06.972 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.976+0000 7f67a7797740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5563d9c30230 0x5563d9c32620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:06.972 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.976+0000 7f67a7797740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5563d9b67400 con 0x5563d9c27ba0 2024-04-02T14:46:06.972 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.976+0000 7f67a7797740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x5563d9b4bf10 con 0x5563d9c30230 2024-04-02T14:46:06.972 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.976+0000 7f67a7797740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5563d9b8b6f0 con 0x5563d9c26ec0 2024-04-02T14:46:06.972 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.976+0000 7f679de00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5563d9c26ec0 0x5563d9c27340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:06.972 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.976+0000 7f679e800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5563d9c27ba0 0x5563d9c268f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:06.972 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.976+0000 7f679de00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5563d9c26ec0 0x5563d9c27340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:58422/0 (socket says 172.21.15.90:58422) 2024-04-02T14:46:06.972 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.976+0000 7f679de00640 1 -- 172.21.15.90:0/2765964651 learned_addr learned my addr 172.21.15.90:0/2765964651 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:46:06.972 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.976+0000 7f679d400640 1 --2- 172.21.15.90:0/2765964651 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5563d9c30230 0x5563d9c32620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:06.972 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.976+0000 7f679de00640 1 -- 172.21.15.90:0/2765964651 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5563d9c30230 msgr2=0x5563d9c32620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:46:06.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.976+0000 7f679de00640 1 --2- 172.21.15.90:0/2765964651 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5563d9c30230 0x5563d9c32620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:06.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.976+0000 7f679de00640 1 -- 172.21.15.90:0/2765964651 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5563d9c27ba0 msgr2=0x5563d9c268f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:46:06.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.976+0000 7f679de00640 1 --2- 172.21.15.90:0/2765964651 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5563d9c27ba0 0x5563d9c268f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:06.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.977+0000 7f679de00640 1 -- 172.21.15.90:0/2765964651 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5563d989f3f0 con 0x5563d9c26ec0 2024-04-02T14:46:06.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.977+0000 7f679e800640 1 --2- 172.21.15.90:0/2765964651 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5563d9c27ba0 0x5563d9c268f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:46:06.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.978+0000 7f679de00640 1 --2- 172.21.15.90:0/2765964651 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5563d9c26ec0 0x5563d9c27340 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f6790000c00 tx=0x7f6790009560 comp rx=0 tx=0).ready entity=mon.1 client_cookie=27def3681d014042 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:46:06.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.978+0000 7f6796a00640 1 --2- 172.21.15.90:0/2570259483 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f67840a3d20 0x7f67840a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:06.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.978+0000 7f678b000640 1 -- 172.21.15.90:0/2765964651 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6790043070 con 0x5563d9c26ec0 2024-04-02T14:46:06.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.978+0000 7f678b000640 1 -- 172.21.15.90:0/2765964651 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f679002c6f0 con 0x5563d9c26ec0 2024-04-02T14:46:06.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.978+0000 7f678b000640 1 -- 172.21.15.90:0/2765964651 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6790004a60 con 0x5563d9c26ec0 2024-04-02T14:46:06.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.978+0000 7f6796a00640 1 -- 172.21.15.90:0/2570259483 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 8594+0+0 (secure 0 0 0) 0x7f678c0d0d20 con 0x7f67a0103450 2024-04-02T14:46:06.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.979+0000 7f679e800640 1 --2- 172.21.15.90:0/2570259483 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f67840a3d20 0x7f67840a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:06.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.979+0000 7f67a7797740 1 -- 172.21.15.90:0/2765964651 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5563d9c26ec0 msgr2=0x5563d9c27340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:06.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.979+0000 7f67a7797740 1 --2- 172.21.15.90:0/2765964651 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5563d9c26ec0 0x5563d9c27340 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f6790000c00 tx=0x7f6790009560 comp rx=0 tx=0).stop 2024-04-02T14:46:06.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.980+0000 7f67a7797740 1 -- 172.21.15.90:0/2765964651 shutdown_connections 2024-04-02T14:46:06.976 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.980+0000 7f67a7797740 1 --2- 172.21.15.90:0/2765964651 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5563d9c30230 0x5563d9c32620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:06.976 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.980+0000 7f67a7797740 1 --2- 172.21.15.90:0/2765964651 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5563d9c26ec0 0x5563d9c27340 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:06.976 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.980+0000 7f67a7797740 1 --2- 172.21.15.90:0/2765964651 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5563d9c27ba0 0x5563d9c268f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:06.976 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.980+0000 7f67a7797740 1 -- 172.21.15.90:0/2765964651 >> 172.21.15.90:0/2765964651 conn(0x5563d9beda50 msgr2=0x5563d9c21910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:46:06.976 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.980+0000 7f679e800640 1 --2- 172.21.15.90:0/2570259483 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f67840a3d20 0x7f67840a61e0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f6798005fd0 tx=0x7f6798009500 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:46:06.976 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.980+0000 7f67a7797740 1 -- 172.21.15.90:0/2765964651 shutdown_connections 2024-04-02T14:46:06.976 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.980+0000 7f67a7797740 1 -- 172.21.15.90:0/2765964651 wait complete. 2024-04-02T14:46:06.976 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.980+0000 7f67a7797740 1 -- start start 2024-04-02T14:46:06.976 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.981+0000 7f67a7797740 1 Processor -- start 2024-04-02T14:46:06.979 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.981+0000 7f67a7797740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5563d9c26ec0 0x5563d9c2d280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:06.979 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.981+0000 7f67a7797740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5563d9c27ba0 0x5563d9c2d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:06.979 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.981+0000 7f67a7797740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5563d9c30230 0x5563d9c2dd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:06.979 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.981+0000 7f67a7797740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5563d9b876a0 con 0x5563d9c26ec0 2024-04-02T14:46:06.979 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.981+0000 7f67a7797740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x5563d9b33980 con 0x5563d9c27ba0 2024-04-02T14:46:06.979 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.981+0000 7f67a7797740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5563d9b46cb0 con 0x5563d9c30230 2024-04-02T14:46:06.979 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.981+0000 7f67a7797740 10 client.0 _ll_register_callbacks cb 0x5563d9b466a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:46:06.979 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.981+0000 7f679e800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5563d9c30230 0x5563d9c2dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:06.979 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.981+0000 7f679d400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5563d9c27ba0 0x5563d9c2d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:06.980 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.981+0000 7f679e800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5563d9c30230 0x5563d9c2dd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:58438/0 (socket says 172.21.15.90:58438) 2024-04-02T14:46:06.980 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.981+0000 7f679e800640 1 -- 172.21.15.90:0/1636201791 learned_addr learned my addr 172.21.15.90:0/1636201791 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:46:06.980 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.981+0000 7f679d400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5563d9c27ba0 0x5563d9c2d7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:52676/0 (socket says 172.21.15.90:52676) 2024-04-02T14:46:06.980 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.982+0000 7f679de00640 1 --2- 172.21.15.90:0/1636201791 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5563d9c26ec0 0x5563d9c2d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:06.980 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.982+0000 7f679e800640 1 -- 172.21.15.90:0/1636201791 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5563d9c27ba0 msgr2=0x5563d9c2d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:46:06.980 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.982+0000 7f679e800640 1 --2- 172.21.15.90:0/1636201791 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5563d9c27ba0 0x5563d9c2d7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:06.980 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.982+0000 7f679e800640 1 -- 172.21.15.90:0/1636201791 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5563d9c26ec0 msgr2=0x5563d9c2d280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:46:06.980 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.982+0000 7f679e800640 1 --2- 172.21.15.90:0/1636201791 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5563d9c26ec0 0x5563d9c2d280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:06.980 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.982+0000 7f679e800640 1 -- 172.21.15.90:0/1636201791 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5563d99697d0 con 0x5563d9c30230 2024-04-02T14:46:06.980 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.982+0000 7f679e800640 1 --2- 172.21.15.90:0/1636201791 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5563d9c30230 0x5563d9c2dd00 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f6798009b80 tx=0x7f6798004ab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e689ab011650afe6 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:46:06.980 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.982+0000 7f677f400640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:46:06.980 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.982+0000 7f677f400640 1 -- 172.21.15.90:0/1636201791 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6798003b80 con 0x5563d9c30230 2024-04-02T14:46:06.980 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.983+0000 7f677f400640 1 -- 172.21.15.90:0/1636201791 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 8594+0+0 (secure 0 0 0) 0x7f6798003d20 con 0x5563d9c30230 2024-04-02T14:46:06.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.983+0000 7f67a7797740 1 -- 172.21.15.90:0/1636201791 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f679002c6f0 con 0x5563d9c30230 2024-04-02T14:46:06.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.983+0000 7f677f400640 1 -- 172.21.15.90:0/1636201791 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f6798004050 con 0x5563d9c30230 2024-04-02T14:46:06.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.983+0000 7f67a7797740 10 client.28798 fetch_fsmap learned FSMap version 27 2024-04-02T14:46:06.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.983+0000 7f67a7797740 1 -- 172.21.15.90:0/1636201791 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5563d9c3f190 con 0x5563d9c30230 2024-04-02T14:46:06.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.983+0000 7f677f400640 1 -- 172.21.15.90:0/1636201791 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f679002c6f0 con 0x5563d9c30230 2024-04-02T14:46:06.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.983+0000 7f677f400640 1 -- 172.21.15.90:0/1636201791 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f6798027020 con 0x5563d9c30230 2024-04-02T14:46:06.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.984+0000 7f67a7797740 10 client.28798 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:46:06.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.984+0000 7f67a7797740 10 client.28798 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:46:06.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.984+0000 7f67a7797740 20 client.28798 populate_metadata read hostname 'smithi090' 2024-04-02T14:46:06.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.984+0000 7f67a7797740 1 --2- 172.21.15.90:0/1636201791 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5563d9c40650 0x5563d9c42ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:06.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.984+0000 7f67a7797740 4 client.28798 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:46:06.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.984+0000 7f67a7797740 1 -- 172.21.15.90:0/1636201791 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5563d9b4b040 con 0x5563d9c40650 2024-04-02T14:46:06.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.984+0000 7f679de00640 1 --2- 172.21.15.90:0/1636201791 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5563d9c40650 0x5563d9c42ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:06.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.984+0000 7f679de00640 1 --2- 172.21.15.90:0/1636201791 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5563d9c40650 0x5563d9c42ab0 crc :-1 s=READY pgs=294 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=faa67bd8a583aaf8 server_cookie=85a8b4b5773f719c in_seq=0 out_seq=0 2024-04-02T14:46:06.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.984+0000 7f677f400640 10 client.28798 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:46:06.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.985+0000 7f677f400640 1 -- 172.21.15.90:0/1636201791 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5563d9b4b040 con 0x5563d9c40650 2024-04-02T14:46:06.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.985+0000 7f677f400640 10 client.28798 handle_command_reply: tid=0 2024-04-02T14:46:06.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.985+0000 7f67a7797740 1 client.28798 shutdown 2024-04-02T14:46:06.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.985+0000 7f67a7797740 1 -- 172.21.15.90:0/1636201791 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5563d9c30230 msgr2=0x5563d9c2dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:06.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.985+0000 7f67a7797740 1 --2- 172.21.15.90:0/1636201791 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5563d9c30230 0x5563d9c2dd00 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f6798009b80 tx=0x7f6798004ab0 comp rx=0 tx=0).stop 2024-04-02T14:46:06.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.985+0000 7f67a7797740 1 -- 172.21.15.90:0/1636201791 shutdown_connections 2024-04-02T14:46:06.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.985+0000 7f67a7797740 1 --2- 172.21.15.90:0/1636201791 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5563d9c40650 0x5563d9c42ab0 crc :-1 s=READY pgs=294 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:06.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.985+0000 7f67a7797740 1 --2- 172.21.15.90:0/1636201791 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5563d9c30230 0x5563d9c2dd00 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:06.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.985+0000 7f67a7797740 1 --2- 172.21.15.90:0/1636201791 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5563d9c27ba0 0x5563d9c2d7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:06.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.985+0000 7f67a7797740 1 --2- 172.21.15.90:0/1636201791 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5563d9c26ec0 0x5563d9c2d280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:06.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.985+0000 7f67a7797740 1 -- 172.21.15.90:0/1636201791 >> 172.21.15.90:0/1636201791 conn(0x5563d9beda50 msgr2=0x5563d9c23660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:46:06.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.986+0000 7f677f400640 0 client.28798 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:46:06.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.986+0000 7f67a7797740 1 -- 172.21.15.90:0/1636201791 shutdown_connections 2024-04-02T14:46:06.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.986+0000 7f67a7797740 1 -- 172.21.15.90:0/1636201791 wait complete. 2024-04-02T14:46:06.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:06.986+0000 7f67a7797740 20 client.28798 trim_cache size 0 max 16384 2024-04-02T14:46:07.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:07.010+0000 7f67a7797740 1 Processor -- start 2024-04-02T14:46:07.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:07.010+0000 7f67a7797740 1 -- start start 2024-04-02T14:46:07.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:07.010+0000 7f67a7797740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5563d9c26ec0 0x5563d9c2f9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:07.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:07.010+0000 7f67a7797740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5563d9c27ba0 0x5563d9c3bc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:07.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:07.010+0000 7f67a7797740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5563d9c30230 0x5563d9c48380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:07.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:07.010+0000 7f67a7797740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f67a01af860 con 0x5563d9c27ba0 2024-04-02T14:46:07.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:07.010+0000 7f67a7797740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f67a01b13a0 con 0x5563d9c26ec0 2024-04-02T14:46:07.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:07.010+0000 7f67a7797740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f67a01aff30 con 0x5563d9c30230 2024-04-02T14:46:07.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:07.011+0000 7f679de00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5563d9c26ec0 0x5563d9c2f9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:07.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:07.011+0000 7f679d400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5563d9c27ba0 0x5563d9c3bc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:07.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:07.012+0000 7f679d400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5563d9c27ba0 0x5563d9c3bc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:60372/0 (socket says 172.21.15.90:60372) 2024-04-02T14:46:07.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:07.012+0000 7f679d400640 1 -- 172.21.15.90:0/4194969247 learned_addr learned my addr 172.21.15.90:0/4194969247 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:46:07.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:07.013+0000 7f679de00640 1 -- 172.21.15.90:0/4194969247 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5563d9c30230 msgr2=0x5563d9c48380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:46:07.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:07.013+0000 7f679de00640 1 --2- 172.21.15.90:0/4194969247 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5563d9c30230 0x5563d9c48380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:07.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:07.013+0000 7f679de00640 1 -- 172.21.15.90:0/4194969247 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5563d9c27ba0 msgr2=0x5563d9c3bc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:46:07.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:07.013+0000 7f679de00640 1 --2- 172.21.15.90:0/4194969247 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5563d9c27ba0 0x5563d9c3bc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:07.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:07.013+0000 7f679de00640 1 -- 172.21.15.90:0/4194969247 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5563d9c2b7d0 con 0x5563d9c26ec0 2024-04-02T14:46:07.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:07.014+0000 7f679de00640 1 --2- 172.21.15.90:0/4194969247 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5563d9c26ec0 0x5563d9c2f9b0 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f679002c2e0 tx=0x7f679002bcd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8ba51621580de4f8 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:46:07.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:07.015+0000 7f678a600640 1 -- 172.21.15.90:0/4194969247 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6790051070 con 0x5563d9c26ec0 2024-04-02T14:46:07.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:07.015+0000 7f678a600640 1 -- 172.21.15.90:0/4194969247 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6790004c80 con 0x5563d9c26ec0 2024-04-02T14:46:07.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:07.015+0000 7f678a600640 1 -- 172.21.15.90:0/4194969247 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6790051070 con 0x5563d9c26ec0 2024-04-02T14:46:07.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:07.018+0000 7f67a7797740 1 -- 172.21.15.90:0/4194969247 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5563d9c26ec0 msgr2=0x5563d9c2f9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:07.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:07.018+0000 7f67a7797740 1 --2- 172.21.15.90:0/4194969247 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5563d9c26ec0 0x5563d9c2f9b0 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f679002c2e0 tx=0x7f679002bcd0 comp rx=0 tx=0).stop 2024-04-02T14:46:07.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:07.021+0000 7f67a7797740 1 -- 172.21.15.90:0/4194969247 shutdown_connections 2024-04-02T14:46:07.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:07.021+0000 7f67a7797740 1 --2- 172.21.15.90:0/4194969247 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5563d9c30230 0x5563d9c48380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:07.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:07.021+0000 7f67a7797740 1 --2- 172.21.15.90:0/4194969247 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5563d9c27ba0 0x5563d9c3bc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:07.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:07.021+0000 7f67a7797740 1 --2- 172.21.15.90:0/4194969247 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5563d9c26ec0 0x5563d9c2f9b0 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:07.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:07.021+0000 7f67a7797740 1 -- 172.21.15.90:0/4194969247 >> 172.21.15.90:0/4194969247 conn(0x5563d9beda50 msgr2=0x5563d9c21440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:46:07.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:07.021+0000 7f67a7797740 1 -- 172.21.15.90:0/4194969247 shutdown_connections 2024-04-02T14:46:07.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:07.022+0000 7f67a7797740 1 -- 172.21.15.90:0/4194969247 wait complete. 2024-04-02T14:46:07.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:07.022+0000 7f67a7797740 1 -- start start 2024-04-02T14:46:07.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:07.022+0000 7f67a7797740 1 Processor -- start 2024-04-02T14:46:07.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:07.024+0000 7f67a7797740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5563d9c26ec0 0x5563d9c386a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:07.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:07.024+0000 7f67a7797740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5563d9c27ba0 0x5563d9c30c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:07.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:07.025+0000 7f67a7797740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5563d9c30230 0x5563d9c313b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:07.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:07.025+0000 7f67a7797740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f67a01b0cd0 con 0x5563d9c27ba0 2024-04-02T14:46:07.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:07.025+0000 7f67a7797740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f67a01b0600 con 0x5563d9c30230 2024-04-02T14:46:07.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:07.025+0000 7f67a7797740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f67a01ad860 con 0x5563d9c26ec0 2024-04-02T14:46:07.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:07.025+0000 7f67a7797740 10 client.0 _ll_register_callbacks cb 0x7f6798009cf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:46:07.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:07.025+0000 7f679e800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5563d9c30230 0x5563d9c313b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:07.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:07.025+0000 7f679e800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5563d9c30230 0x5563d9c313b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:52692/0 (socket says 172.21.15.90:52692) 2024-04-02T14:46:07.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:07.025+0000 7f679e800640 1 -- 172.21.15.90:0/833236045 learned_addr learned my addr 172.21.15.90:0/833236045 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:46:07.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:07.025+0000 7f679de00640 1 --2- 172.21.15.90:0/833236045 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5563d9c26ec0 0x5563d9c386a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:07.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:07.026+0000 7f679d400640 1 --2- 172.21.15.90:0/833236045 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5563d9c27ba0 0x5563d9c30c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:07.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:07.026+0000 7f679e800640 1 -- 172.21.15.90:0/833236045 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5563d9c26ec0 msgr2=0x5563d9c386a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:46:07.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:07.026+0000 7f679e800640 1 --2- 172.21.15.90:0/833236045 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5563d9c26ec0 0x5563d9c386a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:07.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:07.026+0000 7f679e800640 1 -- 172.21.15.90:0/833236045 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5563d9c27ba0 msgr2=0x5563d9c30c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:46:07.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:07.026+0000 7f679e800640 1 --2- 172.21.15.90:0/833236045 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5563d9c27ba0 0x5563d9c30c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:07.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:07.026+0000 7f679e800640 1 -- 172.21.15.90:0/833236045 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5563d9c361e0 con 0x5563d9c30230 2024-04-02T14:46:07.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:07.026+0000 7f679de00640 1 --2- 172.21.15.90:0/833236045 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5563d9c26ec0 0x5563d9c386a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:46:07.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:07.026+0000 7f677f400640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:46:07.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:07.026+0000 7f679e800640 1 --2- 172.21.15.90:0/833236045 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5563d9c30230 0x5563d9c313b0 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f679800ff40 tx=0x7f6798009740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dbec6e210eb62a9d server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:46:07.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:07.027+0000 7f677f400640 1 -- 172.21.15.90:0/833236045 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6798011790 con 0x5563d9c30230 2024-04-02T14:46:07.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:07.027+0000 7f67a7797740 1 -- 172.21.15.90:0/833236045 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6790004c80 con 0x5563d9c30230 2024-04-02T14:46:07.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:07.027+0000 7f677f400640 1 -- 172.21.15.90:0/833236045 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 8594+0+0 (secure 0 0 0) 0x7f67980123f0 con 0x5563d9c30230 2024-04-02T14:46:07.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:07.029+0000 7f677f400640 1 -- 172.21.15.90:0/833236045 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f6798011790 con 0x5563d9c30230 2024-04-02T14:46:07.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:07.029+0000 7f67a7797740 10 client.28991 fetch_fsmap learned FSMap version 27 2024-04-02T14:46:07.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:07.029+0000 7f67a7797740 1 -- 172.21.15.90:0/833236045 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5563d9caf520 con 0x5563d9c30230 2024-04-02T14:46:07.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:07.029+0000 7f677f400640 1 -- 172.21.15.90:0/833236045 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6790004c80 con 0x5563d9c30230 2024-04-02T14:46:07.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:07.031+0000 7f677f400640 1 -- 172.21.15.90:0/833236045 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f6798038020 con 0x5563d9c30230 2024-04-02T14:46:07.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:07.031+0000 7f67a7797740 10 client.28991 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:46:07.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:07.031+0000 7f67a7797740 10 client.28991 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:46:07.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:07.031+0000 7f67a7797740 20 client.28991 populate_metadata read hostname 'smithi090' 2024-04-02T14:46:07.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:07.031+0000 7f67a7797740 1 --2- 172.21.15.90:0/833236045 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5563d9cafc20 0x5563d9cb2040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:07.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:07.031+0000 7f67a7797740 4 client.28991 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-02T14:46:07.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:07.031+0000 7f67a7797740 1 -- 172.21.15.90:0/833236045 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5563d9b98c30 con 0x5563d9cafc20 2024-04-02T14:46:07.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:07.033+0000 7f679de00640 1 --2- 172.21.15.90:0/833236045 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5563d9cafc20 0x5563d9cb2040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:07.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:07.033+0000 7f679de00640 1 --2- 172.21.15.90:0/833236045 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5563d9cafc20 0x5563d9cb2040 crc :-1 s=READY pgs=295 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1d46662bfca26c66 server_cookie=1aaad00b1dd662d5 in_seq=0 out_seq=0 2024-04-02T14:46:07.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:07.033+0000 7f677f400640 10 client.28991 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:46:07.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:07.034+0000 7f677f400640 1 -- 172.21.15.90:0/833236045 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5563d9b98c30 con 0x5563d9cafc20 2024-04-02T14:46:07.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:07.034+0000 7f677f400640 10 client.28991 handle_command_reply: tid=0 2024-04-02T14:46:07.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:07.034+0000 7f67a7797740 1 client.28991 shutdown 2024-04-02T14:46:07.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:07.034+0000 7f67a7797740 1 -- 172.21.15.90:0/833236045 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5563d9c30230 msgr2=0x5563d9c313b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:07.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:07.034+0000 7f67a7797740 1 --2- 172.21.15.90:0/833236045 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5563d9c30230 0x5563d9c313b0 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f679800ff40 tx=0x7f6798009740 comp rx=0 tx=0).stop 2024-04-02T14:46:07.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:07.034+0000 7f67a7797740 1 -- 172.21.15.90:0/833236045 shutdown_connections 2024-04-02T14:46:07.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:07.034+0000 7f67a7797740 1 --2- 172.21.15.90:0/833236045 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5563d9cafc20 0x5563d9cb2040 crc :-1 s=READY pgs=295 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:07.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:07.034+0000 7f67a7797740 1 --2- 172.21.15.90:0/833236045 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5563d9c30230 0x5563d9c313b0 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:07.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:07.034+0000 7f67a7797740 1 --2- 172.21.15.90:0/833236045 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5563d9c27ba0 0x5563d9c30c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:07.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:07.034+0000 7f67a7797740 1 --2- 172.21.15.90:0/833236045 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5563d9c26ec0 0x5563d9c386a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:07.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:07.035+0000 7f67a7797740 1 -- 172.21.15.90:0/833236045 >> 172.21.15.90:0/833236045 conn(0x5563d9beda50 msgr2=0x5563d9c20260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:46:07.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:07.035+0000 7f677f400640 0 client.28991 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:46:07.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:07.035+0000 7f67a7797740 1 -- 172.21.15.90:0/833236045 shutdown_connections 2024-04-02T14:46:07.031 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:07.035+0000 7f67a7797740 1 -- 172.21.15.90:0/833236045 wait complete. 2024-04-02T14:46:07.031 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:07.035+0000 7f67a7797740 20 client.28991 trim_cache size 0 max 16384 2024-04-02T14:46:07.031 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-02T14:46:07.031 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:07.037+0000 7f679f200640 1 -- 172.21.15.90:0/2570259483 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f67840a3d20 msgr2=0x7f67840a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:07.032 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:07.037+0000 7f679f200640 1 --2- 172.21.15.90:0/2570259483 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f67840a3d20 0x7f67840a61e0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f6798005fd0 tx=0x7f6798009500 comp rx=0 tx=0).stop 2024-04-02T14:46:07.032 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:07.037+0000 7f679f200640 1 -- 172.21.15.90:0/2570259483 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f67a0103450 msgr2=0x7f67a019a9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:07.033 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:07.037+0000 7f679f200640 1 --2- 172.21.15.90:0/2570259483 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f67a0103450 0x7f67a019a9f0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f678c00e9f0 tx=0x7f678c00edf0 comp rx=0 tx=0).stop 2024-04-02T14:46:07.033 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:07.038+0000 7f679f200640 1 -- 172.21.15.90:0/2570259483 shutdown_connections 2024-04-02T14:46:07.033 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:07.038+0000 7f679f200640 1 --2- 172.21.15.90:0/2570259483 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f67840a3d20 0x7f67840a61e0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:07.033 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:07.038+0000 7f679f200640 1 --2- 172.21.15.90:0/2570259483 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f67a010bf50 0x7f67a019ed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:07.033 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:07.038+0000 7f679f200640 1 --2- 172.21.15.90:0/2570259483 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f67a0103450 0x7f67a019a9f0 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:07.033 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:07.038+0000 7f679f200640 1 --2- 172.21.15.90:0/2570259483 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f67a0102b30 0x7f67a019a4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:07.033 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:07.038+0000 7f679f200640 1 -- 172.21.15.90:0/2570259483 >> 172.21.15.90:0/2570259483 conn(0x7f67a00fc350 msgr2=0x7f67a00fcfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:46:07.033 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:07.038+0000 7f679f200640 1 -- 172.21.15.90:0/2570259483 shutdown_connections 2024-04-02T14:46:07.034 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:07.038+0000 7f679f200640 1 -- 172.21.15.90:0/2570259483 wait complete. 2024-04-02T14:46:08.077 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-02T14:46:08.078 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T14:46:08.078 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-02T14:46:08.104 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:46:07 smithi092 ceph-mon[36497]: pgmap v348: 97 pgs: 97 active+clean; 4.7 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 7.9 MiB/s wr, 833 op/s 2024-04-02T14:46:08.104 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:46:07 smithi092 ceph-mon[36497]: osdmap e100: 12 total, 12 up, 12 in 2024-04-02T14:46:08.139 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:46:07 smithi090 ceph-mon[31674]: pgmap v348: 97 pgs: 97 active+clean; 4.7 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 7.9 MiB/s wr, 833 op/s 2024-04-02T14:46:08.139 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:46:07 smithi090 ceph-mon[31674]: osdmap e100: 12 total, 12 up, 12 in 2024-04-02T14:46:08.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:46:07 smithi177 ceph-mon[36431]: pgmap v348: 97 pgs: 97 active+clean; 4.7 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 7.9 MiB/s wr, 833 op/s 2024-04-02T14:46:08.181 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:46:07 smithi177 ceph-mon[36431]: osdmap e100: 12 total, 12 up, 12 in 2024-04-02T14:46:08.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.284+0000 7fd934000640 1 -- 172.21.15.90:0/3781739780 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd92c100910 msgr2=0x7fd92c100cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:08.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.284+0000 7fd934000640 1 --2- 172.21.15.90:0/3781739780 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd92c100910 0x7fd92c100cf0 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7fd920009900 tx=0x7fd92002bc40 comp rx=0 tx=0).stop 2024-04-02T14:46:08.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.284+0000 7fd934000640 1 -- 172.21.15.90:0/3781739780 shutdown_connections 2024-04-02T14:46:08.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.284+0000 7fd934000640 1 --2- 172.21.15.90:0/3781739780 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd92c10e4c0 0x7fd92c1108b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:08.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.284+0000 7fd934000640 1 --2- 172.21.15.90:0/3781739780 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd92c101230 0x7fd92c10df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:08.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.284+0000 7fd934000640 1 --2- 172.21.15.90:0/3781739780 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd92c100910 0x7fd92c100cf0 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:08.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.284+0000 7fd934000640 1 -- 172.21.15.90:0/3781739780 >> 172.21.15.90:0/3781739780 conn(0x7fd92c077b60 msgr2=0x7fd92c077f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:46:08.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.284+0000 7fd934000640 1 -- 172.21.15.90:0/3781739780 shutdown_connections 2024-04-02T14:46:08.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.284+0000 7fd934000640 1 -- 172.21.15.90:0/3781739780 wait complete. 2024-04-02T14:46:08.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.285+0000 7fd934000640 1 Processor -- start 2024-04-02T14:46:08.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.285+0000 7fd934000640 1 -- start start 2024-04-02T14:46:08.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.285+0000 7fd934000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd92c100910 0x7fd92c19a5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:08.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.285+0000 7fd934000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd92c101230 0x7fd92c19ab10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:08.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.285+0000 7fd934000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd92c10e4c0 0x7fd92c19ee90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:08.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.286+0000 7fd934000640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fd92c114130 con 0x7fd92c10e4c0 2024-04-02T14:46:08.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.286+0000 7fd934000640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fd92c113fb0 con 0x7fd92c100910 2024-04-02T14:46:08.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.286+0000 7fd934000640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fd92c1142b0 con 0x7fd92c101230 2024-04-02T14:46:08.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.286+0000 7fd931600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd92c100910 0x7fd92c19a5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:08.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.286+0000 7fd931600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd92c100910 0x7fd92c19a5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:52710/0 (socket says 172.21.15.90:52710) 2024-04-02T14:46:08.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.286+0000 7fd931600640 1 -- 172.21.15.90:0/1015617899 learned_addr learned my addr 172.21.15.90:0/1015617899 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:46:08.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.286+0000 7fd930c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd92c101230 0x7fd92c19ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:08.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.286+0000 7fd932000640 1 --2- 172.21.15.90:0/1015617899 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd92c10e4c0 0x7fd92c19ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:08.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.286+0000 7fd931600640 1 -- 172.21.15.90:0/1015617899 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd92c101230 msgr2=0x7fd92c19ab10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:08.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.286+0000 7fd931600640 1 --2- 172.21.15.90:0/1015617899 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd92c101230 0x7fd92c19ab10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:08.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.286+0000 7fd931600640 1 -- 172.21.15.90:0/1015617899 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd92c10e4c0 msgr2=0x7fd92c19ee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:08.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.286+0000 7fd931600640 1 --2- 172.21.15.90:0/1015617899 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd92c10e4c0 0x7fd92c19ee90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:08.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.286+0000 7fd931600640 1 -- 172.21.15.90:0/1015617899 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd92c19f5c0 con 0x7fd92c100910 2024-04-02T14:46:08.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.286+0000 7fd932000640 1 --2- 172.21.15.90:0/1015617899 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd92c10e4c0 0x7fd92c19ee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:46:08.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.286+0000 7fd930c00640 1 --2- 172.21.15.90:0/1015617899 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd92c101230 0x7fd92c19ab10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:46:08.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.287+0000 7fd931600640 1 --2- 172.21.15.90:0/1015617899 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd92c100910 0x7fd92c19a5d0 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7fd920005ab0 tx=0x7fd9200042f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:46:08.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.287+0000 7fd92a000640 1 -- 172.21.15.90:0/1015617899 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd920041070 con 0x7fd92c100910 2024-04-02T14:46:08.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.287+0000 7fd92a000640 1 -- 172.21.15.90:0/1015617899 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd92002cca0 con 0x7fd92c100910 2024-04-02T14:46:08.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.287+0000 7fd92a000640 1 -- 172.21.15.90:0/1015617899 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd920003d90 con 0x7fd92c100910 2024-04-02T14:46:08.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.287+0000 7fd934000640 1 -- 172.21.15.90:0/1015617899 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd92c19f850 con 0x7fd92c100910 2024-04-02T14:46:08.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.287+0000 7fd934000640 1 -- 172.21.15.90:0/1015617899 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd92c1a7140 con 0x7fd92c100910 2024-04-02T14:46:08.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.291+0000 7fd92a000640 1 -- 172.21.15.90:0/1015617899 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fd920044070 con 0x7fd92c100910 2024-04-02T14:46:08.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.292+0000 7fd936d74740 1 Processor -- start 2024-04-02T14:46:08.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.293+0000 7fd936d74740 1 -- start start 2024-04-02T14:46:08.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.293+0000 7fd936d74740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563d11e0d030 0x563d11e0bd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:08.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.293+0000 7fd936d74740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563d11e0c350 0x563d11e0c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:08.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.293+0000 7fd936d74740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563d11e156c0 0x563d11e17ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:08.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.293+0000 7fd936d74740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x563d11d70310 con 0x563d11e156c0 2024-04-02T14:46:08.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.293+0000 7fd936d74740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x563d11d18690 con 0x563d11e0d030 2024-04-02T14:46:08.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.293+0000 7fd936d74740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x563d11d6c2c0 con 0x563d11e0c350 2024-04-02T14:46:08.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.293+0000 7fd932000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563d11e0d030 0x563d11e0bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:08.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.293+0000 7fd931600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563d11e0c350 0x563d11e0c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:08.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.293+0000 7fd932000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563d11e0d030 0x563d11e0bd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:52722/0 (socket says 172.21.15.90:52722) 2024-04-02T14:46:08.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.293+0000 7fd932000640 1 -- 172.21.15.90:0/1364852653 learned_addr learned my addr 172.21.15.90:0/1364852653 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:46:08.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.294+0000 7fd930c00640 1 --2- 172.21.15.90:0/1364852653 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563d11e156c0 0x563d11e17ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:08.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.294+0000 7fd92a000640 1 --2- 172.21.15.90:0/1015617899 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fd9140a3e20 0x7fd9140a62e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:08.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.294+0000 7fd92a000640 1 -- 172.21.15.90:0/1015617899 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 8594+0+0 (secure 0 0 0) 0x7fd9200f2110 con 0x7fd92c100910 2024-04-02T14:46:08.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.294+0000 7fd931600640 1 -- 172.21.15.90:0/1364852653 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563d11e0d030 msgr2=0x563d11e0bd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:46:08.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.294+0000 7fd931600640 1 --2- 172.21.15.90:0/1364852653 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563d11e0d030 0x563d11e0bd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:08.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.294+0000 7fd931600640 1 -- 172.21.15.90:0/1364852653 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563d11e156c0 msgr2=0x563d11e17ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:46:08.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.294+0000 7fd931600640 1 --2- 172.21.15.90:0/1364852653 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563d11e156c0 0x563d11e17ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:08.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.294+0000 7fd931600640 1 -- 172.21.15.90:0/1364852653 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563d11b4c7e0 con 0x563d11e0c350 2024-04-02T14:46:08.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.294+0000 7fd932000640 1 --2- 172.21.15.90:0/1364852653 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563d11e0d030 0x563d11e0bd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:46:08.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.294+0000 7fd932000640 1 --2- 172.21.15.90:0/1015617899 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fd9140a3e20 0x7fd9140a62e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:08.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.294+0000 7fd931600640 1 --2- 172.21.15.90:0/1364852653 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563d11e0c350 0x563d11e0c7d0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7fd9200ef8c0 tx=0x7fd92002cf90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dd15cc768ff172ab server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:46:08.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.294+0000 7fd91e600640 1 -- 172.21.15.90:0/1364852653 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd92004b030 con 0x563d11e0c350 2024-04-02T14:46:08.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.295+0000 7fd91e600640 1 -- 172.21.15.90:0/1364852653 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd920004c80 con 0x563d11e0c350 2024-04-02T14:46:08.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.295+0000 7fd91e600640 1 -- 172.21.15.90:0/1364852653 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9200be810 con 0x563d11e0c350 2024-04-02T14:46:08.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.295+0000 7fd936d74740 1 -- 172.21.15.90:0/1364852653 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563d11e0c350 msgr2=0x563d11e0c7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:08.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.295+0000 7fd936d74740 1 --2- 172.21.15.90:0/1364852653 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563d11e0c350 0x563d11e0c7d0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7fd9200ef8c0 tx=0x7fd92002cf90 comp rx=0 tx=0).stop 2024-04-02T14:46:08.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.295+0000 7fd932000640 1 --2- 172.21.15.90:0/1015617899 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fd9140a3e20 0x7fd9140a62e0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fd924014710 tx=0x7fd924004a40 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:46:08.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.295+0000 7fd936d74740 1 -- 172.21.15.90:0/1364852653 shutdown_connections 2024-04-02T14:46:08.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.295+0000 7fd936d74740 1 --2- 172.21.15.90:0/1364852653 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563d11e156c0 0x563d11e17ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:08.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.295+0000 7fd936d74740 1 --2- 172.21.15.90:0/1364852653 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563d11e0c350 0x563d11e0c7d0 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:08.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.295+0000 7fd936d74740 1 --2- 172.21.15.90:0/1364852653 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563d11e0d030 0x563d11e0bd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:08.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.295+0000 7fd936d74740 1 -- 172.21.15.90:0/1364852653 >> 172.21.15.90:0/1364852653 conn(0x563d11da0270 msgr2=0x563d11e06da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:46:08.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.295+0000 7fd936d74740 1 -- 172.21.15.90:0/1364852653 shutdown_connections 2024-04-02T14:46:08.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.296+0000 7fd936d74740 1 -- 172.21.15.90:0/1364852653 wait complete. 2024-04-02T14:46:08.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.296+0000 7fd936d74740 1 -- start start 2024-04-02T14:46:08.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.296+0000 7fd936d74740 1 Processor -- start 2024-04-02T14:46:08.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.296+0000 7fd936d74740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563d11e0c350 0x563d11e12670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:08.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.296+0000 7fd936d74740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563d11e0d030 0x563d11e12bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:08.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.296+0000 7fd936d74740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563d11e156c0 0x563d11e130f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:08.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.297+0000 7fd936d74740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x563d11d6c2c0 con 0x563d11e0d030 2024-04-02T14:46:08.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.297+0000 7fd936d74740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x563d11d2ba60 con 0x563d11e156c0 2024-04-02T14:46:08.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.297+0000 7fd936d74740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x563d11e182f0 con 0x563d11e0c350 2024-04-02T14:46:08.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.297+0000 7fd936d74740 10 client.0 _ll_register_callbacks cb 0x563d11d31030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:46:08.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.297+0000 7fd931600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563d11e0c350 0x563d11e12670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:08.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.297+0000 7fd930c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563d11e0d030 0x563d11e12bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:08.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.297+0000 7fd932000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563d11e156c0 0x563d11e130f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:08.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.297+0000 7fd931600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563d11e0c350 0x563d11e12670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:58484/0 (socket says 172.21.15.90:58484) 2024-04-02T14:46:08.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.297+0000 7fd931600640 1 -- 172.21.15.90:0/182920417 learned_addr learned my addr 172.21.15.90:0/182920417 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:46:08.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.297+0000 7fd932000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563d11e156c0 0x563d11e130f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:52734/0 (socket says 172.21.15.90:52734) 2024-04-02T14:46:08.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.297+0000 7fd931600640 1 -- 172.21.15.90:0/182920417 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563d11e156c0 msgr2=0x563d11e130f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:46:08.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.297+0000 7fd931600640 1 --2- 172.21.15.90:0/182920417 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563d11e156c0 0x563d11e130f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:08.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.297+0000 7fd931600640 1 -- 172.21.15.90:0/182920417 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563d11e0d030 msgr2=0x563d11e12bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:46:08.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.297+0000 7fd931600640 1 --2- 172.21.15.90:0/182920417 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563d11e0d030 0x563d11e12bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:08.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.297+0000 7fd931600640 1 -- 172.21.15.90:0/182920417 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563d11bacf70 con 0x563d11e0c350 2024-04-02T14:46:08.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.297+0000 7fd932000640 1 --2- 172.21.15.90:0/182920417 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563d11e156c0 0x563d11e130f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:46:08.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.297+0000 7fd931600640 1 --2- 172.21.15.90:0/182920417 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563d11e0c350 0x563d11e12670 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7fd9200bdbe0 tx=0x7fd920005bd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9479f47ed3600d8e server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:46:08.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.297+0000 7fd912a00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:46:08.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.298+0000 7fd912a00640 1 -- 172.21.15.90:0/182920417 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd92004b070 con 0x563d11e0c350 2024-04-02T14:46:08.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.298+0000 7fd912a00640 1 -- 172.21.15.90:0/182920417 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 8594+0+0 (secure 0 0 0) 0x7fd9200ef9a0 con 0x563d11e0c350 2024-04-02T14:46:08.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.298+0000 7fd936d74740 1 -- 172.21.15.90:0/182920417 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd920004c80 con 0x563d11e0c350 2024-04-02T14:46:08.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.298+0000 7fd912a00640 1 -- 172.21.15.90:0/182920417 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fd9200bdee0 con 0x563d11e0c350 2024-04-02T14:46:08.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.298+0000 7fd936d74740 10 client.28813 fetch_fsmap learned FSMap version 27 2024-04-02T14:46:08.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.298+0000 7fd936d74740 1 -- 172.21.15.90:0/182920417 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x563d11a44170 con 0x563d11e0c350 2024-04-02T14:46:08.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.298+0000 7fd912a00640 1 -- 172.21.15.90:0/182920417 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd920004c80 con 0x563d11e0c350 2024-04-02T14:46:08.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.298+0000 7fd912a00640 1 -- 172.21.15.90:0/182920417 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fd9200c68f0 con 0x563d11e0c350 2024-04-02T14:46:08.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.298+0000 7fd936d74740 10 client.28813 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:46:08.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.298+0000 7fd936d74740 10 client.28813 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:46:08.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.298+0000 7fd936d74740 20 client.28813 populate_metadata read hostname 'smithi090' 2024-04-02T14:46:08.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.298+0000 7fd936d74740 1 --2- 172.21.15.90:0/182920417 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x563d11e25650 0x563d11e27a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:08.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.298+0000 7fd936d74740 4 client.28813 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:46:08.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.298+0000 7fd936d74740 1 -- 172.21.15.90:0/182920417 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563d11d7d850 con 0x563d11e25650 2024-04-02T14:46:08.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.299+0000 7fd930c00640 1 --2- 172.21.15.90:0/182920417 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x563d11e25650 0x563d11e27a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:08.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.299+0000 7fd930c00640 1 --2- 172.21.15.90:0/182920417 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x563d11e25650 0x563d11e27a40 crc :-1 s=READY pgs=296 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9e25016fa6712823 server_cookie=19035c4e73353580 in_seq=0 out_seq=0 2024-04-02T14:46:08.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.299+0000 7fd912a00640 10 client.28813 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:46:08.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.300+0000 7fd912a00640 1 -- 172.21.15.90:0/182920417 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x563d11d7d850 con 0x563d11e25650 2024-04-02T14:46:08.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.300+0000 7fd912a00640 10 client.28813 handle_command_reply: tid=0 2024-04-02T14:46:08.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.300+0000 7fd936d74740 1 client.28813 shutdown 2024-04-02T14:46:08.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.300+0000 7fd936d74740 1 -- 172.21.15.90:0/182920417 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563d11e0c350 msgr2=0x563d11e12670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:08.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.300+0000 7fd936d74740 1 --2- 172.21.15.90:0/182920417 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563d11e0c350 0x563d11e12670 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7fd9200bdbe0 tx=0x7fd920005bd0 comp rx=0 tx=0).stop 2024-04-02T14:46:08.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.300+0000 7fd936d74740 1 -- 172.21.15.90:0/182920417 shutdown_connections 2024-04-02T14:46:08.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.300+0000 7fd936d74740 1 --2- 172.21.15.90:0/182920417 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x563d11e25650 0x563d11e27a40 crc :-1 s=READY pgs=296 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:08.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.300+0000 7fd936d74740 1 --2- 172.21.15.90:0/182920417 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563d11e156c0 0x563d11e130f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:08.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.300+0000 7fd912a00640 0 client.28813 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:46:08.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.300+0000 7fd936d74740 1 --2- 172.21.15.90:0/182920417 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563d11e0d030 0x563d11e12bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:08.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.300+0000 7fd936d74740 1 --2- 172.21.15.90:0/182920417 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563d11e0c350 0x563d11e12670 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:08.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.300+0000 7fd936d74740 1 -- 172.21.15.90:0/182920417 >> 172.21.15.90:0/182920417 conn(0x563d11da0270 msgr2=0x563d11e08af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:46:08.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.300+0000 7fd936d74740 1 -- 172.21.15.90:0/182920417 shutdown_connections 2024-04-02T14:46:08.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.300+0000 7fd936d74740 1 -- 172.21.15.90:0/182920417 wait complete. 2024-04-02T14:46:08.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.300+0000 7fd936d74740 20 client.28813 trim_cache size 0 max 16384 2024-04-02T14:46:08.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.320+0000 7fd936d74740 1 Processor -- start 2024-04-02T14:46:08.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.320+0000 7fd936d74740 1 -- start start 2024-04-02T14:46:08.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.320+0000 7fd936d74740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563d11e0c350 0x563d11e14da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:08.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.321+0000 7fd936d74740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563d11e0d030 0x563d11e210d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:08.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.321+0000 7fd936d74740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563d11e156c0 0x563d11e2d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:08.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.321+0000 7fd936d74740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fd92c1afaf0 con 0x563d11e0d030 2024-04-02T14:46:08.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.321+0000 7fd936d74740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fd92c1b1630 con 0x563d11e156c0 2024-04-02T14:46:08.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.321+0000 7fd936d74740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fd92c1b01c0 con 0x563d11e0c350 2024-04-02T14:46:08.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.321+0000 7fd931600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563d11e0c350 0x563d11e14da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:08.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.321+0000 7fd932000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563d11e156c0 0x563d11e2d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:08.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.321+0000 7fd930c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563d11e0d030 0x563d11e210d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:08.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.321+0000 7fd931600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563d11e0c350 0x563d11e14da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:58490/0 (socket says 172.21.15.90:58490) 2024-04-02T14:46:08.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.321+0000 7fd931600640 1 -- 172.21.15.90:0/2243820471 learned_addr learned my addr 172.21.15.90:0/2243820471 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:46:08.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.321+0000 7fd931600640 1 -- 172.21.15.90:0/2243820471 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563d11e156c0 msgr2=0x563d11e2d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:46:08.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.321+0000 7fd931600640 1 --2- 172.21.15.90:0/2243820471 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563d11e156c0 0x563d11e2d2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:08.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.321+0000 7fd931600640 1 -- 172.21.15.90:0/2243820471 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563d11e0d030 msgr2=0x563d11e210d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:46:08.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.321+0000 7fd931600640 1 --2- 172.21.15.90:0/2243820471 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563d11e0d030 0x563d11e210d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:08.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.321+0000 7fd931600640 1 -- 172.21.15.90:0/2243820471 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563d11e135d0 con 0x563d11e0c350 2024-04-02T14:46:08.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.322+0000 7fd931600640 1 --2- 172.21.15.90:0/2243820471 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563d11e0c350 0x563d11e14da0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7fd92002c830 tx=0x7fd920009260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bb0fc7ea25df965b server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:46:08.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.322+0000 7fd91dc00640 1 -- 172.21.15.90:0/2243820471 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd92004b070 con 0x563d11e0c350 2024-04-02T14:46:08.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.322+0000 7fd91dc00640 1 -- 172.21.15.90:0/2243820471 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd920004c80 con 0x563d11e0c350 2024-04-02T14:46:08.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.322+0000 7fd91dc00640 1 -- 172.21.15.90:0/2243820471 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9200bdc50 con 0x563d11e0c350 2024-04-02T14:46:08.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.323+0000 7fd936d74740 1 -- 172.21.15.90:0/2243820471 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563d11e0c350 msgr2=0x563d11e14da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:08.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.323+0000 7fd936d74740 1 --2- 172.21.15.90:0/2243820471 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563d11e0c350 0x563d11e14da0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7fd92002c830 tx=0x7fd920009260 comp rx=0 tx=0).stop 2024-04-02T14:46:08.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.323+0000 7fd936d74740 1 -- 172.21.15.90:0/2243820471 shutdown_connections 2024-04-02T14:46:08.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.323+0000 7fd936d74740 1 --2- 172.21.15.90:0/2243820471 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563d11e156c0 0x563d11e2d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:08.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.323+0000 7fd936d74740 1 --2- 172.21.15.90:0/2243820471 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563d11e0d030 0x563d11e210d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:08.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.323+0000 7fd936d74740 1 --2- 172.21.15.90:0/2243820471 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563d11e0c350 0x563d11e14da0 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:08.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.323+0000 7fd936d74740 1 -- 172.21.15.90:0/2243820471 >> 172.21.15.90:0/2243820471 conn(0x563d11da0270 msgr2=0x563d11e12ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:46:08.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.323+0000 7fd936d74740 1 -- 172.21.15.90:0/2243820471 shutdown_connections 2024-04-02T14:46:08.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.323+0000 7fd936d74740 1 -- 172.21.15.90:0/2243820471 wait complete. 2024-04-02T14:46:08.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.323+0000 7fd936d74740 1 -- start start 2024-04-02T14:46:08.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.323+0000 7fd936d74740 1 Processor -- start 2024-04-02T14:46:08.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.324+0000 7fd936d74740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563d11e0c350 0x563d11e1d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:08.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.324+0000 7fd936d74740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563d11e0d030 0x563d11e185a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:08.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.324+0000 7fd936d74740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563d11e156c0 0x563d11e18ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:08.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.324+0000 7fd936d74740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fd92c1b0f60 con 0x563d11e156c0 2024-04-02T14:46:08.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.324+0000 7fd936d74740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fd92c1b0890 con 0x563d11e0d030 2024-04-02T14:46:08.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.324+0000 7fd936d74740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fd92c1adaf0 con 0x563d11e0c350 2024-04-02T14:46:08.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.324+0000 7fd936d74740 10 client.0 _ll_register_callbacks cb 0x7fd918009b30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:46:08.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.324+0000 7fd930c00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563d11e0d030 0x563d11e185a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:08.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.324+0000 7fd931600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563d11e0c350 0x563d11e1d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:08.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.324+0000 7fd930c00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563d11e0d030 0x563d11e185a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:52742/0 (socket says 172.21.15.90:52742) 2024-04-02T14:46:08.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.324+0000 7fd930c00640 1 -- 172.21.15.90:0/3443890281 learned_addr learned my addr 172.21.15.90:0/3443890281 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:46:08.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.324+0000 7fd932000640 1 --2- 172.21.15.90:0/3443890281 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563d11e156c0 0x563d11e18ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:08.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.324+0000 7fd930c00640 1 -- 172.21.15.90:0/3443890281 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563d11e0c350 msgr2=0x563d11e1d7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:46:08.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.324+0000 7fd930c00640 1 --2- 172.21.15.90:0/3443890281 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563d11e0c350 0x563d11e1d7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:08.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.324+0000 7fd930c00640 1 -- 172.21.15.90:0/3443890281 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563d11e156c0 msgr2=0x563d11e18ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:46:08.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.324+0000 7fd930c00640 1 --2- 172.21.15.90:0/3443890281 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563d11e156c0 0x563d11e18ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:08.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.324+0000 7fd930c00640 1 -- 172.21.15.90:0/3443890281 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563d11e19020 con 0x563d11e0d030 2024-04-02T14:46:08.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.324+0000 7fd931600640 1 --2- 172.21.15.90:0/3443890281 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563d11e0c350 0x563d11e1d7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:46:08.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.325+0000 7fd930c00640 1 --2- 172.21.15.90:0/3443890281 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563d11e0d030 0x563d11e185a0 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x563d11e13c40 tx=0x7fd918012b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=56e886041bcc28e6 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:46:08.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.325+0000 7fd912a00640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:46:08.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.325+0000 7fd912a00640 1 -- 172.21.15.90:0/3443890281 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd918004230 con 0x563d11e0d030 2024-04-02T14:46:08.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.325+0000 7fd936d74740 1 -- 172.21.15.90:0/3443890281 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd920004c80 con 0x563d11e0d030 2024-04-02T14:46:08.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.325+0000 7fd912a00640 1 -- 172.21.15.90:0/3443890281 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 8594+0+0 (secure 0 0 0) 0x7fd9180043d0 con 0x563d11e0d030 2024-04-02T14:46:08.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.325+0000 7fd912a00640 1 -- 172.21.15.90:0/3443890281 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fd918030050 con 0x563d11e0d030 2024-04-02T14:46:08.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.325+0000 7fd936d74740 10 client.29009 fetch_fsmap learned FSMap version 27 2024-04-02T14:46:08.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.325+0000 7fd936d74740 1 -- 172.21.15.90:0/3443890281 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x563d11e8c8e0 con 0x563d11e0d030 2024-04-02T14:46:08.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.325+0000 7fd912a00640 1 -- 172.21.15.90:0/3443890281 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd920004c80 con 0x563d11e0d030 2024-04-02T14:46:08.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.326+0000 7fd912a00640 1 -- 172.21.15.90:0/3443890281 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fd918003ae0 con 0x563d11e0d030 2024-04-02T14:46:08.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.326+0000 7fd936d74740 10 client.29009 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:46:08.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.326+0000 7fd936d74740 10 client.29009 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:46:08.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.326+0000 7fd936d74740 20 client.29009 populate_metadata read hostname 'smithi090' 2024-04-02T14:46:08.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.326+0000 7fd936d74740 1 --2- 172.21.15.90:0/3443890281 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x563d11e8d040 0x563d11e8f460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:08.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.326+0000 7fd936d74740 4 client.29009 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-02T14:46:08.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.326+0000 7fd936d74740 1 -- 172.21.15.90:0/3443890281 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x563d11d77ba0 con 0x563d11e8d040 2024-04-02T14:46:08.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.326+0000 7fd931600640 1 --2- 172.21.15.90:0/3443890281 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x563d11e8d040 0x563d11e8f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:08.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.326+0000 7fd931600640 1 --2- 172.21.15.90:0/3443890281 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x563d11e8d040 0x563d11e8f460 crc :-1 s=READY pgs=297 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=321c388aed1a3d2 server_cookie=e53dbcc05b728cc7 in_seq=0 out_seq=0 2024-04-02T14:46:08.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.326+0000 7fd912a00640 10 client.29009 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:46:08.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.327+0000 7fd912a00640 1 -- 172.21.15.90:0/3443890281 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x563d11d77ba0 con 0x563d11e8d040 2024-04-02T14:46:08.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.327+0000 7fd912a00640 10 client.29009 handle_command_reply: tid=0 2024-04-02T14:46:08.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.327+0000 7fd936d74740 1 client.29009 shutdown 2024-04-02T14:46:08.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.327+0000 7fd936d74740 1 -- 172.21.15.90:0/3443890281 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563d11e0d030 msgr2=0x563d11e185a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:08.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.327+0000 7fd936d74740 1 --2- 172.21.15.90:0/3443890281 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563d11e0d030 0x563d11e185a0 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x563d11e13c40 tx=0x7fd918012b80 comp rx=0 tx=0).stop 2024-04-02T14:46:08.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.328+0000 7fd936d74740 1 -- 172.21.15.90:0/3443890281 shutdown_connections 2024-04-02T14:46:08.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.328+0000 7fd936d74740 1 --2- 172.21.15.90:0/3443890281 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x563d11e8d040 0x563d11e8f460 crc :-1 s=READY pgs=297 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:08.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.328+0000 7fd936d74740 1 --2- 172.21.15.90:0/3443890281 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563d11e156c0 0x563d11e18ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:08.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.328+0000 7fd936d74740 1 --2- 172.21.15.90:0/3443890281 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563d11e0d030 0x563d11e185a0 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:08.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.328+0000 7fd936d74740 1 --2- 172.21.15.90:0/3443890281 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563d11e0c350 0x563d11e1d7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:08.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.328+0000 7fd912a00640 0 client.29009 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:46:08.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.328+0000 7fd936d74740 1 -- 172.21.15.90:0/3443890281 >> 172.21.15.90:0/3443890281 conn(0x563d11da0270 msgr2=0x563d11e05590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:46:08.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.328+0000 7fd936d74740 1 -- 172.21.15.90:0/3443890281 shutdown_connections 2024-04-02T14:46:08.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.328+0000 7fd936d74740 1 -- 172.21.15.90:0/3443890281 wait complete. 2024-04-02T14:46:08.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.328+0000 7fd936d74740 20 client.29009 trim_cache size 0 max 16384 2024-04-02T14:46:08.327 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-02T14:46:08.327 INFO:teuthology.orchestra.run.smithi090.stdout: "return_code": 0, 2024-04-02T14:46:08.327 INFO:teuthology.orchestra.run.smithi090.stdout: "scrub_tag": "ab1f6b44-0ab2-4a42-a230-d5ba5b727f3f", 2024-04-02T14:46:08.327 INFO:teuthology.orchestra.run.smithi090.stdout: "mode": "asynchronous" 2024-04-02T14:46:08.328 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-02T14:46:08.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.329+0000 7fd934000640 1 -- 172.21.15.90:0/1015617899 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fd9140a3e20 msgr2=0x7fd9140a62e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:08.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.329+0000 7fd934000640 1 --2- 172.21.15.90:0/1015617899 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fd9140a3e20 0x7fd9140a62e0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fd924014710 tx=0x7fd924004a40 comp rx=0 tx=0).stop 2024-04-02T14:46:08.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.329+0000 7fd934000640 1 -- 172.21.15.90:0/1015617899 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd92c100910 msgr2=0x7fd92c19a5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:08.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.329+0000 7fd934000640 1 --2- 172.21.15.90:0/1015617899 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd92c100910 0x7fd92c19a5d0 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7fd920005ab0 tx=0x7fd9200042f0 comp rx=0 tx=0).stop 2024-04-02T14:46:08.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.329+0000 7fd934000640 1 -- 172.21.15.90:0/1015617899 shutdown_connections 2024-04-02T14:46:08.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.329+0000 7fd934000640 1 --2- 172.21.15.90:0/1015617899 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fd9140a3e20 0x7fd9140a62e0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:08.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.329+0000 7fd934000640 1 --2- 172.21.15.90:0/1015617899 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd92c10e4c0 0x7fd92c19ee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:08.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.329+0000 7fd934000640 1 --2- 172.21.15.90:0/1015617899 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd92c101230 0x7fd92c19ab10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:08.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.329+0000 7fd934000640 1 --2- 172.21.15.90:0/1015617899 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd92c100910 0x7fd92c19a5d0 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:08.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.329+0000 7fd934000640 1 -- 172.21.15.90:0/1015617899 >> 172.21.15.90:0/1015617899 conn(0x7fd92c077b60 msgr2=0x7fd92c10f930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:46:08.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.329+0000 7fd934000640 1 -- 172.21.15.90:0/1015617899 shutdown_connections 2024-04-02T14:46:08.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.329+0000 7fd934000640 1 -- 172.21.15.90:0/1015617899 wait complete. 2024-04-02T14:46:08.340 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-02T14:46:08.340 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-02T14:46:08.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.571+0000 7fdd2ec00640 1 -- 172.21.15.90:0/2396183331 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdd281047a0 msgr2=0x7fdd281087e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:08.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.571+0000 7fdd2ec00640 1 --2- 172.21.15.90:0/2396183331 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdd281047a0 0x7fdd281087e0 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7fdd14009900 tx=0x7fdd1402bc30 comp rx=0 tx=0).stop 2024-04-02T14:46:08.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.571+0000 7fdd2ec00640 1 -- 172.21.15.90:0/2396183331 shutdown_connections 2024-04-02T14:46:08.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.571+0000 7fdd2ec00640 1 --2- 172.21.15.90:0/2396183331 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdd28108ec0 0x7fdd28110a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:08.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.571+0000 7fdd2ec00640 1 --2- 172.21.15.90:0/2396183331 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdd281047a0 0x7fdd281087e0 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:08.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.571+0000 7fdd2ec00640 1 --2- 172.21.15.90:0/2396183331 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdd28103df0 0x7fdd281041d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:08.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.571+0000 7fdd2ec00640 1 -- 172.21.15.90:0/2396183331 >> 172.21.15.90:0/2396183331 conn(0x7fdd280ff610 msgr2=0x7fdd28101a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:46:08.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.571+0000 7fdd2ec00640 1 -- 172.21.15.90:0/2396183331 shutdown_connections 2024-04-02T14:46:08.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.571+0000 7fdd2ec00640 1 -- 172.21.15.90:0/2396183331 wait complete. 2024-04-02T14:46:08.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.572+0000 7fdd2ec00640 1 Processor -- start 2024-04-02T14:46:08.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.572+0000 7fdd2ec00640 1 -- start start 2024-04-02T14:46:08.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.573+0000 7fdd2ec00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdd28103df0 0x7fdd2819a500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:08.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.573+0000 7fdd2ec00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdd28108ec0 0x7fdd2819aa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:08.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.573+0000 7fdd2ec00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdd2819ed70 0x7fdd2819f220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:08.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.573+0000 7fdd2ec00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fdd28113fc0 con 0x7fdd2819ed70 2024-04-02T14:46:08.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.573+0000 7fdd2ec00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fdd28113e40 con 0x7fdd28103df0 2024-04-02T14:46:08.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.573+0000 7fdd2ec00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fdd28114140 con 0x7fdd28108ec0 2024-04-02T14:46:08.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.573+0000 7fdd2d800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdd28103df0 0x7fdd2819a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:08.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.573+0000 7fdd2d800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdd28103df0 0x7fdd2819a500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:52770/0 (socket says 172.21.15.90:52770) 2024-04-02T14:46:08.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.573+0000 7fdd2d800640 1 -- 172.21.15.90:0/2893545894 learned_addr learned my addr 172.21.15.90:0/2893545894 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:46:08.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.573+0000 7fdd2ce00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdd28108ec0 0x7fdd2819aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:08.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.573+0000 7fdd2e200640 1 --2- 172.21.15.90:0/2893545894 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdd2819ed70 0x7fdd2819f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:08.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.573+0000 7fdd2ce00640 1 -- 172.21.15.90:0/2893545894 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdd28103df0 msgr2=0x7fdd2819a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:08.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.573+0000 7fdd2ce00640 1 --2- 172.21.15.90:0/2893545894 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdd28103df0 0x7fdd2819a500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:08.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.573+0000 7fdd2ce00640 1 -- 172.21.15.90:0/2893545894 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdd2819ed70 msgr2=0x7fdd2819f220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:08.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.573+0000 7fdd2ce00640 1 --2- 172.21.15.90:0/2893545894 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdd2819ed70 0x7fdd2819f220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:08.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.573+0000 7fdd2ce00640 1 -- 172.21.15.90:0/2893545894 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd2819f860 con 0x7fdd28108ec0 2024-04-02T14:46:08.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.573+0000 7fdd2d800640 1 --2- 172.21.15.90:0/2893545894 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdd28103df0 0x7fdd2819a500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:46:08.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.574+0000 7fdd2ce00640 1 --2- 172.21.15.90:0/2893545894 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdd28108ec0 0x7fdd2819aa40 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7fdd14036040 tx=0x7fdd14005e80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:46:08.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.574+0000 7fdd26000640 1 -- 172.21.15.90:0/2893545894 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd14043020 con 0x7fdd28108ec0 2024-04-02T14:46:08.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.574+0000 7fdd2ec00640 1 -- 172.21.15.90:0/2893545894 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd2819faf0 con 0x7fdd28108ec0 2024-04-02T14:46:08.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.574+0000 7fdd2ec00640 1 -- 172.21.15.90:0/2893545894 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd281a7310 con 0x7fdd28108ec0 2024-04-02T14:46:08.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.574+0000 7fdd26000640 1 -- 172.21.15.90:0/2893545894 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdd14004c80 con 0x7fdd28108ec0 2024-04-02T14:46:08.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.574+0000 7fdd26000640 1 -- 172.21.15.90:0/2893545894 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd140035f0 con 0x7fdd28108ec0 2024-04-02T14:46:08.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.577+0000 7fdd32de1740 1 Processor -- start 2024-04-02T14:46:08.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.577+0000 7fdd32de1740 1 -- start start 2024-04-02T14:46:08.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.578+0000 7fdd32de1740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5649c758d960 0x5649c758dd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:08.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.578+0000 7fdd32de1740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5649c758e280 0x5649c7596920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:08.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.578+0000 7fdd32de1740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5649c7596e60 0x5649c7599250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:08.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.578+0000 7fdd32de1740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5649c74f1770 con 0x5649c758e280 2024-04-02T14:46:08.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.578+0000 7fdd32de1740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x5649c7499700 con 0x5649c758d960 2024-04-02T14:46:08.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.578+0000 7fdd32de1740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5649c74ed720 con 0x5649c7596e60 2024-04-02T14:46:08.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.578+0000 7fdd26000640 1 -- 172.21.15.90:0/2893545894 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fdd14046020 con 0x7fdd28108ec0 2024-04-02T14:46:08.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.578+0000 7fdd2e200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5649c758d960 0x5649c758dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:08.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.578+0000 7fdd2ce00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5649c7596e60 0x5649c7599250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:08.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.578+0000 7fdd2ce00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5649c7596e60 0x5649c7599250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:58540/0 (socket says 172.21.15.90:58540) 2024-04-02T14:46:08.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.578+0000 7fdd2e200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5649c758d960 0x5649c758dd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:52782/0 (socket says 172.21.15.90:52782) 2024-04-02T14:46:08.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.578+0000 7fdd2ce00640 1 -- 172.21.15.90:0/985796420 learned_addr learned my addr 172.21.15.90:0/985796420 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:46:08.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.578+0000 7fdd2d800640 1 --2- 172.21.15.90:0/985796420 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5649c758e280 0x5649c7596920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:08.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.579+0000 7fdd2ce00640 1 -- 172.21.15.90:0/985796420 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5649c758d960 msgr2=0x5649c758dd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:46:08.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.579+0000 7fdd2ce00640 1 --2- 172.21.15.90:0/985796420 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5649c758d960 0x5649c758dd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:08.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.579+0000 7fdd2ce00640 1 -- 172.21.15.90:0/985796420 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5649c758e280 msgr2=0x5649c7596920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:46:08.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.579+0000 7fdd2ce00640 1 --2- 172.21.15.90:0/985796420 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5649c758e280 0x5649c7596920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:08.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.579+0000 7fdd2ce00640 1 -- 172.21.15.90:0/985796420 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5649c72cf7d0 con 0x5649c7596e60 2024-04-02T14:46:08.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.579+0000 7fdd2d800640 1 --2- 172.21.15.90:0/985796420 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5649c758e280 0x5649c7596920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:46:08.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.579+0000 7fdd2ce00640 1 --2- 172.21.15.90:0/985796420 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5649c7596e60 0x5649c7599250 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7fdd140ef5f0 tx=0x7fdd140f5f10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=517f7e8acc9d0f60 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:46:08.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.579+0000 7fdd1aa00640 1 -- 172.21.15.90:0/985796420 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd140fac30 con 0x5649c7596e60 2024-04-02T14:46:08.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.579+0000 7fdd1aa00640 1 -- 172.21.15.90:0/985796420 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdd140ff070 con 0x5649c7596e60 2024-04-02T14:46:08.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.579+0000 7fdd1aa00640 1 -- 172.21.15.90:0/985796420 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd14102670 con 0x5649c7596e60 2024-04-02T14:46:08.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.580+0000 7fdd26000640 1 --2- 172.21.15.90:0/2893545894 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fdd100a3d20 0x7fdd100a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:08.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.580+0000 7fdd32de1740 1 -- 172.21.15.90:0/985796420 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5649c7596e60 msgr2=0x5649c7599250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:08.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.580+0000 7fdd32de1740 1 --2- 172.21.15.90:0/985796420 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5649c7596e60 0x5649c7599250 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7fdd140ef5f0 tx=0x7fdd140f5f10 comp rx=0 tx=0).stop 2024-04-02T14:46:08.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.580+0000 7fdd26000640 1 -- 172.21.15.90:0/2893545894 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 8594+0+0 (secure 0 0 0) 0x7fdd140f1f00 con 0x7fdd28108ec0 2024-04-02T14:46:08.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.580+0000 7fdd2e200640 1 --2- 172.21.15.90:0/2893545894 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fdd100a3d20 0x7fdd100a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:08.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.580+0000 7fdd32de1740 1 -- 172.21.15.90:0/985796420 shutdown_connections 2024-04-02T14:46:08.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.580+0000 7fdd32de1740 1 --2- 172.21.15.90:0/985796420 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5649c7596e60 0x5649c7599250 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:08.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.580+0000 7fdd32de1740 1 --2- 172.21.15.90:0/985796420 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5649c758e280 0x5649c7596920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:08.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.580+0000 7fdd32de1740 1 --2- 172.21.15.90:0/985796420 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5649c758d960 0x5649c758dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:08.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.580+0000 7fdd32de1740 1 -- 172.21.15.90:0/985796420 >> 172.21.15.90:0/985796420 conn(0x5649c7588090 msgr2=0x5649c7588470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:46:08.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.580+0000 7fdd32de1740 1 -- 172.21.15.90:0/985796420 shutdown_connections 2024-04-02T14:46:08.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.580+0000 7fdd32de1740 1 -- 172.21.15.90:0/985796420 wait complete. 2024-04-02T14:46:08.582 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.581+0000 7fdd2e200640 1 --2- 172.21.15.90:0/2893545894 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fdd100a3d20 0x7fdd100a61e0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fdd20005fd0 tx=0x7fdd20009500 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:46:08.582 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.581+0000 7fdd32de1740 1 -- start start 2024-04-02T14:46:08.582 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.581+0000 7fdd32de1740 1 Processor -- start 2024-04-02T14:46:08.583 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.581+0000 7fdd32de1740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5649c758d960 0x5649c7593e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:08.583 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.581+0000 7fdd32de1740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5649c758e280 0x5649c75943a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:08.583 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.581+0000 7fdd32de1740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5649c7596e60 0x5649c75948e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:08.583 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.581+0000 7fdd32de1740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5649c7599a10 con 0x5649c758d960 2024-04-02T14:46:08.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.581+0000 7fdd32de1740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x5649c74acac0 con 0x5649c758e280 2024-04-02T14:46:08.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.581+0000 7fdd32de1740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5649c7599b90 con 0x5649c7596e60 2024-04-02T14:46:08.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.582+0000 7fdd32de1740 10 client.0 _ll_register_callbacks cb 0x5649c74af180 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:46:08.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.582+0000 7fdd2ce00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5649c758e280 0x5649c75943a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:08.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.582+0000 7fdd2e200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5649c7596e60 0x5649c75948e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:08.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.582+0000 7fdd2ce00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5649c758e280 0x5649c75943a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:52794/0 (socket says 172.21.15.90:52794) 2024-04-02T14:46:08.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.582+0000 7fdd2ce00640 1 -- 172.21.15.90:0/41185236 learned_addr learned my addr 172.21.15.90:0/41185236 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:46:08.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.582+0000 7fdd2d800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5649c758d960 0x5649c7593e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:08.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.582+0000 7fdd2ce00640 1 -- 172.21.15.90:0/41185236 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5649c7596e60 msgr2=0x5649c75948e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:46:08.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.582+0000 7fdd2ce00640 1 --2- 172.21.15.90:0/41185236 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5649c7596e60 0x5649c75948e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:08.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.582+0000 7fdd2ce00640 1 -- 172.21.15.90:0/41185236 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5649c758d960 msgr2=0x5649c7593e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:46:08.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.582+0000 7fdd2ce00640 1 --2- 172.21.15.90:0/41185236 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5649c758d960 0x5649c7593e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:08.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.582+0000 7fdd2ce00640 1 -- 172.21.15.90:0/41185236 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5649c72053f0 con 0x5649c758e280 2024-04-02T14:46:08.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.582+0000 7fdd2e200640 1 --2- 172.21.15.90:0/41185236 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5649c7596e60 0x5649c75948e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:46:08.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.582+0000 7fdd2ce00640 1 --2- 172.21.15.90:0/41185236 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5649c758e280 0x5649c75943a0 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7fdd140ba360 tx=0x7fdd1402bc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b2853117c2a23d3f server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:46:08.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.582+0000 7fdd0f400640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:46:08.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.583+0000 7fdd0f400640 1 -- 172.21.15.90:0/41185236 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd140ba1a0 con 0x5649c758e280 2024-04-02T14:46:08.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.583+0000 7fdd0f400640 1 -- 172.21.15.90:0/41185236 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 8594+0+0 (secure 0 0 0) 0x7fdd140ef820 con 0x5649c758e280 2024-04-02T14:46:08.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.583+0000 7fdd32de1740 1 -- 172.21.15.90:0/41185236 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdd140ff070 con 0x5649c758e280 2024-04-02T14:46:08.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.583+0000 7fdd0f400640 1 -- 172.21.15.90:0/41185236 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fdd140bd860 con 0x5649c758e280 2024-04-02T14:46:08.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.583+0000 7fdd32de1740 10 client.29021 fetch_fsmap learned FSMap version 27 2024-04-02T14:46:08.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.583+0000 7fdd32de1740 1 -- 172.21.15.90:0/41185236 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5649c71c50f0 con 0x5649c758e280 2024-04-02T14:46:08.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.583+0000 7fdd0f400640 1 -- 172.21.15.90:0/41185236 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdd140ff070 con 0x5649c758e280 2024-04-02T14:46:08.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.583+0000 7fdd0f400640 1 -- 172.21.15.90:0/41185236 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fdd140f3030 con 0x5649c758e280 2024-04-02T14:46:08.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.583+0000 7fdd32de1740 10 client.29021 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:46:08.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.583+0000 7fdd32de1740 10 client.29021 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:46:08.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.583+0000 7fdd32de1740 20 client.29021 populate_metadata read hostname 'smithi090' 2024-04-02T14:46:08.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.583+0000 7fdd32de1740 1 --2- 172.21.15.90:0/41185236 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5649c75a6a60 0x5649c75a8e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:08.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.583+0000 7fdd32de1740 4 client.29021 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:46:08.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.583+0000 7fdd32de1740 1 -- 172.21.15.90:0/41185236 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5649c7492630 con 0x5649c75a6a60 2024-04-02T14:46:08.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.584+0000 7fdd2d800640 1 --2- 172.21.15.90:0/41185236 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5649c75a6a60 0x5649c75a8e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:08.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.584+0000 7fdd2d800640 1 --2- 172.21.15.90:0/41185236 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5649c75a6a60 0x5649c75a8e50 crc :-1 s=READY pgs=298 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5bb3fddb1cf0189d server_cookie=7a8f76ced7460acc in_seq=0 out_seq=0 2024-04-02T14:46:08.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.584+0000 7fdd0f400640 10 client.29021 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:46:08.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.585+0000 7fdd0f400640 1 -- 172.21.15.90:0/41185236 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5649c7492630 con 0x5649c75a6a60 2024-04-02T14:46:08.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.585+0000 7fdd0f400640 10 client.29021 handle_command_reply: tid=0 2024-04-02T14:46:08.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.585+0000 7fdd32de1740 1 client.29021 shutdown 2024-04-02T14:46:08.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.585+0000 7fdd32de1740 1 -- 172.21.15.90:0/41185236 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5649c758e280 msgr2=0x5649c75943a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:08.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.585+0000 7fdd32de1740 1 --2- 172.21.15.90:0/41185236 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5649c758e280 0x5649c75943a0 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7fdd140ba360 tx=0x7fdd1402bc30 comp rx=0 tx=0).stop 2024-04-02T14:46:08.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.585+0000 7fdd32de1740 1 -- 172.21.15.90:0/41185236 shutdown_connections 2024-04-02T14:46:08.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.585+0000 7fdd32de1740 1 --2- 172.21.15.90:0/41185236 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5649c75a6a60 0x5649c75a8e50 crc :-1 s=READY pgs=298 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:08.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.585+0000 7fdd32de1740 1 --2- 172.21.15.90:0/41185236 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5649c7596e60 0x5649c75948e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:08.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.585+0000 7fdd32de1740 1 --2- 172.21.15.90:0/41185236 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5649c758e280 0x5649c75943a0 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:08.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.585+0000 7fdd32de1740 1 --2- 172.21.15.90:0/41185236 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5649c758d960 0x5649c7593e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:08.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.585+0000 7fdd32de1740 1 -- 172.21.15.90:0/41185236 >> 172.21.15.90:0/41185236 conn(0x5649c7588090 msgr2=0x5649c7589de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:46:08.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.585+0000 7fdd0f400640 0 client.29021 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:46:08.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.585+0000 7fdd32de1740 1 -- 172.21.15.90:0/41185236 shutdown_connections 2024-04-02T14:46:08.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.585+0000 7fdd32de1740 1 -- 172.21.15.90:0/41185236 wait complete. 2024-04-02T14:46:08.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.586+0000 7fdd32de1740 20 client.29021 trim_cache size 0 max 16384 2024-04-02T14:46:08.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.608+0000 7fdd32de1740 1 Processor -- start 2024-04-02T14:46:08.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.608+0000 7fdd32de1740 1 -- start start 2024-04-02T14:46:08.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.608+0000 7fdd32de1740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5649c758d960 0x5649c7596590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:08.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.608+0000 7fdd32de1740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5649c758e280 0x5649c75a28d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:08.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.608+0000 7fdd32de1740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5649c7596e60 0x5649c7587190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:08.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.608+0000 7fdd32de1740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fdd0800e820 con 0x5649c758d960 2024-04-02T14:46:08.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.608+0000 7fdd32de1740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fdd08010360 con 0x5649c7596e60 2024-04-02T14:46:08.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.608+0000 7fdd32de1740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fdd0800eef0 con 0x5649c758e280 2024-04-02T14:46:08.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.608+0000 7fdd2e200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5649c7596e60 0x5649c7587190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:08.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.608+0000 7fdd2ce00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5649c758e280 0x5649c75a28d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:08.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.608+0000 7fdd2d800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5649c758d960 0x5649c7596590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:08.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.608+0000 7fdd2e200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5649c7596e60 0x5649c7587190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:52796/0 (socket says 172.21.15.90:52796) 2024-04-02T14:46:08.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.608+0000 7fdd2e200640 1 -- 172.21.15.90:0/1030607764 learned_addr learned my addr 172.21.15.90:0/1030607764 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:46:08.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.608+0000 7fdd2ce00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5649c758e280 0x5649c75a28d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:58560/0 (socket says 172.21.15.90:58560) 2024-04-02T14:46:08.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.609+0000 7fdd2e200640 1 -- 172.21.15.90:0/1030607764 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5649c758e280 msgr2=0x5649c75a28d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:46:08.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.609+0000 7fdd2e200640 1 --2- 172.21.15.90:0/1030607764 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5649c758e280 0x5649c75a28d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:08.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.609+0000 7fdd2e200640 1 -- 172.21.15.90:0/1030607764 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5649c758d960 msgr2=0x5649c7596590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:46:08.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.609+0000 7fdd2e200640 1 --2- 172.21.15.90:0/1030607764 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5649c758d960 0x5649c7596590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:08.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.609+0000 7fdd2e200640 1 -- 172.21.15.90:0/1030607764 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5649c758e0b0 con 0x5649c7596e60 2024-04-02T14:46:08.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.609+0000 7fdd2e200640 1 --2- 172.21.15.90:0/1030607764 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5649c7596e60 0x5649c7587190 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7fdd200042b0 tx=0x7fdd2001a000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2a68af3553eb731e server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:46:08.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.609+0000 7fdd1a000640 1 -- 172.21.15.90:0/1030607764 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd2000fa00 con 0x5649c7596e60 2024-04-02T14:46:08.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.609+0000 7fdd1a000640 1 -- 172.21.15.90:0/1030607764 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdd2000fba0 con 0x5649c7596e60 2024-04-02T14:46:08.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.609+0000 7fdd1a000640 1 -- 172.21.15.90:0/1030607764 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd2001e690 con 0x5649c7596e60 2024-04-02T14:46:08.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.610+0000 7fdd32de1740 1 -- 172.21.15.90:0/1030607764 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5649c7596e60 msgr2=0x5649c7587190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:08.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.610+0000 7fdd32de1740 1 --2- 172.21.15.90:0/1030607764 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5649c7596e60 0x5649c7587190 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7fdd200042b0 tx=0x7fdd2001a000 comp rx=0 tx=0).stop 2024-04-02T14:46:08.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.610+0000 7fdd32de1740 1 -- 172.21.15.90:0/1030607764 shutdown_connections 2024-04-02T14:46:08.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.610+0000 7fdd32de1740 1 --2- 172.21.15.90:0/1030607764 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5649c7596e60 0x5649c7587190 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:08.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.610+0000 7fdd32de1740 1 --2- 172.21.15.90:0/1030607764 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5649c758e280 0x5649c75a28d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:08.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.610+0000 7fdd32de1740 1 --2- 172.21.15.90:0/1030607764 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5649c758d960 0x5649c7596590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:08.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.610+0000 7fdd32de1740 1 -- 172.21.15.90:0/1030607764 >> 172.21.15.90:0/1030607764 conn(0x5649c7588090 msgr2=0x5649c7586a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:46:08.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.610+0000 7fdd32de1740 1 -- 172.21.15.90:0/1030607764 shutdown_connections 2024-04-02T14:46:08.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.610+0000 7fdd32de1740 1 -- 172.21.15.90:0/1030607764 wait complete. 2024-04-02T14:46:08.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.611+0000 7fdd32de1740 1 -- start start 2024-04-02T14:46:08.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.611+0000 7fdd32de1740 1 Processor -- start 2024-04-02T14:46:08.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.611+0000 7fdd32de1740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5649c758d960 0x5649c759af20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:08.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.611+0000 7fdd32de1740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5649c758e280 0x5649c759b460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:08.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.611+0000 7fdd32de1740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5649c7596e60 0x5649c760be70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:08.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.612+0000 7fdd32de1740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fdd0800fc90 con 0x5649c758d960 2024-04-02T14:46:08.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.612+0000 7fdd32de1740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fdd0800f5c0 con 0x5649c7596e60 2024-04-02T14:46:08.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.612+0000 7fdd32de1740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fdd0800c820 con 0x5649c758e280 2024-04-02T14:46:08.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.612+0000 7fdd32de1740 10 client.0 _ll_register_callbacks cb 0x7fdd20009b30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:46:08.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.612+0000 7fdd2e200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5649c7596e60 0x5649c760be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:08.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.612+0000 7fdd2e200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5649c7596e60 0x5649c760be70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:52806/0 (socket says 172.21.15.90:52806) 2024-04-02T14:46:08.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.612+0000 7fdd2e200640 1 -- 172.21.15.90:0/1612875290 learned_addr learned my addr 172.21.15.90:0/1612875290 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:46:08.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.612+0000 7fdd2d800640 1 --2- 172.21.15.90:0/1612875290 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5649c758d960 0x5649c759af20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:08.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.612+0000 7fdd2ce00640 1 --2- 172.21.15.90:0/1612875290 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5649c758e280 0x5649c759b460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:08.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.613+0000 7fdd2e200640 1 -- 172.21.15.90:0/1612875290 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5649c758e280 msgr2=0x5649c759b460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:46:08.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.613+0000 7fdd2e200640 1 --2- 172.21.15.90:0/1612875290 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5649c758e280 0x5649c759b460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:08.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.613+0000 7fdd2e200640 1 -- 172.21.15.90:0/1612875290 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5649c758d960 msgr2=0x5649c759af20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:46:08.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.613+0000 7fdd2e200640 1 --2- 172.21.15.90:0/1612875290 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5649c758d960 0x5649c759af20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:08.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.613+0000 7fdd2e200640 1 -- 172.21.15.90:0/1612875290 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5649c760c3b0 con 0x5649c7596e60 2024-04-02T14:46:08.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.613+0000 7fdd2ce00640 1 --2- 172.21.15.90:0/1612875290 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5649c758e280 0x5649c759b460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:46:08.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.613+0000 7fdd2e200640 1 --2- 172.21.15.90:0/1612875290 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5649c7596e60 0x5649c760be70 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7fdd20000f80 tx=0x7fdd2000fa00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=538a09e5c8f8eac6 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:46:08.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.613+0000 7fdd0f400640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:46:08.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.613+0000 7fdd0f400640 1 -- 172.21.15.90:0/1612875290 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd20004c40 con 0x5649c7596e60 2024-04-02T14:46:08.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.613+0000 7fdd32de1740 1 -- 172.21.15.90:0/1612875290 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdd2000fba0 con 0x5649c7596e60 2024-04-02T14:46:08.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.613+0000 7fdd0f400640 1 -- 172.21.15.90:0/1612875290 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 8594+0+0 (secure 0 0 0) 0x7fdd20004de0 con 0x5649c7596e60 2024-04-02T14:46:08.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.614+0000 7fdd0f400640 1 -- 172.21.15.90:0/1612875290 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fdd20029480 con 0x5649c7596e60 2024-04-02T14:46:08.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.614+0000 7fdd32de1740 10 client.29027 fetch_fsmap learned FSMap version 27 2024-04-02T14:46:08.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.614+0000 7fdd32de1740 1 -- 172.21.15.90:0/1612875290 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5649c760d6c0 con 0x5649c7596e60 2024-04-02T14:46:08.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.614+0000 7fdd0f400640 1 -- 172.21.15.90:0/1612875290 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdd2000fba0 con 0x5649c7596e60 2024-04-02T14:46:08.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.615+0000 7fdd0f400640 1 -- 172.21.15.90:0/1612875290 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fdd20030070 con 0x5649c7596e60 2024-04-02T14:46:08.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.615+0000 7fdd32de1740 10 client.29027 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:46:08.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.615+0000 7fdd32de1740 10 client.29027 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:46:08.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.615+0000 7fdd32de1740 20 client.29027 populate_metadata read hostname 'smithi090' 2024-04-02T14:46:08.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.615+0000 7fdd32de1740 1 --2- 172.21.15.90:0/1612875290 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5649c760dc70 0x5649c7610090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:08.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.615+0000 7fdd32de1740 4 client.29027 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-02T14:46:08.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.615+0000 7fdd32de1740 1 -- 172.21.15.90:0/1612875290 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5649c74fecb0 con 0x5649c760dc70 2024-04-02T14:46:08.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.615+0000 7fdd2d800640 1 --2- 172.21.15.90:0/1612875290 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5649c760dc70 0x5649c7610090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:08.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.616+0000 7fdd0f400640 10 client.29027 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:46:08.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.616+0000 7fdd2d800640 1 --2- 172.21.15.90:0/1612875290 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5649c760dc70 0x5649c7610090 crc :-1 s=READY pgs=299 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6b4e56f21b24ae44 server_cookie=a3ce714a7da89440 in_seq=0 out_seq=0 2024-04-02T14:46:08.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.616+0000 7fdd0f400640 1 -- 172.21.15.90:0/1612875290 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+267 (crc 0 0 0) 0x5649c74fecb0 con 0x5649c760dc70 2024-04-02T14:46:08.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.616+0000 7fdd0f400640 10 client.29027 handle_command_reply: tid=0 2024-04-02T14:46:08.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.616+0000 7fdd32de1740 1 client.29027 shutdown 2024-04-02T14:46:08.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.616+0000 7fdd32de1740 1 -- 172.21.15.90:0/1612875290 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5649c7596e60 msgr2=0x5649c760be70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:08.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.616+0000 7fdd32de1740 1 --2- 172.21.15.90:0/1612875290 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5649c7596e60 0x5649c760be70 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7fdd20000f80 tx=0x7fdd2000fa00 comp rx=0 tx=0).stop 2024-04-02T14:46:08.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.616+0000 7fdd32de1740 1 -- 172.21.15.90:0/1612875290 shutdown_connections 2024-04-02T14:46:08.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.616+0000 7fdd32de1740 1 --2- 172.21.15.90:0/1612875290 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5649c760dc70 0x5649c7610090 crc :-1 s=READY pgs=299 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:08.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.616+0000 7fdd32de1740 1 --2- 172.21.15.90:0/1612875290 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5649c7596e60 0x5649c760be70 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:08.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.616+0000 7fdd0f400640 0 client.29027 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:46:08.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.616+0000 7fdd32de1740 1 --2- 172.21.15.90:0/1612875290 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5649c758e280 0x5649c759b460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:08.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.616+0000 7fdd32de1740 1 --2- 172.21.15.90:0/1612875290 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5649c758d960 0x5649c759af20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:08.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.616+0000 7fdd32de1740 1 -- 172.21.15.90:0/1612875290 >> 172.21.15.90:0/1612875290 conn(0x5649c7588090 msgr2=0x5649c7587310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:46:08.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.617+0000 7fdd32de1740 1 -- 172.21.15.90:0/1612875290 shutdown_connections 2024-04-02T14:46:08.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.617+0000 7fdd32de1740 1 -- 172.21.15.90:0/1612875290 wait complete. 2024-04-02T14:46:08.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.617+0000 7fdd32de1740 20 client.29027 trim_cache size 0 max 16384 2024-04-02T14:46:08.616 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-02T14:46:08.616 INFO:teuthology.orchestra.run.smithi090.stdout: "status": "scrub active (316 inodes in the stack)", 2024-04-02T14:46:08.616 INFO:teuthology.orchestra.run.smithi090.stdout: "scrubs": { 2024-04-02T14:46:08.616 INFO:teuthology.orchestra.run.smithi090.stdout: "ab1f6b44-0ab2-4a42-a230-d5ba5b727f3f": { 2024-04-02T14:46:08.616 INFO:teuthology.orchestra.run.smithi090.stdout: "path": "/", 2024-04-02T14:46:08.616 INFO:teuthology.orchestra.run.smithi090.stdout: "tag": "ab1f6b44-0ab2-4a42-a230-d5ba5b727f3f", 2024-04-02T14:46:08.616 INFO:teuthology.orchestra.run.smithi090.stdout: "options": "recursive,force" 2024-04-02T14:46:08.616 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:46:08.616 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:46:08.616 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-02T14:46:08.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.617+0000 7fdd2ec00640 1 -- 172.21.15.90:0/2893545894 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fdd100a3d20 msgr2=0x7fdd100a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:08.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.617+0000 7fdd2ec00640 1 --2- 172.21.15.90:0/2893545894 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fdd100a3d20 0x7fdd100a61e0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fdd20005fd0 tx=0x7fdd20009500 comp rx=0 tx=0).stop 2024-04-02T14:46:08.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.617+0000 7fdd2ec00640 1 -- 172.21.15.90:0/2893545894 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdd28108ec0 msgr2=0x7fdd2819aa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:08.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.617+0000 7fdd2ec00640 1 --2- 172.21.15.90:0/2893545894 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdd28108ec0 0x7fdd2819aa40 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7fdd14036040 tx=0x7fdd14005e80 comp rx=0 tx=0).stop 2024-04-02T14:46:08.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.617+0000 7fdd2ec00640 1 -- 172.21.15.90:0/2893545894 shutdown_connections 2024-04-02T14:46:08.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.617+0000 7fdd2ec00640 1 --2- 172.21.15.90:0/2893545894 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fdd100a3d20 0x7fdd100a61e0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:08.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.617+0000 7fdd2ec00640 1 --2- 172.21.15.90:0/2893545894 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdd2819ed70 0x7fdd2819f220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:08.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.617+0000 7fdd2ec00640 1 --2- 172.21.15.90:0/2893545894 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdd28108ec0 0x7fdd2819aa40 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:08.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.617+0000 7fdd2ec00640 1 --2- 172.21.15.90:0/2893545894 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdd28103df0 0x7fdd2819a500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:08.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.617+0000 7fdd2ec00640 1 -- 172.21.15.90:0/2893545894 >> 172.21.15.90:0/2893545894 conn(0x7fdd280ff610 msgr2=0x7fdd28101960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:46:08.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.618+0000 7fdd2ec00640 1 -- 172.21.15.90:0/2893545894 shutdown_connections 2024-04-02T14:46:08.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:08.618+0000 7fdd2ec00640 1 -- 172.21.15.90:0/2893545894 wait complete. 2024-04-02T14:46:08.638 INFO:tasks.cephfs.filesystem:scrub status for tag:ab1f6b44-0ab2-4a42-a230-d5ba5b727f3f - {'path': '/', 'tag': 'ab1f6b44-0ab2-4a42-a230-d5ba5b727f3f', 'options': 'recursive,force'} 2024-04-02T14:46:10.104 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:46:09 smithi092 ceph-mon[36497]: scrub queued for path: / 2024-04-02T14:46:10.105 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:46:09 smithi092 ceph-mon[36497]: scrub summary: idle+waiting paths [/] 2024-04-02T14:46:10.105 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:46:09 smithi092 ceph-mon[36497]: scrub summary: active paths [/] 2024-04-02T14:46:10.105 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:46:09 smithi092 ceph-mon[36497]: pgmap v350: 97 pgs: 2 active+clean+snaptrim, 5 active+clean+snaptrim_wait, 90 active+clean; 4.7 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 10 MiB/s wr, 1.09k op/s 2024-04-02T14:46:10.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:46:09 smithi177 ceph-mon[36431]: scrub queued for path: / 2024-04-02T14:46:10.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:46:09 smithi177 ceph-mon[36431]: scrub summary: idle+waiting paths [/] 2024-04-02T14:46:10.181 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:46:09 smithi177 ceph-mon[36431]: scrub summary: active paths [/] 2024-04-02T14:46:10.181 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:46:09 smithi177 ceph-mon[36431]: pgmap v350: 97 pgs: 2 active+clean+snaptrim, 5 active+clean+snaptrim_wait, 90 active+clean; 4.7 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 10 MiB/s wr, 1.09k op/s 2024-04-02T14:46:10.182 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:46:09 smithi090 ceph-mon[31674]: scrub queued for path: / 2024-04-02T14:46:10.182 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:46:09 smithi090 ceph-mon[31674]: scrub summary: idle+waiting paths [/] 2024-04-02T14:46:10.182 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:46:09 smithi090 ceph-mon[31674]: scrub summary: active paths [/] 2024-04-02T14:46:10.182 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:46:09 smithi090 ceph-mon[31674]: pgmap v350: 97 pgs: 2 active+clean+snaptrim, 5 active+clean+snaptrim_wait, 90 active+clean; 4.7 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 10 MiB/s wr, 1.09k op/s 2024-04-02T14:46:11.104 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:46:10 smithi092 ceph-mon[36497]: scrub summary: idle+waiting paths [/] 2024-04-02T14:46:11.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:46:10 smithi177 ceph-mon[36431]: scrub summary: idle+waiting paths [/] 2024-04-02T14:46:11.182 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:46:10 smithi090 ceph-mon[31674]: scrub summary: idle+waiting paths [/] 2024-04-02T14:46:11.827 INFO:teuthology.orchestra.run.smithi090.stderr:progress: 640.0 s, 30.0 tps, lat 1641.387 ms stddev 1039.182 2024-04-02T14:46:12.104 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:46:11 smithi092 ceph-mon[36497]: pgmap v351: 97 pgs: 22 active+clean+snaptrim, 40 active+clean+snaptrim_wait, 35 active+clean; 4.5 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 15 MiB/s wr, 1.58k op/s 2024-04-02T14:46:12.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:46:11 smithi177 ceph-mon[36431]: pgmap v351: 97 pgs: 22 active+clean+snaptrim, 40 active+clean+snaptrim_wait, 35 active+clean; 4.5 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 15 MiB/s wr, 1.58k op/s 2024-04-02T14:46:12.182 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:46:11 smithi090 ceph-mon[31674]: pgmap v351: 97 pgs: 22 active+clean+snaptrim, 40 active+clean+snaptrim_wait, 35 active+clean; 4.5 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 15 MiB/s wr, 1.58k op/s 2024-04-02T14:46:13.083 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:46:12 smithi090 ceph-mon[31674]: scrub summary: idle 2024-04-02T14:46:13.104 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:46:12 smithi092 ceph-mon[36497]: scrub summary: idle 2024-04-02T14:46:13.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:46:12 smithi177 ceph-mon[36431]: scrub summary: idle 2024-04-02T14:46:14.104 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:46:13 smithi092 ceph-mon[36497]: pgmap v352: 97 pgs: 22 active+clean+snaptrim, 40 active+clean+snaptrim_wait, 35 active+clean; 4.5 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 15 MiB/s wr, 1.58k op/s 2024-04-02T14:46:14.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:46:13 smithi177 ceph-mon[36431]: pgmap v352: 97 pgs: 22 active+clean+snaptrim, 40 active+clean+snaptrim_wait, 35 active+clean; 4.5 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 15 MiB/s wr, 1.58k op/s 2024-04-02T14:46:14.182 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:46:13 smithi090 ceph-mon[31674]: pgmap v352: 97 pgs: 22 active+clean+snaptrim, 40 active+clean+snaptrim_wait, 35 active+clean; 4.5 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 15 MiB/s wr, 1.58k op/s 2024-04-02T14:46:16.354 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:46:16 smithi092 ceph-mon[36497]: pgmap v353: 97 pgs: 97 active+clean; 4.2 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 10 MiB/s wr, 1.10k op/s 2024-04-02T14:46:16.355 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:46:16 smithi092 ceph-mon[36497]: osdmap e101: 12 total, 12 up, 12 in 2024-04-02T14:46:16.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:46:16 smithi177 ceph-mon[36431]: pgmap v353: 97 pgs: 97 active+clean; 4.2 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 10 MiB/s wr, 1.10k op/s 2024-04-02T14:46:16.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:46:16 smithi177 ceph-mon[36431]: osdmap e101: 12 total, 12 up, 12 in 2024-04-02T14:46:16.432 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:46:16 smithi090 ceph-mon[31674]: pgmap v353: 97 pgs: 97 active+clean; 4.2 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 10 MiB/s wr, 1.10k op/s 2024-04-02T14:46:16.432 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:46:16 smithi090 ceph-mon[31674]: osdmap e101: 12 total, 12 up, 12 in 2024-04-02T14:46:16.825 INFO:teuthology.orchestra.run.smithi090.stderr:progress: 645.0 s, 1240.7 tps, lat 123.938 ms stddev 482.494 2024-04-02T14:46:18.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:46:18 smithi177 ceph-mon[36431]: pgmap v355: 97 pgs: 97 active+clean; 4.2 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 10 MiB/s wr, 1.11k op/s 2024-04-02T14:46:18.433 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:46:18 smithi090 ceph-mon[31674]: pgmap v355: 97 pgs: 97 active+clean; 4.2 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 10 MiB/s wr, 1.11k op/s 2024-04-02T14:46:18.604 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:46:18 smithi092 ceph-mon[36497]: pgmap v355: 97 pgs: 97 active+clean; 4.2 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 10 MiB/s wr, 1.11k op/s 2024-04-02T14:46:19.121 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.125+0000 7f44eb800640 1 -- 172.21.15.90:0/4153909780 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f44e4103f20 msgr2=0x7f44e4104300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:19.121 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.125+0000 7f44eb800640 1 --2- 172.21.15.90:0/4153909780 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f44e4103f20 0x7f44e4104300 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f44d4009900 tx=0x7f44d402bc40 comp rx=0 tx=0).stop 2024-04-02T14:46:19.123 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.127+0000 7f44eb800640 1 -- 172.21.15.90:0/4153909780 shutdown_connections 2024-04-02T14:46:19.123 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.127+0000 7f44eb800640 1 --2- 172.21.15.90:0/4153909780 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f44e4108ff0 0x7f44e4110bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:19.124 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.127+0000 7f44eb800640 1 --2- 172.21.15.90:0/4153909780 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f44e41048d0 0x7f44e4108910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:19.124 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.127+0000 7f44eb800640 1 --2- 172.21.15.90:0/4153909780 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f44e4103f20 0x7f44e4104300 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:19.124 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.127+0000 7f44eb800640 1 -- 172.21.15.90:0/4153909780 >> 172.21.15.90:0/4153909780 conn(0x7f44e40ff780 msgr2=0x7f44e4101ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:46:19.124 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.128+0000 7f44eb800640 1 -- 172.21.15.90:0/4153909780 shutdown_connections 2024-04-02T14:46:19.124 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.128+0000 7f44eb800640 1 -- 172.21.15.90:0/4153909780 wait complete. 2024-04-02T14:46:19.126 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.131+0000 7f44eb800640 1 Processor -- start 2024-04-02T14:46:19.127 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.131+0000 7f44eb800640 1 -- start start 2024-04-02T14:46:19.128 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.132+0000 7f44eb800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f44e4103f20 0x7f44e411a210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:19.128 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.132+0000 7f44eb800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f44e4108ff0 0x7f44e4110030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:19.128 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.132+0000 7f44eb800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f44e4110570 0x7f44e4110a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:19.128 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.132+0000 7f44eb800640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f44e4077ce0 con 0x7f44e4108ff0 2024-04-02T14:46:19.128 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.132+0000 7f44eb800640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f44e4077b60 con 0x7f44e4103f20 2024-04-02T14:46:19.129 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.132+0000 7f44eb800640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f44e4077e60 con 0x7f44e4110570 2024-04-02T14:46:19.131 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.132+0000 7f44e9800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f44e4110570 0x7f44e4110a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:19.131 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.133+0000 7f44e9800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f44e4110570 0x7f44e4110a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:53834/0 (socket says 172.21.15.90:53834) 2024-04-02T14:46:19.131 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.133+0000 7f44e9800640 1 -- 172.21.15.90:0/900189990 learned_addr learned my addr 172.21.15.90:0/900189990 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:46:19.131 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.133+0000 7f44e8e00640 1 --2- 172.21.15.90:0/900189990 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f44e4103f20 0x7f44e411a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:19.131 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.133+0000 7f44dbe00640 1 --2- 172.21.15.90:0/900189990 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f44e4108ff0 0x7f44e4110030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:19.132 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.133+0000 7f44e9800640 1 -- 172.21.15.90:0/900189990 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f44e4103f20 msgr2=0x7f44e411a210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:19.132 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.133+0000 7f44e9800640 1 --2- 172.21.15.90:0/900189990 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f44e4103f20 0x7f44e411a210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:19.132 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.133+0000 7f44e9800640 1 -- 172.21.15.90:0/900189990 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f44e4108ff0 msgr2=0x7f44e4110030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:19.132 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.133+0000 7f44e9800640 1 --2- 172.21.15.90:0/900189990 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f44e4108ff0 0x7f44e4110030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:19.132 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.133+0000 7f44e9800640 1 -- 172.21.15.90:0/900189990 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44e4115d40 con 0x7f44e4110570 2024-04-02T14:46:19.132 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.133+0000 7f44dbe00640 1 --2- 172.21.15.90:0/900189990 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f44e4108ff0 0x7f44e4110030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:46:19.133 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.133+0000 7f44e8e00640 1 --2- 172.21.15.90:0/900189990 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f44e4103f20 0x7f44e411a210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:46:19.133 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.134+0000 7f44e9800640 1 --2- 172.21.15.90:0/900189990 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f44e4110570 0x7f44e4110a20 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f44dc00b970 tx=0x7f44dc00be30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:46:19.133 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.134+0000 7f44e2000640 1 -- 172.21.15.90:0/900189990 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44dc019070 con 0x7f44e4110570 2024-04-02T14:46:19.133 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.134+0000 7f44e2000640 1 -- 172.21.15.90:0/900189990 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f44dc00cb40 con 0x7f44e4110570 2024-04-02T14:46:19.133 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.134+0000 7f44e2000640 1 -- 172.21.15.90:0/900189990 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44dc007650 con 0x7f44e4110570 2024-04-02T14:46:19.133 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.134+0000 7f44eb800640 1 -- 172.21.15.90:0/900189990 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f44e4116030 con 0x7f44e4110570 2024-04-02T14:46:19.134 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.135+0000 7f44eb800640 1 -- 172.21.15.90:0/900189990 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44e4116530 con 0x7f44e4110570 2024-04-02T14:46:19.136 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.140+0000 7f44e2000640 1 -- 172.21.15.90:0/900189990 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f44dc01c070 con 0x7f44e4110570 2024-04-02T14:46:19.138 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.143+0000 7f44e2000640 1 --2- 172.21.15.90:0/900189990 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f44c80a3ec0 0x7f44c80a6380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:19.139 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.143+0000 7f44e2000640 1 -- 172.21.15.90:0/900189990 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 8550+0+0 (secure 0 0 0) 0x7f44dc0cfda0 con 0x7f44e4110570 2024-04-02T14:46:19.139 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.143+0000 7f44e8e00640 1 --2- 172.21.15.90:0/900189990 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f44c80a3ec0 0x7f44c80a6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:19.140 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.143+0000 7f44ee45f740 1 Processor -- start 2024-04-02T14:46:19.140 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.144+0000 7f44ee45f740 1 -- start start 2024-04-02T14:46:19.140 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.144+0000 7f44e8e00640 1 --2- 172.21.15.90:0/900189990 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f44c80a3ec0 0x7f44c80a6380 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f44e4112510 tx=0x7f44d4005a20 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:46:19.140 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.144+0000 7f44ee45f740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557f79f02eb0 0x557f79f03290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:19.140 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.144+0000 7f44ee45f740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557f79f037d0 0x557f79f0be70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:19.140 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.144+0000 7f44ee45f740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557f79f0c3b0 0x557f79f0e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:19.140 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.144+0000 7f44ee45f740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x557f79e67d70 con 0x557f79f0c3b0 2024-04-02T14:46:19.141 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.144+0000 7f44ee45f740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x557f79e102a0 con 0x557f79f02eb0 2024-04-02T14:46:19.141 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.144+0000 7f44ee45f740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x557f79e63d20 con 0x557f79f037d0 2024-04-02T14:46:19.141 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.144+0000 7f44e8e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557f79f02eb0 0x557f79f03290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:19.141 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.144+0000 7f44e8e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557f79f02eb0 0x557f79f03290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:35924/0 (socket says 172.21.15.90:35924) 2024-04-02T14:46:19.141 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.144+0000 7f44e8e00640 1 -- 172.21.15.90:0/998133598 learned_addr learned my addr 172.21.15.90:0/998133598 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:46:19.142 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.145+0000 7f44dbe00640 1 --2- 172.21.15.90:0/998133598 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557f79f037d0 0x557f79f0be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:19.142 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.145+0000 7f44e9800640 1 --2- 172.21.15.90:0/998133598 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557f79f0c3b0 0x557f79f0e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:19.142 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.145+0000 7f44e8e00640 1 -- 172.21.15.90:0/998133598 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557f79f037d0 msgr2=0x557f79f0be70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:46:19.142 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.145+0000 7f44e8e00640 1 --2- 172.21.15.90:0/998133598 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557f79f037d0 0x557f79f0be70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:19.142 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.145+0000 7f44e8e00640 1 -- 172.21.15.90:0/998133598 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557f79f0c3b0 msgr2=0x557f79f0e7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:46:19.142 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.145+0000 7f44e8e00640 1 --2- 172.21.15.90:0/998133598 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557f79f0c3b0 0x557f79f0e7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:19.142 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.145+0000 7f44e8e00640 1 -- 172.21.15.90:0/998133598 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557f79e24840 con 0x557f79f02eb0 2024-04-02T14:46:19.142 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.145+0000 7f44dbe00640 1 --2- 172.21.15.90:0/998133598 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557f79f037d0 0x557f79f0be70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:46:19.143 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.145+0000 7f44e9800640 1 --2- 172.21.15.90:0/998133598 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557f79f0c3b0 0x557f79f0e7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:46:19.143 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.146+0000 7f44e8e00640 1 --2- 172.21.15.90:0/998133598 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557f79f02eb0 0x557f79f03290 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f44d4005ec0 tx=0x7f44d403d000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=102181cccf1bbe75 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:46:19.143 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.147+0000 7f44da000640 1 -- 172.21.15.90:0/998133598 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44d4005040 con 0x557f79f02eb0 2024-04-02T14:46:19.143 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.147+0000 7f44da000640 1 -- 172.21.15.90:0/998133598 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f44d404e070 con 0x557f79f02eb0 2024-04-02T14:46:19.143 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.147+0000 7f44da000640 1 -- 172.21.15.90:0/998133598 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44d4003850 con 0x557f79f02eb0 2024-04-02T14:46:19.144 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.148+0000 7f44ee45f740 1 -- 172.21.15.90:0/998133598 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557f79f02eb0 msgr2=0x557f79f03290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:19.144 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.148+0000 7f44ee45f740 1 --2- 172.21.15.90:0/998133598 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557f79f02eb0 0x557f79f03290 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f44d4005ec0 tx=0x7f44d403d000 comp rx=0 tx=0).stop 2024-04-02T14:46:19.144 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.148+0000 7f44ee45f740 1 -- 172.21.15.90:0/998133598 shutdown_connections 2024-04-02T14:46:19.145 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.148+0000 7f44ee45f740 1 --2- 172.21.15.90:0/998133598 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557f79f0c3b0 0x557f79f0e7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:19.145 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.148+0000 7f44ee45f740 1 --2- 172.21.15.90:0/998133598 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557f79f037d0 0x557f79f0be70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:19.145 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.148+0000 7f44ee45f740 1 --2- 172.21.15.90:0/998133598 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557f79f02eb0 0x557f79f03290 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:19.145 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.148+0000 7f44ee45f740 1 -- 172.21.15.90:0/998133598 >> 172.21.15.90:0/998133598 conn(0x557f79efd5e0 msgr2=0x557f79efd9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:46:19.146 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.149+0000 7f44ee45f740 1 -- 172.21.15.90:0/998133598 shutdown_connections 2024-04-02T14:46:19.146 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.149+0000 7f44ee45f740 1 -- 172.21.15.90:0/998133598 wait complete. 2024-04-02T14:46:19.146 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.149+0000 7f44ee45f740 1 -- start start 2024-04-02T14:46:19.146 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.150+0000 7f44ee45f740 1 Processor -- start 2024-04-02T14:46:19.148 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.151+0000 7f44ee45f740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557f79f02eb0 0x557f79f094b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:19.148 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.152+0000 7f44ee45f740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557f79f037d0 0x557f79f099f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:19.148 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.152+0000 7f44ee45f740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557f79f0c3b0 0x557f79f09f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:19.148 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.152+0000 7f44e8e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557f79f02eb0 0x557f79f094b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:19.148 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.152+0000 7f44e8e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557f79f02eb0 0x557f79f094b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:53866/0 (socket says 172.21.15.90:53866) 2024-04-02T14:46:19.148 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.152+0000 7f44dbe00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557f79f037d0 0x557f79f099f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:19.148 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.152+0000 7f44dbe00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557f79f037d0 0x557f79f099f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:35928/0 (socket says 172.21.15.90:35928) 2024-04-02T14:46:19.149 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.152+0000 7f44e8e00640 1 -- 172.21.15.90:0/2335621660 learned_addr learned my addr 172.21.15.90:0/2335621660 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:46:19.149 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.152+0000 7f44ee45f740 1 -- 172.21.15.90:0/2335621660 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x557f79f0f140 con 0x557f79f0c3b0 2024-04-02T14:46:19.149 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.152+0000 7f44ee45f740 1 -- 172.21.15.90:0/2335621660 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x557f79e23660 con 0x557f79f037d0 2024-04-02T14:46:19.149 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.152+0000 7f44ee45f740 1 -- 172.21.15.90:0/2335621660 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x557f79f0f2c0 con 0x557f79f02eb0 2024-04-02T14:46:19.149 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.152+0000 7f44e9800640 1 --2- 172.21.15.90:0/2335621660 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557f79f0c3b0 0x557f79f09f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:19.150 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.153+0000 7f44ee45f740 10 client.0 _ll_register_callbacks cb 0x557f79e23050 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:46:19.150 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.153+0000 7f44dbe00640 1 -- 172.21.15.90:0/2335621660 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557f79f02eb0 msgr2=0x557f79f094b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:46:19.150 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.154+0000 7f44dbe00640 1 --2- 172.21.15.90:0/2335621660 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557f79f02eb0 0x557f79f094b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:19.150 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.154+0000 7f44dbe00640 1 -- 172.21.15.90:0/2335621660 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557f79f0c3b0 msgr2=0x557f79f09f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:46:19.150 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.154+0000 7f44dbe00640 1 --2- 172.21.15.90:0/2335621660 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557f79f0c3b0 0x557f79f09f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:19.151 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.154+0000 7f44dbe00640 1 -- 172.21.15.90:0/2335621660 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557f79e32e80 con 0x557f79f037d0 2024-04-02T14:46:19.151 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.154+0000 7f44ee45f740 1 -- 172.21.15.90:0/2335621660 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f44d404e070 con 0x557f79f037d0 2024-04-02T14:46:19.151 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.154+0000 7f44e9800640 1 --2- 172.21.15.90:0/2335621660 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557f79f0c3b0 0x557f79f09f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:46:19.151 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.154+0000 7f44e8e00640 1 --2- 172.21.15.90:0/2335621660 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557f79f02eb0 0x557f79f094b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:46:19.151 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.154+0000 7f44dbe00640 1 --2- 172.21.15.90:0/2335621660 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557f79f037d0 0x557f79f099f0 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f44e4101cc0 tx=0x7f44cc013bf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4adb74b0f027a18c server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:46:19.152 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.155+0000 7f44d2a00640 10 client.29039 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:46:19.152 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.155+0000 7f44d2a00640 1 -- 172.21.15.90:0/2335621660 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44cc011da0 con 0x557f79f037d0 2024-04-02T14:46:19.152 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.155+0000 7f44d2a00640 1 -- 172.21.15.90:0/2335621660 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 8550+0+0 (secure 0 0 0) 0x7f44cc011ba0 con 0x557f79f037d0 2024-04-02T14:46:19.152 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.155+0000 7f44d2a00640 1 -- 172.21.15.90:0/2335621660 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f44cc011da0 con 0x557f79f037d0 2024-04-02T14:46:19.153 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.155+0000 7f44ee45f740 10 client.29039 fetch_fsmap learned FSMap version 27 2024-04-02T14:46:19.153 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.155+0000 7f44ee45f740 1 -- 172.21.15.90:0/2335621660 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({fsmap=27}) -- 0x557f79b7c210 con 0x557f79f037d0 2024-04-02T14:46:19.153 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.156+0000 7f44d2a00640 1 -- 172.21.15.90:0/2335621660 <== mon.2 v2:172.21.15.92:3300/0 4 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f44cc01a030 con 0x557f79f037d0 2024-04-02T14:46:19.153 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.156+0000 7f44ee45f740 10 client.29039 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:46:19.153 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.156+0000 7f44ee45f740 10 client.29039 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-02T14:46:19.153 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.156+0000 7f44ee45f740 20 client.29039 populate_metadata read hostname 'smithi090' 2024-04-02T14:46:19.153 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.156+0000 7f44ee45f740 1 --2- 172.21.15.90:0/2335621660 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x557f79f1c690 0x557f79f1eac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:19.153 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.156+0000 7f44ee45f740 4 client.29039 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:46:19.153 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.156+0000 7f44ee45f740 1 -- 172.21.15.90:0/2335621660 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557f79e17cd0 con 0x557f79f1c690 2024-04-02T14:46:19.153 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.156+0000 7f44e8e00640 1 --2- 172.21.15.90:0/2335621660 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x557f79f1c690 0x557f79f1eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:19.154 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.158+0000 7f44d2a00640 10 client.29039 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:46:19.154 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.158+0000 7f44e8e00640 1 --2- 172.21.15.90:0/2335621660 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x557f79f1c690 0x557f79f1eac0 crc :-1 s=READY pgs=300 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ab0f990fedd82b48 server_cookie=f9022cd817974ffe in_seq=0 out_seq=0 2024-04-02T14:46:19.154 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.158+0000 7f44d2a00640 1 -- 172.21.15.90:0/2335621660 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x557f79e17cd0 con 0x557f79f1c690 2024-04-02T14:46:19.155 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.158+0000 7f44d2a00640 10 client.29039 handle_command_reply: tid=0 2024-04-02T14:46:19.155 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.159+0000 7f44ee45f740 1 client.29039 shutdown 2024-04-02T14:46:19.155 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.159+0000 7f44ee45f740 1 -- 172.21.15.90:0/2335621660 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557f79f037d0 msgr2=0x557f79f099f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:19.155 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.159+0000 7f44ee45f740 1 --2- 172.21.15.90:0/2335621660 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557f79f037d0 0x557f79f099f0 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f44e4101cc0 tx=0x7f44cc013bf0 comp rx=0 tx=0).stop 2024-04-02T14:46:19.156 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.159+0000 7f44ee45f740 1 -- 172.21.15.90:0/2335621660 shutdown_connections 2024-04-02T14:46:19.156 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.159+0000 7f44ee45f740 1 --2- 172.21.15.90:0/2335621660 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x557f79f1c690 0x557f79f1eac0 crc :-1 s=READY pgs=300 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:19.156 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.160+0000 7f44d2a00640 0 client.29039 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:46:19.156 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.160+0000 7f44ee45f740 1 --2- 172.21.15.90:0/2335621660 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557f79f0c3b0 0x557f79f09f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:19.156 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.160+0000 7f44ee45f740 1 --2- 172.21.15.90:0/2335621660 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557f79f037d0 0x557f79f099f0 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:19.156 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.160+0000 7f44ee45f740 1 --2- 172.21.15.90:0/2335621660 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557f79f02eb0 0x557f79f094b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:19.156 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.160+0000 7f44ee45f740 1 -- 172.21.15.90:0/2335621660 >> 172.21.15.90:0/2335621660 conn(0x557f79efd5e0 msgr2=0x557f79eff330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:46:19.156 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.160+0000 7f44ee45f740 1 -- 172.21.15.90:0/2335621660 shutdown_connections 2024-04-02T14:46:19.156 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.160+0000 7f44ee45f740 1 -- 172.21.15.90:0/2335621660 wait complete. 2024-04-02T14:46:19.157 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.160+0000 7f44ee45f740 20 client.29039 trim_cache size 0 max 16384 2024-04-02T14:46:19.186 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.190+0000 7f44ee45f740 1 Processor -- start 2024-04-02T14:46:19.187 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.191+0000 7f44ee45f740 1 -- start start 2024-04-02T14:46:19.187 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.191+0000 7f44ee45f740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557f79f02eb0 0x557f79f0bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:19.187 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.191+0000 7f44ee45f740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557f79f037d0 0x557f79f09d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:19.187 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.191+0000 7f44ee45f740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557f79f0c3b0 0x557f79f243c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:19.188 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.192+0000 7f44e8e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557f79f02eb0 0x557f79f0bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:19.188 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.192+0000 7f44ee45f740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f44e41bc6e0 con 0x557f79f0c3b0 2024-04-02T14:46:19.188 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.192+0000 7f44ee45f740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f44e41be220 con 0x557f79f02eb0 2024-04-02T14:46:19.188 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.192+0000 7f44e9800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557f79f0c3b0 0x557f79f243c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:19.188 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.192+0000 7f44e9800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557f79f0c3b0 0x557f79f243c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41748/0 (socket says 172.21.15.90:41748) 2024-04-02T14:46:19.189 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.192+0000 7f44e9800640 1 -- 172.21.15.90:0/3475358275 learned_addr learned my addr 172.21.15.90:0/3475358275 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:46:19.189 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.193+0000 7f44ee45f740 1 -- 172.21.15.90:0/3475358275 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f44e41bcdb0 con 0x557f79f037d0 2024-04-02T14:46:19.189 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.193+0000 7f44dbe00640 1 --2- 172.21.15.90:0/3475358275 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557f79f037d0 0x557f79f09d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:19.190 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.194+0000 7f44e9800640 1 -- 172.21.15.90:0/3475358275 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557f79f037d0 msgr2=0x557f79f09d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:46:19.190 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.194+0000 7f44e9800640 1 --2- 172.21.15.90:0/3475358275 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557f79f037d0 0x557f79f09d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:19.190 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.194+0000 7f44e9800640 1 -- 172.21.15.90:0/3475358275 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557f79f02eb0 msgr2=0x557f79f0bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:46:19.190 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.194+0000 7f44e9800640 1 --2- 172.21.15.90:0/3475358275 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557f79f02eb0 0x557f79f0bbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:19.190 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.194+0000 7f44e9800640 1 -- 172.21.15.90:0/3475358275 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557f79f03600 con 0x557f79f0c3b0 2024-04-02T14:46:19.190 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.194+0000 7f44e8e00640 1 --2- 172.21.15.90:0/3475358275 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557f79f02eb0 0x557f79f0bbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:46:19.191 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.194+0000 7f44dbe00640 1 --2- 172.21.15.90:0/3475358275 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557f79f037d0 0x557f79f09d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:46:19.191 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.195+0000 7f44e9800640 1 --2- 172.21.15.90:0/3475358275 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557f79f0c3b0 0x557f79f243c0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f44dc09bad0 tx=0x7f44dc0bc040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=edc17d193cf5ba3d server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:46:19.191 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.196+0000 7f44d9600640 1 -- 172.21.15.90:0/3475358275 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44dc0a4a40 con 0x557f79f0c3b0 2024-04-02T14:46:19.192 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.196+0000 7f44d9600640 1 -- 172.21.15.90:0/3475358275 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f44dc0b5070 con 0x557f79f0c3b0 2024-04-02T14:46:19.193 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.196+0000 7f44d9600640 1 -- 172.21.15.90:0/3475358275 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44dc0a45b0 con 0x557f79f0c3b0 2024-04-02T14:46:19.193 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.197+0000 7f44ee45f740 1 -- 172.21.15.90:0/3475358275 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557f79f0c3b0 msgr2=0x557f79f243c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:19.193 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.197+0000 7f44ee45f740 1 --2- 172.21.15.90:0/3475358275 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557f79f0c3b0 0x557f79f243c0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f44dc09bad0 tx=0x7f44dc0bc040 comp rx=0 tx=0).stop 2024-04-02T14:46:19.193 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.198+0000 7f44ee45f740 1 -- 172.21.15.90:0/3475358275 shutdown_connections 2024-04-02T14:46:19.194 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.198+0000 7f44ee45f740 1 --2- 172.21.15.90:0/3475358275 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557f79f0c3b0 0x557f79f243c0 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:19.194 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.198+0000 7f44ee45f740 1 --2- 172.21.15.90:0/3475358275 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557f79f037d0 0x557f79f09d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:19.194 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.198+0000 7f44ee45f740 1 --2- 172.21.15.90:0/3475358275 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557f79f02eb0 0x557f79f0bbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:19.194 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.198+0000 7f44ee45f740 1 -- 172.21.15.90:0/3475358275 >> 172.21.15.90:0/3475358275 conn(0x557f79efd5e0 msgr2=0x557f79efe810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:46:19.194 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.198+0000 7f44ee45f740 1 -- 172.21.15.90:0/3475358275 shutdown_connections 2024-04-02T14:46:19.195 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.199+0000 7f44ee45f740 1 -- 172.21.15.90:0/3475358275 wait complete. 2024-04-02T14:46:19.195 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.199+0000 7f44ee45f740 1 -- start start 2024-04-02T14:46:19.196 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.199+0000 7f44ee45f740 1 Processor -- start 2024-04-02T14:46:19.196 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.201+0000 7f44ee45f740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557f79f02eb0 0x557f79f10620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:19.197 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.201+0000 7f44ee45f740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557f79f037d0 0x557f79f10b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:19.197 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.201+0000 7f44ee45f740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557f79f0c3b0 0x557f79f81c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:19.197 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.201+0000 7f44ee45f740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f44e41bdb50 con 0x557f79f02eb0 2024-04-02T14:46:19.197 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.201+0000 7f44ee45f740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f44e41bd480 con 0x557f79f0c3b0 2024-04-02T14:46:19.197 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.201+0000 7f44ee45f740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f44e41ba6e0 con 0x557f79f037d0 2024-04-02T14:46:19.197 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.201+0000 7f44ee45f740 10 client.0 _ll_register_callbacks cb 0x7f44cc009d40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:46:19.198 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.201+0000 7f44dbe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557f79f037d0 0x557f79f10b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:19.198 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.201+0000 7f44dbe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557f79f037d0 0x557f79f10b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:53894/0 (socket says 172.21.15.90:53894) 2024-04-02T14:46:19.198 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.201+0000 7f44dbe00640 1 -- 172.21.15.90:0/3135114021 learned_addr learned my addr 172.21.15.90:0/3135114021 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:46:19.199 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.202+0000 7f44e9800640 1 --2- 172.21.15.90:0/3135114021 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557f79f0c3b0 0x557f79f81c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:19.199 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.202+0000 7f44e8e00640 1 --2- 172.21.15.90:0/3135114021 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557f79f02eb0 0x557f79f10620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:19.199 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.202+0000 7f44e9800640 1 -- 172.21.15.90:0/3135114021 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557f79f037d0 msgr2=0x557f79f10b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:46:19.199 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.202+0000 7f44e9800640 1 --2- 172.21.15.90:0/3135114021 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557f79f037d0 0x557f79f10b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:19.199 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.202+0000 7f44e9800640 1 -- 172.21.15.90:0/3135114021 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557f79f02eb0 msgr2=0x557f79f10620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:46:19.199 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.202+0000 7f44e9800640 1 --2- 172.21.15.90:0/3135114021 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557f79f02eb0 0x557f79f10620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:19.199 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.202+0000 7f44e9800640 1 -- 172.21.15.90:0/3135114021 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557f79c48620 con 0x557f79f0c3b0 2024-04-02T14:46:19.199 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.202+0000 7f44e8e00640 1 --2- 172.21.15.90:0/3135114021 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557f79f02eb0 0x557f79f10620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:46:19.199 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.202+0000 7f44e9800640 1 --2- 172.21.15.90:0/3135114021 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557f79f0c3b0 0x557f79f81c30 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f44dc09bad0 tx=0x7f44dc0a0300 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7c7bc7a252f1a468 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:46:19.199 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.202+0000 7f44d2a00640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:46:19.199 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.203+0000 7f44d2a00640 1 -- 172.21.15.90:0/3135114021 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44dc0a4a40 con 0x557f79f0c3b0 2024-04-02T14:46:19.199 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.203+0000 7f44d2a00640 1 -- 172.21.15.90:0/3135114021 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 8550+0+0 (secure 0 0 0) 0x7f44dc0b5220 con 0x557f79f0c3b0 2024-04-02T14:46:19.200 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.203+0000 7f44ee45f740 1 -- 172.21.15.90:0/3135114021 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f44dc0b5070 con 0x557f79f0c3b0 2024-04-02T14:46:19.200 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.204+0000 7f44d2a00640 1 -- 172.21.15.90:0/3135114021 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f44dc0850a0 con 0x557f79f0c3b0 2024-04-02T14:46:19.200 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.204+0000 7f44ee45f740 10 client.29045 fetch_fsmap learned FSMap version 27 2024-04-02T14:46:19.200 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.204+0000 7f44ee45f740 1 -- 172.21.15.90:0/3135114021 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x557f79f83260 con 0x557f79f0c3b0 2024-04-02T14:46:19.200 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.205+0000 7f44d2a00640 1 -- 172.21.15.90:0/3135114021 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f44dc0b5070 con 0x557f79f0c3b0 2024-04-02T14:46:19.200 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.205+0000 7f44d2a00640 1 -- 172.21.15.90:0/3135114021 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f44dc082030 con 0x557f79f0c3b0 2024-04-02T14:46:19.201 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.205+0000 7f44ee45f740 10 client.29045 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:46:19.201 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.205+0000 7f44ee45f740 10 client.29045 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-02T14:46:19.201 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.205+0000 7f44ee45f740 20 client.29045 populate_metadata read hostname 'smithi090' 2024-04-02T14:46:19.201 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.205+0000 7f44ee45f740 1 --2- 172.21.15.90:0/3135114021 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x557f79f1c690 0x557f79f85890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:19.201 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.205+0000 7f44ee45f740 4 client.29045 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-02T14:46:19.201 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.205+0000 7f44ee45f740 1 -- 172.21.15.90:0/3135114021 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "flush journal"}) -- 0x557f79e752b0 con 0x557f79f1c690 2024-04-02T14:46:19.201 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.205+0000 7f44e8e00640 1 --2- 172.21.15.90:0/3135114021 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x557f79f1c690 0x557f79f85890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:19.202 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.206+0000 7f44e8e00640 1 --2- 172.21.15.90:0/3135114021 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x557f79f1c690 0x557f79f85890 crc :-1 s=READY pgs=301 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a99f5c0a942cb5b7 server_cookie=a77a4cc0fea3abc5 in_seq=0 out_seq=0 2024-04-02T14:46:19.202 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.206+0000 7f44d2a00640 10 client.29045 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:46:19.220 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.224+0000 7f44d2a00640 1 -- 172.21.15.90:0/3135114021 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x557f79e752b0 con 0x557f79f1c690 2024-04-02T14:46:19.220 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.224+0000 7f44d2a00640 10 client.29045 handle_command_reply: tid=0 2024-04-02T14:46:19.220 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.224+0000 7f44ee45f740 1 client.29045 shutdown 2024-04-02T14:46:19.221 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.225+0000 7f44ee45f740 1 -- 172.21.15.90:0/3135114021 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557f79f0c3b0 msgr2=0x557f79f81c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:19.221 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.225+0000 7f44ee45f740 1 --2- 172.21.15.90:0/3135114021 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557f79f0c3b0 0x557f79f81c30 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f44dc09bad0 tx=0x7f44dc0a0300 comp rx=0 tx=0).stop 2024-04-02T14:46:19.222 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.226+0000 7f44ee45f740 1 -- 172.21.15.90:0/3135114021 shutdown_connections 2024-04-02T14:46:19.222 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.226+0000 7f44ee45f740 1 --2- 172.21.15.90:0/3135114021 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x557f79f1c690 0x557f79f85890 crc :-1 s=READY pgs=301 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:19.222 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.226+0000 7f44ee45f740 1 --2- 172.21.15.90:0/3135114021 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557f79f0c3b0 0x557f79f81c30 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:19.222 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.226+0000 7f44ee45f740 1 --2- 172.21.15.90:0/3135114021 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557f79f037d0 0x557f79f10b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:19.222 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.226+0000 7f44ee45f740 1 --2- 172.21.15.90:0/3135114021 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557f79f02eb0 0x557f79f10620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:19.223 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.226+0000 7f44ee45f740 1 -- 172.21.15.90:0/3135114021 >> 172.21.15.90:0/3135114021 conn(0x557f79efd5e0 msgr2=0x557f79efef80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:46:19.223 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.226+0000 7f44d2a00640 0 client.29045 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:46:19.223 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.226+0000 7f44ee45f740 1 -- 172.21.15.90:0/3135114021 shutdown_connections 2024-04-02T14:46:19.223 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.226+0000 7f44ee45f740 1 -- 172.21.15.90:0/3135114021 wait complete. 2024-04-02T14:46:19.223 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.226+0000 7f44ee45f740 20 client.29045 trim_cache size 0 max 16384 2024-04-02T14:46:19.223 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:{ 2024-04-02T14:46:19.223 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "message": "", 2024-04-02T14:46:19.223 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "return_code": 0 2024-04-02T14:46:19.223 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:} 2024-04-02T14:46:19.224 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.227+0000 7f44eb800640 1 -- 172.21.15.90:0/900189990 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f44c80a3ec0 msgr2=0x7f44c80a6380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:19.224 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.227+0000 7f44eb800640 1 --2- 172.21.15.90:0/900189990 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f44c80a3ec0 0x7f44c80a6380 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f44e4112510 tx=0x7f44d4005a20 comp rx=0 tx=0).stop 2024-04-02T14:46:19.224 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.227+0000 7f44eb800640 1 -- 172.21.15.90:0/900189990 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f44e4110570 msgr2=0x7f44e4110a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:19.224 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.227+0000 7f44eb800640 1 --2- 172.21.15.90:0/900189990 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f44e4110570 0x7f44e4110a20 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f44dc00b970 tx=0x7f44dc00be30 comp rx=0 tx=0).stop 2024-04-02T14:46:19.224 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.228+0000 7f44eb800640 1 -- 172.21.15.90:0/900189990 shutdown_connections 2024-04-02T14:46:19.224 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.228+0000 7f44eb800640 1 --2- 172.21.15.90:0/900189990 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f44c80a3ec0 0x7f44c80a6380 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:19.224 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.228+0000 7f44eb800640 1 --2- 172.21.15.90:0/900189990 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f44e4110570 0x7f44e4110a20 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:19.224 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.228+0000 7f44eb800640 1 --2- 172.21.15.90:0/900189990 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f44e4108ff0 0x7f44e4110030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:19.225 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.228+0000 7f44eb800640 1 --2- 172.21.15.90:0/900189990 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f44e4103f20 0x7f44e411a210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:19.225 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.228+0000 7f44eb800640 1 -- 172.21.15.90:0/900189990 >> 172.21.15.90:0/900189990 conn(0x7f44e40ff780 msgr2=0x7f44e4079300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:46:19.225 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.228+0000 7f44eb800640 1 -- 172.21.15.90:0/900189990 shutdown_connections 2024-04-02T14:46:19.225 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:19.228+0000 7f44eb800640 1 -- 172.21.15.90:0/900189990 wait complete. 2024-04-02T14:46:20.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:46:20 smithi177 ceph-mon[36431]: pgmap v356: 97 pgs: 97 active+clean; 4.2 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 9.5 MiB/s wr, 1.04k op/s 2024-04-02T14:46:20.432 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:46:20 smithi090 ceph-mon[31674]: pgmap v356: 97 pgs: 97 active+clean; 4.2 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 9.5 MiB/s wr, 1.04k op/s 2024-04-02T14:46:20.604 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:46:20 smithi092 ceph-mon[36497]: pgmap v356: 97 pgs: 97 active+clean; 4.2 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 9.5 MiB/s wr, 1.04k op/s 2024-04-02T14:46:21.825 INFO:teuthology.orchestra.run.smithi090.stderr:progress: 650.0 s, 2544.0 tps, lat 38.949 ms stddev 50.029 2024-04-02T14:46:22.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:46:22 smithi177 ceph-mon[36431]: pgmap v357: 97 pgs: 97 active+clean; 4.2 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 3.9 MiB/s wr, 450 op/s 2024-04-02T14:46:22.432 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:46:22 smithi090 ceph-mon[31674]: pgmap v357: 97 pgs: 97 active+clean; 4.2 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 3.9 MiB/s wr, 450 op/s 2024-04-02T14:46:22.604 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:46:22 smithi092 ceph-mon[36497]: pgmap v357: 97 pgs: 97 active+clean; 4.2 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 3.9 MiB/s wr, 450 op/s 2024-04-02T14:46:24.431 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:46:24 smithi177 ceph-mon[36431]: pgmap v358: 97 pgs: 97 active+clean; 4.2 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 3.9 MiB/s wr, 450 op/s 2024-04-02T14:46:24.431 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:46:24 smithi177 ceph-mon[36431]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T14:46:24.431 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:46:24 smithi177 ceph-mon[36431]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' 2024-04-02T14:46:24.431 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:46:24 smithi177 ceph-mon[36431]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' 2024-04-02T14:46:24.431 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:46:24 smithi177 ceph-mon[36431]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' 2024-04-02T14:46:24.431 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:46:24 smithi177 ceph-mon[36431]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' 2024-04-02T14:46:24.435 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:46:24 smithi090 ceph-mon[31674]: pgmap v358: 97 pgs: 97 active+clean; 4.2 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 3.9 MiB/s wr, 450 op/s 2024-04-02T14:46:24.435 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:46:24 smithi090 ceph-mon[31674]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T14:46:24.435 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:46:24 smithi090 ceph-mon[31674]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' 2024-04-02T14:46:24.435 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:46:24 smithi090 ceph-mon[31674]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' 2024-04-02T14:46:24.435 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:46:24 smithi090 ceph-mon[31674]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' 2024-04-02T14:46:24.435 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:46:24 smithi090 ceph-mon[31674]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' 2024-04-02T14:46:24.604 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:46:24 smithi092 ceph-mon[36497]: pgmap v358: 97 pgs: 97 active+clean; 4.2 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 3.9 MiB/s wr, 450 op/s 2024-04-02T14:46:24.604 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:46:24 smithi092 ceph-mon[36497]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T14:46:24.605 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:46:24 smithi092 ceph-mon[36497]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' 2024-04-02T14:46:24.605 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:46:24 smithi092 ceph-mon[36497]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' 2024-04-02T14:46:24.605 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:46:24 smithi092 ceph-mon[36497]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' 2024-04-02T14:46:24.605 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:46:24 smithi092 ceph-mon[36497]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' 2024-04-02T14:46:25.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:46:25 smithi177 ceph-mon[36431]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:46:25.431 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:46:25 smithi177 ceph-mon[36431]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T14:46:25.431 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:46:25 smithi177 ceph-mon[36431]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' 2024-04-02T14:46:25.432 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:46:25 smithi090 ceph-mon[31674]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:46:25.432 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:46:25 smithi090 ceph-mon[31674]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T14:46:25.432 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:46:25 smithi090 ceph-mon[31674]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' 2024-04-02T14:46:25.604 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:46:25 smithi092 ceph-mon[36497]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:46:25.604 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:46:25 smithi092 ceph-mon[36497]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T14:46:25.604 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:46:25 smithi092 ceph-mon[36497]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' 2024-04-02T14:46:26.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:46:26 smithi177 ceph-mon[36431]: pgmap v359: 97 pgs: 97 active+clean; 4.2 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 6.6 MiB/s wr, 673 op/s 2024-04-02T14:46:26.432 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:46:26 smithi090 ceph-mon[31674]: pgmap v359: 97 pgs: 97 active+clean; 4.2 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 6.6 MiB/s wr, 673 op/s 2024-04-02T14:46:26.604 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:46:26 smithi092 ceph-mon[36497]: pgmap v359: 97 pgs: 97 active+clean; 4.2 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 6.6 MiB/s wr, 673 op/s 2024-04-02T14:46:26.825 INFO:teuthology.orchestra.run.smithi090.stderr:progress: 655.0 s, 1699.2 tps, lat 59.492 ms stddev 74.244 2024-04-02T14:46:28.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:46:28 smithi177 ceph-mon[36431]: pgmap v360: 97 pgs: 97 active+clean; 4.2 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 5.7 MiB/s wr, 587 op/s 2024-04-02T14:46:28.432 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:46:28 smithi090 ceph-mon[31674]: pgmap v360: 97 pgs: 97 active+clean; 4.2 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 5.7 MiB/s wr, 587 op/s 2024-04-02T14:46:28.604 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:46:28 smithi092 ceph-mon[36497]: pgmap v360: 97 pgs: 97 active+clean; 4.2 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 5.7 MiB/s wr, 587 op/s 2024-04-02T14:46:30.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:46:30 smithi177 ceph-mon[36431]: pgmap v361: 97 pgs: 97 active+clean; 4.2 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 5.7 MiB/s wr, 592 op/s 2024-04-02T14:46:30.432 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:46:30 smithi090 ceph-mon[31674]: pgmap v361: 97 pgs: 97 active+clean; 4.2 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 5.7 MiB/s wr, 592 op/s 2024-04-02T14:46:30.604 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:46:30 smithi092 ceph-mon[36497]: pgmap v361: 97 pgs: 97 active+clean; 4.2 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 5.7 MiB/s wr, 592 op/s 2024-04-02T14:46:31.825 INFO:teuthology.orchestra.run.smithi090.stderr:progress: 660.0 s, 1735.4 tps, lat 57.677 ms stddev 66.447 2024-04-02T14:46:32.402 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.405+0000 7f21cea00640 1 -- 172.21.15.90:0/2601755380 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f21c8109010 msgr2=0x7f21c8110be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:32.402 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.405+0000 7f21cea00640 1 --2- 172.21.15.90:0/2601755380 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f21c8109010 0x7f21c8110be0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f21b4009900 tx=0x7f21b402bc30 comp rx=0 tx=0).stop 2024-04-02T14:46:32.402 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.406+0000 7f21cea00640 1 -- 172.21.15.90:0/2601755380 shutdown_connections 2024-04-02T14:46:32.402 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.406+0000 7f21cea00640 1 --2- 172.21.15.90:0/2601755380 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f21c8109010 0x7f21c8110be0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:32.402 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.406+0000 7f21cea00640 1 --2- 172.21.15.90:0/2601755380 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f21c81048f0 0x7f21c8108930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:32.402 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.406+0000 7f21cea00640 1 --2- 172.21.15.90:0/2601755380 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f21c8103f40 0x7f21c8104320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:32.402 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.406+0000 7f21cea00640 1 -- 172.21.15.90:0/2601755380 >> 172.21.15.90:0/2601755380 conn(0x7f21c80ff7c0 msgr2=0x7f21c8101be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:46:32.402 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.406+0000 7f21cea00640 1 -- 172.21.15.90:0/2601755380 shutdown_connections 2024-04-02T14:46:32.403 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.406+0000 7f21cea00640 1 -- 172.21.15.90:0/2601755380 wait complete. 2024-04-02T14:46:32.403 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.407+0000 7f21cea00640 1 Processor -- start 2024-04-02T14:46:32.403 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.407+0000 7f21cea00640 1 -- start start 2024-04-02T14:46:32.404 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.407+0000 7f21cea00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f21c8103f40 0x7f21c819a500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:32.404 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.407+0000 7f21cea00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f21c8109010 0x7f21c819aa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:32.404 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.407+0000 7f21cea00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f21c819edc0 0x7f21c819f270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:32.404 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.407+0000 7f21cea00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f21c8114120 con 0x7f21c8109010 2024-04-02T14:46:32.404 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.407+0000 7f21cea00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f21c8113fa0 con 0x7f21c8103f40 2024-04-02T14:46:32.404 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.407+0000 7f21cea00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f21c81142a0 con 0x7f21c819edc0 2024-04-02T14:46:32.404 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.408+0000 7f21c7e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f21c8103f40 0x7f21c819a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:32.405 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.408+0000 7f21c7e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f21c8103f40 0x7f21c819a500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:43842/0 (socket says 172.21.15.90:43842) 2024-04-02T14:46:32.405 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.408+0000 7f21c7e00640 1 -- 172.21.15.90:0/188712178 learned_addr learned my addr 172.21.15.90:0/188712178 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:46:32.405 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.408+0000 7f21c7400640 1 --2- 172.21.15.90:0/188712178 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f21c8109010 0x7f21c819aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:32.405 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.408+0000 7f21c7e00640 1 -- 172.21.15.90:0/188712178 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f21c819edc0 msgr2=0x7f21c819f270 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-02T14:46:32.405 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.408+0000 7f21c7e00640 1 --2- 172.21.15.90:0/188712178 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f21c819edc0 0x7f21c819f270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:32.405 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.408+0000 7f21c7e00640 1 -- 172.21.15.90:0/188712178 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f21c8109010 msgr2=0x7f21c819aa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:32.405 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.408+0000 7f21c7e00640 1 --2- 172.21.15.90:0/188712178 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f21c8109010 0x7f21c819aa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:32.405 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.408+0000 7f21c7e00640 1 -- 172.21.15.90:0/188712178 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21c819f8b0 con 0x7f21c8103f40 2024-04-02T14:46:32.405 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.408+0000 7f21c7400640 1 --2- 172.21.15.90:0/188712178 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f21c8109010 0x7f21c819aa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:46:32.405 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.409+0000 7f21c7e00640 1 --2- 172.21.15.90:0/188712178 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f21c8103f40 0x7f21c819a500 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f21c000ebc0 tx=0x7f21c000c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:46:32.406 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.409+0000 7f21c4c00640 1 -- 172.21.15.90:0/188712178 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21c0010630 con 0x7f21c8103f40 2024-04-02T14:46:32.406 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.409+0000 7f21c4c00640 1 -- 172.21.15.90:0/188712178 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f21c0010cb0 con 0x7f21c8103f40 2024-04-02T14:46:32.406 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.409+0000 7f21c4c00640 1 -- 172.21.15.90:0/188712178 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21c0018c40 con 0x7f21c8103f40 2024-04-02T14:46:32.406 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.409+0000 7f21cea00640 1 -- 172.21.15.90:0/188712178 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f21c819fba0 con 0x7f21c8103f40 2024-04-02T14:46:32.406 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.409+0000 7f21cea00640 1 -- 172.21.15.90:0/188712178 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21c81a7340 con 0x7f21c8103f40 2024-04-02T14:46:32.409 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.413+0000 7f21d1767740 1 Processor -- start 2024-04-02T14:46:32.409 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.413+0000 7f21c4c00640 1 -- 172.21.15.90:0/188712178 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f21c00107d0 con 0x7f21c8103f40 2024-04-02T14:46:32.409 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.414+0000 7f21d1767740 1 -- start start 2024-04-02T14:46:32.411 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.414+0000 7f21d1767740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559abcfa7cc0 0x559abcfa80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:32.411 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.415+0000 7f21d1767740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559abcfa8670 0x559abcfb9680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:32.411 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.415+0000 7f21d1767740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559abcfb9bc0 0x559abcfbbfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:32.411 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.415+0000 7f21d1767740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x559abcf0cd70 con 0x559abcfa7cc0 2024-04-02T14:46:32.411 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.415+0000 7f21d1767740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x559abceb52a0 con 0x559abcfb9bc0 2024-04-02T14:46:32.411 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.415+0000 7f21d1767740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x559abcf08d20 con 0x559abcfa8670 2024-04-02T14:46:32.411 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.415+0000 7f21c7e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559abcfa8670 0x559abcfb9680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:32.411 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.415+0000 7f21c7e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559abcfa8670 0x559abcfb9680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:32842/0 (socket says 172.21.15.90:32842) 2024-04-02T14:46:32.411 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.415+0000 7f21c7e00640 1 -- 172.21.15.90:0/1547286953 learned_addr learned my addr 172.21.15.90:0/1547286953 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:46:32.411 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.415+0000 7f21c7400640 1 --2- 172.21.15.90:0/1547286953 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559abcfb9bc0 0x559abcfbbfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:32.411 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.415+0000 7f21cca00640 1 --2- 172.21.15.90:0/1547286953 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559abcfa7cc0 0x559abcfa80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:32.412 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.416+0000 7f21c7e00640 1 -- 172.21.15.90:0/1547286953 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559abcfb9bc0 msgr2=0x559abcfbbfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:46:32.412 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.416+0000 7f21c7e00640 1 --2- 172.21.15.90:0/1547286953 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559abcfb9bc0 0x559abcfbbfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:32.413 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.416+0000 7f21c7e00640 1 -- 172.21.15.90:0/1547286953 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559abcfa7cc0 msgr2=0x559abcfa80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:46:32.413 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.416+0000 7f21c7e00640 1 --2- 172.21.15.90:0/1547286953 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559abcfa7cc0 0x559abcfa80a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:32.413 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.416+0000 7f21c7e00640 1 -- 172.21.15.90:0/1547286953 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559abcec9840 con 0x559abcfa8670 2024-04-02T14:46:32.413 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.416+0000 7f21cca00640 1 --2- 172.21.15.90:0/1547286953 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559abcfa7cc0 0x559abcfa80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:46:32.413 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.416+0000 7f21c4c00640 1 --2- 172.21.15.90:0/188712178 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f21ac0a3e40 0x7f21ac0a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:32.413 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.416+0000 7f21c4c00640 1 -- 172.21.15.90:0/188712178 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 8550+0+0 (secure 0 0 0) 0x7f21c00d10b0 con 0x7f21c8103f40 2024-04-02T14:46:32.413 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.416+0000 7f21cca00640 1 --2- 172.21.15.90:0/188712178 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f21ac0a3e40 0x7f21ac0a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:32.413 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.416+0000 7f21c7e00640 1 --2- 172.21.15.90:0/1547286953 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559abcfa8670 0x559abcfb9680 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f21c00ce720 tx=0x7f21c00d9910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9438cc63e904a47d server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:46:32.413 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.416+0000 7f21bd200640 1 -- 172.21.15.90:0/1547286953 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21c0010e60 con 0x559abcfa8670 2024-04-02T14:46:32.413 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.416+0000 7f21bd200640 1 -- 172.21.15.90:0/1547286953 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f21c009d730 con 0x559abcfa8670 2024-04-02T14:46:32.413 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.416+0000 7f21bd200640 1 -- 172.21.15.90:0/1547286953 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21c009c990 con 0x559abcfa8670 2024-04-02T14:46:32.414 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.418+0000 7f21d1767740 1 -- 172.21.15.90:0/1547286953 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559abcfa8670 msgr2=0x559abcfb9680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:32.414 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.418+0000 7f21d1767740 1 --2- 172.21.15.90:0/1547286953 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559abcfa8670 0x559abcfb9680 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f21c00ce720 tx=0x7f21c00d9910 comp rx=0 tx=0).stop 2024-04-02T14:46:32.415 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.418+0000 7f21cca00640 1 --2- 172.21.15.90:0/188712178 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f21ac0a3e40 0x7f21ac0a6300 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f21b4005ab0 tx=0x7f21b402e040 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:46:32.415 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.419+0000 7f21d1767740 1 -- 172.21.15.90:0/1547286953 shutdown_connections 2024-04-02T14:46:32.415 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.419+0000 7f21d1767740 1 --2- 172.21.15.90:0/1547286953 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559abcfb9bc0 0x559abcfbbfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:32.415 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.419+0000 7f21d1767740 1 --2- 172.21.15.90:0/1547286953 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559abcfa8670 0x559abcfb9680 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:32.415 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.419+0000 7f21d1767740 1 --2- 172.21.15.90:0/1547286953 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559abcfa7cc0 0x559abcfa80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:32.415 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.419+0000 7f21d1767740 1 -- 172.21.15.90:0/1547286953 >> 172.21.15.90:0/1547286953 conn(0x559abcfa25e0 msgr2=0x559abcfa29c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:46:32.415 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.419+0000 7f21d1767740 1 -- 172.21.15.90:0/1547286953 shutdown_connections 2024-04-02T14:46:32.415 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.419+0000 7f21d1767740 1 -- 172.21.15.90:0/1547286953 wait complete. 2024-04-02T14:46:32.416 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.419+0000 7f21d1767740 1 -- start start 2024-04-02T14:46:32.416 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.420+0000 7f21d1767740 1 Processor -- start 2024-04-02T14:46:32.416 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.420+0000 7f21d1767740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559abcfa7cc0 0x559abcfb6c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:32.416 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.420+0000 7f21d1767740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559abcfa8670 0x559abcfb7180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:32.416 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.420+0000 7f21d1767740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559abcfb9bc0 0x559abcfb76c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:32.416 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.420+0000 7f21d1767740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x559abcfbc980 con 0x559abcfb9bc0 2024-04-02T14:46:32.416 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.420+0000 7f21d1767740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x559abcec8660 con 0x559abcfa7cc0 2024-04-02T14:46:32.416 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.420+0000 7f21d1767740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x559abcfbcb00 con 0x559abcfa8670 2024-04-02T14:46:32.417 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.420+0000 7f21d1767740 10 client.0 _ll_register_callbacks cb 0x559abcec8050 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:46:32.417 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.421+0000 7f21c7400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559abcfa8670 0x559abcfb7180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:32.417 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.421+0000 7f21c7400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559abcfa8670 0x559abcfb7180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:32854/0 (socket says 172.21.15.90:32854) 2024-04-02T14:46:32.417 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.421+0000 7f21c7400640 1 -- 172.21.15.90:0/2300517043 learned_addr learned my addr 172.21.15.90:0/2300517043 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:46:32.417 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.421+0000 7f21cca00640 1 --2- 172.21.15.90:0/2300517043 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559abcfb9bc0 0x559abcfb76c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:32.417 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.421+0000 7f21c7e00640 1 --2- 172.21.15.90:0/2300517043 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559abcfa7cc0 0x559abcfb6c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:32.418 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.421+0000 7f21c7400640 1 -- 172.21.15.90:0/2300517043 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559abcfa7cc0 msgr2=0x559abcfb6c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:46:32.418 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.421+0000 7f21c7400640 1 --2- 172.21.15.90:0/2300517043 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559abcfa7cc0 0x559abcfb6c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:32.418 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.421+0000 7f21c7400640 1 -- 172.21.15.90:0/2300517043 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559abcfb9bc0 msgr2=0x559abcfb76c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:46:32.418 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.421+0000 7f21c7400640 1 --2- 172.21.15.90:0/2300517043 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559abcfb9bc0 0x559abcfb76c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:32.418 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.421+0000 7f21c7400640 1 -- 172.21.15.90:0/2300517043 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559abced7e80 con 0x559abcfa8670 2024-04-02T14:46:32.418 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.422+0000 7f21c7e00640 1 --2- 172.21.15.90:0/2300517043 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559abcfa7cc0 0x559abcfb6c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:46:32.419 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.422+0000 7f21c7400640 1 --2- 172.21.15.90:0/2300517043 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559abcfa8670 0x559abcfb7180 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f21b8013510 tx=0x7f21b80139d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=322d7efc4922a142 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:46:32.419 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.422+0000 7f21b1600640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:46:32.419 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.423+0000 7f21b1600640 1 -- 172.21.15.90:0/2300517043 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21b8004020 con 0x559abcfa8670 2024-04-02T14:46:32.419 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.423+0000 7f21b1600640 1 -- 172.21.15.90:0/2300517043 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 8550+0+0 (secure 0 0 0) 0x7f21b80041c0 con 0x559abcfa8670 2024-04-02T14:46:32.419 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.423+0000 7f21d1767740 1 -- 172.21.15.90:0/2300517043 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f21c009d730 con 0x559abcfa8670 2024-04-02T14:46:32.419 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.424+0000 7f21b1600640 1 -- 172.21.15.90:0/2300517043 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f21b80043c0 con 0x559abcfa8670 2024-04-02T14:46:32.419 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.424+0000 7f21d1767740 10 client.28861 fetch_fsmap learned FSMap version 27 2024-04-02T14:46:32.419 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.424+0000 7f21d1767740 1 -- 172.21.15.90:0/2300517043 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x559abcc21210 con 0x559abcfa8670 2024-04-02T14:46:32.420 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.424+0000 7f21b1600640 1 -- 172.21.15.90:0/2300517043 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f21c009d730 con 0x559abcfa8670 2024-04-02T14:46:32.421 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.424+0000 7f21b1600640 1 -- 172.21.15.90:0/2300517043 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f21b800f040 con 0x559abcfa8670 2024-04-02T14:46:32.421 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.424+0000 7f21d1767740 10 client.28861 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:46:32.421 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.424+0000 7f21d1767740 10 client.28861 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-02T14:46:32.421 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.424+0000 7f21d1767740 20 client.28861 populate_metadata read hostname 'smithi090' 2024-04-02T14:46:32.421 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.424+0000 7f21d1767740 1 --2- 172.21.15.90:0/2300517043 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x559abcfca030 0x559abcfcc420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:32.421 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.424+0000 7f21d1767740 4 client.28861 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:46:32.421 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.424+0000 7f21d1767740 1 -- 172.21.15.90:0/2300517043 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559abcebccd0 con 0x559abcfca030 2024-04-02T14:46:32.421 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.425+0000 7f21c7e00640 1 --2- 172.21.15.90:0/2300517043 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x559abcfca030 0x559abcfcc420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:32.421 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.425+0000 7f21c7e00640 1 --2- 172.21.15.90:0/2300517043 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x559abcfca030 0x559abcfcc420 crc :-1 s=READY pgs=302 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6b769862c0442a72 server_cookie=e19e7b8cd10c55ad in_seq=0 out_seq=0 2024-04-02T14:46:32.421 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.425+0000 7f21b1600640 10 client.28861 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:46:32.422 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.426+0000 7f21b1600640 1 -- 172.21.15.90:0/2300517043 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x7f21c007be50 con 0x559abcfca030 2024-04-02T14:46:32.422 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.426+0000 7f21b1600640 10 client.28861 handle_command_reply: tid=0 2024-04-02T14:46:32.422 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.426+0000 7f21d1767740 1 client.28861 shutdown 2024-04-02T14:46:32.422 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.426+0000 7f21d1767740 1 -- 172.21.15.90:0/2300517043 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559abcfa8670 msgr2=0x559abcfb7180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:32.422 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.426+0000 7f21d1767740 1 --2- 172.21.15.90:0/2300517043 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559abcfa8670 0x559abcfb7180 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f21b8013510 tx=0x7f21b80139d0 comp rx=0 tx=0).stop 2024-04-02T14:46:32.422 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.426+0000 7f21d1767740 1 -- 172.21.15.90:0/2300517043 shutdown_connections 2024-04-02T14:46:32.423 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.426+0000 7f21d1767740 1 --2- 172.21.15.90:0/2300517043 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x559abcfca030 0x559abcfcc420 crc :-1 s=READY pgs=302 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:32.423 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.426+0000 7f21d1767740 1 --2- 172.21.15.90:0/2300517043 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559abcfb9bc0 0x559abcfb76c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:32.423 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.426+0000 7f21d1767740 1 --2- 172.21.15.90:0/2300517043 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559abcfa8670 0x559abcfb7180 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:32.423 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.426+0000 7f21d1767740 1 --2- 172.21.15.90:0/2300517043 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559abcfa7cc0 0x559abcfb6c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:32.423 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.426+0000 7f21d1767740 1 -- 172.21.15.90:0/2300517043 >> 172.21.15.90:0/2300517043 conn(0x559abcfa25e0 msgr2=0x559abcfa4330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:46:32.423 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.426+0000 7f21b1600640 0 client.28861 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:46:32.423 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.427+0000 7f21d1767740 1 -- 172.21.15.90:0/2300517043 shutdown_connections 2024-04-02T14:46:32.423 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.427+0000 7f21d1767740 1 -- 172.21.15.90:0/2300517043 wait complete. 2024-04-02T14:46:32.424 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.428+0000 7f21d1767740 20 client.28861 trim_cache size 0 max 16384 2024-04-02T14:46:32.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:46:32 smithi177 ceph-mon[36431]: pgmap v362: 97 pgs: 97 active+clean; 4.3 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 7.8 MiB/s wr, 831 op/s 2024-04-02T14:46:32.432 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:46:32 smithi090 ceph-mon[31674]: pgmap v362: 97 pgs: 97 active+clean; 4.3 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 7.8 MiB/s wr, 831 op/s 2024-04-02T14:46:32.446 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.450+0000 7f21d1767740 1 Processor -- start 2024-04-02T14:46:32.450 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.450+0000 7f21d1767740 1 -- start start 2024-04-02T14:46:32.450 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.451+0000 7f21d1767740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559abcfa7cc0 0x559abcfb9370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:32.450 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.451+0000 7f21d1767740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559abcfa8670 0x559abcfc58b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:32.450 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.451+0000 7f21d1767740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559abcfb9bc0 0x559abcfd1d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:32.450 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.451+0000 7f21d1767740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f21c81ab770 con 0x559abcfa7cc0 2024-04-02T14:46:32.450 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.451+0000 7f21d1767740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f21c81ad2b0 con 0x559abcfa8670 2024-04-02T14:46:32.450 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.451+0000 7f21d1767740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f21c81abe40 con 0x559abcfb9bc0 2024-04-02T14:46:32.450 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.452+0000 7f21c7400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559abcfa8670 0x559abcfc58b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:32.450 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.452+0000 7f21c7400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559abcfa8670 0x559abcfc58b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:43866/0 (socket says 172.21.15.90:43866) 2024-04-02T14:46:32.450 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.452+0000 7f21c7400640 1 -- 172.21.15.90:0/385519631 learned_addr learned my addr 172.21.15.90:0/385519631 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:46:32.451 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.453+0000 7f21c7e00640 1 --2- 172.21.15.90:0/385519631 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559abcfa7cc0 0x559abcfb9370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:32.451 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.453+0000 7f21c7e00640 1 -- 172.21.15.90:0/385519631 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559abcfb9bc0 msgr2=0x559abcfd1d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:46:32.451 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.453+0000 7f21c7e00640 1 --2- 172.21.15.90:0/385519631 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559abcfb9bc0 0x559abcfd1d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:32.451 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.453+0000 7f21c7e00640 1 -- 172.21.15.90:0/385519631 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559abcfa8670 msgr2=0x559abcfc58b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:46:32.451 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.453+0000 7f21c7e00640 1 --2- 172.21.15.90:0/385519631 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559abcfa8670 0x559abcfc58b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:32.451 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.453+0000 7f21c7e00640 1 -- 172.21.15.90:0/385519631 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559abcced620 con 0x559abcfa7cc0 2024-04-02T14:46:32.451 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.454+0000 7f21c7e00640 1 --2- 172.21.15.90:0/385519631 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559abcfa7cc0 0x559abcfb9370 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f21c00ce720 tx=0x7f21c0009200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6852c37801f49bf1 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:46:32.451 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.455+0000 7f21b3e00640 1 -- 172.21.15.90:0/385519631 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21c0010e60 con 0x559abcfa7cc0 2024-04-02T14:46:32.451 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.455+0000 7f21b3e00640 1 -- 172.21.15.90:0/385519631 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x559abcebccd0 con 0x559abcfa7cc0 2024-04-02T14:46:32.452 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.455+0000 7f21b3e00640 1 -- 172.21.15.90:0/385519631 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21c009d030 con 0x559abcfa7cc0 2024-04-02T14:46:32.453 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.457+0000 7f21d1767740 1 -- 172.21.15.90:0/385519631 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559abcfa7cc0 msgr2=0x559abcfb9370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:32.453 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.457+0000 7f21d1767740 1 --2- 172.21.15.90:0/385519631 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559abcfa7cc0 0x559abcfb9370 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f21c00ce720 tx=0x7f21c0009200 comp rx=0 tx=0).stop 2024-04-02T14:46:32.454 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.458+0000 7f21d1767740 1 -- 172.21.15.90:0/385519631 shutdown_connections 2024-04-02T14:46:32.454 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.458+0000 7f21d1767740 1 --2- 172.21.15.90:0/385519631 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559abcfb9bc0 0x559abcfd1d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:32.455 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.458+0000 7f21d1767740 1 --2- 172.21.15.90:0/385519631 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559abcfa8670 0x559abcfc58b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:32.455 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.458+0000 7f21d1767740 1 --2- 172.21.15.90:0/385519631 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559abcfa7cc0 0x559abcfb9370 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:32.455 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.458+0000 7f21d1767740 1 -- 172.21.15.90:0/385519631 >> 172.21.15.90:0/385519631 conn(0x559abcfa25e0 msgr2=0x559abcfa29c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:46:32.455 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.459+0000 7f21d1767740 1 -- 172.21.15.90:0/385519631 shutdown_connections 2024-04-02T14:46:32.455 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.459+0000 7f21d1767740 1 -- 172.21.15.90:0/385519631 wait complete. 2024-04-02T14:46:32.457 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.461+0000 7f21d1767740 1 -- start start 2024-04-02T14:46:32.457 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.461+0000 7f21d1767740 1 Processor -- start 2024-04-02T14:46:32.457 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.462+0000 7f21d1767740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559abcfa7cc0 0x559abcfc1dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:32.460 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.462+0000 7f21d1767740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559abcfa8670 0x559abcfc2300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:32.460 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.462+0000 7f21d1767740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559abcfb9bc0 0x559abcfb3e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:32.460 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.462+0000 7f21d1767740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f21c81acbe0 con 0x559abcfa7cc0 2024-04-02T14:46:32.461 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.462+0000 7f21d1767740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f21c81ac510 con 0x559abcfb9bc0 2024-04-02T14:46:32.461 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.462+0000 7f21d1767740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f21c81a97d0 con 0x559abcfa8670 2024-04-02T14:46:32.461 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.462+0000 7f21d1767740 10 client.0 _ll_register_callbacks cb 0x7f21b8009c90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:46:32.461 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.463+0000 7f21cca00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559abcfb9bc0 0x559abcfb3e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:32.462 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.463+0000 7f21cca00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559abcfb9bc0 0x559abcfb3e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:43876/0 (socket says 172.21.15.90:43876) 2024-04-02T14:46:32.462 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.463+0000 7f21cca00640 1 -- 172.21.15.90:0/989202675 learned_addr learned my addr 172.21.15.90:0/989202675 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:46:32.462 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.463+0000 7f21c7400640 1 --2- 172.21.15.90:0/989202675 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559abcfa8670 0x559abcfc2300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:32.462 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.463+0000 7f21c7e00640 1 --2- 172.21.15.90:0/989202675 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559abcfa7cc0 0x559abcfc1dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:32.462 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.464+0000 7f21c7400640 1 -- 172.21.15.90:0/989202675 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559abcfb9bc0 msgr2=0x559abcfb3e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:46:32.462 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.464+0000 7f21c7400640 1 --2- 172.21.15.90:0/989202675 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559abcfb9bc0 0x559abcfb3e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:32.462 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.464+0000 7f21c7400640 1 -- 172.21.15.90:0/989202675 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559abcfa7cc0 msgr2=0x559abcfc1dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:46:32.462 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.464+0000 7f21c7400640 1 --2- 172.21.15.90:0/989202675 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559abcfa7cc0 0x559abcfc1dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:32.463 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.464+0000 7f21c7400640 1 -- 172.21.15.90:0/989202675 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559abcfb4350 con 0x559abcfa8670 2024-04-02T14:46:32.463 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.464+0000 7f21cca00640 1 --2- 172.21.15.90:0/989202675 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559abcfb9bc0 0x559abcfb3e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:46:32.463 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.464+0000 7f21c7e00640 1 --2- 172.21.15.90:0/989202675 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559abcfa7cc0 0x559abcfc1dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:46:32.463 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.465+0000 7f21c7400640 1 --2- 172.21.15.90:0/989202675 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559abcfa8670 0x559abcfc2300 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f21b8001eb0 tx=0x7f21b8005cc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=11ef2d109332cc51 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:46:32.463 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.465+0000 7f21b1600640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:46:32.463 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.466+0000 7f21b1600640 1 -- 172.21.15.90:0/989202675 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21b8031060 con 0x559abcfa8670 2024-04-02T14:46:32.464 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.466+0000 7f21b1600640 1 -- 172.21.15.90:0/989202675 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 8550+0+0 (secure 0 0 0) 0x7f21b8011cc0 con 0x559abcfa8670 2024-04-02T14:46:32.464 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.468+0000 7f21d1767740 1 -- 172.21.15.90:0/989202675 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x559abcebccd0 con 0x559abcfa8670 2024-04-02T14:46:32.465 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.470+0000 7f21b1600640 1 -- 172.21.15.90:0/989202675 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f21b802c450 con 0x559abcfa8670 2024-04-02T14:46:32.465 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.470+0000 7f21d1767740 10 client.28864 fetch_fsmap learned FSMap version 27 2024-04-02T14:46:32.465 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.470+0000 7f21d1767740 1 -- 172.21.15.90:0/989202675 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x559abd041d20 con 0x559abcfa8670 2024-04-02T14:46:32.468 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.472+0000 7f21b1600640 1 -- 172.21.15.90:0/989202675 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x559abcebccd0 con 0x559abcfa8670 2024-04-02T14:46:32.473 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.473+0000 7f21b1600640 1 -- 172.21.15.90:0/989202675 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f21b8023720 con 0x559abcfa8670 2024-04-02T14:46:32.473 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.473+0000 7f21d1767740 10 client.28864 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:46:32.473 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.473+0000 7f21d1767740 10 client.28864 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-02T14:46:32.473 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.473+0000 7f21d1767740 20 client.28864 populate_metadata read hostname 'smithi090' 2024-04-02T14:46:32.473 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.473+0000 7f21d1767740 1 --2- 172.21.15.90:0/989202675 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x559abd042480 0x559abd0448a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:32.473 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.473+0000 7f21d1767740 4 client.28864 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-02T14:46:32.474 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.473+0000 7f21d1767740 1 -- 172.21.15.90:0/989202675 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "flush journal"}) -- 0x559abcf1a2b0 con 0x559abd042480 2024-04-02T14:46:32.474 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.473+0000 7f21c7e00640 1 --2- 172.21.15.90:0/989202675 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x559abd042480 0x559abd0448a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:32.474 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.474+0000 7f21c7e00640 1 --2- 172.21.15.90:0/989202675 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x559abd042480 0x559abd0448a0 crc :-1 s=READY pgs=303 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=44e346a436d9a4aa server_cookie=7a3574ad2e2a4841 in_seq=0 out_seq=0 2024-04-02T14:46:32.474 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.474+0000 7f21b1600640 10 client.28864 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:46:32.585 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.588+0000 7f21b1600640 1 -- 172.21.15.90:0/989202675 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x559abcf1a2b0 con 0x559abd042480 2024-04-02T14:46:32.585 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.588+0000 7f21b1600640 10 client.28864 handle_command_reply: tid=0 2024-04-02T14:46:32.585 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.588+0000 7f21d1767740 1 client.28864 shutdown 2024-04-02T14:46:32.585 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.588+0000 7f21d1767740 1 -- 172.21.15.90:0/989202675 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559abcfa8670 msgr2=0x559abcfc2300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:32.585 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.588+0000 7f21d1767740 1 --2- 172.21.15.90:0/989202675 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559abcfa8670 0x559abcfc2300 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f21b8001eb0 tx=0x7f21b8005cc0 comp rx=0 tx=0).stop 2024-04-02T14:46:32.585 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.588+0000 7f21d1767740 1 -- 172.21.15.90:0/989202675 shutdown_connections 2024-04-02T14:46:32.585 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.588+0000 7f21d1767740 1 --2- 172.21.15.90:0/989202675 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x559abd042480 0x559abd0448a0 crc :-1 s=READY pgs=303 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:32.585 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.588+0000 7f21d1767740 1 --2- 172.21.15.90:0/989202675 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559abcfb9bc0 0x559abcfb3e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:32.585 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.588+0000 7f21d1767740 1 --2- 172.21.15.90:0/989202675 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559abcfa8670 0x559abcfc2300 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:32.585 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.588+0000 7f21d1767740 1 --2- 172.21.15.90:0/989202675 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559abcfa7cc0 0x559abcfc1dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:32.586 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.588+0000 7f21d1767740 1 -- 172.21.15.90:0/989202675 >> 172.21.15.90:0/989202675 conn(0x559abcfa25e0 msgr2=0x559abcfb6be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:46:32.586 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.588+0000 7f21b1600640 0 client.28864 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:46:32.586 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.589+0000 7f21d1767740 1 -- 172.21.15.90:0/989202675 shutdown_connections 2024-04-02T14:46:32.586 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.589+0000 7f21d1767740 1 -- 172.21.15.90:0/989202675 wait complete. 2024-04-02T14:46:32.586 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.589+0000 7f21d1767740 20 client.28864 trim_cache size 0 max 16384 2024-04-02T14:46:32.586 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:{ 2024-04-02T14:46:32.586 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "message": "", 2024-04-02T14:46:32.586 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "return_code": 0 2024-04-02T14:46:32.587 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:} 2024-04-02T14:46:32.587 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.590+0000 7f21cea00640 1 -- 172.21.15.90:0/188712178 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f21ac0a3e40 msgr2=0x7f21ac0a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:32.587 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.590+0000 7f21cea00640 1 --2- 172.21.15.90:0/188712178 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f21ac0a3e40 0x7f21ac0a6300 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f21b4005ab0 tx=0x7f21b402e040 comp rx=0 tx=0).stop 2024-04-02T14:46:32.587 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.590+0000 7f21cea00640 1 -- 172.21.15.90:0/188712178 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f21c8103f40 msgr2=0x7f21c819a500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:32.587 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.590+0000 7f21cea00640 1 --2- 172.21.15.90:0/188712178 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f21c8103f40 0x7f21c819a500 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f21c000ebc0 tx=0x7f21c000c6a0 comp rx=0 tx=0).stop 2024-04-02T14:46:32.587 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.590+0000 7f21cea00640 1 -- 172.21.15.90:0/188712178 shutdown_connections 2024-04-02T14:46:32.588 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.590+0000 7f21cea00640 1 --2- 172.21.15.90:0/188712178 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f21ac0a3e40 0x7f21ac0a6300 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:32.588 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.590+0000 7f21cea00640 1 --2- 172.21.15.90:0/188712178 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f21c819edc0 0x7f21c819f270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:32.588 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.590+0000 7f21cea00640 1 --2- 172.21.15.90:0/188712178 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f21c8109010 0x7f21c819aa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:32.588 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.590+0000 7f21cea00640 1 --2- 172.21.15.90:0/188712178 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f21c8103f40 0x7f21c819a500 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:32.588 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.590+0000 7f21cea00640 1 -- 172.21.15.90:0/188712178 >> 172.21.15.90:0/188712178 conn(0x7f21c80ff7c0 msgr2=0x7f21c8100160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:46:32.588 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.590+0000 7f21cea00640 1 -- 172.21.15.90:0/188712178 shutdown_connections 2024-04-02T14:46:32.588 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:32.591+0000 7f21cea00640 1 -- 172.21.15.90:0/188712178 wait complete. 2024-04-02T14:46:32.604 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:46:32 smithi092 ceph-mon[36497]: pgmap v362: 97 pgs: 97 active+clean; 4.3 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 7.8 MiB/s wr, 831 op/s 2024-04-02T14:46:34.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:46:34 smithi177 ceph-mon[36431]: pgmap v363: 97 pgs: 97 active+clean; 4.3 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 6.0 MiB/s wr, 622 op/s 2024-04-02T14:46:34.432 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:46:34 smithi090 ceph-mon[31674]: pgmap v363: 97 pgs: 97 active+clean; 4.3 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 6.0 MiB/s wr, 622 op/s 2024-04-02T14:46:34.604 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:46:34 smithi092 ceph-mon[36497]: pgmap v363: 97 pgs: 97 active+clean; 4.3 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 6.0 MiB/s wr, 622 op/s 2024-04-02T14:46:36.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:46:36 smithi177 ceph-mon[36431]: pgmap v364: 97 pgs: 97 active+clean; 4.3 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 8.8 MiB/s wr, 894 op/s 2024-04-02T14:46:36.432 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:46:36 smithi090 ceph-mon[31674]: pgmap v364: 97 pgs: 97 active+clean; 4.3 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 8.8 MiB/s wr, 894 op/s 2024-04-02T14:46:36.604 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:46:36 smithi092 ceph-mon[36497]: pgmap v364: 97 pgs: 97 active+clean; 4.3 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 8.8 MiB/s wr, 894 op/s 2024-04-02T14:46:36.825 INFO:teuthology.orchestra.run.smithi090.stderr:progress: 665.0 s, 1788.4 tps, lat 54.466 ms stddev 60.409 2024-04-02T14:46:38.432 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:46:38 smithi090 ceph-mon[31674]: pgmap v365: 97 pgs: 97 active+clean; 4.3 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 5.8 MiB/s wr, 601 op/s 2024-04-02T14:46:38.604 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:46:38 smithi092 ceph-mon[36497]: pgmap v365: 97 pgs: 97 active+clean; 4.3 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 5.8 MiB/s wr, 601 op/s 2024-04-02T14:46:38.639 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-02T14:46:38.640 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-02T14:46:38.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:46:38 smithi177 ceph-mon[36431]: pgmap v365: 97 pgs: 97 active+clean; 4.3 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 5.8 MiB/s wr, 601 op/s 2024-04-02T14:46:38.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.901+0000 7f9e8bc00640 1 -- 172.21.15.90:0/4243798017 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9e84108cc0 msgr2=0x7f9e8410cd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:38.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.901+0000 7f9e8bc00640 1 --2- 172.21.15.90:0/4243798017 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9e84108cc0 0x7f9e8410cd00 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f9e70009900 tx=0x7f9e7002bc30 comp rx=0 tx=0).stop 2024-04-02T14:46:38.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.901+0000 7f9e8bc00640 1 -- 172.21.15.90:0/4243798017 shutdown_connections 2024-04-02T14:46:38.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.901+0000 7f9e8bc00640 1 --2- 172.21.15.90:0/4243798017 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9e8410d3e0 0x7f9e84114fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:38.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.901+0000 7f9e8bc00640 1 --2- 172.21.15.90:0/4243798017 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9e84108cc0 0x7f9e8410cd00 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:38.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.901+0000 7f9e8bc00640 1 --2- 172.21.15.90:0/4243798017 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9e84108310 0x7f9e841086f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:38.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.901+0000 7f9e8bc00640 1 -- 172.21.15.90:0/4243798017 >> 172.21.15.90:0/4243798017 conn(0x7f9e84076040 msgr2=0x7f9e84078460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:46:38.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.901+0000 7f9e8bc00640 1 -- 172.21.15.90:0/4243798017 shutdown_connections 2024-04-02T14:46:38.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.901+0000 7f9e8bc00640 1 -- 172.21.15.90:0/4243798017 wait complete. 2024-04-02T14:46:38.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.902+0000 7f9e8bc00640 1 Processor -- start 2024-04-02T14:46:38.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.902+0000 7f9e8bc00640 1 -- start start 2024-04-02T14:46:38.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.903+0000 7f9e8bc00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9e84108310 0x7f9e841a46b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:38.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.903+0000 7f9e8bc00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9e8410d3e0 0x7f9e841a4bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:38.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.903+0000 7f9e8bc00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9e8419e820 0x7f9e8419ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:38.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.903+0000 7f9e8bc00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f9e841184c0 con 0x7f9e84108310 2024-04-02T14:46:38.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.903+0000 7f9e8bc00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f9e84118340 con 0x7f9e8410d3e0 2024-04-02T14:46:38.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.903+0000 7f9e8bc00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f9e84118640 con 0x7f9e8419e820 2024-04-02T14:46:38.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.904+0000 7f9e83e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9e8410d3e0 0x7f9e841a4bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:38.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.904+0000 7f9e89c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9e8419e820 0x7f9e8419ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:38.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.904+0000 7f9e83e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9e8410d3e0 0x7f9e841a4bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:47910/0 (socket says 172.21.15.90:47910) 2024-04-02T14:46:38.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.904+0000 7f9e83e00640 1 -- 172.21.15.90:0/3566591071 learned_addr learned my addr 172.21.15.90:0/3566591071 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:46:38.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.904+0000 7f9e83e00640 1 -- 172.21.15.90:0/3566591071 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9e8419e820 msgr2=0x7f9e8419ecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:38.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.904+0000 7f9e83e00640 1 --2- 172.21.15.90:0/3566591071 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9e8419e820 0x7f9e8419ecd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:38.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.904+0000 7f9e83e00640 1 -- 172.21.15.90:0/3566591071 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9e84108310 msgr2=0x7f9e841a46b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:38.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.904+0000 7f9e89200640 1 --2- 172.21.15.90:0/3566591071 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9e84108310 0x7f9e841a46b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:38.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.904+0000 7f9e83e00640 1 --2- 172.21.15.90:0/3566591071 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9e84108310 0x7f9e841a46b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:38.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.904+0000 7f9e83e00640 1 -- 172.21.15.90:0/3566591071 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e8419f3b0 con 0x7f9e8410d3e0 2024-04-02T14:46:38.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.905+0000 7f9e83e00640 1 --2- 172.21.15.90:0/3566591071 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9e8410d3e0 0x7f9e841a4bf0 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f9e7002c240 tx=0x7f9e70004360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:46:38.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.906+0000 7f9e81600640 1 -- 172.21.15.90:0/3566591071 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e70043020 con 0x7f9e8410d3e0 2024-04-02T14:46:38.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.906+0000 7f9e81600640 1 -- 172.21.15.90:0/3566591071 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9e700054e0 con 0x7f9e8410d3e0 2024-04-02T14:46:38.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.906+0000 7f9e81600640 1 -- 172.21.15.90:0/3566591071 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e70003b90 con 0x7f9e8410d3e0 2024-04-02T14:46:38.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.907+0000 7f9e8bc00640 1 -- 172.21.15.90:0/3566591071 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e8419f640 con 0x7f9e8410d3e0 2024-04-02T14:46:38.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.908+0000 7f9e8bc00640 1 -- 172.21.15.90:0/3566591071 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e841ab7d0 con 0x7f9e8410d3e0 2024-04-02T14:46:38.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.914+0000 7f9e8e98c740 1 Processor -- start 2024-04-02T14:46:38.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.915+0000 7f9e8e98c740 1 -- start start 2024-04-02T14:46:38.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.915+0000 7f9e81600640 1 -- 172.21.15.90:0/3566591071 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f9e70046020 con 0x7f9e8410d3e0 2024-04-02T14:46:38.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.915+0000 7f9e8e98c740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d012c637b0 0x55d012c63b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:38.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.915+0000 7f9e8e98c740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d012c640d0 0x55d012c6c770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:38.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.915+0000 7f9e8e98c740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d012c6ccb0 0x55d012c6f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:38.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.915+0000 7f9e8e98c740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55d012bc7770 con 0x55d012c6ccb0 2024-04-02T14:46:38.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.915+0000 7f9e8e98c740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55d012b6f700 con 0x55d012c637b0 2024-04-02T14:46:38.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.915+0000 7f9e8e98c740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55d012bc3720 con 0x55d012c640d0 2024-04-02T14:46:38.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.915+0000 7f9e89200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d012c640d0 0x55d012c6c770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:38.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.915+0000 7f9e89200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d012c640d0 0x55d012c6c770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:48290/0 (socket says 172.21.15.90:48290) 2024-04-02T14:46:38.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.915+0000 7f9e89200640 1 -- 172.21.15.90:0/2354725713 learned_addr learned my addr 172.21.15.90:0/2354725713 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:46:38.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.915+0000 7f9e89c00640 1 --2- 172.21.15.90:0/2354725713 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d012c637b0 0x55d012c63b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:38.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.916+0000 7f9e83e00640 1 --2- 172.21.15.90:0/2354725713 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d012c6ccb0 0x55d012c6f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:38.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.916+0000 7f9e89c00640 1 -- 172.21.15.90:0/2354725713 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d012c640d0 msgr2=0x55d012c6c770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:46:38.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.916+0000 7f9e89c00640 1 --2- 172.21.15.90:0/2354725713 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d012c640d0 0x55d012c6c770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:38.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.916+0000 7f9e89c00640 1 -- 172.21.15.90:0/2354725713 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d012c6ccb0 msgr2=0x55d012c6f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:46:38.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.916+0000 7f9e89c00640 1 --2- 172.21.15.90:0/2354725713 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d012c6ccb0 0x55d012c6f0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:38.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.916+0000 7f9e89c00640 1 -- 172.21.15.90:0/2354725713 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d0129a57d0 con 0x55d012c637b0 2024-04-02T14:46:38.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.916+0000 7f9e89c00640 1 --2- 172.21.15.90:0/2354725713 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d012c637b0 0x55d012c63b90 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f9e7c009eb0 tx=0x7f9e7c010740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=397254fc9c8ac65c server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:46:38.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.916+0000 7f9e76000640 1 -- 172.21.15.90:0/2354725713 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e7c016070 con 0x55d012c637b0 2024-04-02T14:46:38.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.917+0000 7f9e76000640 1 -- 172.21.15.90:0/2354725713 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9e7c004d20 con 0x55d012c637b0 2024-04-02T14:46:38.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.917+0000 7f9e76000640 1 -- 172.21.15.90:0/2354725713 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e7c012810 con 0x55d012c637b0 2024-04-02T14:46:38.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.917+0000 7f9e81600640 1 --2- 172.21.15.90:0/3566591071 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f9e6c0a3e70 0x7f9e6c0a6330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:38.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.917+0000 7f9e81600640 1 -- 172.21.15.90:0/3566591071 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 8550+0+0 (secure 0 0 0) 0x7f9e700f2720 con 0x7f9e8410d3e0 2024-04-02T14:46:38.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.917+0000 7f9e8e98c740 1 -- 172.21.15.90:0/2354725713 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d012c637b0 msgr2=0x55d012c63b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:38.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.917+0000 7f9e8e98c740 1 --2- 172.21.15.90:0/2354725713 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d012c637b0 0x55d012c63b90 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f9e7c009eb0 tx=0x7f9e7c010740 comp rx=0 tx=0).stop 2024-04-02T14:46:38.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.918+0000 7f9e8e98c740 1 -- 172.21.15.90:0/2354725713 shutdown_connections 2024-04-02T14:46:38.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.918+0000 7f9e8e98c740 1 --2- 172.21.15.90:0/2354725713 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d012c6ccb0 0x55d012c6f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:38.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.918+0000 7f9e8e98c740 1 --2- 172.21.15.90:0/2354725713 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d012c640d0 0x55d012c6c770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:38.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.918+0000 7f9e8e98c740 1 --2- 172.21.15.90:0/2354725713 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d012c637b0 0x55d012c63b90 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:38.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.918+0000 7f9e8e98c740 1 -- 172.21.15.90:0/2354725713 >> 172.21.15.90:0/2354725713 conn(0x55d012c5dee0 msgr2=0x55d012c5e2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:46:38.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.918+0000 7f9e89200640 1 --2- 172.21.15.90:0/3566591071 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f9e6c0a3e70 0x7f9e6c0a6330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:38.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.918+0000 7f9e8e98c740 1 -- 172.21.15.90:0/2354725713 shutdown_connections 2024-04-02T14:46:38.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.918+0000 7f9e8e98c740 1 -- 172.21.15.90:0/2354725713 wait complete. 2024-04-02T14:46:38.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.918+0000 7f9e89200640 1 --2- 172.21.15.90:0/3566591071 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f9e6c0a3e70 0x7f9e6c0a6330 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x55d012c63280 tx=0x7f9e78005f90 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:46:38.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.918+0000 7f9e8e98c740 1 -- start start 2024-04-02T14:46:38.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.919+0000 7f9e8e98c740 1 Processor -- start 2024-04-02T14:46:38.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.919+0000 7f9e8e98c740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d012c637b0 0x55d012c69cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:38.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.919+0000 7f9e8e98c740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d012c640d0 0x55d012c6a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:38.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.919+0000 7f9e8e98c740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d012c6ccb0 0x55d012c6a730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:38.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.919+0000 7f9e8e98c740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55d012c6faa0 con 0x55d012c640d0 2024-04-02T14:46:38.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.919+0000 7f9e8e98c740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55d012b82ac0 con 0x55d012c637b0 2024-04-02T14:46:38.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.919+0000 7f9e8e98c740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55d012c6fc20 con 0x55d012c6ccb0 2024-04-02T14:46:38.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.920+0000 7f9e8e98c740 10 client.0 _ll_register_callbacks cb 0x55d012b85180 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:46:38.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.920+0000 7f9e89200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d012c637b0 0x55d012c69cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:38.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.920+0000 7f9e89200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d012c637b0 0x55d012c69cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:47920/0 (socket says 172.21.15.90:47920) 2024-04-02T14:46:38.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.920+0000 7f9e89200640 1 -- 172.21.15.90:0/3689837573 learned_addr learned my addr 172.21.15.90:0/3689837573 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:46:38.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.921+0000 7f9e89200640 1 -- 172.21.15.90:0/3689837573 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d012c6ccb0 msgr2=0x55d012c6a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:46:38.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.921+0000 7f9e89200640 1 --2- 172.21.15.90:0/3689837573 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d012c6ccb0 0x55d012c6a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:38.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.921+0000 7f9e89200640 1 -- 172.21.15.90:0/3689837573 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d012c640d0 msgr2=0x55d012c6a1f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-02T14:46:38.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.921+0000 7f9e83e00640 1 --2- 172.21.15.90:0/3689837573 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d012c640d0 0x55d012c6a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:38.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.921+0000 7f9e89200640 1 --2- 172.21.15.90:0/3689837573 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d012c640d0 0x55d012c6a1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:38.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.921+0000 7f9e89200640 1 -- 172.21.15.90:0/3689837573 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d0128db3f0 con 0x55d012c637b0 2024-04-02T14:46:38.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.921+0000 7f9e83e00640 1 --2- 172.21.15.90:0/3689837573 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d012c640d0 0x55d012c6a1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:46:38.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.922+0000 7f9e89200640 1 --2- 172.21.15.90:0/3689837573 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d012c637b0 0x55d012c69cb0 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f9e7801e8d0 tx=0x7f9e780046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=32dc4410ea811762 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:46:38.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.922+0000 7f9e6aa00640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:46:38.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.922+0000 7f9e6aa00640 1 -- 172.21.15.90:0/3689837573 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e7802b060 con 0x55d012c637b0 2024-04-02T14:46:38.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.922+0000 7f9e6aa00640 1 -- 172.21.15.90:0/3689837573 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 8550+0+0 (secure 0 0 0) 0x7f9e78011b70 con 0x55d012c637b0 2024-04-02T14:46:38.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.922+0000 7f9e8e98c740 1 -- 172.21.15.90:0/3689837573 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9e7c004d20 con 0x55d012c637b0 2024-04-02T14:46:38.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.923+0000 7f9e6aa00640 1 -- 172.21.15.90:0/3689837573 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f9e78011d70 con 0x55d012c637b0 2024-04-02T14:46:38.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.923+0000 7f9e8e98c740 10 client.29075 fetch_fsmap learned FSMap version 27 2024-04-02T14:46:38.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.923+0000 7f9e8e98c740 1 -- 172.21.15.90:0/3689837573 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55d01289b0f0 con 0x55d012c637b0 2024-04-02T14:46:38.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.923+0000 7f9e6aa00640 1 -- 172.21.15.90:0/3689837573 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9e7c004d20 con 0x55d012c637b0 2024-04-02T14:46:38.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.923+0000 7f9e6aa00640 1 -- 172.21.15.90:0/3689837573 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f9e7801a040 con 0x55d012c637b0 2024-04-02T14:46:38.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.924+0000 7f9e8e98c740 10 client.29075 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:46:38.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.924+0000 7f9e8e98c740 10 client.29075 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:46:38.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.924+0000 7f9e8e98c740 20 client.29075 populate_metadata read hostname 'smithi090' 2024-04-02T14:46:38.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.924+0000 7f9e8e98c740 1 --2- 172.21.15.90:0/3689837573 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55d012c8dc50 0x55d012c900e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:38.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.924+0000 7f9e8e98c740 4 client.29075 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:46:38.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.924+0000 7f9e8e98c740 1 -- 172.21.15.90:0/3689837573 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d012b68630 con 0x55d012c8dc50 2024-04-02T14:46:38.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.924+0000 7f9e83e00640 1 --2- 172.21.15.90:0/3689837573 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55d012c8dc50 0x55d012c900e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:38.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.925+0000 7f9e83e00640 1 --2- 172.21.15.90:0/3689837573 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55d012c8dc50 0x55d012c900e0 crc :-1 s=READY pgs=304 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d6a86a2e3b07e99e server_cookie=7dd9e282dfe7948 in_seq=0 out_seq=0 2024-04-02T14:46:38.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.925+0000 7f9e6aa00640 10 client.29075 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:46:38.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.926+0000 7f9e6aa00640 1 -- 172.21.15.90:0/3689837573 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d012b68630 con 0x55d012c8dc50 2024-04-02T14:46:38.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.926+0000 7f9e6aa00640 10 client.29075 handle_command_reply: tid=0 2024-04-02T14:46:38.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.926+0000 7f9e8e98c740 1 client.29075 shutdown 2024-04-02T14:46:38.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.926+0000 7f9e8e98c740 1 -- 172.21.15.90:0/3689837573 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d012c637b0 msgr2=0x55d012c69cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:38.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.926+0000 7f9e8e98c740 1 --2- 172.21.15.90:0/3689837573 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d012c637b0 0x55d012c69cb0 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f9e7801e8d0 tx=0x7f9e780046c0 comp rx=0 tx=0).stop 2024-04-02T14:46:38.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.926+0000 7f9e8e98c740 1 -- 172.21.15.90:0/3689837573 shutdown_connections 2024-04-02T14:46:38.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.926+0000 7f9e8e98c740 1 --2- 172.21.15.90:0/3689837573 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55d012c8dc50 0x55d012c900e0 crc :-1 s=READY pgs=304 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:38.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.926+0000 7f9e8e98c740 1 --2- 172.21.15.90:0/3689837573 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d012c6ccb0 0x55d012c6a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:38.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.926+0000 7f9e8e98c740 1 --2- 172.21.15.90:0/3689837573 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d012c640d0 0x55d012c6a1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:38.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.926+0000 7f9e8e98c740 1 --2- 172.21.15.90:0/3689837573 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d012c637b0 0x55d012c69cb0 secure :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f9e7801e8d0 tx=0x7f9e780046c0 comp rx=0 tx=0).stop 2024-04-02T14:46:38.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.926+0000 7f9e8e98c740 1 -- 172.21.15.90:0/3689837573 >> 172.21.15.90:0/3689837573 conn(0x55d012c5dee0 msgr2=0x55d012c5fc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:46:38.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.926+0000 7f9e6aa00640 0 client.29075 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:46:38.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.927+0000 7f9e8e98c740 1 -- 172.21.15.90:0/3689837573 shutdown_connections 2024-04-02T14:46:38.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.927+0000 7f9e8e98c740 1 -- 172.21.15.90:0/3689837573 wait complete. 2024-04-02T14:46:38.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.927+0000 7f9e8e98c740 20 client.29075 trim_cache size 0 max 16384 2024-04-02T14:46:38.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.959+0000 7f9e8e98c740 1 Processor -- start 2024-04-02T14:46:38.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.959+0000 7f9e8e98c740 1 -- start start 2024-04-02T14:46:38.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.960+0000 7f9e8e98c740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d012c640d0 0x55d012c6c3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:38.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.960+0000 7f9e8e98c740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d012c6ccb0 0x55d012c78990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:38.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.960+0000 7f9e8e98c740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d012c6a380 0x55d012c6a800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:38.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.960+0000 7f9e8e98c740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f9e840827e0 con 0x55d012c640d0 2024-04-02T14:46:38.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.960+0000 7f9e8e98c740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f9e841ad1f0 con 0x55d012c6a380 2024-04-02T14:46:38.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.960+0000 7f9e8e98c740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f9e841abd80 con 0x55d012c6ccb0 2024-04-02T14:46:38.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.960+0000 7f9e89200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d012c640d0 0x55d012c6c3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:38.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.960+0000 7f9e89c00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d012c6a380 0x55d012c6a800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:38.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.960+0000 7f9e89200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d012c640d0 0x55d012c6c3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:39988/0 (socket says 172.21.15.90:39988) 2024-04-02T14:46:38.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.960+0000 7f9e89200640 1 -- 172.21.15.90:0/1846674033 learned_addr learned my addr 172.21.15.90:0/1846674033 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:46:38.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.960+0000 7f9e83e00640 1 --2- 172.21.15.90:0/1846674033 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d012c6ccb0 0x55d012c78990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:38.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.961+0000 7f9e89c00640 1 -- 172.21.15.90:0/1846674033 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d012c6ccb0 msgr2=0x55d012c78990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:46:38.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.961+0000 7f9e89c00640 1 --2- 172.21.15.90:0/1846674033 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d012c6ccb0 0x55d012c78990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:38.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.961+0000 7f9e89c00640 1 -- 172.21.15.90:0/1846674033 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d012c640d0 msgr2=0x55d012c6c3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:46:38.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.961+0000 7f9e89c00640 1 --2- 172.21.15.90:0/1846674033 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d012c640d0 0x55d012c6c3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:38.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.961+0000 7f9e89c00640 1 -- 172.21.15.90:0/1846674033 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d012c64a50 con 0x55d012c6a380 2024-04-02T14:46:38.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.961+0000 7f9e89c00640 1 --2- 172.21.15.90:0/1846674033 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d012c6a380 0x55d012c6a800 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f9e7c000f80 tx=0x7f9e7c009c30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=103ac89345a1c770 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:46:38.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.961+0000 7f9e75600640 1 -- 172.21.15.90:0/1846674033 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e7c016030 con 0x55d012c6a380 2024-04-02T14:46:38.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.961+0000 7f9e75600640 1 -- 172.21.15.90:0/1846674033 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9e7c003c70 con 0x55d012c6a380 2024-04-02T14:46:38.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.962+0000 7f9e75600640 1 -- 172.21.15.90:0/1846674033 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e7c0128f0 con 0x55d012c6a380 2024-04-02T14:46:38.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.962+0000 7f9e8e98c740 1 -- 172.21.15.90:0/1846674033 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d012c6a380 msgr2=0x55d012c6a800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:38.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.962+0000 7f9e8e98c740 1 --2- 172.21.15.90:0/1846674033 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d012c6a380 0x55d012c6a800 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f9e7c000f80 tx=0x7f9e7c009c30 comp rx=0 tx=0).stop 2024-04-02T14:46:38.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.962+0000 7f9e8e98c740 1 -- 172.21.15.90:0/1846674033 shutdown_connections 2024-04-02T14:46:38.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.962+0000 7f9e8e98c740 1 --2- 172.21.15.90:0/1846674033 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d012c6a380 0x55d012c6a800 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:38.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.962+0000 7f9e8e98c740 1 --2- 172.21.15.90:0/1846674033 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d012c6ccb0 0x55d012c78990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:38.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.962+0000 7f9e8e98c740 1 --2- 172.21.15.90:0/1846674033 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d012c640d0 0x55d012c6c3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:38.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.962+0000 7f9e8e98c740 1 -- 172.21.15.90:0/1846674033 >> 172.21.15.90:0/1846674033 conn(0x55d012c5dee0 msgr2=0x55d012c69c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:46:38.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.963+0000 7f9e8e98c740 1 -- 172.21.15.90:0/1846674033 shutdown_connections 2024-04-02T14:46:38.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.963+0000 7f9e8e98c740 1 -- 172.21.15.90:0/1846674033 wait complete. 2024-04-02T14:46:38.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.963+0000 7f9e8e98c740 1 -- start start 2024-04-02T14:46:38.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.963+0000 7f9e8e98c740 1 Processor -- start 2024-04-02T14:46:38.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.964+0000 7f9e8e98c740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d012c640d0 0x55d012c70b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:38.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.964+0000 7f9e8e98c740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d012c6a380 0x55d012c71040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:38.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.964+0000 7f9e8e98c740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d012c6ccb0 0x55d012c71580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:38.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.964+0000 7f9e8e98c740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f9e841acb20 con 0x55d012c6ccb0 2024-04-02T14:46:38.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.964+0000 7f9e8e98c740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f9e841ac450 con 0x55d012c640d0 2024-04-02T14:46:38.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.964+0000 7f9e8e98c740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f9e840807e0 con 0x55d012c6a380 2024-04-02T14:46:38.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.964+0000 7f9e8e98c740 10 client.0 _ll_register_callbacks cb 0x7f9e78009a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:46:38.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.964+0000 7f9e89c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d012c6ccb0 0x55d012c71580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:38.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.964+0000 7f9e83e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d012c6a380 0x55d012c71040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:38.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.964+0000 7f9e89200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d012c640d0 0x55d012c70b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:38.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.964+0000 7f9e83e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d012c6a380 0x55d012c71040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:48312/0 (socket says 172.21.15.90:48312) 2024-04-02T14:46:38.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.964+0000 7f9e89200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d012c640d0 0x55d012c70b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:47942/0 (socket says 172.21.15.90:47942) 2024-04-02T14:46:38.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.964+0000 7f9e83e00640 1 -- 172.21.15.90:0/1370634685 learned_addr learned my addr 172.21.15.90:0/1370634685 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:46:38.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.964+0000 7f9e89200640 1 -- 172.21.15.90:0/1370634685 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d012c6a380 msgr2=0x55d012c71040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:46:38.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.964+0000 7f9e89200640 1 --2- 172.21.15.90:0/1370634685 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d012c6a380 0x55d012c71040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:38.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.964+0000 7f9e89200640 1 -- 172.21.15.90:0/1370634685 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d012c6ccb0 msgr2=0x55d012c71580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:46:38.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.964+0000 7f9e89200640 1 --2- 172.21.15.90:0/1370634685 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d012c6ccb0 0x55d012c71580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:38.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.964+0000 7f9e89200640 1 -- 172.21.15.90:0/1370634685 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d012cf3470 con 0x55d012c640d0 2024-04-02T14:46:38.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.965+0000 7f9e89200640 1 --2- 172.21.15.90:0/1370634685 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d012c640d0 0x55d012c70b00 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f9e7801b580 tx=0x7f9e7801bdf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bc7bd629da673b5 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:46:38.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.965+0000 7f9e6aa00640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:46:38.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.965+0000 7f9e6aa00640 1 -- 172.21.15.90:0/1370634685 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e78035080 con 0x55d012c640d0 2024-04-02T14:46:38.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.965+0000 7f9e6aa00640 1 -- 172.21.15.90:0/1370634685 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 8550+0+0 (secure 0 0 0) 0x7f9e78005250 con 0x55d012c640d0 2024-04-02T14:46:38.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.965+0000 7f9e8e98c740 1 -- 172.21.15.90:0/1370634685 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9e7c003c70 con 0x55d012c640d0 2024-04-02T14:46:38.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.965+0000 7f9e6aa00640 1 -- 172.21.15.90:0/1370634685 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f9e7802b030 con 0x55d012c640d0 2024-04-02T14:46:38.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.965+0000 7f9e8e98c740 10 client.29081 fetch_fsmap learned FSMap version 27 2024-04-02T14:46:38.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.965+0000 7f9e8e98c740 1 -- 172.21.15.90:0/1370634685 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55d012cf4760 con 0x55d012c640d0 2024-04-02T14:46:38.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.966+0000 7f9e6aa00640 1 -- 172.21.15.90:0/1370634685 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9e7c003c70 con 0x55d012c640d0 2024-04-02T14:46:38.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.966+0000 7f9e6aa00640 1 -- 172.21.15.90:0/1370634685 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f9e7801a040 con 0x55d012c640d0 2024-04-02T14:46:38.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.966+0000 7f9e8e98c740 10 client.29081 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:46:38.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.966+0000 7f9e8e98c740 10 client.29081 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:46:38.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.966+0000 7f9e8e98c740 20 client.29081 populate_metadata read hostname 'smithi090' 2024-04-02T14:46:38.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.966+0000 7f9e8e98c740 1 --2- 172.21.15.90:0/1370634685 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55d012cf4ea0 0x55d012cf7290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:38.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.966+0000 7f9e8e98c740 4 client.29081 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-02T14:46:38.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.966+0000 7f9e8e98c740 1 -- 172.21.15.90:0/1370634685 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55d012bd4cb0 con 0x55d012cf4ea0 2024-04-02T14:46:38.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.966+0000 7f9e83e00640 1 --2- 172.21.15.90:0/1370634685 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55d012cf4ea0 0x55d012cf7290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:38.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.969+0000 7f9e83e00640 1 --2- 172.21.15.90:0/1370634685 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55d012cf4ea0 0x55d012cf7290 crc :-1 s=READY pgs=305 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ac38293d8559221f server_cookie=9603466589faaf42 in_seq=0 out_seq=0 2024-04-02T14:46:38.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.969+0000 7f9e6aa00640 10 client.29081 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:46:38.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.969+0000 7f9e6aa00640 1 -- 172.21.15.90:0/1370634685 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55d012bd4cb0 con 0x55d012cf4ea0 2024-04-02T14:46:38.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.969+0000 7f9e6aa00640 10 client.29081 handle_command_reply: tid=0 2024-04-02T14:46:38.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.970+0000 7f9e8e98c740 1 client.29081 shutdown 2024-04-02T14:46:38.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.970+0000 7f9e8e98c740 1 -- 172.21.15.90:0/1370634685 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d012c640d0 msgr2=0x55d012c70b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:38.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.970+0000 7f9e8e98c740 1 --2- 172.21.15.90:0/1370634685 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d012c640d0 0x55d012c70b00 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f9e7801b580 tx=0x7f9e7801bdf0 comp rx=0 tx=0).stop 2024-04-02T14:46:38.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.970+0000 7f9e8e98c740 1 -- 172.21.15.90:0/1370634685 shutdown_connections 2024-04-02T14:46:38.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.970+0000 7f9e8e98c740 1 --2- 172.21.15.90:0/1370634685 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55d012cf4ea0 0x55d012cf7290 crc :-1 s=READY pgs=305 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:38.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.970+0000 7f9e8e98c740 1 --2- 172.21.15.90:0/1370634685 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d012c6ccb0 0x55d012c71580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:38.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.970+0000 7f9e8e98c740 1 --2- 172.21.15.90:0/1370634685 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d012c6a380 0x55d012c71040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:38.969 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.970+0000 7f9e8e98c740 1 --2- 172.21.15.90:0/1370634685 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d012c640d0 0x55d012c70b00 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:38.969 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.970+0000 7f9e8e98c740 1 -- 172.21.15.90:0/1370634685 >> 172.21.15.90:0/1370634685 conn(0x55d012c5dee0 msgr2=0x55d012c5f1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:46:38.970 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.970+0000 7f9e6aa00640 0 client.29081 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:46:38.970 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.970+0000 7f9e8e98c740 1 -- 172.21.15.90:0/1370634685 shutdown_connections 2024-04-02T14:46:38.970 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.970+0000 7f9e8e98c740 1 -- 172.21.15.90:0/1370634685 wait complete. 2024-04-02T14:46:38.970 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.971+0000 7f9e8e98c740 20 client.29081 trim_cache size 0 max 16384 2024-04-02T14:46:38.972 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-02T14:46:38.972 INFO:teuthology.orchestra.run.smithi090.stdout: "status": "no active scrubs running", 2024-04-02T14:46:38.972 INFO:teuthology.orchestra.run.smithi090.stdout: "scrubs": {} 2024-04-02T14:46:38.973 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-02T14:46:38.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.971+0000 7f9e8bc00640 1 -- 172.21.15.90:0/3566591071 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f9e6c0a3e70 msgr2=0x7f9e6c0a6330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:38.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.971+0000 7f9e8bc00640 1 --2- 172.21.15.90:0/3566591071 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f9e6c0a3e70 0x7f9e6c0a6330 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x55d012c63280 tx=0x7f9e78005f90 comp rx=0 tx=0).stop 2024-04-02T14:46:38.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.971+0000 7f9e8bc00640 1 -- 172.21.15.90:0/3566591071 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9e8410d3e0 msgr2=0x7f9e841a4bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:38.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.971+0000 7f9e8bc00640 1 --2- 172.21.15.90:0/3566591071 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9e8410d3e0 0x7f9e841a4bf0 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f9e7002c240 tx=0x7f9e70004360 comp rx=0 tx=0).stop 2024-04-02T14:46:38.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.972+0000 7f9e8bc00640 1 -- 172.21.15.90:0/3566591071 shutdown_connections 2024-04-02T14:46:38.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.972+0000 7f9e8bc00640 1 --2- 172.21.15.90:0/3566591071 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f9e6c0a3e70 0x7f9e6c0a6330 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:38.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.972+0000 7f9e8bc00640 1 --2- 172.21.15.90:0/3566591071 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9e8419e820 0x7f9e8419ecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:38.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.972+0000 7f9e8bc00640 1 --2- 172.21.15.90:0/3566591071 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9e8410d3e0 0x7f9e841a4bf0 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:38.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.972+0000 7f9e8bc00640 1 --2- 172.21.15.90:0/3566591071 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9e84108310 0x7f9e841a46b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:38.976 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.972+0000 7f9e8bc00640 1 -- 172.21.15.90:0/3566591071 >> 172.21.15.90:0/3566591071 conn(0x7f9e84076040 msgr2=0x7f9e84078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:46:38.976 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.972+0000 7f9e8bc00640 1 -- 172.21.15.90:0/3566591071 shutdown_connections 2024-04-02T14:46:38.976 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:38.972+0000 7f9e8bc00640 1 -- 172.21.15.90:0/3566591071 wait complete. 2024-04-02T14:46:38.987 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-02T14:46:38.987 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T14:46:39.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:39.272+0000 7fdfdc000640 1 -- 172.21.15.90:0/1455953223 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdfd4108fd0 msgr2=0x7fdfd4110ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:39.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:39.272+0000 7fdfdc000640 1 --2- 172.21.15.90:0/1455953223 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdfd4108fd0 0x7fdfd4110ba0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7fdfcc009ca0 tx=0x7fdfcc02c120 comp rx=0 tx=0).stop 2024-04-02T14:46:39.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:39.272+0000 7fdfdc000640 1 -- 172.21.15.90:0/1455953223 shutdown_connections 2024-04-02T14:46:39.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:39.272+0000 7fdfdc000640 1 --2- 172.21.15.90:0/1455953223 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdfd4108fd0 0x7fdfd4110ba0 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:39.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:39.272+0000 7fdfdc000640 1 --2- 172.21.15.90:0/1455953223 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdfd41048b0 0x7fdfd41088f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:39.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:39.272+0000 7fdfdc000640 1 --2- 172.21.15.90:0/1455953223 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdfd4103f00 0x7fdfd41042e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:39.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:39.272+0000 7fdfdc000640 1 -- 172.21.15.90:0/1455953223 >> 172.21.15.90:0/1455953223 conn(0x7fdfd40ff760 msgr2=0x7fdfd4101b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:46:39.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:39.273+0000 7fdfdc000640 1 -- 172.21.15.90:0/1455953223 shutdown_connections 2024-04-02T14:46:39.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:39.273+0000 7fdfdc000640 1 -- 172.21.15.90:0/1455953223 wait complete. 2024-04-02T14:46:39.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:39.274+0000 7fdfdc000640 1 Processor -- start 2024-04-02T14:46:39.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:39.274+0000 7fdfdc000640 1 -- start start 2024-04-02T14:46:39.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:39.275+0000 7fdfdc000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdfd4103f00 0x7fdfd419a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:39.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:39.275+0000 7fdfdc000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdfd4108fd0 0x7fdfd419aa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:39.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:39.275+0000 7fdfdc000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdfd419ed80 0x7fdfd419f230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:39.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:39.275+0000 7fdfd9600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdfd4103f00 0x7fdfd419a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:39.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:39.275+0000 7fdfd9600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdfd4103f00 0x7fdfd419a4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:47966/0 (socket says 172.21.15.90:47966) 2024-04-02T14:46:39.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:39.275+0000 7fdfd8c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdfd4108fd0 0x7fdfd419aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:39.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:39.275+0000 7fdfd8c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdfd4108fd0 0x7fdfd419aa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:48344/0 (socket says 172.21.15.90:48344) 2024-04-02T14:46:39.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:39.276+0000 7fdfd8c00640 1 -- 172.21.15.90:0/3981764374 learned_addr learned my addr 172.21.15.90:0/3981764374 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:46:39.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:39.276+0000 7fdfdc000640 1 -- 172.21.15.90:0/3981764374 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fdfd4114160 con 0x7fdfd419ed80 2024-04-02T14:46:39.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:39.276+0000 7fdfdc000640 1 -- 172.21.15.90:0/3981764374 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fdfd4113fe0 con 0x7fdfd4103f00 2024-04-02T14:46:39.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:39.276+0000 7fdfdc000640 1 -- 172.21.15.90:0/3981764374 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fdfd41142e0 con 0x7fdfd4108fd0 2024-04-02T14:46:39.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:39.276+0000 7fdfda000640 1 --2- 172.21.15.90:0/3981764374 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdfd419ed80 0x7fdfd419f230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:39.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:39.276+0000 7fdfd8c00640 1 -- 172.21.15.90:0/3981764374 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdfd4103f00 msgr2=0x7fdfd419a4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:39.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:39.276+0000 7fdfd8c00640 1 --2- 172.21.15.90:0/3981764374 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdfd4103f00 0x7fdfd419a4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:39.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:39.276+0000 7fdfd8c00640 1 -- 172.21.15.90:0/3981764374 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdfd419ed80 msgr2=0x7fdfd419f230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:39.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:39.276+0000 7fdfd8c00640 1 --2- 172.21.15.90:0/3981764374 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdfd419ed80 0x7fdfd419f230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:39.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:39.276+0000 7fdfd8c00640 1 -- 172.21.15.90:0/3981764374 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdfd419f870 con 0x7fdfd4108fd0 2024-04-02T14:46:39.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:39.277+0000 7fdfd8c00640 1 --2- 172.21.15.90:0/3981764374 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdfd4108fd0 0x7fdfd419aa00 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7fdfc800b470 tx=0x7fdfc800b870 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:46:39.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:39.278+0000 7fdfd2000640 1 -- 172.21.15.90:0/3981764374 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfc8015020 con 0x7fdfd4108fd0 2024-04-02T14:46:39.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:39.278+0000 7fdfd2000640 1 -- 172.21.15.90:0/3981764374 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdfc8011d10 con 0x7fdfd4108fd0 2024-04-02T14:46:39.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:39.278+0000 7fdfd2000640 1 -- 172.21.15.90:0/3981764374 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfc800f2e0 con 0x7fdfd4108fd0 2024-04-02T14:46:39.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:39.278+0000 7fdfdc000640 1 -- 172.21.15.90:0/3981764374 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdfd419fb60 con 0x7fdfd4108fd0 2024-04-02T14:46:39.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:39.279+0000 7fdfdc000640 1 -- 172.21.15.90:0/3981764374 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdfd41a7300 con 0x7fdfd4108fd0 2024-04-02T14:46:39.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:39.281+0000 7fdfdc000640 1 -- 172.21.15.90:0/3981764374 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdfd4068d40 con 0x7fdfd4108fd0 2024-04-02T14:46:39.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:39.283+0000 7fdfd2000640 1 -- 172.21.15.90:0/3981764374 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fdfc8018020 con 0x7fdfd4108fd0 2024-04-02T14:46:39.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:39.286+0000 7fdfd2000640 1 --2- 172.21.15.90:0/3981764374 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fdfbc0a3da0 0x7fdfbc0a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:39.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:39.287+0000 7fdfd2000640 1 -- 172.21.15.90:0/3981764374 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 8550+0+0 (secure 0 0 0) 0x7fdfc80cff70 con 0x7fdfd4108fd0 2024-04-02T14:46:39.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:39.287+0000 7fdfd9600640 1 --2- 172.21.15.90:0/3981764374 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fdfbc0a3da0 0x7fdfbc0a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:39.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:39.288+0000 7fdfd9600640 1 --2- 172.21.15.90:0/3981764374 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fdfbc0a3da0 0x7fdfbc0a6260 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fdfc0009990 tx=0x7fdfc0009110 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:46:39.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:39.292+0000 7fdfd2000640 1 -- 172.21.15.90:0/3981764374 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7fdfc801f020 con 0x7fdfd4108fd0 2024-04-02T14:46:39.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:39.740+0000 7fdfdc000640 1 -- 172.21.15.90:0/3981764374 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdfd41048b0 con 0x7fdfd4108fd0 2024-04-02T14:46:39.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:39.741+0000 7fdfd2000640 1 -- 172.21.15.90:0/3981764374 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11927 (secure 0 0 0) 0x7fdfc809c700 con 0x7fdfd4108fd0 2024-04-02T14:46:39.737 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-02T14:46:39.737 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14412,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6837/2852160309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6836","nonce":2852160309},{"type":"v1","addr":"172.21.15.90:6837","nonce":2852160309}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14478,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6841/1919376858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6840","nonce":1919376858},{"type":"v1","addr":"172.21.15.90:6841","nonce":1919376858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-02T14:32:22.319083+0000","modified":"2024-04-02T14:34:52.266152+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24590,"mds_1":24560,"mds_2":24547,"mds_3":24539,"mds_4":24523},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24454":{"gid":24454,"name":"l","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6833/1564319431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":1564319431},{"type":"v1","addr":"172.21.15.177:6833","nonce":1564319431}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.92:6833/1570337372","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":1570337372},{"type":"v1","addr":"172.21.15.92:6833","nonce":1570337372}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"j","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.90:6835/1277124891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6834","nonce":1277124891},{"type":"v1","addr":"172.21.15.90:6835","nonce":1277124891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.92:6835/860721164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":860721164},{"type":"v1","addr":"172.21.15.92:6835","nonce":860721164}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24511":{"gid":24511,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6835/3177244988","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":3177244988},{"type":"v1","addr":"172.21.15.177:6835","nonce":3177244988}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"c","rank":4,"incarnation":25,"state":"up:active","state_seq":9,"addr":"172.21.15.177:6837/4223414858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":4223414858},{"type":"v1","addr":"172.21.15.177:6837","nonce":4223414858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24539":{"gid":24539,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":8,"addr":"172.21.15.92:6837/1270635939","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":1270635939},{"type":"v1","addr":"172.21.15.92:6837","nonce":1270635939}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24547":{"gid":24547,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":7,"addr":"172.21.15.177:6839/664895663","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":664895663},{"type":"v1","addr":"172.21.15.177:6839","nonce":664895663}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"g","rank":1,"incarnation":19,"state":"up:active","state_seq":5,"addr":"172.21.15.90:6839/1498915870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6838","nonce":1498915870},{"type":"v1","addr":"172.21.15.90:6839","nonce":1498915870}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24590":{"gid":24590,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.92:6839/1054888219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":1054888219},{"type":"v1","addr":"172.21.15.92:6839","nonce":1054888219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24590,"qdb_cluster":[24547,24590,24523,24539,24560]},"id":1}]} 2024-04-02T14:46:39.737 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 27 2024-04-02T14:46:39.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:39.744+0000 7fdfdc000640 1 -- 172.21.15.90:0/3981764374 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fdfbc0a3da0 msgr2=0x7fdfbc0a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:39.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:39.744+0000 7fdfdc000640 1 --2- 172.21.15.90:0/3981764374 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fdfbc0a3da0 0x7fdfbc0a6260 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fdfc0009990 tx=0x7fdfc0009110 comp rx=0 tx=0).stop 2024-04-02T14:46:39.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:39.744+0000 7fdfdc000640 1 -- 172.21.15.90:0/3981764374 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdfd4108fd0 msgr2=0x7fdfd419aa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:39.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:39.744+0000 7fdfdc000640 1 --2- 172.21.15.90:0/3981764374 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdfd4108fd0 0x7fdfd419aa00 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7fdfc800b470 tx=0x7fdfc800b870 comp rx=0 tx=0).stop 2024-04-02T14:46:39.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:39.744+0000 7fdfdc000640 1 -- 172.21.15.90:0/3981764374 shutdown_connections 2024-04-02T14:46:39.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:39.744+0000 7fdfdc000640 1 --2- 172.21.15.90:0/3981764374 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fdfbc0a3da0 0x7fdfbc0a6260 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:39.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:39.744+0000 7fdfdc000640 1 --2- 172.21.15.90:0/3981764374 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdfd419ed80 0x7fdfd419f230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:39.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:39.744+0000 7fdfdc000640 1 --2- 172.21.15.90:0/3981764374 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdfd4108fd0 0x7fdfd419aa00 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:39.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:39.744+0000 7fdfdc000640 1 --2- 172.21.15.90:0/3981764374 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdfd4103f00 0x7fdfd419a4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:39.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:39.744+0000 7fdfdc000640 1 -- 172.21.15.90:0/3981764374 >> 172.21.15.90:0/3981764374 conn(0x7fdfd40ff760 msgr2=0x7fdfd41000b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:46:39.741 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:39.745+0000 7fdfdc000640 1 -- 172.21.15.90:0/3981764374 shutdown_connections 2024-04-02T14:46:39.741 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:39.745+0000 7fdfdc000640 1 -- 172.21.15.90:0/3981764374 wait complete. 2024-04-02T14:46:39.769 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-04-02T14:46:40.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.075+0000 7f1b3e000640 1 -- 172.21.15.90:0/2121568394 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1b30005670 msgr2=0x7f1b300a2b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:40.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.075+0000 7f1b3e000640 1 --2- 172.21.15.90:0/2121568394 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1b30005670 0x7f1b300a2b90 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f1b380677d0 tx=0x7f1b38097a10 comp rx=0 tx=0).stop 2024-04-02T14:46:40.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.075+0000 7f1b3e000640 1 -- 172.21.15.90:0/2121568394 shutdown_connections 2024-04-02T14:46:40.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.075+0000 7f1b3e000640 1 --2- 172.21.15.90:0/2121568394 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1b30005670 0x7f1b300a2b90 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:40.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.075+0000 7f1b3e000640 1 --2- 172.21.15.90:0/2121568394 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1b30004cb0 0x7f1b30005130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:40.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.075+0000 7f1b3e000640 1 --2- 172.21.15.90:0/2121568394 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1b3009be10 0x7f1b3009c1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:40.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.075+0000 7f1b3e000640 1 -- 172.21.15.90:0/2121568394 >> 172.21.15.90:0/2121568394 conn(0x7f1b3000a740 msgr2=0x7f1b3000ab50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:46:40.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.075+0000 7f1b3e000640 1 -- 172.21.15.90:0/2121568394 shutdown_connections 2024-04-02T14:46:40.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.076+0000 7f1b3e000640 1 -- 172.21.15.90:0/2121568394 wait complete. 2024-04-02T14:46:40.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.077+0000 7f1b3e000640 1 Processor -- start 2024-04-02T14:46:40.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.077+0000 7f1b3e000640 1 -- start start 2024-04-02T14:46:40.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.078+0000 7f1b3e000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1b30004cb0 0x7f1b300aa270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:40.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.078+0000 7f1b3e000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1b30005670 0x7f1b300aa7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:40.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.078+0000 7f1b3cc00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1b30004cb0 0x7f1b300aa270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:40.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.078+0000 7f1b3cc00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1b30004cb0 0x7f1b300aa270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:47972/0 (socket says 172.21.15.90:47972) 2024-04-02T14:46:40.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.079+0000 7f1b3e000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1b3009be10 0x7f1b300a53d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:40.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.079+0000 7f1b3e000640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f1b300a3dc0 con 0x7f1b30005670 2024-04-02T14:46:40.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.079+0000 7f1b3e000640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f1b300a3c40 con 0x7f1b30004cb0 2024-04-02T14:46:40.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.079+0000 7f1b3e000640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f1b300a3f40 con 0x7f1b3009be10 2024-04-02T14:46:40.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.079+0000 7f1b3cc00640 1 -- 172.21.15.90:0/1985735605 learned_addr learned my addr 172.21.15.90:0/1985735605 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:46:40.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.079+0000 7f1b3d600640 1 --2- 172.21.15.90:0/1985735605 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1b3009be10 0x7f1b300a53d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:40.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.079+0000 7f1b3cc00640 1 -- 172.21.15.90:0/1985735605 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1b3009be10 msgr2=0x7f1b300a53d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:40.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.079+0000 7f1b3cc00640 1 --2- 172.21.15.90:0/1985735605 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1b3009be10 0x7f1b300a53d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:40.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.080+0000 7f1b3cc00640 1 -- 172.21.15.90:0/1985735605 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1b30005670 msgr2=0x7f1b300aa7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:40.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.080+0000 7f1b3cc00640 1 --2- 172.21.15.90:0/1985735605 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1b30005670 0x7f1b300aa7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:40.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.080+0000 7f1b3cc00640 1 -- 172.21.15.90:0/1985735605 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b300a5c90 con 0x7f1b30004cb0 2024-04-02T14:46:40.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.080+0000 7f1b3cc00640 1 --2- 172.21.15.90:0/1985735605 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1b30004cb0 0x7f1b300aa270 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f1b2c00b420 tx=0x7f1b2c00b8e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:46:40.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.080+0000 7f1b35600640 1 -- 172.21.15.90:0/1985735605 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b2c011830 con 0x7f1b30004cb0 2024-04-02T14:46:40.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.081+0000 7f1b35600640 1 -- 172.21.15.90:0/1985735605 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1b2c016070 con 0x7f1b30004cb0 2024-04-02T14:46:40.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.081+0000 7f1b3e000640 1 -- 172.21.15.90:0/1985735605 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b300a5f80 con 0x7f1b30004cb0 2024-04-02T14:46:40.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.081+0000 7f1b35600640 1 -- 172.21.15.90:0/1985735605 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b2c00f2e0 con 0x7f1b30004cb0 2024-04-02T14:46:40.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.081+0000 7f1b3e000640 1 -- 172.21.15.90:0/1985735605 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b300ae150 con 0x7f1b30004cb0 2024-04-02T14:46:40.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.086+0000 7f1b35600640 1 -- 172.21.15.90:0/1985735605 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f1b2c0119d0 con 0x7f1b30004cb0 2024-04-02T14:46:40.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.089+0000 7f1b423a5740 1 Processor -- start 2024-04-02T14:46:40.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.089+0000 7f1b423a5740 1 -- start start 2024-04-02T14:46:40.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.089+0000 7f1b423a5740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560cc0d83ba0 0x560cc0d828f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:40.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.090+0000 7f1b423a5740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560cc0d82ec0 0x560cc0d83340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:40.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.090+0000 7f1b423a5740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560cc0d8c230 0x560cc0d8e620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:40.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.090+0000 7f1b423a5740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x560cc0cc3400 con 0x560cc0d8c230 2024-04-02T14:46:40.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.090+0000 7f1b423a5740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x560cc0ca7f10 con 0x560cc0d83ba0 2024-04-02T14:46:40.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.090+0000 7f1b423a5740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x560cc0ce76f0 con 0x560cc0d82ec0 2024-04-02T14:46:40.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.090+0000 7f1b3d600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560cc0d83ba0 0x560cc0d828f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:40.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.090+0000 7f1b3d600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560cc0d83ba0 0x560cc0d828f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:47980/0 (socket says 172.21.15.90:47980) 2024-04-02T14:46:40.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.090+0000 7f1b3d600640 1 -- 172.21.15.90:0/638581771 learned_addr learned my addr 172.21.15.90:0/638581771 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:46:40.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.090+0000 7f1b3cc00640 1 --2- 172.21.15.90:0/638581771 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560cc0d82ec0 0x560cc0d83340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:40.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.090+0000 7f1b35600640 1 --2- 172.21.15.90:0/1985735605 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f1b200a3de0 0x7f1b200a62a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:40.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.091+0000 7f1b3d600640 1 --2- 172.21.15.90:0/1985735605 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f1b200a3de0 0x7f1b200a62a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:40.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.091+0000 7f1b3d600640 1 -- 172.21.15.90:0/638581771 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560cc0d82ec0 msgr2=0x560cc0d83340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:46:40.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.091+0000 7f1b3d600640 1 --2- 172.21.15.90:0/638581771 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560cc0d82ec0 0x560cc0d83340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:40.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.091+0000 7f1b3d600640 1 -- 172.21.15.90:0/638581771 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560cc0d8c230 msgr2=0x560cc0d8e620 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-02T14:46:40.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.091+0000 7f1b35600640 1 -- 172.21.15.90:0/1985735605 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 8550+0+0 (secure 0 0 0) 0x7f1b2c0cfcf0 con 0x7f1b30004cb0 2024-04-02T14:46:40.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.091+0000 7f1b3d600640 1 --2- 172.21.15.90:0/638581771 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560cc0d8c230 0x560cc0d8e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:40.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.091+0000 7f1b3d600640 1 -- 172.21.15.90:0/638581771 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560cc09fb3f0 con 0x560cc0d83ba0 2024-04-02T14:46:40.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.093+0000 7f1b3d600640 1 --2- 172.21.15.90:0/638581771 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560cc0d83ba0 0x560cc0d828f0 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f1b38050610 tx=0x7f1b3809bce0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d8f4016393417eb server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:46:40.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.093+0000 7f1b29c00640 1 -- 172.21.15.90:0/638581771 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b380b7070 con 0x560cc0d83ba0 2024-04-02T14:46:40.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.093+0000 7f1b29c00640 1 -- 172.21.15.90:0/638581771 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1b3809b840 con 0x560cc0d83ba0 2024-04-02T14:46:40.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.093+0000 7f1b29c00640 1 -- 172.21.15.90:0/638581771 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b380b1c40 con 0x560cc0d83ba0 2024-04-02T14:46:40.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.096+0000 7f1b423a5740 1 -- 172.21.15.90:0/638581771 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560cc0d83ba0 msgr2=0x560cc0d828f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:40.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.096+0000 7f1b3d600640 1 --2- 172.21.15.90:0/1985735605 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f1b200a3de0 0x7f1b200a62a0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f1b380b0000 tx=0x7f1b380ae000 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:46:40.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.096+0000 7f1b423a5740 1 --2- 172.21.15.90:0/638581771 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560cc0d83ba0 0x560cc0d828f0 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f1b38050610 tx=0x7f1b3809bce0 comp rx=0 tx=0).stop 2024-04-02T14:46:40.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.096+0000 7f1b423a5740 1 -- 172.21.15.90:0/638581771 shutdown_connections 2024-04-02T14:46:40.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.096+0000 7f1b423a5740 1 --2- 172.21.15.90:0/638581771 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560cc0d8c230 0x560cc0d8e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:40.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.096+0000 7f1b423a5740 1 --2- 172.21.15.90:0/638581771 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560cc0d82ec0 0x560cc0d83340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:40.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.096+0000 7f1b423a5740 1 --2- 172.21.15.90:0/638581771 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560cc0d83ba0 0x560cc0d828f0 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:40.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.096+0000 7f1b423a5740 1 -- 172.21.15.90:0/638581771 >> 172.21.15.90:0/638581771 conn(0x560cc0d49a50 msgr2=0x560cc0d7d910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:46:40.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.098+0000 7f1b423a5740 1 -- 172.21.15.90:0/638581771 shutdown_connections 2024-04-02T14:46:40.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.098+0000 7f1b423a5740 1 -- 172.21.15.90:0/638581771 wait complete. 2024-04-02T14:46:40.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.099+0000 7f1b423a5740 1 -- start start 2024-04-02T14:46:40.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.099+0000 7f1b423a5740 1 Processor -- start 2024-04-02T14:46:40.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.100+0000 7f1b423a5740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560cc0d82ec0 0x560cc0d891a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:40.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.100+0000 7f1b423a5740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560cc0d83ba0 0x560cc0d896e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:40.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.100+0000 7f1b423a5740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560cc0d8c230 0x560cc0d89c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:40.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.100+0000 7f1b423a5740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x560cc0ce36a0 con 0x560cc0d83ba0 2024-04-02T14:46:40.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.100+0000 7f1b423a5740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x560cc0c8f980 con 0x560cc0d82ec0 2024-04-02T14:46:40.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.100+0000 7f1b423a5740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x560cc0ca2cb0 con 0x560cc0d8c230 2024-04-02T14:46:40.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.100+0000 7f1b423a5740 10 client.0 _ll_register_callbacks cb 0x560cc0ca26a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:46:40.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.100+0000 7f1b3d600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560cc0d8c230 0x560cc0d89c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:40.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.100+0000 7f1b3d600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560cc0d8c230 0x560cc0d89c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:48390/0 (socket says 172.21.15.90:48390) 2024-04-02T14:46:40.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.100+0000 7f1b3d600640 1 -- 172.21.15.90:0/253411018 learned_addr learned my addr 172.21.15.90:0/253411018 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:46:40.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.101+0000 7f1b37e00640 1 --2- 172.21.15.90:0/253411018 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560cc0d83ba0 0x560cc0d896e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:40.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.101+0000 7f1b3cc00640 1 --2- 172.21.15.90:0/253411018 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560cc0d82ec0 0x560cc0d891a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:40.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.101+0000 7f1b3cc00640 1 -- 172.21.15.90:0/253411018 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560cc0d8c230 msgr2=0x560cc0d89c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:46:40.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.101+0000 7f1b3cc00640 1 --2- 172.21.15.90:0/253411018 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560cc0d8c230 0x560cc0d89c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:40.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.101+0000 7f1b3cc00640 1 -- 172.21.15.90:0/253411018 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560cc0d83ba0 msgr2=0x560cc0d896e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:46:40.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.101+0000 7f1b3cc00640 1 --2- 172.21.15.90:0/253411018 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560cc0d83ba0 0x560cc0d896e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:40.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.101+0000 7f1b3cc00640 1 -- 172.21.15.90:0/253411018 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560cc0ac57d0 con 0x560cc0d82ec0 2024-04-02T14:46:40.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.101+0000 7f1b37e00640 1 --2- 172.21.15.90:0/253411018 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560cc0d83ba0 0x560cc0d896e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:46:40.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.102+0000 7f1b3cc00640 1 --2- 172.21.15.90:0/253411018 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560cc0d82ec0 0x560cc0d891a0 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f1b2c0cd530 tx=0x7f1b2c0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=63a7aea99cdae7f6 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:46:40.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.103+0000 7f1b1e000640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:46:40.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.103+0000 7f1b1e000640 1 -- 172.21.15.90:0/253411018 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b2c0a0130 con 0x560cc0d82ec0 2024-04-02T14:46:40.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.103+0000 7f1b423a5740 1 -- 172.21.15.90:0/253411018 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1b3809b840 con 0x560cc0d82ec0 2024-04-02T14:46:40.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.105+0000 7f1b1e000640 1 -- 172.21.15.90:0/253411018 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 8550+0+0 (secure 0 0 0) 0x7f1b2c0cdb70 con 0x560cc0d82ec0 2024-04-02T14:46:40.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.105+0000 7f1b1e000640 1 -- 172.21.15.90:0/253411018 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f1b2c0b52d0 con 0x560cc0d82ec0 2024-04-02T14:46:40.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.105+0000 7f1b423a5740 10 client.29096 fetch_fsmap learned FSMap version 27 2024-04-02T14:46:40.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.105+0000 7f1b423a5740 1 -- 172.21.15.90:0/253411018 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x560cc0d9b120 con 0x560cc0d82ec0 2024-04-02T14:46:40.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.106+0000 7f1b1e000640 1 -- 172.21.15.90:0/253411018 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1b3809b840 con 0x560cc0d82ec0 2024-04-02T14:46:40.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.107+0000 7f1b1e000640 1 -- 172.21.15.90:0/253411018 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f1b2c09b1d0 con 0x560cc0d82ec0 2024-04-02T14:46:40.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.107+0000 7f1b423a5740 10 client.29096 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:46:40.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.107+0000 7f1b423a5740 10 client.29096 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.c 2024-04-02T14:46:40.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.107+0000 7f1b423a5740 20 client.29096 populate_metadata read hostname 'smithi090' 2024-04-02T14:46:40.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.107+0000 7f1b423a5740 1 --2- 172.21.15.90:0/253411018 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x560cc0d9c590 0x560cc0d9ea10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:40.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.107+0000 7f1b423a5740 4 client.29096 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:46:40.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.107+0000 7f1b423a5740 1 -- 172.21.15.90:0/253411018 --> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560cc0ca7040 con 0x560cc0d9c590 2024-04-02T14:46:40.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.108+0000 7f1b37e00640 1 --2- 172.21.15.90:0/253411018 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x560cc0d9c590 0x560cc0d9ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:40.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.112+0000 7f1b37e00640 1 --2- 172.21.15.90:0/253411018 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x560cc0d9c590 0x560cc0d9ea10 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=4fcfe259f2ecd47b server_cookie=b2e8ca54d6d6c047 in_seq=0 out_seq=0 2024-04-02T14:46:40.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.112+0000 7f1b1e000640 10 client.29096 ms_handle_connect on v2:172.21.15.177:6836/4223414858 2024-04-02T14:46:40.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.113+0000 7f1b1e000640 1 -- 172.21.15.90:0/253411018 <== mds.4 v2:172.21.15.177:6836/4223414858 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x560cc0ca7040 con 0x560cc0d9c590 2024-04-02T14:46:40.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.113+0000 7f1b1e000640 10 client.29096 handle_command_reply: tid=0 2024-04-02T14:46:40.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.114+0000 7f1b423a5740 1 client.29096 shutdown 2024-04-02T14:46:40.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.114+0000 7f1b423a5740 1 -- 172.21.15.90:0/253411018 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560cc0d82ec0 msgr2=0x560cc0d891a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:40.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.114+0000 7f1b423a5740 1 --2- 172.21.15.90:0/253411018 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560cc0d82ec0 0x560cc0d891a0 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f1b2c0cd530 tx=0x7f1b2c0046c0 comp rx=0 tx=0).stop 2024-04-02T14:46:40.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.114+0000 7f1b423a5740 1 -- 172.21.15.90:0/253411018 shutdown_connections 2024-04-02T14:46:40.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.114+0000 7f1b423a5740 1 --2- 172.21.15.90:0/253411018 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560cc0d8c230 0x560cc0d89c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:40.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.114+0000 7f1b423a5740 1 --2- 172.21.15.90:0/253411018 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x560cc0d9c590 0x560cc0d9ea10 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:40.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.114+0000 7f1b423a5740 1 --2- 172.21.15.90:0/253411018 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560cc0d83ba0 0x560cc0d896e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:40.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.114+0000 7f1b423a5740 1 --2- 172.21.15.90:0/253411018 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560cc0d82ec0 0x560cc0d891a0 secure :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f1b2c0cd530 tx=0x7f1b2c0046c0 comp rx=0 tx=0).stop 2024-04-02T14:46:40.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.114+0000 7f1b423a5740 1 -- 172.21.15.90:0/253411018 >> 172.21.15.90:0/253411018 conn(0x560cc0d49a50 msgr2=0x560cc0d7f660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:46:40.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.114+0000 7f1b1e000640 0 client.29096 ms_handle_reset on v2:172.21.15.177:6836/4223414858 2024-04-02T14:46:40.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.115+0000 7f1b423a5740 1 -- 172.21.15.90:0/253411018 shutdown_connections 2024-04-02T14:46:40.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.115+0000 7f1b423a5740 1 -- 172.21.15.90:0/253411018 wait complete. 2024-04-02T14:46:40.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.115+0000 7f1b423a5740 20 client.29096 trim_cache size 0 max 16384 2024-04-02T14:46:40.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.137+0000 7f1b423a5740 1 Processor -- start 2024-04-02T14:46:40.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.137+0000 7f1b423a5740 1 -- start start 2024-04-02T14:46:40.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.137+0000 7f1b423a5740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560cc0d83ba0 0x560cc0d8b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:40.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.137+0000 7f1b423a5740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560cc0d8c230 0x560cc0d97cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:40.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.137+0000 7f1b423a5740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560cc0d7c970 0x560cc0da4320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:40.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.137+0000 7f1b423a5740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f1b1800e810 con 0x560cc0d8c230 2024-04-02T14:46:40.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.137+0000 7f1b423a5740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f1b18010350 con 0x560cc0d7c970 2024-04-02T14:46:40.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.137+0000 7f1b423a5740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f1b1800eee0 con 0x560cc0d83ba0 2024-04-02T14:46:40.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.138+0000 7f1b3cc00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560cc0d83ba0 0x560cc0d8b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:40.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.138+0000 7f1b3d600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560cc0d7c970 0x560cc0da4320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:40.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.138+0000 7f1b3cc00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560cc0d83ba0 0x560cc0d8b8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:48402/0 (socket says 172.21.15.90:48402) 2024-04-02T14:46:40.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.138+0000 7f1b3cc00640 1 -- 172.21.15.90:0/2417282233 learned_addr learned my addr 172.21.15.90:0/2417282233 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:46:40.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.138+0000 7f1b3d600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560cc0d7c970 0x560cc0da4320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:47992/0 (socket says 172.21.15.90:47992) 2024-04-02T14:46:40.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.138+0000 7f1b37e00640 1 --2- 172.21.15.90:0/2417282233 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560cc0d8c230 0x560cc0d97cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:40.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.139+0000 7f1b3d600640 1 -- 172.21.15.90:0/2417282233 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560cc0d83ba0 msgr2=0x560cc0d8b8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:46:40.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.139+0000 7f1b3d600640 1 --2- 172.21.15.90:0/2417282233 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560cc0d83ba0 0x560cc0d8b8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:40.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.139+0000 7f1b3d600640 1 -- 172.21.15.90:0/2417282233 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560cc0d8c230 msgr2=0x560cc0d97cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:46:40.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.139+0000 7f1b3d600640 1 --2- 172.21.15.90:0/2417282233 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560cc0d8c230 0x560cc0d97cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:40.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.139+0000 7f1b3d600640 1 -- 172.21.15.90:0/2417282233 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560cc0d9f090 con 0x560cc0d7c970 2024-04-02T14:46:40.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.139+0000 7f1b37e00640 1 --2- 172.21.15.90:0/2417282233 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560cc0d8c230 0x560cc0d97cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:46:40.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.139+0000 7f1b3d600640 1 --2- 172.21.15.90:0/2417282233 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560cc0d7c970 0x560cc0da4320 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f1b3804fb30 tx=0x7f1b380a6000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b0bda4a8a21694b0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:46:40.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.139+0000 7f1b29200640 1 -- 172.21.15.90:0/2417282233 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b380b7040 con 0x560cc0d7c970 2024-04-02T14:46:40.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.139+0000 7f1b29200640 1 -- 172.21.15.90:0/2417282233 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1b380b1460 con 0x560cc0d7c970 2024-04-02T14:46:40.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.139+0000 7f1b29200640 1 -- 172.21.15.90:0/2417282233 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b380a86b0 con 0x560cc0d7c970 2024-04-02T14:46:40.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.140+0000 7f1b423a5740 1 -- 172.21.15.90:0/2417282233 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560cc0d7c970 msgr2=0x560cc0da4320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:40.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.141+0000 7f1b423a5740 1 --2- 172.21.15.90:0/2417282233 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560cc0d7c970 0x560cc0da4320 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f1b3804fb30 tx=0x7f1b380a6000 comp rx=0 tx=0).stop 2024-04-02T14:46:40.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.141+0000 7f1b423a5740 1 -- 172.21.15.90:0/2417282233 shutdown_connections 2024-04-02T14:46:40.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.141+0000 7f1b423a5740 1 --2- 172.21.15.90:0/2417282233 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560cc0d7c970 0x560cc0da4320 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:40.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.141+0000 7f1b423a5740 1 --2- 172.21.15.90:0/2417282233 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560cc0d8c230 0x560cc0d97cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:40.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.141+0000 7f1b423a5740 1 --2- 172.21.15.90:0/2417282233 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560cc0d83ba0 0x560cc0d8b8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:40.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.141+0000 7f1b423a5740 1 -- 172.21.15.90:0/2417282233 >> 172.21.15.90:0/2417282233 conn(0x560cc0d49a50 msgr2=0x560cc0d7c260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:46:40.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.141+0000 7f1b423a5740 1 -- 172.21.15.90:0/2417282233 shutdown_connections 2024-04-02T14:46:40.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.141+0000 7f1b423a5740 1 -- 172.21.15.90:0/2417282233 wait complete. 2024-04-02T14:46:40.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.142+0000 7f1b423a5740 1 -- start start 2024-04-02T14:46:40.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.142+0000 7f1b423a5740 1 Processor -- start 2024-04-02T14:46:40.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.143+0000 7f1b423a5740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560cc0d7c970 0x560cc0d941f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:40.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.143+0000 7f1b423a5740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560cc0d83ba0 0x560cc0d94730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:40.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.143+0000 7f1b423a5740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560cc0d8c230 0x560cc0d90770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:40.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.143+0000 7f1b423a5740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f1b1800fc80 con 0x560cc0d8c230 2024-04-02T14:46:40.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.143+0000 7f1b423a5740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f1b1800f5b0 con 0x560cc0d83ba0 2024-04-02T14:46:40.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.143+0000 7f1b423a5740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f1b1800c810 con 0x560cc0d7c970 2024-04-02T14:46:40.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.143+0000 7f1b423a5740 10 client.0 _ll_register_callbacks cb 0x7f1b2c0108a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:46:40.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.143+0000 7f1b37e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560cc0d83ba0 0x560cc0d94730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:40.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.143+0000 7f1b37e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560cc0d83ba0 0x560cc0d94730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:48008/0 (socket says 172.21.15.90:48008) 2024-04-02T14:46:40.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.143+0000 7f1b37e00640 1 -- 172.21.15.90:0/2605020356 learned_addr learned my addr 172.21.15.90:0/2605020356 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:46:40.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.143+0000 7f1b3cc00640 1 --2- 172.21.15.90:0/2605020356 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560cc0d7c970 0x560cc0d941f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:40.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.144+0000 7f1b37e00640 1 -- 172.21.15.90:0/2605020356 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560cc0d7c970 msgr2=0x560cc0d941f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:46:40.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.144+0000 7f1b37e00640 1 --2- 172.21.15.90:0/2605020356 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560cc0d7c970 0x560cc0d941f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:40.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.144+0000 7f1b3d600640 1 --2- 172.21.15.90:0/2605020356 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560cc0d8c230 0x560cc0d90770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:40.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.144+0000 7f1b37e00640 1 -- 172.21.15.90:0/2605020356 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560cc0d8c230 msgr2=0x560cc0d90770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:46:40.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.144+0000 7f1b37e00640 1 --2- 172.21.15.90:0/2605020356 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560cc0d8c230 0x560cc0d90770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:40.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.144+0000 7f1b37e00640 1 -- 172.21.15.90:0/2605020356 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560cc0d90f70 con 0x560cc0d83ba0 2024-04-02T14:46:40.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.144+0000 7f1b3d600640 1 --2- 172.21.15.90:0/2605020356 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560cc0d8c230 0x560cc0d90770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:46:40.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.144+0000 7f1b37e00640 1 --2- 172.21.15.90:0/2605020356 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560cc0d83ba0 0x560cc0d94730 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f1b24005dc0 tx=0x7f1b24011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=129a0f25b64927b8 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:46:40.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.144+0000 7f1b1e000640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:46:40.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.144+0000 7f1b1e000640 1 -- 172.21.15.90:0/2605020356 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b24019bc0 con 0x560cc0d83ba0 2024-04-02T14:46:40.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.145+0000 7f1b1e000640 1 -- 172.21.15.90:0/2605020356 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 8550+0+0 (secure 0 0 0) 0x7f1b24019d60 con 0x560cc0d83ba0 2024-04-02T14:46:40.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.145+0000 7f1b423a5740 1 -- 172.21.15.90:0/2605020356 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1b380b1460 con 0x560cc0d83ba0 2024-04-02T14:46:40.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.145+0000 7f1b1e000640 1 -- 172.21.15.90:0/2605020356 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f1b24026050 con 0x560cc0d83ba0 2024-04-02T14:46:40.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.145+0000 7f1b423a5740 10 client.29102 fetch_fsmap learned FSMap version 27 2024-04-02T14:46:40.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.145+0000 7f1b423a5740 1 -- 172.21.15.90:0/2605020356 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x560cc0e037f0 con 0x560cc0d83ba0 2024-04-02T14:46:40.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.146+0000 7f1b1e000640 1 -- 172.21.15.90:0/2605020356 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1b380b1460 con 0x560cc0d83ba0 2024-04-02T14:46:40.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.146+0000 7f1b1e000640 1 -- 172.21.15.90:0/2605020356 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f1b240047a0 con 0x560cc0d83ba0 2024-04-02T14:46:40.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.146+0000 7f1b423a5740 10 client.29102 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:46:40.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.146+0000 7f1b423a5740 10 client.29102 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.c 2024-04-02T14:46:40.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.146+0000 7f1b423a5740 20 client.29102 populate_metadata read hostname 'smithi090' 2024-04-02T14:46:40.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.146+0000 7f1b423a5740 1 --2- 172.21.15.90:0/2605020356 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x560cc0e03ff0 0x560cc0e06410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:40.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.146+0000 7f1b423a5740 4 client.29102 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-02T14:46:40.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.146+0000 7f1b423a5740 1 -- 172.21.15.90:0/2605020356 --> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] -- command(tid 0: {"prefix": "damage ls"}) -- 0x560cc0cf4c30 con 0x560cc0e03ff0 2024-04-02T14:46:40.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.146+0000 7f1b3cc00640 1 --2- 172.21.15.90:0/2605020356 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x560cc0e03ff0 0x560cc0e06410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:40.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.147+0000 7f1b1e000640 10 client.29102 ms_handle_connect on v2:172.21.15.177:6836/4223414858 2024-04-02T14:46:40.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.147+0000 7f1b3cc00640 1 --2- 172.21.15.90:0/2605020356 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x560cc0e03ff0 0x560cc0e06410 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=76ea497dc3567aa5 server_cookie=4136894bc2069bd0 in_seq=0 out_seq=0 2024-04-02T14:46:40.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.147+0000 7f1b1e000640 1 -- 172.21.15.90:0/2605020356 <== mds.4 v2:172.21.15.177:6836/4223414858 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x560cc0cf4c30 con 0x560cc0e03ff0 2024-04-02T14:46:40.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.147+0000 7f1b1e000640 10 client.29102 handle_command_reply: tid=0 2024-04-02T14:46:40.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.147+0000 7f1b423a5740 1 client.29102 shutdown 2024-04-02T14:46:40.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.148+0000 7f1b423a5740 1 -- 172.21.15.90:0/2605020356 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560cc0d83ba0 msgr2=0x560cc0d94730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:40.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.148+0000 7f1b423a5740 1 --2- 172.21.15.90:0/2605020356 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560cc0d83ba0 0x560cc0d94730 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f1b24005dc0 tx=0x7f1b24011810 comp rx=0 tx=0).stop 2024-04-02T14:46:40.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.148+0000 7f1b423a5740 1 -- 172.21.15.90:0/2605020356 shutdown_connections 2024-04-02T14:46:40.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.148+0000 7f1b423a5740 1 --2- 172.21.15.90:0/2605020356 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x560cc0e03ff0 0x560cc0e06410 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:40.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.148+0000 7f1b423a5740 1 --2- 172.21.15.90:0/2605020356 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560cc0d8c230 0x560cc0d90770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:40.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.148+0000 7f1b423a5740 1 --2- 172.21.15.90:0/2605020356 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560cc0d83ba0 0x560cc0d94730 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:40.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.148+0000 7f1b423a5740 1 --2- 172.21.15.90:0/2605020356 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560cc0d7c970 0x560cc0d941f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:40.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.148+0000 7f1b423a5740 1 -- 172.21.15.90:0/2605020356 >> 172.21.15.90:0/2605020356 conn(0x560cc0d49a50 msgr2=0x560cc0d89140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:46:40.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.148+0000 7f1b1e000640 0 client.29102 ms_handle_reset on v2:172.21.15.177:6836/4223414858 2024-04-02T14:46:40.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.148+0000 7f1b423a5740 1 -- 172.21.15.90:0/2605020356 shutdown_connections 2024-04-02T14:46:40.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.148+0000 7f1b423a5740 1 -- 172.21.15.90:0/2605020356 wait complete. 2024-04-02T14:46:40.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.148+0000 7f1b423a5740 20 client.29102 trim_cache size 0 max 16384 2024-04-02T14:46:40.146 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-02T14:46:40.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.150+0000 7f1b3e000640 1 -- 172.21.15.90:0/1985735605 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f1b200a3de0 msgr2=0x7f1b200a62a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:40.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.150+0000 7f1b3e000640 1 --2- 172.21.15.90:0/1985735605 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f1b200a3de0 0x7f1b200a62a0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f1b380b0000 tx=0x7f1b380ae000 comp rx=0 tx=0).stop 2024-04-02T14:46:40.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.150+0000 7f1b3e000640 1 -- 172.21.15.90:0/1985735605 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1b30004cb0 msgr2=0x7f1b300aa270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:40.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.150+0000 7f1b3e000640 1 --2- 172.21.15.90:0/1985735605 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1b30004cb0 0x7f1b300aa270 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f1b2c00b420 tx=0x7f1b2c00b8e0 comp rx=0 tx=0).stop 2024-04-02T14:46:40.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.151+0000 7f1b3e000640 1 -- 172.21.15.90:0/1985735605 shutdown_connections 2024-04-02T14:46:40.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.151+0000 7f1b3e000640 1 --2- 172.21.15.90:0/1985735605 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f1b200a3de0 0x7f1b200a62a0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:40.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.151+0000 7f1b3e000640 1 --2- 172.21.15.90:0/1985735605 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1b3009be10 0x7f1b300a53d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:40.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.151+0000 7f1b3e000640 1 --2- 172.21.15.90:0/1985735605 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1b30005670 0x7f1b300aa7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:40.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.151+0000 7f1b3e000640 1 --2- 172.21.15.90:0/1985735605 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1b30004cb0 0x7f1b300aa270 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:40.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.151+0000 7f1b3e000640 1 -- 172.21.15.90:0/1985735605 >> 172.21.15.90:0/1985735605 conn(0x7f1b3000a740 msgr2=0x7f1b30093b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:46:40.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.151+0000 7f1b3e000640 1 -- 172.21.15.90:0/1985735605 shutdown_connections 2024-04-02T14:46:40.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.152+0000 7f1b3e000640 1 -- 172.21.15.90:0/1985735605 wait complete. 2024-04-02T14:46:40.179 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-04-02T14:46:40.432 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:46:40 smithi090 ceph-mon[31674]: pgmap v366: 97 pgs: 97 active+clean; 4.3 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 8.0 MiB/s wr, 780 op/s 2024-04-02T14:46:40.432 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:46:40 smithi090 ceph-mon[31674]: from='client.? 172.21.15.90:0/3981764374' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-02T14:46:40.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.481+0000 7f156e400640 1 -- 172.21.15.90:0/1906364826 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f15680747f0 msgr2=0x7f156810ddf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:40.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.481+0000 7f156e400640 1 --2- 172.21.15.90:0/1906364826 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f15680747f0 0x7f156810ddf0 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f1554009900 tx=0x7f155402bcd0 comp rx=0 tx=0).stop 2024-04-02T14:46:40.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.482+0000 7f156e400640 1 -- 172.21.15.90:0/1906364826 shutdown_connections 2024-04-02T14:46:40.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.482+0000 7f156e400640 1 --2- 172.21.15.90:0/1906364826 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f156810e330 0x7f1568110740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:40.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.482+0000 7f156e400640 1 --2- 172.21.15.90:0/1906364826 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f15680747f0 0x7f156810ddf0 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:40.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.482+0000 7f156e400640 1 --2- 172.21.15.90:0/1906364826 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1568073ed0 0x7f15680742b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:40.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.482+0000 7f156e400640 1 -- 172.21.15.90:0/1906364826 >> 172.21.15.90:0/1906364826 conn(0x7f1568078f00 msgr2=0x7f1568079310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:46:40.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.483+0000 7f156e400640 1 -- 172.21.15.90:0/1906364826 shutdown_connections 2024-04-02T14:46:40.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.483+0000 7f156e400640 1 -- 172.21.15.90:0/1906364826 wait complete. 2024-04-02T14:46:40.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.485+0000 7f156e400640 1 Processor -- start 2024-04-02T14:46:40.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.485+0000 7f156e400640 1 -- start start 2024-04-02T14:46:40.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.485+0000 7f156e400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1568073ed0 0x7f156810da70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:40.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.485+0000 7f156e400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f15680747f0 0x7f1568104a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:40.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.485+0000 7f156e400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f156810e330 0x7f1568104fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:40.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.486+0000 7f156e400640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f1568103640 con 0x7f1568073ed0 2024-04-02T14:46:40.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.486+0000 7f156e400640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f15681034c0 con 0x7f15680747f0 2024-04-02T14:46:40.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.486+0000 7f156e400640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f15681037c0 con 0x7f156810e330 2024-04-02T14:46:40.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.486+0000 7f1567e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f156810e330 0x7f1568104fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:40.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.486+0000 7f1566a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f15680747f0 0x7f1568104a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:40.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.486+0000 7f1566a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f15680747f0 0x7f1568104a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:48022/0 (socket says 172.21.15.90:48022) 2024-04-02T14:46:40.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.486+0000 7f1566a00640 1 -- 172.21.15.90:0/4278934071 learned_addr learned my addr 172.21.15.90:0/4278934071 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:46:40.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.486+0000 7f1567400640 1 --2- 172.21.15.90:0/4278934071 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1568073ed0 0x7f156810da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:40.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.487+0000 7f1566a00640 1 -- 172.21.15.90:0/4278934071 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f156810e330 msgr2=0x7f1568104fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:40.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.487+0000 7f1566a00640 1 --2- 172.21.15.90:0/4278934071 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f156810e330 0x7f1568104fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:40.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.487+0000 7f1566a00640 1 -- 172.21.15.90:0/4278934071 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1568073ed0 msgr2=0x7f156810da70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:40.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.487+0000 7f1566a00640 1 --2- 172.21.15.90:0/4278934071 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1568073ed0 0x7f156810da70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:40.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.487+0000 7f1566a00640 1 -- 172.21.15.90:0/4278934071 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15681056e0 con 0x7f15680747f0 2024-04-02T14:46:40.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.487+0000 7f1567400640 1 --2- 172.21.15.90:0/4278934071 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1568073ed0 0x7f156810da70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:46:40.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.487+0000 7f1566a00640 1 --2- 172.21.15.90:0/4278934071 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f15680747f0 0x7f1568104a70 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f1554005940 tx=0x7f155402cd10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:46:40.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.488+0000 7f155fe00640 1 -- 172.21.15.90:0/4278934071 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1554041070 con 0x7f15680747f0 2024-04-02T14:46:40.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.488+0000 7f156e400640 1 -- 172.21.15.90:0/4278934071 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f15681094b0 con 0x7f15680747f0 2024-04-02T14:46:40.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.488+0000 7f155fe00640 1 -- 172.21.15.90:0/4278934071 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1554004450 con 0x7f15680747f0 2024-04-02T14:46:40.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.488+0000 7f155fe00640 1 -- 172.21.15.90:0/4278934071 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1554034440 con 0x7f15680747f0 2024-04-02T14:46:40.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.489+0000 7f156e400640 1 -- 172.21.15.90:0/4278934071 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15681099f0 con 0x7f15680747f0 2024-04-02T14:46:40.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.496+0000 7f155fe00640 1 -- 172.21.15.90:0/4278934071 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f1554044070 con 0x7f15680747f0 2024-04-02T14:46:40.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.499+0000 7f1571211740 1 Processor -- start 2024-04-02T14:46:40.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.499+0000 7f155fe00640 1 --2- 172.21.15.90:0/4278934071 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f154c0a3da0 0x7f154c0a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:40.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.499+0000 7f155fe00640 1 -- 172.21.15.90:0/4278934071 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 8550+0+0 (secure 0 0 0) 0x7f15540f23e0 con 0x7f15680747f0 2024-04-02T14:46:40.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.500+0000 7f1571211740 1 -- start start 2024-04-02T14:46:40.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.500+0000 7f1567e00640 1 --2- 172.21.15.90:0/4278934071 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f154c0a3da0 0x7f154c0a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:40.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.500+0000 7f1571211740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562ef61689f0 0x562ef6167740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:40.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.500+0000 7f1571211740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562ef6167d10 0x562ef6168190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:40.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.500+0000 7f1571211740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x562ef6171080 0x562ef6173470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:40.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.500+0000 7f1571211740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x562ef60a8400 con 0x562ef61689f0 2024-04-02T14:46:40.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.500+0000 7f1571211740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x562ef608cf10 con 0x562ef6171080 2024-04-02T14:46:40.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.500+0000 7f1571211740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x562ef60cc6f0 con 0x562ef6167d10 2024-04-02T14:46:40.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.501+0000 7f1566a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562ef6167d10 0x562ef6168190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:40.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.501+0000 7f1567400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562ef61689f0 0x562ef6167740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:40.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.501+0000 7f1567e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x562ef6171080 0x562ef6173470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:40.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.501+0000 7f1567400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562ef61689f0 0x562ef6167740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:40108/0 (socket says 172.21.15.90:40108) 2024-04-02T14:46:40.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.501+0000 7f1567400640 1 -- 172.21.15.90:0/1064800375 learned_addr learned my addr 172.21.15.90:0/1064800375 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:46:40.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.502+0000 7f1566a00640 1 -- 172.21.15.90:0/1064800375 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x562ef6171080 msgr2=0x562ef6173470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:46:40.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.502+0000 7f1566a00640 1 --2- 172.21.15.90:0/1064800375 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x562ef6171080 0x562ef6173470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:40.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.502+0000 7f1566a00640 1 -- 172.21.15.90:0/1064800375 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562ef61689f0 msgr2=0x562ef6167740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:46:40.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.502+0000 7f1566a00640 1 --2- 172.21.15.90:0/1064800375 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562ef61689f0 0x562ef6167740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:40.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.502+0000 7f1566a00640 1 -- 172.21.15.90:0/1064800375 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562ef5de03f0 con 0x562ef6167d10 2024-04-02T14:46:40.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.502+0000 7f1567400640 1 --2- 172.21.15.90:0/1064800375 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562ef61689f0 0x562ef6167740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:46:40.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.502+0000 7f1566a00640 1 --2- 172.21.15.90:0/1064800375 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562ef6167d10 0x562ef6168190 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f15540efbc0 tx=0x7f155402bcd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7b4ef4378bccc545 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:46:40.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.503+0000 7f155cc00640 1 -- 172.21.15.90:0/1064800375 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f155404b040 con 0x562ef6167d10 2024-04-02T14:46:40.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.503+0000 7f155cc00640 1 -- 172.21.15.90:0/1064800375 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f15540be410 con 0x562ef6167d10 2024-04-02T14:46:40.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.503+0000 7f155cc00640 1 -- 172.21.15.90:0/1064800375 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15540efd30 con 0x562ef6167d10 2024-04-02T14:46:40.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.504+0000 7f1567e00640 1 --2- 172.21.15.90:0/4278934071 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f154c0a3da0 0x7f154c0a6260 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f156806b3a0 tx=0x7f1560009a50 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:46:40.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.504+0000 7f1571211740 1 -- 172.21.15.90:0/1064800375 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562ef6167d10 msgr2=0x562ef6168190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:40.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.504+0000 7f1571211740 1 --2- 172.21.15.90:0/1064800375 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562ef6167d10 0x562ef6168190 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f15540efbc0 tx=0x7f155402bcd0 comp rx=0 tx=0).stop 2024-04-02T14:46:40.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.506+0000 7f1571211740 1 -- 172.21.15.90:0/1064800375 shutdown_connections 2024-04-02T14:46:40.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.506+0000 7f1571211740 1 --2- 172.21.15.90:0/1064800375 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x562ef6171080 0x562ef6173470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:40.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.506+0000 7f1571211740 1 --2- 172.21.15.90:0/1064800375 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562ef6167d10 0x562ef6168190 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:40.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.506+0000 7f1571211740 1 --2- 172.21.15.90:0/1064800375 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562ef61689f0 0x562ef6167740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:40.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.506+0000 7f1571211740 1 -- 172.21.15.90:0/1064800375 >> 172.21.15.90:0/1064800375 conn(0x562ef612ea50 msgr2=0x562ef6162760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:46:40.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.507+0000 7f1571211740 1 -- 172.21.15.90:0/1064800375 shutdown_connections 2024-04-02T14:46:40.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.507+0000 7f1571211740 1 -- 172.21.15.90:0/1064800375 wait complete. 2024-04-02T14:46:40.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.508+0000 7f1571211740 1 -- start start 2024-04-02T14:46:40.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.509+0000 7f1571211740 1 Processor -- start 2024-04-02T14:46:40.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.510+0000 7f1571211740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562ef6167d10 0x562ef616e0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:40.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.510+0000 7f1571211740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562ef61689f0 0x562ef616e630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:40.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.510+0000 7f1571211740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x562ef6171080 0x562ef616eb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:40.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.510+0000 7f1571211740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x562ef60c86a0 con 0x562ef61689f0 2024-04-02T14:46:40.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.510+0000 7f1571211740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x562ef6074980 con 0x562ef6171080 2024-04-02T14:46:40.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.510+0000 7f1571211740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x562ef6087cb0 con 0x562ef6167d10 2024-04-02T14:46:40.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.510+0000 7f1571211740 10 client.0 _ll_register_callbacks cb 0x562ef60876a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:46:40.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.510+0000 7f1566a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562ef61689f0 0x562ef616e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:40.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.510+0000 7f1566a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562ef61689f0 0x562ef616e630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:40110/0 (socket says 172.21.15.90:40110) 2024-04-02T14:46:40.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.510+0000 7f1566a00640 1 -- 172.21.15.90:0/2343307036 learned_addr learned my addr 172.21.15.90:0/2343307036 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:46:40.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.511+0000 7f1566a00640 1 -- 172.21.15.90:0/2343307036 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562ef6167d10 msgr2=0x562ef616e0f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-02T14:46:40.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.511+0000 7f1566a00640 1 --2- 172.21.15.90:0/2343307036 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562ef6167d10 0x562ef616e0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:40.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.511+0000 7f1566a00640 1 -- 172.21.15.90:0/2343307036 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x562ef6171080 msgr2=0x562ef616eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:46:40.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.511+0000 7f1567e00640 1 --2- 172.21.15.90:0/2343307036 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x562ef6171080 0x562ef616eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:40.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.511+0000 7f1566a00640 1 --2- 172.21.15.90:0/2343307036 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x562ef6171080 0x562ef616eb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:40.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.511+0000 7f1566a00640 1 -- 172.21.15.90:0/2343307036 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562ef5eaa7d0 con 0x562ef61689f0 2024-04-02T14:46:40.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.512+0000 7f1566a00640 1 --2- 172.21.15.90:0/2343307036 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562ef61689f0 0x562ef616e630 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f15540ba870 tx=0x7f15540efb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0f9c5c6f86bc9d7 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:46:40.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.512+0000 7f1551600640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-02T14:46:40.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.512+0000 7f1551600640 1 -- 172.21.15.90:0/2343307036 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15540d6020 con 0x562ef61689f0 2024-04-02T14:46:40.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.512+0000 7f1551600640 1 -- 172.21.15.90:0/2343307036 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 8550+0+0 (secure 0 0 0) 0x7f15540c6bf0 con 0x562ef61689f0 2024-04-02T14:46:40.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.512+0000 7f1571211740 1 -- 172.21.15.90:0/2343307036 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f15540be410 con 0x562ef61689f0 2024-04-02T14:46:40.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.512+0000 7f1551600640 1 -- 172.21.15.90:0/2343307036 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f155404b070 con 0x562ef61689f0 2024-04-02T14:46:40.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.512+0000 7f1571211740 10 client.18054 fetch_fsmap learned FSMap version 27 2024-04-02T14:46:40.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.512+0000 7f1571211740 1 -- 172.21.15.90:0/2343307036 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x562ef617fcb0 con 0x562ef61689f0 2024-04-02T14:46:40.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.513+0000 7f1551600640 1 -- 172.21.15.90:0/2343307036 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f15540be410 con 0x562ef61689f0 2024-04-02T14:46:40.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.513+0000 7f1551600640 1 -- 172.21.15.90:0/2343307036 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f15540bd2d0 con 0x562ef61689f0 2024-04-02T14:46:40.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.513+0000 7f1571211740 10 client.18054 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:46:40.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.513+0000 7f1571211740 10 client.18054 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-04-02T14:46:40.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.513+0000 7f1571211740 20 client.18054 populate_metadata read hostname 'smithi090' 2024-04-02T14:46:40.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.513+0000 7f1571211740 1 --2- 172.21.15.90:0/2343307036 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x562ef6181000 0x562ef61833f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:40.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.513+0000 7f1571211740 4 client.18054 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:46:40.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.513+0000 7f1571211740 1 -- 172.21.15.90:0/2343307036 --> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562ef608c040 con 0x562ef6181000 2024-04-02T14:46:40.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.515+0000 7f1567400640 1 --2- 172.21.15.90:0/2343307036 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x562ef6181000 0x562ef61833f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:40.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.517+0000 7f1567400640 1 --2- 172.21.15.90:0/2343307036 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x562ef6181000 0x562ef61833f0 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=9d44a6b9972bcca1 server_cookie=e0c9118d5e1df277 in_seq=0 out_seq=0 2024-04-02T14:46:40.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.517+0000 7f1551600640 10 client.18054 ms_handle_connect on v2:172.21.15.92:6836/1270635939 2024-04-02T14:46:40.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.518+0000 7f1551600640 1 -- 172.21.15.90:0/2343307036 <== mds.3 v2:172.21.15.92:6836/1270635939 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x562ef608c040 con 0x562ef6181000 2024-04-02T14:46:40.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.518+0000 7f1551600640 10 client.18054 handle_command_reply: tid=0 2024-04-02T14:46:40.514 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.518+0000 7f1571211740 1 client.18054 shutdown 2024-04-02T14:46:40.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.519+0000 7f1571211740 1 -- 172.21.15.90:0/2343307036 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562ef61689f0 msgr2=0x562ef616e630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:40.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.519+0000 7f1571211740 1 --2- 172.21.15.90:0/2343307036 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562ef61689f0 0x562ef616e630 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f15540ba870 tx=0x7f15540efb30 comp rx=0 tx=0).stop 2024-04-02T14:46:40.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.519+0000 7f1571211740 1 -- 172.21.15.90:0/2343307036 shutdown_connections 2024-04-02T14:46:40.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.519+0000 7f1571211740 1 --2- 172.21.15.90:0/2343307036 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x562ef6181000 0x562ef61833f0 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:40.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.519+0000 7f1571211740 1 --2- 172.21.15.90:0/2343307036 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x562ef6171080 0x562ef616eb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:40.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.519+0000 7f1571211740 1 --2- 172.21.15.90:0/2343307036 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562ef61689f0 0x562ef616e630 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:40.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.519+0000 7f1571211740 1 --2- 172.21.15.90:0/2343307036 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562ef6167d10 0x562ef616e0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:40.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.519+0000 7f1551600640 0 client.18054 ms_handle_reset on v2:172.21.15.92:6836/1270635939 2024-04-02T14:46:40.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.519+0000 7f1571211740 1 -- 172.21.15.90:0/2343307036 >> 172.21.15.90:0/2343307036 conn(0x562ef612ea50 msgr2=0x562ef61644b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:46:40.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.521+0000 7f1571211740 1 -- 172.21.15.90:0/2343307036 shutdown_connections 2024-04-02T14:46:40.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.521+0000 7f1571211740 1 -- 172.21.15.90:0/2343307036 wait complete. 2024-04-02T14:46:40.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.521+0000 7f1571211740 20 client.18054 trim_cache size 0 max 16384 2024-04-02T14:46:40.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.545+0000 7f1571211740 1 Processor -- start 2024-04-02T14:46:40.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.545+0000 7f1571211740 1 -- start start 2024-04-02T14:46:40.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.545+0000 7f1571211740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562ef6167d10 0x562ef6188c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:40.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.545+0000 7f1571211740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x562ef61689f0 0x562ef6189160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:40.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.545+0000 7f1571211740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562ef6171080 0x562ef6161010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:40.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.545+0000 7f1571211740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f15681bc6e0 con 0x562ef6167d10 2024-04-02T14:46:40.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.545+0000 7f1571211740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f15681be220 con 0x562ef61689f0 2024-04-02T14:46:40.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.545+0000 7f1571211740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f15681bcdb0 con 0x562ef6171080 2024-04-02T14:46:40.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.545+0000 7f1567e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562ef6171080 0x562ef6161010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:40.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.546+0000 7f1567e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562ef6171080 0x562ef6161010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:48452/0 (socket says 172.21.15.90:48452) 2024-04-02T14:46:40.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.546+0000 7f1567e00640 1 -- 172.21.15.90:0/3065442042 learned_addr learned my addr 172.21.15.90:0/3065442042 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:46:40.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.546+0000 7f1567400640 1 --2- 172.21.15.90:0/3065442042 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562ef6167d10 0x562ef6188c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:40.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.546+0000 7f1566a00640 1 --2- 172.21.15.90:0/3065442042 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x562ef61689f0 0x562ef6189160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:40.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.546+0000 7f1567e00640 1 -- 172.21.15.90:0/3065442042 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x562ef61689f0 msgr2=0x562ef6189160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:46:40.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.546+0000 7f1567e00640 1 --2- 172.21.15.90:0/3065442042 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x562ef61689f0 0x562ef6189160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:40.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.546+0000 7f1567e00640 1 -- 172.21.15.90:0/3065442042 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562ef6167d10 msgr2=0x562ef6188c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:46:40.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.546+0000 7f1567e00640 1 --2- 172.21.15.90:0/3065442042 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562ef6167d10 0x562ef6188c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:40.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.546+0000 7f1567e00640 1 -- 172.21.15.90:0/3065442042 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562ef6166d40 con 0x562ef6171080 2024-04-02T14:46:40.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.546+0000 7f1567400640 1 --2- 172.21.15.90:0/3065442042 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562ef6167d10 0x562ef6188c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:46:40.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.546+0000 7f1567e00640 1 --2- 172.21.15.90:0/3065442042 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562ef6171080 0x562ef6161010 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f1560015f90 tx=0x7f1560004ab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5f3fd47a322f8336 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:46:40.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.547+0000 7f1553e00640 1 -- 172.21.15.90:0/3065442042 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15600150f0 con 0x562ef6171080 2024-04-02T14:46:40.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.547+0000 7f1553e00640 1 -- 172.21.15.90:0/3065442042 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1560003e50 con 0x562ef6171080 2024-04-02T14:46:40.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.547+0000 7f1553e00640 1 -- 172.21.15.90:0/3065442042 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1560004020 con 0x562ef6171080 2024-04-02T14:46:40.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.548+0000 7f1571211740 1 -- 172.21.15.90:0/3065442042 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562ef6171080 msgr2=0x562ef6161010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:40.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.548+0000 7f1571211740 1 --2- 172.21.15.90:0/3065442042 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562ef6171080 0x562ef6161010 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f1560015f90 tx=0x7f1560004ab0 comp rx=0 tx=0).stop 2024-04-02T14:46:40.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.548+0000 7f1571211740 1 -- 172.21.15.90:0/3065442042 shutdown_connections 2024-04-02T14:46:40.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.548+0000 7f1571211740 1 --2- 172.21.15.90:0/3065442042 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562ef6171080 0x562ef6161010 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:40.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.548+0000 7f1571211740 1 --2- 172.21.15.90:0/3065442042 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x562ef61689f0 0x562ef6189160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:40.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.549+0000 7f1571211740 1 --2- 172.21.15.90:0/3065442042 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562ef6167d10 0x562ef6188c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:40.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.549+0000 7f1571211740 1 -- 172.21.15.90:0/3065442042 >> 172.21.15.90:0/3065442042 conn(0x562ef612ea50 msgr2=0x562ef6162290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:46:40.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.549+0000 7f1571211740 1 -- 172.21.15.90:0/3065442042 shutdown_connections 2024-04-02T14:46:40.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.549+0000 7f1571211740 1 -- 172.21.15.90:0/3065442042 wait complete. 2024-04-02T14:46:40.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.549+0000 7f1571211740 1 -- start start 2024-04-02T14:46:40.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.550+0000 7f1571211740 1 Processor -- start 2024-04-02T14:46:40.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.550+0000 7f1571211740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x562ef6167d10 0x562ef616bda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:40.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.550+0000 7f1571211740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562ef61689f0 0x562ef6178d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:40.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.550+0000 7f1571211740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562ef6171080 0x562ef61720c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:40.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.550+0000 7f1571211740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f15681bdb50 con 0x562ef6171080 2024-04-02T14:46:40.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.550+0000 7f1571211740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f15681bd480 con 0x562ef6167d10 2024-04-02T14:46:40.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.550+0000 7f1571211740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f15681ba6e0 con 0x562ef61689f0 2024-04-02T14:46:40.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.550+0000 7f1571211740 10 client.0 _ll_register_callbacks cb 0x7f15540ba7a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:46:40.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.550+0000 7f1567400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x562ef6167d10 0x562ef616bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:40.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.550+0000 7f1567400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x562ef6167d10 0x562ef616bda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:48064/0 (socket says 172.21.15.90:48064) 2024-04-02T14:46:40.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.550+0000 7f1567400640 1 -- 172.21.15.90:0/1034079412 learned_addr learned my addr 172.21.15.90:0/1034079412 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:46:40.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.551+0000 7f1567e00640 1 --2- 172.21.15.90:0/1034079412 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562ef6171080 0x562ef61720c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:40.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.551+0000 7f1566a00640 1 --2- 172.21.15.90:0/1034079412 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562ef61689f0 0x562ef6178d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:40.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.551+0000 7f1567e00640 1 -- 172.21.15.90:0/1034079412 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562ef61689f0 msgr2=0x562ef6178d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:46:40.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.552+0000 7f1567e00640 1 --2- 172.21.15.90:0/1034079412 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562ef61689f0 0x562ef6178d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:40.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.552+0000 7f1567e00640 1 -- 172.21.15.90:0/1034079412 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x562ef6167d10 msgr2=0x562ef616bda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:46:40.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.552+0000 7f1567e00640 1 --2- 172.21.15.90:0/1034079412 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x562ef6167d10 0x562ef616bda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:40.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.552+0000 7f1567e00640 1 -- 172.21.15.90:0/1034079412 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562ef6172600 con 0x562ef6171080 2024-04-02T14:46:40.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.552+0000 7f1566a00640 1 --2- 172.21.15.90:0/1034079412 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562ef61689f0 0x562ef6178d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:46:40.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.553+0000 7f1567e00640 1 --2- 172.21.15.90:0/1034079412 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562ef6171080 0x562ef61720c0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f15600127e0 tx=0x7f1560012f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f29616836ceb977b server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:46:40.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.553+0000 7f1551600640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-02T14:46:40.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.554+0000 7f1551600640 1 -- 172.21.15.90:0/1034079412 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1560030070 con 0x562ef6171080 2024-04-02T14:46:40.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.554+0000 7f1571211740 1 -- 172.21.15.90:0/1034079412 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1560003e50 con 0x562ef6171080 2024-04-02T14:46:40.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.554+0000 7f1551600640 1 -- 172.21.15.90:0/1034079412 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 8550+0+0 (secure 0 0 0) 0x7f1560003b80 con 0x562ef6171080 2024-04-02T14:46:40.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.554+0000 7f1551600640 1 -- 172.21.15.90:0/1034079412 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f15600156c0 con 0x562ef6171080 2024-04-02T14:46:40.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.554+0000 7f1571211740 10 client.18057 fetch_fsmap learned FSMap version 27 2024-04-02T14:46:40.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.554+0000 7f1571211740 1 -- 172.21.15.90:0/1034079412 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x562ef61e7a80 con 0x562ef6171080 2024-04-02T14:46:40.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.555+0000 7f1551600640 1 -- 172.21.15.90:0/1034079412 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1560003e50 con 0x562ef6171080 2024-04-02T14:46:40.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.555+0000 7f1551600640 1 -- 172.21.15.90:0/1034079412 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f156000f040 con 0x562ef6171080 2024-04-02T14:46:40.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.555+0000 7f1571211740 10 client.18057 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:46:40.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.555+0000 7f1571211740 10 client.18057 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-04-02T14:46:40.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.555+0000 7f1571211740 20 client.18057 populate_metadata read hostname 'smithi090' 2024-04-02T14:46:40.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.555+0000 7f1571211740 1 --2- 172.21.15.90:0/1034079412 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x562ef61e8190 0x562ef61ea5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:40.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.556+0000 7f1571211740 4 client.18057 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-02T14:46:40.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.556+0000 7f1571211740 1 -- 172.21.15.90:0/1034079412 --> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] -- command(tid 0: {"prefix": "damage ls"}) -- 0x562ef60d9c30 con 0x562ef61e8190 2024-04-02T14:46:40.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.556+0000 7f1567400640 1 --2- 172.21.15.90:0/1034079412 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x562ef61e8190 0x562ef61ea5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:40.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.557+0000 7f1551600640 10 client.18057 ms_handle_connect on v2:172.21.15.92:6836/1270635939 2024-04-02T14:46:40.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.557+0000 7f1567400640 1 --2- 172.21.15.90:0/1034079412 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x562ef61e8190 0x562ef61ea5b0 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=1880138b1f06639c server_cookie=8bd12f5b75b5dde9 in_seq=0 out_seq=0 2024-04-02T14:46:40.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.558+0000 7f1551600640 1 -- 172.21.15.90:0/1034079412 <== mds.3 v2:172.21.15.92:6836/1270635939 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x562ef60d9c30 con 0x562ef61e8190 2024-04-02T14:46:40.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.558+0000 7f1551600640 10 client.18057 handle_command_reply: tid=0 2024-04-02T14:46:40.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.558+0000 7f1571211740 1 client.18057 shutdown 2024-04-02T14:46:40.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.558+0000 7f1571211740 1 -- 172.21.15.90:0/1034079412 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562ef6171080 msgr2=0x562ef61720c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:40.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.558+0000 7f1571211740 1 --2- 172.21.15.90:0/1034079412 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562ef6171080 0x562ef61720c0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f15600127e0 tx=0x7f1560012f90 comp rx=0 tx=0).stop 2024-04-02T14:46:40.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.559+0000 7f1571211740 1 -- 172.21.15.90:0/1034079412 shutdown_connections 2024-04-02T14:46:40.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.559+0000 7f1571211740 1 --2- 172.21.15.90:0/1034079412 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x562ef61e8190 0x562ef61ea5b0 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:40.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.559+0000 7f1571211740 1 --2- 172.21.15.90:0/1034079412 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x562ef6171080 0x562ef61720c0 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:40.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.559+0000 7f1571211740 1 --2- 172.21.15.90:0/1034079412 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x562ef61689f0 0x562ef6178d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:40.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.559+0000 7f1571211740 1 --2- 172.21.15.90:0/1034079412 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x562ef6167d10 0x562ef616bda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:40.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.559+0000 7f1551600640 0 client.18057 ms_handle_reset on v2:172.21.15.92:6836/1270635939 2024-04-02T14:46:40.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.559+0000 7f1571211740 1 -- 172.21.15.90:0/1034079412 >> 172.21.15.90:0/1034079412 conn(0x562ef612ea50 msgr2=0x562ef6161010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:46:40.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.559+0000 7f1571211740 1 -- 172.21.15.90:0/1034079412 shutdown_connections 2024-04-02T14:46:40.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.559+0000 7f1571211740 1 -- 172.21.15.90:0/1034079412 wait complete. 2024-04-02T14:46:40.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.559+0000 7f1571211740 20 client.18057 trim_cache size 0 max 16384 2024-04-02T14:46:40.563 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-02T14:46:40.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.560+0000 7f156e400640 1 -- 172.21.15.90:0/4278934071 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f154c0a3da0 msgr2=0x7f154c0a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:40.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.560+0000 7f156e400640 1 --2- 172.21.15.90:0/4278934071 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f154c0a3da0 0x7f154c0a6260 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f156806b3a0 tx=0x7f1560009a50 comp rx=0 tx=0).stop 2024-04-02T14:46:40.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.560+0000 7f156e400640 1 -- 172.21.15.90:0/4278934071 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f15680747f0 msgr2=0x7f1568104a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:40.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.560+0000 7f156e400640 1 --2- 172.21.15.90:0/4278934071 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f15680747f0 0x7f1568104a70 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f1554005940 tx=0x7f155402cd10 comp rx=0 tx=0).stop 2024-04-02T14:46:40.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.561+0000 7f156e400640 1 -- 172.21.15.90:0/4278934071 shutdown_connections 2024-04-02T14:46:40.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.561+0000 7f156e400640 1 --2- 172.21.15.90:0/4278934071 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f154c0a3da0 0x7f154c0a6260 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:40.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.561+0000 7f156e400640 1 --2- 172.21.15.90:0/4278934071 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f156810e330 0x7f1568104fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:40.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.561+0000 7f156e400640 1 --2- 172.21.15.90:0/4278934071 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f15680747f0 0x7f1568104a70 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:40.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.561+0000 7f156e400640 1 --2- 172.21.15.90:0/4278934071 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1568073ed0 0x7f156810da70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:40.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.561+0000 7f156e400640 1 -- 172.21.15.90:0/4278934071 >> 172.21.15.90:0/4278934071 conn(0x7f1568078f00 msgr2=0x7f15680fe9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:46:40.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.562+0000 7f156e400640 1 -- 172.21.15.90:0/4278934071 shutdown_connections 2024-04-02T14:46:40.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.563+0000 7f156e400640 1 -- 172.21.15.90:0/4278934071 wait complete. 2024-04-02T14:46:40.583 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-02T14:46:40.604 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:46:40 smithi092 ceph-mon[36497]: pgmap v366: 97 pgs: 97 active+clean; 4.3 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 8.0 MiB/s wr, 780 op/s 2024-04-02T14:46:40.604 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:46:40 smithi092 ceph-mon[36497]: from='client.? 172.21.15.90:0/3981764374' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-02T14:46:40.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:46:40 smithi177 ceph-mon[36431]: pgmap v366: 97 pgs: 97 active+clean; 4.3 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 8.0 MiB/s wr, 780 op/s 2024-04-02T14:46:40.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:46:40 smithi177 ceph-mon[36431]: from='client.? 172.21.15.90:0/3981764374' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-02T14:46:40.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.885+0000 7fa774600640 1 -- 172.21.15.90:0/710881089 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa76c0747f0 msgr2=0x7fa76c10ddf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:40.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.885+0000 7fa774600640 1 --2- 172.21.15.90:0/710881089 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa76c0747f0 0x7fa76c10ddf0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7fa758009900 tx=0x7fa75802bcd0 comp rx=0 tx=0).stop 2024-04-02T14:46:40.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.889+0000 7fa774600640 1 -- 172.21.15.90:0/710881089 shutdown_connections 2024-04-02T14:46:40.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.889+0000 7fa774600640 1 --2- 172.21.15.90:0/710881089 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa76c10e330 0x7fa76c110740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:40.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.889+0000 7fa774600640 1 --2- 172.21.15.90:0/710881089 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa76c0747f0 0x7fa76c10ddf0 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:40.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.889+0000 7fa774600640 1 --2- 172.21.15.90:0/710881089 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa76c073ed0 0x7fa76c0742b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:40.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.889+0000 7fa774600640 1 -- 172.21.15.90:0/710881089 >> 172.21.15.90:0/710881089 conn(0x7fa76c078f00 msgr2=0x7fa76c079310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:46:40.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.890+0000 7fa774600640 1 -- 172.21.15.90:0/710881089 shutdown_connections 2024-04-02T14:46:40.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.894+0000 7fa774600640 1 -- 172.21.15.90:0/710881089 wait complete. 2024-04-02T14:46:40.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.895+0000 7fa774600640 1 Processor -- start 2024-04-02T14:46:40.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.896+0000 7fa774600640 1 -- start start 2024-04-02T14:46:40.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.897+0000 7fa774600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa76c073ed0 0x7fa76c19a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:40.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.897+0000 7fa774600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa76c0747f0 0x7fa76c19abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:40.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.897+0000 7fa774600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa76c10e330 0x7fa76c19ef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:40.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.897+0000 7fa774600640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fa76c114160 con 0x7fa76c073ed0 2024-04-02T14:46:40.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.897+0000 7fa774600640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fa76c113fe0 con 0x7fa76c0747f0 2024-04-02T14:46:40.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.897+0000 7fa774600640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fa76c1142e0 con 0x7fa76c10e330 2024-04-02T14:46:40.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.897+0000 7fa771200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa76c0747f0 0x7fa76c19abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:40.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.897+0000 7fa772600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa76c10e330 0x7fa76c19ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:40.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.897+0000 7fa771200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa76c0747f0 0x7fa76c19abe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:48080/0 (socket says 172.21.15.90:48080) 2024-04-02T14:46:40.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.897+0000 7fa771200640 1 -- 172.21.15.90:0/3163956178 learned_addr learned my addr 172.21.15.90:0/3163956178 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:46:40.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.897+0000 7fa772600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa76c10e330 0x7fa76c19ef10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:48494/0 (socket says 172.21.15.90:48494) 2024-04-02T14:46:40.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.898+0000 7fa771c00640 1 --2- 172.21.15.90:0/3163956178 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa76c073ed0 0x7fa76c19a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:40.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.898+0000 7fa772600640 1 -- 172.21.15.90:0/3163956178 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa76c0747f0 msgr2=0x7fa76c19abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:40.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.898+0000 7fa772600640 1 --2- 172.21.15.90:0/3163956178 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa76c0747f0 0x7fa76c19abe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:40.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.898+0000 7fa772600640 1 -- 172.21.15.90:0/3163956178 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa76c073ed0 msgr2=0x7fa76c19a6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:40.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.898+0000 7fa772600640 1 --2- 172.21.15.90:0/3163956178 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa76c073ed0 0x7fa76c19a6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:40.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.898+0000 7fa772600640 1 -- 172.21.15.90:0/3163956178 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa76c19f640 con 0x7fa76c10e330 2024-04-02T14:46:40.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.898+0000 7fa771c00640 1 --2- 172.21.15.90:0/3163956178 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa76c073ed0 0x7fa76c19a6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:46:40.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.898+0000 7fa772600640 1 --2- 172.21.15.90:0/3163956178 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa76c10e330 0x7fa76c19ef10 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7fa76c074e20 tx=0x7fa76400ef10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:46:40.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.899+0000 7fa76a000640 1 -- 172.21.15.90:0/3163956178 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa76400cdf0 con 0x7fa76c10e330 2024-04-02T14:46:40.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.899+0000 7fa76a000640 1 -- 172.21.15.90:0/3163956178 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa764005020 con 0x7fa76c10e330 2024-04-02T14:46:40.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.899+0000 7fa76a000640 1 -- 172.21.15.90:0/3163956178 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa764010670 con 0x7fa76c10e330 2024-04-02T14:46:40.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.901+0000 7fa774600640 1 -- 172.21.15.90:0/3163956178 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa76c19f930 con 0x7fa76c10e330 2024-04-02T14:46:40.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.901+0000 7fa774600640 1 -- 172.21.15.90:0/3163956178 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa76c101980 con 0x7fa76c10e330 2024-04-02T14:46:40.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.905+0000 7fa77725f740 1 Processor -- start 2024-04-02T14:46:40.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.906+0000 7fa76a000640 1 -- 172.21.15.90:0/3163956178 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fa7640054d0 con 0x7fa76c10e330 2024-04-02T14:46:40.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.909+0000 7fa76a000640 1 --2- 172.21.15.90:0/3163956178 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fa7540a3da0 0x7fa7540a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:40.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.909+0000 7fa76a000640 1 -- 172.21.15.90:0/3163956178 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 8550+0+0 (secure 0 0 0) 0x7fa7640d12b0 con 0x7fa76c10e330 2024-04-02T14:46:40.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.910+0000 7fa77725f740 1 -- start start 2024-04-02T14:46:40.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.910+0000 7fa77725f740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56385274b9f0 0x56385274a740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:40.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.910+0000 7fa77725f740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56385274ad10 0x56385274b190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:40.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.910+0000 7fa77725f740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563852754080 0x563852756470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:40.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.910+0000 7fa77725f740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x56385268b400 con 0x563852754080 2024-04-02T14:46:40.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.910+0000 7fa77725f740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x56385266ff10 con 0x56385274ad10 2024-04-02T14:46:40.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.910+0000 7fa77725f740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5638526af6f0 con 0x56385274b9f0 2024-04-02T14:46:40.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.910+0000 7fa771200640 1 --2- 172.21.15.90:0/3163956178 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fa7540a3da0 0x7fa7540a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:40.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.911+0000 7fa772600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563852754080 0x563852756470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:40.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.911+0000 7fa772600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563852754080 0x563852756470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:40166/0 (socket says 172.21.15.90:40166) 2024-04-02T14:46:40.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.911+0000 7fa772600640 1 -- 172.21.15.90:0/1033654411 learned_addr learned my addr 172.21.15.90:0/1033654411 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:46:40.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.911+0000 7fa771c00640 1 --2- 172.21.15.90:0/1033654411 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56385274b9f0 0x56385274a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:40.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.911+0000 7fa772600640 1 -- 172.21.15.90:0/1033654411 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56385274b9f0 msgr2=0x56385274a740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:46:40.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.911+0000 7fa772600640 1 --2- 172.21.15.90:0/1033654411 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56385274b9f0 0x56385274a740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:40.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.911+0000 7fa772600640 1 -- 172.21.15.90:0/1033654411 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56385274ad10 msgr2=0x56385274b190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:46:40.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.911+0000 7fa772600640 1 --2- 172.21.15.90:0/1033654411 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56385274ad10 0x56385274b190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:40.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.911+0000 7fa772600640 1 -- 172.21.15.90:0/1033654411 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5638523c33f0 con 0x563852754080 2024-04-02T14:46:40.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.912+0000 7fa772600640 1 --2- 172.21.15.90:0/1033654411 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563852754080 0x563852756470 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fa764005fd0 tx=0x7fa76409da70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=647958b8237b7c7e server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:46:40.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.913+0000 7fa771200640 1 --2- 172.21.15.90:0/3163956178 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fa7540a3da0 0x7fa7540a6260 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x56385274bf40 tx=0x7fa758005df0 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:46:40.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.914+0000 7fa75ea00640 1 -- 172.21.15.90:0/1033654411 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7640b5020 con 0x563852754080 2024-04-02T14:46:40.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.914+0000 7fa75ea00640 1 -- 172.21.15.90:0/1033654411 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa7640a5ea0 con 0x563852754080 2024-04-02T14:46:40.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.915+0000 7fa77725f740 1 -- 172.21.15.90:0/1033654411 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563852754080 msgr2=0x563852756470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:40.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.915+0000 7fa77725f740 1 --2- 172.21.15.90:0/1033654411 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563852754080 0x563852756470 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fa764005fd0 tx=0x7fa76409da70 comp rx=0 tx=0).stop 2024-04-02T14:46:40.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.915+0000 7fa77725f740 1 -- 172.21.15.90:0/1033654411 shutdown_connections 2024-04-02T14:46:40.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.915+0000 7fa77725f740 1 --2- 172.21.15.90:0/1033654411 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563852754080 0x563852756470 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:40.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.915+0000 7fa77725f740 1 --2- 172.21.15.90:0/1033654411 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56385274ad10 0x56385274b190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:40.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.915+0000 7fa77725f740 1 --2- 172.21.15.90:0/1033654411 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56385274b9f0 0x56385274a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:40.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.915+0000 7fa77725f740 1 -- 172.21.15.90:0/1033654411 >> 172.21.15.90:0/1033654411 conn(0x563852711a50 msgr2=0x563852745760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:46:40.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.916+0000 7fa77725f740 1 -- 172.21.15.90:0/1033654411 shutdown_connections 2024-04-02T14:46:40.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.916+0000 7fa77725f740 1 -- 172.21.15.90:0/1033654411 wait complete. 2024-04-02T14:46:40.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.917+0000 7fa77725f740 1 -- start start 2024-04-02T14:46:40.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.917+0000 7fa77725f740 1 Processor -- start 2024-04-02T14:46:40.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.918+0000 7fa77725f740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56385274ad10 0x563852753990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:40.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.918+0000 7fa77725f740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56385274b9f0 0x563852750c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:40.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.918+0000 7fa77725f740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5638527511c0 0x563852751620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:40.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.919+0000 7fa771c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56385274ad10 0x563852753990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:40.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.919+0000 7fa771200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56385274b9f0 0x563852750c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:40.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.919+0000 7fa771200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56385274b9f0 0x563852750c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:48094/0 (socket says 172.21.15.90:48094) 2024-04-02T14:46:40.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.919+0000 7fa772600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5638527511c0 0x563852751620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:40.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.919+0000 7fa771c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56385274ad10 0x563852753990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:40180/0 (socket says 172.21.15.90:40180) 2024-04-02T14:46:40.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.919+0000 7fa772600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5638527511c0 0x563852751620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:48514/0 (socket says 172.21.15.90:48514) 2024-04-02T14:46:40.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.919+0000 7fa771200640 1 -- 172.21.15.90:0/3234898460 learned_addr learned my addr 172.21.15.90:0/3234898460 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:46:40.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.919+0000 7fa77725f740 1 -- 172.21.15.90:0/3234898460 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5638526ab6a0 con 0x56385274ad10 2024-04-02T14:46:40.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.919+0000 7fa77725f740 1 -- 172.21.15.90:0/3234898460 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x563852657980 con 0x56385274b9f0 2024-04-02T14:46:40.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.919+0000 7fa77725f740 1 -- 172.21.15.90:0/3234898460 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x56385266acb0 con 0x5638527511c0 2024-04-02T14:46:40.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.920+0000 7fa77725f740 10 client.0 _ll_register_callbacks cb 0x56385266a6a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:46:40.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.920+0000 7fa772600640 1 -- 172.21.15.90:0/3234898460 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56385274b9f0 msgr2=0x563852750c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:46:40.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.920+0000 7fa772600640 1 --2- 172.21.15.90:0/3234898460 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56385274b9f0 0x563852750c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:40.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.920+0000 7fa772600640 1 -- 172.21.15.90:0/3234898460 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56385274ad10 msgr2=0x563852753990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:46:40.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.920+0000 7fa772600640 1 --2- 172.21.15.90:0/3234898460 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56385274ad10 0x563852753990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:40.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.920+0000 7fa772600640 1 -- 172.21.15.90:0/3234898460 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56385248d7d0 con 0x5638527511c0 2024-04-02T14:46:40.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.921+0000 7fa772600640 1 --2- 172.21.15.90:0/3234898460 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5638527511c0 0x563852751620 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7fa7640cefe0 tx=0x7fa76409cdf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d09c428fc33e5419 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:46:40.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.921+0000 7fa753400640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:46:40.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.921+0000 7fa753400640 1 -- 172.21.15.90:0/3234898460 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7640ba070 con 0x5638527511c0 2024-04-02T14:46:40.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.921+0000 7fa77725f740 1 -- 172.21.15.90:0/3234898460 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x56385266f040 con 0x5638527511c0 2024-04-02T14:46:40.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.921+0000 7fa753400640 1 -- 172.21.15.90:0/3234898460 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 8550+0+0 (secure 0 0 0) 0x7fa7640998a0 con 0x5638527511c0 2024-04-02T14:46:40.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.922+0000 7fa753400640 1 -- 172.21.15.90:0/3234898460 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fa7640b5040 con 0x5638527511c0 2024-04-02T14:46:40.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.923+0000 7fa77725f740 10 client.28930 fetch_fsmap learned FSMap version 27 2024-04-02T14:46:40.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.923+0000 7fa77725f740 1 -- 172.21.15.90:0/3234898460 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5638527630d0 con 0x5638527511c0 2024-04-02T14:46:40.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.923+0000 7fa753400640 1 -- 172.21.15.90:0/3234898460 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x56385266f040 con 0x5638527511c0 2024-04-02T14:46:40.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.923+0000 7fa753400640 1 -- 172.21.15.90:0/3234898460 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fa764020070 con 0x5638527511c0 2024-04-02T14:46:40.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.924+0000 7fa77725f740 10 client.28930 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:46:40.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.924+0000 7fa77725f740 10 client.28930 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-04-02T14:46:40.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.924+0000 7fa77725f740 20 client.28930 populate_metadata read hostname 'smithi090' 2024-04-02T14:46:40.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.924+0000 7fa77725f740 1 --2- 172.21.15.90:0/3234898460 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x563852764420 0x563852766810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:40.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.924+0000 7fa77725f740 4 client.28930 mds_command: new command op to 24547 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:46:40.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.924+0000 7fa77725f740 1 -- 172.21.15.90:0/3234898460 --> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5638526bcc30 con 0x563852764420 2024-04-02T14:46:40.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.924+0000 7fa771c00640 1 --2- 172.21.15.90:0/3234898460 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x563852764420 0x563852766810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:40.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.925+0000 7fa771c00640 1 --2- 172.21.15.90:0/3234898460 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x563852764420 0x563852766810 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=f6e642b9b2b84729 server_cookie=bbc5c471fab3ecc0 in_seq=0 out_seq=0 2024-04-02T14:46:40.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.925+0000 7fa753400640 10 client.28930 ms_handle_connect on v2:172.21.15.177:6838/664895663 2024-04-02T14:46:40.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.926+0000 7fa753400640 1 -- 172.21.15.90:0/3234898460 <== mds.2 v2:172.21.15.177:6838/664895663 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5638526bcc30 con 0x563852764420 2024-04-02T14:46:40.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.926+0000 7fa753400640 10 client.28930 handle_command_reply: tid=0 2024-04-02T14:46:40.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.926+0000 7fa77725f740 1 client.28930 shutdown 2024-04-02T14:46:40.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.926+0000 7fa77725f740 1 -- 172.21.15.90:0/3234898460 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5638527511c0 msgr2=0x563852751620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:40.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.926+0000 7fa77725f740 1 --2- 172.21.15.90:0/3234898460 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5638527511c0 0x563852751620 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7fa7640cefe0 tx=0x7fa76409cdf0 comp rx=0 tx=0).stop 2024-04-02T14:46:40.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.927+0000 7fa77725f740 1 -- 172.21.15.90:0/3234898460 shutdown_connections 2024-04-02T14:46:40.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.927+0000 7fa77725f740 1 --2- 172.21.15.90:0/3234898460 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x563852764420 0x563852766810 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:40.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.927+0000 7fa77725f740 1 --2- 172.21.15.90:0/3234898460 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5638527511c0 0x563852751620 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:40.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.927+0000 7fa77725f740 1 --2- 172.21.15.90:0/3234898460 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56385274b9f0 0x563852750c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:40.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.927+0000 7fa77725f740 1 --2- 172.21.15.90:0/3234898460 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56385274ad10 0x563852753990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:40.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.927+0000 7fa753400640 0 client.28930 ms_handle_reset on v2:172.21.15.177:6838/664895663 2024-04-02T14:46:40.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.927+0000 7fa77725f740 1 -- 172.21.15.90:0/3234898460 >> 172.21.15.90:0/3234898460 conn(0x563852711a50 msgr2=0x5638527474b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:46:40.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.927+0000 7fa77725f740 1 -- 172.21.15.90:0/3234898460 shutdown_connections 2024-04-02T14:46:40.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.927+0000 7fa77725f740 1 -- 172.21.15.90:0/3234898460 wait complete. 2024-04-02T14:46:40.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.928+0000 7fa77725f740 20 client.28930 trim_cache size 0 max 16384 2024-04-02T14:46:40.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.970+0000 7fa77725f740 1 Processor -- start 2024-04-02T14:46:40.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.970+0000 7fa77725f740 1 -- start start 2024-04-02T14:46:40.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.970+0000 7fa77725f740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563852711a50 0x5638527564d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:40.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.971+0000 7fa77725f740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56385274ad10 0x56385274fde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:40.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.971+0000 7fa77725f740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56385274b9f0 0x5638527472c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:40.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.971+0000 7fa77725f740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fa76c1b3f00 con 0x563852711a50 2024-04-02T14:46:40.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.971+0000 7fa77725f740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fa76c1b5a40 con 0x56385274ad10 2024-04-02T14:46:40.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.971+0000 7fa77725f740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fa76c1b45d0 con 0x56385274b9f0 2024-04-02T14:46:40.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.971+0000 7fa772600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56385274b9f0 0x5638527472c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:40.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.971+0000 7fa772600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56385274b9f0 0x5638527472c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:48530/0 (socket says 172.21.15.90:48530) 2024-04-02T14:46:40.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.971+0000 7fa772600640 1 -- 172.21.15.90:0/3345870362 learned_addr learned my addr 172.21.15.90:0/3345870362 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:46:40.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.972+0000 7fa771c00640 1 --2- 172.21.15.90:0/3345870362 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563852711a50 0x5638527564d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:40.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.973+0000 7fa771200640 1 --2- 172.21.15.90:0/3345870362 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56385274ad10 0x56385274fde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:40.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.973+0000 7fa772600640 1 -- 172.21.15.90:0/3345870362 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56385274ad10 msgr2=0x56385274fde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:46:40.969 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.973+0000 7fa772600640 1 --2- 172.21.15.90:0/3345870362 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56385274ad10 0x56385274fde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:40.969 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.973+0000 7fa772600640 1 -- 172.21.15.90:0/3345870362 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563852711a50 msgr2=0x5638527564d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:46:40.969 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.973+0000 7fa772600640 1 --2- 172.21.15.90:0/3345870362 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563852711a50 0x5638527564d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:40.969 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.973+0000 7fa772600640 1 -- 172.21.15.90:0/3345870362 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563852749d40 con 0x56385274b9f0 2024-04-02T14:46:40.969 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.973+0000 7fa772600640 1 --2- 172.21.15.90:0/3345870362 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56385274b9f0 0x5638527472c0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7fa764009fd0 tx=0x7fa76409b740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=93f240d73b376818 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:46:40.970 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.974+0000 7fa75e000640 1 -- 172.21.15.90:0/3345870362 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7640ba070 con 0x56385274b9f0 2024-04-02T14:46:40.970 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.974+0000 7fa75e000640 1 -- 172.21.15.90:0/3345870362 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa7640a29e0 con 0x56385274b9f0 2024-04-02T14:46:40.970 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.974+0000 7fa75e000640 1 -- 172.21.15.90:0/3345870362 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7640862e0 con 0x56385274b9f0 2024-04-02T14:46:40.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.975+0000 7fa77725f740 1 -- 172.21.15.90:0/3345870362 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56385274b9f0 msgr2=0x5638527472c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:40.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.975+0000 7fa77725f740 1 --2- 172.21.15.90:0/3345870362 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56385274b9f0 0x5638527472c0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7fa764009fd0 tx=0x7fa76409b740 comp rx=0 tx=0).stop 2024-04-02T14:46:40.972 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.975+0000 7fa77725f740 1 -- 172.21.15.90:0/3345870362 shutdown_connections 2024-04-02T14:46:40.972 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.975+0000 7fa77725f740 1 --2- 172.21.15.90:0/3345870362 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56385274b9f0 0x5638527472c0 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:40.972 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.975+0000 7fa77725f740 1 --2- 172.21.15.90:0/3345870362 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56385274ad10 0x56385274fde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:40.972 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.975+0000 7fa77725f740 1 --2- 172.21.15.90:0/3345870362 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563852711a50 0x5638527564d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:40.972 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.975+0000 7fa77725f740 1 -- 172.21.15.90:0/3345870362 >> 172.21.15.90:0/3345870362 conn(0x563852740b80 msgr2=0x563852746bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:46:40.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.977+0000 7fa77725f740 1 -- 172.21.15.90:0/3345870362 shutdown_connections 2024-04-02T14:46:40.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.977+0000 7fa77725f740 1 -- 172.21.15.90:0/3345870362 wait complete. 2024-04-02T14:46:40.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.977+0000 7fa77725f740 1 -- start start 2024-04-02T14:46:40.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.978+0000 7fa77725f740 1 Processor -- start 2024-04-02T14:46:40.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.978+0000 7fa77725f740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563852711a50 0x563852769d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:40.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.978+0000 7fa77725f740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56385274ad10 0x563852765090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:40.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.978+0000 7fa77725f740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56385274b9f0 0x5638527656a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:40.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.978+0000 7fa77725f740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fa76c1b5370 con 0x563852711a50 2024-04-02T14:46:40.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.978+0000 7fa77725f740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fa76c1b4ca0 con 0x56385274ad10 2024-04-02T14:46:40.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.978+0000 7fa77725f740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fa76c1b1f00 con 0x56385274b9f0 2024-04-02T14:46:40.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.979+0000 7fa77725f740 10 client.0 _ll_register_callbacks cb 0x7fa7640a1720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:46:40.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.979+0000 7fa771c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563852711a50 0x563852769d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:40.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.979+0000 7fa772600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56385274b9f0 0x5638527656a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:40.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.979+0000 7fa772600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56385274b9f0 0x5638527656a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:48546/0 (socket says 172.21.15.90:48546) 2024-04-02T14:46:40.976 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.979+0000 7fa772600640 1 -- 172.21.15.90:0/3414851013 learned_addr learned my addr 172.21.15.90:0/3414851013 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:46:40.976 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.979+0000 7fa771c00640 1 --2- 172.21.15.90:0/3414851013 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563852711a50 0x563852769d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:40188/0 (socket says 172.21.15.90:40188) 2024-04-02T14:46:40.976 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.979+0000 7fa771200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56385274ad10 0x563852765090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:40.976 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.979+0000 7fa772600640 1 -- 172.21.15.90:0/3414851013 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56385274ad10 msgr2=0x563852765090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:46:40.976 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.979+0000 7fa772600640 1 --2- 172.21.15.90:0/3414851013 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56385274ad10 0x563852765090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:40.976 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.979+0000 7fa772600640 1 -- 172.21.15.90:0/3414851013 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563852711a50 msgr2=0x563852769d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:46:40.976 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.979+0000 7fa772600640 1 --2- 172.21.15.90:0/3414851013 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563852711a50 0x563852769d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:40.976 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.979+0000 7fa772600640 1 -- 172.21.15.90:0/3414851013 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563852765ed0 con 0x56385274b9f0 2024-04-02T14:46:40.976 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.980+0000 7fa772600640 1 --2- 172.21.15.90:0/3414851013 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56385274b9f0 0x5638527656a0 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7fa764005fd0 tx=0x7fa7640991b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fa04137b5b29a6ca server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:46:40.976 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.980+0000 7fa753400640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:46:40.976 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.980+0000 7fa753400640 1 -- 172.21.15.90:0/3414851013 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7640ba070 con 0x56385274b9f0 2024-04-02T14:46:40.978 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.980+0000 7fa753400640 1 -- 172.21.15.90:0/3414851013 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 8550+0+0 (secure 0 0 0) 0x7fa76409c0f0 con 0x56385274b9f0 2024-04-02T14:46:40.978 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.980+0000 7fa77725f740 1 -- 172.21.15.90:0/3414851013 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa7640a29e0 con 0x56385274b9f0 2024-04-02T14:46:40.978 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.981+0000 7fa753400640 1 -- 172.21.15.90:0/3414851013 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fa76409c350 con 0x56385274b9f0 2024-04-02T14:46:40.978 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.981+0000 7fa77725f740 10 client.28936 fetch_fsmap learned FSMap version 27 2024-04-02T14:46:40.978 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.981+0000 7fa77725f740 1 -- 172.21.15.90:0/3414851013 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x563852765ed0 con 0x56385274b9f0 2024-04-02T14:46:40.978 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.981+0000 7fa753400640 1 -- 172.21.15.90:0/3414851013 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa7640a29e0 con 0x56385274b9f0 2024-04-02T14:46:40.978 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.982+0000 7fa753400640 1 -- 172.21.15.90:0/3414851013 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fa764020030 con 0x56385274b9f0 2024-04-02T14:46:40.978 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.982+0000 7fa77725f740 10 client.28936 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:46:40.979 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.982+0000 7fa77725f740 10 client.28936 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-04-02T14:46:40.979 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.982+0000 7fa77725f740 20 client.28936 populate_metadata read hostname 'smithi090' 2024-04-02T14:46:40.979 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.982+0000 7fa77725f740 1 --2- 172.21.15.90:0/3414851013 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x5638527511c0 0x5638527dd990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:40.979 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.982+0000 7fa77725f740 4 client.28936 mds_command: new command op to 24547 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-02T14:46:40.979 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.982+0000 7fa77725f740 1 -- 172.21.15.90:0/3414851013 --> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5638526b6f80 con 0x5638527511c0 2024-04-02T14:46:40.980 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.983+0000 7fa771c00640 1 --2- 172.21.15.90:0/3414851013 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x5638527511c0 0x5638527dd990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:40.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.987+0000 7fa771c00640 1 --2- 172.21.15.90:0/3414851013 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x5638527511c0 0x5638527dd990 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=12fe112b803a8e35 server_cookie=8de95adf6ef990c in_seq=0 out_seq=0 2024-04-02T14:46:40.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.987+0000 7fa753400640 10 client.28936 ms_handle_connect on v2:172.21.15.177:6838/664895663 2024-04-02T14:46:40.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.987+0000 7fa753400640 1 -- 172.21.15.90:0/3414851013 <== mds.2 v2:172.21.15.177:6838/664895663 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5638526b6f80 con 0x5638527511c0 2024-04-02T14:46:40.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.987+0000 7fa753400640 10 client.28936 handle_command_reply: tid=0 2024-04-02T14:46:40.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.987+0000 7fa77725f740 1 client.28936 shutdown 2024-04-02T14:46:40.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.988+0000 7fa77725f740 1 -- 172.21.15.90:0/3414851013 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56385274b9f0 msgr2=0x5638527656a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:40.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.988+0000 7fa77725f740 1 --2- 172.21.15.90:0/3414851013 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56385274b9f0 0x5638527656a0 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7fa764005fd0 tx=0x7fa7640991b0 comp rx=0 tx=0).stop 2024-04-02T14:46:40.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.988+0000 7fa77725f740 1 -- 172.21.15.90:0/3414851013 shutdown_connections 2024-04-02T14:46:40.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.988+0000 7fa77725f740 1 --2- 172.21.15.90:0/3414851013 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x5638527511c0 0x5638527dd990 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:40.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.988+0000 7fa77725f740 1 --2- 172.21.15.90:0/3414851013 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56385274b9f0 0x5638527656a0 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:40.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.988+0000 7fa77725f740 1 --2- 172.21.15.90:0/3414851013 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56385274ad10 0x563852765090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:40.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.988+0000 7fa77725f740 1 --2- 172.21.15.90:0/3414851013 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563852711a50 0x563852769d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:40.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.988+0000 7fa753400640 0 client.28936 ms_handle_reset on v2:172.21.15.177:6838/664895663 2024-04-02T14:46:40.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.988+0000 7fa77725f740 1 -- 172.21.15.90:0/3414851013 >> 172.21.15.90:0/3414851013 conn(0x563852740b80 msgr2=0x56385275f920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:46:40.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.989+0000 7fa77725f740 1 -- 172.21.15.90:0/3414851013 shutdown_connections 2024-04-02T14:46:40.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.990+0000 7fa77725f740 1 -- 172.21.15.90:0/3414851013 wait complete. 2024-04-02T14:46:40.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.990+0000 7fa77725f740 20 client.28936 trim_cache size 0 max 16384 2024-04-02T14:46:40.987 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-02T14:46:40.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.992+0000 7fa774600640 1 -- 172.21.15.90:0/3163956178 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fa7540a3da0 msgr2=0x7fa7540a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:40.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.992+0000 7fa774600640 1 --2- 172.21.15.90:0/3163956178 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fa7540a3da0 0x7fa7540a6260 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x56385274bf40 tx=0x7fa758005df0 comp rx=0 tx=0).stop 2024-04-02T14:46:40.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.992+0000 7fa774600640 1 -- 172.21.15.90:0/3163956178 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa76c10e330 msgr2=0x7fa76c19ef10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:40.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.992+0000 7fa774600640 1 --2- 172.21.15.90:0/3163956178 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa76c10e330 0x7fa76c19ef10 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7fa76c074e20 tx=0x7fa76400ef10 comp rx=0 tx=0).stop 2024-04-02T14:46:40.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.993+0000 7fa774600640 1 -- 172.21.15.90:0/3163956178 shutdown_connections 2024-04-02T14:46:40.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.993+0000 7fa774600640 1 --2- 172.21.15.90:0/3163956178 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fa7540a3da0 0x7fa7540a6260 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:40.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.993+0000 7fa774600640 1 --2- 172.21.15.90:0/3163956178 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa76c10e330 0x7fa76c19ef10 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:40.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.993+0000 7fa774600640 1 --2- 172.21.15.90:0/3163956178 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa76c0747f0 0x7fa76c19abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:40.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.993+0000 7fa774600640 1 --2- 172.21.15.90:0/3163956178 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa76c073ed0 0x7fa76c19a6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:40.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.993+0000 7fa774600640 1 -- 172.21.15.90:0/3163956178 >> 172.21.15.90:0/3163956178 conn(0x7fa76c078f00 msgr2=0x7fa76c0fe9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:46:40.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.993+0000 7fa774600640 1 -- 172.21.15.90:0/3163956178 shutdown_connections 2024-04-02T14:46:40.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:40.993+0000 7fa774600640 1 -- 172.21.15.90:0/3163956178 wait complete. 2024-04-02T14:46:41.017 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-02T14:46:41.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.314+0000 7f7c71600640 1 -- 172.21.15.90:0/232647315 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7c6c108310 msgr2=0x7f7c6c1086f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:41.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.314+0000 7f7c71600640 1 --2- 172.21.15.90:0/232647315 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7c6c108310 0x7f7c6c1086f0 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f7c5c009860 tx=0x7f7c5c02bb70 comp rx=0 tx=0).stop 2024-04-02T14:46:41.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.314+0000 7f7c71600640 1 -- 172.21.15.90:0/232647315 shutdown_connections 2024-04-02T14:46:41.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.314+0000 7f7c71600640 1 --2- 172.21.15.90:0/232647315 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7c6c10d3e0 0x7f7c6c114fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:41.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.314+0000 7f7c71600640 1 --2- 172.21.15.90:0/232647315 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7c6c108cc0 0x7f7c6c10cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:41.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.314+0000 7f7c71600640 1 --2- 172.21.15.90:0/232647315 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7c6c108310 0x7f7c6c1086f0 secure :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f7c5c009860 tx=0x7f7c5c02bb70 comp rx=0 tx=0).stop 2024-04-02T14:46:41.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.314+0000 7f7c71600640 1 -- 172.21.15.90:0/232647315 >> 172.21.15.90:0/232647315 conn(0x7f7c6c076040 msgr2=0x7f7c6c078460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:46:41.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.315+0000 7f7c71600640 1 -- 172.21.15.90:0/232647315 shutdown_connections 2024-04-02T14:46:41.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.315+0000 7f7c71600640 1 -- 172.21.15.90:0/232647315 wait complete. 2024-04-02T14:46:41.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.316+0000 7f7c71600640 1 Processor -- start 2024-04-02T14:46:41.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.317+0000 7f7c71600640 1 -- start start 2024-04-02T14:46:41.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.318+0000 7f7c71600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7c6c10d3e0 0x7f7c6c114e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:41.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.318+0000 7f7c71600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7c6c111040 0x7f7c6c1114c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:41.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.318+0000 7f7c71600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7c6c111a00 0x7f7c6c1a2860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:41.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.318+0000 7f7c71600640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f7c6c07cce0 con 0x7f7c6c111040 2024-04-02T14:46:41.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.318+0000 7f7c71600640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f7c6c07cb60 con 0x7f7c6c10d3e0 2024-04-02T14:46:41.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.318+0000 7f7c71600640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f7c6c07ce60 con 0x7f7c6c111a00 2024-04-02T14:46:41.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.318+0000 7f7c6aa00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7c6c10d3e0 0x7f7c6c114e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:41.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.318+0000 7f7c6b400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7c6c111a00 0x7f7c6c1a2860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:41.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.319+0000 7f7c6b400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7c6c111a00 0x7f7c6c1a2860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:48572/0 (socket says 172.21.15.90:48572) 2024-04-02T14:46:41.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.319+0000 7f7c6b400640 1 -- 172.21.15.90:0/3453017319 learned_addr learned my addr 172.21.15.90:0/3453017319 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:46:41.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.319+0000 7f7c6aa00640 1 -- 172.21.15.90:0/3453017319 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7c6c111a00 msgr2=0x7f7c6c1a2860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:41.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.319+0000 7f7c6aa00640 1 --2- 172.21.15.90:0/3453017319 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7c6c111a00 0x7f7c6c1a2860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:41.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.319+0000 7f7c6aa00640 1 -- 172.21.15.90:0/3453017319 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7c6c111040 msgr2=0x7f7c6c1114c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:41.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.319+0000 7f7c6aa00640 1 --2- 172.21.15.90:0/3453017319 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7c6c111040 0x7f7c6c1114c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:41.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.319+0000 7f7c6aa00640 1 -- 172.21.15.90:0/3453017319 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c6c1a2ee0 con 0x7f7c6c10d3e0 2024-04-02T14:46:41.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.321+0000 7f7c6aa00640 1 --2- 172.21.15.90:0/3453017319 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7c6c10d3e0 0x7f7c6c114e90 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f7c6c109530 tx=0x7f7c5c034670 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:46:41.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.322+0000 7f7c63400640 1 -- 172.21.15.90:0/3453017319 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c5c039040 con 0x7f7c6c10d3e0 2024-04-02T14:46:41.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.322+0000 7f7c63400640 1 -- 172.21.15.90:0/3453017319 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7c5c02c9f0 con 0x7f7c6c10d3e0 2024-04-02T14:46:41.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.322+0000 7f7c71600640 1 -- 172.21.15.90:0/3453017319 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c6c1a3110 con 0x7f7c6c10d3e0 2024-04-02T14:46:41.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.322+0000 7f7c63400640 1 -- 172.21.15.90:0/3453017319 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c5c004920 con 0x7f7c6c10d3e0 2024-04-02T14:46:41.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.322+0000 7f7c71600640 1 -- 172.21.15.90:0/3453017319 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c6c1a3570 con 0x7f7c6c10d3e0 2024-04-02T14:46:41.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.328+0000 7f7c741fb740 1 Processor -- start 2024-04-02T14:46:41.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.328+0000 7f7c63400640 1 -- 172.21.15.90:0/3453017319 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f7c5c044020 con 0x7f7c6c10d3e0 2024-04-02T14:46:41.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.328+0000 7f7c741fb740 1 -- start start 2024-04-02T14:46:41.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.329+0000 7f7c741fb740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561116d8e9f0 0x561116d8d740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:41.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.330+0000 7f7c741fb740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561116d8dd10 0x561116d8e190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:41.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.330+0000 7f7c6b400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561116d8e9f0 0x561116d8d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:41.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.330+0000 7f7c6b400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561116d8e9f0 0x561116d8d740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:48136/0 (socket says 172.21.15.90:48136) 2024-04-02T14:46:41.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.331+0000 7f7c741fb740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561116d97080 0x561116d99470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:41.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.331+0000 7f7c741fb740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x561116cce400 con 0x561116d97080 2024-04-02T14:46:41.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.331+0000 7f7c741fb740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x561116cb2f10 con 0x561116d8e9f0 2024-04-02T14:46:41.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.331+0000 7f7c741fb740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x561116cf26f0 con 0x561116d8dd10 2024-04-02T14:46:41.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.331+0000 7f7c63400640 1 --2- 172.21.15.90:0/3453017319 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f7c500a3da0 0x7f7c500a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:41.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.331+0000 7f7c63400640 1 -- 172.21.15.90:0/3453017319 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 8550+0+0 (secure 0 0 0) 0x7f7c5c0f11d0 con 0x7f7c6c10d3e0 2024-04-02T14:46:41.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.332+0000 7f7c6a000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561116d97080 0x561116d99470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:41.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.332+0000 7f7c6a000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561116d97080 0x561116d99470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:40226/0 (socket says 172.21.15.90:40226) 2024-04-02T14:46:41.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.332+0000 7f7c6a000640 1 -- 172.21.15.90:0/715586585 learned_addr learned my addr 172.21.15.90:0/715586585 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:46:41.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.333+0000 7f7c6a000640 1 -- 172.21.15.90:0/715586585 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561116d8dd10 msgr2=0x561116d8e190 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-02T14:46:41.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.333+0000 7f7c6b400640 1 --2- 172.21.15.90:0/3453017319 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f7c500a3da0 0x7f7c500a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:41.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.333+0000 7f7c6a000640 1 --2- 172.21.15.90:0/715586585 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561116d8dd10 0x561116d8e190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:41.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.333+0000 7f7c6a000640 1 -- 172.21.15.90:0/715586585 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561116d8e9f0 msgr2=0x561116d8d740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:46:41.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.333+0000 7f7c6a000640 1 --2- 172.21.15.90:0/715586585 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561116d8e9f0 0x561116d8d740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:41.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.333+0000 7f7c6a000640 1 -- 172.21.15.90:0/715586585 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561116a063f0 con 0x561116d97080 2024-04-02T14:46:41.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.334+0000 7f7c6a000640 1 --2- 172.21.15.90:0/715586585 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561116d97080 0x561116d99470 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f7c54004650 tx=0x7f7c5401f360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6389fec0637bc81 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:46:41.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.334+0000 7f7c6b400640 1 --2- 172.21.15.90:0/3453017319 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f7c500a3da0 0x7f7c500a6260 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f7c64009fd0 tx=0x7f7c64004ab0 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:46:41.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.334+0000 7f7c5b400640 1 -- 172.21.15.90:0/715586585 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c5401fd70 con 0x561116d97080 2024-04-02T14:46:41.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.334+0000 7f7c5b400640 1 -- 172.21.15.90:0/715586585 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7c54005500 con 0x561116d97080 2024-04-02T14:46:41.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.334+0000 7f7c5b400640 1 -- 172.21.15.90:0/715586585 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c540216a0 con 0x561116d97080 2024-04-02T14:46:41.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.335+0000 7f7c741fb740 1 -- 172.21.15.90:0/715586585 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561116d97080 msgr2=0x561116d99470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:41.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.335+0000 7f7c741fb740 1 --2- 172.21.15.90:0/715586585 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561116d97080 0x561116d99470 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f7c54004650 tx=0x7f7c5401f360 comp rx=0 tx=0).stop 2024-04-02T14:46:41.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.337+0000 7f7c741fb740 1 -- 172.21.15.90:0/715586585 shutdown_connections 2024-04-02T14:46:41.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.337+0000 7f7c741fb740 1 --2- 172.21.15.90:0/715586585 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561116d97080 0x561116d99470 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:41.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.337+0000 7f7c741fb740 1 --2- 172.21.15.90:0/715586585 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561116d8dd10 0x561116d8e190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:41.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.337+0000 7f7c741fb740 1 --2- 172.21.15.90:0/715586585 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561116d8e9f0 0x561116d8d740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:41.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.337+0000 7f7c741fb740 1 -- 172.21.15.90:0/715586585 >> 172.21.15.90:0/715586585 conn(0x561116d54a50 msgr2=0x561116d88760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:46:41.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.337+0000 7f7c741fb740 1 -- 172.21.15.90:0/715586585 shutdown_connections 2024-04-02T14:46:41.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.337+0000 7f7c741fb740 1 -- 172.21.15.90:0/715586585 wait complete. 2024-04-02T14:46:41.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.341+0000 7f7c741fb740 1 -- start start 2024-04-02T14:46:41.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.341+0000 7f7c741fb740 1 Processor -- start 2024-04-02T14:46:41.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.344+0000 7f7c741fb740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561116d8dd10 0x561116d940d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:41.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.344+0000 7f7c741fb740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561116d8e9f0 0x561116d94610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:41.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.344+0000 7f7c741fb740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561116d97080 0x561116d94b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:41.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.345+0000 7f7c741fb740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x561116cee6a0 con 0x561116d8dd10 2024-04-02T14:46:41.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.345+0000 7f7c741fb740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x561116c9a980 con 0x561116d97080 2024-04-02T14:46:41.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.345+0000 7f7c741fb740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x561116cadcb0 con 0x561116d8e9f0 2024-04-02T14:46:41.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.345+0000 7f7c741fb740 10 client.0 _ll_register_callbacks cb 0x561116cad6a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:46:41.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.346+0000 7f7c6a000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561116d8e9f0 0x561116d94610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:41.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.347+0000 7f7c6aa00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561116d8dd10 0x561116d940d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:41.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.347+0000 7f7c6aa00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561116d8dd10 0x561116d940d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:40240/0 (socket says 172.21.15.90:40240) 2024-04-02T14:46:41.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.347+0000 7f7c6aa00640 1 -- 172.21.15.90:0/3352899105 learned_addr learned my addr 172.21.15.90:0/3352899105 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:46:41.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.347+0000 7f7c6b400640 1 --2- 172.21.15.90:0/3352899105 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561116d97080 0x561116d94b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:41.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.348+0000 7f7c6aa00640 1 -- 172.21.15.90:0/3352899105 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561116d8e9f0 msgr2=0x561116d94610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:46:41.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.348+0000 7f7c6aa00640 1 --2- 172.21.15.90:0/3352899105 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561116d8e9f0 0x561116d94610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:41.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.348+0000 7f7c6aa00640 1 -- 172.21.15.90:0/3352899105 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561116d97080 msgr2=0x561116d94b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:46:41.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.348+0000 7f7c6aa00640 1 --2- 172.21.15.90:0/3352899105 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561116d97080 0x561116d94b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:41.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.348+0000 7f7c6aa00640 1 -- 172.21.15.90:0/3352899105 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561116ad07d0 con 0x561116d8dd10 2024-04-02T14:46:41.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.348+0000 7f7c6b400640 1 --2- 172.21.15.90:0/3352899105 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561116d97080 0x561116d94b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:46:41.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.350+0000 7f7c6aa00640 1 --2- 172.21.15.90:0/3352899105 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561116d8dd10 0x561116d940d0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f7c5c0c5940 tx=0x7f7c5c0bd090 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22f1bdb3cdf45787 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:46:41.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.350+0000 7f7c4fe00640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-02T14:46:41.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.350+0000 7f7c4fe00640 1 -- 172.21.15.90:0/3352899105 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c5c0bd900 con 0x561116d8dd10 2024-04-02T14:46:41.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.351+0000 7f7c4fe00640 1 -- 172.21.15.90:0/3352899105 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 8550+0+0 (secure 0 0 0) 0x7f7c5c0eeb20 con 0x561116d8dd10 2024-04-02T14:46:41.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.351+0000 7f7c741fb740 1 -- 172.21.15.90:0/3352899105 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7c54005500 con 0x561116d8dd10 2024-04-02T14:46:41.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.353+0000 7f7c4fe00640 1 -- 172.21.15.90:0/3352899105 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f7c5c0dc050 con 0x561116d8dd10 2024-04-02T14:46:41.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.354+0000 7f7c741fb740 10 client.18078 fetch_fsmap learned FSMap version 27 2024-04-02T14:46:41.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.354+0000 7f7c741fb740 1 -- 172.21.15.90:0/3352899105 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x561116da6030 con 0x561116d8dd10 2024-04-02T14:46:41.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.355+0000 7f7c4fe00640 1 -- 172.21.15.90:0/3352899105 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7c54005500 con 0x561116d8dd10 2024-04-02T14:46:41.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.355+0000 7f7c4fe00640 1 -- 172.21.15.90:0/3352899105 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f7c5c03c070 con 0x561116d8dd10 2024-04-02T14:46:41.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.355+0000 7f7c741fb740 10 client.18078 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:46:41.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.355+0000 7f7c741fb740 10 client.18078 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2024-04-02T14:46:41.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.355+0000 7f7c741fb740 20 client.18078 populate_metadata read hostname 'smithi090' 2024-04-02T14:46:41.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.355+0000 7f7c741fb740 1 --2- 172.21.15.90:0/3352899105 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x561116da7490 0x561116da98a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:41.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.355+0000 7f7c741fb740 4 client.18078 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:46:41.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.355+0000 7f7c741fb740 1 -- 172.21.15.90:0/3352899105 --> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561116cb2040 con 0x561116da7490 2024-04-02T14:46:41.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.356+0000 7f7c6a000640 1 --2- 172.21.15.90:0/3352899105 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x561116da7490 0x561116da98a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:41.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.358+0000 7f7c4fe00640 10 client.18078 ms_handle_connect on v2:172.21.15.90:6838/1498915870 2024-04-02T14:46:41.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.358+0000 7f7c6a000640 1 --2- 172.21.15.90:0/3352899105 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x561116da7490 0x561116da98a0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=a9d85b024f1f0ffc server_cookie=909df6bda2eeab29 in_seq=0 out_seq=0 2024-04-02T14:46:41.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.359+0000 7f7c4fe00640 1 -- 172.21.15.90:0/3352899105 <== mds.1 v2:172.21.15.90:6838/1498915870 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x561116cb2040 con 0x561116da7490 2024-04-02T14:46:41.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.359+0000 7f7c4fe00640 10 client.18078 handle_command_reply: tid=0 2024-04-02T14:46:41.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.359+0000 7f7c741fb740 1 client.18078 shutdown 2024-04-02T14:46:41.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.360+0000 7f7c741fb740 1 -- 172.21.15.90:0/3352899105 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561116d8dd10 msgr2=0x561116d940d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:41.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.360+0000 7f7c741fb740 1 --2- 172.21.15.90:0/3352899105 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561116d8dd10 0x561116d940d0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f7c5c0c5940 tx=0x7f7c5c0bd090 comp rx=0 tx=0).stop 2024-04-02T14:46:41.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.361+0000 7f7c741fb740 1 -- 172.21.15.90:0/3352899105 shutdown_connections 2024-04-02T14:46:41.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.361+0000 7f7c741fb740 1 --2- 172.21.15.90:0/3352899105 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x561116da7490 0x561116da98a0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:41.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.361+0000 7f7c741fb740 1 --2- 172.21.15.90:0/3352899105 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561116d97080 0x561116d94b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:41.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.361+0000 7f7c741fb740 1 --2- 172.21.15.90:0/3352899105 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561116d8e9f0 0x561116d94610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:41.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.361+0000 7f7c741fb740 1 --2- 172.21.15.90:0/3352899105 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561116d8dd10 0x561116d940d0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:41.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.361+0000 7f7c741fb740 1 -- 172.21.15.90:0/3352899105 >> 172.21.15.90:0/3352899105 conn(0x561116d54a50 msgr2=0x561116d8a4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:46:41.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.361+0000 7f7c4fe00640 0 client.18078 ms_handle_reset on v2:172.21.15.90:6838/1498915870 2024-04-02T14:46:41.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.362+0000 7f7c741fb740 1 -- 172.21.15.90:0/3352899105 shutdown_connections 2024-04-02T14:46:41.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.362+0000 7f7c741fb740 1 -- 172.21.15.90:0/3352899105 wait complete. 2024-04-02T14:46:41.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.362+0000 7f7c741fb740 20 client.18078 trim_cache size 0 max 16384 2024-04-02T14:46:41.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.387+0000 7f7c741fb740 1 Processor -- start 2024-04-02T14:46:41.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.387+0000 7f7c741fb740 1 -- start start 2024-04-02T14:46:41.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.388+0000 7f7c741fb740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561116d8dd10 0x561116d96800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:41.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.388+0000 7f7c741fb740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561116d8e9f0 0x561116da2b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:41.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.388+0000 7f7c741fb740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561116d97080 0x561116daf110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:41.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.388+0000 7f7c741fb740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f7c6c073a20 con 0x561116d97080 2024-04-02T14:46:41.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.388+0000 7f7c741fb740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f7c6c1a5000 con 0x561116d8e9f0 2024-04-02T14:46:41.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.388+0000 7f7c741fb740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f7c6c1a3b90 con 0x561116d8dd10 2024-04-02T14:46:41.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.388+0000 7f7c6aa00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561116d8dd10 0x561116d96800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:41.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.388+0000 7f7c6aa00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561116d8dd10 0x561116d96800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:48598/0 (socket says 172.21.15.90:48598) 2024-04-02T14:46:41.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.388+0000 7f7c6aa00640 1 -- 172.21.15.90:0/2596962723 learned_addr learned my addr 172.21.15.90:0/2596962723 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:46:41.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.389+0000 7f7c6a000640 1 --2- 172.21.15.90:0/2596962723 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561116d8e9f0 0x561116da2b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:41.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.389+0000 7f7c6b400640 1 --2- 172.21.15.90:0/2596962723 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561116d97080 0x561116daf110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:41.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.389+0000 7f7c6aa00640 1 -- 172.21.15.90:0/2596962723 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561116d8e9f0 msgr2=0x561116da2b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:46:41.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.389+0000 7f7c6aa00640 1 --2- 172.21.15.90:0/2596962723 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561116d8e9f0 0x561116da2b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:41.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.389+0000 7f7c6aa00640 1 -- 172.21.15.90:0/2596962723 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561116d97080 msgr2=0x561116daf110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:46:41.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.389+0000 7f7c6aa00640 1 --2- 172.21.15.90:0/2596962723 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561116d97080 0x561116daf110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:41.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.389+0000 7f7c6aa00640 1 -- 172.21.15.90:0/2596962723 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561116d84020 con 0x561116d8dd10 2024-04-02T14:46:41.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.390+0000 7f7c6b400640 1 --2- 172.21.15.90:0/2596962723 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561116d97080 0x561116daf110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:46:41.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.390+0000 7f7c6aa00640 1 --2- 172.21.15.90:0/2596962723 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561116d8dd10 0x561116d96800 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f7c5c0bac40 tx=0x7f7c5c0b94a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a049aa8d87946702 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:46:41.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.390+0000 7f7c5aa00640 1 -- 172.21.15.90:0/2596962723 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c5c0a0020 con 0x561116d8dd10 2024-04-02T14:46:41.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.390+0000 7f7c5aa00640 1 -- 172.21.15.90:0/2596962723 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7c5c0bd900 con 0x561116d8dd10 2024-04-02T14:46:41.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.390+0000 7f7c5aa00640 1 -- 172.21.15.90:0/2596962723 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c5c0a0020 con 0x561116d8dd10 2024-04-02T14:46:41.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.391+0000 7f7c741fb740 1 -- 172.21.15.90:0/2596962723 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561116d8dd10 msgr2=0x561116d96800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:41.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.391+0000 7f7c741fb740 1 --2- 172.21.15.90:0/2596962723 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561116d8dd10 0x561116d96800 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f7c5c0bac40 tx=0x7f7c5c0b94a0 comp rx=0 tx=0).stop 2024-04-02T14:46:41.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.392+0000 7f7c741fb740 1 -- 172.21.15.90:0/2596962723 shutdown_connections 2024-04-02T14:46:41.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.392+0000 7f7c741fb740 1 --2- 172.21.15.90:0/2596962723 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561116d97080 0x561116daf110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:41.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.392+0000 7f7c741fb740 1 --2- 172.21.15.90:0/2596962723 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561116d8e9f0 0x561116da2b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:41.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.392+0000 7f7c741fb740 1 --2- 172.21.15.90:0/2596962723 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561116d8dd10 0x561116d96800 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:41.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.392+0000 7f7c741fb740 1 -- 172.21.15.90:0/2596962723 >> 172.21.15.90:0/2596962723 conn(0x561116d54a50 msgr2=0x561116d929a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:46:41.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.392+0000 7f7c741fb740 1 -- 172.21.15.90:0/2596962723 shutdown_connections 2024-04-02T14:46:41.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.393+0000 7f7c741fb740 1 -- 172.21.15.90:0/2596962723 wait complete. 2024-04-02T14:46:41.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.393+0000 7f7c741fb740 1 -- start start 2024-04-02T14:46:41.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.394+0000 7f7c741fb740 1 Processor -- start 2024-04-02T14:46:41.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.395+0000 7f7c741fb740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561116d8dd10 0x561116d97a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:41.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.395+0000 7f7c741fb740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561116d8e9f0 0x561116d97fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:41.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.395+0000 7f7c741fb740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561116d97080 0x561116d9e070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:41.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.396+0000 7f7c6aa00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561116d8dd10 0x561116d97a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:41.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.396+0000 7f7c741fb740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f7c6c1a4930 con 0x561116d8e9f0 2024-04-02T14:46:41.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.396+0000 7f7c741fb740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f7c6c1a4260 con 0x561116d97080 2024-04-02T14:46:41.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.396+0000 7f7c741fb740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f7c6c071a20 con 0x561116d8dd10 2024-04-02T14:46:41.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.396+0000 7f7c741fb740 10 client.0 _ll_register_callbacks cb 0x561116cad6a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:46:41.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.396+0000 7f7c6aa00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561116d8dd10 0x561116d97a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:48612/0 (socket says 172.21.15.90:48612) 2024-04-02T14:46:41.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.396+0000 7f7c6aa00640 1 -- 172.21.15.90:0/3140547342 learned_addr learned my addr 172.21.15.90:0/3140547342 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:46:41.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.396+0000 7f7c6b400640 1 --2- 172.21.15.90:0/3140547342 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561116d97080 0x561116d9e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:41.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.396+0000 7f7c6a000640 1 --2- 172.21.15.90:0/3140547342 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561116d8e9f0 0x561116d97fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:41.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.396+0000 7f7c6aa00640 1 -- 172.21.15.90:0/3140547342 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561116d97080 msgr2=0x561116d9e070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:46:41.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.396+0000 7f7c6aa00640 1 --2- 172.21.15.90:0/3140547342 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561116d97080 0x561116d9e070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:41.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.396+0000 7f7c6aa00640 1 -- 172.21.15.90:0/3140547342 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561116d8e9f0 msgr2=0x561116d97fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:46:41.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.396+0000 7f7c6aa00640 1 --2- 172.21.15.90:0/3140547342 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561116d8e9f0 0x561116d97fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:41.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.396+0000 7f7c6aa00640 1 -- 172.21.15.90:0/3140547342 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561116d9e5b0 con 0x561116d8dd10 2024-04-02T14:46:41.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.396+0000 7f7c6b400640 1 --2- 172.21.15.90:0/3140547342 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561116d97080 0x561116d9e070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:46:41.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.396+0000 7f7c6a000640 1 --2- 172.21.15.90:0/3140547342 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561116d8e9f0 0x561116d97fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:46:41.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.397+0000 7f7c6aa00640 1 --2- 172.21.15.90:0/3140547342 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561116d8dd10 0x561116d97a80 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f7c5c0bd190 tx=0x7f7c5c0eed50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c14edb61bb11a575 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:46:41.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.397+0000 7f7c4fe00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:46:41.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.397+0000 7f7c4fe00640 1 -- 172.21.15.90:0/3140547342 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c5c0a0070 con 0x561116d8dd10 2024-04-02T14:46:41.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.398+0000 7f7c4fe00640 1 -- 172.21.15.90:0/3140547342 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 8550+0+0 (secure 0 0 0) 0x7f7c5c0c59e0 con 0x561116d8dd10 2024-04-02T14:46:41.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.398+0000 7f7c741fb740 1 -- 172.21.15.90:0/3140547342 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7c5c0bd900 con 0x561116d8dd10 2024-04-02T14:46:41.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.398+0000 7f7c4fe00640 1 -- 172.21.15.90:0/3140547342 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f7c5c0c1380 con 0x561116d8dd10 2024-04-02T14:46:41.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.399+0000 7f7c741fb740 10 client.28951 fetch_fsmap learned FSMap version 27 2024-04-02T14:46:41.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.399+0000 7f7c741fb740 1 -- 172.21.15.90:0/3140547342 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x561116d9e5b0 con 0x561116d8dd10 2024-04-02T14:46:41.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.399+0000 7f7c4fe00640 1 -- 172.21.15.90:0/3140547342 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7c5c0bd900 con 0x561116d8dd10 2024-04-02T14:46:41.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.400+0000 7f7c4fe00640 1 -- 172.21.15.90:0/3140547342 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f7c5c03c070 con 0x561116d8dd10 2024-04-02T14:46:41.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.400+0000 7f7c741fb740 10 client.28951 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:46:41.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.400+0000 7f7c741fb740 10 client.28951 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2024-04-02T14:46:41.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.400+0000 7f7c741fb740 20 client.28951 populate_metadata read hostname 'smithi090' 2024-04-02T14:46:41.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.400+0000 7f7c741fb740 1 --2- 172.21.15.90:0/3140547342 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x561116e0d9d0 0x561116e0fdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:41.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.400+0000 7f7c741fb740 4 client.28951 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-02T14:46:41.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.400+0000 7f7c741fb740 1 -- 172.21.15.90:0/3140547342 --> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] -- command(tid 0: {"prefix": "damage ls"}) -- 0x561116cffc30 con 0x561116e0d9d0 2024-04-02T14:46:41.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.401+0000 7f7c6a000640 1 --2- 172.21.15.90:0/3140547342 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x561116e0d9d0 0x561116e0fdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:41.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.402+0000 7f7c6a000640 1 --2- 172.21.15.90:0/3140547342 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x561116e0d9d0 0x561116e0fdc0 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=388cd7186acbc96 server_cookie=68ecd3c784d6020a in_seq=0 out_seq=0 2024-04-02T14:46:41.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.402+0000 7f7c4fe00640 10 client.28951 ms_handle_connect on v2:172.21.15.90:6838/1498915870 2024-04-02T14:46:41.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.405+0000 7f7c4fe00640 1 -- 172.21.15.90:0/3140547342 <== mds.1 v2:172.21.15.90:6838/1498915870 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x561116cffc30 con 0x561116e0d9d0 2024-04-02T14:46:41.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.405+0000 7f7c4fe00640 10 client.28951 handle_command_reply: tid=0 2024-04-02T14:46:41.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.405+0000 7f7c741fb740 1 client.28951 shutdown 2024-04-02T14:46:41.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.405+0000 7f7c741fb740 1 -- 172.21.15.90:0/3140547342 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561116d8dd10 msgr2=0x561116d97a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:41.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.405+0000 7f7c741fb740 1 --2- 172.21.15.90:0/3140547342 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561116d8dd10 0x561116d97a80 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f7c5c0bd190 tx=0x7f7c5c0eed50 comp rx=0 tx=0).stop 2024-04-02T14:46:41.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.405+0000 7f7c741fb740 1 -- 172.21.15.90:0/3140547342 shutdown_connections 2024-04-02T14:46:41.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.405+0000 7f7c741fb740 1 --2- 172.21.15.90:0/3140547342 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x561116e0d9d0 0x561116e0fdc0 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:41.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.405+0000 7f7c741fb740 1 --2- 172.21.15.90:0/3140547342 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x561116d97080 0x561116d9e070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:41.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.405+0000 7f7c4fe00640 0 client.28951 ms_handle_reset on v2:172.21.15.90:6838/1498915870 2024-04-02T14:46:41.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.405+0000 7f7c741fb740 1 --2- 172.21.15.90:0/3140547342 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561116d8e9f0 0x561116d97fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:41.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.405+0000 7f7c741fb740 1 --2- 172.21.15.90:0/3140547342 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561116d8dd10 0x561116d97a80 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:41.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.405+0000 7f7c741fb740 1 -- 172.21.15.90:0/3140547342 >> 172.21.15.90:0/3140547342 conn(0x561116d54a50 msgr2=0x561116d878a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:46:41.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.406+0000 7f7c741fb740 1 -- 172.21.15.90:0/3140547342 shutdown_connections 2024-04-02T14:46:41.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.406+0000 7f7c741fb740 1 -- 172.21.15.90:0/3140547342 wait complete. 2024-04-02T14:46:41.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.406+0000 7f7c741fb740 20 client.28951 trim_cache size 0 max 16384 2024-04-02T14:46:41.403 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-02T14:46:41.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.407+0000 7f7c71600640 1 -- 172.21.15.90:0/3453017319 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f7c500a3da0 msgr2=0x7f7c500a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:41.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.407+0000 7f7c71600640 1 --2- 172.21.15.90:0/3453017319 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f7c500a3da0 0x7f7c500a6260 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f7c64009fd0 tx=0x7f7c64004ab0 comp rx=0 tx=0).stop 2024-04-02T14:46:41.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.407+0000 7f7c71600640 1 -- 172.21.15.90:0/3453017319 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7c6c10d3e0 msgr2=0x7f7c6c114e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:41.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.407+0000 7f7c71600640 1 --2- 172.21.15.90:0/3453017319 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7c6c10d3e0 0x7f7c6c114e90 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f7c6c109530 tx=0x7f7c5c034670 comp rx=0 tx=0).stop 2024-04-02T14:46:41.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.408+0000 7f7c71600640 1 -- 172.21.15.90:0/3453017319 shutdown_connections 2024-04-02T14:46:41.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.408+0000 7f7c71600640 1 --2- 172.21.15.90:0/3453017319 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f7c500a3da0 0x7f7c500a6260 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:41.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.408+0000 7f7c71600640 1 --2- 172.21.15.90:0/3453017319 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7c6c111a00 0x7f7c6c1a2860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:41.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.408+0000 7f7c71600640 1 --2- 172.21.15.90:0/3453017319 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7c6c111040 0x7f7c6c1114c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:41.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.408+0000 7f7c71600640 1 --2- 172.21.15.90:0/3453017319 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7c6c10d3e0 0x7f7c6c114e90 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:41.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.408+0000 7f7c71600640 1 -- 172.21.15.90:0/3453017319 >> 172.21.15.90:0/3453017319 conn(0x7f7c6c076040 msgr2=0x7f7c6c077440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:46:41.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.408+0000 7f7c71600640 1 -- 172.21.15.90:0/3453017319 shutdown_connections 2024-04-02T14:46:41.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.408+0000 7f7c71600640 1 -- 172.21.15.90:0/3453017319 wait complete. 2024-04-02T14:46:41.430 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-02T14:46:41.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.702+0000 7f83d9000640 1 -- 172.21.15.90:0/1160863652 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f83d4108ce0 msgr2=0x7f83d41090c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:41.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.702+0000 7f83d9000640 1 --2- 172.21.15.90:0/1160863652 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f83d4108ce0 0x7f83d41090c0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f83c4009900 tx=0x7f83c402bc30 comp rx=0 tx=0).stop 2024-04-02T14:46:41.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.704+0000 7f83d9000640 1 -- 172.21.15.90:0/1160863652 shutdown_connections 2024-04-02T14:46:41.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.704+0000 7f83d9000640 1 --2- 172.21.15.90:0/1160863652 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f83d4114c50 0x7f83d4117040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:41.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.704+0000 7f83d9000640 1 --2- 172.21.15.90:0/1160863652 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f83d4109600 0x7f83d4114710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:41.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.704+0000 7f83d9000640 1 --2- 172.21.15.90:0/1160863652 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f83d4108ce0 0x7f83d41090c0 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:41.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.704+0000 7f83d9000640 1 -- 172.21.15.90:0/1160863652 >> 172.21.15.90:0/1160863652 conn(0x7f83d406cbe0 msgr2=0x7f83d406cff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:46:41.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.705+0000 7f83d9000640 1 -- 172.21.15.90:0/1160863652 shutdown_connections 2024-04-02T14:46:41.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.705+0000 7f83d9000640 1 -- 172.21.15.90:0/1160863652 wait complete. 2024-04-02T14:46:41.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.707+0000 7f83d9000640 1 Processor -- start 2024-04-02T14:46:41.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.707+0000 7f83d9000640 1 -- start start 2024-04-02T14:46:41.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.708+0000 7f83d9000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f83d4108ce0 0x7f83d41a2e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:41.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.708+0000 7f83d9000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f83d4109600 0x7f83d41a33d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:41.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.708+0000 7f83d9000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f83d4114c50 0x7f83d41a76b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:41.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.708+0000 7f83d9000640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f83d411a710 con 0x7f83d4114c50 2024-04-02T14:46:41.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.708+0000 7f83d9000640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f83d411a590 con 0x7f83d4109600 2024-04-02T14:46:41.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.708+0000 7f83d9000640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f83d411a890 con 0x7f83d4108ce0 2024-04-02T14:46:41.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.709+0000 7f83cbe00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f83d4109600 0x7f83d41a33d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:41.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.709+0000 7f83cbe00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f83d4109600 0x7f83d41a33d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:48176/0 (socket says 172.21.15.90:48176) 2024-04-02T14:46:41.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.709+0000 7f83cbe00640 1 -- 172.21.15.90:0/2658390853 learned_addr learned my addr 172.21.15.90:0/2658390853 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:46:41.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.709+0000 7f83d2800640 1 --2- 172.21.15.90:0/2658390853 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f83d4114c50 0x7f83d41a76b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:41.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.709+0000 7f83cbe00640 1 -- 172.21.15.90:0/2658390853 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f83d4108ce0 msgr2=0x7f83d41a2e90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T14:46:41.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.709+0000 7f83cbe00640 1 --2- 172.21.15.90:0/2658390853 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f83d4108ce0 0x7f83d41a2e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:41.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.709+0000 7f83cbe00640 1 -- 172.21.15.90:0/2658390853 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f83d4114c50 msgr2=0x7f83d41a76b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:41.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.709+0000 7f83cbe00640 1 --2- 172.21.15.90:0/2658390853 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f83d4114c50 0x7f83d41a76b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:41.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.709+0000 7f83cbe00640 1 -- 172.21.15.90:0/2658390853 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83d41a7de0 con 0x7f83d4109600 2024-04-02T14:46:41.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.710+0000 7f83cbe00640 1 --2- 172.21.15.90:0/2658390853 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f83d4109600 0x7f83d41a33d0 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f83bc00b4a0 tx=0x7f83bc00b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:46:41.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.711+0000 7f83caa00640 1 -- 172.21.15.90:0/2658390853 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83bc014070 con 0x7f83d4109600 2024-04-02T14:46:41.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.711+0000 7f83d9000640 1 -- 172.21.15.90:0/2658390853 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83d41a80d0 con 0x7f83d4109600 2024-04-02T14:46:41.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.711+0000 7f83d9000640 1 -- 172.21.15.90:0/2658390853 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83d41af960 con 0x7f83d4109600 2024-04-02T14:46:41.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.711+0000 7f83caa00640 1 -- 172.21.15.90:0/2658390853 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f83bc010af0 con 0x7f83d4109600 2024-04-02T14:46:41.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.711+0000 7f83caa00640 1 -- 172.21.15.90:0/2658390853 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83bc017d10 con 0x7f83d4109600 2024-04-02T14:46:41.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.719+0000 7f83caa00640 1 -- 172.21.15.90:0/2658390853 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f83bc017070 con 0x7f83d4109600 2024-04-02T14:46:41.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.721+0000 7f83caa00640 1 --2- 172.21.15.90:0/2658390853 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f83b80a3e20 0x7f83b80a62e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:41.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.721+0000 7f83caa00640 1 -- 172.21.15.90:0/2658390853 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 8550+0+0 (secure 0 0 0) 0x7f83bc0cf970 con 0x7f83d4109600 2024-04-02T14:46:41.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.722+0000 7f83dbd99740 1 Processor -- start 2024-04-02T14:46:41.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.722+0000 7f83dbd99740 1 -- start start 2024-04-02T14:46:41.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.723+0000 7f83dbd99740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5613f74dd200 0x5613f74dbf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:41.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.723+0000 7f83dbd99740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5613f74dc520 0x5613f74dc9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:41.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.723+0000 7f83dbd99740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5613f74d6150 0x5613f74f07c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:41.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.723+0000 7f83dbd99740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5613f7414400 con 0x5613f74d6150 2024-04-02T14:46:41.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.723+0000 7f83dbd99740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x5613f73f8f10 con 0x5613f74dd200 2024-04-02T14:46:41.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.723+0000 7f83dbd99740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5613f74386f0 con 0x5613f74dc520 2024-04-02T14:46:41.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.723+0000 7f83d1e00640 1 --2- 172.21.15.90:0/2658390853 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f83b80a3e20 0x7f83b80a62e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:41.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.723+0000 7f83d2800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5613f74d6150 0x5613f74f07c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:41.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.723+0000 7f83d2800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5613f74d6150 0x5613f74f07c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:40298/0 (socket says 172.21.15.90:40298) 2024-04-02T14:46:41.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.723+0000 7f83d2800640 1 -- 172.21.15.90:0/3247097891 learned_addr learned my addr 172.21.15.90:0/3247097891 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:46:41.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.724+0000 7f83d1e00640 1 --2- 172.21.15.90:0/3247097891 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5613f74dd200 0x5613f74dbf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:41.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.724+0000 7f83d2800640 1 -- 172.21.15.90:0/3247097891 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5613f74dc520 msgr2=0x5613f74dc9a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-02T14:46:41.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.724+0000 7f83d1e00640 1 --2- 172.21.15.90:0/2658390853 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f83b80a3e20 0x7f83b80a62e0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f83c40096c0 tx=0x7f83c402c7e0 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:46:41.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.724+0000 7f83cbe00640 1 --2- 172.21.15.90:0/3247097891 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5613f74dc520 0x5613f74dc9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:41.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.724+0000 7f83d2800640 1 --2- 172.21.15.90:0/3247097891 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5613f74dc520 0x5613f74dc9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:41.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.724+0000 7f83d2800640 1 -- 172.21.15.90:0/3247097891 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5613f74dd200 msgr2=0x5613f74dbf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:46:41.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.724+0000 7f83d2800640 1 --2- 172.21.15.90:0/3247097891 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5613f74dd200 0x5613f74dbf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:41.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.724+0000 7f83d2800640 1 -- 172.21.15.90:0/3247097891 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5613f714c3f0 con 0x5613f74d6150 2024-04-02T14:46:41.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.724+0000 7f83d1e00640 1 --2- 172.21.15.90:0/3247097891 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5613f74dd200 0x5613f74dbf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:46:41.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.727+0000 7f83d2800640 1 --2- 172.21.15.90:0/3247097891 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5613f74d6150 0x5613f74f07c0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f83cc00f390 tx=0x7f83cc00f850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8cff90bd08f2e9ce server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:46:41.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.729+0000 7f83c3400640 1 -- 172.21.15.90:0/3247097891 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83cc015070 con 0x5613f74d6150 2024-04-02T14:46:41.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.729+0000 7f83c3400640 1 -- 172.21.15.90:0/3247097891 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f83cc011be0 con 0x5613f74d6150 2024-04-02T14:46:41.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.730+0000 7f83dbd99740 1 -- 172.21.15.90:0/3247097891 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5613f74d6150 msgr2=0x5613f74f07c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:41.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.730+0000 7f83dbd99740 1 --2- 172.21.15.90:0/3247097891 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5613f74d6150 0x5613f74f07c0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f83cc00f390 tx=0x7f83cc00f850 comp rx=0 tx=0).stop 2024-04-02T14:46:41.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.730+0000 7f83dbd99740 1 -- 172.21.15.90:0/3247097891 shutdown_connections 2024-04-02T14:46:41.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.730+0000 7f83dbd99740 1 --2- 172.21.15.90:0/3247097891 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5613f74d6150 0x5613f74f07c0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:41.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.730+0000 7f83dbd99740 1 --2- 172.21.15.90:0/3247097891 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5613f74dc520 0x5613f74dc9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:41.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.730+0000 7f83dbd99740 1 --2- 172.21.15.90:0/3247097891 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5613f74dd200 0x5613f74dbf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:41.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.730+0000 7f83dbd99740 1 -- 172.21.15.90:0/3247097891 >> 172.21.15.90:0/3247097891 conn(0x5613f749aa50 msgr2=0x5613f74d6f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:46:41.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.732+0000 7f83dbd99740 1 -- 172.21.15.90:0/3247097891 shutdown_connections 2024-04-02T14:46:41.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.732+0000 7f83dbd99740 1 -- 172.21.15.90:0/3247097891 wait complete. 2024-04-02T14:46:41.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.733+0000 7f83dbd99740 1 -- start start 2024-04-02T14:46:41.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.733+0000 7f83dbd99740 1 Processor -- start 2024-04-02T14:46:41.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.734+0000 7f83dbd99740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5613f74d6150 0x5613f74d2f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:41.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.734+0000 7f83dbd99740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5613f74dc520 0x5613f74d34c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:41.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.734+0000 7f83dbd99740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5613f74dd200 0x5613f74d3a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:41.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.735+0000 7f83d1e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5613f74d6150 0x5613f74d2f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:41.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.735+0000 7f83d1e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5613f74d6150 0x5613f74d2f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:48192/0 (socket says 172.21.15.90:48192) 2024-04-02T14:46:41.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.735+0000 7f83cbe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5613f74dc520 0x5613f74d34c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:41.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.735+0000 7f83cbe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5613f74dc520 0x5613f74d34c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:48634/0 (socket says 172.21.15.90:48634) 2024-04-02T14:46:41.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.735+0000 7f83cbe00640 1 -- 172.21.15.90:0/3900279857 learned_addr learned my addr 172.21.15.90:0/3900279857 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:46:41.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.735+0000 7f83dbd99740 1 -- 172.21.15.90:0/3900279857 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5613f74346a0 con 0x5613f74dd200 2024-04-02T14:46:41.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.735+0000 7f83dbd99740 1 -- 172.21.15.90:0/3900279857 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x5613f73e0980 con 0x5613f74d6150 2024-04-02T14:46:41.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.735+0000 7f83dbd99740 1 -- 172.21.15.90:0/3900279857 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5613f73f3cb0 con 0x5613f74dc520 2024-04-02T14:46:41.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.735+0000 7f83dbd99740 10 client.0 _ll_register_callbacks cb 0x5613f73f36a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:46:41.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.735+0000 7f83cbe00640 1 -- 172.21.15.90:0/3900279857 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5613f74d6150 msgr2=0x5613f74d2f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:46:41.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.735+0000 7f83cbe00640 1 --2- 172.21.15.90:0/3900279857 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5613f74d6150 0x5613f74d2f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:41.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.735+0000 7f83cbe00640 1 -- 172.21.15.90:0/3900279857 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5613f74dd200 msgr2=0x5613f74d3a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:46:41.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.735+0000 7f83cbe00640 1 --2- 172.21.15.90:0/3900279857 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5613f74dd200 0x5613f74d3a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:41.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.735+0000 7f83cbe00640 1 -- 172.21.15.90:0/3900279857 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5613f72167d0 con 0x5613f74dc520 2024-04-02T14:46:41.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.736+0000 7f83cbe00640 1 --2- 172.21.15.90:0/3900279857 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5613f74dc520 0x5613f74d34c0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f83bc00b470 tx=0x7f83bc09c1e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fe9067e5faec91bd server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:46:41.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.736+0000 7f83b7e00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:46:41.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.736+0000 7f83b7e00640 1 -- 172.21.15.90:0/3900279857 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83bc01e070 con 0x5613f74dc520 2024-04-02T14:46:41.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.738+0000 7f83b7e00640 1 -- 172.21.15.90:0/3900279857 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 8550+0+0 (secure 0 0 0) 0x7f83bc0cd2c0 con 0x5613f74dc520 2024-04-02T14:46:41.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.739+0000 7f83dbd99740 1 -- 172.21.15.90:0/3900279857 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f83cc011be0 con 0x5613f74dc520 2024-04-02T14:46:41.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.740+0000 7f83b7e00640 1 -- 172.21.15.90:0/3900279857 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f83bc0b7050 con 0x5613f74dc520 2024-04-02T14:46:41.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.740+0000 7f83dbd99740 10 client.28954 fetch_fsmap learned FSMap version 27 2024-04-02T14:46:41.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.740+0000 7f83dbd99740 1 -- 172.21.15.90:0/3900279857 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5613f74fcf70 con 0x5613f74dc520 2024-04-02T14:46:41.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.740+0000 7f83b7e00640 1 -- 172.21.15.90:0/3900279857 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f83cc011be0 con 0x5613f74dc520 2024-04-02T14:46:41.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.741+0000 7f83b7e00640 1 -- 172.21.15.90:0/3900279857 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f83bc0860f0 con 0x5613f74dc520 2024-04-02T14:46:41.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.741+0000 7f83dbd99740 10 client.28954 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:46:41.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.741+0000 7f83dbd99740 10 client.28954 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:46:41.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.741+0000 7f83dbd99740 20 client.28954 populate_metadata read hostname 'smithi090' 2024-04-02T14:46:41.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.741+0000 7f83dbd99740 1 --2- 172.21.15.90:0/3900279857 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5613f74fe430 0x5613f75008c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:41.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.741+0000 7f83dbd99740 4 client.28954 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:46:41.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.741+0000 7f83dbd99740 1 -- 172.21.15.90:0/3900279857 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5613f73f8040 con 0x5613f74fe430 2024-04-02T14:46:41.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.742+0000 7f83d1e00640 1 --2- 172.21.15.90:0/3900279857 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5613f74fe430 0x5613f75008c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:41.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.743+0000 7f83d1e00640 1 --2- 172.21.15.90:0/3900279857 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5613f74fe430 0x5613f75008c0 crc :-1 s=READY pgs=306 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6a7dd23a2af4283e server_cookie=b303d20733970fd6 in_seq=0 out_seq=0 2024-04-02T14:46:41.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.743+0000 7f83b7e00640 10 client.28954 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:46:41.741 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.745+0000 7f83b7e00640 1 -- 172.21.15.90:0/3900279857 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5613f73f8040 con 0x5613f74fe430 2024-04-02T14:46:41.741 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.745+0000 7f83b7e00640 10 client.28954 handle_command_reply: tid=0 2024-04-02T14:46:41.741 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.745+0000 7f83dbd99740 1 client.28954 shutdown 2024-04-02T14:46:41.741 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.746+0000 7f83dbd99740 1 -- 172.21.15.90:0/3900279857 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5613f74dc520 msgr2=0x5613f74d34c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:41.741 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.746+0000 7f83dbd99740 1 --2- 172.21.15.90:0/3900279857 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5613f74dc520 0x5613f74d34c0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f83bc00b470 tx=0x7f83bc09c1e0 comp rx=0 tx=0).stop 2024-04-02T14:46:41.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.746+0000 7f83dbd99740 1 -- 172.21.15.90:0/3900279857 shutdown_connections 2024-04-02T14:46:41.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.746+0000 7f83dbd99740 1 --2- 172.21.15.90:0/3900279857 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5613f74fe430 0x5613f75008c0 crc :-1 s=READY pgs=306 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:41.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.746+0000 7f83dbd99740 1 --2- 172.21.15.90:0/3900279857 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5613f74dd200 0x5613f74d3a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:41.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.746+0000 7f83b7e00640 0 client.28954 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:46:41.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.746+0000 7f83dbd99740 1 --2- 172.21.15.90:0/3900279857 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5613f74dc520 0x5613f74d34c0 secure :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f83bc00b470 tx=0x7f83bc09c1e0 comp rx=0 tx=0).stop 2024-04-02T14:46:41.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.746+0000 7f83dbd99740 1 --2- 172.21.15.90:0/3900279857 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5613f74d6150 0x5613f74d2f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:41.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.746+0000 7f83dbd99740 1 -- 172.21.15.90:0/3900279857 >> 172.21.15.90:0/3900279857 conn(0x5613f749aa50 msgr2=0x5613f74d8cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:46:41.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.746+0000 7f83dbd99740 1 -- 172.21.15.90:0/3900279857 shutdown_connections 2024-04-02T14:46:41.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.746+0000 7f83dbd99740 1 -- 172.21.15.90:0/3900279857 wait complete. 2024-04-02T14:46:41.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.747+0000 7f83dbd99740 20 client.28954 trim_cache size 0 max 16384 2024-04-02T14:46:41.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.782+0000 7f83dbd99740 1 Processor -- start 2024-04-02T14:46:41.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.782+0000 7f83dbd99740 1 -- start start 2024-04-02T14:46:41.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.782+0000 7f83dbd99740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5613f74d6150 0x5613f74f9c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:41.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.782+0000 7f83dbd99740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5613f74dd200 0x5613f75061e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:41.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.783+0000 7f83dbd99740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5613f74d71b0 0x5613f7506720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:41.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.783+0000 7f83dbd99740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f83b000e7d0 con 0x5613f74d71b0 2024-04-02T14:46:41.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.783+0000 7f83dbd99740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f83b0010310 con 0x5613f74dd200 2024-04-02T14:46:41.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.783+0000 7f83dbd99740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f83b000eea0 con 0x5613f74d6150 2024-04-02T14:46:41.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.783+0000 7f83d1e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5613f74d6150 0x5613f74f9c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:41.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.783+0000 7f83d1e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5613f74d6150 0x5613f74f9c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:48644/0 (socket says 172.21.15.90:48644) 2024-04-02T14:46:41.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.783+0000 7f83d1e00640 1 -- 172.21.15.90:0/3720444887 learned_addr learned my addr 172.21.15.90:0/3720444887 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:46:41.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.783+0000 7f83cbe00640 1 --2- 172.21.15.90:0/3720444887 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5613f74dd200 0x5613f75061e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:41.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.784+0000 7f83d2800640 1 --2- 172.21.15.90:0/3720444887 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5613f74d71b0 0x5613f7506720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:41.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.784+0000 7f83d1e00640 1 -- 172.21.15.90:0/3720444887 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5613f74dd200 msgr2=0x5613f75061e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:46:41.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.784+0000 7f83d1e00640 1 --2- 172.21.15.90:0/3720444887 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5613f74dd200 0x5613f75061e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:41.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.784+0000 7f83d1e00640 1 -- 172.21.15.90:0/3720444887 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5613f74d71b0 msgr2=0x5613f7506720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:46:41.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.784+0000 7f83d1e00640 1 --2- 172.21.15.90:0/3720444887 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5613f74d71b0 0x5613f7506720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:41.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.784+0000 7f83d1e00640 1 -- 172.21.15.90:0/3720444887 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5613f74d14f0 con 0x5613f74d6150 2024-04-02T14:46:41.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.784+0000 7f83d1e00640 1 --2- 172.21.15.90:0/3720444887 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5613f74d6150 0x5613f74f9c70 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f83c4009230 tx=0x7f83c4045000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7a0ffe2877aff6a8 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:46:41.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.784+0000 7f83c2a00640 1 -- 172.21.15.90:0/3720444887 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83c4005050 con 0x5613f74d6150 2024-04-02T14:46:41.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.785+0000 7f83c2a00640 1 -- 172.21.15.90:0/3720444887 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f83c4057030 con 0x5613f74d6150 2024-04-02T14:46:41.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.785+0000 7f83c2a00640 1 -- 172.21.15.90:0/3720444887 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83c40046c0 con 0x5613f74d6150 2024-04-02T14:46:41.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.786+0000 7f83dbd99740 1 -- 172.21.15.90:0/3720444887 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5613f74d6150 msgr2=0x5613f74f9c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:41.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.787+0000 7f83dbd99740 1 --2- 172.21.15.90:0/3720444887 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5613f74d6150 0x5613f74f9c70 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f83c4009230 tx=0x7f83c4045000 comp rx=0 tx=0).stop 2024-04-02T14:46:41.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.787+0000 7f83dbd99740 1 -- 172.21.15.90:0/3720444887 shutdown_connections 2024-04-02T14:46:41.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.787+0000 7f83dbd99740 1 --2- 172.21.15.90:0/3720444887 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5613f74d71b0 0x5613f7506720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:41.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.787+0000 7f83dbd99740 1 --2- 172.21.15.90:0/3720444887 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5613f74dd200 0x5613f75061e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:41.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.787+0000 7f83dbd99740 1 --2- 172.21.15.90:0/3720444887 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5613f74d6150 0x5613f74f9c70 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:41.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.787+0000 7f83dbd99740 1 -- 172.21.15.90:0/3720444887 >> 172.21.15.90:0/3720444887 conn(0x5613f749aa50 msgr2=0x5613f74d6aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:46:41.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.789+0000 7f83dbd99740 1 -- 172.21.15.90:0/3720444887 shutdown_connections 2024-04-02T14:46:41.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.789+0000 7f83dbd99740 1 -- 172.21.15.90:0/3720444887 wait complete. 2024-04-02T14:46:41.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.789+0000 7f83dbd99740 1 -- start start 2024-04-02T14:46:41.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.790+0000 7f83dbd99740 1 Processor -- start 2024-04-02T14:46:41.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.791+0000 7f83dbd99740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5613f74d6150 0x5613f74f6180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:41.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.791+0000 7f83dbd99740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5613f74d71b0 0x5613f74f66c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:41.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.791+0000 7f83dbd99740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5613f74dd200 0x5613f74f3150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:41.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.791+0000 7f83cbe00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5613f74d71b0 0x5613f74f66c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:41.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.791+0000 7f83cbe00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5613f74d71b0 0x5613f74f66c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:48208/0 (socket says 172.21.15.90:48208) 2024-04-02T14:46:41.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.792+0000 7f83d1e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5613f74d6150 0x5613f74f6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:41.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.792+0000 7f83d1e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5613f74d6150 0x5613f74f6180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:48648/0 (socket says 172.21.15.90:48648) 2024-04-02T14:46:41.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.792+0000 7f83cbe00640 1 -- 172.21.15.90:0/3850262897 learned_addr learned my addr 172.21.15.90:0/3850262897 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:46:41.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.792+0000 7f83d2800640 1 --2- 172.21.15.90:0/3850262897 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5613f74dd200 0x5613f74f3150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:41.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.793+0000 7f83dbd99740 1 -- 172.21.15.90:0/3850262897 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f83b000fc40 con 0x5613f74dd200 2024-04-02T14:46:41.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.793+0000 7f83dbd99740 1 -- 172.21.15.90:0/3850262897 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f83b000f570 con 0x5613f74d71b0 2024-04-02T14:46:41.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.793+0000 7f83dbd99740 1 -- 172.21.15.90:0/3850262897 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f83b000c7d0 con 0x5613f74d6150 2024-04-02T14:46:41.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.793+0000 7f83cbe00640 1 -- 172.21.15.90:0/3850262897 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5613f74d6150 msgr2=0x5613f74f6180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:46:41.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.793+0000 7f83cbe00640 1 --2- 172.21.15.90:0/3850262897 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5613f74d6150 0x5613f74f6180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:41.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.794+0000 7f83cbe00640 1 -- 172.21.15.90:0/3850262897 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5613f74dd200 msgr2=0x5613f74f3150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:46:41.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.794+0000 7f83cbe00640 1 --2- 172.21.15.90:0/3850262897 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5613f74dd200 0x5613f74f3150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:41.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.794+0000 7f83cbe00640 1 -- 172.21.15.90:0/3850262897 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5613f74f3950 con 0x5613f74d71b0 2024-04-02T14:46:41.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.794+0000 7f83dbd99740 10 client.0 _ll_register_callbacks cb 0x7f83bc0092b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:46:41.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.794+0000 7f83dbd99740 1 -- 172.21.15.90:0/3850262897 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f83c4057030 con 0x5613f74d71b0 2024-04-02T14:46:41.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.795+0000 7f83cbe00640 1 --2- 172.21.15.90:0/3850262897 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5613f74d71b0 0x5613f74f66c0 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f83bc09b930 tx=0x7f83bc0047d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4c5d62045c27f7bb server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:46:41.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.795+0000 7f83b7e00640 10 client.29165 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:46:41.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.796+0000 7f83b7e00640 1 -- 172.21.15.90:0/3850262897 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83bc01e070 con 0x5613f74d71b0 2024-04-02T14:46:41.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.797+0000 7f83b7e00640 1 -- 172.21.15.90:0/3850262897 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 8550+0+0 (secure 0 0 0) 0x7f83bc09aff0 con 0x5613f74d71b0 2024-04-02T14:46:41.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.797+0000 7f83b7e00640 1 -- 172.21.15.90:0/3850262897 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f83bc09b770 con 0x5613f74d71b0 2024-04-02T14:46:41.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.797+0000 7f83dbd99740 10 client.29165 fetch_fsmap learned FSMap version 27 2024-04-02T14:46:41.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.797+0000 7f83dbd99740 1 -- 172.21.15.90:0/3850262897 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({fsmap=27}) -- 0x5613f7565290 con 0x5613f74d71b0 2024-04-02T14:46:41.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.798+0000 7f83b7e00640 1 -- 172.21.15.90:0/3850262897 <== mon.2 v2:172.21.15.92:3300/0 4 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f83bc010610 con 0x5613f74d71b0 2024-04-02T14:46:41.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.798+0000 7f83dbd99740 10 client.29165 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:46:41.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.798+0000 7f83dbd99740 10 client.29165 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:46:41.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.798+0000 7f83dbd99740 20 client.29165 populate_metadata read hostname 'smithi090' 2024-04-02T14:46:41.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.798+0000 7f83dbd99740 1 --2- 172.21.15.90:0/3850262897 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5613f7565900 0x5613f7567d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:41.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.798+0000 7f83dbd99740 4 client.29165 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-02T14:46:41.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.798+0000 7f83dbd99740 1 -- 172.21.15.90:0/3850262897 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5613f7445c30 con 0x5613f7565900 2024-04-02T14:46:41.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.799+0000 7f83d1e00640 1 --2- 172.21.15.90:0/3850262897 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5613f7565900 0x5613f7567d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:41.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.800+0000 7f83d1e00640 1 --2- 172.21.15.90:0/3850262897 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5613f7565900 0x5613f7567d20 crc :-1 s=READY pgs=307 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8d21da8f1ea2b6a5 server_cookie=1c7d2e0e4ac11a91 in_seq=0 out_seq=0 2024-04-02T14:46:41.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.800+0000 7f83b7e00640 10 client.29165 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:46:41.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.801+0000 7f83b7e00640 1 -- 172.21.15.90:0/3850262897 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5613f7445c30 con 0x5613f7565900 2024-04-02T14:46:41.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.802+0000 7f83b7e00640 10 client.29165 handle_command_reply: tid=0 2024-04-02T14:46:41.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.802+0000 7f83dbd99740 1 client.29165 shutdown 2024-04-02T14:46:41.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.802+0000 7f83dbd99740 1 -- 172.21.15.90:0/3850262897 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5613f74d71b0 msgr2=0x5613f74f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:41.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.802+0000 7f83dbd99740 1 --2- 172.21.15.90:0/3850262897 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5613f74d71b0 0x5613f74f66c0 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f83bc09b930 tx=0x7f83bc0047d0 comp rx=0 tx=0).stop 2024-04-02T14:46:41.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.802+0000 7f83dbd99740 1 -- 172.21.15.90:0/3850262897 shutdown_connections 2024-04-02T14:46:41.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.802+0000 7f83dbd99740 1 --2- 172.21.15.90:0/3850262897 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5613f7565900 0x5613f7567d20 crc :-1 s=READY pgs=307 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:41.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.802+0000 7f83dbd99740 1 --2- 172.21.15.90:0/3850262897 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5613f74dd200 0x5613f74f3150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:41.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.802+0000 7f83dbd99740 1 --2- 172.21.15.90:0/3850262897 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5613f74d71b0 0x5613f74f66c0 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:41.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.802+0000 7f83dbd99740 1 --2- 172.21.15.90:0/3850262897 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5613f74d6150 0x5613f74f6180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:41.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.802+0000 7f83dbd99740 1 -- 172.21.15.90:0/3850262897 >> 172.21.15.90:0/3850262897 conn(0x5613f749aa50 msgr2=0x5613f74d1870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:46:41.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.802+0000 7f83b7e00640 0 client.29165 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:46:41.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.802+0000 7f83dbd99740 1 -- 172.21.15.90:0/3850262897 shutdown_connections 2024-04-02T14:46:41.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.803+0000 7f83dbd99740 1 -- 172.21.15.90:0/3850262897 wait complete. 2024-04-02T14:46:41.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.803+0000 7f83dbd99740 20 client.29165 trim_cache size 0 max 16384 2024-04-02T14:46:41.800 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-02T14:46:41.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.804+0000 7f83d9000640 1 -- 172.21.15.90:0/2658390853 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f83b80a3e20 msgr2=0x7f83b80a62e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:41.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.804+0000 7f83d9000640 1 --2- 172.21.15.90:0/2658390853 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f83b80a3e20 0x7f83b80a62e0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f83c40096c0 tx=0x7f83c402c7e0 comp rx=0 tx=0).stop 2024-04-02T14:46:41.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.804+0000 7f83d9000640 1 -- 172.21.15.90:0/2658390853 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f83d4109600 msgr2=0x7f83d41a33d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:41.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.804+0000 7f83d9000640 1 --2- 172.21.15.90:0/2658390853 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f83d4109600 0x7f83d41a33d0 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f83bc00b4a0 tx=0x7f83bc00b960 comp rx=0 tx=0).stop 2024-04-02T14:46:41.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.805+0000 7f83d9000640 1 -- 172.21.15.90:0/2658390853 shutdown_connections 2024-04-02T14:46:41.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.805+0000 7f83d9000640 1 --2- 172.21.15.90:0/2658390853 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f83b80a3e20 0x7f83b80a62e0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:41.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.805+0000 7f83d9000640 1 --2- 172.21.15.90:0/2658390853 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f83d4114c50 0x7f83d41a76b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:41.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.805+0000 7f83d9000640 1 --2- 172.21.15.90:0/2658390853 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f83d4109600 0x7f83d41a33d0 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:41.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.805+0000 7f83d9000640 1 --2- 172.21.15.90:0/2658390853 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f83d4108ce0 0x7f83d41a2e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:41.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.805+0000 7f83d9000640 1 -- 172.21.15.90:0/2658390853 >> 172.21.15.90:0/2658390853 conn(0x7f83d406cbe0 msgr2=0x7f83d40705c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:46:41.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.806+0000 7f83d9000640 1 -- 172.21.15.90:0/2658390853 shutdown_connections 2024-04-02T14:46:41.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:41.806+0000 7f83d9000640 1 -- 172.21.15.90:0/2658390853 wait complete. 2024-04-02T14:46:41.825 INFO:teuthology.orchestra.run.smithi090.stderr:progress: 670.0 s, 1344.8 tps, lat 75.759 ms stddev 142.233 2024-04-02T14:46:42.604 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:46:42 smithi092 ceph-mon[36497]: pgmap v367: 97 pgs: 97 active+clean; 4.4 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 22 MiB/s wr, 1.88k op/s 2024-04-02T14:46:42.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:46:42 smithi177 ceph-mon[36431]: pgmap v367: 97 pgs: 97 active+clean; 4.4 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 22 MiB/s wr, 1.88k op/s 2024-04-02T14:46:42.682 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:46:42 smithi090 ceph-mon[31674]: pgmap v367: 97 pgs: 97 active+clean; 4.4 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 22 MiB/s wr, 1.88k op/s 2024-04-02T14:46:42.829 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-02T14:46:42.830 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T14:46:42.830 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-02T14:46:43.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.114+0000 7f0bd9600640 1 -- 172.21.15.90:0/3292790527 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0bd4100970 msgr2=0x7f0bd4100df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:43.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.114+0000 7f0bd9600640 1 --2- 172.21.15.90:0/3292790527 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0bd4100970 0x7f0bd4100df0 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f0bc0009860 tx=0x7f0bc002bb70 comp rx=0 tx=0).stop 2024-04-02T14:46:43.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.115+0000 7f0bd9600640 1 -- 172.21.15.90:0/3292790527 shutdown_connections 2024-04-02T14:46:43.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.115+0000 7f0bd9600640 1 --2- 172.21.15.90:0/3292790527 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0bd4101330 0x7f0bd410d840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:43.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.115+0000 7f0bd9600640 1 --2- 172.21.15.90:0/3292790527 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0bd4100970 0x7f0bd4100df0 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:43.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.115+0000 7f0bd9600640 1 --2- 172.21.15.90:0/3292790527 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0bd4106980 0x7f0bd4106d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:43.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.115+0000 7f0bd9600640 1 -- 172.21.15.90:0/3292790527 >> 172.21.15.90:0/3292790527 conn(0x7f0bd4077b00 msgr2=0x7f0bd4077f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:46:43.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.115+0000 7f0bd9600640 1 -- 172.21.15.90:0/3292790527 shutdown_connections 2024-04-02T14:46:43.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.115+0000 7f0bd9600640 1 -- 172.21.15.90:0/3292790527 wait complete. 2024-04-02T14:46:43.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.116+0000 7f0bd9600640 1 Processor -- start 2024-04-02T14:46:43.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.116+0000 7f0bd9600640 1 -- start start 2024-04-02T14:46:43.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.117+0000 7f0bd9600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0bd4100970 0x7f0bd410c990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:43.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.117+0000 7f0bd9600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0bd4101330 0x7f0bd410ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:43.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.117+0000 7f0bd9600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0bd4106980 0x7f0bd419ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:43.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.117+0000 7f0bd9600640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f0bd4069380 con 0x7f0bd4101330 2024-04-02T14:46:43.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.117+0000 7f0bd9600640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f0bd4069200 con 0x7f0bd4100970 2024-04-02T14:46:43.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.117+0000 7f0bd9600640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f0bd4069500 con 0x7f0bd4106980 2024-04-02T14:46:43.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.118+0000 7f0bd2a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0bd4100970 0x7f0bd410c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:43.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.118+0000 7f0bd3400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0bd4106980 0x7f0bd419ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:43.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.118+0000 7f0bd2a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0bd4100970 0x7f0bd410c990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:48222/0 (socket says 172.21.15.90:48222) 2024-04-02T14:46:43.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.118+0000 7f0bd2a00640 1 -- 172.21.15.90:0/1686528436 learned_addr learned my addr 172.21.15.90:0/1686528436 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:46:43.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.118+0000 7f0bd2000640 1 --2- 172.21.15.90:0/1686528436 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0bd4101330 0x7f0bd410ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:43.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.118+0000 7f0bd2a00640 1 -- 172.21.15.90:0/1686528436 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0bd4106980 msgr2=0x7f0bd419ee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:43.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.118+0000 7f0bd2a00640 1 --2- 172.21.15.90:0/1686528436 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0bd4106980 0x7f0bd419ee80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:43.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.118+0000 7f0bd2a00640 1 -- 172.21.15.90:0/1686528436 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0bd4101330 msgr2=0x7f0bd410ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:43.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.118+0000 7f0bd2a00640 1 --2- 172.21.15.90:0/1686528436 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0bd4101330 0x7f0bd410ced0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:43.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.118+0000 7f0bd2a00640 1 -- 172.21.15.90:0/1686528436 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0bd419f5b0 con 0x7f0bd4100970 2024-04-02T14:46:43.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.118+0000 7f0bd3400640 1 --2- 172.21.15.90:0/1686528436 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0bd4106980 0x7f0bd419ee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:46:43.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.119+0000 7f0bd2a00640 1 --2- 172.21.15.90:0/1686528436 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0bd4100970 0x7f0bd410c990 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f0bc400d950 tx=0x7f0bc400de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:46:43.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.119+0000 7f0bcb400640 1 -- 172.21.15.90:0/1686528436 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bc4017030 con 0x7f0bd4100970 2024-04-02T14:46:43.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.119+0000 7f0bd9600640 1 -- 172.21.15.90:0/1686528436 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0bd419f8a0 con 0x7f0bd4100970 2024-04-02T14:46:43.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.119+0000 7f0bcb400640 1 -- 172.21.15.90:0/1686528436 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0bc4004020 con 0x7f0bd4100970 2024-04-02T14:46:43.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.119+0000 7f0bcb400640 1 -- 172.21.15.90:0/1686528436 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bc4003d80 con 0x7f0bd4100970 2024-04-02T14:46:43.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.120+0000 7f0bd9600640 1 -- 172.21.15.90:0/1686528436 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0bd41a7130 con 0x7f0bd4100970 2024-04-02T14:46:43.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.124+0000 7f0bdc3a4740 1 Processor -- start 2024-04-02T14:46:43.120 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.124+0000 7f0bdc3a4740 1 -- start start 2024-04-02T14:46:43.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.125+0000 7f0bdc3a4740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55861f26f030 0x55861f26dd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:43.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.125+0000 7f0bdc3a4740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55861f26e350 0x55861f26e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:43.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.125+0000 7f0bdc3a4740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55861f2776c0 0x55861f279ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:43.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.125+0000 7f0bdc3a4740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55861f1d2310 con 0x55861f26f030 2024-04-02T14:46:43.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.125+0000 7f0bdc3a4740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55861f17a690 con 0x55861f2776c0 2024-04-02T14:46:43.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.125+0000 7f0bdc3a4740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55861f1ce2c0 con 0x55861f26e350 2024-04-02T14:46:43.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.125+0000 7f0bd2000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55861f2776c0 0x55861f279ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:43.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.125+0000 7f0bd2000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55861f2776c0 0x55861f279ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:48238/0 (socket says 172.21.15.90:48238) 2024-04-02T14:46:43.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.125+0000 7f0bd2000640 1 -- 172.21.15.90:0/881693869 learned_addr learned my addr 172.21.15.90:0/881693869 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:46:43.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.126+0000 7f0bd2000640 1 -- 172.21.15.90:0/881693869 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55861f26e350 msgr2=0x55861f26e7d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-02T14:46:43.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.126+0000 7f0bd2000640 1 --2- 172.21.15.90:0/881693869 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55861f26e350 0x55861f26e7d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:43.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.126+0000 7f0bd2000640 1 -- 172.21.15.90:0/881693869 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55861f26f030 msgr2=0x55861f26dd80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-02T14:46:43.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.126+0000 7f0bd3400640 1 --2- 172.21.15.90:0/881693869 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55861f26f030 0x55861f26dd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:43.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.126+0000 7f0bd2000640 1 --2- 172.21.15.90:0/881693869 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55861f26f030 0x55861f26dd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:43.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.126+0000 7f0bd2000640 1 -- 172.21.15.90:0/881693869 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55861efae7e0 con 0x55861f2776c0 2024-04-02T14:46:43.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.127+0000 7f0bd3400640 1 --2- 172.21.15.90:0/881693869 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55861f26f030 0x55861f26dd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:46:43.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.127+0000 7f0bd2000640 1 --2- 172.21.15.90:0/881693869 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55861f2776c0 0x55861f279ab0 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f0bc0005e00 tx=0x7f0bc0032340 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ddc09ba6a956e73a server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:46:43.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.127+0000 7f0bbf400640 1 -- 172.21.15.90:0/881693869 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bc0039040 con 0x55861f2776c0 2024-04-02T14:46:43.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.127+0000 7f0bbf400640 1 -- 172.21.15.90:0/881693869 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0bc0034da0 con 0x55861f2776c0 2024-04-02T14:46:43.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.127+0000 7f0bbf400640 1 -- 172.21.15.90:0/881693869 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bc002c890 con 0x55861f2776c0 2024-04-02T14:46:43.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.128+0000 7f0bcb400640 1 -- 172.21.15.90:0/1686528436 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f0bc401a070 con 0x7f0bd4100970 2024-04-02T14:46:43.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.128+0000 7f0bdc3a4740 1 -- 172.21.15.90:0/881693869 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55861f2776c0 msgr2=0x55861f279ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:43.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.129+0000 7f0bdc3a4740 1 --2- 172.21.15.90:0/881693869 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55861f2776c0 0x55861f279ab0 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f0bc0005e00 tx=0x7f0bc0032340 comp rx=0 tx=0).stop 2024-04-02T14:46:43.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.130+0000 7f0bcb400640 1 --2- 172.21.15.90:0/1686528436 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f0bb80a3d20 0x7f0bb80a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:43.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.130+0000 7f0bcb400640 1 -- 172.21.15.90:0/1686528436 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 8550+0+0 (secure 0 0 0) 0x7f0bc40cfbb0 con 0x7f0bd4100970 2024-04-02T14:46:43.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.131+0000 7f0bd3400640 1 --2- 172.21.15.90:0/1686528436 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f0bb80a3d20 0x7f0bb80a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:43.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.132+0000 7f0bdc3a4740 1 -- 172.21.15.90:0/881693869 shutdown_connections 2024-04-02T14:46:43.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.132+0000 7f0bdc3a4740 1 --2- 172.21.15.90:0/881693869 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55861f2776c0 0x55861f279ab0 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:43.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.132+0000 7f0bdc3a4740 1 --2- 172.21.15.90:0/881693869 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55861f26e350 0x55861f26e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:43.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.132+0000 7f0bdc3a4740 1 --2- 172.21.15.90:0/881693869 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55861f26f030 0x55861f26dd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:43.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.132+0000 7f0bdc3a4740 1 -- 172.21.15.90:0/881693869 >> 172.21.15.90:0/881693869 conn(0x55861f202270 msgr2=0x55861f268da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:46:43.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.132+0000 7f0bdc3a4740 1 -- 172.21.15.90:0/881693869 shutdown_connections 2024-04-02T14:46:43.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.132+0000 7f0bdc3a4740 1 -- 172.21.15.90:0/881693869 wait complete. 2024-04-02T14:46:43.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.132+0000 7f0bd3400640 1 --2- 172.21.15.90:0/1686528436 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f0bb80a3d20 0x7f0bb80a61e0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f0bcc005fd0 tx=0x7f0bcc009600 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:46:43.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.134+0000 7f0bdc3a4740 1 -- start start 2024-04-02T14:46:43.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.134+0000 7f0bdc3a4740 1 Processor -- start 2024-04-02T14:46:43.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.135+0000 7f0bdc3a4740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55861f26e350 0x55861f274720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:43.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.135+0000 7f0bdc3a4740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55861f26f030 0x55861f274c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:43.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.135+0000 7f0bdc3a4740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55861f2776c0 0x55861f2751a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:43.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.135+0000 7f0bdc3a4740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55861f282ce0 con 0x55861f26f030 2024-04-02T14:46:43.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.135+0000 7f0bdc3a4740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55861f18da60 con 0x55861f2776c0 2024-04-02T14:46:43.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.136+0000 7f0bdc3a4740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55861f282e60 con 0x55861f26e350 2024-04-02T14:46:43.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.136+0000 7f0bd2a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55861f26e350 0x55861f274720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:43.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.136+0000 7f0bd2a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55861f26e350 0x55861f274720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:48674/0 (socket says 172.21.15.90:48674) 2024-04-02T14:46:43.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.136+0000 7f0bd2a00640 1 -- 172.21.15.90:0/2973525933 learned_addr learned my addr 172.21.15.90:0/2973525933 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:46:43.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.136+0000 7f0bdc3a4740 10 client.0 _ll_register_callbacks cb 0x55861f193030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:46:43.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.136+0000 7f0bd3400640 1 --2- 172.21.15.90:0/2973525933 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55861f2776c0 0x55861f2751a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:43.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.136+0000 7f0bd2000640 1 --2- 172.21.15.90:0/2973525933 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55861f26f030 0x55861f274c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:43.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.137+0000 7f0bd2a00640 1 -- 172.21.15.90:0/2973525933 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55861f2776c0 msgr2=0x55861f2751a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:46:43.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.137+0000 7f0bd2a00640 1 --2- 172.21.15.90:0/2973525933 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55861f2776c0 0x55861f2751a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:43.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.137+0000 7f0bd2a00640 1 -- 172.21.15.90:0/2973525933 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55861f26f030 msgr2=0x55861f274c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:46:43.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.137+0000 7f0bd2a00640 1 --2- 172.21.15.90:0/2973525933 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55861f26f030 0x55861f274c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:43.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.137+0000 7f0bd2a00640 1 -- 172.21.15.90:0/2973525933 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55861f00ef70 con 0x55861f26e350 2024-04-02T14:46:43.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.137+0000 7f0bd2a00640 1 --2- 172.21.15.90:0/2973525933 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55861f26e350 0x55861f274720 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f0bc4009610 tx=0x7f0bc409b430 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6c145458a50b6230 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:46:43.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.137+0000 7f0bb3e00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:46:43.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.137+0000 7f0bb3e00640 1 -- 172.21.15.90:0/2973525933 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bc409fe80 con 0x55861f26e350 2024-04-02T14:46:43.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.137+0000 7f0bb3e00640 1 -- 172.21.15.90:0/2973525933 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 8550+0+0 (secure 0 0 0) 0x7f0bc40cd500 con 0x55861f26e350 2024-04-02T14:46:43.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.137+0000 7f0bdc3a4740 1 -- 172.21.15.90:0/2973525933 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0bc0034da0 con 0x55861f26e350 2024-04-02T14:46:43.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.138+0000 7f0bb3e00640 1 -- 172.21.15.90:0/2973525933 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f0bc40a4740 con 0x55861f26e350 2024-04-02T14:46:43.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.138+0000 7f0bdc3a4740 10 client.28969 fetch_fsmap learned FSMap version 27 2024-04-02T14:46:43.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.138+0000 7f0bdc3a4740 1 -- 172.21.15.90:0/2973525933 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55861eea6170 con 0x55861f26e350 2024-04-02T14:46:43.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.138+0000 7f0bb3e00640 1 -- 172.21.15.90:0/2973525933 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0bc0034da0 con 0x55861f26e350 2024-04-02T14:46:43.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.138+0000 7f0bb3e00640 1 -- 172.21.15.90:0/2973525933 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f0bc409af20 con 0x55861f26e350 2024-04-02T14:46:43.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.138+0000 7f0bdc3a4740 10 client.28969 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:46:43.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.138+0000 7f0bdc3a4740 10 client.28969 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:46:43.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.138+0000 7f0bdc3a4740 20 client.28969 populate_metadata read hostname 'smithi090' 2024-04-02T14:46:43.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.138+0000 7f0bdc3a4740 1 --2- 172.21.15.90:0/2973525933 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55861f2a13d0 0x55861f2a37c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:43.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.138+0000 7f0bdc3a4740 4 client.28969 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:46:43.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.138+0000 7f0bdc3a4740 1 -- 172.21.15.90:0/2973525933 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55861f1df850 con 0x55861f2a13d0 2024-04-02T14:46:43.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.139+0000 7f0bd2000640 1 --2- 172.21.15.90:0/2973525933 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55861f2a13d0 0x55861f2a37c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:43.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.140+0000 7f0bd2000640 1 --2- 172.21.15.90:0/2973525933 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55861f2a13d0 0x55861f2a37c0 crc :-1 s=READY pgs=308 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=69f9b52369e13bb6 server_cookie=a34de2ad8f85035a in_seq=0 out_seq=0 2024-04-02T14:46:43.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.140+0000 7f0bb3e00640 10 client.28969 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:46:43.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.141+0000 7f0bb3e00640 1 -- 172.21.15.90:0/2973525933 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55861f1df850 con 0x55861f2a13d0 2024-04-02T14:46:43.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.141+0000 7f0bb3e00640 10 client.28969 handle_command_reply: tid=0 2024-04-02T14:46:43.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.141+0000 7f0bdc3a4740 1 client.28969 shutdown 2024-04-02T14:46:43.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.142+0000 7f0bdc3a4740 1 -- 172.21.15.90:0/2973525933 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55861f26e350 msgr2=0x55861f274720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:43.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.142+0000 7f0bdc3a4740 1 --2- 172.21.15.90:0/2973525933 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55861f26e350 0x55861f274720 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f0bc4009610 tx=0x7f0bc409b430 comp rx=0 tx=0).stop 2024-04-02T14:46:43.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.142+0000 7f0bdc3a4740 1 -- 172.21.15.90:0/2973525933 shutdown_connections 2024-04-02T14:46:43.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.142+0000 7f0bdc3a4740 1 --2- 172.21.15.90:0/2973525933 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55861f2a13d0 0x55861f2a37c0 crc :-1 s=READY pgs=308 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:43.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.142+0000 7f0bdc3a4740 1 --2- 172.21.15.90:0/2973525933 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55861f2776c0 0x55861f2751a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:43.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.142+0000 7f0bdc3a4740 1 --2- 172.21.15.90:0/2973525933 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55861f26f030 0x55861f274c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:43.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.142+0000 7f0bdc3a4740 1 --2- 172.21.15.90:0/2973525933 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55861f26e350 0x55861f274720 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:43.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.142+0000 7f0bb3e00640 0 client.28969 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:46:43.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.142+0000 7f0bdc3a4740 1 -- 172.21.15.90:0/2973525933 >> 172.21.15.90:0/2973525933 conn(0x55861f202270 msgr2=0x55861f26aaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:46:43.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.142+0000 7f0bdc3a4740 1 -- 172.21.15.90:0/2973525933 shutdown_connections 2024-04-02T14:46:43.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.142+0000 7f0bdc3a4740 1 -- 172.21.15.90:0/2973525933 wait complete. 2024-04-02T14:46:43.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.142+0000 7f0bdc3a4740 20 client.28969 trim_cache size 0 max 16384 2024-04-02T14:46:43.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.171+0000 7f0bdc3a4740 1 Processor -- start 2024-04-02T14:46:43.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.172+0000 7f0bdc3a4740 1 -- start start 2024-04-02T14:46:43.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.172+0000 7f0bdc3a4740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55861f26e350 0x55861f276e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:43.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.172+0000 7f0bdc3a4740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55861f26f030 0x55861f267ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:43.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.172+0000 7f0bdc3a4740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55861f2776c0 0x55861f274fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:43.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.173+0000 7f0bdc3a4740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f0bd406f800 con 0x55861f26e350 2024-04-02T14:46:43.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.173+0000 7f0bdc3a4740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f0bd4071340 con 0x55861f2776c0 2024-04-02T14:46:43.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.173+0000 7f0bdc3a4740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f0bd406fed0 con 0x55861f26f030 2024-04-02T14:46:43.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.175+0000 7f0bd2a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55861f26e350 0x55861f276e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:43.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.177+0000 7f0bd2a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55861f26e350 0x55861f276e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:40350/0 (socket says 172.21.15.90:40350) 2024-04-02T14:46:43.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.177+0000 7f0bd2a00640 1 -- 172.21.15.90:0/174541618 learned_addr learned my addr 172.21.15.90:0/174541618 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:46:43.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.178+0000 7f0bd2000640 1 --2- 172.21.15.90:0/174541618 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55861f26f030 0x55861f267ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:43.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.178+0000 7f0bd2a00640 1 -- 172.21.15.90:0/174541618 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55861f26f030 msgr2=0x55861f267ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:46:43.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.179+0000 7f0bd3400640 1 --2- 172.21.15.90:0/174541618 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55861f2776c0 0x55861f274fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:43.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.180+0000 7f0bd2a00640 1 --2- 172.21.15.90:0/174541618 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55861f26f030 0x55861f267ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:43.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.180+0000 7f0bd2a00640 1 -- 172.21.15.90:0/174541618 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55861f2776c0 msgr2=0x55861f274fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:46:43.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.180+0000 7f0bd2a00640 1 --2- 172.21.15.90:0/174541618 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55861f2776c0 0x55861f274fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:43.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.180+0000 7f0bd2a00640 1 -- 172.21.15.90:0/174541618 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55861f2702e0 con 0x55861f26e350 2024-04-02T14:46:43.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.180+0000 7f0bd3400640 1 --2- 172.21.15.90:0/174541618 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55861f2776c0 0x55861f274fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:46:43.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.180+0000 7f0bd2000640 1 --2- 172.21.15.90:0/174541618 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55861f26f030 0x55861f267ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:46:43.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.181+0000 7f0bd2a00640 1 --2- 172.21.15.90:0/174541618 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55861f26e350 0x55861f276e50 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f0bc40a4900 tx=0x7f0bc409bc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e8f2c65591c4e5 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:46:43.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.181+0000 7f0bbea00640 1 -- 172.21.15.90:0/174541618 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bc4014070 con 0x55861f26e350 2024-04-02T14:46:43.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.182+0000 7f0bbea00640 1 -- 172.21.15.90:0/174541618 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0bc409ad30 con 0x55861f26e350 2024-04-02T14:46:43.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.182+0000 7f0bbea00640 1 -- 172.21.15.90:0/174541618 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bc40041d0 con 0x55861f26e350 2024-04-02T14:46:43.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.183+0000 7f0bdc3a4740 1 -- 172.21.15.90:0/174541618 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55861f26e350 msgr2=0x55861f276e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:43.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.183+0000 7f0bdc3a4740 1 --2- 172.21.15.90:0/174541618 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55861f26e350 0x55861f276e50 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f0bc40a4900 tx=0x7f0bc409bc50 comp rx=0 tx=0).stop 2024-04-02T14:46:43.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.183+0000 7f0bdc3a4740 1 -- 172.21.15.90:0/174541618 shutdown_connections 2024-04-02T14:46:43.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.183+0000 7f0bdc3a4740 1 --2- 172.21.15.90:0/174541618 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55861f2776c0 0x55861f274fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:43.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.183+0000 7f0bdc3a4740 1 --2- 172.21.15.90:0/174541618 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55861f26f030 0x55861f267ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:43.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.183+0000 7f0bdc3a4740 1 --2- 172.21.15.90:0/174541618 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55861f26e350 0x55861f276e50 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:43.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.183+0000 7f0bdc3a4740 1 -- 172.21.15.90:0/174541618 >> 172.21.15.90:0/174541618 conn(0x55861f202270 msgr2=0x55861f267590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:46:43.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.183+0000 7f0bdc3a4740 1 -- 172.21.15.90:0/174541618 shutdown_connections 2024-04-02T14:46:43.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.183+0000 7f0bdc3a4740 1 -- 172.21.15.90:0/174541618 wait complete. 2024-04-02T14:46:43.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.184+0000 7f0bdc3a4740 1 -- start start 2024-04-02T14:46:43.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.184+0000 7f0bdc3a4740 1 Processor -- start 2024-04-02T14:46:43.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.185+0000 7f0bdc3a4740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55861f26f030 0x55861f288640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:43.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.185+0000 7f0bdc3a4740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55861f2776c0 0x55861f2830e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:43.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.186+0000 7f0bdc3a4740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55861f283780 0x55861f3078d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:43.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.186+0000 7f0bd2a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55861f26f030 0x55861f288640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:43.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.186+0000 7f0bd2a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55861f26f030 0x55861f288640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:48260/0 (socket says 172.21.15.90:48260) 2024-04-02T14:46:43.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.186+0000 7f0bdc3a4740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f0bd4070c70 con 0x55861f283780 2024-04-02T14:46:43.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.186+0000 7f0bdc3a4740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f0bd40705a0 con 0x55861f26f030 2024-04-02T14:46:43.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.186+0000 7f0bdc3a4740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f0bd406d820 con 0x55861f2776c0 2024-04-02T14:46:43.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.186+0000 7f0bd2a00640 1 -- 172.21.15.90:0/2355812170 learned_addr learned my addr 172.21.15.90:0/2355812170 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:46:43.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.186+0000 7f0bdc3a4740 10 client.0 _ll_register_callbacks cb 0x7f0bc4003f20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:46:43.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.187+0000 7f0bd2000640 1 --2- 172.21.15.90:0/2355812170 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55861f2776c0 0x55861f2830e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:43.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.187+0000 7f0bd3400640 1 --2- 172.21.15.90:0/2355812170 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55861f283780 0x55861f3078d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:43.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.188+0000 7f0bd2000640 1 -- 172.21.15.90:0/2355812170 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55861f26f030 msgr2=0x55861f288640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:46:43.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.188+0000 7f0bd2000640 1 --2- 172.21.15.90:0/2355812170 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55861f26f030 0x55861f288640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:43.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.188+0000 7f0bd2000640 1 -- 172.21.15.90:0/2355812170 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55861f283780 msgr2=0x55861f3078d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:46:43.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.188+0000 7f0bd2000640 1 --2- 172.21.15.90:0/2355812170 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55861f283780 0x55861f3078d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:43.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.188+0000 7f0bd2000640 1 -- 172.21.15.90:0/2355812170 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55861f283ef0 con 0x55861f2776c0 2024-04-02T14:46:43.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.188+0000 7f0bd3400640 1 --2- 172.21.15.90:0/2355812170 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55861f283780 0x55861f3078d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:46:43.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.189+0000 7f0bb3e00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:46:43.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.189+0000 7f0bd2000640 1 --2- 172.21.15.90:0/2355812170 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55861f2776c0 0x55861f2830e0 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f0bc002c0b0 tx=0x7f0bc00349b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6d702e248a0bbc8d server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:46:43.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.190+0000 7f0bb3e00640 1 -- 172.21.15.90:0/2355812170 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bc0039040 con 0x55861f2776c0 2024-04-02T14:46:43.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.190+0000 7f0bb3e00640 1 -- 172.21.15.90:0/2355812170 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 8550+0+0 (secure 0 0 0) 0x7f0bc0034540 con 0x55861f2776c0 2024-04-02T14:46:43.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.193+0000 7f0bdc3a4740 1 -- 172.21.15.90:0/2355812170 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0bc409ad30 con 0x55861f2776c0 2024-04-02T14:46:43.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.193+0000 7f0bb3e00640 1 -- 172.21.15.90:0/2355812170 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f0bc0055050 con 0x55861f2776c0 2024-04-02T14:46:43.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.193+0000 7f0bdc3a4740 10 client.28972 fetch_fsmap learned FSMap version 27 2024-04-02T14:46:43.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.193+0000 7f0bdc3a4740 1 -- 172.21.15.90:0/2355812170 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55861f308df0 con 0x55861f2776c0 2024-04-02T14:46:43.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.194+0000 7f0bb3e00640 1 -- 172.21.15.90:0/2355812170 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0bc409ad30 con 0x55861f2776c0 2024-04-02T14:46:43.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.194+0000 7f0bb3e00640 1 -- 172.21.15.90:0/2355812170 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f0bc0005030 con 0x55861f2776c0 2024-04-02T14:46:43.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.194+0000 7f0bdc3a4740 10 client.28972 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:46:43.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.194+0000 7f0bdc3a4740 10 client.28972 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:46:43.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.194+0000 7f0bdc3a4740 20 client.28972 populate_metadata read hostname 'smithi090' 2024-04-02T14:46:43.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.194+0000 7f0bdc3a4740 1 --2- 172.21.15.90:0/2355812170 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55861f3090e0 0x55861f30b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:43.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.194+0000 7f0bdc3a4740 4 client.28972 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-02T14:46:43.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.194+0000 7f0bdc3a4740 1 -- 172.21.15.90:0/2355812170 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55861f1d9ba0 con 0x55861f3090e0 2024-04-02T14:46:43.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.195+0000 7f0bd2a00640 1 --2- 172.21.15.90:0/2355812170 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55861f3090e0 0x55861f30b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:43.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.196+0000 7f0bd2a00640 1 --2- 172.21.15.90:0/2355812170 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55861f3090e0 0x55861f30b4d0 crc :-1 s=READY pgs=309 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eee1dd99a1b3620a server_cookie=599baa588f2be998 in_seq=0 out_seq=0 2024-04-02T14:46:43.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.196+0000 7f0bb3e00640 10 client.28972 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:46:43.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.197+0000 7f0bb3e00640 1 -- 172.21.15.90:0/2355812170 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55861f1d9ba0 con 0x55861f3090e0 2024-04-02T14:46:43.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.197+0000 7f0bb3e00640 10 client.28972 handle_command_reply: tid=0 2024-04-02T14:46:43.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.198+0000 7f0bdc3a4740 1 client.28972 shutdown 2024-04-02T14:46:43.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.198+0000 7f0bdc3a4740 1 -- 172.21.15.90:0/2355812170 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55861f2776c0 msgr2=0x55861f2830e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:43.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.199+0000 7f0bdc3a4740 1 --2- 172.21.15.90:0/2355812170 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55861f2776c0 0x55861f2830e0 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f0bc002c0b0 tx=0x7f0bc00349b0 comp rx=0 tx=0).stop 2024-04-02T14:46:43.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.199+0000 7f0bdc3a4740 1 -- 172.21.15.90:0/2355812170 shutdown_connections 2024-04-02T14:46:43.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.199+0000 7f0bdc3a4740 1 --2- 172.21.15.90:0/2355812170 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55861f3090e0 0x55861f30b4d0 crc :-1 s=READY pgs=309 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:43.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.199+0000 7f0bdc3a4740 1 --2- 172.21.15.90:0/2355812170 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55861f283780 0x55861f3078d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:43.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.199+0000 7f0bdc3a4740 1 --2- 172.21.15.90:0/2355812170 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55861f2776c0 0x55861f2830e0 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:43.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.199+0000 7f0bdc3a4740 1 --2- 172.21.15.90:0/2355812170 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55861f26f030 0x55861f288640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:43.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.199+0000 7f0bb3e00640 0 client.28972 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:46:43.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.199+0000 7f0bdc3a4740 1 -- 172.21.15.90:0/2355812170 >> 172.21.15.90:0/2355812170 conn(0x55861f202270 msgr2=0x55861f267e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:46:43.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.200+0000 7f0bdc3a4740 1 -- 172.21.15.90:0/2355812170 shutdown_connections 2024-04-02T14:46:43.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.201+0000 7f0bdc3a4740 1 -- 172.21.15.90:0/2355812170 wait complete. 2024-04-02T14:46:43.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.201+0000 7f0bdc3a4740 20 client.28972 trim_cache size 0 max 16384 2024-04-02T14:46:43.197 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-02T14:46:43.197 INFO:teuthology.orchestra.run.smithi090.stdout: "return_code": 0, 2024-04-02T14:46:43.197 INFO:teuthology.orchestra.run.smithi090.stdout: "scrub_tag": "14ff3309-2615-4b00-935d-0f067b927c5d", 2024-04-02T14:46:43.197 INFO:teuthology.orchestra.run.smithi090.stdout: "mode": "asynchronous" 2024-04-02T14:46:43.197 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-02T14:46:43.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.202+0000 7f0bd9600640 1 -- 172.21.15.90:0/1686528436 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f0bb80a3d20 msgr2=0x7f0bb80a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:43.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.202+0000 7f0bd9600640 1 --2- 172.21.15.90:0/1686528436 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f0bb80a3d20 0x7f0bb80a61e0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f0bcc005fd0 tx=0x7f0bcc009600 comp rx=0 tx=0).stop 2024-04-02T14:46:43.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.202+0000 7f0bd9600640 1 -- 172.21.15.90:0/1686528436 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0bd4100970 msgr2=0x7f0bd410c990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:43.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.202+0000 7f0bd9600640 1 --2- 172.21.15.90:0/1686528436 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0bd4100970 0x7f0bd410c990 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f0bc400d950 tx=0x7f0bc400de10 comp rx=0 tx=0).stop 2024-04-02T14:46:43.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.202+0000 7f0bd9600640 1 -- 172.21.15.90:0/1686528436 shutdown_connections 2024-04-02T14:46:43.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.202+0000 7f0bd9600640 1 --2- 172.21.15.90:0/1686528436 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f0bb80a3d20 0x7f0bb80a61e0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:43.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.202+0000 7f0bd9600640 1 --2- 172.21.15.90:0/1686528436 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0bd4106980 0x7f0bd419ee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:43.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.202+0000 7f0bd9600640 1 --2- 172.21.15.90:0/1686528436 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0bd4101330 0x7f0bd410ced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:43.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.202+0000 7f0bd9600640 1 --2- 172.21.15.90:0/1686528436 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0bd4100970 0x7f0bd410c990 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:43.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.202+0000 7f0bd9600640 1 -- 172.21.15.90:0/1686528436 >> 172.21.15.90:0/1686528436 conn(0x7f0bd4077b00 msgr2=0x7f0bd4110e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:46:43.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.202+0000 7f0bd9600640 1 -- 172.21.15.90:0/1686528436 shutdown_connections 2024-04-02T14:46:43.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.202+0000 7f0bd9600640 1 -- 172.21.15.90:0/1686528436 wait complete. 2024-04-02T14:46:43.236 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-02T14:46:43.237 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-02T14:46:43.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.512+0000 7fabebc00640 1 -- 172.21.15.90:0/3264623978 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fabe4101a90 msgr2=0x7fabe4101f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:43.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.512+0000 7fabebc00640 1 --2- 172.21.15.90:0/3264623978 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fabe4101a90 0x7fabe4101f10 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7fabd0009930 tx=0x7fabd002bc20 comp rx=0 tx=0).stop 2024-04-02T14:46:43.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.512+0000 7fabebc00640 1 -- 172.21.15.90:0/3264623978 shutdown_connections 2024-04-02T14:46:43.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.512+0000 7fabebc00640 1 --2- 172.21.15.90:0/3264623978 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fabe4102450 0x7fabe410e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:43.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.512+0000 7fabebc00640 1 --2- 172.21.15.90:0/3264623978 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fabe4101a90 0x7fabe4101f10 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:43.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.512+0000 7fabebc00640 1 --2- 172.21.15.90:0/3264623978 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fabe4107aa0 0x7fabe4107e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:43.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.512+0000 7fabebc00640 1 -- 172.21.15.90:0/3264623978 >> 172.21.15.90:0/3264623978 conn(0x7fabe4078f00 msgr2=0x7fabe4079310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:46:43.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.513+0000 7fabebc00640 1 -- 172.21.15.90:0/3264623978 shutdown_connections 2024-04-02T14:46:43.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.513+0000 7fabebc00640 1 -- 172.21.15.90:0/3264623978 wait complete. 2024-04-02T14:46:43.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.514+0000 7fabebc00640 1 Processor -- start 2024-04-02T14:46:43.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.514+0000 7fabebc00640 1 -- start start 2024-04-02T14:46:43.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.514+0000 7fabebc00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fabe4101a90 0x7fabe4196240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:43.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.514+0000 7fabebc00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fabe4102450 0x7fabe4196780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:43.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.515+0000 7fabe9200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fabe4101a90 0x7fabe4196240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:43.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.514+0000 7fabebc00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fabe4107aa0 0x7fabe419aab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:43.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.515+0000 7fabe9200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fabe4101a90 0x7fabe4196240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:48276/0 (socket says 172.21.15.90:48276) 2024-04-02T14:46:43.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.515+0000 7fabebc00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fabe410fd00 con 0x7fabe4107aa0 2024-04-02T14:46:43.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.515+0000 7fabebc00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fabe410fb80 con 0x7fabe4101a90 2024-04-02T14:46:43.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.515+0000 7fabebc00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fabe410fe80 con 0x7fabe4102450 2024-04-02T14:46:43.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.515+0000 7fabe9200640 1 -- 172.21.15.90:0/972990088 learned_addr learned my addr 172.21.15.90:0/972990088 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:46:43.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.515+0000 7fabe9c00640 1 --2- 172.21.15.90:0/972990088 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fabe4107aa0 0x7fabe419aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:43.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.515+0000 7fabe3e00640 1 --2- 172.21.15.90:0/972990088 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fabe4102450 0x7fabe4196780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:43.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.516+0000 7fabe3e00640 1 -- 172.21.15.90:0/972990088 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fabe4101a90 msgr2=0x7fabe4196240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:43.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.516+0000 7fabe3e00640 1 --2- 172.21.15.90:0/972990088 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fabe4101a90 0x7fabe4196240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:43.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.516+0000 7fabe3e00640 1 -- 172.21.15.90:0/972990088 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fabe4107aa0 msgr2=0x7fabe419aab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:43.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.516+0000 7fabe3e00640 1 --2- 172.21.15.90:0/972990088 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fabe4107aa0 0x7fabe419aab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:43.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.516+0000 7fabe3e00640 1 -- 172.21.15.90:0/972990088 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fabe419b1e0 con 0x7fabe4102450 2024-04-02T14:46:43.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.516+0000 7fabe9200640 1 --2- 172.21.15.90:0/972990088 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fabe4101a90 0x7fabe4196240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:46:43.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.517+0000 7fabe3e00640 1 --2- 172.21.15.90:0/972990088 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fabe4102450 0x7fabe4196780 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7fabd0036040 tx=0x7fabd00043b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:46:43.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.517+0000 7fabe1600640 1 -- 172.21.15.90:0/972990088 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabd0043020 con 0x7fabe4102450 2024-04-02T14:46:43.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.517+0000 7fabe1600640 1 -- 172.21.15.90:0/972990088 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fabd00054e0 con 0x7fabe4102450 2024-04-02T14:46:43.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.517+0000 7fabe1600640 1 -- 172.21.15.90:0/972990088 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabd0003a80 con 0x7fabe4102450 2024-04-02T14:46:43.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.517+0000 7fabebc00640 1 -- 172.21.15.90:0/972990088 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fabe419b470 con 0x7fabe4102450 2024-04-02T14:46:43.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.517+0000 7fabebc00640 1 -- 172.21.15.90:0/972990088 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fabe41a2d60 con 0x7fabe4102450 2024-04-02T14:46:43.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.522+0000 7fabe1600640 1 -- 172.21.15.90:0/972990088 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fabd0046020 con 0x7fabe4102450 2024-04-02T14:46:43.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.523+0000 7fabee970740 1 Processor -- start 2024-04-02T14:46:43.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.523+0000 7fabee970740 1 -- start start 2024-04-02T14:46:43.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.523+0000 7fabee970740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cb49e537b0 0x55cb49e53b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:43.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.523+0000 7fabee970740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55cb49e540d0 0x55cb49e5c770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:43.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.523+0000 7fabee970740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cb49e5ccb0 0x55cb49e5f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:43.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.523+0000 7fabee970740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55cb49db7770 con 0x55cb49e5ccb0 2024-04-02T14:46:43.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.523+0000 7fabee970740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55cb49d5f700 con 0x55cb49e540d0 2024-04-02T14:46:43.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.523+0000 7fabee970740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55cb49db3720 con 0x55cb49e537b0 2024-04-02T14:46:43.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.523+0000 7fabe9c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cb49e537b0 0x55cb49e53b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:43.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.523+0000 7fabe9200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55cb49e540d0 0x55cb49e5c770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:43.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.523+0000 7fabe9c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cb49e537b0 0x55cb49e53b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:48728/0 (socket says 172.21.15.90:48728) 2024-04-02T14:46:43.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.524+0000 7fabe9c00640 1 -- 172.21.15.90:0/3412614463 learned_addr learned my addr 172.21.15.90:0/3412614463 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:46:43.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.524+0000 7fabe3e00640 1 --2- 172.21.15.90:0/3412614463 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cb49e5ccb0 0x55cb49e5f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:43.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.524+0000 7fabe9c00640 1 -- 172.21.15.90:0/3412614463 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55cb49e540d0 msgr2=0x55cb49e5c770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:46:43.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.524+0000 7fabe9c00640 1 --2- 172.21.15.90:0/3412614463 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55cb49e540d0 0x55cb49e5c770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:43.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.524+0000 7fabe9c00640 1 -- 172.21.15.90:0/3412614463 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cb49e5ccb0 msgr2=0x55cb49e5f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:46:43.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.524+0000 7fabe9c00640 1 --2- 172.21.15.90:0/3412614463 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cb49e5ccb0 0x55cb49e5f0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:43.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.524+0000 7fabe9c00640 1 -- 172.21.15.90:0/3412614463 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cb49b957d0 con 0x55cb49e537b0 2024-04-02T14:46:43.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.524+0000 7fabe3e00640 1 --2- 172.21.15.90:0/3412614463 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cb49e5ccb0 0x55cb49e5f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:46:43.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.525+0000 7fabe9c00640 1 --2- 172.21.15.90:0/3412614463 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cb49e537b0 0x55cb49e53b90 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7fabdc009fd0 tx=0x7fabdc0118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=26dafa6cc9729404 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:46:43.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.525+0000 7fabd6000640 1 -- 172.21.15.90:0/3412614463 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabdc01e070 con 0x55cb49e537b0 2024-04-02T14:46:43.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.525+0000 7fabd6000640 1 -- 172.21.15.90:0/3412614463 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fabdc003c30 con 0x55cb49e537b0 2024-04-02T14:46:43.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.525+0000 7fabd6000640 1 -- 172.21.15.90:0/3412614463 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabdc00f9e0 con 0x55cb49e537b0 2024-04-02T14:46:43.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.525+0000 7fabe1600640 1 --2- 172.21.15.90:0/972990088 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fabcc0a81f0 0x7fabcc0aa6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:43.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.526+0000 7fabe1600640 1 -- 172.21.15.90:0/972990088 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 8550+0+0 (secure 0 0 0) 0x7fabd00f2600 con 0x7fabe4102450 2024-04-02T14:46:43.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.526+0000 7fabee970740 1 -- 172.21.15.90:0/3412614463 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cb49e537b0 msgr2=0x55cb49e53b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:43.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.526+0000 7fabee970740 1 --2- 172.21.15.90:0/3412614463 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cb49e537b0 0x55cb49e53b90 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7fabdc009fd0 tx=0x7fabdc0118d0 comp rx=0 tx=0).stop 2024-04-02T14:46:43.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.526+0000 7fabee970740 1 -- 172.21.15.90:0/3412614463 shutdown_connections 2024-04-02T14:46:43.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.526+0000 7fabee970740 1 --2- 172.21.15.90:0/3412614463 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cb49e5ccb0 0x55cb49e5f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:43.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.526+0000 7fabee970740 1 --2- 172.21.15.90:0/3412614463 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55cb49e540d0 0x55cb49e5c770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:43.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.526+0000 7fabee970740 1 --2- 172.21.15.90:0/3412614463 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cb49e537b0 0x55cb49e53b90 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:43.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.526+0000 7fabee970740 1 -- 172.21.15.90:0/3412614463 >> 172.21.15.90:0/3412614463 conn(0x55cb49e4dee0 msgr2=0x55cb49e4e2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:46:43.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.526+0000 7fabe9200640 1 --2- 172.21.15.90:0/972990088 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fabcc0a81f0 0x7fabcc0aa6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:43.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.526+0000 7fabee970740 1 -- 172.21.15.90:0/3412614463 shutdown_connections 2024-04-02T14:46:43.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.526+0000 7fabee970740 1 -- 172.21.15.90:0/3412614463 wait complete. 2024-04-02T14:46:43.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.527+0000 7fabee970740 1 -- start start 2024-04-02T14:46:43.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.527+0000 7fabe9200640 1 --2- 172.21.15.90:0/972990088 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fabcc0a81f0 0x7fabcc0aa6b0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x55cb49e53280 tx=0x7fabd8005f90 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:46:43.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.527+0000 7fabee970740 1 Processor -- start 2024-04-02T14:46:43.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.528+0000 7fabee970740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cb49e537b0 0x55cb49e59c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:43.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.528+0000 7fabee970740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55cb49e540d0 0x55cb49e5a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:43.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.528+0000 7fabee970740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cb49e5ccb0 0x55cb49e5a710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:43.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.528+0000 7fabee970740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55cb49e5faa0 con 0x55cb49e537b0 2024-04-02T14:46:43.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.528+0000 7fabee970740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55cb49d72ac0 con 0x55cb49e540d0 2024-04-02T14:46:43.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.528+0000 7fabee970740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55cb49e5fc20 con 0x55cb49e5ccb0 2024-04-02T14:46:43.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.528+0000 7fabee970740 10 client.0 _ll_register_callbacks cb 0x55cb49d75180 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:46:43.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.528+0000 7fabe3e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55cb49e540d0 0x55cb49e5a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:43.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.528+0000 7fabe9c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cb49e5ccb0 0x55cb49e5a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:43.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.528+0000 7fabe9c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cb49e5ccb0 0x55cb49e5a710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:48736/0 (socket says 172.21.15.90:48736) 2024-04-02T14:46:43.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.528+0000 7fabe9c00640 1 -- 172.21.15.90:0/1140070623 learned_addr learned my addr 172.21.15.90:0/1140070623 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:46:43.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.528+0000 7fabe9200640 1 --2- 172.21.15.90:0/1140070623 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cb49e537b0 0x55cb49e59c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:43.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.530+0000 7fabe9200640 1 -- 172.21.15.90:0/1140070623 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cb49e5ccb0 msgr2=0x55cb49e5a710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:46:43.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.530+0000 7fabe9200640 1 --2- 172.21.15.90:0/1140070623 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cb49e5ccb0 0x55cb49e5a710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:43.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.530+0000 7fabe9200640 1 -- 172.21.15.90:0/1140070623 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55cb49e540d0 msgr2=0x55cb49e5a1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:46:43.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.530+0000 7fabe9200640 1 --2- 172.21.15.90:0/1140070623 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55cb49e540d0 0x55cb49e5a1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:43.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.530+0000 7fabe9200640 1 -- 172.21.15.90:0/1140070623 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cb49acb3f0 con 0x55cb49e537b0 2024-04-02T14:46:43.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.531+0000 7fabe9200640 1 --2- 172.21.15.90:0/1140070623 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cb49e537b0 0x55cb49e59c90 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fabd8009620 tx=0x7fabd8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=959b29b4a533d0ad server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:46:43.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.531+0000 7fabcaa00640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-02T14:46:43.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.531+0000 7fabcaa00640 1 -- 172.21.15.90:0/1140070623 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabd8003dc0 con 0x55cb49e537b0 2024-04-02T14:46:43.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.531+0000 7fabcaa00640 1 -- 172.21.15.90:0/1140070623 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 8550+0+0 (secure 0 0 0) 0x7fabd800fa60 con 0x55cb49e537b0 2024-04-02T14:46:43.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.531+0000 7fabee970740 1 -- 172.21.15.90:0/1140070623 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fabdc003c30 con 0x55cb49e537b0 2024-04-02T14:46:43.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.532+0000 7fabcaa00640 1 -- 172.21.15.90:0/1140070623 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fabd800fc60 con 0x55cb49e537b0 2024-04-02T14:46:43.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.532+0000 7fabee970740 10 client.18120 fetch_fsmap learned FSMap version 27 2024-04-02T14:46:43.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.532+0000 7fabee970740 1 -- 172.21.15.90:0/1140070623 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55cb49a8b0f0 con 0x55cb49e537b0 2024-04-02T14:46:43.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.532+0000 7fabcaa00640 1 -- 172.21.15.90:0/1140070623 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fabdc003c30 con 0x55cb49e537b0 2024-04-02T14:46:43.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.533+0000 7fabcaa00640 1 -- 172.21.15.90:0/1140070623 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fabd8027020 con 0x55cb49e537b0 2024-04-02T14:46:43.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.533+0000 7fabee970740 10 client.18120 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:46:43.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.533+0000 7fabee970740 10 client.18120 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:46:43.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.533+0000 7fabee970740 20 client.18120 populate_metadata read hostname 'smithi090' 2024-04-02T14:46:43.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.533+0000 7fabee970740 1 --2- 172.21.15.90:0/1140070623 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55cb49e75520 0x55cb49e77930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:43.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.533+0000 7fabee970740 4 client.18120 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:46:43.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.533+0000 7fabee970740 1 -- 172.21.15.90:0/1140070623 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cb49d58630 con 0x55cb49e75520 2024-04-02T14:46:43.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.534+0000 7fabe3e00640 1 --2- 172.21.15.90:0/1140070623 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55cb49e75520 0x55cb49e77930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:43.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.534+0000 7fabcaa00640 10 client.18120 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:46:43.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.534+0000 7fabe3e00640 1 --2- 172.21.15.90:0/1140070623 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55cb49e75520 0x55cb49e77930 crc :-1 s=READY pgs=310 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ae77447676852de7 server_cookie=9ce333ff1d4b32d4 in_seq=0 out_seq=0 2024-04-02T14:46:43.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.535+0000 7fabcaa00640 1 -- 172.21.15.90:0/1140070623 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55cb49d58630 con 0x55cb49e75520 2024-04-02T14:46:43.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.535+0000 7fabcaa00640 10 client.18120 handle_command_reply: tid=0 2024-04-02T14:46:43.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.535+0000 7fabee970740 1 client.18120 shutdown 2024-04-02T14:46:43.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.535+0000 7fabee970740 1 -- 172.21.15.90:0/1140070623 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cb49e537b0 msgr2=0x55cb49e59c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:43.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.535+0000 7fabee970740 1 --2- 172.21.15.90:0/1140070623 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cb49e537b0 0x55cb49e59c90 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fabd8009620 tx=0x7fabd8004730 comp rx=0 tx=0).stop 2024-04-02T14:46:43.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.535+0000 7fabee970740 1 -- 172.21.15.90:0/1140070623 shutdown_connections 2024-04-02T14:46:43.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.535+0000 7fabee970740 1 --2- 172.21.15.90:0/1140070623 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55cb49e75520 0x55cb49e77930 crc :-1 s=READY pgs=310 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:43.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.535+0000 7fabee970740 1 --2- 172.21.15.90:0/1140070623 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cb49e5ccb0 0x55cb49e5a710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:43.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.535+0000 7fabee970740 1 --2- 172.21.15.90:0/1140070623 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55cb49e540d0 0x55cb49e5a1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:43.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.535+0000 7fabee970740 1 --2- 172.21.15.90:0/1140070623 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cb49e537b0 0x55cb49e59c90 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:43.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.535+0000 7fabee970740 1 -- 172.21.15.90:0/1140070623 >> 172.21.15.90:0/1140070623 conn(0x55cb49e4dee0 msgr2=0x55cb49e4fc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:46:43.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.536+0000 7fabcaa00640 0 client.18120 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:46:43.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.536+0000 7fabee970740 1 -- 172.21.15.90:0/1140070623 shutdown_connections 2024-04-02T14:46:43.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.536+0000 7fabee970740 1 -- 172.21.15.90:0/1140070623 wait complete. 2024-04-02T14:46:43.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.536+0000 7fabee970740 20 client.18120 trim_cache size 0 max 16384 2024-04-02T14:46:43.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.564+0000 7fabee970740 1 Processor -- start 2024-04-02T14:46:43.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.564+0000 7fabee970740 1 -- start start 2024-04-02T14:46:43.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.564+0000 7fabee970740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cb49e537b0 0x55cb49e5c3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:43.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.564+0000 7fabee970740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55cb49e540d0 0x55cb49e689c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:43.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.564+0000 7fabee970740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cb49e5ccb0 0x55cb49e58e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:43.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.564+0000 7fabee970740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fabe41ab720 con 0x55cb49e537b0 2024-04-02T14:46:43.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.564+0000 7fabee970740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fabe41ad260 con 0x55cb49e540d0 2024-04-02T14:46:43.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.564+0000 7fabee970740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fabe41abdf0 con 0x55cb49e5ccb0 2024-04-02T14:46:43.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.564+0000 7fabe3e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55cb49e540d0 0x55cb49e689c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:43.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.564+0000 7fabe9c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cb49e5ccb0 0x55cb49e58e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:43.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.564+0000 7fabe9200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cb49e537b0 0x55cb49e5c3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:43.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.564+0000 7fabe3e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55cb49e540d0 0x55cb49e689c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:48312/0 (socket says 172.21.15.90:48312) 2024-04-02T14:46:43.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.564+0000 7fabe9c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cb49e5ccb0 0x55cb49e58e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:48746/0 (socket says 172.21.15.90:48746) 2024-04-02T14:46:43.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.564+0000 7fabe3e00640 1 -- 172.21.15.90:0/1985371996 learned_addr learned my addr 172.21.15.90:0/1985371996 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:46:43.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.565+0000 7fabe3e00640 1 -- 172.21.15.90:0/1985371996 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cb49e5ccb0 msgr2=0x55cb49e58e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:46:43.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.565+0000 7fabe3e00640 1 --2- 172.21.15.90:0/1985371996 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cb49e5ccb0 0x55cb49e58e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:43.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.565+0000 7fabe3e00640 1 -- 172.21.15.90:0/1985371996 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cb49e537b0 msgr2=0x55cb49e5c3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:46:43.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.565+0000 7fabe3e00640 1 --2- 172.21.15.90:0/1985371996 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cb49e537b0 0x55cb49e5c3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:43.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.565+0000 7fabe3e00640 1 -- 172.21.15.90:0/1985371996 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cb49e54b40 con 0x55cb49e540d0 2024-04-02T14:46:43.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.565+0000 7fabe3e00640 1 --2- 172.21.15.90:0/1985371996 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55cb49e540d0 0x55cb49e689c0 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7fabd0009930 tx=0x7fabd00a9040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=84903f75a22dcbfc server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:46:43.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.566+0000 7fabd5600640 1 -- 172.21.15.90:0/1985371996 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabd00ba8d0 con 0x55cb49e540d0 2024-04-02T14:46:43.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.566+0000 7fabd5600640 1 -- 172.21.15.90:0/1985371996 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fabd0005690 con 0x55cb49e540d0 2024-04-02T14:46:43.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.566+0000 7fabd5600640 1 -- 172.21.15.90:0/1985371996 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabd00ba8d0 con 0x55cb49e540d0 2024-04-02T14:46:43.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.567+0000 7fabee970740 1 -- 172.21.15.90:0/1985371996 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55cb49e540d0 msgr2=0x55cb49e689c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:43.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.567+0000 7fabee970740 1 --2- 172.21.15.90:0/1985371996 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55cb49e540d0 0x55cb49e689c0 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7fabd0009930 tx=0x7fabd00a9040 comp rx=0 tx=0).stop 2024-04-02T14:46:43.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.568+0000 7fabee970740 1 -- 172.21.15.90:0/1985371996 shutdown_connections 2024-04-02T14:46:43.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.568+0000 7fabee970740 1 --2- 172.21.15.90:0/1985371996 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cb49e5ccb0 0x55cb49e58e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:43.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.568+0000 7fabee970740 1 --2- 172.21.15.90:0/1985371996 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55cb49e540d0 0x55cb49e689c0 unknown :-1 s=CLOSED pgs=855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:43.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.568+0000 7fabee970740 1 --2- 172.21.15.90:0/1985371996 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cb49e537b0 0x55cb49e5c3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:43.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.568+0000 7fabee970740 1 -- 172.21.15.90:0/1985371996 >> 172.21.15.90:0/1985371996 conn(0x55cb49e4dee0 msgr2=0x55cb49e4c6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:46:43.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.569+0000 7fabee970740 1 -- 172.21.15.90:0/1985371996 shutdown_connections 2024-04-02T14:46:43.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.569+0000 7fabee970740 1 -- 172.21.15.90:0/1985371996 wait complete. 2024-04-02T14:46:43.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.569+0000 7fabee970740 1 -- start start 2024-04-02T14:46:43.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.569+0000 7fabee970740 1 Processor -- start 2024-04-02T14:46:43.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.570+0000 7fabee970740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cb49e537b0 0x55cb49e56ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:43.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.570+0000 7fabee970740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cb49e540d0 0x55cb49e570e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:43.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.570+0000 7fabee970740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55cb49e5ccb0 0x55cb49e57620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:43.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.570+0000 7fabee970740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fabe41acb90 con 0x55cb49e537b0 2024-04-02T14:46:43.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.570+0000 7fabee970740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fabe41ac4c0 con 0x55cb49e5ccb0 2024-04-02T14:46:43.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.570+0000 7fabee970740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fabe41a9720 con 0x55cb49e540d0 2024-04-02T14:46:43.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.570+0000 7fabee970740 10 client.0 _ll_register_callbacks cb 0x7fabd8009e90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:46:43.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.571+0000 7fabe3e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cb49e540d0 0x55cb49e570e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:43.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.571+0000 7fabe9200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cb49e537b0 0x55cb49e56ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:43.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.571+0000 7fabe9c00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55cb49e5ccb0 0x55cb49e57620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:43.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.571+0000 7fabe3e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cb49e540d0 0x55cb49e570e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:48750/0 (socket says 172.21.15.90:48750) 2024-04-02T14:46:43.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.571+0000 7fabe3e00640 1 -- 172.21.15.90:0/2964593427 learned_addr learned my addr 172.21.15.90:0/2964593427 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:46:43.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.571+0000 7fabe3e00640 1 -- 172.21.15.90:0/2964593427 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55cb49e5ccb0 msgr2=0x55cb49e57620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:46:43.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.571+0000 7fabe3e00640 1 --2- 172.21.15.90:0/2964593427 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55cb49e5ccb0 0x55cb49e57620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:43.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.571+0000 7fabe3e00640 1 -- 172.21.15.90:0/2964593427 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cb49e537b0 msgr2=0x55cb49e56ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:46:43.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.571+0000 7fabe3e00640 1 --2- 172.21.15.90:0/2964593427 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cb49e537b0 0x55cb49e56ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:43.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.571+0000 7fabe3e00640 1 -- 172.21.15.90:0/2964593427 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cb49edacc0 con 0x55cb49e540d0 2024-04-02T14:46:43.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.571+0000 7fabe3e00640 1 --2- 172.21.15.90:0/2964593427 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cb49e540d0 0x55cb49e570e0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7fabd0005c20 tx=0x7fabd002ce20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c4ccf3a7c3f07f46 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:46:43.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.571+0000 7fabcaa00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:46:43.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.571+0000 7fabcaa00640 1 -- 172.21.15.90:0/2964593427 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabd00bdf90 con 0x55cb49e540d0 2024-04-02T14:46:43.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.571+0000 7fabcaa00640 1 -- 172.21.15.90:0/2964593427 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 8550+0+0 (secure 0 0 0) 0x7fabd002ca80 con 0x55cb49e540d0 2024-04-02T14:46:43.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.571+0000 7fabee970740 1 -- 172.21.15.90:0/2964593427 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55cb49dc4cb0 con 0x55cb49e540d0 2024-04-02T14:46:43.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.572+0000 7fabcaa00640 1 -- 172.21.15.90:0/2964593427 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fabd00be170 con 0x55cb49e540d0 2024-04-02T14:46:43.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.572+0000 7fabee970740 10 client.28990 fetch_fsmap learned FSMap version 27 2024-04-02T14:46:43.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.572+0000 7fabee970740 1 -- 172.21.15.90:0/2964593427 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55cb49edbfb0 con 0x55cb49e540d0 2024-04-02T14:46:43.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.572+0000 7fabcaa00640 1 -- 172.21.15.90:0/2964593427 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55cb49dc4cb0 con 0x55cb49e540d0 2024-04-02T14:46:43.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.572+0000 7fabcaa00640 1 -- 172.21.15.90:0/2964593427 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fabd00effe0 con 0x55cb49e540d0 2024-04-02T14:46:43.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.572+0000 7fabee970740 10 client.28990 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:46:43.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.572+0000 7fabee970740 10 client.28990 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:46:43.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.572+0000 7fabee970740 20 client.28990 populate_metadata read hostname 'smithi090' 2024-04-02T14:46:43.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.572+0000 7fabee970740 1 --2- 172.21.15.90:0/2964593427 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55cb49edc4a0 0x55cb49ede890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:43.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.572+0000 7fabee970740 4 client.28990 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-02T14:46:43.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.572+0000 7fabee970740 1 -- 172.21.15.90:0/2964593427 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55cb49dbf000 con 0x55cb49edc4a0 2024-04-02T14:46:43.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.572+0000 7fabe9200640 1 --2- 172.21.15.90:0/2964593427 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55cb49edc4a0 0x55cb49ede890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:43.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.573+0000 7fabe9200640 1 --2- 172.21.15.90:0/2964593427 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55cb49edc4a0 0x55cb49ede890 crc :-1 s=READY pgs=311 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f38d2ef7a5dc0600 server_cookie=e839a2369eae8798 in_seq=0 out_seq=0 2024-04-02T14:46:43.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.573+0000 7fabcaa00640 10 client.28990 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:46:43.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.573+0000 7fabcaa00640 1 -- 172.21.15.90:0/2964593427 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55cb49dbf000 con 0x55cb49edc4a0 2024-04-02T14:46:43.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.573+0000 7fabcaa00640 10 client.28990 handle_command_reply: tid=0 2024-04-02T14:46:43.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.573+0000 7fabee970740 1 client.28990 shutdown 2024-04-02T14:46:43.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.574+0000 7fabee970740 1 -- 172.21.15.90:0/2964593427 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cb49e540d0 msgr2=0x55cb49e570e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:43.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.574+0000 7fabee970740 1 --2- 172.21.15.90:0/2964593427 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cb49e540d0 0x55cb49e570e0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7fabd0005c20 tx=0x7fabd002ce20 comp rx=0 tx=0).stop 2024-04-02T14:46:43.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.576+0000 7fabee970740 1 -- 172.21.15.90:0/2964593427 shutdown_connections 2024-04-02T14:46:43.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.576+0000 7fabee970740 1 --2- 172.21.15.90:0/2964593427 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55cb49edc4a0 0x55cb49ede890 crc :-1 s=READY pgs=311 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:43.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.576+0000 7fabee970740 1 --2- 172.21.15.90:0/2964593427 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55cb49e5ccb0 0x55cb49e57620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:43.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.576+0000 7fabee970740 1 --2- 172.21.15.90:0/2964593427 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cb49e540d0 0x55cb49e570e0 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:43.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.576+0000 7fabee970740 1 --2- 172.21.15.90:0/2964593427 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cb49e537b0 0x55cb49e56ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:43.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.576+0000 7fabee970740 1 -- 172.21.15.90:0/2964593427 >> 172.21.15.90:0/2964593427 conn(0x55cb49e4dee0 msgr2=0x55cb49e4cf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:46:43.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.576+0000 7fabcaa00640 0 client.28990 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:46:43.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.576+0000 7fabee970740 1 -- 172.21.15.90:0/2964593427 shutdown_connections 2024-04-02T14:46:43.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.576+0000 7fabee970740 1 -- 172.21.15.90:0/2964593427 wait complete. 2024-04-02T14:46:43.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.577+0000 7fabee970740 20 client.28990 trim_cache size 0 max 16384 2024-04-02T14:46:43.573 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-02T14:46:43.573 INFO:teuthology.orchestra.run.smithi090.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-02T14:46:43.573 INFO:teuthology.orchestra.run.smithi090.stdout: "scrubs": { 2024-04-02T14:46:43.573 INFO:teuthology.orchestra.run.smithi090.stdout: "14ff3309-2615-4b00-935d-0f067b927c5d": { 2024-04-02T14:46:43.573 INFO:teuthology.orchestra.run.smithi090.stdout: "path": "/", 2024-04-02T14:46:43.573 INFO:teuthology.orchestra.run.smithi090.stdout: "tag": "14ff3309-2615-4b00-935d-0f067b927c5d", 2024-04-02T14:46:43.573 INFO:teuthology.orchestra.run.smithi090.stdout: "options": "recursive,force" 2024-04-02T14:46:43.574 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:46:43.574 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:46:43.574 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-02T14:46:43.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.577+0000 7fabebc00640 1 -- 172.21.15.90:0/972990088 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fabcc0a81f0 msgr2=0x7fabcc0aa6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:43.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.577+0000 7fabebc00640 1 --2- 172.21.15.90:0/972990088 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fabcc0a81f0 0x7fabcc0aa6b0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x55cb49e53280 tx=0x7fabd8005f90 comp rx=0 tx=0).stop 2024-04-02T14:46:43.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.577+0000 7fabebc00640 1 -- 172.21.15.90:0/972990088 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fabe4102450 msgr2=0x7fabe4196780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:43.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.577+0000 7fabebc00640 1 --2- 172.21.15.90:0/972990088 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fabe4102450 0x7fabe4196780 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7fabd0036040 tx=0x7fabd00043b0 comp rx=0 tx=0).stop 2024-04-02T14:46:43.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.578+0000 7fabebc00640 1 -- 172.21.15.90:0/972990088 shutdown_connections 2024-04-02T14:46:43.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.578+0000 7fabebc00640 1 --2- 172.21.15.90:0/972990088 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fabcc0a81f0 0x7fabcc0aa6b0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:43.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.578+0000 7fabebc00640 1 --2- 172.21.15.90:0/972990088 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fabe4107aa0 0x7fabe419aab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:43.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.578+0000 7fabebc00640 1 --2- 172.21.15.90:0/972990088 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fabe4102450 0x7fabe4196780 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:43.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.578+0000 7fabebc00640 1 --2- 172.21.15.90:0/972990088 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fabe4101a90 0x7fabe4196240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:43.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.578+0000 7fabebc00640 1 -- 172.21.15.90:0/972990088 >> 172.21.15.90:0/972990088 conn(0x7fabe4078f00 msgr2=0x7fabe4104400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:46:43.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.578+0000 7fabebc00640 1 -- 172.21.15.90:0/972990088 shutdown_connections 2024-04-02T14:46:43.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:46:43.578+0000 7fabebc00640 1 -- 172.21.15.90:0/972990088 wait complete. 2024-04-02T14:46:43.605 INFO:tasks.cephfs.filesystem:scrub status for tag:14ff3309-2615-4b00-935d-0f067b927c5d - {'path': '/', 'tag': '14ff3309-2615-4b00-935d-0f067b927c5d', 'options': 'recursive,force'} 2024-04-02T14:46:44.604 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:46:44 smithi092 ceph-mon[36497]: pgmap v368: 97 pgs: 97 active+clean; 4.4 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 19 MiB/s wr, 1.58k op/s 2024-04-02T14:46:44.604 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:46:44 smithi092 ceph-mon[36497]: scrub queued for path: / 2024-04-02T14:46:44.604 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:46:44 smithi092 ceph-mon[36497]: scrub summary: idle+waiting paths [/] 2024-04-02T14:46:44.604 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:46:44 smithi092 ceph-mon[36497]: scrub summary: active paths [/] 2024-04-02T14:46:44.604 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:46:44 smithi092 ceph-mon[36497]: scrub summary: idle+waiting paths [/] 2024-04-02T14:46:44.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:46:44 smithi177 ceph-mon[36431]: pgmap v368: 97 pgs: 97 active+clean; 4.4 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 19 MiB/s wr, 1.58k op/s 2024-04-02T14:46:44.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:46:44 smithi177 ceph-mon[36431]: scrub queued for path: / 2024-04-02T14:46:44.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:46:44 smithi177 ceph-mon[36431]: scrub summary: idle+waiting paths [/] 2024-04-02T14:46:44.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:46:44 smithi177 ceph-mon[36431]: scrub summary: active paths [/] 2024-04-02T14:46:44.681 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:46:44 smithi177 ceph-mon[36431]: scrub summary: idle+waiting paths [/] 2024-04-02T14:46:44.682 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:46:44 smithi090 ceph-mon[31674]: pgmap v368: 97 pgs: 97 active+clean; 4.4 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 19 MiB/s wr, 1.58k op/s 2024-04-02T14:46:44.682 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:46:44 smithi090 ceph-mon[31674]: scrub queued for path: / 2024-04-02T14:46:44.682 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:46:44 smithi090 ceph-mon[31674]: scrub summary: idle+waiting paths [/] 2024-04-02T14:46:44.682 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:46:44 smithi090 ceph-mon[31674]: scrub summary: active paths [/] 2024-04-02T14:46:44.682 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:46:44 smithi090 ceph-mon[31674]: scrub summary: idle+waiting paths [/] 2024-04-02T14:46:45.758 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.762+0000 7f532aa00640 1 -- 172.21.15.90:0/1016847896 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f532410b550 msgr2=0x7f532410b930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:45.758 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.762+0000 7f532aa00640 1 --2- 172.21.15.90:0/1016847896 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f532410b550 0x7f532410b930 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f5314009900 tx=0x7f531402bd50 comp rx=0 tx=0).stop 2024-04-02T14:46:45.758 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.762+0000 7f532aa00640 1 -- 172.21.15.90:0/1016847896 shutdown_connections 2024-04-02T14:46:45.758 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.762+0000 7f532aa00640 1 --2- 172.21.15.90:0/1016847896 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5324114890 0x7f5324116d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:45.758 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.762+0000 7f532aa00640 1 --2- 172.21.15.90:0/1016847896 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f532410be70 0x7f5324114350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:45.758 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.762+0000 7f532aa00640 1 --2- 172.21.15.90:0/1016847896 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f532410b550 0x7f532410b930 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:45.759 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.762+0000 7f532aa00640 1 -- 172.21.15.90:0/1016847896 >> 172.21.15.90:0/1016847896 conn(0x7f532406b8b0 msgr2=0x7f532406dcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:46:45.759 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.763+0000 7f532aa00640 1 -- 172.21.15.90:0/1016847896 shutdown_connections 2024-04-02T14:46:45.759 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.763+0000 7f532aa00640 1 -- 172.21.15.90:0/1016847896 wait complete. 2024-04-02T14:46:45.759 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.764+0000 7f532aa00640 1 Processor -- start 2024-04-02T14:46:45.760 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.764+0000 7f532aa00640 1 -- start start 2024-04-02T14:46:45.760 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.765+0000 7f532aa00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f532410b550 0x7f53241091a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:45.760 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.765+0000 7f532aa00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f532410be70 0x7f53241096e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:45.760 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.765+0000 7f532aa00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5324114890 0x7f5324109c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:45.760 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.765+0000 7f532aa00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f532411a6c0 con 0x7f5324114890 2024-04-02T14:46:45.761 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.765+0000 7f532aa00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f532411a540 con 0x7f532410be70 2024-04-02T14:46:45.761 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.765+0000 7f532aa00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f532411a840 con 0x7f532410b550 2024-04-02T14:46:45.761 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.765+0000 7f5323e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f532410b550 0x7f53241091a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:45.761 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.765+0000 7f5323400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f532410be70 0x7f53241096e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:45.761 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.765+0000 7f5323e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f532410b550 0x7f53241091a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:34852/0 (socket says 172.21.15.90:34852) 2024-04-02T14:46:45.761 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.765+0000 7f5323e00640 1 -- 172.21.15.90:0/649388391 learned_addr learned my addr 172.21.15.90:0/649388391 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:46:45.761 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.765+0000 7f5328a00640 1 --2- 172.21.15.90:0/649388391 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5324114890 0x7f5324109c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:45.761 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.765+0000 7f5323e00640 1 -- 172.21.15.90:0/649388391 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f532410be70 msgr2=0x7f53241096e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:45.761 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.765+0000 7f5323e00640 1 --2- 172.21.15.90:0/649388391 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f532410be70 0x7f53241096e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:45.761 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.765+0000 7f5323e00640 1 -- 172.21.15.90:0/649388391 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5324114890 msgr2=0x7f5324109c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:45.762 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.765+0000 7f5323e00640 1 --2- 172.21.15.90:0/649388391 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5324114890 0x7f5324109c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:45.762 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.765+0000 7f5323e00640 1 -- 172.21.15.90:0/649388391 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53241aefe0 con 0x7f532410b550 2024-04-02T14:46:45.762 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.765+0000 7f5323400640 1 --2- 172.21.15.90:0/649388391 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f532410be70 0x7f53241096e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:46:45.762 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.765+0000 7f5328a00640 1 --2- 172.21.15.90:0/649388391 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5324114890 0x7f5324109c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:46:45.762 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.766+0000 7f5323e00640 1 --2- 172.21.15.90:0/649388391 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f532410b550 0x7f53241091a0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f5314000c00 tx=0x7f5314004360 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:46:45.763 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.767+0000 7f5320c00640 1 -- 172.21.15.90:0/649388391 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5314042020 con 0x7f532410b550 2024-04-02T14:46:45.763 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.767+0000 7f5320c00640 1 -- 172.21.15.90:0/649388391 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f53140054e0 con 0x7f532410b550 2024-04-02T14:46:45.763 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.767+0000 7f5320c00640 1 -- 172.21.15.90:0/649388391 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5314003bb0 con 0x7f532410b550 2024-04-02T14:46:45.763 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.767+0000 7f532aa00640 1 -- 172.21.15.90:0/649388391 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53241af270 con 0x7f532410b550 2024-04-02T14:46:45.763 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.767+0000 7f532aa00640 1 -- 172.21.15.90:0/649388391 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53241a08d0 con 0x7f532410b550 2024-04-02T14:46:45.768 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.772+0000 7f5320c00640 1 -- 172.21.15.90:0/649388391 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f5314045020 con 0x7f532410b550 2024-04-02T14:46:45.769 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.774+0000 7f532d629740 1 Processor -- start 2024-04-02T14:46:45.769 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.774+0000 7f5320c00640 1 --2- 172.21.15.90:0/649388391 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f53080a3ec0 0x7f53080a6380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:45.770 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.774+0000 7f5320c00640 1 -- 172.21.15.90:0/649388391 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 8550+0+0 (secure 0 0 0) 0x7f53140f2310 con 0x7f532410b550 2024-04-02T14:46:45.770 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.774+0000 7f532d629740 1 -- start start 2024-04-02T14:46:45.770 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.774+0000 7f532d629740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55903db67eb0 0x55903db68290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:45.770 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.774+0000 7f532d629740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55903db687d0 0x55903db70e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:45.770 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.774+0000 7f532d629740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55903db713b0 0x55903db737a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:45.770 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.774+0000 7f532d629740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55903daccd70 con 0x55903db713b0 2024-04-02T14:46:45.770 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.774+0000 7f532d629740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55903da752a0 con 0x55903db687d0 2024-04-02T14:46:45.771 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.774+0000 7f532d629740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55903dac8d20 con 0x55903db67eb0 2024-04-02T14:46:45.772 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.775+0000 7f5323400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55903db687d0 0x55903db70e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:45.772 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.775+0000 7f5323e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55903db67eb0 0x55903db68290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:45.772 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.775+0000 7f5323e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55903db67eb0 0x55903db68290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:34862/0 (socket says 172.21.15.90:34862) 2024-04-02T14:46:45.772 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.775+0000 7f5323400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55903db687d0 0x55903db70e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:34674/0 (socket says 172.21.15.90:34674) 2024-04-02T14:46:45.772 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.775+0000 7f5323e00640 1 -- 172.21.15.90:0/3493992184 learned_addr learned my addr 172.21.15.90:0/3493992184 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:46:45.772 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.775+0000 7f5323400640 1 --2- 172.21.15.90:0/649388391 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f53080a3ec0 0x7f53080a6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:45.772 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.775+0000 7f5328a00640 1 --2- 172.21.15.90:0/3493992184 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55903db713b0 0x55903db737a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:45.772 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.776+0000 7f5323e00640 1 -- 172.21.15.90:0/3493992184 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55903db687d0 msgr2=0x55903db70e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:46:45.772 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.776+0000 7f5323e00640 1 --2- 172.21.15.90:0/3493992184 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55903db687d0 0x55903db70e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:45.772 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.776+0000 7f5323e00640 1 -- 172.21.15.90:0/3493992184 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55903db713b0 msgr2=0x55903db737a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:46:45.772 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.776+0000 7f5323e00640 1 --2- 172.21.15.90:0/3493992184 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55903db713b0 0x55903db737a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:45.772 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.776+0000 7f5323e00640 1 -- 172.21.15.90:0/3493992184 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55903da89840 con 0x55903db67eb0 2024-04-02T14:46:45.773 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.776+0000 7f5328a00640 1 --2- 172.21.15.90:0/3493992184 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55903db713b0 0x55903db737a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:46:45.773 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.776+0000 7f5323e00640 1 --2- 172.21.15.90:0/3493992184 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55903db67eb0 0x55903db68290 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f53140ba7a0 tx=0x7f5314005d50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f0e49e809f355be0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:46:45.773 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.776+0000 7f5319200640 1 -- 172.21.15.90:0/3493992184 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53140d7070 con 0x55903db67eb0 2024-04-02T14:46:45.773 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.776+0000 7f5319200640 1 -- 172.21.15.90:0/3493992184 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f53140be340 con 0x55903db67eb0 2024-04-02T14:46:45.773 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.776+0000 7f5319200640 1 -- 172.21.15.90:0/3493992184 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f531402ccb0 con 0x55903db67eb0 2024-04-02T14:46:45.774 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.777+0000 7f532d629740 1 -- 172.21.15.90:0/3493992184 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55903db67eb0 msgr2=0x55903db68290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:45.774 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.777+0000 7f532d629740 1 --2- 172.21.15.90:0/3493992184 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55903db67eb0 0x55903db68290 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f53140ba7a0 tx=0x7f5314005d50 comp rx=0 tx=0).stop 2024-04-02T14:46:45.774 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.777+0000 7f5323400640 1 --2- 172.21.15.90:0/649388391 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f53080a3ec0 0x7f53080a6380 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x55903db67980 tx=0x7f5310005f90 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:46:45.774 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.777+0000 7f532d629740 1 -- 172.21.15.90:0/3493992184 shutdown_connections 2024-04-02T14:46:45.774 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.777+0000 7f532d629740 1 --2- 172.21.15.90:0/3493992184 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55903db713b0 0x55903db737a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:45.774 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.777+0000 7f532d629740 1 --2- 172.21.15.90:0/3493992184 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55903db687d0 0x55903db70e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:45.774 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.777+0000 7f532d629740 1 --2- 172.21.15.90:0/3493992184 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55903db67eb0 0x55903db68290 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:45.774 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.777+0000 7f532d629740 1 -- 172.21.15.90:0/3493992184 >> 172.21.15.90:0/3493992184 conn(0x55903db625e0 msgr2=0x55903db629c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:46:45.774 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.777+0000 7f532d629740 1 -- 172.21.15.90:0/3493992184 shutdown_connections 2024-04-02T14:46:45.774 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.777+0000 7f532d629740 1 -- 172.21.15.90:0/3493992184 wait complete. 2024-04-02T14:46:45.775 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.779+0000 7f532d629740 1 -- start start 2024-04-02T14:46:45.775 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.779+0000 7f532d629740 1 Processor -- start 2024-04-02T14:46:45.775 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.780+0000 7f532d629740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55903db67eb0 0x55903db6e4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:45.775 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.780+0000 7f532d629740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55903db687d0 0x55903db6e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:45.775 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.780+0000 7f532d629740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55903db713b0 0x55903db6ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:45.776 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.780+0000 7f532d629740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55903db73f60 con 0x55903db687d0 2024-04-02T14:46:45.776 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.780+0000 7f532d629740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55903da88660 con 0x55903db713b0 2024-04-02T14:46:45.776 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.780+0000 7f532d629740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55903db740e0 con 0x55903db67eb0 2024-04-02T14:46:45.776 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.780+0000 7f5323400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55903db687d0 0x55903db6e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:45.776 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.780+0000 7f5323400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55903db687d0 0x55903db6e9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:36252/0 (socket says 172.21.15.90:36252) 2024-04-02T14:46:45.776 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.780+0000 7f5323400640 1 -- 172.21.15.90:0/2096668307 learned_addr learned my addr 172.21.15.90:0/2096668307 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:46:45.776 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.780+0000 7f532d629740 10 client.0 _ll_register_callbacks cb 0x55903da88050 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:46:45.776 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.780+0000 7f5328a00640 1 --2- 172.21.15.90:0/2096668307 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55903db713b0 0x55903db6ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:45.777 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.781+0000 7f5323e00640 1 --2- 172.21.15.90:0/2096668307 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55903db67eb0 0x55903db6e4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:45.778 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.781+0000 7f5323400640 1 -- 172.21.15.90:0/2096668307 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55903db67eb0 msgr2=0x55903db6e4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:46:45.778 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.781+0000 7f5323400640 1 --2- 172.21.15.90:0/2096668307 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55903db67eb0 0x55903db6e4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:45.778 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.781+0000 7f5323400640 1 -- 172.21.15.90:0/2096668307 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55903db713b0 msgr2=0x55903db6ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:46:45.778 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.781+0000 7f5323400640 1 --2- 172.21.15.90:0/2096668307 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55903db713b0 0x55903db6ef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:45.778 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.781+0000 7f5323400640 1 -- 172.21.15.90:0/2096668307 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55903da97e80 con 0x55903db687d0 2024-04-02T14:46:45.778 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.781+0000 7f5328a00640 1 --2- 172.21.15.90:0/2096668307 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55903db713b0 0x55903db6ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:46:45.778 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.781+0000 7f5323e00640 1 --2- 172.21.15.90:0/2096668307 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55903db67eb0 0x55903db6e4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:46:45.778 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.781+0000 7f5323400640 1 --2- 172.21.15.90:0/2096668307 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55903db687d0 0x55903db6e9e0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f53100119d0 tx=0x7f5310004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=defe8a563661b5bb server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:46:45.778 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.781+0000 7f530d600640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-02T14:46:45.778 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.782+0000 7f530d600640 1 -- 172.21.15.90:0/2096668307 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f531001f460 con 0x55903db687d0 2024-04-02T14:46:45.779 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.782+0000 7f532d629740 1 -- 172.21.15.90:0/2096668307 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f53140be340 con 0x55903db687d0 2024-04-02T14:46:45.779 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.782+0000 7f530d600640 1 -- 172.21.15.90:0/2096668307 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 8550+0+0 (secure 0 0 0) 0x7f531001f600 con 0x55903db687d0 2024-04-02T14:46:45.779 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.782+0000 7f530d600640 1 -- 172.21.15.90:0/2096668307 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f531001f800 con 0x55903db687d0 2024-04-02T14:46:45.779 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.782+0000 7f532d629740 10 client.18135 fetch_fsmap learned FSMap version 27 2024-04-02T14:46:45.779 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.782+0000 7f532d629740 1 -- 172.21.15.90:0/2096668307 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55903d7e1210 con 0x55903db687d0 2024-04-02T14:46:45.780 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.783+0000 7f530d600640 1 -- 172.21.15.90:0/2096668307 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f53140be340 con 0x55903db687d0 2024-04-02T14:46:45.780 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.783+0000 7f530d600640 1 -- 172.21.15.90:0/2096668307 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f5310028070 con 0x55903db687d0 2024-04-02T14:46:45.780 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.783+0000 7f532d629740 10 client.18135 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:46:45.780 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.783+0000 7f532d629740 10 client.18135 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-02T14:46:45.780 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.783+0000 7f532d629740 20 client.18135 populate_metadata read hostname 'smithi090' 2024-04-02T14:46:45.780 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.783+0000 7f532d629740 1 --2- 172.21.15.90:0/2096668307 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55903db81490 0x55903db838e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:45.780 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.783+0000 7f532d629740 4 client.18135 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:46:45.780 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.783+0000 7f532d629740 1 -- 172.21.15.90:0/2096668307 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55903da7ccd0 con 0x55903db81490 2024-04-02T14:46:45.780 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.784+0000 7f5323e00640 1 --2- 172.21.15.90:0/2096668307 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55903db81490 0x55903db838e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:45.780 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.785+0000 7f5323e00640 1 --2- 172.21.15.90:0/2096668307 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55903db81490 0x55903db838e0 crc :-1 s=READY pgs=312 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f57cddcb5b9b0d45 server_cookie=fa374a7f9f1edbcc in_seq=0 out_seq=0 2024-04-02T14:46:45.781 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.785+0000 7f530d600640 10 client.18135 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:46:45.781 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.786+0000 7f530d600640 1 -- 172.21.15.90:0/2096668307 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55903da7ccd0 con 0x55903db81490 2024-04-02T14:46:45.781 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.786+0000 7f530d600640 10 client.18135 handle_command_reply: tid=0 2024-04-02T14:46:45.781 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.786+0000 7f532d629740 1 client.18135 shutdown 2024-04-02T14:46:45.782 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.786+0000 7f532d629740 1 -- 172.21.15.90:0/2096668307 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55903db687d0 msgr2=0x55903db6e9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:45.782 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.786+0000 7f532d629740 1 --2- 172.21.15.90:0/2096668307 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55903db687d0 0x55903db6e9e0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f53100119d0 tx=0x7f5310004730 comp rx=0 tx=0).stop 2024-04-02T14:46:45.782 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.786+0000 7f532d629740 1 -- 172.21.15.90:0/2096668307 shutdown_connections 2024-04-02T14:46:45.782 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.786+0000 7f532d629740 1 --2- 172.21.15.90:0/2096668307 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55903db81490 0x55903db838e0 crc :-1 s=READY pgs=312 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:45.782 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.786+0000 7f532d629740 1 --2- 172.21.15.90:0/2096668307 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55903db713b0 0x55903db6ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:45.782 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.786+0000 7f532d629740 1 --2- 172.21.15.90:0/2096668307 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55903db687d0 0x55903db6e9e0 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:45.782 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.786+0000 7f532d629740 1 --2- 172.21.15.90:0/2096668307 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55903db67eb0 0x55903db6e4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:45.782 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.786+0000 7f532d629740 1 -- 172.21.15.90:0/2096668307 >> 172.21.15.90:0/2096668307 conn(0x55903db625e0 msgr2=0x55903db64330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:46:45.782 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.786+0000 7f530d600640 0 client.18135 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:46:45.782 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.787+0000 7f532d629740 1 -- 172.21.15.90:0/2096668307 shutdown_connections 2024-04-02T14:46:45.783 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.788+0000 7f532d629740 1 -- 172.21.15.90:0/2096668307 wait complete. 2024-04-02T14:46:45.784 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.788+0000 7f532d629740 20 client.18135 trim_cache size 0 max 16384 2024-04-02T14:46:45.808 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.812+0000 7f532d629740 1 Processor -- start 2024-04-02T14:46:45.808 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.812+0000 7f532d629740 1 -- start start 2024-04-02T14:46:45.808 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.812+0000 7f532d629740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55903db67eb0 0x55903db70bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:45.808 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.813+0000 7f532d629740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55903db687d0 0x55903db7cd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:45.808 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.813+0000 7f532d629740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55903db713b0 0x55903db891f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:45.808 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.813+0000 7f532d629740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f53241b3e30 con 0x55903db687d0 2024-04-02T14:46:45.809 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.813+0000 7f532d629740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f53241b5970 con 0x55903db67eb0 2024-04-02T14:46:45.809 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.813+0000 7f532d629740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f53241b4500 con 0x55903db713b0 2024-04-02T14:46:45.809 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.813+0000 7f5323e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55903db67eb0 0x55903db70bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:45.810 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.813+0000 7f5323400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55903db687d0 0x55903db7cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:45.810 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.813+0000 7f5323e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55903db67eb0 0x55903db70bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:34690/0 (socket says 172.21.15.90:34690) 2024-04-02T14:46:45.810 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.813+0000 7f5323e00640 1 -- 172.21.15.90:0/2963600183 learned_addr learned my addr 172.21.15.90:0/2963600183 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:46:45.810 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.813+0000 7f5328a00640 1 --2- 172.21.15.90:0/2963600183 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55903db713b0 0x55903db891f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:45.810 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.814+0000 7f5323e00640 1 -- 172.21.15.90:0/2963600183 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55903db713b0 msgr2=0x55903db891f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:46:45.810 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.814+0000 7f5323e00640 1 --2- 172.21.15.90:0/2963600183 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55903db713b0 0x55903db891f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:45.810 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.814+0000 7f5323e00640 1 -- 172.21.15.90:0/2963600183 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55903db687d0 msgr2=0x55903db7cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:46:45.810 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.814+0000 7f5323e00640 1 --2- 172.21.15.90:0/2963600183 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55903db687d0 0x55903db7cd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:45.810 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.814+0000 7f5323e00640 1 -- 172.21.15.90:0/2963600183 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55903d8ad620 con 0x55903db67eb0 2024-04-02T14:46:45.810 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.814+0000 7f5323e00640 1 --2- 172.21.15.90:0/2963600183 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55903db67eb0 0x55903db70bd0 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f53140be2d0 tx=0x7f5314003680 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e317bb39c9ae4632 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:46:45.811 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.814+0000 7f530fe00640 1 -- 172.21.15.90:0/2963600183 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53140d7040 con 0x55903db67eb0 2024-04-02T14:46:45.811 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.814+0000 7f530fe00640 1 -- 172.21.15.90:0/2963600183 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f53140bc3e0 con 0x55903db67eb0 2024-04-02T14:46:45.811 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.814+0000 7f530fe00640 1 -- 172.21.15.90:0/2963600183 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f531402ccb0 con 0x55903db67eb0 2024-04-02T14:46:45.811 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.816+0000 7f532d629740 1 -- 172.21.15.90:0/2963600183 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55903db67eb0 msgr2=0x55903db70bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:45.811 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.816+0000 7f532d629740 1 --2- 172.21.15.90:0/2963600183 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55903db67eb0 0x55903db70bd0 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f53140be2d0 tx=0x7f5314003680 comp rx=0 tx=0).stop 2024-04-02T14:46:45.812 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.816+0000 7f532d629740 1 -- 172.21.15.90:0/2963600183 shutdown_connections 2024-04-02T14:46:45.812 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.816+0000 7f532d629740 1 --2- 172.21.15.90:0/2963600183 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55903db713b0 0x55903db891f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:45.812 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.816+0000 7f532d629740 1 --2- 172.21.15.90:0/2963600183 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55903db687d0 0x55903db7cd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:45.812 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.816+0000 7f532d629740 1 --2- 172.21.15.90:0/2963600183 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55903db67eb0 0x55903db70bd0 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:45.812 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.816+0000 7f532d629740 1 -- 172.21.15.90:0/2963600183 >> 172.21.15.90:0/2963600183 conn(0x55903db625e0 msgr2=0x55903db6ecf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:46:45.812 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.816+0000 7f532d629740 1 -- 172.21.15.90:0/2963600183 shutdown_connections 2024-04-02T14:46:45.812 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.816+0000 7f532d629740 1 -- 172.21.15.90:0/2963600183 wait complete. 2024-04-02T14:46:45.814 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.816+0000 7f532d629740 1 -- start start 2024-04-02T14:46:45.814 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.816+0000 7f532d629740 1 Processor -- start 2024-04-02T14:46:45.814 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.817+0000 7f532d629740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55903db67eb0 0x55903db6b540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:45.814 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.817+0000 7f532d629740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55903db687d0 0x55903db6ba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:45.814 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.817+0000 7f532d629740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55903db713b0 0x55903dbe6a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:45.814 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.817+0000 7f532d629740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f53241b52a0 con 0x55903db67eb0 2024-04-02T14:46:45.815 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.817+0000 7f532d629740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f53241b4bd0 con 0x55903db713b0 2024-04-02T14:46:45.815 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.817+0000 7f532d629740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f53241b1e70 con 0x55903db687d0 2024-04-02T14:46:45.815 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.817+0000 7f532d629740 10 client.0 _ll_register_callbacks cb 0x7f5310011920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:46:45.815 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.817+0000 7f5328a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55903db713b0 0x55903dbe6a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:45.815 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.817+0000 7f5328a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55903db713b0 0x55903dbe6a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:34694/0 (socket says 172.21.15.90:34694) 2024-04-02T14:46:45.815 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.817+0000 7f5328a00640 1 -- 172.21.15.90:0/3959395452 learned_addr learned my addr 172.21.15.90:0/3959395452 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:46:45.815 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.817+0000 7f5323e00640 1 --2- 172.21.15.90:0/3959395452 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55903db67eb0 0x55903db6b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:45.815 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.817+0000 7f5323400640 1 --2- 172.21.15.90:0/3959395452 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55903db687d0 0x55903db6ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:45.815 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.818+0000 7f5328a00640 1 -- 172.21.15.90:0/3959395452 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55903db687d0 msgr2=0x55903db6ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:46:45.815 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.818+0000 7f5328a00640 1 --2- 172.21.15.90:0/3959395452 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55903db687d0 0x55903db6ba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:45.815 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.818+0000 7f5328a00640 1 -- 172.21.15.90:0/3959395452 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55903db67eb0 msgr2=0x55903db6b540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:46:45.815 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.818+0000 7f5328a00640 1 --2- 172.21.15.90:0/3959395452 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55903db67eb0 0x55903db6b540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:45.815 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.818+0000 7f5328a00640 1 -- 172.21.15.90:0/3959395452 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55903dbe6fa0 con 0x55903db713b0 2024-04-02T14:46:45.815 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.818+0000 7f5323e00640 1 --2- 172.21.15.90:0/3959395452 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55903db67eb0 0x55903db6b540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:46:45.815 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.818+0000 7f5328a00640 1 --2- 172.21.15.90:0/3959395452 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55903db713b0 0x55903dbe6a60 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f531c005fd0 tx=0x7f531c012b60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a8c290c0d8531370 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:46:45.816 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.818+0000 7f530d600640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:46:45.816 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.819+0000 7f530d600640 1 -- 172.21.15.90:0/3959395452 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f531c004d20 con 0x55903db713b0 2024-04-02T14:46:45.816 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.819+0000 7f530d600640 1 -- 172.21.15.90:0/3959395452 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 8550+0+0 (secure 0 0 0) 0x7f531c00fa00 con 0x55903db713b0 2024-04-02T14:46:45.816 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.819+0000 7f532d629740 1 -- 172.21.15.90:0/3959395452 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55903dada2b0 con 0x55903db713b0 2024-04-02T14:46:45.817 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.819+0000 7f530d600640 1 -- 172.21.15.90:0/3959395452 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f531c00fc00 con 0x55903db713b0 2024-04-02T14:46:45.817 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.819+0000 7f532d629740 10 client.29216 fetch_fsmap learned FSMap version 27 2024-04-02T14:46:45.817 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.819+0000 7f532d629740 1 -- 172.21.15.90:0/3959395452 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55903dbe8150 con 0x55903db713b0 2024-04-02T14:46:45.817 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.820+0000 7f530d600640 1 -- 172.21.15.90:0/3959395452 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55903dada2b0 con 0x55903db713b0 2024-04-02T14:46:45.817 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.820+0000 7f530d600640 1 -- 172.21.15.90:0/3959395452 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f531c01d070 con 0x55903db713b0 2024-04-02T14:46:45.817 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.820+0000 7f532d629740 10 client.29216 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:46:45.817 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.820+0000 7f532d629740 10 client.29216 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-02T14:46:45.817 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.820+0000 7f532d629740 20 client.29216 populate_metadata read hostname 'smithi090' 2024-04-02T14:46:45.817 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.820+0000 7f532d629740 1 --2- 172.21.15.90:0/3959395452 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55903dbe8900 0x55903dbead20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:45.817 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.820+0000 7f532d629740 4 client.29216 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-02T14:46:45.817 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.820+0000 7f532d629740 1 -- 172.21.15.90:0/3959395452 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55903dad4600 con 0x55903dbe8900 2024-04-02T14:46:45.817 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.820+0000 7f5323e00640 1 --2- 172.21.15.90:0/3959395452 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55903dbe8900 0x55903dbead20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:45.817 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.821+0000 7f5323e00640 1 --2- 172.21.15.90:0/3959395452 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55903dbe8900 0x55903dbead20 crc :-1 s=READY pgs=313 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eaa64b796cb4c073 server_cookie=516ee1fb47b25f54 in_seq=0 out_seq=0 2024-04-02T14:46:45.817 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.821+0000 7f530d600640 10 client.29216 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:46:45.834 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.838+0000 7f530d600640 1 -- 172.21.15.90:0/3959395452 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55903dad4600 con 0x55903dbe8900 2024-04-02T14:46:45.834 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.838+0000 7f530d600640 10 client.29216 handle_command_reply: tid=0 2024-04-02T14:46:45.834 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.838+0000 7f532d629740 1 client.29216 shutdown 2024-04-02T14:46:45.834 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.838+0000 7f532d629740 1 -- 172.21.15.90:0/3959395452 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55903db713b0 msgr2=0x55903dbe6a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:45.834 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.838+0000 7f532d629740 1 --2- 172.21.15.90:0/3959395452 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55903db713b0 0x55903dbe6a60 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f531c005fd0 tx=0x7f531c012b60 comp rx=0 tx=0).stop 2024-04-02T14:46:45.835 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.839+0000 7f532d629740 1 -- 172.21.15.90:0/3959395452 shutdown_connections 2024-04-02T14:46:45.835 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.839+0000 7f532d629740 1 --2- 172.21.15.90:0/3959395452 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55903dbe8900 0x55903dbead20 crc :-1 s=READY pgs=313 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:45.835 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.839+0000 7f532d629740 1 --2- 172.21.15.90:0/3959395452 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55903db713b0 0x55903dbe6a60 unknown :-1 s=CLOSED pgs=857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:45.835 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.839+0000 7f532d629740 1 --2- 172.21.15.90:0/3959395452 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55903db687d0 0x55903db6ba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:45.835 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.839+0000 7f532d629740 1 --2- 172.21.15.90:0/3959395452 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55903db67eb0 0x55903db6b540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:45.836 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.839+0000 7f532d629740 1 -- 172.21.15.90:0/3959395452 >> 172.21.15.90:0/3959395452 conn(0x55903db625e0 msgr2=0x55903db638c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:46:45.836 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.839+0000 7f530d600640 0 client.29216 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:46:45.836 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.840+0000 7f532d629740 1 -- 172.21.15.90:0/3959395452 shutdown_connections 2024-04-02T14:46:45.836 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.840+0000 7f532d629740 1 -- 172.21.15.90:0/3959395452 wait complete. 2024-04-02T14:46:45.836 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.840+0000 7f532d629740 20 client.29216 trim_cache size 0 max 16384 2024-04-02T14:46:45.836 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:{ 2024-04-02T14:46:45.836 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "message": "", 2024-04-02T14:46:45.836 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "return_code": 0 2024-04-02T14:46:45.836 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:} 2024-04-02T14:46:45.837 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.840+0000 7f532aa00640 1 -- 172.21.15.90:0/649388391 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f53080a3ec0 msgr2=0x7f53080a6380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:45.837 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.840+0000 7f532aa00640 1 --2- 172.21.15.90:0/649388391 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f53080a3ec0 0x7f53080a6380 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x55903db67980 tx=0x7f5310005f90 comp rx=0 tx=0).stop 2024-04-02T14:46:45.837 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.840+0000 7f532aa00640 1 -- 172.21.15.90:0/649388391 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f532410b550 msgr2=0x7f53241091a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:45.837 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.840+0000 7f532aa00640 1 --2- 172.21.15.90:0/649388391 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f532410b550 0x7f53241091a0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f5314000c00 tx=0x7f5314004360 comp rx=0 tx=0).stop 2024-04-02T14:46:45.837 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.841+0000 7f532aa00640 1 -- 172.21.15.90:0/649388391 shutdown_connections 2024-04-02T14:46:45.837 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.841+0000 7f532aa00640 1 --2- 172.21.15.90:0/649388391 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f53080a3ec0 0x7f53080a6380 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:45.837 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.841+0000 7f532aa00640 1 --2- 172.21.15.90:0/649388391 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5324114890 0x7f5324109c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:45.838 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.841+0000 7f532aa00640 1 --2- 172.21.15.90:0/649388391 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f532410be70 0x7f53241096e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:45.838 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.841+0000 7f532aa00640 1 --2- 172.21.15.90:0/649388391 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f532410b550 0x7f53241091a0 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:45.838 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.841+0000 7f532aa00640 1 -- 172.21.15.90:0/649388391 >> 172.21.15.90:0/649388391 conn(0x7f532406b8b0 msgr2=0x7f532406c010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:46:45.838 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.841+0000 7f532aa00640 1 -- 172.21.15.90:0/649388391 shutdown_connections 2024-04-02T14:46:45.838 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:45.841+0000 7f532aa00640 1 -- 172.21.15.90:0/649388391 wait complete. 2024-04-02T14:46:46.604 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:46:46 smithi092 ceph-mon[36497]: pgmap v369: 97 pgs: 97 active+clean; 4.5 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 21 MiB/s wr, 1.94k op/s 2024-04-02T14:46:46.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:46:46 smithi177 ceph-mon[36431]: pgmap v369: 97 pgs: 97 active+clean; 4.5 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 21 MiB/s wr, 1.94k op/s 2024-04-02T14:46:46.682 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:46:46 smithi090 ceph-mon[31674]: pgmap v369: 97 pgs: 97 active+clean; 4.5 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 21 MiB/s wr, 1.94k op/s 2024-04-02T14:46:46.825 INFO:teuthology.orchestra.run.smithi090.stderr:progress: 675.0 s, 2001.2 tps, lat 50.213 ms stddev 63.458 2024-04-02T14:46:47.604 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:46:47 smithi092 ceph-mon[36497]: scrub summary: idle 2024-04-02T14:46:47.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:46:47 smithi177 ceph-mon[36431]: scrub summary: idle 2024-04-02T14:46:47.682 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:46:47 smithi090 ceph-mon[31674]: scrub summary: idle 2024-04-02T14:46:48.604 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:46:48 smithi092 ceph-mon[36497]: pgmap v370: 97 pgs: 97 active+clean; 4.5 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 19 MiB/s wr, 1.66k op/s 2024-04-02T14:46:48.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:46:48 smithi177 ceph-mon[36431]: pgmap v370: 97 pgs: 97 active+clean; 4.5 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 19 MiB/s wr, 1.66k op/s 2024-04-02T14:46:48.681 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:46:48 smithi090 ceph-mon[31674]: pgmap v370: 97 pgs: 97 active+clean; 4.5 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 19 MiB/s wr, 1.66k op/s 2024-04-02T14:46:50.604 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:46:50 smithi092 ceph-mon[36497]: pgmap v371: 97 pgs: 97 active+clean; 4.5 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 19 MiB/s wr, 1.68k op/s 2024-04-02T14:46:50.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:46:50 smithi177 ceph-mon[36431]: pgmap v371: 97 pgs: 97 active+clean; 4.5 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 19 MiB/s wr, 1.68k op/s 2024-04-02T14:46:50.681 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:46:50 smithi090 ceph-mon[31674]: pgmap v371: 97 pgs: 97 active+clean; 4.5 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 19 MiB/s wr, 1.68k op/s 2024-04-02T14:46:51.825 INFO:teuthology.orchestra.run.smithi090.stderr:progress: 680.0 s, 1835.2 tps, lat 54.368 ms stddev 63.982 2024-04-02T14:46:52.604 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:46:52 smithi092 ceph-mon[36497]: pgmap v372: 97 pgs: 97 active+clean; 4.5 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 19 MiB/s wr, 1.77k op/s 2024-04-02T14:46:52.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:46:52 smithi177 ceph-mon[36431]: pgmap v372: 97 pgs: 97 active+clean; 4.5 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 19 MiB/s wr, 1.77k op/s 2024-04-02T14:46:52.681 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:46:52 smithi090 ceph-mon[31674]: pgmap v372: 97 pgs: 97 active+clean; 4.5 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 19 MiB/s wr, 1.77k op/s 2024-04-02T14:46:54.604 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:46:54 smithi092 ceph-mon[36497]: pgmap v373: 97 pgs: 97 active+clean; 4.5 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 5.1 MiB/s wr, 637 op/s 2024-04-02T14:46:54.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:46:54 smithi177 ceph-mon[36431]: pgmap v373: 97 pgs: 97 active+clean; 4.5 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 5.1 MiB/s wr, 637 op/s 2024-04-02T14:46:54.681 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:46:54 smithi090 ceph-mon[31674]: pgmap v373: 97 pgs: 97 active+clean; 4.5 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 5.1 MiB/s wr, 637 op/s 2024-04-02T14:46:56.604 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:46:56 smithi092 ceph-mon[36497]: pgmap v374: 97 pgs: 97 active+clean; 4.5 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 7.5 MiB/s wr, 904 op/s 2024-04-02T14:46:56.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:46:56 smithi177 ceph-mon[36431]: pgmap v374: 97 pgs: 97 active+clean; 4.5 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 7.5 MiB/s wr, 904 op/s 2024-04-02T14:46:56.681 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:46:56 smithi090 ceph-mon[31674]: pgmap v374: 97 pgs: 97 active+clean; 4.5 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 7.5 MiB/s wr, 904 op/s 2024-04-02T14:46:56.825 INFO:teuthology.orchestra.run.smithi090.stderr:progress: 685.0 s, 1816.8 tps, lat 54.547 ms stddev 62.531 2024-04-02T14:46:58.604 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:46:58 smithi092 ceph-mon[36497]: pgmap v375: 97 pgs: 97 active+clean; 4.5 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 5.1 MiB/s wr, 550 op/s 2024-04-02T14:46:58.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:46:58 smithi177 ceph-mon[36431]: pgmap v375: 97 pgs: 97 active+clean; 4.5 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 5.1 MiB/s wr, 550 op/s 2024-04-02T14:46:58.681 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:46:58 smithi090 ceph-mon[31674]: pgmap v375: 97 pgs: 97 active+clean; 4.5 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 5.1 MiB/s wr, 550 op/s 2024-04-02T14:46:59.000 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.005+0000 7f299e000640 1 -- 172.21.15.90:0/1585386260 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f299810d410 msgr2=0x7f2998114fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:59.000 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.005+0000 7f299e000640 1 --2- 172.21.15.90:0/1585386260 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f299810d410 0x7f2998114fe0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f2990009ca0 tx=0x7f299002c100 comp rx=0 tx=0).stop 2024-04-02T14:46:59.000 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.005+0000 7f299e000640 1 -- 172.21.15.90:0/1585386260 shutdown_connections 2024-04-02T14:46:59.000 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.005+0000 7f299e000640 1 --2- 172.21.15.90:0/1585386260 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f299810d410 0x7f2998114fe0 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:59.000 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.005+0000 7f299e000640 1 --2- 172.21.15.90:0/1585386260 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2998108cf0 0x7f299810cd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:59.000 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.005+0000 7f299e000640 1 --2- 172.21.15.90:0/1585386260 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2998108340 0x7f2998108720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:59.000 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.005+0000 7f299e000640 1 -- 172.21.15.90:0/1585386260 >> 172.21.15.90:0/1585386260 conn(0x7f2998076070 msgr2=0x7f2998078490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:46:59.000 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.005+0000 7f299e000640 1 -- 172.21.15.90:0/1585386260 shutdown_connections 2024-04-02T14:46:59.000 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.005+0000 7f299e000640 1 -- 172.21.15.90:0/1585386260 wait complete. 2024-04-02T14:46:59.001 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.006+0000 7f299e000640 1 Processor -- start 2024-04-02T14:46:59.001 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.007+0000 7f299e000640 1 -- start start 2024-04-02T14:46:59.002 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.007+0000 7f299e000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2998108340 0x7f29981a4630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:59.002 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.007+0000 7f299e000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f299810d410 0x7f29981a4b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:59.002 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.007+0000 7f299e000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f299819e6f0 0x7f299819eb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:59.002 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.008+0000 7f299e000640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f29981184b0 con 0x7f2998108340 2024-04-02T14:46:59.002 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.008+0000 7f299e000640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f2998118330 con 0x7f299819e6f0 2024-04-02T14:46:59.003 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.008+0000 7f299e000640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f2998118630 con 0x7f299810d410 2024-04-02T14:46:59.003 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.008+0000 7f2996a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f299810d410 0x7f29981a4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:59.003 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.008+0000 7f2996a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f299810d410 0x7f29981a4b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:50344/0 (socket says 172.21.15.90:50344) 2024-04-02T14:46:59.003 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.008+0000 7f2997e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f299819e6f0 0x7f299819eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:59.003 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.008+0000 7f2996a00640 1 -- 172.21.15.90:0/4251808198 learned_addr learned my addr 172.21.15.90:0/4251808198 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:46:59.003 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.008+0000 7f2997400640 1 --2- 172.21.15.90:0/4251808198 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2998108340 0x7f29981a4630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:59.004 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.008+0000 7f2996a00640 1 -- 172.21.15.90:0/4251808198 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f299819e6f0 msgr2=0x7f299819eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:59.004 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.008+0000 7f2996a00640 1 --2- 172.21.15.90:0/4251808198 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f299819e6f0 0x7f299819eb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:59.004 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.008+0000 7f2996a00640 1 -- 172.21.15.90:0/4251808198 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2998108340 msgr2=0x7f29981a4630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:59.004 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.008+0000 7f2996a00640 1 --2- 172.21.15.90:0/4251808198 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2998108340 0x7f29981a4630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:59.004 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.008+0000 7f2996a00640 1 -- 172.21.15.90:0/4251808198 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f299819f2f0 con 0x7f299810d410 2024-04-02T14:46:59.004 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.008+0000 7f2997400640 1 --2- 172.21.15.90:0/4251808198 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2998108340 0x7f29981a4630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:46:59.004 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.008+0000 7f2997e00640 1 --2- 172.21.15.90:0/4251808198 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f299819e6f0 0x7f299819eb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:46:59.005 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.009+0000 7f2996a00640 1 --2- 172.21.15.90:0/4251808198 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f299810d410 0x7f29981a4b70 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f298400e9c0 tx=0x7f298400ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:46:59.005 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.009+0000 7f298fe00640 1 -- 172.21.15.90:0/4251808198 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f298400cd00 con 0x7f299810d410 2024-04-02T14:46:59.005 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.009+0000 7f298fe00640 1 -- 172.21.15.90:0/4251808198 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2984005500 con 0x7f299810d410 2024-04-02T14:46:59.005 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.009+0000 7f299e000640 1 -- 172.21.15.90:0/4251808198 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29981ab3d0 con 0x7f299810d410 2024-04-02T14:46:59.005 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.009+0000 7f298fe00640 1 -- 172.21.15.90:0/4251808198 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2984010430 con 0x7f299810d410 2024-04-02T14:46:59.005 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.009+0000 7f299e000640 1 -- 172.21.15.90:0/4251808198 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29981ab830 con 0x7f299810d410 2024-04-02T14:46:59.009 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.014+0000 7f29a0c42740 1 Processor -- start 2024-04-02T14:46:59.009 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.014+0000 7f29a0c42740 1 -- start start 2024-04-02T14:46:59.010 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.015+0000 7f29a0c42740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5647e9cafeb0 0x5647e9cb0290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:59.010 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.015+0000 7f29a0c42740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5647e9cb07d0 0x5647e9cb8e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:59.010 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.015+0000 7f29a0c42740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5647e9cb93b0 0x5647e9cbb7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:59.010 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.015+0000 7f29a0c42740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5647e9c14d70 con 0x5647e9cb93b0 2024-04-02T14:46:59.010 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.015+0000 7f29a0c42740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x5647e9bbd2a0 con 0x5647e9cb07d0 2024-04-02T14:46:59.010 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.015+0000 7f29a0c42740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5647e9c10d20 con 0x5647e9cafeb0 2024-04-02T14:46:59.010 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.015+0000 7f2997400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5647e9cb07d0 0x5647e9cb8e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:59.010 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.015+0000 7f2997400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5647e9cb07d0 0x5647e9cb8e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:49780/0 (socket says 172.21.15.90:49780) 2024-04-02T14:46:59.011 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.015+0000 7f2997400640 1 -- 172.21.15.90:0/857563828 learned_addr learned my addr 172.21.15.90:0/857563828 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:46:59.011 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.015+0000 7f2997400640 1 -- 172.21.15.90:0/857563828 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5647e9cafeb0 msgr2=0x5647e9cb0290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:46:59.011 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.015+0000 7f2997e00640 1 --2- 172.21.15.90:0/857563828 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5647e9cafeb0 0x5647e9cb0290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:59.011 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.016+0000 7f2997400640 1 --2- 172.21.15.90:0/857563828 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5647e9cafeb0 0x5647e9cb0290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:59.011 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.016+0000 7f2997400640 1 -- 172.21.15.90:0/857563828 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5647e9cb93b0 msgr2=0x5647e9cbb7a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-02T14:46:59.011 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.016+0000 7f2997400640 1 --2- 172.21.15.90:0/857563828 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5647e9cb93b0 0x5647e9cbb7a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:59.011 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.016+0000 7f2997400640 1 -- 172.21.15.90:0/857563828 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5647e9bd1840 con 0x5647e9cb07d0 2024-04-02T14:46:59.012 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.016+0000 7f298fe00640 1 -- 172.21.15.90:0/4251808198 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f2984005000 con 0x7f299810d410 2024-04-02T14:46:59.012 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.016+0000 7f2997400640 1 --2- 172.21.15.90:0/857563828 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5647e9cb07d0 0x5647e9cb8e70 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f2988009d50 tx=0x7f29880117b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d014226af37219db server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:46:59.012 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.017+0000 7f2983e00640 1 -- 172.21.15.90:0/857563828 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2988012650 con 0x5647e9cb07d0 2024-04-02T14:46:59.012 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.017+0000 7f2983e00640 1 -- 172.21.15.90:0/857563828 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2988009e40 con 0x5647e9cb07d0 2024-04-02T14:46:59.012 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.017+0000 7f2983e00640 1 -- 172.21.15.90:0/857563828 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2988012650 con 0x5647e9cb07d0 2024-04-02T14:46:59.013 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.018+0000 7f29a0c42740 1 -- 172.21.15.90:0/857563828 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5647e9cb07d0 msgr2=0x5647e9cb8e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:59.013 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.018+0000 7f29a0c42740 1 --2- 172.21.15.90:0/857563828 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5647e9cb07d0 0x5647e9cb8e70 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f2988009d50 tx=0x7f29880117b0 comp rx=0 tx=0).stop 2024-04-02T14:46:59.013 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.018+0000 7f29a0c42740 1 -- 172.21.15.90:0/857563828 shutdown_connections 2024-04-02T14:46:59.013 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.018+0000 7f29a0c42740 1 --2- 172.21.15.90:0/857563828 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5647e9cb93b0 0x5647e9cbb7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:59.013 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.018+0000 7f29a0c42740 1 --2- 172.21.15.90:0/857563828 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5647e9cb07d0 0x5647e9cb8e70 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:59.014 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.018+0000 7f29a0c42740 1 --2- 172.21.15.90:0/857563828 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5647e9cafeb0 0x5647e9cb0290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:59.014 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.018+0000 7f29a0c42740 1 -- 172.21.15.90:0/857563828 >> 172.21.15.90:0/857563828 conn(0x5647e9caa5e0 msgr2=0x5647e9caa9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:46:59.014 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.018+0000 7f29a0c42740 1 -- 172.21.15.90:0/857563828 shutdown_connections 2024-04-02T14:46:59.014 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.018+0000 7f29a0c42740 1 -- 172.21.15.90:0/857563828 wait complete. 2024-04-02T14:46:59.014 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.019+0000 7f298fe00640 1 --2- 172.21.15.90:0/4251808198 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f297c0a3da0 0x7f297c0a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:59.014 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.019+0000 7f298fe00640 1 -- 172.21.15.90:0/4251808198 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 8550+0+0 (secure 0 0 0) 0x7f29840d0c60 con 0x7f299810d410 2024-04-02T14:46:59.014 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.019+0000 7f29a0c42740 1 -- start start 2024-04-02T14:46:59.014 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.019+0000 7f2997400640 1 --2- 172.21.15.90:0/4251808198 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f297c0a3da0 0x7f297c0a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:59.016 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.019+0000 7f29a0c42740 1 Processor -- start 2024-04-02T14:46:59.016 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.020+0000 7f29a0c42740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5647e9cafeb0 0x5647e9cb6420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:59.016 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.020+0000 7f29a0c42740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5647e9cb07d0 0x5647e9cb6960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:59.016 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.020+0000 7f29a0c42740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5647e9cb93b0 0x5647e9cb6ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:59.016 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.020+0000 7f29a0c42740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5647e9cbc1a0 con 0x5647e9cb07d0 2024-04-02T14:46:59.016 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.020+0000 7f29a0c42740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x5647e9bd0660 con 0x5647e9cafeb0 2024-04-02T14:46:59.016 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.020+0000 7f29a0c42740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5647e9cbc320 con 0x5647e9cb93b0 2024-04-02T14:46:59.016 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.020+0000 7f29a0c42740 10 client.0 _ll_register_callbacks cb 0x5647e9bd0050 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:46:59.016 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.020+0000 7f2997400640 1 --2- 172.21.15.90:0/4251808198 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f297c0a3da0 0x7f297c0a6260 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f298800f7e0 tx=0x7f29880057f0 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:46:59.016 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.020+0000 7f2997400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5647e9cafeb0 0x5647e9cb6420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:59.016 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.020+0000 7f2997400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5647e9cafeb0 0x5647e9cb6420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:49790/0 (socket says 172.21.15.90:49790) 2024-04-02T14:46:59.017 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.020+0000 7f2997400640 1 -- 172.21.15.90:0/5782832 learned_addr learned my addr 172.21.15.90:0/5782832 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:46:59.017 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.020+0000 7f2996a00640 1 --2- 172.21.15.90:0/5782832 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5647e9cb07d0 0x5647e9cb6960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:59.017 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.020+0000 7f2997e00640 1 --2- 172.21.15.90:0/5782832 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5647e9cb93b0 0x5647e9cb6ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:59.017 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.021+0000 7f2997400640 1 -- 172.21.15.90:0/5782832 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5647e9cb93b0 msgr2=0x5647e9cb6ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:46:59.017 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.021+0000 7f2997400640 1 --2- 172.21.15.90:0/5782832 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5647e9cb93b0 0x5647e9cb6ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:59.017 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.021+0000 7f2997400640 1 -- 172.21.15.90:0/5782832 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5647e9cb07d0 msgr2=0x5647e9cb6960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:46:59.017 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.021+0000 7f2997400640 1 --2- 172.21.15.90:0/5782832 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5647e9cb07d0 0x5647e9cb6960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:59.017 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.021+0000 7f2997400640 1 -- 172.21.15.90:0/5782832 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5647e9bdfe80 con 0x5647e9cafeb0 2024-04-02T14:46:59.022 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.026+0000 7f2997400640 1 --2- 172.21.15.90:0/5782832 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5647e9cafeb0 0x5647e9cb6420 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f2988004270 tx=0x7f298800f8e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=713dd39e5f2b03b8 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:46:59.022 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.027+0000 7f2980c00640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:46:59.024 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.030+0000 7f2980c00640 1 -- 172.21.15.90:0/5782832 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2988034060 con 0x5647e9cafeb0 2024-04-02T14:46:59.024 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.030+0000 7f2980c00640 1 -- 172.21.15.90:0/5782832 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 8550+0+0 (secure 0 0 0) 0x7f2988005460 con 0x5647e9cafeb0 2024-04-02T14:46:59.025 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.031+0000 7f29a0c42740 1 -- 172.21.15.90:0/5782832 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5647e9bc4cd0 con 0x5647e9cafeb0 2024-04-02T14:46:59.028 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.033+0000 7f2980c00640 1 -- 172.21.15.90:0/5782832 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f29880046c0 con 0x5647e9cafeb0 2024-04-02T14:46:59.028 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.033+0000 7f29a0c42740 10 client.29228 fetch_fsmap learned FSMap version 27 2024-04-02T14:46:59.028 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.033+0000 7f29a0c42740 1 -- 172.21.15.90:0/5782832 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5647e9929210 con 0x5647e9cafeb0 2024-04-02T14:46:59.032 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.038+0000 7f2980c00640 1 -- 172.21.15.90:0/5782832 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x5647e9bc4cd0 con 0x5647e9cafeb0 2024-04-02T14:46:59.032 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.038+0000 7f2980c00640 1 -- 172.21.15.90:0/5782832 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f2988003ad0 con 0x5647e9cafeb0 2024-04-02T14:46:59.032 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.038+0000 7f29a0c42740 10 client.29228 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:46:59.032 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.038+0000 7f29a0c42740 10 client.29228 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-02T14:46:59.032 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.038+0000 7f29a0c42740 20 client.29228 populate_metadata read hostname 'smithi090' 2024-04-02T14:46:59.033 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.038+0000 7f29a0c42740 1 --2- 172.21.15.90:0/5782832 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5647e9cc9700 0x5647e9ccbaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:59.033 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.038+0000 7f29a0c42740 4 client.29228 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:46:59.033 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.038+0000 7f29a0c42740 1 -- 172.21.15.90:0/5782832 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5647e9c222b0 con 0x5647e9cc9700 2024-04-02T14:46:59.034 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.040+0000 7f2996a00640 1 --2- 172.21.15.90:0/5782832 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5647e9cc9700 0x5647e9ccbaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:59.038 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.044+0000 7f2980c00640 10 client.29228 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:46:59.038 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.044+0000 7f2996a00640 1 --2- 172.21.15.90:0/5782832 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5647e9cc9700 0x5647e9ccbaf0 crc :-1 s=READY pgs=314 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ded34c60092cea73 server_cookie=609265db54722d9b in_seq=0 out_seq=0 2024-04-02T14:46:59.040 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.045+0000 7f2980c00640 1 -- 172.21.15.90:0/5782832 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5647e9c222b0 con 0x5647e9cc9700 2024-04-02T14:46:59.040 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.045+0000 7f2980c00640 10 client.29228 handle_command_reply: tid=0 2024-04-02T14:46:59.040 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.045+0000 7f29a0c42740 1 client.29228 shutdown 2024-04-02T14:46:59.040 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.045+0000 7f29a0c42740 1 -- 172.21.15.90:0/5782832 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5647e9cafeb0 msgr2=0x5647e9cb6420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:59.040 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.045+0000 7f29a0c42740 1 --2- 172.21.15.90:0/5782832 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5647e9cafeb0 0x5647e9cb6420 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f2988004270 tx=0x7f298800f8e0 comp rx=0 tx=0).stop 2024-04-02T14:46:59.041 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.046+0000 7f29a0c42740 1 -- 172.21.15.90:0/5782832 shutdown_connections 2024-04-02T14:46:59.041 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.046+0000 7f29a0c42740 1 --2- 172.21.15.90:0/5782832 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5647e9cc9700 0x5647e9ccbaf0 crc :-1 s=READY pgs=314 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:59.041 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.046+0000 7f29a0c42740 1 --2- 172.21.15.90:0/5782832 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5647e9cb93b0 0x5647e9cb6ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:59.041 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.046+0000 7f29a0c42740 1 --2- 172.21.15.90:0/5782832 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5647e9cb07d0 0x5647e9cb6960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:59.041 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.046+0000 7f29a0c42740 1 --2- 172.21.15.90:0/5782832 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5647e9cafeb0 0x5647e9cb6420 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:59.041 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.046+0000 7f29a0c42740 1 -- 172.21.15.90:0/5782832 >> 172.21.15.90:0/5782832 conn(0x5647e9caa5e0 msgr2=0x5647e9cac330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:46:59.041 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.046+0000 7f2980c00640 0 client.29228 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:46:59.041 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.046+0000 7f29a0c42740 1 -- 172.21.15.90:0/5782832 shutdown_connections 2024-04-02T14:46:59.041 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.046+0000 7f29a0c42740 1 -- 172.21.15.90:0/5782832 wait complete. 2024-04-02T14:46:59.041 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.046+0000 7f29a0c42740 20 client.29228 trim_cache size 0 max 16384 2024-04-02T14:46:59.093 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.099+0000 7f29a0c42740 1 Processor -- start 2024-04-02T14:46:59.093 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.099+0000 7f29a0c42740 1 -- start start 2024-04-02T14:46:59.094 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.099+0000 7f29a0c42740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5647e9cb07d0 0x5647e9cc5060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:59.094 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.099+0000 7f29a0c42740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5647e9cb93b0 0x5647e9cb6d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:59.094 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.099+0000 7f29a0c42740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5647e9cd1260 0x5647e9cd16c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:59.094 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.099+0000 7f29a0c42740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f29981b3eb0 con 0x5647e9cb93b0 2024-04-02T14:46:59.095 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.099+0000 7f29a0c42740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f29981b59f0 con 0x5647e9cd1260 2024-04-02T14:46:59.095 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.099+0000 7f29a0c42740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f29981b4580 con 0x5647e9cb07d0 2024-04-02T14:46:59.095 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.100+0000 7f2997400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5647e9cb07d0 0x5647e9cc5060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:59.095 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.100+0000 7f2997400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5647e9cb07d0 0x5647e9cc5060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:50358/0 (socket says 172.21.15.90:50358) 2024-04-02T14:46:59.095 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.100+0000 7f2997400640 1 -- 172.21.15.90:0/3894870826 learned_addr learned my addr 172.21.15.90:0/3894870826 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:46:59.095 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.100+0000 7f2997e00640 1 --2- 172.21.15.90:0/3894870826 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5647e9cd1260 0x5647e9cd16c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:59.095 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.100+0000 7f2996a00640 1 --2- 172.21.15.90:0/3894870826 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5647e9cb93b0 0x5647e9cb6d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:59.096 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.101+0000 7f2997e00640 1 -- 172.21.15.90:0/3894870826 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5647e9cb07d0 msgr2=0x5647e9cc5060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:46:59.096 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.101+0000 7f2997e00640 1 --2- 172.21.15.90:0/3894870826 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5647e9cb07d0 0x5647e9cc5060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:59.096 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.101+0000 7f2997e00640 1 -- 172.21.15.90:0/3894870826 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5647e9cb93b0 msgr2=0x5647e9cb6d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:46:59.096 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.101+0000 7f2997e00640 1 --2- 172.21.15.90:0/3894870826 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5647e9cb93b0 0x5647e9cb6d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:59.097 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.101+0000 7f2997e00640 1 -- 172.21.15.90:0/3894870826 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5647e99f5620 con 0x5647e9cd1260 2024-04-02T14:46:59.097 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.102+0000 7f2997400640 1 --2- 172.21.15.90:0/3894870826 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5647e9cb07d0 0x5647e9cc5060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:46:59.097 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.103+0000 7f2997e00640 1 --2- 172.21.15.90:0/3894870826 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5647e9cd1260 0x5647e9cd16c0 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f2990009bf0 tx=0x7f29900323d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e3d0b9ef297ec7e7 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:46:59.098 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.103+0000 7f2983400640 1 -- 172.21.15.90:0/3894870826 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2990045030 con 0x5647e9cd1260 2024-04-02T14:46:59.098 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.104+0000 7f2983400640 1 -- 172.21.15.90:0/3894870826 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f29900098c0 con 0x5647e9cd1260 2024-04-02T14:46:59.098 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.104+0000 7f2983400640 1 -- 172.21.15.90:0/3894870826 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2990039040 con 0x5647e9cd1260 2024-04-02T14:46:59.100 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.106+0000 7f29a0c42740 1 -- 172.21.15.90:0/3894870826 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5647e9cd1260 msgr2=0x5647e9cd16c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:59.100 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.106+0000 7f29a0c42740 1 --2- 172.21.15.90:0/3894870826 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5647e9cd1260 0x5647e9cd16c0 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f2990009bf0 tx=0x7f29900323d0 comp rx=0 tx=0).stop 2024-04-02T14:46:59.101 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.106+0000 7f29a0c42740 1 -- 172.21.15.90:0/3894870826 shutdown_connections 2024-04-02T14:46:59.101 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.106+0000 7f29a0c42740 1 --2- 172.21.15.90:0/3894870826 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5647e9cd1260 0x5647e9cd16c0 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:59.101 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.106+0000 7f29a0c42740 1 --2- 172.21.15.90:0/3894870826 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5647e9cb93b0 0x5647e9cb6d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:59.101 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.106+0000 7f29a0c42740 1 --2- 172.21.15.90:0/3894870826 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5647e9cb07d0 0x5647e9cc5060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:59.101 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.106+0000 7f29a0c42740 1 -- 172.21.15.90:0/3894870826 >> 172.21.15.90:0/3894870826 conn(0x5647e9caa5e0 msgr2=0x5647e9ca8db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:46:59.101 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.107+0000 7f29a0c42740 1 -- 172.21.15.90:0/3894870826 shutdown_connections 2024-04-02T14:46:59.101 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.107+0000 7f29a0c42740 1 -- 172.21.15.90:0/3894870826 wait complete. 2024-04-02T14:46:59.102 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.107+0000 7f29a0c42740 1 -- start start 2024-04-02T14:46:59.102 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.108+0000 7f29a0c42740 1 Processor -- start 2024-04-02T14:46:59.103 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.108+0000 7f29a0c42740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5647e9cb07d0 0x5647e9cc1760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:59.103 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.108+0000 7f29a0c42740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5647e9cb93b0 0x5647e9cb2c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:59.103 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.108+0000 7f29a0c42740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5647e9cd1260 0x5647e9cb3150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:59.103 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.108+0000 7f29a0c42740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f29981b5320 con 0x5647e9cb93b0 2024-04-02T14:46:59.103 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.108+0000 7f29a0c42740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f29981b4c50 con 0x5647e9cb07d0 2024-04-02T14:46:59.103 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.108+0000 7f29a0c42740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f29981b1eb0 con 0x5647e9cd1260 2024-04-02T14:46:59.103 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.108+0000 7f29a0c42740 10 client.0 _ll_register_callbacks cb 0x7f298800fe60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:46:59.103 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.109+0000 7f2997400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5647e9cb07d0 0x5647e9cc1760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:59.105 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.109+0000 7f2997400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5647e9cb07d0 0x5647e9cc1760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:49806/0 (socket says 172.21.15.90:49806) 2024-04-02T14:46:59.105 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.109+0000 7f2997400640 1 -- 172.21.15.90:0/1681771190 learned_addr learned my addr 172.21.15.90:0/1681771190 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:46:59.105 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.109+0000 7f2996a00640 1 --2- 172.21.15.90:0/1681771190 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5647e9cb93b0 0x5647e9cb2c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:59.105 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.109+0000 7f2997400640 1 -- 172.21.15.90:0/1681771190 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5647e9cd1260 msgr2=0x5647e9cb3150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:46:59.105 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.109+0000 7f2997400640 1 --2- 172.21.15.90:0/1681771190 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5647e9cd1260 0x5647e9cb3150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:59.105 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.109+0000 7f2997400640 1 -- 172.21.15.90:0/1681771190 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5647e9cb93b0 msgr2=0x5647e9cb2c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:46:59.105 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.109+0000 7f2997400640 1 --2- 172.21.15.90:0/1681771190 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5647e9cb93b0 0x5647e9cb2c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:59.105 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.109+0000 7f2997400640 1 -- 172.21.15.90:0/1681771190 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5647e9cb3790 con 0x5647e9cb07d0 2024-04-02T14:46:59.105 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.110+0000 7f2997400640 1 --2- 172.21.15.90:0/1681771190 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5647e9cb07d0 0x5647e9cc1760 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f2988003f20 tx=0x7f298800f5f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9ea0e8373159f4cd server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:46:59.106 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.110+0000 7f2980c00640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:46:59.106 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.110+0000 7f2980c00640 1 -- 172.21.15.90:0/1681771190 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2988034040 con 0x5647e9cb07d0 2024-04-02T14:46:59.106 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.110+0000 7f2980c00640 1 -- 172.21.15.90:0/1681771190 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 8550+0+0 (secure 0 0 0) 0x7f2988005460 con 0x5647e9cb07d0 2024-04-02T14:46:59.106 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.110+0000 7f29a0c42740 1 -- 172.21.15.90:0/1681771190 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f29900098c0 con 0x5647e9cb07d0 2024-04-02T14:46:59.106 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.110+0000 7f2980c00640 1 -- 172.21.15.90:0/1681771190 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f298803e080 con 0x5647e9cb07d0 2024-04-02T14:46:59.106 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.111+0000 7f29a0c42740 10 client.29234 fetch_fsmap learned FSMap version 27 2024-04-02T14:46:59.106 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.111+0000 7f29a0c42740 1 -- 172.21.15.90:0/1681771190 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5647e9d2f950 con 0x5647e9cb07d0 2024-04-02T14:46:59.106 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.111+0000 7f2980c00640 1 -- 172.21.15.90:0/1681771190 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f29900098c0 con 0x5647e9cb07d0 2024-04-02T14:46:59.107 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.111+0000 7f2980c00640 1 -- 172.21.15.90:0/1681771190 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f2988023b20 con 0x5647e9cb07d0 2024-04-02T14:46:59.107 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.111+0000 7f29a0c42740 10 client.29234 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:46:59.107 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.111+0000 7f29a0c42740 10 client.29234 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-02T14:46:59.107 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.111+0000 7f29a0c42740 20 client.29234 populate_metadata read hostname 'smithi090' 2024-04-02T14:46:59.107 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.111+0000 7f29a0c42740 1 --2- 172.21.15.90:0/1681771190 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5647e9d2ffc0 0x5647e9d32410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:46:59.107 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.111+0000 7f29a0c42740 4 client.29234 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-02T14:46:59.107 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.111+0000 7f29a0c42740 1 -- 172.21.15.90:0/1681771190 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5647e9c1c600 con 0x5647e9d2ffc0 2024-04-02T14:46:59.108 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.112+0000 7f2996a00640 1 --2- 172.21.15.90:0/1681771190 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5647e9d2ffc0 0x5647e9d32410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:46:59.108 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.112+0000 7f2996a00640 1 --2- 172.21.15.90:0/1681771190 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5647e9d2ffc0 0x5647e9d32410 crc :-1 s=READY pgs=315 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bf0da6f3b663efb2 server_cookie=a2d80ad92002e1f4 in_seq=0 out_seq=0 2024-04-02T14:46:59.108 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.112+0000 7f2980c00640 10 client.29234 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:46:59.128 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.133+0000 7f2980c00640 1 -- 172.21.15.90:0/1681771190 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5647e9c1c600 con 0x5647e9d2ffc0 2024-04-02T14:46:59.128 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.133+0000 7f2980c00640 10 client.29234 handle_command_reply: tid=0 2024-04-02T14:46:59.128 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.133+0000 7f29a0c42740 1 client.29234 shutdown 2024-04-02T14:46:59.128 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.133+0000 7f29a0c42740 1 -- 172.21.15.90:0/1681771190 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5647e9cb07d0 msgr2=0x5647e9cc1760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:59.128 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.133+0000 7f29a0c42740 1 --2- 172.21.15.90:0/1681771190 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5647e9cb07d0 0x5647e9cc1760 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f2988003f20 tx=0x7f298800f5f0 comp rx=0 tx=0).stop 2024-04-02T14:46:59.128 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.133+0000 7f29a0c42740 1 -- 172.21.15.90:0/1681771190 shutdown_connections 2024-04-02T14:46:59.128 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.133+0000 7f29a0c42740 1 --2- 172.21.15.90:0/1681771190 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5647e9d2ffc0 0x5647e9d32410 crc :-1 s=READY pgs=315 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:59.128 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.133+0000 7f29a0c42740 1 --2- 172.21.15.90:0/1681771190 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5647e9cd1260 0x5647e9cb3150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:59.129 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.133+0000 7f29a0c42740 1 --2- 172.21.15.90:0/1681771190 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5647e9cb93b0 0x5647e9cb2c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:59.129 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.133+0000 7f29a0c42740 1 --2- 172.21.15.90:0/1681771190 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5647e9cb07d0 0x5647e9cc1760 unknown :-1 s=CLOSED pgs=861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:59.129 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.133+0000 7f2980c00640 0 client.29234 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:46:59.129 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.133+0000 7f29a0c42740 1 -- 172.21.15.90:0/1681771190 >> 172.21.15.90:0/1681771190 conn(0x5647e9caa5e0 msgr2=0x5647e9cd0ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:46:59.129 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.133+0000 7f29a0c42740 1 -- 172.21.15.90:0/1681771190 shutdown_connections 2024-04-02T14:46:59.129 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.133+0000 7f29a0c42740 1 -- 172.21.15.90:0/1681771190 wait complete. 2024-04-02T14:46:59.129 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.134+0000 7f29a0c42740 20 client.29234 trim_cache size 0 max 16384 2024-04-02T14:46:59.129 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:{ 2024-04-02T14:46:59.129 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "message": "", 2024-04-02T14:46:59.129 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "return_code": 0 2024-04-02T14:46:59.130 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:} 2024-04-02T14:46:59.130 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.134+0000 7f299e000640 1 -- 172.21.15.90:0/4251808198 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f297c0a3da0 msgr2=0x7f297c0a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:59.130 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.134+0000 7f299e000640 1 --2- 172.21.15.90:0/4251808198 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f297c0a3da0 0x7f297c0a6260 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f298800f7e0 tx=0x7f29880057f0 comp rx=0 tx=0).stop 2024-04-02T14:46:59.130 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.134+0000 7f299e000640 1 -- 172.21.15.90:0/4251808198 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f299810d410 msgr2=0x7f29981a4b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:46:59.130 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.134+0000 7f299e000640 1 --2- 172.21.15.90:0/4251808198 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f299810d410 0x7f29981a4b70 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f298400e9c0 tx=0x7f298400ee80 comp rx=0 tx=0).stop 2024-04-02T14:46:59.131 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.134+0000 7f299e000640 1 -- 172.21.15.90:0/4251808198 shutdown_connections 2024-04-02T14:46:59.131 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.134+0000 7f299e000640 1 --2- 172.21.15.90:0/4251808198 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f297c0a3da0 0x7f297c0a6260 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:59.131 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.134+0000 7f299e000640 1 --2- 172.21.15.90:0/4251808198 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f299819e6f0 0x7f299819eb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:59.131 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.134+0000 7f299e000640 1 --2- 172.21.15.90:0/4251808198 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f299810d410 0x7f29981a4b70 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:59.131 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.134+0000 7f299e000640 1 --2- 172.21.15.90:0/4251808198 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2998108340 0x7f29981a4630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:46:59.131 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.134+0000 7f299e000640 1 -- 172.21.15.90:0/4251808198 >> 172.21.15.90:0/4251808198 conn(0x7f2998076070 msgr2=0x7f2998076bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:46:59.131 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.134+0000 7f299e000640 1 -- 172.21.15.90:0/4251808198 shutdown_connections 2024-04-02T14:46:59.131 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:46:59.134+0000 7f299e000640 1 -- 172.21.15.90:0/4251808198 wait complete. 2024-04-02T14:47:00.604 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:47:00 smithi092 ceph-mon[36497]: pgmap v376: 97 pgs: 97 active+clean; 4.5 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 5.2 MiB/s wr, 572 op/s 2024-04-02T14:47:00.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:47:00 smithi177 ceph-mon[36431]: pgmap v376: 97 pgs: 97 active+clean; 4.5 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 5.2 MiB/s wr, 572 op/s 2024-04-02T14:47:00.681 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:47:00 smithi090 ceph-mon[31674]: pgmap v376: 97 pgs: 97 active+clean; 4.5 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 5.2 MiB/s wr, 572 op/s 2024-04-02T14:47:01.604 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:47:01 smithi092 ceph-mon[36497]: pgmap v377: 97 pgs: 97 active+clean; 4.5 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 8.2 MiB/s wr, 858 op/s 2024-04-02T14:47:01.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:47:01 smithi177 ceph-mon[36431]: pgmap v377: 97 pgs: 97 active+clean; 4.5 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 8.2 MiB/s wr, 858 op/s 2024-04-02T14:47:01.682 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:47:01 smithi090 ceph-mon[31674]: pgmap v377: 97 pgs: 97 active+clean; 4.5 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 8.2 MiB/s wr, 858 op/s 2024-04-02T14:47:01.825 INFO:teuthology.orchestra.run.smithi090.stderr:progress: 690.0 s, 1389.7 tps, lat 67.521 ms stddev 91.074 2024-04-02T14:47:03.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:47:03 smithi177 ceph-mon[36431]: pgmap v378: 97 pgs: 97 active+clean; 4.5 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 5.7 MiB/s wr, 595 op/s 2024-04-02T14:47:03.932 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:47:03 smithi090 ceph-mon[31674]: pgmap v378: 97 pgs: 97 active+clean; 4.5 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 5.7 MiB/s wr, 595 op/s 2024-04-02T14:47:04.104 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:47:03 smithi092 ceph-mon[36497]: pgmap v378: 97 pgs: 97 active+clean; 4.5 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 5.7 MiB/s wr, 595 op/s 2024-04-02T14:47:05.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:47:05 smithi177 ceph-mon[36431]: pgmap v379: 97 pgs: 97 active+clean; 4.5 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 6.8 MiB/s wr, 758 op/s 2024-04-02T14:47:05.932 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:47:05 smithi090 ceph-mon[31674]: pgmap v379: 97 pgs: 97 active+clean; 4.5 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 6.8 MiB/s wr, 758 op/s 2024-04-02T14:47:06.104 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:47:05 smithi092 ceph-mon[36497]: pgmap v379: 97 pgs: 97 active+clean; 4.5 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 6.8 MiB/s wr, 758 op/s 2024-04-02T14:47:06.825 INFO:teuthology.orchestra.run.smithi090.stderr:progress: 695.0 s, 173.4 tps, lat 458.636 ms stddev 270.785 2024-04-02T14:47:07.930 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:47:07 smithi177 ceph-mon[36431]: pgmap v380: 97 pgs: 97 active+clean; 4.5 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 4.4 MiB/s wr, 491 op/s 2024-04-02T14:47:07.931 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:47:07 smithi090 ceph-mon[31674]: pgmap v380: 97 pgs: 97 active+clean; 4.5 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 4.4 MiB/s wr, 491 op/s 2024-04-02T14:47:08.104 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:47:07 smithi092 ceph-mon[36497]: pgmap v380: 97 pgs: 97 active+clean; 4.5 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 4.4 MiB/s wr, 491 op/s 2024-04-02T14:47:09.104 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:47:08 smithi092 ceph-mon[36497]: osdmap e102: 12 total, 12 up, 12 in 2024-04-02T14:47:09.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:47:08 smithi177 ceph-mon[36431]: osdmap e102: 12 total, 12 up, 12 in 2024-04-02T14:47:09.181 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:47:08 smithi090 ceph-mon[31674]: osdmap e102: 12 total, 12 up, 12 in 2024-04-02T14:47:10.104 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:47:09 smithi092 ceph-mon[36497]: pgmap v382: 97 pgs: 5 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 90 active+clean; 4.5 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 6.9 MiB/s wr, 715 op/s 2024-04-02T14:47:10.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:47:09 smithi177 ceph-mon[36431]: pgmap v382: 97 pgs: 5 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 90 active+clean; 4.5 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 6.9 MiB/s wr, 715 op/s 2024-04-02T14:47:10.181 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:47:09 smithi090 ceph-mon[31674]: pgmap v382: 97 pgs: 5 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 90 active+clean; 4.5 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 6.9 MiB/s wr, 715 op/s 2024-04-02T14:47:11.825 INFO:teuthology.orchestra.run.smithi090.stderr:progress: 700.0 s, 374.2 tps, lat 341.068 ms stddev 693.438 2024-04-02T14:47:12.104 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:47:11 smithi092 ceph-mon[36497]: pgmap v383: 97 pgs: 40 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 35 active+clean; 4.4 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 17 MiB/s wr, 1.50k op/s 2024-04-02T14:47:12.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:47:11 smithi177 ceph-mon[36431]: pgmap v383: 97 pgs: 40 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 35 active+clean; 4.4 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 17 MiB/s wr, 1.50k op/s 2024-04-02T14:47:12.181 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:47:11 smithi090 ceph-mon[31674]: pgmap v383: 97 pgs: 40 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 35 active+clean; 4.4 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 17 MiB/s wr, 1.50k op/s 2024-04-02T14:47:12.299 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.304+0000 7f498a400640 1 -- 172.21.15.90:0/1329718124 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4984112970 msgr2=0x7f4984114d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:12.299 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.304+0000 7f498a400640 1 --2- 172.21.15.90:0/1329718124 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4984112970 0x7f4984114d60 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f4974009930 tx=0x7f497402bc30 comp rx=0 tx=0).stop 2024-04-02T14:47:12.299 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.304+0000 7f498a400640 1 -- 172.21.15.90:0/1329718124 shutdown_connections 2024-04-02T14:47:12.299 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.304+0000 7f498a400640 1 --2- 172.21.15.90:0/1329718124 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4984112970 0x7f4984114d60 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:12.299 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.304+0000 7f498a400640 1 --2- 172.21.15.90:0/1329718124 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4984076950 0x7f4984112430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:12.299 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.304+0000 7f498a400640 1 --2- 172.21.15.90:0/1329718124 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4984076030 0x7f4984076410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:12.299 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.304+0000 7f498a400640 1 -- 172.21.15.90:0/1329718124 >> 172.21.15.90:0/1329718124 conn(0x7f49840ff7c0 msgr2=0x7f4984101be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:47:12.299 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.304+0000 7f498a400640 1 -- 172.21.15.90:0/1329718124 shutdown_connections 2024-04-02T14:47:12.299 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.304+0000 7f498a400640 1 -- 172.21.15.90:0/1329718124 wait complete. 2024-04-02T14:47:12.300 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.306+0000 7f498a400640 1 Processor -- start 2024-04-02T14:47:12.300 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.306+0000 7f498a400640 1 -- start start 2024-04-02T14:47:12.301 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.306+0000 7f498a400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4984076030 0x7f49841a4680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:12.301 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.306+0000 7f498a400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4984076950 0x7f49841a4bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:12.301 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.306+0000 7f498a400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4984112970 0x7f498419e740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:12.301 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.306+0000 7f498a400640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f49841184e0 con 0x7f4984112970 2024-04-02T14:47:12.302 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.306+0000 7f498a400640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f4984118360 con 0x7f4984076030 2024-04-02T14:47:12.302 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.306+0000 7f498a400640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f4984118660 con 0x7f4984076950 2024-04-02T14:47:12.302 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.307+0000 7f4983400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4984076030 0x7f49841a4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:12.302 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.307+0000 7f4982a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4984076950 0x7f49841a4bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:12.302 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.307+0000 7f4982a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4984076950 0x7f49841a4bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:35124/0 (socket says 172.21.15.90:35124) 2024-04-02T14:47:12.302 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.307+0000 7f4983400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4984076030 0x7f49841a4680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:49090/0 (socket says 172.21.15.90:49090) 2024-04-02T14:47:12.302 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.307+0000 7f4982a00640 1 -- 172.21.15.90:0/239137831 learned_addr learned my addr 172.21.15.90:0/239137831 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:47:12.303 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.307+0000 7f4982a00640 1 -- 172.21.15.90:0/239137831 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4984076030 msgr2=0x7f49841a4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:12.303 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.308+0000 7f4982a00640 1 --2- 172.21.15.90:0/239137831 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4984076030 0x7f49841a4680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:12.303 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.308+0000 7f4982a00640 1 -- 172.21.15.90:0/239137831 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4984112970 msgr2=0x7f498419e740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:12.303 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.308+0000 7f4982a00640 1 --2- 172.21.15.90:0/239137831 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4984112970 0x7f498419e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:12.303 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.308+0000 7f4982a00640 1 -- 172.21.15.90:0/239137831 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f498419f000 con 0x7f4984076950 2024-04-02T14:47:12.303 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.308+0000 7f4982a00640 1 --2- 172.21.15.90:0/239137831 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4984076950 0x7f49841a4bc0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f497000d950 tx=0x7f497000de10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:47:12.303 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.308+0000 7f497be00640 1 -- 172.21.15.90:0/239137831 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4970017070 con 0x7f4984076950 2024-04-02T14:47:12.303 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.308+0000 7f498a400640 1 -- 172.21.15.90:0/239137831 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f498419f2f0 con 0x7f4984076950 2024-04-02T14:47:12.304 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.308+0000 7f497be00640 1 -- 172.21.15.90:0/239137831 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4970009c30 con 0x7f4984076950 2024-04-02T14:47:12.304 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.309+0000 7f498a400640 1 -- 172.21.15.90:0/239137831 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49841ab600 con 0x7f4984076950 2024-04-02T14:47:12.304 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.309+0000 7f497be00640 1 -- 172.21.15.90:0/239137831 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f497000f6b0 con 0x7f4984076950 2024-04-02T14:47:12.311 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.317+0000 7f498d179740 1 Processor -- start 2024-04-02T14:47:12.311 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.318+0000 7f497be00640 1 -- 172.21.15.90:0/239137831 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f497000f850 con 0x7f4984076950 2024-04-02T14:47:12.313 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.318+0000 7f498d179740 1 -- start start 2024-04-02T14:47:12.313 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.319+0000 7f498d179740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c983d00eb0 0x55c983d01290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:12.313 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.319+0000 7f498d179740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c983d017d0 0x55c983d09e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:12.313 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.319+0000 7f498d179740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c983d0a3b0 0x55c983d0c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:12.313 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.319+0000 7f498d179740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55c983c65d70 con 0x55c983d0a3b0 2024-04-02T14:47:12.313 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.319+0000 7f498d179740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55c983c0e2a0 con 0x55c983d017d0 2024-04-02T14:47:12.313 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.319+0000 7f498d179740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55c983c61d20 con 0x55c983d00eb0 2024-04-02T14:47:12.314 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.320+0000 7f4983e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c983d00eb0 0x55c983d01290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:12.314 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.320+0000 7f4983e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c983d00eb0 0x55c983d01290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:35134/0 (socket says 172.21.15.90:35134) 2024-04-02T14:47:12.314 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.320+0000 7f4983e00640 1 -- 172.21.15.90:0/3074598658 learned_addr learned my addr 172.21.15.90:0/3074598658 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:47:12.314 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.320+0000 7f4982a00640 1 --2- 172.21.15.90:0/3074598658 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c983d0a3b0 0x55c983d0c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:12.314 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.320+0000 7f4983400640 1 --2- 172.21.15.90:0/3074598658 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c983d017d0 0x55c983d09e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:12.315 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.321+0000 7f4983e00640 1 -- 172.21.15.90:0/3074598658 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c983d017d0 msgr2=0x55c983d09e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:47:12.315 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.321+0000 7f4983e00640 1 --2- 172.21.15.90:0/3074598658 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c983d017d0 0x55c983d09e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:12.315 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.321+0000 7f4983e00640 1 -- 172.21.15.90:0/3074598658 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c983d0a3b0 msgr2=0x55c983d0c7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:47:12.315 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.321+0000 7f4983e00640 1 --2- 172.21.15.90:0/3074598658 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c983d0a3b0 0x55c983d0c7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:12.316 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.321+0000 7f4983e00640 1 -- 172.21.15.90:0/3074598658 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c983c22840 con 0x55c983d00eb0 2024-04-02T14:47:12.316 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.321+0000 7f4982a00640 1 --2- 172.21.15.90:0/3074598658 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c983d0a3b0 0x55c983d0c7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:47:12.316 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.321+0000 7f497be00640 1 --2- 172.21.15.90:0/239137831 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f49680a3e40 0x7f49680a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:12.316 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.322+0000 7f4983e00640 1 --2- 172.21.15.90:0/3074598658 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c983d00eb0 0x55c983d01290 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f4974009900 tx=0x7f4974005d70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e1fc33eee17dc861 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:47:12.317 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.322+0000 7f4983400640 1 --2- 172.21.15.90:0/239137831 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f49680a3e40 0x7f49680a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:12.317 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.322+0000 7f497be00640 1 -- 172.21.15.90:0/239137831 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 8594+0+0 (secure 0 0 0) 0x7f49700d0670 con 0x7f4984076950 2024-04-02T14:47:12.317 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.323+0000 7f4978c00640 1 -- 172.21.15.90:0/3074598658 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4974042020 con 0x55c983d00eb0 2024-04-02T14:47:12.318 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.324+0000 7f4978c00640 1 -- 172.21.15.90:0/3074598658 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f497402c880 con 0x55c983d00eb0 2024-04-02T14:47:12.318 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.324+0000 7f4978c00640 1 -- 172.21.15.90:0/3074598658 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4974003b90 con 0x55c983d00eb0 2024-04-02T14:47:12.319 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.325+0000 7f498d179740 1 -- 172.21.15.90:0/3074598658 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c983d00eb0 msgr2=0x55c983d01290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:12.319 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.325+0000 7f498d179740 1 --2- 172.21.15.90:0/3074598658 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c983d00eb0 0x55c983d01290 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f4974009900 tx=0x7f4974005d70 comp rx=0 tx=0).stop 2024-04-02T14:47:12.319 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.325+0000 7f4983400640 1 --2- 172.21.15.90:0/239137831 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f49680a3e40 0x7f49680a6300 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x55c983d00980 tx=0x7f497c0097c0 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:47:12.319 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.325+0000 7f498d179740 1 -- 172.21.15.90:0/3074598658 shutdown_connections 2024-04-02T14:47:12.319 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.325+0000 7f498d179740 1 --2- 172.21.15.90:0/3074598658 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c983d0a3b0 0x55c983d0c7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:12.320 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.325+0000 7f498d179740 1 --2- 172.21.15.90:0/3074598658 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c983d017d0 0x55c983d09e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:12.320 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.325+0000 7f498d179740 1 --2- 172.21.15.90:0/3074598658 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c983d00eb0 0x55c983d01290 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:12.320 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.325+0000 7f498d179740 1 -- 172.21.15.90:0/3074598658 >> 172.21.15.90:0/3074598658 conn(0x55c983cfb5e0 msgr2=0x55c983cfb9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:47:12.320 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.325+0000 7f498d179740 1 -- 172.21.15.90:0/3074598658 shutdown_connections 2024-04-02T14:47:12.320 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.326+0000 7f498d179740 1 -- 172.21.15.90:0/3074598658 wait complete. 2024-04-02T14:47:12.320 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.326+0000 7f498d179740 1 -- start start 2024-04-02T14:47:12.321 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.326+0000 7f498d179740 1 Processor -- start 2024-04-02T14:47:12.322 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.328+0000 7f498d179740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c983d00eb0 0x55c983d07370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:12.322 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.328+0000 7f498d179740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c983d017d0 0x55c983d078b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:12.323 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.328+0000 7f498d179740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c983d0a3b0 0x55c983d07df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:12.323 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.328+0000 7f4983400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c983d00eb0 0x55c983d07370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:12.323 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.328+0000 7f4983e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c983d0a3b0 0x55c983d07df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:12.323 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.328+0000 7f498d179740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55c983d1e150 con 0x55c983d017d0 2024-04-02T14:47:12.323 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.329+0000 7f498d179740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55c983c21660 con 0x55c983d00eb0 2024-04-02T14:47:12.323 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.329+0000 7f498d179740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55c983d1e2d0 con 0x55c983d0a3b0 2024-04-02T14:47:12.323 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.329+0000 7f4983e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c983d0a3b0 0x55c983d07df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:35142/0 (socket says 172.21.15.90:35142) 2024-04-02T14:47:12.324 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.329+0000 7f4983e00640 1 -- 172.21.15.90:0/2820469098 learned_addr learned my addr 172.21.15.90:0/2820469098 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:47:12.324 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.329+0000 7f498d179740 10 client.0 _ll_register_callbacks cb 0x55c983c21050 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:47:12.324 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.329+0000 7f4983e00640 1 -- 172.21.15.90:0/2820469098 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c983d00eb0 msgr2=0x55c983d07370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:47:12.324 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.329+0000 7f4983e00640 1 --2- 172.21.15.90:0/2820469098 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c983d00eb0 0x55c983d07370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:12.324 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.329+0000 7f4983e00640 1 -- 172.21.15.90:0/2820469098 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c983d017d0 msgr2=0x55c983d078b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:47:12.324 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.329+0000 7f4983e00640 1 --2- 172.21.15.90:0/2820469098 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c983d017d0 0x55c983d078b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:12.324 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.329+0000 7f4983e00640 1 -- 172.21.15.90:0/2820469098 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c983c30e80 con 0x55c983d0a3b0 2024-04-02T14:47:12.326 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.330+0000 7f496d600640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:47:12.326 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.330+0000 7f4983e00640 1 --2- 172.21.15.90:0/2820469098 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c983d0a3b0 0x55c983d07df0 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f49740057c0 tx=0x7f49740035f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2a79a411861f106d server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:47:12.326 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.332+0000 7f496d600640 1 -- 172.21.15.90:0/2820469098 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4974042030 con 0x55c983d0a3b0 2024-04-02T14:47:12.326 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.332+0000 7f496d600640 1 -- 172.21.15.90:0/2820469098 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 8594+0+0 (secure 0 0 0) 0x7f49740038d0 con 0x55c983d0a3b0 2024-04-02T14:47:12.326 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.332+0000 7f498d179740 1 -- 172.21.15.90:0/2820469098 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f497402c880 con 0x55c983d0a3b0 2024-04-02T14:47:12.327 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.333+0000 7f496d600640 1 -- 172.21.15.90:0/2820469098 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f4974003ad0 con 0x55c983d0a3b0 2024-04-02T14:47:12.328 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.334+0000 7f498d179740 10 client.29032 fetch_fsmap learned FSMap version 27 2024-04-02T14:47:12.328 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.334+0000 7f498d179740 1 -- 172.21.15.90:0/2820469098 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55c98397a210 con 0x55c983d0a3b0 2024-04-02T14:47:12.330 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.335+0000 7f496d600640 1 -- 172.21.15.90:0/2820469098 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f497402c880 con 0x55c983d0a3b0 2024-04-02T14:47:12.330 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.335+0000 7f496d600640 1 -- 172.21.15.90:0/2820469098 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f497405a020 con 0x55c983d0a3b0 2024-04-02T14:47:12.330 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.335+0000 7f498d179740 10 client.29032 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:47:12.330 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.335+0000 7f498d179740 10 client.29032 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-02T14:47:12.330 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.335+0000 7f498d179740 20 client.29032 populate_metadata read hostname 'smithi090' 2024-04-02T14:47:12.330 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.335+0000 7f498d179740 1 --2- 172.21.15.90:0/2820469098 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55c983d2b550 0x55c983d2d940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:12.330 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.335+0000 7f498d179740 4 client.29032 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:47:12.330 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.335+0000 7f498d179740 1 -- 172.21.15.90:0/2820469098 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c983c15cd0 con 0x55c983d2b550 2024-04-02T14:47:12.330 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.336+0000 7f4983400640 1 --2- 172.21.15.90:0/2820469098 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55c983d2b550 0x55c983d2d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:12.331 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.337+0000 7f4983400640 1 --2- 172.21.15.90:0/2820469098 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55c983d2b550 0x55c983d2d940 crc :-1 s=READY pgs=316 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a7414fe2c4c4b7b9 server_cookie=80888526039b3e5f in_seq=0 out_seq=0 2024-04-02T14:47:12.331 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.337+0000 7f496d600640 10 client.29032 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:47:12.333 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.339+0000 7f496d600640 1 -- 172.21.15.90:0/2820469098 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c983c15cd0 con 0x55c983d2b550 2024-04-02T14:47:12.333 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.339+0000 7f496d600640 10 client.29032 handle_command_reply: tid=0 2024-04-02T14:47:12.333 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.339+0000 7f498d179740 1 client.29032 shutdown 2024-04-02T14:47:12.333 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.339+0000 7f498d179740 1 -- 172.21.15.90:0/2820469098 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c983d0a3b0 msgr2=0x55c983d07df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:12.334 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.339+0000 7f498d179740 1 --2- 172.21.15.90:0/2820469098 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c983d0a3b0 0x55c983d07df0 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f49740057c0 tx=0x7f49740035f0 comp rx=0 tx=0).stop 2024-04-02T14:47:12.334 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.340+0000 7f498d179740 1 -- 172.21.15.90:0/2820469098 shutdown_connections 2024-04-02T14:47:12.334 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.340+0000 7f498d179740 1 --2- 172.21.15.90:0/2820469098 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55c983d2b550 0x55c983d2d940 crc :-1 s=READY pgs=316 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:12.334 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.340+0000 7f498d179740 1 --2- 172.21.15.90:0/2820469098 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c983d0a3b0 0x55c983d07df0 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:12.334 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.340+0000 7f498d179740 1 --2- 172.21.15.90:0/2820469098 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c983d017d0 0x55c983d078b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:12.334 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.340+0000 7f496d600640 0 client.29032 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:47:12.335 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.340+0000 7f498d179740 1 --2- 172.21.15.90:0/2820469098 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c983d00eb0 0x55c983d07370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:12.335 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.340+0000 7f498d179740 1 -- 172.21.15.90:0/2820469098 >> 172.21.15.90:0/2820469098 conn(0x55c983cfb5e0 msgr2=0x55c983cfd330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:47:12.336 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.342+0000 7f498d179740 1 -- 172.21.15.90:0/2820469098 shutdown_connections 2024-04-02T14:47:12.337 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.343+0000 7f498d179740 1 -- 172.21.15.90:0/2820469098 wait complete. 2024-04-02T14:47:12.337 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.343+0000 7f498d179740 20 client.29032 trim_cache size 0 max 16384 2024-04-02T14:47:12.373 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.378+0000 7f498d179740 1 Processor -- start 2024-04-02T14:47:12.373 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.379+0000 7f498d179740 1 -- start start 2024-04-02T14:47:12.373 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.379+0000 7f498d179740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c983d00eb0 0x55c983d09aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:12.373 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.379+0000 7f498d179740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c983d017d0 0x55c983d27070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:12.373 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.379+0000 7f498d179740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c983d0a3b0 0x55c983cfb9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:12.375 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.380+0000 7f4982a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c983d017d0 0x55c983d27070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:12.375 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.380+0000 7f4982a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c983d017d0 0x55c983d27070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:35144/0 (socket says 172.21.15.90:35144) 2024-04-02T14:47:12.375 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.380+0000 7f4982a00640 1 -- 172.21.15.90:0/2103184336 learned_addr learned my addr 172.21.15.90:0/2103184336 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:47:12.375 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.380+0000 7f498d179740 1 -- 172.21.15.90:0/2103184336 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f496400e930 con 0x55c983d0a3b0 2024-04-02T14:47:12.375 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.380+0000 7f498d179740 1 -- 172.21.15.90:0/2103184336 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f4964010470 con 0x55c983d00eb0 2024-04-02T14:47:12.375 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.380+0000 7f498d179740 1 -- 172.21.15.90:0/2103184336 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f496400f000 con 0x55c983d017d0 2024-04-02T14:47:12.376 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.380+0000 7f4983400640 1 --2- 172.21.15.90:0/2103184336 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c983d00eb0 0x55c983d09aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:12.376 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.380+0000 7f4983e00640 1 --2- 172.21.15.90:0/2103184336 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c983d0a3b0 0x55c983cfb9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:12.376 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.380+0000 7f4982a00640 1 -- 172.21.15.90:0/2103184336 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c983d00eb0 msgr2=0x55c983d09aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:47:12.376 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.380+0000 7f4982a00640 1 --2- 172.21.15.90:0/2103184336 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c983d00eb0 0x55c983d09aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:12.376 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.380+0000 7f4982a00640 1 -- 172.21.15.90:0/2103184336 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c983d0a3b0 msgr2=0x55c983cfb9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:47:12.376 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.380+0000 7f4982a00640 1 --2- 172.21.15.90:0/2103184336 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c983d0a3b0 0x55c983cfb9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:12.376 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.380+0000 7f4982a00640 1 -- 172.21.15.90:0/2103184336 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c983a46620 con 0x55c983d017d0 2024-04-02T14:47:12.376 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.380+0000 7f4983e00640 1 --2- 172.21.15.90:0/2103184336 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c983d0a3b0 0x55c983cfb9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:47:12.376 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.381+0000 7f4982a00640 1 --2- 172.21.15.90:0/2103184336 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c983d017d0 0x55c983d27070 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f4970010fd0 tx=0x7f4970005880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8d76a7f09c3ef9ab server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:47:12.376 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.381+0000 7f496fe00640 1 -- 172.21.15.90:0/2103184336 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4970020030 con 0x55c983d017d0 2024-04-02T14:47:12.376 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.381+0000 7f496fe00640 1 -- 172.21.15.90:0/2103184336 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f497009ccc0 con 0x55c983d017d0 2024-04-02T14:47:12.376 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.381+0000 7f496fe00640 1 -- 172.21.15.90:0/2103184336 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49700a5230 con 0x55c983d017d0 2024-04-02T14:47:12.377 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.382+0000 7f498d179740 1 -- 172.21.15.90:0/2103184336 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c983d017d0 msgr2=0x55c983d27070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:12.377 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.382+0000 7f498d179740 1 --2- 172.21.15.90:0/2103184336 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c983d017d0 0x55c983d27070 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f4970010fd0 tx=0x7f4970005880 comp rx=0 tx=0).stop 2024-04-02T14:47:12.377 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.382+0000 7f498d179740 1 -- 172.21.15.90:0/2103184336 shutdown_connections 2024-04-02T14:47:12.377 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.382+0000 7f498d179740 1 --2- 172.21.15.90:0/2103184336 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c983d0a3b0 0x55c983cfb9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:12.377 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.382+0000 7f498d179740 1 --2- 172.21.15.90:0/2103184336 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c983d017d0 0x55c983d27070 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:12.377 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.382+0000 7f498d179740 1 --2- 172.21.15.90:0/2103184336 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c983d00eb0 0x55c983d09aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:12.378 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.382+0000 7f498d179740 1 -- 172.21.15.90:0/2103184336 >> 172.21.15.90:0/2103184336 conn(0x55c983cfb5e0 msgr2=0x55c983d01cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:47:12.379 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.383+0000 7f498d179740 1 -- 172.21.15.90:0/2103184336 shutdown_connections 2024-04-02T14:47:12.379 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.383+0000 7f498d179740 1 -- 172.21.15.90:0/2103184336 wait complete. 2024-04-02T14:47:12.379 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.383+0000 7f498d179740 1 -- start start 2024-04-02T14:47:12.379 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.384+0000 7f498d179740 1 Processor -- start 2024-04-02T14:47:12.380 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.386+0000 7f498d179740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c983d00eb0 0x55c983d23870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:12.380 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.386+0000 7f498d179740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c983d017d0 0x55c983d03cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:12.381 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.386+0000 7f498d179740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c983d0a3b0 0x55c983d04200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:12.381 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.386+0000 7f498d179740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f496400fda0 con 0x55c983d00eb0 2024-04-02T14:47:12.381 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.386+0000 7f498d179740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f496400f6d0 con 0x55c983d0a3b0 2024-04-02T14:47:12.381 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.386+0000 7f498d179740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f496400c930 con 0x55c983d017d0 2024-04-02T14:47:12.381 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.386+0000 7f498d179740 10 client.0 _ll_register_callbacks cb 0x7f497402cda0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:47:12.381 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.386+0000 7f4982a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c983d017d0 0x55c983d03cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:12.382 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.386+0000 7f4982a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c983d017d0 0x55c983d03cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:35146/0 (socket says 172.21.15.90:35146) 2024-04-02T14:47:12.382 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.387+0000 7f4982a00640 1 -- 172.21.15.90:0/1662807314 learned_addr learned my addr 172.21.15.90:0/1662807314 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:47:12.382 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.387+0000 7f4983e00640 1 --2- 172.21.15.90:0/1662807314 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c983d0a3b0 0x55c983d04200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:12.382 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.387+0000 7f4983400640 1 --2- 172.21.15.90:0/1662807314 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c983d00eb0 0x55c983d23870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:12.382 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.387+0000 7f4983e00640 1 -- 172.21.15.90:0/1662807314 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c983d017d0 msgr2=0x55c983d03cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:47:12.382 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.387+0000 7f4983e00640 1 --2- 172.21.15.90:0/1662807314 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c983d017d0 0x55c983d03cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:12.382 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.387+0000 7f4983e00640 1 -- 172.21.15.90:0/1662807314 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c983d00eb0 msgr2=0x55c983d23870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:47:12.382 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.387+0000 7f4983e00640 1 --2- 172.21.15.90:0/1662807314 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c983d00eb0 0x55c983d23870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:12.382 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.387+0000 7f4983e00640 1 -- 172.21.15.90:0/1662807314 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c983d048c0 con 0x55c983d0a3b0 2024-04-02T14:47:12.382 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.387+0000 7f4982a00640 1 --2- 172.21.15.90:0/1662807314 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c983d017d0 0x55c983d03cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:47:12.382 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.387+0000 7f4983400640 1 --2- 172.21.15.90:0/1662807314 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c983d00eb0 0x55c983d23870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:47:12.383 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.387+0000 7f4983e00640 1 --2- 172.21.15.90:0/1662807314 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c983d0a3b0 0x55c983d04200 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f497402c170 tx=0x7f49740057f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ec7bd19e38123661 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:47:12.383 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.387+0000 7f496d600640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:47:12.384 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.388+0000 7f496d600640 1 -- 172.21.15.90:0/1662807314 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4974042040 con 0x55c983d0a3b0 2024-04-02T14:47:12.384 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.388+0000 7f496d600640 1 -- 172.21.15.90:0/1662807314 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 8594+0+0 (secure 0 0 0) 0x7f4974047a30 con 0x55c983d0a3b0 2024-04-02T14:47:12.384 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.388+0000 7f498d179740 1 -- 172.21.15.90:0/1662807314 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f497009ccc0 con 0x55c983d0a3b0 2024-04-02T14:47:12.384 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.389+0000 7f496d600640 1 -- 172.21.15.90:0/1662807314 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f4974000e60 con 0x55c983d0a3b0 2024-04-02T14:47:12.384 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.389+0000 7f498d179740 10 client.29252 fetch_fsmap learned FSMap version 27 2024-04-02T14:47:12.384 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.389+0000 7f498d179740 1 -- 172.21.15.90:0/1662807314 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55c983d91770 con 0x55c983d0a3b0 2024-04-02T14:47:12.384 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.389+0000 7f496d600640 1 -- 172.21.15.90:0/1662807314 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f497009ccc0 con 0x55c983d0a3b0 2024-04-02T14:47:12.384 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.390+0000 7f496d600640 1 -- 172.21.15.90:0/1662807314 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f497405a070 con 0x55c983d0a3b0 2024-04-02T14:47:12.384 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.390+0000 7f498d179740 10 client.29252 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:47:12.385 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.390+0000 7f498d179740 10 client.29252 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-02T14:47:12.385 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.390+0000 7f498d179740 20 client.29252 populate_metadata read hostname 'smithi090' 2024-04-02T14:47:12.385 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.390+0000 7f498d179740 1 --2- 172.21.15.90:0/1662807314 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55c983d2b550 0x55c983d93e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:12.385 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.390+0000 7f498d179740 4 client.29252 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-02T14:47:12.385 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.390+0000 7f498d179740 1 -- 172.21.15.90:0/1662807314 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55c983c732b0 con 0x55c983d2b550 2024-04-02T14:47:12.385 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.390+0000 7f4983400640 1 --2- 172.21.15.90:0/1662807314 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55c983d2b550 0x55c983d93e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:12.385 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.391+0000 7f4983400640 1 --2- 172.21.15.90:0/1662807314 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55c983d2b550 0x55c983d93e80 crc :-1 s=READY pgs=317 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=58580ac796f89562 server_cookie=cefa16c64df58dad in_seq=0 out_seq=0 2024-04-02T14:47:12.385 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.391+0000 7f496d600640 10 client.29252 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:47:12.407 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.411+0000 7f496d600640 1 -- 172.21.15.90:0/1662807314 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55c983c732b0 con 0x55c983d2b550 2024-04-02T14:47:12.407 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.411+0000 7f496d600640 10 client.29252 handle_command_reply: tid=0 2024-04-02T14:47:12.407 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.412+0000 7f498d179740 1 client.29252 shutdown 2024-04-02T14:47:12.407 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.412+0000 7f498d179740 1 -- 172.21.15.90:0/1662807314 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c983d0a3b0 msgr2=0x55c983d04200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:12.407 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.412+0000 7f498d179740 1 --2- 172.21.15.90:0/1662807314 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c983d0a3b0 0x55c983d04200 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f497402c170 tx=0x7f49740057f0 comp rx=0 tx=0).stop 2024-04-02T14:47:12.407 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.412+0000 7f498d179740 1 -- 172.21.15.90:0/1662807314 shutdown_connections 2024-04-02T14:47:12.407 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.412+0000 7f498d179740 1 --2- 172.21.15.90:0/1662807314 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55c983d2b550 0x55c983d93e80 crc :-1 s=READY pgs=317 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:12.407 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.412+0000 7f498d179740 1 --2- 172.21.15.90:0/1662807314 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c983d0a3b0 0x55c983d04200 unknown :-1 s=CLOSED pgs=862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:12.408 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.412+0000 7f498d179740 1 --2- 172.21.15.90:0/1662807314 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c983d017d0 0x55c983d03cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:12.408 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.412+0000 7f498d179740 1 --2- 172.21.15.90:0/1662807314 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c983d00eb0 0x55c983d23870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:12.408 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.412+0000 7f498d179740 1 -- 172.21.15.90:0/1662807314 >> 172.21.15.90:0/1662807314 conn(0x55c983cfb5e0 msgr2=0x55c983cfb9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:47:12.408 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.412+0000 7f496d600640 0 client.29252 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:47:12.408 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.413+0000 7f498d179740 1 -- 172.21.15.90:0/1662807314 shutdown_connections 2024-04-02T14:47:12.408 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.413+0000 7f498d179740 1 -- 172.21.15.90:0/1662807314 wait complete. 2024-04-02T14:47:12.408 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.413+0000 7f498d179740 20 client.29252 trim_cache size 0 max 16384 2024-04-02T14:47:12.408 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:{ 2024-04-02T14:47:12.408 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "message": "", 2024-04-02T14:47:12.408 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "return_code": 0 2024-04-02T14:47:12.409 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:} 2024-04-02T14:47:12.409 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.413+0000 7f498a400640 1 -- 172.21.15.90:0/239137831 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f49680a3e40 msgr2=0x7f49680a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:12.409 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.413+0000 7f498a400640 1 --2- 172.21.15.90:0/239137831 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f49680a3e40 0x7f49680a6300 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x55c983d00980 tx=0x7f497c0097c0 comp rx=0 tx=0).stop 2024-04-02T14:47:12.409 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.413+0000 7f498a400640 1 -- 172.21.15.90:0/239137831 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4984076950 msgr2=0x7f49841a4bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:12.409 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.413+0000 7f498a400640 1 --2- 172.21.15.90:0/239137831 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4984076950 0x7f49841a4bc0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f497000d950 tx=0x7f497000de10 comp rx=0 tx=0).stop 2024-04-02T14:47:12.410 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.414+0000 7f498a400640 1 -- 172.21.15.90:0/239137831 shutdown_connections 2024-04-02T14:47:12.410 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.414+0000 7f498a400640 1 --2- 172.21.15.90:0/239137831 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f49680a3e40 0x7f49680a6300 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:12.410 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.414+0000 7f498a400640 1 --2- 172.21.15.90:0/239137831 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4984112970 0x7f498419e740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:12.410 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.414+0000 7f498a400640 1 --2- 172.21.15.90:0/239137831 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4984076950 0x7f49841a4bc0 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:12.410 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.414+0000 7f498a400640 1 --2- 172.21.15.90:0/239137831 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4984076030 0x7f49841a4680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:12.410 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.414+0000 7f498a400640 1 -- 172.21.15.90:0/239137831 >> 172.21.15.90:0/239137831 conn(0x7f49840ff7c0 msgr2=0x7f4984101bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:47:12.410 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.414+0000 7f498a400640 1 -- 172.21.15.90:0/239137831 shutdown_connections 2024-04-02T14:47:12.410 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:12.414+0000 7f498a400640 1 -- 172.21.15.90:0/239137831 wait complete. 2024-04-02T14:47:13.607 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-02T14:47:13.607 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-02T14:47:13.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.884+0000 7f06e3000640 1 -- 172.21.15.90:0/3230895043 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f06dc068d00 msgr2=0x7f06dc069180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:13.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.884+0000 7f06e3000640 1 --2- 172.21.15.90:0/3230895043 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f06dc068d00 0x7f06dc069180 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f06c8009890 tx=0x7f06c802bb70 comp rx=0 tx=0).stop 2024-04-02T14:47:13.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.885+0000 7f06e3000640 1 -- 172.21.15.90:0/3230895043 shutdown_connections 2024-04-02T14:47:13.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.885+0000 7f06e3000640 1 --2- 172.21.15.90:0/3230895043 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f06dc0696c0 0x7f06dc10e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:13.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.885+0000 7f06e3000640 1 --2- 172.21.15.90:0/3230895043 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f06dc068d00 0x7f06dc069180 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:13.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.885+0000 7f06e3000640 1 --2- 172.21.15.90:0/3230895043 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f06dc107b50 0x7f06dc107f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:13.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.885+0000 7f06e3000640 1 -- 172.21.15.90:0/3230895043 >> 172.21.15.90:0/3230895043 conn(0x7f06dc077b00 msgr2=0x7f06dc077f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:47:13.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.885+0000 7f06e3000640 1 -- 172.21.15.90:0/3230895043 shutdown_connections 2024-04-02T14:47:13.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.886+0000 7f06e3000640 1 -- 172.21.15.90:0/3230895043 wait complete. 2024-04-02T14:47:13.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.887+0000 7f06e3000640 1 Processor -- start 2024-04-02T14:47:13.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.887+0000 7f06e3000640 1 -- start start 2024-04-02T14:47:13.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.887+0000 7f06e3000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f06dc068d00 0x7f06dc19a590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:13.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.887+0000 7f06e3000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f06dc0696c0 0x7f06dc19aad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:13.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.887+0000 7f06e3000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f06dc107b50 0x7f06dc19ee50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:13.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.887+0000 7f06e3000640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f06dc111e70 con 0x7f06dc0696c0 2024-04-02T14:47:13.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.888+0000 7f06e3000640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f06dc111cf0 con 0x7f06dc107b50 2024-04-02T14:47:13.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.888+0000 7f06e3000640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f06dc111ff0 con 0x7f06dc068d00 2024-04-02T14:47:13.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.888+0000 7f06dbe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f06dc068d00 0x7f06dc19a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:13.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.888+0000 7f06db400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f06dc0696c0 0x7f06dc19aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:13.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.888+0000 7f06dbe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f06dc068d00 0x7f06dc19a590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:59450/0 (socket says 172.21.15.90:59450) 2024-04-02T14:47:13.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.888+0000 7f06dbe00640 1 -- 172.21.15.90:0/1979828414 learned_addr learned my addr 172.21.15.90:0/1979828414 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:47:13.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.888+0000 7f06e1000640 1 --2- 172.21.15.90:0/1979828414 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f06dc107b50 0x7f06dc19ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:13.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.888+0000 7f06db400640 1 -- 172.21.15.90:0/1979828414 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f06dc068d00 msgr2=0x7f06dc19a590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:13.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.888+0000 7f06db400640 1 --2- 172.21.15.90:0/1979828414 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f06dc068d00 0x7f06dc19a590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:13.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.888+0000 7f06db400640 1 -- 172.21.15.90:0/1979828414 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f06dc107b50 msgr2=0x7f06dc19ee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:13.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.888+0000 7f06db400640 1 --2- 172.21.15.90:0/1979828414 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f06dc107b50 0x7f06dc19ee50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:13.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.888+0000 7f06db400640 1 -- 172.21.15.90:0/1979828414 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06dc19f580 con 0x7f06dc0696c0 2024-04-02T14:47:13.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.888+0000 7f06e1000640 1 --2- 172.21.15.90:0/1979828414 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f06dc107b50 0x7f06dc19ee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:47:13.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.888+0000 7f06dbe00640 1 --2- 172.21.15.90:0/1979828414 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f06dc068d00 0x7f06dc19a590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:47:13.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.889+0000 7f06db400640 1 --2- 172.21.15.90:0/1979828414 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f06dc0696c0 0x7f06dc19aad0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f06c8037040 tx=0x7f06c8004360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:47:13.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.889+0000 7f06d8c00640 1 -- 172.21.15.90:0/1979828414 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06c8041020 con 0x7f06dc0696c0 2024-04-02T14:47:13.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.889+0000 7f06e3000640 1 -- 172.21.15.90:0/1979828414 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06dc19f810 con 0x7f06dc0696c0 2024-04-02T14:47:13.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.889+0000 7f06d8c00640 1 -- 172.21.15.90:0/1979828414 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f06c802c9f0 con 0x7f06dc0696c0 2024-04-02T14:47:13.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.889+0000 7f06d8c00640 1 -- 172.21.15.90:0/1979828414 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06c8039040 con 0x7f06dc0696c0 2024-04-02T14:47:13.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.889+0000 7f06e3000640 1 -- 172.21.15.90:0/1979828414 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06dc1a7100 con 0x7f06dc0696c0 2024-04-02T14:47:13.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.893+0000 7f06e5daa740 1 Processor -- start 2024-04-02T14:47:13.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.894+0000 7f06e5daa740 1 -- start start 2024-04-02T14:47:13.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.894+0000 7f06d8c00640 1 -- 172.21.15.90:0/1979828414 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f06c8044020 con 0x7f06dc0696c0 2024-04-02T14:47:13.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.894+0000 7f06e5daa740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c850ed17b0 0x55c850ed1b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:13.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.894+0000 7f06e5daa740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c850ed20d0 0x55c850eda770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:13.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.894+0000 7f06e5daa740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c850edacb0 0x55c850edd0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:13.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.894+0000 7f06e5daa740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55c850e35770 con 0x55c850edacb0 2024-04-02T14:47:13.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.894+0000 7f06e5daa740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55c850ddd700 con 0x55c850ed20d0 2024-04-02T14:47:13.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.894+0000 7f06e5daa740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55c850e31720 con 0x55c850ed17b0 2024-04-02T14:47:13.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.894+0000 7f06e1000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c850ed17b0 0x55c850ed1b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:13.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.894+0000 7f06e1000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c850ed17b0 0x55c850ed1b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:59462/0 (socket says 172.21.15.90:59462) 2024-04-02T14:47:13.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.894+0000 7f06e1000640 1 -- 172.21.15.90:0/4260367692 learned_addr learned my addr 172.21.15.90:0/4260367692 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:47:13.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.895+0000 7f06dbe00640 1 --2- 172.21.15.90:0/4260367692 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c850ed20d0 0x55c850eda770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:13.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.895+0000 7f06e1000640 1 -- 172.21.15.90:0/4260367692 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c850ed20d0 msgr2=0x55c850eda770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:47:13.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.895+0000 7f06e1000640 1 --2- 172.21.15.90:0/4260367692 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c850ed20d0 0x55c850eda770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:13.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.895+0000 7f06e1000640 1 -- 172.21.15.90:0/4260367692 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c850edacb0 msgr2=0x55c850edd0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:47:13.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.895+0000 7f06db400640 1 --2- 172.21.15.90:0/4260367692 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c850edacb0 0x55c850edd0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:13.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.895+0000 7f06e1000640 1 --2- 172.21.15.90:0/4260367692 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c850edacb0 0x55c850edd0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:13.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.895+0000 7f06e1000640 1 -- 172.21.15.90:0/4260367692 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c850c137d0 con 0x55c850ed17b0 2024-04-02T14:47:13.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.895+0000 7f06dbe00640 1 --2- 172.21.15.90:0/4260367692 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c850ed20d0 0x55c850eda770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:47:13.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.895+0000 7f06db400640 1 --2- 172.21.15.90:0/4260367692 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c850edacb0 0x55c850edd0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:47:13.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.895+0000 7f06e1000640 1 --2- 172.21.15.90:0/4260367692 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c850ed17b0 0x55c850ed1b90 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f06d4011810 tx=0x7f06d4011cd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a7d2833bbc041b08 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:47:13.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.897+0000 7f06d8c00640 1 --2- 172.21.15.90:0/1979828414 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f06c00a3d20 0x7f06c00a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:13.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.897+0000 7f06d1600640 1 -- 172.21.15.90:0/4260367692 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06d4011310 con 0x55c850ed17b0 2024-04-02T14:47:13.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.897+0000 7f06d1600640 1 -- 172.21.15.90:0/4260367692 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f06d400f9a0 con 0x55c850ed17b0 2024-04-02T14:47:13.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.897+0000 7f06d1600640 1 -- 172.21.15.90:0/4260367692 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06d4005720 con 0x55c850ed17b0 2024-04-02T14:47:13.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.897+0000 7f06d8c00640 1 -- 172.21.15.90:0/1979828414 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 8594+0+0 (secure 0 0 0) 0x7f06c80f1ed0 con 0x7f06dc0696c0 2024-04-02T14:47:13.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.898+0000 7f06dbe00640 1 --2- 172.21.15.90:0/1979828414 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f06c00a3d20 0x7f06c00a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:13.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.898+0000 7f06e5daa740 1 -- 172.21.15.90:0/4260367692 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c850ed17b0 msgr2=0x55c850ed1b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:13.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.898+0000 7f06e5daa740 1 --2- 172.21.15.90:0/4260367692 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c850ed17b0 0x55c850ed1b90 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f06d4011810 tx=0x7f06d4011cd0 comp rx=0 tx=0).stop 2024-04-02T14:47:13.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.899+0000 7f06dbe00640 1 --2- 172.21.15.90:0/1979828414 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f06c00a3d20 0x7f06c00a61e0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x55c850ed1280 tx=0x7f06cc009260 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:47:13.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.899+0000 7f06e5daa740 1 -- 172.21.15.90:0/4260367692 shutdown_connections 2024-04-02T14:47:13.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.899+0000 7f06e5daa740 1 --2- 172.21.15.90:0/4260367692 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c850edacb0 0x55c850edd0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:13.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.899+0000 7f06e5daa740 1 --2- 172.21.15.90:0/4260367692 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c850ed20d0 0x55c850eda770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:13.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.899+0000 7f06e5daa740 1 --2- 172.21.15.90:0/4260367692 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c850ed17b0 0x55c850ed1b90 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:13.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.899+0000 7f06e5daa740 1 -- 172.21.15.90:0/4260367692 >> 172.21.15.90:0/4260367692 conn(0x55c850ecbee0 msgr2=0x55c850ecc2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:47:13.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.899+0000 7f06e5daa740 1 -- 172.21.15.90:0/4260367692 shutdown_connections 2024-04-02T14:47:13.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.900+0000 7f06e5daa740 1 -- 172.21.15.90:0/4260367692 wait complete. 2024-04-02T14:47:13.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.901+0000 7f06e5daa740 1 -- start start 2024-04-02T14:47:13.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.901+0000 7f06e5daa740 1 Processor -- start 2024-04-02T14:47:13.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.902+0000 7f06e5daa740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c850ed17b0 0x55c850ed7cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:13.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.902+0000 7f06e5daa740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c850ed20d0 0x55c850ed81f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:13.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.902+0000 7f06e5daa740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c850edacb0 0x55c850ed8730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:13.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.902+0000 7f06dbe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c850ed17b0 0x55c850ed7cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:13.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.902+0000 7f06dbe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c850ed17b0 0x55c850ed7cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:59478/0 (socket says 172.21.15.90:59478) 2024-04-02T14:47:13.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.903+0000 7f06db400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c850ed20d0 0x55c850ed81f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:13.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.903+0000 7f06dbe00640 1 -- 172.21.15.90:0/3212271517 learned_addr learned my addr 172.21.15.90:0/3212271517 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:47:13.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.903+0000 7f06e5daa740 1 -- 172.21.15.90:0/3212271517 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55c850eddaa0 con 0x55c850ed20d0 2024-04-02T14:47:13.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.903+0000 7f06e5daa740 1 -- 172.21.15.90:0/3212271517 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55c850df0ac0 con 0x55c850edacb0 2024-04-02T14:47:13.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.903+0000 7f06e5daa740 1 -- 172.21.15.90:0/3212271517 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55c850eddc20 con 0x55c850ed17b0 2024-04-02T14:47:13.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.903+0000 7f06e1000640 1 --2- 172.21.15.90:0/3212271517 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c850edacb0 0x55c850ed8730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:13.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.904+0000 7f06db400640 1 -- 172.21.15.90:0/3212271517 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c850ed17b0 msgr2=0x55c850ed7cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:47:13.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.904+0000 7f06db400640 1 --2- 172.21.15.90:0/3212271517 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c850ed17b0 0x55c850ed7cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:13.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.904+0000 7f06db400640 1 -- 172.21.15.90:0/3212271517 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c850edacb0 msgr2=0x55c850ed8730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:47:13.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.904+0000 7f06db400640 1 --2- 172.21.15.90:0/3212271517 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c850edacb0 0x55c850ed8730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:13.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.904+0000 7f06db400640 1 -- 172.21.15.90:0/3212271517 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c850b493f0 con 0x55c850ed20d0 2024-04-02T14:47:13.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.904+0000 7f06e5daa740 10 client.0 _ll_register_callbacks cb 0x55c850df3180 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:47:13.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.904+0000 7f06e5daa740 1 -- 172.21.15.90:0/3212271517 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f06d400f9a0 con 0x55c850ed20d0 2024-04-02T14:47:13.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.904+0000 7f06db400640 1 --2- 172.21.15.90:0/3212271517 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c850ed20d0 0x55c850ed81f0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f06c80ef6e0 tx=0x7f06c8032640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec100bfef2cab8fd server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:47:13.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.904+0000 7f06c6000640 10 client.18174 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-02T14:47:13.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.906+0000 7f06c6000640 1 -- 172.21.15.90:0/3212271517 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06c80bd890 con 0x55c850ed20d0 2024-04-02T14:47:13.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.907+0000 7f06c6000640 1 -- 172.21.15.90:0/3212271517 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 8594+0+0 (secure 0 0 0) 0x7f06c80ba5a0 con 0x55c850ed20d0 2024-04-02T14:47:13.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.907+0000 7f06c6000640 1 -- 172.21.15.90:0/3212271517 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f06c80be630 con 0x55c850ed20d0 2024-04-02T14:47:13.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.909+0000 7f06e5daa740 10 client.18174 fetch_fsmap learned FSMap version 27 2024-04-02T14:47:13.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.909+0000 7f06e5daa740 1 -- 172.21.15.90:0/3212271517 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({fsmap=27}) -- 0x55c850b090f0 con 0x55c850ed20d0 2024-04-02T14:47:13.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.911+0000 7f06c6000640 1 -- 172.21.15.90:0/3212271517 <== mon.0 v2:172.21.15.90:3300/0 4 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f06c80c1030 con 0x55c850ed20d0 2024-04-02T14:47:13.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.911+0000 7f06e5daa740 10 client.18174 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:47:13.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.911+0000 7f06e5daa740 10 client.18174 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:47:13.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.911+0000 7f06e5daa740 20 client.18174 populate_metadata read hostname 'smithi090' 2024-04-02T14:47:13.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.911+0000 7f06e5daa740 1 --2- 172.21.15.90:0/3212271517 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55c850eea920 0x55c850eecdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:13.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.911+0000 7f06e5daa740 4 client.18174 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:47:13.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.911+0000 7f06e5daa740 1 -- 172.21.15.90:0/3212271517 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c850dd6630 con 0x55c850eea920 2024-04-02T14:47:13.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.916+0000 7f06dbe00640 1 --2- 172.21.15.90:0/3212271517 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55c850eea920 0x55c850eecdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:13.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.917+0000 7f06dbe00640 1 --2- 172.21.15.90:0/3212271517 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55c850eea920 0x55c850eecdb0 crc :-1 s=READY pgs=318 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7172a27af449a52b server_cookie=b12b1e9f47253b87 in_seq=0 out_seq=0 2024-04-02T14:47:13.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.917+0000 7f06c6000640 10 client.18174 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:47:13.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.920+0000 7f06c6000640 1 -- 172.21.15.90:0/3212271517 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c850dd6630 con 0x55c850eea920 2024-04-02T14:47:13.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.920+0000 7f06c6000640 10 client.18174 handle_command_reply: tid=0 2024-04-02T14:47:13.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.920+0000 7f06e5daa740 1 client.18174 shutdown 2024-04-02T14:47:13.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.921+0000 7f06e5daa740 1 -- 172.21.15.90:0/3212271517 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c850ed20d0 msgr2=0x55c850ed81f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:13.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.921+0000 7f06e5daa740 1 --2- 172.21.15.90:0/3212271517 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c850ed20d0 0x55c850ed81f0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f06c80ef6e0 tx=0x7f06c8032640 comp rx=0 tx=0).stop 2024-04-02T14:47:13.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.921+0000 7f06e5daa740 1 -- 172.21.15.90:0/3212271517 shutdown_connections 2024-04-02T14:47:13.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.921+0000 7f06e5daa740 1 --2- 172.21.15.90:0/3212271517 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55c850eea920 0x55c850eecdb0 crc :-1 s=READY pgs=318 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:13.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.921+0000 7f06e5daa740 1 --2- 172.21.15.90:0/3212271517 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c850edacb0 0x55c850ed8730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:13.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.921+0000 7f06e5daa740 1 --2- 172.21.15.90:0/3212271517 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c850ed20d0 0x55c850ed81f0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:13.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.921+0000 7f06e5daa740 1 --2- 172.21.15.90:0/3212271517 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c850ed17b0 0x55c850ed7cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:13.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.921+0000 7f06e5daa740 1 -- 172.21.15.90:0/3212271517 >> 172.21.15.90:0/3212271517 conn(0x55c850ecbee0 msgr2=0x55c850ecdc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:47:13.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.921+0000 7f06c6000640 0 client.18174 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:47:13.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.921+0000 7f06e5daa740 1 -- 172.21.15.90:0/3212271517 shutdown_connections 2024-04-02T14:47:13.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.922+0000 7f06e5daa740 1 -- 172.21.15.90:0/3212271517 wait complete. 2024-04-02T14:47:13.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.925+0000 7f06e5daa740 20 client.18174 trim_cache size 0 max 16384 2024-04-02T14:47:13.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.966+0000 7f06e5daa740 1 Processor -- start 2024-04-02T14:47:13.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.966+0000 7f06e5daa740 1 -- start start 2024-04-02T14:47:13.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.966+0000 7f06e5daa740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c850ed17b0 0x55c850ee6960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:13.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.966+0000 7f06e5daa740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c850ed20d0 0x55c850ed8500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:13.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.967+0000 7f06e5daa740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c850edacb0 0x55c850ef26c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:13.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.967+0000 7f06e5daa740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f06dc06f980 con 0x55c850ed17b0 2024-04-02T14:47:13.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.967+0000 7f06e5daa740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f06dc0714c0 con 0x55c850ed20d0 2024-04-02T14:47:13.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.967+0000 7f06e5daa740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f06dc070050 con 0x55c850edacb0 2024-04-02T14:47:13.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.967+0000 7f06db400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c850ed20d0 0x55c850ed8500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:13.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.967+0000 7f06db400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c850ed20d0 0x55c850ed8500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:52956/0 (socket says 172.21.15.90:52956) 2024-04-02T14:47:13.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.967+0000 7f06db400640 1 -- 172.21.15.90:0/2725274034 learned_addr learned my addr 172.21.15.90:0/2725274034 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:47:13.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.968+0000 7f06e1000640 1 --2- 172.21.15.90:0/2725274034 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c850edacb0 0x55c850ef26c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:13.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.968+0000 7f06dbe00640 1 --2- 172.21.15.90:0/2725274034 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c850ed17b0 0x55c850ee6960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:13.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.968+0000 7f06db400640 1 -- 172.21.15.90:0/2725274034 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c850edacb0 msgr2=0x55c850ef26c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:47:13.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.968+0000 7f06db400640 1 --2- 172.21.15.90:0/2725274034 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c850edacb0 0x55c850ef26c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:13.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.968+0000 7f06db400640 1 -- 172.21.15.90:0/2725274034 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c850ed17b0 msgr2=0x55c850ee6960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:47:13.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.968+0000 7f06db400640 1 --2- 172.21.15.90:0/2725274034 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c850ed17b0 0x55c850ee6960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:13.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.968+0000 7f06db400640 1 -- 172.21.15.90:0/2725274034 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c850eed430 con 0x55c850ed20d0 2024-04-02T14:47:13.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.969+0000 7f06dbe00640 1 --2- 172.21.15.90:0/2725274034 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c850ed17b0 0x55c850ee6960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:47:13.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.969+0000 7f06e1000640 1 --2- 172.21.15.90:0/2725274034 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c850edacb0 0x55c850ef26c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:47:13.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.969+0000 7f06db400640 1 --2- 172.21.15.90:0/2725274034 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c850ed20d0 0x55c850ed8500 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f06c802c0b0 tx=0x7f06c80a7070 comp rx=0 tx=0).ready entity=mon.2 client_cookie=50905f5faa8ac957 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:47:13.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.970+0000 7f06d0c00640 1 -- 172.21.15.90:0/2725274034 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06c80c6ba0 con 0x55c850ed20d0 2024-04-02T14:47:13.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.970+0000 7f06d0c00640 1 -- 172.21.15.90:0/2725274034 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f06d400f9a0 con 0x55c850ed20d0 2024-04-02T14:47:13.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.970+0000 7f06d0c00640 1 -- 172.21.15.90:0/2725274034 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06c80c6970 con 0x55c850ed20d0 2024-04-02T14:47:13.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.971+0000 7f06e5daa740 1 -- 172.21.15.90:0/2725274034 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c850ed20d0 msgr2=0x55c850ed8500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:13.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.971+0000 7f06e5daa740 1 --2- 172.21.15.90:0/2725274034 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c850ed20d0 0x55c850ed8500 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f06c802c0b0 tx=0x7f06c80a7070 comp rx=0 tx=0).stop 2024-04-02T14:47:13.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.971+0000 7f06e5daa740 1 -- 172.21.15.90:0/2725274034 shutdown_connections 2024-04-02T14:47:13.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.971+0000 7f06e5daa740 1 --2- 172.21.15.90:0/2725274034 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c850edacb0 0x55c850ef26c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:13.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.971+0000 7f06e5daa740 1 --2- 172.21.15.90:0/2725274034 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c850ed20d0 0x55c850ed8500 secure :-1 s=CLOSED pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f06c802c0b0 tx=0x7f06c80a7070 comp rx=0 tx=0).stop 2024-04-02T14:47:13.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.971+0000 7f06e5daa740 1 --2- 172.21.15.90:0/2725274034 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c850ed17b0 0x55c850ee6960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:13.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.971+0000 7f06e5daa740 1 -- 172.21.15.90:0/2725274034 >> 172.21.15.90:0/2725274034 conn(0x55c850ecbee0 msgr2=0x55c850eca6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:47:13.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.973+0000 7f06e5daa740 1 -- 172.21.15.90:0/2725274034 shutdown_connections 2024-04-02T14:47:13.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.973+0000 7f06e5daa740 1 -- 172.21.15.90:0/2725274034 wait complete. 2024-04-02T14:47:13.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.974+0000 7f06e5daa740 1 -- start start 2024-04-02T14:47:13.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.974+0000 7f06e5daa740 1 Processor -- start 2024-04-02T14:47:13.969 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.975+0000 7f06e5daa740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c850ed17b0 0x55c850edea30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:13.969 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.975+0000 7f06e5daa740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c850edacb0 0x55c850edef70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:13.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.975+0000 7f06e5daa740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c850edf4b0 0x55c850f4ff30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:13.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.976+0000 7f06e5daa740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f06dc070df0 con 0x55c850edacb0 2024-04-02T14:47:13.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.976+0000 7f06e5daa740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f06dc070720 con 0x55c850edf4b0 2024-04-02T14:47:13.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.976+0000 7f06e5daa740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f06dc06d980 con 0x55c850ed17b0 2024-04-02T14:47:13.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.976+0000 7f06e5daa740 10 client.0 _ll_register_callbacks cb 0x7f06c80ba280 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:47:13.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.976+0000 7f06db400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c850edacb0 0x55c850edef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:13.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.976+0000 7f06db400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c850edacb0 0x55c850edef70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:36920/0 (socket says 172.21.15.90:36920) 2024-04-02T14:47:13.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.976+0000 7f06db400640 1 -- 172.21.15.90:0/2617072609 learned_addr learned my addr 172.21.15.90:0/2617072609 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:47:13.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.976+0000 7f06e1000640 1 --2- 172.21.15.90:0/2617072609 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c850edf4b0 0x55c850f4ff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:13.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.976+0000 7f06db400640 1 -- 172.21.15.90:0/2617072609 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c850ed17b0 msgr2=0x55c850edea30 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-02T14:47:13.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.976+0000 7f06dbe00640 1 --2- 172.21.15.90:0/2617072609 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c850ed17b0 0x55c850edea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:13.972 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.976+0000 7f06db400640 1 --2- 172.21.15.90:0/2617072609 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c850ed17b0 0x55c850edea30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:13.972 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.976+0000 7f06db400640 1 -- 172.21.15.90:0/2617072609 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c850edf4b0 msgr2=0x55c850f4ff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:47:13.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.977+0000 7f06db400640 1 --2- 172.21.15.90:0/2617072609 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c850edf4b0 0x55c850f4ff30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:13.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.977+0000 7f06db400640 1 -- 172.21.15.90:0/2617072609 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c850f50570 con 0x55c850edacb0 2024-04-02T14:47:13.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.977+0000 7f06dbe00640 1 --2- 172.21.15.90:0/2617072609 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c850ed17b0 0x55c850edea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:47:13.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.977+0000 7f06e1000640 1 --2- 172.21.15.90:0/2617072609 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c850edf4b0 0x55c850f4ff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:47:13.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.977+0000 7f06db400640 1 --2- 172.21.15.90:0/2617072609 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c850edacb0 0x55c850edef70 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f06c80ef790 tx=0x7f06c80f20d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9f0707b0912e498 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:47:13.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.977+0000 7f06c6000640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-02T14:47:13.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.978+0000 7f06c6000640 1 -- 172.21.15.90:0/2617072609 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06c80c6910 con 0x55c850edacb0 2024-04-02T14:47:13.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.978+0000 7f06e5daa740 1 -- 172.21.15.90:0/2617072609 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f06d400f9a0 con 0x55c850edacb0 2024-04-02T14:47:13.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.979+0000 7f06c6000640 1 -- 172.21.15.90:0/2617072609 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 8594+0+0 (secure 0 0 0) 0x7f06c80c6710 con 0x55c850edacb0 2024-04-02T14:47:13.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.979+0000 7f06c6000640 1 -- 172.21.15.90:0/2617072609 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f06c80c6910 con 0x55c850edacb0 2024-04-02T14:47:13.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.980+0000 7f06e5daa740 10 client.18177 fetch_fsmap learned FSMap version 27 2024-04-02T14:47:13.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.980+0000 7f06e5daa740 1 -- 172.21.15.90:0/2617072609 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55c850f50570 con 0x55c850edacb0 2024-04-02T14:47:13.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.980+0000 7f06c6000640 1 -- 172.21.15.90:0/2617072609 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f06d400f9a0 con 0x55c850edacb0 2024-04-02T14:47:13.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.981+0000 7f06c6000640 1 -- 172.21.15.90:0/2617072609 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f06c80c1070 con 0x55c850edacb0 2024-04-02T14:47:13.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.981+0000 7f06e5daa740 10 client.18177 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:47:13.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.981+0000 7f06e5daa740 10 client.18177 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:47:13.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.981+0000 7f06e5daa740 20 client.18177 populate_metadata read hostname 'smithi090' 2024-04-02T14:47:13.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.981+0000 7f06e5daa740 1 --2- 172.21.15.90:0/2617072609 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55c850f51b10 0x55c850f53f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:13.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.981+0000 7f06e5daa740 4 client.18177 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-02T14:47:13.976 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.981+0000 7f06e5daa740 1 -- 172.21.15.90:0/2617072609 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55c850e42cb0 con 0x55c850f51b10 2024-04-02T14:47:13.976 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.981+0000 7f06dbe00640 1 --2- 172.21.15.90:0/2617072609 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55c850f51b10 0x55c850f53f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:13.976 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.982+0000 7f06dbe00640 1 --2- 172.21.15.90:0/2617072609 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55c850f51b10 0x55c850f53f00 crc :-1 s=READY pgs=319 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3cc75355114398dc server_cookie=b47ba84a72df8b5b in_seq=0 out_seq=0 2024-04-02T14:47:13.976 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.982+0000 7f06c6000640 10 client.18177 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:47:13.977 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.982+0000 7f06c6000640 1 -- 172.21.15.90:0/2617072609 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55c850e42cb0 con 0x55c850f51b10 2024-04-02T14:47:13.977 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.982+0000 7f06c6000640 10 client.18177 handle_command_reply: tid=0 2024-04-02T14:47:13.977 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.982+0000 7f06e5daa740 1 client.18177 shutdown 2024-04-02T14:47:13.977 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.983+0000 7f06e5daa740 1 -- 172.21.15.90:0/2617072609 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c850edacb0 msgr2=0x55c850edef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:13.977 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.983+0000 7f06e5daa740 1 --2- 172.21.15.90:0/2617072609 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c850edacb0 0x55c850edef70 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f06c80ef790 tx=0x7f06c80f20d0 comp rx=0 tx=0).stop 2024-04-02T14:47:13.978 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.983+0000 7f06e5daa740 1 -- 172.21.15.90:0/2617072609 shutdown_connections 2024-04-02T14:47:13.978 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.983+0000 7f06e5daa740 1 --2- 172.21.15.90:0/2617072609 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55c850f51b10 0x55c850f53f00 crc :-1 s=READY pgs=319 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:13.978 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.983+0000 7f06e5daa740 1 --2- 172.21.15.90:0/2617072609 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c850edf4b0 0x55c850f4ff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:13.978 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.983+0000 7f06e5daa740 1 --2- 172.21.15.90:0/2617072609 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c850edacb0 0x55c850edef70 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:13.978 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.984+0000 7f06c6000640 0 client.18177 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:47:13.978 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.983+0000 7f06e5daa740 1 --2- 172.21.15.90:0/2617072609 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c850ed17b0 0x55c850edea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:13.978 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.984+0000 7f06e5daa740 1 -- 172.21.15.90:0/2617072609 >> 172.21.15.90:0/2617072609 conn(0x55c850ecbee0 msgr2=0x55c850ed8380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:47:13.979 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.984+0000 7f06e5daa740 1 -- 172.21.15.90:0/2617072609 shutdown_connections 2024-04-02T14:47:13.979 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.984+0000 7f06e5daa740 1 -- 172.21.15.90:0/2617072609 wait complete. 2024-04-02T14:47:13.979 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.985+0000 7f06e5daa740 20 client.18177 trim_cache size 0 max 16384 2024-04-02T14:47:13.980 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-02T14:47:13.980 INFO:teuthology.orchestra.run.smithi090.stdout: "status": "no active scrubs running", 2024-04-02T14:47:13.980 INFO:teuthology.orchestra.run.smithi090.stdout: "scrubs": {} 2024-04-02T14:47:13.980 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-02T14:47:13.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.985+0000 7f06e3000640 1 -- 172.21.15.90:0/1979828414 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f06c00a3d20 msgr2=0x7f06c00a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:13.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.985+0000 7f06e3000640 1 --2- 172.21.15.90:0/1979828414 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f06c00a3d20 0x7f06c00a61e0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x55c850ed1280 tx=0x7f06cc009260 comp rx=0 tx=0).stop 2024-04-02T14:47:13.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.985+0000 7f06e3000640 1 -- 172.21.15.90:0/1979828414 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f06dc0696c0 msgr2=0x7f06dc19aad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:13.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.985+0000 7f06e3000640 1 --2- 172.21.15.90:0/1979828414 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f06dc0696c0 0x7f06dc19aad0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f06c8037040 tx=0x7f06c8004360 comp rx=0 tx=0).stop 2024-04-02T14:47:13.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.985+0000 7f06e3000640 1 -- 172.21.15.90:0/1979828414 shutdown_connections 2024-04-02T14:47:13.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.985+0000 7f06e3000640 1 --2- 172.21.15.90:0/1979828414 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f06c00a3d20 0x7f06c00a61e0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:13.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.985+0000 7f06e3000640 1 --2- 172.21.15.90:0/1979828414 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f06dc107b50 0x7f06dc19ee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:13.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.985+0000 7f06e3000640 1 --2- 172.21.15.90:0/1979828414 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f06dc0696c0 0x7f06dc19aad0 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:13.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.985+0000 7f06e3000640 1 --2- 172.21.15.90:0/1979828414 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f06dc068d00 0x7f06dc19a590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:13.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.985+0000 7f06e3000640 1 -- 172.21.15.90:0/1979828414 >> 172.21.15.90:0/1979828414 conn(0x7f06dc077b00 msgr2=0x7f06dc10e650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:47:13.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.986+0000 7f06e3000640 1 -- 172.21.15.90:0/1979828414 shutdown_connections 2024-04-02T14:47:13.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:13.986+0000 7f06e3000640 1 -- 172.21.15.90:0/1979828414 wait complete. 2024-04-02T14:47:13.984 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:47:13 smithi090 ceph-mon[31674]: pgmap v384: 97 pgs: 40 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 35 active+clean; 4.4 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 17 MiB/s wr, 1.50k op/s 2024-04-02T14:47:14.005 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-02T14:47:14.005 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T14:47:14.104 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:47:13 smithi092 ceph-mon[36497]: pgmap v384: 97 pgs: 40 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 35 active+clean; 4.4 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 17 MiB/s wr, 1.50k op/s 2024-04-02T14:47:14.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:47:13 smithi177 ceph-mon[36431]: pgmap v384: 97 pgs: 40 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 35 active+clean; 4.4 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 17 MiB/s wr, 1.50k op/s 2024-04-02T14:47:14.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:14.284+0000 7f4a9c000640 1 -- 172.21.15.90:0/4031314895 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4a94103f20 msgr2=0x7f4a94104300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:14.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:14.284+0000 7f4a9c000640 1 --2- 172.21.15.90:0/4031314895 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4a94103f20 0x7f4a94104300 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f4a88009ca0 tx=0x7f4a8802c100 comp rx=0 tx=0).stop 2024-04-02T14:47:14.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:14.284+0000 7f4a9c000640 1 -- 172.21.15.90:0/4031314895 shutdown_connections 2024-04-02T14:47:14.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:14.284+0000 7f4a9c000640 1 --2- 172.21.15.90:0/4031314895 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4a94108ff0 0x7f4a94110bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:14.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:14.284+0000 7f4a9c000640 1 --2- 172.21.15.90:0/4031314895 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4a941048d0 0x7f4a94108910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:14.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:14.284+0000 7f4a9c000640 1 --2- 172.21.15.90:0/4031314895 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4a94103f20 0x7f4a94104300 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:14.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:14.284+0000 7f4a9c000640 1 -- 172.21.15.90:0/4031314895 >> 172.21.15.90:0/4031314895 conn(0x7f4a940ff780 msgr2=0x7f4a94101ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:47:14.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:14.284+0000 7f4a9c000640 1 -- 172.21.15.90:0/4031314895 shutdown_connections 2024-04-02T14:47:14.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:14.284+0000 7f4a9c000640 1 -- 172.21.15.90:0/4031314895 wait complete. 2024-04-02T14:47:14.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:14.285+0000 7f4a9c000640 1 Processor -- start 2024-04-02T14:47:14.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:14.285+0000 7f4a9c000640 1 -- start start 2024-04-02T14:47:14.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:14.286+0000 7f4a9c000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4a94103f20 0x7f4a9419a630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:14.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:14.286+0000 7f4a9c000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4a94108ff0 0x7f4a9419ab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:14.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:14.286+0000 7f4a9c000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4a9419eef0 0x7f4a9419f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:14.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:14.286+0000 7f4a9c000640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f4a94114150 con 0x7f4a9419eef0 2024-04-02T14:47:14.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:14.286+0000 7f4a9c000640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f4a94113fd0 con 0x7f4a94103f20 2024-04-02T14:47:14.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:14.286+0000 7f4a9c000640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f4a941142d0 con 0x7f4a94108ff0 2024-04-02T14:47:14.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:14.286+0000 7f4a98c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4a94108ff0 0x7f4a9419ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:14.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:14.286+0000 7f4a98c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4a94108ff0 0x7f4a9419ab70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:59526/0 (socket says 172.21.15.90:59526) 2024-04-02T14:47:14.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:14.286+0000 7f4a9a000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4a9419eef0 0x7f4a9419f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:14.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:14.286+0000 7f4a98c00640 1 -- 172.21.15.90:0/2182759054 learned_addr learned my addr 172.21.15.90:0/2182759054 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:47:14.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:14.286+0000 7f4a99600640 1 --2- 172.21.15.90:0/2182759054 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4a94103f20 0x7f4a9419a630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:14.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:14.286+0000 7f4a98c00640 1 -- 172.21.15.90:0/2182759054 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4a94103f20 msgr2=0x7f4a9419a630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:14.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:14.286+0000 7f4a98c00640 1 --2- 172.21.15.90:0/2182759054 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4a94103f20 0x7f4a9419a630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:14.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:14.287+0000 7f4a98c00640 1 -- 172.21.15.90:0/2182759054 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4a9419eef0 msgr2=0x7f4a9419f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:14.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:14.287+0000 7f4a98c00640 1 --2- 172.21.15.90:0/2182759054 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4a9419eef0 0x7f4a9419f3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:14.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:14.287+0000 7f4a98c00640 1 -- 172.21.15.90:0/2182759054 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a9419f9e0 con 0x7f4a94108ff0 2024-04-02T14:47:14.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:14.287+0000 7f4a9a000640 1 --2- 172.21.15.90:0/2182759054 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4a9419eef0 0x7f4a9419f3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:47:14.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:14.288+0000 7f4a98c00640 1 --2- 172.21.15.90:0/2182759054 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4a94108ff0 0x7f4a9419ab70 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f4a8000b450 tx=0x7f4a8000b910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:47:14.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:14.289+0000 7f4a92000640 1 -- 172.21.15.90:0/2182759054 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a80015020 con 0x7f4a94108ff0 2024-04-02T14:47:14.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:14.289+0000 7f4a92000640 1 -- 172.21.15.90:0/2182759054 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4a80011d70 con 0x7f4a94108ff0 2024-04-02T14:47:14.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:14.289+0000 7f4a92000640 1 -- 172.21.15.90:0/2182759054 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a8000f340 con 0x7f4a94108ff0 2024-04-02T14:47:14.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:14.289+0000 7f4a9c000640 1 -- 172.21.15.90:0/2182759054 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a9419fcd0 con 0x7f4a94108ff0 2024-04-02T14:47:14.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:14.289+0000 7f4a9c000640 1 -- 172.21.15.90:0/2182759054 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a941a7470 con 0x7f4a94108ff0 2024-04-02T14:47:14.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:14.290+0000 7f4a9c000640 1 -- 172.21.15.90:0/2182759054 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4a94068d40 con 0x7f4a94108ff0 2024-04-02T14:47:14.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:14.294+0000 7f4a92000640 1 -- 172.21.15.90:0/2182759054 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f4a80018020 con 0x7f4a94108ff0 2024-04-02T14:47:14.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:14.295+0000 7f4a92000640 1 --2- 172.21.15.90:0/2182759054 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f4a7c0a3ea0 0x7f4a7c0a6360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:14.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:14.295+0000 7f4a92000640 1 -- 172.21.15.90:0/2182759054 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 8594+0+0 (secure 0 0 0) 0x7f4a800d0940 con 0x7f4a94108ff0 2024-04-02T14:47:14.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:14.295+0000 7f4a99600640 1 --2- 172.21.15.90:0/2182759054 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f4a7c0a3ea0 0x7f4a7c0a6360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:14.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:14.297+0000 7f4a99600640 1 --2- 172.21.15.90:0/2182759054 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f4a7c0a3ea0 0x7f4a7c0a6360 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f4a88000f80 tx=0x7f4a880324c0 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:47:14.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:14.297+0000 7f4a92000640 1 -- 172.21.15.90:0/2182759054 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7f4a8001f020 con 0x7f4a94108ff0 2024-04-02T14:47:14.690 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:14.696+0000 7f4a9c000640 1 -- 172.21.15.90:0/2182759054 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4a9419b980 con 0x7f4a94108ff0 2024-04-02T14:47:14.691 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:14.697+0000 7f4a92000640 1 -- 172.21.15.90:0/2182759054 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11927 (secure 0 0 0) 0x7f4a8000d040 con 0x7f4a94108ff0 2024-04-02T14:47:14.692 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-02T14:47:14.692 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14412,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6837/2852160309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6836","nonce":2852160309},{"type":"v1","addr":"172.21.15.90:6837","nonce":2852160309}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14478,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6841/1919376858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6840","nonce":1919376858},{"type":"v1","addr":"172.21.15.90:6841","nonce":1919376858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-02T14:32:22.319083+0000","modified":"2024-04-02T14:34:52.266152+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24590,"mds_1":24560,"mds_2":24547,"mds_3":24539,"mds_4":24523},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24454":{"gid":24454,"name":"l","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6833/1564319431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":1564319431},{"type":"v1","addr":"172.21.15.177:6833","nonce":1564319431}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.92:6833/1570337372","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":1570337372},{"type":"v1","addr":"172.21.15.92:6833","nonce":1570337372}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"j","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.90:6835/1277124891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6834","nonce":1277124891},{"type":"v1","addr":"172.21.15.90:6835","nonce":1277124891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.92:6835/860721164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":860721164},{"type":"v1","addr":"172.21.15.92:6835","nonce":860721164}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24511":{"gid":24511,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6835/3177244988","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":3177244988},{"type":"v1","addr":"172.21.15.177:6835","nonce":3177244988}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"c","rank":4,"incarnation":25,"state":"up:active","state_seq":9,"addr":"172.21.15.177:6837/4223414858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":4223414858},{"type":"v1","addr":"172.21.15.177:6837","nonce":4223414858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24539":{"gid":24539,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":8,"addr":"172.21.15.92:6837/1270635939","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":1270635939},{"type":"v1","addr":"172.21.15.92:6837","nonce":1270635939}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24547":{"gid":24547,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":7,"addr":"172.21.15.177:6839/664895663","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":664895663},{"type":"v1","addr":"172.21.15.177:6839","nonce":664895663}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"g","rank":1,"incarnation":19,"state":"up:active","state_seq":5,"addr":"172.21.15.90:6839/1498915870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6838","nonce":1498915870},{"type":"v1","addr":"172.21.15.90:6839","nonce":1498915870}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24590":{"gid":24590,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.92:6839/1054888219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":1054888219},{"type":"v1","addr":"172.21.15.92:6839","nonce":1054888219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24590,"qdb_cluster":[24547,24590,24523,24539,24560]},"id":1}]} 2024-04-02T14:47:14.692 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 27 2024-04-02T14:47:14.695 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:14.700+0000 7f4a9c000640 1 -- 172.21.15.90:0/2182759054 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f4a7c0a3ea0 msgr2=0x7f4a7c0a6360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:14.695 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:14.700+0000 7f4a9c000640 1 --2- 172.21.15.90:0/2182759054 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f4a7c0a3ea0 0x7f4a7c0a6360 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f4a88000f80 tx=0x7f4a880324c0 comp rx=0 tx=0).stop 2024-04-02T14:47:14.695 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:14.700+0000 7f4a9c000640 1 -- 172.21.15.90:0/2182759054 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4a94108ff0 msgr2=0x7f4a9419ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:14.695 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:14.700+0000 7f4a9c000640 1 --2- 172.21.15.90:0/2182759054 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4a94108ff0 0x7f4a9419ab70 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f4a8000b450 tx=0x7f4a8000b910 comp rx=0 tx=0).stop 2024-04-02T14:47:14.695 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:14.700+0000 7f4a9c000640 1 -- 172.21.15.90:0/2182759054 shutdown_connections 2024-04-02T14:47:14.695 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:14.700+0000 7f4a9c000640 1 --2- 172.21.15.90:0/2182759054 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f4a7c0a3ea0 0x7f4a7c0a6360 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:14.695 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:14.700+0000 7f4a9c000640 1 --2- 172.21.15.90:0/2182759054 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4a9419eef0 0x7f4a9419f3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:14.695 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:14.700+0000 7f4a9c000640 1 --2- 172.21.15.90:0/2182759054 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4a94108ff0 0x7f4a9419ab70 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:14.695 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:14.700+0000 7f4a9c000640 1 --2- 172.21.15.90:0/2182759054 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4a94103f20 0x7f4a9419a630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:14.695 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:14.700+0000 7f4a9c000640 1 -- 172.21.15.90:0/2182759054 >> 172.21.15.90:0/2182759054 conn(0x7f4a940ff780 msgr2=0x7f4a94101ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:47:14.695 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:14.700+0000 7f4a9c000640 1 -- 172.21.15.90:0/2182759054 shutdown_connections 2024-04-02T14:47:14.695 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:14.700+0000 7f4a9c000640 1 -- 172.21.15.90:0/2182759054 wait complete. 2024-04-02T14:47:14.728 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-04-02T14:47:14.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:14.997+0000 7fb026800640 1 -- 172.21.15.90:0/2594617221 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb0200747c0 msgr2=0x7fb020112560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:14.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:14.997+0000 7fb026800640 1 --2- 172.21.15.90:0/2594617221 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb0200747c0 0x7fb020112560 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7fb008009900 tx=0x7fb00802bcd0 comp rx=0 tx=0).stop 2024-04-02T14:47:14.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:14.998+0000 7fb026800640 1 -- 172.21.15.90:0/2594617221 shutdown_connections 2024-04-02T14:47:14.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:14.998+0000 7fb026800640 1 --2- 172.21.15.90:0/2594617221 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb020112aa0 0x7fb020114e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:14.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:14.998+0000 7fb026800640 1 --2- 172.21.15.90:0/2594617221 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb0200747c0 0x7fb020112560 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:14.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:14.998+0000 7fb026800640 1 --2- 172.21.15.90:0/2594617221 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb020073ea0 0x7fb020074280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:14.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:14.998+0000 7fb026800640 1 -- 172.21.15.90:0/2594617221 >> 172.21.15.90:0/2594617221 conn(0x7fb020078f00 msgr2=0x7fb020079310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:47:14.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:14.998+0000 7fb026800640 1 -- 172.21.15.90:0/2594617221 shutdown_connections 2024-04-02T14:47:14.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:14.998+0000 7fb026800640 1 -- 172.21.15.90:0/2594617221 wait complete. 2024-04-02T14:47:14.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:14.999+0000 7fb026800640 1 Processor -- start 2024-04-02T14:47:14.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:14.999+0000 7fb026800640 1 -- start start 2024-04-02T14:47:14.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.000+0000 7fb026800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb020073ea0 0x7fb0201a48b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:14.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.000+0000 7fb026800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb020112aa0 0x7fb0201a4df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:14.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.000+0000 7fb026800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb02019ea60 0x7fb02019ef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:14.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.000+0000 7fb026800640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fb0201184d0 con 0x7fb020073ea0 2024-04-02T14:47:14.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.000+0000 7fb026800640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fb020118350 con 0x7fb020112aa0 2024-04-02T14:47:14.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.000+0000 7fb026800640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fb020118650 con 0x7fb02019ea60 2024-04-02T14:47:14.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.001+0000 7fb01ea00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb020112aa0 0x7fb0201a4df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:14.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.001+0000 7fb01ea00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb020112aa0 0x7fb0201a4df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:53004/0 (socket says 172.21.15.90:53004) 2024-04-02T14:47:14.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.001+0000 7fb01ea00640 1 -- 172.21.15.90:0/3678327674 learned_addr learned my addr 172.21.15.90:0/3678327674 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:47:14.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.001+0000 7fb01f400640 1 --2- 172.21.15.90:0/3678327674 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb020073ea0 0x7fb0201a48b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:14.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.001+0000 7fb01fe00640 1 --2- 172.21.15.90:0/3678327674 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb02019ea60 0x7fb02019ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:14.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.002+0000 7fb01ea00640 1 -- 172.21.15.90:0/3678327674 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb02019ea60 msgr2=0x7fb02019ef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:14.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.002+0000 7fb01ea00640 1 --2- 172.21.15.90:0/3678327674 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb02019ea60 0x7fb02019ef10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:14.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.002+0000 7fb01ea00640 1 -- 172.21.15.90:0/3678327674 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb020073ea0 msgr2=0x7fb0201a48b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:14.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.002+0000 7fb01ea00640 1 --2- 172.21.15.90:0/3678327674 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb020073ea0 0x7fb0201a48b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:14.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.002+0000 7fb01ea00640 1 -- 172.21.15.90:0/3678327674 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb02019f580 con 0x7fb020112aa0 2024-04-02T14:47:14.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.002+0000 7fb01ea00640 1 --2- 172.21.15.90:0/3678327674 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb020112aa0 0x7fb0201a4df0 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7fb008005ab0 tx=0x7fb00802c880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:47:14.997 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.003+0000 7fb017e00640 1 -- 172.21.15.90:0/3678327674 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb008041020 con 0x7fb020112aa0 2024-04-02T14:47:14.997 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.003+0000 7fb017e00640 1 -- 172.21.15.90:0/3678327674 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb0080054e0 con 0x7fb020112aa0 2024-04-02T14:47:14.997 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.003+0000 7fb017e00640 1 -- 172.21.15.90:0/3678327674 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb008003a80 con 0x7fb020112aa0 2024-04-02T14:47:14.997 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.003+0000 7fb026800640 1 -- 172.21.15.90:0/3678327674 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb02019f810 con 0x7fb020112aa0 2024-04-02T14:47:14.997 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.003+0000 7fb026800640 1 -- 172.21.15.90:0/3678327674 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0201aba10 con 0x7fb020112aa0 2024-04-02T14:47:15.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.006+0000 7fb029577740 1 Processor -- start 2024-04-02T14:47:15.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.007+0000 7fb029577740 1 -- start start 2024-04-02T14:47:15.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.007+0000 7fb029577740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e5934729f0 0x55e593471740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:15.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.007+0000 7fb029577740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e593471d10 0x55e593472190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:15.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.007+0000 7fb029577740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e59347b080 0x55e59347d470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:15.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.007+0000 7fb029577740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55e5933b2400 con 0x55e5934729f0 2024-04-02T14:47:15.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.007+0000 7fb029577740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55e593396f10 con 0x55e59347b080 2024-04-02T14:47:15.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.007+0000 7fb029577740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55e5933d66f0 con 0x55e593471d10 2024-04-02T14:47:15.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.007+0000 7fb017e00640 1 -- 172.21.15.90:0/3678327674 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fb008044020 con 0x7fb020112aa0 2024-04-02T14:47:15.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.008+0000 7fb01f400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e593471d10 0x55e593472190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:15.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.008+0000 7fb01f400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e593471d10 0x55e593472190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:59548/0 (socket says 172.21.15.90:59548) 2024-04-02T14:47:15.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.008+0000 7fb01f400640 1 -- 172.21.15.90:0/4081122182 learned_addr learned my addr 172.21.15.90:0/4081122182 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:47:15.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.008+0000 7fb01fe00640 1 --2- 172.21.15.90:0/4081122182 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e5934729f0 0x55e593471740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:15.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.008+0000 7fb01f400640 1 -- 172.21.15.90:0/4081122182 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e59347b080 msgr2=0x55e59347d470 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-02T14:47:15.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.008+0000 7fb01f400640 1 --2- 172.21.15.90:0/4081122182 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e59347b080 0x55e59347d470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:15.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.008+0000 7fb01f400640 1 -- 172.21.15.90:0/4081122182 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e5934729f0 msgr2=0x55e593471740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:47:15.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.008+0000 7fb01f400640 1 --2- 172.21.15.90:0/4081122182 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e5934729f0 0x55e593471740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:15.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.008+0000 7fb01f400640 1 -- 172.21.15.90:0/4081122182 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e5930ea3f0 con 0x55e593471d10 2024-04-02T14:47:15.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.008+0000 7fb01fe00640 1 --2- 172.21.15.90:0/4081122182 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e5934729f0 0x55e593471740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:47:15.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.009+0000 7fb01f400640 1 --2- 172.21.15.90:0/4081122182 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e593471d10 0x55e593472190 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7fb010009f50 tx=0x7fb010010800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2aace2c2ce0611b server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:47:15.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.010+0000 7fb014c00640 1 -- 172.21.15.90:0/4081122182 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb010016070 con 0x55e593471d10 2024-04-02T14:47:15.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.010+0000 7fb017e00640 1 --2- 172.21.15.90:0/3678327674 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fb0040a3e70 0x7fb0040a6330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:15.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.010+0000 7fb014c00640 1 -- 172.21.15.90:0/4081122182 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb010012930 con 0x55e593471d10 2024-04-02T14:47:15.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.010+0000 7fb014c00640 1 -- 172.21.15.90:0/4081122182 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb010011a50 con 0x55e593471d10 2024-04-02T14:47:15.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.010+0000 7fb017e00640 1 -- 172.21.15.90:0/3678327674 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 8594+0+0 (secure 0 0 0) 0x7fb0080f1ef0 con 0x7fb020112aa0 2024-04-02T14:47:15.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.010+0000 7fb01fe00640 1 --2- 172.21.15.90:0/3678327674 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fb0040a3e70 0x7fb0040a6330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:15.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.011+0000 7fb029577740 1 -- 172.21.15.90:0/4081122182 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e593471d10 msgr2=0x55e593472190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:15.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.011+0000 7fb029577740 1 --2- 172.21.15.90:0/4081122182 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e593471d10 0x55e593472190 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7fb010009f50 tx=0x7fb010010800 comp rx=0 tx=0).stop 2024-04-02T14:47:15.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.011+0000 7fb029577740 1 -- 172.21.15.90:0/4081122182 shutdown_connections 2024-04-02T14:47:15.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.011+0000 7fb029577740 1 --2- 172.21.15.90:0/4081122182 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e59347b080 0x55e59347d470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:15.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.011+0000 7fb029577740 1 --2- 172.21.15.90:0/4081122182 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e593471d10 0x55e593472190 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:15.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.011+0000 7fb029577740 1 --2- 172.21.15.90:0/4081122182 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e5934729f0 0x55e593471740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:15.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.011+0000 7fb01fe00640 1 --2- 172.21.15.90:0/3678327674 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fb0040a3e70 0x7fb0040a6330 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fb02006b3a0 tx=0x7fb018005f90 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:47:15.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.011+0000 7fb029577740 1 -- 172.21.15.90:0/4081122182 >> 172.21.15.90:0/4081122182 conn(0x55e593438a50 msgr2=0x55e59346c760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:47:15.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.011+0000 7fb029577740 1 -- 172.21.15.90:0/4081122182 shutdown_connections 2024-04-02T14:47:15.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.011+0000 7fb029577740 1 -- 172.21.15.90:0/4081122182 wait complete. 2024-04-02T14:47:15.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.012+0000 7fb029577740 1 -- start start 2024-04-02T14:47:15.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.012+0000 7fb029577740 1 Processor -- start 2024-04-02T14:47:15.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.012+0000 7fb029577740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e593471d10 0x55e5934780e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:15.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.012+0000 7fb029577740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e5934729f0 0x55e593478620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:15.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.012+0000 7fb029577740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e59347b080 0x55e593478b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:15.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.012+0000 7fb029577740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55e5933d26a0 con 0x55e59347b080 2024-04-02T14:47:15.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.012+0000 7fb029577740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55e59337e980 con 0x55e5934729f0 2024-04-02T14:47:15.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.012+0000 7fb029577740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55e593391cb0 con 0x55e593471d10 2024-04-02T14:47:15.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.012+0000 7fb029577740 10 client.0 _ll_register_callbacks cb 0x55e5933916a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:47:15.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.013+0000 7fb01ea00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e5934729f0 0x55e593478620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:15.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.013+0000 7fb01ea00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e5934729f0 0x55e593478620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:53010/0 (socket says 172.21.15.90:53010) 2024-04-02T14:47:15.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.013+0000 7fb01ea00640 1 -- 172.21.15.90:0/3967180493 learned_addr learned my addr 172.21.15.90:0/3967180493 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:47:15.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.013+0000 7fb01fe00640 1 --2- 172.21.15.90:0/3967180493 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e59347b080 0x55e593478b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:15.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.013+0000 7fb01f400640 1 --2- 172.21.15.90:0/3967180493 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e593471d10 0x55e5934780e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:15.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.013+0000 7fb01ea00640 1 -- 172.21.15.90:0/3967180493 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e593471d10 msgr2=0x55e5934780e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:47:15.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.013+0000 7fb01ea00640 1 --2- 172.21.15.90:0/3967180493 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e593471d10 0x55e5934780e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:15.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.013+0000 7fb01ea00640 1 -- 172.21.15.90:0/3967180493 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e59347b080 msgr2=0x55e593478b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:47:15.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.013+0000 7fb01ea00640 1 --2- 172.21.15.90:0/3967180493 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e59347b080 0x55e593478b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:15.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.013+0000 7fb01ea00640 1 -- 172.21.15.90:0/3967180493 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e5931b47d0 con 0x55e5934729f0 2024-04-02T14:47:15.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.013+0000 7fb01ea00640 1 --2- 172.21.15.90:0/3967180493 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e5934729f0 0x55e593478620 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7fb0080ba8c0 tx=0x7fb0080bdaf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=127414c6bc306cd server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:47:15.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.013+0000 7fb00d600640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:47:15.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.014+0000 7fb00d600640 1 -- 172.21.15.90:0/3967180493 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb008032340 con 0x55e5934729f0 2024-04-02T14:47:15.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.014+0000 7fb00d600640 1 -- 172.21.15.90:0/3967180493 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 8594+0+0 (secure 0 0 0) 0x7fb0080d7310 con 0x55e5934729f0 2024-04-02T14:47:15.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.014+0000 7fb029577740 1 -- 172.21.15.90:0/3967180493 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb010012930 con 0x55e5934729f0 2024-04-02T14:47:15.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.015+0000 7fb00d600640 1 -- 172.21.15.90:0/3967180493 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fb0080be5f0 con 0x55e5934729f0 2024-04-02T14:47:15.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.015+0000 7fb029577740 10 client.29279 fetch_fsmap learned FSMap version 27 2024-04-02T14:47:15.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.015+0000 7fb029577740 1 -- 172.21.15.90:0/3967180493 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55e593489fc0 con 0x55e5934729f0 2024-04-02T14:47:15.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.015+0000 7fb00d600640 1 -- 172.21.15.90:0/3967180493 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb010012930 con 0x55e5934729f0 2024-04-02T14:47:15.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.016+0000 7fb00d600640 1 -- 172.21.15.90:0/3967180493 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fb0080efce0 con 0x55e5934729f0 2024-04-02T14:47:15.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.016+0000 7fb029577740 10 client.29279 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:47:15.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.016+0000 7fb029577740 10 client.29279 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.c 2024-04-02T14:47:15.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.016+0000 7fb029577740 20 client.29279 populate_metadata read hostname 'smithi090' 2024-04-02T14:47:15.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.016+0000 7fb029577740 1 --2- 172.21.15.90:0/3967180493 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x55e59348b400 0x55e59348d830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:15.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.016+0000 7fb029577740 4 client.29279 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:47:15.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.016+0000 7fb029577740 1 -- 172.21.15.90:0/3967180493 --> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e593396040 con 0x55e59348b400 2024-04-02T14:47:15.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.017+0000 7fb01f400640 1 --2- 172.21.15.90:0/3967180493 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x55e59348b400 0x55e59348d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:15.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.017+0000 7fb01f400640 1 --2- 172.21.15.90:0/3967180493 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x55e59348b400 0x55e59348d830 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=495628f4c9de45ff server_cookie=d3276fad378f79bc in_seq=0 out_seq=0 2024-04-02T14:47:15.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.017+0000 7fb00d600640 10 client.29279 ms_handle_connect on v2:172.21.15.177:6836/4223414858 2024-04-02T14:47:15.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.018+0000 7fb00d600640 1 -- 172.21.15.90:0/3967180493 <== mds.4 v2:172.21.15.177:6836/4223414858 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e593396040 con 0x55e59348b400 2024-04-02T14:47:15.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.018+0000 7fb00d600640 10 client.29279 handle_command_reply: tid=0 2024-04-02T14:47:15.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.018+0000 7fb029577740 1 client.29279 shutdown 2024-04-02T14:47:15.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.019+0000 7fb029577740 1 -- 172.21.15.90:0/3967180493 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e5934729f0 msgr2=0x55e593478620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:15.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.019+0000 7fb029577740 1 --2- 172.21.15.90:0/3967180493 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e5934729f0 0x55e593478620 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7fb0080ba8c0 tx=0x7fb0080bdaf0 comp rx=0 tx=0).stop 2024-04-02T14:47:15.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.019+0000 7fb029577740 1 -- 172.21.15.90:0/3967180493 shutdown_connections 2024-04-02T14:47:15.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.019+0000 7fb029577740 1 --2- 172.21.15.90:0/3967180493 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x55e59348b400 0x55e59348d830 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:15.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.019+0000 7fb029577740 1 --2- 172.21.15.90:0/3967180493 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e59347b080 0x55e593478b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:15.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.019+0000 7fb029577740 1 --2- 172.21.15.90:0/3967180493 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e5934729f0 0x55e593478620 unknown :-1 s=CLOSED pgs=865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:15.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.019+0000 7fb029577740 1 --2- 172.21.15.90:0/3967180493 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e593471d10 0x55e5934780e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:15.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.019+0000 7fb00d600640 0 client.29279 ms_handle_reset on v2:172.21.15.177:6836/4223414858 2024-04-02T14:47:15.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.019+0000 7fb029577740 1 -- 172.21.15.90:0/3967180493 >> 172.21.15.90:0/3967180493 conn(0x55e593438a50 msgr2=0x55e59346e4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:47:15.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.019+0000 7fb029577740 1 -- 172.21.15.90:0/3967180493 shutdown_connections 2024-04-02T14:47:15.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.019+0000 7fb029577740 1 -- 172.21.15.90:0/3967180493 wait complete. 2024-04-02T14:47:15.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.019+0000 7fb029577740 20 client.29279 trim_cache size 0 max 16384 2024-04-02T14:47:15.032 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.038+0000 7fb029577740 1 Processor -- start 2024-04-02T14:47:15.033 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.038+0000 7fb029577740 1 -- start start 2024-04-02T14:47:15.033 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.038+0000 7fb029577740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e593471d10 0x55e59347a810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:15.033 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.038+0000 7fb029577740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e5934729f0 0x55e593486b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:15.033 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.038+0000 7fb029577740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e59347b080 0x55e593477240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:15.033 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.038+0000 7fb029577740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fb0201b45c0 con 0x55e59347b080 2024-04-02T14:47:15.033 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.038+0000 7fb029577740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fb0201b6100 con 0x55e5934729f0 2024-04-02T14:47:15.033 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.038+0000 7fb029577740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fb0201b4c90 con 0x55e593471d10 2024-04-02T14:47:15.033 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.038+0000 7fb01f400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e593471d10 0x55e59347a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:15.033 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.038+0000 7fb01f400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e593471d10 0x55e59347a810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:59562/0 (socket says 172.21.15.90:59562) 2024-04-02T14:47:15.033 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.038+0000 7fb01f400640 1 -- 172.21.15.90:0/381931436 learned_addr learned my addr 172.21.15.90:0/381931436 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:47:15.033 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.039+0000 7fb01ea00640 1 --2- 172.21.15.90:0/381931436 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e5934729f0 0x55e593486b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:15.033 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.039+0000 7fb01fe00640 1 --2- 172.21.15.90:0/381931436 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e59347b080 0x55e593477240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:15.034 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.039+0000 7fb01f400640 1 -- 172.21.15.90:0/381931436 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e5934729f0 msgr2=0x55e593486b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:47:15.034 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.039+0000 7fb01f400640 1 --2- 172.21.15.90:0/381931436 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e5934729f0 0x55e593486b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:15.034 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.039+0000 7fb01f400640 1 -- 172.21.15.90:0/381931436 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e59347b080 msgr2=0x55e593477240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:47:15.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.039+0000 7fb01f400640 1 --2- 172.21.15.90:0/381931436 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e59347b080 0x55e593477240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:15.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.039+0000 7fb01f400640 1 -- 172.21.15.90:0/381931436 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e593476630 con 0x55e593471d10 2024-04-02T14:47:15.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.039+0000 7fb01fe00640 1 --2- 172.21.15.90:0/381931436 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e59347b080 0x55e593477240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:47:15.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.039+0000 7fb01f400640 1 --2- 172.21.15.90:0/381931436 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e593471d10 0x55e59347a810 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7fb010009f50 tx=0x7fb010009630 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3036ec768ae1b282 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:47:15.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.040+0000 7fb00fe00640 1 -- 172.21.15.90:0/381931436 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb010016040 con 0x55e593471d10 2024-04-02T14:47:15.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.040+0000 7fb00fe00640 1 -- 172.21.15.90:0/381931436 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb01002a6a0 con 0x55e593471d10 2024-04-02T14:47:15.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.040+0000 7fb00fe00640 1 -- 172.21.15.90:0/381931436 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb010011870 con 0x55e593471d10 2024-04-02T14:47:15.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.042+0000 7fb029577740 1 -- 172.21.15.90:0/381931436 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e593471d10 msgr2=0x55e59347a810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:15.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.042+0000 7fb029577740 1 --2- 172.21.15.90:0/381931436 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e593471d10 0x55e59347a810 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7fb010009f50 tx=0x7fb010009630 comp rx=0 tx=0).stop 2024-04-02T14:47:15.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.042+0000 7fb029577740 1 -- 172.21.15.90:0/381931436 shutdown_connections 2024-04-02T14:47:15.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.042+0000 7fb029577740 1 --2- 172.21.15.90:0/381931436 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e59347b080 0x55e593477240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:15.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.042+0000 7fb029577740 1 --2- 172.21.15.90:0/381931436 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e5934729f0 0x55e593486b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:15.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.042+0000 7fb029577740 1 --2- 172.21.15.90:0/381931436 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e593471d10 0x55e59347a810 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:15.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.042+0000 7fb029577740 1 -- 172.21.15.90:0/381931436 >> 172.21.15.90:0/381931436 conn(0x55e593438a50 msgr2=0x55e59346dbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:47:15.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.042+0000 7fb029577740 1 -- 172.21.15.90:0/381931436 shutdown_connections 2024-04-02T14:47:15.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.042+0000 7fb029577740 1 -- 172.21.15.90:0/381931436 wait complete. 2024-04-02T14:47:15.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.043+0000 7fb029577740 1 -- start start 2024-04-02T14:47:15.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.043+0000 7fb029577740 1 Processor -- start 2024-04-02T14:47:15.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.043+0000 7fb029577740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e593471d10 0x55e5934835c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:15.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.043+0000 7fb029577740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e5934729f0 0x55e59347bbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:15.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.043+0000 7fb029577740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e59347b080 0x55e59347c310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:15.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.043+0000 7fb029577740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fb0201b5a30 con 0x55e5934729f0 2024-04-02T14:47:15.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.043+0000 7fb029577740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fb0201b5360 con 0x55e593471d10 2024-04-02T14:47:15.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.043+0000 7fb029577740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fb0201b25c0 con 0x55e59347b080 2024-04-02T14:47:15.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.043+0000 7fb029577740 10 client.0 _ll_register_callbacks cb 0x7fb0080b9b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:47:15.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.044+0000 7fb01f400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e593471d10 0x55e5934835c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:15.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.044+0000 7fb01ea00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e5934729f0 0x55e59347bbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:15.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.044+0000 7fb01f400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e593471d10 0x55e5934835c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:53030/0 (socket says 172.21.15.90:53030) 2024-04-02T14:47:15.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.044+0000 7fb01f400640 1 -- 172.21.15.90:0/4294079090 learned_addr learned my addr 172.21.15.90:0/4294079090 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:47:15.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.044+0000 7fb01fe00640 1 --2- 172.21.15.90:0/4294079090 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e59347b080 0x55e59347c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:15.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.044+0000 7fb01ea00640 1 -- 172.21.15.90:0/4294079090 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e59347b080 msgr2=0x55e59347c310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:47:15.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.044+0000 7fb01ea00640 1 --2- 172.21.15.90:0/4294079090 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e59347b080 0x55e59347c310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:15.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.044+0000 7fb01ea00640 1 -- 172.21.15.90:0/4294079090 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e593471d10 msgr2=0x55e5934835c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:47:15.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.044+0000 7fb01ea00640 1 --2- 172.21.15.90:0/4294079090 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e593471d10 0x55e5934835c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:15.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.044+0000 7fb01ea00640 1 -- 172.21.15.90:0/4294079090 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e593481100 con 0x55e5934729f0 2024-04-02T14:47:15.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.044+0000 7fb01f400640 1 --2- 172.21.15.90:0/4294079090 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e593471d10 0x55e5934835c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:47:15.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.044+0000 7fb01ea00640 1 --2- 172.21.15.90:0/4294079090 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e5934729f0 0x55e59347bbd0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fb0080ba350 tx=0x7fb008000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf986c84e19a3f2a server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:47:15.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.044+0000 7fb00d600640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-02T14:47:15.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.045+0000 7fb00d600640 1 -- 172.21.15.90:0/4294079090 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0080d7470 con 0x55e5934729f0 2024-04-02T14:47:15.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.045+0000 7fb00d600640 1 -- 172.21.15.90:0/4294079090 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 8594+0+0 (secure 0 0 0) 0x7fb0080bd540 con 0x55e5934729f0 2024-04-02T14:47:15.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.045+0000 7fb029577740 1 -- 172.21.15.90:0/4294079090 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb01002a6a0 con 0x55e5934729f0 2024-04-02T14:47:15.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.045+0000 7fb00d600640 1 -- 172.21.15.90:0/4294079090 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fb008032340 con 0x55e5934729f0 2024-04-02T14:47:15.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.045+0000 7fb029577740 10 client.18198 fetch_fsmap learned FSMap version 27 2024-04-02T14:47:15.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.045+0000 7fb029577740 1 -- 172.21.15.90:0/4294079090 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55e593481100 con 0x55e5934729f0 2024-04-02T14:47:15.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.045+0000 7fb00d600640 1 -- 172.21.15.90:0/4294079090 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb01002a6a0 con 0x55e5934729f0 2024-04-02T14:47:15.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.046+0000 7fb00d600640 1 -- 172.21.15.90:0/4294079090 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fb0080efce0 con 0x55e5934729f0 2024-04-02T14:47:15.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.046+0000 7fb029577740 10 client.18198 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:47:15.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.046+0000 7fb029577740 10 client.18198 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.c 2024-04-02T14:47:15.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.046+0000 7fb029577740 20 client.18198 populate_metadata read hostname 'smithi090' 2024-04-02T14:47:15.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.046+0000 7fb029577740 1 --2- 172.21.15.90:0/4294079090 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x55e5934f1ef0 0x55e5934f4310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:15.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.046+0000 7fb029577740 4 client.18198 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-02T14:47:15.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.046+0000 7fb029577740 1 -- 172.21.15.90:0/4294079090 --> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55e5933e3c30 con 0x55e5934f1ef0 2024-04-02T14:47:15.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.047+0000 7fb01f400640 1 --2- 172.21.15.90:0/4294079090 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x55e5934f1ef0 0x55e5934f4310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:15.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.047+0000 7fb00d600640 10 client.18198 ms_handle_connect on v2:172.21.15.177:6836/4223414858 2024-04-02T14:47:15.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.047+0000 7fb01f400640 1 --2- 172.21.15.90:0/4294079090 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x55e5934f1ef0 0x55e5934f4310 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=516f7ef59833ad98 server_cookie=a39faf950304b519 in_seq=0 out_seq=0 2024-04-02T14:47:15.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.048+0000 7fb00d600640 1 -- 172.21.15.90:0/4294079090 <== mds.4 v2:172.21.15.177:6836/4223414858 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55e5933e3c30 con 0x55e5934f1ef0 2024-04-02T14:47:15.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.048+0000 7fb00d600640 10 client.18198 handle_command_reply: tid=0 2024-04-02T14:47:15.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.048+0000 7fb029577740 1 client.18198 shutdown 2024-04-02T14:47:15.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.048+0000 7fb029577740 1 -- 172.21.15.90:0/4294079090 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e5934729f0 msgr2=0x55e59347bbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:15.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.048+0000 7fb029577740 1 --2- 172.21.15.90:0/4294079090 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e5934729f0 0x55e59347bbd0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fb0080ba350 tx=0x7fb008000bc0 comp rx=0 tx=0).stop 2024-04-02T14:47:15.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.048+0000 7fb029577740 1 -- 172.21.15.90:0/4294079090 shutdown_connections 2024-04-02T14:47:15.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.048+0000 7fb029577740 1 --2- 172.21.15.90:0/4294079090 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e59347b080 0x55e59347c310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:15.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.049+0000 7fb029577740 1 --2- 172.21.15.90:0/4294079090 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x55e5934f1ef0 0x55e5934f4310 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:15.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.049+0000 7fb029577740 1 --2- 172.21.15.90:0/4294079090 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e5934729f0 0x55e59347bbd0 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:15.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.049+0000 7fb029577740 1 --2- 172.21.15.90:0/4294079090 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e593471d10 0x55e5934835c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:15.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.049+0000 7fb029577740 1 -- 172.21.15.90:0/4294079090 >> 172.21.15.90:0/4294079090 conn(0x55e593438a50 msgr2=0x55e593478910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:47:15.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.049+0000 7fb00d600640 0 client.18198 ms_handle_reset on v2:172.21.15.177:6836/4223414858 2024-04-02T14:47:15.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.049+0000 7fb029577740 1 -- 172.21.15.90:0/4294079090 shutdown_connections 2024-04-02T14:47:15.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.049+0000 7fb029577740 1 -- 172.21.15.90:0/4294079090 wait complete. 2024-04-02T14:47:15.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.049+0000 7fb029577740 20 client.18198 trim_cache size 0 max 16384 2024-04-02T14:47:15.045 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-02T14:47:15.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.050+0000 7fb026800640 1 -- 172.21.15.90:0/3678327674 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fb0040a3e70 msgr2=0x7fb0040a6330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:15.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.050+0000 7fb026800640 1 --2- 172.21.15.90:0/3678327674 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fb0040a3e70 0x7fb0040a6330 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fb02006b3a0 tx=0x7fb018005f90 comp rx=0 tx=0).stop 2024-04-02T14:47:15.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.050+0000 7fb026800640 1 -- 172.21.15.90:0/3678327674 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb020112aa0 msgr2=0x7fb0201a4df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:15.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.050+0000 7fb026800640 1 --2- 172.21.15.90:0/3678327674 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb020112aa0 0x7fb0201a4df0 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7fb008005ab0 tx=0x7fb00802c880 comp rx=0 tx=0).stop 2024-04-02T14:47:15.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.050+0000 7fb026800640 1 -- 172.21.15.90:0/3678327674 shutdown_connections 2024-04-02T14:47:15.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.050+0000 7fb026800640 1 --2- 172.21.15.90:0/3678327674 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fb0040a3e70 0x7fb0040a6330 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:15.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.050+0000 7fb026800640 1 --2- 172.21.15.90:0/3678327674 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb02019ea60 0x7fb02019ef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:15.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.050+0000 7fb026800640 1 --2- 172.21.15.90:0/3678327674 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb020112aa0 0x7fb0201a4df0 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:15.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.050+0000 7fb026800640 1 --2- 172.21.15.90:0/3678327674 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb020073ea0 0x7fb0201a48b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:15.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.050+0000 7fb026800640 1 -- 172.21.15.90:0/3678327674 >> 172.21.15.90:0/3678327674 conn(0x7fb020078f00 msgr2=0x7fb0200fefa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:47:15.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.050+0000 7fb026800640 1 -- 172.21.15.90:0/3678327674 shutdown_connections 2024-04-02T14:47:15.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.050+0000 7fb026800640 1 -- 172.21.15.90:0/3678327674 wait complete. 2024-04-02T14:47:15.059 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:47:14 smithi090 ceph-mon[31674]: from='client.? 172.21.15.90:0/2182759054' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-02T14:47:15.063 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-04-02T14:47:15.103 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:47:14 smithi092 ceph-mon[36497]: from='client.? 172.21.15.90:0/2182759054' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-02T14:47:15.180 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:47:14 smithi177 ceph-mon[36431]: from='client.? 172.21.15.90:0/2182759054' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-02T14:47:15.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.333+0000 7fba08200640 1 -- 172.21.15.90:0/3494642218 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fba00103f10 msgr2=0x7fba001042f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:15.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.333+0000 7fba08200640 1 --2- 172.21.15.90:0/3494642218 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fba00103f10 0x7fba001042f0 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7fb9f4009900 tx=0x7fb9f402bc50 comp rx=0 tx=0).stop 2024-04-02T14:47:15.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.333+0000 7fba08200640 1 -- 172.21.15.90:0/3494642218 shutdown_connections 2024-04-02T14:47:15.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.333+0000 7fba08200640 1 --2- 172.21.15.90:0/3494642218 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fba00108fe0 0x7fba00110bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:15.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.333+0000 7fba08200640 1 --2- 172.21.15.90:0/3494642218 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fba001048c0 0x7fba00108900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:15.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.333+0000 7fba08200640 1 --2- 172.21.15.90:0/3494642218 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fba00103f10 0x7fba001042f0 unknown :-1 s=CLOSED pgs=866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:15.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.333+0000 7fba08200640 1 -- 172.21.15.90:0/3494642218 >> 172.21.15.90:0/3494642218 conn(0x7fba000ff7b0 msgr2=0x7fba00101bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:47:15.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.333+0000 7fba08200640 1 -- 172.21.15.90:0/3494642218 shutdown_connections 2024-04-02T14:47:15.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.333+0000 7fba08200640 1 -- 172.21.15.90:0/3494642218 wait complete. 2024-04-02T14:47:15.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.334+0000 7fba08200640 1 Processor -- start 2024-04-02T14:47:15.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.334+0000 7fba08200640 1 -- start start 2024-04-02T14:47:15.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.334+0000 7fba08200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fba00103f10 0x7fba0019a610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:15.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.334+0000 7fba08200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fba00108fe0 0x7fba0019ab50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:15.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.334+0000 7fba08200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fba0019eed0 0x7fba0019f380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:15.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.334+0000 7fba08200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fba001140d0 con 0x7fba0019eed0 2024-04-02T14:47:15.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.334+0000 7fba08200640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fba00113f50 con 0x7fba00103f10 2024-04-02T14:47:15.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.334+0000 7fba08200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fba00114250 con 0x7fba00108fe0 2024-04-02T14:47:15.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.335+0000 7fba05800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fba00103f10 0x7fba0019a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:15.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.335+0000 7fba06200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fba0019eed0 0x7fba0019f380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:15.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.335+0000 7fba06200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fba0019eed0 0x7fba0019f380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:36996/0 (socket says 172.21.15.90:36996) 2024-04-02T14:47:15.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.335+0000 7fba06200640 1 -- 172.21.15.90:0/141378004 learned_addr learned my addr 172.21.15.90:0/141378004 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:47:15.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.335+0000 7fba04e00640 1 --2- 172.21.15.90:0/141378004 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fba00108fe0 0x7fba0019ab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:15.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.336+0000 7fba04e00640 1 -- 172.21.15.90:0/141378004 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fba00103f10 msgr2=0x7fba0019a610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:15.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.336+0000 7fba04e00640 1 --2- 172.21.15.90:0/141378004 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fba00103f10 0x7fba0019a610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:15.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.336+0000 7fba04e00640 1 -- 172.21.15.90:0/141378004 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fba0019eed0 msgr2=0x7fba0019f380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:15.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.336+0000 7fba04e00640 1 --2- 172.21.15.90:0/141378004 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fba0019eed0 0x7fba0019f380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:15.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.336+0000 7fba04e00640 1 -- 172.21.15.90:0/141378004 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba0019f9c0 con 0x7fba00108fe0 2024-04-02T14:47:15.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.337+0000 7fba04e00640 1 --2- 172.21.15.90:0/141378004 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fba00108fe0 0x7fba0019ab50 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7fb9ec00e9c0 tx=0x7fb9ec00ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:47:15.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.343+0000 7fb9fe000640 1 -- 172.21.15.90:0/141378004 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9ec00ccc0 con 0x7fba00108fe0 2024-04-02T14:47:15.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.343+0000 7fb9fe000640 1 -- 172.21.15.90:0/141378004 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb9ec010910 con 0x7fba00108fe0 2024-04-02T14:47:15.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.343+0000 7fb9fe000640 1 -- 172.21.15.90:0/141378004 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9ec0188c0 con 0x7fba00108fe0 2024-04-02T14:47:15.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.343+0000 7fba08200640 1 -- 172.21.15.90:0/141378004 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba0019fcb0 con 0x7fba00108fe0 2024-04-02T14:47:15.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.343+0000 7fba08200640 1 -- 172.21.15.90:0/141378004 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba00075f40 con 0x7fba00108fe0 2024-04-02T14:47:15.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.348+0000 7fba0ae76740 1 Processor -- start 2024-04-02T14:47:15.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.349+0000 7fba0ae76740 1 -- start start 2024-04-02T14:47:15.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.349+0000 7fba0ae76740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5598a2d129f0 0x5598a2d11740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:15.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.352+0000 7fba0ae76740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5598a2d11d10 0x5598a2d12190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:15.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.353+0000 7fba0ae76740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5598a2d1b080 0x5598a2d1d470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:15.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.353+0000 7fba06200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5598a2d129f0 0x5598a2d11740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:15.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.353+0000 7fba06200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5598a2d129f0 0x5598a2d11740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:59586/0 (socket says 172.21.15.90:59586) 2024-04-02T14:47:15.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.353+0000 7fba06200640 1 -- 172.21.15.90:0/3770294932 learned_addr learned my addr 172.21.15.90:0/3770294932 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:47:15.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.353+0000 7fba0ae76740 1 -- 172.21.15.90:0/3770294932 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5598a2c52400 con 0x5598a2d1b080 2024-04-02T14:47:15.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.353+0000 7fba0ae76740 1 -- 172.21.15.90:0/3770294932 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x5598a2c36f10 con 0x5598a2d11d10 2024-04-02T14:47:15.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.353+0000 7fba0ae76740 1 -- 172.21.15.90:0/3770294932 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5598a2c766f0 con 0x5598a2d129f0 2024-04-02T14:47:15.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.353+0000 7fb9fe000640 1 -- 172.21.15.90:0/141378004 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fb9ec018a60 con 0x7fba00108fe0 2024-04-02T14:47:15.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.353+0000 7fba05800640 1 --2- 172.21.15.90:0/3770294932 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5598a2d11d10 0x5598a2d12190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:15.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.353+0000 7fba06200640 1 -- 172.21.15.90:0/3770294932 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5598a2d11d10 msgr2=0x5598a2d12190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:47:15.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.353+0000 7fba06200640 1 --2- 172.21.15.90:0/3770294932 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5598a2d11d10 0x5598a2d12190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:15.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.353+0000 7fba06200640 1 -- 172.21.15.90:0/3770294932 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5598a2d1b080 msgr2=0x5598a2d1d470 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-02T14:47:15.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.355+0000 7fb9fe000640 1 --2- 172.21.15.90:0/141378004 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fb9e80a3da0 0x7fb9e80a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:15.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.356+0000 7fb9fe000640 1 -- 172.21.15.90:0/141378004 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 8550+0+0 (secure 0 0 0) 0x7fb9ec0d0c10 con 0x7fba00108fe0 2024-04-02T14:47:15.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.356+0000 7fba04e00640 1 --2- 172.21.15.90:0/3770294932 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5598a2d1b080 0x5598a2d1d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:15.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.356+0000 7fba06200640 1 --2- 172.21.15.90:0/3770294932 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5598a2d1b080 0x5598a2d1d470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:15.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.356+0000 7fba06200640 1 -- 172.21.15.90:0/3770294932 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5598a298a3f0 con 0x5598a2d129f0 2024-04-02T14:47:15.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.356+0000 7fba06200640 1 --2- 172.21.15.90:0/3770294932 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5598a2d129f0 0x5598a2d11740 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7fb9f80094d0 tx=0x7fb9f8011870 comp rx=0 tx=0).ready entity=mon.1 client_cookie=100d0fce52dd8f4f server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:47:15.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.357+0000 7fba05800640 1 --2- 172.21.15.90:0/141378004 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fb9e80a3da0 0x7fb9e80a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:15.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.357+0000 7fb9f2a00640 1 -- 172.21.15.90:0/3770294932 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9f8012710 con 0x5598a2d129f0 2024-04-02T14:47:15.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.359+0000 7fb9f2a00640 1 -- 172.21.15.90:0/3770294932 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb9f8012d90 con 0x5598a2d129f0 2024-04-02T14:47:15.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.359+0000 7fb9f2a00640 1 -- 172.21.15.90:0/3770294932 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9f801a3f0 con 0x5598a2d129f0 2024-04-02T14:47:15.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.360+0000 7fba0ae76740 1 -- 172.21.15.90:0/3770294932 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5598a2d129f0 msgr2=0x5598a2d11740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:15.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.360+0000 7fba0ae76740 1 --2- 172.21.15.90:0/3770294932 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5598a2d129f0 0x5598a2d11740 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7fb9f80094d0 tx=0x7fb9f8011870 comp rx=0 tx=0).stop 2024-04-02T14:47:15.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.360+0000 7fba0ae76740 1 -- 172.21.15.90:0/3770294932 shutdown_connections 2024-04-02T14:47:15.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.360+0000 7fba0ae76740 1 --2- 172.21.15.90:0/3770294932 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5598a2d1b080 0x5598a2d1d470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:15.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.360+0000 7fba0ae76740 1 --2- 172.21.15.90:0/3770294932 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5598a2d11d10 0x5598a2d12190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:15.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.360+0000 7fba0ae76740 1 --2- 172.21.15.90:0/3770294932 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5598a2d129f0 0x5598a2d11740 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:15.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.360+0000 7fba0ae76740 1 -- 172.21.15.90:0/3770294932 >> 172.21.15.90:0/3770294932 conn(0x5598a2cd8a50 msgr2=0x5598a2d0c760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:47:15.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.361+0000 7fba0ae76740 1 -- 172.21.15.90:0/3770294932 shutdown_connections 2024-04-02T14:47:15.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.363+0000 7fba0ae76740 1 -- 172.21.15.90:0/3770294932 wait complete. 2024-04-02T14:47:15.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.363+0000 7fba05800640 1 --2- 172.21.15.90:0/141378004 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fb9e80a3da0 0x7fb9e80a6260 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fb9f4006010 tx=0x7fb9f402e040 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:47:15.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.363+0000 7fba0ae76740 1 -- start start 2024-04-02T14:47:15.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.364+0000 7fba0ae76740 1 Processor -- start 2024-04-02T14:47:15.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.366+0000 7fba0ae76740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5598a2d11d10 0x5598a2d18090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:15.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.366+0000 7fba0ae76740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5598a2d129f0 0x5598a2d185d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:15.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.366+0000 7fba0ae76740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5598a2d1b080 0x5598a2d18b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:15.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.366+0000 7fba0ae76740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5598a2c726a0 con 0x5598a2d11d10 2024-04-02T14:47:15.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.366+0000 7fba0ae76740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x5598a2c1e980 con 0x5598a2d1b080 2024-04-02T14:47:15.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.366+0000 7fba0ae76740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5598a2c31cb0 con 0x5598a2d129f0 2024-04-02T14:47:15.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.366+0000 7fba0ae76740 10 client.0 _ll_register_callbacks cb 0x5598a2c316a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:47:15.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.366+0000 7fba04e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5598a2d129f0 0x5598a2d185d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:15.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.366+0000 7fba04e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5598a2d129f0 0x5598a2d185d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:59596/0 (socket says 172.21.15.90:59596) 2024-04-02T14:47:15.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.366+0000 7fba04e00640 1 -- 172.21.15.90:0/1186489767 learned_addr learned my addr 172.21.15.90:0/1186489767 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:47:15.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.367+0000 7fba06200640 1 --2- 172.21.15.90:0/1186489767 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5598a2d1b080 0x5598a2d18b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:15.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.367+0000 7fba05800640 1 --2- 172.21.15.90:0/1186489767 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5598a2d11d10 0x5598a2d18090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:15.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.368+0000 7fba04e00640 1 -- 172.21.15.90:0/1186489767 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5598a2d1b080 msgr2=0x5598a2d18b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:47:15.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.368+0000 7fba04e00640 1 --2- 172.21.15.90:0/1186489767 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5598a2d1b080 0x5598a2d18b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:15.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.368+0000 7fba04e00640 1 -- 172.21.15.90:0/1186489767 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5598a2d11d10 msgr2=0x5598a2d18090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:47:15.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.368+0000 7fba04e00640 1 --2- 172.21.15.90:0/1186489767 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5598a2d11d10 0x5598a2d18090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:15.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.368+0000 7fba04e00640 1 -- 172.21.15.90:0/1186489767 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5598a2a547d0 con 0x5598a2d129f0 2024-04-02T14:47:15.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.368+0000 7fba05800640 1 --2- 172.21.15.90:0/1186489767 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5598a2d11d10 0x5598a2d18090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:47:15.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.369+0000 7fba04e00640 1 --2- 172.21.15.90:0/1186489767 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5598a2d129f0 0x5598a2d185d0 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7fb9ec099560 tx=0x7fb9ec09c5e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=924650bcc3a66f38 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:47:15.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.369+0000 7fb9e7400640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:47:15.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.370+0000 7fb9e7400640 1 -- 172.21.15.90:0/1186489767 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9ec0a5800 con 0x5598a2d129f0 2024-04-02T14:47:15.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.370+0000 7fb9e7400640 1 -- 172.21.15.90:0/1186489767 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 8550+0+0 (secure 0 0 0) 0x7fb9ec010ac0 con 0x5598a2d129f0 2024-04-02T14:47:15.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.370+0000 7fba0ae76740 1 -- 172.21.15.90:0/1186489767 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb9f8012d90 con 0x5598a2d129f0 2024-04-02T14:47:15.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.371+0000 7fb9e7400640 1 -- 172.21.15.90:0/1186489767 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fb9ec09bfe0 con 0x5598a2d129f0 2024-04-02T14:47:15.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.371+0000 7fba0ae76740 10 client.29086 fetch_fsmap learned FSMap version 27 2024-04-02T14:47:15.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.371+0000 7fba0ae76740 1 -- 172.21.15.90:0/1186489767 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5598a2d29f90 con 0x5598a2d129f0 2024-04-02T14:47:15.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.371+0000 7fb9e7400640 1 -- 172.21.15.90:0/1186489767 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb9f8012d90 con 0x5598a2d129f0 2024-04-02T14:47:15.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.373+0000 7fb9e7400640 1 -- 172.21.15.90:0/1186489767 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fb9ec098ae0 con 0x5598a2d129f0 2024-04-02T14:47:15.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.374+0000 7fba0ae76740 10 client.29086 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:47:15.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.374+0000 7fba0ae76740 10 client.29086 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-04-02T14:47:15.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.374+0000 7fba0ae76740 20 client.29086 populate_metadata read hostname 'smithi090' 2024-04-02T14:47:15.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.374+0000 7fba0ae76740 1 --2- 172.21.15.90:0/1186489767 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x5598a2d2b450 0x5598a2d2d840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:15.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.374+0000 7fba0ae76740 4 client.29086 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:47:15.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.374+0000 7fba0ae76740 1 -- 172.21.15.90:0/1186489767 --> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5598a2c36040 con 0x5598a2d2b450 2024-04-02T14:47:15.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.374+0000 7fba05800640 1 --2- 172.21.15.90:0/1186489767 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x5598a2d2b450 0x5598a2d2d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:15.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.375+0000 7fba05800640 1 --2- 172.21.15.90:0/1186489767 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x5598a2d2b450 0x5598a2d2d840 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=5eee81dbfd668591 server_cookie=b6b380e92add8898 in_seq=0 out_seq=0 2024-04-02T14:47:15.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.375+0000 7fb9e7400640 10 client.29086 ms_handle_connect on v2:172.21.15.92:6836/1270635939 2024-04-02T14:47:15.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.376+0000 7fb9e7400640 1 -- 172.21.15.90:0/1186489767 <== mds.3 v2:172.21.15.92:6836/1270635939 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5598a2c36040 con 0x5598a2d2b450 2024-04-02T14:47:15.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.377+0000 7fb9e7400640 10 client.29086 handle_command_reply: tid=0 2024-04-02T14:47:15.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.377+0000 7fba0ae76740 1 client.29086 shutdown 2024-04-02T14:47:15.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.378+0000 7fba0ae76740 1 -- 172.21.15.90:0/1186489767 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5598a2d129f0 msgr2=0x5598a2d185d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:15.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.378+0000 7fba0ae76740 1 --2- 172.21.15.90:0/1186489767 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5598a2d129f0 0x5598a2d185d0 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7fb9ec099560 tx=0x7fb9ec09c5e0 comp rx=0 tx=0).stop 2024-04-02T14:47:15.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.378+0000 7fba0ae76740 1 -- 172.21.15.90:0/1186489767 shutdown_connections 2024-04-02T14:47:15.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.378+0000 7fba0ae76740 1 --2- 172.21.15.90:0/1186489767 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x5598a2d2b450 0x5598a2d2d840 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:15.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.378+0000 7fba0ae76740 1 --2- 172.21.15.90:0/1186489767 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5598a2d1b080 0x5598a2d18b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:15.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.378+0000 7fba0ae76740 1 --2- 172.21.15.90:0/1186489767 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5598a2d129f0 0x5598a2d185d0 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:15.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.378+0000 7fba0ae76740 1 --2- 172.21.15.90:0/1186489767 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5598a2d11d10 0x5598a2d18090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:15.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.378+0000 7fba0ae76740 1 -- 172.21.15.90:0/1186489767 >> 172.21.15.90:0/1186489767 conn(0x5598a2cd8a50 msgr2=0x5598a2d0e4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:47:15.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.379+0000 7fb9e7400640 0 client.29086 ms_handle_reset on v2:172.21.15.92:6836/1270635939 2024-04-02T14:47:15.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.379+0000 7fba0ae76740 1 -- 172.21.15.90:0/1186489767 shutdown_connections 2024-04-02T14:47:15.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.380+0000 7fba0ae76740 1 -- 172.21.15.90:0/1186489767 wait complete. 2024-04-02T14:47:15.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.380+0000 7fba0ae76740 20 client.29086 trim_cache size 0 max 16384 2024-04-02T14:47:15.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.400+0000 7fba0ae76740 1 Processor -- start 2024-04-02T14:47:15.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.400+0000 7fba0ae76740 1 -- start start 2024-04-02T14:47:15.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.400+0000 7fba0ae76740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5598a2d11d10 0x5598a2d1a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:15.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.400+0000 7fba0ae76740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5598a2d129f0 0x5598a2d26af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:15.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.400+0000 7fba0ae76740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5598a2d1b080 0x5598a2d0b720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:15.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.401+0000 7fba0ae76740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fba001b40f0 con 0x5598a2d1b080 2024-04-02T14:47:15.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.401+0000 7fba0ae76740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fba001b5c30 con 0x5598a2d11d10 2024-04-02T14:47:15.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.401+0000 7fba0ae76740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fba001b47c0 con 0x5598a2d129f0 2024-04-02T14:47:15.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.401+0000 7fba06200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5598a2d1b080 0x5598a2d0b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:15.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.401+0000 7fba06200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5598a2d1b080 0x5598a2d0b720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:37012/0 (socket says 172.21.15.90:37012) 2024-04-02T14:47:15.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.401+0000 7fba06200640 1 -- 172.21.15.90:0/2403231093 learned_addr learned my addr 172.21.15.90:0/2403231093 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:47:15.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.401+0000 7fba05800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5598a2d11d10 0x5598a2d1a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:15.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.401+0000 7fba04e00640 1 --2- 172.21.15.90:0/2403231093 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5598a2d129f0 0x5598a2d26af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:15.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.401+0000 7fba06200640 1 -- 172.21.15.90:0/2403231093 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5598a2d129f0 msgr2=0x5598a2d26af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:47:15.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.401+0000 7fba06200640 1 --2- 172.21.15.90:0/2403231093 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5598a2d129f0 0x5598a2d26af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:15.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.401+0000 7fba06200640 1 -- 172.21.15.90:0/2403231093 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5598a2d11d10 msgr2=0x5598a2d1a7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:47:15.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.401+0000 7fba06200640 1 --2- 172.21.15.90:0/2403231093 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5598a2d11d10 0x5598a2d1a7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:15.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.401+0000 7fba06200640 1 -- 172.21.15.90:0/2403231093 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5598a2d10d40 con 0x5598a2d1b080 2024-04-02T14:47:15.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.402+0000 7fba06200640 1 --2- 172.21.15.90:0/2403231093 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5598a2d1b080 0x5598a2d0b720 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fb9f800fc20 tx=0x7fb9f8009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3cab01530b57146 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:47:15.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.402+0000 7fb9f2000640 1 -- 172.21.15.90:0/2403231093 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9f802b020 con 0x5598a2d1b080 2024-04-02T14:47:15.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.402+0000 7fb9f2000640 1 -- 172.21.15.90:0/2403231093 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb9f800fc50 con 0x5598a2d1b080 2024-04-02T14:47:15.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.402+0000 7fb9f2000640 1 -- 172.21.15.90:0/2403231093 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9f8012770 con 0x5598a2d1b080 2024-04-02T14:47:15.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.403+0000 7fba0ae76740 1 -- 172.21.15.90:0/2403231093 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5598a2d1b080 msgr2=0x5598a2d0b720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:15.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.403+0000 7fba0ae76740 1 --2- 172.21.15.90:0/2403231093 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5598a2d1b080 0x5598a2d0b720 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fb9f800fc20 tx=0x7fb9f8009b20 comp rx=0 tx=0).stop 2024-04-02T14:47:15.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.403+0000 7fba0ae76740 1 -- 172.21.15.90:0/2403231093 shutdown_connections 2024-04-02T14:47:15.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.403+0000 7fba0ae76740 1 --2- 172.21.15.90:0/2403231093 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5598a2d1b080 0x5598a2d0b720 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:15.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.403+0000 7fba0ae76740 1 --2- 172.21.15.90:0/2403231093 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5598a2d129f0 0x5598a2d26af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:15.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.403+0000 7fba0ae76740 1 --2- 172.21.15.90:0/2403231093 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5598a2d11d10 0x5598a2d1a7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:15.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.403+0000 7fba0ae76740 1 -- 172.21.15.90:0/2403231093 >> 172.21.15.90:0/2403231093 conn(0x5598a2cd8a50 msgr2=0x5598a2d0b010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:47:15.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.403+0000 7fba0ae76740 1 -- 172.21.15.90:0/2403231093 shutdown_connections 2024-04-02T14:47:15.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.403+0000 7fba0ae76740 1 -- 172.21.15.90:0/2403231093 wait complete. 2024-04-02T14:47:15.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.404+0000 7fba0ae76740 1 -- start start 2024-04-02T14:47:15.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.404+0000 7fba0ae76740 1 Processor -- start 2024-04-02T14:47:15.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.404+0000 7fba0ae76740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5598a2d11d10 0x5598a2d1c440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:15.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.404+0000 7fba0ae76740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5598a2d129f0 0x5598a2d1c980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:15.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.404+0000 7fba0ae76740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5598a2d1b080 0x5598a2d21750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:15.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.404+0000 7fba0ae76740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fba001b5560 con 0x5598a2d1b080 2024-04-02T14:47:15.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.404+0000 7fba0ae76740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fba001b4e90 con 0x5598a2d129f0 2024-04-02T14:47:15.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.404+0000 7fba0ae76740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fba001b2110 con 0x5598a2d11d10 2024-04-02T14:47:15.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.404+0000 7fba0ae76740 10 client.0 _ll_register_callbacks cb 0x7fb9ec09c6d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:47:15.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.404+0000 7fba05800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5598a2d11d10 0x5598a2d1c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:15.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.404+0000 7fba05800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5598a2d11d10 0x5598a2d1c440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:59616/0 (socket says 172.21.15.90:59616) 2024-04-02T14:47:15.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.405+0000 7fba05800640 1 -- 172.21.15.90:0/3821089902 learned_addr learned my addr 172.21.15.90:0/3821089902 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:47:15.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.405+0000 7fba04e00640 1 --2- 172.21.15.90:0/3821089902 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5598a2d129f0 0x5598a2d1c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:15.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.405+0000 7fba06200640 1 --2- 172.21.15.90:0/3821089902 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5598a2d1b080 0x5598a2d21750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:15.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.405+0000 7fba05800640 1 -- 172.21.15.90:0/3821089902 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5598a2d129f0 msgr2=0x5598a2d1c980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:47:15.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.405+0000 7fba05800640 1 --2- 172.21.15.90:0/3821089902 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5598a2d129f0 0x5598a2d1c980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:15.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.405+0000 7fba05800640 1 -- 172.21.15.90:0/3821089902 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5598a2d1b080 msgr2=0x5598a2d21750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:47:15.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.405+0000 7fba05800640 1 --2- 172.21.15.90:0/3821089902 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5598a2d1b080 0x5598a2d21750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:15.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.405+0000 7fba05800640 1 -- 172.21.15.90:0/3821089902 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5598a2d21d90 con 0x5598a2d11d10 2024-04-02T14:47:15.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.405+0000 7fba05800640 1 --2- 172.21.15.90:0/3821089902 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5598a2d11d10 0x5598a2d1c440 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7fb9f402e010 tx=0x7fb9f4009260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=634fbb0e81c9c2f server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:47:15.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.405+0000 7fb9e7400640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:47:15.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.406+0000 7fb9e7400640 1 -- 172.21.15.90:0/3821089902 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9f402ccb0 con 0x5598a2d11d10 2024-04-02T14:47:15.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.406+0000 7fb9e7400640 1 -- 172.21.15.90:0/3821089902 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 8550+0+0 (secure 0 0 0) 0x7fb9f40046c0 con 0x5598a2d11d10 2024-04-02T14:47:15.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.406+0000 7fba0ae76740 1 -- 172.21.15.90:0/3821089902 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb9f800fc50 con 0x5598a2d11d10 2024-04-02T14:47:15.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.406+0000 7fb9e7400640 1 -- 172.21.15.90:0/3821089902 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fb9f40048c0 con 0x5598a2d11d10 2024-04-02T14:47:15.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.406+0000 7fba0ae76740 10 client.29092 fetch_fsmap learned FSMap version 27 2024-04-02T14:47:15.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.406+0000 7fba0ae76740 1 -- 172.21.15.90:0/3821089902 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5598a2d921a0 con 0x5598a2d11d10 2024-04-02T14:47:15.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.406+0000 7fb9e7400640 1 -- 172.21.15.90:0/3821089902 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb9f800fc50 con 0x5598a2d11d10 2024-04-02T14:47:15.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.407+0000 7fb9e7400640 1 -- 172.21.15.90:0/3821089902 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fb9f405a070 con 0x5598a2d11d10 2024-04-02T14:47:15.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.407+0000 7fba0ae76740 10 client.29092 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:47:15.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.407+0000 7fba0ae76740 10 client.29092 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-04-02T14:47:15.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.407+0000 7fba0ae76740 20 client.29092 populate_metadata read hostname 'smithi090' 2024-04-02T14:47:15.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.407+0000 7fba0ae76740 1 --2- 172.21.15.90:0/3821089902 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x5598a2d928d0 0x5598a2d94cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:15.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.407+0000 7fba0ae76740 4 client.29092 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-02T14:47:15.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.407+0000 7fba0ae76740 1 -- 172.21.15.90:0/3821089902 --> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5598a2c83c30 con 0x5598a2d928d0 2024-04-02T14:47:15.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.408+0000 7fba04e00640 1 --2- 172.21.15.90:0/3821089902 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x5598a2d928d0 0x5598a2d94cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:15.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.408+0000 7fba04e00640 1 --2- 172.21.15.90:0/3821089902 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x5598a2d928d0 0x5598a2d94cf0 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=5ad5fbaa41a4628f server_cookie=bc9b275869fe77bc in_seq=0 out_seq=0 2024-04-02T14:47:15.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.409+0000 7fb9e7400640 10 client.29092 ms_handle_connect on v2:172.21.15.92:6836/1270635939 2024-04-02T14:47:15.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.409+0000 7fb9e7400640 1 -- 172.21.15.90:0/3821089902 <== mds.3 v2:172.21.15.92:6836/1270635939 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5598a2c83c30 con 0x5598a2d928d0 2024-04-02T14:47:15.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.409+0000 7fb9e7400640 10 client.29092 handle_command_reply: tid=0 2024-04-02T14:47:15.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.409+0000 7fba0ae76740 1 client.29092 shutdown 2024-04-02T14:47:15.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.410+0000 7fba0ae76740 1 -- 172.21.15.90:0/3821089902 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5598a2d11d10 msgr2=0x5598a2d1c440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:15.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.410+0000 7fba0ae76740 1 --2- 172.21.15.90:0/3821089902 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5598a2d11d10 0x5598a2d1c440 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7fb9f402e010 tx=0x7fb9f4009260 comp rx=0 tx=0).stop 2024-04-02T14:47:15.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.410+0000 7fba0ae76740 1 -- 172.21.15.90:0/3821089902 shutdown_connections 2024-04-02T14:47:15.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.410+0000 7fba0ae76740 1 --2- 172.21.15.90:0/3821089902 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x5598a2d928d0 0x5598a2d94cf0 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:15.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.410+0000 7fba0ae76740 1 --2- 172.21.15.90:0/3821089902 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5598a2d1b080 0x5598a2d21750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:15.406 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.410+0000 7fba0ae76740 1 --2- 172.21.15.90:0/3821089902 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5598a2d129f0 0x5598a2d1c980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:15.406 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.410+0000 7fb9e7400640 0 client.29092 ms_handle_reset on v2:172.21.15.92:6836/1270635939 2024-04-02T14:47:15.406 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.410+0000 7fba0ae76740 1 --2- 172.21.15.90:0/3821089902 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5598a2d11d10 0x5598a2d1c440 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:15.406 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.410+0000 7fba0ae76740 1 -- 172.21.15.90:0/3821089902 >> 172.21.15.90:0/3821089902 conn(0x5598a2cd8a50 msgr2=0x5598a2d0e440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:47:15.406 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.411+0000 7fba0ae76740 1 -- 172.21.15.90:0/3821089902 shutdown_connections 2024-04-02T14:47:15.406 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.411+0000 7fba0ae76740 1 -- 172.21.15.90:0/3821089902 wait complete. 2024-04-02T14:47:15.406 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.411+0000 7fba0ae76740 20 client.29092 trim_cache size 0 max 16384 2024-04-02T14:47:15.406 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-02T14:47:15.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.412+0000 7fba08200640 1 -- 172.21.15.90:0/141378004 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fb9e80a3da0 msgr2=0x7fb9e80a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:15.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.412+0000 7fba08200640 1 --2- 172.21.15.90:0/141378004 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fb9e80a3da0 0x7fb9e80a6260 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fb9f4006010 tx=0x7fb9f402e040 comp rx=0 tx=0).stop 2024-04-02T14:47:15.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.412+0000 7fba08200640 1 -- 172.21.15.90:0/141378004 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fba00108fe0 msgr2=0x7fba0019ab50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:15.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.412+0000 7fba08200640 1 --2- 172.21.15.90:0/141378004 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fba00108fe0 0x7fba0019ab50 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7fb9ec00e9c0 tx=0x7fb9ec00ee80 comp rx=0 tx=0).stop 2024-04-02T14:47:15.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.412+0000 7fba08200640 1 -- 172.21.15.90:0/141378004 shutdown_connections 2024-04-02T14:47:15.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.412+0000 7fba08200640 1 --2- 172.21.15.90:0/141378004 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fb9e80a3da0 0x7fb9e80a6260 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:15.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.412+0000 7fba08200640 1 --2- 172.21.15.90:0/141378004 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fba0019eed0 0x7fba0019f380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:15.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.412+0000 7fba08200640 1 --2- 172.21.15.90:0/141378004 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fba00108fe0 0x7fba0019ab50 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:15.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.412+0000 7fba08200640 1 --2- 172.21.15.90:0/141378004 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fba00103f10 0x7fba0019a610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:15.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.412+0000 7fba08200640 1 -- 172.21.15.90:0/141378004 >> 172.21.15.90:0/141378004 conn(0x7fba000ff7b0 msgr2=0x7fba00101b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:47:15.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.412+0000 7fba08200640 1 -- 172.21.15.90:0/141378004 shutdown_connections 2024-04-02T14:47:15.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.412+0000 7fba08200640 1 -- 172.21.15.90:0/141378004 wait complete. 2024-04-02T14:47:15.432 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-02T14:47:15.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.686+0000 7f14d4600640 1 -- 172.21.15.90:0/3950213744 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f14cc073ed0 msgr2=0x7f14cc0742b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:15.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.686+0000 7f14d4600640 1 --2- 172.21.15.90:0/3950213744 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f14cc073ed0 0x7f14cc0742b0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f14b4009860 tx=0x7f14b402bc10 comp rx=0 tx=0).stop 2024-04-02T14:47:15.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.687+0000 7f14d4600640 1 -- 172.21.15.90:0/3950213744 shutdown_connections 2024-04-02T14:47:15.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.687+0000 7f14d4600640 1 --2- 172.21.15.90:0/3950213744 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f14cc10e330 0x7f14cc110740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:15.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.687+0000 7f14d4600640 1 --2- 172.21.15.90:0/3950213744 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f14cc0747f0 0x7f14cc10ddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:15.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.687+0000 7f14d4600640 1 --2- 172.21.15.90:0/3950213744 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f14cc073ed0 0x7f14cc0742b0 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:15.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.687+0000 7f14d4600640 1 -- 172.21.15.90:0/3950213744 >> 172.21.15.90:0/3950213744 conn(0x7f14cc0fd560 msgr2=0x7f14cc0ff980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:47:15.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.687+0000 7f14d4600640 1 -- 172.21.15.90:0/3950213744 shutdown_connections 2024-04-02T14:47:15.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.688+0000 7f14d4600640 1 -- 172.21.15.90:0/3950213744 wait complete. 2024-04-02T14:47:15.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.689+0000 7f14d4600640 1 Processor -- start 2024-04-02T14:47:15.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.689+0000 7f14d4600640 1 -- start start 2024-04-02T14:47:15.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.689+0000 7f14d4600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f14cc073ed0 0x7f14cc19a630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:15.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.690+0000 7f14d4600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f14cc0747f0 0x7f14cc19ab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:15.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.690+0000 7f14d4600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f14cc10e330 0x7f14cc19eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:15.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.690+0000 7f14d4600640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f14cc1140b0 con 0x7f14cc073ed0 2024-04-02T14:47:15.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.690+0000 7f14d4600640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f14cc113f30 con 0x7f14cc10e330 2024-04-02T14:47:15.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.690+0000 7f14d4600640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f14cc114230 con 0x7f14cc0747f0 2024-04-02T14:47:15.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.690+0000 7f14d1200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f14cc0747f0 0x7f14cc19ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:15.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.690+0000 7f14d1c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f14cc073ed0 0x7f14cc19a630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:15.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.690+0000 7f14d1200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f14cc0747f0 0x7f14cc19ab70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:59628/0 (socket says 172.21.15.90:59628) 2024-04-02T14:47:15.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.690+0000 7f14d1200640 1 -- 172.21.15.90:0/2998190289 learned_addr learned my addr 172.21.15.90:0/2998190289 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:47:15.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.690+0000 7f14d2600640 1 --2- 172.21.15.90:0/2998190289 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f14cc10e330 0x7f14cc19eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:15.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.690+0000 7f14d1200640 1 -- 172.21.15.90:0/2998190289 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f14cc10e330 msgr2=0x7f14cc19eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:15.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.690+0000 7f14d1200640 1 --2- 172.21.15.90:0/2998190289 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f14cc10e330 0x7f14cc19eef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:15.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.690+0000 7f14d1200640 1 -- 172.21.15.90:0/2998190289 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f14cc073ed0 msgr2=0x7f14cc19a630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:15.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.690+0000 7f14d1200640 1 --2- 172.21.15.90:0/2998190289 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f14cc073ed0 0x7f14cc19a630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:15.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.690+0000 7f14d1200640 1 -- 172.21.15.90:0/2998190289 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14cc19f620 con 0x7f14cc0747f0 2024-04-02T14:47:15.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.691+0000 7f14d1200640 1 --2- 172.21.15.90:0/2998190289 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f14cc0747f0 0x7f14cc19ab70 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f14bc00eb90 tx=0x7f14bc00ef90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:47:15.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.691+0000 7f14ca000640 1 -- 172.21.15.90:0/2998190289 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14bc004b20 con 0x7f14cc0747f0 2024-04-02T14:47:15.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.691+0000 7f14ca000640 1 -- 172.21.15.90:0/2998190289 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f14bc004cc0 con 0x7f14cc0747f0 2024-04-02T14:47:15.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.691+0000 7f14d4600640 1 -- 172.21.15.90:0/2998190289 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14cc19f910 con 0x7f14cc0747f0 2024-04-02T14:47:15.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.691+0000 7f14ca000640 1 -- 172.21.15.90:0/2998190289 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14bc010720 con 0x7f14cc0747f0 2024-04-02T14:47:15.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.691+0000 7f14d4600640 1 -- 172.21.15.90:0/2998190289 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14cc1a71a0 con 0x7f14cc0747f0 2024-04-02T14:47:15.690 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.696+0000 7f14ca000640 1 -- 172.21.15.90:0/2998190289 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f14bc005020 con 0x7f14cc0747f0 2024-04-02T14:47:15.691 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.697+0000 7f14d73c4740 1 Processor -- start 2024-04-02T14:47:15.691 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.697+0000 7f14d73c4740 1 -- start start 2024-04-02T14:47:15.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.697+0000 7f14d73c4740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559a1499d9f0 0x559a1499c740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:15.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.698+0000 7f14d73c4740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559a1499cd10 0x559a1499d190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:15.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.698+0000 7f14d73c4740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559a149a6080 0x559a149a8470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:15.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.698+0000 7f14d73c4740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x559a148dd400 con 0x559a149a6080 2024-04-02T14:47:15.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.698+0000 7f14d73c4740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x559a148c1f10 con 0x559a1499d9f0 2024-04-02T14:47:15.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.698+0000 7f14d73c4740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x559a149016f0 con 0x559a1499cd10 2024-04-02T14:47:15.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.698+0000 7f14d2600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559a1499d9f0 0x559a1499c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:15.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.698+0000 7f14d2600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559a1499d9f0 0x559a1499c740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:53134/0 (socket says 172.21.15.90:53134) 2024-04-02T14:47:15.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.698+0000 7f14d2600640 1 -- 172.21.15.90:0/3680956513 learned_addr learned my addr 172.21.15.90:0/3680956513 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:47:15.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.698+0000 7f14d1c00640 1 --2- 172.21.15.90:0/3680956513 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559a1499cd10 0x559a1499d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:15.694 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.698+0000 7f14d1200640 1 --2- 172.21.15.90:0/3680956513 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559a149a6080 0x559a149a8470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:15.694 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.698+0000 7f14ca000640 1 --2- 172.21.15.90:0/2998190289 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f14b00a3da0 0x7f14b00a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:15.694 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.698+0000 7f14d1c00640 1 -- 172.21.15.90:0/3680956513 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559a1499d9f0 msgr2=0x559a1499c740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:47:15.694 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.698+0000 7f14d1c00640 1 --2- 172.21.15.90:0/3680956513 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559a1499d9f0 0x559a1499c740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:15.694 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.698+0000 7f14d1c00640 1 -- 172.21.15.90:0/3680956513 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559a149a6080 msgr2=0x559a149a8470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:47:15.694 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.698+0000 7f14d1c00640 1 --2- 172.21.15.90:0/3680956513 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559a149a6080 0x559a149a8470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:15.694 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.698+0000 7f14d1c00640 1 -- 172.21.15.90:0/3680956513 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559a146153f0 con 0x559a1499cd10 2024-04-02T14:47:15.694 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.698+0000 7f14d2600640 1 --2- 172.21.15.90:0/3680956513 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559a1499d9f0 0x559a1499c740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:47:15.694 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.699+0000 7f14ca000640 1 -- 172.21.15.90:0/2998190289 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 8550+0+0 (secure 0 0 0) 0x7f14bc0d0dc0 con 0x7f14cc0747f0 2024-04-02T14:47:15.694 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.699+0000 7f14d1c00640 1 --2- 172.21.15.90:0/3680956513 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559a1499cd10 0x559a1499d190 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f14b4037040 tx=0x7f14b4009510 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7c1b4b5fd777b14c server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:47:15.694 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.699+0000 7f14d2600640 1 --2- 172.21.15.90:0/2998190289 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f14b00a3da0 0x7f14b00a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:15.695 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.699+0000 7f14c2a00640 1 -- 172.21.15.90:0/3680956513 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14b4043020 con 0x559a1499cd10 2024-04-02T14:47:15.695 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.699+0000 7f14c2a00640 1 -- 172.21.15.90:0/3680956513 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f14b4034440 con 0x559a1499cd10 2024-04-02T14:47:15.695 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.699+0000 7f14c2a00640 1 -- 172.21.15.90:0/3680956513 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14b4039040 con 0x559a1499cd10 2024-04-02T14:47:15.695 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.700+0000 7f14d73c4740 1 -- 172.21.15.90:0/3680956513 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559a1499cd10 msgr2=0x559a1499d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:15.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.700+0000 7f14d73c4740 1 --2- 172.21.15.90:0/3680956513 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559a1499cd10 0x559a1499d190 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f14b4037040 tx=0x7f14b4009510 comp rx=0 tx=0).stop 2024-04-02T14:47:15.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.700+0000 7f14d2600640 1 --2- 172.21.15.90:0/2998190289 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f14b00a3da0 0x7f14b00a6260 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f14c4004040 tx=0x7f14c40046c0 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:47:15.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.702+0000 7f14d73c4740 1 -- 172.21.15.90:0/3680956513 shutdown_connections 2024-04-02T14:47:15.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.702+0000 7f14d73c4740 1 --2- 172.21.15.90:0/3680956513 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559a149a6080 0x559a149a8470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:15.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.702+0000 7f14d73c4740 1 --2- 172.21.15.90:0/3680956513 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559a1499cd10 0x559a1499d190 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:15.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.702+0000 7f14d73c4740 1 --2- 172.21.15.90:0/3680956513 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559a1499d9f0 0x559a1499c740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:15.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.702+0000 7f14d73c4740 1 -- 172.21.15.90:0/3680956513 >> 172.21.15.90:0/3680956513 conn(0x559a14963a50 msgr2=0x559a14997760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:47:15.697 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.702+0000 7f14d73c4740 1 -- 172.21.15.90:0/3680956513 shutdown_connections 2024-04-02T14:47:15.697 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.702+0000 7f14d73c4740 1 -- 172.21.15.90:0/3680956513 wait complete. 2024-04-02T14:47:15.697 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.703+0000 7f14d73c4740 1 -- start start 2024-04-02T14:47:15.697 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.703+0000 7f14d73c4740 1 Processor -- start 2024-04-02T14:47:15.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.704+0000 7f14d73c4740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559a1499cd10 0x559a149a3080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:15.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.704+0000 7f14d73c4740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559a1499d9f0 0x559a149a35c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:15.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.704+0000 7f14d73c4740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559a149a6080 0x559a149a3b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:15.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.704+0000 7f14d73c4740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x559a148fd6a0 con 0x559a149a6080 2024-04-02T14:47:15.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.704+0000 7f14d73c4740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x559a148a9980 con 0x559a1499d9f0 2024-04-02T14:47:15.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.704+0000 7f14d73c4740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x559a148bccb0 con 0x559a1499cd10 2024-04-02T14:47:15.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.704+0000 7f14d73c4740 10 client.0 _ll_register_callbacks cb 0x559a148bc6a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:47:15.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.704+0000 7f14d1c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559a1499cd10 0x559a149a3080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:15.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.704+0000 7f14d1c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559a1499cd10 0x559a149a3080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:59646/0 (socket says 172.21.15.90:59646) 2024-04-02T14:47:15.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.704+0000 7f14d1c00640 1 -- 172.21.15.90:0/1555755370 learned_addr learned my addr 172.21.15.90:0/1555755370 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:47:15.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.704+0000 7f14d1200640 1 --2- 172.21.15.90:0/1555755370 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559a1499d9f0 0x559a149a35c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:15.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.704+0000 7f14d2600640 1 --2- 172.21.15.90:0/1555755370 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559a149a6080 0x559a149a3b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:15.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.704+0000 7f14d1c00640 1 -- 172.21.15.90:0/1555755370 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559a1499d9f0 msgr2=0x559a149a35c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:47:15.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.704+0000 7f14d1c00640 1 --2- 172.21.15.90:0/1555755370 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559a1499d9f0 0x559a149a35c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:15.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.704+0000 7f14d1c00640 1 -- 172.21.15.90:0/1555755370 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559a149a6080 msgr2=0x559a149a3b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:47:15.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.704+0000 7f14d1c00640 1 --2- 172.21.15.90:0/1555755370 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559a149a6080 0x559a149a3b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:15.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.704+0000 7f14d1c00640 1 -- 172.21.15.90:0/1555755370 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559a146df7d0 con 0x559a1499cd10 2024-04-02T14:47:15.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.704+0000 7f14d2600640 1 --2- 172.21.15.90:0/1555755370 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559a149a6080 0x559a149a3b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:47:15.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.705+0000 7f14d1c00640 1 --2- 172.21.15.90:0/1555755370 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559a1499cd10 0x559a149a3080 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f14b4005e00 tx=0x7f14b40349a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=990063d4ed35430e server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:47:15.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.705+0000 7f14bb400640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:47:15.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.705+0000 7f14bb400640 1 -- 172.21.15.90:0/1555755370 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14b4043030 con 0x559a1499cd10 2024-04-02T14:47:15.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.705+0000 7f14bb400640 1 -- 172.21.15.90:0/1555755370 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 8550+0+0 (secure 0 0 0) 0x7f14b402c950 con 0x559a1499cd10 2024-04-02T14:47:15.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.705+0000 7f14d73c4740 1 -- 172.21.15.90:0/1555755370 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f14b4034440 con 0x559a1499cd10 2024-04-02T14:47:15.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.705+0000 7f14bb400640 1 -- 172.21.15.90:0/1555755370 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f14b4039040 con 0x559a1499cd10 2024-04-02T14:47:15.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.705+0000 7f14d73c4740 10 client.29104 fetch_fsmap learned FSMap version 27 2024-04-02T14:47:15.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.705+0000 7f14d73c4740 1 -- 172.21.15.90:0/1555755370 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x559a149b4ed0 con 0x559a1499cd10 2024-04-02T14:47:15.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.705+0000 7f14bb400640 1 -- 172.21.15.90:0/1555755370 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f14b4034440 con 0x559a1499cd10 2024-04-02T14:47:15.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.705+0000 7f14bb400640 1 -- 172.21.15.90:0/1555755370 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f14b4052430 con 0x559a1499cd10 2024-04-02T14:47:15.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.706+0000 7f14d73c4740 10 client.29104 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:47:15.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.706+0000 7f14d73c4740 10 client.29104 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-04-02T14:47:15.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.706+0000 7f14d73c4740 20 client.29104 populate_metadata read hostname 'smithi090' 2024-04-02T14:47:15.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.706+0000 7f14d73c4740 1 --2- 172.21.15.90:0/1555755370 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x559a149b6220 0x559a149b8610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:15.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.706+0000 7f14d73c4740 4 client.29104 mds_command: new command op to 24547 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:47:15.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.706+0000 7f14d73c4740 1 -- 172.21.15.90:0/1555755370 --> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559a148c1040 con 0x559a149b6220 2024-04-02T14:47:15.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.706+0000 7f14d1200640 1 --2- 172.21.15.90:0/1555755370 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x559a149b6220 0x559a149b8610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:15.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.707+0000 7f14d1200640 1 --2- 172.21.15.90:0/1555755370 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x559a149b6220 0x559a149b8610 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=56a97e79c8ba6a0 server_cookie=47af495eeb382297 in_seq=0 out_seq=0 2024-04-02T14:47:15.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.707+0000 7f14bb400640 10 client.29104 ms_handle_connect on v2:172.21.15.177:6838/664895663 2024-04-02T14:47:15.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.708+0000 7f14bb400640 1 -- 172.21.15.90:0/1555755370 <== mds.2 v2:172.21.15.177:6838/664895663 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x559a148c1040 con 0x559a149b6220 2024-04-02T14:47:15.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.708+0000 7f14bb400640 10 client.29104 handle_command_reply: tid=0 2024-04-02T14:47:15.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.708+0000 7f14d73c4740 1 client.29104 shutdown 2024-04-02T14:47:15.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.709+0000 7f14d73c4740 1 -- 172.21.15.90:0/1555755370 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559a1499cd10 msgr2=0x559a149a3080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:15.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.709+0000 7f14d73c4740 1 --2- 172.21.15.90:0/1555755370 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559a1499cd10 0x559a149a3080 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f14b4005e00 tx=0x7f14b40349a0 comp rx=0 tx=0).stop 2024-04-02T14:47:15.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.709+0000 7f14d73c4740 1 -- 172.21.15.90:0/1555755370 shutdown_connections 2024-04-02T14:47:15.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.709+0000 7f14d73c4740 1 --2- 172.21.15.90:0/1555755370 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x559a149b6220 0x559a149b8610 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:15.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.709+0000 7f14d73c4740 1 --2- 172.21.15.90:0/1555755370 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559a149a6080 0x559a149a3b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:15.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.709+0000 7f14d73c4740 1 --2- 172.21.15.90:0/1555755370 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559a1499d9f0 0x559a149a35c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:15.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.709+0000 7f14d73c4740 1 --2- 172.21.15.90:0/1555755370 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559a1499cd10 0x559a149a3080 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:15.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.709+0000 7f14d73c4740 1 -- 172.21.15.90:0/1555755370 >> 172.21.15.90:0/1555755370 conn(0x559a14963a50 msgr2=0x559a149994b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:47:15.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.709+0000 7f14bb400640 0 client.29104 ms_handle_reset on v2:172.21.15.177:6838/664895663 2024-04-02T14:47:15.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.709+0000 7f14d73c4740 1 -- 172.21.15.90:0/1555755370 shutdown_connections 2024-04-02T14:47:15.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.710+0000 7f14d73c4740 1 -- 172.21.15.90:0/1555755370 wait complete. 2024-04-02T14:47:15.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.710+0000 7f14d73c4740 20 client.29104 trim_cache size 0 max 16384 2024-04-02T14:47:15.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.729+0000 7f14d73c4740 1 Processor -- start 2024-04-02T14:47:15.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.730+0000 7f14d73c4740 1 -- start start 2024-04-02T14:47:15.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.730+0000 7f14d73c4740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559a1499cd10 0x559a149a57b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:15.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.730+0000 7f14d73c4740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559a1499d9f0 0x559a149a22f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:15.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.730+0000 7f14d73c4740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559a149a6080 0x559a149a7470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:15.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.730+0000 7f14d73c4740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f14cc1afa80 con 0x559a149a6080 2024-04-02T14:47:15.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.730+0000 7f14d73c4740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f14cc1b15c0 con 0x559a1499d9f0 2024-04-02T14:47:15.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.730+0000 7f14d73c4740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f14cc1b0150 con 0x559a1499cd10 2024-04-02T14:47:15.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.730+0000 7f14d1c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559a1499cd10 0x559a149a57b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:15.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.730+0000 7f14d1c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559a1499cd10 0x559a149a57b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:59660/0 (socket says 172.21.15.90:59660) 2024-04-02T14:47:15.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.730+0000 7f14d1c00640 1 -- 172.21.15.90:0/727325160 learned_addr learned my addr 172.21.15.90:0/727325160 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:47:15.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.730+0000 7f14d1200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559a1499d9f0 0x559a149a22f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:15.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.730+0000 7f14d2600640 1 --2- 172.21.15.90:0/727325160 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559a149a6080 0x559a149a7470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:15.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.730+0000 7f14d1c00640 1 -- 172.21.15.90:0/727325160 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559a1499d9f0 msgr2=0x559a149a22f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:47:15.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.730+0000 7f14d1c00640 1 --2- 172.21.15.90:0/727325160 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559a1499d9f0 0x559a149a22f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:15.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.731+0000 7f14d1c00640 1 -- 172.21.15.90:0/727325160 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559a149a6080 msgr2=0x559a149a7470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:47:15.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.731+0000 7f14d1c00640 1 --2- 172.21.15.90:0/727325160 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559a149a6080 0x559a149a7470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:15.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.731+0000 7f14d1c00640 1 -- 172.21.15.90:0/727325160 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559a149999a0 con 0x559a1499cd10 2024-04-02T14:47:15.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.731+0000 7f14d1200640 1 --2- 172.21.15.90:0/727325160 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559a1499d9f0 0x559a149a22f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:47:15.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.731+0000 7f14d1c00640 1 --2- 172.21.15.90:0/727325160 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559a1499cd10 0x559a149a57b0 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f14b402c150 tx=0x7f14b4009510 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c3e7a57daaa86837 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:47:15.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.731+0000 7f14c2000640 1 -- 172.21.15.90:0/727325160 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14b405e070 con 0x559a1499cd10 2024-04-02T14:47:15.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.731+0000 7f14c2000640 1 -- 172.21.15.90:0/727325160 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f14b4034440 con 0x559a1499cd10 2024-04-02T14:47:15.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.731+0000 7f14c2000640 1 -- 172.21.15.90:0/727325160 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14b4043070 con 0x559a1499cd10 2024-04-02T14:47:15.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.732+0000 7f14d73c4740 1 -- 172.21.15.90:0/727325160 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559a1499cd10 msgr2=0x559a149a57b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:15.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.732+0000 7f14d73c4740 1 --2- 172.21.15.90:0/727325160 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559a1499cd10 0x559a149a57b0 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f14b402c150 tx=0x7f14b4009510 comp rx=0 tx=0).stop 2024-04-02T14:47:15.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.732+0000 7f14d73c4740 1 -- 172.21.15.90:0/727325160 shutdown_connections 2024-04-02T14:47:15.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.732+0000 7f14d73c4740 1 --2- 172.21.15.90:0/727325160 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559a149a6080 0x559a149a7470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:15.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.732+0000 7f14d73c4740 1 --2- 172.21.15.90:0/727325160 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559a1499d9f0 0x559a149a22f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:15.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.732+0000 7f14d73c4740 1 --2- 172.21.15.90:0/727325160 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559a1499cd10 0x559a149a57b0 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:15.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.732+0000 7f14d73c4740 1 -- 172.21.15.90:0/727325160 >> 172.21.15.90:0/727325160 conn(0x559a14963a50 msgr2=0x559a149b1830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:47:15.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.732+0000 7f14d73c4740 1 -- 172.21.15.90:0/727325160 shutdown_connections 2024-04-02T14:47:15.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.732+0000 7f14d73c4740 1 -- 172.21.15.90:0/727325160 wait complete. 2024-04-02T14:47:15.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.732+0000 7f14d73c4740 1 -- start start 2024-04-02T14:47:15.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.733+0000 7f14d73c4740 1 Processor -- start 2024-04-02T14:47:15.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.733+0000 7f14d73c4740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559a1499cd10 0x559a1499feb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:15.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.733+0000 7f14d73c4740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559a1499d9f0 0x559a149a03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:15.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.733+0000 7f14d73c4740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559a149a6080 0x559a149a0930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:15.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.733+0000 7f14d73c4740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f14cc1b0ef0 con 0x559a1499cd10 2024-04-02T14:47:15.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.733+0000 7f14d73c4740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f14cc1b0820 con 0x559a149a6080 2024-04-02T14:47:15.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.733+0000 7f14d73c4740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f14cc1ada80 con 0x559a1499d9f0 2024-04-02T14:47:15.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.733+0000 7f14d73c4740 10 client.0 _ll_register_callbacks cb 0x7f14b40096b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:47:15.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.734+0000 7f14d1c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559a1499cd10 0x559a1499feb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:15.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.734+0000 7f14d2600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559a149a6080 0x559a149a0930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:15.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.734+0000 7f14d2600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559a149a6080 0x559a149a0930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:53160/0 (socket says 172.21.15.90:53160) 2024-04-02T14:47:15.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.734+0000 7f14d2600640 1 -- 172.21.15.90:0/2757110415 learned_addr learned my addr 172.21.15.90:0/2757110415 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:47:15.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.734+0000 7f14d1200640 1 --2- 172.21.15.90:0/2757110415 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559a1499d9f0 0x559a149a03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:15.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.735+0000 7f14d1c00640 1 -- 172.21.15.90:0/2757110415 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559a1499d9f0 msgr2=0x559a149a03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:47:15.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.735+0000 7f14d1c00640 1 --2- 172.21.15.90:0/2757110415 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559a1499d9f0 0x559a149a03f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:15.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.736+0000 7f14d1c00640 1 -- 172.21.15.90:0/2757110415 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559a149a6080 msgr2=0x559a149a0930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:47:15.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.736+0000 7f14d1c00640 1 --2- 172.21.15.90:0/2757110415 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559a149a6080 0x559a149a0930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:15.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.736+0000 7f14d1c00640 1 -- 172.21.15.90:0/2757110415 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559a14a1b5a0 con 0x559a1499cd10 2024-04-02T14:47:15.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.736+0000 7f14d1200640 1 --2- 172.21.15.90:0/2757110415 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559a1499d9f0 0x559a149a03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:47:15.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.736+0000 7f14d2600640 1 --2- 172.21.15.90:0/2757110415 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559a149a6080 0x559a149a0930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:47:15.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.737+0000 7f14d1c00640 1 --2- 172.21.15.90:0/2757110415 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559a1499cd10 0x559a1499feb0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f14b402c220 tx=0x7f14b40349a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61de98da329169a6 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:47:15.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.737+0000 7f14bb400640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-02T14:47:15.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.737+0000 7f14bb400640 1 -- 172.21.15.90:0/2757110415 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14b4043070 con 0x559a1499cd10 2024-04-02T14:47:15.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.737+0000 7f14bb400640 1 -- 172.21.15.90:0/2757110415 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 8550+0+0 (secure 0 0 0) 0x7f14b402c850 con 0x559a1499cd10 2024-04-02T14:47:15.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.737+0000 7f14d73c4740 1 -- 172.21.15.90:0/2757110415 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f14b4034440 con 0x559a1499cd10 2024-04-02T14:47:15.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.737+0000 7f14bb400640 1 -- 172.21.15.90:0/2757110415 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f14b405e040 con 0x559a1499cd10 2024-04-02T14:47:15.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.737+0000 7f14d73c4740 10 client.18231 fetch_fsmap learned FSMap version 27 2024-04-02T14:47:15.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.737+0000 7f14d73c4740 1 -- 172.21.15.90:0/2757110415 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x559a14a1c860 con 0x559a1499cd10 2024-04-02T14:47:15.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.737+0000 7f14bb400640 1 -- 172.21.15.90:0/2757110415 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f14b4034440 con 0x559a1499cd10 2024-04-02T14:47:15.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.738+0000 7f14bb400640 1 -- 172.21.15.90:0/2757110415 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f14b403fc90 con 0x559a1499cd10 2024-04-02T14:47:15.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.738+0000 7f14d73c4740 10 client.18231 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:47:15.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.738+0000 7f14d73c4740 10 client.18231 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-04-02T14:47:15.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.738+0000 7f14d73c4740 20 client.18231 populate_metadata read hostname 'smithi090' 2024-04-02T14:47:15.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.738+0000 7f14d73c4740 1 --2- 172.21.15.90:0/2757110415 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x559a14a1ccd0 0x559a14a1f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:15.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.738+0000 7f14d73c4740 4 client.18231 mds_command: new command op to 24547 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-02T14:47:15.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.738+0000 7f14d73c4740 1 -- 172.21.15.90:0/2757110415 --> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] -- command(tid 0: {"prefix": "damage ls"}) -- 0x559a1490ec30 con 0x559a14a1ccd0 2024-04-02T14:47:15.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.738+0000 7f14d1200640 1 --2- 172.21.15.90:0/2757110415 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x559a14a1ccd0 0x559a14a1f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:15.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.739+0000 7f14d1200640 1 --2- 172.21.15.90:0/2757110415 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x559a14a1ccd0 0x559a14a1f0c0 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=f8c198dff934a20c server_cookie=9363b3a59fc20111 in_seq=0 out_seq=0 2024-04-02T14:47:15.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.739+0000 7f14bb400640 10 client.18231 ms_handle_connect on v2:172.21.15.177:6838/664895663 2024-04-02T14:47:15.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.739+0000 7f14bb400640 1 -- 172.21.15.90:0/2757110415 <== mds.2 v2:172.21.15.177:6838/664895663 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x559a1490ec30 con 0x559a14a1ccd0 2024-04-02T14:47:15.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.739+0000 7f14bb400640 10 client.18231 handle_command_reply: tid=0 2024-04-02T14:47:15.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.739+0000 7f14d73c4740 1 client.18231 shutdown 2024-04-02T14:47:15.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.739+0000 7f14d73c4740 1 -- 172.21.15.90:0/2757110415 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559a1499cd10 msgr2=0x559a1499feb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:15.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.739+0000 7f14d73c4740 1 --2- 172.21.15.90:0/2757110415 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559a1499cd10 0x559a1499feb0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f14b402c220 tx=0x7f14b40349a0 comp rx=0 tx=0).stop 2024-04-02T14:47:15.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.739+0000 7f14d73c4740 1 -- 172.21.15.90:0/2757110415 shutdown_connections 2024-04-02T14:47:15.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.739+0000 7f14d73c4740 1 --2- 172.21.15.90:0/2757110415 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x559a14a1ccd0 0x559a14a1f0c0 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:15.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.739+0000 7f14d73c4740 1 --2- 172.21.15.90:0/2757110415 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559a149a6080 0x559a149a0930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:15.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.739+0000 7f14d73c4740 1 --2- 172.21.15.90:0/2757110415 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559a1499d9f0 0x559a149a03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:15.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.739+0000 7f14d73c4740 1 --2- 172.21.15.90:0/2757110415 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559a1499cd10 0x559a1499feb0 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:15.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.739+0000 7f14d73c4740 1 -- 172.21.15.90:0/2757110415 >> 172.21.15.90:0/2757110415 conn(0x559a14963a50 msgr2=0x559a14996010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:47:15.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.739+0000 7f14bb400640 0 client.18231 ms_handle_reset on v2:172.21.15.177:6838/664895663 2024-04-02T14:47:15.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.739+0000 7f14d73c4740 1 -- 172.21.15.90:0/2757110415 shutdown_connections 2024-04-02T14:47:15.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.739+0000 7f14d73c4740 1 -- 172.21.15.90:0/2757110415 wait complete. 2024-04-02T14:47:15.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.740+0000 7f14d73c4740 20 client.18231 trim_cache size 0 max 16384 2024-04-02T14:47:15.736 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-02T14:47:15.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.740+0000 7f14d4600640 1 -- 172.21.15.90:0/2998190289 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f14b00a3da0 msgr2=0x7f14b00a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:15.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.740+0000 7f14d4600640 1 --2- 172.21.15.90:0/2998190289 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f14b00a3da0 0x7f14b00a6260 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f14c4004040 tx=0x7f14c40046c0 comp rx=0 tx=0).stop 2024-04-02T14:47:15.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.740+0000 7f14d4600640 1 -- 172.21.15.90:0/2998190289 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f14cc0747f0 msgr2=0x7f14cc19ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:15.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.740+0000 7f14d4600640 1 --2- 172.21.15.90:0/2998190289 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f14cc0747f0 0x7f14cc19ab70 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f14bc00eb90 tx=0x7f14bc00ef90 comp rx=0 tx=0).stop 2024-04-02T14:47:15.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.740+0000 7f14d4600640 1 -- 172.21.15.90:0/2998190289 shutdown_connections 2024-04-02T14:47:15.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.740+0000 7f14d4600640 1 --2- 172.21.15.90:0/2998190289 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f14b00a3da0 0x7f14b00a6260 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:15.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.740+0000 7f14d4600640 1 --2- 172.21.15.90:0/2998190289 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f14cc10e330 0x7f14cc19eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:15.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.740+0000 7f14d4600640 1 --2- 172.21.15.90:0/2998190289 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f14cc0747f0 0x7f14cc19ab70 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:15.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.740+0000 7f14d4600640 1 --2- 172.21.15.90:0/2998190289 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f14cc073ed0 0x7f14cc19a630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:15.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.740+0000 7f14d4600640 1 -- 172.21.15.90:0/2998190289 >> 172.21.15.90:0/2998190289 conn(0x7f14cc0fd560 msgr2=0x7f14cc0fe2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:47:15.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.741+0000 7f14d4600640 1 -- 172.21.15.90:0/2998190289 shutdown_connections 2024-04-02T14:47:15.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:15.741+0000 7f14d4600640 1 -- 172.21.15.90:0/2998190289 wait complete. 2024-04-02T14:47:15.763 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-02T14:47:16.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.028+0000 7fe5bba00640 1 -- 172.21.15.90:0/2147136690 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe5b4108ce0 msgr2=0x7fe5b41090c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:16.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.028+0000 7fe5bba00640 1 --2- 172.21.15.90:0/2147136690 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe5b4108ce0 0x7fe5b41090c0 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7fe5ac009ca0 tx=0x7fe5ac02c120 comp rx=0 tx=0).stop 2024-04-02T14:47:16.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.028+0000 7fe5bba00640 1 -- 172.21.15.90:0/2147136690 shutdown_connections 2024-04-02T14:47:16.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.028+0000 7fe5bba00640 1 --2- 172.21.15.90:0/2147136690 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe5b4114c50 0x7fe5b4117040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:16.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.028+0000 7fe5bba00640 1 --2- 172.21.15.90:0/2147136690 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe5b4109600 0x7fe5b4114710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:16.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.028+0000 7fe5bba00640 1 --2- 172.21.15.90:0/2147136690 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe5b4108ce0 0x7fe5b41090c0 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:16.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.028+0000 7fe5bba00640 1 -- 172.21.15.90:0/2147136690 >> 172.21.15.90:0/2147136690 conn(0x7fe5b406cbe0 msgr2=0x7fe5b406cff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:47:16.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.029+0000 7fe5bba00640 1 -- 172.21.15.90:0/2147136690 shutdown_connections 2024-04-02T14:47:16.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.030+0000 7fe5bba00640 1 -- 172.21.15.90:0/2147136690 wait complete. 2024-04-02T14:47:16.025 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.031+0000 7fe5bba00640 1 Processor -- start 2024-04-02T14:47:16.025 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.031+0000 7fe5bba00640 1 -- start start 2024-04-02T14:47:16.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.032+0000 7fe5bba00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe5b4108ce0 0x7fe5b41a68f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:16.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.032+0000 7fe5bba00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe5b4109600 0x7fe5b41a6e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:16.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.033+0000 7fe5bba00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe5b4114c50 0x7fe5b41a0aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:16.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.033+0000 7fe5bba00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fe5b411a720 con 0x7fe5b4114c50 2024-04-02T14:47:16.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.033+0000 7fe5bba00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fe5b411a5a0 con 0x7fe5b4108ce0 2024-04-02T14:47:16.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.033+0000 7fe5bba00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fe5b411a8a0 con 0x7fe5b4109600 2024-04-02T14:47:16.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.033+0000 7fe5b3e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe5b4109600 0x7fe5b41a6e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:16.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.033+0000 7fe5b9a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe5b4114c50 0x7fe5b41a0aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:16.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.034+0000 7fe5b9a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe5b4114c50 0x7fe5b41a0aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:37074/0 (socket says 172.21.15.90:37074) 2024-04-02T14:47:16.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.033+0000 7fe5b9000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe5b4108ce0 0x7fe5b41a68f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:16.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.034+0000 7fe5b9000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe5b4108ce0 0x7fe5b41a68f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:53186/0 (socket says 172.21.15.90:53186) 2024-04-02T14:47:16.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.034+0000 7fe5b3e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe5b4109600 0x7fe5b41a6e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:59694/0 (socket says 172.21.15.90:59694) 2024-04-02T14:47:16.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.034+0000 7fe5b9000640 1 -- 172.21.15.90:0/1907339052 learned_addr learned my addr 172.21.15.90:0/1907339052 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:47:16.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.035+0000 7fe5b9000640 1 -- 172.21.15.90:0/1907339052 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe5b4109600 msgr2=0x7fe5b41a6e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:16.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.035+0000 7fe5b9000640 1 --2- 172.21.15.90:0/1907339052 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe5b4109600 0x7fe5b41a6e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:16.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.035+0000 7fe5b9000640 1 -- 172.21.15.90:0/1907339052 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe5b4114c50 msgr2=0x7fe5b41a0aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:16.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.035+0000 7fe5b9000640 1 --2- 172.21.15.90:0/1907339052 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe5b4114c50 0x7fe5b41a0aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:16.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.035+0000 7fe5b9000640 1 -- 172.21.15.90:0/1907339052 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5b41a1220 con 0x7fe5b4108ce0 2024-04-02T14:47:16.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.035+0000 7fe5b9a00640 1 --2- 172.21.15.90:0/1907339052 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe5b4114c50 0x7fe5b41a0aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:47:16.032 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.038+0000 7fe5b9000640 1 --2- 172.21.15.90:0/1907339052 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe5b4108ce0 0x7fe5b41a68f0 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7fe5ac02c6b0 tx=0x7fe5ac035d10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:47:16.033 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.038+0000 7fe5b1600640 1 -- 172.21.15.90:0/1907339052 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5ac039040 con 0x7fe5b4108ce0 2024-04-02T14:47:16.033 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.038+0000 7fe5bba00640 1 -- 172.21.15.90:0/1907339052 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe5b41a14b0 con 0x7fe5b4108ce0 2024-04-02T14:47:16.033 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.039+0000 7fe5bba00640 1 -- 172.21.15.90:0/1907339052 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5b41ad7b0 con 0x7fe5b4108ce0 2024-04-02T14:47:16.033 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.039+0000 7fe5b1600640 1 -- 172.21.15.90:0/1907339052 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe5ac044070 con 0x7fe5b4108ce0 2024-04-02T14:47:16.033 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.039+0000 7fe5b1600640 1 -- 172.21.15.90:0/1907339052 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5ac0042a0 con 0x7fe5b4108ce0 2024-04-02T14:47:16.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.045+0000 7fe5b1600640 1 -- 172.21.15.90:0/1907339052 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fe5ac004440 con 0x7fe5b4108ce0 2024-04-02T14:47:16.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.046+0000 7fe5be7e3740 1 Processor -- start 2024-04-02T14:47:16.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.047+0000 7fe5b1600640 1 --2- 172.21.15.90:0/1907339052 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fe5980a3de0 0x7fe5980a62a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:16.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.047+0000 7fe5b1600640 1 -- 172.21.15.90:0/1907339052 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 8550+0+0 (secure 0 0 0) 0x7fe5ac0f2910 con 0x7fe5b4108ce0 2024-04-02T14:47:16.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.048+0000 7fe5b9a00640 1 --2- 172.21.15.90:0/1907339052 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fe5980a3de0 0x7fe5980a62a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:16.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.049+0000 7fe5be7e3740 1 -- start start 2024-04-02T14:47:16.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.049+0000 7fe5b9a00640 1 --2- 172.21.15.90:0/1907339052 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fe5980a3de0 0x7fe5980a62a0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fe5b406af20 tx=0x7fe5a000b2e0 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:47:16.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.050+0000 7fe5be7e3740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557a26c409f0 0x557a26c3f740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:16.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.050+0000 7fe5be7e3740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557a26c3fd10 0x557a26c40190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:16.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.050+0000 7fe5be7e3740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557a26c51bf0 0x557a26c53fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:16.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.050+0000 7fe5be7e3740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x557a26b80400 con 0x557a26c409f0 2024-04-02T14:47:16.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.050+0000 7fe5be7e3740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x557a26b64f10 con 0x557a26c51bf0 2024-04-02T14:47:16.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.050+0000 7fe5be7e3740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x557a26ba46f0 con 0x557a26c3fd10 2024-04-02T14:47:16.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.050+0000 7fe5b3e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557a26c3fd10 0x557a26c40190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:16.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.050+0000 7fe5b9a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557a26c51bf0 0x557a26c53fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:16.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.050+0000 7fe5b3e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557a26c3fd10 0x557a26c40190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:59698/0 (socket says 172.21.15.90:59698) 2024-04-02T14:47:16.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.050+0000 7fe5b3e00640 1 -- 172.21.15.90:0/1552070770 learned_addr learned my addr 172.21.15.90:0/1552070770 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:47:16.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.050+0000 7fe5b9a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557a26c51bf0 0x557a26c53fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:53202/0 (socket says 172.21.15.90:53202) 2024-04-02T14:47:16.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.051+0000 7fe5b9000640 1 --2- 172.21.15.90:0/1552070770 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557a26c409f0 0x557a26c3f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:16.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.051+0000 7fe5b9a00640 1 -- 172.21.15.90:0/1552070770 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557a26c3fd10 msgr2=0x557a26c40190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:47:16.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.051+0000 7fe5b9a00640 1 --2- 172.21.15.90:0/1552070770 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557a26c3fd10 0x557a26c40190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:16.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.051+0000 7fe5b9a00640 1 -- 172.21.15.90:0/1552070770 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557a26c409f0 msgr2=0x557a26c3f740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:47:16.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.051+0000 7fe5b9a00640 1 --2- 172.21.15.90:0/1552070770 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557a26c409f0 0x557a26c3f740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:16.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.051+0000 7fe5b9a00640 1 -- 172.21.15.90:0/1552070770 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557a268b83f0 con 0x557a26c51bf0 2024-04-02T14:47:16.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.051+0000 7fe5b9a00640 1 --2- 172.21.15.90:0/1552070770 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557a26c51bf0 0x557a26c53fe0 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7fe5a0010c30 tx=0x7fe5a0013000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=258ee34af99dbe81 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:47:16.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.052+0000 7fe5aa000640 1 -- 172.21.15.90:0/1552070770 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5a001c700 con 0x557a26c51bf0 2024-04-02T14:47:16.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.052+0000 7fe5aa000640 1 -- 172.21.15.90:0/1552070770 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe5a001cd80 con 0x557a26c51bf0 2024-04-02T14:47:16.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.052+0000 7fe5aa000640 1 -- 172.21.15.90:0/1552070770 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5a001e420 con 0x557a26c51bf0 2024-04-02T14:47:16.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.054+0000 7fe5be7e3740 1 -- 172.21.15.90:0/1552070770 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557a26c51bf0 msgr2=0x557a26c53fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:16.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.054+0000 7fe5be7e3740 1 --2- 172.21.15.90:0/1552070770 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557a26c51bf0 0x557a26c53fe0 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7fe5a0010c30 tx=0x7fe5a0013000 comp rx=0 tx=0).stop 2024-04-02T14:47:16.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.055+0000 7fe5be7e3740 1 -- 172.21.15.90:0/1552070770 shutdown_connections 2024-04-02T14:47:16.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.055+0000 7fe5be7e3740 1 --2- 172.21.15.90:0/1552070770 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557a26c51bf0 0x557a26c53fe0 unknown :-1 s=CLOSED pgs=868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:16.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.055+0000 7fe5be7e3740 1 --2- 172.21.15.90:0/1552070770 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557a26c3fd10 0x557a26c40190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:16.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.055+0000 7fe5be7e3740 1 --2- 172.21.15.90:0/1552070770 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557a26c409f0 0x557a26c3f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:16.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.055+0000 7fe5be7e3740 1 -- 172.21.15.90:0/1552070770 >> 172.21.15.90:0/1552070770 conn(0x557a26c06a50 msgr2=0x557a26c3a760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:47:16.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.056+0000 7fe5be7e3740 1 -- 172.21.15.90:0/1552070770 shutdown_connections 2024-04-02T14:47:16.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.056+0000 7fe5be7e3740 1 -- 172.21.15.90:0/1552070770 wait complete. 2024-04-02T14:47:16.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.057+0000 7fe5be7e3740 1 -- start start 2024-04-02T14:47:16.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.057+0000 7fe5be7e3740 1 Processor -- start 2024-04-02T14:47:16.052 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.058+0000 7fe5be7e3740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557a26c3fd10 0x557a26c46410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:16.052 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.058+0000 7fe5be7e3740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557a26c409f0 0x557a26c46950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:16.052 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.058+0000 7fe5be7e3740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557a26c51bf0 0x557a26c46e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:16.052 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.058+0000 7fe5be7e3740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x557a26ba06a0 con 0x557a26c409f0 2024-04-02T14:47:16.052 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.058+0000 7fe5be7e3740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x557a26b4c980 con 0x557a26c3fd10 2024-04-02T14:47:16.052 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.058+0000 7fe5be7e3740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x557a26b5fcb0 con 0x557a26c51bf0 2024-04-02T14:47:16.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.058+0000 7fe5be7e3740 10 client.0 _ll_register_callbacks cb 0x557a26b5f6a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:47:16.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.058+0000 7fe5b9000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557a26c3fd10 0x557a26c46410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:16.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.058+0000 7fe5b9000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557a26c3fd10 0x557a26c46410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:53206/0 (socket says 172.21.15.90:53206) 2024-04-02T14:47:16.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.058+0000 7fe5b9000640 1 -- 172.21.15.90:0/1555063066 learned_addr learned my addr 172.21.15.90:0/1555063066 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:47:16.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.059+0000 7fe5b9a00640 1 --2- 172.21.15.90:0/1555063066 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557a26c51bf0 0x557a26c46e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:16.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.059+0000 7fe5b3e00640 1 --2- 172.21.15.90:0/1555063066 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557a26c409f0 0x557a26c46950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:16.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.059+0000 7fe5b9a00640 1 -- 172.21.15.90:0/1555063066 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557a26c3fd10 msgr2=0x557a26c46410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:47:16.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.059+0000 7fe5b9a00640 1 --2- 172.21.15.90:0/1555063066 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557a26c3fd10 0x557a26c46410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:16.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.059+0000 7fe5b9a00640 1 -- 172.21.15.90:0/1555063066 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557a26c409f0 msgr2=0x557a26c46950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:47:16.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.059+0000 7fe5b9a00640 1 --2- 172.21.15.90:0/1555063066 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557a26c409f0 0x557a26c46950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:16.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.059+0000 7fe5b9a00640 1 -- 172.21.15.90:0/1555063066 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557a269827d0 con 0x557a26c51bf0 2024-04-02T14:47:16.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.059+0000 7fe5b3e00640 1 --2- 172.21.15.90:0/1555063066 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557a26c409f0 0x557a26c46950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:47:16.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.060+0000 7fe5b9a00640 1 --2- 172.21.15.90:0/1555063066 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557a26c51bf0 0x557a26c46e90 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7fe5a0011fd0 tx=0x7fe5a00276b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fe3e66a8ec95fec7 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:47:16.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.060+0000 7fe59ea00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:47:16.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.060+0000 7fe59ea00640 1 -- 172.21.15.90:0/1555063066 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5a0036060 con 0x557a26c51bf0 2024-04-02T14:47:16.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.060+0000 7fe59ea00640 1 -- 172.21.15.90:0/1555063066 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 8550+0+0 (secure 0 0 0) 0x7fe5a0004250 con 0x557a26c51bf0 2024-04-02T14:47:16.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.060+0000 7fe5be7e3740 1 -- 172.21.15.90:0/1555063066 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe5a001cd80 con 0x557a26c51bf0 2024-04-02T14:47:16.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.061+0000 7fe59ea00640 1 -- 172.21.15.90:0/1555063066 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fe5a0004450 con 0x557a26c51bf0 2024-04-02T14:47:16.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.061+0000 7fe5be7e3740 10 client.29122 fetch_fsmap learned FSMap version 27 2024-04-02T14:47:16.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.061+0000 7fe5be7e3740 1 -- 172.21.15.90:0/1555063066 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x557a26c607c0 con 0x557a26c51bf0 2024-04-02T14:47:16.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.061+0000 7fe59ea00640 1 -- 172.21.15.90:0/1555063066 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe5a001cd80 con 0x557a26c51bf0 2024-04-02T14:47:16.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.061+0000 7fe59ea00640 1 -- 172.21.15.90:0/1555063066 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fe5a0003900 con 0x557a26c51bf0 2024-04-02T14:47:16.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.061+0000 7fe5be7e3740 10 client.29122 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:47:16.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.061+0000 7fe5be7e3740 10 client.29122 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2024-04-02T14:47:16.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.061+0000 7fe5be7e3740 20 client.29122 populate_metadata read hostname 'smithi090' 2024-04-02T14:47:16.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.061+0000 7fe5be7e3740 1 --2- 172.21.15.90:0/1555063066 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x557a26c61c20 0x557a26c64010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:16.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.061+0000 7fe5be7e3740 4 client.29122 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:47:16.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.061+0000 7fe5be7e3740 1 -- 172.21.15.90:0/1555063066 --> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557a26b64040 con 0x557a26c61c20 2024-04-02T14:47:16.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.062+0000 7fe5b9000640 1 --2- 172.21.15.90:0/1555063066 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x557a26c61c20 0x557a26c64010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:16.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.063+0000 7fe5b9000640 1 --2- 172.21.15.90:0/1555063066 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x557a26c61c20 0x557a26c64010 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=66dbf9e04c758c80 server_cookie=c3b96d52768e2a9a in_seq=0 out_seq=0 2024-04-02T14:47:16.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.063+0000 7fe59ea00640 10 client.29122 ms_handle_connect on v2:172.21.15.90:6838/1498915870 2024-04-02T14:47:16.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.064+0000 7fe59ea00640 1 -- 172.21.15.90:0/1555063066 <== mds.1 v2:172.21.15.90:6838/1498915870 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x557a26b64040 con 0x557a26c61c20 2024-04-02T14:47:16.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.064+0000 7fe59ea00640 10 client.29122 handle_command_reply: tid=0 2024-04-02T14:47:16.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.064+0000 7fe5be7e3740 1 client.29122 shutdown 2024-04-02T14:47:16.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.065+0000 7fe5be7e3740 1 -- 172.21.15.90:0/1555063066 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557a26c51bf0 msgr2=0x557a26c46e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:16.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.065+0000 7fe5be7e3740 1 --2- 172.21.15.90:0/1555063066 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557a26c51bf0 0x557a26c46e90 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7fe5a0011fd0 tx=0x7fe5a00276b0 comp rx=0 tx=0).stop 2024-04-02T14:47:16.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.065+0000 7fe5be7e3740 1 -- 172.21.15.90:0/1555063066 shutdown_connections 2024-04-02T14:47:16.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.065+0000 7fe5be7e3740 1 --2- 172.21.15.90:0/1555063066 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x557a26c61c20 0x557a26c64010 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:16.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.065+0000 7fe5be7e3740 1 --2- 172.21.15.90:0/1555063066 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557a26c51bf0 0x557a26c46e90 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:16.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.065+0000 7fe5be7e3740 1 --2- 172.21.15.90:0/1555063066 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557a26c409f0 0x557a26c46950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:16.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.065+0000 7fe5be7e3740 1 --2- 172.21.15.90:0/1555063066 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557a26c3fd10 0x557a26c46410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:16.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.065+0000 7fe5be7e3740 1 -- 172.21.15.90:0/1555063066 >> 172.21.15.90:0/1555063066 conn(0x557a26c06a50 msgr2=0x557a26c3c4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:47:16.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.065+0000 7fe59ea00640 0 client.29122 ms_handle_reset on v2:172.21.15.90:6838/1498915870 2024-04-02T14:47:16.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.066+0000 7fe5be7e3740 1 -- 172.21.15.90:0/1555063066 shutdown_connections 2024-04-02T14:47:16.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.066+0000 7fe5be7e3740 1 -- 172.21.15.90:0/1555063066 wait complete. 2024-04-02T14:47:16.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.066+0000 7fe5be7e3740 20 client.29122 trim_cache size 0 max 16384 2024-04-02T14:47:16.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.096+0000 7fe5be7e3740 1 Processor -- start 2024-04-02T14:47:16.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.096+0000 7fe5be7e3740 1 -- start start 2024-04-02T14:47:16.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.096+0000 7fe5be7e3740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557a26c3fd10 0x557a26c48b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:16.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.096+0000 7fe5be7e3740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557a26c409f0 0x557a26c46cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:16.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.096+0000 7fe5be7e3740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557a26c51bf0 0x557a26c39720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:16.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.097+0000 7fe5be7e3740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fe59400e7e0 con 0x557a26c409f0 2024-04-02T14:47:16.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.097+0000 7fe5be7e3740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fe594010320 con 0x557a26c3fd10 2024-04-02T14:47:16.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.097+0000 7fe5be7e3740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fe59400eeb0 con 0x557a26c51bf0 2024-04-02T14:47:16.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.097+0000 7fe5b9a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557a26c51bf0 0x557a26c39720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:16.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.097+0000 7fe5b9a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557a26c51bf0 0x557a26c39720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:59708/0 (socket says 172.21.15.90:59708) 2024-04-02T14:47:16.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.097+0000 7fe5b9a00640 1 -- 172.21.15.90:0/3188786202 learned_addr learned my addr 172.21.15.90:0/3188786202 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:47:16.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.097+0000 7fe5b3e00640 1 --2- 172.21.15.90:0/3188786202 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557a26c409f0 0x557a26c46cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:16.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.097+0000 7fe5b9000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557a26c3fd10 0x557a26c48b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:16.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.097+0000 7fe5b9a00640 1 -- 172.21.15.90:0/3188786202 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557a26c3fd10 msgr2=0x557a26c48b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:47:16.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.097+0000 7fe5b9a00640 1 --2- 172.21.15.90:0/3188786202 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557a26c3fd10 0x557a26c48b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:16.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.097+0000 7fe5b9a00640 1 -- 172.21.15.90:0/3188786202 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557a26c409f0 msgr2=0x557a26c46cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:47:16.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.097+0000 7fe5b9a00640 1 --2- 172.21.15.90:0/3188786202 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557a26c409f0 0x557a26c46cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:16.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.097+0000 7fe5b9a00640 1 -- 172.21.15.90:0/3188786202 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557a26c3ed40 con 0x557a26c51bf0 2024-04-02T14:47:16.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.098+0000 7fe5b9a00640 1 --2- 172.21.15.90:0/3188786202 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557a26c51bf0 0x557a26c39720 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7fe5a0011f80 tx=0x7fe5a0005c50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1937601d20313c8c server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:47:16.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.098+0000 7fe5a9600640 1 -- 172.21.15.90:0/3188786202 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5a0036030 con 0x557a26c51bf0 2024-04-02T14:47:16.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.098+0000 7fe5a9600640 1 -- 172.21.15.90:0/3188786202 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe5a0004250 con 0x557a26c51bf0 2024-04-02T14:47:16.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.098+0000 7fe5a9600640 1 -- 172.21.15.90:0/3188786202 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5a003a750 con 0x557a26c51bf0 2024-04-02T14:47:16.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.099+0000 7fe5be7e3740 1 -- 172.21.15.90:0/3188786202 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557a26c51bf0 msgr2=0x557a26c39720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:16.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.099+0000 7fe5be7e3740 1 --2- 172.21.15.90:0/3188786202 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557a26c51bf0 0x557a26c39720 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7fe5a0011f80 tx=0x7fe5a0005c50 comp rx=0 tx=0).stop 2024-04-02T14:47:16.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.099+0000 7fe5be7e3740 1 -- 172.21.15.90:0/3188786202 shutdown_connections 2024-04-02T14:47:16.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.099+0000 7fe5be7e3740 1 --2- 172.21.15.90:0/3188786202 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557a26c51bf0 0x557a26c39720 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:16.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.099+0000 7fe5be7e3740 1 --2- 172.21.15.90:0/3188786202 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557a26c409f0 0x557a26c46cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:16.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.099+0000 7fe5be7e3740 1 --2- 172.21.15.90:0/3188786202 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557a26c3fd10 0x557a26c48b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:16.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.099+0000 7fe5be7e3740 1 -- 172.21.15.90:0/3188786202 >> 172.21.15.90:0/3188786202 conn(0x557a26c06a50 msgr2=0x557a26c39010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:47:16.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.099+0000 7fe5be7e3740 1 -- 172.21.15.90:0/3188786202 shutdown_connections 2024-04-02T14:47:16.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.099+0000 7fe5be7e3740 1 -- 172.21.15.90:0/3188786202 wait complete. 2024-04-02T14:47:16.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.099+0000 7fe5be7e3740 1 -- start start 2024-04-02T14:47:16.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.099+0000 7fe5be7e3740 1 Processor -- start 2024-04-02T14:47:16.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.100+0000 7fe5be7e3740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557a26c3fd10 0x557a26c598c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:16.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.100+0000 7fe5be7e3740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557a26c409f0 0x557a26c59e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:16.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.100+0000 7fe5be7e3740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557a26c51bf0 0x557a26c56890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:16.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.100+0000 7fe5be7e3740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fe59400fc50 con 0x557a26c3fd10 2024-04-02T14:47:16.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.100+0000 7fe5be7e3740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fe59400f580 con 0x557a26c51bf0 2024-04-02T14:47:16.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.100+0000 7fe5be7e3740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fe59400c7e0 con 0x557a26c409f0 2024-04-02T14:47:16.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.100+0000 7fe5be7e3740 10 client.0 _ll_register_callbacks cb 0x7fe5a0011440 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:47:16.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.100+0000 7fe5b9a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557a26c51bf0 0x557a26c56890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:16.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.100+0000 7fe5b3e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557a26c409f0 0x557a26c59e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:16.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.100+0000 7fe5b9a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557a26c51bf0 0x557a26c56890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:53216/0 (socket says 172.21.15.90:53216) 2024-04-02T14:47:16.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.100+0000 7fe5b9a00640 1 -- 172.21.15.90:0/658140061 learned_addr learned my addr 172.21.15.90:0/658140061 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:47:16.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.100+0000 7fe5b9000640 1 --2- 172.21.15.90:0/658140061 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557a26c3fd10 0x557a26c598c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:16.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.101+0000 7fe5b3e00640 1 -- 172.21.15.90:0/658140061 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557a26c51bf0 msgr2=0x557a26c56890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:47:16.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.101+0000 7fe5b3e00640 1 --2- 172.21.15.90:0/658140061 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557a26c51bf0 0x557a26c56890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:16.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.101+0000 7fe5b3e00640 1 -- 172.21.15.90:0/658140061 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557a26c3fd10 msgr2=0x557a26c598c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:47:16.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.101+0000 7fe5b3e00640 1 --2- 172.21.15.90:0/658140061 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557a26c3fd10 0x557a26c598c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:16.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.101+0000 7fe5b3e00640 1 -- 172.21.15.90:0/658140061 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557a26c56ed0 con 0x557a26c409f0 2024-04-02T14:47:16.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.102+0000 7fe5b3e00640 1 --2- 172.21.15.90:0/658140061 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557a26c409f0 0x557a26c59e00 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x557a26c45c70 tx=0x7fe5a40167a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=46ee3a136896137e server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:47:16.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.102+0000 7fe59ea00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:47:16.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.103+0000 7fe59ea00640 1 -- 172.21.15.90:0/658140061 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5a4017440 con 0x557a26c409f0 2024-04-02T14:47:16.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.103+0000 7fe59ea00640 1 -- 172.21.15.90:0/658140061 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 8550+0+0 (secure 0 0 0) 0x7fe5a40175e0 con 0x557a26c409f0 2024-04-02T14:47:16.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.103+0000 7fe5be7e3740 1 -- 172.21.15.90:0/658140061 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe5a0004250 con 0x557a26c409f0 2024-04-02T14:47:16.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.103+0000 7fe59ea00640 1 -- 172.21.15.90:0/658140061 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fe5a40177e0 con 0x557a26c409f0 2024-04-02T14:47:16.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.103+0000 7fe5be7e3740 10 client.29128 fetch_fsmap learned FSMap version 27 2024-04-02T14:47:16.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.103+0000 7fe5be7e3740 1 -- 172.21.15.90:0/658140061 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x557a26cc8800 con 0x557a26c409f0 2024-04-02T14:47:16.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.104+0000 7fe59ea00640 1 -- 172.21.15.90:0/658140061 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe5a0004250 con 0x557a26c409f0 2024-04-02T14:47:16.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.104+0000 7fe59ea00640 1 -- 172.21.15.90:0/658140061 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fe5a4020bb0 con 0x557a26c409f0 2024-04-02T14:47:16.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.104+0000 7fe5be7e3740 10 client.29128 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:47:16.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.104+0000 7fe5be7e3740 10 client.29128 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2024-04-02T14:47:16.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.104+0000 7fe5be7e3740 20 client.29128 populate_metadata read hostname 'smithi090' 2024-04-02T14:47:16.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.104+0000 7fe5be7e3740 1 --2- 172.21.15.90:0/658140061 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x557a26cc8fd0 0x557a26ccb3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:16.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.104+0000 7fe5be7e3740 4 client.29128 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-02T14:47:16.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.104+0000 7fe5be7e3740 1 -- 172.21.15.90:0/658140061 --> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] -- command(tid 0: {"prefix": "damage ls"}) -- 0x557a26bb1c30 con 0x557a26cc8fd0 2024-04-02T14:47:16.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.104+0000 7fe5b9000640 1 --2- 172.21.15.90:0/658140061 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x557a26cc8fd0 0x557a26ccb3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:16.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.105+0000 7fe5b9000640 1 --2- 172.21.15.90:0/658140061 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x557a26cc8fd0 0x557a26ccb3f0 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=dd51b33c4859387f server_cookie=5591a9ba45f8a96 in_seq=0 out_seq=0 2024-04-02T14:47:16.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.105+0000 7fe59ea00640 10 client.29128 ms_handle_connect on v2:172.21.15.90:6838/1498915870 2024-04-02T14:47:16.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.105+0000 7fe59ea00640 1 -- 172.21.15.90:0/658140061 <== mds.1 v2:172.21.15.90:6838/1498915870 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x557a26bb1c30 con 0x557a26cc8fd0 2024-04-02T14:47:16.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.105+0000 7fe59ea00640 10 client.29128 handle_command_reply: tid=0 2024-04-02T14:47:16.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.105+0000 7fe5be7e3740 1 client.29128 shutdown 2024-04-02T14:47:16.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.105+0000 7fe5be7e3740 1 -- 172.21.15.90:0/658140061 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557a26c409f0 msgr2=0x557a26c59e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:16.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.105+0000 7fe5be7e3740 1 --2- 172.21.15.90:0/658140061 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557a26c409f0 0x557a26c59e00 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x557a26c45c70 tx=0x7fe5a40167a0 comp rx=0 tx=0).stop 2024-04-02T14:47:16.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.105+0000 7fe5be7e3740 1 -- 172.21.15.90:0/658140061 shutdown_connections 2024-04-02T14:47:16.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.105+0000 7fe5be7e3740 1 --2- 172.21.15.90:0/658140061 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x557a26cc8fd0 0x557a26ccb3f0 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:16.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.105+0000 7fe5be7e3740 1 --2- 172.21.15.90:0/658140061 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557a26c51bf0 0x557a26c56890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:16.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.105+0000 7fe5be7e3740 1 --2- 172.21.15.90:0/658140061 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557a26c409f0 0x557a26c59e00 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:16.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.105+0000 7fe5be7e3740 1 --2- 172.21.15.90:0/658140061 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557a26c3fd10 0x557a26c598c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:16.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.105+0000 7fe59ea00640 0 client.29128 ms_handle_reset on v2:172.21.15.90:6838/1498915870 2024-04-02T14:47:16.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.105+0000 7fe5be7e3740 1 -- 172.21.15.90:0/658140061 >> 172.21.15.90:0/658140061 conn(0x557a26c06a50 msgr2=0x557a26c398a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:47:16.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.106+0000 7fe5be7e3740 1 -- 172.21.15.90:0/658140061 shutdown_connections 2024-04-02T14:47:16.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.106+0000 7fe5be7e3740 1 -- 172.21.15.90:0/658140061 wait complete. 2024-04-02T14:47:16.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.106+0000 7fe5be7e3740 20 client.29128 trim_cache size 0 max 16384 2024-04-02T14:47:16.102 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-02T14:47:16.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.106+0000 7fe5bba00640 1 -- 172.21.15.90:0/1907339052 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fe5980a3de0 msgr2=0x7fe5980a62a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:16.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.106+0000 7fe5bba00640 1 --2- 172.21.15.90:0/1907339052 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fe5980a3de0 0x7fe5980a62a0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fe5b406af20 tx=0x7fe5a000b2e0 comp rx=0 tx=0).stop 2024-04-02T14:47:16.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.106+0000 7fe5bba00640 1 -- 172.21.15.90:0/1907339052 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe5b4108ce0 msgr2=0x7fe5b41a68f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:16.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.106+0000 7fe5bba00640 1 --2- 172.21.15.90:0/1907339052 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe5b4108ce0 0x7fe5b41a68f0 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7fe5ac02c6b0 tx=0x7fe5ac035d10 comp rx=0 tx=0).stop 2024-04-02T14:47:16.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.107+0000 7fe5bba00640 1 -- 172.21.15.90:0/1907339052 shutdown_connections 2024-04-02T14:47:16.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.107+0000 7fe5bba00640 1 --2- 172.21.15.90:0/1907339052 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fe5980a3de0 0x7fe5980a62a0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:16.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.107+0000 7fe5bba00640 1 --2- 172.21.15.90:0/1907339052 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe5b4114c50 0x7fe5b41a0aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:16.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.107+0000 7fe5bba00640 1 --2- 172.21.15.90:0/1907339052 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe5b4109600 0x7fe5b41a6e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:16.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.107+0000 7fe5bba00640 1 --2- 172.21.15.90:0/1907339052 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe5b4108ce0 0x7fe5b41a68f0 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:16.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.107+0000 7fe5bba00640 1 -- 172.21.15.90:0/1907339052 >> 172.21.15.90:0/1907339052 conn(0x7fe5b406cbe0 msgr2=0x7fe5b4070800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:47:16.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.107+0000 7fe5bba00640 1 -- 172.21.15.90:0/1907339052 shutdown_connections 2024-04-02T14:47:16.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.107+0000 7fe5bba00640 1 -- 172.21.15.90:0/1907339052 wait complete. 2024-04-02T14:47:16.120 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-02T14:47:16.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.363+0000 7f09d5400640 1 -- 172.21.15.90:0/4172092373 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f09d0069240 msgr2=0x7f09d00696a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:16.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.363+0000 7f09d5400640 1 --2- 172.21.15.90:0/4172092373 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f09d0069240 0x7f09d00696a0 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f09bc009890 tx=0x7f09bc02bb70 comp rx=0 tx=0).stop 2024-04-02T14:47:16.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.363+0000 7f09d5400640 1 -- 172.21.15.90:0/4172092373 shutdown_connections 2024-04-02T14:47:16.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.363+0000 7f09d5400640 1 --2- 172.21.15.90:0/4172092373 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f09d01045c0 0x7f09d0104a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:16.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.363+0000 7f09d5400640 1 --2- 172.21.15.90:0/4172092373 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f09d0069240 0x7f09d00696a0 unknown :-1 s=CLOSED pgs=869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:16.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.363+0000 7f09d5400640 1 --2- 172.21.15.90:0/4172092373 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f09d0106a50 0x7f09d0068d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:16.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.363+0000 7f09d5400640 1 -- 172.21.15.90:0/4172092373 >> 172.21.15.90:0/4172092373 conn(0x7f09d0077b00 msgr2=0x7f09d0077f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:47:16.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.363+0000 7f09d5400640 1 -- 172.21.15.90:0/4172092373 shutdown_connections 2024-04-02T14:47:16.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.363+0000 7f09d5400640 1 -- 172.21.15.90:0/4172092373 wait complete. 2024-04-02T14:47:16.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.364+0000 7f09d5400640 1 Processor -- start 2024-04-02T14:47:16.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.365+0000 7f09d5400640 1 -- start start 2024-04-02T14:47:16.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.365+0000 7f09d5400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f09d0069240 0x7f09d01a46a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:16.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.365+0000 7f09d5400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f09d01045c0 0x7f09d01a4be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:16.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.365+0000 7f09d5400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f09d0106a50 0x7f09d019e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:16.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.365+0000 7f09d5400640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f09d0118430 con 0x7f09d01045c0 2024-04-02T14:47:16.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.365+0000 7f09d5400640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f09d01182b0 con 0x7f09d0106a50 2024-04-02T14:47:16.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.365+0000 7f09d5400640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f09d01185b0 con 0x7f09d0069240 2024-04-02T14:47:16.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.365+0000 7f09cf200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f09d0106a50 0x7f09d019e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:16.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.365+0000 7f09cde00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f09d01045c0 0x7f09d01a4be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:16.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.365+0000 7f09cf200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f09d0106a50 0x7f09d019e7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:53224/0 (socket says 172.21.15.90:53224) 2024-04-02T14:47:16.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.365+0000 7f09cf200640 1 -- 172.21.15.90:0/3138095912 learned_addr learned my addr 172.21.15.90:0/3138095912 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:47:16.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.365+0000 7f09ce800640 1 --2- 172.21.15.90:0/3138095912 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f09d0069240 0x7f09d01a46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:16.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.366+0000 7f09cf200640 1 -- 172.21.15.90:0/3138095912 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f09d0069240 msgr2=0x7f09d01a46a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:16.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.366+0000 7f09cf200640 1 --2- 172.21.15.90:0/3138095912 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f09d0069240 0x7f09d01a46a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:16.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.366+0000 7f09cf200640 1 -- 172.21.15.90:0/3138095912 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f09d01045c0 msgr2=0x7f09d01a4be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:16.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.366+0000 7f09cf200640 1 --2- 172.21.15.90:0/3138095912 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f09d01045c0 0x7f09d01a4be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:16.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.366+0000 7f09cf200640 1 -- 172.21.15.90:0/3138095912 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09d019f070 con 0x7f09d0106a50 2024-04-02T14:47:16.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.366+0000 7f09cf200640 1 --2- 172.21.15.90:0/3138095912 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f09d0106a50 0x7f09d019e7b0 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f09c8009fd0 tx=0x7f09c800c740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:47:16.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.366+0000 7f09c7400640 1 -- 172.21.15.90:0/3138095912 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09c8004b20 con 0x7f09d0106a50 2024-04-02T14:47:16.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.366+0000 7f09c7400640 1 -- 172.21.15.90:0/3138095912 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f09c8003e50 con 0x7f09d0106a50 2024-04-02T14:47:16.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.366+0000 7f09c7400640 1 -- 172.21.15.90:0/3138095912 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09c800b570 con 0x7f09d0106a50 2024-04-02T14:47:16.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.366+0000 7f09d5400640 1 -- 172.21.15.90:0/3138095912 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09d019f360 con 0x7f09d0106a50 2024-04-02T14:47:16.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.366+0000 7f09d5400640 1 -- 172.21.15.90:0/3138095912 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09d01ab620 con 0x7f09d0106a50 2024-04-02T14:47:16.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.374+0000 7f09c7400640 1 -- 172.21.15.90:0/3138095912 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f09c801d020 con 0x7f09d0106a50 2024-04-02T14:47:16.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.377+0000 7f09d8235740 1 Processor -- start 2024-04-02T14:47:16.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.379+0000 7f09c7400640 1 --2- 172.21.15.90:0/3138095912 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f09b40a3d20 0x7f09b40a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:16.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.379+0000 7f09c7400640 1 -- 172.21.15.90:0/3138095912 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 8550+0+0 (secure 0 0 0) 0x7f09c80d07c0 con 0x7f09d0106a50 2024-04-02T14:47:16.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.380+0000 7f09cde00640 1 --2- 172.21.15.90:0/3138095912 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f09b40a3d20 0x7f09b40a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:16.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.380+0000 7f09d8235740 1 -- start start 2024-04-02T14:47:16.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.381+0000 7f09d8235740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d4ff6ef9f0 0x55d4ff6ee740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:16.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.381+0000 7f09d8235740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d4ff6eed10 0x55d4ff6ef190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:16.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.381+0000 7f09d8235740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d4ff6f8080 0x55d4ff6fa470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:16.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.382+0000 7f09d8235740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55d4ff62f400 con 0x55d4ff6ef9f0 2024-04-02T14:47:16.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.382+0000 7f09d8235740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55d4ff613f10 con 0x55d4ff6f8080 2024-04-02T14:47:16.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.382+0000 7f09d8235740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55d4ff6536f0 con 0x55d4ff6eed10 2024-04-02T14:47:16.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.384+0000 7f09cf200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d4ff6f8080 0x55d4ff6fa470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:16.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.384+0000 7f09cf200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d4ff6f8080 0x55d4ff6fa470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:53232/0 (socket says 172.21.15.90:53232) 2024-04-02T14:47:16.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.384+0000 7f09cf200640 1 -- 172.21.15.90:0/3054242812 learned_addr learned my addr 172.21.15.90:0/3054242812 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:47:16.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.385+0000 7f09cde00640 1 --2- 172.21.15.90:0/3054242812 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d4ff6eed10 0x55d4ff6ef190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:16.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.385+0000 7f09cf200640 1 -- 172.21.15.90:0/3054242812 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d4ff6eed10 msgr2=0x55d4ff6ef190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:47:16.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.385+0000 7f09cf200640 1 --2- 172.21.15.90:0/3054242812 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d4ff6eed10 0x55d4ff6ef190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:16.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.385+0000 7f09cf200640 1 -- 172.21.15.90:0/3054242812 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d4ff6ef9f0 msgr2=0x55d4ff6ee740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:47:16.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.385+0000 7f09cf200640 1 --2- 172.21.15.90:0/3054242812 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d4ff6ef9f0 0x55d4ff6ee740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:16.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.385+0000 7f09cf200640 1 -- 172.21.15.90:0/3054242812 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d4ff3673f0 con 0x55d4ff6f8080 2024-04-02T14:47:16.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.386+0000 7f09cf200640 1 --2- 172.21.15.90:0/3054242812 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d4ff6f8080 0x55d4ff6fa470 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f09c80a4e80 tx=0x7f09c809c340 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b71aaef59540fc79 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:47:16.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.386+0000 7f09bb400640 1 -- 172.21.15.90:0/3054242812 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09c809c740 con 0x55d4ff6f8080 2024-04-02T14:47:16.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.386+0000 7f09cde00640 1 --2- 172.21.15.90:0/3138095912 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f09b40a3d20 0x7f09b40a61e0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f09bc0097b0 tx=0x7f09bc0058d0 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:47:16.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.389+0000 7f09bb400640 1 -- 172.21.15.90:0/3054242812 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f09c80a0af0 con 0x55d4ff6f8080 2024-04-02T14:47:16.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.390+0000 7f09d8235740 1 -- 172.21.15.90:0/3054242812 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d4ff6f8080 msgr2=0x55d4ff6fa470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:16.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.390+0000 7f09d8235740 1 --2- 172.21.15.90:0/3054242812 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d4ff6f8080 0x55d4ff6fa470 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f09c80a4e80 tx=0x7f09c809c340 comp rx=0 tx=0).stop 2024-04-02T14:47:16.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.390+0000 7f09d8235740 1 -- 172.21.15.90:0/3054242812 shutdown_connections 2024-04-02T14:47:16.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.393+0000 7f09d8235740 1 --2- 172.21.15.90:0/3054242812 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d4ff6f8080 0x55d4ff6fa470 unknown :-1 s=CLOSED pgs=871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:16.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.393+0000 7f09d8235740 1 --2- 172.21.15.90:0/3054242812 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d4ff6eed10 0x55d4ff6ef190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:16.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.393+0000 7f09d8235740 1 --2- 172.21.15.90:0/3054242812 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d4ff6ef9f0 0x55d4ff6ee740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:16.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.393+0000 7f09d8235740 1 -- 172.21.15.90:0/3054242812 >> 172.21.15.90:0/3054242812 conn(0x55d4ff6b5a50 msgr2=0x55d4ff6e9760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:47:16.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.393+0000 7f09d8235740 1 -- 172.21.15.90:0/3054242812 shutdown_connections 2024-04-02T14:47:16.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.393+0000 7f09d8235740 1 -- 172.21.15.90:0/3054242812 wait complete. 2024-04-02T14:47:16.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.394+0000 7f09d8235740 1 -- start start 2024-04-02T14:47:16.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.394+0000 7f09d8235740 1 Processor -- start 2024-04-02T14:47:16.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.396+0000 7f09d8235740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d4ff6eed10 0x55d4ff6f4fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:16.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.396+0000 7f09d8235740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d4ff6ef9f0 0x55d4ff6f5510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:16.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.396+0000 7f09ce800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d4ff6eed10 0x55d4ff6f4fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:16.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.396+0000 7f09ce800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d4ff6eed10 0x55d4ff6f4fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:37138/0 (socket says 172.21.15.90:37138) 2024-04-02T14:47:16.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.396+0000 7f09cde00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d4ff6ef9f0 0x55d4ff6f5510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:16.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.396+0000 7f09cde00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d4ff6ef9f0 0x55d4ff6f5510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:53246/0 (socket says 172.21.15.90:53246) 2024-04-02T14:47:16.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.397+0000 7f09ce800640 1 -- 172.21.15.90:0/3246809859 learned_addr learned my addr 172.21.15.90:0/3246809859 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:47:16.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.398+0000 7f09d8235740 1 --2- 172.21.15.90:0/3246809859 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d4ff6f8080 0x55d4ff6f5a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:16.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.398+0000 7f09d8235740 1 -- 172.21.15.90:0/3246809859 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55d4ff64f6a0 con 0x55d4ff6eed10 2024-04-02T14:47:16.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.398+0000 7f09d8235740 1 -- 172.21.15.90:0/3246809859 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55d4ff5fb980 con 0x55d4ff6ef9f0 2024-04-02T14:47:16.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.398+0000 7f09d8235740 1 -- 172.21.15.90:0/3246809859 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55d4ff60ecb0 con 0x55d4ff6f8080 2024-04-02T14:47:16.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.398+0000 7f09d8235740 10 client.0 _ll_register_callbacks cb 0x55d4ff60e6a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:47:16.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.398+0000 7f09cde00640 1 -- 172.21.15.90:0/3246809859 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d4ff6f8080 msgr2=0x55d4ff6f5a50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-02T14:47:16.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.398+0000 7f09cde00640 1 --2- 172.21.15.90:0/3246809859 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d4ff6f8080 0x55d4ff6f5a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:16.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.398+0000 7f09cde00640 1 -- 172.21.15.90:0/3246809859 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d4ff6eed10 msgr2=0x55d4ff6f4fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:47:16.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.398+0000 7f09cde00640 1 --2- 172.21.15.90:0/3246809859 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d4ff6eed10 0x55d4ff6f4fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:16.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.398+0000 7f09cde00640 1 -- 172.21.15.90:0/3246809859 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d4ff4317d0 con 0x55d4ff6ef9f0 2024-04-02T14:47:16.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.400+0000 7f09cde00640 1 --2- 172.21.15.90:0/3246809859 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d4ff6ef9f0 0x55d4ff6f5510 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f09bc02c180 tx=0x7f09bc02bb70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=12d11370ed62c69e server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:47:16.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.400+0000 7f09afe00640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:47:16.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.400+0000 7f09afe00640 1 -- 172.21.15.90:0/3246809859 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09bc04d070 con 0x55d4ff6ef9f0 2024-04-02T14:47:16.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.400+0000 7f09afe00640 1 -- 172.21.15.90:0/3246809859 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 8550+0+0 (secure 0 0 0) 0x7f09bc02c710 con 0x55d4ff6ef9f0 2024-04-02T14:47:16.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.400+0000 7f09d8235740 1 -- 172.21.15.90:0/3246809859 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f09c80a0af0 con 0x55d4ff6ef9f0 2024-04-02T14:47:16.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.401+0000 7f09afe00640 1 -- 172.21.15.90:0/3246809859 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f09bc057050 con 0x55d4ff6ef9f0 2024-04-02T14:47:16.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.401+0000 7f09d8235740 10 client.29348 fetch_fsmap learned FSMap version 27 2024-04-02T14:47:16.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.401+0000 7f09d8235740 1 -- 172.21.15.90:0/3246809859 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55d4ff706fa0 con 0x55d4ff6ef9f0 2024-04-02T14:47:16.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.402+0000 7f09afe00640 1 -- 172.21.15.90:0/3246809859 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f09c80a0af0 con 0x55d4ff6ef9f0 2024-04-02T14:47:16.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.402+0000 7f09afe00640 1 -- 172.21.15.90:0/3246809859 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f09bc0035f0 con 0x55d4ff6ef9f0 2024-04-02T14:47:16.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.403+0000 7f09d8235740 10 client.29348 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:47:16.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.403+0000 7f09d8235740 10 client.29348 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:47:16.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.403+0000 7f09d8235740 20 client.29348 populate_metadata read hostname 'smithi090' 2024-04-02T14:47:16.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.403+0000 7f09d8235740 1 --2- 172.21.15.90:0/3246809859 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55d4ff7084b0 0x55d4ff70a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:16.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.403+0000 7f09d8235740 4 client.29348 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:47:16.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.403+0000 7f09d8235740 1 -- 172.21.15.90:0/3246809859 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d4ff613040 con 0x55d4ff7084b0 2024-04-02T14:47:16.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.405+0000 7f09ce800640 1 --2- 172.21.15.90:0/3246809859 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55d4ff7084b0 0x55d4ff70a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:16.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.409+0000 7f09ce800640 1 --2- 172.21.15.90:0/3246809859 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55d4ff7084b0 0x55d4ff70a8c0 crc :-1 s=READY pgs=320 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=79a6383b38d57f6 server_cookie=9a63a3441d5ba936 in_seq=0 out_seq=0 2024-04-02T14:47:16.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.409+0000 7f09afe00640 10 client.29348 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:47:16.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.410+0000 7f09afe00640 1 -- 172.21.15.90:0/3246809859 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d4ff613040 con 0x55d4ff7084b0 2024-04-02T14:47:16.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.410+0000 7f09afe00640 10 client.29348 handle_command_reply: tid=0 2024-04-02T14:47:16.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.410+0000 7f09d8235740 1 client.29348 shutdown 2024-04-02T14:47:16.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.413+0000 7f09d8235740 1 -- 172.21.15.90:0/3246809859 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d4ff6ef9f0 msgr2=0x55d4ff6f5510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:16.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.413+0000 7f09d8235740 1 --2- 172.21.15.90:0/3246809859 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d4ff6ef9f0 0x55d4ff6f5510 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f09bc02c180 tx=0x7f09bc02bb70 comp rx=0 tx=0).stop 2024-04-02T14:47:16.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.413+0000 7f09d8235740 1 -- 172.21.15.90:0/3246809859 shutdown_connections 2024-04-02T14:47:16.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.413+0000 7f09d8235740 1 --2- 172.21.15.90:0/3246809859 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55d4ff7084b0 0x55d4ff70a8c0 crc :-1 s=READY pgs=320 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:16.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.413+0000 7f09d8235740 1 --2- 172.21.15.90:0/3246809859 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d4ff6f8080 0x55d4ff6f5a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:16.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.413+0000 7f09afe00640 0 client.29348 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:47:16.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.413+0000 7f09d8235740 1 --2- 172.21.15.90:0/3246809859 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d4ff6ef9f0 0x55d4ff6f5510 unknown :-1 s=CLOSED pgs=872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:16.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.413+0000 7f09d8235740 1 --2- 172.21.15.90:0/3246809859 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d4ff6eed10 0x55d4ff6f4fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:16.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.413+0000 7f09d8235740 1 -- 172.21.15.90:0/3246809859 >> 172.21.15.90:0/3246809859 conn(0x55d4ff6b5a50 msgr2=0x55d4ff6eb4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:47:16.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.414+0000 7f09d8235740 1 -- 172.21.15.90:0/3246809859 shutdown_connections 2024-04-02T14:47:16.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.414+0000 7f09d8235740 1 -- 172.21.15.90:0/3246809859 wait complete. 2024-04-02T14:47:16.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.414+0000 7f09d8235740 20 client.29348 trim_cache size 0 max 16384 2024-04-02T14:47:16.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:47:16 smithi177 ceph-mon[36431]: pgmap v385: 97 pgs: 97 active+clean; 4.1 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 18 MiB/s wr, 1.60k op/s 2024-04-02T14:47:16.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:47:16 smithi177 ceph-mon[36431]: osdmap e103: 12 total, 12 up, 12 in 2024-04-02T14:47:16.434 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:47:16 smithi090 ceph-mon[31674]: pgmap v385: 97 pgs: 97 active+clean; 4.1 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 18 MiB/s wr, 1.60k op/s 2024-04-02T14:47:16.434 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:47:16 smithi090 ceph-mon[31674]: osdmap e103: 12 total, 12 up, 12 in 2024-04-02T14:47:16.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.442+0000 7f09d8235740 1 Processor -- start 2024-04-02T14:47:16.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.442+0000 7f09d8235740 1 -- start start 2024-04-02T14:47:16.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.443+0000 7f09d8235740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d4ff6eed10 0x55d4ff6f7700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:16.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.443+0000 7f09d8235740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d4ff6ef9f0 0x55d4ff703be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:16.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.443+0000 7f09d8235740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d4ff6f8080 0x55d4ff6f4130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:16.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.443+0000 7f09d8235740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f09d01b3f50 con 0x55d4ff6ef9f0 2024-04-02T14:47:16.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.443+0000 7f09d8235740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f09d01b5a90 con 0x55d4ff6f8080 2024-04-02T14:47:16.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.443+0000 7f09d8235740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f09d01b4620 con 0x55d4ff6eed10 2024-04-02T14:47:16.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.443+0000 7f09ce800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d4ff6eed10 0x55d4ff6f7700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:16.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.443+0000 7f09cf200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d4ff6f8080 0x55d4ff6f4130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:16.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.443+0000 7f09ce800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d4ff6eed10 0x55d4ff6f7700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:59748/0 (socket says 172.21.15.90:59748) 2024-04-02T14:47:16.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.443+0000 7f09ce800640 1 -- 172.21.15.90:0/1129619885 learned_addr learned my addr 172.21.15.90:0/1129619885 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:47:16.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.444+0000 7f09cf200640 1 -- 172.21.15.90:0/1129619885 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d4ff6eed10 msgr2=0x55d4ff6f7700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:47:16.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.444+0000 7f09cf200640 1 --2- 172.21.15.90:0/1129619885 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d4ff6eed10 0x55d4ff6f7700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:16.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.444+0000 7f09cf200640 1 -- 172.21.15.90:0/1129619885 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d4ff6ef9f0 msgr2=0x55d4ff703be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:47:16.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.444+0000 7f09cde00640 1 --2- 172.21.15.90:0/1129619885 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d4ff6ef9f0 0x55d4ff703be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:16.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.444+0000 7f09cf200640 1 --2- 172.21.15.90:0/1129619885 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d4ff6ef9f0 0x55d4ff703be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:16.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.444+0000 7f09cf200640 1 -- 172.21.15.90:0/1129619885 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d4ff70aef0 con 0x55d4ff6f8080 2024-04-02T14:47:16.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.445+0000 7f09cde00640 1 --2- 172.21.15.90:0/1129619885 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d4ff6ef9f0 0x55d4ff703be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:47:16.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.445+0000 7f09cf200640 1 --2- 172.21.15.90:0/1129619885 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d4ff6f8080 0x55d4ff6f4130 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f09c80a5570 tx=0x7f09c8009650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6f1d72a2b02e04b1 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:47:16.440 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.446+0000 7f09baa00640 1 -- 172.21.15.90:0/1129619885 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09c80a0af0 con 0x55d4ff6f8080 2024-04-02T14:47:16.440 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.446+0000 7f09baa00640 1 -- 172.21.15.90:0/1129619885 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f09c80ba070 con 0x55d4ff6f8080 2024-04-02T14:47:16.440 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.446+0000 7f09baa00640 1 -- 172.21.15.90:0/1129619885 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09c809bb90 con 0x55d4ff6f8080 2024-04-02T14:47:16.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.447+0000 7f09d8235740 1 -- 172.21.15.90:0/1129619885 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d4ff6f8080 msgr2=0x55d4ff6f4130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:16.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.447+0000 7f09d8235740 1 --2- 172.21.15.90:0/1129619885 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d4ff6f8080 0x55d4ff6f4130 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f09c80a5570 tx=0x7f09c8009650 comp rx=0 tx=0).stop 2024-04-02T14:47:16.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.447+0000 7f09d8235740 1 -- 172.21.15.90:0/1129619885 shutdown_connections 2024-04-02T14:47:16.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.447+0000 7f09d8235740 1 --2- 172.21.15.90:0/1129619885 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d4ff6f8080 0x55d4ff6f4130 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:16.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.447+0000 7f09d8235740 1 --2- 172.21.15.90:0/1129619885 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d4ff6ef9f0 0x55d4ff703be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:16.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.447+0000 7f09d8235740 1 --2- 172.21.15.90:0/1129619885 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d4ff6eed10 0x55d4ff6f7700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:16.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.447+0000 7f09d8235740 1 -- 172.21.15.90:0/1129619885 >> 172.21.15.90:0/1129619885 conn(0x55d4ff6b5a50 msgr2=0x55d4ff6f4f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:47:16.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.450+0000 7f09d8235740 1 -- 172.21.15.90:0/1129619885 shutdown_connections 2024-04-02T14:47:16.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.450+0000 7f09d8235740 1 -- 172.21.15.90:0/1129619885 wait complete. 2024-04-02T14:47:16.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.450+0000 7f09d8235740 1 -- start start 2024-04-02T14:47:16.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.451+0000 7f09d8235740 1 Processor -- start 2024-04-02T14:47:16.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.452+0000 7f09d8235740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d4ff6eed10 0x55d4ff6f1dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:16.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.452+0000 7f09d8235740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d4ff6ef9f0 0x55d4ff6f2310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:16.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.452+0000 7f09ce800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d4ff6eed10 0x55d4ff6f1dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:16.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.453+0000 7f09d8235740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d4ff6f8080 0x55d4ff6f2850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:16.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.453+0000 7f09ce800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d4ff6eed10 0x55d4ff6f1dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:53272/0 (socket says 172.21.15.90:53272) 2024-04-02T14:47:16.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.453+0000 7f09d8235740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f09d01b53c0 con 0x55d4ff6ef9f0 2024-04-02T14:47:16.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.453+0000 7f09d8235740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f09d01b4cf0 con 0x55d4ff6eed10 2024-04-02T14:47:16.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.453+0000 7f09d8235740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f09d01b1f50 con 0x55d4ff6f8080 2024-04-02T14:47:16.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.453+0000 7f09ce800640 1 -- 172.21.15.90:0/3851439136 learned_addr learned my addr 172.21.15.90:0/3851439136 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:47:16.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.453+0000 7f09d8235740 10 client.0 _ll_register_callbacks cb 0x7f09bc004080 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:47:16.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.454+0000 7f09cde00640 1 --2- 172.21.15.90:0/3851439136 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d4ff6ef9f0 0x55d4ff6f2310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:16.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.454+0000 7f09ce800640 1 -- 172.21.15.90:0/3851439136 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d4ff6f8080 msgr2=0x55d4ff6f2850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:47:16.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.454+0000 7f09ce800640 1 --2- 172.21.15.90:0/3851439136 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d4ff6f8080 0x55d4ff6f2850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:16.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.454+0000 7f09ce800640 1 -- 172.21.15.90:0/3851439136 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d4ff6ef9f0 msgr2=0x55d4ff6f2310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:47:16.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.454+0000 7f09ce800640 1 --2- 172.21.15.90:0/3851439136 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d4ff6ef9f0 0x55d4ff6f2310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:16.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.454+0000 7f09ce800640 1 -- 172.21.15.90:0/3851439136 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d4ff76db60 con 0x55d4ff6eed10 2024-04-02T14:47:16.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.454+0000 7f09cde00640 1 --2- 172.21.15.90:0/3851439136 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d4ff6ef9f0 0x55d4ff6f2310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:47:16.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.454+0000 7f09ce800640 1 --2- 172.21.15.90:0/3851439136 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d4ff6eed10 0x55d4ff6f1dd0 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f09c001ae50 tx=0x7f09c001a600 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a923744b08c4b95d server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:47:16.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.454+0000 7f09afe00640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:47:16.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.455+0000 7f09afe00640 1 -- 172.21.15.90:0/3851439136 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09c0027060 con 0x55d4ff6eed10 2024-04-02T14:47:16.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.455+0000 7f09afe00640 1 -- 172.21.15.90:0/3851439136 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 8550+0+0 (secure 0 0 0) 0x7f09c0011910 con 0x55d4ff6eed10 2024-04-02T14:47:16.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.455+0000 7f09d8235740 1 -- 172.21.15.90:0/3851439136 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f09c80ba070 con 0x55d4ff6eed10 2024-04-02T14:47:16.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.455+0000 7f09afe00640 1 -- 172.21.15.90:0/3851439136 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f09c0011b10 con 0x55d4ff6eed10 2024-04-02T14:47:16.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.455+0000 7f09d8235740 10 client.29354 fetch_fsmap learned FSMap version 27 2024-04-02T14:47:16.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.455+0000 7f09d8235740 1 -- 172.21.15.90:0/3851439136 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55d4ff76ee10 con 0x55d4ff6eed10 2024-04-02T14:47:16.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.457+0000 7f09afe00640 1 -- 172.21.15.90:0/3851439136 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f09c80ba070 con 0x55d4ff6eed10 2024-04-02T14:47:16.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.457+0000 7f09afe00640 1 -- 172.21.15.90:0/3851439136 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f09c0021030 con 0x55d4ff6eed10 2024-04-02T14:47:16.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.457+0000 7f09d8235740 10 client.29354 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:47:16.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.457+0000 7f09d8235740 10 client.29354 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:47:16.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.457+0000 7f09d8235740 20 client.29354 populate_metadata read hostname 'smithi090' 2024-04-02T14:47:16.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.457+0000 7f09d8235740 1 --2- 172.21.15.90:0/3851439136 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55d4ff7084b0 0x55d4ff7715a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:16.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.457+0000 7f09d8235740 4 client.29354 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-02T14:47:16.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.457+0000 7f09d8235740 1 -- 172.21.15.90:0/3851439136 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55d4ff660c30 con 0x55d4ff7084b0 2024-04-02T14:47:16.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.457+0000 7f09cde00640 1 --2- 172.21.15.90:0/3851439136 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55d4ff7084b0 0x55d4ff7715a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:16.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.458+0000 7f09cde00640 1 --2- 172.21.15.90:0/3851439136 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55d4ff7084b0 0x55d4ff7715a0 crc :-1 s=READY pgs=321 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f0a40c16019e8955 server_cookie=135a48c9cb58bde5 in_seq=0 out_seq=0 2024-04-02T14:47:16.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.458+0000 7f09afe00640 10 client.29354 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:47:16.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.458+0000 7f09afe00640 1 -- 172.21.15.90:0/3851439136 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55d4ff660c30 con 0x55d4ff7084b0 2024-04-02T14:47:16.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.458+0000 7f09afe00640 10 client.29354 handle_command_reply: tid=0 2024-04-02T14:47:16.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.458+0000 7f09d8235740 1 client.29354 shutdown 2024-04-02T14:47:16.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.459+0000 7f09d8235740 1 -- 172.21.15.90:0/3851439136 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d4ff6eed10 msgr2=0x55d4ff6f1dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:16.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.459+0000 7f09d8235740 1 --2- 172.21.15.90:0/3851439136 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d4ff6eed10 0x55d4ff6f1dd0 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f09c001ae50 tx=0x7f09c001a600 comp rx=0 tx=0).stop 2024-04-02T14:47:16.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.459+0000 7f09d8235740 1 -- 172.21.15.90:0/3851439136 shutdown_connections 2024-04-02T14:47:16.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.459+0000 7f09d8235740 1 --2- 172.21.15.90:0/3851439136 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55d4ff7084b0 0x55d4ff7715a0 crc :-1 s=READY pgs=321 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:16.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.459+0000 7f09d8235740 1 --2- 172.21.15.90:0/3851439136 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d4ff6f8080 0x55d4ff6f2850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:16.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.459+0000 7f09d8235740 1 --2- 172.21.15.90:0/3851439136 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d4ff6ef9f0 0x55d4ff6f2310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:16.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.459+0000 7f09d8235740 1 --2- 172.21.15.90:0/3851439136 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d4ff6eed10 0x55d4ff6f1dd0 unknown :-1 s=CLOSED pgs=874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:16.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.459+0000 7f09afe00640 0 client.29354 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:47:16.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.459+0000 7f09d8235740 1 -- 172.21.15.90:0/3851439136 >> 172.21.15.90:0/3851439136 conn(0x55d4ff6b5a50 msgr2=0x55d4ff6f3fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:47:16.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.461+0000 7f09d8235740 1 -- 172.21.15.90:0/3851439136 shutdown_connections 2024-04-02T14:47:16.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.461+0000 7f09d8235740 1 -- 172.21.15.90:0/3851439136 wait complete. 2024-04-02T14:47:16.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.462+0000 7f09d8235740 20 client.29354 trim_cache size 0 max 16384 2024-04-02T14:47:16.456 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-02T14:47:16.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.463+0000 7f09d5400640 1 -- 172.21.15.90:0/3138095912 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f09b40a3d20 msgr2=0x7f09b40a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:16.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.463+0000 7f09d5400640 1 --2- 172.21.15.90:0/3138095912 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f09b40a3d20 0x7f09b40a61e0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f09bc0097b0 tx=0x7f09bc0058d0 comp rx=0 tx=0).stop 2024-04-02T14:47:16.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.463+0000 7f09d5400640 1 -- 172.21.15.90:0/3138095912 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f09d0106a50 msgr2=0x7f09d019e7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:16.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.463+0000 7f09d5400640 1 --2- 172.21.15.90:0/3138095912 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f09d0106a50 0x7f09d019e7b0 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f09c8009fd0 tx=0x7f09c800c740 comp rx=0 tx=0).stop 2024-04-02T14:47:16.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.463+0000 7f09d5400640 1 -- 172.21.15.90:0/3138095912 shutdown_connections 2024-04-02T14:47:16.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.463+0000 7f09d5400640 1 --2- 172.21.15.90:0/3138095912 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f09b40a3d20 0x7f09b40a61e0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:16.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.463+0000 7f09d5400640 1 --2- 172.21.15.90:0/3138095912 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f09d0106a50 0x7f09d019e7b0 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:16.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.463+0000 7f09d5400640 1 --2- 172.21.15.90:0/3138095912 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f09d01045c0 0x7f09d01a4be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:16.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.463+0000 7f09d5400640 1 --2- 172.21.15.90:0/3138095912 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f09d0069240 0x7f09d01a46a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:16.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.463+0000 7f09d5400640 1 -- 172.21.15.90:0/3138095912 >> 172.21.15.90:0/3138095912 conn(0x7f09d0077b00 msgr2=0x7f09d0101910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:47:16.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.463+0000 7f09d5400640 1 -- 172.21.15.90:0/3138095912 shutdown_connections 2024-04-02T14:47:16.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:16.463+0000 7f09d5400640 1 -- 172.21.15.90:0/3138095912 wait complete. 2024-04-02T14:47:16.604 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:47:16 smithi092 ceph-mon[36497]: pgmap v385: 97 pgs: 97 active+clean; 4.1 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 18 MiB/s wr, 1.60k op/s 2024-04-02T14:47:16.604 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:47:16 smithi092 ceph-mon[36497]: osdmap e103: 12 total, 12 up, 12 in 2024-04-02T14:47:16.824 INFO:teuthology.orchestra.run.smithi090.stderr:progress: 705.0 s, 1667.4 tps, lat 59.250 ms stddev 74.177 2024-04-02T14:47:17.484 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-02T14:47:17.484 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T14:47:17.484 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-02T14:47:17.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.726+0000 7fef74400640 1 -- 172.21.15.90:0/901009458 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fef6c103f10 msgr2=0x7fef6c1042f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:17.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.726+0000 7fef74400640 1 --2- 172.21.15.90:0/901009458 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fef6c103f10 0x7fef6c1042f0 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7fef60009900 tx=0x7fef6002bc40 comp rx=0 tx=0).stop 2024-04-02T14:47:17.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.726+0000 7fef74400640 1 -- 172.21.15.90:0/901009458 shutdown_connections 2024-04-02T14:47:17.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.726+0000 7fef74400640 1 --2- 172.21.15.90:0/901009458 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fef6c108fe0 0x7fef6c110bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:17.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.726+0000 7fef74400640 1 --2- 172.21.15.90:0/901009458 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fef6c1048c0 0x7fef6c108900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:17.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.726+0000 7fef74400640 1 --2- 172.21.15.90:0/901009458 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fef6c103f10 0x7fef6c1042f0 unknown :-1 s=CLOSED pgs=875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:17.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.726+0000 7fef74400640 1 -- 172.21.15.90:0/901009458 >> 172.21.15.90:0/901009458 conn(0x7fef6c0ff750 msgr2=0x7fef6c101b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:47:17.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.726+0000 7fef74400640 1 -- 172.21.15.90:0/901009458 shutdown_connections 2024-04-02T14:47:17.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.726+0000 7fef74400640 1 -- 172.21.15.90:0/901009458 wait complete. 2024-04-02T14:47:17.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.728+0000 7fef74400640 1 Processor -- start 2024-04-02T14:47:17.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.728+0000 7fef74400640 1 -- start start 2024-04-02T14:47:17.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.729+0000 7fef74400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fef6c103f10 0x7fef6c19a610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:17.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.729+0000 7fef74400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fef6c108fe0 0x7fef6c19ab50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:17.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.729+0000 7fef74400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fef6c19eed0 0x7fef6c19f380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:17.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.729+0000 7fef74400640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fef6c114090 con 0x7fef6c19eed0 2024-04-02T14:47:17.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.729+0000 7fef74400640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fef6c113f10 con 0x7fef6c103f10 2024-04-02T14:47:17.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.729+0000 7fef74400640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fef6c114210 con 0x7fef6c108fe0 2024-04-02T14:47:17.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.729+0000 7fef71000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fef6c108fe0 0x7fef6c19ab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:17.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.729+0000 7fef71000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fef6c108fe0 0x7fef6c19ab50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:59776/0 (socket says 172.21.15.90:59776) 2024-04-02T14:47:17.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.729+0000 7fef71a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fef6c103f10 0x7fef6c19a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:17.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.729+0000 7fef71000640 1 -- 172.21.15.90:0/4273477586 learned_addr learned my addr 172.21.15.90:0/4273477586 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:47:17.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.730+0000 7fef72400640 1 --2- 172.21.15.90:0/4273477586 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fef6c19eed0 0x7fef6c19f380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:17.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.730+0000 7fef71000640 1 -- 172.21.15.90:0/4273477586 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fef6c103f10 msgr2=0x7fef6c19a610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:17.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.730+0000 7fef71000640 1 --2- 172.21.15.90:0/4273477586 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fef6c103f10 0x7fef6c19a610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:17.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.730+0000 7fef71000640 1 -- 172.21.15.90:0/4273477586 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fef6c19eed0 msgr2=0x7fef6c19f380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:17.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.730+0000 7fef71000640 1 --2- 172.21.15.90:0/4273477586 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fef6c19eed0 0x7fef6c19f380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:17.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.730+0000 7fef71000640 1 -- 172.21.15.90:0/4273477586 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef6c19f9c0 con 0x7fef6c108fe0 2024-04-02T14:47:17.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.730+0000 7fef71000640 1 --2- 172.21.15.90:0/4273477586 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fef6c108fe0 0x7fef6c19ab50 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7fef58005fd0 tx=0x7fef5800dde0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:47:17.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.730+0000 7fef6a000640 1 -- 172.21.15.90:0/4273477586 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef58017070 con 0x7fef6c108fe0 2024-04-02T14:47:17.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.731+0000 7fef74400640 1 -- 172.21.15.90:0/4273477586 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef6c19fcb0 con 0x7fef6c108fe0 2024-04-02T14:47:17.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.731+0000 7fef6a000640 1 -- 172.21.15.90:0/4273477586 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fef58009c30 con 0x7fef6c108fe0 2024-04-02T14:47:17.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.731+0000 7fef6a000640 1 -- 172.21.15.90:0/4273477586 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef58010460 con 0x7fef6c108fe0 2024-04-02T14:47:17.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.731+0000 7fef74400640 1 -- 172.21.15.90:0/4273477586 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef6c1a74a0 con 0x7fef6c108fe0 2024-04-02T14:47:17.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.735+0000 7fef77206740 1 Processor -- start 2024-04-02T14:47:17.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.735+0000 7fef77206740 1 -- start start 2024-04-02T14:47:17.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.735+0000 7fef77206740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5559d8822030 0x5559d8820d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:17.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.735+0000 7fef77206740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5559d8821350 0x5559d88217d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:17.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.736+0000 7fef77206740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5559d8832ed0 0x5559d88352c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:17.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.736+0000 7fef77206740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5559d8785310 con 0x5559d8832ed0 2024-04-02T14:47:17.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.736+0000 7fef77206740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x5559d872d690 con 0x5559d8822030 2024-04-02T14:47:17.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.736+0000 7fef72400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5559d8822030 0x5559d8820d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:17.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.736+0000 7fef77206740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5559d87812c0 con 0x5559d8821350 2024-04-02T14:47:17.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.736+0000 7fef72400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5559d8822030 0x5559d8820d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:53298/0 (socket says 172.21.15.90:53298) 2024-04-02T14:47:17.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.736+0000 7fef72400640 1 -- 172.21.15.90:0/3428617394 learned_addr learned my addr 172.21.15.90:0/3428617394 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:47:17.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.736+0000 7fef71a00640 1 --2- 172.21.15.90:0/3428617394 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5559d8821350 0x5559d88217d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:17.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.736+0000 7fef72400640 1 -- 172.21.15.90:0/3428617394 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5559d8821350 msgr2=0x5559d88217d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:47:17.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.736+0000 7fef72400640 1 --2- 172.21.15.90:0/3428617394 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5559d8821350 0x5559d88217d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:17.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.736+0000 7fef72400640 1 -- 172.21.15.90:0/3428617394 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5559d8832ed0 msgr2=0x5559d88352c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-02T14:47:17.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.736+0000 7fef72400640 1 --2- 172.21.15.90:0/3428617394 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5559d8832ed0 0x5559d88352c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:17.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.736+0000 7fef72400640 1 -- 172.21.15.90:0/3428617394 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5559d85617e0 con 0x5559d8822030 2024-04-02T14:47:17.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.737+0000 7fef6a000640 1 -- 172.21.15.90:0/4273477586 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fef58003e70 con 0x7fef6c108fe0 2024-04-02T14:47:17.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.737+0000 7fef72400640 1 --2- 172.21.15.90:0/3428617394 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5559d8822030 0x5559d8820d80 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7fef64009f20 tx=0x7fef64010860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=84912b139b03c1a9 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:47:17.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.737+0000 7fef5ea00640 1 -- 172.21.15.90:0/3428617394 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef64011700 con 0x5559d8822030 2024-04-02T14:47:17.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.737+0000 7fef5ea00640 1 -- 172.21.15.90:0/3428617394 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fef64011d80 con 0x5559d8822030 2024-04-02T14:47:17.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.737+0000 7fef5ea00640 1 -- 172.21.15.90:0/3428617394 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef6401a380 con 0x5559d8822030 2024-04-02T14:47:17.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.738+0000 7fef77206740 1 -- 172.21.15.90:0/3428617394 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5559d8822030 msgr2=0x5559d8820d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:17.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.738+0000 7fef77206740 1 --2- 172.21.15.90:0/3428617394 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5559d8822030 0x5559d8820d80 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7fef64009f20 tx=0x7fef64010860 comp rx=0 tx=0).stop 2024-04-02T14:47:17.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.738+0000 7fef77206740 1 -- 172.21.15.90:0/3428617394 shutdown_connections 2024-04-02T14:47:17.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.738+0000 7fef77206740 1 --2- 172.21.15.90:0/3428617394 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5559d8832ed0 0x5559d88352c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:17.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.738+0000 7fef77206740 1 --2- 172.21.15.90:0/3428617394 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5559d8821350 0x5559d88217d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:17.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.738+0000 7fef77206740 1 --2- 172.21.15.90:0/3428617394 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5559d8822030 0x5559d8820d80 unknown :-1 s=CLOSED pgs=876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:17.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.738+0000 7fef77206740 1 -- 172.21.15.90:0/3428617394 >> 172.21.15.90:0/3428617394 conn(0x5559d87b5270 msgr2=0x5559d881bda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:47:17.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.738+0000 7fef77206740 1 -- 172.21.15.90:0/3428617394 shutdown_connections 2024-04-02T14:47:17.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.738+0000 7fef77206740 1 -- 172.21.15.90:0/3428617394 wait complete. 2024-04-02T14:47:17.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.738+0000 7fef6a000640 1 --2- 172.21.15.90:0/4273477586 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fef540a3c30 0x7fef540a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:17.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.738+0000 7fef77206740 1 -- start start 2024-04-02T14:47:17.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.738+0000 7fef6a000640 1 -- 172.21.15.90:0/4273477586 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 8550+0+0 (secure 0 0 0) 0x7fef580d0540 con 0x7fef6c108fe0 2024-04-02T14:47:17.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.739+0000 7fef77206740 1 Processor -- start 2024-04-02T14:47:17.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.739+0000 7fef72400640 1 --2- 172.21.15.90:0/4273477586 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fef540a3c30 0x7fef540a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:17.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.739+0000 7fef77206740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5559d8821350 0x5559d882fed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:17.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.739+0000 7fef77206740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5559d8822030 0x5559d8830410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:17.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.739+0000 7fef77206740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5559d8832ed0 0x5559d8830950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:17.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.739+0000 7fef77206740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5559d8835d40 con 0x5559d8832ed0 2024-04-02T14:47:17.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.739+0000 7fef77206740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x5559d8740a60 con 0x5559d8822030 2024-04-02T14:47:17.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.739+0000 7fef77206740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5559d8835ec0 con 0x5559d8821350 2024-04-02T14:47:17.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.740+0000 7fef77206740 10 client.0 _ll_register_callbacks cb 0x5559d8746030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:47:17.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.740+0000 7fef72400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5559d8832ed0 0x5559d8830950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:17.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.740+0000 7fef71000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5559d8822030 0x5559d8830410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:17.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.740+0000 7fef71000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5559d8822030 0x5559d8830410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:53306/0 (socket says 172.21.15.90:53306) 2024-04-02T14:47:17.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.740+0000 7fef71000640 1 -- 172.21.15.90:0/2883609584 learned_addr learned my addr 172.21.15.90:0/2883609584 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:47:17.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.740+0000 7fef72400640 1 --2- 172.21.15.90:0/4273477586 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fef540a3c30 0x7fef540a60f0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fef6401c9e0 tx=0x7fef64000f40 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:47:17.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.740+0000 7fef71000640 1 -- 172.21.15.90:0/2883609584 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5559d8821350 msgr2=0x5559d882fed0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-02T14:47:17.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.740+0000 7fef71000640 1 --2- 172.21.15.90:0/2883609584 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5559d8821350 0x5559d882fed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:17.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.740+0000 7fef71000640 1 -- 172.21.15.90:0/2883609584 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5559d8832ed0 msgr2=0x5559d8830950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:47:17.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.740+0000 7fef71000640 1 --2- 172.21.15.90:0/2883609584 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5559d8832ed0 0x5559d8830950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:17.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.740+0000 7fef71000640 1 -- 172.21.15.90:0/2883609584 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5559d85c1f70 con 0x5559d8822030 2024-04-02T14:47:17.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.740+0000 7fef72400640 1 --2- 172.21.15.90:0/2883609584 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5559d8832ed0 0x5559d8830950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:47:17.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.741+0000 7fef71000640 1 --2- 172.21.15.90:0/2883609584 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5559d8822030 0x5559d8830410 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7fef58009610 tx=0x7fef5809cb60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5f998143af869251 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:47:17.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.741+0000 7fef53400640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:47:17.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.742+0000 7fef53400640 1 -- 172.21.15.90:0/2883609584 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef580a0810 con 0x5559d8822030 2024-04-02T14:47:17.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.742+0000 7fef77206740 1 -- 172.21.15.90:0/2883609584 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fef64011d80 con 0x5559d8822030 2024-04-02T14:47:17.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.742+0000 7fef53400640 1 -- 172.21.15.90:0/2883609584 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 8550+0+0 (secure 0 0 0) 0x7fef58098ad0 con 0x5559d8822030 2024-04-02T14:47:17.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.743+0000 7fef53400640 1 -- 172.21.15.90:0/2883609584 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fef580a0810 con 0x5559d8822030 2024-04-02T14:47:17.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.743+0000 7fef77206740 10 client.29366 fetch_fsmap learned FSMap version 27 2024-04-02T14:47:17.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.743+0000 7fef77206740 1 -- 172.21.15.90:0/2883609584 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5559d8459170 con 0x5559d8822030 2024-04-02T14:47:17.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.743+0000 7fef53400640 1 -- 172.21.15.90:0/2883609584 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fef64011d80 con 0x5559d8822030 2024-04-02T14:47:17.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.745+0000 7fef53400640 1 -- 172.21.15.90:0/2883609584 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fef5809b8b0 con 0x5559d8822030 2024-04-02T14:47:17.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.745+0000 7fef77206740 10 client.29366 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:47:17.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.745+0000 7fef77206740 10 client.29366 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:47:17.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.745+0000 7fef77206740 20 client.29366 populate_metadata read hostname 'smithi090' 2024-04-02T14:47:17.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.745+0000 7fef77206740 1 --2- 172.21.15.90:0/2883609584 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5559d8843340 0x5559d8845730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:17.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.745+0000 7fef77206740 4 client.29366 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:47:17.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.745+0000 7fef77206740 1 -- 172.21.15.90:0/2883609584 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5559d8792850 con 0x5559d8843340 2024-04-02T14:47:17.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.745+0000 7fef71a00640 1 --2- 172.21.15.90:0/2883609584 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5559d8843340 0x5559d8845730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:17.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.746+0000 7fef71a00640 1 --2- 172.21.15.90:0/2883609584 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5559d8843340 0x5559d8845730 crc :-1 s=READY pgs=322 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cd9bf2523c176518 server_cookie=d7d0bd6db34979a9 in_seq=0 out_seq=0 2024-04-02T14:47:17.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.746+0000 7fef53400640 10 client.29366 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:47:17.741 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.746+0000 7fef53400640 1 -- 172.21.15.90:0/2883609584 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5559d8792850 con 0x5559d8843340 2024-04-02T14:47:17.741 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.746+0000 7fef53400640 10 client.29366 handle_command_reply: tid=0 2024-04-02T14:47:17.741 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.746+0000 7fef77206740 1 client.29366 shutdown 2024-04-02T14:47:17.741 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.747+0000 7fef77206740 1 -- 172.21.15.90:0/2883609584 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5559d8822030 msgr2=0x5559d8830410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:17.741 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.747+0000 7fef77206740 1 --2- 172.21.15.90:0/2883609584 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5559d8822030 0x5559d8830410 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7fef58009610 tx=0x7fef5809cb60 comp rx=0 tx=0).stop 2024-04-02T14:47:17.741 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.747+0000 7fef77206740 1 -- 172.21.15.90:0/2883609584 shutdown_connections 2024-04-02T14:47:17.741 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.747+0000 7fef77206740 1 --2- 172.21.15.90:0/2883609584 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5559d8843340 0x5559d8845730 crc :-1 s=READY pgs=322 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:17.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.747+0000 7fef77206740 1 --2- 172.21.15.90:0/2883609584 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5559d8832ed0 0x5559d8830950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:17.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.747+0000 7fef77206740 1 --2- 172.21.15.90:0/2883609584 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5559d8822030 0x5559d8830410 unknown :-1 s=CLOSED pgs=877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:17.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.747+0000 7fef77206740 1 --2- 172.21.15.90:0/2883609584 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5559d8821350 0x5559d882fed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:17.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.747+0000 7fef77206740 1 -- 172.21.15.90:0/2883609584 >> 172.21.15.90:0/2883609584 conn(0x5559d87b5270 msgr2=0x5559d881daf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:47:17.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.747+0000 7fef53400640 0 client.29366 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:47:17.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.747+0000 7fef77206740 1 -- 172.21.15.90:0/2883609584 shutdown_connections 2024-04-02T14:47:17.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.747+0000 7fef77206740 1 -- 172.21.15.90:0/2883609584 wait complete. 2024-04-02T14:47:17.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.747+0000 7fef77206740 20 client.29366 trim_cache size 0 max 16384 2024-04-02T14:47:17.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.773+0000 7fef77206740 1 Processor -- start 2024-04-02T14:47:17.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.773+0000 7fef77206740 1 -- start start 2024-04-02T14:47:17.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.774+0000 7fef77206740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5559d8821350 0x5559d8832600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:17.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.774+0000 7fef77206740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5559d8822030 0x5559d8830790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:17.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.774+0000 7fef77206740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5559d8832ed0 0x5559d884b040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:17.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.774+0000 7fef77206740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fef6c1ab7e0 con 0x5559d8822030 2024-04-02T14:47:17.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.774+0000 7fef77206740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fef6c1ad320 con 0x5559d8821350 2024-04-02T14:47:17.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.774+0000 7fef77206740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fef6c1abeb0 con 0x5559d8832ed0 2024-04-02T14:47:17.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.774+0000 7fef72400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5559d8832ed0 0x5559d884b040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:17.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.774+0000 7fef71a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5559d8821350 0x5559d8832600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:17.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.774+0000 7fef72400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5559d8832ed0 0x5559d884b040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:59804/0 (socket says 172.21.15.90:59804) 2024-04-02T14:47:17.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.774+0000 7fef71a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5559d8821350 0x5559d8832600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:53312/0 (socket says 172.21.15.90:53312) 2024-04-02T14:47:17.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.774+0000 7fef72400640 1 -- 172.21.15.90:0/3298942401 learned_addr learned my addr 172.21.15.90:0/3298942401 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:47:17.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.775+0000 7fef71a00640 1 -- 172.21.15.90:0/3298942401 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5559d8832ed0 msgr2=0x5559d884b040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:47:17.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.775+0000 7fef71a00640 1 --2- 172.21.15.90:0/3298942401 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5559d8832ed0 0x5559d884b040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:17.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.775+0000 7fef71a00640 1 -- 172.21.15.90:0/3298942401 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5559d8822030 msgr2=0x5559d8830790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:47:17.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.775+0000 7fef71000640 1 --2- 172.21.15.90:0/3298942401 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5559d8822030 0x5559d8830790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:17.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.775+0000 7fef71a00640 1 --2- 172.21.15.90:0/3298942401 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5559d8822030 0x5559d8830790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:17.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.775+0000 7fef71a00640 1 -- 172.21.15.90:0/3298942401 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5559d8830cd0 con 0x5559d8821350 2024-04-02T14:47:17.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.775+0000 7fef72400640 1 --2- 172.21.15.90:0/3298942401 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5559d8832ed0 0x5559d884b040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:47:17.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.775+0000 7fef71000640 1 --2- 172.21.15.90:0/3298942401 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5559d8822030 0x5559d8830790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:47:17.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.775+0000 7fef71a00640 1 --2- 172.21.15.90:0/3298942401 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5559d8821350 0x5559d8832600 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7fef60000c00 tx=0x7fef6002e040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4d4289b6e1b5c9bf server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:47:17.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.775+0000 7fef5e000640 1 -- 172.21.15.90:0/3298942401 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef60046c90 con 0x5559d8821350 2024-04-02T14:47:17.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.775+0000 7fef5e000640 1 -- 172.21.15.90:0/3298942401 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fef6004c070 con 0x5559d8821350 2024-04-02T14:47:17.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.775+0000 7fef5e000640 1 -- 172.21.15.90:0/3298942401 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef600046c0 con 0x5559d8821350 2024-04-02T14:47:17.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.776+0000 7fef77206740 1 -- 172.21.15.90:0/3298942401 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5559d8821350 msgr2=0x5559d8832600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:17.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.776+0000 7fef77206740 1 --2- 172.21.15.90:0/3298942401 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5559d8821350 0x5559d8832600 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7fef60000c00 tx=0x7fef6002e040 comp rx=0 tx=0).stop 2024-04-02T14:47:17.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.776+0000 7fef77206740 1 -- 172.21.15.90:0/3298942401 shutdown_connections 2024-04-02T14:47:17.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.776+0000 7fef77206740 1 --2- 172.21.15.90:0/3298942401 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5559d8832ed0 0x5559d884b040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:17.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.776+0000 7fef77206740 1 --2- 172.21.15.90:0/3298942401 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5559d8822030 0x5559d8830790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:17.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.776+0000 7fef77206740 1 --2- 172.21.15.90:0/3298942401 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5559d8821350 0x5559d8832600 unknown :-1 s=CLOSED pgs=878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:17.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.776+0000 7fef77206740 1 -- 172.21.15.90:0/3298942401 >> 172.21.15.90:0/3298942401 conn(0x5559d87b5270 msgr2=0x5559d881a590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:47:17.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.776+0000 7fef77206740 1 -- 172.21.15.90:0/3298942401 shutdown_connections 2024-04-02T14:47:17.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.776+0000 7fef77206740 1 -- 172.21.15.90:0/3298942401 wait complete. 2024-04-02T14:47:17.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.777+0000 7fef77206740 1 -- start start 2024-04-02T14:47:17.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.777+0000 7fef77206740 1 Processor -- start 2024-04-02T14:47:17.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.777+0000 7fef77206740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5559d8821350 0x5559d883b450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:17.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.777+0000 7fef77206740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5559d8822030 0x5559d8836110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:17.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.777+0000 7fef77206740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5559d8832ed0 0x5559d8836650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:17.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.777+0000 7fef77206740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fef6c1acc50 con 0x5559d8822030 2024-04-02T14:47:17.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.777+0000 7fef77206740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fef6c1ac580 con 0x5559d8821350 2024-04-02T14:47:17.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.777+0000 7fef77206740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fef6c1a9780 con 0x5559d8832ed0 2024-04-02T14:47:17.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.777+0000 7fef77206740 10 client.0 _ll_register_callbacks cb 0x7fef58003810 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:47:17.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.778+0000 7fef71000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5559d8822030 0x5559d8836110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:17.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.778+0000 7fef71a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5559d8821350 0x5559d883b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:17.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.778+0000 7fef72400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5559d8832ed0 0x5559d8836650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:17.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.778+0000 7fef72400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5559d8832ed0 0x5559d8836650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:59818/0 (socket says 172.21.15.90:59818) 2024-04-02T14:47:17.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.778+0000 7fef71a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5559d8821350 0x5559d883b450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:53316/0 (socket says 172.21.15.90:53316) 2024-04-02T14:47:17.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.778+0000 7fef72400640 1 -- 172.21.15.90:0/3314953561 learned_addr learned my addr 172.21.15.90:0/3314953561 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:47:17.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.778+0000 7fef71000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5559d8822030 0x5559d8836110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:37204/0 (socket says 172.21.15.90:37204) 2024-04-02T14:47:17.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.778+0000 7fef71a00640 1 -- 172.21.15.90:0/3314953561 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5559d8832ed0 msgr2=0x5559d8836650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:47:17.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.778+0000 7fef71a00640 1 --2- 172.21.15.90:0/3314953561 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5559d8832ed0 0x5559d8836650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:17.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.778+0000 7fef71a00640 1 -- 172.21.15.90:0/3314953561 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5559d8822030 msgr2=0x5559d8836110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:47:17.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.778+0000 7fef71a00640 1 --2- 172.21.15.90:0/3314953561 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5559d8822030 0x5559d8836110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:17.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.778+0000 7fef71a00640 1 -- 172.21.15.90:0/3314953561 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5559d8836dc0 con 0x5559d8821350 2024-04-02T14:47:17.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.778+0000 7fef72400640 1 --2- 172.21.15.90:0/3314953561 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5559d8832ed0 0x5559d8836650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:47:17.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.778+0000 7fef71000640 1 --2- 172.21.15.90:0/3314953561 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5559d8822030 0x5559d8836110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:47:17.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.778+0000 7fef71a00640 1 --2- 172.21.15.90:0/3314953561 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5559d8821350 0x5559d883b450 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7fef60009230 tx=0x7fef6002bc40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8bcc1a245a6296fe server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:47:17.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.778+0000 7fef53400640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:47:17.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.778+0000 7fef53400640 1 -- 172.21.15.90:0/3314953561 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef6002cc20 con 0x5559d8821350 2024-04-02T14:47:17.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.779+0000 7fef53400640 1 -- 172.21.15.90:0/3314953561 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 8550+0+0 (secure 0 0 0) 0x7fef60045040 con 0x5559d8821350 2024-04-02T14:47:17.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.779+0000 7fef77206740 1 -- 172.21.15.90:0/3314953561 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fef6004c070 con 0x5559d8821350 2024-04-02T14:47:17.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.779+0000 7fef53400640 1 -- 172.21.15.90:0/3314953561 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fef60004040 con 0x5559d8821350 2024-04-02T14:47:17.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.779+0000 7fef77206740 10 client.29372 fetch_fsmap learned FSMap version 27 2024-04-02T14:47:17.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.779+0000 7fef77206740 1 -- 172.21.15.90:0/3314953561 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5559d88aa7a0 con 0x5559d8821350 2024-04-02T14:47:17.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.779+0000 7fef53400640 1 -- 172.21.15.90:0/3314953561 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fef6004c070 con 0x5559d8821350 2024-04-02T14:47:17.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.779+0000 7fef53400640 1 -- 172.21.15.90:0/3314953561 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fef600464d0 con 0x5559d8821350 2024-04-02T14:47:17.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.780+0000 7fef77206740 10 client.29372 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:47:17.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.780+0000 7fef77206740 10 client.29372 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:47:17.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.780+0000 7fef77206740 20 client.29372 populate_metadata read hostname 'smithi090' 2024-04-02T14:47:17.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.780+0000 7fef77206740 1 --2- 172.21.15.90:0/3314953561 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5559d88aadb0 0x5559d88ad1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:17.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.780+0000 7fef77206740 4 client.29372 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-02T14:47:17.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.780+0000 7fef77206740 1 -- 172.21.15.90:0/3314953561 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5559d878cba0 con 0x5559d88aadb0 2024-04-02T14:47:17.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.780+0000 7fef71000640 1 --2- 172.21.15.90:0/3314953561 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5559d88aadb0 0x5559d88ad1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:17.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.782+0000 7fef71000640 1 --2- 172.21.15.90:0/3314953561 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5559d88aadb0 0x5559d88ad1a0 crc :-1 s=READY pgs=323 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6c954d8d49d94961 server_cookie=9f81186aaad23f16 in_seq=0 out_seq=0 2024-04-02T14:47:17.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.782+0000 7fef53400640 10 client.29372 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:47:17.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.784+0000 7fef53400640 1 -- 172.21.15.90:0/3314953561 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5559d878cba0 con 0x5559d88aadb0 2024-04-02T14:47:17.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.784+0000 7fef53400640 10 client.29372 handle_command_reply: tid=0 2024-04-02T14:47:17.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.784+0000 7fef77206740 1 client.29372 shutdown 2024-04-02T14:47:17.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.784+0000 7fef77206740 1 -- 172.21.15.90:0/3314953561 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5559d8821350 msgr2=0x5559d883b450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:17.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.784+0000 7fef77206740 1 --2- 172.21.15.90:0/3314953561 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5559d8821350 0x5559d883b450 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7fef60009230 tx=0x7fef6002bc40 comp rx=0 tx=0).stop 2024-04-02T14:47:17.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.784+0000 7fef77206740 1 -- 172.21.15.90:0/3314953561 shutdown_connections 2024-04-02T14:47:17.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.784+0000 7fef77206740 1 --2- 172.21.15.90:0/3314953561 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5559d88aadb0 0x5559d88ad1a0 crc :-1 s=READY pgs=323 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:17.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.784+0000 7fef77206740 1 --2- 172.21.15.90:0/3314953561 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5559d8832ed0 0x5559d8836650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:17.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.784+0000 7fef77206740 1 --2- 172.21.15.90:0/3314953561 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5559d8822030 0x5559d8836110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:17.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.784+0000 7fef77206740 1 --2- 172.21.15.90:0/3314953561 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5559d8821350 0x5559d883b450 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:17.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.784+0000 7fef77206740 1 -- 172.21.15.90:0/3314953561 >> 172.21.15.90:0/3314953561 conn(0x5559d87b5270 msgr2=0x5559d881ae20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:47:17.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.784+0000 7fef53400640 0 client.29372 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:47:17.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.784+0000 7fef77206740 1 -- 172.21.15.90:0/3314953561 shutdown_connections 2024-04-02T14:47:17.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.784+0000 7fef77206740 1 -- 172.21.15.90:0/3314953561 wait complete. 2024-04-02T14:47:17.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.784+0000 7fef77206740 20 client.29372 trim_cache size 0 max 16384 2024-04-02T14:47:17.780 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-02T14:47:17.780 INFO:teuthology.orchestra.run.smithi090.stdout: "return_code": 0, 2024-04-02T14:47:17.780 INFO:teuthology.orchestra.run.smithi090.stdout: "scrub_tag": "bef6617f-ab82-4906-84e9-a8da40685075", 2024-04-02T14:47:17.780 INFO:teuthology.orchestra.run.smithi090.stdout: "mode": "asynchronous" 2024-04-02T14:47:17.780 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-02T14:47:17.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.785+0000 7fef74400640 1 -- 172.21.15.90:0/4273477586 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fef540a3c30 msgr2=0x7fef540a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:17.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.785+0000 7fef74400640 1 --2- 172.21.15.90:0/4273477586 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fef540a3c30 0x7fef540a60f0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fef6401c9e0 tx=0x7fef64000f40 comp rx=0 tx=0).stop 2024-04-02T14:47:17.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.785+0000 7fef74400640 1 -- 172.21.15.90:0/4273477586 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fef6c108fe0 msgr2=0x7fef6c19ab50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:17.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.785+0000 7fef74400640 1 --2- 172.21.15.90:0/4273477586 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fef6c108fe0 0x7fef6c19ab50 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7fef58005fd0 tx=0x7fef5800dde0 comp rx=0 tx=0).stop 2024-04-02T14:47:17.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.785+0000 7fef74400640 1 -- 172.21.15.90:0/4273477586 shutdown_connections 2024-04-02T14:47:17.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.785+0000 7fef74400640 1 --2- 172.21.15.90:0/4273477586 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fef540a3c30 0x7fef540a60f0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:17.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.785+0000 7fef74400640 1 --2- 172.21.15.90:0/4273477586 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fef6c19eed0 0x7fef6c19f380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:17.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.785+0000 7fef74400640 1 --2- 172.21.15.90:0/4273477586 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fef6c108fe0 0x7fef6c19ab50 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:17.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.785+0000 7fef74400640 1 --2- 172.21.15.90:0/4273477586 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fef6c103f10 0x7fef6c19a610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:17.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.785+0000 7fef74400640 1 -- 172.21.15.90:0/4273477586 >> 172.21.15.90:0/4273477586 conn(0x7fef6c0ff750 msgr2=0x7fef6c100b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:47:17.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.785+0000 7fef74400640 1 -- 172.21.15.90:0/4273477586 shutdown_connections 2024-04-02T14:47:17.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:17.785+0000 7fef74400640 1 -- 172.21.15.90:0/4273477586 wait complete. 2024-04-02T14:47:17.800 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-02T14:47:17.800 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-02T14:47:18.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.113+0000 7f6353800640 1 -- 172.21.15.90:0/1175338605 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f634c102450 msgr2=0x7f634c10e960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:18.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.113+0000 7f6353800640 1 --2- 172.21.15.90:0/1175338605 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f634c102450 0x7f634c10e960 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f6338009860 tx=0x7f633802bb70 comp rx=0 tx=0).stop 2024-04-02T14:47:18.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.113+0000 7f6353800640 1 -- 172.21.15.90:0/1175338605 shutdown_connections 2024-04-02T14:47:18.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.113+0000 7f6353800640 1 --2- 172.21.15.90:0/1175338605 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f634c102450 0x7f634c10e960 unknown :-1 s=CLOSED pgs=880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:18.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.113+0000 7f6353800640 1 --2- 172.21.15.90:0/1175338605 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f634c101a90 0x7f634c101f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:18.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.113+0000 7f6353800640 1 --2- 172.21.15.90:0/1175338605 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f634c107aa0 0x7f634c107e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:18.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.113+0000 7f6353800640 1 -- 172.21.15.90:0/1175338605 >> 172.21.15.90:0/1175338605 conn(0x7f634c078f00 msgr2=0x7f634c079310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:47:18.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.114+0000 7f6353800640 1 -- 172.21.15.90:0/1175338605 shutdown_connections 2024-04-02T14:47:18.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.114+0000 7f6353800640 1 -- 172.21.15.90:0/1175338605 wait complete. 2024-04-02T14:47:18.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.115+0000 7f6353800640 1 Processor -- start 2024-04-02T14:47:18.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.116+0000 7f6353800640 1 -- start start 2024-04-02T14:47:18.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.117+0000 7f6353800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f634c101a90 0x7f634c19a540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:18.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.117+0000 7f6353800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f634c102450 0x7f634c19aa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:18.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.117+0000 7f6353800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f634c107aa0 0x7f634c19ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:18.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.117+0000 7f6353800640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f634c111eb0 con 0x7f634c101a90 2024-04-02T14:47:18.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.117+0000 7f6353800640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f634c111d30 con 0x7f634c102450 2024-04-02T14:47:18.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.117+0000 7f6353800640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f634c112030 con 0x7f634c107aa0 2024-04-02T14:47:18.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.117+0000 7f634be00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f634c102450 0x7f634c19aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:18.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.117+0000 7f6351800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f634c107aa0 0x7f634c19ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:18.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.117+0000 7f634be00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f634c102450 0x7f634c19aa80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:53326/0 (socket says 172.21.15.90:53326) 2024-04-02T14:47:18.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.117+0000 7f6350e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f634c101a90 0x7f634c19a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:18.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.117+0000 7f634be00640 1 -- 172.21.15.90:0/2212012607 learned_addr learned my addr 172.21.15.90:0/2212012607 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:47:18.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.118+0000 7f634be00640 1 -- 172.21.15.90:0/2212012607 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f634c107aa0 msgr2=0x7f634c19ee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:18.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.118+0000 7f634be00640 1 --2- 172.21.15.90:0/2212012607 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f634c107aa0 0x7f634c19ee00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:18.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.118+0000 7f634be00640 1 -- 172.21.15.90:0/2212012607 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f634c101a90 msgr2=0x7f634c19a540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:18.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.118+0000 7f634be00640 1 --2- 172.21.15.90:0/2212012607 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f634c101a90 0x7f634c19a540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:18.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.118+0000 7f634be00640 1 -- 172.21.15.90:0/2212012607 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f634c19f530 con 0x7f634c102450 2024-04-02T14:47:18.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.119+0000 7f634be00640 1 --2- 172.21.15.90:0/2212012607 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f634c102450 0x7f634c19aa80 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f633c00e9c0 tx=0x7f633c00ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:47:18.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.120+0000 7f6349600640 1 -- 172.21.15.90:0/2212012607 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f633c00ccc0 con 0x7f634c102450 2024-04-02T14:47:18.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.120+0000 7f6349600640 1 -- 172.21.15.90:0/2212012607 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f633c005500 con 0x7f634c102450 2024-04-02T14:47:18.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.120+0000 7f6349600640 1 -- 172.21.15.90:0/2212012607 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f633c010430 con 0x7f634c102450 2024-04-02T14:47:18.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.120+0000 7f6353800640 1 -- 172.21.15.90:0/2212012607 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f634c19f820 con 0x7f634c102450 2024-04-02T14:47:18.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.120+0000 7f6353800640 1 -- 172.21.15.90:0/2212012607 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f634c1a70b0 con 0x7f634c102450 2024-04-02T14:47:18.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.125+0000 7f635641c740 1 Processor -- start 2024-04-02T14:47:18.120 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.126+0000 7f635641c740 1 -- start start 2024-04-02T14:47:18.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.126+0000 7f6349600640 1 -- 172.21.15.90:0/2212012607 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f633c0105d0 con 0x7f634c102450 2024-04-02T14:47:18.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.126+0000 7f635641c740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5612b82ae7b0 0x5612b82aeb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:18.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.126+0000 7f635641c740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5612b82af0d0 0x5612b82b7770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:18.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.126+0000 7f635641c740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5612b82b7cb0 0x5612b82ba0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:18.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.126+0000 7f635641c740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5612b8212770 con 0x5612b82b7cb0 2024-04-02T14:47:18.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.126+0000 7f635641c740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x5612b81ba700 con 0x5612b82af0d0 2024-04-02T14:47:18.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.126+0000 7f635641c740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5612b820e720 con 0x5612b82ae7b0 2024-04-02T14:47:18.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.127+0000 7f6350e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5612b82af0d0 0x5612b82b7770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:18.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.127+0000 7f6351800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5612b82ae7b0 0x5612b82aeb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:18.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.127+0000 7f6350e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5612b82af0d0 0x5612b82b7770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:53328/0 (socket says 172.21.15.90:53328) 2024-04-02T14:47:18.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.127+0000 7f6350e00640 1 -- 172.21.15.90:0/616122700 learned_addr learned my addr 172.21.15.90:0/616122700 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:47:18.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.127+0000 7f634be00640 1 --2- 172.21.15.90:0/616122700 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5612b82b7cb0 0x5612b82ba0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:18.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.127+0000 7f6350e00640 1 -- 172.21.15.90:0/616122700 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5612b82ae7b0 msgr2=0x5612b82aeb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:47:18.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.127+0000 7f6350e00640 1 --2- 172.21.15.90:0/616122700 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5612b82ae7b0 0x5612b82aeb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:18.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.127+0000 7f6350e00640 1 -- 172.21.15.90:0/616122700 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5612b82b7cb0 msgr2=0x5612b82ba0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:47:18.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.128+0000 7f6350e00640 1 --2- 172.21.15.90:0/616122700 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5612b82b7cb0 0x5612b82ba0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:18.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.128+0000 7f6350e00640 1 -- 172.21.15.90:0/616122700 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5612b7ff07d0 con 0x5612b82af0d0 2024-04-02T14:47:18.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.128+0000 7f6351800640 1 --2- 172.21.15.90:0/616122700 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5612b82ae7b0 0x5612b82aeb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:47:18.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.128+0000 7f634be00640 1 --2- 172.21.15.90:0/616122700 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5612b82b7cb0 0x5612b82ba0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:47:18.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.128+0000 7f6350e00640 1 --2- 172.21.15.90:0/616122700 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5612b82af0d0 0x5612b82b7770 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7f6344009fd0 tx=0x7f63440118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9efff64d7109498a server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:47:18.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.128+0000 7f6342000640 1 -- 172.21.15.90:0/616122700 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f634401e070 con 0x5612b82af0d0 2024-04-02T14:47:18.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.129+0000 7f6342000640 1 -- 172.21.15.90:0/616122700 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6344004030 con 0x5612b82af0d0 2024-04-02T14:47:18.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.129+0000 7f6342000640 1 -- 172.21.15.90:0/616122700 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f634400f780 con 0x5612b82af0d0 2024-04-02T14:47:18.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.130+0000 7f635641c740 1 -- 172.21.15.90:0/616122700 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5612b82af0d0 msgr2=0x5612b82b7770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:18.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.130+0000 7f635641c740 1 --2- 172.21.15.90:0/616122700 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5612b82af0d0 0x5612b82b7770 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7f6344009fd0 tx=0x7f63440118d0 comp rx=0 tx=0).stop 2024-04-02T14:47:18.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.130+0000 7f635641c740 1 -- 172.21.15.90:0/616122700 shutdown_connections 2024-04-02T14:47:18.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.130+0000 7f635641c740 1 --2- 172.21.15.90:0/616122700 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5612b82b7cb0 0x5612b82ba0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:18.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.130+0000 7f635641c740 1 --2- 172.21.15.90:0/616122700 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5612b82af0d0 0x5612b82b7770 unknown :-1 s=CLOSED pgs=882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:18.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.130+0000 7f635641c740 1 --2- 172.21.15.90:0/616122700 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5612b82ae7b0 0x5612b82aeb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:18.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.130+0000 7f635641c740 1 -- 172.21.15.90:0/616122700 >> 172.21.15.90:0/616122700 conn(0x5612b82a8ee0 msgr2=0x5612b82a92c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:47:18.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.130+0000 7f635641c740 1 -- 172.21.15.90:0/616122700 shutdown_connections 2024-04-02T14:47:18.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.130+0000 7f635641c740 1 -- 172.21.15.90:0/616122700 wait complete. 2024-04-02T14:47:18.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.131+0000 7f635641c740 1 -- start start 2024-04-02T14:47:18.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.132+0000 7f635641c740 1 Processor -- start 2024-04-02T14:47:18.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.132+0000 7f635641c740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5612b82ae7b0 0x5612b82b4ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:18.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.132+0000 7f635641c740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5612b82af0d0 0x5612b82b5220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:18.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.132+0000 7f6349600640 1 --2- 172.21.15.90:0/2212012607 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f63300a3e30 0x7f63300a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:18.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.132+0000 7f635641c740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5612b82b7cb0 0x5612b82b5760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:18.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.133+0000 7f635641c740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5612b82c32b0 con 0x5612b82ae7b0 2024-04-02T14:47:18.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.133+0000 7f635641c740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x5612b81cdac0 con 0x5612b82b7cb0 2024-04-02T14:47:18.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.133+0000 7f635641c740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5612b82c3430 con 0x5612b82af0d0 2024-04-02T14:47:18.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.133+0000 7f6349600640 1 -- 172.21.15.90:0/2212012607 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 8550+0+0 (secure 0 0 0) 0x7f633c0d0aa0 con 0x7f634c102450 2024-04-02T14:47:18.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.133+0000 7f635641c740 10 client.0 _ll_register_callbacks cb 0x5612b81d0180 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:47:18.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.133+0000 7f6350e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5612b82af0d0 0x5612b82b5220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:18.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.133+0000 7f6350e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5612b82af0d0 0x5612b82b5220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:59862/0 (socket says 172.21.15.90:59862) 2024-04-02T14:47:18.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.133+0000 7f6350e00640 1 -- 172.21.15.90:0/3723657525 learned_addr learned my addr 172.21.15.90:0/3723657525 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:47:18.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.133+0000 7f6351800640 1 --2- 172.21.15.90:0/3723657525 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5612b82ae7b0 0x5612b82b4ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:18.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.133+0000 7f634be00640 1 --2- 172.21.15.90:0/2212012607 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f63300a3e30 0x7f63300a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:18.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.133+0000 7f6351800640 1 --2- 172.21.15.90:0/3723657525 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5612b82b7cb0 0x5612b82b5760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:18.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.133+0000 7f6350e00640 1 -- 172.21.15.90:0/3723657525 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5612b82b7cb0 msgr2=0x5612b82b5760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:47:18.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.133+0000 7f6350e00640 1 --2- 172.21.15.90:0/3723657525 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5612b82b7cb0 0x5612b82b5760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:18.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.133+0000 7f6350e00640 1 -- 172.21.15.90:0/3723657525 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5612b82ae7b0 msgr2=0x5612b82b4ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:47:18.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.133+0000 7f6350e00640 1 --2- 172.21.15.90:0/3723657525 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5612b82ae7b0 0x5612b82b4ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:18.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.133+0000 7f6350e00640 1 -- 172.21.15.90:0/3723657525 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5612b7f263f0 con 0x5612b82af0d0 2024-04-02T14:47:18.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.133+0000 7f6351800640 1 --2- 172.21.15.90:0/3723657525 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5612b82b7cb0 0x5612b82b5760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:47:18.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.134+0000 7f6350e00640 1 --2- 172.21.15.90:0/3723657525 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5612b82af0d0 0x5612b82b5220 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f6344006010 tx=0x7f6344004410 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ba04b92fb023e3e0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:47:18.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.134+0000 7f6336a00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:47:18.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.134+0000 7f634be00640 1 --2- 172.21.15.90:0/2212012607 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f63300a3e30 0x7f63300a62f0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x5612b82aa070 tx=0x7f633c0d6530 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:47:18.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.134+0000 7f6336a00640 1 -- 172.21.15.90:0/3723657525 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f634401e070 con 0x5612b82af0d0 2024-04-02T14:47:18.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.134+0000 7f6336a00640 1 -- 172.21.15.90:0/3723657525 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 8550+0+0 (secure 0 0 0) 0x7f6344004c50 con 0x5612b82af0d0 2024-04-02T14:47:18.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.134+0000 7f635641c740 1 -- 172.21.15.90:0/3723657525 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6344004030 con 0x5612b82af0d0 2024-04-02T14:47:18.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.134+0000 7f6336a00640 1 -- 172.21.15.90:0/3723657525 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f6344030050 con 0x5612b82af0d0 2024-04-02T14:47:18.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.134+0000 7f635641c740 10 client.29164 fetch_fsmap learned FSMap version 27 2024-04-02T14:47:18.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.135+0000 7f635641c740 1 -- 172.21.15.90:0/3723657525 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5612b7ee60f0 con 0x5612b82af0d0 2024-04-02T14:47:18.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.135+0000 7f6336a00640 1 -- 172.21.15.90:0/3723657525 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6344004030 con 0x5612b82af0d0 2024-04-02T14:47:18.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.135+0000 7f6336a00640 1 -- 172.21.15.90:0/3723657525 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f634402b030 con 0x5612b82af0d0 2024-04-02T14:47:18.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.135+0000 7f635641c740 10 client.29164 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:47:18.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.135+0000 7f635641c740 10 client.29164 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:47:18.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.135+0000 7f635641c740 20 client.29164 populate_metadata read hostname 'smithi090' 2024-04-02T14:47:18.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.135+0000 7f635641c740 1 --2- 172.21.15.90:0/3723657525 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5612b82d03d0 0x5612b82d27c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:18.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.135+0000 7f635641c740 4 client.29164 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:47:18.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.135+0000 7f635641c740 1 -- 172.21.15.90:0/3723657525 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5612b81b3630 con 0x5612b82d03d0 2024-04-02T14:47:18.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.135+0000 7f6351800640 1 --2- 172.21.15.90:0/3723657525 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5612b82d03d0 0x5612b82d27c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:18.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.136+0000 7f6351800640 1 --2- 172.21.15.90:0/3723657525 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5612b82d03d0 0x5612b82d27c0 crc :-1 s=READY pgs=324 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c4a712e256f87588 server_cookie=505b27f05fcaf60b in_seq=0 out_seq=0 2024-04-02T14:47:18.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.136+0000 7f6336a00640 10 client.29164 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:47:18.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.137+0000 7f6336a00640 1 -- 172.21.15.90:0/3723657525 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5612b81b3630 con 0x5612b82d03d0 2024-04-02T14:47:18.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.137+0000 7f6336a00640 10 client.29164 handle_command_reply: tid=0 2024-04-02T14:47:18.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.137+0000 7f635641c740 1 client.29164 shutdown 2024-04-02T14:47:18.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.138+0000 7f635641c740 1 -- 172.21.15.90:0/3723657525 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5612b82af0d0 msgr2=0x5612b82b5220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:18.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.138+0000 7f635641c740 1 --2- 172.21.15.90:0/3723657525 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5612b82af0d0 0x5612b82b5220 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f6344006010 tx=0x7f6344004410 comp rx=0 tx=0).stop 2024-04-02T14:47:18.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.138+0000 7f635641c740 1 -- 172.21.15.90:0/3723657525 shutdown_connections 2024-04-02T14:47:18.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.138+0000 7f635641c740 1 --2- 172.21.15.90:0/3723657525 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5612b82d03d0 0x5612b82d27c0 crc :-1 s=READY pgs=324 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:18.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.138+0000 7f635641c740 1 --2- 172.21.15.90:0/3723657525 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5612b82b7cb0 0x5612b82b5760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:18.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.138+0000 7f6336a00640 0 client.29164 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:47:18.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.138+0000 7f635641c740 1 --2- 172.21.15.90:0/3723657525 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5612b82af0d0 0x5612b82b5220 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:18.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.138+0000 7f635641c740 1 --2- 172.21.15.90:0/3723657525 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5612b82ae7b0 0x5612b82b4ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:18.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.138+0000 7f635641c740 1 -- 172.21.15.90:0/3723657525 >> 172.21.15.90:0/3723657525 conn(0x5612b82a8ee0 msgr2=0x5612b82aac30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:47:18.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.138+0000 7f635641c740 1 -- 172.21.15.90:0/3723657525 shutdown_connections 2024-04-02T14:47:18.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.138+0000 7f635641c740 1 -- 172.21.15.90:0/3723657525 wait complete. 2024-04-02T14:47:18.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.138+0000 7f635641c740 20 client.29164 trim_cache size 0 max 16384 2024-04-02T14:47:18.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.166+0000 7f635641c740 1 Processor -- start 2024-04-02T14:47:18.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.166+0000 7f635641c740 1 -- start start 2024-04-02T14:47:18.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.166+0000 7f635641c740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5612b82a8ee0 0x5612b82b7410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:18.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.166+0000 7f635641c740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5612b82ae7b0 0x5612b82b3c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:18.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.166+0000 7f635641c740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5612b82af0d0 0x5612b82cc1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:18.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.166+0000 7f635641c740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f634c1ab740 con 0x5612b82a8ee0 2024-04-02T14:47:18.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.166+0000 7f635641c740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f634c1ad280 con 0x5612b82ae7b0 2024-04-02T14:47:18.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.166+0000 7f635641c740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f634c1abe10 con 0x5612b82af0d0 2024-04-02T14:47:18.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.166+0000 7f6351800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5612b82a8ee0 0x5612b82b7410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:18.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.166+0000 7f6351800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5612b82a8ee0 0x5612b82b7410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:37248/0 (socket says 172.21.15.90:37248) 2024-04-02T14:47:18.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.166+0000 7f6351800640 1 -- 172.21.15.90:0/3505440077 learned_addr learned my addr 172.21.15.90:0/3505440077 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:47:18.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.166+0000 7f6350e00640 1 --2- 172.21.15.90:0/3505440077 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5612b82ae7b0 0x5612b82b3c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:18.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.167+0000 7f6351800640 1 --2- 172.21.15.90:0/3505440077 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5612b82af0d0 0x5612b82cc1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:18.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.167+0000 7f6351800640 1 -- 172.21.15.90:0/3505440077 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5612b82af0d0 msgr2=0x5612b82cc1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:47:18.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.167+0000 7f6351800640 1 --2- 172.21.15.90:0/3505440077 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5612b82af0d0 0x5612b82cc1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:18.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.167+0000 7f6351800640 1 -- 172.21.15.90:0/3505440077 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5612b82ae7b0 msgr2=0x5612b82b3c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:47:18.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.167+0000 7f6351800640 1 --2- 172.21.15.90:0/3505440077 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5612b82ae7b0 0x5612b82b3c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:18.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.167+0000 7f6351800640 1 -- 172.21.15.90:0/3505440077 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5612b82c3190 con 0x5612b82a8ee0 2024-04-02T14:47:18.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.168+0000 7f6351800640 1 --2- 172.21.15.90:0/3505440077 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5612b82a8ee0 0x5612b82b7410 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f633802bbb0 tx=0x7f6338005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9188e89320cad5d server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:47:18.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.169+0000 7f6341600640 1 -- 172.21.15.90:0/3505440077 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f633804d070 con 0x5612b82a8ee0 2024-04-02T14:47:18.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.169+0000 7f6341600640 1 -- 172.21.15.90:0/3505440077 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6338034da0 con 0x5612b82a8ee0 2024-04-02T14:47:18.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.170+0000 7f6341600640 1 -- 172.21.15.90:0/3505440077 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f633804d070 con 0x5612b82a8ee0 2024-04-02T14:47:18.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.171+0000 7f635641c740 1 -- 172.21.15.90:0/3505440077 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5612b82a8ee0 msgr2=0x5612b82b7410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:18.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.171+0000 7f635641c740 1 --2- 172.21.15.90:0/3505440077 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5612b82a8ee0 0x5612b82b7410 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f633802bbb0 tx=0x7f6338005f90 comp rx=0 tx=0).stop 2024-04-02T14:47:18.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.171+0000 7f635641c740 1 -- 172.21.15.90:0/3505440077 shutdown_connections 2024-04-02T14:47:18.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.171+0000 7f635641c740 1 --2- 172.21.15.90:0/3505440077 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5612b82af0d0 0x5612b82cc1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:18.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.171+0000 7f635641c740 1 --2- 172.21.15.90:0/3505440077 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5612b82ae7b0 0x5612b82b3c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:18.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.171+0000 7f635641c740 1 --2- 172.21.15.90:0/3505440077 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5612b82a8ee0 0x5612b82b7410 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:18.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.171+0000 7f635641c740 1 -- 172.21.15.90:0/3505440077 >> 172.21.15.90:0/3505440077 conn(0x5612b82d2b30 msgr2=0x5612b82b5530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:47:18.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.171+0000 7f635641c740 1 -- 172.21.15.90:0/3505440077 shutdown_connections 2024-04-02T14:47:18.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.171+0000 7f635641c740 1 -- 172.21.15.90:0/3505440077 wait complete. 2024-04-02T14:47:18.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.172+0000 7f635641c740 1 -- start start 2024-04-02T14:47:18.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.172+0000 7f635641c740 1 Processor -- start 2024-04-02T14:47:18.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.172+0000 7f635641c740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5612b82a8ee0 0x5612b82c86b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:18.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.172+0000 7f635641c740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5612b82ae7b0 0x5612b82c8bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:18.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.172+0000 7f635641c740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5612b82af0d0 0x5612b82c5680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:18.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.172+0000 7f635641c740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f634c1acbb0 con 0x5612b82ae7b0 2024-04-02T14:47:18.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.172+0000 7f635641c740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f634c1ac4e0 con 0x5612b82af0d0 2024-04-02T14:47:18.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.172+0000 7f635641c740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f634c1a9760 con 0x5612b82a8ee0 2024-04-02T14:47:18.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.172+0000 7f635641c740 10 client.0 _ll_register_callbacks cb 0x7f634401c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:47:18.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.173+0000 7f6350e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5612b82ae7b0 0x5612b82c8bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:18.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.173+0000 7f6351800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5612b82a8ee0 0x5612b82c86b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:18.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.173+0000 7f6351800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5612b82a8ee0 0x5612b82c86b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:59886/0 (socket says 172.21.15.90:59886) 2024-04-02T14:47:18.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.173+0000 7f6351800640 1 -- 172.21.15.90:0/2804090610 learned_addr learned my addr 172.21.15.90:0/2804090610 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:47:18.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.173+0000 7f6351800640 1 --2- 172.21.15.90:0/2804090610 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5612b82af0d0 0x5612b82c5680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:18.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.173+0000 7f6350e00640 1 -- 172.21.15.90:0/2804090610 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5612b82a8ee0 msgr2=0x5612b82c86b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:47:18.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.173+0000 7f6350e00640 1 --2- 172.21.15.90:0/2804090610 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5612b82a8ee0 0x5612b82c86b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:18.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.173+0000 7f6350e00640 1 -- 172.21.15.90:0/2804090610 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5612b82af0d0 msgr2=0x5612b82c5680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:47:18.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.173+0000 7f6350e00640 1 --2- 172.21.15.90:0/2804090610 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5612b82af0d0 0x5612b82c5680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:18.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.173+0000 7f6350e00640 1 -- 172.21.15.90:0/2804090610 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5612b82c5e80 con 0x5612b82ae7b0 2024-04-02T14:47:18.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.173+0000 7f6351800640 1 --2- 172.21.15.90:0/2804090610 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5612b82af0d0 0x5612b82c5680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:47:18.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.173+0000 7f6350e00640 1 --2- 172.21.15.90:0/2804090610 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5612b82ae7b0 0x5612b82c8bf0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f6344005a30 tx=0x7f63440044f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26937c579f4362a9 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:47:18.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.173+0000 7f6336a00640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-02T14:47:18.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.174+0000 7f6336a00640 1 -- 172.21.15.90:0/2804090610 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6344039060 con 0x5612b82ae7b0 2024-04-02T14:47:18.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.174+0000 7f6336a00640 1 -- 172.21.15.90:0/2804090610 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 8550+0+0 (secure 0 0 0) 0x7f634400f5b0 con 0x5612b82ae7b0 2024-04-02T14:47:18.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.174+0000 7f635641c740 1 -- 172.21.15.90:0/2804090610 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6338034da0 con 0x5612b82ae7b0 2024-04-02T14:47:18.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.174+0000 7f6336a00640 1 -- 172.21.15.90:0/2804090610 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f634401e040 con 0x5612b82ae7b0 2024-04-02T14:47:18.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.174+0000 7f635641c740 10 client.18282 fetch_fsmap learned FSMap version 27 2024-04-02T14:47:18.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.174+0000 7f635641c740 1 -- 172.21.15.90:0/2804090610 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5612b8337200 con 0x5612b82ae7b0 2024-04-02T14:47:18.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.175+0000 7f6336a00640 1 -- 172.21.15.90:0/2804090610 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6338034da0 con 0x5612b82ae7b0 2024-04-02T14:47:18.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.175+0000 7f6336a00640 1 -- 172.21.15.90:0/2804090610 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f634402b070 con 0x5612b82ae7b0 2024-04-02T14:47:18.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.176+0000 7f635641c740 10 client.18282 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:47:18.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.176+0000 7f635641c740 10 client.18282 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:47:18.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.176+0000 7f635641c740 20 client.18282 populate_metadata read hostname 'smithi090' 2024-04-02T14:47:18.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.176+0000 7f635641c740 1 --2- 172.21.15.90:0/2804090610 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5612b82b7cb0 0x5612b83399a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:18.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.177+0000 7f635641c740 4 client.18282 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-02T14:47:18.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.177+0000 7f635641c740 1 -- 172.21.15.90:0/2804090610 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5612b821fcb0 con 0x5612b82b7cb0 2024-04-02T14:47:18.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.177+0000 7f6351800640 1 --2- 172.21.15.90:0/2804090610 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5612b82b7cb0 0x5612b83399a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:18.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.178+0000 7f6351800640 1 --2- 172.21.15.90:0/2804090610 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5612b82b7cb0 0x5612b83399a0 crc :-1 s=READY pgs=325 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b0fdefcc1b1f77b5 server_cookie=972cf755b592e779 in_seq=0 out_seq=0 2024-04-02T14:47:18.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.178+0000 7f6336a00640 10 client.18282 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:47:18.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.178+0000 7f6336a00640 1 -- 172.21.15.90:0/2804090610 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x5612b821fcb0 con 0x5612b82b7cb0 2024-04-02T14:47:18.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.178+0000 7f6336a00640 10 client.18282 handle_command_reply: tid=0 2024-04-02T14:47:18.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.178+0000 7f635641c740 1 client.18282 shutdown 2024-04-02T14:47:18.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.179+0000 7f635641c740 1 -- 172.21.15.90:0/2804090610 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5612b82ae7b0 msgr2=0x5612b82c8bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:18.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.179+0000 7f635641c740 1 --2- 172.21.15.90:0/2804090610 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5612b82ae7b0 0x5612b82c8bf0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f6344005a30 tx=0x7f63440044f0 comp rx=0 tx=0).stop 2024-04-02T14:47:18.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.179+0000 7f635641c740 1 -- 172.21.15.90:0/2804090610 shutdown_connections 2024-04-02T14:47:18.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.179+0000 7f635641c740 1 --2- 172.21.15.90:0/2804090610 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5612b82b7cb0 0x5612b83399a0 crc :-1 s=READY pgs=325 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:18.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.179+0000 7f635641c740 1 --2- 172.21.15.90:0/2804090610 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5612b82af0d0 0x5612b82c5680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:18.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.179+0000 7f635641c740 1 --2- 172.21.15.90:0/2804090610 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5612b82ae7b0 0x5612b82c8bf0 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:18.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.179+0000 7f635641c740 1 --2- 172.21.15.90:0/2804090610 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5612b82a8ee0 0x5612b82c86b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:18.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.179+0000 7f635641c740 1 -- 172.21.15.90:0/2804090610 >> 172.21.15.90:0/2804090610 conn(0x5612b82d2b30 msgr2=0x5612b82a76d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:47:18.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.179+0000 7f6336a00640 0 client.18282 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:47:18.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.179+0000 7f635641c740 1 -- 172.21.15.90:0/2804090610 shutdown_connections 2024-04-02T14:47:18.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.179+0000 7f635641c740 1 -- 172.21.15.90:0/2804090610 wait complete. 2024-04-02T14:47:18.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.179+0000 7f635641c740 20 client.18282 trim_cache size 0 max 16384 2024-04-02T14:47:18.176 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-02T14:47:18.176 INFO:teuthology.orchestra.run.smithi090.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-02T14:47:18.176 INFO:teuthology.orchestra.run.smithi090.stdout: "scrubs": { 2024-04-02T14:47:18.176 INFO:teuthology.orchestra.run.smithi090.stdout: "bef6617f-ab82-4906-84e9-a8da40685075": { 2024-04-02T14:47:18.176 INFO:teuthology.orchestra.run.smithi090.stdout: "path": "/", 2024-04-02T14:47:18.176 INFO:teuthology.orchestra.run.smithi090.stdout: "tag": "bef6617f-ab82-4906-84e9-a8da40685075", 2024-04-02T14:47:18.176 INFO:teuthology.orchestra.run.smithi090.stdout: "options": "recursive,force" 2024-04-02T14:47:18.176 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:47:18.176 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:47:18.176 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-02T14:47:18.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.180+0000 7f6353800640 1 -- 172.21.15.90:0/2212012607 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f63300a3e30 msgr2=0x7f63300a62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:18.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.180+0000 7f6353800640 1 --2- 172.21.15.90:0/2212012607 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f63300a3e30 0x7f63300a62f0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x5612b82aa070 tx=0x7f633c0d6530 comp rx=0 tx=0).stop 2024-04-02T14:47:18.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.180+0000 7f6353800640 1 -- 172.21.15.90:0/2212012607 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f634c102450 msgr2=0x7f634c19aa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:18.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.180+0000 7f6353800640 1 --2- 172.21.15.90:0/2212012607 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f634c102450 0x7f634c19aa80 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f633c00e9c0 tx=0x7f633c00ee80 comp rx=0 tx=0).stop 2024-04-02T14:47:18.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.180+0000 7f6353800640 1 -- 172.21.15.90:0/2212012607 shutdown_connections 2024-04-02T14:47:18.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.180+0000 7f6353800640 1 --2- 172.21.15.90:0/2212012607 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f63300a3e30 0x7f63300a62f0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:18.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.180+0000 7f6353800640 1 --2- 172.21.15.90:0/2212012607 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f634c107aa0 0x7f634c19ee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:18.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.180+0000 7f6353800640 1 --2- 172.21.15.90:0/2212012607 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f634c102450 0x7f634c19aa80 unknown :-1 s=CLOSED pgs=881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:18.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.180+0000 7f6353800640 1 --2- 172.21.15.90:0/2212012607 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f634c101a90 0x7f634c19a540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:18.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.180+0000 7f6353800640 1 -- 172.21.15.90:0/2212012607 >> 172.21.15.90:0/2212012607 conn(0x7f634c078f00 msgr2=0x7f634c10d1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:47:18.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.180+0000 7f6353800640 1 -- 172.21.15.90:0/2212012607 shutdown_connections 2024-04-02T14:47:18.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:18.180+0000 7f6353800640 1 -- 172.21.15.90:0/2212012607 wait complete. 2024-04-02T14:47:18.181 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:47:18 smithi090 ceph-mon[31674]: pgmap v387: 97 pgs: 97 active+clean; 4.1 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 20 MiB/s wr, 1.79k op/s 2024-04-02T14:47:18.181 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:47:18 smithi090 ceph-mon[31674]: scrub queued for path: / 2024-04-02T14:47:18.181 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:47:18 smithi090 ceph-mon[31674]: scrub summary: idle+waiting paths [/] 2024-04-02T14:47:18.181 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:47:18 smithi090 ceph-mon[31674]: scrub summary: active paths [/] 2024-04-02T14:47:18.195 INFO:tasks.cephfs.filesystem:scrub status for tag:bef6617f-ab82-4906-84e9-a8da40685075 - {'path': '/', 'tag': 'bef6617f-ab82-4906-84e9-a8da40685075', 'options': 'recursive,force'} 2024-04-02T14:47:18.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:47:18 smithi177 ceph-mon[36431]: pgmap v387: 97 pgs: 97 active+clean; 4.1 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 20 MiB/s wr, 1.79k op/s 2024-04-02T14:47:18.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:47:18 smithi177 ceph-mon[36431]: scrub queued for path: / 2024-04-02T14:47:18.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:47:18 smithi177 ceph-mon[36431]: scrub summary: idle+waiting paths [/] 2024-04-02T14:47:18.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:47:18 smithi177 ceph-mon[36431]: scrub summary: active paths [/] 2024-04-02T14:47:18.604 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:47:18 smithi092 ceph-mon[36497]: pgmap v387: 97 pgs: 97 active+clean; 4.1 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 20 MiB/s wr, 1.79k op/s 2024-04-02T14:47:18.604 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:47:18 smithi092 ceph-mon[36497]: scrub queued for path: / 2024-04-02T14:47:18.604 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:47:18 smithi092 ceph-mon[36497]: scrub summary: idle+waiting paths [/] 2024-04-02T14:47:18.604 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:47:18 smithi092 ceph-mon[36497]: scrub summary: active paths [/] 2024-04-02T14:47:19.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:47:19 smithi177 ceph-mon[36431]: scrub summary: idle+waiting paths [/] 2024-04-02T14:47:19.431 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:47:19 smithi090 ceph-mon[31674]: scrub summary: idle+waiting paths [/] 2024-04-02T14:47:19.604 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:47:19 smithi092 ceph-mon[36497]: scrub summary: idle+waiting paths [/] 2024-04-02T14:47:20.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:47:20 smithi177 ceph-mon[36431]: pgmap v388: 97 pgs: 97 active+clean; 4.1 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 17 MiB/s wr, 1.50k op/s 2024-04-02T14:47:20.431 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:47:20 smithi090 ceph-mon[31674]: pgmap v388: 97 pgs: 97 active+clean; 4.1 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 17 MiB/s wr, 1.50k op/s 2024-04-02T14:47:20.604 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:47:20 smithi092 ceph-mon[36497]: pgmap v388: 97 pgs: 97 active+clean; 4.1 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 17 MiB/s wr, 1.50k op/s 2024-04-02T14:47:21.824 INFO:teuthology.orchestra.run.smithi090.stderr:progress: 710.0 s, 1383.2 tps, lat 72.076 ms stddev 86.632 2024-04-02T14:47:22.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:47:22 smithi177 ceph-mon[36431]: pgmap v389: 97 pgs: 97 active+clean; 4.1 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 146 KiB/s rd, 5.3 MiB/s wr, 716 op/s 2024-04-02T14:47:22.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:47:22 smithi177 ceph-mon[36431]: scrub summary: idle 2024-04-02T14:47:22.431 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:47:22 smithi090 ceph-mon[31674]: pgmap v389: 97 pgs: 97 active+clean; 4.1 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 146 KiB/s rd, 5.3 MiB/s wr, 716 op/s 2024-04-02T14:47:22.431 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:47:22 smithi090 ceph-mon[31674]: scrub summary: idle 2024-04-02T14:47:22.604 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:47:22 smithi092 ceph-mon[36497]: pgmap v389: 97 pgs: 97 active+clean; 4.1 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 146 KiB/s rd, 5.3 MiB/s wr, 716 op/s 2024-04-02T14:47:22.604 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:47:22 smithi092 ceph-mon[36497]: scrub summary: idle 2024-04-02T14:47:24.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:47:24 smithi177 ceph-mon[36431]: pgmap v390: 97 pgs: 97 active+clean; 4.1 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 146 KiB/s rd, 5.3 MiB/s wr, 716 op/s 2024-04-02T14:47:24.431 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:47:24 smithi090 ceph-mon[31674]: pgmap v390: 97 pgs: 97 active+clean; 4.1 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 146 KiB/s rd, 5.3 MiB/s wr, 716 op/s 2024-04-02T14:47:24.604 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:47:24 smithi092 ceph-mon[36497]: pgmap v390: 97 pgs: 97 active+clean; 4.1 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 146 KiB/s rd, 5.3 MiB/s wr, 716 op/s 2024-04-02T14:47:25.611 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.616+0000 7f7ca3800640 1 -- 172.21.15.90:0/317190873 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7c9c109670 msgr2=0x7f7c9c114710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:25.612 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.616+0000 7f7ca3800640 1 --2- 172.21.15.90:0/317190873 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7c9c109670 0x7f7c9c114710 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f7c88009930 tx=0x7f7c8802bc30 comp rx=0 tx=0).stop 2024-04-02T14:47:25.612 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.616+0000 7f7ca3800640 1 -- 172.21.15.90:0/317190873 shutdown_connections 2024-04-02T14:47:25.612 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.616+0000 7f7ca3800640 1 --2- 172.21.15.90:0/317190873 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7c9c114c50 0x7f7c9c117040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:25.612 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.616+0000 7f7ca3800640 1 --2- 172.21.15.90:0/317190873 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7c9c109670 0x7f7c9c114710 unknown :-1 s=CLOSED pgs=883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:25.612 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.616+0000 7f7ca3800640 1 --2- 172.21.15.90:0/317190873 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7c9c108d50 0x7f7c9c109130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:25.612 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.616+0000 7f7ca3800640 1 -- 172.21.15.90:0/317190873 >> 172.21.15.90:0/317190873 conn(0x7f7c9c06cba0 msgr2=0x7f7c9c06cfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:47:25.613 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.617+0000 7f7ca3800640 1 -- 172.21.15.90:0/317190873 shutdown_connections 2024-04-02T14:47:25.613 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.617+0000 7f7ca3800640 1 -- 172.21.15.90:0/317190873 wait complete. 2024-04-02T14:47:25.614 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.619+0000 7f7ca3800640 1 Processor -- start 2024-04-02T14:47:25.614 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.619+0000 7f7ca3800640 1 -- start start 2024-04-02T14:47:25.615 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.620+0000 7f7ca3800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7c9c108d50 0x7f7c9c11e800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:25.615 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.620+0000 7f7ca3800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7c9c109670 0x7f7c9c11ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:25.615 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.620+0000 7f7ca3800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7c9c114c50 0x7f7c9c124510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:25.615 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.620+0000 7f7ca3800640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f7c9c10c530 con 0x7f7c9c114c50 2024-04-02T14:47:25.616 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.620+0000 7f7ca3800640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f7c9c10c3b0 con 0x7f7c9c108d50 2024-04-02T14:47:25.616 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.620+0000 7f7ca3800640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f7c9c118600 con 0x7f7c9c109670 2024-04-02T14:47:25.616 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.620+0000 7f7c9be00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7c9c109670 0x7f7c9c11ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:25.616 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.620+0000 7f7c9be00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7c9c109670 0x7f7c9c11ed40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:38652/0 (socket says 172.21.15.90:38652) 2024-04-02T14:47:25.616 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.620+0000 7f7c9be00640 1 -- 172.21.15.90:0/2650930364 learned_addr learned my addr 172.21.15.90:0/2650930364 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:47:25.616 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.620+0000 7f7ca0e00640 1 --2- 172.21.15.90:0/2650930364 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7c9c108d50 0x7f7c9c11e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:25.616 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.621+0000 7f7ca1800640 1 --2- 172.21.15.90:0/2650930364 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7c9c114c50 0x7f7c9c124510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:25.616 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.621+0000 7f7ca1800640 1 -- 172.21.15.90:0/2650930364 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7c9c109670 msgr2=0x7f7c9c11ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:25.617 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.621+0000 7f7ca1800640 1 --2- 172.21.15.90:0/2650930364 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7c9c109670 0x7f7c9c11ed40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:25.617 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.621+0000 7f7ca1800640 1 -- 172.21.15.90:0/2650930364 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7c9c108d50 msgr2=0x7f7c9c11e800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:25.617 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.621+0000 7f7ca1800640 1 --2- 172.21.15.90:0/2650930364 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7c9c108d50 0x7f7c9c11e800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:25.617 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.621+0000 7f7ca1800640 1 -- 172.21.15.90:0/2650930364 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c9c124dd0 con 0x7f7c9c114c50 2024-04-02T14:47:25.617 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.621+0000 7f7ca0e00640 1 --2- 172.21.15.90:0/2650930364 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7c9c108d50 0x7f7c9c11e800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:47:25.617 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.622+0000 7f7ca1800640 1 --2- 172.21.15.90:0/2650930364 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7c9c114c50 0x7f7c9c124510 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f7c8c00e970 tx=0x7f7c8c00ee30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:47:25.618 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.622+0000 7f7c99600640 1 -- 172.21.15.90:0/2650930364 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c8c00ccc0 con 0x7f7c9c114c50 2024-04-02T14:47:25.618 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.623+0000 7f7c99600640 1 -- 172.21.15.90:0/2650930364 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7c8c0035f0 con 0x7f7c9c114c50 2024-04-02T14:47:25.618 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.623+0000 7f7ca3800640 1 -- 172.21.15.90:0/2650930364 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c9c119910 con 0x7f7c9c114c50 2024-04-02T14:47:25.618 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.623+0000 7f7c99600640 1 -- 172.21.15.90:0/2650930364 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c8c010690 con 0x7f7c9c114c50 2024-04-02T14:47:25.618 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.623+0000 7f7ca3800640 1 -- 172.21.15.90:0/2650930364 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c9c119eb0 con 0x7f7c9c114c50 2024-04-02T14:47:25.628 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.633+0000 7f7ca643b740 1 Processor -- start 2024-04-02T14:47:25.629 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.633+0000 7f7c99600640 1 -- 172.21.15.90:0/2650930364 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f7c8c010830 con 0x7f7c9c114c50 2024-04-02T14:47:25.629 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.634+0000 7f7ca643b740 1 -- start start 2024-04-02T14:47:25.629 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.634+0000 7f7ca643b740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5640cc266eb0 0x5640cc267290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:25.629 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.634+0000 7f7ca643b740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5640cc2677d0 0x5640cc26fe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:25.629 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.634+0000 7f7ca643b740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5640cc2703b0 0x5640cc2727a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:25.630 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.634+0000 7f7ca643b740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5640cc1cbd70 con 0x5640cc2677d0 2024-04-02T14:47:25.630 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.634+0000 7f7ca643b740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x5640cc1742a0 con 0x5640cc2703b0 2024-04-02T14:47:25.630 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.634+0000 7f7ca643b740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5640cc1c7d20 con 0x5640cc266eb0 2024-04-02T14:47:25.631 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.635+0000 7f7c9be00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5640cc266eb0 0x5640cc267290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:25.631 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.635+0000 7f7c9be00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5640cc266eb0 0x5640cc267290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:38658/0 (socket says 172.21.15.90:38658) 2024-04-02T14:47:25.631 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.635+0000 7f7c9be00640 1 -- 172.21.15.90:0/693679917 learned_addr learned my addr 172.21.15.90:0/693679917 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:47:25.632 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.635+0000 7f7c9be00640 1 --2- 172.21.15.90:0/693679917 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5640cc2703b0 0x5640cc2727a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:25.632 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.636+0000 7f7ca0e00640 1 --2- 172.21.15.90:0/693679917 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5640cc2677d0 0x5640cc26fe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:25.632 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.636+0000 7f7c9be00640 1 -- 172.21.15.90:0/693679917 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5640cc2703b0 msgr2=0x5640cc2727a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:47:25.632 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.636+0000 7f7c9be00640 1 --2- 172.21.15.90:0/693679917 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5640cc2703b0 0x5640cc2727a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:25.632 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.636+0000 7f7c9be00640 1 -- 172.21.15.90:0/693679917 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5640cc2677d0 msgr2=0x5640cc26fe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:47:25.632 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.636+0000 7f7c9be00640 1 --2- 172.21.15.90:0/693679917 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5640cc2677d0 0x5640cc26fe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:25.632 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.636+0000 7f7c9be00640 1 -- 172.21.15.90:0/693679917 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5640cc188840 con 0x5640cc266eb0 2024-04-02T14:47:25.632 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.636+0000 7f7ca0e00640 1 --2- 172.21.15.90:0/693679917 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5640cc2677d0 0x5640cc26fe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:47:25.633 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.637+0000 7f7c9be00640 1 --2- 172.21.15.90:0/693679917 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5640cc266eb0 0x5640cc267290 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f7c88005a90 tx=0x7f7c88005a20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2c8c950b50e8a8d2 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:47:25.633 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.637+0000 7f7c99600640 1 --2- 172.21.15.90:0/2650930364 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f7c800a3f10 0x7f7c800a63d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:25.633 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.637+0000 7f7c99600640 1 -- 172.21.15.90:0/2650930364 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 8550+0+0 (secure 0 0 0) 0x7f7c8c0d0860 con 0x7f7c9c114c50 2024-04-02T14:47:25.633 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.637+0000 7f7c92000640 1 -- 172.21.15.90:0/693679917 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c8802c880 con 0x5640cc266eb0 2024-04-02T14:47:25.633 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.637+0000 7f7c92000640 1 -- 172.21.15.90:0/693679917 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7c88004ba0 con 0x5640cc266eb0 2024-04-02T14:47:25.633 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.637+0000 7f7c92000640 1 -- 172.21.15.90:0/693679917 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c880039e0 con 0x5640cc266eb0 2024-04-02T14:47:25.634 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.639+0000 7f7ca0e00640 1 --2- 172.21.15.90:0/2650930364 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f7c800a3f10 0x7f7c800a63d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:25.634 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.639+0000 7f7ca643b740 1 -- 172.21.15.90:0/693679917 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5640cc266eb0 msgr2=0x5640cc267290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:25.634 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.639+0000 7f7ca643b740 1 --2- 172.21.15.90:0/693679917 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5640cc266eb0 0x5640cc267290 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f7c88005a90 tx=0x7f7c88005a20 comp rx=0 tx=0).stop 2024-04-02T14:47:25.637 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.641+0000 7f7ca0e00640 1 --2- 172.21.15.90:0/2650930364 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f7c800a3f10 0x7f7c800a63d0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f7c94007fd0 tx=0x7f7c940075b0 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:47:25.637 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.641+0000 7f7ca643b740 1 -- 172.21.15.90:0/693679917 shutdown_connections 2024-04-02T14:47:25.637 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.641+0000 7f7ca643b740 1 --2- 172.21.15.90:0/693679917 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5640cc2703b0 0x5640cc2727a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:25.637 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.641+0000 7f7ca643b740 1 --2- 172.21.15.90:0/693679917 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5640cc2677d0 0x5640cc26fe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:25.637 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.641+0000 7f7ca643b740 1 --2- 172.21.15.90:0/693679917 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5640cc266eb0 0x5640cc267290 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:25.637 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.641+0000 7f7ca643b740 1 -- 172.21.15.90:0/693679917 >> 172.21.15.90:0/693679917 conn(0x5640cc2615e0 msgr2=0x5640cc2619c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:47:25.637 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.642+0000 7f7ca643b740 1 -- 172.21.15.90:0/693679917 shutdown_connections 2024-04-02T14:47:25.637 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.642+0000 7f7ca643b740 1 -- 172.21.15.90:0/693679917 wait complete. 2024-04-02T14:47:25.639 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.643+0000 7f7ca643b740 1 -- start start 2024-04-02T14:47:25.639 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.644+0000 7f7ca643b740 1 Processor -- start 2024-04-02T14:47:25.641 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.645+0000 7f7ca643b740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5640cc266eb0 0x5640cc26d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:25.641 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.645+0000 7f7ca643b740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5640cc2677d0 0x5640cc26d900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:25.641 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.645+0000 7f7ca643b740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5640cc2703b0 0x5640cc26de40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:25.641 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.645+0000 7f7ca643b740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5640cc273140 con 0x5640cc2703b0 2024-04-02T14:47:25.641 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.645+0000 7f7ca643b740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x5640cc187660 con 0x5640cc2677d0 2024-04-02T14:47:25.641 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.645+0000 7f7ca643b740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5640cc2732c0 con 0x5640cc266eb0 2024-04-02T14:47:25.641 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.645+0000 7f7ca643b740 10 client.0 _ll_register_callbacks cb 0x5640cc187050 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:47:25.642 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.646+0000 7f7c9be00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5640cc2677d0 0x5640cc26d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:25.642 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.646+0000 7f7c9be00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5640cc2677d0 0x5640cc26d900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:47402/0 (socket says 172.21.15.90:47402) 2024-04-02T14:47:25.642 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.646+0000 7f7c9be00640 1 -- 172.21.15.90:0/4022989280 learned_addr learned my addr 172.21.15.90:0/4022989280 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:47:25.642 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.647+0000 7f7ca1800640 1 --2- 172.21.15.90:0/4022989280 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5640cc2703b0 0x5640cc26de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:25.642 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.647+0000 7f7c9be00640 1 -- 172.21.15.90:0/4022989280 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5640cc266eb0 msgr2=0x5640cc26d3c0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-02T14:47:25.642 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.647+0000 7f7c9be00640 1 --2- 172.21.15.90:0/4022989280 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5640cc266eb0 0x5640cc26d3c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:25.643 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.647+0000 7f7c9be00640 1 -- 172.21.15.90:0/4022989280 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5640cc2703b0 msgr2=0x5640cc26de40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:47:25.643 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.647+0000 7f7c9be00640 1 --2- 172.21.15.90:0/4022989280 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5640cc2703b0 0x5640cc26de40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:25.643 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.647+0000 7f7c9be00640 1 -- 172.21.15.90:0/4022989280 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5640cc196e80 con 0x5640cc2677d0 2024-04-02T14:47:25.643 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.647+0000 7f7c9be00640 1 --2- 172.21.15.90:0/4022989280 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5640cc2677d0 0x5640cc26d900 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f7c880048f0 tx=0x7f7c88004dc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6acd3a17640e8717 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:47:25.643 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.647+0000 7f7c86a00640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:47:25.643 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.648+0000 7f7c86a00640 1 -- 172.21.15.90:0/4022989280 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c8802cc90 con 0x5640cc2677d0 2024-04-02T14:47:25.644 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.648+0000 7f7c86a00640 1 -- 172.21.15.90:0/4022989280 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 8550+0+0 (secure 0 0 0) 0x7f7c88003760 con 0x5640cc2677d0 2024-04-02T14:47:25.644 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.648+0000 7f7ca643b740 1 -- 172.21.15.90:0/4022989280 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7c88004ba0 con 0x5640cc2677d0 2024-04-02T14:47:25.644 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.649+0000 7f7c86a00640 1 -- 172.21.15.90:0/4022989280 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f7c8805e050 con 0x5640cc2677d0 2024-04-02T14:47:25.644 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.649+0000 7f7ca643b740 10 client.29399 fetch_fsmap learned FSMap version 27 2024-04-02T14:47:25.644 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.649+0000 7f7ca643b740 1 -- 172.21.15.90:0/4022989280 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5640cbee0210 con 0x5640cc2677d0 2024-04-02T14:47:25.645 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.650+0000 7f7c86a00640 1 -- 172.21.15.90:0/4022989280 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7c88004ba0 con 0x5640cc2677d0 2024-04-02T14:47:25.645 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.650+0000 7f7c86a00640 1 -- 172.21.15.90:0/4022989280 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f7c88005360 con 0x5640cc2677d0 2024-04-02T14:47:25.645 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.650+0000 7f7ca643b740 10 client.29399 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:47:25.645 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.650+0000 7f7ca643b740 10 client.29399 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-02T14:47:25.645 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.650+0000 7f7ca643b740 20 client.29399 populate_metadata read hostname 'smithi090' 2024-04-02T14:47:25.645 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.650+0000 7f7ca643b740 1 --2- 172.21.15.90:0/4022989280 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5640cc280820 0x5640cc282c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:25.645 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.650+0000 7f7ca643b740 4 client.29399 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:47:25.645 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.650+0000 7f7ca643b740 1 -- 172.21.15.90:0/4022989280 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5640cc17bcd0 con 0x5640cc280820 2024-04-02T14:47:25.646 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.651+0000 7f7ca0e00640 1 --2- 172.21.15.90:0/4022989280 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5640cc280820 0x5640cc282c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:25.646 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.651+0000 7f7c86a00640 10 client.29399 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:47:25.646 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.651+0000 7f7ca0e00640 1 --2- 172.21.15.90:0/4022989280 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5640cc280820 0x5640cc282c80 crc :-1 s=READY pgs=326 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=60929ce9f0ae834c server_cookie=ee82f4c87521223f in_seq=0 out_seq=0 2024-04-02T14:47:25.647 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.652+0000 7f7c86a00640 1 -- 172.21.15.90:0/4022989280 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5640cc17bcd0 con 0x5640cc280820 2024-04-02T14:47:25.647 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.652+0000 7f7c86a00640 10 client.29399 handle_command_reply: tid=0 2024-04-02T14:47:25.647 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.652+0000 7f7ca643b740 1 client.29399 shutdown 2024-04-02T14:47:25.647 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.652+0000 7f7ca643b740 1 -- 172.21.15.90:0/4022989280 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5640cc2677d0 msgr2=0x5640cc26d900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:25.648 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.652+0000 7f7ca643b740 1 --2- 172.21.15.90:0/4022989280 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5640cc2677d0 0x5640cc26d900 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f7c880048f0 tx=0x7f7c88004dc0 comp rx=0 tx=0).stop 2024-04-02T14:47:25.648 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.653+0000 7f7ca643b740 1 -- 172.21.15.90:0/4022989280 shutdown_connections 2024-04-02T14:47:25.648 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.653+0000 7f7ca643b740 1 --2- 172.21.15.90:0/4022989280 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5640cc280820 0x5640cc282c80 crc :-1 s=READY pgs=326 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:25.648 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.653+0000 7f7ca643b740 1 --2- 172.21.15.90:0/4022989280 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5640cc2703b0 0x5640cc26de40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:25.648 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.653+0000 7f7ca643b740 1 --2- 172.21.15.90:0/4022989280 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5640cc2677d0 0x5640cc26d900 unknown :-1 s=CLOSED pgs=884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:25.648 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.653+0000 7f7ca643b740 1 --2- 172.21.15.90:0/4022989280 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5640cc266eb0 0x5640cc26d3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:25.648 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.653+0000 7f7ca643b740 1 -- 172.21.15.90:0/4022989280 >> 172.21.15.90:0/4022989280 conn(0x5640cc2615e0 msgr2=0x5640cc263330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:47:25.648 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.653+0000 7f7c86a00640 0 client.29399 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:47:25.648 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.653+0000 7f7ca643b740 1 -- 172.21.15.90:0/4022989280 shutdown_connections 2024-04-02T14:47:25.649 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.654+0000 7f7ca643b740 1 -- 172.21.15.90:0/4022989280 wait complete. 2024-04-02T14:47:25.649 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.654+0000 7f7ca643b740 20 client.29399 trim_cache size 0 max 16384 2024-04-02T14:47:25.677 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.681+0000 7f7ca643b740 1 Processor -- start 2024-04-02T14:47:25.677 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.682+0000 7f7ca643b740 1 -- start start 2024-04-02T14:47:25.677 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.682+0000 7f7ca643b740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5640cc266eb0 0x5640cc26faf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:25.678 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.682+0000 7f7ca643b740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5640cc2677d0 0x5640cc27c090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:25.678 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.682+0000 7f7ca643b740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5640cc2703b0 0x5640cc26dc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:25.678 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.682+0000 7f7ca643b740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f7c7c00e8a0 con 0x5640cc2677d0 2024-04-02T14:47:25.678 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.682+0000 7f7ca643b740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f7c7c0103e0 con 0x5640cc266eb0 2024-04-02T14:47:25.678 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.682+0000 7f7ca643b740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f7c7c00ef70 con 0x5640cc2703b0 2024-04-02T14:47:25.678 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.682+0000 7f7c9be00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5640cc2677d0 0x5640cc27c090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:25.678 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.682+0000 7f7ca0e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5640cc266eb0 0x5640cc26faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:25.678 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.682+0000 7f7ca0e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5640cc266eb0 0x5640cc26faf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:47408/0 (socket says 172.21.15.90:47408) 2024-04-02T14:47:25.678 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.682+0000 7f7c9be00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5640cc2677d0 0x5640cc27c090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:42798/0 (socket says 172.21.15.90:42798) 2024-04-02T14:47:25.678 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.682+0000 7f7ca0e00640 1 -- 172.21.15.90:0/3843794293 learned_addr learned my addr 172.21.15.90:0/3843794293 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:47:25.678 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.683+0000 7f7ca1800640 1 --2- 172.21.15.90:0/3843794293 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5640cc2703b0 0x5640cc26dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:25.679 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.683+0000 7f7ca0e00640 1 -- 172.21.15.90:0/3843794293 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5640cc2703b0 msgr2=0x5640cc26dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:47:25.679 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.683+0000 7f7ca0e00640 1 --2- 172.21.15.90:0/3843794293 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5640cc2703b0 0x5640cc26dc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:25.679 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.683+0000 7f7ca0e00640 1 -- 172.21.15.90:0/3843794293 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5640cc2677d0 msgr2=0x5640cc27c090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:47:25.679 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.683+0000 7f7ca0e00640 1 --2- 172.21.15.90:0/3843794293 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5640cc2677d0 0x5640cc27c090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:25.679 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.683+0000 7f7ca0e00640 1 -- 172.21.15.90:0/3843794293 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5640cbfac620 con 0x5640cc266eb0 2024-04-02T14:47:25.679 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.683+0000 7f7c9be00640 1 --2- 172.21.15.90:0/3843794293 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5640cc2677d0 0x5640cc27c090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:47:25.680 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.683+0000 7f7ca0e00640 1 --2- 172.21.15.90:0/3843794293 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5640cc266eb0 0x5640cc26faf0 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x5640cc283500 tx=0x7f7c94013000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1ed8cea2dc698569 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:47:25.680 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.683+0000 7f7c91600640 1 -- 172.21.15.90:0/3843794293 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c9402b030 con 0x5640cc266eb0 2024-04-02T14:47:25.680 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.684+0000 7f7c91600640 1 -- 172.21.15.90:0/3843794293 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7c9400edc0 con 0x5640cc266eb0 2024-04-02T14:47:25.680 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.684+0000 7f7c91600640 1 -- 172.21.15.90:0/3843794293 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c94017070 con 0x5640cc266eb0 2024-04-02T14:47:25.680 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.684+0000 7f7ca643b740 1 -- 172.21.15.90:0/3843794293 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5640cc266eb0 msgr2=0x5640cc26faf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:25.681 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.684+0000 7f7ca643b740 1 --2- 172.21.15.90:0/3843794293 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5640cc266eb0 0x5640cc26faf0 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x5640cc283500 tx=0x7f7c94013000 comp rx=0 tx=0).stop 2024-04-02T14:47:25.681 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.685+0000 7f7ca643b740 1 -- 172.21.15.90:0/3843794293 shutdown_connections 2024-04-02T14:47:25.681 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.685+0000 7f7ca643b740 1 --2- 172.21.15.90:0/3843794293 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5640cc2703b0 0x5640cc26dc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:25.681 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.685+0000 7f7ca643b740 1 --2- 172.21.15.90:0/3843794293 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5640cc2677d0 0x5640cc27c090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:25.681 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.685+0000 7f7ca643b740 1 --2- 172.21.15.90:0/3843794293 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5640cc266eb0 0x5640cc26faf0 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:25.681 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.685+0000 7f7ca643b740 1 -- 172.21.15.90:0/3843794293 >> 172.21.15.90:0/3843794293 conn(0x5640cc2615e0 msgr2=0x5640cc267bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:47:25.681 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.685+0000 7f7ca643b740 1 -- 172.21.15.90:0/3843794293 shutdown_connections 2024-04-02T14:47:25.681 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.685+0000 7f7ca643b740 1 -- 172.21.15.90:0/3843794293 wait complete. 2024-04-02T14:47:25.682 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.686+0000 7f7ca643b740 1 -- start start 2024-04-02T14:47:25.682 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.686+0000 7f7ca643b740 1 Processor -- start 2024-04-02T14:47:25.685 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.690+0000 7f7ca643b740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5640cc266eb0 0x5640cc2785a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:25.686 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.690+0000 7f7ca643b740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5640cc2677d0 0x5640cc278ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:25.686 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.690+0000 7f7ca643b740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5640cc2703b0 0x5640cc2697f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:25.686 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.691+0000 7f7ca643b740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f7c7c00fd10 con 0x5640cc2677d0 2024-04-02T14:47:25.687 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.691+0000 7f7ca643b740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f7c7c00f640 con 0x5640cc266eb0 2024-04-02T14:47:25.687 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.691+0000 7f7ca643b740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f7c7c00c8a0 con 0x5640cc2703b0 2024-04-02T14:47:25.687 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.691+0000 7f7ca643b740 10 client.0 _ll_register_callbacks cb 0x7f7c88004580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:47:25.687 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.691+0000 7f7ca1800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5640cc2703b0 0x5640cc2697f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:25.687 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.691+0000 7f7ca1800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5640cc2703b0 0x5640cc2697f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:38684/0 (socket says 172.21.15.90:38684) 2024-04-02T14:47:25.687 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.691+0000 7f7ca1800640 1 -- 172.21.15.90:0/2033407150 learned_addr learned my addr 172.21.15.90:0/2033407150 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:47:25.687 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.692+0000 7f7ca1800640 1 -- 172.21.15.90:0/2033407150 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5640cc266eb0 msgr2=0x5640cc2785a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:47:25.687 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.692+0000 7f7ca1800640 1 --2- 172.21.15.90:0/2033407150 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5640cc266eb0 0x5640cc2785a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:25.688 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.692+0000 7f7ca1800640 1 -- 172.21.15.90:0/2033407150 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5640cc2677d0 msgr2=0x5640cc278ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:47:25.688 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.692+0000 7f7ca1800640 1 --2- 172.21.15.90:0/2033407150 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5640cc2677d0 0x5640cc278ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:25.688 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.692+0000 7f7ca1800640 1 -- 172.21.15.90:0/2033407150 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5640cc269fc0 con 0x5640cc2703b0 2024-04-02T14:47:25.688 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.692+0000 7f7ca1800640 1 --2- 172.21.15.90:0/2033407150 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5640cc2703b0 0x5640cc2697f0 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f7c8c0cdfd0 tx=0x7f7c8c09e0b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9c2ea6fa84f0dfa8 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:47:25.688 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.693+0000 7f7c86a00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:47:25.692 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.696+0000 7f7c86a00640 1 -- 172.21.15.90:0/2033407150 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c8c0bb060 con 0x5640cc2703b0 2024-04-02T14:47:25.692 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.696+0000 7f7c86a00640 1 -- 172.21.15.90:0/2033407150 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 8550+0+0 (secure 0 0 0) 0x7f7c8c09ebb0 con 0x5640cc2703b0 2024-04-02T14:47:25.692 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.697+0000 7f7ca643b740 1 -- 172.21.15.90:0/2033407150 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7c9400edc0 con 0x5640cc2703b0 2024-04-02T14:47:25.693 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.698+0000 7f7c86a00640 1 -- 172.21.15.90:0/2033407150 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f7c8c09edb0 con 0x5640cc2703b0 2024-04-02T14:47:25.695 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.700+0000 7f7ca643b740 10 client.29185 fetch_fsmap learned FSMap version 27 2024-04-02T14:47:25.695 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.700+0000 7f7ca643b740 1 -- 172.21.15.90:0/2033407150 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5640cc2e6930 con 0x5640cc2703b0 2024-04-02T14:47:25.696 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.701+0000 7f7c86a00640 1 -- 172.21.15.90:0/2033407150 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7c9400edc0 con 0x5640cc2703b0 2024-04-02T14:47:25.696 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.701+0000 7f7c86a00640 1 -- 172.21.15.90:0/2033407150 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f7c8c0995d0 con 0x5640cc2703b0 2024-04-02T14:47:25.696 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.701+0000 7f7ca643b740 10 client.29185 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:47:25.696 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.701+0000 7f7ca643b740 10 client.29185 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-02T14:47:25.697 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.701+0000 7f7ca643b740 20 client.29185 populate_metadata read hostname 'smithi090' 2024-04-02T14:47:25.697 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.701+0000 7f7ca643b740 1 --2- 172.21.15.90:0/2033407150 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5640cc2e6fe0 0x5640cc2e9440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:25.697 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.701+0000 7f7ca643b740 4 client.29185 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-02T14:47:25.697 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.701+0000 7f7ca643b740 1 -- 172.21.15.90:0/2033407150 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5640cc1d92b0 con 0x5640cc2e6fe0 2024-04-02T14:47:25.701 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.705+0000 7f7ca0e00640 1 --2- 172.21.15.90:0/2033407150 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5640cc2e6fe0 0x5640cc2e9440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:25.702 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.707+0000 7f7ca0e00640 1 --2- 172.21.15.90:0/2033407150 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5640cc2e6fe0 0x5640cc2e9440 crc :-1 s=READY pgs=327 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=aecc612c1cd05b35 server_cookie=5cf6638574277b0b in_seq=0 out_seq=0 2024-04-02T14:47:25.703 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.707+0000 7f7c86a00640 10 client.29185 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:47:25.763 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.767+0000 7f7c86a00640 1 -- 172.21.15.90:0/2033407150 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5640cc1d92b0 con 0x5640cc2e6fe0 2024-04-02T14:47:25.763 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.767+0000 7f7c86a00640 10 client.29185 handle_command_reply: tid=0 2024-04-02T14:47:25.763 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.767+0000 7f7ca643b740 1 client.29185 shutdown 2024-04-02T14:47:25.763 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.767+0000 7f7ca643b740 1 -- 172.21.15.90:0/2033407150 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5640cc2703b0 msgr2=0x5640cc2697f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:25.763 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.767+0000 7f7ca643b740 1 --2- 172.21.15.90:0/2033407150 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5640cc2703b0 0x5640cc2697f0 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f7c8c0cdfd0 tx=0x7f7c8c09e0b0 comp rx=0 tx=0).stop 2024-04-02T14:47:25.763 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.767+0000 7f7ca643b740 1 -- 172.21.15.90:0/2033407150 shutdown_connections 2024-04-02T14:47:25.763 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.767+0000 7f7ca643b740 1 --2- 172.21.15.90:0/2033407150 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5640cc2e6fe0 0x5640cc2e9440 crc :-1 s=READY pgs=327 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:25.763 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.767+0000 7f7c86a00640 0 client.29185 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:47:25.763 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.767+0000 7f7ca643b740 1 --2- 172.21.15.90:0/2033407150 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5640cc2703b0 0x5640cc2697f0 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:25.763 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.767+0000 7f7ca643b740 1 --2- 172.21.15.90:0/2033407150 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5640cc2677d0 0x5640cc278ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:25.763 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.767+0000 7f7ca643b740 1 --2- 172.21.15.90:0/2033407150 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5640cc266eb0 0x5640cc2785a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:25.764 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.767+0000 7f7ca643b740 1 -- 172.21.15.90:0/2033407150 >> 172.21.15.90:0/2033407150 conn(0x5640cc2615e0 msgr2=0x5640cc262930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:47:25.764 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.767+0000 7f7ca643b740 1 -- 172.21.15.90:0/2033407150 shutdown_connections 2024-04-02T14:47:25.764 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.767+0000 7f7ca643b740 1 -- 172.21.15.90:0/2033407150 wait complete. 2024-04-02T14:47:25.764 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.768+0000 7f7ca643b740 20 client.29185 trim_cache size 0 max 16384 2024-04-02T14:47:25.764 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:{ 2024-04-02T14:47:25.764 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "message": "", 2024-04-02T14:47:25.764 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "return_code": 0 2024-04-02T14:47:25.764 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:} 2024-04-02T14:47:25.765 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.768+0000 7f7ca3800640 1 -- 172.21.15.90:0/2650930364 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f7c800a3f10 msgr2=0x7f7c800a63d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:25.765 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.768+0000 7f7ca3800640 1 --2- 172.21.15.90:0/2650930364 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f7c800a3f10 0x7f7c800a63d0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f7c94007fd0 tx=0x7f7c940075b0 comp rx=0 tx=0).stop 2024-04-02T14:47:25.765 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.768+0000 7f7ca3800640 1 -- 172.21.15.90:0/2650930364 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7c9c114c50 msgr2=0x7f7c9c124510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:25.765 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.768+0000 7f7ca3800640 1 --2- 172.21.15.90:0/2650930364 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7c9c114c50 0x7f7c9c124510 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f7c8c00e970 tx=0x7f7c8c00ee30 comp rx=0 tx=0).stop 2024-04-02T14:47:25.765 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.768+0000 7f7ca3800640 1 -- 172.21.15.90:0/2650930364 shutdown_connections 2024-04-02T14:47:25.765 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.768+0000 7f7ca3800640 1 --2- 172.21.15.90:0/2650930364 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f7c800a3f10 0x7f7c800a63d0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:25.765 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.768+0000 7f7ca3800640 1 --2- 172.21.15.90:0/2650930364 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7c9c114c50 0x7f7c9c124510 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:25.765 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.768+0000 7f7ca3800640 1 --2- 172.21.15.90:0/2650930364 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7c9c109670 0x7f7c9c11ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:25.766 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.768+0000 7f7ca3800640 1 --2- 172.21.15.90:0/2650930364 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7c9c108d50 0x7f7c9c11e800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:25.766 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.768+0000 7f7ca3800640 1 -- 172.21.15.90:0/2650930364 >> 172.21.15.90:0/2650930364 conn(0x7f7c9c06cba0 msgr2=0x7f7c9c0706b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:47:25.766 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.769+0000 7f7ca3800640 1 -- 172.21.15.90:0/2650930364 shutdown_connections 2024-04-02T14:47:25.766 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:25.769+0000 7f7ca3800640 1 -- 172.21.15.90:0/2650930364 wait complete. 2024-04-02T14:47:26.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:47:26 smithi177 ceph-mon[36431]: pgmap v391: 97 pgs: 97 active+clean; 4.1 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 156 KiB/s rd, 5.5 MiB/s wr, 713 op/s 2024-04-02T14:47:26.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:47:26 smithi177 ceph-mon[36431]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T14:47:26.431 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:47:26 smithi090 ceph-mon[31674]: pgmap v391: 97 pgs: 97 active+clean; 4.1 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 156 KiB/s rd, 5.5 MiB/s wr, 713 op/s 2024-04-02T14:47:26.431 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:47:26 smithi090 ceph-mon[31674]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T14:47:26.604 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:47:26 smithi092 ceph-mon[36497]: pgmap v391: 97 pgs: 97 active+clean; 4.1 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 156 KiB/s rd, 5.5 MiB/s wr, 713 op/s 2024-04-02T14:47:26.604 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:47:26 smithi092 ceph-mon[36497]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T14:47:26.824 INFO:teuthology.orchestra.run.smithi090.stderr:progress: 715.0 s, 1643.6 tps, lat 61.738 ms stddev 68.899 2024-04-02T14:47:27.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:47:27 smithi177 ceph-mon[36431]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:47:27.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:47:27 smithi177 ceph-mon[36431]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T14:47:27.430 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:47:27 smithi177 ceph-mon[36431]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' 2024-04-02T14:47:27.431 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:47:27 smithi090 ceph-mon[31674]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:47:27.431 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:47:27 smithi090 ceph-mon[31674]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T14:47:27.431 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:47:27 smithi090 ceph-mon[31674]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' 2024-04-02T14:47:27.604 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:47:27 smithi092 ceph-mon[36497]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:47:27.604 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:47:27 smithi092 ceph-mon[36497]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T14:47:27.604 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:47:27 smithi092 ceph-mon[36497]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' 2024-04-02T14:47:28.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:47:28 smithi177 ceph-mon[36431]: pgmap v392: 97 pgs: 97 active+clean; 4.1 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 4.8 MiB/s wr, 621 op/s 2024-04-02T14:47:28.431 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:47:28 smithi090 ceph-mon[31674]: pgmap v392: 97 pgs: 97 active+clean; 4.1 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 4.8 MiB/s wr, 621 op/s 2024-04-02T14:47:28.603 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:47:28 smithi092 ceph-mon[36497]: pgmap v392: 97 pgs: 97 active+clean; 4.1 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 4.8 MiB/s wr, 621 op/s 2024-04-02T14:47:30.432 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:47:30 smithi090 ceph-mon[31674]: pgmap v393: 97 pgs: 97 active+clean; 4.1 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 146 KiB/s rd, 5.2 MiB/s wr, 642 op/s 2024-04-02T14:47:30.604 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:47:30 smithi092 ceph-mon[36497]: pgmap v393: 97 pgs: 97 active+clean; 4.1 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 146 KiB/s rd, 5.2 MiB/s wr, 642 op/s 2024-04-02T14:47:30.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:47:30 smithi177 ceph-mon[36431]: pgmap v393: 97 pgs: 97 active+clean; 4.1 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 146 KiB/s rd, 5.2 MiB/s wr, 642 op/s 2024-04-02T14:47:31.824 INFO:teuthology.orchestra.run.smithi090.stderr:progress: 720.0 s, 2096.6 tps, lat 47.839 ms stddev 64.562 2024-04-02T14:47:32.603 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:47:32 smithi092 ceph-mon[36497]: pgmap v394: 97 pgs: 97 active+clean; 4.2 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 7.9 MiB/s wr, 888 op/s 2024-04-02T14:47:32.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:47:32 smithi177 ceph-mon[36431]: pgmap v394: 97 pgs: 97 active+clean; 4.2 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 7.9 MiB/s wr, 888 op/s 2024-04-02T14:47:32.681 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:47:32 smithi090 ceph-mon[31674]: pgmap v394: 97 pgs: 97 active+clean; 4.2 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 7.9 MiB/s wr, 888 op/s 2024-04-02T14:47:34.603 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:47:34 smithi092 ceph-mon[36497]: pgmap v395: 97 pgs: 97 active+clean; 4.2 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 5.9 MiB/s wr, 587 op/s 2024-04-02T14:47:34.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:47:34 smithi177 ceph-mon[36431]: pgmap v395: 97 pgs: 97 active+clean; 4.2 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 5.9 MiB/s wr, 587 op/s 2024-04-02T14:47:34.681 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:47:34 smithi090 ceph-mon[31674]: pgmap v395: 97 pgs: 97 active+clean; 4.2 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 5.9 MiB/s wr, 587 op/s 2024-04-02T14:47:36.603 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:47:36 smithi092 ceph-mon[36497]: pgmap v396: 97 pgs: 97 active+clean; 4.2 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 8.7 MiB/s wr, 874 op/s 2024-04-02T14:47:36.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:47:36 smithi177 ceph-mon[36431]: pgmap v396: 97 pgs: 97 active+clean; 4.2 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 8.7 MiB/s wr, 874 op/s 2024-04-02T14:47:36.681 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:47:36 smithi090 ceph-mon[31674]: pgmap v396: 97 pgs: 97 active+clean; 4.2 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 8.7 MiB/s wr, 874 op/s 2024-04-02T14:47:36.825 INFO:teuthology.orchestra.run.smithi090.stderr:progress: 725.0 s, 2254.5 tps, lat 44.121 ms stddev 56.840 2024-04-02T14:47:38.603 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:47:38 smithi092 ceph-mon[36497]: pgmap v397: 97 pgs: 97 active+clean; 4.2 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 6.4 MiB/s wr, 628 op/s 2024-04-02T14:47:38.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:47:38 smithi177 ceph-mon[36431]: pgmap v397: 97 pgs: 97 active+clean; 4.2 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 6.4 MiB/s wr, 628 op/s 2024-04-02T14:47:38.681 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:47:38 smithi090 ceph-mon[31674]: pgmap v397: 97 pgs: 97 active+clean; 4.2 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 6.4 MiB/s wr, 628 op/s 2024-04-02T14:47:38.951 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:38.956+0000 7fe6f5e00640 1 -- 172.21.15.90:0/1202288121 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe6f00747f0 msgr2=0x7fe6f010de10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:38.951 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:38.956+0000 7fe6f5e00640 1 --2- 172.21.15.90:0/1202288121 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe6f00747f0 0x7fe6f010de10 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fe6e0009860 tx=0x7fe6e002bc00 comp rx=0 tx=0).stop 2024-04-02T14:47:38.951 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:38.956+0000 7fe6f5e00640 1 -- 172.21.15.90:0/1202288121 shutdown_connections 2024-04-02T14:47:38.951 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:38.956+0000 7fe6f5e00640 1 --2- 172.21.15.90:0/1202288121 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe6f010e350 0x7fe6f0110760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:38.952 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:38.956+0000 7fe6f5e00640 1 --2- 172.21.15.90:0/1202288121 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe6f00747f0 0x7fe6f010de10 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:38.952 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:38.956+0000 7fe6f5e00640 1 --2- 172.21.15.90:0/1202288121 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe6f0073ed0 0x7fe6f00742b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:38.952 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:38.956+0000 7fe6f5e00640 1 -- 172.21.15.90:0/1202288121 >> 172.21.15.90:0/1202288121 conn(0x7fe6f00fd5c0 msgr2=0x7fe6f00ff9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:47:38.952 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:38.956+0000 7fe6f5e00640 1 -- 172.21.15.90:0/1202288121 shutdown_connections 2024-04-02T14:47:38.952 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:38.956+0000 7fe6f5e00640 1 -- 172.21.15.90:0/1202288121 wait complete. 2024-04-02T14:47:38.954 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:38.958+0000 7fe6f5e00640 1 Processor -- start 2024-04-02T14:47:38.955 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:38.960+0000 7fe6f5e00640 1 -- start start 2024-04-02T14:47:38.955 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:38.960+0000 7fe6f5e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe6f0073ed0 0x7fe6f0101a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:38.955 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:38.960+0000 7fe6f5e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe6f00747f0 0x7fe6f010db20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:38.955 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:38.960+0000 7fe6f5e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe6f010e350 0x7fe6f0101f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:38.956 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:38.960+0000 7fe6f5e00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fe6f006bf70 con 0x7fe6f0073ed0 2024-04-02T14:47:38.956 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:38.961+0000 7fe6ee000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe6f00747f0 0x7fe6f010db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:38.956 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:38.961+0000 7fe6ee000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe6f00747f0 0x7fe6f010db20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:51244/0 (socket says 172.21.15.90:51244) 2024-04-02T14:47:38.956 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:38.961+0000 7fe6ee000640 1 -- 172.21.15.90:0/3233754247 learned_addr learned my addr 172.21.15.90:0/3233754247 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:47:38.956 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:38.961+0000 7fe6ef400640 1 --2- 172.21.15.90:0/3233754247 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe6f010e350 0x7fe6f0101f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:38.956 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:38.961+0000 7fe6eea00640 1 --2- 172.21.15.90:0/3233754247 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe6f0073ed0 0x7fe6f0101a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:38.956 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:38.961+0000 7fe6f5e00640 1 -- 172.21.15.90:0/3233754247 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fe6f006bdf0 con 0x7fe6f00747f0 2024-04-02T14:47:38.956 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:38.961+0000 7fe6f5e00640 1 -- 172.21.15.90:0/3233754247 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fe6f006c0f0 con 0x7fe6f010e350 2024-04-02T14:47:38.957 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:38.961+0000 7fe6ef400640 1 -- 172.21.15.90:0/3233754247 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe6f00747f0 msgr2=0x7fe6f010db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:38.957 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:38.961+0000 7fe6ef400640 1 --2- 172.21.15.90:0/3233754247 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe6f00747f0 0x7fe6f010db20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:38.957 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:38.961+0000 7fe6ef400640 1 -- 172.21.15.90:0/3233754247 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe6f0073ed0 msgr2=0x7fe6f0101a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:38.957 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:38.961+0000 7fe6ef400640 1 --2- 172.21.15.90:0/3233754247 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe6f0073ed0 0x7fe6f0101a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:38.957 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:38.961+0000 7fe6ef400640 1 -- 172.21.15.90:0/3233754247 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6f0105b90 con 0x7fe6f010e350 2024-04-02T14:47:38.957 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:38.962+0000 7fe6ef400640 1 --2- 172.21.15.90:0/3233754247 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe6f010e350 0x7fe6f0101f40 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7fe6f0074e20 tx=0x7fe6dc00eba0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:47:38.957 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:38.962+0000 7fe6f4a00640 1 -- 172.21.15.90:0/3233754247 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6dc004230 con 0x7fe6f010e350 2024-04-02T14:47:38.958 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:38.962+0000 7fe6f4a00640 1 -- 172.21.15.90:0/3233754247 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe6dc0043d0 con 0x7fe6f010e350 2024-04-02T14:47:38.958 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:38.962+0000 7fe6f4a00640 1 -- 172.21.15.90:0/3233754247 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6dc004a00 con 0x7fe6f010e350 2024-04-02T14:47:38.958 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:38.963+0000 7fe6f5e00640 1 -- 172.21.15.90:0/3233754247 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe6f01025a0 con 0x7fe6f010e350 2024-04-02T14:47:38.958 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:38.963+0000 7fe6f5e00640 1 -- 172.21.15.90:0/3233754247 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6f01b3dd0 con 0x7fe6f010e350 2024-04-02T14:47:38.964 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:38.969+0000 7fe6f8a11740 1 Processor -- start 2024-04-02T14:47:38.967 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:38.972+0000 7fe6f4a00640 1 -- 172.21.15.90:0/3233754247 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fe6dc00c880 con 0x7fe6f010e350 2024-04-02T14:47:38.969 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:38.972+0000 7fe6f8a11740 1 -- start start 2024-04-02T14:47:38.969 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:38.974+0000 7fe6f4a00640 1 --2- 172.21.15.90:0/3233754247 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fe6d40a3e90 0x7fe6d40a6350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:38.969 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:38.974+0000 7fe6f4a00640 1 -- 172.21.15.90:0/3233754247 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 8550+0+0 (secure 0 0 0) 0x7fe6dc0d0810 con 0x7fe6f010e350 2024-04-02T14:47:38.969 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:38.974+0000 7fe6f8a11740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a7b4709eb0 0x55a7b470a290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:38.969 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:38.974+0000 7fe6f8a11740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a7b470a7d0 0x55a7b4712e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:38.969 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:38.974+0000 7fe6f8a11740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a7b47133b0 0x55a7b47157a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:38.969 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:38.974+0000 7fe6f8a11740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55a7b466ed70 con 0x55a7b470a7d0 2024-04-02T14:47:38.969 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:38.974+0000 7fe6f8a11740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55a7b46172a0 con 0x55a7b4709eb0 2024-04-02T14:47:38.969 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:38.974+0000 7fe6f8a11740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55a7b466ad20 con 0x55a7b47133b0 2024-04-02T14:47:38.970 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:38.974+0000 7fe6ef400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a7b47133b0 0x55a7b47157a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:38.970 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:38.974+0000 7fe6eea00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a7b4709eb0 0x55a7b470a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:38.970 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:38.975+0000 7fe6ef400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a7b47133b0 0x55a7b47157a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:45094/0 (socket says 172.21.15.90:45094) 2024-04-02T14:47:38.970 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:38.975+0000 7fe6ef400640 1 -- 172.21.15.90:0/124024951 learned_addr learned my addr 172.21.15.90:0/124024951 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:47:38.970 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:38.975+0000 7fe6ee000640 1 --2- 172.21.15.90:0/124024951 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a7b470a7d0 0x55a7b4712e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:38.971 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:38.975+0000 7fe6eea00640 1 -- 172.21.15.90:0/124024951 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a7b47133b0 msgr2=0x55a7b47157a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:47:38.971 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:38.975+0000 7fe6eea00640 1 --2- 172.21.15.90:0/124024951 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a7b47133b0 0x55a7b47157a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:38.971 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:38.975+0000 7fe6eea00640 1 -- 172.21.15.90:0/124024951 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a7b470a7d0 msgr2=0x55a7b4712e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:47:38.971 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:38.975+0000 7fe6eea00640 1 --2- 172.21.15.90:0/124024951 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a7b470a7d0 0x55a7b4712e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:38.972 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:38.975+0000 7fe6eea00640 1 -- 172.21.15.90:0/124024951 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a7b462b840 con 0x55a7b4709eb0 2024-04-02T14:47:38.972 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:38.976+0000 7fe6ef400640 1 --2- 172.21.15.90:0/124024951 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a7b47133b0 0x55a7b47157a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:47:38.972 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:38.976+0000 7fe6ee000640 1 --2- 172.21.15.90:0/3233754247 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fe6d40a3e90 0x7fe6d40a6350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:38.972 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:38.977+0000 7fe6eea00640 1 --2- 172.21.15.90:0/124024951 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a7b4709eb0 0x55a7b470a290 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7fe6e8009fa0 tx=0x7fe6e80118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cbb75bd5b7022daa server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:47:38.972 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:38.977+0000 7fe6ee000640 1 --2- 172.21.15.90:0/3233754247 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fe6d40a3e90 0x7fe6d40a6350 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fe6e00057c0 tx=0x7fe6e002bc00 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:47:38.972 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:38.978+0000 7fe6dbe00640 1 -- 172.21.15.90:0/124024951 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6e8003a30 con 0x55a7b4709eb0 2024-04-02T14:47:38.975 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:38.978+0000 7fe6dbe00640 1 -- 172.21.15.90:0/124024951 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe6e8003bd0 con 0x55a7b4709eb0 2024-04-02T14:47:38.975 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:38.979+0000 7fe6f8a11740 1 -- 172.21.15.90:0/124024951 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a7b4709eb0 msgr2=0x55a7b470a290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:38.975 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:38.979+0000 7fe6dbe00640 1 -- 172.21.15.90:0/124024951 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6e8003a30 con 0x55a7b4709eb0 2024-04-02T14:47:38.975 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:38.979+0000 7fe6f8a11740 1 --2- 172.21.15.90:0/124024951 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a7b4709eb0 0x55a7b470a290 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7fe6e8009fa0 tx=0x7fe6e80118d0 comp rx=0 tx=0).stop 2024-04-02T14:47:38.975 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:38.980+0000 7fe6f8a11740 1 -- 172.21.15.90:0/124024951 shutdown_connections 2024-04-02T14:47:38.975 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:38.980+0000 7fe6f8a11740 1 --2- 172.21.15.90:0/124024951 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a7b47133b0 0x55a7b47157a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:38.975 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:38.980+0000 7fe6f8a11740 1 --2- 172.21.15.90:0/124024951 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a7b470a7d0 0x55a7b4712e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:38.975 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:38.980+0000 7fe6f8a11740 1 --2- 172.21.15.90:0/124024951 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a7b4709eb0 0x55a7b470a290 unknown :-1 s=CLOSED pgs=886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:38.975 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:38.980+0000 7fe6f8a11740 1 -- 172.21.15.90:0/124024951 >> 172.21.15.90:0/124024951 conn(0x55a7b47045e0 msgr2=0x55a7b47049c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:47:38.976 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:38.980+0000 7fe6f8a11740 1 -- 172.21.15.90:0/124024951 shutdown_connections 2024-04-02T14:47:38.976 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:38.981+0000 7fe6f8a11740 1 -- 172.21.15.90:0/124024951 wait complete. 2024-04-02T14:47:38.976 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:38.981+0000 7fe6f8a11740 1 -- start start 2024-04-02T14:47:38.977 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:38.982+0000 7fe6f8a11740 1 Processor -- start 2024-04-02T14:47:38.977 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:38.983+0000 7fe6f8a11740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a7b4709eb0 0x55a7b4710370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:38.978 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:38.983+0000 7fe6f8a11740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a7b470a7d0 0x55a7b47108b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:38.978 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:38.983+0000 7fe6f8a11740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a7b47133b0 0x55a7b4710df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:38.978 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:38.983+0000 7fe6f8a11740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55a7b47161f0 con 0x55a7b4709eb0 2024-04-02T14:47:38.978 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:38.983+0000 7fe6f8a11740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55a7b462a660 con 0x55a7b470a7d0 2024-04-02T14:47:38.978 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:38.983+0000 7fe6f8a11740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55a7b4716370 con 0x55a7b47133b0 2024-04-02T14:47:38.978 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:38.983+0000 7fe6f8a11740 10 client.0 _ll_register_callbacks cb 0x55a7b462a050 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:47:38.979 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:38.983+0000 7fe6ef400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a7b47133b0 0x55a7b4710df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:38.979 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:38.983+0000 7fe6ee000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a7b470a7d0 0x55a7b47108b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:38.979 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:38.983+0000 7fe6ef400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a7b47133b0 0x55a7b4710df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:45106/0 (socket says 172.21.15.90:45106) 2024-04-02T14:47:38.979 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:38.983+0000 7fe6ef400640 1 -- 172.21.15.90:0/1083459165 learned_addr learned my addr 172.21.15.90:0/1083459165 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:47:38.980 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:38.984+0000 7fe6ef400640 1 -- 172.21.15.90:0/1083459165 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a7b470a7d0 msgr2=0x55a7b47108b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:47:38.980 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:38.984+0000 7fe6ef400640 1 --2- 172.21.15.90:0/1083459165 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a7b470a7d0 0x55a7b47108b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:38.980 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:38.984+0000 7fe6eea00640 1 --2- 172.21.15.90:0/1083459165 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a7b4709eb0 0x55a7b4710370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:38.980 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:38.984+0000 7fe6ef400640 1 -- 172.21.15.90:0/1083459165 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a7b4709eb0 msgr2=0x55a7b4710370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:47:38.980 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:38.984+0000 7fe6ef400640 1 --2- 172.21.15.90:0/1083459165 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a7b4709eb0 0x55a7b4710370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:38.980 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:38.984+0000 7fe6ef400640 1 -- 172.21.15.90:0/1083459165 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a7b4639e80 con 0x55a7b47133b0 2024-04-02T14:47:38.980 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:38.984+0000 7fe6eea00640 1 --2- 172.21.15.90:0/1083459165 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a7b4709eb0 0x55a7b4710370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:47:38.980 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:38.984+0000 7fe6ef400640 1 --2- 172.21.15.90:0/1083459165 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a7b47133b0 0x55a7b4710df0 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7fe6dc09c430 tx=0x7fe6dc0057e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eaf0127b42de0bad server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:47:38.980 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:38.984+0000 7fe6d8c00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:47:38.980 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:38.984+0000 7fe6d8c00640 1 -- 172.21.15.90:0/1083459165 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6dc0a0ae0 con 0x55a7b47133b0 2024-04-02T14:47:38.981 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:38.984+0000 7fe6d8c00640 1 -- 172.21.15.90:0/1083459165 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 8550+0+0 (secure 0 0 0) 0x7fe6dc098da0 con 0x55a7b47133b0 2024-04-02T14:47:38.981 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:38.984+0000 7fe6f8a11740 1 -- 172.21.15.90:0/1083459165 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe6e8003bd0 con 0x55a7b47133b0 2024-04-02T14:47:38.981 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:38.985+0000 7fe6d8c00640 1 -- 172.21.15.90:0/1083459165 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fe6dc080050 con 0x55a7b47133b0 2024-04-02T14:47:38.981 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:38.985+0000 7fe6f8a11740 10 client.29197 fetch_fsmap learned FSMap version 27 2024-04-02T14:47:38.981 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:38.985+0000 7fe6f8a11740 1 -- 172.21.15.90:0/1083459165 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55a7b4383210 con 0x55a7b47133b0 2024-04-02T14:47:38.981 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:38.985+0000 7fe6d8c00640 1 -- 172.21.15.90:0/1083459165 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe6e8003bd0 con 0x55a7b47133b0 2024-04-02T14:47:38.982 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:38.986+0000 7fe6d8c00640 1 -- 172.21.15.90:0/1083459165 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fe6dc020030 con 0x55a7b47133b0 2024-04-02T14:47:38.982 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:38.986+0000 7fe6f8a11740 10 client.29197 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:47:38.982 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:38.986+0000 7fe6f8a11740 10 client.29197 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-02T14:47:38.982 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:38.986+0000 7fe6f8a11740 20 client.29197 populate_metadata read hostname 'smithi090' 2024-04-02T14:47:38.982 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:38.986+0000 7fe6f8a11740 1 --2- 172.21.15.90:0/1083459165 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55a7b4723a30 0x55a7b4725e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:38.982 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:38.986+0000 7fe6f8a11740 4 client.29197 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:47:38.982 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:38.986+0000 7fe6f8a11740 1 -- 172.21.15.90:0/1083459165 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a7b461ecd0 con 0x55a7b4723a30 2024-04-02T14:47:38.982 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:38.986+0000 7fe6eea00640 1 --2- 172.21.15.90:0/1083459165 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55a7b4723a30 0x55a7b4725e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:38.983 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:38.988+0000 7fe6d8c00640 10 client.29197 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:47:38.983 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:38.988+0000 7fe6eea00640 1 --2- 172.21.15.90:0/1083459165 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55a7b4723a30 0x55a7b4725e80 crc :-1 s=READY pgs=328 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=93430253db920ae1 server_cookie=53d41849c3be3ffa in_seq=0 out_seq=0 2024-04-02T14:47:38.983 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:38.989+0000 7fe6d8c00640 1 -- 172.21.15.90:0/1083459165 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a7b461ecd0 con 0x55a7b4723a30 2024-04-02T14:47:38.983 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:38.989+0000 7fe6d8c00640 10 client.29197 handle_command_reply: tid=0 2024-04-02T14:47:38.984 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:38.989+0000 7fe6f8a11740 1 client.29197 shutdown 2024-04-02T14:47:38.984 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:38.989+0000 7fe6f8a11740 1 -- 172.21.15.90:0/1083459165 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a7b47133b0 msgr2=0x55a7b4710df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:38.984 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:38.989+0000 7fe6f8a11740 1 --2- 172.21.15.90:0/1083459165 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a7b47133b0 0x55a7b4710df0 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7fe6dc09c430 tx=0x7fe6dc0057e0 comp rx=0 tx=0).stop 2024-04-02T14:47:38.984 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:38.989+0000 7fe6f8a11740 1 -- 172.21.15.90:0/1083459165 shutdown_connections 2024-04-02T14:47:38.984 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:38.989+0000 7fe6f8a11740 1 --2- 172.21.15.90:0/1083459165 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55a7b4723a30 0x55a7b4725e80 crc :-1 s=READY pgs=328 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:38.985 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:38.989+0000 7fe6f8a11740 1 --2- 172.21.15.90:0/1083459165 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a7b47133b0 0x55a7b4710df0 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:38.985 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:38.989+0000 7fe6f8a11740 1 --2- 172.21.15.90:0/1083459165 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a7b470a7d0 0x55a7b47108b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:38.985 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:38.989+0000 7fe6d8c00640 0 client.29197 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:47:38.985 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:38.989+0000 7fe6f8a11740 1 --2- 172.21.15.90:0/1083459165 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a7b4709eb0 0x55a7b4710370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:38.985 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:38.989+0000 7fe6f8a11740 1 -- 172.21.15.90:0/1083459165 >> 172.21.15.90:0/1083459165 conn(0x55a7b47045e0 msgr2=0x55a7b4706330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:47:38.985 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:38.989+0000 7fe6f8a11740 1 -- 172.21.15.90:0/1083459165 shutdown_connections 2024-04-02T14:47:38.985 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:38.990+0000 7fe6f8a11740 1 -- 172.21.15.90:0/1083459165 wait complete. 2024-04-02T14:47:38.985 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:38.990+0000 7fe6f8a11740 20 client.29197 trim_cache size 0 max 16384 2024-04-02T14:47:39.008 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:39.013+0000 7fe6f8a11740 1 Processor -- start 2024-04-02T14:47:39.009 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:39.013+0000 7fe6f8a11740 1 -- start start 2024-04-02T14:47:39.009 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:39.014+0000 7fe6f8a11740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a7b4709eb0 0x55a7b4712aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:39.009 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:39.014+0000 7fe6f8a11740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a7b470a7d0 0x55a7b471f100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:39.009 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:39.014+0000 7fe6f8a11740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a7b47133b0 0x55a7b47034c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:39.009 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:39.014+0000 7fe6f8a11740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fe6f01bc6f0 con 0x55a7b470a7d0 2024-04-02T14:47:39.009 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:39.014+0000 7fe6f8a11740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fe6f01be230 con 0x55a7b47133b0 2024-04-02T14:47:39.009 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:39.014+0000 7fe6f8a11740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fe6f01bcdc0 con 0x55a7b4709eb0 2024-04-02T14:47:39.010 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:39.015+0000 7fe6ef400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a7b47133b0 0x55a7b47034c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:39.010 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:39.015+0000 7fe6ef400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a7b47133b0 0x55a7b47034c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:51264/0 (socket says 172.21.15.90:51264) 2024-04-02T14:47:39.010 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:39.015+0000 7fe6ef400640 1 -- 172.21.15.90:0/584502722 learned_addr learned my addr 172.21.15.90:0/584502722 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:47:39.010 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:39.015+0000 7fe6eea00640 1 --2- 172.21.15.90:0/584502722 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a7b4709eb0 0x55a7b4712aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:39.011 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:39.015+0000 7fe6ef400640 1 -- 172.21.15.90:0/584502722 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a7b4709eb0 msgr2=0x55a7b4712aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:47:39.011 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:39.015+0000 7fe6ef400640 1 --2- 172.21.15.90:0/584502722 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a7b4709eb0 0x55a7b4712aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:39.011 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:39.015+0000 7fe6ef400640 1 -- 172.21.15.90:0/584502722 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a7b470a7d0 msgr2=0x55a7b471f100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:47:39.011 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:39.015+0000 7fe6ef400640 1 --2- 172.21.15.90:0/584502722 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a7b470a7d0 0x55a7b471f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:39.011 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:39.015+0000 7fe6ef400640 1 -- 172.21.15.90:0/584502722 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a7b444f620 con 0x55a7b47133b0 2024-04-02T14:47:39.011 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:39.016+0000 7fe6ef400640 1 --2- 172.21.15.90:0/584502722 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a7b47133b0 0x55a7b47034c0 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7fe6dc09c6c0 tx=0x7fe6dc09ac40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6bc3fa3defb5c6b5 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:47:39.011 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:39.016+0000 7fe6db400640 1 -- 172.21.15.90:0/584502722 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6dc0a0ae0 con 0x55a7b47133b0 2024-04-02T14:47:39.011 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:39.016+0000 7fe6db400640 1 -- 172.21.15.90:0/584502722 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe6dc0a53a0 con 0x55a7b47133b0 2024-04-02T14:47:39.011 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:39.016+0000 7fe6db400640 1 -- 172.21.15.90:0/584502722 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6dc0a1c70 con 0x55a7b47133b0 2024-04-02T14:47:39.012 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:39.017+0000 7fe6f8a11740 1 -- 172.21.15.90:0/584502722 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a7b47133b0 msgr2=0x55a7b47034c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:39.012 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:39.017+0000 7fe6f8a11740 1 --2- 172.21.15.90:0/584502722 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a7b47133b0 0x55a7b47034c0 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7fe6dc09c6c0 tx=0x7fe6dc09ac40 comp rx=0 tx=0).stop 2024-04-02T14:47:39.013 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:39.017+0000 7fe6f8a11740 1 -- 172.21.15.90:0/584502722 shutdown_connections 2024-04-02T14:47:39.013 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:39.017+0000 7fe6f8a11740 1 --2- 172.21.15.90:0/584502722 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a7b47133b0 0x55a7b47034c0 unknown :-1 s=CLOSED pgs=887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:39.013 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:39.017+0000 7fe6f8a11740 1 --2- 172.21.15.90:0/584502722 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a7b470a7d0 0x55a7b471f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:39.013 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:39.017+0000 7fe6f8a11740 1 --2- 172.21.15.90:0/584502722 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a7b4709eb0 0x55a7b4712aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:39.013 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:39.018+0000 7fe6f8a11740 1 -- 172.21.15.90:0/584502722 >> 172.21.15.90:0/584502722 conn(0x55a7b47045e0 msgr2=0x55a7b4702db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:47:39.013 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:39.018+0000 7fe6f8a11740 1 -- 172.21.15.90:0/584502722 shutdown_connections 2024-04-02T14:47:39.013 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:39.018+0000 7fe6f8a11740 1 -- 172.21.15.90:0/584502722 wait complete. 2024-04-02T14:47:39.014 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:39.018+0000 7fe6f8a11740 1 -- start start 2024-04-02T14:47:39.014 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:39.018+0000 7fe6f8a11740 1 Processor -- start 2024-04-02T14:47:39.014 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:39.019+0000 7fe6f8a11740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a7b4709eb0 0x55a7b470d300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:39.014 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:39.019+0000 7fe6f8a11740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a7b470a7d0 0x55a7b470d840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:39.014 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:39.019+0000 7fe6f8a11740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a7b47133b0 0x55a7b470dd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:39.014 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:39.019+0000 7fe6f8a11740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fe6f01bdb60 con 0x55a7b470a7d0 2024-04-02T14:47:39.014 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:39.019+0000 7fe6f8a11740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fe6f01bd490 con 0x55a7b4709eb0 2024-04-02T14:47:39.014 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:39.019+0000 7fe6f8a11740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fe6f01ba6f0 con 0x55a7b47133b0 2024-04-02T14:47:39.014 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:39.019+0000 7fe6f8a11740 10 client.0 _ll_register_callbacks cb 0x7fe6dc09c120 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:47:39.015 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:39.019+0000 7fe6ee000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a7b470a7d0 0x55a7b470d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:39.015 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:39.019+0000 7fe6eea00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a7b4709eb0 0x55a7b470d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:39.015 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:39.019+0000 7fe6ee000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a7b470a7d0 0x55a7b470d840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:58792/0 (socket says 172.21.15.90:58792) 2024-04-02T14:47:39.015 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:39.019+0000 7fe6ee000640 1 -- 172.21.15.90:0/52894069 learned_addr learned my addr 172.21.15.90:0/52894069 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:47:39.015 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:39.019+0000 7fe6eea00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a7b4709eb0 0x55a7b470d300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:51270/0 (socket says 172.21.15.90:51270) 2024-04-02T14:47:39.015 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:39.020+0000 7fe6ef400640 1 --2- 172.21.15.90:0/52894069 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a7b47133b0 0x55a7b470dd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:39.015 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:39.020+0000 7fe6eea00640 1 -- 172.21.15.90:0/52894069 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a7b47133b0 msgr2=0x55a7b470dd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:47:39.015 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:39.020+0000 7fe6eea00640 1 --2- 172.21.15.90:0/52894069 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a7b47133b0 0x55a7b470dd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:39.016 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:39.020+0000 7fe6eea00640 1 -- 172.21.15.90:0/52894069 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a7b470a7d0 msgr2=0x55a7b470d840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:47:39.016 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:39.020+0000 7fe6eea00640 1 --2- 172.21.15.90:0/52894069 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a7b470a7d0 0x55a7b470d840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:39.016 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:39.020+0000 7fe6eea00640 1 -- 172.21.15.90:0/52894069 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a7b4789040 con 0x55a7b4709eb0 2024-04-02T14:47:39.016 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:39.020+0000 7fe6eea00640 1 --2- 172.21.15.90:0/52894069 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a7b4709eb0 0x55a7b470d300 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x55a7b4716250 tx=0x7fe6e8003b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dca3e2b0a4638cfa server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:47:39.016 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:39.020+0000 7fe6d8c00640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:47:39.016 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:39.021+0000 7fe6d8c00640 1 -- 172.21.15.90:0/52894069 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6e802c070 con 0x55a7b4709eb0 2024-04-02T14:47:39.017 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:39.022+0000 7fe6d8c00640 1 -- 172.21.15.90:0/52894069 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 8550+0+0 (secure 0 0 0) 0x7fe6e80281d0 con 0x55a7b4709eb0 2024-04-02T14:47:39.017 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:39.022+0000 7fe6f8a11740 1 -- 172.21.15.90:0/52894069 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe6dc0a53a0 con 0x55a7b4709eb0 2024-04-02T14:47:39.017 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:39.022+0000 7fe6d8c00640 1 -- 172.21.15.90:0/52894069 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fe6e8036050 con 0x55a7b4709eb0 2024-04-02T14:47:39.017 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:39.023+0000 7fe6f8a11740 10 client.29420 fetch_fsmap learned FSMap version 27 2024-04-02T14:47:39.018 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:39.023+0000 7fe6f8a11740 1 -- 172.21.15.90:0/52894069 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55a7b478a390 con 0x55a7b4709eb0 2024-04-02T14:47:39.018 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:39.023+0000 7fe6d8c00640 1 -- 172.21.15.90:0/52894069 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe6e80288d0 con 0x55a7b4709eb0 2024-04-02T14:47:39.019 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:39.024+0000 7fe6d8c00640 1 -- 172.21.15.90:0/52894069 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fe6e8004040 con 0x55a7b4709eb0 2024-04-02T14:47:39.019 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:39.024+0000 7fe6f8a11740 10 client.29420 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:47:39.019 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:39.024+0000 7fe6f8a11740 10 client.29420 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-02T14:47:39.019 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:39.024+0000 7fe6f8a11740 20 client.29420 populate_metadata read hostname 'smithi090' 2024-04-02T14:47:39.019 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:39.024+0000 7fe6f8a11740 1 --2- 172.21.15.90:0/52894069 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55a7b478a970 0x55a7b478cd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:39.019 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:39.024+0000 7fe6f8a11740 4 client.29420 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-02T14:47:39.019 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:39.024+0000 7fe6f8a11740 1 -- 172.21.15.90:0/52894069 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7fe6dc0a53a0 con 0x55a7b478a970 2024-04-02T14:47:39.019 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:39.024+0000 7fe6ee000640 1 --2- 172.21.15.90:0/52894069 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55a7b478a970 0x55a7b478cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:39.020 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:39.025+0000 7fe6ee000640 1 --2- 172.21.15.90:0/52894069 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55a7b478a970 0x55a7b478cd60 crc :-1 s=READY pgs=329 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=84147d1b29f34dd4 server_cookie=6cc7051049205e72 in_seq=0 out_seq=0 2024-04-02T14:47:39.020 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:39.025+0000 7fe6d8c00640 10 client.29420 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:47:39.044 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:39.049+0000 7fe6d8c00640 1 -- 172.21.15.90:0/52894069 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7fe6dc0a53a0 con 0x55a7b478a970 2024-04-02T14:47:39.163 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:39.050+0000 7fe6d8c00640 10 client.29420 handle_command_reply: tid=0 2024-04-02T14:47:39.163 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:39.050+0000 7fe6f8a11740 1 client.29420 shutdown 2024-04-02T14:47:39.164 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:39.051+0000 7fe6f8a11740 1 -- 172.21.15.90:0/52894069 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a7b4709eb0 msgr2=0x55a7b470d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:39.164 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:39.051+0000 7fe6f8a11740 1 --2- 172.21.15.90:0/52894069 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a7b4709eb0 0x55a7b470d300 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x55a7b4716250 tx=0x7fe6e8003b50 comp rx=0 tx=0).stop 2024-04-02T14:47:39.164 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:39.051+0000 7fe6f8a11740 1 -- 172.21.15.90:0/52894069 shutdown_connections 2024-04-02T14:47:39.165 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:39.051+0000 7fe6f8a11740 1 --2- 172.21.15.90:0/52894069 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55a7b478a970 0x55a7b478cd60 crc :-1 s=READY pgs=329 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:39.165 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:39.051+0000 7fe6f8a11740 1 --2- 172.21.15.90:0/52894069 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a7b47133b0 0x55a7b470dd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:39.165 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:39.051+0000 7fe6f8a11740 1 --2- 172.21.15.90:0/52894069 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a7b470a7d0 0x55a7b470d840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:39.165 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:39.051+0000 7fe6f8a11740 1 --2- 172.21.15.90:0/52894069 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a7b4709eb0 0x55a7b470d300 unknown :-1 s=CLOSED pgs=888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:39.165 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:39.051+0000 7fe6d8c00640 0 client.29420 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:47:39.165 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:39.051+0000 7fe6f8a11740 1 -- 172.21.15.90:0/52894069 >> 172.21.15.90:0/52894069 conn(0x55a7b47045e0 msgr2=0x55a7b47062a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:47:39.165 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:39.052+0000 7fe6f8a11740 1 -- 172.21.15.90:0/52894069 shutdown_connections 2024-04-02T14:47:39.165 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:39.052+0000 7fe6f8a11740 1 -- 172.21.15.90:0/52894069 wait complete. 2024-04-02T14:47:39.165 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:39.052+0000 7fe6f8a11740 20 client.29420 trim_cache size 0 max 16384 2024-04-02T14:47:39.166 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:{ 2024-04-02T14:47:39.166 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "message": "", 2024-04-02T14:47:39.166 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "return_code": 0 2024-04-02T14:47:39.166 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:} 2024-04-02T14:47:39.167 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:39.053+0000 7fe6f5e00640 1 -- 172.21.15.90:0/3233754247 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fe6d40a3e90 msgr2=0x7fe6d40a6350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:39.167 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:39.053+0000 7fe6f5e00640 1 --2- 172.21.15.90:0/3233754247 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fe6d40a3e90 0x7fe6d40a6350 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fe6e00057c0 tx=0x7fe6e002bc00 comp rx=0 tx=0).stop 2024-04-02T14:47:39.167 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:39.053+0000 7fe6f5e00640 1 -- 172.21.15.90:0/3233754247 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe6f010e350 msgr2=0x7fe6f0101f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:39.167 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:39.053+0000 7fe6f5e00640 1 --2- 172.21.15.90:0/3233754247 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe6f010e350 0x7fe6f0101f40 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7fe6f0074e20 tx=0x7fe6dc00eba0 comp rx=0 tx=0).stop 2024-04-02T14:47:39.167 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:39.053+0000 7fe6f5e00640 1 -- 172.21.15.90:0/3233754247 shutdown_connections 2024-04-02T14:47:39.167 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:39.053+0000 7fe6f5e00640 1 --2- 172.21.15.90:0/3233754247 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fe6d40a3e90 0x7fe6d40a6350 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:39.167 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:39.053+0000 7fe6f5e00640 1 --2- 172.21.15.90:0/3233754247 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe6f010e350 0x7fe6f0101f40 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:39.168 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:39.053+0000 7fe6f5e00640 1 --2- 172.21.15.90:0/3233754247 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe6f00747f0 0x7fe6f010db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:39.168 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:39.053+0000 7fe6f5e00640 1 --2- 172.21.15.90:0/3233754247 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe6f0073ed0 0x7fe6f0101a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:39.168 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:39.053+0000 7fe6f5e00640 1 -- 172.21.15.90:0/3233754247 >> 172.21.15.90:0/3233754247 conn(0x7fe6f00fd5c0 msgr2=0x7fe6f00feec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:47:39.168 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:39.054+0000 7fe6f5e00640 1 -- 172.21.15.90:0/3233754247 shutdown_connections 2024-04-02T14:47:39.168 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:39.054+0000 7fe6f5e00640 1 -- 172.21.15.90:0/3233754247 wait complete. 2024-04-02T14:47:40.603 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:47:40 smithi092 ceph-mon[36497]: pgmap v398: 97 pgs: 97 active+clean; 4.2 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 6.6 MiB/s wr, 649 op/s 2024-04-02T14:47:40.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:47:40 smithi177 ceph-mon[36431]: pgmap v398: 97 pgs: 97 active+clean; 4.2 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 6.6 MiB/s wr, 649 op/s 2024-04-02T14:47:40.681 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:47:40 smithi090 ceph-mon[31674]: pgmap v398: 97 pgs: 97 active+clean; 4.2 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 6.6 MiB/s wr, 649 op/s 2024-04-02T14:47:41.824 INFO:teuthology.orchestra.run.smithi090.stderr:progress: 730.0 s, 1636.5 tps, lat 59.322 ms stddev 67.010 2024-04-02T14:47:42.603 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:47:42 smithi092 ceph-mon[36497]: pgmap v399: 97 pgs: 97 active+clean; 4.2 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 8.3 MiB/s wr, 854 op/s 2024-04-02T14:47:42.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:47:42 smithi177 ceph-mon[36431]: pgmap v399: 97 pgs: 97 active+clean; 4.2 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 8.3 MiB/s wr, 854 op/s 2024-04-02T14:47:42.682 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:47:42 smithi090 ceph-mon[31674]: pgmap v399: 97 pgs: 97 active+clean; 4.2 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 8.3 MiB/s wr, 854 op/s 2024-04-02T14:47:44.483 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:47:44 smithi090 ceph-mon[31674]: pgmap v400: 97 pgs: 97 active+clean; 4.2 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 5.2 MiB/s wr, 560 op/s 2024-04-02T14:47:44.603 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:47:44 smithi092 ceph-mon[36497]: pgmap v400: 97 pgs: 97 active+clean; 4.2 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 5.2 MiB/s wr, 560 op/s 2024-04-02T14:47:44.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:47:44 smithi177 ceph-mon[36431]: pgmap v400: 97 pgs: 97 active+clean; 4.2 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 5.2 MiB/s wr, 560 op/s 2024-04-02T14:47:46.603 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:47:46 smithi092 ceph-mon[36497]: pgmap v401: 97 pgs: 97 active+clean; 4.4 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 26 MiB/s wr, 1.50k op/s 2024-04-02T14:47:46.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:47:46 smithi177 ceph-mon[36431]: pgmap v401: 97 pgs: 97 active+clean; 4.4 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 26 MiB/s wr, 1.50k op/s 2024-04-02T14:47:46.681 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:47:46 smithi090 ceph-mon[31674]: pgmap v401: 97 pgs: 97 active+clean; 4.4 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 26 MiB/s wr, 1.50k op/s 2024-04-02T14:47:46.824 INFO:teuthology.orchestra.run.smithi090.stderr:progress: 735.0 s, 778.6 tps, lat 129.611 ms stddev 222.852 2024-04-02T14:47:48.196 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-02T14:47:48.196 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-02T14:47:48.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.493+0000 7f3003e00640 1 -- 172.21.15.90:0/1934084987 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f30040747c0 msgr2=0x7f3004112430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:48.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.493+0000 7f3003e00640 1 --2- 172.21.15.90:0/1934084987 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f30040747c0 0x7f3004112430 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f2ff4009ca0 tx=0x7f2ff402c180 comp rx=0 tx=0).stop 2024-04-02T14:47:48.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.493+0000 7f3003e00640 1 -- 172.21.15.90:0/1934084987 shutdown_connections 2024-04-02T14:47:48.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.493+0000 7f3003e00640 1 --2- 172.21.15.90:0/1934084987 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3004112970 0x7f3004114d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:48.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.493+0000 7f3003e00640 1 --2- 172.21.15.90:0/1934084987 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f30040747c0 0x7f3004112430 unknown :-1 s=CLOSED pgs=889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:48.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.493+0000 7f3003e00640 1 --2- 172.21.15.90:0/1934084987 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3004073ea0 0x7f3004074280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:48.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.493+0000 7f3003e00640 1 -- 172.21.15.90:0/1934084987 >> 172.21.15.90:0/1934084987 conn(0x7f30040fd470 msgr2=0x7f30040ff890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:47:48.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.493+0000 7f3003e00640 1 -- 172.21.15.90:0/1934084987 shutdown_connections 2024-04-02T14:47:48.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.493+0000 7f3003e00640 1 -- 172.21.15.90:0/1934084987 wait complete. 2024-04-02T14:47:48.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.495+0000 7f3003e00640 1 Processor -- start 2024-04-02T14:47:48.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.496+0000 7f3003e00640 1 -- start start 2024-04-02T14:47:48.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.496+0000 7f3003e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3004073ea0 0x7f3004111e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:48.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.496+0000 7f3003e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f30040747c0 0x7f30041123b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:48.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.496+0000 7f3003e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3004112970 0x7f300410b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:48.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.496+0000 7f3003e00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f3004118340 con 0x7f30040747c0 2024-04-02T14:47:48.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.496+0000 7f3003e00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f30041181c0 con 0x7f3004073ea0 2024-04-02T14:47:48.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.496+0000 7f3003e00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f30041184c0 con 0x7f3004112970 2024-04-02T14:47:48.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.496+0000 7f3002a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3004073ea0 0x7f3004111e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:48.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.496+0000 7f3002a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3004073ea0 0x7f3004111e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:49958/0 (socket says 172.21.15.90:49958) 2024-04-02T14:47:48.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.496+0000 7f3002a00640 1 -- 172.21.15.90:0/2525950038 learned_addr learned my addr 172.21.15.90:0/2525950038 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:47:48.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.497+0000 7f3002000640 1 --2- 172.21.15.90:0/2525950038 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f30040747c0 0x7f30041123b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:48.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.497+0000 7f3003400640 1 --2- 172.21.15.90:0/2525950038 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3004112970 0x7f300410b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:48.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.497+0000 7f3002a00640 1 -- 172.21.15.90:0/2525950038 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3004112970 msgr2=0x7f300410b0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:48.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.497+0000 7f3002a00640 1 --2- 172.21.15.90:0/2525950038 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3004112970 0x7f300410b0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:48.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.497+0000 7f3002a00640 1 -- 172.21.15.90:0/2525950038 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f30040747c0 msgr2=0x7f30041123b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:48.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.497+0000 7f3002a00640 1 --2- 172.21.15.90:0/2525950038 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f30040747c0 0x7f30041123b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:48.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.497+0000 7f3002a00640 1 -- 172.21.15.90:0/2525950038 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f300410b990 con 0x7f3004073ea0 2024-04-02T14:47:48.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.497+0000 7f3002a00640 1 --2- 172.21.15.90:0/2525950038 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3004073ea0 0x7f3004111e70 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f2ffc00c920 tx=0x7f2ffc00cde0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:47:48.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.497+0000 7f2ffb400640 1 -- 172.21.15.90:0/2525950038 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ffc007c70 con 0x7f3004073ea0 2024-04-02T14:47:48.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.497+0000 7f2ffb400640 1 -- 172.21.15.90:0/2525950038 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2ffc007e10 con 0x7f3004073ea0 2024-04-02T14:47:48.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.498+0000 7f3003e00640 1 -- 172.21.15.90:0/2525950038 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f300410bc80 con 0x7f3004073ea0 2024-04-02T14:47:48.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.498+0000 7f2ffb400640 1 -- 172.21.15.90:0/2525950038 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ffc0189a0 con 0x7f3004073ea0 2024-04-02T14:47:48.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.498+0000 7f3003e00640 1 -- 172.21.15.90:0/2525950038 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f300410fc20 con 0x7f3004073ea0 2024-04-02T14:47:48.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.501+0000 7f300c7c4740 1 Processor -- start 2024-04-02T14:47:48.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.503+0000 7f2ffb400640 1 -- 172.21.15.90:0/2525950038 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f2ffc018b40 con 0x7f3004073ea0 2024-04-02T14:47:48.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.503+0000 7f300c7c4740 1 -- start start 2024-04-02T14:47:48.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.503+0000 7f300c7c4740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56489a21f960 0x56489a21fd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:48.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.503+0000 7f300c7c4740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56489a220280 0x56489a228920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:48.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.504+0000 7f300c7c4740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56489a228e60 0x56489a22b250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:48.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.504+0000 7f300c7c4740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x56489a183770 con 0x56489a21f960 2024-04-02T14:47:48.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.504+0000 7f300c7c4740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x56489a12b700 con 0x56489a228e60 2024-04-02T14:47:48.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.504+0000 7f300c7c4740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x56489a17f720 con 0x56489a220280 2024-04-02T14:47:48.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.504+0000 7f3003400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56489a21f960 0x56489a21fd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:48.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.504+0000 7f3003400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56489a21f960 0x56489a21fd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33798/0 (socket says 172.21.15.90:33798) 2024-04-02T14:47:48.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.504+0000 7f3003400640 1 -- 172.21.15.90:0/1188397109 learned_addr learned my addr 172.21.15.90:0/1188397109 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:47:48.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.504+0000 7f3002000640 1 --2- 172.21.15.90:0/1188397109 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56489a228e60 0x56489a22b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:48.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.504+0000 7f3002a00640 1 --2- 172.21.15.90:0/1188397109 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56489a220280 0x56489a228920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:48.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.504+0000 7f3003400640 1 -- 172.21.15.90:0/1188397109 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56489a220280 msgr2=0x56489a228920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:47:48.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.504+0000 7f3003400640 1 --2- 172.21.15.90:0/1188397109 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56489a220280 0x56489a228920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:48.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.504+0000 7f3003400640 1 -- 172.21.15.90:0/1188397109 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56489a228e60 msgr2=0x56489a22b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:47:48.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.504+0000 7f3003400640 1 --2- 172.21.15.90:0/1188397109 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56489a228e60 0x56489a22b250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:48.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.504+0000 7f3003400640 1 -- 172.21.15.90:0/1188397109 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564899f617d0 con 0x56489a21f960 2024-04-02T14:47:48.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.505+0000 7f3003400640 1 --2- 172.21.15.90:0/1188397109 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56489a21f960 0x56489a21fd40 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f2fec00ab60 tx=0x7f2fec00f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dbed53c8047d01e4 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:47:48.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.505+0000 7f2ffb400640 1 --2- 172.21.15.90:0/2525950038 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f2fe80a3e20 0x7f2fe80a62e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:48.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.505+0000 7f2ffb400640 1 -- 172.21.15.90:0/2525950038 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 8550+0+0 (secure 0 0 0) 0x7f2ffc0d0880 con 0x7f3004073ea0 2024-04-02T14:47:48.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.505+0000 7f2ff3e00640 1 -- 172.21.15.90:0/1188397109 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fec00ae60 con 0x56489a21f960 2024-04-02T14:47:48.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.505+0000 7f2ff3e00640 1 -- 172.21.15.90:0/1188397109 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2fec0047a0 con 0x56489a21f960 2024-04-02T14:47:48.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.505+0000 7f2ff3e00640 1 -- 172.21.15.90:0/1188397109 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fec018800 con 0x56489a21f960 2024-04-02T14:47:48.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.505+0000 7f3002a00640 1 --2- 172.21.15.90:0/2525950038 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f2fe80a3e20 0x7f2fe80a62e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:48.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.506+0000 7f300c7c4740 1 -- 172.21.15.90:0/1188397109 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56489a21f960 msgr2=0x56489a21fd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:48.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.506+0000 7f3002a00640 1 --2- 172.21.15.90:0/2525950038 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f2fe80a3e20 0x7f2fe80a62e0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x56489a21f430 tx=0x7f2ffc005e40 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:47:48.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.506+0000 7f300c7c4740 1 --2- 172.21.15.90:0/1188397109 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56489a21f960 0x56489a21fd40 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f2fec00ab60 tx=0x7f2fec00f790 comp rx=0 tx=0).stop 2024-04-02T14:47:48.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.506+0000 7f300c7c4740 1 -- 172.21.15.90:0/1188397109 shutdown_connections 2024-04-02T14:47:48.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.506+0000 7f300c7c4740 1 --2- 172.21.15.90:0/1188397109 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56489a228e60 0x56489a22b250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:48.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.506+0000 7f300c7c4740 1 --2- 172.21.15.90:0/1188397109 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56489a220280 0x56489a228920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:48.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.506+0000 7f300c7c4740 1 --2- 172.21.15.90:0/1188397109 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56489a21f960 0x56489a21fd40 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:48.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.506+0000 7f300c7c4740 1 -- 172.21.15.90:0/1188397109 >> 172.21.15.90:0/1188397109 conn(0x56489a21a090 msgr2=0x56489a21a470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:47:48.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.506+0000 7f300c7c4740 1 -- 172.21.15.90:0/1188397109 shutdown_connections 2024-04-02T14:47:48.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.507+0000 7f300c7c4740 1 -- 172.21.15.90:0/1188397109 wait complete. 2024-04-02T14:47:48.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.507+0000 7f300c7c4740 1 -- start start 2024-04-02T14:47:48.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.507+0000 7f300c7c4740 1 Processor -- start 2024-04-02T14:47:48.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.508+0000 7f300c7c4740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56489a21f960 0x56489a225ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:48.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.508+0000 7f300c7c4740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56489a220280 0x56489a226430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:48.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.508+0000 7f300c7c4740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56489a228e60 0x56489a226970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:48.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.508+0000 7f300c7c4740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x56489a22bc50 con 0x56489a21f960 2024-04-02T14:47:48.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.508+0000 7f300c7c4740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x56489a13eac0 con 0x56489a220280 2024-04-02T14:47:48.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.508+0000 7f300c7c4740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x56489a22bdd0 con 0x56489a228e60 2024-04-02T14:47:48.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.509+0000 7f300c7c4740 10 client.0 _ll_register_callbacks cb 0x56489a141180 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:47:48.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.509+0000 7f3003400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56489a21f960 0x56489a225ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:48.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.509+0000 7f3002000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56489a220280 0x56489a226430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:48.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.509+0000 7f3002000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56489a220280 0x56489a226430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:49984/0 (socket says 172.21.15.90:49984) 2024-04-02T14:47:48.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.509+0000 7f3002000640 1 -- 172.21.15.90:0/1328695028 learned_addr learned my addr 172.21.15.90:0/1328695028 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:47:48.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.510+0000 7f3003400640 1 --2- 172.21.15.90:0/1328695028 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56489a228e60 0x56489a226970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:48.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.510+0000 7f3003400640 1 -- 172.21.15.90:0/1328695028 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56489a228e60 msgr2=0x56489a226970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:47:48.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.510+0000 7f3003400640 1 --2- 172.21.15.90:0/1328695028 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56489a228e60 0x56489a226970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:48.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.510+0000 7f3003400640 1 -- 172.21.15.90:0/1328695028 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56489a220280 msgr2=0x56489a226430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:47:48.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.510+0000 7f3003400640 1 --2- 172.21.15.90:0/1328695028 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56489a220280 0x56489a226430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:48.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.510+0000 7f3003400640 1 -- 172.21.15.90:0/1328695028 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564899e973f0 con 0x56489a21f960 2024-04-02T14:47:48.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.510+0000 7f3002000640 1 --2- 172.21.15.90:0/1328695028 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56489a220280 0x56489a226430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:47:48.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.511+0000 7f3003400640 1 --2- 172.21.15.90:0/1328695028 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56489a21f960 0x56489a225ef0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f2fec000c00 tx=0x7f2fec00a880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83a126c2036ccca4 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:47:48.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.511+0000 7f2ff0c00640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-02T14:47:48.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.511+0000 7f2ff0c00640 1 -- 172.21.15.90:0/1328695028 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fec02e060 con 0x56489a21f960 2024-04-02T14:47:48.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.511+0000 7f2ff0c00640 1 -- 172.21.15.90:0/1328695028 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 8550+0+0 (secure 0 0 0) 0x7f2fec004fa0 con 0x56489a21f960 2024-04-02T14:47:48.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.511+0000 7f300c7c4740 1 -- 172.21.15.90:0/1328695028 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2fec0047a0 con 0x56489a21f960 2024-04-02T14:47:48.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.511+0000 7f2ff0c00640 1 -- 172.21.15.90:0/1328695028 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f2fec0051a0 con 0x56489a21f960 2024-04-02T14:47:48.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.511+0000 7f300c7c4740 10 client.18312 fetch_fsmap learned FSMap version 27 2024-04-02T14:47:48.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.511+0000 7f300c7c4740 1 -- 172.21.15.90:0/1328695028 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x564899e570f0 con 0x56489a21f960 2024-04-02T14:47:48.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.512+0000 7f2ff0c00640 1 -- 172.21.15.90:0/1328695028 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2fec0047a0 con 0x56489a21f960 2024-04-02T14:47:48.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.512+0000 7f2ff0c00640 1 -- 172.21.15.90:0/1328695028 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f2fec024040 con 0x56489a21f960 2024-04-02T14:47:48.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.512+0000 7f300c7c4740 10 client.18312 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:47:48.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.512+0000 7f300c7c4740 10 client.18312 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:47:48.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.512+0000 7f300c7c4740 20 client.18312 populate_metadata read hostname 'smithi090' 2024-04-02T14:47:48.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.512+0000 7f300c7c4740 1 --2- 172.21.15.90:0/1328695028 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x56489a238e50 0x56489a23b240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:48.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.512+0000 7f300c7c4740 4 client.18312 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:47:48.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.512+0000 7f300c7c4740 1 -- 172.21.15.90:0/1328695028 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56489a124630 con 0x56489a238e50 2024-04-02T14:47:48.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.512+0000 7f3002000640 1 --2- 172.21.15.90:0/1328695028 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x56489a238e50 0x56489a23b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:48.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.513+0000 7f3002000640 1 --2- 172.21.15.90:0/1328695028 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x56489a238e50 0x56489a23b240 crc :-1 s=READY pgs=330 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ace0530343554662 server_cookie=ce8695e3919e2298 in_seq=0 out_seq=0 2024-04-02T14:47:48.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.513+0000 7f2ff0c00640 10 client.18312 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:47:48.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.513+0000 7f2ff0c00640 1 -- 172.21.15.90:0/1328695028 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56489a124630 con 0x56489a238e50 2024-04-02T14:47:48.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.513+0000 7f2ff0c00640 10 client.18312 handle_command_reply: tid=0 2024-04-02T14:47:48.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.513+0000 7f300c7c4740 1 client.18312 shutdown 2024-04-02T14:47:48.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.514+0000 7f300c7c4740 1 -- 172.21.15.90:0/1328695028 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56489a21f960 msgr2=0x56489a225ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:48.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.514+0000 7f300c7c4740 1 --2- 172.21.15.90:0/1328695028 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56489a21f960 0x56489a225ef0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f2fec000c00 tx=0x7f2fec00a880 comp rx=0 tx=0).stop 2024-04-02T14:47:48.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.514+0000 7f300c7c4740 1 -- 172.21.15.90:0/1328695028 shutdown_connections 2024-04-02T14:47:48.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.514+0000 7f300c7c4740 1 --2- 172.21.15.90:0/1328695028 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x56489a238e50 0x56489a23b240 crc :-1 s=READY pgs=330 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:48.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.514+0000 7f300c7c4740 1 --2- 172.21.15.90:0/1328695028 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56489a228e60 0x56489a226970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:48.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.514+0000 7f300c7c4740 1 --2- 172.21.15.90:0/1328695028 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56489a220280 0x56489a226430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:48.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.514+0000 7f300c7c4740 1 --2- 172.21.15.90:0/1328695028 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56489a21f960 0x56489a225ef0 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:48.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.514+0000 7f300c7c4740 1 -- 172.21.15.90:0/1328695028 >> 172.21.15.90:0/1328695028 conn(0x56489a21a090 msgr2=0x56489a219310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:47:48.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.514+0000 7f2ff0c00640 0 client.18312 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:47:48.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.514+0000 7f300c7c4740 1 -- 172.21.15.90:0/1328695028 shutdown_connections 2024-04-02T14:47:48.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.514+0000 7f300c7c4740 1 -- 172.21.15.90:0/1328695028 wait complete. 2024-04-02T14:47:48.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.514+0000 7f300c7c4740 20 client.18312 trim_cache size 0 max 16384 2024-04-02T14:47:48.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.543+0000 7f300c7c4740 1 Processor -- start 2024-04-02T14:47:48.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.543+0000 7f300c7c4740 1 -- start start 2024-04-02T14:47:48.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.543+0000 7f300c7c4740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56489a21f960 0x56489a228620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:48.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.543+0000 7f300c7c4740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56489a220280 0x56489a2256a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:48.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.543+0000 7f300c7c4740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56489a228e60 0x56489a23e260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:48.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.543+0000 7f300c7c4740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f30041bc480 con 0x56489a21f960 2024-04-02T14:47:48.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.543+0000 7f300c7c4740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f30041bdfc0 con 0x56489a220280 2024-04-02T14:47:48.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.543+0000 7f300c7c4740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f30041bcb50 con 0x56489a228e60 2024-04-02T14:47:48.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.544+0000 7f3003400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56489a21f960 0x56489a228620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:48.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.544+0000 7f3002000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56489a220280 0x56489a2256a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:48.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.544+0000 7f3003400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56489a21f960 0x56489a228620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33806/0 (socket says 172.21.15.90:33806) 2024-04-02T14:47:48.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.544+0000 7f3003400640 1 -- 172.21.15.90:0/375434848 learned_addr learned my addr 172.21.15.90:0/375434848 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:47:48.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.544+0000 7f3002000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56489a220280 0x56489a2256a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:49996/0 (socket says 172.21.15.90:49996) 2024-04-02T14:47:48.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.544+0000 7f3003400640 1 --2- 172.21.15.90:0/375434848 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56489a228e60 0x56489a23e260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:48.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.544+0000 7f3002000640 1 -- 172.21.15.90:0/375434848 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56489a228e60 msgr2=0x56489a23e260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:47:48.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.544+0000 7f3002000640 1 --2- 172.21.15.90:0/375434848 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56489a228e60 0x56489a23e260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:48.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.544+0000 7f3002000640 1 -- 172.21.15.90:0/375434848 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56489a21f960 msgr2=0x56489a228620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:47:48.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.544+0000 7f3002000640 1 --2- 172.21.15.90:0/375434848 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56489a21f960 0x56489a228620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:48.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.544+0000 7f3002000640 1 -- 172.21.15.90:0/375434848 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56489a2200b0 con 0x56489a220280 2024-04-02T14:47:48.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.544+0000 7f3002000640 1 --2- 172.21.15.90:0/375434848 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56489a220280 0x56489a2256a0 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7f2ff40323b0 tx=0x7f2ff402e040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=31f927a0714d4c74 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:47:48.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.544+0000 7f2ff3400640 1 -- 172.21.15.90:0/375434848 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ff4039040 con 0x56489a220280 2024-04-02T14:47:48.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.545+0000 7f2ff3400640 1 -- 172.21.15.90:0/375434848 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2ff404e070 con 0x56489a220280 2024-04-02T14:47:48.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.545+0000 7f2ff3400640 1 -- 172.21.15.90:0/375434848 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ff40042a0 con 0x56489a220280 2024-04-02T14:47:48.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.545+0000 7f300c7c4740 1 -- 172.21.15.90:0/375434848 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56489a220280 msgr2=0x56489a2256a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:48.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.545+0000 7f300c7c4740 1 --2- 172.21.15.90:0/375434848 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56489a220280 0x56489a2256a0 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7f2ff40323b0 tx=0x7f2ff402e040 comp rx=0 tx=0).stop 2024-04-02T14:47:48.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.546+0000 7f300c7c4740 1 -- 172.21.15.90:0/375434848 shutdown_connections 2024-04-02T14:47:48.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.546+0000 7f300c7c4740 1 --2- 172.21.15.90:0/375434848 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56489a228e60 0x56489a23e260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:48.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.546+0000 7f300c7c4740 1 --2- 172.21.15.90:0/375434848 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56489a220280 0x56489a2256a0 unknown :-1 s=CLOSED pgs=891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:48.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.546+0000 7f300c7c4740 1 --2- 172.21.15.90:0/375434848 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56489a21f960 0x56489a228620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:48.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.546+0000 7f300c7c4740 1 -- 172.21.15.90:0/375434848 >> 172.21.15.90:0/375434848 conn(0x56489a21a090 msgr2=0x56489a2348b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:47:48.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.546+0000 7f300c7c4740 1 -- 172.21.15.90:0/375434848 shutdown_connections 2024-04-02T14:47:48.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.547+0000 7f300c7c4740 1 -- 172.21.15.90:0/375434848 wait complete. 2024-04-02T14:47:48.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.547+0000 7f300c7c4740 1 -- start start 2024-04-02T14:47:48.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.548+0000 7f300c7c4740 1 Processor -- start 2024-04-02T14:47:48.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.549+0000 7f300c7c4740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56489a21f960 0x56489a23e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:48.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.549+0000 7f300c7c4740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56489a220280 0x56489a23bb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:48.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.549+0000 7f300c7c4740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56489a228e60 0x56489a23c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:48.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.549+0000 7f300c7c4740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f30041bd8f0 con 0x56489a228e60 2024-04-02T14:47:48.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.549+0000 7f300c7c4740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f30041bd220 con 0x56489a220280 2024-04-02T14:47:48.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.549+0000 7f300c7c4740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f30041ba480 con 0x56489a21f960 2024-04-02T14:47:48.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.549+0000 7f300c7c4740 10 client.0 _ll_register_callbacks cb 0x7f2ff4009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:47:48.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.550+0000 7f3003400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56489a21f960 0x56489a23e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:48.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.550+0000 7f3002000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56489a220280 0x56489a23bb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:48.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.550+0000 7f3002000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56489a220280 0x56489a23bb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:50004/0 (socket says 172.21.15.90:50004) 2024-04-02T14:47:48.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.550+0000 7f3002000640 1 -- 172.21.15.90:0/41805846 learned_addr learned my addr 172.21.15.90:0/41805846 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:47:48.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.550+0000 7f3003400640 1 --2- 172.21.15.90:0/41805846 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56489a228e60 0x56489a23c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:48.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.551+0000 7f3002000640 1 -- 172.21.15.90:0/41805846 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56489a21f960 msgr2=0x56489a23e7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:47:48.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.551+0000 7f3002000640 1 --2- 172.21.15.90:0/41805846 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56489a21f960 0x56489a23e7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:48.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.551+0000 7f3002000640 1 -- 172.21.15.90:0/41805846 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56489a228e60 msgr2=0x56489a23c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:47:48.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.551+0000 7f3002000640 1 --2- 172.21.15.90:0/41805846 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56489a228e60 0x56489a23c2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:48.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.551+0000 7f3002000640 1 -- 172.21.15.90:0/41805846 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56489a29e030 con 0x56489a220280 2024-04-02T14:47:48.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.551+0000 7f3003400640 1 --2- 172.21.15.90:0/41805846 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56489a21f960 0x56489a23e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:47:48.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.551+0000 7f3002000640 1 --2- 172.21.15.90:0/41805846 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56489a220280 0x56489a23bb80 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7f2ff404b040 tx=0x7f2ff4003ef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=edfe51fdef7c73ae server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:47:48.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.551+0000 7f2ff0c00640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:47:48.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.552+0000 7f2ff0c00640 1 -- 172.21.15.90:0/41805846 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ff4039040 con 0x56489a220280 2024-04-02T14:47:48.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.553+0000 7f2ff0c00640 1 -- 172.21.15.90:0/41805846 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 8550+0+0 (secure 0 0 0) 0x7f2ff4047040 con 0x56489a220280 2024-04-02T14:47:48.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.553+0000 7f300c7c4740 1 -- 172.21.15.90:0/41805846 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2ff404e070 con 0x56489a220280 2024-04-02T14:47:48.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.553+0000 7f2ff0c00640 1 -- 172.21.15.90:0/41805846 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f2ff405f050 con 0x56489a220280 2024-04-02T14:47:48.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.553+0000 7f300c7c4740 10 client.29438 fetch_fsmap learned FSMap version 27 2024-04-02T14:47:48.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.553+0000 7f300c7c4740 1 -- 172.21.15.90:0/41805846 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x56489a29f200 con 0x56489a220280 2024-04-02T14:47:48.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.554+0000 7f2ff0c00640 1 -- 172.21.15.90:0/41805846 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2ff404e070 con 0x56489a220280 2024-04-02T14:47:48.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.554+0000 7f2ff0c00640 1 -- 172.21.15.90:0/41805846 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f2ff4004620 con 0x56489a220280 2024-04-02T14:47:48.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.554+0000 7f300c7c4740 10 client.29438 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:47:48.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.554+0000 7f300c7c4740 10 client.29438 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:47:48.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.554+0000 7f300c7c4740 20 client.29438 populate_metadata read hostname 'smithi090' 2024-04-02T14:47:48.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.554+0000 7f300c7c4740 1 --2- 172.21.15.90:0/41805846 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x56489a29f8c0 0x56489a2a1ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:48.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.554+0000 7f300c7c4740 4 client.29438 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-02T14:47:48.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.554+0000 7f300c7c4740 1 -- 172.21.15.90:0/41805846 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "scrub status"}) -- 0x56489a190cb0 con 0x56489a29f8c0 2024-04-02T14:47:48.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.555+0000 7f3003400640 1 --2- 172.21.15.90:0/41805846 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x56489a29f8c0 0x56489a2a1ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:48.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.555+0000 7f3003400640 1 --2- 172.21.15.90:0/41805846 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x56489a29f8c0 0x56489a2a1ce0 crc :-1 s=READY pgs=331 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dafbc14020ad4f42 server_cookie=b20e13cc2c35e0cc in_seq=0 out_seq=0 2024-04-02T14:47:48.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.555+0000 7f2ff0c00640 10 client.29438 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:47:48.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.555+0000 7f2ff0c00640 1 -- 172.21.15.90:0/41805846 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x56489a190cb0 con 0x56489a29f8c0 2024-04-02T14:47:48.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.555+0000 7f2ff0c00640 10 client.29438 handle_command_reply: tid=0 2024-04-02T14:47:48.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.555+0000 7f300c7c4740 1 client.29438 shutdown 2024-04-02T14:47:48.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.556+0000 7f300c7c4740 1 -- 172.21.15.90:0/41805846 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56489a220280 msgr2=0x56489a23bb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:48.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.556+0000 7f300c7c4740 1 --2- 172.21.15.90:0/41805846 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56489a220280 0x56489a23bb80 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7f2ff404b040 tx=0x7f2ff4003ef0 comp rx=0 tx=0).stop 2024-04-02T14:47:48.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.556+0000 7f300c7c4740 1 -- 172.21.15.90:0/41805846 shutdown_connections 2024-04-02T14:47:48.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.556+0000 7f300c7c4740 1 --2- 172.21.15.90:0/41805846 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x56489a29f8c0 0x56489a2a1ce0 crc :-1 s=READY pgs=331 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:48.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.556+0000 7f300c7c4740 1 --2- 172.21.15.90:0/41805846 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56489a228e60 0x56489a23c2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:48.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.556+0000 7f300c7c4740 1 --2- 172.21.15.90:0/41805846 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56489a220280 0x56489a23bb80 unknown :-1 s=CLOSED pgs=892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:48.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.556+0000 7f300c7c4740 1 --2- 172.21.15.90:0/41805846 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56489a21f960 0x56489a23e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:48.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.556+0000 7f300c7c4740 1 -- 172.21.15.90:0/41805846 >> 172.21.15.90:0/41805846 conn(0x56489a21a090 msgr2=0x56489a225eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:47:48.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.556+0000 7f2ff0c00640 0 client.29438 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:47:48.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.556+0000 7f300c7c4740 1 -- 172.21.15.90:0/41805846 shutdown_connections 2024-04-02T14:47:48.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.556+0000 7f300c7c4740 1 -- 172.21.15.90:0/41805846 wait complete. 2024-04-02T14:47:48.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.557+0000 7f300c7c4740 20 client.29438 trim_cache size 0 max 16384 2024-04-02T14:47:48.553 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-02T14:47:48.553 INFO:teuthology.orchestra.run.smithi090.stdout: "status": "no active scrubs running", 2024-04-02T14:47:48.553 INFO:teuthology.orchestra.run.smithi090.stdout: "scrubs": {} 2024-04-02T14:47:48.553 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-02T14:47:48.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.559+0000 7f3003e00640 1 -- 172.21.15.90:0/2525950038 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f2fe80a3e20 msgr2=0x7f2fe80a62e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:48.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.559+0000 7f3003e00640 1 --2- 172.21.15.90:0/2525950038 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f2fe80a3e20 0x7f2fe80a62e0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x56489a21f430 tx=0x7f2ffc005e40 comp rx=0 tx=0).stop 2024-04-02T14:47:48.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.559+0000 7f3003e00640 1 -- 172.21.15.90:0/2525950038 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3004073ea0 msgr2=0x7f3004111e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:48.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.559+0000 7f3003e00640 1 --2- 172.21.15.90:0/2525950038 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3004073ea0 0x7f3004111e70 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f2ffc00c920 tx=0x7f2ffc00cde0 comp rx=0 tx=0).stop 2024-04-02T14:47:48.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.560+0000 7f3003e00640 1 -- 172.21.15.90:0/2525950038 shutdown_connections 2024-04-02T14:47:48.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.560+0000 7f3003e00640 1 --2- 172.21.15.90:0/2525950038 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f2fe80a3e20 0x7f2fe80a62e0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:48.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.560+0000 7f3003e00640 1 --2- 172.21.15.90:0/2525950038 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3004112970 0x7f300410b0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:48.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.560+0000 7f3003e00640 1 --2- 172.21.15.90:0/2525950038 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f30040747c0 0x7f30041123b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:48.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.560+0000 7f3003e00640 1 --2- 172.21.15.90:0/2525950038 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3004073ea0 0x7f3004111e70 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:48.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.560+0000 7f3003e00640 1 -- 172.21.15.90:0/2525950038 >> 172.21.15.90:0/2525950038 conn(0x7f30040fd470 msgr2=0x7f30040fe0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:47:48.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.560+0000 7f3003e00640 1 -- 172.21.15.90:0/2525950038 shutdown_connections 2024-04-02T14:47:48.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.560+0000 7f3003e00640 1 -- 172.21.15.90:0/2525950038 wait complete. 2024-04-02T14:47:48.576 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:47:48 smithi090 ceph-mon[31674]: pgmap v402: 97 pgs: 97 active+clean; 4.5 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 30 MiB/s wr, 1.52k op/s 2024-04-02T14:47:48.582 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-02T14:47:48.583 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T14:47:48.603 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:47:48 smithi092 ceph-mon[36497]: pgmap v402: 97 pgs: 97 active+clean; 4.5 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 30 MiB/s wr, 1.52k op/s 2024-04-02T14:47:48.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:47:48 smithi177 ceph-mon[36431]: pgmap v402: 97 pgs: 97 active+clean; 4.5 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 30 MiB/s wr, 1.52k op/s 2024-04-02T14:47:48.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.875+0000 7f7083000640 1 -- 172.21.15.90:0/2292873468 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f707c068c60 msgr2=0x7f707c0690c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:48.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.875+0000 7f7083000640 1 --2- 172.21.15.90:0/2292873468 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f707c068c60 0x7f707c0690c0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f706800fb40 tx=0x7f706802fdb0 comp rx=0 tx=0).stop 2024-04-02T14:47:48.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.875+0000 7f7083000640 1 -- 172.21.15.90:0/2292873468 shutdown_connections 2024-04-02T14:47:48.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.875+0000 7f7083000640 1 --2- 172.21.15.90:0/2292873468 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f707c069600 0x7f707c10e7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:48.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.875+0000 7f7083000640 1 --2- 172.21.15.90:0/2292873468 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f707c068c60 0x7f707c0690c0 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:48.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.875+0000 7f7083000640 1 --2- 172.21.15.90:0/2292873468 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f707c107950 0x7f707c107d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:48.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.875+0000 7f7083000640 1 -- 172.21.15.90:0/2292873468 >> 172.21.15.90:0/2292873468 conn(0x7f707c076410 msgr2=0x7f707c076820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:47:48.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.876+0000 7f7083000640 1 -- 172.21.15.90:0/2292873468 shutdown_connections 2024-04-02T14:47:48.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.876+0000 7f7083000640 1 -- 172.21.15.90:0/2292873468 wait complete. 2024-04-02T14:47:48.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.877+0000 7f7083000640 1 Processor -- start 2024-04-02T14:47:48.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.878+0000 7f7083000640 1 -- start start 2024-04-02T14:47:48.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.878+0000 7f7083000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f707c068c60 0x7f707c1a45a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:48.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.878+0000 7f7083000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f707c069600 0x7f707c1a4ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:48.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.878+0000 7f7083000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f707c107950 0x7f707c19e660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:48.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.878+0000 7f7083000640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f707c118380 con 0x7f707c068c60 2024-04-02T14:47:48.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.878+0000 7f7083000640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f707c118200 con 0x7f707c069600 2024-04-02T14:47:48.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.878+0000 7f7083000640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f707c118500 con 0x7f707c107950 2024-04-02T14:47:48.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.878+0000 7f7081200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f707c069600 0x7f707c1a4ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:48.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.878+0000 7f7081200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f707c069600 0x7f707c1a4ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:50030/0 (socket says 172.21.15.90:50030) 2024-04-02T14:47:48.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.878+0000 7f7081200640 1 -- 172.21.15.90:0/2162411399 learned_addr learned my addr 172.21.15.90:0/2162411399 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:47:48.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.878+0000 7f7081c00640 1 --2- 172.21.15.90:0/2162411399 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f707c068c60 0x7f707c1a45a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:48.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.879+0000 7f7082600640 1 --2- 172.21.15.90:0/2162411399 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f707c107950 0x7f707c19e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:48.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.879+0000 7f7081200640 1 -- 172.21.15.90:0/2162411399 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f707c107950 msgr2=0x7f707c19e660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:48.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.879+0000 7f7081200640 1 --2- 172.21.15.90:0/2162411399 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f707c107950 0x7f707c19e660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:48.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.879+0000 7f7081200640 1 -- 172.21.15.90:0/2162411399 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f707c068c60 msgr2=0x7f707c1a45a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:48.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.879+0000 7f7081200640 1 --2- 172.21.15.90:0/2162411399 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f707c068c60 0x7f707c1a45a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:48.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.879+0000 7f7081200640 1 -- 172.21.15.90:0/2162411399 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f707c19ef20 con 0x7f707c069600 2024-04-02T14:47:48.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.879+0000 7f7081c00640 1 --2- 172.21.15.90:0/2162411399 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f707c068c60 0x7f707c1a45a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:47:48.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.879+0000 7f7081200640 1 --2- 172.21.15.90:0/2162411399 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f707c069600 0x7f707c1a4ae0 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f7068000c00 tx=0x7f7068005720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:47:48.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.879+0000 7f707a000640 1 -- 172.21.15.90:0/2162411399 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7068045070 con 0x7f707c069600 2024-04-02T14:47:48.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.880+0000 7f707a000640 1 -- 172.21.15.90:0/2162411399 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7068031d30 con 0x7f707c069600 2024-04-02T14:47:48.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.880+0000 7f707a000640 1 -- 172.21.15.90:0/2162411399 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f706800ea40 con 0x7f707c069600 2024-04-02T14:47:48.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.880+0000 7f7083000640 1 -- 172.21.15.90:0/2162411399 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f707c19f1b0 con 0x7f707c069600 2024-04-02T14:47:48.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.880+0000 7f7083000640 1 -- 172.21.15.90:0/2162411399 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f707c1ab340 con 0x7f707c069600 2024-04-02T14:47:48.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.881+0000 7f7083000640 1 -- 172.21.15.90:0/2162411399 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f707c103010 con 0x7f707c069600 2024-04-02T14:47:48.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.885+0000 7f707a000640 1 -- 172.21.15.90:0/2162411399 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f7068048070 con 0x7f707c069600 2024-04-02T14:47:48.882 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.886+0000 7f707a000640 1 --2- 172.21.15.90:0/2162411399 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f70640a3d60 0x7f70640a6220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:48.882 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.886+0000 7f707a000640 1 -- 172.21.15.90:0/2162411399 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 8550+0+0 (secure 0 0 0) 0x7f70680f1e00 con 0x7f707c069600 2024-04-02T14:47:48.882 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.887+0000 7f7081c00640 1 --2- 172.21.15.90:0/2162411399 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f70640a3d60 0x7f70640a6220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:48.882 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.887+0000 7f7081c00640 1 --2- 172.21.15.90:0/2162411399 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f70640a3d60 0x7f70640a6220 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f7070009ba0 tx=0x7f7070005f90 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:47:48.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:48.891+0000 7f707a000640 1 -- 172.21.15.90:0/2162411399 <== mon.2 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7f70680be590 con 0x7f707c069600 2024-04-02T14:47:49.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.287+0000 7f7083000640 1 -- 172.21.15.90:0/2162411399 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f707c068810 con 0x7f707c069600 2024-04-02T14:47:49.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.289+0000 7f707a000640 1 -- 172.21.15.90:0/2162411399 <== mon.2 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11927 (secure 0 0 0) 0x7f70680bde90 con 0x7f707c069600 2024-04-02T14:47:49.286 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-02T14:47:49.286 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14412,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6837/2852160309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6836","nonce":2852160309},{"type":"v1","addr":"172.21.15.90:6837","nonce":2852160309}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14478,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6841/1919376858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6840","nonce":1919376858},{"type":"v1","addr":"172.21.15.90:6841","nonce":1919376858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-02T14:32:22.319083+0000","modified":"2024-04-02T14:34:52.266152+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24590,"mds_1":24560,"mds_2":24547,"mds_3":24539,"mds_4":24523},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24454":{"gid":24454,"name":"l","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6833/1564319431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":1564319431},{"type":"v1","addr":"172.21.15.177:6833","nonce":1564319431}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.92:6833/1570337372","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":1570337372},{"type":"v1","addr":"172.21.15.92:6833","nonce":1570337372}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"j","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.90:6835/1277124891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6834","nonce":1277124891},{"type":"v1","addr":"172.21.15.90:6835","nonce":1277124891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.92:6835/860721164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":860721164},{"type":"v1","addr":"172.21.15.92:6835","nonce":860721164}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24511":{"gid":24511,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6835/3177244988","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":3177244988},{"type":"v1","addr":"172.21.15.177:6835","nonce":3177244988}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"c","rank":4,"incarnation":25,"state":"up:active","state_seq":9,"addr":"172.21.15.177:6837/4223414858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":4223414858},{"type":"v1","addr":"172.21.15.177:6837","nonce":4223414858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24539":{"gid":24539,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":8,"addr":"172.21.15.92:6837/1270635939","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":1270635939},{"type":"v1","addr":"172.21.15.92:6837","nonce":1270635939}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24547":{"gid":24547,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":7,"addr":"172.21.15.177:6839/664895663","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":664895663},{"type":"v1","addr":"172.21.15.177:6839","nonce":664895663}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"g","rank":1,"incarnation":19,"state":"up:active","state_seq":5,"addr":"172.21.15.90:6839/1498915870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6838","nonce":1498915870},{"type":"v1","addr":"172.21.15.90:6839","nonce":1498915870}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24590":{"gid":24590,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.92:6839/1054888219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":1054888219},{"type":"v1","addr":"172.21.15.92:6839","nonce":1054888219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24590,"qdb_cluster":[24547,24590,24523,24539,24560]},"id":1}]} 2024-04-02T14:47:49.289 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 27 2024-04-02T14:47:49.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.296+0000 7f7083000640 1 -- 172.21.15.90:0/2162411399 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f70640a3d60 msgr2=0x7f70640a6220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:49.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.296+0000 7f7083000640 1 --2- 172.21.15.90:0/2162411399 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f70640a3d60 0x7f70640a6220 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f7070009ba0 tx=0x7f7070005f90 comp rx=0 tx=0).stop 2024-04-02T14:47:49.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.296+0000 7f7083000640 1 -- 172.21.15.90:0/2162411399 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f707c069600 msgr2=0x7f707c1a4ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:49.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.296+0000 7f7083000640 1 --2- 172.21.15.90:0/2162411399 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f707c069600 0x7f707c1a4ae0 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f7068000c00 tx=0x7f7068005720 comp rx=0 tx=0).stop 2024-04-02T14:47:49.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.297+0000 7f7083000640 1 -- 172.21.15.90:0/2162411399 shutdown_connections 2024-04-02T14:47:49.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.297+0000 7f7083000640 1 --2- 172.21.15.90:0/2162411399 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f70640a3d60 0x7f70640a6220 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:49.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.297+0000 7f7083000640 1 --2- 172.21.15.90:0/2162411399 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f707c107950 0x7f707c19e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:49.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.297+0000 7f7083000640 1 --2- 172.21.15.90:0/2162411399 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f707c069600 0x7f707c1a4ae0 secure :-1 s=CLOSED pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f7068000c00 tx=0x7f7068005720 comp rx=0 tx=0).stop 2024-04-02T14:47:49.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.297+0000 7f7083000640 1 --2- 172.21.15.90:0/2162411399 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f707c068c60 0x7f707c1a45a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:49.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.297+0000 7f7083000640 1 -- 172.21.15.90:0/2162411399 >> 172.21.15.90:0/2162411399 conn(0x7f707c076410 msgr2=0x7f707c0feb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:47:49.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.299+0000 7f7083000640 1 -- 172.21.15.90:0/2162411399 shutdown_connections 2024-04-02T14:47:49.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.299+0000 7f7083000640 1 -- 172.21.15.90:0/2162411399 wait complete. 2024-04-02T14:47:49.325 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-04-02T14:47:49.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.590+0000 7f0889200640 1 -- 172.21.15.90:0/3336256867 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f08841048c0 msgr2=0x7f0884108900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:49.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.590+0000 7f0889200640 1 --2- 172.21.15.90:0/3336256867 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f08841048c0 0x7f0884108900 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f0870009900 tx=0x7f087002bc30 comp rx=0 tx=0).stop 2024-04-02T14:47:49.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.591+0000 7f0889200640 1 -- 172.21.15.90:0/3336256867 shutdown_connections 2024-04-02T14:47:49.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.591+0000 7f0889200640 1 --2- 172.21.15.90:0/3336256867 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0884108fe0 0x7f0884110bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:49.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.591+0000 7f0889200640 1 --2- 172.21.15.90:0/3336256867 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f08841048c0 0x7f0884108900 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:49.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.591+0000 7f0889200640 1 --2- 172.21.15.90:0/3336256867 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0884103f10 0x7f08841042f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:49.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.591+0000 7f0889200640 1 -- 172.21.15.90:0/3336256867 >> 172.21.15.90:0/3336256867 conn(0x7f08840ff770 msgr2=0x7f0884101b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:47:49.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.591+0000 7f0889200640 1 -- 172.21.15.90:0/3336256867 shutdown_connections 2024-04-02T14:47:49.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.592+0000 7f0889200640 1 -- 172.21.15.90:0/3336256867 wait complete. 2024-04-02T14:47:49.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.593+0000 7f0889200640 1 Processor -- start 2024-04-02T14:47:49.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.593+0000 7f0889200640 1 -- start start 2024-04-02T14:47:49.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.593+0000 7f0889200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0884103f10 0x7f088419a600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:49.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.593+0000 7f0889200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0884108fe0 0x7f088419ab40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:49.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.593+0000 7f0889200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f088419ee70 0x7f088419f320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:49.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.593+0000 7f0889200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f08841140c0 con 0x7f0884103f10 2024-04-02T14:47:49.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.593+0000 7f0889200640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f0884113f40 con 0x7f0884108fe0 2024-04-02T14:47:49.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.593+0000 7f0889200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f0884114240 con 0x7f088419ee70 2024-04-02T14:47:49.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.594+0000 7f0881e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0884103f10 0x7f088419a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:49.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.594+0000 7f0882800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f088419ee70 0x7f088419f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:49.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.594+0000 7f0882800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f088419ee70 0x7f088419f320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:34520/0 (socket says 172.21.15.90:34520) 2024-04-02T14:47:49.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.594+0000 7f0882800640 1 -- 172.21.15.90:0/1569095777 learned_addr learned my addr 172.21.15.90:0/1569095777 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:47:49.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.594+0000 7f0881400640 1 --2- 172.21.15.90:0/1569095777 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0884108fe0 0x7f088419ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:49.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.594+0000 7f0882800640 1 -- 172.21.15.90:0/1569095777 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0884108fe0 msgr2=0x7f088419ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:49.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.594+0000 7f0882800640 1 --2- 172.21.15.90:0/1569095777 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0884108fe0 0x7f088419ab40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:49.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.594+0000 7f0882800640 1 -- 172.21.15.90:0/1569095777 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0884103f10 msgr2=0x7f088419a600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:49.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.594+0000 7f0882800640 1 --2- 172.21.15.90:0/1569095777 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0884103f10 0x7f088419a600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:49.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.594+0000 7f0882800640 1 -- 172.21.15.90:0/1569095777 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f088419f960 con 0x7f088419ee70 2024-04-02T14:47:49.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.594+0000 7f0881e00640 1 --2- 172.21.15.90:0/1569095777 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0884103f10 0x7f088419a600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:47:49.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.595+0000 7f0882800640 1 --2- 172.21.15.90:0/1569095777 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f088419ee70 0x7f088419f320 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f087c00eb10 tx=0x7f087c00c2d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:47:49.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.595+0000 7f087aa00640 1 -- 172.21.15.90:0/1569095777 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f087c004b20 con 0x7f088419ee70 2024-04-02T14:47:49.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.595+0000 7f087aa00640 1 -- 172.21.15.90:0/1569095777 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f087c004cc0 con 0x7f088419ee70 2024-04-02T14:47:49.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.595+0000 7f087aa00640 1 -- 172.21.15.90:0/1569095777 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f087c010760 con 0x7f088419ee70 2024-04-02T14:47:49.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.595+0000 7f0889200640 1 -- 172.21.15.90:0/1569095777 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f088419fc50 con 0x7f088419ee70 2024-04-02T14:47:49.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.595+0000 7f0889200640 1 -- 172.21.15.90:0/1569095777 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f08841a7410 con 0x7f088419ee70 2024-04-02T14:47:49.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.600+0000 7f087aa00640 1 -- 172.21.15.90:0/1569095777 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f087c005020 con 0x7f088419ee70 2024-04-02T14:47:49.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.601+0000 7f088bfa0740 1 Processor -- start 2024-04-02T14:47:49.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.601+0000 7f088bfa0740 1 -- start start 2024-04-02T14:47:49.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.601+0000 7f088bfa0740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cb94bc69f0 0x55cb94bc5740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:49.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.601+0000 7f088bfa0740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cb94bc5d10 0x55cb94bc6190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:49.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.601+0000 7f088bfa0740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55cb94bd7890 0x55cb94bd9c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:49.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.601+0000 7f088bfa0740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55cb94b06400 con 0x55cb94bc5d10 2024-04-02T14:47:49.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.601+0000 7f088bfa0740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55cb94aeaf10 con 0x55cb94bd7890 2024-04-02T14:47:49.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.601+0000 7f088bfa0740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55cb94b2a6f0 con 0x55cb94bc69f0 2024-04-02T14:47:49.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.602+0000 7f087aa00640 1 --2- 172.21.15.90:0/1569095777 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f08680a3d60 0x7f08680a6220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:49.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.602+0000 7f0881400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cb94bc69f0 0x55cb94bc5740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:49.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.602+0000 7f0881e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cb94bc5d10 0x55cb94bc6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:49.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.602+0000 7f0881400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cb94bc69f0 0x55cb94bc5740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:34534/0 (socket says 172.21.15.90:34534) 2024-04-02T14:47:49.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.602+0000 7f0881400640 1 -- 172.21.15.90:0/3371207321 learned_addr learned my addr 172.21.15.90:0/3371207321 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:47:49.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.602+0000 7f0881400640 1 --2- 172.21.15.90:0/3371207321 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55cb94bd7890 0x55cb94bd9c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:49.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.602+0000 7f087aa00640 1 -- 172.21.15.90:0/1569095777 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 8550+0+0 (secure 0 0 0) 0x7f087c0d0e80 con 0x7f088419ee70 2024-04-02T14:47:49.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.602+0000 7f0882800640 1 --2- 172.21.15.90:0/1569095777 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f08680a3d60 0x7f08680a6220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:49.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.602+0000 7f0881400640 1 -- 172.21.15.90:0/3371207321 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55cb94bd7890 msgr2=0x55cb94bd9c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:47:49.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.602+0000 7f0881400640 1 --2- 172.21.15.90:0/3371207321 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55cb94bd7890 0x55cb94bd9c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:49.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.602+0000 7f0881400640 1 -- 172.21.15.90:0/3371207321 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cb94bc5d10 msgr2=0x55cb94bc6190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:47:49.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.602+0000 7f0881400640 1 --2- 172.21.15.90:0/3371207321 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cb94bc5d10 0x55cb94bc6190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:49.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.602+0000 7f0881400640 1 -- 172.21.15.90:0/3371207321 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cb9483e3f0 con 0x55cb94bc69f0 2024-04-02T14:47:49.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.603+0000 7f0881400640 1 --2- 172.21.15.90:0/3371207321 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cb94bc69f0 0x55cb94bc5740 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f0870006010 tx=0x7f087002bc30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d4cbd87152b21013 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:47:49.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.603+0000 7f086f400640 1 -- 172.21.15.90:0/3371207321 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f087002c880 con 0x55cb94bc69f0 2024-04-02T14:47:49.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.603+0000 7f086f400640 1 -- 172.21.15.90:0/3371207321 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0870005530 con 0x55cb94bc69f0 2024-04-02T14:47:49.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.603+0000 7f086f400640 1 -- 172.21.15.90:0/3371207321 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08700049b0 con 0x55cb94bc69f0 2024-04-02T14:47:49.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.603+0000 7f0882800640 1 --2- 172.21.15.90:0/1569095777 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f08680a3d60 0x7f08680a6220 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f087c0a5540 tx=0x7f087c005b40 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:47:49.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.604+0000 7f088bfa0740 1 -- 172.21.15.90:0/3371207321 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cb94bc69f0 msgr2=0x55cb94bc5740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:49.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.604+0000 7f088bfa0740 1 --2- 172.21.15.90:0/3371207321 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cb94bc69f0 0x55cb94bc5740 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f0870006010 tx=0x7f087002bc30 comp rx=0 tx=0).stop 2024-04-02T14:47:49.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.605+0000 7f088bfa0740 1 -- 172.21.15.90:0/3371207321 shutdown_connections 2024-04-02T14:47:49.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.605+0000 7f088bfa0740 1 --2- 172.21.15.90:0/3371207321 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55cb94bd7890 0x55cb94bd9c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:49.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.605+0000 7f088bfa0740 1 --2- 172.21.15.90:0/3371207321 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cb94bc5d10 0x55cb94bc6190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:49.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.605+0000 7f088bfa0740 1 --2- 172.21.15.90:0/3371207321 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cb94bc69f0 0x55cb94bc5740 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:49.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.605+0000 7f088bfa0740 1 -- 172.21.15.90:0/3371207321 >> 172.21.15.90:0/3371207321 conn(0x55cb94b8ca50 msgr2=0x55cb94bc0760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:47:49.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.606+0000 7f088bfa0740 1 -- 172.21.15.90:0/3371207321 shutdown_connections 2024-04-02T14:47:49.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.606+0000 7f088bfa0740 1 -- 172.21.15.90:0/3371207321 wait complete. 2024-04-02T14:47:49.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.607+0000 7f088bfa0740 1 -- start start 2024-04-02T14:47:49.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.608+0000 7f088bfa0740 1 Processor -- start 2024-04-02T14:47:49.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.609+0000 7f088bfa0740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cb94bc5d10 0x55cb94bd48a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:49.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.609+0000 7f088bfa0740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55cb94bc69f0 0x55cb94bd4de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:49.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.609+0000 7f0881400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cb94bc5d10 0x55cb94bd48a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:49.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.609+0000 7f0881400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cb94bc5d10 0x55cb94bd48a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:34546/0 (socket says 172.21.15.90:34546) 2024-04-02T14:47:49.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.610+0000 7f088bfa0740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cb94bd7890 0x55cb94bd5320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:49.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.610+0000 7f0881400640 1 -- 172.21.15.90:0/3778804836 learned_addr learned my addr 172.21.15.90:0/3778804836 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:47:49.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.610+0000 7f088bfa0740 1 -- 172.21.15.90:0/3778804836 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55cb94b266a0 con 0x55cb94bd7890 2024-04-02T14:47:49.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.610+0000 7f088bfa0740 1 -- 172.21.15.90:0/3778804836 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55cb94ad2980 con 0x55cb94bc69f0 2024-04-02T14:47:49.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.610+0000 7f088bfa0740 1 -- 172.21.15.90:0/3778804836 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55cb94ae5cb0 con 0x55cb94bc5d10 2024-04-02T14:47:49.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.610+0000 7f088bfa0740 10 client.0 _ll_register_callbacks cb 0x55cb94ae56a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:47:49.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.610+0000 7f0881e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55cb94bc69f0 0x55cb94bd4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:49.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.610+0000 7f0881400640 1 --2- 172.21.15.90:0/3778804836 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cb94bd7890 0x55cb94bd5320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:49.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.610+0000 7f0881400640 1 -- 172.21.15.90:0/3778804836 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55cb94bc69f0 msgr2=0x55cb94bd4de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:47:49.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.610+0000 7f0881400640 1 --2- 172.21.15.90:0/3778804836 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55cb94bc69f0 0x55cb94bd4de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:49.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.610+0000 7f0881400640 1 -- 172.21.15.90:0/3778804836 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cb94bd7890 msgr2=0x55cb94bd5320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:47:49.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.610+0000 7f0881400640 1 --2- 172.21.15.90:0/3778804836 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cb94bd7890 0x55cb94bd5320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:49.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.610+0000 7f0881400640 1 -- 172.21.15.90:0/3778804836 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cb949087d0 con 0x55cb94bc5d10 2024-04-02T14:47:49.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.610+0000 7f0881400640 1 --2- 172.21.15.90:0/3778804836 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cb94bc5d10 0x55cb94bd48a0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f08700096c0 tx=0x7f087002c2b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=717bd068a971745b server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:47:49.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.611+0000 7f0863e00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:47:49.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.611+0000 7f0863e00640 1 -- 172.21.15.90:0/3778804836 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0870057060 con 0x55cb94bc5d10 2024-04-02T14:47:49.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.611+0000 7f0863e00640 1 -- 172.21.15.90:0/3778804836 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 8550+0+0 (secure 0 0 0) 0x7f0870003b40 con 0x55cb94bc5d10 2024-04-02T14:47:49.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.612+0000 7f088bfa0740 1 -- 172.21.15.90:0/3778804836 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0870005530 con 0x55cb94bc5d10 2024-04-02T14:47:49.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.613+0000 7f0863e00640 1 -- 172.21.15.90:0/3778804836 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f0870003d40 con 0x55cb94bc5d10 2024-04-02T14:47:49.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.613+0000 7f088bfa0740 10 client.29236 fetch_fsmap learned FSMap version 27 2024-04-02T14:47:49.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.613+0000 7f088bfa0740 1 -- 172.21.15.90:0/3778804836 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55cb94be66c0 con 0x55cb94bc5d10 2024-04-02T14:47:49.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.613+0000 7f0863e00640 1 -- 172.21.15.90:0/3778804836 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0870005530 con 0x55cb94bc5d10 2024-04-02T14:47:49.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.614+0000 7f0863e00640 1 -- 172.21.15.90:0/3778804836 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f0870034440 con 0x55cb94bc5d10 2024-04-02T14:47:49.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.614+0000 7f088bfa0740 10 client.29236 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:47:49.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.614+0000 7f088bfa0740 10 client.29236 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.c 2024-04-02T14:47:49.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.614+0000 7f088bfa0740 20 client.29236 populate_metadata read hostname 'smithi090' 2024-04-02T14:47:49.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.614+0000 7f088bfa0740 1 --2- 172.21.15.90:0/3778804836 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x55cb94be7a10 0x55cb94be9e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:49.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.614+0000 7f088bfa0740 4 client.29236 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:47:49.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.614+0000 7f088bfa0740 1 -- 172.21.15.90:0/3778804836 --> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cb94aea040 con 0x55cb94be7a10 2024-04-02T14:47:49.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.615+0000 7f0881e00640 1 --2- 172.21.15.90:0/3778804836 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x55cb94be7a10 0x55cb94be9e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:49.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.615+0000 7f0881e00640 1 --2- 172.21.15.90:0/3778804836 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x55cb94be7a10 0x55cb94be9e00 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=c0fe39f4e12834a0 server_cookie=de58339ee7ecbd85 in_seq=0 out_seq=0 2024-04-02T14:47:49.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.615+0000 7f0863e00640 10 client.29236 ms_handle_connect on v2:172.21.15.177:6836/4223414858 2024-04-02T14:47:49.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.616+0000 7f0863e00640 1 -- 172.21.15.90:0/3778804836 <== mds.4 v2:172.21.15.177:6836/4223414858 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55cb94aea040 con 0x55cb94be7a10 2024-04-02T14:47:49.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.616+0000 7f0863e00640 10 client.29236 handle_command_reply: tid=0 2024-04-02T14:47:49.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.616+0000 7f088bfa0740 1 client.29236 shutdown 2024-04-02T14:47:49.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.616+0000 7f088bfa0740 1 -- 172.21.15.90:0/3778804836 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cb94bc5d10 msgr2=0x55cb94bd48a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:49.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.616+0000 7f088bfa0740 1 --2- 172.21.15.90:0/3778804836 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cb94bc5d10 0x55cb94bd48a0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f08700096c0 tx=0x7f087002c2b0 comp rx=0 tx=0).stop 2024-04-02T14:47:49.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.616+0000 7f088bfa0740 1 -- 172.21.15.90:0/3778804836 shutdown_connections 2024-04-02T14:47:49.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.616+0000 7f088bfa0740 1 --2- 172.21.15.90:0/3778804836 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x55cb94be7a10 0x55cb94be9e00 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:49.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.616+0000 7f088bfa0740 1 --2- 172.21.15.90:0/3778804836 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cb94bd7890 0x55cb94bd5320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:49.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.616+0000 7f088bfa0740 1 --2- 172.21.15.90:0/3778804836 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55cb94bc69f0 0x55cb94bd4de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:49.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.616+0000 7f088bfa0740 1 --2- 172.21.15.90:0/3778804836 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cb94bc5d10 0x55cb94bd48a0 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:49.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.616+0000 7f088bfa0740 1 -- 172.21.15.90:0/3778804836 >> 172.21.15.90:0/3778804836 conn(0x55cb94b8ca50 msgr2=0x55cb94bc24b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:47:49.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.616+0000 7f0863e00640 0 client.29236 ms_handle_reset on v2:172.21.15.177:6836/4223414858 2024-04-02T14:47:49.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.617+0000 7f088bfa0740 1 -- 172.21.15.90:0/3778804836 shutdown_connections 2024-04-02T14:47:49.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.617+0000 7f088bfa0740 1 -- 172.21.15.90:0/3778804836 wait complete. 2024-04-02T14:47:49.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.617+0000 7f088bfa0740 20 client.29236 trim_cache size 0 max 16384 2024-04-02T14:47:49.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.638+0000 7f088bfa0740 1 Processor -- start 2024-04-02T14:47:49.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.639+0000 7f088bfa0740 1 -- start start 2024-04-02T14:47:49.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.639+0000 7f088bfa0740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55cb94bc69f0 0x55cb94bd8c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:49.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.639+0000 7f088bfa0740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cb94bd7890 0x55cb94bd91c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:49.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.639+0000 7f088bfa0740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cb94bc09a0 0x55cb94bd4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:49.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.639+0000 7f088bfa0740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f08841ab6d0 con 0x55cb94bd7890 2024-04-02T14:47:49.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.639+0000 7f088bfa0740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f08841ad210 con 0x55cb94bc69f0 2024-04-02T14:47:49.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.639+0000 7f088bfa0740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f08841abda0 con 0x55cb94bc09a0 2024-04-02T14:47:49.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.639+0000 7f0881400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55cb94bc69f0 0x55cb94bd8c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:49.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.639+0000 7f0881400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55cb94bc69f0 0x55cb94bd8c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:50062/0 (socket says 172.21.15.90:50062) 2024-04-02T14:47:49.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.639+0000 7f0881400640 1 -- 172.21.15.90:0/705861643 learned_addr learned my addr 172.21.15.90:0/705861643 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:47:49.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.639+0000 7f0881e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cb94bd7890 0x55cb94bd91c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:49.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.640+0000 7f0881400640 1 --2- 172.21.15.90:0/705861643 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cb94bc09a0 0x55cb94bd4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:49.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.640+0000 7f0881e00640 1 -- 172.21.15.90:0/705861643 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cb94bc09a0 msgr2=0x55cb94bd4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:47:49.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.640+0000 7f0881e00640 1 --2- 172.21.15.90:0/705861643 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cb94bc09a0 0x55cb94bd4860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:49.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.640+0000 7f0881e00640 1 -- 172.21.15.90:0/705861643 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55cb94bc69f0 msgr2=0x55cb94bd8c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:47:49.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.640+0000 7f0881e00640 1 --2- 172.21.15.90:0/705861643 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55cb94bc69f0 0x55cb94bd8c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:49.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.640+0000 7f0881e00640 1 -- 172.21.15.90:0/705861643 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cb94bc4d40 con 0x55cb94bd7890 2024-04-02T14:47:49.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.640+0000 7f0881400640 1 --2- 172.21.15.90:0/705861643 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55cb94bc69f0 0x55cb94bd8c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:47:49.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.640+0000 7f0881e00640 1 --2- 172.21.15.90:0/705861643 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cb94bd7890 0x55cb94bd91c0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x55cb94ad2aa0 tx=0x7f08740046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1416b4636cdf6585 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:47:49.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.640+0000 7f086ea00640 1 -- 172.21.15.90:0/705861643 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0874012310 con 0x55cb94bd7890 2024-04-02T14:47:49.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.640+0000 7f086ea00640 1 -- 172.21.15.90:0/705861643 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f08740124b0 con 0x55cb94bd7890 2024-04-02T14:47:49.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.640+0000 7f086ea00640 1 -- 172.21.15.90:0/705861643 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0874010670 con 0x55cb94bd7890 2024-04-02T14:47:49.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.641+0000 7f088bfa0740 1 -- 172.21.15.90:0/705861643 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cb94bd7890 msgr2=0x55cb94bd91c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:49.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.641+0000 7f088bfa0740 1 --2- 172.21.15.90:0/705861643 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cb94bd7890 0x55cb94bd91c0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x55cb94ad2aa0 tx=0x7f08740046c0 comp rx=0 tx=0).stop 2024-04-02T14:47:49.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.641+0000 7f088bfa0740 1 -- 172.21.15.90:0/705861643 shutdown_connections 2024-04-02T14:47:49.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.641+0000 7f088bfa0740 1 --2- 172.21.15.90:0/705861643 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cb94bc09a0 0x55cb94bd4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:49.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.641+0000 7f088bfa0740 1 --2- 172.21.15.90:0/705861643 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cb94bd7890 0x55cb94bd91c0 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:49.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.641+0000 7f088bfa0740 1 --2- 172.21.15.90:0/705861643 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55cb94bc69f0 0x55cb94bd8c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:49.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.641+0000 7f088bfa0740 1 -- 172.21.15.90:0/705861643 >> 172.21.15.90:0/705861643 conn(0x55cb94b8ca50 msgr2=0x55cb94bc0290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:47:49.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.641+0000 7f088bfa0740 1 -- 172.21.15.90:0/705861643 shutdown_connections 2024-04-02T14:47:49.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.641+0000 7f088bfa0740 1 -- 172.21.15.90:0/705861643 wait complete. 2024-04-02T14:47:49.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.642+0000 7f088bfa0740 1 -- start start 2024-04-02T14:47:49.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.642+0000 7f088bfa0740 1 Processor -- start 2024-04-02T14:47:49.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.643+0000 7f088bfa0740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cb94bc09a0 0x55cb94beccc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:49.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.643+0000 7f088bfa0740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cb94bc69f0 0x55cb94be80d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:49.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.643+0000 7f088bfa0740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55cb94bd7890 0x55cb94be8610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:49.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.643+0000 7f088bfa0740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f08841acb40 con 0x55cb94bc09a0 2024-04-02T14:47:49.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.643+0000 7f088bfa0740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f08841ac470 con 0x55cb94bd7890 2024-04-02T14:47:49.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.643+0000 7f088bfa0740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f08841a96d0 con 0x55cb94bc69f0 2024-04-02T14:47:49.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.643+0000 7f088bfa0740 10 client.0 _ll_register_callbacks cb 0x7f0870004400 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:47:49.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.643+0000 7f0881400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cb94bc09a0 0x55cb94beccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:49.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.643+0000 7f0881400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cb94bc09a0 0x55cb94beccc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33882/0 (socket says 172.21.15.90:33882) 2024-04-02T14:47:49.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.643+0000 7f0881400640 1 -- 172.21.15.90:0/729546183 learned_addr learned my addr 172.21.15.90:0/729546183 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:47:49.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.643+0000 7f0881400640 1 --2- 172.21.15.90:0/729546183 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55cb94bd7890 0x55cb94be8610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:49.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.643+0000 7f0881e00640 1 --2- 172.21.15.90:0/729546183 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cb94bc69f0 0x55cb94be80d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:49.640 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.644+0000 7f0881400640 1 -- 172.21.15.90:0/729546183 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cb94bc69f0 msgr2=0x55cb94be80d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:47:49.640 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.644+0000 7f0881400640 1 --2- 172.21.15.90:0/729546183 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cb94bc69f0 0x55cb94be80d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:49.640 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.644+0000 7f0881400640 1 -- 172.21.15.90:0/729546183 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55cb94bd7890 msgr2=0x55cb94be8610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:47:49.640 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.644+0000 7f0881400640 1 --2- 172.21.15.90:0/729546183 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55cb94bd7890 0x55cb94be8610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:49.640 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.644+0000 7f0881400640 1 -- 172.21.15.90:0/729546183 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cb94be8cb0 con 0x55cb94bc09a0 2024-04-02T14:47:49.640 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.644+0000 7f0881400640 1 --2- 172.21.15.90:0/729546183 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cb94bc09a0 0x55cb94beccc0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f087002c420 tx=0x7f087002bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7a9e95c098b58c5 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:47:49.640 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.645+0000 7f0863e00640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-02T14:47:49.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.646+0000 7f0863e00640 1 -- 172.21.15.90:0/729546183 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0870057040 con 0x55cb94bc09a0 2024-04-02T14:47:49.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.646+0000 7f0863e00640 1 -- 172.21.15.90:0/729546183 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 8550+0+0 (secure 0 0 0) 0x7f0870034440 con 0x55cb94bc09a0 2024-04-02T14:47:49.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.646+0000 7f088bfa0740 1 -- 172.21.15.90:0/729546183 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f08740124b0 con 0x55cb94bc09a0 2024-04-02T14:47:49.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.647+0000 7f0863e00640 1 -- 172.21.15.90:0/729546183 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f0870034a80 con 0x55cb94bc09a0 2024-04-02T14:47:49.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.647+0000 7f088bfa0740 10 client.18342 fetch_fsmap learned FSMap version 27 2024-04-02T14:47:49.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.647+0000 7f088bfa0740 1 -- 172.21.15.90:0/729546183 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55cb94c4da90 con 0x55cb94bc09a0 2024-04-02T14:47:49.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.648+0000 7f0863e00640 1 -- 172.21.15.90:0/729546183 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f08740124b0 con 0x55cb94bc09a0 2024-04-02T14:47:49.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.648+0000 7f0863e00640 1 -- 172.21.15.90:0/729546183 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f087006f920 con 0x55cb94bc09a0 2024-04-02T14:47:49.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.651+0000 7f088bfa0740 10 client.18342 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:47:49.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.651+0000 7f088bfa0740 10 client.18342 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.c 2024-04-02T14:47:49.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.651+0000 7f088bfa0740 20 client.18342 populate_metadata read hostname 'smithi090' 2024-04-02T14:47:49.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.651+0000 7f088bfa0740 1 --2- 172.21.15.90:0/729546183 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x55cb94c4e110 0x55cb94c50530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:49.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.651+0000 7f088bfa0740 4 client.18342 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-02T14:47:49.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.651+0000 7f088bfa0740 1 -- 172.21.15.90:0/729546183 --> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55cb94b37c30 con 0x55cb94c4e110 2024-04-02T14:47:49.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.652+0000 7f0881e00640 1 --2- 172.21.15.90:0/729546183 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x55cb94c4e110 0x55cb94c50530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:49.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.654+0000 7f0881e00640 1 --2- 172.21.15.90:0/729546183 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x55cb94c4e110 0x55cb94c50530 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=761820f8fbdab45b server_cookie=4db32b1724d88ff2 in_seq=0 out_seq=0 2024-04-02T14:47:49.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.655+0000 7f0863e00640 10 client.18342 ms_handle_connect on v2:172.21.15.177:6836/4223414858 2024-04-02T14:47:49.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.655+0000 7f0863e00640 1 -- 172.21.15.90:0/729546183 <== mds.4 v2:172.21.15.177:6836/4223414858 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55cb94b37c30 con 0x55cb94c4e110 2024-04-02T14:47:49.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.655+0000 7f0863e00640 10 client.18342 handle_command_reply: tid=0 2024-04-02T14:47:49.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.655+0000 7f088bfa0740 1 client.18342 shutdown 2024-04-02T14:47:49.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.655+0000 7f088bfa0740 1 -- 172.21.15.90:0/729546183 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cb94bc09a0 msgr2=0x55cb94beccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:49.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.655+0000 7f088bfa0740 1 --2- 172.21.15.90:0/729546183 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cb94bc09a0 0x55cb94beccc0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f087002c420 tx=0x7f087002bc30 comp rx=0 tx=0).stop 2024-04-02T14:47:49.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.658+0000 7f088bfa0740 1 -- 172.21.15.90:0/729546183 shutdown_connections 2024-04-02T14:47:49.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.658+0000 7f088bfa0740 1 --2- 172.21.15.90:0/729546183 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55cb94bd7890 0x55cb94be8610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:49.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.658+0000 7f088bfa0740 1 --2- 172.21.15.90:0/729546183 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55cb94bc69f0 0x55cb94be80d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:49.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.658+0000 7f088bfa0740 1 --2- 172.21.15.90:0/729546183 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x55cb94c4e110 0x55cb94c50530 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:49.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.658+0000 7f088bfa0740 1 --2- 172.21.15.90:0/729546183 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55cb94bc09a0 0x55cb94beccc0 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:49.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.658+0000 7f088bfa0740 1 -- 172.21.15.90:0/729546183 >> 172.21.15.90:0/729546183 conn(0x55cb94b8ca50 msgr2=0x55cb94bd4840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:47:49.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.658+0000 7f0863e00640 0 client.18342 ms_handle_reset on v2:172.21.15.177:6836/4223414858 2024-04-02T14:47:49.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.659+0000 7f088bfa0740 1 -- 172.21.15.90:0/729546183 shutdown_connections 2024-04-02T14:47:49.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.659+0000 7f088bfa0740 1 -- 172.21.15.90:0/729546183 wait complete. 2024-04-02T14:47:49.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.660+0000 7f088bfa0740 20 client.18342 trim_cache size 0 max 16384 2024-04-02T14:47:49.655 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-02T14:47:49.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.661+0000 7f0889200640 1 -- 172.21.15.90:0/1569095777 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f08680a3d60 msgr2=0x7f08680a6220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:49.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.661+0000 7f0889200640 1 --2- 172.21.15.90:0/1569095777 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f08680a3d60 0x7f08680a6220 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f087c0a5540 tx=0x7f087c005b40 comp rx=0 tx=0).stop 2024-04-02T14:47:49.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.661+0000 7f0889200640 1 -- 172.21.15.90:0/1569095777 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f088419ee70 msgr2=0x7f088419f320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:49.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.661+0000 7f0889200640 1 --2- 172.21.15.90:0/1569095777 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f088419ee70 0x7f088419f320 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f087c00eb10 tx=0x7f087c00c2d0 comp rx=0 tx=0).stop 2024-04-02T14:47:49.657 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.663+0000 7f0889200640 1 -- 172.21.15.90:0/1569095777 shutdown_connections 2024-04-02T14:47:49.658 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.663+0000 7f0889200640 1 --2- 172.21.15.90:0/1569095777 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f08680a3d60 0x7f08680a6220 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:49.658 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.663+0000 7f0889200640 1 --2- 172.21.15.90:0/1569095777 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f088419ee70 0x7f088419f320 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:49.658 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.663+0000 7f0889200640 1 --2- 172.21.15.90:0/1569095777 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0884108fe0 0x7f088419ab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:49.658 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.663+0000 7f0889200640 1 --2- 172.21.15.90:0/1569095777 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0884103f10 0x7f088419a600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:49.658 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.663+0000 7f0889200640 1 -- 172.21.15.90:0/1569095777 >> 172.21.15.90:0/1569095777 conn(0x7f08840ff770 msgr2=0x7f0884100b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:47:49.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.665+0000 7f0889200640 1 -- 172.21.15.90:0/1569095777 shutdown_connections 2024-04-02T14:47:49.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.665+0000 7f0889200640 1 -- 172.21.15.90:0/1569095777 wait complete. 2024-04-02T14:47:49.690 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-04-02T14:47:49.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.961+0000 7fe4e6a00640 1 -- 172.21.15.90:0/778971757 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe4e0103f10 msgr2=0x7fe4e01042f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:49.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.961+0000 7fe4e6a00640 1 --2- 172.21.15.90:0/778971757 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe4e0103f10 0x7fe4e01042f0 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7fe4d0009900 tx=0x7fe4d002bc30 comp rx=0 tx=0).stop 2024-04-02T14:47:49.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.962+0000 7fe4e6a00640 1 -- 172.21.15.90:0/778971757 shutdown_connections 2024-04-02T14:47:49.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.962+0000 7fe4e6a00640 1 --2- 172.21.15.90:0/778971757 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe4e0108fe0 0x7fe4e0110bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:49.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.962+0000 7fe4e6a00640 1 --2- 172.21.15.90:0/778971757 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe4e01048c0 0x7fe4e0108900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:49.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.962+0000 7fe4e6a00640 1 --2- 172.21.15.90:0/778971757 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe4e0103f10 0x7fe4e01042f0 unknown :-1 s=CLOSED pgs=894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:49.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.962+0000 7fe4e6a00640 1 -- 172.21.15.90:0/778971757 >> 172.21.15.90:0/778971757 conn(0x7fe4e00ff7b0 msgr2=0x7fe4e0101bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:47:49.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.962+0000 7fe4e6a00640 1 -- 172.21.15.90:0/778971757 shutdown_connections 2024-04-02T14:47:49.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.963+0000 7fe4e6a00640 1 -- 172.21.15.90:0/778971757 wait complete. 2024-04-02T14:47:49.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.964+0000 7fe4e6a00640 1 Processor -- start 2024-04-02T14:47:49.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.964+0000 7fe4e6a00640 1 -- start start 2024-04-02T14:47:49.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.965+0000 7fe4e6a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe4e0103f10 0x7fe4e0110a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:49.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.965+0000 7fe4e6a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe4e0108fe0 0x7fe4e010dad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:49.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.965+0000 7fe4e6a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe4e010e010 0x7fe4e010e4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:49.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.965+0000 7fe4e6a00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fe4e01140e0 con 0x7fe4e0103f10 2024-04-02T14:47:49.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.965+0000 7fe4e6a00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fe4e0113f60 con 0x7fe4e010e010 2024-04-02T14:47:49.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.965+0000 7fe4e6a00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fe4e0114260 con 0x7fe4e0108fe0 2024-04-02T14:47:49.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.965+0000 7fe4e4a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe4e010e010 0x7fe4e010e4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:49.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.965+0000 7fe4e4a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe4e010e010 0x7fe4e010e4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:50108/0 (socket says 172.21.15.90:50108) 2024-04-02T14:47:49.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.965+0000 7fe4e4a00640 1 -- 172.21.15.90:0/4152071845 learned_addr learned my addr 172.21.15.90:0/4152071845 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:47:49.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.965+0000 7fe4df400640 1 --2- 172.21.15.90:0/4152071845 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe4e0108fe0 0x7fe4e010dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:49.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.965+0000 7fe4dfe00640 1 --2- 172.21.15.90:0/4152071845 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe4e0103f10 0x7fe4e0110a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:49.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.966+0000 7fe4df400640 1 -- 172.21.15.90:0/4152071845 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe4e010e010 msgr2=0x7fe4e010e4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:49.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.966+0000 7fe4df400640 1 --2- 172.21.15.90:0/4152071845 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe4e010e010 0x7fe4e010e4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:49.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.966+0000 7fe4df400640 1 -- 172.21.15.90:0/4152071845 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe4e0103f10 msgr2=0x7fe4e0110a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:49.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.966+0000 7fe4df400640 1 --2- 172.21.15.90:0/4152071845 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe4e0103f10 0x7fe4e0110a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:49.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.966+0000 7fe4df400640 1 -- 172.21.15.90:0/4152071845 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4e01ab010 con 0x7fe4e0108fe0 2024-04-02T14:47:49.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.966+0000 7fe4dfe00640 1 --2- 172.21.15.90:0/4152071845 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe4e0103f10 0x7fe4e0110a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:47:49.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.966+0000 7fe4df400640 1 --2- 172.21.15.90:0/4152071845 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe4e0108fe0 0x7fe4e010dad0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7fe4cc00e9c0 tx=0x7fe4cc00ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:47:49.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.967+0000 7fe4dcc00640 1 -- 172.21.15.90:0/4152071845 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4cc00ccc0 con 0x7fe4e0108fe0 2024-04-02T14:47:49.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.967+0000 7fe4dcc00640 1 -- 172.21.15.90:0/4152071845 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe4cc010910 con 0x7fe4e0108fe0 2024-04-02T14:47:49.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.967+0000 7fe4e6a00640 1 -- 172.21.15.90:0/4152071845 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4e01ab300 con 0x7fe4e0108fe0 2024-04-02T14:47:49.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.967+0000 7fe4dcc00640 1 -- 172.21.15.90:0/4152071845 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4cc0188c0 con 0x7fe4e0108fe0 2024-04-02T14:47:49.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.967+0000 7fe4e6a00640 1 -- 172.21.15.90:0/4152071845 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4e01ab850 con 0x7fe4e0108fe0 2024-04-02T14:47:49.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.972+0000 7fe4e97cc740 1 Processor -- start 2024-04-02T14:47:49.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.973+0000 7fe4e97cc740 1 -- start start 2024-04-02T14:47:49.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.973+0000 7fe4dcc00640 1 -- 172.21.15.90:0/4152071845 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fe4cc018a60 con 0x7fe4e0108fe0 2024-04-02T14:47:49.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.973+0000 7fe4e97cc740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558eaaf379f0 0x558eaaf36740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:49.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.973+0000 7fe4e97cc740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558eaaf36d10 0x558eaaf37190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:49.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.973+0000 7fe4e97cc740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558eaaf40080 0x558eaaf42470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:49.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.973+0000 7fe4e97cc740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x558eaae77400 con 0x558eaaf40080 2024-04-02T14:47:49.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.973+0000 7fe4e97cc740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x558eaae5bf10 con 0x558eaaf379f0 2024-04-02T14:47:49.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.973+0000 7fe4e97cc740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x558eaae9b6f0 con 0x558eaaf36d10 2024-04-02T14:47:49.969 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.974+0000 7fe4dfe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558eaaf36d10 0x558eaaf37190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:49.969 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.974+0000 7fe4dfe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558eaaf36d10 0x558eaaf37190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:34606/0 (socket says 172.21.15.90:34606) 2024-04-02T14:47:49.970 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.974+0000 7fe4dfe00640 1 -- 172.21.15.90:0/2635627015 learned_addr learned my addr 172.21.15.90:0/2635627015 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:47:49.970 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.974+0000 7fe4e4a00640 1 --2- 172.21.15.90:0/2635627015 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558eaaf379f0 0x558eaaf36740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:49.970 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.974+0000 7fe4df400640 1 --2- 172.21.15.90:0/2635627015 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558eaaf40080 0x558eaaf42470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:49.970 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.974+0000 7fe4e4a00640 1 -- 172.21.15.90:0/2635627015 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558eaaf36d10 msgr2=0x558eaaf37190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:47:49.970 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.974+0000 7fe4e4a00640 1 --2- 172.21.15.90:0/2635627015 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558eaaf36d10 0x558eaaf37190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:49.970 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.974+0000 7fe4e4a00640 1 -- 172.21.15.90:0/2635627015 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558eaaf40080 msgr2=0x558eaaf42470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:47:49.970 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.974+0000 7fe4e4a00640 1 --2- 172.21.15.90:0/2635627015 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558eaaf40080 0x558eaaf42470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:49.970 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.974+0000 7fe4e4a00640 1 -- 172.21.15.90:0/2635627015 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558eaabaf3f0 con 0x558eaaf379f0 2024-04-02T14:47:49.970 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.974+0000 7fe4df400640 1 --2- 172.21.15.90:0/2635627015 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558eaaf40080 0x558eaaf42470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:47:49.970 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.975+0000 7fe4dcc00640 1 --2- 172.21.15.90:0/4152071845 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fe4c40a3ea0 0x7fe4c40a6360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:49.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.975+0000 7fe4e4a00640 1 --2- 172.21.15.90:0/2635627015 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558eaaf379f0 0x558eaaf36740 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7fe4d8009f30 tx=0x7fe4d80118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eced4b526a26c283 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:47:49.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.975+0000 7fe4dcc00640 1 -- 172.21.15.90:0/4152071845 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 8550+0+0 (secure 0 0 0) 0x7fe4cc0d0af0 con 0x7fe4e0108fe0 2024-04-02T14:47:49.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.975+0000 7fe4d5200640 1 -- 172.21.15.90:0/2635627015 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4d8004b20 con 0x558eaaf379f0 2024-04-02T14:47:49.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.975+0000 7fe4dfe00640 1 --2- 172.21.15.90:0/4152071845 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fe4c40a3ea0 0x7fe4c40a6360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:49.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.975+0000 7fe4d5200640 1 -- 172.21.15.90:0/2635627015 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe4d8004cc0 con 0x558eaaf379f0 2024-04-02T14:47:49.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.975+0000 7fe4d5200640 1 -- 172.21.15.90:0/2635627015 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4d8005310 con 0x558eaaf379f0 2024-04-02T14:47:49.972 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.976+0000 7fe4e97cc740 1 -- 172.21.15.90:0/2635627015 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558eaaf379f0 msgr2=0x558eaaf36740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:49.972 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.976+0000 7fe4dfe00640 1 --2- 172.21.15.90:0/4152071845 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fe4c40a3ea0 0x7fe4c40a6360 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fe4d00096c0 tx=0x7fe4d002df90 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:47:49.972 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.976+0000 7fe4e97cc740 1 --2- 172.21.15.90:0/2635627015 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558eaaf379f0 0x558eaaf36740 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7fe4d8009f30 tx=0x7fe4d80118d0 comp rx=0 tx=0).stop 2024-04-02T14:47:49.972 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.976+0000 7fe4e97cc740 1 -- 172.21.15.90:0/2635627015 shutdown_connections 2024-04-02T14:47:49.972 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.976+0000 7fe4e97cc740 1 --2- 172.21.15.90:0/2635627015 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558eaaf40080 0x558eaaf42470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:49.972 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.976+0000 7fe4e97cc740 1 --2- 172.21.15.90:0/2635627015 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558eaaf36d10 0x558eaaf37190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:49.972 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.976+0000 7fe4e97cc740 1 --2- 172.21.15.90:0/2635627015 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558eaaf379f0 0x558eaaf36740 unknown :-1 s=CLOSED pgs=895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:49.972 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.976+0000 7fe4e97cc740 1 -- 172.21.15.90:0/2635627015 >> 172.21.15.90:0/2635627015 conn(0x558eaaefda50 msgr2=0x558eaaf31760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:47:49.972 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.976+0000 7fe4e97cc740 1 -- 172.21.15.90:0/2635627015 shutdown_connections 2024-04-02T14:47:49.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.977+0000 7fe4e97cc740 1 -- 172.21.15.90:0/2635627015 wait complete. 2024-04-02T14:47:49.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.977+0000 7fe4e97cc740 1 -- start start 2024-04-02T14:47:49.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.977+0000 7fe4e97cc740 1 Processor -- start 2024-04-02T14:47:49.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.978+0000 7fe4e97cc740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558eaaf36d10 0x558eaaf3cf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:49.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.978+0000 7fe4e97cc740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558eaaf379f0 0x558eaaf3d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:49.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.978+0000 7fe4e97cc740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558eaaf40080 0x558eaaf3d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:49.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.978+0000 7fe4e97cc740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x558eaae976a0 con 0x558eaaf379f0 2024-04-02T14:47:49.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.978+0000 7fe4e97cc740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x558eaae43980 con 0x558eaaf40080 2024-04-02T14:47:49.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.978+0000 7fe4e97cc740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x558eaae56cb0 con 0x558eaaf36d10 2024-04-02T14:47:49.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.978+0000 7fe4e97cc740 10 client.0 _ll_register_callbacks cb 0x558eaae566a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:47:49.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.978+0000 7fe4dfe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558eaaf36d10 0x558eaaf3cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:49.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.978+0000 7fe4df400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558eaaf379f0 0x558eaaf3d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:49.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.978+0000 7fe4e4a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558eaaf40080 0x558eaaf3d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:49.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.978+0000 7fe4e4a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558eaaf40080 0x558eaaf3d9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:50116/0 (socket says 172.21.15.90:50116) 2024-04-02T14:47:49.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.978+0000 7fe4e4a00640 1 -- 172.21.15.90:0/3839152752 learned_addr learned my addr 172.21.15.90:0/3839152752 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:47:49.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.978+0000 7fe4dfe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558eaaf36d10 0x558eaaf3cf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:34612/0 (socket says 172.21.15.90:34612) 2024-04-02T14:47:49.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.979+0000 7fe4e4a00640 1 -- 172.21.15.90:0/3839152752 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558eaaf36d10 msgr2=0x558eaaf3cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:47:49.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.979+0000 7fe4e4a00640 1 --2- 172.21.15.90:0/3839152752 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558eaaf36d10 0x558eaaf3cf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:49.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.979+0000 7fe4e4a00640 1 -- 172.21.15.90:0/3839152752 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558eaaf379f0 msgr2=0x558eaaf3d4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:47:49.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.979+0000 7fe4e4a00640 1 --2- 172.21.15.90:0/3839152752 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558eaaf379f0 0x558eaaf3d4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:49.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.979+0000 7fe4e4a00640 1 -- 172.21.15.90:0/3839152752 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558eaac797d0 con 0x558eaaf40080 2024-04-02T14:47:49.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.979+0000 7fe4df400640 1 --2- 172.21.15.90:0/3839152752 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558eaaf379f0 0x558eaaf3d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:47:49.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.979+0000 7fe4e4a00640 1 --2- 172.21.15.90:0/3839152752 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558eaaf40080 0x558eaaf3d9e0 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7fe4d801b040 tx=0x7fe4d800fc40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=81a63af38a39f56b server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:47:49.976 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.979+0000 7fe4c9600640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:47:49.977 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.980+0000 7fe4c9600640 1 -- 172.21.15.90:0/3839152752 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4d8028070 con 0x558eaaf40080 2024-04-02T14:47:49.977 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.980+0000 7fe4e97cc740 1 -- 172.21.15.90:0/3839152752 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe4d8004cc0 con 0x558eaaf40080 2024-04-02T14:47:49.977 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.980+0000 7fe4c9600640 1 -- 172.21.15.90:0/3839152752 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 8550+0+0 (secure 0 0 0) 0x7fe4d8005090 con 0x558eaaf40080 2024-04-02T14:47:49.977 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.980+0000 7fe4c9600640 1 -- 172.21.15.90:0/3839152752 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fe4d8034050 con 0x558eaaf40080 2024-04-02T14:47:49.977 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.980+0000 7fe4e97cc740 10 client.29474 fetch_fsmap learned FSMap version 27 2024-04-02T14:47:49.977 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.980+0000 7fe4e97cc740 1 -- 172.21.15.90:0/3839152752 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x558eaaf4ee00 con 0x558eaaf40080 2024-04-02T14:47:49.978 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.980+0000 7fe4c9600640 1 -- 172.21.15.90:0/3839152752 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe4d8004cc0 con 0x558eaaf40080 2024-04-02T14:47:49.978 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.981+0000 7fe4c9600640 1 -- 172.21.15.90:0/3839152752 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fe4d8024740 con 0x558eaaf40080 2024-04-02T14:47:49.978 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.981+0000 7fe4e97cc740 10 client.29474 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:47:49.978 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.981+0000 7fe4e97cc740 10 client.29474 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-04-02T14:47:49.978 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.981+0000 7fe4e97cc740 20 client.29474 populate_metadata read hostname 'smithi090' 2024-04-02T14:47:49.978 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.981+0000 7fe4e97cc740 1 --2- 172.21.15.90:0/3839152752 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x558eaaf50210 0x558eaaf52600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:49.978 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.981+0000 7fe4e97cc740 4 client.29474 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:47:49.978 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.981+0000 7fe4e97cc740 1 -- 172.21.15.90:0/3839152752 --> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558eaae5b040 con 0x558eaaf50210 2024-04-02T14:47:49.978 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.981+0000 7fe4dfe00640 1 --2- 172.21.15.90:0/3839152752 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x558eaaf50210 0x558eaaf52600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:49.978 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.982+0000 7fe4c9600640 10 client.29474 ms_handle_connect on v2:172.21.15.92:6836/1270635939 2024-04-02T14:47:49.978 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.982+0000 7fe4dfe00640 1 --2- 172.21.15.90:0/3839152752 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x558eaaf50210 0x558eaaf52600 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=51f8ccfb02d10c3c server_cookie=6f1d4b2a1beff25c in_seq=0 out_seq=0 2024-04-02T14:47:49.978 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.983+0000 7fe4c9600640 1 -- 172.21.15.90:0/3839152752 <== mds.3 v2:172.21.15.92:6836/1270635939 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x558eaae5b040 con 0x558eaaf50210 2024-04-02T14:47:49.978 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.983+0000 7fe4c9600640 10 client.29474 handle_command_reply: tid=0 2024-04-02T14:47:49.978 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.983+0000 7fe4e97cc740 1 client.29474 shutdown 2024-04-02T14:47:49.979 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.983+0000 7fe4e97cc740 1 -- 172.21.15.90:0/3839152752 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558eaaf40080 msgr2=0x558eaaf3d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:49.979 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.983+0000 7fe4e97cc740 1 --2- 172.21.15.90:0/3839152752 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558eaaf40080 0x558eaaf3d9e0 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7fe4d801b040 tx=0x7fe4d800fc40 comp rx=0 tx=0).stop 2024-04-02T14:47:49.979 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.983+0000 7fe4e97cc740 1 -- 172.21.15.90:0/3839152752 shutdown_connections 2024-04-02T14:47:49.979 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.983+0000 7fe4e97cc740 1 --2- 172.21.15.90:0/3839152752 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x558eaaf50210 0x558eaaf52600 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:49.979 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.983+0000 7fe4e97cc740 1 --2- 172.21.15.90:0/3839152752 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558eaaf40080 0x558eaaf3d9e0 unknown :-1 s=CLOSED pgs=896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:49.979 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.983+0000 7fe4e97cc740 1 --2- 172.21.15.90:0/3839152752 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558eaaf379f0 0x558eaaf3d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:49.980 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.983+0000 7fe4c9600640 0 client.29474 ms_handle_reset on v2:172.21.15.92:6836/1270635939 2024-04-02T14:47:49.980 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.983+0000 7fe4e97cc740 1 --2- 172.21.15.90:0/3839152752 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558eaaf36d10 0x558eaaf3cf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:49.980 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.983+0000 7fe4e97cc740 1 -- 172.21.15.90:0/3839152752 >> 172.21.15.90:0/3839152752 conn(0x558eaaefda50 msgr2=0x558eaaf334b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:47:49.980 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.983+0000 7fe4e97cc740 1 -- 172.21.15.90:0/3839152752 shutdown_connections 2024-04-02T14:47:49.980 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.983+0000 7fe4e97cc740 1 -- 172.21.15.90:0/3839152752 wait complete. 2024-04-02T14:47:49.980 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:49.984+0000 7fe4e97cc740 20 client.29474 trim_cache size 0 max 16384 2024-04-02T14:47:49.997 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.002+0000 7fe4e97cc740 1 Processor -- start 2024-04-02T14:47:49.997 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.002+0000 7fe4e97cc740 1 -- start start 2024-04-02T14:47:49.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.002+0000 7fe4e97cc740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558eaaf36d10 0x558eaaf3f690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:49.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.002+0000 7fe4e97cc740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558eaaf379f0 0x558eaaf4baf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:49.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.002+0000 7fe4e97cc740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558eaaf40080 0x558eaaf57dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:49.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.003+0000 7fe4e97cc740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fe4e01b3f20 con 0x558eaaf36d10 2024-04-02T14:47:49.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.003+0000 7fe4e97cc740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fe4e01b5a60 con 0x558eaaf40080 2024-04-02T14:47:49.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.003+0000 7fe4e97cc740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fe4e01b45f0 con 0x558eaaf379f0 2024-04-02T14:47:49.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.003+0000 7fe4e4a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558eaaf40080 0x558eaaf57dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:49.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.003+0000 7fe4e4a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558eaaf40080 0x558eaaf57dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:50124/0 (socket says 172.21.15.90:50124) 2024-04-02T14:47:49.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.003+0000 7fe4e4a00640 1 -- 172.21.15.90:0/2236010105 learned_addr learned my addr 172.21.15.90:0/2236010105 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:47:49.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.003+0000 7fe4df400640 1 --2- 172.21.15.90:0/2236010105 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558eaaf379f0 0x558eaaf4baf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:49.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.003+0000 7fe4dfe00640 1 --2- 172.21.15.90:0/2236010105 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558eaaf36d10 0x558eaaf3f690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:49.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.003+0000 7fe4e4a00640 1 -- 172.21.15.90:0/2236010105 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558eaaf379f0 msgr2=0x558eaaf4baf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:47:50.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.003+0000 7fe4e4a00640 1 --2- 172.21.15.90:0/2236010105 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558eaaf379f0 0x558eaaf4baf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:50.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.003+0000 7fe4e4a00640 1 -- 172.21.15.90:0/2236010105 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558eaaf36d10 msgr2=0x558eaaf3f690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:47:50.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.003+0000 7fe4e4a00640 1 --2- 172.21.15.90:0/2236010105 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558eaaf36d10 0x558eaaf3f690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:50.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.003+0000 7fe4e4a00640 1 -- 172.21.15.90:0/2236010105 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558eaaf52b90 con 0x558eaaf40080 2024-04-02T14:47:50.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.003+0000 7fe4dfe00640 1 --2- 172.21.15.90:0/2236010105 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558eaaf36d10 0x558eaaf3f690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:47:50.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.004+0000 7fe4e4a00640 1 --2- 172.21.15.90:0/2236010105 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558eaaf40080 0x558eaaf57dc0 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7fe4d8009aa0 tx=0x7fe4d8009480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5a41b53fb6bf87ae server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:47:50.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.004+0000 7fe4cbe00640 1 -- 172.21.15.90:0/2236010105 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4d8028070 con 0x558eaaf40080 2024-04-02T14:47:50.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.004+0000 7fe4cbe00640 1 -- 172.21.15.90:0/2236010105 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe4d8022510 con 0x558eaaf40080 2024-04-02T14:47:50.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.004+0000 7fe4cbe00640 1 -- 172.21.15.90:0/2236010105 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4d8024920 con 0x558eaaf40080 2024-04-02T14:47:50.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.005+0000 7fe4e97cc740 1 -- 172.21.15.90:0/2236010105 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558eaaf40080 msgr2=0x558eaaf57dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:50.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.005+0000 7fe4e97cc740 1 --2- 172.21.15.90:0/2236010105 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558eaaf40080 0x558eaaf57dc0 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7fe4d8009aa0 tx=0x7fe4d8009480 comp rx=0 tx=0).stop 2024-04-02T14:47:50.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.005+0000 7fe4e97cc740 1 -- 172.21.15.90:0/2236010105 shutdown_connections 2024-04-02T14:47:50.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.005+0000 7fe4e97cc740 1 --2- 172.21.15.90:0/2236010105 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558eaaf40080 0x558eaaf57dc0 unknown :-1 s=CLOSED pgs=897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:50.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.005+0000 7fe4e97cc740 1 --2- 172.21.15.90:0/2236010105 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558eaaf379f0 0x558eaaf4baf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:50.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.005+0000 7fe4e97cc740 1 --2- 172.21.15.90:0/2236010105 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558eaaf36d10 0x558eaaf3f690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:50.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.005+0000 7fe4e97cc740 1 -- 172.21.15.90:0/2236010105 >> 172.21.15.90:0/2236010105 conn(0x558eaaefda50 msgr2=0x558eaaf333d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:47:50.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.005+0000 7fe4e97cc740 1 -- 172.21.15.90:0/2236010105 shutdown_connections 2024-04-02T14:47:50.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.005+0000 7fe4e97cc740 1 -- 172.21.15.90:0/2236010105 wait complete. 2024-04-02T14:47:50.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.006+0000 7fe4e97cc740 1 -- start start 2024-04-02T14:47:50.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.007+0000 7fe4e97cc740 1 Processor -- start 2024-04-02T14:47:50.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.009+0000 7fe4e97cc740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558eaaf36d10 0x558eaaf481f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:50.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.010+0000 7fe4e97cc740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558eaaf379f0 0x558eaaf397f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:50.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.010+0000 7fe4e97cc740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558eaaf40080 0x558eaaf39d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:50.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.010+0000 7fe4e97cc740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fe4e01b5390 con 0x558eaaf379f0 2024-04-02T14:47:50.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.010+0000 7fe4e97cc740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fe4e01b4cc0 con 0x558eaaf40080 2024-04-02T14:47:50.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.010+0000 7fe4e97cc740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fe4e01b1f20 con 0x558eaaf36d10 2024-04-02T14:47:50.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.010+0000 7fe4e97cc740 10 client.0 _ll_register_callbacks cb 0x7fe4d80093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:47:50.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.010+0000 7fe4dfe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558eaaf36d10 0x558eaaf481f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:50.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.010+0000 7fe4dfe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558eaaf36d10 0x558eaaf481f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:34624/0 (socket says 172.21.15.90:34624) 2024-04-02T14:47:50.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.010+0000 7fe4dfe00640 1 -- 172.21.15.90:0/566553120 learned_addr learned my addr 172.21.15.90:0/566553120 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:47:50.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.010+0000 7fe4e4a00640 1 --2- 172.21.15.90:0/566553120 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558eaaf40080 0x558eaaf39d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:50.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.010+0000 7fe4dfe00640 1 -- 172.21.15.90:0/566553120 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558eaaf40080 msgr2=0x558eaaf39d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:47:50.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.010+0000 7fe4dfe00640 1 --2- 172.21.15.90:0/566553120 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558eaaf40080 0x558eaaf39d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:50.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.010+0000 7fe4df400640 1 --2- 172.21.15.90:0/566553120 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558eaaf379f0 0x558eaaf397f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:50.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.010+0000 7fe4dfe00640 1 -- 172.21.15.90:0/566553120 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558eaaf379f0 msgr2=0x558eaaf397f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:47:50.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.010+0000 7fe4dfe00640 1 --2- 172.21.15.90:0/566553120 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558eaaf379f0 0x558eaaf397f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:50.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.010+0000 7fe4dfe00640 1 -- 172.21.15.90:0/566553120 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558eaaf35d40 con 0x558eaaf36d10 2024-04-02T14:47:50.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.011+0000 7fe4df400640 1 --2- 172.21.15.90:0/566553120 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558eaaf379f0 0x558eaaf397f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:47:50.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.011+0000 7fe4dfe00640 1 --2- 172.21.15.90:0/566553120 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558eaaf36d10 0x558eaaf481f0 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7fe4d0000c00 tx=0x7fe4d0042070 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a8071f40370f820b server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:47:50.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.011+0000 7fe4c9600640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:47:50.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.011+0000 7fe4c9600640 1 -- 172.21.15.90:0/566553120 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4d0046070 con 0x558eaaf36d10 2024-04-02T14:47:50.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.012+0000 7fe4c9600640 1 -- 172.21.15.90:0/566553120 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 8550+0+0 (secure 0 0 0) 0x7fe4d00583f0 con 0x558eaaf36d10 2024-04-02T14:47:50.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.012+0000 7fe4e97cc740 1 -- 172.21.15.90:0/566553120 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe4d8022510 con 0x558eaaf36d10 2024-04-02T14:47:50.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.012+0000 7fe4c9600640 1 -- 172.21.15.90:0/566553120 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fe4d002cd30 con 0x558eaaf36d10 2024-04-02T14:47:50.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.012+0000 7fe4e97cc740 10 client.29260 fetch_fsmap learned FSMap version 27 2024-04-02T14:47:50.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.012+0000 7fe4e97cc740 1 -- 172.21.15.90:0/566553120 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x558eaafb6190 con 0x558eaaf36d10 2024-04-02T14:47:50.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.012+0000 7fe4c9600640 1 -- 172.21.15.90:0/566553120 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe4d8022510 con 0x558eaaf36d10 2024-04-02T14:47:50.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.012+0000 7fe4c9600640 1 -- 172.21.15.90:0/566553120 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fe4d0003d40 con 0x558eaaf36d10 2024-04-02T14:47:50.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.013+0000 7fe4e97cc740 10 client.29260 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:47:50.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.013+0000 7fe4e97cc740 10 client.29260 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-04-02T14:47:50.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.013+0000 7fe4e97cc740 20 client.29260 populate_metadata read hostname 'smithi090' 2024-04-02T14:47:50.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.013+0000 7fe4e97cc740 1 --2- 172.21.15.90:0/566553120 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x558eaafb6890 0x558eaafb8cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:50.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.013+0000 7fe4e97cc740 4 client.29260 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-02T14:47:50.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.013+0000 7fe4e97cc740 1 -- 172.21.15.90:0/566553120 --> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] -- command(tid 0: {"prefix": "damage ls"}) -- 0x558eaaea8c30 con 0x558eaafb6890 2024-04-02T14:47:50.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.013+0000 7fe4df400640 1 --2- 172.21.15.90:0/566553120 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x558eaafb6890 0x558eaafb8cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:50.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.014+0000 7fe4df400640 1 --2- 172.21.15.90:0/566553120 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x558eaafb6890 0x558eaafb8cb0 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=dae55dfa2567caef server_cookie=df1b8900951492a2 in_seq=0 out_seq=0 2024-04-02T14:47:50.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.014+0000 7fe4c9600640 10 client.29260 ms_handle_connect on v2:172.21.15.92:6836/1270635939 2024-04-02T14:47:50.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.014+0000 7fe4c9600640 1 -- 172.21.15.90:0/566553120 <== mds.3 v2:172.21.15.92:6836/1270635939 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x558eaaea8c30 con 0x558eaafb6890 2024-04-02T14:47:50.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.014+0000 7fe4c9600640 10 client.29260 handle_command_reply: tid=0 2024-04-02T14:47:50.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.014+0000 7fe4e97cc740 1 client.29260 shutdown 2024-04-02T14:47:50.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.014+0000 7fe4e97cc740 1 -- 172.21.15.90:0/566553120 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558eaaf36d10 msgr2=0x558eaaf481f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:50.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.014+0000 7fe4e97cc740 1 --2- 172.21.15.90:0/566553120 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558eaaf36d10 0x558eaaf481f0 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7fe4d0000c00 tx=0x7fe4d0042070 comp rx=0 tx=0).stop 2024-04-02T14:47:50.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.014+0000 7fe4e97cc740 1 -- 172.21.15.90:0/566553120 shutdown_connections 2024-04-02T14:47:50.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.014+0000 7fe4e97cc740 1 --2- 172.21.15.90:0/566553120 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x558eaafb6890 0x558eaafb8cb0 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:50.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.014+0000 7fe4e97cc740 1 --2- 172.21.15.90:0/566553120 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558eaaf40080 0x558eaaf39d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:50.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.014+0000 7fe4e97cc740 1 --2- 172.21.15.90:0/566553120 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558eaaf379f0 0x558eaaf397f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:50.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.014+0000 7fe4c9600640 0 client.29260 ms_handle_reset on v2:172.21.15.92:6836/1270635939 2024-04-02T14:47:50.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.014+0000 7fe4e97cc740 1 --2- 172.21.15.90:0/566553120 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558eaaf36d10 0x558eaaf481f0 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:50.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.014+0000 7fe4e97cc740 1 -- 172.21.15.90:0/566553120 >> 172.21.15.90:0/566553120 conn(0x558eaaefda50 msgr2=0x558eaaf30010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:47:50.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.014+0000 7fe4e97cc740 1 -- 172.21.15.90:0/566553120 shutdown_connections 2024-04-02T14:47:50.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.014+0000 7fe4e97cc740 1 -- 172.21.15.90:0/566553120 wait complete. 2024-04-02T14:47:50.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.014+0000 7fe4e97cc740 20 client.29260 trim_cache size 0 max 16384 2024-04-02T14:47:50.011 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-02T14:47:50.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.015+0000 7fe4e6a00640 1 -- 172.21.15.90:0/4152071845 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fe4c40a3ea0 msgr2=0x7fe4c40a6360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:50.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.015+0000 7fe4e6a00640 1 --2- 172.21.15.90:0/4152071845 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fe4c40a3ea0 0x7fe4c40a6360 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fe4d00096c0 tx=0x7fe4d002df90 comp rx=0 tx=0).stop 2024-04-02T14:47:50.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.015+0000 7fe4e6a00640 1 -- 172.21.15.90:0/4152071845 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe4e0108fe0 msgr2=0x7fe4e010dad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:50.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.015+0000 7fe4e6a00640 1 --2- 172.21.15.90:0/4152071845 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe4e0108fe0 0x7fe4e010dad0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7fe4cc00e9c0 tx=0x7fe4cc00ee80 comp rx=0 tx=0).stop 2024-04-02T14:47:50.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.015+0000 7fe4e6a00640 1 -- 172.21.15.90:0/4152071845 shutdown_connections 2024-04-02T14:47:50.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.015+0000 7fe4e6a00640 1 --2- 172.21.15.90:0/4152071845 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fe4c40a3ea0 0x7fe4c40a6360 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:50.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.015+0000 7fe4e6a00640 1 --2- 172.21.15.90:0/4152071845 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe4e010e010 0x7fe4e010e4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:50.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.015+0000 7fe4e6a00640 1 --2- 172.21.15.90:0/4152071845 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe4e0108fe0 0x7fe4e010dad0 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:50.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.015+0000 7fe4e6a00640 1 --2- 172.21.15.90:0/4152071845 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe4e0103f10 0x7fe4e0110a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:50.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.015+0000 7fe4e6a00640 1 -- 172.21.15.90:0/4152071845 >> 172.21.15.90:0/4152071845 conn(0x7fe4e00ff7b0 msgr2=0x7fe4e01003a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:47:50.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.015+0000 7fe4e6a00640 1 -- 172.21.15.90:0/4152071845 shutdown_connections 2024-04-02T14:47:50.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.016+0000 7fe4e6a00640 1 -- 172.21.15.90:0/4152071845 wait complete. 2024-04-02T14:47:50.029 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-02T14:47:50.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.293+0000 7f34d2400640 1 -- 172.21.15.90:0/2897262230 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f34cc114c50 msgr2=0x7f34cc117040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:50.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.293+0000 7f34d2400640 1 --2- 172.21.15.90:0/2897262230 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f34cc114c50 0x7f34cc117040 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f34bc009900 tx=0x7f34bc02bc60 comp rx=0 tx=0).stop 2024-04-02T14:47:50.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.296+0000 7f34d2400640 1 -- 172.21.15.90:0/2897262230 shutdown_connections 2024-04-02T14:47:50.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.296+0000 7f34d2400640 1 --2- 172.21.15.90:0/2897262230 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f34cc114c50 0x7f34cc117040 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:50.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.296+0000 7f34d2400640 1 --2- 172.21.15.90:0/2897262230 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f34cc109600 0x7f34cc114710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:50.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.296+0000 7f34d2400640 1 --2- 172.21.15.90:0/2897262230 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f34cc108ce0 0x7f34cc1090c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:50.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.296+0000 7f34d2400640 1 -- 172.21.15.90:0/2897262230 >> 172.21.15.90:0/2897262230 conn(0x7f34cc06cbe0 msgr2=0x7f34cc06cff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:47:50.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.299+0000 7f34d2400640 1 -- 172.21.15.90:0/2897262230 shutdown_connections 2024-04-02T14:47:50.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.299+0000 7f34d2400640 1 -- 172.21.15.90:0/2897262230 wait complete. 2024-04-02T14:47:50.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.300+0000 7f34d2400640 1 Processor -- start 2024-04-02T14:47:50.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.300+0000 7f34d2400640 1 -- start start 2024-04-02T14:47:50.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.301+0000 7f34d2400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f34cc108ce0 0x7f34cc1a2e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:50.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.301+0000 7f34d2400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f34cc109600 0x7f34cc1a3360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:50.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.301+0000 7f34d2400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f34cc114c50 0x7f34cc1a7690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:50.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.301+0000 7f34d2400640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f34cc10b010 con 0x7f34cc114c50 2024-04-02T14:47:50.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.301+0000 7f34d2400640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f34cc10ae90 con 0x7f34cc108ce0 2024-04-02T14:47:50.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.301+0000 7f34d2400640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f34cc10b190 con 0x7f34cc109600 2024-04-02T14:47:50.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.303+0000 7f34caa00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f34cc109600 0x7f34cc1a3360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:50.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.303+0000 7f34caa00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f34cc109600 0x7f34cc1a3360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:34638/0 (socket says 172.21.15.90:34638) 2024-04-02T14:47:50.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.303+0000 7f34caa00640 1 -- 172.21.15.90:0/52509435 learned_addr learned my addr 172.21.15.90:0/52509435 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:47:50.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.303+0000 7f34caa00640 1 -- 172.21.15.90:0/52509435 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f34cc108ce0 msgr2=0x7f34cc1a2e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:50.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.303+0000 7f34caa00640 1 --2- 172.21.15.90:0/52509435 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f34cc108ce0 0x7f34cc1a2e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:50.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.303+0000 7f34caa00640 1 -- 172.21.15.90:0/52509435 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f34cc114c50 msgr2=0x7f34cc1a7690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:50.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.303+0000 7f34caa00640 1 --2- 172.21.15.90:0/52509435 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f34cc114c50 0x7f34cc1a7690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:50.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.303+0000 7f34caa00640 1 -- 172.21.15.90:0/52509435 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34cc1a7dc0 con 0x7f34cc109600 2024-04-02T14:47:50.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.306+0000 7f34caa00640 1 --2- 172.21.15.90:0/52509435 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f34cc109600 0x7f34cc1a3360 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f34b400b4a0 tx=0x7f34b400b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:47:50.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.307+0000 7f34c3e00640 1 -- 172.21.15.90:0/52509435 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34b4014070 con 0x7f34cc109600 2024-04-02T14:47:50.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.308+0000 7f34c3e00640 1 -- 172.21.15.90:0/52509435 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f34b4010af0 con 0x7f34cc109600 2024-04-02T14:47:50.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.308+0000 7f34c3e00640 1 -- 172.21.15.90:0/52509435 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34b4017d10 con 0x7f34cc109600 2024-04-02T14:47:50.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.308+0000 7f34d2400640 1 -- 172.21.15.90:0/52509435 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34cc1a80b0 con 0x7f34cc109600 2024-04-02T14:47:50.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.309+0000 7f34d2400640 1 -- 172.21.15.90:0/52509435 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34cc1af940 con 0x7f34cc109600 2024-04-02T14:47:50.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.316+0000 7f34d516b740 1 Processor -- start 2024-04-02T14:47:50.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.317+0000 7f34d516b740 1 -- start start 2024-04-02T14:47:50.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.317+0000 7f34d516b740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c3f8e949f0 0x55c3f8e93740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:50.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.318+0000 7f34d516b740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c3f8e93d10 0x55c3f8e94190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:50.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.318+0000 7f34d516b740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c3f8e9d080 0x55c3f8e9f470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:50.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.318+0000 7f34d516b740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55c3f8dd4400 con 0x55c3f8e949f0 2024-04-02T14:47:50.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.318+0000 7f34d516b740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55c3f8db8f10 con 0x55c3f8e93d10 2024-04-02T14:47:50.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.318+0000 7f34d516b740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55c3f8df86f0 con 0x55c3f8e9d080 2024-04-02T14:47:50.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.318+0000 7f34cbe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c3f8e949f0 0x55c3f8e93740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:50.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.318+0000 7f34cbe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c3f8e949f0 0x55c3f8e93740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33946/0 (socket says 172.21.15.90:33946) 2024-04-02T14:47:50.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.318+0000 7f34cbe00640 1 -- 172.21.15.90:0/1945348679 learned_addr learned my addr 172.21.15.90:0/1945348679 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:47:50.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.318+0000 7f34caa00640 1 --2- 172.21.15.90:0/1945348679 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c3f8e9d080 0x55c3f8e9f470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:50.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.319+0000 7f34cbe00640 1 -- 172.21.15.90:0/1945348679 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c3f8e9d080 msgr2=0x55c3f8e9f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:47:50.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.319+0000 7f34cbe00640 1 --2- 172.21.15.90:0/1945348679 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c3f8e9d080 0x55c3f8e9f470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:50.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.319+0000 7f34cbe00640 1 -- 172.21.15.90:0/1945348679 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c3f8e93d10 msgr2=0x55c3f8e94190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:47:50.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.319+0000 7f34cb400640 1 --2- 172.21.15.90:0/1945348679 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c3f8e93d10 0x55c3f8e94190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:50.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.319+0000 7f34cbe00640 1 --2- 172.21.15.90:0/1945348679 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c3f8e93d10 0x55c3f8e94190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:50.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.319+0000 7f34cbe00640 1 -- 172.21.15.90:0/1945348679 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c3f8b0c3f0 con 0x55c3f8e949f0 2024-04-02T14:47:50.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.319+0000 7f34cb400640 1 --2- 172.21.15.90:0/1945348679 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c3f8e93d10 0x55c3f8e94190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:47:50.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.320+0000 7f34cbe00640 1 --2- 172.21.15.90:0/1945348679 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c3f8e949f0 0x55c3f8e93740 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f34bc005ab0 tx=0x7f34bc005e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34e01e6df85bcf55 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:47:50.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.320+0000 7f34c0c00640 1 -- 172.21.15.90:0/1945348679 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34bc039040 con 0x55c3f8e949f0 2024-04-02T14:47:50.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.320+0000 7f34c0c00640 1 -- 172.21.15.90:0/1945348679 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f34bc042070 con 0x55c3f8e949f0 2024-04-02T14:47:50.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.320+0000 7f34c0c00640 1 -- 172.21.15.90:0/1945348679 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34bc0035f0 con 0x55c3f8e949f0 2024-04-02T14:47:50.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.321+0000 7f34c3e00640 1 -- 172.21.15.90:0/52509435 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f34b4017070 con 0x7f34cc109600 2024-04-02T14:47:50.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.321+0000 7f34d516b740 1 -- 172.21.15.90:0/1945348679 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c3f8e949f0 msgr2=0x55c3f8e93740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:50.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.321+0000 7f34d516b740 1 --2- 172.21.15.90:0/1945348679 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c3f8e949f0 0x55c3f8e93740 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f34bc005ab0 tx=0x7f34bc005e00 comp rx=0 tx=0).stop 2024-04-02T14:47:50.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.322+0000 7f34d516b740 1 -- 172.21.15.90:0/1945348679 shutdown_connections 2024-04-02T14:47:50.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.322+0000 7f34d516b740 1 --2- 172.21.15.90:0/1945348679 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c3f8e9d080 0x55c3f8e9f470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:50.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.322+0000 7f34d516b740 1 --2- 172.21.15.90:0/1945348679 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c3f8e93d10 0x55c3f8e94190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:50.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.322+0000 7f34d516b740 1 --2- 172.21.15.90:0/1945348679 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c3f8e949f0 0x55c3f8e93740 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:50.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.322+0000 7f34d516b740 1 -- 172.21.15.90:0/1945348679 >> 172.21.15.90:0/1945348679 conn(0x55c3f8e5aa50 msgr2=0x55c3f8e8e760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:47:50.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.323+0000 7f34d516b740 1 -- 172.21.15.90:0/1945348679 shutdown_connections 2024-04-02T14:47:50.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.323+0000 7f34d516b740 1 -- 172.21.15.90:0/1945348679 wait complete. 2024-04-02T14:47:50.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.324+0000 7f34d516b740 1 -- start start 2024-04-02T14:47:50.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.324+0000 7f34d516b740 1 Processor -- start 2024-04-02T14:47:50.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.325+0000 7f34c3e00640 1 --2- 172.21.15.90:0/52509435 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f34b00a3e00 0x7f34b00a62c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:50.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.325+0000 7f34d516b740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c3f8e93d10 0x55c3f8e9a0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:50.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.325+0000 7f34d516b740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c3f8e949f0 0x55c3f8e9a610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:50.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.325+0000 7f34d516b740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c3f8e9d080 0x55c3f8e9ab50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:50.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.325+0000 7f34d516b740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55c3f8df46a0 con 0x55c3f8e949f0 2024-04-02T14:47:50.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.325+0000 7f34d516b740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55c3f8da0980 con 0x55c3f8e9d080 2024-04-02T14:47:50.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.325+0000 7f34d516b740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55c3f8db3cb0 con 0x55c3f8e93d10 2024-04-02T14:47:50.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.325+0000 7f34c3e00640 1 -- 172.21.15.90:0/52509435 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 8550+0+0 (secure 0 0 0) 0x7f34b40cf230 con 0x7f34cc109600 2024-04-02T14:47:50.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.325+0000 7f34d516b740 10 client.0 _ll_register_callbacks cb 0x55c3f8db36a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:47:50.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.325+0000 7f34cbe00640 1 --2- 172.21.15.90:0/52509435 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f34b00a3e00 0x7f34b00a62c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:50.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.326+0000 7f34cbe00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c3f8e9d080 0x55c3f8e9ab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:50.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.326+0000 7f34cbe00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c3f8e9d080 0x55c3f8e9ab50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:50176/0 (socket says 172.21.15.90:50176) 2024-04-02T14:47:50.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.326+0000 7f34cbe00640 1 -- 172.21.15.90:0/3863329020 learned_addr learned my addr 172.21.15.90:0/3863329020 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:47:50.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.326+0000 7f34cb400640 1 --2- 172.21.15.90:0/3863329020 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c3f8e93d10 0x55c3f8e9a0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:50.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.326+0000 7f34caa00640 1 --2- 172.21.15.90:0/3863329020 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c3f8e949f0 0x55c3f8e9a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:50.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.327+0000 7f34cbe00640 1 -- 172.21.15.90:0/3863329020 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c3f8e93d10 msgr2=0x55c3f8e9a0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:47:50.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.327+0000 7f34cbe00640 1 --2- 172.21.15.90:0/3863329020 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c3f8e93d10 0x55c3f8e9a0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:50.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.327+0000 7f34cbe00640 1 -- 172.21.15.90:0/3863329020 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c3f8e949f0 msgr2=0x55c3f8e9a610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:47:50.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.327+0000 7f34cbe00640 1 --2- 172.21.15.90:0/3863329020 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c3f8e949f0 0x55c3f8e9a610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:50.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.327+0000 7f34cbe00640 1 -- 172.21.15.90:0/3863329020 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c3f8bd67d0 con 0x55c3f8e9d080 2024-04-02T14:47:50.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.327+0000 7f34cbe00640 1 --2- 172.21.15.90:0/52509435 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f34b00a3e00 0x7f34b00a62c0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f34bc004c30 tx=0x7f34bc004bc0 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:47:50.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.327+0000 7f34cbe00640 1 --2- 172.21.15.90:0/3863329020 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c3f8e9d080 0x55c3f8e9ab50 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f34bc041000 tx=0x7f34bc005e00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bea04aa6208b985b server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:47:50.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.327+0000 7f34b9600640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:47:50.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.327+0000 7f34b9600640 1 -- 172.21.15.90:0/3863329020 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34bc004870 con 0x55c3f8e9d080 2024-04-02T14:47:50.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.327+0000 7f34b9600640 1 -- 172.21.15.90:0/3863329020 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 8550+0+0 (secure 0 0 0) 0x7f34bc003dc0 con 0x55c3f8e9d080 2024-04-02T14:47:50.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.327+0000 7f34d516b740 1 -- 172.21.15.90:0/3863329020 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f34bc042070 con 0x55c3f8e9d080 2024-04-02T14:47:50.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.328+0000 7f34b9600640 1 -- 172.21.15.90:0/3863329020 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f34bc004c90 con 0x55c3f8e9d080 2024-04-02T14:47:50.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.328+0000 7f34d516b740 10 client.29486 fetch_fsmap learned FSMap version 27 2024-04-02T14:47:50.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.328+0000 7f34d516b740 1 -- 172.21.15.90:0/3863329020 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55c3f8eabea0 con 0x55c3f8e9d080 2024-04-02T14:47:50.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.329+0000 7f34b9600640 1 -- 172.21.15.90:0/3863329020 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f34bc042070 con 0x55c3f8e9d080 2024-04-02T14:47:50.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.329+0000 7f34b9600640 1 -- 172.21.15.90:0/3863329020 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f34bc03c070 con 0x55c3f8e9d080 2024-04-02T14:47:50.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.330+0000 7f34d516b740 10 client.29486 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:47:50.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.330+0000 7f34d516b740 10 client.29486 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-04-02T14:47:50.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.330+0000 7f34d516b740 20 client.29486 populate_metadata read hostname 'smithi090' 2024-04-02T14:47:50.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.330+0000 7f34d516b740 1 --2- 172.21.15.90:0/3863329020 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x55c3f8ead1f0 0x55c3f8eaf5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:50.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.330+0000 7f34d516b740 4 client.29486 mds_command: new command op to 24547 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:47:50.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.330+0000 7f34d516b740 1 -- 172.21.15.90:0/3863329020 --> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c3f8db8040 con 0x55c3f8ead1f0 2024-04-02T14:47:50.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.331+0000 7f34cb400640 1 --2- 172.21.15.90:0/3863329020 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x55c3f8ead1f0 0x55c3f8eaf5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:50.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.332+0000 7f34b9600640 10 client.29486 ms_handle_connect on v2:172.21.15.177:6838/664895663 2024-04-02T14:47:50.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.332+0000 7f34cb400640 1 --2- 172.21.15.90:0/3863329020 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x55c3f8ead1f0 0x55c3f8eaf5e0 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=5266f56381563462 server_cookie=e9661c8c3823a603 in_seq=0 out_seq=0 2024-04-02T14:47:50.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.333+0000 7f34b9600640 1 -- 172.21.15.90:0/3863329020 <== mds.2 v2:172.21.15.177:6838/664895663 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c3f8db8040 con 0x55c3f8ead1f0 2024-04-02T14:47:50.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.333+0000 7f34b9600640 10 client.29486 handle_command_reply: tid=0 2024-04-02T14:47:50.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.333+0000 7f34d516b740 1 client.29486 shutdown 2024-04-02T14:47:50.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.334+0000 7f34d516b740 1 -- 172.21.15.90:0/3863329020 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c3f8e9d080 msgr2=0x55c3f8e9ab50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:50.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.334+0000 7f34d516b740 1 --2- 172.21.15.90:0/3863329020 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c3f8e9d080 0x55c3f8e9ab50 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f34bc041000 tx=0x7f34bc005e00 comp rx=0 tx=0).stop 2024-04-02T14:47:50.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.334+0000 7f34d516b740 1 -- 172.21.15.90:0/3863329020 shutdown_connections 2024-04-02T14:47:50.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.334+0000 7f34d516b740 1 --2- 172.21.15.90:0/3863329020 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x55c3f8ead1f0 0x55c3f8eaf5e0 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:50.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.334+0000 7f34d516b740 1 --2- 172.21.15.90:0/3863329020 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c3f8e9d080 0x55c3f8e9ab50 unknown :-1 s=CLOSED pgs=898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:50.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.334+0000 7f34b9600640 0 client.29486 ms_handle_reset on v2:172.21.15.177:6838/664895663 2024-04-02T14:47:50.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.334+0000 7f34d516b740 1 --2- 172.21.15.90:0/3863329020 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c3f8e949f0 0x55c3f8e9a610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:50.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.334+0000 7f34d516b740 1 --2- 172.21.15.90:0/3863329020 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c3f8e93d10 0x55c3f8e9a0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:50.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.334+0000 7f34d516b740 1 -- 172.21.15.90:0/3863329020 >> 172.21.15.90:0/3863329020 conn(0x55c3f8e5aa50 msgr2=0x55c3f8e904b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:47:50.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.334+0000 7f34d516b740 1 -- 172.21.15.90:0/3863329020 shutdown_connections 2024-04-02T14:47:50.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.334+0000 7f34d516b740 1 -- 172.21.15.90:0/3863329020 wait complete. 2024-04-02T14:47:50.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.335+0000 7f34d516b740 20 client.29486 trim_cache size 0 max 16384 2024-04-02T14:47:50.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.356+0000 7f34d516b740 1 Processor -- start 2024-04-02T14:47:50.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.356+0000 7f34d516b740 1 -- start start 2024-04-02T14:47:50.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.356+0000 7f34d516b740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c3f8e93d10 0x55c3f8ea9650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:50.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.356+0000 7f34d516b740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c3f8e949f0 0x55c3f8ea8a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:50.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.356+0000 7f34d516b740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c3f8e9d080 0x55c3f8e903d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:50.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.356+0000 7f34d516b740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f34cc1b8270 con 0x55c3f8e9d080 2024-04-02T14:47:50.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.356+0000 7f34d516b740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f34cc1b9db0 con 0x55c3f8e93d10 2024-04-02T14:47:50.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.356+0000 7f34d516b740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f34cc1b8940 con 0x55c3f8e949f0 2024-04-02T14:47:50.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.357+0000 7f34cb400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c3f8e93d10 0x55c3f8ea9650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:50.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.357+0000 7f34cb400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c3f8e93d10 0x55c3f8ea9650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:50186/0 (socket says 172.21.15.90:50186) 2024-04-02T14:47:50.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.357+0000 7f34cb400640 1 -- 172.21.15.90:0/3893037422 learned_addr learned my addr 172.21.15.90:0/3893037422 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:47:50.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.357+0000 7f34cbe00640 1 --2- 172.21.15.90:0/3893037422 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c3f8e9d080 0x55c3f8e903d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:50.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.357+0000 7f34caa00640 1 --2- 172.21.15.90:0/3893037422 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c3f8e949f0 0x55c3f8ea8a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:50.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.357+0000 7f34caa00640 1 -- 172.21.15.90:0/3893037422 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c3f8e93d10 msgr2=0x55c3f8ea9650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:47:50.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.357+0000 7f34caa00640 1 --2- 172.21.15.90:0/3893037422 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c3f8e93d10 0x55c3f8ea9650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:50.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.357+0000 7f34caa00640 1 -- 172.21.15.90:0/3893037422 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c3f8e9d080 msgr2=0x55c3f8e903d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:47:50.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.357+0000 7f34caa00640 1 --2- 172.21.15.90:0/3893037422 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c3f8e9d080 0x55c3f8e903d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:50.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.357+0000 7f34caa00640 1 -- 172.21.15.90:0/3893037422 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c3f8e8e970 con 0x55c3f8e949f0 2024-04-02T14:47:50.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.358+0000 7f34caa00640 1 --2- 172.21.15.90:0/3893037422 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c3f8e949f0 0x55c3f8ea8a90 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f34b40cca10 tx=0x7f34b4017730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ed2ba43363a796cd server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:47:50.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.358+0000 7f34bbe00640 1 -- 172.21.15.90:0/3893037422 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34b409b960 con 0x55c3f8e949f0 2024-04-02T14:47:50.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.358+0000 7f34bbe00640 1 -- 172.21.15.90:0/3893037422 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f34b4044060 con 0x55c3f8e949f0 2024-04-02T14:47:50.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.358+0000 7f34bbe00640 1 -- 172.21.15.90:0/3893037422 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34b40978d0 con 0x55c3f8e949f0 2024-04-02T14:47:50.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.359+0000 7f34d516b740 1 -- 172.21.15.90:0/3893037422 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c3f8e949f0 msgr2=0x55c3f8ea8a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:50.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.359+0000 7f34d516b740 1 --2- 172.21.15.90:0/3893037422 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c3f8e949f0 0x55c3f8ea8a90 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f34b40cca10 tx=0x7f34b4017730 comp rx=0 tx=0).stop 2024-04-02T14:47:50.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.359+0000 7f34d516b740 1 -- 172.21.15.90:0/3893037422 shutdown_connections 2024-04-02T14:47:50.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.359+0000 7f34d516b740 1 --2- 172.21.15.90:0/3893037422 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c3f8e9d080 0x55c3f8e903d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:50.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.359+0000 7f34d516b740 1 --2- 172.21.15.90:0/3893037422 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c3f8e949f0 0x55c3f8ea8a90 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:50.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.359+0000 7f34d516b740 1 --2- 172.21.15.90:0/3893037422 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c3f8e93d10 0x55c3f8ea9650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:50.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.359+0000 7f34d516b740 1 -- 172.21.15.90:0/3893037422 >> 172.21.15.90:0/3893037422 conn(0x55c3f8e5aa50 msgr2=0x55c3f8e9a090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:47:50.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.359+0000 7f34d516b740 1 -- 172.21.15.90:0/3893037422 shutdown_connections 2024-04-02T14:47:50.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.359+0000 7f34d516b740 1 -- 172.21.15.90:0/3893037422 wait complete. 2024-04-02T14:47:50.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.359+0000 7f34d516b740 1 -- start start 2024-04-02T14:47:50.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.360+0000 7f34d516b740 1 Processor -- start 2024-04-02T14:47:50.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.360+0000 7f34d516b740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c3f8e93d10 0x55c3f8e9d460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:50.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.360+0000 7f34d516b740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c3f8e949f0 0x55c3f8e9d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:50.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.360+0000 7f34d516b740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c3f8e9d080 0x55c3f8e9dee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:50.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.360+0000 7f34d516b740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f34cc1b96e0 con 0x55c3f8e93d10 2024-04-02T14:47:50.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.360+0000 7f34d516b740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f34cc1b9010 con 0x55c3f8e9d080 2024-04-02T14:47:50.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.360+0000 7f34d516b740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f34cc1b6270 con 0x55c3f8e949f0 2024-04-02T14:47:50.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.360+0000 7f34d516b740 10 client.0 _ll_register_callbacks cb 0x7f34bc02ce90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:47:50.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.361+0000 7f34cbe00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c3f8e9d080 0x55c3f8e9dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:50.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.361+0000 7f34caa00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c3f8e949f0 0x55c3f8e9d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:50.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.361+0000 7f34caa00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c3f8e949f0 0x55c3f8e9d9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:34662/0 (socket says 172.21.15.90:34662) 2024-04-02T14:47:50.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.361+0000 7f34caa00640 1 -- 172.21.15.90:0/2649032117 learned_addr learned my addr 172.21.15.90:0/2649032117 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:47:50.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.361+0000 7f34cb400640 1 --2- 172.21.15.90:0/2649032117 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c3f8e93d10 0x55c3f8e9d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:50.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.361+0000 7f34caa00640 1 -- 172.21.15.90:0/2649032117 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c3f8e9d080 msgr2=0x55c3f8e9dee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:47:50.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.361+0000 7f34caa00640 1 --2- 172.21.15.90:0/2649032117 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c3f8e9d080 0x55c3f8e9dee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:50.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.361+0000 7f34caa00640 1 -- 172.21.15.90:0/2649032117 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c3f8e93d10 msgr2=0x55c3f8e9d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:47:50.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.361+0000 7f34caa00640 1 --2- 172.21.15.90:0/2649032117 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c3f8e93d10 0x55c3f8e9d460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:50.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.361+0000 7f34caa00640 1 -- 172.21.15.90:0/2649032117 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c3f8ea4240 con 0x55c3f8e949f0 2024-04-02T14:47:50.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.361+0000 7f34cb400640 1 --2- 172.21.15.90:0/2649032117 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c3f8e93d10 0x55c3f8e9d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:47:50.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.362+0000 7f34caa00640 1 --2- 172.21.15.90:0/2649032117 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c3f8e949f0 0x55c3f8e9d9a0 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f34b400b470 tx=0x7f34b4017590 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c811a676dd4c1f76 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:47:50.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.362+0000 7f34b9600640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:47:50.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.362+0000 7f34b9600640 1 -- 172.21.15.90:0/2649032117 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34b40b3020 con 0x55c3f8e949f0 2024-04-02T14:47:50.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.362+0000 7f34b9600640 1 -- 172.21.15.90:0/2649032117 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 8550+0+0 (secure 0 0 0) 0x7f34b409f170 con 0x55c3f8e949f0 2024-04-02T14:47:50.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.362+0000 7f34d516b740 1 -- 172.21.15.90:0/2649032117 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f34b4044060 con 0x55c3f8e949f0 2024-04-02T14:47:50.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.362+0000 7f34b9600640 1 -- 172.21.15.90:0/2649032117 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f34b40a6050 con 0x55c3f8e949f0 2024-04-02T14:47:50.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.362+0000 7f34d516b740 10 client.29278 fetch_fsmap learned FSMap version 27 2024-04-02T14:47:50.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.362+0000 7f34d516b740 1 -- 172.21.15.90:0/2649032117 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55c3f8ea5500 con 0x55c3f8e949f0 2024-04-02T14:47:50.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.362+0000 7f34b9600640 1 -- 172.21.15.90:0/2649032117 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f34b40ccd80 con 0x55c3f8e949f0 2024-04-02T14:47:50.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.363+0000 7f34b9600640 1 -- 172.21.15.90:0/2649032117 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f34b409a6b0 con 0x55c3f8e949f0 2024-04-02T14:47:50.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.363+0000 7f34d516b740 10 client.29278 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:47:50.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.363+0000 7f34d516b740 10 client.29278 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-04-02T14:47:50.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.363+0000 7f34d516b740 20 client.29278 populate_metadata read hostname 'smithi090' 2024-04-02T14:47:50.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.363+0000 7f34d516b740 1 --2- 172.21.15.90:0/2649032117 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x55c3f8ead1f0 0x55c3f8f1ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:50.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.363+0000 7f34d516b740 4 client.29278 mds_command: new command op to 24547 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-02T14:47:50.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.363+0000 7f34d516b740 1 -- 172.21.15.90:0/2649032117 --> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7f34b4044060 con 0x55c3f8ead1f0 2024-04-02T14:47:50.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.363+0000 7f34cb400640 1 --2- 172.21.15.90:0/2649032117 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x55c3f8ead1f0 0x55c3f8f1ddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:50.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.364+0000 7f34b9600640 10 client.29278 ms_handle_connect on v2:172.21.15.177:6838/664895663 2024-04-02T14:47:50.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.364+0000 7f34cb400640 1 --2- 172.21.15.90:0/2649032117 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x55c3f8ead1f0 0x55c3f8f1ddd0 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=39d415a43e622dab server_cookie=de78eb3e2f587042 in_seq=0 out_seq=0 2024-04-02T14:47:50.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.364+0000 7f34b9600640 1 -- 172.21.15.90:0/2649032117 <== mds.2 v2:172.21.15.177:6838/664895663 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7f34b4044060 con 0x55c3f8ead1f0 2024-04-02T14:47:50.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.364+0000 7f34b9600640 10 client.29278 handle_command_reply: tid=0 2024-04-02T14:47:50.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.364+0000 7f34d516b740 1 client.29278 shutdown 2024-04-02T14:47:50.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.365+0000 7f34d516b740 1 -- 172.21.15.90:0/2649032117 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c3f8e949f0 msgr2=0x55c3f8e9d9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:50.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.365+0000 7f34d516b740 1 --2- 172.21.15.90:0/2649032117 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c3f8e949f0 0x55c3f8e9d9a0 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f34b400b470 tx=0x7f34b4017590 comp rx=0 tx=0).stop 2024-04-02T14:47:50.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.365+0000 7f34d516b740 1 -- 172.21.15.90:0/2649032117 shutdown_connections 2024-04-02T14:47:50.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.365+0000 7f34d516b740 1 --2- 172.21.15.90:0/2649032117 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x55c3f8ead1f0 0x55c3f8f1ddd0 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:50.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.365+0000 7f34d516b740 1 --2- 172.21.15.90:0/2649032117 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c3f8e9d080 0x55c3f8e9dee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:50.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.365+0000 7f34d516b740 1 --2- 172.21.15.90:0/2649032117 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c3f8e949f0 0x55c3f8e9d9a0 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:50.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.365+0000 7f34d516b740 1 --2- 172.21.15.90:0/2649032117 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c3f8e93d10 0x55c3f8e9d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:50.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.365+0000 7f34d516b740 1 -- 172.21.15.90:0/2649032117 >> 172.21.15.90:0/2649032117 conn(0x55c3f8e5aa50 msgr2=0x55c3f8e8d8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:47:50.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.365+0000 7f34b9600640 0 client.29278 ms_handle_reset on v2:172.21.15.177:6838/664895663 2024-04-02T14:47:50.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.365+0000 7f34d516b740 1 -- 172.21.15.90:0/2649032117 shutdown_connections 2024-04-02T14:47:50.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.366+0000 7f34d516b740 1 -- 172.21.15.90:0/2649032117 wait complete. 2024-04-02T14:47:50.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.366+0000 7f34d516b740 20 client.29278 trim_cache size 0 max 16384 2024-04-02T14:47:50.363 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-02T14:47:50.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.367+0000 7f34d2400640 1 -- 172.21.15.90:0/52509435 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f34b00a3e00 msgr2=0x7f34b00a62c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:50.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.367+0000 7f34d2400640 1 --2- 172.21.15.90:0/52509435 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f34b00a3e00 0x7f34b00a62c0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f34bc004c30 tx=0x7f34bc004bc0 comp rx=0 tx=0).stop 2024-04-02T14:47:50.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.367+0000 7f34d2400640 1 -- 172.21.15.90:0/52509435 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f34cc109600 msgr2=0x7f34cc1a3360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:50.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.367+0000 7f34d2400640 1 --2- 172.21.15.90:0/52509435 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f34cc109600 0x7f34cc1a3360 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f34b400b4a0 tx=0x7f34b400b960 comp rx=0 tx=0).stop 2024-04-02T14:47:50.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.367+0000 7f34d2400640 1 -- 172.21.15.90:0/52509435 shutdown_connections 2024-04-02T14:47:50.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.367+0000 7f34d2400640 1 --2- 172.21.15.90:0/52509435 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f34b00a3e00 0x7f34b00a62c0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:50.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.367+0000 7f34d2400640 1 --2- 172.21.15.90:0/52509435 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f34cc114c50 0x7f34cc1a7690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:50.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.367+0000 7f34d2400640 1 --2- 172.21.15.90:0/52509435 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f34cc109600 0x7f34cc1a3360 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:50.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.367+0000 7f34d2400640 1 --2- 172.21.15.90:0/52509435 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f34cc108ce0 0x7f34cc1a2e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:50.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.367+0000 7f34d2400640 1 -- 172.21.15.90:0/52509435 >> 172.21.15.90:0/52509435 conn(0x7f34cc06cbe0 msgr2=0x7f34cc116c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:47:50.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.368+0000 7f34d2400640 1 -- 172.21.15.90:0/52509435 shutdown_connections 2024-04-02T14:47:50.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.368+0000 7f34d2400640 1 -- 172.21.15.90:0/52509435 wait complete. 2024-04-02T14:47:50.389 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:47:50 smithi090 ceph-mon[31674]: pgmap v403: 97 pgs: 97 active+clean; 4.5 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 31 MiB/s wr, 1.56k op/s 2024-04-02T14:47:50.389 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:47:50 smithi090 ceph-mon[31674]: from='client.? 172.21.15.90:0/2162411399' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-02T14:47:50.393 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-02T14:47:50.603 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:47:50 smithi092 ceph-mon[36497]: pgmap v403: 97 pgs: 97 active+clean; 4.5 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 31 MiB/s wr, 1.56k op/s 2024-04-02T14:47:50.604 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:47:50 smithi092 ceph-mon[36497]: from='client.? 172.21.15.90:0/2162411399' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-02T14:47:50.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.672+0000 7fa1bf000640 1 -- 172.21.15.90:0/320790073 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa1b810af70 msgr2=0x7fa1b810d380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:50.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.672+0000 7fa1bf000640 1 --2- 172.21.15.90:0/320790073 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa1b810af70 0x7fa1b810d380 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7fa1b0009ca0 tx=0x7fa1b002c100 comp rx=0 tx=0).stop 2024-04-02T14:47:50.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.672+0000 7fa1bf000640 1 -- 172.21.15.90:0/320790073 shutdown_connections 2024-04-02T14:47:50.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.672+0000 7fa1bf000640 1 --2- 172.21.15.90:0/320790073 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa1b810af70 0x7fa1b810d380 unknown :-1 s=CLOSED pgs=899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:50.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.672+0000 7fa1bf000640 1 --2- 172.21.15.90:0/320790073 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa1b8102420 0x7fa1b810a900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:50.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.672+0000 7fa1bf000640 1 --2- 172.21.15.90:0/320790073 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa1b8101b00 0x7fa1b8101ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:50.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.672+0000 7fa1bf000640 1 -- 172.21.15.90:0/320790073 >> 172.21.15.90:0/320790073 conn(0x7fa1b80756c0 msgr2=0x7fa1b8075ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:47:50.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.672+0000 7fa1bf000640 1 -- 172.21.15.90:0/320790073 shutdown_connections 2024-04-02T14:47:50.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.672+0000 7fa1bf000640 1 -- 172.21.15.90:0/320790073 wait complete. 2024-04-02T14:47:50.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.673+0000 7fa1bf000640 1 Processor -- start 2024-04-02T14:47:50.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.673+0000 7fa1bf000640 1 -- start start 2024-04-02T14:47:50.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.673+0000 7fa1bf000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa1b8101b00 0x7fa1b819a510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:50.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.673+0000 7fa1bf000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa1b8102420 0x7fa1b819aa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:50.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.673+0000 7fa1bf000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa1b810af70 0x7fa1b819ed80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:50.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.673+0000 7fa1bf000640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fa1b810fbe0 con 0x7fa1b810af70 2024-04-02T14:47:50.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.673+0000 7fa1bf000640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fa1b810fa60 con 0x7fa1b8102420 2024-04-02T14:47:50.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.673+0000 7fa1bf000640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fa1b810fd60 con 0x7fa1b8101b00 2024-04-02T14:47:50.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.674+0000 7fa1bd200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa1b8102420 0x7fa1b819aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:50.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.674+0000 7fa1bdc00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa1b8101b00 0x7fa1b819a510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:50.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.674+0000 7fa1bd200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa1b8102420 0x7fa1b819aa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:50216/0 (socket says 172.21.15.90:50216) 2024-04-02T14:47:50.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.674+0000 7fa1bd200640 1 -- 172.21.15.90:0/331731874 learned_addr learned my addr 172.21.15.90:0/331731874 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:47:50.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.674+0000 7fa1be600640 1 --2- 172.21.15.90:0/331731874 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa1b810af70 0x7fa1b819ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:50.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.674+0000 7fa1bd200640 1 -- 172.21.15.90:0/331731874 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa1b8101b00 msgr2=0x7fa1b819a510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:50.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.674+0000 7fa1bd200640 1 --2- 172.21.15.90:0/331731874 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa1b8101b00 0x7fa1b819a510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:50.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.674+0000 7fa1bd200640 1 -- 172.21.15.90:0/331731874 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa1b810af70 msgr2=0x7fa1b819ed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:50.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.674+0000 7fa1bd200640 1 --2- 172.21.15.90:0/331731874 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa1b810af70 0x7fa1b819ed80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:50.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.674+0000 7fa1bd200640 1 -- 172.21.15.90:0/331731874 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1b819f4b0 con 0x7fa1b8102420 2024-04-02T14:47:50.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.675+0000 7fa1bd200640 1 --2- 172.21.15.90:0/331731874 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa1b8102420 0x7fa1b819aa50 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7fa1a4005d00 tx=0x7fa1a400dde0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:47:50.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.675+0000 7fa1b6000640 1 -- 172.21.15.90:0/331731874 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1a4017070 con 0x7fa1b8102420 2024-04-02T14:47:50.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.675+0000 7fa1b6000640 1 -- 172.21.15.90:0/331731874 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa1a4009c30 con 0x7fa1b8102420 2024-04-02T14:47:50.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.675+0000 7fa1b6000640 1 -- 172.21.15.90:0/331731874 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1a400f620 con 0x7fa1b8102420 2024-04-02T14:47:50.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.675+0000 7fa1bf000640 1 -- 172.21.15.90:0/331731874 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa1b819f7a0 con 0x7fa1b8102420 2024-04-02T14:47:50.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.675+0000 7fa1bf000640 1 -- 172.21.15.90:0/331731874 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1b81a7030 con 0x7fa1b8102420 2024-04-02T14:47:50.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.681+0000 7fa1b6000640 1 -- 172.21.15.90:0/331731874 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fa1a401a070 con 0x7fa1b8102420 2024-04-02T14:47:50.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.683+0000 7fa1c33e7740 1 Processor -- start 2024-04-02T14:47:50.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.683+0000 7fa1c33e7740 1 -- start start 2024-04-02T14:47:50.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.683+0000 7fa1b6000640 1 --2- 172.21.15.90:0/331731874 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fa1a00a3d90 0x7fa1a00a6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:50.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.684+0000 7fa1c33e7740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f41c497ba0 0x55f41c4968f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:50.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.684+0000 7fa1b6000640 1 -- 172.21.15.90:0/331731874 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 8550+0+0 (secure 0 0 0) 0x7fa1a40d0130 con 0x7fa1b8102420 2024-04-02T14:47:50.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:47:50 smithi177 ceph-mon[36431]: pgmap v403: 97 pgs: 97 active+clean; 4.5 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 31 MiB/s wr, 1.56k op/s 2024-04-02T14:47:50.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:47:50 smithi177 ceph-mon[36431]: from='client.? 172.21.15.90:0/2162411399' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-02T14:47:50.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.684+0000 7fa1c33e7740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f41c496ec0 0x55f41c497340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:50.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.684+0000 7fa1c33e7740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f41c4a0230 0x55f41c4a2620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:50.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.684+0000 7fa1c33e7740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55f41c3d7400 con 0x55f41c4a0230 2024-04-02T14:47:50.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.684+0000 7fa1c33e7740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55f41c3bbf10 con 0x55f41c496ec0 2024-04-02T14:47:50.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.684+0000 7fa1c33e7740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55f41c3fb6f0 con 0x55f41c497ba0 2024-04-02T14:47:50.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.684+0000 7fa1bdc00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f41c497ba0 0x55f41c4968f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:50.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.684+0000 7fa1bdc00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f41c497ba0 0x55f41c4968f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:34694/0 (socket says 172.21.15.90:34694) 2024-04-02T14:47:50.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.684+0000 7fa1bdc00640 1 -- 172.21.15.90:0/2094676083 learned_addr learned my addr 172.21.15.90:0/2094676083 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:47:50.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.684+0000 7fa1be600640 1 --2- 172.21.15.90:0/2094676083 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f41c4a0230 0x55f41c4a2620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:50.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.684+0000 7fa1bd200640 1 --2- 172.21.15.90:0/2094676083 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f41c496ec0 0x55f41c497340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:50.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.684+0000 7fa1be600640 1 --2- 172.21.15.90:0/331731874 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fa1a00a3d90 0x7fa1a00a6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:50.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.684+0000 7fa1bdc00640 1 -- 172.21.15.90:0/2094676083 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f41c496ec0 msgr2=0x55f41c497340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:47:50.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.684+0000 7fa1bdc00640 1 --2- 172.21.15.90:0/2094676083 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f41c496ec0 0x55f41c497340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:50.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.684+0000 7fa1bdc00640 1 -- 172.21.15.90:0/2094676083 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f41c4a0230 msgr2=0x55f41c4a2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:47:50.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.684+0000 7fa1bdc00640 1 --2- 172.21.15.90:0/2094676083 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f41c4a0230 0x55f41c4a2620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:50.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.685+0000 7fa1bdc00640 1 -- 172.21.15.90:0/2094676083 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f41c10f3f0 con 0x55f41c497ba0 2024-04-02T14:47:50.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.685+0000 7fa1bd200640 1 --2- 172.21.15.90:0/2094676083 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f41c496ec0 0x55f41c497340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:47:50.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.685+0000 7fa1bdc00640 1 --2- 172.21.15.90:0/2094676083 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f41c497ba0 0x55f41c4968f0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7fa1ac00ac20 tx=0x7fa1ac00f790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7b853b01e1b6db86 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:47:50.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.685+0000 7fa1aaa00640 1 -- 172.21.15.90:0/2094676083 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1ac01d070 con 0x55f41c497ba0 2024-04-02T14:47:50.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.685+0000 7fa1aaa00640 1 -- 172.21.15.90:0/2094676083 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa1ac018b20 con 0x55f41c497ba0 2024-04-02T14:47:50.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.685+0000 7fa1aaa00640 1 -- 172.21.15.90:0/2094676083 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1ac010c00 con 0x55f41c497ba0 2024-04-02T14:47:50.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.687+0000 7fa1be600640 1 --2- 172.21.15.90:0/331731874 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fa1a00a3d90 0x7fa1a00a6250 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fa1b002c640 tx=0x7fa1b002c100 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:47:50.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.687+0000 7fa1c33e7740 1 -- 172.21.15.90:0/2094676083 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f41c497ba0 msgr2=0x55f41c4968f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:50.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.687+0000 7fa1c33e7740 1 --2- 172.21.15.90:0/2094676083 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f41c497ba0 0x55f41c4968f0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7fa1ac00ac20 tx=0x7fa1ac00f790 comp rx=0 tx=0).stop 2024-04-02T14:47:50.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.687+0000 7fa1c33e7740 1 -- 172.21.15.90:0/2094676083 shutdown_connections 2024-04-02T14:47:50.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.687+0000 7fa1c33e7740 1 --2- 172.21.15.90:0/2094676083 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f41c4a0230 0x55f41c4a2620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:50.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.687+0000 7fa1c33e7740 1 --2- 172.21.15.90:0/2094676083 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f41c496ec0 0x55f41c497340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:50.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.687+0000 7fa1c33e7740 1 --2- 172.21.15.90:0/2094676083 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f41c497ba0 0x55f41c4968f0 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:50.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.687+0000 7fa1c33e7740 1 -- 172.21.15.90:0/2094676083 >> 172.21.15.90:0/2094676083 conn(0x55f41c45da50 msgr2=0x55f41c491910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:47:50.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.687+0000 7fa1c33e7740 1 -- 172.21.15.90:0/2094676083 shutdown_connections 2024-04-02T14:47:50.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.687+0000 7fa1c33e7740 1 -- 172.21.15.90:0/2094676083 wait complete. 2024-04-02T14:47:50.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.688+0000 7fa1c33e7740 1 -- start start 2024-04-02T14:47:50.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.688+0000 7fa1c33e7740 1 Processor -- start 2024-04-02T14:47:50.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.688+0000 7fa1c33e7740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f41c496ec0 0x55f41c49d210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:50.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.688+0000 7fa1c33e7740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f41c497ba0 0x55f41c49d750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:50.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.688+0000 7fa1c33e7740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f41c4a0230 0x55f41c49dc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:50.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.688+0000 7fa1c33e7740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55f41c3f76a0 con 0x55f41c497ba0 2024-04-02T14:47:50.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.688+0000 7fa1c33e7740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55f41c3a3980 con 0x55f41c4a0230 2024-04-02T14:47:50.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.688+0000 7fa1c33e7740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55f41c3b6cb0 con 0x55f41c496ec0 2024-04-02T14:47:50.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.689+0000 7fa1c33e7740 10 client.0 _ll_register_callbacks cb 0x55f41c3b66a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:47:50.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.689+0000 7fa1be600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f41c4a0230 0x55f41c49dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:50.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.689+0000 7fa1be600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f41c4a0230 0x55f41c49dc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:50240/0 (socket says 172.21.15.90:50240) 2024-04-02T14:47:50.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.689+0000 7fa1be600640 1 -- 172.21.15.90:0/609634153 learned_addr learned my addr 172.21.15.90:0/609634153 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:47:50.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.689+0000 7fa1bdc00640 1 --2- 172.21.15.90:0/609634153 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f41c496ec0 0x55f41c49d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:50.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.689+0000 7fa1bd200640 1 --2- 172.21.15.90:0/609634153 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f41c497ba0 0x55f41c49d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:50.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.689+0000 7fa1bdc00640 1 -- 172.21.15.90:0/609634153 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f41c4a0230 msgr2=0x55f41c49dc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:47:50.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.689+0000 7fa1bdc00640 1 --2- 172.21.15.90:0/609634153 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f41c4a0230 0x55f41c49dc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:50.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.689+0000 7fa1bdc00640 1 -- 172.21.15.90:0/609634153 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f41c497ba0 msgr2=0x55f41c49d750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:47:50.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.689+0000 7fa1bdc00640 1 --2- 172.21.15.90:0/609634153 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f41c497ba0 0x55f41c49d750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:50.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.689+0000 7fa1bdc00640 1 -- 172.21.15.90:0/609634153 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f41c1d97d0 con 0x55f41c496ec0 2024-04-02T14:47:50.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.689+0000 7fa1be600640 1 --2- 172.21.15.90:0/609634153 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f41c4a0230 0x55f41c49dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:47:50.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.690+0000 7fa1bdc00640 1 --2- 172.21.15.90:0/609634153 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f41c496ec0 0x55f41c49d210 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7fa1ac00ac20 tx=0x7fa1ac00a950 comp rx=0 tx=0).ready entity=mon.1 client_cookie=df24c097cf30e951 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:47:50.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.690+0000 7fa19f400640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:47:50.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.690+0000 7fa19f400640 1 -- 172.21.15.90:0/609634153 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1ac027030 con 0x55f41c496ec0 2024-04-02T14:47:50.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.690+0000 7fa19f400640 1 -- 172.21.15.90:0/609634153 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 8550+0+0 (secure 0 0 0) 0x7fa1ac004020 con 0x55f41c496ec0 2024-04-02T14:47:50.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.690+0000 7fa1c33e7740 1 -- 172.21.15.90:0/609634153 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa1ac018b20 con 0x55f41c496ec0 2024-04-02T14:47:50.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.690+0000 7fa19f400640 1 -- 172.21.15.90:0/609634153 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fa1ac031050 con 0x55f41c496ec0 2024-04-02T14:47:50.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.690+0000 7fa1c33e7740 10 client.29290 fetch_fsmap learned FSMap version 27 2024-04-02T14:47:50.690 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.690+0000 7fa1c33e7740 1 -- 172.21.15.90:0/609634153 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55f41c4b78e0 con 0x55f41c496ec0 2024-04-02T14:47:50.690 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.690+0000 7fa19f400640 1 -- 172.21.15.90:0/609634153 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa1ac018b20 con 0x55f41c496ec0 2024-04-02T14:47:50.690 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.691+0000 7fa19f400640 1 -- 172.21.15.90:0/609634153 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fa1ac005040 con 0x55f41c496ec0 2024-04-02T14:47:50.690 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.691+0000 7fa1c33e7740 10 client.29290 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:47:50.690 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.691+0000 7fa1c33e7740 10 client.29290 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2024-04-02T14:47:50.690 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.691+0000 7fa1c33e7740 20 client.29290 populate_metadata read hostname 'smithi090' 2024-04-02T14:47:50.690 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.691+0000 7fa1c33e7740 1 --2- 172.21.15.90:0/609634153 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x55f41c4b8ce0 0x55f41c4bb0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:50.690 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.691+0000 7fa1c33e7740 4 client.29290 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:47:50.690 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.691+0000 7fa1c33e7740 1 -- 172.21.15.90:0/609634153 --> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f41c3bb040 con 0x55f41c4b8ce0 2024-04-02T14:47:50.690 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.691+0000 7fa1bd200640 1 --2- 172.21.15.90:0/609634153 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x55f41c4b8ce0 0x55f41c4bb0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:50.691 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.692+0000 7fa19f400640 10 client.29290 ms_handle_connect on v2:172.21.15.90:6838/1498915870 2024-04-02T14:47:50.691 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.692+0000 7fa1bd200640 1 --2- 172.21.15.90:0/609634153 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x55f41c4b8ce0 0x55f41c4bb0d0 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=8e927a74ab0360f6 server_cookie=aaa492668a7f815b in_seq=0 out_seq=0 2024-04-02T14:47:50.691 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.693+0000 7fa19f400640 1 -- 172.21.15.90:0/609634153 <== mds.1 v2:172.21.15.90:6838/1498915870 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f41c3bb040 con 0x55f41c4b8ce0 2024-04-02T14:47:50.691 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.693+0000 7fa19f400640 10 client.29290 handle_command_reply: tid=0 2024-04-02T14:47:50.691 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.693+0000 7fa1c33e7740 1 client.29290 shutdown 2024-04-02T14:47:50.691 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.694+0000 7fa1c33e7740 1 -- 172.21.15.90:0/609634153 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f41c496ec0 msgr2=0x55f41c49d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:50.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.694+0000 7fa1c33e7740 1 --2- 172.21.15.90:0/609634153 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f41c496ec0 0x55f41c49d210 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7fa1ac00ac20 tx=0x7fa1ac00a950 comp rx=0 tx=0).stop 2024-04-02T14:47:50.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.694+0000 7fa1c33e7740 1 -- 172.21.15.90:0/609634153 shutdown_connections 2024-04-02T14:47:50.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.694+0000 7fa1c33e7740 1 --2- 172.21.15.90:0/609634153 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x55f41c4b8ce0 0x55f41c4bb0d0 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:50.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.694+0000 7fa1c33e7740 1 --2- 172.21.15.90:0/609634153 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f41c4a0230 0x55f41c49dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:50.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.694+0000 7fa1c33e7740 1 --2- 172.21.15.90:0/609634153 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f41c497ba0 0x55f41c49d750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:50.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.694+0000 7fa1c33e7740 1 --2- 172.21.15.90:0/609634153 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f41c496ec0 0x55f41c49d210 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:50.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.694+0000 7fa1c33e7740 1 -- 172.21.15.90:0/609634153 >> 172.21.15.90:0/609634153 conn(0x55f41c45da50 msgr2=0x55f41c492dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:47:50.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.694+0000 7fa19f400640 0 client.29290 ms_handle_reset on v2:172.21.15.90:6838/1498915870 2024-04-02T14:47:50.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.695+0000 7fa1c33e7740 1 -- 172.21.15.90:0/609634153 shutdown_connections 2024-04-02T14:47:50.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.695+0000 7fa1c33e7740 1 -- 172.21.15.90:0/609634153 wait complete. 2024-04-02T14:47:50.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.696+0000 7fa1c33e7740 20 client.29290 trim_cache size 0 max 16384 2024-04-02T14:47:50.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.731+0000 7fa1c33e7740 1 Processor -- start 2024-04-02T14:47:50.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.732+0000 7fa1c33e7740 1 -- start start 2024-04-02T14:47:50.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.732+0000 7fa1c33e7740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f41c496ec0 0x55f41c49f940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:50.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.732+0000 7fa1c33e7740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f41c497ba0 0x55f41c4abca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:50.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.732+0000 7fa1c33e7740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f41c4a0230 0x55f41c4c0700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:50.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.732+0000 7fa1c33e7740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fa1b81ab570 con 0x55f41c4a0230 2024-04-02T14:47:50.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.732+0000 7fa1c33e7740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fa1b81ad0b0 con 0x55f41c496ec0 2024-04-02T14:47:50.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.732+0000 7fa1c33e7740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fa1b81abc40 con 0x55f41c497ba0 2024-04-02T14:47:50.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.732+0000 7fa1bd200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f41c497ba0 0x55f41c4abca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:50.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.733+0000 7fa1bd200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f41c497ba0 0x55f41c4abca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:34720/0 (socket says 172.21.15.90:34720) 2024-04-02T14:47:50.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.733+0000 7fa1bd200640 1 -- 172.21.15.90:0/568184127 learned_addr learned my addr 172.21.15.90:0/568184127 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:47:50.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.733+0000 7fa1bdc00640 1 --2- 172.21.15.90:0/568184127 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f41c496ec0 0x55f41c49f940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:50.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.734+0000 7fa1bdc00640 1 -- 172.21.15.90:0/568184127 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f41c497ba0 msgr2=0x55f41c4abca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:47:50.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.734+0000 7fa1bdc00640 1 --2- 172.21.15.90:0/568184127 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f41c497ba0 0x55f41c4abca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:50.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.734+0000 7fa1bdc00640 1 -- 172.21.15.90:0/568184127 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f41c4a0230 msgr2=0x55f41c4c0700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:47:50.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.734+0000 7fa1bdc00640 1 --2- 172.21.15.90:0/568184127 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f41c4a0230 0x55f41c4c0700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:50.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.734+0000 7fa1bdc00640 1 -- 172.21.15.90:0/568184127 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f41c495ef0 con 0x55f41c496ec0 2024-04-02T14:47:50.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.735+0000 7fa1bdc00640 1 --2- 172.21.15.90:0/568184127 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f41c496ec0 0x55f41c49f940 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7fa1ac00a310 tx=0x7fa1ac010550 comp rx=0 tx=0).ready entity=mon.2 client_cookie=814629e6c5bb6046 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:47:50.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.735+0000 7fa1aa000640 1 -- 172.21.15.90:0/568184127 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1ac027070 con 0x55f41c496ec0 2024-04-02T14:47:50.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.735+0000 7fa1aa000640 1 -- 172.21.15.90:0/568184127 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa1ac005630 con 0x55f41c496ec0 2024-04-02T14:47:50.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.735+0000 7fa1aa000640 1 -- 172.21.15.90:0/568184127 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1ac01d040 con 0x55f41c496ec0 2024-04-02T14:47:50.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.736+0000 7fa1c33e7740 1 -- 172.21.15.90:0/568184127 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f41c496ec0 msgr2=0x55f41c49f940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:50.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.736+0000 7fa1c33e7740 1 --2- 172.21.15.90:0/568184127 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f41c496ec0 0x55f41c49f940 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7fa1ac00a310 tx=0x7fa1ac010550 comp rx=0 tx=0).stop 2024-04-02T14:47:50.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.736+0000 7fa1c33e7740 1 -- 172.21.15.90:0/568184127 shutdown_connections 2024-04-02T14:47:50.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.736+0000 7fa1c33e7740 1 --2- 172.21.15.90:0/568184127 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f41c4a0230 0x55f41c4c0700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:50.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.736+0000 7fa1c33e7740 1 --2- 172.21.15.90:0/568184127 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f41c497ba0 0x55f41c4abca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:50.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.736+0000 7fa1c33e7740 1 --2- 172.21.15.90:0/568184127 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f41c496ec0 0x55f41c49f940 unknown :-1 s=CLOSED pgs=901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:50.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.736+0000 7fa1c33e7740 1 -- 172.21.15.90:0/568184127 >> 172.21.15.90:0/568184127 conn(0x55f41c45da50 msgr2=0x55f41c491440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:47:50.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.739+0000 7fa1c33e7740 1 -- 172.21.15.90:0/568184127 shutdown_connections 2024-04-02T14:47:50.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.741+0000 7fa1c33e7740 1 -- 172.21.15.90:0/568184127 wait complete. 2024-04-02T14:47:50.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.744+0000 7fa1c33e7740 1 -- start start 2024-04-02T14:47:50.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.744+0000 7fa1c33e7740 1 Processor -- start 2024-04-02T14:47:50.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.745+0000 7fa1c33e7740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f41c496ec0 0x55f41c4a0c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:50.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.745+0000 7fa1c33e7740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f41c497ba0 0x55f41c4a1150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:50.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.745+0000 7fa1c33e7740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f41c4a0230 0x55f41c4a71b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:50.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.746+0000 7fa1bdc00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f41c496ec0 0x55f41c4a0c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:50.741 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.746+0000 7fa1bdc00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f41c496ec0 0x55f41c4a0c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:34726/0 (socket says 172.21.15.90:34726) 2024-04-02T14:47:50.741 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.746+0000 7fa1bdc00640 1 -- 172.21.15.90:0/3896194081 learned_addr learned my addr 172.21.15.90:0/3896194081 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:47:50.741 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.746+0000 7fa1c33e7740 1 -- 172.21.15.90:0/3896194081 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fa1b81ac9e0 con 0x55f41c497ba0 2024-04-02T14:47:50.741 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.746+0000 7fa1be600640 1 --2- 172.21.15.90:0/3896194081 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f41c4a0230 0x55f41c4a71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:50.741 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.746+0000 7fa1c33e7740 1 -- 172.21.15.90:0/3896194081 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fa1b81ac310 con 0x55f41c4a0230 2024-04-02T14:47:50.741 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.747+0000 7fa1c33e7740 1 -- 172.21.15.90:0/3896194081 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fa1b81a9590 con 0x55f41c496ec0 2024-04-02T14:47:50.741 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.747+0000 7fa1c33e7740 10 client.0 _ll_register_callbacks cb 0x7fa1ac00aaa0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:47:50.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.747+0000 7fa1bd200640 1 --2- 172.21.15.90:0/3896194081 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f41c497ba0 0x55f41c4a1150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:50.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.748+0000 7fa1bd200640 1 -- 172.21.15.90:0/3896194081 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f41c496ec0 msgr2=0x55f41c4a0c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:47:50.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.748+0000 7fa1bd200640 1 --2- 172.21.15.90:0/3896194081 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f41c496ec0 0x55f41c4a0c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:50.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.748+0000 7fa1bd200640 1 -- 172.21.15.90:0/3896194081 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f41c4a0230 msgr2=0x55f41c4a71b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:47:50.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.748+0000 7fa1bd200640 1 --2- 172.21.15.90:0/3896194081 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f41c4a0230 0x55f41c4a71b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:50.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.748+0000 7fa1bd200640 1 -- 172.21.15.90:0/3896194081 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f41c4a77a0 con 0x55f41c497ba0 2024-04-02T14:47:50.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.748+0000 7fa1bd200640 1 --2- 172.21.15.90:0/3896194081 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f41c497ba0 0x55f41c4a1150 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7fa1a4009610 tx=0x7fa1a4010c70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4ad35401c548f24 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:47:50.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.748+0000 7fa19f400640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-02T14:47:50.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.748+0000 7fa19f400640 1 -- 172.21.15.90:0/3896194081 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1a4021040 con 0x55f41c497ba0 2024-04-02T14:47:50.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.748+0000 7fa19f400640 1 -- 172.21.15.90:0/3896194081 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 8550+0+0 (secure 0 0 0) 0x7fa1a40987c0 con 0x55f41c497ba0 2024-04-02T14:47:50.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.749+0000 7fa1c33e7740 1 -- 172.21.15.90:0/3896194081 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55f41c408c30 con 0x55f41c497ba0 2024-04-02T14:47:50.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.749+0000 7fa19f400640 1 -- 172.21.15.90:0/3896194081 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fa1a40a4dc0 con 0x55f41c497ba0 2024-04-02T14:47:50.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.749+0000 7fa1c33e7740 10 client.18378 fetch_fsmap learned FSMap version 27 2024-04-02T14:47:50.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.749+0000 7fa1c33e7740 1 -- 172.21.15.90:0/3896194081 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55f41c4a8a40 con 0x55f41c497ba0 2024-04-02T14:47:50.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.749+0000 7fa19f400640 1 -- 172.21.15.90:0/3896194081 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55f41c408c30 con 0x55f41c497ba0 2024-04-02T14:47:50.745 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.750+0000 7fa19f400640 1 -- 172.21.15.90:0/3896194081 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fa1ac019ce0 con 0x55f41c497ba0 2024-04-02T14:47:50.745 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.750+0000 7fa1c33e7740 10 client.18378 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:47:50.746 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.750+0000 7fa1c33e7740 10 client.18378 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2024-04-02T14:47:50.746 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.750+0000 7fa1c33e7740 20 client.18378 populate_metadata read hostname 'smithi090' 2024-04-02T14:47:50.746 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.750+0000 7fa1c33e7740 1 --2- 172.21.15.90:0/3896194081 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x55f41c51f4d0 0x55f41c5218f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:50.746 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.750+0000 7fa1c33e7740 4 client.18378 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-02T14:47:50.746 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.750+0000 7fa1c33e7740 1 -- 172.21.15.90:0/3896194081 --> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55f41c402f80 con 0x55f41c51f4d0 2024-04-02T14:47:50.746 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.751+0000 7fa1bdc00640 1 --2- 172.21.15.90:0/3896194081 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x55f41c51f4d0 0x55f41c5218f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:50.746 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.751+0000 7fa1bdc00640 1 --2- 172.21.15.90:0/3896194081 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x55f41c51f4d0 0x55f41c5218f0 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=2e4c48396b02ffe9 server_cookie=52f175cc89150e4a in_seq=0 out_seq=0 2024-04-02T14:47:50.746 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.751+0000 7fa19f400640 10 client.18378 ms_handle_connect on v2:172.21.15.90:6838/1498915870 2024-04-02T14:47:50.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.752+0000 7fa19f400640 1 -- 172.21.15.90:0/3896194081 <== mds.1 v2:172.21.15.90:6838/1498915870 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55f41c402f80 con 0x55f41c51f4d0 2024-04-02T14:47:50.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.752+0000 7fa19f400640 10 client.18378 handle_command_reply: tid=0 2024-04-02T14:47:50.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.752+0000 7fa1c33e7740 1 client.18378 shutdown 2024-04-02T14:47:50.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.752+0000 7fa1c33e7740 1 -- 172.21.15.90:0/3896194081 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f41c497ba0 msgr2=0x55f41c4a1150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:50.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.752+0000 7fa1c33e7740 1 --2- 172.21.15.90:0/3896194081 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f41c497ba0 0x55f41c4a1150 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7fa1a4009610 tx=0x7fa1a4010c70 comp rx=0 tx=0).stop 2024-04-02T14:47:50.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.752+0000 7fa1c33e7740 1 -- 172.21.15.90:0/3896194081 shutdown_connections 2024-04-02T14:47:50.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.752+0000 7fa1c33e7740 1 --2- 172.21.15.90:0/3896194081 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x55f41c51f4d0 0x55f41c5218f0 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:50.750 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-02T14:47:50.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.752+0000 7fa1c33e7740 1 --2- 172.21.15.90:0/3896194081 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f41c4a0230 0x55f41c4a71b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:50.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.752+0000 7fa1c33e7740 1 --2- 172.21.15.90:0/3896194081 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f41c497ba0 0x55f41c4a1150 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:50.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.752+0000 7fa1c33e7740 1 --2- 172.21.15.90:0/3896194081 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f41c496ec0 0x55f41c4a0c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:50.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.752+0000 7fa1c33e7740 1 -- 172.21.15.90:0/3896194081 >> 172.21.15.90:0/3896194081 conn(0x55f41c45da50 msgr2=0x55f41c492be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:47:50.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.753+0000 7fa19f400640 0 client.18378 ms_handle_reset on v2:172.21.15.90:6838/1498915870 2024-04-02T14:47:50.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.753+0000 7fa1c33e7740 1 -- 172.21.15.90:0/3896194081 shutdown_connections 2024-04-02T14:47:50.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.753+0000 7fa1c33e7740 1 -- 172.21.15.90:0/3896194081 wait complete. 2024-04-02T14:47:50.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.754+0000 7fa1c33e7740 20 client.18378 trim_cache size 0 max 16384 2024-04-02T14:47:50.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.755+0000 7fa1bf000640 1 -- 172.21.15.90:0/331731874 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fa1a00a3d90 msgr2=0x7fa1a00a6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:50.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.755+0000 7fa1bf000640 1 --2- 172.21.15.90:0/331731874 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fa1a00a3d90 0x7fa1a00a6250 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fa1b002c640 tx=0x7fa1b002c100 comp rx=0 tx=0).stop 2024-04-02T14:47:50.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.755+0000 7fa1bf000640 1 -- 172.21.15.90:0/331731874 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa1b8102420 msgr2=0x7fa1b819aa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:50.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.755+0000 7fa1bf000640 1 --2- 172.21.15.90:0/331731874 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa1b8102420 0x7fa1b819aa50 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7fa1a4005d00 tx=0x7fa1a400dde0 comp rx=0 tx=0).stop 2024-04-02T14:47:50.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.756+0000 7fa1bf000640 1 -- 172.21.15.90:0/331731874 shutdown_connections 2024-04-02T14:47:50.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.756+0000 7fa1bf000640 1 --2- 172.21.15.90:0/331731874 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fa1a00a3d90 0x7fa1a00a6250 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:50.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.756+0000 7fa1bf000640 1 --2- 172.21.15.90:0/331731874 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa1b810af70 0x7fa1b819ed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:50.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.756+0000 7fa1bf000640 1 --2- 172.21.15.90:0/331731874 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa1b8102420 0x7fa1b819aa50 unknown :-1 s=CLOSED pgs=900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:50.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.756+0000 7fa1bf000640 1 --2- 172.21.15.90:0/331731874 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa1b8101b00 0x7fa1b819a510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:50.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.756+0000 7fa1bf000640 1 -- 172.21.15.90:0/331731874 >> 172.21.15.90:0/331731874 conn(0x7fa1b80756c0 msgr2=0x7fa1b80fcf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:47:50.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.756+0000 7fa1bf000640 1 -- 172.21.15.90:0/331731874 shutdown_connections 2024-04-02T14:47:50.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:50.756+0000 7fa1bf000640 1 -- 172.21.15.90:0/331731874 wait complete. 2024-04-02T14:47:50.774 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-02T14:47:51.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.045+0000 7f78cd000640 1 -- 172.21.15.90:0/3278238124 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f78c8100900 msgr2=0x7f78c8104940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:51.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.045+0000 7f78cd000640 1 --2- 172.21.15.90:0/3278238124 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f78c8100900 0x7f78c8104940 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f78b40074e0 tx=0x7f78b402c8b0 comp rx=0 tx=0).stop 2024-04-02T14:47:51.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.045+0000 7f78cd000640 1 -- 172.21.15.90:0/3278238124 shutdown_connections 2024-04-02T14:47:51.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.045+0000 7f78cd000640 1 --2- 172.21.15.90:0/3278238124 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f78c8105020 0x7f78c810cbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:51.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.045+0000 7f78cd000640 1 --2- 172.21.15.90:0/3278238124 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f78c8100900 0x7f78c8104940 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:51.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.045+0000 7f78cd000640 1 --2- 172.21.15.90:0/3278238124 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f78c80fff50 0x7f78c8100330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:51.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.045+0000 7f78cd000640 1 -- 172.21.15.90:0/3278238124 >> 172.21.15.90:0/3278238124 conn(0x7f78c8076b50 msgr2=0x7f78c80fdee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:47:51.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.046+0000 7f78cd000640 1 -- 172.21.15.90:0/3278238124 shutdown_connections 2024-04-02T14:47:51.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.046+0000 7f78cd000640 1 -- 172.21.15.90:0/3278238124 wait complete. 2024-04-02T14:47:51.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.047+0000 7f78cd000640 1 Processor -- start 2024-04-02T14:47:51.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.049+0000 7f78cd000640 1 -- start start 2024-04-02T14:47:51.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.049+0000 7f78cd000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f78c80fff50 0x7f78c819a580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:51.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.049+0000 7f78cd000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f78c8105020 0x7f78c819aac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:51.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.049+0000 7f78cd000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f78c819ee40 0x7f78c819f2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:51.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.049+0000 7f78cd000640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f78c8114080 con 0x7f78c8105020 2024-04-02T14:47:51.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.049+0000 7f78cd000640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f78c8113f00 con 0x7f78c80fff50 2024-04-02T14:47:51.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.049+0000 7f78cd000640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f78c8114200 con 0x7f78c819ee40 2024-04-02T14:47:51.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.049+0000 7f78c5e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f78c80fff50 0x7f78c819a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:51.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.049+0000 7f78c6800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f78c819ee40 0x7f78c819f2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:51.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.049+0000 7f78c5e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f78c80fff50 0x7f78c819a580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:50268/0 (socket says 172.21.15.90:50268) 2024-04-02T14:47:51.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.049+0000 7f78c5e00640 1 -- 172.21.15.90:0/1921224773 learned_addr learned my addr 172.21.15.90:0/1921224773 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:47:51.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.050+0000 7f78c5400640 1 --2- 172.21.15.90:0/1921224773 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f78c8105020 0x7f78c819aac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:51.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.050+0000 7f78c5e00640 1 -- 172.21.15.90:0/1921224773 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f78c819ee40 msgr2=0x7f78c819f2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:51.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.050+0000 7f78c5e00640 1 --2- 172.21.15.90:0/1921224773 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f78c819ee40 0x7f78c819f2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:51.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.050+0000 7f78c5e00640 1 -- 172.21.15.90:0/1921224773 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f78c8105020 msgr2=0x7f78c819aac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:51.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.050+0000 7f78c5e00640 1 --2- 172.21.15.90:0/1921224773 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f78c8105020 0x7f78c819aac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:51.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.050+0000 7f78c5e00640 1 -- 172.21.15.90:0/1921224773 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78c819f930 con 0x7f78c80fff50 2024-04-02T14:47:51.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.050+0000 7f78c5e00640 1 --2- 172.21.15.90:0/1921224773 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f78c80fff50 0x7f78c819a580 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7f78b800ea60 tx=0x7f78b800ef20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:47:51.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.050+0000 7f78bea00640 1 -- 172.21.15.90:0/1921224773 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78b800cde0 con 0x7f78c80fff50 2024-04-02T14:47:51.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.051+0000 7f78bea00640 1 -- 172.21.15.90:0/1921224773 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f78b80035f0 con 0x7f78c80fff50 2024-04-02T14:47:51.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.051+0000 7f78cd000640 1 -- 172.21.15.90:0/1921224773 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78c819fc20 con 0x7f78c80fff50 2024-04-02T14:47:51.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.051+0000 7f78bea00640 1 -- 172.21.15.90:0/1921224773 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78b8010670 con 0x7f78c80fff50 2024-04-02T14:47:51.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.051+0000 7f78cd000640 1 -- 172.21.15.90:0/1921224773 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78c806b840 con 0x7f78c80fff50 2024-04-02T14:47:51.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.055+0000 7f78bea00640 1 -- 172.21.15.90:0/1921224773 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f78b8004020 con 0x7f78c80fff50 2024-04-02T14:47:51.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.056+0000 7f78cfdc5740 1 Processor -- start 2024-04-02T14:47:51.052 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.057+0000 7f78cfdc5740 1 -- start start 2024-04-02T14:47:51.052 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.057+0000 7f78cfdc5740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b33a8d7200 0x55b33a8d5f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:51.052 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.057+0000 7f78bea00640 1 --2- 172.21.15.90:0/1921224773 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f78ac0a3d50 0x7f78ac0a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:51.052 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.057+0000 7f78bea00640 1 -- 172.21.15.90:0/1921224773 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 8550+0+0 (secure 0 0 0) 0x7f78b80d0c60 con 0x7f78c80fff50 2024-04-02T14:47:51.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.057+0000 7f78cfdc5740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b33a8d6520 0x55b33a8d69a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:51.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.057+0000 7f78cfdc5740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b33a8df890 0x55b33a8e1c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:51.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.057+0000 7f78c5e00640 1 --2- 172.21.15.90:0/1921224773 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f78ac0a3d50 0x7f78ac0a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:51.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.058+0000 7f78cfdc5740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55b33a80e400 con 0x55b33a8df890 2024-04-02T14:47:51.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.058+0000 7f78cfdc5740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55b33a7f2f10 con 0x55b33a8d6520 2024-04-02T14:47:51.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.058+0000 7f78cfdc5740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55b33a8326f0 con 0x55b33a8d7200 2024-04-02T14:47:51.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.058+0000 7f78c5400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b33a8d6520 0x55b33a8d69a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:51.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.058+0000 7f78c5400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b33a8d6520 0x55b33a8d69a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:50284/0 (socket says 172.21.15.90:50284) 2024-04-02T14:47:51.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.058+0000 7f78c5400640 1 -- 172.21.15.90:0/3505931000 learned_addr learned my addr 172.21.15.90:0/3505931000 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:47:51.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.058+0000 7f78c5e00640 1 --2- 172.21.15.90:0/1921224773 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f78ac0a3d50 0x7f78ac0a6210 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f78b80ce390 tx=0x7f78b8003e30 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:47:51.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.058+0000 7f78c6800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b33a8d7200 0x55b33a8d5f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:51.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.059+0000 7f78c5400640 1 -- 172.21.15.90:0/3505931000 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b33a8d7200 msgr2=0x55b33a8d5f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:47:51.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.059+0000 7f78c5400640 1 --2- 172.21.15.90:0/3505931000 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b33a8d7200 0x55b33a8d5f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:51.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.059+0000 7f78c5400640 1 -- 172.21.15.90:0/3505931000 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b33a8df890 msgr2=0x55b33a8e1c80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-02T14:47:51.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.060+0000 7f78c6800640 1 --2- 172.21.15.90:0/3505931000 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b33a8df890 0x55b33a8e1c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:51.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.060+0000 7f78c5400640 1 --2- 172.21.15.90:0/3505931000 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b33a8df890 0x55b33a8e1c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:51.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.060+0000 7f78c5400640 1 -- 172.21.15.90:0/3505931000 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b33a5463f0 con 0x55b33a8d6520 2024-04-02T14:47:51.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.060+0000 7f78c6800640 1 --2- 172.21.15.90:0/3505931000 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b33a8df890 0x55b33a8e1c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:47:51.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.061+0000 7f78c5400640 1 --2- 172.21.15.90:0/3505931000 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b33a8d6520 0x55b33a8d69a0 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f78b4035000 tx=0x7f78b400a140 comp rx=0 tx=0).ready entity=mon.2 client_cookie=abcadec5f8a674a4 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:47:51.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.061+0000 7f78b3400640 1 -- 172.21.15.90:0/3505931000 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78b400a2e0 con 0x55b33a8d6520 2024-04-02T14:47:51.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.061+0000 7f78b3400640 1 -- 172.21.15.90:0/3505931000 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f78b400a5d0 con 0x55b33a8d6520 2024-04-02T14:47:51.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.061+0000 7f78b3400640 1 -- 172.21.15.90:0/3505931000 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78b4004920 con 0x55b33a8d6520 2024-04-02T14:47:51.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.062+0000 7f78cfdc5740 1 -- 172.21.15.90:0/3505931000 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b33a8d6520 msgr2=0x55b33a8d69a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:51.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.062+0000 7f78cfdc5740 1 --2- 172.21.15.90:0/3505931000 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b33a8d6520 0x55b33a8d69a0 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f78b4035000 tx=0x7f78b400a140 comp rx=0 tx=0).stop 2024-04-02T14:47:51.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.063+0000 7f78cfdc5740 1 -- 172.21.15.90:0/3505931000 shutdown_connections 2024-04-02T14:47:51.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.063+0000 7f78cfdc5740 1 --2- 172.21.15.90:0/3505931000 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b33a8df890 0x55b33a8e1c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:51.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.063+0000 7f78cfdc5740 1 --2- 172.21.15.90:0/3505931000 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b33a8d6520 0x55b33a8d69a0 unknown :-1 s=CLOSED pgs=903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:51.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.063+0000 7f78cfdc5740 1 --2- 172.21.15.90:0/3505931000 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b33a8d7200 0x55b33a8d5f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:51.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.063+0000 7f78cfdc5740 1 -- 172.21.15.90:0/3505931000 >> 172.21.15.90:0/3505931000 conn(0x55b33a894a50 msgr2=0x55b33a8d0f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:47:51.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.064+0000 7f78cfdc5740 1 -- 172.21.15.90:0/3505931000 shutdown_connections 2024-04-02T14:47:51.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.064+0000 7f78cfdc5740 1 -- 172.21.15.90:0/3505931000 wait complete. 2024-04-02T14:47:51.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.065+0000 7f78cfdc5740 1 -- start start 2024-04-02T14:47:51.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.065+0000 7f78cfdc5740 1 Processor -- start 2024-04-02T14:47:51.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.066+0000 7f78cfdc5740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b33a8d6520 0x55b33a8dc890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:51.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.066+0000 7f78cfdc5740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b33a8d7200 0x55b33a8dcdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:51.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.066+0000 7f78c6800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b33a8d6520 0x55b33a8dc890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:51.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.066+0000 7f78c6800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b33a8d6520 0x55b33a8dc890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:50298/0 (socket says 172.21.15.90:50298) 2024-04-02T14:47:51.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.066+0000 7f78c6800640 1 -- 172.21.15.90:0/3396758800 learned_addr learned my addr 172.21.15.90:0/3396758800 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:47:51.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.066+0000 7f78cfdc5740 1 --2- 172.21.15.90:0/3396758800 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b33a8df890 0x55b33a8dd310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:51.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.066+0000 7f78cfdc5740 1 -- 172.21.15.90:0/3396758800 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55b33a82e6a0 con 0x55b33a8d7200 2024-04-02T14:47:51.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.066+0000 7f78cfdc5740 1 -- 172.21.15.90:0/3396758800 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55b33a7da980 con 0x55b33a8d6520 2024-04-02T14:47:51.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.066+0000 7f78cfdc5740 1 -- 172.21.15.90:0/3396758800 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55b33a7edcb0 con 0x55b33a8df890 2024-04-02T14:47:51.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.067+0000 7f78c6800640 1 --2- 172.21.15.90:0/3396758800 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b33a8df890 0x55b33a8dd310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:51.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.067+0000 7f78c6800640 1 -- 172.21.15.90:0/3396758800 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b33a8df890 msgr2=0x55b33a8dd310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:47:51.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.067+0000 7f78c6800640 1 --2- 172.21.15.90:0/3396758800 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b33a8df890 0x55b33a8dd310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:51.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.067+0000 7f78c6800640 1 -- 172.21.15.90:0/3396758800 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b33a8d7200 msgr2=0x55b33a8dcdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:47:51.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.067+0000 7f78c6800640 1 --2- 172.21.15.90:0/3396758800 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b33a8d7200 0x55b33a8dcdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:51.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.067+0000 7f78c6800640 1 -- 172.21.15.90:0/3396758800 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b33a6107d0 con 0x55b33a8d6520 2024-04-02T14:47:51.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.067+0000 7f78cfdc5740 10 client.0 _ll_register_callbacks cb 0x55b33a7ed6a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:47:51.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.067+0000 7f78cfdc5740 1 -- 172.21.15.90:0/3396758800 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f78b400a5d0 con 0x55b33a8d6520 2024-04-02T14:47:51.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.068+0000 7f78a7e00640 10 client.29522 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:47:51.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.068+0000 7f78c6800640 1 --2- 172.21.15.90:0/3396758800 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b33a8d6520 0x55b33a8dc890 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7f78c0012e70 tx=0x7f78c0004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=75a1b1403e72ff0d server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:47:51.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.069+0000 7f78a7e00640 1 -- 172.21.15.90:0/3396758800 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78c0025050 con 0x55b33a8d6520 2024-04-02T14:47:51.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.069+0000 7f78a7e00640 1 -- 172.21.15.90:0/3396758800 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 8550+0+0 (secure 0 0 0) 0x7f78c000f910 con 0x55b33a8d6520 2024-04-02T14:47:51.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.069+0000 7f78a7e00640 1 -- 172.21.15.90:0/3396758800 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f78c001b070 con 0x55b33a8d6520 2024-04-02T14:47:51.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.069+0000 7f78cfdc5740 10 client.29522 fetch_fsmap learned FSMap version 27 2024-04-02T14:47:51.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.069+0000 7f78cfdc5740 1 -- 172.21.15.90:0/3396758800 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({fsmap=27}) -- 0x55b33a8ee640 con 0x55b33a8d6520 2024-04-02T14:47:51.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.070+0000 7f78a7e00640 1 -- 172.21.15.90:0/3396758800 <== mon.2 v2:172.21.15.92:3300/0 4 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f78c000fb10 con 0x55b33a8d6520 2024-04-02T14:47:51.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.070+0000 7f78cfdc5740 10 client.29522 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:47:51.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.070+0000 7f78cfdc5740 10 client.29522 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:47:51.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.070+0000 7f78cfdc5740 20 client.29522 populate_metadata read hostname 'smithi090' 2024-04-02T14:47:51.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.070+0000 7f78cfdc5740 1 --2- 172.21.15.90:0/3396758800 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55b33a8ef9f0 0x55b33a8f1de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:51.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.070+0000 7f78cfdc5740 4 client.29522 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:47:51.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.070+0000 7f78cfdc5740 1 -- 172.21.15.90:0/3396758800 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b33a7f2040 con 0x55b33a8ef9f0 2024-04-02T14:47:51.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.070+0000 7f78c5400640 1 --2- 172.21.15.90:0/3396758800 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55b33a8ef9f0 0x55b33a8f1de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:51.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.071+0000 7f78c5400640 1 --2- 172.21.15.90:0/3396758800 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55b33a8ef9f0 0x55b33a8f1de0 crc :-1 s=READY pgs=332 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b0db3c534f40ed3d server_cookie=64fc62ccd784e471 in_seq=0 out_seq=0 2024-04-02T14:47:51.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.071+0000 7f78a7e00640 10 client.29522 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:47:51.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.072+0000 7f78a7e00640 1 -- 172.21.15.90:0/3396758800 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b33a7f2040 con 0x55b33a8ef9f0 2024-04-02T14:47:51.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.072+0000 7f78a7e00640 10 client.29522 handle_command_reply: tid=0 2024-04-02T14:47:51.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.072+0000 7f78cfdc5740 1 client.29522 shutdown 2024-04-02T14:47:51.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.072+0000 7f78cfdc5740 1 -- 172.21.15.90:0/3396758800 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b33a8d6520 msgr2=0x55b33a8dc890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:51.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.072+0000 7f78cfdc5740 1 --2- 172.21.15.90:0/3396758800 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b33a8d6520 0x55b33a8dc890 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7f78c0012e70 tx=0x7f78c0004a40 comp rx=0 tx=0).stop 2024-04-02T14:47:51.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.072+0000 7f78cfdc5740 1 -- 172.21.15.90:0/3396758800 shutdown_connections 2024-04-02T14:47:51.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.072+0000 7f78cfdc5740 1 --2- 172.21.15.90:0/3396758800 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55b33a8ef9f0 0x55b33a8f1de0 crc :-1 s=READY pgs=332 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:51.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.072+0000 7f78cfdc5740 1 --2- 172.21.15.90:0/3396758800 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b33a8df890 0x55b33a8dd310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:51.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.072+0000 7f78cfdc5740 1 --2- 172.21.15.90:0/3396758800 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b33a8d7200 0x55b33a8dcdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:51.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.072+0000 7f78cfdc5740 1 --2- 172.21.15.90:0/3396758800 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b33a8d6520 0x55b33a8dc890 unknown :-1 s=CLOSED pgs=904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:51.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.072+0000 7f78cfdc5740 1 -- 172.21.15.90:0/3396758800 >> 172.21.15.90:0/3396758800 conn(0x55b33a894a50 msgr2=0x55b33a8d2cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:47:51.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.072+0000 7f78a7e00640 0 client.29522 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:47:51.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.072+0000 7f78cfdc5740 1 -- 172.21.15.90:0/3396758800 shutdown_connections 2024-04-02T14:47:51.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.072+0000 7f78cfdc5740 1 -- 172.21.15.90:0/3396758800 wait complete. 2024-04-02T14:47:51.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.073+0000 7f78cfdc5740 20 client.29522 trim_cache size 0 max 16384 2024-04-02T14:47:51.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.091+0000 7f78cfdc5740 1 Processor -- start 2024-04-02T14:47:51.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.091+0000 7f78cfdc5740 1 -- start start 2024-04-02T14:47:51.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.091+0000 7f78cfdc5740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b33a8d6520 0x55b33a8defc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:51.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.092+0000 7f78cfdc5740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b33a8d7200 0x55b33a8eb2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:51.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.092+0000 7f78cfdc5740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b33a8df890 0x55b33a8f7660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:51.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.092+0000 7f78cfdc5740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f78c806fb60 con 0x55b33a8d7200 2024-04-02T14:47:51.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.092+0000 7f78cfdc5740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f78c80716a0 con 0x55b33a8d6520 2024-04-02T14:47:51.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.092+0000 7f78cfdc5740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f78c8070230 con 0x55b33a8df890 2024-04-02T14:47:51.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.092+0000 7f78c6800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b33a8d6520 0x55b33a8defc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:51.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.092+0000 7f78c5400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b33a8d7200 0x55b33a8eb2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:51.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.092+0000 7f78c6800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b33a8d6520 0x55b33a8defc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:50312/0 (socket says 172.21.15.90:50312) 2024-04-02T14:47:51.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.092+0000 7f78c6800640 1 -- 172.21.15.90:0/913862884 learned_addr learned my addr 172.21.15.90:0/913862884 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:47:51.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.092+0000 7f78c6800640 1 --2- 172.21.15.90:0/913862884 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b33a8df890 0x55b33a8f7660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:51.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.092+0000 7f78c6800640 1 -- 172.21.15.90:0/913862884 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b33a8df890 msgr2=0x55b33a8f7660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:47:51.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.092+0000 7f78c6800640 1 --2- 172.21.15.90:0/913862884 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b33a8df890 0x55b33a8f7660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:51.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.092+0000 7f78c6800640 1 -- 172.21.15.90:0/913862884 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b33a8d7200 msgr2=0x55b33a8eb2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:47:51.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.092+0000 7f78c6800640 1 --2- 172.21.15.90:0/913862884 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b33a8d7200 0x55b33a8eb2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:51.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.092+0000 7f78c6800640 1 -- 172.21.15.90:0/913862884 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b33a8d8450 con 0x55b33a8d6520 2024-04-02T14:47:51.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.093+0000 7f78c6800640 1 --2- 172.21.15.90:0/913862884 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b33a8d6520 0x55b33a8defc0 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f78c0004a80 tx=0x7f78c0019000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d5dd1075f159208 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:47:51.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.093+0000 7f78b2a00640 1 -- 172.21.15.90:0/913862884 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78c0025050 con 0x55b33a8d6520 2024-04-02T14:47:51.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.093+0000 7f78b2a00640 1 -- 172.21.15.90:0/913862884 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f78c002e070 con 0x55b33a8d6520 2024-04-02T14:47:51.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.093+0000 7f78b2a00640 1 -- 172.21.15.90:0/913862884 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78c0016030 con 0x55b33a8d6520 2024-04-02T14:47:51.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.094+0000 7f78cfdc5740 1 -- 172.21.15.90:0/913862884 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b33a8d6520 msgr2=0x55b33a8defc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:51.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.094+0000 7f78cfdc5740 1 --2- 172.21.15.90:0/913862884 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b33a8d6520 0x55b33a8defc0 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f78c0004a80 tx=0x7f78c0019000 comp rx=0 tx=0).stop 2024-04-02T14:47:51.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.094+0000 7f78cfdc5740 1 -- 172.21.15.90:0/913862884 shutdown_connections 2024-04-02T14:47:51.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.094+0000 7f78cfdc5740 1 --2- 172.21.15.90:0/913862884 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b33a8df890 0x55b33a8f7660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:51.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.094+0000 7f78cfdc5740 1 --2- 172.21.15.90:0/913862884 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b33a8d7200 0x55b33a8eb2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:51.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.094+0000 7f78cfdc5740 1 --2- 172.21.15.90:0/913862884 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b33a8d6520 0x55b33a8defc0 unknown :-1 s=CLOSED pgs=905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:51.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.094+0000 7f78cfdc5740 1 -- 172.21.15.90:0/913862884 >> 172.21.15.90:0/913862884 conn(0x55b33a894a50 msgr2=0x55b33a8c78a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:47:51.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.094+0000 7f78cfdc5740 1 -- 172.21.15.90:0/913862884 shutdown_connections 2024-04-02T14:47:51.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.094+0000 7f78cfdc5740 1 -- 172.21.15.90:0/913862884 wait complete. 2024-04-02T14:47:51.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.094+0000 7f78cfdc5740 1 -- start start 2024-04-02T14:47:51.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.094+0000 7f78cfdc5740 1 Processor -- start 2024-04-02T14:47:51.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.095+0000 7f78cfdc5740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b33a8d6520 0x55b33a8e7770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:51.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.095+0000 7f78cfdc5740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b33a8d7200 0x55b33a8e7cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:51.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.095+0000 7f78cfdc5740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b33a8df890 0x55b33a8e2720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:51.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.095+0000 7f78cfdc5740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f78c8070fd0 con 0x55b33a8df890 2024-04-02T14:47:51.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.095+0000 7f78cfdc5740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f78c8070900 con 0x55b33a8d6520 2024-04-02T14:47:51.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.095+0000 7f78cfdc5740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f78c806db60 con 0x55b33a8d7200 2024-04-02T14:47:51.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.095+0000 7f78cfdc5740 10 client.0 _ll_register_callbacks cb 0x7f78c0012d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:47:51.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.095+0000 7f78c5400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b33a8d7200 0x55b33a8e7cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:51.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.095+0000 7f78c5400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b33a8d7200 0x55b33a8e7cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:34780/0 (socket says 172.21.15.90:34780) 2024-04-02T14:47:51.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.095+0000 7f78c5400640 1 -- 172.21.15.90:0/801591969 learned_addr learned my addr 172.21.15.90:0/801591969 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:47:51.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.095+0000 7f78c6800640 1 --2- 172.21.15.90:0/801591969 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b33a8d6520 0x55b33a8e7770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:51.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.095+0000 7f78c6800640 1 --2- 172.21.15.90:0/801591969 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b33a8df890 0x55b33a8e2720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:51.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.095+0000 7f78c5400640 1 -- 172.21.15.90:0/801591969 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b33a8d6520 msgr2=0x55b33a8e7770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:47:51.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.095+0000 7f78c5400640 1 --2- 172.21.15.90:0/801591969 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b33a8d6520 0x55b33a8e7770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:51.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.095+0000 7f78c5400640 1 -- 172.21.15.90:0/801591969 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b33a8df890 msgr2=0x55b33a8e2720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:47:51.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.095+0000 7f78c5400640 1 --2- 172.21.15.90:0/801591969 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b33a8df890 0x55b33a8e2720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:51.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.095+0000 7f78c5400640 1 -- 172.21.15.90:0/801591969 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b33a8e2d90 con 0x55b33a8d7200 2024-04-02T14:47:51.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.096+0000 7f78c6800640 1 --2- 172.21.15.90:0/801591969 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b33a8d6520 0x55b33a8e7770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:47:51.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.096+0000 7f78c5400640 1 --2- 172.21.15.90:0/801591969 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b33a8d7200 0x55b33a8e7cb0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f78b4007320 tx=0x7f78b400a470 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2da6a9727ae344b9 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:47:51.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.096+0000 7f78a7e00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:47:51.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.096+0000 7f78a7e00640 1 -- 172.21.15.90:0/801591969 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78b4054060 con 0x55b33a8d7200 2024-04-02T14:47:51.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.096+0000 7f78a7e00640 1 -- 172.21.15.90:0/801591969 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 8550+0+0 (secure 0 0 0) 0x7f78b400a780 con 0x55b33a8d7200 2024-04-02T14:47:51.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.096+0000 7f78cfdc5740 1 -- 172.21.15.90:0/801591969 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f78c002e070 con 0x55b33a8d7200 2024-04-02T14:47:51.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.096+0000 7f78a7e00640 1 -- 172.21.15.90:0/801591969 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f78b400a980 con 0x55b33a8d7200 2024-04-02T14:47:51.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.096+0000 7f78cfdc5740 10 client.29311 fetch_fsmap learned FSMap version 27 2024-04-02T14:47:51.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.096+0000 7f78cfdc5740 1 -- 172.21.15.90:0/801591969 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55b33a956810 con 0x55b33a8d7200 2024-04-02T14:47:51.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.096+0000 7f78a7e00640 1 -- 172.21.15.90:0/801591969 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f78c002e070 con 0x55b33a8d7200 2024-04-02T14:47:51.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.097+0000 7f78a7e00640 1 -- 172.21.15.90:0/801591969 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f78b404c070 con 0x55b33a8d7200 2024-04-02T14:47:51.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.097+0000 7f78cfdc5740 10 client.29311 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:47:51.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.097+0000 7f78cfdc5740 10 client.29311 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:47:51.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.097+0000 7f78cfdc5740 20 client.29311 populate_metadata read hostname 'smithi090' 2024-04-02T14:47:51.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.097+0000 7f78cfdc5740 1 --2- 172.21.15.90:0/801591969 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55b33a956ba0 0x55b33a958f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:51.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.097+0000 7f78cfdc5740 4 client.29311 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-02T14:47:51.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.097+0000 7f78cfdc5740 1 -- 172.21.15.90:0/801591969 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55b33a83fc30 con 0x55b33a956ba0 2024-04-02T14:47:51.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.097+0000 7f78c6800640 1 --2- 172.21.15.90:0/801591969 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55b33a956ba0 0x55b33a958f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:51.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.098+0000 7f78c6800640 1 --2- 172.21.15.90:0/801591969 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55b33a956ba0 0x55b33a958f90 crc :-1 s=READY pgs=333 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1e695b3734222564 server_cookie=7a9a08c2c304175f in_seq=0 out_seq=0 2024-04-02T14:47:51.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.098+0000 7f78a7e00640 10 client.29311 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:47:51.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.098+0000 7f78a7e00640 1 -- 172.21.15.90:0/801591969 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55b33a83fc30 con 0x55b33a956ba0 2024-04-02T14:47:51.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.099+0000 7f78a7e00640 10 client.29311 handle_command_reply: tid=0 2024-04-02T14:47:51.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.099+0000 7f78cfdc5740 1 client.29311 shutdown 2024-04-02T14:47:51.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.099+0000 7f78cfdc5740 1 -- 172.21.15.90:0/801591969 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b33a8d7200 msgr2=0x55b33a8e7cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:51.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.099+0000 7f78cfdc5740 1 --2- 172.21.15.90:0/801591969 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b33a8d7200 0x55b33a8e7cb0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f78b4007320 tx=0x7f78b400a470 comp rx=0 tx=0).stop 2024-04-02T14:47:51.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.100+0000 7f78cfdc5740 1 -- 172.21.15.90:0/801591969 shutdown_connections 2024-04-02T14:47:51.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.100+0000 7f78cfdc5740 1 --2- 172.21.15.90:0/801591969 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55b33a956ba0 0x55b33a958f90 crc :-1 s=READY pgs=333 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:51.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.100+0000 7f78cfdc5740 1 --2- 172.21.15.90:0/801591969 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b33a8df890 0x55b33a8e2720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:51.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.100+0000 7f78cfdc5740 1 --2- 172.21.15.90:0/801591969 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b33a8d7200 0x55b33a8e7cb0 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:51.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.100+0000 7f78cfdc5740 1 --2- 172.21.15.90:0/801591969 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b33a8d6520 0x55b33a8e7770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:51.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.100+0000 7f78cfdc5740 1 -- 172.21.15.90:0/801591969 >> 172.21.15.90:0/801591969 conn(0x55b33a894a50 msgr2=0x55b33a8dc830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:47:51.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.100+0000 7f78a7e00640 0 client.29311 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:47:51.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.100+0000 7f78cfdc5740 1 -- 172.21.15.90:0/801591969 shutdown_connections 2024-04-02T14:47:51.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.100+0000 7f78cfdc5740 1 -- 172.21.15.90:0/801591969 wait complete. 2024-04-02T14:47:51.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.100+0000 7f78cfdc5740 20 client.29311 trim_cache size 0 max 16384 2024-04-02T14:47:51.099 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-02T14:47:51.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.101+0000 7f78cd000640 1 -- 172.21.15.90:0/1921224773 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f78ac0a3d50 msgr2=0x7f78ac0a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:51.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.101+0000 7f78cd000640 1 --2- 172.21.15.90:0/1921224773 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f78ac0a3d50 0x7f78ac0a6210 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f78b80ce390 tx=0x7f78b8003e30 comp rx=0 tx=0).stop 2024-04-02T14:47:51.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.102+0000 7f78cd000640 1 -- 172.21.15.90:0/1921224773 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f78c80fff50 msgr2=0x7f78c819a580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:51.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.102+0000 7f78cd000640 1 --2- 172.21.15.90:0/1921224773 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f78c80fff50 0x7f78c819a580 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7f78b800ea60 tx=0x7f78b800ef20 comp rx=0 tx=0).stop 2024-04-02T14:47:51.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.102+0000 7f78cd000640 1 -- 172.21.15.90:0/1921224773 shutdown_connections 2024-04-02T14:47:51.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.102+0000 7f78cd000640 1 --2- 172.21.15.90:0/1921224773 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f78ac0a3d50 0x7f78ac0a6210 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:51.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.102+0000 7f78cd000640 1 --2- 172.21.15.90:0/1921224773 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f78c819ee40 0x7f78c819f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:51.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.102+0000 7f78cd000640 1 --2- 172.21.15.90:0/1921224773 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f78c8105020 0x7f78c819aac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:51.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.102+0000 7f78cd000640 1 --2- 172.21.15.90:0/1921224773 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f78c80fff50 0x7f78c819a580 unknown :-1 s=CLOSED pgs=902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:51.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.102+0000 7f78cd000640 1 -- 172.21.15.90:0/1921224773 >> 172.21.15.90:0/1921224773 conn(0x7f78c8076b50 msgr2=0x7f78c81031a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:47:51.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.102+0000 7f78cd000640 1 -- 172.21.15.90:0/1921224773 shutdown_connections 2024-04-02T14:47:51.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:51.102+0000 7f78cd000640 1 -- 172.21.15.90:0/1921224773 wait complete. 2024-04-02T14:47:51.824 INFO:teuthology.orchestra.run.smithi090.stderr:progress: 740.0 s, 1237.6 tps, lat 82.271 ms stddev 97.605 2024-04-02T14:47:52.121 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-02T14:47:52.121 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T14:47:52.121 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-02T14:47:52.237 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.242+0000 7fcc0ac00640 1 -- 172.21.15.90:0/2947883638 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcc0406a300 msgr2=0x7fcc0406a780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:52.238 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.242+0000 7fcc0ac00640 1 --2- 172.21.15.90:0/2947883638 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcc0406a300 0x7fcc0406a780 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7fcbf0009890 tx=0x7fcbf002bb70 comp rx=0 tx=0).stop 2024-04-02T14:47:52.239 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.244+0000 7fcc0ac00640 1 -- 172.21.15.90:0/2947883638 shutdown_connections 2024-04-02T14:47:52.239 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.244+0000 7fcc0ac00640 1 --2- 172.21.15.90:0/2947883638 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcc0406acc0 0x7fcc0410e990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:52.239 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.244+0000 7fcc0ac00640 1 --2- 172.21.15.90:0/2947883638 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcc0406a300 0x7fcc0406a780 unknown :-1 s=CLOSED pgs=906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:52.239 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.244+0000 7fcc0ac00640 1 --2- 172.21.15.90:0/2947883638 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcc04107b80 0x7fcc04107f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:52.239 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.244+0000 7fcc0ac00640 1 -- 172.21.15.90:0/2947883638 >> 172.21.15.90:0/2947883638 conn(0x7fcc040777d0 msgr2=0x7fcc04077be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:47:52.239 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.244+0000 7fcc0ac00640 1 -- 172.21.15.90:0/2947883638 shutdown_connections 2024-04-02T14:47:52.240 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.245+0000 7fcc0ac00640 1 -- 172.21.15.90:0/2947883638 wait complete. 2024-04-02T14:47:52.241 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.246+0000 7fcc0ac00640 1 Processor -- start 2024-04-02T14:47:52.241 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.246+0000 7fcc0ac00640 1 -- start start 2024-04-02T14:47:52.241 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.247+0000 7fcc0ac00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcc0406a300 0x7fcc0419a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:52.241 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.247+0000 7fcc0ac00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcc0406acc0 0x7fcc0419aa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:52.242 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.247+0000 7fcc0ac00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcc04107b80 0x7fcc0419eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:52.242 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.247+0000 7fcc0ac00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fcc04111eb0 con 0x7fcc0406a300 2024-04-02T14:47:52.242 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.247+0000 7fcc0ac00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fcc04111d30 con 0x7fcc04107b80 2024-04-02T14:47:52.242 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.247+0000 7fcc0ac00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fcc04112030 con 0x7fcc0406acc0 2024-04-02T14:47:52.242 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.247+0000 7fcc08c00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcc04107b80 0x7fcc0419eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:52.242 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.248+0000 7fcc03400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcc0406acc0 0x7fcc0419aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:52.243 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.248+0000 7fcc03400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcc0406acc0 0x7fcc0419aa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:34804/0 (socket says 172.21.15.90:34804) 2024-04-02T14:47:52.243 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.248+0000 7fcc03400640 1 -- 172.21.15.90:0/3855204406 learned_addr learned my addr 172.21.15.90:0/3855204406 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:47:52.243 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.248+0000 7fcc03e00640 1 --2- 172.21.15.90:0/3855204406 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcc0406a300 0x7fcc0419a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:52.245 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.248+0000 7fcc08c00640 1 -- 172.21.15.90:0/3855204406 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcc0406acc0 msgr2=0x7fcc0419aa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:52.245 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.248+0000 7fcc08c00640 1 --2- 172.21.15.90:0/3855204406 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcc0406acc0 0x7fcc0419aa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:52.245 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.248+0000 7fcc08c00640 1 -- 172.21.15.90:0/3855204406 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcc0406a300 msgr2=0x7fcc0419a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:52.245 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.248+0000 7fcc08c00640 1 --2- 172.21.15.90:0/3855204406 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcc0406a300 0x7fcc0419a4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:52.245 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.248+0000 7fcc08c00640 1 -- 172.21.15.90:0/3855204406 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc0419f4d0 con 0x7fcc04107b80 2024-04-02T14:47:52.245 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.248+0000 7fcc03400640 1 --2- 172.21.15.90:0/3855204406 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcc0406acc0 0x7fcc0419aa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:47:52.245 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.249+0000 7fcc03e00640 1 --2- 172.21.15.90:0/3855204406 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcc0406a300 0x7fcc0419a4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:47:52.245 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.249+0000 7fcc08c00640 1 --2- 172.21.15.90:0/3855204406 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcc04107b80 0x7fcc0419eda0 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7fcbfc009fd0 tx=0x7fcbfc00ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:47:52.245 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.250+0000 7fcc00c00640 1 -- 172.21.15.90:0/3855204406 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbfc00ce20 con 0x7fcc04107b80 2024-04-02T14:47:52.245 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.250+0000 7fcc00c00640 1 -- 172.21.15.90:0/3855204406 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcbfc005020 con 0x7fcc04107b80 2024-04-02T14:47:52.245 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.250+0000 7fcc00c00640 1 -- 172.21.15.90:0/3855204406 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbfc010720 con 0x7fcc04107b80 2024-04-02T14:47:52.246 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.250+0000 7fcc0ac00640 1 -- 172.21.15.90:0/3855204406 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc0419f7c0 con 0x7fcc04107b80 2024-04-02T14:47:52.246 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.250+0000 7fcc0ac00640 1 -- 172.21.15.90:0/3855204406 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc041a7050 con 0x7fcc04107b80 2024-04-02T14:47:52.250 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.255+0000 7fcc0d9d4740 1 Processor -- start 2024-04-02T14:47:52.251 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.256+0000 7fcc00c00640 1 -- 172.21.15.90:0/3855204406 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fcbfc005540 con 0x7fcc04107b80 2024-04-02T14:47:52.251 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.256+0000 7fcc0d9d4740 1 -- start start 2024-04-02T14:47:52.251 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.257+0000 7fcc0d9d4740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c1abe5aeb0 0x55c1abe5b290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:52.251 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.257+0000 7fcc0d9d4740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c1abe5b7d0 0x55c1abe63e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:52.251 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.257+0000 7fcc0d9d4740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c1abe643b0 0x55c1abe667a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:52.252 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.257+0000 7fcc0d9d4740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55c1abdbfd70 con 0x55c1abe5b7d0 2024-04-02T14:47:52.252 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.257+0000 7fcc0d9d4740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55c1abd682a0 con 0x55c1abe5aeb0 2024-04-02T14:47:52.252 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.257+0000 7fcc0d9d4740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55c1abdbbd20 con 0x55c1abe643b0 2024-04-02T14:47:52.253 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.258+0000 7fcc03400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c1abe643b0 0x55c1abe667a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:52.253 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.258+0000 7fcc03400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c1abe5aeb0 0x55c1abe5b290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:52.253 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.258+0000 7fcc03400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c1abe5aeb0 0x55c1abe5b290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:50346/0 (socket says 172.21.15.90:50346) 2024-04-02T14:47:52.253 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.258+0000 7fcc03400640 1 -- 172.21.15.90:0/987727649 learned_addr learned my addr 172.21.15.90:0/987727649 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:47:52.254 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.259+0000 7fcc03e00640 1 --2- 172.21.15.90:0/987727649 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c1abe5b7d0 0x55c1abe63e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:52.255 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.259+0000 7fcc00c00640 1 --2- 172.21.15.90:0/3855204406 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fcbe80a3e40 0x7fcbe80a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:52.255 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.260+0000 7fcc03400640 1 -- 172.21.15.90:0/987727649 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c1abe643b0 msgr2=0x55c1abe667a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:47:52.255 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.260+0000 7fcc03400640 1 --2- 172.21.15.90:0/987727649 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c1abe643b0 0x55c1abe667a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:52.255 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.260+0000 7fcc03400640 1 -- 172.21.15.90:0/987727649 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c1abe5b7d0 msgr2=0x55c1abe63e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:47:52.255 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.260+0000 7fcc03400640 1 --2- 172.21.15.90:0/987727649 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c1abe5b7d0 0x55c1abe63e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:52.256 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.260+0000 7fcc03400640 1 -- 172.21.15.90:0/987727649 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c1abd7c840 con 0x55c1abe5aeb0 2024-04-02T14:47:52.256 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.260+0000 7fcc00c00640 1 -- 172.21.15.90:0/3855204406 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 8550+0+0 (secure 0 0 0) 0x7fcbfc0d0f00 con 0x7fcc04107b80 2024-04-02T14:47:52.256 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.260+0000 7fcc08c00640 1 --2- 172.21.15.90:0/3855204406 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fcbe80a3e40 0x7fcbe80a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:52.256 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.260+0000 7fcc03400640 1 --2- 172.21.15.90:0/987727649 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c1abe5aeb0 0x55c1abe5b290 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7fcbf0004200 tx=0x7fcbf00056d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fc66250c3a7ff026 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:47:52.256 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.261+0000 7fcc08c00640 1 --2- 172.21.15.90:0/3855204406 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fcbe80a3e40 0x7fcbe80a6300 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fcbfc0a55c0 tx=0x7fcbfc005f70 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:47:52.257 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.262+0000 7fcbf9200640 1 -- 172.21.15.90:0/987727649 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbf0039040 con 0x55c1abe5aeb0 2024-04-02T14:47:52.257 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.262+0000 7fcbf9200640 1 -- 172.21.15.90:0/987727649 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcbf0049070 con 0x55c1abe5aeb0 2024-04-02T14:47:52.257 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.262+0000 7fcbf9200640 1 -- 172.21.15.90:0/987727649 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbf0032640 con 0x55c1abe5aeb0 2024-04-02T14:47:52.258 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.263+0000 7fcc0d9d4740 1 -- 172.21.15.90:0/987727649 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c1abe5aeb0 msgr2=0x55c1abe5b290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:52.258 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.263+0000 7fcc0d9d4740 1 --2- 172.21.15.90:0/987727649 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c1abe5aeb0 0x55c1abe5b290 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7fcbf0004200 tx=0x7fcbf00056d0 comp rx=0 tx=0).stop 2024-04-02T14:47:52.259 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.265+0000 7fcc0d9d4740 1 -- 172.21.15.90:0/987727649 shutdown_connections 2024-04-02T14:47:52.259 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.265+0000 7fcc0d9d4740 1 --2- 172.21.15.90:0/987727649 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c1abe643b0 0x55c1abe667a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:52.259 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.265+0000 7fcc0d9d4740 1 --2- 172.21.15.90:0/987727649 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c1abe5b7d0 0x55c1abe63e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:52.260 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.265+0000 7fcc0d9d4740 1 --2- 172.21.15.90:0/987727649 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c1abe5aeb0 0x55c1abe5b290 unknown :-1 s=CLOSED pgs=908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:52.260 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.265+0000 7fcc0d9d4740 1 -- 172.21.15.90:0/987727649 >> 172.21.15.90:0/987727649 conn(0x55c1abe555e0 msgr2=0x55c1abe559c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:47:52.260 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.266+0000 7fcc0d9d4740 1 -- 172.21.15.90:0/987727649 shutdown_connections 2024-04-02T14:47:52.260 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.266+0000 7fcc0d9d4740 1 -- 172.21.15.90:0/987727649 wait complete. 2024-04-02T14:47:52.261 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.266+0000 7fcc0d9d4740 1 -- start start 2024-04-02T14:47:52.261 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.267+0000 7fcc0d9d4740 1 Processor -- start 2024-04-02T14:47:52.264 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.268+0000 7fcc0d9d4740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c1abe5aeb0 0x55c1abe61400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:52.264 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.268+0000 7fcc0d9d4740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c1abe5b7d0 0x55c1abe61940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:52.264 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.268+0000 7fcc0d9d4740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c1abe643b0 0x55c1abe61e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:52.264 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.268+0000 7fcc0d9d4740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55c1abe67140 con 0x55c1abe5aeb0 2024-04-02T14:47:52.264 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.268+0000 7fcc0d9d4740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55c1abd7b660 con 0x55c1abe643b0 2024-04-02T14:47:52.264 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.268+0000 7fcc0d9d4740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55c1abe672c0 con 0x55c1abe5b7d0 2024-04-02T14:47:52.264 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.268+0000 7fcc0d9d4740 10 client.0 _ll_register_callbacks cb 0x55c1abd7b050 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:47:52.265 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.270+0000 7fcc03400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c1abe5aeb0 0x55c1abe61400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:52.265 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.270+0000 7fcc03400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c1abe5aeb0 0x55c1abe61400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34114/0 (socket says 172.21.15.90:34114) 2024-04-02T14:47:52.265 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.270+0000 7fcc03400640 1 -- 172.21.15.90:0/1997064793 learned_addr learned my addr 172.21.15.90:0/1997064793 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:47:52.265 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.271+0000 7fcc03e00640 1 --2- 172.21.15.90:0/1997064793 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c1abe5b7d0 0x55c1abe61940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:52.266 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.271+0000 7fcc03400640 1 --2- 172.21.15.90:0/1997064793 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c1abe643b0 0x55c1abe61e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:52.266 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.271+0000 7fcc03400640 1 -- 172.21.15.90:0/1997064793 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c1abe5b7d0 msgr2=0x55c1abe61940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:47:52.266 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.271+0000 7fcc03400640 1 --2- 172.21.15.90:0/1997064793 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c1abe5b7d0 0x55c1abe61940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:52.266 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.271+0000 7fcc03400640 1 -- 172.21.15.90:0/1997064793 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c1abe643b0 msgr2=0x55c1abe61e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:47:52.266 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.271+0000 7fcc03400640 1 --2- 172.21.15.90:0/1997064793 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c1abe643b0 0x55c1abe61e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:52.267 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.271+0000 7fcc03400640 1 -- 172.21.15.90:0/1997064793 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c1abd8ae80 con 0x55c1abe5aeb0 2024-04-02T14:47:52.268 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.273+0000 7fcc03400640 1 --2- 172.21.15.90:0/1997064793 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c1abe5aeb0 0x55c1abe61400 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fcbf00099b0 tx=0x7fcbf0045040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6518ad34d9c59e9e server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:47:52.268 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.273+0000 7fcbed600640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-02T14:47:52.268 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.274+0000 7fcbed600640 1 -- 172.21.15.90:0/1997064793 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbf0055060 con 0x55c1abe5aeb0 2024-04-02T14:47:52.269 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.274+0000 7fcc0d9d4740 1 -- 172.21.15.90:0/1997064793 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcbf0049070 con 0x55c1abe5aeb0 2024-04-02T14:47:52.270 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.274+0000 7fcbed600640 1 -- 172.21.15.90:0/1997064793 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 8550+0+0 (secure 0 0 0) 0x7fcbf002c720 con 0x55c1abe5aeb0 2024-04-02T14:47:52.270 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.275+0000 7fcbed600640 1 -- 172.21.15.90:0/1997064793 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fcbf0055060 con 0x55c1abe5aeb0 2024-04-02T14:47:52.270 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.275+0000 7fcc0d9d4740 10 client.18402 fetch_fsmap learned FSMap version 27 2024-04-02T14:47:52.270 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.275+0000 7fcc0d9d4740 1 -- 172.21.15.90:0/1997064793 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55c1abad4210 con 0x55c1abe5aeb0 2024-04-02T14:47:52.271 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.276+0000 7fcbed600640 1 -- 172.21.15.90:0/1997064793 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcbf0049070 con 0x55c1abe5aeb0 2024-04-02T14:47:52.271 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.276+0000 7fcbed600640 1 -- 172.21.15.90:0/1997064793 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fcbf003c070 con 0x55c1abe5aeb0 2024-04-02T14:47:52.271 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.276+0000 7fcc0d9d4740 10 client.18402 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:47:52.271 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.276+0000 7fcc0d9d4740 10 client.18402 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-02T14:47:52.271 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.276+0000 7fcc0d9d4740 20 client.18402 populate_metadata read hostname 'smithi090' 2024-04-02T14:47:52.271 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.276+0000 7fcc0d9d4740 1 --2- 172.21.15.90:0/1997064793 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55c1abe7cdf0 0x55c1abe7f1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:52.271 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.276+0000 7fcc0d9d4740 4 client.18402 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:47:52.271 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.276+0000 7fcc0d9d4740 1 -- 172.21.15.90:0/1997064793 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c1abd6fcd0 con 0x55c1abe7cdf0 2024-04-02T14:47:52.272 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.277+0000 7fcc03e00640 1 --2- 172.21.15.90:0/1997064793 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55c1abe7cdf0 0x55c1abe7f1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:52.272 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.277+0000 7fcc03e00640 1 --2- 172.21.15.90:0/1997064793 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55c1abe7cdf0 0x55c1abe7f1e0 crc :-1 s=READY pgs=334 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=249cb5dec65e16ab server_cookie=5c4815c28ffa78a0 in_seq=0 out_seq=0 2024-04-02T14:47:52.272 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.277+0000 7fcbed600640 10 client.18402 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:47:52.273 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.278+0000 7fcbed600640 1 -- 172.21.15.90:0/1997064793 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c1abd6fcd0 con 0x55c1abe7cdf0 2024-04-02T14:47:52.273 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.278+0000 7fcbed600640 10 client.18402 handle_command_reply: tid=0 2024-04-02T14:47:52.273 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.278+0000 7fcc0d9d4740 1 client.18402 shutdown 2024-04-02T14:47:52.273 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.278+0000 7fcc0d9d4740 1 -- 172.21.15.90:0/1997064793 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c1abe5aeb0 msgr2=0x55c1abe61400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:52.273 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.278+0000 7fcc0d9d4740 1 --2- 172.21.15.90:0/1997064793 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c1abe5aeb0 0x55c1abe61400 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fcbf00099b0 tx=0x7fcbf0045040 comp rx=0 tx=0).stop 2024-04-02T14:47:52.274 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.278+0000 7fcc0d9d4740 1 -- 172.21.15.90:0/1997064793 shutdown_connections 2024-04-02T14:47:52.274 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.278+0000 7fcc0d9d4740 1 --2- 172.21.15.90:0/1997064793 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55c1abe7cdf0 0x55c1abe7f1e0 crc :-1 s=READY pgs=334 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:52.274 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.278+0000 7fcc0d9d4740 1 --2- 172.21.15.90:0/1997064793 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c1abe643b0 0x55c1abe61e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:52.274 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.278+0000 7fcc0d9d4740 1 --2- 172.21.15.90:0/1997064793 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c1abe5b7d0 0x55c1abe61940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:52.274 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.278+0000 7fcc0d9d4740 1 --2- 172.21.15.90:0/1997064793 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c1abe5aeb0 0x55c1abe61400 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:52.274 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.279+0000 7fcc0d9d4740 1 -- 172.21.15.90:0/1997064793 >> 172.21.15.90:0/1997064793 conn(0x55c1abe555e0 msgr2=0x55c1abe57330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:47:52.274 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.279+0000 7fcbed600640 0 client.18402 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:47:52.275 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.279+0000 7fcc0d9d4740 1 -- 172.21.15.90:0/1997064793 shutdown_connections 2024-04-02T14:47:52.275 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.279+0000 7fcc0d9d4740 1 -- 172.21.15.90:0/1997064793 wait complete. 2024-04-02T14:47:52.275 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.279+0000 7fcc0d9d4740 20 client.18402 trim_cache size 0 max 16384 2024-04-02T14:47:52.309 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.314+0000 7fcc0d9d4740 1 Processor -- start 2024-04-02T14:47:52.309 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.315+0000 7fcc0d9d4740 1 -- start start 2024-04-02T14:47:52.310 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.315+0000 7fcc0d9d4740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c1abe5b7d0 0x55c1abe63b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:52.311 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.315+0000 7fcc0d9d4740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c1abe643b0 0x55c1abe60560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:52.311 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.315+0000 7fcc0d9d4740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c1abe5e680 0x55c1abe5eae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:52.311 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.315+0000 7fcc0d9d4740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fcc041ab610 con 0x55c1abe643b0 2024-04-02T14:47:52.311 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.315+0000 7fcc0d9d4740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fcc041ad150 con 0x55c1abe5e680 2024-04-02T14:47:52.311 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.315+0000 7fcc0d9d4740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fcc041abce0 con 0x55c1abe5b7d0 2024-04-02T14:47:52.311 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.316+0000 7fcc03e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c1abe643b0 0x55c1abe60560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:52.311 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.316+0000 7fcc03400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c1abe5b7d0 0x55c1abe63b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:52.311 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.316+0000 7fcc03400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c1abe5e680 0x55c1abe5eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:52.311 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.316+0000 7fcc03400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c1abe5e680 0x55c1abe5eae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:50364/0 (socket says 172.21.15.90:50364) 2024-04-02T14:47:52.311 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.316+0000 7fcc03400640 1 -- 172.21.15.90:0/53778383 learned_addr learned my addr 172.21.15.90:0/53778383 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:47:52.311 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.316+0000 7fcc03e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c1abe643b0 0x55c1abe60560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34118/0 (socket says 172.21.15.90:34118) 2024-04-02T14:47:52.312 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.317+0000 7fcc03400640 1 -- 172.21.15.90:0/53778383 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c1abe5b7d0 msgr2=0x55c1abe63b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:47:52.312 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.317+0000 7fcc03400640 1 --2- 172.21.15.90:0/53778383 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c1abe5b7d0 0x55c1abe63b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:52.312 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.317+0000 7fcc03400640 1 -- 172.21.15.90:0/53778383 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c1abe643b0 msgr2=0x55c1abe60560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:47:52.312 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.317+0000 7fcc03400640 1 --2- 172.21.15.90:0/53778383 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c1abe643b0 0x55c1abe60560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:52.312 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.317+0000 7fcc03400640 1 -- 172.21.15.90:0/53778383 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c1abba0620 con 0x55c1abe5e680 2024-04-02T14:47:52.312 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.317+0000 7fcc03400640 1 --2- 172.21.15.90:0/53778383 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c1abe5e680 0x55c1abe5eae0 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7fcbf0006010 tx=0x7fcbf0005bd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fdfecbb2cde7516a server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:47:52.313 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.318+0000 7fcbefe00640 1 -- 172.21.15.90:0/53778383 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbf0039070 con 0x55c1abe5e680 2024-04-02T14:47:52.313 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.318+0000 7fcbefe00640 1 -- 172.21.15.90:0/53778383 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcbf0049070 con 0x55c1abe5e680 2024-04-02T14:47:52.313 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.318+0000 7fcbefe00640 1 -- 172.21.15.90:0/53778383 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbf00346a0 con 0x55c1abe5e680 2024-04-02T14:47:52.314 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.319+0000 7fcc0d9d4740 1 -- 172.21.15.90:0/53778383 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c1abe5e680 msgr2=0x55c1abe5eae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:52.315 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.319+0000 7fcc0d9d4740 1 --2- 172.21.15.90:0/53778383 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c1abe5e680 0x55c1abe5eae0 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7fcbf0006010 tx=0x7fcbf0005bd0 comp rx=0 tx=0).stop 2024-04-02T14:47:52.315 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.320+0000 7fcc0d9d4740 1 -- 172.21.15.90:0/53778383 shutdown_connections 2024-04-02T14:47:52.315 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.320+0000 7fcc0d9d4740 1 --2- 172.21.15.90:0/53778383 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c1abe5e680 0x55c1abe5eae0 unknown :-1 s=CLOSED pgs=909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:52.315 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.320+0000 7fcc0d9d4740 1 --2- 172.21.15.90:0/53778383 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c1abe643b0 0x55c1abe60560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:52.315 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.320+0000 7fcc0d9d4740 1 --2- 172.21.15.90:0/53778383 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c1abe5b7d0 0x55c1abe63b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:52.315 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.320+0000 7fcc0d9d4740 1 -- 172.21.15.90:0/53778383 >> 172.21.15.90:0/53778383 conn(0x55c1abe555e0 msgr2=0x55c1abe5bbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:47:52.316 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.321+0000 7fcc0d9d4740 1 -- 172.21.15.90:0/53778383 shutdown_connections 2024-04-02T14:47:52.316 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.321+0000 7fcc0d9d4740 1 -- 172.21.15.90:0/53778383 wait complete. 2024-04-02T14:47:52.316 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.321+0000 7fcc0d9d4740 1 -- start start 2024-04-02T14:47:52.316 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.321+0000 7fcc0d9d4740 1 Processor -- start 2024-04-02T14:47:52.317 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.322+0000 7fcc0d9d4740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c1abe5b7d0 0x55c1abe82540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:52.317 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.322+0000 7fcc0d9d4740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c1abe5e680 0x55c1abe7f8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:52.317 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.322+0000 7fcc0d9d4740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c1abe643b0 0x55c1abe7fe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:52.317 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.322+0000 7fcc0d9d4740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fcc041aca80 con 0x55c1abe5b7d0 2024-04-02T14:47:52.317 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.322+0000 7fcc0d9d4740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fcc041ac3b0 con 0x55c1abe5e680 2024-04-02T14:47:52.317 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.322+0000 7fcc0d9d4740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fcc041a9630 con 0x55c1abe643b0 2024-04-02T14:47:52.317 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.322+0000 7fcc0d9d4740 10 client.0 _ll_register_callbacks cb 0x7fcbf00096b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:47:52.318 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.323+0000 7fcc03e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c1abe5e680 0x55c1abe7f8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:52.318 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.323+0000 7fcc03400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c1abe5b7d0 0x55c1abe82540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:52.318 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.323+0000 7fcc03e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c1abe5e680 0x55c1abe7f8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:50366/0 (socket says 172.21.15.90:50366) 2024-04-02T14:47:52.318 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.323+0000 7fcc03e00640 1 -- 172.21.15.90:0/1793939965 learned_addr learned my addr 172.21.15.90:0/1793939965 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:47:52.319 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.324+0000 7fcc03400640 1 --2- 172.21.15.90:0/1793939965 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c1abe643b0 0x55c1abe7fe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:52.320 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.325+0000 7fcc03e00640 1 -- 172.21.15.90:0/1793939965 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c1abe643b0 msgr2=0x55c1abe7fe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:47:52.320 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.325+0000 7fcc03e00640 1 --2- 172.21.15.90:0/1793939965 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c1abe643b0 0x55c1abe7fe10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:52.320 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.325+0000 7fcc03e00640 1 -- 172.21.15.90:0/1793939965 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c1abe5b7d0 msgr2=0x55c1abe82540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:47:52.320 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.325+0000 7fcc03e00640 1 --2- 172.21.15.90:0/1793939965 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c1abe5b7d0 0x55c1abe82540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:52.321 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.326+0000 7fcc03e00640 1 -- 172.21.15.90:0/1793939965 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c1abe80610 con 0x55c1abe5e680 2024-04-02T14:47:52.321 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.326+0000 7fcc03400640 1 --2- 172.21.15.90:0/1793939965 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c1abe5b7d0 0x55c1abe82540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:47:52.321 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.326+0000 7fcc03e00640 1 --2- 172.21.15.90:0/1793939965 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c1abe5e680 0x55c1abe7f8d0 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x55c1abe60c60 tx=0x7fcbf40204d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c61b12fb1e8d4940 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:47:52.321 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.326+0000 7fcbed600640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:47:52.321 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.326+0000 7fcbed600640 1 -- 172.21.15.90:0/1793939965 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbf4004230 con 0x55c1abe5e680 2024-04-02T14:47:52.322 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.326+0000 7fcbed600640 1 -- 172.21.15.90:0/1793939965 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 8550+0+0 (secure 0 0 0) 0x7fcbf40043d0 con 0x55c1abe5e680 2024-04-02T14:47:52.322 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.327+0000 7fcc0d9d4740 1 -- 172.21.15.90:0/1793939965 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcbf0049070 con 0x55c1abe5e680 2024-04-02T14:47:52.322 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.327+0000 7fcbed600640 1 -- 172.21.15.90:0/1793939965 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fcbf4030050 con 0x55c1abe5e680 2024-04-02T14:47:52.322 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.327+0000 7fcc0d9d4740 10 client.29546 fetch_fsmap learned FSMap version 27 2024-04-02T14:47:52.323 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.327+0000 7fcc0d9d4740 1 -- 172.21.15.90:0/1793939965 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55c1abee2fe0 con 0x55c1abe5e680 2024-04-02T14:47:52.323 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.328+0000 7fcbed600640 1 -- 172.21.15.90:0/1793939965 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcbf0049070 con 0x55c1abe5e680 2024-04-02T14:47:52.323 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.328+0000 7fcbed600640 1 -- 172.21.15.90:0/1793939965 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fcbf4003be0 con 0x55c1abe5e680 2024-04-02T14:47:52.323 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.328+0000 7fcc0d9d4740 10 client.29546 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:47:52.323 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.328+0000 7fcc0d9d4740 10 client.29546 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-02T14:47:52.323 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.328+0000 7fcc0d9d4740 20 client.29546 populate_metadata read hostname 'smithi090' 2024-04-02T14:47:52.323 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.328+0000 7fcc0d9d4740 1 --2- 172.21.15.90:0/1793939965 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55c1abee3740 0x55c1abee5b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:52.323 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.328+0000 7fcc0d9d4740 4 client.29546 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-02T14:47:52.323 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.328+0000 7fcc0d9d4740 1 -- 172.21.15.90:0/1793939965 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55c1abdcd2b0 con 0x55c1abee3740 2024-04-02T14:47:52.324 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.329+0000 7fcc03400640 1 --2- 172.21.15.90:0/1793939965 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55c1abee3740 0x55c1abee5b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:52.325 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.330+0000 7fcbed600640 10 client.29546 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:47:52.325 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.330+0000 7fcc03400640 1 --2- 172.21.15.90:0/1793939965 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55c1abee3740 0x55c1abee5b60 crc :-1 s=READY pgs=335 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d47a9bc2a928f829 server_cookie=1d484bfcf5c8cdd2 in_seq=0 out_seq=0 2024-04-02T14:47:52.445 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.448+0000 7fcbed600640 1 -- 172.21.15.90:0/1793939965 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55c1abdcd2b0 con 0x55c1abee3740 2024-04-02T14:47:52.445 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.448+0000 7fcbed600640 10 client.29546 handle_command_reply: tid=0 2024-04-02T14:47:52.445 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.448+0000 7fcc0d9d4740 1 client.29546 shutdown 2024-04-02T14:47:52.445 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.448+0000 7fcc0d9d4740 1 -- 172.21.15.90:0/1793939965 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c1abe5e680 msgr2=0x55c1abe7f8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:52.445 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.448+0000 7fcc0d9d4740 1 --2- 172.21.15.90:0/1793939965 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c1abe5e680 0x55c1abe7f8d0 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x55c1abe60c60 tx=0x7fcbf40204d0 comp rx=0 tx=0).stop 2024-04-02T14:47:52.445 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.449+0000 7fcc0d9d4740 1 -- 172.21.15.90:0/1793939965 shutdown_connections 2024-04-02T14:47:52.446 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.449+0000 7fcc0d9d4740 1 --2- 172.21.15.90:0/1793939965 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55c1abee3740 0x55c1abee5b60 crc :-1 s=READY pgs=335 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:52.446 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.449+0000 7fcc0d9d4740 1 --2- 172.21.15.90:0/1793939965 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c1abe643b0 0x55c1abe7fe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:52.446 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.449+0000 7fcc0d9d4740 1 --2- 172.21.15.90:0/1793939965 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c1abe5e680 0x55c1abe7f8d0 unknown :-1 s=CLOSED pgs=910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:52.446 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.449+0000 7fcc0d9d4740 1 --2- 172.21.15.90:0/1793939965 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c1abe5b7d0 0x55c1abe82540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:52.446 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.449+0000 7fcc0d9d4740 1 -- 172.21.15.90:0/1793939965 >> 172.21.15.90:0/1793939965 conn(0x55c1abe555e0 msgr2=0x55c1abe5fcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:47:52.446 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.449+0000 7fcbed600640 0 client.29546 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:47:52.446 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.449+0000 7fcc0d9d4740 1 -- 172.21.15.90:0/1793939965 shutdown_connections 2024-04-02T14:47:52.446 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.450+0000 7fcc0d9d4740 1 -- 172.21.15.90:0/1793939965 wait complete. 2024-04-02T14:47:52.446 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.450+0000 7fcc0d9d4740 20 client.29546 trim_cache size 0 max 16384 2024-04-02T14:47:52.447 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:{ 2024-04-02T14:47:52.448 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "message": "", 2024-04-02T14:47:52.448 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "return_code": 0 2024-04-02T14:47:52.448 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:} 2024-04-02T14:47:52.448 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.453+0000 7fcc0ac00640 1 -- 172.21.15.90:0/3855204406 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fcbe80a3e40 msgr2=0x7fcbe80a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:52.448 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.453+0000 7fcc0ac00640 1 --2- 172.21.15.90:0/3855204406 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fcbe80a3e40 0x7fcbe80a6300 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fcbfc0a55c0 tx=0x7fcbfc005f70 comp rx=0 tx=0).stop 2024-04-02T14:47:52.448 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.453+0000 7fcc0ac00640 1 -- 172.21.15.90:0/3855204406 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcc04107b80 msgr2=0x7fcc0419eda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:52.448 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.453+0000 7fcc0ac00640 1 --2- 172.21.15.90:0/3855204406 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcc04107b80 0x7fcc0419eda0 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7fcbfc009fd0 tx=0x7fcbfc00ef10 comp rx=0 tx=0).stop 2024-04-02T14:47:52.448 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.453+0000 7fcc0ac00640 1 -- 172.21.15.90:0/3855204406 shutdown_connections 2024-04-02T14:47:52.448 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.453+0000 7fcc0ac00640 1 --2- 172.21.15.90:0/3855204406 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fcbe80a3e40 0x7fcbe80a6300 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:52.449 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.453+0000 7fcc0ac00640 1 --2- 172.21.15.90:0/3855204406 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcc04107b80 0x7fcc0419eda0 unknown :-1 s=CLOSED pgs=907 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:52.449 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.453+0000 7fcc0ac00640 1 --2- 172.21.15.90:0/3855204406 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcc0406acc0 0x7fcc0419aa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:52.449 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.453+0000 7fcc0ac00640 1 --2- 172.21.15.90:0/3855204406 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcc0406a300 0x7fcc0419a4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:52.449 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.453+0000 7fcc0ac00640 1 -- 172.21.15.90:0/3855204406 >> 172.21.15.90:0/3855204406 conn(0x7fcc040777d0 msgr2=0x7fcc040fdc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:47:52.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.454+0000 7f85e8600640 1 -- 172.21.15.90:0/2099025400 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f85e0110320 msgr2=0x7f85e010de10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:52.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.454+0000 7f85e8600640 1 --2- 172.21.15.90:0/2099025400 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f85e0110320 0x7f85e010de10 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f85d8009900 tx=0x7f85d802bc50 comp rx=0 tx=0).stop 2024-04-02T14:47:52.450 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.455+0000 7fcc0ac00640 1 -- 172.21.15.90:0/3855204406 shutdown_connections 2024-04-02T14:47:52.450 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:47:52.455+0000 7fcc0ac00640 1 -- 172.21.15.90:0/3855204406 wait complete. 2024-04-02T14:47:52.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.455+0000 7f85e8600640 1 -- 172.21.15.90:0/2099025400 shutdown_connections 2024-04-02T14:47:52.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.455+0000 7f85e8600640 1 --2- 172.21.15.90:0/2099025400 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f85e011b2e0 0x7f85e011d6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:52.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.455+0000 7f85e8600640 1 --2- 172.21.15.90:0/2099025400 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f85e010e350 0x7f85e010e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:52.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.455+0000 7f85e8600640 1 --2- 172.21.15.90:0/2099025400 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f85e0110320 0x7f85e010de10 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:52.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.455+0000 7f85e8600640 1 -- 172.21.15.90:0/2099025400 >> 172.21.15.90:0/2099025400 conn(0x7f85e006cbe0 msgr2=0x7f85e006cff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:47:52.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.456+0000 7f85e8600640 1 -- 172.21.15.90:0/2099025400 shutdown_connections 2024-04-02T14:47:52.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.456+0000 7f85e8600640 1 -- 172.21.15.90:0/2099025400 wait complete. 2024-04-02T14:47:52.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.458+0000 7f85e8600640 1 Processor -- start 2024-04-02T14:47:52.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.459+0000 7f85e8600640 1 -- start start 2024-04-02T14:47:52.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.460+0000 7f85e8600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f85e010e350 0x7f85e01a71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:52.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.460+0000 7f85e8600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f85e0110320 0x7f85e01a76f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:52.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.460+0000 7f85e8600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f85e011b2e0 0x7f85e01aba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:52.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.460+0000 7f85e8600640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f85e0120c90 con 0x7f85e010e350 2024-04-02T14:47:52.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.460+0000 7f85e8600640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f85e0120b10 con 0x7f85e011b2e0 2024-04-02T14:47:52.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.460+0000 7f85e8600640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f85e0120e10 con 0x7f85e0110320 2024-04-02T14:47:52.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.460+0000 7f85e5200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f85e0110320 0x7f85e01a76f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:52.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.460+0000 7f85e6600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f85e011b2e0 0x7f85e01aba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:52.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.460+0000 7f85e6600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f85e011b2e0 0x7f85e01aba70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:50394/0 (socket says 172.21.15.90:50394) 2024-04-02T14:47:52.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.460+0000 7f85e6600640 1 -- 172.21.15.90:0/180568992 learned_addr learned my addr 172.21.15.90:0/180568992 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:47:52.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.460+0000 7f85e5c00640 1 --2- 172.21.15.90:0/180568992 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f85e010e350 0x7f85e01a71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:52.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.461+0000 7f85e6600640 1 -- 172.21.15.90:0/180568992 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f85e0110320 msgr2=0x7f85e01a76f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:52.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.461+0000 7f85e6600640 1 --2- 172.21.15.90:0/180568992 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f85e0110320 0x7f85e01a76f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:52.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.461+0000 7f85e6600640 1 -- 172.21.15.90:0/180568992 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f85e010e350 msgr2=0x7f85e01a71b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:52.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.461+0000 7f85e6600640 1 --2- 172.21.15.90:0/180568992 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f85e010e350 0x7f85e01a71b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:52.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.461+0000 7f85e6600640 1 -- 172.21.15.90:0/180568992 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85e01ac1a0 con 0x7f85e011b2e0 2024-04-02T14:47:52.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.461+0000 7f85e5c00640 1 --2- 172.21.15.90:0/180568992 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f85e010e350 0x7f85e01a71b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:47:52.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.461+0000 7f85e6600640 1 --2- 172.21.15.90:0/180568992 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f85e011b2e0 0x7f85e01aba70 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7f85d000eaa0 tx=0x7f85d000ef60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:47:52.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.462+0000 7f85de000640 1 -- 172.21.15.90:0/180568992 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85d000cdb0 con 0x7f85e011b2e0 2024-04-02T14:47:52.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.462+0000 7f85de000640 1 -- 172.21.15.90:0/180568992 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f85d0005020 con 0x7f85e011b2e0 2024-04-02T14:47:52.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.462+0000 7f85de000640 1 -- 172.21.15.90:0/180568992 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85d00116b0 con 0x7f85e011b2e0 2024-04-02T14:47:52.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.463+0000 7f85e8600640 1 -- 172.21.15.90:0/180568992 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f85e01ac490 con 0x7f85e011b2e0 2024-04-02T14:47:52.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.463+0000 7f85e8600640 1 -- 172.21.15.90:0/180568992 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f85e01b3d20 con 0x7f85e011b2e0 2024-04-02T14:47:52.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.469+0000 7f85de000640 1 -- 172.21.15.90:0/180568992 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f85d0011850 con 0x7f85e011b2e0 2024-04-02T14:47:52.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.471+0000 7f85eb216740 1 Processor -- start 2024-04-02T14:47:52.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.471+0000 7f85eb216740 1 -- start start 2024-04-02T14:47:52.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.471+0000 7f85eb216740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558ba1bb1030 0x558ba1bafd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:52.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.471+0000 7f85eb216740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558ba1bb0350 0x558ba1bb07d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:52.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.471+0000 7f85eb216740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558ba1bb96c0 0x558ba1bbbab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:52.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.471+0000 7f85eb216740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x558ba1b14310 con 0x558ba1bb0350 2024-04-02T14:47:52.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.471+0000 7f85eb216740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x558ba1abc690 con 0x558ba1bb1030 2024-04-02T14:47:52.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.471+0000 7f85eb216740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x558ba1b102c0 con 0x558ba1bb96c0 2024-04-02T14:47:52.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.472+0000 7f85e5200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558ba1bb1030 0x558ba1bafd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:52.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.472+0000 7f85e5200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558ba1bb1030 0x558ba1bafd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:50398/0 (socket says 172.21.15.90:50398) 2024-04-02T14:47:52.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.472+0000 7f85e5200640 1 -- 172.21.15.90:0/4088845916 learned_addr learned my addr 172.21.15.90:0/4088845916 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:47:52.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.472+0000 7f85e5200640 1 --2- 172.21.15.90:0/4088845916 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558ba1bb96c0 0x558ba1bbbab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:52.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.472+0000 7f85e5c00640 1 --2- 172.21.15.90:0/4088845916 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558ba1bb0350 0x558ba1bb07d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:52.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.472+0000 7f85e5200640 1 -- 172.21.15.90:0/4088845916 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558ba1bb96c0 msgr2=0x558ba1bbbab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:47:52.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.472+0000 7f85e5200640 1 --2- 172.21.15.90:0/4088845916 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558ba1bb96c0 0x558ba1bbbab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:52.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.472+0000 7f85e5200640 1 -- 172.21.15.90:0/4088845916 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558ba1bb0350 msgr2=0x558ba1bb07d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:47:52.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.472+0000 7f85e5200640 1 --2- 172.21.15.90:0/4088845916 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558ba1bb0350 0x558ba1bb07d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:52.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.472+0000 7f85e5200640 1 -- 172.21.15.90:0/4088845916 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558ba18f07e0 con 0x558ba1bb1030 2024-04-02T14:47:52.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.472+0000 7f85e5200640 1 --2- 172.21.15.90:0/4088845916 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558ba1bb1030 0x558ba1bafd80 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f85cc0135a0 tx=0x7f85cc009850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=da99979348f9bbdb server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:47:52.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.473+0000 7f85d6a00640 1 -- 172.21.15.90:0/4088845916 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85cc004050 con 0x558ba1bb1030 2024-04-02T14:47:52.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.473+0000 7f85d6a00640 1 -- 172.21.15.90:0/4088845916 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f85cc0041f0 con 0x558ba1bb1030 2024-04-02T14:47:52.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.473+0000 7f85d6a00640 1 -- 172.21.15.90:0/4088845916 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85cc00f7c0 con 0x558ba1bb1030 2024-04-02T14:47:52.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.473+0000 7f85de000640 1 --2- 172.21.15.90:0/180568992 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f85c40a3e70 0x7f85c40a6330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:52.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.473+0000 7f85de000640 1 -- 172.21.15.90:0/180568992 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 8550+0+0 (secure 0 0 0) 0x7f85d00d08e0 con 0x7f85e011b2e0 2024-04-02T14:47:52.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.473+0000 7f85eb216740 1 -- 172.21.15.90:0/4088845916 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558ba1bb1030 msgr2=0x558ba1bafd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:52.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.473+0000 7f85eb216740 1 --2- 172.21.15.90:0/4088845916 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558ba1bb1030 0x558ba1bafd80 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f85cc0135a0 tx=0x7f85cc009850 comp rx=0 tx=0).stop 2024-04-02T14:47:52.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.474+0000 7f85eb216740 1 -- 172.21.15.90:0/4088845916 shutdown_connections 2024-04-02T14:47:52.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.474+0000 7f85eb216740 1 --2- 172.21.15.90:0/4088845916 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558ba1bb96c0 0x558ba1bbbab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:52.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.474+0000 7f85eb216740 1 --2- 172.21.15.90:0/4088845916 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558ba1bb0350 0x558ba1bb07d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:52.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.474+0000 7f85eb216740 1 --2- 172.21.15.90:0/4088845916 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558ba1bb1030 0x558ba1bafd80 unknown :-1 s=CLOSED pgs=912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:52.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.474+0000 7f85eb216740 1 -- 172.21.15.90:0/4088845916 >> 172.21.15.90:0/4088845916 conn(0x558ba1b44270 msgr2=0x558ba1baada0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:47:52.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.474+0000 7f85eb216740 1 -- 172.21.15.90:0/4088845916 shutdown_connections 2024-04-02T14:47:52.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.475+0000 7f85eb216740 1 -- 172.21.15.90:0/4088845916 wait complete. 2024-04-02T14:47:52.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.475+0000 7f85e5c00640 1 --2- 172.21.15.90:0/180568992 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f85c40a3e70 0x7f85c40a6330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:52.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.475+0000 7f85eb216740 1 -- start start 2024-04-02T14:47:52.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.476+0000 7f85eb216740 1 Processor -- start 2024-04-02T14:47:52.473 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.478+0000 7f85eb216740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558ba1bb0350 0x558ba1bb66b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:52.473 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.478+0000 7f85eb216740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558ba1bb1030 0x558ba1bb6bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:52.473 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.478+0000 7f85eb216740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558ba1bb96c0 0x558ba1bb7130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:52.473 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.478+0000 7f85e5200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558ba1bb1030 0x558ba1bb6bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:52.473 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.478+0000 7f85e5200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558ba1bb1030 0x558ba1bb6bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:34854/0 (socket says 172.21.15.90:34854) 2024-04-02T14:47:52.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.478+0000 7f85e5200640 1 -- 172.21.15.90:0/1545978765 learned_addr learned my addr 172.21.15.90:0/1545978765 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:47:52.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.479+0000 7f85eb216740 1 -- 172.21.15.90:0/1545978765 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x558ba1bbc4d0 con 0x558ba1bb0350 2024-04-02T14:47:52.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.479+0000 7f85eb216740 1 -- 172.21.15.90:0/1545978765 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x558ba1acfa60 con 0x558ba1bb96c0 2024-04-02T14:47:52.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.479+0000 7f85eb216740 1 -- 172.21.15.90:0/1545978765 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x558ba1bbc650 con 0x558ba1bb1030 2024-04-02T14:47:52.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.479+0000 7f85e6600640 1 --2- 172.21.15.90:0/1545978765 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558ba1bb96c0 0x558ba1bb7130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:52.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.479+0000 7f85eb216740 10 client.0 _ll_register_callbacks cb 0x558ba1ad5030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:47:52.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.479+0000 7f85e6600640 1 -- 172.21.15.90:0/1545978765 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558ba1bb1030 msgr2=0x558ba1bb6bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:47:52.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.479+0000 7f85e6600640 1 --2- 172.21.15.90:0/1545978765 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558ba1bb1030 0x558ba1bb6bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:52.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.479+0000 7f85e6600640 1 -- 172.21.15.90:0/1545978765 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558ba1bb0350 msgr2=0x558ba1bb66b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-02T14:47:52.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.479+0000 7f85e6600640 1 --2- 172.21.15.90:0/1545978765 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558ba1bb0350 0x558ba1bb66b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:52.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.479+0000 7f85e6600640 1 -- 172.21.15.90:0/1545978765 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558ba1950f70 con 0x558ba1bb96c0 2024-04-02T14:47:52.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.479+0000 7f85eb216740 1 -- 172.21.15.90:0/1545978765 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f85cc0041f0 con 0x558ba1bb96c0 2024-04-02T14:47:52.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.480+0000 7f85e6600640 1 --2- 172.21.15.90:0/1545978765 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558ba1bb96c0 0x558ba1bb7130 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f85d00cdfe0 tx=0x7f85d009d0a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a5b2e75a721ce6d1 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:47:52.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.480+0000 7f85cb400640 10 client.29558 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:47:52.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.480+0000 7f85cb400640 1 -- 172.21.15.90:0/1545978765 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85d001f040 con 0x558ba1bb96c0 2024-04-02T14:47:52.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.480+0000 7f85cb400640 1 -- 172.21.15.90:0/1545978765 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 8550+0+0 (secure 0 0 0) 0x7f85d00a5040 con 0x558ba1bb96c0 2024-04-02T14:47:52.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.480+0000 7f85cb400640 1 -- 172.21.15.90:0/1545978765 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f85d00a5240 con 0x558ba1bb96c0 2024-04-02T14:47:52.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.480+0000 7f85eb216740 10 client.29558 fetch_fsmap learned FSMap version 27 2024-04-02T14:47:52.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.480+0000 7f85e5c00640 1 --2- 172.21.15.90:0/180568992 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f85c40a3e70 0x7f85c40a6330 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f85d802c190 tx=0x7f85d8005de0 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:47:52.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.480+0000 7f85eb216740 1 -- 172.21.15.90:0/1545978765 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({fsmap=27}) -- 0x558ba17e8170 con 0x558ba1bb96c0 2024-04-02T14:47:52.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.481+0000 7f85cb400640 1 -- 172.21.15.90:0/1545978765 <== mon.2 v2:172.21.15.92:3300/0 4 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f85d000f040 con 0x558ba1bb96c0 2024-04-02T14:47:52.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.481+0000 7f85eb216740 10 client.29558 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:47:52.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.481+0000 7f85eb216740 10 client.29558 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:47:52.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.481+0000 7f85eb216740 20 client.29558 populate_metadata read hostname 'smithi090' 2024-04-02T14:47:52.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.481+0000 7f85eb216740 1 --2- 172.21.15.90:0/1545978765 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x558ba1bc99a0 0x558ba1bcbdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:52.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.481+0000 7f85eb216740 4 client.29558 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:47:52.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.481+0000 7f85eb216740 1 -- 172.21.15.90:0/1545978765 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558ba1b21850 con 0x558ba1bc99a0 2024-04-02T14:47:52.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.481+0000 7f85e5c00640 1 --2- 172.21.15.90:0/1545978765 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x558ba1bc99a0 0x558ba1bcbdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:52.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.482+0000 7f85cb400640 10 client.29558 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:47:52.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.482+0000 7f85e5c00640 1 --2- 172.21.15.90:0/1545978765 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x558ba1bc99a0 0x558ba1bcbdf0 crc :-1 s=READY pgs=336 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=574e6341d974d73b server_cookie=5dad79c4597626fd in_seq=0 out_seq=0 2024-04-02T14:47:52.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.483+0000 7f85cb400640 1 -- 172.21.15.90:0/1545978765 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x558ba1b21850 con 0x558ba1bc99a0 2024-04-02T14:47:52.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.483+0000 7f85cb400640 10 client.29558 handle_command_reply: tid=0 2024-04-02T14:47:52.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.483+0000 7f85eb216740 1 client.29558 shutdown 2024-04-02T14:47:52.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.483+0000 7f85eb216740 1 -- 172.21.15.90:0/1545978765 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558ba1bb96c0 msgr2=0x558ba1bb7130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:52.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.483+0000 7f85eb216740 1 --2- 172.21.15.90:0/1545978765 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558ba1bb96c0 0x558ba1bb7130 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f85d00cdfe0 tx=0x7f85d009d0a0 comp rx=0 tx=0).stop 2024-04-02T14:47:52.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.483+0000 7f85eb216740 1 -- 172.21.15.90:0/1545978765 shutdown_connections 2024-04-02T14:47:52.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.483+0000 7f85eb216740 1 --2- 172.21.15.90:0/1545978765 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x558ba1bc99a0 0x558ba1bcbdf0 crc :-1 s=READY pgs=336 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:52.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.483+0000 7f85eb216740 1 --2- 172.21.15.90:0/1545978765 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558ba1bb96c0 0x558ba1bb7130 unknown :-1 s=CLOSED pgs=913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:52.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.483+0000 7f85eb216740 1 --2- 172.21.15.90:0/1545978765 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558ba1bb1030 0x558ba1bb6bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:52.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.483+0000 7f85eb216740 1 --2- 172.21.15.90:0/1545978765 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558ba1bb0350 0x558ba1bb66b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:52.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.483+0000 7f85cb400640 0 client.29558 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:47:52.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.483+0000 7f85eb216740 1 -- 172.21.15.90:0/1545978765 >> 172.21.15.90:0/1545978765 conn(0x558ba1b44270 msgr2=0x558ba1bacaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:47:52.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.484+0000 7f85eb216740 1 -- 172.21.15.90:0/1545978765 shutdown_connections 2024-04-02T14:47:52.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.485+0000 7f85eb216740 1 -- 172.21.15.90:0/1545978765 wait complete. 2024-04-02T14:47:52.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.486+0000 7f85eb216740 20 client.29558 trim_cache size 0 max 16384 2024-04-02T14:47:52.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.521+0000 7f85eb216740 1 Processor -- start 2024-04-02T14:47:52.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.521+0000 7f85eb216740 1 -- start start 2024-04-02T14:47:52.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.522+0000 7f85eb216740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558ba1bb0350 0x558ba1bc53d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:52.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.522+0000 7f85eb216740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558ba1bb1030 0x558ba1bac900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:52.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.522+0000 7f85eb216740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558ba1bb96c0 0x558ba1bd16c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:52.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.522+0000 7f85eb216740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f85e01bc720 con 0x558ba1bb96c0 2024-04-02T14:47:52.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.522+0000 7f85eb216740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f85e01be260 con 0x558ba1bb1030 2024-04-02T14:47:52.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.522+0000 7f85eb216740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f85e01bcdf0 con 0x558ba1bb0350 2024-04-02T14:47:52.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.522+0000 7f85e5c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558ba1bb0350 0x558ba1bc53d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:52.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.522+0000 7f85e5200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558ba1bb1030 0x558ba1bac900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:52.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.522+0000 7f85e5c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558ba1bb0350 0x558ba1bc53d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:34870/0 (socket says 172.21.15.90:34870) 2024-04-02T14:47:52.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.522+0000 7f85e5c00640 1 -- 172.21.15.90:0/3270286923 learned_addr learned my addr 172.21.15.90:0/3270286923 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:47:52.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.522+0000 7f85e6600640 1 --2- 172.21.15.90:0/3270286923 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558ba1bb96c0 0x558ba1bd16c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:52.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.522+0000 7f85e5c00640 1 -- 172.21.15.90:0/3270286923 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558ba1bb1030 msgr2=0x558ba1bac900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:47:52.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.522+0000 7f85e5c00640 1 --2- 172.21.15.90:0/3270286923 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558ba1bb1030 0x558ba1bac900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:52.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.523+0000 7f85e5c00640 1 -- 172.21.15.90:0/3270286923 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558ba1bb96c0 msgr2=0x558ba1bd16c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:47:52.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.523+0000 7f85e5c00640 1 --2- 172.21.15.90:0/3270286923 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558ba1bb96c0 0x558ba1bd16c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:52.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.523+0000 7f85e5c00640 1 -- 172.21.15.90:0/3270286923 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558ba1baf380 con 0x558ba1bb0350 2024-04-02T14:47:52.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.523+0000 7f85e6600640 1 --2- 172.21.15.90:0/3270286923 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558ba1bb96c0 0x558ba1bd16c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:47:52.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.523+0000 7f85e5c00640 1 --2- 172.21.15.90:0/3270286923 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558ba1bb0350 0x558ba1bc53d0 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f85d802c260 tx=0x7f85d803e000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=14bab1b986ecb6e server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:47:52.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.524+0000 7f85d6000640 1 -- 172.21.15.90:0/3270286923 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85d8056070 con 0x558ba1bb0350 2024-04-02T14:47:52.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.524+0000 7f85d6000640 1 -- 172.21.15.90:0/3270286923 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f85d8044030 con 0x558ba1bb0350 2024-04-02T14:47:52.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.524+0000 7f85d6000640 1 -- 172.21.15.90:0/3270286923 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85d802c850 con 0x558ba1bb0350 2024-04-02T14:47:52.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.526+0000 7f85eb216740 1 -- 172.21.15.90:0/3270286923 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558ba1bb0350 msgr2=0x558ba1bc53d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:52.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.526+0000 7f85eb216740 1 --2- 172.21.15.90:0/3270286923 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558ba1bb0350 0x558ba1bc53d0 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f85d802c260 tx=0x7f85d803e000 comp rx=0 tx=0).stop 2024-04-02T14:47:52.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.526+0000 7f85eb216740 1 -- 172.21.15.90:0/3270286923 shutdown_connections 2024-04-02T14:47:52.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.526+0000 7f85eb216740 1 --2- 172.21.15.90:0/3270286923 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558ba1bb96c0 0x558ba1bd16c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:52.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.526+0000 7f85eb216740 1 --2- 172.21.15.90:0/3270286923 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558ba1bb1030 0x558ba1bac900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:52.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.526+0000 7f85eb216740 1 --2- 172.21.15.90:0/3270286923 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558ba1bb0350 0x558ba1bc53d0 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:52.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.526+0000 7f85eb216740 1 -- 172.21.15.90:0/3270286923 >> 172.21.15.90:0/3270286923 conn(0x558ba1b44270 msgr2=0x558ba1bac1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:47:52.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.527+0000 7f85eb216740 1 -- 172.21.15.90:0/3270286923 shutdown_connections 2024-04-02T14:47:52.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.527+0000 7f85eb216740 1 -- 172.21.15.90:0/3270286923 wait complete. 2024-04-02T14:47:52.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.527+0000 7f85eb216740 1 -- start start 2024-04-02T14:47:52.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.528+0000 7f85eb216740 1 Processor -- start 2024-04-02T14:47:52.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.528+0000 7f85eb216740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558ba1bb0350 0x558ba1bb3d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:52.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.528+0000 7f85eb216740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558ba1bb1030 0x558ba1bb42d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:52.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.528+0000 7f85eb216740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558ba1bb96c0 0x558ba1bb29a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:52.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.528+0000 7f85eb216740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f85e01bdb90 con 0x558ba1bb96c0 2024-04-02T14:47:52.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.528+0000 7f85eb216740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f85e01bd4c0 con 0x558ba1bb1030 2024-04-02T14:47:52.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.528+0000 7f85eb216740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f85e01ba720 con 0x558ba1bb0350 2024-04-02T14:47:52.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.528+0000 7f85eb216740 10 client.0 _ll_register_callbacks cb 0x7f85d0098b00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:47:52.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.529+0000 7f85e5c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558ba1bb0350 0x558ba1bb3d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:52.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.529+0000 7f85e5c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558ba1bb0350 0x558ba1bb3d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:34878/0 (socket says 172.21.15.90:34878) 2024-04-02T14:47:52.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.529+0000 7f85e5c00640 1 -- 172.21.15.90:0/2402275106 learned_addr learned my addr 172.21.15.90:0/2402275106 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:47:52.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.529+0000 7f85e6600640 1 --2- 172.21.15.90:0/2402275106 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558ba1bb96c0 0x558ba1bb29a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:52.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.529+0000 7f85e5200640 1 --2- 172.21.15.90:0/2402275106 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558ba1bb1030 0x558ba1bb42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:52.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.529+0000 7f85e5c00640 1 -- 172.21.15.90:0/2402275106 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558ba1bb1030 msgr2=0x558ba1bb42d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:47:52.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.529+0000 7f85e5c00640 1 --2- 172.21.15.90:0/2402275106 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558ba1bb1030 0x558ba1bb42d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:52.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.529+0000 7f85e5c00640 1 -- 172.21.15.90:0/2402275106 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558ba1bb96c0 msgr2=0x558ba1bb29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:47:52.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.529+0000 7f85e5c00640 1 --2- 172.21.15.90:0/2402275106 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558ba1bb96c0 0x558ba1bb29a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:52.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.529+0000 7f85e5c00640 1 -- 172.21.15.90:0/2402275106 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558ba1bc2190 con 0x558ba1bb0350 2024-04-02T14:47:52.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.529+0000 7f85e6600640 1 --2- 172.21.15.90:0/2402275106 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558ba1bb96c0 0x558ba1bb29a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:47:52.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.529+0000 7f85e5c00640 1 --2- 172.21.15.90:0/2402275106 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558ba1bb0350 0x558ba1bb3d90 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f85d8009850 tx=0x7f85d8004b90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c5c4250a89aa6fe2 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:47:52.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.529+0000 7f85cb400640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:47:52.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.530+0000 7f85cb400640 1 -- 172.21.15.90:0/2402275106 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85d8056070 con 0x558ba1bb0350 2024-04-02T14:47:52.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.530+0000 7f85eb216740 1 -- 172.21.15.90:0/2402275106 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f85d8044030 con 0x558ba1bb0350 2024-04-02T14:47:52.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.530+0000 7f85cb400640 1 -- 172.21.15.90:0/2402275106 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 8550+0+0 (secure 0 0 0) 0x7f85d8003ac0 con 0x558ba1bb0350 2024-04-02T14:47:52.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.531+0000 7f85cb400640 1 -- 172.21.15.90:0/2402275106 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f85d8003cc0 con 0x558ba1bb0350 2024-04-02T14:47:52.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.531+0000 7f85eb216740 10 client.29347 fetch_fsmap learned FSMap version 27 2024-04-02T14:47:52.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.531+0000 7f85eb216740 1 -- 172.21.15.90:0/2402275106 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x558ba1c314d0 con 0x558ba1bb0350 2024-04-02T14:47:52.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.531+0000 7f85cb400640 1 -- 172.21.15.90:0/2402275106 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f85d8044030 con 0x558ba1bb0350 2024-04-02T14:47:52.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.532+0000 7f85cb400640 1 -- 172.21.15.90:0/2402275106 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f85d802cae0 con 0x558ba1bb0350 2024-04-02T14:47:52.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.532+0000 7f85eb216740 10 client.29347 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:47:52.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.532+0000 7f85eb216740 10 client.29347 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:47:52.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.532+0000 7f85eb216740 20 client.29347 populate_metadata read hostname 'smithi090' 2024-04-02T14:47:52.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.532+0000 7f85eb216740 1 --2- 172.21.15.90:0/2402275106 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x558ba1c31ae0 0x558ba1c33ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:52.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.533+0000 7f85eb216740 4 client.29347 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-02T14:47:52.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.533+0000 7f85eb216740 1 -- 172.21.15.90:0/2402275106 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x558ba1b1bba0 con 0x558ba1c31ae0 2024-04-02T14:47:52.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.533+0000 7f85e5200640 1 --2- 172.21.15.90:0/2402275106 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x558ba1c31ae0 0x558ba1c33ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:52.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.533+0000 7f85cb400640 10 client.29347 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:47:52.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.533+0000 7f85e5200640 1 --2- 172.21.15.90:0/2402275106 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x558ba1c31ae0 0x558ba1c33ed0 crc :-1 s=READY pgs=337 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5b445366586eb1c server_cookie=50338d801050bfdd in_seq=0 out_seq=0 2024-04-02T14:47:52.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.535+0000 7f85cb400640 1 -- 172.21.15.90:0/2402275106 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x558ba1b1bba0 con 0x558ba1c31ae0 2024-04-02T14:47:52.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.535+0000 7f85cb400640 10 client.29347 handle_command_reply: tid=0 2024-04-02T14:47:52.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.535+0000 7f85eb216740 1 client.29347 shutdown 2024-04-02T14:47:52.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.535+0000 7f85eb216740 1 -- 172.21.15.90:0/2402275106 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558ba1bb0350 msgr2=0x558ba1bb3d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:52.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.535+0000 7f85eb216740 1 --2- 172.21.15.90:0/2402275106 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558ba1bb0350 0x558ba1bb3d90 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f85d8009850 tx=0x7f85d8004b90 comp rx=0 tx=0).stop 2024-04-02T14:47:52.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.535+0000 7f85eb216740 1 -- 172.21.15.90:0/2402275106 shutdown_connections 2024-04-02T14:47:52.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.535+0000 7f85eb216740 1 --2- 172.21.15.90:0/2402275106 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x558ba1c31ae0 0x558ba1c33ed0 crc :-1 s=READY pgs=337 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:52.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.535+0000 7f85eb216740 1 --2- 172.21.15.90:0/2402275106 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558ba1bb96c0 0x558ba1bb29a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:52.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.535+0000 7f85eb216740 1 --2- 172.21.15.90:0/2402275106 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x558ba1bb1030 0x558ba1bb42d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:52.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.535+0000 7f85cb400640 0 client.29347 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:47:52.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.535+0000 7f85eb216740 1 --2- 172.21.15.90:0/2402275106 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558ba1bb0350 0x558ba1bb3d90 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:52.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.535+0000 7f85eb216740 1 -- 172.21.15.90:0/2402275106 >> 172.21.15.90:0/2402275106 conn(0x558ba1b44270 msgr2=0x558ba1bb6f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:47:52.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.536+0000 7f85eb216740 1 -- 172.21.15.90:0/2402275106 shutdown_connections 2024-04-02T14:47:52.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.537+0000 7f85eb216740 1 -- 172.21.15.90:0/2402275106 wait complete. 2024-04-02T14:47:52.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.537+0000 7f85eb216740 20 client.29347 trim_cache size 0 max 16384 2024-04-02T14:47:52.532 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-02T14:47:52.532 INFO:teuthology.orchestra.run.smithi090.stdout: "return_code": 0, 2024-04-02T14:47:52.532 INFO:teuthology.orchestra.run.smithi090.stdout: "scrub_tag": "7ef8edb4-83cf-4332-a60d-24f59e2bbde4", 2024-04-02T14:47:52.532 INFO:teuthology.orchestra.run.smithi090.stdout: "mode": "asynchronous" 2024-04-02T14:47:52.532 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-02T14:47:52.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.538+0000 7f85e8600640 1 -- 172.21.15.90:0/180568992 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f85c40a3e70 msgr2=0x7f85c40a6330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:52.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.538+0000 7f85e8600640 1 --2- 172.21.15.90:0/180568992 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f85c40a3e70 0x7f85c40a6330 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f85d802c190 tx=0x7f85d8005de0 comp rx=0 tx=0).stop 2024-04-02T14:47:52.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.538+0000 7f85e8600640 1 -- 172.21.15.90:0/180568992 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f85e011b2e0 msgr2=0x7f85e01aba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:52.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.538+0000 7f85e8600640 1 --2- 172.21.15.90:0/180568992 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f85e011b2e0 0x7f85e01aba70 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7f85d000eaa0 tx=0x7f85d000ef60 comp rx=0 tx=0).stop 2024-04-02T14:47:52.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.538+0000 7f85e8600640 1 -- 172.21.15.90:0/180568992 shutdown_connections 2024-04-02T14:47:52.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.538+0000 7f85e8600640 1 --2- 172.21.15.90:0/180568992 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f85c40a3e70 0x7f85c40a6330 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:52.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.538+0000 7f85e8600640 1 --2- 172.21.15.90:0/180568992 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f85e011b2e0 0x7f85e01aba70 unknown :-1 s=CLOSED pgs=911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:52.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.538+0000 7f85e8600640 1 --2- 172.21.15.90:0/180568992 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f85e0110320 0x7f85e01a76f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:52.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.538+0000 7f85e8600640 1 --2- 172.21.15.90:0/180568992 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f85e010e350 0x7f85e01a71b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:52.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.538+0000 7f85e8600640 1 -- 172.21.15.90:0/180568992 >> 172.21.15.90:0/180568992 conn(0x7f85e006cbe0 msgr2=0x7f85e00725d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:47:52.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.538+0000 7f85e8600640 1 -- 172.21.15.90:0/180568992 shutdown_connections 2024-04-02T14:47:52.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.538+0000 7f85e8600640 1 -- 172.21.15.90:0/180568992 wait complete. 2024-04-02T14:47:52.553 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:47:52 smithi090 ceph-mon[31674]: pgmap v404: 97 pgs: 97 active+clean; 4.5 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 35 MiB/s wr, 1.82k op/s 2024-04-02T14:47:52.556 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-02T14:47:52.557 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-02T14:47:52.603 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:47:52 smithi092 ceph-mon[36497]: pgmap v404: 97 pgs: 97 active+clean; 4.5 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 35 MiB/s wr, 1.82k op/s 2024-04-02T14:47:52.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:47:52 smithi177 ceph-mon[36431]: pgmap v404: 97 pgs: 97 active+clean; 4.5 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 35 MiB/s wr, 1.82k op/s 2024-04-02T14:47:52.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.895+0000 7fc5cea00640 1 -- 172.21.15.90:0/2806090196 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc5c810e2c0 msgr2=0x7fc5c8110710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:52.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.895+0000 7fc5cea00640 1 --2- 172.21.15.90:0/2806090196 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc5c810e2c0 0x7fc5c8110710 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7fc5c0009ca0 tx=0x7fc5c002c190 comp rx=0 tx=0).stop 2024-04-02T14:47:52.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.896+0000 7fc5cea00640 1 -- 172.21.15.90:0/2806090196 shutdown_connections 2024-04-02T14:47:52.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.896+0000 7fc5cea00640 1 --2- 172.21.15.90:0/2806090196 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc5c810e2c0 0x7fc5c8110710 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:52.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.896+0000 7fc5cea00640 1 --2- 172.21.15.90:0/2806090196 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc5c8074850 0x7fc5c810dd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:52.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.896+0000 7fc5cea00640 1 --2- 172.21.15.90:0/2806090196 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc5c8073f30 0x7fc5c8074310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:52.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.896+0000 7fc5cea00640 1 -- 172.21.15.90:0/2806090196 >> 172.21.15.90:0/2806090196 conn(0x7fc5c80fd590 msgr2=0x7fc5c80ff9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:47:52.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.896+0000 7fc5cea00640 1 -- 172.21.15.90:0/2806090196 shutdown_connections 2024-04-02T14:47:52.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.896+0000 7fc5cea00640 1 -- 172.21.15.90:0/2806090196 wait complete. 2024-04-02T14:47:52.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.897+0000 7fc5cea00640 1 Processor -- start 2024-04-02T14:47:52.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.897+0000 7fc5cea00640 1 -- start start 2024-04-02T14:47:52.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.898+0000 7fc5cea00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc5c8073f30 0x7fc5c819a560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:52.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.898+0000 7fc5cea00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc5c8074850 0x7fc5c819aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:52.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.898+0000 7fc5cea00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc5c810e2c0 0x7fc5c819ee20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:52.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.898+0000 7fc5cea00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fc5c8114060 con 0x7fc5c8073f30 2024-04-02T14:47:52.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.898+0000 7fc5cea00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fc5c8113ee0 con 0x7fc5c810e2c0 2024-04-02T14:47:52.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.898+0000 7fc5cea00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fc5c81141e0 con 0x7fc5c8074850 2024-04-02T14:47:52.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.898+0000 7fc5c7e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc5c8073f30 0x7fc5c819a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:52.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.898+0000 7fc5cca00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc5c810e2c0 0x7fc5c819ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:52.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.898+0000 7fc5c7e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc5c8073f30 0x7fc5c819a560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34196/0 (socket says 172.21.15.90:34196) 2024-04-02T14:47:52.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.898+0000 7fc5c7e00640 1 -- 172.21.15.90:0/2263427907 learned_addr learned my addr 172.21.15.90:0/2263427907 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:47:52.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.898+0000 7fc5c7400640 1 --2- 172.21.15.90:0/2263427907 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc5c8074850 0x7fc5c819aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:52.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.899+0000 7fc5cca00640 1 -- 172.21.15.90:0/2263427907 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc5c8074850 msgr2=0x7fc5c819aaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:52.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.899+0000 7fc5cca00640 1 --2- 172.21.15.90:0/2263427907 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc5c8074850 0x7fc5c819aaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:52.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.899+0000 7fc5cca00640 1 -- 172.21.15.90:0/2263427907 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc5c8073f30 msgr2=0x7fc5c819a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:52.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.899+0000 7fc5cca00640 1 --2- 172.21.15.90:0/2263427907 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc5c8073f30 0x7fc5c819a560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:52.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.899+0000 7fc5cca00640 1 -- 172.21.15.90:0/2263427907 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5c819f550 con 0x7fc5c810e2c0 2024-04-02T14:47:52.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.899+0000 7fc5c7400640 1 --2- 172.21.15.90:0/2263427907 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc5c8074850 0x7fc5c819aaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:47:52.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.899+0000 7fc5c7e00640 1 --2- 172.21.15.90:0/2263427907 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc5c8073f30 0x7fc5c819a560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:47:52.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.899+0000 7fc5cca00640 1 --2- 172.21.15.90:0/2263427907 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc5c810e2c0 0x7fc5c819ee20 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7fc5c0009bf0 tx=0x7fc5c0035d80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:47:52.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.900+0000 7fc5c4c00640 1 -- 172.21.15.90:0/2263427907 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5c0039040 con 0x7fc5c810e2c0 2024-04-02T14:47:52.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.900+0000 7fc5cea00640 1 -- 172.21.15.90:0/2263427907 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5c819f7e0 con 0x7fc5c810e2c0 2024-04-02T14:47:52.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.900+0000 7fc5cea00640 1 -- 172.21.15.90:0/2263427907 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5c81a70d0 con 0x7fc5c810e2c0 2024-04-02T14:47:52.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.900+0000 7fc5c4c00640 1 -- 172.21.15.90:0/2263427907 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc5c0044070 con 0x7fc5c810e2c0 2024-04-02T14:47:52.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.900+0000 7fc5c4c00640 1 -- 172.21.15.90:0/2263427907 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5c00328d0 con 0x7fc5c810e2c0 2024-04-02T14:47:52.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.904+0000 7fc5d1793740 1 Processor -- start 2024-04-02T14:47:52.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.905+0000 7fc5c4c00640 1 -- 172.21.15.90:0/2263427907 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fc5c0032a70 con 0x7fc5c810e2c0 2024-04-02T14:47:52.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.905+0000 7fc5d1793740 1 -- start start 2024-04-02T14:47:52.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.906+0000 7fc5d1793740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b6a9a297b0 0x55b6a9a29b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:52.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.907+0000 7fc5d1793740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b6a9a2a0d0 0x55b6a9a32770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:52.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.907+0000 7fc5d1793740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b6a9a32cb0 0x55b6a9a350a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:52.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.907+0000 7fc5d1793740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55b6a998d770 con 0x55b6a9a2a0d0 2024-04-02T14:47:52.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.907+0000 7fc5d1793740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55b6a9935700 con 0x55b6a9a297b0 2024-04-02T14:47:52.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.907+0000 7fc5d1793740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55b6a9989720 con 0x55b6a9a32cb0 2024-04-02T14:47:52.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.907+0000 7fc5c7400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b6a9a297b0 0x55b6a9a29b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:52.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.907+0000 7fc5c7400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b6a9a297b0 0x55b6a9a29b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:50424/0 (socket says 172.21.15.90:50424) 2024-04-02T14:47:52.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.907+0000 7fc5c7400640 1 -- 172.21.15.90:0/295186568 learned_addr learned my addr 172.21.15.90:0/295186568 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:47:52.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.907+0000 7fc5c7e00640 1 --2- 172.21.15.90:0/295186568 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b6a9a2a0d0 0x55b6a9a32770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:52.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.907+0000 7fc5c7400640 1 --2- 172.21.15.90:0/295186568 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b6a9a32cb0 0x55b6a9a350a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:52.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.907+0000 7fc5c7400640 1 -- 172.21.15.90:0/295186568 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b6a9a32cb0 msgr2=0x55b6a9a350a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:47:52.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.907+0000 7fc5c7400640 1 --2- 172.21.15.90:0/295186568 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b6a9a32cb0 0x55b6a9a350a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:52.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.907+0000 7fc5c7400640 1 -- 172.21.15.90:0/295186568 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b6a9a2a0d0 msgr2=0x55b6a9a32770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:47:52.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.907+0000 7fc5c7400640 1 --2- 172.21.15.90:0/295186568 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b6a9a2a0d0 0x55b6a9a32770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:52.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.908+0000 7fc5c7400640 1 -- 172.21.15.90:0/295186568 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b6a976b7d0 con 0x55b6a9a297b0 2024-04-02T14:47:52.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.908+0000 7fc5c7400640 1 --2- 172.21.15.90:0/295186568 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b6a9a297b0 0x55b6a9a29b90 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7fc5b4013910 tx=0x7fc5b4005850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=98a88594dd144434 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:47:52.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.909+0000 7fc5bd200640 1 -- 172.21.15.90:0/295186568 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5b4004050 con 0x55b6a9a297b0 2024-04-02T14:47:52.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.909+0000 7fc5bd200640 1 -- 172.21.15.90:0/295186568 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc5b400fe20 con 0x55b6a9a297b0 2024-04-02T14:47:52.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.909+0000 7fc5bd200640 1 -- 172.21.15.90:0/295186568 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5b4003b20 con 0x55b6a9a297b0 2024-04-02T14:47:52.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.909+0000 7fc5c4c00640 1 --2- 172.21.15.90:0/2263427907 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fc5ac0a3da0 0x7fc5ac0a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:52.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.909+0000 7fc5c4c00640 1 -- 172.21.15.90:0/2263427907 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 8550+0+0 (secure 0 0 0) 0x7fc5c00f2ac0 con 0x7fc5c810e2c0 2024-04-02T14:47:52.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.909+0000 7fc5c7e00640 1 --2- 172.21.15.90:0/2263427907 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fc5ac0a3da0 0x7fc5ac0a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:52.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.910+0000 7fc5d1793740 1 -- 172.21.15.90:0/295186568 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b6a9a297b0 msgr2=0x55b6a9a29b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:52.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.910+0000 7fc5d1793740 1 --2- 172.21.15.90:0/295186568 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b6a9a297b0 0x55b6a9a29b90 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7fc5b4013910 tx=0x7fc5b4005850 comp rx=0 tx=0).stop 2024-04-02T14:47:52.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.910+0000 7fc5d1793740 1 -- 172.21.15.90:0/295186568 shutdown_connections 2024-04-02T14:47:52.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.910+0000 7fc5d1793740 1 --2- 172.21.15.90:0/295186568 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b6a9a32cb0 0x55b6a9a350a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:52.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.910+0000 7fc5d1793740 1 --2- 172.21.15.90:0/295186568 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b6a9a2a0d0 0x55b6a9a32770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:52.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.910+0000 7fc5d1793740 1 --2- 172.21.15.90:0/295186568 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b6a9a297b0 0x55b6a9a29b90 unknown :-1 s=CLOSED pgs=915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:52.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.910+0000 7fc5d1793740 1 -- 172.21.15.90:0/295186568 >> 172.21.15.90:0/295186568 conn(0x55b6a9a23ee0 msgr2=0x55b6a9a242c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:47:52.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.910+0000 7fc5d1793740 1 -- 172.21.15.90:0/295186568 shutdown_connections 2024-04-02T14:47:52.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.910+0000 7fc5d1793740 1 -- 172.21.15.90:0/295186568 wait complete. 2024-04-02T14:47:52.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.910+0000 7fc5c7e00640 1 --2- 172.21.15.90:0/2263427907 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fc5ac0a3da0 0x7fc5ac0a6260 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x55b6a9a29280 tx=0x7fc5b8005f90 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:47:52.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.910+0000 7fc5d1793740 1 -- start start 2024-04-02T14:47:52.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.911+0000 7fc5d1793740 1 Processor -- start 2024-04-02T14:47:52.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.911+0000 7fc5d1793740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b6a9a297b0 0x55b6a9a2fda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:52.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.911+0000 7fc5d1793740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b6a9a2a0d0 0x55b6a9a302e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:52.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.911+0000 7fc5d1793740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b6a9a32cb0 0x55b6a9a30820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:52.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.911+0000 7fc5d1793740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55b6a9a35aa0 con 0x55b6a9a2a0d0 2024-04-02T14:47:52.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.911+0000 7fc5d1793740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55b6a9948ac0 con 0x55b6a9a32cb0 2024-04-02T14:47:52.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.911+0000 7fc5d1793740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55b6a9a35c20 con 0x55b6a9a297b0 2024-04-02T14:47:52.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.911+0000 7fc5d1793740 10 client.0 _ll_register_callbacks cb 0x55b6a994b180 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:47:52.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.912+0000 7fc5c7e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b6a9a297b0 0x55b6a9a2fda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:52.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.912+0000 7fc5cca00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b6a9a32cb0 0x55b6a9a30820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:52.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.912+0000 7fc5c7400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b6a9a2a0d0 0x55b6a9a302e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:52.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.912+0000 7fc5c7e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b6a9a297b0 0x55b6a9a2fda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:34918/0 (socket says 172.21.15.90:34918) 2024-04-02T14:47:52.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.912+0000 7fc5c7e00640 1 -- 172.21.15.90:0/2288532021 learned_addr learned my addr 172.21.15.90:0/2288532021 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:47:52.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.912+0000 7fc5cca00640 1 -- 172.21.15.90:0/2288532021 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b6a9a297b0 msgr2=0x55b6a9a2fda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:47:52.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.912+0000 7fc5cca00640 1 --2- 172.21.15.90:0/2288532021 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b6a9a297b0 0x55b6a9a2fda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:52.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.912+0000 7fc5cca00640 1 -- 172.21.15.90:0/2288532021 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b6a9a2a0d0 msgr2=0x55b6a9a302e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:47:52.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.912+0000 7fc5cca00640 1 --2- 172.21.15.90:0/2288532021 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b6a9a2a0d0 0x55b6a9a302e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:52.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.912+0000 7fc5cca00640 1 -- 172.21.15.90:0/2288532021 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b6a96a13f0 con 0x55b6a9a32cb0 2024-04-02T14:47:52.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.913+0000 7fc5c7e00640 1 --2- 172.21.15.90:0/2288532021 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b6a9a297b0 0x55b6a9a2fda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:47:52.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.913+0000 7fc5cca00640 1 --2- 172.21.15.90:0/2288532021 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b6a9a32cb0 0x55b6a9a30820 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7fc5c00f0300 tx=0x7fc5c00be710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=abb37581f7043ae4 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:47:52.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.913+0000 7fc5b1600640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:47:52.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.913+0000 7fc5b1600640 1 -- 172.21.15.90:0/2288532021 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5c00beb50 con 0x55b6a9a32cb0 2024-04-02T14:47:52.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.913+0000 7fc5b1600640 1 -- 172.21.15.90:0/2288532021 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 8550+0+0 (secure 0 0 0) 0x7fc5c0032d60 con 0x55b6a9a32cb0 2024-04-02T14:47:52.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.913+0000 7fc5d1793740 1 -- 172.21.15.90:0/2288532021 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc5b400fe20 con 0x55b6a9a32cb0 2024-04-02T14:47:52.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.914+0000 7fc5b1600640 1 -- 172.21.15.90:0/2288532021 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fc5c00c2f00 con 0x55b6a9a32cb0 2024-04-02T14:47:52.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.914+0000 7fc5d1793740 10 client.29573 fetch_fsmap learned FSMap version 27 2024-04-02T14:47:52.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.914+0000 7fc5d1793740 1 -- 172.21.15.90:0/2288532021 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55b6a96610f0 con 0x55b6a9a32cb0 2024-04-02T14:47:52.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.914+0000 7fc5b1600640 1 -- 172.21.15.90:0/2288532021 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc5b400fe20 con 0x55b6a9a32cb0 2024-04-02T14:47:52.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.915+0000 7fc5b1600640 1 -- 172.21.15.90:0/2288532021 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fc5c003c030 con 0x55b6a9a32cb0 2024-04-02T14:47:52.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.915+0000 7fc5d1793740 10 client.29573 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:47:52.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.915+0000 7fc5d1793740 10 client.29573 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:47:52.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.915+0000 7fc5d1793740 20 client.29573 populate_metadata read hostname 'smithi090' 2024-04-02T14:47:52.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.915+0000 7fc5d1793740 1 --2- 172.21.15.90:0/2288532021 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55b6a9a42d30 0x55b6a9a45190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:52.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.915+0000 7fc5d1793740 4 client.29573 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:47:52.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.915+0000 7fc5d1793740 1 -- 172.21.15.90:0/2288532021 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b6a992e630 con 0x55b6a9a42d30 2024-04-02T14:47:52.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.916+0000 7fc5c7e00640 1 --2- 172.21.15.90:0/2288532021 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55b6a9a42d30 0x55b6a9a45190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:52.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.916+0000 7fc5c7e00640 1 --2- 172.21.15.90:0/2288532021 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55b6a9a42d30 0x55b6a9a45190 crc :-1 s=READY pgs=338 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=71b7a0fd5677feab server_cookie=834e0b51645c0aa4 in_seq=0 out_seq=0 2024-04-02T14:47:52.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.916+0000 7fc5b1600640 10 client.29573 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:47:52.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.917+0000 7fc5b1600640 1 -- 172.21.15.90:0/2288532021 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b6a992e630 con 0x55b6a9a42d30 2024-04-02T14:47:52.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.917+0000 7fc5b1600640 10 client.29573 handle_command_reply: tid=0 2024-04-02T14:47:52.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.917+0000 7fc5d1793740 1 client.29573 shutdown 2024-04-02T14:47:52.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.917+0000 7fc5d1793740 1 -- 172.21.15.90:0/2288532021 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b6a9a32cb0 msgr2=0x55b6a9a30820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:52.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.917+0000 7fc5d1793740 1 --2- 172.21.15.90:0/2288532021 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b6a9a32cb0 0x55b6a9a30820 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7fc5c00f0300 tx=0x7fc5c00be710 comp rx=0 tx=0).stop 2024-04-02T14:47:52.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.917+0000 7fc5d1793740 1 -- 172.21.15.90:0/2288532021 shutdown_connections 2024-04-02T14:47:52.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.917+0000 7fc5d1793740 1 --2- 172.21.15.90:0/2288532021 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55b6a9a42d30 0x55b6a9a45190 crc :-1 s=READY pgs=338 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:52.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.917+0000 7fc5b1600640 0 client.29573 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:47:52.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.917+0000 7fc5d1793740 1 --2- 172.21.15.90:0/2288532021 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b6a9a32cb0 0x55b6a9a30820 unknown :-1 s=CLOSED pgs=916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:52.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.917+0000 7fc5d1793740 1 --2- 172.21.15.90:0/2288532021 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b6a9a2a0d0 0x55b6a9a302e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:52.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.917+0000 7fc5d1793740 1 --2- 172.21.15.90:0/2288532021 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b6a9a297b0 0x55b6a9a2fda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:52.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.917+0000 7fc5d1793740 1 -- 172.21.15.90:0/2288532021 >> 172.21.15.90:0/2288532021 conn(0x55b6a9a23ee0 msgr2=0x55b6a9a25c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:47:52.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.918+0000 7fc5d1793740 1 -- 172.21.15.90:0/2288532021 shutdown_connections 2024-04-02T14:47:52.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.918+0000 7fc5d1793740 1 -- 172.21.15.90:0/2288532021 wait complete. 2024-04-02T14:47:52.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.919+0000 7fc5d1793740 20 client.29573 trim_cache size 0 max 16384 2024-04-02T14:47:52.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.945+0000 7fc5d1793740 1 Processor -- start 2024-04-02T14:47:52.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.945+0000 7fc5d1793740 1 -- start start 2024-04-02T14:47:52.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.946+0000 7fc5d1793740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b6a9a297b0 0x55b6a9a2ece0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:52.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.946+0000 7fc5d1793740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b6a9a2a0d0 0x55b6a9a3e990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:52.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.946+0000 7fc5d1793740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b6a9a32cb0 0x55b6a9a30640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:52.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.946+0000 7fc5d1793740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fc5c81af9b0 con 0x55b6a9a2a0d0 2024-04-02T14:47:52.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.946+0000 7fc5d1793740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fc5c81b14f0 con 0x55b6a9a297b0 2024-04-02T14:47:52.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.946+0000 7fc5d1793740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fc5c81b0080 con 0x55b6a9a32cb0 2024-04-02T14:47:52.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.946+0000 7fc5cca00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b6a9a32cb0 0x55b6a9a30640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:52.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.947+0000 7fc5cca00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b6a9a32cb0 0x55b6a9a30640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:34924/0 (socket says 172.21.15.90:34924) 2024-04-02T14:47:52.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.947+0000 7fc5cca00640 1 -- 172.21.15.90:0/1128078160 learned_addr learned my addr 172.21.15.90:0/1128078160 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:47:52.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.947+0000 7fc5c7e00640 1 --2- 172.21.15.90:0/1128078160 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b6a9a297b0 0x55b6a9a2ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:52.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.947+0000 7fc5cca00640 1 -- 172.21.15.90:0/1128078160 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b6a9a297b0 msgr2=0x55b6a9a2ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:47:52.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.947+0000 7fc5cca00640 1 --2- 172.21.15.90:0/1128078160 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b6a9a297b0 0x55b6a9a2ece0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:52.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.947+0000 7fc5cca00640 1 -- 172.21.15.90:0/1128078160 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b6a9a2a0d0 msgr2=0x55b6a9a3e990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:47:52.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.947+0000 7fc5cca00640 1 --2- 172.21.15.90:0/1128078160 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b6a9a2a0d0 0x55b6a9a3e990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:52.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.947+0000 7fc5cca00640 1 -- 172.21.15.90:0/1128078160 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b6a9a45810 con 0x55b6a9a32cb0 2024-04-02T14:47:52.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.948+0000 7fc5cca00640 1 --2- 172.21.15.90:0/1128078160 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b6a9a32cb0 0x55b6a9a30640 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7fc5c0009ca0 tx=0x7fc5c00df300 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c5046042b089354d server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:47:52.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.948+0000 7fc5b3e00640 1 -- 172.21.15.90:0/1128078160 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5c00beb50 con 0x55b6a9a32cb0 2024-04-02T14:47:52.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.948+0000 7fc5b3e00640 1 -- 172.21.15.90:0/1128078160 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc5c00c77c0 con 0x55b6a9a32cb0 2024-04-02T14:47:52.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.948+0000 7fc5b3e00640 1 -- 172.21.15.90:0/1128078160 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5c00c7480 con 0x55b6a9a32cb0 2024-04-02T14:47:52.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.949+0000 7fc5d1793740 1 -- 172.21.15.90:0/1128078160 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b6a9a32cb0 msgr2=0x55b6a9a30640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:52.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.949+0000 7fc5d1793740 1 --2- 172.21.15.90:0/1128078160 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b6a9a32cb0 0x55b6a9a30640 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7fc5c0009ca0 tx=0x7fc5c00df300 comp rx=0 tx=0).stop 2024-04-02T14:47:52.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.949+0000 7fc5d1793740 1 -- 172.21.15.90:0/1128078160 shutdown_connections 2024-04-02T14:47:52.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.949+0000 7fc5d1793740 1 --2- 172.21.15.90:0/1128078160 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b6a9a32cb0 0x55b6a9a30640 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:52.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.949+0000 7fc5d1793740 1 --2- 172.21.15.90:0/1128078160 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b6a9a2a0d0 0x55b6a9a3e990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:52.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.949+0000 7fc5d1793740 1 --2- 172.21.15.90:0/1128078160 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b6a9a297b0 0x55b6a9a2ece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:52.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.949+0000 7fc5d1793740 1 -- 172.21.15.90:0/1128078160 >> 172.21.15.90:0/1128078160 conn(0x55b6a9a23ee0 msgr2=0x55b6a9a25ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:47:52.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.949+0000 7fc5d1793740 1 -- 172.21.15.90:0/1128078160 shutdown_connections 2024-04-02T14:47:52.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.949+0000 7fc5d1793740 1 -- 172.21.15.90:0/1128078160 wait complete. 2024-04-02T14:47:52.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.950+0000 7fc5d1793740 1 -- start start 2024-04-02T14:47:52.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.950+0000 7fc5d1793740 1 Processor -- start 2024-04-02T14:47:52.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.950+0000 7fc5d1793740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b6a9a297b0 0x55b6a9a3aea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:52.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.950+0000 7fc5d1793740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b6a9a2a0d0 0x55b6a9a3b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:52.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.950+0000 7fc5d1793740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b6a9a32cb0 0x55b6a9aa2250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:52.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.950+0000 7fc5d1793740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fc5c81b0e20 con 0x55b6a9a32cb0 2024-04-02T14:47:52.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.950+0000 7fc5d1793740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fc5c81b0750 con 0x55b6a9a2a0d0 2024-04-02T14:47:52.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.950+0000 7fc5d1793740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fc5c81ad9b0 con 0x55b6a9a297b0 2024-04-02T14:47:52.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.950+0000 7fc5d1793740 10 client.0 _ll_register_callbacks cb 0x7fc5c00be920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:47:52.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.951+0000 7fc5c7400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b6a9a2a0d0 0x55b6a9a3b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:52.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.951+0000 7fc5c7400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b6a9a2a0d0 0x55b6a9a3b3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:50450/0 (socket says 172.21.15.90:50450) 2024-04-02T14:47:52.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.951+0000 7fc5c7400640 1 -- 172.21.15.90:0/30812777 learned_addr learned my addr 172.21.15.90:0/30812777 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:47:52.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.951+0000 7fc5c7e00640 1 --2- 172.21.15.90:0/30812777 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b6a9a297b0 0x55b6a9a3aea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:52.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.951+0000 7fc5c7400640 1 -- 172.21.15.90:0/30812777 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b6a9a297b0 msgr2=0x55b6a9a3aea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:47:52.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.951+0000 7fc5c7400640 1 --2- 172.21.15.90:0/30812777 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b6a9a297b0 0x55b6a9a3aea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:52.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.951+0000 7fc5c7400640 1 -- 172.21.15.90:0/30812777 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b6a9a32cb0 msgr2=0x55b6a9aa2250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:47:52.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.951+0000 7fc5c7400640 1 --2- 172.21.15.90:0/30812777 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b6a9a32cb0 0x55b6a9aa2250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:52.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.951+0000 7fc5c7400640 1 -- 172.21.15.90:0/30812777 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b6a9aa2790 con 0x55b6a9a2a0d0 2024-04-02T14:47:52.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.952+0000 7fc5c7400640 1 --2- 172.21.15.90:0/30812777 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b6a9a2a0d0 0x55b6a9a3b3e0 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7fc5b4009b50 tx=0x7fc5b40056b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1716b8774cf6d261 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:47:52.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.952+0000 7fc5b1600640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:47:52.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.952+0000 7fc5b1600640 1 -- 172.21.15.90:0/30812777 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5b400f940 con 0x55b6a9a2a0d0 2024-04-02T14:47:52.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.953+0000 7fc5b1600640 1 -- 172.21.15.90:0/30812777 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 8550+0+0 (secure 0 0 0) 0x7fc5b400fae0 con 0x55b6a9a2a0d0 2024-04-02T14:47:52.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.953+0000 7fc5d1793740 1 -- 172.21.15.90:0/30812777 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc5c00c77c0 con 0x55b6a9a2a0d0 2024-04-02T14:47:52.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.953+0000 7fc5b1600640 1 -- 172.21.15.90:0/30812777 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fc5b400fce0 con 0x55b6a9a2a0d0 2024-04-02T14:47:52.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.953+0000 7fc5d1793740 10 client.29579 fetch_fsmap learned FSMap version 27 2024-04-02T14:47:52.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.953+0000 7fc5d1793740 1 -- 172.21.15.90:0/30812777 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55b6a9ab2340 con 0x55b6a9a2a0d0 2024-04-02T14:47:52.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.953+0000 7fc5b1600640 1 -- 172.21.15.90:0/30812777 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc5c00c77c0 con 0x55b6a9a2a0d0 2024-04-02T14:47:52.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.953+0000 7fc5b1600640 1 -- 172.21.15.90:0/30812777 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fc5b401e070 con 0x55b6a9a2a0d0 2024-04-02T14:47:52.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.953+0000 7fc5d1793740 10 client.29579 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:47:52.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.953+0000 7fc5d1793740 10 client.29579 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:47:52.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.953+0000 7fc5d1793740 20 client.29579 populate_metadata read hostname 'smithi090' 2024-04-02T14:47:52.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.953+0000 7fc5d1793740 1 --2- 172.21.15.90:0/30812777 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55b6a9ab28c0 0x55b6a9ab4ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:47:52.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.953+0000 7fc5d1793740 4 client.29579 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-02T14:47:52.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.953+0000 7fc5d1793740 1 -- 172.21.15.90:0/30812777 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55b6a999acb0 con 0x55b6a9ab28c0 2024-04-02T14:47:52.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.954+0000 7fc5c7e00640 1 --2- 172.21.15.90:0/30812777 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55b6a9ab28c0 0x55b6a9ab4ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:47:52.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.954+0000 7fc5b1600640 10 client.29579 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:47:52.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.954+0000 7fc5c7e00640 1 --2- 172.21.15.90:0/30812777 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55b6a9ab28c0 0x55b6a9ab4ce0 crc :-1 s=READY pgs=339 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=71c05fd4d2a76f7f server_cookie=a962d3ff090f7fa6 in_seq=0 out_seq=0 2024-04-02T14:47:52.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.955+0000 7fc5b1600640 1 -- 172.21.15.90:0/30812777 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55b6a999acb0 con 0x55b6a9ab28c0 2024-04-02T14:47:52.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.955+0000 7fc5b1600640 10 client.29579 handle_command_reply: tid=0 2024-04-02T14:47:52.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.955+0000 7fc5d1793740 1 client.29579 shutdown 2024-04-02T14:47:52.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.955+0000 7fc5d1793740 1 -- 172.21.15.90:0/30812777 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b6a9a2a0d0 msgr2=0x55b6a9a3b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:52.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.955+0000 7fc5d1793740 1 --2- 172.21.15.90:0/30812777 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b6a9a2a0d0 0x55b6a9a3b3e0 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7fc5b4009b50 tx=0x7fc5b40056b0 comp rx=0 tx=0).stop 2024-04-02T14:47:52.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.955+0000 7fc5d1793740 1 -- 172.21.15.90:0/30812777 shutdown_connections 2024-04-02T14:47:52.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.955+0000 7fc5d1793740 1 --2- 172.21.15.90:0/30812777 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55b6a9ab28c0 0x55b6a9ab4ce0 crc :-1 s=READY pgs=339 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:52.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.955+0000 7fc5d1793740 1 --2- 172.21.15.90:0/30812777 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b6a9a32cb0 0x55b6a9aa2250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:52.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.955+0000 7fc5d1793740 1 --2- 172.21.15.90:0/30812777 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b6a9a2a0d0 0x55b6a9a3b3e0 unknown :-1 s=CLOSED pgs=917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:52.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.955+0000 7fc5d1793740 1 --2- 172.21.15.90:0/30812777 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b6a9a297b0 0x55b6a9a3aea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:52.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.955+0000 7fc5b1600640 0 client.29579 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:47:52.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.955+0000 7fc5d1793740 1 -- 172.21.15.90:0/30812777 >> 172.21.15.90:0/30812777 conn(0x55b6a9a23ee0 msgr2=0x55b6a9a226d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:47:52.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.955+0000 7fc5d1793740 1 -- 172.21.15.90:0/30812777 shutdown_connections 2024-04-02T14:47:52.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.955+0000 7fc5d1793740 1 -- 172.21.15.90:0/30812777 wait complete. 2024-04-02T14:47:52.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.955+0000 7fc5d1793740 20 client.29579 trim_cache size 0 max 16384 2024-04-02T14:47:52.953 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-02T14:47:52.953 INFO:teuthology.orchestra.run.smithi090.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-02T14:47:52.953 INFO:teuthology.orchestra.run.smithi090.stdout: "scrubs": { 2024-04-02T14:47:52.953 INFO:teuthology.orchestra.run.smithi090.stdout: "7ef8edb4-83cf-4332-a60d-24f59e2bbde4": { 2024-04-02T14:47:52.953 INFO:teuthology.orchestra.run.smithi090.stdout: "path": "/", 2024-04-02T14:47:52.953 INFO:teuthology.orchestra.run.smithi090.stdout: "tag": "7ef8edb4-83cf-4332-a60d-24f59e2bbde4", 2024-04-02T14:47:52.953 INFO:teuthology.orchestra.run.smithi090.stdout: "options": "recursive,force" 2024-04-02T14:47:52.953 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:47:52.953 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:47:52.953 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-02T14:47:52.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.956+0000 7fc5cea00640 1 -- 172.21.15.90:0/2263427907 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fc5ac0a3da0 msgr2=0x7fc5ac0a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:52.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.956+0000 7fc5cea00640 1 --2- 172.21.15.90:0/2263427907 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fc5ac0a3da0 0x7fc5ac0a6260 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x55b6a9a29280 tx=0x7fc5b8005f90 comp rx=0 tx=0).stop 2024-04-02T14:47:52.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.956+0000 7fc5cea00640 1 -- 172.21.15.90:0/2263427907 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc5c810e2c0 msgr2=0x7fc5c819ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:47:52.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.956+0000 7fc5cea00640 1 --2- 172.21.15.90:0/2263427907 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc5c810e2c0 0x7fc5c819ee20 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7fc5c0009bf0 tx=0x7fc5c0035d80 comp rx=0 tx=0).stop 2024-04-02T14:47:52.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.956+0000 7fc5cea00640 1 -- 172.21.15.90:0/2263427907 shutdown_connections 2024-04-02T14:47:52.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.956+0000 7fc5cea00640 1 --2- 172.21.15.90:0/2263427907 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fc5ac0a3da0 0x7fc5ac0a6260 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:52.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.956+0000 7fc5cea00640 1 --2- 172.21.15.90:0/2263427907 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc5c810e2c0 0x7fc5c819ee20 unknown :-1 s=CLOSED pgs=914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:52.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.956+0000 7fc5cea00640 1 --2- 172.21.15.90:0/2263427907 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc5c8074850 0x7fc5c819aaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:52.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.956+0000 7fc5cea00640 1 --2- 172.21.15.90:0/2263427907 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc5c8073f30 0x7fc5c819a560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:47:52.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.956+0000 7fc5cea00640 1 -- 172.21.15.90:0/2263427907 >> 172.21.15.90:0/2263427907 conn(0x7fc5c80fd590 msgr2=0x7fc5c80fe300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:47:52.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.956+0000 7fc5cea00640 1 -- 172.21.15.90:0/2263427907 shutdown_connections 2024-04-02T14:47:52.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:47:52.956+0000 7fc5cea00640 1 -- 172.21.15.90:0/2263427907 wait complete. 2024-04-02T14:47:52.969 INFO:tasks.cephfs.filesystem:scrub status for tag:7ef8edb4-83cf-4332-a60d-24f59e2bbde4 - {'path': '/', 'tag': '7ef8edb4-83cf-4332-a60d-24f59e2bbde4', 'options': 'recursive,force'} 2024-04-02T14:47:53.603 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:47:53 smithi092 ceph-mon[36497]: scrub queued for path: / 2024-04-02T14:47:53.603 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:47:53 smithi092 ceph-mon[36497]: scrub summary: idle+waiting paths [/] 2024-04-02T14:47:53.604 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:47:53 smithi092 ceph-mon[36497]: scrub summary: active paths [/] 2024-04-02T14:47:53.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:47:53 smithi177 ceph-mon[36431]: scrub queued for path: / 2024-04-02T14:47:53.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:47:53 smithi177 ceph-mon[36431]: scrub summary: idle+waiting paths [/] 2024-04-02T14:47:53.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:47:53 smithi177 ceph-mon[36431]: scrub summary: active paths [/] 2024-04-02T14:47:53.681 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:47:53 smithi090 ceph-mon[31674]: scrub queued for path: / 2024-04-02T14:47:53.681 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:47:53 smithi090 ceph-mon[31674]: scrub summary: idle+waiting paths [/] 2024-04-02T14:47:53.681 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:47:53 smithi090 ceph-mon[31674]: scrub summary: active paths [/] 2024-04-02T14:47:54.603 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:47:54 smithi092 ceph-mon[36497]: pgmap v405: 97 pgs: 97 active+clean; 4.5 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 33 MiB/s wr, 1.56k op/s 2024-04-02T14:47:54.603 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:47:54 smithi092 ceph-mon[36497]: scrub summary: idle+waiting paths [/] 2024-04-02T14:47:54.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:47:54 smithi177 ceph-mon[36431]: pgmap v405: 97 pgs: 97 active+clean; 4.5 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 33 MiB/s wr, 1.56k op/s 2024-04-02T14:47:54.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:47:54 smithi177 ceph-mon[36431]: scrub summary: idle+waiting paths [/] 2024-04-02T14:47:54.681 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:47:54 smithi090 ceph-mon[31674]: pgmap v405: 97 pgs: 97 active+clean; 4.5 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 33 MiB/s wr, 1.56k op/s 2024-04-02T14:47:54.681 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:47:54 smithi090 ceph-mon[31674]: scrub summary: idle+waiting paths [/] 2024-04-02T14:47:56.603 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:47:56 smithi092 ceph-mon[36497]: pgmap v406: 97 pgs: 97 active+clean; 4.6 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 38 MiB/s wr, 1.81k op/s 2024-04-02T14:47:56.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:47:56 smithi177 ceph-mon[36431]: pgmap v406: 97 pgs: 97 active+clean; 4.6 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 38 MiB/s wr, 1.81k op/s 2024-04-02T14:47:56.681 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:47:56 smithi090 ceph-mon[31674]: pgmap v406: 97 pgs: 97 active+clean; 4.6 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 38 MiB/s wr, 1.81k op/s 2024-04-02T14:47:56.824 INFO:teuthology.orchestra.run.smithi090.stderr:progress: 745.0 s, 1390.6 tps, lat 71.369 ms stddev 88.896 2024-04-02T14:47:57.603 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:47:57 smithi092 ceph-mon[36497]: scrub summary: idle 2024-04-02T14:47:57.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:47:57 smithi177 ceph-mon[36431]: scrub summary: idle 2024-04-02T14:47:57.681 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:47:57 smithi090 ceph-mon[31674]: scrub summary: idle 2024-04-02T14:47:58.603 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:47:58 smithi092 ceph-mon[36497]: pgmap v407: 97 pgs: 97 active+clean; 4.6 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 19 MiB/s wr, 992 op/s 2024-04-02T14:47:58.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:47:58 smithi177 ceph-mon[36431]: pgmap v407: 97 pgs: 97 active+clean; 4.6 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 19 MiB/s wr, 992 op/s 2024-04-02T14:47:58.681 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:47:58 smithi090 ceph-mon[31674]: pgmap v407: 97 pgs: 97 active+clean; 4.6 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 19 MiB/s wr, 992 op/s 2024-04-02T14:47:59.603 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:47:59 smithi092 ceph-mon[36497]: pgmap v408: 97 pgs: 97 active+clean; 4.6 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 12 MiB/s wr, 708 op/s 2024-04-02T14:47:59.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:47:59 smithi177 ceph-mon[36431]: pgmap v408: 97 pgs: 97 active+clean; 4.6 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 12 MiB/s wr, 708 op/s 2024-04-02T14:47:59.681 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:47:59 smithi090 ceph-mon[31674]: pgmap v408: 97 pgs: 97 active+clean; 4.6 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 12 MiB/s wr, 708 op/s 2024-04-02T14:48:01.824 INFO:teuthology.orchestra.run.smithi090.stderr:progress: 750.0 s, 1133.2 tps, lat 75.309 ms stddev 95.631 2024-04-02T14:48:02.103 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:48:01 smithi092 ceph-mon[36497]: pgmap v409: 97 pgs: 97 active+clean; 4.6 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 17 MiB/s wr, 973 op/s 2024-04-02T14:48:02.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:48:01 smithi177 ceph-mon[36431]: pgmap v409: 97 pgs: 97 active+clean; 4.6 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 17 MiB/s wr, 973 op/s 2024-04-02T14:48:02.181 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:48:01 smithi090 ceph-mon[31674]: pgmap v409: 97 pgs: 97 active+clean; 4.6 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 17 MiB/s wr, 973 op/s 2024-04-02T14:48:04.103 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:48:03 smithi092 ceph-mon[36497]: pgmap v410: 97 pgs: 97 active+clean; 4.6 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 12 MiB/s wr, 699 op/s 2024-04-02T14:48:04.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:48:03 smithi177 ceph-mon[36431]: pgmap v410: 97 pgs: 97 active+clean; 4.6 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 12 MiB/s wr, 699 op/s 2024-04-02T14:48:04.181 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:48:03 smithi090 ceph-mon[31674]: pgmap v410: 97 pgs: 97 active+clean; 4.6 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 12 MiB/s wr, 699 op/s 2024-04-02T14:48:05.655 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.661+0000 7f5b5d800640 1 -- 172.21.15.90:0/3137166455 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5b58073ea0 msgr2=0x7f5b58074280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:48:05.655 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.661+0000 7f5b5d800640 1 --2- 172.21.15.90:0/3137166455 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5b58073ea0 0x7f5b58074280 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f5b50009ca0 tx=0x7f5b5002c190 comp rx=0 tx=0).stop 2024-04-02T14:48:05.655 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.661+0000 7f5b5d800640 1 -- 172.21.15.90:0/3137166455 shutdown_connections 2024-04-02T14:48:05.655 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.661+0000 7f5b5d800640 1 --2- 172.21.15.90:0/3137166455 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5b58112ac0 0x7f5b58114eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:05.655 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.661+0000 7f5b5d800640 1 --2- 172.21.15.90:0/3137166455 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5b580747c0 0x7f5b58112580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:05.655 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.661+0000 7f5b5d800640 1 --2- 172.21.15.90:0/3137166455 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5b58073ea0 0x7f5b58074280 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:05.655 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.661+0000 7f5b5d800640 1 -- 172.21.15.90:0/3137166455 >> 172.21.15.90:0/3137166455 conn(0x7f5b580fd5c0 msgr2=0x7f5b580ff9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:48:05.656 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.661+0000 7f5b5d800640 1 -- 172.21.15.90:0/3137166455 shutdown_connections 2024-04-02T14:48:05.656 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.661+0000 7f5b5d800640 1 -- 172.21.15.90:0/3137166455 wait complete. 2024-04-02T14:48:05.656 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.662+0000 7f5b5d800640 1 Processor -- start 2024-04-02T14:48:05.656 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.662+0000 7f5b5d800640 1 -- start start 2024-04-02T14:48:05.656 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.662+0000 7f5b5d800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5b58073ea0 0x7f5b581a4640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:05.657 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.662+0000 7f5b5d800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5b580747c0 0x7f5b581a4b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:05.657 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.663+0000 7f5b5d800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5b58112ac0 0x7f5b5819e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:05.657 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.663+0000 7f5b5d800640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f5b58118490 con 0x7f5b580747c0 2024-04-02T14:48:05.657 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.663+0000 7f5b5d800640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f5b58118310 con 0x7f5b58112ac0 2024-04-02T14:48:05.657 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.663+0000 7f5b5d800640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f5b58118610 con 0x7f5b58073ea0 2024-04-02T14:48:05.657 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.663+0000 7f5b56000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5b580747c0 0x7f5b581a4b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:05.657 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.663+0000 7f5b56a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5b58073ea0 0x7f5b581a4640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:05.657 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.663+0000 7f5b56000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5b580747c0 0x7f5b581a4b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:35232/0 (socket says 172.21.15.90:35232) 2024-04-02T14:48:05.657 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.663+0000 7f5b56000640 1 -- 172.21.15.90:0/4171403981 learned_addr learned my addr 172.21.15.90:0/4171403981 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:48:05.657 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.663+0000 7f5b57400640 1 --2- 172.21.15.90:0/4171403981 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5b58112ac0 0x7f5b5819e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:05.658 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.663+0000 7f5b56a00640 1 -- 172.21.15.90:0/4171403981 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5b58112ac0 msgr2=0x7f5b5819e7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:48:05.658 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.663+0000 7f5b56a00640 1 --2- 172.21.15.90:0/4171403981 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5b58112ac0 0x7f5b5819e7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:05.658 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.663+0000 7f5b56a00640 1 -- 172.21.15.90:0/4171403981 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5b580747c0 msgr2=0x7f5b581a4b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:48:05.658 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.663+0000 7f5b56a00640 1 --2- 172.21.15.90:0/4171403981 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5b580747c0 0x7f5b581a4b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:05.658 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.663+0000 7f5b56a00640 1 -- 172.21.15.90:0/4171403981 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b5819ef80 con 0x7f5b58073ea0 2024-04-02T14:48:05.658 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.664+0000 7f5b56a00640 1 --2- 172.21.15.90:0/4171403981 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5b58073ea0 0x7f5b581a4640 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f5b50006010 tx=0x7f5b50035d90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:48:05.659 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.665+0000 7f5b4f400640 1 -- 172.21.15.90:0/4171403981 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b50039040 con 0x7f5b58073ea0 2024-04-02T14:48:05.659 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.665+0000 7f5b4f400640 1 -- 172.21.15.90:0/4171403981 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5b50003e50 con 0x7f5b58073ea0 2024-04-02T14:48:05.659 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.665+0000 7f5b4f400640 1 -- 172.21.15.90:0/4171403981 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b500328d0 con 0x7f5b58073ea0 2024-04-02T14:48:05.659 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.665+0000 7f5b5d800640 1 -- 172.21.15.90:0/4171403981 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b5819f210 con 0x7f5b58073ea0 2024-04-02T14:48:05.659 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.666+0000 7f5b5d800640 1 -- 172.21.15.90:0/4171403981 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b581ab490 con 0x7f5b58073ea0 2024-04-02T14:48:05.662 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.669+0000 7f5b6044b740 1 Processor -- start 2024-04-02T14:48:05.662 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.669+0000 7f5b4f400640 1 -- 172.21.15.90:0/4171403981 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f5b50046030 con 0x7f5b58073ea0 2024-04-02T14:48:05.662 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.669+0000 7f5b6044b740 1 -- start start 2024-04-02T14:48:05.663 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.669+0000 7f5b6044b740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a7ab100eb0 0x55a7ab101290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:05.663 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.669+0000 7f5b6044b740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a7ab1017d0 0x55a7ab109e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:05.664 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.669+0000 7f5b6044b740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a7ab10a3b0 0x55a7ab10c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:05.664 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.669+0000 7f5b6044b740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55a7ab065d70 con 0x55a7ab1017d0 2024-04-02T14:48:05.664 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.669+0000 7f5b6044b740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55a7ab00e2a0 con 0x55a7ab10a3b0 2024-04-02T14:48:05.664 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.669+0000 7f5b6044b740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55a7ab061d20 con 0x55a7ab100eb0 2024-04-02T14:48:05.664 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.670+0000 7f5b56000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a7ab10a3b0 0x55a7ab10c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:05.664 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.670+0000 7f5b56000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a7ab10a3b0 0x55a7ab10c7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:38628/0 (socket says 172.21.15.90:38628) 2024-04-02T14:48:05.664 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.670+0000 7f5b56000640 1 -- 172.21.15.90:0/2706847179 learned_addr learned my addr 172.21.15.90:0/2706847179 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:48:05.665 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.671+0000 7f5b57400640 1 --2- 172.21.15.90:0/2706847179 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a7ab100eb0 0x55a7ab101290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:05.665 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.671+0000 7f5b4f400640 1 --2- 172.21.15.90:0/4171403981 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f5b3c0a3f20 0x7f5b3c0a63e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:05.665 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.671+0000 7f5b4f400640 1 -- 172.21.15.90:0/4171403981 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 8550+0+0 (secure 0 0 0) 0x7f5b500f1e80 con 0x7f5b58073ea0 2024-04-02T14:48:05.666 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.672+0000 7f5b56000640 1 -- 172.21.15.90:0/2706847179 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a7ab100eb0 msgr2=0x55a7ab101290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:48:05.666 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.672+0000 7f5b56a00640 1 --2- 172.21.15.90:0/2706847179 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a7ab1017d0 0x55a7ab109e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:05.666 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.672+0000 7f5b56000640 1 --2- 172.21.15.90:0/2706847179 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a7ab100eb0 0x55a7ab101290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:05.666 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.672+0000 7f5b56000640 1 -- 172.21.15.90:0/2706847179 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a7ab1017d0 msgr2=0x55a7ab109e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:48:05.666 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.672+0000 7f5b56000640 1 --2- 172.21.15.90:0/2706847179 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a7ab1017d0 0x55a7ab109e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:05.666 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.672+0000 7f5b56000640 1 -- 172.21.15.90:0/2706847179 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a7ab022840 con 0x55a7ab10a3b0 2024-04-02T14:48:05.666 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.672+0000 7f5b57400640 1 --2- 172.21.15.90:0/2706847179 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a7ab100eb0 0x55a7ab101290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:48:05.666 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.672+0000 7f5b56a00640 1 --2- 172.21.15.90:0/2706847179 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a7ab1017d0 0x55a7ab109e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:48:05.667 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.673+0000 7f5b57400640 1 --2- 172.21.15.90:0/4171403981 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f5b3c0a3f20 0x7f5b3c0a63e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:05.667 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.673+0000 7f5b56000640 1 --2- 172.21.15.90:0/2706847179 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a7ab10a3b0 0x55a7ab10c7a0 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7f5b44009d00 tx=0x7f5b44011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=72d0a3aff4141ee4 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:48:05.667 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.673+0000 7f5b43e00640 1 -- 172.21.15.90:0/2706847179 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b44004220 con 0x55a7ab10a3b0 2024-04-02T14:48:05.667 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.673+0000 7f5b43e00640 1 -- 172.21.15.90:0/2706847179 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5b440043c0 con 0x55a7ab10a3b0 2024-04-02T14:48:05.667 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.673+0000 7f5b43e00640 1 -- 172.21.15.90:0/2706847179 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b440035f0 con 0x55a7ab10a3b0 2024-04-02T14:48:05.668 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.674+0000 7f5b6044b740 1 -- 172.21.15.90:0/2706847179 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a7ab10a3b0 msgr2=0x55a7ab10c7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:48:05.668 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.674+0000 7f5b57400640 1 --2- 172.21.15.90:0/4171403981 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f5b3c0a3f20 0x7f5b3c0a63e0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f5b5806b1b0 tx=0x7f5b480046c0 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:48:05.668 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.674+0000 7f5b6044b740 1 --2- 172.21.15.90:0/2706847179 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a7ab10a3b0 0x55a7ab10c7a0 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7f5b44009d00 tx=0x7f5b44011810 comp rx=0 tx=0).stop 2024-04-02T14:48:05.668 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.674+0000 7f5b6044b740 1 -- 172.21.15.90:0/2706847179 shutdown_connections 2024-04-02T14:48:05.668 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.674+0000 7f5b6044b740 1 --2- 172.21.15.90:0/2706847179 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a7ab10a3b0 0x55a7ab10c7a0 unknown :-1 s=CLOSED pgs=918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:05.668 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.674+0000 7f5b6044b740 1 --2- 172.21.15.90:0/2706847179 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a7ab1017d0 0x55a7ab109e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:05.668 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.674+0000 7f5b6044b740 1 --2- 172.21.15.90:0/2706847179 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a7ab100eb0 0x55a7ab101290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:05.668 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.674+0000 7f5b6044b740 1 -- 172.21.15.90:0/2706847179 >> 172.21.15.90:0/2706847179 conn(0x55a7ab0fb5e0 msgr2=0x55a7ab0fb9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:48:05.669 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.674+0000 7f5b6044b740 1 -- 172.21.15.90:0/2706847179 shutdown_connections 2024-04-02T14:48:05.669 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.674+0000 7f5b6044b740 1 -- 172.21.15.90:0/2706847179 wait complete. 2024-04-02T14:48:05.669 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.675+0000 7f5b6044b740 1 -- start start 2024-04-02T14:48:05.669 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.676+0000 7f5b6044b740 1 Processor -- start 2024-04-02T14:48:05.671 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.677+0000 7f5b6044b740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a7ab100eb0 0x55a7ab107370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:05.671 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.677+0000 7f5b6044b740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a7ab1017d0 0x55a7ab1078b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:05.671 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.677+0000 7f5b6044b740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a7ab10a3b0 0x55a7ab107df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:05.671 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.677+0000 7f5b56a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a7ab100eb0 0x55a7ab107370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:05.672 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.677+0000 7f5b56a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a7ab100eb0 0x55a7ab107370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:38638/0 (socket says 172.21.15.90:38638) 2024-04-02T14:48:05.672 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.677+0000 7f5b56000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a7ab1017d0 0x55a7ab1078b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:05.672 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.677+0000 7f5b56000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a7ab1017d0 0x55a7ab1078b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:59458/0 (socket says 172.21.15.90:59458) 2024-04-02T14:48:05.672 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.677+0000 7f5b57400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a7ab10a3b0 0x55a7ab107df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:05.672 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.677+0000 7f5b57400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a7ab10a3b0 0x55a7ab107df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:35248/0 (socket says 172.21.15.90:35248) 2024-04-02T14:48:05.672 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.677+0000 7f5b6044b740 1 -- 172.21.15.90:0/1618661318 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55a7ab10d1a0 con 0x55a7ab10a3b0 2024-04-02T14:48:05.672 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.677+0000 7f5b56a00640 1 -- 172.21.15.90:0/1618661318 learned_addr learned my addr 172.21.15.90:0/1618661318 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:48:05.672 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.677+0000 7f5b6044b740 1 -- 172.21.15.90:0/1618661318 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55a7ab021660 con 0x55a7ab100eb0 2024-04-02T14:48:05.672 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.677+0000 7f5b6044b740 1 -- 172.21.15.90:0/1618661318 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55a7ab10d320 con 0x55a7ab1017d0 2024-04-02T14:48:05.672 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.677+0000 7f5b6044b740 10 client.0 _ll_register_callbacks cb 0x55a7ab021050 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:48:05.672 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.678+0000 7f5b56000640 1 -- 172.21.15.90:0/1618661318 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a7ab100eb0 msgr2=0x55a7ab107370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:48:05.672 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.678+0000 7f5b56000640 1 --2- 172.21.15.90:0/1618661318 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a7ab100eb0 0x55a7ab107370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:05.672 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.678+0000 7f5b56000640 1 -- 172.21.15.90:0/1618661318 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a7ab10a3b0 msgr2=0x55a7ab107df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:48:05.672 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.678+0000 7f5b56000640 1 --2- 172.21.15.90:0/1618661318 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a7ab10a3b0 0x55a7ab107df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:05.672 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.678+0000 7f5b56000640 1 -- 172.21.15.90:0/1618661318 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a7ab030e80 con 0x55a7ab1017d0 2024-04-02T14:48:05.674 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.678+0000 7f5b56000640 1 --2- 172.21.15.90:0/1618661318 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a7ab1017d0 0x55a7ab1078b0 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f5b4401b040 tx=0x7f5b44005350 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d3b8ae42b4921208 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:48:05.674 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.678+0000 7f5b40c00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:48:05.674 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.678+0000 7f5b40c00640 1 -- 172.21.15.90:0/1618661318 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b44027030 con 0x55a7ab1017d0 2024-04-02T14:48:05.674 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.678+0000 7f5b6044b740 1 -- 172.21.15.90:0/1618661318 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5b440043c0 con 0x55a7ab1017d0 2024-04-02T14:48:05.674 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.679+0000 7f5b40c00640 1 -- 172.21.15.90:0/1618661318 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 8550+0+0 (secure 0 0 0) 0x7f5b44019430 con 0x55a7ab1017d0 2024-04-02T14:48:05.674 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.679+0000 7f5b40c00640 1 -- 172.21.15.90:0/1618661318 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f5b4401d040 con 0x55a7ab1017d0 2024-04-02T14:48:05.674 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.679+0000 7f5b6044b740 10 client.29374 fetch_fsmap learned FSMap version 27 2024-04-02T14:48:05.675 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.679+0000 7f5b6044b740 1 -- 172.21.15.90:0/1618661318 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55a7aad7a210 con 0x55a7ab1017d0 2024-04-02T14:48:05.675 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.679+0000 7f5b40c00640 1 -- 172.21.15.90:0/1618661318 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5b440043c0 con 0x55a7ab1017d0 2024-04-02T14:48:05.675 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.679+0000 7f5b40c00640 1 -- 172.21.15.90:0/1618661318 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f5b44019b10 con 0x55a7ab1017d0 2024-04-02T14:48:05.675 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.679+0000 7f5b6044b740 10 client.29374 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:48:05.675 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.679+0000 7f5b6044b740 10 client.29374 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-02T14:48:05.675 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.679+0000 7f5b6044b740 20 client.29374 populate_metadata read hostname 'smithi090' 2024-04-02T14:48:05.675 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.679+0000 7f5b6044b740 1 --2- 172.21.15.90:0/1618661318 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55a7ab11a5f0 0x55a7ab11c9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:05.675 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.679+0000 7f5b6044b740 4 client.29374 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:48:05.675 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.679+0000 7f5b6044b740 1 -- 172.21.15.90:0/1618661318 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a7ab015cd0 con 0x55a7ab11a5f0 2024-04-02T14:48:05.675 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.680+0000 7f5b56a00640 1 --2- 172.21.15.90:0/1618661318 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55a7ab11a5f0 0x55a7ab11c9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:05.675 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.680+0000 7f5b56a00640 1 --2- 172.21.15.90:0/1618661318 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55a7ab11a5f0 0x55a7ab11c9e0 crc :-1 s=READY pgs=340 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d84ba5d1bbbbd30f server_cookie=a959e98ff684ee87 in_seq=0 out_seq=0 2024-04-02T14:48:05.675 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.680+0000 7f5b40c00640 10 client.29374 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:48:05.676 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.681+0000 7f5b40c00640 1 -- 172.21.15.90:0/1618661318 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a7ab015cd0 con 0x55a7ab11a5f0 2024-04-02T14:48:05.676 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.681+0000 7f5b40c00640 10 client.29374 handle_command_reply: tid=0 2024-04-02T14:48:05.676 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.681+0000 7f5b6044b740 1 client.29374 shutdown 2024-04-02T14:48:05.677 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.682+0000 7f5b6044b740 1 -- 172.21.15.90:0/1618661318 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a7ab1017d0 msgr2=0x55a7ab1078b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:48:05.677 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.682+0000 7f5b6044b740 1 --2- 172.21.15.90:0/1618661318 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a7ab1017d0 0x55a7ab1078b0 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f5b4401b040 tx=0x7f5b44005350 comp rx=0 tx=0).stop 2024-04-02T14:48:05.677 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.682+0000 7f5b6044b740 1 -- 172.21.15.90:0/1618661318 shutdown_connections 2024-04-02T14:48:05.677 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.682+0000 7f5b6044b740 1 --2- 172.21.15.90:0/1618661318 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55a7ab11a5f0 0x55a7ab11c9e0 crc :-1 s=READY pgs=340 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:05.677 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.682+0000 7f5b6044b740 1 --2- 172.21.15.90:0/1618661318 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a7ab10a3b0 0x55a7ab107df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:05.677 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.682+0000 7f5b6044b740 1 --2- 172.21.15.90:0/1618661318 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a7ab1017d0 0x55a7ab1078b0 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:05.677 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.682+0000 7f5b6044b740 1 --2- 172.21.15.90:0/1618661318 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a7ab100eb0 0x55a7ab107370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:05.677 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.682+0000 7f5b40c00640 0 client.29374 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:48:05.677 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.682+0000 7f5b6044b740 1 -- 172.21.15.90:0/1618661318 >> 172.21.15.90:0/1618661318 conn(0x55a7ab0fb5e0 msgr2=0x55a7ab0fd330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:48:05.677 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.683+0000 7f5b6044b740 1 -- 172.21.15.90:0/1618661318 shutdown_connections 2024-04-02T14:48:05.677 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.683+0000 7f5b6044b740 1 -- 172.21.15.90:0/1618661318 wait complete. 2024-04-02T14:48:05.677 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.683+0000 7f5b6044b740 20 client.29374 trim_cache size 0 max 16384 2024-04-02T14:48:05.721 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.727+0000 7f5b6044b740 1 Processor -- start 2024-04-02T14:48:05.722 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.728+0000 7f5b6044b740 1 -- start start 2024-04-02T14:48:05.722 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.729+0000 7f5b6044b740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a7ab0fb5e0 0x55a7ab109aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:05.722 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.729+0000 7f5b6044b740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a7ab100eb0 0x55a7ab116090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:05.723 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.729+0000 7f5b6044b740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a7ab1017d0 0x55a7ab0fd250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:05.723 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.729+0000 7f5b6044b740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f5b581b3d90 con 0x55a7ab0fb5e0 2024-04-02T14:48:05.723 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.729+0000 7f5b6044b740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f5b581b58d0 con 0x55a7ab100eb0 2024-04-02T14:48:05.723 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.729+0000 7f5b6044b740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f5b581b4460 con 0x55a7ab1017d0 2024-04-02T14:48:05.723 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.730+0000 7f5b57400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a7ab1017d0 0x55a7ab0fd250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:05.724 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.730+0000 7f5b56a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a7ab0fb5e0 0x55a7ab109aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:05.724 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.730+0000 7f5b56a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a7ab0fb5e0 0x55a7ab109aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:35256/0 (socket says 172.21.15.90:35256) 2024-04-02T14:48:05.724 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.730+0000 7f5b56a00640 1 -- 172.21.15.90:0/3503117636 learned_addr learned my addr 172.21.15.90:0/3503117636 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:48:05.724 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.730+0000 7f5b56000640 1 --2- 172.21.15.90:0/3503117636 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a7ab100eb0 0x55a7ab116090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:05.724 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.731+0000 7f5b56000640 1 -- 172.21.15.90:0/3503117636 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a7ab1017d0 msgr2=0x55a7ab0fd250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:48:05.725 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.731+0000 7f5b56000640 1 --2- 172.21.15.90:0/3503117636 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a7ab1017d0 0x55a7ab0fd250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:05.725 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.731+0000 7f5b56000640 1 -- 172.21.15.90:0/3503117636 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a7ab0fb5e0 msgr2=0x55a7ab109aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:48:05.725 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.731+0000 7f5b56000640 1 --2- 172.21.15.90:0/3503117636 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a7ab0fb5e0 0x55a7ab109aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:05.725 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.731+0000 7f5b56000640 1 -- 172.21.15.90:0/3503117636 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a7ab106340 con 0x55a7ab100eb0 2024-04-02T14:48:05.725 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.731+0000 7f5b56a00640 1 --2- 172.21.15.90:0/3503117636 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a7ab0fb5e0 0x55a7ab109aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:48:05.726 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.732+0000 7f5b56000640 1 --2- 172.21.15.90:0/3503117636 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a7ab100eb0 0x55a7ab116090 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f5b44019f00 tx=0x7f5b44005d30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fe03767232a0ab4a server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:48:05.726 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.732+0000 7f5b43400640 1 -- 172.21.15.90:0/3503117636 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b44038070 con 0x55a7ab100eb0 2024-04-02T14:48:05.726 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.733+0000 7f5b43400640 1 -- 172.21.15.90:0/3503117636 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5b4402e5f0 con 0x55a7ab100eb0 2024-04-02T14:48:05.726 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.733+0000 7f5b43400640 1 -- 172.21.15.90:0/3503117636 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b44027070 con 0x55a7ab100eb0 2024-04-02T14:48:05.730 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.734+0000 7f5b6044b740 1 -- 172.21.15.90:0/3503117636 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a7ab100eb0 msgr2=0x55a7ab116090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:48:05.731 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.734+0000 7f5b6044b740 1 --2- 172.21.15.90:0/3503117636 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a7ab100eb0 0x55a7ab116090 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f5b44019f00 tx=0x7f5b44005d30 comp rx=0 tx=0).stop 2024-04-02T14:48:05.731 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.734+0000 7f5b6044b740 1 -- 172.21.15.90:0/3503117636 shutdown_connections 2024-04-02T14:48:05.731 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.734+0000 7f5b6044b740 1 --2- 172.21.15.90:0/3503117636 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a7ab1017d0 0x55a7ab0fd250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:05.731 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.734+0000 7f5b6044b740 1 --2- 172.21.15.90:0/3503117636 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a7ab100eb0 0x55a7ab116090 unknown :-1 s=CLOSED pgs=919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:05.731 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.734+0000 7f5b6044b740 1 --2- 172.21.15.90:0/3503117636 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a7ab0fb5e0 0x55a7ab109aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:05.731 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.734+0000 7f5b6044b740 1 -- 172.21.15.90:0/3503117636 >> 172.21.15.90:0/3503117636 conn(0x55a7ab11cd50 msgr2=0x55a7ab0fb9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:48:05.732 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.735+0000 7f5b6044b740 1 -- 172.21.15.90:0/3503117636 shutdown_connections 2024-04-02T14:48:05.732 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.736+0000 7f5b6044b740 1 -- 172.21.15.90:0/3503117636 wait complete. 2024-04-02T14:48:05.732 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.736+0000 7f5b6044b740 1 -- start start 2024-04-02T14:48:05.732 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.737+0000 7f5b6044b740 1 Processor -- start 2024-04-02T14:48:05.734 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.739+0000 7f5b6044b740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a7ab0fb5e0 0x55a7ab104250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:05.734 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.739+0000 7f5b6044b740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a7ab100eb0 0x55a7ab104790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:05.734 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.739+0000 7f5b6044b740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a7ab1017d0 0x55a7ab104cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:05.734 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.739+0000 7f5b6044b740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f5b581b5200 con 0x55a7ab0fb5e0 2024-04-02T14:48:05.735 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.740+0000 7f5b56a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a7ab0fb5e0 0x55a7ab104250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:05.735 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.740+0000 7f5b57400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a7ab1017d0 0x55a7ab104cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:05.735 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.740+0000 7f5b56a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a7ab0fb5e0 0x55a7ab104250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:35270/0 (socket says 172.21.15.90:35270) 2024-04-02T14:48:05.735 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.740+0000 7f5b56a00640 1 -- 172.21.15.90:0/2130390874 learned_addr learned my addr 172.21.15.90:0/2130390874 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:48:05.735 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.740+0000 7f5b6044b740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f5b581b4b30 con 0x55a7ab1017d0 2024-04-02T14:48:05.735 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.740+0000 7f5b56000640 1 --2- 172.21.15.90:0/2130390874 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a7ab100eb0 0x55a7ab104790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:05.735 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.740+0000 7f5b6044b740 1 -- 172.21.15.90:0/2130390874 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f5b581b1d90 con 0x55a7ab100eb0 2024-04-02T14:48:05.735 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.740+0000 7f5b6044b740 10 client.0 _ll_register_callbacks cb 0x7f5b44004fe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:48:05.736 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.740+0000 7f5b56a00640 1 -- 172.21.15.90:0/2130390874 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a7ab100eb0 msgr2=0x55a7ab104790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:48:05.737 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.740+0000 7f5b56a00640 1 --2- 172.21.15.90:0/2130390874 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a7ab100eb0 0x55a7ab104790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:05.737 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.740+0000 7f5b56a00640 1 -- 172.21.15.90:0/2130390874 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a7ab1017d0 msgr2=0x55a7ab104cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:48:05.737 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.741+0000 7f5b56a00640 1 --2- 172.21.15.90:0/2130390874 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a7ab1017d0 0x55a7ab104cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:05.737 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.741+0000 7f5b56a00640 1 -- 172.21.15.90:0/2130390874 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a7aae46620 con 0x55a7ab0fb5e0 2024-04-02T14:48:05.737 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.741+0000 7f5b57400640 1 --2- 172.21.15.90:0/2130390874 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a7ab1017d0 0x55a7ab104cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:48:05.737 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.741+0000 7f5b56a00640 1 --2- 172.21.15.90:0/2130390874 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a7ab0fb5e0 0x55a7ab104250 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f5b500ba3a0 tx=0x7f5b5002c190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a0bf5f7c6536e78 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:48:05.737 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.741+0000 7f5b40c00640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-02T14:48:05.737 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.742+0000 7f5b40c00640 1 -- 172.21.15.90:0/2130390874 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b500db070 con 0x55a7ab0fb5e0 2024-04-02T14:48:05.738 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.742+0000 7f5b40c00640 1 -- 172.21.15.90:0/2130390874 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 8550+0+0 (secure 0 0 0) 0x7f5b500041d0 con 0x55a7ab0fb5e0 2024-04-02T14:48:05.738 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.742+0000 7f5b6044b740 1 -- 172.21.15.90:0/2130390874 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5b4402e5f0 con 0x55a7ab0fb5e0 2024-04-02T14:48:05.739 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.743+0000 7f5b40c00640 1 -- 172.21.15.90:0/2130390874 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f5b500c22f0 con 0x55a7ab0fb5e0 2024-04-02T14:48:05.739 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.744+0000 7f5b6044b740 10 client.18444 fetch_fsmap learned FSMap version 27 2024-04-02T14:48:05.739 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.744+0000 7f5b6044b740 1 -- 172.21.15.90:0/2130390874 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55a7ab180d80 con 0x55a7ab0fb5e0 2024-04-02T14:48:05.739 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.744+0000 7f5b40c00640 1 -- 172.21.15.90:0/2130390874 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5b4402e5f0 con 0x55a7ab0fb5e0 2024-04-02T14:48:05.739 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.744+0000 7f5b40c00640 1 -- 172.21.15.90:0/2130390874 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f5b5003c030 con 0x55a7ab0fb5e0 2024-04-02T14:48:05.739 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.745+0000 7f5b6044b740 10 client.18444 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:48:05.740 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.745+0000 7f5b6044b740 10 client.18444 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-02T14:48:05.740 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.745+0000 7f5b6044b740 20 client.18444 populate_metadata read hostname 'smithi090' 2024-04-02T14:48:05.740 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.745+0000 7f5b6044b740 1 --2- 172.21.15.90:0/2130390874 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55a7ab10a3b0 0x55a7ab1834c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:05.740 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.745+0000 7f5b6044b740 4 client.18444 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-02T14:48:05.740 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.745+0000 7f5b6044b740 1 -- 172.21.15.90:0/2130390874 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55a7ab0732b0 con 0x55a7ab10a3b0 2024-04-02T14:48:05.740 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.745+0000 7f5b56000640 1 --2- 172.21.15.90:0/2130390874 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55a7ab10a3b0 0x55a7ab1834c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:05.741 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.746+0000 7f5b56000640 1 --2- 172.21.15.90:0/2130390874 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55a7ab10a3b0 0x55a7ab1834c0 crc :-1 s=READY pgs=341 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=40519fec5c704d7 server_cookie=aee8e0daba638f43 in_seq=0 out_seq=0 2024-04-02T14:48:05.741 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.746+0000 7f5b40c00640 10 client.18444 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:48:05.763 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.768+0000 7f5b40c00640 1 -- 172.21.15.90:0/2130390874 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55a7ab0732b0 con 0x55a7ab10a3b0 2024-04-02T14:48:05.763 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.768+0000 7f5b40c00640 10 client.18444 handle_command_reply: tid=0 2024-04-02T14:48:05.763 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.768+0000 7f5b6044b740 1 client.18444 shutdown 2024-04-02T14:48:05.763 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.769+0000 7f5b6044b740 1 -- 172.21.15.90:0/2130390874 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a7ab0fb5e0 msgr2=0x55a7ab104250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:48:05.763 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.769+0000 7f5b6044b740 1 --2- 172.21.15.90:0/2130390874 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a7ab0fb5e0 0x55a7ab104250 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f5b500ba3a0 tx=0x7f5b5002c190 comp rx=0 tx=0).stop 2024-04-02T14:48:05.763 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.769+0000 7f5b6044b740 1 -- 172.21.15.90:0/2130390874 shutdown_connections 2024-04-02T14:48:05.764 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.769+0000 7f5b6044b740 1 --2- 172.21.15.90:0/2130390874 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55a7ab10a3b0 0x55a7ab1834c0 crc :-1 s=READY pgs=341 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:05.764 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.769+0000 7f5b6044b740 1 --2- 172.21.15.90:0/2130390874 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a7ab1017d0 0x55a7ab104cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:05.764 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.769+0000 7f5b6044b740 1 --2- 172.21.15.90:0/2130390874 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a7ab100eb0 0x55a7ab104790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:05.764 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.769+0000 7f5b6044b740 1 --2- 172.21.15.90:0/2130390874 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a7ab0fb5e0 0x55a7ab104250 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:05.764 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.769+0000 7f5b6044b740 1 -- 172.21.15.90:0/2130390874 >> 172.21.15.90:0/2130390874 conn(0x55a7ab11cd50 msgr2=0x55a7ab0fd250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:48:05.764 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.769+0000 7f5b40c00640 0 client.18444 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:48:05.764 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.769+0000 7f5b6044b740 1 -- 172.21.15.90:0/2130390874 shutdown_connections 2024-04-02T14:48:05.764 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.769+0000 7f5b6044b740 1 -- 172.21.15.90:0/2130390874 wait complete. 2024-04-02T14:48:05.764 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.769+0000 7f5b6044b740 20 client.18444 trim_cache size 0 max 16384 2024-04-02T14:48:05.764 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:{ 2024-04-02T14:48:05.764 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "message": "", 2024-04-02T14:48:05.765 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "return_code": 0 2024-04-02T14:48:05.765 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:} 2024-04-02T14:48:05.765 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.770+0000 7f5b5d800640 1 -- 172.21.15.90:0/4171403981 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f5b3c0a3f20 msgr2=0x7f5b3c0a63e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:48:05.765 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.770+0000 7f5b5d800640 1 --2- 172.21.15.90:0/4171403981 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f5b3c0a3f20 0x7f5b3c0a63e0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f5b5806b1b0 tx=0x7f5b480046c0 comp rx=0 tx=0).stop 2024-04-02T14:48:05.765 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.770+0000 7f5b5d800640 1 -- 172.21.15.90:0/4171403981 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5b58073ea0 msgr2=0x7f5b581a4640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:48:05.766 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.770+0000 7f5b5d800640 1 --2- 172.21.15.90:0/4171403981 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5b58073ea0 0x7f5b581a4640 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f5b50006010 tx=0x7f5b50035d90 comp rx=0 tx=0).stop 2024-04-02T14:48:05.766 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.770+0000 7f5b5d800640 1 -- 172.21.15.90:0/4171403981 shutdown_connections 2024-04-02T14:48:05.766 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.770+0000 7f5b5d800640 1 --2- 172.21.15.90:0/4171403981 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f5b3c0a3f20 0x7f5b3c0a63e0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:05.766 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.770+0000 7f5b5d800640 1 --2- 172.21.15.90:0/4171403981 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5b58112ac0 0x7f5b5819e7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:05.766 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.770+0000 7f5b5d800640 1 --2- 172.21.15.90:0/4171403981 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5b580747c0 0x7f5b581a4b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:05.766 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.770+0000 7f5b5d800640 1 --2- 172.21.15.90:0/4171403981 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5b58073ea0 0x7f5b581a4640 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:05.766 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.770+0000 7f5b5d800640 1 -- 172.21.15.90:0/4171403981 >> 172.21.15.90:0/4171403981 conn(0x7f5b580fd5c0 msgr2=0x7f5b580fed10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:48:05.766 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.770+0000 7f5b5d800640 1 -- 172.21.15.90:0/4171403981 shutdown_connections 2024-04-02T14:48:05.766 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:05.770+0000 7f5b5d800640 1 -- 172.21.15.90:0/4171403981 wait complete. 2024-04-02T14:48:06.103 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:48:05 smithi092 ceph-mon[36497]: pgmap v411: 97 pgs: 97 active+clean; 4.7 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 159 KiB/s rd, 19 MiB/s wr, 1.26k op/s 2024-04-02T14:48:06.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:48:05 smithi177 ceph-mon[36431]: pgmap v411: 97 pgs: 97 active+clean; 4.7 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 159 KiB/s rd, 19 MiB/s wr, 1.26k op/s 2024-04-02T14:48:06.180 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:48:05 smithi090 ceph-mon[31674]: pgmap v411: 97 pgs: 97 active+clean; 4.7 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 159 KiB/s rd, 19 MiB/s wr, 1.26k op/s 2024-04-02T14:48:06.824 INFO:teuthology.orchestra.run.smithi090.stderr:progress: 755.0 s, 1691.6 tps, lat 66.815 ms stddev 122.983 2024-04-02T14:48:08.103 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:48:07 smithi092 ceph-mon[36497]: pgmap v412: 97 pgs: 97 active+clean; 4.7 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 18 MiB/s wr, 1.27k op/s 2024-04-02T14:48:08.103 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:48:07 smithi092 ceph-mon[36497]: osdmap e104: 12 total, 12 up, 12 in 2024-04-02T14:48:08.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:48:07 smithi177 ceph-mon[36431]: pgmap v412: 97 pgs: 97 active+clean; 4.7 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 18 MiB/s wr, 1.27k op/s 2024-04-02T14:48:08.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:48:07 smithi177 ceph-mon[36431]: osdmap e104: 12 total, 12 up, 12 in 2024-04-02T14:48:08.181 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:48:07 smithi090 ceph-mon[31674]: pgmap v412: 97 pgs: 97 active+clean; 4.7 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 18 MiB/s wr, 1.27k op/s 2024-04-02T14:48:08.181 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:48:07 smithi090 ceph-mon[31674]: osdmap e104: 12 total, 12 up, 12 in 2024-04-02T14:48:10.103 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:48:09 smithi092 ceph-mon[36497]: pgmap v414: 97 pgs: 5 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 90 active+clean; 4.7 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 20 MiB/s wr, 1.38k op/s 2024-04-02T14:48:10.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:48:09 smithi177 ceph-mon[36431]: pgmap v414: 97 pgs: 5 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 90 active+clean; 4.7 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 20 MiB/s wr, 1.38k op/s 2024-04-02T14:48:10.181 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:48:09 smithi090 ceph-mon[31674]: pgmap v414: 97 pgs: 5 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 90 active+clean; 4.7 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 20 MiB/s wr, 1.38k op/s 2024-04-02T14:48:11.824 INFO:teuthology.orchestra.run.smithi090.stderr:progress: 760.0 s, 288.8 tps, lat 350.157 ms stddev 735.503 2024-04-02T14:48:12.103 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:48:11 smithi092 ceph-mon[36497]: pgmap v415: 97 pgs: 28 active+clean+snaptrim_wait, 17 active+clean+snaptrim, 52 active+clean; 4.3 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 17 MiB/s wr, 1.23k op/s 2024-04-02T14:48:12.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:48:11 smithi177 ceph-mon[36431]: pgmap v415: 97 pgs: 28 active+clean+snaptrim_wait, 17 active+clean+snaptrim, 52 active+clean; 4.3 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 17 MiB/s wr, 1.23k op/s 2024-04-02T14:48:12.180 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:48:11 smithi090 ceph-mon[31674]: pgmap v415: 97 pgs: 28 active+clean+snaptrim_wait, 17 active+clean+snaptrim, 52 active+clean; 4.3 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 17 MiB/s wr, 1.23k op/s 2024-04-02T14:48:14.103 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:48:13 smithi092 ceph-mon[36497]: pgmap v416: 97 pgs: 28 active+clean+snaptrim_wait, 17 active+clean+snaptrim, 52 active+clean; 4.3 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 17 MiB/s wr, 1.23k op/s 2024-04-02T14:48:14.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:48:13 smithi177 ceph-mon[36431]: pgmap v416: 97 pgs: 28 active+clean+snaptrim_wait, 17 active+clean+snaptrim, 52 active+clean; 4.3 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 17 MiB/s wr, 1.23k op/s 2024-04-02T14:48:14.180 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:48:13 smithi090 ceph-mon[31674]: pgmap v416: 97 pgs: 28 active+clean+snaptrim_wait, 17 active+clean+snaptrim, 52 active+clean; 4.3 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 17 MiB/s wr, 1.23k op/s 2024-04-02T14:48:16.103 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:48:15 smithi092 ceph-mon[36497]: pgmap v417: 97 pgs: 11 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 81 active+clean; 4.2 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 9.6 MiB/s wr, 681 op/s 2024-04-02T14:48:16.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:48:15 smithi177 ceph-mon[36431]: pgmap v417: 97 pgs: 11 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 81 active+clean; 4.2 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 9.6 MiB/s wr, 681 op/s 2024-04-02T14:48:16.181 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:48:15 smithi090 ceph-mon[31674]: pgmap v417: 97 pgs: 11 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 81 active+clean; 4.2 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 9.6 MiB/s wr, 681 op/s 2024-04-02T14:48:16.824 INFO:teuthology.orchestra.run.smithi090.stderr:progress: 765.0 s, 1530.8 tps, lat 66.255 ms stddev 80.599 2024-04-02T14:48:18.103 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:48:17 smithi092 ceph-mon[36497]: pgmap v418: 97 pgs: 97 active+clean; 4.2 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 8.0 MiB/s wr, 463 op/s 2024-04-02T14:48:18.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:48:17 smithi177 ceph-mon[36431]: pgmap v418: 97 pgs: 97 active+clean; 4.2 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 8.0 MiB/s wr, 463 op/s 2024-04-02T14:48:18.181 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:48:17 smithi090 ceph-mon[31674]: pgmap v418: 97 pgs: 97 active+clean; 4.2 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 8.0 MiB/s wr, 463 op/s 2024-04-02T14:48:18.945 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:18.951+0000 7f5652600640 1 -- 172.21.15.90:0/4085611319 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f564c0747f0 msgr2=0x7f564c10de10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:48:18.945 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:18.951+0000 7f5652600640 1 --2- 172.21.15.90:0/4085611319 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f564c0747f0 0x7f564c10de10 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7f5634009900 tx=0x7f563402bcc0 comp rx=0 tx=0).stop 2024-04-02T14:48:18.945 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:18.951+0000 7f5652600640 1 -- 172.21.15.90:0/4085611319 shutdown_connections 2024-04-02T14:48:18.945 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:18.951+0000 7f5652600640 1 --2- 172.21.15.90:0/4085611319 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f564c10e350 0x7f564c110760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:18.946 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:18.951+0000 7f5652600640 1 --2- 172.21.15.90:0/4085611319 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f564c0747f0 0x7f564c10de10 unknown :-1 s=CLOSED pgs=920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:18.946 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:18.951+0000 7f5652600640 1 --2- 172.21.15.90:0/4085611319 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f564c073ed0 0x7f564c0742b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:18.946 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:18.951+0000 7f5652600640 1 -- 172.21.15.90:0/4085611319 >> 172.21.15.90:0/4085611319 conn(0x7f564c0fd5c0 msgr2=0x7f564c0ff9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:48:18.946 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:18.952+0000 7f5652600640 1 -- 172.21.15.90:0/4085611319 shutdown_connections 2024-04-02T14:48:18.946 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:18.952+0000 7f5652600640 1 -- 172.21.15.90:0/4085611319 wait complete. 2024-04-02T14:48:18.947 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:18.953+0000 7f5652600640 1 Processor -- start 2024-04-02T14:48:18.948 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:18.954+0000 7f5652600640 1 -- start start 2024-04-02T14:48:18.950 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:18.955+0000 7f5652600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f564c073ed0 0x7f564c19a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:18.950 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:18.955+0000 7f5652600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f564c0747f0 0x7f564c19aae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:18.950 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:18.955+0000 7f5652600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f564c10e350 0x7f564c19ee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:18.950 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:18.955+0000 7f5652600640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f564c114180 con 0x7f564c073ed0 2024-04-02T14:48:18.950 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:18.955+0000 7f5652600640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f564c114000 con 0x7f564c10e350 2024-04-02T14:48:18.950 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:18.955+0000 7f5652600640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f564c114300 con 0x7f564c0747f0 2024-04-02T14:48:18.950 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:18.955+0000 7f564aa00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f564c0747f0 0x7f564c19aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:18.950 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:18.955+0000 7f564aa00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f564c0747f0 0x7f564c19aae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:46490/0 (socket says 172.21.15.90:46490) 2024-04-02T14:48:18.950 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:18.955+0000 7f564aa00640 1 -- 172.21.15.90:0/25398519 learned_addr learned my addr 172.21.15.90:0/25398519 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:48:18.950 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:18.955+0000 7f564be00640 1 --2- 172.21.15.90:0/25398519 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f564c10e350 0x7f564c19ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:18.950 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:18.956+0000 7f564aa00640 1 -- 172.21.15.90:0/25398519 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f564c10e350 msgr2=0x7f564c19ee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:48:18.950 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:18.956+0000 7f564aa00640 1 --2- 172.21.15.90:0/25398519 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f564c10e350 0x7f564c19ee10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:18.951 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:18.956+0000 7f564aa00640 1 -- 172.21.15.90:0/25398519 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f564c073ed0 msgr2=0x7f564c19a5a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T14:48:18.951 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:18.956+0000 7f564b400640 1 --2- 172.21.15.90:0/25398519 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f564c073ed0 0x7f564c19a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:18.951 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:18.956+0000 7f564aa00640 1 --2- 172.21.15.90:0/25398519 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f564c073ed0 0x7f564c19a5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:18.951 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:18.956+0000 7f564aa00640 1 -- 172.21.15.90:0/25398519 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f564c19f540 con 0x7f564c0747f0 2024-04-02T14:48:18.952 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:18.956+0000 7f564aa00640 1 --2- 172.21.15.90:0/25398519 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f564c0747f0 0x7f564c19aae0 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f563402c2d0 tx=0x7f563402c7a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:48:18.952 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:18.957+0000 7f5643e00640 1 -- 172.21.15.90:0/25398519 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5634043020 con 0x7f564c0747f0 2024-04-02T14:48:18.952 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:18.957+0000 7f5643e00640 1 -- 172.21.15.90:0/25398519 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f56340054e0 con 0x7f564c0747f0 2024-04-02T14:48:18.952 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:18.957+0000 7f5643e00640 1 -- 172.21.15.90:0/25398519 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5634003d70 con 0x7f564c0747f0 2024-04-02T14:48:18.952 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:18.957+0000 7f5652600640 1 -- 172.21.15.90:0/25398519 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f564c19f7d0 con 0x7f564c0747f0 2024-04-02T14:48:18.952 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:18.958+0000 7f5652600640 1 -- 172.21.15.90:0/25398519 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f564c1a70c0 con 0x7f564c0747f0 2024-04-02T14:48:18.958 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:18.964+0000 7f5655439740 1 Processor -- start 2024-04-02T14:48:18.959 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:18.965+0000 7f5655439740 1 -- start start 2024-04-02T14:48:18.959 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:18.965+0000 7f5655439740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556829ed3cc0 0x556829ed40a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:18.959 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:18.965+0000 7f5655439740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556829ed4670 0x556829ee5680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:18.959 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:18.966+0000 7f5655439740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556829ee5bc0 0x556829ee7fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:18.959 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:18.966+0000 7f5655439740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x556829e38d70 con 0x556829ee5bc0 2024-04-02T14:48:18.959 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:18.966+0000 7f5655439740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x556829de12a0 con 0x556829ed3cc0 2024-04-02T14:48:18.959 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:18.966+0000 7f5655439740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x556829e34d20 con 0x556829ed4670 2024-04-02T14:48:18.960 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:18.967+0000 7f5643e00640 1 -- 172.21.15.90:0/25398519 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f5634046020 con 0x7f564c0747f0 2024-04-02T14:48:18.960 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:18.967+0000 7f564be00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556829ed3cc0 0x556829ed40a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:18.960 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:18.967+0000 7f564be00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556829ed3cc0 0x556829ed40a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:36994/0 (socket says 172.21.15.90:36994) 2024-04-02T14:48:18.960 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:18.967+0000 7f564be00640 1 -- 172.21.15.90:0/1701734846 learned_addr learned my addr 172.21.15.90:0/1701734846 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:48:18.961 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:18.967+0000 7f564b400640 1 --2- 172.21.15.90:0/1701734846 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556829ed4670 0x556829ee5680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:18.962 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:18.969+0000 7f5643e00640 1 --2- 172.21.15.90:0/25398519 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f56300a3ec0 0x7f56300a6380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:18.962 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:18.969+0000 7f564aa00640 1 --2- 172.21.15.90:0/1701734846 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556829ee5bc0 0x556829ee7fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:18.963 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:18.969+0000 7f564b400640 1 -- 172.21.15.90:0/1701734846 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556829ed3cc0 msgr2=0x556829ed40a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:48:18.963 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:18.969+0000 7f564b400640 1 --2- 172.21.15.90:0/1701734846 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556829ed3cc0 0x556829ed40a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:18.963 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:18.969+0000 7f564b400640 1 -- 172.21.15.90:0/1701734846 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556829ee5bc0 msgr2=0x556829ee7fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:48:18.963 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:18.969+0000 7f564b400640 1 --2- 172.21.15.90:0/1701734846 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556829ee5bc0 0x556829ee7fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:18.963 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:18.969+0000 7f564b400640 1 -- 172.21.15.90:0/1701734846 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556829df5840 con 0x556829ed4670 2024-04-02T14:48:18.963 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:18.969+0000 7f5643e00640 1 -- 172.21.15.90:0/25398519 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 8594+0+0 (secure 0 0 0) 0x7f56340f2990 con 0x7f564c0747f0 2024-04-02T14:48:18.964 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:18.970+0000 7f564b400640 1 --2- 172.21.15.90:0/1701734846 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556829ed4670 0x556829ee5680 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f563c00a860 tx=0x7f563c00ad20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9ac4dc81f9c0ade0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:48:18.964 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:18.971+0000 7f564be00640 1 --2- 172.21.15.90:0/25398519 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f56300a3ec0 0x7f56300a6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:18.965 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:18.971+0000 7f5640c00640 1 -- 172.21.15.90:0/1701734846 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f563c015070 con 0x556829ed4670 2024-04-02T14:48:18.965 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:18.971+0000 7f5640c00640 1 -- 172.21.15.90:0/1701734846 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f563c00cb40 con 0x556829ed4670 2024-04-02T14:48:18.965 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:18.971+0000 7f5640c00640 1 -- 172.21.15.90:0/1701734846 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f563c0185c0 con 0x556829ed4670 2024-04-02T14:48:18.967 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:18.972+0000 7f5655439740 1 -- 172.21.15.90:0/1701734846 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556829ed4670 msgr2=0x556829ee5680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:48:18.967 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:18.972+0000 7f5655439740 1 --2- 172.21.15.90:0/1701734846 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556829ed4670 0x556829ee5680 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f563c00a860 tx=0x7f563c00ad20 comp rx=0 tx=0).stop 2024-04-02T14:48:18.968 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:18.974+0000 7f5655439740 1 -- 172.21.15.90:0/1701734846 shutdown_connections 2024-04-02T14:48:18.968 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:18.974+0000 7f5655439740 1 --2- 172.21.15.90:0/1701734846 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556829ee5bc0 0x556829ee7fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:18.968 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:18.974+0000 7f5655439740 1 --2- 172.21.15.90:0/1701734846 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556829ed4670 0x556829ee5680 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:18.968 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:18.974+0000 7f5655439740 1 --2- 172.21.15.90:0/1701734846 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556829ed3cc0 0x556829ed40a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:18.968 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:18.974+0000 7f5655439740 1 -- 172.21.15.90:0/1701734846 >> 172.21.15.90:0/1701734846 conn(0x556829ece5e0 msgr2=0x556829ece9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:48:18.968 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:18.974+0000 7f564be00640 1 --2- 172.21.15.90:0/25398519 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f56300a3ec0 0x7f56300a6380 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f5644007f90 tx=0x7f5644004a40 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:48:18.968 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:18.974+0000 7f5655439740 1 -- 172.21.15.90:0/1701734846 shutdown_connections 2024-04-02T14:48:18.968 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:18.975+0000 7f5655439740 1 -- 172.21.15.90:0/1701734846 wait complete. 2024-04-02T14:48:18.969 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:18.975+0000 7f5655439740 1 -- start start 2024-04-02T14:48:18.969 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:18.975+0000 7f5655439740 1 Processor -- start 2024-04-02T14:48:18.969 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:18.976+0000 7f5655439740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556829ed3cc0 0x556829ee2c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:18.970 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:18.976+0000 7f5655439740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556829ed4670 0x556829ee31a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:18.970 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:18.976+0000 7f5655439740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556829ee5bc0 0x556829ee36e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:18.971 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:18.977+0000 7f564b400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556829ed3cc0 0x556829ee2c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:18.971 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:18.977+0000 7f5655439740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x556829ee8980 con 0x556829ed4670 2024-04-02T14:48:18.971 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:18.977+0000 7f5655439740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x556829df4660 con 0x556829ee5bc0 2024-04-02T14:48:18.971 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:18.977+0000 7f5655439740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x556829ee8b00 con 0x556829ed3cc0 2024-04-02T14:48:18.971 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:18.977+0000 7f5655439740 10 client.0 _ll_register_callbacks cb 0x556829df4050 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:48:18.972 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:18.977+0000 7f564be00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556829ee5bc0 0x556829ee36e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:18.972 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:18.977+0000 7f564b400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556829ed3cc0 0x556829ee2c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:46502/0 (socket says 172.21.15.90:46502) 2024-04-02T14:48:18.972 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:18.977+0000 7f564b400640 1 -- 172.21.15.90:0/3869545101 learned_addr learned my addr 172.21.15.90:0/3869545101 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:48:18.973 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:18.978+0000 7f564aa00640 1 --2- 172.21.15.90:0/3869545101 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556829ed4670 0x556829ee31a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:18.973 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:18.979+0000 7f564be00640 1 -- 172.21.15.90:0/3869545101 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556829ed3cc0 msgr2=0x556829ee2c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:48:18.973 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:18.979+0000 7f564be00640 1 --2- 172.21.15.90:0/3869545101 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556829ed3cc0 0x556829ee2c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:18.973 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:18.979+0000 7f564be00640 1 -- 172.21.15.90:0/3869545101 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556829ed4670 msgr2=0x556829ee31a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:48:18.973 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:18.979+0000 7f564be00640 1 --2- 172.21.15.90:0/3869545101 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556829ed4670 0x556829ee31a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:18.973 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:18.979+0000 7f564be00640 1 -- 172.21.15.90:0/3869545101 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556829e03e80 con 0x556829ee5bc0 2024-04-02T14:48:18.974 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:18.981+0000 7f564be00640 1 --2- 172.21.15.90:0/3869545101 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556829ee5bc0 0x556829ee36e0 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7f5644012ee0 tx=0x7f5644004ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5ab81f417138f116 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:48:18.974 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:18.981+0000 7f5639600640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:48:18.976 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:18.981+0000 7f5639600640 1 -- 172.21.15.90:0/3869545101 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5644027670 con 0x556829ee5bc0 2024-04-02T14:48:18.976 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:18.982+0000 7f5655439740 1 -- 172.21.15.90:0/3869545101 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f563c00cb40 con 0x556829ee5bc0 2024-04-02T14:48:18.977 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:18.983+0000 7f5639600640 1 -- 172.21.15.90:0/3869545101 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 8594+0+0 (secure 0 0 0) 0x7f5644027470 con 0x556829ee5bc0 2024-04-02T14:48:18.977 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:18.983+0000 7f5639600640 1 -- 172.21.15.90:0/3869545101 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f5644027670 con 0x556829ee5bc0 2024-04-02T14:48:18.977 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:18.983+0000 7f5655439740 10 client.29609 fetch_fsmap learned FSMap version 27 2024-04-02T14:48:18.977 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:18.983+0000 7f5655439740 1 -- 172.21.15.90:0/3869545101 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x556829b4d210 con 0x556829ee5bc0 2024-04-02T14:48:18.977 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:18.984+0000 7f5639600640 1 -- 172.21.15.90:0/3869545101 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f563c00cb40 con 0x556829ee5bc0 2024-04-02T14:48:18.978 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:18.984+0000 7f5639600640 1 -- 172.21.15.90:0/3869545101 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f564401e8b0 con 0x556829ee5bc0 2024-04-02T14:48:18.978 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:18.984+0000 7f5655439740 10 client.29609 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:48:18.978 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:18.984+0000 7f5655439740 10 client.29609 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-02T14:48:18.978 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:18.984+0000 7f5655439740 20 client.29609 populate_metadata read hostname 'smithi090' 2024-04-02T14:48:18.978 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:18.984+0000 7f5655439740 1 --2- 172.21.15.90:0/3869545101 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x556829ef6010 0x556829ef8400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:18.978 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:18.984+0000 7f5655439740 4 client.29609 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:48:18.978 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:18.984+0000 7f5655439740 1 -- 172.21.15.90:0/3869545101 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556829de8cd0 con 0x556829ef6010 2024-04-02T14:48:18.978 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:18.985+0000 7f564b400640 1 --2- 172.21.15.90:0/3869545101 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x556829ef6010 0x556829ef8400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:18.979 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:18.986+0000 7f564b400640 1 --2- 172.21.15.90:0/3869545101 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x556829ef6010 0x556829ef8400 crc :-1 s=READY pgs=342 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=749a5149ac35db98 server_cookie=adb3405fefc47b1d in_seq=0 out_seq=0 2024-04-02T14:48:18.979 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:18.986+0000 7f5639600640 10 client.29609 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:48:18.980 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:18.987+0000 7f5639600640 1 -- 172.21.15.90:0/3869545101 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x556829de8cd0 con 0x556829ef6010 2024-04-02T14:48:18.980 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:18.987+0000 7f5639600640 10 client.29609 handle_command_reply: tid=0 2024-04-02T14:48:18.980 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:18.987+0000 7f5655439740 1 client.29609 shutdown 2024-04-02T14:48:18.981 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:18.987+0000 7f5655439740 1 -- 172.21.15.90:0/3869545101 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556829ee5bc0 msgr2=0x556829ee36e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:48:18.981 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:18.987+0000 7f5655439740 1 --2- 172.21.15.90:0/3869545101 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556829ee5bc0 0x556829ee36e0 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7f5644012ee0 tx=0x7f5644004ab0 comp rx=0 tx=0).stop 2024-04-02T14:48:18.982 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:18.988+0000 7f5655439740 1 -- 172.21.15.90:0/3869545101 shutdown_connections 2024-04-02T14:48:18.982 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:18.988+0000 7f5655439740 1 --2- 172.21.15.90:0/3869545101 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x556829ef6010 0x556829ef8400 crc :-1 s=READY pgs=342 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:18.982 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:18.988+0000 7f5655439740 1 --2- 172.21.15.90:0/3869545101 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556829ee5bc0 0x556829ee36e0 unknown :-1 s=CLOSED pgs=921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:18.982 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:18.988+0000 7f5655439740 1 --2- 172.21.15.90:0/3869545101 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556829ed4670 0x556829ee31a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:18.983 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:18.988+0000 7f5655439740 1 --2- 172.21.15.90:0/3869545101 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556829ed3cc0 0x556829ee2c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:18.983 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:18.988+0000 7f5639600640 0 client.29609 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:48:18.983 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:18.988+0000 7f5655439740 1 -- 172.21.15.90:0/3869545101 >> 172.21.15.90:0/3869545101 conn(0x556829ece5e0 msgr2=0x556829ed0330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:48:18.983 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:18.989+0000 7f5655439740 1 -- 172.21.15.90:0/3869545101 shutdown_connections 2024-04-02T14:48:18.984 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:18.991+0000 7f5655439740 1 -- 172.21.15.90:0/3869545101 wait complete. 2024-04-02T14:48:18.984 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:18.991+0000 7f5655439740 20 client.29609 trim_cache size 0 max 16384 2024-04-02T14:48:19.024 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:19.030+0000 7f5655439740 1 Processor -- start 2024-04-02T14:48:19.025 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:19.031+0000 7f5655439740 1 -- start start 2024-04-02T14:48:19.025 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:19.031+0000 7f5655439740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556829ed3cc0 0x556829ef1880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:19.025 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:19.031+0000 7f5655439740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556829ed4670 0x556829ee3500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:19.025 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:19.031+0000 7f5655439740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556829ee5bc0 0x556829efdd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:19.026 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:19.031+0000 7f5655439740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f564c1ab680 con 0x556829ee5bc0 2024-04-02T14:48:19.026 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:19.031+0000 7f5655439740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f564c1ad1c0 con 0x556829ed4670 2024-04-02T14:48:19.026 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:19.031+0000 7f5655439740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f564c1abd50 con 0x556829ed3cc0 2024-04-02T14:48:19.026 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:19.032+0000 7f564b400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556829ed3cc0 0x556829ef1880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:19.026 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:19.032+0000 7f564aa00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556829ed4670 0x556829ee3500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:19.026 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:19.032+0000 7f564b400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556829ed3cc0 0x556829ef1880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:46508/0 (socket says 172.21.15.90:46508) 2024-04-02T14:48:19.026 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:19.032+0000 7f564aa00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556829ed4670 0x556829ee3500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:37008/0 (socket says 172.21.15.90:37008) 2024-04-02T14:48:19.026 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:19.032+0000 7f564b400640 1 -- 172.21.15.90:0/549080345 learned_addr learned my addr 172.21.15.90:0/549080345 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:48:19.026 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:19.032+0000 7f564be00640 1 --2- 172.21.15.90:0/549080345 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556829ee5bc0 0x556829efdd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:19.027 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:19.032+0000 7f564b400640 1 -- 172.21.15.90:0/549080345 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556829ed4670 msgr2=0x556829ee3500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:48:19.027 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:19.032+0000 7f564b400640 1 --2- 172.21.15.90:0/549080345 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556829ed4670 0x556829ee3500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:19.027 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:19.032+0000 7f564b400640 1 -- 172.21.15.90:0/549080345 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556829ee5bc0 msgr2=0x556829efdd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:48:19.027 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:19.032+0000 7f564b400640 1 --2- 172.21.15.90:0/549080345 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556829ee5bc0 0x556829efdd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:19.027 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:19.032+0000 7f564b400640 1 -- 172.21.15.90:0/549080345 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556829ef8770 con 0x556829ed3cc0 2024-04-02T14:48:19.027 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:19.033+0000 7f564b400640 1 --2- 172.21.15.90:0/549080345 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556829ed3cc0 0x556829ef1880 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f563c00a830 tx=0x7f563c008040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=131a3bb0f2212969 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:48:19.027 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:19.033+0000 7f563be00640 1 -- 172.21.15.90:0/549080345 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f563c02b070 con 0x556829ed3cc0 2024-04-02T14:48:19.028 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:19.033+0000 7f563be00640 1 -- 172.21.15.90:0/549080345 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f563c0118d0 con 0x556829ed3cc0 2024-04-02T14:48:19.028 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:19.033+0000 7f563be00640 1 -- 172.21.15.90:0/549080345 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f563c015040 con 0x556829ed3cc0 2024-04-02T14:48:19.028 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:19.034+0000 7f5655439740 1 -- 172.21.15.90:0/549080345 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556829ed3cc0 msgr2=0x556829ef1880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:48:19.028 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:19.034+0000 7f5655439740 1 --2- 172.21.15.90:0/549080345 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556829ed3cc0 0x556829ef1880 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f563c00a830 tx=0x7f563c008040 comp rx=0 tx=0).stop 2024-04-02T14:48:19.029 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:19.034+0000 7f5655439740 1 -- 172.21.15.90:0/549080345 shutdown_connections 2024-04-02T14:48:19.029 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:19.034+0000 7f5655439740 1 --2- 172.21.15.90:0/549080345 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556829ee5bc0 0x556829efdd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:19.029 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:19.034+0000 7f5655439740 1 --2- 172.21.15.90:0/549080345 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556829ed4670 0x556829ee3500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:19.029 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:19.034+0000 7f5655439740 1 --2- 172.21.15.90:0/549080345 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556829ed3cc0 0x556829ef1880 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:19.029 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:19.034+0000 7f5655439740 1 -- 172.21.15.90:0/549080345 >> 172.21.15.90:0/549080345 conn(0x556829ece5e0 msgr2=0x556829ece9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:48:19.029 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:19.034+0000 7f5655439740 1 -- 172.21.15.90:0/549080345 shutdown_connections 2024-04-02T14:48:19.029 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:19.034+0000 7f5655439740 1 -- 172.21.15.90:0/549080345 wait complete. 2024-04-02T14:48:19.029 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:19.034+0000 7f5655439740 1 -- start start 2024-04-02T14:48:19.029 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:19.035+0000 7f5655439740 1 Processor -- start 2024-04-02T14:48:19.032 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:19.036+0000 7f5655439740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556829ed3cc0 0x556829ee96d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:19.032 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:19.036+0000 7f5655439740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556829ed4670 0x556829ee9c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:19.032 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:19.036+0000 7f5655439740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556829ee5bc0 0x556829eea150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:19.032 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:19.036+0000 7f5655439740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f564c1acaf0 con 0x556829ed4670 2024-04-02T14:48:19.032 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:19.036+0000 7f5655439740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f564c1ac420 con 0x556829ee5bc0 2024-04-02T14:48:19.032 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:19.036+0000 7f5655439740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f564c1a96a0 con 0x556829ed3cc0 2024-04-02T14:48:19.032 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:19.036+0000 7f5655439740 10 client.0 _ll_register_callbacks cb 0x7f5644012b90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:48:19.032 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:19.036+0000 7f564be00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556829ee5bc0 0x556829eea150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:19.032 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:19.036+0000 7f564b400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556829ed3cc0 0x556829ee96d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:19.032 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:19.036+0000 7f564aa00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556829ed4670 0x556829ee9c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:19.032 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:19.036+0000 7f564b400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556829ed3cc0 0x556829ee96d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:46518/0 (socket says 172.21.15.90:46518) 2024-04-02T14:48:19.033 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:19.036+0000 7f564b400640 1 -- 172.21.15.90:0/990103930 learned_addr learned my addr 172.21.15.90:0/990103930 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:48:19.033 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:19.036+0000 7f564b400640 1 -- 172.21.15.90:0/990103930 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556829ee5bc0 msgr2=0x556829eea150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:48:19.033 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:19.036+0000 7f564b400640 1 --2- 172.21.15.90:0/990103930 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556829ee5bc0 0x556829eea150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:19.033 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:19.036+0000 7f564b400640 1 -- 172.21.15.90:0/990103930 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556829ed4670 msgr2=0x556829ee9c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:48:19.033 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:19.036+0000 7f564b400640 1 --2- 172.21.15.90:0/990103930 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556829ed4670 0x556829ee9c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:19.033 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:19.036+0000 7f564b400640 1 -- 172.21.15.90:0/990103930 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556829c19620 con 0x556829ed3cc0 2024-04-02T14:48:19.033 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:19.036+0000 7f564aa00640 1 --2- 172.21.15.90:0/990103930 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556829ed4670 0x556829ee9c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:48:19.033 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:19.037+0000 7f564b400640 1 --2- 172.21.15.90:0/990103930 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556829ed3cc0 0x556829ee96d0 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f563c00ad60 tx=0x7f563c006e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=57ab8c99a08a9165 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:48:19.033 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:19.037+0000 7f5639600640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:48:19.033 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:19.037+0000 7f5639600640 1 -- 172.21.15.90:0/990103930 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f563c034060 con 0x556829ed3cc0 2024-04-02T14:48:19.033 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:19.037+0000 7f5639600640 1 -- 172.21.15.90:0/990103930 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 8594+0+0 (secure 0 0 0) 0x7f563c0113f0 con 0x556829ed3cc0 2024-04-02T14:48:19.033 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:19.037+0000 7f5655439740 1 -- 172.21.15.90:0/990103930 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f563c0118d0 con 0x556829ed3cc0 2024-04-02T14:48:19.033 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:19.037+0000 7f5639600640 1 -- 172.21.15.90:0/990103930 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f563c015050 con 0x556829ed3cc0 2024-04-02T14:48:19.034 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:19.037+0000 7f5655439740 10 client.29395 fetch_fsmap learned FSMap version 27 2024-04-02T14:48:19.034 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:19.037+0000 7f5655439740 1 -- 172.21.15.90:0/990103930 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x556829f5c5d0 con 0x556829ed3cc0 2024-04-02T14:48:19.034 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:19.038+0000 7f5639600640 1 -- 172.21.15.90:0/990103930 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f563c0118d0 con 0x556829ed3cc0 2024-04-02T14:48:19.034 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:19.038+0000 7f5639600640 1 -- 172.21.15.90:0/990103930 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f563c0266c0 con 0x556829ed3cc0 2024-04-02T14:48:19.034 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:19.038+0000 7f5655439740 10 client.29395 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:48:19.034 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:19.038+0000 7f5655439740 10 client.29395 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-02T14:48:19.034 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:19.038+0000 7f5655439740 20 client.29395 populate_metadata read hostname 'smithi090' 2024-04-02T14:48:19.034 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:19.038+0000 7f5655439740 1 --2- 172.21.15.90:0/990103930 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x556829f5c800 0x556829f5ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:19.034 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:19.038+0000 7f5655439740 4 client.29395 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-02T14:48:19.034 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:19.038+0000 7f5655439740 1 -- 172.21.15.90:0/990103930 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "flush journal"}) -- 0x556829e462b0 con 0x556829f5c800 2024-04-02T14:48:19.034 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:19.038+0000 7f564aa00640 1 --2- 172.21.15.90:0/990103930 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x556829f5c800 0x556829f5ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:19.035 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:19.039+0000 7f564aa00640 1 --2- 172.21.15.90:0/990103930 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x556829f5c800 0x556829f5ebf0 crc :-1 s=READY pgs=343 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a457d6e3f45d7fac server_cookie=36f075e3edd67b8f in_seq=0 out_seq=0 2024-04-02T14:48:19.035 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:19.039+0000 7f5639600640 10 client.29395 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:48:19.053 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:19.059+0000 7f5639600640 1 -- 172.21.15.90:0/990103930 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x556829e462b0 con 0x556829f5c800 2024-04-02T14:48:19.053 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:19.059+0000 7f5639600640 10 client.29395 handle_command_reply: tid=0 2024-04-02T14:48:19.053 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:19.059+0000 7f5655439740 1 client.29395 shutdown 2024-04-02T14:48:19.053 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:19.059+0000 7f5655439740 1 -- 172.21.15.90:0/990103930 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556829ed3cc0 msgr2=0x556829ee96d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:48:19.053 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:19.059+0000 7f5655439740 1 --2- 172.21.15.90:0/990103930 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556829ed3cc0 0x556829ee96d0 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f563c00ad60 tx=0x7f563c006e30 comp rx=0 tx=0).stop 2024-04-02T14:48:19.053 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:19.059+0000 7f5655439740 1 -- 172.21.15.90:0/990103930 shutdown_connections 2024-04-02T14:48:19.054 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:19.059+0000 7f5655439740 1 --2- 172.21.15.90:0/990103930 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x556829f5c800 0x556829f5ebf0 crc :-1 s=READY pgs=343 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:19.054 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:19.059+0000 7f5655439740 1 --2- 172.21.15.90:0/990103930 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556829ee5bc0 0x556829eea150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:19.054 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:19.059+0000 7f5655439740 1 --2- 172.21.15.90:0/990103930 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556829ed4670 0x556829ee9c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:19.054 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:19.059+0000 7f5655439740 1 --2- 172.21.15.90:0/990103930 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556829ed3cc0 0x556829ee96d0 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:19.054 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:19.059+0000 7f5639600640 0 client.29395 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:48:19.054 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:19.059+0000 7f5655439740 1 -- 172.21.15.90:0/990103930 >> 172.21.15.90:0/990103930 conn(0x556829ece5e0 msgr2=0x556829eccdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:48:19.054 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:19.060+0000 7f5655439740 1 -- 172.21.15.90:0/990103930 shutdown_connections 2024-04-02T14:48:19.054 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:19.060+0000 7f5655439740 1 -- 172.21.15.90:0/990103930 wait complete. 2024-04-02T14:48:19.054 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:19.060+0000 7f5655439740 20 client.29395 trim_cache size 0 max 16384 2024-04-02T14:48:19.055 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:{ 2024-04-02T14:48:19.055 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "message": "", 2024-04-02T14:48:19.055 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "return_code": 0 2024-04-02T14:48:19.055 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:} 2024-04-02T14:48:19.055 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:19.061+0000 7f5652600640 1 -- 172.21.15.90:0/25398519 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f56300a3ec0 msgr2=0x7f56300a6380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:48:19.056 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:19.061+0000 7f5652600640 1 --2- 172.21.15.90:0/25398519 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f56300a3ec0 0x7f56300a6380 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f5644007f90 tx=0x7f5644004a40 comp rx=0 tx=0).stop 2024-04-02T14:48:19.056 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:19.061+0000 7f5652600640 1 -- 172.21.15.90:0/25398519 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f564c0747f0 msgr2=0x7f564c19aae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:48:19.056 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:19.061+0000 7f5652600640 1 --2- 172.21.15.90:0/25398519 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f564c0747f0 0x7f564c19aae0 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f563402c2d0 tx=0x7f563402c7a0 comp rx=0 tx=0).stop 2024-04-02T14:48:19.056 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:19.061+0000 7f5652600640 1 -- 172.21.15.90:0/25398519 shutdown_connections 2024-04-02T14:48:19.056 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:19.061+0000 7f5652600640 1 --2- 172.21.15.90:0/25398519 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f56300a3ec0 0x7f56300a6380 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:19.056 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:19.061+0000 7f5652600640 1 --2- 172.21.15.90:0/25398519 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f564c10e350 0x7f564c19ee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:19.056 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:19.061+0000 7f5652600640 1 --2- 172.21.15.90:0/25398519 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f564c0747f0 0x7f564c19aae0 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:19.056 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:19.061+0000 7f5652600640 1 --2- 172.21.15.90:0/25398519 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f564c073ed0 0x7f564c19a5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:19.056 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:19.061+0000 7f5652600640 1 -- 172.21.15.90:0/25398519 >> 172.21.15.90:0/25398519 conn(0x7f564c0fd5c0 msgr2=0x7f564c0fdae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:48:19.056 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:19.061+0000 7f5652600640 1 -- 172.21.15.90:0/25398519 shutdown_connections 2024-04-02T14:48:19.056 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:19.061+0000 7f5652600640 1 -- 172.21.15.90:0/25398519 wait complete. 2024-04-02T14:48:20.105 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:48:19 smithi092 ceph-mon[36497]: pgmap v419: 97 pgs: 97 active+clean; 4.2 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 6.7 MiB/s wr, 394 op/s 2024-04-02T14:48:20.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:48:19 smithi177 ceph-mon[36431]: pgmap v419: 97 pgs: 97 active+clean; 4.2 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 6.7 MiB/s wr, 394 op/s 2024-04-02T14:48:20.181 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:48:19 smithi090 ceph-mon[31674]: pgmap v419: 97 pgs: 97 active+clean; 4.2 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 6.7 MiB/s wr, 394 op/s 2024-04-02T14:48:21.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:48:21 smithi177 ceph-mon[36431]: osdmap e105: 12 total, 12 up, 12 in 2024-04-02T14:48:21.431 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:48:21 smithi090 ceph-mon[31674]: osdmap e105: 12 total, 12 up, 12 in 2024-04-02T14:48:21.603 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:48:21 smithi092 ceph-mon[36497]: osdmap e105: 12 total, 12 up, 12 in 2024-04-02T14:48:21.824 INFO:teuthology.orchestra.run.smithi090.stderr:progress: 770.0 s, 1443.0 tps, lat 68.688 ms stddev 83.299 2024-04-02T14:48:22.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:48:22 smithi177 ceph-mon[36431]: pgmap v421: 97 pgs: 97 active+clean; 4.2 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 9.6 MiB/s wr, 528 op/s 2024-04-02T14:48:22.431 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:48:22 smithi090 ceph-mon[31674]: pgmap v421: 97 pgs: 97 active+clean; 4.2 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 9.6 MiB/s wr, 528 op/s 2024-04-02T14:48:22.603 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:48:22 smithi092 ceph-mon[36497]: pgmap v421: 97 pgs: 97 active+clean; 4.2 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 9.6 MiB/s wr, 528 op/s 2024-04-02T14:48:22.971 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-02T14:48:22.971 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-02T14:48:23.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.271+0000 7fb022400640 1 -- 172.21.15.90:0/3423048554 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb01c0fff50 msgr2=0x7fb01c100330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:48:23.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.271+0000 7fb022400640 1 --2- 172.21.15.90:0/3423048554 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb01c0fff50 0x7fb01c100330 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7fb014005fd0 tx=0x7fb01402cc50 comp rx=0 tx=0).stop 2024-04-02T14:48:23.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.271+0000 7fb022400640 1 -- 172.21.15.90:0/3423048554 shutdown_connections 2024-04-02T14:48:23.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.271+0000 7fb022400640 1 --2- 172.21.15.90:0/3423048554 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb01c105020 0x7fb01c10cbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:23.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.271+0000 7fb022400640 1 --2- 172.21.15.90:0/3423048554 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb01c100900 0x7fb01c104940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:23.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.271+0000 7fb022400640 1 --2- 172.21.15.90:0/3423048554 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb01c0fff50 0x7fb01c100330 unknown :-1 s=CLOSED pgs=922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:23.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.271+0000 7fb022400640 1 -- 172.21.15.90:0/3423048554 >> 172.21.15.90:0/3423048554 conn(0x7fb01c076b50 msgr2=0x7fb01c0fdee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:48:23.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.271+0000 7fb022400640 1 -- 172.21.15.90:0/3423048554 shutdown_connections 2024-04-02T14:48:23.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.271+0000 7fb022400640 1 -- 172.21.15.90:0/3423048554 wait complete. 2024-04-02T14:48:23.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.273+0000 7fb022400640 1 Processor -- start 2024-04-02T14:48:23.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.273+0000 7fb022400640 1 -- start start 2024-04-02T14:48:23.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.275+0000 7fb022400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb01c0fff50 0x7fb01c073360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:23.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.275+0000 7fb022400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb01c105020 0x7fb01c06c430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:23.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.275+0000 7fb022400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb01c06c970 0x7fb01c06ce20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:23.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.275+0000 7fb022400640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fb01c10fcd0 con 0x7fb01c06c970 2024-04-02T14:48:23.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.275+0000 7fb022400640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fb01c10fb50 con 0x7fb01c0fff50 2024-04-02T14:48:23.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.275+0000 7fb022400640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fb01c10fe50 con 0x7fb01c105020 2024-04-02T14:48:23.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.275+0000 7fb01b400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb01c0fff50 0x7fb01c073360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:23.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.275+0000 7fb01b400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb01c0fff50 0x7fb01c073360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:37030/0 (socket says 172.21.15.90:37030) 2024-04-02T14:48:23.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.275+0000 7fb01b400640 1 -- 172.21.15.90:0/3246976606 learned_addr learned my addr 172.21.15.90:0/3246976606 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:48:23.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.275+0000 7fb01be00640 1 --2- 172.21.15.90:0/3246976606 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb01c06c970 0x7fb01c06ce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:23.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.276+0000 7fb01aa00640 1 --2- 172.21.15.90:0/3246976606 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb01c105020 0x7fb01c06c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:23.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.276+0000 7fb01b400640 1 -- 172.21.15.90:0/3246976606 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb01c105020 msgr2=0x7fb01c06c430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:48:23.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.276+0000 7fb01b400640 1 --2- 172.21.15.90:0/3246976606 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb01c105020 0x7fb01c06c430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:23.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.276+0000 7fb01b400640 1 -- 172.21.15.90:0/3246976606 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb01c06c970 msgr2=0x7fb01c06ce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:48:23.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.276+0000 7fb01b400640 1 --2- 172.21.15.90:0/3246976606 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb01c06c970 0x7fb01c06ce20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:23.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.276+0000 7fb01b400640 1 -- 172.21.15.90:0/3246976606 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb01c070e80 con 0x7fb01c0fff50 2024-04-02T14:48:23.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.276+0000 7fb01b400640 1 --2- 172.21.15.90:0/3246976606 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb01c0fff50 0x7fb01c073360 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7fb014005e30 tx=0x7fb014007d00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:48:23.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.277+0000 7fb013e00640 1 -- 172.21.15.90:0/3246976606 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb014044070 con 0x7fb01c0fff50 2024-04-02T14:48:23.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.277+0000 7fb013e00640 1 -- 172.21.15.90:0/3246976606 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb01400a4e0 con 0x7fb01c0fff50 2024-04-02T14:48:23.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.277+0000 7fb022400640 1 -- 172.21.15.90:0/3246976606 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb01c071110 con 0x7fb01c0fff50 2024-04-02T14:48:23.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.277+0000 7fb013e00640 1 -- 172.21.15.90:0/3246976606 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb014035950 con 0x7fb01c0fff50 2024-04-02T14:48:23.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.277+0000 7fb022400640 1 -- 172.21.15.90:0/3246976606 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb01c0716b0 con 0x7fb01c0fff50 2024-04-02T14:48:23.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.281+0000 7fb0251a0740 1 Processor -- start 2024-04-02T14:48:23.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.282+0000 7fb0251a0740 1 -- start start 2024-04-02T14:48:23.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.282+0000 7fb013e00640 1 -- 172.21.15.90:0/3246976606 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fb014047070 con 0x7fb01c0fff50 2024-04-02T14:48:23.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.282+0000 7fb0251a0740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f0a9224fc0 0x55f0a92253a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:23.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.282+0000 7fb0251a0740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f0a92258e0 0x55f0a922df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:23.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.282+0000 7fb0251a0740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f0a922e4c0 0x55f0a92308b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:23.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.282+0000 7fb0251a0740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55f0a9180770 con 0x55f0a922e4c0 2024-04-02T14:48:23.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.282+0000 7fb0251a0740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55f0a9128700 con 0x55f0a92258e0 2024-04-02T14:48:23.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.282+0000 7fb0251a0740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55f0a917c720 con 0x55f0a9224fc0 2024-04-02T14:48:23.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.283+0000 7fb01be00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f0a9224fc0 0x55f0a92253a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:23.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.283+0000 7fb01b400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f0a92258e0 0x55f0a922df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:23.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.283+0000 7fb01be00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f0a9224fc0 0x55f0a92253a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:46532/0 (socket says 172.21.15.90:46532) 2024-04-02T14:48:23.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.283+0000 7fb01be00640 1 -- 172.21.15.90:0/3559507768 learned_addr learned my addr 172.21.15.90:0/3559507768 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:48:23.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.283+0000 7fb01aa00640 1 --2- 172.21.15.90:0/3559507768 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f0a922e4c0 0x55f0a92308b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:23.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.283+0000 7fb01be00640 1 -- 172.21.15.90:0/3559507768 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f0a92258e0 msgr2=0x55f0a922df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:48:23.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.283+0000 7fb01be00640 1 --2- 172.21.15.90:0/3559507768 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f0a92258e0 0x55f0a922df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:23.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.283+0000 7fb01be00640 1 -- 172.21.15.90:0/3559507768 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f0a922e4c0 msgr2=0x55f0a92308b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:48:23.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.283+0000 7fb01be00640 1 --2- 172.21.15.90:0/3559507768 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f0a922e4c0 0x55f0a92308b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:23.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.283+0000 7fb01be00640 1 -- 172.21.15.90:0/3559507768 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f0a8f5e7d0 con 0x55f0a9224fc0 2024-04-02T14:48:23.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.284+0000 7fb01aa00640 1 --2- 172.21.15.90:0/3559507768 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f0a922e4c0 0x55f0a92308b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:48:23.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.284+0000 7fb01be00640 1 --2- 172.21.15.90:0/3559507768 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f0a9224fc0 0x55f0a92253a0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7fb00c00bc70 tx=0x7fb00c00f730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5e69a2e2d8e4a60f server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:48:23.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.284+0000 7fb010c00640 1 -- 172.21.15.90:0/3559507768 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb00c015070 con 0x55f0a9224fc0 2024-04-02T14:48:23.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.284+0000 7fb010c00640 1 -- 172.21.15.90:0/3559507768 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb00c01a940 con 0x55f0a9224fc0 2024-04-02T14:48:23.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.284+0000 7fb010c00640 1 -- 172.21.15.90:0/3559507768 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb00c0119d0 con 0x55f0a9224fc0 2024-04-02T14:48:23.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.284+0000 7fb013e00640 1 --2- 172.21.15.90:0/3246976606 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fb0000a3da0 0x7fb0000a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:23.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.285+0000 7fb01b400640 1 --2- 172.21.15.90:0/3246976606 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fb0000a3da0 0x7fb0000a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:23.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.285+0000 7fb013e00640 1 -- 172.21.15.90:0/3246976606 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 8550+0+0 (secure 0 0 0) 0x7fb0140f1ce0 con 0x7fb01c0fff50 2024-04-02T14:48:23.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.285+0000 7fb0251a0740 1 -- 172.21.15.90:0/3559507768 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f0a9224fc0 msgr2=0x55f0a92253a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:48:23.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.285+0000 7fb0251a0740 1 --2- 172.21.15.90:0/3559507768 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f0a9224fc0 0x55f0a92253a0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7fb00c00bc70 tx=0x7fb00c00f730 comp rx=0 tx=0).stop 2024-04-02T14:48:23.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.285+0000 7fb0251a0740 1 -- 172.21.15.90:0/3559507768 shutdown_connections 2024-04-02T14:48:23.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.285+0000 7fb0251a0740 1 --2- 172.21.15.90:0/3559507768 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f0a922e4c0 0x55f0a92308b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:23.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.285+0000 7fb0251a0740 1 --2- 172.21.15.90:0/3559507768 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f0a92258e0 0x55f0a922df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:23.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.285+0000 7fb0251a0740 1 --2- 172.21.15.90:0/3559507768 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f0a9224fc0 0x55f0a92253a0 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:23.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.285+0000 7fb0251a0740 1 -- 172.21.15.90:0/3559507768 >> 172.21.15.90:0/3559507768 conn(0x55f0a921f6f0 msgr2=0x55f0a921fad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:48:23.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.286+0000 7fb0251a0740 1 -- 172.21.15.90:0/3559507768 shutdown_connections 2024-04-02T14:48:23.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.286+0000 7fb0251a0740 1 -- 172.21.15.90:0/3559507768 wait complete. 2024-04-02T14:48:23.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.286+0000 7fb01b400640 1 --2- 172.21.15.90:0/3246976606 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fb0000a3da0 0x7fb0000a6260 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x55f0a9224a90 tx=0x7fb0140f5790 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:48:23.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.286+0000 7fb0251a0740 1 -- start start 2024-04-02T14:48:23.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.286+0000 7fb0251a0740 1 Processor -- start 2024-04-02T14:48:23.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.287+0000 7fb0251a0740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f0a9224fc0 0x55f0a922b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:23.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.287+0000 7fb0251a0740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f0a92258e0 0x55f0a922ba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:23.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.287+0000 7fb0251a0740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f0a922e4c0 0x55f0a922bf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:23.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.287+0000 7fb0251a0740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55f0a92312b0 con 0x55f0a92258e0 2024-04-02T14:48:23.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.287+0000 7fb0251a0740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55f0a913bac0 con 0x55f0a922e4c0 2024-04-02T14:48:23.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.287+0000 7fb0251a0740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55f0a9231430 con 0x55f0a9224fc0 2024-04-02T14:48:23.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.287+0000 7fb0251a0740 10 client.0 _ll_register_callbacks cb 0x55f0a913e180 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:48:23.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.287+0000 7fb01aa00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f0a92258e0 0x55f0a922ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:23.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.287+0000 7fb01aa00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f0a92258e0 0x55f0a922ba30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41676/0 (socket says 172.21.15.90:41676) 2024-04-02T14:48:23.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.287+0000 7fb01aa00640 1 -- 172.21.15.90:0/1826479964 learned_addr learned my addr 172.21.15.90:0/1826479964 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:48:23.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.288+0000 7fb01be00640 1 --2- 172.21.15.90:0/1826479964 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f0a9224fc0 0x55f0a922b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:23.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.288+0000 7fb01be00640 1 --2- 172.21.15.90:0/1826479964 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f0a922e4c0 0x55f0a922bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:23.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.288+0000 7fb01aa00640 1 -- 172.21.15.90:0/1826479964 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f0a9224fc0 msgr2=0x55f0a922b4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:48:23.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.288+0000 7fb01aa00640 1 --2- 172.21.15.90:0/1826479964 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f0a9224fc0 0x55f0a922b4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:23.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.288+0000 7fb01aa00640 1 -- 172.21.15.90:0/1826479964 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f0a922e4c0 msgr2=0x55f0a922bf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:48:23.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.288+0000 7fb01aa00640 1 --2- 172.21.15.90:0/1826479964 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f0a922e4c0 0x55f0a922bf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:23.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.288+0000 7fb01aa00640 1 -- 172.21.15.90:0/1826479964 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f0a8e943f0 con 0x55f0a92258e0 2024-04-02T14:48:23.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.288+0000 7fb01be00640 1 --2- 172.21.15.90:0/1826479964 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f0a922e4c0 0x55f0a922bf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:48:23.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.288+0000 7fb01aa00640 1 --2- 172.21.15.90:0/1826479964 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f0a92258e0 0x55f0a922ba30 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fb00800d220 tx=0x7fb00800d6e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd36c5bcb907aee9 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:48:23.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.288+0000 7fb005600640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-02T14:48:23.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.289+0000 7fb005600640 1 -- 172.21.15.90:0/1826479964 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb008014790 con 0x55f0a92258e0 2024-04-02T14:48:23.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.289+0000 7fb005600640 1 -- 172.21.15.90:0/1826479964 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 8550+0+0 (secure 0 0 0) 0x7fb008014930 con 0x55f0a92258e0 2024-04-02T14:48:23.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.289+0000 7fb0251a0740 1 -- 172.21.15.90:0/1826479964 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb00c01a940 con 0x55f0a92258e0 2024-04-02T14:48:23.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.289+0000 7fb005600640 1 -- 172.21.15.90:0/1826479964 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fb008014b30 con 0x55f0a92258e0 2024-04-02T14:48:23.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.289+0000 7fb0251a0740 10 client.18462 fetch_fsmap learned FSMap version 27 2024-04-02T14:48:23.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.290+0000 7fb0251a0740 1 -- 172.21.15.90:0/1826479964 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55f0a8e540f0 con 0x55f0a92258e0 2024-04-02T14:48:23.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.291+0000 7fb005600640 1 -- 172.21.15.90:0/1826479964 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb00c01a940 con 0x55f0a92258e0 2024-04-02T14:48:23.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.291+0000 7fb005600640 1 -- 172.21.15.90:0/1826479964 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fb00801d350 con 0x55f0a92258e0 2024-04-02T14:48:23.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.291+0000 7fb0251a0740 10 client.18462 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:48:23.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.291+0000 7fb0251a0740 10 client.18462 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:48:23.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.291+0000 7fb0251a0740 20 client.18462 populate_metadata read hostname 'smithi090' 2024-04-02T14:48:23.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.291+0000 7fb0251a0740 1 --2- 172.21.15.90:0/1826479964 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55f0a923e550 0x55f0a9240970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:23.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.291+0000 7fb0251a0740 4 client.18462 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:48:23.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.291+0000 7fb0251a0740 1 -- 172.21.15.90:0/1826479964 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f0a9121630 con 0x55f0a923e550 2024-04-02T14:48:23.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.291+0000 7fb01be00640 1 --2- 172.21.15.90:0/1826479964 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55f0a923e550 0x55f0a9240970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:23.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.292+0000 7fb01be00640 1 --2- 172.21.15.90:0/1826479964 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55f0a923e550 0x55f0a9240970 crc :-1 s=READY pgs=344 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e9486ed5ffab4d49 server_cookie=713ff97e792b967a in_seq=0 out_seq=0 2024-04-02T14:48:23.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.292+0000 7fb005600640 10 client.18462 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:48:23.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.293+0000 7fb005600640 1 -- 172.21.15.90:0/1826479964 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f0a9121630 con 0x55f0a923e550 2024-04-02T14:48:23.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.293+0000 7fb005600640 10 client.18462 handle_command_reply: tid=0 2024-04-02T14:48:23.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.293+0000 7fb0251a0740 1 client.18462 shutdown 2024-04-02T14:48:23.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.293+0000 7fb0251a0740 1 -- 172.21.15.90:0/1826479964 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f0a92258e0 msgr2=0x55f0a922ba30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:48:23.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.293+0000 7fb0251a0740 1 --2- 172.21.15.90:0/1826479964 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f0a92258e0 0x55f0a922ba30 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fb00800d220 tx=0x7fb00800d6e0 comp rx=0 tx=0).stop 2024-04-02T14:48:23.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.293+0000 7fb0251a0740 1 -- 172.21.15.90:0/1826479964 shutdown_connections 2024-04-02T14:48:23.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.293+0000 7fb0251a0740 1 --2- 172.21.15.90:0/1826479964 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55f0a923e550 0x55f0a9240970 crc :-1 s=READY pgs=344 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:23.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.293+0000 7fb0251a0740 1 --2- 172.21.15.90:0/1826479964 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f0a922e4c0 0x55f0a922bf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:23.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.293+0000 7fb005600640 0 client.18462 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:48:23.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.293+0000 7fb0251a0740 1 --2- 172.21.15.90:0/1826479964 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f0a92258e0 0x55f0a922ba30 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:23.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.293+0000 7fb0251a0740 1 --2- 172.21.15.90:0/1826479964 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f0a9224fc0 0x55f0a922b4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:23.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.293+0000 7fb0251a0740 1 -- 172.21.15.90:0/1826479964 >> 172.21.15.90:0/1826479964 conn(0x55f0a921f6f0 msgr2=0x55f0a9221440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:48:23.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.294+0000 7fb0251a0740 1 -- 172.21.15.90:0/1826479964 shutdown_connections 2024-04-02T14:48:23.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.294+0000 7fb0251a0740 1 -- 172.21.15.90:0/1826479964 wait complete. 2024-04-02T14:48:23.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.294+0000 7fb0251a0740 20 client.18462 trim_cache size 0 max 16384 2024-04-02T14:48:23.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.322+0000 7fb0251a0740 1 Processor -- start 2024-04-02T14:48:23.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.322+0000 7fb0251a0740 1 -- start start 2024-04-02T14:48:23.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.323+0000 7fb0251a0740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f0a9224fc0 0x55f0a922dc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:23.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.323+0000 7fb0251a0740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f0a92258e0 0x55f0a923a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:23.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.323+0000 7fb0251a0740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f0a922e4c0 0x55f0a922a450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:23.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.323+0000 7fb0251a0740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fb01c1b3f90 con 0x55f0a92258e0 2024-04-02T14:48:23.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.323+0000 7fb0251a0740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fb01c1b5ad0 con 0x55f0a9224fc0 2024-04-02T14:48:23.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.323+0000 7fb0251a0740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fb01c1b4660 con 0x55f0a922e4c0 2024-04-02T14:48:23.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.323+0000 7fb01be00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f0a9224fc0 0x55f0a922dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:23.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.323+0000 7fb01be00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f0a9224fc0 0x55f0a922dc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:37064/0 (socket says 172.21.15.90:37064) 2024-04-02T14:48:23.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.323+0000 7fb01be00640 1 -- 172.21.15.90:0/3990289848 learned_addr learned my addr 172.21.15.90:0/3990289848 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:48:23.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.323+0000 7fb01be00640 1 --2- 172.21.15.90:0/3990289848 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f0a922e4c0 0x55f0a922a450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:23.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.323+0000 7fb01be00640 1 -- 172.21.15.90:0/3990289848 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f0a922e4c0 msgr2=0x55f0a922a450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:48:23.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.323+0000 7fb01be00640 1 --2- 172.21.15.90:0/3990289848 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f0a922e4c0 0x55f0a922a450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:23.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.323+0000 7fb01be00640 1 -- 172.21.15.90:0/3990289848 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f0a92258e0 msgr2=0x55f0a923a1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:48:23.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.323+0000 7fb01aa00640 1 --2- 172.21.15.90:0/3990289848 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f0a92258e0 0x55f0a923a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:23.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.323+0000 7fb01be00640 1 --2- 172.21.15.90:0/3990289848 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f0a92258e0 0x55f0a923a1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:23.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.323+0000 7fb01be00640 1 -- 172.21.15.90:0/3990289848 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f0a92308b0 con 0x55f0a9224fc0 2024-04-02T14:48:23.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.323+0000 7fb01aa00640 1 --2- 172.21.15.90:0/3990289848 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f0a92258e0 0x55f0a923a1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:48:23.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.324+0000 7fb01be00640 1 --2- 172.21.15.90:0/3990289848 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f0a9224fc0 0x55f0a922dc20 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7fb00c005fd0 tx=0x7fb00c011550 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3021b01326d75329 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:48:23.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.324+0000 7fb007e00640 1 -- 172.21.15.90:0/3990289848 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb00c015040 con 0x55f0a9224fc0 2024-04-02T14:48:23.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.324+0000 7fb007e00640 1 -- 172.21.15.90:0/3990289848 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb00c005520 con 0x55f0a9224fc0 2024-04-02T14:48:23.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.324+0000 7fb007e00640 1 -- 172.21.15.90:0/3990289848 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb00c011700 con 0x55f0a9224fc0 2024-04-02T14:48:23.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.325+0000 7fb0251a0740 1 -- 172.21.15.90:0/3990289848 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f0a9224fc0 msgr2=0x55f0a922dc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:48:23.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.325+0000 7fb0251a0740 1 --2- 172.21.15.90:0/3990289848 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f0a9224fc0 0x55f0a922dc20 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7fb00c005fd0 tx=0x7fb00c011550 comp rx=0 tx=0).stop 2024-04-02T14:48:23.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.325+0000 7fb0251a0740 1 -- 172.21.15.90:0/3990289848 shutdown_connections 2024-04-02T14:48:23.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.325+0000 7fb0251a0740 1 --2- 172.21.15.90:0/3990289848 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f0a922e4c0 0x55f0a922a450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:23.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.325+0000 7fb0251a0740 1 --2- 172.21.15.90:0/3990289848 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f0a92258e0 0x55f0a923a1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:23.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.325+0000 7fb0251a0740 1 --2- 172.21.15.90:0/3990289848 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f0a9224fc0 0x55f0a922dc20 unknown :-1 s=CLOSED pgs=924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:23.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.325+0000 7fb0251a0740 1 -- 172.21.15.90:0/3990289848 >> 172.21.15.90:0/3990289848 conn(0x55f0a921f6f0 msgr2=0x55f0a9215f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:48:23.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.325+0000 7fb0251a0740 1 -- 172.21.15.90:0/3990289848 shutdown_connections 2024-04-02T14:48:23.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.325+0000 7fb0251a0740 1 -- 172.21.15.90:0/3990289848 wait complete. 2024-04-02T14:48:23.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.325+0000 7fb0251a0740 1 -- start start 2024-04-02T14:48:23.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.325+0000 7fb0251a0740 1 Processor -- start 2024-04-02T14:48:23.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.326+0000 7fb0251a0740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f0a9224fc0 0x55f0a92284d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:23.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.326+0000 7fb0251a0740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f0a92258e0 0x55f0a9228a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:23.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.326+0000 7fb0251a0740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f0a922e4c0 0x55f0a9228f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:23.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.326+0000 7fb0251a0740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fb01c1b5400 con 0x55f0a922e4c0 2024-04-02T14:48:23.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.326+0000 7fb0251a0740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fb01c1b4d30 con 0x55f0a92258e0 2024-04-02T14:48:23.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.326+0000 7fb0251a0740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fb01c1b1f90 con 0x55f0a9224fc0 2024-04-02T14:48:23.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.326+0000 7fb0251a0740 10 client.0 _ll_register_callbacks cb 0x7fb00800b140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:48:23.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.326+0000 7fb01aa00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f0a92258e0 0x55f0a9228a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:23.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.326+0000 7fb01aa00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f0a92258e0 0x55f0a9228a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:37078/0 (socket says 172.21.15.90:37078) 2024-04-02T14:48:23.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.326+0000 7fb01aa00640 1 -- 172.21.15.90:0/4113570073 learned_addr learned my addr 172.21.15.90:0/4113570073 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:48:23.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.326+0000 7fb01be00640 1 --2- 172.21.15.90:0/4113570073 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f0a9224fc0 0x55f0a92284d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:23.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.326+0000 7fb01be00640 1 --2- 172.21.15.90:0/4113570073 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f0a922e4c0 0x55f0a9228f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:23.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.326+0000 7fb01aa00640 1 -- 172.21.15.90:0/4113570073 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f0a9224fc0 msgr2=0x55f0a92284d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:48:23.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.326+0000 7fb01aa00640 1 --2- 172.21.15.90:0/4113570073 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f0a9224fc0 0x55f0a92284d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:23.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.326+0000 7fb01aa00640 1 -- 172.21.15.90:0/4113570073 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f0a922e4c0 msgr2=0x55f0a9228f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:48:23.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.326+0000 7fb01aa00640 1 --2- 172.21.15.90:0/4113570073 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f0a922e4c0 0x55f0a9228f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:23.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.326+0000 7fb01aa00640 1 -- 172.21.15.90:0/4113570073 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f0a92a3d80 con 0x55f0a92258e0 2024-04-02T14:48:23.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.327+0000 7fb01be00640 1 --2- 172.21.15.90:0/4113570073 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f0a922e4c0 0x55f0a9228f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:48:23.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.327+0000 7fb01aa00640 1 --2- 172.21.15.90:0/4113570073 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f0a92258e0 0x55f0a9228a10 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x55f0a922cac0 tx=0x7fb00800bc70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9c0fdf6bbbed44d8 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:48:23.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.327+0000 7fb005600640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:48:23.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.327+0000 7fb005600640 1 -- 172.21.15.90:0/4113570073 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb00801d550 con 0x55f0a92258e0 2024-04-02T14:48:23.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.328+0000 7fb0251a0740 1 -- 172.21.15.90:0/4113570073 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb00c005520 con 0x55f0a92258e0 2024-04-02T14:48:23.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.328+0000 7fb005600640 1 -- 172.21.15.90:0/4113570073 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 8550+0+0 (secure 0 0 0) 0x7fb00801d9e0 con 0x55f0a92258e0 2024-04-02T14:48:23.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.328+0000 7fb005600640 1 -- 172.21.15.90:0/4113570073 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fb00801dc00 con 0x55f0a92258e0 2024-04-02T14:48:23.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.328+0000 7fb0251a0740 10 client.29633 fetch_fsmap learned FSMap version 27 2024-04-02T14:48:23.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.328+0000 7fb0251a0740 1 -- 172.21.15.90:0/4113570073 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55f0a92a5000 con 0x55f0a92258e0 2024-04-02T14:48:23.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.328+0000 7fb005600640 1 -- 172.21.15.90:0/4113570073 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb00c005520 con 0x55f0a92258e0 2024-04-02T14:48:23.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.329+0000 7fb005600640 1 -- 172.21.15.90:0/4113570073 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fb008028b80 con 0x55f0a92258e0 2024-04-02T14:48:23.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.329+0000 7fb0251a0740 10 client.29633 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:48:23.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.329+0000 7fb0251a0740 10 client.29633 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:48:23.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.329+0000 7fb0251a0740 20 client.29633 populate_metadata read hostname 'smithi090' 2024-04-02T14:48:23.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.329+0000 7fb0251a0740 1 --2- 172.21.15.90:0/4113570073 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55f0a92a57b0 0x55f0a92a7bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:23.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.329+0000 7fb0251a0740 4 client.29633 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-02T14:48:23.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.329+0000 7fb0251a0740 1 -- 172.21.15.90:0/4113570073 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55f0a918dcb0 con 0x55f0a92a57b0 2024-04-02T14:48:23.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.329+0000 7fb01be00640 1 --2- 172.21.15.90:0/4113570073 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55f0a92a57b0 0x55f0a92a7bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:23.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.329+0000 7fb01be00640 1 --2- 172.21.15.90:0/4113570073 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55f0a92a57b0 0x55f0a92a7bd0 crc :-1 s=READY pgs=345 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f186062465e63dee server_cookie=ba6a7dea613af6b in_seq=0 out_seq=0 2024-04-02T14:48:23.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.330+0000 7fb005600640 10 client.29633 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:48:23.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.330+0000 7fb005600640 1 -- 172.21.15.90:0/4113570073 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55f0a918dcb0 con 0x55f0a92a57b0 2024-04-02T14:48:23.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.330+0000 7fb005600640 10 client.29633 handle_command_reply: tid=0 2024-04-02T14:48:23.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.330+0000 7fb0251a0740 1 client.29633 shutdown 2024-04-02T14:48:23.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.330+0000 7fb0251a0740 1 -- 172.21.15.90:0/4113570073 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f0a92258e0 msgr2=0x55f0a9228a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:48:23.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.330+0000 7fb0251a0740 1 --2- 172.21.15.90:0/4113570073 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f0a92258e0 0x55f0a9228a10 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x55f0a922cac0 tx=0x7fb00800bc70 comp rx=0 tx=0).stop 2024-04-02T14:48:23.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.330+0000 7fb0251a0740 1 -- 172.21.15.90:0/4113570073 shutdown_connections 2024-04-02T14:48:23.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.330+0000 7fb0251a0740 1 --2- 172.21.15.90:0/4113570073 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55f0a92a57b0 0x55f0a92a7bd0 crc :-1 s=READY pgs=345 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:23.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.330+0000 7fb0251a0740 1 --2- 172.21.15.90:0/4113570073 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f0a922e4c0 0x55f0a9228f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:23.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.330+0000 7fb0251a0740 1 --2- 172.21.15.90:0/4113570073 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f0a92258e0 0x55f0a9228a10 unknown :-1 s=CLOSED pgs=925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:23.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.330+0000 7fb0251a0740 1 --2- 172.21.15.90:0/4113570073 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f0a9224fc0 0x55f0a92284d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:23.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.330+0000 7fb005600640 0 client.29633 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:48:23.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.330+0000 7fb0251a0740 1 -- 172.21.15.90:0/4113570073 >> 172.21.15.90:0/4113570073 conn(0x55f0a921f6f0 msgr2=0x55f0a922b2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:48:23.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.330+0000 7fb0251a0740 1 -- 172.21.15.90:0/4113570073 shutdown_connections 2024-04-02T14:48:23.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.330+0000 7fb0251a0740 1 -- 172.21.15.90:0/4113570073 wait complete. 2024-04-02T14:48:23.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.330+0000 7fb0251a0740 20 client.29633 trim_cache size 0 max 16384 2024-04-02T14:48:23.328 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-02T14:48:23.328 INFO:teuthology.orchestra.run.smithi090.stdout: "status": "no active scrubs running", 2024-04-02T14:48:23.328 INFO:teuthology.orchestra.run.smithi090.stdout: "scrubs": {} 2024-04-02T14:48:23.328 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-02T14:48:23.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.331+0000 7fb022400640 1 -- 172.21.15.90:0/3246976606 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fb0000a3da0 msgr2=0x7fb0000a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:48:23.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.331+0000 7fb022400640 1 --2- 172.21.15.90:0/3246976606 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fb0000a3da0 0x7fb0000a6260 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x55f0a9224a90 tx=0x7fb0140f5790 comp rx=0 tx=0).stop 2024-04-02T14:48:23.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.331+0000 7fb022400640 1 -- 172.21.15.90:0/3246976606 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb01c0fff50 msgr2=0x7fb01c073360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:48:23.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.331+0000 7fb022400640 1 --2- 172.21.15.90:0/3246976606 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb01c0fff50 0x7fb01c073360 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7fb014005e30 tx=0x7fb014007d00 comp rx=0 tx=0).stop 2024-04-02T14:48:23.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.331+0000 7fb022400640 1 -- 172.21.15.90:0/3246976606 shutdown_connections 2024-04-02T14:48:23.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.331+0000 7fb022400640 1 --2- 172.21.15.90:0/3246976606 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fb0000a3da0 0x7fb0000a6260 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:23.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.331+0000 7fb022400640 1 --2- 172.21.15.90:0/3246976606 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb01c06c970 0x7fb01c06ce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:23.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.331+0000 7fb022400640 1 --2- 172.21.15.90:0/3246976606 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb01c105020 0x7fb01c06c430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:23.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.331+0000 7fb022400640 1 --2- 172.21.15.90:0/3246976606 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb01c0fff50 0x7fb01c073360 unknown :-1 s=CLOSED pgs=923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:23.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.331+0000 7fb022400640 1 -- 172.21.15.90:0/3246976606 >> 172.21.15.90:0/3246976606 conn(0x7fb01c076b50 msgr2=0x7fb01c103260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:48:23.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.331+0000 7fb022400640 1 -- 172.21.15.90:0/3246976606 shutdown_connections 2024-04-02T14:48:23.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.331+0000 7fb022400640 1 -- 172.21.15.90:0/3246976606 wait complete. 2024-04-02T14:48:23.344 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-02T14:48:23.344 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T14:48:23.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.594+0000 7f2edce00640 1 -- 172.21.15.90:0/2763785453 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2ed8108320 msgr2=0x7f2ed8108700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:48:23.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.594+0000 7f2edce00640 1 --2- 172.21.15.90:0/2763785453 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2ed8108320 0x7f2ed8108700 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f2ec8009900 tx=0x7f2ec802bc50 comp rx=0 tx=0).stop 2024-04-02T14:48:23.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.594+0000 7f2edce00640 1 -- 172.21.15.90:0/2763785453 shutdown_connections 2024-04-02T14:48:23.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.594+0000 7f2edce00640 1 --2- 172.21.15.90:0/2763785453 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2ed810d3f0 0x7f2ed8114fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:23.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.594+0000 7f2edce00640 1 --2- 172.21.15.90:0/2763785453 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2ed8108cd0 0x7f2ed810cd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:23.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.594+0000 7f2edce00640 1 --2- 172.21.15.90:0/2763785453 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2ed8108320 0x7f2ed8108700 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:23.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.594+0000 7f2edce00640 1 -- 172.21.15.90:0/2763785453 >> 172.21.15.90:0/2763785453 conn(0x7f2ed8076050 msgr2=0x7f2ed8078470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:48:23.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.594+0000 7f2edce00640 1 -- 172.21.15.90:0/2763785453 shutdown_connections 2024-04-02T14:48:23.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.595+0000 7f2edce00640 1 -- 172.21.15.90:0/2763785453 wait complete. 2024-04-02T14:48:23.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.596+0000 7f2edce00640 1 Processor -- start 2024-04-02T14:48:23.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.596+0000 7f2edce00640 1 -- start start 2024-04-02T14:48:23.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.597+0000 7f2edce00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2ed8108320 0x7f2ed81a47b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:23.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.597+0000 7f2edce00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2ed810d3f0 0x7f2ed81a4cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:23.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.597+0000 7f2edce00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2ed819e960 0x7f2ed819ee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:23.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.597+0000 7f2edce00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f2ed81184a0 con 0x7f2ed810d3f0 2024-04-02T14:48:23.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.597+0000 7f2edce00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f2ed8118320 con 0x7f2ed8108320 2024-04-02T14:48:23.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.597+0000 7f2edce00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f2ed8118620 con 0x7f2ed819e960 2024-04-02T14:48:23.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.597+0000 7f2ed6800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2ed819e960 0x7f2ed819ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:23.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.597+0000 7f2ed6800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2ed819e960 0x7f2ed819ee10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:46580/0 (socket says 172.21.15.90:46580) 2024-04-02T14:48:23.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.597+0000 7f2ed6800640 1 -- 172.21.15.90:0/816800392 learned_addr learned my addr 172.21.15.90:0/816800392 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:48:23.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.598+0000 7f2ed5400640 1 --2- 172.21.15.90:0/816800392 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2ed810d3f0 0x7f2ed81a4cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:23.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.598+0000 7f2ed6800640 1 -- 172.21.15.90:0/816800392 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2ed8108320 msgr2=0x7f2ed81a47b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:48:23.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.598+0000 7f2ed5e00640 1 --2- 172.21.15.90:0/816800392 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2ed8108320 0x7f2ed81a47b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:23.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.598+0000 7f2ed6800640 1 --2- 172.21.15.90:0/816800392 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2ed8108320 0x7f2ed81a47b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:23.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.598+0000 7f2ed6800640 1 -- 172.21.15.90:0/816800392 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2ed810d3f0 msgr2=0x7f2ed81a4cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:48:23.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.598+0000 7f2ed6800640 1 --2- 172.21.15.90:0/816800392 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2ed810d3f0 0x7f2ed81a4cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:23.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.598+0000 7f2ed6800640 1 -- 172.21.15.90:0/816800392 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ed819f480 con 0x7f2ed819e960 2024-04-02T14:48:23.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.598+0000 7f2ed5400640 1 --2- 172.21.15.90:0/816800392 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2ed810d3f0 0x7f2ed81a4cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:48:23.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.598+0000 7f2ed5e00640 1 --2- 172.21.15.90:0/816800392 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2ed8108320 0x7f2ed81a47b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:48:23.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.598+0000 7f2ed6800640 1 --2- 172.21.15.90:0/816800392 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2ed819e960 0x7f2ed819ee10 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f2ed000efd0 tx=0x7f2ed000c6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:48:23.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.598+0000 7f2ecea00640 1 -- 172.21.15.90:0/816800392 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ed0003b70 con 0x7f2ed819e960 2024-04-02T14:48:23.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.598+0000 7f2ecea00640 1 -- 172.21.15.90:0/816800392 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2ed0003d10 con 0x7f2ed819e960 2024-04-02T14:48:23.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.598+0000 7f2ecea00640 1 -- 172.21.15.90:0/816800392 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ed00109f0 con 0x7f2ed819e960 2024-04-02T14:48:23.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.598+0000 7f2edce00640 1 -- 172.21.15.90:0/816800392 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ed819f770 con 0x7f2ed819e960 2024-04-02T14:48:23.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.599+0000 7f2edce00640 1 -- 172.21.15.90:0/816800392 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ed81ab910 con 0x7f2ed819e960 2024-04-02T14:48:23.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.600+0000 7f2edce00640 1 -- 172.21.15.90:0/816800392 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2ed8068d40 con 0x7f2ed819e960 2024-04-02T14:48:23.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.602+0000 7f2ecea00640 1 -- 172.21.15.90:0/816800392 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f2ed0004020 con 0x7f2ed819e960 2024-04-02T14:48:23.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.603+0000 7f2ecea00640 1 --2- 172.21.15.90:0/816800392 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f2ebc0a3d20 0x7f2ebc0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:23.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.604+0000 7f2ecea00640 1 -- 172.21.15.90:0/816800392 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 8550+0+0 (secure 0 0 0) 0x7f2ed00d0d60 con 0x7f2ed819e960 2024-04-02T14:48:23.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.604+0000 7f2ed5e00640 1 --2- 172.21.15.90:0/816800392 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f2ebc0a3d20 0x7f2ebc0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:23.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.604+0000 7f2ed5e00640 1 --2- 172.21.15.90:0/816800392 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f2ebc0a3d20 0x7f2ebc0a61e0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f2ec8036040 tx=0x7f2ec8000bc0 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:48:23.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:23.608+0000 7f2ecea00640 1 -- 172.21.15.90:0/816800392 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7f2ed009d3e0 con 0x7f2ed819e960 2024-04-02T14:48:23.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.001+0000 7f2edce00640 1 -- 172.21.15.90:0/816800392 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2ed8108cd0 con 0x7f2ed819e960 2024-04-02T14:48:23.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.002+0000 7f2ecea00640 1 -- 172.21.15.90:0/816800392 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11927 (secure 0 0 0) 0x7f2ed009cce0 con 0x7f2ed819e960 2024-04-02T14:48:23.996 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-02T14:48:23.996 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14412,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6837/2852160309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6836","nonce":2852160309},{"type":"v1","addr":"172.21.15.90:6837","nonce":2852160309}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14478,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6841/1919376858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6840","nonce":1919376858},{"type":"v1","addr":"172.21.15.90:6841","nonce":1919376858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-02T14:32:22.319083+0000","modified":"2024-04-02T14:34:52.266152+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24590,"mds_1":24560,"mds_2":24547,"mds_3":24539,"mds_4":24523},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24454":{"gid":24454,"name":"l","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6833/1564319431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":1564319431},{"type":"v1","addr":"172.21.15.177:6833","nonce":1564319431}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.92:6833/1570337372","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":1570337372},{"type":"v1","addr":"172.21.15.92:6833","nonce":1570337372}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"j","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.90:6835/1277124891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6834","nonce":1277124891},{"type":"v1","addr":"172.21.15.90:6835","nonce":1277124891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.92:6835/860721164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":860721164},{"type":"v1","addr":"172.21.15.92:6835","nonce":860721164}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24511":{"gid":24511,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6835/3177244988","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":3177244988},{"type":"v1","addr":"172.21.15.177:6835","nonce":3177244988}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"c","rank":4,"incarnation":25,"state":"up:active","state_seq":9,"addr":"172.21.15.177:6837/4223414858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":4223414858},{"type":"v1","addr":"172.21.15.177:6837","nonce":4223414858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24539":{"gid":24539,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":8,"addr":"172.21.15.92:6837/1270635939","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":1270635939},{"type":"v1","addr":"172.21.15.92:6837","nonce":1270635939}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24547":{"gid":24547,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":7,"addr":"172.21.15.177:6839/664895663","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":664895663},{"type":"v1","addr":"172.21.15.177:6839","nonce":664895663}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"g","rank":1,"incarnation":19,"state":"up:active","state_seq":5,"addr":"172.21.15.90:6839/1498915870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6838","nonce":1498915870},{"type":"v1","addr":"172.21.15.90:6839","nonce":1498915870}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24590":{"gid":24590,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.92:6839/1054888219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":1054888219},{"type":"v1","addr":"172.21.15.92:6839","nonce":1054888219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24590,"qdb_cluster":[24547,24590,24523,24539,24560]},"id":1}]} 2024-04-02T14:48:23.996 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 27 2024-04-02T14:48:23.998 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.005+0000 7f2edce00640 1 -- 172.21.15.90:0/816800392 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f2ebc0a3d20 msgr2=0x7f2ebc0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:48:23.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.005+0000 7f2edce00640 1 --2- 172.21.15.90:0/816800392 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f2ebc0a3d20 0x7f2ebc0a61e0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f2ec8036040 tx=0x7f2ec8000bc0 comp rx=0 tx=0).stop 2024-04-02T14:48:23.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.005+0000 7f2edce00640 1 -- 172.21.15.90:0/816800392 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2ed819e960 msgr2=0x7f2ed819ee10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:48:23.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.005+0000 7f2edce00640 1 --2- 172.21.15.90:0/816800392 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2ed819e960 0x7f2ed819ee10 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f2ed000efd0 tx=0x7f2ed000c6a0 comp rx=0 tx=0).stop 2024-04-02T14:48:23.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.005+0000 7f2edce00640 1 -- 172.21.15.90:0/816800392 shutdown_connections 2024-04-02T14:48:23.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.005+0000 7f2edce00640 1 --2- 172.21.15.90:0/816800392 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f2ebc0a3d20 0x7f2ebc0a61e0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:23.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.005+0000 7f2edce00640 1 --2- 172.21.15.90:0/816800392 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2ed819e960 0x7f2ed819ee10 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:23.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.005+0000 7f2edce00640 1 --2- 172.21.15.90:0/816800392 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2ed810d3f0 0x7f2ed81a4cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:23.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.005+0000 7f2edce00640 1 --2- 172.21.15.90:0/816800392 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2ed8108320 0x7f2ed81a47b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:23.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.005+0000 7f2edce00640 1 -- 172.21.15.90:0/816800392 >> 172.21.15.90:0/816800392 conn(0x7f2ed8076050 msgr2=0x7f2ed80783d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:48:24.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.006+0000 7f2edce00640 1 -- 172.21.15.90:0/816800392 shutdown_connections 2024-04-02T14:48:24.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.006+0000 7f2edce00640 1 -- 172.21.15.90:0/816800392 wait complete. 2024-04-02T14:48:24.021 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-04-02T14:48:24.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.318+0000 7f1e79c00640 1 -- 172.21.15.90:0/629481766 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1e74108310 msgr2=0x7f1e741086f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:48:24.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.318+0000 7f1e79c00640 1 --2- 172.21.15.90:0/629481766 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1e74108310 0x7f1e741086f0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f1e64009900 tx=0x7f1e6402bc40 comp rx=0 tx=0).stop 2024-04-02T14:48:24.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.318+0000 7f1e79c00640 1 -- 172.21.15.90:0/629481766 shutdown_connections 2024-04-02T14:48:24.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.318+0000 7f1e79c00640 1 --2- 172.21.15.90:0/629481766 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1e7410d3e0 0x7f1e74114fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:24.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.318+0000 7f1e79c00640 1 --2- 172.21.15.90:0/629481766 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1e74108cc0 0x7f1e7410cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:24.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.318+0000 7f1e79c00640 1 --2- 172.21.15.90:0/629481766 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1e74108310 0x7f1e741086f0 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:24.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.318+0000 7f1e79c00640 1 -- 172.21.15.90:0/629481766 >> 172.21.15.90:0/629481766 conn(0x7f1e74076040 msgr2=0x7f1e74078460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:48:24.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.318+0000 7f1e79c00640 1 -- 172.21.15.90:0/629481766 shutdown_connections 2024-04-02T14:48:24.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.319+0000 7f1e79c00640 1 -- 172.21.15.90:0/629481766 wait complete. 2024-04-02T14:48:24.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.320+0000 7f1e79c00640 1 Processor -- start 2024-04-02T14:48:24.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.320+0000 7f1e79c00640 1 -- start start 2024-04-02T14:48:24.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.321+0000 7f1e79c00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1e74108310 0x7f1e741a46b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:24.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.321+0000 7f1e79c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1e7410d3e0 0x7f1e741a4bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:24.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.321+0000 7f1e79c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1e7419e770 0x7f1e7419ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:24.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.321+0000 7f1e79c00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f1e74118470 con 0x7f1e7410d3e0 2024-04-02T14:48:24.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.321+0000 7f1e79c00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f1e741182f0 con 0x7f1e74108310 2024-04-02T14:48:24.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.321+0000 7f1e79c00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f1e741185f0 con 0x7f1e7419e770 2024-04-02T14:48:24.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.322+0000 7f1e72a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1e74108310 0x7f1e741a46b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:24.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.322+0000 7f1e73400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1e7419e770 0x7f1e7419ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:24.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.322+0000 7f1e72a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1e74108310 0x7f1e741a46b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:54212/0 (socket says 172.21.15.90:54212) 2024-04-02T14:48:24.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.322+0000 7f1e72a00640 1 -- 172.21.15.90:0/3847129613 learned_addr learned my addr 172.21.15.90:0/3847129613 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:48:24.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.322+0000 7f1e72000640 1 --2- 172.21.15.90:0/3847129613 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1e7410d3e0 0x7f1e741a4bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:24.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.322+0000 7f1e73400640 1 -- 172.21.15.90:0/3847129613 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1e74108310 msgr2=0x7f1e741a46b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:48:24.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.322+0000 7f1e73400640 1 --2- 172.21.15.90:0/3847129613 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1e74108310 0x7f1e741a46b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:24.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.322+0000 7f1e73400640 1 -- 172.21.15.90:0/3847129613 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1e7410d3e0 msgr2=0x7f1e741a4bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:48:24.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.322+0000 7f1e73400640 1 --2- 172.21.15.90:0/3847129613 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1e7410d3e0 0x7f1e741a4bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:24.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.322+0000 7f1e73400640 1 -- 172.21.15.90:0/3847129613 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e7419f370 con 0x7f1e7419e770 2024-04-02T14:48:24.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.322+0000 7f1e73400640 1 --2- 172.21.15.90:0/3847129613 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1e7419e770 0x7f1e7419ebf0 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f1e6c00cab0 tx=0x7f1e6c00cf70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:48:24.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.323+0000 7f1e6b400640 1 -- 172.21.15.90:0/3847129613 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e6c00ae20 con 0x7f1e7419e770 2024-04-02T14:48:24.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.323+0000 7f1e6b400640 1 -- 172.21.15.90:0/3847129613 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1e6c00f930 con 0x7f1e7419e770 2024-04-02T14:48:24.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.323+0000 7f1e6b400640 1 -- 172.21.15.90:0/3847129613 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e6c018ad0 con 0x7f1e7419e770 2024-04-02T14:48:24.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.323+0000 7f1e79c00640 1 -- 172.21.15.90:0/3847129613 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e741ab450 con 0x7f1e7419e770 2024-04-02T14:48:24.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.323+0000 7f1e79c00640 1 -- 172.21.15.90:0/3847129613 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e741ab800 con 0x7f1e7419e770 2024-04-02T14:48:24.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.329+0000 7f1e6b400640 1 -- 172.21.15.90:0/3847129613 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f1e6c00fae0 con 0x7f1e7419e770 2024-04-02T14:48:24.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.331+0000 7f1e7c9ba740 1 Processor -- start 2024-04-02T14:48:24.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.331+0000 7f1e7c9ba740 1 -- start start 2024-04-02T14:48:24.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.331+0000 7f1e7c9ba740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f34e5e19f0 0x55f34e5e0740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:24.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.331+0000 7f1e7c9ba740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f34e5e0d10 0x55f34e5e1190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:24.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.331+0000 7f1e6b400640 1 --2- 172.21.15.90:0/3847129613 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f1e580a3e20 0x7f1e580a62e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:24.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.331+0000 7f1e7c9ba740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f34e5ea080 0x55f34e5ec470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:24.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.331+0000 7f1e7c9ba740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55f34e521400 con 0x55f34e5e19f0 2024-04-02T14:48:24.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.331+0000 7f1e7c9ba740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55f34e505f10 con 0x55f34e5e0d10 2024-04-02T14:48:24.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.331+0000 7f1e7c9ba740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55f34e5456f0 con 0x55f34e5ea080 2024-04-02T14:48:24.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.332+0000 7f1e6b400640 1 -- 172.21.15.90:0/3847129613 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 8550+0+0 (secure 0 0 0) 0x7f1e6c0d0b60 con 0x7f1e7419e770 2024-04-02T14:48:24.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.332+0000 7f1e73400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f34e5ea080 0x55f34e5ec470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:24.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.332+0000 7f1e73400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f34e5ea080 0x55f34e5ec470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:38142/0 (socket says 172.21.15.90:38142) 2024-04-02T14:48:24.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.332+0000 7f1e73400640 1 -- 172.21.15.90:0/2862935617 learned_addr learned my addr 172.21.15.90:0/2862935617 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:48:24.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.332+0000 7f1e72a00640 1 --2- 172.21.15.90:0/2862935617 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f34e5e0d10 0x55f34e5e1190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:24.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.332+0000 7f1e72000640 1 --2- 172.21.15.90:0/3847129613 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f1e580a3e20 0x7f1e580a62e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:24.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.332+0000 7f1e72000640 1 --2- 172.21.15.90:0/2862935617 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f34e5e19f0 0x55f34e5e0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:24.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.333+0000 7f1e72a00640 1 -- 172.21.15.90:0/2862935617 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f34e5ea080 msgr2=0x55f34e5ec470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:48:24.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.333+0000 7f1e72a00640 1 --2- 172.21.15.90:0/2862935617 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f34e5ea080 0x55f34e5ec470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:24.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.333+0000 7f1e72a00640 1 -- 172.21.15.90:0/2862935617 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f34e5e19f0 msgr2=0x55f34e5e0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:48:24.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.333+0000 7f1e72a00640 1 --2- 172.21.15.90:0/2862935617 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f34e5e19f0 0x55f34e5e0740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:24.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.333+0000 7f1e72a00640 1 -- 172.21.15.90:0/2862935617 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f34e2593f0 con 0x55f34e5e0d10 2024-04-02T14:48:24.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.333+0000 7f1e72000640 1 --2- 172.21.15.90:0/3847129613 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f1e580a3e20 0x7f1e580a62e0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f1e60005d00 tx=0x7f1e6000b9e0 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:48:24.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.334+0000 7f1e72a00640 1 --2- 172.21.15.90:0/2862935617 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f34e5e0d10 0x55f34e5e1190 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7f1e64009230 tx=0x7f1e6402c660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=75372ab29ccd6f93 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:48:24.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.334+0000 7f1e5fe00640 1 -- 172.21.15.90:0/2862935617 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e64042070 con 0x55f34e5e0d10 2024-04-02T14:48:24.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.334+0000 7f1e5fe00640 1 -- 172.21.15.90:0/2862935617 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1e6402cbd0 con 0x55f34e5e0d10 2024-04-02T14:48:24.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.334+0000 7f1e5fe00640 1 -- 172.21.15.90:0/2862935617 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e64005120 con 0x55f34e5e0d10 2024-04-02T14:48:24.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.335+0000 7f1e7c9ba740 1 -- 172.21.15.90:0/2862935617 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f34e5e0d10 msgr2=0x55f34e5e1190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:48:24.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.335+0000 7f1e7c9ba740 1 --2- 172.21.15.90:0/2862935617 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f34e5e0d10 0x55f34e5e1190 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7f1e64009230 tx=0x7f1e6402c660 comp rx=0 tx=0).stop 2024-04-02T14:48:24.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.336+0000 7f1e7c9ba740 1 -- 172.21.15.90:0/2862935617 shutdown_connections 2024-04-02T14:48:24.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.336+0000 7f1e7c9ba740 1 --2- 172.21.15.90:0/2862935617 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f34e5ea080 0x55f34e5ec470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:24.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.336+0000 7f1e7c9ba740 1 --2- 172.21.15.90:0/2862935617 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f34e5e0d10 0x55f34e5e1190 unknown :-1 s=CLOSED pgs=926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:24.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.336+0000 7f1e7c9ba740 1 --2- 172.21.15.90:0/2862935617 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f34e5e19f0 0x55f34e5e0740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:24.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.336+0000 7f1e7c9ba740 1 -- 172.21.15.90:0/2862935617 >> 172.21.15.90:0/2862935617 conn(0x55f34e5a7a50 msgr2=0x55f34e5db760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:48:24.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.336+0000 7f1e7c9ba740 1 -- 172.21.15.90:0/2862935617 shutdown_connections 2024-04-02T14:48:24.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.336+0000 7f1e7c9ba740 1 -- 172.21.15.90:0/2862935617 wait complete. 2024-04-02T14:48:24.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.338+0000 7f1e7c9ba740 1 -- start start 2024-04-02T14:48:24.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.338+0000 7f1e7c9ba740 1 Processor -- start 2024-04-02T14:48:24.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.339+0000 7f1e7c9ba740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f34e5e0d10 0x55f34e5e7000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:24.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.339+0000 7f1e7c9ba740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f34e5e19f0 0x55f34e5e7540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:24.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.339+0000 7f1e7c9ba740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f34e5ea080 0x55f34e5e7a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:24.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.339+0000 7f1e7c9ba740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55f34e5416a0 con 0x55f34e5e0d10 2024-04-02T14:48:24.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.339+0000 7f1e7c9ba740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55f34e4ed980 con 0x55f34e5ea080 2024-04-02T14:48:24.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.339+0000 7f1e7c9ba740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55f34e500cb0 con 0x55f34e5e19f0 2024-04-02T14:48:24.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.339+0000 7f1e7c9ba740 10 client.0 _ll_register_callbacks cb 0x55f34e5006a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:48:24.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.339+0000 7f1e72000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f34e5e19f0 0x55f34e5e7540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:24.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.339+0000 7f1e72a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f34e5e0d10 0x55f34e5e7000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:24.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.339+0000 7f1e72000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f34e5e19f0 0x55f34e5e7540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:38144/0 (socket says 172.21.15.90:38144) 2024-04-02T14:48:24.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.339+0000 7f1e72a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f34e5e0d10 0x55f34e5e7000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41928/0 (socket says 172.21.15.90:41928) 2024-04-02T14:48:24.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.339+0000 7f1e72000640 1 -- 172.21.15.90:0/213143879 learned_addr learned my addr 172.21.15.90:0/213143879 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:48:24.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.339+0000 7f1e73400640 1 --2- 172.21.15.90:0/213143879 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f34e5ea080 0x55f34e5e7a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:24.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.340+0000 7f1e72a00640 1 -- 172.21.15.90:0/213143879 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f34e5e19f0 msgr2=0x55f34e5e7540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:48:24.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.340+0000 7f1e72a00640 1 --2- 172.21.15.90:0/213143879 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f34e5e19f0 0x55f34e5e7540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:24.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.340+0000 7f1e72a00640 1 -- 172.21.15.90:0/213143879 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f34e5ea080 msgr2=0x55f34e5e7a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:48:24.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.340+0000 7f1e72a00640 1 --2- 172.21.15.90:0/213143879 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f34e5ea080 0x55f34e5e7a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:24.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.340+0000 7f1e72a00640 1 -- 172.21.15.90:0/213143879 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f34e3237d0 con 0x55f34e5e0d10 2024-04-02T14:48:24.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.340+0000 7f1e73400640 1 --2- 172.21.15.90:0/213143879 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f34e5ea080 0x55f34e5e7a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:48:24.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.340+0000 7f1e72a00640 1 --2- 172.21.15.90:0/213143879 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f34e5e0d10 0x55f34e5e7000 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f1e64005940 tx=0x7f1e64004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4eb88b1eb08a29a9 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:48:24.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.341+0000 7f1e5cc00640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-02T14:48:24.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.341+0000 7f1e5cc00640 1 -- 172.21.15.90:0/213143879 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e64042070 con 0x55f34e5e0d10 2024-04-02T14:48:24.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.341+0000 7f1e5cc00640 1 -- 172.21.15.90:0/213143879 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 8550+0+0 (secure 0 0 0) 0x7f1e64004df0 con 0x55f34e5e0d10 2024-04-02T14:48:24.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.342+0000 7f1e7c9ba740 1 -- 172.21.15.90:0/213143879 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1e6402cbd0 con 0x55f34e5e0d10 2024-04-02T14:48:24.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.342+0000 7f1e5cc00640 1 -- 172.21.15.90:0/213143879 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f1e64004ff0 con 0x55f34e5e0d10 2024-04-02T14:48:24.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.343+0000 7f1e7c9ba740 10 client.18477 fetch_fsmap learned FSMap version 27 2024-04-02T14:48:24.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.343+0000 7f1e7c9ba740 1 -- 172.21.15.90:0/213143879 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55f34e5f8e50 con 0x55f34e5e0d10 2024-04-02T14:48:24.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.343+0000 7f1e5cc00640 1 -- 172.21.15.90:0/213143879 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1e6402cbd0 con 0x55f34e5e0d10 2024-04-02T14:48:24.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.343+0000 7f1e5cc00640 1 -- 172.21.15.90:0/213143879 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f1e6404f450 con 0x55f34e5e0d10 2024-04-02T14:48:24.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.343+0000 7f1e7c9ba740 10 client.18477 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:48:24.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.343+0000 7f1e7c9ba740 10 client.18477 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.c 2024-04-02T14:48:24.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.344+0000 7f1e7c9ba740 20 client.18477 populate_metadata read hostname 'smithi090' 2024-04-02T14:48:24.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.344+0000 7f1e7c9ba740 1 --2- 172.21.15.90:0/213143879 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x55f34e5fa1a0 0x55f34e5fc590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:24.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.344+0000 7f1e7c9ba740 4 client.18477 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:48:24.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.344+0000 7f1e7c9ba740 1 -- 172.21.15.90:0/213143879 --> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f34e505040 con 0x55f34e5fa1a0 2024-04-02T14:48:24.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.344+0000 7f1e72000640 1 --2- 172.21.15.90:0/213143879 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x55f34e5fa1a0 0x55f34e5fc590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:24.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.345+0000 7f1e72000640 1 --2- 172.21.15.90:0/213143879 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x55f34e5fa1a0 0x55f34e5fc590 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=3c6d9ea28fde0567 server_cookie=a769339339cfda67 in_seq=0 out_seq=0 2024-04-02T14:48:24.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.345+0000 7f1e5cc00640 10 client.18477 ms_handle_connect on v2:172.21.15.177:6836/4223414858 2024-04-02T14:48:24.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.346+0000 7f1e5cc00640 1 -- 172.21.15.90:0/213143879 <== mds.4 v2:172.21.15.177:6836/4223414858 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f34e505040 con 0x55f34e5fa1a0 2024-04-02T14:48:24.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.346+0000 7f1e5cc00640 10 client.18477 handle_command_reply: tid=0 2024-04-02T14:48:24.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.346+0000 7f1e7c9ba740 1 client.18477 shutdown 2024-04-02T14:48:24.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.347+0000 7f1e7c9ba740 1 -- 172.21.15.90:0/213143879 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f34e5e0d10 msgr2=0x55f34e5e7000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:48:24.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.347+0000 7f1e7c9ba740 1 --2- 172.21.15.90:0/213143879 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f34e5e0d10 0x55f34e5e7000 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f1e64005940 tx=0x7f1e64004270 comp rx=0 tx=0).stop 2024-04-02T14:48:24.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.347+0000 7f1e7c9ba740 1 -- 172.21.15.90:0/213143879 shutdown_connections 2024-04-02T14:48:24.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.347+0000 7f1e7c9ba740 1 --2- 172.21.15.90:0/213143879 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f34e5ea080 0x55f34e5e7a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:24.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.347+0000 7f1e7c9ba740 1 --2- 172.21.15.90:0/213143879 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f34e5e19f0 0x55f34e5e7540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:24.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.347+0000 7f1e7c9ba740 1 --2- 172.21.15.90:0/213143879 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x55f34e5fa1a0 0x55f34e5fc590 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:24.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.347+0000 7f1e7c9ba740 1 --2- 172.21.15.90:0/213143879 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f34e5e0d10 0x55f34e5e7000 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:24.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.347+0000 7f1e7c9ba740 1 -- 172.21.15.90:0/213143879 >> 172.21.15.90:0/213143879 conn(0x55f34e5a7a50 msgr2=0x55f34e5dd4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:48:24.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.348+0000 7f1e5cc00640 0 client.18477 ms_handle_reset on v2:172.21.15.177:6836/4223414858 2024-04-02T14:48:24.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.348+0000 7f1e7c9ba740 1 -- 172.21.15.90:0/213143879 shutdown_connections 2024-04-02T14:48:24.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.348+0000 7f1e7c9ba740 1 -- 172.21.15.90:0/213143879 wait complete. 2024-04-02T14:48:24.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.348+0000 7f1e7c9ba740 20 client.18477 trim_cache size 0 max 16384 2024-04-02T14:48:24.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.374+0000 7f1e7c9ba740 1 Processor -- start 2024-04-02T14:48:24.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.374+0000 7f1e7c9ba740 1 -- start start 2024-04-02T14:48:24.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.374+0000 7f1e7c9ba740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f34e5e0d10 0x55f34e5e9730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:24.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.374+0000 7f1e7c9ba740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f34e5e19f0 0x55f34e5db900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:24.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.374+0000 7f1e7c9ba740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f34e5ea080 0x55f34e5e6160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:24.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.374+0000 7f1e7c9ba740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f1e741b3e10 con 0x55f34e5ea080 2024-04-02T14:48:24.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.374+0000 7f1e7c9ba740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f1e741b5950 con 0x55f34e5e0d10 2024-04-02T14:48:24.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.374+0000 7f1e7c9ba740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f1e741b44e0 con 0x55f34e5e19f0 2024-04-02T14:48:24.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.375+0000 7f1e72000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f34e5e19f0 0x55f34e5db900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:24.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.375+0000 7f1e72000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f34e5e19f0 0x55f34e5db900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:38154/0 (socket says 172.21.15.90:38154) 2024-04-02T14:48:24.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.375+0000 7f1e72000640 1 -- 172.21.15.90:0/3565187042 learned_addr learned my addr 172.21.15.90:0/3565187042 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:48:24.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.375+0000 7f1e72a00640 1 --2- 172.21.15.90:0/3565187042 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f34e5e0d10 0x55f34e5e9730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:24.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.375+0000 7f1e72000640 1 -- 172.21.15.90:0/3565187042 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f34e5e0d10 msgr2=0x55f34e5e9730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:48:24.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.375+0000 7f1e73400640 1 --2- 172.21.15.90:0/3565187042 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f34e5ea080 0x55f34e5e6160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:24.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.375+0000 7f1e72000640 1 --2- 172.21.15.90:0/3565187042 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f34e5e0d10 0x55f34e5e9730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:24.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.375+0000 7f1e72000640 1 -- 172.21.15.90:0/3565187042 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f34e5ea080 msgr2=0x55f34e5e6160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:48:24.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.376+0000 7f1e72000640 1 --2- 172.21.15.90:0/3565187042 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f34e5ea080 0x55f34e5e6160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:24.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.376+0000 7f1e72000640 1 -- 172.21.15.90:0/3565187042 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f34e5dfd40 con 0x55f34e5e19f0 2024-04-02T14:48:24.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.376+0000 7f1e73400640 1 --2- 172.21.15.90:0/3565187042 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f34e5ea080 0x55f34e5e6160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:48:24.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.376+0000 7f1e72000640 1 --2- 172.21.15.90:0/3565187042 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f34e5e19f0 0x55f34e5db900 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f1e6000b660 tx=0x7f1e60009000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=83cecf3c6dd5ee16 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:48:24.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.377+0000 7f1e5f400640 1 -- 172.21.15.90:0/3565187042 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e6002b500 con 0x55f34e5e19f0 2024-04-02T14:48:24.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.377+0000 7f1e5f400640 1 -- 172.21.15.90:0/3565187042 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1e60030070 con 0x55f34e5e19f0 2024-04-02T14:48:24.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.377+0000 7f1e5f400640 1 -- 172.21.15.90:0/3565187042 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e60010430 con 0x55f34e5e19f0 2024-04-02T14:48:24.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.377+0000 7f1e7c9ba740 1 -- 172.21.15.90:0/3565187042 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f34e5e19f0 msgr2=0x55f34e5db900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:48:24.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.377+0000 7f1e7c9ba740 1 --2- 172.21.15.90:0/3565187042 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f34e5e19f0 0x55f34e5db900 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f1e6000b660 tx=0x7f1e60009000 comp rx=0 tx=0).stop 2024-04-02T14:48:24.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.378+0000 7f1e7c9ba740 1 -- 172.21.15.90:0/3565187042 shutdown_connections 2024-04-02T14:48:24.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.378+0000 7f1e7c9ba740 1 --2- 172.21.15.90:0/3565187042 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f34e5ea080 0x55f34e5e6160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:24.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.378+0000 7f1e7c9ba740 1 --2- 172.21.15.90:0/3565187042 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f34e5e19f0 0x55f34e5db900 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:24.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.378+0000 7f1e7c9ba740 1 --2- 172.21.15.90:0/3565187042 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f34e5e0d10 0x55f34e5e9730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:24.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.378+0000 7f1e7c9ba740 1 -- 172.21.15.90:0/3565187042 >> 172.21.15.90:0/3565187042 conn(0x55f34e5a7a50 msgr2=0x55f34e5f4fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:48:24.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.378+0000 7f1e7c9ba740 1 -- 172.21.15.90:0/3565187042 shutdown_connections 2024-04-02T14:48:24.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.379+0000 7f1e7c9ba740 1 -- 172.21.15.90:0/3565187042 wait complete. 2024-04-02T14:48:24.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.379+0000 7f1e7c9ba740 1 -- start start 2024-04-02T14:48:24.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.380+0000 7f1e7c9ba740 1 Processor -- start 2024-04-02T14:48:24.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.380+0000 7f1e7c9ba740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f34e5e0d10 0x55f34e5ff310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:24.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.380+0000 7f1e7c9ba740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f34e5e19f0 0x55f34e5fa780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:24.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.380+0000 7f1e7c9ba740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f34e5ea080 0x55f34e5facc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:24.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.380+0000 7f1e7c9ba740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f1e741b5280 con 0x55f34e5e0d10 2024-04-02T14:48:24.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.380+0000 7f1e7c9ba740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f1e741b4bb0 con 0x55f34e5ea080 2024-04-02T14:48:24.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.380+0000 7f1e7c9ba740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f1e741b1e10 con 0x55f34e5e19f0 2024-04-02T14:48:24.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.380+0000 7f1e7c9ba740 10 client.0 _ll_register_callbacks cb 0x7f1e6402c340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:48:24.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.381+0000 7f1e72000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f34e5e19f0 0x55f34e5fa780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:24.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.381+0000 7f1e72000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f34e5e19f0 0x55f34e5fa780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:38158/0 (socket says 172.21.15.90:38158) 2024-04-02T14:48:24.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.381+0000 7f1e72000640 1 -- 172.21.15.90:0/2672133055 learned_addr learned my addr 172.21.15.90:0/2672133055 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:48:24.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.381+0000 7f1e72a00640 1 --2- 172.21.15.90:0/2672133055 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f34e5e0d10 0x55f34e5ff310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:24.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.381+0000 7f1e73400640 1 --2- 172.21.15.90:0/2672133055 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f34e5ea080 0x55f34e5facc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:24.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.381+0000 7f1e72000640 1 -- 172.21.15.90:0/2672133055 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f34e5ea080 msgr2=0x55f34e5facc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:48:24.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.381+0000 7f1e72000640 1 --2- 172.21.15.90:0/2672133055 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f34e5ea080 0x55f34e5facc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:24.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.381+0000 7f1e72000640 1 -- 172.21.15.90:0/2672133055 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f34e5e0d10 msgr2=0x55f34e5ff310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:48:24.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.381+0000 7f1e72000640 1 --2- 172.21.15.90:0/2672133055 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f34e5e0d10 0x55f34e5ff310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:24.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.381+0000 7f1e72000640 1 -- 172.21.15.90:0/2672133055 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f34e5fb230 con 0x55f34e5e19f0 2024-04-02T14:48:24.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.381+0000 7f1e73400640 1 --2- 172.21.15.90:0/2672133055 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f34e5ea080 0x55f34e5facc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:48:24.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.381+0000 7f1e72000640 1 --2- 172.21.15.90:0/2672133055 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f34e5e19f0 0x55f34e5fa780 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f1e60029c10 tx=0x7f1e60029cf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a4d27e29db6a6ac0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:48:24.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.381+0000 7f1e5cc00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:48:24.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.382+0000 7f1e5cc00640 1 -- 172.21.15.90:0/2672133055 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e6002b710 con 0x55f34e5e19f0 2024-04-02T14:48:24.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.382+0000 7f1e5cc00640 1 -- 172.21.15.90:0/2672133055 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 8550+0+0 (secure 0 0 0) 0x7f1e6002b8b0 con 0x55f34e5e19f0 2024-04-02T14:48:24.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.382+0000 7f1e7c9ba740 1 -- 172.21.15.90:0/2672133055 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1e60030070 con 0x55f34e5e19f0 2024-04-02T14:48:24.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.382+0000 7f1e5cc00640 1 -- 172.21.15.90:0/2672133055 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f1e60029470 con 0x55f34e5e19f0 2024-04-02T14:48:24.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.382+0000 7f1e7c9ba740 10 client.29434 fetch_fsmap learned FSMap version 27 2024-04-02T14:48:24.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.382+0000 7f1e7c9ba740 1 -- 172.21.15.90:0/2672133055 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55f34e660070 con 0x55f34e5e19f0 2024-04-02T14:48:24.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.382+0000 7f1e5cc00640 1 -- 172.21.15.90:0/2672133055 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1e60030070 con 0x55f34e5e19f0 2024-04-02T14:48:24.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.383+0000 7f1e5cc00640 1 -- 172.21.15.90:0/2672133055 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f1e6000f040 con 0x55f34e5e19f0 2024-04-02T14:48:24.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.383+0000 7f1e7c9ba740 10 client.29434 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:48:24.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.383+0000 7f1e7c9ba740 10 client.29434 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.c 2024-04-02T14:48:24.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.383+0000 7f1e7c9ba740 20 client.29434 populate_metadata read hostname 'smithi090' 2024-04-02T14:48:24.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.383+0000 7f1e7c9ba740 1 --2- 172.21.15.90:0/2672133055 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x55f34e6607c0 0x55f34e662be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:24.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.383+0000 7f1e7c9ba740 4 client.29434 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-02T14:48:24.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.383+0000 7f1e7c9ba740 1 -- 172.21.15.90:0/2672133055 --> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55f34e552c30 con 0x55f34e6607c0 2024-04-02T14:48:24.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.384+0000 7f1e72a00640 1 --2- 172.21.15.90:0/2672133055 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x55f34e6607c0 0x55f34e662be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:24.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.385+0000 7f1e72a00640 1 --2- 172.21.15.90:0/2672133055 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x55f34e6607c0 0x55f34e662be0 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=48a5f4046bd827eb server_cookie=564650276b58c103 in_seq=0 out_seq=0 2024-04-02T14:48:24.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.385+0000 7f1e5cc00640 10 client.29434 ms_handle_connect on v2:172.21.15.177:6836/4223414858 2024-04-02T14:48:24.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.386+0000 7f1e5cc00640 1 -- 172.21.15.90:0/2672133055 <== mds.4 v2:172.21.15.177:6836/4223414858 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55f34e552c30 con 0x55f34e6607c0 2024-04-02T14:48:24.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.386+0000 7f1e5cc00640 10 client.29434 handle_command_reply: tid=0 2024-04-02T14:48:24.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.386+0000 7f1e7c9ba740 1 client.29434 shutdown 2024-04-02T14:48:24.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.386+0000 7f1e7c9ba740 1 -- 172.21.15.90:0/2672133055 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f34e5e19f0 msgr2=0x55f34e5fa780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:48:24.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.386+0000 7f1e7c9ba740 1 --2- 172.21.15.90:0/2672133055 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f34e5e19f0 0x55f34e5fa780 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f1e60029c10 tx=0x7f1e60029cf0 comp rx=0 tx=0).stop 2024-04-02T14:48:24.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.386+0000 7f1e7c9ba740 1 -- 172.21.15.90:0/2672133055 shutdown_connections 2024-04-02T14:48:24.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.386+0000 7f1e7c9ba740 1 --2- 172.21.15.90:0/2672133055 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f34e5ea080 0x55f34e5facc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:24.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.386+0000 7f1e7c9ba740 1 --2- 172.21.15.90:0/2672133055 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f34e5e19f0 0x55f34e5fa780 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:24.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.386+0000 7f1e7c9ba740 1 --2- 172.21.15.90:0/2672133055 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x55f34e6607c0 0x55f34e662be0 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:24.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.386+0000 7f1e7c9ba740 1 --2- 172.21.15.90:0/2672133055 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f34e5e0d10 0x55f34e5ff310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:24.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.386+0000 7f1e7c9ba740 1 -- 172.21.15.90:0/2672133055 >> 172.21.15.90:0/2672133055 conn(0x55f34e5a7a50 msgr2=0x55f34e5dcb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:48:24.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.386+0000 7f1e5cc00640 0 client.29434 ms_handle_reset on v2:172.21.15.177:6836/4223414858 2024-04-02T14:48:24.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.387+0000 7f1e7c9ba740 1 -- 172.21.15.90:0/2672133055 shutdown_connections 2024-04-02T14:48:24.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.388+0000 7f1e7c9ba740 1 -- 172.21.15.90:0/2672133055 wait complete. 2024-04-02T14:48:24.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.388+0000 7f1e7c9ba740 20 client.29434 trim_cache size 0 max 16384 2024-04-02T14:48:24.383 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-02T14:48:24.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.389+0000 7f1e79c00640 1 -- 172.21.15.90:0/3847129613 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f1e580a3e20 msgr2=0x7f1e580a62e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:48:24.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.389+0000 7f1e79c00640 1 --2- 172.21.15.90:0/3847129613 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f1e580a3e20 0x7f1e580a62e0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f1e60005d00 tx=0x7f1e6000b9e0 comp rx=0 tx=0).stop 2024-04-02T14:48:24.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.389+0000 7f1e79c00640 1 -- 172.21.15.90:0/3847129613 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1e7419e770 msgr2=0x7f1e7419ebf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:48:24.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.389+0000 7f1e79c00640 1 --2- 172.21.15.90:0/3847129613 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1e7419e770 0x7f1e7419ebf0 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f1e6c00cab0 tx=0x7f1e6c00cf70 comp rx=0 tx=0).stop 2024-04-02T14:48:24.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.389+0000 7f1e79c00640 1 -- 172.21.15.90:0/3847129613 shutdown_connections 2024-04-02T14:48:24.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.389+0000 7f1e79c00640 1 --2- 172.21.15.90:0/3847129613 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f1e580a3e20 0x7f1e580a62e0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:24.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.389+0000 7f1e79c00640 1 --2- 172.21.15.90:0/3847129613 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1e7419e770 0x7f1e7419ebf0 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:24.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.389+0000 7f1e79c00640 1 --2- 172.21.15.90:0/3847129613 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1e7410d3e0 0x7f1e741a4bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:24.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.389+0000 7f1e79c00640 1 --2- 172.21.15.90:0/3847129613 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1e74108310 0x7f1e741a46b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:24.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.389+0000 7f1e79c00640 1 -- 172.21.15.90:0/3847129613 >> 172.21.15.90:0/3847129613 conn(0x7f1e74076040 msgr2=0x7f1e740783a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:48:24.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.390+0000 7f1e79c00640 1 -- 172.21.15.90:0/3847129613 shutdown_connections 2024-04-02T14:48:24.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.390+0000 7f1e79c00640 1 -- 172.21.15.90:0/3847129613 wait complete. 2024-04-02T14:48:24.410 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:48:24 smithi090 ceph-mon[31674]: pgmap v422: 97 pgs: 97 active+clean; 4.2 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 9.6 MiB/s wr, 528 op/s 2024-04-02T14:48:24.410 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:48:24 smithi090 ceph-mon[31674]: from='client.? 172.21.15.90:0/816800392' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-02T14:48:24.414 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-04-02T14:48:24.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:48:24 smithi177 ceph-mon[36431]: pgmap v422: 97 pgs: 97 active+clean; 4.2 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 9.6 MiB/s wr, 528 op/s 2024-04-02T14:48:24.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:48:24 smithi177 ceph-mon[36431]: from='client.? 172.21.15.90:0/816800392' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-02T14:48:24.603 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:48:24 smithi092 ceph-mon[36497]: pgmap v422: 97 pgs: 97 active+clean; 4.2 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 9.6 MiB/s wr, 528 op/s 2024-04-02T14:48:24.603 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:48:24 smithi092 ceph-mon[36497]: from='client.? 172.21.15.90:0/816800392' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-02T14:48:24.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.724+0000 7f42ca200640 1 -- 172.21.15.90:0/2012228047 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f42c4101a90 msgr2=0x7f42c4101f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:48:24.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.724+0000 7f42ca200640 1 --2- 172.21.15.90:0/2012228047 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f42c4101a90 0x7f42c4101f10 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f42b0009930 tx=0x7f42b002bc20 comp rx=0 tx=0).stop 2024-04-02T14:48:24.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.724+0000 7f42ca200640 1 -- 172.21.15.90:0/2012228047 shutdown_connections 2024-04-02T14:48:24.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.724+0000 7f42ca200640 1 --2- 172.21.15.90:0/2012228047 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f42c4102450 0x7f42c410e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:24.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.724+0000 7f42ca200640 1 --2- 172.21.15.90:0/2012228047 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f42c4101a90 0x7f42c4101f10 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:24.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.724+0000 7f42ca200640 1 --2- 172.21.15.90:0/2012228047 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f42c4107aa0 0x7f42c4107e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:24.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.724+0000 7f42ca200640 1 -- 172.21.15.90:0/2012228047 >> 172.21.15.90:0/2012228047 conn(0x7f42c4078f00 msgr2=0x7f42c4079310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:48:24.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.724+0000 7f42ca200640 1 -- 172.21.15.90:0/2012228047 shutdown_connections 2024-04-02T14:48:24.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.724+0000 7f42ca200640 1 -- 172.21.15.90:0/2012228047 wait complete. 2024-04-02T14:48:24.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.726+0000 7f42ca200640 1 Processor -- start 2024-04-02T14:48:24.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.726+0000 7f42ca200640 1 -- start start 2024-04-02T14:48:24.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.726+0000 7f42ca200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f42c4101a90 0x7f42c41089e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:24.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.726+0000 7f42ca200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f42c4102450 0x7f42c4108f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:24.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.726+0000 7f42ca200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f42c4107aa0 0x7f42c4109460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:24.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.726+0000 7f42ca200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f42c4111ed0 con 0x7f42c4107aa0 2024-04-02T14:48:24.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.726+0000 7f42ca200640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f42c4111d50 con 0x7f42c4101a90 2024-04-02T14:48:24.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.727+0000 7f42ca200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f42c4112050 con 0x7f42c4102450 2024-04-02T14:48:24.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.727+0000 7f42c2a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f42c4102450 0x7f42c4108f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:24.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.727+0000 7f42c3400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f42c4101a90 0x7f42c41089e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:24.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.727+0000 7f42c2a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f42c4102450 0x7f42c4108f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:38178/0 (socket says 172.21.15.90:38178) 2024-04-02T14:48:24.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.727+0000 7f42c2a00640 1 -- 172.21.15.90:0/1012879831 learned_addr learned my addr 172.21.15.90:0/1012879831 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:48:24.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.727+0000 7f42c3400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f42c4101a90 0x7f42c41089e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:54246/0 (socket says 172.21.15.90:54246) 2024-04-02T14:48:24.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.727+0000 7f42c2a00640 1 -- 172.21.15.90:0/1012879831 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f42c4101a90 msgr2=0x7f42c41089e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:48:24.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.727+0000 7f42c2a00640 1 --2- 172.21.15.90:0/1012879831 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f42c4101a90 0x7f42c41089e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:24.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.727+0000 7f42c2a00640 1 -- 172.21.15.90:0/1012879831 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f42c4107aa0 msgr2=0x7f42c4109460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:48:24.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.727+0000 7f42c2a00640 1 --2- 172.21.15.90:0/1012879831 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f42c4107aa0 0x7f42c4109460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:24.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.727+0000 7f42c2a00640 1 -- 172.21.15.90:0/1012879831 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42c419a3b0 con 0x7f42c4102450 2024-04-02T14:48:24.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.728+0000 7f42c2a00640 1 --2- 172.21.15.90:0/1012879831 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f42c4102450 0x7f42c4108f20 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f42b0009900 tx=0x7f42b00043b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:48:24.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.729+0000 7f42bbe00640 1 -- 172.21.15.90:0/1012879831 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42b0043020 con 0x7f42c4102450 2024-04-02T14:48:24.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.729+0000 7f42bbe00640 1 -- 172.21.15.90:0/1012879831 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f42b00054e0 con 0x7f42c4102450 2024-04-02T14:48:24.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.729+0000 7f42bbe00640 1 -- 172.21.15.90:0/1012879831 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42b0003c50 con 0x7f42c4102450 2024-04-02T14:48:24.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.730+0000 7f42ca200640 1 -- 172.21.15.90:0/1012879831 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f42c419a640 con 0x7f42c4102450 2024-04-02T14:48:24.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.730+0000 7f42ca200640 1 -- 172.21.15.90:0/1012879831 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42c419ab40 con 0x7f42c4102450 2024-04-02T14:48:24.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.735+0000 7f42bbe00640 1 -- 172.21.15.90:0/1012879831 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f42b0046020 con 0x7f42c4102450 2024-04-02T14:48:24.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.738+0000 7f42ccf65740 1 Processor -- start 2024-04-02T14:48:24.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.738+0000 7f42bbe00640 1 --2- 172.21.15.90:0/1012879831 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f42a80a3da0 0x7f42a80a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:24.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.738+0000 7f42ccf65740 1 -- start start 2024-04-02T14:48:24.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.739+0000 7f42bbe00640 1 -- 172.21.15.90:0/1012879831 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 8550+0+0 (secure 0 0 0) 0x7f42b00f1ea0 con 0x7f42c4102450 2024-04-02T14:48:24.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.739+0000 7f42ccf65740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557e74df49f0 0x557e74df3740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:24.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.739+0000 7f42ccf65740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557e74df3d10 0x557e74df4190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:24.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.739+0000 7f42ccf65740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557e74dfd080 0x557e74dff470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:24.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.739+0000 7f42ccf65740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x557e74d34400 con 0x557e74df49f0 2024-04-02T14:48:24.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.739+0000 7f42ccf65740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x557e74d18f10 con 0x557e74dfd080 2024-04-02T14:48:24.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.739+0000 7f42ccf65740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x557e74d586f0 con 0x557e74df3d10 2024-04-02T14:48:24.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.740+0000 7f42c3e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557e74dfd080 0x557e74dff470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:24.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.740+0000 7f42c3e00640 1 --2- 172.21.15.90:0/1012879831 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f42a80a3da0 0x7f42a80a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:24.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.740+0000 7f42c3e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557e74dfd080 0x557e74dff470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:54254/0 (socket says 172.21.15.90:54254) 2024-04-02T14:48:24.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.740+0000 7f42c3e00640 1 -- 172.21.15.90:0/1775894101 learned_addr learned my addr 172.21.15.90:0/1775894101 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:48:24.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.740+0000 7f42c3400640 1 --2- 172.21.15.90:0/1775894101 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557e74df49f0 0x557e74df3740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:24.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.741+0000 7f42c2a00640 1 --2- 172.21.15.90:0/1775894101 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557e74df3d10 0x557e74df4190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:24.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.742+0000 7f42c3400640 1 -- 172.21.15.90:0/1775894101 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557e74df3d10 msgr2=0x557e74df4190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:48:24.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.743+0000 7f42c3400640 1 --2- 172.21.15.90:0/1775894101 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557e74df3d10 0x557e74df4190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:24.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.743+0000 7f42c3400640 1 -- 172.21.15.90:0/1775894101 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557e74dfd080 msgr2=0x557e74dff470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:48:24.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.743+0000 7f42c3400640 1 --2- 172.21.15.90:0/1775894101 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557e74dfd080 0x557e74dff470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:24.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.743+0000 7f42c3400640 1 -- 172.21.15.90:0/1775894101 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557e74a6c3f0 con 0x557e74df49f0 2024-04-02T14:48:24.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.743+0000 7f42c3e00640 1 --2- 172.21.15.90:0/1775894101 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557e74dfd080 0x557e74dff470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:48:24.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.743+0000 7f42c2a00640 1 --2- 172.21.15.90:0/1775894101 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557e74df3d10 0x557e74df4190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:48:24.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.743+0000 7f42c3e00640 1 --2- 172.21.15.90:0/1012879831 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f42a80a3da0 0x7f42a80a6260 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f42b400bfd0 tx=0x7f42b4005b80 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:48:24.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.744+0000 7f42c3400640 1 --2- 172.21.15.90:0/1775894101 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557e74df49f0 0x557e74df3740 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f42bc009f30 tx=0x7f42bc010800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5bb833efc60fb68 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:48:24.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.744+0000 7f42afe00640 1 -- 172.21.15.90:0/1775894101 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42bc016070 con 0x557e74df49f0 2024-04-02T14:48:24.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.744+0000 7f42afe00640 1 -- 172.21.15.90:0/1775894101 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f42bc004d20 con 0x557e74df49f0 2024-04-02T14:48:24.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.744+0000 7f42afe00640 1 -- 172.21.15.90:0/1775894101 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42bc0053c0 con 0x557e74df49f0 2024-04-02T14:48:24.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.745+0000 7f42ccf65740 1 -- 172.21.15.90:0/1775894101 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557e74df49f0 msgr2=0x557e74df3740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:48:24.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.745+0000 7f42ccf65740 1 --2- 172.21.15.90:0/1775894101 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557e74df49f0 0x557e74df3740 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f42bc009f30 tx=0x7f42bc010800 comp rx=0 tx=0).stop 2024-04-02T14:48:24.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.746+0000 7f42ccf65740 1 -- 172.21.15.90:0/1775894101 shutdown_connections 2024-04-02T14:48:24.741 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.746+0000 7f42ccf65740 1 --2- 172.21.15.90:0/1775894101 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557e74dfd080 0x557e74dff470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:24.741 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.746+0000 7f42ccf65740 1 --2- 172.21.15.90:0/1775894101 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557e74df3d10 0x557e74df4190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:24.741 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.746+0000 7f42ccf65740 1 --2- 172.21.15.90:0/1775894101 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557e74df49f0 0x557e74df3740 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:24.741 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.746+0000 7f42ccf65740 1 -- 172.21.15.90:0/1775894101 >> 172.21.15.90:0/1775894101 conn(0x557e74dbaa50 msgr2=0x557e74dee760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:48:24.741 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.746+0000 7f42ccf65740 1 -- 172.21.15.90:0/1775894101 shutdown_connections 2024-04-02T14:48:24.741 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.746+0000 7f42ccf65740 1 -- 172.21.15.90:0/1775894101 wait complete. 2024-04-02T14:48:24.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.747+0000 7f42ccf65740 1 -- start start 2024-04-02T14:48:24.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.748+0000 7f42ccf65740 1 Processor -- start 2024-04-02T14:48:24.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.748+0000 7f42ccf65740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557e74df3d10 0x557e74dfa070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:24.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.748+0000 7f42ccf65740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557e74dfd080 0x557e74dfa5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:24.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.748+0000 7f42ccf65740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557e74dfaaf0 0x557e74e090d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:24.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.748+0000 7f42ccf65740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x557e74d546a0 con 0x557e74dfaaf0 2024-04-02T14:48:24.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.748+0000 7f42ccf65740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x557e74d00980 con 0x557e74df3d10 2024-04-02T14:48:24.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.748+0000 7f42ccf65740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x557e74d13cb0 con 0x557e74dfd080 2024-04-02T14:48:24.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.749+0000 7f42ccf65740 10 client.0 _ll_register_callbacks cb 0x557e74d136a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:48:24.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.749+0000 7f42c3400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557e74df3d10 0x557e74dfa070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:24.745 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.749+0000 7f42c3400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557e74df3d10 0x557e74dfa070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:54262/0 (socket says 172.21.15.90:54262) 2024-04-02T14:48:24.745 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.749+0000 7f42c3400640 1 -- 172.21.15.90:0/1700569480 learned_addr learned my addr 172.21.15.90:0/1700569480 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:48:24.745 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.750+0000 7f42c2a00640 1 --2- 172.21.15.90:0/1700569480 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557e74dfd080 0x557e74dfa5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:24.745 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.750+0000 7f42c3e00640 1 --2- 172.21.15.90:0/1700569480 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557e74dfaaf0 0x557e74e090d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:24.745 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.750+0000 7f42c3400640 1 -- 172.21.15.90:0/1700569480 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557e74dfd080 msgr2=0x557e74dfa5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:48:24.745 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.750+0000 7f42c3400640 1 --2- 172.21.15.90:0/1700569480 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557e74dfd080 0x557e74dfa5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:24.746 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.751+0000 7f42c3400640 1 -- 172.21.15.90:0/1700569480 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557e74dfaaf0 msgr2=0x557e74e090d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:48:24.746 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.751+0000 7f42c3400640 1 --2- 172.21.15.90:0/1700569480 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557e74dfaaf0 0x557e74e090d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:24.746 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.751+0000 7f42c3400640 1 -- 172.21.15.90:0/1700569480 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557e74b367d0 con 0x557e74df3d10 2024-04-02T14:48:24.746 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.751+0000 7f42c3e00640 1 --2- 172.21.15.90:0/1700569480 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557e74dfaaf0 0x557e74e090d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:48:24.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.752+0000 7f42c3400640 1 --2- 172.21.15.90:0/1700569480 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557e74df3d10 0x557e74dfa070 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7f42bc012f30 tx=0x7f42bc004f40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7dc1d74e0b37316f server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:48:24.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.752+0000 7f42acc00640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:48:24.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.753+0000 7f42acc00640 1 -- 172.21.15.90:0/1700569480 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42bc016070 con 0x557e74df3d10 2024-04-02T14:48:24.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.753+0000 7f42ccf65740 1 -- 172.21.15.90:0/1700569480 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f42bc004d20 con 0x557e74df3d10 2024-04-02T14:48:24.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.753+0000 7f42acc00640 1 -- 172.21.15.90:0/1700569480 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 8550+0+0 (secure 0 0 0) 0x7f42bc012720 con 0x557e74df3d10 2024-04-02T14:48:24.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.753+0000 7f42acc00640 1 -- 172.21.15.90:0/1700569480 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f42bc016070 con 0x557e74df3d10 2024-04-02T14:48:24.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.753+0000 7f42ccf65740 10 client.29660 fetch_fsmap learned FSMap version 27 2024-04-02T14:48:24.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.753+0000 7f42ccf65740 1 -- 172.21.15.90:0/1700569480 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x557e74e0c3c0 con 0x557e74df3d10 2024-04-02T14:48:24.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.753+0000 7f42acc00640 1 -- 172.21.15.90:0/1700569480 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f42bc004d20 con 0x557e74df3d10 2024-04-02T14:48:24.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.754+0000 7f42acc00640 1 -- 172.21.15.90:0/1700569480 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f42bc00f040 con 0x557e74df3d10 2024-04-02T14:48:24.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.754+0000 7f42ccf65740 10 client.29660 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:48:24.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.754+0000 7f42ccf65740 10 client.29660 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-04-02T14:48:24.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.754+0000 7f42ccf65740 20 client.29660 populate_metadata read hostname 'smithi090' 2024-04-02T14:48:24.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.754+0000 7f42ccf65740 1 --2- 172.21.15.90:0/1700569480 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x557e74e0d710 0x557e74e0fb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:24.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.754+0000 7f42ccf65740 4 client.29660 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:48:24.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.754+0000 7f42ccf65740 1 -- 172.21.15.90:0/1700569480 --> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557e74d18040 con 0x557e74e0d710 2024-04-02T14:48:24.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.754+0000 7f42c2a00640 1 --2- 172.21.15.90:0/1700569480 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x557e74e0d710 0x557e74e0fb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:24.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.755+0000 7f42c2a00640 1 --2- 172.21.15.90:0/1700569480 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x557e74e0d710 0x557e74e0fb00 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=d48a203e1e70766c server_cookie=7a45fb7d27edb072 in_seq=0 out_seq=0 2024-04-02T14:48:24.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.755+0000 7f42acc00640 10 client.29660 ms_handle_connect on v2:172.21.15.92:6836/1270635939 2024-04-02T14:48:24.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.755+0000 7f42acc00640 1 -- 172.21.15.90:0/1700569480 <== mds.3 v2:172.21.15.92:6836/1270635939 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x557e74d18040 con 0x557e74e0d710 2024-04-02T14:48:24.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.755+0000 7f42acc00640 10 client.29660 handle_command_reply: tid=0 2024-04-02T14:48:24.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.755+0000 7f42ccf65740 1 client.29660 shutdown 2024-04-02T14:48:24.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.756+0000 7f42ccf65740 1 -- 172.21.15.90:0/1700569480 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557e74df3d10 msgr2=0x557e74dfa070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:48:24.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.756+0000 7f42ccf65740 1 --2- 172.21.15.90:0/1700569480 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557e74df3d10 0x557e74dfa070 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7f42bc012f30 tx=0x7f42bc004f40 comp rx=0 tx=0).stop 2024-04-02T14:48:24.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.756+0000 7f42ccf65740 1 -- 172.21.15.90:0/1700569480 shutdown_connections 2024-04-02T14:48:24.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.756+0000 7f42ccf65740 1 --2- 172.21.15.90:0/1700569480 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x557e74e0d710 0x557e74e0fb00 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:24.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.756+0000 7f42ccf65740 1 --2- 172.21.15.90:0/1700569480 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557e74dfaaf0 0x557e74e090d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:24.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.756+0000 7f42ccf65740 1 --2- 172.21.15.90:0/1700569480 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557e74dfd080 0x557e74dfa5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:24.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.756+0000 7f42ccf65740 1 --2- 172.21.15.90:0/1700569480 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557e74df3d10 0x557e74dfa070 unknown :-1 s=CLOSED pgs=927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:24.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.756+0000 7f42ccf65740 1 -- 172.21.15.90:0/1700569480 >> 172.21.15.90:0/1700569480 conn(0x557e74dbaa50 msgr2=0x557e74df04b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:48:24.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.756+0000 7f42acc00640 0 client.29660 ms_handle_reset on v2:172.21.15.92:6836/1270635939 2024-04-02T14:48:24.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.756+0000 7f42ccf65740 1 -- 172.21.15.90:0/1700569480 shutdown_connections 2024-04-02T14:48:24.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.756+0000 7f42ccf65740 1 -- 172.21.15.90:0/1700569480 wait complete. 2024-04-02T14:48:24.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.756+0000 7f42ccf65740 20 client.29660 trim_cache size 0 max 16384 2024-04-02T14:48:24.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.774+0000 7f42ccf65740 1 Processor -- start 2024-04-02T14:48:24.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.774+0000 7f42ccf65740 1 -- start start 2024-04-02T14:48:24.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.774+0000 7f42ccf65740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557e74dfaaf0 0x557e74e15130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:24.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.774+0000 7f42ccf65740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557e74dfd080 0x557e74e15670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:24.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.774+0000 7f42ccf65740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557e74df0770 0x557e74e08b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:24.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.774+0000 7f42ccf65740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f42c41bc6f0 con 0x557e74dfd080 2024-04-02T14:48:24.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.774+0000 7f42ccf65740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f42c41be230 con 0x557e74dfaaf0 2024-04-02T14:48:24.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.774+0000 7f42ccf65740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f42c41bcdc0 con 0x557e74df0770 2024-04-02T14:48:24.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.775+0000 7f42c3400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557e74dfaaf0 0x557e74e15130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:24.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.775+0000 7f42c3e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557e74df0770 0x557e74e08b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:24.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.775+0000 7f42c3400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557e74dfaaf0 0x557e74e15130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:54278/0 (socket says 172.21.15.90:54278) 2024-04-02T14:48:24.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.775+0000 7f42c3e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557e74df0770 0x557e74e08b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:38194/0 (socket says 172.21.15.90:38194) 2024-04-02T14:48:24.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.775+0000 7f42c3400640 1 -- 172.21.15.90:0/2779674694 learned_addr learned my addr 172.21.15.90:0/2779674694 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:48:24.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.775+0000 7f42c2a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557e74dfd080 0x557e74e15670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:24.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.775+0000 7f42c3400640 1 -- 172.21.15.90:0/2779674694 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557e74df0770 msgr2=0x557e74e08b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:48:24.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.775+0000 7f42c3400640 1 --2- 172.21.15.90:0/2779674694 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557e74df0770 0x557e74e08b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:24.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.775+0000 7f42c3400640 1 -- 172.21.15.90:0/2779674694 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557e74dfd080 msgr2=0x557e74e15670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:48:24.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.775+0000 7f42c3400640 1 --2- 172.21.15.90:0/2779674694 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557e74dfd080 0x557e74e15670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:24.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.775+0000 7f42c3400640 1 -- 172.21.15.90:0/2779674694 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557e74df85c0 con 0x557e74dfaaf0 2024-04-02T14:48:24.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.775+0000 7f42c3e00640 1 --2- 172.21.15.90:0/2779674694 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557e74df0770 0x557e74e08b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:48:24.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.775+0000 7f42c2a00640 1 --2- 172.21.15.90:0/2779674694 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557e74dfd080 0x557e74e15670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:48:24.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.776+0000 7f42c3400640 1 --2- 172.21.15.90:0/2779674694 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557e74dfaaf0 0x557e74e15130 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7f42bc010d00 tx=0x7f42bc004420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d43a5463b20efb17 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:48:24.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.776+0000 7f42af400640 1 -- 172.21.15.90:0/2779674694 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42bc0045a0 con 0x557e74dfaaf0 2024-04-02T14:48:24.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.776+0000 7f42af400640 1 -- 172.21.15.90:0/2779674694 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f42bc004cd0 con 0x557e74dfaaf0 2024-04-02T14:48:24.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.776+0000 7f42af400640 1 -- 172.21.15.90:0/2779674694 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42bc012bc0 con 0x557e74dfaaf0 2024-04-02T14:48:24.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.777+0000 7f42ccf65740 1 -- 172.21.15.90:0/2779674694 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557e74dfaaf0 msgr2=0x557e74e15130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:48:24.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.777+0000 7f42ccf65740 1 --2- 172.21.15.90:0/2779674694 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557e74dfaaf0 0x557e74e15130 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7f42bc010d00 tx=0x7f42bc004420 comp rx=0 tx=0).stop 2024-04-02T14:48:24.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.778+0000 7f42ccf65740 1 -- 172.21.15.90:0/2779674694 shutdown_connections 2024-04-02T14:48:24.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.778+0000 7f42ccf65740 1 --2- 172.21.15.90:0/2779674694 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557e74df0770 0x557e74e08b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:24.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.778+0000 7f42ccf65740 1 --2- 172.21.15.90:0/2779674694 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557e74dfd080 0x557e74e15670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:24.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.778+0000 7f42ccf65740 1 --2- 172.21.15.90:0/2779674694 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557e74dfaaf0 0x557e74e15130 unknown :-1 s=CLOSED pgs=928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:24.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.778+0000 7f42ccf65740 1 -- 172.21.15.90:0/2779674694 >> 172.21.15.90:0/2779674694 conn(0x557e74dbaa50 msgr2=0x557e74dee290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:48:24.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.779+0000 7f42ccf65740 1 -- 172.21.15.90:0/2779674694 shutdown_connections 2024-04-02T14:48:24.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.779+0000 7f42ccf65740 1 -- 172.21.15.90:0/2779674694 wait complete. 2024-04-02T14:48:24.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.780+0000 7f42ccf65740 1 -- start start 2024-04-02T14:48:24.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.780+0000 7f42ccf65740 1 Processor -- start 2024-04-02T14:48:24.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.781+0000 7f42ccf65740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557e74df0770 0x557e74dfec10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:24.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.781+0000 7f42ccf65740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557e74dfaaf0 0x557e74dfd460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:24.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.781+0000 7f42ccf65740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557e74dfd080 0x557e74dfd9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:24.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.781+0000 7f42ccf65740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f42c41bdb60 con 0x557e74dfd080 2024-04-02T14:48:24.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.781+0000 7f42ccf65740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f42c41bd490 con 0x557e74df0770 2024-04-02T14:48:24.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.781+0000 7f42ccf65740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f42c4068d40 con 0x557e74dfaaf0 2024-04-02T14:48:24.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.781+0000 7f42ccf65740 10 client.0 _ll_register_callbacks cb 0x7f42bc009db0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:48:24.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.781+0000 7f42c3400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557e74df0770 0x557e74dfec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:24.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.781+0000 7f42c3400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557e74df0770 0x557e74dfec10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:54284/0 (socket says 172.21.15.90:54284) 2024-04-02T14:48:24.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.781+0000 7f42c3400640 1 -- 172.21.15.90:0/956901226 learned_addr learned my addr 172.21.15.90:0/956901226 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:48:24.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.781+0000 7f42c3e00640 1 --2- 172.21.15.90:0/956901226 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557e74dfd080 0x557e74dfd9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:24.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.781+0000 7f42c3400640 1 -- 172.21.15.90:0/956901226 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557e74dfaaf0 msgr2=0x557e74dfd460 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-02T14:48:24.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.782+0000 7f42c3400640 1 --2- 172.21.15.90:0/956901226 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557e74dfaaf0 0x557e74dfd460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:24.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.782+0000 7f42c3400640 1 -- 172.21.15.90:0/956901226 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557e74dfd080 msgr2=0x557e74dfd9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:48:24.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.782+0000 7f42c3400640 1 --2- 172.21.15.90:0/956901226 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557e74dfd080 0x557e74dfd9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:24.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.782+0000 7f42c3400640 1 -- 172.21.15.90:0/956901226 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557e74dfe1d0 con 0x557e74df0770 2024-04-02T14:48:24.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.782+0000 7f42c3e00640 1 --2- 172.21.15.90:0/956901226 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557e74dfd080 0x557e74dfd9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:48:24.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.782+0000 7f42c3400640 1 --2- 172.21.15.90:0/956901226 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557e74df0770 0x557e74dfec10 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7f42bc012760 tx=0x7f42bc004940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=76d762ce91d3fd56 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:48:24.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.782+0000 7f42acc00640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:48:24.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.782+0000 7f42acc00640 1 -- 172.21.15.90:0/956901226 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42bc003df0 con 0x557e74df0770 2024-04-02T14:48:24.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.782+0000 7f42acc00640 1 -- 172.21.15.90:0/956901226 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 8550+0+0 (secure 0 0 0) 0x7f42bc009aa0 con 0x557e74df0770 2024-04-02T14:48:24.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.782+0000 7f42ccf65740 1 -- 172.21.15.90:0/956901226 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f42bc004cd0 con 0x557e74df0770 2024-04-02T14:48:24.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.783+0000 7f42acc00640 1 -- 172.21.15.90:0/956901226 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f42bc0127c0 con 0x557e74df0770 2024-04-02T14:48:24.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.783+0000 7f42ccf65740 10 client.29666 fetch_fsmap learned FSMap version 27 2024-04-02T14:48:24.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.783+0000 7f42ccf65740 1 -- 172.21.15.90:0/956901226 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x557e74e05040 con 0x557e74df0770 2024-04-02T14:48:24.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.783+0000 7f42acc00640 1 -- 172.21.15.90:0/956901226 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f42bc004cd0 con 0x557e74df0770 2024-04-02T14:48:24.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.783+0000 7f42acc00640 1 -- 172.21.15.90:0/956901226 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f42bc024c30 con 0x557e74df0770 2024-04-02T14:48:24.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.783+0000 7f42ccf65740 10 client.29666 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:48:24.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.783+0000 7f42ccf65740 10 client.29666 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-04-02T14:48:24.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.783+0000 7f42ccf65740 20 client.29666 populate_metadata read hostname 'smithi090' 2024-04-02T14:48:24.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.783+0000 7f42ccf65740 1 --2- 172.21.15.90:0/956901226 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x557e74e739a0 0x557e74e75d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:24.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.783+0000 7f42ccf65740 4 client.29666 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-02T14:48:24.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.783+0000 7f42ccf65740 1 -- 172.21.15.90:0/956901226 --> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] -- command(tid 0: {"prefix": "damage ls"}) -- 0x557e74d65c30 con 0x557e74e739a0 2024-04-02T14:48:24.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.784+0000 7f42c2a00640 1 --2- 172.21.15.90:0/956901226 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x557e74e739a0 0x557e74e75d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:24.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.786+0000 7f42acc00640 10 client.29666 ms_handle_connect on v2:172.21.15.92:6836/1270635939 2024-04-02T14:48:24.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.786+0000 7f42c2a00640 1 --2- 172.21.15.90:0/956901226 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x557e74e739a0 0x557e74e75d90 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=6f7ddb24a186c0a8 server_cookie=f30214b04f351e9 in_seq=0 out_seq=0 2024-04-02T14:48:24.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.786+0000 7f42acc00640 1 -- 172.21.15.90:0/956901226 <== mds.3 v2:172.21.15.92:6836/1270635939 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x557e74d65c30 con 0x557e74e739a0 2024-04-02T14:48:24.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.786+0000 7f42acc00640 10 client.29666 handle_command_reply: tid=0 2024-04-02T14:48:24.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.787+0000 7f42ccf65740 1 client.29666 shutdown 2024-04-02T14:48:24.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.787+0000 7f42ccf65740 1 -- 172.21.15.90:0/956901226 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557e74df0770 msgr2=0x557e74dfec10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:48:24.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.787+0000 7f42ccf65740 1 --2- 172.21.15.90:0/956901226 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557e74df0770 0x557e74dfec10 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7f42bc012760 tx=0x7f42bc004940 comp rx=0 tx=0).stop 2024-04-02T14:48:24.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.788+0000 7f42ccf65740 1 -- 172.21.15.90:0/956901226 shutdown_connections 2024-04-02T14:48:24.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.788+0000 7f42ccf65740 1 --2- 172.21.15.90:0/956901226 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x557e74e739a0 0x557e74e75d90 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:24.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.788+0000 7f42ccf65740 1 --2- 172.21.15.90:0/956901226 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557e74dfd080 0x557e74dfd9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:24.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.788+0000 7f42ccf65740 1 --2- 172.21.15.90:0/956901226 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557e74dfaaf0 0x557e74dfd460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:24.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.788+0000 7f42acc00640 0 client.29666 ms_handle_reset on v2:172.21.15.92:6836/1270635939 2024-04-02T14:48:24.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.788+0000 7f42ccf65740 1 --2- 172.21.15.90:0/956901226 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x557e74df0770 0x557e74dfec10 unknown :-1 s=CLOSED pgs=929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:24.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.788+0000 7f42ccf65740 1 -- 172.21.15.90:0/956901226 >> 172.21.15.90:0/956901226 conn(0x557e74dbaa50 msgr2=0x557e74ded010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:48:24.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.788+0000 7f42ccf65740 1 -- 172.21.15.90:0/956901226 shutdown_connections 2024-04-02T14:48:24.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.788+0000 7f42ccf65740 1 -- 172.21.15.90:0/956901226 wait complete. 2024-04-02T14:48:24.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.788+0000 7f42ccf65740 20 client.29666 trim_cache size 0 max 16384 2024-04-02T14:48:24.785 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-02T14:48:24.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.789+0000 7f42ca200640 1 -- 172.21.15.90:0/1012879831 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f42a80a3da0 msgr2=0x7f42a80a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:48:24.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.789+0000 7f42ca200640 1 --2- 172.21.15.90:0/1012879831 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f42a80a3da0 0x7f42a80a6260 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f42b400bfd0 tx=0x7f42b4005b80 comp rx=0 tx=0).stop 2024-04-02T14:48:24.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.789+0000 7f42ca200640 1 -- 172.21.15.90:0/1012879831 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f42c4102450 msgr2=0x7f42c4108f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:48:24.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.789+0000 7f42ca200640 1 --2- 172.21.15.90:0/1012879831 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f42c4102450 0x7f42c4108f20 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f42b0009900 tx=0x7f42b00043b0 comp rx=0 tx=0).stop 2024-04-02T14:48:24.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.789+0000 7f42ca200640 1 -- 172.21.15.90:0/1012879831 shutdown_connections 2024-04-02T14:48:24.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.789+0000 7f42ca200640 1 --2- 172.21.15.90:0/1012879831 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f42a80a3da0 0x7f42a80a6260 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:24.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.790+0000 7f42ca200640 1 --2- 172.21.15.90:0/1012879831 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f42c4107aa0 0x7f42c4109460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:24.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.790+0000 7f42ca200640 1 --2- 172.21.15.90:0/1012879831 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f42c4102450 0x7f42c4108f20 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:24.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.790+0000 7f42ca200640 1 --2- 172.21.15.90:0/1012879831 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f42c4101a90 0x7f42c41089e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:24.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.790+0000 7f42ca200640 1 -- 172.21.15.90:0/1012879831 >> 172.21.15.90:0/1012879831 conn(0x7f42c4078f00 msgr2=0x7f42c4103590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:48:24.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.790+0000 7f42ca200640 1 -- 172.21.15.90:0/1012879831 shutdown_connections 2024-04-02T14:48:24.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:24.790+0000 7f42ca200640 1 -- 172.21.15.90:0/1012879831 wait complete. 2024-04-02T14:48:24.808 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-02T14:48:25.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.104+0000 7f4c27400640 1 -- 172.21.15.90:0/462624912 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4c20103f10 msgr2=0x7f4c201042f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:48:25.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.104+0000 7f4c27400640 1 --2- 172.21.15.90:0/462624912 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4c20103f10 0x7f4c201042f0 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f4c14009900 tx=0x7f4c1402bc40 comp rx=0 tx=0).stop 2024-04-02T14:48:25.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.104+0000 7f4c27400640 1 -- 172.21.15.90:0/462624912 shutdown_connections 2024-04-02T14:48:25.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.104+0000 7f4c27400640 1 --2- 172.21.15.90:0/462624912 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4c20108fe0 0x7f4c20110bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:25.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.104+0000 7f4c27400640 1 --2- 172.21.15.90:0/462624912 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4c201048c0 0x7f4c20108900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:25.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.104+0000 7f4c27400640 1 --2- 172.21.15.90:0/462624912 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4c20103f10 0x7f4c201042f0 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:25.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.104+0000 7f4c27400640 1 -- 172.21.15.90:0/462624912 >> 172.21.15.90:0/462624912 conn(0x7f4c200ff770 msgr2=0x7f4c20101b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:48:25.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.104+0000 7f4c27400640 1 -- 172.21.15.90:0/462624912 shutdown_connections 2024-04-02T14:48:25.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.104+0000 7f4c27400640 1 -- 172.21.15.90:0/462624912 wait complete. 2024-04-02T14:48:25.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.106+0000 7f4c27400640 1 Processor -- start 2024-04-02T14:48:25.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.106+0000 7f4c27400640 1 -- start start 2024-04-02T14:48:25.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.106+0000 7f4c27400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4c20103f10 0x7f4c20076010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:25.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.107+0000 7f4c27400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4c20108fe0 0x7f4c20076550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:25.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.107+0000 7f4c27400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4c20076a90 0x7f4c201aabf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:25.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.107+0000 7f4c27400640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f4c20114140 con 0x7f4c20103f10 2024-04-02T14:48:25.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.107+0000 7f4c27400640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f4c20113fc0 con 0x7f4c20076a90 2024-04-02T14:48:25.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.107+0000 7f4c27400640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f4c201142c0 con 0x7f4c20108fe0 2024-04-02T14:48:25.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.107+0000 7f4c25400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4c20076a90 0x7f4c201aabf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:25.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.107+0000 7f4c24a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4c20103f10 0x7f4c20076010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:25.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.107+0000 7f4c25400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4c20076a90 0x7f4c201aabf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:54296/0 (socket says 172.21.15.90:54296) 2024-04-02T14:48:25.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.107+0000 7f4c25400640 1 -- 172.21.15.90:0/4003639036 learned_addr learned my addr 172.21.15.90:0/4003639036 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:48:25.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.107+0000 7f4c1fe00640 1 --2- 172.21.15.90:0/4003639036 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4c20108fe0 0x7f4c20076550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:25.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.108+0000 7f4c25400640 1 -- 172.21.15.90:0/4003639036 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4c20108fe0 msgr2=0x7f4c20076550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:48:25.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.108+0000 7f4c25400640 1 --2- 172.21.15.90:0/4003639036 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4c20108fe0 0x7f4c20076550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:25.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.108+0000 7f4c25400640 1 -- 172.21.15.90:0/4003639036 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4c20103f10 msgr2=0x7f4c20076010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:48:25.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.108+0000 7f4c25400640 1 --2- 172.21.15.90:0/4003639036 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4c20103f10 0x7f4c20076010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:25.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.108+0000 7f4c25400640 1 -- 172.21.15.90:0/4003639036 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c201ab130 con 0x7f4c20076a90 2024-04-02T14:48:25.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.108+0000 7f4c25400640 1 --2- 172.21.15.90:0/4003639036 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4c20076a90 0x7f4c201aabf0 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f4c1800eb10 tx=0x7f4c1800ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:48:25.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.109+0000 7f4c1d600640 1 -- 172.21.15.90:0/4003639036 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c1800ce20 con 0x7f4c20076a90 2024-04-02T14:48:25.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.109+0000 7f4c27400640 1 -- 172.21.15.90:0/4003639036 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c201ab420 con 0x7f4c20076a90 2024-04-02T14:48:25.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.110+0000 7f4c27400640 1 -- 172.21.15.90:0/4003639036 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c201ab940 con 0x7f4c20076a90 2024-04-02T14:48:25.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.110+0000 7f4c1d600640 1 -- 172.21.15.90:0/4003639036 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4c18005020 con 0x7f4c20076a90 2024-04-02T14:48:25.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.110+0000 7f4c1d600640 1 -- 172.21.15.90:0/4003639036 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c1800ce20 con 0x7f4c20076a90 2024-04-02T14:48:25.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.115+0000 7f4c2a050740 1 Processor -- start 2024-04-02T14:48:25.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.115+0000 7f4c1d600640 1 -- 172.21.15.90:0/4003639036 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f4c18004a40 con 0x7f4c20076a90 2024-04-02T14:48:25.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.117+0000 7f4c2a050740 1 -- start start 2024-04-02T14:48:25.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.117+0000 7f4c2a050740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d0ce9529f0 0x55d0ce951740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:25.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.117+0000 7f4c2a050740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d0ce951d10 0x55d0ce952190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:25.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.118+0000 7f4c2a050740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d0ce95b080 0x55d0ce95d470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:25.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.118+0000 7f4c2a050740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55d0ce892400 con 0x55d0ce95b080 2024-04-02T14:48:25.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.118+0000 7f4c2a050740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55d0ce876f10 con 0x55d0ce9529f0 2024-04-02T14:48:25.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.118+0000 7f4c2a050740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55d0ce8b66f0 con 0x55d0ce951d10 2024-04-02T14:48:25.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.118+0000 7f4c24a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d0ce951d10 0x55d0ce952190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:25.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.118+0000 7f4c1fe00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d0ce9529f0 0x55d0ce951740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:25.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.118+0000 7f4c1fe00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d0ce9529f0 0x55d0ce951740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:54308/0 (socket says 172.21.15.90:54308) 2024-04-02T14:48:25.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.118+0000 7f4c1fe00640 1 -- 172.21.15.90:0/1593409745 learned_addr learned my addr 172.21.15.90:0/1593409745 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:48:25.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.118+0000 7f4c1fe00640 1 --2- 172.21.15.90:0/1593409745 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d0ce95b080 0x55d0ce95d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:25.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.119+0000 7f4c24a00640 1 -- 172.21.15.90:0/1593409745 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d0ce9529f0 msgr2=0x55d0ce951740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:48:25.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.119+0000 7f4c24a00640 1 --2- 172.21.15.90:0/1593409745 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d0ce9529f0 0x55d0ce951740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:25.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.119+0000 7f4c24a00640 1 -- 172.21.15.90:0/1593409745 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d0ce95b080 msgr2=0x55d0ce95d470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:48:25.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.119+0000 7f4c24a00640 1 --2- 172.21.15.90:0/1593409745 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d0ce95b080 0x55d0ce95d470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:25.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.119+0000 7f4c24a00640 1 -- 172.21.15.90:0/1593409745 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d0ce5ca3f0 con 0x55d0ce951d10 2024-04-02T14:48:25.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.119+0000 7f4c1fe00640 1 --2- 172.21.15.90:0/1593409745 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d0ce95b080 0x55d0ce95d470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:48:25.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.119+0000 7f4c24a00640 1 --2- 172.21.15.90:0/1593409745 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d0ce951d10 0x55d0ce952190 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f4c1402c180 tx=0x7f4c14000bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=adcc0438a324e14b server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:48:25.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.120+0000 7f4c1d600640 1 --2- 172.21.15.90:0/4003639036 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f4c080a3fa0 0x7f4c080a6460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:25.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.120+0000 7f4c1d600640 1 -- 172.21.15.90:0/4003639036 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 8550+0+0 (secure 0 0 0) 0x7f4c180d08a0 con 0x7f4c20076a90 2024-04-02T14:48:25.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.120+0000 7f4c11c00640 1 -- 172.21.15.90:0/1593409745 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c14042070 con 0x55d0ce951d10 2024-04-02T14:48:25.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.120+0000 7f4c1fe00640 1 --2- 172.21.15.90:0/4003639036 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f4c080a3fa0 0x7f4c080a6460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:25.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.120+0000 7f4c11c00640 1 -- 172.21.15.90:0/1593409745 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4c14004240 con 0x55d0ce951d10 2024-04-02T14:48:25.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.120+0000 7f4c11c00640 1 -- 172.21.15.90:0/1593409745 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c14004f80 con 0x55d0ce951d10 2024-04-02T14:48:25.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.121+0000 7f4c2a050740 1 -- 172.21.15.90:0/1593409745 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d0ce951d10 msgr2=0x55d0ce952190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:48:25.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.121+0000 7f4c1fe00640 1 --2- 172.21.15.90:0/4003639036 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f4c080a3fa0 0x7f4c080a6460 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f4c0c009fd0 tx=0x7f4c0c008040 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:48:25.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.121+0000 7f4c2a050740 1 --2- 172.21.15.90:0/1593409745 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d0ce951d10 0x55d0ce952190 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f4c1402c180 tx=0x7f4c14000bc0 comp rx=0 tx=0).stop 2024-04-02T14:48:25.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.121+0000 7f4c2a050740 1 -- 172.21.15.90:0/1593409745 shutdown_connections 2024-04-02T14:48:25.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.121+0000 7f4c2a050740 1 --2- 172.21.15.90:0/1593409745 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d0ce95b080 0x55d0ce95d470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:25.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.121+0000 7f4c2a050740 1 --2- 172.21.15.90:0/1593409745 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d0ce951d10 0x55d0ce952190 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:25.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.121+0000 7f4c2a050740 1 --2- 172.21.15.90:0/1593409745 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d0ce9529f0 0x55d0ce951740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:25.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.121+0000 7f4c2a050740 1 -- 172.21.15.90:0/1593409745 >> 172.21.15.90:0/1593409745 conn(0x55d0ce918a50 msgr2=0x55d0ce94c760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:48:25.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.121+0000 7f4c2a050740 1 -- 172.21.15.90:0/1593409745 shutdown_connections 2024-04-02T14:48:25.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.121+0000 7f4c2a050740 1 -- 172.21.15.90:0/1593409745 wait complete. 2024-04-02T14:48:25.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.122+0000 7f4c2a050740 1 -- start start 2024-04-02T14:48:25.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.122+0000 7f4c2a050740 1 Processor -- start 2024-04-02T14:48:25.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.123+0000 7f4c2a050740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d0ce951d10 0x55d0ce957fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:25.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.123+0000 7f4c2a050740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d0ce9529f0 0x55d0ce9584f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:25.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.123+0000 7f4c2a050740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d0ce95b080 0x55d0ce958a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:25.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.123+0000 7f4c2a050740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55d0ce8b26a0 con 0x55d0ce9529f0 2024-04-02T14:48:25.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.123+0000 7f4c2a050740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55d0ce85e980 con 0x55d0ce95b080 2024-04-02T14:48:25.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.123+0000 7f4c2a050740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55d0ce871cb0 con 0x55d0ce951d10 2024-04-02T14:48:25.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.123+0000 7f4c2a050740 10 client.0 _ll_register_callbacks cb 0x55d0ce8716a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:48:25.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.123+0000 7f4c24a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d0ce951d10 0x55d0ce957fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:25.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.123+0000 7f4c25400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d0ce95b080 0x55d0ce958a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:25.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.124+0000 7f4c24a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d0ce951d10 0x55d0ce957fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:38224/0 (socket says 172.21.15.90:38224) 2024-04-02T14:48:25.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.124+0000 7f4c24a00640 1 -- 172.21.15.90:0/2011250608 learned_addr learned my addr 172.21.15.90:0/2011250608 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:48:25.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.124+0000 7f4c25400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d0ce95b080 0x55d0ce958a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:54310/0 (socket says 172.21.15.90:54310) 2024-04-02T14:48:25.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.123+0000 7f4c1fe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d0ce9529f0 0x55d0ce9584f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:25.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.124+0000 7f4c24a00640 1 -- 172.21.15.90:0/2011250608 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d0ce95b080 msgr2=0x55d0ce958a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:48:25.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.124+0000 7f4c24a00640 1 --2- 172.21.15.90:0/2011250608 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d0ce95b080 0x55d0ce958a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:25.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.124+0000 7f4c24a00640 1 -- 172.21.15.90:0/2011250608 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d0ce9529f0 msgr2=0x55d0ce9584f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:48:25.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.124+0000 7f4c24a00640 1 --2- 172.21.15.90:0/2011250608 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d0ce9529f0 0x55d0ce9584f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:25.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.124+0000 7f4c24a00640 1 -- 172.21.15.90:0/2011250608 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d0ce6947d0 con 0x55d0ce951d10 2024-04-02T14:48:25.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.126+0000 7f4c24a00640 1 --2- 172.21.15.90:0/2011250608 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d0ce951d10 0x55d0ce957fb0 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f4c14036040 tx=0x7f4c140058f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6face3ee409e6797 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:48:25.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.126+0000 7f4c06000640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:48:25.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.126+0000 7f4c06000640 1 -- 172.21.15.90:0/2011250608 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c14042040 con 0x55d0ce951d10 2024-04-02T14:48:25.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.126+0000 7f4c06000640 1 -- 172.21.15.90:0/2011250608 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 8550+0+0 (secure 0 0 0) 0x7f4c1402cca0 con 0x55d0ce951d10 2024-04-02T14:48:25.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.127+0000 7f4c2a050740 1 -- 172.21.15.90:0/2011250608 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4c14004240 con 0x55d0ce951d10 2024-04-02T14:48:25.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.127+0000 7f4c06000640 1 -- 172.21.15.90:0/2011250608 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f4c14004f80 con 0x55d0ce951d10 2024-04-02T14:48:25.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.128+0000 7f4c2a050740 10 client.29461 fetch_fsmap learned FSMap version 27 2024-04-02T14:48:25.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.128+0000 7f4c2a050740 1 -- 172.21.15.90:0/2011250608 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55d0ce969e70 con 0x55d0ce951d10 2024-04-02T14:48:25.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.129+0000 7f4c06000640 1 -- 172.21.15.90:0/2011250608 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4c14004240 con 0x55d0ce951d10 2024-04-02T14:48:25.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.129+0000 7f4c06000640 1 -- 172.21.15.90:0/2011250608 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f4c14047b90 con 0x55d0ce951d10 2024-04-02T14:48:25.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.129+0000 7f4c2a050740 10 client.29461 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:48:25.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.129+0000 7f4c2a050740 10 client.29461 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-04-02T14:48:25.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.129+0000 7f4c2a050740 20 client.29461 populate_metadata read hostname 'smithi090' 2024-04-02T14:48:25.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.129+0000 7f4c2a050740 1 --2- 172.21.15.90:0/2011250608 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x55d0ce96b1c0 0x55d0ce96d5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:25.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.129+0000 7f4c2a050740 4 client.29461 mds_command: new command op to 24547 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:48:25.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.129+0000 7f4c2a050740 1 -- 172.21.15.90:0/2011250608 --> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d0ce876040 con 0x55d0ce96b1c0 2024-04-02T14:48:25.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.129+0000 7f4c1fe00640 1 --2- 172.21.15.90:0/2011250608 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x55d0ce96b1c0 0x55d0ce96d5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:25.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.130+0000 7f4c06000640 10 client.29461 ms_handle_connect on v2:172.21.15.177:6838/664895663 2024-04-02T14:48:25.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.130+0000 7f4c1fe00640 1 --2- 172.21.15.90:0/2011250608 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x55d0ce96b1c0 0x55d0ce96d5b0 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=6c62b76d5867c593 server_cookie=77eb33f40594a3c8 in_seq=0 out_seq=0 2024-04-02T14:48:25.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.131+0000 7f4c06000640 1 -- 172.21.15.90:0/2011250608 <== mds.2 v2:172.21.15.177:6838/664895663 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d0ce876040 con 0x55d0ce96b1c0 2024-04-02T14:48:25.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.131+0000 7f4c06000640 10 client.29461 handle_command_reply: tid=0 2024-04-02T14:48:25.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.131+0000 7f4c2a050740 1 client.29461 shutdown 2024-04-02T14:48:25.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.132+0000 7f4c2a050740 1 -- 172.21.15.90:0/2011250608 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d0ce951d10 msgr2=0x55d0ce957fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:48:25.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.132+0000 7f4c2a050740 1 --2- 172.21.15.90:0/2011250608 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d0ce951d10 0x55d0ce957fb0 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f4c14036040 tx=0x7f4c140058f0 comp rx=0 tx=0).stop 2024-04-02T14:48:25.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.132+0000 7f4c2a050740 1 -- 172.21.15.90:0/2011250608 shutdown_connections 2024-04-02T14:48:25.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.132+0000 7f4c2a050740 1 --2- 172.21.15.90:0/2011250608 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x55d0ce96b1c0 0x55d0ce96d5b0 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:25.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.132+0000 7f4c2a050740 1 --2- 172.21.15.90:0/2011250608 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d0ce95b080 0x55d0ce958a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:25.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.132+0000 7f4c2a050740 1 --2- 172.21.15.90:0/2011250608 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d0ce9529f0 0x55d0ce9584f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:25.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.132+0000 7f4c2a050740 1 --2- 172.21.15.90:0/2011250608 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d0ce951d10 0x55d0ce957fb0 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:25.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.132+0000 7f4c2a050740 1 -- 172.21.15.90:0/2011250608 >> 172.21.15.90:0/2011250608 conn(0x55d0ce918a50 msgr2=0x55d0ce94e4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:48:25.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.132+0000 7f4c06000640 0 client.29461 ms_handle_reset on v2:172.21.15.177:6838/664895663 2024-04-02T14:48:25.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.133+0000 7f4c2a050740 1 -- 172.21.15.90:0/2011250608 shutdown_connections 2024-04-02T14:48:25.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.133+0000 7f4c2a050740 1 -- 172.21.15.90:0/2011250608 wait complete. 2024-04-02T14:48:25.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.133+0000 7f4c2a050740 20 client.29461 trim_cache size 0 max 16384 2024-04-02T14:48:25.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.157+0000 7f4c2a050740 1 Processor -- start 2024-04-02T14:48:25.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.158+0000 7f4c2a050740 1 -- start start 2024-04-02T14:48:25.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.158+0000 7f4c2a050740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d0ce951d10 0x55d0ce95a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:25.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.158+0000 7f4c2a050740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d0ce9529f0 0x55d0ce966af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:25.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.158+0000 7f4c2a050740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d0ce95b080 0x55d0ce957130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:25.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.158+0000 7f4c2a050740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f4c0000e7e0 con 0x55d0ce9529f0 2024-04-02T14:48:25.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.158+0000 7f4c2a050740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f4c00010320 con 0x55d0ce95b080 2024-04-02T14:48:25.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.158+0000 7f4c2a050740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f4c0000eeb0 con 0x55d0ce951d10 2024-04-02T14:48:25.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.158+0000 7f4c25400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d0ce95b080 0x55d0ce957130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:25.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.158+0000 7f4c24a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d0ce951d10 0x55d0ce95a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:25.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.159+0000 7f4c25400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d0ce95b080 0x55d0ce957130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:54316/0 (socket says 172.21.15.90:54316) 2024-04-02T14:48:25.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.159+0000 7f4c25400640 1 -- 172.21.15.90:0/3019320159 learned_addr learned my addr 172.21.15.90:0/3019320159 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:48:25.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.159+0000 7f4c25400640 1 -- 172.21.15.90:0/3019320159 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d0ce951d10 msgr2=0x55d0ce95a6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:48:25.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.159+0000 7f4c25400640 1 --2- 172.21.15.90:0/3019320159 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d0ce951d10 0x55d0ce95a6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:25.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.159+0000 7f4c25400640 1 -- 172.21.15.90:0/3019320159 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d0ce9529f0 msgr2=0x55d0ce966af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:48:25.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.159+0000 7f4c25400640 1 --2- 172.21.15.90:0/3019320159 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d0ce9529f0 0x55d0ce966af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:25.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.160+0000 7f4c25400640 1 -- 172.21.15.90:0/3019320159 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d0ce96d920 con 0x55d0ce95b080 2024-04-02T14:48:25.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.160+0000 7f4c25400640 1 --2- 172.21.15.90:0/3019320159 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d0ce95b080 0x55d0ce957130 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7f4c18098c20 tx=0x7f4c1809cf80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=32993c330a1a36f1 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:48:25.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.160+0000 7f4c11200640 1 -- 172.21.15.90:0/3019320159 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c180a5430 con 0x55d0ce95b080 2024-04-02T14:48:25.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.160+0000 7f4c11200640 1 -- 172.21.15.90:0/3019320159 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4c180870b0 con 0x55d0ce95b080 2024-04-02T14:48:25.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.160+0000 7f4c11200640 1 -- 172.21.15.90:0/3019320159 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c180a4fa0 con 0x55d0ce95b080 2024-04-02T14:48:25.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.161+0000 7f4c2a050740 1 -- 172.21.15.90:0/3019320159 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d0ce95b080 msgr2=0x55d0ce957130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:48:25.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.161+0000 7f4c2a050740 1 --2- 172.21.15.90:0/3019320159 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d0ce95b080 0x55d0ce957130 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7f4c18098c20 tx=0x7f4c1809cf80 comp rx=0 tx=0).stop 2024-04-02T14:48:25.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.162+0000 7f4c2a050740 1 -- 172.21.15.90:0/3019320159 shutdown_connections 2024-04-02T14:48:25.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.162+0000 7f4c2a050740 1 --2- 172.21.15.90:0/3019320159 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d0ce95b080 0x55d0ce957130 unknown :-1 s=CLOSED pgs=931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:25.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.162+0000 7f4c2a050740 1 --2- 172.21.15.90:0/3019320159 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d0ce9529f0 0x55d0ce966af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:25.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.162+0000 7f4c2a050740 1 --2- 172.21.15.90:0/3019320159 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d0ce951d10 0x55d0ce95a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:25.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.162+0000 7f4c2a050740 1 -- 172.21.15.90:0/3019320159 >> 172.21.15.90:0/3019320159 conn(0x55d0ce918a50 msgr2=0x55d0ce94b010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:48:25.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.162+0000 7f4c2a050740 1 -- 172.21.15.90:0/3019320159 shutdown_connections 2024-04-02T14:48:25.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.162+0000 7f4c2a050740 1 -- 172.21.15.90:0/3019320159 wait complete. 2024-04-02T14:48:25.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.163+0000 7f4c2a050740 1 -- start start 2024-04-02T14:48:25.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.163+0000 7f4c2a050740 1 Processor -- start 2024-04-02T14:48:25.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.164+0000 7f4c2a050740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d0ce951d10 0x55d0ce954d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:25.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.164+0000 7f4c2a050740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d0ce9529f0 0x55d0ce955260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:25.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.164+0000 7f4c2a050740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d0ce95b080 0x55d0ce9557a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:25.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.164+0000 7f4c2a050740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f4c0000fc50 con 0x55d0ce9529f0 2024-04-02T14:48:25.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.164+0000 7f4c2a050740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f4c0000f580 con 0x55d0ce951d10 2024-04-02T14:48:25.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.164+0000 7f4c2a050740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f4c0000c7e0 con 0x55d0ce95b080 2024-04-02T14:48:25.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.164+0000 7f4c2a050740 10 client.0 _ll_register_callbacks cb 0x7f4c140095f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:48:25.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.164+0000 7f4c24a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d0ce951d10 0x55d0ce954d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:25.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.164+0000 7f4c24a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d0ce951d10 0x55d0ce954d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:54326/0 (socket says 172.21.15.90:54326) 2024-04-02T14:48:25.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.164+0000 7f4c24a00640 1 -- 172.21.15.90:0/4236296846 learned_addr learned my addr 172.21.15.90:0/4236296846 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:48:25.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.164+0000 7f4c25400640 1 --2- 172.21.15.90:0/4236296846 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d0ce95b080 0x55d0ce9557a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:25.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.164+0000 7f4c1fe00640 1 --2- 172.21.15.90:0/4236296846 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d0ce9529f0 0x55d0ce955260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:25.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.164+0000 7f4c24a00640 1 -- 172.21.15.90:0/4236296846 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d0ce95b080 msgr2=0x55d0ce9557a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:48:25.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.165+0000 7f4c24a00640 1 --2- 172.21.15.90:0/4236296846 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d0ce95b080 0x55d0ce9557a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:25.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.165+0000 7f4c24a00640 1 -- 172.21.15.90:0/4236296846 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d0ce9529f0 msgr2=0x55d0ce955260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:48:25.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.165+0000 7f4c24a00640 1 --2- 172.21.15.90:0/4236296846 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d0ce9529f0 0x55d0ce955260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:25.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.165+0000 7f4c24a00640 1 -- 172.21.15.90:0/4236296846 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d0ce9d03f0 con 0x55d0ce951d10 2024-04-02T14:48:25.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.165+0000 7f4c25400640 1 --2- 172.21.15.90:0/4236296846 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d0ce95b080 0x55d0ce9557a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:48:25.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.165+0000 7f4c24a00640 1 --2- 172.21.15.90:0/4236296846 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d0ce951d10 0x55d0ce954d20 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f4c14050520 tx=0x7f4c14004a00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ac3ea406d80dbce5 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:48:25.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.165+0000 7f4c06000640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:48:25.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.165+0000 7f4c06000640 1 -- 172.21.15.90:0/4236296846 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c14042040 con 0x55d0ce951d10 2024-04-02T14:48:25.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.165+0000 7f4c2a050740 1 -- 172.21.15.90:0/4236296846 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4c180870b0 con 0x55d0ce951d10 2024-04-02T14:48:25.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.165+0000 7f4c06000640 1 -- 172.21.15.90:0/4236296846 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 8550+0+0 (secure 0 0 0) 0x7f4c14005330 con 0x55d0ce951d10 2024-04-02T14:48:25.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.165+0000 7f4c06000640 1 -- 172.21.15.90:0/4236296846 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f4c14005530 con 0x55d0ce951d10 2024-04-02T14:48:25.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.165+0000 7f4c2a050740 10 client.29684 fetch_fsmap learned FSMap version 27 2024-04-02T14:48:25.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.165+0000 7f4c2a050740 1 -- 172.21.15.90:0/4236296846 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55d0ce9d16d0 con 0x55d0ce951d10 2024-04-02T14:48:25.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.166+0000 7f4c06000640 1 -- 172.21.15.90:0/4236296846 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4c180870b0 con 0x55d0ce951d10 2024-04-02T14:48:25.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.166+0000 7f4c06000640 1 -- 172.21.15.90:0/4236296846 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f4c140478f0 con 0x55d0ce951d10 2024-04-02T14:48:25.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.166+0000 7f4c2a050740 10 client.29684 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:48:25.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.166+0000 7f4c2a050740 10 client.29684 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-04-02T14:48:25.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.166+0000 7f4c2a050740 20 client.29684 populate_metadata read hostname 'smithi090' 2024-04-02T14:48:25.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.166+0000 7f4c2a050740 1 --2- 172.21.15.90:0/4236296846 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x55d0ce9d1d60 0x55d0ce9d4180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:25.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.166+0000 7f4c2a050740 4 client.29684 mds_command: new command op to 24547 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-02T14:48:25.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.166+0000 7f4c2a050740 1 -- 172.21.15.90:0/4236296846 --> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55d0ce8c3c30 con 0x55d0ce9d1d60 2024-04-02T14:48:25.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.166+0000 7f4c1fe00640 1 --2- 172.21.15.90:0/4236296846 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x55d0ce9d1d60 0x55d0ce9d4180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:25.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.167+0000 7f4c1fe00640 1 --2- 172.21.15.90:0/4236296846 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x55d0ce9d1d60 0x55d0ce9d4180 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=e1b0157bd7e821d5 server_cookie=5d1aa51c8c32f77c in_seq=0 out_seq=0 2024-04-02T14:48:25.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.167+0000 7f4c06000640 10 client.29684 ms_handle_connect on v2:172.21.15.177:6838/664895663 2024-04-02T14:48:25.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.167+0000 7f4c06000640 1 -- 172.21.15.90:0/4236296846 <== mds.2 v2:172.21.15.177:6838/664895663 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55d0ce8c3c30 con 0x55d0ce9d1d60 2024-04-02T14:48:25.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.167+0000 7f4c06000640 10 client.29684 handle_command_reply: tid=0 2024-04-02T14:48:25.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.167+0000 7f4c2a050740 1 client.29684 shutdown 2024-04-02T14:48:25.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.167+0000 7f4c2a050740 1 -- 172.21.15.90:0/4236296846 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d0ce951d10 msgr2=0x55d0ce954d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:48:25.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.167+0000 7f4c2a050740 1 --2- 172.21.15.90:0/4236296846 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d0ce951d10 0x55d0ce954d20 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f4c14050520 tx=0x7f4c14004a00 comp rx=0 tx=0).stop 2024-04-02T14:48:25.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.167+0000 7f4c2a050740 1 -- 172.21.15.90:0/4236296846 shutdown_connections 2024-04-02T14:48:25.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.167+0000 7f4c2a050740 1 --2- 172.21.15.90:0/4236296846 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x55d0ce9d1d60 0x55d0ce9d4180 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:25.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.167+0000 7f4c2a050740 1 --2- 172.21.15.90:0/4236296846 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d0ce95b080 0x55d0ce9557a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:25.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.167+0000 7f4c2a050740 1 --2- 172.21.15.90:0/4236296846 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d0ce9529f0 0x55d0ce955260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:25.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.167+0000 7f4c2a050740 1 --2- 172.21.15.90:0/4236296846 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d0ce951d10 0x55d0ce954d20 unknown :-1 s=CLOSED pgs=932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:25.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.167+0000 7f4c2a050740 1 -- 172.21.15.90:0/4236296846 >> 172.21.15.90:0/4236296846 conn(0x55d0ce918a50 msgr2=0x55d0ce958800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:48:25.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.167+0000 7f4c06000640 0 client.29684 ms_handle_reset on v2:172.21.15.177:6838/664895663 2024-04-02T14:48:25.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.168+0000 7f4c2a050740 1 -- 172.21.15.90:0/4236296846 shutdown_connections 2024-04-02T14:48:25.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.168+0000 7f4c2a050740 1 -- 172.21.15.90:0/4236296846 wait complete. 2024-04-02T14:48:25.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.168+0000 7f4c2a050740 20 client.29684 trim_cache size 0 max 16384 2024-04-02T14:48:25.166 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-02T14:48:25.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.169+0000 7f4c27400640 1 -- 172.21.15.90:0/4003639036 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f4c080a3fa0 msgr2=0x7f4c080a6460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:48:25.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.169+0000 7f4c27400640 1 --2- 172.21.15.90:0/4003639036 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f4c080a3fa0 0x7f4c080a6460 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f4c0c009fd0 tx=0x7f4c0c008040 comp rx=0 tx=0).stop 2024-04-02T14:48:25.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.169+0000 7f4c27400640 1 -- 172.21.15.90:0/4003639036 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4c20076a90 msgr2=0x7f4c201aabf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:48:25.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.169+0000 7f4c27400640 1 --2- 172.21.15.90:0/4003639036 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4c20076a90 0x7f4c201aabf0 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f4c1800eb10 tx=0x7f4c1800ef10 comp rx=0 tx=0).stop 2024-04-02T14:48:25.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.170+0000 7f4c27400640 1 -- 172.21.15.90:0/4003639036 shutdown_connections 2024-04-02T14:48:25.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.170+0000 7f4c27400640 1 --2- 172.21.15.90:0/4003639036 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f4c080a3fa0 0x7f4c080a6460 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:25.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.170+0000 7f4c27400640 1 --2- 172.21.15.90:0/4003639036 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4c20076a90 0x7f4c201aabf0 unknown :-1 s=CLOSED pgs=930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:25.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.170+0000 7f4c27400640 1 --2- 172.21.15.90:0/4003639036 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4c20108fe0 0x7f4c20076550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:25.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.170+0000 7f4c27400640 1 --2- 172.21.15.90:0/4003639036 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4c20103f10 0x7f4c20076010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:25.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.170+0000 7f4c27400640 1 -- 172.21.15.90:0/4003639036 >> 172.21.15.90:0/4003639036 conn(0x7f4c200ff770 msgr2=0x7f4c20101ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:48:25.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.171+0000 7f4c27400640 1 -- 172.21.15.90:0/4003639036 shutdown_connections 2024-04-02T14:48:25.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.171+0000 7f4c27400640 1 -- 172.21.15.90:0/4003639036 wait complete. 2024-04-02T14:48:25.187 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-02T14:48:25.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.448+0000 7f1b00200640 1 -- 172.21.15.90:0/2966800228 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1af8073ea0 msgr2=0x7f1af8074280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:48:25.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.448+0000 7f1b00200640 1 --2- 172.21.15.90:0/2966800228 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1af8073ea0 0x7f1af8074280 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f1aec009900 tx=0x7f1aec02bcc0 comp rx=0 tx=0).stop 2024-04-02T14:48:25.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.448+0000 7f1b00200640 1 -- 172.21.15.90:0/2966800228 shutdown_connections 2024-04-02T14:48:25.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.448+0000 7f1b00200640 1 --2- 172.21.15.90:0/2966800228 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1af8112a90 0x7f1af8114e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:25.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.448+0000 7f1b00200640 1 --2- 172.21.15.90:0/2966800228 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1af80747c0 0x7f1af8112550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:25.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.448+0000 7f1b00200640 1 --2- 172.21.15.90:0/2966800228 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1af8073ea0 0x7f1af8074280 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:25.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.448+0000 7f1b00200640 1 -- 172.21.15.90:0/2966800228 >> 172.21.15.90:0/2966800228 conn(0x7f1af80fd590 msgr2=0x7f1af80ff9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:48:25.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.448+0000 7f1b00200640 1 -- 172.21.15.90:0/2966800228 shutdown_connections 2024-04-02T14:48:25.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.448+0000 7f1b00200640 1 -- 172.21.15.90:0/2966800228 wait complete. 2024-04-02T14:48:25.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.449+0000 7f1b00200640 1 Processor -- start 2024-04-02T14:48:25.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.450+0000 7f1b00200640 1 -- start start 2024-04-02T14:48:25.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.450+0000 7f1b00200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1af8073ea0 0x7f1af81a46c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:25.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.450+0000 7f1b00200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1af80747c0 0x7f1af81a4c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:25.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.450+0000 7f1b00200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1af8112a90 0x7f1af819e870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:25.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.450+0000 7f1b00200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f1af81184d0 con 0x7f1af80747c0 2024-04-02T14:48:25.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.450+0000 7f1b00200640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f1af8118350 con 0x7f1af8073ea0 2024-04-02T14:48:25.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.450+0000 7f1b00200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f1af8118650 con 0x7f1af8112a90 2024-04-02T14:48:25.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.450+0000 7f1afd800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1af8073ea0 0x7f1af81a46c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:25.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.450+0000 7f1afd800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1af8073ea0 0x7f1af81a46c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:54350/0 (socket says 172.21.15.90:54350) 2024-04-02T14:48:25.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.450+0000 7f1afd800640 1 -- 172.21.15.90:0/3982679153 learned_addr learned my addr 172.21.15.90:0/3982679153 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:48:25.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.450+0000 7f1afe200640 1 --2- 172.21.15.90:0/3982679153 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1af8112a90 0x7f1af819e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:25.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.450+0000 7f1afce00640 1 --2- 172.21.15.90:0/3982679153 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1af80747c0 0x7f1af81a4c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:25.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.451+0000 7f1afe200640 1 -- 172.21.15.90:0/3982679153 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1af8073ea0 msgr2=0x7f1af81a46c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:48:25.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.451+0000 7f1afe200640 1 --2- 172.21.15.90:0/3982679153 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1af8073ea0 0x7f1af81a46c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:25.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.451+0000 7f1afe200640 1 -- 172.21.15.90:0/3982679153 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1af80747c0 msgr2=0x7f1af81a4c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:48:25.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.451+0000 7f1afe200640 1 --2- 172.21.15.90:0/3982679153 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1af80747c0 0x7f1af81a4c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:25.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.451+0000 7f1afe200640 1 -- 172.21.15.90:0/3982679153 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1af819eff0 con 0x7f1af8112a90 2024-04-02T14:48:25.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.451+0000 7f1afd800640 1 --2- 172.21.15.90:0/3982679153 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1af8073ea0 0x7f1af81a46c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:48:25.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.451+0000 7f1afe200640 1 --2- 172.21.15.90:0/3982679153 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1af8112a90 0x7f1af819e870 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f1af0005fd0 tx=0x7f1af000c6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:48:25.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.452+0000 7f1af6000640 1 -- 172.21.15.90:0/3982679153 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1af0003b70 con 0x7f1af8112a90 2024-04-02T14:48:25.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.452+0000 7f1af6000640 1 -- 172.21.15.90:0/3982679153 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1af0003d10 con 0x7f1af8112a90 2024-04-02T14:48:25.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.452+0000 7f1af6000640 1 -- 172.21.15.90:0/3982679153 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1af0010830 con 0x7f1af8112a90 2024-04-02T14:48:25.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.452+0000 7f1b00200640 1 -- 172.21.15.90:0/3982679153 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1af819f2e0 con 0x7f1af8112a90 2024-04-02T14:48:25.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.452+0000 7f1b00200640 1 -- 172.21.15.90:0/3982679153 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1af81ab580 con 0x7f1af8112a90 2024-04-02T14:48:25.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.456+0000 7f1b02e13740 1 Processor -- start 2024-04-02T14:48:25.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.458+0000 7f1af6000640 1 -- 172.21.15.90:0/3982679153 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f1af0004020 con 0x7f1af8112a90 2024-04-02T14:48:25.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.459+0000 7f1b02e13740 1 -- start start 2024-04-02T14:48:25.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.459+0000 7f1b02e13740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a6785c39f0 0x55a6785c2740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:25.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.459+0000 7f1b02e13740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a6785c2d10 0x55a6785c3190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:25.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.459+0000 7f1b02e13740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a6785cc080 0x55a6785ce470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:25.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.459+0000 7f1b02e13740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55a678503400 con 0x55a6785c2d10 2024-04-02T14:48:25.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.459+0000 7f1b02e13740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55a6784e7f10 con 0x55a6785c39f0 2024-04-02T14:48:25.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.459+0000 7f1b02e13740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55a6785276f0 con 0x55a6785cc080 2024-04-02T14:48:25.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.459+0000 7f1afce00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a6785c39f0 0x55a6785c2740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:25.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.459+0000 7f1afce00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a6785c39f0 0x55a6785c2740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:54356/0 (socket says 172.21.15.90:54356) 2024-04-02T14:48:25.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.459+0000 7f1afce00640 1 -- 172.21.15.90:0/1915212145 learned_addr learned my addr 172.21.15.90:0/1915212145 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:48:25.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.459+0000 7f1afd800640 1 --2- 172.21.15.90:0/1915212145 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a6785c2d10 0x55a6785c3190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:25.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.459+0000 7f1afce00640 1 --2- 172.21.15.90:0/1915212145 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a6785cc080 0x55a6785ce470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:25.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.460+0000 7f1afce00640 1 -- 172.21.15.90:0/1915212145 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a6785cc080 msgr2=0x55a6785ce470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:48:25.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.460+0000 7f1afce00640 1 --2- 172.21.15.90:0/1915212145 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a6785cc080 0x55a6785ce470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:25.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.460+0000 7f1afce00640 1 -- 172.21.15.90:0/1915212145 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a6785c2d10 msgr2=0x55a6785c3190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:48:25.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.460+0000 7f1afce00640 1 --2- 172.21.15.90:0/1915212145 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a6785c2d10 0x55a6785c3190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:25.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.460+0000 7f1afce00640 1 -- 172.21.15.90:0/1915212145 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a67823b3f0 con 0x55a6785c39f0 2024-04-02T14:48:25.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.460+0000 7f1afd800640 1 --2- 172.21.15.90:0/1915212145 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a6785c2d10 0x55a6785c3190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:48:25.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.460+0000 7f1afce00640 1 --2- 172.21.15.90:0/1915212145 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a6785c39f0 0x55a6785c2740 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f1ae4013560 tx=0x7f1ae4005850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f952579134a09670 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:48:25.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.460+0000 7f1aeaa00640 1 -- 172.21.15.90:0/1915212145 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ae4004050 con 0x55a6785c39f0 2024-04-02T14:48:25.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.460+0000 7f1aeaa00640 1 -- 172.21.15.90:0/1915212145 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1ae4005500 con 0x55a6785c39f0 2024-04-02T14:48:25.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.460+0000 7f1aeaa00640 1 -- 172.21.15.90:0/1915212145 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ae4004ce0 con 0x55a6785c39f0 2024-04-02T14:48:25.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.461+0000 7f1af6000640 1 --2- 172.21.15.90:0/3982679153 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f1ae00a3da0 0x7f1ae00a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:25.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.461+0000 7f1af6000640 1 -- 172.21.15.90:0/3982679153 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 8550+0+0 (secure 0 0 0) 0x7f1af00d0d40 con 0x7f1af8112a90 2024-04-02T14:48:25.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.461+0000 7f1afd800640 1 --2- 172.21.15.90:0/3982679153 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f1ae00a3da0 0x7f1ae00a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:25.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.462+0000 7f1b02e13740 1 -- 172.21.15.90:0/1915212145 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a6785c39f0 msgr2=0x55a6785c2740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:48:25.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.462+0000 7f1b02e13740 1 --2- 172.21.15.90:0/1915212145 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a6785c39f0 0x55a6785c2740 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f1ae4013560 tx=0x7f1ae4005850 comp rx=0 tx=0).stop 2024-04-02T14:48:25.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.463+0000 7f1b02e13740 1 -- 172.21.15.90:0/1915212145 shutdown_connections 2024-04-02T14:48:25.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.463+0000 7f1b02e13740 1 --2- 172.21.15.90:0/1915212145 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a6785cc080 0x55a6785ce470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:25.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.463+0000 7f1b02e13740 1 --2- 172.21.15.90:0/1915212145 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a6785c2d10 0x55a6785c3190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:25.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.463+0000 7f1b02e13740 1 --2- 172.21.15.90:0/1915212145 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a6785c39f0 0x55a6785c2740 unknown :-1 s=CLOSED pgs=933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:25.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.463+0000 7f1b02e13740 1 -- 172.21.15.90:0/1915212145 >> 172.21.15.90:0/1915212145 conn(0x55a678589a50 msgr2=0x55a6785bd760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:48:25.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.465+0000 7f1b02e13740 1 -- 172.21.15.90:0/1915212145 shutdown_connections 2024-04-02T14:48:25.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.466+0000 7f1afd800640 1 --2- 172.21.15.90:0/3982679153 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f1ae00a3da0 0x7f1ae00a6260 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x55a6785c3f40 tx=0x7f1aec005a20 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:48:25.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.466+0000 7f1b02e13740 1 -- 172.21.15.90:0/1915212145 wait complete. 2024-04-02T14:48:25.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.468+0000 7f1b02e13740 1 -- start start 2024-04-02T14:48:25.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.469+0000 7f1b02e13740 1 Processor -- start 2024-04-02T14:48:25.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.472+0000 7f1b02e13740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a6785c2d10 0x55a6785c9080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:25.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.473+0000 7f1afd800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a6785c2d10 0x55a6785c9080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:25.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.473+0000 7f1afd800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a6785c2d10 0x55a6785c9080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:54360/0 (socket says 172.21.15.90:54360) 2024-04-02T14:48:25.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.477+0000 7f1afd800640 1 -- 172.21.15.90:0/823248705 learned_addr learned my addr 172.21.15.90:0/823248705 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:48:25.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.477+0000 7f1b02e13740 1 --2- 172.21.15.90:0/823248705 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a6785c39f0 0x55a6785c95c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:25.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.477+0000 7f1b02e13740 1 --2- 172.21.15.90:0/823248705 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a6785cc080 0x55a6785c9b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:25.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.477+0000 7f1b02e13740 1 -- 172.21.15.90:0/823248705 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55a6785236a0 con 0x55a6785cc080 2024-04-02T14:48:25.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.477+0000 7f1b02e13740 1 -- 172.21.15.90:0/823248705 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55a6784cf980 con 0x55a6785c2d10 2024-04-02T14:48:25.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.477+0000 7f1b02e13740 1 -- 172.21.15.90:0/823248705 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55a6784e2cb0 con 0x55a6785c39f0 2024-04-02T14:48:25.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.477+0000 7f1afce00640 1 --2- 172.21.15.90:0/823248705 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a6785c39f0 0x55a6785c95c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:25.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.478+0000 7f1b02e13740 10 client.0 _ll_register_callbacks cb 0x55a6784e26a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:48:25.473 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.478+0000 7f1afce00640 1 -- 172.21.15.90:0/823248705 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a6785c2d10 msgr2=0x55a6785c9080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:48:25.473 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.478+0000 7f1afce00640 1 --2- 172.21.15.90:0/823248705 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a6785c2d10 0x55a6785c9080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:25.473 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.478+0000 7f1afce00640 1 -- 172.21.15.90:0/823248705 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a6785cc080 msgr2=0x55a6785c9b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:48:25.473 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.478+0000 7f1afce00640 1 --2- 172.21.15.90:0/823248705 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a6785cc080 0x55a6785c9b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:25.473 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.478+0000 7f1afce00640 1 -- 172.21.15.90:0/823248705 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a6783057d0 con 0x55a6785c39f0 2024-04-02T14:48:25.473 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.478+0000 7f1afd800640 1 --2- 172.21.15.90:0/823248705 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a6785c2d10 0x55a6785c9080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:48:25.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.481+0000 7f1afce00640 1 --2- 172.21.15.90:0/823248705 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a6785c39f0 0x55a6785c95c0 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f1ae4006010 tx=0x7f1ae40048b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=af0c611def492d6b server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:48:25.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.481+0000 7f1adf400640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:48:25.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.481+0000 7f1adf400640 1 -- 172.21.15.90:0/823248705 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ae4004bf0 con 0x55a6785c39f0 2024-04-02T14:48:25.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.481+0000 7f1adf400640 1 -- 172.21.15.90:0/823248705 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 8550+0+0 (secure 0 0 0) 0x7f1ae4004d90 con 0x55a6785c39f0 2024-04-02T14:48:25.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.481+0000 7f1b02e13740 1 -- 172.21.15.90:0/823248705 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1ae4005500 con 0x55a6785c39f0 2024-04-02T14:48:25.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.482+0000 7f1adf400640 1 -- 172.21.15.90:0/823248705 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f1ae400f500 con 0x55a6785c39f0 2024-04-02T14:48:25.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.482+0000 7f1b02e13740 10 client.29479 fetch_fsmap learned FSMap version 27 2024-04-02T14:48:25.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.482+0000 7f1b02e13740 1 -- 172.21.15.90:0/823248705 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55a6785daf30 con 0x55a6785c39f0 2024-04-02T14:48:25.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.484+0000 7f1adf400640 1 -- 172.21.15.90:0/823248705 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1ae4005500 con 0x55a6785c39f0 2024-04-02T14:48:25.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.484+0000 7f1adf400640 1 -- 172.21.15.90:0/823248705 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f1ae401d020 con 0x55a6785c39f0 2024-04-02T14:48:25.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.487+0000 7f1b02e13740 10 client.29479 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:48:25.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.487+0000 7f1b02e13740 10 client.29479 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2024-04-02T14:48:25.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.487+0000 7f1b02e13740 20 client.29479 populate_metadata read hostname 'smithi090' 2024-04-02T14:48:25.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.487+0000 7f1b02e13740 1 --2- 172.21.15.90:0/823248705 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x55a6785dc280 0x55a6785de670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:25.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.487+0000 7f1b02e13740 4 client.29479 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:48:25.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.487+0000 7f1b02e13740 1 -- 172.21.15.90:0/823248705 --> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a6784e7040 con 0x55a6785dc280 2024-04-02T14:48:25.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.488+0000 7f1afd800640 1 --2- 172.21.15.90:0/823248705 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x55a6785dc280 0x55a6785de670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:25.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.490+0000 7f1adf400640 10 client.29479 ms_handle_connect on v2:172.21.15.90:6838/1498915870 2024-04-02T14:48:25.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.490+0000 7f1afd800640 1 --2- 172.21.15.90:0/823248705 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x55a6785dc280 0x55a6785de670 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=c557f85d0f8bd0a server_cookie=cc8b6e44f9be913f in_seq=0 out_seq=0 2024-04-02T14:48:25.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.494+0000 7f1adf400640 1 -- 172.21.15.90:0/823248705 <== mds.1 v2:172.21.15.90:6838/1498915870 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a6784e7040 con 0x55a6785dc280 2024-04-02T14:48:25.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.494+0000 7f1adf400640 10 client.29479 handle_command_reply: tid=0 2024-04-02T14:48:25.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.494+0000 7f1b02e13740 1 client.29479 shutdown 2024-04-02T14:48:25.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.496+0000 7f1b02e13740 1 -- 172.21.15.90:0/823248705 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a6785c39f0 msgr2=0x55a6785c95c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:48:25.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.496+0000 7f1b02e13740 1 --2- 172.21.15.90:0/823248705 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a6785c39f0 0x55a6785c95c0 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f1ae4006010 tx=0x7f1ae40048b0 comp rx=0 tx=0).stop 2024-04-02T14:48:25.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.496+0000 7f1b02e13740 1 -- 172.21.15.90:0/823248705 shutdown_connections 2024-04-02T14:48:25.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.496+0000 7f1b02e13740 1 --2- 172.21.15.90:0/823248705 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x55a6785dc280 0x55a6785de670 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:25.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.496+0000 7f1b02e13740 1 --2- 172.21.15.90:0/823248705 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a6785cc080 0x55a6785c9b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:25.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.496+0000 7f1b02e13740 1 --2- 172.21.15.90:0/823248705 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a6785c39f0 0x55a6785c95c0 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:25.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.496+0000 7f1adf400640 0 client.29479 ms_handle_reset on v2:172.21.15.90:6838/1498915870 2024-04-02T14:48:25.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.496+0000 7f1b02e13740 1 --2- 172.21.15.90:0/823248705 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a6785c2d10 0x55a6785c9080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:25.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.496+0000 7f1b02e13740 1 -- 172.21.15.90:0/823248705 >> 172.21.15.90:0/823248705 conn(0x55a678589a50 msgr2=0x55a6785bf4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:48:25.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.497+0000 7f1b02e13740 1 -- 172.21.15.90:0/823248705 shutdown_connections 2024-04-02T14:48:25.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.497+0000 7f1b02e13740 1 -- 172.21.15.90:0/823248705 wait complete. 2024-04-02T14:48:25.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.497+0000 7f1b02e13740 20 client.29479 trim_cache size 0 max 16384 2024-04-02T14:48:25.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.532+0000 7f1b02e13740 1 Processor -- start 2024-04-02T14:48:25.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.532+0000 7f1b02e13740 1 -- start start 2024-04-02T14:48:25.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.533+0000 7f1b02e13740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a6785c2d10 0x55a6785c89a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:25.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.533+0000 7f1b02e13740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a6785c39f0 0x55a6785d7b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:25.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.533+0000 7f1b02e13740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a6785cc080 0x55a6785caeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:25.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.533+0000 7f1b02e13740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f1af81b3fd0 con 0x55a6785cc080 2024-04-02T14:48:25.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.533+0000 7f1b02e13740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f1af81b5b10 con 0x55a6785c2d10 2024-04-02T14:48:25.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.533+0000 7f1b02e13740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f1af81b46a0 con 0x55a6785c39f0 2024-04-02T14:48:25.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.533+0000 7f1afce00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a6785c39f0 0x55a6785d7b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:25.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.533+0000 7f1afce00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a6785c39f0 0x55a6785d7b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:38306/0 (socket says 172.21.15.90:38306) 2024-04-02T14:48:25.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.533+0000 7f1afce00640 1 -- 172.21.15.90:0/3171743634 learned_addr learned my addr 172.21.15.90:0/3171743634 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:48:25.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.534+0000 7f1afce00640 1 -- 172.21.15.90:0/3171743634 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a6785c2d10 msgr2=0x55a6785c89a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-02T14:48:25.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.534+0000 7f1afce00640 1 --2- 172.21.15.90:0/3171743634 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a6785c2d10 0x55a6785c89a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:25.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.534+0000 7f1afe200640 1 --2- 172.21.15.90:0/3171743634 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a6785cc080 0x55a6785caeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:25.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.534+0000 7f1afce00640 1 -- 172.21.15.90:0/3171743634 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a6785cc080 msgr2=0x55a6785caeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:48:25.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.535+0000 7f1afce00640 1 --2- 172.21.15.90:0/3171743634 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a6785cc080 0x55a6785caeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:25.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.535+0000 7f1afce00640 1 -- 172.21.15.90:0/3171743634 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a6785c1d40 con 0x55a6785c39f0 2024-04-02T14:48:25.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.535+0000 7f1afe200640 1 --2- 172.21.15.90:0/3171743634 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a6785cc080 0x55a6785caeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:48:25.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.536+0000 7f1afce00640 1 --2- 172.21.15.90:0/3171743634 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a6785c39f0 0x55a6785d7b20 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f1ae40049b0 tx=0x7f1ae4009f50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=23f21ccec8baa75c server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:48:25.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.536+0000 7f1aea000640 1 -- 172.21.15.90:0/3171743634 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ae4037020 con 0x55a6785c39f0 2024-04-02T14:48:25.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.536+0000 7f1aea000640 1 -- 172.21.15.90:0/3171743634 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1ae4005460 con 0x55a6785c39f0 2024-04-02T14:48:25.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.536+0000 7f1aea000640 1 -- 172.21.15.90:0/3171743634 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ae401f8a0 con 0x55a6785c39f0 2024-04-02T14:48:25.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.537+0000 7f1b02e13740 1 -- 172.21.15.90:0/3171743634 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a6785c39f0 msgr2=0x55a6785d7b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:48:25.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.537+0000 7f1b02e13740 1 --2- 172.21.15.90:0/3171743634 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a6785c39f0 0x55a6785d7b20 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f1ae40049b0 tx=0x7f1ae4009f50 comp rx=0 tx=0).stop 2024-04-02T14:48:25.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.538+0000 7f1b02e13740 1 -- 172.21.15.90:0/3171743634 shutdown_connections 2024-04-02T14:48:25.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.538+0000 7f1b02e13740 1 --2- 172.21.15.90:0/3171743634 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a6785cc080 0x55a6785caeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:25.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.538+0000 7f1b02e13740 1 --2- 172.21.15.90:0/3171743634 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a6785c39f0 0x55a6785d7b20 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:25.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.538+0000 7f1b02e13740 1 --2- 172.21.15.90:0/3171743634 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a6785c2d10 0x55a6785c89a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:25.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.538+0000 7f1b02e13740 1 -- 172.21.15.90:0/3171743634 >> 172.21.15.90:0/3171743634 conn(0x55a678589a50 msgr2=0x55a6785ca7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:48:25.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.538+0000 7f1b02e13740 1 -- 172.21.15.90:0/3171743634 shutdown_connections 2024-04-02T14:48:25.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.539+0000 7f1b02e13740 1 -- 172.21.15.90:0/3171743634 wait complete. 2024-04-02T14:48:25.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.539+0000 7f1b02e13740 1 -- start start 2024-04-02T14:48:25.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.539+0000 7f1b02e13740 1 Processor -- start 2024-04-02T14:48:25.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.542+0000 7f1b02e13740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a6785c2d10 0x55a6785d3e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:25.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.544+0000 7f1b02e13740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a6785c39f0 0x55a6785d4350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:25.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.544+0000 7f1b02e13740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a6785cc080 0x55a6785c6340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:25.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.544+0000 7f1afe200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a6785cc080 0x55a6785c6340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:25.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.544+0000 7f1afe200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a6785cc080 0x55a6785c6340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:42112/0 (socket says 172.21.15.90:42112) 2024-04-02T14:48:25.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.545+0000 7f1afe200640 1 -- 172.21.15.90:0/331862915 learned_addr learned my addr 172.21.15.90:0/331862915 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:48:25.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.545+0000 7f1b02e13740 1 -- 172.21.15.90:0/331862915 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f1af81b5440 con 0x55a6785cc080 2024-04-02T14:48:25.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.545+0000 7f1b02e13740 1 -- 172.21.15.90:0/331862915 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f1af81b4d70 con 0x55a6785c2d10 2024-04-02T14:48:25.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.545+0000 7f1b02e13740 1 -- 172.21.15.90:0/331862915 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f1af81b1fd0 con 0x55a6785c39f0 2024-04-02T14:48:25.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.545+0000 7f1b02e13740 10 client.0 _ll_register_callbacks cb 0x7f1ae4005330 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:48:25.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.545+0000 7f1afd800640 1 --2- 172.21.15.90:0/331862915 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a6785c2d10 0x55a6785d3e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:25.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.546+0000 7f1afce00640 1 --2- 172.21.15.90:0/331862915 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a6785c39f0 0x55a6785d4350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:25.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.547+0000 7f1afe200640 1 -- 172.21.15.90:0/331862915 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a6785c39f0 msgr2=0x55a6785d4350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:48:25.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.547+0000 7f1afe200640 1 --2- 172.21.15.90:0/331862915 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a6785c39f0 0x55a6785d4350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:25.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.547+0000 7f1afe200640 1 -- 172.21.15.90:0/331862915 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a6785c2d10 msgr2=0x55a6785d3e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:48:25.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.547+0000 7f1afe200640 1 --2- 172.21.15.90:0/331862915 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a6785c2d10 0x55a6785d3e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:25.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.548+0000 7f1afe200640 1 -- 172.21.15.90:0/331862915 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a6785c69e0 con 0x55a6785cc080 2024-04-02T14:48:25.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.548+0000 7f1afd800640 1 --2- 172.21.15.90:0/331862915 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a6785c2d10 0x55a6785d3e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:48:25.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.549+0000 7f1afe200640 1 --2- 172.21.15.90:0/331862915 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a6785cc080 0x55a6785c6340 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f1af000cba0 tx=0x7f1af009c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38698e8dd2769c36 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:48:25.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.549+0000 7f1adf400640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-02T14:48:25.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.550+0000 7f1adf400640 1 -- 172.21.15.90:0/331862915 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1af009cc60 con 0x55a6785cc080 2024-04-02T14:48:25.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.550+0000 7f1adf400640 1 -- 172.21.15.90:0/331862915 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 8550+0+0 (secure 0 0 0) 0x7f1af00bd070 con 0x55a6785cc080 2024-04-02T14:48:25.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.550+0000 7f1b02e13740 1 -- 172.21.15.90:0/331862915 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1ae4005460 con 0x55a6785cc080 2024-04-02T14:48:25.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.551+0000 7f1adf400640 1 -- 172.21.15.90:0/331862915 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f1af00a1010 con 0x55a6785cc080 2024-04-02T14:48:25.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.552+0000 7f1b02e13740 10 client.18516 fetch_fsmap learned FSMap version 27 2024-04-02T14:48:25.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.552+0000 7f1b02e13740 1 -- 172.21.15.90:0/331862915 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55a6785cce40 con 0x55a6785cc080 2024-04-02T14:48:25.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.552+0000 7f1adf400640 1 -- 172.21.15.90:0/331862915 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1ae4005460 con 0x55a6785cc080 2024-04-02T14:48:25.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.553+0000 7f1adf400640 1 -- 172.21.15.90:0/331862915 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f1af00ceb60 con 0x55a6785cc080 2024-04-02T14:48:25.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.553+0000 7f1b02e13740 10 client.18516 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:48:25.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.553+0000 7f1b02e13740 10 client.18516 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2024-04-02T14:48:25.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.554+0000 7f1b02e13740 20 client.18516 populate_metadata read hostname 'smithi090' 2024-04-02T14:48:25.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.554+0000 7f1b02e13740 1 --2- 172.21.15.90:0/331862915 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x55a6786426c0 0x55a678644ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:25.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.554+0000 7f1b02e13740 4 client.18516 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-02T14:48:25.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.554+0000 7f1b02e13740 1 -- 172.21.15.90:0/331862915 --> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55a678534c30 con 0x55a6786426c0 2024-04-02T14:48:25.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.554+0000 7f1afd800640 1 --2- 172.21.15.90:0/331862915 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x55a6786426c0 0x55a678644ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:25.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.558+0000 7f1afd800640 1 --2- 172.21.15.90:0/331862915 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x55a6786426c0 0x55a678644ab0 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=cee6bff262eff851 server_cookie=41be4afe6162f174 in_seq=0 out_seq=0 2024-04-02T14:48:25.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.558+0000 7f1adf400640 10 client.18516 ms_handle_connect on v2:172.21.15.90:6838/1498915870 2024-04-02T14:48:25.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.559+0000 7f1adf400640 1 -- 172.21.15.90:0/331862915 <== mds.1 v2:172.21.15.90:6838/1498915870 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55a678534c30 con 0x55a6786426c0 2024-04-02T14:48:25.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.559+0000 7f1adf400640 10 client.18516 handle_command_reply: tid=0 2024-04-02T14:48:25.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.559+0000 7f1b02e13740 1 client.18516 shutdown 2024-04-02T14:48:25.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.559+0000 7f1b02e13740 1 -- 172.21.15.90:0/331862915 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a6785cc080 msgr2=0x55a6785c6340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:48:25.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.559+0000 7f1b02e13740 1 --2- 172.21.15.90:0/331862915 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a6785cc080 0x55a6785c6340 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f1af000cba0 tx=0x7f1af009c890 comp rx=0 tx=0).stop 2024-04-02T14:48:25.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.559+0000 7f1b02e13740 1 -- 172.21.15.90:0/331862915 shutdown_connections 2024-04-02T14:48:25.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.559+0000 7f1b02e13740 1 --2- 172.21.15.90:0/331862915 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x55a6786426c0 0x55a678644ab0 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:25.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.559+0000 7f1b02e13740 1 --2- 172.21.15.90:0/331862915 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a6785cc080 0x55a6785c6340 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:25.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.559+0000 7f1b02e13740 1 --2- 172.21.15.90:0/331862915 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a6785c39f0 0x55a6785d4350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:25.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.559+0000 7f1b02e13740 1 --2- 172.21.15.90:0/331862915 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a6785c2d10 0x55a6785d3e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:25.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.559+0000 7f1b02e13740 1 -- 172.21.15.90:0/331862915 >> 172.21.15.90:0/331862915 conn(0x55a678589a50 msgr2=0x55a6785bf3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:48:25.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.560+0000 7f1adf400640 0 client.18516 ms_handle_reset on v2:172.21.15.90:6838/1498915870 2024-04-02T14:48:25.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.560+0000 7f1b02e13740 1 -- 172.21.15.90:0/331862915 shutdown_connections 2024-04-02T14:48:25.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.560+0000 7f1b02e13740 1 -- 172.21.15.90:0/331862915 wait complete. 2024-04-02T14:48:25.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.560+0000 7f1b02e13740 20 client.18516 trim_cache size 0 max 16384 2024-04-02T14:48:25.556 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-02T14:48:25.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.561+0000 7f1b00200640 1 -- 172.21.15.90:0/3982679153 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f1ae00a3da0 msgr2=0x7f1ae00a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:48:25.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.561+0000 7f1b00200640 1 --2- 172.21.15.90:0/3982679153 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f1ae00a3da0 0x7f1ae00a6260 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x55a6785c3f40 tx=0x7f1aec005a20 comp rx=0 tx=0).stop 2024-04-02T14:48:25.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.561+0000 7f1b00200640 1 -- 172.21.15.90:0/3982679153 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1af8112a90 msgr2=0x7f1af819e870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:48:25.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.561+0000 7f1b00200640 1 --2- 172.21.15.90:0/3982679153 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1af8112a90 0x7f1af819e870 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f1af0005fd0 tx=0x7f1af000c6a0 comp rx=0 tx=0).stop 2024-04-02T14:48:25.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.561+0000 7f1b00200640 1 -- 172.21.15.90:0/3982679153 shutdown_connections 2024-04-02T14:48:25.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.561+0000 7f1b00200640 1 --2- 172.21.15.90:0/3982679153 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f1ae00a3da0 0x7f1ae00a6260 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:25.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.561+0000 7f1b00200640 1 --2- 172.21.15.90:0/3982679153 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1af8112a90 0x7f1af819e870 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:25.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.561+0000 7f1b00200640 1 --2- 172.21.15.90:0/3982679153 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1af80747c0 0x7f1af81a4c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:25.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.561+0000 7f1b00200640 1 --2- 172.21.15.90:0/3982679153 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1af8073ea0 0x7f1af81a46c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:25.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.561+0000 7f1b00200640 1 -- 172.21.15.90:0/3982679153 >> 172.21.15.90:0/3982679153 conn(0x7f1af80fd590 msgr2=0x7f1af80fedf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:48:25.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.562+0000 7f1b00200640 1 -- 172.21.15.90:0/3982679153 shutdown_connections 2024-04-02T14:48:25.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.562+0000 7f1b00200640 1 -- 172.21.15.90:0/3982679153 wait complete. 2024-04-02T14:48:25.586 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-02T14:48:25.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.835+0000 7f501e400640 1 -- 172.21.15.90:0/3999350815 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5018073eb0 msgr2=0x7f5018074290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:48:25.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.835+0000 7f501e400640 1 --2- 172.21.15.90:0/3999350815 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5018073eb0 0x7f5018074290 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f5008009900 tx=0x7f500802bcc0 comp rx=0 tx=0).stop 2024-04-02T14:48:25.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.835+0000 7f501e400640 1 -- 172.21.15.90:0/3999350815 shutdown_connections 2024-04-02T14:48:25.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.835+0000 7f501e400640 1 --2- 172.21.15.90:0/3999350815 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f501810e350 0x7f5018110740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:25.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.835+0000 7f501e400640 1 --2- 172.21.15.90:0/3999350815 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f50180747d0 0x7f501810de10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:25.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.835+0000 7f501e400640 1 --2- 172.21.15.90:0/3999350815 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5018073eb0 0x7f5018074290 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:25.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.836+0000 7f501e400640 1 -- 172.21.15.90:0/3999350815 >> 172.21.15.90:0/3999350815 conn(0x7f5018078f00 msgr2=0x7f5018079310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:48:25.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.836+0000 7f501e400640 1 -- 172.21.15.90:0/3999350815 shutdown_connections 2024-04-02T14:48:25.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.836+0000 7f501e400640 1 -- 172.21.15.90:0/3999350815 wait complete. 2024-04-02T14:48:25.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.838+0000 7f501e400640 1 Processor -- start 2024-04-02T14:48:25.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.839+0000 7f501e400640 1 -- start start 2024-04-02T14:48:25.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.839+0000 7f501e400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5018073eb0 0x7f501819a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:25.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.839+0000 7f501e400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f50180747d0 0x7f501819ab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:25.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.839+0000 7f501e400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f501810e350 0x7f501819eea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:25.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.839+0000 7f501e400640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f50181140a0 con 0x7f501810e350 2024-04-02T14:48:25.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.839+0000 7f501e400640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f5018113f20 con 0x7f50180747d0 2024-04-02T14:48:25.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.839+0000 7f501e400640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f5018114220 con 0x7f5018073eb0 2024-04-02T14:48:25.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.839+0000 7f5017400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5018073eb0 0x7f501819a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:25.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.839+0000 7f5017400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5018073eb0 0x7f501819a5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:38318/0 (socket says 172.21.15.90:38318) 2024-04-02T14:48:25.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.839+0000 7f5017400640 1 -- 172.21.15.90:0/2253480381 learned_addr learned my addr 172.21.15.90:0/2253480381 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:48:25.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.840+0000 7f500fe00640 1 --2- 172.21.15.90:0/2253480381 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f50180747d0 0x7f501819ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:25.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.840+0000 7f5017e00640 1 --2- 172.21.15.90:0/2253480381 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f501810e350 0x7f501819eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:25.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.840+0000 7f5017400640 1 -- 172.21.15.90:0/2253480381 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f50180747d0 msgr2=0x7f501819ab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:48:25.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.840+0000 7f5017400640 1 --2- 172.21.15.90:0/2253480381 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f50180747d0 0x7f501819ab20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:25.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.840+0000 7f5017400640 1 -- 172.21.15.90:0/2253480381 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f501810e350 msgr2=0x7f501819eea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:48:25.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.840+0000 7f5017400640 1 --2- 172.21.15.90:0/2253480381 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f501810e350 0x7f501819eea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:25.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.840+0000 7f5017400640 1 -- 172.21.15.90:0/2253480381 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f501819f5d0 con 0x7f5018073eb0 2024-04-02T14:48:25.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.840+0000 7f5017e00640 1 --2- 172.21.15.90:0/2253480381 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f501810e350 0x7f501819eea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:48:25.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.840+0000 7f5017400640 1 --2- 172.21.15.90:0/2253480381 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5018073eb0 0x7f501819a5e0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f5008000c00 tx=0x7f5008004360 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:48:25.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.841+0000 7f5014c00640 1 -- 172.21.15.90:0/2253480381 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5008042020 con 0x7f5018073eb0 2024-04-02T14:48:25.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.841+0000 7f501e400640 1 -- 172.21.15.90:0/2253480381 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f501819f860 con 0x7f5018073eb0 2024-04-02T14:48:25.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.841+0000 7f5014c00640 1 -- 172.21.15.90:0/2253480381 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5008005000 con 0x7f5018073eb0 2024-04-02T14:48:25.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.841+0000 7f501e400640 1 -- 172.21.15.90:0/2253480381 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50181a7150 con 0x7f5018073eb0 2024-04-02T14:48:25.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.841+0000 7f5014c00640 1 -- 172.21.15.90:0/2253480381 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5008003b20 con 0x7f5018073eb0 2024-04-02T14:48:25.841 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.846+0000 7f5014c00640 1 -- 172.21.15.90:0/2253480381 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f5008045020 con 0x7f5018073eb0 2024-04-02T14:48:25.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.847+0000 7f5021230740 1 Processor -- start 2024-04-02T14:48:25.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.847+0000 7f5021230740 1 -- start start 2024-04-02T14:48:25.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.847+0000 7f5021230740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c0d727a9f0 0x55c0d7279740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:25.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.847+0000 7f5021230740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c0d7279d10 0x55c0d727a190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:25.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.848+0000 7f5021230740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c0d7283080 0x55c0d7285470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:25.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.848+0000 7f5021230740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55c0d71ba400 con 0x55c0d7279d10 2024-04-02T14:48:25.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.848+0000 7f5021230740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55c0d719ef10 con 0x55c0d727a9f0 2024-04-02T14:48:25.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.848+0000 7f5021230740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55c0d71de6f0 con 0x55c0d7283080 2024-04-02T14:48:25.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.848+0000 7f5017e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c0d727a9f0 0x55c0d7279740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:25.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.848+0000 7f5017400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c0d7279d10 0x55c0d727a190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:25.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.848+0000 7f5017e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c0d727a9f0 0x55c0d7279740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:54404/0 (socket says 172.21.15.90:54404) 2024-04-02T14:48:25.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.848+0000 7f5017e00640 1 -- 172.21.15.90:0/3526886652 learned_addr learned my addr 172.21.15.90:0/3526886652 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:48:25.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.848+0000 7f500fe00640 1 --2- 172.21.15.90:0/3526886652 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c0d7283080 0x55c0d7285470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:25.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.848+0000 7f5017e00640 1 -- 172.21.15.90:0/3526886652 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c0d7283080 msgr2=0x55c0d7285470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:48:25.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.848+0000 7f5017e00640 1 --2- 172.21.15.90:0/3526886652 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c0d7283080 0x55c0d7285470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:25.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.848+0000 7f5017e00640 1 -- 172.21.15.90:0/3526886652 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c0d7279d10 msgr2=0x55c0d727a190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:48:25.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.848+0000 7f5017e00640 1 --2- 172.21.15.90:0/3526886652 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c0d7279d10 0x55c0d727a190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:25.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.848+0000 7f5017e00640 1 -- 172.21.15.90:0/3526886652 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c0d6ef23f0 con 0x55c0d727a9f0 2024-04-02T14:48:25.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.848+0000 7f5017400640 1 --2- 172.21.15.90:0/3526886652 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c0d7279d10 0x55c0d727a190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:48:25.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.848+0000 7f5014c00640 1 --2- 172.21.15.90:0/2253480381 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f4ffc0a3e20 0x7f4ffc0a62e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:25.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.849+0000 7f5017e00640 1 --2- 172.21.15.90:0/3526886652 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c0d727a9f0 0x55c0d7279740 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7f5010009f10 tx=0x7f501000f730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c4090ea44d1f80fa server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:48:25.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.849+0000 7f5014c00640 1 -- 172.21.15.90:0/2253480381 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 8550+0+0 (secure 0 0 0) 0x7f50080f2380 con 0x7f5018073eb0 2024-04-02T14:48:25.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.849+0000 7f5017400640 1 --2- 172.21.15.90:0/2253480381 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f4ffc0a3e20 0x7f4ffc0a62e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:25.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.849+0000 7f500cc00640 1 -- 172.21.15.90:0/3526886652 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5010003b70 con 0x55c0d727a9f0 2024-04-02T14:48:25.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.849+0000 7f500cc00640 1 -- 172.21.15.90:0/3526886652 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5010003d10 con 0x55c0d727a9f0 2024-04-02T14:48:25.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.849+0000 7f500cc00640 1 -- 172.21.15.90:0/3526886652 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5010011a70 con 0x55c0d727a9f0 2024-04-02T14:48:25.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.850+0000 7f5021230740 1 -- 172.21.15.90:0/3526886652 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c0d727a9f0 msgr2=0x55c0d7279740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:48:25.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.850+0000 7f5017400640 1 --2- 172.21.15.90:0/2253480381 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f4ffc0a3e20 0x7f4ffc0a62e0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x55c0d727af40 tx=0x7f5008009260 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:48:25.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.850+0000 7f5021230740 1 --2- 172.21.15.90:0/3526886652 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c0d727a9f0 0x55c0d7279740 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7f5010009f10 tx=0x7f501000f730 comp rx=0 tx=0).stop 2024-04-02T14:48:25.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.850+0000 7f5021230740 1 -- 172.21.15.90:0/3526886652 shutdown_connections 2024-04-02T14:48:25.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.850+0000 7f5021230740 1 --2- 172.21.15.90:0/3526886652 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c0d7283080 0x55c0d7285470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:25.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.850+0000 7f5021230740 1 --2- 172.21.15.90:0/3526886652 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c0d7279d10 0x55c0d727a190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:25.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.850+0000 7f5021230740 1 --2- 172.21.15.90:0/3526886652 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c0d727a9f0 0x55c0d7279740 unknown :-1 s=CLOSED pgs=934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:25.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.850+0000 7f5021230740 1 -- 172.21.15.90:0/3526886652 >> 172.21.15.90:0/3526886652 conn(0x55c0d7240a50 msgr2=0x55c0d7274760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:48:25.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.850+0000 7f5021230740 1 -- 172.21.15.90:0/3526886652 shutdown_connections 2024-04-02T14:48:25.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.850+0000 7f5021230740 1 -- 172.21.15.90:0/3526886652 wait complete. 2024-04-02T14:48:25.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.851+0000 7f5021230740 1 -- start start 2024-04-02T14:48:25.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.851+0000 7f5021230740 1 Processor -- start 2024-04-02T14:48:25.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.851+0000 7f5021230740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c0d7279d10 0x55c0d7280090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:25.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.851+0000 7f5021230740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c0d727a9f0 0x55c0d72805d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:25.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.851+0000 7f5021230740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c0d7283080 0x55c0d7280b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:25.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.851+0000 7f5021230740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55c0d71da6a0 con 0x55c0d727a9f0 2024-04-02T14:48:25.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.851+0000 7f5021230740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55c0d7186980 con 0x55c0d7279d10 2024-04-02T14:48:25.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.851+0000 7f5021230740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55c0d7199cb0 con 0x55c0d7283080 2024-04-02T14:48:25.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.851+0000 7f5021230740 10 client.0 _ll_register_callbacks cb 0x55c0d71996a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:48:25.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.852+0000 7f500fe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c0d727a9f0 0x55c0d72805d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:25.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.852+0000 7f5017e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c0d7279d10 0x55c0d7280090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:25.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.852+0000 7f500fe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c0d727a9f0 0x55c0d72805d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:42162/0 (socket says 172.21.15.90:42162) 2024-04-02T14:48:25.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.852+0000 7f500fe00640 1 -- 172.21.15.90:0/4166412227 learned_addr learned my addr 172.21.15.90:0/4166412227 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:48:25.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.852+0000 7f5017e00640 1 --2- 172.21.15.90:0/4166412227 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c0d7283080 0x55c0d7280b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:25.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.852+0000 7f5017e00640 1 -- 172.21.15.90:0/4166412227 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c0d7283080 msgr2=0x55c0d7280b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:48:25.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.852+0000 7f5017e00640 1 --2- 172.21.15.90:0/4166412227 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c0d7283080 0x55c0d7280b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:25.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.852+0000 7f5017e00640 1 -- 172.21.15.90:0/4166412227 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c0d727a9f0 msgr2=0x55c0d72805d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:48:25.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.852+0000 7f5017e00640 1 --2- 172.21.15.90:0/4166412227 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c0d727a9f0 0x55c0d72805d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:25.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.852+0000 7f5017e00640 1 -- 172.21.15.90:0/4166412227 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c0d6fbc7d0 con 0x55c0d7279d10 2024-04-02T14:48:25.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.852+0000 7f500fe00640 1 --2- 172.21.15.90:0/4166412227 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c0d727a9f0 0x55c0d72805d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:48:25.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.852+0000 7f5017e00640 1 --2- 172.21.15.90:0/4166412227 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c0d7279d10 0x55c0d7280090 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7f501000fc30 tx=0x7f50100116f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9c71d8b6fad0b5ea server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:48:25.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.852+0000 7f5005600640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:48:25.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.852+0000 7f5005600640 1 -- 172.21.15.90:0/4166412227 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5010004e50 con 0x55c0d7279d10 2024-04-02T14:48:25.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.853+0000 7f5005600640 1 -- 172.21.15.90:0/4166412227 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 8550+0+0 (secure 0 0 0) 0x7f50100117f0 con 0x55c0d7279d10 2024-04-02T14:48:25.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.853+0000 7f5021230740 1 -- 172.21.15.90:0/4166412227 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5010003d10 con 0x55c0d7279d10 2024-04-02T14:48:25.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.853+0000 7f5005600640 1 -- 172.21.15.90:0/4166412227 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f5010030050 con 0x55c0d7279d10 2024-04-02T14:48:25.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.853+0000 7f5021230740 10 client.29711 fetch_fsmap learned FSMap version 27 2024-04-02T14:48:25.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.853+0000 7f5021230740 1 -- 172.21.15.90:0/4166412227 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55c0d7291f60 con 0x55c0d7279d10 2024-04-02T14:48:25.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.853+0000 7f5005600640 1 -- 172.21.15.90:0/4166412227 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5010003d10 con 0x55c0d7279d10 2024-04-02T14:48:25.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.853+0000 7f5005600640 1 -- 172.21.15.90:0/4166412227 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f5010004510 con 0x55c0d7279d10 2024-04-02T14:48:25.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.854+0000 7f5021230740 10 client.29711 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:48:25.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.854+0000 7f5021230740 10 client.29711 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:48:25.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.854+0000 7f5021230740 20 client.29711 populate_metadata read hostname 'smithi090' 2024-04-02T14:48:25.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.854+0000 7f5021230740 1 --2- 172.21.15.90:0/4166412227 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55c0d7293360 0x55c0d7295750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:25.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.854+0000 7f5021230740 4 client.29711 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:48:25.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.854+0000 7f5021230740 1 -- 172.21.15.90:0/4166412227 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c0d719e040 con 0x55c0d7293360 2024-04-02T14:48:25.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.854+0000 7f500fe00640 1 --2- 172.21.15.90:0/4166412227 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55c0d7293360 0x55c0d7295750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:25.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.854+0000 7f500fe00640 1 --2- 172.21.15.90:0/4166412227 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55c0d7293360 0x55c0d7295750 crc :-1 s=READY pgs=346 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=abbb7eba960ecec1 server_cookie=87a2a784f0bfd21e in_seq=0 out_seq=0 2024-04-02T14:48:25.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.854+0000 7f5005600640 10 client.29711 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:48:25.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.855+0000 7f5005600640 1 -- 172.21.15.90:0/4166412227 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c0d719e040 con 0x55c0d7293360 2024-04-02T14:48:25.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.855+0000 7f5005600640 10 client.29711 handle_command_reply: tid=0 2024-04-02T14:48:25.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.855+0000 7f5021230740 1 client.29711 shutdown 2024-04-02T14:48:25.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.855+0000 7f5021230740 1 -- 172.21.15.90:0/4166412227 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c0d7279d10 msgr2=0x55c0d7280090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:48:25.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.855+0000 7f5021230740 1 --2- 172.21.15.90:0/4166412227 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c0d7279d10 0x55c0d7280090 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7f501000fc30 tx=0x7f50100116f0 comp rx=0 tx=0).stop 2024-04-02T14:48:25.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.855+0000 7f5021230740 1 -- 172.21.15.90:0/4166412227 shutdown_connections 2024-04-02T14:48:25.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.855+0000 7f5021230740 1 --2- 172.21.15.90:0/4166412227 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55c0d7293360 0x55c0d7295750 crc :-1 s=READY pgs=346 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:25.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.855+0000 7f5021230740 1 --2- 172.21.15.90:0/4166412227 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c0d7283080 0x55c0d7280b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:25.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.855+0000 7f5021230740 1 --2- 172.21.15.90:0/4166412227 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c0d727a9f0 0x55c0d72805d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:25.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.855+0000 7f5021230740 1 --2- 172.21.15.90:0/4166412227 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c0d7279d10 0x55c0d7280090 unknown :-1 s=CLOSED pgs=935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:25.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.855+0000 7f5021230740 1 -- 172.21.15.90:0/4166412227 >> 172.21.15.90:0/4166412227 conn(0x55c0d7240a50 msgr2=0x55c0d72764b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:48:25.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.855+0000 7f5005600640 0 client.29711 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:48:25.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.855+0000 7f5021230740 1 -- 172.21.15.90:0/4166412227 shutdown_connections 2024-04-02T14:48:25.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.856+0000 7f5021230740 1 -- 172.21.15.90:0/4166412227 wait complete. 2024-04-02T14:48:25.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.856+0000 7f5021230740 20 client.29711 trim_cache size 0 max 16384 2024-04-02T14:48:25.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.874+0000 7f5021230740 1 Processor -- start 2024-04-02T14:48:25.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.874+0000 7f5021230740 1 -- start start 2024-04-02T14:48:25.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.875+0000 7f5021230740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c0d7279d10 0x55c0d72827c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:25.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.875+0000 7f5021230740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c0d727a9f0 0x55c0d728eb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:25.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.875+0000 7f5021230740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c0d7283080 0x55c0d7280930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:25.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.875+0000 7f5021230740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f50181afac0 con 0x55c0d727a9f0 2024-04-02T14:48:25.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.875+0000 7f5021230740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f50181b1600 con 0x55c0d7283080 2024-04-02T14:48:25.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.875+0000 7f5021230740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f50181b0190 con 0x55c0d7279d10 2024-04-02T14:48:25.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.875+0000 7f5017e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c0d7279d10 0x55c0d72827c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:25.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.875+0000 7f5017e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c0d7279d10 0x55c0d72827c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:38350/0 (socket says 172.21.15.90:38350) 2024-04-02T14:48:25.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.876+0000 7f5017e00640 1 -- 172.21.15.90:0/2340705126 learned_addr learned my addr 172.21.15.90:0/2340705126 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:48:25.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.876+0000 7f500fe00640 1 --2- 172.21.15.90:0/2340705126 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c0d727a9f0 0x55c0d728eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:25.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.876+0000 7f5017e00640 1 --2- 172.21.15.90:0/2340705126 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c0d7283080 0x55c0d7280930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:25.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.876+0000 7f5017e00640 1 -- 172.21.15.90:0/2340705126 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c0d7283080 msgr2=0x55c0d7280930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:48:25.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.876+0000 7f5017e00640 1 --2- 172.21.15.90:0/2340705126 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c0d7283080 0x55c0d7280930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:25.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.876+0000 7f5017e00640 1 -- 172.21.15.90:0/2340705126 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c0d727a9f0 msgr2=0x55c0d728eb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:48:25.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.876+0000 7f5017e00640 1 --2- 172.21.15.90:0/2340705126 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c0d727a9f0 0x55c0d728eb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:25.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.876+0000 7f5017e00640 1 -- 172.21.15.90:0/2340705126 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c0d7278d40 con 0x55c0d7279d10 2024-04-02T14:48:25.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.877+0000 7f5017e00640 1 --2- 172.21.15.90:0/2340705126 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c0d7279d10 0x55c0d72827c0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f5010005d70 tx=0x7f5010005f70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cacd9dbf572bf85f server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:48:25.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.877+0000 7f5007e00640 1 -- 172.21.15.90:0/2340705126 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50100114b0 con 0x55c0d7279d10 2024-04-02T14:48:25.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.877+0000 7f5007e00640 1 -- 172.21.15.90:0/2340705126 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5010026040 con 0x55c0d7279d10 2024-04-02T14:48:25.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.877+0000 7f5007e00640 1 -- 172.21.15.90:0/2340705126 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5010005710 con 0x55c0d7279d10 2024-04-02T14:48:25.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.877+0000 7f5021230740 1 -- 172.21.15.90:0/2340705126 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c0d7279d10 msgr2=0x55c0d72827c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:48:25.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.877+0000 7f5021230740 1 --2- 172.21.15.90:0/2340705126 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c0d7279d10 0x55c0d72827c0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f5010005d70 tx=0x7f5010005f70 comp rx=0 tx=0).stop 2024-04-02T14:48:25.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.878+0000 7f5021230740 1 -- 172.21.15.90:0/2340705126 shutdown_connections 2024-04-02T14:48:25.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.878+0000 7f5021230740 1 --2- 172.21.15.90:0/2340705126 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c0d7283080 0x55c0d7280930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:25.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.878+0000 7f5021230740 1 --2- 172.21.15.90:0/2340705126 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c0d727a9f0 0x55c0d728eb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:25.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.878+0000 7f5021230740 1 --2- 172.21.15.90:0/2340705126 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c0d7279d10 0x55c0d72827c0 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:25.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.878+0000 7f5021230740 1 -- 172.21.15.90:0/2340705126 >> 172.21.15.90:0/2340705126 conn(0x55c0d7240a50 msgr2=0x55c0d7276440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:48:25.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.878+0000 7f5021230740 1 -- 172.21.15.90:0/2340705126 shutdown_connections 2024-04-02T14:48:25.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.878+0000 7f5021230740 1 -- 172.21.15.90:0/2340705126 wait complete. 2024-04-02T14:48:25.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.878+0000 7f5021230740 1 -- start start 2024-04-02T14:48:25.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.878+0000 7f5021230740 1 Processor -- start 2024-04-02T14:48:25.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.879+0000 7f5021230740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c0d7279d10 0x55c0d727d500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:25.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.879+0000 7f5021230740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c0d727a9f0 0x55c0d727da40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:25.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.879+0000 7f5021230740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c0d7283080 0x55c0d72f6610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:25.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.879+0000 7f5021230740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f50181b0f30 con 0x55c0d7283080 2024-04-02T14:48:25.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.879+0000 7f5021230740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f50181b0860 con 0x55c0d727a9f0 2024-04-02T14:48:25.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.879+0000 7f5021230740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f50181adac0 con 0x55c0d7279d10 2024-04-02T14:48:25.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.879+0000 7f5021230740 10 client.0 _ll_register_callbacks cb 0x7f5010009de0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:48:25.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.879+0000 7f5017e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c0d7279d10 0x55c0d727d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:25.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.879+0000 7f5017e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c0d7283080 0x55c0d72f6610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:25.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.879+0000 7f500fe00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c0d727a9f0 0x55c0d727da40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:25.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.879+0000 7f5017e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c0d7279d10 0x55c0d727d500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:38352/0 (socket says 172.21.15.90:38352) 2024-04-02T14:48:25.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.879+0000 7f5017e00640 1 -- 172.21.15.90:0/1348576536 learned_addr learned my addr 172.21.15.90:0/1348576536 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:48:25.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.880+0000 7f500fe00640 1 -- 172.21.15.90:0/1348576536 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c0d7279d10 msgr2=0x55c0d727d500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:48:25.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.880+0000 7f500fe00640 1 --2- 172.21.15.90:0/1348576536 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c0d7279d10 0x55c0d727d500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:25.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.880+0000 7f500fe00640 1 -- 172.21.15.90:0/1348576536 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c0d7283080 msgr2=0x55c0d72f6610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:48:25.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.881+0000 7f500fe00640 1 --2- 172.21.15.90:0/1348576536 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c0d7283080 0x55c0d72f6610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:25.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.882+0000 7f500fe00640 1 -- 172.21.15.90:0/1348576536 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c0d72f6c00 con 0x55c0d727a9f0 2024-04-02T14:48:25.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.882+0000 7f5017e00640 1 --2- 172.21.15.90:0/1348576536 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c0d7283080 0x55c0d72f6610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:48:25.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.882+0000 7f500fe00640 1 --2- 172.21.15.90:0/1348576536 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c0d727a9f0 0x55c0d727da40 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x55c0d7295cf0 tx=0x7f5000012ba0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=545b593ea11424b1 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:48:25.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.882+0000 7f5005600640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:48:25.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.882+0000 7f5005600640 1 -- 172.21.15.90:0/1348576536 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5000010ae0 con 0x55c0d727a9f0 2024-04-02T14:48:25.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.882+0000 7f5021230740 1 -- 172.21.15.90:0/1348576536 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5010026040 con 0x55c0d727a9f0 2024-04-02T14:48:25.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.882+0000 7f5005600640 1 -- 172.21.15.90:0/1348576536 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 8550+0+0 (secure 0 0 0) 0x7f5000010c80 con 0x55c0d727a9f0 2024-04-02T14:48:25.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.883+0000 7f5005600640 1 -- 172.21.15.90:0/1348576536 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f5000012e50 con 0x55c0d727a9f0 2024-04-02T14:48:25.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.883+0000 7f5021230740 10 client.29717 fetch_fsmap learned FSMap version 27 2024-04-02T14:48:25.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.883+0000 7f5021230740 1 -- 172.21.15.90:0/1348576536 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55c0d72841b0 con 0x55c0d727a9f0 2024-04-02T14:48:25.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.883+0000 7f5005600640 1 -- 172.21.15.90:0/1348576536 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5010026040 con 0x55c0d727a9f0 2024-04-02T14:48:25.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.883+0000 7f5005600640 1 -- 172.21.15.90:0/1348576536 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f5000004ba0 con 0x55c0d727a9f0 2024-04-02T14:48:25.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.883+0000 7f5021230740 10 client.29717 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:48:25.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.883+0000 7f5021230740 10 client.29717 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:48:25.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.883+0000 7f5021230740 20 client.29717 populate_metadata read hostname 'smithi090' 2024-04-02T14:48:25.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.883+0000 7f5021230740 1 --2- 172.21.15.90:0/1348576536 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55c0d728a070 0x55c0d728a480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:25.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.883+0000 7f5021230740 4 client.29717 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-02T14:48:25.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.883+0000 7f5021230740 1 -- 172.21.15.90:0/1348576536 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55c0d71ebc30 con 0x55c0d728a070 2024-04-02T14:48:25.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.884+0000 7f5017e00640 1 --2- 172.21.15.90:0/1348576536 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55c0d728a070 0x55c0d728a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:25.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.884+0000 7f5017e00640 1 --2- 172.21.15.90:0/1348576536 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55c0d728a070 0x55c0d728a480 crc :-1 s=READY pgs=347 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a2105124b23fd57a server_cookie=da7fd80914847e37 in_seq=0 out_seq=0 2024-04-02T14:48:25.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.884+0000 7f5005600640 10 client.29717 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:48:25.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.885+0000 7f5005600640 1 -- 172.21.15.90:0/1348576536 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55c0d71ebc30 con 0x55c0d728a070 2024-04-02T14:48:25.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.885+0000 7f5005600640 10 client.29717 handle_command_reply: tid=0 2024-04-02T14:48:25.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.885+0000 7f5021230740 1 client.29717 shutdown 2024-04-02T14:48:25.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.885+0000 7f5021230740 1 -- 172.21.15.90:0/1348576536 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c0d727a9f0 msgr2=0x55c0d727da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:48:25.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.885+0000 7f5021230740 1 --2- 172.21.15.90:0/1348576536 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c0d727a9f0 0x55c0d727da40 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x55c0d7295cf0 tx=0x7f5000012ba0 comp rx=0 tx=0).stop 2024-04-02T14:48:25.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.885+0000 7f5021230740 1 -- 172.21.15.90:0/1348576536 shutdown_connections 2024-04-02T14:48:25.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.885+0000 7f5021230740 1 --2- 172.21.15.90:0/1348576536 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55c0d728a070 0x55c0d728a480 crc :-1 s=READY pgs=347 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:25.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.885+0000 7f5021230740 1 --2- 172.21.15.90:0/1348576536 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55c0d7283080 0x55c0d72f6610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:25.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.885+0000 7f5021230740 1 --2- 172.21.15.90:0/1348576536 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55c0d727a9f0 0x55c0d727da40 unknown :-1 s=CLOSED pgs=936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:25.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.885+0000 7f5021230740 1 --2- 172.21.15.90:0/1348576536 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c0d7279d10 0x55c0d727d500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:25.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.885+0000 7f5005600640 0 client.29717 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:48:25.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.885+0000 7f5021230740 1 -- 172.21.15.90:0/1348576536 >> 172.21.15.90:0/1348576536 conn(0x55c0d7240a50 msgr2=0x55c0d7273010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:48:25.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.885+0000 7f5021230740 1 -- 172.21.15.90:0/1348576536 shutdown_connections 2024-04-02T14:48:25.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.885+0000 7f5021230740 1 -- 172.21.15.90:0/1348576536 wait complete. 2024-04-02T14:48:25.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.885+0000 7f5021230740 20 client.29717 trim_cache size 0 max 16384 2024-04-02T14:48:25.882 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-02T14:48:25.882 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.886+0000 7f501e400640 1 -- 172.21.15.90:0/2253480381 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f4ffc0a3e20 msgr2=0x7f4ffc0a62e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:48:25.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.886+0000 7f501e400640 1 --2- 172.21.15.90:0/2253480381 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f4ffc0a3e20 0x7f4ffc0a62e0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x55c0d727af40 tx=0x7f5008009260 comp rx=0 tx=0).stop 2024-04-02T14:48:25.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.886+0000 7f501e400640 1 -- 172.21.15.90:0/2253480381 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5018073eb0 msgr2=0x7f501819a5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:48:25.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.886+0000 7f501e400640 1 --2- 172.21.15.90:0/2253480381 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5018073eb0 0x7f501819a5e0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f5008000c00 tx=0x7f5008004360 comp rx=0 tx=0).stop 2024-04-02T14:48:25.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.886+0000 7f501e400640 1 -- 172.21.15.90:0/2253480381 shutdown_connections 2024-04-02T14:48:25.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.886+0000 7f501e400640 1 --2- 172.21.15.90:0/2253480381 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f4ffc0a3e20 0x7f4ffc0a62e0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:25.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.886+0000 7f501e400640 1 --2- 172.21.15.90:0/2253480381 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f501810e350 0x7f501819eea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:25.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.886+0000 7f501e400640 1 --2- 172.21.15.90:0/2253480381 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f50180747d0 0x7f501819ab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:25.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.886+0000 7f501e400640 1 --2- 172.21.15.90:0/2253480381 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5018073eb0 0x7f501819a5e0 secure :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f5008000c00 tx=0x7f5008004360 comp rx=0 tx=0).stop 2024-04-02T14:48:25.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.886+0000 7f501e400640 1 -- 172.21.15.90:0/2253480381 >> 172.21.15.90:0/2253480381 conn(0x7f5018078f00 msgr2=0x7f50180fe9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:48:25.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.886+0000 7f501e400640 1 -- 172.21.15.90:0/2253480381 shutdown_connections 2024-04-02T14:48:25.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:25.886+0000 7f501e400640 1 -- 172.21.15.90:0/2253480381 wait complete. 2024-04-02T14:48:26.429 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:48:26 smithi177 ceph-mon[36431]: pgmap v423: 97 pgs: 97 active+clean; 4.2 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 10 MiB/s wr, 603 op/s 2024-04-02T14:48:26.431 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:48:26 smithi090 ceph-mon[31674]: pgmap v423: 97 pgs: 97 active+clean; 4.2 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 10 MiB/s wr, 603 op/s 2024-04-02T14:48:26.603 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:48:26 smithi092 ceph-mon[36497]: pgmap v423: 97 pgs: 97 active+clean; 4.2 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 10 MiB/s wr, 603 op/s 2024-04-02T14:48:26.824 INFO:teuthology.orchestra.run.smithi090.stderr:progress: 775.0 s, 1259.0 tps, lat 79.361 ms stddev 94.460 2024-04-02T14:48:26.908 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-02T14:48:26.908 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T14:48:26.908 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-02T14:48:27.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.209+0000 7f1a01c00640 1 -- 172.21.15.90:0/2055293214 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f19fc1045c0 msgr2=0x7f19fc104a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:48:27.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.209+0000 7f1a01c00640 1 --2- 172.21.15.90:0/2055293214 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f19fc1045c0 0x7f19fc104a40 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f19f4009be0 tx=0x7f19f402bef0 comp rx=0 tx=0).stop 2024-04-02T14:48:27.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.209+0000 7f1a01c00640 1 -- 172.21.15.90:0/2055293214 shutdown_connections 2024-04-02T14:48:27.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.209+0000 7f1a01c00640 1 --2- 172.21.15.90:0/2055293214 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f19fc1045c0 0x7f19fc104a40 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:27.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.209+0000 7f1a01c00640 1 --2- 172.21.15.90:0/2055293214 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f19fc069240 0x7f19fc0696a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:27.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.209+0000 7f1a01c00640 1 --2- 172.21.15.90:0/2055293214 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f19fc106a50 0x7f19fc068d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:27.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.209+0000 7f1a01c00640 1 -- 172.21.15.90:0/2055293214 >> 172.21.15.90:0/2055293214 conn(0x7f19fc077b00 msgr2=0x7f19fc077f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:48:27.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.210+0000 7f1a01c00640 1 -- 172.21.15.90:0/2055293214 shutdown_connections 2024-04-02T14:48:27.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.210+0000 7f1a01c00640 1 -- 172.21.15.90:0/2055293214 wait complete. 2024-04-02T14:48:27.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.211+0000 7f1a01c00640 1 Processor -- start 2024-04-02T14:48:27.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.212+0000 7f1a01c00640 1 -- start start 2024-04-02T14:48:27.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.212+0000 7f1a01c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f19fc069240 0x7f19fc073250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:27.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.212+0000 7f1a01c00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f19fc1045c0 0x7f19fc073790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:27.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.212+0000 7f1a01c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f19fc106a50 0x7f19fc06c4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:27.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.212+0000 7f1a01c00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f19fc118480 con 0x7f19fc069240 2024-04-02T14:48:27.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.212+0000 7f1a01c00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f19fc118300 con 0x7f19fc1045c0 2024-04-02T14:48:27.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.212+0000 7f1a01c00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f19fc118600 con 0x7f19fc106a50 2024-04-02T14:48:27.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.213+0000 7f19faa00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f19fc069240 0x7f19fc073250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:27.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.213+0000 7f19faa00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f19fc069240 0x7f19fc073250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:42204/0 (socket says 172.21.15.90:42204) 2024-04-02T14:48:27.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.213+0000 7f19faa00640 1 -- 172.21.15.90:0/793038828 learned_addr learned my addr 172.21.15.90:0/793038828 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:48:27.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.213+0000 7f19fb400640 1 --2- 172.21.15.90:0/793038828 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f19fc106a50 0x7f19fc06c4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:27.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.213+0000 7f19fa000640 1 --2- 172.21.15.90:0/793038828 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f19fc1045c0 0x7f19fc073790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:27.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.213+0000 7f19fb400640 1 -- 172.21.15.90:0/793038828 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f19fc1045c0 msgr2=0x7f19fc073790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:48:27.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.213+0000 7f19fb400640 1 --2- 172.21.15.90:0/793038828 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f19fc1045c0 0x7f19fc073790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:27.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.213+0000 7f19fb400640 1 -- 172.21.15.90:0/793038828 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f19fc069240 msgr2=0x7f19fc073250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:48:27.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.213+0000 7f19fb400640 1 --2- 172.21.15.90:0/793038828 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f19fc069240 0x7f19fc073250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:27.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.213+0000 7f19fb400640 1 -- 172.21.15.90:0/793038828 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19fc06cd70 con 0x7f19fc106a50 2024-04-02T14:48:27.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.214+0000 7f19faa00640 1 --2- 172.21.15.90:0/793038828 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f19fc069240 0x7f19fc073250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:48:27.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.214+0000 7f19fb400640 1 --2- 172.21.15.90:0/793038828 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f19fc106a50 0x7f19fc06c4b0 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f19f402c430 tx=0x7f19f4033780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:48:27.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.214+0000 7f19f3400640 1 -- 172.21.15.90:0/793038828 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19f4043020 con 0x7f19fc106a50 2024-04-02T14:48:27.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.215+0000 7f19f3400640 1 -- 172.21.15.90:0/793038828 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f19f40338f0 con 0x7f19fc106a50 2024-04-02T14:48:27.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.215+0000 7f19f3400640 1 -- 172.21.15.90:0/793038828 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19f403e6b0 con 0x7f19fc106a50 2024-04-02T14:48:27.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.215+0000 7f1a01c00640 1 -- 172.21.15.90:0/793038828 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f19fc06cfa0 con 0x7f19fc106a50 2024-04-02T14:48:27.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.215+0000 7f1a01c00640 1 -- 172.21.15.90:0/793038828 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19fc070f50 con 0x7f19fc106a50 2024-04-02T14:48:27.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.220+0000 7f19f3400640 1 -- 172.21.15.90:0/793038828 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f19f4046020 con 0x7f19fc106a50 2024-04-02T14:48:27.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.221+0000 7f1a04858740 1 Processor -- start 2024-04-02T14:48:27.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.222+0000 7f1a04858740 1 -- start start 2024-04-02T14:48:27.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.222+0000 7f1a04858740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f2dbead030 0x55f2dbeabd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:27.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.222+0000 7f1a04858740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f2dbeac350 0x55f2dbeac7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:27.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.222+0000 7f1a04858740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f2dbeb56c0 0x55f2dbeb7ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:27.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.222+0000 7f1a04858740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55f2dbe10310 con 0x55f2dbead030 2024-04-02T14:48:27.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.222+0000 7f1a04858740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55f2dbdb8690 con 0x55f2dbeb56c0 2024-04-02T14:48:27.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.222+0000 7f1a04858740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55f2dbe0c2c0 con 0x55f2dbeac350 2024-04-02T14:48:27.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.223+0000 7f19faa00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f2dbeac350 0x55f2dbeac7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:27.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.223+0000 7f19faa00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f2dbeac350 0x55f2dbeac7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:38372/0 (socket says 172.21.15.90:38372) 2024-04-02T14:48:27.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.223+0000 7f19faa00640 1 -- 172.21.15.90:0/3945615148 learned_addr learned my addr 172.21.15.90:0/3945615148 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:48:27.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.225+0000 7f19faa00640 1 -- 172.21.15.90:0/3945615148 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f2dbeb56c0 msgr2=0x55f2dbeb7ab0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-02T14:48:27.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.225+0000 7f19faa00640 1 --2- 172.21.15.90:0/3945615148 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f2dbeb56c0 0x55f2dbeb7ab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:27.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.225+0000 7f19faa00640 1 -- 172.21.15.90:0/3945615148 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f2dbead030 msgr2=0x55f2dbeabd80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-02T14:48:27.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.226+0000 7f19fa000640 1 --2- 172.21.15.90:0/3945615148 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f2dbead030 0x55f2dbeabd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:27.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.226+0000 7f19faa00640 1 --2- 172.21.15.90:0/3945615148 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f2dbead030 0x55f2dbeabd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:27.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.226+0000 7f19faa00640 1 -- 172.21.15.90:0/3945615148 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f2dbbec7e0 con 0x55f2dbeac350 2024-04-02T14:48:27.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.226+0000 7f19f3400640 1 --2- 172.21.15.90:0/793038828 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f19e00a3d20 0x7f19e00a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:27.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.227+0000 7f19faa00640 1 --2- 172.21.15.90:0/3945615148 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f2dbeac350 0x55f2dbeac7d0 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f19ec00bd50 tx=0x7f19ec00f730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d2bda15b7465df9b server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:48:27.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.227+0000 7f19f3400640 1 -- 172.21.15.90:0/793038828 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 8550+0+0 (secure 0 0 0) 0x7f19f40f2f80 con 0x7f19fc106a50 2024-04-02T14:48:27.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.228+0000 7f19fa000640 1 --2- 172.21.15.90:0/793038828 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f19e00a3d20 0x7f19e00a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:27.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.228+0000 7f19e7e00640 1 -- 172.21.15.90:0/3945615148 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19ec004020 con 0x55f2dbeac350 2024-04-02T14:48:27.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.228+0000 7f19e7e00640 1 -- 172.21.15.90:0/3945615148 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f19ec0041c0 con 0x55f2dbeac350 2024-04-02T14:48:27.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.228+0000 7f19e7e00640 1 -- 172.21.15.90:0/3945615148 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19ec0118e0 con 0x55f2dbeac350 2024-04-02T14:48:27.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.229+0000 7f1a04858740 1 -- 172.21.15.90:0/3945615148 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f2dbeac350 msgr2=0x55f2dbeac7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:48:27.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.229+0000 7f1a04858740 1 --2- 172.21.15.90:0/3945615148 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f2dbeac350 0x55f2dbeac7d0 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f19ec00bd50 tx=0x7f19ec00f730 comp rx=0 tx=0).stop 2024-04-02T14:48:27.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.229+0000 7f1a04858740 1 -- 172.21.15.90:0/3945615148 shutdown_connections 2024-04-02T14:48:27.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.229+0000 7f1a04858740 1 --2- 172.21.15.90:0/3945615148 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f2dbeb56c0 0x55f2dbeb7ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:27.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.229+0000 7f1a04858740 1 --2- 172.21.15.90:0/3945615148 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f2dbeac350 0x55f2dbeac7d0 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:27.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.229+0000 7f1a04858740 1 --2- 172.21.15.90:0/3945615148 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f2dbead030 0x55f2dbeabd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:27.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.229+0000 7f1a04858740 1 -- 172.21.15.90:0/3945615148 >> 172.21.15.90:0/3945615148 conn(0x55f2dbe40270 msgr2=0x55f2dbea6da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:48:27.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.230+0000 7f1a04858740 1 -- 172.21.15.90:0/3945615148 shutdown_connections 2024-04-02T14:48:27.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.230+0000 7f1a04858740 1 -- 172.21.15.90:0/3945615148 wait complete. 2024-04-02T14:48:27.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.230+0000 7f19fa000640 1 --2- 172.21.15.90:0/793038828 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f19e00a3d20 0x7f19e00a61e0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f19e8005fd0 tx=0x7f19e8005850 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:48:27.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.231+0000 7f1a04858740 1 -- start start 2024-04-02T14:48:27.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.231+0000 7f1a04858740 1 Processor -- start 2024-04-02T14:48:27.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.231+0000 7f1a04858740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f2dbeac350 0x55f2dbeb26c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:27.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.231+0000 7f1a04858740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f2dbead030 0x55f2dbeb2c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:27.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.231+0000 7f1a04858740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f2dbeb56c0 0x55f2dbeb3140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:27.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.231+0000 7f1a04858740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55f2dbec9540 con 0x55f2dbeac350 2024-04-02T14:48:27.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.231+0000 7f1a04858740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55f2dbdcba60 con 0x55f2dbead030 2024-04-02T14:48:27.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.231+0000 7f1a04858740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55f2dbec96c0 con 0x55f2dbeb56c0 2024-04-02T14:48:27.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.231+0000 7f1a04858740 10 client.0 _ll_register_callbacks cb 0x55f2dbdd1030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:48:27.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.232+0000 7f19faa00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f2dbeac350 0x55f2dbeb26c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:27.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.232+0000 7f19fa000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f2dbead030 0x55f2dbeb2c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:27.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.232+0000 7f19fa000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f2dbead030 0x55f2dbeb2c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:54470/0 (socket says 172.21.15.90:54470) 2024-04-02T14:48:27.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.232+0000 7f19fa000640 1 -- 172.21.15.90:0/4022035999 learned_addr learned my addr 172.21.15.90:0/4022035999 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:48:27.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.232+0000 7f19fb400640 1 --2- 172.21.15.90:0/4022035999 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f2dbeb56c0 0x55f2dbeb3140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:27.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.232+0000 7f19fa000640 1 -- 172.21.15.90:0/4022035999 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f2dbeb56c0 msgr2=0x55f2dbeb3140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:48:27.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.232+0000 7f19fa000640 1 --2- 172.21.15.90:0/4022035999 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f2dbeb56c0 0x55f2dbeb3140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:27.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.232+0000 7f19fa000640 1 -- 172.21.15.90:0/4022035999 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f2dbeac350 msgr2=0x55f2dbeb26c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:48:27.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.232+0000 7f19fa000640 1 --2- 172.21.15.90:0/4022035999 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f2dbeac350 0x55f2dbeb26c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:27.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.232+0000 7f19fa000640 1 -- 172.21.15.90:0/4022035999 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f2dbc4cf70 con 0x55f2dbead030 2024-04-02T14:48:27.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.233+0000 7f19faa00640 1 --2- 172.21.15.90:0/4022035999 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f2dbeac350 0x55f2dbeb26c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:48:27.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.233+0000 7f19fa000640 1 --2- 172.21.15.90:0/4022035999 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f2dbead030 0x55f2dbeb2c00 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7f19e8010fb0 tx=0x7f19e8004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bb7017f40aab95fc server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:48:27.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.233+0000 7f19e4c00640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:48:27.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.234+0000 7f19e4c00640 1 -- 172.21.15.90:0/4022035999 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19e8023070 con 0x55f2dbead030 2024-04-02T14:48:27.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.235+0000 7f19e4c00640 1 -- 172.21.15.90:0/4022035999 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 8550+0+0 (secure 0 0 0) 0x7f19e801f870 con 0x55f2dbead030 2024-04-02T14:48:27.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.235+0000 7f1a04858740 1 -- 172.21.15.90:0/4022035999 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f19ec0041c0 con 0x55f2dbead030 2024-04-02T14:48:27.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.235+0000 7f19e4c00640 1 -- 172.21.15.90:0/4022035999 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f19e802d050 con 0x55f2dbead030 2024-04-02T14:48:27.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.235+0000 7f1a04858740 10 client.29726 fetch_fsmap learned FSMap version 27 2024-04-02T14:48:27.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.235+0000 7f1a04858740 1 -- 172.21.15.90:0/4022035999 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55f2dbae4170 con 0x55f2dbead030 2024-04-02T14:48:27.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.236+0000 7f19e4c00640 1 -- 172.21.15.90:0/4022035999 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f19ec0041c0 con 0x55f2dbead030 2024-04-02T14:48:27.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.236+0000 7f19e4c00640 1 -- 172.21.15.90:0/4022035999 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f19e800f040 con 0x55f2dbead030 2024-04-02T14:48:27.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.236+0000 7f1a04858740 10 client.29726 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:48:27.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.236+0000 7f1a04858740 10 client.29726 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:48:27.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.236+0000 7f1a04858740 20 client.29726 populate_metadata read hostname 'smithi090' 2024-04-02T14:48:27.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.236+0000 7f1a04858740 1 --2- 172.21.15.90:0/4022035999 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55f2dbed6c70 0x55f2dbed90c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:27.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.237+0000 7f1a04858740 4 client.29726 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:48:27.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.237+0000 7f1a04858740 1 -- 172.21.15.90:0/4022035999 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f2dbe1d850 con 0x55f2dbed6c70 2024-04-02T14:48:27.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.237+0000 7f19faa00640 1 --2- 172.21.15.90:0/4022035999 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55f2dbed6c70 0x55f2dbed90c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:27.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.239+0000 7f19e4c00640 10 client.29726 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:48:27.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.239+0000 7f19faa00640 1 --2- 172.21.15.90:0/4022035999 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55f2dbed6c70 0x55f2dbed90c0 crc :-1 s=READY pgs=348 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b662c1814193fc35 server_cookie=32a2f7bd87e24c84 in_seq=0 out_seq=0 2024-04-02T14:48:27.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.239+0000 7f19e4c00640 1 -- 172.21.15.90:0/4022035999 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f2dbe1d850 con 0x55f2dbed6c70 2024-04-02T14:48:27.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.240+0000 7f19e4c00640 10 client.29726 handle_command_reply: tid=0 2024-04-02T14:48:27.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.240+0000 7f1a04858740 1 client.29726 shutdown 2024-04-02T14:48:27.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.240+0000 7f1a04858740 1 -- 172.21.15.90:0/4022035999 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f2dbead030 msgr2=0x55f2dbeb2c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:48:27.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.240+0000 7f1a04858740 1 --2- 172.21.15.90:0/4022035999 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f2dbead030 0x55f2dbeb2c00 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7f19e8010fb0 tx=0x7f19e8004730 comp rx=0 tx=0).stop 2024-04-02T14:48:27.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.240+0000 7f1a04858740 1 -- 172.21.15.90:0/4022035999 shutdown_connections 2024-04-02T14:48:27.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.240+0000 7f1a04858740 1 --2- 172.21.15.90:0/4022035999 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55f2dbed6c70 0x55f2dbed90c0 crc :-1 s=READY pgs=348 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:27.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.240+0000 7f1a04858740 1 --2- 172.21.15.90:0/4022035999 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f2dbeb56c0 0x55f2dbeb3140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:27.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.240+0000 7f1a04858740 1 --2- 172.21.15.90:0/4022035999 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f2dbead030 0x55f2dbeb2c00 unknown :-1 s=CLOSED pgs=937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:27.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.240+0000 7f1a04858740 1 --2- 172.21.15.90:0/4022035999 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f2dbeac350 0x55f2dbeb26c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:27.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.240+0000 7f1a04858740 1 -- 172.21.15.90:0/4022035999 >> 172.21.15.90:0/4022035999 conn(0x55f2dbe40270 msgr2=0x55f2dbea8af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:48:27.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.240+0000 7f19e4c00640 0 client.29726 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:48:27.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.240+0000 7f1a04858740 1 -- 172.21.15.90:0/4022035999 shutdown_connections 2024-04-02T14:48:27.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.240+0000 7f1a04858740 1 -- 172.21.15.90:0/4022035999 wait complete. 2024-04-02T14:48:27.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.241+0000 7f1a04858740 20 client.29726 trim_cache size 0 max 16384 2024-04-02T14:48:27.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.276+0000 7f1a04858740 1 Processor -- start 2024-04-02T14:48:27.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.277+0000 7f1a04858740 1 -- start start 2024-04-02T14:48:27.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.277+0000 7f1a04858740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f2dbeac350 0x55f2dbeb4df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:27.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.277+0000 7f1a04858740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f2dbead030 0x55f2dbed2440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:27.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.277+0000 7f1a04858740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f2dbeb56c0 0x55f2dbede9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:27.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.277+0000 7f1a04858740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f19fc10fe00 con 0x55f2dbead030 2024-04-02T14:48:27.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.277+0000 7f1a04858740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f19fc1b5a30 con 0x55f2dbeb56c0 2024-04-02T14:48:27.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.277+0000 7f1a04858740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f19fc1104d0 con 0x55f2dbeac350 2024-04-02T14:48:27.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.277+0000 7f19fb400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f2dbeb56c0 0x55f2dbede9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:27.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.277+0000 7f19faa00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f2dbeac350 0x55f2dbeb4df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:27.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.277+0000 7f19fb400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f2dbeb56c0 0x55f2dbede9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:54486/0 (socket says 172.21.15.90:54486) 2024-04-02T14:48:27.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.277+0000 7f19fb400640 1 -- 172.21.15.90:0/3945257043 learned_addr learned my addr 172.21.15.90:0/3945257043 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:48:27.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.278+0000 7f19fb400640 1 -- 172.21.15.90:0/3945257043 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f2dbeac350 msgr2=0x55f2dbeb4df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:48:27.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.278+0000 7f19fb400640 1 --2- 172.21.15.90:0/3945257043 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f2dbeac350 0x55f2dbeb4df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:27.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.278+0000 7f19fb400640 1 -- 172.21.15.90:0/3945257043 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f2dbead030 msgr2=0x55f2dbed2440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:48:27.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.278+0000 7f19fa000640 1 --2- 172.21.15.90:0/3945257043 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f2dbead030 0x55f2dbed2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:27.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.278+0000 7f19fb400640 1 --2- 172.21.15.90:0/3945257043 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f2dbead030 0x55f2dbed2440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:27.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.278+0000 7f19fb400640 1 -- 172.21.15.90:0/3945257043 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f2dbed57c0 con 0x55f2dbeb56c0 2024-04-02T14:48:27.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.278+0000 7f19fa000640 1 --2- 172.21.15.90:0/3945257043 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f2dbead030 0x55f2dbed2440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:48:27.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.278+0000 7f19fb400640 1 --2- 172.21.15.90:0/3945257043 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f2dbeb56c0 0x55f2dbede9d0 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7f19f4005ee0 tx=0x7f19f40bb380 comp rx=0 tx=0).ready entity=mon.2 client_cookie=52bc25ecb1cc5753 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:48:27.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.278+0000 7f19e7400640 1 -- 172.21.15.90:0/3945257043 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19f40dd070 con 0x55f2dbeb56c0 2024-04-02T14:48:27.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.279+0000 7f19e7400640 1 -- 172.21.15.90:0/3945257043 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f19f40bf6b0 con 0x55f2dbeb56c0 2024-04-02T14:48:27.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.279+0000 7f19e7400640 1 -- 172.21.15.90:0/3945257043 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19f40d8070 con 0x55f2dbeb56c0 2024-04-02T14:48:27.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.279+0000 7f1a04858740 1 -- 172.21.15.90:0/3945257043 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f2dbeb56c0 msgr2=0x55f2dbede9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:48:27.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.279+0000 7f1a04858740 1 --2- 172.21.15.90:0/3945257043 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f2dbeb56c0 0x55f2dbede9d0 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7f19f4005ee0 tx=0x7f19f40bb380 comp rx=0 tx=0).stop 2024-04-02T14:48:27.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.280+0000 7f1a04858740 1 -- 172.21.15.90:0/3945257043 shutdown_connections 2024-04-02T14:48:27.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.280+0000 7f1a04858740 1 --2- 172.21.15.90:0/3945257043 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f2dbeb56c0 0x55f2dbede9d0 unknown :-1 s=CLOSED pgs=938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:27.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.280+0000 7f1a04858740 1 --2- 172.21.15.90:0/3945257043 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f2dbead030 0x55f2dbed2440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:27.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.280+0000 7f1a04858740 1 --2- 172.21.15.90:0/3945257043 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f2dbeac350 0x55f2dbeb4df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:27.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.280+0000 7f1a04858740 1 -- 172.21.15.90:0/3945257043 >> 172.21.15.90:0/3945257043 conn(0x55f2dbe40270 msgr2=0x55f2dbea8a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:48:27.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.280+0000 7f1a04858740 1 -- 172.21.15.90:0/3945257043 shutdown_connections 2024-04-02T14:48:27.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.281+0000 7f1a04858740 1 -- 172.21.15.90:0/3945257043 wait complete. 2024-04-02T14:48:27.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.281+0000 7f1a04858740 1 -- start start 2024-04-02T14:48:27.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.282+0000 7f1a04858740 1 Processor -- start 2024-04-02T14:48:27.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.282+0000 7f1a04858740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f2dbeac350 0x55f2dbecebb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:27.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.282+0000 7f1a04858740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f2dbead030 0x55f2dbeae930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:27.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.282+0000 7f1a04858740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f2dbeb56c0 0x55f2dbeaee70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:27.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.282+0000 7f1a04858740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f19fc1b5360 con 0x55f2dbeac350 2024-04-02T14:48:27.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.282+0000 7f1a04858740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f19fc1b4c90 con 0x55f2dbeb56c0 2024-04-02T14:48:27.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.282+0000 7f1a04858740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f19fc10de00 con 0x55f2dbead030 2024-04-02T14:48:27.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.282+0000 7f1a04858740 10 client.0 _ll_register_callbacks cb 0x7f19e8016050 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:48:27.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.282+0000 7f19fb400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f2dbeb56c0 0x55f2dbeaee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:27.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.283+0000 7f19fb400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f2dbeb56c0 0x55f2dbeaee70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:54488/0 (socket says 172.21.15.90:54488) 2024-04-02T14:48:27.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.283+0000 7f19fb400640 1 -- 172.21.15.90:0/2353250752 learned_addr learned my addr 172.21.15.90:0/2353250752 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:48:27.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.283+0000 7f19faa00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f2dbeac350 0x55f2dbecebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:27.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.283+0000 7f19fa000640 1 --2- 172.21.15.90:0/2353250752 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f2dbead030 0x55f2dbeae930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:27.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.283+0000 7f19fb400640 1 -- 172.21.15.90:0/2353250752 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f2dbead030 msgr2=0x55f2dbeae930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:48:27.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.283+0000 7f19fb400640 1 --2- 172.21.15.90:0/2353250752 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f2dbead030 0x55f2dbeae930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:27.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.283+0000 7f19fb400640 1 -- 172.21.15.90:0/2353250752 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f2dbeac350 msgr2=0x55f2dbecebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:48:27.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.283+0000 7f19fb400640 1 --2- 172.21.15.90:0/2353250752 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f2dbeac350 0x55f2dbecebb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:27.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.283+0000 7f19fb400640 1 -- 172.21.15.90:0/2353250752 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f2dbeaf4e0 con 0x55f2dbeb56c0 2024-04-02T14:48:27.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.283+0000 7f19fb400640 1 --2- 172.21.15.90:0/2353250752 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f2dbeb56c0 0x55f2dbeaee70 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f19f40bb410 tx=0x7f19f4004700 comp rx=0 tx=0).ready entity=mon.2 client_cookie=26ff51da6cb123f6 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:48:27.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.283+0000 7f19e4c00640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:48:27.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.283+0000 7f19e4c00640 1 -- 172.21.15.90:0/2353250752 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19f40dd070 con 0x55f2dbeb56c0 2024-04-02T14:48:27.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.283+0000 7f19e4c00640 1 -- 172.21.15.90:0/2353250752 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 8550+0+0 (secure 0 0 0) 0x7f19f40c7790 con 0x55f2dbeb56c0 2024-04-02T14:48:27.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.283+0000 7f1a04858740 1 -- 172.21.15.90:0/2353250752 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f19f40bf6b0 con 0x55f2dbeb56c0 2024-04-02T14:48:27.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.284+0000 7f19e4c00640 1 -- 172.21.15.90:0/2353250752 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f19f40d8040 con 0x55f2dbeb56c0 2024-04-02T14:48:27.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.284+0000 7f1a04858740 10 client.29732 fetch_fsmap learned FSMap version 27 2024-04-02T14:48:27.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.284+0000 7f1a04858740 1 -- 172.21.15.90:0/2353250752 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55f2dbf3e250 con 0x55f2dbeb56c0 2024-04-02T14:48:27.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.284+0000 7f19e4c00640 1 -- 172.21.15.90:0/2353250752 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f19f40bf6b0 con 0x55f2dbeb56c0 2024-04-02T14:48:27.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.284+0000 7f19e4c00640 1 -- 172.21.15.90:0/2353250752 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f19f4004b40 con 0x55f2dbeb56c0 2024-04-02T14:48:27.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.284+0000 7f1a04858740 10 client.29732 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:48:27.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.284+0000 7f1a04858740 10 client.29732 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:48:27.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.284+0000 7f1a04858740 20 client.29732 populate_metadata read hostname 'smithi090' 2024-04-02T14:48:27.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.284+0000 7f1a04858740 1 --2- 172.21.15.90:0/2353250752 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55f2dbf3e670 0x55f2dbf40a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:27.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.284+0000 7f1a04858740 4 client.29732 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-02T14:48:27.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.284+0000 7f1a04858740 1 -- 172.21.15.90:0/2353250752 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55f2dbe17ba0 con 0x55f2dbf3e670 2024-04-02T14:48:27.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.284+0000 7f19faa00640 1 --2- 172.21.15.90:0/2353250752 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55f2dbf3e670 0x55f2dbf40a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:27.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.285+0000 7f19e4c00640 10 client.29732 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:48:27.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.285+0000 7f19faa00640 1 --2- 172.21.15.90:0/2353250752 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55f2dbf3e670 0x55f2dbf40a60 crc :-1 s=READY pgs=349 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a3fd8545227d8a59 server_cookie=9a0861a3d0658ef3 in_seq=0 out_seq=0 2024-04-02T14:48:27.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.286+0000 7f19e4c00640 1 -- 172.21.15.90:0/2353250752 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55f2dbe17ba0 con 0x55f2dbf3e670 2024-04-02T14:48:27.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.286+0000 7f19e4c00640 10 client.29732 handle_command_reply: tid=0 2024-04-02T14:48:27.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.286+0000 7f1a04858740 1 client.29732 shutdown 2024-04-02T14:48:27.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.287+0000 7f1a04858740 1 -- 172.21.15.90:0/2353250752 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f2dbeb56c0 msgr2=0x55f2dbeaee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:48:27.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.287+0000 7f1a04858740 1 --2- 172.21.15.90:0/2353250752 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f2dbeb56c0 0x55f2dbeaee70 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f19f40bb410 tx=0x7f19f4004700 comp rx=0 tx=0).stop 2024-04-02T14:48:27.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.288+0000 7f1a04858740 1 -- 172.21.15.90:0/2353250752 shutdown_connections 2024-04-02T14:48:27.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.288+0000 7f1a04858740 1 --2- 172.21.15.90:0/2353250752 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55f2dbf3e670 0x55f2dbf40a60 crc :-1 s=READY pgs=349 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:27.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.288+0000 7f1a04858740 1 --2- 172.21.15.90:0/2353250752 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f2dbeb56c0 0x55f2dbeaee70 unknown :-1 s=CLOSED pgs=939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:27.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.288+0000 7f1a04858740 1 --2- 172.21.15.90:0/2353250752 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f2dbead030 0x55f2dbeae930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:27.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.288+0000 7f1a04858740 1 --2- 172.21.15.90:0/2353250752 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f2dbeac350 0x55f2dbecebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:27.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.288+0000 7f1a04858740 1 -- 172.21.15.90:0/2353250752 >> 172.21.15.90:0/2353250752 conn(0x55f2dbe40270 msgr2=0x55f2dbeb25e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:48:27.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.288+0000 7f19e4c00640 0 client.29732 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:48:27.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.288+0000 7f1a04858740 1 -- 172.21.15.90:0/2353250752 shutdown_connections 2024-04-02T14:48:27.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.289+0000 7f1a04858740 1 -- 172.21.15.90:0/2353250752 wait complete. 2024-04-02T14:48:27.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.289+0000 7f1a04858740 20 client.29732 trim_cache size 0 max 16384 2024-04-02T14:48:27.284 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-02T14:48:27.284 INFO:teuthology.orchestra.run.smithi090.stdout: "return_code": 0, 2024-04-02T14:48:27.284 INFO:teuthology.orchestra.run.smithi090.stdout: "scrub_tag": "088a64b5-afdf-4d58-abe6-bff35505816b", 2024-04-02T14:48:27.284 INFO:teuthology.orchestra.run.smithi090.stdout: "mode": "asynchronous" 2024-04-02T14:48:27.285 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-02T14:48:27.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.290+0000 7f1a01c00640 1 -- 172.21.15.90:0/793038828 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f19e00a3d20 msgr2=0x7f19e00a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:48:27.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.290+0000 7f1a01c00640 1 --2- 172.21.15.90:0/793038828 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f19e00a3d20 0x7f19e00a61e0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f19e8005fd0 tx=0x7f19e8005850 comp rx=0 tx=0).stop 2024-04-02T14:48:27.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.290+0000 7f1a01c00640 1 -- 172.21.15.90:0/793038828 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f19fc106a50 msgr2=0x7f19fc06c4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:48:27.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.290+0000 7f1a01c00640 1 --2- 172.21.15.90:0/793038828 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f19fc106a50 0x7f19fc06c4b0 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f19f402c430 tx=0x7f19f4033780 comp rx=0 tx=0).stop 2024-04-02T14:48:27.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.290+0000 7f1a01c00640 1 -- 172.21.15.90:0/793038828 shutdown_connections 2024-04-02T14:48:27.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.290+0000 7f1a01c00640 1 --2- 172.21.15.90:0/793038828 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f19e00a3d20 0x7f19e00a61e0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:27.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.290+0000 7f1a01c00640 1 --2- 172.21.15.90:0/793038828 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f19fc106a50 0x7f19fc06c4b0 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:27.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.290+0000 7f1a01c00640 1 --2- 172.21.15.90:0/793038828 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f19fc1045c0 0x7f19fc073790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:27.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.290+0000 7f1a01c00640 1 --2- 172.21.15.90:0/793038828 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f19fc069240 0x7f19fc073250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:27.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.290+0000 7f1a01c00640 1 -- 172.21.15.90:0/793038828 >> 172.21.15.90:0/793038828 conn(0x7f19fc077b00 msgr2=0x7f19fc1018b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:48:27.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.291+0000 7f1a01c00640 1 -- 172.21.15.90:0/793038828 shutdown_connections 2024-04-02T14:48:27.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.291+0000 7f1a01c00640 1 -- 172.21.15.90:0/793038828 wait complete. 2024-04-02T14:48:27.306 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:48:27 smithi090 ceph-mon[31674]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T14:48:27.312 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-02T14:48:27.312 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-02T14:48:27.603 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:48:27 smithi092 ceph-mon[36497]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T14:48:27.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:48:27 smithi177 ceph-mon[36431]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T14:48:27.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.691+0000 7f346ce00640 1 -- 172.21.15.90:0/757769110 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3468108cc0 msgr2=0x7f346810cd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:48:27.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.691+0000 7f346ce00640 1 --2- 172.21.15.90:0/757769110 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3468108cc0 0x7f346810cd00 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f3454009900 tx=0x7f345402bc80 comp rx=0 tx=0).stop 2024-04-02T14:48:27.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.692+0000 7f346ce00640 1 -- 172.21.15.90:0/757769110 shutdown_connections 2024-04-02T14:48:27.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.692+0000 7f346ce00640 1 --2- 172.21.15.90:0/757769110 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f346810d3e0 0x7f3468114fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:27.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.692+0000 7f346ce00640 1 --2- 172.21.15.90:0/757769110 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3468108cc0 0x7f346810cd00 unknown :-1 s=CLOSED pgs=940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:27.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.692+0000 7f346ce00640 1 --2- 172.21.15.90:0/757769110 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3468108310 0x7f34681086f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:27.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.692+0000 7f346ce00640 1 -- 172.21.15.90:0/757769110 >> 172.21.15.90:0/757769110 conn(0x7f3468076040 msgr2=0x7f3468078460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:48:27.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.692+0000 7f346ce00640 1 -- 172.21.15.90:0/757769110 shutdown_connections 2024-04-02T14:48:27.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.692+0000 7f346ce00640 1 -- 172.21.15.90:0/757769110 wait complete. 2024-04-02T14:48:27.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.693+0000 7f346ce00640 1 Processor -- start 2024-04-02T14:48:27.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.694+0000 7f346ce00640 1 -- start start 2024-04-02T14:48:27.690 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.695+0000 7f346ce00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3468108310 0x7f34681a4690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:27.690 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.695+0000 7f346ce00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f346810d3e0 0x7f34681a4bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:27.690 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.695+0000 7f346ce00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f346819e800 0x7f346819ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:27.690 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.695+0000 7f346ce00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f34681184a0 con 0x7f346810d3e0 2024-04-02T14:48:27.690 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.695+0000 7f346ce00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f3468118320 con 0x7f3468108310 2024-04-02T14:48:27.690 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.695+0000 7f346ce00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f3468118620 con 0x7f346819e800 2024-04-02T14:48:27.691 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.696+0000 7f3465e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3468108310 0x7f34681a4690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:27.691 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.696+0000 7f3465e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3468108310 0x7f34681a4690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:54506/0 (socket says 172.21.15.90:54506) 2024-04-02T14:48:27.691 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.696+0000 7f3465e00640 1 -- 172.21.15.90:0/2050718799 learned_addr learned my addr 172.21.15.90:0/2050718799 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:48:27.691 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.696+0000 7f3465400640 1 --2- 172.21.15.90:0/2050718799 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f346810d3e0 0x7f34681a4bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:27.691 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.696+0000 7f3466800640 1 --2- 172.21.15.90:0/2050718799 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f346819e800 0x7f346819ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:27.691 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.697+0000 7f3465400640 1 -- 172.21.15.90:0/2050718799 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f346819e800 msgr2=0x7f346819ecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:48:27.691 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.697+0000 7f3465400640 1 --2- 172.21.15.90:0/2050718799 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f346819e800 0x7f346819ecb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:27.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.697+0000 7f3465400640 1 -- 172.21.15.90:0/2050718799 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3468108310 msgr2=0x7f34681a4690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:48:27.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.697+0000 7f3465400640 1 --2- 172.21.15.90:0/2050718799 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3468108310 0x7f34681a4690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:27.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.697+0000 7f3465400640 1 -- 172.21.15.90:0/2050718799 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f346819f390 con 0x7f346810d3e0 2024-04-02T14:48:27.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.697+0000 7f3465e00640 1 --2- 172.21.15.90:0/2050718799 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3468108310 0x7f34681a4690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:48:27.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.697+0000 7f3465400640 1 --2- 172.21.15.90:0/2050718799 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f346810d3e0 0x7f34681a4bd0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f3454009850 tx=0x7f345402ea40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:48:27.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.698+0000 7f345ea00640 1 -- 172.21.15.90:0/2050718799 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3454041020 con 0x7f346810d3e0 2024-04-02T14:48:27.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.698+0000 7f345ea00640 1 -- 172.21.15.90:0/2050718799 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f345402edc0 con 0x7f346810d3e0 2024-04-02T14:48:27.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.698+0000 7f346ce00640 1 -- 172.21.15.90:0/2050718799 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f346819f620 con 0x7f346810d3e0 2024-04-02T14:48:27.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.698+0000 7f345ea00640 1 -- 172.21.15.90:0/2050718799 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3454004c80 con 0x7f346810d3e0 2024-04-02T14:48:27.694 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.699+0000 7f346ce00640 1 -- 172.21.15.90:0/2050718799 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34681ab7d0 con 0x7f346810d3e0 2024-04-02T14:48:27.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.705+0000 7f346fa12740 1 Processor -- start 2024-04-02T14:48:27.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.706+0000 7f346fa12740 1 -- start start 2024-04-02T14:48:27.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.706+0000 7f345ea00640 1 -- 172.21.15.90:0/2050718799 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f3454044020 con 0x7f346810d3e0 2024-04-02T14:48:27.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.706+0000 7f346fa12740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b5a04187b0 0x55b5a0418b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:27.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.706+0000 7f346fa12740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b5a04190d0 0x55b5a0421770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:27.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.706+0000 7f346fa12740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b5a0421cb0 0x55b5a04240a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:27.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.706+0000 7f346fa12740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55b5a037c770 con 0x55b5a04190d0 2024-04-02T14:48:27.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.706+0000 7f346fa12740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55b5a0324700 con 0x55b5a04187b0 2024-04-02T14:48:27.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.706+0000 7f346fa12740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55b5a0378720 con 0x55b5a0421cb0 2024-04-02T14:48:27.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.708+0000 7f3466800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b5a04187b0 0x55b5a0418b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:27.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.708+0000 7f3466800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b5a04187b0 0x55b5a0418b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:54516/0 (socket says 172.21.15.90:54516) 2024-04-02T14:48:27.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.708+0000 7f3466800640 1 -- 172.21.15.90:0/714838982 learned_addr learned my addr 172.21.15.90:0/714838982 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:48:27.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.708+0000 7f3465400640 1 --2- 172.21.15.90:0/714838982 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b5a0421cb0 0x55b5a04240a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:27.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.708+0000 7f3465e00640 1 --2- 172.21.15.90:0/714838982 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b5a04190d0 0x55b5a0421770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:27.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.708+0000 7f3466800640 1 -- 172.21.15.90:0/714838982 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b5a0421cb0 msgr2=0x55b5a04240a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:48:27.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.708+0000 7f3466800640 1 --2- 172.21.15.90:0/714838982 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b5a0421cb0 0x55b5a04240a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:27.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.708+0000 7f3466800640 1 -- 172.21.15.90:0/714838982 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b5a04190d0 msgr2=0x55b5a0421770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:48:27.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.709+0000 7f3466800640 1 --2- 172.21.15.90:0/714838982 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b5a04190d0 0x55b5a0421770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:27.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.709+0000 7f3466800640 1 -- 172.21.15.90:0/714838982 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b5a015a7d0 con 0x55b5a04187b0 2024-04-02T14:48:27.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.709+0000 7f3465e00640 1 --2- 172.21.15.90:0/714838982 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b5a04190d0 0x55b5a0421770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:48:27.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.709+0000 7f3466800640 1 --2- 172.21.15.90:0/714838982 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b5a04187b0 0x55b5a0418b90 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f3460009f70 tx=0x7f3460010800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a4f27443bf96dc65 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:48:27.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.709+0000 7f345ea00640 1 --2- 172.21.15.90:0/2050718799 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f344c0a3e60 0x7f344c0a6320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:27.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.710+0000 7f3453400640 1 -- 172.21.15.90:0/714838982 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3460016070 con 0x55b5a04187b0 2024-04-02T14:48:27.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.710+0000 7f3453400640 1 -- 172.21.15.90:0/714838982 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3460004d20 con 0x55b5a04187b0 2024-04-02T14:48:27.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.710+0000 7f3453400640 1 -- 172.21.15.90:0/714838982 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3460012910 con 0x55b5a04187b0 2024-04-02T14:48:27.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.710+0000 7f3465e00640 1 --2- 172.21.15.90:0/2050718799 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f344c0a3e60 0x7f344c0a6320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:27.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.712+0000 7f345ea00640 1 -- 172.21.15.90:0/2050718799 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 8550+0+0 (secure 0 0 0) 0x7f34540f16d0 con 0x7f346810d3e0 2024-04-02T14:48:27.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.713+0000 7f346fa12740 1 -- 172.21.15.90:0/714838982 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b5a04187b0 msgr2=0x55b5a0418b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:48:27.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.713+0000 7f346fa12740 1 --2- 172.21.15.90:0/714838982 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b5a04187b0 0x55b5a0418b90 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f3460009f70 tx=0x7f3460010800 comp rx=0 tx=0).stop 2024-04-02T14:48:27.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.713+0000 7f3465e00640 1 --2- 172.21.15.90:0/2050718799 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f344c0a3e60 0x7f344c0a6320 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x55b5a0418280 tx=0x7f3458005f90 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:48:27.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.713+0000 7f346fa12740 1 -- 172.21.15.90:0/714838982 shutdown_connections 2024-04-02T14:48:27.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.713+0000 7f346fa12740 1 --2- 172.21.15.90:0/714838982 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b5a0421cb0 0x55b5a04240a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:27.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.713+0000 7f346fa12740 1 --2- 172.21.15.90:0/714838982 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b5a04190d0 0x55b5a0421770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:27.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.713+0000 7f346fa12740 1 --2- 172.21.15.90:0/714838982 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b5a04187b0 0x55b5a0418b90 unknown :-1 s=CLOSED pgs=941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:27.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.713+0000 7f346fa12740 1 -- 172.21.15.90:0/714838982 >> 172.21.15.90:0/714838982 conn(0x55b5a0412ee0 msgr2=0x55b5a04132c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:48:27.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.713+0000 7f346fa12740 1 -- 172.21.15.90:0/714838982 shutdown_connections 2024-04-02T14:48:27.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.713+0000 7f346fa12740 1 -- 172.21.15.90:0/714838982 wait complete. 2024-04-02T14:48:27.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.714+0000 7f346fa12740 1 -- start start 2024-04-02T14:48:27.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.715+0000 7f346fa12740 1 Processor -- start 2024-04-02T14:48:27.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.715+0000 7f346fa12740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b5a04187b0 0x55b5a041edc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:27.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.715+0000 7f346fa12740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b5a04190d0 0x55b5a041f300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:27.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.716+0000 7f346fa12740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b5a0421cb0 0x55b5a041f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:27.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.716+0000 7f346fa12740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55b5a0424aa0 con 0x55b5a0421cb0 2024-04-02T14:48:27.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.716+0000 7f346fa12740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55b5a0337ac0 con 0x55b5a04190d0 2024-04-02T14:48:27.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.716+0000 7f346fa12740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55b5a0424c20 con 0x55b5a04187b0 2024-04-02T14:48:27.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.716+0000 7f346fa12740 10 client.0 _ll_register_callbacks cb 0x55b5a033a180 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:48:27.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.716+0000 7f3465400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b5a04190d0 0x55b5a041f300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:27.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.716+0000 7f3465400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b5a04190d0 0x55b5a041f300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:54520/0 (socket says 172.21.15.90:54520) 2024-04-02T14:48:27.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.716+0000 7f3465400640 1 -- 172.21.15.90:0/1194160337 learned_addr learned my addr 172.21.15.90:0/1194160337 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:48:27.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.716+0000 7f3465e00640 1 --2- 172.21.15.90:0/1194160337 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b5a04187b0 0x55b5a041edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:27.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.717+0000 7f3466800640 1 --2- 172.21.15.90:0/1194160337 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b5a0421cb0 0x55b5a041f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:27.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.717+0000 7f3465400640 1 -- 172.21.15.90:0/1194160337 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b5a04187b0 msgr2=0x55b5a041edc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:48:27.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.717+0000 7f3465400640 1 --2- 172.21.15.90:0/1194160337 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b5a04187b0 0x55b5a041edc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:27.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.717+0000 7f3465400640 1 -- 172.21.15.90:0/1194160337 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b5a0421cb0 msgr2=0x55b5a041f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:48:27.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.717+0000 7f3465400640 1 --2- 172.21.15.90:0/1194160337 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b5a0421cb0 0x55b5a041f840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:27.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.718+0000 7f3465400640 1 -- 172.21.15.90:0/1194160337 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b5a00903f0 con 0x55b5a04190d0 2024-04-02T14:48:27.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.718+0000 7f3466800640 1 --2- 172.21.15.90:0/1194160337 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b5a0421cb0 0x55b5a041f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:48:27.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.718+0000 7f3465400640 1 --2- 172.21.15.90:0/1194160337 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b5a04190d0 0x55b5a041f300 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f34540c5e40 tx=0x7f34540092e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=adb522afe71ed63f server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:48:27.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.719+0000 7f3447e00640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:48:27.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.719+0000 7f3447e00640 1 -- 172.21.15.90:0/1194160337 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34540bd4d0 con 0x55b5a04190d0 2024-04-02T14:48:27.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.719+0000 7f346fa12740 1 -- 172.21.15.90:0/1194160337 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3460004d20 con 0x55b5a04190d0 2024-04-02T14:48:27.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.719+0000 7f3447e00640 1 -- 172.21.15.90:0/1194160337 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 8550+0+0 (secure 0 0 0) 0x7f34540c5ee0 con 0x55b5a04190d0 2024-04-02T14:48:27.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.720+0000 7f3447e00640 1 -- 172.21.15.90:0/1194160337 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f34540bd060 con 0x55b5a04190d0 2024-04-02T14:48:27.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.720+0000 7f346fa12740 10 client.29744 fetch_fsmap learned FSMap version 27 2024-04-02T14:48:27.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.720+0000 7f346fa12740 1 -- 172.21.15.90:0/1194160337 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55b5a00500f0 con 0x55b5a04190d0 2024-04-02T14:48:27.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.721+0000 7f3447e00640 1 -- 172.21.15.90:0/1194160337 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3460004d20 con 0x55b5a04190d0 2024-04-02T14:48:27.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.721+0000 7f3447e00640 1 -- 172.21.15.90:0/1194160337 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f34540d6050 con 0x55b5a04190d0 2024-04-02T14:48:27.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.721+0000 7f346fa12740 10 client.29744 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:48:27.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.721+0000 7f346fa12740 10 client.29744 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:48:27.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.721+0000 7f346fa12740 20 client.29744 populate_metadata read hostname 'smithi090' 2024-04-02T14:48:27.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.721+0000 7f346fa12740 1 --2- 172.21.15.90:0/1194160337 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55b5a0431d70 0x55b5a04341d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:27.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.721+0000 7f346fa12740 4 client.29744 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:48:27.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.721+0000 7f346fa12740 1 -- 172.21.15.90:0/1194160337 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b5a031d630 con 0x55b5a0431d70 2024-04-02T14:48:27.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.722+0000 7f3465e00640 1 --2- 172.21.15.90:0/1194160337 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55b5a0431d70 0x55b5a04341d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:27.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.723+0000 7f3465e00640 1 --2- 172.21.15.90:0/1194160337 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55b5a0431d70 0x55b5a04341d0 crc :-1 s=READY pgs=350 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=400b889aea3dd473 server_cookie=45d72233c6b18b9f in_seq=0 out_seq=0 2024-04-02T14:48:27.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.723+0000 7f3447e00640 10 client.29744 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:48:27.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.724+0000 7f3447e00640 1 -- 172.21.15.90:0/1194160337 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b5a031d630 con 0x55b5a0431d70 2024-04-02T14:48:27.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.724+0000 7f3447e00640 10 client.29744 handle_command_reply: tid=0 2024-04-02T14:48:27.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.724+0000 7f346fa12740 1 client.29744 shutdown 2024-04-02T14:48:27.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.725+0000 7f346fa12740 1 -- 172.21.15.90:0/1194160337 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b5a04190d0 msgr2=0x55b5a041f300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:48:27.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.725+0000 7f346fa12740 1 --2- 172.21.15.90:0/1194160337 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b5a04190d0 0x55b5a041f300 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f34540c5e40 tx=0x7f34540092e0 comp rx=0 tx=0).stop 2024-04-02T14:48:27.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.725+0000 7f346fa12740 1 -- 172.21.15.90:0/1194160337 shutdown_connections 2024-04-02T14:48:27.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.725+0000 7f346fa12740 1 --2- 172.21.15.90:0/1194160337 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55b5a0431d70 0x55b5a04341d0 crc :-1 s=READY pgs=350 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:27.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.725+0000 7f346fa12740 1 --2- 172.21.15.90:0/1194160337 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b5a0421cb0 0x55b5a041f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:27.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.725+0000 7f346fa12740 1 --2- 172.21.15.90:0/1194160337 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b5a04190d0 0x55b5a041f300 unknown :-1 s=CLOSED pgs=942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:27.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.725+0000 7f346fa12740 1 --2- 172.21.15.90:0/1194160337 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b5a04187b0 0x55b5a041edc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:27.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.725+0000 7f3447e00640 0 client.29744 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:48:27.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.725+0000 7f346fa12740 1 -- 172.21.15.90:0/1194160337 >> 172.21.15.90:0/1194160337 conn(0x55b5a0412ee0 msgr2=0x55b5a0414c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:48:27.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.725+0000 7f346fa12740 1 -- 172.21.15.90:0/1194160337 shutdown_connections 2024-04-02T14:48:27.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.725+0000 7f346fa12740 1 -- 172.21.15.90:0/1194160337 wait complete. 2024-04-02T14:48:27.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.725+0000 7f346fa12740 20 client.29744 trim_cache size 0 max 16384 2024-04-02T14:48:27.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.762+0000 7f346fa12740 1 Processor -- start 2024-04-02T14:48:27.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.762+0000 7f346fa12740 1 -- start start 2024-04-02T14:48:27.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.763+0000 7f346fa12740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b5a04187b0 0x55b5a04214f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:27.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.763+0000 7f346fa12740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b5a04190d0 0x55b5a042d960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:27.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.763+0000 7f346fa12740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b5a0421cb0 0x55b5a041df20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:27.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.763+0000 7f346fa12740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f34681b3ed0 con 0x55b5a0421cb0 2024-04-02T14:48:27.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.763+0000 7f346fa12740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f34681b5a10 con 0x55b5a04187b0 2024-04-02T14:48:27.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.763+0000 7f346fa12740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f34681b45a0 con 0x55b5a04190d0 2024-04-02T14:48:27.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.764+0000 7f3465400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b5a04190d0 0x55b5a042d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:27.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.764+0000 7f3465400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b5a04190d0 0x55b5a042d960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:38424/0 (socket says 172.21.15.90:38424) 2024-04-02T14:48:27.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.764+0000 7f3465400640 1 -- 172.21.15.90:0/3511911515 learned_addr learned my addr 172.21.15.90:0/3511911515 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:48:27.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.765+0000 7f3466800640 1 --2- 172.21.15.90:0/3511911515 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b5a0421cb0 0x55b5a041df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:27.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.765+0000 7f3465400640 1 -- 172.21.15.90:0/3511911515 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b5a04187b0 msgr2=0x55b5a04214f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:48:27.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.765+0000 7f3465400640 1 --2- 172.21.15.90:0/3511911515 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b5a04187b0 0x55b5a04214f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:27.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.765+0000 7f3465400640 1 -- 172.21.15.90:0/3511911515 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b5a0421cb0 msgr2=0x55b5a041df20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:48:27.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.765+0000 7f3465400640 1 --2- 172.21.15.90:0/3511911515 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b5a0421cb0 0x55b5a041df20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:27.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.765+0000 7f3465400640 1 -- 172.21.15.90:0/3511911515 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b5a0424980 con 0x55b5a04190d0 2024-04-02T14:48:27.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.766+0000 7f3465400640 1 --2- 172.21.15.90:0/3511911515 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b5a04190d0 0x55b5a042d960 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f34540bde50 tx=0x7f345402cb50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1b51519c82450a69 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:48:27.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.766+0000 7f3452a00640 1 -- 172.21.15.90:0/3511911515 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34540da020 con 0x55b5a04190d0 2024-04-02T14:48:27.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.766+0000 7f3452a00640 1 -- 172.21.15.90:0/3511911515 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3454009530 con 0x55b5a04190d0 2024-04-02T14:48:27.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.766+0000 7f3452a00640 1 -- 172.21.15.90:0/3511911515 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34540bd240 con 0x55b5a04190d0 2024-04-02T14:48:27.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.767+0000 7f346fa12740 1 -- 172.21.15.90:0/3511911515 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b5a04190d0 msgr2=0x55b5a042d960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:48:27.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.767+0000 7f346fa12740 1 --2- 172.21.15.90:0/3511911515 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b5a04190d0 0x55b5a042d960 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f34540bde50 tx=0x7f345402cb50 comp rx=0 tx=0).stop 2024-04-02T14:48:27.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.768+0000 7f346fa12740 1 -- 172.21.15.90:0/3511911515 shutdown_connections 2024-04-02T14:48:27.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.768+0000 7f346fa12740 1 --2- 172.21.15.90:0/3511911515 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b5a0421cb0 0x55b5a041df20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:27.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.768+0000 7f346fa12740 1 --2- 172.21.15.90:0/3511911515 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b5a04190d0 0x55b5a042d960 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:27.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.768+0000 7f346fa12740 1 --2- 172.21.15.90:0/3511911515 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b5a04187b0 0x55b5a04214f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:27.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.768+0000 7f346fa12740 1 -- 172.21.15.90:0/3511911515 >> 172.21.15.90:0/3511911515 conn(0x55b5a0412ee0 msgr2=0x55b5a0414310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:48:27.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.768+0000 7f346fa12740 1 -- 172.21.15.90:0/3511911515 shutdown_connections 2024-04-02T14:48:27.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.768+0000 7f346fa12740 1 -- 172.21.15.90:0/3511911515 wait complete. 2024-04-02T14:48:27.764 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.768+0000 7f346fa12740 1 -- start start 2024-04-02T14:48:27.764 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.769+0000 7f346fa12740 1 Processor -- start 2024-04-02T14:48:27.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.769+0000 7f346fa12740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b5a04187b0 0x55b5a04257d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:27.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.769+0000 7f346fa12740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b5a04190d0 0x55b5a0425d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:27.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.769+0000 7f346fa12740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b5a0421cb0 0x55b5a0426250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:27.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.770+0000 7f346fa12740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f34681b5340 con 0x55b5a04190d0 2024-04-02T14:48:27.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.770+0000 7f346fa12740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f34681b4c70 con 0x55b5a0421cb0 2024-04-02T14:48:27.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.770+0000 7f346fa12740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f34681b1ed0 con 0x55b5a04187b0 2024-04-02T14:48:27.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.770+0000 7f346fa12740 10 client.0 _ll_register_callbacks cb 0x7f3454009880 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:48:27.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.770+0000 7f3465e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b5a04187b0 0x55b5a04257d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:27.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.770+0000 7f3465e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b5a04187b0 0x55b5a04257d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:38440/0 (socket says 172.21.15.90:38440) 2024-04-02T14:48:27.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.770+0000 7f3465e00640 1 -- 172.21.15.90:0/1170821628 learned_addr learned my addr 172.21.15.90:0/1170821628 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:48:27.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.770+0000 7f3465400640 1 --2- 172.21.15.90:0/1170821628 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b5a04190d0 0x55b5a0425d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:27.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.770+0000 7f3465e00640 1 -- 172.21.15.90:0/1170821628 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b5a0421cb0 msgr2=0x55b5a0426250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:48:27.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.770+0000 7f3465e00640 1 --2- 172.21.15.90:0/1170821628 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b5a0421cb0 0x55b5a0426250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:27.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.770+0000 7f3465e00640 1 -- 172.21.15.90:0/1170821628 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b5a04190d0 msgr2=0x55b5a0425d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:48:27.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.770+0000 7f3465e00640 1 --2- 172.21.15.90:0/1170821628 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b5a04190d0 0x55b5a0425d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:27.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.770+0000 7f3465e00640 1 -- 172.21.15.90:0/1170821628 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b5a0497310 con 0x55b5a04187b0 2024-04-02T14:48:27.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.771+0000 7f3447e00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:48:27.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.771+0000 7f3465e00640 1 --2- 172.21.15.90:0/1170821628 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b5a04187b0 0x55b5a04257d0 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f345800f9e0 tx=0x7f345800fa90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bb08f322a503fe29 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:48:27.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.772+0000 7f3447e00640 1 -- 172.21.15.90:0/1170821628 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3458004020 con 0x55b5a04187b0 2024-04-02T14:48:27.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.772+0000 7f3447e00640 1 -- 172.21.15.90:0/1170821628 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 8550+0+0 (secure 0 0 0) 0x7f3458018030 con 0x55b5a04187b0 2024-04-02T14:48:27.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.775+0000 7f346fa12740 1 -- 172.21.15.90:0/1170821628 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55b5a0389cb0 con 0x55b5a04187b0 2024-04-02T14:48:27.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.775+0000 7f3447e00640 1 -- 172.21.15.90:0/1170821628 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f3458004020 con 0x55b5a04187b0 2024-04-02T14:48:27.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.775+0000 7f346fa12740 10 client.29539 fetch_fsmap learned FSMap version 27 2024-04-02T14:48:27.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.775+0000 7f346fa12740 1 -- 172.21.15.90:0/1170821628 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55b5a04985f0 con 0x55b5a04187b0 2024-04-02T14:48:27.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.776+0000 7f3447e00640 1 -- 172.21.15.90:0/1170821628 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55b5a0389cb0 con 0x55b5a04187b0 2024-04-02T14:48:27.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.776+0000 7f3447e00640 1 -- 172.21.15.90:0/1170821628 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f34580035f0 con 0x55b5a04187b0 2024-04-02T14:48:27.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.776+0000 7f346fa12740 10 client.29539 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:48:27.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.776+0000 7f346fa12740 10 client.29539 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:48:27.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.776+0000 7f346fa12740 20 client.29539 populate_metadata read hostname 'smithi090' 2024-04-02T14:48:27.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.777+0000 7f346fa12740 1 --2- 172.21.15.90:0/1170821628 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55b5a0498da0 0x55b5a049b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:27.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.777+0000 7f346fa12740 4 client.29539 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-02T14:48:27.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.777+0000 7f346fa12740 1 -- 172.21.15.90:0/1170821628 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55b5a0384000 con 0x55b5a0498da0 2024-04-02T14:48:27.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.777+0000 7f3465400640 1 --2- 172.21.15.90:0/1170821628 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55b5a0498da0 0x55b5a049b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:27.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.780+0000 7f3465400640 1 --2- 172.21.15.90:0/1170821628 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55b5a0498da0 0x55b5a049b1c0 crc :-1 s=READY pgs=351 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d6a2da6a60db0ef0 server_cookie=a7cf81ac3159390a in_seq=0 out_seq=0 2024-04-02T14:48:27.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.780+0000 7f3447e00640 10 client.29539 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:48:27.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.780+0000 7f3447e00640 1 -- 172.21.15.90:0/1170821628 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55b5a0384000 con 0x55b5a0498da0 2024-04-02T14:48:27.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.780+0000 7f3447e00640 10 client.29539 handle_command_reply: tid=0 2024-04-02T14:48:27.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.780+0000 7f346fa12740 1 client.29539 shutdown 2024-04-02T14:48:27.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.780+0000 7f346fa12740 1 -- 172.21.15.90:0/1170821628 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b5a04187b0 msgr2=0x55b5a04257d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:48:27.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.780+0000 7f346fa12740 1 --2- 172.21.15.90:0/1170821628 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b5a04187b0 0x55b5a04257d0 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f345800f9e0 tx=0x7f345800fa90 comp rx=0 tx=0).stop 2024-04-02T14:48:27.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.781+0000 7f346fa12740 1 -- 172.21.15.90:0/1170821628 shutdown_connections 2024-04-02T14:48:27.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.781+0000 7f346fa12740 1 --2- 172.21.15.90:0/1170821628 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55b5a0498da0 0x55b5a049b1c0 crc :-1 s=READY pgs=351 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:27.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.781+0000 7f346fa12740 1 --2- 172.21.15.90:0/1170821628 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b5a0421cb0 0x55b5a0426250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:27.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.781+0000 7f3447e00640 0 client.29539 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:48:27.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.781+0000 7f346fa12740 1 --2- 172.21.15.90:0/1170821628 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b5a04190d0 0x55b5a0425d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:27.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.781+0000 7f346fa12740 1 --2- 172.21.15.90:0/1170821628 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b5a04187b0 0x55b5a04257d0 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:27.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.781+0000 7f346fa12740 1 -- 172.21.15.90:0/1170821628 >> 172.21.15.90:0/1170821628 conn(0x55b5a0412ee0 msgr2=0x55b5a04116d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:48:27.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.782+0000 7f346fa12740 1 -- 172.21.15.90:0/1170821628 shutdown_connections 2024-04-02T14:48:27.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.782+0000 7f346fa12740 1 -- 172.21.15.90:0/1170821628 wait complete. 2024-04-02T14:48:27.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.782+0000 7f346fa12740 20 client.29539 trim_cache size 0 max 16384 2024-04-02T14:48:27.778 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-02T14:48:27.778 INFO:teuthology.orchestra.run.smithi090.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-02T14:48:27.778 INFO:teuthology.orchestra.run.smithi090.stdout: "scrubs": { 2024-04-02T14:48:27.779 INFO:teuthology.orchestra.run.smithi090.stdout: "088a64b5-afdf-4d58-abe6-bff35505816b": { 2024-04-02T14:48:27.779 INFO:teuthology.orchestra.run.smithi090.stdout: "path": "/", 2024-04-02T14:48:27.779 INFO:teuthology.orchestra.run.smithi090.stdout: "tag": "088a64b5-afdf-4d58-abe6-bff35505816b", 2024-04-02T14:48:27.779 INFO:teuthology.orchestra.run.smithi090.stdout: "options": "recursive,force" 2024-04-02T14:48:27.779 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:48:27.779 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:48:27.779 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-02T14:48:27.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.785+0000 7f346ce00640 1 -- 172.21.15.90:0/2050718799 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f344c0a3e60 msgr2=0x7f344c0a6320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:48:27.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.785+0000 7f346ce00640 1 --2- 172.21.15.90:0/2050718799 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f344c0a3e60 0x7f344c0a6320 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x55b5a0418280 tx=0x7f3458005f90 comp rx=0 tx=0).stop 2024-04-02T14:48:27.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.785+0000 7f346ce00640 1 -- 172.21.15.90:0/2050718799 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f346810d3e0 msgr2=0x7f34681a4bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:48:27.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.785+0000 7f346ce00640 1 --2- 172.21.15.90:0/2050718799 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f346810d3e0 0x7f34681a4bd0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f3454009850 tx=0x7f345402ea40 comp rx=0 tx=0).stop 2024-04-02T14:48:27.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.785+0000 7f346ce00640 1 -- 172.21.15.90:0/2050718799 shutdown_connections 2024-04-02T14:48:27.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.785+0000 7f346ce00640 1 --2- 172.21.15.90:0/2050718799 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f344c0a3e60 0x7f344c0a6320 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:27.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.785+0000 7f346ce00640 1 --2- 172.21.15.90:0/2050718799 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f346819e800 0x7f346819ecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:27.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.785+0000 7f346ce00640 1 --2- 172.21.15.90:0/2050718799 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f346810d3e0 0x7f34681a4bd0 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:27.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.785+0000 7f346ce00640 1 --2- 172.21.15.90:0/2050718799 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3468108310 0x7f34681a4690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:27.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.785+0000 7f346ce00640 1 -- 172.21.15.90:0/2050718799 >> 172.21.15.90:0/2050718799 conn(0x7f3468076040 msgr2=0x7f34680783c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:48:27.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.787+0000 7f346ce00640 1 -- 172.21.15.90:0/2050718799 shutdown_connections 2024-04-02T14:48:27.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:27.787+0000 7f346ce00640 1 -- 172.21.15.90:0/2050718799 wait complete. 2024-04-02T14:48:27.818 INFO:tasks.cephfs.filesystem:scrub status for tag:088a64b5-afdf-4d58-abe6-bff35505816b - {'path': '/', 'tag': '088a64b5-afdf-4d58-abe6-bff35505816b', 'options': 'recursive,force'} 2024-04-02T14:48:28.603 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:48:28 smithi092 ceph-mon[36497]: pgmap v424: 97 pgs: 97 active+clean; 4.2 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 11 MiB/s wr, 631 op/s 2024-04-02T14:48:28.603 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:48:28 smithi092 ceph-mon[36497]: scrub queued for path: / 2024-04-02T14:48:28.603 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:48:28 smithi092 ceph-mon[36497]: scrub summary: idle+waiting paths [/] 2024-04-02T14:48:28.603 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:48:28 smithi092 ceph-mon[36497]: scrub summary: active paths [/] 2024-04-02T14:48:28.603 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:48:28 smithi092 ceph-mon[36497]: scrub summary: idle+waiting paths [/] 2024-04-02T14:48:28.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:48:28 smithi177 ceph-mon[36431]: pgmap v424: 97 pgs: 97 active+clean; 4.2 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 11 MiB/s wr, 631 op/s 2024-04-02T14:48:28.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:48:28 smithi177 ceph-mon[36431]: scrub queued for path: / 2024-04-02T14:48:28.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:48:28 smithi177 ceph-mon[36431]: scrub summary: idle+waiting paths [/] 2024-04-02T14:48:28.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:48:28 smithi177 ceph-mon[36431]: scrub summary: active paths [/] 2024-04-02T14:48:28.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:48:28 smithi177 ceph-mon[36431]: scrub summary: idle+waiting paths [/] 2024-04-02T14:48:28.681 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:48:28 smithi090 ceph-mon[31674]: pgmap v424: 97 pgs: 97 active+clean; 4.2 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 11 MiB/s wr, 631 op/s 2024-04-02T14:48:28.681 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:48:28 smithi090 ceph-mon[31674]: scrub queued for path: / 2024-04-02T14:48:28.681 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:48:28 smithi090 ceph-mon[31674]: scrub summary: idle+waiting paths [/] 2024-04-02T14:48:28.681 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:48:28 smithi090 ceph-mon[31674]: scrub summary: active paths [/] 2024-04-02T14:48:28.681 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:48:28 smithi090 ceph-mon[31674]: scrub summary: idle+waiting paths [/] 2024-04-02T14:48:29.603 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:48:29 smithi092 ceph-mon[36497]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:48:29.603 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:48:29 smithi092 ceph-mon[36497]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T14:48:29.603 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:48:29 smithi092 ceph-mon[36497]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' 2024-04-02T14:48:29.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:48:29 smithi177 ceph-mon[36431]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:48:29.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:48:29 smithi177 ceph-mon[36431]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T14:48:29.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:48:29 smithi177 ceph-mon[36431]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' 2024-04-02T14:48:29.681 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:48:29 smithi090 ceph-mon[31674]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:48:29.681 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:48:29 smithi090 ceph-mon[31674]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T14:48:29.681 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:48:29 smithi090 ceph-mon[31674]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' 2024-04-02T14:48:30.603 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:48:30 smithi092 ceph-mon[36497]: pgmap v425: 97 pgs: 97 active+clean; 4.2 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 11 MiB/s wr, 626 op/s 2024-04-02T14:48:30.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:48:30 smithi177 ceph-mon[36431]: pgmap v425: 97 pgs: 97 active+clean; 4.2 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 11 MiB/s wr, 626 op/s 2024-04-02T14:48:30.680 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:48:30 smithi090 ceph-mon[31674]: pgmap v425: 97 pgs: 97 active+clean; 4.2 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 11 MiB/s wr, 626 op/s 2024-04-02T14:48:31.824 INFO:teuthology.orchestra.run.smithi090.stderr:progress: 780.0 s, 1411.8 tps, lat 70.747 ms stddev 86.181 2024-04-02T14:48:32.238 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.243+0000 7f3076400640 1 -- 172.21.15.90:0/1913297396 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3070109420 msgr2=0x7f3070114710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:48:32.238 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.243+0000 7f3076400640 1 --2- 172.21.15.90:0/1913297396 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3070109420 0x7f3070114710 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f3068009cd0 tx=0x7f306802c0e0 comp rx=0 tx=0).stop 2024-04-02T14:48:32.239 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.243+0000 7f3076400640 1 -- 172.21.15.90:0/1913297396 shutdown_connections 2024-04-02T14:48:32.239 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.243+0000 7f3076400640 1 --2- 172.21.15.90:0/1913297396 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3070114c50 0x7f3070117040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:32.239 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.243+0000 7f3076400640 1 --2- 172.21.15.90:0/1913297396 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3070109420 0x7f3070114710 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:32.239 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.243+0000 7f3076400640 1 --2- 172.21.15.90:0/1913297396 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3070108b00 0x7f3070108ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:32.239 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.244+0000 7f3076400640 1 -- 172.21.15.90:0/1913297396 >> 172.21.15.90:0/1913297396 conn(0x7f307006d030 msgr2=0x7f307006d440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:48:32.239 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.244+0000 7f3076400640 1 -- 172.21.15.90:0/1913297396 shutdown_connections 2024-04-02T14:48:32.239 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.245+0000 7f3076400640 1 -- 172.21.15.90:0/1913297396 wait complete. 2024-04-02T14:48:32.240 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.246+0000 7f3076400640 1 Processor -- start 2024-04-02T14:48:32.241 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.246+0000 7f3076400640 1 -- start start 2024-04-02T14:48:32.241 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.247+0000 7f3076400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3070108b00 0x7f30701140a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:32.241 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.247+0000 7f3076400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3070109420 0x7f30701145e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:32.241 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.247+0000 7f3076400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3070114c50 0x7f307010b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:32.241 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.247+0000 7f3076400640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f307011a720 con 0x7f3070108b00 2024-04-02T14:48:32.241 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.247+0000 7f3076400640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f307011a5a0 con 0x7f3070114c50 2024-04-02T14:48:32.242 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.247+0000 7f3076400640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f307011a8a0 con 0x7f3070109420 2024-04-02T14:48:32.242 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.247+0000 7f306ea00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3070109420 0x7f30701145e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:32.242 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.247+0000 7f306ea00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3070109420 0x7f30701145e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:38458/0 (socket says 172.21.15.90:38458) 2024-04-02T14:48:32.243 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.247+0000 7f306ea00640 1 -- 172.21.15.90:0/2197538459 learned_addr learned my addr 172.21.15.90:0/2197538459 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:48:32.243 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.247+0000 7f306fe00640 1 --2- 172.21.15.90:0/2197538459 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3070114c50 0x7f307010b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:32.243 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.248+0000 7f306ea00640 1 -- 172.21.15.90:0/2197538459 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3070114c50 msgr2=0x7f307010b910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:48:32.243 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.248+0000 7f306ea00640 1 --2- 172.21.15.90:0/2197538459 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3070114c50 0x7f307010b910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:32.243 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.248+0000 7f306ea00640 1 -- 172.21.15.90:0/2197538459 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3070108b00 msgr2=0x7f30701140a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:48:32.243 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.248+0000 7f306ea00640 1 --2- 172.21.15.90:0/2197538459 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3070108b00 0x7f30701140a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:32.243 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.248+0000 7f306ea00640 1 -- 172.21.15.90:0/2197538459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f307010c1d0 con 0x7f3070109420 2024-04-02T14:48:32.243 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.248+0000 7f306ea00640 1 --2- 172.21.15.90:0/2197538459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3070109420 0x7f30701145e0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f3068037040 tx=0x7f3068035d10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:48:32.244 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.249+0000 7f3067e00640 1 -- 172.21.15.90:0/2197538459 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3068039040 con 0x7f3070109420 2024-04-02T14:48:32.244 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.250+0000 7f3076400640 1 -- 172.21.15.90:0/2197538459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f307010c460 con 0x7f3070109420 2024-04-02T14:48:32.244 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.250+0000 7f3076400640 1 -- 172.21.15.90:0/2197538459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f307010fdd0 con 0x7f3070109420 2024-04-02T14:48:32.244 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.249+0000 7f3067e00640 1 -- 172.21.15.90:0/2197538459 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3068044070 con 0x7f3070109420 2024-04-02T14:48:32.244 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.250+0000 7f3067e00640 1 -- 172.21.15.90:0/2197538459 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3068004600 con 0x7f3070109420 2024-04-02T14:48:32.248 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.254+0000 7f307917c740 1 Processor -- start 2024-04-02T14:48:32.250 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.255+0000 7f307917c740 1 -- start start 2024-04-02T14:48:32.251 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.256+0000 7f307917c740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55adcc393eb0 0x55adcc394290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:32.251 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.256+0000 7f307917c740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55adcc3947d0 0x55adcc39ce70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:32.251 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.256+0000 7f307917c740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55adcc39d3b0 0x55adcc39f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:32.251 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.256+0000 7f307917c740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55adcc2f8d70 con 0x55adcc393eb0 2024-04-02T14:48:32.251 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.256+0000 7f307917c740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55adcc2a12a0 con 0x55adcc39d3b0 2024-04-02T14:48:32.251 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.256+0000 7f307917c740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55adcc2f4d20 con 0x55adcc3947d0 2024-04-02T14:48:32.256 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.256+0000 7f3067e00640 1 -- 172.21.15.90:0/2197538459 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f3068049460 con 0x7f3070109420 2024-04-02T14:48:32.256 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.258+0000 7f306ea00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55adcc39d3b0 0x55adcc39f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:32.256 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.258+0000 7f306ea00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55adcc39d3b0 0x55adcc39f7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:54548/0 (socket says 172.21.15.90:54548) 2024-04-02T14:48:32.256 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.258+0000 7f306ea00640 1 -- 172.21.15.90:0/3070743857 learned_addr learned my addr 172.21.15.90:0/3070743857 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:48:32.257 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.259+0000 7f306ea00640 1 -- 172.21.15.90:0/3070743857 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55adcc3947d0 msgr2=0x55adcc39ce70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-02T14:48:32.257 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.259+0000 7f306ea00640 1 --2- 172.21.15.90:0/3070743857 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55adcc3947d0 0x55adcc39ce70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:32.257 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.259+0000 7f306ea00640 1 -- 172.21.15.90:0/3070743857 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55adcc393eb0 msgr2=0x55adcc394290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:48:32.257 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.259+0000 7f306ea00640 1 --2- 172.21.15.90:0/3070743857 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55adcc393eb0 0x55adcc394290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:32.257 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.259+0000 7f306ea00640 1 -- 172.21.15.90:0/3070743857 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55adcc2b5840 con 0x55adcc39d3b0 2024-04-02T14:48:32.257 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.259+0000 7f306ea00640 1 --2- 172.21.15.90:0/3070743857 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55adcc39d3b0 0x55adcc39f7a0 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f30680f04b0 tx=0x7f30680f6db0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=451321eb41e45c61 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:48:32.257 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.260+0000 7f3064c00640 1 -- 172.21.15.90:0/3070743857 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3068102070 con 0x55adcc39d3b0 2024-04-02T14:48:32.257 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.260+0000 7f3064c00640 1 -- 172.21.15.90:0/3070743857 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f30680f9480 con 0x55adcc39d3b0 2024-04-02T14:48:32.257 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.260+0000 7f3064c00640 1 -- 172.21.15.90:0/3070743857 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30680f8640 con 0x55adcc39d3b0 2024-04-02T14:48:32.257 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.261+0000 7f3067e00640 1 --2- 172.21.15.90:0/2197538459 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f30540a3f10 0x7f30540a63d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:32.257 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.261+0000 7f3067e00640 1 -- 172.21.15.90:0/2197538459 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 8550+0+0 (secure 0 0 0) 0x7f30680f2d90 con 0x7f3070109420 2024-04-02T14:48:32.257 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.261+0000 7f307917c740 1 -- 172.21.15.90:0/3070743857 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55adcc39d3b0 msgr2=0x55adcc39f7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:48:32.257 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.261+0000 7f307917c740 1 --2- 172.21.15.90:0/3070743857 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55adcc39d3b0 0x55adcc39f7a0 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f30680f04b0 tx=0x7f30680f6db0 comp rx=0 tx=0).stop 2024-04-02T14:48:32.258 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.262+0000 7f306fe00640 1 --2- 172.21.15.90:0/2197538459 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f30540a3f10 0x7f30540a63d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:32.259 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.263+0000 7f307917c740 1 -- 172.21.15.90:0/3070743857 shutdown_connections 2024-04-02T14:48:32.259 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.263+0000 7f307917c740 1 --2- 172.21.15.90:0/3070743857 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55adcc39d3b0 0x55adcc39f7a0 unknown :-1 s=CLOSED pgs=943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:32.259 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.263+0000 7f307917c740 1 --2- 172.21.15.90:0/3070743857 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55adcc3947d0 0x55adcc39ce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:32.259 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.263+0000 7f307917c740 1 --2- 172.21.15.90:0/3070743857 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55adcc393eb0 0x55adcc394290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:32.259 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.263+0000 7f307917c740 1 -- 172.21.15.90:0/3070743857 >> 172.21.15.90:0/3070743857 conn(0x55adcc38e5e0 msgr2=0x55adcc38e9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:48:32.259 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.264+0000 7f306fe00640 1 --2- 172.21.15.90:0/2197538459 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f30540a3f10 0x7f30540a63d0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f306000b830 tx=0x7f3060008040 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:48:32.259 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.264+0000 7f307917c740 1 -- 172.21.15.90:0/3070743857 shutdown_connections 2024-04-02T14:48:32.259 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.264+0000 7f307917c740 1 -- 172.21.15.90:0/3070743857 wait complete. 2024-04-02T14:48:32.260 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.265+0000 7f307917c740 1 -- start start 2024-04-02T14:48:32.260 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.265+0000 7f307917c740 1 Processor -- start 2024-04-02T14:48:32.261 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.266+0000 7f307917c740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55adcc393eb0 0x55adcc39a4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:32.261 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.266+0000 7f307917c740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55adcc3947d0 0x55adcc39aa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:32.261 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.266+0000 7f307917c740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55adcc39d3b0 0x55adcc39af50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:32.261 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.266+0000 7f307917c740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55adcc39ff60 con 0x55adcc3947d0 2024-04-02T14:48:32.261 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.266+0000 7f307917c740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55adcc2b4660 con 0x55adcc393eb0 2024-04-02T14:48:32.261 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.266+0000 7f307917c740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55adcc3a00e0 con 0x55adcc39d3b0 2024-04-02T14:48:32.261 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.266+0000 7f307917c740 10 client.0 _ll_register_callbacks cb 0x55adcc2b4050 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:48:32.261 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.266+0000 7f306f400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55adcc393eb0 0x55adcc39a4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:32.262 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.267+0000 7f306f400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55adcc393eb0 0x55adcc39a4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:54564/0 (socket says 172.21.15.90:54564) 2024-04-02T14:48:32.262 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.267+0000 7f306f400640 1 -- 172.21.15.90:0/386736194 learned_addr learned my addr 172.21.15.90:0/386736194 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:48:32.262 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.267+0000 7f306fe00640 1 --2- 172.21.15.90:0/386736194 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55adcc39d3b0 0x55adcc39af50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:32.262 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.267+0000 7f306ea00640 1 --2- 172.21.15.90:0/386736194 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55adcc3947d0 0x55adcc39aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:32.263 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.268+0000 7f306fe00640 1 -- 172.21.15.90:0/386736194 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55adcc393eb0 msgr2=0x55adcc39a4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:48:32.263 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.268+0000 7f306fe00640 1 --2- 172.21.15.90:0/386736194 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55adcc393eb0 0x55adcc39a4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:32.263 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.268+0000 7f306fe00640 1 -- 172.21.15.90:0/386736194 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55adcc3947d0 msgr2=0x55adcc39aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:48:32.263 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.268+0000 7f306fe00640 1 --2- 172.21.15.90:0/386736194 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55adcc3947d0 0x55adcc39aa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:32.263 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.268+0000 7f306fe00640 1 -- 172.21.15.90:0/386736194 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55adcc2c3e80 con 0x55adcc39d3b0 2024-04-02T14:48:32.264 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.268+0000 7f306f400640 1 --2- 172.21.15.90:0/386736194 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55adcc393eb0 0x55adcc39a4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:48:32.264 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.268+0000 7f306fe00640 1 --2- 172.21.15.90:0/386736194 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55adcc39d3b0 0x55adcc39af50 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f306000d530 tx=0x7f3060006a30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1da2d9f8e69dcd07 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:48:32.264 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.268+0000 7f3059600640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:48:32.264 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.270+0000 7f3059600640 1 -- 172.21.15.90:0/386736194 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3060024060 con 0x55adcc39d3b0 2024-04-02T14:48:32.265 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.270+0000 7f3059600640 1 -- 172.21.15.90:0/386736194 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 8550+0+0 (secure 0 0 0) 0x7f306000f040 con 0x55adcc39d3b0 2024-04-02T14:48:32.265 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.270+0000 7f307917c740 1 -- 172.21.15.90:0/386736194 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f30680f9480 con 0x55adcc39d3b0 2024-04-02T14:48:32.266 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.271+0000 7f3059600640 1 -- 172.21.15.90:0/386736194 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f3060024060 con 0x55adcc39d3b0 2024-04-02T14:48:32.267 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.272+0000 7f307917c740 10 client.29548 fetch_fsmap learned FSMap version 27 2024-04-02T14:48:32.267 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.272+0000 7f307917c740 1 -- 172.21.15.90:0/386736194 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55adcc00d210 con 0x55adcc39d3b0 2024-04-02T14:48:32.267 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.272+0000 7f3059600640 1 -- 172.21.15.90:0/386736194 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f30680f9480 con 0x55adcc39d3b0 2024-04-02T14:48:32.267 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.273+0000 7f3059600640 1 -- 172.21.15.90:0/386736194 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f3060013050 con 0x55adcc39d3b0 2024-04-02T14:48:32.268 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.273+0000 7f307917c740 10 client.29548 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:48:32.268 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.273+0000 7f307917c740 10 client.29548 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-02T14:48:32.268 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.273+0000 7f307917c740 20 client.29548 populate_metadata read hostname 'smithi090' 2024-04-02T14:48:32.268 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.273+0000 7f307917c740 1 --2- 172.21.15.90:0/386736194 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55adcc3ad660 0x55adcc3afa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:32.268 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.273+0000 7f307917c740 4 client.29548 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:48:32.268 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.273+0000 7f307917c740 1 -- 172.21.15.90:0/386736194 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55adcc2a8cd0 con 0x55adcc3ad660 2024-04-02T14:48:32.268 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.273+0000 7f306f400640 1 --2- 172.21.15.90:0/386736194 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55adcc3ad660 0x55adcc3afa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:32.268 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.274+0000 7f306f400640 1 --2- 172.21.15.90:0/386736194 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55adcc3ad660 0x55adcc3afa70 crc :-1 s=READY pgs=352 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=635ad405b2c1c83f server_cookie=b92c16459626cbb8 in_seq=0 out_seq=0 2024-04-02T14:48:32.269 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.274+0000 7f3059600640 10 client.29548 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:48:32.270 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.275+0000 7f3059600640 1 -- 172.21.15.90:0/386736194 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55adcc2a8cd0 con 0x55adcc3ad660 2024-04-02T14:48:32.270 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.275+0000 7f3059600640 10 client.29548 handle_command_reply: tid=0 2024-04-02T14:48:32.272 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.277+0000 7f307917c740 1 client.29548 shutdown 2024-04-02T14:48:32.272 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.277+0000 7f307917c740 1 -- 172.21.15.90:0/386736194 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55adcc39d3b0 msgr2=0x55adcc39af50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:48:32.272 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.277+0000 7f307917c740 1 --2- 172.21.15.90:0/386736194 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55adcc39d3b0 0x55adcc39af50 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f306000d530 tx=0x7f3060006a30 comp rx=0 tx=0).stop 2024-04-02T14:48:32.272 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.277+0000 7f307917c740 1 -- 172.21.15.90:0/386736194 shutdown_connections 2024-04-02T14:48:32.272 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.277+0000 7f307917c740 1 --2- 172.21.15.90:0/386736194 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55adcc3ad660 0x55adcc3afa70 crc :-1 s=READY pgs=352 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:32.272 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.277+0000 7f307917c740 1 --2- 172.21.15.90:0/386736194 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55adcc39d3b0 0x55adcc39af50 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:32.272 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.277+0000 7f307917c740 1 --2- 172.21.15.90:0/386736194 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55adcc3947d0 0x55adcc39aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:32.272 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.277+0000 7f3059600640 0 client.29548 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:48:32.272 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.277+0000 7f307917c740 1 --2- 172.21.15.90:0/386736194 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55adcc393eb0 0x55adcc39a4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:32.272 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.277+0000 7f307917c740 1 -- 172.21.15.90:0/386736194 >> 172.21.15.90:0/386736194 conn(0x55adcc38e5e0 msgr2=0x55adcc390330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:48:32.273 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.278+0000 7f307917c740 1 -- 172.21.15.90:0/386736194 shutdown_connections 2024-04-02T14:48:32.273 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.278+0000 7f307917c740 1 -- 172.21.15.90:0/386736194 wait complete. 2024-04-02T14:48:32.273 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.278+0000 7f307917c740 20 client.29548 trim_cache size 0 max 16384 2024-04-02T14:48:32.297 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.302+0000 7f307917c740 1 Processor -- start 2024-04-02T14:48:32.297 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.302+0000 7f307917c740 1 -- start start 2024-04-02T14:48:32.298 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.302+0000 7f307917c740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55adcc393eb0 0x55adcc39cc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:32.298 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.302+0000 7f307917c740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55adcc3947d0 0x55adcc3a8e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:32.298 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.303+0000 7f307917c740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55adcc39d3b0 0x55adcc38d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:32.298 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.303+0000 7f307917c740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f305000e820 con 0x55adcc39d3b0 2024-04-02T14:48:32.298 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.303+0000 7f307917c740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f3050010360 con 0x55adcc3947d0 2024-04-02T14:48:32.298 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.303+0000 7f307917c740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f305000eef0 con 0x55adcc393eb0 2024-04-02T14:48:32.298 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.303+0000 7f306f400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55adcc393eb0 0x55adcc39cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:32.298 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.303+0000 7f306f400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55adcc393eb0 0x55adcc39cc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:38464/0 (socket says 172.21.15.90:38464) 2024-04-02T14:48:32.298 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.303+0000 7f306f400640 1 -- 172.21.15.90:0/1489976269 learned_addr learned my addr 172.21.15.90:0/1489976269 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:48:32.298 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.303+0000 7f306fe00640 1 --2- 172.21.15.90:0/1489976269 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55adcc39d3b0 0x55adcc38d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:32.298 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.303+0000 7f306ea00640 1 --2- 172.21.15.90:0/1489976269 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55adcc3947d0 0x55adcc3a8e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:32.298 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.303+0000 7f306f400640 1 -- 172.21.15.90:0/1489976269 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55adcc3947d0 msgr2=0x55adcc3a8e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:48:32.299 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.303+0000 7f306f400640 1 --2- 172.21.15.90:0/1489976269 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55adcc3947d0 0x55adcc3a8e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:32.299 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.303+0000 7f306f400640 1 -- 172.21.15.90:0/1489976269 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55adcc39d3b0 msgr2=0x55adcc38d4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:48:32.299 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.303+0000 7f306f400640 1 --2- 172.21.15.90:0/1489976269 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55adcc39d3b0 0x55adcc38d4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:32.299 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.303+0000 7f306f400640 1 -- 172.21.15.90:0/1489976269 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55adcc0d9620 con 0x55adcc393eb0 2024-04-02T14:48:32.300 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.303+0000 7f306f400640 1 --2- 172.21.15.90:0/1489976269 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55adcc393eb0 0x55adcc39cc00 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x55adcc3b0370 tx=0x7f305c005dc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2575a6b488513766 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:48:32.300 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.304+0000 7f305be00640 1 -- 172.21.15.90:0/1489976269 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f305c004020 con 0x55adcc393eb0 2024-04-02T14:48:32.301 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.304+0000 7f305be00640 1 -- 172.21.15.90:0/1489976269 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f305c01a8c0 con 0x55adcc393eb0 2024-04-02T14:48:32.301 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.304+0000 7f305be00640 1 -- 172.21.15.90:0/1489976269 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f305c003c30 con 0x55adcc393eb0 2024-04-02T14:48:32.301 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.304+0000 7f307917c740 1 -- 172.21.15.90:0/1489976269 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55adcc393eb0 msgr2=0x55adcc39cc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:48:32.301 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.304+0000 7f307917c740 1 --2- 172.21.15.90:0/1489976269 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55adcc393eb0 0x55adcc39cc00 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x55adcc3b0370 tx=0x7f305c005dc0 comp rx=0 tx=0).stop 2024-04-02T14:48:32.301 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.304+0000 7f307917c740 1 -- 172.21.15.90:0/1489976269 shutdown_connections 2024-04-02T14:48:32.301 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.304+0000 7f307917c740 1 --2- 172.21.15.90:0/1489976269 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55adcc39d3b0 0x55adcc38d4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:32.301 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.304+0000 7f307917c740 1 --2- 172.21.15.90:0/1489976269 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55adcc3947d0 0x55adcc3a8e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:32.301 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.304+0000 7f307917c740 1 --2- 172.21.15.90:0/1489976269 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55adcc393eb0 0x55adcc39cc00 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:32.301 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.304+0000 7f307917c740 1 -- 172.21.15.90:0/1489976269 >> 172.21.15.90:0/1489976269 conn(0x55adcc38e5e0 msgr2=0x55adcc38cdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:48:32.301 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.304+0000 7f307917c740 1 -- 172.21.15.90:0/1489976269 shutdown_connections 2024-04-02T14:48:32.301 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.305+0000 7f307917c740 1 -- 172.21.15.90:0/1489976269 wait complete. 2024-04-02T14:48:32.301 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.305+0000 7f307917c740 1 -- start start 2024-04-02T14:48:32.301 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.305+0000 7f307917c740 1 Processor -- start 2024-04-02T14:48:32.301 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.306+0000 7f307917c740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55adcc393eb0 0x55adcc3a1000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:32.302 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.306+0000 7f307917c740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55adcc3947d0 0x55adcc3a1540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:32.302 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.306+0000 7f307917c740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55adcc39d3b0 0x55adcc3a1aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:32.302 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.306+0000 7f307917c740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f305000fc90 con 0x55adcc3947d0 2024-04-02T14:48:32.302 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.306+0000 7f307917c740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f305000f5c0 con 0x55adcc39d3b0 2024-04-02T14:48:32.302 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.306+0000 7f307917c740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f305000c820 con 0x55adcc393eb0 2024-04-02T14:48:32.302 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.306+0000 7f307917c740 10 client.0 _ll_register_callbacks cb 0x7f306000da10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:48:32.304 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.306+0000 7f306fe00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55adcc39d3b0 0x55adcc3a1aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:32.304 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.306+0000 7f306fe00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55adcc39d3b0 0x55adcc3a1aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:54576/0 (socket says 172.21.15.90:54576) 2024-04-02T14:48:32.304 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.306+0000 7f306fe00640 1 -- 172.21.15.90:0/3110023804 learned_addr learned my addr 172.21.15.90:0/3110023804 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:48:32.304 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.307+0000 7f306f400640 1 --2- 172.21.15.90:0/3110023804 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55adcc393eb0 0x55adcc3a1000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:32.305 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.307+0000 7f306ea00640 1 --2- 172.21.15.90:0/3110023804 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55adcc3947d0 0x55adcc3a1540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:32.305 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.307+0000 7f306fe00640 1 -- 172.21.15.90:0/3110023804 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55adcc393eb0 msgr2=0x55adcc3a1000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:48:32.305 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.307+0000 7f306fe00640 1 --2- 172.21.15.90:0/3110023804 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55adcc393eb0 0x55adcc3a1000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:32.305 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.307+0000 7f306fe00640 1 -- 172.21.15.90:0/3110023804 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55adcc3947d0 msgr2=0x55adcc3a1540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:48:32.305 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.307+0000 7f306fe00640 1 --2- 172.21.15.90:0/3110023804 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55adcc3947d0 0x55adcc3a1540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:32.305 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.307+0000 7f306fe00640 1 -- 172.21.15.90:0/3110023804 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55adcc412f60 con 0x55adcc39d3b0 2024-04-02T14:48:32.305 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.307+0000 7f306f400640 1 --2- 172.21.15.90:0/3110023804 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55adcc393eb0 0x55adcc3a1000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:48:32.305 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.307+0000 7f306fe00640 1 --2- 172.21.15.90:0/3110023804 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55adcc39d3b0 0x55adcc3a1aa0 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f3060004700 tx=0x7f3060014320 comp rx=0 tx=0).ready entity=mon.2 client_cookie=349a135a0c715f45 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:48:32.305 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.307+0000 7f3059600640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:48:32.305 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.308+0000 7f3059600640 1 -- 172.21.15.90:0/3110023804 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f306002e080 con 0x55adcc39d3b0 2024-04-02T14:48:32.305 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.308+0000 7f307917c740 1 -- 172.21.15.90:0/3110023804 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f305c01a8c0 con 0x55adcc39d3b0 2024-04-02T14:48:32.305 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.308+0000 7f3059600640 1 -- 172.21.15.90:0/3110023804 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 8550+0+0 (secure 0 0 0) 0x7f3060005bd0 con 0x55adcc39d3b0 2024-04-02T14:48:32.305 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.308+0000 7f3059600640 1 -- 172.21.15.90:0/3110023804 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f3060005dd0 con 0x55adcc39d3b0 2024-04-02T14:48:32.305 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.308+0000 7f307917c740 10 client.29759 fetch_fsmap learned FSMap version 27 2024-04-02T14:48:32.306 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.308+0000 7f307917c740 1 -- 172.21.15.90:0/3110023804 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55adcc414210 con 0x55adcc39d3b0 2024-04-02T14:48:32.306 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.308+0000 7f3059600640 1 -- 172.21.15.90:0/3110023804 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f305c01a8c0 con 0x55adcc39d3b0 2024-04-02T14:48:32.306 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.308+0000 7f3059600640 1 -- 172.21.15.90:0/3110023804 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f3060014ba0 con 0x55adcc39d3b0 2024-04-02T14:48:32.306 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.308+0000 7f307917c740 10 client.29759 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:48:32.306 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.308+0000 7f307917c740 10 client.29759 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-02T14:48:32.306 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.308+0000 7f307917c740 20 client.29759 populate_metadata read hostname 'smithi090' 2024-04-02T14:48:32.306 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.308+0000 7f307917c740 1 --2- 172.21.15.90:0/3110023804 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55adcc414990 0x55adcc416db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:32.306 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.309+0000 7f307917c740 4 client.29759 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-02T14:48:32.306 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.309+0000 7f307917c740 1 -- 172.21.15.90:0/3110023804 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55adcc3062b0 con 0x55adcc414990 2024-04-02T14:48:32.306 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.309+0000 7f306f400640 1 --2- 172.21.15.90:0/3110023804 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55adcc414990 0x55adcc416db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:32.306 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.309+0000 7f306f400640 1 --2- 172.21.15.90:0/3110023804 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55adcc414990 0x55adcc416db0 crc :-1 s=READY pgs=353 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a4c8bfa6dc1c4fd8 server_cookie=9fcd8e15e122c3d2 in_seq=0 out_seq=0 2024-04-02T14:48:32.306 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.309+0000 7f3059600640 10 client.29759 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:48:32.339 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.345+0000 7f3059600640 1 -- 172.21.15.90:0/3110023804 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55adcc3062b0 con 0x55adcc414990 2024-04-02T14:48:32.340 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.345+0000 7f3059600640 10 client.29759 handle_command_reply: tid=0 2024-04-02T14:48:32.340 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.345+0000 7f307917c740 1 client.29759 shutdown 2024-04-02T14:48:32.340 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.345+0000 7f307917c740 1 -- 172.21.15.90:0/3110023804 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55adcc39d3b0 msgr2=0x55adcc3a1aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:48:32.340 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.345+0000 7f307917c740 1 --2- 172.21.15.90:0/3110023804 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55adcc39d3b0 0x55adcc3a1aa0 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f3060004700 tx=0x7f3060014320 comp rx=0 tx=0).stop 2024-04-02T14:48:32.340 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.345+0000 7f307917c740 1 -- 172.21.15.90:0/3110023804 shutdown_connections 2024-04-02T14:48:32.340 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.345+0000 7f307917c740 1 --2- 172.21.15.90:0/3110023804 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55adcc414990 0x55adcc416db0 crc :-1 s=READY pgs=353 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:32.340 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.345+0000 7f307917c740 1 --2- 172.21.15.90:0/3110023804 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55adcc39d3b0 0x55adcc3a1aa0 unknown :-1 s=CLOSED pgs=944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:32.340 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.345+0000 7f307917c740 1 --2- 172.21.15.90:0/3110023804 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55adcc3947d0 0x55adcc3a1540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:32.340 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.345+0000 7f307917c740 1 --2- 172.21.15.90:0/3110023804 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55adcc393eb0 0x55adcc3a1000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:32.340 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.345+0000 7f307917c740 1 -- 172.21.15.90:0/3110023804 >> 172.21.15.90:0/3110023804 conn(0x55adcc38e5e0 msgr2=0x55adcc38d640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:48:32.340 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.345+0000 7f3059600640 0 client.29759 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:48:32.340 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.345+0000 7f307917c740 1 -- 172.21.15.90:0/3110023804 shutdown_connections 2024-04-02T14:48:32.341 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.345+0000 7f307917c740 1 -- 172.21.15.90:0/3110023804 wait complete. 2024-04-02T14:48:32.341 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.346+0000 7f307917c740 20 client.29759 trim_cache size 0 max 16384 2024-04-02T14:48:32.341 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:{ 2024-04-02T14:48:32.341 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "message": "", 2024-04-02T14:48:32.341 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "return_code": 0 2024-04-02T14:48:32.342 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:} 2024-04-02T14:48:32.342 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.348+0000 7f3076400640 1 -- 172.21.15.90:0/2197538459 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f30540a3f10 msgr2=0x7f30540a63d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:48:32.342 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.348+0000 7f3076400640 1 --2- 172.21.15.90:0/2197538459 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f30540a3f10 0x7f30540a63d0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f306000b830 tx=0x7f3060008040 comp rx=0 tx=0).stop 2024-04-02T14:48:32.342 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.348+0000 7f3076400640 1 -- 172.21.15.90:0/2197538459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3070109420 msgr2=0x7f30701145e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:48:32.343 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.348+0000 7f3076400640 1 --2- 172.21.15.90:0/2197538459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3070109420 0x7f30701145e0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f3068037040 tx=0x7f3068035d10 comp rx=0 tx=0).stop 2024-04-02T14:48:32.343 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.348+0000 7f3076400640 1 -- 172.21.15.90:0/2197538459 shutdown_connections 2024-04-02T14:48:32.343 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.348+0000 7f3076400640 1 --2- 172.21.15.90:0/2197538459 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f30540a3f10 0x7f30540a63d0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:32.343 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.348+0000 7f3076400640 1 --2- 172.21.15.90:0/2197538459 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3070114c50 0x7f307010b910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:32.343 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.348+0000 7f3076400640 1 --2- 172.21.15.90:0/2197538459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3070109420 0x7f30701145e0 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:32.343 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.348+0000 7f3076400640 1 --2- 172.21.15.90:0/2197538459 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3070108b00 0x7f30701140a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:32.343 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.348+0000 7f3076400640 1 -- 172.21.15.90:0/2197538459 >> 172.21.15.90:0/2197538459 conn(0x7f307006d030 msgr2=0x7f307010a4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:48:32.343 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.348+0000 7f3076400640 1 -- 172.21.15.90:0/2197538459 shutdown_connections 2024-04-02T14:48:32.343 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:32.348+0000 7f3076400640 1 -- 172.21.15.90:0/2197538459 wait complete. 2024-04-02T14:48:32.481 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:48:32 smithi177 ceph-mon[36431]: pgmap v426: 97 pgs: 97 active+clean; 4.3 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 160 KiB/s rd, 10 MiB/s wr, 770 op/s 2024-04-02T14:48:32.481 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:48:32 smithi177 ceph-mon[36431]: scrub summary: idle 2024-04-02T14:48:32.603 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:48:32 smithi092 ceph-mon[36497]: pgmap v426: 97 pgs: 97 active+clean; 4.3 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 160 KiB/s rd, 10 MiB/s wr, 770 op/s 2024-04-02T14:48:32.603 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:48:32 smithi092 ceph-mon[36497]: scrub summary: idle 2024-04-02T14:48:32.681 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:48:32 smithi090 ceph-mon[31674]: pgmap v426: 97 pgs: 97 active+clean; 4.3 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 160 KiB/s rd, 10 MiB/s wr, 770 op/s 2024-04-02T14:48:32.681 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:48:32 smithi090 ceph-mon[31674]: scrub summary: idle 2024-04-02T14:48:34.603 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:48:34 smithi092 ceph-mon[36497]: pgmap v427: 97 pgs: 97 active+clean; 4.3 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 8.8 MiB/s wr, 673 op/s 2024-04-02T14:48:34.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:48:34 smithi177 ceph-mon[36431]: pgmap v427: 97 pgs: 97 active+clean; 4.3 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 8.8 MiB/s wr, 673 op/s 2024-04-02T14:48:34.680 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:48:34 smithi090 ceph-mon[31674]: pgmap v427: 97 pgs: 97 active+clean; 4.3 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 8.8 MiB/s wr, 673 op/s 2024-04-02T14:48:36.603 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:48:36 smithi092 ceph-mon[36497]: pgmap v428: 97 pgs: 97 active+clean; 4.3 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 167 KiB/s rd, 12 MiB/s wr, 900 op/s 2024-04-02T14:48:36.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:48:36 smithi177 ceph-mon[36431]: pgmap v428: 97 pgs: 97 active+clean; 4.3 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 167 KiB/s rd, 12 MiB/s wr, 900 op/s 2024-04-02T14:48:36.681 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:48:36 smithi090 ceph-mon[31674]: pgmap v428: 97 pgs: 97 active+clean; 4.3 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 167 KiB/s rd, 12 MiB/s wr, 900 op/s 2024-04-02T14:48:36.824 INFO:teuthology.orchestra.run.smithi090.stderr:progress: 785.0 s, 1615.3 tps, lat 54.886 ms stddev 69.469 2024-04-02T14:48:38.603 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:48:38 smithi092 ceph-mon[36497]: pgmap v429: 97 pgs: 97 active+clean; 4.3 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 11 MiB/s wr, 821 op/s 2024-04-02T14:48:38.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:48:38 smithi177 ceph-mon[36431]: pgmap v429: 97 pgs: 97 active+clean; 4.3 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 11 MiB/s wr, 821 op/s 2024-04-02T14:48:38.682 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:48:38 smithi090 ceph-mon[31674]: pgmap v429: 97 pgs: 97 active+clean; 4.3 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 11 MiB/s wr, 821 op/s 2024-04-02T14:48:40.603 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:48:40 smithi092 ceph-mon[36497]: pgmap v430: 97 pgs: 97 active+clean; 4.3 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 8.8 MiB/s wr, 727 op/s 2024-04-02T14:48:40.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:48:40 smithi177 ceph-mon[36431]: pgmap v430: 97 pgs: 97 active+clean; 4.3 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 8.8 MiB/s wr, 727 op/s 2024-04-02T14:48:40.680 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:48:40 smithi090 ceph-mon[31674]: pgmap v430: 97 pgs: 97 active+clean; 4.3 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 8.8 MiB/s wr, 727 op/s 2024-04-02T14:48:41.824 INFO:teuthology.orchestra.run.smithi090.stderr:progress: 790.0 s, 1393.7 tps, lat 78.909 ms stddev 123.772 2024-04-02T14:48:42.603 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:48:42 smithi092 ceph-mon[36497]: pgmap v431: 97 pgs: 97 active+clean; 4.5 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 252 KiB/s rd, 25 MiB/s wr, 1.97k op/s 2024-04-02T14:48:42.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:48:42 smithi177 ceph-mon[36431]: pgmap v431: 97 pgs: 97 active+clean; 4.5 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 252 KiB/s rd, 25 MiB/s wr, 1.97k op/s 2024-04-02T14:48:42.680 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:48:42 smithi090 ceph-mon[31674]: pgmap v431: 97 pgs: 97 active+clean; 4.5 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 252 KiB/s rd, 25 MiB/s wr, 1.97k op/s 2024-04-02T14:48:44.603 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:48:44 smithi092 ceph-mon[36497]: pgmap v432: 97 pgs: 97 active+clean; 4.5 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 145 KiB/s rd, 21 MiB/s wr, 1.55k op/s 2024-04-02T14:48:44.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:48:44 smithi177 ceph-mon[36431]: pgmap v432: 97 pgs: 97 active+clean; 4.5 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 145 KiB/s rd, 21 MiB/s wr, 1.55k op/s 2024-04-02T14:48:44.680 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:48:44 smithi090 ceph-mon[31674]: pgmap v432: 97 pgs: 97 active+clean; 4.5 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 145 KiB/s rd, 21 MiB/s wr, 1.55k op/s 2024-04-02T14:48:45.529 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.534+0000 7f2a68800640 1 -- 172.21.15.90:0/2911969789 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2a60112590 msgr2=0x7f2a60112970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:48:45.529 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.534+0000 7f2a68800640 1 --2- 172.21.15.90:0/2911969789 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2a60112590 0x7f2a60112970 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f2a54009930 tx=0x7f2a5402bc20 comp rx=0 tx=0).stop 2024-04-02T14:48:45.529 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.534+0000 7f2a68800640 1 -- 172.21.15.90:0/2911969789 shutdown_connections 2024-04-02T14:48:45.529 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.534+0000 7f2a68800640 1 --2- 172.21.15.90:0/2911969789 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2a6010ce90 0x7f2a60119350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:45.529 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.534+0000 7f2a68800640 1 --2- 172.21.15.90:0/2911969789 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2a6010c4d0 0x7f2a6010c950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:45.529 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.534+0000 7f2a68800640 1 --2- 172.21.15.90:0/2911969789 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2a60112590 0x7f2a60112970 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:45.529 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.534+0000 7f2a68800640 1 -- 172.21.15.90:0/2911969789 >> 172.21.15.90:0/2911969789 conn(0x7f2a6006cd10 msgr2=0x7f2a6010ab10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:48:45.530 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.534+0000 7f2a68800640 1 -- 172.21.15.90:0/2911969789 shutdown_connections 2024-04-02T14:48:45.530 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.534+0000 7f2a68800640 1 -- 172.21.15.90:0/2911969789 wait complete. 2024-04-02T14:48:45.531 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.536+0000 7f2a68800640 1 Processor -- start 2024-04-02T14:48:45.531 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.536+0000 7f2a68800640 1 -- start start 2024-04-02T14:48:45.531 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.536+0000 7f2a68800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2a6010c4d0 0x7f2a601a88d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:45.531 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.536+0000 7f2a68800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2a6010ce90 0x7f2a601a8e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:45.531 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.536+0000 7f2a68800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2a60112590 0x7f2a601a2a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:45.531 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.536+0000 7f2a68800640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f2a6011c8c0 con 0x7f2a60112590 2024-04-02T14:48:45.531 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.536+0000 7f2a68800640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f2a6011c740 con 0x7f2a6010ce90 2024-04-02T14:48:45.531 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.536+0000 7f2a68800640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f2a6011ca40 con 0x7f2a6010c4d0 2024-04-02T14:48:45.531 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.536+0000 7f2a65e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2a6010c4d0 0x7f2a601a88d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:45.531 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.536+0000 7f2a65400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2a6010ce90 0x7f2a601a8e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:45.532 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.536+0000 7f2a65e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2a6010c4d0 0x7f2a601a88d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:58322/0 (socket says 172.21.15.90:58322) 2024-04-02T14:48:45.532 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.537+0000 7f2a65400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2a6010ce90 0x7f2a601a8e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:51216/0 (socket says 172.21.15.90:51216) 2024-04-02T14:48:45.532 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.537+0000 7f2a65400640 1 -- 172.21.15.90:0/752286433 learned_addr learned my addr 172.21.15.90:0/752286433 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:48:45.532 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.537+0000 7f2a66800640 1 --2- 172.21.15.90:0/752286433 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2a60112590 0x7f2a601a2a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:45.532 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.537+0000 7f2a65400640 1 -- 172.21.15.90:0/752286433 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2a6010c4d0 msgr2=0x7f2a601a88d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:48:45.533 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.537+0000 7f2a65400640 1 --2- 172.21.15.90:0/752286433 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2a6010c4d0 0x7f2a601a88d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:45.533 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.537+0000 7f2a65400640 1 -- 172.21.15.90:0/752286433 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2a60112590 msgr2=0x7f2a601a2a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:48:45.533 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.537+0000 7f2a65400640 1 --2- 172.21.15.90:0/752286433 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2a60112590 0x7f2a601a2a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:45.533 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.537+0000 7f2a65400640 1 -- 172.21.15.90:0/752286433 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a601a3200 con 0x7f2a6010ce90 2024-04-02T14:48:45.533 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.537+0000 7f2a65400640 1 --2- 172.21.15.90:0/752286433 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2a6010ce90 0x7f2a601a8e10 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7f2a5000d950 tx=0x7f2a5000de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:48:45.533 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.538+0000 7f2a5aa00640 1 -- 172.21.15.90:0/752286433 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a50017030 con 0x7f2a6010ce90 2024-04-02T14:48:45.533 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.538+0000 7f2a5aa00640 1 -- 172.21.15.90:0/752286433 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2a50004020 con 0x7f2a6010ce90 2024-04-02T14:48:45.533 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.538+0000 7f2a5aa00640 1 -- 172.21.15.90:0/752286433 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a500047a0 con 0x7f2a6010ce90 2024-04-02T14:48:45.533 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.538+0000 7f2a68800640 1 -- 172.21.15.90:0/752286433 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a601a34f0 con 0x7f2a6010ce90 2024-04-02T14:48:45.533 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.538+0000 7f2a68800640 1 -- 172.21.15.90:0/752286433 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a601af790 con 0x7f2a6010ce90 2024-04-02T14:48:45.535 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.541+0000 7f2a6b569740 1 Processor -- start 2024-04-02T14:48:45.535 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.541+0000 7f2a5aa00640 1 -- 172.21.15.90:0/752286433 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f2a500041d0 con 0x7f2a6010ce90 2024-04-02T14:48:45.535 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.541+0000 7f2a6b569740 1 -- start start 2024-04-02T14:48:45.536 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.541+0000 7f2a6b569740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5601e9081eb0 0x5601e9082290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:45.536 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.541+0000 7f2a6b569740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5601e90827d0 0x5601e908ae70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:45.536 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.542+0000 7f2a6b569740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5601e908b3b0 0x5601e908d7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:45.537 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.542+0000 7f2a6b569740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5601e8fe6d70 con 0x5601e908b3b0 2024-04-02T14:48:45.537 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.542+0000 7f2a6b569740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x5601e8f8f2a0 con 0x5601e9081eb0 2024-04-02T14:48:45.537 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.542+0000 7f2a6b569740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5601e8fe2d20 con 0x5601e90827d0 2024-04-02T14:48:45.537 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.542+0000 7f2a66800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5601e9081eb0 0x5601e9082290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:45.537 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.542+0000 7f2a65400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5601e908b3b0 0x5601e908d7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:45.537 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.542+0000 7f2a66800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5601e9081eb0 0x5601e9082290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:51220/0 (socket says 172.21.15.90:51220) 2024-04-02T14:48:45.537 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.542+0000 7f2a66800640 1 -- 172.21.15.90:0/2993310531 learned_addr learned my addr 172.21.15.90:0/2993310531 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:48:45.537 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.542+0000 7f2a65e00640 1 --2- 172.21.15.90:0/2993310531 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5601e90827d0 0x5601e908ae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:45.537 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.542+0000 7f2a66800640 1 -- 172.21.15.90:0/2993310531 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5601e90827d0 msgr2=0x5601e908ae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:48:45.537 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.542+0000 7f2a66800640 1 --2- 172.21.15.90:0/2993310531 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5601e90827d0 0x5601e908ae70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:45.537 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.542+0000 7f2a66800640 1 -- 172.21.15.90:0/2993310531 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5601e908b3b0 msgr2=0x5601e908d7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:48:45.537 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.542+0000 7f2a66800640 1 --2- 172.21.15.90:0/2993310531 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5601e908b3b0 0x5601e908d7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:45.537 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.542+0000 7f2a66800640 1 -- 172.21.15.90:0/2993310531 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5601e8fa3840 con 0x5601e9081eb0 2024-04-02T14:48:45.538 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.543+0000 7f2a66800640 1 --2- 172.21.15.90:0/2993310531 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5601e9081eb0 0x5601e9082290 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7f2a5c009f50 tx=0x7f2a5c0118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a4004b7a0eb7f4ae server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:48:45.538 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.543+0000 7f2a4f000640 1 -- 172.21.15.90:0/2993310531 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a5c004b20 con 0x5601e9081eb0 2024-04-02T14:48:45.538 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.543+0000 7f2a4f000640 1 -- 172.21.15.90:0/2993310531 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2a5c004cc0 con 0x5601e9081eb0 2024-04-02T14:48:45.539 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.543+0000 7f2a4f000640 1 -- 172.21.15.90:0/2993310531 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a5c005020 con 0x5601e9081eb0 2024-04-02T14:48:45.539 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.543+0000 7f2a5aa00640 1 --2- 172.21.15.90:0/752286433 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f2a480a3e40 0x7f2a480a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:45.539 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.544+0000 7f2a6b569740 1 -- 172.21.15.90:0/2993310531 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5601e9081eb0 msgr2=0x5601e9082290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:48:45.539 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.544+0000 7f2a6b569740 1 --2- 172.21.15.90:0/2993310531 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5601e9081eb0 0x5601e9082290 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7f2a5c009f50 tx=0x7f2a5c0118d0 comp rx=0 tx=0).stop 2024-04-02T14:48:45.539 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.544+0000 7f2a5aa00640 1 -- 172.21.15.90:0/752286433 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 8550+0+0 (secure 0 0 0) 0x7f2a500d0190 con 0x7f2a6010ce90 2024-04-02T14:48:45.539 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.544+0000 7f2a65e00640 1 --2- 172.21.15.90:0/752286433 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f2a480a3e40 0x7f2a480a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:45.539 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.544+0000 7f2a6b569740 1 -- 172.21.15.90:0/2993310531 shutdown_connections 2024-04-02T14:48:45.539 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.544+0000 7f2a6b569740 1 --2- 172.21.15.90:0/2993310531 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5601e908b3b0 0x5601e908d7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:45.539 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.544+0000 7f2a6b569740 1 --2- 172.21.15.90:0/2993310531 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5601e90827d0 0x5601e908ae70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:45.539 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.544+0000 7f2a6b569740 1 --2- 172.21.15.90:0/2993310531 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5601e9081eb0 0x5601e9082290 unknown :-1 s=CLOSED pgs=946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:45.539 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.544+0000 7f2a6b569740 1 -- 172.21.15.90:0/2993310531 >> 172.21.15.90:0/2993310531 conn(0x5601e907c5e0 msgr2=0x5601e907c9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:48:45.539 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.544+0000 7f2a6b569740 1 -- 172.21.15.90:0/2993310531 shutdown_connections 2024-04-02T14:48:45.540 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.544+0000 7f2a6b569740 1 -- 172.21.15.90:0/2993310531 wait complete. 2024-04-02T14:48:45.540 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.545+0000 7f2a6b569740 1 -- start start 2024-04-02T14:48:45.540 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.545+0000 7f2a65e00640 1 --2- 172.21.15.90:0/752286433 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f2a480a3e40 0x7f2a480a6300 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f2a54009900 tx=0x7f2a54005a20 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:48:45.540 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.545+0000 7f2a6b569740 1 Processor -- start 2024-04-02T14:48:45.542 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.547+0000 7f2a6b569740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5601e9081eb0 0x5601e9088400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:45.542 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.547+0000 7f2a6b569740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5601e90827d0 0x5601e9088940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:45.542 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.547+0000 7f2a6b569740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5601e908b3b0 0x5601e9088e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:45.542 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.547+0000 7f2a6b569740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5601e908e1a0 con 0x5601e90827d0 2024-04-02T14:48:45.542 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.547+0000 7f2a6b569740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x5601e8fa2660 con 0x5601e908b3b0 2024-04-02T14:48:45.542 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.547+0000 7f2a6b569740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5601e908e320 con 0x5601e9081eb0 2024-04-02T14:48:45.543 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.547+0000 7f2a6b569740 10 client.0 _ll_register_callbacks cb 0x5601e8fa2050 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:48:45.543 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.548+0000 7f2a65400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5601e90827d0 0x5601e9088940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:45.543 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.548+0000 7f2a65e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5601e9081eb0 0x5601e9088400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:45.543 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.548+0000 7f2a65400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5601e90827d0 0x5601e9088940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:59884/0 (socket says 172.21.15.90:59884) 2024-04-02T14:48:45.543 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.548+0000 7f2a65400640 1 -- 172.21.15.90:0/4164454048 learned_addr learned my addr 172.21.15.90:0/4164454048 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:48:45.544 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.548+0000 7f2a66800640 1 --2- 172.21.15.90:0/4164454048 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5601e908b3b0 0x5601e9088e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:45.544 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.548+0000 7f2a65400640 1 -- 172.21.15.90:0/4164454048 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5601e9081eb0 msgr2=0x5601e9088400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:48:45.544 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.548+0000 7f2a65400640 1 --2- 172.21.15.90:0/4164454048 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5601e9081eb0 0x5601e9088400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:45.544 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.548+0000 7f2a65400640 1 -- 172.21.15.90:0/4164454048 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5601e908b3b0 msgr2=0x5601e9088e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:48:45.544 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.548+0000 7f2a65400640 1 --2- 172.21.15.90:0/4164454048 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5601e908b3b0 0x5601e9088e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:45.544 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.548+0000 7f2a65400640 1 -- 172.21.15.90:0/4164454048 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5601e8fb1e80 con 0x5601e90827d0 2024-04-02T14:48:45.544 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.549+0000 7f2a65400640 1 --2- 172.21.15.90:0/4164454048 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5601e90827d0 0x5601e9088940 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f2a50009610 tx=0x7f2a500d38d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba2e7f0b9c76a9b server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:48:45.544 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.549+0000 7f2a43400640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-02T14:48:45.544 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.549+0000 7f2a43400640 1 -- 172.21.15.90:0/4164454048 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a5009bee0 con 0x5601e90827d0 2024-04-02T14:48:45.544 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.549+0000 7f2a43400640 1 -- 172.21.15.90:0/4164454048 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 8550+0+0 (secure 0 0 0) 0x7f2a500a01e0 con 0x5601e90827d0 2024-04-02T14:48:45.545 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.549+0000 7f2a6b569740 1 -- 172.21.15.90:0/4164454048 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2a5c004cc0 con 0x5601e90827d0 2024-04-02T14:48:45.545 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.549+0000 7f2a43400640 1 -- 172.21.15.90:0/4164454048 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f2a5009ba70 con 0x5601e90827d0 2024-04-02T14:48:45.545 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.549+0000 7f2a6b569740 10 client.18573 fetch_fsmap learned FSMap version 27 2024-04-02T14:48:45.545 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.549+0000 7f2a6b569740 1 -- 172.21.15.90:0/4164454048 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5601e8cfb210 con 0x5601e90827d0 2024-04-02T14:48:45.546 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.550+0000 7f2a43400640 1 -- 172.21.15.90:0/4164454048 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2a5c004cc0 con 0x5601e90827d0 2024-04-02T14:48:45.546 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.551+0000 7f2a43400640 1 -- 172.21.15.90:0/4164454048 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f2a500b0bb0 con 0x5601e90827d0 2024-04-02T14:48:45.546 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.551+0000 7f2a6b569740 10 client.18573 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:48:45.546 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.551+0000 7f2a6b569740 10 client.18573 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-02T14:48:45.546 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.551+0000 7f2a6b569740 20 client.18573 populate_metadata read hostname 'smithi090' 2024-04-02T14:48:45.546 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.551+0000 7f2a6b569740 1 --2- 172.21.15.90:0/4164454048 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5601e90a3fc0 0x5601e90a63b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:45.546 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.551+0000 7f2a6b569740 4 client.18573 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:48:45.546 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.551+0000 7f2a6b569740 1 -- 172.21.15.90:0/4164454048 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5601e8f96cd0 con 0x5601e90a3fc0 2024-04-02T14:48:45.546 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.551+0000 7f2a65e00640 1 --2- 172.21.15.90:0/4164454048 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5601e90a3fc0 0x5601e90a63b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:45.547 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.552+0000 7f2a65e00640 1 --2- 172.21.15.90:0/4164454048 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5601e90a3fc0 0x5601e90a63b0 crc :-1 s=READY pgs=354 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4274ac5c69ee315b server_cookie=a4fb28b6b671deac in_seq=0 out_seq=0 2024-04-02T14:48:45.547 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.552+0000 7f2a43400640 10 client.18573 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:48:45.548 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.552+0000 7f2a43400640 1 -- 172.21.15.90:0/4164454048 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5601e8f96cd0 con 0x5601e90a3fc0 2024-04-02T14:48:45.548 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.552+0000 7f2a43400640 10 client.18573 handle_command_reply: tid=0 2024-04-02T14:48:45.548 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.552+0000 7f2a6b569740 1 client.18573 shutdown 2024-04-02T14:48:45.548 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.553+0000 7f2a6b569740 1 -- 172.21.15.90:0/4164454048 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5601e90827d0 msgr2=0x5601e9088940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:48:45.548 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.553+0000 7f2a6b569740 1 --2- 172.21.15.90:0/4164454048 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5601e90827d0 0x5601e9088940 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f2a50009610 tx=0x7f2a500d38d0 comp rx=0 tx=0).stop 2024-04-02T14:48:45.548 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.553+0000 7f2a6b569740 1 -- 172.21.15.90:0/4164454048 shutdown_connections 2024-04-02T14:48:45.548 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.553+0000 7f2a6b569740 1 --2- 172.21.15.90:0/4164454048 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5601e90a3fc0 0x5601e90a63b0 crc :-1 s=READY pgs=354 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:45.548 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.553+0000 7f2a6b569740 1 --2- 172.21.15.90:0/4164454048 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5601e908b3b0 0x5601e9088e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:45.548 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.553+0000 7f2a6b569740 1 --2- 172.21.15.90:0/4164454048 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5601e90827d0 0x5601e9088940 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:45.548 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.553+0000 7f2a6b569740 1 --2- 172.21.15.90:0/4164454048 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5601e9081eb0 0x5601e9088400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:45.549 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.553+0000 7f2a6b569740 1 -- 172.21.15.90:0/4164454048 >> 172.21.15.90:0/4164454048 conn(0x5601e907c5e0 msgr2=0x5601e907e330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:48:45.549 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.553+0000 7f2a43400640 0 client.18573 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:48:45.549 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.553+0000 7f2a6b569740 1 -- 172.21.15.90:0/4164454048 shutdown_connections 2024-04-02T14:48:45.549 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.553+0000 7f2a6b569740 1 -- 172.21.15.90:0/4164454048 wait complete. 2024-04-02T14:48:45.549 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.553+0000 7f2a6b569740 20 client.18573 trim_cache size 0 max 16384 2024-04-02T14:48:45.576 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.580+0000 7f2a6b569740 1 Processor -- start 2024-04-02T14:48:45.576 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.580+0000 7f2a6b569740 1 -- start start 2024-04-02T14:48:45.576 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.580+0000 7f2a6b569740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5601e907c5e0 0x5601e908aac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:45.576 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.580+0000 7f2a6b569740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5601e9081eb0 0x5601e9097060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:45.576 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.580+0000 7f2a6b569740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5601e90827d0 0x5601e9088ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:45.576 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.580+0000 7f2a6b569740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f2a601b3e70 con 0x5601e90827d0 2024-04-02T14:48:45.576 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.580+0000 7f2a6b569740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f2a601b59b0 con 0x5601e9081eb0 2024-04-02T14:48:45.576 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.580+0000 7f2a6b569740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f2a601b4540 con 0x5601e907c5e0 2024-04-02T14:48:45.576 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.581+0000 7f2a65400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5601e9081eb0 0x5601e9097060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:45.576 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.581+0000 7f2a65400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5601e9081eb0 0x5601e9097060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:51246/0 (socket says 172.21.15.90:51246) 2024-04-02T14:48:45.576 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.581+0000 7f2a65400640 1 -- 172.21.15.90:0/3517978728 learned_addr learned my addr 172.21.15.90:0/3517978728 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:48:45.576 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.581+0000 7f2a65e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5601e907c5e0 0x5601e908aac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:45.577 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.581+0000 7f2a66800640 1 --2- 172.21.15.90:0/3517978728 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5601e90827d0 0x5601e9088ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:45.577 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.581+0000 7f2a65400640 1 -- 172.21.15.90:0/3517978728 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5601e907c5e0 msgr2=0x5601e908aac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:48:45.577 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.581+0000 7f2a65400640 1 --2- 172.21.15.90:0/3517978728 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5601e907c5e0 0x5601e908aac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:45.577 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.581+0000 7f2a65400640 1 -- 172.21.15.90:0/3517978728 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5601e90827d0 msgr2=0x5601e9088ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:48:45.577 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.581+0000 7f2a65400640 1 --2- 172.21.15.90:0/3517978728 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5601e90827d0 0x5601e9088ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:45.577 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.581+0000 7f2a65400640 1 -- 172.21.15.90:0/3517978728 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5601e8dc7620 con 0x5601e9081eb0 2024-04-02T14:48:45.577 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.581+0000 7f2a66800640 1 --2- 172.21.15.90:0/3517978728 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5601e90827d0 0x5601e9088ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:48:45.577 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.581+0000 7f2a65400640 1 --2- 172.21.15.90:0/3517978728 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5601e9081eb0 0x5601e9097060 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7f2a50009fd0 tx=0x7f2a500d00d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1efab08a3cce48f7 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:48:45.577 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.581+0000 7f2a4e600640 1 -- 172.21.15.90:0/3517978728 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a5009bc50 con 0x5601e9081eb0 2024-04-02T14:48:45.577 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.582+0000 7f2a4e600640 1 -- 172.21.15.90:0/3517978728 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2a5009c810 con 0x5601e9081eb0 2024-04-02T14:48:45.577 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.582+0000 7f2a4e600640 1 -- 172.21.15.90:0/3517978728 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a5009b650 con 0x5601e9081eb0 2024-04-02T14:48:45.578 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.582+0000 7f2a6b569740 1 -- 172.21.15.90:0/3517978728 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5601e9081eb0 msgr2=0x5601e9097060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:48:45.578 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.582+0000 7f2a6b569740 1 --2- 172.21.15.90:0/3517978728 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5601e9081eb0 0x5601e9097060 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7f2a50009fd0 tx=0x7f2a500d00d0 comp rx=0 tx=0).stop 2024-04-02T14:48:45.578 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.583+0000 7f2a6b569740 1 -- 172.21.15.90:0/3517978728 shutdown_connections 2024-04-02T14:48:45.579 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.583+0000 7f2a6b569740 1 --2- 172.21.15.90:0/3517978728 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5601e90827d0 0x5601e9088ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:45.579 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.583+0000 7f2a6b569740 1 --2- 172.21.15.90:0/3517978728 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5601e9081eb0 0x5601e9097060 unknown :-1 s=CLOSED pgs=947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:45.579 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.583+0000 7f2a6b569740 1 --2- 172.21.15.90:0/3517978728 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5601e907c5e0 0x5601e908aac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:45.579 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.583+0000 7f2a6b569740 1 -- 172.21.15.90:0/3517978728 >> 172.21.15.90:0/3517978728 conn(0x5601e90a67a0 msgr2=0x5601e90883c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:48:45.579 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.583+0000 7f2a6b569740 1 -- 172.21.15.90:0/3517978728 shutdown_connections 2024-04-02T14:48:45.579 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.583+0000 7f2a6b569740 1 -- 172.21.15.90:0/3517978728 wait complete. 2024-04-02T14:48:45.579 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.583+0000 7f2a6b569740 1 -- start start 2024-04-02T14:48:45.579 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.583+0000 7f2a6b569740 1 Processor -- start 2024-04-02T14:48:45.580 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.586+0000 7f2a6b569740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5601e907c5e0 0x5601e90850e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:45.583 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.586+0000 7f2a6b569740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5601e9081eb0 0x5601e9085620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:45.583 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.586+0000 7f2a6b569740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5601e90827d0 0x5601e9085b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:45.583 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.587+0000 7f2a6b569740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f2a601b52e0 con 0x5601e9081eb0 2024-04-02T14:48:45.583 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.587+0000 7f2a6b569740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f2a601b4c10 con 0x5601e907c5e0 2024-04-02T14:48:45.583 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.587+0000 7f2a6b569740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f2a601b1e90 con 0x5601e90827d0 2024-04-02T14:48:45.583 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.587+0000 7f2a6b569740 10 client.0 _ll_register_callbacks cb 0x7f2a500d39d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:48:45.583 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.587+0000 7f2a65e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5601e907c5e0 0x5601e90850e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:45.583 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.587+0000 7f2a65e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5601e907c5e0 0x5601e90850e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:51260/0 (socket says 172.21.15.90:51260) 2024-04-02T14:48:45.583 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.587+0000 7f2a65e00640 1 -- 172.21.15.90:0/479515649 learned_addr learned my addr 172.21.15.90:0/479515649 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:48:45.583 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.587+0000 7f2a65400640 1 --2- 172.21.15.90:0/479515649 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5601e9081eb0 0x5601e9085620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:45.583 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.587+0000 7f2a66800640 1 --2- 172.21.15.90:0/479515649 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5601e90827d0 0x5601e9085b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:45.583 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.587+0000 7f2a65e00640 1 -- 172.21.15.90:0/479515649 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5601e90827d0 msgr2=0x5601e9085b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:48:45.583 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.587+0000 7f2a65e00640 1 --2- 172.21.15.90:0/479515649 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5601e90827d0 0x5601e9085b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:45.583 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.587+0000 7f2a65e00640 1 -- 172.21.15.90:0/479515649 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5601e9081eb0 msgr2=0x5601e9085620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:48:45.584 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.587+0000 7f2a65e00640 1 --2- 172.21.15.90:0/479515649 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5601e9081eb0 0x5601e9085620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:45.584 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.587+0000 7f2a65e00640 1 -- 172.21.15.90:0/479515649 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5601e9109450 con 0x5601e907c5e0 2024-04-02T14:48:45.584 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.587+0000 7f2a65400640 1 --2- 172.21.15.90:0/479515649 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5601e9081eb0 0x5601e9085620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:48:45.584 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.587+0000 7f2a65e00640 1 --2- 172.21.15.90:0/479515649 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5601e907c5e0 0x5601e90850e0 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7f2a5402c230 tx=0x7f2a54009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e32e8d0bfda5a4a3 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:48:45.584 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.587+0000 7f2a43400640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:48:45.584 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.588+0000 7f2a43400640 1 -- 172.21.15.90:0/479515649 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a54046070 con 0x5601e907c5e0 2024-04-02T14:48:45.584 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.588+0000 7f2a43400640 1 -- 172.21.15.90:0/479515649 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 8550+0+0 (secure 0 0 0) 0x7f2a5402cbb0 con 0x5601e907c5e0 2024-04-02T14:48:45.584 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.588+0000 7f2a6b569740 1 -- 172.21.15.90:0/479515649 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2a5009c810 con 0x5601e907c5e0 2024-04-02T14:48:45.584 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.588+0000 7f2a43400640 1 -- 172.21.15.90:0/479515649 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f2a540583f0 con 0x5601e907c5e0 2024-04-02T14:48:45.584 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.588+0000 7f2a6b569740 10 client.29777 fetch_fsmap learned FSMap version 27 2024-04-02T14:48:45.584 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.588+0000 7f2a6b569740 1 -- 172.21.15.90:0/479515649 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5601e910a730 con 0x5601e907c5e0 2024-04-02T14:48:45.584 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.588+0000 7f2a43400640 1 -- 172.21.15.90:0/479515649 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2a5009c810 con 0x5601e907c5e0 2024-04-02T14:48:45.584 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.588+0000 7f2a43400640 1 -- 172.21.15.90:0/479515649 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f2a54004f70 con 0x5601e907c5e0 2024-04-02T14:48:45.584 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.588+0000 7f2a6b569740 10 client.29777 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:48:45.584 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.588+0000 7f2a6b569740 10 client.29777 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-02T14:48:45.585 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.588+0000 7f2a6b569740 20 client.29777 populate_metadata read hostname 'smithi090' 2024-04-02T14:48:45.585 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.588+0000 7f2a6b569740 1 --2- 172.21.15.90:0/479515649 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5601e908b3b0 0x5601e910cf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:45.585 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.589+0000 7f2a6b569740 4 client.29777 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-02T14:48:45.585 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.589+0000 7f2a6b569740 1 -- 172.21.15.90:0/479515649 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5601e8ff42b0 con 0x5601e908b3b0 2024-04-02T14:48:45.585 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.589+0000 7f2a65400640 1 --2- 172.21.15.90:0/479515649 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5601e908b3b0 0x5601e910cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:45.585 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.589+0000 7f2a65400640 1 --2- 172.21.15.90:0/479515649 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5601e908b3b0 0x5601e910cf70 crc :-1 s=READY pgs=355 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=279e27ea62253539 server_cookie=223d707fd65469c6 in_seq=0 out_seq=0 2024-04-02T14:48:45.586 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.589+0000 7f2a43400640 10 client.29777 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:48:45.680 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.685+0000 7f2a43400640 1 -- 172.21.15.90:0/479515649 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5601e8ff42b0 con 0x5601e908b3b0 2024-04-02T14:48:45.680 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.685+0000 7f2a43400640 10 client.29777 handle_command_reply: tid=0 2024-04-02T14:48:45.681 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.685+0000 7f2a6b569740 1 client.29777 shutdown 2024-04-02T14:48:45.681 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.686+0000 7f2a6b569740 1 -- 172.21.15.90:0/479515649 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5601e907c5e0 msgr2=0x5601e90850e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:48:45.681 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.686+0000 7f2a6b569740 1 --2- 172.21.15.90:0/479515649 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5601e907c5e0 0x5601e90850e0 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7f2a5402c230 tx=0x7f2a54009260 comp rx=0 tx=0).stop 2024-04-02T14:48:45.681 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.686+0000 7f2a6b569740 1 -- 172.21.15.90:0/479515649 shutdown_connections 2024-04-02T14:48:45.681 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.686+0000 7f2a6b569740 1 --2- 172.21.15.90:0/479515649 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5601e908b3b0 0x5601e910cf70 crc :-1 s=READY pgs=355 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:45.681 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.686+0000 7f2a6b569740 1 --2- 172.21.15.90:0/479515649 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5601e90827d0 0x5601e9085b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:45.681 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.686+0000 7f2a6b569740 1 --2- 172.21.15.90:0/479515649 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5601e9081eb0 0x5601e9085620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:45.681 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.686+0000 7f2a6b569740 1 --2- 172.21.15.90:0/479515649 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5601e907c5e0 0x5601e90850e0 unknown :-1 s=CLOSED pgs=948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:45.681 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.686+0000 7f2a43400640 0 client.29777 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:48:45.681 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.686+0000 7f2a6b569740 1 -- 172.21.15.90:0/479515649 >> 172.21.15.90:0/479515649 conn(0x5601e90a67a0 msgr2=0x5601e9088c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:48:45.682 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.686+0000 7f2a6b569740 1 -- 172.21.15.90:0/479515649 shutdown_connections 2024-04-02T14:48:45.682 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.687+0000 7f2a6b569740 1 -- 172.21.15.90:0/479515649 wait complete. 2024-04-02T14:48:45.682 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.687+0000 7f2a6b569740 20 client.29777 trim_cache size 0 max 16384 2024-04-02T14:48:45.682 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:{ 2024-04-02T14:48:45.682 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "message": "", 2024-04-02T14:48:45.682 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "return_code": 0 2024-04-02T14:48:45.682 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:} 2024-04-02T14:48:45.683 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.688+0000 7f2a68800640 1 -- 172.21.15.90:0/752286433 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f2a480a3e40 msgr2=0x7f2a480a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:48:45.683 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.688+0000 7f2a68800640 1 --2- 172.21.15.90:0/752286433 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f2a480a3e40 0x7f2a480a6300 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f2a54009900 tx=0x7f2a54005a20 comp rx=0 tx=0).stop 2024-04-02T14:48:45.683 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.688+0000 7f2a68800640 1 -- 172.21.15.90:0/752286433 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2a6010ce90 msgr2=0x7f2a601a8e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:48:45.683 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.688+0000 7f2a68800640 1 --2- 172.21.15.90:0/752286433 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2a6010ce90 0x7f2a601a8e10 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7f2a5000d950 tx=0x7f2a5000de10 comp rx=0 tx=0).stop 2024-04-02T14:48:45.683 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.689+0000 7f2a68800640 1 -- 172.21.15.90:0/752286433 shutdown_connections 2024-04-02T14:48:45.683 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.689+0000 7f2a68800640 1 --2- 172.21.15.90:0/752286433 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f2a480a3e40 0x7f2a480a6300 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:45.683 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.689+0000 7f2a68800640 1 --2- 172.21.15.90:0/752286433 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2a60112590 0x7f2a601a2a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:45.684 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.689+0000 7f2a68800640 1 --2- 172.21.15.90:0/752286433 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2a6010ce90 0x7f2a601a8e10 secure :-1 s=CLOSED pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7f2a5000d950 tx=0x7f2a5000de10 comp rx=0 tx=0).stop 2024-04-02T14:48:45.684 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.689+0000 7f2a68800640 1 --2- 172.21.15.90:0/752286433 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2a6010c4d0 0x7f2a601a88d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:45.684 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.689+0000 7f2a68800640 1 -- 172.21.15.90:0/752286433 >> 172.21.15.90:0/752286433 conn(0x7f2a6006cd10 msgr2=0x7f2a6010ac40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:48:45.684 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.689+0000 7f2a68800640 1 -- 172.21.15.90:0/752286433 shutdown_connections 2024-04-02T14:48:45.684 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:45.689+0000 7f2a68800640 1 -- 172.21.15.90:0/752286433 wait complete. 2024-04-02T14:48:46.603 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:48:46 smithi092 ceph-mon[36497]: pgmap v433: 97 pgs: 97 active+clean; 4.5 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 220 KiB/s rd, 25 MiB/s wr, 1.93k op/s 2024-04-02T14:48:46.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:48:46 smithi177 ceph-mon[36431]: pgmap v433: 97 pgs: 97 active+clean; 4.5 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 220 KiB/s rd, 25 MiB/s wr, 1.93k op/s 2024-04-02T14:48:46.680 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:48:46 smithi090 ceph-mon[31674]: pgmap v433: 97 pgs: 97 active+clean; 4.5 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 220 KiB/s rd, 25 MiB/s wr, 1.93k op/s 2024-04-02T14:48:46.824 INFO:teuthology.orchestra.run.smithi090.stderr:progress: 795.0 s, 1463.4 tps, lat 69.586 ms stddev 94.075 2024-04-02T14:48:48.603 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:48:48 smithi092 ceph-mon[36497]: pgmap v434: 97 pgs: 97 active+clean; 4.5 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 194 KiB/s rd, 24 MiB/s wr, 1.89k op/s 2024-04-02T14:48:48.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:48:48 smithi177 ceph-mon[36431]: pgmap v434: 97 pgs: 97 active+clean; 4.5 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 194 KiB/s rd, 24 MiB/s wr, 1.89k op/s 2024-04-02T14:48:48.680 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:48:48 smithi090 ceph-mon[31674]: pgmap v434: 97 pgs: 97 active+clean; 4.5 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 194 KiB/s rd, 24 MiB/s wr, 1.89k op/s 2024-04-02T14:48:50.603 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:48:50 smithi092 ceph-mon[36497]: pgmap v435: 97 pgs: 97 active+clean; 4.5 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 193 KiB/s rd, 23 MiB/s wr, 1.80k op/s 2024-04-02T14:48:50.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:48:50 smithi177 ceph-mon[36431]: pgmap v435: 97 pgs: 97 active+clean; 4.5 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 193 KiB/s rd, 23 MiB/s wr, 1.80k op/s 2024-04-02T14:48:50.680 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:48:50 smithi090 ceph-mon[31674]: pgmap v435: 97 pgs: 97 active+clean; 4.5 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 193 KiB/s rd, 23 MiB/s wr, 1.80k op/s 2024-04-02T14:48:51.824 INFO:teuthology.orchestra.run.smithi090.stderr:progress: 800.0 s, 1492.4 tps, lat 66.566 ms stddev 73.315 2024-04-02T14:48:52.603 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:48:52 smithi092 ceph-mon[36497]: pgmap v436: 97 pgs: 97 active+clean; 4.6 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 207 KiB/s rd, 26 MiB/s wr, 2.08k op/s 2024-04-02T14:48:52.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:48:52 smithi177 ceph-mon[36431]: pgmap v436: 97 pgs: 97 active+clean; 4.6 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 207 KiB/s rd, 26 MiB/s wr, 2.08k op/s 2024-04-02T14:48:52.681 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:48:52 smithi090 ceph-mon[31674]: pgmap v436: 97 pgs: 97 active+clean; 4.6 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 207 KiB/s rd, 26 MiB/s wr, 2.08k op/s 2024-04-02T14:48:54.603 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:48:54 smithi092 ceph-mon[36497]: pgmap v437: 97 pgs: 97 active+clean; 4.6 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 9.6 MiB/s wr, 840 op/s 2024-04-02T14:48:54.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:48:54 smithi177 ceph-mon[36431]: pgmap v437: 97 pgs: 97 active+clean; 4.6 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 9.6 MiB/s wr, 840 op/s 2024-04-02T14:48:54.681 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:48:54 smithi090 ceph-mon[31674]: pgmap v437: 97 pgs: 97 active+clean; 4.6 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 9.6 MiB/s wr, 840 op/s 2024-04-02T14:48:56.603 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:48:56 smithi092 ceph-mon[36497]: pgmap v438: 97 pgs: 97 active+clean; 4.6 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 11 MiB/s wr, 1k op/s 2024-04-02T14:48:56.678 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:48:56 smithi177 ceph-mon[36431]: pgmap v438: 97 pgs: 97 active+clean; 4.6 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 11 MiB/s wr, 1k op/s 2024-04-02T14:48:56.680 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:48:56 smithi090 ceph-mon[31674]: pgmap v438: 97 pgs: 97 active+clean; 4.6 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 11 MiB/s wr, 1k op/s 2024-04-02T14:48:56.824 INFO:teuthology.orchestra.run.smithi090.stderr:progress: 805.0 s, 1540.4 tps, lat 65.593 ms stddev 81.473 2024-04-02T14:48:57.818 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-02T14:48:57.819 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-02T14:48:58.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.080+0000 7f7589c00640 1 -- 172.21.15.90:0/3055893792 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f758410e4a0 msgr2=0x7f7584110890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:48:58.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.080+0000 7f7589c00640 1 --2- 172.21.15.90:0/3055893792 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f758410e4a0 0x7f7584110890 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f757c009860 tx=0x7f757c02bb70 comp rx=0 tx=0).stop 2024-04-02T14:48:58.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.080+0000 7f7589c00640 1 -- 172.21.15.90:0/3055893792 shutdown_connections 2024-04-02T14:48:58.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.080+0000 7f7589c00640 1 --2- 172.21.15.90:0/3055893792 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f758410e4a0 0x7f7584110890 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:58.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.080+0000 7f7589c00640 1 --2- 172.21.15.90:0/3055893792 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7584069600 0x7f758410df60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:58.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.080+0000 7f7589c00640 1 --2- 172.21.15.90:0/3055893792 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7584068ce0 0x7f75840690c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:58.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.080+0000 7f7589c00640 1 -- 172.21.15.90:0/3055893792 >> 172.21.15.90:0/3055893792 conn(0x7f7584077b00 msgr2=0x7f7584077f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:48:58.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.080+0000 7f7589c00640 1 -- 172.21.15.90:0/3055893792 shutdown_connections 2024-04-02T14:48:58.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.080+0000 7f7589c00640 1 -- 172.21.15.90:0/3055893792 wait complete. 2024-04-02T14:48:58.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.081+0000 7f7589c00640 1 Processor -- start 2024-04-02T14:48:58.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.081+0000 7f7589c00640 1 -- start start 2024-04-02T14:48:58.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.081+0000 7f7589c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7584068ce0 0x7f758410c130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:58.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.081+0000 7f7589c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7584069600 0x7f758410c670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:58.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.081+0000 7f7589c00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f758410e4a0 0x7f758410cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:58.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.081+0000 7f7589c00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f7584103550 con 0x7f7584068ce0 2024-04-02T14:48:58.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.082+0000 7f7589c00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f75841033d0 con 0x7f758410e4a0 2024-04-02T14:48:58.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.082+0000 7f7589c00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f75841036d0 con 0x7f7584069600 2024-04-02T14:48:58.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.082+0000 7f7583400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f758410e4a0 0x7f758410cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:58.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.082+0000 7f7582000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7584069600 0x7f758410c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:58.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.082+0000 7f7583400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f758410e4a0 0x7f758410cbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:42002/0 (socket says 172.21.15.90:42002) 2024-04-02T14:48:58.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.082+0000 7f7583400640 1 -- 172.21.15.90:0/4265553887 learned_addr learned my addr 172.21.15.90:0/4265553887 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:48:58.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.082+0000 7f7582a00640 1 --2- 172.21.15.90:0/4265553887 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7584068ce0 0x7f758410c130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:58.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.082+0000 7f7583400640 1 -- 172.21.15.90:0/4265553887 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7584069600 msgr2=0x7f758410c670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:48:58.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.082+0000 7f7583400640 1 --2- 172.21.15.90:0/4265553887 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7584069600 0x7f758410c670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:58.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.082+0000 7f7583400640 1 -- 172.21.15.90:0/4265553887 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7584068ce0 msgr2=0x7f758410c130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:48:58.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.082+0000 7f7583400640 1 --2- 172.21.15.90:0/4265553887 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7584068ce0 0x7f758410c130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:58.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.082+0000 7f7583400640 1 -- 172.21.15.90:0/4265553887 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f758406b620 con 0x7f758410e4a0 2024-04-02T14:48:58.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.082+0000 7f7582000640 1 --2- 172.21.15.90:0/4265553887 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7584069600 0x7f758410c670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:48:58.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.082+0000 7f7583400640 1 --2- 172.21.15.90:0/4265553887 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f758410e4a0 0x7f758410cbb0 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7f757c02c180 tx=0x7f757c0045e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:48:58.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.083+0000 7f757b400640 1 -- 172.21.15.90:0/4265553887 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f757c041020 con 0x7f758410e4a0 2024-04-02T14:48:58.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.083+0000 7f757b400640 1 -- 172.21.15.90:0/4265553887 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f757c02c9f0 con 0x7f758410e4a0 2024-04-02T14:48:58.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.083+0000 7f757b400640 1 -- 172.21.15.90:0/4265553887 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f757c039040 con 0x7f758410e4a0 2024-04-02T14:48:58.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.083+0000 7f7589c00640 1 -- 172.21.15.90:0/4265553887 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f758406b8b0 con 0x7f758410e4a0 2024-04-02T14:48:58.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.083+0000 7f7589c00640 1 -- 172.21.15.90:0/4265553887 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f758406bd50 con 0x7f758410e4a0 2024-04-02T14:48:58.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.088+0000 7f757b400640 1 -- 172.21.15.90:0/4265553887 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f757c044020 con 0x7f758410e4a0 2024-04-02T14:48:58.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.088+0000 7f758c9b2740 1 Processor -- start 2024-04-02T14:48:58.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.088+0000 7f758c9b2740 1 -- start start 2024-04-02T14:48:58.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.088+0000 7f758c9b2740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563621d477b0 0x563621d47b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:58.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.088+0000 7f758c9b2740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563621d480d0 0x563621d50770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:58.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.088+0000 7f758c9b2740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563621d50cb0 0x563621d530a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:58.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.088+0000 7f758c9b2740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x563621cab770 con 0x563621d50cb0 2024-04-02T14:48:58.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.088+0000 7f758c9b2740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x563621c53700 con 0x563621d480d0 2024-04-02T14:48:58.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.088+0000 7f758c9b2740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x563621ca7720 con 0x563621d477b0 2024-04-02T14:48:58.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.089+0000 7f7582a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563621d480d0 0x563621d50770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:58.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.089+0000 7f7582a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563621d480d0 0x563621d50770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:42010/0 (socket says 172.21.15.90:42010) 2024-04-02T14:48:58.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.089+0000 7f7582a00640 1 -- 172.21.15.90:0/377186270 learned_addr learned my addr 172.21.15.90:0/377186270 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:48:58.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.089+0000 7f7582a00640 1 -- 172.21.15.90:0/377186270 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563621d477b0 msgr2=0x563621d47b90 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-02T14:48:58.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.089+0000 7f7582a00640 1 --2- 172.21.15.90:0/377186270 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563621d477b0 0x563621d47b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:58.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.089+0000 7f7582a00640 1 -- 172.21.15.90:0/377186270 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563621d50cb0 msgr2=0x563621d530a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:48:58.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.089+0000 7f7582000640 1 --2- 172.21.15.90:0/377186270 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563621d50cb0 0x563621d530a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:58.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.089+0000 7f757b400640 1 --2- 172.21.15.90:0/4265553887 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f75680a3de0 0x7f75680a62a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:58.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.089+0000 7f7582a00640 1 --2- 172.21.15.90:0/377186270 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563621d50cb0 0x563621d530a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:58.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.089+0000 7f7582a00640 1 -- 172.21.15.90:0/377186270 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563621a897d0 con 0x563621d480d0 2024-04-02T14:48:58.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.089+0000 7f7582000640 1 --2- 172.21.15.90:0/377186270 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563621d50cb0 0x563621d530a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:48:58.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.089+0000 7f757b400640 1 -- 172.21.15.90:0/4265553887 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 8550+0+0 (secure 0 0 0) 0x7f757c0f1eb0 con 0x7f758410e4a0 2024-04-02T14:48:58.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.090+0000 7f7582a00640 1 --2- 172.21.15.90:0/377186270 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563621d480d0 0x563621d50770 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f757400a4b0 tx=0x7f75740096d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4b95ab9c701604de server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:48:58.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.090+0000 7f7582000640 1 --2- 172.21.15.90:0/4265553887 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f75680a3de0 0x7f75680a62a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:58.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.090+0000 7f756fe00640 1 -- 172.21.15.90:0/377186270 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7574018020 con 0x563621d480d0 2024-04-02T14:48:58.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.090+0000 7f756fe00640 1 -- 172.21.15.90:0/377186270 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7574014cf0 con 0x563621d480d0 2024-04-02T14:48:58.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.090+0000 7f756fe00640 1 -- 172.21.15.90:0/377186270 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7574013e50 con 0x563621d480d0 2024-04-02T14:48:58.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.091+0000 7f758c9b2740 1 -- 172.21.15.90:0/377186270 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563621d480d0 msgr2=0x563621d50770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:48:58.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.091+0000 7f7582000640 1 --2- 172.21.15.90:0/4265553887 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f75680a3de0 0x7f75680a62a0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x563621d43070 tx=0x7f75700046c0 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:48:58.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.091+0000 7f758c9b2740 1 --2- 172.21.15.90:0/377186270 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563621d480d0 0x563621d50770 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f757400a4b0 tx=0x7f75740096d0 comp rx=0 tx=0).stop 2024-04-02T14:48:58.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.091+0000 7f758c9b2740 1 -- 172.21.15.90:0/377186270 shutdown_connections 2024-04-02T14:48:58.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.091+0000 7f758c9b2740 1 --2- 172.21.15.90:0/377186270 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563621d50cb0 0x563621d530a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:58.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.091+0000 7f758c9b2740 1 --2- 172.21.15.90:0/377186270 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563621d480d0 0x563621d50770 unknown :-1 s=CLOSED pgs=950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:58.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.091+0000 7f758c9b2740 1 --2- 172.21.15.90:0/377186270 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563621d477b0 0x563621d47b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:58.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.091+0000 7f758c9b2740 1 -- 172.21.15.90:0/377186270 >> 172.21.15.90:0/377186270 conn(0x563621d41ee0 msgr2=0x563621d422c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:48:58.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.091+0000 7f758c9b2740 1 -- 172.21.15.90:0/377186270 shutdown_connections 2024-04-02T14:48:58.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.092+0000 7f758c9b2740 1 -- 172.21.15.90:0/377186270 wait complete. 2024-04-02T14:48:58.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.092+0000 7f758c9b2740 1 -- start start 2024-04-02T14:48:58.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.092+0000 7f758c9b2740 1 Processor -- start 2024-04-02T14:48:58.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.093+0000 7f758c9b2740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563621d477b0 0x563621d4dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:58.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.093+0000 7f758c9b2740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563621d480d0 0x563621d4e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:58.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.093+0000 7f758c9b2740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563621d50cb0 0x563621d4e730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:58.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.093+0000 7f758c9b2740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x563621d53aa0 con 0x563621d50cb0 2024-04-02T14:48:58.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.093+0000 7f758c9b2740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x563621c66ac0 con 0x563621d477b0 2024-04-02T14:48:58.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.093+0000 7f758c9b2740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x563621d53c20 con 0x563621d480d0 2024-04-02T14:48:58.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.093+0000 7f758c9b2740 10 client.0 _ll_register_callbacks cb 0x563621c69180 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:48:58.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.093+0000 7f7582a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563621d477b0 0x563621d4dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:58.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.093+0000 7f7582a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563621d477b0 0x563621d4dcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:42022/0 (socket says 172.21.15.90:42022) 2024-04-02T14:48:58.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.093+0000 7f7582a00640 1 -- 172.21.15.90:0/1105609567 learned_addr learned my addr 172.21.15.90:0/1105609567 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:48:58.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.094+0000 7f7582000640 1 --2- 172.21.15.90:0/1105609567 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563621d480d0 0x563621d4e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:58.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.094+0000 7f7583400640 1 --2- 172.21.15.90:0/1105609567 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563621d50cb0 0x563621d4e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:58.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.094+0000 7f7582a00640 1 -- 172.21.15.90:0/1105609567 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563621d480d0 msgr2=0x563621d4e1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:48:58.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.094+0000 7f7582a00640 1 --2- 172.21.15.90:0/1105609567 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563621d480d0 0x563621d4e1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:58.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.094+0000 7f7582a00640 1 -- 172.21.15.90:0/1105609567 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563621d50cb0 msgr2=0x563621d4e730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:48:58.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.094+0000 7f7582a00640 1 --2- 172.21.15.90:0/1105609567 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563621d50cb0 0x563621d4e730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:58.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.094+0000 7f7582a00640 1 -- 172.21.15.90:0/1105609567 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5636219bf3f0 con 0x563621d477b0 2024-04-02T14:48:58.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.094+0000 7f7582000640 1 --2- 172.21.15.90:0/1105609567 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563621d480d0 0x563621d4e1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:48:58.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.095+0000 7f7582a00640 1 --2- 172.21.15.90:0/1105609567 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563621d477b0 0x563621d4dcb0 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f7574013a70 tx=0x7f7574013e00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b8a128b7d66eb7c3 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:48:58.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.095+0000 7f756cc00640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:48:58.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.095+0000 7f756cc00640 1 -- 172.21.15.90:0/1105609567 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7574018040 con 0x563621d477b0 2024-04-02T14:48:58.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.095+0000 7f756cc00640 1 -- 172.21.15.90:0/1105609567 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 8550+0+0 (secure 0 0 0) 0x7f7574014a20 con 0x563621d477b0 2024-04-02T14:48:58.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.095+0000 7f758c9b2740 1 -- 172.21.15.90:0/1105609567 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7574014cf0 con 0x563621d477b0 2024-04-02T14:48:58.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.096+0000 7f756cc00640 1 -- 172.21.15.90:0/1105609567 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f7574009ca0 con 0x563621d477b0 2024-04-02T14:48:58.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.096+0000 7f758c9b2740 10 client.29789 fetch_fsmap learned FSMap version 27 2024-04-02T14:48:58.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.096+0000 7f758c9b2740 1 -- 172.21.15.90:0/1105609567 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x56362197f0f0 con 0x563621d477b0 2024-04-02T14:48:58.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.096+0000 7f756cc00640 1 -- 172.21.15.90:0/1105609567 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7574014cf0 con 0x563621d477b0 2024-04-02T14:48:58.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.096+0000 7f756cc00640 1 -- 172.21.15.90:0/1105609567 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f757401e4c0 con 0x563621d477b0 2024-04-02T14:48:58.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.096+0000 7f758c9b2740 10 client.29789 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:48:58.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.096+0000 7f758c9b2740 10 client.29789 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:48:58.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.096+0000 7f758c9b2740 20 client.29789 populate_metadata read hostname 'smithi090' 2024-04-02T14:48:58.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.096+0000 7f758c9b2740 1 --2- 172.21.15.90:0/1105609567 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x563621d60ac0 0x563621d62eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:58.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.096+0000 7f758c9b2740 4 client.29789 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:48:58.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.096+0000 7f758c9b2740 1 -- 172.21.15.90:0/1105609567 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563621c4c630 con 0x563621d60ac0 2024-04-02T14:48:58.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.097+0000 7f7582000640 1 --2- 172.21.15.90:0/1105609567 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x563621d60ac0 0x563621d62eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:58.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.097+0000 7f7582000640 1 --2- 172.21.15.90:0/1105609567 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x563621d60ac0 0x563621d62eb0 crc :-1 s=READY pgs=356 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dcd07dd884ad9d12 server_cookie=2f5d114cd0c21b24 in_seq=0 out_seq=0 2024-04-02T14:48:58.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.097+0000 7f756cc00640 10 client.29789 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:48:58.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.098+0000 7f756cc00640 1 -- 172.21.15.90:0/1105609567 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x563621c4c630 con 0x563621d60ac0 2024-04-02T14:48:58.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.098+0000 7f756cc00640 10 client.29789 handle_command_reply: tid=0 2024-04-02T14:48:58.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.098+0000 7f758c9b2740 1 client.29789 shutdown 2024-04-02T14:48:58.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.098+0000 7f758c9b2740 1 -- 172.21.15.90:0/1105609567 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563621d477b0 msgr2=0x563621d4dcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:48:58.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.098+0000 7f758c9b2740 1 --2- 172.21.15.90:0/1105609567 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563621d477b0 0x563621d4dcb0 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f7574013a70 tx=0x7f7574013e00 comp rx=0 tx=0).stop 2024-04-02T14:48:58.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.098+0000 7f758c9b2740 1 -- 172.21.15.90:0/1105609567 shutdown_connections 2024-04-02T14:48:58.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.098+0000 7f758c9b2740 1 --2- 172.21.15.90:0/1105609567 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x563621d60ac0 0x563621d62eb0 crc :-1 s=READY pgs=356 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:58.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.098+0000 7f758c9b2740 1 --2- 172.21.15.90:0/1105609567 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563621d50cb0 0x563621d4e730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:58.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.098+0000 7f758c9b2740 1 --2- 172.21.15.90:0/1105609567 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563621d480d0 0x563621d4e1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:58.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.098+0000 7f758c9b2740 1 --2- 172.21.15.90:0/1105609567 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563621d477b0 0x563621d4dcb0 unknown :-1 s=CLOSED pgs=951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:58.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.098+0000 7f758c9b2740 1 -- 172.21.15.90:0/1105609567 >> 172.21.15.90:0/1105609567 conn(0x563621d41ee0 msgr2=0x563621d43c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:48:58.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.099+0000 7f756cc00640 0 client.29789 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:48:58.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.099+0000 7f758c9b2740 1 -- 172.21.15.90:0/1105609567 shutdown_connections 2024-04-02T14:48:58.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.099+0000 7f758c9b2740 1 -- 172.21.15.90:0/1105609567 wait complete. 2024-04-02T14:48:58.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.099+0000 7f758c9b2740 20 client.29789 trim_cache size 0 max 16384 2024-04-02T14:48:58.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.131+0000 7f758c9b2740 1 Processor -- start 2024-04-02T14:48:58.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.131+0000 7f758c9b2740 1 -- start start 2024-04-02T14:48:58.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.131+0000 7f758c9b2740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563621d477b0 0x563621d503e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:58.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.131+0000 7f758c9b2740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563621d480d0 0x563621d5c960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:58.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.131+0000 7f758c9b2740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563621d50cb0 0x563621d4ce50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:58.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.131+0000 7f758c9b2740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f75841bc740 con 0x563621d50cb0 2024-04-02T14:48:58.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.131+0000 7f758c9b2740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f75841be280 con 0x563621d480d0 2024-04-02T14:48:58.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.131+0000 7f758c9b2740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f75841bce10 con 0x563621d477b0 2024-04-02T14:48:58.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.132+0000 7f7582a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563621d477b0 0x563621d503e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:58.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.132+0000 7f7582a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563621d477b0 0x563621d503e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:59012/0 (socket says 172.21.15.90:59012) 2024-04-02T14:48:58.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.132+0000 7f7582a00640 1 -- 172.21.15.90:0/3401046170 learned_addr learned my addr 172.21.15.90:0/3401046170 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:48:58.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.132+0000 7f7582000640 1 --2- 172.21.15.90:0/3401046170 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563621d480d0 0x563621d5c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:58.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.132+0000 7f7583400640 1 --2- 172.21.15.90:0/3401046170 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563621d50cb0 0x563621d4ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:58.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.132+0000 7f7582a00640 1 -- 172.21.15.90:0/3401046170 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563621d480d0 msgr2=0x563621d5c960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:48:58.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.132+0000 7f7582a00640 1 --2- 172.21.15.90:0/3401046170 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563621d480d0 0x563621d5c960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:58.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.132+0000 7f7582a00640 1 -- 172.21.15.90:0/3401046170 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563621d50cb0 msgr2=0x563621d4ce50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:48:58.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.132+0000 7f7582a00640 1 --2- 172.21.15.90:0/3401046170 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563621d50cb0 0x563621d4ce50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:58.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.132+0000 7f7582a00640 1 -- 172.21.15.90:0/3401046170 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563621d532d0 con 0x563621d477b0 2024-04-02T14:48:58.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.132+0000 7f7582a00640 1 --2- 172.21.15.90:0/3401046170 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563621d477b0 0x563621d503e0 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f7574009980 tx=0x7f75740096d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=efd7ac9cb1afc2e4 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:48:58.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.133+0000 7f756f400640 1 -- 172.21.15.90:0/3401046170 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7574018040 con 0x563621d477b0 2024-04-02T14:48:58.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.133+0000 7f756f400640 1 -- 172.21.15.90:0/3401046170 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7574009ad0 con 0x563621d477b0 2024-04-02T14:48:58.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.133+0000 7f756f400640 1 -- 172.21.15.90:0/3401046170 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7574029480 con 0x563621d477b0 2024-04-02T14:48:58.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.134+0000 7f758c9b2740 1 -- 172.21.15.90:0/3401046170 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563621d477b0 msgr2=0x563621d503e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:48:58.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.134+0000 7f758c9b2740 1 --2- 172.21.15.90:0/3401046170 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563621d477b0 0x563621d503e0 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f7574009980 tx=0x7f75740096d0 comp rx=0 tx=0).stop 2024-04-02T14:48:58.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.134+0000 7f758c9b2740 1 -- 172.21.15.90:0/3401046170 shutdown_connections 2024-04-02T14:48:58.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.134+0000 7f758c9b2740 1 --2- 172.21.15.90:0/3401046170 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563621d50cb0 0x563621d4ce50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:58.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.134+0000 7f758c9b2740 1 --2- 172.21.15.90:0/3401046170 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563621d480d0 0x563621d5c960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:58.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.134+0000 7f758c9b2740 1 --2- 172.21.15.90:0/3401046170 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563621d477b0 0x563621d503e0 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:58.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.134+0000 7f758c9b2740 1 -- 172.21.15.90:0/3401046170 >> 172.21.15.90:0/3401046170 conn(0x563621d41ee0 msgr2=0x563621d4dc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:48:58.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.134+0000 7f758c9b2740 1 -- 172.21.15.90:0/3401046170 shutdown_connections 2024-04-02T14:48:58.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.134+0000 7f758c9b2740 1 -- 172.21.15.90:0/3401046170 wait complete. 2024-04-02T14:48:58.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.134+0000 7f758c9b2740 1 -- start start 2024-04-02T14:48:58.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.134+0000 7f758c9b2740 1 Processor -- start 2024-04-02T14:48:58.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.135+0000 7f758c9b2740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563621d477b0 0x563621d4a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:58.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.135+0000 7f758c9b2740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563621d480d0 0x563621d4af20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:58.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.135+0000 7f758c9b2740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563621d50cb0 0x563621d4b460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:58.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.135+0000 7f758c9b2740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f75841bdbb0 con 0x563621d477b0 2024-04-02T14:48:58.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.135+0000 7f758c9b2740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f75841bd4e0 con 0x563621d50cb0 2024-04-02T14:48:58.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.135+0000 7f758c9b2740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f7584072870 con 0x563621d480d0 2024-04-02T14:48:58.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.135+0000 7f758c9b2740 10 client.0 _ll_register_callbacks cb 0x7f7574009f00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:48:58.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.136+0000 7f7582000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563621d480d0 0x563621d4af20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:58.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.136+0000 7f7583400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563621d50cb0 0x563621d4b460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:58.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.136+0000 7f7582000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563621d480d0 0x563621d4af20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:59022/0 (socket says 172.21.15.90:59022) 2024-04-02T14:48:58.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.136+0000 7f7582000640 1 -- 172.21.15.90:0/2340174557 learned_addr learned my addr 172.21.15.90:0/2340174557 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:48:58.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.136+0000 7f7582000640 1 -- 172.21.15.90:0/2340174557 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563621d50cb0 msgr2=0x563621d4b460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:48:58.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.136+0000 7f7582000640 1 --2- 172.21.15.90:0/2340174557 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563621d50cb0 0x563621d4b460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:58.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.136+0000 7f7582000640 1 -- 172.21.15.90:0/2340174557 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563621d477b0 msgr2=0x563621d4a9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:48:58.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.136+0000 7f7582000640 1 --2- 172.21.15.90:0/2340174557 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563621d477b0 0x563621d4a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:58.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.136+0000 7f7582000640 1 -- 172.21.15.90:0/2340174557 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563621dc5cf0 con 0x563621d480d0 2024-04-02T14:48:58.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.137+0000 7f7582000640 1 --2- 172.21.15.90:0/2340174557 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563621d480d0 0x563621d4af20 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f758410dbc0 tx=0x7f7570009260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9ecb560ff3d5cdf server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:48:58.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.137+0000 7f756cc00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:48:58.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.137+0000 7f756cc00640 1 -- 172.21.15.90:0/2340174557 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f757002c4a0 con 0x563621d480d0 2024-04-02T14:48:58.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.138+0000 7f756cc00640 1 -- 172.21.15.90:0/2340174557 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 8550+0+0 (secure 0 0 0) 0x7f7570016050 con 0x563621d480d0 2024-04-02T14:48:58.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.138+0000 7f758c9b2740 1 -- 172.21.15.90:0/2340174557 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7574009ad0 con 0x563621d480d0 2024-04-02T14:48:58.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.138+0000 7f756cc00640 1 -- 172.21.15.90:0/2340174557 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f757003c050 con 0x563621d480d0 2024-04-02T14:48:58.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.138+0000 7f758c9b2740 10 client.29584 fetch_fsmap learned FSMap version 27 2024-04-02T14:48:58.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.138+0000 7f758c9b2740 1 -- 172.21.15.90:0/2340174557 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x563621dc7060 con 0x563621d480d0 2024-04-02T14:48:58.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.139+0000 7f756cc00640 1 -- 172.21.15.90:0/2340174557 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7574009ad0 con 0x563621d480d0 2024-04-02T14:48:58.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.140+0000 7f756cc00640 1 -- 172.21.15.90:0/2340174557 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f757001bce0 con 0x563621d480d0 2024-04-02T14:48:58.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.140+0000 7f758c9b2740 10 client.29584 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:48:58.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.140+0000 7f758c9b2740 10 client.29584 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:48:58.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.140+0000 7f758c9b2740 20 client.29584 populate_metadata read hostname 'smithi090' 2024-04-02T14:48:58.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.140+0000 7f758c9b2740 1 --2- 172.21.15.90:0/2340174557 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x563621dc7790 0x563621dc9bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:58.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.140+0000 7f758c9b2740 4 client.29584 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-02T14:48:58.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.140+0000 7f758c9b2740 1 -- 172.21.15.90:0/2340174557 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "scrub status"}) -- 0x563621cb8cb0 con 0x563621dc7790 2024-04-02T14:48:58.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.140+0000 7f7582a00640 1 --2- 172.21.15.90:0/2340174557 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x563621dc7790 0x563621dc9bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:58.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.141+0000 7f756cc00640 10 client.29584 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:48:58.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.141+0000 7f7582a00640 1 --2- 172.21.15.90:0/2340174557 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x563621dc7790 0x563621dc9bb0 crc :-1 s=READY pgs=357 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8ab7346058c4df3a server_cookie=300fd71b7fa7f50c in_seq=0 out_seq=0 2024-04-02T14:48:58.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.141+0000 7f756cc00640 1 -- 172.21.15.90:0/2340174557 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x563621cb8cb0 con 0x563621dc7790 2024-04-02T14:48:58.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.141+0000 7f756cc00640 10 client.29584 handle_command_reply: tid=0 2024-04-02T14:48:58.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.141+0000 7f758c9b2740 1 client.29584 shutdown 2024-04-02T14:48:58.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.141+0000 7f758c9b2740 1 -- 172.21.15.90:0/2340174557 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563621d480d0 msgr2=0x563621d4af20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:48:58.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.141+0000 7f758c9b2740 1 --2- 172.21.15.90:0/2340174557 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563621d480d0 0x563621d4af20 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f758410dbc0 tx=0x7f7570009260 comp rx=0 tx=0).stop 2024-04-02T14:48:58.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.141+0000 7f758c9b2740 1 -- 172.21.15.90:0/2340174557 shutdown_connections 2024-04-02T14:48:58.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.141+0000 7f758c9b2740 1 --2- 172.21.15.90:0/2340174557 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x563621dc7790 0x563621dc9bb0 crc :-1 s=READY pgs=357 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:58.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.141+0000 7f758c9b2740 1 --2- 172.21.15.90:0/2340174557 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563621d50cb0 0x563621d4b460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:58.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.141+0000 7f758c9b2740 1 --2- 172.21.15.90:0/2340174557 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563621d480d0 0x563621d4af20 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:58.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.141+0000 7f758c9b2740 1 --2- 172.21.15.90:0/2340174557 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x563621d477b0 0x563621d4a9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:58.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.141+0000 7f756cc00640 0 client.29584 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:48:58.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.141+0000 7f758c9b2740 1 -- 172.21.15.90:0/2340174557 >> 172.21.15.90:0/2340174557 conn(0x563621d41ee0 msgr2=0x563621d4ccd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:48:58.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.141+0000 7f758c9b2740 1 -- 172.21.15.90:0/2340174557 shutdown_connections 2024-04-02T14:48:58.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.141+0000 7f758c9b2740 1 -- 172.21.15.90:0/2340174557 wait complete. 2024-04-02T14:48:58.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.141+0000 7f758c9b2740 20 client.29584 trim_cache size 0 max 16384 2024-04-02T14:48:58.137 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-02T14:48:58.138 INFO:teuthology.orchestra.run.smithi090.stdout: "status": "no active scrubs running", 2024-04-02T14:48:58.138 INFO:teuthology.orchestra.run.smithi090.stdout: "scrubs": {} 2024-04-02T14:48:58.138 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-02T14:48:58.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.142+0000 7f7589c00640 1 -- 172.21.15.90:0/4265553887 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f75680a3de0 msgr2=0x7f75680a62a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:48:58.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.142+0000 7f7589c00640 1 --2- 172.21.15.90:0/4265553887 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f75680a3de0 0x7f75680a62a0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x563621d43070 tx=0x7f75700046c0 comp rx=0 tx=0).stop 2024-04-02T14:48:58.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.142+0000 7f7589c00640 1 -- 172.21.15.90:0/4265553887 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f758410e4a0 msgr2=0x7f758410cbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:48:58.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.142+0000 7f7589c00640 1 --2- 172.21.15.90:0/4265553887 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f758410e4a0 0x7f758410cbb0 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7f757c02c180 tx=0x7f757c0045e0 comp rx=0 tx=0).stop 2024-04-02T14:48:58.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.142+0000 7f7589c00640 1 -- 172.21.15.90:0/4265553887 shutdown_connections 2024-04-02T14:48:58.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.142+0000 7f7589c00640 1 --2- 172.21.15.90:0/4265553887 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f75680a3de0 0x7f75680a62a0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:58.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.142+0000 7f7589c00640 1 --2- 172.21.15.90:0/4265553887 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f758410e4a0 0x7f758410cbb0 unknown :-1 s=CLOSED pgs=949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:58.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.142+0000 7f7589c00640 1 --2- 172.21.15.90:0/4265553887 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7584069600 0x7f758410c670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:58.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.142+0000 7f7589c00640 1 --2- 172.21.15.90:0/4265553887 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7584068ce0 0x7f758410c130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:58.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.142+0000 7f7589c00640 1 -- 172.21.15.90:0/4265553887 >> 172.21.15.90:0/4265553887 conn(0x7f7584077b00 msgr2=0x7f758410fa50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:48:58.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.142+0000 7f7589c00640 1 -- 172.21.15.90:0/4265553887 shutdown_connections 2024-04-02T14:48:58.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.142+0000 7f7589c00640 1 -- 172.21.15.90:0/4265553887 wait complete. 2024-04-02T14:48:58.156 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-02T14:48:58.156 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T14:48:58.429 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.435+0000 7fa72a000640 1 -- 172.21.15.90:0/3997093618 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa7241048b0 msgr2=0x7fa7241088f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:48:58.429 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.435+0000 7fa72a000640 1 --2- 172.21.15.90:0/3997093618 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa7241048b0 0x7fa7241088f0 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7fa710009860 tx=0x7fa71002bb70 comp rx=0 tx=0).stop 2024-04-02T14:48:58.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.436+0000 7fa72a000640 1 -- 172.21.15.90:0/3997093618 shutdown_connections 2024-04-02T14:48:58.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.436+0000 7fa72a000640 1 --2- 172.21.15.90:0/3997093618 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa724108fd0 0x7fa724110ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:58.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.436+0000 7fa72a000640 1 --2- 172.21.15.90:0/3997093618 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa7241048b0 0x7fa7241088f0 unknown :-1 s=CLOSED pgs=952 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:58.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.436+0000 7fa72a000640 1 --2- 172.21.15.90:0/3997093618 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa724103f00 0x7fa7241042e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:58.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.436+0000 7fa72a000640 1 -- 172.21.15.90:0/3997093618 >> 172.21.15.90:0/3997093618 conn(0x7fa7240ff760 msgr2=0x7fa724101b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:48:58.431 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:48:58 smithi090 ceph-mon[31674]: pgmap v439: 97 pgs: 97 active+clean; 4.6 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 8.2 MiB/s wr, 708 op/s 2024-04-02T14:48:58.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.437+0000 7fa72a000640 1 -- 172.21.15.90:0/3997093618 shutdown_connections 2024-04-02T14:48:58.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.437+0000 7fa72a000640 1 -- 172.21.15.90:0/3997093618 wait complete. 2024-04-02T14:48:58.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.439+0000 7fa72a000640 1 Processor -- start 2024-04-02T14:48:58.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.439+0000 7fa72a000640 1 -- start start 2024-04-02T14:48:58.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.440+0000 7fa72a000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa724103f00 0x7fa724110b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:58.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.440+0000 7fa72a000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa724108fd0 0x7fa72410dc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:58.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.440+0000 7fa72a000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa72410e320 0x7fa7241a0560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:58.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.440+0000 7fa72a000640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fa724114040 con 0x7fa72410e320 2024-04-02T14:48:58.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.440+0000 7fa72a000640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fa724113ec0 con 0x7fa724108fd0 2024-04-02T14:48:58.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.440+0000 7fa72a000640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fa7241141c0 con 0x7fa724103f00 2024-04-02T14:48:58.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.440+0000 7fa723400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa724103f00 0x7fa724110b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:58.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.440+0000 7fa722a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa724108fd0 0x7fa72410dc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:58.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.440+0000 7fa723400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa724103f00 0x7fa724110b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:59042/0 (socket says 172.21.15.90:59042) 2024-04-02T14:48:58.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.440+0000 7fa723400640 1 -- 172.21.15.90:0/314532437 learned_addr learned my addr 172.21.15.90:0/314532437 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:48:58.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.440+0000 7fa723e00640 1 --2- 172.21.15.90:0/314532437 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa72410e320 0x7fa7241a0560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:58.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.441+0000 7fa722a00640 1 -- 172.21.15.90:0/314532437 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa724103f00 msgr2=0x7fa724110b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:48:58.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.441+0000 7fa722a00640 1 --2- 172.21.15.90:0/314532437 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa724103f00 0x7fa724110b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:58.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.441+0000 7fa722a00640 1 -- 172.21.15.90:0/314532437 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa72410e320 msgr2=0x7fa7241a0560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:48:58.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.441+0000 7fa722a00640 1 --2- 172.21.15.90:0/314532437 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa72410e320 0x7fa7241a0560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:58.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.441+0000 7fa722a00640 1 -- 172.21.15.90:0/314532437 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7241a0aa0 con 0x7fa724108fd0 2024-04-02T14:48:58.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.441+0000 7fa723400640 1 --2- 172.21.15.90:0/314532437 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa724103f00 0x7fa724110b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:48:58.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.441+0000 7fa723e00640 1 --2- 172.21.15.90:0/314532437 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa72410e320 0x7fa7241a0560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:48:58.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.441+0000 7fa722a00640 1 --2- 172.21.15.90:0/314532437 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa724108fd0 0x7fa72410dc80 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7fa71002c0b0 tx=0x7fa7100045b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:48:58.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.442+0000 7fa71be00640 1 -- 172.21.15.90:0/314532437 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa710041020 con 0x7fa724108fd0 2024-04-02T14:48:58.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.442+0000 7fa71be00640 1 -- 172.21.15.90:0/314532437 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa71002c9f0 con 0x7fa724108fd0 2024-04-02T14:48:58.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.442+0000 7fa71be00640 1 -- 172.21.15.90:0/314532437 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa710039040 con 0x7fa724108fd0 2024-04-02T14:48:58.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.442+0000 7fa72a000640 1 -- 172.21.15.90:0/314532437 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7241a0cd0 con 0x7fa724108fd0 2024-04-02T14:48:58.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.442+0000 7fa72a000640 1 -- 172.21.15.90:0/314532437 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7241a1170 con 0x7fa724108fd0 2024-04-02T14:48:58.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.444+0000 7fa72a000640 1 -- 172.21.15.90:0/314532437 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa7241053d0 con 0x7fa724108fd0 2024-04-02T14:48:58.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.446+0000 7fa71be00640 1 -- 172.21.15.90:0/314532437 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fa710044020 con 0x7fa724108fd0 2024-04-02T14:48:58.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.447+0000 7fa71be00640 1 --2- 172.21.15.90:0/314532437 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fa7080a3da0 0x7fa7080a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:58.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.448+0000 7fa71be00640 1 -- 172.21.15.90:0/314532437 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 8550+0+0 (secure 0 0 0) 0x7fa7100f1d00 con 0x7fa724108fd0 2024-04-02T14:48:58.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.448+0000 7fa723400640 1 --2- 172.21.15.90:0/314532437 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fa7080a3da0 0x7fa7080a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:58.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.448+0000 7fa723400640 1 --2- 172.21.15.90:0/314532437 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fa7080a3da0 0x7fa7080a6260 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fa7140096c0 tx=0x7fa714009260 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:48:58.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.452+0000 7fa71be00640 1 -- 172.21.15.90:0/314532437 <== mon.2 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7fa71004b020 con 0x7fa724108fd0 2024-04-02T14:48:58.603 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:48:58 smithi092 ceph-mon[36497]: pgmap v439: 97 pgs: 97 active+clean; 4.6 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 8.2 MiB/s wr, 708 op/s 2024-04-02T14:48:58.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:48:58 smithi177 ceph-mon[36431]: pgmap v439: 97 pgs: 97 active+clean; 4.6 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 8.2 MiB/s wr, 708 op/s 2024-04-02T14:48:58.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.841+0000 7fa72a000640 1 -- 172.21.15.90:0/314532437 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa72410f110 con 0x7fa724108fd0 2024-04-02T14:48:58.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.844+0000 7fa71be00640 1 -- 172.21.15.90:0/314532437 <== mon.2 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11927 (secure 0 0 0) 0x7fa7100be490 con 0x7fa724108fd0 2024-04-02T14:48:58.839 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-02T14:48:58.839 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14412,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6837/2852160309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6836","nonce":2852160309},{"type":"v1","addr":"172.21.15.90:6837","nonce":2852160309}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14478,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6841/1919376858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6840","nonce":1919376858},{"type":"v1","addr":"172.21.15.90:6841","nonce":1919376858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-02T14:32:22.319083+0000","modified":"2024-04-02T14:34:52.266152+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24590,"mds_1":24560,"mds_2":24547,"mds_3":24539,"mds_4":24523},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24454":{"gid":24454,"name":"l","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6833/1564319431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":1564319431},{"type":"v1","addr":"172.21.15.177:6833","nonce":1564319431}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.92:6833/1570337372","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":1570337372},{"type":"v1","addr":"172.21.15.92:6833","nonce":1570337372}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"j","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.90:6835/1277124891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6834","nonce":1277124891},{"type":"v1","addr":"172.21.15.90:6835","nonce":1277124891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.92:6835/860721164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":860721164},{"type":"v1","addr":"172.21.15.92:6835","nonce":860721164}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24511":{"gid":24511,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6835/3177244988","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":3177244988},{"type":"v1","addr":"172.21.15.177:6835","nonce":3177244988}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"c","rank":4,"incarnation":25,"state":"up:active","state_seq":9,"addr":"172.21.15.177:6837/4223414858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":4223414858},{"type":"v1","addr":"172.21.15.177:6837","nonce":4223414858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24539":{"gid":24539,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":8,"addr":"172.21.15.92:6837/1270635939","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":1270635939},{"type":"v1","addr":"172.21.15.92:6837","nonce":1270635939}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24547":{"gid":24547,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":7,"addr":"172.21.15.177:6839/664895663","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":664895663},{"type":"v1","addr":"172.21.15.177:6839","nonce":664895663}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"g","rank":1,"incarnation":19,"state":"up:active","state_seq":5,"addr":"172.21.15.90:6839/1498915870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6838","nonce":1498915870},{"type":"v1","addr":"172.21.15.90:6839","nonce":1498915870}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24590":{"gid":24590,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.92:6839/1054888219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":1054888219},{"type":"v1","addr":"172.21.15.92:6839","nonce":1054888219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24590,"qdb_cluster":[24547,24590,24523,24539,24560]},"id":1}]} 2024-04-02T14:48:58.839 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 27 2024-04-02T14:48:58.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.847+0000 7fa72a000640 1 -- 172.21.15.90:0/314532437 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fa7080a3da0 msgr2=0x7fa7080a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:48:58.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.847+0000 7fa72a000640 1 --2- 172.21.15.90:0/314532437 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fa7080a3da0 0x7fa7080a6260 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fa7140096c0 tx=0x7fa714009260 comp rx=0 tx=0).stop 2024-04-02T14:48:58.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.848+0000 7fa72a000640 1 -- 172.21.15.90:0/314532437 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa724108fd0 msgr2=0x7fa72410dc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:48:58.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.848+0000 7fa72a000640 1 --2- 172.21.15.90:0/314532437 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa724108fd0 0x7fa72410dc80 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7fa71002c0b0 tx=0x7fa7100045b0 comp rx=0 tx=0).stop 2024-04-02T14:48:58.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.848+0000 7fa72a000640 1 -- 172.21.15.90:0/314532437 shutdown_connections 2024-04-02T14:48:58.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.848+0000 7fa72a000640 1 --2- 172.21.15.90:0/314532437 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fa7080a3da0 0x7fa7080a6260 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:58.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.848+0000 7fa72a000640 1 --2- 172.21.15.90:0/314532437 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa72410e320 0x7fa7241a0560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:58.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.848+0000 7fa72a000640 1 --2- 172.21.15.90:0/314532437 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa724108fd0 0x7fa72410dc80 unknown :-1 s=CLOSED pgs=953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:58.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.848+0000 7fa72a000640 1 --2- 172.21.15.90:0/314532437 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa724103f00 0x7fa724110b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:58.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.848+0000 7fa72a000640 1 -- 172.21.15.90:0/314532437 >> 172.21.15.90:0/314532437 conn(0x7fa7240ff760 msgr2=0x7fa724100a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:48:58.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.848+0000 7fa72a000640 1 -- 172.21.15.90:0/314532437 shutdown_connections 2024-04-02T14:48:58.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:58.848+0000 7fa72a000640 1 -- 172.21.15.90:0/314532437 wait complete. 2024-04-02T14:48:58.857 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.863+0000 7fbcd0e00640 1 -- 172.21.15.90:0/2269352038 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbccc109010 msgr2=0x7fbccc110be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:48:58.858 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.863+0000 7fbcc8a00640 1 -- 172.21.15.90:0/2269352038 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcc4032990 con 0x7fbccc109010 2024-04-02T14:48:58.859 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.864+0000 7fbcd0e00640 1 --2- 172.21.15.90:0/2269352038 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbccc109010 0x7fbccc110be0 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7fbcc4009ca0 tx=0x7fbcc402c100 comp rx=0 tx=0).stop 2024-04-02T14:48:58.859 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.864+0000 7fbcd0e00640 1 -- 172.21.15.90:0/2269352038 shutdown_connections 2024-04-02T14:48:58.859 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.864+0000 7fbcd0e00640 1 --2- 172.21.15.90:0/2269352038 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbccc109010 0x7fbccc110be0 unknown :-1 s=CLOSED pgs=954 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:58.859 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.864+0000 7fbcd0e00640 1 --2- 172.21.15.90:0/2269352038 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbccc1048f0 0x7fbccc108930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:58.859 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.864+0000 7fbcd0e00640 1 --2- 172.21.15.90:0/2269352038 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbccc103f40 0x7fbccc104320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:58.859 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.865+0000 7fbcd0e00640 1 -- 172.21.15.90:0/2269352038 >> 172.21.15.90:0/2269352038 conn(0x7fbccc0ff7c0 msgr2=0x7fbccc101be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:48:58.859 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.865+0000 7fbcd0e00640 1 -- 172.21.15.90:0/2269352038 shutdown_connections 2024-04-02T14:48:58.859 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.865+0000 7fbcd0e00640 1 -- 172.21.15.90:0/2269352038 wait complete. 2024-04-02T14:48:58.860 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.866+0000 7fbcd0e00640 1 Processor -- start 2024-04-02T14:48:58.860 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.866+0000 7fbcd0e00640 1 -- start start 2024-04-02T14:48:58.861 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.867+0000 7fbcd0e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbccc103f40 0x7fbccc10f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:58.861 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.867+0000 7fbcd0e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbccc109010 0x7fbccc10f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:58.861 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.867+0000 7fbcd0e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbccc10fd80 0x7fbccc1aa4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:58.861 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.867+0000 7fbcd0e00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fbccc077120 con 0x7fbccc10fd80 2024-04-02T14:48:58.862 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.867+0000 7fbcd0e00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fbccc076fa0 con 0x7fbccc109010 2024-04-02T14:48:58.862 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.867+0000 7fbcd0e00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fbccc0772a0 con 0x7fbccc103f40 2024-04-02T14:48:58.862 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.867+0000 7fbcc9400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbccc109010 0x7fbccc10f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:58.862 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.867+0000 7fbcc9400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbccc109010 0x7fbccc10f6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:42084/0 (socket says 172.21.15.90:42084) 2024-04-02T14:48:58.862 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.867+0000 7fbcc9400640 1 -- 172.21.15.90:0/2724020256 learned_addr learned my addr 172.21.15.90:0/2724020256 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:48:58.862 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.867+0000 7fbcc9e00640 1 --2- 172.21.15.90:0/2724020256 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbccc103f40 0x7fbccc10f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:58.863 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.868+0000 7fbcc9400640 1 -- 172.21.15.90:0/2724020256 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbccc103f40 msgr2=0x7fbccc10f1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:48:58.863 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.868+0000 7fbcc9400640 1 --2- 172.21.15.90:0/2724020256 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbccc103f40 0x7fbccc10f1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:58.863 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.868+0000 7fbcc9400640 1 -- 172.21.15.90:0/2724020256 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbccc10fd80 msgr2=0x7fbccc1aa4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:48:58.863 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.868+0000 7fbcc9400640 1 --2- 172.21.15.90:0/2724020256 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbccc10fd80 0x7fbccc1aa4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:58.863 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.868+0000 7fbcc9400640 1 -- 172.21.15.90:0/2724020256 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbccc1aad50 con 0x7fbccc109010 2024-04-02T14:48:58.863 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.868+0000 7fbcc9400640 1 --2- 172.21.15.90:0/2724020256 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbccc109010 0x7fbccc10f6e0 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7fbcb800d770 tx=0x7fbcb800dc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:48:58.863 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.868+0000 7fbcc2a00640 1 -- 172.21.15.90:0/2724020256 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcb8017070 con 0x7fbccc109010 2024-04-02T14:48:58.863 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.868+0000 7fbcc2a00640 1 -- 172.21.15.90:0/2724020256 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbcb8009c30 con 0x7fbccc109010 2024-04-02T14:48:58.863 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.868+0000 7fbcc2a00640 1 -- 172.21.15.90:0/2724020256 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcb800f620 con 0x7fbccc109010 2024-04-02T14:48:58.864 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.868+0000 7fbcd0e00640 1 -- 172.21.15.90:0/2724020256 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbccc1aafe0 con 0x7fbccc109010 2024-04-02T14:48:58.864 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.869+0000 7fbcd0e00640 1 -- 172.21.15.90:0/2724020256 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbccc1ab4b0 con 0x7fbccc109010 2024-04-02T14:48:58.865 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-04-02T14:48:58.871 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.873+0000 7fbcd3b76740 1 Processor -- start 2024-04-02T14:48:58.871 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.874+0000 7fbcc2a00640 1 -- 172.21.15.90:0/2724020256 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fbcb801a070 con 0x7fbccc109010 2024-04-02T14:48:58.871 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.874+0000 7fbcd3b76740 1 -- start start 2024-04-02T14:48:58.871 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.874+0000 7fbcd3b76740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5602f1889eb0 0x5602f188a290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:58.872 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.874+0000 7fbcd3b76740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5602f188a7d0 0x5602f1892e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:58.872 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.874+0000 7fbcd3b76740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5602f18933b0 0x5602f18957a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:58.872 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.874+0000 7fbcd3b76740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5602f17eed70 con 0x5602f18933b0 2024-04-02T14:48:58.872 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.874+0000 7fbcd3b76740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x5602f17972a0 con 0x5602f1889eb0 2024-04-02T14:48:58.872 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.874+0000 7fbcd3b76740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5602f17ead20 con 0x5602f188a7d0 2024-04-02T14:48:58.872 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.875+0000 7fbcca800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5602f1889eb0 0x5602f188a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:58.872 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.875+0000 7fbcca800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5602f1889eb0 0x5602f188a290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:42092/0 (socket says 172.21.15.90:42092) 2024-04-02T14:48:58.872 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.875+0000 7fbcca800640 1 -- 172.21.15.90:0/2010077242 learned_addr learned my addr 172.21.15.90:0/2010077242 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:48:58.872 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.875+0000 7fbcc9400640 1 --2- 172.21.15.90:0/2010077242 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5602f18933b0 0x5602f18957a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:58.872 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.875+0000 7fbcc9e00640 1 --2- 172.21.15.90:0/2010077242 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5602f188a7d0 0x5602f1892e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:58.872 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.875+0000 7fbcca800640 1 -- 172.21.15.90:0/2010077242 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5602f188a7d0 msgr2=0x5602f1892e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:48:58.872 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.875+0000 7fbcca800640 1 --2- 172.21.15.90:0/2010077242 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5602f188a7d0 0x5602f1892e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:58.872 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.875+0000 7fbcca800640 1 -- 172.21.15.90:0/2010077242 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5602f18933b0 msgr2=0x5602f18957a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:48:58.873 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.875+0000 7fbcca800640 1 --2- 172.21.15.90:0/2010077242 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5602f18933b0 0x5602f18957a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:58.873 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.875+0000 7fbcca800640 1 -- 172.21.15.90:0/2010077242 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5602f17ab840 con 0x5602f1889eb0 2024-04-02T14:48:58.873 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.875+0000 7fbcc9400640 1 --2- 172.21.15.90:0/2010077242 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5602f18933b0 0x5602f18957a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:48:58.873 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.875+0000 7fbcc9e00640 1 --2- 172.21.15.90:0/2010077242 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5602f188a7d0 0x5602f1892e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:48:58.873 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.876+0000 7fbcca800640 1 --2- 172.21.15.90:0/2010077242 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5602f1889eb0 0x5602f188a290 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7fbcc4005cc0 tx=0x7fbcc4005ef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=69f09c9cbfb3930 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:48:58.873 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.876+0000 7fbcb7400640 1 -- 172.21.15.90:0/2010077242 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcc4041020 con 0x5602f1889eb0 2024-04-02T14:48:58.873 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.876+0000 7fbcb7400640 1 -- 172.21.15.90:0/2010077242 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbcc4032ce0 con 0x5602f1889eb0 2024-04-02T14:48:58.873 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.876+0000 7fbcb7400640 1 -- 172.21.15.90:0/2010077242 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcc4039040 con 0x5602f1889eb0 2024-04-02T14:48:58.873 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.876+0000 7fbcc2a00640 1 --2- 172.21.15.90:0/2724020256 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fbcb00a3e90 0x7fbcb00a6350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:58.873 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.876+0000 7fbcc2a00640 1 -- 172.21.15.90:0/2724020256 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 8550+0+0 (secure 0 0 0) 0x7fbcb80d0250 con 0x7fbccc109010 2024-04-02T14:48:58.873 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.877+0000 7fbcc9e00640 1 --2- 172.21.15.90:0/2724020256 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fbcb00a3e90 0x7fbcb00a6350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:58.873 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.877+0000 7fbcd3b76740 1 -- 172.21.15.90:0/2010077242 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5602f1889eb0 msgr2=0x5602f188a290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:48:58.873 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.877+0000 7fbcd3b76740 1 --2- 172.21.15.90:0/2010077242 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5602f1889eb0 0x5602f188a290 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7fbcc4005cc0 tx=0x7fbcc4005ef0 comp rx=0 tx=0).stop 2024-04-02T14:48:58.873 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.877+0000 7fbcd3b76740 1 -- 172.21.15.90:0/2010077242 shutdown_connections 2024-04-02T14:48:58.873 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.877+0000 7fbcd3b76740 1 --2- 172.21.15.90:0/2010077242 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5602f18933b0 0x5602f18957a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:58.874 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.877+0000 7fbcd3b76740 1 --2- 172.21.15.90:0/2010077242 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5602f188a7d0 0x5602f1892e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:58.874 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.877+0000 7fbcd3b76740 1 --2- 172.21.15.90:0/2010077242 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5602f1889eb0 0x5602f188a290 unknown :-1 s=CLOSED pgs=956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:58.874 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.877+0000 7fbcd3b76740 1 -- 172.21.15.90:0/2010077242 >> 172.21.15.90:0/2010077242 conn(0x5602f18845e0 msgr2=0x5602f18849c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:48:58.876 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.878+0000 7fbcd3b76740 1 -- 172.21.15.90:0/2010077242 shutdown_connections 2024-04-02T14:48:58.876 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.878+0000 7fbcd3b76740 1 -- 172.21.15.90:0/2010077242 wait complete. 2024-04-02T14:48:58.876 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.878+0000 7fbcd3b76740 1 -- start start 2024-04-02T14:48:58.876 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.878+0000 7fbcd3b76740 1 Processor -- start 2024-04-02T14:48:58.876 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.879+0000 7fbcc9e00640 1 --2- 172.21.15.90:0/2724020256 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fbcb00a3e90 0x7fbcb00a6350 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x5602f1889980 tx=0x7fbcbc005f90 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:48:58.876 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.879+0000 7fbcd3b76740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5602f1889eb0 0x5602f18903e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:58.876 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.879+0000 7fbcd3b76740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5602f188a7d0 0x5602f1890920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:58.876 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.879+0000 7fbcd3b76740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5602f18933b0 0x5602f1890e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:58.876 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.879+0000 7fbcd3b76740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5602f1896140 con 0x5602f188a7d0 2024-04-02T14:48:58.876 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.879+0000 7fbcd3b76740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x5602f17aa660 con 0x5602f18933b0 2024-04-02T14:48:58.876 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.879+0000 7fbcd3b76740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5602f18962c0 con 0x5602f1889eb0 2024-04-02T14:48:58.877 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.879+0000 7fbcd3b76740 10 client.0 _ll_register_callbacks cb 0x5602f17aa050 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:48:58.877 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.879+0000 7fbcc9e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5602f1889eb0 0x5602f18903e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:58.877 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.879+0000 7fbcc9e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5602f1889eb0 0x5602f18903e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:59066/0 (socket says 172.21.15.90:59066) 2024-04-02T14:48:58.877 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.879+0000 7fbcc9e00640 1 -- 172.21.15.90:0/3842012574 learned_addr learned my addr 172.21.15.90:0/3842012574 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:48:58.877 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.879+0000 7fbcc9400640 1 --2- 172.21.15.90:0/3842012574 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5602f188a7d0 0x5602f1890920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:58.877 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.880+0000 7fbcca800640 1 --2- 172.21.15.90:0/3842012574 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5602f18933b0 0x5602f1890e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:58.877 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.880+0000 7fbcca800640 1 -- 172.21.15.90:0/3842012574 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5602f1889eb0 msgr2=0x5602f18903e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:48:58.877 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.880+0000 7fbcca800640 1 --2- 172.21.15.90:0/3842012574 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5602f1889eb0 0x5602f18903e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:58.877 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.880+0000 7fbcca800640 1 -- 172.21.15.90:0/3842012574 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5602f188a7d0 msgr2=0x5602f1890920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:48:58.877 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.880+0000 7fbcca800640 1 --2- 172.21.15.90:0/3842012574 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5602f188a7d0 0x5602f1890920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:58.877 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.880+0000 7fbcca800640 1 -- 172.21.15.90:0/3842012574 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5602f17b9e80 con 0x5602f18933b0 2024-04-02T14:48:58.877 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.881+0000 7fbcca800640 1 --2- 172.21.15.90:0/3842012574 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5602f18933b0 0x5602f1890e60 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7fbcc4009bf0 tx=0x7fbcc4005de0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6ec4ceb6baacde1e server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:48:58.877 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.881+0000 7fbcabe00640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:48:58.877 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.881+0000 7fbcabe00640 1 -- 172.21.15.90:0/3842012574 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcc4039050 con 0x5602f18933b0 2024-04-02T14:48:58.878 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.881+0000 7fbcabe00640 1 -- 172.21.15.90:0/3842012574 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 8550+0+0 (secure 0 0 0) 0x7fbcc40040a0 con 0x5602f18933b0 2024-04-02T14:48:58.878 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.882+0000 7fbcd3b76740 1 -- 172.21.15.90:0/3842012574 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbcc4032ce0 con 0x5602f18933b0 2024-04-02T14:48:58.878 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.882+0000 7fbcabe00640 1 -- 172.21.15.90:0/3842012574 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fbcc4032e90 con 0x5602f18933b0 2024-04-02T14:48:58.878 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.882+0000 7fbcd3b76740 10 client.29813 fetch_fsmap learned FSMap version 27 2024-04-02T14:48:58.878 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.882+0000 7fbcd3b76740 1 -- 172.21.15.90:0/3842012574 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5602f1503210 con 0x5602f18933b0 2024-04-02T14:48:58.879 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.883+0000 7fbcabe00640 1 -- 172.21.15.90:0/3842012574 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbcc4032ce0 con 0x5602f18933b0 2024-04-02T14:48:58.879 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.883+0000 7fbcabe00640 1 -- 172.21.15.90:0/3842012574 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fbcc403c030 con 0x5602f18933b0 2024-04-02T14:48:58.879 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.884+0000 7fbcd3b76740 10 client.29813 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:48:58.879 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.884+0000 7fbcd3b76740 10 client.29813 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-02T14:48:58.879 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.884+0000 7fbcd3b76740 20 client.29813 populate_metadata read hostname 'smithi090' 2024-04-02T14:48:58.879 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.884+0000 7fbcd3b76740 1 --2- 172.21.15.90:0/3842012574 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5602f18a34d0 0x5602f18a58c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:58.879 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.884+0000 7fbcd3b76740 4 client.29813 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:48:58.879 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.884+0000 7fbcd3b76740 1 -- 172.21.15.90:0/3842012574 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5602f179ecd0 con 0x5602f18a34d0 2024-04-02T14:48:58.879 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.885+0000 7fbcc9e00640 1 --2- 172.21.15.90:0/3842012574 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5602f18a34d0 0x5602f18a58c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:58.880 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.885+0000 7fbcc9e00640 1 --2- 172.21.15.90:0/3842012574 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5602f18a34d0 0x5602f18a58c0 crc :-1 s=READY pgs=358 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=23d1617f7f6b1565 server_cookie=12e03137adfdf58e in_seq=0 out_seq=0 2024-04-02T14:48:58.880 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.885+0000 7fbcabe00640 10 client.29813 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:48:58.880 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.886+0000 7fbcabe00640 1 -- 172.21.15.90:0/3842012574 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5602f179ecd0 con 0x5602f18a34d0 2024-04-02T14:48:58.880 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.886+0000 7fbcabe00640 10 client.29813 handle_command_reply: tid=0 2024-04-02T14:48:58.881 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.886+0000 7fbcd3b76740 1 client.29813 shutdown 2024-04-02T14:48:58.881 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.886+0000 7fbcd3b76740 1 -- 172.21.15.90:0/3842012574 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5602f18933b0 msgr2=0x5602f1890e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:48:58.881 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.886+0000 7fbcd3b76740 1 --2- 172.21.15.90:0/3842012574 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5602f18933b0 0x5602f1890e60 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7fbcc4009bf0 tx=0x7fbcc4005de0 comp rx=0 tx=0).stop 2024-04-02T14:48:58.881 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.887+0000 7fbcd3b76740 1 -- 172.21.15.90:0/3842012574 shutdown_connections 2024-04-02T14:48:58.881 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.887+0000 7fbcd3b76740 1 --2- 172.21.15.90:0/3842012574 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5602f18a34d0 0x5602f18a58c0 crc :-1 s=READY pgs=358 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:58.882 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.887+0000 7fbcd3b76740 1 --2- 172.21.15.90:0/3842012574 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5602f18933b0 0x5602f1890e60 unknown :-1 s=CLOSED pgs=957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:58.882 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.887+0000 7fbcd3b76740 1 --2- 172.21.15.90:0/3842012574 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5602f188a7d0 0x5602f1890920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:58.882 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.887+0000 7fbcd3b76740 1 --2- 172.21.15.90:0/3842012574 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5602f1889eb0 0x5602f18903e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:58.882 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.887+0000 7fbcabe00640 0 client.29813 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:48:58.882 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.887+0000 7fbcd3b76740 1 -- 172.21.15.90:0/3842012574 >> 172.21.15.90:0/3842012574 conn(0x5602f18845e0 msgr2=0x5602f1886330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:48:58.882 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.887+0000 7fbcd3b76740 1 -- 172.21.15.90:0/3842012574 shutdown_connections 2024-04-02T14:48:58.882 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.887+0000 7fbcd3b76740 1 -- 172.21.15.90:0/3842012574 wait complete. 2024-04-02T14:48:58.882 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.887+0000 7fbcd3b76740 20 client.29813 trim_cache size 0 max 16384 2024-04-02T14:48:58.906 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.911+0000 7fbcd3b76740 1 Processor -- start 2024-04-02T14:48:58.906 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.912+0000 7fbcd3b76740 1 -- start start 2024-04-02T14:48:58.907 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.913+0000 7fbcd3b76740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5602f1889eb0 0x5602f1892b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:58.907 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.913+0000 7fbcd3b76740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5602f188a7d0 0x5602f189f000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:58.907 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.913+0000 7fbcd3b76740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5602f18933b0 0x5602f1890ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:58.908 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.913+0000 7fbcd3b76740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fbccc1b3b70 con 0x5602f1889eb0 2024-04-02T14:48:58.908 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.913+0000 7fbcd3b76740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fbccc1b56b0 con 0x5602f188a7d0 2024-04-02T14:48:58.908 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.914+0000 7fbcd3b76740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fbccc1b4240 con 0x5602f18933b0 2024-04-02T14:48:58.908 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.914+0000 7fbcca800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5602f18933b0 0x5602f1890ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:58.908 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.914+0000 7fbcc9400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5602f188a7d0 0x5602f189f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:58.908 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.914+0000 7fbcca800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5602f18933b0 0x5602f1890ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:59076/0 (socket says 172.21.15.90:59076) 2024-04-02T14:48:58.908 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.914+0000 7fbcca800640 1 -- 172.21.15.90:0/3934657454 learned_addr learned my addr 172.21.15.90:0/3934657454 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:48:58.909 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.914+0000 7fbcc9400640 1 -- 172.21.15.90:0/3934657454 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5602f18933b0 msgr2=0x5602f1890ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:48:58.909 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.914+0000 7fbcc9400640 1 --2- 172.21.15.90:0/3934657454 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5602f18933b0 0x5602f1890ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:58.909 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.914+0000 7fbcc9400640 1 -- 172.21.15.90:0/3934657454 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5602f1889eb0 msgr2=0x5602f1892b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:48:58.909 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.914+0000 7fbcc9400640 1 --2- 172.21.15.90:0/3934657454 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5602f1889eb0 0x5602f1892b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:58.909 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.914+0000 7fbcc9400640 1 -- 172.21.15.90:0/3934657454 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5602f188f3b0 con 0x5602f188a7d0 2024-04-02T14:48:58.910 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.916+0000 7fbcc9400640 1 --2- 172.21.15.90:0/3934657454 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5602f188a7d0 0x5602f189f000 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7fbcb80cd980 tx=0x7fbcb809bd10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1d2c29a0081147d3 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:48:58.911 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.916+0000 7fbcb6a00640 1 -- 172.21.15.90:0/3934657454 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcb8021040 con 0x5602f188a7d0 2024-04-02T14:48:58.911 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.916+0000 7fbcb6a00640 1 -- 172.21.15.90:0/3934657454 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbcb809c8d0 con 0x5602f188a7d0 2024-04-02T14:48:58.911 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.916+0000 7fbcb6a00640 1 -- 172.21.15.90:0/3934657454 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcb8098840 con 0x5602f188a7d0 2024-04-02T14:48:58.912 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.917+0000 7fbcd3b76740 1 -- 172.21.15.90:0/3934657454 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5602f188a7d0 msgr2=0x5602f189f000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:48:58.912 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.917+0000 7fbcd3b76740 1 --2- 172.21.15.90:0/3934657454 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5602f188a7d0 0x5602f189f000 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7fbcb80cd980 tx=0x7fbcb809bd10 comp rx=0 tx=0).stop 2024-04-02T14:48:58.912 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.918+0000 7fbcd3b76740 1 -- 172.21.15.90:0/3934657454 shutdown_connections 2024-04-02T14:48:58.912 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.918+0000 7fbcd3b76740 1 --2- 172.21.15.90:0/3934657454 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5602f18933b0 0x5602f1890ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:58.912 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.918+0000 7fbcd3b76740 1 --2- 172.21.15.90:0/3934657454 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5602f188a7d0 0x5602f189f000 unknown :-1 s=CLOSED pgs=958 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:58.912 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.918+0000 7fbcd3b76740 1 --2- 172.21.15.90:0/3934657454 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5602f1889eb0 0x5602f1892b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:58.912 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.918+0000 7fbcd3b76740 1 -- 172.21.15.90:0/3934657454 >> 172.21.15.90:0/3934657454 conn(0x5602f18845e0 msgr2=0x5602f18903a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:48:58.912 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.918+0000 7fbcd3b76740 1 -- 172.21.15.90:0/3934657454 shutdown_connections 2024-04-02T14:48:58.913 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.918+0000 7fbcd3b76740 1 -- 172.21.15.90:0/3934657454 wait complete. 2024-04-02T14:48:58.913 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.918+0000 7fbcd3b76740 1 -- start start 2024-04-02T14:48:58.913 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.919+0000 7fbcd3b76740 1 Processor -- start 2024-04-02T14:48:58.914 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.919+0000 7fbcd3b76740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5602f1889eb0 0x5602f188d440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:58.914 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.919+0000 7fbcd3b76740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5602f188a7d0 0x5602f188d980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:58.914 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.919+0000 7fbcd3b76740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5602f18933b0 0x5602f19086d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:58.914 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.919+0000 7fbcd3b76740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fbccc1b4fe0 con 0x5602f1889eb0 2024-04-02T14:48:58.914 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.919+0000 7fbcd3b76740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fbccc1b4910 con 0x5602f18933b0 2024-04-02T14:48:58.914 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.919+0000 7fbcd3b76740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fbccc1b1b70 con 0x5602f188a7d0 2024-04-02T14:48:58.914 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.919+0000 7fbcd3b76740 10 client.0 _ll_register_callbacks cb 0x7fbcc40329e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:48:58.914 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.920+0000 7fbcc9400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5602f188a7d0 0x5602f188d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:58.915 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.920+0000 7fbcc9e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5602f1889eb0 0x5602f188d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:58.915 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.920+0000 7fbcc9400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5602f188a7d0 0x5602f188d980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:59092/0 (socket says 172.21.15.90:59092) 2024-04-02T14:48:58.915 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.920+0000 7fbcc9400640 1 -- 172.21.15.90:0/2554421628 learned_addr learned my addr 172.21.15.90:0/2554421628 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:48:58.915 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.920+0000 7fbcca800640 1 --2- 172.21.15.90:0/2554421628 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5602f18933b0 0x5602f19086d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:58.915 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.920+0000 7fbcc9400640 1 -- 172.21.15.90:0/2554421628 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5602f18933b0 msgr2=0x5602f19086d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:48:58.915 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.921+0000 7fbcc9400640 1 --2- 172.21.15.90:0/2554421628 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5602f18933b0 0x5602f19086d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:58.915 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.921+0000 7fbcc9400640 1 -- 172.21.15.90:0/2554421628 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5602f1889eb0 msgr2=0x5602f188d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:48:58.915 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.921+0000 7fbcc9400640 1 --2- 172.21.15.90:0/2554421628 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5602f1889eb0 0x5602f188d440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:58.915 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.921+0000 7fbcc9400640 1 -- 172.21.15.90:0/2554421628 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5602f15cf620 con 0x5602f188a7d0 2024-04-02T14:48:58.915 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.921+0000 7fbcca800640 1 --2- 172.21.15.90:0/2554421628 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5602f18933b0 0x5602f19086d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:48:58.916 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.922+0000 7fbcc9400640 1 --2- 172.21.15.90:0/2554421628 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5602f188a7d0 0x5602f188d980 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7fbcb80cda30 tx=0x7fbcb80cde30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=34196b6c91524b9a server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:48:58.916 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.922+0000 7fbcabe00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:48:58.917 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.922+0000 7fbcabe00640 1 -- 172.21.15.90:0/2554421628 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcb80b4020 con 0x5602f188a7d0 2024-04-02T14:48:58.917 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.922+0000 7fbcabe00640 1 -- 172.21.15.90:0/2554421628 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 8550+0+0 (secure 0 0 0) 0x7fbcb80a0120 con 0x5602f188a7d0 2024-04-02T14:48:58.917 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.923+0000 7fbcd3b76740 1 -- 172.21.15.90:0/2554421628 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbcb809c8d0 con 0x5602f188a7d0 2024-04-02T14:48:58.917 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.923+0000 7fbcabe00640 1 -- 172.21.15.90:0/2554421628 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fbcb8021050 con 0x5602f188a7d0 2024-04-02T14:48:58.917 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.923+0000 7fbcd3b76740 10 client.29605 fetch_fsmap learned FSMap version 27 2024-04-02T14:48:58.917 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.923+0000 7fbcd3b76740 1 -- 172.21.15.90:0/2554421628 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5602f1909b50 con 0x5602f188a7d0 2024-04-02T14:48:58.918 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.924+0000 7fbcabe00640 1 -- 172.21.15.90:0/2554421628 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbcb809c8d0 con 0x5602f188a7d0 2024-04-02T14:48:58.919 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.924+0000 7fbcabe00640 1 -- 172.21.15.90:0/2554421628 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fbcb809b620 con 0x5602f188a7d0 2024-04-02T14:48:58.919 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.924+0000 7fbcd3b76740 10 client.29605 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:48:58.919 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.924+0000 7fbcd3b76740 10 client.29605 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-02T14:48:58.919 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.924+0000 7fbcd3b76740 20 client.29605 populate_metadata read hostname 'smithi090' 2024-04-02T14:48:58.919 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.924+0000 7fbcd3b76740 1 --2- 172.21.15.90:0/2554421628 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5602f190a1d0 0x5602f190c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:58.919 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.924+0000 7fbcd3b76740 4 client.29605 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-02T14:48:58.919 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.924+0000 7fbcd3b76740 1 -- 172.21.15.90:0/2554421628 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5602f17fc2b0 con 0x5602f190a1d0 2024-04-02T14:48:58.919 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.924+0000 7fbcc9e00640 1 --2- 172.21.15.90:0/2554421628 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5602f190a1d0 0x5602f190c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:58.919 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.925+0000 7fbcc9e00640 1 --2- 172.21.15.90:0/2554421628 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5602f190a1d0 0x5602f190c5f0 crc :-1 s=READY pgs=359 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=216f86e4bc449125 server_cookie=aece75732dca8cbd in_seq=0 out_seq=0 2024-04-02T14:48:58.920 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.925+0000 7fbcabe00640 10 client.29605 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:48:58.942 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.948+0000 7fbcabe00640 1 -- 172.21.15.90:0/2554421628 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5602f17fc2b0 con 0x5602f190a1d0 2024-04-02T14:48:58.942 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.948+0000 7fbcabe00640 10 client.29605 handle_command_reply: tid=0 2024-04-02T14:48:58.942 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.948+0000 7fbcd3b76740 1 client.29605 shutdown 2024-04-02T14:48:58.943 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.949+0000 7fbcd3b76740 1 -- 172.21.15.90:0/2554421628 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5602f188a7d0 msgr2=0x5602f188d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:48:58.943 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.949+0000 7fbcd3b76740 1 --2- 172.21.15.90:0/2554421628 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5602f188a7d0 0x5602f188d980 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7fbcb80cda30 tx=0x7fbcb80cde30 comp rx=0 tx=0).stop 2024-04-02T14:48:58.944 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.950+0000 7fbcd3b76740 1 -- 172.21.15.90:0/2554421628 shutdown_connections 2024-04-02T14:48:58.944 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.950+0000 7fbcd3b76740 1 --2- 172.21.15.90:0/2554421628 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5602f190a1d0 0x5602f190c5f0 crc :-1 s=READY pgs=359 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:58.944 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.950+0000 7fbcd3b76740 1 --2- 172.21.15.90:0/2554421628 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5602f18933b0 0x5602f19086d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:58.944 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.950+0000 7fbcd3b76740 1 --2- 172.21.15.90:0/2554421628 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5602f188a7d0 0x5602f188d980 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:58.944 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.950+0000 7fbcd3b76740 1 --2- 172.21.15.90:0/2554421628 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5602f1889eb0 0x5602f188d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:58.944 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.950+0000 7fbcd3b76740 1 -- 172.21.15.90:0/2554421628 >> 172.21.15.90:0/2554421628 conn(0x5602f18845e0 msgr2=0x5602f1890c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:48:58.945 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.950+0000 7fbcabe00640 0 client.29605 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:48:58.945 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.950+0000 7fbcd3b76740 1 -- 172.21.15.90:0/2554421628 shutdown_connections 2024-04-02T14:48:58.945 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.950+0000 7fbcd3b76740 1 -- 172.21.15.90:0/2554421628 wait complete. 2024-04-02T14:48:58.945 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.951+0000 7fbcd3b76740 20 client.29605 trim_cache size 0 max 16384 2024-04-02T14:48:58.946 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:{ 2024-04-02T14:48:58.946 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "message": "", 2024-04-02T14:48:58.946 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "return_code": 0 2024-04-02T14:48:58.946 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:} 2024-04-02T14:48:58.947 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.952+0000 7fbcd0e00640 1 -- 172.21.15.90:0/2724020256 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fbcb00a3e90 msgr2=0x7fbcb00a6350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:48:58.947 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.952+0000 7fbcd0e00640 1 --2- 172.21.15.90:0/2724020256 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fbcb00a3e90 0x7fbcb00a6350 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x5602f1889980 tx=0x7fbcbc005f90 comp rx=0 tx=0).stop 2024-04-02T14:48:58.947 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.952+0000 7fbcd0e00640 1 -- 172.21.15.90:0/2724020256 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbccc109010 msgr2=0x7fbccc10f6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:48:58.947 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.952+0000 7fbcd0e00640 1 --2- 172.21.15.90:0/2724020256 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbccc109010 0x7fbccc10f6e0 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7fbcb800d770 tx=0x7fbcb800dc30 comp rx=0 tx=0).stop 2024-04-02T14:48:58.947 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.952+0000 7fbcd0e00640 1 -- 172.21.15.90:0/2724020256 shutdown_connections 2024-04-02T14:48:58.947 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.952+0000 7fbcd0e00640 1 --2- 172.21.15.90:0/2724020256 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fbcb00a3e90 0x7fbcb00a6350 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:58.947 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.952+0000 7fbcd0e00640 1 --2- 172.21.15.90:0/2724020256 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbccc10fd80 0x7fbccc1aa4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:58.947 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.952+0000 7fbcd0e00640 1 --2- 172.21.15.90:0/2724020256 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbccc109010 0x7fbccc10f6e0 unknown :-1 s=CLOSED pgs=955 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:58.947 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.952+0000 7fbcd0e00640 1 --2- 172.21.15.90:0/2724020256 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbccc103f40 0x7fbccc10f1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:58.947 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.952+0000 7fbcd0e00640 1 -- 172.21.15.90:0/2724020256 >> 172.21.15.90:0/2724020256 conn(0x7fbccc0ff7c0 msgr2=0x7fbccc101750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:48:58.947 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.953+0000 7fbcd0e00640 1 -- 172.21.15.90:0/2724020256 shutdown_connections 2024-04-02T14:48:58.947 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:48:58.953+0000 7fbcd0e00640 1 -- 172.21.15.90:0/2724020256 wait complete. 2024-04-02T14:48:59.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.139+0000 7fae5da00640 1 -- 172.21.15.90:0/112645475 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fae58107a90 msgr2=0x7fae58107e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:48:59.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.139+0000 7fae5da00640 1 --2- 172.21.15.90:0/112645475 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fae58107a90 0x7fae58107e70 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7fae48009930 tx=0x7fae4802bc30 comp rx=0 tx=0).stop 2024-04-02T14:48:59.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.139+0000 7fae5da00640 1 -- 172.21.15.90:0/112645475 shutdown_connections 2024-04-02T14:48:59.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.139+0000 7fae5da00640 1 --2- 172.21.15.90:0/112645475 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fae58102440 0x7fae5810e950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:59.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.139+0000 7fae5da00640 1 --2- 172.21.15.90:0/112645475 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fae58101a80 0x7fae58101f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:59.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.139+0000 7fae5da00640 1 --2- 172.21.15.90:0/112645475 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fae58107a90 0x7fae58107e70 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:59.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.139+0000 7fae5da00640 1 -- 172.21.15.90:0/112645475 >> 172.21.15.90:0/112645475 conn(0x7fae580fd590 msgr2=0x7fae580ff9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:48:59.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.140+0000 7fae5da00640 1 -- 172.21.15.90:0/112645475 shutdown_connections 2024-04-02T14:48:59.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.140+0000 7fae5da00640 1 -- 172.21.15.90:0/112645475 wait complete. 2024-04-02T14:48:59.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.141+0000 7fae5da00640 1 Processor -- start 2024-04-02T14:48:59.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.141+0000 7fae5da00640 1 -- start start 2024-04-02T14:48:59.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.142+0000 7fae5da00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fae58101a80 0x7fae581a46d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:59.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.142+0000 7fae5da00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fae58102440 0x7fae581a4c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:59.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.142+0000 7fae5da00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fae58107a90 0x7fae5810ca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:59.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.142+0000 7fae5da00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fae58111ef0 con 0x7fae58102440 2024-04-02T14:48:59.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.142+0000 7fae5da00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fae58111d70 con 0x7fae58107a90 2024-04-02T14:48:59.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.142+0000 7fae5da00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fae58112070 con 0x7fae58101a80 2024-04-02T14:48:59.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.142+0000 7fae57400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fae58107a90 0x7fae5810ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:59.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.142+0000 7fae57400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fae58107a90 0x7fae5810ca20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:42144/0 (socket says 172.21.15.90:42144) 2024-04-02T14:48:59.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.142+0000 7fae57400640 1 -- 172.21.15.90:0/1927486046 learned_addr learned my addr 172.21.15.90:0/1927486046 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:48:59.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.142+0000 7fae56000640 1 --2- 172.21.15.90:0/1927486046 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fae58102440 0x7fae581a4c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:59.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.142+0000 7fae56a00640 1 --2- 172.21.15.90:0/1927486046 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fae58101a80 0x7fae581a46d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:59.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.143+0000 7fae57400640 1 -- 172.21.15.90:0/1927486046 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fae58101a80 msgr2=0x7fae581a46d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:48:59.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.143+0000 7fae57400640 1 --2- 172.21.15.90:0/1927486046 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fae58101a80 0x7fae581a46d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:59.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.143+0000 7fae57400640 1 -- 172.21.15.90:0/1927486046 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fae58102440 msgr2=0x7fae581a4c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:48:59.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.143+0000 7fae57400640 1 --2- 172.21.15.90:0/1927486046 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fae58102440 0x7fae581a4c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:59.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.143+0000 7fae57400640 1 -- 172.21.15.90:0/1927486046 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae5810d1f0 con 0x7fae58107a90 2024-04-02T14:48:59.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.143+0000 7fae57400640 1 --2- 172.21.15.90:0/1927486046 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fae58107a90 0x7fae5810ca20 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7fae5000eaa0 tx=0x7fae5000ef60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:48:59.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.143+0000 7fae4f400640 1 -- 172.21.15.90:0/1927486046 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae5000cdb0 con 0x7fae58107a90 2024-04-02T14:48:59.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.143+0000 7fae4f400640 1 -- 172.21.15.90:0/1927486046 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fae50005020 con 0x7fae58107a90 2024-04-02T14:48:59.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.143+0000 7fae5da00640 1 -- 172.21.15.90:0/1927486046 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae5810d4e0 con 0x7fae58107a90 2024-04-02T14:48:59.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.143+0000 7fae4f400640 1 -- 172.21.15.90:0/1927486046 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae500116b0 con 0x7fae58107a90 2024-04-02T14:48:59.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.143+0000 7fae5da00640 1 -- 172.21.15.90:0/1927486046 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fae581ab520 con 0x7fae58107a90 2024-04-02T14:48:59.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.148+0000 7fae607a8740 1 Processor -- start 2024-04-02T14:48:59.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.149+0000 7fae4f400640 1 -- 172.21.15.90:0/1927486046 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fae50005560 con 0x7fae58107a90 2024-04-02T14:48:59.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.149+0000 7fae607a8740 1 -- start start 2024-04-02T14:48:59.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.150+0000 7fae607a8740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f4d9ea49f0 0x55f4d9ea3740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:59.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.150+0000 7fae607a8740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f4d9ea3d10 0x55f4d9ea4190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:59.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.150+0000 7fae607a8740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f4d9ead080 0x55f4d9eaf470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:59.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.150+0000 7fae607a8740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55f4d9de4400 con 0x55f4d9ea49f0 2024-04-02T14:48:59.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.150+0000 7fae607a8740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55f4d9dc8f10 con 0x55f4d9ead080 2024-04-02T14:48:59.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.150+0000 7fae607a8740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55f4d9e086f0 con 0x55f4d9ea3d10 2024-04-02T14:48:59.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.150+0000 7fae56a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f4d9ea3d10 0x55f4d9ea4190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:59.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.150+0000 7fae56a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f4d9ea3d10 0x55f4d9ea4190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:59116/0 (socket says 172.21.15.90:59116) 2024-04-02T14:48:59.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.150+0000 7fae56a00640 1 -- 172.21.15.90:0/1985275422 learned_addr learned my addr 172.21.15.90:0/1985275422 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:48:59.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.151+0000 7fae56000640 1 --2- 172.21.15.90:0/1985275422 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f4d9ea49f0 0x55f4d9ea3740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:59.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.151+0000 7fae56a00640 1 -- 172.21.15.90:0/1985275422 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f4d9ead080 msgr2=0x55f4d9eaf470 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-02T14:48:59.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.151+0000 7fae56a00640 1 --2- 172.21.15.90:0/1985275422 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f4d9ead080 0x55f4d9eaf470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:59.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.151+0000 7fae56a00640 1 -- 172.21.15.90:0/1985275422 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f4d9ea49f0 msgr2=0x55f4d9ea3740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:48:59.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.151+0000 7fae56a00640 1 --2- 172.21.15.90:0/1985275422 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f4d9ea49f0 0x55f4d9ea3740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:59.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.151+0000 7fae56a00640 1 -- 172.21.15.90:0/1985275422 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f4d9b1c3f0 con 0x55f4d9ea3d10 2024-04-02T14:48:59.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.151+0000 7fae56a00640 1 --2- 172.21.15.90:0/1985275422 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f4d9ea3d10 0x55f4d9ea4190 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7fae48000c00 tx=0x7fae480095e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c67a49f5f3359f30 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:48:59.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.151+0000 7fae43e00640 1 -- 172.21.15.90:0/1985275422 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae48045070 con 0x55f4d9ea3d10 2024-04-02T14:48:59.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.151+0000 7fae4f400640 1 --2- 172.21.15.90:0/1927486046 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fae3c0a3da0 0x7fae3c0a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:59.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.152+0000 7fae4f400640 1 -- 172.21.15.90:0/1927486046 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 8550+0+0 (secure 0 0 0) 0x7fae500d0920 con 0x7fae58107a90 2024-04-02T14:48:59.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.152+0000 7fae43e00640 1 -- 172.21.15.90:0/1985275422 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fae4802c880 con 0x55f4d9ea3d10 2024-04-02T14:48:59.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.152+0000 7fae43e00640 1 -- 172.21.15.90:0/1985275422 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae48004b60 con 0x55f4d9ea3d10 2024-04-02T14:48:59.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.152+0000 7fae56000640 1 --2- 172.21.15.90:0/1927486046 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fae3c0a3da0 0x7fae3c0a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:59.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.153+0000 7fae607a8740 1 -- 172.21.15.90:0/1985275422 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f4d9ea3d10 msgr2=0x55f4d9ea4190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:48:59.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.153+0000 7fae56000640 1 --2- 172.21.15.90:0/1927486046 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fae3c0a3da0 0x7fae3c0a6260 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fae5810ec80 tx=0x7fae44005f90 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:48:59.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.153+0000 7fae607a8740 1 --2- 172.21.15.90:0/1985275422 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f4d9ea3d10 0x55f4d9ea4190 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7fae48000c00 tx=0x7fae480095e0 comp rx=0 tx=0).stop 2024-04-02T14:48:59.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.153+0000 7fae607a8740 1 -- 172.21.15.90:0/1985275422 shutdown_connections 2024-04-02T14:48:59.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.153+0000 7fae607a8740 1 --2- 172.21.15.90:0/1985275422 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f4d9ead080 0x55f4d9eaf470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:59.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.153+0000 7fae607a8740 1 --2- 172.21.15.90:0/1985275422 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f4d9ea3d10 0x55f4d9ea4190 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:59.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.153+0000 7fae607a8740 1 --2- 172.21.15.90:0/1985275422 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f4d9ea49f0 0x55f4d9ea3740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:59.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.153+0000 7fae607a8740 1 -- 172.21.15.90:0/1985275422 >> 172.21.15.90:0/1985275422 conn(0x55f4d9e6aa50 msgr2=0x55f4d9e9e760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:48:59.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.154+0000 7fae607a8740 1 -- 172.21.15.90:0/1985275422 shutdown_connections 2024-04-02T14:48:59.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.154+0000 7fae607a8740 1 -- 172.21.15.90:0/1985275422 wait complete. 2024-04-02T14:48:59.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.154+0000 7fae607a8740 1 -- start start 2024-04-02T14:48:59.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.155+0000 7fae607a8740 1 Processor -- start 2024-04-02T14:48:59.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.155+0000 7fae607a8740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f4d9ea3d10 0x55f4d9ea9ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:59.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.156+0000 7fae607a8740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f4d9ea49f0 0x55f4d9eaa530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:59.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.156+0000 7fae607a8740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f4d9ead080 0x55f4d9eaaa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:59.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.156+0000 7fae607a8740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55f4d9e046a0 con 0x55f4d9ea49f0 2024-04-02T14:48:59.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.156+0000 7fae607a8740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55f4d9db0980 con 0x55f4d9ea3d10 2024-04-02T14:48:59.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.156+0000 7fae607a8740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55f4d9dc3cb0 con 0x55f4d9ead080 2024-04-02T14:48:59.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.156+0000 7fae607a8740 10 client.0 _ll_register_callbacks cb 0x55f4d9dc36a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:48:59.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.156+0000 7fae57400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f4d9ead080 0x55f4d9eaaa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:59.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.156+0000 7fae57400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f4d9ead080 0x55f4d9eaaa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:59130/0 (socket says 172.21.15.90:59130) 2024-04-02T14:48:59.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.156+0000 7fae57400640 1 -- 172.21.15.90:0/3477580461 learned_addr learned my addr 172.21.15.90:0/3477580461 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:48:59.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.156+0000 7fae56a00640 1 --2- 172.21.15.90:0/3477580461 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f4d9ea3d10 0x55f4d9ea9ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:59.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.156+0000 7fae57400640 1 -- 172.21.15.90:0/3477580461 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f4d9ea3d10 msgr2=0x55f4d9ea9ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:48:59.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.156+0000 7fae57400640 1 --2- 172.21.15.90:0/3477580461 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f4d9ea3d10 0x55f4d9ea9ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:59.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.156+0000 7fae57400640 1 -- 172.21.15.90:0/3477580461 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f4d9ea49f0 msgr2=0x55f4d9eaa530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:48:59.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.156+0000 7fae57400640 1 --2- 172.21.15.90:0/3477580461 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f4d9ea49f0 0x55f4d9eaa530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:59.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.156+0000 7fae57400640 1 -- 172.21.15.90:0/3477580461 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f4d9be67d0 con 0x55f4d9ead080 2024-04-02T14:48:59.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.157+0000 7fae57400640 1 --2- 172.21.15.90:0/3477580461 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f4d9ead080 0x55f4d9eaaa70 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7fae500ce100 tx=0x7fae500a0d10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7588c5a3f6adce36 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:48:59.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.157+0000 7fae40c00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:48:59.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.157+0000 7fae40c00640 1 -- 172.21.15.90:0/3477580461 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae5001f040 con 0x55f4d9ead080 2024-04-02T14:48:59.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.157+0000 7fae40c00640 1 -- 172.21.15.90:0/3477580461 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 8550+0+0 (secure 0 0 0) 0x7fae500870c0 con 0x55f4d9ead080 2024-04-02T14:48:59.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.157+0000 7fae607a8740 1 -- 172.21.15.90:0/3477580461 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fae4802c880 con 0x55f4d9ead080 2024-04-02T14:48:59.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.158+0000 7fae40c00640 1 -- 172.21.15.90:0/3477580461 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fae500872c0 con 0x55f4d9ead080 2024-04-02T14:48:59.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.158+0000 7fae607a8740 10 client.29617 fetch_fsmap learned FSMap version 27 2024-04-02T14:48:59.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.158+0000 7fae607a8740 1 -- 172.21.15.90:0/3477580461 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55f4d9ebbef0 con 0x55f4d9ead080 2024-04-02T14:48:59.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.158+0000 7fae40c00640 1 -- 172.21.15.90:0/3477580461 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fae4802c880 con 0x55f4d9ead080 2024-04-02T14:48:59.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.158+0000 7fae40c00640 1 -- 172.21.15.90:0/3477580461 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fae5000f040 con 0x55f4d9ead080 2024-04-02T14:48:59.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.158+0000 7fae607a8740 10 client.29617 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:48:59.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.158+0000 7fae607a8740 10 client.29617 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.c 2024-04-02T14:48:59.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.158+0000 7fae607a8740 20 client.29617 populate_metadata read hostname 'smithi090' 2024-04-02T14:48:59.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.158+0000 7fae607a8740 1 --2- 172.21.15.90:0/3477580461 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x55f4d9ebd410 0x55f4d9ebf860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:59.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.158+0000 7fae607a8740 4 client.29617 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:48:59.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.158+0000 7fae607a8740 1 -- 172.21.15.90:0/3477580461 --> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f4d9dc8040 con 0x55f4d9ebd410 2024-04-02T14:48:59.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.159+0000 7fae56a00640 1 --2- 172.21.15.90:0/3477580461 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x55f4d9ebd410 0x55f4d9ebf860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:59.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.159+0000 7fae56a00640 1 --2- 172.21.15.90:0/3477580461 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x55f4d9ebd410 0x55f4d9ebf860 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=ffc2e0132702989f server_cookie=3e50de52e5258036 in_seq=0 out_seq=0 2024-04-02T14:48:59.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.159+0000 7fae40c00640 10 client.29617 ms_handle_connect on v2:172.21.15.177:6836/4223414858 2024-04-02T14:48:59.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.160+0000 7fae40c00640 1 -- 172.21.15.90:0/3477580461 <== mds.4 v2:172.21.15.177:6836/4223414858 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f4d9dc8040 con 0x55f4d9ebd410 2024-04-02T14:48:59.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.160+0000 7fae40c00640 10 client.29617 handle_command_reply: tid=0 2024-04-02T14:48:59.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.160+0000 7fae607a8740 1 client.29617 shutdown 2024-04-02T14:48:59.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.160+0000 7fae607a8740 1 -- 172.21.15.90:0/3477580461 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f4d9ead080 msgr2=0x55f4d9eaaa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:48:59.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.160+0000 7fae607a8740 1 --2- 172.21.15.90:0/3477580461 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f4d9ead080 0x55f4d9eaaa70 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7fae500ce100 tx=0x7fae500a0d10 comp rx=0 tx=0).stop 2024-04-02T14:48:59.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.160+0000 7fae607a8740 1 -- 172.21.15.90:0/3477580461 shutdown_connections 2024-04-02T14:48:59.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.160+0000 7fae607a8740 1 --2- 172.21.15.90:0/3477580461 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f4d9ead080 0x55f4d9eaaa70 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:59.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.160+0000 7fae607a8740 1 --2- 172.21.15.90:0/3477580461 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x55f4d9ebd410 0x55f4d9ebf860 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:59.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.160+0000 7fae607a8740 1 --2- 172.21.15.90:0/3477580461 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f4d9ea49f0 0x55f4d9eaa530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:59.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.160+0000 7fae607a8740 1 --2- 172.21.15.90:0/3477580461 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f4d9ea3d10 0x55f4d9ea9ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:59.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.160+0000 7fae607a8740 1 -- 172.21.15.90:0/3477580461 >> 172.21.15.90:0/3477580461 conn(0x55f4d9e6aa50 msgr2=0x55f4d9ea04b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:48:59.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.160+0000 7fae40c00640 0 client.29617 ms_handle_reset on v2:172.21.15.177:6836/4223414858 2024-04-02T14:48:59.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.160+0000 7fae607a8740 1 -- 172.21.15.90:0/3477580461 shutdown_connections 2024-04-02T14:48:59.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.160+0000 7fae607a8740 1 -- 172.21.15.90:0/3477580461 wait complete. 2024-04-02T14:48:59.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.161+0000 7fae607a8740 20 client.29617 trim_cache size 0 max 16384 2024-04-02T14:48:59.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.180+0000 7fae607a8740 1 Processor -- start 2024-04-02T14:48:59.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.180+0000 7fae607a8740 1 -- start start 2024-04-02T14:48:59.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.181+0000 7fae607a8740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f4d9ea3d10 0x55f4d9eb8af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:59.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.181+0000 7fae607a8740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f4d9ea49f0 0x55f4d9ea98f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:59.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.181+0000 7fae607a8740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f4d9ead080 0x55f4d9ec5130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:59.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.181+0000 7fae607a8740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fae581b3e00 con 0x55f4d9ea3d10 2024-04-02T14:48:59.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.181+0000 7fae607a8740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fae581b5940 con 0x55f4d9ea49f0 2024-04-02T14:48:59.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.181+0000 7fae607a8740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fae581b44d0 con 0x55f4d9ead080 2024-04-02T14:48:59.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.181+0000 7fae57400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f4d9ead080 0x55f4d9ec5130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:59.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.181+0000 7fae56a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f4d9ea3d10 0x55f4d9eb8af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:59.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.181+0000 7fae56a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f4d9ea3d10 0x55f4d9eb8af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41316/0 (socket says 172.21.15.90:41316) 2024-04-02T14:48:59.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.181+0000 7fae56a00640 1 -- 172.21.15.90:0/819047353 learned_addr learned my addr 172.21.15.90:0/819047353 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:48:59.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.181+0000 7fae57400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f4d9ead080 0x55f4d9ec5130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:59140/0 (socket says 172.21.15.90:59140) 2024-04-02T14:48:59.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.181+0000 7fae56000640 1 --2- 172.21.15.90:0/819047353 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f4d9ea49f0 0x55f4d9ea98f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:59.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.182+0000 7fae57400640 1 -- 172.21.15.90:0/819047353 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f4d9ea49f0 msgr2=0x55f4d9ea98f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:48:59.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.182+0000 7fae57400640 1 --2- 172.21.15.90:0/819047353 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f4d9ea49f0 0x55f4d9ea98f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:59.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.182+0000 7fae57400640 1 -- 172.21.15.90:0/819047353 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f4d9ea3d10 msgr2=0x55f4d9eb8af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:48:59.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.182+0000 7fae57400640 1 --2- 172.21.15.90:0/819047353 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f4d9ea3d10 0x55f4d9eb8af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:59.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.182+0000 7fae57400640 1 -- 172.21.15.90:0/819047353 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f4d9ea2d40 con 0x55f4d9ead080 2024-04-02T14:48:59.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.182+0000 7fae56a00640 1 --2- 172.21.15.90:0/819047353 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f4d9ea3d10 0x55f4d9eb8af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:48:59.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.182+0000 7fae57400640 1 --2- 172.21.15.90:0/819047353 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f4d9ead080 0x55f4d9ec5130 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7fae500ce650 tx=0x7fae500054d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3ccd699cb06ca00d server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:48:59.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.182+0000 7fae43400640 1 -- 172.21.15.90:0/819047353 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae5001f070 con 0x55f4d9ead080 2024-04-02T14:48:59.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.182+0000 7fae43400640 1 -- 172.21.15.90:0/819047353 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fae5009c040 con 0x55f4d9ead080 2024-04-02T14:48:59.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.182+0000 7fae43400640 1 -- 172.21.15.90:0/819047353 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae500a52e0 con 0x55f4d9ead080 2024-04-02T14:48:59.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.183+0000 7fae607a8740 1 -- 172.21.15.90:0/819047353 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f4d9ead080 msgr2=0x55f4d9ec5130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:48:59.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.183+0000 7fae607a8740 1 --2- 172.21.15.90:0/819047353 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f4d9ead080 0x55f4d9ec5130 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7fae500ce650 tx=0x7fae500054d0 comp rx=0 tx=0).stop 2024-04-02T14:48:59.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.183+0000 7fae607a8740 1 -- 172.21.15.90:0/819047353 shutdown_connections 2024-04-02T14:48:59.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.183+0000 7fae607a8740 1 --2- 172.21.15.90:0/819047353 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f4d9ead080 0x55f4d9ec5130 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:59.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.183+0000 7fae607a8740 1 --2- 172.21.15.90:0/819047353 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f4d9ea49f0 0x55f4d9ea98f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:59.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.183+0000 7fae607a8740 1 --2- 172.21.15.90:0/819047353 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f4d9ea3d10 0x55f4d9eb8af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:59.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.183+0000 7fae607a8740 1 -- 172.21.15.90:0/819047353 >> 172.21.15.90:0/819047353 conn(0x55f4d9e6aa50 msgr2=0x55f4d9ea0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:48:59.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.184+0000 7fae607a8740 1 -- 172.21.15.90:0/819047353 shutdown_connections 2024-04-02T14:48:59.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.184+0000 7fae607a8740 1 -- 172.21.15.90:0/819047353 wait complete. 2024-04-02T14:48:59.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.184+0000 7fae607a8740 1 -- start start 2024-04-02T14:48:59.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.185+0000 7fae607a8740 1 Processor -- start 2024-04-02T14:48:59.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.186+0000 7fae607a8740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f4d9ea3d10 0x55f4d9eb5000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:59.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.186+0000 7fae607a8740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f4d9ea49f0 0x55f4d9eb5540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:59.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.186+0000 7fae607a8740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f4d9ead080 0x55f4d9ea6460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:59.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.186+0000 7fae607a8740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fae581b5270 con 0x55f4d9ea49f0 2024-04-02T14:48:59.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.186+0000 7fae607a8740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fae581b4ba0 con 0x55f4d9ead080 2024-04-02T14:48:59.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.186+0000 7fae607a8740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fae581b1e00 con 0x55f4d9ea3d10 2024-04-02T14:48:59.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.186+0000 7fae607a8740 10 client.0 _ll_register_callbacks cb 0x7fae5009c950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:48:59.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.186+0000 7fae57400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f4d9ead080 0x55f4d9ea6460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:59.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.186+0000 7fae57400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f4d9ead080 0x55f4d9ea6460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:42174/0 (socket says 172.21.15.90:42174) 2024-04-02T14:48:59.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.186+0000 7fae57400640 1 -- 172.21.15.90:0/1551501719 learned_addr learned my addr 172.21.15.90:0/1551501719 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:48:59.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.187+0000 7fae57400640 1 -- 172.21.15.90:0/1551501719 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f4d9ea3d10 msgr2=0x55f4d9eb5000 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-02T14:48:59.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.187+0000 7fae57400640 1 --2- 172.21.15.90:0/1551501719 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f4d9ea3d10 0x55f4d9eb5000 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:59.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.187+0000 7fae57400640 1 -- 172.21.15.90:0/1551501719 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f4d9ea49f0 msgr2=0x55f4d9eb5540 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-02T14:48:59.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.187+0000 7fae56000640 1 --2- 172.21.15.90:0/1551501719 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f4d9ea49f0 0x55f4d9eb5540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:59.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.187+0000 7fae57400640 1 --2- 172.21.15.90:0/1551501719 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f4d9ea49f0 0x55f4d9eb5540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:59.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.187+0000 7fae57400640 1 -- 172.21.15.90:0/1551501719 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f4d9ea6c10 con 0x55f4d9ead080 2024-04-02T14:48:59.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.187+0000 7fae56000640 1 --2- 172.21.15.90:0/1551501719 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f4d9ea49f0 0x55f4d9eb5540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:48:59.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.188+0000 7fae57400640 1 --2- 172.21.15.90:0/1551501719 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f4d9ead080 0x55f4d9ea6460 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7fae5009c600 tx=0x7fae500a53f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=42227f82c53bb94 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:48:59.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.188+0000 7fae40c00640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:48:59.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.188+0000 7fae40c00640 1 -- 172.21.15.90:0/1551501719 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae5001f070 con 0x55f4d9ead080 2024-04-02T14:48:59.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.188+0000 7fae40c00640 1 -- 172.21.15.90:0/1551501719 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 8550+0+0 (secure 0 0 0) 0x7fae500986a0 con 0x55f4d9ead080 2024-04-02T14:48:59.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.188+0000 7fae607a8740 1 -- 172.21.15.90:0/1551501719 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fae5009c040 con 0x55f4d9ead080 2024-04-02T14:48:59.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.188+0000 7fae40c00640 1 -- 172.21.15.90:0/1551501719 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fae500988a0 con 0x55f4d9ead080 2024-04-02T14:48:59.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.188+0000 7fae607a8740 10 client.29831 fetch_fsmap learned FSMap version 27 2024-04-02T14:48:59.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.188+0000 7fae607a8740 1 -- 172.21.15.90:0/1551501719 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55f4d9ea6c10 con 0x55f4d9ead080 2024-04-02T14:48:59.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.189+0000 7fae40c00640 1 -- 172.21.15.90:0/1551501719 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fae5009c040 con 0x55f4d9ead080 2024-04-02T14:48:59.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.189+0000 7fae40c00640 1 -- 172.21.15.90:0/1551501719 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fae50098040 con 0x55f4d9ead080 2024-04-02T14:48:59.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.189+0000 7fae607a8740 10 client.29831 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:48:59.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.189+0000 7fae607a8740 10 client.29831 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.c 2024-04-02T14:48:59.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.189+0000 7fae607a8740 20 client.29831 populate_metadata read hostname 'smithi090' 2024-04-02T14:48:59.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.189+0000 7fae607a8740 1 --2- 172.21.15.90:0/1551501719 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x55f4d9f34720 0x55f4d9f36b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:59.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.189+0000 7fae607a8740 4 client.29831 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-02T14:48:59.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.189+0000 7fae607a8740 1 -- 172.21.15.90:0/1551501719 --> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55f4d9e15c30 con 0x55f4d9f34720 2024-04-02T14:48:59.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.189+0000 7fae56a00640 1 --2- 172.21.15.90:0/1551501719 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x55f4d9f34720 0x55f4d9f36b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:59.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.190+0000 7fae40c00640 10 client.29831 ms_handle_connect on v2:172.21.15.177:6836/4223414858 2024-04-02T14:48:59.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.190+0000 7fae56a00640 1 --2- 172.21.15.90:0/1551501719 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x55f4d9f34720 0x55f4d9f36b40 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=e738cef4e856f122 server_cookie=850ef7b31f1686c5 in_seq=0 out_seq=0 2024-04-02T14:48:59.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.190+0000 7fae40c00640 1 -- 172.21.15.90:0/1551501719 <== mds.4 v2:172.21.15.177:6836/4223414858 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55f4d9e15c30 con 0x55f4d9f34720 2024-04-02T14:48:59.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.190+0000 7fae40c00640 10 client.29831 handle_command_reply: tid=0 2024-04-02T14:48:59.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.190+0000 7fae607a8740 1 client.29831 shutdown 2024-04-02T14:48:59.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.190+0000 7fae607a8740 1 -- 172.21.15.90:0/1551501719 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f4d9ead080 msgr2=0x55f4d9ea6460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:48:59.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.190+0000 7fae607a8740 1 --2- 172.21.15.90:0/1551501719 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f4d9ead080 0x55f4d9ea6460 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7fae5009c600 tx=0x7fae500a53f0 comp rx=0 tx=0).stop 2024-04-02T14:48:59.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.191+0000 7fae607a8740 1 -- 172.21.15.90:0/1551501719 shutdown_connections 2024-04-02T14:48:59.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.191+0000 7fae607a8740 1 --2- 172.21.15.90:0/1551501719 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f4d9ead080 0x55f4d9ea6460 unknown :-1 s=CLOSED pgs=960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:59.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.191+0000 7fae607a8740 1 --2- 172.21.15.90:0/1551501719 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x55f4d9f34720 0x55f4d9f36b40 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:59.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.191+0000 7fae607a8740 1 --2- 172.21.15.90:0/1551501719 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f4d9ea49f0 0x55f4d9eb5540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:59.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.191+0000 7fae607a8740 1 --2- 172.21.15.90:0/1551501719 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f4d9ea3d10 0x55f4d9eb5000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:59.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.191+0000 7fae607a8740 1 -- 172.21.15.90:0/1551501719 >> 172.21.15.90:0/1551501719 conn(0x55f4d9e6aa50 msgr2=0x55f4d9ea9fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:48:59.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.191+0000 7fae40c00640 0 client.29831 ms_handle_reset on v2:172.21.15.177:6836/4223414858 2024-04-02T14:48:59.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.192+0000 7fae607a8740 1 -- 172.21.15.90:0/1551501719 shutdown_connections 2024-04-02T14:48:59.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.192+0000 7fae607a8740 1 -- 172.21.15.90:0/1551501719 wait complete. 2024-04-02T14:48:59.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.193+0000 7fae607a8740 20 client.29831 trim_cache size 0 max 16384 2024-04-02T14:48:59.188 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-02T14:48:59.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.195+0000 7fae5da00640 1 -- 172.21.15.90:0/1927486046 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fae3c0a3da0 msgr2=0x7fae3c0a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:48:59.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.195+0000 7fae5da00640 1 --2- 172.21.15.90:0/1927486046 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fae3c0a3da0 0x7fae3c0a6260 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fae5810ec80 tx=0x7fae44005f90 comp rx=0 tx=0).stop 2024-04-02T14:48:59.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.195+0000 7fae5da00640 1 -- 172.21.15.90:0/1927486046 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fae58107a90 msgr2=0x7fae5810ca20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:48:59.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.195+0000 7fae5da00640 1 --2- 172.21.15.90:0/1927486046 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fae58107a90 0x7fae5810ca20 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7fae5000eaa0 tx=0x7fae5000ef60 comp rx=0 tx=0).stop 2024-04-02T14:48:59.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.195+0000 7fae5da00640 1 -- 172.21.15.90:0/1927486046 shutdown_connections 2024-04-02T14:48:59.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.195+0000 7fae5da00640 1 --2- 172.21.15.90:0/1927486046 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fae3c0a3da0 0x7fae3c0a6260 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:59.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.195+0000 7fae5da00640 1 --2- 172.21.15.90:0/1927486046 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fae58107a90 0x7fae5810ca20 unknown :-1 s=CLOSED pgs=959 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:59.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.195+0000 7fae5da00640 1 --2- 172.21.15.90:0/1927486046 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fae58102440 0x7fae581a4c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:59.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.195+0000 7fae5da00640 1 --2- 172.21.15.90:0/1927486046 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fae58101a80 0x7fae581a46d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:59.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.195+0000 7fae5da00640 1 -- 172.21.15.90:0/1927486046 >> 172.21.15.90:0/1927486046 conn(0x7fae580fd590 msgr2=0x7fae58074210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:48:59.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.195+0000 7fae5da00640 1 -- 172.21.15.90:0/1927486046 shutdown_connections 2024-04-02T14:48:59.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.195+0000 7fae5da00640 1 -- 172.21.15.90:0/1927486046 wait complete. 2024-04-02T14:48:59.219 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-04-02T14:48:59.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.520+0000 7fe4afa00640 1 -- 172.21.15.90:0/3997639204 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe4a8108310 msgr2=0x7fe4a81086f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:48:59.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.520+0000 7fe4afa00640 1 --2- 172.21.15.90:0/3997639204 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe4a8108310 0x7fe4a81086f0 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7fe4a0009900 tx=0x7fe4a002bc30 comp rx=0 tx=0).stop 2024-04-02T14:48:59.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.520+0000 7fe4afa00640 1 -- 172.21.15.90:0/3997639204 shutdown_connections 2024-04-02T14:48:59.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.520+0000 7fe4afa00640 1 --2- 172.21.15.90:0/3997639204 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe4a810d3e0 0x7fe4a8114fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:59.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.520+0000 7fe4afa00640 1 --2- 172.21.15.90:0/3997639204 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe4a8108cc0 0x7fe4a810cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:59.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.520+0000 7fe4afa00640 1 --2- 172.21.15.90:0/3997639204 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe4a8108310 0x7fe4a81086f0 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:59.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.520+0000 7fe4afa00640 1 -- 172.21.15.90:0/3997639204 >> 172.21.15.90:0/3997639204 conn(0x7fe4a8076040 msgr2=0x7fe4a8078460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:48:59.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.521+0000 7fe4afa00640 1 -- 172.21.15.90:0/3997639204 shutdown_connections 2024-04-02T14:48:59.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.521+0000 7fe4afa00640 1 -- 172.21.15.90:0/3997639204 wait complete. 2024-04-02T14:48:59.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.522+0000 7fe4afa00640 1 Processor -- start 2024-04-02T14:48:59.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.522+0000 7fe4afa00640 1 -- start start 2024-04-02T14:48:59.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.522+0000 7fe4afa00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe4a8108310 0x7fe4a81a4680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:59.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.522+0000 7fe4afa00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe4a810d3e0 0x7fe4a81a4bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:59.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.522+0000 7fe4afa00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe4a819e7f0 0x7fe4a819eca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:59.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.522+0000 7fe4afa00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fe4a81184f0 con 0x7fe4a8108310 2024-04-02T14:48:59.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.522+0000 7fe4afa00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fe4a8118370 con 0x7fe4a819e7f0 2024-04-02T14:48:59.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.522+0000 7fe4afa00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fe4a8118670 con 0x7fe4a810d3e0 2024-04-02T14:48:59.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.523+0000 7fe4a7e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe4a810d3e0 0x7fe4a81a4bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:59.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.523+0000 7fe4ada00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe4a819e7f0 0x7fe4a819eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:59.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.523+0000 7fe4a7e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe4a810d3e0 0x7fe4a81a4bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:59156/0 (socket says 172.21.15.90:59156) 2024-04-02T14:48:59.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.523+0000 7fe4a7e00640 1 -- 172.21.15.90:0/3105814601 learned_addr learned my addr 172.21.15.90:0/3105814601 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:48:59.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.523+0000 7fe4ad000640 1 --2- 172.21.15.90:0/3105814601 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe4a8108310 0x7fe4a81a4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:59.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.523+0000 7fe4ada00640 1 -- 172.21.15.90:0/3105814601 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe4a810d3e0 msgr2=0x7fe4a81a4bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:48:59.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.523+0000 7fe4ada00640 1 --2- 172.21.15.90:0/3105814601 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe4a810d3e0 0x7fe4a81a4bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:59.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.523+0000 7fe4ada00640 1 -- 172.21.15.90:0/3105814601 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe4a8108310 msgr2=0x7fe4a81a4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:48:59.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.523+0000 7fe4ada00640 1 --2- 172.21.15.90:0/3105814601 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe4a8108310 0x7fe4a81a4680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:59.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.523+0000 7fe4ada00640 1 -- 172.21.15.90:0/3105814601 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4a819f380 con 0x7fe4a819e7f0 2024-04-02T14:48:59.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.523+0000 7fe4a7e00640 1 --2- 172.21.15.90:0/3105814601 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe4a810d3e0 0x7fe4a81a4bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:48:59.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.523+0000 7fe4ada00640 1 --2- 172.21.15.90:0/3105814601 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe4a819e7f0 0x7fe4a819eca0 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7fe49000e970 tx=0x7fe49000ee30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:48:59.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.524+0000 7fe4a5600640 1 -- 172.21.15.90:0/3105814601 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe49000ccc0 con 0x7fe4a819e7f0 2024-04-02T14:48:59.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.524+0000 7fe4a5600640 1 -- 172.21.15.90:0/3105814601 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe490005020 con 0x7fe4a819e7f0 2024-04-02T14:48:59.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.524+0000 7fe4afa00640 1 -- 172.21.15.90:0/3105814601 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4a819f670 con 0x7fe4a819e7f0 2024-04-02T14:48:59.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.524+0000 7fe4a5600640 1 -- 172.21.15.90:0/3105814601 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe490004a00 con 0x7fe4a819e7f0 2024-04-02T14:48:59.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.524+0000 7fe4afa00640 1 -- 172.21.15.90:0/3105814601 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4a81ab850 con 0x7fe4a819e7f0 2024-04-02T14:48:59.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.527+0000 7fe4b27dc740 1 Processor -- start 2024-04-02T14:48:59.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.527+0000 7fe4a5600640 1 -- 172.21.15.90:0/3105814601 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fe4900051d0 con 0x7fe4a819e7f0 2024-04-02T14:48:59.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.527+0000 7fe4b27dc740 1 -- start start 2024-04-02T14:48:59.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.528+0000 7fe4b27dc740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d8fd13a9f0 0x55d8fd139740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:59.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.528+0000 7fe4b27dc740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d8fd139d10 0x55d8fd13a190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:59.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.528+0000 7fe4b27dc740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d8fd143080 0x55d8fd145470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:59.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.528+0000 7fe4b27dc740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55d8fd07a400 con 0x55d8fd143080 2024-04-02T14:48:59.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.528+0000 7fe4b27dc740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55d8fd05ef10 con 0x55d8fd13a9f0 2024-04-02T14:48:59.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.528+0000 7fe4b27dc740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55d8fd09e6f0 con 0x55d8fd139d10 2024-04-02T14:48:59.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.528+0000 7fe4a7e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d8fd13a9f0 0x55d8fd139740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:59.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.528+0000 7fe4ad000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d8fd139d10 0x55d8fd13a190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:59.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.528+0000 7fe4a7e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d8fd13a9f0 0x55d8fd139740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:42192/0 (socket says 172.21.15.90:42192) 2024-04-02T14:48:59.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.528+0000 7fe4a7e00640 1 -- 172.21.15.90:0/1509074824 learned_addr learned my addr 172.21.15.90:0/1509074824 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:48:59.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.528+0000 7fe4a7e00640 1 --2- 172.21.15.90:0/1509074824 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d8fd143080 0x55d8fd145470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:59.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.528+0000 7fe4ad000640 1 -- 172.21.15.90:0/1509074824 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d8fd13a9f0 msgr2=0x55d8fd139740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:48:59.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.528+0000 7fe4ad000640 1 --2- 172.21.15.90:0/1509074824 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d8fd13a9f0 0x55d8fd139740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:59.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.528+0000 7fe4ad000640 1 -- 172.21.15.90:0/1509074824 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d8fd143080 msgr2=0x55d8fd145470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:48:59.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.528+0000 7fe4ad000640 1 --2- 172.21.15.90:0/1509074824 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d8fd143080 0x55d8fd145470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:59.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.528+0000 7fe4ad000640 1 -- 172.21.15.90:0/1509074824 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d8fcdb23f0 con 0x55d8fd139d10 2024-04-02T14:48:59.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.529+0000 7fe4a5600640 1 --2- 172.21.15.90:0/3105814601 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fe48c0a3da0 0x7fe48c0a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:59.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.529+0000 7fe4ad000640 1 --2- 172.21.15.90:0/1509074824 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d8fd139d10 0x55d8fd13a190 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7fe4a002c170 tx=0x7fe4a0005be0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1fa654a543658a75 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:48:59.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.529+0000 7fe4a5600640 1 -- 172.21.15.90:0/3105814601 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 8550+0+0 (secure 0 0 0) 0x7fe4900d0750 con 0x7fe4a819e7f0 2024-04-02T14:48:59.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.529+0000 7fe4a7e00640 1 --2- 172.21.15.90:0/3105814601 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fe48c0a3da0 0x7fe48c0a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:59.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.529+0000 7fe49e000640 1 -- 172.21.15.90:0/1509074824 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4a0045070 con 0x55d8fd139d10 2024-04-02T14:48:59.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.529+0000 7fe49e000640 1 -- 172.21.15.90:0/1509074824 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe4a002c650 con 0x55d8fd139d10 2024-04-02T14:48:59.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.529+0000 7fe49e000640 1 -- 172.21.15.90:0/1509074824 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4a0004b90 con 0x55d8fd139d10 2024-04-02T14:48:59.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.530+0000 7fe4b27dc740 1 -- 172.21.15.90:0/1509074824 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d8fd139d10 msgr2=0x55d8fd13a190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:48:59.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.530+0000 7fe4b27dc740 1 --2- 172.21.15.90:0/1509074824 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d8fd139d10 0x55d8fd13a190 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7fe4a002c170 tx=0x7fe4a0005be0 comp rx=0 tx=0).stop 2024-04-02T14:48:59.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.530+0000 7fe4a7e00640 1 --2- 172.21.15.90:0/3105814601 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fe48c0a3da0 0x7fe48c0a6260 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fe498003c10 tx=0x7fe498009500 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:48:59.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.530+0000 7fe4b27dc740 1 -- 172.21.15.90:0/1509074824 shutdown_connections 2024-04-02T14:48:59.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.530+0000 7fe4b27dc740 1 --2- 172.21.15.90:0/1509074824 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d8fd143080 0x55d8fd145470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:59.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.530+0000 7fe4b27dc740 1 --2- 172.21.15.90:0/1509074824 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d8fd139d10 0x55d8fd13a190 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:59.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.530+0000 7fe4b27dc740 1 --2- 172.21.15.90:0/1509074824 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d8fd13a9f0 0x55d8fd139740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:59.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.530+0000 7fe4b27dc740 1 -- 172.21.15.90:0/1509074824 >> 172.21.15.90:0/1509074824 conn(0x55d8fd100a50 msgr2=0x55d8fd134760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:48:59.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.530+0000 7fe4b27dc740 1 -- 172.21.15.90:0/1509074824 shutdown_connections 2024-04-02T14:48:59.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.530+0000 7fe4b27dc740 1 -- 172.21.15.90:0/1509074824 wait complete. 2024-04-02T14:48:59.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.531+0000 7fe4b27dc740 1 -- start start 2024-04-02T14:48:59.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.531+0000 7fe4b27dc740 1 Processor -- start 2024-04-02T14:48:59.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.531+0000 7fe4b27dc740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d8fd139d10 0x55d8fd140040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:59.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.531+0000 7fe4b27dc740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d8fd13a9f0 0x55d8fd140580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:59.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.531+0000 7fe4b27dc740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d8fd143080 0x55d8fd140ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:59.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.531+0000 7fe4b27dc740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55d8fd09a6a0 con 0x55d8fd139d10 2024-04-02T14:48:59.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.531+0000 7fe4b27dc740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55d8fd046980 con 0x55d8fd13a9f0 2024-04-02T14:48:59.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.531+0000 7fe4b27dc740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55d8fd059cb0 con 0x55d8fd143080 2024-04-02T14:48:59.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.531+0000 7fe4b27dc740 10 client.0 _ll_register_callbacks cb 0x55d8fd0596a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:48:59.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.531+0000 7fe4ad000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d8fd139d10 0x55d8fd140040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:59.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.531+0000 7fe4ad000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d8fd139d10 0x55d8fd140040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41362/0 (socket says 172.21.15.90:41362) 2024-04-02T14:48:59.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.531+0000 7fe4ad000640 1 -- 172.21.15.90:0/153118020 learned_addr learned my addr 172.21.15.90:0/153118020 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:48:59.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.531+0000 7fe4a7e00640 1 --2- 172.21.15.90:0/153118020 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d8fd13a9f0 0x55d8fd140580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:59.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.532+0000 7fe4ada00640 1 --2- 172.21.15.90:0/153118020 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d8fd143080 0x55d8fd140ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:59.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.532+0000 7fe4a7e00640 1 -- 172.21.15.90:0/153118020 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d8fd143080 msgr2=0x55d8fd140ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:48:59.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.532+0000 7fe4a7e00640 1 --2- 172.21.15.90:0/153118020 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d8fd143080 0x55d8fd140ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:59.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.532+0000 7fe4a7e00640 1 -- 172.21.15.90:0/153118020 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d8fd139d10 msgr2=0x55d8fd140040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:48:59.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.532+0000 7fe4a7e00640 1 --2- 172.21.15.90:0/153118020 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d8fd139d10 0x55d8fd140040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:59.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.532+0000 7fe4a7e00640 1 -- 172.21.15.90:0/153118020 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d8fce7c7d0 con 0x55d8fd13a9f0 2024-04-02T14:48:59.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.532+0000 7fe4ada00640 1 --2- 172.21.15.90:0/153118020 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d8fd143080 0x55d8fd140ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:48:59.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.532+0000 7fe4a7e00640 1 --2- 172.21.15.90:0/153118020 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d8fd13a9f0 0x55d8fd140580 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7fe4980048a0 tx=0x7fe498004ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=22580b84fb5909f9 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:48:59.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.532+0000 7fe496a00640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:48:59.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.532+0000 7fe496a00640 1 -- 172.21.15.90:0/153118020 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe49802f060 con 0x55d8fd13a9f0 2024-04-02T14:48:59.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.533+0000 7fe496a00640 1 -- 172.21.15.90:0/153118020 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 8550+0+0 (secure 0 0 0) 0x7fe49801c030 con 0x55d8fd13a9f0 2024-04-02T14:48:59.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.533+0000 7fe4b27dc740 1 -- 172.21.15.90:0/153118020 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe4a002c650 con 0x55d8fd13a9f0 2024-04-02T14:48:59.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.533+0000 7fe496a00640 1 -- 172.21.15.90:0/153118020 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fe498021a90 con 0x55d8fd13a9f0 2024-04-02T14:48:59.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.533+0000 7fe4b27dc740 10 client.29843 fetch_fsmap learned FSMap version 27 2024-04-02T14:48:59.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.533+0000 7fe4b27dc740 1 -- 172.21.15.90:0/153118020 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55d8fd151f20 con 0x55d8fd13a9f0 2024-04-02T14:48:59.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.534+0000 7fe496a00640 1 -- 172.21.15.90:0/153118020 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe4a002c650 con 0x55d8fd13a9f0 2024-04-02T14:48:59.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.535+0000 7fe496a00640 1 -- 172.21.15.90:0/153118020 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fe49801f3e0 con 0x55d8fd13a9f0 2024-04-02T14:48:59.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.535+0000 7fe4b27dc740 10 client.29843 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:48:59.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.535+0000 7fe4b27dc740 10 client.29843 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-04-02T14:48:59.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.535+0000 7fe4b27dc740 20 client.29843 populate_metadata read hostname 'smithi090' 2024-04-02T14:48:59.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.535+0000 7fe4b27dc740 1 --2- 172.21.15.90:0/153118020 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x55d8fd153430 0x55d8fd155860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:59.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.535+0000 7fe4b27dc740 4 client.29843 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:48:59.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.535+0000 7fe4b27dc740 1 -- 172.21.15.90:0/153118020 --> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d8fd05e040 con 0x55d8fd153430 2024-04-02T14:48:59.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.535+0000 7fe4ad000640 1 --2- 172.21.15.90:0/153118020 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x55d8fd153430 0x55d8fd155860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:59.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.536+0000 7fe4ad000640 1 --2- 172.21.15.90:0/153118020 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x55d8fd153430 0x55d8fd155860 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=48d9921664a70f54 server_cookie=1e9d57495d2638cd in_seq=0 out_seq=0 2024-04-02T14:48:59.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.536+0000 7fe496a00640 10 client.29843 ms_handle_connect on v2:172.21.15.92:6836/1270635939 2024-04-02T14:48:59.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.537+0000 7fe496a00640 1 -- 172.21.15.90:0/153118020 <== mds.3 v2:172.21.15.92:6836/1270635939 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d8fd05e040 con 0x55d8fd153430 2024-04-02T14:48:59.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.537+0000 7fe496a00640 10 client.29843 handle_command_reply: tid=0 2024-04-02T14:48:59.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.537+0000 7fe4b27dc740 1 client.29843 shutdown 2024-04-02T14:48:59.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.537+0000 7fe4b27dc740 1 -- 172.21.15.90:0/153118020 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d8fd13a9f0 msgr2=0x55d8fd140580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:48:59.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.537+0000 7fe4b27dc740 1 --2- 172.21.15.90:0/153118020 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d8fd13a9f0 0x55d8fd140580 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7fe4980048a0 tx=0x7fe498004ab0 comp rx=0 tx=0).stop 2024-04-02T14:48:59.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.537+0000 7fe4b27dc740 1 -- 172.21.15.90:0/153118020 shutdown_connections 2024-04-02T14:48:59.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.537+0000 7fe4b27dc740 1 --2- 172.21.15.90:0/153118020 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x55d8fd153430 0x55d8fd155860 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:59.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.537+0000 7fe4b27dc740 1 --2- 172.21.15.90:0/153118020 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d8fd143080 0x55d8fd140ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:59.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.537+0000 7fe4b27dc740 1 --2- 172.21.15.90:0/153118020 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d8fd13a9f0 0x55d8fd140580 unknown :-1 s=CLOSED pgs=962 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:59.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.537+0000 7fe4b27dc740 1 --2- 172.21.15.90:0/153118020 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d8fd139d10 0x55d8fd140040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:59.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.537+0000 7fe4b27dc740 1 -- 172.21.15.90:0/153118020 >> 172.21.15.90:0/153118020 conn(0x55d8fd100a50 msgr2=0x55d8fd1364b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:48:59.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.537+0000 7fe496a00640 0 client.29843 ms_handle_reset on v2:172.21.15.92:6836/1270635939 2024-04-02T14:48:59.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.537+0000 7fe4b27dc740 1 -- 172.21.15.90:0/153118020 shutdown_connections 2024-04-02T14:48:59.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.537+0000 7fe4b27dc740 1 -- 172.21.15.90:0/153118020 wait complete. 2024-04-02T14:48:59.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.537+0000 7fe4b27dc740 20 client.29843 trim_cache size 0 max 16384 2024-04-02T14:48:59.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.558+0000 7fe4b27dc740 1 Processor -- start 2024-04-02T14:48:59.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.559+0000 7fe4b27dc740 1 -- start start 2024-04-02T14:48:59.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.559+0000 7fe4b27dc740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d8fd139d10 0x55d8fd142770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:59.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.559+0000 7fe4b27dc740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d8fd13a9f0 0x55d8fd14eac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:59.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.559+0000 7fe4b27dc740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d8fd143080 0x55d8fd15b130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:59.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.559+0000 7fe4b27dc740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fe4a81b3e50 con 0x55d8fd139d10 2024-04-02T14:48:59.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.559+0000 7fe4b27dc740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fe4a81b5990 con 0x55d8fd13a9f0 2024-04-02T14:48:59.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.559+0000 7fe4b27dc740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fe4a81b4520 con 0x55d8fd143080 2024-04-02T14:48:59.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.559+0000 7fe4ad000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d8fd139d10 0x55d8fd142770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:59.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.559+0000 7fe4ad000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d8fd139d10 0x55d8fd142770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41370/0 (socket says 172.21.15.90:41370) 2024-04-02T14:48:59.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.559+0000 7fe4ad000640 1 -- 172.21.15.90:0/2700441431 learned_addr learned my addr 172.21.15.90:0/2700441431 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:48:59.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.559+0000 7fe4ada00640 1 --2- 172.21.15.90:0/2700441431 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d8fd143080 0x55d8fd15b130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:59.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.559+0000 7fe4a7e00640 1 --2- 172.21.15.90:0/2700441431 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d8fd13a9f0 0x55d8fd14eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:59.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.560+0000 7fe4ad000640 1 -- 172.21.15.90:0/2700441431 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d8fd143080 msgr2=0x55d8fd15b130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:48:59.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.560+0000 7fe4ad000640 1 --2- 172.21.15.90:0/2700441431 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d8fd143080 0x55d8fd15b130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:59.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.560+0000 7fe4ad000640 1 -- 172.21.15.90:0/2700441431 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d8fd13a9f0 msgr2=0x55d8fd14eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:48:59.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.560+0000 7fe4ad000640 1 --2- 172.21.15.90:0/2700441431 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d8fd13a9f0 0x55d8fd14eac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:59.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.560+0000 7fe4ad000640 1 -- 172.21.15.90:0/2700441431 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d8fd138d40 con 0x55d8fd139d10 2024-04-02T14:48:59.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.560+0000 7fe4ada00640 1 --2- 172.21.15.90:0/2700441431 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d8fd143080 0x55d8fd15b130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:48:59.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.560+0000 7fe4ad000640 1 --2- 172.21.15.90:0/2700441431 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d8fd139d10 0x55d8fd142770 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fe4a002c240 tx=0x7fe4a0000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ae97ecbc363756a server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:48:59.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.560+0000 7fe49d600640 1 -- 172.21.15.90:0/2700441431 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4a0045040 con 0x55d8fd139d10 2024-04-02T14:48:59.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.560+0000 7fe49d600640 1 -- 172.21.15.90:0/2700441431 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe4a002c5d0 con 0x55d8fd139d10 2024-04-02T14:48:59.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.560+0000 7fe49d600640 1 -- 172.21.15.90:0/2700441431 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4a004ac40 con 0x55d8fd139d10 2024-04-02T14:48:59.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.561+0000 7fe4b27dc740 1 -- 172.21.15.90:0/2700441431 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d8fd139d10 msgr2=0x55d8fd142770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:48:59.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.561+0000 7fe4b27dc740 1 --2- 172.21.15.90:0/2700441431 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d8fd139d10 0x55d8fd142770 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fe4a002c240 tx=0x7fe4a0000bc0 comp rx=0 tx=0).stop 2024-04-02T14:48:59.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.561+0000 7fe4b27dc740 1 -- 172.21.15.90:0/2700441431 shutdown_connections 2024-04-02T14:48:59.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.561+0000 7fe4b27dc740 1 --2- 172.21.15.90:0/2700441431 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d8fd143080 0x55d8fd15b130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:59.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.561+0000 7fe4b27dc740 1 --2- 172.21.15.90:0/2700441431 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d8fd13a9f0 0x55d8fd14eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:59.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.561+0000 7fe4b27dc740 1 --2- 172.21.15.90:0/2700441431 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d8fd139d10 0x55d8fd142770 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:59.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.561+0000 7fe4b27dc740 1 -- 172.21.15.90:0/2700441431 >> 172.21.15.90:0/2700441431 conn(0x55d8fd100a50 msgr2=0x55d8fd134290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:48:59.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.561+0000 7fe4b27dc740 1 -- 172.21.15.90:0/2700441431 shutdown_connections 2024-04-02T14:48:59.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.561+0000 7fe4b27dc740 1 -- 172.21.15.90:0/2700441431 wait complete. 2024-04-02T14:48:59.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.562+0000 7fe4b27dc740 1 -- start start 2024-04-02T14:48:59.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.562+0000 7fe4b27dc740 1 Processor -- start 2024-04-02T14:48:59.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.563+0000 7fe4b27dc740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d8fd139d10 0x55d8fd14b4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:59.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.563+0000 7fe4b27dc740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d8fd13a9f0 0x55d8fd13d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:59.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.563+0000 7fe4b27dc740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d8fd143080 0x55d8fd13d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:59.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.563+0000 7fe4b27dc740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fe4a81b52c0 con 0x55d8fd139d10 2024-04-02T14:48:59.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.563+0000 7fe4b27dc740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fe4a81b4bf0 con 0x55d8fd13a9f0 2024-04-02T14:48:59.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.563+0000 7fe4b27dc740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fe4a81b1e50 con 0x55d8fd143080 2024-04-02T14:48:59.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.563+0000 7fe4b27dc740 10 client.0 _ll_register_callbacks cb 0x7fe498011830 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:48:59.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.564+0000 7fe4a7e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d8fd13a9f0 0x55d8fd13d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:59.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.564+0000 7fe4a7e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d8fd13a9f0 0x55d8fd13d3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:42212/0 (socket says 172.21.15.90:42212) 2024-04-02T14:48:59.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.564+0000 7fe4a7e00640 1 -- 172.21.15.90:0/2160238972 learned_addr learned my addr 172.21.15.90:0/2160238972 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:48:59.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.564+0000 7fe4ad000640 1 --2- 172.21.15.90:0/2160238972 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d8fd139d10 0x55d8fd14b4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:59.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.564+0000 7fe4ada00640 1 --2- 172.21.15.90:0/2160238972 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d8fd143080 0x55d8fd13d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:59.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.564+0000 7fe4ad000640 1 -- 172.21.15.90:0/2160238972 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d8fd143080 msgr2=0x55d8fd13d9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:48:59.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.564+0000 7fe4ad000640 1 --2- 172.21.15.90:0/2160238972 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d8fd143080 0x55d8fd13d9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:59.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.564+0000 7fe4ad000640 1 -- 172.21.15.90:0/2160238972 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d8fd13a9f0 msgr2=0x55d8fd13d3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:48:59.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.564+0000 7fe4ad000640 1 --2- 172.21.15.90:0/2160238972 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d8fd13a9f0 0x55d8fd13d3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:59.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.564+0000 7fe4ad000640 1 -- 172.21.15.90:0/2160238972 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d8fd13c390 con 0x55d8fd139d10 2024-04-02T14:48:59.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.565+0000 7fe4a7e00640 1 --2- 172.21.15.90:0/2160238972 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d8fd13a9f0 0x55d8fd13d3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:48:59.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.565+0000 7fe4ad000640 1 --2- 172.21.15.90:0/2160238972 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d8fd139d10 0x55d8fd14b4b0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fe4a002c240 tx=0x7fe4a002c2e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=293024920aa1f5c0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:48:59.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.565+0000 7fe496a00640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-02T14:48:59.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.565+0000 7fe496a00640 1 -- 172.21.15.90:0/2160238972 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4a0045040 con 0x55d8fd139d10 2024-04-02T14:48:59.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.565+0000 7fe496a00640 1 -- 172.21.15.90:0/2160238972 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 8550+0+0 (secure 0 0 0) 0x7fe4a002c880 con 0x55d8fd139d10 2024-04-02T14:48:59.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.565+0000 7fe4b27dc740 1 -- 172.21.15.90:0/2160238972 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe4a002c5d0 con 0x55d8fd139d10 2024-04-02T14:48:59.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.566+0000 7fe496a00640 1 -- 172.21.15.90:0/2160238972 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fe4a004a650 con 0x55d8fd139d10 2024-04-02T14:48:59.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.566+0000 7fe4b27dc740 10 client.18633 fetch_fsmap learned FSMap version 27 2024-04-02T14:48:59.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.566+0000 7fe4b27dc740 1 -- 172.21.15.90:0/2160238972 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55d8fd1b9c00 con 0x55d8fd139d10 2024-04-02T14:48:59.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.566+0000 7fe496a00640 1 -- 172.21.15.90:0/2160238972 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe4a002c5d0 con 0x55d8fd139d10 2024-04-02T14:48:59.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.566+0000 7fe496a00640 1 -- 172.21.15.90:0/2160238972 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fe4a00346e0 con 0x55d8fd139d10 2024-04-02T14:48:59.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.567+0000 7fe4b27dc740 10 client.18633 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:48:59.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.567+0000 7fe4b27dc740 10 client.18633 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-04-02T14:48:59.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.567+0000 7fe4b27dc740 20 client.18633 populate_metadata read hostname 'smithi090' 2024-04-02T14:48:59.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.567+0000 7fe4b27dc740 1 --2- 172.21.15.90:0/2160238972 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x55d8fd1b9ed0 0x55d8fd1bc2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:59.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.567+0000 7fe4b27dc740 4 client.18633 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-02T14:48:59.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.567+0000 7fe4b27dc740 1 -- 172.21.15.90:0/2160238972 --> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55d8fd0abc30 con 0x55d8fd1b9ed0 2024-04-02T14:48:59.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.567+0000 7fe4a7e00640 1 --2- 172.21.15.90:0/2160238972 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x55d8fd1b9ed0 0x55d8fd1bc2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:59.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.567+0000 7fe4a7e00640 1 --2- 172.21.15.90:0/2160238972 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x55d8fd1b9ed0 0x55d8fd1bc2c0 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=1bce2c0c0a233da4 server_cookie=670ca578c2563047 in_seq=0 out_seq=0 2024-04-02T14:48:59.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.567+0000 7fe496a00640 10 client.18633 ms_handle_connect on v2:172.21.15.92:6836/1270635939 2024-04-02T14:48:59.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.568+0000 7fe496a00640 1 -- 172.21.15.90:0/2160238972 <== mds.3 v2:172.21.15.92:6836/1270635939 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55d8fd0abc30 con 0x55d8fd1b9ed0 2024-04-02T14:48:59.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.568+0000 7fe496a00640 10 client.18633 handle_command_reply: tid=0 2024-04-02T14:48:59.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.568+0000 7fe4b27dc740 1 client.18633 shutdown 2024-04-02T14:48:59.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.568+0000 7fe4b27dc740 1 -- 172.21.15.90:0/2160238972 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d8fd139d10 msgr2=0x55d8fd14b4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:48:59.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.568+0000 7fe4b27dc740 1 --2- 172.21.15.90:0/2160238972 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d8fd139d10 0x55d8fd14b4b0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fe4a002c240 tx=0x7fe4a002c2e0 comp rx=0 tx=0).stop 2024-04-02T14:48:59.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.568+0000 7fe4b27dc740 1 -- 172.21.15.90:0/2160238972 shutdown_connections 2024-04-02T14:48:59.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.568+0000 7fe4b27dc740 1 --2- 172.21.15.90:0/2160238972 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x55d8fd1b9ed0 0x55d8fd1bc2c0 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:59.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.568+0000 7fe4b27dc740 1 --2- 172.21.15.90:0/2160238972 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d8fd143080 0x55d8fd13d9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:59.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.568+0000 7fe4b27dc740 1 --2- 172.21.15.90:0/2160238972 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d8fd13a9f0 0x55d8fd13d3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:59.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.568+0000 7fe496a00640 0 client.18633 ms_handle_reset on v2:172.21.15.92:6836/1270635939 2024-04-02T14:48:59.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.568+0000 7fe4b27dc740 1 --2- 172.21.15.90:0/2160238972 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d8fd139d10 0x55d8fd14b4b0 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:59.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.568+0000 7fe4b27dc740 1 -- 172.21.15.90:0/2160238972 >> 172.21.15.90:0/2160238972 conn(0x55d8fd100a50 msgr2=0x55d8fd133010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:48:59.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.568+0000 7fe4b27dc740 1 -- 172.21.15.90:0/2160238972 shutdown_connections 2024-04-02T14:48:59.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.568+0000 7fe4b27dc740 1 -- 172.21.15.90:0/2160238972 wait complete. 2024-04-02T14:48:59.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.569+0000 7fe4b27dc740 20 client.18633 trim_cache size 0 max 16384 2024-04-02T14:48:59.565 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-02T14:48:59.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.569+0000 7fe4afa00640 1 -- 172.21.15.90:0/3105814601 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fe48c0a3da0 msgr2=0x7fe48c0a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:48:59.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.569+0000 7fe4afa00640 1 --2- 172.21.15.90:0/3105814601 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fe48c0a3da0 0x7fe48c0a6260 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fe498003c10 tx=0x7fe498009500 comp rx=0 tx=0).stop 2024-04-02T14:48:59.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.569+0000 7fe4afa00640 1 -- 172.21.15.90:0/3105814601 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe4a819e7f0 msgr2=0x7fe4a819eca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:48:59.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.569+0000 7fe4afa00640 1 --2- 172.21.15.90:0/3105814601 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe4a819e7f0 0x7fe4a819eca0 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7fe49000e970 tx=0x7fe49000ee30 comp rx=0 tx=0).stop 2024-04-02T14:48:59.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.569+0000 7fe4afa00640 1 -- 172.21.15.90:0/3105814601 shutdown_connections 2024-04-02T14:48:59.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.569+0000 7fe4afa00640 1 --2- 172.21.15.90:0/3105814601 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fe48c0a3da0 0x7fe48c0a6260 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:59.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.569+0000 7fe4afa00640 1 --2- 172.21.15.90:0/3105814601 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe4a819e7f0 0x7fe4a819eca0 unknown :-1 s=CLOSED pgs=961 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:59.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.569+0000 7fe4afa00640 1 --2- 172.21.15.90:0/3105814601 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe4a810d3e0 0x7fe4a81a4bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:59.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.569+0000 7fe4afa00640 1 --2- 172.21.15.90:0/3105814601 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe4a8108310 0x7fe4a81a4680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:59.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.569+0000 7fe4afa00640 1 -- 172.21.15.90:0/3105814601 >> 172.21.15.90:0/3105814601 conn(0x7fe4a8076040 msgr2=0x7fe4a80782f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:48:59.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.570+0000 7fe4afa00640 1 -- 172.21.15.90:0/3105814601 shutdown_connections 2024-04-02T14:48:59.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.570+0000 7fe4afa00640 1 -- 172.21.15.90:0/3105814601 wait complete. 2024-04-02T14:48:59.580 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:48:59 smithi090 ceph-mon[31674]: from='client.? 172.21.15.90:0/314532437' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-02T14:48:59.584 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-02T14:48:59.603 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:48:59 smithi092 ceph-mon[36497]: from='client.? 172.21.15.90:0/314532437' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-02T14:48:59.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:48:59 smithi177 ceph-mon[36431]: from='client.? 172.21.15.90:0/314532437' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-02T14:48:59.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.854+0000 7fd638a00640 1 -- 172.21.15.90:0/1478593745 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd62402cc80 con 0x7fd63c108ce0 2024-04-02T14:48:59.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.854+0000 7fd641000640 1 -- 172.21.15.90:0/1478593745 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd63c108ce0 msgr2=0x7fd63c1090c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:48:59.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.854+0000 7fd641000640 1 --2- 172.21.15.90:0/1478593745 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd63c108ce0 0x7fd63c1090c0 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7fd624009900 tx=0x7fd62402bc30 comp rx=0 tx=0).stop 2024-04-02T14:48:59.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.858+0000 7fd641000640 1 -- 172.21.15.90:0/1478593745 shutdown_connections 2024-04-02T14:48:59.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.858+0000 7fd641000640 1 --2- 172.21.15.90:0/1478593745 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd63c114c50 0x7fd63c117040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:59.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.858+0000 7fd641000640 1 --2- 172.21.15.90:0/1478593745 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd63c109600 0x7fd63c114710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:59.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.858+0000 7fd641000640 1 --2- 172.21.15.90:0/1478593745 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd63c108ce0 0x7fd63c1090c0 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:59.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.858+0000 7fd641000640 1 -- 172.21.15.90:0/1478593745 >> 172.21.15.90:0/1478593745 conn(0x7fd63c06cbe0 msgr2=0x7fd63c06cff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:48:59.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.858+0000 7fd641000640 1 -- 172.21.15.90:0/1478593745 shutdown_connections 2024-04-02T14:48:59.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.859+0000 7fd641000640 1 -- 172.21.15.90:0/1478593745 wait complete. 2024-04-02T14:48:59.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.860+0000 7fd641000640 1 Processor -- start 2024-04-02T14:48:59.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.860+0000 7fd641000640 1 -- start start 2024-04-02T14:48:59.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.861+0000 7fd641000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd63c108ce0 0x7fd63c1a6850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:59.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.861+0000 7fd641000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd63c109600 0x7fd63c1a6d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:59.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.861+0000 7fd641000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd63c114c50 0x7fd63c1a0a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:59.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.861+0000 7fd641000640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fd63c11a710 con 0x7fd63c114c50 2024-04-02T14:48:59.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.861+0000 7fd641000640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fd63c11a590 con 0x7fd63c109600 2024-04-02T14:48:59.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.861+0000 7fd641000640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fd63c11a890 con 0x7fd63c108ce0 2024-04-02T14:48:59.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.861+0000 7fd639400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd63c109600 0x7fd63c1a6d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:59.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.861+0000 7fd639400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd63c109600 0x7fd63c1a6d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:42222/0 (socket says 172.21.15.90:42222) 2024-04-02T14:48:59.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.861+0000 7fd639400640 1 -- 172.21.15.90:0/3286149445 learned_addr learned my addr 172.21.15.90:0/3286149445 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:48:59.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.861+0000 7fd639e00640 1 --2- 172.21.15.90:0/3286149445 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd63c108ce0 0x7fd63c1a6850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:59.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.861+0000 7fd63a800640 1 --2- 172.21.15.90:0/3286149445 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd63c114c50 0x7fd63c1a0a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:59.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.863+0000 7fd639e00640 1 -- 172.21.15.90:0/3286149445 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd63c109600 msgr2=0x7fd63c1a6d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:48:59.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.863+0000 7fd639e00640 1 --2- 172.21.15.90:0/3286149445 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd63c109600 0x7fd63c1a6d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:59.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.863+0000 7fd639e00640 1 -- 172.21.15.90:0/3286149445 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd63c114c50 msgr2=0x7fd63c1a0a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:48:59.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.863+0000 7fd639e00640 1 --2- 172.21.15.90:0/3286149445 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd63c114c50 0x7fd63c1a0a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:59.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.863+0000 7fd639e00640 1 -- 172.21.15.90:0/3286149445 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd63c1a1180 con 0x7fd63c108ce0 2024-04-02T14:48:59.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.866+0000 7fd639e00640 1 --2- 172.21.15.90:0/3286149445 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd63c108ce0 0x7fd63c1a6850 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7fd624036040 tx=0x7fd62402cbf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:48:59.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.866+0000 7fd632a00640 1 -- 172.21.15.90:0/3286149445 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd624041070 con 0x7fd63c108ce0 2024-04-02T14:48:59.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.866+0000 7fd632a00640 1 -- 172.21.15.90:0/3286149445 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd624004450 con 0x7fd63c108ce0 2024-04-02T14:48:59.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.866+0000 7fd632a00640 1 -- 172.21.15.90:0/3286149445 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd624003e30 con 0x7fd63c108ce0 2024-04-02T14:48:59.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.866+0000 7fd641000640 1 -- 172.21.15.90:0/3286149445 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd63c1a1410 con 0x7fd63c108ce0 2024-04-02T14:48:59.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.867+0000 7fd641000640 1 -- 172.21.15.90:0/3286149445 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd63c10a4a0 con 0x7fd63c108ce0 2024-04-02T14:48:59.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.872+0000 7fd643de6740 1 Processor -- start 2024-04-02T14:48:59.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.872+0000 7fd643de6740 1 -- start start 2024-04-02T14:48:59.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.872+0000 7fd632a00640 1 -- 172.21.15.90:0/3286149445 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fd624044070 con 0x7fd63c108ce0 2024-04-02T14:48:59.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.872+0000 7fd643de6740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56391d5059f0 0x56391d504740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:59.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.873+0000 7fd643de6740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56391d504d10 0x56391d505190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:59.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.873+0000 7fd63a800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56391d5059f0 0x56391d504740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:59.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.873+0000 7fd63a800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56391d5059f0 0x56391d504740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:59218/0 (socket says 172.21.15.90:59218) 2024-04-02T14:48:59.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.873+0000 7fd63a800640 1 -- 172.21.15.90:0/2649071382 learned_addr learned my addr 172.21.15.90:0/2649071382 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:48:59.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.873+0000 7fd643de6740 1 --2- 172.21.15.90:0/2649071382 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56391d50e080 0x56391d510470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:59.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.873+0000 7fd643de6740 1 -- 172.21.15.90:0/2649071382 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x56391d445400 con 0x56391d50e080 2024-04-02T14:48:59.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.873+0000 7fd643de6740 1 -- 172.21.15.90:0/2649071382 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x56391d429f10 con 0x56391d504d10 2024-04-02T14:48:59.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.873+0000 7fd643de6740 1 -- 172.21.15.90:0/2649071382 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x56391d4696f0 con 0x56391d5059f0 2024-04-02T14:48:59.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.874+0000 7fd639e00640 1 --2- 172.21.15.90:0/2649071382 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56391d504d10 0x56391d505190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:59.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.874+0000 7fd632a00640 1 --2- 172.21.15.90:0/3286149445 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fd6200a3da0 0x7fd6200a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:59.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.874+0000 7fd632a00640 1 -- 172.21.15.90:0/3286149445 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 8550+0+0 (secure 0 0 0) 0x7fd6240f2680 con 0x7fd63c108ce0 2024-04-02T14:48:59.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.875+0000 7fd639400640 1 --2- 172.21.15.90:0/2649071382 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56391d50e080 0x56391d510470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:59.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.875+0000 7fd639e00640 1 -- 172.21.15.90:0/2649071382 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56391d5059f0 msgr2=0x56391d504740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:48:59.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.875+0000 7fd639e00640 1 --2- 172.21.15.90:0/2649071382 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56391d5059f0 0x56391d504740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:59.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.875+0000 7fd639e00640 1 -- 172.21.15.90:0/2649071382 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56391d50e080 msgr2=0x56391d510470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:48:59.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.875+0000 7fd639e00640 1 --2- 172.21.15.90:0/2649071382 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56391d50e080 0x56391d510470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:59.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.875+0000 7fd639e00640 1 -- 172.21.15.90:0/2649071382 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56391d17d3f0 con 0x56391d504d10 2024-04-02T14:48:59.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.875+0000 7fd63a800640 1 --2- 172.21.15.90:0/2649071382 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56391d5059f0 0x56391d504740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:48:59.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.875+0000 7fd639400640 1 --2- 172.21.15.90:0/2649071382 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56391d50e080 0x56391d510470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:48:59.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.875+0000 7fd63a800640 1 --2- 172.21.15.90:0/3286149445 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fd6200a3da0 0x7fd6200a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:59.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.876+0000 7fd639e00640 1 --2- 172.21.15.90:0/2649071382 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56391d504d10 0x56391d505190 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7fd6240efec0 tx=0x7fd624000bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d7be2d282e1834ff server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:48:59.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.876+0000 7fd62b400640 1 -- 172.21.15.90:0/2649071382 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd62404b070 con 0x56391d504d10 2024-04-02T14:48:59.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.876+0000 7fd63a800640 1 --2- 172.21.15.90:0/3286149445 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fd6200a3da0 0x7fd6200a6260 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fd6340125f0 tx=0x7fd634004a40 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:48:59.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.877+0000 7fd62b400640 1 -- 172.21.15.90:0/2649071382 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd6240bee10 con 0x56391d504d10 2024-04-02T14:48:59.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.878+0000 7fd643de6740 1 -- 172.21.15.90:0/2649071382 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56391d504d10 msgr2=0x56391d505190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:48:59.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.878+0000 7fd62b400640 1 -- 172.21.15.90:0/2649071382 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd62404b070 con 0x56391d504d10 2024-04-02T14:48:59.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.879+0000 7fd643de6740 1 --2- 172.21.15.90:0/2649071382 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56391d504d10 0x56391d505190 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7fd6240efec0 tx=0x7fd624000bc0 comp rx=0 tx=0).stop 2024-04-02T14:48:59.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.881+0000 7fd643de6740 1 -- 172.21.15.90:0/2649071382 shutdown_connections 2024-04-02T14:48:59.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.881+0000 7fd643de6740 1 --2- 172.21.15.90:0/2649071382 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56391d50e080 0x56391d510470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:59.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.881+0000 7fd643de6740 1 --2- 172.21.15.90:0/2649071382 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56391d504d10 0x56391d505190 unknown :-1 s=CLOSED pgs=963 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:59.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.881+0000 7fd643de6740 1 --2- 172.21.15.90:0/2649071382 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56391d5059f0 0x56391d504740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:59.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.881+0000 7fd643de6740 1 -- 172.21.15.90:0/2649071382 >> 172.21.15.90:0/2649071382 conn(0x56391d4cba50 msgr2=0x56391d4ff760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:48:59.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.882+0000 7fd643de6740 1 -- 172.21.15.90:0/2649071382 shutdown_connections 2024-04-02T14:48:59.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.882+0000 7fd643de6740 1 -- 172.21.15.90:0/2649071382 wait complete. 2024-04-02T14:48:59.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.882+0000 7fd643de6740 1 -- start start 2024-04-02T14:48:59.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.883+0000 7fd643de6740 1 Processor -- start 2024-04-02T14:48:59.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.884+0000 7fd643de6740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56391d504d10 0x56391d50b000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:59.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.885+0000 7fd643de6740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56391d5059f0 0x56391d50b540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:59.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.885+0000 7fd639e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56391d504d10 0x56391d50b000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:59.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.885+0000 7fd639e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56391d504d10 0x56391d50b000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:59230/0 (socket says 172.21.15.90:59230) 2024-04-02T14:48:59.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.885+0000 7fd639e00640 1 -- 172.21.15.90:0/3869966123 learned_addr learned my addr 172.21.15.90:0/3869966123 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:48:59.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.885+0000 7fd639400640 1 --2- 172.21.15.90:0/3869966123 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56391d5059f0 0x56391d50b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:59.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.886+0000 7fd643de6740 1 --2- 172.21.15.90:0/3869966123 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56391d50e080 0x56391d50ba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:59.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.886+0000 7fd643de6740 1 -- 172.21.15.90:0/3869966123 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x56391d4656a0 con 0x56391d50e080 2024-04-02T14:48:59.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.886+0000 7fd643de6740 1 -- 172.21.15.90:0/3869966123 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x56391d411980 con 0x56391d5059f0 2024-04-02T14:48:59.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.886+0000 7fd643de6740 1 -- 172.21.15.90:0/3869966123 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x56391d424cb0 con 0x56391d504d10 2024-04-02T14:48:59.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.886+0000 7fd643de6740 10 client.0 _ll_register_callbacks cb 0x56391d4246a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:48:59.882 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.886+0000 7fd63a800640 1 --2- 172.21.15.90:0/3869966123 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56391d50e080 0x56391d50ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:59.882 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.886+0000 7fd639400640 1 -- 172.21.15.90:0/3869966123 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56391d504d10 msgr2=0x56391d50b000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:48:59.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.886+0000 7fd639400640 1 --2- 172.21.15.90:0/3869966123 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56391d504d10 0x56391d50b000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:59.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.886+0000 7fd639400640 1 -- 172.21.15.90:0/3869966123 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56391d50e080 msgr2=0x56391d50ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:48:59.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.886+0000 7fd639400640 1 --2- 172.21.15.90:0/3869966123 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56391d50e080 0x56391d50ba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:59.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.887+0000 7fd639400640 1 -- 172.21.15.90:0/3869966123 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56391d2477d0 con 0x56391d5059f0 2024-04-02T14:48:59.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.887+0000 7fd639400640 1 --2- 172.21.15.90:0/3869966123 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56391d5059f0 0x56391d50b540 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7fd62c00e1f0 tx=0x7fd62c00e6b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f4d182187a25f1d7 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:48:59.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.887+0000 7fd61fe00640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:48:59.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.888+0000 7fd61fe00640 1 -- 172.21.15.90:0/3869966123 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd62c017020 con 0x56391d5059f0 2024-04-02T14:48:59.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.888+0000 7fd643de6740 1 -- 172.21.15.90:0/3869966123 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd6240bee10 con 0x56391d5059f0 2024-04-02T14:48:59.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.888+0000 7fd61fe00640 1 -- 172.21.15.90:0/3869966123 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 8550+0+0 (secure 0 0 0) 0x7fd62c013410 con 0x56391d5059f0 2024-04-02T14:48:59.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.889+0000 7fd61fe00640 1 -- 172.21.15.90:0/3869966123 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fd62c022050 con 0x56391d5059f0 2024-04-02T14:48:59.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.889+0000 7fd643de6740 10 client.29861 fetch_fsmap learned FSMap version 27 2024-04-02T14:48:59.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.889+0000 7fd643de6740 1 -- 172.21.15.90:0/3869966123 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x56391d5252a0 con 0x56391d5059f0 2024-04-02T14:48:59.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.890+0000 7fd61fe00640 1 -- 172.21.15.90:0/3869966123 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd6240bee10 con 0x56391d5059f0 2024-04-02T14:48:59.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.890+0000 7fd61fe00640 1 -- 172.21.15.90:0/3869966123 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fd62c012050 con 0x56391d5059f0 2024-04-02T14:48:59.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.890+0000 7fd643de6740 10 client.29861 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:48:59.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.890+0000 7fd643de6740 10 client.29861 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-04-02T14:48:59.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.890+0000 7fd643de6740 20 client.29861 populate_metadata read hostname 'smithi090' 2024-04-02T14:48:59.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.890+0000 7fd643de6740 1 --2- 172.21.15.90:0/3869966123 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x56391d5267a0 0x56391d528bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:59.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.890+0000 7fd643de6740 4 client.29861 mds_command: new command op to 24547 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:48:59.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.890+0000 7fd643de6740 1 -- 172.21.15.90:0/3869966123 --> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56391d429040 con 0x56391d5267a0 2024-04-02T14:48:59.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.891+0000 7fd639e00640 1 --2- 172.21.15.90:0/3869966123 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x56391d5267a0 0x56391d528bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:59.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.891+0000 7fd639e00640 1 --2- 172.21.15.90:0/3869966123 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x56391d5267a0 0x56391d528bb0 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=f75a3211a7b57f10 server_cookie=e0eced6241f9bee in_seq=0 out_seq=0 2024-04-02T14:48:59.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.891+0000 7fd61fe00640 10 client.29861 ms_handle_connect on v2:172.21.15.177:6838/664895663 2024-04-02T14:48:59.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.892+0000 7fd61fe00640 1 -- 172.21.15.90:0/3869966123 <== mds.2 v2:172.21.15.177:6838/664895663 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56391d429040 con 0x56391d5267a0 2024-04-02T14:48:59.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.892+0000 7fd61fe00640 10 client.29861 handle_command_reply: tid=0 2024-04-02T14:48:59.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.892+0000 7fd643de6740 1 client.29861 shutdown 2024-04-02T14:48:59.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.893+0000 7fd643de6740 1 -- 172.21.15.90:0/3869966123 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56391d5059f0 msgr2=0x56391d50b540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:48:59.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.893+0000 7fd643de6740 1 --2- 172.21.15.90:0/3869966123 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56391d5059f0 0x56391d50b540 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7fd62c00e1f0 tx=0x7fd62c00e6b0 comp rx=0 tx=0).stop 2024-04-02T14:48:59.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.894+0000 7fd643de6740 1 -- 172.21.15.90:0/3869966123 shutdown_connections 2024-04-02T14:48:59.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.895+0000 7fd643de6740 1 --2- 172.21.15.90:0/3869966123 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x56391d5267a0 0x56391d528bb0 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:59.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.895+0000 7fd643de6740 1 --2- 172.21.15.90:0/3869966123 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56391d50e080 0x56391d50ba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:59.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.895+0000 7fd643de6740 1 --2- 172.21.15.90:0/3869966123 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56391d5059f0 0x56391d50b540 unknown :-1 s=CLOSED pgs=964 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:59.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.895+0000 7fd643de6740 1 --2- 172.21.15.90:0/3869966123 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56391d504d10 0x56391d50b000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:59.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.895+0000 7fd643de6740 1 -- 172.21.15.90:0/3869966123 >> 172.21.15.90:0/3869966123 conn(0x56391d4cba50 msgr2=0x56391d5014b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:48:59.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.895+0000 7fd61fe00640 0 client.29861 ms_handle_reset on v2:172.21.15.177:6838/664895663 2024-04-02T14:48:59.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.896+0000 7fd643de6740 1 -- 172.21.15.90:0/3869966123 shutdown_connections 2024-04-02T14:48:59.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.896+0000 7fd643de6740 1 -- 172.21.15.90:0/3869966123 wait complete. 2024-04-02T14:48:59.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.896+0000 7fd643de6740 20 client.29861 trim_cache size 0 max 16384 2024-04-02T14:48:59.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.919+0000 7fd643de6740 1 Processor -- start 2024-04-02T14:48:59.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.919+0000 7fd643de6740 1 -- start start 2024-04-02T14:48:59.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.920+0000 7fd643de6740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56391d504d10 0x56391d522150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:59.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.920+0000 7fd643de6740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56391d5059f0 0x56391d50b8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:59.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.920+0000 7fd643de6740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56391d50e080 0x56391d50ce30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:59.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.920+0000 7fd639400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56391d5059f0 0x56391d50b8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:59.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.920+0000 7fd639400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56391d5059f0 0x56391d50b8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:59238/0 (socket says 172.21.15.90:59238) 2024-04-02T14:48:59.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.920+0000 7fd639400640 1 -- 172.21.15.90:0/1973964297 learned_addr learned my addr 172.21.15.90:0/1973964297 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:48:59.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.920+0000 7fd643de6740 1 -- 172.21.15.90:0/1973964297 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fd63c1b3f20 con 0x56391d504d10 2024-04-02T14:48:59.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.920+0000 7fd643de6740 1 -- 172.21.15.90:0/1973964297 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fd63c1b5a60 con 0x56391d50e080 2024-04-02T14:48:59.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.920+0000 7fd643de6740 1 -- 172.21.15.90:0/1973964297 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fd63c1b45f0 con 0x56391d5059f0 2024-04-02T14:48:59.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.921+0000 7fd639e00640 1 --2- 172.21.15.90:0/1973964297 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56391d504d10 0x56391d522150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:59.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.921+0000 7fd639400640 1 -- 172.21.15.90:0/1973964297 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56391d50e080 msgr2=0x56391d50ce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:48:59.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.921+0000 7fd639400640 1 --2- 172.21.15.90:0/1973964297 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56391d50e080 0x56391d50ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:59.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.921+0000 7fd639400640 1 -- 172.21.15.90:0/1973964297 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56391d504d10 msgr2=0x56391d522150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:48:59.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.921+0000 7fd639400640 1 --2- 172.21.15.90:0/1973964297 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56391d504d10 0x56391d522150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:59.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.921+0000 7fd639400640 1 -- 172.21.15.90:0/1973964297 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56391d503d40 con 0x56391d5059f0 2024-04-02T14:48:59.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.921+0000 7fd639400640 1 --2- 172.21.15.90:0/1973964297 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56391d5059f0 0x56391d50b8a0 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7fd62c000c00 tx=0x7fd62c00ef60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=25c0e1681d1a0a16 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:48:59.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.921+0000 7fd62aa00640 1 -- 172.21.15.90:0/1973964297 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd62c02a070 con 0x56391d5059f0 2024-04-02T14:48:59.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.922+0000 7fd62aa00640 1 -- 172.21.15.90:0/1973964297 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd62c01bc80 con 0x56391d5059f0 2024-04-02T14:48:59.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.922+0000 7fd62aa00640 1 -- 172.21.15.90:0/1973964297 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd62c017020 con 0x56391d5059f0 2024-04-02T14:48:59.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.924+0000 7fd643de6740 1 -- 172.21.15.90:0/1973964297 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56391d5059f0 msgr2=0x56391d50b8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:48:59.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.924+0000 7fd643de6740 1 --2- 172.21.15.90:0/1973964297 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56391d5059f0 0x56391d50b8a0 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7fd62c000c00 tx=0x7fd62c00ef60 comp rx=0 tx=0).stop 2024-04-02T14:48:59.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.924+0000 7fd643de6740 1 -- 172.21.15.90:0/1973964297 shutdown_connections 2024-04-02T14:48:59.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.924+0000 7fd643de6740 1 --2- 172.21.15.90:0/1973964297 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56391d50e080 0x56391d50ce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:59.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.924+0000 7fd643de6740 1 --2- 172.21.15.90:0/1973964297 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56391d5059f0 0x56391d50b8a0 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:59.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.924+0000 7fd643de6740 1 --2- 172.21.15.90:0/1973964297 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56391d504d10 0x56391d522150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:59.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.924+0000 7fd643de6740 1 -- 172.21.15.90:0/1973964297 >> 172.21.15.90:0/1973964297 conn(0x56391d4cba50 msgr2=0x56391d50c720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:48:59.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.924+0000 7fd643de6740 1 -- 172.21.15.90:0/1973964297 shutdown_connections 2024-04-02T14:48:59.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.924+0000 7fd643de6740 1 -- 172.21.15.90:0/1973964297 wait complete. 2024-04-02T14:48:59.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.924+0000 7fd643de6740 1 -- start start 2024-04-02T14:48:59.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.924+0000 7fd643de6740 1 Processor -- start 2024-04-02T14:48:59.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.925+0000 7fd643de6740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56391d504d10 0x56391d51e730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:59.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.925+0000 7fd643de6740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56391d5059f0 0x56391d519c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:59.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.925+0000 7fd643de6740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56391d50e080 0x56391d51a180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:59.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.925+0000 7fd643de6740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fd63c1b5390 con 0x56391d504d10 2024-04-02T14:48:59.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.925+0000 7fd643de6740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fd63c1b4cc0 con 0x56391d50e080 2024-04-02T14:48:59.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.925+0000 7fd643de6740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fd63c1b1f20 con 0x56391d5059f0 2024-04-02T14:48:59.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.925+0000 7fd643de6740 10 client.0 _ll_register_callbacks cb 0x7fd62c009c20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:48:59.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.926+0000 7fd639400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56391d5059f0 0x56391d519c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:59.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.926+0000 7fd639e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56391d504d10 0x56391d51e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:59.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.926+0000 7fd639e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56391d504d10 0x56391d51e730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41440/0 (socket says 172.21.15.90:41440) 2024-04-02T14:48:59.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.926+0000 7fd639e00640 1 -- 172.21.15.90:0/268363775 learned_addr learned my addr 172.21.15.90:0/268363775 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:48:59.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.926+0000 7fd63a800640 1 --2- 172.21.15.90:0/268363775 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56391d50e080 0x56391d51a180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:59.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.927+0000 7fd639400640 1 -- 172.21.15.90:0/268363775 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56391d50e080 msgr2=0x56391d51a180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:48:59.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.927+0000 7fd639400640 1 --2- 172.21.15.90:0/268363775 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56391d50e080 0x56391d51a180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:59.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.927+0000 7fd639400640 1 -- 172.21.15.90:0/268363775 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56391d504d10 msgr2=0x56391d51e730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:48:59.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.927+0000 7fd639400640 1 --2- 172.21.15.90:0/268363775 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56391d504d10 0x56391d51e730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:59.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.927+0000 7fd639400640 1 -- 172.21.15.90:0/268363775 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56391d51a6c0 con 0x56391d5059f0 2024-04-02T14:48:59.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.927+0000 7fd639400640 1 --2- 172.21.15.90:0/268363775 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56391d5059f0 0x56391d519c40 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x56391d50c190 tx=0x7fd62c00efd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cb5dbae2bfbd4e10 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:48:59.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.927+0000 7fd61fe00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:48:59.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.927+0000 7fd61fe00640 1 -- 172.21.15.90:0/268363775 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd62c017040 con 0x56391d5059f0 2024-04-02T14:48:59.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.927+0000 7fd61fe00640 1 -- 172.21.15.90:0/268363775 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 8550+0+0 (secure 0 0 0) 0x7fd62c026cf0 con 0x56391d5059f0 2024-04-02T14:48:59.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.927+0000 7fd643de6740 1 -- 172.21.15.90:0/268363775 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd62c01bc80 con 0x56391d5059f0 2024-04-02T14:48:59.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.928+0000 7fd61fe00640 1 -- 172.21.15.90:0/268363775 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fd62c02a040 con 0x56391d5059f0 2024-04-02T14:48:59.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.928+0000 7fd643de6740 10 client.29656 fetch_fsmap learned FSMap version 27 2024-04-02T14:48:59.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.928+0000 7fd643de6740 1 -- 172.21.15.90:0/268363775 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x56391d58ca60 con 0x56391d5059f0 2024-04-02T14:48:59.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.928+0000 7fd61fe00640 1 -- 172.21.15.90:0/268363775 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd62c01bc80 con 0x56391d5059f0 2024-04-02T14:48:59.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.928+0000 7fd61fe00640 1 -- 172.21.15.90:0/268363775 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fd62c012050 con 0x56391d5059f0 2024-04-02T14:48:59.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.928+0000 7fd643de6740 10 client.29656 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:48:59.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.928+0000 7fd643de6740 10 client.29656 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-04-02T14:48:59.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.928+0000 7fd643de6740 20 client.29656 populate_metadata read hostname 'smithi090' 2024-04-02T14:48:59.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.928+0000 7fd643de6740 1 --2- 172.21.15.90:0/268363775 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x56391d58d0b0 0x56391d58f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:48:59.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.928+0000 7fd643de6740 4 client.29656 mds_command: new command op to 24547 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-02T14:48:59.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.928+0000 7fd643de6740 1 -- 172.21.15.90:0/268363775 --> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56391d476c30 con 0x56391d58d0b0 2024-04-02T14:48:59.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.929+0000 7fd639e00640 1 --2- 172.21.15.90:0/268363775 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x56391d58d0b0 0x56391d58f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:48:59.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.929+0000 7fd639e00640 1 --2- 172.21.15.90:0/268363775 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x56391d58d0b0 0x56391d58f4d0 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=1ecf5676012dc15d server_cookie=f37a39136c8d4605 in_seq=0 out_seq=0 2024-04-02T14:48:59.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.929+0000 7fd61fe00640 10 client.29656 ms_handle_connect on v2:172.21.15.177:6838/664895663 2024-04-02T14:48:59.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.929+0000 7fd61fe00640 1 -- 172.21.15.90:0/268363775 <== mds.2 v2:172.21.15.177:6838/664895663 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56391d476c30 con 0x56391d58d0b0 2024-04-02T14:48:59.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.929+0000 7fd61fe00640 10 client.29656 handle_command_reply: tid=0 2024-04-02T14:48:59.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.929+0000 7fd643de6740 1 client.29656 shutdown 2024-04-02T14:48:59.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.930+0000 7fd643de6740 1 -- 172.21.15.90:0/268363775 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56391d5059f0 msgr2=0x56391d519c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:48:59.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.930+0000 7fd643de6740 1 --2- 172.21.15.90:0/268363775 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56391d5059f0 0x56391d519c40 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x56391d50c190 tx=0x7fd62c00efd0 comp rx=0 tx=0).stop 2024-04-02T14:48:59.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.930+0000 7fd643de6740 1 -- 172.21.15.90:0/268363775 shutdown_connections 2024-04-02T14:48:59.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.930+0000 7fd643de6740 1 --2- 172.21.15.90:0/268363775 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x56391d58d0b0 0x56391d58f4d0 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:59.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.930+0000 7fd643de6740 1 --2- 172.21.15.90:0/268363775 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56391d50e080 0x56391d51a180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:59.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.930+0000 7fd643de6740 1 --2- 172.21.15.90:0/268363775 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56391d5059f0 0x56391d519c40 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:59.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.930+0000 7fd643de6740 1 --2- 172.21.15.90:0/268363775 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56391d504d10 0x56391d51e730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:59.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.930+0000 7fd61fe00640 0 client.29656 ms_handle_reset on v2:172.21.15.177:6838/664895663 2024-04-02T14:48:59.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.930+0000 7fd643de6740 1 -- 172.21.15.90:0/268363775 >> 172.21.15.90:0/268363775 conn(0x56391d4cba50 msgr2=0x56391d500870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:48:59.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.930+0000 7fd643de6740 1 -- 172.21.15.90:0/268363775 shutdown_connections 2024-04-02T14:48:59.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.930+0000 7fd643de6740 1 -- 172.21.15.90:0/268363775 wait complete. 2024-04-02T14:48:59.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.930+0000 7fd643de6740 20 client.29656 trim_cache size 0 max 16384 2024-04-02T14:48:59.927 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-02T14:48:59.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.932+0000 7fd641000640 1 -- 172.21.15.90:0/3286149445 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fd6200a3da0 msgr2=0x7fd6200a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:48:59.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.932+0000 7fd641000640 1 --2- 172.21.15.90:0/3286149445 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fd6200a3da0 0x7fd6200a6260 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fd6340125f0 tx=0x7fd634004a40 comp rx=0 tx=0).stop 2024-04-02T14:48:59.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.932+0000 7fd641000640 1 -- 172.21.15.90:0/3286149445 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd63c108ce0 msgr2=0x7fd63c1a6850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:48:59.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.932+0000 7fd641000640 1 --2- 172.21.15.90:0/3286149445 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd63c108ce0 0x7fd63c1a6850 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7fd624036040 tx=0x7fd62402cbf0 comp rx=0 tx=0).stop 2024-04-02T14:48:59.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.932+0000 7fd641000640 1 -- 172.21.15.90:0/3286149445 shutdown_connections 2024-04-02T14:48:59.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.932+0000 7fd641000640 1 --2- 172.21.15.90:0/3286149445 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fd6200a3da0 0x7fd6200a6260 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:59.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.932+0000 7fd641000640 1 --2- 172.21.15.90:0/3286149445 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd63c114c50 0x7fd63c1a0a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:59.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.932+0000 7fd641000640 1 --2- 172.21.15.90:0/3286149445 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd63c109600 0x7fd63c1a6d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:59.929 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.932+0000 7fd641000640 1 --2- 172.21.15.90:0/3286149445 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd63c108ce0 0x7fd63c1a6850 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:48:59.929 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.932+0000 7fd641000640 1 -- 172.21.15.90:0/3286149445 >> 172.21.15.90:0/3286149445 conn(0x7fd63c06cbe0 msgr2=0x7fd63c0707a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:48:59.929 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.932+0000 7fd641000640 1 -- 172.21.15.90:0/3286149445 shutdown_connections 2024-04-02T14:48:59.929 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:48:59.932+0000 7fd641000640 1 -- 172.21.15.90:0/3286149445 wait complete. 2024-04-02T14:48:59.946 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-02T14:49:00.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.235+0000 7f3edbe00640 1 -- 172.21.15.90:0/2339102338 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3ed4076950 msgr2=0x7f3ed4112400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:49:00.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.235+0000 7f3edbe00640 1 --2- 172.21.15.90:0/2339102338 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3ed4076950 0x7f3ed4112400 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7f3ec8009930 tx=0x7f3ec802bc30 comp rx=0 tx=0).stop 2024-04-02T14:49:00.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.235+0000 7f3edbe00640 1 -- 172.21.15.90:0/2339102338 shutdown_connections 2024-04-02T14:49:00.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.235+0000 7f3edbe00640 1 --2- 172.21.15.90:0/2339102338 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3ed4112940 0x7f3ed4114d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:00.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.235+0000 7f3edbe00640 1 --2- 172.21.15.90:0/2339102338 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3ed4076950 0x7f3ed4112400 unknown :-1 s=CLOSED pgs=965 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:00.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.236+0000 7f3edbe00640 1 --2- 172.21.15.90:0/2339102338 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3ed4076030 0x7f3ed4076410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:00.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.236+0000 7f3edbe00640 1 -- 172.21.15.90:0/2339102338 >> 172.21.15.90:0/2339102338 conn(0x7f3ed40ff7b0 msgr2=0x7f3ed4101bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:49:00.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.236+0000 7f3edbe00640 1 -- 172.21.15.90:0/2339102338 shutdown_connections 2024-04-02T14:49:00.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.236+0000 7f3edbe00640 1 -- 172.21.15.90:0/2339102338 wait complete. 2024-04-02T14:49:00.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.237+0000 7f3edbe00640 1 Processor -- start 2024-04-02T14:49:00.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.239+0000 7f3edbe00640 1 -- start start 2024-04-02T14:49:00.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.239+0000 7f3edbe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3ed4076030 0x7f3ed4111f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:00.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.240+0000 7f3edbe00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3ed4076950 0x7f3ed410afe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:00.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.240+0000 7f3edbe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3ed4112940 0x7f3ed410b520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:00.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.240+0000 7f3ed8a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3ed4076950 0x7f3ed410afe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:00.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.240+0000 7f3ed8a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3ed4076950 0x7f3ed410afe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:42272/0 (socket says 172.21.15.90:42272) 2024-04-02T14:49:00.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.240+0000 7f3ed8a00640 1 -- 172.21.15.90:0/3382440547 learned_addr learned my addr 172.21.15.90:0/3382440547 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:49:00.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.240+0000 7f3ed9e00640 1 --2- 172.21.15.90:0/3382440547 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3ed4112940 0x7f3ed410b520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:00.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.240+0000 7f3edbe00640 1 -- 172.21.15.90:0/3382440547 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f3ed41184f0 con 0x7f3ed4076030 2024-04-02T14:49:00.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.241+0000 7f3ed9400640 1 --2- 172.21.15.90:0/3382440547 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3ed4076030 0x7f3ed4111f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:00.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.241+0000 7f3edbe00640 1 -- 172.21.15.90:0/3382440547 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f3ed4118370 con 0x7f3ed4076950 2024-04-02T14:49:00.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.241+0000 7f3edbe00640 1 -- 172.21.15.90:0/3382440547 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f3ed4118670 con 0x7f3ed4112940 2024-04-02T14:49:00.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.242+0000 7f3ed8a00640 1 -- 172.21.15.90:0/3382440547 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3ed4112940 msgr2=0x7f3ed410b520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:49:00.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.242+0000 7f3ed8a00640 1 --2- 172.21.15.90:0/3382440547 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3ed4112940 0x7f3ed410b520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:00.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.242+0000 7f3ed8a00640 1 -- 172.21.15.90:0/3382440547 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3ed4076030 msgr2=0x7f3ed4111f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:49:00.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.242+0000 7f3ed8a00640 1 --2- 172.21.15.90:0/3382440547 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3ed4076030 0x7f3ed4111f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:00.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.242+0000 7f3ed8a00640 1 -- 172.21.15.90:0/3382440547 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ed410bb60 con 0x7f3ed4076950 2024-04-02T14:49:00.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.242+0000 7f3ed8a00640 1 --2- 172.21.15.90:0/3382440547 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3ed4076950 0x7f3ed410afe0 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7f3ec8009900 tx=0x7f3ec8004360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:49:00.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.244+0000 7f3ed2000640 1 -- 172.21.15.90:0/3382440547 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ec8043020 con 0x7f3ed4076950 2024-04-02T14:49:00.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.244+0000 7f3ed2000640 1 -- 172.21.15.90:0/3382440547 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3ec80054e0 con 0x7f3ed4076950 2024-04-02T14:49:00.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.244+0000 7f3ed2000640 1 -- 172.21.15.90:0/3382440547 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ec8003b80 con 0x7f3ed4076950 2024-04-02T14:49:00.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.244+0000 7f3edbe00640 1 -- 172.21.15.90:0/3382440547 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ed410bdf0 con 0x7f3ed4076950 2024-04-02T14:49:00.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.244+0000 7f3edbe00640 1 -- 172.21.15.90:0/3382440547 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ed410fd80 con 0x7f3ed4076950 2024-04-02T14:49:00.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.249+0000 7f3ed2000640 1 -- 172.21.15.90:0/3382440547 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f3ec8046020 con 0x7f3ed4076950 2024-04-02T14:49:00.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.253+0000 7f3ed2000640 1 --2- 172.21.15.90:0/3382440547 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f3ebc0a3e20 0x7f3ebc0a62e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:00.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.253+0000 7f3ed2000640 1 -- 172.21.15.90:0/3382440547 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 8550+0+0 (secure 0 0 0) 0x7f3ec80f37c0 con 0x7f3ed4076950 2024-04-02T14:49:00.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.254+0000 7f3ed9400640 1 --2- 172.21.15.90:0/3382440547 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f3ebc0a3e20 0x7f3ebc0a62e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:00.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.255+0000 7f3ed9400640 1 --2- 172.21.15.90:0/3382440547 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f3ebc0a3e20 0x7f3ebc0a62e0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f3ec4005d00 tx=0x7f3ec400b410 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:49:00.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.260+0000 7f3edea82740 1 Processor -- start 2024-04-02T14:49:00.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.260+0000 7f3edea82740 1 -- start start 2024-04-02T14:49:00.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.260+0000 7f3edea82740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5636309ed9f0 0x5636309ec740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:00.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.260+0000 7f3edea82740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5636309ecd10 0x5636309ed190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:00.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.260+0000 7f3edea82740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5636309f6080 0x5636309f8470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:00.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.262+0000 7f3ed8a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5636309ecd10 0x5636309ed190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:00.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.262+0000 7f3ed8a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5636309ecd10 0x5636309ed190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:42284/0 (socket says 172.21.15.90:42284) 2024-04-02T14:49:00.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.262+0000 7f3ed8a00640 1 -- 172.21.15.90:0/4090407894 learned_addr learned my addr 172.21.15.90:0/4090407894 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:49:00.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.262+0000 7f3ed9e00640 1 --2- 172.21.15.90:0/4090407894 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5636309f6080 0x5636309f8470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:00.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.262+0000 7f3edea82740 1 -- 172.21.15.90:0/4090407894 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x56363092d400 con 0x5636309ed9f0 2024-04-02T14:49:00.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.262+0000 7f3edea82740 1 -- 172.21.15.90:0/4090407894 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x563630911f10 con 0x5636309ecd10 2024-04-02T14:49:00.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.262+0000 7f3edea82740 1 -- 172.21.15.90:0/4090407894 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5636309516f0 con 0x5636309f6080 2024-04-02T14:49:00.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.262+0000 7f3ed8a00640 1 -- 172.21.15.90:0/4090407894 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5636309f6080 msgr2=0x5636309f8470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:49:00.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.262+0000 7f3ed8a00640 1 --2- 172.21.15.90:0/4090407894 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5636309f6080 0x5636309f8470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:00.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.263+0000 7f3ed8a00640 1 -- 172.21.15.90:0/4090407894 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5636309ed9f0 msgr2=0x5636309ec740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:49:00.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.263+0000 7f3ed8a00640 1 --2- 172.21.15.90:0/4090407894 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5636309ed9f0 0x5636309ec740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:00.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.263+0000 7f3ed8a00640 1 -- 172.21.15.90:0/4090407894 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5636306653f0 con 0x5636309ecd10 2024-04-02T14:49:00.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.264+0000 7f3ed8a00640 1 --2- 172.21.15.90:0/4090407894 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5636309ecd10 0x5636309ed190 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7f3ec80bf780 tx=0x7f3ec8004ee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=57bf7c8eda188ef1 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:49:00.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.264+0000 7f3ec2600640 1 -- 172.21.15.90:0/4090407894 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ec80bfef0 con 0x5636309ecd10 2024-04-02T14:49:00.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.264+0000 7f3ec2600640 1 -- 172.21.15.90:0/4090407894 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3ec80bf7f0 con 0x5636309ecd10 2024-04-02T14:49:00.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.264+0000 7f3ec2600640 1 -- 172.21.15.90:0/4090407894 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ec80bbe60 con 0x5636309ecd10 2024-04-02T14:49:00.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.266+0000 7f3edea82740 1 -- 172.21.15.90:0/4090407894 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5636309ecd10 msgr2=0x5636309ed190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:49:00.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.266+0000 7f3edea82740 1 --2- 172.21.15.90:0/4090407894 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5636309ecd10 0x5636309ed190 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7f3ec80bf780 tx=0x7f3ec8004ee0 comp rx=0 tx=0).stop 2024-04-02T14:49:00.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.266+0000 7f3edea82740 1 -- 172.21.15.90:0/4090407894 shutdown_connections 2024-04-02T14:49:00.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.266+0000 7f3edea82740 1 --2- 172.21.15.90:0/4090407894 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5636309f6080 0x5636309f8470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:00.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.266+0000 7f3edea82740 1 --2- 172.21.15.90:0/4090407894 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5636309ecd10 0x5636309ed190 unknown :-1 s=CLOSED pgs=967 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:00.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.266+0000 7f3edea82740 1 --2- 172.21.15.90:0/4090407894 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5636309ed9f0 0x5636309ec740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:00.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.266+0000 7f3edea82740 1 -- 172.21.15.90:0/4090407894 >> 172.21.15.90:0/4090407894 conn(0x5636309b3a50 msgr2=0x5636309e7760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:49:00.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.266+0000 7f3edea82740 1 -- 172.21.15.90:0/4090407894 shutdown_connections 2024-04-02T14:49:00.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.267+0000 7f3edea82740 1 -- 172.21.15.90:0/4090407894 wait complete. 2024-04-02T14:49:00.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.268+0000 7f3edea82740 1 -- start start 2024-04-02T14:49:00.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.268+0000 7f3edea82740 1 Processor -- start 2024-04-02T14:49:00.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.270+0000 7f3edea82740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5636309ecd10 0x5636309f3160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:00.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.271+0000 7f3edea82740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5636309ed9f0 0x5636309f36a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:00.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.271+0000 7f3edea82740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5636309f6080 0x5636309f3be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:00.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.271+0000 7f3edea82740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x56363094d6a0 con 0x5636309f6080 2024-04-02T14:49:00.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.271+0000 7f3edea82740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x5636308f9980 con 0x5636309ed9f0 2024-04-02T14:49:00.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.271+0000 7f3edea82740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x56363090ccb0 con 0x5636309ecd10 2024-04-02T14:49:00.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.271+0000 7f3edea82740 10 client.0 _ll_register_callbacks cb 0x56363090c6a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:49:00.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.271+0000 7f3ed9400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5636309ecd10 0x5636309f3160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:00.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.271+0000 7f3ed9400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5636309ecd10 0x5636309f3160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:59284/0 (socket says 172.21.15.90:59284) 2024-04-02T14:49:00.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.271+0000 7f3ed9400640 1 -- 172.21.15.90:0/953998498 learned_addr learned my addr 172.21.15.90:0/953998498 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:49:00.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.272+0000 7f3ed8a00640 1 --2- 172.21.15.90:0/953998498 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5636309ed9f0 0x5636309f36a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:00.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.272+0000 7f3ed9e00640 1 --2- 172.21.15.90:0/953998498 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5636309f6080 0x5636309f3be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:00.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.273+0000 7f3ed8a00640 1 -- 172.21.15.90:0/953998498 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5636309ecd10 msgr2=0x5636309f3160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:49:00.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.273+0000 7f3ed8a00640 1 --2- 172.21.15.90:0/953998498 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5636309ecd10 0x5636309f3160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:00.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.273+0000 7f3ed8a00640 1 -- 172.21.15.90:0/953998498 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5636309f6080 msgr2=0x5636309f3be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:49:00.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.273+0000 7f3ed8a00640 1 --2- 172.21.15.90:0/953998498 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5636309f6080 0x5636309f3be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:00.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.273+0000 7f3ed8a00640 1 -- 172.21.15.90:0/953998498 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56363072f7d0 con 0x5636309ed9f0 2024-04-02T14:49:00.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.274+0000 7f3ed8a00640 1 --2- 172.21.15.90:0/953998498 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5636309ed9f0 0x5636309f36a0 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x5636309edf40 tx=0x7f3ec80bf6f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a9963f3767ca67b0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:49:00.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.274+0000 7f3eb6a00640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:49:00.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.275+0000 7f3eb6a00640 1 -- 172.21.15.90:0/953998498 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ec80c3ba0 con 0x5636309ed9f0 2024-04-02T14:49:00.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.275+0000 7f3eb6a00640 1 -- 172.21.15.90:0/953998498 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 8550+0+0 (secure 0 0 0) 0x7f3ec80bdaf0 con 0x5636309ed9f0 2024-04-02T14:49:00.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.275+0000 7f3edea82740 1 -- 172.21.15.90:0/953998498 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3ec80bf7f0 con 0x5636309ed9f0 2024-04-02T14:49:00.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.275+0000 7f3eb6a00640 1 -- 172.21.15.90:0/953998498 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f3ec80c8460 con 0x5636309ed9f0 2024-04-02T14:49:00.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.275+0000 7f3edea82740 10 client.29876 fetch_fsmap learned FSMap version 27 2024-04-02T14:49:00.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.275+0000 7f3edea82740 1 -- 172.21.15.90:0/953998498 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x563630a04cc0 con 0x5636309ed9f0 2024-04-02T14:49:00.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.276+0000 7f3eb6a00640 1 -- 172.21.15.90:0/953998498 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3ec80bf7f0 con 0x5636309ed9f0 2024-04-02T14:49:00.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.276+0000 7f3eb6a00640 1 -- 172.21.15.90:0/953998498 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f3ec80bec40 con 0x5636309ed9f0 2024-04-02T14:49:00.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.276+0000 7f3edea82740 10 client.29876 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:49:00.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.276+0000 7f3edea82740 10 client.29876 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2024-04-02T14:49:00.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.277+0000 7f3edea82740 20 client.29876 populate_metadata read hostname 'smithi090' 2024-04-02T14:49:00.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.277+0000 7f3edea82740 1 --2- 172.21.15.90:0/953998498 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x563630a06010 0x563630a08400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:00.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.277+0000 7f3edea82740 4 client.29876 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:49:00.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.277+0000 7f3edea82740 1 -- 172.21.15.90:0/953998498 --> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563630911040 con 0x563630a06010 2024-04-02T14:49:00.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.277+0000 7f3ed9400640 1 --2- 172.21.15.90:0/953998498 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x563630a06010 0x563630a08400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:00.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.278+0000 7f3ed9400640 1 --2- 172.21.15.90:0/953998498 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x563630a06010 0x563630a08400 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=11af8825ef6920c7 server_cookie=a845b6ecd0c49fa8 in_seq=0 out_seq=0 2024-04-02T14:49:00.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.278+0000 7f3eb6a00640 10 client.29876 ms_handle_connect on v2:172.21.15.90:6838/1498915870 2024-04-02T14:49:00.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.281+0000 7f3eb6a00640 1 -- 172.21.15.90:0/953998498 <== mds.1 v2:172.21.15.90:6838/1498915870 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x563630911040 con 0x563630a06010 2024-04-02T14:49:00.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.281+0000 7f3eb6a00640 10 client.29876 handle_command_reply: tid=0 2024-04-02T14:49:00.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.281+0000 7f3edea82740 1 client.29876 shutdown 2024-04-02T14:49:00.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.283+0000 7f3edea82740 1 -- 172.21.15.90:0/953998498 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5636309ed9f0 msgr2=0x5636309f36a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:49:00.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.283+0000 7f3edea82740 1 --2- 172.21.15.90:0/953998498 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5636309ed9f0 0x5636309f36a0 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x5636309edf40 tx=0x7f3ec80bf6f0 comp rx=0 tx=0).stop 2024-04-02T14:49:00.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.283+0000 7f3edea82740 1 -- 172.21.15.90:0/953998498 shutdown_connections 2024-04-02T14:49:00.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.284+0000 7f3edea82740 1 --2- 172.21.15.90:0/953998498 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x563630a06010 0x563630a08400 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:00.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.284+0000 7f3edea82740 1 --2- 172.21.15.90:0/953998498 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5636309f6080 0x5636309f3be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:00.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.284+0000 7f3edea82740 1 --2- 172.21.15.90:0/953998498 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5636309ed9f0 0x5636309f36a0 unknown :-1 s=CLOSED pgs=968 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:00.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.284+0000 7f3edea82740 1 --2- 172.21.15.90:0/953998498 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5636309ecd10 0x5636309f3160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:00.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.284+0000 7f3edea82740 1 -- 172.21.15.90:0/953998498 >> 172.21.15.90:0/953998498 conn(0x5636309b3a50 msgr2=0x5636309e94b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:49:00.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.284+0000 7f3eb6a00640 0 client.29876 ms_handle_reset on v2:172.21.15.90:6838/1498915870 2024-04-02T14:49:00.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.285+0000 7f3edea82740 1 -- 172.21.15.90:0/953998498 shutdown_connections 2024-04-02T14:49:00.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.286+0000 7f3edea82740 1 -- 172.21.15.90:0/953998498 wait complete. 2024-04-02T14:49:00.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.286+0000 7f3edea82740 20 client.29876 trim_cache size 0 max 16384 2024-04-02T14:49:00.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.321+0000 7f3edea82740 1 Processor -- start 2024-04-02T14:49:00.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.321+0000 7f3edea82740 1 -- start start 2024-04-02T14:49:00.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.321+0000 7f3edea82740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5636309ecd10 0x563630a01780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:00.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.321+0000 7f3edea82740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5636309f6080 0x5636309f3a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:00.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.321+0000 7f3edea82740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563630a0dc60 0x563630a0e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:00.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.321+0000 7f3edea82740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f3ed41bc800 con 0x5636309ecd10 2024-04-02T14:49:00.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.321+0000 7f3edea82740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f3ed41be340 con 0x563630a0dc60 2024-04-02T14:49:00.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.321+0000 7f3edea82740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f3ed41bced0 con 0x5636309f6080 2024-04-02T14:49:00.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.322+0000 7f3ed9e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563630a0dc60 0x563630a0e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:00.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.322+0000 7f3ed9e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563630a0dc60 0x563630a0e0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:42298/0 (socket says 172.21.15.90:42298) 2024-04-02T14:49:00.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.322+0000 7f3ed9e00640 1 -- 172.21.15.90:0/259390116 learned_addr learned my addr 172.21.15.90:0/259390116 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:49:00.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.322+0000 7f3ed9400640 1 --2- 172.21.15.90:0/259390116 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5636309ecd10 0x563630a01780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:00.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.322+0000 7f3ed9e00640 1 -- 172.21.15.90:0/259390116 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5636309f6080 msgr2=0x5636309f3a00 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-02T14:49:00.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.322+0000 7f3ed9e00640 1 --2- 172.21.15.90:0/259390116 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5636309f6080 0x5636309f3a00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:00.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.322+0000 7f3ed9e00640 1 -- 172.21.15.90:0/259390116 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5636309ecd10 msgr2=0x563630a01780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:49:00.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.322+0000 7f3ed9e00640 1 --2- 172.21.15.90:0/259390116 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5636309ecd10 0x563630a01780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:00.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.322+0000 7f3ed9e00640 1 -- 172.21.15.90:0/259390116 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5636309ebd40 con 0x563630a0dc60 2024-04-02T14:49:00.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.322+0000 7f3ed9e00640 1 --2- 172.21.15.90:0/259390116 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563630a0dc60 0x563630a0e0e0 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7f3ecc003aa0 tx=0x7f3ecc003f60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=86f90487714efba9 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:49:00.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.324+0000 7f3ec1c00640 1 -- 172.21.15.90:0/259390116 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ecc020070 con 0x563630a0dc60 2024-04-02T14:49:00.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.324+0000 7f3ec1c00640 1 -- 172.21.15.90:0/259390116 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3ecc010910 con 0x563630a0dc60 2024-04-02T14:49:00.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.324+0000 7f3ec1c00640 1 -- 172.21.15.90:0/259390116 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ecc01ba30 con 0x563630a0dc60 2024-04-02T14:49:00.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.325+0000 7f3edea82740 1 -- 172.21.15.90:0/259390116 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563630a0dc60 msgr2=0x563630a0e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:49:00.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.325+0000 7f3edea82740 1 --2- 172.21.15.90:0/259390116 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563630a0dc60 0x563630a0e0e0 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7f3ecc003aa0 tx=0x7f3ecc003f60 comp rx=0 tx=0).stop 2024-04-02T14:49:00.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.326+0000 7f3edea82740 1 -- 172.21.15.90:0/259390116 shutdown_connections 2024-04-02T14:49:00.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.326+0000 7f3edea82740 1 --2- 172.21.15.90:0/259390116 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x563630a0dc60 0x563630a0e0e0 unknown :-1 s=CLOSED pgs=969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:00.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.326+0000 7f3edea82740 1 --2- 172.21.15.90:0/259390116 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5636309f6080 0x5636309f3a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:00.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.326+0000 7f3edea82740 1 --2- 172.21.15.90:0/259390116 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5636309ecd10 0x563630a01780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:00.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.326+0000 7f3edea82740 1 -- 172.21.15.90:0/259390116 >> 172.21.15.90:0/259390116 conn(0x5636309b3a50 msgr2=0x5636309e93d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:49:00.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.327+0000 7f3edea82740 1 -- 172.21.15.90:0/259390116 shutdown_connections 2024-04-02T14:49:00.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.327+0000 7f3edea82740 1 -- 172.21.15.90:0/259390116 wait complete. 2024-04-02T14:49:00.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.328+0000 7f3edea82740 1 -- start start 2024-04-02T14:49:00.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.329+0000 7f3edea82740 1 Processor -- start 2024-04-02T14:49:00.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.330+0000 7f3edea82740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5636309ecd10 0x5636309fdc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:00.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.330+0000 7f3edea82740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5636309f6080 0x5636309fe1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:00.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.330+0000 7f3edea82740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563630a0dc60 0x5636309ef3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:00.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.330+0000 7f3edea82740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f3ed41bdc70 con 0x5636309ecd10 2024-04-02T14:49:00.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.330+0000 7f3edea82740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f3ed41bd5a0 con 0x5636309f6080 2024-04-02T14:49:00.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.330+0000 7f3edea82740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f3ed41ba800 con 0x563630a0dc60 2024-04-02T14:49:00.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.330+0000 7f3edea82740 10 client.0 _ll_register_callbacks cb 0x7f3ec802ca90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:49:00.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.331+0000 7f3ed8a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5636309f6080 0x5636309fe1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:00.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.331+0000 7f3ed8a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5636309f6080 0x5636309fe1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:42302/0 (socket says 172.21.15.90:42302) 2024-04-02T14:49:00.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.332+0000 7f3ed9e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563630a0dc60 0x5636309ef3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:00.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.332+0000 7f3ed9e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563630a0dc60 0x5636309ef3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:59310/0 (socket says 172.21.15.90:59310) 2024-04-02T14:49:00.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.332+0000 7f3ed9e00640 1 -- 172.21.15.90:0/117254309 learned_addr learned my addr 172.21.15.90:0/117254309 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:49:00.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.332+0000 7f3ed9400640 1 --2- 172.21.15.90:0/117254309 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5636309ecd10 0x5636309fdc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:00.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.333+0000 7f3ed8a00640 1 -- 172.21.15.90:0/117254309 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563630a0dc60 msgr2=0x5636309ef3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:49:00.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.333+0000 7f3ed8a00640 1 --2- 172.21.15.90:0/117254309 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563630a0dc60 0x5636309ef3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:00.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.333+0000 7f3ed8a00640 1 -- 172.21.15.90:0/117254309 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5636309ecd10 msgr2=0x5636309fdc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:49:00.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.333+0000 7f3ed8a00640 1 --2- 172.21.15.90:0/117254309 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5636309ecd10 0x5636309fdc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:00.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.333+0000 7f3ed8a00640 1 -- 172.21.15.90:0/117254309 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5636309ef9c0 con 0x5636309f6080 2024-04-02T14:49:00.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.333+0000 7f3ed9e00640 1 --2- 172.21.15.90:0/117254309 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563630a0dc60 0x5636309ef3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:49:00.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.334+0000 7f3ed8a00640 1 --2- 172.21.15.90:0/117254309 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5636309f6080 0x5636309fe1d0 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x5636309f4730 tx=0x7f3ec80035f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=11c1043866f475b2 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:49:00.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.334+0000 7f3eb6a00640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:49:00.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.336+0000 7f3eb6a00640 1 -- 172.21.15.90:0/117254309 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ec80bf150 con 0x5636309f6080 2024-04-02T14:49:00.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.336+0000 7f3eb6a00640 1 -- 172.21.15.90:0/117254309 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 8550+0+0 (secure 0 0 0) 0x7f3ec80c7fd0 con 0x5636309f6080 2024-04-02T14:49:00.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.336+0000 7f3edea82740 1 -- 172.21.15.90:0/117254309 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3ecc010910 con 0x5636309f6080 2024-04-02T14:49:00.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.336+0000 7f3eb6a00640 1 -- 172.21.15.90:0/117254309 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f3ec80bfef0 con 0x5636309f6080 2024-04-02T14:49:00.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.336+0000 7f3edea82740 10 client.29882 fetch_fsmap learned FSMap version 27 2024-04-02T14:49:00.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.336+0000 7f3edea82740 1 -- 172.21.15.90:0/117254309 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5636309ef9c0 con 0x5636309f6080 2024-04-02T14:49:00.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.337+0000 7f3eb6a00640 1 -- 172.21.15.90:0/117254309 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3ecc010910 con 0x5636309f6080 2024-04-02T14:49:00.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.337+0000 7f3eb6a00640 1 -- 172.21.15.90:0/117254309 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f3ec80be6b0 con 0x5636309f6080 2024-04-02T14:49:00.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.338+0000 7f3edea82740 10 client.29882 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:49:00.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.338+0000 7f3edea82740 10 client.29882 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2024-04-02T14:49:00.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.338+0000 7f3edea82740 20 client.29882 populate_metadata read hostname 'smithi090' 2024-04-02T14:49:00.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.338+0000 7f3edea82740 1 --2- 172.21.15.90:0/117254309 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x5636309f0fb0 0x563630a6f4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:00.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.338+0000 7f3edea82740 4 client.29882 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-02T14:49:00.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.338+0000 7f3edea82740 1 -- 172.21.15.90:0/117254309 --> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56363095ec30 con 0x5636309f0fb0 2024-04-02T14:49:00.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.338+0000 7f3ed9400640 1 --2- 172.21.15.90:0/117254309 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x5636309f0fb0 0x563630a6f4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:00.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.340+0000 7f3ed9400640 1 --2- 172.21.15.90:0/117254309 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x5636309f0fb0 0x563630a6f4f0 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=be26cd90526b2d82 server_cookie=2f12e7f6c82b25fc in_seq=0 out_seq=0 2024-04-02T14:49:00.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.340+0000 7f3eb6a00640 10 client.29882 ms_handle_connect on v2:172.21.15.90:6838/1498915870 2024-04-02T14:49:00.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.341+0000 7f3eb6a00640 1 -- 172.21.15.90:0/117254309 <== mds.1 v2:172.21.15.90:6838/1498915870 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56363095ec30 con 0x5636309f0fb0 2024-04-02T14:49:00.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.341+0000 7f3eb6a00640 10 client.29882 handle_command_reply: tid=0 2024-04-02T14:49:00.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.341+0000 7f3edea82740 1 client.29882 shutdown 2024-04-02T14:49:00.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.342+0000 7f3edea82740 1 -- 172.21.15.90:0/117254309 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5636309f6080 msgr2=0x5636309fe1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:49:00.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.342+0000 7f3edea82740 1 --2- 172.21.15.90:0/117254309 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5636309f6080 0x5636309fe1d0 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x5636309f4730 tx=0x7f3ec80035f0 comp rx=0 tx=0).stop 2024-04-02T14:49:00.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.343+0000 7f3edea82740 1 -- 172.21.15.90:0/117254309 shutdown_connections 2024-04-02T14:49:00.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.343+0000 7f3edea82740 1 --2- 172.21.15.90:0/117254309 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x5636309f0fb0 0x563630a6f4f0 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:00.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.343+0000 7f3edea82740 1 --2- 172.21.15.90:0/117254309 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563630a0dc60 0x5636309ef3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:00.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.343+0000 7f3edea82740 1 --2- 172.21.15.90:0/117254309 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5636309f6080 0x5636309fe1d0 unknown :-1 s=CLOSED pgs=970 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:00.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.343+0000 7f3edea82740 1 --2- 172.21.15.90:0/117254309 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5636309ecd10 0x5636309fdc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:00.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.343+0000 7f3eb6a00640 0 client.29882 ms_handle_reset on v2:172.21.15.90:6838/1498915870 2024-04-02T14:49:00.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.343+0000 7f3edea82740 1 -- 172.21.15.90:0/117254309 >> 172.21.15.90:0/117254309 conn(0x5636309b3a50 msgr2=0x5636309e6010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:49:00.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.343+0000 7f3edea82740 1 -- 172.21.15.90:0/117254309 shutdown_connections 2024-04-02T14:49:00.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.343+0000 7f3edea82740 1 -- 172.21.15.90:0/117254309 wait complete. 2024-04-02T14:49:00.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.344+0000 7f3edea82740 20 client.29882 trim_cache size 0 max 16384 2024-04-02T14:49:00.339 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-02T14:49:00.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.347+0000 7f3edbe00640 1 -- 172.21.15.90:0/3382440547 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f3ebc0a3e20 msgr2=0x7f3ebc0a62e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:49:00.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.347+0000 7f3edbe00640 1 --2- 172.21.15.90:0/3382440547 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f3ebc0a3e20 0x7f3ebc0a62e0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f3ec4005d00 tx=0x7f3ec400b410 comp rx=0 tx=0).stop 2024-04-02T14:49:00.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.347+0000 7f3edbe00640 1 -- 172.21.15.90:0/3382440547 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3ed4076950 msgr2=0x7f3ed410afe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:49:00.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.347+0000 7f3edbe00640 1 --2- 172.21.15.90:0/3382440547 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3ed4076950 0x7f3ed410afe0 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7f3ec8009900 tx=0x7f3ec8004360 comp rx=0 tx=0).stop 2024-04-02T14:49:00.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.347+0000 7f3edbe00640 1 -- 172.21.15.90:0/3382440547 shutdown_connections 2024-04-02T14:49:00.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.347+0000 7f3edbe00640 1 --2- 172.21.15.90:0/3382440547 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f3ebc0a3e20 0x7f3ebc0a62e0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:00.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.347+0000 7f3edbe00640 1 --2- 172.21.15.90:0/3382440547 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3ed4112940 0x7f3ed410b520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:00.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.347+0000 7f3edbe00640 1 --2- 172.21.15.90:0/3382440547 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3ed4076950 0x7f3ed410afe0 unknown :-1 s=CLOSED pgs=966 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:00.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.347+0000 7f3edbe00640 1 --2- 172.21.15.90:0/3382440547 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3ed4076030 0x7f3ed4111f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:00.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.347+0000 7f3edbe00640 1 -- 172.21.15.90:0/3382440547 >> 172.21.15.90:0/3382440547 conn(0x7f3ed40ff7b0 msgr2=0x7f3ed4101ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:49:00.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.347+0000 7f3edbe00640 1 -- 172.21.15.90:0/3382440547 shutdown_connections 2024-04-02T14:49:00.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.348+0000 7f3edbe00640 1 -- 172.21.15.90:0/3382440547 wait complete. 2024-04-02T14:49:00.366 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-02T14:49:00.603 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:49:00 smithi092 ceph-mon[36497]: pgmap v440: 97 pgs: 97 active+clean; 4.6 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 6.3 MiB/s wr, 523 op/s 2024-04-02T14:49:00.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.649+0000 7f0a63200640 1 -- 172.21.15.90:0/3156345144 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0a6410e210 msgr2=0x7f0a64110620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:49:00.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.649+0000 7f0a63200640 1 --2- 172.21.15.90:0/3156345144 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0a6410e210 0x7f0a64110620 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7f0a5c009ca0 tx=0x7f0a5c02c190 comp rx=0 tx=0).stop 2024-04-02T14:49:00.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.649+0000 7f0a63200640 1 -- 172.21.15.90:0/3156345144 shutdown_connections 2024-04-02T14:49:00.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.649+0000 7f0a63200640 1 --2- 172.21.15.90:0/3156345144 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0a6410e210 0x7f0a64110620 unknown :-1 s=CLOSED pgs=971 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:00.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.649+0000 7f0a63200640 1 --2- 172.21.15.90:0/3156345144 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0a640747f0 0x7f0a6410dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:00.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.649+0000 7f0a63200640 1 --2- 172.21.15.90:0/3156345144 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0a64073ed0 0x7f0a640742b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:00.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.649+0000 7f0a63200640 1 -- 172.21.15.90:0/3156345144 >> 172.21.15.90:0/3156345144 conn(0x7f0a640fd460 msgr2=0x7f0a640ff880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:49:00.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.649+0000 7f0a63200640 1 -- 172.21.15.90:0/3156345144 shutdown_connections 2024-04-02T14:49:00.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.650+0000 7f0a63200640 1 -- 172.21.15.90:0/3156345144 wait complete. 2024-04-02T14:49:00.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.650+0000 7f0a63200640 1 Processor -- start 2024-04-02T14:49:00.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.651+0000 7f0a63200640 1 -- start start 2024-04-02T14:49:00.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.651+0000 7f0a63200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0a64073ed0 0x7f0a6419a490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:00.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.651+0000 7f0a63200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0a640747f0 0x7f0a6419a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:00.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.651+0000 7f0a63200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0a6410e210 0x7f0a6419ed00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:00.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.651+0000 7f0a63200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f0a64113f70 con 0x7f0a640747f0 2024-04-02T14:49:00.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.651+0000 7f0a63200640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f0a64113df0 con 0x7f0a64073ed0 2024-04-02T14:49:00.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.651+0000 7f0a63200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f0a641140f0 con 0x7f0a6410e210 2024-04-02T14:49:00.646 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.652+0000 7f0a62800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0a6410e210 0x7f0a6419ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:00.646 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.652+0000 7f0a62800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0a6410e210 0x7f0a6419ed00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:59340/0 (socket says 172.21.15.90:59340) 2024-04-02T14:49:00.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.652+0000 7f0a62800640 1 -- 172.21.15.90:0/3057124826 learned_addr learned my addr 172.21.15.90:0/3057124826 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:49:00.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.652+0000 7f0a61e00640 1 --2- 172.21.15.90:0/3057124826 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0a64073ed0 0x7f0a6419a490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:00.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.652+0000 7f0a5be00640 1 --2- 172.21.15.90:0/3057124826 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0a640747f0 0x7f0a6419a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:00.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.652+0000 7f0a62800640 1 -- 172.21.15.90:0/3057124826 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0a64073ed0 msgr2=0x7f0a6419a490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:49:00.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.652+0000 7f0a62800640 1 --2- 172.21.15.90:0/3057124826 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0a64073ed0 0x7f0a6419a490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:00.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.652+0000 7f0a62800640 1 -- 172.21.15.90:0/3057124826 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0a640747f0 msgr2=0x7f0a6419a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:49:00.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.652+0000 7f0a62800640 1 --2- 172.21.15.90:0/3057124826 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0a640747f0 0x7f0a6419a9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:00.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.652+0000 7f0a62800640 1 -- 172.21.15.90:0/3057124826 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a6419f430 con 0x7f0a6410e210 2024-04-02T14:49:00.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.653+0000 7f0a62800640 1 --2- 172.21.15.90:0/3057124826 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0a6410e210 0x7f0a6419ed00 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f0a5c005e30 tx=0x7f0a5c035e50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:49:00.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.653+0000 7f0a5aa00640 1 -- 172.21.15.90:0/3057124826 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a5c039040 con 0x7f0a6410e210 2024-04-02T14:49:00.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.653+0000 7f0a63200640 1 -- 172.21.15.90:0/3057124826 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a6419f6c0 con 0x7f0a6410e210 2024-04-02T14:49:00.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.653+0000 7f0a5aa00640 1 -- 172.21.15.90:0/3057124826 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0a5c044070 con 0x7f0a6410e210 2024-04-02T14:49:00.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.653+0000 7f0a5aa00640 1 -- 172.21.15.90:0/3057124826 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a5c032940 con 0x7f0a6410e210 2024-04-02T14:49:00.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.653+0000 7f0a63200640 1 -- 172.21.15.90:0/3057124826 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a641a6fb0 con 0x7f0a6410e210 2024-04-02T14:49:00.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.658+0000 7f0a6be0c740 1 Processor -- start 2024-04-02T14:49:00.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.658+0000 7f0a5aa00640 1 -- 172.21.15.90:0/3057124826 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f0a5c032ae0 con 0x7f0a6410e210 2024-04-02T14:49:00.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.658+0000 7f0a6be0c740 1 -- start start 2024-04-02T14:49:00.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.658+0000 7f0a6be0c740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f07df60ba0 0x55f07df5f8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:00.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.658+0000 7f0a6be0c740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f07df5fec0 0x55f07df60340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:00.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.658+0000 7f0a6be0c740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f07df7a240 0x55f07df7c630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:00.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.658+0000 7f0a6be0c740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55f07dea0400 con 0x55f07df60ba0 2024-04-02T14:49:00.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.658+0000 7f0a6be0c740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55f07de84f10 con 0x55f07df7a240 2024-04-02T14:49:00.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.658+0000 7f0a6be0c740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55f07dec46f0 con 0x55f07df5fec0 2024-04-02T14:49:00.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.659+0000 7f0a61e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f07df5fec0 0x55f07df60340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:00.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.659+0000 7f0a61e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f07df5fec0 0x55f07df60340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:59344/0 (socket says 172.21.15.90:59344) 2024-04-02T14:49:00.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.659+0000 7f0a61e00640 1 -- 172.21.15.90:0/238850579 learned_addr learned my addr 172.21.15.90:0/238850579 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:49:00.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.659+0000 7f0a5be00640 1 --2- 172.21.15.90:0/238850579 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f07df60ba0 0x55f07df5f8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:00.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.660+0000 7f0a61e00640 1 -- 172.21.15.90:0/238850579 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f07df7a240 msgr2=0x55f07df7c630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:49:00.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.660+0000 7f0a61e00640 1 --2- 172.21.15.90:0/238850579 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f07df7a240 0x55f07df7c630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:00.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.660+0000 7f0a61e00640 1 -- 172.21.15.90:0/238850579 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f07df60ba0 msgr2=0x55f07df5f8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:49:00.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.660+0000 7f0a61e00640 1 --2- 172.21.15.90:0/238850579 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f07df60ba0 0x55f07df5f8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:00.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.660+0000 7f0a61e00640 1 -- 172.21.15.90:0/238850579 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f07dbd83f0 con 0x55f07df5fec0 2024-04-02T14:49:00.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.660+0000 7f0a5aa00640 1 --2- 172.21.15.90:0/3057124826 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f0a480a3d20 0x7f0a480a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:00.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.661+0000 7f0a5aa00640 1 -- 172.21.15.90:0/3057124826 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 8550+0+0 (secure 0 0 0) 0x7f0a5c0f2ba0 con 0x7f0a6410e210 2024-04-02T14:49:00.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.663+0000 7f0a5be00640 1 --2- 172.21.15.90:0/3057124826 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f0a480a3d20 0x7f0a480a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:00.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.664+0000 7f0a61e00640 1 --2- 172.21.15.90:0/238850579 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f07df5fec0 0x55f07df60340 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f0a54009cf0 tx=0x7f0a54011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=45d4f841aa9a95a9 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:49:00.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.665+0000 7f0a53400640 1 -- 172.21.15.90:0/238850579 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a54004230 con 0x55f07df5fec0 2024-04-02T14:49:00.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.665+0000 7f0a53400640 1 -- 172.21.15.90:0/238850579 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0a540043d0 con 0x55f07df5fec0 2024-04-02T14:49:00.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.665+0000 7f0a53400640 1 -- 172.21.15.90:0/238850579 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a54019960 con 0x55f07df5fec0 2024-04-02T14:49:00.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.666+0000 7f0a6be0c740 1 -- 172.21.15.90:0/238850579 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f07df5fec0 msgr2=0x55f07df60340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:49:00.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.666+0000 7f0a6be0c740 1 --2- 172.21.15.90:0/238850579 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f07df5fec0 0x55f07df60340 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f0a54009cf0 tx=0x7f0a54011810 comp rx=0 tx=0).stop 2024-04-02T14:49:00.660 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.666+0000 7f0a5be00640 1 --2- 172.21.15.90:0/3057124826 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f0a480a3d20 0x7f0a480a61e0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f0a640ffaf0 tx=0x7f0a4c00ba00 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:49:00.660 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.667+0000 7f0a6be0c740 1 -- 172.21.15.90:0/238850579 shutdown_connections 2024-04-02T14:49:00.660 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.667+0000 7f0a6be0c740 1 --2- 172.21.15.90:0/238850579 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f07df7a240 0x55f07df7c630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:00.660 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.667+0000 7f0a6be0c740 1 --2- 172.21.15.90:0/238850579 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f07df5fec0 0x55f07df60340 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:00.660 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.667+0000 7f0a6be0c740 1 --2- 172.21.15.90:0/238850579 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f07df60ba0 0x55f07df5f8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:00.660 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.667+0000 7f0a6be0c740 1 -- 172.21.15.90:0/238850579 >> 172.21.15.90:0/238850579 conn(0x55f07df26a50 msgr2=0x55f07df5a910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:49:00.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.667+0000 7f0a6be0c740 1 -- 172.21.15.90:0/238850579 shutdown_connections 2024-04-02T14:49:00.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.667+0000 7f0a6be0c740 1 -- 172.21.15.90:0/238850579 wait complete. 2024-04-02T14:49:00.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.669+0000 7f0a6be0c740 1 -- start start 2024-04-02T14:49:00.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.670+0000 7f0a6be0c740 1 Processor -- start 2024-04-02T14:49:00.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.671+0000 7f0a6be0c740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f07df5fec0 0x55f07df772b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:00.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.671+0000 7f0a6be0c740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f07df60ba0 0x55f07df777f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:00.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.671+0000 7f0a6be0c740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f07df7a240 0x55f07df77d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:00.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.671+0000 7f0a6be0c740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55f07dec06a0 con 0x55f07df7a240 2024-04-02T14:49:00.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.671+0000 7f0a6be0c740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55f07de6c980 con 0x55f07df5fec0 2024-04-02T14:49:00.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.671+0000 7f0a6be0c740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55f07de7fcb0 con 0x55f07df60ba0 2024-04-02T14:49:00.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.671+0000 7f0a6be0c740 10 client.0 _ll_register_callbacks cb 0x55f07de7f6a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:49:00.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.671+0000 7f0a5be00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f07df60ba0 0x55f07df777f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:00.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.671+0000 7f0a5be00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f07df60ba0 0x55f07df777f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:59360/0 (socket says 172.21.15.90:59360) 2024-04-02T14:49:00.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.671+0000 7f0a5be00640 1 -- 172.21.15.90:0/3426009180 learned_addr learned my addr 172.21.15.90:0/3426009180 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:49:00.666 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.672+0000 7f0a62800640 1 --2- 172.21.15.90:0/3426009180 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f07df7a240 0x55f07df77d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:00.666 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.672+0000 7f0a5be00640 1 -- 172.21.15.90:0/3426009180 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f07df5fec0 msgr2=0x55f07df772b0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-02T14:49:00.666 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.672+0000 7f0a5be00640 1 --2- 172.21.15.90:0/3426009180 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f07df5fec0 0x55f07df772b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:00.666 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.672+0000 7f0a5be00640 1 -- 172.21.15.90:0/3426009180 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f07df7a240 msgr2=0x55f07df77d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:49:00.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.672+0000 7f0a5be00640 1 --2- 172.21.15.90:0/3426009180 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f07df7a240 0x55f07df77d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:00.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.672+0000 7f0a5be00640 1 -- 172.21.15.90:0/3426009180 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f07dca27d0 con 0x55f07df60ba0 2024-04-02T14:49:00.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.673+0000 7f0a5be00640 1 --2- 172.21.15.90:0/3426009180 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f07df60ba0 0x55f07df777f0 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f0a4c01d8e0 tx=0x7f0a4c0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=abc0fcb2671b2077 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:49:00.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.673+0000 7f0a47e00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:49:00.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.674+0000 7f0a47e00640 1 -- 172.21.15.90:0/3426009180 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a4c01de60 con 0x55f07df60ba0 2024-04-02T14:49:00.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.674+0000 7f0a47e00640 1 -- 172.21.15.90:0/3426009180 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 8550+0+0 (secure 0 0 0) 0x7f0a4c010b60 con 0x55f07df60ba0 2024-04-02T14:49:00.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.674+0000 7f0a6be0c740 1 -- 172.21.15.90:0/3426009180 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0a540043d0 con 0x55f07df60ba0 2024-04-02T14:49:00.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.675+0000 7f0a47e00640 1 -- 172.21.15.90:0/3426009180 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f0a4c010d60 con 0x55f07df60ba0 2024-04-02T14:49:00.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.675+0000 7f0a6be0c740 10 client.29683 fetch_fsmap learned FSMap version 27 2024-04-02T14:49:00.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.675+0000 7f0a6be0c740 1 -- 172.21.15.90:0/3426009180 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55f07df891d0 con 0x55f07df60ba0 2024-04-02T14:49:00.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.675+0000 7f0a47e00640 1 -- 172.21.15.90:0/3426009180 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0a540043d0 con 0x55f07df60ba0 2024-04-02T14:49:00.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.676+0000 7f0a47e00640 1 -- 172.21.15.90:0/3426009180 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f0a4c019040 con 0x55f07df60ba0 2024-04-02T14:49:00.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.677+0000 7f0a6be0c740 10 client.29683 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:49:00.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.677+0000 7f0a6be0c740 10 client.29683 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:49:00.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.677+0000 7f0a6be0c740 20 client.29683 populate_metadata read hostname 'smithi090' 2024-04-02T14:49:00.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.677+0000 7f0a6be0c740 1 --2- 172.21.15.90:0/3426009180 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55f07df8a730 0x55f07df8cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:00.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.677+0000 7f0a6be0c740 4 client.29683 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:49:00.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.677+0000 7f0a6be0c740 1 -- 172.21.15.90:0/3426009180 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f07de84040 con 0x55f07df8a730 2024-04-02T14:49:00.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.680+0000 7f0a61e00640 1 --2- 172.21.15.90:0/3426009180 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55f07df8a730 0x55f07df8cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:00.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.681+0000 7f0a61e00640 1 --2- 172.21.15.90:0/3426009180 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55f07df8a730 0x55f07df8cbd0 crc :-1 s=READY pgs=360 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5583a31b1486df1 server_cookie=40143530dd1079d8 in_seq=0 out_seq=0 2024-04-02T14:49:00.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.681+0000 7f0a47e00640 10 client.29683 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:49:00.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.682+0000 7f0a47e00640 1 -- 172.21.15.90:0/3426009180 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f07de84040 con 0x55f07df8a730 2024-04-02T14:49:00.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.683+0000 7f0a47e00640 10 client.29683 handle_command_reply: tid=0 2024-04-02T14:49:00.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.683+0000 7f0a6be0c740 1 client.29683 shutdown 2024-04-02T14:49:00.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.684+0000 7f0a6be0c740 1 -- 172.21.15.90:0/3426009180 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f07df60ba0 msgr2=0x55f07df777f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:49:00.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.684+0000 7f0a6be0c740 1 --2- 172.21.15.90:0/3426009180 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f07df60ba0 0x55f07df777f0 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f0a4c01d8e0 tx=0x7f0a4c0046c0 comp rx=0 tx=0).stop 2024-04-02T14:49:00.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.684+0000 7f0a6be0c740 1 -- 172.21.15.90:0/3426009180 shutdown_connections 2024-04-02T14:49:00.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.684+0000 7f0a6be0c740 1 --2- 172.21.15.90:0/3426009180 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55f07df8a730 0x55f07df8cbd0 crc :-1 s=READY pgs=360 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:00.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.684+0000 7f0a6be0c740 1 --2- 172.21.15.90:0/3426009180 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f07df7a240 0x55f07df77d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:00.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.684+0000 7f0a6be0c740 1 --2- 172.21.15.90:0/3426009180 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f07df60ba0 0x55f07df777f0 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:00.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.684+0000 7f0a6be0c740 1 --2- 172.21.15.90:0/3426009180 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f07df5fec0 0x55f07df772b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:00.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.684+0000 7f0a6be0c740 1 -- 172.21.15.90:0/3426009180 >> 172.21.15.90:0/3426009180 conn(0x55f07df26a50 msgr2=0x55f07df5c660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:49:00.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.684+0000 7f0a47e00640 0 client.29683 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:49:00.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.684+0000 7f0a6be0c740 1 -- 172.21.15.90:0/3426009180 shutdown_connections 2024-04-02T14:49:00.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.684+0000 7f0a6be0c740 1 -- 172.21.15.90:0/3426009180 wait complete. 2024-04-02T14:49:00.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.684+0000 7f0a6be0c740 20 client.29683 trim_cache size 0 max 16384 2024-04-02T14:49:00.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:49:00 smithi177 ceph-mon[36431]: pgmap v440: 97 pgs: 97 active+clean; 4.6 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 6.3 MiB/s wr, 523 op/s 2024-04-02T14:49:00.680 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:49:00 smithi090 ceph-mon[31674]: pgmap v440: 97 pgs: 97 active+clean; 4.6 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 6.3 MiB/s wr, 523 op/s 2024-04-02T14:49:00.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.713+0000 7f0a6be0c740 1 Processor -- start 2024-04-02T14:49:00.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.713+0000 7f0a6be0c740 1 -- start start 2024-04-02T14:49:00.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.713+0000 7f0a6be0c740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f07df5fec0 0x55f07df799e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:00.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.713+0000 7f0a6be0c740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f07df60ba0 0x55f07df85cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:00.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.713+0000 7f0a6be0c740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f07df7a240 0x55f07df5c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:00.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.713+0000 7f0a6be0c740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f0a641a7180 con 0x55f07df7a240 2024-04-02T14:49:00.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.713+0000 7f0a6be0c740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f0a641a8bf0 con 0x55f07df5fec0 2024-04-02T14:49:00.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.713+0000 7f0a6be0c740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f0a641a7780 con 0x55f07df60ba0 2024-04-02T14:49:00.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.713+0000 7f0a5be00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f07df60ba0 0x55f07df85cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:00.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.713+0000 7f0a5be00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f07df60ba0 0x55f07df85cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:59364/0 (socket says 172.21.15.90:59364) 2024-04-02T14:49:00.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.713+0000 7f0a5be00640 1 -- 172.21.15.90:0/3109471573 learned_addr learned my addr 172.21.15.90:0/3109471573 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:49:00.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.714+0000 7f0a5be00640 1 -- 172.21.15.90:0/3109471573 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f07df5fec0 msgr2=0x55f07df799e0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-02T14:49:00.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.714+0000 7f0a5be00640 1 --2- 172.21.15.90:0/3109471573 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f07df5fec0 0x55f07df799e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:00.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.714+0000 7f0a5be00640 1 -- 172.21.15.90:0/3109471573 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f07df7a240 msgr2=0x55f07df5c5f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-02T14:49:00.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.715+0000 7f0a62800640 1 --2- 172.21.15.90:0/3109471573 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f07df7a240 0x55f07df5c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:00.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.715+0000 7f0a5be00640 1 --2- 172.21.15.90:0/3109471573 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f07df7a240 0x55f07df5c5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:00.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.715+0000 7f0a5be00640 1 -- 172.21.15.90:0/3109471573 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f07df75800 con 0x55f07df60ba0 2024-04-02T14:49:00.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.715+0000 7f0a5be00640 1 --2- 172.21.15.90:0/3109471573 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f07df60ba0 0x55f07df85cb0 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f0a4c0256f0 tx=0x7f0a4c010f80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8ddb4174b3e1e63a server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:49:00.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.716+0000 7f0a52a00640 1 -- 172.21.15.90:0/3109471573 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a4c01de60 con 0x55f07df60ba0 2024-04-02T14:49:00.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.716+0000 7f0a52a00640 1 -- 172.21.15.90:0/3109471573 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0a4c010b60 con 0x55f07df60ba0 2024-04-02T14:49:00.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.716+0000 7f0a52a00640 1 -- 172.21.15.90:0/3109471573 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a4c01bdc0 con 0x55f07df60ba0 2024-04-02T14:49:00.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.717+0000 7f0a6be0c740 1 -- 172.21.15.90:0/3109471573 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f07df60ba0 msgr2=0x55f07df85cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:49:00.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.717+0000 7f0a6be0c740 1 --2- 172.21.15.90:0/3109471573 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f07df60ba0 0x55f07df85cb0 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f0a4c0256f0 tx=0x7f0a4c010f80 comp rx=0 tx=0).stop 2024-04-02T14:49:00.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.717+0000 7f0a6be0c740 1 -- 172.21.15.90:0/3109471573 shutdown_connections 2024-04-02T14:49:00.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.717+0000 7f0a6be0c740 1 --2- 172.21.15.90:0/3109471573 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f07df7a240 0x55f07df5c5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:00.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.717+0000 7f0a6be0c740 1 --2- 172.21.15.90:0/3109471573 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f07df60ba0 0x55f07df85cb0 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:00.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.717+0000 7f0a6be0c740 1 --2- 172.21.15.90:0/3109471573 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f07df5fec0 0x55f07df799e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:00.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.717+0000 7f0a6be0c740 1 -- 172.21.15.90:0/3109471573 >> 172.21.15.90:0/3109471573 conn(0x55f07df26a50 msgr2=0x55f07df75a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:49:00.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.718+0000 7f0a6be0c740 1 -- 172.21.15.90:0/3109471573 shutdown_connections 2024-04-02T14:49:00.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.718+0000 7f0a6be0c740 1 -- 172.21.15.90:0/3109471573 wait complete. 2024-04-02T14:49:00.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.718+0000 7f0a6be0c740 1 -- start start 2024-04-02T14:49:00.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.719+0000 7f0a6be0c740 1 Processor -- start 2024-04-02T14:49:00.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.720+0000 7f0a6be0c740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f07df5fec0 0x55f07df82690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:00.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.720+0000 7f0a6be0c740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f07df7a240 0x55f07df748f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:00.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.720+0000 7f0a6be0c740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f07df74ed0 0x55f07df7b080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:00.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.720+0000 7f0a6be0c740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f0a641a8520 con 0x55f07df5fec0 2024-04-02T14:49:00.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.720+0000 7f0a6be0c740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f0a641a7e50 con 0x55f07df7a240 2024-04-02T14:49:00.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.720+0000 7f0a6be0c740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f0a64071c50 con 0x55f07df74ed0 2024-04-02T14:49:00.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.720+0000 7f0a6be0c740 10 client.0 _ll_register_callbacks cb 0x7f0a4c01d560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:49:00.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.721+0000 7f0a5be00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f07df7a240 0x55f07df748f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:00.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.721+0000 7f0a5be00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f07df7a240 0x55f07df748f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:42362/0 (socket says 172.21.15.90:42362) 2024-04-02T14:49:00.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.721+0000 7f0a5be00640 1 -- 172.21.15.90:0/2613291243 learned_addr learned my addr 172.21.15.90:0/2613291243 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:49:00.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.721+0000 7f0a62800640 1 --2- 172.21.15.90:0/2613291243 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f07df74ed0 0x55f07df7b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:00.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.721+0000 7f0a61e00640 1 --2- 172.21.15.90:0/2613291243 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f07df5fec0 0x55f07df82690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:00.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.721+0000 7f0a5be00640 1 -- 172.21.15.90:0/2613291243 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f07df74ed0 msgr2=0x55f07df7b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:49:00.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.721+0000 7f0a5be00640 1 --2- 172.21.15.90:0/2613291243 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f07df74ed0 0x55f07df7b080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:00.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.721+0000 7f0a5be00640 1 -- 172.21.15.90:0/2613291243 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f07df5fec0 msgr2=0x55f07df82690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:49:00.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.721+0000 7f0a5be00640 1 --2- 172.21.15.90:0/2613291243 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f07df5fec0 0x55f07df82690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:00.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.721+0000 7f0a5be00640 1 -- 172.21.15.90:0/2613291243 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f07df7b6c0 con 0x55f07df7a240 2024-04-02T14:49:00.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.722+0000 7f0a5be00640 1 --2- 172.21.15.90:0/2613291243 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f07df7a240 0x55f07df748f0 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7f0a4c005d00 tx=0x7f0a4c00b540 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f57d525c1762b08a server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:49:00.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.722+0000 7f0a47e00640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:49:00.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.723+0000 7f0a47e00640 1 -- 172.21.15.90:0/2613291243 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a4c03e070 con 0x55f07df7a240 2024-04-02T14:49:00.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.723+0000 7f0a47e00640 1 -- 172.21.15.90:0/2613291243 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 8550+0+0 (secure 0 0 0) 0x7f0a4c01b6f0 con 0x55f07df7a240 2024-04-02T14:49:00.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.723+0000 7f0a6be0c740 1 -- 172.21.15.90:0/2613291243 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0a4c010b60 con 0x55f07df7a240 2024-04-02T14:49:00.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.723+0000 7f0a47e00640 1 -- 172.21.15.90:0/2613291243 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f0a4c01de60 con 0x55f07df7a240 2024-04-02T14:49:00.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.723+0000 7f0a6be0c740 10 client.29891 fetch_fsmap learned FSMap version 27 2024-04-02T14:49:00.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.723+0000 7f0a6be0c740 1 -- 172.21.15.90:0/2613291243 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55f07dff9e60 con 0x55f07df7a240 2024-04-02T14:49:00.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.724+0000 7f0a47e00640 1 -- 172.21.15.90:0/2613291243 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0a4c010b60 con 0x55f07df7a240 2024-04-02T14:49:00.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.724+0000 7f0a47e00640 1 -- 172.21.15.90:0/2613291243 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f0a4c019040 con 0x55f07df7a240 2024-04-02T14:49:00.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.724+0000 7f0a6be0c740 10 client.29891 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:49:00.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.724+0000 7f0a6be0c740 10 client.29891 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:49:00.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.724+0000 7f0a6be0c740 20 client.29891 populate_metadata read hostname 'smithi090' 2024-04-02T14:49:00.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.724+0000 7f0a6be0c740 1 --2- 172.21.15.90:0/2613291243 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55f07dffa4d0 0x55f07dffc8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:00.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.724+0000 7f0a6be0c740 4 client.29891 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-02T14:49:00.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.724+0000 7f0a6be0c740 1 -- 172.21.15.90:0/2613291243 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55f07ded1c30 con 0x55f07dffa4d0 2024-04-02T14:49:00.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.724+0000 7f0a61e00640 1 --2- 172.21.15.90:0/2613291243 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55f07dffa4d0 0x55f07dffc8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:00.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.725+0000 7f0a61e00640 1 --2- 172.21.15.90:0/2613291243 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55f07dffa4d0 0x55f07dffc8f0 crc :-1 s=READY pgs=361 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5f056f833fa536e server_cookie=4c63a11e104041a0 in_seq=0 out_seq=0 2024-04-02T14:49:00.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.725+0000 7f0a47e00640 10 client.29891 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:49:00.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.725+0000 7f0a47e00640 1 -- 172.21.15.90:0/2613291243 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55f07ded1c30 con 0x55f07dffa4d0 2024-04-02T14:49:00.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.725+0000 7f0a47e00640 10 client.29891 handle_command_reply: tid=0 2024-04-02T14:49:00.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.725+0000 7f0a6be0c740 1 client.29891 shutdown 2024-04-02T14:49:00.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.725+0000 7f0a6be0c740 1 -- 172.21.15.90:0/2613291243 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f07df7a240 msgr2=0x55f07df748f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:49:00.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.725+0000 7f0a6be0c740 1 --2- 172.21.15.90:0/2613291243 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f07df7a240 0x55f07df748f0 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7f0a4c005d00 tx=0x7f0a4c00b540 comp rx=0 tx=0).stop 2024-04-02T14:49:00.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.726+0000 7f0a6be0c740 1 -- 172.21.15.90:0/2613291243 shutdown_connections 2024-04-02T14:49:00.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.726+0000 7f0a6be0c740 1 --2- 172.21.15.90:0/2613291243 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55f07dffa4d0 0x55f07dffc8f0 crc :-1 s=READY pgs=361 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:00.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.726+0000 7f0a6be0c740 1 --2- 172.21.15.90:0/2613291243 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f07df74ed0 0x55f07df7b080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:00.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.726+0000 7f0a6be0c740 1 --2- 172.21.15.90:0/2613291243 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f07df7a240 0x55f07df748f0 unknown :-1 s=CLOSED pgs=972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:00.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.726+0000 7f0a6be0c740 1 --2- 172.21.15.90:0/2613291243 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f07df5fec0 0x55f07df82690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:00.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.726+0000 7f0a47e00640 0 client.29891 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:49:00.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.726+0000 7f0a6be0c740 1 -- 172.21.15.90:0/2613291243 >> 172.21.15.90:0/2613291243 conn(0x55f07df26a50 msgr2=0x55f07df59af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:49:00.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.726+0000 7f0a6be0c740 1 -- 172.21.15.90:0/2613291243 shutdown_connections 2024-04-02T14:49:00.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.726+0000 7f0a6be0c740 1 -- 172.21.15.90:0/2613291243 wait complete. 2024-04-02T14:49:00.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.726+0000 7f0a6be0c740 20 client.29891 trim_cache size 0 max 16384 2024-04-02T14:49:00.721 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-02T14:49:00.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.727+0000 7f0a63200640 1 -- 172.21.15.90:0/3057124826 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f0a480a3d20 msgr2=0x7f0a480a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:49:00.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.727+0000 7f0a63200640 1 --2- 172.21.15.90:0/3057124826 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f0a480a3d20 0x7f0a480a61e0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f0a640ffaf0 tx=0x7f0a4c00ba00 comp rx=0 tx=0).stop 2024-04-02T14:49:00.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.727+0000 7f0a63200640 1 -- 172.21.15.90:0/3057124826 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0a6410e210 msgr2=0x7f0a6419ed00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:49:00.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.727+0000 7f0a63200640 1 --2- 172.21.15.90:0/3057124826 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0a6410e210 0x7f0a6419ed00 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f0a5c005e30 tx=0x7f0a5c035e50 comp rx=0 tx=0).stop 2024-04-02T14:49:00.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.727+0000 7f0a63200640 1 -- 172.21.15.90:0/3057124826 shutdown_connections 2024-04-02T14:49:00.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.727+0000 7f0a63200640 1 --2- 172.21.15.90:0/3057124826 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f0a480a3d20 0x7f0a480a61e0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:00.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.727+0000 7f0a63200640 1 --2- 172.21.15.90:0/3057124826 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0a6410e210 0x7f0a6419ed00 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:00.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.727+0000 7f0a63200640 1 --2- 172.21.15.90:0/3057124826 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0a640747f0 0x7f0a6419a9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:00.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.727+0000 7f0a63200640 1 --2- 172.21.15.90:0/3057124826 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0a64073ed0 0x7f0a6419a490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:00.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.727+0000 7f0a63200640 1 -- 172.21.15.90:0/3057124826 >> 172.21.15.90:0/3057124826 conn(0x7f0a640fd460 msgr2=0x7f0a640fec90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:49:00.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.727+0000 7f0a63200640 1 -- 172.21.15.90:0/3057124826 shutdown_connections 2024-04-02T14:49:00.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:00.728+0000 7f0a63200640 1 -- 172.21.15.90:0/3057124826 wait complete. 2024-04-02T14:49:01.743 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-02T14:49:01.743 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T14:49:01.743 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-02T14:49:01.823 INFO:teuthology.orchestra.run.smithi090.stderr:progress: 810.0 s, 1020.4 tps, lat 89.400 ms stddev 138.437 2024-04-02T14:49:02.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.063+0000 7ff7a3200640 1 -- 172.21.15.90:0/1096470578 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff79c11b2e0 msgr2=0x7ff79c11d6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:49:02.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.063+0000 7ff7a3200640 1 --2- 172.21.15.90:0/1096470578 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff79c11b2e0 0x7ff79c11d6d0 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7ff794009ca0 tx=0x7ff79402c100 comp rx=0 tx=0).stop 2024-04-02T14:49:02.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.065+0000 7ff7a3200640 1 -- 172.21.15.90:0/1096470578 shutdown_connections 2024-04-02T14:49:02.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.065+0000 7ff7a3200640 1 --2- 172.21.15.90:0/1096470578 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff79c11b2e0 0x7ff79c11d6d0 unknown :-1 s=CLOSED pgs=973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:02.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.065+0000 7ff7a3200640 1 --2- 172.21.15.90:0/1096470578 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff79c10e350 0x7ff79c10e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:02.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.065+0000 7ff7a3200640 1 --2- 172.21.15.90:0/1096470578 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff79c110320 0x7ff79c10de10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:02.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.065+0000 7ff7a3200640 1 -- 172.21.15.90:0/1096470578 >> 172.21.15.90:0/1096470578 conn(0x7ff79c06cbe0 msgr2=0x7ff79c06cff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:49:02.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.066+0000 7ff7a3200640 1 -- 172.21.15.90:0/1096470578 shutdown_connections 2024-04-02T14:49:02.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.066+0000 7ff7a3200640 1 -- 172.21.15.90:0/1096470578 wait complete. 2024-04-02T14:49:02.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.068+0000 7ff7a3200640 1 Processor -- start 2024-04-02T14:49:02.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.068+0000 7ff7a3200640 1 -- start start 2024-04-02T14:49:02.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.069+0000 7ff7a3200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff79c10e350 0x7ff79c1189c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:02.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.069+0000 7ff7a3200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff79c110320 0x7ff79c111a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:02.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.069+0000 7ff7a3200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff79c11b2e0 0x7ff79c111f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:02.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.070+0000 7ff79be00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff79c10e350 0x7ff79c1189c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:02.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.070+0000 7ff79b400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff79c110320 0x7ff79c111a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:02.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.070+0000 7ff7a1200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff79c11b2e0 0x7ff79c111f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:02.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.070+0000 7ff79b400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff79c110320 0x7ff79c111a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:42384/0 (socket says 172.21.15.90:42384) 2024-04-02T14:49:02.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.070+0000 7ff7a1200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff79c11b2e0 0x7ff79c111f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:59406/0 (socket says 172.21.15.90:59406) 2024-04-02T14:49:02.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.070+0000 7ff79b400640 1 -- 172.21.15.90:0/2266466641 learned_addr learned my addr 172.21.15.90:0/2266466641 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:49:02.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.070+0000 7ff7a3200640 1 -- 172.21.15.90:0/2266466641 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7ff79c120c40 con 0x7ff79c10e350 2024-04-02T14:49:02.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.070+0000 7ff7a3200640 1 -- 172.21.15.90:0/2266466641 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7ff79c120ac0 con 0x7ff79c110320 2024-04-02T14:49:02.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.070+0000 7ff7a3200640 1 -- 172.21.15.90:0/2266466641 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7ff79c120dc0 con 0x7ff79c11b2e0 2024-04-02T14:49:02.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.071+0000 7ff7a1200640 1 -- 172.21.15.90:0/2266466641 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff79c110320 msgr2=0x7ff79c111a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:49:02.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.071+0000 7ff7a1200640 1 --2- 172.21.15.90:0/2266466641 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff79c110320 0x7ff79c111a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:02.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.071+0000 7ff7a1200640 1 -- 172.21.15.90:0/2266466641 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff79c10e350 msgr2=0x7ff79c1189c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:49:02.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.071+0000 7ff7a1200640 1 --2- 172.21.15.90:0/2266466641 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff79c10e350 0x7ff79c1189c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:02.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.071+0000 7ff7a1200640 1 -- 172.21.15.90:0/2266466641 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff79c1126b0 con 0x7ff79c11b2e0 2024-04-02T14:49:02.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.071+0000 7ff79b400640 1 --2- 172.21.15.90:0/2266466641 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff79c110320 0x7ff79c111a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:49:02.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.072+0000 7ff7a1200640 1 --2- 172.21.15.90:0/2266466641 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff79c11b2e0 0x7ff79c111f80 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7ff794037040 tx=0x7ff794035d10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:49:02.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.072+0000 7ff798c00640 1 -- 172.21.15.90:0/2266466641 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff794039040 con 0x7ff79c11b2e0 2024-04-02T14:49:02.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.072+0000 7ff798c00640 1 -- 172.21.15.90:0/2266466641 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff794044070 con 0x7ff79c11b2e0 2024-04-02T14:49:02.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.072+0000 7ff798c00640 1 -- 172.21.15.90:0/2266466641 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff794004280 con 0x7ff79c11b2e0 2024-04-02T14:49:02.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.072+0000 7ff7a3200640 1 -- 172.21.15.90:0/2266466641 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff79c1164e0 con 0x7ff79c11b2e0 2024-04-02T14:49:02.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.073+0000 7ff7a3200640 1 -- 172.21.15.90:0/2266466641 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff79c116980 con 0x7ff79c11b2e0 2024-04-02T14:49:02.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.078+0000 7ff7a5f9b740 1 Processor -- start 2024-04-02T14:49:02.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.079+0000 7ff798c00640 1 -- 172.21.15.90:0/2266466641 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7ff794003970 con 0x7ff79c11b2e0 2024-04-02T14:49:02.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.079+0000 7ff7a5f9b740 1 -- start start 2024-04-02T14:49:02.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.080+0000 7ff7a5f9b740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56360fc55030 0x56360fc53d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:02.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.080+0000 7ff7a5f9b740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56360fc54350 0x56360fc547d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:02.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.080+0000 7ff7a5f9b740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56360fc5d6c0 0x56360fc5fab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:02.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.080+0000 7ff7a5f9b740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x56360fbb8310 con 0x56360fc55030 2024-04-02T14:49:02.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.080+0000 7ff7a5f9b740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x56360fb60690 con 0x56360fc5d6c0 2024-04-02T14:49:02.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.080+0000 7ff7a5f9b740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x56360fbb42c0 con 0x56360fc54350 2024-04-02T14:49:02.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.081+0000 7ff79be00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56360fc54350 0x56360fc547d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:02.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.081+0000 7ff79be00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56360fc54350 0x56360fc547d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:59408/0 (socket says 172.21.15.90:59408) 2024-04-02T14:49:02.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.081+0000 7ff798c00640 1 --2- 172.21.15.90:0/2266466641 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7ff7800a3e60 0x7ff7800a6320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:02.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.081+0000 7ff79be00640 1 -- 172.21.15.90:0/2496469396 learned_addr learned my addr 172.21.15.90:0/2496469396 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:49:02.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.081+0000 7ff79b400640 1 --2- 172.21.15.90:0/2496469396 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56360fc5d6c0 0x56360fc5fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:02.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.081+0000 7ff79b400640 1 --2- 172.21.15.90:0/2496469396 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56360fc55030 0x56360fc53d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:02.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.082+0000 7ff79be00640 1 -- 172.21.15.90:0/2496469396 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56360fc5d6c0 msgr2=0x56360fc5fab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:49:02.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.082+0000 7ff79be00640 1 --2- 172.21.15.90:0/2496469396 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56360fc5d6c0 0x56360fc5fab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:02.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.082+0000 7ff79be00640 1 -- 172.21.15.90:0/2496469396 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56360fc55030 msgr2=0x56360fc53d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:49:02.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.082+0000 7ff79be00640 1 --2- 172.21.15.90:0/2496469396 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56360fc55030 0x56360fc53d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:02.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.082+0000 7ff79be00640 1 -- 172.21.15.90:0/2496469396 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56360f9947e0 con 0x56360fc54350 2024-04-02T14:49:02.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.081+0000 7ff798c00640 1 -- 172.21.15.90:0/2266466641 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 8550+0+0 (secure 0 0 0) 0x7ff7940f25c0 con 0x7ff79c11b2e0 2024-04-02T14:49:02.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.082+0000 7ff7a1200640 1 --2- 172.21.15.90:0/2266466641 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7ff7800a3e60 0x7ff7800a6320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:02.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.083+0000 7ff7a1200640 1 --2- 172.21.15.90:0/2266466641 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7ff7800a3e60 0x7ff7800a6320 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7ff7940c6d30 tx=0x7ff79402c100 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:49:02.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.083+0000 7ff79be00640 1 --2- 172.21.15.90:0/2496469396 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56360fc54350 0x56360fc547d0 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7ff78c009ad0 tx=0x7ff78c009f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a47818d27f6d9a3e server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:49:02.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.083+0000 7ff791600640 1 -- 172.21.15.90:0/2496469396 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff78c00fd20 con 0x56360fc54350 2024-04-02T14:49:02.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.084+0000 7ff791600640 1 -- 172.21.15.90:0/2496469396 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff78c01a960 con 0x56360fc54350 2024-04-02T14:49:02.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.084+0000 7ff791600640 1 -- 172.21.15.90:0/2496469396 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff78c011a50 con 0x56360fc54350 2024-04-02T14:49:02.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.084+0000 7ff7a5f9b740 1 -- 172.21.15.90:0/2496469396 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56360fc54350 msgr2=0x56360fc547d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:49:02.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.084+0000 7ff7a5f9b740 1 --2- 172.21.15.90:0/2496469396 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56360fc54350 0x56360fc547d0 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7ff78c009ad0 tx=0x7ff78c009f90 comp rx=0 tx=0).stop 2024-04-02T14:49:02.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.085+0000 7ff7a5f9b740 1 -- 172.21.15.90:0/2496469396 shutdown_connections 2024-04-02T14:49:02.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.085+0000 7ff7a5f9b740 1 --2- 172.21.15.90:0/2496469396 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56360fc5d6c0 0x56360fc5fab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:02.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.085+0000 7ff7a5f9b740 1 --2- 172.21.15.90:0/2496469396 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56360fc54350 0x56360fc547d0 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:02.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.085+0000 7ff7a5f9b740 1 --2- 172.21.15.90:0/2496469396 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56360fc55030 0x56360fc53d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:02.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.085+0000 7ff7a5f9b740 1 -- 172.21.15.90:0/2496469396 >> 172.21.15.90:0/2496469396 conn(0x56360fbe8270 msgr2=0x56360fc4eda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:49:02.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.085+0000 7ff7a5f9b740 1 -- 172.21.15.90:0/2496469396 shutdown_connections 2024-04-02T14:49:02.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.086+0000 7ff7a5f9b740 1 -- 172.21.15.90:0/2496469396 wait complete. 2024-04-02T14:49:02.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.086+0000 7ff7a5f9b740 1 -- start start 2024-04-02T14:49:02.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.087+0000 7ff7a5f9b740 1 Processor -- start 2024-04-02T14:49:02.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.088+0000 7ff7a5f9b740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56360fc54350 0x56360fc5a5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:02.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.088+0000 7ff7a5f9b740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56360fc55030 0x56360fc5ab10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:02.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.089+0000 7ff7a5f9b740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56360fc5d6c0 0x56360fc5b050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:02.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.089+0000 7ff7a5f9b740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x56360fc60580 con 0x56360fc55030 2024-04-02T14:49:02.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.089+0000 7ff7a5f9b740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x56360fb73a60 con 0x56360fc54350 2024-04-02T14:49:02.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.089+0000 7ff7a5f9b740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x56360fc60700 con 0x56360fc5d6c0 2024-04-02T14:49:02.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.089+0000 7ff7a5f9b740 10 client.0 _ll_register_callbacks cb 0x56360fb79030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:49:02.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.089+0000 7ff79b400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56360fc54350 0x56360fc5a5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:02.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.089+0000 7ff79b400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56360fc54350 0x56360fc5a5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:42400/0 (socket says 172.21.15.90:42400) 2024-04-02T14:49:02.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.089+0000 7ff79b400640 1 -- 172.21.15.90:0/102658370 learned_addr learned my addr 172.21.15.90:0/102658370 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:49:02.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.090+0000 7ff79b400640 1 --2- 172.21.15.90:0/102658370 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56360fc5d6c0 0x56360fc5b050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:02.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.090+0000 7ff79b400640 1 -- 172.21.15.90:0/102658370 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56360fc5d6c0 msgr2=0x56360fc5b050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:49:02.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.090+0000 7ff79b400640 1 --2- 172.21.15.90:0/102658370 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56360fc5d6c0 0x56360fc5b050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:02.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.090+0000 7ff79b400640 1 -- 172.21.15.90:0/102658370 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56360fc55030 msgr2=0x56360fc5ab10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:49:02.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.090+0000 7ff79b400640 1 --2- 172.21.15.90:0/102658370 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56360fc55030 0x56360fc5ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:02.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.090+0000 7ff79b400640 1 -- 172.21.15.90:0/102658370 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56360f9f4f70 con 0x56360fc54350 2024-04-02T14:49:02.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.091+0000 7ff79b400640 1 --2- 172.21.15.90:0/102658370 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56360fc54350 0x56360fc5a5d0 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7ff78400b770 tx=0x7ff78400b480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fd5e8de29ff402a6 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:49:02.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.091+0000 7ff78a000640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:49:02.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.092+0000 7ff78a000640 1 -- 172.21.15.90:0/102658370 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff784012ad0 con 0x56360fc54350 2024-04-02T14:49:02.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.092+0000 7ff7a5f9b740 1 -- 172.21.15.90:0/102658370 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff78c01a960 con 0x56360fc54350 2024-04-02T14:49:02.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.092+0000 7ff78a000640 1 -- 172.21.15.90:0/102658370 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 8550+0+0 (secure 0 0 0) 0x7ff784012c70 con 0x56360fc54350 2024-04-02T14:49:02.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.093+0000 7ff78a000640 1 -- 172.21.15.90:0/102658370 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7ff784005020 con 0x56360fc54350 2024-04-02T14:49:02.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.093+0000 7ff7a5f9b740 10 client.29903 fetch_fsmap learned FSMap version 27 2024-04-02T14:49:02.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.093+0000 7ff7a5f9b740 1 -- 172.21.15.90:0/102658370 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x56360f88c170 con 0x56360fc54350 2024-04-02T14:49:02.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.093+0000 7ff78a000640 1 -- 172.21.15.90:0/102658370 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff78c01a960 con 0x56360fc54350 2024-04-02T14:49:02.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.094+0000 7ff78a000640 1 -- 172.21.15.90:0/102658370 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7ff784028020 con 0x56360fc54350 2024-04-02T14:49:02.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.094+0000 7ff7a5f9b740 10 client.29903 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:49:02.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.094+0000 7ff7a5f9b740 10 client.29903 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:49:02.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.094+0000 7ff7a5f9b740 20 client.29903 populate_metadata read hostname 'smithi090' 2024-04-02T14:49:02.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.094+0000 7ff7a5f9b740 1 --2- 172.21.15.90:0/102658370 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x56360fc6dbb0 0x56360fc6ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:02.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.095+0000 7ff7a5f9b740 4 client.29903 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:49:02.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.095+0000 7ff7a5f9b740 1 -- 172.21.15.90:0/102658370 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56360fbc5850 con 0x56360fc6dbb0 2024-04-02T14:49:02.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.095+0000 7ff79be00640 1 --2- 172.21.15.90:0/102658370 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x56360fc6dbb0 0x56360fc6ffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:02.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.096+0000 7ff79be00640 1 --2- 172.21.15.90:0/102658370 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x56360fc6dbb0 0x56360fc6ffe0 crc :-1 s=READY pgs=362 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=818c03f81885ad0 server_cookie=2984b80aea8d8f5 in_seq=0 out_seq=0 2024-04-02T14:49:02.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.096+0000 7ff78a000640 10 client.29903 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:49:02.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.097+0000 7ff78a000640 1 -- 172.21.15.90:0/102658370 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56360fbc5850 con 0x56360fc6dbb0 2024-04-02T14:49:02.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.097+0000 7ff78a000640 10 client.29903 handle_command_reply: tid=0 2024-04-02T14:49:02.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.097+0000 7ff7a5f9b740 1 client.29903 shutdown 2024-04-02T14:49:02.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.097+0000 7ff7a5f9b740 1 -- 172.21.15.90:0/102658370 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56360fc54350 msgr2=0x56360fc5a5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:49:02.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.097+0000 7ff7a5f9b740 1 --2- 172.21.15.90:0/102658370 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56360fc54350 0x56360fc5a5d0 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7ff78400b770 tx=0x7ff78400b480 comp rx=0 tx=0).stop 2024-04-02T14:49:02.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.097+0000 7ff7a5f9b740 1 -- 172.21.15.90:0/102658370 shutdown_connections 2024-04-02T14:49:02.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.098+0000 7ff7a5f9b740 1 --2- 172.21.15.90:0/102658370 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x56360fc6dbb0 0x56360fc6ffe0 crc :-1 s=READY pgs=362 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:02.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.098+0000 7ff7a5f9b740 1 --2- 172.21.15.90:0/102658370 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56360fc5d6c0 0x56360fc5b050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:02.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.098+0000 7ff78a000640 0 client.29903 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:49:02.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.098+0000 7ff7a5f9b740 1 --2- 172.21.15.90:0/102658370 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56360fc55030 0x56360fc5ab10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:02.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.098+0000 7ff7a5f9b740 1 --2- 172.21.15.90:0/102658370 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56360fc54350 0x56360fc5a5d0 unknown :-1 s=CLOSED pgs=974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:02.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.098+0000 7ff7a5f9b740 1 -- 172.21.15.90:0/102658370 >> 172.21.15.90:0/102658370 conn(0x56360fbe8270 msgr2=0x56360fc50af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:49:02.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.098+0000 7ff7a5f9b740 1 -- 172.21.15.90:0/102658370 shutdown_connections 2024-04-02T14:49:02.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.100+0000 7ff7a5f9b740 1 -- 172.21.15.90:0/102658370 wait complete. 2024-04-02T14:49:02.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.100+0000 7ff7a5f9b740 20 client.29903 trim_cache size 0 max 16384 2024-04-02T14:49:02.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.133+0000 7ff7a5f9b740 1 Processor -- start 2024-04-02T14:49:02.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.133+0000 7ff7a5f9b740 1 -- start start 2024-04-02T14:49:02.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.133+0000 7ff7a5f9b740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56360fc54350 0x56360fc5cd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:02.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.133+0000 7ff7a5f9b740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56360fc55030 0x56360fc5aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:02.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.133+0000 7ff7a5f9b740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56360fc5d6c0 0x56360fc50870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:02.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.134+0000 7ff7a5f9b740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7ff79c1c4f60 con 0x56360fc5d6c0 2024-04-02T14:49:02.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.134+0000 7ff7a5f9b740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7ff79c1c6aa0 con 0x56360fc54350 2024-04-02T14:49:02.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.134+0000 7ff7a5f9b740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7ff79c1c5630 con 0x56360fc55030 2024-04-02T14:49:02.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.134+0000 7ff79b400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56360fc54350 0x56360fc5cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:02.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.134+0000 7ff79b400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56360fc54350 0x56360fc5cd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:42406/0 (socket says 172.21.15.90:42406) 2024-04-02T14:49:02.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.134+0000 7ff79b400640 1 -- 172.21.15.90:0/2113685720 learned_addr learned my addr 172.21.15.90:0/2113685720 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:49:02.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.134+0000 7ff79b400640 1 --2- 172.21.15.90:0/2113685720 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56360fc5d6c0 0x56360fc50870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:02.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.134+0000 7ff79be00640 1 --2- 172.21.15.90:0/2113685720 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56360fc55030 0x56360fc5aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:02.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.135+0000 7ff79be00640 1 -- 172.21.15.90:0/2113685720 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56360fc54350 msgr2=0x56360fc5cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:49:02.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.135+0000 7ff79be00640 1 --2- 172.21.15.90:0/2113685720 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56360fc54350 0x56360fc5cd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:02.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.135+0000 7ff79be00640 1 -- 172.21.15.90:0/2113685720 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56360fc5d6c0 msgr2=0x56360fc50870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:49:02.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.135+0000 7ff79be00640 1 --2- 172.21.15.90:0/2113685720 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56360fc5d6c0 0x56360fc50870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:02.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.136+0000 7ff79be00640 1 -- 172.21.15.90:0/2113685720 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56360fc4e5e0 con 0x56360fc55030 2024-04-02T14:49:02.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.136+0000 7ff79b400640 1 --2- 172.21.15.90:0/2113685720 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56360fc5d6c0 0x56360fc50870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:49:02.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.136+0000 7ff79be00640 1 --2- 172.21.15.90:0/2113685720 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56360fc55030 0x56360fc5aeb0 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x56360fc5bba0 tx=0x7ff78c011550 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aa528b08790445db server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:49:02.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.137+0000 7ff790c00640 1 -- 172.21.15.90:0/2113685720 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff78c005850 con 0x56360fc55030 2024-04-02T14:49:02.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.137+0000 7ff790c00640 1 -- 172.21.15.90:0/2113685720 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff78c005520 con 0x56360fc55030 2024-04-02T14:49:02.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.137+0000 7ff790c00640 1 -- 172.21.15.90:0/2113685720 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff78c011ae0 con 0x56360fc55030 2024-04-02T14:49:02.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.137+0000 7ff7a5f9b740 1 -- 172.21.15.90:0/2113685720 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56360fc55030 msgr2=0x56360fc5aeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:49:02.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.137+0000 7ff7a5f9b740 1 --2- 172.21.15.90:0/2113685720 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56360fc55030 0x56360fc5aeb0 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x56360fc5bba0 tx=0x7ff78c011550 comp rx=0 tx=0).stop 2024-04-02T14:49:02.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.138+0000 7ff7a5f9b740 1 -- 172.21.15.90:0/2113685720 shutdown_connections 2024-04-02T14:49:02.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.138+0000 7ff7a5f9b740 1 --2- 172.21.15.90:0/2113685720 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56360fc5d6c0 0x56360fc50870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:02.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.138+0000 7ff7a5f9b740 1 --2- 172.21.15.90:0/2113685720 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56360fc55030 0x56360fc5aeb0 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:02.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.138+0000 7ff7a5f9b740 1 --2- 172.21.15.90:0/2113685720 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56360fc54350 0x56360fc5cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:02.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.138+0000 7ff7a5f9b740 1 -- 172.21.15.90:0/2113685720 >> 172.21.15.90:0/2113685720 conn(0x56360fbe8270 msgr2=0x56360fc50160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:49:02.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.138+0000 7ff7a5f9b740 1 -- 172.21.15.90:0/2113685720 shutdown_connections 2024-04-02T14:49:02.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.139+0000 7ff7a5f9b740 1 -- 172.21.15.90:0/2113685720 wait complete. 2024-04-02T14:49:02.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.139+0000 7ff7a5f9b740 1 -- start start 2024-04-02T14:49:02.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.139+0000 7ff7a5f9b740 1 Processor -- start 2024-04-02T14:49:02.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.140+0000 7ff7a5f9b740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56360fc54350 0x56360fc659e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:02.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.140+0000 7ff7a5f9b740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56360fc55030 0x56360fc65f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:02.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.140+0000 7ff7a5f9b740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56360fc5d6c0 0x56360fc629d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:02.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.140+0000 7ff7a5f9b740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7ff79c1c63d0 con 0x56360fc5d6c0 2024-04-02T14:49:02.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.140+0000 7ff7a5f9b740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7ff79c1c5d00 con 0x56360fc55030 2024-04-02T14:49:02.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.140+0000 7ff7a5f9b740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7ff79c1c2f00 con 0x56360fc54350 2024-04-02T14:49:02.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.140+0000 7ff7a5f9b740 10 client.0 _ll_register_callbacks cb 0x7ff78400bd40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:49:02.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.141+0000 7ff79be00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56360fc55030 0x56360fc65f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:02.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.141+0000 7ff79be00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56360fc55030 0x56360fc65f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:42414/0 (socket says 172.21.15.90:42414) 2024-04-02T14:49:02.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.141+0000 7ff79be00640 1 -- 172.21.15.90:0/3447459477 learned_addr learned my addr 172.21.15.90:0/3447459477 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:49:02.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.141+0000 7ff79b400640 1 --2- 172.21.15.90:0/3447459477 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56360fc54350 0x56360fc659e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:02.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.141+0000 7ff79be00640 1 -- 172.21.15.90:0/3447459477 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56360fc54350 msgr2=0x56360fc659e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:49:02.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.141+0000 7ff79be00640 1 --2- 172.21.15.90:0/3447459477 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56360fc54350 0x56360fc659e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:02.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.141+0000 7ff79be00640 1 -- 172.21.15.90:0/3447459477 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56360fc5d6c0 msgr2=0x56360fc629d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:49:02.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.141+0000 7ff79be00640 1 --2- 172.21.15.90:0/3447459477 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56360fc5d6c0 0x56360fc629d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:02.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.141+0000 7ff79be00640 1 -- 172.21.15.90:0/3447459477 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56360fc63200 con 0x56360fc55030 2024-04-02T14:49:02.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.142+0000 7ff79be00640 1 --2- 172.21.15.90:0/3447459477 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56360fc55030 0x56360fc65f20 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7ff78c00a010 tx=0x7ff78c00f920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=207b742b10223a0c server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:49:02.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.142+0000 7ff78a000640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:49:02.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.143+0000 7ff78a000640 1 -- 172.21.15.90:0/3447459477 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff78c034060 con 0x56360fc55030 2024-04-02T14:49:02.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.143+0000 7ff78a000640 1 -- 172.21.15.90:0/3447459477 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 8550+0+0 (secure 0 0 0) 0x7ff78c00fb20 con 0x56360fc55030 2024-04-02T14:49:02.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.143+0000 7ff7a5f9b740 1 -- 172.21.15.90:0/3447459477 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff78c005520 con 0x56360fc55030 2024-04-02T14:49:02.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.144+0000 7ff78a000640 1 -- 172.21.15.90:0/3447459477 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7ff78c02b040 con 0x56360fc55030 2024-04-02T14:49:02.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.144+0000 7ff7a5f9b740 10 client.29909 fetch_fsmap learned FSMap version 27 2024-04-02T14:49:02.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.144+0000 7ff7a5f9b740 1 -- 172.21.15.90:0/3447459477 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x56360fcd4c90 con 0x56360fc55030 2024-04-02T14:49:02.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.145+0000 7ff78a000640 1 -- 172.21.15.90:0/3447459477 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff78c005520 con 0x56360fc55030 2024-04-02T14:49:02.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.145+0000 7ff78a000640 1 -- 172.21.15.90:0/3447459477 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7ff78c011b90 con 0x56360fc55030 2024-04-02T14:49:02.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.145+0000 7ff7a5f9b740 10 client.29909 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:49:02.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.145+0000 7ff7a5f9b740 10 client.29909 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:49:02.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.145+0000 7ff7a5f9b740 20 client.29909 populate_metadata read hostname 'smithi090' 2024-04-02T14:49:02.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.145+0000 7ff7a5f9b740 1 --2- 172.21.15.90:0/3447459477 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x56360fc6dbb0 0x56360fcd70d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:02.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.145+0000 7ff7a5f9b740 4 client.29909 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-02T14:49:02.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.145+0000 7ff7a5f9b740 1 -- 172.21.15.90:0/3447459477 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x56360fbbfba0 con 0x56360fc6dbb0 2024-04-02T14:49:02.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.146+0000 7ff79b400640 1 --2- 172.21.15.90:0/3447459477 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x56360fc6dbb0 0x56360fcd70d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:02.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.147+0000 7ff79b400640 1 --2- 172.21.15.90:0/3447459477 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x56360fc6dbb0 0x56360fcd70d0 crc :-1 s=READY pgs=363 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a570ec0c9b715801 server_cookie=a0ff9e31d7d9c124 in_seq=0 out_seq=0 2024-04-02T14:49:02.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.147+0000 7ff78a000640 10 client.29909 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:49:02.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.148+0000 7ff78a000640 1 -- 172.21.15.90:0/3447459477 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x56360fbbfba0 con 0x56360fc6dbb0 2024-04-02T14:49:02.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.148+0000 7ff78a000640 10 client.29909 handle_command_reply: tid=0 2024-04-02T14:49:02.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.148+0000 7ff7a5f9b740 1 client.29909 shutdown 2024-04-02T14:49:02.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.149+0000 7ff7a5f9b740 1 -- 172.21.15.90:0/3447459477 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56360fc55030 msgr2=0x56360fc65f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:49:02.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.149+0000 7ff7a5f9b740 1 --2- 172.21.15.90:0/3447459477 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56360fc55030 0x56360fc65f20 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7ff78c00a010 tx=0x7ff78c00f920 comp rx=0 tx=0).stop 2024-04-02T14:49:02.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.149+0000 7ff7a5f9b740 1 -- 172.21.15.90:0/3447459477 shutdown_connections 2024-04-02T14:49:02.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.149+0000 7ff7a5f9b740 1 --2- 172.21.15.90:0/3447459477 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x56360fc6dbb0 0x56360fcd70d0 crc :-1 s=READY pgs=363 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:02.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.149+0000 7ff78a000640 0 client.29909 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:49:02.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.149+0000 7ff7a5f9b740 1 --2- 172.21.15.90:0/3447459477 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56360fc5d6c0 0x56360fc629d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:02.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.149+0000 7ff7a5f9b740 1 --2- 172.21.15.90:0/3447459477 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56360fc55030 0x56360fc65f20 unknown :-1 s=CLOSED pgs=975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:02.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.149+0000 7ff7a5f9b740 1 --2- 172.21.15.90:0/3447459477 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56360fc54350 0x56360fc659e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:02.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.149+0000 7ff7a5f9b740 1 -- 172.21.15.90:0/3447459477 >> 172.21.15.90:0/3447459477 conn(0x56360fbe8270 msgr2=0x56360fc507d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:49:02.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.149+0000 7ff7a5f9b740 1 -- 172.21.15.90:0/3447459477 shutdown_connections 2024-04-02T14:49:02.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.150+0000 7ff7a5f9b740 1 -- 172.21.15.90:0/3447459477 wait complete. 2024-04-02T14:49:02.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.150+0000 7ff7a5f9b740 20 client.29909 trim_cache size 0 max 16384 2024-04-02T14:49:02.144 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-02T14:49:02.144 INFO:teuthology.orchestra.run.smithi090.stdout: "return_code": 0, 2024-04-02T14:49:02.145 INFO:teuthology.orchestra.run.smithi090.stdout: "scrub_tag": "a49b09f4-6159-4c9f-a8d5-db0b16e4eab6", 2024-04-02T14:49:02.145 INFO:teuthology.orchestra.run.smithi090.stdout: "mode": "asynchronous" 2024-04-02T14:49:02.145 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-02T14:49:02.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.152+0000 7ff7a3200640 1 -- 172.21.15.90:0/2266466641 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7ff7800a3e60 msgr2=0x7ff7800a6320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:49:02.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.152+0000 7ff7a3200640 1 --2- 172.21.15.90:0/2266466641 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7ff7800a3e60 0x7ff7800a6320 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7ff7940c6d30 tx=0x7ff79402c100 comp rx=0 tx=0).stop 2024-04-02T14:49:02.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.152+0000 7ff7a3200640 1 -- 172.21.15.90:0/2266466641 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff79c11b2e0 msgr2=0x7ff79c111f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:49:02.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.152+0000 7ff7a3200640 1 --2- 172.21.15.90:0/2266466641 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff79c11b2e0 0x7ff79c111f80 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7ff794037040 tx=0x7ff794035d10 comp rx=0 tx=0).stop 2024-04-02T14:49:02.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.152+0000 7ff7a3200640 1 -- 172.21.15.90:0/2266466641 shutdown_connections 2024-04-02T14:49:02.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.152+0000 7ff7a3200640 1 --2- 172.21.15.90:0/2266466641 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7ff7800a3e60 0x7ff7800a6320 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:02.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.152+0000 7ff7a3200640 1 --2- 172.21.15.90:0/2266466641 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff79c11b2e0 0x7ff79c111f80 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:02.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.152+0000 7ff7a3200640 1 --2- 172.21.15.90:0/2266466641 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff79c110320 0x7ff79c111a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:02.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.152+0000 7ff7a3200640 1 --2- 172.21.15.90:0/2266466641 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff79c10e350 0x7ff79c1189c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:02.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.152+0000 7ff7a3200640 1 -- 172.21.15.90:0/2266466641 >> 172.21.15.90:0/2266466641 conn(0x7ff79c06cbe0 msgr2=0x7ff79c0725d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:49:02.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.152+0000 7ff7a3200640 1 -- 172.21.15.90:0/2266466641 shutdown_connections 2024-04-02T14:49:02.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.153+0000 7ff7a3200640 1 -- 172.21.15.90:0/2266466641 wait complete. 2024-04-02T14:49:02.180 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-02T14:49:02.180 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-02T14:49:02.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.403+0000 7f22df600640 1 -- 172.21.15.90:0/1760886934 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f22d8103f10 msgr2=0x7f22d81042f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:49:02.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.403+0000 7f22df600640 1 --2- 172.21.15.90:0/1760886934 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f22d8103f10 0x7f22d81042f0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f22cc009900 tx=0x7f22cc02bc50 comp rx=0 tx=0).stop 2024-04-02T14:49:02.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.403+0000 7f22df600640 1 -- 172.21.15.90:0/1760886934 shutdown_connections 2024-04-02T14:49:02.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.403+0000 7f22df600640 1 --2- 172.21.15.90:0/1760886934 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f22d8108fe0 0x7f22d8110bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:02.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.403+0000 7f22df600640 1 --2- 172.21.15.90:0/1760886934 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f22d81048c0 0x7f22d8108900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:02.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.403+0000 7f22df600640 1 --2- 172.21.15.90:0/1760886934 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f22d8103f10 0x7f22d81042f0 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:02.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.403+0000 7f22df600640 1 -- 172.21.15.90:0/1760886934 >> 172.21.15.90:0/1760886934 conn(0x7f22d80ff7b0 msgr2=0x7f22d8101bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:49:02.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.403+0000 7f22df600640 1 -- 172.21.15.90:0/1760886934 shutdown_connections 2024-04-02T14:49:02.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.403+0000 7f22df600640 1 -- 172.21.15.90:0/1760886934 wait complete. 2024-04-02T14:49:02.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.404+0000 7f22df600640 1 Processor -- start 2024-04-02T14:49:02.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.404+0000 7f22df600640 1 -- start start 2024-04-02T14:49:02.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.405+0000 7f22df600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f22d8103f10 0x7f22d819a610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:02.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.405+0000 7f22df600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f22d8108fe0 0x7f22d819ab50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:02.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.405+0000 7f22df600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f22d819eed0 0x7f22d819f380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:02.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.405+0000 7f22df600640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f22d81140f0 con 0x7f22d8108fe0 2024-04-02T14:49:02.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.405+0000 7f22df600640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f22d8113f70 con 0x7f22d819eed0 2024-04-02T14:49:02.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.405+0000 7f22df600640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f22d8114270 con 0x7f22d8103f10 2024-04-02T14:49:02.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.405+0000 7f22dd600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f22d819eed0 0x7f22d819f380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:02.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.405+0000 7f22d7e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f22d8108fe0 0x7f22d819ab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:02.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.405+0000 7f22dd600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f22d819eed0 0x7f22d819f380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:42440/0 (socket says 172.21.15.90:42440) 2024-04-02T14:49:02.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.405+0000 7f22dd600640 1 -- 172.21.15.90:0/1246533252 learned_addr learned my addr 172.21.15.90:0/1246533252 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:49:02.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.405+0000 7f22dcc00640 1 --2- 172.21.15.90:0/1246533252 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f22d8103f10 0x7f22d819a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:02.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.405+0000 7f22dd600640 1 -- 172.21.15.90:0/1246533252 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f22d8103f10 msgr2=0x7f22d819a610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:49:02.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.405+0000 7f22dd600640 1 --2- 172.21.15.90:0/1246533252 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f22d8103f10 0x7f22d819a610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:02.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.405+0000 7f22dd600640 1 -- 172.21.15.90:0/1246533252 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f22d8108fe0 msgr2=0x7f22d819ab50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:49:02.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.405+0000 7f22dd600640 1 --2- 172.21.15.90:0/1246533252 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f22d8108fe0 0x7f22d819ab50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:02.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.405+0000 7f22dd600640 1 -- 172.21.15.90:0/1246533252 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22d819f9c0 con 0x7f22d819eed0 2024-04-02T14:49:02.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.405+0000 7f22d7e00640 1 --2- 172.21.15.90:0/1246533252 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f22d8108fe0 0x7f22d819ab50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:49:02.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.406+0000 7f22dd600640 1 --2- 172.21.15.90:0/1246533252 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f22d819eed0 0x7f22d819f380 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7f22d000efd0 tx=0x7f22d000c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:49:02.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.406+0000 7f22d5600640 1 -- 172.21.15.90:0/1246533252 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22d0003b70 con 0x7f22d819eed0 2024-04-02T14:49:02.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.406+0000 7f22d5600640 1 -- 172.21.15.90:0/1246533252 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f22d0003d10 con 0x7f22d819eed0 2024-04-02T14:49:02.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.406+0000 7f22df600640 1 -- 172.21.15.90:0/1246533252 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22d819fcb0 con 0x7f22d819eed0 2024-04-02T14:49:02.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.406+0000 7f22d5600640 1 -- 172.21.15.90:0/1246533252 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22d00109f0 con 0x7f22d819eed0 2024-04-02T14:49:02.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.406+0000 7f22df600640 1 -- 172.21.15.90:0/1246533252 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22d81a7450 con 0x7f22d819eed0 2024-04-02T14:49:02.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.409+0000 7f22e222e740 1 Processor -- start 2024-04-02T14:49:02.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.409+0000 7f22d5600640 1 -- 172.21.15.90:0/1246533252 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f22d0004020 con 0x7f22d819eed0 2024-04-02T14:49:02.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.409+0000 7f22e222e740 1 -- start start 2024-04-02T14:49:02.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.410+0000 7f22e222e740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d24391b7b0 0x55d24391bb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:02.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.410+0000 7f22e222e740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d24391c0d0 0x55d243924770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:02.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.410+0000 7f22e222e740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d243924cb0 0x55d2439270a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:02.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.410+0000 7f22e222e740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55d24387f770 con 0x55d24391b7b0 2024-04-02T14:49:02.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.410+0000 7f22e222e740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55d243827700 con 0x55d243924cb0 2024-04-02T14:49:02.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.410+0000 7f22e222e740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55d24387b720 con 0x55d24391c0d0 2024-04-02T14:49:02.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.410+0000 7f22dcc00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d24391c0d0 0x55d243924770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:02.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.410+0000 7f22dcc00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d24391c0d0 0x55d243924770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:59466/0 (socket says 172.21.15.90:59466) 2024-04-02T14:49:02.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.410+0000 7f22d7e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d24391b7b0 0x55d24391bb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:02.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.410+0000 7f22dcc00640 1 -- 172.21.15.90:0/1060357740 learned_addr learned my addr 172.21.15.90:0/1060357740 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:49:02.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.410+0000 7f22d7e00640 1 --2- 172.21.15.90:0/1060357740 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d243924cb0 0x55d2439270a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:02.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.410+0000 7f22dcc00640 1 -- 172.21.15.90:0/1060357740 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d243924cb0 msgr2=0x55d2439270a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:49:02.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.410+0000 7f22dcc00640 1 --2- 172.21.15.90:0/1060357740 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d243924cb0 0x55d2439270a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:02.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.410+0000 7f22dcc00640 1 -- 172.21.15.90:0/1060357740 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d24391b7b0 msgr2=0x55d24391bb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:49:02.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.410+0000 7f22dcc00640 1 --2- 172.21.15.90:0/1060357740 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d24391b7b0 0x55d24391bb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:02.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.410+0000 7f22dcc00640 1 -- 172.21.15.90:0/1060357740 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d24365d7d0 con 0x55d24391c0d0 2024-04-02T14:49:02.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.411+0000 7f22dcc00640 1 --2- 172.21.15.90:0/1060357740 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d24391c0d0 0x55d243924770 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f22cc009900 tx=0x7f22cc0095e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=655f7195a4b805f1 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:49:02.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.411+0000 7f22c9c00640 1 -- 172.21.15.90:0/1060357740 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22cc042070 con 0x55d24391c0d0 2024-04-02T14:49:02.406 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.411+0000 7f22c9c00640 1 -- 172.21.15.90:0/1060357740 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f22cc02c6b0 con 0x55d24391c0d0 2024-04-02T14:49:02.406 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.411+0000 7f22c9c00640 1 -- 172.21.15.90:0/1060357740 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22cc004f60 con 0x55d24391c0d0 2024-04-02T14:49:02.406 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.411+0000 7f22d5600640 1 --2- 172.21.15.90:0/1246533252 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f22c00a3d20 0x7f22c00a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:02.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.411+0000 7f22d5600640 1 -- 172.21.15.90:0/1246533252 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 8550+0+0 (secure 0 0 0) 0x7f22d00d0c40 con 0x7f22d819eed0 2024-04-02T14:49:02.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.412+0000 7f22d7e00640 1 --2- 172.21.15.90:0/1246533252 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f22c00a3d20 0x7f22c00a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:02.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.412+0000 7f22e222e740 1 -- 172.21.15.90:0/1060357740 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d24391c0d0 msgr2=0x55d243924770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:49:02.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.412+0000 7f22e222e740 1 --2- 172.21.15.90:0/1060357740 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d24391c0d0 0x55d243924770 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f22cc009900 tx=0x7f22cc0095e0 comp rx=0 tx=0).stop 2024-04-02T14:49:02.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.412+0000 7f22e222e740 1 -- 172.21.15.90:0/1060357740 shutdown_connections 2024-04-02T14:49:02.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.412+0000 7f22e222e740 1 --2- 172.21.15.90:0/1060357740 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d243924cb0 0x55d2439270a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:02.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.412+0000 7f22e222e740 1 --2- 172.21.15.90:0/1060357740 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d24391c0d0 0x55d243924770 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:02.410 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.412+0000 7f22e222e740 1 --2- 172.21.15.90:0/1060357740 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d24391b7b0 0x55d24391bb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:02.410 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.412+0000 7f22e222e740 1 -- 172.21.15.90:0/1060357740 >> 172.21.15.90:0/1060357740 conn(0x55d243915ee0 msgr2=0x55d2439162c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:49:02.410 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.412+0000 7f22e222e740 1 -- 172.21.15.90:0/1060357740 shutdown_connections 2024-04-02T14:49:02.410 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.412+0000 7f22d7e00640 1 --2- 172.21.15.90:0/1246533252 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f22c00a3d20 0x7f22c00a61e0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f22c4009710 tx=0x7f22c4015040 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:49:02.410 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.412+0000 7f22e222e740 1 -- 172.21.15.90:0/1060357740 wait complete. 2024-04-02T14:49:02.410 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.413+0000 7f22e222e740 1 -- start start 2024-04-02T14:49:02.410 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.413+0000 7f22e222e740 1 Processor -- start 2024-04-02T14:49:02.410 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.413+0000 7f22e222e740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d24391b7b0 0x55d243921d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:02.410 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.413+0000 7f22e222e740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d24391c0d0 0x55d2439222b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:02.410 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.413+0000 7f22e222e740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d243924cb0 0x55d2439227f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:02.410 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.413+0000 7f22e222e740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55d243927a40 con 0x55d24391b7b0 2024-04-02T14:49:02.410 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.413+0000 7f22e222e740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55d24383aac0 con 0x55d243924cb0 2024-04-02T14:49:02.410 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.413+0000 7f22e222e740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55d243927bc0 con 0x55d24391c0d0 2024-04-02T14:49:02.410 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.413+0000 7f22e222e740 10 client.0 _ll_register_callbacks cb 0x55d24383d180 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:49:02.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.413+0000 7f22dcc00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d24391b7b0 0x55d243921d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:02.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.414+0000 7f22d7e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d24391c0d0 0x55d2439222b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:02.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.414+0000 7f22dd600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d243924cb0 0x55d2439227f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:02.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.414+0000 7f22dcc00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d24391b7b0 0x55d243921d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41656/0 (socket says 172.21.15.90:41656) 2024-04-02T14:49:02.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.414+0000 7f22dcc00640 1 -- 172.21.15.90:0/2665167527 learned_addr learned my addr 172.21.15.90:0/2665167527 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:49:02.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.414+0000 7f22dcc00640 1 -- 172.21.15.90:0/2665167527 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d24391c0d0 msgr2=0x55d2439222b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:49:02.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.414+0000 7f22dcc00640 1 --2- 172.21.15.90:0/2665167527 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d24391c0d0 0x55d2439222b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:02.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.414+0000 7f22dcc00640 1 -- 172.21.15.90:0/2665167527 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d243924cb0 msgr2=0x55d2439227f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:49:02.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.414+0000 7f22dcc00640 1 --2- 172.21.15.90:0/2665167527 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d243924cb0 0x55d2439227f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:02.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.414+0000 7f22dcc00640 1 -- 172.21.15.90:0/2665167527 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d2435933f0 con 0x55d24391b7b0 2024-04-02T14:49:02.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.414+0000 7f22dd600640 1 --2- 172.21.15.90:0/2665167527 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d243924cb0 0x55d2439227f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:49:02.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.414+0000 7f22dcc00640 1 --2- 172.21.15.90:0/2665167527 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d24391b7b0 0x55d243921d70 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f22cc02c190 tx=0x7f22cc02bc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=190e7d56a8541b81 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:49:02.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.414+0000 7f22be000640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-02T14:49:02.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.414+0000 7f22be000640 1 -- 172.21.15.90:0/2665167527 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22cc042040 con 0x55d24391b7b0 2024-04-02T14:49:02.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.414+0000 7f22be000640 1 -- 172.21.15.90:0/2665167527 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 8550+0+0 (secure 0 0 0) 0x7f22cc005160 con 0x55d24391b7b0 2024-04-02T14:49:02.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.415+0000 7f22e222e740 1 -- 172.21.15.90:0/2665167527 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55d243820630 con 0x55d24391b7b0 2024-04-02T14:49:02.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.415+0000 7f22be000640 1 -- 172.21.15.90:0/2665167527 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f22cc005360 con 0x55d24391b7b0 2024-04-02T14:49:02.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.415+0000 7f22e222e740 10 client.18684 fetch_fsmap learned FSMap version 27 2024-04-02T14:49:02.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.415+0000 7f22e222e740 1 -- 172.21.15.90:0/2665167527 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55d2435530f0 con 0x55d24391b7b0 2024-04-02T14:49:02.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.415+0000 7f22be000640 1 -- 172.21.15.90:0/2665167527 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55d243820630 con 0x55d24391b7b0 2024-04-02T14:49:02.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.415+0000 7f22be000640 1 -- 172.21.15.90:0/2665167527 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f22cc0524c0 con 0x55d24391b7b0 2024-04-02T14:49:02.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.415+0000 7f22e222e740 10 client.18684 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:49:02.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.415+0000 7f22e222e740 10 client.18684 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:49:02.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.415+0000 7f22e222e740 20 client.18684 populate_metadata read hostname 'smithi090' 2024-04-02T14:49:02.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.415+0000 7f22e222e740 1 --2- 172.21.15.90:0/2665167527 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55d243934b90 0x55d243936f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:02.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.415+0000 7f22e222e740 4 client.18684 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:49:02.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.415+0000 7f22e222e740 1 -- 172.21.15.90:0/2665167527 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d24388ccb0 con 0x55d243934b90 2024-04-02T14:49:02.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.416+0000 7f22d7e00640 1 --2- 172.21.15.90:0/2665167527 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55d243934b90 0x55d243936f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:02.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.416+0000 7f22be000640 10 client.18684 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:49:02.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.416+0000 7f22d7e00640 1 --2- 172.21.15.90:0/2665167527 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55d243934b90 0x55d243936f80 crc :-1 s=READY pgs=364 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=730c8d7fae0ae39c server_cookie=d8de4ad87091db06 in_seq=0 out_seq=0 2024-04-02T14:49:02.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.417+0000 7f22be000640 1 -- 172.21.15.90:0/2665167527 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d24388ccb0 con 0x55d243934b90 2024-04-02T14:49:02.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.417+0000 7f22be000640 10 client.18684 handle_command_reply: tid=0 2024-04-02T14:49:02.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.417+0000 7f22e222e740 1 client.18684 shutdown 2024-04-02T14:49:02.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.417+0000 7f22e222e740 1 -- 172.21.15.90:0/2665167527 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d24391b7b0 msgr2=0x55d243921d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:49:02.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.417+0000 7f22e222e740 1 --2- 172.21.15.90:0/2665167527 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d24391b7b0 0x55d243921d70 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f22cc02c190 tx=0x7f22cc02bc50 comp rx=0 tx=0).stop 2024-04-02T14:49:02.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.417+0000 7f22e222e740 1 -- 172.21.15.90:0/2665167527 shutdown_connections 2024-04-02T14:49:02.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.417+0000 7f22e222e740 1 --2- 172.21.15.90:0/2665167527 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55d243934b90 0x55d243936f80 crc :-1 s=READY pgs=364 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:02.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.417+0000 7f22e222e740 1 --2- 172.21.15.90:0/2665167527 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d243924cb0 0x55d2439227f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:02.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.417+0000 7f22e222e740 1 --2- 172.21.15.90:0/2665167527 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d24391c0d0 0x55d2439222b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:02.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.417+0000 7f22e222e740 1 --2- 172.21.15.90:0/2665167527 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d24391b7b0 0x55d243921d70 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:02.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.417+0000 7f22be000640 0 client.18684 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:49:02.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.417+0000 7f22e222e740 1 -- 172.21.15.90:0/2665167527 >> 172.21.15.90:0/2665167527 conn(0x55d243915ee0 msgr2=0x55d243917c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:49:02.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.417+0000 7f22e222e740 1 -- 172.21.15.90:0/2665167527 shutdown_connections 2024-04-02T14:49:02.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.417+0000 7f22e222e740 1 -- 172.21.15.90:0/2665167527 wait complete. 2024-04-02T14:49:02.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.417+0000 7f22e222e740 20 client.18684 trim_cache size 0 max 16384 2024-04-02T14:49:02.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.436+0000 7f22e222e740 1 Processor -- start 2024-04-02T14:49:02.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.437+0000 7f22e222e740 1 -- start start 2024-04-02T14:49:02.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.437+0000 7f22e222e740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d24391b7b0 0x55d2439244a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:02.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.437+0000 7f22e222e740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d24391c0d0 0x55d243930930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:02.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.437+0000 7f22e222e740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d243924cb0 0x55d243914de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:02.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.437+0000 7f22e222e740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f22d81afa70 con 0x55d24391c0d0 2024-04-02T14:49:02.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.437+0000 7f22e222e740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f22d81b15b0 con 0x55d243924cb0 2024-04-02T14:49:02.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.437+0000 7f22e222e740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f22d81b0140 con 0x55d24391b7b0 2024-04-02T14:49:02.431 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:49:02 smithi090 ceph-mon[31674]: pgmap v441: 97 pgs: 97 active+clean; 4.6 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 9.7 MiB/s wr, 796 op/s 2024-04-02T14:49:02.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.437+0000 7f22dcc00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d24391b7b0 0x55d2439244a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:02.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.437+0000 7f22dcc00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d24391b7b0 0x55d2439244a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:59482/0 (socket says 172.21.15.90:59482) 2024-04-02T14:49:02.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.437+0000 7f22dcc00640 1 -- 172.21.15.90:0/3550515597 learned_addr learned my addr 172.21.15.90:0/3550515597 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:49:02.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.437+0000 7f22dd600640 1 --2- 172.21.15.90:0/3550515597 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d243924cb0 0x55d243914de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:02.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.437+0000 7f22d7e00640 1 --2- 172.21.15.90:0/3550515597 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d24391c0d0 0x55d243930930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:02.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.438+0000 7f22dcc00640 1 -- 172.21.15.90:0/3550515597 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d243924cb0 msgr2=0x55d243914de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:49:02.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.438+0000 7f22dcc00640 1 --2- 172.21.15.90:0/3550515597 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d243924cb0 0x55d243914de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:02.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.438+0000 7f22dcc00640 1 -- 172.21.15.90:0/3550515597 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d24391c0d0 msgr2=0x55d243930930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:49:02.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.438+0000 7f22dcc00640 1 --2- 172.21.15.90:0/3550515597 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d24391c0d0 0x55d243930930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:02.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.438+0000 7f22dcc00640 1 -- 172.21.15.90:0/3550515597 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d24391bf00 con 0x55d24391b7b0 2024-04-02T14:49:02.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.438+0000 7f22d7e00640 1 --2- 172.21.15.90:0/3550515597 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d24391c0d0 0x55d243930930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:49:02.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.438+0000 7f22dcc00640 1 --2- 172.21.15.90:0/3550515597 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d24391b7b0 0x55d2439244a0 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f22cc009900 tx=0x7f22cc0095e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b197f24396e7555e server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:49:02.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.438+0000 7f22c9200640 1 -- 172.21.15.90:0/3550515597 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22cc05c070 con 0x55d24391b7b0 2024-04-02T14:49:02.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.438+0000 7f22c9200640 1 -- 172.21.15.90:0/3550515597 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f22cc0048d0 con 0x55d24391b7b0 2024-04-02T14:49:02.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.438+0000 7f22c9200640 1 -- 172.21.15.90:0/3550515597 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22cc042040 con 0x55d24391b7b0 2024-04-02T14:49:02.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.439+0000 7f22e222e740 1 -- 172.21.15.90:0/3550515597 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d24391b7b0 msgr2=0x55d2439244a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:49:02.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.439+0000 7f22e222e740 1 --2- 172.21.15.90:0/3550515597 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d24391b7b0 0x55d2439244a0 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f22cc009900 tx=0x7f22cc0095e0 comp rx=0 tx=0).stop 2024-04-02T14:49:02.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.439+0000 7f22e222e740 1 -- 172.21.15.90:0/3550515597 shutdown_connections 2024-04-02T14:49:02.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.439+0000 7f22e222e740 1 --2- 172.21.15.90:0/3550515597 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d243924cb0 0x55d243914de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:02.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.439+0000 7f22e222e740 1 --2- 172.21.15.90:0/3550515597 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d24391c0d0 0x55d243930930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:02.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.439+0000 7f22e222e740 1 --2- 172.21.15.90:0/3550515597 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d24391b7b0 0x55d2439244a0 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:02.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.439+0000 7f22e222e740 1 -- 172.21.15.90:0/3550515597 >> 172.21.15.90:0/3550515597 conn(0x55d243915ee0 msgr2=0x55d2439146d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:49:02.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.439+0000 7f22e222e740 1 -- 172.21.15.90:0/3550515597 shutdown_connections 2024-04-02T14:49:02.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.439+0000 7f22e222e740 1 -- 172.21.15.90:0/3550515597 wait complete. 2024-04-02T14:49:02.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.439+0000 7f22e222e740 1 -- start start 2024-04-02T14:49:02.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.439+0000 7f22e222e740 1 Processor -- start 2024-04-02T14:49:02.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.440+0000 7f22e222e740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d24391b7b0 0x55d2439288d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:02.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.440+0000 7f22e222e740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d24391c0d0 0x55d243928e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:02.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.440+0000 7f22e222e740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d243924cb0 0x55d243929350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:02.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.440+0000 7f22e222e740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f22d81b0ee0 con 0x55d243924cb0 2024-04-02T14:49:02.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.440+0000 7f22e222e740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f22d81b0810 con 0x55d24391c0d0 2024-04-02T14:49:02.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.440+0000 7f22e222e740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f22d81ada70 con 0x55d24391b7b0 2024-04-02T14:49:02.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.440+0000 7f22e222e740 10 client.0 _ll_register_callbacks cb 0x7f22c4009e10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:49:02.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.440+0000 7f22d7e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d24391c0d0 0x55d243928e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:02.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.440+0000 7f22dcc00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d24391b7b0 0x55d2439288d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:02.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.440+0000 7f22d7e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d24391c0d0 0x55d243928e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:42468/0 (socket says 172.21.15.90:42468) 2024-04-02T14:49:02.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.440+0000 7f22d7e00640 1 -- 172.21.15.90:0/1304069034 learned_addr learned my addr 172.21.15.90:0/1304069034 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:49:02.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.440+0000 7f22dd600640 1 --2- 172.21.15.90:0/1304069034 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d243924cb0 0x55d243929350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:02.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.441+0000 7f22dd600640 1 -- 172.21.15.90:0/1304069034 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d24391b7b0 msgr2=0x55d2439288d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:49:02.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.441+0000 7f22dd600640 1 --2- 172.21.15.90:0/1304069034 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d24391b7b0 0x55d2439288d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:02.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.441+0000 7f22dd600640 1 -- 172.21.15.90:0/1304069034 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d24391c0d0 msgr2=0x55d243928e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:49:02.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.441+0000 7f22dd600640 1 --2- 172.21.15.90:0/1304069034 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d24391c0d0 0x55d243928e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:02.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.441+0000 7f22dd600640 1 -- 172.21.15.90:0/1304069034 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d2439a2650 con 0x55d243924cb0 2024-04-02T14:49:02.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.441+0000 7f22dcc00640 1 --2- 172.21.15.90:0/1304069034 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d24391b7b0 0x55d2439288d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:49:02.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.441+0000 7f22d7e00640 1 --2- 172.21.15.90:0/1304069034 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d24391c0d0 0x55d243928e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:49:02.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.441+0000 7f22dd600640 1 --2- 172.21.15.90:0/1304069034 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d243924cb0 0x55d243929350 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f22d00ce2e0 tx=0x7f22d0004920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3bc0a746db80cda server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:49:02.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.441+0000 7f22be000640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-02T14:49:02.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.441+0000 7f22be000640 1 -- 172.21.15.90:0/1304069034 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22d009cb60 con 0x55d243924cb0 2024-04-02T14:49:02.440 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.441+0000 7f22be000640 1 -- 172.21.15.90:0/1304069034 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 8550+0+0 (secure 0 0 0) 0x7f22d0004310 con 0x55d243924cb0 2024-04-02T14:49:02.440 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.441+0000 7f22e222e740 1 -- 172.21.15.90:0/1304069034 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f22cc0048d0 con 0x55d243924cb0 2024-04-02T14:49:02.440 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.441+0000 7f22be000640 1 -- 172.21.15.90:0/1304069034 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f22d00a0f10 con 0x55d243924cb0 2024-04-02T14:49:02.440 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.441+0000 7f22e222e740 10 client.18690 fetch_fsmap learned FSMap version 27 2024-04-02T14:49:02.440 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.441+0000 7f22e222e740 1 -- 172.21.15.90:0/1304069034 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55d2439a3940 con 0x55d243924cb0 2024-04-02T14:49:02.440 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.442+0000 7f22be000640 1 -- 172.21.15.90:0/1304069034 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f22cc0048d0 con 0x55d243924cb0 2024-04-02T14:49:02.440 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.442+0000 7f22be000640 1 -- 172.21.15.90:0/1304069034 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f22d001f020 con 0x55d243924cb0 2024-04-02T14:49:02.440 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.442+0000 7f22e222e740 10 client.18690 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:49:02.440 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.442+0000 7f22e222e740 10 client.18690 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:49:02.440 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.442+0000 7f22e222e740 20 client.18690 populate_metadata read hostname 'smithi090' 2024-04-02T14:49:02.440 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.442+0000 7f22e222e740 1 --2- 172.21.15.90:0/1304069034 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55d2439a4010 0x55d2439a6400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:02.440 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.442+0000 7f22e222e740 4 client.18690 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-02T14:49:02.440 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.442+0000 7f22e222e740 1 -- 172.21.15.90:0/1304069034 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55d243887000 con 0x55d2439a4010 2024-04-02T14:49:02.440 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.442+0000 7f22dcc00640 1 --2- 172.21.15.90:0/1304069034 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55d2439a4010 0x55d2439a6400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:02.440 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.443+0000 7f22dcc00640 1 --2- 172.21.15.90:0/1304069034 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55d2439a4010 0x55d2439a6400 crc :-1 s=READY pgs=365 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=43cc44ec2051fc5c server_cookie=66eeb28ee6607c2a in_seq=0 out_seq=0 2024-04-02T14:49:02.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.443+0000 7f22be000640 10 client.18690 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:49:02.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.443+0000 7f22be000640 1 -- 172.21.15.90:0/1304069034 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+265 (crc 0 0 0) 0x55d243887000 con 0x55d2439a4010 2024-04-02T14:49:02.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.443+0000 7f22be000640 10 client.18690 handle_command_reply: tid=0 2024-04-02T14:49:02.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.443+0000 7f22e222e740 1 client.18690 shutdown 2024-04-02T14:49:02.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.443+0000 7f22e222e740 1 -- 172.21.15.90:0/1304069034 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d243924cb0 msgr2=0x55d243929350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:49:02.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.443+0000 7f22e222e740 1 --2- 172.21.15.90:0/1304069034 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d243924cb0 0x55d243929350 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f22d00ce2e0 tx=0x7f22d0004920 comp rx=0 tx=0).stop 2024-04-02T14:49:02.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.443+0000 7f22e222e740 1 -- 172.21.15.90:0/1304069034 shutdown_connections 2024-04-02T14:49:02.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.443+0000 7f22e222e740 1 --2- 172.21.15.90:0/1304069034 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55d2439a4010 0x55d2439a6400 crc :-1 s=READY pgs=365 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:02.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.443+0000 7f22e222e740 1 --2- 172.21.15.90:0/1304069034 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d243924cb0 0x55d243929350 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:02.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.443+0000 7f22e222e740 1 --2- 172.21.15.90:0/1304069034 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d24391c0d0 0x55d243928e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:02.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.443+0000 7f22e222e740 1 --2- 172.21.15.90:0/1304069034 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d24391b7b0 0x55d2439288d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:02.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.443+0000 7f22e222e740 1 -- 172.21.15.90:0/1304069034 >> 172.21.15.90:0/1304069034 conn(0x55d243915ee0 msgr2=0x55d243914f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:49:02.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.443+0000 7f22be000640 0 client.18690 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:49:02.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.443+0000 7f22e222e740 1 -- 172.21.15.90:0/1304069034 shutdown_connections 2024-04-02T14:49:02.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.443+0000 7f22e222e740 1 -- 172.21.15.90:0/1304069034 wait complete. 2024-04-02T14:49:02.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.444+0000 7f22e222e740 20 client.18690 trim_cache size 0 max 16384 2024-04-02T14:49:02.442 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-02T14:49:02.442 INFO:teuthology.orchestra.run.smithi090.stdout: "status": "scrub active (0 inodes in the stack)", 2024-04-02T14:49:02.442 INFO:teuthology.orchestra.run.smithi090.stdout: "scrubs": { 2024-04-02T14:49:02.442 INFO:teuthology.orchestra.run.smithi090.stdout: "a49b09f4-6159-4c9f-a8d5-db0b16e4eab6": { 2024-04-02T14:49:02.442 INFO:teuthology.orchestra.run.smithi090.stdout: "path": "/", 2024-04-02T14:49:02.442 INFO:teuthology.orchestra.run.smithi090.stdout: "tag": "a49b09f4-6159-4c9f-a8d5-db0b16e4eab6", 2024-04-02T14:49:02.442 INFO:teuthology.orchestra.run.smithi090.stdout: "options": "recursive,force" 2024-04-02T14:49:02.442 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:49:02.442 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:49:02.443 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-02T14:49:02.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.444+0000 7f22df600640 1 -- 172.21.15.90:0/1246533252 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f22c00a3d20 msgr2=0x7f22c00a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:49:02.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.444+0000 7f22df600640 1 --2- 172.21.15.90:0/1246533252 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f22c00a3d20 0x7f22c00a61e0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f22c4009710 tx=0x7f22c4015040 comp rx=0 tx=0).stop 2024-04-02T14:49:02.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.444+0000 7f22df600640 1 -- 172.21.15.90:0/1246533252 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f22d819eed0 msgr2=0x7f22d819f380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:49:02.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.444+0000 7f22df600640 1 --2- 172.21.15.90:0/1246533252 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f22d819eed0 0x7f22d819f380 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7f22d000efd0 tx=0x7f22d000c6a0 comp rx=0 tx=0).stop 2024-04-02T14:49:02.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.444+0000 7f22df600640 1 -- 172.21.15.90:0/1246533252 shutdown_connections 2024-04-02T14:49:02.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.444+0000 7f22df600640 1 --2- 172.21.15.90:0/1246533252 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f22c00a3d20 0x7f22c00a61e0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:02.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.444+0000 7f22df600640 1 --2- 172.21.15.90:0/1246533252 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f22d819eed0 0x7f22d819f380 unknown :-1 s=CLOSED pgs=976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:02.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.444+0000 7f22df600640 1 --2- 172.21.15.90:0/1246533252 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f22d8108fe0 0x7f22d819ab50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:02.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.444+0000 7f22df600640 1 --2- 172.21.15.90:0/1246533252 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f22d8103f10 0x7f22d819a610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:02.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.444+0000 7f22df600640 1 -- 172.21.15.90:0/1246533252 >> 172.21.15.90:0/1246533252 conn(0x7f22d80ff7b0 msgr2=0x7f22d8101ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:49:02.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.445+0000 7f22df600640 1 -- 172.21.15.90:0/1246533252 shutdown_connections 2024-04-02T14:49:02.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:02.445+0000 7f22df600640 1 -- 172.21.15.90:0/1246533252 wait complete. 2024-04-02T14:49:02.454 INFO:tasks.cephfs.filesystem:scrub status for tag:a49b09f4-6159-4c9f-a8d5-db0b16e4eab6 - {'path': '/', 'tag': 'a49b09f4-6159-4c9f-a8d5-db0b16e4eab6', 'options': 'recursive,force'} 2024-04-02T14:49:02.603 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:49:02 smithi092 ceph-mon[36497]: pgmap v441: 97 pgs: 97 active+clean; 4.6 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 9.7 MiB/s wr, 796 op/s 2024-04-02T14:49:02.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:49:02 smithi177 ceph-mon[36431]: pgmap v441: 97 pgs: 97 active+clean; 4.6 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 9.7 MiB/s wr, 796 op/s 2024-04-02T14:49:03.603 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:49:03 smithi092 ceph-mon[36497]: scrub queued for path: / 2024-04-02T14:49:03.603 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:49:03 smithi092 ceph-mon[36497]: scrub summary: idle+waiting paths [/] 2024-04-02T14:49:03.603 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:49:03 smithi092 ceph-mon[36497]: scrub summary: active paths [/] 2024-04-02T14:49:03.603 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:49:03 smithi092 ceph-mon[36497]: scrub summary: idle+waiting paths [/] 2024-04-02T14:49:03.678 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:49:03 smithi177 ceph-mon[36431]: scrub queued for path: / 2024-04-02T14:49:03.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:49:03 smithi177 ceph-mon[36431]: scrub summary: idle+waiting paths [/] 2024-04-02T14:49:03.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:49:03 smithi177 ceph-mon[36431]: scrub summary: active paths [/] 2024-04-02T14:49:03.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:49:03 smithi177 ceph-mon[36431]: scrub summary: idle+waiting paths [/] 2024-04-02T14:49:03.681 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:49:03 smithi090 ceph-mon[31674]: scrub queued for path: / 2024-04-02T14:49:03.682 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:49:03 smithi090 ceph-mon[31674]: scrub summary: idle+waiting paths [/] 2024-04-02T14:49:03.682 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:49:03 smithi090 ceph-mon[31674]: scrub summary: active paths [/] 2024-04-02T14:49:03.682 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:49:03 smithi090 ceph-mon[31674]: scrub summary: idle+waiting paths [/] 2024-04-02T14:49:04.602 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:49:04 smithi092 ceph-mon[36497]: pgmap v442: 97 pgs: 97 active+clean; 4.6 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 6.2 MiB/s wr, 517 op/s 2024-04-02T14:49:04.678 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:49:04 smithi177 ceph-mon[36431]: pgmap v442: 97 pgs: 97 active+clean; 4.6 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 6.2 MiB/s wr, 517 op/s 2024-04-02T14:49:04.681 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:49:04 smithi090 ceph-mon[31674]: pgmap v442: 97 pgs: 97 active+clean; 4.6 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 6.2 MiB/s wr, 517 op/s 2024-04-02T14:49:05.602 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:49:05 smithi092 ceph-mon[36497]: pgmap v443: 97 pgs: 97 active+clean; 4.6 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 13 MiB/s wr, 1.25k op/s 2024-04-02T14:49:05.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:49:05 smithi177 ceph-mon[36431]: pgmap v443: 97 pgs: 97 active+clean; 4.6 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 13 MiB/s wr, 1.25k op/s 2024-04-02T14:49:05.680 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:49:05 smithi090 ceph-mon[31674]: pgmap v443: 97 pgs: 97 active+clean; 4.6 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 13 MiB/s wr, 1.25k op/s 2024-04-02T14:49:06.823 INFO:teuthology.orchestra.run.smithi090.stderr:progress: 815.0 s, 1079.6 tps, lat 99.046 ms stddev 224.869 2024-04-02T14:49:08.103 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:49:07 smithi092 ceph-mon[36497]: pgmap v444: 97 pgs: 97 active+clean; 4.7 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 15 MiB/s wr, 1.47k op/s 2024-04-02T14:49:08.103 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:49:07 smithi092 ceph-mon[36497]: scrub summary: idle 2024-04-02T14:49:08.103 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:49:07 smithi092 ceph-mon[36497]: osdmap e106: 12 total, 12 up, 12 in 2024-04-02T14:49:08.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:49:07 smithi177 ceph-mon[36431]: pgmap v444: 97 pgs: 97 active+clean; 4.7 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 15 MiB/s wr, 1.47k op/s 2024-04-02T14:49:08.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:49:07 smithi177 ceph-mon[36431]: scrub summary: idle 2024-04-02T14:49:08.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:49:07 smithi177 ceph-mon[36431]: osdmap e106: 12 total, 12 up, 12 in 2024-04-02T14:49:08.180 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:49:07 smithi090 ceph-mon[31674]: pgmap v444: 97 pgs: 97 active+clean; 4.7 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 15 MiB/s wr, 1.47k op/s 2024-04-02T14:49:08.180 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:49:07 smithi090 ceph-mon[31674]: scrub summary: idle 2024-04-02T14:49:08.181 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:49:07 smithi090 ceph-mon[31674]: osdmap e106: 12 total, 12 up, 12 in 2024-04-02T14:49:10.102 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:49:09 smithi092 ceph-mon[36497]: pgmap v446: 97 pgs: 2 active+clean+snaptrim, 5 active+clean+snaptrim_wait, 90 active+clean; 4.6 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 156 KiB/s rd, 17 MiB/s wr, 1.69k op/s 2024-04-02T14:49:10.178 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:49:09 smithi177 ceph-mon[36431]: pgmap v446: 97 pgs: 2 active+clean+snaptrim, 5 active+clean+snaptrim_wait, 90 active+clean; 4.6 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 156 KiB/s rd, 17 MiB/s wr, 1.69k op/s 2024-04-02T14:49:10.180 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:49:09 smithi090 ceph-mon[31674]: pgmap v446: 97 pgs: 2 active+clean+snaptrim, 5 active+clean+snaptrim_wait, 90 active+clean; 4.6 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 156 KiB/s rd, 17 MiB/s wr, 1.69k op/s 2024-04-02T14:49:11.824 INFO:teuthology.orchestra.run.smithi090.stderr:progress: 820.0 s, 40.8 tps, lat 1884.706 ms stddev 1548.698 2024-04-02T14:49:12.102 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:49:11 smithi092 ceph-mon[36497]: pgmap v447: 97 pgs: 22 active+clean+snaptrim, 38 active+clean+snaptrim_wait, 37 active+clean; 4.4 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 159 KiB/s rd, 15 MiB/s wr, 1.55k op/s 2024-04-02T14:49:12.165 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.171+0000 7fcd27c00640 1 -- 172.21.15.90:0/2381504461 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcd2010d410 msgr2=0x7fcd20114fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:49:12.165 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.171+0000 7fcd27c00640 1 --2- 172.21.15.90:0/2381504461 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcd2010d410 0x7fcd20114fe0 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7fcd18009ca0 tx=0x7fcd1802c100 comp rx=0 tx=0).stop 2024-04-02T14:49:12.165 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.171+0000 7fcd27c00640 1 -- 172.21.15.90:0/2381504461 shutdown_connections 2024-04-02T14:49:12.165 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.171+0000 7fcd27c00640 1 --2- 172.21.15.90:0/2381504461 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcd2010d410 0x7fcd20114fe0 unknown :-1 s=CLOSED pgs=977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:12.165 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.171+0000 7fcd27c00640 1 --2- 172.21.15.90:0/2381504461 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcd20108cf0 0x7fcd2010cd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:12.165 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.171+0000 7fcd27c00640 1 --2- 172.21.15.90:0/2381504461 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcd20108340 0x7fcd20108720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:12.165 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.171+0000 7fcd27c00640 1 -- 172.21.15.90:0/2381504461 >> 172.21.15.90:0/2381504461 conn(0x7fcd20076070 msgr2=0x7fcd20078490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:49:12.165 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.172+0000 7fcd27c00640 1 -- 172.21.15.90:0/2381504461 shutdown_connections 2024-04-02T14:49:12.165 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.172+0000 7fcd27c00640 1 -- 172.21.15.90:0/2381504461 wait complete. 2024-04-02T14:49:12.166 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.173+0000 7fcd27c00640 1 Processor -- start 2024-04-02T14:49:12.167 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.174+0000 7fcd27c00640 1 -- start start 2024-04-02T14:49:12.168 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.175+0000 7fcd27c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcd20108340 0x7fcd201a45f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:12.168 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.175+0000 7fcd27c00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcd2010d410 0x7fcd201a4b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:12.169 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.175+0000 7fcd27c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcd2019e6b0 0x7fcd2019eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:12.169 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.175+0000 7fcd27c00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fcd201184d0 con 0x7fcd20108340 2024-04-02T14:49:12.169 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.175+0000 7fcd27c00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fcd20118350 con 0x7fcd2010d410 2024-04-02T14:49:12.169 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.175+0000 7fcd27c00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fcd20118650 con 0x7fcd2019e6b0 2024-04-02T14:49:12.169 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.175+0000 7fcd25c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcd2019e6b0 0x7fcd2019eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:12.169 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.175+0000 7fcd1fe00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcd2010d410 0x7fcd201a4b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:12.169 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.175+0000 7fcd25c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcd2019e6b0 0x7fcd2019eb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:45036/0 (socket says 172.21.15.90:45036) 2024-04-02T14:49:12.169 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.175+0000 7fcd25c00640 1 -- 172.21.15.90:0/62667901 learned_addr learned my addr 172.21.15.90:0/62667901 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:49:12.169 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.176+0000 7fcd25200640 1 --2- 172.21.15.90:0/62667901 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcd20108340 0x7fcd201a45f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:12.170 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.176+0000 7fcd25c00640 1 -- 172.21.15.90:0/62667901 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcd2010d410 msgr2=0x7fcd201a4b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:49:12.170 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.176+0000 7fcd25c00640 1 --2- 172.21.15.90:0/62667901 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcd2010d410 0x7fcd201a4b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:12.170 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.176+0000 7fcd25c00640 1 -- 172.21.15.90:0/62667901 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcd20108340 msgr2=0x7fcd201a45f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:49:12.170 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.176+0000 7fcd25c00640 1 --2- 172.21.15.90:0/62667901 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcd20108340 0x7fcd201a45f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:12.170 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.176+0000 7fcd25c00640 1 -- 172.21.15.90:0/62667901 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd2019f2b0 con 0x7fcd2019e6b0 2024-04-02T14:49:12.170 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.177+0000 7fcd25c00640 1 --2- 172.21.15.90:0/62667901 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcd2019e6b0 0x7fcd2019eb30 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7fcd180348f0 tx=0x7fcd18035d10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:49:12.173 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.177+0000 7fcd1d600640 1 -- 172.21.15.90:0/62667901 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd18039040 con 0x7fcd2019e6b0 2024-04-02T14:49:12.173 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.177+0000 7fcd1d600640 1 -- 172.21.15.90:0/62667901 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcd18044070 con 0x7fcd2019e6b0 2024-04-02T14:49:12.173 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.177+0000 7fcd1d600640 1 -- 172.21.15.90:0/62667901 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd180042a0 con 0x7fcd2019e6b0 2024-04-02T14:49:12.173 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.177+0000 7fcd27c00640 1 -- 172.21.15.90:0/62667901 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcd2019f540 con 0x7fcd2019e6b0 2024-04-02T14:49:12.173 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.178+0000 7fcd27c00640 1 -- 172.21.15.90:0/62667901 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcd201ab680 con 0x7fcd2019e6b0 2024-04-02T14:49:12.178 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:49:11 smithi177 ceph-mon[36431]: pgmap v447: 97 pgs: 22 active+clean+snaptrim, 38 active+clean+snaptrim_wait, 37 active+clean; 4.4 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 159 KiB/s rd, 15 MiB/s wr, 1.55k op/s 2024-04-02T14:49:12.180 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.185+0000 7fcd2a9fe740 1 Processor -- start 2024-04-02T14:49:12.181 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:49:11 smithi090 ceph-mon[31674]: pgmap v447: 97 pgs: 22 active+clean+snaptrim, 38 active+clean+snaptrim_wait, 37 active+clean; 4.4 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 159 KiB/s rd, 15 MiB/s wr, 1.55k op/s 2024-04-02T14:49:12.185 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.189+0000 7fcd1d600640 1 -- 172.21.15.90:0/62667901 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fcd18004440 con 0x7fcd2019e6b0 2024-04-02T14:49:12.185 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.191+0000 7fcd2a9fe740 1 -- start start 2024-04-02T14:49:12.185 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.191+0000 7fcd2a9fe740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5650a9c54eb0 0x5650a9c55290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:12.185 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.191+0000 7fcd2a9fe740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5650a9c557d0 0x5650a9c5de70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:12.185 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.191+0000 7fcd2a9fe740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5650a9c5e3b0 0x5650a9c607a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:12.186 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.191+0000 7fcd2a9fe740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5650a9bb9d70 con 0x5650a9c54eb0 2024-04-02T14:49:12.186 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.191+0000 7fcd2a9fe740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x5650a9b622a0 con 0x5650a9c557d0 2024-04-02T14:49:12.186 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.191+0000 7fcd2a9fe740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5650a9bb5d20 con 0x5650a9c5e3b0 2024-04-02T14:49:12.186 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.192+0000 7fcd1d600640 1 --2- 172.21.15.90:0/62667901 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fcd080a3f70 0x7fcd080a6430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:12.186 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.192+0000 7fcd1d600640 1 -- 172.21.15.90:0/62667901 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 8594+0+0 (secure 0 0 0) 0x7fcd180f2640 con 0x7fcd2019e6b0 2024-04-02T14:49:12.189 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.193+0000 7fcd1fe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5650a9c5e3b0 0x5650a9c607a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:12.189 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.193+0000 7fcd1fe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5650a9c5e3b0 0x5650a9c607a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:45050/0 (socket says 172.21.15.90:45050) 2024-04-02T14:49:12.189 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.193+0000 7fcd1fe00640 1 -- 172.21.15.90:0/2770311555 learned_addr learned my addr 172.21.15.90:0/2770311555 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:49:12.189 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.193+0000 7fcd25200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5650a9c557d0 0x5650a9c5de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:12.189 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.193+0000 7fcd1fe00640 1 -- 172.21.15.90:0/2770311555 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5650a9c557d0 msgr2=0x5650a9c5de70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:49:12.190 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.193+0000 7fcd1fe00640 1 --2- 172.21.15.90:0/2770311555 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5650a9c557d0 0x5650a9c5de70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:12.190 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.193+0000 7fcd1fe00640 1 -- 172.21.15.90:0/2770311555 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5650a9c54eb0 msgr2=0x5650a9c55290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:49:12.190 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.193+0000 7fcd1fe00640 1 --2- 172.21.15.90:0/2770311555 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5650a9c54eb0 0x5650a9c55290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:12.190 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.193+0000 7fcd1fe00640 1 -- 172.21.15.90:0/2770311555 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5650a9b76840 con 0x5650a9c5e3b0 2024-04-02T14:49:12.190 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.194+0000 7fcd1fe00640 1 --2- 172.21.15.90:0/2770311555 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5650a9c5e3b0 0x5650a9c607a0 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7fcd0c00ac60 tx=0x7fcd0c011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c73dfa314ef4c206 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:49:12.190 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.194+0000 7fcd25c00640 1 --2- 172.21.15.90:0/62667901 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fcd080a3f70 0x7fcd080a6430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:12.190 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.194+0000 7fcd12000640 1 -- 172.21.15.90:0/2770311555 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd0c005020 con 0x5650a9c5e3b0 2024-04-02T14:49:12.190 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.194+0000 7fcd12000640 1 -- 172.21.15.90:0/2770311555 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcd0c0047a0 con 0x5650a9c5e3b0 2024-04-02T14:49:12.190 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.194+0000 7fcd12000640 1 -- 172.21.15.90:0/2770311555 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd0c00f930 con 0x5650a9c5e3b0 2024-04-02T14:49:12.190 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.195+0000 7fcd2a9fe740 1 -- 172.21.15.90:0/2770311555 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5650a9c5e3b0 msgr2=0x5650a9c607a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:49:12.191 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.195+0000 7fcd2a9fe740 1 --2- 172.21.15.90:0/2770311555 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5650a9c5e3b0 0x5650a9c607a0 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7fcd0c00ac60 tx=0x7fcd0c011810 comp rx=0 tx=0).stop 2024-04-02T14:49:12.191 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.198+0000 7fcd2a9fe740 1 -- 172.21.15.90:0/2770311555 shutdown_connections 2024-04-02T14:49:12.191 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.198+0000 7fcd2a9fe740 1 --2- 172.21.15.90:0/2770311555 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5650a9c5e3b0 0x5650a9c607a0 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:12.192 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.198+0000 7fcd2a9fe740 1 --2- 172.21.15.90:0/2770311555 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5650a9c557d0 0x5650a9c5de70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:12.192 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.198+0000 7fcd2a9fe740 1 --2- 172.21.15.90:0/2770311555 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5650a9c54eb0 0x5650a9c55290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:12.192 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.199+0000 7fcd2a9fe740 1 -- 172.21.15.90:0/2770311555 >> 172.21.15.90:0/2770311555 conn(0x5650a9c4f5e0 msgr2=0x5650a9c4f9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:49:12.192 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.199+0000 7fcd2a9fe740 1 -- 172.21.15.90:0/2770311555 shutdown_connections 2024-04-02T14:49:12.192 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.199+0000 7fcd2a9fe740 1 -- 172.21.15.90:0/2770311555 wait complete. 2024-04-02T14:49:12.198 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.200+0000 7fcd2a9fe740 1 -- start start 2024-04-02T14:49:12.198 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.200+0000 7fcd25c00640 1 --2- 172.21.15.90:0/62667901 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fcd080a3f70 0x7fcd080a6430 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fcd180bab00 tx=0x7fcd1802c920 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:49:12.198 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.200+0000 7fcd2a9fe740 1 Processor -- start 2024-04-02T14:49:12.198 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.201+0000 7fcd2a9fe740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5650a9c54eb0 0x5650a9c5b460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:12.198 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.201+0000 7fcd2a9fe740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5650a9c557d0 0x5650a9c5b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:12.198 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.201+0000 7fcd2a9fe740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5650a9c5e3b0 0x5650a9c5bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:12.198 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.201+0000 7fcd2a9fe740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5650a9c721b0 con 0x5650a9c557d0 2024-04-02T14:49:12.198 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.201+0000 7fcd2a9fe740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x5650a9b75660 con 0x5650a9c54eb0 2024-04-02T14:49:12.198 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.201+0000 7fcd2a9fe740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5650a9c72330 con 0x5650a9c5e3b0 2024-04-02T14:49:12.199 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.201+0000 7fcd2a9fe740 10 client.0 _ll_register_callbacks cb 0x5650a9b75050 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:49:12.199 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.202+0000 7fcd1fe00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5650a9c54eb0 0x5650a9c5b460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:12.199 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.202+0000 7fcd1fe00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5650a9c54eb0 0x5650a9c5b460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:46474/0 (socket says 172.21.15.90:46474) 2024-04-02T14:49:12.199 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.202+0000 7fcd1fe00640 1 -- 172.21.15.90:0/1685216264 learned_addr learned my addr 172.21.15.90:0/1685216264 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:49:12.199 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.203+0000 7fcd1fe00640 1 --2- 172.21.15.90:0/1685216264 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5650a9c5e3b0 0x5650a9c5bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:12.199 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.203+0000 7fcd1fe00640 1 -- 172.21.15.90:0/1685216264 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5650a9c5e3b0 msgr2=0x5650a9c5bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:49:12.199 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.203+0000 7fcd1fe00640 1 --2- 172.21.15.90:0/1685216264 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5650a9c5e3b0 0x5650a9c5bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:12.199 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.203+0000 7fcd1fe00640 1 -- 172.21.15.90:0/1685216264 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5650a9c557d0 msgr2=0x5650a9c5b9a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-02T14:49:12.199 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.203+0000 7fcd1fe00640 1 --2- 172.21.15.90:0/1685216264 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5650a9c557d0 0x5650a9c5b9a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:12.199 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.203+0000 7fcd1fe00640 1 -- 172.21.15.90:0/1685216264 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5650a9b84e80 con 0x5650a9c54eb0 2024-04-02T14:49:12.199 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.203+0000 7fcd1fe00640 1 --2- 172.21.15.90:0/1685216264 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5650a9c54eb0 0x5650a9c5b460 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7fcd0c004770 tx=0x7fcd0c0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dada5019fa1be67e server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:49:12.199 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.203+0000 7fcd06a00640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:49:12.199 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.204+0000 7fcd06a00640 1 -- 172.21.15.90:0/1685216264 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd0c04e060 con 0x5650a9c54eb0 2024-04-02T14:49:12.199 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.204+0000 7fcd2a9fe740 1 -- 172.21.15.90:0/1685216264 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcd0c0047a0 con 0x5650a9c54eb0 2024-04-02T14:49:12.200 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.204+0000 7fcd06a00640 1 -- 172.21.15.90:0/1685216264 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 8594+0+0 (secure 0 0 0) 0x7fcd0c004b60 con 0x5650a9c54eb0 2024-04-02T14:49:12.200 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.205+0000 7fcd06a00640 1 -- 172.21.15.90:0/1685216264 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fcd0c004d60 con 0x5650a9c54eb0 2024-04-02T14:49:12.200 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.206+0000 7fcd2a9fe740 10 client.29939 fetch_fsmap learned FSMap version 27 2024-04-02T14:49:12.200 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.206+0000 7fcd2a9fe740 1 -- 172.21.15.90:0/1685216264 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5650a98ce210 con 0x5650a9c54eb0 2024-04-02T14:49:12.200 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.206+0000 7fcd06a00640 1 -- 172.21.15.90:0/1685216264 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcd0c0047a0 con 0x5650a9c54eb0 2024-04-02T14:49:12.205 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.209+0000 7fcd06a00640 1 -- 172.21.15.90:0/1685216264 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fcd0c021070 con 0x5650a9c54eb0 2024-04-02T14:49:12.205 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.209+0000 7fcd2a9fe740 10 client.29939 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:49:12.205 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.209+0000 7fcd2a9fe740 10 client.29939 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-02T14:49:12.205 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.209+0000 7fcd2a9fe740 20 client.29939 populate_metadata read hostname 'smithi090' 2024-04-02T14:49:12.205 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.209+0000 7fcd2a9fe740 1 --2- 172.21.15.90:0/1685216264 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5650a9c62030 0x5650a9c644b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:12.205 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.209+0000 7fcd2a9fe740 4 client.29939 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:49:12.205 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.209+0000 7fcd2a9fe740 1 -- 172.21.15.90:0/1685216264 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5650a9b69cd0 con 0x5650a9c62030 2024-04-02T14:49:12.208 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.213+0000 7fcd25200640 1 --2- 172.21.15.90:0/1685216264 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5650a9c62030 0x5650a9c644b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:12.210 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.216+0000 7fcd25200640 1 --2- 172.21.15.90:0/1685216264 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5650a9c62030 0x5650a9c644b0 crc :-1 s=READY pgs=366 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=59534ea4f3548ceb server_cookie=a4185980e4d95627 in_seq=0 out_seq=0 2024-04-02T14:49:12.210 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.216+0000 7fcd06a00640 10 client.29939 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:49:12.213 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.220+0000 7fcd06a00640 1 -- 172.21.15.90:0/1685216264 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5650a9b69cd0 con 0x5650a9c62030 2024-04-02T14:49:12.213 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.220+0000 7fcd06a00640 10 client.29939 handle_command_reply: tid=0 2024-04-02T14:49:12.213 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.220+0000 7fcd2a9fe740 1 client.29939 shutdown 2024-04-02T14:49:12.214 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.220+0000 7fcd2a9fe740 1 -- 172.21.15.90:0/1685216264 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5650a9c54eb0 msgr2=0x5650a9c5b460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:49:12.214 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.220+0000 7fcd2a9fe740 1 --2- 172.21.15.90:0/1685216264 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5650a9c54eb0 0x5650a9c5b460 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7fcd0c004770 tx=0x7fcd0c0046c0 comp rx=0 tx=0).stop 2024-04-02T14:49:12.214 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.220+0000 7fcd2a9fe740 1 -- 172.21.15.90:0/1685216264 shutdown_connections 2024-04-02T14:49:12.214 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.220+0000 7fcd2a9fe740 1 --2- 172.21.15.90:0/1685216264 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5650a9c62030 0x5650a9c644b0 crc :-1 s=READY pgs=366 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:12.214 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.220+0000 7fcd2a9fe740 1 --2- 172.21.15.90:0/1685216264 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5650a9c5e3b0 0x5650a9c5bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:12.214 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.221+0000 7fcd2a9fe740 1 --2- 172.21.15.90:0/1685216264 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5650a9c557d0 0x5650a9c5b9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:12.215 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.221+0000 7fcd2a9fe740 1 --2- 172.21.15.90:0/1685216264 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5650a9c54eb0 0x5650a9c5b460 unknown :-1 s=CLOSED pgs=978 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:12.215 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.221+0000 7fcd2a9fe740 1 -- 172.21.15.90:0/1685216264 >> 172.21.15.90:0/1685216264 conn(0x5650a9c4f5e0 msgr2=0x5650a9c51330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:49:12.215 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.221+0000 7fcd06a00640 0 client.29939 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:49:12.215 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.221+0000 7fcd2a9fe740 1 -- 172.21.15.90:0/1685216264 shutdown_connections 2024-04-02T14:49:12.215 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.221+0000 7fcd2a9fe740 1 -- 172.21.15.90:0/1685216264 wait complete. 2024-04-02T14:49:12.215 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.221+0000 7fcd2a9fe740 20 client.29939 trim_cache size 0 max 16384 2024-04-02T14:49:12.266 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.273+0000 7fcd2a9fe740 1 Processor -- start 2024-04-02T14:49:12.268 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.273+0000 7fcd2a9fe740 1 -- start start 2024-04-02T14:49:12.268 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.273+0000 7fcd2a9fe740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5650a9c54eb0 0x5650a9c7b070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:12.268 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.273+0000 7fcd2a9fe740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5650a9c557d0 0x5650a9c5bd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:12.268 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.274+0000 7fcd2a9fe740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5650a9c5e3b0 0x5650a9c4e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:12.268 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.274+0000 7fcd2a9fe740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fcd20073910 con 0x5650a9c54eb0 2024-04-02T14:49:12.268 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.274+0000 7fcd2a9fe740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fcd201ad0c0 con 0x5650a9c557d0 2024-04-02T14:49:12.268 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.274+0000 7fcd2a9fe740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fcd201abc50 con 0x5650a9c5e3b0 2024-04-02T14:49:12.268 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.274+0000 7fcd25200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5650a9c557d0 0x5650a9c5bd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:12.268 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.274+0000 7fcd25200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5650a9c557d0 0x5650a9c5bd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:46478/0 (socket says 172.21.15.90:46478) 2024-04-02T14:49:12.268 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.274+0000 7fcd25200640 1 -- 172.21.15.90:0/1180161810 learned_addr learned my addr 172.21.15.90:0/1180161810 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:49:12.268 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.274+0000 7fcd25200640 1 -- 172.21.15.90:0/1180161810 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5650a9c5e3b0 msgr2=0x5650a9c4e4c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-02T14:49:12.268 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.274+0000 7fcd25200640 1 --2- 172.21.15.90:0/1180161810 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5650a9c5e3b0 0x5650a9c4e4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:12.269 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.274+0000 7fcd25200640 1 -- 172.21.15.90:0/1180161810 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5650a9c54eb0 msgr2=0x5650a9c7b070 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-02T14:49:12.269 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.274+0000 7fcd25200640 1 --2- 172.21.15.90:0/1180161810 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5650a9c54eb0 0x5650a9c7b070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:12.269 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.274+0000 7fcd25200640 1 -- 172.21.15.90:0/1180161810 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5650a999a620 con 0x5650a9c557d0 2024-04-02T14:49:12.269 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.275+0000 7fcd25200640 1 --2- 172.21.15.90:0/1180161810 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5650a9c557d0 0x5650a9c5bd00 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x5650a9c54980 tx=0x7fcd14005f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2988aa4070a0e60c server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:49:12.269 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.275+0000 7fcd11600640 1 -- 172.21.15.90:0/1180161810 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd14013930 con 0x5650a9c557d0 2024-04-02T14:49:12.269 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.275+0000 7fcd11600640 1 -- 172.21.15.90:0/1180161810 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcd1401e030 con 0x5650a9c557d0 2024-04-02T14:49:12.269 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.275+0000 7fcd11600640 1 -- 172.21.15.90:0/1180161810 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd14011740 con 0x5650a9c557d0 2024-04-02T14:49:12.271 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.277+0000 7fcd2a9fe740 1 -- 172.21.15.90:0/1180161810 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5650a9c557d0 msgr2=0x5650a9c5bd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:49:12.271 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.277+0000 7fcd2a9fe740 1 --2- 172.21.15.90:0/1180161810 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5650a9c557d0 0x5650a9c5bd00 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x5650a9c54980 tx=0x7fcd14005f90 comp rx=0 tx=0).stop 2024-04-02T14:49:12.271 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.278+0000 7fcd2a9fe740 1 -- 172.21.15.90:0/1180161810 shutdown_connections 2024-04-02T14:49:12.272 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.279+0000 7fcd2a9fe740 1 --2- 172.21.15.90:0/1180161810 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5650a9c5e3b0 0x5650a9c4e4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:12.272 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.279+0000 7fcd2a9fe740 1 --2- 172.21.15.90:0/1180161810 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5650a9c557d0 0x5650a9c5bd00 unknown :-1 s=CLOSED pgs=979 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:12.272 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.279+0000 7fcd2a9fe740 1 --2- 172.21.15.90:0/1180161810 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5650a9c54eb0 0x5650a9c7b070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:12.273 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.279+0000 7fcd2a9fe740 1 -- 172.21.15.90:0/1180161810 >> 172.21.15.90:0/1180161810 conn(0x5650a9c4f5e0 msgr2=0x5650a9c4ddb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:49:12.273 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.280+0000 7fcd2a9fe740 1 -- 172.21.15.90:0/1180161810 shutdown_connections 2024-04-02T14:49:12.275 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.281+0000 7fcd2a9fe740 1 -- 172.21.15.90:0/1180161810 wait complete. 2024-04-02T14:49:12.275 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.281+0000 7fcd2a9fe740 1 -- start start 2024-04-02T14:49:12.275 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.281+0000 7fcd2a9fe740 1 Processor -- start 2024-04-02T14:49:12.279 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.286+0000 7fcd2a9fe740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5650a9c54eb0 0x5650a9c58d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:12.281 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.288+0000 7fcd2a9fe740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5650a9c557d0 0x5650a9cd16c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:12.284 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.289+0000 7fcd2a9fe740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5650a9c5e3b0 0x5650a9cd1c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:12.284 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.289+0000 7fcd2a9fe740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fcd201ac9f0 con 0x5650a9c557d0 2024-04-02T14:49:12.284 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.289+0000 7fcd2a9fe740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fcd201ac320 con 0x5650a9c5e3b0 2024-04-02T14:49:12.284 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.289+0000 7fcd2a9fe740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fcd2004f860 con 0x5650a9c54eb0 2024-04-02T14:49:12.284 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.289+0000 7fcd2a9fe740 10 client.0 _ll_register_callbacks cb 0x7fcd0c005560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:49:12.284 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.289+0000 7fcd1fe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5650a9c54eb0 0x5650a9c58d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:12.284 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.289+0000 7fcd1fe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5650a9c54eb0 0x5650a9c58d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:45068/0 (socket says 172.21.15.90:45068) 2024-04-02T14:49:12.284 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.290+0000 7fcd1fe00640 1 -- 172.21.15.90:0/1904551037 learned_addr learned my addr 172.21.15.90:0/1904551037 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:49:12.285 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.291+0000 7fcd1fe00640 1 --2- 172.21.15.90:0/1904551037 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5650a9c5e3b0 0x5650a9cd1c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:12.287 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.293+0000 7fcd1fe00640 1 -- 172.21.15.90:0/1904551037 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5650a9c5e3b0 msgr2=0x5650a9cd1c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:49:12.289 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.293+0000 7fcd1fe00640 1 --2- 172.21.15.90:0/1904551037 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5650a9c5e3b0 0x5650a9cd1c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:12.289 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.293+0000 7fcd1fe00640 1 -- 172.21.15.90:0/1904551037 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5650a9c557d0 msgr2=0x5650a9cd16c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:49:12.290 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.293+0000 7fcd1fe00640 1 --2- 172.21.15.90:0/1904551037 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5650a9c557d0 0x5650a9cd16c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:12.290 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.293+0000 7fcd1fe00640 1 -- 172.21.15.90:0/1904551037 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5650a9cd2140 con 0x5650a9c54eb0 2024-04-02T14:49:12.290 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.295+0000 7fcd1fe00640 1 --2- 172.21.15.90:0/1904551037 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5650a9c54eb0 0x5650a9c58d20 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7fcd0c011d90 tx=0x7fcd0c00abe0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7bb9df3d60f8c76f server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:49:12.290 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.295+0000 7fcd06a00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:49:12.292 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.299+0000 7fcd06a00640 1 -- 172.21.15.90:0/1904551037 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd0c025df0 con 0x5650a9c54eb0 2024-04-02T14:49:12.292 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.299+0000 7fcd06a00640 1 -- 172.21.15.90:0/1904551037 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 8594+0+0 (secure 0 0 0) 0x7fcd0c01e040 con 0x5650a9c54eb0 2024-04-02T14:49:12.293 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.300+0000 7fcd2a9fe740 1 -- 172.21.15.90:0/1904551037 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcd1401e030 con 0x5650a9c54eb0 2024-04-02T14:49:12.293 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.300+0000 7fcd06a00640 1 -- 172.21.15.90:0/1904551037 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fcd0c04e030 con 0x5650a9c54eb0 2024-04-02T14:49:12.293 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.300+0000 7fcd2a9fe740 10 client.29740 fetch_fsmap learned FSMap version 27 2024-04-02T14:49:12.294 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.301+0000 7fcd2a9fe740 1 -- 172.21.15.90:0/1904551037 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5650a9c71520 con 0x5650a9c54eb0 2024-04-02T14:49:12.294 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.301+0000 7fcd06a00640 1 -- 172.21.15.90:0/1904551037 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcd1401e030 con 0x5650a9c54eb0 2024-04-02T14:49:12.294 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.301+0000 7fcd06a00640 1 -- 172.21.15.90:0/1904551037 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fcd0c021050 con 0x5650a9c54eb0 2024-04-02T14:49:12.296 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.303+0000 7fcd2a9fe740 10 client.29740 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:49:12.296 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.303+0000 7fcd2a9fe740 10 client.29740 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-02T14:49:12.296 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.303+0000 7fcd2a9fe740 20 client.29740 populate_metadata read hostname 'smithi090' 2024-04-02T14:49:12.296 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.303+0000 7fcd2a9fe740 1 --2- 172.21.15.90:0/1904551037 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5650a9c768a0 0x5650a9c76cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:12.296 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.303+0000 7fcd2a9fe740 4 client.29740 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-02T14:49:12.297 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.303+0000 7fcd2a9fe740 1 -- 172.21.15.90:0/1904551037 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5650a9bc72b0 con 0x5650a9c768a0 2024-04-02T14:49:12.297 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.304+0000 7fcd25200640 1 --2- 172.21.15.90:0/1904551037 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5650a9c768a0 0x5650a9c76cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:12.298 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.305+0000 7fcd25200640 1 --2- 172.21.15.90:0/1904551037 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5650a9c768a0 0x5650a9c76cb0 crc :-1 s=READY pgs=367 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2b70ecaed7c2a772 server_cookie=de7e22d4dc4c8548 in_seq=0 out_seq=0 2024-04-02T14:49:12.298 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.305+0000 7fcd06a00640 10 client.29740 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:49:12.747 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.753+0000 7fcd06a00640 1 -- 172.21.15.90:0/1904551037 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5650a9bc72b0 con 0x5650a9c768a0 2024-04-02T14:49:12.747 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.753+0000 7fcd06a00640 10 client.29740 handle_command_reply: tid=0 2024-04-02T14:49:12.747 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.754+0000 7fcd2a9fe740 1 client.29740 shutdown 2024-04-02T14:49:12.748 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.754+0000 7fcd2a9fe740 1 -- 172.21.15.90:0/1904551037 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5650a9c54eb0 msgr2=0x5650a9c58d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:49:12.748 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.754+0000 7fcd2a9fe740 1 --2- 172.21.15.90:0/1904551037 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5650a9c54eb0 0x5650a9c58d20 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7fcd0c011d90 tx=0x7fcd0c00abe0 comp rx=0 tx=0).stop 2024-04-02T14:49:12.748 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.755+0000 7fcd2a9fe740 1 -- 172.21.15.90:0/1904551037 shutdown_connections 2024-04-02T14:49:12.748 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.755+0000 7fcd2a9fe740 1 --2- 172.21.15.90:0/1904551037 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5650a9c768a0 0x5650a9c76cb0 crc :-1 s=READY pgs=367 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:12.748 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.755+0000 7fcd2a9fe740 1 --2- 172.21.15.90:0/1904551037 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5650a9c5e3b0 0x5650a9cd1c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:12.748 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.755+0000 7fcd2a9fe740 1 --2- 172.21.15.90:0/1904551037 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5650a9c557d0 0x5650a9cd16c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:12.748 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.755+0000 7fcd2a9fe740 1 --2- 172.21.15.90:0/1904551037 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5650a9c54eb0 0x5650a9c58d20 secure :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7fcd0c011d90 tx=0x7fcd0c00abe0 comp rx=0 tx=0).stop 2024-04-02T14:49:12.748 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.755+0000 7fcd2a9fe740 1 -- 172.21.15.90:0/1904551037 >> 172.21.15.90:0/1904551037 conn(0x5650a9c4f5e0 msgr2=0x5650a9c51210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:49:12.749 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.755+0000 7fcd06a00640 0 client.29740 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:49:12.749 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.755+0000 7fcd2a9fe740 1 -- 172.21.15.90:0/1904551037 shutdown_connections 2024-04-02T14:49:12.749 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.755+0000 7fcd2a9fe740 1 -- 172.21.15.90:0/1904551037 wait complete. 2024-04-02T14:49:12.749 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.756+0000 7fcd2a9fe740 20 client.29740 trim_cache size 0 max 16384 2024-04-02T14:49:12.749 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:{ 2024-04-02T14:49:12.749 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "message": "", 2024-04-02T14:49:12.750 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "return_code": 0 2024-04-02T14:49:12.750 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:} 2024-04-02T14:49:12.750 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.757+0000 7fcd27c00640 1 -- 172.21.15.90:0/62667901 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fcd080a3f70 msgr2=0x7fcd080a6430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:49:12.751 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.757+0000 7fcd27c00640 1 --2- 172.21.15.90:0/62667901 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fcd080a3f70 0x7fcd080a6430 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fcd180bab00 tx=0x7fcd1802c920 comp rx=0 tx=0).stop 2024-04-02T14:49:12.751 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.757+0000 7fcd27c00640 1 -- 172.21.15.90:0/62667901 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcd2019e6b0 msgr2=0x7fcd2019eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:49:12.751 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.757+0000 7fcd27c00640 1 --2- 172.21.15.90:0/62667901 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcd2019e6b0 0x7fcd2019eb30 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7fcd180348f0 tx=0x7fcd18035d10 comp rx=0 tx=0).stop 2024-04-02T14:49:12.751 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.757+0000 7fcd27c00640 1 -- 172.21.15.90:0/62667901 shutdown_connections 2024-04-02T14:49:12.751 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.757+0000 7fcd27c00640 1 --2- 172.21.15.90:0/62667901 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fcd080a3f70 0x7fcd080a6430 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:12.751 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.757+0000 7fcd27c00640 1 --2- 172.21.15.90:0/62667901 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcd2019e6b0 0x7fcd2019eb30 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:12.751 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.757+0000 7fcd27c00640 1 --2- 172.21.15.90:0/62667901 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcd2010d410 0x7fcd201a4b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:12.751 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.757+0000 7fcd27c00640 1 --2- 172.21.15.90:0/62667901 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcd20108340 0x7fcd201a45f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:12.751 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.757+0000 7fcd27c00640 1 -- 172.21.15.90:0/62667901 >> 172.21.15.90:0/62667901 conn(0x7fcd20076070 msgr2=0x7fcd20076b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:49:12.751 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.758+0000 7fcd27c00640 1 -- 172.21.15.90:0/62667901 shutdown_connections 2024-04-02T14:49:12.751 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:12.758+0000 7fcd27c00640 1 -- 172.21.15.90:0/62667901 wait complete. 2024-04-02T14:49:14.103 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:49:13 smithi092 ceph-mon[36497]: pgmap v448: 97 pgs: 22 active+clean+snaptrim, 38 active+clean+snaptrim_wait, 37 active+clean; 4.4 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 159 KiB/s rd, 15 MiB/s wr, 1.55k op/s 2024-04-02T14:49:14.178 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:49:13 smithi177 ceph-mon[36431]: pgmap v448: 97 pgs: 22 active+clean+snaptrim, 38 active+clean+snaptrim_wait, 37 active+clean; 4.4 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 159 KiB/s rd, 15 MiB/s wr, 1.55k op/s 2024-04-02T14:49:14.180 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:49:13 smithi090 ceph-mon[31674]: pgmap v448: 97 pgs: 22 active+clean+snaptrim, 38 active+clean+snaptrim_wait, 37 active+clean; 4.4 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 159 KiB/s rd, 15 MiB/s wr, 1.55k op/s 2024-04-02T14:49:16.103 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:49:15 smithi092 ceph-mon[36497]: pgmap v449: 97 pgs: 6 active+clean+snaptrim, 15 active+clean+snaptrim_wait, 76 active+clean; 4.2 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 8.3 MiB/s wr, 810 op/s 2024-04-02T14:49:16.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:49:15 smithi177 ceph-mon[36431]: pgmap v449: 97 pgs: 6 active+clean+snaptrim, 15 active+clean+snaptrim_wait, 76 active+clean; 4.2 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 8.3 MiB/s wr, 810 op/s 2024-04-02T14:49:16.180 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:49:15 smithi090 ceph-mon[31674]: pgmap v449: 97 pgs: 6 active+clean+snaptrim, 15 active+clean+snaptrim_wait, 76 active+clean; 4.2 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 8.3 MiB/s wr, 810 op/s 2024-04-02T14:49:16.823 INFO:teuthology.orchestra.run.smithi090.stderr:progress: 825.0 s, 1770.2 tps, lat 69.834 ms stddev 250.748 2024-04-02T14:49:18.102 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:49:17 smithi092 ceph-mon[36497]: pgmap v450: 97 pgs: 97 active+clean; 4.1 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 4.0 MiB/s wr, 407 op/s 2024-04-02T14:49:18.179 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:49:17 smithi177 ceph-mon[36431]: pgmap v450: 97 pgs: 97 active+clean; 4.1 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 4.0 MiB/s wr, 407 op/s 2024-04-02T14:49:18.181 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:49:17 smithi090 ceph-mon[31674]: pgmap v450: 97 pgs: 97 active+clean; 4.1 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 4.0 MiB/s wr, 407 op/s 2024-04-02T14:49:20.102 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:49:19 smithi092 ceph-mon[36497]: pgmap v451: 97 pgs: 97 active+clean; 4.1 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 3.4 MiB/s wr, 341 op/s 2024-04-02T14:49:20.178 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:49:19 smithi177 ceph-mon[36431]: pgmap v451: 97 pgs: 97 active+clean; 4.1 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 3.4 MiB/s wr, 341 op/s 2024-04-02T14:49:20.180 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:49:19 smithi090 ceph-mon[31674]: pgmap v451: 97 pgs: 97 active+clean; 4.1 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 3.4 MiB/s wr, 341 op/s 2024-04-02T14:49:21.428 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:49:21 smithi177 ceph-mon[36431]: osdmap e107: 12 total, 12 up, 12 in 2024-04-02T14:49:21.431 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:49:21 smithi090 ceph-mon[31674]: osdmap e107: 12 total, 12 up, 12 in 2024-04-02T14:49:21.602 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:49:21 smithi092 ceph-mon[36497]: osdmap e107: 12 total, 12 up, 12 in 2024-04-02T14:49:21.823 INFO:teuthology.orchestra.run.smithi090.stderr:progress: 830.0 s, 1439.6 tps, lat 70.102 ms stddev 85.992 2024-04-02T14:49:22.428 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:49:22 smithi177 ceph-mon[36431]: pgmap v453: 97 pgs: 97 active+clean; 4.1 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 5.8 MiB/s wr, 521 op/s 2024-04-02T14:49:22.430 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:49:22 smithi090 ceph-mon[31674]: pgmap v453: 97 pgs: 97 active+clean; 4.1 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 5.8 MiB/s wr, 521 op/s 2024-04-02T14:49:22.603 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:49:22 smithi092 ceph-mon[36497]: pgmap v453: 97 pgs: 97 active+clean; 4.1 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 5.8 MiB/s wr, 521 op/s 2024-04-02T14:49:24.428 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:49:24 smithi177 ceph-mon[36431]: pgmap v454: 97 pgs: 97 active+clean; 4.1 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 5.8 MiB/s wr, 521 op/s 2024-04-02T14:49:24.430 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:49:24 smithi090 ceph-mon[31674]: pgmap v454: 97 pgs: 97 active+clean; 4.1 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 5.8 MiB/s wr, 521 op/s 2024-04-02T14:49:24.602 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:49:24 smithi092 ceph-mon[36497]: pgmap v454: 97 pgs: 97 active+clean; 4.1 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 5.8 MiB/s wr, 521 op/s 2024-04-02T14:49:25.933 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:25.940+0000 7feb0b200640 1 -- 172.21.15.90:0/363935669 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7feb04112970 msgr2=0x7feb04114d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:49:25.933 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:25.940+0000 7feb0b200640 1 --2- 172.21.15.90:0/363935669 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7feb04112970 0x7feb04114d60 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7feafc009c10 tx=0x7feafc02bef0 comp rx=0 tx=0).stop 2024-04-02T14:49:25.933 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:25.940+0000 7feb0b200640 1 -- 172.21.15.90:0/363935669 shutdown_connections 2024-04-02T14:49:25.933 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:25.940+0000 7feb0b200640 1 --2- 172.21.15.90:0/363935669 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7feb04112970 0x7feb04114d60 unknown :-1 s=CLOSED pgs=980 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:25.933 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:25.940+0000 7feb0b200640 1 --2- 172.21.15.90:0/363935669 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7feb04076950 0x7feb04112430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:25.933 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:25.940+0000 7feb0b200640 1 --2- 172.21.15.90:0/363935669 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7feb04076030 0x7feb04076410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:25.934 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:25.940+0000 7feb0b200640 1 -- 172.21.15.90:0/363935669 >> 172.21.15.90:0/363935669 conn(0x7feb040ff7c0 msgr2=0x7feb04101be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:49:25.934 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:25.941+0000 7feb0b200640 1 -- 172.21.15.90:0/363935669 shutdown_connections 2024-04-02T14:49:25.934 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:25.941+0000 7feb0b200640 1 -- 172.21.15.90:0/363935669 wait complete. 2024-04-02T14:49:25.935 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:25.942+0000 7feb0b200640 1 Processor -- start 2024-04-02T14:49:25.936 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:25.943+0000 7feb0b200640 1 -- start start 2024-04-02T14:49:25.938 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:25.943+0000 7feb0b200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7feb04076030 0x7feb041a46a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:25.938 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:25.943+0000 7feb0b200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7feb04076950 0x7feb041a4be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:25.938 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:25.943+0000 7feb0b200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7feb04112970 0x7feb0419e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:25.938 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:25.943+0000 7feb0b200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7feb04118500 con 0x7feb04076950 2024-04-02T14:49:25.938 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:25.943+0000 7feb0b200640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7feb04118380 con 0x7feb04112970 2024-04-02T14:49:25.938 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:25.944+0000 7feb0b200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7feb04118680 con 0x7feb04076030 2024-04-02T14:49:25.938 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:25.944+0000 7feb03e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7feb04076030 0x7feb041a46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:25.938 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:25.944+0000 7feb03e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7feb04076030 0x7feb041a46a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:34966/0 (socket says 172.21.15.90:34966) 2024-04-02T14:49:25.938 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:25.944+0000 7feb03e00640 1 -- 172.21.15.90:0/3505900346 learned_addr learned my addr 172.21.15.90:0/3505900346 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:49:25.939 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:25.944+0000 7feb09200640 1 --2- 172.21.15.90:0/3505900346 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7feb04112970 0x7feb0419e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:25.939 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:25.945+0000 7feb03e00640 1 -- 172.21.15.90:0/3505900346 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7feb04112970 msgr2=0x7feb0419e760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:49:25.939 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:25.945+0000 7feb03e00640 1 --2- 172.21.15.90:0/3505900346 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7feb04112970 0x7feb0419e760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:25.939 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:25.945+0000 7feb03e00640 1 -- 172.21.15.90:0/3505900346 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7feb04076950 msgr2=0x7feb041a4be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:49:25.939 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:25.945+0000 7feb03e00640 1 --2- 172.21.15.90:0/3505900346 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7feb04076950 0x7feb041a4be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:25.939 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:25.945+0000 7feb03e00640 1 -- 172.21.15.90:0/3505900346 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb0419f020 con 0x7feb04076030 2024-04-02T14:49:25.939 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:25.945+0000 7feb03e00640 1 --2- 172.21.15.90:0/3505900346 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7feb04076030 0x7feb041a46a0 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7feaf401e780 tx=0x7feaf401ec40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:49:25.939 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:25.946+0000 7feb00c00640 1 -- 172.21.15.90:0/3505900346 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feaf4028070 con 0x7feb04076030 2024-04-02T14:49:25.940 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:25.947+0000 7feb00c00640 1 -- 172.21.15.90:0/3505900346 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feaf4005500 con 0x7feb04076030 2024-04-02T14:49:25.940 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:25.947+0000 7feb00c00640 1 -- 172.21.15.90:0/3505900346 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feaf4004e40 con 0x7feb04076030 2024-04-02T14:49:25.941 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:25.948+0000 7feb0b200640 1 -- 172.21.15.90:0/3505900346 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feb0419f2b0 con 0x7feb04076030 2024-04-02T14:49:25.941 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:25.948+0000 7feb0b200640 1 -- 172.21.15.90:0/3505900346 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feb0406b520 con 0x7feb04076030 2024-04-02T14:49:25.950 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:25.955+0000 7feb00c00640 1 -- 172.21.15.90:0/3505900346 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7feaf4005000 con 0x7feb04076030 2024-04-02T14:49:25.952 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:25.959+0000 7feb00c00640 1 --2- 172.21.15.90:0/3505900346 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7feae80a3e40 0x7feae80a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:25.953 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:25.960+0000 7feb03400640 1 --2- 172.21.15.90:0/3505900346 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7feae80a3e40 0x7feae80a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:25.955 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:25.962+0000 7feb0dfd6740 1 Processor -- start 2024-04-02T14:49:25.956 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:25.963+0000 7feb00c00640 1 -- 172.21.15.90:0/3505900346 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 8550+0+0 (secure 0 0 0) 0x7feaf40e14b0 con 0x7feb04076030 2024-04-02T14:49:25.956 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:25.963+0000 7feb03400640 1 --2- 172.21.15.90:0/3505900346 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7feae80a3e40 0x7feae80a6300 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7feaf0005fd0 tx=0x7feaf0005f40 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:49:25.958 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:25.965+0000 7feb0dfd6740 1 -- start start 2024-04-02T14:49:25.959 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:25.966+0000 7feb0dfd6740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ecf1540eb0 0x55ecf1541290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:25.959 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:25.966+0000 7feb0dfd6740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ecf15417d0 0x55ecf1549e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:25.959 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:25.966+0000 7feb0dfd6740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ecf154a3b0 0x55ecf154c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:25.959 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:25.966+0000 7feb0dfd6740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55ecf14a5d70 con 0x55ecf15417d0 2024-04-02T14:49:25.959 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:25.966+0000 7feb0dfd6740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55ecf144e2a0 con 0x55ecf154a3b0 2024-04-02T14:49:25.960 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:25.966+0000 7feb0dfd6740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55ecf14a1d20 con 0x55ecf1540eb0 2024-04-02T14:49:25.960 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:25.967+0000 7feb03e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ecf1540eb0 0x55ecf1541290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:25.960 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:25.967+0000 7feb09200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ecf154a3b0 0x55ecf154c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:25.962 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:25.967+0000 7feb09200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ecf154a3b0 0x55ecf154c7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:37440/0 (socket says 172.21.15.90:37440) 2024-04-02T14:49:25.962 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:25.967+0000 7feb09200640 1 -- 172.21.15.90:0/883740463 learned_addr learned my addr 172.21.15.90:0/883740463 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:49:25.962 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:25.967+0000 7feb09200640 1 -- 172.21.15.90:0/883740463 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ecf1540eb0 msgr2=0x55ecf1541290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:49:25.962 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:25.967+0000 7feb09200640 1 --2- 172.21.15.90:0/883740463 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ecf1540eb0 0x55ecf1541290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:25.962 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:25.967+0000 7feb09200640 1 -- 172.21.15.90:0/883740463 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ecf15417d0 msgr2=0x55ecf1549e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:49:25.962 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:25.969+0000 7feb03400640 1 --2- 172.21.15.90:0/883740463 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ecf15417d0 0x55ecf1549e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:25.962 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:25.969+0000 7feb09200640 1 --2- 172.21.15.90:0/883740463 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ecf15417d0 0x55ecf1549e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:25.963 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:25.969+0000 7feb09200640 1 -- 172.21.15.90:0/883740463 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ecf1462840 con 0x55ecf154a3b0 2024-04-02T14:49:25.963 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:25.969+0000 7feb03400640 1 --2- 172.21.15.90:0/883740463 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ecf15417d0 0x55ecf1549e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:49:25.963 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:25.970+0000 7feb09200640 1 --2- 172.21.15.90:0/883740463 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ecf154a3b0 0x55ecf154c7a0 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7feafc033420 tx=0x7feafc033b40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5c744fcbbe75da67 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:49:25.964 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:25.971+0000 7feaf9600640 1 -- 172.21.15.90:0/883740463 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feafc043030 con 0x55ecf154a3b0 2024-04-02T14:49:25.964 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:25.972+0000 7feaf9600640 1 -- 172.21.15.90:0/883740463 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feafc033cb0 con 0x55ecf154a3b0 2024-04-02T14:49:25.964 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:25.972+0000 7feaf9600640 1 -- 172.21.15.90:0/883740463 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feafc02c890 con 0x55ecf154a3b0 2024-04-02T14:49:25.966 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:25.973+0000 7feb0dfd6740 1 -- 172.21.15.90:0/883740463 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ecf154a3b0 msgr2=0x55ecf154c7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:49:25.966 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:25.973+0000 7feb0dfd6740 1 --2- 172.21.15.90:0/883740463 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ecf154a3b0 0x55ecf154c7a0 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7feafc033420 tx=0x7feafc033b40 comp rx=0 tx=0).stop 2024-04-02T14:49:25.967 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:25.974+0000 7feb0dfd6740 1 -- 172.21.15.90:0/883740463 shutdown_connections 2024-04-02T14:49:25.967 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:25.974+0000 7feb0dfd6740 1 --2- 172.21.15.90:0/883740463 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ecf154a3b0 0x55ecf154c7a0 unknown :-1 s=CLOSED pgs=981 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:25.967 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:25.974+0000 7feb0dfd6740 1 --2- 172.21.15.90:0/883740463 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ecf15417d0 0x55ecf1549e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:25.967 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:25.974+0000 7feb0dfd6740 1 --2- 172.21.15.90:0/883740463 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ecf1540eb0 0x55ecf1541290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:25.967 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:25.974+0000 7feb0dfd6740 1 -- 172.21.15.90:0/883740463 >> 172.21.15.90:0/883740463 conn(0x55ecf153b5e0 msgr2=0x55ecf153b9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:49:25.967 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:25.974+0000 7feb0dfd6740 1 -- 172.21.15.90:0/883740463 shutdown_connections 2024-04-02T14:49:25.967 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:25.974+0000 7feb0dfd6740 1 -- 172.21.15.90:0/883740463 wait complete. 2024-04-02T14:49:25.969 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:25.976+0000 7feb0dfd6740 1 -- start start 2024-04-02T14:49:25.969 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:25.976+0000 7feb0dfd6740 1 Processor -- start 2024-04-02T14:49:25.970 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:25.977+0000 7feb0dfd6740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ecf1540eb0 0x55ecf1547470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:25.970 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:25.977+0000 7feb0dfd6740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ecf15417d0 0x55ecf15479b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:25.970 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:25.977+0000 7feb0dfd6740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ecf154a3b0 0x55ecf1547ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:25.971 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:25.978+0000 7feb0dfd6740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55ecf154d140 con 0x55ecf1540eb0 2024-04-02T14:49:25.971 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:25.978+0000 7feb0dfd6740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55ecf1461660 con 0x55ecf15417d0 2024-04-02T14:49:25.972 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:25.978+0000 7feb0dfd6740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55ecf154d2c0 con 0x55ecf154a3b0 2024-04-02T14:49:25.972 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:25.978+0000 7feb09200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ecf154a3b0 0x55ecf1547ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:25.972 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:25.978+0000 7feb09200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ecf154a3b0 0x55ecf1547ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:34980/0 (socket says 172.21.15.90:34980) 2024-04-02T14:49:25.972 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:25.978+0000 7feb09200640 1 -- 172.21.15.90:0/2083583775 learned_addr learned my addr 172.21.15.90:0/2083583775 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:49:25.972 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:25.978+0000 7feb03400640 1 --2- 172.21.15.90:0/2083583775 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ecf15417d0 0x55ecf15479b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:25.972 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:25.979+0000 7feb0dfd6740 10 client.0 _ll_register_callbacks cb 0x55ecf1461050 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:49:25.972 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:25.979+0000 7feb09200640 1 -- 172.21.15.90:0/2083583775 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ecf15417d0 msgr2=0x55ecf15479b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:49:25.972 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:25.979+0000 7feb09200640 1 --2- 172.21.15.90:0/2083583775 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ecf15417d0 0x55ecf15479b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:25.972 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:25.979+0000 7feb09200640 1 -- 172.21.15.90:0/2083583775 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ecf1540eb0 msgr2=0x55ecf1547470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:49:25.972 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:25.979+0000 7feb09200640 1 --2- 172.21.15.90:0/2083583775 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ecf1540eb0 0x55ecf1547470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:25.972 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:25.979+0000 7feb09200640 1 -- 172.21.15.90:0/2083583775 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ecf1470e80 con 0x55ecf154a3b0 2024-04-02T14:49:25.973 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:25.979+0000 7feb09200640 1 --2- 172.21.15.90:0/2083583775 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ecf154a3b0 0x55ecf1547ef0 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7feafc009b30 tx=0x7feafc033f50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1e629df53f57b0f server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:49:25.973 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:25.980+0000 7feaee000640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:49:25.973 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:25.980+0000 7feaee000640 1 -- 172.21.15.90:0/2083583775 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feafc04c070 con 0x55ecf154a3b0 2024-04-02T14:49:25.973 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:25.980+0000 7feaee000640 1 -- 172.21.15.90:0/2083583775 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 8550+0+0 (secure 0 0 0) 0x7feafc033840 con 0x55ecf154a3b0 2024-04-02T14:49:25.973 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:25.980+0000 7feb0dfd6740 1 -- 172.21.15.90:0/2083583775 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7feafc033cb0 con 0x55ecf154a3b0 2024-04-02T14:49:25.974 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:25.981+0000 7feaee000640 1 -- 172.21.15.90:0/2083583775 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7feafc056050 con 0x55ecf154a3b0 2024-04-02T14:49:25.974 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:25.981+0000 7feb0dfd6740 10 client.29752 fetch_fsmap learned FSMap version 27 2024-04-02T14:49:25.974 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:25.981+0000 7feb0dfd6740 1 -- 172.21.15.90:0/2083583775 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55ecf11ba210 con 0x55ecf154a3b0 2024-04-02T14:49:25.976 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:25.983+0000 7feaee000640 1 -- 172.21.15.90:0/2083583775 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feafc033cb0 con 0x55ecf154a3b0 2024-04-02T14:49:25.976 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:25.983+0000 7feaee000640 1 -- 172.21.15.90:0/2083583775 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7feafc051440 con 0x55ecf154a3b0 2024-04-02T14:49:25.976 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:25.983+0000 7feb0dfd6740 10 client.29752 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:49:25.976 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:25.983+0000 7feb0dfd6740 10 client.29752 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-02T14:49:25.977 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:25.983+0000 7feb0dfd6740 20 client.29752 populate_metadata read hostname 'smithi090' 2024-04-02T14:49:25.977 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:25.983+0000 7feb0dfd6740 1 --2- 172.21.15.90:0/2083583775 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55ecf155a540 0x55ecf155c930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:25.977 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:25.984+0000 7feb0dfd6740 4 client.29752 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:49:25.977 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:25.984+0000 7feb0dfd6740 1 -- 172.21.15.90:0/2083583775 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ecf1455cd0 con 0x55ecf155a540 2024-04-02T14:49:25.977 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:25.984+0000 7feb03e00640 1 --2- 172.21.15.90:0/2083583775 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55ecf155a540 0x55ecf155c930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:25.977 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:25.984+0000 7feb03e00640 1 --2- 172.21.15.90:0/2083583775 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55ecf155a540 0x55ecf155c930 crc :-1 s=READY pgs=368 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=705bf88f0ac8c1d5 server_cookie=ea49e1b007dc0f55 in_seq=0 out_seq=0 2024-04-02T14:49:25.977 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:25.984+0000 7feaee000640 10 client.29752 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:49:25.978 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:25.985+0000 7feaee000640 1 -- 172.21.15.90:0/2083583775 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ecf1455cd0 con 0x55ecf155a540 2024-04-02T14:49:25.978 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:25.985+0000 7feaee000640 10 client.29752 handle_command_reply: tid=0 2024-04-02T14:49:25.978 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:25.985+0000 7feb0dfd6740 1 client.29752 shutdown 2024-04-02T14:49:25.978 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:25.986+0000 7feb0dfd6740 1 -- 172.21.15.90:0/2083583775 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ecf154a3b0 msgr2=0x55ecf1547ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:49:25.978 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:25.986+0000 7feb0dfd6740 1 --2- 172.21.15.90:0/2083583775 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ecf154a3b0 0x55ecf1547ef0 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7feafc009b30 tx=0x7feafc033f50 comp rx=0 tx=0).stop 2024-04-02T14:49:25.979 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:25.986+0000 7feb0dfd6740 1 -- 172.21.15.90:0/2083583775 shutdown_connections 2024-04-02T14:49:25.979 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:25.986+0000 7feb0dfd6740 1 --2- 172.21.15.90:0/2083583775 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55ecf155a540 0x55ecf155c930 crc :-1 s=READY pgs=368 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:25.979 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:25.986+0000 7feb0dfd6740 1 --2- 172.21.15.90:0/2083583775 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ecf154a3b0 0x55ecf1547ef0 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:25.979 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:25.986+0000 7feaee000640 0 client.29752 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:49:25.979 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:25.986+0000 7feb0dfd6740 1 --2- 172.21.15.90:0/2083583775 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ecf15417d0 0x55ecf15479b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:25.979 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:25.986+0000 7feb0dfd6740 1 --2- 172.21.15.90:0/2083583775 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ecf1540eb0 0x55ecf1547470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:25.979 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:25.986+0000 7feb0dfd6740 1 -- 172.21.15.90:0/2083583775 >> 172.21.15.90:0/2083583775 conn(0x55ecf153b5e0 msgr2=0x55ecf153d330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:49:25.980 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:25.987+0000 7feb0dfd6740 1 -- 172.21.15.90:0/2083583775 shutdown_connections 2024-04-02T14:49:25.980 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:25.987+0000 7feb0dfd6740 1 -- 172.21.15.90:0/2083583775 wait complete. 2024-04-02T14:49:25.980 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:25.987+0000 7feb0dfd6740 20 client.29752 trim_cache size 0 max 16384 2024-04-02T14:49:26.008 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:26.015+0000 7feb0dfd6740 1 Processor -- start 2024-04-02T14:49:26.008 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:26.015+0000 7feb0dfd6740 1 -- start start 2024-04-02T14:49:26.009 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:26.016+0000 7feb0dfd6740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ecf1540eb0 0x55ecf1549ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:26.009 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:26.016+0000 7feb0dfd6740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ecf15417d0 0x55ecf1556000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:26.009 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:26.016+0000 7feb0dfd6740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ecf154a3b0 0x55ecf1561f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:26.009 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:26.016+0000 7feb0dfd6740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7feb041ab740 con 0x55ecf154a3b0 2024-04-02T14:49:26.009 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:26.016+0000 7feb0dfd6740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7feb041ad280 con 0x55ecf1540eb0 2024-04-02T14:49:26.009 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:26.016+0000 7feb0dfd6740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7feb041abe10 con 0x55ecf15417d0 2024-04-02T14:49:26.009 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:26.016+0000 7feb03400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ecf15417d0 0x55ecf1556000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:26.010 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:26.016+0000 7feb03400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ecf15417d0 0x55ecf1556000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:34982/0 (socket says 172.21.15.90:34982) 2024-04-02T14:49:26.010 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:26.016+0000 7feb03400640 1 -- 172.21.15.90:0/2781289368 learned_addr learned my addr 172.21.15.90:0/2781289368 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:49:26.010 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:26.016+0000 7feb09200640 1 --2- 172.21.15.90:0/2781289368 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ecf154a3b0 0x55ecf1561f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:26.010 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:26.017+0000 7feb03e00640 1 --2- 172.21.15.90:0/2781289368 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ecf1540eb0 0x55ecf1549ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:26.010 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:26.017+0000 7feb09200640 1 -- 172.21.15.90:0/2781289368 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ecf15417d0 msgr2=0x55ecf1556000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:49:26.010 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:26.017+0000 7feb09200640 1 --2- 172.21.15.90:0/2781289368 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ecf15417d0 0x55ecf1556000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:26.011 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:26.017+0000 7feb09200640 1 -- 172.21.15.90:0/2781289368 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ecf1540eb0 msgr2=0x55ecf1549ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:49:26.011 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:26.017+0000 7feb09200640 1 --2- 172.21.15.90:0/2781289368 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ecf1540eb0 0x55ecf1549ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:26.011 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:26.017+0000 7feb09200640 1 -- 172.21.15.90:0/2781289368 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ecf1286620 con 0x55ecf154a3b0 2024-04-02T14:49:26.011 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:26.018+0000 7feb09200640 1 --2- 172.21.15.90:0/2781289368 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ecf154a3b0 0x55ecf1561f30 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7feafc02c430 tx=0x7feafc033b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9207cc883416621 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:49:26.011 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:26.018+0000 7feaf8c00640 1 -- 172.21.15.90:0/2781289368 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feafc04c040 con 0x55ecf154a3b0 2024-04-02T14:49:26.011 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:26.018+0000 7feaf8c00640 1 -- 172.21.15.90:0/2781289368 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feafc051440 con 0x55ecf154a3b0 2024-04-02T14:49:26.011 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:26.018+0000 7feaf8c00640 1 -- 172.21.15.90:0/2781289368 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feafc043040 con 0x55ecf154a3b0 2024-04-02T14:49:26.013 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:26.019+0000 7feb0dfd6740 1 -- 172.21.15.90:0/2781289368 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ecf154a3b0 msgr2=0x55ecf1561f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:49:26.013 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:26.019+0000 7feb0dfd6740 1 --2- 172.21.15.90:0/2781289368 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ecf154a3b0 0x55ecf1561f30 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7feafc02c430 tx=0x7feafc033b40 comp rx=0 tx=0).stop 2024-04-02T14:49:26.013 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:26.020+0000 7feb0dfd6740 1 -- 172.21.15.90:0/2781289368 shutdown_connections 2024-04-02T14:49:26.013 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:26.020+0000 7feb0dfd6740 1 --2- 172.21.15.90:0/2781289368 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ecf154a3b0 0x55ecf1561f30 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:26.013 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:26.020+0000 7feb0dfd6740 1 --2- 172.21.15.90:0/2781289368 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ecf15417d0 0x55ecf1556000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:26.013 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:26.020+0000 7feb0dfd6740 1 --2- 172.21.15.90:0/2781289368 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ecf1540eb0 0x55ecf1549ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:26.013 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:26.020+0000 7feb0dfd6740 1 -- 172.21.15.90:0/2781289368 >> 172.21.15.90:0/2781289368 conn(0x55ecf153b5e0 msgr2=0x55ecf153b9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:49:26.013 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:26.020+0000 7feb0dfd6740 1 -- 172.21.15.90:0/2781289368 shutdown_connections 2024-04-02T14:49:26.013 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:26.020+0000 7feb0dfd6740 1 -- 172.21.15.90:0/2781289368 wait complete. 2024-04-02T14:49:26.014 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:26.021+0000 7feb0dfd6740 1 -- start start 2024-04-02T14:49:26.014 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:26.021+0000 7feb0dfd6740 1 Processor -- start 2024-04-02T14:49:26.016 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:26.021+0000 7feb0dfd6740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ecf1540eb0 0x55ecf1544040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:26.016 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:26.021+0000 7feb0dfd6740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ecf15417d0 0x55ecf1544580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:26.016 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:26.022+0000 7feb0dfd6740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ecf154a3b0 0x55ecf1544ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:26.016 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:26.022+0000 7feb0dfd6740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7feb041acbb0 con 0x55ecf1540eb0 2024-04-02T14:49:26.016 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:26.022+0000 7feb0dfd6740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7feb041ac4e0 con 0x55ecf154a3b0 2024-04-02T14:49:26.016 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:26.022+0000 7feb0dfd6740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7feb0404f860 con 0x55ecf15417d0 2024-04-02T14:49:26.016 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:26.022+0000 7feb0dfd6740 10 client.0 _ll_register_callbacks cb 0x7feafc033e60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:49:26.016 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:26.022+0000 7feb03e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ecf1540eb0 0x55ecf1544040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:26.016 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:26.022+0000 7feb03e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ecf1540eb0 0x55ecf1544040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:44600/0 (socket says 172.21.15.90:44600) 2024-04-02T14:49:26.016 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:26.022+0000 7feb03e00640 1 -- 172.21.15.90:0/3728081181 learned_addr learned my addr 172.21.15.90:0/3728081181 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:49:26.016 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:26.022+0000 7feb03400640 1 --2- 172.21.15.90:0/3728081181 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ecf15417d0 0x55ecf1544580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:26.016 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:26.022+0000 7feb09200640 1 --2- 172.21.15.90:0/3728081181 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ecf154a3b0 0x55ecf1544ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:26.017 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:26.022+0000 7feb03e00640 1 -- 172.21.15.90:0/3728081181 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ecf15417d0 msgr2=0x55ecf1544580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:49:26.017 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:26.022+0000 7feb03e00640 1 --2- 172.21.15.90:0/3728081181 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ecf15417d0 0x55ecf1544580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:26.017 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:26.022+0000 7feb03e00640 1 -- 172.21.15.90:0/3728081181 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ecf154a3b0 msgr2=0x55ecf1544ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:49:26.017 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:26.022+0000 7feb03e00640 1 --2- 172.21.15.90:0/3728081181 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ecf154a3b0 0x55ecf1544ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:26.017 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:26.022+0000 7feb03e00640 1 -- 172.21.15.90:0/3728081181 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ecf15bf7a0 con 0x55ecf1540eb0 2024-04-02T14:49:26.017 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:26.023+0000 7feb03e00640 1 --2- 172.21.15.90:0/3728081181 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ecf1540eb0 0x55ecf1544040 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7feaf40ad3c0 tx=0x7feaf40adc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=334a054bb41c23e1 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:49:26.017 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:26.023+0000 7feaee000640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-02T14:49:26.017 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:26.023+0000 7feaee000640 1 -- 172.21.15.90:0/3728081181 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feaf40b60a0 con 0x55ecf1540eb0 2024-04-02T14:49:26.017 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:26.023+0000 7feaee000640 1 -- 172.21.15.90:0/3728081181 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 8550+0+0 (secure 0 0 0) 0x7feaf40056b0 con 0x55ecf1540eb0 2024-04-02T14:49:26.017 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:26.023+0000 7feb0dfd6740 1 -- 172.21.15.90:0/3728081181 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7feafc051440 con 0x55ecf1540eb0 2024-04-02T14:49:26.018 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:26.023+0000 7feaee000640 1 -- 172.21.15.90:0/3728081181 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7feaf40ab730 con 0x55ecf1540eb0 2024-04-02T14:49:26.018 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:26.023+0000 7feb0dfd6740 10 client.18699 fetch_fsmap learned FSMap version 27 2024-04-02T14:49:26.018 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:26.023+0000 7feb0dfd6740 1 -- 172.21.15.90:0/3728081181 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55ecf15c0a10 con 0x55ecf1540eb0 2024-04-02T14:49:26.018 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:26.023+0000 7feaee000640 1 -- 172.21.15.90:0/3728081181 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feafc051440 con 0x55ecf1540eb0 2024-04-02T14:49:26.018 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:26.024+0000 7feaee000640 1 -- 172.21.15.90:0/3728081181 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7feaf40b7070 con 0x55ecf1540eb0 2024-04-02T14:49:26.018 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:26.024+0000 7feb0dfd6740 10 client.18699 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:49:26.018 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:26.024+0000 7feb0dfd6740 10 client.18699 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-02T14:49:26.019 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:26.024+0000 7feb0dfd6740 20 client.18699 populate_metadata read hostname 'smithi090' 2024-04-02T14:49:26.019 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:26.024+0000 7feb0dfd6740 1 --2- 172.21.15.90:0/3728081181 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55ecf155a540 0x55ecf15c3180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:26.019 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:26.024+0000 7feb0dfd6740 4 client.18699 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-02T14:49:26.019 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:26.024+0000 7feb0dfd6740 1 -- 172.21.15.90:0/3728081181 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55ecf14b32b0 con 0x55ecf155a540 2024-04-02T14:49:26.019 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:26.024+0000 7feb03400640 1 --2- 172.21.15.90:0/3728081181 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55ecf155a540 0x55ecf15c3180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:26.019 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:26.024+0000 7feb03400640 1 --2- 172.21.15.90:0/3728081181 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55ecf155a540 0x55ecf15c3180 crc :-1 s=READY pgs=369 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a485bedbff56758b server_cookie=d60c601d34c64d14 in_seq=0 out_seq=0 2024-04-02T14:49:26.019 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:26.024+0000 7feaee000640 10 client.18699 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:49:26.056 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:26.062+0000 7feaee000640 1 -- 172.21.15.90:0/3728081181 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55ecf14b32b0 con 0x55ecf155a540 2024-04-02T14:49:26.056 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:26.062+0000 7feaee000640 10 client.18699 handle_command_reply: tid=0 2024-04-02T14:49:26.056 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:26.062+0000 7feb0dfd6740 1 client.18699 shutdown 2024-04-02T14:49:26.056 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:26.063+0000 7feb0dfd6740 1 -- 172.21.15.90:0/3728081181 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ecf1540eb0 msgr2=0x55ecf1544040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:49:26.056 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:26.063+0000 7feb0dfd6740 1 --2- 172.21.15.90:0/3728081181 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ecf1540eb0 0x55ecf1544040 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7feaf40ad3c0 tx=0x7feaf40adc70 comp rx=0 tx=0).stop 2024-04-02T14:49:26.056 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:26.063+0000 7feb0dfd6740 1 -- 172.21.15.90:0/3728081181 shutdown_connections 2024-04-02T14:49:26.056 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:26.063+0000 7feb0dfd6740 1 --2- 172.21.15.90:0/3728081181 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55ecf155a540 0x55ecf15c3180 crc :-1 s=READY pgs=369 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:26.056 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:26.063+0000 7feb0dfd6740 1 --2- 172.21.15.90:0/3728081181 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55ecf154a3b0 0x55ecf1544ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:26.056 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:26.063+0000 7feb0dfd6740 1 --2- 172.21.15.90:0/3728081181 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ecf15417d0 0x55ecf1544580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:26.056 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:26.063+0000 7feb0dfd6740 1 --2- 172.21.15.90:0/3728081181 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55ecf1540eb0 0x55ecf1544040 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:26.056 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:26.063+0000 7feb0dfd6740 1 -- 172.21.15.90:0/3728081181 >> 172.21.15.90:0/3728081181 conn(0x55ecf153b5e0 msgr2=0x55ecf1547410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:49:26.057 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:26.063+0000 7feaee000640 0 client.18699 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:49:26.057 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:26.063+0000 7feb0dfd6740 1 -- 172.21.15.90:0/3728081181 shutdown_connections 2024-04-02T14:49:26.057 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:26.063+0000 7feb0dfd6740 1 -- 172.21.15.90:0/3728081181 wait complete. 2024-04-02T14:49:26.057 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:26.063+0000 7feb0dfd6740 20 client.18699 trim_cache size 0 max 16384 2024-04-02T14:49:26.057 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:{ 2024-04-02T14:49:26.057 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "message": "", 2024-04-02T14:49:26.057 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "return_code": 0 2024-04-02T14:49:26.057 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:} 2024-04-02T14:49:26.058 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:26.064+0000 7feb0b200640 1 -- 172.21.15.90:0/3505900346 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7feae80a3e40 msgr2=0x7feae80a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:49:26.058 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:26.064+0000 7feb0b200640 1 --2- 172.21.15.90:0/3505900346 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7feae80a3e40 0x7feae80a6300 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7feaf0005fd0 tx=0x7feaf0005f40 comp rx=0 tx=0).stop 2024-04-02T14:49:26.058 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:26.064+0000 7feb0b200640 1 -- 172.21.15.90:0/3505900346 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7feb04076030 msgr2=0x7feb041a46a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:49:26.058 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:26.064+0000 7feb0b200640 1 --2- 172.21.15.90:0/3505900346 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7feb04076030 0x7feb041a46a0 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7feaf401e780 tx=0x7feaf401ec40 comp rx=0 tx=0).stop 2024-04-02T14:49:26.058 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:26.065+0000 7feb0b200640 1 -- 172.21.15.90:0/3505900346 shutdown_connections 2024-04-02T14:49:26.058 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:26.065+0000 7feb0b200640 1 --2- 172.21.15.90:0/3505900346 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7feae80a3e40 0x7feae80a6300 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:26.058 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:26.065+0000 7feb0b200640 1 --2- 172.21.15.90:0/3505900346 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7feb04112970 0x7feb0419e760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:26.058 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:26.065+0000 7feb0b200640 1 --2- 172.21.15.90:0/3505900346 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7feb04076950 0x7feb041a4be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:26.058 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:26.065+0000 7feb0b200640 1 --2- 172.21.15.90:0/3505900346 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7feb04076030 0x7feb041a46a0 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:26.058 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:26.065+0000 7feb0b200640 1 -- 172.21.15.90:0/3505900346 >> 172.21.15.90:0/3505900346 conn(0x7feb040ff7c0 msgr2=0x7feb04101bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:49:26.058 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:26.065+0000 7feb0b200640 1 -- 172.21.15.90:0/3505900346 shutdown_connections 2024-04-02T14:49:26.058 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:26.065+0000 7feb0b200640 1 -- 172.21.15.90:0/3505900346 wait complete. 2024-04-02T14:49:26.428 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:49:26 smithi177 ceph-mon[36431]: pgmap v455: 97 pgs: 97 active+clean; 4.2 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 5.6 MiB/s wr, 555 op/s 2024-04-02T14:49:26.430 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:49:26 smithi090 ceph-mon[31674]: pgmap v455: 97 pgs: 97 active+clean; 4.2 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 5.6 MiB/s wr, 555 op/s 2024-04-02T14:49:26.602 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:49:26 smithi092 ceph-mon[36497]: pgmap v455: 97 pgs: 97 active+clean; 4.2 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 5.6 MiB/s wr, 555 op/s 2024-04-02T14:49:26.823 INFO:teuthology.orchestra.run.smithi090.stderr:progress: 835.0 s, 1221.4 tps, lat 80.348 ms stddev 91.280 2024-04-02T14:49:28.428 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:49:28 smithi177 ceph-mon[36431]: pgmap v456: 97 pgs: 97 active+clean; 4.2 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 7.1 MiB/s wr, 635 op/s 2024-04-02T14:49:28.430 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:49:28 smithi090 ceph-mon[31674]: pgmap v456: 97 pgs: 97 active+clean; 4.2 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 7.1 MiB/s wr, 635 op/s 2024-04-02T14:49:28.602 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:49:28 smithi092 ceph-mon[36497]: pgmap v456: 97 pgs: 97 active+clean; 4.2 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 7.1 MiB/s wr, 635 op/s 2024-04-02T14:49:30.602 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:49:30 smithi092 ceph-mon[36497]: pgmap v457: 97 pgs: 97 active+clean; 4.2 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 7.1 MiB/s wr, 635 op/s 2024-04-02T14:49:30.603 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:49:30 smithi092 ceph-mon[36497]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T14:49:30.678 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:49:30 smithi177 ceph-mon[36431]: pgmap v457: 97 pgs: 97 active+clean; 4.2 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 7.1 MiB/s wr, 635 op/s 2024-04-02T14:49:30.678 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:49:30 smithi177 ceph-mon[36431]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T14:49:30.680 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:49:30 smithi090 ceph-mon[31674]: pgmap v457: 97 pgs: 97 active+clean; 4.2 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 7.1 MiB/s wr, 635 op/s 2024-04-02T14:49:30.680 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:49:30 smithi090 ceph-mon[31674]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T14:49:31.602 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:49:31 smithi092 ceph-mon[36497]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:49:31.602 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:49:31 smithi092 ceph-mon[36497]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T14:49:31.602 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:49:31 smithi092 ceph-mon[36497]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' 2024-04-02T14:49:31.678 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:49:31 smithi177 ceph-mon[36431]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:49:31.678 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:49:31 smithi177 ceph-mon[36431]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T14:49:31.678 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:49:31 smithi177 ceph-mon[36431]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' 2024-04-02T14:49:31.680 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:49:31 smithi090 ceph-mon[31674]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:49:31.680 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:49:31 smithi090 ceph-mon[31674]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T14:49:31.680 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:49:31 smithi090 ceph-mon[31674]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' 2024-04-02T14:49:31.823 INFO:teuthology.orchestra.run.smithi090.stderr:progress: 840.0 s, 1376.4 tps, lat 73.453 ms stddev 104.065 2024-04-02T14:49:32.455 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-02T14:49:32.456 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-02T14:49:32.541 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:49:32 smithi090 ceph-mon[31674]: pgmap v458: 97 pgs: 97 active+clean; 4.2 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 5.8 MiB/s wr, 575 op/s 2024-04-02T14:49:32.579 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:49:32 smithi177 ceph-mon[36431]: pgmap v458: 97 pgs: 97 active+clean; 4.2 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 5.8 MiB/s wr, 575 op/s 2024-04-02T14:49:32.602 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:49:32 smithi092 ceph-mon[36497]: pgmap v458: 97 pgs: 97 active+clean; 4.2 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 5.8 MiB/s wr, 575 op/s 2024-04-02T14:49:32.694 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.700+0000 7f685be00640 1 -- 172.21.15.90:0/741183105 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6850005220 con 0x7f685c107aa0 2024-04-02T14:49:32.694 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.700+0000 7f6864200640 1 -- 172.21.15.90:0/741183105 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f685c107aa0 msgr2=0x7f685c107e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:49:32.694 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.700+0000 7f6864200640 1 --2- 172.21.15.90:0/741183105 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f685c107aa0 0x7f685c107e80 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f6850009930 tx=0x7f685002bc30 comp rx=0 tx=0).stop 2024-04-02T14:49:32.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.701+0000 7f6864200640 1 -- 172.21.15.90:0/741183105 shutdown_connections 2024-04-02T14:49:32.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.701+0000 7f6864200640 1 --2- 172.21.15.90:0/741183105 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f685c102450 0x7f685c10e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:32.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.701+0000 7f6864200640 1 --2- 172.21.15.90:0/741183105 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f685c101a90 0x7f685c101f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:32.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.701+0000 7f6864200640 1 --2- 172.21.15.90:0/741183105 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f685c107aa0 0x7f685c107e80 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:32.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.701+0000 7f6864200640 1 -- 172.21.15.90:0/741183105 >> 172.21.15.90:0/741183105 conn(0x7f685c078f00 msgr2=0x7f685c079310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:49:32.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.701+0000 7f6864200640 1 -- 172.21.15.90:0/741183105 shutdown_connections 2024-04-02T14:49:32.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.701+0000 7f6864200640 1 -- 172.21.15.90:0/741183105 wait complete. 2024-04-02T14:49:32.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.702+0000 7f6864200640 1 Processor -- start 2024-04-02T14:49:32.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.702+0000 7f6864200640 1 -- start start 2024-04-02T14:49:32.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.703+0000 7f6864200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f685c101a90 0x7f685c19a410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:32.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.703+0000 7f6864200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f685c102450 0x7f685c19a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:32.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.703+0000 7f6864200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f685c107aa0 0x7f685c19ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:32.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.703+0000 7f6864200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f685c111f00 con 0x7f685c101a90 2024-04-02T14:49:32.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.703+0000 7f6864200640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f685c111d80 con 0x7f685c107aa0 2024-04-02T14:49:32.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.703+0000 7f6864200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f685c112080 con 0x7f685c102450 2024-04-02T14:49:32.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.703+0000 7f6860e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f685c102450 0x7f685c19a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:32.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.703+0000 7f6861800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f685c101a90 0x7f685c19a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:32.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.703+0000 7f6860e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f685c102450 0x7f685c19a950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:35014/0 (socket says 172.21.15.90:35014) 2024-04-02T14:49:32.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.703+0000 7f6860e00640 1 -- 172.21.15.90:0/2749993770 learned_addr learned my addr 172.21.15.90:0/2749993770 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:49:32.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.703+0000 7f6862200640 1 --2- 172.21.15.90:0/2749993770 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f685c107aa0 0x7f685c19ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:32.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.704+0000 7f6860e00640 1 -- 172.21.15.90:0/2749993770 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f685c107aa0 msgr2=0x7f685c19ecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:49:32.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.704+0000 7f6860e00640 1 --2- 172.21.15.90:0/2749993770 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f685c107aa0 0x7f685c19ecd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:32.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.704+0000 7f6860e00640 1 -- 172.21.15.90:0/2749993770 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f685c101a90 msgr2=0x7f685c19a410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:49:32.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.704+0000 7f6860e00640 1 --2- 172.21.15.90:0/2749993770 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f685c101a90 0x7f685c19a410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:32.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.704+0000 7f6860e00640 1 -- 172.21.15.90:0/2749993770 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f685c19f400 con 0x7f685c102450 2024-04-02T14:49:32.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.704+0000 7f6860e00640 1 --2- 172.21.15.90:0/2749993770 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f685c102450 0x7f685c19a950 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f684800e9f0 tx=0x7f684800edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:49:32.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.704+0000 7f685a000640 1 -- 172.21.15.90:0/2749993770 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f684800cd80 con 0x7f685c102450 2024-04-02T14:49:32.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.704+0000 7f6864200640 1 -- 172.21.15.90:0/2749993770 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f685c19f6f0 con 0x7f685c102450 2024-04-02T14:49:32.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.704+0000 7f685a000640 1 -- 172.21.15.90:0/2749993770 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6848005500 con 0x7f685c102450 2024-04-02T14:49:32.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.704+0000 7f685a000640 1 -- 172.21.15.90:0/2749993770 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6848010690 con 0x7f685c102450 2024-04-02T14:49:32.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.704+0000 7f6864200640 1 -- 172.21.15.90:0/2749993770 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f685c1a6f80 con 0x7f685c102450 2024-04-02T14:49:32.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.708+0000 7f685a000640 1 -- 172.21.15.90:0/2749993770 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f6848004f60 con 0x7f685c102450 2024-04-02T14:49:32.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.709+0000 7f6866fa6740 1 Processor -- start 2024-04-02T14:49:32.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.709+0000 7f6866fa6740 1 -- start start 2024-04-02T14:49:32.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.713+0000 7f6866fa6740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e319b8b7b0 0x55e319b8bb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:32.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.713+0000 7f6866fa6740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e319b8c0d0 0x55e319b94770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:32.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.713+0000 7f6866fa6740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e319b94cb0 0x55e319b970a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:32.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.713+0000 7f6866fa6740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55e319aef770 con 0x55e319b94cb0 2024-04-02T14:49:32.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.713+0000 7f6866fa6740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55e319a97700 con 0x55e319b8c0d0 2024-04-02T14:49:32.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.713+0000 7f6866fa6740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55e319aeb720 con 0x55e319b8b7b0 2024-04-02T14:49:32.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.713+0000 7f6862200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e319b8b7b0 0x55e319b8bb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:32.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.713+0000 7f6862200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e319b8b7b0 0x55e319b8bb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:35022/0 (socket says 172.21.15.90:35022) 2024-04-02T14:49:32.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.713+0000 7f6862200640 1 -- 172.21.15.90:0/3415256589 learned_addr learned my addr 172.21.15.90:0/3415256589 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:49:32.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.714+0000 7f685a000640 1 --2- 172.21.15.90:0/2749993770 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f68440a3d20 0x7f68440a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:32.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.714+0000 7f685a000640 1 -- 172.21.15.90:0/2749993770 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 8550+0+0 (secure 0 0 0) 0x7f68480d0c60 con 0x7f685c102450 2024-04-02T14:49:32.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.715+0000 7f6860e00640 1 --2- 172.21.15.90:0/3415256589 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e319b94cb0 0x55e319b970a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:32.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.715+0000 7f6861800640 1 --2- 172.21.15.90:0/3415256589 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e319b8c0d0 0x55e319b94770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:32.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.715+0000 7f6861800640 1 -- 172.21.15.90:0/3415256589 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e319b8b7b0 msgr2=0x55e319b8bb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:49:32.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.715+0000 7f6861800640 1 --2- 172.21.15.90:0/3415256589 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e319b8b7b0 0x55e319b8bb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:32.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.715+0000 7f6861800640 1 -- 172.21.15.90:0/3415256589 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e319b94cb0 msgr2=0x55e319b970a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:49:32.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.715+0000 7f6861800640 1 --2- 172.21.15.90:0/3415256589 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e319b94cb0 0x55e319b970a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:32.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.715+0000 7f6861800640 1 -- 172.21.15.90:0/3415256589 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e3198cd7d0 con 0x55e319b8c0d0 2024-04-02T14:49:32.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.716+0000 7f6862200640 1 --2- 172.21.15.90:0/2749993770 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f68440a3d20 0x7f68440a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:32.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.716+0000 7f6861800640 1 --2- 172.21.15.90:0/3415256589 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e319b8c0d0 0x55e319b94770 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7f685002c170 tx=0x7f6850005d50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=231fec087d033b49 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:49:32.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.717+0000 7f6862200640 1 --2- 172.21.15.90:0/2749993770 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f68440a3d20 0x7f68440a61e0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f6854010610 tx=0x7f6854004a40 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:49:32.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.718+0000 7f684ea00640 1 -- 172.21.15.90:0/3415256589 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6850045070 con 0x55e319b8c0d0 2024-04-02T14:49:32.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.718+0000 7f684ea00640 1 -- 172.21.15.90:0/3415256589 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f685002c650 con 0x55e319b8c0d0 2024-04-02T14:49:32.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.718+0000 7f684ea00640 1 -- 172.21.15.90:0/3415256589 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6850005170 con 0x55e319b8c0d0 2024-04-02T14:49:32.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.719+0000 7f6866fa6740 1 -- 172.21.15.90:0/3415256589 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e319b8c0d0 msgr2=0x55e319b94770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:49:32.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.719+0000 7f6866fa6740 1 --2- 172.21.15.90:0/3415256589 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e319b8c0d0 0x55e319b94770 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7f685002c170 tx=0x7f6850005d50 comp rx=0 tx=0).stop 2024-04-02T14:49:32.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.720+0000 7f6866fa6740 1 -- 172.21.15.90:0/3415256589 shutdown_connections 2024-04-02T14:49:32.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.720+0000 7f6866fa6740 1 --2- 172.21.15.90:0/3415256589 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e319b94cb0 0x55e319b970a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:32.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.720+0000 7f6866fa6740 1 --2- 172.21.15.90:0/3415256589 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e319b8c0d0 0x55e319b94770 unknown :-1 s=CLOSED pgs=982 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:32.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.720+0000 7f6866fa6740 1 --2- 172.21.15.90:0/3415256589 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e319b8b7b0 0x55e319b8bb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:32.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.720+0000 7f6866fa6740 1 -- 172.21.15.90:0/3415256589 >> 172.21.15.90:0/3415256589 conn(0x55e319b85ee0 msgr2=0x55e319b862c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:49:32.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.724+0000 7f6866fa6740 1 -- 172.21.15.90:0/3415256589 shutdown_connections 2024-04-02T14:49:32.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.724+0000 7f6866fa6740 1 -- 172.21.15.90:0/3415256589 wait complete. 2024-04-02T14:49:32.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.728+0000 7f6866fa6740 1 -- start start 2024-04-02T14:49:32.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.728+0000 7f6866fa6740 1 Processor -- start 2024-04-02T14:49:32.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.729+0000 7f6866fa6740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e319b8b7b0 0x55e319b91c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:32.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.729+0000 7f6866fa6740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e319b8c0d0 0x55e319b921c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:32.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.729+0000 7f6866fa6740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e319b94cb0 0x55e319b92700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:32.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.729+0000 7f6860e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e319b8c0d0 0x55e319b921c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:32.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.730+0000 7f6861800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e319b8b7b0 0x55e319b91c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:32.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.730+0000 7f6861800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e319b8b7b0 0x55e319b91c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:37504/0 (socket says 172.21.15.90:37504) 2024-04-02T14:49:32.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.730+0000 7f6860e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e319b8c0d0 0x55e319b921c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:35038/0 (socket says 172.21.15.90:35038) 2024-04-02T14:49:32.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.730+0000 7f6866fa6740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55e319b97a90 con 0x55e319b94cb0 2024-04-02T14:49:32.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.730+0000 7f6866fa6740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55e319aaaac0 con 0x55e319b8b7b0 2024-04-02T14:49:32.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.730+0000 7f6866fa6740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55e319b97c10 con 0x55e319b8c0d0 2024-04-02T14:49:32.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.730+0000 7f6866fa6740 10 client.0 _ll_register_callbacks cb 0x55e319aad180 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:49:32.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.730+0000 7f6861800640 1 -- 172.21.15.90:0/3862989215 learned_addr learned my addr 172.21.15.90:0/3862989215 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:49:32.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.731+0000 7f6860e00640 1 -- 172.21.15.90:0/3862989215 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e319b8b7b0 msgr2=0x55e319b91c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:49:32.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.731+0000 7f6860e00640 1 --2- 172.21.15.90:0/3862989215 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e319b8b7b0 0x55e319b91c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:32.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.731+0000 7f6860e00640 1 -- 172.21.15.90:0/3862989215 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e319b94cb0 msgr2=0x55e319b92700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:49:32.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.731+0000 7f6860e00640 1 --2- 172.21.15.90:0/3862989215 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e319b94cb0 0x55e319b92700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:32.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.731+0000 7f6860e00640 1 -- 172.21.15.90:0/3862989215 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e3198033f0 con 0x55e319b8c0d0 2024-04-02T14:49:32.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.731+0000 7f6843400640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:49:32.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.731+0000 7f6860e00640 1 --2- 172.21.15.90:0/3862989215 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e319b8c0d0 0x55e319b921c0 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f68480ce330 tx=0x7f684809d3c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f8611d8cfe3582ab server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:49:32.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.732+0000 7f6843400640 1 -- 172.21.15.90:0/3862989215 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68480bd0c0 con 0x55e319b8c0d0 2024-04-02T14:49:32.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.732+0000 7f6866fa6740 1 -- 172.21.15.90:0/3862989215 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f685002c650 con 0x55e319b8c0d0 2024-04-02T14:49:32.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.732+0000 7f6843400640 1 -- 172.21.15.90:0/3862989215 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 8550+0+0 (secure 0 0 0) 0x7f68480b6060 con 0x55e319b8c0d0 2024-04-02T14:49:32.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.732+0000 7f6843400640 1 -- 172.21.15.90:0/3862989215 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f68480bd0c0 con 0x55e319b8c0d0 2024-04-02T14:49:32.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.732+0000 7f6866fa6740 10 client.29770 fetch_fsmap learned FSMap version 27 2024-04-02T14:49:32.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.732+0000 7f6866fa6740 1 -- 172.21.15.90:0/3862989215 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55e3197c30f0 con 0x55e319b8c0d0 2024-04-02T14:49:32.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.733+0000 7f6843400640 1 -- 172.21.15.90:0/3862989215 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f685002c650 con 0x55e319b8c0d0 2024-04-02T14:49:32.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.733+0000 7f6843400640 1 -- 172.21.15.90:0/3862989215 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f68480a5360 con 0x55e319b8c0d0 2024-04-02T14:49:32.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.733+0000 7f6866fa6740 10 client.29770 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:49:32.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.733+0000 7f6866fa6740 10 client.29770 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:49:32.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.733+0000 7f6866fa6740 20 client.29770 populate_metadata read hostname 'smithi090' 2024-04-02T14:49:32.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.733+0000 7f6866fa6740 1 --2- 172.21.15.90:0/3862989215 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55e319b9bcc0 0x55e319b9e0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:32.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.733+0000 7f6866fa6740 4 client.29770 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:49:32.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.733+0000 7f6866fa6740 1 -- 172.21.15.90:0/3862989215 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e319a90630 con 0x55e319b9bcc0 2024-04-02T14:49:32.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.733+0000 7f6861800640 1 --2- 172.21.15.90:0/3862989215 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55e319b9bcc0 0x55e319b9e0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:32.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.734+0000 7f6861800640 1 --2- 172.21.15.90:0/3862989215 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55e319b9bcc0 0x55e319b9e0f0 crc :-1 s=READY pgs=370 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ae78c146b9d1b208 server_cookie=cb7a9a100e4a19d5 in_seq=0 out_seq=0 2024-04-02T14:49:32.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.734+0000 7f6843400640 10 client.29770 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:49:32.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.735+0000 7f6843400640 1 -- 172.21.15.90:0/3862989215 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e319a90630 con 0x55e319b9bcc0 2024-04-02T14:49:32.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.735+0000 7f6843400640 10 client.29770 handle_command_reply: tid=0 2024-04-02T14:49:32.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.735+0000 7f6866fa6740 1 client.29770 shutdown 2024-04-02T14:49:32.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.735+0000 7f6866fa6740 1 -- 172.21.15.90:0/3862989215 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e319b8c0d0 msgr2=0x55e319b921c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:49:32.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.735+0000 7f6866fa6740 1 --2- 172.21.15.90:0/3862989215 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e319b8c0d0 0x55e319b921c0 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f68480ce330 tx=0x7f684809d3c0 comp rx=0 tx=0).stop 2024-04-02T14:49:32.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.736+0000 7f6866fa6740 1 -- 172.21.15.90:0/3862989215 shutdown_connections 2024-04-02T14:49:32.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.736+0000 7f6866fa6740 1 --2- 172.21.15.90:0/3862989215 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55e319b9bcc0 0x55e319b9e0f0 crc :-1 s=READY pgs=370 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:32.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.736+0000 7f6866fa6740 1 --2- 172.21.15.90:0/3862989215 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e319b94cb0 0x55e319b92700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:32.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.736+0000 7f6866fa6740 1 --2- 172.21.15.90:0/3862989215 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e319b8c0d0 0x55e319b921c0 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:32.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.736+0000 7f6843400640 0 client.29770 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:49:32.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.736+0000 7f6866fa6740 1 --2- 172.21.15.90:0/3862989215 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e319b8b7b0 0x55e319b91c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:32.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.736+0000 7f6866fa6740 1 -- 172.21.15.90:0/3862989215 >> 172.21.15.90:0/3862989215 conn(0x55e319b85ee0 msgr2=0x55e319b87c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:49:32.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.739+0000 7f6866fa6740 1 -- 172.21.15.90:0/3862989215 shutdown_connections 2024-04-02T14:49:32.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.739+0000 7f6866fa6740 1 -- 172.21.15.90:0/3862989215 wait complete. 2024-04-02T14:49:32.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.740+0000 7f6866fa6740 20 client.29770 trim_cache size 0 max 16384 2024-04-02T14:49:32.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.796+0000 7f6866fa6740 1 Processor -- start 2024-04-02T14:49:32.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.796+0000 7f6866fa6740 1 -- start start 2024-04-02T14:49:32.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.796+0000 7f6866fa6740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e319b8b7b0 0x55e319b943b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:32.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.796+0000 7f6866fa6740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e319b8c0d0 0x55e319b90c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:32.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.797+0000 7f6861800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e319b8b7b0 0x55e319b943b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:32.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.797+0000 7f6861800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e319b8b7b0 0x55e319b943b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:35054/0 (socket says 172.21.15.90:35054) 2024-04-02T14:49:32.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.797+0000 7f6866fa6740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e319b94cb0 0x55e319baebf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:32.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.797+0000 7f6860e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e319b8c0d0 0x55e319b90c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:32.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.797+0000 7f6866fa6740 1 -- 172.21.15.90:0/840033134 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f685c1ab6d0 con 0x55e319b8c0d0 2024-04-02T14:49:32.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.797+0000 7f6861800640 1 -- 172.21.15.90:0/840033134 learned_addr learned my addr 172.21.15.90:0/840033134 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:49:32.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.798+0000 7f6866fa6740 1 -- 172.21.15.90:0/840033134 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f685c1ad210 con 0x55e319b94cb0 2024-04-02T14:49:32.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.798+0000 7f6866fa6740 1 -- 172.21.15.90:0/840033134 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f685c1abda0 con 0x55e319b8b7b0 2024-04-02T14:49:32.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.798+0000 7f6862200640 1 --2- 172.21.15.90:0/840033134 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e319b94cb0 0x55e319baebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:32.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.798+0000 7f6861800640 1 -- 172.21.15.90:0/840033134 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e319b94cb0 msgr2=0x55e319baebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:49:32.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.798+0000 7f6861800640 1 --2- 172.21.15.90:0/840033134 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e319b94cb0 0x55e319baebf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:32.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.798+0000 7f6861800640 1 -- 172.21.15.90:0/840033134 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e319b8c0d0 msgr2=0x55e319b90c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:49:32.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.798+0000 7f6861800640 1 --2- 172.21.15.90:0/840033134 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e319b8c0d0 0x55e319b90c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:32.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.798+0000 7f6861800640 1 -- 172.21.15.90:0/840033134 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e319b8b180 con 0x55e319b8b7b0 2024-04-02T14:49:32.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.799+0000 7f6861800640 1 --2- 172.21.15.90:0/840033134 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e319b8b7b0 0x55e319b943b0 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f6850009850 tx=0x7f6850005fd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=76e33558a1fb7746 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:49:32.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.799+0000 7f684e000640 1 -- 172.21.15.90:0/840033134 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6850045040 con 0x55e319b8b7b0 2024-04-02T14:49:32.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.799+0000 7f684e000640 1 -- 172.21.15.90:0/840033134 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f685005f070 con 0x55e319b8b7b0 2024-04-02T14:49:32.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.799+0000 7f684e000640 1 -- 172.21.15.90:0/840033134 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f685005b6c0 con 0x55e319b8b7b0 2024-04-02T14:49:32.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.802+0000 7f6866fa6740 1 -- 172.21.15.90:0/840033134 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e319b8b7b0 msgr2=0x55e319b943b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:49:32.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.802+0000 7f6866fa6740 1 --2- 172.21.15.90:0/840033134 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e319b8b7b0 0x55e319b943b0 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f6850009850 tx=0x7f6850005fd0 comp rx=0 tx=0).stop 2024-04-02T14:49:32.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.802+0000 7f6866fa6740 1 -- 172.21.15.90:0/840033134 shutdown_connections 2024-04-02T14:49:32.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.802+0000 7f6866fa6740 1 --2- 172.21.15.90:0/840033134 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e319b94cb0 0x55e319baebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:32.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.802+0000 7f6866fa6740 1 --2- 172.21.15.90:0/840033134 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e319b8c0d0 0x55e319b90c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:32.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.802+0000 7f6866fa6740 1 --2- 172.21.15.90:0/840033134 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e319b8b7b0 0x55e319b943b0 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:32.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.802+0000 7f6866fa6740 1 -- 172.21.15.90:0/840033134 >> 172.21.15.90:0/840033134 conn(0x55e319b85ee0 msgr2=0x55e319b91c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:49:32.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.803+0000 7f6866fa6740 1 -- 172.21.15.90:0/840033134 shutdown_connections 2024-04-02T14:49:32.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.803+0000 7f6866fa6740 1 -- 172.21.15.90:0/840033134 wait complete. 2024-04-02T14:49:32.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.804+0000 7f6866fa6740 1 -- start start 2024-04-02T14:49:32.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.805+0000 7f6866fa6740 1 Processor -- start 2024-04-02T14:49:32.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.807+0000 7f6866fa6740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e319b8b7b0 0x55e319ba16d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:32.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.807+0000 7f6866fa6740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e319b8c0d0 0x55e319ba1c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:32.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.807+0000 7f6866fa6740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e319b94cb0 0x55e319c0a000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:32.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.807+0000 7f6862200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e319b94cb0 0x55e319c0a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:32.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.808+0000 7f6862200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e319b94cb0 0x55e319c0a000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:37524/0 (socket says 172.21.15.90:37524) 2024-04-02T14:49:32.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.808+0000 7f6862200640 1 -- 172.21.15.90:0/3347744432 learned_addr learned my addr 172.21.15.90:0/3347744432 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:49:32.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.808+0000 7f6866fa6740 1 -- 172.21.15.90:0/3347744432 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f685c1acb40 con 0x55e319b8c0d0 2024-04-02T14:49:32.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.808+0000 7f6866fa6740 1 -- 172.21.15.90:0/3347744432 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f685c1ac470 con 0x55e319b94cb0 2024-04-02T14:49:32.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.808+0000 7f6866fa6740 1 -- 172.21.15.90:0/3347744432 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f685c1a9690 con 0x55e319b8b7b0 2024-04-02T14:49:32.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.808+0000 7f6866fa6740 10 client.0 _ll_register_callbacks cb 0x7f684809c9e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:49:32.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.808+0000 7f6861800640 1 --2- 172.21.15.90:0/3347744432 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e319b8b7b0 0x55e319ba16d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:32.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.809+0000 7f6862200640 1 -- 172.21.15.90:0/3347744432 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e319b8b7b0 msgr2=0x55e319ba16d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:49:32.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.809+0000 7f6862200640 1 --2- 172.21.15.90:0/3347744432 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e319b8b7b0 0x55e319ba16d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:32.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.809+0000 7f6862200640 1 -- 172.21.15.90:0/3347744432 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e319b8c0d0 msgr2=0x55e319ba1c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:49:32.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.809+0000 7f6860e00640 1 --2- 172.21.15.90:0/3347744432 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e319b8c0d0 0x55e319ba1c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:32.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.809+0000 7f6862200640 1 --2- 172.21.15.90:0/3347744432 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e319b8c0d0 0x55e319ba1c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:32.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.809+0000 7f6862200640 1 -- 172.21.15.90:0/3347744432 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e319c0a540 con 0x55e319b94cb0 2024-04-02T14:49:32.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.810+0000 7f6862200640 1 --2- 172.21.15.90:0/3347744432 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e319b94cb0 0x55e319c0a000 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7f685400fbc0 tx=0x7f6854009500 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6d7b41a9383c2e65 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:49:32.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.810+0000 7f6843400640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:49:32.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.811+0000 7f6843400640 1 -- 172.21.15.90:0/3347744432 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f685402d060 con 0x55e319b94cb0 2024-04-02T14:49:32.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.811+0000 7f6843400640 1 -- 172.21.15.90:0/3347744432 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 8550+0+0 (secure 0 0 0) 0x7f685401b420 con 0x55e319b94cb0 2024-04-02T14:49:32.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.811+0000 7f6866fa6740 1 -- 172.21.15.90:0/3347744432 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f685005f070 con 0x55e319b94cb0 2024-04-02T14:49:32.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.812+0000 7f6843400640 1 -- 172.21.15.90:0/3347744432 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f685401b620 con 0x55e319b94cb0 2024-04-02T14:49:32.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.812+0000 7f6866fa6740 10 client.29981 fetch_fsmap learned FSMap version 27 2024-04-02T14:49:32.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.812+0000 7f6866fa6740 1 -- 172.21.15.90:0/3347744432 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55e319c0b750 con 0x55e319b94cb0 2024-04-02T14:49:32.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.812+0000 7f6843400640 1 -- 172.21.15.90:0/3347744432 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f685005f070 con 0x55e319b94cb0 2024-04-02T14:49:32.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.813+0000 7f6843400640 1 -- 172.21.15.90:0/3347744432 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f6854027030 con 0x55e319b94cb0 2024-04-02T14:49:32.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.815+0000 7f6866fa6740 10 client.29981 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:49:32.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.815+0000 7f6866fa6740 10 client.29981 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:49:32.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.815+0000 7f6866fa6740 20 client.29981 populate_metadata read hostname 'smithi090' 2024-04-02T14:49:32.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.815+0000 7f6866fa6740 1 --2- 172.21.15.90:0/3347744432 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55e319c0be00 0x55e319c0e220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:32.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.815+0000 7f6866fa6740 4 client.29981 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-02T14:49:32.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.815+0000 7f6866fa6740 1 -- 172.21.15.90:0/3347744432 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55e319afccb0 con 0x55e319c0be00 2024-04-02T14:49:32.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.816+0000 7f6861800640 1 --2- 172.21.15.90:0/3347744432 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55e319c0be00 0x55e319c0e220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:32.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.816+0000 7f6861800640 1 --2- 172.21.15.90:0/3347744432 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55e319c0be00 0x55e319c0e220 crc :-1 s=READY pgs=371 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c9ac4ece7825d28c server_cookie=7935a7687e2ea248 in_seq=0 out_seq=0 2024-04-02T14:49:32.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.817+0000 7f6843400640 10 client.29981 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:49:32.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.817+0000 7f6843400640 1 -- 172.21.15.90:0/3347744432 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55e319afccb0 con 0x55e319c0be00 2024-04-02T14:49:32.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.817+0000 7f6843400640 10 client.29981 handle_command_reply: tid=0 2024-04-02T14:49:32.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.817+0000 7f6866fa6740 1 client.29981 shutdown 2024-04-02T14:49:32.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.817+0000 7f6866fa6740 1 -- 172.21.15.90:0/3347744432 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e319b94cb0 msgr2=0x55e319c0a000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:49:32.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.817+0000 7f6866fa6740 1 --2- 172.21.15.90:0/3347744432 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e319b94cb0 0x55e319c0a000 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7f685400fbc0 tx=0x7f6854009500 comp rx=0 tx=0).stop 2024-04-02T14:49:32.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.817+0000 7f6866fa6740 1 -- 172.21.15.90:0/3347744432 shutdown_connections 2024-04-02T14:49:32.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.817+0000 7f6866fa6740 1 --2- 172.21.15.90:0/3347744432 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55e319c0be00 0x55e319c0e220 crc :-1 s=READY pgs=371 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:32.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.817+0000 7f6866fa6740 1 --2- 172.21.15.90:0/3347744432 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e319b94cb0 0x55e319c0a000 unknown :-1 s=CLOSED pgs=983 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:32.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.817+0000 7f6866fa6740 1 --2- 172.21.15.90:0/3347744432 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e319b8c0d0 0x55e319ba1c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:32.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.817+0000 7f6866fa6740 1 --2- 172.21.15.90:0/3347744432 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e319b8b7b0 0x55e319ba16d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:32.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.817+0000 7f6866fa6740 1 -- 172.21.15.90:0/3347744432 >> 172.21.15.90:0/3347744432 conn(0x55e319b85ee0 msgr2=0x55e319baea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:49:32.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.818+0000 7f6843400640 0 client.29981 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:49:32.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.818+0000 7f6866fa6740 1 -- 172.21.15.90:0/3347744432 shutdown_connections 2024-04-02T14:49:32.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.818+0000 7f6866fa6740 1 -- 172.21.15.90:0/3347744432 wait complete. 2024-04-02T14:49:32.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.818+0000 7f6866fa6740 20 client.29981 trim_cache size 0 max 16384 2024-04-02T14:49:32.813 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-02T14:49:32.814 INFO:teuthology.orchestra.run.smithi090.stdout: "status": "no active scrubs running", 2024-04-02T14:49:32.814 INFO:teuthology.orchestra.run.smithi090.stdout: "scrubs": {} 2024-04-02T14:49:32.814 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-02T14:49:32.815 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.819+0000 7f6864200640 1 -- 172.21.15.90:0/2749993770 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f68440a3d20 msgr2=0x7f68440a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:49:32.815 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.819+0000 7f6864200640 1 --2- 172.21.15.90:0/2749993770 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f68440a3d20 0x7f68440a61e0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f6854010610 tx=0x7f6854004a40 comp rx=0 tx=0).stop 2024-04-02T14:49:32.815 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.819+0000 7f6864200640 1 -- 172.21.15.90:0/2749993770 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f685c102450 msgr2=0x7f685c19a950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:49:32.815 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.819+0000 7f6864200640 1 --2- 172.21.15.90:0/2749993770 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f685c102450 0x7f685c19a950 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f684800e9f0 tx=0x7f684800edf0 comp rx=0 tx=0).stop 2024-04-02T14:49:32.815 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.819+0000 7f6864200640 1 -- 172.21.15.90:0/2749993770 shutdown_connections 2024-04-02T14:49:32.815 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.819+0000 7f6864200640 1 --2- 172.21.15.90:0/2749993770 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f68440a3d20 0x7f68440a61e0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:32.815 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.819+0000 7f6864200640 1 --2- 172.21.15.90:0/2749993770 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f685c107aa0 0x7f685c19ecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:32.815 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.819+0000 7f6864200640 1 --2- 172.21.15.90:0/2749993770 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f685c102450 0x7f685c19a950 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:32.815 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.819+0000 7f6864200640 1 --2- 172.21.15.90:0/2749993770 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f685c101a90 0x7f685c19a410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:32.815 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.819+0000 7f6864200640 1 -- 172.21.15.90:0/2749993770 >> 172.21.15.90:0/2749993770 conn(0x7f685c078f00 msgr2=0x7f685c10d200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:49:32.815 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.819+0000 7f6864200640 1 -- 172.21.15.90:0/2749993770 shutdown_connections 2024-04-02T14:49:32.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:32.820+0000 7f6864200640 1 -- 172.21.15.90:0/2749993770 wait complete. 2024-04-02T14:49:32.838 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-02T14:49:32.838 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T14:49:33.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.085+0000 7fbf75200640 1 -- 172.21.15.90:0/208263764 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbf70101200 msgr2=0x7fbf70107940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:49:33.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.085+0000 7fbf75200640 1 --2- 172.21.15.90:0/208263764 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbf70101200 0x7fbf70107940 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7fbf5c009900 tx=0x7fbf5c02bc50 comp rx=0 tx=0).stop 2024-04-02T14:49:33.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.085+0000 7fbf75200640 1 -- 172.21.15.90:0/208263764 shutdown_connections 2024-04-02T14:49:33.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.085+0000 7fbf75200640 1 --2- 172.21.15.90:0/208263764 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbf70108020 0x7fbf7010e9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:33.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.085+0000 7fbf75200640 1 --2- 172.21.15.90:0/208263764 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbf70101200 0x7fbf70107940 unknown :-1 s=CLOSED pgs=984 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:33.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.085+0000 7fbf75200640 1 --2- 172.21.15.90:0/208263764 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbf70100850 0x7fbf70100c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:33.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.085+0000 7fbf75200640 1 -- 172.21.15.90:0/208263764 >> 172.21.15.90:0/208263764 conn(0x7fbf70075d20 msgr2=0x7fbf70076130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:49:33.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.085+0000 7fbf75200640 1 -- 172.21.15.90:0/208263764 shutdown_connections 2024-04-02T14:49:33.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.085+0000 7fbf75200640 1 -- 172.21.15.90:0/208263764 wait complete. 2024-04-02T14:49:33.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.086+0000 7fbf75200640 1 Processor -- start 2024-04-02T14:49:33.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.086+0000 7fbf75200640 1 -- start start 2024-04-02T14:49:33.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.086+0000 7fbf75200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbf70100850 0x7fbf701017c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:33.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.086+0000 7fbf75200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbf70108020 0x7fbf70101d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:33.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.087+0000 7fbf75200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbf70102240 0x7fbf701a65f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:33.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.087+0000 7fbf75200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fbf70111e80 con 0x7fbf70108020 2024-04-02T14:49:33.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.087+0000 7fbf75200640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fbf70111d00 con 0x7fbf70102240 2024-04-02T14:49:33.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.087+0000 7fbf75200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fbf70112000 con 0x7fbf70100850 2024-04-02T14:49:33.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.087+0000 7fbf6e800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbf70102240 0x7fbf701a65f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:33.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.087+0000 7fbf6de00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbf70100850 0x7fbf701017c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:33.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.087+0000 7fbf6e800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbf70102240 0x7fbf701a65f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:37550/0 (socket says 172.21.15.90:37550) 2024-04-02T14:49:33.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.087+0000 7fbf6e800640 1 -- 172.21.15.90:0/1595934776 learned_addr learned my addr 172.21.15.90:0/1595934776 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:49:33.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.087+0000 7fbf6d400640 1 --2- 172.21.15.90:0/1595934776 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbf70108020 0x7fbf70101d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:33.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.088+0000 7fbf6e800640 1 -- 172.21.15.90:0/1595934776 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbf70100850 msgr2=0x7fbf701017c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:49:33.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.088+0000 7fbf6e800640 1 --2- 172.21.15.90:0/1595934776 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbf70100850 0x7fbf701017c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:33.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.088+0000 7fbf6e800640 1 -- 172.21.15.90:0/1595934776 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbf70108020 msgr2=0x7fbf70101d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:49:33.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.088+0000 7fbf6e800640 1 --2- 172.21.15.90:0/1595934776 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbf70108020 0x7fbf70101d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:33.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.088+0000 7fbf6e800640 1 -- 172.21.15.90:0/1595934776 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf701a6cf0 con 0x7fbf70102240 2024-04-02T14:49:33.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.088+0000 7fbf6d400640 1 --2- 172.21.15.90:0/1595934776 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbf70108020 0x7fbf70101d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:49:33.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.088+0000 7fbf6de00640 1 --2- 172.21.15.90:0/1595934776 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbf70100850 0x7fbf701017c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:49:33.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.088+0000 7fbf6e800640 1 --2- 172.21.15.90:0/1595934776 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbf70102240 0x7fbf701a65f0 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7fbf6801fb20 tx=0x7fbf6800c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:49:33.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.089+0000 7fbf66a00640 1 -- 172.21.15.90:0/1595934776 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf68004b20 con 0x7fbf70102240 2024-04-02T14:49:33.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.089+0000 7fbf66a00640 1 -- 172.21.15.90:0/1595934776 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbf68004cc0 con 0x7fbf70102240 2024-04-02T14:49:33.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.089+0000 7fbf66a00640 1 -- 172.21.15.90:0/1595934776 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf68021840 con 0x7fbf70102240 2024-04-02T14:49:33.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.090+0000 7fbf75200640 1 -- 172.21.15.90:0/1595934776 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbf7006b580 con 0x7fbf70102240 2024-04-02T14:49:33.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.090+0000 7fbf75200640 1 -- 172.21.15.90:0/1595934776 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbf7006ba50 con 0x7fbf70102240 2024-04-02T14:49:33.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.091+0000 7fbf75200640 1 -- 172.21.15.90:0/1595934776 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbf70068060 con 0x7fbf70102240 2024-04-02T14:49:33.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.093+0000 7fbf66a00640 1 -- 172.21.15.90:0/1595934776 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fbf68005020 con 0x7fbf70102240 2024-04-02T14:49:33.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.096+0000 7fbf66a00640 1 --2- 172.21.15.90:0/1595934776 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fbf540a3ea0 0x7fbf540a6360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:33.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.096+0000 7fbf66a00640 1 -- 172.21.15.90:0/1595934776 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 8550+0+0 (secure 0 0 0) 0x7fbf680e1d60 con 0x7fbf70102240 2024-04-02T14:49:33.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.096+0000 7fbf6de00640 1 --2- 172.21.15.90:0/1595934776 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fbf540a3ea0 0x7fbf540a6360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:33.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.097+0000 7fbf6de00640 1 --2- 172.21.15.90:0/1595934776 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fbf540a3ea0 0x7fbf540a6360 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fbf60009fc0 tx=0x7fbf60005f90 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:49:33.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.099+0000 7fbf66a00640 1 -- 172.21.15.90:0/1595934776 <== mon.2 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7fbf680ae380 con 0x7fbf70102240 2024-04-02T14:49:33.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.496+0000 7fbf75200640 1 -- 172.21.15.90:0/1595934776 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbf70109fb0 con 0x7fbf70102240 2024-04-02T14:49:33.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.497+0000 7fbf66a00640 1 -- 172.21.15.90:0/1595934776 <== mon.2 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11927 (secure 0 0 0) 0x7fbf680adc80 con 0x7fbf70102240 2024-04-02T14:49:33.491 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-02T14:49:33.491 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14412,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6837/2852160309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6836","nonce":2852160309},{"type":"v1","addr":"172.21.15.90:6837","nonce":2852160309}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14478,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6841/1919376858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6840","nonce":1919376858},{"type":"v1","addr":"172.21.15.90:6841","nonce":1919376858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-02T14:32:22.319083+0000","modified":"2024-04-02T14:34:52.266152+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24590,"mds_1":24560,"mds_2":24547,"mds_3":24539,"mds_4":24523},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24454":{"gid":24454,"name":"l","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6833/1564319431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":1564319431},{"type":"v1","addr":"172.21.15.177:6833","nonce":1564319431}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.92:6833/1570337372","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":1570337372},{"type":"v1","addr":"172.21.15.92:6833","nonce":1570337372}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"j","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.90:6835/1277124891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6834","nonce":1277124891},{"type":"v1","addr":"172.21.15.90:6835","nonce":1277124891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.92:6835/860721164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":860721164},{"type":"v1","addr":"172.21.15.92:6835","nonce":860721164}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24511":{"gid":24511,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6835/3177244988","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":3177244988},{"type":"v1","addr":"172.21.15.177:6835","nonce":3177244988}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"c","rank":4,"incarnation":25,"state":"up:active","state_seq":9,"addr":"172.21.15.177:6837/4223414858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":4223414858},{"type":"v1","addr":"172.21.15.177:6837","nonce":4223414858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24539":{"gid":24539,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":8,"addr":"172.21.15.92:6837/1270635939","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":1270635939},{"type":"v1","addr":"172.21.15.92:6837","nonce":1270635939}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24547":{"gid":24547,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":7,"addr":"172.21.15.177:6839/664895663","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":664895663},{"type":"v1","addr":"172.21.15.177:6839","nonce":664895663}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"g","rank":1,"incarnation":19,"state":"up:active","state_seq":5,"addr":"172.21.15.90:6839/1498915870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6838","nonce":1498915870},{"type":"v1","addr":"172.21.15.90:6839","nonce":1498915870}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24590":{"gid":24590,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.92:6839/1054888219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":1054888219},{"type":"v1","addr":"172.21.15.92:6839","nonce":1054888219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24590,"qdb_cluster":[24547,24590,24523,24539,24560]},"id":1}]} 2024-04-02T14:49:33.492 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 27 2024-04-02T14:49:33.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.500+0000 7fbf75200640 1 -- 172.21.15.90:0/1595934776 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fbf540a3ea0 msgr2=0x7fbf540a6360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:49:33.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.500+0000 7fbf75200640 1 --2- 172.21.15.90:0/1595934776 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fbf540a3ea0 0x7fbf540a6360 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fbf60009fc0 tx=0x7fbf60005f90 comp rx=0 tx=0).stop 2024-04-02T14:49:33.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.500+0000 7fbf75200640 1 -- 172.21.15.90:0/1595934776 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbf70102240 msgr2=0x7fbf701a65f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:49:33.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.500+0000 7fbf75200640 1 --2- 172.21.15.90:0/1595934776 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbf70102240 0x7fbf701a65f0 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7fbf6801fb20 tx=0x7fbf6800c2d0 comp rx=0 tx=0).stop 2024-04-02T14:49:33.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.500+0000 7fbf75200640 1 -- 172.21.15.90:0/1595934776 shutdown_connections 2024-04-02T14:49:33.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.500+0000 7fbf75200640 1 --2- 172.21.15.90:0/1595934776 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fbf540a3ea0 0x7fbf540a6360 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:33.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.500+0000 7fbf75200640 1 --2- 172.21.15.90:0/1595934776 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbf70102240 0x7fbf701a65f0 unknown :-1 s=CLOSED pgs=985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:33.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.500+0000 7fbf75200640 1 --2- 172.21.15.90:0/1595934776 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbf70108020 0x7fbf70101d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:33.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.500+0000 7fbf75200640 1 --2- 172.21.15.90:0/1595934776 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbf70100850 0x7fbf701017c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:33.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.500+0000 7fbf75200640 1 -- 172.21.15.90:0/1595934776 >> 172.21.15.90:0/1595934776 conn(0x7fbf70075d20 msgr2=0x7fbf70105c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:49:33.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.500+0000 7fbf75200640 1 -- 172.21.15.90:0/1595934776 shutdown_connections 2024-04-02T14:49:33.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.500+0000 7fbf75200640 1 -- 172.21.15.90:0/1595934776 wait complete. 2024-04-02T14:49:33.518 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-04-02T14:49:33.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.786+0000 7ff615800640 1 -- 172.21.15.90:0/867855579 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff6101048c0 msgr2=0x7ff610108900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:49:33.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.786+0000 7ff615800640 1 --2- 172.21.15.90:0/867855579 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff6101048c0 0x7ff610108900 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7ff5f8009900 tx=0x7ff5f802bc50 comp rx=0 tx=0).stop 2024-04-02T14:49:33.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.786+0000 7ff615800640 1 -- 172.21.15.90:0/867855579 shutdown_connections 2024-04-02T14:49:33.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.786+0000 7ff615800640 1 --2- 172.21.15.90:0/867855579 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff610108fe0 0x7ff610110bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:33.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.786+0000 7ff615800640 1 --2- 172.21.15.90:0/867855579 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff6101048c0 0x7ff610108900 unknown :-1 s=CLOSED pgs=986 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:33.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.786+0000 7ff615800640 1 --2- 172.21.15.90:0/867855579 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff610103f10 0x7ff6101042f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:33.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.786+0000 7ff615800640 1 -- 172.21.15.90:0/867855579 >> 172.21.15.90:0/867855579 conn(0x7ff6100ff7b0 msgr2=0x7ff610101bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:49:33.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.786+0000 7ff615800640 1 -- 172.21.15.90:0/867855579 shutdown_connections 2024-04-02T14:49:33.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.786+0000 7ff615800640 1 -- 172.21.15.90:0/867855579 wait complete. 2024-04-02T14:49:33.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.787+0000 7ff615800640 1 Processor -- start 2024-04-02T14:49:33.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.789+0000 7ff615800640 1 -- start start 2024-04-02T14:49:33.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.789+0000 7ff615800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff610103f10 0x7ff61019a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:33.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.789+0000 7ff615800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff610108fe0 0x7ff61019abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:33.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.789+0000 7ff615800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff61019ef60 0x7ff61019f410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:33.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.789+0000 7ff615800640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7ff610114140 con 0x7ff61019ef60 2024-04-02T14:49:33.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.789+0000 7ff615800640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7ff610113fc0 con 0x7ff610108fe0 2024-04-02T14:49:33.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.789+0000 7ff615800640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7ff6101142c0 con 0x7ff610103f10 2024-04-02T14:49:33.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.789+0000 7ff60ea00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff610103f10 0x7ff61019a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:33.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.790+0000 7ff60e000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff610108fe0 0x7ff61019abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:33.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.790+0000 7ff60ea00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff610103f10 0x7ff61019a6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:35098/0 (socket says 172.21.15.90:35098) 2024-04-02T14:49:33.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.790+0000 7ff60ea00640 1 -- 172.21.15.90:0/1837000856 learned_addr learned my addr 172.21.15.90:0/1837000856 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:49:33.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.790+0000 7ff60e000640 1 -- 172.21.15.90:0/1837000856 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff610103f10 msgr2=0x7ff61019a6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:49:33.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.790+0000 7ff60e000640 1 --2- 172.21.15.90:0/1837000856 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff610103f10 0x7ff61019a6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:33.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.790+0000 7ff60e000640 1 -- 172.21.15.90:0/1837000856 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff61019ef60 msgr2=0x7ff61019f410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:49:33.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.790+0000 7ff60e000640 1 --2- 172.21.15.90:0/1837000856 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff61019ef60 0x7ff61019f410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:33.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.790+0000 7ff60e000640 1 -- 172.21.15.90:0/1837000856 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff61019fa50 con 0x7ff610108fe0 2024-04-02T14:49:33.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.791+0000 7ff60e000640 1 --2- 172.21.15.90:0/1837000856 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff610108fe0 0x7ff61019abe0 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7ff5f802c190 tx=0x7ff5f80042f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:49:33.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.793+0000 7ff607400640 1 -- 172.21.15.90:0/1837000856 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5f8041070 con 0x7ff610108fe0 2024-04-02T14:49:33.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.793+0000 7ff607400640 1 -- 172.21.15.90:0/1837000856 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff5f802ccb0 con 0x7ff610108fe0 2024-04-02T14:49:33.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.793+0000 7ff607400640 1 -- 172.21.15.90:0/1837000856 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5f8003ca0 con 0x7ff610108fe0 2024-04-02T14:49:33.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.793+0000 7ff615800640 1 -- 172.21.15.90:0/1837000856 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff61019fce0 con 0x7ff610108fe0 2024-04-02T14:49:33.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.793+0000 7ff615800640 1 -- 172.21.15.90:0/1837000856 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6101a74b0 con 0x7ff610108fe0 2024-04-02T14:49:33.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.800+0000 7ff6185f7740 1 Processor -- start 2024-04-02T14:49:33.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.800+0000 7ff607400640 1 -- 172.21.15.90:0/1837000856 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7ff5f8044070 con 0x7ff610108fe0 2024-04-02T14:49:33.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.800+0000 7ff6185f7740 1 -- start start 2024-04-02T14:49:33.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.800+0000 7ff6185f7740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5602039819f0 0x560203980740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:33.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.800+0000 7ff6185f7740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560203980d10 0x560203981190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:33.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.800+0000 7ff6185f7740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56020398a080 0x56020398c470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:33.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.800+0000 7ff6185f7740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5602038c1400 con 0x56020398a080 2024-04-02T14:49:33.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.800+0000 7ff6185f7740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x5602038a5f10 con 0x560203980d10 2024-04-02T14:49:33.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.800+0000 7ff6185f7740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5602038e56f0 con 0x5602039819f0 2024-04-02T14:49:33.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.801+0000 7ff60f400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5602039819f0 0x560203980740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:33.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.801+0000 7ff60f400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5602039819f0 0x560203980740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:35100/0 (socket says 172.21.15.90:35100) 2024-04-02T14:49:33.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.801+0000 7ff60f400640 1 -- 172.21.15.90:0/1616764659 learned_addr learned my addr 172.21.15.90:0/1616764659 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:49:33.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.801+0000 7ff60f400640 1 -- 172.21.15.90:0/1616764659 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560203980d10 msgr2=0x560203981190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:49:33.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.801+0000 7ff60f400640 1 --2- 172.21.15.90:0/1616764659 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560203980d10 0x560203981190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:33.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.801+0000 7ff60f400640 1 -- 172.21.15.90:0/1616764659 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56020398a080 msgr2=0x56020398c470 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-02T14:49:33.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.801+0000 7ff60e000640 1 --2- 172.21.15.90:0/1616764659 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56020398a080 0x56020398c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:33.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.802+0000 7ff60f400640 1 --2- 172.21.15.90:0/1616764659 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56020398a080 0x56020398c470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:33.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.802+0000 7ff60f400640 1 -- 172.21.15.90:0/1616764659 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5602035f93f0 con 0x5602039819f0 2024-04-02T14:49:33.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.802+0000 7ff60e000640 1 --2- 172.21.15.90:0/1616764659 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56020398a080 0x56020398c470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:49:33.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.802+0000 7ff607400640 1 --2- 172.21.15.90:0/1837000856 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7ff5f40b4db0 0x7ff5f40b7270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:33.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.803+0000 7ff60f400640 1 --2- 172.21.15.90:0/1616764659 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5602039819f0 0x560203980740 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7ff608009d00 tx=0x7ff60800b420 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eef9a99d55fa674b server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:49:33.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.803+0000 7ff607400640 1 -- 172.21.15.90:0/1837000856 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 8550+0+0 (secure 0 0 0) 0x7ff5f80f2010 con 0x7ff610108fe0 2024-04-02T14:49:33.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.804+0000 7ff5ffe00640 1 -- 172.21.15.90:0/1616764659 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff60800bb30 con 0x5602039819f0 2024-04-02T14:49:33.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.804+0000 7ff5ffe00640 1 -- 172.21.15.90:0/1616764659 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff608017070 con 0x5602039819f0 2024-04-02T14:49:33.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.804+0000 7ff5ffe00640 1 -- 172.21.15.90:0/1616764659 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6080115f0 con 0x5602039819f0 2024-04-02T14:49:33.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.805+0000 7ff6185f7740 1 -- 172.21.15.90:0/1616764659 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5602039819f0 msgr2=0x560203980740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:49:33.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.805+0000 7ff6185f7740 1 --2- 172.21.15.90:0/1616764659 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5602039819f0 0x560203980740 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7ff608009d00 tx=0x7ff60800b420 comp rx=0 tx=0).stop 2024-04-02T14:49:33.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.806+0000 7ff60ea00640 1 --2- 172.21.15.90:0/1837000856 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7ff5f40b4db0 0x7ff5f40b7270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:33.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.807+0000 7ff6185f7740 1 -- 172.21.15.90:0/1616764659 shutdown_connections 2024-04-02T14:49:33.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.807+0000 7ff6185f7740 1 --2- 172.21.15.90:0/1616764659 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56020398a080 0x56020398c470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:33.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.807+0000 7ff6185f7740 1 --2- 172.21.15.90:0/1616764659 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560203980d10 0x560203981190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:33.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.807+0000 7ff6185f7740 1 --2- 172.21.15.90:0/1616764659 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5602039819f0 0x560203980740 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:33.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.807+0000 7ff6185f7740 1 -- 172.21.15.90:0/1616764659 >> 172.21.15.90:0/1616764659 conn(0x560203947a50 msgr2=0x56020397b760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:49:33.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.807+0000 7ff60ea00640 1 --2- 172.21.15.90:0/1837000856 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7ff5f40b4db0 0x7ff5f40b7270 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x560203981f40 tx=0x7ff600009900 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:49:33.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.808+0000 7ff6185f7740 1 -- 172.21.15.90:0/1616764659 shutdown_connections 2024-04-02T14:49:33.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.808+0000 7ff6185f7740 1 -- 172.21.15.90:0/1616764659 wait complete. 2024-04-02T14:49:33.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.808+0000 7ff6185f7740 1 -- start start 2024-04-02T14:49:33.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.808+0000 7ff6185f7740 1 Processor -- start 2024-04-02T14:49:33.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.810+0000 7ff6185f7740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560203980d10 0x5602039870d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:33.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.810+0000 7ff6185f7740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5602039819f0 0x560203987610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:33.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.810+0000 7ff6185f7740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56020398a080 0x560203987b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:33.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.810+0000 7ff6185f7740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5602038e16a0 con 0x56020398a080 2024-04-02T14:49:33.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.810+0000 7ff6185f7740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x56020388d980 con 0x560203980d10 2024-04-02T14:49:33.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.810+0000 7ff6185f7740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5602038a0cb0 con 0x5602039819f0 2024-04-02T14:49:33.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.810+0000 7ff6185f7740 10 client.0 _ll_register_callbacks cb 0x5602038a06a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:49:33.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.811+0000 7ff60e000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5602039819f0 0x560203987610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:33.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.811+0000 7ff60e000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5602039819f0 0x560203987610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:35116/0 (socket says 172.21.15.90:35116) 2024-04-02T14:49:33.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.811+0000 7ff60e000640 1 -- 172.21.15.90:0/3218570153 learned_addr learned my addr 172.21.15.90:0/3218570153 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:49:33.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.811+0000 7ff60e000640 1 -- 172.21.15.90:0/3218570153 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560203980d10 msgr2=0x5602039870d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-02T14:49:33.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.811+0000 7ff60e000640 1 --2- 172.21.15.90:0/3218570153 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560203980d10 0x5602039870d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:33.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.811+0000 7ff60e000640 1 -- 172.21.15.90:0/3218570153 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56020398a080 msgr2=0x560203987b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:49:33.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.811+0000 7ff60e000640 1 --2- 172.21.15.90:0/3218570153 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56020398a080 0x560203987b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:33.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.811+0000 7ff60e000640 1 -- 172.21.15.90:0/3218570153 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5602036c37d0 con 0x5602039819f0 2024-04-02T14:49:33.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.812+0000 7ff60e000640 1 --2- 172.21.15.90:0/3218570153 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5602039819f0 0x560203987610 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7ff5f80ba500 tx=0x7ff5f802bc50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7b9ec9667cf7722d server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:49:33.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.812+0000 7ff5fcc00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:49:33.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.812+0000 7ff5fcc00640 1 -- 172.21.15.90:0/3218570153 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5f804b040 con 0x5602039819f0 2024-04-02T14:49:33.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.813+0000 7ff6185f7740 1 -- 172.21.15.90:0/3218570153 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff608017070 con 0x5602039819f0 2024-04-02T14:49:33.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.813+0000 7ff5fcc00640 1 -- 172.21.15.90:0/3218570153 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 8550+0+0 (secure 0 0 0) 0x7ff5f80f4110 con 0x5602039819f0 2024-04-02T14:49:33.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.813+0000 7ff5fcc00640 1 -- 172.21.15.90:0/3218570153 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7ff5f80bda00 con 0x5602039819f0 2024-04-02T14:49:33.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.813+0000 7ff6185f7740 10 client.29794 fetch_fsmap learned FSMap version 27 2024-04-02T14:49:33.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.813+0000 7ff6185f7740 1 -- 172.21.15.90:0/3218570153 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x560203998f00 con 0x5602039819f0 2024-04-02T14:49:33.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.814+0000 7ff5fcc00640 1 -- 172.21.15.90:0/3218570153 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff608017070 con 0x5602039819f0 2024-04-02T14:49:33.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.814+0000 7ff5fcc00640 1 -- 172.21.15.90:0/3218570153 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7ff5f80c6880 con 0x5602039819f0 2024-04-02T14:49:33.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.814+0000 7ff6185f7740 10 client.29794 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:49:33.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.814+0000 7ff6185f7740 10 client.29794 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.c 2024-04-02T14:49:33.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.814+0000 7ff6185f7740 20 client.29794 populate_metadata read hostname 'smithi090' 2024-04-02T14:49:33.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.814+0000 7ff6185f7740 1 --2- 172.21.15.90:0/3218570153 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x56020399a3b0 0x56020399c840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:33.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.814+0000 7ff6185f7740 4 client.29794 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:49:33.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.814+0000 7ff6185f7740 1 -- 172.21.15.90:0/3218570153 --> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5602038a5040 con 0x56020399a3b0 2024-04-02T14:49:33.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.815+0000 7ff60ea00640 1 --2- 172.21.15.90:0/3218570153 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x56020399a3b0 0x56020399c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:33.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.817+0000 7ff60ea00640 1 --2- 172.21.15.90:0/3218570153 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x56020399a3b0 0x56020399c840 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=b5de280d40b94425 server_cookie=2f7f62426a1b22be in_seq=0 out_seq=0 2024-04-02T14:49:33.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.817+0000 7ff5fcc00640 10 client.29794 ms_handle_connect on v2:172.21.15.177:6836/4223414858 2024-04-02T14:49:33.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.818+0000 7ff5fcc00640 1 -- 172.21.15.90:0/3218570153 <== mds.4 v2:172.21.15.177:6836/4223414858 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5602038a5040 con 0x56020399a3b0 2024-04-02T14:49:33.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.818+0000 7ff5fcc00640 10 client.29794 handle_command_reply: tid=0 2024-04-02T14:49:33.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.819+0000 7ff6185f7740 1 client.29794 shutdown 2024-04-02T14:49:33.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.822+0000 7ff6185f7740 1 -- 172.21.15.90:0/3218570153 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5602039819f0 msgr2=0x560203987610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:49:33.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.822+0000 7ff6185f7740 1 --2- 172.21.15.90:0/3218570153 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5602039819f0 0x560203987610 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7ff5f80ba500 tx=0x7ff5f802bc50 comp rx=0 tx=0).stop 2024-04-02T14:49:33.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.822+0000 7ff6185f7740 1 -- 172.21.15.90:0/3218570153 shutdown_connections 2024-04-02T14:49:33.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.822+0000 7ff6185f7740 1 --2- 172.21.15.90:0/3218570153 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x56020399a3b0 0x56020399c840 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:33.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.822+0000 7ff6185f7740 1 --2- 172.21.15.90:0/3218570153 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56020398a080 0x560203987b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:33.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.822+0000 7ff6185f7740 1 --2- 172.21.15.90:0/3218570153 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5602039819f0 0x560203987610 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:33.817 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.822+0000 7ff6185f7740 1 --2- 172.21.15.90:0/3218570153 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560203980d10 0x5602039870d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:33.817 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.822+0000 7ff5fcc00640 0 client.29794 ms_handle_reset on v2:172.21.15.177:6836/4223414858 2024-04-02T14:49:33.817 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.822+0000 7ff6185f7740 1 -- 172.21.15.90:0/3218570153 >> 172.21.15.90:0/3218570153 conn(0x560203947a50 msgr2=0x56020397d4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:49:33.817 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.823+0000 7ff6185f7740 1 -- 172.21.15.90:0/3218570153 shutdown_connections 2024-04-02T14:49:33.817 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.823+0000 7ff6185f7740 1 -- 172.21.15.90:0/3218570153 wait complete. 2024-04-02T14:49:33.817 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.823+0000 7ff6185f7740 20 client.29794 trim_cache size 0 max 16384 2024-04-02T14:49:33.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.857+0000 7ff6185f7740 1 Processor -- start 2024-04-02T14:49:33.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.858+0000 7ff6185f7740 1 -- start start 2024-04-02T14:49:33.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.859+0000 7ff6185f7740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560203980d10 0x560203989800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:33.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.859+0000 7ff6185f7740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5602039819f0 0x560203995af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:33.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.859+0000 7ff6185f7740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56020398a080 0x560203982390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:33.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.859+0000 7ff6185f7740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7ff6100738e0 con 0x5602039819f0 2024-04-02T14:49:33.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.859+0000 7ff6185f7740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7ff6101a8e90 con 0x56020398a080 2024-04-02T14:49:33.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.859+0000 7ff6185f7740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7ff6101a7a20 con 0x560203980d10 2024-04-02T14:49:33.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.859+0000 7ff60ea00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560203980d10 0x560203989800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:33.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.859+0000 7ff60ea00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560203980d10 0x560203989800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:53114/0 (socket says 172.21.15.90:53114) 2024-04-02T14:49:33.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.859+0000 7ff60ea00640 1 -- 172.21.15.90:0/845323609 learned_addr learned my addr 172.21.15.90:0/845323609 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:49:33.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.859+0000 7ff60f400640 1 --2- 172.21.15.90:0/845323609 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56020398a080 0x560203982390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:33.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.860+0000 7ff60e000640 1 --2- 172.21.15.90:0/845323609 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5602039819f0 0x560203995af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:33.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.860+0000 7ff60ea00640 1 -- 172.21.15.90:0/845323609 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56020398a080 msgr2=0x560203982390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:49:33.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.860+0000 7ff60ea00640 1 --2- 172.21.15.90:0/845323609 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56020398a080 0x560203982390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:33.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.860+0000 7ff60ea00640 1 -- 172.21.15.90:0/845323609 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5602039819f0 msgr2=0x560203995af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:49:33.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.860+0000 7ff60ea00640 1 --2- 172.21.15.90:0/845323609 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5602039819f0 0x560203995af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:33.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.860+0000 7ff60ea00640 1 -- 172.21.15.90:0/845323609 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56020397fd40 con 0x560203980d10 2024-04-02T14:49:33.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.860+0000 7ff60ea00640 1 --2- 172.21.15.90:0/845323609 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560203980d10 0x560203989800 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7ff600011ca0 tx=0x7ff600014000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f5f28f683783bc20 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:49:33.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.861+0000 7ff5ff400640 1 -- 172.21.15.90:0/845323609 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff60002b030 con 0x560203980d10 2024-04-02T14:49:33.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.861+0000 7ff5ff400640 1 -- 172.21.15.90:0/845323609 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff6000268d0 con 0x560203980d10 2024-04-02T14:49:33.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.861+0000 7ff5ff400640 1 -- 172.21.15.90:0/845323609 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff60001d410 con 0x560203980d10 2024-04-02T14:49:33.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.861+0000 7ff6185f7740 1 -- 172.21.15.90:0/845323609 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560203980d10 msgr2=0x560203989800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:49:33.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.861+0000 7ff6185f7740 1 --2- 172.21.15.90:0/845323609 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560203980d10 0x560203989800 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7ff600011ca0 tx=0x7ff600014000 comp rx=0 tx=0).stop 2024-04-02T14:49:33.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.862+0000 7ff6185f7740 1 -- 172.21.15.90:0/845323609 shutdown_connections 2024-04-02T14:49:33.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.862+0000 7ff6185f7740 1 --2- 172.21.15.90:0/845323609 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56020398a080 0x560203982390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:33.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.862+0000 7ff6185f7740 1 --2- 172.21.15.90:0/845323609 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5602039819f0 0x560203995af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:33.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.862+0000 7ff6185f7740 1 --2- 172.21.15.90:0/845323609 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560203980d10 0x560203989800 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:33.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.862+0000 7ff6185f7740 1 -- 172.21.15.90:0/845323609 >> 172.21.15.90:0/845323609 conn(0x560203947a50 msgr2=0x56020397cb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:49:33.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.862+0000 7ff6185f7740 1 -- 172.21.15.90:0/845323609 shutdown_connections 2024-04-02T14:49:33.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.862+0000 7ff6185f7740 1 -- 172.21.15.90:0/845323609 wait complete. 2024-04-02T14:49:33.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.862+0000 7ff6185f7740 1 -- start start 2024-04-02T14:49:33.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.862+0000 7ff6185f7740 1 Processor -- start 2024-04-02T14:49:33.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.863+0000 7ff6185f7740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560203980d10 0x56020398a520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:33.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.863+0000 7ff6185f7740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5602039819f0 0x56020398aa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:33.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.863+0000 7ff6185f7740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56020398a080 0x560203991000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:33.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.863+0000 7ff6185f7740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7ff6101a87c0 con 0x5602039819f0 2024-04-02T14:49:33.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.863+0000 7ff6185f7740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7ff6101a80f0 con 0x56020398a080 2024-04-02T14:49:33.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.863+0000 7ff6185f7740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7ff6100718e0 con 0x560203980d10 2024-04-02T14:49:33.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.863+0000 7ff6185f7740 10 client.0 _ll_register_callbacks cb 0x7ff5f8009560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:49:33.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.864+0000 7ff60ea00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560203980d10 0x56020398a520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:33.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.864+0000 7ff60ea00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560203980d10 0x56020398a520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:53120/0 (socket says 172.21.15.90:53120) 2024-04-02T14:49:33.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.864+0000 7ff60ea00640 1 -- 172.21.15.90:0/3014812257 learned_addr learned my addr 172.21.15.90:0/3014812257 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:49:33.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.864+0000 7ff60f400640 1 --2- 172.21.15.90:0/3014812257 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56020398a080 0x560203991000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:33.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.864+0000 7ff60e000640 1 --2- 172.21.15.90:0/3014812257 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5602039819f0 0x56020398aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:33.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.864+0000 7ff60f400640 1 -- 172.21.15.90:0/3014812257 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560203980d10 msgr2=0x56020398a520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:49:33.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.864+0000 7ff60f400640 1 --2- 172.21.15.90:0/3014812257 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560203980d10 0x56020398a520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:33.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.864+0000 7ff60f400640 1 -- 172.21.15.90:0/3014812257 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5602039819f0 msgr2=0x56020398aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:49:33.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.864+0000 7ff60f400640 1 --2- 172.21.15.90:0/3014812257 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5602039819f0 0x56020398aa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:33.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.865+0000 7ff60f400640 1 -- 172.21.15.90:0/3014812257 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560203991540 con 0x56020398a080 2024-04-02T14:49:33.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.865+0000 7ff60e000640 1 --2- 172.21.15.90:0/3014812257 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5602039819f0 0x56020398aa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:49:33.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.865+0000 7ff60f400640 1 --2- 172.21.15.90:0/3014812257 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56020398a080 0x560203991000 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7ff60800b920 tx=0x7ff608012660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b8b16d802e56fee server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:49:33.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.865+0000 7ff5fcc00640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:49:33.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.865+0000 7ff5fcc00640 1 -- 172.21.15.90:0/3014812257 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff608021070 con 0x56020398a080 2024-04-02T14:49:33.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.865+0000 7ff5fcc00640 1 -- 172.21.15.90:0/3014812257 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 8550+0+0 (secure 0 0 0) 0x7ff608003c30 con 0x56020398a080 2024-04-02T14:49:33.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.865+0000 7ff6185f7740 1 -- 172.21.15.90:0/3014812257 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff6000268d0 con 0x56020398a080 2024-04-02T14:49:33.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.865+0000 7ff5fcc00640 1 -- 172.21.15.90:0/3014812257 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7ff608003e30 con 0x56020398a080 2024-04-02T14:49:33.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.866+0000 7ff6185f7740 10 client.29999 fetch_fsmap learned FSMap version 27 2024-04-02T14:49:33.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.866+0000 7ff6185f7740 1 -- 172.21.15.90:0/3014812257 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x560203992840 con 0x56020398a080 2024-04-02T14:49:33.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.866+0000 7ff5fcc00640 1 -- 172.21.15.90:0/3014812257 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff6000268d0 con 0x56020398a080 2024-04-02T14:49:33.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.866+0000 7ff5fcc00640 1 -- 172.21.15.90:0/3014812257 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7ff608011070 con 0x56020398a080 2024-04-02T14:49:33.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.866+0000 7ff6185f7740 10 client.29999 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:49:33.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.866+0000 7ff6185f7740 10 client.29999 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.c 2024-04-02T14:49:33.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.866+0000 7ff6185f7740 20 client.29999 populate_metadata read hostname 'smithi090' 2024-04-02T14:49:33.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.866+0000 7ff6185f7740 1 --2- 172.21.15.90:0/3014812257 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x560203a092d0 0x560203a0b6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:33.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.866+0000 7ff6185f7740 4 client.29999 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-02T14:49:33.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.866+0000 7ff6185f7740 1 -- 172.21.15.90:0/3014812257 --> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5602038f2c30 con 0x560203a092d0 2024-04-02T14:49:33.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.867+0000 7ff60ea00640 1 --2- 172.21.15.90:0/3014812257 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x560203a092d0 0x560203a0b6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:33.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.867+0000 7ff5fcc00640 10 client.29999 ms_handle_connect on v2:172.21.15.177:6836/4223414858 2024-04-02T14:49:33.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.867+0000 7ff60ea00640 1 --2- 172.21.15.90:0/3014812257 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x560203a092d0 0x560203a0b6f0 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=ee8d8473dfb36c5d server_cookie=8427c5d28dbc3973 in_seq=0 out_seq=0 2024-04-02T14:49:33.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.868+0000 7ff5fcc00640 1 -- 172.21.15.90:0/3014812257 <== mds.4 v2:172.21.15.177:6836/4223414858 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5602038f2c30 con 0x560203a092d0 2024-04-02T14:49:33.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.868+0000 7ff5fcc00640 10 client.29999 handle_command_reply: tid=0 2024-04-02T14:49:33.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.868+0000 7ff6185f7740 1 client.29999 shutdown 2024-04-02T14:49:33.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.868+0000 7ff6185f7740 1 -- 172.21.15.90:0/3014812257 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56020398a080 msgr2=0x560203991000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:49:33.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.868+0000 7ff6185f7740 1 --2- 172.21.15.90:0/3014812257 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56020398a080 0x560203991000 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7ff60800b920 tx=0x7ff608012660 comp rx=0 tx=0).stop 2024-04-02T14:49:33.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.868+0000 7ff6185f7740 1 -- 172.21.15.90:0/3014812257 shutdown_connections 2024-04-02T14:49:33.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.868+0000 7ff6185f7740 1 --2- 172.21.15.90:0/3014812257 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56020398a080 0x560203991000 unknown :-1 s=CLOSED pgs=988 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:33.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.868+0000 7ff6185f7740 1 --2- 172.21.15.90:0/3014812257 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x560203a092d0 0x560203a0b6f0 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:33.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.868+0000 7ff6185f7740 1 --2- 172.21.15.90:0/3014812257 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5602039819f0 0x56020398aa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:33.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.868+0000 7ff6185f7740 1 --2- 172.21.15.90:0/3014812257 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560203980d10 0x56020398a520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:33.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.868+0000 7ff6185f7740 1 -- 172.21.15.90:0/3014812257 >> 172.21.15.90:0/3014812257 conn(0x560203947a50 msgr2=0x56020397d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:49:33.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.868+0000 7ff5fcc00640 0 client.29999 ms_handle_reset on v2:172.21.15.177:6836/4223414858 2024-04-02T14:49:33.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.868+0000 7ff6185f7740 1 -- 172.21.15.90:0/3014812257 shutdown_connections 2024-04-02T14:49:33.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.868+0000 7ff6185f7740 1 -- 172.21.15.90:0/3014812257 wait complete. 2024-04-02T14:49:33.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.869+0000 7ff6185f7740 20 client.29999 trim_cache size 0 max 16384 2024-04-02T14:49:33.865 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-02T14:49:33.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.871+0000 7ff615800640 1 -- 172.21.15.90:0/1837000856 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7ff5f40b4db0 msgr2=0x7ff5f40b7270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:49:33.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.871+0000 7ff615800640 1 --2- 172.21.15.90:0/1837000856 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7ff5f40b4db0 0x7ff5f40b7270 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x560203981f40 tx=0x7ff600009900 comp rx=0 tx=0).stop 2024-04-02T14:49:33.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.872+0000 7ff615800640 1 -- 172.21.15.90:0/1837000856 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff610108fe0 msgr2=0x7ff61019abe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:49:33.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.872+0000 7ff615800640 1 --2- 172.21.15.90:0/1837000856 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff610108fe0 0x7ff61019abe0 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7ff5f802c190 tx=0x7ff5f80042f0 comp rx=0 tx=0).stop 2024-04-02T14:49:33.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.872+0000 7ff615800640 1 -- 172.21.15.90:0/1837000856 shutdown_connections 2024-04-02T14:49:33.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.872+0000 7ff615800640 1 --2- 172.21.15.90:0/1837000856 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7ff5f40b4db0 0x7ff5f40b7270 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:33.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.872+0000 7ff615800640 1 --2- 172.21.15.90:0/1837000856 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff61019ef60 0x7ff61019f410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:33.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.872+0000 7ff615800640 1 --2- 172.21.15.90:0/1837000856 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff610108fe0 0x7ff61019abe0 unknown :-1 s=CLOSED pgs=987 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:33.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.872+0000 7ff615800640 1 --2- 172.21.15.90:0/1837000856 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff610103f10 0x7ff61019a6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:33.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.872+0000 7ff615800640 1 -- 172.21.15.90:0/1837000856 >> 172.21.15.90:0/1837000856 conn(0x7ff6100ff7b0 msgr2=0x7ff610100ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:49:33.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.873+0000 7ff615800640 1 -- 172.21.15.90:0/1837000856 shutdown_connections 2024-04-02T14:49:33.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:33.873+0000 7ff615800640 1 -- 172.21.15.90:0/1837000856 wait complete. 2024-04-02T14:49:33.889 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-04-02T14:49:34.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.137+0000 7ff53e200640 1 -- 172.21.15.90:0/314859186 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff538103f10 msgr2=0x7ff5381042f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:49:34.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.137+0000 7ff53e200640 1 --2- 172.21.15.90:0/314859186 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff538103f10 0x7ff5381042f0 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7ff528009900 tx=0x7ff52802bc50 comp rx=0 tx=0).stop 2024-04-02T14:49:34.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.138+0000 7ff53e200640 1 -- 172.21.15.90:0/314859186 shutdown_connections 2024-04-02T14:49:34.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.138+0000 7ff53e200640 1 --2- 172.21.15.90:0/314859186 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff538108fe0 0x7ff538110bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:34.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.138+0000 7ff53e200640 1 --2- 172.21.15.90:0/314859186 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff5381048c0 0x7ff538108900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:34.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.138+0000 7ff53e200640 1 --2- 172.21.15.90:0/314859186 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff538103f10 0x7ff5381042f0 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:34.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.138+0000 7ff53e200640 1 -- 172.21.15.90:0/314859186 >> 172.21.15.90:0/314859186 conn(0x7ff5380ff7b0 msgr2=0x7ff538101bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:49:34.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.138+0000 7ff53e200640 1 -- 172.21.15.90:0/314859186 shutdown_connections 2024-04-02T14:49:34.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.142+0000 7ff53e200640 1 -- 172.21.15.90:0/314859186 wait complete. 2024-04-02T14:49:34.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.146+0000 7ff53e200640 1 Processor -- start 2024-04-02T14:49:34.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.153+0000 7ff53e200640 1 -- start start 2024-04-02T14:49:34.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.153+0000 7ff53e200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff538103f10 0x7ff53819a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:34.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.153+0000 7ff53e200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff538108fe0 0x7ff53819ab30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:34.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.154+0000 7ff53e200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff53819eeb0 0x7ff53819f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:34.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.154+0000 7ff53e200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7ff5381140d0 con 0x7ff538103f10 2024-04-02T14:49:34.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.154+0000 7ff53e200640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7ff538113f50 con 0x7ff538108fe0 2024-04-02T14:49:34.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.154+0000 7ff53e200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7ff538114250 con 0x7ff53819eeb0 2024-04-02T14:49:34.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.154+0000 7ff537400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff538103f10 0x7ff53819a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:34.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.154+0000 7ff536a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff538108fe0 0x7ff53819ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:34.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.154+0000 7ff536a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff538108fe0 0x7ff53819ab30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:50044/0 (socket says 172.21.15.90:50044) 2024-04-02T14:49:34.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.154+0000 7ff536a00640 1 -- 172.21.15.90:0/2517678158 learned_addr learned my addr 172.21.15.90:0/2517678158 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:49:34.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.154+0000 7ff537e00640 1 --2- 172.21.15.90:0/2517678158 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff53819eeb0 0x7ff53819f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:34.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.157+0000 7ff536a00640 1 -- 172.21.15.90:0/2517678158 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff53819eeb0 msgr2=0x7ff53819f360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:49:34.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.157+0000 7ff536a00640 1 --2- 172.21.15.90:0/2517678158 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff53819eeb0 0x7ff53819f360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:34.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.157+0000 7ff536a00640 1 -- 172.21.15.90:0/2517678158 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff538103f10 msgr2=0x7ff53819a5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:49:34.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.157+0000 7ff536a00640 1 --2- 172.21.15.90:0/2517678158 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff538103f10 0x7ff53819a5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:34.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.157+0000 7ff536a00640 1 -- 172.21.15.90:0/2517678158 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff53819f9a0 con 0x7ff538108fe0 2024-04-02T14:49:34.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.157+0000 7ff537e00640 1 --2- 172.21.15.90:0/2517678158 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff53819eeb0 0x7ff53819f360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:49:34.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.162+0000 7ff536a00640 1 --2- 172.21.15.90:0/2517678158 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff538108fe0 0x7ff53819ab30 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7ff52400e9a0 tx=0x7ff52400ee60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:49:34.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.162+0000 7ff52fe00640 1 -- 172.21.15.90:0/2517678158 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff52400cd80 con 0x7ff538108fe0 2024-04-02T14:49:34.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.163+0000 7ff53e200640 1 -- 172.21.15.90:0/2517678158 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff53819fc90 con 0x7ff538108fe0 2024-04-02T14:49:34.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.163+0000 7ff53e200640 1 -- 172.21.15.90:0/2517678158 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5381a74d0 con 0x7ff538108fe0 2024-04-02T14:49:34.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.170+0000 7ff52fe00640 1 -- 172.21.15.90:0/2517678158 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff524009450 con 0x7ff538108fe0 2024-04-02T14:49:34.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.170+0000 7ff52fe00640 1 -- 172.21.15.90:0/2517678158 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff524010430 con 0x7ff538108fe0 2024-04-02T14:49:34.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.174+0000 7ff540f85740 1 Processor -- start 2024-04-02T14:49:34.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.174+0000 7ff540f85740 1 -- start start 2024-04-02T14:49:34.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.174+0000 7ff540f85740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559dcddef9f0 0x559dcddee740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:34.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.175+0000 7ff540f85740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559dcddeed10 0x559dcddef190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:34.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.175+0000 7ff540f85740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559dcddf8080 0x559dcddfa470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:34.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.175+0000 7ff540f85740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x559dcdd2f400 con 0x559dcddeed10 2024-04-02T14:49:34.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.175+0000 7ff540f85740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x559dcdd13f10 con 0x559dcddf8080 2024-04-02T14:49:34.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.175+0000 7ff540f85740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x559dcdd536f0 con 0x559dcddef9f0 2024-04-02T14:49:34.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.176+0000 7ff537400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559dcddeed10 0x559dcddef190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:34.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.176+0000 7ff537400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559dcddeed10 0x559dcddef190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:46176/0 (socket says 172.21.15.90:46176) 2024-04-02T14:49:34.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.176+0000 7ff537400640 1 -- 172.21.15.90:0/1052368820 learned_addr learned my addr 172.21.15.90:0/1052368820 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:49:34.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.176+0000 7ff52fe00640 1 -- 172.21.15.90:0/2517678158 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7ff5240106d0 con 0x7ff538108fe0 2024-04-02T14:49:34.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.177+0000 7ff537400640 1 -- 172.21.15.90:0/1052368820 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559dcddef9f0 msgr2=0x559dcddee740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:49:34.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.177+0000 7ff537400640 1 --2- 172.21.15.90:0/1052368820 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559dcddef9f0 0x559dcddee740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:34.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.177+0000 7ff537400640 1 -- 172.21.15.90:0/1052368820 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559dcddf8080 msgr2=0x559dcddfa470 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-02T14:49:34.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.177+0000 7ff537400640 1 --2- 172.21.15.90:0/1052368820 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559dcddf8080 0x559dcddfa470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:34.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.177+0000 7ff537400640 1 -- 172.21.15.90:0/1052368820 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559dcda673f0 con 0x559dcddeed10 2024-04-02T14:49:34.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.178+0000 7ff537400640 1 --2- 172.21.15.90:0/1052368820 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559dcddeed10 0x559dcddef190 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7ff52802c1e0 tx=0x7ff5280095f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6573b9928b6b4c28 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:49:34.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.178+0000 7ff52fe00640 1 --2- 172.21.15.90:0/2517678158 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7ff51c0a3de0 0x7ff51c0a62a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:34.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.179+0000 7ff52fe00640 1 -- 172.21.15.90:0/2517678158 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 8550+0+0 (secure 0 0 0) 0x7ff5240d0990 con 0x7ff538108fe0 2024-04-02T14:49:34.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.179+0000 7ff523e00640 1 -- 172.21.15.90:0/1052368820 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff528043d20 con 0x559dcddeed10 2024-04-02T14:49:34.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.179+0000 7ff523e00640 1 -- 172.21.15.90:0/1052368820 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff52802c6f0 con 0x559dcddeed10 2024-04-02T14:49:34.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.179+0000 7ff523e00640 1 -- 172.21.15.90:0/1052368820 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5280053e0 con 0x559dcddeed10 2024-04-02T14:49:34.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.184+0000 7ff540f85740 1 -- 172.21.15.90:0/1052368820 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559dcddeed10 msgr2=0x559dcddef190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:49:34.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.184+0000 7ff540f85740 1 --2- 172.21.15.90:0/1052368820 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559dcddeed10 0x559dcddef190 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7ff52802c1e0 tx=0x7ff5280095f0 comp rx=0 tx=0).stop 2024-04-02T14:49:34.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.184+0000 7ff537e00640 1 --2- 172.21.15.90:0/2517678158 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7ff51c0a3de0 0x7ff51c0a62a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:34.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.185+0000 7ff537e00640 1 --2- 172.21.15.90:0/2517678158 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7ff51c0a3de0 0x7ff51c0a62a0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7ff530009fd0 tx=0x7ff530009e80 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:49:34.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.185+0000 7ff540f85740 1 -- 172.21.15.90:0/1052368820 shutdown_connections 2024-04-02T14:49:34.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.185+0000 7ff540f85740 1 --2- 172.21.15.90:0/1052368820 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559dcddf8080 0x559dcddfa470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:34.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.186+0000 7ff540f85740 1 --2- 172.21.15.90:0/1052368820 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559dcddeed10 0x559dcddef190 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:34.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.186+0000 7ff540f85740 1 --2- 172.21.15.90:0/1052368820 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559dcddef9f0 0x559dcddee740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:34.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.186+0000 7ff540f85740 1 -- 172.21.15.90:0/1052368820 >> 172.21.15.90:0/1052368820 conn(0x559dcddb5a50 msgr2=0x559dcdde9760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:49:34.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.188+0000 7ff540f85740 1 -- 172.21.15.90:0/1052368820 shutdown_connections 2024-04-02T14:49:34.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.188+0000 7ff540f85740 1 -- 172.21.15.90:0/1052368820 wait complete. 2024-04-02T14:49:34.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.189+0000 7ff540f85740 1 -- start start 2024-04-02T14:49:34.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.190+0000 7ff540f85740 1 Processor -- start 2024-04-02T14:49:34.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.191+0000 7ff540f85740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559dcddeed10 0x559dcddf4ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:34.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.191+0000 7ff540f85740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559dcddef9f0 0x559dcddf5530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:34.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.191+0000 7ff540f85740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559dcddf8080 0x559dcddf5a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:34.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.191+0000 7ff540f85740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x559dcdd4f6a0 con 0x559dcddef9f0 2024-04-02T14:49:34.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.191+0000 7ff540f85740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x559dcdcfb980 con 0x559dcddf8080 2024-04-02T14:49:34.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.191+0000 7ff540f85740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x559dcdd0ecb0 con 0x559dcddeed10 2024-04-02T14:49:34.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.193+0000 7ff536a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559dcddef9f0 0x559dcddf5530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:34.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.193+0000 7ff537400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559dcddeed10 0x559dcddf4ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:34.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.193+0000 7ff536a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559dcddef9f0 0x559dcddf5530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:46182/0 (socket says 172.21.15.90:46182) 2024-04-02T14:49:34.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.193+0000 7ff536a00640 1 -- 172.21.15.90:0/2227565385 learned_addr learned my addr 172.21.15.90:0/2227565385 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:49:34.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.193+0000 7ff537400640 1 -- 172.21.15.90:0/2227565385 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559dcddf8080 msgr2=0x559dcddf5a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:49:34.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.193+0000 7ff537400640 1 --2- 172.21.15.90:0/2227565385 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559dcddf8080 0x559dcddf5a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:34.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.193+0000 7ff537400640 1 -- 172.21.15.90:0/2227565385 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559dcddef9f0 msgr2=0x559dcddf5530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:49:34.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.193+0000 7ff537400640 1 --2- 172.21.15.90:0/2227565385 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559dcddef9f0 0x559dcddf5530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:34.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.193+0000 7ff537400640 1 -- 172.21.15.90:0/2227565385 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559dcdb317d0 con 0x559dcddeed10 2024-04-02T14:49:34.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.193+0000 7ff540f85740 10 client.0 _ll_register_callbacks cb 0x559dcdd0e6a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:49:34.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.194+0000 7ff540f85740 1 -- 172.21.15.90:0/2227565385 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff52802c6f0 con 0x559dcddeed10 2024-04-02T14:49:34.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.194+0000 7ff537400640 1 --2- 172.21.15.90:0/2227565385 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559dcddeed10 0x559dcddf4ff0 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7ff52802c260 tx=0x7ff5280043b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=17fcbc07c346cc92 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:49:34.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.194+0000 7ff520c00640 10 client.29809 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:49:34.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.195+0000 7ff520c00640 1 -- 172.21.15.90:0/2227565385 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff528005530 con 0x559dcddeed10 2024-04-02T14:49:34.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.195+0000 7ff520c00640 1 -- 172.21.15.90:0/2227565385 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 8550+0+0 (secure 0 0 0) 0x7ff5280046c0 con 0x559dcddeed10 2024-04-02T14:49:34.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.195+0000 7ff520c00640 1 -- 172.21.15.90:0/2227565385 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7ff528004900 con 0x559dcddeed10 2024-04-02T14:49:34.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.195+0000 7ff540f85740 10 client.29809 fetch_fsmap learned FSMap version 27 2024-04-02T14:49:34.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.196+0000 7ff540f85740 1 -- 172.21.15.90:0/2227565385 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({fsmap=27}) -- 0x559dcde17d10 con 0x559dcddeed10 2024-04-02T14:49:34.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.197+0000 7ff520c00640 1 -- 172.21.15.90:0/2227565385 <== mon.1 v2:172.21.15.177:3300/0 4 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7ff528004aa0 con 0x559dcddeed10 2024-04-02T14:49:34.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.197+0000 7ff540f85740 10 client.29809 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:49:34.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.197+0000 7ff540f85740 10 client.29809 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-04-02T14:49:34.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.197+0000 7ff540f85740 20 client.29809 populate_metadata read hostname 'smithi090' 2024-04-02T14:49:34.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.197+0000 7ff540f85740 1 --2- 172.21.15.90:0/2227565385 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x559dcde19270 0x559dcde1b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:34.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.197+0000 7ff540f85740 4 client.29809 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:49:34.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.197+0000 7ff540f85740 1 -- 172.21.15.90:0/2227565385 --> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559dcdd13040 con 0x559dcde19270 2024-04-02T14:49:34.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.198+0000 7ff536a00640 1 --2- 172.21.15.90:0/2227565385 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x559dcde19270 0x559dcde1b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:34.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.198+0000 7ff520c00640 10 client.29809 ms_handle_connect on v2:172.21.15.92:6836/1270635939 2024-04-02T14:49:34.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.198+0000 7ff536a00640 1 --2- 172.21.15.90:0/2227565385 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x559dcde19270 0x559dcde1b6d0 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=b3fc9972c98cfa0e server_cookie=d1b144955359bb4c in_seq=0 out_seq=0 2024-04-02T14:49:34.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.199+0000 7ff520c00640 1 -- 172.21.15.90:0/2227565385 <== mds.3 v2:172.21.15.92:6836/1270635939 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x559dcdd13040 con 0x559dcde19270 2024-04-02T14:49:34.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.199+0000 7ff520c00640 10 client.29809 handle_command_reply: tid=0 2024-04-02T14:49:34.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.200+0000 7ff540f85740 1 client.29809 shutdown 2024-04-02T14:49:34.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.200+0000 7ff540f85740 1 -- 172.21.15.90:0/2227565385 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559dcddeed10 msgr2=0x559dcddf4ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:49:34.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.200+0000 7ff540f85740 1 --2- 172.21.15.90:0/2227565385 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559dcddeed10 0x559dcddf4ff0 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7ff52802c260 tx=0x7ff5280043b0 comp rx=0 tx=0).stop 2024-04-02T14:49:34.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.200+0000 7ff540f85740 1 -- 172.21.15.90:0/2227565385 shutdown_connections 2024-04-02T14:49:34.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.200+0000 7ff540f85740 1 --2- 172.21.15.90:0/2227565385 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x559dcde19270 0x559dcde1b6d0 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:34.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.200+0000 7ff540f85740 1 --2- 172.21.15.90:0/2227565385 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559dcddf8080 0x559dcddf5a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:34.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.200+0000 7ff540f85740 1 --2- 172.21.15.90:0/2227565385 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559dcddef9f0 0x559dcddf5530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:34.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.200+0000 7ff540f85740 1 --2- 172.21.15.90:0/2227565385 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559dcddeed10 0x559dcddf4ff0 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:34.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.200+0000 7ff520c00640 0 client.29809 ms_handle_reset on v2:172.21.15.92:6836/1270635939 2024-04-02T14:49:34.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.200+0000 7ff540f85740 1 -- 172.21.15.90:0/2227565385 >> 172.21.15.90:0/2227565385 conn(0x559dcddb5a50 msgr2=0x559dcddeb4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:49:34.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.201+0000 7ff540f85740 1 -- 172.21.15.90:0/2227565385 shutdown_connections 2024-04-02T14:49:34.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.202+0000 7ff540f85740 1 -- 172.21.15.90:0/2227565385 wait complete. 2024-04-02T14:49:34.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.203+0000 7ff540f85740 20 client.29809 trim_cache size 0 max 16384 2024-04-02T14:49:34.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.241+0000 7ff540f85740 1 Processor -- start 2024-04-02T14:49:34.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.242+0000 7ff540f85740 1 -- start start 2024-04-02T14:49:34.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.242+0000 7ff540f85740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559dcddeed10 0x559dcddf7720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:34.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.242+0000 7ff540f85740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559dcddef9f0 0x559dcde03ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:34.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.242+0000 7ff540f85740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559dcddf8080 0x559dcddeb250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:34.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.242+0000 7ff540f85740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7ff51800e7c0 con 0x559dcddeed10 2024-04-02T14:49:34.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.242+0000 7ff540f85740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7ff518010300 con 0x559dcddef9f0 2024-04-02T14:49:34.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.242+0000 7ff540f85740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7ff51800ee90 con 0x559dcddf8080 2024-04-02T14:49:34.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.242+0000 7ff536a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559dcddef9f0 0x559dcde03ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:34.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.242+0000 7ff537e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559dcddf8080 0x559dcddeb250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:34.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.242+0000 7ff536a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559dcddef9f0 0x559dcde03ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:50076/0 (socket says 172.21.15.90:50076) 2024-04-02T14:49:34.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.242+0000 7ff536a00640 1 -- 172.21.15.90:0/1905623338 learned_addr learned my addr 172.21.15.90:0/1905623338 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:49:34.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.243+0000 7ff537400640 1 --2- 172.21.15.90:0/1905623338 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559dcddeed10 0x559dcddf7720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:34.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.243+0000 7ff537e00640 1 -- 172.21.15.90:0/1905623338 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559dcddef9f0 msgr2=0x559dcde03ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:49:34.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.243+0000 7ff537e00640 1 --2- 172.21.15.90:0/1905623338 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559dcddef9f0 0x559dcde03ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:34.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.243+0000 7ff537e00640 1 -- 172.21.15.90:0/1905623338 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559dcddeed10 msgr2=0x559dcddf7720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:49:34.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.243+0000 7ff537e00640 1 --2- 172.21.15.90:0/1905623338 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559dcddeed10 0x559dcddf7720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:34.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.243+0000 7ff537e00640 1 -- 172.21.15.90:0/1905623338 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559dcddedd40 con 0x559dcddf8080 2024-04-02T14:49:34.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.244+0000 7ff537e00640 1 --2- 172.21.15.90:0/1905623338 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559dcddf8080 0x559dcddeb250 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7ff530005fd0 tx=0x7ff530013040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=da227245ae724a7a server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:49:34.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.244+0000 7ff523400640 1 -- 172.21.15.90:0/1905623338 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff530004040 con 0x559dcddf8080 2024-04-02T14:49:34.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.244+0000 7ff523400640 1 -- 172.21.15.90:0/1905623338 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff5300046c0 con 0x559dcddf8080 2024-04-02T14:49:34.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.244+0000 7ff523400640 1 -- 172.21.15.90:0/1905623338 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff530004040 con 0x559dcddf8080 2024-04-02T14:49:34.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.245+0000 7ff540f85740 1 -- 172.21.15.90:0/1905623338 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559dcddf8080 msgr2=0x559dcddeb250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:49:34.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.245+0000 7ff540f85740 1 --2- 172.21.15.90:0/1905623338 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559dcddf8080 0x559dcddeb250 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7ff530005fd0 tx=0x7ff530013040 comp rx=0 tx=0).stop 2024-04-02T14:49:34.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.247+0000 7ff540f85740 1 -- 172.21.15.90:0/1905623338 shutdown_connections 2024-04-02T14:49:34.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.247+0000 7ff540f85740 1 --2- 172.21.15.90:0/1905623338 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559dcddf8080 0x559dcddeb250 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:34.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.247+0000 7ff540f85740 1 --2- 172.21.15.90:0/1905623338 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559dcddef9f0 0x559dcde03ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:34.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.247+0000 7ff540f85740 1 --2- 172.21.15.90:0/1905623338 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559dcddeed10 0x559dcddf7720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:34.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.247+0000 7ff540f85740 1 -- 172.21.15.90:0/1905623338 >> 172.21.15.90:0/1905623338 conn(0x559dcddb5a50 msgr2=0x559dcddeab40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:49:34.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.247+0000 7ff540f85740 1 -- 172.21.15.90:0/1905623338 shutdown_connections 2024-04-02T14:49:34.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.247+0000 7ff540f85740 1 -- 172.21.15.90:0/1905623338 wait complete. 2024-04-02T14:49:34.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.248+0000 7ff540f85740 1 -- start start 2024-04-02T14:49:34.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.248+0000 7ff540f85740 1 Processor -- start 2024-04-02T14:49:34.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.249+0000 7ff540f85740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559dcddeed10 0x559dcddf8e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:34.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.249+0000 7ff540f85740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559dcddef9f0 0x559dcddf9340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:34.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.249+0000 7ff540f85740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559dcddf8080 0x559dcddf9880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:34.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.249+0000 7ff540f85740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7ff51800fc30 con 0x559dcddf8080 2024-04-02T14:49:34.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.249+0000 7ff540f85740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7ff51800f560 con 0x559dcddeed10 2024-04-02T14:49:34.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.249+0000 7ff540f85740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7ff51800c7c0 con 0x559dcddef9f0 2024-04-02T14:49:34.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.249+0000 7ff540f85740 10 client.0 _ll_register_callbacks cb 0x7ff528004040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:49:34.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.249+0000 7ff536a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559dcddef9f0 0x559dcddf9340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:34.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.249+0000 7ff536a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559dcddef9f0 0x559dcddf9340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:53146/0 (socket says 172.21.15.90:53146) 2024-04-02T14:49:34.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.249+0000 7ff536a00640 1 -- 172.21.15.90:0/1299325817 learned_addr learned my addr 172.21.15.90:0/1299325817 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:49:34.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.249+0000 7ff537400640 1 --2- 172.21.15.90:0/1299325817 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559dcddeed10 0x559dcddf8e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:34.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.250+0000 7ff536a00640 1 -- 172.21.15.90:0/1299325817 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559dcddeed10 msgr2=0x559dcddf8e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:49:34.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.250+0000 7ff536a00640 1 --2- 172.21.15.90:0/1299325817 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559dcddeed10 0x559dcddf8e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:34.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.250+0000 7ff536a00640 1 -- 172.21.15.90:0/1299325817 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559dcddf8080 msgr2=0x559dcddf9880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:49:34.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.250+0000 7ff537e00640 1 --2- 172.21.15.90:0/1299325817 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559dcddf8080 0x559dcddf9880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:34.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.250+0000 7ff536a00640 1 --2- 172.21.15.90:0/1299325817 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559dcddf8080 0x559dcddf9880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:34.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.250+0000 7ff536a00640 1 -- 172.21.15.90:0/1299325817 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559dcddfe490 con 0x559dcddef9f0 2024-04-02T14:49:34.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.250+0000 7ff537e00640 1 --2- 172.21.15.90:0/1299325817 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559dcddf8080 0x559dcddf9880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:49:34.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.251+0000 7ff536a00640 1 --2- 172.21.15.90:0/1299325817 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559dcddef9f0 0x559dcddf9340 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7ff5240106a0 tx=0x7ff52400cf90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=31a83a24f06273ee server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:49:34.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.251+0000 7ff520c00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:49:34.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.251+0000 7ff520c00640 1 -- 172.21.15.90:0/1299325817 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5240a6740 con 0x559dcddef9f0 2024-04-02T14:49:34.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.251+0000 7ff520c00640 1 -- 172.21.15.90:0/1299325817 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 8550+0+0 (secure 0 0 0) 0x7ff5240ce340 con 0x559dcddef9f0 2024-04-02T14:49:34.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.251+0000 7ff540f85740 1 -- 172.21.15.90:0/1299325817 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff5300046c0 con 0x559dcddef9f0 2024-04-02T14:49:34.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.251+0000 7ff520c00640 1 -- 172.21.15.90:0/1299325817 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7ff52409a140 con 0x559dcddef9f0 2024-04-02T14:49:34.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.252+0000 7ff540f85740 10 client.29815 fetch_fsmap learned FSMap version 27 2024-04-02T14:49:34.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.252+0000 7ff540f85740 1 -- 172.21.15.90:0/1299325817 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x559dcde7fd10 con 0x559dcddef9f0 2024-04-02T14:49:34.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.252+0000 7ff520c00640 1 -- 172.21.15.90:0/1299325817 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff5300046c0 con 0x559dcddef9f0 2024-04-02T14:49:34.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.252+0000 7ff520c00640 1 -- 172.21.15.90:0/1299325817 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7ff5240b9030 con 0x559dcddef9f0 2024-04-02T14:49:34.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.252+0000 7ff540f85740 10 client.29815 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:49:34.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.253+0000 7ff540f85740 10 client.29815 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-04-02T14:49:34.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.253+0000 7ff540f85740 20 client.29815 populate_metadata read hostname 'smithi090' 2024-04-02T14:49:34.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.253+0000 7ff540f85740 1 --2- 172.21.15.90:0/1299325817 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x559dcde80320 0x559dcde827a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:34.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.253+0000 7ff540f85740 4 client.29815 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-02T14:49:34.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.253+0000 7ff540f85740 1 -- 172.21.15.90:0/1299325817 --> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] -- command(tid 0: {"prefix": "damage ls"}) -- 0x559dcdd60c30 con 0x559dcde80320 2024-04-02T14:49:34.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.253+0000 7ff537400640 1 --2- 172.21.15.90:0/1299325817 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x559dcde80320 0x559dcde827a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:34.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.257+0000 7ff537400640 1 --2- 172.21.15.90:0/1299325817 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x559dcde80320 0x559dcde827a0 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=9105b599d796acd1 server_cookie=f23e4efde8729a0a in_seq=0 out_seq=0 2024-04-02T14:49:34.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.257+0000 7ff520c00640 10 client.29815 ms_handle_connect on v2:172.21.15.92:6836/1270635939 2024-04-02T14:49:34.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.257+0000 7ff520c00640 1 -- 172.21.15.90:0/1299325817 <== mds.3 v2:172.21.15.92:6836/1270635939 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x559dcdd60c30 con 0x559dcde80320 2024-04-02T14:49:34.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.257+0000 7ff520c00640 10 client.29815 handle_command_reply: tid=0 2024-04-02T14:49:34.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.257+0000 7ff540f85740 1 client.29815 shutdown 2024-04-02T14:49:34.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.258+0000 7ff540f85740 1 -- 172.21.15.90:0/1299325817 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559dcddef9f0 msgr2=0x559dcddf9340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:49:34.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.258+0000 7ff540f85740 1 --2- 172.21.15.90:0/1299325817 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559dcddef9f0 0x559dcddf9340 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7ff5240106a0 tx=0x7ff52400cf90 comp rx=0 tx=0).stop 2024-04-02T14:49:34.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.258+0000 7ff540f85740 1 -- 172.21.15.90:0/1299325817 shutdown_connections 2024-04-02T14:49:34.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.258+0000 7ff540f85740 1 --2- 172.21.15.90:0/1299325817 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x559dcde80320 0x559dcde827a0 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:34.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.258+0000 7ff540f85740 1 --2- 172.21.15.90:0/1299325817 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x559dcddf8080 0x559dcddf9880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:34.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.258+0000 7ff540f85740 1 --2- 172.21.15.90:0/1299325817 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559dcddef9f0 0x559dcddf9340 secure :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7ff5240106a0 tx=0x7ff52400cf90 comp rx=0 tx=0).stop 2024-04-02T14:49:34.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.258+0000 7ff540f85740 1 --2- 172.21.15.90:0/1299325817 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x559dcddeed10 0x559dcddf8e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:34.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.258+0000 7ff540f85740 1 -- 172.21.15.90:0/1299325817 >> 172.21.15.90:0/1299325817 conn(0x559dcddb5a50 msgr2=0x559dcddf5840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:49:34.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.258+0000 7ff520c00640 0 client.29815 ms_handle_reset on v2:172.21.15.92:6836/1270635939 2024-04-02T14:49:34.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.259+0000 7ff540f85740 1 -- 172.21.15.90:0/1299325817 shutdown_connections 2024-04-02T14:49:34.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.259+0000 7ff540f85740 1 -- 172.21.15.90:0/1299325817 wait complete. 2024-04-02T14:49:34.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.259+0000 7ff540f85740 20 client.29815 trim_cache size 0 max 16384 2024-04-02T14:49:34.254 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-02T14:49:34.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.261+0000 7ff53e200640 1 -- 172.21.15.90:0/2517678158 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7ff51c0a3de0 msgr2=0x7ff51c0a62a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:49:34.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.261+0000 7ff53e200640 1 --2- 172.21.15.90:0/2517678158 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7ff51c0a3de0 0x7ff51c0a62a0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7ff530009fd0 tx=0x7ff530009e80 comp rx=0 tx=0).stop 2024-04-02T14:49:34.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.261+0000 7ff53e200640 1 -- 172.21.15.90:0/2517678158 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff538108fe0 msgr2=0x7ff53819ab30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:49:34.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.261+0000 7ff53e200640 1 --2- 172.21.15.90:0/2517678158 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff538108fe0 0x7ff53819ab30 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7ff52400e9a0 tx=0x7ff52400ee60 comp rx=0 tx=0).stop 2024-04-02T14:49:34.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.261+0000 7ff53e200640 1 -- 172.21.15.90:0/2517678158 shutdown_connections 2024-04-02T14:49:34.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.261+0000 7ff53e200640 1 --2- 172.21.15.90:0/2517678158 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7ff51c0a3de0 0x7ff51c0a62a0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:34.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.261+0000 7ff53e200640 1 --2- 172.21.15.90:0/2517678158 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff53819eeb0 0x7ff53819f360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:34.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.261+0000 7ff53e200640 1 --2- 172.21.15.90:0/2517678158 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff538108fe0 0x7ff53819ab30 unknown :-1 s=CLOSED pgs=989 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:34.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.261+0000 7ff53e200640 1 --2- 172.21.15.90:0/2517678158 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff538103f10 0x7ff53819a5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:34.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.261+0000 7ff53e200640 1 -- 172.21.15.90:0/2517678158 >> 172.21.15.90:0/2517678158 conn(0x7ff5380ff7b0 msgr2=0x7ff538101ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:49:34.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.261+0000 7ff53e200640 1 -- 172.21.15.90:0/2517678158 shutdown_connections 2024-04-02T14:49:34.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.261+0000 7ff53e200640 1 -- 172.21.15.90:0/2517678158 wait complete. 2024-04-02T14:49:34.276 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-02T14:49:34.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.534+0000 7f20b5000640 1 -- 172.21.15.90:0/4087319354 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f20b0073ed0 msgr2=0x7f20b00742b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:49:34.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.534+0000 7f20b5000640 1 --2- 172.21.15.90:0/4087319354 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f20b0073ed0 0x7f20b00742b0 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7f2098009900 tx=0x7f209802bcf0 comp rx=0 tx=0).stop 2024-04-02T14:49:34.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.535+0000 7f20b5000640 1 -- 172.21.15.90:0/4087319354 shutdown_connections 2024-04-02T14:49:34.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.535+0000 7f20b5000640 1 --2- 172.21.15.90:0/4087319354 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f20b010e330 0x7f20b0110740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:34.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.535+0000 7f20b5000640 1 --2- 172.21.15.90:0/4087319354 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f20b00747f0 0x7f20b010ddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:34.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.535+0000 7f20b5000640 1 --2- 172.21.15.90:0/4087319354 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f20b0073ed0 0x7f20b00742b0 unknown :-1 s=CLOSED pgs=990 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:34.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.535+0000 7f20b5000640 1 -- 172.21.15.90:0/4087319354 >> 172.21.15.90:0/4087319354 conn(0x7f20b0078f00 msgr2=0x7f20b0079310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:49:34.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.535+0000 7f20b5000640 1 -- 172.21.15.90:0/4087319354 shutdown_connections 2024-04-02T14:49:34.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.535+0000 7f20b5000640 1 -- 172.21.15.90:0/4087319354 wait complete. 2024-04-02T14:49:34.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.536+0000 7f20b5000640 1 Processor -- start 2024-04-02T14:49:34.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.536+0000 7f20b5000640 1 -- start start 2024-04-02T14:49:34.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.536+0000 7f20b5000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f20b0073ed0 0x7f20b01a4630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:34.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.536+0000 7f20b5000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f20b00747f0 0x7f20b01a4b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:34.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.536+0000 7f20b5000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f20b010e330 0x7f20b010d2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:34.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.536+0000 7f20b5000640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f20b0103760 con 0x7f20b010e330 2024-04-02T14:49:34.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.537+0000 7f20b5000640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f20b01035e0 con 0x7f20b0073ed0 2024-04-02T14:49:34.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.537+0000 7f20b5000640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f20b01038e0 con 0x7f20b00747f0 2024-04-02T14:49:34.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.537+0000 7f20ade00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f20b0073ed0 0x7f20b01a4630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:34.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.537+0000 7f20a7e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f20b00747f0 0x7f20b01a4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:34.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.537+0000 7f20ade00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f20b0073ed0 0x7f20b01a4630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:50108/0 (socket says 172.21.15.90:50108) 2024-04-02T14:49:34.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.537+0000 7f20ade00640 1 -- 172.21.15.90:0/2895500523 learned_addr learned my addr 172.21.15.90:0/2895500523 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:49:34.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.537+0000 7f20ae800640 1 --2- 172.21.15.90:0/2895500523 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f20b010e330 0x7f20b010d2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:34.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.537+0000 7f20ade00640 1 -- 172.21.15.90:0/2895500523 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f20b00747f0 msgr2=0x7f20b01a4b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:49:34.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.537+0000 7f20ade00640 1 --2- 172.21.15.90:0/2895500523 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f20b00747f0 0x7f20b01a4b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:34.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.537+0000 7f20ade00640 1 -- 172.21.15.90:0/2895500523 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f20b010e330 msgr2=0x7f20b010d2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:49:34.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.537+0000 7f20ade00640 1 --2- 172.21.15.90:0/2895500523 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f20b010e330 0x7f20b010d2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:34.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.537+0000 7f20ade00640 1 -- 172.21.15.90:0/2895500523 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20b010da30 con 0x7f20b0073ed0 2024-04-02T14:49:34.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.537+0000 7f20a7e00640 1 --2- 172.21.15.90:0/2895500523 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f20b00747f0 0x7f20b01a4b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:49:34.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.537+0000 7f20ade00640 1 --2- 172.21.15.90:0/2895500523 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f20b0073ed0 0x7f20b01a4630 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7f2098005ab0 tx=0x7f209802c7d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:49:34.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.538+0000 7f20a6a00640 1 -- 172.21.15.90:0/2895500523 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2098043020 con 0x7f20b0073ed0 2024-04-02T14:49:34.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.538+0000 7f20a6a00640 1 -- 172.21.15.90:0/2895500523 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f20980054e0 con 0x7f20b0073ed0 2024-04-02T14:49:34.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.538+0000 7f20b5000640 1 -- 172.21.15.90:0/2895500523 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f20b010dcc0 con 0x7f20b0073ed0 2024-04-02T14:49:34.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.538+0000 7f20a6a00640 1 -- 172.21.15.90:0/2895500523 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2098003be0 con 0x7f20b0073ed0 2024-04-02T14:49:34.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.538+0000 7f20b5000640 1 -- 172.21.15.90:0/2895500523 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20b01ab4d0 con 0x7f20b0073ed0 2024-04-02T14:49:34.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.541+0000 7f20a6a00640 1 -- 172.21.15.90:0/2895500523 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f2098046020 con 0x7f20b0073ed0 2024-04-02T14:49:34.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.542+0000 7f20b7e2d740 1 Processor -- start 2024-04-02T14:49:34.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.543+0000 7f20b7e2d740 1 -- start start 2024-04-02T14:49:34.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.545+0000 7f20b7e2d740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560d19e329f0 0x560d19e31740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:34.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.546+0000 7f20b7e2d740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560d19e31d10 0x560d19e32190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:34.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.546+0000 7f20b7e2d740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560d19e3b080 0x560d19e3d470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:34.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.546+0000 7f20b7e2d740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x560d19d72400 con 0x560d19e329f0 2024-04-02T14:49:34.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.546+0000 7f20b7e2d740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x560d19d56f10 con 0x560d19e31d10 2024-04-02T14:49:34.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.546+0000 7f20b7e2d740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x560d19d966f0 con 0x560d19e3b080 2024-04-02T14:49:34.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.546+0000 7f20ade00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560d19e31d10 0x560d19e32190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:34.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.546+0000 7f20ade00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560d19e31d10 0x560d19e32190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:50124/0 (socket says 172.21.15.90:50124) 2024-04-02T14:49:34.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.546+0000 7f20ade00640 1 -- 172.21.15.90:0/344744244 learned_addr learned my addr 172.21.15.90:0/344744244 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:49:34.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.547+0000 7f20ae800640 1 --2- 172.21.15.90:0/344744244 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560d19e329f0 0x560d19e31740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:34.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.547+0000 7f20a7e00640 1 --2- 172.21.15.90:0/344744244 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560d19e3b080 0x560d19e3d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:34.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.547+0000 7f20ade00640 1 -- 172.21.15.90:0/344744244 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560d19e3b080 msgr2=0x560d19e3d470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:49:34.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.547+0000 7f20a6a00640 1 --2- 172.21.15.90:0/2895500523 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f20940a3d20 0x7f20940a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:34.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.547+0000 7f20ade00640 1 --2- 172.21.15.90:0/344744244 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560d19e3b080 0x560d19e3d470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:34.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.547+0000 7f20ade00640 1 -- 172.21.15.90:0/344744244 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560d19e329f0 msgr2=0x560d19e31740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:49:34.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.547+0000 7f20ade00640 1 --2- 172.21.15.90:0/344744244 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560d19e329f0 0x560d19e31740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:34.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.547+0000 7f20ade00640 1 -- 172.21.15.90:0/344744244 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560d19aaa3f0 con 0x560d19e31d10 2024-04-02T14:49:34.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.547+0000 7f20ae800640 1 --2- 172.21.15.90:0/344744244 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560d19e329f0 0x560d19e31740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:49:34.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.547+0000 7f20a7e00640 1 --2- 172.21.15.90:0/344744244 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560d19e3b080 0x560d19e3d470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:49:34.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.548+0000 7f20ade00640 1 --2- 172.21.15.90:0/344744244 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560d19e31d10 0x560d19e32190 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7f20980eff60 tx=0x7f20980e9980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=941d0d8fae58bd73 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:49:34.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.548+0000 7f20ae800640 1 --2- 172.21.15.90:0/2895500523 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f20940a3d20 0x7f20940a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:34.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.548+0000 7f209f400640 1 -- 172.21.15.90:0/344744244 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20980be640 con 0x560d19e31d10 2024-04-02T14:49:34.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.548+0000 7f209f400640 1 -- 172.21.15.90:0/344744244 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f209803a400 con 0x560d19e31d10 2024-04-02T14:49:34.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.548+0000 7f209f400640 1 -- 172.21.15.90:0/344744244 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20980b5eb0 con 0x560d19e31d10 2024-04-02T14:49:34.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.548+0000 7f20a6a00640 1 -- 172.21.15.90:0/2895500523 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 8550+0+0 (secure 0 0 0) 0x7f20980f2840 con 0x7f20b0073ed0 2024-04-02T14:49:34.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.549+0000 7f20b7e2d740 1 -- 172.21.15.90:0/344744244 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560d19e31d10 msgr2=0x560d19e32190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:49:34.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.549+0000 7f20b7e2d740 1 --2- 172.21.15.90:0/344744244 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560d19e31d10 0x560d19e32190 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7f20980eff60 tx=0x7f20980e9980 comp rx=0 tx=0).stop 2024-04-02T14:49:34.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.549+0000 7f20ae800640 1 --2- 172.21.15.90:0/2895500523 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f20940a3d20 0x7f20940a61e0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f20b006b3a0 tx=0x7f20a8004b50 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:49:34.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.549+0000 7f20b7e2d740 1 -- 172.21.15.90:0/344744244 shutdown_connections 2024-04-02T14:49:34.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.549+0000 7f20b7e2d740 1 --2- 172.21.15.90:0/344744244 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560d19e3b080 0x560d19e3d470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:34.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.549+0000 7f20b7e2d740 1 --2- 172.21.15.90:0/344744244 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560d19e31d10 0x560d19e32190 unknown :-1 s=CLOSED pgs=992 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:34.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.549+0000 7f20b7e2d740 1 --2- 172.21.15.90:0/344744244 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560d19e329f0 0x560d19e31740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:34.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.549+0000 7f20b7e2d740 1 -- 172.21.15.90:0/344744244 >> 172.21.15.90:0/344744244 conn(0x560d19df8a50 msgr2=0x560d19e2c760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:49:34.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.549+0000 7f20b7e2d740 1 -- 172.21.15.90:0/344744244 shutdown_connections 2024-04-02T14:49:34.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.549+0000 7f20b7e2d740 1 -- 172.21.15.90:0/344744244 wait complete. 2024-04-02T14:49:34.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.550+0000 7f20b7e2d740 1 -- start start 2024-04-02T14:49:34.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.550+0000 7f20b7e2d740 1 Processor -- start 2024-04-02T14:49:34.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.550+0000 7f20b7e2d740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560d19e31d10 0x560d19e380a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:34.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.550+0000 7f20b7e2d740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560d19e329f0 0x560d19e385e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:34.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.550+0000 7f20b7e2d740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560d19e3b080 0x560d19e38b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:34.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.550+0000 7f20b7e2d740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x560d19d926a0 con 0x560d19e31d10 2024-04-02T14:49:34.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.550+0000 7f20b7e2d740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x560d19d3e980 con 0x560d19e329f0 2024-04-02T14:49:34.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.550+0000 7f20b7e2d740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x560d19d51cb0 con 0x560d19e3b080 2024-04-02T14:49:34.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.551+0000 7f20b7e2d740 10 client.0 _ll_register_callbacks cb 0x560d19d516a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:49:34.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.551+0000 7f20a7e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560d19e329f0 0x560d19e385e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:34.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.551+0000 7f20a7e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560d19e329f0 0x560d19e385e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:50132/0 (socket says 172.21.15.90:50132) 2024-04-02T14:49:34.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.551+0000 7f20a7e00640 1 -- 172.21.15.90:0/2257633479 learned_addr learned my addr 172.21.15.90:0/2257633479 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:49:34.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.551+0000 7f20ade00640 1 --2- 172.21.15.90:0/2257633479 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560d19e31d10 0x560d19e380a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:34.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.551+0000 7f20a7e00640 1 -- 172.21.15.90:0/2257633479 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560d19e3b080 msgr2=0x560d19e38b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:49:34.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.551+0000 7f20a7e00640 1 --2- 172.21.15.90:0/2257633479 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560d19e3b080 0x560d19e38b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:34.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.551+0000 7f20a7e00640 1 -- 172.21.15.90:0/2257633479 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560d19e31d10 msgr2=0x560d19e380a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:49:34.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.551+0000 7f20a7e00640 1 --2- 172.21.15.90:0/2257633479 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560d19e31d10 0x560d19e380a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:34.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.551+0000 7f20a7e00640 1 -- 172.21.15.90:0/2257633479 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560d19b747d0 con 0x560d19e329f0 2024-04-02T14:49:34.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.551+0000 7f20a7e00640 1 --2- 172.21.15.90:0/2257633479 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560d19e329f0 0x560d19e385e0 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7f20a0012ca0 tx=0x7f20a00107a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=72431c55dee7f04f server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:49:34.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.551+0000 7f2093e00640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:49:34.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.552+0000 7f2093e00640 1 -- 172.21.15.90:0/2257633479 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20a0005220 con 0x560d19e329f0 2024-04-02T14:49:34.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.552+0000 7f2093e00640 1 -- 172.21.15.90:0/2257633479 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 8550+0+0 (secure 0 0 0) 0x7f20a00053c0 con 0x560d19e329f0 2024-04-02T14:49:34.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.552+0000 7f20b7e2d740 1 -- 172.21.15.90:0/2257633479 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f209803a400 con 0x560d19e329f0 2024-04-02T14:49:34.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.552+0000 7f2093e00640 1 -- 172.21.15.90:0/2257633479 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f20a00055c0 con 0x560d19e329f0 2024-04-02T14:49:34.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.552+0000 7f20b7e2d740 10 client.30023 fetch_fsmap learned FSMap version 27 2024-04-02T14:49:34.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.552+0000 7f20b7e2d740 1 -- 172.21.15.90:0/2257633479 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x560d19e49c70 con 0x560d19e329f0 2024-04-02T14:49:34.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.552+0000 7f2093e00640 1 -- 172.21.15.90:0/2257633479 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f209803a400 con 0x560d19e329f0 2024-04-02T14:49:34.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.552+0000 7f2093e00640 1 -- 172.21.15.90:0/2257633479 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f20a000eb50 con 0x560d19e329f0 2024-04-02T14:49:34.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.553+0000 7f20b7e2d740 10 client.30023 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:49:34.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.553+0000 7f20b7e2d740 10 client.30023 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-04-02T14:49:34.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.553+0000 7f20b7e2d740 20 client.30023 populate_metadata read hostname 'smithi090' 2024-04-02T14:49:34.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.553+0000 7f20b7e2d740 1 --2- 172.21.15.90:0/2257633479 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x560d19e4b1f0 0x560d19e4d690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:34.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.553+0000 7f20b7e2d740 4 client.30023 mds_command: new command op to 24547 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:49:34.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.553+0000 7f20b7e2d740 1 -- 172.21.15.90:0/2257633479 --> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560d19d56040 con 0x560d19e4b1f0 2024-04-02T14:49:34.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.553+0000 7f20ade00640 1 --2- 172.21.15.90:0/2257633479 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x560d19e4b1f0 0x560d19e4d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:34.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.553+0000 7f2093e00640 10 client.30023 ms_handle_connect on v2:172.21.15.177:6838/664895663 2024-04-02T14:49:34.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.553+0000 7f20ade00640 1 --2- 172.21.15.90:0/2257633479 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x560d19e4b1f0 0x560d19e4d690 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=95be9950d18f298f server_cookie=69cf3e88a2e9a1bc in_seq=0 out_seq=0 2024-04-02T14:49:34.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.555+0000 7f2093e00640 1 -- 172.21.15.90:0/2257633479 <== mds.2 v2:172.21.15.177:6838/664895663 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x560d19d56040 con 0x560d19e4b1f0 2024-04-02T14:49:34.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.555+0000 7f2093e00640 10 client.30023 handle_command_reply: tid=0 2024-04-02T14:49:34.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.555+0000 7f20b7e2d740 1 client.30023 shutdown 2024-04-02T14:49:34.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.557+0000 7f20b7e2d740 1 -- 172.21.15.90:0/2257633479 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560d19e329f0 msgr2=0x560d19e385e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:49:34.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.557+0000 7f20b7e2d740 1 --2- 172.21.15.90:0/2257633479 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560d19e329f0 0x560d19e385e0 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7f20a0012ca0 tx=0x7f20a00107a0 comp rx=0 tx=0).stop 2024-04-02T14:49:34.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.557+0000 7f20b7e2d740 1 -- 172.21.15.90:0/2257633479 shutdown_connections 2024-04-02T14:49:34.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.557+0000 7f20b7e2d740 1 --2- 172.21.15.90:0/2257633479 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x560d19e4b1f0 0x560d19e4d690 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:34.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.557+0000 7f20b7e2d740 1 --2- 172.21.15.90:0/2257633479 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560d19e3b080 0x560d19e38b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:34.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.557+0000 7f20b7e2d740 1 --2- 172.21.15.90:0/2257633479 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560d19e329f0 0x560d19e385e0 unknown :-1 s=CLOSED pgs=993 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:34.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.557+0000 7f20b7e2d740 1 --2- 172.21.15.90:0/2257633479 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560d19e31d10 0x560d19e380a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:34.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.557+0000 7f20b7e2d740 1 -- 172.21.15.90:0/2257633479 >> 172.21.15.90:0/2257633479 conn(0x560d19df8a50 msgr2=0x560d19e2e4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:49:34.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.557+0000 7f2093e00640 0 client.30023 ms_handle_reset on v2:172.21.15.177:6838/664895663 2024-04-02T14:49:34.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.558+0000 7f20b7e2d740 1 -- 172.21.15.90:0/2257633479 shutdown_connections 2024-04-02T14:49:34.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.558+0000 7f20b7e2d740 1 -- 172.21.15.90:0/2257633479 wait complete. 2024-04-02T14:49:34.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.558+0000 7f20b7e2d740 20 client.30023 trim_cache size 0 max 16384 2024-04-02T14:49:34.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.596+0000 7f20b7e2d740 1 Processor -- start 2024-04-02T14:49:34.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.596+0000 7f20b7e2d740 1 -- start start 2024-04-02T14:49:34.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.597+0000 7f20b7e2d740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560d19e31d10 0x560d19e3a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:34.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.597+0000 7f20b7e2d740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560d19e329f0 0x560d19e46780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:34.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.597+0000 7f20b7e2d740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560d19e3b080 0x560d19e37200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:34.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.597+0000 7f20b7e2d740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f20b01b3e70 con 0x560d19e31d10 2024-04-02T14:49:34.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.597+0000 7f20b7e2d740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f20b01b59b0 con 0x560d19e329f0 2024-04-02T14:49:34.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.597+0000 7f20b7e2d740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f20b01b4540 con 0x560d19e3b080 2024-04-02T14:49:34.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.598+0000 7f20a7e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560d19e329f0 0x560d19e46780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:34.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.598+0000 7f20a7e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560d19e329f0 0x560d19e46780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:50138/0 (socket says 172.21.15.90:50138) 2024-04-02T14:49:34.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.598+0000 7f20a7e00640 1 -- 172.21.15.90:0/3692843541 learned_addr learned my addr 172.21.15.90:0/3692843541 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:49:34.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.598+0000 7f20a7e00640 1 -- 172.21.15.90:0/3692843541 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560d19e3b080 msgr2=0x560d19e37200 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-02T14:49:34.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.598+0000 7f20a7e00640 1 --2- 172.21.15.90:0/3692843541 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560d19e3b080 0x560d19e37200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:34.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.598+0000 7f20a7e00640 1 -- 172.21.15.90:0/3692843541 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560d19e31d10 msgr2=0x560d19e3a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:49:34.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.598+0000 7f20a7e00640 1 --2- 172.21.15.90:0/3692843541 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560d19e31d10 0x560d19e3a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:34.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.598+0000 7f20a7e00640 1 -- 172.21.15.90:0/3692843541 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560d19e30d40 con 0x560d19e329f0 2024-04-02T14:49:34.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.599+0000 7f20a7e00640 1 --2- 172.21.15.90:0/3692843541 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560d19e329f0 0x560d19e46780 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7f20a0000c00 tx=0x7f20a0005760 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dba94ec2a241ff2d server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:49:34.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.600+0000 7f209ea00640 1 -- 172.21.15.90:0/3692843541 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20a00236b0 con 0x560d19e329f0 2024-04-02T14:49:34.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.600+0000 7f209ea00640 1 -- 172.21.15.90:0/3692843541 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f20a0023850 con 0x560d19e329f0 2024-04-02T14:49:34.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.600+0000 7f209ea00640 1 -- 172.21.15.90:0/3692843541 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20a000ec30 con 0x560d19e329f0 2024-04-02T14:49:34.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.601+0000 7f20b7e2d740 1 -- 172.21.15.90:0/3692843541 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560d19e329f0 msgr2=0x560d19e46780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:49:34.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.601+0000 7f20b7e2d740 1 --2- 172.21.15.90:0/3692843541 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560d19e329f0 0x560d19e46780 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7f20a0000c00 tx=0x7f20a0005760 comp rx=0 tx=0).stop 2024-04-02T14:49:34.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.601+0000 7f20b7e2d740 1 -- 172.21.15.90:0/3692843541 shutdown_connections 2024-04-02T14:49:34.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.601+0000 7f20b7e2d740 1 --2- 172.21.15.90:0/3692843541 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560d19e3b080 0x560d19e37200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:34.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.601+0000 7f20b7e2d740 1 --2- 172.21.15.90:0/3692843541 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560d19e329f0 0x560d19e46780 unknown :-1 s=CLOSED pgs=994 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:34.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.601+0000 7f20b7e2d740 1 --2- 172.21.15.90:0/3692843541 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560d19e31d10 0x560d19e3a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:34.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.601+0000 7f20b7e2d740 1 -- 172.21.15.90:0/3692843541 >> 172.21.15.90:0/3692843541 conn(0x560d19df8a50 msgr2=0x560d19e2dbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:49:34.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.601+0000 7f20b7e2d740 1 -- 172.21.15.90:0/3692843541 shutdown_connections 2024-04-02T14:49:34.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.602+0000 7f20b7e2d740 1 -- 172.21.15.90:0/3692843541 wait complete. 2024-04-02T14:49:34.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.602+0000 7f20b7e2d740 1 -- start start 2024-04-02T14:49:34.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.602+0000 7f20b7e2d740 1 Processor -- start 2024-04-02T14:49:34.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.603+0000 7f20b7e2d740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560d19e31d10 0x560d19e3e710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:34.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.603+0000 7f20b7e2d740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560d19e329f0 0x560d19e3ec50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:34.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.603+0000 7f20b7e2d740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560d19e3b080 0x560d19e3f190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:34.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.603+0000 7f20b7e2d740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f20b01b52e0 con 0x560d19e31d10 2024-04-02T14:49:34.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.603+0000 7f20b7e2d740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f20b01b4c10 con 0x560d19e329f0 2024-04-02T14:49:34.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.603+0000 7f20b7e2d740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f20b01b1e70 con 0x560d19e3b080 2024-04-02T14:49:34.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.603+0000 7f20b7e2d740 10 client.0 _ll_register_callbacks cb 0x7f20a0012830 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:49:34.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.604+0000 7f20ade00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560d19e31d10 0x560d19e3e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:34.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.604+0000 7f20ade00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560d19e31d10 0x560d19e3e710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:46248/0 (socket says 172.21.15.90:46248) 2024-04-02T14:49:34.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.604+0000 7f20ade00640 1 -- 172.21.15.90:0/1124137340 learned_addr learned my addr 172.21.15.90:0/1124137340 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:49:34.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.604+0000 7f20ae800640 1 --2- 172.21.15.90:0/1124137340 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560d19e3b080 0x560d19e3f190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:34.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.604+0000 7f20a7e00640 1 --2- 172.21.15.90:0/1124137340 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560d19e329f0 0x560d19e3ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:34.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.605+0000 7f20ae800640 1 -- 172.21.15.90:0/1124137340 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560d19e329f0 msgr2=0x560d19e3ec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:49:34.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.605+0000 7f20ae800640 1 --2- 172.21.15.90:0/1124137340 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560d19e329f0 0x560d19e3ec50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:34.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.605+0000 7f20ae800640 1 -- 172.21.15.90:0/1124137340 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560d19e31d10 msgr2=0x560d19e3e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:49:34.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.605+0000 7f20ae800640 1 --2- 172.21.15.90:0/1124137340 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560d19e31d10 0x560d19e3e710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:34.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.605+0000 7f20ae800640 1 -- 172.21.15.90:0/1124137340 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560d19ec1830 con 0x560d19e3b080 2024-04-02T14:49:34.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.605+0000 7f20a7e00640 1 --2- 172.21.15.90:0/1124137340 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560d19e329f0 0x560d19e3ec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:49:34.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.605+0000 7f20ae800640 1 --2- 172.21.15.90:0/1124137340 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560d19e3b080 0x560d19e3f190 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x560d19e27770 tx=0x7f20a8009650 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ca6a46abe4e5d1d6 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:49:34.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.605+0000 7f2093e00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:49:34.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.606+0000 7f2093e00640 1 -- 172.21.15.90:0/1124137340 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20a8026060 con 0x560d19e3b080 2024-04-02T14:49:34.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.606+0000 7f2093e00640 1 -- 172.21.15.90:0/1124137340 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 8550+0+0 (secure 0 0 0) 0x7f20a8021600 con 0x560d19e3b080 2024-04-02T14:49:34.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.606+0000 7f20b7e2d740 1 -- 172.21.15.90:0/1124137340 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f20a0023850 con 0x560d19e3b080 2024-04-02T14:49:34.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.607+0000 7f2093e00640 1 -- 172.21.15.90:0/1124137340 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f20a8021800 con 0x560d19e3b080 2024-04-02T14:49:34.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.607+0000 7f20b7e2d740 10 client.29824 fetch_fsmap learned FSMap version 27 2024-04-02T14:49:34.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.608+0000 7f20b7e2d740 1 -- 172.21.15.90:0/1124137340 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x560d19ed3ba0 con 0x560d19e3b080 2024-04-02T14:49:34.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.608+0000 7f2093e00640 1 -- 172.21.15.90:0/1124137340 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f20a0023850 con 0x560d19e3b080 2024-04-02T14:49:34.602 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:49:34 smithi092 ceph-mon[36497]: pgmap v459: 97 pgs: 97 active+clean; 4.2 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 5.1 MiB/s wr, 504 op/s 2024-04-02T14:49:34.602 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:49:34 smithi092 ceph-mon[36497]: from='client.? 172.21.15.90:0/1595934776' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-02T14:49:34.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.608+0000 7f2093e00640 1 -- 172.21.15.90:0/1124137340 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f20a80203a0 con 0x560d19e3b080 2024-04-02T14:49:34.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.609+0000 7f20b7e2d740 10 client.29824 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:49:34.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.609+0000 7f20b7e2d740 10 client.29824 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-04-02T14:49:34.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.609+0000 7f20b7e2d740 20 client.29824 populate_metadata read hostname 'smithi090' 2024-04-02T14:49:34.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.609+0000 7f20b7e2d740 1 --2- 172.21.15.90:0/1124137340 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x560d19ed4300 0x560d19ed6720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:34.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.609+0000 7f20b7e2d740 4 client.29824 mds_command: new command op to 24547 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-02T14:49:34.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.609+0000 7f20b7e2d740 1 -- 172.21.15.90:0/1124137340 --> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] -- command(tid 0: {"prefix": "damage ls"}) -- 0x560d19da3c30 con 0x560d19ed4300 2024-04-02T14:49:34.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.609+0000 7f20ade00640 1 --2- 172.21.15.90:0/1124137340 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x560d19ed4300 0x560d19ed6720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:34.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.610+0000 7f2093e00640 10 client.29824 ms_handle_connect on v2:172.21.15.177:6838/664895663 2024-04-02T14:49:34.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.610+0000 7f20ade00640 1 --2- 172.21.15.90:0/1124137340 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x560d19ed4300 0x560d19ed6720 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=823e756c1c26865c server_cookie=5401abe6a37400e5 in_seq=0 out_seq=0 2024-04-02T14:49:34.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.610+0000 7f2093e00640 1 -- 172.21.15.90:0/1124137340 <== mds.2 v2:172.21.15.177:6838/664895663 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x560d19da3c30 con 0x560d19ed4300 2024-04-02T14:49:34.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.610+0000 7f2093e00640 10 client.29824 handle_command_reply: tid=0 2024-04-02T14:49:34.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.610+0000 7f20b7e2d740 1 client.29824 shutdown 2024-04-02T14:49:34.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.611+0000 7f20b7e2d740 1 -- 172.21.15.90:0/1124137340 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560d19e3b080 msgr2=0x560d19e3f190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:49:34.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.611+0000 7f20b7e2d740 1 --2- 172.21.15.90:0/1124137340 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560d19e3b080 0x560d19e3f190 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x560d19e27770 tx=0x7f20a8009650 comp rx=0 tx=0).stop 2024-04-02T14:49:34.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.611+0000 7f20b7e2d740 1 -- 172.21.15.90:0/1124137340 shutdown_connections 2024-04-02T14:49:34.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.611+0000 7f20b7e2d740 1 --2- 172.21.15.90:0/1124137340 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x560d19ed4300 0x560d19ed6720 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:34.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.612+0000 7f20b7e2d740 1 --2- 172.21.15.90:0/1124137340 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560d19e3b080 0x560d19e3f190 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:34.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.612+0000 7f2093e00640 0 client.29824 ms_handle_reset on v2:172.21.15.177:6838/664895663 2024-04-02T14:49:34.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.612+0000 7f20b7e2d740 1 --2- 172.21.15.90:0/1124137340 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560d19e329f0 0x560d19e3ec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:34.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.612+0000 7f20b7e2d740 1 --2- 172.21.15.90:0/1124137340 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560d19e31d10 0x560d19e3e710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:34.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.612+0000 7f20b7e2d740 1 -- 172.21.15.90:0/1124137340 >> 172.21.15.90:0/1124137340 conn(0x560d19df8a50 msgr2=0x560d19e38040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:49:34.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.612+0000 7f20b7e2d740 1 -- 172.21.15.90:0/1124137340 shutdown_connections 2024-04-02T14:49:34.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.612+0000 7f20b7e2d740 1 -- 172.21.15.90:0/1124137340 wait complete. 2024-04-02T14:49:34.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.612+0000 7f20b7e2d740 20 client.29824 trim_cache size 0 max 16384 2024-04-02T14:49:34.607 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-02T14:49:34.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.613+0000 7f20b5000640 1 -- 172.21.15.90:0/2895500523 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f20940a3d20 msgr2=0x7f20940a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:49:34.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.613+0000 7f20b5000640 1 --2- 172.21.15.90:0/2895500523 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f20940a3d20 0x7f20940a61e0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f20b006b3a0 tx=0x7f20a8004b50 comp rx=0 tx=0).stop 2024-04-02T14:49:34.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.613+0000 7f20b5000640 1 -- 172.21.15.90:0/2895500523 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f20b0073ed0 msgr2=0x7f20b01a4630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:49:34.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.613+0000 7f20b5000640 1 --2- 172.21.15.90:0/2895500523 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f20b0073ed0 0x7f20b01a4630 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7f2098005ab0 tx=0x7f209802c7d0 comp rx=0 tx=0).stop 2024-04-02T14:49:34.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.614+0000 7f20b5000640 1 -- 172.21.15.90:0/2895500523 shutdown_connections 2024-04-02T14:49:34.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.614+0000 7f20b5000640 1 --2- 172.21.15.90:0/2895500523 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f20940a3d20 0x7f20940a61e0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:34.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.614+0000 7f20b5000640 1 --2- 172.21.15.90:0/2895500523 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f20b010e330 0x7f20b010d2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:34.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.614+0000 7f20b5000640 1 --2- 172.21.15.90:0/2895500523 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f20b00747f0 0x7f20b01a4b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:34.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.614+0000 7f20b5000640 1 --2- 172.21.15.90:0/2895500523 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f20b0073ed0 0x7f20b01a4630 unknown :-1 s=CLOSED pgs=991 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:34.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.614+0000 7f20b5000640 1 -- 172.21.15.90:0/2895500523 >> 172.21.15.90:0/2895500523 conn(0x7f20b0078f00 msgr2=0x7f20b00fe920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:49:34.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.614+0000 7f20b5000640 1 -- 172.21.15.90:0/2895500523 shutdown_connections 2024-04-02T14:49:34.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.614+0000 7f20b5000640 1 -- 172.21.15.90:0/2895500523 wait complete. 2024-04-02T14:49:34.640 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:49:34 smithi090 ceph-mon[31674]: pgmap v459: 97 pgs: 97 active+clean; 4.2 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 5.1 MiB/s wr, 504 op/s 2024-04-02T14:49:34.640 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:49:34 smithi090 ceph-mon[31674]: from='client.? 172.21.15.90:0/1595934776' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-02T14:49:34.646 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-02T14:49:34.678 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:49:34 smithi177 ceph-mon[36431]: pgmap v459: 97 pgs: 97 active+clean; 4.2 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 5.1 MiB/s wr, 504 op/s 2024-04-02T14:49:34.678 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:49:34 smithi177 ceph-mon[36431]: from='client.? 172.21.15.90:0/1595934776' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-02T14:49:34.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.903+0000 7fbbfd400640 1 -- 172.21.15.90:0/4037993972 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbbf8105020 msgr2=0x7fbbf810cbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:49:34.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.903+0000 7fbbfd400640 1 --2- 172.21.15.90:0/4037993972 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbbf8105020 0x7fbbf810cbf0 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7fbbf0005fd0 tx=0x7fbbf002cc30 comp rx=0 tx=0).stop 2024-04-02T14:49:34.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.904+0000 7fbbfd400640 1 -- 172.21.15.90:0/4037993972 shutdown_connections 2024-04-02T14:49:34.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.904+0000 7fbbfd400640 1 --2- 172.21.15.90:0/4037993972 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbbf8105020 0x7fbbf810cbf0 unknown :-1 s=CLOSED pgs=995 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:34.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.904+0000 7fbbfd400640 1 --2- 172.21.15.90:0/4037993972 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbbf8100900 0x7fbbf8104940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:34.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.904+0000 7fbbfd400640 1 --2- 172.21.15.90:0/4037993972 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbbf80fff50 0x7fbbf8100330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:34.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.904+0000 7fbbfd400640 1 -- 172.21.15.90:0/4037993972 >> 172.21.15.90:0/4037993972 conn(0x7fbbf8076b50 msgr2=0x7fbbf80fdee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:49:34.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.904+0000 7fbbfd400640 1 -- 172.21.15.90:0/4037993972 shutdown_connections 2024-04-02T14:49:34.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.904+0000 7fbbfd400640 1 -- 172.21.15.90:0/4037993972 wait complete. 2024-04-02T14:49:34.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.905+0000 7fbbfd400640 1 Processor -- start 2024-04-02T14:49:34.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.905+0000 7fbbfd400640 1 -- start start 2024-04-02T14:49:34.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.906+0000 7fbbfd400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbbf80fff50 0x7fbbf81982a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:34.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.906+0000 7fbbfd400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbbf8105020 0x7fbbf81987e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:34.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.906+0000 7fbbfd400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbbf819cb60 0x7fbbf819d010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:34.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.906+0000 7fbbfd400640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fbbf8111e60 con 0x7fbbf80fff50 2024-04-02T14:49:34.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.906+0000 7fbbfd400640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fbbf8111ce0 con 0x7fbbf819cb60 2024-04-02T14:49:34.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.906+0000 7fbbfd400640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fbbf8111fe0 con 0x7fbbf8105020 2024-04-02T14:49:34.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.906+0000 7fbbf6800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbbf80fff50 0x7fbbf81982a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:34.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.906+0000 7fbbefe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbbf8105020 0x7fbbf81987e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:34.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.906+0000 7fbbf7200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbbf819cb60 0x7fbbf819d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:34.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.906+0000 7fbbf7200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbbf819cb60 0x7fbbf819d010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:50170/0 (socket says 172.21.15.90:50170) 2024-04-02T14:49:34.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.906+0000 7fbbf7200640 1 -- 172.21.15.90:0/4129618447 learned_addr learned my addr 172.21.15.90:0/4129618447 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:49:34.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.906+0000 7fbbefe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbbf8105020 0x7fbbf81987e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:53218/0 (socket says 172.21.15.90:53218) 2024-04-02T14:49:34.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.907+0000 7fbbf7200640 1 -- 172.21.15.90:0/4129618447 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbbf8105020 msgr2=0x7fbbf81987e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:49:34.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.907+0000 7fbbf7200640 1 --2- 172.21.15.90:0/4129618447 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbbf8105020 0x7fbbf81987e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:34.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.907+0000 7fbbf7200640 1 -- 172.21.15.90:0/4129618447 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbbf80fff50 msgr2=0x7fbbf81982a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:49:34.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.907+0000 7fbbf7200640 1 --2- 172.21.15.90:0/4129618447 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbbf80fff50 0x7fbbf81982a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:34.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.907+0000 7fbbf7200640 1 -- 172.21.15.90:0/4129618447 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbbf819d650 con 0x7fbbf819cb60 2024-04-02T14:49:34.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.907+0000 7fbbf7200640 1 --2- 172.21.15.90:0/4129618447 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbbf819cb60 0x7fbbf819d010 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7fbbf000ab50 tx=0x7fbbf0007c80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:49:34.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.907+0000 7fbbef400640 1 -- 172.21.15.90:0/4129618447 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbf0044070 con 0x7fbbf819cb60 2024-04-02T14:49:34.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.907+0000 7fbbef400640 1 -- 172.21.15.90:0/4129618447 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbbf000a2e0 con 0x7fbbf819cb60 2024-04-02T14:49:34.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.907+0000 7fbbef400640 1 -- 172.21.15.90:0/4129618447 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbf00356f0 con 0x7fbbf819cb60 2024-04-02T14:49:34.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.907+0000 7fbbfd400640 1 -- 172.21.15.90:0/4129618447 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbbf819d8e0 con 0x7fbbf819cb60 2024-04-02T14:49:34.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.907+0000 7fbbfd400640 1 -- 172.21.15.90:0/4129618447 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbbf810d9d0 con 0x7fbbf819cb60 2024-04-02T14:49:34.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.915+0000 7fbbef400640 1 -- 172.21.15.90:0/4129618447 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fbbf0047070 con 0x7fbbf819cb60 2024-04-02T14:49:34.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.915+0000 7fbc00014740 1 Processor -- start 2024-04-02T14:49:34.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.916+0000 7fbc00014740 1 -- start start 2024-04-02T14:49:34.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.916+0000 7fbc00014740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5612cabc3200 0x5612cabc1f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:34.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.916+0000 7fbc00014740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5612cabc2520 0x5612cabc29a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:34.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.916+0000 7fbc00014740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5612cabcb890 0x5612cabcdc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:34.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.916+0000 7fbc00014740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5612caafa400 con 0x5612cabc3200 2024-04-02T14:49:34.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.916+0000 7fbc00014740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x5612caadef10 con 0x5612cabcb890 2024-04-02T14:49:34.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.916+0000 7fbc00014740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5612cab1e6f0 con 0x5612cabc2520 2024-04-02T14:49:34.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.916+0000 7fbbefe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5612cabc3200 0x5612cabc1f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:34.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.917+0000 7fbbefe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5612cabc3200 0x5612cabc1f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:46282/0 (socket says 172.21.15.90:46282) 2024-04-02T14:49:34.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.917+0000 7fbbefe00640 1 -- 172.21.15.90:0/2867512869 learned_addr learned my addr 172.21.15.90:0/2867512869 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:49:34.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.917+0000 7fbbf6800640 1 --2- 172.21.15.90:0/2867512869 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5612cabc2520 0x5612cabc29a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:34.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.917+0000 7fbbefe00640 1 --2- 172.21.15.90:0/2867512869 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5612cabcb890 0x5612cabcdc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:34.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.917+0000 7fbbef400640 1 --2- 172.21.15.90:0/4129618447 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fbbdc0a3da0 0x7fbbdc0a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:34.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.917+0000 7fbbf6800640 1 -- 172.21.15.90:0/2867512869 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5612cabcb890 msgr2=0x5612cabcdc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:49:34.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.917+0000 7fbbf6800640 1 --2- 172.21.15.90:0/2867512869 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5612cabcb890 0x5612cabcdc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:34.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.917+0000 7fbbf6800640 1 -- 172.21.15.90:0/2867512869 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5612cabc3200 msgr2=0x5612cabc1f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:49:34.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.917+0000 7fbbf6800640 1 --2- 172.21.15.90:0/2867512869 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5612cabc3200 0x5612cabc1f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:34.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.917+0000 7fbbf6800640 1 -- 172.21.15.90:0/2867512869 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5612ca8323f0 con 0x5612cabc2520 2024-04-02T14:49:34.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.917+0000 7fbbf6800640 1 --2- 172.21.15.90:0/2867512869 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5612cabc2520 0x5612cabc29a0 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7fbbe800abf0 tx=0x7fbbe800f790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9d748bd527411945 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:49:34.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.918+0000 7fbbef400640 1 -- 172.21.15.90:0/4129618447 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 8550+0+0 (secure 0 0 0) 0x7fbbf00f18a0 con 0x7fbbf819cb60 2024-04-02T14:49:34.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.918+0000 7fbbe7400640 1 -- 172.21.15.90:0/2867512869 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbe8005220 con 0x5612cabc2520 2024-04-02T14:49:34.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.918+0000 7fbbe7400640 1 -- 172.21.15.90:0/2867512869 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbbe8004c80 con 0x5612cabc2520 2024-04-02T14:49:34.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.918+0000 7fbbf7200640 1 --2- 172.21.15.90:0/4129618447 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fbbdc0a3da0 0x7fbbdc0a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:34.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.918+0000 7fbbe7400640 1 -- 172.21.15.90:0/2867512869 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbe80187e0 con 0x5612cabc2520 2024-04-02T14:49:34.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.919+0000 7fbc00014740 1 -- 172.21.15.90:0/2867512869 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5612cabc2520 msgr2=0x5612cabc29a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:49:34.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.919+0000 7fbc00014740 1 --2- 172.21.15.90:0/2867512869 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5612cabc2520 0x5612cabc29a0 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7fbbe800abf0 tx=0x7fbbe800f790 comp rx=0 tx=0).stop 2024-04-02T14:49:34.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.919+0000 7fbc00014740 1 -- 172.21.15.90:0/2867512869 shutdown_connections 2024-04-02T14:49:34.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.919+0000 7fbc00014740 1 --2- 172.21.15.90:0/2867512869 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5612cabcb890 0x5612cabcdc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:34.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.919+0000 7fbc00014740 1 --2- 172.21.15.90:0/2867512869 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5612cabc2520 0x5612cabc29a0 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:34.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.919+0000 7fbc00014740 1 --2- 172.21.15.90:0/2867512869 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5612cabc3200 0x5612cabc1f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:34.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.919+0000 7fbc00014740 1 -- 172.21.15.90:0/2867512869 >> 172.21.15.90:0/2867512869 conn(0x5612cab80a50 msgr2=0x5612cabbcf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:49:34.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.919+0000 7fbc00014740 1 -- 172.21.15.90:0/2867512869 shutdown_connections 2024-04-02T14:49:34.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.919+0000 7fbbf7200640 1 --2- 172.21.15.90:0/4129618447 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fbbdc0a3da0 0x7fbbdc0a6260 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fbbf00b9d30 tx=0x7fbbf0035560 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:49:34.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.919+0000 7fbc00014740 1 -- 172.21.15.90:0/2867512869 wait complete. 2024-04-02T14:49:34.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.920+0000 7fbc00014740 1 -- start start 2024-04-02T14:49:34.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.920+0000 7fbc00014740 1 Processor -- start 2024-04-02T14:49:34.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.920+0000 7fbc00014740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5612cabc2520 0x5612cabc8810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:34.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.920+0000 7fbc00014740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5612cabc3200 0x5612cabc8d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:34.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.920+0000 7fbc00014740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5612cabcb890 0x5612cabc9290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:34.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.920+0000 7fbc00014740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5612cab1a6a0 con 0x5612cabc3200 2024-04-02T14:49:34.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.920+0000 7fbc00014740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x5612caac6980 con 0x5612cabcb890 2024-04-02T14:49:34.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.920+0000 7fbc00014740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5612caad9cb0 con 0x5612cabc2520 2024-04-02T14:49:34.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.920+0000 7fbc00014740 10 client.0 _ll_register_callbacks cb 0x5612caad96a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:49:34.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.921+0000 7fbbefe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5612cabc2520 0x5612cabc8810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:34.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.921+0000 7fbbf6800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5612cabc3200 0x5612cabc8d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:34.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.921+0000 7fbbefe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5612cabc2520 0x5612cabc8810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:53242/0 (socket says 172.21.15.90:53242) 2024-04-02T14:49:34.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.921+0000 7fbbefe00640 1 -- 172.21.15.90:0/4273303176 learned_addr learned my addr 172.21.15.90:0/4273303176 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:49:34.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.921+0000 7fbbefe00640 1 --2- 172.21.15.90:0/4273303176 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5612cabcb890 0x5612cabc9290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:34.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.921+0000 7fbbf6800640 1 -- 172.21.15.90:0/4273303176 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5612cabc2520 msgr2=0x5612cabc8810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:49:34.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.921+0000 7fbbf6800640 1 --2- 172.21.15.90:0/4273303176 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5612cabc2520 0x5612cabc8810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:34.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.921+0000 7fbbf6800640 1 -- 172.21.15.90:0/4273303176 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5612cabcb890 msgr2=0x5612cabc9290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:49:34.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.921+0000 7fbbf6800640 1 --2- 172.21.15.90:0/4273303176 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5612cabcb890 0x5612cabc9290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:34.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.921+0000 7fbbf6800640 1 -- 172.21.15.90:0/4273303176 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5612ca8fc7d0 con 0x5612cabc3200 2024-04-02T14:49:34.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.921+0000 7fbbefe00640 1 --2- 172.21.15.90:0/4273303176 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5612cabc2520 0x5612cabc8810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:49:34.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.921+0000 7fbbf6800640 1 --2- 172.21.15.90:0/4273303176 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5612cabc3200 0x5612cabc8d50 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fbbe8000c00 tx=0x7fbbe8005200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27b3f8b5924b70fd server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:49:34.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.921+0000 7fbbdbe00640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-02T14:49:34.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.922+0000 7fbbdbe00640 1 -- 172.21.15.90:0/4273303176 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbe8027070 con 0x5612cabc3200 2024-04-02T14:49:34.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.922+0000 7fbbdbe00640 1 -- 172.21.15.90:0/4273303176 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 8550+0+0 (secure 0 0 0) 0x7fbbe8018520 con 0x5612cabc3200 2024-04-02T14:49:34.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.922+0000 7fbc00014740 1 -- 172.21.15.90:0/4273303176 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbbe8004c80 con 0x5612cabc3200 2024-04-02T14:49:34.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.922+0000 7fbbdbe00640 1 -- 172.21.15.90:0/4273303176 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fbbe8033050 con 0x5612cabc3200 2024-04-02T14:49:34.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.922+0000 7fbc00014740 10 client.18756 fetch_fsmap learned FSMap version 27 2024-04-02T14:49:34.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.922+0000 7fbc00014740 1 -- 172.21.15.90:0/4273303176 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5612cabda720 con 0x5612cabc3200 2024-04-02T14:49:34.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.922+0000 7fbbdbe00640 1 -- 172.21.15.90:0/4273303176 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbbe8004c80 con 0x5612cabc3200 2024-04-02T14:49:34.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.923+0000 7fbbdbe00640 1 -- 172.21.15.90:0/4273303176 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fbbe8023430 con 0x5612cabc3200 2024-04-02T14:49:34.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.923+0000 7fbc00014740 10 client.18756 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:49:34.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.923+0000 7fbc00014740 10 client.18756 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2024-04-02T14:49:34.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.923+0000 7fbc00014740 20 client.18756 populate_metadata read hostname 'smithi090' 2024-04-02T14:49:34.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.923+0000 7fbc00014740 1 --2- 172.21.15.90:0/4273303176 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x5612cabdbad0 0x5612cabddec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:34.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.923+0000 7fbc00014740 4 client.18756 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:49:34.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.923+0000 7fbc00014740 1 -- 172.21.15.90:0/4273303176 --> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5612caade040 con 0x5612cabdbad0 2024-04-02T14:49:34.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.924+0000 7fbbefe00640 1 --2- 172.21.15.90:0/4273303176 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x5612cabdbad0 0x5612cabddec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:34.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.927+0000 7fbbefe00640 1 --2- 172.21.15.90:0/4273303176 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x5612cabdbad0 0x5612cabddec0 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=a7bc479b5513bd0c server_cookie=10bafbca84e5bad9 in_seq=0 out_seq=0 2024-04-02T14:49:34.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.927+0000 7fbbdbe00640 10 client.18756 ms_handle_connect on v2:172.21.15.90:6838/1498915870 2024-04-02T14:49:34.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.929+0000 7fbbdbe00640 1 -- 172.21.15.90:0/4273303176 <== mds.1 v2:172.21.15.90:6838/1498915870 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5612caade040 con 0x5612cabdbad0 2024-04-02T14:49:34.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.929+0000 7fbbdbe00640 10 client.18756 handle_command_reply: tid=0 2024-04-02T14:49:34.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.929+0000 7fbc00014740 1 client.18756 shutdown 2024-04-02T14:49:34.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.929+0000 7fbc00014740 1 -- 172.21.15.90:0/4273303176 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5612cabc3200 msgr2=0x5612cabc8d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:49:34.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.929+0000 7fbc00014740 1 --2- 172.21.15.90:0/4273303176 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5612cabc3200 0x5612cabc8d50 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fbbe8000c00 tx=0x7fbbe8005200 comp rx=0 tx=0).stop 2024-04-02T14:49:34.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.929+0000 7fbc00014740 1 -- 172.21.15.90:0/4273303176 shutdown_connections 2024-04-02T14:49:34.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.929+0000 7fbc00014740 1 --2- 172.21.15.90:0/4273303176 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x5612cabdbad0 0x5612cabddec0 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:34.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.929+0000 7fbc00014740 1 --2- 172.21.15.90:0/4273303176 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5612cabcb890 0x5612cabc9290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:34.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.929+0000 7fbc00014740 1 --2- 172.21.15.90:0/4273303176 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5612cabc3200 0x5612cabc8d50 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:34.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.929+0000 7fbc00014740 1 --2- 172.21.15.90:0/4273303176 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5612cabc2520 0x5612cabc8810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:34.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.929+0000 7fbbdbe00640 0 client.18756 ms_handle_reset on v2:172.21.15.90:6838/1498915870 2024-04-02T14:49:34.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.929+0000 7fbc00014740 1 -- 172.21.15.90:0/4273303176 >> 172.21.15.90:0/4273303176 conn(0x5612cab80a50 msgr2=0x5612cabb38a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:49:34.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.930+0000 7fbc00014740 1 -- 172.21.15.90:0/4273303176 shutdown_connections 2024-04-02T14:49:34.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.930+0000 7fbc00014740 1 -- 172.21.15.90:0/4273303176 wait complete. 2024-04-02T14:49:34.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.930+0000 7fbc00014740 20 client.18756 trim_cache size 0 max 16384 2024-04-02T14:49:34.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.952+0000 7fbc00014740 1 Processor -- start 2024-04-02T14:49:34.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.952+0000 7fbc00014740 1 -- start start 2024-04-02T14:49:34.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.953+0000 7fbc00014740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5612cabc2520 0x5612cabcaf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:34.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.953+0000 7fbc00014740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5612cabc3200 0x5612cabd7360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:34.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.953+0000 7fbc00014740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5612cabcb890 0x5612cabb3720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:34.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.953+0000 7fbc00014740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fbbf81ab600 con 0x5612cabc3200 2024-04-02T14:49:34.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.953+0000 7fbc00014740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fbbf81ad140 con 0x5612cabc2520 2024-04-02T14:49:34.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.953+0000 7fbc00014740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fbbf81abcd0 con 0x5612cabcb890 2024-04-02T14:49:34.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.953+0000 7fbbf6800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5612cabc3200 0x5612cabd7360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:34.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.953+0000 7fbbefe00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5612cabc2520 0x5612cabcaf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:34.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.953+0000 7fbbf6800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5612cabc3200 0x5612cabd7360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:46312/0 (socket says 172.21.15.90:46312) 2024-04-02T14:49:34.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.953+0000 7fbbf6800640 1 -- 172.21.15.90:0/104912373 learned_addr learned my addr 172.21.15.90:0/104912373 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:49:34.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.953+0000 7fbbefe00640 1 --2- 172.21.15.90:0/104912373 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5612cabcb890 0x5612cabb3720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:34.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.954+0000 7fbbefe00640 1 -- 172.21.15.90:0/104912373 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5612cabcb890 msgr2=0x5612cabb3720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:49:34.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.954+0000 7fbbefe00640 1 --2- 172.21.15.90:0/104912373 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5612cabcb890 0x5612cabb3720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:34.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.954+0000 7fbbefe00640 1 -- 172.21.15.90:0/104912373 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5612cabc3200 msgr2=0x5612cabd7360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:49:34.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.954+0000 7fbbefe00640 1 --2- 172.21.15.90:0/104912373 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5612cabc3200 0x5612cabd7360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:34.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.954+0000 7fbbefe00640 1 -- 172.21.15.90:0/104912373 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5612cabc1550 con 0x5612cabc2520 2024-04-02T14:49:34.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.954+0000 7fbbf6800640 1 --2- 172.21.15.90:0/104912373 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5612cabc3200 0x5612cabd7360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:49:34.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.954+0000 7fbbefe00640 1 --2- 172.21.15.90:0/104912373 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5612cabc2520 0x5612cabcaf40 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x5612caac6a70 tx=0x7fbbe00046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b86d4acc3ce10efb server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:49:34.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.955+0000 7fbbe6a00640 1 -- 172.21.15.90:0/104912373 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbe0003ac0 con 0x5612cabc2520 2024-04-02T14:49:34.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.955+0000 7fbbe6a00640 1 -- 172.21.15.90:0/104912373 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbbe0005500 con 0x5612cabc2520 2024-04-02T14:49:34.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.955+0000 7fbbe6a00640 1 -- 172.21.15.90:0/104912373 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbe0004c70 con 0x5612cabc2520 2024-04-02T14:49:34.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.955+0000 7fbc00014740 1 -- 172.21.15.90:0/104912373 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5612cabc2520 msgr2=0x5612cabcaf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:49:34.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.955+0000 7fbc00014740 1 --2- 172.21.15.90:0/104912373 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5612cabc2520 0x5612cabcaf40 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x5612caac6a70 tx=0x7fbbe00046c0 comp rx=0 tx=0).stop 2024-04-02T14:49:34.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.956+0000 7fbc00014740 1 -- 172.21.15.90:0/104912373 shutdown_connections 2024-04-02T14:49:34.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.956+0000 7fbc00014740 1 --2- 172.21.15.90:0/104912373 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5612cabcb890 0x5612cabb3720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:34.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.956+0000 7fbc00014740 1 --2- 172.21.15.90:0/104912373 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5612cabc3200 0x5612cabd7360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:34.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.956+0000 7fbc00014740 1 --2- 172.21.15.90:0/104912373 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5612cabc2520 0x5612cabcaf40 unknown :-1 s=CLOSED pgs=997 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:34.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.956+0000 7fbc00014740 1 -- 172.21.15.90:0/104912373 >> 172.21.15.90:0/104912373 conn(0x5612cab80a50 msgr2=0x5612cabb3010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:49:34.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.956+0000 7fbc00014740 1 -- 172.21.15.90:0/104912373 shutdown_connections 2024-04-02T14:49:34.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.956+0000 7fbc00014740 1 -- 172.21.15.90:0/104912373 wait complete. 2024-04-02T14:49:34.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.956+0000 7fbc00014740 1 -- start start 2024-04-02T14:49:34.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.956+0000 7fbc00014740 1 Processor -- start 2024-04-02T14:49:34.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.957+0000 7fbc00014740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5612cabc2520 0x5612cabd3c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:34.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.957+0000 7fbc00014740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5612cabc3200 0x5612cabcc250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:34.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.957+0000 7fbc00014740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5612cabcb890 0x5612cabcc790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:34.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.957+0000 7fbc00014740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fbbf81aca70 con 0x5612cabcb890 2024-04-02T14:49:34.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.957+0000 7fbc00014740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fbbf81ac3a0 con 0x5612cabc3200 2024-04-02T14:49:34.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.957+0000 7fbc00014740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fbbf81a9600 con 0x5612cabc2520 2024-04-02T14:49:34.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.957+0000 7fbc00014740 10 client.0 _ll_register_callbacks cb 0x7fbbe800aa40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:49:34.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.957+0000 7fbbefe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5612cabc2520 0x5612cabd3c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:34.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.957+0000 7fbbf6800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5612cabc3200 0x5612cabcc250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:34.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.957+0000 7fbbf6800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5612cabc3200 0x5612cabcc250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:50202/0 (socket says 172.21.15.90:50202) 2024-04-02T14:49:34.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.957+0000 7fbbf6800640 1 -- 172.21.15.90:0/1730641361 learned_addr learned my addr 172.21.15.90:0/1730641361 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:49:34.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.957+0000 7fbbefe00640 1 --2- 172.21.15.90:0/1730641361 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5612cabcb890 0x5612cabcc790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:34.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.957+0000 7fbbf6800640 1 -- 172.21.15.90:0/1730641361 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5612cabc2520 msgr2=0x5612cabd3c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:49:34.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.957+0000 7fbbf6800640 1 --2- 172.21.15.90:0/1730641361 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5612cabc2520 0x5612cabd3c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:34.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.957+0000 7fbbf6800640 1 -- 172.21.15.90:0/1730641361 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5612cabcb890 msgr2=0x5612cabcc790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:49:34.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.957+0000 7fbbf6800640 1 --2- 172.21.15.90:0/1730641361 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5612cabcb890 0x5612cabcc790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:34.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.957+0000 7fbbf6800640 1 -- 172.21.15.90:0/1730641361 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5612cabccfc0 con 0x5612cabc3200 2024-04-02T14:49:34.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.957+0000 7fbbefe00640 1 --2- 172.21.15.90:0/1730641361 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5612cabcb890 0x5612cabcc790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:49:34.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.957+0000 7fbbf6800640 1 --2- 172.21.15.90:0/1730641361 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5612cabc3200 0x5612cabcc250 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7fbbe800f7d0 tx=0x7fbbe8005ed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3732eb88033f693b server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:49:34.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.957+0000 7fbbdbe00640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:49:34.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.958+0000 7fbbdbe00640 1 -- 172.21.15.90:0/1730641361 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbe803c060 con 0x5612cabc3200 2024-04-02T14:49:34.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.958+0000 7fbbdbe00640 1 -- 172.21.15.90:0/1730641361 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 8550+0+0 (secure 0 0 0) 0x7fbbe800a630 con 0x5612cabc3200 2024-04-02T14:49:34.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.958+0000 7fbc00014740 1 -- 172.21.15.90:0/1730641361 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbbe0005500 con 0x5612cabc3200 2024-04-02T14:49:34.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.958+0000 7fbbdbe00640 1 -- 172.21.15.90:0/1730641361 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fbbe8027040 con 0x5612cabc3200 2024-04-02T14:49:34.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.958+0000 7fbc00014740 10 client.30047 fetch_fsmap learned FSMap version 27 2024-04-02T14:49:34.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.958+0000 7fbc00014740 1 -- 172.21.15.90:0/1730641361 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5612cac41d60 con 0x5612cabc3200 2024-04-02T14:49:34.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.958+0000 7fbbdbe00640 1 -- 172.21.15.90:0/1730641361 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbbe0005500 con 0x5612cabc3200 2024-04-02T14:49:34.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.958+0000 7fbbdbe00640 1 -- 172.21.15.90:0/1730641361 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fbbe802c990 con 0x5612cabc3200 2024-04-02T14:49:34.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.958+0000 7fbc00014740 10 client.30047 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:49:34.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.958+0000 7fbc00014740 10 client.30047 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2024-04-02T14:49:34.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.958+0000 7fbc00014740 20 client.30047 populate_metadata read hostname 'smithi090' 2024-04-02T14:49:34.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.958+0000 7fbc00014740 1 --2- 172.21.15.90:0/1730641361 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x5612cac422f0 0x5612cac44710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:34.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.958+0000 7fbc00014740 4 client.30047 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-02T14:49:34.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.958+0000 7fbc00014740 1 -- 172.21.15.90:0/1730641361 --> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5612cab2bc30 con 0x5612cac422f0 2024-04-02T14:49:34.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.959+0000 7fbbefe00640 1 --2- 172.21.15.90:0/1730641361 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x5612cac422f0 0x5612cac44710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:34.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.960+0000 7fbbefe00640 1 --2- 172.21.15.90:0/1730641361 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x5612cac422f0 0x5612cac44710 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=9e5cc896c179aea2 server_cookie=3a6d11d533a82772 in_seq=0 out_seq=0 2024-04-02T14:49:34.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.961+0000 7fbbdbe00640 10 client.30047 ms_handle_connect on v2:172.21.15.90:6838/1498915870 2024-04-02T14:49:34.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.961+0000 7fbbdbe00640 1 -- 172.21.15.90:0/1730641361 <== mds.1 v2:172.21.15.90:6838/1498915870 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5612cab2bc30 con 0x5612cac422f0 2024-04-02T14:49:34.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.962+0000 7fbbdbe00640 10 client.30047 handle_command_reply: tid=0 2024-04-02T14:49:34.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.962+0000 7fbc00014740 1 client.30047 shutdown 2024-04-02T14:49:34.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.962+0000 7fbc00014740 1 -- 172.21.15.90:0/1730641361 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5612cabc3200 msgr2=0x5612cabcc250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:49:34.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.962+0000 7fbc00014740 1 --2- 172.21.15.90:0/1730641361 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5612cabc3200 0x5612cabcc250 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7fbbe800f7d0 tx=0x7fbbe8005ed0 comp rx=0 tx=0).stop 2024-04-02T14:49:34.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.963+0000 7fbc00014740 1 -- 172.21.15.90:0/1730641361 shutdown_connections 2024-04-02T14:49:34.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.963+0000 7fbc00014740 1 --2- 172.21.15.90:0/1730641361 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x5612cac422f0 0x5612cac44710 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:34.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.963+0000 7fbc00014740 1 --2- 172.21.15.90:0/1730641361 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5612cabcb890 0x5612cabcc790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:34.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.963+0000 7fbc00014740 1 --2- 172.21.15.90:0/1730641361 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5612cabc3200 0x5612cabcc250 unknown :-1 s=CLOSED pgs=998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:34.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.963+0000 7fbc00014740 1 --2- 172.21.15.90:0/1730641361 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5612cabc2520 0x5612cabd3c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:34.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.963+0000 7fbc00014740 1 -- 172.21.15.90:0/1730641361 >> 172.21.15.90:0/1730641361 conn(0x5612cab80a50 msgr2=0x5612cabb38a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:49:34.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.963+0000 7fbbdbe00640 0 client.30047 ms_handle_reset on v2:172.21.15.90:6838/1498915870 2024-04-02T14:49:34.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.964+0000 7fbc00014740 1 -- 172.21.15.90:0/1730641361 shutdown_connections 2024-04-02T14:49:34.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.964+0000 7fbc00014740 1 -- 172.21.15.90:0/1730641361 wait complete. 2024-04-02T14:49:34.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.964+0000 7fbc00014740 20 client.30047 trim_cache size 0 max 16384 2024-04-02T14:49:34.961 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-02T14:49:34.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.965+0000 7fbbfd400640 1 -- 172.21.15.90:0/4129618447 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fbbdc0a3da0 msgr2=0x7fbbdc0a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:49:34.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.965+0000 7fbbfd400640 1 --2- 172.21.15.90:0/4129618447 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fbbdc0a3da0 0x7fbbdc0a6260 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fbbf00b9d30 tx=0x7fbbf0035560 comp rx=0 tx=0).stop 2024-04-02T14:49:34.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.965+0000 7fbbfd400640 1 -- 172.21.15.90:0/4129618447 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbbf819cb60 msgr2=0x7fbbf819d010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:49:34.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.965+0000 7fbbfd400640 1 --2- 172.21.15.90:0/4129618447 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbbf819cb60 0x7fbbf819d010 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7fbbf000ab50 tx=0x7fbbf0007c80 comp rx=0 tx=0).stop 2024-04-02T14:49:34.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.966+0000 7fbbfd400640 1 -- 172.21.15.90:0/4129618447 shutdown_connections 2024-04-02T14:49:34.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.966+0000 7fbbfd400640 1 --2- 172.21.15.90:0/4129618447 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fbbdc0a3da0 0x7fbbdc0a6260 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:34.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.966+0000 7fbbfd400640 1 --2- 172.21.15.90:0/4129618447 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbbf819cb60 0x7fbbf819d010 unknown :-1 s=CLOSED pgs=996 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:34.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.966+0000 7fbbfd400640 1 --2- 172.21.15.90:0/4129618447 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbbf8105020 0x7fbbf81987e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:34.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.966+0000 7fbbfd400640 1 --2- 172.21.15.90:0/4129618447 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbbf80fff50 0x7fbbf81982a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:34.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.966+0000 7fbbfd400640 1 -- 172.21.15.90:0/4129618447 >> 172.21.15.90:0/4129618447 conn(0x7fbbf8076b50 msgr2=0x7fbbf8103020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:49:34.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.966+0000 7fbbfd400640 1 -- 172.21.15.90:0/4129618447 shutdown_connections 2024-04-02T14:49:34.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:34.966+0000 7fbbfd400640 1 -- 172.21.15.90:0/4129618447 wait complete. 2024-04-02T14:49:34.994 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-02T14:49:35.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.287+0000 7fdc2be00640 1 -- 172.21.15.90:0/3549721261 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdc24108fe0 msgr2=0x7fdc24110bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:49:35.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.287+0000 7fdc2be00640 1 --2- 172.21.15.90:0/3549721261 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdc24108fe0 0x7fdc24110bb0 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7fdc1c009ca0 tx=0x7fdc1c02c120 comp rx=0 tx=0).stop 2024-04-02T14:49:35.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.287+0000 7fdc2be00640 1 -- 172.21.15.90:0/3549721261 shutdown_connections 2024-04-02T14:49:35.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.287+0000 7fdc2be00640 1 --2- 172.21.15.90:0/3549721261 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdc24108fe0 0x7fdc24110bb0 unknown :-1 s=CLOSED pgs=999 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:35.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.287+0000 7fdc2be00640 1 --2- 172.21.15.90:0/3549721261 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdc241048c0 0x7fdc24108900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:35.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.287+0000 7fdc2be00640 1 --2- 172.21.15.90:0/3549721261 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdc24103f10 0x7fdc241042f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:35.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.287+0000 7fdc2be00640 1 -- 172.21.15.90:0/3549721261 >> 172.21.15.90:0/3549721261 conn(0x7fdc240ff7b0 msgr2=0x7fdc24101bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:49:35.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.288+0000 7fdc2be00640 1 -- 172.21.15.90:0/3549721261 shutdown_connections 2024-04-02T14:49:35.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.288+0000 7fdc2be00640 1 -- 172.21.15.90:0/3549721261 wait complete. 2024-04-02T14:49:35.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.289+0000 7fdc2be00640 1 Processor -- start 2024-04-02T14:49:35.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.289+0000 7fdc2be00640 1 -- start start 2024-04-02T14:49:35.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.289+0000 7fdc2be00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdc24103f10 0x7fdc2419a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:35.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.289+0000 7fdc2be00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdc24108fe0 0x7fdc2419acd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:35.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.290+0000 7fdc28a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdc24108fe0 0x7fdc2419acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:35.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.290+0000 7fdc2be00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdc2419f000 0x7fdc2419f4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:35.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.290+0000 7fdc2be00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fdc24114190 con 0x7fdc2419f000 2024-04-02T14:49:35.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.290+0000 7fdc2be00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fdc24114010 con 0x7fdc24108fe0 2024-04-02T14:49:35.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.290+0000 7fdc28a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdc24108fe0 0x7fdc2419acd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:50212/0 (socket says 172.21.15.90:50212) 2024-04-02T14:49:35.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.290+0000 7fdc28a00640 1 -- 172.21.15.90:0/210546496 learned_addr learned my addr 172.21.15.90:0/210546496 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:49:35.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.290+0000 7fdc29400640 1 --2- 172.21.15.90:0/210546496 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdc24103f10 0x7fdc2419a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:35.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.290+0000 7fdc2be00640 1 -- 172.21.15.90:0/210546496 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fdc24114310 con 0x7fdc24103f10 2024-04-02T14:49:35.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.291+0000 7fdc29e00640 1 --2- 172.21.15.90:0/210546496 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdc2419f000 0x7fdc2419f4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:35.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.291+0000 7fdc28a00640 1 -- 172.21.15.90:0/210546496 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdc24103f10 msgr2=0x7fdc2419a790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:49:35.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.291+0000 7fdc28a00640 1 --2- 172.21.15.90:0/210546496 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdc24103f10 0x7fdc2419a790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:35.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.291+0000 7fdc28a00640 1 -- 172.21.15.90:0/210546496 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdc2419f000 msgr2=0x7fdc2419f4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:49:35.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.291+0000 7fdc28a00640 1 --2- 172.21.15.90:0/210546496 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdc2419f000 0x7fdc2419f4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:35.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.291+0000 7fdc28a00640 1 -- 172.21.15.90:0/210546496 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc2419faf0 con 0x7fdc24108fe0 2024-04-02T14:49:35.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.292+0000 7fdc28a00640 1 --2- 172.21.15.90:0/210546496 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdc24108fe0 0x7fdc2419acd0 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7fdc1000b450 tx=0x7fdc1000b910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:49:35.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.292+0000 7fdc22000640 1 -- 172.21.15.90:0/210546496 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc10015020 con 0x7fdc24108fe0 2024-04-02T14:49:35.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.292+0000 7fdc22000640 1 -- 172.21.15.90:0/210546496 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdc10011d70 con 0x7fdc24108fe0 2024-04-02T14:49:35.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.292+0000 7fdc22000640 1 -- 172.21.15.90:0/210546496 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc1000f340 con 0x7fdc24108fe0 2024-04-02T14:49:35.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.292+0000 7fdc2be00640 1 -- 172.21.15.90:0/210546496 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc2419fde0 con 0x7fdc24108fe0 2024-04-02T14:49:35.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.292+0000 7fdc2be00640 1 -- 172.21.15.90:0/210546496 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc241a7580 con 0x7fdc24108fe0 2024-04-02T14:49:35.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.296+0000 7fdc22000640 1 -- 172.21.15.90:0/210546496 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fdc10018020 con 0x7fdc24108fe0 2024-04-02T14:49:35.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.297+0000 7fdc22000640 1 --2- 172.21.15.90:0/210546496 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fdc0c0a3e60 0x7fdc0c0a6320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:35.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.298+0000 7fdc22000640 1 -- 172.21.15.90:0/210546496 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 8550+0+0 (secure 0 0 0) 0x7fdc100d00e0 con 0x7fdc24108fe0 2024-04-02T14:49:35.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.298+0000 7fdc29400640 1 --2- 172.21.15.90:0/210546496 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fdc0c0a3e60 0x7fdc0c0a6320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:35.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.298+0000 7fdc2e9f2740 1 Processor -- start 2024-04-02T14:49:35.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.298+0000 7fdc2e9f2740 1 -- start start 2024-04-02T14:49:35.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.298+0000 7fdc29400640 1 --2- 172.21.15.90:0/210546496 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fdc0c0a3e60 0x7fdc0c0a6320 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fdc18006fd0 tx=0x7fdc180090e0 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:49:35.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.299+0000 7fdc2e9f2740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b6f1a829f0 0x55b6f1a81740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:35.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.299+0000 7fdc2e9f2740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b6f1a81d10 0x55b6f1a82190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:35.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.299+0000 7fdc2e9f2740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b6f1a8b080 0x55b6f1a8d470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:35.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.299+0000 7fdc2e9f2740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55b6f19c2400 con 0x55b6f1a829f0 2024-04-02T14:49:35.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.299+0000 7fdc2e9f2740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55b6f19a6f10 con 0x55b6f1a8b080 2024-04-02T14:49:35.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.299+0000 7fdc2e9f2740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55b6f19e66f0 con 0x55b6f1a81d10 2024-04-02T14:49:35.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.299+0000 7fdc28a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b6f1a81d10 0x55b6f1a82190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:35.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.299+0000 7fdc29400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b6f1a829f0 0x55b6f1a81740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:35.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.299+0000 7fdc29e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b6f1a8b080 0x55b6f1a8d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:35.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.299+0000 7fdc28a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b6f1a81d10 0x55b6f1a82190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:53292/0 (socket says 172.21.15.90:53292) 2024-04-02T14:49:35.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.299+0000 7fdc28a00640 1 -- 172.21.15.90:0/2543305354 learned_addr learned my addr 172.21.15.90:0/2543305354 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:49:35.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.299+0000 7fdc28a00640 1 -- 172.21.15.90:0/2543305354 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b6f1a8b080 msgr2=0x55b6f1a8d470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:49:35.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.299+0000 7fdc28a00640 1 --2- 172.21.15.90:0/2543305354 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b6f1a8b080 0x55b6f1a8d470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:35.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.299+0000 7fdc28a00640 1 -- 172.21.15.90:0/2543305354 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b6f1a829f0 msgr2=0x55b6f1a81740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:49:35.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.299+0000 7fdc28a00640 1 --2- 172.21.15.90:0/2543305354 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b6f1a829f0 0x55b6f1a81740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:35.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.299+0000 7fdc28a00640 1 -- 172.21.15.90:0/2543305354 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b6f16fa3f0 con 0x55b6f1a81d10 2024-04-02T14:49:35.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.299+0000 7fdc29e00640 1 --2- 172.21.15.90:0/2543305354 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b6f1a8b080 0x55b6f1a8d470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:49:35.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.300+0000 7fdc28a00640 1 --2- 172.21.15.90:0/2543305354 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b6f1a81d10 0x55b6f1a82190 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7fdc10098570 tx=0x7fdc10010810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a969dbd5a41a872e server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:49:35.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.300+0000 7fdc16600640 1 -- 172.21.15.90:0/2543305354 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc100b5070 con 0x55b6f1a81d10 2024-04-02T14:49:35.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.300+0000 7fdc16600640 1 -- 172.21.15.90:0/2543305354 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdc100a04c0 con 0x55b6f1a81d10 2024-04-02T14:49:35.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.300+0000 7fdc16600640 1 -- 172.21.15.90:0/2543305354 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc10098910 con 0x55b6f1a81d10 2024-04-02T14:49:35.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.301+0000 7fdc2e9f2740 1 -- 172.21.15.90:0/2543305354 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b6f1a81d10 msgr2=0x55b6f1a82190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:49:35.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.301+0000 7fdc2e9f2740 1 --2- 172.21.15.90:0/2543305354 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b6f1a81d10 0x55b6f1a82190 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7fdc10098570 tx=0x7fdc10010810 comp rx=0 tx=0).stop 2024-04-02T14:49:35.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.301+0000 7fdc2e9f2740 1 -- 172.21.15.90:0/2543305354 shutdown_connections 2024-04-02T14:49:35.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.301+0000 7fdc2e9f2740 1 --2- 172.21.15.90:0/2543305354 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b6f1a8b080 0x55b6f1a8d470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:35.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.301+0000 7fdc2e9f2740 1 --2- 172.21.15.90:0/2543305354 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b6f1a81d10 0x55b6f1a82190 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:35.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.301+0000 7fdc2e9f2740 1 --2- 172.21.15.90:0/2543305354 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b6f1a829f0 0x55b6f1a81740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:35.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.301+0000 7fdc2e9f2740 1 -- 172.21.15.90:0/2543305354 >> 172.21.15.90:0/2543305354 conn(0x55b6f1a48a50 msgr2=0x55b6f1a7c760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:49:35.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.302+0000 7fdc2e9f2740 1 -- 172.21.15.90:0/2543305354 shutdown_connections 2024-04-02T14:49:35.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.302+0000 7fdc2e9f2740 1 -- 172.21.15.90:0/2543305354 wait complete. 2024-04-02T14:49:35.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.302+0000 7fdc2e9f2740 1 -- start start 2024-04-02T14:49:35.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.303+0000 7fdc2e9f2740 1 Processor -- start 2024-04-02T14:49:35.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.303+0000 7fdc2e9f2740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b6f1a81d10 0x55b6f1a88100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:35.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.303+0000 7fdc2e9f2740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b6f1a829f0 0x55b6f1a88640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:35.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.303+0000 7fdc2e9f2740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b6f1a8b080 0x55b6f1a88b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:35.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.303+0000 7fdc2e9f2740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55b6f19e26a0 con 0x55b6f1a8b080 2024-04-02T14:49:35.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.303+0000 7fdc2e9f2740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55b6f198e980 con 0x55b6f1a81d10 2024-04-02T14:49:35.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.303+0000 7fdc2e9f2740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55b6f19a1cb0 con 0x55b6f1a829f0 2024-04-02T14:49:35.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.303+0000 7fdc2e9f2740 10 client.0 _ll_register_callbacks cb 0x55b6f19a16a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:49:35.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.304+0000 7fdc28a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b6f1a829f0 0x55b6f1a88640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:35.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.304+0000 7fdc28a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b6f1a829f0 0x55b6f1a88640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:53300/0 (socket says 172.21.15.90:53300) 2024-04-02T14:49:35.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.304+0000 7fdc29e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b6f1a8b080 0x55b6f1a88b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:35.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.304+0000 7fdc28a00640 1 -- 172.21.15.90:0/392360463 learned_addr learned my addr 172.21.15.90:0/392360463 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:49:35.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.304+0000 7fdc29400640 1 --2- 172.21.15.90:0/392360463 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b6f1a81d10 0x55b6f1a88100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:35.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.304+0000 7fdc28a00640 1 -- 172.21.15.90:0/392360463 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b6f1a81d10 msgr2=0x55b6f1a88100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:49:35.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.304+0000 7fdc28a00640 1 --2- 172.21.15.90:0/392360463 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b6f1a81d10 0x55b6f1a88100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:35.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.304+0000 7fdc28a00640 1 -- 172.21.15.90:0/392360463 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b6f1a8b080 msgr2=0x55b6f1a88b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:49:35.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.304+0000 7fdc28a00640 1 --2- 172.21.15.90:0/392360463 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b6f1a8b080 0x55b6f1a88b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:35.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.304+0000 7fdc28a00640 1 -- 172.21.15.90:0/392360463 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b6f17c47d0 con 0x55b6f1a829f0 2024-04-02T14:49:35.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.304+0000 7fdc28a00640 1 --2- 172.21.15.90:0/392360463 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b6f1a829f0 0x55b6f1a88640 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7fdc10009fd0 tx=0x7fdc10098780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d61c338452871dea server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:49:35.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.304+0000 7fdc0aa00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:49:35.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.304+0000 7fdc0aa00640 1 -- 172.21.15.90:0/392360463 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc1007f060 con 0x55b6f1a829f0 2024-04-02T14:49:35.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.304+0000 7fdc0aa00640 1 -- 172.21.15.90:0/392360463 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 8550+0+0 (secure 0 0 0) 0x7fdc100a0170 con 0x55b6f1a829f0 2024-04-02T14:49:35.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.304+0000 7fdc2e9f2740 1 -- 172.21.15.90:0/392360463 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdc100a04c0 con 0x55b6f1a829f0 2024-04-02T14:49:35.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.305+0000 7fdc0aa00640 1 -- 172.21.15.90:0/392360463 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fdc100bb070 con 0x55b6f1a829f0 2024-04-02T14:49:35.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.305+0000 7fdc2e9f2740 10 client.29848 fetch_fsmap learned FSMap version 27 2024-04-02T14:49:35.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.305+0000 7fdc2e9f2740 1 -- 172.21.15.90:0/392360463 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55b6f1aaaf60 con 0x55b6f1a829f0 2024-04-02T14:49:35.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.305+0000 7fdc0aa00640 1 -- 172.21.15.90:0/392360463 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdc100a04c0 con 0x55b6f1a829f0 2024-04-02T14:49:35.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.306+0000 7fdc0aa00640 1 -- 172.21.15.90:0/392360463 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fdc1009b6f0 con 0x55b6f1a829f0 2024-04-02T14:49:35.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.306+0000 7fdc2e9f2740 10 client.29848 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:49:35.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.306+0000 7fdc2e9f2740 10 client.29848 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:49:35.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.306+0000 7fdc2e9f2740 20 client.29848 populate_metadata read hostname 'smithi090' 2024-04-02T14:49:35.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.306+0000 7fdc2e9f2740 1 --2- 172.21.15.90:0/392360463 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55b6f1aac2b0 0x55b6f1aae6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:35.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.306+0000 7fdc2e9f2740 4 client.29848 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:49:35.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.306+0000 7fdc2e9f2740 1 -- 172.21.15.90:0/392360463 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b6f19a6040 con 0x55b6f1aac2b0 2024-04-02T14:49:35.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.306+0000 7fdc29400640 1 --2- 172.21.15.90:0/392360463 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55b6f1aac2b0 0x55b6f1aae6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:35.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.307+0000 7fdc29400640 1 --2- 172.21.15.90:0/392360463 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55b6f1aac2b0 0x55b6f1aae6a0 crc :-1 s=READY pgs=372 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a37dd2a4a52bc5f1 server_cookie=94ed637906881048 in_seq=0 out_seq=0 2024-04-02T14:49:35.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.307+0000 7fdc0aa00640 10 client.29848 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:49:35.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.308+0000 7fdc0aa00640 1 -- 172.21.15.90:0/392360463 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b6f19a6040 con 0x55b6f1aac2b0 2024-04-02T14:49:35.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.308+0000 7fdc0aa00640 10 client.29848 handle_command_reply: tid=0 2024-04-02T14:49:35.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.308+0000 7fdc2e9f2740 1 client.29848 shutdown 2024-04-02T14:49:35.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.308+0000 7fdc2e9f2740 1 -- 172.21.15.90:0/392360463 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b6f1a829f0 msgr2=0x55b6f1a88640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:49:35.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.308+0000 7fdc2e9f2740 1 --2- 172.21.15.90:0/392360463 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b6f1a829f0 0x55b6f1a88640 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7fdc10009fd0 tx=0x7fdc10098780 comp rx=0 tx=0).stop 2024-04-02T14:49:35.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.308+0000 7fdc2e9f2740 1 -- 172.21.15.90:0/392360463 shutdown_connections 2024-04-02T14:49:35.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.308+0000 7fdc2e9f2740 1 --2- 172.21.15.90:0/392360463 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55b6f1aac2b0 0x55b6f1aae6a0 crc :-1 s=READY pgs=372 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:35.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.309+0000 7fdc2e9f2740 1 --2- 172.21.15.90:0/392360463 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b6f1a8b080 0x55b6f1a88b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:35.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.309+0000 7fdc2e9f2740 1 --2- 172.21.15.90:0/392360463 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b6f1a829f0 0x55b6f1a88640 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:35.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.309+0000 7fdc2e9f2740 1 --2- 172.21.15.90:0/392360463 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b6f1a81d10 0x55b6f1a88100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:35.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.309+0000 7fdc2e9f2740 1 -- 172.21.15.90:0/392360463 >> 172.21.15.90:0/392360463 conn(0x55b6f1a48a50 msgr2=0x55b6f1a7e4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:49:35.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.309+0000 7fdc0aa00640 0 client.29848 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:49:35.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.309+0000 7fdc2e9f2740 1 -- 172.21.15.90:0/392360463 shutdown_connections 2024-04-02T14:49:35.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.309+0000 7fdc2e9f2740 1 -- 172.21.15.90:0/392360463 wait complete. 2024-04-02T14:49:35.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.309+0000 7fdc2e9f2740 20 client.29848 trim_cache size 0 max 16384 2024-04-02T14:49:35.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.335+0000 7fdc2e9f2740 1 Processor -- start 2024-04-02T14:49:35.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.336+0000 7fdc2e9f2740 1 -- start start 2024-04-02T14:49:35.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.336+0000 7fdc2e9f2740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b6f1a81d10 0x55b6f1a8a830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:35.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.336+0000 7fdc2e9f2740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b6f1a829f0 0x55b6f1aa76c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:35.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.336+0000 7fdc2e9f2740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b6f1a8b080 0x55b6f1a8c470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:35.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.336+0000 7fdc2e9f2740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fdc2406f4f0 con 0x55b6f1a829f0 2024-04-02T14:49:35.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.336+0000 7fdc2e9f2740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fdc24071030 con 0x55b6f1a8b080 2024-04-02T14:49:35.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.336+0000 7fdc2e9f2740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fdc2406fbc0 con 0x55b6f1a81d10 2024-04-02T14:49:35.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.336+0000 7fdc29400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b6f1a81d10 0x55b6f1a8a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:35.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.336+0000 7fdc28a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b6f1a829f0 0x55b6f1aa76c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:35.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.336+0000 7fdc29400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b6f1a81d10 0x55b6f1a8a830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:53310/0 (socket says 172.21.15.90:53310) 2024-04-02T14:49:35.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.336+0000 7fdc29400640 1 -- 172.21.15.90:0/2660384831 learned_addr learned my addr 172.21.15.90:0/2660384831 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:49:35.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.336+0000 7fdc29e00640 1 --2- 172.21.15.90:0/2660384831 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b6f1a8b080 0x55b6f1a8c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:35.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.336+0000 7fdc29400640 1 -- 172.21.15.90:0/2660384831 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b6f1a8b080 msgr2=0x55b6f1a8c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:49:35.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.337+0000 7fdc29400640 1 --2- 172.21.15.90:0/2660384831 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b6f1a8b080 0x55b6f1a8c470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:35.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.337+0000 7fdc29400640 1 -- 172.21.15.90:0/2660384831 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b6f1a829f0 msgr2=0x55b6f1aa76c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:49:35.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.337+0000 7fdc29400640 1 --2- 172.21.15.90:0/2660384831 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b6f1a829f0 0x55b6f1aa76c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:35.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.337+0000 7fdc29400640 1 -- 172.21.15.90:0/2660384831 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b6f1a86650 con 0x55b6f1a81d10 2024-04-02T14:49:35.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.337+0000 7fdc29e00640 1 --2- 172.21.15.90:0/2660384831 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b6f1a8b080 0x55b6f1a8c470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:49:35.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.337+0000 7fdc29400640 1 --2- 172.21.15.90:0/2660384831 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b6f1a81d10 0x55b6f1a8a830 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7fdc1800e960 tx=0x7fdc18013000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a16f580ba440089b server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:49:35.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.337+0000 7fdc15c00640 1 -- 172.21.15.90:0/2660384831 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc1802f070 con 0x55b6f1a81d10 2024-04-02T14:49:35.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.337+0000 7fdc15c00640 1 -- 172.21.15.90:0/2660384831 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdc1800edf0 con 0x55b6f1a81d10 2024-04-02T14:49:35.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.337+0000 7fdc15c00640 1 -- 172.21.15.90:0/2660384831 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc1801b040 con 0x55b6f1a81d10 2024-04-02T14:49:35.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.338+0000 7fdc2e9f2740 1 -- 172.21.15.90:0/2660384831 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b6f1a81d10 msgr2=0x55b6f1a8a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:49:35.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.338+0000 7fdc2e9f2740 1 --2- 172.21.15.90:0/2660384831 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b6f1a81d10 0x55b6f1a8a830 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7fdc1800e960 tx=0x7fdc18013000 comp rx=0 tx=0).stop 2024-04-02T14:49:35.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.339+0000 7fdc2e9f2740 1 -- 172.21.15.90:0/2660384831 shutdown_connections 2024-04-02T14:49:35.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.339+0000 7fdc2e9f2740 1 --2- 172.21.15.90:0/2660384831 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b6f1a8b080 0x55b6f1a8c470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:35.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.339+0000 7fdc2e9f2740 1 --2- 172.21.15.90:0/2660384831 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b6f1a829f0 0x55b6f1aa76c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:35.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.339+0000 7fdc2e9f2740 1 --2- 172.21.15.90:0/2660384831 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b6f1a81d10 0x55b6f1a8a830 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:35.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.339+0000 7fdc2e9f2740 1 -- 172.21.15.90:0/2660384831 >> 172.21.15.90:0/2660384831 conn(0x55b6f1a48a50 msgr2=0x55b6f1aa6fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:49:35.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.339+0000 7fdc2e9f2740 1 -- 172.21.15.90:0/2660384831 shutdown_connections 2024-04-02T14:49:35.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.339+0000 7fdc2e9f2740 1 -- 172.21.15.90:0/2660384831 wait complete. 2024-04-02T14:49:35.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.339+0000 7fdc2e9f2740 1 -- start start 2024-04-02T14:49:35.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.340+0000 7fdc2e9f2740 1 Processor -- start 2024-04-02T14:49:35.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.341+0000 7fdc2e9f2740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b6f1a81d10 0x55b6f1aad110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:35.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.341+0000 7fdc2e9f2740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b6f1a829f0 0x55b6f1aad650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:35.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.341+0000 7fdc2e9f2740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b6f1a8b080 0x55b6f1aadb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:35.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.341+0000 7fdc2e9f2740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fdc24070960 con 0x55b6f1a81d10 2024-04-02T14:49:35.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.341+0000 7fdc28a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b6f1a829f0 0x55b6f1aad650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:35.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.341+0000 7fdc28a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b6f1a829f0 0x55b6f1aad650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:50260/0 (socket says 172.21.15.90:50260) 2024-04-02T14:49:35.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.341+0000 7fdc28a00640 1 -- 172.21.15.90:0/3864560499 learned_addr learned my addr 172.21.15.90:0/3864560499 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:49:35.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.341+0000 7fdc29400640 1 --2- 172.21.15.90:0/3864560499 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b6f1a81d10 0x55b6f1aad110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:35.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.341+0000 7fdc29e00640 1 --2- 172.21.15.90:0/3864560499 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b6f1a8b080 0x55b6f1aadb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:35.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.341+0000 7fdc2e9f2740 1 -- 172.21.15.90:0/3864560499 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fdc24070290 con 0x55b6f1a829f0 2024-04-02T14:49:35.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.341+0000 7fdc2e9f2740 1 -- 172.21.15.90:0/3864560499 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fdc2406d4f0 con 0x55b6f1a8b080 2024-04-02T14:49:35.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.341+0000 7fdc2e9f2740 10 client.0 _ll_register_callbacks cb 0x7fdc1c009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:49:35.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.341+0000 7fdc28a00640 1 -- 172.21.15.90:0/3864560499 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b6f1a8b080 msgr2=0x55b6f1aadb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:49:35.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.341+0000 7fdc28a00640 1 --2- 172.21.15.90:0/3864560499 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b6f1a8b080 0x55b6f1aadb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:35.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.341+0000 7fdc28a00640 1 -- 172.21.15.90:0/3864560499 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b6f1a81d10 msgr2=0x55b6f1aad110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:49:35.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.341+0000 7fdc28a00640 1 --2- 172.21.15.90:0/3864560499 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b6f1a81d10 0x55b6f1aad110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:35.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.341+0000 7fdc28a00640 1 -- 172.21.15.90:0/3864560499 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b6f1b114b0 con 0x55b6f1a829f0 2024-04-02T14:49:35.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.342+0000 7fdc2e9f2740 1 -- 172.21.15.90:0/3864560499 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdc1800edf0 con 0x55b6f1a829f0 2024-04-02T14:49:35.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.342+0000 7fdc28a00640 1 --2- 172.21.15.90:0/3864560499 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b6f1a829f0 0x55b6f1aad650 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7fdc1009be10 tx=0x7fdc100047d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=70c5d574556c0d19 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:49:35.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.342+0000 7fdc0aa00640 10 client.30065 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:49:35.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.342+0000 7fdc0aa00640 1 -- 172.21.15.90:0/3864560499 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc1007f040 con 0x55b6f1a829f0 2024-04-02T14:49:35.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.342+0000 7fdc0aa00640 1 -- 172.21.15.90:0/3864560499 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 8550+0+0 (secure 0 0 0) 0x7fdc100b0db0 con 0x55b6f1a829f0 2024-04-02T14:49:35.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.342+0000 7fdc0aa00640 1 -- 172.21.15.90:0/3864560499 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fdc100b5040 con 0x55b6f1a829f0 2024-04-02T14:49:35.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.342+0000 7fdc2e9f2740 10 client.30065 fetch_fsmap learned FSMap version 27 2024-04-02T14:49:35.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.342+0000 7fdc2e9f2740 1 -- 172.21.15.90:0/3864560499 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({fsmap=27}) -- 0x55b6f1b12620 con 0x55b6f1a829f0 2024-04-02T14:49:35.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.343+0000 7fdc0aa00640 1 -- 172.21.15.90:0/3864560499 <== mon.2 v2:172.21.15.92:3300/0 4 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fdc1009b560 con 0x55b6f1a829f0 2024-04-02T14:49:35.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.343+0000 7fdc2e9f2740 10 client.30065 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:49:35.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.343+0000 7fdc2e9f2740 10 client.30065 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:49:35.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.343+0000 7fdc2e9f2740 20 client.30065 populate_metadata read hostname 'smithi090' 2024-04-02T14:49:35.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.343+0000 7fdc2e9f2740 1 --2- 172.21.15.90:0/3864560499 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55b6f1b12a30 0x55b6f1b14e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:35.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.343+0000 7fdc2e9f2740 4 client.30065 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-02T14:49:35.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.343+0000 7fdc2e9f2740 1 -- 172.21.15.90:0/3864560499 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55b6f19f3c30 con 0x55b6f1b12a30 2024-04-02T14:49:35.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.343+0000 7fdc29400640 1 --2- 172.21.15.90:0/3864560499 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55b6f1b12a30 0x55b6f1b14e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:35.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.344+0000 7fdc0aa00640 10 client.30065 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:49:35.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.344+0000 7fdc29400640 1 --2- 172.21.15.90:0/3864560499 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55b6f1b12a30 0x55b6f1b14e50 crc :-1 s=READY pgs=373 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=95afff80d4a14360 server_cookie=3a9839a4d580d819 in_seq=0 out_seq=0 2024-04-02T14:49:35.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.344+0000 7fdc0aa00640 1 -- 172.21.15.90:0/3864560499 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55b6f19f3c30 con 0x55b6f1b12a30 2024-04-02T14:49:35.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.344+0000 7fdc0aa00640 10 client.30065 handle_command_reply: tid=0 2024-04-02T14:49:35.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.345+0000 7fdc2e9f2740 1 client.30065 shutdown 2024-04-02T14:49:35.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.345+0000 7fdc2e9f2740 1 -- 172.21.15.90:0/3864560499 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b6f1a829f0 msgr2=0x55b6f1aad650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:49:35.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.345+0000 7fdc2e9f2740 1 --2- 172.21.15.90:0/3864560499 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b6f1a829f0 0x55b6f1aad650 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7fdc1009be10 tx=0x7fdc100047d0 comp rx=0 tx=0).stop 2024-04-02T14:49:35.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.346+0000 7fdc2e9f2740 1 -- 172.21.15.90:0/3864560499 shutdown_connections 2024-04-02T14:49:35.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.346+0000 7fdc2e9f2740 1 --2- 172.21.15.90:0/3864560499 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55b6f1b12a30 0x55b6f1b14e50 crc :-1 s=READY pgs=373 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:35.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.346+0000 7fdc2e9f2740 1 --2- 172.21.15.90:0/3864560499 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b6f1a8b080 0x55b6f1aadb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:35.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.346+0000 7fdc2e9f2740 1 --2- 172.21.15.90:0/3864560499 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55b6f1a829f0 0x55b6f1aad650 unknown :-1 s=CLOSED pgs=1001 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:35.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.346+0000 7fdc2e9f2740 1 --2- 172.21.15.90:0/3864560499 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55b6f1a81d10 0x55b6f1aad110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:35.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.346+0000 7fdc0aa00640 0 client.30065 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:49:35.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.346+0000 7fdc2e9f2740 1 -- 172.21.15.90:0/3864560499 >> 172.21.15.90:0/3864560499 conn(0x55b6f1a48a50 msgr2=0x55b6f1a7b8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:49:35.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.346+0000 7fdc2e9f2740 1 -- 172.21.15.90:0/3864560499 shutdown_connections 2024-04-02T14:49:35.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.346+0000 7fdc2e9f2740 1 -- 172.21.15.90:0/3864560499 wait complete. 2024-04-02T14:49:35.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.346+0000 7fdc2e9f2740 20 client.30065 trim_cache size 0 max 16384 2024-04-02T14:49:35.342 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-02T14:49:35.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.347+0000 7fdc2be00640 1 -- 172.21.15.90:0/210546496 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fdc0c0a3e60 msgr2=0x7fdc0c0a6320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:49:35.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.347+0000 7fdc2be00640 1 --2- 172.21.15.90:0/210546496 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fdc0c0a3e60 0x7fdc0c0a6320 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fdc18006fd0 tx=0x7fdc180090e0 comp rx=0 tx=0).stop 2024-04-02T14:49:35.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.347+0000 7fdc2be00640 1 -- 172.21.15.90:0/210546496 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdc24108fe0 msgr2=0x7fdc2419acd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:49:35.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.347+0000 7fdc2be00640 1 --2- 172.21.15.90:0/210546496 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdc24108fe0 0x7fdc2419acd0 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7fdc1000b450 tx=0x7fdc1000b910 comp rx=0 tx=0).stop 2024-04-02T14:49:35.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.347+0000 7fdc2be00640 1 -- 172.21.15.90:0/210546496 shutdown_connections 2024-04-02T14:49:35.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.347+0000 7fdc2be00640 1 --2- 172.21.15.90:0/210546496 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fdc0c0a3e60 0x7fdc0c0a6320 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:35.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.347+0000 7fdc2be00640 1 --2- 172.21.15.90:0/210546496 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdc2419f000 0x7fdc2419f4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:35.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.347+0000 7fdc2be00640 1 --2- 172.21.15.90:0/210546496 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdc24108fe0 0x7fdc2419acd0 unknown :-1 s=CLOSED pgs=1000 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:35.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.347+0000 7fdc2be00640 1 --2- 172.21.15.90:0/210546496 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdc24103f10 0x7fdc2419a790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:35.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.347+0000 7fdc2be00640 1 -- 172.21.15.90:0/210546496 >> 172.21.15.90:0/210546496 conn(0x7fdc240ff7b0 msgr2=0x7fdc24100c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:49:35.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.347+0000 7fdc2be00640 1 -- 172.21.15.90:0/210546496 shutdown_connections 2024-04-02T14:49:35.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:35.347+0000 7fdc2be00640 1 -- 172.21.15.90:0/210546496 wait complete. 2024-04-02T14:49:36.362 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-02T14:49:36.362 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T14:49:36.362 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-02T14:49:36.602 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:49:36 smithi092 ceph-mon[36497]: pgmap v460: 97 pgs: 97 active+clean; 4.2 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 7.5 MiB/s wr, 728 op/s 2024-04-02T14:49:36.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.658+0000 7efc1da00640 1 -- 172.21.15.90:0/508162515 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efc18114c50 msgr2=0x7efc18117040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:49:36.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.658+0000 7efc1da00640 1 --2- 172.21.15.90:0/508162515 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efc18114c50 0x7efc18117040 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7efc04009860 tx=0x7efc0402bb70 comp rx=0 tx=0).stop 2024-04-02T14:49:36.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.658+0000 7efc1da00640 1 -- 172.21.15.90:0/508162515 shutdown_connections 2024-04-02T14:49:36.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.658+0000 7efc1da00640 1 --2- 172.21.15.90:0/508162515 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efc18114c50 0x7efc18117040 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:36.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.658+0000 7efc1da00640 1 --2- 172.21.15.90:0/508162515 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efc18109600 0x7efc18114710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:36.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.658+0000 7efc1da00640 1 --2- 172.21.15.90:0/508162515 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efc18108ce0 0x7efc181090c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:36.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.658+0000 7efc1da00640 1 -- 172.21.15.90:0/508162515 >> 172.21.15.90:0/508162515 conn(0x7efc1806cbe0 msgr2=0x7efc1806cff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:49:36.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.659+0000 7efc1da00640 1 -- 172.21.15.90:0/508162515 shutdown_connections 2024-04-02T14:49:36.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.661+0000 7efc1da00640 1 -- 172.21.15.90:0/508162515 wait complete. 2024-04-02T14:49:36.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.663+0000 7efc1da00640 1 Processor -- start 2024-04-02T14:49:36.657 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.663+0000 7efc1da00640 1 -- start start 2024-04-02T14:49:36.657 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.663+0000 7efc1da00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efc18108ce0 0x7efc1810c780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:36.657 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.663+0000 7efc1da00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efc18109600 0x7efc1810ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:36.658 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.663+0000 7efc1da00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efc18114c50 0x7efc1810d200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:36.658 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.663+0000 7efc1da00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7efc1811a690 con 0x7efc18109600 2024-04-02T14:49:36.658 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.663+0000 7efc1da00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7efc1811a510 con 0x7efc18108ce0 2024-04-02T14:49:36.658 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.663+0000 7efc1da00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7efc1811a810 con 0x7efc18114c50 2024-04-02T14:49:36.658 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.664+0000 7efc16a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efc18108ce0 0x7efc1810c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:36.658 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.664+0000 7efc16a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efc18108ce0 0x7efc1810c780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:50274/0 (socket says 172.21.15.90:50274) 2024-04-02T14:49:36.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.664+0000 7efc16a00640 1 -- 172.21.15.90:0/2195408808 learned_addr learned my addr 172.21.15.90:0/2195408808 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:49:36.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.664+0000 7efc17400640 1 --2- 172.21.15.90:0/2195408808 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efc18114c50 0x7efc1810d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:36.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.664+0000 7efc16000640 1 --2- 172.21.15.90:0/2195408808 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efc18109600 0x7efc1810ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:36.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.664+0000 7efc16a00640 1 -- 172.21.15.90:0/2195408808 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efc18114c50 msgr2=0x7efc1810d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:49:36.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.664+0000 7efc16a00640 1 --2- 172.21.15.90:0/2195408808 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efc18114c50 0x7efc1810d200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:36.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.664+0000 7efc16a00640 1 -- 172.21.15.90:0/2195408808 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efc18109600 msgr2=0x7efc1810ccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:49:36.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.664+0000 7efc16a00640 1 --2- 172.21.15.90:0/2195408808 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efc18109600 0x7efc1810ccc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:36.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.664+0000 7efc16a00640 1 -- 172.21.15.90:0/2195408808 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc181b34c0 con 0x7efc18108ce0 2024-04-02T14:49:36.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.664+0000 7efc17400640 1 --2- 172.21.15.90:0/2195408808 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efc18114c50 0x7efc1810d200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:49:36.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.668+0000 7efc16a00640 1 --2- 172.21.15.90:0/2195408808 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efc18108ce0 0x7efc1810c780 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7efc0800e9f0 tx=0x7efc0800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:49:36.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.671+0000 7efc0f400640 1 -- 172.21.15.90:0/2195408808 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc0800ccc0 con 0x7efc18108ce0 2024-04-02T14:49:36.666 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.671+0000 7efc1da00640 1 -- 172.21.15.90:0/2195408808 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efc181b37b0 con 0x7efc18108ce0 2024-04-02T14:49:36.666 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.672+0000 7efc1da00640 1 -- 172.21.15.90:0/2195408808 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efc181b3d50 con 0x7efc18108ce0 2024-04-02T14:49:36.666 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.672+0000 7efc0f400640 1 -- 172.21.15.90:0/2195408808 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efc0800ce60 con 0x7efc18108ce0 2024-04-02T14:49:36.666 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.672+0000 7efc0f400640 1 -- 172.21.15.90:0/2195408808 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc08010690 con 0x7efc18108ce0 2024-04-02T14:49:36.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.676+0000 7efc20621740 1 Processor -- start 2024-04-02T14:49:36.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.677+0000 7efc20621740 1 -- start start 2024-04-02T14:49:36.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.677+0000 7efc20621740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f19566d030 0x55f19566bd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:36.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.677+0000 7efc20621740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f19566c350 0x55f19566c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:36.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.677+0000 7efc20621740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f1956756c0 0x55f195677ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:36.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.677+0000 7efc20621740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55f1955d0310 con 0x55f1956756c0 2024-04-02T14:49:36.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.677+0000 7efc20621740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55f195578690 con 0x55f19566c350 2024-04-02T14:49:36.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.677+0000 7efc20621740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55f1955cc2c0 con 0x55f19566d030 2024-04-02T14:49:36.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.677+0000 7efc0f400640 1 -- 172.21.15.90:0/2195408808 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7efc08010830 con 0x7efc18108ce0 2024-04-02T14:49:36.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.677+0000 7efc17400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f19566d030 0x55f19566bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:36.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.677+0000 7efc17400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f19566d030 0x55f19566bd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:53350/0 (socket says 172.21.15.90:53350) 2024-04-02T14:49:36.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.677+0000 7efc17400640 1 -- 172.21.15.90:0/4283689874 learned_addr learned my addr 172.21.15.90:0/4283689874 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:49:36.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.678+0000 7efc16000640 1 --2- 172.21.15.90:0/4283689874 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f1956756c0 0x55f195677ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:36.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.678+0000 7efc17400640 1 -- 172.21.15.90:0/4283689874 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f19566c350 msgr2=0x55f19566c7d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-02T14:49:36.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.678+0000 7efc17400640 1 --2- 172.21.15.90:0/4283689874 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f19566c350 0x55f19566c7d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:36.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.678+0000 7efc17400640 1 -- 172.21.15.90:0/4283689874 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f1956756c0 msgr2=0x55f195677ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:49:36.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.678+0000 7efc17400640 1 --2- 172.21.15.90:0/4283689874 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f1956756c0 0x55f195677ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:36.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.678+0000 7efc17400640 1 -- 172.21.15.90:0/4283689874 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f1953ac7e0 con 0x55f19566d030 2024-04-02T14:49:36.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.679+0000 7efc17400640 1 --2- 172.21.15.90:0/4283689874 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f19566d030 0x55f19566bd80 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7efc040057c0 tx=0x7efc04005710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7ff4cd62c1ded31d server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:49:36.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.679+0000 7efc03e00640 1 -- 172.21.15.90:0/4283689874 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc04044020 con 0x55f19566d030 2024-04-02T14:49:36.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.680+0000 7efc0f400640 1 --2- 172.21.15.90:0/2195408808 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7efbfc0b4db0 0x7efbfc0b7270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:36.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.680+0000 7efc03e00640 1 -- 172.21.15.90:0/4283689874 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efc04004240 con 0x55f19566d030 2024-04-02T14:49:36.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.680+0000 7efc03e00640 1 -- 172.21.15.90:0/4283689874 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc04039040 con 0x55f19566d030 2024-04-02T14:49:36.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.681+0000 7efc16a00640 1 --2- 172.21.15.90:0/2195408808 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7efbfc0b4db0 0x7efbfc0b7270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:36.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.682+0000 7efc0f400640 1 -- 172.21.15.90:0/2195408808 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 8550+0+0 (secure 0 0 0) 0x7efc08005020 con 0x7efc18108ce0 2024-04-02T14:49:36.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.683+0000 7efc16a00640 1 --2- 172.21.15.90:0/2195408808 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7efbfc0b4db0 0x7efbfc0b7270 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7efc08098590 tx=0x7efc080a1440 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:49:36.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.683+0000 7efc20621740 1 -- 172.21.15.90:0/4283689874 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f19566d030 msgr2=0x55f19566bd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:49:36.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.683+0000 7efc20621740 1 --2- 172.21.15.90:0/4283689874 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f19566d030 0x55f19566bd80 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7efc040057c0 tx=0x7efc04005710 comp rx=0 tx=0).stop 2024-04-02T14:49:36.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.684+0000 7efc20621740 1 -- 172.21.15.90:0/4283689874 shutdown_connections 2024-04-02T14:49:36.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.684+0000 7efc20621740 1 --2- 172.21.15.90:0/4283689874 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f1956756c0 0x55f195677ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:36.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.684+0000 7efc20621740 1 --2- 172.21.15.90:0/4283689874 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f19566c350 0x55f19566c7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:36.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.684+0000 7efc20621740 1 --2- 172.21.15.90:0/4283689874 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f19566d030 0x55f19566bd80 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:36.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.684+0000 7efc20621740 1 -- 172.21.15.90:0/4283689874 >> 172.21.15.90:0/4283689874 conn(0x55f195600270 msgr2=0x55f195666da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:49:36.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.684+0000 7efc20621740 1 -- 172.21.15.90:0/4283689874 shutdown_connections 2024-04-02T14:49:36.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:49:36 smithi177 ceph-mon[36431]: pgmap v460: 97 pgs: 97 active+clean; 4.2 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 7.5 MiB/s wr, 728 op/s 2024-04-02T14:49:36.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.684+0000 7efc20621740 1 -- 172.21.15.90:0/4283689874 wait complete. 2024-04-02T14:49:36.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.685+0000 7efc20621740 1 -- start start 2024-04-02T14:49:36.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.686+0000 7efc20621740 1 Processor -- start 2024-04-02T14:49:36.680 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:49:36 smithi090 ceph-mon[31674]: pgmap v460: 97 pgs: 97 active+clean; 4.2 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 7.5 MiB/s wr, 728 op/s 2024-04-02T14:49:36.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.687+0000 7efc20621740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f19566c350 0x55f195672620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:36.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.687+0000 7efc20621740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f19566d030 0x55f195672b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:36.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.687+0000 7efc20621740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f1956756c0 0x55f1956730a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:36.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.687+0000 7efc20621740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55f1956784d0 con 0x55f19566c350 2024-04-02T14:49:36.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.687+0000 7efc20621740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55f19558ba60 con 0x55f19566d030 2024-04-02T14:49:36.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.687+0000 7efc20621740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55f195678650 con 0x55f1956756c0 2024-04-02T14:49:36.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.687+0000 7efc20621740 10 client.0 _ll_register_callbacks cb 0x55f195591030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:49:36.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.688+0000 7efc17400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f19566c350 0x55f195672620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:36.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.688+0000 7efc16000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f19566d030 0x55f195672b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:36.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.688+0000 7efc16000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f19566d030 0x55f195672b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:50288/0 (socket says 172.21.15.90:50288) 2024-04-02T14:49:36.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.688+0000 7efc16000640 1 -- 172.21.15.90:0/2395974884 learned_addr learned my addr 172.21.15.90:0/2395974884 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:49:36.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.689+0000 7efc17400640 1 --2- 172.21.15.90:0/2395974884 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f1956756c0 0x55f1956730a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:36.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.689+0000 7efc16000640 1 -- 172.21.15.90:0/2395974884 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f1956756c0 msgr2=0x55f1956730a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:49:36.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.689+0000 7efc16000640 1 --2- 172.21.15.90:0/2395974884 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f1956756c0 0x55f1956730a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:36.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.689+0000 7efc16000640 1 -- 172.21.15.90:0/2395974884 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f19566c350 msgr2=0x55f195672620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:49:36.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.689+0000 7efc16000640 1 --2- 172.21.15.90:0/2395974884 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f19566c350 0x55f195672620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:36.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.689+0000 7efc16000640 1 -- 172.21.15.90:0/2395974884 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f19540cf70 con 0x55f19566d030 2024-04-02T14:49:36.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.690+0000 7efc17400640 1 --2- 172.21.15.90:0/2395974884 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f1956756c0 0x55f1956730a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:49:36.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.690+0000 7efc16000640 1 --2- 172.21.15.90:0/2395974884 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f19566d030 0x55f195672b60 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7efc1000e8b0 tx=0x7efc1000ed70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8197603dec039abf server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:49:36.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.690+0000 7efc00c00640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:49:36.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.690+0000 7efc00c00640 1 -- 172.21.15.90:0/2395974884 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc10018020 con 0x55f19566d030 2024-04-02T14:49:36.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.691+0000 7efc00c00640 1 -- 172.21.15.90:0/2395974884 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 8550+0+0 (secure 0 0 0) 0x7efc1000cb80 con 0x55f19566d030 2024-04-02T14:49:36.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.691+0000 7efc20621740 1 -- 172.21.15.90:0/2395974884 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efc04004240 con 0x55f19566d030 2024-04-02T14:49:36.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.693+0000 7efc00c00640 1 -- 172.21.15.90:0/2395974884 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7efc1000eea0 con 0x55f19566d030 2024-04-02T14:49:36.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.693+0000 7efc20621740 10 client.30074 fetch_fsmap learned FSMap version 27 2024-04-02T14:49:36.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.693+0000 7efc20621740 1 -- 172.21.15.90:0/2395974884 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55f1952a4170 con 0x55f19566d030 2024-04-02T14:49:36.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.694+0000 7efc00c00640 1 -- 172.21.15.90:0/2395974884 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efc04004240 con 0x55f19566d030 2024-04-02T14:49:36.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.694+0000 7efc00c00640 1 -- 172.21.15.90:0/2395974884 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7efc10025020 con 0x55f19566d030 2024-04-02T14:49:36.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.694+0000 7efc20621740 10 client.30074 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:49:36.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.694+0000 7efc20621740 10 client.30074 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:49:36.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.694+0000 7efc20621740 20 client.30074 populate_metadata read hostname 'smithi090' 2024-04-02T14:49:36.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.694+0000 7efc20621740 1 --2- 172.21.15.90:0/2395974884 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55f195685930 0x55f195687d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:36.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.695+0000 7efc20621740 4 client.30074 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:49:36.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.695+0000 7efc20621740 1 -- 172.21.15.90:0/2395974884 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f1955dd850 con 0x55f195685930 2024-04-02T14:49:36.690 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.697+0000 7efc17400640 1 --2- 172.21.15.90:0/2395974884 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55f195685930 0x55f195687d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:36.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.698+0000 7efc17400640 1 --2- 172.21.15.90:0/2395974884 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55f195685930 0x55f195687d60 crc :-1 s=READY pgs=374 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=348e4d6782bf0fa0 server_cookie=7d5968a27c652f4b in_seq=0 out_seq=0 2024-04-02T14:49:36.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.698+0000 7efc00c00640 10 client.30074 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:49:36.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.703+0000 7efc00c00640 1 -- 172.21.15.90:0/2395974884 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f1955dd850 con 0x55f195685930 2024-04-02T14:49:36.697 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.703+0000 7efc00c00640 10 client.30074 handle_command_reply: tid=0 2024-04-02T14:49:36.697 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.703+0000 7efc20621740 1 client.30074 shutdown 2024-04-02T14:49:36.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.705+0000 7efc20621740 1 -- 172.21.15.90:0/2395974884 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f19566d030 msgr2=0x55f195672b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:49:36.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.705+0000 7efc20621740 1 --2- 172.21.15.90:0/2395974884 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f19566d030 0x55f195672b60 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7efc1000e8b0 tx=0x7efc1000ed70 comp rx=0 tx=0).stop 2024-04-02T14:49:36.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.705+0000 7efc20621740 1 -- 172.21.15.90:0/2395974884 shutdown_connections 2024-04-02T14:49:36.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.705+0000 7efc20621740 1 --2- 172.21.15.90:0/2395974884 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55f195685930 0x55f195687d60 crc :-1 s=READY pgs=374 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:36.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.705+0000 7efc20621740 1 --2- 172.21.15.90:0/2395974884 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f1956756c0 0x55f1956730a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:36.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.705+0000 7efc20621740 1 --2- 172.21.15.90:0/2395974884 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f19566d030 0x55f195672b60 unknown :-1 s=CLOSED pgs=1003 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:36.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.705+0000 7efc20621740 1 --2- 172.21.15.90:0/2395974884 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f19566c350 0x55f195672620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:36.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.705+0000 7efc20621740 1 -- 172.21.15.90:0/2395974884 >> 172.21.15.90:0/2395974884 conn(0x55f195600270 msgr2=0x55f195668af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:49:36.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.705+0000 7efc00c00640 0 client.30074 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:49:36.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.706+0000 7efc20621740 1 -- 172.21.15.90:0/2395974884 shutdown_connections 2024-04-02T14:49:36.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.706+0000 7efc20621740 1 -- 172.21.15.90:0/2395974884 wait complete. 2024-04-02T14:49:36.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.706+0000 7efc20621740 20 client.30074 trim_cache size 0 max 16384 2024-04-02T14:49:36.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.740+0000 7efc20621740 1 Processor -- start 2024-04-02T14:49:36.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.740+0000 7efc20621740 1 -- start start 2024-04-02T14:49:36.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.740+0000 7efc20621740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f19566c350 0x55f195681430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:36.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.740+0000 7efc20621740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f19566d030 0x55f195672ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:36.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.740+0000 7efc20621740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f1956756c0 0x55f19568d670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:36.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.740+0000 7efc20621740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7efbf800e8a0 con 0x55f1956756c0 2024-04-02T14:49:36.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.740+0000 7efc20621740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7efbf80103e0 con 0x55f19566d030 2024-04-02T14:49:36.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.740+0000 7efc20621740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7efbf800ef70 con 0x55f19566c350 2024-04-02T14:49:36.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.741+0000 7efc17400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f19566c350 0x55f195681430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:36.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.741+0000 7efc17400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f19566c350 0x55f195681430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:53376/0 (socket says 172.21.15.90:53376) 2024-04-02T14:49:36.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.741+0000 7efc17400640 1 -- 172.21.15.90:0/4046651229 learned_addr learned my addr 172.21.15.90:0/4046651229 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:49:36.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.741+0000 7efc16000640 1 --2- 172.21.15.90:0/4046651229 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f19566d030 0x55f195672ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:36.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.741+0000 7efc17400640 1 -- 172.21.15.90:0/4046651229 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f19566d030 msgr2=0x55f195672ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:49:36.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.741+0000 7efc17400640 1 --2- 172.21.15.90:0/4046651229 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f19566d030 0x55f195672ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:36.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.741+0000 7efc17400640 1 -- 172.21.15.90:0/4046651229 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f1956756c0 msgr2=0x55f19568d670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:49:36.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.741+0000 7efc17400640 1 --2- 172.21.15.90:0/4046651229 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f1956756c0 0x55f19568d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:36.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.741+0000 7efc17400640 1 -- 172.21.15.90:0/4046651229 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f19566b380 con 0x55f19566c350 2024-04-02T14:49:36.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.741+0000 7efc16000640 1 --2- 172.21.15.90:0/4046651229 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f19566d030 0x55f195672ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:49:36.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.742+0000 7efc17400640 1 --2- 172.21.15.90:0/4046651229 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f19566c350 0x55f195681430 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7efc04009860 tx=0x7efc0402c690 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a3948e784ddb4bab server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:49:36.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.742+0000 7efc03400640 1 -- 172.21.15.90:0/4046651229 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc04059070 con 0x55f19566c350 2024-04-02T14:49:36.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.742+0000 7efc03400640 1 -- 172.21.15.90:0/4046651229 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efc040044d0 con 0x55f19566c350 2024-04-02T14:49:36.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.742+0000 7efc03400640 1 -- 172.21.15.90:0/4046651229 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc04044040 con 0x55f19566c350 2024-04-02T14:49:36.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.743+0000 7efc20621740 1 -- 172.21.15.90:0/4046651229 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f19566c350 msgr2=0x55f195681430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:49:36.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.743+0000 7efc20621740 1 --2- 172.21.15.90:0/4046651229 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f19566c350 0x55f195681430 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7efc04009860 tx=0x7efc0402c690 comp rx=0 tx=0).stop 2024-04-02T14:49:36.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.743+0000 7efc20621740 1 -- 172.21.15.90:0/4046651229 shutdown_connections 2024-04-02T14:49:36.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.743+0000 7efc20621740 1 --2- 172.21.15.90:0/4046651229 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f1956756c0 0x55f19568d670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:36.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.743+0000 7efc20621740 1 --2- 172.21.15.90:0/4046651229 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f19566d030 0x55f195672ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:36.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.743+0000 7efc20621740 1 --2- 172.21.15.90:0/4046651229 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f19566c350 0x55f195681430 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:36.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.743+0000 7efc20621740 1 -- 172.21.15.90:0/4046651229 >> 172.21.15.90:0/4046651229 conn(0x55f195600270 msgr2=0x55f195668a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:49:36.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.743+0000 7efc20621740 1 -- 172.21.15.90:0/4046651229 shutdown_connections 2024-04-02T14:49:36.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.743+0000 7efc20621740 1 -- 172.21.15.90:0/4046651229 wait complete. 2024-04-02T14:49:36.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.743+0000 7efc20621740 1 -- start start 2024-04-02T14:49:36.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.743+0000 7efc20621740 1 Processor -- start 2024-04-02T14:49:36.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.744+0000 7efc20621740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f19566c350 0x55f19567d940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:36.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.744+0000 7efc20621740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f19566d030 0x55f19567de80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:36.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.744+0000 7efc20621740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f1956756c0 0x55f19567a910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:36.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.744+0000 7efc20621740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7efbf800fd10 con 0x55f19566c350 2024-04-02T14:49:36.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.744+0000 7efc20621740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7efbf800f640 con 0x55f1956756c0 2024-04-02T14:49:36.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.744+0000 7efc20621740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7efbf800c8a0 con 0x55f19566d030 2024-04-02T14:49:36.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.744+0000 7efc20621740 10 client.0 _ll_register_callbacks cb 0x7efc10008040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:49:36.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.744+0000 7efc16000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f19566d030 0x55f19567de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:36.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.744+0000 7efc16000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f19566d030 0x55f19567de80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:53382/0 (socket says 172.21.15.90:53382) 2024-04-02T14:49:36.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.744+0000 7efc16000640 1 -- 172.21.15.90:0/1405806386 learned_addr learned my addr 172.21.15.90:0/1405806386 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:49:36.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.744+0000 7efc17400640 1 --2- 172.21.15.90:0/1405806386 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f19566c350 0x55f19567d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:36.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.744+0000 7efc17400640 1 --2- 172.21.15.90:0/1405806386 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f1956756c0 0x55f19567a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:36.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.745+0000 7efc16000640 1 -- 172.21.15.90:0/1405806386 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f1956756c0 msgr2=0x55f19567a910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:49:36.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.746+0000 7efc16000640 1 --2- 172.21.15.90:0/1405806386 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f1956756c0 0x55f19567a910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:36.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.746+0000 7efc16000640 1 -- 172.21.15.90:0/1405806386 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f19566c350 msgr2=0x55f19567d940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:49:36.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.746+0000 7efc16000640 1 --2- 172.21.15.90:0/1405806386 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f19566c350 0x55f19567d940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:36.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.746+0000 7efc16000640 1 -- 172.21.15.90:0/1405806386 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f19567afa0 con 0x55f19566d030 2024-04-02T14:49:36.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.746+0000 7efc17400640 1 --2- 172.21.15.90:0/1405806386 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f19566c350 0x55f19567d940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:49:36.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.746+0000 7efc00c00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:49:36.741 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.746+0000 7efc16000640 1 --2- 172.21.15.90:0/1405806386 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f19566d030 0x55f19567de80 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x55f195671e80 tx=0x7efc1000c680 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d0d9cbfed25bdb6e server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:49:36.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.747+0000 7efc00c00640 1 -- 172.21.15.90:0/1405806386 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc10018040 con 0x55f19566d030 2024-04-02T14:49:36.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.747+0000 7efc00c00640 1 -- 172.21.15.90:0/1405806386 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 8550+0+0 (secure 0 0 0) 0x7efc1000c980 con 0x55f19566d030 2024-04-02T14:49:36.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.747+0000 7efc20621740 1 -- 172.21.15.90:0/1405806386 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efc040044d0 con 0x55f19566d030 2024-04-02T14:49:36.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.747+0000 7efc00c00640 1 -- 172.21.15.90:0/1405806386 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7efc1000cbc0 con 0x55f19566d030 2024-04-02T14:49:36.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.747+0000 7efc20621740 10 client.29869 fetch_fsmap learned FSMap version 27 2024-04-02T14:49:36.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.747+0000 7efc20621740 1 -- 172.21.15.90:0/1405806386 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55f1956ec940 con 0x55f19566d030 2024-04-02T14:49:36.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.748+0000 7efc00c00640 1 -- 172.21.15.90:0/1405806386 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efc040044d0 con 0x55f19566d030 2024-04-02T14:49:36.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.748+0000 7efc00c00640 1 -- 172.21.15.90:0/1405806386 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7efc10025040 con 0x55f19566d030 2024-04-02T14:49:36.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.748+0000 7efc20621740 10 client.29869 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:49:36.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.748+0000 7efc20621740 10 client.29869 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:49:36.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.748+0000 7efc20621740 20 client.29869 populate_metadata read hostname 'smithi090' 2024-04-02T14:49:36.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.748+0000 7efc20621740 1 --2- 172.21.15.90:0/1405806386 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55f1956ed0c0 0x55f1956ef4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:36.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.748+0000 7efc20621740 4 client.29869 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-02T14:49:36.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.748+0000 7efc20621740 1 -- 172.21.15.90:0/1405806386 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55f1955d7ba0 con 0x55f1956ed0c0 2024-04-02T14:49:36.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.749+0000 7efc17400640 1 --2- 172.21.15.90:0/1405806386 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55f1956ed0c0 0x55f1956ef4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:36.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.749+0000 7efc17400640 1 --2- 172.21.15.90:0/1405806386 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55f1956ed0c0 0x55f1956ef4e0 crc :-1 s=READY pgs=375 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4ee63d401fe6cc07 server_cookie=9fc27d6cb46977ed in_seq=0 out_seq=0 2024-04-02T14:49:36.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.749+0000 7efc00c00640 10 client.29869 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:49:36.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.750+0000 7efc00c00640 1 -- 172.21.15.90:0/1405806386 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55f1955d7ba0 con 0x55f1956ed0c0 2024-04-02T14:49:36.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.750+0000 7efc00c00640 10 client.29869 handle_command_reply: tid=0 2024-04-02T14:49:36.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.750+0000 7efc20621740 1 client.29869 shutdown 2024-04-02T14:49:36.745 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.751+0000 7efc20621740 1 -- 172.21.15.90:0/1405806386 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f19566d030 msgr2=0x55f19567de80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:49:36.745 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.751+0000 7efc20621740 1 --2- 172.21.15.90:0/1405806386 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f19566d030 0x55f19567de80 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x55f195671e80 tx=0x7efc1000c680 comp rx=0 tx=0).stop 2024-04-02T14:49:36.745 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.751+0000 7efc20621740 1 -- 172.21.15.90:0/1405806386 shutdown_connections 2024-04-02T14:49:36.745 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.751+0000 7efc20621740 1 --2- 172.21.15.90:0/1405806386 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55f1956ed0c0 0x55f1956ef4e0 crc :-1 s=READY pgs=375 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:36.745 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.751+0000 7efc20621740 1 --2- 172.21.15.90:0/1405806386 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55f1956756c0 0x55f19567a910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:36.745 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.751+0000 7efc20621740 1 --2- 172.21.15.90:0/1405806386 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f19566d030 0x55f19567de80 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:36.745 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.751+0000 7efc20621740 1 --2- 172.21.15.90:0/1405806386 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f19566c350 0x55f19567d940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:36.745 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.751+0000 7efc20621740 1 -- 172.21.15.90:0/1405806386 >> 172.21.15.90:0/1405806386 conn(0x55f195600270 msgr2=0x55f195667fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:49:36.745 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.751+0000 7efc00c00640 0 client.29869 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:49:36.746 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.751+0000 7efc20621740 1 -- 172.21.15.90:0/1405806386 shutdown_connections 2024-04-02T14:49:36.746 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.752+0000 7efc20621740 1 -- 172.21.15.90:0/1405806386 wait complete. 2024-04-02T14:49:36.746 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.752+0000 7efc20621740 20 client.29869 trim_cache size 0 max 16384 2024-04-02T14:49:36.747 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-02T14:49:36.747 INFO:teuthology.orchestra.run.smithi090.stdout: "return_code": 0, 2024-04-02T14:49:36.747 INFO:teuthology.orchestra.run.smithi090.stdout: "scrub_tag": "703b31e0-ef88-41f1-a3d1-2dd9bcdc6912", 2024-04-02T14:49:36.747 INFO:teuthology.orchestra.run.smithi090.stdout: "mode": "asynchronous" 2024-04-02T14:49:36.747 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-02T14:49:36.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.752+0000 7efc1da00640 1 -- 172.21.15.90:0/2195408808 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7efbfc0b4db0 msgr2=0x7efbfc0b7270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:49:36.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.752+0000 7efc1da00640 1 --2- 172.21.15.90:0/2195408808 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7efbfc0b4db0 0x7efbfc0b7270 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7efc08098590 tx=0x7efc080a1440 comp rx=0 tx=0).stop 2024-04-02T14:49:36.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.752+0000 7efc1da00640 1 -- 172.21.15.90:0/2195408808 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efc18108ce0 msgr2=0x7efc1810c780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:49:36.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.752+0000 7efc1da00640 1 --2- 172.21.15.90:0/2195408808 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efc18108ce0 0x7efc1810c780 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7efc0800e9f0 tx=0x7efc0800eeb0 comp rx=0 tx=0).stop 2024-04-02T14:49:36.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.753+0000 7efc1da00640 1 -- 172.21.15.90:0/2195408808 shutdown_connections 2024-04-02T14:49:36.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.753+0000 7efc1da00640 1 --2- 172.21.15.90:0/2195408808 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7efbfc0b4db0 0x7efbfc0b7270 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:36.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.753+0000 7efc1da00640 1 --2- 172.21.15.90:0/2195408808 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efc18114c50 0x7efc1810d200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:36.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.753+0000 7efc1da00640 1 --2- 172.21.15.90:0/2195408808 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efc18109600 0x7efc1810ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:36.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.753+0000 7efc1da00640 1 --2- 172.21.15.90:0/2195408808 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efc18108ce0 0x7efc1810c780 unknown :-1 s=CLOSED pgs=1002 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:36.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.753+0000 7efc1da00640 1 -- 172.21.15.90:0/2195408808 >> 172.21.15.90:0/2195408808 conn(0x7efc1806cbe0 msgr2=0x7efc1810a4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:49:36.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.753+0000 7efc1da00640 1 -- 172.21.15.90:0/2195408808 shutdown_connections 2024-04-02T14:49:36.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:36.753+0000 7efc1da00640 1 -- 172.21.15.90:0/2195408808 wait complete. 2024-04-02T14:49:36.766 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-02T14:49:36.766 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-02T14:49:36.824 INFO:teuthology.orchestra.run.smithi090.stderr:progress: 845.0 s, 1280.8 tps, lat 78.574 ms stddev 89.381 2024-04-02T14:49:37.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.024+0000 7fc7e5600640 1 -- 172.21.15.90:0/3493503505 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc7e0076030 msgr2=0x7fc7e0076410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:49:37.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.024+0000 7fc7e5600640 1 --2- 172.21.15.90:0/3493503505 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc7e0076030 0x7fc7e0076410 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7fc7d0009930 tx=0x7fc7d002bc60 comp rx=0 tx=0).stop 2024-04-02T14:49:37.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.028+0000 7fc7e5600640 1 -- 172.21.15.90:0/3493503505 shutdown_connections 2024-04-02T14:49:37.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.028+0000 7fc7e5600640 1 --2- 172.21.15.90:0/3493503505 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc7e0112940 0x7fc7e0114d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:37.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.028+0000 7fc7e5600640 1 --2- 172.21.15.90:0/3493503505 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc7e0076950 0x7fc7e0112400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:37.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.028+0000 7fc7e5600640 1 --2- 172.21.15.90:0/3493503505 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc7e0076030 0x7fc7e0076410 unknown :-1 s=CLOSED pgs=1004 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:37.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.028+0000 7fc7e5600640 1 -- 172.21.15.90:0/3493503505 >> 172.21.15.90:0/3493503505 conn(0x7fc7e00ff7b0 msgr2=0x7fc7e0101bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:49:37.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.028+0000 7fc7e5600640 1 -- 172.21.15.90:0/3493503505 shutdown_connections 2024-04-02T14:49:37.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.028+0000 7fc7e5600640 1 -- 172.21.15.90:0/3493503505 wait complete. 2024-04-02T14:49:37.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.030+0000 7fc7e5600640 1 Processor -- start 2024-04-02T14:49:37.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.034+0000 7fc7e5600640 1 -- start start 2024-04-02T14:49:37.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.034+0000 7fc7e5600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc7e0076030 0x7fc7e01a4750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:37.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.034+0000 7fc7e5600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc7e0076950 0x7fc7e01a4c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:37.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.034+0000 7fc7e5600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc7e0112940 0x7fc7e019e900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:37.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.034+0000 7fc7e5600640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fc7e0118520 con 0x7fc7e0076030 2024-04-02T14:49:37.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.034+0000 7fc7e5600640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fc7e01183a0 con 0x7fc7e0076950 2024-04-02T14:49:37.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.034+0000 7fc7e5600640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fc7e01186a0 con 0x7fc7e0112940 2024-04-02T14:49:37.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.035+0000 7fc7df400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc7e0112940 0x7fc7e019e900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:37.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.035+0000 7fc7df400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc7e0112940 0x7fc7e019e900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:53394/0 (socket says 172.21.15.90:53394) 2024-04-02T14:49:37.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.035+0000 7fc7df400640 1 -- 172.21.15.90:0/2471588409 learned_addr learned my addr 172.21.15.90:0/2471588409 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:49:37.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.035+0000 7fc7df400640 1 -- 172.21.15.90:0/2471588409 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc7e0076950 msgr2=0x7fc7e01a4c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:49:37.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.035+0000 7fc7de000640 1 --2- 172.21.15.90:0/2471588409 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc7e0076950 0x7fc7e01a4c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:37.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.036+0000 7fc7df400640 1 --2- 172.21.15.90:0/2471588409 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc7e0076950 0x7fc7e01a4c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:37.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.036+0000 7fc7df400640 1 -- 172.21.15.90:0/2471588409 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc7e0076030 msgr2=0x7fc7e01a4750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:49:37.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.036+0000 7fc7df400640 1 --2- 172.21.15.90:0/2471588409 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc7e0076030 0x7fc7e01a4750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:37.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.036+0000 7fc7df400640 1 -- 172.21.15.90:0/2471588409 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7e019f080 con 0x7fc7e0112940 2024-04-02T14:49:37.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.036+0000 7fc7de000640 1 --2- 172.21.15.90:0/2471588409 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc7e0076950 0x7fc7e01a4c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:49:37.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.036+0000 7fc7df400640 1 --2- 172.21.15.90:0/2471588409 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc7e0112940 0x7fc7e019e900 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7fc7d800eb40 tx=0x7fc7d800c2d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:49:37.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.036+0000 7fc7d7400640 1 -- 172.21.15.90:0/2471588409 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7d8004b20 con 0x7fc7e0112940 2024-04-02T14:49:37.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.036+0000 7fc7d7400640 1 -- 172.21.15.90:0/2471588409 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc7d8004cc0 con 0x7fc7e0112940 2024-04-02T14:49:37.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.036+0000 7fc7d7400640 1 -- 172.21.15.90:0/2471588409 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7d80187f0 con 0x7fc7e0112940 2024-04-02T14:49:37.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.038+0000 7fc7e5600640 1 -- 172.21.15.90:0/2471588409 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc7e019f370 con 0x7fc7e0112940 2024-04-02T14:49:37.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.038+0000 7fc7e5600640 1 -- 172.21.15.90:0/2471588409 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7e01ab610 con 0x7fc7e0112940 2024-04-02T14:49:37.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.046+0000 7fc7d7400640 1 -- 172.21.15.90:0/2471588409 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fc7d8005000 con 0x7fc7e0112940 2024-04-02T14:49:37.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.050+0000 7fc7e83be740 1 Processor -- start 2024-04-02T14:49:37.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.050+0000 7fc7e83be740 1 -- start start 2024-04-02T14:49:37.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.051+0000 7fc7e83be740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56321c1d57b0 0x56321c1d5b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:37.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.051+0000 7fc7e83be740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56321c1d60d0 0x56321c1de770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:37.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.051+0000 7fc7e83be740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56321c1decb0 0x56321c1e10a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:37.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.051+0000 7fc7e83be740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x56321c139770 con 0x56321c1d60d0 2024-04-02T14:49:37.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.051+0000 7fc7e83be740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x56321c0e1700 con 0x56321c1d57b0 2024-04-02T14:49:37.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.051+0000 7fc7e83be740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x56321c135720 con 0x56321c1decb0 2024-04-02T14:49:37.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.052+0000 7fc7de000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56321c1d57b0 0x56321c1d5b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:37.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.052+0000 7fc7de000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56321c1d57b0 0x56321c1d5b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:50340/0 (socket says 172.21.15.90:50340) 2024-04-02T14:49:37.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.052+0000 7fc7de000640 1 -- 172.21.15.90:0/4020505865 learned_addr learned my addr 172.21.15.90:0/4020505865 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:49:37.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.052+0000 7fc7de000640 1 --2- 172.21.15.90:0/4020505865 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56321c1decb0 0x56321c1e10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:37.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.052+0000 7fc7dea00640 1 --2- 172.21.15.90:0/4020505865 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56321c1d60d0 0x56321c1de770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:37.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.052+0000 7fc7d7400640 1 --2- 172.21.15.90:0/2471588409 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fc7c40a3da0 0x7fc7c40a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:37.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.052+0000 7fc7de000640 1 -- 172.21.15.90:0/4020505865 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56321c1decb0 msgr2=0x56321c1e10a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:49:37.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.052+0000 7fc7de000640 1 --2- 172.21.15.90:0/4020505865 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56321c1decb0 0x56321c1e10a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:37.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.052+0000 7fc7de000640 1 -- 172.21.15.90:0/4020505865 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56321c1d60d0 msgr2=0x56321c1de770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:49:37.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.052+0000 7fc7de000640 1 --2- 172.21.15.90:0/4020505865 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56321c1d60d0 0x56321c1de770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:37.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.052+0000 7fc7de000640 1 -- 172.21.15.90:0/4020505865 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56321bf177d0 con 0x56321c1d57b0 2024-04-02T14:49:37.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.053+0000 7fc7d7400640 1 -- 172.21.15.90:0/2471588409 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 8550+0+0 (secure 0 0 0) 0x7fc7d80d1420 con 0x7fc7e0112940 2024-04-02T14:49:37.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.053+0000 7fc7de000640 1 --2- 172.21.15.90:0/4020505865 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56321c1d57b0 0x56321c1d5b90 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7fc7cc00afc0 tx=0x7fc7cc00a150 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c605cdd83606640c server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:49:37.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.053+0000 7fc7cb400640 1 -- 172.21.15.90:0/4020505865 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7cc00f040 con 0x56321c1d57b0 2024-04-02T14:49:37.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.053+0000 7fc7cb400640 1 -- 172.21.15.90:0/4020505865 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc7cc0036b0 con 0x56321c1d57b0 2024-04-02T14:49:37.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.053+0000 7fc7cb400640 1 -- 172.21.15.90:0/4020505865 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7cc0186b0 con 0x56321c1d57b0 2024-04-02T14:49:37.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.053+0000 7fc7df400640 1 --2- 172.21.15.90:0/2471588409 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fc7c40a3da0 0x7fc7c40a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:37.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.054+0000 7fc7e83be740 1 -- 172.21.15.90:0/4020505865 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56321c1d57b0 msgr2=0x56321c1d5b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:49:37.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.054+0000 7fc7e83be740 1 --2- 172.21.15.90:0/4020505865 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56321c1d57b0 0x56321c1d5b90 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7fc7cc00afc0 tx=0x7fc7cc00a150 comp rx=0 tx=0).stop 2024-04-02T14:49:37.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.055+0000 7fc7df400640 1 --2- 172.21.15.90:0/2471588409 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fc7c40a3da0 0x7fc7c40a6260 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fc7d8099800 tx=0x7fc7d8099040 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:49:37.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.056+0000 7fc7e83be740 1 -- 172.21.15.90:0/4020505865 shutdown_connections 2024-04-02T14:49:37.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.057+0000 7fc7e83be740 1 --2- 172.21.15.90:0/4020505865 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56321c1decb0 0x56321c1e10a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:37.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.057+0000 7fc7e83be740 1 --2- 172.21.15.90:0/4020505865 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56321c1d60d0 0x56321c1de770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:37.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.057+0000 7fc7e83be740 1 --2- 172.21.15.90:0/4020505865 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56321c1d57b0 0x56321c1d5b90 unknown :-1 s=CLOSED pgs=1005 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:37.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.057+0000 7fc7e83be740 1 -- 172.21.15.90:0/4020505865 >> 172.21.15.90:0/4020505865 conn(0x56321c1cfee0 msgr2=0x56321c1d02c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:49:37.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.057+0000 7fc7e83be740 1 -- 172.21.15.90:0/4020505865 shutdown_connections 2024-04-02T14:49:37.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.057+0000 7fc7e83be740 1 -- 172.21.15.90:0/4020505865 wait complete. 2024-04-02T14:49:37.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.058+0000 7fc7e83be740 1 -- start start 2024-04-02T14:49:37.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.058+0000 7fc7e83be740 1 Processor -- start 2024-04-02T14:49:37.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.059+0000 7fc7e83be740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56321c1d57b0 0x56321c1dbd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:37.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.059+0000 7fc7e83be740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56321c1d60d0 0x56321c1dc250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:37.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.059+0000 7fc7e83be740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56321c1decb0 0x56321c1dc790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:37.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.059+0000 7fc7e83be740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x56321c1e1a40 con 0x56321c1decb0 2024-04-02T14:49:37.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.059+0000 7fc7e83be740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x56321c0f4ac0 con 0x56321c1d57b0 2024-04-02T14:49:37.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.059+0000 7fc7e83be740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x56321c1e1bc0 con 0x56321c1d60d0 2024-04-02T14:49:37.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.059+0000 7fc7e83be740 10 client.0 _ll_register_callbacks cb 0x56321c0f7180 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:49:37.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.059+0000 7fc7de000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56321c1d57b0 0x56321c1dbd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:37.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.060+0000 7fc7de000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56321c1d57b0 0x56321c1dbd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:50356/0 (socket says 172.21.15.90:50356) 2024-04-02T14:49:37.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.060+0000 7fc7de000640 1 -- 172.21.15.90:0/474826909 learned_addr learned my addr 172.21.15.90:0/474826909 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:49:37.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.060+0000 7fc7de000640 1 --2- 172.21.15.90:0/474826909 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56321c1decb0 0x56321c1dc790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:37.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.061+0000 7fc7de000640 1 -- 172.21.15.90:0/474826909 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56321c1d60d0 msgr2=0x56321c1dc250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:49:37.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.061+0000 7fc7de000640 1 --2- 172.21.15.90:0/474826909 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56321c1d60d0 0x56321c1dc250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:37.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.061+0000 7fc7de000640 1 -- 172.21.15.90:0/474826909 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56321c1decb0 msgr2=0x56321c1dc790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:49:37.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.061+0000 7fc7de000640 1 --2- 172.21.15.90:0/474826909 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56321c1decb0 0x56321c1dc790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:37.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.061+0000 7fc7de000640 1 -- 172.21.15.90:0/474826909 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56321be4d3f0 con 0x56321c1d57b0 2024-04-02T14:49:37.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.061+0000 7fc7de000640 1 --2- 172.21.15.90:0/474826909 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56321c1d57b0 0x56321c1dbd10 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7fc7cc01a040 tx=0x7fc7cc00aa90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a1bb21f6827ba7d7 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:49:37.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.061+0000 7fc7bfe00640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:49:37.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.062+0000 7fc7bfe00640 1 -- 172.21.15.90:0/474826909 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7cc02b020 con 0x56321c1d57b0 2024-04-02T14:49:37.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.062+0000 7fc7bfe00640 1 -- 172.21.15.90:0/474826909 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 8550+0+0 (secure 0 0 0) 0x7fc7cc018470 con 0x56321c1d57b0 2024-04-02T14:49:37.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.062+0000 7fc7e83be740 1 -- 172.21.15.90:0/474826909 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc7cc0036b0 con 0x56321c1d57b0 2024-04-02T14:49:37.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.064+0000 7fc7bfe00640 1 -- 172.21.15.90:0/474826909 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fc7cc018670 con 0x56321c1d57b0 2024-04-02T14:49:37.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.065+0000 7fc7e83be740 10 client.30089 fetch_fsmap learned FSMap version 27 2024-04-02T14:49:37.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.065+0000 7fc7e83be740 1 -- 172.21.15.90:0/474826909 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x56321be0d0f0 con 0x56321c1d57b0 2024-04-02T14:49:37.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.066+0000 7fc7bfe00640 1 -- 172.21.15.90:0/474826909 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc7cc0036b0 con 0x56321c1d57b0 2024-04-02T14:49:37.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.067+0000 7fc7bfe00640 1 -- 172.21.15.90:0/474826909 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fc7cc018810 con 0x56321c1d57b0 2024-04-02T14:49:37.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.068+0000 7fc7e83be740 10 client.30089 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:49:37.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.068+0000 7fc7e83be740 10 client.30089 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:49:37.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.068+0000 7fc7e83be740 20 client.30089 populate_metadata read hostname 'smithi090' 2024-04-02T14:49:37.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.068+0000 7fc7e83be740 1 --2- 172.21.15.90:0/474826909 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x56321c1ee9b0 0x56321c1f0da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:37.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.068+0000 7fc7e83be740 4 client.30089 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:49:37.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.068+0000 7fc7e83be740 1 -- 172.21.15.90:0/474826909 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56321c0da630 con 0x56321c1ee9b0 2024-04-02T14:49:37.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.070+0000 7fc7dea00640 1 --2- 172.21.15.90:0/474826909 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x56321c1ee9b0 0x56321c1f0da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:37.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.074+0000 7fc7dea00640 1 --2- 172.21.15.90:0/474826909 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x56321c1ee9b0 0x56321c1f0da0 crc :-1 s=READY pgs=376 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d0328cbd8ca50d53 server_cookie=ec468835e41ac61b in_seq=0 out_seq=0 2024-04-02T14:49:37.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.074+0000 7fc7bfe00640 10 client.30089 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:49:37.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.075+0000 7fc7bfe00640 1 -- 172.21.15.90:0/474826909 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56321c0da630 con 0x56321c1ee9b0 2024-04-02T14:49:37.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.075+0000 7fc7bfe00640 10 client.30089 handle_command_reply: tid=0 2024-04-02T14:49:37.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.075+0000 7fc7e83be740 1 client.30089 shutdown 2024-04-02T14:49:37.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.076+0000 7fc7e83be740 1 -- 172.21.15.90:0/474826909 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56321c1d57b0 msgr2=0x56321c1dbd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:49:37.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.076+0000 7fc7e83be740 1 --2- 172.21.15.90:0/474826909 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56321c1d57b0 0x56321c1dbd10 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7fc7cc01a040 tx=0x7fc7cc00aa90 comp rx=0 tx=0).stop 2024-04-02T14:49:37.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.076+0000 7fc7e83be740 1 -- 172.21.15.90:0/474826909 shutdown_connections 2024-04-02T14:49:37.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.076+0000 7fc7e83be740 1 --2- 172.21.15.90:0/474826909 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x56321c1ee9b0 0x56321c1f0da0 crc :-1 s=READY pgs=376 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:37.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.076+0000 7fc7e83be740 1 --2- 172.21.15.90:0/474826909 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56321c1decb0 0x56321c1dc790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:37.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.076+0000 7fc7e83be740 1 --2- 172.21.15.90:0/474826909 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56321c1d60d0 0x56321c1dc250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:37.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.076+0000 7fc7e83be740 1 --2- 172.21.15.90:0/474826909 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56321c1d57b0 0x56321c1dbd10 unknown :-1 s=CLOSED pgs=1006 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:37.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.076+0000 7fc7e83be740 1 -- 172.21.15.90:0/474826909 >> 172.21.15.90:0/474826909 conn(0x56321c1cfee0 msgr2=0x56321c1d1c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:49:37.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.076+0000 7fc7bfe00640 0 client.30089 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:49:37.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.077+0000 7fc7e83be740 1 -- 172.21.15.90:0/474826909 shutdown_connections 2024-04-02T14:49:37.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.077+0000 7fc7e83be740 1 -- 172.21.15.90:0/474826909 wait complete. 2024-04-02T14:49:37.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.078+0000 7fc7e83be740 20 client.30089 trim_cache size 0 max 16384 2024-04-02T14:49:37.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.125+0000 7fc7e83be740 1 Processor -- start 2024-04-02T14:49:37.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.126+0000 7fc7e83be740 1 -- start start 2024-04-02T14:49:37.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.126+0000 7fc7e83be740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56321c1d57b0 0x56321c1cede0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:37.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.126+0000 7fc7e83be740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56321c1d60d0 0x56321c1cf320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:37.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.126+0000 7fc7e83be740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56321c1decb0 0x56321c1dc260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:37.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.126+0000 7fc7e83be740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fc7e01b4130 con 0x56321c1d57b0 2024-04-02T14:49:37.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.126+0000 7fc7e83be740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fc7e01b5c70 con 0x56321c1d60d0 2024-04-02T14:49:37.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.126+0000 7fc7e83be740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fc7e01b4800 con 0x56321c1decb0 2024-04-02T14:49:37.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.127+0000 7fc7dea00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56321c1d60d0 0x56321c1cf320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:37.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.127+0000 7fc7dea00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56321c1d60d0 0x56321c1cf320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:50360/0 (socket says 172.21.15.90:50360) 2024-04-02T14:49:37.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.127+0000 7fc7dea00640 1 -- 172.21.15.90:0/2780992953 learned_addr learned my addr 172.21.15.90:0/2780992953 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:49:37.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.127+0000 7fc7de000640 1 --2- 172.21.15.90:0/2780992953 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56321c1d57b0 0x56321c1cede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:37.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.127+0000 7fc7dea00640 1 -- 172.21.15.90:0/2780992953 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56321c1decb0 msgr2=0x56321c1dc260 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-02T14:49:37.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.127+0000 7fc7dea00640 1 --2- 172.21.15.90:0/2780992953 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56321c1decb0 0x56321c1dc260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:37.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.127+0000 7fc7dea00640 1 -- 172.21.15.90:0/2780992953 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56321c1d57b0 msgr2=0x56321c1cede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:49:37.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.127+0000 7fc7dea00640 1 --2- 172.21.15.90:0/2780992953 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56321c1d57b0 0x56321c1cede0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:37.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.128+0000 7fc7dea00640 1 -- 172.21.15.90:0/2780992953 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56321c1e12d0 con 0x56321c1d60d0 2024-04-02T14:49:37.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.128+0000 7fc7dea00640 1 --2- 172.21.15.90:0/2780992953 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56321c1d60d0 0x56321c1cf320 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7fc7d002c3d0 tx=0x7fc7d0005e70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f380cdf14ca1e838 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:49:37.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.129+0000 7fc7caa00640 1 -- 172.21.15.90:0/2780992953 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7d0049070 con 0x56321c1d60d0 2024-04-02T14:49:37.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.129+0000 7fc7caa00640 1 -- 172.21.15.90:0/2780992953 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc7d0004a80 con 0x56321c1d60d0 2024-04-02T14:49:37.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.129+0000 7fc7caa00640 1 -- 172.21.15.90:0/2780992953 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7d0039040 con 0x56321c1d60d0 2024-04-02T14:49:37.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.131+0000 7fc7e83be740 1 -- 172.21.15.90:0/2780992953 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56321c1d60d0 msgr2=0x56321c1cf320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:49:37.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.131+0000 7fc7e83be740 1 --2- 172.21.15.90:0/2780992953 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56321c1d60d0 0x56321c1cf320 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7fc7d002c3d0 tx=0x7fc7d0005e70 comp rx=0 tx=0).stop 2024-04-02T14:49:37.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.132+0000 7fc7e83be740 1 -- 172.21.15.90:0/2780992953 shutdown_connections 2024-04-02T14:49:37.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.132+0000 7fc7e83be740 1 --2- 172.21.15.90:0/2780992953 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56321c1decb0 0x56321c1dc260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:37.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.132+0000 7fc7e83be740 1 --2- 172.21.15.90:0/2780992953 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56321c1d60d0 0x56321c1cf320 unknown :-1 s=CLOSED pgs=1007 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:37.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.132+0000 7fc7e83be740 1 --2- 172.21.15.90:0/2780992953 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56321c1d57b0 0x56321c1cede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:37.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.132+0000 7fc7e83be740 1 -- 172.21.15.90:0/2780992953 >> 172.21.15.90:0/2780992953 conn(0x56321c1cfee0 msgr2=0x56321c1ce6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:49:37.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.133+0000 7fc7e83be740 1 -- 172.21.15.90:0/2780992953 shutdown_connections 2024-04-02T14:49:37.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.133+0000 7fc7e83be740 1 -- 172.21.15.90:0/2780992953 wait complete. 2024-04-02T14:49:37.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.134+0000 7fc7e83be740 1 -- start start 2024-04-02T14:49:37.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.135+0000 7fc7e83be740 1 Processor -- start 2024-04-02T14:49:37.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.136+0000 7fc7e83be740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56321c1d57b0 0x56321c1f1c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:37.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.136+0000 7fc7e83be740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56321c1d60d0 0x56321c1f2160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:37.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.136+0000 7fc7e83be740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56321c1decb0 0x56321c253b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:37.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.136+0000 7fc7e83be740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fc7e01b55a0 con 0x56321c1decb0 2024-04-02T14:49:37.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.136+0000 7fc7e83be740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fc7e01b4ed0 con 0x56321c1d57b0 2024-04-02T14:49:37.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.136+0000 7fc7e83be740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fc7e01b20f0 con 0x56321c1d60d0 2024-04-02T14:49:37.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.136+0000 7fc7e83be740 10 client.0 _ll_register_callbacks cb 0x7fc7d80093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:49:37.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.137+0000 7fc7dea00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56321c1d60d0 0x56321c1f2160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:37.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.137+0000 7fc7dea00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56321c1d60d0 0x56321c1f2160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:53424/0 (socket says 172.21.15.90:53424) 2024-04-02T14:49:37.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.137+0000 7fc7dea00640 1 -- 172.21.15.90:0/4261562422 learned_addr learned my addr 172.21.15.90:0/4261562422 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:49:37.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.137+0000 7fc7de000640 1 --2- 172.21.15.90:0/4261562422 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56321c1decb0 0x56321c253b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:37.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.137+0000 7fc7de000640 1 --2- 172.21.15.90:0/4261562422 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56321c1d57b0 0x56321c1f1c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:37.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.138+0000 7fc7dea00640 1 -- 172.21.15.90:0/4261562422 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56321c1d57b0 msgr2=0x56321c1f1c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:49:37.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.138+0000 7fc7dea00640 1 --2- 172.21.15.90:0/4261562422 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56321c1d57b0 0x56321c1f1c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:37.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.138+0000 7fc7dea00640 1 -- 172.21.15.90:0/4261562422 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56321c1decb0 msgr2=0x56321c253b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:49:37.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.138+0000 7fc7dea00640 1 --2- 172.21.15.90:0/4261562422 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56321c1decb0 0x56321c253b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:37.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.138+0000 7fc7dea00640 1 -- 172.21.15.90:0/4261562422 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56321c2540d0 con 0x56321c1d60d0 2024-04-02T14:49:37.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.138+0000 7fc7de000640 1 --2- 172.21.15.90:0/4261562422 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56321c1decb0 0x56321c253b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:49:37.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.139+0000 7fc7dea00640 1 --2- 172.21.15.90:0/4261562422 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56321c1d60d0 0x56321c1f2160 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7fc7d0005940 tx=0x7fc7d0004880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=392076421fe6f49f server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:49:37.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.139+0000 7fc7bfe00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:49:37.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.139+0000 7fc7bfe00640 1 -- 172.21.15.90:0/4261562422 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7d0049070 con 0x56321c1d60d0 2024-04-02T14:49:37.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.139+0000 7fc7bfe00640 1 -- 172.21.15.90:0/4261562422 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 8550+0+0 (secure 0 0 0) 0x7fc7d002ce00 con 0x56321c1d60d0 2024-04-02T14:49:37.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.140+0000 7fc7e83be740 1 -- 172.21.15.90:0/4261562422 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc7d0004a80 con 0x56321c1d60d0 2024-04-02T14:49:37.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.140+0000 7fc7bfe00640 1 -- 172.21.15.90:0/4261562422 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fc7d002c800 con 0x56321c1d60d0 2024-04-02T14:49:37.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.140+0000 7fc7e83be740 10 client.29881 fetch_fsmap learned FSMap version 27 2024-04-02T14:49:37.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.140+0000 7fc7e83be740 1 -- 172.21.15.90:0/4261562422 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x56321c2552a0 con 0x56321c1d60d0 2024-04-02T14:49:37.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.141+0000 7fc7bfe00640 1 -- 172.21.15.90:0/4261562422 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc7d0004a80 con 0x56321c1d60d0 2024-04-02T14:49:37.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.141+0000 7fc7bfe00640 1 -- 172.21.15.90:0/4261562422 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fc7d0044760 con 0x56321c1d60d0 2024-04-02T14:49:37.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.141+0000 7fc7e83be740 10 client.29881 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:49:37.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.141+0000 7fc7e83be740 10 client.29881 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:49:37.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.141+0000 7fc7e83be740 20 client.29881 populate_metadata read hostname 'smithi090' 2024-04-02T14:49:37.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.141+0000 7fc7e83be740 1 --2- 172.21.15.90:0/4261562422 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x56321c255970 0x56321c257d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:37.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.141+0000 7fc7e83be740 4 client.29881 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-02T14:49:37.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.141+0000 7fc7e83be740 1 -- 172.21.15.90:0/4261562422 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "scrub status"}) -- 0x56321c146cb0 con 0x56321c255970 2024-04-02T14:49:37.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.141+0000 7fc7de000640 1 --2- 172.21.15.90:0/4261562422 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x56321c255970 0x56321c257d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:37.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.142+0000 7fc7bfe00640 10 client.29881 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:49:37.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.142+0000 7fc7de000640 1 --2- 172.21.15.90:0/4261562422 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x56321c255970 0x56321c257d90 crc :-1 s=READY pgs=377 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4db61de2a7f2ace server_cookie=9061e67ed541fd5b in_seq=0 out_seq=0 2024-04-02T14:49:37.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.143+0000 7fc7bfe00640 1 -- 172.21.15.90:0/4261562422 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x56321c146cb0 con 0x56321c255970 2024-04-02T14:49:37.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.143+0000 7fc7bfe00640 10 client.29881 handle_command_reply: tid=0 2024-04-02T14:49:37.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.143+0000 7fc7e83be740 1 client.29881 shutdown 2024-04-02T14:49:37.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.143+0000 7fc7e83be740 1 -- 172.21.15.90:0/4261562422 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56321c1d60d0 msgr2=0x56321c1f2160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:49:37.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.143+0000 7fc7e83be740 1 --2- 172.21.15.90:0/4261562422 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56321c1d60d0 0x56321c1f2160 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7fc7d0005940 tx=0x7fc7d0004880 comp rx=0 tx=0).stop 2024-04-02T14:49:37.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.144+0000 7fc7e83be740 1 -- 172.21.15.90:0/4261562422 shutdown_connections 2024-04-02T14:49:37.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.144+0000 7fc7e83be740 1 --2- 172.21.15.90:0/4261562422 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x56321c255970 0x56321c257d90 crc :-1 s=READY pgs=377 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:37.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.144+0000 7fc7e83be740 1 --2- 172.21.15.90:0/4261562422 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56321c1decb0 0x56321c253b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:37.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.144+0000 7fc7bfe00640 0 client.29881 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:49:37.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.144+0000 7fc7e83be740 1 --2- 172.21.15.90:0/4261562422 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56321c1d60d0 0x56321c1f2160 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:37.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.144+0000 7fc7e83be740 1 --2- 172.21.15.90:0/4261562422 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56321c1d57b0 0x56321c1f1c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:37.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.144+0000 7fc7e83be740 1 -- 172.21.15.90:0/4261562422 >> 172.21.15.90:0/4261562422 conn(0x56321c1cfee0 msgr2=0x56321c1cef60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:49:37.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.144+0000 7fc7e83be740 1 -- 172.21.15.90:0/4261562422 shutdown_connections 2024-04-02T14:49:37.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.144+0000 7fc7e83be740 1 -- 172.21.15.90:0/4261562422 wait complete. 2024-04-02T14:49:37.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.144+0000 7fc7e83be740 20 client.29881 trim_cache size 0 max 16384 2024-04-02T14:49:37.491 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-02T14:49:37.491 INFO:teuthology.orchestra.run.smithi090.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-02T14:49:37.492 INFO:teuthology.orchestra.run.smithi090.stdout: "scrubs": { 2024-04-02T14:49:37.492 INFO:teuthology.orchestra.run.smithi090.stdout: "703b31e0-ef88-41f1-a3d1-2dd9bcdc6912": { 2024-04-02T14:49:37.492 INFO:teuthology.orchestra.run.smithi090.stdout: "path": "/", 2024-04-02T14:49:37.492 INFO:teuthology.orchestra.run.smithi090.stdout: "tag": "703b31e0-ef88-41f1-a3d1-2dd9bcdc6912", 2024-04-02T14:49:37.492 INFO:teuthology.orchestra.run.smithi090.stdout: "options": "recursive,force" 2024-04-02T14:49:37.492 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:49:37.492 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:49:37.492 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-02T14:49:37.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.145+0000 7fc7e5600640 1 -- 172.21.15.90:0/2471588409 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fc7c40a3da0 msgr2=0x7fc7c40a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:49:37.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.145+0000 7fc7e5600640 1 --2- 172.21.15.90:0/2471588409 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fc7c40a3da0 0x7fc7c40a6260 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fc7d8099800 tx=0x7fc7d8099040 comp rx=0 tx=0).stop 2024-04-02T14:49:37.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.145+0000 7fc7e5600640 1 -- 172.21.15.90:0/2471588409 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc7e0112940 msgr2=0x7fc7e019e900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:49:37.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.145+0000 7fc7e5600640 1 --2- 172.21.15.90:0/2471588409 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc7e0112940 0x7fc7e019e900 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7fc7d800eb40 tx=0x7fc7d800c2d0 comp rx=0 tx=0).stop 2024-04-02T14:49:37.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.145+0000 7fc7e5600640 1 -- 172.21.15.90:0/2471588409 shutdown_connections 2024-04-02T14:49:37.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.145+0000 7fc7e5600640 1 --2- 172.21.15.90:0/2471588409 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fc7c40a3da0 0x7fc7c40a6260 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:37.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.145+0000 7fc7e5600640 1 --2- 172.21.15.90:0/2471588409 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc7e0112940 0x7fc7e019e900 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:37.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.145+0000 7fc7e5600640 1 --2- 172.21.15.90:0/2471588409 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc7e0076950 0x7fc7e01a4c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:37.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.145+0000 7fc7e5600640 1 --2- 172.21.15.90:0/2471588409 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc7e0076030 0x7fc7e01a4750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:37.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.145+0000 7fc7e5600640 1 -- 172.21.15.90:0/2471588409 >> 172.21.15.90:0/2471588409 conn(0x7fc7e00ff7b0 msgr2=0x7fc7e0101ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:49:37.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.146+0000 7fc7e5600640 1 -- 172.21.15.90:0/2471588409 shutdown_connections 2024-04-02T14:49:37.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:49:37.146+0000 7fc7e5600640 1 -- 172.21.15.90:0/2471588409 wait complete. 2024-04-02T14:49:37.494 INFO:tasks.cephfs.filesystem:scrub status for tag:703b31e0-ef88-41f1-a3d1-2dd9bcdc6912 - {'path': '/', 'tag': '703b31e0-ef88-41f1-a3d1-2dd9bcdc6912', 'options': 'recursive,force'} 2024-04-02T14:49:37.494 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:49:37 smithi090 ceph-mon[31674]: scrub queued for path: / 2024-04-02T14:49:37.494 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:49:37 smithi090 ceph-mon[31674]: scrub summary: idle+waiting paths [/] 2024-04-02T14:49:37.495 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:49:37 smithi090 ceph-mon[31674]: scrub summary: active paths [/] 2024-04-02T14:49:37.606 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:49:37 smithi092 ceph-mon[36497]: scrub queued for path: / 2024-04-02T14:49:37.606 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:49:37 smithi092 ceph-mon[36497]: scrub summary: idle+waiting paths [/] 2024-04-02T14:49:37.606 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:49:37 smithi092 ceph-mon[36497]: scrub summary: active paths [/] 2024-04-02T14:49:37.678 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:49:37 smithi177 ceph-mon[36431]: scrub queued for path: / 2024-04-02T14:49:37.678 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:49:37 smithi177 ceph-mon[36431]: scrub summary: idle+waiting paths [/] 2024-04-02T14:49:37.678 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:49:37 smithi177 ceph-mon[36431]: scrub summary: active paths [/] 2024-04-02T14:49:38.602 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:49:38 smithi092 ceph-mon[36497]: pgmap v461: 97 pgs: 97 active+clean; 4.2 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 7.2 MiB/s wr, 686 op/s 2024-04-02T14:49:38.603 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:49:38 smithi092 ceph-mon[36497]: scrub summary: idle+waiting paths [/] 2024-04-02T14:49:38.678 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:49:38 smithi177 ceph-mon[36431]: pgmap v461: 97 pgs: 97 active+clean; 4.2 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 7.2 MiB/s wr, 686 op/s 2024-04-02T14:49:38.678 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:49:38 smithi177 ceph-mon[36431]: scrub summary: idle+waiting paths [/] 2024-04-02T14:49:38.680 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:49:38 smithi090 ceph-mon[31674]: pgmap v461: 97 pgs: 97 active+clean; 4.2 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 7.2 MiB/s wr, 686 op/s 2024-04-02T14:49:38.681 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:49:38 smithi090 ceph-mon[31674]: scrub summary: idle+waiting paths [/] 2024-04-02T14:49:39.220 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.226+0000 7f4c46e00640 1 -- 172.21.15.90:0/3891580547 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4c401078f0 msgr2=0x7f4c40107cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:49:39.220 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.226+0000 7f4c46e00640 1 --2- 172.21.15.90:0/3891580547 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4c401078f0 0x7f4c40107cd0 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f4c34009930 tx=0x7f4c3402bc30 comp rx=0 tx=0).stop 2024-04-02T14:49:39.221 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.226+0000 7f4c46e00640 1 -- 172.21.15.90:0/3891580547 shutdown_connections 2024-04-02T14:49:39.221 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.226+0000 7f4c46e00640 1 --2- 172.21.15.90:0/3891580547 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4c401022a0 0x7f4c4010e7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:39.221 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.226+0000 7f4c46e00640 1 --2- 172.21.15.90:0/3891580547 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4c401018e0 0x7f4c40101d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:39.221 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.227+0000 7f4c46e00640 1 --2- 172.21.15.90:0/3891580547 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4c401078f0 0x7f4c40107cd0 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:39.221 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.227+0000 7f4c46e00640 1 -- 172.21.15.90:0/3891580547 >> 172.21.15.90:0/3891580547 conn(0x7f4c400fd450 msgr2=0x7f4c400ff870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:49:39.221 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.227+0000 7f4c46e00640 1 -- 172.21.15.90:0/3891580547 shutdown_connections 2024-04-02T14:49:39.221 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.227+0000 7f4c46e00640 1 -- 172.21.15.90:0/3891580547 wait complete. 2024-04-02T14:49:39.222 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.228+0000 7f4c46e00640 1 Processor -- start 2024-04-02T14:49:39.222 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.228+0000 7f4c46e00640 1 -- start start 2024-04-02T14:49:39.224 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.229+0000 7f4c46e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4c401018e0 0x7f4c40198200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:39.224 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.229+0000 7f4c46e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4c401022a0 0x7f4c40198740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:39.224 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.229+0000 7f4c46e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4c401078f0 0x7f4c4019cac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:39.224 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.229+0000 7f4c46e00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f4c40111cc0 con 0x7f4c401018e0 2024-04-02T14:49:39.224 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.229+0000 7f4c46e00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f4c40111b40 con 0x7f4c401078f0 2024-04-02T14:49:39.224 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.229+0000 7f4c46e00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f4c40111e40 con 0x7f4c401022a0 2024-04-02T14:49:39.224 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.229+0000 7f4c45a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4c401018e0 0x7f4c40198200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:39.224 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.229+0000 7f4c46400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4c401078f0 0x7f4c4019cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:39.224 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.229+0000 7f4c46400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4c401078f0 0x7f4c4019cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:50378/0 (socket says 172.21.15.90:50378) 2024-04-02T14:49:39.224 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.229+0000 7f4c46400640 1 -- 172.21.15.90:0/4133944810 learned_addr learned my addr 172.21.15.90:0/4133944810 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:49:39.224 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.229+0000 7f4c45000640 1 --2- 172.21.15.90:0/4133944810 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4c401022a0 0x7f4c40198740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:39.225 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.229+0000 7f4c46400640 1 -- 172.21.15.90:0/4133944810 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4c401022a0 msgr2=0x7f4c40198740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:49:39.225 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.229+0000 7f4c46400640 1 --2- 172.21.15.90:0/4133944810 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4c401022a0 0x7f4c40198740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:39.225 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.229+0000 7f4c46400640 1 -- 172.21.15.90:0/4133944810 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4c401018e0 msgr2=0x7f4c40198200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:49:39.225 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.229+0000 7f4c46400640 1 --2- 172.21.15.90:0/4133944810 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4c401018e0 0x7f4c40198200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:39.225 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.229+0000 7f4c46400640 1 -- 172.21.15.90:0/4133944810 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c4019d1f0 con 0x7f4c401078f0 2024-04-02T14:49:39.225 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.230+0000 7f4c46400640 1 --2- 172.21.15.90:0/4133944810 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4c401078f0 0x7f4c4019cac0 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7f4c3800eb10 tx=0x7f4c3800c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:49:39.225 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.230+0000 7f4c3e000640 1 -- 172.21.15.90:0/4133944810 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c38004b20 con 0x7f4c401078f0 2024-04-02T14:49:39.225 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.230+0000 7f4c46e00640 1 -- 172.21.15.90:0/4133944810 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c4019d4e0 con 0x7f4c401078f0 2024-04-02T14:49:39.225 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.230+0000 7f4c3e000640 1 -- 172.21.15.90:0/4133944810 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4c38004cc0 con 0x7f4c401078f0 2024-04-02T14:49:39.225 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.230+0000 7f4c3e000640 1 -- 172.21.15.90:0/4133944810 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c38010760 con 0x7f4c401078f0 2024-04-02T14:49:39.225 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.230+0000 7f4c46e00640 1 -- 172.21.15.90:0/4133944810 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c401a4d70 con 0x7f4c401078f0 2024-04-02T14:49:39.227 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.233+0000 7f4c4b231740 1 Processor -- start 2024-04-02T14:49:39.228 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.234+0000 7f4c3e000640 1 -- 172.21.15.90:0/4133944810 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f4c38005020 con 0x7f4c401078f0 2024-04-02T14:49:39.228 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.235+0000 7f4c4b231740 1 -- start start 2024-04-02T14:49:39.230 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.236+0000 7f4c4b231740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eb434f54f0 0x55eb434f58d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:39.230 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.236+0000 7f4c4b231740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eb434f5e10 0x55eb434fe4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:39.230 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.236+0000 7f4c4b231740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55eb434fe9f0 0x55eb43500de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:39.230 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.236+0000 7f4c4b231740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55eb43455d70 con 0x55eb434f5e10 2024-04-02T14:49:39.230 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.236+0000 7f4c4b231740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55eb433fe2a0 con 0x55eb434fe9f0 2024-04-02T14:49:39.231 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.236+0000 7f4c4b231740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55eb43451d20 con 0x55eb434f54f0 2024-04-02T14:49:39.236 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.237+0000 7f4c3e000640 1 --2- 172.21.15.90:0/4133944810 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f4c280a3d90 0x7f4c280a6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:39.236 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.237+0000 7f4c45000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eb434f54f0 0x55eb434f58d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:39.236 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.237+0000 7f4c45000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eb434f54f0 0x55eb434f58d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:53450/0 (socket says 172.21.15.90:53450) 2024-04-02T14:49:39.236 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.237+0000 7f4c45000640 1 -- 172.21.15.90:0/2001875857 learned_addr learned my addr 172.21.15.90:0/2001875857 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:49:39.236 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.237+0000 7f4c45a00640 1 --2- 172.21.15.90:0/2001875857 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eb434f5e10 0x55eb434fe4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:39.236 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.237+0000 7f4c3e000640 1 -- 172.21.15.90:0/4133944810 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 8550+0+0 (secure 0 0 0) 0x7f4c380d0cd0 con 0x7f4c401078f0 2024-04-02T14:49:39.236 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.237+0000 7f4c45000640 1 --2- 172.21.15.90:0/2001875857 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55eb434fe9f0 0x55eb43500de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:39.236 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.237+0000 7f4c45a00640 1 -- 172.21.15.90:0/2001875857 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eb434f54f0 msgr2=0x55eb434f58d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:49:39.237 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.237+0000 7f4c45a00640 1 --2- 172.21.15.90:0/2001875857 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eb434f54f0 0x55eb434f58d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:39.237 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.237+0000 7f4c45a00640 1 -- 172.21.15.90:0/2001875857 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55eb434fe9f0 msgr2=0x55eb43500de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:49:39.237 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.237+0000 7f4c45a00640 1 --2- 172.21.15.90:0/2001875857 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55eb434fe9f0 0x55eb43500de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:39.237 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.237+0000 7f4c45a00640 1 -- 172.21.15.90:0/2001875857 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55eb43412840 con 0x55eb434f5e10 2024-04-02T14:49:39.237 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.238+0000 7f4c45000640 1 --2- 172.21.15.90:0/2001875857 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55eb434fe9f0 0x55eb43500de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:49:39.237 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.238+0000 7f4c46400640 1 --2- 172.21.15.90:0/4133944810 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f4c280a3d90 0x7f4c280a6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:39.237 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.238+0000 7f4c45a00640 1 --2- 172.21.15.90:0/2001875857 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eb434f5e10 0x55eb434fe4b0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f4c34009230 tx=0x7f4c34005bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c744e733dc426ba server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:49:39.237 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.239+0000 7f4c32a00640 1 -- 172.21.15.90:0/2001875857 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c34042070 con 0x55eb434f5e10 2024-04-02T14:49:39.237 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.239+0000 7f4c32a00640 1 -- 172.21.15.90:0/2001875857 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4c3402c690 con 0x55eb434f5e10 2024-04-02T14:49:39.237 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.239+0000 7f4c32a00640 1 -- 172.21.15.90:0/2001875857 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c34004f50 con 0x55eb434f5e10 2024-04-02T14:49:39.237 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.239+0000 7f4c4b231740 1 -- 172.21.15.90:0/2001875857 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eb434f5e10 msgr2=0x55eb434fe4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:49:39.237 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.239+0000 7f4c4b231740 1 --2- 172.21.15.90:0/2001875857 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eb434f5e10 0x55eb434fe4b0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f4c34009230 tx=0x7f4c34005bf0 comp rx=0 tx=0).stop 2024-04-02T14:49:39.237 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.239+0000 7f4c46400640 1 --2- 172.21.15.90:0/4133944810 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f4c280a3d90 0x7f4c280a6250 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f4c380ce4b0 tx=0x7f4c38005b40 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:49:39.238 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.240+0000 7f4c4b231740 1 -- 172.21.15.90:0/2001875857 shutdown_connections 2024-04-02T14:49:39.238 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.240+0000 7f4c4b231740 1 --2- 172.21.15.90:0/2001875857 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55eb434fe9f0 0x55eb43500de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:39.238 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.240+0000 7f4c4b231740 1 --2- 172.21.15.90:0/2001875857 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eb434f5e10 0x55eb434fe4b0 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:39.238 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.240+0000 7f4c4b231740 1 --2- 172.21.15.90:0/2001875857 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eb434f54f0 0x55eb434f58d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:39.238 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.240+0000 7f4c4b231740 1 -- 172.21.15.90:0/2001875857 >> 172.21.15.90:0/2001875857 conn(0x55eb434efc20 msgr2=0x55eb434f0000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:49:39.238 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.240+0000 7f4c4b231740 1 -- 172.21.15.90:0/2001875857 shutdown_connections 2024-04-02T14:49:39.238 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.240+0000 7f4c4b231740 1 -- 172.21.15.90:0/2001875857 wait complete. 2024-04-02T14:49:39.238 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.241+0000 7f4c4b231740 1 -- start start 2024-04-02T14:49:39.238 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.241+0000 7f4c4b231740 1 Processor -- start 2024-04-02T14:49:39.238 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.241+0000 7f4c4b231740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eb434f54f0 0x55eb434f7c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:39.238 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.241+0000 7f4c4b231740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eb434f5e10 0x55eb434f8180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:39.238 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.241+0000 7f4c4b231740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55eb434fe9f0 0x55eb434fc400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:39.238 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.241+0000 7f4c4b231740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55eb434eb240 con 0x55eb434f5e10 2024-04-02T14:49:39.238 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.241+0000 7f4c4b231740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55eb43411660 con 0x55eb434fe9f0 2024-04-02T14:49:39.239 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.241+0000 7f4c4b231740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55eb434eb3c0 con 0x55eb434f54f0 2024-04-02T14:49:39.239 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.241+0000 7f4c4b231740 10 client.0 _ll_register_callbacks cb 0x55eb43411050 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:49:39.239 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.242+0000 7f4c45a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eb434f5e10 0x55eb434f8180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:39.239 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.242+0000 7f4c45a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eb434f5e10 0x55eb434f8180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:46518/0 (socket says 172.21.15.90:46518) 2024-04-02T14:49:39.239 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.242+0000 7f4c45a00640 1 -- 172.21.15.90:0/836039719 learned_addr learned my addr 172.21.15.90:0/836039719 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:49:39.239 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.242+0000 7f4c45000640 1 --2- 172.21.15.90:0/836039719 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eb434f54f0 0x55eb434f7c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:39.239 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.242+0000 7f4c45000640 1 --2- 172.21.15.90:0/836039719 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55eb434fe9f0 0x55eb434fc400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:39.239 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.242+0000 7f4c45a00640 1 -- 172.21.15.90:0/836039719 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eb434f54f0 msgr2=0x55eb434f7c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:49:39.239 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.242+0000 7f4c45a00640 1 --2- 172.21.15.90:0/836039719 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eb434f54f0 0x55eb434f7c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:39.239 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.242+0000 7f4c45a00640 1 -- 172.21.15.90:0/836039719 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55eb434fe9f0 msgr2=0x55eb434fc400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:49:39.239 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.242+0000 7f4c45a00640 1 --2- 172.21.15.90:0/836039719 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55eb434fe9f0 0x55eb434fc400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:39.239 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.242+0000 7f4c45a00640 1 -- 172.21.15.90:0/836039719 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55eb43420e80 con 0x55eb434f5e10 2024-04-02T14:49:39.239 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.242+0000 7f4c45000640 1 --2- 172.21.15.90:0/836039719 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55eb434fe9f0 0x55eb434fc400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:49:39.239 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.242+0000 7f4c45a00640 1 --2- 172.21.15.90:0/836039719 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eb434f5e10 0x55eb434f8180 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f4c34036040 tx=0x7f4c34004e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3cfa1a739c2857 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:49:39.239 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.242+0000 7f4c27400640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-02T14:49:39.241 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.243+0000 7f4c27400640 1 -- 172.21.15.90:0/836039719 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c34042040 con 0x55eb434f5e10 2024-04-02T14:49:39.241 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.243+0000 7f4c27400640 1 -- 172.21.15.90:0/836039719 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 8550+0+0 (secure 0 0 0) 0x7f4c34004040 con 0x55eb434f5e10 2024-04-02T14:49:39.241 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.243+0000 7f4c4b231740 1 -- 172.21.15.90:0/836039719 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55eb43405cd0 con 0x55eb434f5e10 2024-04-02T14:49:39.241 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.244+0000 7f4c27400640 1 -- 172.21.15.90:0/836039719 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f4c34004240 con 0x55eb434f5e10 2024-04-02T14:49:39.241 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.245+0000 7f4c4b231740 10 client.18810 fetch_fsmap learned FSMap version 27 2024-04-02T14:49:39.241 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.245+0000 7f4c4b231740 1 -- 172.21.15.90:0/836039719 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55eb4316a210 con 0x55eb434f5e10 2024-04-02T14:49:39.241 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.245+0000 7f4c27400640 1 -- 172.21.15.90:0/836039719 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55eb43405cd0 con 0x55eb434f5e10 2024-04-02T14:49:39.242 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.246+0000 7f4c27400640 1 -- 172.21.15.90:0/836039719 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f4c340038b0 con 0x55eb434f5e10 2024-04-02T14:49:39.242 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.246+0000 7f4c4b231740 10 client.18810 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:49:39.242 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.246+0000 7f4c4b231740 10 client.18810 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-02T14:49:39.242 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.246+0000 7f4c4b231740 20 client.18810 populate_metadata read hostname 'smithi090' 2024-04-02T14:49:39.242 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.246+0000 7f4c4b231740 1 --2- 172.21.15.90:0/836039719 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55eb4350a840 0x55eb4350cc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:39.242 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.246+0000 7f4c4b231740 4 client.18810 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:49:39.242 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.246+0000 7f4c4b231740 1 -- 172.21.15.90:0/836039719 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55eb434632b0 con 0x55eb4350a840 2024-04-02T14:49:39.242 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.246+0000 7f4c45000640 1 --2- 172.21.15.90:0/836039719 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55eb4350a840 0x55eb4350cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:39.242 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.247+0000 7f4c45000640 1 --2- 172.21.15.90:0/836039719 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55eb4350a840 0x55eb4350cc30 crc :-1 s=READY pgs=378 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5e30f354c6ac3cc9 server_cookie=e3e2e706fbd4e48 in_seq=0 out_seq=0 2024-04-02T14:49:39.242 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.247+0000 7f4c27400640 10 client.18810 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:49:39.242 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.247+0000 7f4c27400640 1 -- 172.21.15.90:0/836039719 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55eb434632b0 con 0x55eb4350a840 2024-04-02T14:49:39.242 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.247+0000 7f4c27400640 10 client.18810 handle_command_reply: tid=0 2024-04-02T14:49:39.242 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.247+0000 7f4c4b231740 1 client.18810 shutdown 2024-04-02T14:49:39.243 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.248+0000 7f4c4b231740 1 -- 172.21.15.90:0/836039719 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eb434f5e10 msgr2=0x55eb434f8180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:49:39.243 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.248+0000 7f4c4b231740 1 --2- 172.21.15.90:0/836039719 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eb434f5e10 0x55eb434f8180 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f4c34036040 tx=0x7f4c34004e30 comp rx=0 tx=0).stop 2024-04-02T14:49:39.244 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.248+0000 7f4c4b231740 1 -- 172.21.15.90:0/836039719 shutdown_connections 2024-04-02T14:49:39.244 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.248+0000 7f4c4b231740 1 --2- 172.21.15.90:0/836039719 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55eb4350a840 0x55eb4350cc30 crc :-1 s=READY pgs=378 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:39.244 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.248+0000 7f4c4b231740 1 --2- 172.21.15.90:0/836039719 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55eb434fe9f0 0x55eb434fc400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:39.244 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.248+0000 7f4c4b231740 1 --2- 172.21.15.90:0/836039719 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eb434f5e10 0x55eb434f8180 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:39.244 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.248+0000 7f4c4b231740 1 --2- 172.21.15.90:0/836039719 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eb434f54f0 0x55eb434f7c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:39.244 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.248+0000 7f4c4b231740 1 -- 172.21.15.90:0/836039719 >> 172.21.15.90:0/836039719 conn(0x55eb434efc20 msgr2=0x55eb434f1970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:49:39.244 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.248+0000 7f4c27400640 0 client.18810 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:49:39.244 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.249+0000 7f4c4b231740 1 -- 172.21.15.90:0/836039719 shutdown_connections 2024-04-02T14:49:39.244 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.249+0000 7f4c4b231740 1 -- 172.21.15.90:0/836039719 wait complete. 2024-04-02T14:49:39.244 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.250+0000 7f4c4b231740 20 client.18810 trim_cache size 0 max 16384 2024-04-02T14:49:39.267 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.273+0000 7f4c4b231740 1 Processor -- start 2024-04-02T14:49:39.267 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.273+0000 7f4c4b231740 1 -- start start 2024-04-02T14:49:39.267 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.273+0000 7f4c4b231740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55eb434f54f0 0x55eb434fc260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:39.268 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.273+0000 7f4c4b231740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eb434f5e10 0x55eb434ea6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:39.268 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.273+0000 7f4c45000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55eb434f54f0 0x55eb434fc260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:39.268 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.274+0000 7f4c45000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55eb434f54f0 0x55eb434fc260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:50408/0 (socket says 172.21.15.90:50408) 2024-04-02T14:49:39.268 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.274+0000 7f4c4b231740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eb434fe9f0 0x55eb43512260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:39.268 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.274+0000 7f4c4b231740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f4c401ab550 con 0x55eb434f5e10 2024-04-02T14:49:39.268 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.274+0000 7f4c45000640 1 -- 172.21.15.90:0/3145662901 learned_addr learned my addr 172.21.15.90:0/3145662901 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:49:39.268 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.274+0000 7f4c4b231740 1 -- 172.21.15.90:0/3145662901 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f4c401ad090 con 0x55eb434f54f0 2024-04-02T14:49:39.268 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.274+0000 7f4c4b231740 1 -- 172.21.15.90:0/3145662901 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f4c401abc20 con 0x55eb434fe9f0 2024-04-02T14:49:39.271 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.274+0000 7f4c45000640 1 --2- 172.21.15.90:0/3145662901 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eb434fe9f0 0x55eb43512260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:39.271 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.274+0000 7f4c45000640 1 -- 172.21.15.90:0/3145662901 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eb434fe9f0 msgr2=0x55eb43512260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:49:39.271 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.274+0000 7f4c45000640 1 --2- 172.21.15.90:0/3145662901 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eb434fe9f0 0x55eb43512260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:39.271 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.274+0000 7f4c45000640 1 -- 172.21.15.90:0/3145662901 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eb434f5e10 msgr2=0x55eb434ea6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:49:39.272 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.274+0000 7f4c45000640 1 --2- 172.21.15.90:0/3145662901 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eb434f5e10 0x55eb434ea6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:39.272 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.274+0000 7f4c45000640 1 -- 172.21.15.90:0/3145662901 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55eb43236620 con 0x55eb434f54f0 2024-04-02T14:49:39.272 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.274+0000 7f4c45000640 1 --2- 172.21.15.90:0/3145662901 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55eb434f54f0 0x55eb434fc260 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7f4c2c005fd0 tx=0x7f4c2c004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d3e3edb7e28b4906 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:49:39.272 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.275+0000 7f4c32000640 1 -- 172.21.15.90:0/3145662901 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c2c00fc80 con 0x55eb434f54f0 2024-04-02T14:49:39.272 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.275+0000 7f4c32000640 1 -- 172.21.15.90:0/3145662901 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4c2c00fe20 con 0x55eb434f54f0 2024-04-02T14:49:39.272 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.275+0000 7f4c32000640 1 -- 172.21.15.90:0/3145662901 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c2c0047a0 con 0x55eb434f54f0 2024-04-02T14:49:39.272 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.278+0000 7f4c4b231740 1 -- 172.21.15.90:0/3145662901 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55eb434f54f0 msgr2=0x55eb434fc260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:49:39.272 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.278+0000 7f4c4b231740 1 --2- 172.21.15.90:0/3145662901 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55eb434f54f0 0x55eb434fc260 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7f4c2c005fd0 tx=0x7f4c2c004730 comp rx=0 tx=0).stop 2024-04-02T14:49:39.278 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.285+0000 7f4c4b231740 1 -- 172.21.15.90:0/3145662901 shutdown_connections 2024-04-02T14:49:39.279 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.285+0000 7f4c4b231740 1 --2- 172.21.15.90:0/3145662901 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eb434fe9f0 0x55eb43512260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:39.279 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.285+0000 7f4c4b231740 1 --2- 172.21.15.90:0/3145662901 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eb434f5e10 0x55eb434ea6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:39.279 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.285+0000 7f4c4b231740 1 --2- 172.21.15.90:0/3145662901 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55eb434f54f0 0x55eb434fc260 unknown :-1 s=CLOSED pgs=1009 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:39.279 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.285+0000 7f4c4b231740 1 -- 172.21.15.90:0/3145662901 >> 172.21.15.90:0/3145662901 conn(0x55eb434efc20 msgr2=0x55eb434e9fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:49:39.279 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.286+0000 7f4c4b231740 1 -- 172.21.15.90:0/3145662901 shutdown_connections 2024-04-02T14:49:39.279 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.286+0000 7f4c4b231740 1 -- 172.21.15.90:0/3145662901 wait complete. 2024-04-02T14:49:39.280 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.286+0000 7f4c4b231740 1 -- start start 2024-04-02T14:49:39.280 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.286+0000 7f4c4b231740 1 Processor -- start 2024-04-02T14:49:39.281 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.287+0000 7f4c4b231740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55eb434f54f0 0x55eb43503550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:39.281 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.287+0000 7f4c4b231740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eb434f5e10 0x55eb43503a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:39.281 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.287+0000 7f4c4b231740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eb434fe9f0 0x55eb43580ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:39.281 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.287+0000 7f4c4b231740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f4c401ac9c0 con 0x55eb434f5e10 2024-04-02T14:49:39.281 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.287+0000 7f4c4b231740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f4c401ac2f0 con 0x55eb434f54f0 2024-04-02T14:49:39.281 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.287+0000 7f4c4b231740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f4c401a9550 con 0x55eb434fe9f0 2024-04-02T14:49:39.281 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.287+0000 7f4c4b231740 10 client.0 _ll_register_callbacks cb 0x7f4c34004ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:49:39.289 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.295+0000 7f4c45000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55eb434f54f0 0x55eb43503550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:39.289 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.295+0000 7f4c45000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55eb434f54f0 0x55eb43503550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:50412/0 (socket says 172.21.15.90:50412) 2024-04-02T14:49:39.289 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.295+0000 7f4c45000640 1 -- 172.21.15.90:0/1106833197 learned_addr learned my addr 172.21.15.90:0/1106833197 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:49:39.289 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.295+0000 7f4c45000640 1 --2- 172.21.15.90:0/1106833197 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eb434fe9f0 0x55eb43580ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:39.291 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.297+0000 7f4c45a00640 1 --2- 172.21.15.90:0/1106833197 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eb434f5e10 0x55eb43503a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:39.292 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.298+0000 7f4c45000640 1 -- 172.21.15.90:0/1106833197 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eb434fe9f0 msgr2=0x55eb43580ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:49:39.292 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.298+0000 7f4c45000640 1 --2- 172.21.15.90:0/1106833197 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eb434fe9f0 0x55eb43580ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:39.292 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.298+0000 7f4c45000640 1 -- 172.21.15.90:0/1106833197 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eb434f5e10 msgr2=0x55eb43503a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:49:39.292 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.298+0000 7f4c45000640 1 --2- 172.21.15.90:0/1106833197 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eb434f5e10 0x55eb43503a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:39.292 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.298+0000 7f4c45000640 1 -- 172.21.15.90:0/1106833197 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55eb43504320 con 0x55eb434f54f0 2024-04-02T14:49:39.292 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.298+0000 7f4c45a00640 1 --2- 172.21.15.90:0/1106833197 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eb434f5e10 0x55eb43503a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:49:39.295 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.300+0000 7f4c45000640 1 --2- 172.21.15.90:0/1106833197 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55eb434f54f0 0x55eb43503550 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7f4c2c0051c0 tx=0x7f4c2c01c040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=77192088c8eb4503 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:49:39.296 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.301+0000 7f4c27400640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:49:39.296 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.301+0000 7f4c27400640 1 -- 172.21.15.90:0/1106833197 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c2c005430 con 0x55eb434f54f0 2024-04-02T14:49:39.296 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.301+0000 7f4c4b231740 1 -- 172.21.15.90:0/1106833197 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4c2c00fe20 con 0x55eb434f54f0 2024-04-02T14:49:39.296 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.301+0000 7f4c27400640 1 -- 172.21.15.90:0/1106833197 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 8550+0+0 (secure 0 0 0) 0x7f4c2c004020 con 0x55eb434f54f0 2024-04-02T14:49:39.296 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.301+0000 7f4c27400640 1 -- 172.21.15.90:0/1106833197 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f4c2c049050 con 0x55eb434f54f0 2024-04-02T14:49:39.296 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.301+0000 7f4c4b231740 10 client.30110 fetch_fsmap learned FSMap version 27 2024-04-02T14:49:39.296 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.301+0000 7f4c4b231740 1 -- 172.21.15.90:0/1106833197 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55eb43582160 con 0x55eb434f54f0 2024-04-02T14:49:39.296 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.302+0000 7f4c27400640 1 -- 172.21.15.90:0/1106833197 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4c2c00fe20 con 0x55eb434f54f0 2024-04-02T14:49:39.297 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.302+0000 7f4c27400640 1 -- 172.21.15.90:0/1106833197 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f4c2c04c020 con 0x55eb434f54f0 2024-04-02T14:49:39.297 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.302+0000 7f4c4b231740 10 client.30110 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:49:39.297 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.302+0000 7f4c4b231740 10 client.30110 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-02T14:49:39.297 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.302+0000 7f4c4b231740 20 client.30110 populate_metadata read hostname 'smithi090' 2024-04-02T14:49:39.297 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.302+0000 7f4c4b231740 1 --2- 172.21.15.90:0/1106833197 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55eb43582870 0x55eb43584c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:39.297 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.303+0000 7f4c4b231740 4 client.30110 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-02T14:49:39.297 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.303+0000 7f4c4b231740 1 -- 172.21.15.90:0/1106833197 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55eb4345d600 con 0x55eb43582870 2024-04-02T14:49:39.297 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.303+0000 7f4c45a00640 1 --2- 172.21.15.90:0/1106833197 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55eb43582870 0x55eb43584c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:39.298 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.304+0000 7f4c27400640 10 client.30110 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:49:39.298 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.304+0000 7f4c45a00640 1 --2- 172.21.15.90:0/1106833197 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55eb43582870 0x55eb43584c90 crc :-1 s=READY pgs=379 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f08500ca95dda2e2 server_cookie=b7735cba8fcf06c6 in_seq=0 out_seq=0 2024-04-02T14:49:39.408 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.413+0000 7f4c27400640 1 -- 172.21.15.90:0/1106833197 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55eb4345d600 con 0x55eb43582870 2024-04-02T14:49:39.408 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.413+0000 7f4c27400640 10 client.30110 handle_command_reply: tid=0 2024-04-02T14:49:39.408 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.413+0000 7f4c4b231740 1 client.30110 shutdown 2024-04-02T14:49:39.408 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.414+0000 7f4c4b231740 1 -- 172.21.15.90:0/1106833197 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55eb434f54f0 msgr2=0x55eb43503550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:49:39.408 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.414+0000 7f4c4b231740 1 --2- 172.21.15.90:0/1106833197 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55eb434f54f0 0x55eb43503550 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7f4c2c0051c0 tx=0x7f4c2c01c040 comp rx=0 tx=0).stop 2024-04-02T14:49:39.408 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.414+0000 7f4c4b231740 1 -- 172.21.15.90:0/1106833197 shutdown_connections 2024-04-02T14:49:39.408 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.414+0000 7f4c4b231740 1 --2- 172.21.15.90:0/1106833197 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55eb43582870 0x55eb43584c90 crc :-1 s=READY pgs=379 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:39.408 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.414+0000 7f4c4b231740 1 --2- 172.21.15.90:0/1106833197 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eb434fe9f0 0x55eb43580ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:39.408 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.414+0000 7f4c4b231740 1 --2- 172.21.15.90:0/1106833197 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eb434f5e10 0x55eb43503a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:39.408 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.414+0000 7f4c4b231740 1 --2- 172.21.15.90:0/1106833197 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55eb434f54f0 0x55eb43503550 unknown :-1 s=CLOSED pgs=1010 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:39.408 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.414+0000 7f4c4b231740 1 -- 172.21.15.90:0/1106833197 >> 172.21.15.90:0/1106833197 conn(0x55eb434efc20 msgr2=0x55eb434ea870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:49:39.408 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.414+0000 7f4c27400640 0 client.30110 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:49:39.409 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.414+0000 7f4c4b231740 1 -- 172.21.15.90:0/1106833197 shutdown_connections 2024-04-02T14:49:39.409 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.414+0000 7f4c4b231740 1 -- 172.21.15.90:0/1106833197 wait complete. 2024-04-02T14:49:39.409 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.415+0000 7f4c4b231740 20 client.30110 trim_cache size 0 max 16384 2024-04-02T14:49:39.410 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:{ 2024-04-02T14:49:39.410 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "message": "", 2024-04-02T14:49:39.410 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "return_code": 0 2024-04-02T14:49:39.410 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:} 2024-04-02T14:49:39.410 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.416+0000 7f4c46e00640 1 -- 172.21.15.90:0/4133944810 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f4c280a3d90 msgr2=0x7f4c280a6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:49:39.410 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.416+0000 7f4c46e00640 1 --2- 172.21.15.90:0/4133944810 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f4c280a3d90 0x7f4c280a6250 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f4c380ce4b0 tx=0x7f4c38005b40 comp rx=0 tx=0).stop 2024-04-02T14:49:39.411 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.416+0000 7f4c46e00640 1 -- 172.21.15.90:0/4133944810 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4c401078f0 msgr2=0x7f4c4019cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:49:39.411 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.416+0000 7f4c46e00640 1 --2- 172.21.15.90:0/4133944810 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4c401078f0 0x7f4c4019cac0 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7f4c3800eb10 tx=0x7f4c3800c2d0 comp rx=0 tx=0).stop 2024-04-02T14:49:39.411 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.416+0000 7f4c46e00640 1 -- 172.21.15.90:0/4133944810 shutdown_connections 2024-04-02T14:49:39.411 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.416+0000 7f4c46e00640 1 --2- 172.21.15.90:0/4133944810 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f4c280a3d90 0x7f4c280a6250 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:39.411 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.416+0000 7f4c46e00640 1 --2- 172.21.15.90:0/4133944810 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4c401078f0 0x7f4c4019cac0 unknown :-1 s=CLOSED pgs=1008 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:39.411 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.416+0000 7f4c46e00640 1 --2- 172.21.15.90:0/4133944810 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4c401022a0 0x7f4c40198740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:39.411 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.416+0000 7f4c46e00640 1 --2- 172.21.15.90:0/4133944810 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4c401018e0 0x7f4c40198200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:39.411 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.416+0000 7f4c46e00640 1 -- 172.21.15.90:0/4133944810 >> 172.21.15.90:0/4133944810 conn(0x7f4c400fd450 msgr2=0x7f4c400fdcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:49:39.411 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.417+0000 7f4c46e00640 1 -- 172.21.15.90:0/4133944810 shutdown_connections 2024-04-02T14:49:39.411 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:39.417+0000 7f4c46e00640 1 -- 172.21.15.90:0/4133944810 wait complete. 2024-04-02T14:49:40.602 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:49:40 smithi092 ceph-mon[36497]: pgmap v462: 97 pgs: 97 active+clean; 4.2 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 5.6 MiB/s wr, 571 op/s 2024-04-02T14:49:40.678 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:49:40 smithi177 ceph-mon[36431]: pgmap v462: 97 pgs: 97 active+clean; 4.2 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 5.6 MiB/s wr, 571 op/s 2024-04-02T14:49:40.680 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:49:40 smithi090 ceph-mon[31674]: pgmap v462: 97 pgs: 97 active+clean; 4.2 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 5.6 MiB/s wr, 571 op/s 2024-04-02T14:49:41.823 INFO:teuthology.orchestra.run.smithi090.stderr:progress: 850.0 s, 1826.0 tps, lat 53.682 ms stddev 66.449 2024-04-02T14:49:42.603 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:49:42 smithi092 ceph-mon[36497]: pgmap v463: 97 pgs: 97 active+clean; 4.3 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 9.6 MiB/s wr, 915 op/s 2024-04-02T14:49:42.603 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:49:42 smithi092 ceph-mon[36497]: scrub summary: idle 2024-04-02T14:49:42.678 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:49:42 smithi177 ceph-mon[36431]: pgmap v463: 97 pgs: 97 active+clean; 4.3 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 9.6 MiB/s wr, 915 op/s 2024-04-02T14:49:42.678 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:49:42 smithi177 ceph-mon[36431]: scrub summary: idle 2024-04-02T14:49:42.681 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:49:42 smithi090 ceph-mon[31674]: pgmap v463: 97 pgs: 97 active+clean; 4.3 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 9.6 MiB/s wr, 915 op/s 2024-04-02T14:49:42.681 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:49:42 smithi090 ceph-mon[31674]: scrub summary: idle 2024-04-02T14:49:44.602 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:49:44 smithi092 ceph-mon[36497]: pgmap v464: 97 pgs: 97 active+clean; 4.3 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 7.2 MiB/s wr, 663 op/s 2024-04-02T14:49:44.678 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:49:44 smithi177 ceph-mon[36431]: pgmap v464: 97 pgs: 97 active+clean; 4.3 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 7.2 MiB/s wr, 663 op/s 2024-04-02T14:49:44.680 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:49:44 smithi090 ceph-mon[31674]: pgmap v464: 97 pgs: 97 active+clean; 4.3 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 7.2 MiB/s wr, 663 op/s 2024-04-02T14:49:46.602 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:49:46 smithi092 ceph-mon[36497]: pgmap v465: 97 pgs: 97 active+clean; 4.4 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 18 MiB/s wr, 1.55k op/s 2024-04-02T14:49:46.678 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:49:46 smithi177 ceph-mon[36431]: pgmap v465: 97 pgs: 97 active+clean; 4.4 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 18 MiB/s wr, 1.55k op/s 2024-04-02T14:49:46.680 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:49:46 smithi090 ceph-mon[31674]: pgmap v465: 97 pgs: 97 active+clean; 4.4 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 18 MiB/s wr, 1.55k op/s 2024-04-02T14:49:46.823 INFO:teuthology.orchestra.run.smithi090.stderr:progress: 855.0 s, 1651.4 tps, lat 60.548 ms stddev 130.730 2024-04-02T14:49:48.602 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:49:48 smithi092 ceph-mon[36497]: pgmap v466: 97 pgs: 97 active+clean; 4.4 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 20 MiB/s wr, 1.74k op/s 2024-04-02T14:49:48.678 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:49:48 smithi177 ceph-mon[36431]: pgmap v466: 97 pgs: 97 active+clean; 4.4 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 20 MiB/s wr, 1.74k op/s 2024-04-02T14:49:48.680 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:49:48 smithi090 ceph-mon[31674]: pgmap v466: 97 pgs: 97 active+clean; 4.4 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 20 MiB/s wr, 1.74k op/s 2024-04-02T14:49:50.602 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:49:50 smithi092 ceph-mon[36497]: pgmap v467: 97 pgs: 97 active+clean; 4.4 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 20 MiB/s wr, 1.65k op/s 2024-04-02T14:49:50.678 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:49:50 smithi177 ceph-mon[36431]: pgmap v467: 97 pgs: 97 active+clean; 4.4 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 20 MiB/s wr, 1.65k op/s 2024-04-02T14:49:50.680 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:49:50 smithi090 ceph-mon[31674]: pgmap v467: 97 pgs: 97 active+clean; 4.4 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 20 MiB/s wr, 1.65k op/s 2024-04-02T14:49:51.823 INFO:teuthology.orchestra.run.smithi090.stderr:progress: 860.0 s, 1674.2 tps, lat 61.085 ms stddev 74.060 2024-04-02T14:49:52.589 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.595+0000 7f33df400640 1 -- 172.21.15.90:0/4014710061 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f33d8108340 msgr2=0x7f33d8108720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:49:52.589 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.595+0000 7f33df400640 1 --2- 172.21.15.90:0/4014710061 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f33d8108340 0x7f33d8108720 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f33cc009900 tx=0x7f33cc02bc60 comp rx=0 tx=0).stop 2024-04-02T14:49:52.589 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.595+0000 7f33df400640 1 -- 172.21.15.90:0/4014710061 shutdown_connections 2024-04-02T14:49:52.589 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.595+0000 7f33df400640 1 --2- 172.21.15.90:0/4014710061 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f33d810d410 0x7f33d8114fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:52.589 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.595+0000 7f33df400640 1 --2- 172.21.15.90:0/4014710061 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f33d8108cf0 0x7f33d810cd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:52.589 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.595+0000 7f33df400640 1 --2- 172.21.15.90:0/4014710061 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f33d8108340 0x7f33d8108720 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:52.589 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.595+0000 7f33df400640 1 -- 172.21.15.90:0/4014710061 >> 172.21.15.90:0/4014710061 conn(0x7f33d8076070 msgr2=0x7f33d8078490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:49:52.590 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.595+0000 7f33df400640 1 -- 172.21.15.90:0/4014710061 shutdown_connections 2024-04-02T14:49:52.590 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.595+0000 7f33df400640 1 -- 172.21.15.90:0/4014710061 wait complete. 2024-04-02T14:49:52.591 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.597+0000 7f33df400640 1 Processor -- start 2024-04-02T14:49:52.591 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.597+0000 7f33df400640 1 -- start start 2024-04-02T14:49:52.592 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.598+0000 7f33df400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f33d8108340 0x7f33d81a4620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:52.592 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.598+0000 7f33df400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f33d810d410 0x7f33d81a4b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:52.592 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.598+0000 7f33df400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f33d819e7e0 0x7f33d819ec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:52.592 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.598+0000 7f33df400640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f33d8118540 con 0x7f33d8108340 2024-04-02T14:49:52.592 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.598+0000 7f33df400640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f33d81183c0 con 0x7f33d819e7e0 2024-04-02T14:49:52.592 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.598+0000 7f33d7e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f33d810d410 0x7f33d81a4b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:52.593 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.598+0000 7f33dd400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f33d819e7e0 0x7f33d819ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:52.593 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.598+0000 7f33dca00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f33d8108340 0x7f33d81a4620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:52.593 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.598+0000 7f33d7e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f33d810d410 0x7f33d81a4b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:37324/0 (socket says 172.21.15.90:37324) 2024-04-02T14:49:52.593 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.598+0000 7f33d7e00640 1 -- 172.21.15.90:0/1472176450 learned_addr learned my addr 172.21.15.90:0/1472176450 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:49:52.593 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.598+0000 7f33df400640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f33d81186c0 con 0x7f33d810d410 2024-04-02T14:49:52.594 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.599+0000 7f33d7e00640 1 -- 172.21.15.90:0/1472176450 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f33d819e7e0 msgr2=0x7f33d819ec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:49:52.594 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.599+0000 7f33d7e00640 1 --2- 172.21.15.90:0/1472176450 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f33d819e7e0 0x7f33d819ec90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:52.594 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.599+0000 7f33d7e00640 1 -- 172.21.15.90:0/1472176450 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f33d8108340 msgr2=0x7f33d81a4620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:49:52.594 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.599+0000 7f33d7e00640 1 --2- 172.21.15.90:0/1472176450 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f33d8108340 0x7f33d81a4620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:52.594 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.599+0000 7f33d7e00640 1 -- 172.21.15.90:0/1472176450 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33d819f370 con 0x7f33d810d410 2024-04-02T14:49:52.594 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.599+0000 7f33dca00640 1 --2- 172.21.15.90:0/1472176450 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f33d8108340 0x7f33d81a4620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:49:52.594 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.600+0000 7f33d7e00640 1 --2- 172.21.15.90:0/1472176450 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f33d810d410 0x7f33d81a4b60 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f33c800e9f0 tx=0x7f33c800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:49:52.594 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.600+0000 7f33d5600640 1 -- 172.21.15.90:0/1472176450 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33c800ccc0 con 0x7f33d810d410 2024-04-02T14:49:52.594 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.600+0000 7f33df400640 1 -- 172.21.15.90:0/1472176450 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f33d819f660 con 0x7f33d810d410 2024-04-02T14:49:52.594 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.600+0000 7f33d5600640 1 -- 172.21.15.90:0/1472176450 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f33c8005500 con 0x7f33d810d410 2024-04-02T14:49:52.594 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.600+0000 7f33d5600640 1 -- 172.21.15.90:0/1472176450 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33c8010690 con 0x7f33d810d410 2024-04-02T14:49:52.595 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.600+0000 7f33df400640 1 -- 172.21.15.90:0/1472176450 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f33d81ab7f0 con 0x7f33d810d410 2024-04-02T14:49:52.599 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.605+0000 7f33d5600640 1 -- 172.21.15.90:0/1472176450 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f33c8010830 con 0x7f33d810d410 2024-04-02T14:49:52.599 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.606+0000 7f33e2072740 1 Processor -- start 2024-04-02T14:49:52.600 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.606+0000 7f33e2072740 1 -- start start 2024-04-02T14:49:52.600 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.607+0000 7f33e2072740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5578ee118eb0 0x5578ee119290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:52.601 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.607+0000 7f33e2072740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5578ee1197d0 0x5578ee121e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:52.601 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.607+0000 7f33e2072740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5578ee1223b0 0x5578ee1247a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:52.601 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.607+0000 7f33e2072740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5578ee07dd70 con 0x5578ee1223b0 2024-04-02T14:49:52.601 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.607+0000 7f33e2072740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x5578ee0262a0 con 0x5578ee118eb0 2024-04-02T14:49:52.601 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.607+0000 7f33e2072740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5578ee079d20 con 0x5578ee1197d0 2024-04-02T14:49:52.602 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:49:52 smithi092 ceph-mon[36497]: pgmap v468: 97 pgs: 97 active+clean; 4.4 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 161 KiB/s rd, 23 MiB/s wr, 1.91k op/s 2024-04-02T14:49:52.603 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.608+0000 7f33dca00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5578ee1197d0 0x5578ee121e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:52.603 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.608+0000 7f33dca00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5578ee1197d0 0x5578ee121e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:37336/0 (socket says 172.21.15.90:37336) 2024-04-02T14:49:52.603 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.608+0000 7f33dca00640 1 -- 172.21.15.90:0/3390017480 learned_addr learned my addr 172.21.15.90:0/3390017480 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:49:52.603 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.608+0000 7f33dd400640 1 --2- 172.21.15.90:0/3390017480 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5578ee118eb0 0x5578ee119290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:52.603 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.608+0000 7f33d5600640 1 --2- 172.21.15.90:0/1472176450 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f33c00a3e40 0x7f33c00a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:52.603 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.608+0000 7f33dca00640 1 -- 172.21.15.90:0/3390017480 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5578ee118eb0 msgr2=0x5578ee119290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:49:52.603 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.608+0000 7f33dca00640 1 --2- 172.21.15.90:0/3390017480 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5578ee118eb0 0x5578ee119290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:52.604 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.608+0000 7f33dca00640 1 -- 172.21.15.90:0/3390017480 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5578ee1223b0 msgr2=0x5578ee1247a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:49:52.604 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.608+0000 7f33dca00640 1 --2- 172.21.15.90:0/3390017480 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5578ee1223b0 0x5578ee1247a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:52.604 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.608+0000 7f33dca00640 1 -- 172.21.15.90:0/3390017480 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5578ee03a840 con 0x5578ee1197d0 2024-04-02T14:49:52.604 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.609+0000 7f33dca00640 1 --2- 172.21.15.90:0/3390017480 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5578ee1197d0 0x5578ee121e70 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f33cc005e00 tx=0x7f33cc02c6b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9862e8fd29d379a2 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:49:52.604 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.609+0000 7f33dd400640 1 --2- 172.21.15.90:0/1472176450 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f33c00a3e40 0x7f33c00a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:52.604 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.609+0000 7f33d5600640 1 -- 172.21.15.90:0/1472176450 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 8550+0+0 (secure 0 0 0) 0x7f33c80d0b30 con 0x7f33d810d410 2024-04-02T14:49:52.604 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.609+0000 7f33c5c00640 1 -- 172.21.15.90:0/3390017480 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33cc045070 con 0x5578ee1197d0 2024-04-02T14:49:52.604 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.609+0000 7f33c5c00640 1 -- 172.21.15.90:0/3390017480 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f33cc004040 con 0x5578ee1197d0 2024-04-02T14:49:52.604 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.609+0000 7f33c5c00640 1 -- 172.21.15.90:0/3390017480 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33cc003c10 con 0x5578ee1197d0 2024-04-02T14:49:52.605 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.610+0000 7f33e2072740 1 -- 172.21.15.90:0/3390017480 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5578ee1197d0 msgr2=0x5578ee121e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:49:52.605 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.610+0000 7f33dd400640 1 --2- 172.21.15.90:0/1472176450 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f33c00a3e40 0x7f33c00a6300 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f33d000a4c0 tx=0x7f33d0004b50 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:49:52.605 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.610+0000 7f33e2072740 1 --2- 172.21.15.90:0/3390017480 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5578ee1197d0 0x5578ee121e70 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f33cc005e00 tx=0x7f33cc02c6b0 comp rx=0 tx=0).stop 2024-04-02T14:49:52.605 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.611+0000 7f33e2072740 1 -- 172.21.15.90:0/3390017480 shutdown_connections 2024-04-02T14:49:52.605 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.611+0000 7f33e2072740 1 --2- 172.21.15.90:0/3390017480 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5578ee1223b0 0x5578ee1247a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:52.605 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.611+0000 7f33e2072740 1 --2- 172.21.15.90:0/3390017480 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5578ee1197d0 0x5578ee121e70 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:52.605 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.611+0000 7f33e2072740 1 --2- 172.21.15.90:0/3390017480 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5578ee118eb0 0x5578ee119290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:52.605 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.611+0000 7f33e2072740 1 -- 172.21.15.90:0/3390017480 >> 172.21.15.90:0/3390017480 conn(0x5578ee1135e0 msgr2=0x5578ee1139c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:49:52.606 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.611+0000 7f33e2072740 1 -- 172.21.15.90:0/3390017480 shutdown_connections 2024-04-02T14:49:52.606 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.611+0000 7f33e2072740 1 -- 172.21.15.90:0/3390017480 wait complete. 2024-04-02T14:49:52.606 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.611+0000 7f33e2072740 1 -- start start 2024-04-02T14:49:52.606 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.611+0000 7f33e2072740 1 Processor -- start 2024-04-02T14:49:52.606 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.612+0000 7f33e2072740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5578ee118eb0 0x5578ee11f430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:52.607 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.612+0000 7f33e2072740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5578ee1197d0 0x5578ee11f970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:52.607 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.612+0000 7f33e2072740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5578ee1223b0 0x5578ee11feb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:52.607 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.612+0000 7f33e2072740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5578ee125140 con 0x5578ee118eb0 2024-04-02T14:49:52.607 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.612+0000 7f33d7e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5578ee1197d0 0x5578ee11f970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:52.607 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.612+0000 7f33dd400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5578ee1223b0 0x5578ee11feb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:52.607 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.612+0000 7f33e2072740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x5578ee039660 con 0x5578ee1197d0 2024-04-02T14:49:52.607 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.613+0000 7f33e2072740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5578ee1252c0 con 0x5578ee1223b0 2024-04-02T14:49:52.607 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.613+0000 7f33e2072740 10 client.0 _ll_register_callbacks cb 0x5578ee039050 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:49:52.607 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.612+0000 7f33dd400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5578ee1223b0 0x5578ee11feb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:37348/0 (socket says 172.21.15.90:37348) 2024-04-02T14:49:52.607 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.613+0000 7f33dd400640 1 -- 172.21.15.90:0/3578011398 learned_addr learned my addr 172.21.15.90:0/3578011398 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:49:52.608 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.612+0000 7f33d7e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5578ee1197d0 0x5578ee11f970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:54704/0 (socket says 172.21.15.90:54704) 2024-04-02T14:49:52.608 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.613+0000 7f33dd400640 1 -- 172.21.15.90:0/3578011398 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5578ee1197d0 msgr2=0x5578ee11f970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:49:52.608 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.613+0000 7f33dd400640 1 --2- 172.21.15.90:0/3578011398 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5578ee1197d0 0x5578ee11f970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:52.608 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.613+0000 7f33dd400640 1 -- 172.21.15.90:0/3578011398 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5578ee118eb0 msgr2=0x5578ee11f430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:49:52.608 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.612+0000 7f33dca00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5578ee118eb0 0x5578ee11f430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:52.608 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.614+0000 7f33dd400640 1 --2- 172.21.15.90:0/3578011398 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5578ee118eb0 0x5578ee11f430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:52.608 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.614+0000 7f33dd400640 1 -- 172.21.15.90:0/3578011398 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5578ee048e80 con 0x5578ee1223b0 2024-04-02T14:49:52.608 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.614+0000 7f33dca00640 1 --2- 172.21.15.90:0/3578011398 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5578ee118eb0 0x5578ee11f430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:49:52.608 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.615+0000 7f33dd400640 1 --2- 172.21.15.90:0/3578011398 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5578ee1223b0 0x5578ee11feb0 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f33d0016e20 tx=0x7f33d000d040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5f93ce47c349e173 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:49:52.609 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.615+0000 7f33ba000640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:49:52.609 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.615+0000 7f33ba000640 1 -- 172.21.15.90:0/3578011398 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33d0027060 con 0x5578ee1223b0 2024-04-02T14:49:52.609 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.615+0000 7f33ba000640 1 -- 172.21.15.90:0/3578011398 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 8550+0+0 (secure 0 0 0) 0x7f33d0022610 con 0x5578ee1223b0 2024-04-02T14:49:52.609 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.616+0000 7f33e2072740 1 -- 172.21.15.90:0/3578011398 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f33cc004040 con 0x5578ee1223b0 2024-04-02T14:49:52.610 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.616+0000 7f33ba000640 1 -- 172.21.15.90:0/3578011398 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f33d0022810 con 0x5578ee1223b0 2024-04-02T14:49:52.610 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.616+0000 7f33e2072740 10 client.29911 fetch_fsmap learned FSMap version 27 2024-04-02T14:49:52.610 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.616+0000 7f33e2072740 1 -- 172.21.15.90:0/3578011398 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5578edd92210 con 0x5578ee1223b0 2024-04-02T14:49:52.611 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.617+0000 7f33ba000640 1 -- 172.21.15.90:0/3578011398 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f33cc004040 con 0x5578ee1223b0 2024-04-02T14:49:52.611 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.617+0000 7f33ba000640 1 -- 172.21.15.90:0/3578011398 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f33d00219f0 con 0x5578ee1223b0 2024-04-02T14:49:52.611 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.617+0000 7f33e2072740 10 client.29911 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:49:52.611 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.617+0000 7f33e2072740 10 client.29911 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-02T14:49:52.611 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.617+0000 7f33e2072740 20 client.29911 populate_metadata read hostname 'smithi090' 2024-04-02T14:49:52.611 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.617+0000 7f33e2072740 1 --2- 172.21.15.90:0/3578011398 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5578ee132730 0x5578ee134b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:52.611 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.617+0000 7f33e2072740 4 client.29911 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:49:52.611 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.617+0000 7f33e2072740 1 -- 172.21.15.90:0/3578011398 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5578ee02dcd0 con 0x5578ee132730 2024-04-02T14:49:52.611 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.617+0000 7f33dca00640 1 --2- 172.21.15.90:0/3578011398 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5578ee132730 0x5578ee134b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:52.612 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.618+0000 7f33dca00640 1 --2- 172.21.15.90:0/3578011398 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5578ee132730 0x5578ee134b20 crc :-1 s=READY pgs=380 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=34007cab47f2505 server_cookie=89e4dfae7aa14cee in_seq=0 out_seq=0 2024-04-02T14:49:52.612 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.618+0000 7f33ba000640 10 client.29911 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:49:52.613 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.619+0000 7f33ba000640 1 -- 172.21.15.90:0/3578011398 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5578ee02dcd0 con 0x5578ee132730 2024-04-02T14:49:52.613 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.619+0000 7f33ba000640 10 client.29911 handle_command_reply: tid=0 2024-04-02T14:49:52.613 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.619+0000 7f33e2072740 1 client.29911 shutdown 2024-04-02T14:49:52.614 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.620+0000 7f33e2072740 1 -- 172.21.15.90:0/3578011398 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5578ee1223b0 msgr2=0x5578ee11feb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:49:52.614 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.620+0000 7f33e2072740 1 --2- 172.21.15.90:0/3578011398 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5578ee1223b0 0x5578ee11feb0 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f33d0016e20 tx=0x7f33d000d040 comp rx=0 tx=0).stop 2024-04-02T14:49:52.614 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.620+0000 7f33e2072740 1 -- 172.21.15.90:0/3578011398 shutdown_connections 2024-04-02T14:49:52.614 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.620+0000 7f33e2072740 1 --2- 172.21.15.90:0/3578011398 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5578ee132730 0x5578ee134b20 crc :-1 s=READY pgs=380 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:52.614 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.620+0000 7f33ba000640 0 client.29911 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:49:52.615 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.620+0000 7f33e2072740 1 --2- 172.21.15.90:0/3578011398 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5578ee1223b0 0x5578ee11feb0 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:52.615 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.620+0000 7f33e2072740 1 --2- 172.21.15.90:0/3578011398 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5578ee1197d0 0x5578ee11f970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:52.615 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.620+0000 7f33e2072740 1 --2- 172.21.15.90:0/3578011398 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5578ee118eb0 0x5578ee11f430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:52.615 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.620+0000 7f33e2072740 1 -- 172.21.15.90:0/3578011398 >> 172.21.15.90:0/3578011398 conn(0x5578ee1135e0 msgr2=0x5578ee115330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:49:52.615 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.621+0000 7f33e2072740 1 -- 172.21.15.90:0/3578011398 shutdown_connections 2024-04-02T14:49:52.615 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.621+0000 7f33e2072740 1 -- 172.21.15.90:0/3578011398 wait complete. 2024-04-02T14:49:52.615 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.621+0000 7f33e2072740 20 client.29911 trim_cache size 0 max 16384 2024-04-02T14:49:52.643 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.648+0000 7f33e2072740 1 Processor -- start 2024-04-02T14:49:52.643 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.648+0000 7f33e2072740 1 -- start start 2024-04-02T14:49:52.643 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.648+0000 7f33e2072740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5578ee118eb0 0x5578ee121b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:52.643 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.648+0000 7f33e2072740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5578ee1197d0 0x5578ee12e000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:52.643 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.648+0000 7f33e2072740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5578ee1223b0 0x5578ee11fb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:52.643 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.648+0000 7f33e2072740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f33d81b3de0 con 0x5578ee118eb0 2024-04-02T14:49:52.643 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.648+0000 7f33e2072740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f33d81b5920 con 0x5578ee1197d0 2024-04-02T14:49:52.643 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.648+0000 7f33e2072740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f33d81b44b0 con 0x5578ee1223b0 2024-04-02T14:49:52.643 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.648+0000 7f33dd400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5578ee1223b0 0x5578ee11fb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:52.643 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.648+0000 7f33dd400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5578ee1223b0 0x5578ee11fb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:37354/0 (socket says 172.21.15.90:37354) 2024-04-02T14:49:52.644 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.648+0000 7f33dd400640 1 -- 172.21.15.90:0/4188033211 learned_addr learned my addr 172.21.15.90:0/4188033211 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:49:52.644 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.648+0000 7f33dca00640 1 --2- 172.21.15.90:0/4188033211 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5578ee118eb0 0x5578ee121b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:52.644 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.648+0000 7f33d7e00640 1 --2- 172.21.15.90:0/4188033211 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5578ee1197d0 0x5578ee12e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:52.644 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.649+0000 7f33dd400640 1 -- 172.21.15.90:0/4188033211 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5578ee1197d0 msgr2=0x5578ee12e000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:49:52.644 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.649+0000 7f33dd400640 1 --2- 172.21.15.90:0/4188033211 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5578ee1197d0 0x5578ee12e000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:52.644 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.649+0000 7f33dd400640 1 -- 172.21.15.90:0/4188033211 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5578ee118eb0 msgr2=0x5578ee121b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:49:52.644 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.649+0000 7f33dd400640 1 --2- 172.21.15.90:0/4188033211 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5578ee118eb0 0x5578ee121b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:52.644 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.649+0000 7f33dd400640 1 -- 172.21.15.90:0/4188033211 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5578ee124b10 con 0x5578ee1223b0 2024-04-02T14:49:52.644 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.649+0000 7f33dca00640 1 --2- 172.21.15.90:0/4188033211 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5578ee118eb0 0x5578ee121b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:49:52.644 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.649+0000 7f33dd400640 1 --2- 172.21.15.90:0/4188033211 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5578ee1223b0 0x5578ee11fb00 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f33d0021e10 tx=0x7f33d0004a00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bcda13aa2f1d82b server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:49:52.644 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.649+0000 7f33c5200640 1 -- 172.21.15.90:0/4188033211 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33d003a070 con 0x5578ee1223b0 2024-04-02T14:49:52.644 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.649+0000 7f33c5200640 1 -- 172.21.15.90:0/4188033211 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f33d001d050 con 0x5578ee1223b0 2024-04-02T14:49:52.644 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.649+0000 7f33c5200640 1 -- 172.21.15.90:0/4188033211 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33d0031080 con 0x5578ee1223b0 2024-04-02T14:49:52.645 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.650+0000 7f33e2072740 1 -- 172.21.15.90:0/4188033211 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5578ee1223b0 msgr2=0x5578ee11fb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:49:52.645 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.650+0000 7f33e2072740 1 --2- 172.21.15.90:0/4188033211 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5578ee1223b0 0x5578ee11fb00 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f33d0021e10 tx=0x7f33d0004a00 comp rx=0 tx=0).stop 2024-04-02T14:49:52.645 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.650+0000 7f33e2072740 1 -- 172.21.15.90:0/4188033211 shutdown_connections 2024-04-02T14:49:52.645 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.650+0000 7f33e2072740 1 --2- 172.21.15.90:0/4188033211 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5578ee1223b0 0x5578ee11fb00 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:52.645 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.650+0000 7f33e2072740 1 --2- 172.21.15.90:0/4188033211 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5578ee1197d0 0x5578ee12e000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:52.645 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.650+0000 7f33e2072740 1 --2- 172.21.15.90:0/4188033211 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5578ee118eb0 0x5578ee121b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:52.646 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.650+0000 7f33e2072740 1 -- 172.21.15.90:0/4188033211 >> 172.21.15.90:0/4188033211 conn(0x5578ee1135e0 msgr2=0x5578ee11f3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:49:52.646 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.652+0000 7f33e2072740 1 -- 172.21.15.90:0/4188033211 shutdown_connections 2024-04-02T14:49:52.646 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.652+0000 7f33e2072740 1 -- 172.21.15.90:0/4188033211 wait complete. 2024-04-02T14:49:52.646 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.652+0000 7f33e2072740 1 -- start start 2024-04-02T14:49:52.646 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.653+0000 7f33e2072740 1 Processor -- start 2024-04-02T14:49:52.647 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.653+0000 7f33e2072740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5578ee118eb0 0x5578ee12a510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:52.647 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.653+0000 7f33e2072740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5578ee1197d0 0x5578ee12aa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:52.647 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.653+0000 7f33e2072740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5578ee1223b0 0x5578ee1274e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:52.647 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.653+0000 7f33e2072740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f33d81b5250 con 0x5578ee118eb0 2024-04-02T14:49:52.647 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.653+0000 7f33e2072740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f33d81b4b80 con 0x5578ee1197d0 2024-04-02T14:49:52.648 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.653+0000 7f33e2072740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f33d81b1de0 con 0x5578ee1223b0 2024-04-02T14:49:52.648 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.653+0000 7f33e2072740 10 client.0 _ll_register_callbacks cb 0x7f33d0016a70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:49:52.648 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.654+0000 7f33d7e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5578ee1197d0 0x5578ee12aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:52.649 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.654+0000 7f33d7e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5578ee1197d0 0x5578ee12aa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:54716/0 (socket says 172.21.15.90:54716) 2024-04-02T14:49:52.649 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.654+0000 7f33d7e00640 1 -- 172.21.15.90:0/1013434728 learned_addr learned my addr 172.21.15.90:0/1013434728 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:49:52.649 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.654+0000 7f33dca00640 1 --2- 172.21.15.90:0/1013434728 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5578ee118eb0 0x5578ee12a510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:52.649 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.654+0000 7f33d7e00640 1 -- 172.21.15.90:0/1013434728 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5578ee1223b0 msgr2=0x5578ee1274e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:49:52.649 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.654+0000 7f33d7e00640 1 --2- 172.21.15.90:0/1013434728 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5578ee1223b0 0x5578ee1274e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:52.649 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.654+0000 7f33d7e00640 1 -- 172.21.15.90:0/1013434728 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5578ee118eb0 msgr2=0x5578ee12a510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:49:52.649 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.654+0000 7f33d7e00640 1 --2- 172.21.15.90:0/1013434728 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5578ee118eb0 0x5578ee12a510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:52.649 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.654+0000 7f33d7e00640 1 -- 172.21.15.90:0/1013434728 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5578ede5e620 con 0x5578ee1197d0 2024-04-02T14:49:52.649 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.654+0000 7f33dca00640 1 --2- 172.21.15.90:0/1013434728 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5578ee118eb0 0x5578ee12a510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:49:52.649 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.655+0000 7f33d7e00640 1 --2- 172.21.15.90:0/1013434728 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5578ee1197d0 0x5578ee12aa50 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7f33c800cf10 tx=0x7f33c809c6b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d7d7dfbcdbe7d186 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:49:52.650 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.655+0000 7f33ba000640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:49:52.650 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.655+0000 7f33ba000640 1 -- 172.21.15.90:0/1013434728 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33c80b5020 con 0x5578ee1197d0 2024-04-02T14:49:52.650 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.655+0000 7f33ba000640 1 -- 172.21.15.90:0/1013434728 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 8550+0+0 (secure 0 0 0) 0x7f33c80046c0 con 0x5578ee1197d0 2024-04-02T14:49:52.651 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.655+0000 7f33e2072740 1 -- 172.21.15.90:0/1013434728 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f33d001d050 con 0x5578ee1197d0 2024-04-02T14:49:52.651 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.656+0000 7f33ba000640 1 -- 172.21.15.90:0/1013434728 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f33c80048c0 con 0x5578ee1197d0 2024-04-02T14:49:52.651 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.656+0000 7f33e2072740 10 client.30128 fetch_fsmap learned FSMap version 27 2024-04-02T14:49:52.651 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.656+0000 7f33e2072740 1 -- 172.21.15.90:0/1013434728 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5578ee1994c0 con 0x5578ee1197d0 2024-04-02T14:49:52.651 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.656+0000 7f33ba000640 1 -- 172.21.15.90:0/1013434728 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f33d001d050 con 0x5578ee1197d0 2024-04-02T14:49:52.651 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.657+0000 7f33ba000640 1 -- 172.21.15.90:0/1013434728 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f33c80a5320 con 0x5578ee1197d0 2024-04-02T14:49:52.651 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.657+0000 7f33e2072740 10 client.30128 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:49:52.651 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.657+0000 7f33e2072740 10 client.30128 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-02T14:49:52.651 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.657+0000 7f33e2072740 20 client.30128 populate_metadata read hostname 'smithi090' 2024-04-02T14:49:52.652 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.657+0000 7f33e2072740 1 --2- 172.21.15.90:0/1013434728 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5578ee199c40 0x5578ee19c060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:49:52.652 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.657+0000 7f33e2072740 4 client.30128 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-02T14:49:52.652 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.657+0000 7f33e2072740 1 -- 172.21.15.90:0/1013434728 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5578ee08b2b0 con 0x5578ee199c40 2024-04-02T14:49:52.652 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.658+0000 7f33dca00640 1 --2- 172.21.15.90:0/1013434728 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5578ee199c40 0x5578ee19c060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:49:52.652 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.658+0000 7f33ba000640 10 client.30128 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:49:52.653 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.658+0000 7f33dca00640 1 --2- 172.21.15.90:0/1013434728 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5578ee199c40 0x5578ee19c060 crc :-1 s=READY pgs=381 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3b14aed8297413fc server_cookie=b9bcbd0fc550a256 in_seq=0 out_seq=0 2024-04-02T14:49:52.675 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.680+0000 7f33ba000640 1 -- 172.21.15.90:0/1013434728 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5578ee08b2b0 con 0x5578ee199c40 2024-04-02T14:49:52.675 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.680+0000 7f33ba000640 10 client.30128 handle_command_reply: tid=0 2024-04-02T14:49:52.675 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.680+0000 7f33e2072740 1 client.30128 shutdown 2024-04-02T14:49:52.675 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.680+0000 7f33e2072740 1 -- 172.21.15.90:0/1013434728 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5578ee1197d0 msgr2=0x5578ee12aa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:49:52.675 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.680+0000 7f33e2072740 1 --2- 172.21.15.90:0/1013434728 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5578ee1197d0 0x5578ee12aa50 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7f33c800cf10 tx=0x7f33c809c6b0 comp rx=0 tx=0).stop 2024-04-02T14:49:52.675 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.681+0000 7f33e2072740 1 -- 172.21.15.90:0/1013434728 shutdown_connections 2024-04-02T14:49:52.676 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.681+0000 7f33e2072740 1 --2- 172.21.15.90:0/1013434728 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5578ee199c40 0x5578ee19c060 crc :-1 s=READY pgs=381 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:52.676 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.681+0000 7f33e2072740 1 --2- 172.21.15.90:0/1013434728 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5578ee1223b0 0x5578ee1274e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:52.676 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.681+0000 7f33e2072740 1 --2- 172.21.15.90:0/1013434728 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5578ee1197d0 0x5578ee12aa50 unknown :-1 s=CLOSED pgs=1011 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:52.676 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.681+0000 7f33e2072740 1 --2- 172.21.15.90:0/1013434728 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5578ee118eb0 0x5578ee12a510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:52.676 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.681+0000 7f33ba000640 0 client.30128 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:49:52.676 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.681+0000 7f33e2072740 1 -- 172.21.15.90:0/1013434728 >> 172.21.15.90:0/1013434728 conn(0x5578ee1135e0 msgr2=0x5578ee112640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:49:52.676 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.681+0000 7f33e2072740 1 -- 172.21.15.90:0/1013434728 shutdown_connections 2024-04-02T14:49:52.676 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.681+0000 7f33e2072740 1 -- 172.21.15.90:0/1013434728 wait complete. 2024-04-02T14:49:52.676 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.681+0000 7f33e2072740 20 client.30128 trim_cache size 0 max 16384 2024-04-02T14:49:52.677 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:{ 2024-04-02T14:49:52.677 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "message": "", 2024-04-02T14:49:52.677 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "return_code": 0 2024-04-02T14:49:52.677 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:} 2024-04-02T14:49:52.678 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.682+0000 7f33df400640 1 -- 172.21.15.90:0/1472176450 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f33c00a3e40 msgr2=0x7f33c00a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:49:52.678 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.682+0000 7f33df400640 1 --2- 172.21.15.90:0/1472176450 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f33c00a3e40 0x7f33c00a6300 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f33d000a4c0 tx=0x7f33d0004b50 comp rx=0 tx=0).stop 2024-04-02T14:49:52.678 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.682+0000 7f33df400640 1 -- 172.21.15.90:0/1472176450 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f33d810d410 msgr2=0x7f33d81a4b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:49:52.678 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.682+0000 7f33df400640 1 --2- 172.21.15.90:0/1472176450 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f33d810d410 0x7f33d81a4b60 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f33c800e9f0 tx=0x7f33c800eeb0 comp rx=0 tx=0).stop 2024-04-02T14:49:52.678 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.682+0000 7f33df400640 1 -- 172.21.15.90:0/1472176450 shutdown_connections 2024-04-02T14:49:52.678 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.682+0000 7f33df400640 1 --2- 172.21.15.90:0/1472176450 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f33c00a3e40 0x7f33c00a6300 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:52.678 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.682+0000 7f33df400640 1 --2- 172.21.15.90:0/1472176450 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f33d819e7e0 0x7f33d819ec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:52.678 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.682+0000 7f33df400640 1 --2- 172.21.15.90:0/1472176450 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f33d810d410 0x7f33d81a4b60 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:52.678 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.682+0000 7f33df400640 1 --2- 172.21.15.90:0/1472176450 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f33d8108340 0x7f33d81a4620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:49:52.678 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.682+0000 7f33df400640 1 -- 172.21.15.90:0/1472176450 >> 172.21.15.90:0/1472176450 conn(0x7f33d8076070 msgr2=0x7f33d8076c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:49:52.678 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.682+0000 7f33df400640 1 -- 172.21.15.90:0/1472176450 shutdown_connections 2024-04-02T14:49:52.678 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:49:52.682+0000 7f33df400640 1 -- 172.21.15.90:0/1472176450 wait complete. 2024-04-02T14:49:52.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:49:52 smithi177 ceph-mon[36431]: pgmap v468: 97 pgs: 97 active+clean; 4.4 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 161 KiB/s rd, 23 MiB/s wr, 1.91k op/s 2024-04-02T14:49:52.680 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:49:52 smithi090 ceph-mon[31674]: pgmap v468: 97 pgs: 97 active+clean; 4.4 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 161 KiB/s rd, 23 MiB/s wr, 1.91k op/s 2024-04-02T14:49:54.602 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:49:54 smithi092 ceph-mon[36497]: pgmap v469: 97 pgs: 97 active+clean; 4.4 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 19 MiB/s wr, 1.56k op/s 2024-04-02T14:49:54.678 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:49:54 smithi177 ceph-mon[36431]: pgmap v469: 97 pgs: 97 active+clean; 4.4 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 19 MiB/s wr, 1.56k op/s 2024-04-02T14:49:54.680 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:49:54 smithi090 ceph-mon[31674]: pgmap v469: 97 pgs: 97 active+clean; 4.4 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 19 MiB/s wr, 1.56k op/s 2024-04-02T14:49:56.602 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:49:56 smithi092 ceph-mon[36497]: pgmap v470: 97 pgs: 97 active+clean; 4.4 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 21 MiB/s wr, 1.76k op/s 2024-04-02T14:49:56.678 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:49:56 smithi177 ceph-mon[36431]: pgmap v470: 97 pgs: 97 active+clean; 4.4 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 21 MiB/s wr, 1.76k op/s 2024-04-02T14:49:56.683 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:49:56 smithi090 ceph-mon[31674]: pgmap v470: 97 pgs: 97 active+clean; 4.4 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 21 MiB/s wr, 1.76k op/s 2024-04-02T14:49:56.823 INFO:teuthology.orchestra.run.smithi090.stderr:progress: 865.0 s, 2034.8 tps, lat 48.878 ms stddev 58.349 2024-04-02T14:49:58.602 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:49:58 smithi092 ceph-mon[36497]: pgmap v471: 97 pgs: 97 active+clean; 4.5 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 10 MiB/s wr, 951 op/s 2024-04-02T14:49:58.678 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:49:58 smithi177 ceph-mon[36431]: pgmap v471: 97 pgs: 97 active+clean; 4.5 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 10 MiB/s wr, 951 op/s 2024-04-02T14:49:58.680 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:49:58 smithi090 ceph-mon[31674]: pgmap v471: 97 pgs: 97 active+clean; 4.5 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 10 MiB/s wr, 951 op/s 2024-04-02T14:50:00.602 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:50:00 smithi092 ceph-mon[36497]: pgmap v472: 97 pgs: 97 active+clean; 4.5 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 5.7 MiB/s wr, 534 op/s 2024-04-02T14:50:00.602 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:50:00 smithi092 ceph-mon[36497]: overall HEALTH_OK 2024-04-02T14:50:00.678 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:50:00 smithi177 ceph-mon[36431]: pgmap v472: 97 pgs: 97 active+clean; 4.5 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 5.7 MiB/s wr, 534 op/s 2024-04-02T14:50:00.678 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:50:00 smithi177 ceph-mon[36431]: overall HEALTH_OK 2024-04-02T14:50:00.680 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:50:00 smithi090 ceph-mon[31674]: pgmap v472: 97 pgs: 97 active+clean; 4.5 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 5.7 MiB/s wr, 534 op/s 2024-04-02T14:50:00.680 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:50:00 smithi090 ceph-mon[31674]: overall HEALTH_OK 2024-04-02T14:50:01.823 INFO:teuthology.orchestra.run.smithi090.stderr:progress: 870.0 s, 1392.2 tps, lat 69.297 ms stddev 93.789 2024-04-02T14:50:02.602 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:50:02 smithi092 ceph-mon[36497]: pgmap v473: 97 pgs: 97 active+clean; 4.5 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 8.6 MiB/s wr, 813 op/s 2024-04-02T14:50:02.678 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:50:02 smithi177 ceph-mon[36431]: pgmap v473: 97 pgs: 97 active+clean; 4.5 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 8.6 MiB/s wr, 813 op/s 2024-04-02T14:50:02.680 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:50:02 smithi090 ceph-mon[31674]: pgmap v473: 97 pgs: 97 active+clean; 4.5 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 8.6 MiB/s wr, 813 op/s 2024-04-02T14:50:03.678 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:50:03 smithi177 ceph-mon[36431]: osdmap e108: 12 total, 12 up, 12 in 2024-04-02T14:50:03.678 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:50:03 smithi177 ceph-mon[36431]: pgmap v475: 97 pgs: 97 active+clean; 4.5 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 6.7 MiB/s wr, 658 op/s 2024-04-02T14:50:03.680 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:50:03 smithi090 ceph-mon[31674]: osdmap e108: 12 total, 12 up, 12 in 2024-04-02T14:50:03.680 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:50:03 smithi090 ceph-mon[31674]: pgmap v475: 97 pgs: 97 active+clean; 4.5 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 6.7 MiB/s wr, 658 op/s 2024-04-02T14:50:03.852 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:50:03 smithi092 ceph-mon[36497]: osdmap e108: 12 total, 12 up, 12 in 2024-04-02T14:50:03.852 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:50:03 smithi092 ceph-mon[36497]: pgmap v475: 97 pgs: 97 active+clean; 4.5 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 6.7 MiB/s wr, 658 op/s 2024-04-02T14:50:05.828 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.834+0000 7fed83c00640 1 -- 172.21.15.90:0/3166407779 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fed7c107ac0 msgr2=0x7fed7c107ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:05.828 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.834+0000 7fed83c00640 1 --2- 172.21.15.90:0/3166407779 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fed7c107ac0 0x7fed7c107ea0 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7fed70009900 tx=0x7fed7002bc30 comp rx=0 tx=0).stop 2024-04-02T14:50:05.828 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.835+0000 7fed83c00640 1 -- 172.21.15.90:0/3166407779 shutdown_connections 2024-04-02T14:50:05.828 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.835+0000 7fed83c00640 1 --2- 172.21.15.90:0/3166407779 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fed7c102470 0x7fed7c10e980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:05.828 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.835+0000 7fed83c00640 1 --2- 172.21.15.90:0/3166407779 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fed7c101ab0 0x7fed7c101f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:05.828 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.835+0000 7fed83c00640 1 --2- 172.21.15.90:0/3166407779 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fed7c107ac0 0x7fed7c107ea0 unknown :-1 s=CLOSED pgs=1012 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:05.828 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.835+0000 7fed83c00640 1 -- 172.21.15.90:0/3166407779 >> 172.21.15.90:0/3166407779 conn(0x7fed7c0fd5c0 msgr2=0x7fed7c0ff9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:50:05.828 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.835+0000 7fed83c00640 1 -- 172.21.15.90:0/3166407779 shutdown_connections 2024-04-02T14:50:05.828 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.835+0000 7fed83c00640 1 -- 172.21.15.90:0/3166407779 wait complete. 2024-04-02T14:50:05.829 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.836+0000 7fed83c00640 1 Processor -- start 2024-04-02T14:50:05.830 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.836+0000 7fed83c00640 1 -- start start 2024-04-02T14:50:05.830 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.836+0000 7fed83c00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fed7c101ab0 0x7fed7c1982c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:05.830 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.836+0000 7fed83c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fed7c102470 0x7fed7c198800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:05.830 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.836+0000 7fed83c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fed7c107ac0 0x7fed7c19cb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:05.830 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.836+0000 7fed83c00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fed7c111f20 con 0x7fed7c102470 2024-04-02T14:50:05.830 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.836+0000 7fed83c00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fed7c111da0 con 0x7fed7c101ab0 2024-04-02T14:50:05.830 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.836+0000 7fed83c00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fed7c1120a0 con 0x7fed7c107ac0 2024-04-02T14:50:05.830 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.837+0000 7fed7be00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fed7c102470 0x7fed7c198800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:05.830 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.837+0000 7fed7be00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fed7c102470 0x7fed7c198800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41148/0 (socket says 172.21.15.90:41148) 2024-04-02T14:50:05.830 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.837+0000 7fed7be00640 1 -- 172.21.15.90:0/3785836099 learned_addr learned my addr 172.21.15.90:0/3785836099 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:50:05.830 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.837+0000 7fed81200640 1 --2- 172.21.15.90:0/3785836099 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fed7c101ab0 0x7fed7c1982c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:05.830 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.837+0000 7fed81c00640 1 --2- 172.21.15.90:0/3785836099 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fed7c107ac0 0x7fed7c19cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:05.831 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.837+0000 7fed7be00640 1 -- 172.21.15.90:0/3785836099 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fed7c107ac0 msgr2=0x7fed7c19cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:05.831 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.837+0000 7fed7be00640 1 --2- 172.21.15.90:0/3785836099 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fed7c107ac0 0x7fed7c19cb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:05.831 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.837+0000 7fed7be00640 1 -- 172.21.15.90:0/3785836099 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fed7c101ab0 msgr2=0x7fed7c1982c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:05.831 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.837+0000 7fed7be00640 1 --2- 172.21.15.90:0/3785836099 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fed7c101ab0 0x7fed7c1982c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:05.832 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.837+0000 7fed7be00640 1 -- 172.21.15.90:0/3785836099 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed7c19d2b0 con 0x7fed7c102470 2024-04-02T14:50:05.832 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.837+0000 7fed81200640 1 --2- 172.21.15.90:0/3785836099 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fed7c101ab0 0x7fed7c1982c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:50:05.832 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.838+0000 7fed7be00640 1 --2- 172.21.15.90:0/3785836099 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fed7c102470 0x7fed7c198800 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fed6800e9f0 tx=0x7fed6800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:50:05.832 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.838+0000 7fed79600640 1 -- 172.21.15.90:0/3785836099 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed6800cd80 con 0x7fed7c102470 2024-04-02T14:50:05.832 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.838+0000 7fed79600640 1 -- 172.21.15.90:0/3785836099 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fed68010910 con 0x7fed7c102470 2024-04-02T14:50:05.832 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.838+0000 7fed83c00640 1 -- 172.21.15.90:0/3785836099 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed7c19d5a0 con 0x7fed7c102470 2024-04-02T14:50:05.832 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.838+0000 7fed79600640 1 -- 172.21.15.90:0/3785836099 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed68018990 con 0x7fed7c102470 2024-04-02T14:50:05.832 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.838+0000 7fed83c00640 1 -- 172.21.15.90:0/3785836099 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed7c1a4e30 con 0x7fed7c102470 2024-04-02T14:50:05.834 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.841+0000 7fed8696b740 1 Processor -- start 2024-04-02T14:50:05.834 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.841+0000 7fed79600640 1 -- 172.21.15.90:0/3785836099 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fed68010ac0 con 0x7fed7c102470 2024-04-02T14:50:05.834 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.842+0000 7fed8696b740 1 -- start start 2024-04-02T14:50:05.835 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.842+0000 7fed8696b740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bfaea3b550 0x55bfaea3b930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:05.835 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.842+0000 7fed8696b740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55bfaea3bf00 0x55bfaea28090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:05.835 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.842+0000 7fed8696b740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bfaea286b0 0x55bfaea28b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:05.835 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.842+0000 7fed8696b740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55bfae992d70 con 0x55bfaea3b550 2024-04-02T14:50:05.836 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.842+0000 7fed8696b740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55bfae93b2a0 con 0x55bfaea3bf00 2024-04-02T14:50:05.836 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.842+0000 7fed8696b740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55bfae98ed20 con 0x55bfaea286b0 2024-04-02T14:50:05.836 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.843+0000 7fed81200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55bfaea3bf00 0x55bfaea28090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:05.836 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.843+0000 7fed81200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55bfaea3bf00 0x55bfaea28090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:47502/0 (socket says 172.21.15.90:47502) 2024-04-02T14:50:05.836 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.843+0000 7fed81200640 1 -- 172.21.15.90:0/4176048570 learned_addr learned my addr 172.21.15.90:0/4176048570 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:50:05.836 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.843+0000 7fed7be00640 1 --2- 172.21.15.90:0/4176048570 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bfaea286b0 0x55bfaea28b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:05.836 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.843+0000 7fed81c00640 1 --2- 172.21.15.90:0/4176048570 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bfaea3b550 0x55bfaea3b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:05.837 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.843+0000 7fed7be00640 1 -- 172.21.15.90:0/4176048570 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55bfaea3bf00 msgr2=0x55bfaea28090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:50:05.837 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.843+0000 7fed7be00640 1 --2- 172.21.15.90:0/4176048570 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55bfaea3bf00 0x55bfaea28090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:05.837 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.843+0000 7fed7be00640 1 -- 172.21.15.90:0/4176048570 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bfaea3b550 msgr2=0x55bfaea3b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:50:05.837 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.843+0000 7fed7be00640 1 --2- 172.21.15.90:0/4176048570 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bfaea3b550 0x55bfaea3b930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:05.837 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.843+0000 7fed7be00640 1 -- 172.21.15.90:0/4176048570 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bfae94f840 con 0x55bfaea286b0 2024-04-02T14:50:05.837 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.843+0000 7fed81c00640 1 --2- 172.21.15.90:0/4176048570 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bfaea3b550 0x55bfaea3b930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:50:05.838 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.843+0000 7fed79600640 1 --2- 172.21.15.90:0/3785836099 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fed640a3e40 0x7fed640a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:05.838 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.844+0000 7fed7be00640 1 --2- 172.21.15.90:0/4176048570 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bfaea286b0 0x55bfaea28b10 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7fed680ce1d0 tx=0x7fed680d95f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9312d17fa4a72fc2 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:50:05.838 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.844+0000 7fed79600640 1 -- 172.21.15.90:0/3785836099 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 8594+0+0 (secure 0 0 0) 0x7fed680d0b30 con 0x7fed7c102470 2024-04-02T14:50:05.838 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.844+0000 7fed81c00640 1 --2- 172.21.15.90:0/3785836099 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fed640a3e40 0x7fed640a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:05.838 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.844+0000 7fed6e000640 1 -- 172.21.15.90:0/4176048570 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed68026030 con 0x55bfaea286b0 2024-04-02T14:50:05.838 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.844+0000 7fed6e000640 1 -- 172.21.15.90:0/4176048570 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fed680d6050 con 0x55bfaea286b0 2024-04-02T14:50:05.838 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.844+0000 7fed6e000640 1 -- 172.21.15.90:0/4176048570 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed6809c8a0 con 0x55bfaea286b0 2024-04-02T14:50:05.838 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.845+0000 7fed8696b740 1 -- 172.21.15.90:0/4176048570 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bfaea286b0 msgr2=0x55bfaea28b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:05.838 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.845+0000 7fed81c00640 1 --2- 172.21.15.90:0/3785836099 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fed640a3e40 0x7fed640a6300 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x55bfaea291d0 tx=0x7fed74009500 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:50:05.838 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.845+0000 7fed8696b740 1 --2- 172.21.15.90:0/4176048570 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bfaea286b0 0x55bfaea28b10 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7fed680ce1d0 tx=0x7fed680d95f0 comp rx=0 tx=0).stop 2024-04-02T14:50:05.841 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.845+0000 7fed8696b740 1 -- 172.21.15.90:0/4176048570 shutdown_connections 2024-04-02T14:50:05.841 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.845+0000 7fed8696b740 1 --2- 172.21.15.90:0/4176048570 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bfaea286b0 0x55bfaea28b10 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:05.841 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.845+0000 7fed8696b740 1 --2- 172.21.15.90:0/4176048570 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55bfaea3bf00 0x55bfaea28090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:05.841 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.845+0000 7fed8696b740 1 --2- 172.21.15.90:0/4176048570 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bfaea3b550 0x55bfaea3b930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:05.841 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.845+0000 7fed8696b740 1 -- 172.21.15.90:0/4176048570 >> 172.21.15.90:0/4176048570 conn(0x55bfaea2ab50 msgr2=0x55bfaea2af30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:50:05.841 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.845+0000 7fed8696b740 1 -- 172.21.15.90:0/4176048570 shutdown_connections 2024-04-02T14:50:05.841 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.845+0000 7fed8696b740 1 -- 172.21.15.90:0/4176048570 wait complete. 2024-04-02T14:50:05.841 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.846+0000 7fed8696b740 1 -- start start 2024-04-02T14:50:05.841 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.846+0000 7fed8696b740 1 Processor -- start 2024-04-02T14:50:05.841 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.846+0000 7fed8696b740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bfaea286b0 0x55bfaea3f510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:05.841 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.846+0000 7fed8696b740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bfaea3b550 0x55bfaea3fa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:05.841 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.846+0000 7fed8696b740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55bfaea3bf00 0x55bfaea3ff90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:05.841 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.846+0000 7fed8696b740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55bfaea42990 con 0x55bfaea3b550 2024-04-02T14:50:05.841 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.847+0000 7fed8696b740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55bfae94e660 con 0x55bfaea3bf00 2024-04-02T14:50:05.842 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.847+0000 7fed8696b740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55bfaea42b10 con 0x55bfaea286b0 2024-04-02T14:50:05.842 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.847+0000 7fed8696b740 10 client.0 _ll_register_callbacks cb 0x55bfae94e050 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:50:05.842 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.847+0000 7fed81200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bfaea286b0 0x55bfaea3f510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:05.842 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.847+0000 7fed81200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bfaea286b0 0x55bfaea3f510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:45824/0 (socket says 172.21.15.90:45824) 2024-04-02T14:50:05.842 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.847+0000 7fed81200640 1 -- 172.21.15.90:0/4230847318 learned_addr learned my addr 172.21.15.90:0/4230847318 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:50:05.842 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.847+0000 7fed81c00640 1 --2- 172.21.15.90:0/4230847318 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55bfaea3bf00 0x55bfaea3ff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:05.842 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.847+0000 7fed7be00640 1 --2- 172.21.15.90:0/4230847318 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bfaea3b550 0x55bfaea3fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:05.842 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.847+0000 7fed81200640 1 -- 172.21.15.90:0/4230847318 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55bfaea3bf00 msgr2=0x55bfaea3ff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:50:05.842 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.847+0000 7fed81200640 1 --2- 172.21.15.90:0/4230847318 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55bfaea3bf00 0x55bfaea3ff90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:05.842 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.847+0000 7fed81200640 1 -- 172.21.15.90:0/4230847318 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bfaea3b550 msgr2=0x55bfaea3fa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:50:05.842 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.847+0000 7fed81200640 1 --2- 172.21.15.90:0/4230847318 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bfaea3b550 0x55bfaea3fa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:05.842 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.847+0000 7fed81200640 1 -- 172.21.15.90:0/4230847318 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bfae95de80 con 0x55bfaea286b0 2024-04-02T14:50:05.842 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.848+0000 7fed81200640 1 --2- 172.21.15.90:0/4230847318 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bfaea286b0 0x55bfaea3f510 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7fed70036040 tx=0x7fed7002c800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d11e92b517ec6e50 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:50:05.843 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.848+0000 7fed62a00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:50:05.843 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.848+0000 7fed62a00640 1 -- 172.21.15.90:0/4230847318 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed7002c8b0 con 0x55bfaea286b0 2024-04-02T14:50:05.843 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.848+0000 7fed8696b740 1 -- 172.21.15.90:0/4230847318 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fed680d6050 con 0x55bfaea286b0 2024-04-02T14:50:05.843 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.848+0000 7fed62a00640 1 -- 172.21.15.90:0/4230847318 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 8594+0+0 (secure 0 0 0) 0x7fed7002ca50 con 0x55bfaea286b0 2024-04-02T14:50:05.843 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.848+0000 7fed62a00640 1 -- 172.21.15.90:0/4230847318 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fed7002cc50 con 0x55bfaea286b0 2024-04-02T14:50:05.843 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.848+0000 7fed8696b740 10 client.29926 fetch_fsmap learned FSMap version 27 2024-04-02T14:50:05.843 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.848+0000 7fed8696b740 1 -- 172.21.15.90:0/4230847318 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55bfae6a7210 con 0x55bfaea286b0 2024-04-02T14:50:05.844 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.848+0000 7fed62a00640 1 -- 172.21.15.90:0/4230847318 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fed680d6050 con 0x55bfaea286b0 2024-04-02T14:50:05.845 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.849+0000 7fed62a00640 1 -- 172.21.15.90:0/4230847318 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fed70046070 con 0x55bfaea286b0 2024-04-02T14:50:05.845 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.849+0000 7fed8696b740 10 client.29926 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:50:05.845 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.849+0000 7fed8696b740 10 client.29926 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-02T14:50:05.845 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.849+0000 7fed8696b740 20 client.29926 populate_metadata read hostname 'smithi090' 2024-04-02T14:50:05.845 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.849+0000 7fed8696b740 1 --2- 172.21.15.90:0/4230847318 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55bfaea50050 0x55bfaea52440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:05.845 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.849+0000 7fed8696b740 4 client.29926 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:50:05.845 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.849+0000 7fed8696b740 1 -- 172.21.15.90:0/4230847318 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bfae942cd0 con 0x55bfaea50050 2024-04-02T14:50:05.845 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.849+0000 7fed7be00640 1 --2- 172.21.15.90:0/4230847318 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55bfaea50050 0x55bfaea52440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:05.845 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.850+0000 7fed7be00640 1 --2- 172.21.15.90:0/4230847318 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55bfaea50050 0x55bfaea52440 crc :-1 s=READY pgs=382 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a37dc4ab1e010f52 server_cookie=35a9ca6965a93957 in_seq=0 out_seq=0 2024-04-02T14:50:05.845 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.850+0000 7fed62a00640 10 client.29926 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:50:05.845 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.850+0000 7fed62a00640 1 -- 172.21.15.90:0/4230847318 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55bfae942cd0 con 0x55bfaea50050 2024-04-02T14:50:05.845 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.850+0000 7fed62a00640 10 client.29926 handle_command_reply: tid=0 2024-04-02T14:50:05.845 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.850+0000 7fed8696b740 1 client.29926 shutdown 2024-04-02T14:50:05.845 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.851+0000 7fed8696b740 1 -- 172.21.15.90:0/4230847318 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bfaea286b0 msgr2=0x55bfaea3f510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:05.846 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.851+0000 7fed8696b740 1 --2- 172.21.15.90:0/4230847318 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bfaea286b0 0x55bfaea3f510 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7fed70036040 tx=0x7fed7002c800 comp rx=0 tx=0).stop 2024-04-02T14:50:05.846 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.851+0000 7fed8696b740 1 -- 172.21.15.90:0/4230847318 shutdown_connections 2024-04-02T14:50:05.846 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.851+0000 7fed8696b740 1 --2- 172.21.15.90:0/4230847318 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55bfaea50050 0x55bfaea52440 crc :-1 s=READY pgs=382 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:05.846 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.851+0000 7fed8696b740 1 --2- 172.21.15.90:0/4230847318 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55bfaea3bf00 0x55bfaea3ff90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:05.846 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.851+0000 7fed8696b740 1 --2- 172.21.15.90:0/4230847318 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bfaea3b550 0x55bfaea3fa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:05.846 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.851+0000 7fed8696b740 1 --2- 172.21.15.90:0/4230847318 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bfaea286b0 0x55bfaea3f510 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:05.846 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.851+0000 7fed62a00640 0 client.29926 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:50:05.846 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.851+0000 7fed8696b740 1 -- 172.21.15.90:0/4230847318 >> 172.21.15.90:0/4230847318 conn(0x55bfaea2ab50 msgr2=0x55bfaea2a2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:50:05.846 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.851+0000 7fed8696b740 1 -- 172.21.15.90:0/4230847318 shutdown_connections 2024-04-02T14:50:05.846 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.851+0000 7fed8696b740 1 -- 172.21.15.90:0/4230847318 wait complete. 2024-04-02T14:50:05.846 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.851+0000 7fed8696b740 20 client.29926 trim_cache size 0 max 16384 2024-04-02T14:50:05.860 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.867+0000 7fed8696b740 1 Processor -- start 2024-04-02T14:50:05.860 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.867+0000 7fed8696b740 1 -- start start 2024-04-02T14:50:05.860 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.867+0000 7fed8696b740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bfaea286b0 0x55bfaea41c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:05.860 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.867+0000 7fed8696b740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bfaea3b550 0x55bfaea3ac40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:05.861 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.867+0000 7fed8696b740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55bfaea3bf00 0x55bfaea4b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:05.861 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.867+0000 7fed8696b740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fed7c1ab630 con 0x55bfaea3b550 2024-04-02T14:50:05.861 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.867+0000 7fed8696b740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fed7c1ad170 con 0x55bfaea3bf00 2024-04-02T14:50:05.861 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.867+0000 7fed8696b740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fed7c1abd00 con 0x55bfaea286b0 2024-04-02T14:50:05.861 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.868+0000 7fed81200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bfaea286b0 0x55bfaea41c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:05.861 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.868+0000 7fed81200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bfaea286b0 0x55bfaea41c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:45832/0 (socket says 172.21.15.90:45832) 2024-04-02T14:50:05.861 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.868+0000 7fed81200640 1 -- 172.21.15.90:0/637604414 learned_addr learned my addr 172.21.15.90:0/637604414 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:50:05.861 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.868+0000 7fed7be00640 1 --2- 172.21.15.90:0/637604414 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bfaea3b550 0x55bfaea3ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:05.861 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.868+0000 7fed81c00640 1 --2- 172.21.15.90:0/637604414 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55bfaea3bf00 0x55bfaea4b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:05.863 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.868+0000 7fed81200640 1 -- 172.21.15.90:0/637604414 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55bfaea3bf00 msgr2=0x55bfaea4b8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:50:05.863 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.868+0000 7fed81200640 1 --2- 172.21.15.90:0/637604414 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55bfaea3bf00 0x55bfaea4b8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:05.863 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.868+0000 7fed81200640 1 -- 172.21.15.90:0/637604414 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bfaea3b550 msgr2=0x55bfaea3ac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:50:05.863 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.868+0000 7fed81200640 1 --2- 172.21.15.90:0/637604414 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bfaea3b550 0x55bfaea3ac40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:05.863 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.868+0000 7fed81200640 1 -- 172.21.15.90:0/637604414 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bfaea28400 con 0x55bfaea286b0 2024-04-02T14:50:05.863 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.868+0000 7fed81200640 1 --2- 172.21.15.90:0/637604414 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bfaea286b0 0x55bfaea41c40 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7fed70005e80 tx=0x7fed70009680 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d9d6da20e9bd607a server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:50:05.863 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.869+0000 7fed6d600640 1 -- 172.21.15.90:0/637604414 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed7002c8b0 con 0x55bfaea286b0 2024-04-02T14:50:05.863 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.869+0000 7fed6d600640 1 -- 172.21.15.90:0/637604414 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fed7002ca50 con 0x55bfaea286b0 2024-04-02T14:50:05.863 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.869+0000 7fed6d600640 1 -- 172.21.15.90:0/637604414 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed700054b0 con 0x55bfaea286b0 2024-04-02T14:50:05.863 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.869+0000 7fed8696b740 1 -- 172.21.15.90:0/637604414 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bfaea286b0 msgr2=0x55bfaea41c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:05.863 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.869+0000 7fed8696b740 1 --2- 172.21.15.90:0/637604414 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bfaea286b0 0x55bfaea41c40 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7fed70005e80 tx=0x7fed70009680 comp rx=0 tx=0).stop 2024-04-02T14:50:05.863 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.869+0000 7fed8696b740 1 -- 172.21.15.90:0/637604414 shutdown_connections 2024-04-02T14:50:05.864 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.869+0000 7fed8696b740 1 --2- 172.21.15.90:0/637604414 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55bfaea3bf00 0x55bfaea4b8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:05.864 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.869+0000 7fed8696b740 1 --2- 172.21.15.90:0/637604414 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bfaea3b550 0x55bfaea3ac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:05.864 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.869+0000 7fed8696b740 1 --2- 172.21.15.90:0/637604414 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bfaea286b0 0x55bfaea41c40 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:05.864 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.870+0000 7fed8696b740 1 -- 172.21.15.90:0/637604414 >> 172.21.15.90:0/637604414 conn(0x55bfaea2ab50 msgr2=0x55bfaea3a530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:50:05.864 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.870+0000 7fed8696b740 1 -- 172.21.15.90:0/637604414 shutdown_connections 2024-04-02T14:50:05.864 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.870+0000 7fed8696b740 1 -- 172.21.15.90:0/637604414 wait complete. 2024-04-02T14:50:05.864 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.870+0000 7fed8696b740 1 -- start start 2024-04-02T14:50:05.864 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.870+0000 7fed8696b740 1 Processor -- start 2024-04-02T14:50:05.866 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.871+0000 7fed8696b740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bfaea286b0 0x55bfaea47dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:05.866 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.871+0000 7fed8696b740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bfaea3b550 0x55bfaea48310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:05.866 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.871+0000 7fed8696b740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55bfaea3bf00 0x55bfaea570b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:05.867 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.871+0000 7fed8696b740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fed7c1acaa0 con 0x55bfaea3b550 2024-04-02T14:50:05.867 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.871+0000 7fed8696b740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fed7c1ac3d0 con 0x55bfaea3bf00 2024-04-02T14:50:05.867 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.871+0000 7fed8696b740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fed7c1a9630 con 0x55bfaea286b0 2024-04-02T14:50:05.867 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.871+0000 7fed8696b740 10 client.0 _ll_register_callbacks cb 0x7fed70005d50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:50:05.867 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.871+0000 7fed81200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bfaea286b0 0x55bfaea47dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:05.867 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.871+0000 7fed81200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bfaea286b0 0x55bfaea47dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:45834/0 (socket says 172.21.15.90:45834) 2024-04-02T14:50:05.867 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.871+0000 7fed81200640 1 -- 172.21.15.90:0/348396389 learned_addr learned my addr 172.21.15.90:0/348396389 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:50:05.867 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.871+0000 7fed81c00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55bfaea3bf00 0x55bfaea570b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:05.867 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.871+0000 7fed7be00640 1 --2- 172.21.15.90:0/348396389 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bfaea3b550 0x55bfaea48310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:05.867 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.871+0000 7fed81200640 1 -- 172.21.15.90:0/348396389 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55bfaea3bf00 msgr2=0x55bfaea570b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:50:05.867 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.871+0000 7fed81200640 1 --2- 172.21.15.90:0/348396389 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55bfaea3bf00 0x55bfaea570b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:05.867 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.871+0000 7fed81200640 1 -- 172.21.15.90:0/348396389 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bfaea3b550 msgr2=0x55bfaea48310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:50:05.867 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.871+0000 7fed81200640 1 --2- 172.21.15.90:0/348396389 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bfaea3b550 0x55bfaea48310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:05.867 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.871+0000 7fed81200640 1 -- 172.21.15.90:0/348396389 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bfae773620 con 0x55bfaea286b0 2024-04-02T14:50:05.867 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.871+0000 7fed81c00640 1 --2- 172.21.15.90:0/348396389 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55bfaea3bf00 0x55bfaea570b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:50:05.868 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.871+0000 7fed7be00640 1 --2- 172.21.15.90:0/348396389 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bfaea3b550 0x55bfaea48310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:50:05.868 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.872+0000 7fed81200640 1 --2- 172.21.15.90:0/348396389 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bfaea286b0 0x55bfaea47dd0 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7fed7002c170 tx=0x7fed70000bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1a3573d06db7884e server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:50:05.868 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.872+0000 7fed62a00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:50:05.868 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.872+0000 7fed62a00640 1 -- 172.21.15.90:0/348396389 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed7005c060 con 0x55bfaea286b0 2024-04-02T14:50:05.868 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.872+0000 7fed62a00640 1 -- 172.21.15.90:0/348396389 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 8594+0+0 (secure 0 0 0) 0x7fed70005230 con 0x55bfaea286b0 2024-04-02T14:50:05.868 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.872+0000 7fed8696b740 1 -- 172.21.15.90:0/348396389 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fed7002ca50 con 0x55bfaea286b0 2024-04-02T14:50:05.868 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.872+0000 7fed62a00640 1 -- 172.21.15.90:0/348396389 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fed70005430 con 0x55bfaea286b0 2024-04-02T14:50:05.868 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.872+0000 7fed8696b740 10 client.29932 fetch_fsmap learned FSMap version 27 2024-04-02T14:50:05.868 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.872+0000 7fed8696b740 1 -- 172.21.15.90:0/348396389 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55bfaeab6a90 con 0x55bfaea286b0 2024-04-02T14:50:05.868 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.872+0000 7fed62a00640 1 -- 172.21.15.90:0/348396389 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fed7002ca50 con 0x55bfaea286b0 2024-04-02T14:50:05.868 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.873+0000 7fed62a00640 1 -- 172.21.15.90:0/348396389 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fed70046070 con 0x55bfaea286b0 2024-04-02T14:50:05.868 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.873+0000 7fed8696b740 10 client.29932 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:50:05.868 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.873+0000 7fed8696b740 10 client.29932 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-02T14:50:05.868 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.873+0000 7fed8696b740 20 client.29932 populate_metadata read hostname 'smithi090' 2024-04-02T14:50:05.868 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.873+0000 7fed8696b740 1 --2- 172.21.15.90:0/348396389 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55bfaeab6ff0 0x55bfaeab9410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:05.868 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.873+0000 7fed8696b740 4 client.29932 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-02T14:50:05.869 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.873+0000 7fed8696b740 1 -- 172.21.15.90:0/348396389 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55bfae9a02b0 con 0x55bfaeab6ff0 2024-04-02T14:50:05.869 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.873+0000 7fed7be00640 1 --2- 172.21.15.90:0/348396389 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55bfaeab6ff0 0x55bfaeab9410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:05.869 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.874+0000 7fed7be00640 1 --2- 172.21.15.90:0/348396389 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55bfaeab6ff0 0x55bfaeab9410 crc :-1 s=READY pgs=383 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b7a4f30ffe123c30 server_cookie=9b99fb48461b279c in_seq=0 out_seq=0 2024-04-02T14:50:05.869 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:05.874+0000 7fed62a00640 10 client.29932 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:50:06.102 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:50:05 smithi092 ceph-mon[36497]: pgmap v476: 97 pgs: 25 active+clean+snaptrim_wait, 18 active+clean+snaptrim, 54 active+clean; 4.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 14 MiB/s wr, 1.36k op/s 2024-04-02T14:50:06.178 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:50:05 smithi177 ceph-mon[36431]: pgmap v476: 97 pgs: 25 active+clean+snaptrim_wait, 18 active+clean+snaptrim, 54 active+clean; 4.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 14 MiB/s wr, 1.36k op/s 2024-04-02T14:50:06.180 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:50:05 smithi090 ceph-mon[31674]: pgmap v476: 97 pgs: 25 active+clean+snaptrim_wait, 18 active+clean+snaptrim, 54 active+clean; 4.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 14 MiB/s wr, 1.36k op/s 2024-04-02T14:50:06.823 INFO:teuthology.orchestra.run.smithi090.stderr:progress: 875.0 s, 32.6 tps, lat 1039.166 ms stddev 1458.060 2024-04-02T14:50:07.495 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-02T14:50:07.496 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-02T14:50:07.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:07.747+0000 7f6c11a00640 1 -- 172.21.15.90:0/3406715631 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6c0c108ec0 msgr2=0x7f6c0c110a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:07.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:07.747+0000 7f6c11a00640 1 --2- 172.21.15.90:0/3406715631 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6c0c108ec0 0x7f6c0c110a90 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7f6c04009be0 tx=0x7f6c0402bef0 comp rx=0 tx=0).stop 2024-04-02T14:50:07.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:07.747+0000 7f6c11a00640 1 -- 172.21.15.90:0/3406715631 shutdown_connections 2024-04-02T14:50:07.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:07.747+0000 7f6c11a00640 1 --2- 172.21.15.90:0/3406715631 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6c0c108ec0 0x7f6c0c110a90 unknown :-1 s=CLOSED pgs=1013 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:07.741 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:07.747+0000 7f6c11a00640 1 --2- 172.21.15.90:0/3406715631 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6c0c1047a0 0x7f6c0c1087e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:07.741 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:07.747+0000 7f6c11a00640 1 --2- 172.21.15.90:0/3406715631 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6c0c103df0 0x7f6c0c1041d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:07.741 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:07.747+0000 7f6c11a00640 1 -- 172.21.15.90:0/3406715631 >> 172.21.15.90:0/3406715631 conn(0x7f6c0c0ff610 msgr2=0x7f6c0c101a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:50:07.741 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:07.747+0000 7f6c11a00640 1 -- 172.21.15.90:0/3406715631 shutdown_connections 2024-04-02T14:50:07.741 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:07.747+0000 7f6c11a00640 1 -- 172.21.15.90:0/3406715631 wait complete. 2024-04-02T14:50:07.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:07.748+0000 7f6c11a00640 1 Processor -- start 2024-04-02T14:50:07.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:07.748+0000 7f6c11a00640 1 -- start start 2024-04-02T14:50:07.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:07.748+0000 7f6c11a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6c0c103df0 0x7f6c0c19a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:07.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:07.749+0000 7f6c11a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6c0c108ec0 0x7f6c0c19a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:07.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:07.749+0000 7f6c11a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6c0c19eca0 0x7f6c0c19f150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:07.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:07.749+0000 7f6c11a00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f6c0c113f70 con 0x7f6c0c108ec0 2024-04-02T14:50:07.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:07.749+0000 7f6c11a00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f6c0c113df0 con 0x7f6c0c19eca0 2024-04-02T14:50:07.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:07.749+0000 7f6c11a00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f6c0c1140f0 con 0x7f6c0c103df0 2024-04-02T14:50:07.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:07.749+0000 7f6c0be00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6c0c103df0 0x7f6c0c19a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:07.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:07.749+0000 7f6c0b400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6c0c108ec0 0x7f6c0c19a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:07.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:07.749+0000 7f6c0be00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6c0c103df0 0x7f6c0c19a430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:45842/0 (socket says 172.21.15.90:45842) 2024-04-02T14:50:07.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:07.749+0000 7f6c0be00640 1 -- 172.21.15.90:0/3649377175 learned_addr learned my addr 172.21.15.90:0/3649377175 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:50:07.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:07.749+0000 7f6c11000640 1 --2- 172.21.15.90:0/3649377175 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6c0c19eca0 0x7f6c0c19f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:07.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:07.749+0000 7f6c0be00640 1 -- 172.21.15.90:0/3649377175 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6c0c19eca0 msgr2=0x7f6c0c19f150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:07.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:07.749+0000 7f6c0be00640 1 --2- 172.21.15.90:0/3649377175 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6c0c19eca0 0x7f6c0c19f150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:07.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:07.749+0000 7f6c0be00640 1 -- 172.21.15.90:0/3649377175 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6c0c108ec0 msgr2=0x7f6c0c19a970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:07.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:07.749+0000 7f6c0be00640 1 --2- 172.21.15.90:0/3649377175 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6c0c108ec0 0x7f6c0c19a970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:07.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:07.749+0000 7f6c0be00640 1 -- 172.21.15.90:0/3649377175 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c0c19f790 con 0x7f6c0c103df0 2024-04-02T14:50:07.745 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:07.749+0000 7f6c11000640 1 --2- 172.21.15.90:0/3649377175 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6c0c19eca0 0x7f6c0c19f150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:50:07.745 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:07.750+0000 7f6c0be00640 1 --2- 172.21.15.90:0/3649377175 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6c0c103df0 0x7f6c0c19a430 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f6bfc00e9f0 tx=0x7f6bfc00edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:50:07.745 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:07.750+0000 7f6c08c00640 1 -- 172.21.15.90:0/3649377175 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bfc00cd80 con 0x7f6c0c103df0 2024-04-02T14:50:07.745 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:07.750+0000 7f6c08c00640 1 -- 172.21.15.90:0/3649377175 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6bfc005500 con 0x7f6c0c103df0 2024-04-02T14:50:07.745 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:07.750+0000 7f6c11a00640 1 -- 172.21.15.90:0/3649377175 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c0c19fa80 con 0x7f6c0c103df0 2024-04-02T14:50:07.745 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:07.750+0000 7f6c08c00640 1 -- 172.21.15.90:0/3649377175 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bfc010690 con 0x7f6c0c103df0 2024-04-02T14:50:07.745 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:07.750+0000 7f6c11a00640 1 -- 172.21.15.90:0/3649377175 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c0c1a7210 con 0x7f6c0c103df0 2024-04-02T14:50:07.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:07.753+0000 7f6c08c00640 1 -- 172.21.15.90:0/3649377175 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f6bfc0046c0 con 0x7f6c0c103df0 2024-04-02T14:50:07.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:07.753+0000 7f6c15d9f740 1 Processor -- start 2024-04-02T14:50:07.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:07.753+0000 7f6c15d9f740 1 -- start start 2024-04-02T14:50:07.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:07.753+0000 7f6c15d9f740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55bc21881960 0x55bc21881d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:07.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:07.753+0000 7f6c15d9f740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bc21882280 0x55bc2188a920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:07.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:07.753+0000 7f6c15d9f740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bc2188ae60 0x55bc2188d250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:07.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:07.753+0000 7f6c15d9f740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55bc217e5770 con 0x55bc21882280 2024-04-02T14:50:07.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:07.753+0000 7f6c15d9f740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55bc2178d700 con 0x55bc21881960 2024-04-02T14:50:07.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:07.754+0000 7f6c15d9f740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55bc217e1720 con 0x55bc2188ae60 2024-04-02T14:50:07.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:07.754+0000 7f6c11000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55bc21881960 0x55bc21881d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:07.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:07.754+0000 7f6c11000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55bc21881960 0x55bc21881d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:47546/0 (socket says 172.21.15.90:47546) 2024-04-02T14:50:07.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:07.754+0000 7f6c11000640 1 -- 172.21.15.90:0/475052111 learned_addr learned my addr 172.21.15.90:0/475052111 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:50:07.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:07.754+0000 7f6c0b400640 1 --2- 172.21.15.90:0/475052111 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bc2188ae60 0x55bc2188d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:07.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:07.754+0000 7f6c0be00640 1 --2- 172.21.15.90:0/475052111 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bc21882280 0x55bc2188a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:07.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:07.754+0000 7f6c11000640 1 -- 172.21.15.90:0/475052111 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bc2188ae60 msgr2=0x55bc2188d250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:50:07.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:07.754+0000 7f6c11000640 1 --2- 172.21.15.90:0/475052111 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bc2188ae60 0x55bc2188d250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:07.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:07.754+0000 7f6c11000640 1 -- 172.21.15.90:0/475052111 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bc21882280 msgr2=0x55bc2188a920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:50:07.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:07.754+0000 7f6c11000640 1 --2- 172.21.15.90:0/475052111 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bc21882280 0x55bc2188a920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:07.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:07.754+0000 7f6c11000640 1 -- 172.21.15.90:0/475052111 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bc215c37d0 con 0x55bc21881960 2024-04-02T14:50:07.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:07.754+0000 7f6c0b400640 1 --2- 172.21.15.90:0/475052111 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bc2188ae60 0x55bc2188d250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:50:07.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:07.754+0000 7f6c11000640 1 --2- 172.21.15.90:0/475052111 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55bc21881960 0x55bc21881d40 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7f6c04036040 tx=0x7f6c04005c80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3b6058997730e913 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:50:07.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:07.754+0000 7f6c08c00640 1 --2- 172.21.15.90:0/3649377175 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f6bf00a3da0 0x7f6bf00a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:07.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:07.755+0000 7f6c08c00640 1 -- 172.21.15.90:0/3649377175 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 8594+0+0 (secure 0 0 0) 0x7f6bfc0d0c40 con 0x7f6c0c103df0 2024-04-02T14:50:07.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:07.755+0000 7f6c01600640 1 -- 172.21.15.90:0/475052111 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c04043030 con 0x55bc21881960 2024-04-02T14:50:07.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:07.755+0000 7f6c0be00640 1 --2- 172.21.15.90:0/3649377175 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f6bf00a3da0 0x7f6bf00a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:07.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:07.755+0000 7f6c01600640 1 -- 172.21.15.90:0/475052111 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6c04003970 con 0x55bc21881960 2024-04-02T14:50:07.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:07.755+0000 7f6c01600640 1 -- 172.21.15.90:0/475052111 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c0402cb30 con 0x55bc21881960 2024-04-02T14:50:07.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:07.756+0000 7f6c0be00640 1 --2- 172.21.15.90:0/3649377175 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f6bf00a3da0 0x7f6bf00a6260 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x55bc21881430 tx=0x7f6bfc004a40 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:50:07.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:07.755+0000 7f6c15d9f740 1 -- 172.21.15.90:0/475052111 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55bc21881960 msgr2=0x55bc21881d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:07.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:07.756+0000 7f6c15d9f740 1 --2- 172.21.15.90:0/475052111 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55bc21881960 0x55bc21881d40 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7f6c04036040 tx=0x7f6c04005c80 comp rx=0 tx=0).stop 2024-04-02T14:50:07.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:07.756+0000 7f6c15d9f740 1 -- 172.21.15.90:0/475052111 shutdown_connections 2024-04-02T14:50:07.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:07.756+0000 7f6c15d9f740 1 --2- 172.21.15.90:0/475052111 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bc2188ae60 0x55bc2188d250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:07.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:07.756+0000 7f6c15d9f740 1 --2- 172.21.15.90:0/475052111 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bc21882280 0x55bc2188a920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:07.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:07.756+0000 7f6c15d9f740 1 --2- 172.21.15.90:0/475052111 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55bc21881960 0x55bc21881d40 unknown :-1 s=CLOSED pgs=1014 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:07.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:07.756+0000 7f6c15d9f740 1 -- 172.21.15.90:0/475052111 >> 172.21.15.90:0/475052111 conn(0x55bc2187c090 msgr2=0x55bc2187c470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:50:07.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:07.756+0000 7f6c15d9f740 1 -- 172.21.15.90:0/475052111 shutdown_connections 2024-04-02T14:50:07.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:07.756+0000 7f6c15d9f740 1 -- 172.21.15.90:0/475052111 wait complete. 2024-04-02T14:50:07.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:07.756+0000 7f6c15d9f740 1 -- start start 2024-04-02T14:50:07.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:07.756+0000 7f6c15d9f740 1 Processor -- start 2024-04-02T14:50:07.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:07.757+0000 7f6c15d9f740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bc21881960 0x55bc21887ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:07.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:07.757+0000 7f6c15d9f740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bc21882280 0x55bc21888430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:07.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:07.757+0000 7f6c15d9f740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55bc2188ae60 0x55bc21888970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:07.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:07.757+0000 7f6c15d9f740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55bc2188dbf0 con 0x55bc21882280 2024-04-02T14:50:07.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:07.757+0000 7f6c15d9f740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55bc217a0ac0 con 0x55bc2188ae60 2024-04-02T14:50:07.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:07.757+0000 7f6c15d9f740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55bc2188dd70 con 0x55bc21881960 2024-04-02T14:50:07.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:07.757+0000 7f6c15d9f740 10 client.0 _ll_register_callbacks cb 0x55bc217a3180 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:50:07.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:07.757+0000 7f6c11000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bc21881960 0x55bc21887ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:07.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:07.757+0000 7f6c11000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bc21881960 0x55bc21887ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:45856/0 (socket says 172.21.15.90:45856) 2024-04-02T14:50:07.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:07.757+0000 7f6c11000640 1 -- 172.21.15.90:0/4038074664 learned_addr learned my addr 172.21.15.90:0/4038074664 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:50:07.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:07.757+0000 7f6c0b400640 1 --2- 172.21.15.90:0/4038074664 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bc21882280 0x55bc21888430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:07.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:07.757+0000 7f6c11000640 1 --2- 172.21.15.90:0/4038074664 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55bc2188ae60 0x55bc21888970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:07.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:07.758+0000 7f6c11000640 1 -- 172.21.15.90:0/4038074664 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55bc2188ae60 msgr2=0x55bc21888970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:50:07.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:07.758+0000 7f6c11000640 1 --2- 172.21.15.90:0/4038074664 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55bc2188ae60 0x55bc21888970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:07.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:07.758+0000 7f6c11000640 1 -- 172.21.15.90:0/4038074664 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bc21882280 msgr2=0x55bc21888430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:50:07.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:07.758+0000 7f6c11000640 1 --2- 172.21.15.90:0/4038074664 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bc21882280 0x55bc21888430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:07.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:07.758+0000 7f6c11000640 1 -- 172.21.15.90:0/4038074664 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bc214f93f0 con 0x55bc21881960 2024-04-02T14:50:07.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:07.758+0000 7f6c11000640 1 --2- 172.21.15.90:0/4038074664 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bc21881960 0x55bc21887ef0 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f6c040092f0 tx=0x7f6c0402ca10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fc5cb4f7e36b0caf server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:50:07.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:07.758+0000 7f6bf6000640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:50:07.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:07.758+0000 7f6bf6000640 1 -- 172.21.15.90:0/4038074664 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c04043070 con 0x55bc21881960 2024-04-02T14:50:07.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:07.758+0000 7f6bf6000640 1 -- 172.21.15.90:0/4038074664 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 8594+0+0 (secure 0 0 0) 0x7f6c0404d030 con 0x55bc21881960 2024-04-02T14:50:07.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:07.758+0000 7f6c15d9f740 1 -- 172.21.15.90:0/4038074664 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6c04003970 con 0x55bc21881960 2024-04-02T14:50:07.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:07.759+0000 7f6bf6000640 1 -- 172.21.15.90:0/4038074664 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f6c0405a050 con 0x55bc21881960 2024-04-02T14:50:07.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:07.759+0000 7f6c15d9f740 10 client.29944 fetch_fsmap learned FSMap version 27 2024-04-02T14:50:07.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:07.759+0000 7f6c15d9f740 1 -- 172.21.15.90:0/4038074664 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55bc214b90f0 con 0x55bc21881960 2024-04-02T14:50:07.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:07.759+0000 7f6bf6000640 1 -- 172.21.15.90:0/4038074664 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6c04003970 con 0x55bc21881960 2024-04-02T14:50:07.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:07.759+0000 7f6bf6000640 1 -- 172.21.15.90:0/4038074664 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f6c0403f560 con 0x55bc21881960 2024-04-02T14:50:07.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:07.759+0000 7f6c15d9f740 10 client.29944 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:50:07.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:07.759+0000 7f6c15d9f740 10 client.29944 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:50:07.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:07.759+0000 7f6c15d9f740 20 client.29944 populate_metadata read hostname 'smithi090' 2024-04-02T14:50:07.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:07.759+0000 7f6c15d9f740 1 --2- 172.21.15.90:0/4038074664 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55bc2189ace0 0x55bc2189d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:07.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:07.759+0000 7f6c15d9f740 4 client.29944 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:50:07.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:07.759+0000 7f6c15d9f740 1 -- 172.21.15.90:0/4038074664 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bc21786630 con 0x55bc2189ace0 2024-04-02T14:50:07.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:07.760+0000 7f6c0b400640 1 --2- 172.21.15.90:0/4038074664 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55bc2189ace0 0x55bc2189d0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:07.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:07.760+0000 7f6c0b400640 1 --2- 172.21.15.90:0/4038074664 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55bc2189ace0 0x55bc2189d0d0 crc :-1 s=READY pgs=384 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9e28e4c724e6a5f2 server_cookie=359b096c54a3f926 in_seq=0 out_seq=0 2024-04-02T14:50:07.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:07.760+0000 7f6bf6000640 10 client.29944 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:50:07.930 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:50:07 smithi090 ceph-mon[31674]: pgmap v477: 97 pgs: 40 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 4.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 19 MiB/s wr, 1.71k op/s 2024-04-02T14:50:08.102 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:50:07 smithi092 ceph-mon[36497]: pgmap v477: 97 pgs: 40 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 4.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 19 MiB/s wr, 1.71k op/s 2024-04-02T14:50:08.333 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:50:07 smithi177 ceph-mon[36431]: pgmap v477: 97 pgs: 40 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 4.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 19 MiB/s wr, 1.71k op/s 2024-04-02T14:50:10.102 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:50:09 smithi092 ceph-mon[36497]: pgmap v478: 97 pgs: 40 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 4.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 19 MiB/s wr, 1.71k op/s 2024-04-02T14:50:10.178 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:50:09 smithi177 ceph-mon[36431]: pgmap v478: 97 pgs: 40 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 4.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 19 MiB/s wr, 1.71k op/s 2024-04-02T14:50:10.180 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:50:09 smithi090 ceph-mon[31674]: pgmap v478: 97 pgs: 40 active+clean+snaptrim_wait, 24 active+clean+snaptrim, 33 active+clean; 4.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 19 MiB/s wr, 1.71k op/s 2024-04-02T14:50:11.791 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:11.799+0000 7fed62a00640 1 -- 172.21.15.90:0/348396389 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55bfae9a02b0 con 0x55bfaeab6ff0 2024-04-02T14:50:11.791 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:11.799+0000 7fed62a00640 10 client.29932 handle_command_reply: tid=0 2024-04-02T14:50:11.792 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:11.799+0000 7fed8696b740 1 client.29932 shutdown 2024-04-02T14:50:11.792 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:11.799+0000 7fed8696b740 1 -- 172.21.15.90:0/348396389 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bfaea286b0 msgr2=0x55bfaea47dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:11.792 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:11.799+0000 7fed8696b740 1 --2- 172.21.15.90:0/348396389 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bfaea286b0 0x55bfaea47dd0 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7fed7002c170 tx=0x7fed70000bc0 comp rx=0 tx=0).stop 2024-04-02T14:50:11.792 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:11.799+0000 7fed8696b740 1 -- 172.21.15.90:0/348396389 shutdown_connections 2024-04-02T14:50:11.793 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:11.799+0000 7fed8696b740 1 --2- 172.21.15.90:0/348396389 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55bfaeab6ff0 0x55bfaeab9410 crc :-1 s=READY pgs=383 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:11.793 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:11.799+0000 7fed8696b740 1 --2- 172.21.15.90:0/348396389 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55bfaea3bf00 0x55bfaea570b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:11.793 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:11.799+0000 7fed8696b740 1 --2- 172.21.15.90:0/348396389 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bfaea3b550 0x55bfaea48310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:11.793 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:11.799+0000 7fed8696b740 1 --2- 172.21.15.90:0/348396389 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bfaea286b0 0x55bfaea47dd0 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:11.793 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:11.799+0000 7fed8696b740 1 -- 172.21.15.90:0/348396389 >> 172.21.15.90:0/348396389 conn(0x55bfaea2ab50 msgr2=0x55bfaea26db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:50:11.793 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:11.799+0000 7fed62a00640 0 client.29932 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:50:11.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:11.799+0000 7f6bf6000640 1 -- 172.21.15.90:0/4038074664 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55bc21786630 con 0x55bc2189ace0 2024-04-02T14:50:11.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:11.799+0000 7f6bf6000640 10 client.29944 handle_command_reply: tid=0 2024-04-02T14:50:11.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:11.799+0000 7f6c15d9f740 1 client.29944 shutdown 2024-04-02T14:50:11.794 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:11.799+0000 7fed8696b740 1 -- 172.21.15.90:0/348396389 shutdown_connections 2024-04-02T14:50:11.794 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:11.800+0000 7fed8696b740 1 -- 172.21.15.90:0/348396389 wait complete. 2024-04-02T14:50:11.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:11.800+0000 7f6c15d9f740 1 -- 172.21.15.90:0/4038074664 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bc21881960 msgr2=0x55bc21887ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:11.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:11.800+0000 7f6c15d9f740 1 --2- 172.21.15.90:0/4038074664 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bc21881960 0x55bc21887ef0 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f6c040092f0 tx=0x7f6c0402ca10 comp rx=0 tx=0).stop 2024-04-02T14:50:11.795 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:11.800+0000 7fed8696b740 20 client.29932 trim_cache size 0 max 16384 2024-04-02T14:50:11.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:11.800+0000 7f6c15d9f740 1 -- 172.21.15.90:0/4038074664 shutdown_connections 2024-04-02T14:50:11.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:11.800+0000 7f6c15d9f740 1 --2- 172.21.15.90:0/4038074664 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55bc2189ace0 0x55bc2189d0d0 crc :-1 s=READY pgs=384 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:11.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:11.800+0000 7f6c15d9f740 1 --2- 172.21.15.90:0/4038074664 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55bc2188ae60 0x55bc21888970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:11.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:11.800+0000 7f6c15d9f740 1 --2- 172.21.15.90:0/4038074664 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bc21882280 0x55bc21888430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:11.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:11.800+0000 7f6c15d9f740 1 --2- 172.21.15.90:0/4038074664 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bc21881960 0x55bc21887ef0 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:11.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:11.800+0000 7f6c15d9f740 1 -- 172.21.15.90:0/4038074664 >> 172.21.15.90:0/4038074664 conn(0x55bc2187c090 msgr2=0x55bc2187dde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:50:11.796 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:{ 2024-04-02T14:50:11.796 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "message": "", 2024-04-02T14:50:11.796 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "return_code": 0 2024-04-02T14:50:11.796 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:} 2024-04-02T14:50:11.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:11.800+0000 7f6bf6000640 0 client.29944 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:50:11.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:11.800+0000 7f6c15d9f740 1 -- 172.21.15.90:0/4038074664 shutdown_connections 2024-04-02T14:50:11.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:11.800+0000 7f6c15d9f740 1 -- 172.21.15.90:0/4038074664 wait complete. 2024-04-02T14:50:11.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:11.800+0000 7f6c15d9f740 20 client.29944 trim_cache size 0 max 16384 2024-04-02T14:50:11.797 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:11.801+0000 7fed83c00640 1 -- 172.21.15.90:0/3785836099 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fed640a3e40 msgr2=0x7fed640a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:11.798 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:11.801+0000 7fed83c00640 1 --2- 172.21.15.90:0/3785836099 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fed640a3e40 0x7fed640a6300 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x55bfaea291d0 tx=0x7fed74009500 comp rx=0 tx=0).stop 2024-04-02T14:50:11.798 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:11.801+0000 7fed83c00640 1 -- 172.21.15.90:0/3785836099 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fed7c102470 msgr2=0x7fed7c198800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:11.798 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:11.801+0000 7fed83c00640 1 --2- 172.21.15.90:0/3785836099 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fed7c102470 0x7fed7c198800 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fed6800e9f0 tx=0x7fed6800eeb0 comp rx=0 tx=0).stop 2024-04-02T14:50:11.798 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:11.801+0000 7fed83c00640 1 -- 172.21.15.90:0/3785836099 shutdown_connections 2024-04-02T14:50:11.798 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:11.801+0000 7fed83c00640 1 --2- 172.21.15.90:0/3785836099 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fed640a3e40 0x7fed640a6300 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:11.798 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:11.801+0000 7fed83c00640 1 --2- 172.21.15.90:0/3785836099 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fed7c107ac0 0x7fed7c19cb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:11.798 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:11.801+0000 7fed83c00640 1 --2- 172.21.15.90:0/3785836099 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fed7c102470 0x7fed7c198800 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:11.798 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:11.801+0000 7fed83c00640 1 --2- 172.21.15.90:0/3785836099 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fed7c101ab0 0x7fed7c1982c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:11.798 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:11.801+0000 7fed83c00640 1 -- 172.21.15.90:0/3785836099 >> 172.21.15.90:0/3785836099 conn(0x7fed7c0fd5c0 msgr2=0x7fed7c0fdaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:50:11.798 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:11.801+0000 7fed83c00640 1 -- 172.21.15.90:0/3785836099 shutdown_connections 2024-04-02T14:50:11.798 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:11.801+0000 7fed83c00640 1 -- 172.21.15.90:0/3785836099 wait complete. 2024-04-02T14:50:11.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:11.820+0000 7f6c15d9f740 1 Processor -- start 2024-04-02T14:50:11.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:11.820+0000 7f6c15d9f740 1 -- start start 2024-04-02T14:50:11.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:11.821+0000 7f6c15d9f740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55bc21881960 0x55bc2188a620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:11.815 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:11.821+0000 7f6c15d9f740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bc21882280 0x55bc21896ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:11.815 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:11.821+0000 7f6c15d9f740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bc2188ae60 0x55bc21887070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:11.815 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:11.821+0000 7f6c15d9f740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f6c0c1af860 con 0x55bc21882280 2024-04-02T14:50:11.815 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:11.821+0000 7f6c15d9f740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f6c0c1b13a0 con 0x55bc21881960 2024-04-02T14:50:11.815 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:11.821+0000 7f6c15d9f740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f6c0c1aff30 con 0x55bc2188ae60 2024-04-02T14:50:11.815 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:11.821+0000 7f6c11000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55bc21881960 0x55bc2188a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:11.815 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:11.821+0000 7f6c11000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55bc21881960 0x55bc2188a620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:47566/0 (socket says 172.21.15.90:47566) 2024-04-02T14:50:11.815 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:11.821+0000 7f6c11000640 1 -- 172.21.15.90:0/537049777 learned_addr learned my addr 172.21.15.90:0/537049777 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:50:11.815 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:11.821+0000 7f6c11000640 1 --2- 172.21.15.90:0/537049777 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bc2188ae60 0x55bc21887070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:11.815 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:11.821+0000 7f6c0b400640 1 --2- 172.21.15.90:0/537049777 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bc21882280 0x55bc21896ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:11.815 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:11.821+0000 7f6c11000640 1 -- 172.21.15.90:0/537049777 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bc2188ae60 msgr2=0x55bc21887070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:50:11.815 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:11.821+0000 7f6c11000640 1 --2- 172.21.15.90:0/537049777 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bc2188ae60 0x55bc21887070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:11.815 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:11.821+0000 7f6c11000640 1 -- 172.21.15.90:0/537049777 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bc21882280 msgr2=0x55bc21896ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:50:11.815 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:11.821+0000 7f6c11000640 1 --2- 172.21.15.90:0/537049777 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bc21882280 0x55bc21896ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:11.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:11.821+0000 7f6c11000640 1 -- 172.21.15.90:0/537049777 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bc218820b0 con 0x55bc21881960 2024-04-02T14:50:11.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:11.822+0000 7f6c11000640 1 --2- 172.21.15.90:0/537049777 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55bc21881960 0x55bc2188a620 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7f6c04000f80 tx=0x7f6c0404a070 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b28c279c54cb4d13 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:50:11.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:11.822+0000 7f6c00c00640 1 -- 172.21.15.90:0/537049777 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c04043070 con 0x55bc21881960 2024-04-02T14:50:11.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:11.822+0000 7f6c00c00640 1 -- 172.21.15.90:0/537049777 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6c0404c020 con 0x55bc21881960 2024-04-02T14:50:11.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:11.822+0000 7f6c00c00640 1 -- 172.21.15.90:0/537049777 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c0403fd00 con 0x55bc21881960 2024-04-02T14:50:11.820 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:11.823+0000 7f6c15d9f740 1 -- 172.21.15.90:0/537049777 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55bc21881960 msgr2=0x55bc2188a620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:11.820 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:11.823+0000 7f6c15d9f740 1 --2- 172.21.15.90:0/537049777 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55bc21881960 0x55bc2188a620 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7f6c04000f80 tx=0x7f6c0404a070 comp rx=0 tx=0).stop 2024-04-02T14:50:11.820 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:11.823+0000 7f6c15d9f740 1 -- 172.21.15.90:0/537049777 shutdown_connections 2024-04-02T14:50:11.820 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:11.823+0000 7f6c15d9f740 1 --2- 172.21.15.90:0/537049777 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bc2188ae60 0x55bc21887070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:11.820 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:11.823+0000 7f6c15d9f740 1 --2- 172.21.15.90:0/537049777 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bc21882280 0x55bc21896ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:11.821 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:11.823+0000 7f6c15d9f740 1 --2- 172.21.15.90:0/537049777 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55bc21881960 0x55bc2188a620 unknown :-1 s=CLOSED pgs=1015 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:11.821 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:11.823+0000 7f6c15d9f740 1 -- 172.21.15.90:0/537049777 >> 172.21.15.90:0/537049777 conn(0x55bc2187c090 msgr2=0x55bc2187aa80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:50:11.821 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:11.823+0000 7f6c15d9f740 1 -- 172.21.15.90:0/537049777 shutdown_connections 2024-04-02T14:50:11.821 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:11.823+0000 7f6c15d9f740 1 -- 172.21.15.90:0/537049777 wait complete. 2024-04-02T14:50:11.821 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:11.823+0000 7f6c15d9f740 1 -- start start 2024-04-02T14:50:11.821 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:11.823+0000 7f6c15d9f740 1 Processor -- start 2024-04-02T14:50:11.821 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:11.824+0000 7f6c15d9f740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55bc21881960 0x55bc21884ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:11.821 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:11.824+0000 7f6c15d9f740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bc21882280 0x55bc21885020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:11.821 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:11.824+0000 7f6c15d9f740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bc2188ae60 0x55bc21885560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:11.821 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:11.824+0000 7f6c15d9f740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f6c0c1b0cd0 con 0x55bc21882280 2024-04-02T14:50:11.821 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:11.824+0000 7f6c15d9f740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f6c0c1b0600 con 0x55bc21881960 2024-04-02T14:50:11.821 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:11.824+0000 7f6c15d9f740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f6c0c1ad860 con 0x55bc2188ae60 2024-04-02T14:50:11.821 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:11.824+0000 7f6c15d9f740 10 client.0 _ll_register_callbacks cb 0x7f6c04009a60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:50:11.822 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:11.824+0000 7f6c11000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55bc21881960 0x55bc21884ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:11.822 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:11.824+0000 7f6c11000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bc2188ae60 0x55bc21885560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:11.822 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:11.824+0000 7f6c0b400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bc21882280 0x55bc21885020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:11.822 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:11.824+0000 7f6c11000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bc2188ae60 0x55bc21885560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:45866/0 (socket says 172.21.15.90:45866) 2024-04-02T14:50:11.822 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:11.824+0000 7f6c11000640 1 -- 172.21.15.90:0/2638833327 learned_addr learned my addr 172.21.15.90:0/2638833327 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:50:11.822 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:11.824+0000 7f6c11000640 1 -- 172.21.15.90:0/2638833327 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55bc21881960 msgr2=0x55bc21884ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:50:11.822 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:11.824+0000 7f6c11000640 1 --2- 172.21.15.90:0/2638833327 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55bc21881960 0x55bc21884ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:11.822 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:11.824+0000 7f6c11000640 1 -- 172.21.15.90:0/2638833327 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bc21882280 msgr2=0x55bc21885020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:50:11.822 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:11.824+0000 7f6c11000640 1 --2- 172.21.15.90:0/2638833327 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bc21882280 0x55bc21885020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:11.822 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:11.824+0000 7f6c11000640 1 -- 172.21.15.90:0/2638833327 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bc21910fe0 con 0x55bc2188ae60 2024-04-02T14:50:11.822 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:11.825+0000 7f6c11000640 1 --2- 172.21.15.90:0/2638833327 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bc2188ae60 0x55bc21885560 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f6c04009560 tx=0x7f6c0402c970 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f797b91ab59822e9 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:50:11.822 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:11.825+0000 7f6bf6000640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:50:11.822 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:11.825+0000 7f6bf6000640 1 -- 172.21.15.90:0/2638833327 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c04043030 con 0x55bc2188ae60 2024-04-02T14:50:11.822 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:11.825+0000 7f6bf6000640 1 -- 172.21.15.90:0/2638833327 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 8594+0+0 (secure 0 0 0) 0x7f6c0404d040 con 0x55bc2188ae60 2024-04-02T14:50:11.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:11.825+0000 7f6c15d9f740 1 -- 172.21.15.90:0/2638833327 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6c0404c020 con 0x55bc2188ae60 2024-04-02T14:50:11.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:11.825+0000 7f6bf6000640 1 -- 172.21.15.90:0/2638833327 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f6c04065040 con 0x55bc2188ae60 2024-04-02T14:50:11.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:11.825+0000 7f6c15d9f740 10 client.29950 fetch_fsmap learned FSMap version 27 2024-04-02T14:50:11.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:11.825+0000 7f6c15d9f740 1 -- 172.21.15.90:0/2638833327 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55bc21912130 con 0x55bc2188ae60 2024-04-02T14:50:11.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:11.825+0000 7f6bf6000640 1 -- 172.21.15.90:0/2638833327 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6c0404c020 con 0x55bc2188ae60 2024-04-02T14:50:11.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:11.826+0000 7f6bf6000640 1 -- 172.21.15.90:0/2638833327 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f6c04004040 con 0x55bc2188ae60 2024-04-02T14:50:11.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:11.826+0000 7f6c15d9f740 10 client.29950 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:50:11.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:11.826+0000 7f6c15d9f740 10 client.29950 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:50:11.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:11.826+0000 7f6c15d9f740 20 client.29950 populate_metadata read hostname 'smithi090' 2024-04-02T14:50:11.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:11.826+0000 7f6c15d9f740 1 --2- 172.21.15.90:0/2638833327 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55bc21912780 0x55bc21914b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:11.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:11.826+0000 7f6c15d9f740 4 client.29950 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-02T14:50:11.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:11.826+0000 7f6c15d9f740 1 -- 172.21.15.90:0/2638833327 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55bc217f2cb0 con 0x55bc21912780 2024-04-02T14:50:11.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:11.826+0000 7f6c0b400640 1 --2- 172.21.15.90:0/2638833327 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55bc21912780 0x55bc21914b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:11.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:11.826+0000 7f6c0b400640 1 --2- 172.21.15.90:0/2638833327 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55bc21912780 0x55bc21914b70 crc :-1 s=READY pgs=385 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b0d61bdd2e28e776 server_cookie=3343d8288939ed17 in_seq=0 out_seq=0 2024-04-02T14:50:11.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:11.826+0000 7f6bf6000640 10 client.29950 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:50:11.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:11.827+0000 7f6bf6000640 1 -- 172.21.15.90:0/2638833327 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55bc217f2cb0 con 0x55bc21912780 2024-04-02T14:50:11.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:11.827+0000 7f6bf6000640 10 client.29950 handle_command_reply: tid=0 2024-04-02T14:50:11.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:11.827+0000 7f6c15d9f740 1 client.29950 shutdown 2024-04-02T14:50:11.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:11.827+0000 7f6c15d9f740 1 -- 172.21.15.90:0/2638833327 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bc2188ae60 msgr2=0x55bc21885560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:11.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:11.827+0000 7f6c15d9f740 1 --2- 172.21.15.90:0/2638833327 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bc2188ae60 0x55bc21885560 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f6c04009560 tx=0x7f6c0402c970 comp rx=0 tx=0).stop 2024-04-02T14:50:11.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:11.827+0000 7f6c15d9f740 1 -- 172.21.15.90:0/2638833327 shutdown_connections 2024-04-02T14:50:11.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:11.827+0000 7f6c15d9f740 1 --2- 172.21.15.90:0/2638833327 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55bc21912780 0x55bc21914b70 crc :-1 s=READY pgs=385 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:11.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:11.827+0000 7f6c15d9f740 1 --2- 172.21.15.90:0/2638833327 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bc2188ae60 0x55bc21885560 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:11.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:11.827+0000 7f6c15d9f740 1 --2- 172.21.15.90:0/2638833327 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55bc21882280 0x55bc21885020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:11.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:11.827+0000 7f6c15d9f740 1 --2- 172.21.15.90:0/2638833327 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55bc21881960 0x55bc21884ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:11.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:11.827+0000 7f6bf6000640 0 client.29950 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:50:11.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:11.827+0000 7f6c15d9f740 1 -- 172.21.15.90:0/2638833327 >> 172.21.15.90:0/2638833327 conn(0x55bc2187c090 msgr2=0x55bc21888740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:50:11.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:11.827+0000 7f6c15d9f740 1 -- 172.21.15.90:0/2638833327 shutdown_connections 2024-04-02T14:50:11.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:11.827+0000 7f6c15d9f740 1 -- 172.21.15.90:0/2638833327 wait complete. 2024-04-02T14:50:11.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:11.827+0000 7f6c15d9f740 20 client.29950 trim_cache size 0 max 16384 2024-04-02T14:50:11.825 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-02T14:50:11.825 INFO:teuthology.orchestra.run.smithi090.stdout: "status": "no active scrubs running", 2024-04-02T14:50:11.825 INFO:teuthology.orchestra.run.smithi090.stdout: "scrubs": {} 2024-04-02T14:50:11.825 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-02T14:50:11.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:11.828+0000 7f6c11a00640 1 -- 172.21.15.90:0/3649377175 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f6bf00a3da0 msgr2=0x7f6bf00a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:11.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:11.828+0000 7f6c11a00640 1 --2- 172.21.15.90:0/3649377175 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f6bf00a3da0 0x7f6bf00a6260 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x55bc21881430 tx=0x7f6bfc004a40 comp rx=0 tx=0).stop 2024-04-02T14:50:11.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:11.828+0000 7f6c11a00640 1 -- 172.21.15.90:0/3649377175 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6c0c103df0 msgr2=0x7f6c0c19a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:11.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:11.828+0000 7f6c11a00640 1 --2- 172.21.15.90:0/3649377175 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6c0c103df0 0x7f6c0c19a430 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f6bfc00e9f0 tx=0x7f6bfc00edf0 comp rx=0 tx=0).stop 2024-04-02T14:50:11.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:11.828+0000 7f6c11a00640 1 -- 172.21.15.90:0/3649377175 shutdown_connections 2024-04-02T14:50:11.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:11.828+0000 7f6c11a00640 1 --2- 172.21.15.90:0/3649377175 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f6bf00a3da0 0x7f6bf00a6260 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:11.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:11.828+0000 7f6c11a00640 1 --2- 172.21.15.90:0/3649377175 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6c0c19eca0 0x7f6c0c19f150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:11.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:11.828+0000 7f6c11a00640 1 --2- 172.21.15.90:0/3649377175 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6c0c108ec0 0x7f6c0c19a970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:11.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:11.828+0000 7f6c11a00640 1 --2- 172.21.15.90:0/3649377175 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6c0c103df0 0x7f6c0c19a430 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:11.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:11.828+0000 7f6c11a00640 1 -- 172.21.15.90:0/3649377175 >> 172.21.15.90:0/3649377175 conn(0x7f6c0c0ff610 msgr2=0x7f6c0c1018d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:50:11.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:11.828+0000 7f6c11a00640 1 -- 172.21.15.90:0/3649377175 shutdown_connections 2024-04-02T14:50:11.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:11.828+0000 7f6c11a00640 1 -- 172.21.15.90:0/3649377175 wait complete. 2024-04-02T14:50:11.827 INFO:teuthology.orchestra.run.smithi090.stderr:progress: 880.0 s, 446.6 tps, lat 375.347 ms stddev 1272.797 2024-04-02T14:50:11.836 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-02T14:50:11.837 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T14:50:12.102 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:50:11 smithi092 ceph-mon[36497]: pgmap v479: 97 pgs: 10 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 81 active+clean; 4.1 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 15 MiB/s wr, 1.42k op/s 2024-04-02T14:50:12.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.115+0000 7fc3dfe00640 1 -- 172.21.15.90:0/1956798897 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc3d8076950 msgr2=0x7fc3d81123f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:12.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.115+0000 7fc3dfe00640 1 --2- 172.21.15.90:0/1956798897 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc3d8076950 0x7fc3d81123f0 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7fc3c4009930 tx=0x7fc3c402bc30 comp rx=0 tx=0).stop 2024-04-02T14:50:12.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.116+0000 7fc3dfe00640 1 -- 172.21.15.90:0/1956798897 shutdown_connections 2024-04-02T14:50:12.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.116+0000 7fc3dfe00640 1 --2- 172.21.15.90:0/1956798897 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc3d8112930 0x7fc3d8114d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:12.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.116+0000 7fc3dfe00640 1 --2- 172.21.15.90:0/1956798897 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc3d8076950 0x7fc3d81123f0 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:12.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.116+0000 7fc3dfe00640 1 --2- 172.21.15.90:0/1956798897 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc3d8076030 0x7fc3d8076410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:12.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.116+0000 7fc3dfe00640 1 -- 172.21.15.90:0/1956798897 >> 172.21.15.90:0/1956798897 conn(0x7fc3d80ff760 msgr2=0x7fc3d8101b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:50:12.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.117+0000 7fc3dfe00640 1 -- 172.21.15.90:0/1956798897 shutdown_connections 2024-04-02T14:50:12.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.117+0000 7fc3dfe00640 1 -- 172.21.15.90:0/1956798897 wait complete. 2024-04-02T14:50:12.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.119+0000 7fc3dfe00640 1 Processor -- start 2024-04-02T14:50:12.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.119+0000 7fc3dfe00640 1 -- start start 2024-04-02T14:50:12.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.120+0000 7fc3dfe00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc3d8076030 0x7fc3d8111e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:12.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.120+0000 7fc3dfe00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc3d8076950 0x7fc3d81123b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:12.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.120+0000 7fc3dfe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc3d8112930 0x7fc3d810c110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:12.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.120+0000 7fc3dfe00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fc3d8104300 con 0x7fc3d8112930 2024-04-02T14:50:12.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.120+0000 7fc3dfe00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fc3d8104180 con 0x7fc3d8076030 2024-04-02T14:50:12.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.120+0000 7fc3dfe00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fc3d8104480 con 0x7fc3d8076950 2024-04-02T14:50:12.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.120+0000 7fc3dd400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc3d8076030 0x7fc3d8111e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:12.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.120+0000 7fc3dd400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc3d8076030 0x7fc3d8111e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:47590/0 (socket says 172.21.15.90:47590) 2024-04-02T14:50:12.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.120+0000 7fc3dd400640 1 -- 172.21.15.90:0/988048918 learned_addr learned my addr 172.21.15.90:0/988048918 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:50:12.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.120+0000 7fc3dca00640 1 --2- 172.21.15.90:0/988048918 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc3d8076950 0x7fc3d81123b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:12.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.122+0000 7fc3dde00640 1 --2- 172.21.15.90:0/988048918 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc3d8112930 0x7fc3d810c110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:12.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.123+0000 7fc3dca00640 1 -- 172.21.15.90:0/988048918 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc3d8076030 msgr2=0x7fc3d8111e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:12.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.123+0000 7fc3dca00640 1 --2- 172.21.15.90:0/988048918 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc3d8076030 0x7fc3d8111e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:12.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.123+0000 7fc3dca00640 1 -- 172.21.15.90:0/988048918 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc3d8112930 msgr2=0x7fc3d810c110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:12.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.123+0000 7fc3dca00640 1 --2- 172.21.15.90:0/988048918 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc3d8112930 0x7fc3d810c110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:12.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.123+0000 7fc3dca00640 1 -- 172.21.15.90:0/988048918 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3d810c9d0 con 0x7fc3d8076950 2024-04-02T14:50:12.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.123+0000 7fc3dca00640 1 --2- 172.21.15.90:0/988048918 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc3d8076950 0x7fc3d81123b0 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7fc3c4036040 tx=0x7fc3c4004430 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:50:12.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.124+0000 7fc3d6000640 1 -- 172.21.15.90:0/988048918 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3c4043020 con 0x7fc3d8076950 2024-04-02T14:50:12.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.124+0000 7fc3d6000640 1 -- 172.21.15.90:0/988048918 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc3c40054e0 con 0x7fc3d8076950 2024-04-02T14:50:12.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.124+0000 7fc3dfe00640 1 -- 172.21.15.90:0/988048918 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc3d810cc60 con 0x7fc3d8076950 2024-04-02T14:50:12.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.124+0000 7fc3d6000640 1 -- 172.21.15.90:0/988048918 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3c4003c90 con 0x7fc3d8076950 2024-04-02T14:50:12.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.124+0000 7fc3dfe00640 1 -- 172.21.15.90:0/988048918 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3d81b3dc0 con 0x7fc3d8076950 2024-04-02T14:50:12.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.126+0000 7fc3dfe00640 1 -- 172.21.15.90:0/988048918 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc3d8077770 con 0x7fc3d8076950 2024-04-02T14:50:12.120 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.128+0000 7fc3d6000640 1 -- 172.21.15.90:0/988048918 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fc3c4046020 con 0x7fc3d8076950 2024-04-02T14:50:12.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.131+0000 7fc3d6000640 1 --2- 172.21.15.90:0/988048918 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fc3c00a3da0 0x7fc3c00a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:12.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.131+0000 7fc3d6000640 1 -- 172.21.15.90:0/988048918 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 8594+0+0 (secure 0 0 0) 0x7fc3c40f27a0 con 0x7fc3d8076950 2024-04-02T14:50:12.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.134+0000 7fc3dd400640 1 --2- 172.21.15.90:0/988048918 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fc3c00a3da0 0x7fc3c00a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:12.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.135+0000 7fc3dd400640 1 --2- 172.21.15.90:0/988048918 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fc3c00a3da0 0x7fc3c00a6260 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fc3cc009f30 tx=0x7fc3cc0094c0 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:50:12.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.136+0000 7fc3d6000640 1 -- 172.21.15.90:0/988048918 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7fc3c404d020 con 0x7fc3d8076950 2024-04-02T14:50:12.178 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:50:11 smithi177 ceph-mon[36431]: pgmap v479: 97 pgs: 10 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 81 active+clean; 4.1 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 15 MiB/s wr, 1.42k op/s 2024-04-02T14:50:12.182 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:50:11 smithi090 ceph-mon[31674]: pgmap v479: 97 pgs: 10 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 81 active+clean; 4.1 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 15 MiB/s wr, 1.42k op/s 2024-04-02T14:50:12.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.526+0000 7fc3dfe00640 1 -- 172.21.15.90:0/988048918 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc3d81061d0 con 0x7fc3d8076950 2024-04-02T14:50:12.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.528+0000 7fc3d6000640 1 -- 172.21.15.90:0/988048918 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11927 (secure 0 0 0) 0x7fc3c40bef00 con 0x7fc3d8076950 2024-04-02T14:50:12.522 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-02T14:50:12.522 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14412,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6837/2852160309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6836","nonce":2852160309},{"type":"v1","addr":"172.21.15.90:6837","nonce":2852160309}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14478,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6841/1919376858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6840","nonce":1919376858},{"type":"v1","addr":"172.21.15.90:6841","nonce":1919376858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-02T14:32:22.319083+0000","modified":"2024-04-02T14:34:52.266152+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24590,"mds_1":24560,"mds_2":24547,"mds_3":24539,"mds_4":24523},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24454":{"gid":24454,"name":"l","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6833/1564319431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":1564319431},{"type":"v1","addr":"172.21.15.177:6833","nonce":1564319431}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.92:6833/1570337372","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":1570337372},{"type":"v1","addr":"172.21.15.92:6833","nonce":1570337372}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"j","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.90:6835/1277124891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6834","nonce":1277124891},{"type":"v1","addr":"172.21.15.90:6835","nonce":1277124891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.92:6835/860721164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":860721164},{"type":"v1","addr":"172.21.15.92:6835","nonce":860721164}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24511":{"gid":24511,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6835/3177244988","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":3177244988},{"type":"v1","addr":"172.21.15.177:6835","nonce":3177244988}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"c","rank":4,"incarnation":25,"state":"up:active","state_seq":9,"addr":"172.21.15.177:6837/4223414858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":4223414858},{"type":"v1","addr":"172.21.15.177:6837","nonce":4223414858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24539":{"gid":24539,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":8,"addr":"172.21.15.92:6837/1270635939","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":1270635939},{"type":"v1","addr":"172.21.15.92:6837","nonce":1270635939}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24547":{"gid":24547,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":7,"addr":"172.21.15.177:6839/664895663","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":664895663},{"type":"v1","addr":"172.21.15.177:6839","nonce":664895663}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"g","rank":1,"incarnation":19,"state":"up:active","state_seq":5,"addr":"172.21.15.90:6839/1498915870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6838","nonce":1498915870},{"type":"v1","addr":"172.21.15.90:6839","nonce":1498915870}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24590":{"gid":24590,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.92:6839/1054888219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":1054888219},{"type":"v1","addr":"172.21.15.92:6839","nonce":1054888219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24590,"qdb_cluster":[24547,24590,24523,24539,24560]},"id":1}]} 2024-04-02T14:50:12.523 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 27 2024-04-02T14:50:12.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.532+0000 7fc3dfe00640 1 -- 172.21.15.90:0/988048918 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fc3c00a3da0 msgr2=0x7fc3c00a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:12.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.532+0000 7fc3dfe00640 1 --2- 172.21.15.90:0/988048918 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fc3c00a3da0 0x7fc3c00a6260 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fc3cc009f30 tx=0x7fc3cc0094c0 comp rx=0 tx=0).stop 2024-04-02T14:50:12.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.532+0000 7fc3dfe00640 1 -- 172.21.15.90:0/988048918 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc3d8076950 msgr2=0x7fc3d81123b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:12.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.532+0000 7fc3dfe00640 1 --2- 172.21.15.90:0/988048918 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc3d8076950 0x7fc3d81123b0 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7fc3c4036040 tx=0x7fc3c4004430 comp rx=0 tx=0).stop 2024-04-02T14:50:12.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.532+0000 7fc3dfe00640 1 -- 172.21.15.90:0/988048918 shutdown_connections 2024-04-02T14:50:12.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.532+0000 7fc3dfe00640 1 --2- 172.21.15.90:0/988048918 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fc3c00a3da0 0x7fc3c00a6260 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:12.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.532+0000 7fc3dfe00640 1 --2- 172.21.15.90:0/988048918 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc3d8112930 0x7fc3d810c110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:12.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.532+0000 7fc3dfe00640 1 --2- 172.21.15.90:0/988048918 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc3d8076950 0x7fc3d81123b0 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:12.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.532+0000 7fc3dfe00640 1 --2- 172.21.15.90:0/988048918 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc3d8076030 0x7fc3d8111e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:12.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.532+0000 7fc3dfe00640 1 -- 172.21.15.90:0/988048918 >> 172.21.15.90:0/988048918 conn(0x7fc3d80ff760 msgr2=0x7fc3d8113fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:50:12.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.533+0000 7fc3dfe00640 1 -- 172.21.15.90:0/988048918 shutdown_connections 2024-04-02T14:50:12.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.533+0000 7fc3dfe00640 1 -- 172.21.15.90:0/988048918 wait complete. 2024-04-02T14:50:12.555 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-04-02T14:50:12.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.806+0000 7f32d7800640 1 -- 172.21.15.90:0/3379423138 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f32d0107ab0 msgr2=0x7f32d0107e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:12.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.806+0000 7f32d7800640 1 --2- 172.21.15.90:0/3379423138 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f32d0107ab0 0x7f32d0107e90 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f32c4009930 tx=0x7f32c402bc30 comp rx=0 tx=0).stop 2024-04-02T14:50:12.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.806+0000 7f32d7800640 1 -- 172.21.15.90:0/3379423138 shutdown_connections 2024-04-02T14:50:12.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.807+0000 7f32d7800640 1 --2- 172.21.15.90:0/3379423138 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f32d0102460 0x7f32d010e970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:12.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.807+0000 7f32d7800640 1 --2- 172.21.15.90:0/3379423138 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f32d0101aa0 0x7f32d0101f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:12.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.807+0000 7f32d7800640 1 --2- 172.21.15.90:0/3379423138 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f32d0107ab0 0x7f32d0107e90 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:12.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.807+0000 7f32d7800640 1 -- 172.21.15.90:0/3379423138 >> 172.21.15.90:0/3379423138 conn(0x7f32d00fd590 msgr2=0x7f32d00ff9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:50:12.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.807+0000 7f32d7800640 1 -- 172.21.15.90:0/3379423138 shutdown_connections 2024-04-02T14:50:12.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.807+0000 7f32d7800640 1 -- 172.21.15.90:0/3379423138 wait complete. 2024-04-02T14:50:12.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.808+0000 7f32d7800640 1 Processor -- start 2024-04-02T14:50:12.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.809+0000 7f32d7800640 1 -- start start 2024-04-02T14:50:12.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.809+0000 7f32d7800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f32d0101aa0 0x7f32d0198400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:12.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.809+0000 7f32d7800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f32d0102460 0x7f32d0198940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:12.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.809+0000 7f32d7800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f32d0107ab0 0x7f32d019ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:12.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.809+0000 7f32cfe00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f32d0102460 0x7f32d0198940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:12.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.809+0000 7f32cfe00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f32d0102460 0x7f32d0198940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:47598/0 (socket says 172.21.15.90:47598) 2024-04-02T14:50:12.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.809+0000 7f32d4e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f32d0101aa0 0x7f32d0198400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:12.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.810+0000 7f32d7800640 1 -- 172.21.15.90:0/3768885936 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f32d0111ee0 con 0x7f32d0107ab0 2024-04-02T14:50:12.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.810+0000 7f32d7800640 1 -- 172.21.15.90:0/3768885936 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f32d0111d60 con 0x7f32d0102460 2024-04-02T14:50:12.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.810+0000 7f32d7800640 1 -- 172.21.15.90:0/3768885936 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f32d0112060 con 0x7f32d0101aa0 2024-04-02T14:50:12.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.810+0000 7f32d5800640 1 --2- 172.21.15.90:0/3768885936 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f32d0107ab0 0x7f32d019ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:12.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.809+0000 7f32cfe00640 1 -- 172.21.15.90:0/3768885936 learned_addr learned my addr 172.21.15.90:0/3768885936 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:50:12.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.810+0000 7f32d4e00640 1 -- 172.21.15.90:0/3768885936 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f32d0102460 msgr2=0x7f32d0198940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:12.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.810+0000 7f32d4e00640 1 --2- 172.21.15.90:0/3768885936 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f32d0102460 0x7f32d0198940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:12.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.810+0000 7f32d4e00640 1 -- 172.21.15.90:0/3768885936 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f32d0107ab0 msgr2=0x7f32d019ccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:12.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.810+0000 7f32d4e00640 1 --2- 172.21.15.90:0/3768885936 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f32d0107ab0 0x7f32d019ccc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:12.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.810+0000 7f32d4e00640 1 -- 172.21.15.90:0/3768885936 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32d019d3f0 con 0x7f32d0101aa0 2024-04-02T14:50:12.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.810+0000 7f32cfe00640 1 --2- 172.21.15.90:0/3768885936 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f32d0102460 0x7f32d0198940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:50:12.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.811+0000 7f32d4e00640 1 --2- 172.21.15.90:0/3768885936 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f32d0101aa0 0x7f32d0198400 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f32c4036040 tx=0x7f32c40043b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:50:12.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.811+0000 7f32cd600640 1 -- 172.21.15.90:0/3768885936 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32c4043020 con 0x7f32d0101aa0 2024-04-02T14:50:12.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.811+0000 7f32cd600640 1 -- 172.21.15.90:0/3768885936 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f32c40054e0 con 0x7f32d0101aa0 2024-04-02T14:50:12.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.811+0000 7f32d7800640 1 -- 172.21.15.90:0/3768885936 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f32d019d680 con 0x7f32d0101aa0 2024-04-02T14:50:12.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.811+0000 7f32cd600640 1 -- 172.21.15.90:0/3768885936 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32c4003bd0 con 0x7f32d0101aa0 2024-04-02T14:50:12.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.811+0000 7f32d7800640 1 -- 172.21.15.90:0/3768885936 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32d0073d30 con 0x7f32d0101aa0 2024-04-02T14:50:12.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.816+0000 7f32cd600640 1 -- 172.21.15.90:0/3768885936 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f32c4046020 con 0x7f32d0101aa0 2024-04-02T14:50:12.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.817+0000 7f32da5bc740 1 Processor -- start 2024-04-02T14:50:12.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.817+0000 7f32da5bc740 1 -- start start 2024-04-02T14:50:12.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.817+0000 7f32da5bc740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555a429569f0 0x555a42955740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:12.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.817+0000 7f32da5bc740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555a42955d10 0x555a42956190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:12.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.817+0000 7f32da5bc740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555a4295f080 0x555a42961470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:12.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.817+0000 7f32da5bc740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x555a42896400 con 0x555a4295f080 2024-04-02T14:50:12.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.817+0000 7f32da5bc740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x555a4287af10 con 0x555a42955d10 2024-04-02T14:50:12.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.817+0000 7f32da5bc740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x555a428ba6f0 con 0x555a429569f0 2024-04-02T14:50:12.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.817+0000 7f32d5800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555a429569f0 0x555a42955740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:12.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.817+0000 7f32d5800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555a429569f0 0x555a42955740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:45926/0 (socket says 172.21.15.90:45926) 2024-04-02T14:50:12.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.817+0000 7f32d5800640 1 -- 172.21.15.90:0/3340545976 learned_addr learned my addr 172.21.15.90:0/3340545976 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:50:12.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.818+0000 7f32d4e00640 1 --2- 172.21.15.90:0/3340545976 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555a42955d10 0x555a42956190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:12.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.818+0000 7f32cfe00640 1 --2- 172.21.15.90:0/3340545976 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555a4295f080 0x555a42961470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:12.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.818+0000 7f32d5800640 1 -- 172.21.15.90:0/3340545976 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555a42955d10 msgr2=0x555a42956190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:50:12.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.818+0000 7f32d5800640 1 --2- 172.21.15.90:0/3340545976 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555a42955d10 0x555a42956190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:12.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.818+0000 7f32d5800640 1 -- 172.21.15.90:0/3340545976 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555a4295f080 msgr2=0x555a42961470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:50:12.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.818+0000 7f32d5800640 1 --2- 172.21.15.90:0/3340545976 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555a4295f080 0x555a42961470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:12.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.818+0000 7f32d5800640 1 -- 172.21.15.90:0/3340545976 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555a425ce3f0 con 0x555a429569f0 2024-04-02T14:50:12.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.818+0000 7f32d4e00640 1 --2- 172.21.15.90:0/3340545976 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555a42955d10 0x555a42956190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:50:12.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.818+0000 7f32d5800640 1 --2- 172.21.15.90:0/3340545976 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555a429569f0 0x555a42955740 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f32c8009f90 tx=0x7f32c8010860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1051fb5a2b520868 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:50:12.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.818+0000 7f32cd600640 1 --2- 172.21.15.90:0/3768885936 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f32b80a3de0 0x7f32b80a62a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:12.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.818+0000 7f32c2000640 1 -- 172.21.15.90:0/3340545976 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32c8011700 con 0x555a429569f0 2024-04-02T14:50:12.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.819+0000 7f32c2000640 1 -- 172.21.15.90:0/3340545976 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f32c8011d80 con 0x555a429569f0 2024-04-02T14:50:12.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.819+0000 7f32c2000640 1 -- 172.21.15.90:0/3340545976 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32c8018380 con 0x555a429569f0 2024-04-02T14:50:12.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.819+0000 7f32cd600640 1 -- 172.21.15.90:0/3768885936 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 8594+0+0 (secure 0 0 0) 0x7f32c40f27e0 con 0x7f32d0101aa0 2024-04-02T14:50:12.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.819+0000 7f32d4e00640 1 --2- 172.21.15.90:0/3768885936 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f32b80a3de0 0x7f32b80a62a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:12.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.819+0000 7f32da5bc740 1 -- 172.21.15.90:0/3340545976 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555a429569f0 msgr2=0x555a42955740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:12.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.819+0000 7f32da5bc740 1 --2- 172.21.15.90:0/3340545976 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555a429569f0 0x555a42955740 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f32c8009f90 tx=0x7f32c8010860 comp rx=0 tx=0).stop 2024-04-02T14:50:12.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.820+0000 7f32da5bc740 1 -- 172.21.15.90:0/3340545976 shutdown_connections 2024-04-02T14:50:12.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.820+0000 7f32da5bc740 1 --2- 172.21.15.90:0/3340545976 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555a4295f080 0x555a42961470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:12.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.820+0000 7f32da5bc740 1 --2- 172.21.15.90:0/3340545976 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555a42955d10 0x555a42956190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:12.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.820+0000 7f32da5bc740 1 --2- 172.21.15.90:0/3340545976 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555a429569f0 0x555a42955740 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:12.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.820+0000 7f32da5bc740 1 -- 172.21.15.90:0/3340545976 >> 172.21.15.90:0/3340545976 conn(0x555a4291ca50 msgr2=0x555a42950760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:50:12.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.820+0000 7f32da5bc740 1 -- 172.21.15.90:0/3340545976 shutdown_connections 2024-04-02T14:50:12.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.820+0000 7f32da5bc740 1 -- 172.21.15.90:0/3340545976 wait complete. 2024-04-02T14:50:12.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.821+0000 7f32da5bc740 1 -- start start 2024-04-02T14:50:12.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.821+0000 7f32d4e00640 1 --2- 172.21.15.90:0/3768885936 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f32b80a3de0 0x7f32b80a62a0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f32c40f00a0 tx=0x7f32c4000bc0 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:50:12.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.821+0000 7f32da5bc740 1 Processor -- start 2024-04-02T14:50:12.815 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.822+0000 7f32da5bc740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555a42955d10 0x555a4295c0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:12.815 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.822+0000 7f32da5bc740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555a429569f0 0x555a4295c610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:12.815 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.822+0000 7f32da5bc740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555a4295f080 0x555a4295cb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:12.815 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.822+0000 7f32da5bc740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x555a428b66a0 con 0x555a4295f080 2024-04-02T14:50:12.815 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.822+0000 7f32da5bc740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x555a42862980 con 0x555a429569f0 2024-04-02T14:50:12.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.822+0000 7f32da5bc740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x555a42875cb0 con 0x555a42955d10 2024-04-02T14:50:12.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.822+0000 7f32da5bc740 10 client.0 _ll_register_callbacks cb 0x555a428756a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:50:12.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.823+0000 7f32d5800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555a42955d10 0x555a4295c0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:12.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.823+0000 7f32d5800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555a42955d10 0x555a4295c0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:45938/0 (socket says 172.21.15.90:45938) 2024-04-02T14:50:12.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.823+0000 7f32d5800640 1 -- 172.21.15.90:0/2654259230 learned_addr learned my addr 172.21.15.90:0/2654259230 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:50:12.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.823+0000 7f32cfe00640 1 --2- 172.21.15.90:0/2654259230 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555a429569f0 0x555a4295c610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:12.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.823+0000 7f32d5800640 1 -- 172.21.15.90:0/2654259230 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555a429569f0 msgr2=0x555a4295c610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:50:12.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.823+0000 7f32d5800640 1 --2- 172.21.15.90:0/2654259230 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555a429569f0 0x555a4295c610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:12.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.823+0000 7f32d5800640 1 -- 172.21.15.90:0/2654259230 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555a4295f080 msgr2=0x555a4295cb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:50:12.817 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.823+0000 7f32d5800640 1 --2- 172.21.15.90:0/2654259230 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555a4295f080 0x555a4295cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:12.817 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.823+0000 7f32d5800640 1 -- 172.21.15.90:0/2654259230 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555a426987d0 con 0x555a42955d10 2024-04-02T14:50:12.817 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.823+0000 7f32cfe00640 1 --2- 172.21.15.90:0/2654259230 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555a429569f0 0x555a4295c610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:50:12.817 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.824+0000 7f32d5800640 1 --2- 172.21.15.90:0/2654259230 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555a42955d10 0x555a4295c0d0 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f32c8011fd0 tx=0x7f32c8009ce0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1d365b1cc2568d00 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:50:12.817 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.824+0000 7f32b6a00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:50:12.818 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.825+0000 7f32b6a00640 1 -- 172.21.15.90:0/2654259230 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32c8019490 con 0x555a42955d10 2024-04-02T14:50:12.818 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.825+0000 7f32b6a00640 1 -- 172.21.15.90:0/2654259230 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 8594+0+0 (secure 0 0 0) 0x7f32c8003970 con 0x555a42955d10 2024-04-02T14:50:12.818 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.825+0000 7f32da5bc740 1 -- 172.21.15.90:0/2654259230 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f32c8011d80 con 0x555a42955d10 2024-04-02T14:50:12.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.827+0000 7f32b6a00640 1 -- 172.21.15.90:0/2654259230 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f32c8003b70 con 0x555a42955d10 2024-04-02T14:50:12.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.827+0000 7f32da5bc740 10 client.29968 fetch_fsmap learned FSMap version 27 2024-04-02T14:50:12.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.827+0000 7f32da5bc740 1 -- 172.21.15.90:0/2654259230 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x555a42976710 con 0x555a42955d10 2024-04-02T14:50:12.820 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.827+0000 7f32b6a00640 1 -- 172.21.15.90:0/2654259230 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f32c8011d80 con 0x555a42955d10 2024-04-02T14:50:12.821 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.828+0000 7f32b6a00640 1 -- 172.21.15.90:0/2654259230 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f32c801a790 con 0x555a42955d10 2024-04-02T14:50:12.821 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.828+0000 7f32da5bc740 10 client.29968 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:50:12.821 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.828+0000 7f32da5bc740 10 client.29968 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.c 2024-04-02T14:50:12.821 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.828+0000 7f32da5bc740 20 client.29968 populate_metadata read hostname 'smithi090' 2024-04-02T14:50:12.821 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.828+0000 7f32da5bc740 1 --2- 172.21.15.90:0/2654259230 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x555a42977b10 0x555a42979f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:12.821 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.828+0000 7f32da5bc740 4 client.29968 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:50:12.821 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.828+0000 7f32da5bc740 1 -- 172.21.15.90:0/2654259230 --> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555a4287a040 con 0x555a42977b10 2024-04-02T14:50:12.821 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.829+0000 7f32cfe00640 1 --2- 172.21.15.90:0/2654259230 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x555a42977b10 0x555a42979f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:12.822 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.829+0000 7f32cfe00640 1 --2- 172.21.15.90:0/2654259230 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x555a42977b10 0x555a42979f00 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=62c391b6a01a2fce server_cookie=86fce6726f098a9b in_seq=0 out_seq=0 2024-04-02T14:50:12.822 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.829+0000 7f32b6a00640 10 client.29968 ms_handle_connect on v2:172.21.15.177:6836/4223414858 2024-04-02T14:50:12.822 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.830+0000 7f32b6a00640 1 -- 172.21.15.90:0/2654259230 <== mds.4 v2:172.21.15.177:6836/4223414858 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x555a4287a040 con 0x555a42977b10 2024-04-02T14:50:12.822 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.830+0000 7f32b6a00640 10 client.29968 handle_command_reply: tid=0 2024-04-02T14:50:12.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.830+0000 7f32da5bc740 1 client.29968 shutdown 2024-04-02T14:50:12.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.830+0000 7f32da5bc740 1 -- 172.21.15.90:0/2654259230 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555a42955d10 msgr2=0x555a4295c0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:12.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.830+0000 7f32da5bc740 1 --2- 172.21.15.90:0/2654259230 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555a42955d10 0x555a4295c0d0 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f32c8011fd0 tx=0x7f32c8009ce0 comp rx=0 tx=0).stop 2024-04-02T14:50:12.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.830+0000 7f32da5bc740 1 -- 172.21.15.90:0/2654259230 shutdown_connections 2024-04-02T14:50:12.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.830+0000 7f32da5bc740 1 --2- 172.21.15.90:0/2654259230 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x555a42977b10 0x555a42979f00 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:12.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.830+0000 7f32da5bc740 1 --2- 172.21.15.90:0/2654259230 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555a4295f080 0x555a4295cb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:12.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.830+0000 7f32da5bc740 1 --2- 172.21.15.90:0/2654259230 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555a429569f0 0x555a4295c610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:12.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.830+0000 7f32b6a00640 0 client.29968 ms_handle_reset on v2:172.21.15.177:6836/4223414858 2024-04-02T14:50:12.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.830+0000 7f32da5bc740 1 --2- 172.21.15.90:0/2654259230 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555a42955d10 0x555a4295c0d0 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:12.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.830+0000 7f32da5bc740 1 -- 172.21.15.90:0/2654259230 >> 172.21.15.90:0/2654259230 conn(0x555a4291ca50 msgr2=0x555a429524b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:50:12.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.830+0000 7f32da5bc740 1 -- 172.21.15.90:0/2654259230 shutdown_connections 2024-04-02T14:50:12.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.831+0000 7f32da5bc740 1 -- 172.21.15.90:0/2654259230 wait complete. 2024-04-02T14:50:12.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.831+0000 7f32da5bc740 20 client.29968 trim_cache size 0 max 16384 2024-04-02T14:50:12.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.855+0000 7f32da5bc740 1 Processor -- start 2024-04-02T14:50:12.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.855+0000 7f32da5bc740 1 -- start start 2024-04-02T14:50:12.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.855+0000 7f32da5bc740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555a42955d10 0x555a4295e800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:12.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.855+0000 7f32da5bc740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555a429569f0 0x555a42950900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:12.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.855+0000 7f32da5bc740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555a4295f080 0x555a42972fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:12.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.855+0000 7f32da5bc740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f32d01ab8d0 con 0x555a42955d10 2024-04-02T14:50:12.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.855+0000 7f32da5bc740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f32d01ad410 con 0x555a429569f0 2024-04-02T14:50:12.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.855+0000 7f32da5bc740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f32d01abfa0 con 0x555a4295f080 2024-04-02T14:50:12.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.856+0000 7f32cfe00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555a429569f0 0x555a42950900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:12.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.856+0000 7f32cfe00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555a429569f0 0x555a42950900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:47620/0 (socket says 172.21.15.90:47620) 2024-04-02T14:50:12.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.856+0000 7f32cfe00640 1 -- 172.21.15.90:0/2447398793 learned_addr learned my addr 172.21.15.90:0/2447398793 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:50:12.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.856+0000 7f32d5800640 1 --2- 172.21.15.90:0/2447398793 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555a42955d10 0x555a4295e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:12.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.856+0000 7f32d5800640 1 --2- 172.21.15.90:0/2447398793 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555a4295f080 0x555a42972fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:12.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.856+0000 7f32cfe00640 1 -- 172.21.15.90:0/2447398793 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555a4295f080 msgr2=0x555a42972fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:50:12.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.856+0000 7f32cfe00640 1 --2- 172.21.15.90:0/2447398793 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555a4295f080 0x555a42972fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:12.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.856+0000 7f32cfe00640 1 -- 172.21.15.90:0/2447398793 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555a42955d10 msgr2=0x555a4295e800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:50:12.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.856+0000 7f32cfe00640 1 --2- 172.21.15.90:0/2447398793 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555a42955d10 0x555a4295e800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:12.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.856+0000 7f32cfe00640 1 -- 172.21.15.90:0/2447398793 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555a42954d40 con 0x555a429569f0 2024-04-02T14:50:12.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.856+0000 7f32cfe00640 1 --2- 172.21.15.90:0/2447398793 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555a429569f0 0x555a42950900 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7f32bc011010 tx=0x7f32bc00f890 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5bff71904ef52a3c server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:50:12.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.857+0000 7f32c1600640 1 -- 172.21.15.90:0/2447398793 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32bc01d2f0 con 0x555a429569f0 2024-04-02T14:50:12.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.857+0000 7f32c1600640 1 -- 172.21.15.90:0/2447398793 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f32bc0035f0 con 0x555a429569f0 2024-04-02T14:50:12.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.857+0000 7f32c1600640 1 -- 172.21.15.90:0/2447398793 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32bc01b650 con 0x555a429569f0 2024-04-02T14:50:12.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.857+0000 7f32da5bc740 1 -- 172.21.15.90:0/2447398793 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555a429569f0 msgr2=0x555a42950900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:12.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.857+0000 7f32da5bc740 1 --2- 172.21.15.90:0/2447398793 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555a429569f0 0x555a42950900 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7f32bc011010 tx=0x7f32bc00f890 comp rx=0 tx=0).stop 2024-04-02T14:50:12.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.857+0000 7f32da5bc740 1 -- 172.21.15.90:0/2447398793 shutdown_connections 2024-04-02T14:50:12.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.857+0000 7f32da5bc740 1 --2- 172.21.15.90:0/2447398793 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555a4295f080 0x555a42972fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:12.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.857+0000 7f32da5bc740 1 --2- 172.21.15.90:0/2447398793 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555a429569f0 0x555a42950900 unknown :-1 s=CLOSED pgs=1016 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:12.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.857+0000 7f32da5bc740 1 --2- 172.21.15.90:0/2447398793 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555a42955d10 0x555a4295e800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:12.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.857+0000 7f32da5bc740 1 -- 172.21.15.90:0/2447398793 >> 172.21.15.90:0/2447398793 conn(0x555a4291ca50 msgr2=0x555a42972890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:50:12.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.858+0000 7f32da5bc740 1 -- 172.21.15.90:0/2447398793 shutdown_connections 2024-04-02T14:50:12.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.858+0000 7f32da5bc740 1 -- 172.21.15.90:0/2447398793 wait complete. 2024-04-02T14:50:12.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.858+0000 7f32da5bc740 1 -- start start 2024-04-02T14:50:12.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.858+0000 7f32da5bc740 1 Processor -- start 2024-04-02T14:50:12.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.858+0000 7f32da5bc740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555a42955d10 0x555a4297d400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:12.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.858+0000 7f32da5bc740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555a429569f0 0x555a42958c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:12.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.858+0000 7f32da5bc740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555a4295f080 0x555a42959230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:12.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.859+0000 7f32da5bc740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f32d01acd40 con 0x555a42955d10 2024-04-02T14:50:12.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.859+0000 7f32da5bc740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f32d01ac670 con 0x555a429569f0 2024-04-02T14:50:12.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.859+0000 7f32da5bc740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f32d01a98d0 con 0x555a4295f080 2024-04-02T14:50:12.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.859+0000 7f32da5bc740 10 client.0 _ll_register_callbacks cb 0x7f32c8009bf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:50:12.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.859+0000 7f32cfe00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555a429569f0 0x555a42958c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:12.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.859+0000 7f32cfe00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555a429569f0 0x555a42958c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:47630/0 (socket says 172.21.15.90:47630) 2024-04-02T14:50:12.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.859+0000 7f32cfe00640 1 -- 172.21.15.90:0/4275357553 learned_addr learned my addr 172.21.15.90:0/4275357553 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:50:12.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.859+0000 7f32d5800640 1 --2- 172.21.15.90:0/4275357553 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555a42955d10 0x555a4297d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:12.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.859+0000 7f32d5800640 1 --2- 172.21.15.90:0/4275357553 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555a4295f080 0x555a42959230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:12.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.859+0000 7f32cfe00640 1 -- 172.21.15.90:0/4275357553 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555a4295f080 msgr2=0x555a42959230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:50:12.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.859+0000 7f32cfe00640 1 --2- 172.21.15.90:0/4275357553 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555a4295f080 0x555a42959230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:12.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.859+0000 7f32cfe00640 1 -- 172.21.15.90:0/4275357553 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555a42955d10 msgr2=0x555a4297d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:50:12.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.859+0000 7f32cfe00640 1 --2- 172.21.15.90:0/4275357553 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555a42955d10 0x555a4297d400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:12.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.859+0000 7f32cfe00640 1 -- 172.21.15.90:0/4275357553 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555a42959a30 con 0x555a429569f0 2024-04-02T14:50:12.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.859+0000 7f32d5800640 1 --2- 172.21.15.90:0/4275357553 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555a4295f080 0x555a42959230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:50:12.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.859+0000 7f32cfe00640 1 --2- 172.21.15.90:0/4275357553 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555a429569f0 0x555a42958c20 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x555a42950e40 tx=0x7f32bc004580 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e9db497d5001f725 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:50:12.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.859+0000 7f32b6a00640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:50:12.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.860+0000 7f32b6a00640 1 -- 172.21.15.90:0/4275357553 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32bc004020 con 0x555a429569f0 2024-04-02T14:50:12.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.860+0000 7f32b6a00640 1 -- 172.21.15.90:0/4275357553 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 8594+0+0 (secure 0 0 0) 0x7f32bc0037a0 con 0x555a429569f0 2024-04-02T14:50:12.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.860+0000 7f32da5bc740 1 -- 172.21.15.90:0/4275357553 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f32bc0035f0 con 0x555a429569f0 2024-04-02T14:50:12.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.860+0000 7f32b6a00640 1 -- 172.21.15.90:0/4275357553 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f32bc0039a0 con 0x555a429569f0 2024-04-02T14:50:12.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.860+0000 7f32da5bc740 10 client.30182 fetch_fsmap learned FSMap version 27 2024-04-02T14:50:12.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.860+0000 7f32da5bc740 1 -- 172.21.15.90:0/4275357553 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x555a429ddde0 con 0x555a429569f0 2024-04-02T14:50:12.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.860+0000 7f32b6a00640 1 -- 172.21.15.90:0/4275357553 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f32bc0035f0 con 0x555a429569f0 2024-04-02T14:50:12.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.861+0000 7f32b6a00640 1 -- 172.21.15.90:0/4275357553 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f32bc031670 con 0x555a429569f0 2024-04-02T14:50:12.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.861+0000 7f32da5bc740 10 client.30182 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:50:12.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.861+0000 7f32da5bc740 10 client.30182 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.c 2024-04-02T14:50:12.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.861+0000 7f32da5bc740 20 client.30182 populate_metadata read hostname 'smithi090' 2024-04-02T14:50:12.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.861+0000 7f32da5bc740 1 --2- 172.21.15.90:0/4275357553 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x555a429de480 0x555a429e08a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:12.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.861+0000 7f32da5bc740 4 client.30182 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-02T14:50:12.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.861+0000 7f32da5bc740 1 -- 172.21.15.90:0/4275357553 --> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] -- command(tid 0: {"prefix": "damage ls"}) -- 0x555a428c7c30 con 0x555a429de480 2024-04-02T14:50:12.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.861+0000 7f32d5800640 1 --2- 172.21.15.90:0/4275357553 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x555a429de480 0x555a429e08a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:12.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.861+0000 7f32d5800640 1 --2- 172.21.15.90:0/4275357553 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x555a429de480 0x555a429e08a0 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=99ccc865bfc088ca server_cookie=d1ac2aabb59d7fa2 in_seq=0 out_seq=0 2024-04-02T14:50:12.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.861+0000 7f32b6a00640 10 client.30182 ms_handle_connect on v2:172.21.15.177:6836/4223414858 2024-04-02T14:50:12.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.862+0000 7f32b6a00640 1 -- 172.21.15.90:0/4275357553 <== mds.4 v2:172.21.15.177:6836/4223414858 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x555a428c7c30 con 0x555a429de480 2024-04-02T14:50:12.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.862+0000 7f32b6a00640 10 client.30182 handle_command_reply: tid=0 2024-04-02T14:50:12.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.862+0000 7f32da5bc740 1 client.30182 shutdown 2024-04-02T14:50:12.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.862+0000 7f32da5bc740 1 -- 172.21.15.90:0/4275357553 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555a429569f0 msgr2=0x555a42958c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:12.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.862+0000 7f32da5bc740 1 --2- 172.21.15.90:0/4275357553 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555a429569f0 0x555a42958c20 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x555a42950e40 tx=0x7f32bc004580 comp rx=0 tx=0).stop 2024-04-02T14:50:12.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.862+0000 7f32da5bc740 1 -- 172.21.15.90:0/4275357553 shutdown_connections 2024-04-02T14:50:12.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.862+0000 7f32da5bc740 1 --2- 172.21.15.90:0/4275357553 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555a4295f080 0x555a42959230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:12.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.862+0000 7f32da5bc740 1 --2- 172.21.15.90:0/4275357553 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555a429569f0 0x555a42958c20 unknown :-1 s=CLOSED pgs=1017 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:12.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.862+0000 7f32da5bc740 1 --2- 172.21.15.90:0/4275357553 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x555a429de480 0x555a429e08a0 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:12.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.862+0000 7f32b6a00640 0 client.30182 ms_handle_reset on v2:172.21.15.177:6836/4223414858 2024-04-02T14:50:12.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.862+0000 7f32da5bc740 1 --2- 172.21.15.90:0/4275357553 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555a42955d10 0x555a4297d400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:12.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.862+0000 7f32da5bc740 1 -- 172.21.15.90:0/4275357553 >> 172.21.15.90:0/4275357553 conn(0x555a4291ca50 msgr2=0x555a42973120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:50:12.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.862+0000 7f32da5bc740 1 -- 172.21.15.90:0/4275357553 shutdown_connections 2024-04-02T14:50:12.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.862+0000 7f32da5bc740 1 -- 172.21.15.90:0/4275357553 wait complete. 2024-04-02T14:50:12.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.862+0000 7f32da5bc740 20 client.30182 trim_cache size 0 max 16384 2024-04-02T14:50:12.859 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-02T14:50:12.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.863+0000 7f32d7800640 1 -- 172.21.15.90:0/3768885936 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f32b80a3de0 msgr2=0x7f32b80a62a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:12.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.863+0000 7f32d7800640 1 --2- 172.21.15.90:0/3768885936 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f32b80a3de0 0x7f32b80a62a0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f32c40f00a0 tx=0x7f32c4000bc0 comp rx=0 tx=0).stop 2024-04-02T14:50:12.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.863+0000 7f32d7800640 1 -- 172.21.15.90:0/3768885936 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f32d0101aa0 msgr2=0x7f32d0198400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:12.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.863+0000 7f32d7800640 1 --2- 172.21.15.90:0/3768885936 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f32d0101aa0 0x7f32d0198400 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f32c4036040 tx=0x7f32c40043b0 comp rx=0 tx=0).stop 2024-04-02T14:50:12.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.863+0000 7f32d7800640 1 -- 172.21.15.90:0/3768885936 shutdown_connections 2024-04-02T14:50:12.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.863+0000 7f32d7800640 1 --2- 172.21.15.90:0/3768885936 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f32b80a3de0 0x7f32b80a62a0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:12.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.863+0000 7f32d7800640 1 --2- 172.21.15.90:0/3768885936 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f32d0107ab0 0x7f32d019ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:12.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.863+0000 7f32d7800640 1 --2- 172.21.15.90:0/3768885936 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f32d0102460 0x7f32d0198940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:12.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.863+0000 7f32d7800640 1 --2- 172.21.15.90:0/3768885936 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f32d0101aa0 0x7f32d0198400 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:12.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.863+0000 7f32d7800640 1 -- 172.21.15.90:0/3768885936 >> 172.21.15.90:0/3768885936 conn(0x7f32d00fd590 msgr2=0x7f32d00fdea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:50:12.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.863+0000 7f32d7800640 1 -- 172.21.15.90:0/3768885936 shutdown_connections 2024-04-02T14:50:12.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:12.863+0000 7f32d7800640 1 -- 172.21.15.90:0/3768885936 wait complete. 2024-04-02T14:50:12.876 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:50:12 smithi090 ceph-mon[31674]: from='client.? 172.21.15.90:0/988048918' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-02T14:50:12.879 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-04-02T14:50:13.102 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:50:12 smithi092 ceph-mon[36497]: from='client.? 172.21.15.90:0/988048918' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-02T14:50:13.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.150+0000 7fcd44600640 1 -- 172.21.15.90:0/1667029653 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcd3c0747f0 msgr2=0x7fcd3c10ddf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:13.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.150+0000 7fcd44600640 1 --2- 172.21.15.90:0/1667029653 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcd3c0747f0 0x7fcd3c10ddf0 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7fcd28009900 tx=0x7fcd2802bcd0 comp rx=0 tx=0).stop 2024-04-02T14:50:13.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.150+0000 7fcd44600640 1 -- 172.21.15.90:0/1667029653 shutdown_connections 2024-04-02T14:50:13.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.150+0000 7fcd44600640 1 --2- 172.21.15.90:0/1667029653 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcd3c10e330 0x7fcd3c110740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:13.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.150+0000 7fcd44600640 1 --2- 172.21.15.90:0/1667029653 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcd3c0747f0 0x7fcd3c10ddf0 secure :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7fcd28009900 tx=0x7fcd2802bcd0 comp rx=0 tx=0).stop 2024-04-02T14:50:13.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.150+0000 7fcd44600640 1 --2- 172.21.15.90:0/1667029653 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcd3c073ed0 0x7fcd3c0742b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:13.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.150+0000 7fcd44600640 1 -- 172.21.15.90:0/1667029653 >> 172.21.15.90:0/1667029653 conn(0x7fcd3c078f00 msgr2=0x7fcd3c079310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:50:13.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.151+0000 7fcd44600640 1 -- 172.21.15.90:0/1667029653 shutdown_connections 2024-04-02T14:50:13.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.151+0000 7fcd44600640 1 -- 172.21.15.90:0/1667029653 wait complete. 2024-04-02T14:50:13.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.156+0000 7fcd44600640 1 Processor -- start 2024-04-02T14:50:13.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.156+0000 7fcd44600640 1 -- start start 2024-04-02T14:50:13.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.156+0000 7fcd44600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcd3c073ed0 0x7fcd3c19a740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:13.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.157+0000 7fcd44600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcd3c10e330 0x7fcd3c19ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:13.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.157+0000 7fcd44600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcd3c19f000 0x7fcd3c19f4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:13.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.157+0000 7fcd44600640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fcd3c114090 con 0x7fcd3c10e330 2024-04-02T14:50:13.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.157+0000 7fcd44600640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fcd3c113f10 con 0x7fcd3c19f000 2024-04-02T14:50:13.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.157+0000 7fcd44600640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fcd3c114210 con 0x7fcd3c073ed0 2024-04-02T14:50:13.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.157+0000 7fcd42600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcd3c19f000 0x7fcd3c19f4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:13.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.157+0000 7fcd42600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcd3c19f000 0x7fcd3c19f4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:47642/0 (socket says 172.21.15.90:47642) 2024-04-02T14:50:13.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.157+0000 7fcd42600640 1 -- 172.21.15.90:0/4126322225 learned_addr learned my addr 172.21.15.90:0/4126322225 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:50:13.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.157+0000 7fcd41200640 1 --2- 172.21.15.90:0/4126322225 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcd3c10e330 0x7fcd3c19ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:13.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.158+0000 7fcd42600640 1 -- 172.21.15.90:0/4126322225 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcd3c073ed0 msgr2=0x7fcd3c19a740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:13.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.158+0000 7fcd42600640 1 --2- 172.21.15.90:0/4126322225 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcd3c073ed0 0x7fcd3c19a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:13.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.158+0000 7fcd42600640 1 -- 172.21.15.90:0/4126322225 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcd3c10e330 msgr2=0x7fcd3c19ac80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:13.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.158+0000 7fcd42600640 1 --2- 172.21.15.90:0/4126322225 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcd3c10e330 0x7fcd3c19ac80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:13.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.158+0000 7fcd42600640 1 -- 172.21.15.90:0/4126322225 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd3c19faf0 con 0x7fcd3c19f000 2024-04-02T14:50:13.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.158+0000 7fcd41200640 1 --2- 172.21.15.90:0/4126322225 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcd3c10e330 0x7fcd3c19ac80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:50:13.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.159+0000 7fcd42600640 1 --2- 172.21.15.90:0/4126322225 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcd3c19f000 0x7fcd3c19f4b0 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7fcd3400cce0 tx=0x7fcd3400a620 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:50:13.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.159+0000 7fcd3a000640 1 -- 172.21.15.90:0/4126322225 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd3400f450 con 0x7fcd3c19f000 2024-04-02T14:50:13.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.159+0000 7fcd44600640 1 -- 172.21.15.90:0/4126322225 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcd3c19fde0 con 0x7fcd3c19f000 2024-04-02T14:50:13.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.159+0000 7fcd3a000640 1 -- 172.21.15.90:0/4126322225 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcd3400fad0 con 0x7fcd3c19f000 2024-04-02T14:50:13.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.159+0000 7fcd3a000640 1 -- 172.21.15.90:0/4126322225 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd34018b70 con 0x7fcd3c19f000 2024-04-02T14:50:13.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.159+0000 7fcd44600640 1 -- 172.21.15.90:0/4126322225 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcd3c1a75a0 con 0x7fcd3c19f000 2024-04-02T14:50:13.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.164+0000 7fcd3a000640 1 -- 172.21.15.90:0/4126322225 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fcd34018d10 con 0x7fcd3c19f000 2024-04-02T14:50:13.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.165+0000 7fcd47259740 1 Processor -- start 2024-04-02T14:50:13.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.166+0000 7fcd3a000640 1 --2- 172.21.15.90:0/4126322225 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fcd240a3e20 0x7fcd240a62e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:13.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.166+0000 7fcd3a000640 1 -- 172.21.15.90:0/4126322225 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 8594+0+0 (secure 0 0 0) 0x7fcd340d0ca0 con 0x7fcd3c19f000 2024-04-02T14:50:13.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.167+0000 7fcd47259740 1 -- start start 2024-04-02T14:50:13.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.167+0000 7fcd41200640 1 --2- 172.21.15.90:0/4126322225 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fcd240a3e20 0x7fcd240a62e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:13.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.167+0000 7fcd47259740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55eebc7459f0 0x55eebc744740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:13.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.167+0000 7fcd47259740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eebc744d10 0x55eebc745190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:13.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.167+0000 7fcd47259740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eebc756890 0x55eebc758c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:13.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.168+0000 7fcd47259740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55eebc685400 con 0x55eebc744d10 2024-04-02T14:50:13.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.168+0000 7fcd47259740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55eebc669f10 con 0x55eebc7459f0 2024-04-02T14:50:13.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.168+0000 7fcd47259740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55eebc6a96f0 con 0x55eebc756890 2024-04-02T14:50:13.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.168+0000 7fcd42600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eebc756890 0x55eebc758c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:13.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.168+0000 7fcd42600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eebc756890 0x55eebc758c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:45994/0 (socket says 172.21.15.90:45994) 2024-04-02T14:50:13.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.168+0000 7fcd42600640 1 -- 172.21.15.90:0/4262818103 learned_addr learned my addr 172.21.15.90:0/4262818103 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:50:13.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.169+0000 7fcd41c00640 1 --2- 172.21.15.90:0/4262818103 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55eebc7459f0 0x55eebc744740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:13.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.170+0000 7fcd41200640 1 --2- 172.21.15.90:0/4262818103 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eebc744d10 0x55eebc745190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:13.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.171+0000 7fcd41200640 1 -- 172.21.15.90:0/4262818103 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eebc756890 msgr2=0x55eebc758c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:50:13.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.171+0000 7fcd41200640 1 --2- 172.21.15.90:0/4262818103 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eebc756890 0x55eebc758c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:13.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.171+0000 7fcd41200640 1 -- 172.21.15.90:0/4262818103 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55eebc7459f0 msgr2=0x55eebc744740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:50:13.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.171+0000 7fcd41200640 1 --2- 172.21.15.90:0/4262818103 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55eebc7459f0 0x55eebc744740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:13.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.171+0000 7fcd41200640 1 -- 172.21.15.90:0/4262818103 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55eebc3bd3f0 con 0x55eebc744d10 2024-04-02T14:50:13.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.172+0000 7fcd42600640 1 --2- 172.21.15.90:0/4262818103 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eebc756890 0x55eebc758c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:50:13.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.172+0000 7fcd41c00640 1 --2- 172.21.15.90:0/4262818103 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55eebc7459f0 0x55eebc744740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:50:13.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.173+0000 7fcd41200640 1 --2- 172.21.15.90:0/4262818103 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eebc744d10 0x55eebc745190 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fcd28008000 tx=0x7fcd28004220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d4ab507d23d24aa server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:50:13.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.174+0000 7fcd41200640 1 --2- 172.21.15.90:0/4126322225 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fcd240a3e20 0x7fcd240a62e0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fcd28040000 tx=0x7fcd2803d000 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:50:13.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.174+0000 7fcd2ea00640 1 -- 172.21.15.90:0/4262818103 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd28005610 con 0x55eebc744d10 2024-04-02T14:50:13.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.175+0000 7fcd2ea00640 1 -- 172.21.15.90:0/4262818103 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcd28004ba0 con 0x55eebc744d10 2024-04-02T14:50:13.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.175+0000 7fcd2ea00640 1 -- 172.21.15.90:0/4262818103 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd28003a50 con 0x55eebc744d10 2024-04-02T14:50:13.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.175+0000 7fcd47259740 1 -- 172.21.15.90:0/4262818103 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eebc744d10 msgr2=0x55eebc745190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:13.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.175+0000 7fcd47259740 1 --2- 172.21.15.90:0/4262818103 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eebc744d10 0x55eebc745190 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fcd28008000 tx=0x7fcd28004220 comp rx=0 tx=0).stop 2024-04-02T14:50:13.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.176+0000 7fcd47259740 1 -- 172.21.15.90:0/4262818103 shutdown_connections 2024-04-02T14:50:13.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.176+0000 7fcd47259740 1 --2- 172.21.15.90:0/4262818103 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eebc756890 0x55eebc758c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:13.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.176+0000 7fcd47259740 1 --2- 172.21.15.90:0/4262818103 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eebc744d10 0x55eebc745190 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:13.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.176+0000 7fcd47259740 1 --2- 172.21.15.90:0/4262818103 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55eebc7459f0 0x55eebc744740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:13.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.176+0000 7fcd47259740 1 -- 172.21.15.90:0/4262818103 >> 172.21.15.90:0/4262818103 conn(0x55eebc70ba50 msgr2=0x55eebc73f760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:50:13.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.176+0000 7fcd47259740 1 -- 172.21.15.90:0/4262818103 shutdown_connections 2024-04-02T14:50:13.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.176+0000 7fcd47259740 1 -- 172.21.15.90:0/4262818103 wait complete. 2024-04-02T14:50:13.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.177+0000 7fcd47259740 1 -- start start 2024-04-02T14:50:13.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.177+0000 7fcd47259740 1 Processor -- start 2024-04-02T14:50:13.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.178+0000 7fcd47259740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eebc744d10 0x55eebc753880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:13.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.179+0000 7fcd47259740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55eebc7459f0 0x55eebc753dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:13.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.179+0000 7fcd47259740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eebc756890 0x55eebc754300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:13.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.179+0000 7fcd47259740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55eebc6a56a0 con 0x55eebc756890 2024-04-02T14:50:13.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.179+0000 7fcd47259740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55eebc651980 con 0x55eebc7459f0 2024-04-02T14:50:13.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.179+0000 7fcd47259740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55eebc664cb0 con 0x55eebc744d10 2024-04-02T14:50:13.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.179+0000 7fcd47259740 10 client.0 _ll_register_callbacks cb 0x55eebc6646a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:50:13.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.179+0000 7fcd42600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eebc756890 0x55eebc754300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:13.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.180+0000 7fcd41200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55eebc7459f0 0x55eebc753dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:13.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.180+0000 7fcd41200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55eebc7459f0 0x55eebc753dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:47672/0 (socket says 172.21.15.90:47672) 2024-04-02T14:50:13.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.180+0000 7fcd41200640 1 -- 172.21.15.90:0/281949064 learned_addr learned my addr 172.21.15.90:0/281949064 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:50:13.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.180+0000 7fcd41c00640 1 --2- 172.21.15.90:0/281949064 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eebc744d10 0x55eebc753880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:13.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.180+0000 7fcd42600640 1 -- 172.21.15.90:0/281949064 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eebc744d10 msgr2=0x55eebc753880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:50:13.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.180+0000 7fcd42600640 1 --2- 172.21.15.90:0/281949064 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eebc744d10 0x55eebc753880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:13.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.180+0000 7fcd42600640 1 -- 172.21.15.90:0/281949064 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55eebc7459f0 msgr2=0x55eebc753dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:50:13.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.180+0000 7fcd42600640 1 --2- 172.21.15.90:0/281949064 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55eebc7459f0 0x55eebc753dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:13.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.180+0000 7fcd42600640 1 -- 172.21.15.90:0/281949064 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55eebc4877d0 con 0x55eebc756890 2024-04-02T14:50:13.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.181+0000 7fcd42600640 1 --2- 172.21.15.90:0/281949064 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eebc756890 0x55eebc754300 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fcd3400ccb0 tx=0x7fcd3409d290 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe5a01e815025dbf server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:50:13.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.181+0000 7fcd23400640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-02T14:50:13.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.181+0000 7fcd23400640 1 -- 172.21.15.90:0/281949064 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd340b6060 con 0x55eebc756890 2024-04-02T14:50:13.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.181+0000 7fcd23400640 1 -- 172.21.15.90:0/281949064 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 8594+0+0 (secure 0 0 0) 0x7fcd340ce5c0 con 0x55eebc756890 2024-04-02T14:50:13.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.181+0000 7fcd47259740 1 -- 172.21.15.90:0/281949064 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcd28004ba0 con 0x55eebc756890 2024-04-02T14:50:13.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.182+0000 7fcd23400640 1 -- 172.21.15.90:0/281949064 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fcd340a5800 con 0x55eebc756890 2024-04-02T14:50:13.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.182+0000 7fcd47259740 10 client.18882 fetch_fsmap learned FSMap version 27 2024-04-02T14:50:13.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.182+0000 7fcd47259740 1 -- 172.21.15.90:0/281949064 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55eebc765640 con 0x55eebc756890 2024-04-02T14:50:13.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.182+0000 7fcd23400640 1 -- 172.21.15.90:0/281949064 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcd28004ba0 con 0x55eebc756890 2024-04-02T14:50:13.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.182+0000 7fcd23400640 1 -- 172.21.15.90:0/281949064 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fcd34020030 con 0x55eebc756890 2024-04-02T14:50:13.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.183+0000 7fcd47259740 10 client.18882 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:50:13.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.183+0000 7fcd47259740 10 client.18882 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-04-02T14:50:13.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.183+0000 7fcd47259740 20 client.18882 populate_metadata read hostname 'smithi090' 2024-04-02T14:50:13.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.183+0000 7fcd47259740 1 --2- 172.21.15.90:0/281949064 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x55eebc766b00 0x55eebc768f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:13.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.183+0000 7fcd47259740 4 client.18882 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:50:13.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.183+0000 7fcd47259740 1 -- 172.21.15.90:0/281949064 --> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55eebc669040 con 0x55eebc766b00 2024-04-02T14:50:13.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.183+0000 7fcd41c00640 1 --2- 172.21.15.90:0/281949064 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x55eebc766b00 0x55eebc768f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:13.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.184+0000 7fcd41c00640 1 --2- 172.21.15.90:0/281949064 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x55eebc766b00 0x55eebc768f10 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=3d069d5c0671dc7 server_cookie=76722b6b293a5151 in_seq=0 out_seq=0 2024-04-02T14:50:13.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.184+0000 7fcd23400640 10 client.18882 ms_handle_connect on v2:172.21.15.92:6836/1270635939 2024-04-02T14:50:13.178 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:50:12 smithi177 ceph-mon[36431]: from='client.? 172.21.15.90:0/988048918' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-02T14:50:13.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.188+0000 7fcd23400640 1 -- 172.21.15.90:0/281949064 <== mds.3 v2:172.21.15.92:6836/1270635939 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55eebc669040 con 0x55eebc766b00 2024-04-02T14:50:13.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.188+0000 7fcd23400640 10 client.18882 handle_command_reply: tid=0 2024-04-02T14:50:13.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.188+0000 7fcd47259740 1 client.18882 shutdown 2024-04-02T14:50:13.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.189+0000 7fcd47259740 1 -- 172.21.15.90:0/281949064 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eebc756890 msgr2=0x55eebc754300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:13.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.189+0000 7fcd47259740 1 --2- 172.21.15.90:0/281949064 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eebc756890 0x55eebc754300 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fcd3400ccb0 tx=0x7fcd3409d290 comp rx=0 tx=0).stop 2024-04-02T14:50:13.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.190+0000 7fcd47259740 1 -- 172.21.15.90:0/281949064 shutdown_connections 2024-04-02T14:50:13.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.190+0000 7fcd47259740 1 --2- 172.21.15.90:0/281949064 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x55eebc766b00 0x55eebc768f10 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:13.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.190+0000 7fcd47259740 1 --2- 172.21.15.90:0/281949064 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eebc756890 0x55eebc754300 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:13.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.190+0000 7fcd47259740 1 --2- 172.21.15.90:0/281949064 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55eebc7459f0 0x55eebc753dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:13.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.190+0000 7fcd47259740 1 --2- 172.21.15.90:0/281949064 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eebc744d10 0x55eebc753880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:13.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.190+0000 7fcd23400640 0 client.18882 ms_handle_reset on v2:172.21.15.92:6836/1270635939 2024-04-02T14:50:13.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.190+0000 7fcd47259740 1 -- 172.21.15.90:0/281949064 >> 172.21.15.90:0/281949064 conn(0x55eebc70ba50 msgr2=0x55eebc7414b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:50:13.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.190+0000 7fcd47259740 1 -- 172.21.15.90:0/281949064 shutdown_connections 2024-04-02T14:50:13.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.190+0000 7fcd47259740 1 -- 172.21.15.90:0/281949064 wait complete. 2024-04-02T14:50:13.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.191+0000 7fcd47259740 20 client.18882 trim_cache size 0 max 16384 2024-04-02T14:50:13.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.215+0000 7fcd47259740 1 Processor -- start 2024-04-02T14:50:13.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.216+0000 7fcd47259740 1 -- start start 2024-04-02T14:50:13.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.216+0000 7fcd47259740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eebc744d10 0x55eebc755fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:13.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.216+0000 7fcd47259740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55eebc7459f0 0x55eebc7622a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:13.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.216+0000 7fcd47259740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eebc756890 0x55eebc76e830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:13.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.216+0000 7fcd47259740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fcd3c1abc20 con 0x55eebc756890 2024-04-02T14:50:13.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.216+0000 7fcd47259740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fcd3c1ad760 con 0x55eebc7459f0 2024-04-02T14:50:13.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.216+0000 7fcd47259740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fcd3c1ac2f0 con 0x55eebc744d10 2024-04-02T14:50:13.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.217+0000 7fcd41200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55eebc7459f0 0x55eebc7622a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:13.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.217+0000 7fcd41200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55eebc7459f0 0x55eebc7622a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:47676/0 (socket says 172.21.15.90:47676) 2024-04-02T14:50:13.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.217+0000 7fcd41200640 1 -- 172.21.15.90:0/1172225739 learned_addr learned my addr 172.21.15.90:0/1172225739 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:50:13.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.217+0000 7fcd42600640 1 --2- 172.21.15.90:0/1172225739 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eebc756890 0x55eebc76e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:13.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.217+0000 7fcd41200640 1 -- 172.21.15.90:0/1172225739 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eebc744d10 msgr2=0x55eebc755fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:50:13.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.217+0000 7fcd41c00640 1 --2- 172.21.15.90:0/1172225739 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eebc744d10 0x55eebc755fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:13.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.217+0000 7fcd41200640 1 --2- 172.21.15.90:0/1172225739 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eebc744d10 0x55eebc755fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:13.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.217+0000 7fcd41200640 1 -- 172.21.15.90:0/1172225739 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eebc756890 msgr2=0x55eebc76e830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:50:13.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.217+0000 7fcd41200640 1 --2- 172.21.15.90:0/1172225739 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eebc756890 0x55eebc76e830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:13.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.217+0000 7fcd41200640 1 -- 172.21.15.90:0/1172225739 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55eebc73f9a0 con 0x55eebc7459f0 2024-04-02T14:50:13.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.218+0000 7fcd41c00640 1 --2- 172.21.15.90:0/1172225739 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eebc744d10 0x55eebc755fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:50:13.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.218+0000 7fcd41200640 1 --2- 172.21.15.90:0/1172225739 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55eebc7459f0 0x55eebc7622a0 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7fcd28000c00 tx=0x7fcd2804d000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7cb5931976c45a09 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:50:13.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.218+0000 7fcd2e000640 1 -- 172.21.15.90:0/1172225739 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd28005610 con 0x55eebc7459f0 2024-04-02T14:50:13.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.218+0000 7fcd2e000640 1 -- 172.21.15.90:0/1172225739 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcd280046c0 con 0x55eebc7459f0 2024-04-02T14:50:13.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.218+0000 7fcd2e000640 1 -- 172.21.15.90:0/1172225739 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd2803d690 con 0x55eebc7459f0 2024-04-02T14:50:13.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.219+0000 7fcd47259740 1 -- 172.21.15.90:0/1172225739 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55eebc7459f0 msgr2=0x55eebc7622a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:13.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.219+0000 7fcd47259740 1 --2- 172.21.15.90:0/1172225739 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55eebc7459f0 0x55eebc7622a0 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7fcd28000c00 tx=0x7fcd2804d000 comp rx=0 tx=0).stop 2024-04-02T14:50:13.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.219+0000 7fcd47259740 1 -- 172.21.15.90:0/1172225739 shutdown_connections 2024-04-02T14:50:13.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.219+0000 7fcd47259740 1 --2- 172.21.15.90:0/1172225739 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eebc756890 0x55eebc76e830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:13.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.219+0000 7fcd47259740 1 --2- 172.21.15.90:0/1172225739 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55eebc7459f0 0x55eebc7622a0 unknown :-1 s=CLOSED pgs=1019 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:13.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.219+0000 7fcd47259740 1 --2- 172.21.15.90:0/1172225739 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eebc744d10 0x55eebc755fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:13.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.219+0000 7fcd47259740 1 -- 172.21.15.90:0/1172225739 >> 172.21.15.90:0/1172225739 conn(0x55eebc70ba50 msgr2=0x55eebc73f290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:50:13.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.219+0000 7fcd47259740 1 -- 172.21.15.90:0/1172225739 shutdown_connections 2024-04-02T14:50:13.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.220+0000 7fcd47259740 1 -- 172.21.15.90:0/1172225739 wait complete. 2024-04-02T14:50:13.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.220+0000 7fcd47259740 1 -- start start 2024-04-02T14:50:13.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.220+0000 7fcd47259740 1 Processor -- start 2024-04-02T14:50:13.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.221+0000 7fcd47259740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eebc744d10 0x55eebc7c88b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:13.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.221+0000 7fcd47259740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eebc7459f0 0x55eebc75e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:13.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.221+0000 7fcd47259740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55eebc756890 0x55eebc75ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:13.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.221+0000 7fcd47259740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fcd3c1ad090 con 0x55eebc7459f0 2024-04-02T14:50:13.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.221+0000 7fcd47259740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fcd3c1ac9c0 con 0x55eebc756890 2024-04-02T14:50:13.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.221+0000 7fcd47259740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fcd3c1a9c20 con 0x55eebc744d10 2024-04-02T14:50:13.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.221+0000 7fcd47259740 10 client.0 _ll_register_callbacks cb 0x7fcd3409c6d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:50:13.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.221+0000 7fcd41c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eebc744d10 0x55eebc7c88b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:13.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.221+0000 7fcd41c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eebc744d10 0x55eebc7c88b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:46012/0 (socket says 172.21.15.90:46012) 2024-04-02T14:50:13.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.221+0000 7fcd41c00640 1 -- 172.21.15.90:0/3583664865 learned_addr learned my addr 172.21.15.90:0/3583664865 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:50:13.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.221+0000 7fcd42600640 1 --2- 172.21.15.90:0/3583664865 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55eebc756890 0x55eebc75ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:13.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.222+0000 7fcd41c00640 1 -- 172.21.15.90:0/3583664865 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55eebc756890 msgr2=0x55eebc75ecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:50:13.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.222+0000 7fcd41c00640 1 --2- 172.21.15.90:0/3583664865 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55eebc756890 0x55eebc75ecf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:13.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.222+0000 7fcd41c00640 1 -- 172.21.15.90:0/3583664865 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eebc7459f0 msgr2=0x55eebc75e7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:50:13.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.222+0000 7fcd41200640 1 --2- 172.21.15.90:0/3583664865 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eebc7459f0 0x55eebc75e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:13.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.222+0000 7fcd41c00640 1 --2- 172.21.15.90:0/3583664865 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eebc7459f0 0x55eebc75e7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:13.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.222+0000 7fcd41c00640 1 -- 172.21.15.90:0/3583664865 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55eebc750930 con 0x55eebc744d10 2024-04-02T14:50:13.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.222+0000 7fcd42600640 1 --2- 172.21.15.90:0/3583664865 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55eebc756890 0x55eebc75ecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:50:13.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.222+0000 7fcd41200640 1 --2- 172.21.15.90:0/3583664865 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eebc7459f0 0x55eebc75e7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:50:13.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.222+0000 7fcd41c00640 1 --2- 172.21.15.90:0/3583664865 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eebc744d10 0x55eebc7c88b0 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7fcd30000c00 tx=0x7fcd30019f20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=18ccb69feca10557 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:50:13.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.222+0000 7fcd23400640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:50:13.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.222+0000 7fcd23400640 1 -- 172.21.15.90:0/3583664865 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd30004250 con 0x55eebc744d10 2024-04-02T14:50:13.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.222+0000 7fcd23400640 1 -- 172.21.15.90:0/3583664865 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 8594+0+0 (secure 0 0 0) 0x7fcd300043f0 con 0x55eebc744d10 2024-04-02T14:50:13.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.222+0000 7fcd47259740 1 -- 172.21.15.90:0/3583664865 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcd280046c0 con 0x55eebc744d10 2024-04-02T14:50:13.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.223+0000 7fcd23400640 1 -- 172.21.15.90:0/3583664865 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fcd300148a0 con 0x55eebc744d10 2024-04-02T14:50:13.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.223+0000 7fcd47259740 10 client.29980 fetch_fsmap learned FSMap version 27 2024-04-02T14:50:13.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.223+0000 7fcd47259740 1 -- 172.21.15.90:0/3583664865 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55eebc74a430 con 0x55eebc744d10 2024-04-02T14:50:13.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.223+0000 7fcd23400640 1 -- 172.21.15.90:0/3583664865 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcd280046c0 con 0x55eebc744d10 2024-04-02T14:50:13.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.223+0000 7fcd23400640 1 -- 172.21.15.90:0/3583664865 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fcd3000f040 con 0x55eebc744d10 2024-04-02T14:50:13.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.223+0000 7fcd47259740 10 client.29980 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:50:13.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.223+0000 7fcd47259740 10 client.29980 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-04-02T14:50:13.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.223+0000 7fcd47259740 20 client.29980 populate_metadata read hostname 'smithi090' 2024-04-02T14:50:13.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.223+0000 7fcd47259740 1 --2- 172.21.15.90:0/3583664865 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x55eebc74ac30 0x55eebc74d050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:13.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.223+0000 7fcd47259740 4 client.29980 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-02T14:50:13.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.223+0000 7fcd47259740 1 -- 172.21.15.90:0/3583664865 --> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55eebc6b6c30 con 0x55eebc74ac30 2024-04-02T14:50:13.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.224+0000 7fcd41200640 1 --2- 172.21.15.90:0/3583664865 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x55eebc74ac30 0x55eebc74d050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:13.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.224+0000 7fcd23400640 10 client.29980 ms_handle_connect on v2:172.21.15.92:6836/1270635939 2024-04-02T14:50:13.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.224+0000 7fcd41200640 1 --2- 172.21.15.90:0/3583664865 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x55eebc74ac30 0x55eebc74d050 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=7bffa50da95da77c server_cookie=81164ff93b61d8ee in_seq=0 out_seq=0 2024-04-02T14:50:13.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.225+0000 7fcd23400640 1 -- 172.21.15.90:0/3583664865 <== mds.3 v2:172.21.15.92:6836/1270635939 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55eebc6b6c30 con 0x55eebc74ac30 2024-04-02T14:50:13.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.225+0000 7fcd23400640 10 client.29980 handle_command_reply: tid=0 2024-04-02T14:50:13.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.225+0000 7fcd47259740 1 client.29980 shutdown 2024-04-02T14:50:13.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.225+0000 7fcd47259740 1 -- 172.21.15.90:0/3583664865 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eebc744d10 msgr2=0x55eebc7c88b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:13.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.225+0000 7fcd47259740 1 --2- 172.21.15.90:0/3583664865 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eebc744d10 0x55eebc7c88b0 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7fcd30000c00 tx=0x7fcd30019f20 comp rx=0 tx=0).stop 2024-04-02T14:50:13.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.225+0000 7fcd47259740 1 -- 172.21.15.90:0/3583664865 shutdown_connections 2024-04-02T14:50:13.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.225+0000 7fcd47259740 1 --2- 172.21.15.90:0/3583664865 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x55eebc74ac30 0x55eebc74d050 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:13.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.225+0000 7fcd47259740 1 --2- 172.21.15.90:0/3583664865 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55eebc756890 0x55eebc75ecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:13.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.225+0000 7fcd47259740 1 --2- 172.21.15.90:0/3583664865 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55eebc7459f0 0x55eebc75e7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:13.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.225+0000 7fcd23400640 0 client.29980 ms_handle_reset on v2:172.21.15.92:6836/1270635939 2024-04-02T14:50:13.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.225+0000 7fcd47259740 1 --2- 172.21.15.90:0/3583664865 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55eebc744d10 0x55eebc7c88b0 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:13.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.225+0000 7fcd47259740 1 -- 172.21.15.90:0/3583664865 >> 172.21.15.90:0/3583664865 conn(0x55eebc70ba50 msgr2=0x55eebc740bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:50:13.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.225+0000 7fcd47259740 1 -- 172.21.15.90:0/3583664865 shutdown_connections 2024-04-02T14:50:13.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.225+0000 7fcd47259740 1 -- 172.21.15.90:0/3583664865 wait complete. 2024-04-02T14:50:13.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.226+0000 7fcd47259740 20 client.29980 trim_cache size 0 max 16384 2024-04-02T14:50:13.221 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-02T14:50:13.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.227+0000 7fcd44600640 1 -- 172.21.15.90:0/4126322225 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fcd240a3e20 msgr2=0x7fcd240a62e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:13.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.228+0000 7fcd44600640 1 --2- 172.21.15.90:0/4126322225 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fcd240a3e20 0x7fcd240a62e0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fcd28040000 tx=0x7fcd2803d000 comp rx=0 tx=0).stop 2024-04-02T14:50:13.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.228+0000 7fcd44600640 1 -- 172.21.15.90:0/4126322225 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcd3c19f000 msgr2=0x7fcd3c19f4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:13.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.228+0000 7fcd44600640 1 --2- 172.21.15.90:0/4126322225 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcd3c19f000 0x7fcd3c19f4b0 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7fcd3400cce0 tx=0x7fcd3400a620 comp rx=0 tx=0).stop 2024-04-02T14:50:13.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.228+0000 7fcd44600640 1 -- 172.21.15.90:0/4126322225 shutdown_connections 2024-04-02T14:50:13.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.228+0000 7fcd44600640 1 --2- 172.21.15.90:0/4126322225 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fcd240a3e20 0x7fcd240a62e0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:13.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.228+0000 7fcd44600640 1 --2- 172.21.15.90:0/4126322225 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcd3c19f000 0x7fcd3c19f4b0 unknown :-1 s=CLOSED pgs=1018 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:13.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.228+0000 7fcd44600640 1 --2- 172.21.15.90:0/4126322225 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcd3c10e330 0x7fcd3c19ac80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:13.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.228+0000 7fcd44600640 1 --2- 172.21.15.90:0/4126322225 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcd3c073ed0 0x7fcd3c19a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:13.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.228+0000 7fcd44600640 1 -- 172.21.15.90:0/4126322225 >> 172.21.15.90:0/4126322225 conn(0x7fcd3c078f00 msgr2=0x7fcd3c0ffee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:50:13.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.229+0000 7fcd44600640 1 -- 172.21.15.90:0/4126322225 shutdown_connections 2024-04-02T14:50:13.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.229+0000 7fcd44600640 1 -- 172.21.15.90:0/4126322225 wait complete. 2024-04-02T14:50:13.243 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-02T14:50:13.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.491+0000 7f3b42200640 1 -- 172.21.15.90:0/2426886842 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3b3c108ec0 msgr2=0x7f3b3c110a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:13.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.491+0000 7f3b42200640 1 --2- 172.21.15.90:0/2426886842 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3b3c108ec0 0x7f3b3c110a90 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7f3b34009ca0 tx=0x7f3b3402c100 comp rx=0 tx=0).stop 2024-04-02T14:50:13.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.491+0000 7f3b42200640 1 -- 172.21.15.90:0/2426886842 shutdown_connections 2024-04-02T14:50:13.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.491+0000 7f3b42200640 1 --2- 172.21.15.90:0/2426886842 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3b3c108ec0 0x7f3b3c110a90 unknown :-1 s=CLOSED pgs=1020 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:13.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.491+0000 7f3b42200640 1 --2- 172.21.15.90:0/2426886842 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3b3c1047a0 0x7f3b3c1087e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:13.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.491+0000 7f3b42200640 1 --2- 172.21.15.90:0/2426886842 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3b3c103df0 0x7f3b3c1041d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:13.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.491+0000 7f3b42200640 1 -- 172.21.15.90:0/2426886842 >> 172.21.15.90:0/2426886842 conn(0x7f3b3c0ff610 msgr2=0x7f3b3c101a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:50:13.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.491+0000 7f3b42200640 1 -- 172.21.15.90:0/2426886842 shutdown_connections 2024-04-02T14:50:13.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.491+0000 7f3b42200640 1 -- 172.21.15.90:0/2426886842 wait complete. 2024-04-02T14:50:13.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.492+0000 7f3b42200640 1 Processor -- start 2024-04-02T14:50:13.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.492+0000 7f3b42200640 1 -- start start 2024-04-02T14:50:13.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.493+0000 7f3b42200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3b3c103df0 0x7f3b3c19a490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:13.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.493+0000 7f3b42200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3b3c108ec0 0x7f3b3c19a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:13.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.493+0000 7f3b42200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3b3c19ed50 0x7f3b3c19f200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:13.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.493+0000 7f3b42200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f3b3c113f70 con 0x7f3b3c108ec0 2024-04-02T14:50:13.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.493+0000 7f3b42200640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f3b3c113df0 con 0x7f3b3c103df0 2024-04-02T14:50:13.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.493+0000 7f3b42200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f3b3c1140f0 con 0x7f3b3c19ed50 2024-04-02T14:50:13.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.493+0000 7f3b40e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3b3c103df0 0x7f3b3c19a490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:13.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.493+0000 7f3b40e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3b3c103df0 0x7f3b3c19a490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:47720/0 (socket says 172.21.15.90:47720) 2024-04-02T14:50:13.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.493+0000 7f3b41800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3b3c19ed50 0x7f3b3c19f200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:13.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.493+0000 7f3b41800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3b3c19ed50 0x7f3b3c19f200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:46044/0 (socket says 172.21.15.90:46044) 2024-04-02T14:50:13.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.493+0000 7f3b40e00640 1 -- 172.21.15.90:0/1913535589 learned_addr learned my addr 172.21.15.90:0/1913535589 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:50:13.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.493+0000 7f3b3be00640 1 --2- 172.21.15.90:0/1913535589 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3b3c108ec0 0x7f3b3c19a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:13.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.494+0000 7f3b40e00640 1 -- 172.21.15.90:0/1913535589 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3b3c19ed50 msgr2=0x7f3b3c19f200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:13.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.494+0000 7f3b40e00640 1 --2- 172.21.15.90:0/1913535589 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3b3c19ed50 0x7f3b3c19f200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:13.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.494+0000 7f3b40e00640 1 -- 172.21.15.90:0/1913535589 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3b3c108ec0 msgr2=0x7f3b3c19a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:13.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.494+0000 7f3b40e00640 1 --2- 172.21.15.90:0/1913535589 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3b3c108ec0 0x7f3b3c19a9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:13.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.494+0000 7f3b40e00640 1 -- 172.21.15.90:0/1913535589 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b3c19f840 con 0x7f3b3c103df0 2024-04-02T14:50:13.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.495+0000 7f3b40e00640 1 --2- 172.21.15.90:0/1913535589 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3b3c103df0 0x7f3b3c19a490 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7f3b3000e9f0 tx=0x7f3b3000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:50:13.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.496+0000 7f3b39600640 1 -- 172.21.15.90:0/1913535589 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b3000cd80 con 0x7f3b3c103df0 2024-04-02T14:50:13.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.496+0000 7f3b39600640 1 -- 172.21.15.90:0/1913535589 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3b30010910 con 0x7f3b3c103df0 2024-04-02T14:50:13.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.496+0000 7f3b39600640 1 -- 172.21.15.90:0/1913535589 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b3000cd80 con 0x7f3b3c103df0 2024-04-02T14:50:13.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.496+0000 7f3b42200640 1 -- 172.21.15.90:0/1913535589 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b3c19fb30 con 0x7f3b3c103df0 2024-04-02T14:50:13.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.496+0000 7f3b42200640 1 -- 172.21.15.90:0/1913535589 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b3c076150 con 0x7f3b3c103df0 2024-04-02T14:50:13.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.499+0000 7f3b39600640 1 -- 172.21.15.90:0/1913535589 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f3b30010ac0 con 0x7f3b3c103df0 2024-04-02T14:50:13.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.500+0000 7f3b463f3740 1 Processor -- start 2024-04-02T14:50:13.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.501+0000 7f3b39600640 1 --2- 172.21.15.90:0/1913535589 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f3b240a3fa0 0x7f3b240a6460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:13.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.501+0000 7f3b463f3740 1 -- start start 2024-04-02T14:50:13.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.501+0000 7f3b39600640 1 -- 172.21.15.90:0/1913535589 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 8594+0+0 (secure 0 0 0) 0x7f3b300d08f0 con 0x7f3b3c103df0 2024-04-02T14:50:13.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.501+0000 7f3b41800640 1 --2- 172.21.15.90:0/1913535589 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f3b240a3fa0 0x7f3b240a6460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:13.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.501+0000 7f3b463f3740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55da24b33ba0 0x55da24b328f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:13.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.501+0000 7f3b463f3740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55da24b32ec0 0x55da24b33340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:13.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.501+0000 7f3b463f3740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55da24b3c230 0x55da24b3e620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:13.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.501+0000 7f3b463f3740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55da24a73400 con 0x55da24b33ba0 2024-04-02T14:50:13.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.501+0000 7f3b463f3740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55da24a57f10 con 0x55da24b3c230 2024-04-02T14:50:13.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.501+0000 7f3b463f3740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55da24a976f0 con 0x55da24b32ec0 2024-04-02T14:50:13.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.501+0000 7f3b40e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55da24b33ba0 0x55da24b328f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:13.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.501+0000 7f3b41800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55da24b3c230 0x55da24b3e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:13.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.502+0000 7f3b3be00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55da24b32ec0 0x55da24b33340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:13.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.502+0000 7f3b40e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55da24b33ba0 0x55da24b328f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41376/0 (socket says 172.21.15.90:41376) 2024-04-02T14:50:13.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.502+0000 7f3b40e00640 1 -- 172.21.15.90:0/2410026107 learned_addr learned my addr 172.21.15.90:0/2410026107 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:50:13.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.502+0000 7f3b40e00640 1 -- 172.21.15.90:0/2410026107 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55da24b32ec0 msgr2=0x55da24b33340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:50:13.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.502+0000 7f3b40e00640 1 --2- 172.21.15.90:0/2410026107 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55da24b32ec0 0x55da24b33340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:13.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.502+0000 7f3b40e00640 1 -- 172.21.15.90:0/2410026107 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55da24b3c230 msgr2=0x55da24b3e620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:50:13.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.502+0000 7f3b40e00640 1 --2- 172.21.15.90:0/2410026107 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55da24b3c230 0x55da24b3e620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:13.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.502+0000 7f3b40e00640 1 -- 172.21.15.90:0/2410026107 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55da247ab3f0 con 0x55da24b33ba0 2024-04-02T14:50:13.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.502+0000 7f3b3be00640 1 --2- 172.21.15.90:0/2410026107 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55da24b32ec0 0x55da24b33340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:50:13.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.502+0000 7f3b40e00640 1 --2- 172.21.15.90:0/2410026107 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55da24b33ba0 0x55da24b328f0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f3b300991b0 tx=0x7f3b3009c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4056ab8af2ff844 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:50:13.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.502+0000 7f3b41800640 1 --2- 172.21.15.90:0/1913535589 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f3b240a3fa0 0x7f3b240a6460 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f3b34005e30 tx=0x7f3b340099d0 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:50:13.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.502+0000 7f3b2e000640 1 -- 172.21.15.90:0/2410026107 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b300870b0 con 0x55da24b33ba0 2024-04-02T14:50:13.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.503+0000 7f3b2e000640 1 -- 172.21.15.90:0/2410026107 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3b300ce210 con 0x55da24b33ba0 2024-04-02T14:50:13.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.503+0000 7f3b2e000640 1 -- 172.21.15.90:0/2410026107 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b3009bf20 con 0x55da24b33ba0 2024-04-02T14:50:13.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.503+0000 7f3b463f3740 1 -- 172.21.15.90:0/2410026107 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55da24b33ba0 msgr2=0x55da24b328f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:13.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.503+0000 7f3b463f3740 1 --2- 172.21.15.90:0/2410026107 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55da24b33ba0 0x55da24b328f0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f3b300991b0 tx=0x7f3b3009c7e0 comp rx=0 tx=0).stop 2024-04-02T14:50:13.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.503+0000 7f3b463f3740 1 -- 172.21.15.90:0/2410026107 shutdown_connections 2024-04-02T14:50:13.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.503+0000 7f3b463f3740 1 --2- 172.21.15.90:0/2410026107 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55da24b3c230 0x55da24b3e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:13.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.503+0000 7f3b463f3740 1 --2- 172.21.15.90:0/2410026107 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55da24b32ec0 0x55da24b33340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:13.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.503+0000 7f3b463f3740 1 --2- 172.21.15.90:0/2410026107 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55da24b33ba0 0x55da24b328f0 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:13.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.503+0000 7f3b463f3740 1 -- 172.21.15.90:0/2410026107 >> 172.21.15.90:0/2410026107 conn(0x55da24af9a50 msgr2=0x55da24b2d910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:50:13.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.504+0000 7f3b463f3740 1 -- 172.21.15.90:0/2410026107 shutdown_connections 2024-04-02T14:50:13.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.504+0000 7f3b463f3740 1 -- 172.21.15.90:0/2410026107 wait complete. 2024-04-02T14:50:13.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.504+0000 7f3b463f3740 1 -- start start 2024-04-02T14:50:13.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.505+0000 7f3b463f3740 1 Processor -- start 2024-04-02T14:50:13.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.505+0000 7f3b463f3740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55da24b32ec0 0x55da24b391f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:13.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.505+0000 7f3b463f3740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55da24b33ba0 0x55da24b39730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:13.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.505+0000 7f3b463f3740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55da24b3c230 0x55da24b39c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:13.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.505+0000 7f3b463f3740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55da24a936a0 con 0x55da24b33ba0 2024-04-02T14:50:13.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.505+0000 7f3b463f3740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55da24a3f980 con 0x55da24b32ec0 2024-04-02T14:50:13.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.505+0000 7f3b463f3740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55da24a52cb0 con 0x55da24b3c230 2024-04-02T14:50:13.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.505+0000 7f3b463f3740 10 client.0 _ll_register_callbacks cb 0x55da24a526a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:50:13.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.506+0000 7f3b40e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55da24b32ec0 0x55da24b391f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:13.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.506+0000 7f3b40e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55da24b32ec0 0x55da24b391f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:47750/0 (socket says 172.21.15.90:47750) 2024-04-02T14:50:13.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.506+0000 7f3b40e00640 1 -- 172.21.15.90:0/3220585773 learned_addr learned my addr 172.21.15.90:0/3220585773 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:50:13.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.506+0000 7f3b41800640 1 --2- 172.21.15.90:0/3220585773 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55da24b3c230 0x55da24b39c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:13.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.506+0000 7f3b40e00640 1 -- 172.21.15.90:0/3220585773 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55da24b3c230 msgr2=0x55da24b39c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:50:13.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.506+0000 7f3b3be00640 1 --2- 172.21.15.90:0/3220585773 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55da24b33ba0 0x55da24b39730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:13.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.507+0000 7f3b40e00640 1 --2- 172.21.15.90:0/3220585773 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55da24b3c230 0x55da24b39c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:13.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.507+0000 7f3b40e00640 1 -- 172.21.15.90:0/3220585773 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55da24b33ba0 msgr2=0x55da24b39730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:50:13.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.507+0000 7f3b40e00640 1 --2- 172.21.15.90:0/3220585773 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55da24b33ba0 0x55da24b39730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:13.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.507+0000 7f3b40e00640 1 -- 172.21.15.90:0/3220585773 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55da248757d0 con 0x55da24b32ec0 2024-04-02T14:50:13.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.507+0000 7f3b41800640 1 --2- 172.21.15.90:0/3220585773 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55da24b3c230 0x55da24b39c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:50:13.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.510+0000 7f3b40e00640 1 --2- 172.21.15.90:0/3220585773 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55da24b32ec0 0x55da24b391f0 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7f3b3009c4e0 tx=0x7f3b30009770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eb1829b65257f3e4 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:50:13.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.510+0000 7f3b22a00640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:50:13.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.511+0000 7f3b22a00640 1 -- 172.21.15.90:0/3220585773 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b300b9020 con 0x55da24b32ec0 2024-04-02T14:50:13.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.511+0000 7f3b22a00640 1 -- 172.21.15.90:0/3220585773 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 8594+0+0 (secure 0 0 0) 0x7f3b3009beb0 con 0x55da24b32ec0 2024-04-02T14:50:13.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.511+0000 7f3b463f3740 1 -- 172.21.15.90:0/3220585773 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3b300ce210 con 0x55da24b32ec0 2024-04-02T14:50:13.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.511+0000 7f3b22a00640 1 -- 172.21.15.90:0/3220585773 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f3b30010430 con 0x55da24b32ec0 2024-04-02T14:50:13.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.511+0000 7f3b463f3740 10 client.30209 fetch_fsmap learned FSMap version 27 2024-04-02T14:50:13.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.511+0000 7f3b463f3740 1 -- 172.21.15.90:0/3220585773 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55da24b4b040 con 0x55da24b32ec0 2024-04-02T14:50:13.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.518+0000 7f3b22a00640 1 -- 172.21.15.90:0/3220585773 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3b300ce210 con 0x55da24b32ec0 2024-04-02T14:50:13.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.518+0000 7f3b22a00640 1 -- 172.21.15.90:0/3220585773 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f3b3009b6a0 con 0x55da24b32ec0 2024-04-02T14:50:13.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.518+0000 7f3b463f3740 10 client.30209 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:50:13.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.518+0000 7f3b463f3740 10 client.30209 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-04-02T14:50:13.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.518+0000 7f3b463f3740 20 client.30209 populate_metadata read hostname 'smithi090' 2024-04-02T14:50:13.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.518+0000 7f3b463f3740 1 --2- 172.21.15.90:0/3220585773 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x55da24b4c390 0x55da24b4e780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:13.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.518+0000 7f3b463f3740 4 client.30209 mds_command: new command op to 24547 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:50:13.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.518+0000 7f3b463f3740 1 -- 172.21.15.90:0/3220585773 --> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55da24a57040 con 0x55da24b4c390 2024-04-02T14:50:13.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.519+0000 7f3b3be00640 1 --2- 172.21.15.90:0/3220585773 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x55da24b4c390 0x55da24b4e780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:13.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.520+0000 7f3b22a00640 10 client.30209 ms_handle_connect on v2:172.21.15.177:6838/664895663 2024-04-02T14:50:13.514 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.522+0000 7f3b3be00640 1 --2- 172.21.15.90:0/3220585773 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x55da24b4c390 0x55da24b4e780 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=bcc5c3d8446534e8 server_cookie=61423d0802626852 in_seq=0 out_seq=0 2024-04-02T14:50:13.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.523+0000 7f3b22a00640 1 -- 172.21.15.90:0/3220585773 <== mds.2 v2:172.21.15.177:6838/664895663 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55da24a57040 con 0x55da24b4c390 2024-04-02T14:50:13.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.523+0000 7f3b22a00640 10 client.30209 handle_command_reply: tid=0 2024-04-02T14:50:13.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.523+0000 7f3b463f3740 1 client.30209 shutdown 2024-04-02T14:50:13.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.524+0000 7f3b463f3740 1 -- 172.21.15.90:0/3220585773 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55da24b32ec0 msgr2=0x55da24b391f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:13.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.524+0000 7f3b463f3740 1 --2- 172.21.15.90:0/3220585773 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55da24b32ec0 0x55da24b391f0 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7f3b3009c4e0 tx=0x7f3b30009770 comp rx=0 tx=0).stop 2024-04-02T14:50:13.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.525+0000 7f3b463f3740 1 -- 172.21.15.90:0/3220585773 shutdown_connections 2024-04-02T14:50:13.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.525+0000 7f3b463f3740 1 --2- 172.21.15.90:0/3220585773 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x55da24b4c390 0x55da24b4e780 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:13.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.525+0000 7f3b463f3740 1 --2- 172.21.15.90:0/3220585773 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55da24b3c230 0x55da24b39c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:13.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.525+0000 7f3b463f3740 1 --2- 172.21.15.90:0/3220585773 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55da24b33ba0 0x55da24b39730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:13.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.525+0000 7f3b22a00640 0 client.30209 ms_handle_reset on v2:172.21.15.177:6838/664895663 2024-04-02T14:50:13.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.525+0000 7f3b463f3740 1 --2- 172.21.15.90:0/3220585773 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55da24b32ec0 0x55da24b391f0 unknown :-1 s=CLOSED pgs=1022 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:13.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.525+0000 7f3b463f3740 1 -- 172.21.15.90:0/3220585773 >> 172.21.15.90:0/3220585773 conn(0x55da24af9a50 msgr2=0x55da24b2edd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:50:13.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.526+0000 7f3b463f3740 1 -- 172.21.15.90:0/3220585773 shutdown_connections 2024-04-02T14:50:13.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.526+0000 7f3b463f3740 1 -- 172.21.15.90:0/3220585773 wait complete. 2024-04-02T14:50:13.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.526+0000 7f3b463f3740 20 client.30209 trim_cache size 0 max 16384 2024-04-02T14:50:13.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.558+0000 7f3b463f3740 1 Processor -- start 2024-04-02T14:50:13.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.559+0000 7f3b463f3740 1 -- start start 2024-04-02T14:50:13.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.559+0000 7f3b463f3740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55da24b32ec0 0x55da24b3b920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:13.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.560+0000 7f3b463f3740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55da24b33ba0 0x55da24b39a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:13.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.560+0000 7f3b463f3740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55da24b3c230 0x55da24b380a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:13.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.560+0000 7f3b463f3740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f3b3c1b3f60 con 0x55da24b3c230 2024-04-02T14:50:13.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.560+0000 7f3b463f3740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f3b3c1b5aa0 con 0x55da24b32ec0 2024-04-02T14:50:13.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.560+0000 7f3b463f3740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f3b3c1b4630 con 0x55da24b33ba0 2024-04-02T14:50:13.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.561+0000 7f3b3be00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55da24b33ba0 0x55da24b39a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:13.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.561+0000 7f3b3be00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55da24b33ba0 0x55da24b39a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:46070/0 (socket says 172.21.15.90:46070) 2024-04-02T14:50:13.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.561+0000 7f3b3be00640 1 -- 172.21.15.90:0/757517565 learned_addr learned my addr 172.21.15.90:0/757517565 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:50:13.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.561+0000 7f3b3be00640 1 -- 172.21.15.90:0/757517565 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55da24b32ec0 msgr2=0x55da24b3b920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:50:13.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.561+0000 7f3b3be00640 1 --2- 172.21.15.90:0/757517565 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55da24b32ec0 0x55da24b3b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:13.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.561+0000 7f3b3be00640 1 -- 172.21.15.90:0/757517565 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55da24b3c230 msgr2=0x55da24b380a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:50:13.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.561+0000 7f3b3be00640 1 --2- 172.21.15.90:0/757517565 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55da24b3c230 0x55da24b380a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:13.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.561+0000 7f3b3be00640 1 -- 172.21.15.90:0/757517565 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55da24b31ef0 con 0x55da24b33ba0 2024-04-02T14:50:13.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.563+0000 7f3b3be00640 1 --2- 172.21.15.90:0/757517565 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55da24b33ba0 0x55da24b39a90 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x55da24b340f0 tx=0x7f3b28012820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c1cb2802fa367124 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:50:13.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.563+0000 7f3b2d600640 1 -- 172.21.15.90:0/757517565 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b28012e80 con 0x55da24b33ba0 2024-04-02T14:50:13.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.564+0000 7f3b2d600640 1 -- 172.21.15.90:0/757517565 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3b28005500 con 0x55da24b33ba0 2024-04-02T14:50:13.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.564+0000 7f3b2d600640 1 -- 172.21.15.90:0/757517565 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b2801f0f0 con 0x55da24b33ba0 2024-04-02T14:50:13.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.565+0000 7f3b463f3740 1 -- 172.21.15.90:0/757517565 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55da24b33ba0 msgr2=0x55da24b39a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:13.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.565+0000 7f3b463f3740 1 --2- 172.21.15.90:0/757517565 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55da24b33ba0 0x55da24b39a90 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x55da24b340f0 tx=0x7f3b28012820 comp rx=0 tx=0).stop 2024-04-02T14:50:13.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.566+0000 7f3b463f3740 1 -- 172.21.15.90:0/757517565 shutdown_connections 2024-04-02T14:50:13.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.566+0000 7f3b463f3740 1 --2- 172.21.15.90:0/757517565 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55da24b3c230 0x55da24b380a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:13.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.566+0000 7f3b463f3740 1 --2- 172.21.15.90:0/757517565 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55da24b33ba0 0x55da24b39a90 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:13.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.566+0000 7f3b463f3740 1 --2- 172.21.15.90:0/757517565 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55da24b32ec0 0x55da24b3b920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:13.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.566+0000 7f3b463f3740 1 -- 172.21.15.90:0/757517565 >> 172.21.15.90:0/757517565 conn(0x55da24af9a50 msgr2=0x55da24b37990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:50:13.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.566+0000 7f3b463f3740 1 -- 172.21.15.90:0/757517565 shutdown_connections 2024-04-02T14:50:13.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.566+0000 7f3b463f3740 1 -- 172.21.15.90:0/757517565 wait complete. 2024-04-02T14:50:13.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.567+0000 7f3b463f3740 1 -- start start 2024-04-02T14:50:13.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.567+0000 7f3b463f3740 1 Processor -- start 2024-04-02T14:50:13.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.568+0000 7f3b463f3740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55da24b32ec0 0x55da24b4d1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:13.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.568+0000 7f3b463f3740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55da24b33ba0 0x55da24b4d730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:13.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.568+0000 7f3b463f3740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55da24b3c230 0x55da24b4dc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:13.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.568+0000 7f3b463f3740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f3b3c1b53d0 con 0x55da24b32ec0 2024-04-02T14:50:13.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.568+0000 7f3b463f3740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f3b3c1b4d00 con 0x55da24b33ba0 2024-04-02T14:50:13.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.568+0000 7f3b463f3740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f3b3c1b1f20 con 0x55da24b3c230 2024-04-02T14:50:13.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.568+0000 7f3b463f3740 10 client.0 _ll_register_callbacks cb 0x7f3b3009c1f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:50:13.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.568+0000 7f3b40e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55da24b32ec0 0x55da24b4d1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:13.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.568+0000 7f3b41800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55da24b3c230 0x55da24b4dc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:13.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.568+0000 7f3b41800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55da24b3c230 0x55da24b4dc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:46072/0 (socket says 172.21.15.90:46072) 2024-04-02T14:50:13.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.568+0000 7f3b41800640 1 -- 172.21.15.90:0/3241792015 learned_addr learned my addr 172.21.15.90:0/3241792015 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:50:13.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.569+0000 7f3b3be00640 1 --2- 172.21.15.90:0/3241792015 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55da24b33ba0 0x55da24b4d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:13.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.569+0000 7f3b40e00640 1 -- 172.21.15.90:0/3241792015 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55da24b3c230 msgr2=0x55da24b4dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:50:13.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.569+0000 7f3b40e00640 1 --2- 172.21.15.90:0/3241792015 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55da24b3c230 0x55da24b4dc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:13.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.569+0000 7f3b40e00640 1 -- 172.21.15.90:0/3241792015 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55da24b33ba0 msgr2=0x55da24b4d730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:50:13.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.569+0000 7f3b40e00640 1 --2- 172.21.15.90:0/3241792015 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55da24b33ba0 0x55da24b4d730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:13.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.569+0000 7f3b40e00640 1 -- 172.21.15.90:0/3241792015 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55da24bb1560 con 0x55da24b32ec0 2024-04-02T14:50:13.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.569+0000 7f3b40e00640 1 --2- 172.21.15.90:0/3241792015 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55da24b32ec0 0x55da24b4d1f0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f3b3009aa90 tx=0x7f3b30005f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee49258c4f5f86ac server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:50:13.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.569+0000 7f3b22a00640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-02T14:50:13.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.570+0000 7f3b22a00640 1 -- 172.21.15.90:0/3241792015 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b30010430 con 0x55da24b32ec0 2024-04-02T14:50:13.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.570+0000 7f3b22a00640 1 -- 172.21.15.90:0/3241792015 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 8594+0+0 (secure 0 0 0) 0x7f3b300ce210 con 0x55da24b32ec0 2024-04-02T14:50:13.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.571+0000 7f3b463f3740 1 -- 172.21.15.90:0/3241792015 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3b28005500 con 0x55da24b32ec0 2024-04-02T14:50:13.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.571+0000 7f3b22a00640 1 -- 172.21.15.90:0/3241792015 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f3b300a5450 con 0x55da24b32ec0 2024-04-02T14:50:13.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.571+0000 7f3b463f3740 10 client.18894 fetch_fsmap learned FSMap version 27 2024-04-02T14:50:13.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.571+0000 7f3b463f3740 1 -- 172.21.15.90:0/3241792015 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55da24bb27c0 con 0x55da24b32ec0 2024-04-02T14:50:13.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.572+0000 7f3b22a00640 1 -- 172.21.15.90:0/3241792015 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3b28005500 con 0x55da24b32ec0 2024-04-02T14:50:13.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.572+0000 7f3b22a00640 1 -- 172.21.15.90:0/3241792015 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f3b30098530 con 0x55da24b32ec0 2024-04-02T14:50:13.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.572+0000 7f3b463f3740 10 client.18894 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:50:13.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.572+0000 7f3b463f3740 10 client.18894 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-04-02T14:50:13.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.572+0000 7f3b463f3740 20 client.18894 populate_metadata read hostname 'smithi090' 2024-04-02T14:50:13.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.572+0000 7f3b463f3740 1 --2- 172.21.15.90:0/3241792015 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x55da24bb2d40 0x55da24bb5130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:13.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.572+0000 7f3b463f3740 4 client.18894 mds_command: new command op to 24547 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-02T14:50:13.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.572+0000 7f3b463f3740 1 -- 172.21.15.90:0/3241792015 --> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55da24aa4c30 con 0x55da24bb2d40 2024-04-02T14:50:13.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.573+0000 7f3b3be00640 1 --2- 172.21.15.90:0/3241792015 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x55da24bb2d40 0x55da24bb5130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:13.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.575+0000 7f3b3be00640 1 --2- 172.21.15.90:0/3241792015 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x55da24bb2d40 0x55da24bb5130 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=3440c2f23e7b2a7f server_cookie=e2789b752e705b7 in_seq=0 out_seq=0 2024-04-02T14:50:13.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.575+0000 7f3b22a00640 10 client.18894 ms_handle_connect on v2:172.21.15.177:6838/664895663 2024-04-02T14:50:13.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.576+0000 7f3b22a00640 1 -- 172.21.15.90:0/3241792015 <== mds.2 v2:172.21.15.177:6838/664895663 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55da24aa4c30 con 0x55da24bb2d40 2024-04-02T14:50:13.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.576+0000 7f3b22a00640 10 client.18894 handle_command_reply: tid=0 2024-04-02T14:50:13.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.576+0000 7f3b463f3740 1 client.18894 shutdown 2024-04-02T14:50:13.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.577+0000 7f3b463f3740 1 -- 172.21.15.90:0/3241792015 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55da24b32ec0 msgr2=0x55da24b4d1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:13.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.577+0000 7f3b463f3740 1 --2- 172.21.15.90:0/3241792015 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55da24b32ec0 0x55da24b4d1f0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f3b3009aa90 tx=0x7f3b30005f10 comp rx=0 tx=0).stop 2024-04-02T14:50:13.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.577+0000 7f3b463f3740 1 -- 172.21.15.90:0/3241792015 shutdown_connections 2024-04-02T14:50:13.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.577+0000 7f3b463f3740 1 --2- 172.21.15.90:0/3241792015 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x55da24bb2d40 0x55da24bb5130 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:13.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.577+0000 7f3b463f3740 1 --2- 172.21.15.90:0/3241792015 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55da24b3c230 0x55da24b4dc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:13.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.577+0000 7f3b463f3740 1 --2- 172.21.15.90:0/3241792015 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55da24b33ba0 0x55da24b4d730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:13.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.577+0000 7f3b463f3740 1 --2- 172.21.15.90:0/3241792015 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55da24b32ec0 0x55da24b4d1f0 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:13.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.577+0000 7f3b22a00640 0 client.18894 ms_handle_reset on v2:172.21.15.177:6838/664895663 2024-04-02T14:50:13.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.577+0000 7f3b463f3740 1 -- 172.21.15.90:0/3241792015 >> 172.21.15.90:0/3241792015 conn(0x55da24af9a50 msgr2=0x55da24b38220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:50:13.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.577+0000 7f3b463f3740 1 -- 172.21.15.90:0/3241792015 shutdown_connections 2024-04-02T14:50:13.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.578+0000 7f3b463f3740 1 -- 172.21.15.90:0/3241792015 wait complete. 2024-04-02T14:50:13.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.579+0000 7f3b463f3740 20 client.18894 trim_cache size 0 max 16384 2024-04-02T14:50:13.572 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-02T14:50:13.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.580+0000 7f3b42200640 1 -- 172.21.15.90:0/1913535589 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f3b240a3fa0 msgr2=0x7f3b240a6460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:13.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.580+0000 7f3b42200640 1 --2- 172.21.15.90:0/1913535589 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f3b240a3fa0 0x7f3b240a6460 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f3b34005e30 tx=0x7f3b340099d0 comp rx=0 tx=0).stop 2024-04-02T14:50:13.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.580+0000 7f3b42200640 1 -- 172.21.15.90:0/1913535589 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3b3c103df0 msgr2=0x7f3b3c19a490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:13.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.580+0000 7f3b42200640 1 --2- 172.21.15.90:0/1913535589 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3b3c103df0 0x7f3b3c19a490 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7f3b3000e9f0 tx=0x7f3b3000eeb0 comp rx=0 tx=0).stop 2024-04-02T14:50:13.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.580+0000 7f3b42200640 1 -- 172.21.15.90:0/1913535589 shutdown_connections 2024-04-02T14:50:13.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.582+0000 7f3b42200640 1 --2- 172.21.15.90:0/1913535589 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f3b240a3fa0 0x7f3b240a6460 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:13.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.582+0000 7f3b42200640 1 --2- 172.21.15.90:0/1913535589 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3b3c19ed50 0x7f3b3c19f200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:13.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.582+0000 7f3b42200640 1 --2- 172.21.15.90:0/1913535589 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3b3c108ec0 0x7f3b3c19a9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:13.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.582+0000 7f3b42200640 1 --2- 172.21.15.90:0/1913535589 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3b3c103df0 0x7f3b3c19a490 unknown :-1 s=CLOSED pgs=1021 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:13.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.582+0000 7f3b42200640 1 -- 172.21.15.90:0/1913535589 >> 172.21.15.90:0/1913535589 conn(0x7f3b3c0ff610 msgr2=0x7f3b3c101940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:50:13.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.582+0000 7f3b42200640 1 -- 172.21.15.90:0/1913535589 shutdown_connections 2024-04-02T14:50:13.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.582+0000 7f3b42200640 1 -- 172.21.15.90:0/1913535589 wait complete. 2024-04-02T14:50:13.601 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-02T14:50:13.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.856+0000 7f5e40000640 1 -- 172.21.15.90:0/332756218 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5e38073eb0 msgr2=0x7f5e38074290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:13.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.856+0000 7f5e40000640 1 --2- 172.21.15.90:0/332756218 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5e38073eb0 0x7f5e38074290 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7f5e2c009900 tx=0x7f5e2c02bcc0 comp rx=0 tx=0).stop 2024-04-02T14:50:13.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.856+0000 7f5e40000640 1 -- 172.21.15.90:0/332756218 shutdown_connections 2024-04-02T14:50:13.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.856+0000 7f5e40000640 1 --2- 172.21.15.90:0/332756218 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5e3810e350 0x7f5e38110740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:13.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.856+0000 7f5e40000640 1 --2- 172.21.15.90:0/332756218 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5e380747d0 0x7f5e3810de10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:13.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.856+0000 7f5e40000640 1 --2- 172.21.15.90:0/332756218 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5e38073eb0 0x7f5e38074290 unknown :-1 s=CLOSED pgs=1023 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:13.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.856+0000 7f5e40000640 1 -- 172.21.15.90:0/332756218 >> 172.21.15.90:0/332756218 conn(0x7f5e38078f00 msgr2=0x7f5e38079310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:50:13.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.856+0000 7f5e40000640 1 -- 172.21.15.90:0/332756218 shutdown_connections 2024-04-02T14:50:13.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.856+0000 7f5e40000640 1 -- 172.21.15.90:0/332756218 wait complete. 2024-04-02T14:50:13.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.857+0000 7f5e40000640 1 Processor -- start 2024-04-02T14:50:13.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.858+0000 7f5e40000640 1 -- start start 2024-04-02T14:50:13.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.858+0000 7f5e40000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5e38073eb0 0x7f5e3819a610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:13.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.858+0000 7f5e40000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5e380747d0 0x7f5e3819ab50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:13.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.858+0000 7f5e40000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5e3810e350 0x7f5e3819ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:13.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.858+0000 7f5e40000640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f5e38114030 con 0x7f5e380747d0 2024-04-02T14:50:13.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.858+0000 7f5e40000640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f5e38113eb0 con 0x7f5e3810e350 2024-04-02T14:50:13.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.858+0000 7f5e40000640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f5e381141b0 con 0x7f5e38073eb0 2024-04-02T14:50:13.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.858+0000 7f5e3e000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5e3810e350 0x7f5e3819ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:13.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.858+0000 7f5e3e000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5e3810e350 0x7f5e3819ee80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:47898/0 (socket says 172.21.15.90:47898) 2024-04-02T14:50:13.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.858+0000 7f5e3e000640 1 -- 172.21.15.90:0/1055319694 learned_addr learned my addr 172.21.15.90:0/1055319694 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:50:13.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.859+0000 7f5e3cc00640 1 --2- 172.21.15.90:0/1055319694 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5e380747d0 0x7f5e3819ab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:13.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.859+0000 7f5e3d600640 1 --2- 172.21.15.90:0/1055319694 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5e38073eb0 0x7f5e3819a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:13.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.859+0000 7f5e3e000640 1 -- 172.21.15.90:0/1055319694 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5e38073eb0 msgr2=0x7f5e3819a610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:13.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.859+0000 7f5e3e000640 1 --2- 172.21.15.90:0/1055319694 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5e38073eb0 0x7f5e3819a610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:13.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.859+0000 7f5e3e000640 1 -- 172.21.15.90:0/1055319694 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5e380747d0 msgr2=0x7f5e3819ab50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:13.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.859+0000 7f5e3e000640 1 --2- 172.21.15.90:0/1055319694 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5e380747d0 0x7f5e3819ab50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:13.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.859+0000 7f5e3e000640 1 -- 172.21.15.90:0/1055319694 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e3819f5b0 con 0x7f5e3810e350 2024-04-02T14:50:13.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.859+0000 7f5e3e000640 1 --2- 172.21.15.90:0/1055319694 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5e3810e350 0x7f5e3819ee80 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7f5e3000eb10 tx=0x7f5e3000c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:50:13.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.859+0000 7f5e36000640 1 -- 172.21.15.90:0/1055319694 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e30004b20 con 0x7f5e3810e350 2024-04-02T14:50:13.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.859+0000 7f5e40000640 1 -- 172.21.15.90:0/1055319694 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5e3819f8a0 con 0x7f5e3810e350 2024-04-02T14:50:13.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.859+0000 7f5e36000640 1 -- 172.21.15.90:0/1055319694 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5e30004cc0 con 0x7f5e3810e350 2024-04-02T14:50:13.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.859+0000 7f5e36000640 1 -- 172.21.15.90:0/1055319694 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e30010760 con 0x7f5e3810e350 2024-04-02T14:50:13.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.859+0000 7f5e40000640 1 -- 172.21.15.90:0/1055319694 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5e381a7130 con 0x7f5e3810e350 2024-04-02T14:50:13.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.864+0000 7f5e36000640 1 -- 172.21.15.90:0/1055319694 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f5e30005020 con 0x7f5e3810e350 2024-04-02T14:50:13.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.866+0000 7f5e36000640 1 --2- 172.21.15.90:0/1055319694 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f5e200a3da0 0x7f5e200a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:13.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.866+0000 7f5e36000640 1 -- 172.21.15.90:0/1055319694 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 8594+0+0 (secure 0 0 0) 0x7f5e300d0db0 con 0x7f5e3810e350 2024-04-02T14:50:13.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.866+0000 7f5e3d600640 1 --2- 172.21.15.90:0/1055319694 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f5e200a3da0 0x7f5e200a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:13.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.867+0000 7f5e3d600640 1 --2- 172.21.15.90:0/1055319694 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f5e200a3da0 0x7f5e200a6260 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f5e2c009230 tx=0x7f5e2c02df90 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:50:13.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.867+0000 7f5e42ddb740 1 Processor -- start 2024-04-02T14:50:13.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.867+0000 7f5e42ddb740 1 -- start start 2024-04-02T14:50:13.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.868+0000 7f5e42ddb740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560a12bb29f0 0x560a12bb1740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:13.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.868+0000 7f5e42ddb740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560a12bb1d10 0x560a12bb2190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:13.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.868+0000 7f5e42ddb740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560a12bbb080 0x560a12bbd470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:13.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.868+0000 7f5e42ddb740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x560a12af2400 con 0x560a12bb29f0 2024-04-02T14:50:13.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.868+0000 7f5e42ddb740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x560a12ad6f10 con 0x560a12bbb080 2024-04-02T14:50:13.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.868+0000 7f5e42ddb740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x560a12b166f0 con 0x560a12bb1d10 2024-04-02T14:50:13.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.868+0000 7f5e3e000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560a12bbb080 0x560a12bbd470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:13.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.868+0000 7f5e3d600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560a12bb29f0 0x560a12bb1740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:13.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.868+0000 7f5e3e000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560a12bbb080 0x560a12bbd470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:47902/0 (socket says 172.21.15.90:47902) 2024-04-02T14:50:13.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.868+0000 7f5e3e000640 1 -- 172.21.15.90:0/1479611108 learned_addr learned my addr 172.21.15.90:0/1479611108 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:50:13.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.868+0000 7f5e3cc00640 1 --2- 172.21.15.90:0/1479611108 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560a12bb1d10 0x560a12bb2190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:13.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.869+0000 7f5e3e000640 1 -- 172.21.15.90:0/1479611108 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560a12bb1d10 msgr2=0x560a12bb2190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:50:13.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.869+0000 7f5e3e000640 1 --2- 172.21.15.90:0/1479611108 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560a12bb1d10 0x560a12bb2190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:13.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.869+0000 7f5e3e000640 1 -- 172.21.15.90:0/1479611108 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560a12bb29f0 msgr2=0x560a12bb1740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:50:13.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.869+0000 7f5e3e000640 1 --2- 172.21.15.90:0/1479611108 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560a12bb29f0 0x560a12bb1740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:13.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.869+0000 7f5e3e000640 1 -- 172.21.15.90:0/1479611108 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560a1282a3f0 con 0x560a12bbb080 2024-04-02T14:50:13.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.869+0000 7f5e3e000640 1 --2- 172.21.15.90:0/1479611108 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560a12bbb080 0x560a12bbd470 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7f5e300a53e0 tx=0x7f5e3009c7e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1a7297594892e927 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:50:13.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.869+0000 7f5e2a600640 1 -- 172.21.15.90:0/1479611108 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e300bbe60 con 0x560a12bbb080 2024-04-02T14:50:13.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.869+0000 7f5e2a600640 1 -- 172.21.15.90:0/1479611108 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5e3009cca0 con 0x560a12bbb080 2024-04-02T14:50:13.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.869+0000 7f5e2a600640 1 -- 172.21.15.90:0/1479611108 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e300994a0 con 0x560a12bbb080 2024-04-02T14:50:13.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.870+0000 7f5e42ddb740 1 -- 172.21.15.90:0/1479611108 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560a12bbb080 msgr2=0x560a12bbd470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:13.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.870+0000 7f5e42ddb740 1 --2- 172.21.15.90:0/1479611108 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560a12bbb080 0x560a12bbd470 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7f5e300a53e0 tx=0x7f5e3009c7e0 comp rx=0 tx=0).stop 2024-04-02T14:50:13.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.870+0000 7f5e42ddb740 1 -- 172.21.15.90:0/1479611108 shutdown_connections 2024-04-02T14:50:13.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.870+0000 7f5e42ddb740 1 --2- 172.21.15.90:0/1479611108 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560a12bbb080 0x560a12bbd470 unknown :-1 s=CLOSED pgs=1025 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:13.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.870+0000 7f5e42ddb740 1 --2- 172.21.15.90:0/1479611108 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560a12bb1d10 0x560a12bb2190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:13.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.870+0000 7f5e42ddb740 1 --2- 172.21.15.90:0/1479611108 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560a12bb29f0 0x560a12bb1740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:13.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.870+0000 7f5e42ddb740 1 -- 172.21.15.90:0/1479611108 >> 172.21.15.90:0/1479611108 conn(0x560a12b78a50 msgr2=0x560a12bac760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:50:13.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.870+0000 7f5e42ddb740 1 -- 172.21.15.90:0/1479611108 shutdown_connections 2024-04-02T14:50:13.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.870+0000 7f5e42ddb740 1 -- 172.21.15.90:0/1479611108 wait complete. 2024-04-02T14:50:13.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.871+0000 7f5e42ddb740 1 -- start start 2024-04-02T14:50:13.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.871+0000 7f5e42ddb740 1 Processor -- start 2024-04-02T14:50:13.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.871+0000 7f5e42ddb740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560a12bb1d10 0x560a12bb8110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:13.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.871+0000 7f5e42ddb740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560a12bb29f0 0x560a12bb8650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:13.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.871+0000 7f5e42ddb740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560a12bbb080 0x560a12bb8b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:13.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.871+0000 7f5e42ddb740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x560a12b126a0 con 0x560a12bb29f0 2024-04-02T14:50:13.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.871+0000 7f5e42ddb740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x560a12abe980 con 0x560a12bb1d10 2024-04-02T14:50:13.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.871+0000 7f5e42ddb740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x560a12ad1cb0 con 0x560a12bbb080 2024-04-02T14:50:13.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.871+0000 7f5e42ddb740 10 client.0 _ll_register_callbacks cb 0x560a12ad16a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:50:13.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.872+0000 7f5e3d600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560a12bb1d10 0x560a12bb8110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:13.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.872+0000 7f5e3d600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560a12bb1d10 0x560a12bb8110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:47918/0 (socket says 172.21.15.90:47918) 2024-04-02T14:50:13.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.872+0000 7f5e3d600640 1 -- 172.21.15.90:0/1555096199 learned_addr learned my addr 172.21.15.90:0/1555096199 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:50:13.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.872+0000 7f5e3cc00640 1 --2- 172.21.15.90:0/1555096199 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560a12bb29f0 0x560a12bb8650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:13.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.872+0000 7f5e3e000640 1 --2- 172.21.15.90:0/1555096199 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560a12bbb080 0x560a12bb8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:13.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.872+0000 7f5e3cc00640 1 -- 172.21.15.90:0/1555096199 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560a12bbb080 msgr2=0x560a12bb8b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:50:13.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.872+0000 7f5e3cc00640 1 --2- 172.21.15.90:0/1555096199 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560a12bbb080 0x560a12bb8b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:13.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.872+0000 7f5e3cc00640 1 -- 172.21.15.90:0/1555096199 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560a12bb1d10 msgr2=0x560a12bb8110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:50:13.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.872+0000 7f5e3cc00640 1 --2- 172.21.15.90:0/1555096199 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560a12bb1d10 0x560a12bb8110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:13.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.872+0000 7f5e3cc00640 1 -- 172.21.15.90:0/1555096199 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560a128f47d0 con 0x560a12bb29f0 2024-04-02T14:50:13.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.872+0000 7f5e3d600640 1 --2- 172.21.15.90:0/1555096199 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560a12bb1d10 0x560a12bb8110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:50:13.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.872+0000 7f5e3e000640 1 --2- 172.21.15.90:0/1555096199 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560a12bbb080 0x560a12bb8b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:50:13.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.872+0000 7f5e3cc00640 1 --2- 172.21.15.90:0/1555096199 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560a12bb29f0 0x560a12bb8650 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f5e38110110 tx=0x7f5e24012b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=933a7a6f7b90d54b server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:50:13.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.872+0000 7f5e1ea00640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-02T14:50:13.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.872+0000 7f5e1ea00640 1 -- 172.21.15.90:0/1555096199 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e2400f8c0 con 0x560a12bb29f0 2024-04-02T14:50:13.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.872+0000 7f5e1ea00640 1 -- 172.21.15.90:0/1555096199 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 8594+0+0 (secure 0 0 0) 0x7f5e2400fa60 con 0x560a12bb29f0 2024-04-02T14:50:13.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.873+0000 7f5e42ddb740 1 -- 172.21.15.90:0/1555096199 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5e3009cca0 con 0x560a12bb29f0 2024-04-02T14:50:13.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.873+0000 7f5e1ea00640 1 -- 172.21.15.90:0/1555096199 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f5e2400fc60 con 0x560a12bb29f0 2024-04-02T14:50:13.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.873+0000 7f5e42ddb740 10 client.18906 fetch_fsmap learned FSMap version 27 2024-04-02T14:50:13.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.873+0000 7f5e42ddb740 1 -- 172.21.15.90:0/1555096199 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x560a12bca030 con 0x560a12bb29f0 2024-04-02T14:50:13.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.873+0000 7f5e1ea00640 1 -- 172.21.15.90:0/1555096199 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5e3009cca0 con 0x560a12bb29f0 2024-04-02T14:50:13.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.873+0000 7f5e1ea00640 1 -- 172.21.15.90:0/1555096199 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f5e2401c420 con 0x560a12bb29f0 2024-04-02T14:50:13.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.873+0000 7f5e42ddb740 10 client.18906 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:50:13.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.873+0000 7f5e42ddb740 10 client.18906 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2024-04-02T14:50:13.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.873+0000 7f5e42ddb740 20 client.18906 populate_metadata read hostname 'smithi090' 2024-04-02T14:50:13.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.873+0000 7f5e42ddb740 1 --2- 172.21.15.90:0/1555096199 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x560a12bcb5e0 0x560a12bcda00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:13.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.873+0000 7f5e42ddb740 4 client.18906 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:50:13.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.873+0000 7f5e42ddb740 1 -- 172.21.15.90:0/1555096199 --> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560a12ad6040 con 0x560a12bcb5e0 2024-04-02T14:50:13.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.874+0000 7f5e3d600640 1 --2- 172.21.15.90:0/1555096199 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x560a12bcb5e0 0x560a12bcda00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:13.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.874+0000 7f5e3d600640 1 --2- 172.21.15.90:0/1555096199 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x560a12bcb5e0 0x560a12bcda00 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=1eeb34afcbf3ac3e server_cookie=6aeb61055ab19202 in_seq=0 out_seq=0 2024-04-02T14:50:13.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.874+0000 7f5e1ea00640 10 client.18906 ms_handle_connect on v2:172.21.15.90:6838/1498915870 2024-04-02T14:50:13.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.875+0000 7f5e1ea00640 1 -- 172.21.15.90:0/1555096199 <== mds.1 v2:172.21.15.90:6838/1498915870 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x560a12ad6040 con 0x560a12bcb5e0 2024-04-02T14:50:13.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.875+0000 7f5e1ea00640 10 client.18906 handle_command_reply: tid=0 2024-04-02T14:50:13.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.875+0000 7f5e42ddb740 1 client.18906 shutdown 2024-04-02T14:50:13.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.875+0000 7f5e42ddb740 1 -- 172.21.15.90:0/1555096199 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560a12bb29f0 msgr2=0x560a12bb8650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:13.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.875+0000 7f5e42ddb740 1 --2- 172.21.15.90:0/1555096199 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560a12bb29f0 0x560a12bb8650 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f5e38110110 tx=0x7f5e24012b60 comp rx=0 tx=0).stop 2024-04-02T14:50:13.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.876+0000 7f5e42ddb740 1 -- 172.21.15.90:0/1555096199 shutdown_connections 2024-04-02T14:50:13.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.876+0000 7f5e42ddb740 1 --2- 172.21.15.90:0/1555096199 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x560a12bcb5e0 0x560a12bcda00 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:13.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.876+0000 7f5e42ddb740 1 --2- 172.21.15.90:0/1555096199 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560a12bbb080 0x560a12bb8b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:13.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.876+0000 7f5e42ddb740 1 --2- 172.21.15.90:0/1555096199 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560a12bb29f0 0x560a12bb8650 secure :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f5e38110110 tx=0x7f5e24012b60 comp rx=0 tx=0).stop 2024-04-02T14:50:13.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.876+0000 7f5e42ddb740 1 --2- 172.21.15.90:0/1555096199 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560a12bb1d10 0x560a12bb8110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:13.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.876+0000 7f5e1ea00640 0 client.18906 ms_handle_reset on v2:172.21.15.90:6838/1498915870 2024-04-02T14:50:13.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.876+0000 7f5e42ddb740 1 -- 172.21.15.90:0/1555096199 >> 172.21.15.90:0/1555096199 conn(0x560a12b78a50 msgr2=0x560a12bae4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:50:13.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.876+0000 7f5e42ddb740 1 -- 172.21.15.90:0/1555096199 shutdown_connections 2024-04-02T14:50:13.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.876+0000 7f5e42ddb740 1 -- 172.21.15.90:0/1555096199 wait complete. 2024-04-02T14:50:13.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.876+0000 7f5e42ddb740 20 client.18906 trim_cache size 0 max 16384 2024-04-02T14:50:13.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.896+0000 7f5e42ddb740 1 Processor -- start 2024-04-02T14:50:13.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.897+0000 7f5e42ddb740 1 -- start start 2024-04-02T14:50:13.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.897+0000 7f5e42ddb740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560a12bb1d10 0x560a12bc6af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:13.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.897+0000 7f5e42ddb740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560a12bbb080 0x560a12bab720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:13.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.897+0000 7f5e42ddb740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560a12bd32d0 0x560a12bd3710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:13.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.897+0000 7f5e42ddb740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f5e381afb60 con 0x560a12bd32d0 2024-04-02T14:50:13.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.897+0000 7f5e42ddb740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f5e381b16a0 con 0x560a12bb1d10 2024-04-02T14:50:13.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.897+0000 7f5e42ddb740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f5e381b0230 con 0x560a12bbb080 2024-04-02T14:50:13.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.897+0000 7f5e3cc00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560a12bbb080 0x560a12bab720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:13.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.897+0000 7f5e3d600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560a12bb1d10 0x560a12bc6af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:13.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.897+0000 7f5e3cc00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560a12bbb080 0x560a12bab720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:55124/0 (socket says 172.21.15.90:55124) 2024-04-02T14:50:13.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.897+0000 7f5e3cc00640 1 -- 172.21.15.90:0/3994075616 learned_addr learned my addr 172.21.15.90:0/3994075616 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:50:13.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.897+0000 7f5e3e000640 1 --2- 172.21.15.90:0/3994075616 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560a12bd32d0 0x560a12bd3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:13.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.897+0000 7f5e3cc00640 1 -- 172.21.15.90:0/3994075616 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560a12bb1d10 msgr2=0x560a12bc6af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:50:13.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.897+0000 7f5e3cc00640 1 --2- 172.21.15.90:0/3994075616 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560a12bb1d10 0x560a12bc6af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:13.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.898+0000 7f5e3cc00640 1 -- 172.21.15.90:0/3994075616 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560a12bd32d0 msgr2=0x560a12bd3710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:50:13.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.898+0000 7f5e3cc00640 1 --2- 172.21.15.90:0/3994075616 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560a12bd32d0 0x560a12bd3710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:13.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.898+0000 7f5e3cc00640 1 -- 172.21.15.90:0/3994075616 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560a12bb9200 con 0x560a12bbb080 2024-04-02T14:50:13.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.898+0000 7f5e3d600640 1 --2- 172.21.15.90:0/3994075616 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560a12bb1d10 0x560a12bc6af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:50:13.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.898+0000 7f5e3e000640 1 --2- 172.21.15.90:0/3994075616 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560a12bd32d0 0x560a12bd3710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:50:13.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.898+0000 7f5e3cc00640 1 --2- 172.21.15.90:0/3994075616 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560a12bbb080 0x560a12bab720 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f5e2401ca30 tx=0x7f5e2400ff90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3125457bc7553f8e server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:50:13.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.898+0000 7f5e29c00640 1 -- 172.21.15.90:0/3994075616 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e2401cbb0 con 0x560a12bbb080 2024-04-02T14:50:13.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.898+0000 7f5e29c00640 1 -- 172.21.15.90:0/3994075616 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5e240055b0 con 0x560a12bbb080 2024-04-02T14:50:13.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.898+0000 7f5e29c00640 1 -- 172.21.15.90:0/3994075616 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e24003870 con 0x560a12bbb080 2024-04-02T14:50:13.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.899+0000 7f5e42ddb740 1 -- 172.21.15.90:0/3994075616 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560a12bbb080 msgr2=0x560a12bab720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:13.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.899+0000 7f5e42ddb740 1 --2- 172.21.15.90:0/3994075616 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560a12bbb080 0x560a12bab720 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f5e2401ca30 tx=0x7f5e2400ff90 comp rx=0 tx=0).stop 2024-04-02T14:50:13.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.899+0000 7f5e42ddb740 1 -- 172.21.15.90:0/3994075616 shutdown_connections 2024-04-02T14:50:13.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.899+0000 7f5e42ddb740 1 --2- 172.21.15.90:0/3994075616 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560a12bd32d0 0x560a12bd3710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:13.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.899+0000 7f5e42ddb740 1 --2- 172.21.15.90:0/3994075616 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560a12bbb080 0x560a12bab720 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:13.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.899+0000 7f5e42ddb740 1 --2- 172.21.15.90:0/3994075616 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560a12bb1d10 0x560a12bc6af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:13.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.899+0000 7f5e42ddb740 1 -- 172.21.15.90:0/3994075616 >> 172.21.15.90:0/3994075616 conn(0x560a12b78a50 msgr2=0x560a12bab010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:50:13.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.899+0000 7f5e42ddb740 1 -- 172.21.15.90:0/3994075616 shutdown_connections 2024-04-02T14:50:13.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.899+0000 7f5e42ddb740 1 -- 172.21.15.90:0/3994075616 wait complete. 2024-04-02T14:50:13.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.899+0000 7f5e42ddb740 1 -- start start 2024-04-02T14:50:13.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.900+0000 7f5e42ddb740 1 Processor -- start 2024-04-02T14:50:13.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.900+0000 7f5e42ddb740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560a12bb1d10 0x560a12bbe920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:13.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.900+0000 7f5e42ddb740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560a12bbb080 0x560a12bbee60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:13.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.900+0000 7f5e42ddb740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560a12bd32d0 0x560a12bbf3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:13.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.900+0000 7f5e42ddb740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f5e381b0fd0 con 0x560a12bbb080 2024-04-02T14:50:13.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.900+0000 7f5e42ddb740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f5e381b0900 con 0x560a12bb1d10 2024-04-02T14:50:13.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.900+0000 7f5e42ddb740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f5e381adb60 con 0x560a12bd32d0 2024-04-02T14:50:13.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.900+0000 7f5e42ddb740 10 client.0 _ll_register_callbacks cb 0x7f5e24009ec0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:50:13.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.901+0000 7f5e3e000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560a12bd32d0 0x560a12bbf3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:13.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.901+0000 7f5e3cc00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560a12bbb080 0x560a12bbee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:13.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.901+0000 7f5e3e000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560a12bd32d0 0x560a12bbf3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:55140/0 (socket says 172.21.15.90:55140) 2024-04-02T14:50:13.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.901+0000 7f5e3e000640 1 -- 172.21.15.90:0/1404804849 learned_addr learned my addr 172.21.15.90:0/1404804849 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:50:13.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.901+0000 7f5e3d600640 1 --2- 172.21.15.90:0/1404804849 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560a12bb1d10 0x560a12bbe920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:13.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.902+0000 7f5e3d600640 1 -- 172.21.15.90:0/1404804849 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560a12bd32d0 msgr2=0x560a12bbf3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:50:13.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.902+0000 7f5e3d600640 1 --2- 172.21.15.90:0/1404804849 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560a12bd32d0 0x560a12bbf3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:13.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.902+0000 7f5e3d600640 1 -- 172.21.15.90:0/1404804849 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560a12bbb080 msgr2=0x560a12bbee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:50:13.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.902+0000 7f5e3d600640 1 --2- 172.21.15.90:0/1404804849 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560a12bbb080 0x560a12bbee60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:13.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.902+0000 7f5e3d600640 1 -- 172.21.15.90:0/1404804849 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560a12c39350 con 0x560a12bb1d10 2024-04-02T14:50:13.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.902+0000 7f5e3cc00640 1 --2- 172.21.15.90:0/1404804849 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560a12bbb080 0x560a12bbee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:50:13.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.903+0000 7f5e3d600640 1 --2- 172.21.15.90:0/1404804849 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560a12bb1d10 0x560a12bbe920 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x560a12bce270 tx=0x7f5e2c02bcc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5511516ff3af383b server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:50:13.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.903+0000 7f5e1ea00640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:50:13.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.903+0000 7f5e1ea00640 1 -- 172.21.15.90:0/1404804849 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e2c05b070 con 0x560a12bb1d10 2024-04-02T14:50:13.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.903+0000 7f5e1ea00640 1 -- 172.21.15.90:0/1404804849 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 8594+0+0 (secure 0 0 0) 0x7f5e2c0575b0 con 0x560a12bb1d10 2024-04-02T14:50:13.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.903+0000 7f5e42ddb740 1 -- 172.21.15.90:0/1404804849 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5e240055b0 con 0x560a12bb1d10 2024-04-02T14:50:13.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.904+0000 7f5e1ea00640 1 -- 172.21.15.90:0/1404804849 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f5e2c0044c0 con 0x560a12bb1d10 2024-04-02T14:50:13.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.904+0000 7f5e42ddb740 10 client.30230 fetch_fsmap learned FSMap version 27 2024-04-02T14:50:13.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.904+0000 7f5e42ddb740 1 -- 172.21.15.90:0/1404804849 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x560a12c3a550 con 0x560a12bb1d10 2024-04-02T14:50:13.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.904+0000 7f5e1ea00640 1 -- 172.21.15.90:0/1404804849 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5e240055b0 con 0x560a12bb1d10 2024-04-02T14:50:13.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.904+0000 7f5e1ea00640 1 -- 172.21.15.90:0/1404804849 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f5e2c04b040 con 0x560a12bb1d10 2024-04-02T14:50:13.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.904+0000 7f5e42ddb740 10 client.30230 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:50:13.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.904+0000 7f5e42ddb740 10 client.30230 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2024-04-02T14:50:13.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.904+0000 7f5e42ddb740 20 client.30230 populate_metadata read hostname 'smithi090' 2024-04-02T14:50:13.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.904+0000 7f5e42ddb740 1 --2- 172.21.15.90:0/1404804849 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x560a12c3acc0 0x560a12c3d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:13.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.904+0000 7f5e42ddb740 4 client.30230 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-02T14:50:13.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.904+0000 7f5e42ddb740 1 -- 172.21.15.90:0/1404804849 --> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] -- command(tid 0: {"prefix": "damage ls"}) -- 0x560a12b23c30 con 0x560a12c3acc0 2024-04-02T14:50:13.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.905+0000 7f5e3cc00640 1 --2- 172.21.15.90:0/1404804849 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x560a12c3acc0 0x560a12c3d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:13.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.905+0000 7f5e1ea00640 10 client.30230 ms_handle_connect on v2:172.21.15.90:6838/1498915870 2024-04-02T14:50:13.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.905+0000 7f5e3cc00640 1 --2- 172.21.15.90:0/1404804849 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x560a12c3acc0 0x560a12c3d0b0 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=da9a5233a8afcfc1 server_cookie=b438956f8cd606c8 in_seq=0 out_seq=0 2024-04-02T14:50:13.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.906+0000 7f5e1ea00640 1 -- 172.21.15.90:0/1404804849 <== mds.1 v2:172.21.15.90:6838/1498915870 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x560a12b23c30 con 0x560a12c3acc0 2024-04-02T14:50:13.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.906+0000 7f5e1ea00640 10 client.30230 handle_command_reply: tid=0 2024-04-02T14:50:13.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.906+0000 7f5e42ddb740 1 client.30230 shutdown 2024-04-02T14:50:13.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.906+0000 7f5e42ddb740 1 -- 172.21.15.90:0/1404804849 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560a12bb1d10 msgr2=0x560a12bbe920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:13.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.906+0000 7f5e42ddb740 1 --2- 172.21.15.90:0/1404804849 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560a12bb1d10 0x560a12bbe920 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x560a12bce270 tx=0x7f5e2c02bcc0 comp rx=0 tx=0).stop 2024-04-02T14:50:13.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.906+0000 7f5e42ddb740 1 -- 172.21.15.90:0/1404804849 shutdown_connections 2024-04-02T14:50:13.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.906+0000 7f5e42ddb740 1 --2- 172.21.15.90:0/1404804849 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x560a12c3acc0 0x560a12c3d0b0 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:13.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.906+0000 7f5e42ddb740 1 --2- 172.21.15.90:0/1404804849 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560a12bd32d0 0x560a12bbf3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:13.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.906+0000 7f5e42ddb740 1 --2- 172.21.15.90:0/1404804849 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560a12bbb080 0x560a12bbee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:13.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.906+0000 7f5e1ea00640 0 client.30230 ms_handle_reset on v2:172.21.15.90:6838/1498915870 2024-04-02T14:50:13.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.906+0000 7f5e42ddb740 1 --2- 172.21.15.90:0/1404804849 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560a12bb1d10 0x560a12bbe920 unknown :-1 s=CLOSED pgs=1026 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:13.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.906+0000 7f5e42ddb740 1 -- 172.21.15.90:0/1404804849 >> 172.21.15.90:0/1404804849 conn(0x560a12b78a50 msgr2=0x560a12bd2b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:50:13.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.906+0000 7f5e42ddb740 1 -- 172.21.15.90:0/1404804849 shutdown_connections 2024-04-02T14:50:13.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.906+0000 7f5e42ddb740 1 -- 172.21.15.90:0/1404804849 wait complete. 2024-04-02T14:50:13.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.906+0000 7f5e42ddb740 20 client.30230 trim_cache size 0 max 16384 2024-04-02T14:50:13.901 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-02T14:50:13.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.907+0000 7f5e40000640 1 -- 172.21.15.90:0/1055319694 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f5e200a3da0 msgr2=0x7f5e200a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:13.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.907+0000 7f5e40000640 1 --2- 172.21.15.90:0/1055319694 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f5e200a3da0 0x7f5e200a6260 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f5e2c009230 tx=0x7f5e2c02df90 comp rx=0 tx=0).stop 2024-04-02T14:50:13.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.907+0000 7f5e40000640 1 -- 172.21.15.90:0/1055319694 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5e3810e350 msgr2=0x7f5e3819ee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:13.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.907+0000 7f5e40000640 1 --2- 172.21.15.90:0/1055319694 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5e3810e350 0x7f5e3819ee80 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7f5e3000eb10 tx=0x7f5e3000c2d0 comp rx=0 tx=0).stop 2024-04-02T14:50:13.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.907+0000 7f5e40000640 1 -- 172.21.15.90:0/1055319694 shutdown_connections 2024-04-02T14:50:13.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.907+0000 7f5e40000640 1 --2- 172.21.15.90:0/1055319694 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f5e200a3da0 0x7f5e200a6260 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:13.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.907+0000 7f5e40000640 1 --2- 172.21.15.90:0/1055319694 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5e3810e350 0x7f5e3819ee80 unknown :-1 s=CLOSED pgs=1024 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:13.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.907+0000 7f5e40000640 1 --2- 172.21.15.90:0/1055319694 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5e380747d0 0x7f5e3819ab50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:13.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.907+0000 7f5e40000640 1 --2- 172.21.15.90:0/1055319694 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5e38073eb0 0x7f5e3819a610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:13.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.907+0000 7f5e40000640 1 -- 172.21.15.90:0/1055319694 >> 172.21.15.90:0/1055319694 conn(0x7f5e38078f00 msgr2=0x7f5e380fe9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:50:13.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.908+0000 7f5e40000640 1 -- 172.21.15.90:0/1055319694 shutdown_connections 2024-04-02T14:50:13.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:13.908+0000 7f5e40000640 1 -- 172.21.15.90:0/1055319694 wait complete. 2024-04-02T14:50:13.928 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-02T14:50:13.928 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:50:13 smithi090 ceph-mon[31674]: pgmap v480: 97 pgs: 10 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 81 active+clean; 4.1 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 15 MiB/s wr, 1.37k op/s 2024-04-02T14:50:14.102 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:50:13 smithi092 ceph-mon[36497]: pgmap v480: 97 pgs: 10 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 81 active+clean; 4.1 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 15 MiB/s wr, 1.37k op/s 2024-04-02T14:50:14.178 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:50:13 smithi177 ceph-mon[36431]: pgmap v480: 97 pgs: 10 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 81 active+clean; 4.1 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 15 MiB/s wr, 1.37k op/s 2024-04-02T14:50:14.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.191+0000 7f458dc00640 1 -- 172.21.15.90:0/551939680 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4588108fe0 msgr2=0x7f4588110bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:14.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.191+0000 7f458dc00640 1 --2- 172.21.15.90:0/551939680 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4588108fe0 0x7f4588110bb0 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f4580009ca0 tx=0x7f458002c0f0 comp rx=0 tx=0).stop 2024-04-02T14:50:14.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.192+0000 7f458dc00640 1 -- 172.21.15.90:0/551939680 shutdown_connections 2024-04-02T14:50:14.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.192+0000 7f458dc00640 1 --2- 172.21.15.90:0/551939680 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4588108fe0 0x7f4588110bb0 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:14.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.192+0000 7f458dc00640 1 --2- 172.21.15.90:0/551939680 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f45881048c0 0x7f4588108900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:14.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.192+0000 7f458dc00640 1 --2- 172.21.15.90:0/551939680 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4588103f10 0x7f45881042f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:14.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.192+0000 7f458dc00640 1 -- 172.21.15.90:0/551939680 >> 172.21.15.90:0/551939680 conn(0x7f45880ff7b0 msgr2=0x7f4588101bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:50:14.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.193+0000 7f458dc00640 1 -- 172.21.15.90:0/551939680 shutdown_connections 2024-04-02T14:50:14.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.193+0000 7f458dc00640 1 -- 172.21.15.90:0/551939680 wait complete. 2024-04-02T14:50:14.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.194+0000 7f458dc00640 1 Processor -- start 2024-04-02T14:50:14.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.195+0000 7f458dc00640 1 -- start start 2024-04-02T14:50:14.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.196+0000 7f458dc00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4588103f10 0x7f4588073360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:14.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.196+0000 7f458dc00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4588108fe0 0x7f458806c430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:14.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.196+0000 7f458dc00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f458806c970 0x7f458806ce20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:14.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.196+0000 7f458dc00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f45881140f0 con 0x7f4588103f10 2024-04-02T14:50:14.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.196+0000 7f458dc00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f4588113f70 con 0x7f4588108fe0 2024-04-02T14:50:14.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.196+0000 7f458dc00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f4588114270 con 0x7f458806c970 2024-04-02T14:50:14.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.196+0000 7f4587400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f458806c970 0x7f458806ce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:14.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.196+0000 7f4587400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f458806c970 0x7f458806ce20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:55152/0 (socket says 172.21.15.90:55152) 2024-04-02T14:50:14.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.196+0000 7f4587400640 1 -- 172.21.15.90:0/440216130 learned_addr learned my addr 172.21.15.90:0/440216130 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:50:14.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.197+0000 7f4586a00640 1 --2- 172.21.15.90:0/440216130 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4588103f10 0x7f4588073360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:14.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.197+0000 7f4587400640 1 -- 172.21.15.90:0/440216130 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4588108fe0 msgr2=0x7f458806c430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:14.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.197+0000 7f4587400640 1 --2- 172.21.15.90:0/440216130 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4588108fe0 0x7f458806c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:14.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.197+0000 7f4587400640 1 -- 172.21.15.90:0/440216130 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4588103f10 msgr2=0x7f4588073360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:14.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.197+0000 7f4587400640 1 --2- 172.21.15.90:0/440216130 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4588103f10 0x7f4588073360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:14.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.197+0000 7f4587400640 1 -- 172.21.15.90:0/440216130 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4588070e80 con 0x7f458806c970 2024-04-02T14:50:14.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.197+0000 7f4587400640 1 --2- 172.21.15.90:0/440216130 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f458806c970 0x7f458806ce20 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f4580037040 tx=0x7f4580032ec0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:50:14.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.198+0000 7f457f400640 1 -- 172.21.15.90:0/440216130 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4580041020 con 0x7f458806c970 2024-04-02T14:50:14.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.198+0000 7f457f400640 1 -- 172.21.15.90:0/440216130 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4580034cd0 con 0x7f458806c970 2024-04-02T14:50:14.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.198+0000 7f457f400640 1 -- 172.21.15.90:0/440216130 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4580039040 con 0x7f458806c970 2024-04-02T14:50:14.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.198+0000 7f458dc00640 1 -- 172.21.15.90:0/440216130 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4588071110 con 0x7f458806c970 2024-04-02T14:50:14.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.198+0000 7f458dc00640 1 -- 172.21.15.90:0/440216130 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45880715b0 con 0x7f458806c970 2024-04-02T14:50:14.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.203+0000 7f457f400640 1 -- 172.21.15.90:0/440216130 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f4580044020 con 0x7f458806c970 2024-04-02T14:50:14.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.203+0000 7f4590a2b740 1 Processor -- start 2024-04-02T14:50:14.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.205+0000 7f457f400640 1 --2- 172.21.15.90:0/440216130 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f456c0a3f20 0x7f456c0a63e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:14.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.205+0000 7f457f400640 1 -- 172.21.15.90:0/440216130 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 8594+0+0 (secure 0 0 0) 0x7f45800f2470 con 0x7f458806c970 2024-04-02T14:50:14.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.205+0000 7f4586000640 1 --2- 172.21.15.90:0/440216130 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f456c0a3f20 0x7f456c0a63e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:14.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.206+0000 7f4590a2b740 1 -- start start 2024-04-02T14:50:14.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.207+0000 7f4590a2b740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5601303fc9f0 0x5601303fb740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:14.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.207+0000 7f4590a2b740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5601303fbd10 0x5601303fc190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:14.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.207+0000 7f4590a2b740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560130405080 0x560130407470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:14.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.207+0000 7f4590a2b740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x56013033c400 con 0x560130405080 2024-04-02T14:50:14.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.207+0000 7f4590a2b740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x560130320f10 con 0x5601303fbd10 2024-04-02T14:50:14.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.207+0000 7f4590a2b740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5601303606f0 con 0x5601303fc9f0 2024-04-02T14:50:14.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.207+0000 7f4586000640 1 --2- 172.21.15.90:0/440216130 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f456c0a3f20 0x7f456c0a63e0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f45780058c0 tx=0x7f4578005850 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:50:14.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.207+0000 7f4586a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5601303fc9f0 0x5601303fb740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:14.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.207+0000 7f4586a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5601303fc9f0 0x5601303fb740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:55168/0 (socket says 172.21.15.90:55168) 2024-04-02T14:50:14.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.207+0000 7f4586a00640 1 -- 172.21.15.90:0/1450782217 learned_addr learned my addr 172.21.15.90:0/1450782217 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:50:14.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.207+0000 7f4587400640 1 --2- 172.21.15.90:0/1450782217 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560130405080 0x560130407470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:14.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.208+0000 7f4586a00640 1 -- 172.21.15.90:0/1450782217 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5601303fbd10 msgr2=0x5601303fc190 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-02T14:50:14.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.208+0000 7f4586a00640 1 --2- 172.21.15.90:0/1450782217 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5601303fbd10 0x5601303fc190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:14.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.208+0000 7f4586a00640 1 -- 172.21.15.90:0/1450782217 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560130405080 msgr2=0x560130407470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:50:14.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.208+0000 7f4586a00640 1 --2- 172.21.15.90:0/1450782217 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560130405080 0x560130407470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:14.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.208+0000 7f4586a00640 1 -- 172.21.15.90:0/1450782217 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5601300743f0 con 0x5601303fc9f0 2024-04-02T14:50:14.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.208+0000 7f4587400640 1 --2- 172.21.15.90:0/1450782217 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560130405080 0x560130407470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:50:14.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.209+0000 7f4586a00640 1 --2- 172.21.15.90:0/1450782217 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5601303fc9f0 0x5601303fb740 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f457000a920 tx=0x7f457000ade0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cd3c4e8b2fa33830 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:50:14.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.210+0000 7f4577e00640 1 -- 172.21.15.90:0/1450782217 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4570015070 con 0x5601303fc9f0 2024-04-02T14:50:14.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.210+0000 7f4577e00640 1 -- 172.21.15.90:0/1450782217 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f457000cc00 con 0x5601303fc9f0 2024-04-02T14:50:14.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.210+0000 7f4577e00640 1 -- 172.21.15.90:0/1450782217 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45700105b0 con 0x5601303fc9f0 2024-04-02T14:50:14.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.211+0000 7f4590a2b740 1 -- 172.21.15.90:0/1450782217 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5601303fc9f0 msgr2=0x5601303fb740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:14.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.211+0000 7f4590a2b740 1 --2- 172.21.15.90:0/1450782217 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5601303fc9f0 0x5601303fb740 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f457000a920 tx=0x7f457000ade0 comp rx=0 tx=0).stop 2024-04-02T14:50:14.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.214+0000 7f4590a2b740 1 -- 172.21.15.90:0/1450782217 shutdown_connections 2024-04-02T14:50:14.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.214+0000 7f4590a2b740 1 --2- 172.21.15.90:0/1450782217 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560130405080 0x560130407470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:14.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.214+0000 7f4590a2b740 1 --2- 172.21.15.90:0/1450782217 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5601303fbd10 0x5601303fc190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:14.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.214+0000 7f4590a2b740 1 --2- 172.21.15.90:0/1450782217 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5601303fc9f0 0x5601303fb740 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:14.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.214+0000 7f4590a2b740 1 -- 172.21.15.90:0/1450782217 >> 172.21.15.90:0/1450782217 conn(0x5601303c2a50 msgr2=0x5601303f6760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:50:14.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.214+0000 7f4590a2b740 1 -- 172.21.15.90:0/1450782217 shutdown_connections 2024-04-02T14:50:14.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.214+0000 7f4590a2b740 1 -- 172.21.15.90:0/1450782217 wait complete. 2024-04-02T14:50:14.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.215+0000 7f4590a2b740 1 -- start start 2024-04-02T14:50:14.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.215+0000 7f4590a2b740 1 Processor -- start 2024-04-02T14:50:14.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.216+0000 7f4590a2b740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5601303fbd10 0x560130402130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:14.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.216+0000 7f4590a2b740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560130405080 0x560130402670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:14.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.216+0000 7f4590a2b740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560130402bb0 0x5601304110d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:14.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.216+0000 7f4590a2b740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x56013035c6a0 con 0x560130402bb0 2024-04-02T14:50:14.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.216+0000 7f4590a2b740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x560130308980 con 0x560130405080 2024-04-02T14:50:14.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.216+0000 7f4590a2b740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x56013031bcb0 con 0x5601303fbd10 2024-04-02T14:50:14.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.216+0000 7f4590a2b740 10 client.0 _ll_register_callbacks cb 0x56013031b6a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:50:14.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.216+0000 7f4586a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5601303fbd10 0x560130402130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:14.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.216+0000 7f4586a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5601303fbd10 0x560130402130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:55182/0 (socket says 172.21.15.90:55182) 2024-04-02T14:50:14.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.216+0000 7f4586a00640 1 -- 172.21.15.90:0/4246600992 learned_addr learned my addr 172.21.15.90:0/4246600992 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:50:14.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.216+0000 7f4586000640 1 --2- 172.21.15.90:0/4246600992 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560130405080 0x560130402670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:14.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.217+0000 7f4586a00640 1 -- 172.21.15.90:0/4246600992 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560130405080 msgr2=0x560130402670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:50:14.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.217+0000 7f4586a00640 1 --2- 172.21.15.90:0/4246600992 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560130405080 0x560130402670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:14.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.217+0000 7f4586a00640 1 -- 172.21.15.90:0/4246600992 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560130402bb0 msgr2=0x5601304110d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-02T14:50:14.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.217+0000 7f4586a00640 1 --2- 172.21.15.90:0/4246600992 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560130402bb0 0x5601304110d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:14.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.217+0000 7f4586a00640 1 -- 172.21.15.90:0/4246600992 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56013013e7d0 con 0x5601303fbd10 2024-04-02T14:50:14.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.217+0000 7f4586a00640 1 --2- 172.21.15.90:0/4246600992 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5601303fbd10 0x560130402130 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f4570008010 tx=0x7f4570006a90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=36542f4d488b0fee server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:50:14.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.217+0000 7f4574c00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:50:14.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.217+0000 7f4574c00640 1 -- 172.21.15.90:0/4246600992 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4570029060 con 0x5601303fbd10 2024-04-02T14:50:14.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.218+0000 7f4574c00640 1 -- 172.21.15.90:0/4246600992 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 8594+0+0 (secure 0 0 0) 0x7f457000a430 con 0x5601303fbd10 2024-04-02T14:50:14.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.218+0000 7f4590a2b740 1 -- 172.21.15.90:0/4246600992 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f457000cc00 con 0x5601303fbd10 2024-04-02T14:50:14.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.218+0000 7f4574c00640 1 -- 172.21.15.90:0/4246600992 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f4570015040 con 0x5601303fbd10 2024-04-02T14:50:14.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.219+0000 7f4590a2b740 10 client.30025 fetch_fsmap learned FSMap version 27 2024-04-02T14:50:14.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.219+0000 7f4590a2b740 1 -- 172.21.15.90:0/4246600992 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x560130414490 con 0x5601303fbd10 2024-04-02T14:50:14.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.219+0000 7f4574c00640 1 -- 172.21.15.90:0/4246600992 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f457000cc00 con 0x5601303fbd10 2024-04-02T14:50:14.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.220+0000 7f4574c00640 1 -- 172.21.15.90:0/4246600992 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f457002d3c0 con 0x5601303fbd10 2024-04-02T14:50:14.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.220+0000 7f4590a2b740 10 client.30025 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:50:14.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.220+0000 7f4590a2b740 10 client.30025 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:50:14.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.221+0000 7f4590a2b740 20 client.30025 populate_metadata read hostname 'smithi090' 2024-04-02T14:50:14.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.221+0000 7f4590a2b740 1 --2- 172.21.15.90:0/4246600992 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5601304157e0 0x560130417bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:14.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.221+0000 7f4590a2b740 4 client.30025 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:50:14.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.221+0000 7f4590a2b740 1 -- 172.21.15.90:0/4246600992 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560130320040 con 0x5601304157e0 2024-04-02T14:50:14.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.221+0000 7f4586000640 1 --2- 172.21.15.90:0/4246600992 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5601304157e0 0x560130417bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:14.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.223+0000 7f4574c00640 10 client.30025 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:50:14.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.223+0000 7f4586000640 1 --2- 172.21.15.90:0/4246600992 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5601304157e0 0x560130417bd0 crc :-1 s=READY pgs=386 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=816f5b18e5d02ee4 server_cookie=c93ae3dacda20b8f in_seq=0 out_seq=0 2024-04-02T14:50:14.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.225+0000 7f4574c00640 1 -- 172.21.15.90:0/4246600992 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x560130320040 con 0x5601304157e0 2024-04-02T14:50:14.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.225+0000 7f4574c00640 10 client.30025 handle_command_reply: tid=0 2024-04-02T14:50:14.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.225+0000 7f4590a2b740 1 client.30025 shutdown 2024-04-02T14:50:14.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.226+0000 7f4590a2b740 1 -- 172.21.15.90:0/4246600992 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5601303fbd10 msgr2=0x560130402130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:14.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.226+0000 7f4590a2b740 1 --2- 172.21.15.90:0/4246600992 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5601303fbd10 0x560130402130 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f4570008010 tx=0x7f4570006a90 comp rx=0 tx=0).stop 2024-04-02T14:50:14.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.227+0000 7f4590a2b740 1 -- 172.21.15.90:0/4246600992 shutdown_connections 2024-04-02T14:50:14.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.228+0000 7f4590a2b740 1 --2- 172.21.15.90:0/4246600992 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5601304157e0 0x560130417bd0 crc :-1 s=READY pgs=386 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:14.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.228+0000 7f4590a2b740 1 --2- 172.21.15.90:0/4246600992 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560130402bb0 0x5601304110d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:14.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.228+0000 7f4590a2b740 1 --2- 172.21.15.90:0/4246600992 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560130405080 0x560130402670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:14.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.228+0000 7f4590a2b740 1 --2- 172.21.15.90:0/4246600992 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5601303fbd10 0x560130402130 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:14.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.228+0000 7f4590a2b740 1 -- 172.21.15.90:0/4246600992 >> 172.21.15.90:0/4246600992 conn(0x5601303c2a50 msgr2=0x5601303f84b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:50:14.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.228+0000 7f4574c00640 0 client.30025 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:50:14.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.229+0000 7f4590a2b740 1 -- 172.21.15.90:0/4246600992 shutdown_connections 2024-04-02T14:50:14.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.230+0000 7f4590a2b740 1 -- 172.21.15.90:0/4246600992 wait complete. 2024-04-02T14:50:14.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.230+0000 7f4590a2b740 20 client.30025 trim_cache size 0 max 16384 2024-04-02T14:50:14.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.251+0000 7f4590a2b740 1 Processor -- start 2024-04-02T14:50:14.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.251+0000 7f4590a2b740 1 -- start start 2024-04-02T14:50:14.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.252+0000 7f4590a2b740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5601303fbd10 0x56013041d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:14.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.252+0000 7f4590a2b740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560130402bb0 0x56013041d710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:14.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.252+0000 7f4590a2b740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560130405080 0x560130410b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:14.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.252+0000 7f4590a2b740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f45881bc630 con 0x560130405080 2024-04-02T14:50:14.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.252+0000 7f4590a2b740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f45881be170 con 0x560130402bb0 2024-04-02T14:50:14.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.252+0000 7f4590a2b740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f45881bcd00 con 0x5601303fbd10 2024-04-02T14:50:14.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.252+0000 7f4586000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560130402bb0 0x56013041d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:14.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.252+0000 7f4586000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560130402bb0 0x56013041d710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:47970/0 (socket says 172.21.15.90:47970) 2024-04-02T14:50:14.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.252+0000 7f4586000640 1 -- 172.21.15.90:0/987472870 learned_addr learned my addr 172.21.15.90:0/987472870 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:50:14.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.252+0000 7f4586a00640 1 --2- 172.21.15.90:0/987472870 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5601303fbd10 0x56013041d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:14.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.252+0000 7f4587400640 1 --2- 172.21.15.90:0/987472870 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560130405080 0x560130410b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:14.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.253+0000 7f4586000640 1 -- 172.21.15.90:0/987472870 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5601303fbd10 msgr2=0x56013041d1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:50:14.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.253+0000 7f4586000640 1 --2- 172.21.15.90:0/987472870 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5601303fbd10 0x56013041d1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:14.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.253+0000 7f4586000640 1 -- 172.21.15.90:0/987472870 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560130405080 msgr2=0x560130410b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:50:14.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.253+0000 7f4586000640 1 --2- 172.21.15.90:0/987472870 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560130405080 0x560130410b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:14.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.253+0000 7f4586000640 1 -- 172.21.15.90:0/987472870 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5601303fad40 con 0x560130402bb0 2024-04-02T14:50:14.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.253+0000 7f4586000640 1 --2- 172.21.15.90:0/987472870 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560130402bb0 0x56013041d710 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7f458806dc30 tx=0x7f4578015000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d69ace4c4054343c server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:50:14.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.254+0000 7f4577400640 1 -- 172.21.15.90:0/987472870 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f457800ee80 con 0x560130402bb0 2024-04-02T14:50:14.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.254+0000 7f4577400640 1 -- 172.21.15.90:0/987472870 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f457802b070 con 0x560130402bb0 2024-04-02T14:50:14.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.254+0000 7f4577400640 1 -- 172.21.15.90:0/987472870 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f457801c680 con 0x560130402bb0 2024-04-02T14:50:14.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.254+0000 7f4590a2b740 1 -- 172.21.15.90:0/987472870 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560130402bb0 msgr2=0x56013041d710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:14.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.254+0000 7f4590a2b740 1 --2- 172.21.15.90:0/987472870 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560130402bb0 0x56013041d710 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7f458806dc30 tx=0x7f4578015000 comp rx=0 tx=0).stop 2024-04-02T14:50:14.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.255+0000 7f4590a2b740 1 -- 172.21.15.90:0/987472870 shutdown_connections 2024-04-02T14:50:14.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.255+0000 7f4590a2b740 1 --2- 172.21.15.90:0/987472870 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560130405080 0x560130410b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:14.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.255+0000 7f4590a2b740 1 --2- 172.21.15.90:0/987472870 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560130402bb0 0x56013041d710 unknown :-1 s=CLOSED pgs=1027 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:14.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.255+0000 7f4590a2b740 1 --2- 172.21.15.90:0/987472870 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5601303fbd10 0x56013041d1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:14.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.255+0000 7f4590a2b740 1 -- 172.21.15.90:0/987472870 >> 172.21.15.90:0/987472870 conn(0x5601303c2a50 msgr2=0x5601303f6290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:50:14.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.255+0000 7f4590a2b740 1 -- 172.21.15.90:0/987472870 shutdown_connections 2024-04-02T14:50:14.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.255+0000 7f4590a2b740 1 -- 172.21.15.90:0/987472870 wait complete. 2024-04-02T14:50:14.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.255+0000 7f4590a2b740 1 -- start start 2024-04-02T14:50:14.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.255+0000 7f4590a2b740 1 Processor -- start 2024-04-02T14:50:14.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.256+0000 7f4590a2b740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5601303fbd10 0x560130406bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:14.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.256+0000 7f4590a2b740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560130402bb0 0x560130405460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:14.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.256+0000 7f4590a2b740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560130405080 0x5601304059a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:14.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.256+0000 7f4590a2b740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f45881bdaa0 con 0x5601303fbd10 2024-04-02T14:50:14.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.256+0000 7f4590a2b740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f45881bd3d0 con 0x560130405080 2024-04-02T14:50:14.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.256+0000 7f4590a2b740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f458807a0d0 con 0x560130402bb0 2024-04-02T14:50:14.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.256+0000 7f4590a2b740 10 client.0 _ll_register_callbacks cb 0x7f45700117f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:50:14.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.256+0000 7f4586000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560130402bb0 0x560130405460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:14.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.256+0000 7f4586000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560130402bb0 0x560130405460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:55196/0 (socket says 172.21.15.90:55196) 2024-04-02T14:50:14.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.256+0000 7f4587400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560130405080 0x5601304059a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:14.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.256+0000 7f4586000640 1 -- 172.21.15.90:0/3401600180 learned_addr learned my addr 172.21.15.90:0/3401600180 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:50:14.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.256+0000 7f4586a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5601303fbd10 0x560130406bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:14.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.257+0000 7f4587400640 1 -- 172.21.15.90:0/3401600180 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560130402bb0 msgr2=0x560130405460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:50:14.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.257+0000 7f4587400640 1 --2- 172.21.15.90:0/3401600180 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560130402bb0 0x560130405460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:14.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.257+0000 7f4587400640 1 -- 172.21.15.90:0/3401600180 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5601303fbd10 msgr2=0x560130406bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:50:14.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.257+0000 7f4587400640 1 --2- 172.21.15.90:0/3401600180 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5601303fbd10 0x560130406bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:14.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.257+0000 7f4587400640 1 -- 172.21.15.90:0/3401600180 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5601304061d0 con 0x560130405080 2024-04-02T14:50:14.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.257+0000 7f4586000640 1 --2- 172.21.15.90:0/3401600180 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560130402bb0 0x560130405460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:50:14.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.257+0000 7f4587400640 1 --2- 172.21.15.90:0/3401600180 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560130405080 0x5601304059a0 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7f45800efc20 tx=0x7f45800be070 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7a5403ae5a2d21aa server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:50:14.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.257+0000 7f4574c00640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:50:14.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.257+0000 7f4574c00640 1 -- 172.21.15.90:0/3401600180 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45800effa0 con 0x560130405080 2024-04-02T14:50:14.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.257+0000 7f4590a2b740 1 -- 172.21.15.90:0/3401600180 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f457802b070 con 0x560130405080 2024-04-02T14:50:14.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.257+0000 7f4574c00640 1 -- 172.21.15.90:0/3401600180 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 8594+0+0 (secure 0 0 0) 0x7f45800c6c50 con 0x560130405080 2024-04-02T14:50:14.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.258+0000 7f4574c00640 1 -- 172.21.15.90:0/3401600180 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f45800348f0 con 0x560130405080 2024-04-02T14:50:14.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.258+0000 7f4590a2b740 10 client.30242 fetch_fsmap learned FSMap version 27 2024-04-02T14:50:14.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.258+0000 7f4590a2b740 1 -- 172.21.15.90:0/3401600180 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x56013040d060 con 0x560130405080 2024-04-02T14:50:14.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.258+0000 7f4574c00640 1 -- 172.21.15.90:0/3401600180 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f457802b070 con 0x560130405080 2024-04-02T14:50:14.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.258+0000 7f4574c00640 1 -- 172.21.15.90:0/3401600180 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f458003c030 con 0x560130405080 2024-04-02T14:50:14.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.258+0000 7f4590a2b740 10 client.30242 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:50:14.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.258+0000 7f4590a2b740 10 client.30242 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:50:14.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.258+0000 7f4590a2b740 20 client.30242 populate_metadata read hostname 'smithi090' 2024-04-02T14:50:14.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.258+0000 7f4590a2b740 1 --2- 172.21.15.90:0/3401600180 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x56013040d6e0 0x56013047da50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:14.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.258+0000 7f4590a2b740 4 client.30242 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-02T14:50:14.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.258+0000 7f4590a2b740 1 -- 172.21.15.90:0/3401600180 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56013036dc30 con 0x56013040d6e0 2024-04-02T14:50:14.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.258+0000 7f4586a00640 1 --2- 172.21.15.90:0/3401600180 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x56013040d6e0 0x56013047da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:14.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.259+0000 7f4586a00640 1 --2- 172.21.15.90:0/3401600180 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x56013040d6e0 0x56013047da50 crc :-1 s=READY pgs=387 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d82312aeecb661c7 server_cookie=70d6bf4701f65b83 in_seq=0 out_seq=0 2024-04-02T14:50:14.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.259+0000 7f4574c00640 10 client.30242 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:50:14.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.259+0000 7f4574c00640 1 -- 172.21.15.90:0/3401600180 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56013036dc30 con 0x56013040d6e0 2024-04-02T14:50:14.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.259+0000 7f4574c00640 10 client.30242 handle_command_reply: tid=0 2024-04-02T14:50:14.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.259+0000 7f4590a2b740 1 client.30242 shutdown 2024-04-02T14:50:14.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.259+0000 7f4590a2b740 1 -- 172.21.15.90:0/3401600180 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560130405080 msgr2=0x5601304059a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:14.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.259+0000 7f4590a2b740 1 --2- 172.21.15.90:0/3401600180 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560130405080 0x5601304059a0 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7f45800efc20 tx=0x7f45800be070 comp rx=0 tx=0).stop 2024-04-02T14:50:14.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.259+0000 7f4590a2b740 1 -- 172.21.15.90:0/3401600180 shutdown_connections 2024-04-02T14:50:14.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.259+0000 7f4590a2b740 1 --2- 172.21.15.90:0/3401600180 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x56013040d6e0 0x56013047da50 crc :-1 s=READY pgs=387 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:14.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.259+0000 7f4590a2b740 1 --2- 172.21.15.90:0/3401600180 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x560130405080 0x5601304059a0 unknown :-1 s=CLOSED pgs=1028 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:14.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.259+0000 7f4590a2b740 1 --2- 172.21.15.90:0/3401600180 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560130402bb0 0x560130405460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:14.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.259+0000 7f4590a2b740 1 --2- 172.21.15.90:0/3401600180 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5601303fbd10 0x560130406bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:14.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.259+0000 7f4590a2b740 1 -- 172.21.15.90:0/3401600180 >> 172.21.15.90:0/3401600180 conn(0x5601303c2a50 msgr2=0x5601303f5010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:50:14.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.259+0000 7f4574c00640 0 client.30242 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:50:14.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.259+0000 7f4590a2b740 1 -- 172.21.15.90:0/3401600180 shutdown_connections 2024-04-02T14:50:14.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.260+0000 7f4590a2b740 1 -- 172.21.15.90:0/3401600180 wait complete. 2024-04-02T14:50:14.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.260+0000 7f4590a2b740 20 client.30242 trim_cache size 0 max 16384 2024-04-02T14:50:14.256 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-02T14:50:14.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.261+0000 7f458dc00640 1 -- 172.21.15.90:0/440216130 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f456c0a3f20 msgr2=0x7f456c0a63e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:14.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.261+0000 7f458dc00640 1 --2- 172.21.15.90:0/440216130 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f456c0a3f20 0x7f456c0a63e0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f45780058c0 tx=0x7f4578005850 comp rx=0 tx=0).stop 2024-04-02T14:50:14.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.261+0000 7f458dc00640 1 -- 172.21.15.90:0/440216130 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f458806c970 msgr2=0x7f458806ce20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:14.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.261+0000 7f458dc00640 1 --2- 172.21.15.90:0/440216130 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f458806c970 0x7f458806ce20 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f4580037040 tx=0x7f4580032ec0 comp rx=0 tx=0).stop 2024-04-02T14:50:14.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.262+0000 7f458dc00640 1 -- 172.21.15.90:0/440216130 shutdown_connections 2024-04-02T14:50:14.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.262+0000 7f458dc00640 1 --2- 172.21.15.90:0/440216130 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f456c0a3f20 0x7f456c0a63e0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:14.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.262+0000 7f458dc00640 1 --2- 172.21.15.90:0/440216130 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f458806c970 0x7f458806ce20 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:14.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.262+0000 7f458dc00640 1 --2- 172.21.15.90:0/440216130 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4588108fe0 0x7f458806c430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:14.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.262+0000 7f458dc00640 1 --2- 172.21.15.90:0/440216130 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4588103f10 0x7f4588073360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:14.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.262+0000 7f458dc00640 1 -- 172.21.15.90:0/440216130 >> 172.21.15.90:0/440216130 conn(0x7f45880ff7b0 msgr2=0x7f4588101b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:50:14.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.263+0000 7f458dc00640 1 -- 172.21.15.90:0/440216130 shutdown_connections 2024-04-02T14:50:14.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:14.263+0000 7f458dc00640 1 -- 172.21.15.90:0/440216130 wait complete. 2024-04-02T14:50:15.276 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-02T14:50:15.276 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T14:50:15.276 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-02T14:50:15.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.535+0000 7fa4a0000640 1 -- 172.21.15.90:0/600798238 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa498076950 msgr2=0x7fa498112400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:15.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.535+0000 7fa4a0000640 1 --2- 172.21.15.90:0/600798238 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa498076950 0x7fa498112400 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fa484009860 tx=0x7fa48402bb70 comp rx=0 tx=0).stop 2024-04-02T14:50:15.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.535+0000 7fa4a0000640 1 -- 172.21.15.90:0/600798238 shutdown_connections 2024-04-02T14:50:15.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.535+0000 7fa4a0000640 1 --2- 172.21.15.90:0/600798238 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa498112940 0x7fa498114d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:15.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.535+0000 7fa4a0000640 1 --2- 172.21.15.90:0/600798238 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa498076950 0x7fa498112400 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:15.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.535+0000 7fa4a0000640 1 --2- 172.21.15.90:0/600798238 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa498076030 0x7fa498076410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:15.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.535+0000 7fa4a0000640 1 -- 172.21.15.90:0/600798238 >> 172.21.15.90:0/600798238 conn(0x7fa4980ff7b0 msgr2=0x7fa498101bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:50:15.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.535+0000 7fa4a0000640 1 -- 172.21.15.90:0/600798238 shutdown_connections 2024-04-02T14:50:15.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.535+0000 7fa4a0000640 1 -- 172.21.15.90:0/600798238 wait complete. 2024-04-02T14:50:15.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.536+0000 7fa4a0000640 1 Processor -- start 2024-04-02T14:50:15.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.536+0000 7fa4a0000640 1 -- start start 2024-04-02T14:50:15.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.537+0000 7fa4a0000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa498076030 0x7fa4981a4660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:15.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.537+0000 7fa4a0000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa498076950 0x7fa4981a4ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:15.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.537+0000 7fa4a0000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa498112940 0x7fa49819e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:15.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.537+0000 7fa4a0000640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fa498118490 con 0x7fa498112940 2024-04-02T14:50:15.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.537+0000 7fa4a0000640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fa498118310 con 0x7fa498076950 2024-04-02T14:50:15.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.537+0000 7fa4a0000640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fa498118610 con 0x7fa498076030 2024-04-02T14:50:15.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.537+0000 7fa49d600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa498076030 0x7fa4981a4660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:15.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.537+0000 7fa49d600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa498076030 0x7fa4981a4660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:55212/0 (socket says 172.21.15.90:55212) 2024-04-02T14:50:15.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.537+0000 7fa49d600640 1 -- 172.21.15.90:0/902311748 learned_addr learned my addr 172.21.15.90:0/902311748 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:50:15.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.537+0000 7fa49cc00640 1 --2- 172.21.15.90:0/902311748 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa498076950 0x7fa4981a4ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:15.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.538+0000 7fa49d600640 1 -- 172.21.15.90:0/902311748 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa498076950 msgr2=0x7fa4981a4ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:15.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.538+0000 7fa49d600640 1 --2- 172.21.15.90:0/902311748 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa498076950 0x7fa4981a4ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:15.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.538+0000 7fa49d600640 1 -- 172.21.15.90:0/902311748 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa498112940 msgr2=0x7fa49819e7d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-02T14:50:15.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.538+0000 7fa49e000640 1 --2- 172.21.15.90:0/902311748 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa498112940 0x7fa49819e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:15.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.539+0000 7fa49d600640 1 --2- 172.21.15.90:0/902311748 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa498112940 0x7fa49819e7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:15.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.539+0000 7fa49d600640 1 -- 172.21.15.90:0/902311748 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa49819efa0 con 0x7fa498076030 2024-04-02T14:50:15.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.539+0000 7fa49cc00640 1 --2- 172.21.15.90:0/902311748 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa498076950 0x7fa4981a4ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:50:15.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.539+0000 7fa49e000640 1 --2- 172.21.15.90:0/902311748 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa498112940 0x7fa49819e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:50:15.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.539+0000 7fa49d600640 1 --2- 172.21.15.90:0/902311748 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa498076030 0x7fa4981a4660 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7fa48c00e9c0 tx=0x7fa48c00ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:50:15.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.539+0000 7fa496000640 1 -- 172.21.15.90:0/902311748 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa48c00ccc0 con 0x7fa498076030 2024-04-02T14:50:15.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.539+0000 7fa496000640 1 -- 172.21.15.90:0/902311748 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa48c005500 con 0x7fa498076030 2024-04-02T14:50:15.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.539+0000 7fa496000640 1 -- 172.21.15.90:0/902311748 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa48c010430 con 0x7fa498076030 2024-04-02T14:50:15.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.539+0000 7fa4a0000640 1 -- 172.21.15.90:0/902311748 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa49819f290 con 0x7fa498076030 2024-04-02T14:50:15.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.540+0000 7fa4a0000640 1 -- 172.21.15.90:0/902311748 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4981ab4b0 con 0x7fa498076030 2024-04-02T14:50:15.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.545+0000 7fa4a2c51740 1 Processor -- start 2024-04-02T14:50:15.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.547+0000 7fa4a2c51740 1 -- start start 2024-04-02T14:50:15.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.547+0000 7fa4a2c51740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555fa8761030 0x555fa875fd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:15.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.547+0000 7fa4a2c51740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555fa8760350 0x555fa87607d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:15.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.547+0000 7fa4a2c51740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555fa87696c0 0x555fa876bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:15.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.547+0000 7fa4a2c51740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x555fa86c4310 con 0x555fa8761030 2024-04-02T14:50:15.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.547+0000 7fa4a2c51740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x555fa866c690 con 0x555fa87696c0 2024-04-02T14:50:15.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.548+0000 7fa4a2c51740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x555fa86c02c0 con 0x555fa8760350 2024-04-02T14:50:15.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.549+0000 7fa496000640 1 -- 172.21.15.90:0/902311748 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fa48c0105d0 con 0x7fa498076030 2024-04-02T14:50:15.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.549+0000 7fa49e000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555fa8761030 0x555fa875fd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:15.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.549+0000 7fa49e000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555fa8761030 0x555fa875fd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:54152/0 (socket says 172.21.15.90:54152) 2024-04-02T14:50:15.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.549+0000 7fa49e000640 1 -- 172.21.15.90:0/3859984877 learned_addr learned my addr 172.21.15.90:0/3859984877 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:50:15.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.550+0000 7fa49e000640 1 -- 172.21.15.90:0/3859984877 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555fa8760350 msgr2=0x555fa87607d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:50:15.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.550+0000 7fa49e000640 1 --2- 172.21.15.90:0/3859984877 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555fa8760350 0x555fa87607d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:15.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.550+0000 7fa49e000640 1 -- 172.21.15.90:0/3859984877 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555fa87696c0 msgr2=0x555fa876bab0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-02T14:50:15.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.550+0000 7fa49e000640 1 --2- 172.21.15.90:0/3859984877 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555fa87696c0 0x555fa876bab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:15.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.550+0000 7fa49e000640 1 -- 172.21.15.90:0/3859984877 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555fa84a07e0 con 0x555fa8761030 2024-04-02T14:50:15.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.553+0000 7fa49e000640 1 --2- 172.21.15.90:0/3859984877 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555fa8761030 0x555fa875fd80 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fa490009e10 tx=0x7fa49000f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6efc482c2ddd1ace server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:50:15.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.554+0000 7fa48a600640 1 -- 172.21.15.90:0/3859984877 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa490004d20 con 0x555fa8761030 2024-04-02T14:50:15.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.554+0000 7fa48a600640 1 -- 172.21.15.90:0/3859984877 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa490005500 con 0x555fa8761030 2024-04-02T14:50:15.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.554+0000 7fa48a600640 1 -- 172.21.15.90:0/3859984877 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa490011860 con 0x555fa8761030 2024-04-02T14:50:15.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.555+0000 7fa4a2c51740 1 -- 172.21.15.90:0/3859984877 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555fa8761030 msgr2=0x555fa875fd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:15.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.555+0000 7fa4a2c51740 1 --2- 172.21.15.90:0/3859984877 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555fa8761030 0x555fa875fd80 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fa490009e10 tx=0x7fa49000f730 comp rx=0 tx=0).stop 2024-04-02T14:50:15.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.556+0000 7fa496000640 1 --2- 172.21.15.90:0/902311748 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fa4800a3e20 0x7fa4800a62e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:15.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.556+0000 7fa496000640 1 -- 172.21.15.90:0/902311748 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 8550+0+0 (secure 0 0 0) 0x7fa48c0d0b90 con 0x7fa498076030 2024-04-02T14:50:15.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.558+0000 7fa49e000640 1 --2- 172.21.15.90:0/902311748 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fa4800a3e20 0x7fa4800a62e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:15.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.559+0000 7fa4a2c51740 1 -- 172.21.15.90:0/3859984877 shutdown_connections 2024-04-02T14:50:15.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.559+0000 7fa4a2c51740 1 --2- 172.21.15.90:0/3859984877 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555fa87696c0 0x555fa876bab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:15.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.559+0000 7fa4a2c51740 1 --2- 172.21.15.90:0/3859984877 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555fa8760350 0x555fa87607d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:15.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.559+0000 7fa4a2c51740 1 --2- 172.21.15.90:0/3859984877 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555fa8761030 0x555fa875fd80 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:15.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.559+0000 7fa4a2c51740 1 -- 172.21.15.90:0/3859984877 >> 172.21.15.90:0/3859984877 conn(0x555fa86f4270 msgr2=0x555fa875ada0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:50:15.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.560+0000 7fa4a2c51740 1 -- 172.21.15.90:0/3859984877 shutdown_connections 2024-04-02T14:50:15.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.561+0000 7fa4a2c51740 1 -- 172.21.15.90:0/3859984877 wait complete. 2024-04-02T14:50:15.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.562+0000 7fa49e000640 1 --2- 172.21.15.90:0/902311748 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fa4800a3e20 0x7fa4800a62e0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fa490009de0 tx=0x7fa490004ec0 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:50:15.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.562+0000 7fa4a2c51740 1 -- start start 2024-04-02T14:50:15.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.565+0000 7fa4a2c51740 1 Processor -- start 2024-04-02T14:50:15.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.568+0000 7fa4a2c51740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555fa8760350 0x555fa8766670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:15.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.569+0000 7fa4a2c51740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555fa8761030 0x555fa8766bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:15.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.569+0000 7fa4a2c51740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555fa87696c0 0x555fa87670f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:15.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.569+0000 7fa4a2c51740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x555fa877d540 con 0x555fa87696c0 2024-04-02T14:50:15.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.569+0000 7fa4a2c51740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x555fa867fa60 con 0x555fa8760350 2024-04-02T14:50:15.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.569+0000 7fa4a2c51740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x555fa877d6c0 con 0x555fa8761030 2024-04-02T14:50:15.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.569+0000 7fa4a2c51740 10 client.0 _ll_register_callbacks cb 0x555fa8685030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:50:15.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.569+0000 7fa49cc00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555fa8761030 0x555fa8766bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:15.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.569+0000 7fa49cc00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555fa8761030 0x555fa8766bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:55238/0 (socket says 172.21.15.90:55238) 2024-04-02T14:50:15.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.569+0000 7fa49cc00640 1 -- 172.21.15.90:0/2833520645 learned_addr learned my addr 172.21.15.90:0/2833520645 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:50:15.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.570+0000 7fa49cc00640 1 -- 172.21.15.90:0/2833520645 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555fa8760350 msgr2=0x555fa8766670 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-02T14:50:15.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.570+0000 7fa49cc00640 1 --2- 172.21.15.90:0/2833520645 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555fa8760350 0x555fa8766670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:15.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.570+0000 7fa49cc00640 1 -- 172.21.15.90:0/2833520645 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555fa87696c0 msgr2=0x555fa87670f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-02T14:50:15.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.570+0000 7fa49cc00640 1 --2- 172.21.15.90:0/2833520645 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555fa87696c0 0x555fa87670f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:15.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.570+0000 7fa49cc00640 1 -- 172.21.15.90:0/2833520645 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555fa8500f70 con 0x555fa8761030 2024-04-02T14:50:15.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.571+0000 7fa49cc00640 1 --2- 172.21.15.90:0/2833520645 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555fa8761030 0x555fa8766bb0 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7fa484009780 tx=0x7fa484005a90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=343bcee8b42bc33b server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:50:15.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.571+0000 7fa47ea00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:50:15.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.571+0000 7fa47ea00640 1 -- 172.21.15.90:0/2833520645 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa484044020 con 0x555fa8761030 2024-04-02T14:50:15.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.572+0000 7fa47ea00640 1 -- 172.21.15.90:0/2833520645 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 8550+0+0 (secure 0 0 0) 0x7fa484034640 con 0x555fa8761030 2024-04-02T14:50:15.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.572+0000 7fa4a2c51740 1 -- 172.21.15.90:0/2833520645 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa490005500 con 0x555fa8761030 2024-04-02T14:50:15.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.572+0000 7fa47ea00640 1 -- 172.21.15.90:0/2833520645 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fa484039040 con 0x555fa8761030 2024-04-02T14:50:15.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.572+0000 7fa4a2c51740 10 client.30040 fetch_fsmap learned FSMap version 27 2024-04-02T14:50:15.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.572+0000 7fa4a2c51740 1 -- 172.21.15.90:0/2833520645 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x555fa8398170 con 0x555fa8761030 2024-04-02T14:50:15.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.573+0000 7fa47ea00640 1 -- 172.21.15.90:0/2833520645 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa490005500 con 0x555fa8761030 2024-04-02T14:50:15.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.577+0000 7fa47ea00640 1 -- 172.21.15.90:0/2833520645 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fa48402c510 con 0x555fa8761030 2024-04-02T14:50:15.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.577+0000 7fa4a2c51740 10 client.30040 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:50:15.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.577+0000 7fa4a2c51740 10 client.30040 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:50:15.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.577+0000 7fa4a2c51740 20 client.30040 populate_metadata read hostname 'smithi090' 2024-04-02T14:50:15.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.577+0000 7fa4a2c51740 1 --2- 172.21.15.90:0/2833520645 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x555fa878aae0 0x555fa878cf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:15.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.577+0000 7fa4a2c51740 4 client.30040 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:50:15.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.577+0000 7fa4a2c51740 1 -- 172.21.15.90:0/2833520645 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555fa86d1850 con 0x555fa878aae0 2024-04-02T14:50:15.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.580+0000 7fa49d600640 1 --2- 172.21.15.90:0/2833520645 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x555fa878aae0 0x555fa878cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:15.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.582+0000 7fa49d600640 1 --2- 172.21.15.90:0/2833520645 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x555fa878aae0 0x555fa878cf20 crc :-1 s=READY pgs=388 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eb22ce3ec79d5ce4 server_cookie=d15b0224166a71a4 in_seq=0 out_seq=0 2024-04-02T14:50:15.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.582+0000 7fa47ea00640 10 client.30040 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:50:15.576 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.583+0000 7fa47ea00640 1 -- 172.21.15.90:0/2833520645 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x555fa86d1850 con 0x555fa878aae0 2024-04-02T14:50:15.576 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.584+0000 7fa47ea00640 10 client.30040 handle_command_reply: tid=0 2024-04-02T14:50:15.576 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.584+0000 7fa4a2c51740 1 client.30040 shutdown 2024-04-02T14:50:15.576 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.584+0000 7fa4a2c51740 1 -- 172.21.15.90:0/2833520645 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555fa8761030 msgr2=0x555fa8766bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:15.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.584+0000 7fa4a2c51740 1 --2- 172.21.15.90:0/2833520645 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555fa8761030 0x555fa8766bb0 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7fa484009780 tx=0x7fa484005a90 comp rx=0 tx=0).stop 2024-04-02T14:50:15.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.584+0000 7fa4a2c51740 1 -- 172.21.15.90:0/2833520645 shutdown_connections 2024-04-02T14:50:15.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.584+0000 7fa4a2c51740 1 --2- 172.21.15.90:0/2833520645 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x555fa878aae0 0x555fa878cf20 crc :-1 s=READY pgs=388 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:15.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.584+0000 7fa4a2c51740 1 --2- 172.21.15.90:0/2833520645 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555fa87696c0 0x555fa87670f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:15.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.584+0000 7fa47ea00640 0 client.30040 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:50:15.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.585+0000 7fa4a2c51740 1 --2- 172.21.15.90:0/2833520645 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555fa8761030 0x555fa8766bb0 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:15.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.585+0000 7fa4a2c51740 1 --2- 172.21.15.90:0/2833520645 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555fa8760350 0x555fa8766670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:15.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.585+0000 7fa4a2c51740 1 -- 172.21.15.90:0/2833520645 >> 172.21.15.90:0/2833520645 conn(0x555fa86f4270 msgr2=0x555fa875caf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:50:15.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.585+0000 7fa4a2c51740 1 -- 172.21.15.90:0/2833520645 shutdown_connections 2024-04-02T14:50:15.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.585+0000 7fa4a2c51740 1 -- 172.21.15.90:0/2833520645 wait complete. 2024-04-02T14:50:15.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.586+0000 7fa4a2c51740 20 client.30040 trim_cache size 0 max 16384 2024-04-02T14:50:15.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.623+0000 7fa4a2c51740 1 Processor -- start 2024-04-02T14:50:15.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.623+0000 7fa4a2c51740 1 -- start start 2024-04-02T14:50:15.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.624+0000 7fa4a2c51740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555fa8760350 0x555fa8768da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:15.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.624+0000 7fa4a2c51740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555fa8761030 0x555fa87655b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:15.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.624+0000 7fa4a2c51740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555fa87696c0 0x555fa87864a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:15.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.624+0000 7fa4a2c51740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fa4981ab870 con 0x555fa8760350 2024-04-02T14:50:15.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.624+0000 7fa4a2c51740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fa4981ad3b0 con 0x555fa87696c0 2024-04-02T14:50:15.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.624+0000 7fa4a2c51740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fa4981abf40 con 0x555fa8761030 2024-04-02T14:50:15.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.624+0000 7fa49d600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555fa8760350 0x555fa8768da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:15.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.624+0000 7fa49d600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555fa8760350 0x555fa8768da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:54166/0 (socket says 172.21.15.90:54166) 2024-04-02T14:50:15.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.625+0000 7fa49d600640 1 -- 172.21.15.90:0/3483789946 learned_addr learned my addr 172.21.15.90:0/3483789946 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:50:15.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.625+0000 7fa49d600640 1 -- 172.21.15.90:0/3483789946 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555fa8761030 msgr2=0x555fa87655b0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-02T14:50:15.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.625+0000 7fa49d600640 1 --2- 172.21.15.90:0/3483789946 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555fa8761030 0x555fa87655b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:15.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.625+0000 7fa49d600640 1 -- 172.21.15.90:0/3483789946 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555fa87696c0 msgr2=0x555fa87864a0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-02T14:50:15.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.625+0000 7fa49d600640 1 --2- 172.21.15.90:0/3483789946 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555fa87696c0 0x555fa87864a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:15.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.625+0000 7fa49d600640 1 -- 172.21.15.90:0/3483789946 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555fa8764bc0 con 0x555fa8760350 2024-04-02T14:50:15.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.626+0000 7fa49d600640 1 --2- 172.21.15.90:0/3483789946 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555fa8760350 0x555fa8768da0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fa48c09c7b0 tx=0x7fa48c0bd040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4feb41c4fcaa2a85 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:50:15.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.627+0000 7fa489c00640 1 -- 172.21.15.90:0/3483789946 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa48c0b5070 con 0x555fa8760350 2024-04-02T14:50:15.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.627+0000 7fa489c00640 1 -- 172.21.15.90:0/3483789946 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa48c09cab0 con 0x555fa8760350 2024-04-02T14:50:15.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.627+0000 7fa489c00640 1 -- 172.21.15.90:0/3483789946 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa48c0b5070 con 0x555fa8760350 2024-04-02T14:50:15.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.628+0000 7fa4a2c51740 1 -- 172.21.15.90:0/3483789946 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555fa8760350 msgr2=0x555fa8768da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:15.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.628+0000 7fa4a2c51740 1 --2- 172.21.15.90:0/3483789946 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555fa8760350 0x555fa8768da0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fa48c09c7b0 tx=0x7fa48c0bd040 comp rx=0 tx=0).stop 2024-04-02T14:50:15.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.629+0000 7fa4a2c51740 1 -- 172.21.15.90:0/3483789946 shutdown_connections 2024-04-02T14:50:15.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.629+0000 7fa4a2c51740 1 --2- 172.21.15.90:0/3483789946 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555fa87696c0 0x555fa87864a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:15.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.629+0000 7fa4a2c51740 1 --2- 172.21.15.90:0/3483789946 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555fa8761030 0x555fa87655b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:15.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.629+0000 7fa4a2c51740 1 --2- 172.21.15.90:0/3483789946 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555fa8760350 0x555fa8768da0 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:15.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.629+0000 7fa4a2c51740 1 -- 172.21.15.90:0/3483789946 >> 172.21.15.90:0/3483789946 conn(0x555fa86f4270 msgr2=0x555fa875ca80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:50:15.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.629+0000 7fa4a2c51740 1 -- 172.21.15.90:0/3483789946 shutdown_connections 2024-04-02T14:50:15.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.629+0000 7fa4a2c51740 1 -- 172.21.15.90:0/3483789946 wait complete. 2024-04-02T14:50:15.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.630+0000 7fa4a2c51740 1 -- start start 2024-04-02T14:50:15.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.630+0000 7fa4a2c51740 1 Processor -- start 2024-04-02T14:50:15.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.631+0000 7fa4a2c51740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555fa8760350 0x555fa8782c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:15.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.631+0000 7fa4a2c51740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555fa8761030 0x555fa87ebf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:15.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.631+0000 7fa4a2c51740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555fa87696c0 0x555fa87ec460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:15.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.631+0000 7fa4a2c51740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fa4981acce0 con 0x555fa87696c0 2024-04-02T14:50:15.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.631+0000 7fa4a2c51740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fa4981ac610 con 0x555fa8761030 2024-04-02T14:50:15.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.631+0000 7fa4a2c51740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fa49810be80 con 0x555fa8760350 2024-04-02T14:50:15.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.631+0000 7fa4a2c51740 10 client.0 _ll_register_callbacks cb 0x7fa48c005ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:50:15.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.631+0000 7fa49cc00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555fa8761030 0x555fa87ebf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:15.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.631+0000 7fa49cc00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555fa8761030 0x555fa87ebf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:48038/0 (socket says 172.21.15.90:48038) 2024-04-02T14:50:15.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.631+0000 7fa49cc00640 1 -- 172.21.15.90:0/730856214 learned_addr learned my addr 172.21.15.90:0/730856214 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:50:15.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.631+0000 7fa49d600640 1 --2- 172.21.15.90:0/730856214 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555fa8760350 0x555fa8782c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:15.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.632+0000 7fa49e000640 1 --2- 172.21.15.90:0/730856214 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555fa87696c0 0x555fa87ec460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:15.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.632+0000 7fa49d600640 1 -- 172.21.15.90:0/730856214 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555fa8761030 msgr2=0x555fa87ebf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:50:15.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.632+0000 7fa49d600640 1 --2- 172.21.15.90:0/730856214 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555fa8761030 0x555fa87ebf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:15.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.632+0000 7fa49d600640 1 -- 172.21.15.90:0/730856214 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555fa87696c0 msgr2=0x555fa87ec460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:50:15.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.632+0000 7fa49d600640 1 --2- 172.21.15.90:0/730856214 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555fa87696c0 0x555fa87ec460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:15.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.632+0000 7fa49d600640 1 -- 172.21.15.90:0/730856214 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555fa87ecb50 con 0x555fa8760350 2024-04-02T14:50:15.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.632+0000 7fa49d600640 1 --2- 172.21.15.90:0/730856214 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555fa8760350 0x555fa8782c40 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x555fa8786a60 tx=0x7fa48c009f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a5f897772f6064d5 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:50:15.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.632+0000 7fa47ea00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:50:15.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.632+0000 7fa47ea00640 1 -- 172.21.15.90:0/730856214 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa48c082070 con 0x555fa8760350 2024-04-02T14:50:15.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.633+0000 7fa47ea00640 1 -- 172.21.15.90:0/730856214 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 8550+0+0 (secure 0 0 0) 0x7fa48c099120 con 0x555fa8760350 2024-04-02T14:50:15.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.633+0000 7fa4a2c51740 1 -- 172.21.15.90:0/730856214 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa48c09cab0 con 0x555fa8760350 2024-04-02T14:50:15.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.634+0000 7fa47ea00640 1 -- 172.21.15.90:0/730856214 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fa48c0b5070 con 0x555fa8760350 2024-04-02T14:50:15.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.634+0000 7fa4a2c51740 10 client.30043 fetch_fsmap learned FSMap version 27 2024-04-02T14:50:15.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.634+0000 7fa4a2c51740 1 -- 172.21.15.90:0/730856214 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x555fa87819b0 con 0x555fa8760350 2024-04-02T14:50:15.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.634+0000 7fa47ea00640 1 -- 172.21.15.90:0/730856214 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa48c09cab0 con 0x555fa8760350 2024-04-02T14:50:15.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.635+0000 7fa47ea00640 1 -- 172.21.15.90:0/730856214 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fa48c09bf00 con 0x555fa8760350 2024-04-02T14:50:15.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.635+0000 7fa4a2c51740 10 client.30043 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:50:15.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.635+0000 7fa4a2c51740 10 client.30043 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:50:15.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.635+0000 7fa4a2c51740 20 client.30043 populate_metadata read hostname 'smithi090' 2024-04-02T14:50:15.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.635+0000 7fa4a2c51740 1 --2- 172.21.15.90:0/730856214 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x555fa878aae0 0x555fa8781fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:15.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.635+0000 7fa4a2c51740 4 client.30043 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-02T14:50:15.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.635+0000 7fa4a2c51740 1 -- 172.21.15.90:0/730856214 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x555fa86cbba0 con 0x555fa878aae0 2024-04-02T14:50:15.630 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.637+0000 7fa49cc00640 1 --2- 172.21.15.90:0/730856214 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x555fa878aae0 0x555fa8781fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:15.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.640+0000 7fa47ea00640 10 client.30043 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:50:15.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.640+0000 7fa49cc00640 1 --2- 172.21.15.90:0/730856214 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x555fa878aae0 0x555fa8781fd0 crc :-1 s=READY pgs=389 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=46756b91f3fe1d13 server_cookie=e5180d2c07b50fd9 in_seq=0 out_seq=0 2024-04-02T14:50:15.634 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.642+0000 7fa47ea00640 1 -- 172.21.15.90:0/730856214 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x555fa86cbba0 con 0x555fa878aae0 2024-04-02T14:50:15.634 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.642+0000 7fa47ea00640 10 client.30043 handle_command_reply: tid=0 2024-04-02T14:50:15.634 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.642+0000 7fa4a2c51740 1 client.30043 shutdown 2024-04-02T14:50:15.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.642+0000 7fa4a2c51740 1 -- 172.21.15.90:0/730856214 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555fa8760350 msgr2=0x555fa8782c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:15.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.642+0000 7fa4a2c51740 1 --2- 172.21.15.90:0/730856214 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555fa8760350 0x555fa8782c40 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x555fa8786a60 tx=0x7fa48c009f90 comp rx=0 tx=0).stop 2024-04-02T14:50:15.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.643+0000 7fa4a2c51740 1 -- 172.21.15.90:0/730856214 shutdown_connections 2024-04-02T14:50:15.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.643+0000 7fa4a2c51740 1 --2- 172.21.15.90:0/730856214 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x555fa878aae0 0x555fa8781fd0 crc :-1 s=READY pgs=389 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:15.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.643+0000 7fa4a2c51740 1 --2- 172.21.15.90:0/730856214 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555fa87696c0 0x555fa87ec460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:15.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.643+0000 7fa47ea00640 0 client.30043 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:50:15.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.643+0000 7fa4a2c51740 1 --2- 172.21.15.90:0/730856214 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555fa8761030 0x555fa87ebf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:15.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.643+0000 7fa4a2c51740 1 --2- 172.21.15.90:0/730856214 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555fa8760350 0x555fa8782c40 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:15.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.643+0000 7fa4a2c51740 1 -- 172.21.15.90:0/730856214 >> 172.21.15.90:0/730856214 conn(0x555fa86f4270 msgr2=0x555fa875c0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:50:15.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.644+0000 7fa4a2c51740 1 -- 172.21.15.90:0/730856214 shutdown_connections 2024-04-02T14:50:15.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.644+0000 7fa4a2c51740 1 -- 172.21.15.90:0/730856214 wait complete. 2024-04-02T14:50:15.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.644+0000 7fa4a2c51740 20 client.30043 trim_cache size 0 max 16384 2024-04-02T14:50:15.639 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-02T14:50:15.639 INFO:teuthology.orchestra.run.smithi090.stdout: "return_code": 0, 2024-04-02T14:50:15.639 INFO:teuthology.orchestra.run.smithi090.stdout: "scrub_tag": "cc3a691c-35b3-48eb-b236-e1471312d03a", 2024-04-02T14:50:15.639 INFO:teuthology.orchestra.run.smithi090.stdout: "mode": "asynchronous" 2024-04-02T14:50:15.640 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-02T14:50:15.640 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.646+0000 7fa4a0000640 1 -- 172.21.15.90:0/902311748 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fa4800a3e20 msgr2=0x7fa4800a62e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:15.640 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.646+0000 7fa4a0000640 1 --2- 172.21.15.90:0/902311748 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fa4800a3e20 0x7fa4800a62e0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fa490009de0 tx=0x7fa490004ec0 comp rx=0 tx=0).stop 2024-04-02T14:50:15.640 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.646+0000 7fa4a0000640 1 -- 172.21.15.90:0/902311748 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa498076030 msgr2=0x7fa4981a4660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:15.640 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.646+0000 7fa4a0000640 1 --2- 172.21.15.90:0/902311748 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa498076030 0x7fa4981a4660 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7fa48c00e9c0 tx=0x7fa48c00ee80 comp rx=0 tx=0).stop 2024-04-02T14:50:15.640 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.646+0000 7fa4a0000640 1 -- 172.21.15.90:0/902311748 shutdown_connections 2024-04-02T14:50:15.640 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.646+0000 7fa4a0000640 1 --2- 172.21.15.90:0/902311748 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fa4800a3e20 0x7fa4800a62e0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:15.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.646+0000 7fa4a0000640 1 --2- 172.21.15.90:0/902311748 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa498112940 0x7fa49819e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:15.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.646+0000 7fa4a0000640 1 --2- 172.21.15.90:0/902311748 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa498076950 0x7fa4981a4ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:15.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.646+0000 7fa4a0000640 1 --2- 172.21.15.90:0/902311748 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa498076030 0x7fa4981a4660 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:15.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.646+0000 7fa4a0000640 1 -- 172.21.15.90:0/902311748 >> 172.21.15.90:0/902311748 conn(0x7fa4980ff7b0 msgr2=0x7fa498101ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:50:15.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.647+0000 7fa4a0000640 1 -- 172.21.15.90:0/902311748 shutdown_connections 2024-04-02T14:50:15.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.647+0000 7fa4a0000640 1 -- 172.21.15.90:0/902311748 wait complete. 2024-04-02T14:50:15.666 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-02T14:50:15.666 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-02T14:50:15.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.938+0000 7f12a5000640 1 -- 172.21.15.90:0/687368185 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f12a0106a00 msgr2=0x7f12a006a070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:15.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.938+0000 7f12a5000640 1 --2- 172.21.15.90:0/687368185 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f12a0106a00 0x7f12a006a070 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f1290009900 tx=0x7f129002bc30 comp rx=0 tx=0).stop 2024-04-02T14:50:15.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.938+0000 7f12a5000640 1 -- 172.21.15.90:0/687368185 shutdown_connections 2024-04-02T14:50:15.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.938+0000 7f12a5000640 1 --2- 172.21.15.90:0/687368185 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f12a006ac90 0x7f12a0104500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:15.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.938+0000 7f12a5000640 1 --2- 172.21.15.90:0/687368185 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f12a0106e70 0x7f12a006a5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:15.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.938+0000 7f12a5000640 1 --2- 172.21.15.90:0/687368185 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f12a0106a00 0x7f12a006a070 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:15.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.938+0000 7f12a5000640 1 -- 172.21.15.90:0/687368185 >> 172.21.15.90:0/687368185 conn(0x7f12a00779e0 msgr2=0x7f12a0077df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:50:15.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.938+0000 7f12a5000640 1 -- 172.21.15.90:0/687368185 shutdown_connections 2024-04-02T14:50:15.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.938+0000 7f12a5000640 1 -- 172.21.15.90:0/687368185 wait complete. 2024-04-02T14:50:15.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.940+0000 7f12a5000640 1 Processor -- start 2024-04-02T14:50:15.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.940+0000 7f12a5000640 1 -- start start 2024-04-02T14:50:15.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.940+0000 7f12a5000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f12a0106a00 0x7f12a01a4630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:15.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.940+0000 7f12a5000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f12a0106e70 0x7f12a01a4b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:15.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.940+0000 7f12a5000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f12a019e7a0 0x7f12a019ec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:15.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.940+0000 7f12a5000640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f12a0118500 con 0x7f12a019e7a0 2024-04-02T14:50:15.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.940+0000 7f12a5000640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f12a0118380 con 0x7f12a0106a00 2024-04-02T14:50:15.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.940+0000 7f12a5000640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f12a0118680 con 0x7f12a0106e70 2024-04-02T14:50:15.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.941+0000 7f129de00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f12a0106a00 0x7f12a01a4630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:15.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.941+0000 7f129de00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f12a0106a00 0x7f12a01a4630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:48064/0 (socket says 172.21.15.90:48064) 2024-04-02T14:50:15.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.941+0000 7f129de00640 1 -- 172.21.15.90:0/2920380790 learned_addr learned my addr 172.21.15.90:0/2920380790 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:50:15.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.941+0000 7f129e800640 1 --2- 172.21.15.90:0/2920380790 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f12a019e7a0 0x7f12a019ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:15.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.941+0000 7f129d400640 1 --2- 172.21.15.90:0/2920380790 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f12a0106e70 0x7f12a01a4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:15.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.941+0000 7f129de00640 1 -- 172.21.15.90:0/2920380790 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f12a0106e70 msgr2=0x7f12a01a4b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:15.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.941+0000 7f129de00640 1 --2- 172.21.15.90:0/2920380790 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f12a0106e70 0x7f12a01a4b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:15.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.941+0000 7f129de00640 1 -- 172.21.15.90:0/2920380790 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f12a019e7a0 msgr2=0x7f12a019ec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:15.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.941+0000 7f129de00640 1 --2- 172.21.15.90:0/2920380790 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f12a019e7a0 0x7f12a019ec50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:15.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.941+0000 7f129de00640 1 -- 172.21.15.90:0/2920380790 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12a019f330 con 0x7f12a0106a00 2024-04-02T14:50:15.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.941+0000 7f129e800640 1 --2- 172.21.15.90:0/2920380790 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f12a019e7a0 0x7f12a019ec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:50:15.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.942+0000 7f129de00640 1 --2- 172.21.15.90:0/2920380790 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f12a0106a00 0x7f12a01a4630 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7f129002c240 tx=0x7f12900042f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:50:15.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.942+0000 7f1296a00640 1 -- 172.21.15.90:0/2920380790 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1290043020 con 0x7f12a0106a00 2024-04-02T14:50:15.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.942+0000 7f12a5000640 1 -- 172.21.15.90:0/2920380790 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f12a019f5c0 con 0x7f12a0106a00 2024-04-02T14:50:15.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.942+0000 7f1296a00640 1 -- 172.21.15.90:0/2920380790 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f12900054e0 con 0x7f12a0106a00 2024-04-02T14:50:15.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.942+0000 7f1296a00640 1 -- 172.21.15.90:0/2920380790 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1290003b20 con 0x7f12a0106a00 2024-04-02T14:50:15.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.942+0000 7f12a5000640 1 -- 172.21.15.90:0/2920380790 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12a01ab7f0 con 0x7f12a0106a00 2024-04-02T14:50:15.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.945+0000 7f12a7d99740 1 Processor -- start 2024-04-02T14:50:15.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.946+0000 7f12a7d99740 1 -- start start 2024-04-02T14:50:15.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.947+0000 7f12a7d99740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5648ddac77b0 0x5648ddac7b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:15.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.947+0000 7f12a7d99740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5648ddac80d0 0x5648ddad0770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:15.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.947+0000 7f12a7d99740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5648ddad0cb0 0x5648ddad30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:15.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.947+0000 7f12a7d99740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5648dda2b770 con 0x5648ddac77b0 2024-04-02T14:50:15.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.947+0000 7f12a7d99740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x5648dd9d3700 con 0x5648ddac80d0 2024-04-02T14:50:15.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.947+0000 7f12a7d99740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5648dda27720 con 0x5648ddad0cb0 2024-04-02T14:50:15.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.947+0000 7f129d400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5648ddad0cb0 0x5648ddad30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:15.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.947+0000 7f129d400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5648ddad0cb0 0x5648ddad30a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:55298/0 (socket says 172.21.15.90:55298) 2024-04-02T14:50:15.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.947+0000 7f129d400640 1 -- 172.21.15.90:0/2985313847 learned_addr learned my addr 172.21.15.90:0/2985313847 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:50:15.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.947+0000 7f1296a00640 1 -- 172.21.15.90:0/2920380790 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f1290046020 con 0x7f12a0106a00 2024-04-02T14:50:15.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.948+0000 7f129e800640 1 --2- 172.21.15.90:0/2985313847 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5648ddac77b0 0x5648ddac7b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:15.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.948+0000 7f129d400640 1 -- 172.21.15.90:0/2985313847 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5648ddac80d0 msgr2=0x5648ddad0770 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-02T14:50:15.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.948+0000 7f129d400640 1 --2- 172.21.15.90:0/2985313847 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5648ddac80d0 0x5648ddad0770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:15.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.948+0000 7f129d400640 1 -- 172.21.15.90:0/2985313847 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5648ddac77b0 msgr2=0x5648ddac7b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:50:15.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.948+0000 7f129d400640 1 --2- 172.21.15.90:0/2985313847 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5648ddac77b0 0x5648ddac7b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:15.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.948+0000 7f129d400640 1 -- 172.21.15.90:0/2985313847 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5648dd8097d0 con 0x5648ddad0cb0 2024-04-02T14:50:15.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.949+0000 7f129e800640 1 --2- 172.21.15.90:0/2985313847 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5648ddac77b0 0x5648ddac7b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:50:15.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.949+0000 7f129d400640 1 --2- 172.21.15.90:0/2985313847 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5648ddad0cb0 0x5648ddad30a0 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f128c009b70 tx=0x7f128c0117b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=96e732696da2f554 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:50:15.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.949+0000 7f1296a00640 1 --2- 172.21.15.90:0/2920380790 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f12840a3f20 0x7f12840a63e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:15.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.949+0000 7f128b400640 1 -- 172.21.15.90:0/2985313847 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f128c009e60 con 0x5648ddad0cb0 2024-04-02T14:50:15.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.950+0000 7f128b400640 1 -- 172.21.15.90:0/2985313847 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f128c012930 con 0x5648ddad0cb0 2024-04-02T14:50:15.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.950+0000 7f128b400640 1 -- 172.21.15.90:0/2985313847 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f128c00fbf0 con 0x5648ddad0cb0 2024-04-02T14:50:15.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.950+0000 7f1296a00640 1 -- 172.21.15.90:0/2920380790 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 8550+0+0 (secure 0 0 0) 0x7f12900038c0 con 0x7f12a0106a00 2024-04-02T14:50:15.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.950+0000 7f129e800640 1 --2- 172.21.15.90:0/2920380790 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f12840a3f20 0x7f12840a63e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:15.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.950+0000 7f12a7d99740 1 -- 172.21.15.90:0/2985313847 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5648ddad0cb0 msgr2=0x5648ddad30a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:15.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.950+0000 7f12a7d99740 1 --2- 172.21.15.90:0/2985313847 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5648ddad0cb0 0x5648ddad30a0 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f128c009b70 tx=0x7f128c0117b0 comp rx=0 tx=0).stop 2024-04-02T14:50:15.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.950+0000 7f12a7d99740 1 -- 172.21.15.90:0/2985313847 shutdown_connections 2024-04-02T14:50:15.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.950+0000 7f12a7d99740 1 --2- 172.21.15.90:0/2985313847 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5648ddad0cb0 0x5648ddad30a0 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:15.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.950+0000 7f12a7d99740 1 --2- 172.21.15.90:0/2985313847 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5648ddac80d0 0x5648ddad0770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:15.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.950+0000 7f129e800640 1 --2- 172.21.15.90:0/2920380790 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f12840a3f20 0x7f12840a63e0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f12a0069c50 tx=0x7f1298009500 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:50:15.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.950+0000 7f12a7d99740 1 --2- 172.21.15.90:0/2985313847 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5648ddac77b0 0x5648ddac7b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:15.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.950+0000 7f12a7d99740 1 -- 172.21.15.90:0/2985313847 >> 172.21.15.90:0/2985313847 conn(0x5648ddac1ee0 msgr2=0x5648ddac22c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:50:15.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.951+0000 7f12a7d99740 1 -- 172.21.15.90:0/2985313847 shutdown_connections 2024-04-02T14:50:15.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.951+0000 7f12a7d99740 1 -- 172.21.15.90:0/2985313847 wait complete. 2024-04-02T14:50:15.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.951+0000 7f12a7d99740 1 -- start start 2024-04-02T14:50:15.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.951+0000 7f12a7d99740 1 Processor -- start 2024-04-02T14:50:15.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.952+0000 7f12a7d99740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5648ddac77b0 0x5648ddacdda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:15.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.952+0000 7f12a7d99740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5648ddac80d0 0x5648ddace2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:15.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.952+0000 7f12a7d99740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5648ddad0cb0 0x5648ddace820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:15.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.952+0000 7f12a7d99740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x5648ddad3aa0 con 0x5648ddac80d0 2024-04-02T14:50:15.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.952+0000 7f12a7d99740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x5648dd9e6ac0 con 0x5648ddac77b0 2024-04-02T14:50:15.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.952+0000 7f12a7d99740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5648ddad3c20 con 0x5648ddad0cb0 2024-04-02T14:50:15.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.952+0000 7f12a7d99740 10 client.0 _ll_register_callbacks cb 0x5648dd9e9180 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:50:15.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.952+0000 7f129de00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5648ddac77b0 0x5648ddacdda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:15.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.952+0000 7f129de00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5648ddac77b0 0x5648ddacdda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:48076/0 (socket says 172.21.15.90:48076) 2024-04-02T14:50:15.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.952+0000 7f129de00640 1 -- 172.21.15.90:0/366976078 learned_addr learned my addr 172.21.15.90:0/366976078 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:50:15.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.953+0000 7f129de00640 1 -- 172.21.15.90:0/366976078 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5648ddad0cb0 msgr2=0x5648ddace820 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-02T14:50:15.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.953+0000 7f129de00640 1 --2- 172.21.15.90:0/366976078 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5648ddad0cb0 0x5648ddace820 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:15.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.953+0000 7f129de00640 1 -- 172.21.15.90:0/366976078 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5648ddac80d0 msgr2=0x5648ddace2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:50:15.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.953+0000 7f129de00640 1 --2- 172.21.15.90:0/366976078 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5648ddac80d0 0x5648ddace2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:15.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.953+0000 7f129de00640 1 -- 172.21.15.90:0/366976078 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5648dd73f3f0 con 0x5648ddac77b0 2024-04-02T14:50:15.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.953+0000 7f129de00640 1 --2- 172.21.15.90:0/366976078 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5648ddac77b0 0x5648ddacdda0 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7f1290000c00 tx=0x7f12900be150 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8f091119fc619dc7 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:50:15.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.953+0000 7f127fe00640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:50:15.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.953+0000 7f127fe00640 1 -- 172.21.15.90:0/366976078 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1290005690 con 0x5648ddac77b0 2024-04-02T14:50:15.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.953+0000 7f127fe00640 1 -- 172.21.15.90:0/366976078 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 8550+0+0 (secure 0 0 0) 0x7f12900bac80 con 0x5648ddac77b0 2024-04-02T14:50:15.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.954+0000 7f12a7d99740 1 -- 172.21.15.90:0/366976078 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f128c012930 con 0x5648ddac77b0 2024-04-02T14:50:15.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.954+0000 7f127fe00640 1 -- 172.21.15.90:0/366976078 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f12900bed10 con 0x5648ddac77b0 2024-04-02T14:50:15.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.954+0000 7f12a7d99740 10 client.30260 fetch_fsmap learned FSMap version 27 2024-04-02T14:50:15.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.954+0000 7f12a7d99740 1 -- 172.21.15.90:0/366976078 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5648dd6ff0f0 con 0x5648ddac77b0 2024-04-02T14:50:15.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.954+0000 7f127fe00640 1 -- 172.21.15.90:0/366976078 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f128c012930 con 0x5648ddac77b0 2024-04-02T14:50:15.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.954+0000 7f127fe00640 1 -- 172.21.15.90:0/366976078 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f12900c6df0 con 0x5648ddac77b0 2024-04-02T14:50:15.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.955+0000 7f12a7d99740 10 client.30260 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:50:15.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.955+0000 7f12a7d99740 10 client.30260 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:50:15.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.955+0000 7f12a7d99740 20 client.30260 populate_metadata read hostname 'smithi090' 2024-04-02T14:50:15.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.955+0000 7f12a7d99740 1 --2- 172.21.15.90:0/366976078 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5648ddae0ce0 0x5648ddae3180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:15.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.955+0000 7f12a7d99740 4 client.30260 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:50:15.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.955+0000 7f12a7d99740 1 -- 172.21.15.90:0/366976078 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5648dd9cc630 con 0x5648ddae0ce0 2024-04-02T14:50:15.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.955+0000 7f129d400640 1 --2- 172.21.15.90:0/366976078 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5648ddae0ce0 0x5648ddae3180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:15.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.955+0000 7f129d400640 1 --2- 172.21.15.90:0/366976078 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5648ddae0ce0 0x5648ddae3180 crc :-1 s=READY pgs=390 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=224565c36b22d9d1 server_cookie=85bdb8973e4ef560 in_seq=0 out_seq=0 2024-04-02T14:50:15.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.956+0000 7f127fe00640 10 client.30260 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:50:15.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.956+0000 7f127fe00640 1 -- 172.21.15.90:0/366976078 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5648dd9cc630 con 0x5648ddae0ce0 2024-04-02T14:50:15.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.956+0000 7f127fe00640 10 client.30260 handle_command_reply: tid=0 2024-04-02T14:50:15.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.956+0000 7f12a7d99740 1 client.30260 shutdown 2024-04-02T14:50:15.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.956+0000 7f12a7d99740 1 -- 172.21.15.90:0/366976078 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5648ddac77b0 msgr2=0x5648ddacdda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:15.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.956+0000 7f12a7d99740 1 --2- 172.21.15.90:0/366976078 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5648ddac77b0 0x5648ddacdda0 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7f1290000c00 tx=0x7f12900be150 comp rx=0 tx=0).stop 2024-04-02T14:50:15.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.957+0000 7f12a7d99740 1 -- 172.21.15.90:0/366976078 shutdown_connections 2024-04-02T14:50:15.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.957+0000 7f12a7d99740 1 --2- 172.21.15.90:0/366976078 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5648ddae0ce0 0x5648ddae3180 crc :-1 s=READY pgs=390 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:15.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.957+0000 7f12a7d99740 1 --2- 172.21.15.90:0/366976078 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5648ddad0cb0 0x5648ddace820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:15.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.957+0000 7f12a7d99740 1 --2- 172.21.15.90:0/366976078 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5648ddac80d0 0x5648ddace2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:15.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.957+0000 7f12a7d99740 1 --2- 172.21.15.90:0/366976078 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5648ddac77b0 0x5648ddacdda0 unknown :-1 s=CLOSED pgs=1030 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:15.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.957+0000 7f127fe00640 0 client.30260 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:50:15.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.957+0000 7f12a7d99740 1 -- 172.21.15.90:0/366976078 >> 172.21.15.90:0/366976078 conn(0x5648ddac1ee0 msgr2=0x5648ddac3c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:50:15.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.957+0000 7f12a7d99740 1 -- 172.21.15.90:0/366976078 shutdown_connections 2024-04-02T14:50:15.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.957+0000 7f12a7d99740 1 -- 172.21.15.90:0/366976078 wait complete. 2024-04-02T14:50:15.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.957+0000 7f12a7d99740 20 client.30260 trim_cache size 0 max 16384 2024-04-02T14:50:15.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.989+0000 7f12a7d99740 1 Processor -- start 2024-04-02T14:50:15.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.989+0000 7f12a7d99740 1 -- start start 2024-04-02T14:50:15.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.989+0000 7f12a7d99740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5648ddac77b0 0x5648ddad04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:15.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.989+0000 7f12a7d99740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5648ddac80d0 0x5648ddadc960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:15.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.989+0000 7f12a7d99740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5648ddad0cb0 0x5648ddace470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:15.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.989+0000 7f12a7d99740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f12a0073900 con 0x5648ddac77b0 2024-04-02T14:50:15.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.989+0000 7f12a7d99740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f12a01ad230 con 0x5648ddac80d0 2024-04-02T14:50:15.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.989+0000 7f12a7d99740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f12a01abdc0 con 0x5648ddad0cb0 2024-04-02T14:50:15.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.991+0000 7f129e800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5648ddad0cb0 0x5648ddace470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:15.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.991+0000 7f129e800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5648ddad0cb0 0x5648ddace470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:55322/0 (socket says 172.21.15.90:55322) 2024-04-02T14:50:15.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.991+0000 7f129e800640 1 -- 172.21.15.90:0/2408247686 learned_addr learned my addr 172.21.15.90:0/2408247686 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:50:15.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.992+0000 7f129d400640 1 --2- 172.21.15.90:0/2408247686 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5648ddac80d0 0x5648ddadc960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:15.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.992+0000 7f129e800640 1 -- 172.21.15.90:0/2408247686 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5648ddac80d0 msgr2=0x5648ddadc960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:50:15.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.992+0000 7f129e800640 1 --2- 172.21.15.90:0/2408247686 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5648ddac80d0 0x5648ddadc960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:15.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.992+0000 7f129e800640 1 -- 172.21.15.90:0/2408247686 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5648ddac77b0 msgr2=0x5648ddad04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:50:15.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.992+0000 7f129e800640 1 --2- 172.21.15.90:0/2408247686 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5648ddac77b0 0x5648ddad04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:15.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.992+0000 7f129e800640 1 -- 172.21.15.90:0/2408247686 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5648ddac2860 con 0x5648ddad0cb0 2024-04-02T14:50:15.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.992+0000 7f129e800640 1 --2- 172.21.15.90:0/2408247686 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5648ddad0cb0 0x5648ddace470 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f129800fb50 tx=0x7f1298017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ba3573b42dab9bdf server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:50:15.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.993+0000 7f128aa00640 1 -- 172.21.15.90:0/2408247686 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1298004100 con 0x5648ddad0cb0 2024-04-02T14:50:15.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.993+0000 7f128aa00640 1 -- 172.21.15.90:0/2408247686 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1298004780 con 0x5648ddad0cb0 2024-04-02T14:50:15.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.993+0000 7f128aa00640 1 -- 172.21.15.90:0/2408247686 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1298004100 con 0x5648ddad0cb0 2024-04-02T14:50:15.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.994+0000 7f12a7d99740 1 -- 172.21.15.90:0/2408247686 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5648ddad0cb0 msgr2=0x5648ddace470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:15.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.994+0000 7f12a7d99740 1 --2- 172.21.15.90:0/2408247686 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5648ddad0cb0 0x5648ddace470 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f129800fb50 tx=0x7f1298017000 comp rx=0 tx=0).stop 2024-04-02T14:50:15.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.994+0000 7f12a7d99740 1 -- 172.21.15.90:0/2408247686 shutdown_connections 2024-04-02T14:50:15.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.994+0000 7f12a7d99740 1 --2- 172.21.15.90:0/2408247686 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5648ddad0cb0 0x5648ddace470 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:15.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.994+0000 7f12a7d99740 1 --2- 172.21.15.90:0/2408247686 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5648ddac80d0 0x5648ddadc960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:15.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.994+0000 7f12a7d99740 1 --2- 172.21.15.90:0/2408247686 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5648ddac77b0 0x5648ddad04d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:15.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.994+0000 7f12a7d99740 1 -- 172.21.15.90:0/2408247686 >> 172.21.15.90:0/2408247686 conn(0x5648ddac1ee0 msgr2=0x5648ddacdd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:50:15.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.994+0000 7f12a7d99740 1 -- 172.21.15.90:0/2408247686 shutdown_connections 2024-04-02T14:50:15.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.995+0000 7f12a7d99740 1 -- 172.21.15.90:0/2408247686 wait complete. 2024-04-02T14:50:15.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.995+0000 7f12a7d99740 1 -- start start 2024-04-02T14:50:15.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.996+0000 7f12a7d99740 1 Processor -- start 2024-04-02T14:50:15.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.996+0000 7f12a7d99740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5648ddac77b0 0x5648ddacaad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:15.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.996+0000 7f12a7d99740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5648ddac80d0 0x5648ddacb010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:15.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.996+0000 7f12a7d99740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5648ddad0cb0 0x5648ddacb550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:15.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.996+0000 7f12a7d99740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f12a01acb60 con 0x5648ddad0cb0 2024-04-02T14:50:15.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.996+0000 7f12a7d99740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f12a01ac490 con 0x5648ddac80d0 2024-04-02T14:50:15.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.996+0000 7f12a7d99740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f12a0068d40 con 0x5648ddac77b0 2024-04-02T14:50:15.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.997+0000 7f12a7d99740 10 client.0 _ll_register_callbacks cb 0x7f12900ba8b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:50:15.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.997+0000 7f129de00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5648ddac77b0 0x5648ddacaad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:15.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.997+0000 7f129de00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5648ddac77b0 0x5648ddacaad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:55332/0 (socket says 172.21.15.90:55332) 2024-04-02T14:50:15.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.997+0000 7f129de00640 1 -- 172.21.15.90:0/4196158329 learned_addr learned my addr 172.21.15.90:0/4196158329 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:50:15.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.997+0000 7f129e800640 1 --2- 172.21.15.90:0/4196158329 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5648ddad0cb0 0x5648ddacb550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:15.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.997+0000 7f129de00640 1 -- 172.21.15.90:0/4196158329 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5648ddac80d0 msgr2=0x5648ddacb010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:50:15.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.997+0000 7f129de00640 1 --2- 172.21.15.90:0/4196158329 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5648ddac80d0 0x5648ddacb010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:15.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.997+0000 7f129de00640 1 -- 172.21.15.90:0/4196158329 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5648ddad0cb0 msgr2=0x5648ddacb550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:50:15.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.997+0000 7f129de00640 1 --2- 172.21.15.90:0/4196158329 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5648ddad0cb0 0x5648ddacb550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:15.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.998+0000 7f129de00640 1 -- 172.21.15.90:0/4196158329 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5648ddb46400 con 0x5648ddac77b0 2024-04-02T14:50:15.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.998+0000 7f129e800640 1 --2- 172.21.15.90:0/4196158329 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5648ddad0cb0 0x5648ddacb550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:50:15.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.998+0000 7f129de00640 1 --2- 172.21.15.90:0/4196158329 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5648ddac77b0 0x5648ddacaad0 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f12900efe70 tx=0x7f129002bc30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=37ab22b0aa395691 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:50:15.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.998+0000 7f127fe00640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:50:15.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.999+0000 7f127fe00640 1 -- 172.21.15.90:0/4196158329 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12900dd5a0 con 0x5648ddac77b0 2024-04-02T14:50:15.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.999+0000 7f127fe00640 1 -- 172.21.15.90:0/4196158329 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 8550+0+0 (secure 0 0 0) 0x7f12900bda60 con 0x5648ddac77b0 2024-04-02T14:50:15.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:15.999+0000 7f12a7d99740 1 -- 172.21.15.90:0/4196158329 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1298004780 con 0x5648ddac77b0 2024-04-02T14:50:15.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:16.000+0000 7f127fe00640 1 -- 172.21.15.90:0/4196158329 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f1290005690 con 0x5648ddac77b0 2024-04-02T14:50:15.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:16.000+0000 7f12a7d99740 10 client.30058 fetch_fsmap learned FSMap version 27 2024-04-02T14:50:15.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:16.000+0000 7f12a7d99740 1 -- 172.21.15.90:0/4196158329 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5648ddb58720 con 0x5648ddac77b0 2024-04-02T14:50:15.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:16.004+0000 7f127fe00640 1 -- 172.21.15.90:0/4196158329 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1298004780 con 0x5648ddac77b0 2024-04-02T14:50:15.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:16.004+0000 7f127fe00640 1 -- 172.21.15.90:0/4196158329 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f12900ba360 con 0x5648ddac77b0 2024-04-02T14:50:15.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:16.004+0000 7f12a7d99740 10 client.30058 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:50:15.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:16.004+0000 7f12a7d99740 10 client.30058 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:50:15.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:16.004+0000 7f12a7d99740 20 client.30058 populate_metadata read hostname 'smithi090' 2024-04-02T14:50:15.997 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:16.004+0000 7f12a7d99740 1 --2- 172.21.15.90:0/4196158329 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5648ddb58cb0 0x5648ddb5b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:15.997 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:16.004+0000 7f12a7d99740 4 client.30058 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-02T14:50:15.997 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:16.004+0000 7f12a7d99740 1 -- 172.21.15.90:0/4196158329 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5648dda38cb0 con 0x5648ddb58cb0 2024-04-02T14:50:15.997 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:16.004+0000 7f129d400640 1 --2- 172.21.15.90:0/4196158329 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5648ddb58cb0 0x5648ddb5b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:16.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:16.007+0000 7f127fe00640 10 client.30058 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:50:16.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:16.007+0000 7f129d400640 1 --2- 172.21.15.90:0/4196158329 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5648ddb58cb0 0x5648ddb5b0a0 crc :-1 s=READY pgs=391 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b8f96d321afcf161 server_cookie=e158fd3bad525efc in_seq=0 out_seq=0 2024-04-02T14:50:16.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:16.009+0000 7f127fe00640 1 -- 172.21.15.90:0/4196158329 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x5648dda38cb0 con 0x5648ddb58cb0 2024-04-02T14:50:16.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:16.009+0000 7f127fe00640 10 client.30058 handle_command_reply: tid=0 2024-04-02T14:50:16.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:16.009+0000 7f12a7d99740 1 client.30058 shutdown 2024-04-02T14:50:16.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:16.010+0000 7f12a7d99740 1 -- 172.21.15.90:0/4196158329 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5648ddac77b0 msgr2=0x5648ddacaad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:16.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:16.010+0000 7f12a7d99740 1 --2- 172.21.15.90:0/4196158329 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5648ddac77b0 0x5648ddacaad0 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f12900efe70 tx=0x7f129002bc30 comp rx=0 tx=0).stop 2024-04-02T14:50:16.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:16.010+0000 7f12a7d99740 1 -- 172.21.15.90:0/4196158329 shutdown_connections 2024-04-02T14:50:16.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:16.010+0000 7f12a7d99740 1 --2- 172.21.15.90:0/4196158329 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x5648ddb58cb0 0x5648ddb5b0a0 crc :-1 s=READY pgs=391 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:16.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:16.010+0000 7f12a7d99740 1 --2- 172.21.15.90:0/4196158329 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5648ddad0cb0 0x5648ddacb550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:16.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:16.010+0000 7f12a7d99740 1 --2- 172.21.15.90:0/4196158329 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x5648ddac80d0 0x5648ddacb010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:16.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:16.010+0000 7f12a7d99740 1 --2- 172.21.15.90:0/4196158329 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5648ddac77b0 0x5648ddacaad0 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:16.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:16.010+0000 7f127fe00640 0 client.30058 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:50:16.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:16.010+0000 7f12a7d99740 1 -- 172.21.15.90:0/4196158329 >> 172.21.15.90:0/4196158329 conn(0x5648ddac1ee0 msgr2=0x5648ddac3ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:50:16.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:16.012+0000 7f12a7d99740 1 -- 172.21.15.90:0/4196158329 shutdown_connections 2024-04-02T14:50:16.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:16.012+0000 7f12a7d99740 1 -- 172.21.15.90:0/4196158329 wait complete. 2024-04-02T14:50:16.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:16.012+0000 7f12a7d99740 20 client.30058 trim_cache size 0 max 16384 2024-04-02T14:50:16.006 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-02T14:50:16.006 INFO:teuthology.orchestra.run.smithi090.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-02T14:50:16.006 INFO:teuthology.orchestra.run.smithi090.stdout: "scrubs": { 2024-04-02T14:50:16.006 INFO:teuthology.orchestra.run.smithi090.stdout: "cc3a691c-35b3-48eb-b236-e1471312d03a": { 2024-04-02T14:50:16.007 INFO:teuthology.orchestra.run.smithi090.stdout: "path": "/", 2024-04-02T14:50:16.007 INFO:teuthology.orchestra.run.smithi090.stdout: "tag": "cc3a691c-35b3-48eb-b236-e1471312d03a", 2024-04-02T14:50:16.007 INFO:teuthology.orchestra.run.smithi090.stdout: "options": "recursive,force" 2024-04-02T14:50:16.007 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:16.007 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:16.007 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-02T14:50:16.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:16.014+0000 7f12a5000640 1 -- 172.21.15.90:0/2920380790 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f12840a3f20 msgr2=0x7f12840a63e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:16.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:16.014+0000 7f12a5000640 1 --2- 172.21.15.90:0/2920380790 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f12840a3f20 0x7f12840a63e0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f12a0069c50 tx=0x7f1298009500 comp rx=0 tx=0).stop 2024-04-02T14:50:16.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:16.014+0000 7f12a5000640 1 -- 172.21.15.90:0/2920380790 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f12a0106a00 msgr2=0x7f12a01a4630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:16.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:16.014+0000 7f12a5000640 1 --2- 172.21.15.90:0/2920380790 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f12a0106a00 0x7f12a01a4630 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7f129002c240 tx=0x7f12900042f0 comp rx=0 tx=0).stop 2024-04-02T14:50:16.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:16.015+0000 7f12a5000640 1 -- 172.21.15.90:0/2920380790 shutdown_connections 2024-04-02T14:50:16.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:16.015+0000 7f12a5000640 1 --2- 172.21.15.90:0/2920380790 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f12840a3f20 0x7f12840a63e0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:16.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:16.015+0000 7f12a5000640 1 --2- 172.21.15.90:0/2920380790 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f12a019e7a0 0x7f12a019ec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:16.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:16.015+0000 7f12a5000640 1 --2- 172.21.15.90:0/2920380790 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f12a0106e70 0x7f12a01a4b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:16.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:16.015+0000 7f12a5000640 1 --2- 172.21.15.90:0/2920380790 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f12a0106a00 0x7f12a01a4630 unknown :-1 s=CLOSED pgs=1029 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:16.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:16.015+0000 7f12a5000640 1 -- 172.21.15.90:0/2920380790 >> 172.21.15.90:0/2920380790 conn(0x7f12a00779e0 msgr2=0x7f12a01034e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:50:16.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:16.015+0000 7f12a5000640 1 -- 172.21.15.90:0/2920380790 shutdown_connections 2024-04-02T14:50:16.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:16.015+0000 7f12a5000640 1 -- 172.21.15.90:0/2920380790 wait complete. 2024-04-02T14:50:16.044 INFO:tasks.cephfs.filesystem:scrub status for tag:cc3a691c-35b3-48eb-b236-e1471312d03a - {'path': '/', 'tag': 'cc3a691c-35b3-48eb-b236-e1471312d03a', 'options': 'recursive,force'} 2024-04-02T14:50:16.430 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:50:16 smithi090 ceph-mon[31674]: pgmap v481: 97 pgs: 97 active+clean; 4.1 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 14 MiB/s wr, 1.33k op/s 2024-04-02T14:50:16.430 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:50:16 smithi090 ceph-mon[31674]: osdmap e109: 12 total, 12 up, 12 in 2024-04-02T14:50:16.430 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:50:16 smithi090 ceph-mon[31674]: scrub queued for path: / 2024-04-02T14:50:16.430 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:50:16 smithi090 ceph-mon[31674]: scrub summary: idle+waiting paths [/] 2024-04-02T14:50:16.430 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:50:16 smithi090 ceph-mon[31674]: scrub summary: active paths [/] 2024-04-02T14:50:16.602 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:50:16 smithi092 ceph-mon[36497]: pgmap v481: 97 pgs: 97 active+clean; 4.1 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 14 MiB/s wr, 1.33k op/s 2024-04-02T14:50:16.602 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:50:16 smithi092 ceph-mon[36497]: osdmap e109: 12 total, 12 up, 12 in 2024-04-02T14:50:16.602 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:50:16 smithi092 ceph-mon[36497]: scrub queued for path: / 2024-04-02T14:50:16.602 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:50:16 smithi092 ceph-mon[36497]: scrub summary: idle+waiting paths [/] 2024-04-02T14:50:16.602 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:50:16 smithi092 ceph-mon[36497]: scrub summary: active paths [/] 2024-04-02T14:50:16.678 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:50:16 smithi177 ceph-mon[36431]: pgmap v481: 97 pgs: 97 active+clean; 4.1 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 14 MiB/s wr, 1.33k op/s 2024-04-02T14:50:16.678 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:50:16 smithi177 ceph-mon[36431]: osdmap e109: 12 total, 12 up, 12 in 2024-04-02T14:50:16.678 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:50:16 smithi177 ceph-mon[36431]: scrub queued for path: / 2024-04-02T14:50:16.678 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:50:16 smithi177 ceph-mon[36431]: scrub summary: idle+waiting paths [/] 2024-04-02T14:50:16.678 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:50:16 smithi177 ceph-mon[36431]: scrub summary: active paths [/] 2024-04-02T14:50:16.823 INFO:teuthology.orchestra.run.smithi090.stderr:progress: 885.0 s, 1207.8 tps, lat 83.281 ms stddev 104.854 2024-04-02T14:50:17.602 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:50:17 smithi092 ceph-mon[36497]: scrub summary: idle+waiting paths [/] 2024-04-02T14:50:17.602 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:50:17 smithi092 ceph-mon[36497]: scrub summary: idle 2024-04-02T14:50:17.678 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:50:17 smithi177 ceph-mon[36431]: scrub summary: idle+waiting paths [/] 2024-04-02T14:50:17.678 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:50:17 smithi177 ceph-mon[36431]: scrub summary: idle 2024-04-02T14:50:17.679 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:50:17 smithi090 ceph-mon[31674]: scrub summary: idle+waiting paths [/] 2024-04-02T14:50:17.680 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:50:17 smithi090 ceph-mon[31674]: scrub summary: idle 2024-04-02T14:50:18.580 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:50:18 smithi090 ceph-mon[31674]: pgmap v483: 97 pgs: 97 active+clean; 4.1 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 2.7 MiB/s wr, 304 op/s 2024-04-02T14:50:18.602 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:50:18 smithi092 ceph-mon[36497]: pgmap v483: 97 pgs: 97 active+clean; 4.1 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 2.7 MiB/s wr, 304 op/s 2024-04-02T14:50:18.678 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:50:18 smithi177 ceph-mon[36431]: pgmap v483: 97 pgs: 97 active+clean; 4.1 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 2.7 MiB/s wr, 304 op/s 2024-04-02T14:50:20.430 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:50:20 smithi090 ceph-mon[31674]: pgmap v484: 97 pgs: 97 active+clean; 4.1 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 2.7 MiB/s wr, 304 op/s 2024-04-02T14:50:20.602 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:50:20 smithi092 ceph-mon[36497]: pgmap v484: 97 pgs: 97 active+clean; 4.1 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 2.7 MiB/s wr, 304 op/s 2024-04-02T14:50:20.678 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:50:20 smithi177 ceph-mon[36431]: pgmap v484: 97 pgs: 97 active+clean; 4.1 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 2.7 MiB/s wr, 304 op/s 2024-04-02T14:50:21.823 INFO:teuthology.orchestra.run.smithi090.stderr:progress: 890.0 s, 1499.6 tps, lat 67.311 ms stddev 81.114 2024-04-02T14:50:22.602 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:50:22 smithi092 ceph-mon[36497]: pgmap v485: 97 pgs: 97 active+clean; 4.1 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 5.2 MiB/s wr, 673 op/s 2024-04-02T14:50:22.678 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:50:22 smithi177 ceph-mon[36431]: pgmap v485: 97 pgs: 97 active+clean; 4.1 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 5.2 MiB/s wr, 673 op/s 2024-04-02T14:50:22.679 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:50:22 smithi090 ceph-mon[31674]: pgmap v485: 97 pgs: 97 active+clean; 4.1 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 5.2 MiB/s wr, 673 op/s 2024-04-02T14:50:24.602 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:50:24 smithi092 ceph-mon[36497]: pgmap v486: 97 pgs: 97 active+clean; 4.1 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 5.2 MiB/s wr, 673 op/s 2024-04-02T14:50:24.678 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:50:24 smithi177 ceph-mon[36431]: pgmap v486: 97 pgs: 97 active+clean; 4.1 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 5.2 MiB/s wr, 673 op/s 2024-04-02T14:50:24.679 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:50:24 smithi090 ceph-mon[31674]: pgmap v486: 97 pgs: 97 active+clean; 4.1 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 5.2 MiB/s wr, 673 op/s 2024-04-02T14:50:24.969 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:24.976+0000 7fcff8600640 1 -- 172.21.15.90:0/3831296564 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcff0107bd0 msgr2=0x7fcff0107fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:24.969 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:24.976+0000 7fcff8600640 1 --2- 172.21.15.90:0/3831296564 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcff0107bd0 0x7fcff0107fb0 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7fcfe40098c0 tx=0x7fcfe402bb50 comp rx=0 tx=0).stop 2024-04-02T14:50:24.969 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:24.977+0000 7fcff8600640 1 -- 172.21.15.90:0/3831296564 shutdown_connections 2024-04-02T14:50:24.970 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:24.977+0000 7fcff8600640 1 --2- 172.21.15.90:0/3831296564 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcff0069b20 0x7fcff010e990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:24.970 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:24.977+0000 7fcff8600640 1 --2- 172.21.15.90:0/3831296564 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcff0069180 0x7fcff00695e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:24.970 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:24.977+0000 7fcff8600640 1 --2- 172.21.15.90:0/3831296564 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcff0107bd0 0x7fcff0107fb0 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:24.970 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:24.977+0000 7fcff8600640 1 -- 172.21.15.90:0/3831296564 >> 172.21.15.90:0/3831296564 conn(0x7fcff0077850 msgr2=0x7fcff0077c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:50:24.970 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:24.977+0000 7fcff8600640 1 -- 172.21.15.90:0/3831296564 shutdown_connections 2024-04-02T14:50:24.970 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:24.977+0000 7fcff8600640 1 -- 172.21.15.90:0/3831296564 wait complete. 2024-04-02T14:50:24.971 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:24.978+0000 7fcff8600640 1 Processor -- start 2024-04-02T14:50:24.971 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:24.978+0000 7fcff8600640 1 -- start start 2024-04-02T14:50:24.971 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:24.978+0000 7fcff8600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcff0069180 0x7fcff0100d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:24.971 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:24.978+0000 7fcff8600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcff0069b20 0x7fcff0101260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:24.971 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:24.978+0000 7fcff8600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcff0107bd0 0x7fcff01017a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:24.971 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:24.978+0000 7fcff8600640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fcff0111ed0 con 0x7fcff0069180 2024-04-02T14:50:24.972 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:24.978+0000 7fcff8600640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fcff0111d50 con 0x7fcff0107bd0 2024-04-02T14:50:24.972 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:24.978+0000 7fcff8600640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fcff0112050 con 0x7fcff0069b20 2024-04-02T14:50:24.972 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:24.978+0000 7fcff5200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcff0069b20 0x7fcff0101260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:24.972 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:24.978+0000 7fcff5c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcff0069180 0x7fcff0100d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:24.972 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:24.978+0000 7fcff5200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcff0069b20 0x7fcff0101260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:43998/0 (socket says 172.21.15.90:43998) 2024-04-02T14:50:24.972 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:24.978+0000 7fcff5200640 1 -- 172.21.15.90:0/93376278 learned_addr learned my addr 172.21.15.90:0/93376278 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:50:24.972 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:24.979+0000 7fcff6600640 1 --2- 172.21.15.90:0/93376278 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcff0107bd0 0x7fcff01017a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:24.972 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:24.979+0000 7fcff5200640 1 -- 172.21.15.90:0/93376278 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcff0107bd0 msgr2=0x7fcff01017a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:24.972 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:24.979+0000 7fcff5200640 1 --2- 172.21.15.90:0/93376278 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcff0107bd0 0x7fcff01017a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:24.972 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:24.979+0000 7fcff5200640 1 -- 172.21.15.90:0/93376278 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcff0069180 msgr2=0x7fcff0100d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:24.972 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:24.979+0000 7fcff5200640 1 --2- 172.21.15.90:0/93376278 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcff0069180 0x7fcff0100d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:24.972 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:24.979+0000 7fcff5200640 1 -- 172.21.15.90:0/93376278 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcff019c430 con 0x7fcff0069b20 2024-04-02T14:50:24.973 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:24.979+0000 7fcff5200640 1 --2- 172.21.15.90:0/93376278 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcff0069b20 0x7fcff0101260 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7fcfdc00ea60 tx=0x7fcfdc00ef20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:50:24.973 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:24.979+0000 7fcfee000640 1 -- 172.21.15.90:0/93376278 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfdc00cde0 con 0x7fcff0069b20 2024-04-02T14:50:24.973 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:24.980+0000 7fcff8600640 1 -- 172.21.15.90:0/93376278 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcff019c720 con 0x7fcff0069b20 2024-04-02T14:50:24.973 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:24.980+0000 7fcfee000640 1 -- 172.21.15.90:0/93376278 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcfdc0035f0 con 0x7fcff0069b20 2024-04-02T14:50:24.973 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:24.980+0000 7fcff8600640 1 -- 172.21.15.90:0/93376278 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcff019ccc0 con 0x7fcff0069b20 2024-04-02T14:50:24.974 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:24.980+0000 7fcfee000640 1 -- 172.21.15.90:0/93376278 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfdc010670 con 0x7fcff0069b20 2024-04-02T14:50:24.978 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:24.986+0000 7fcfee000640 1 -- 172.21.15.90:0/93376278 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fcfdc021430 con 0x7fcff0069b20 2024-04-02T14:50:24.981 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:24.988+0000 7fcffb381740 1 Processor -- start 2024-04-02T14:50:24.981 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:24.988+0000 7fcfee000640 1 --2- 172.21.15.90:0/93376278 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fcfd80a3e40 0x7fcfd80a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:24.983 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:24.989+0000 7fcfee000640 1 -- 172.21.15.90:0/93376278 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 8550+0+0 (secure 0 0 0) 0x7fcfdc0d09d0 con 0x7fcff0069b20 2024-04-02T14:50:24.983 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:24.989+0000 7fcffb381740 1 -- start start 2024-04-02T14:50:24.984 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:24.989+0000 7fcffb381740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56323f902eb0 0x56323f903290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:24.984 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:24.989+0000 7fcffb381740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56323f9037d0 0x56323f90be70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:24.984 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:24.989+0000 7fcffb381740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56323f90c3b0 0x56323f90e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:24.984 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:24.989+0000 7fcffb381740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x56323f867d70 con 0x56323f9037d0 2024-04-02T14:50:24.984 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:24.989+0000 7fcffb381740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x56323f8102a0 con 0x56323f90c3b0 2024-04-02T14:50:24.984 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:24.989+0000 7fcffb381740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x56323f863d20 con 0x56323f902eb0 2024-04-02T14:50:24.984 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:24.990+0000 7fcff6600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56323f90c3b0 0x56323f90e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:24.984 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:24.990+0000 7fcff6600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56323f90c3b0 0x56323f90e7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:43528/0 (socket says 172.21.15.90:43528) 2024-04-02T14:50:24.985 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:24.990+0000 7fcff6600640 1 -- 172.21.15.90:0/1933797798 learned_addr learned my addr 172.21.15.90:0/1933797798 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:50:24.985 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:24.990+0000 7fcff6600640 1 --2- 172.21.15.90:0/93376278 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fcfd80a3e40 0x7fcfd80a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:24.985 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:24.990+0000 7fcff5c00640 1 --2- 172.21.15.90:0/1933797798 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56323f902eb0 0x56323f903290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:24.985 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:24.991+0000 7fcff6600640 1 --2- 172.21.15.90:0/93376278 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fcfd80a3e40 0x7fcfd80a6300 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fcfe8009590 tx=0x7fcfe8009bf0 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:50:24.985 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:24.991+0000 7fcff5c00640 1 -- 172.21.15.90:0/1933797798 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56323f90c3b0 msgr2=0x56323f90e7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:50:24.985 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:24.991+0000 7fcff5c00640 1 --2- 172.21.15.90:0/1933797798 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56323f90c3b0 0x56323f90e7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:24.985 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:24.991+0000 7fcff5c00640 1 -- 172.21.15.90:0/1933797798 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56323f9037d0 msgr2=0x56323f90be70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:50:24.985 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:24.991+0000 7fcff5c00640 1 --2- 172.21.15.90:0/1933797798 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56323f9037d0 0x56323f90be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:24.985 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:24.991+0000 7fcff5c00640 1 -- 172.21.15.90:0/1933797798 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56323f824840 con 0x56323f902eb0 2024-04-02T14:50:24.985 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:24.992+0000 7fcff5c00640 1 --2- 172.21.15.90:0/1933797798 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56323f902eb0 0x56323f903290 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7fcfe4037040 tx=0x7fcfe402cc30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ce21fc80fa0c6438 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:50:24.990 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:24.998+0000 7fcfe2a00640 1 -- 172.21.15.90:0/1933797798 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfe4055030 con 0x56323f902eb0 2024-04-02T14:50:24.991 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:24.998+0000 7fcfe2a00640 1 -- 172.21.15.90:0/1933797798 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcfe402cda0 con 0x56323f902eb0 2024-04-02T14:50:24.991 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:24.998+0000 7fcfe2a00640 1 -- 172.21.15.90:0/1933797798 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfe4003850 con 0x56323f902eb0 2024-04-02T14:50:24.991 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:24.999+0000 7fcffb381740 1 -- 172.21.15.90:0/1933797798 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56323f902eb0 msgr2=0x56323f903290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:24.991 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:24.999+0000 7fcffb381740 1 --2- 172.21.15.90:0/1933797798 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56323f902eb0 0x56323f903290 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7fcfe4037040 tx=0x7fcfe402cc30 comp rx=0 tx=0).stop 2024-04-02T14:50:24.993 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:25.000+0000 7fcffb381740 1 -- 172.21.15.90:0/1933797798 shutdown_connections 2024-04-02T14:50:24.993 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:25.001+0000 7fcffb381740 1 --2- 172.21.15.90:0/1933797798 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56323f90c3b0 0x56323f90e7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:24.993 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:25.001+0000 7fcffb381740 1 --2- 172.21.15.90:0/1933797798 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56323f9037d0 0x56323f90be70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:24.993 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:25.001+0000 7fcffb381740 1 --2- 172.21.15.90:0/1933797798 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56323f902eb0 0x56323f903290 secure :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7fcfe4037040 tx=0x7fcfe402cc30 comp rx=0 tx=0).stop 2024-04-02T14:50:24.993 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:25.001+0000 7fcffb381740 1 -- 172.21.15.90:0/1933797798 >> 172.21.15.90:0/1933797798 conn(0x56323f8fd5e0 msgr2=0x56323f8fd9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:50:24.995 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:25.002+0000 7fcffb381740 1 -- 172.21.15.90:0/1933797798 shutdown_connections 2024-04-02T14:50:24.995 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:25.002+0000 7fcffb381740 1 -- 172.21.15.90:0/1933797798 wait complete. 2024-04-02T14:50:24.995 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:25.002+0000 7fcffb381740 1 -- start start 2024-04-02T14:50:24.995 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:25.003+0000 7fcffb381740 1 Processor -- start 2024-04-02T14:50:25.000 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:25.006+0000 7fcffb381740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56323f9037d0 0x56323f9095e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:25.000 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:25.006+0000 7fcffb381740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56323f90c3b0 0x56323f909b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:25.000 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:25.007+0000 7fcffb381740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56323f90a0b0 0x56323f918610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:25.001 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:25.007+0000 7fcff5200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56323f90c3b0 0x56323f909b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:25.001 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:25.007+0000 7fcffb381740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x56323f90f140 con 0x56323f9037d0 2024-04-02T14:50:25.001 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:25.007+0000 7fcffb381740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x56323f823660 con 0x56323f90c3b0 2024-04-02T14:50:25.001 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:25.007+0000 7fcffb381740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x56323f90f2c0 con 0x56323f90a0b0 2024-04-02T14:50:25.001 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:25.007+0000 7fcff5c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56323f9037d0 0x56323f9095e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:25.001 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:25.007+0000 7fcffb381740 10 client.0 _ll_register_callbacks cb 0x56323f823050 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:50:25.001 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:25.007+0000 7fcff5c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56323f9037d0 0x56323f9095e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:35180/0 (socket says 172.21.15.90:35180) 2024-04-02T14:50:25.001 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:25.007+0000 7fcff5c00640 1 -- 172.21.15.90:0/2042299506 learned_addr learned my addr 172.21.15.90:0/2042299506 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:50:25.001 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:25.008+0000 7fcff6600640 1 --2- 172.21.15.90:0/2042299506 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56323f90a0b0 0x56323f918610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:25.001 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:25.008+0000 7fcff5200640 1 -- 172.21.15.90:0/2042299506 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56323f90a0b0 msgr2=0x56323f918610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:50:25.001 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:25.008+0000 7fcff5200640 1 --2- 172.21.15.90:0/2042299506 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56323f90a0b0 0x56323f918610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:25.001 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:25.008+0000 7fcff5200640 1 -- 172.21.15.90:0/2042299506 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56323f9037d0 msgr2=0x56323f9095e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:50:25.002 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:25.008+0000 7fcff5200640 1 --2- 172.21.15.90:0/2042299506 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56323f9037d0 0x56323f9095e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:25.002 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:25.008+0000 7fcff5200640 1 -- 172.21.15.90:0/2042299506 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56323f832e80 con 0x56323f90c3b0 2024-04-02T14:50:25.003 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:25.009+0000 7fcff5200640 1 --2- 172.21.15.90:0/2042299506 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56323f90c3b0 0x56323f909b20 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7fcfdc0ce0a0 tx=0x7fcfdc09c380 comp rx=0 tx=0).ready entity=mon.2 client_cookie=162bb6f2c74193a3 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:50:25.003 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:25.009+0000 7fcfd7400640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:50:25.003 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:25.009+0000 7fcfd7400640 1 -- 172.21.15.90:0/2042299506 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfdc0a5560 con 0x56323f90c3b0 2024-04-02T14:50:25.003 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:25.009+0000 7fcfd7400640 1 -- 172.21.15.90:0/2042299506 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 8550+0+0 (secure 0 0 0) 0x7fcfdc0870c0 con 0x56323f90c3b0 2024-04-02T14:50:25.003 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:25.009+0000 7fcffb381740 1 -- 172.21.15.90:0/2042299506 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcfe402cda0 con 0x56323f90c3b0 2024-04-02T14:50:25.003 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:25.010+0000 7fcfd7400640 1 -- 172.21.15.90:0/2042299506 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fcfdc0872e0 con 0x56323f90c3b0 2024-04-02T14:50:25.003 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:25.010+0000 7fcffb381740 10 client.30275 fetch_fsmap learned FSMap version 27 2024-04-02T14:50:25.003 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:25.010+0000 7fcffb381740 1 -- 172.21.15.90:0/2042299506 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x56323f57c210 con 0x56323f90c3b0 2024-04-02T14:50:25.003 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:25.010+0000 7fcfd7400640 1 -- 172.21.15.90:0/2042299506 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcfe402cda0 con 0x56323f90c3b0 2024-04-02T14:50:25.003 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:25.011+0000 7fcfd7400640 1 -- 172.21.15.90:0/2042299506 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fcfdc0bbb40 con 0x56323f90c3b0 2024-04-02T14:50:25.004 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:25.011+0000 7fcffb381740 10 client.30275 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:50:25.004 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:25.011+0000 7fcffb381740 10 client.30275 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-02T14:50:25.004 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:25.011+0000 7fcffb381740 20 client.30275 populate_metadata read hostname 'smithi090' 2024-04-02T14:50:25.004 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:25.011+0000 7fcffb381740 1 --2- 172.21.15.90:0/2042299506 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x56323f91cec0 0x56323f91f2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:25.004 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:25.011+0000 7fcffb381740 4 client.30275 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:50:25.004 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:25.011+0000 7fcffb381740 1 -- 172.21.15.90:0/2042299506 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56323f817cd0 con 0x56323f91cec0 2024-04-02T14:50:25.005 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:25.012+0000 7fcff5c00640 1 --2- 172.21.15.90:0/2042299506 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x56323f91cec0 0x56323f91f2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:25.005 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:25.013+0000 7fcff5c00640 1 --2- 172.21.15.90:0/2042299506 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x56323f91cec0 0x56323f91f2b0 crc :-1 s=READY pgs=392 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7bddddd5d6a86929 server_cookie=e253738ba043831d in_seq=0 out_seq=0 2024-04-02T14:50:25.005 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:25.013+0000 7fcfd7400640 10 client.30275 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:50:25.007 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:25.015+0000 7fcfd7400640 1 -- 172.21.15.90:0/2042299506 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56323f817cd0 con 0x56323f91cec0 2024-04-02T14:50:25.007 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:25.015+0000 7fcfd7400640 10 client.30275 handle_command_reply: tid=0 2024-04-02T14:50:25.007 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:25.015+0000 7fcffb381740 1 client.30275 shutdown 2024-04-02T14:50:25.010 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:25.018+0000 7fcffb381740 1 -- 172.21.15.90:0/2042299506 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56323f90c3b0 msgr2=0x56323f909b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:25.011 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:25.018+0000 7fcffb381740 1 --2- 172.21.15.90:0/2042299506 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56323f90c3b0 0x56323f909b20 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7fcfdc0ce0a0 tx=0x7fcfdc09c380 comp rx=0 tx=0).stop 2024-04-02T14:50:25.012 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:25.019+0000 7fcffb381740 1 -- 172.21.15.90:0/2042299506 shutdown_connections 2024-04-02T14:50:25.012 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:25.019+0000 7fcffb381740 1 --2- 172.21.15.90:0/2042299506 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x56323f91cec0 0x56323f91f2b0 crc :-1 s=READY pgs=392 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:25.012 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:25.019+0000 7fcffb381740 1 --2- 172.21.15.90:0/2042299506 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56323f90a0b0 0x56323f918610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:25.012 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:25.019+0000 7fcffb381740 1 --2- 172.21.15.90:0/2042299506 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56323f90c3b0 0x56323f909b20 unknown :-1 s=CLOSED pgs=1031 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:25.012 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:25.019+0000 7fcffb381740 1 --2- 172.21.15.90:0/2042299506 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56323f9037d0 0x56323f9095e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:25.012 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:25.019+0000 7fcfd7400640 0 client.30275 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:50:25.012 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:25.019+0000 7fcffb381740 1 -- 172.21.15.90:0/2042299506 >> 172.21.15.90:0/2042299506 conn(0x56323f8fd5e0 msgr2=0x56323f8ff330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:50:25.012 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:25.019+0000 7fcffb381740 1 -- 172.21.15.90:0/2042299506 shutdown_connections 2024-04-02T14:50:25.013 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:25.019+0000 7fcffb381740 1 -- 172.21.15.90:0/2042299506 wait complete. 2024-04-02T14:50:25.013 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:25.020+0000 7fcffb381740 20 client.30275 trim_cache size 0 max 16384 2024-04-02T14:50:25.044 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:25.052+0000 7fcffb381740 1 Processor -- start 2024-04-02T14:50:25.045 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:25.052+0000 7fcffb381740 1 -- start start 2024-04-02T14:50:25.045 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:25.053+0000 7fcffb381740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56323f9037d0 0x56323f8fc4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:25.045 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:25.053+0000 7fcffb381740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56323f90a0b0 0x56323f924b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:25.046 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:25.053+0000 7fcffb381740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56323f90c3b0 0x56323f9250c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:25.046 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:25.053+0000 7fcffb381740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fcfd000e800 con 0x56323f90a0b0 2024-04-02T14:50:25.046 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:25.053+0000 7fcffb381740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fcfd0010340 con 0x56323f90c3b0 2024-04-02T14:50:25.046 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:25.053+0000 7fcffb381740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fcfd000eed0 con 0x56323f9037d0 2024-04-02T14:50:25.046 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:25.054+0000 7fcff6600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56323f90c3b0 0x56323f9250c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:25.046 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:25.054+0000 7fcff5c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56323f9037d0 0x56323f8fc4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:25.046 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:25.054+0000 7fcff6600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56323f90c3b0 0x56323f9250c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:43552/0 (socket says 172.21.15.90:43552) 2024-04-02T14:50:25.046 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:25.054+0000 7fcff6600640 1 -- 172.21.15.90:0/2518003849 learned_addr learned my addr 172.21.15.90:0/2518003849 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:50:25.046 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:25.054+0000 7fcff5c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56323f9037d0 0x56323f8fc4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:44028/0 (socket says 172.21.15.90:44028) 2024-04-02T14:50:25.047 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:25.054+0000 7fcff5c00640 1 -- 172.21.15.90:0/2518003849 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56323f90c3b0 msgr2=0x56323f9250c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:50:25.047 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:25.054+0000 7fcff5c00640 1 --2- 172.21.15.90:0/2518003849 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56323f90c3b0 0x56323f9250c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:25.047 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:25.054+0000 7fcff5c00640 1 -- 172.21.15.90:0/2518003849 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56323f90a0b0 msgr2=0x56323f924b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:50:25.047 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:25.054+0000 7fcff5c00640 1 --2- 172.21.15.90:0/2518003849 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56323f90a0b0 0x56323f924b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:25.047 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:25.054+0000 7fcff5c00640 1 -- 172.21.15.90:0/2518003849 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56323f648620 con 0x56323f9037d0 2024-04-02T14:50:25.048 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:25.055+0000 7fcff5c00640 1 --2- 172.21.15.90:0/2518003849 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56323f9037d0 0x56323f8fc4c0 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7fcfe40097b0 tx=0x7fcfe402cfd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=34afeef5c223ae74 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:50:25.048 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:25.055+0000 7fcfe2000640 1 -- 172.21.15.90:0/2518003849 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfe4055040 con 0x56323f9037d0 2024-04-02T14:50:25.048 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:25.056+0000 7fcfe2000640 1 -- 172.21.15.90:0/2518003849 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcfe4056040 con 0x56323f9037d0 2024-04-02T14:50:25.050 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:25.056+0000 7fcfe2000640 1 -- 172.21.15.90:0/2518003849 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfe4003b20 con 0x56323f9037d0 2024-04-02T14:50:25.050 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:25.058+0000 7fcffb381740 1 -- 172.21.15.90:0/2518003849 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56323f9037d0 msgr2=0x56323f8fc4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:25.051 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:25.058+0000 7fcffb381740 1 --2- 172.21.15.90:0/2518003849 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56323f9037d0 0x56323f8fc4c0 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7fcfe40097b0 tx=0x7fcfe402cfd0 comp rx=0 tx=0).stop 2024-04-02T14:50:25.051 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:25.059+0000 7fcffb381740 1 -- 172.21.15.90:0/2518003849 shutdown_connections 2024-04-02T14:50:25.051 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:25.059+0000 7fcffb381740 1 --2- 172.21.15.90:0/2518003849 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56323f90c3b0 0x56323f9250c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:25.051 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:25.059+0000 7fcffb381740 1 --2- 172.21.15.90:0/2518003849 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56323f90a0b0 0x56323f924b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:25.052 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:25.059+0000 7fcffb381740 1 --2- 172.21.15.90:0/2518003849 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56323f9037d0 0x56323f8fc4c0 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:25.052 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:25.059+0000 7fcffb381740 1 -- 172.21.15.90:0/2518003849 >> 172.21.15.90:0/2518003849 conn(0x56323f8fd5e0 msgr2=0x56323f8fbdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:50:25.052 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:25.059+0000 7fcffb381740 1 -- 172.21.15.90:0/2518003849 shutdown_connections 2024-04-02T14:50:25.053 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:25.059+0000 7fcffb381740 1 -- 172.21.15.90:0/2518003849 wait complete. 2024-04-02T14:50:25.053 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:25.060+0000 7fcffb381740 1 -- start start 2024-04-02T14:50:25.053 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:25.060+0000 7fcffb381740 1 Processor -- start 2024-04-02T14:50:25.053 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:25.061+0000 7fcffb381740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56323f9037d0 0x56323f9066b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:25.054 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:25.062+0000 7fcffb381740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56323f90a0b0 0x56323f906bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:25.055 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:25.062+0000 7fcffb381740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56323f90c3b0 0x56323f9823f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:25.055 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:25.062+0000 7fcffb381740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fcfd000fc70 con 0x56323f9037d0 2024-04-02T14:50:25.055 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:25.062+0000 7fcffb381740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fcfd000f5a0 con 0x56323f90a0b0 2024-04-02T14:50:25.055 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:25.062+0000 7fcffb381740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fcfd000c800 con 0x56323f90c3b0 2024-04-02T14:50:25.055 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:25.062+0000 7fcffb381740 10 client.0 _ll_register_callbacks cb 0x7fcfdc0094f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:50:25.056 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:25.062+0000 7fcff5c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56323f9037d0 0x56323f9066b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:25.056 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:25.062+0000 7fcff5c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56323f9037d0 0x56323f9066b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:35190/0 (socket says 172.21.15.90:35190) 2024-04-02T14:50:25.056 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:25.062+0000 7fcff5c00640 1 -- 172.21.15.90:0/71949908 learned_addr learned my addr 172.21.15.90:0/71949908 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:50:25.056 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:25.063+0000 7fcff5200640 1 --2- 172.21.15.90:0/71949908 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56323f90a0b0 0x56323f906bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:25.056 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:25.063+0000 7fcff6600640 1 --2- 172.21.15.90:0/71949908 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56323f90c3b0 0x56323f9823f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:25.056 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:25.063+0000 7fcff5c00640 1 -- 172.21.15.90:0/71949908 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56323f90c3b0 msgr2=0x56323f9823f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:50:25.056 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:25.063+0000 7fcff5c00640 1 --2- 172.21.15.90:0/71949908 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56323f90c3b0 0x56323f9823f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:25.056 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:25.063+0000 7fcff5c00640 1 -- 172.21.15.90:0/71949908 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56323f90a0b0 msgr2=0x56323f906bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:50:25.056 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:25.063+0000 7fcff5c00640 1 --2- 172.21.15.90:0/71949908 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56323f90a0b0 0x56323f906bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:25.056 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:25.063+0000 7fcff5c00640 1 -- 172.21.15.90:0/71949908 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56323f982930 con 0x56323f9037d0 2024-04-02T14:50:25.056 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:25.063+0000 7fcff6600640 1 --2- 172.21.15.90:0/71949908 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56323f90c3b0 0x56323f9823f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-02T14:50:25.057 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:25.063+0000 7fcff5c00640 1 --2- 172.21.15.90:0/71949908 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56323f9037d0 0x56323f9066b0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7fcfe402bb90 tx=0x7fcfe4003660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16ad437db79f5556 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:50:25.057 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:25.064+0000 7fcfd7400640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-02T14:50:25.057 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:25.064+0000 7fcfd7400640 1 -- 172.21.15.90:0/71949908 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfe4055040 con 0x56323f9037d0 2024-04-02T14:50:25.057 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:25.064+0000 7fcfd7400640 1 -- 172.21.15.90:0/71949908 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 8550+0+0 (secure 0 0 0) 0x7fcfe4005030 con 0x56323f9037d0 2024-04-02T14:50:25.057 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:25.064+0000 7fcffb381740 1 -- 172.21.15.90:0/71949908 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcfe4056040 con 0x56323f9037d0 2024-04-02T14:50:25.058 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:25.065+0000 7fcfd7400640 1 -- 172.21.15.90:0/71949908 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fcfe4070050 con 0x56323f9037d0 2024-04-02T14:50:25.058 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:25.065+0000 7fcffb381740 10 client.18954 fetch_fsmap learned FSMap version 27 2024-04-02T14:50:25.058 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:25.065+0000 7fcffb381740 1 -- 172.21.15.90:0/71949908 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x56323f983a80 con 0x56323f9037d0 2024-04-02T14:50:25.059 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:25.065+0000 7fcfd7400640 1 -- 172.21.15.90:0/71949908 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcfe4056040 con 0x56323f9037d0 2024-04-02T14:50:25.060 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:25.067+0000 7fcfd7400640 1 -- 172.21.15.90:0/71949908 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fcfe406b030 con 0x56323f9037d0 2024-04-02T14:50:25.060 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:25.067+0000 7fcffb381740 10 client.18954 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:50:25.060 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:25.067+0000 7fcffb381740 10 client.18954 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-02T14:50:25.060 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:25.067+0000 7fcffb381740 20 client.18954 populate_metadata read hostname 'smithi090' 2024-04-02T14:50:25.060 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:25.067+0000 7fcffb381740 1 --2- 172.21.15.90:0/71949908 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x56323f983d80 0x56323f986170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:25.060 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:25.067+0000 7fcffb381740 4 client.18954 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-02T14:50:25.060 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:25.067+0000 7fcffb381740 1 -- 172.21.15.90:0/71949908 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56323f8752b0 con 0x56323f983d80 2024-04-02T14:50:25.060 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:25.068+0000 7fcff5200640 1 --2- 172.21.15.90:0/71949908 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x56323f983d80 0x56323f986170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:25.061 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:25.069+0000 7fcff5200640 1 --2- 172.21.15.90:0/71949908 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x56323f983d80 0x56323f986170 crc :-1 s=READY pgs=393 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d2cac807b2fd1160 server_cookie=120cbd853caa6d42 in_seq=0 out_seq=0 2024-04-02T14:50:25.061 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:25.069+0000 7fcfd7400640 10 client.18954 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:50:25.120 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:25.127+0000 7fcfd7400640 1 -- 172.21.15.90:0/71949908 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56323f8752b0 con 0x56323f983d80 2024-04-02T14:50:25.120 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:25.127+0000 7fcfd7400640 10 client.18954 handle_command_reply: tid=0 2024-04-02T14:50:25.120 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:25.127+0000 7fcffb381740 1 client.18954 shutdown 2024-04-02T14:50:25.120 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:25.128+0000 7fcffb381740 1 -- 172.21.15.90:0/71949908 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56323f9037d0 msgr2=0x56323f9066b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:25.120 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:25.128+0000 7fcffb381740 1 --2- 172.21.15.90:0/71949908 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56323f9037d0 0x56323f9066b0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7fcfe402bb90 tx=0x7fcfe4003660 comp rx=0 tx=0).stop 2024-04-02T14:50:25.121 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:25.128+0000 7fcffb381740 1 -- 172.21.15.90:0/71949908 shutdown_connections 2024-04-02T14:50:25.121 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:25.128+0000 7fcffb381740 1 --2- 172.21.15.90:0/71949908 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x56323f983d80 0x56323f986170 crc :-1 s=READY pgs=393 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:25.121 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:25.128+0000 7fcffb381740 1 --2- 172.21.15.90:0/71949908 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56323f90c3b0 0x56323f9823f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:25.121 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:25.128+0000 7fcffb381740 1 --2- 172.21.15.90:0/71949908 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x56323f90a0b0 0x56323f906bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:25.121 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:25.128+0000 7fcffb381740 1 --2- 172.21.15.90:0/71949908 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x56323f9037d0 0x56323f9066b0 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:25.121 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:25.128+0000 7fcffb381740 1 -- 172.21.15.90:0/71949908 >> 172.21.15.90:0/71949908 conn(0x56323f8fd5e0 msgr2=0x56323f8fc640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:50:25.121 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:25.128+0000 7fcfd7400640 0 client.18954 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:50:25.122 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:25.128+0000 7fcffb381740 1 -- 172.21.15.90:0/71949908 shutdown_connections 2024-04-02T14:50:25.122 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:25.128+0000 7fcffb381740 1 -- 172.21.15.90:0/71949908 wait complete. 2024-04-02T14:50:25.122 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:25.129+0000 7fcffb381740 20 client.18954 trim_cache size 0 max 16384 2024-04-02T14:50:25.122 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:{ 2024-04-02T14:50:25.122 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "message": "", 2024-04-02T14:50:25.122 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "return_code": 0 2024-04-02T14:50:25.122 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:} 2024-04-02T14:50:25.123 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:25.129+0000 7fcff8600640 1 -- 172.21.15.90:0/93376278 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fcfd80a3e40 msgr2=0x7fcfd80a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:25.123 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:25.129+0000 7fcff8600640 1 --2- 172.21.15.90:0/93376278 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fcfd80a3e40 0x7fcfd80a6300 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fcfe8009590 tx=0x7fcfe8009bf0 comp rx=0 tx=0).stop 2024-04-02T14:50:25.123 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:25.129+0000 7fcff8600640 1 -- 172.21.15.90:0/93376278 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcff0069b20 msgr2=0x7fcff0101260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:25.123 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:25.129+0000 7fcff8600640 1 --2- 172.21.15.90:0/93376278 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcff0069b20 0x7fcff0101260 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7fcfdc00ea60 tx=0x7fcfdc00ef20 comp rx=0 tx=0).stop 2024-04-02T14:50:25.123 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:25.130+0000 7fcff8600640 1 -- 172.21.15.90:0/93376278 shutdown_connections 2024-04-02T14:50:25.123 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:25.130+0000 7fcff8600640 1 --2- 172.21.15.90:0/93376278 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fcfd80a3e40 0x7fcfd80a6300 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:25.124 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:25.130+0000 7fcff8600640 1 --2- 172.21.15.90:0/93376278 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcff0107bd0 0x7fcff01017a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:25.124 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:25.130+0000 7fcff8600640 1 --2- 172.21.15.90:0/93376278 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcff0069b20 0x7fcff0101260 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:25.124 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:25.130+0000 7fcff8600640 1 --2- 172.21.15.90:0/93376278 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcff0069180 0x7fcff0100d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:25.124 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:25.130+0000 7fcff8600640 1 -- 172.21.15.90:0/93376278 >> 172.21.15.90:0/93376278 conn(0x7fcff0077850 msgr2=0x7fcff00fec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:50:25.124 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:25.130+0000 7fcff8600640 1 -- 172.21.15.90:0/93376278 shutdown_connections 2024-04-02T14:50:25.124 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:25.130+0000 7fcff8600640 1 -- 172.21.15.90:0/93376278 wait complete. 2024-04-02T14:50:26.602 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:50:26 smithi092 ceph-mon[36497]: pgmap v487: 97 pgs: 97 active+clean; 4.1 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 5.3 MiB/s wr, 660 op/s 2024-04-02T14:50:26.678 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:50:26 smithi177 ceph-mon[36431]: pgmap v487: 97 pgs: 97 active+clean; 4.1 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 5.3 MiB/s wr, 660 op/s 2024-04-02T14:50:26.680 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:50:26 smithi090 ceph-mon[31674]: pgmap v487: 97 pgs: 97 active+clean; 4.1 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 5.3 MiB/s wr, 660 op/s 2024-04-02T14:50:26.823 INFO:teuthology.orchestra.run.smithi090.stderr:progress: 895.0 s, 1270.4 tps, lat 78.695 ms stddev 87.148 2024-04-02T14:50:28.602 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:50:28 smithi092 ceph-mon[36497]: pgmap v488: 97 pgs: 97 active+clean; 4.1 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 5.4 MiB/s wr, 670 op/s 2024-04-02T14:50:28.678 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:50:28 smithi177 ceph-mon[36431]: pgmap v488: 97 pgs: 97 active+clean; 4.1 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 5.4 MiB/s wr, 670 op/s 2024-04-02T14:50:28.679 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:50:28 smithi090 ceph-mon[31674]: pgmap v488: 97 pgs: 97 active+clean; 4.1 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 5.4 MiB/s wr, 670 op/s 2024-04-02T14:50:30.602 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:50:30 smithi092 ceph-mon[36497]: pgmap v489: 97 pgs: 97 active+clean; 4.1 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 4.9 MiB/s wr, 599 op/s 2024-04-02T14:50:30.678 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:50:30 smithi177 ceph-mon[36431]: pgmap v489: 97 pgs: 97 active+clean; 4.1 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 4.9 MiB/s wr, 599 op/s 2024-04-02T14:50:30.682 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:50:30 smithi090 ceph-mon[31674]: pgmap v489: 97 pgs: 97 active+clean; 4.1 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 4.9 MiB/s wr, 599 op/s 2024-04-02T14:50:31.602 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:50:31 smithi092 ceph-mon[36497]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T14:50:31.678 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:50:31 smithi177 ceph-mon[36431]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T14:50:31.679 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:50:31 smithi090 ceph-mon[31674]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T14:50:31.823 INFO:teuthology.orchestra.run.smithi090.stderr:progress: 900.0 s, 2033.0 tps, lat 49.357 ms stddev 66.139 2024-04-02T14:50:32.074 INFO:teuthology.orchestra.run.smithi090.stdout:transaction type: 2024-04-02T14:50:32.075 INFO:teuthology.orchestra.run.smithi090.stdout:scaling factor: 32 2024-04-02T14:50:32.075 INFO:teuthology.orchestra.run.smithi090.stdout:query mode: simple 2024-04-02T14:50:32.075 INFO:teuthology.orchestra.run.smithi090.stdout:number of clients: 100 2024-04-02T14:50:32.075 INFO:teuthology.orchestra.run.smithi090.stdout:number of threads: 4 2024-04-02T14:50:32.075 INFO:teuthology.orchestra.run.smithi090.stdout:duration: 900 s 2024-04-02T14:50:32.075 INFO:teuthology.orchestra.run.smithi090.stdout:number of transactions actually processed: 1764556 2024-04-02T14:50:32.075 INFO:teuthology.orchestra.run.smithi090.stdout:latency average = 50.984 ms 2024-04-02T14:50:32.075 INFO:teuthology.orchestra.run.smithi090.stdout:latency stddev = 137.180 ms 2024-04-02T14:50:32.075 INFO:teuthology.orchestra.run.smithi090.stdout:tps = 1960.069791 (including connections establishing) 2024-04-02T14:50:32.075 INFO:teuthology.orchestra.run.smithi090.stdout:tps = 1960.084386 (excluding connections establishing) 2024-04-02T14:50:32.093 DEBUG:teuthology.orchestra.run.smithi090:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo systemctl stop postgresql' 2024-04-02T14:50:32.579 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:50:32 smithi177 ceph-mon[36431]: pgmap v490: 97 pgs: 97 active+clean; 4.1 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 150 KiB/s rd, 7.5 MiB/s wr, 875 op/s 2024-04-02T14:50:32.602 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:50:32 smithi092 ceph-mon[36497]: pgmap v490: 97 pgs: 97 active+clean; 4.1 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 150 KiB/s rd, 7.5 MiB/s wr, 875 op/s 2024-04-02T14:50:32.681 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:50:32 smithi090 ceph-mon[31674]: pgmap v490: 97 pgs: 97 active+clean; 4.1 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 150 KiB/s rd, 7.5 MiB/s wr, 875 op/s 2024-04-02T14:50:33.602 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:50:33 smithi092 ceph-mon[36497]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:50:33.602 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:50:33 smithi092 ceph-mon[36497]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T14:50:33.602 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:50:33 smithi092 ceph-mon[36497]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' 2024-04-02T14:50:33.678 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:50:33 smithi177 ceph-mon[36431]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:50:33.678 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:50:33 smithi177 ceph-mon[36431]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T14:50:33.678 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:50:33 smithi177 ceph-mon[36431]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' 2024-04-02T14:50:33.680 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:50:33 smithi090 ceph-mon[31674]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:50:33.680 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:50:33 smithi090 ceph-mon[31674]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T14:50:33.680 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:50:33 smithi090 ceph-mon[31674]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' 2024-04-02T14:50:34.602 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:50:34 smithi092 ceph-mon[36497]: pgmap v491: 97 pgs: 97 active+clean; 4.1 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 5.2 MiB/s wr, 535 op/s 2024-04-02T14:50:34.678 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:50:34 smithi177 ceph-mon[36431]: pgmap v491: 97 pgs: 97 active+clean; 4.1 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 5.2 MiB/s wr, 535 op/s 2024-04-02T14:50:34.680 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:50:34 smithi090 ceph-mon[31674]: pgmap v491: 97 pgs: 97 active+clean; 4.1 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 5.2 MiB/s wr, 535 op/s 2024-04-02T14:50:36.602 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:50:36 smithi092 ceph-mon[36497]: pgmap v492: 97 pgs: 97 active+clean; 4.2 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 7.1 MiB/s wr, 742 op/s 2024-04-02T14:50:36.678 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:50:36 smithi177 ceph-mon[36431]: pgmap v492: 97 pgs: 97 active+clean; 4.2 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 7.1 MiB/s wr, 742 op/s 2024-04-02T14:50:36.679 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:50:36 smithi090 ceph-mon[31674]: pgmap v492: 97 pgs: 97 active+clean; 4.2 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 7.1 MiB/s wr, 742 op/s 2024-04-02T14:50:38.324 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.331+0000 7f4482a00640 1 -- 172.21.15.90:0/2977006594 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f447c1048f0 msgr2=0x7f447c108930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:38.324 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.331+0000 7f4482a00640 1 --2- 172.21.15.90:0/2977006594 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f447c1048f0 0x7f447c108930 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7f4468009900 tx=0x7f446802bc60 comp rx=0 tx=0).stop 2024-04-02T14:50:38.325 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.332+0000 7f4482a00640 1 -- 172.21.15.90:0/2977006594 shutdown_connections 2024-04-02T14:50:38.325 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.332+0000 7f4482a00640 1 --2- 172.21.15.90:0/2977006594 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f447c109010 0x7f447c110be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:38.325 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.332+0000 7f4482a00640 1 --2- 172.21.15.90:0/2977006594 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f447c1048f0 0x7f447c108930 unknown :-1 s=CLOSED pgs=1032 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:38.325 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.332+0000 7f4482a00640 1 --2- 172.21.15.90:0/2977006594 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f447c103f40 0x7f447c104320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:38.325 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.332+0000 7f4482a00640 1 -- 172.21.15.90:0/2977006594 >> 172.21.15.90:0/2977006594 conn(0x7f447c0ff7c0 msgr2=0x7f447c101be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:50:38.327 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.334+0000 7f4482a00640 1 -- 172.21.15.90:0/2977006594 shutdown_connections 2024-04-02T14:50:38.327 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.334+0000 7f4482a00640 1 -- 172.21.15.90:0/2977006594 wait complete. 2024-04-02T14:50:38.328 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.335+0000 7f4482a00640 1 Processor -- start 2024-04-02T14:50:38.329 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.336+0000 7f4482a00640 1 -- start start 2024-04-02T14:50:38.330 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.336+0000 7f4482a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f447c103f40 0x7f447c19a650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:38.330 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.336+0000 7f4482a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f447c109010 0x7f447c19ab90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:38.330 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.337+0000 7f4482a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f447c19ef10 0x7f447c19f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:38.330 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.337+0000 7f4482a00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f447c1141d0 con 0x7f447c109010 2024-04-02T14:50:38.330 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.337+0000 7f4482a00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f447c114050 con 0x7f447c103f40 2024-04-02T14:50:38.330 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.337+0000 7f4482a00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f447c114350 con 0x7f447c19ef10 2024-04-02T14:50:38.331 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.337+0000 7f447be00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f447c103f40 0x7f447c19a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:38.331 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.337+0000 7f447be00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f447c103f40 0x7f447c19a650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:47946/0 (socket says 172.21.15.90:47946) 2024-04-02T14:50:38.331 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.337+0000 7f447be00640 1 -- 172.21.15.90:0/744900266 learned_addr learned my addr 172.21.15.90:0/744900266 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:50:38.331 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.337+0000 7f447b400640 1 --2- 172.21.15.90:0/744900266 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f447c109010 0x7f447c19ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:38.331 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.337+0000 7f4480a00640 1 --2- 172.21.15.90:0/744900266 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f447c19ef10 0x7f447c19f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:38.331 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.338+0000 7f447be00640 1 -- 172.21.15.90:0/744900266 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f447c19ef10 msgr2=0x7f447c19f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:38.331 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.338+0000 7f447be00640 1 --2- 172.21.15.90:0/744900266 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f447c19ef10 0x7f447c19f3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:38.332 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.338+0000 7f447be00640 1 -- 172.21.15.90:0/744900266 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f447c109010 msgr2=0x7f447c19ab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:38.332 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.338+0000 7f447be00640 1 --2- 172.21.15.90:0/744900266 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f447c109010 0x7f447c19ab90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:38.332 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.338+0000 7f447be00640 1 -- 172.21.15.90:0/744900266 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f447c19fa00 con 0x7f447c103f40 2024-04-02T14:50:38.333 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.339+0000 7f447be00640 1 --2- 172.21.15.90:0/744900266 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f447c103f40 0x7f447c19a650 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f446c00b4a0 tx=0x7f446c00b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:50:38.333 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.340+0000 7f4478c00640 1 -- 172.21.15.90:0/744900266 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f446c014070 con 0x7f447c103f40 2024-04-02T14:50:38.333 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.340+0000 7f4478c00640 1 -- 172.21.15.90:0/744900266 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f446c010af0 con 0x7f447c103f40 2024-04-02T14:50:38.333 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.340+0000 7f4478c00640 1 -- 172.21.15.90:0/744900266 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f446c017d10 con 0x7f447c103f40 2024-04-02T14:50:38.334 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.340+0000 7f4482a00640 1 -- 172.21.15.90:0/744900266 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f447c19fcf0 con 0x7f447c103f40 2024-04-02T14:50:38.334 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.340+0000 7f4482a00640 1 -- 172.21.15.90:0/744900266 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f447c0761e0 con 0x7f447c103f40 2024-04-02T14:50:38.339 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.346+0000 7f44857c9740 1 Processor -- start 2024-04-02T14:50:38.340 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.346+0000 7f44857c9740 1 -- start start 2024-04-02T14:50:38.341 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.347+0000 7f44857c9740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564a8fe69eb0 0x564a8fe6a290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:38.341 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.347+0000 7f44857c9740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564a8fe6a7d0 0x564a8fe72e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:38.341 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.347+0000 7f44857c9740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564a8fe733b0 0x564a8fe757a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:38.341 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.347+0000 7f44857c9740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x564a8fdced70 con 0x564a8fe6a7d0 2024-04-02T14:50:38.341 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.347+0000 7f44857c9740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x564a8fd772a0 con 0x564a8fe69eb0 2024-04-02T14:50:38.341 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.347+0000 7f44857c9740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x564a8fdcad20 con 0x564a8fe733b0 2024-04-02T14:50:38.341 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.347+0000 7f4480a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564a8fe69eb0 0x564a8fe6a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:38.341 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.347+0000 7f4480a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564a8fe69eb0 0x564a8fe6a290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:47952/0 (socket says 172.21.15.90:47952) 2024-04-02T14:50:38.341 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.347+0000 7f4480a00640 1 -- 172.21.15.90:0/4222580855 learned_addr learned my addr 172.21.15.90:0/4222580855 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:50:38.341 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.347+0000 7f447b400640 1 --2- 172.21.15.90:0/4222580855 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564a8fe733b0 0x564a8fe757a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:38.342 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.348+0000 7f4480a00640 1 -- 172.21.15.90:0/4222580855 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564a8fe733b0 msgr2=0x564a8fe757a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:50:38.342 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.348+0000 7f4480a00640 1 --2- 172.21.15.90:0/4222580855 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564a8fe733b0 0x564a8fe757a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:38.342 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.348+0000 7f4480a00640 1 -- 172.21.15.90:0/4222580855 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564a8fe6a7d0 msgr2=0x564a8fe72e70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-02T14:50:38.342 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.348+0000 7f4480a00640 1 --2- 172.21.15.90:0/4222580855 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564a8fe6a7d0 0x564a8fe72e70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:38.342 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.348+0000 7f4480a00640 1 -- 172.21.15.90:0/4222580855 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564a8fd8b840 con 0x564a8fe69eb0 2024-04-02T14:50:38.342 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.348+0000 7f4480a00640 1 --2- 172.21.15.90:0/4222580855 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564a8fe69eb0 0x564a8fe6a290 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7f4474009f20 tx=0x7f447400c8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4dcb9d38ae50b23a server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:50:38.343 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.349+0000 7f4471200640 1 -- 172.21.15.90:0/4222580855 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f447401a070 con 0x564a8fe69eb0 2024-04-02T14:50:38.344 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.350+0000 7f4471200640 1 -- 172.21.15.90:0/4222580855 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f44740158f0 con 0x564a8fe69eb0 2024-04-02T14:50:38.344 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.350+0000 7f4471200640 1 -- 172.21.15.90:0/4222580855 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f447400aa30 con 0x564a8fe69eb0 2024-04-02T14:50:38.344 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.350+0000 7f4478c00640 1 -- 172.21.15.90:0/744900266 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f446c017070 con 0x7f447c103f40 2024-04-02T14:50:38.346 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.352+0000 7f44857c9740 1 -- 172.21.15.90:0/4222580855 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564a8fe69eb0 msgr2=0x564a8fe6a290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:38.346 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.352+0000 7f44857c9740 1 --2- 172.21.15.90:0/4222580855 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564a8fe69eb0 0x564a8fe6a290 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7f4474009f20 tx=0x7f447400c8b0 comp rx=0 tx=0).stop 2024-04-02T14:50:38.347 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.354+0000 7f44857c9740 1 -- 172.21.15.90:0/4222580855 shutdown_connections 2024-04-02T14:50:38.347 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.354+0000 7f44857c9740 1 --2- 172.21.15.90:0/4222580855 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564a8fe733b0 0x564a8fe757a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:38.347 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.354+0000 7f44857c9740 1 --2- 172.21.15.90:0/4222580855 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564a8fe6a7d0 0x564a8fe72e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:38.347 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.354+0000 7f44857c9740 1 --2- 172.21.15.90:0/4222580855 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564a8fe69eb0 0x564a8fe6a290 unknown :-1 s=CLOSED pgs=1034 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:38.348 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.354+0000 7f44857c9740 1 -- 172.21.15.90:0/4222580855 >> 172.21.15.90:0/4222580855 conn(0x564a8fe645e0 msgr2=0x564a8fe649c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:50:38.348 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.354+0000 7f44857c9740 1 -- 172.21.15.90:0/4222580855 shutdown_connections 2024-04-02T14:50:38.348 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.354+0000 7f44857c9740 1 -- 172.21.15.90:0/4222580855 wait complete. 2024-04-02T14:50:38.348 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.354+0000 7f4478c00640 1 --2- 172.21.15.90:0/744900266 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f44600a3d20 0x7f44600a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:38.348 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.354+0000 7f44857c9740 1 -- start start 2024-04-02T14:50:38.349 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.355+0000 7f4478c00640 1 -- 172.21.15.90:0/744900266 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 8550+0+0 (secure 0 0 0) 0x7f446c0cfe10 con 0x7f447c103f40 2024-04-02T14:50:38.349 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.355+0000 7f4480a00640 1 --2- 172.21.15.90:0/744900266 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f44600a3d20 0x7f44600a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:38.349 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.356+0000 7f44857c9740 1 Processor -- start 2024-04-02T14:50:38.350 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.357+0000 7f4480a00640 1 --2- 172.21.15.90:0/744900266 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f44600a3d20 0x7f44600a61e0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f4474018040 tx=0x7f4474000f40 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:50:38.353 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.357+0000 7f44857c9740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564a8fe69eb0 0x564a8fe704b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:38.353 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.357+0000 7f44857c9740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564a8fe6a7d0 0x564a8fe709f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:38.353 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.357+0000 7f44857c9740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564a8fe733b0 0x564a8fe70f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:38.353 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.357+0000 7f44857c9740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x564a8fe761a0 con 0x564a8fe733b0 2024-04-02T14:50:38.353 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.357+0000 7f44857c9740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x564a8fd8a660 con 0x564a8fe6a7d0 2024-04-02T14:50:38.354 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.357+0000 7f44857c9740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x564a8fe76320 con 0x564a8fe69eb0 2024-04-02T14:50:38.354 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.357+0000 7f44857c9740 10 client.0 _ll_register_callbacks cb 0x564a8fd8a050 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:50:38.354 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.358+0000 7f4480a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564a8fe733b0 0x564a8fe70f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:38.354 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.358+0000 7f4480a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564a8fe733b0 0x564a8fe70f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:42586/0 (socket says 172.21.15.90:42586) 2024-04-02T14:50:38.354 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.358+0000 7f4480a00640 1 -- 172.21.15.90:0/2941172166 learned_addr learned my addr 172.21.15.90:0/2941172166 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:50:38.354 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.359+0000 7f4480a00640 1 -- 172.21.15.90:0/2941172166 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564a8fe69eb0 msgr2=0x564a8fe704b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-02T14:50:38.354 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.359+0000 7f4480a00640 1 --2- 172.21.15.90:0/2941172166 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564a8fe69eb0 0x564a8fe704b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:38.354 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.359+0000 7f4480a00640 1 -- 172.21.15.90:0/2941172166 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564a8fe6a7d0 msgr2=0x564a8fe709f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-02T14:50:38.354 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.359+0000 7f4480a00640 1 --2- 172.21.15.90:0/2941172166 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564a8fe6a7d0 0x564a8fe709f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:38.354 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.359+0000 7f4480a00640 1 -- 172.21.15.90:0/2941172166 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564a8fd99e80 con 0x564a8fe733b0 2024-04-02T14:50:38.354 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.359+0000 7f4480a00640 1 --2- 172.21.15.90:0/2941172166 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564a8fe733b0 0x564a8fe70f30 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f4474017000 tx=0x7f447400a530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19b19087ca49bb52 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:50:38.354 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.359+0000 7f4465600640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-02T14:50:38.354 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.360+0000 7f4465600640 1 -- 172.21.15.90:0/2941172166 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4474040060 con 0x564a8fe733b0 2024-04-02T14:50:38.354 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.360+0000 7f4465600640 1 -- 172.21.15.90:0/2941172166 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 8550+0+0 (secure 0 0 0) 0x7f4474012040 con 0x564a8fe733b0 2024-04-02T14:50:38.354 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.360+0000 7f44857c9740 1 -- 172.21.15.90:0/2941172166 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f44740158f0 con 0x564a8fe733b0 2024-04-02T14:50:38.356 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.363+0000 7f4465600640 1 -- 172.21.15.90:0/2941172166 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f447401a070 con 0x564a8fe733b0 2024-04-02T14:50:38.356 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.363+0000 7f44857c9740 10 client.18960 fetch_fsmap learned FSMap version 27 2024-04-02T14:50:38.357 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.363+0000 7f44857c9740 1 -- 172.21.15.90:0/2941172166 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x564a8fae3210 con 0x564a8fe733b0 2024-04-02T14:50:38.357 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.364+0000 7f4465600640 1 -- 172.21.15.90:0/2941172166 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f44740158f0 con 0x564a8fe733b0 2024-04-02T14:50:38.357 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.364+0000 7f4465600640 1 -- 172.21.15.90:0/2941172166 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f447401f800 con 0x564a8fe733b0 2024-04-02T14:50:38.358 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.364+0000 7f44857c9740 10 client.18960 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:50:38.358 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.364+0000 7f44857c9740 10 client.18960 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-02T14:50:38.358 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.364+0000 7f44857c9740 20 client.18960 populate_metadata read hostname 'smithi090' 2024-04-02T14:50:38.358 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.364+0000 7f44857c9740 1 --2- 172.21.15.90:0/2941172166 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x564a8fe83760 0x564a8fe85b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:38.358 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.364+0000 7f44857c9740 4 client.18960 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:50:38.358 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.364+0000 7f44857c9740 1 -- 172.21.15.90:0/2941172166 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564a8fd7ecd0 con 0x564a8fe83760 2024-04-02T14:50:38.358 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.364+0000 7f447be00640 1 --2- 172.21.15.90:0/2941172166 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x564a8fe83760 0x564a8fe85b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:38.358 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.365+0000 7f447be00640 1 --2- 172.21.15.90:0/2941172166 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x564a8fe83760 0x564a8fe85b50 crc :-1 s=READY pgs=394 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4899a09b19990edf server_cookie=e490234a53bccdf9 in_seq=0 out_seq=0 2024-04-02T14:50:38.359 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.365+0000 7f4465600640 10 client.18960 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:50:38.363 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.370+0000 7f4465600640 1 -- 172.21.15.90:0/2941172166 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x564a8fd7ecd0 con 0x564a8fe83760 2024-04-02T14:50:38.363 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.370+0000 7f4465600640 10 client.18960 handle_command_reply: tid=0 2024-04-02T14:50:38.363 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.370+0000 7f44857c9740 1 client.18960 shutdown 2024-04-02T14:50:38.364 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.371+0000 7f44857c9740 1 -- 172.21.15.90:0/2941172166 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564a8fe733b0 msgr2=0x564a8fe70f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:38.364 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.371+0000 7f44857c9740 1 --2- 172.21.15.90:0/2941172166 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564a8fe733b0 0x564a8fe70f30 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f4474017000 tx=0x7f447400a530 comp rx=0 tx=0).stop 2024-04-02T14:50:38.365 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.371+0000 7f44857c9740 1 -- 172.21.15.90:0/2941172166 shutdown_connections 2024-04-02T14:50:38.365 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.371+0000 7f44857c9740 1 --2- 172.21.15.90:0/2941172166 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x564a8fe83760 0x564a8fe85b50 crc :-1 s=READY pgs=394 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:38.365 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.371+0000 7f44857c9740 1 --2- 172.21.15.90:0/2941172166 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564a8fe733b0 0x564a8fe70f30 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:38.365 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.371+0000 7f44857c9740 1 --2- 172.21.15.90:0/2941172166 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564a8fe6a7d0 0x564a8fe709f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:38.365 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.371+0000 7f4465600640 0 client.18960 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:50:38.365 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.371+0000 7f44857c9740 1 --2- 172.21.15.90:0/2941172166 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564a8fe69eb0 0x564a8fe704b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:38.365 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.371+0000 7f44857c9740 1 -- 172.21.15.90:0/2941172166 >> 172.21.15.90:0/2941172166 conn(0x564a8fe645e0 msgr2=0x564a8fe66330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:50:38.366 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.372+0000 7f44857c9740 1 -- 172.21.15.90:0/2941172166 shutdown_connections 2024-04-02T14:50:38.366 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.372+0000 7f44857c9740 1 -- 172.21.15.90:0/2941172166 wait complete. 2024-04-02T14:50:38.366 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.372+0000 7f44857c9740 20 client.18960 trim_cache size 0 max 16384 2024-04-02T14:50:38.398 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.405+0000 7f44857c9740 1 Processor -- start 2024-04-02T14:50:38.398 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.405+0000 7f44857c9740 1 -- start start 2024-04-02T14:50:38.399 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.405+0000 7f44857c9740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564a8fe69eb0 0x564a8fe7f060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:38.399 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.405+0000 7f44857c9740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564a8fe6a7d0 0x564a8fe70d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:38.399 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.405+0000 7f44857c9740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564a8fe733b0 0x564a8fe8b180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:38.399 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.405+0000 7f44857c9740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f447c06fa90 con 0x564a8fe733b0 2024-04-02T14:50:38.399 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.405+0000 7f44857c9740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f447c0715d0 con 0x564a8fe69eb0 2024-04-02T14:50:38.399 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.405+0000 7f44857c9740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f447c070160 con 0x564a8fe6a7d0 2024-04-02T14:50:38.400 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.406+0000 7f4480a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564a8fe733b0 0x564a8fe8b180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:38.400 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.406+0000 7f447b400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564a8fe6a7d0 0x564a8fe70d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:38.400 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.406+0000 7f447b400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564a8fe6a7d0 0x564a8fe70d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:49462/0 (socket says 172.21.15.90:49462) 2024-04-02T14:50:38.400 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.406+0000 7f447b400640 1 -- 172.21.15.90:0/1262423656 learned_addr learned my addr 172.21.15.90:0/1262423656 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:50:38.400 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.406+0000 7f447be00640 1 --2- 172.21.15.90:0/1262423656 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564a8fe69eb0 0x564a8fe7f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:38.401 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.407+0000 7f4480a00640 1 -- 172.21.15.90:0/1262423656 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564a8fe6a7d0 msgr2=0x564a8fe70d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:50:38.401 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.407+0000 7f4480a00640 1 --2- 172.21.15.90:0/1262423656 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564a8fe6a7d0 0x564a8fe70d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:38.401 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.407+0000 7f4480a00640 1 -- 172.21.15.90:0/1262423656 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564a8fe69eb0 msgr2=0x564a8fe7f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:50:38.401 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.407+0000 7f4480a00640 1 --2- 172.21.15.90:0/1262423656 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564a8fe69eb0 0x564a8fe7f060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:38.401 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.407+0000 7f4480a00640 1 -- 172.21.15.90:0/1262423656 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564a8fbaf620 con 0x564a8fe733b0 2024-04-02T14:50:38.401 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.408+0000 7f4480a00640 1 --2- 172.21.15.90:0/1262423656 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564a8fe733b0 0x564a8fe8b180 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f447401fc50 tx=0x7f447400a5a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2cdfab196d71700 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:50:38.402 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.408+0000 7f4467e00640 1 -- 172.21.15.90:0/1262423656 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4474040040 con 0x564a8fe733b0 2024-04-02T14:50:38.402 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.408+0000 7f4467e00640 1 -- 172.21.15.90:0/1262423656 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f447401fe40 con 0x564a8fe733b0 2024-04-02T14:50:38.402 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.408+0000 7f4467e00640 1 -- 172.21.15.90:0/1262423656 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4474040040 con 0x564a8fe733b0 2024-04-02T14:50:38.403 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.410+0000 7f44857c9740 1 -- 172.21.15.90:0/1262423656 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564a8fe733b0 msgr2=0x564a8fe8b180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:38.403 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.410+0000 7f44857c9740 1 --2- 172.21.15.90:0/1262423656 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564a8fe733b0 0x564a8fe8b180 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f447401fc50 tx=0x7f447400a5a0 comp rx=0 tx=0).stop 2024-04-02T14:50:38.403 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.410+0000 7f44857c9740 1 -- 172.21.15.90:0/1262423656 shutdown_connections 2024-04-02T14:50:38.404 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.410+0000 7f44857c9740 1 --2- 172.21.15.90:0/1262423656 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564a8fe733b0 0x564a8fe8b180 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:38.404 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.410+0000 7f44857c9740 1 --2- 172.21.15.90:0/1262423656 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564a8fe6a7d0 0x564a8fe70d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:38.404 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.410+0000 7f44857c9740 1 --2- 172.21.15.90:0/1262423656 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564a8fe69eb0 0x564a8fe7f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:38.404 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.410+0000 7f44857c9740 1 -- 172.21.15.90:0/1262423656 >> 172.21.15.90:0/1262423656 conn(0x564a8fe645e0 msgr2=0x564a8fe63640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:50:38.404 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.410+0000 7f44857c9740 1 -- 172.21.15.90:0/1262423656 shutdown_connections 2024-04-02T14:50:38.405 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.412+0000 7f44857c9740 1 -- 172.21.15.90:0/1262423656 wait complete. 2024-04-02T14:50:38.405 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.412+0000 7f44857c9740 1 -- start start 2024-04-02T14:50:38.406 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.412+0000 7f44857c9740 1 Processor -- start 2024-04-02T14:50:38.406 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.413+0000 7f44857c9740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564a8fe69eb0 0x564a8fe7b760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:38.408 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.414+0000 7f44857c9740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564a8fe6a7d0 0x564a8fe6cd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:38.408 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.414+0000 7f44857c9740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564a8fe733b0 0x564a8fe6d280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:38.408 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.414+0000 7f44857c9740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f447c070f00 con 0x564a8fe733b0 2024-04-02T14:50:38.408 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.414+0000 7f44857c9740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f447c070830 con 0x564a8fe69eb0 2024-04-02T14:50:38.408 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.414+0000 7f44857c9740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f447c06dab0 con 0x564a8fe6a7d0 2024-04-02T14:50:38.408 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.414+0000 7f44857c9740 10 client.0 _ll_register_callbacks cb 0x7f4474009df0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:50:38.408 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.414+0000 7f447be00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564a8fe69eb0 0x564a8fe7b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:38.408 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.414+0000 7f447be00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564a8fe69eb0 0x564a8fe7b760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:47972/0 (socket says 172.21.15.90:47972) 2024-04-02T14:50:38.408 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.415+0000 7f447be00640 1 -- 172.21.15.90:0/2157001221 learned_addr learned my addr 172.21.15.90:0/2157001221 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:50:38.410 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.416+0000 7f447b400640 1 --2- 172.21.15.90:0/2157001221 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564a8fe6a7d0 0x564a8fe6cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:38.410 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.416+0000 7f4480a00640 1 --2- 172.21.15.90:0/2157001221 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564a8fe733b0 0x564a8fe6d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:38.410 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.417+0000 7f447be00640 1 -- 172.21.15.90:0/2157001221 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564a8fe6a7d0 msgr2=0x564a8fe6cd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:50:38.410 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.417+0000 7f447be00640 1 --2- 172.21.15.90:0/2157001221 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564a8fe6a7d0 0x564a8fe6cd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:38.410 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.417+0000 7f447be00640 1 -- 172.21.15.90:0/2157001221 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564a8fe733b0 msgr2=0x564a8fe6d280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:50:38.410 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.417+0000 7f447be00640 1 --2- 172.21.15.90:0/2157001221 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564a8fe733b0 0x564a8fe6d280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:38.410 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.417+0000 7f447be00640 1 -- 172.21.15.90:0/2157001221 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564a8fe6d7c0 con 0x564a8fe69eb0 2024-04-02T14:50:38.411 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.417+0000 7f447be00640 1 --2- 172.21.15.90:0/2157001221 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564a8fe69eb0 0x564a8fe7b760 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7f446c00b470 tx=0x7f446c00bf90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a5eab1470e93db7 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:50:38.411 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.417+0000 7f4465600640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:50:38.411 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.418+0000 7f4465600640 1 -- 172.21.15.90:0/2157001221 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f446c0a0250 con 0x564a8fe69eb0 2024-04-02T14:50:38.411 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.418+0000 7f4465600640 1 -- 172.21.15.90:0/2157001221 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 8550+0+0 (secure 0 0 0) 0x7f446c09a1a0 con 0x564a8fe69eb0 2024-04-02T14:50:38.411 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.418+0000 7f44857c9740 1 -- 172.21.15.90:0/2157001221 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f447401fe40 con 0x564a8fe69eb0 2024-04-02T14:50:38.415 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.422+0000 7f4465600640 1 -- 172.21.15.90:0/2157001221 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f446c0a4b10 con 0x564a8fe69eb0 2024-04-02T14:50:38.415 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.422+0000 7f44857c9740 10 client.30293 fetch_fsmap learned FSMap version 27 2024-04-02T14:50:38.415 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.422+0000 7f44857c9740 1 -- 172.21.15.90:0/2157001221 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x564a8fee9720 con 0x564a8fe69eb0 2024-04-02T14:50:38.416 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.423+0000 7f4465600640 1 -- 172.21.15.90:0/2157001221 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f447401fe40 con 0x564a8fe69eb0 2024-04-02T14:50:38.416 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.423+0000 7f4465600640 1 -- 172.21.15.90:0/2157001221 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f446c0b5170 con 0x564a8fe69eb0 2024-04-02T14:50:38.418 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.423+0000 7f44857c9740 10 client.30293 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:50:38.418 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.423+0000 7f44857c9740 10 client.30293 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-02T14:50:38.418 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.423+0000 7f44857c9740 20 client.30293 populate_metadata read hostname 'smithi090' 2024-04-02T14:50:38.418 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.423+0000 7f44857c9740 1 --2- 172.21.15.90:0/2157001221 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x564a8fee9da0 0x564a8feec1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:38.418 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.423+0000 7f44857c9740 4 client.30293 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-02T14:50:38.418 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.423+0000 7f44857c9740 1 -- 172.21.15.90:0/2157001221 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "flush journal"}) -- 0x564a8fddc2b0 con 0x564a8fee9da0 2024-04-02T14:50:38.418 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.423+0000 7f447b400640 1 --2- 172.21.15.90:0/2157001221 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x564a8fee9da0 0x564a8feec1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:38.418 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.424+0000 7f447b400640 1 --2- 172.21.15.90:0/2157001221 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x564a8fee9da0 0x564a8feec1c0 crc :-1 s=READY pgs=395 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=46a4f725d628f066 server_cookie=d06fed9186ef86da in_seq=0 out_seq=0 2024-04-02T14:50:38.418 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:38.424+0000 7f4465600640 10 client.30293 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:50:38.602 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:50:38 smithi092 ceph-mon[36497]: pgmap v493: 97 pgs: 97 active+clean; 4.2 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 7.9 MiB/s wr, 841 op/s 2024-04-02T14:50:38.678 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:50:38 smithi177 ceph-mon[36431]: pgmap v493: 97 pgs: 97 active+clean; 4.2 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 7.9 MiB/s wr, 841 op/s 2024-04-02T14:50:38.679 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:50:38 smithi090 ceph-mon[31674]: pgmap v493: 97 pgs: 97 active+clean; 4.2 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 7.9 MiB/s wr, 841 op/s 2024-04-02T14:50:39.037 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:39.043+0000 7f4465600640 1 -- 172.21.15.90:0/2157001221 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x564a8fddc2b0 con 0x564a8fee9da0 2024-04-02T14:50:39.037 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:39.043+0000 7f4465600640 10 client.30293 handle_command_reply: tid=0 2024-04-02T14:50:39.037 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:39.043+0000 7f44857c9740 1 client.30293 shutdown 2024-04-02T14:50:39.037 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:39.043+0000 7f44857c9740 1 -- 172.21.15.90:0/2157001221 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564a8fe69eb0 msgr2=0x564a8fe7b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:39.037 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:39.043+0000 7f44857c9740 1 --2- 172.21.15.90:0/2157001221 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564a8fe69eb0 0x564a8fe7b760 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7f446c00b470 tx=0x7f446c00bf90 comp rx=0 tx=0).stop 2024-04-02T14:50:39.038 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:39.043+0000 7f44857c9740 1 -- 172.21.15.90:0/2157001221 shutdown_connections 2024-04-02T14:50:39.038 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:39.043+0000 7f44857c9740 1 --2- 172.21.15.90:0/2157001221 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x564a8fee9da0 0x564a8feec1c0 crc :-1 s=READY pgs=395 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:39.038 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:39.044+0000 7f44857c9740 1 --2- 172.21.15.90:0/2157001221 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564a8fe733b0 0x564a8fe6d280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:39.038 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:39.044+0000 7f44857c9740 1 --2- 172.21.15.90:0/2157001221 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564a8fe6a7d0 0x564a8fe6cd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:39.038 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:39.044+0000 7f44857c9740 1 --2- 172.21.15.90:0/2157001221 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564a8fe69eb0 0x564a8fe7b760 secure :-1 s=CLOSED pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7f446c00b470 tx=0x7f446c00bf90 comp rx=0 tx=0).stop 2024-04-02T14:50:39.038 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:39.044+0000 7f44857c9740 1 -- 172.21.15.90:0/2157001221 >> 172.21.15.90:0/2157001221 conn(0x564a8fe645e0 msgr2=0x564a8fe62db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:50:39.038 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:39.044+0000 7f4465600640 0 client.30293 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:50:39.038 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:39.044+0000 7f44857c9740 1 -- 172.21.15.90:0/2157001221 shutdown_connections 2024-04-02T14:50:39.039 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:39.044+0000 7f44857c9740 1 -- 172.21.15.90:0/2157001221 wait complete. 2024-04-02T14:50:39.039 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:39.044+0000 7f44857c9740 20 client.30293 trim_cache size 0 max 16384 2024-04-02T14:50:39.039 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:{ 2024-04-02T14:50:39.039 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "message": "", 2024-04-02T14:50:39.039 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout: "return_code": 0 2024-04-02T14:50:39.039 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stdout:} 2024-04-02T14:50:39.040 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:39.045+0000 7f4482a00640 1 -- 172.21.15.90:0/744900266 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f44600a3d20 msgr2=0x7f44600a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:39.040 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:39.046+0000 7f4482a00640 1 --2- 172.21.15.90:0/744900266 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f44600a3d20 0x7f44600a61e0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f4474018040 tx=0x7f4474000f40 comp rx=0 tx=0).stop 2024-04-02T14:50:39.040 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:39.046+0000 7f4482a00640 1 -- 172.21.15.90:0/744900266 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f447c103f40 msgr2=0x7f447c19a650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:39.040 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:39.046+0000 7f4482a00640 1 --2- 172.21.15.90:0/744900266 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f447c103f40 0x7f447c19a650 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f446c00b4a0 tx=0x7f446c00b960 comp rx=0 tx=0).stop 2024-04-02T14:50:39.040 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:39.046+0000 7f4482a00640 1 -- 172.21.15.90:0/744900266 shutdown_connections 2024-04-02T14:50:39.040 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:39.046+0000 7f4482a00640 1 --2- 172.21.15.90:0/744900266 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f44600a3d20 0x7f44600a61e0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:39.040 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:39.046+0000 7f4482a00640 1 --2- 172.21.15.90:0/744900266 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f447c19ef10 0x7f447c19f3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:39.040 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:39.046+0000 7f4482a00640 1 --2- 172.21.15.90:0/744900266 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f447c109010 0x7f447c19ab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:39.040 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:39.046+0000 7f4482a00640 1 --2- 172.21.15.90:0/744900266 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f447c103f40 0x7f447c19a650 unknown :-1 s=CLOSED pgs=1033 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:39.041 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:39.046+0000 7f4482a00640 1 -- 172.21.15.90:0/744900266 >> 172.21.15.90:0/744900266 conn(0x7f447c0ff7c0 msgr2=0x7f447c100330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:50:39.041 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:39.046+0000 7f4482a00640 1 -- 172.21.15.90:0/744900266 shutdown_connections 2024-04-02T14:50:39.041 INFO:teuthology.task.background_exec.ubuntu@smithi090.front.sepia.ceph.com.smithi090.stderr:2024-04-02T14:50:39.046+0000 7f4482a00640 1 -- 172.21.15.90:0/744900266 wait complete. 2024-04-02T14:50:40.602 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:50:40 smithi092 ceph-mon[36497]: pgmap v494: 97 pgs: 97 active+clean; 4.2 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 6.9 MiB/s wr, 725 op/s 2024-04-02T14:50:40.678 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:50:40 smithi177 ceph-mon[36431]: pgmap v494: 97 pgs: 97 active+clean; 4.2 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 6.9 MiB/s wr, 725 op/s 2024-04-02T14:50:40.679 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:50:40 smithi090 ceph-mon[31674]: pgmap v494: 97 pgs: 97 active+clean; 4.2 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 6.9 MiB/s wr, 725 op/s 2024-04-02T14:50:41.988 DEBUG:teuthology.orchestra.run.smithi090:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ls -lZaR /tmp/cephfs/postgres/' 2024-04-02T14:50:42.152 INFO:teuthology.orchestra.run.smithi090.stdout:/tmp/cephfs/postgres/: 2024-04-02T14:50:42.152 INFO:teuthology.orchestra.run.smithi090.stdout:total 0 2024-04-02T14:50:42.152 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x. 3 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Apr 2 14:35 . 2024-04-02T14:50:42.152 INFO:teuthology.orchestra.run.smithi090.stdout:drwxrwxrwt. 3 root root system_u:object_r:postgresql_db_t:s0 1 Apr 2 14:35 .. 2024-04-02T14:50:42.153 INFO:teuthology.orchestra.run.smithi090.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Apr 2 14:50 data 2024-04-02T14:50:42.153 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-02T14:50:42.153 INFO:teuthology.orchestra.run.smithi090.stdout:/tmp/cephfs/postgres/data: 2024-04-02T14:50:42.153 INFO:teuthology.orchestra.run.smithi090.stdout:total 36 2024-04-02T14:50:42.153 INFO:teuthology.orchestra.run.smithi090.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Apr 2 14:50 . 2024-04-02T14:50:42.153 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x. 3 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Apr 2 14:35 .. 2024-04-02T14:50:42.153 INFO:teuthology.orchestra.run.smithi090.stdout:drwx------. 6 postgres postgres system_u:object_r:postgresql_db_t:s0 4 Apr 2 14:35 base 2024-04-02T14:50:42.153 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 30 Apr 2 14:35 current_logfiles 2024-04-02T14:50:42.153 INFO:teuthology.orchestra.run.smithi090.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 58 Apr 2 14:35 global 2024-04-02T14:50:42.153 INFO:teuthology.orchestra.run.smithi090.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Apr 2 14:35 log 2024-04-02T14:50:42.153 INFO:teuthology.orchestra.run.smithi090.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 pg_commit_ts 2024-04-02T14:50:42.153 INFO:teuthology.orchestra.run.smithi090.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 pg_dynshmem 2024-04-02T14:50:42.153 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 4516 Apr 2 14:35 pg_hba.conf 2024-04-02T14:50:42.153 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 1636 Apr 2 14:35 pg_ident.conf 2024-04-02T14:50:42.153 INFO:teuthology.orchestra.run.smithi090.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Apr 2 14:50 pg_logical 2024-04-02T14:50:42.153 INFO:teuthology.orchestra.run.smithi090.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 2 Apr 2 14:35 pg_multixact 2024-04-02T14:50:42.154 INFO:teuthology.orchestra.run.smithi090.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 pg_notify 2024-04-02T14:50:42.154 INFO:teuthology.orchestra.run.smithi090.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 pg_replslot 2024-04-02T14:50:42.154 INFO:teuthology.orchestra.run.smithi090.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 pg_serial 2024-04-02T14:50:42.154 INFO:teuthology.orchestra.run.smithi090.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 pg_snapshots 2024-04-02T14:50:42.154 INFO:teuthology.orchestra.run.smithi090.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Apr 2 14:50 pg_stat 2024-04-02T14:50:42.154 INFO:teuthology.orchestra.run.smithi090.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:50 pg_stat_tmp 2024-04-02T14:50:42.154 INFO:teuthology.orchestra.run.smithi090.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Apr 2 14:50 pg_subtrans 2024-04-02T14:50:42.154 INFO:teuthology.orchestra.run.smithi090.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 pg_tblspc 2024-04-02T14:50:42.154 INFO:teuthology.orchestra.run.smithi090.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 pg_twophase 2024-04-02T14:50:42.154 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Apr 2 14:35 PG_VERSION 2024-04-02T14:50:42.154 INFO:teuthology.orchestra.run.smithi090.stdout:drwx------. 3 postgres postgres system_u:object_r:postgresql_db_t:s0 65 Apr 2 14:50 pg_wal 2024-04-02T14:50:42.154 INFO:teuthology.orchestra.run.smithi090.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 2 Apr 2 14:42 pg_xact 2024-04-02T14:50:42.154 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 88 Apr 2 14:35 postgresql.auto.conf 2024-04-02T14:50:42.154 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 28082 Apr 2 14:35 postgresql.conf 2024-04-02T14:50:42.155 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 51 Apr 2 14:35 postmaster.opts 2024-04-02T14:50:42.155 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-02T14:50:42.155 INFO:teuthology.orchestra.run.smithi090.stdout:/tmp/cephfs/postgres/data/base: 2024-04-02T14:50:42.155 INFO:teuthology.orchestra.run.smithi090.stdout:total 0 2024-04-02T14:50:42.155 INFO:teuthology.orchestra.run.smithi090.stdout:drwx------. 6 postgres postgres system_u:object_r:postgresql_db_t:s0 4 Apr 2 14:35 . 2024-04-02T14:50:42.155 INFO:teuthology.orchestra.run.smithi090.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Apr 2 14:50 .. 2024-04-02T14:50:42.155 INFO:teuthology.orchestra.run.smithi090.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 289 Apr 2 14:35 1 2024-04-02T14:50:42.155 INFO:teuthology.orchestra.run.smithi090.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 289 Apr 2 14:35 13493 2024-04-02T14:50:42.155 INFO:teuthology.orchestra.run.smithi090.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 305 Apr 2 14:39 13494 2024-04-02T14:50:42.155 INFO:teuthology.orchestra.run.smithi090.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 pgsql_tmp 2024-04-02T14:50:42.155 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-02T14:50:42.155 INFO:teuthology.orchestra.run.smithi090.stdout:/tmp/cephfs/postgres/data/base/1: 2024-04-02T14:50:42.155 INFO:teuthology.orchestra.run.smithi090.stdout:total 7777 2024-04-02T14:50:42.155 INFO:teuthology.orchestra.run.smithi090.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 289 Apr 2 14:35 . 2024-04-02T14:50:42.155 INFO:teuthology.orchestra.run.smithi090.stdout:drwx------. 6 postgres postgres system_u:object_r:postgresql_db_t:s0 4 Apr 2 14:35 .. 2024-04-02T14:50:42.156 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 112 2024-04-02T14:50:42.156 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 113 2024-04-02T14:50:42.156 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 81920 Apr 2 14:35 1247 2024-04-02T14:50:42.156 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 1247_fsm 2024-04-02T14:50:42.156 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 1247_vm 2024-04-02T14:50:42.156 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 434176 Apr 2 14:35 1249 2024-04-02T14:50:42.156 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 1249_fsm 2024-04-02T14:50:42.156 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 1249_vm 2024-04-02T14:50:42.156 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 663552 Apr 2 14:35 1255 2024-04-02T14:50:42.156 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 1255_fsm 2024-04-02T14:50:42.156 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 1255_vm 2024-04-02T14:50:42.156 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 106496 Apr 2 14:35 1259 2024-04-02T14:50:42.156 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 1259_fsm 2024-04-02T14:50:42.156 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 1259_vm 2024-04-02T14:50:42.156 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 65536 Apr 2 14:35 13344 2024-04-02T14:50:42.156 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 13344_fsm 2024-04-02T14:50:42.156 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 13344_vm 2024-04-02T14:50:42.157 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 13346 2024-04-02T14:50:42.157 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 13348 2024-04-02T14:50:42.157 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 13349 2024-04-02T14:50:42.157 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 13349_fsm 2024-04-02T14:50:42.157 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 13349_vm 2024-04-02T14:50:42.157 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 13351 2024-04-02T14:50:42.157 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 13353 2024-04-02T14:50:42.157 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 13354 2024-04-02T14:50:42.157 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 13354_fsm 2024-04-02T14:50:42.157 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 13354_vm 2024-04-02T14:50:42.157 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 13356 2024-04-02T14:50:42.157 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 13358 2024-04-02T14:50:42.157 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 13359 2024-04-02T14:50:42.157 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 13359_fsm 2024-04-02T14:50:42.157 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 13359_vm 2024-04-02T14:50:42.157 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 13361 2024-04-02T14:50:42.158 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 13363 2024-04-02T14:50:42.158 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 1417 2024-04-02T14:50:42.158 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 1418 2024-04-02T14:50:42.158 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 174 2024-04-02T14:50:42.158 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 175 2024-04-02T14:50:42.158 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2187 2024-04-02T14:50:42.158 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 2224 2024-04-02T14:50:42.158 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 2328 2024-04-02T14:50:42.158 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 2336 2024-04-02T14:50:42.158 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2337 2024-04-02T14:50:42.158 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2579 2024-04-02T14:50:42.158 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2600 2024-04-02T14:50:42.158 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 2600_fsm 2024-04-02T14:50:42.158 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2600_vm 2024-04-02T14:50:42.158 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2601 2024-04-02T14:50:42.159 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 2601_fsm 2024-04-02T14:50:42.159 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2601_vm 2024-04-02T14:50:42.159 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Apr 2 14:35 2602 2024-04-02T14:50:42.159 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 2602_fsm 2024-04-02T14:50:42.159 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2602_vm 2024-04-02T14:50:42.159 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 2 14:35 2603 2024-04-02T14:50:42.159 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 2603_fsm 2024-04-02T14:50:42.159 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2603_vm 2024-04-02T14:50:42.159 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 2604 2024-04-02T14:50:42.159 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2605 2024-04-02T14:50:42.159 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 2605_fsm 2024-04-02T14:50:42.159 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2605_vm 2024-04-02T14:50:42.159 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2606 2024-04-02T14:50:42.159 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 2606_fsm 2024-04-02T14:50:42.159 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2606_vm 2024-04-02T14:50:42.159 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2607 2024-04-02T14:50:42.159 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 2607_fsm 2024-04-02T14:50:42.160 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2607_vm 2024-04-02T14:50:42.160 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 466944 Apr 2 14:35 2608 2024-04-02T14:50:42.160 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 2608_fsm 2024-04-02T14:50:42.160 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2608_vm 2024-04-02T14:50:42.160 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 344064 Apr 2 14:35 2609 2024-04-02T14:50:42.160 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 2609_fsm 2024-04-02T14:50:42.160 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2609_vm 2024-04-02T14:50:42.160 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 2 14:35 2610 2024-04-02T14:50:42.160 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 2610_fsm 2024-04-02T14:50:42.160 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2610_vm 2024-04-02T14:50:42.160 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 2611 2024-04-02T14:50:42.160 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2612 2024-04-02T14:50:42.160 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 2612_fsm 2024-04-02T14:50:42.160 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2612_vm 2024-04-02T14:50:42.160 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 2613 2024-04-02T14:50:42.160 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2615 2024-04-02T14:50:42.161 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 2615_fsm 2024-04-02T14:50:42.161 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2615_vm 2024-04-02T14:50:42.161 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2616 2024-04-02T14:50:42.161 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 2616_fsm 2024-04-02T14:50:42.161 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2616_vm 2024-04-02T14:50:42.161 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 114688 Apr 2 14:35 2617 2024-04-02T14:50:42.161 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 2617_fsm 2024-04-02T14:50:42.161 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2617_vm 2024-04-02T14:50:42.161 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 98304 Apr 2 14:35 2618 2024-04-02T14:50:42.161 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 2618_fsm 2024-04-02T14:50:42.161 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2618_vm 2024-04-02T14:50:42.161 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 147456 Apr 2 14:35 2619 2024-04-02T14:50:42.161 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 2619_fsm 2024-04-02T14:50:42.161 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2619_vm 2024-04-02T14:50:42.161 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 2620 2024-04-02T14:50:42.161 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2650 2024-04-02T14:50:42.162 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2651 2024-04-02T14:50:42.162 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2652 2024-04-02T14:50:42.162 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 2 14:35 2653 2024-04-02T14:50:42.162 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 2 14:35 2654 2024-04-02T14:50:42.162 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 2 14:35 2655 2024-04-02T14:50:42.162 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2656 2024-04-02T14:50:42.162 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2657 2024-04-02T14:50:42.162 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 122880 Apr 2 14:35 2658 2024-04-02T14:50:42.162 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 81920 Apr 2 14:35 2659 2024-04-02T14:50:42.162 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2660 2024-04-02T14:50:42.162 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2661 2024-04-02T14:50:42.162 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 2 14:35 2662 2024-04-02T14:50:42.162 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Apr 2 14:35 2663 2024-04-02T14:50:42.162 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2664 2024-04-02T14:50:42.162 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2665 2024-04-02T14:50:42.163 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2666 2024-04-02T14:50:42.163 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2667 2024-04-02T14:50:42.163 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2668 2024-04-02T14:50:42.163 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2669 2024-04-02T14:50:42.163 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2670 2024-04-02T14:50:42.163 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 278528 Apr 2 14:35 2673 2024-04-02T14:50:42.163 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 352256 Apr 2 14:35 2674 2024-04-02T14:50:42.163 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 196608 Apr 2 14:35 2675 2024-04-02T14:50:42.163 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2678 2024-04-02T14:50:42.163 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2679 2024-04-02T14:50:42.163 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2680 2024-04-02T14:50:42.163 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2681 2024-04-02T14:50:42.163 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2682 2024-04-02T14:50:42.163 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2683 2024-04-02T14:50:42.163 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2684 2024-04-02T14:50:42.164 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2685 2024-04-02T14:50:42.164 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2686 2024-04-02T14:50:42.164 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2687 2024-04-02T14:50:42.164 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 2 14:35 2688 2024-04-02T14:50:42.164 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Apr 2 14:35 2689 2024-04-02T14:50:42.164 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 90112 Apr 2 14:35 2690 2024-04-02T14:50:42.164 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 253952 Apr 2 14:35 2691 2024-04-02T14:50:42.164 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2692 2024-04-02T14:50:42.164 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2693 2024-04-02T14:50:42.164 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2696 2024-04-02T14:50:42.164 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2699 2024-04-02T14:50:42.164 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2701 2024-04-02T14:50:42.164 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2702 2024-04-02T14:50:42.164 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 2 14:35 2703 2024-04-02T14:50:42.165 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 2 14:35 2704 2024-04-02T14:50:42.165 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2753 2024-04-02T14:50:42.165 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 2753_fsm 2024-04-02T14:50:42.165 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2753_vm 2024-04-02T14:50:42.165 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2754 2024-04-02T14:50:42.165 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2755 2024-04-02T14:50:42.165 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 2 14:35 2756 2024-04-02T14:50:42.165 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 2 14:35 2757 2024-04-02T14:50:42.165 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 2830 2024-04-02T14:50:42.165 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2831 2024-04-02T14:50:42.165 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 2832 2024-04-02T14:50:42.165 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2833 2024-04-02T14:50:42.165 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 2834 2024-04-02T14:50:42.165 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2835 2024-04-02T14:50:42.165 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 2836 2024-04-02T14:50:42.166 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2837 2024-04-02T14:50:42.166 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 491520 Apr 2 14:35 2838 2024-04-02T14:50:42.166 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 2838_fsm 2024-04-02T14:50:42.166 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2838_vm 2024-04-02T14:50:42.166 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2839 2024-04-02T14:50:42.166 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 2840 2024-04-02T14:50:42.166 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 2840_fsm 2024-04-02T14:50:42.166 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2840_vm 2024-04-02T14:50:42.166 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2841 2024-04-02T14:50:42.166 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 2995 2024-04-02T14:50:42.166 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2996 2024-04-02T14:50:42.166 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3079 2024-04-02T14:50:42.166 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 3079_fsm 2024-04-02T14:50:42.166 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3079_vm 2024-04-02T14:50:42.166 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 3080 2024-04-02T14:50:42.167 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 3081 2024-04-02T14:50:42.167 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 2 14:35 3085 2024-04-02T14:50:42.167 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 3118 2024-04-02T14:50:42.167 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3119 2024-04-02T14:50:42.167 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 57344 Apr 2 14:35 3164 2024-04-02T14:50:42.167 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 3256 2024-04-02T14:50:42.167 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3257 2024-04-02T14:50:42.167 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3258 2024-04-02T14:50:42.167 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 3350 2024-04-02T14:50:42.167 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3351 2024-04-02T14:50:42.167 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3379 2024-04-02T14:50:42.167 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3380 2024-04-02T14:50:42.167 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 3381 2024-04-02T14:50:42.167 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 3394 2024-04-02T14:50:42.167 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 3394_fsm 2024-04-02T14:50:42.168 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3394_vm 2024-04-02T14:50:42.168 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 3395 2024-04-02T14:50:42.168 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 3429 2024-04-02T14:50:42.168 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 3430 2024-04-02T14:50:42.168 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3431 2024-04-02T14:50:42.168 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3433 2024-04-02T14:50:42.168 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 3439 2024-04-02T14:50:42.168 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3440 2024-04-02T14:50:42.168 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 2 14:35 3455 2024-04-02T14:50:42.168 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 221184 Apr 2 14:35 3456 2024-04-02T14:50:42.168 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 3456_fsm 2024-04-02T14:50:42.168 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3456_vm 2024-04-02T14:50:42.168 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 3466 2024-04-02T14:50:42.168 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3467 2024-04-02T14:50:42.168 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3468 2024-04-02T14:50:42.169 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 3501 2024-04-02T14:50:42.169 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3502 2024-04-02T14:50:42.169 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3503 2024-04-02T14:50:42.169 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3534 2024-04-02T14:50:42.169 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3541 2024-04-02T14:50:42.169 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 3541_fsm 2024-04-02T14:50:42.169 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3541_vm 2024-04-02T14:50:42.169 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 3542 2024-04-02T14:50:42.169 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3574 2024-04-02T14:50:42.169 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3575 2024-04-02T14:50:42.169 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 3576 2024-04-02T14:50:42.169 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 3596 2024-04-02T14:50:42.169 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3597 2024-04-02T14:50:42.169 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 3598 2024-04-02T14:50:42.169 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3599 2024-04-02T14:50:42.170 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3600 2024-04-02T14:50:42.170 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 3600_fsm 2024-04-02T14:50:42.170 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3600_vm 2024-04-02T14:50:42.170 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3601 2024-04-02T14:50:42.170 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 3601_fsm 2024-04-02T14:50:42.170 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3601_vm 2024-04-02T14:50:42.170 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3602 2024-04-02T14:50:42.170 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 3602_fsm 2024-04-02T14:50:42.170 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3602_vm 2024-04-02T14:50:42.170 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 3603 2024-04-02T14:50:42.170 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 3603_fsm 2024-04-02T14:50:42.170 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3603_vm 2024-04-02T14:50:42.170 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 3604 2024-04-02T14:50:42.170 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 3605 2024-04-02T14:50:42.171 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 3606 2024-04-02T14:50:42.171 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 3607 2024-04-02T14:50:42.171 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 3608 2024-04-02T14:50:42.171 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 2 14:35 3609 2024-04-02T14:50:42.171 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 3712 2024-04-02T14:50:42.171 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3764 2024-04-02T14:50:42.171 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 3764_fsm 2024-04-02T14:50:42.171 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3764_vm 2024-04-02T14:50:42.171 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 3766 2024-04-02T14:50:42.171 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 3767 2024-04-02T14:50:42.171 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3997 2024-04-02T14:50:42.171 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 4143 2024-04-02T14:50:42.171 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 4144 2024-04-02T14:50:42.171 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 4145 2024-04-02T14:50:42.171 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 4146 2024-04-02T14:50:42.171 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 4147 2024-04-02T14:50:42.171 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 4148 2024-04-02T14:50:42.172 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 4149 2024-04-02T14:50:42.172 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 4150 2024-04-02T14:50:42.172 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 4151 2024-04-02T14:50:42.172 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 4152 2024-04-02T14:50:42.172 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 4153 2024-04-02T14:50:42.172 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 4154 2024-04-02T14:50:42.172 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 4155 2024-04-02T14:50:42.172 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 4156 2024-04-02T14:50:42.172 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 4157 2024-04-02T14:50:42.172 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 4158 2024-04-02T14:50:42.172 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 4159 2024-04-02T14:50:42.172 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 4160 2024-04-02T14:50:42.172 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 4161 2024-04-02T14:50:42.172 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 4162 2024-04-02T14:50:42.173 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 4163 2024-04-02T14:50:42.173 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 4164 2024-04-02T14:50:42.173 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 4165 2024-04-02T14:50:42.173 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 4166 2024-04-02T14:50:42.173 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 4167 2024-04-02T14:50:42.173 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 4168 2024-04-02T14:50:42.173 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 4169 2024-04-02T14:50:42.173 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 4170 2024-04-02T14:50:42.173 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 4171 2024-04-02T14:50:42.174 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 4172 2024-04-02T14:50:42.174 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 4173 2024-04-02T14:50:42.174 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 4174 2024-04-02T14:50:42.174 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 5002 2024-04-02T14:50:42.174 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 548 2024-04-02T14:50:42.174 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 549 2024-04-02T14:50:42.174 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 6102 2024-04-02T14:50:42.174 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 6104 2024-04-02T14:50:42.174 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 6106 2024-04-02T14:50:42.174 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 6110 2024-04-02T14:50:42.174 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 6111 2024-04-02T14:50:42.174 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 6112 2024-04-02T14:50:42.174 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 6113 2024-04-02T14:50:42.174 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 6117 2024-04-02T14:50:42.175 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 826 2024-04-02T14:50:42.175 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 827 2024-04-02T14:50:42.175 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 828 2024-04-02T14:50:42.175 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 512 Apr 2 14:35 pg_filenode.map 2024-04-02T14:50:42.175 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Apr 2 14:35 PG_VERSION 2024-04-02T14:50:42.175 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-02T14:50:42.175 INFO:teuthology.orchestra.run.smithi090.stdout:/tmp/cephfs/postgres/data/base/13493: 2024-04-02T14:50:42.175 INFO:teuthology.orchestra.run.smithi090.stdout:total 7777 2024-04-02T14:50:42.175 INFO:teuthology.orchestra.run.smithi090.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 289 Apr 2 14:35 . 2024-04-02T14:50:42.175 INFO:teuthology.orchestra.run.smithi090.stdout:drwx------. 6 postgres postgres system_u:object_r:postgresql_db_t:s0 4 Apr 2 14:35 .. 2024-04-02T14:50:42.175 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 112 2024-04-02T14:50:42.175 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 113 2024-04-02T14:50:42.175 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 81920 Apr 2 14:35 1247 2024-04-02T14:50:42.175 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 1247_fsm 2024-04-02T14:50:42.175 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 1247_vm 2024-04-02T14:50:42.175 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 434176 Apr 2 14:35 1249 2024-04-02T14:50:42.175 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 1249_fsm 2024-04-02T14:50:42.176 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 1249_vm 2024-04-02T14:50:42.176 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 663552 Apr 2 14:35 1255 2024-04-02T14:50:42.176 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 1255_fsm 2024-04-02T14:50:42.176 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 1255_vm 2024-04-02T14:50:42.176 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 106496 Apr 2 14:35 1259 2024-04-02T14:50:42.176 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 1259_fsm 2024-04-02T14:50:42.176 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 1259_vm 2024-04-02T14:50:42.176 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 65536 Apr 2 14:35 13344 2024-04-02T14:50:42.176 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 13344_fsm 2024-04-02T14:50:42.176 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 13344_vm 2024-04-02T14:50:42.176 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 13346 2024-04-02T14:50:42.176 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 13348 2024-04-02T14:50:42.176 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 13349 2024-04-02T14:50:42.176 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 13349_fsm 2024-04-02T14:50:42.176 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 13349_vm 2024-04-02T14:50:42.176 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 13351 2024-04-02T14:50:42.176 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 13353 2024-04-02T14:50:42.177 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 13354 2024-04-02T14:50:42.177 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 13354_fsm 2024-04-02T14:50:42.177 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 13354_vm 2024-04-02T14:50:42.177 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 13356 2024-04-02T14:50:42.177 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 13358 2024-04-02T14:50:42.177 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 13359 2024-04-02T14:50:42.177 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 13359_fsm 2024-04-02T14:50:42.177 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 13359_vm 2024-04-02T14:50:42.177 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 13361 2024-04-02T14:50:42.177 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 13363 2024-04-02T14:50:42.177 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 1417 2024-04-02T14:50:42.177 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 1418 2024-04-02T14:50:42.177 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 174 2024-04-02T14:50:42.177 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 175 2024-04-02T14:50:42.177 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2187 2024-04-02T14:50:42.177 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 2224 2024-04-02T14:50:42.178 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 2328 2024-04-02T14:50:42.178 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 2336 2024-04-02T14:50:42.178 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2337 2024-04-02T14:50:42.178 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2579 2024-04-02T14:50:42.178 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2600 2024-04-02T14:50:42.178 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 2600_fsm 2024-04-02T14:50:42.178 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2600_vm 2024-04-02T14:50:42.178 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2601 2024-04-02T14:50:42.178 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 2601_fsm 2024-04-02T14:50:42.178 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2601_vm 2024-04-02T14:50:42.178 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Apr 2 14:35 2602 2024-04-02T14:50:42.178 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 2602_fsm 2024-04-02T14:50:42.178 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2602_vm 2024-04-02T14:50:42.178 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 2 14:35 2603 2024-04-02T14:50:42.178 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 2603_fsm 2024-04-02T14:50:42.179 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2603_vm 2024-04-02T14:50:42.179 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 2604 2024-04-02T14:50:42.179 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2605 2024-04-02T14:50:42.179 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 2605_fsm 2024-04-02T14:50:42.179 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2605_vm 2024-04-02T14:50:42.179 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2606 2024-04-02T14:50:42.179 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 2606_fsm 2024-04-02T14:50:42.179 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2606_vm 2024-04-02T14:50:42.179 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2607 2024-04-02T14:50:42.179 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 2607_fsm 2024-04-02T14:50:42.179 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2607_vm 2024-04-02T14:50:42.179 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 466944 Apr 2 14:35 2608 2024-04-02T14:50:42.179 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 2608_fsm 2024-04-02T14:50:42.179 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2608_vm 2024-04-02T14:50:42.179 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 344064 Apr 2 14:35 2609 2024-04-02T14:50:42.180 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 2609_fsm 2024-04-02T14:50:42.180 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2609_vm 2024-04-02T14:50:42.180 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 2 14:35 2610 2024-04-02T14:50:42.180 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 2610_fsm 2024-04-02T14:50:42.180 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2610_vm 2024-04-02T14:50:42.180 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 2611 2024-04-02T14:50:42.180 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2612 2024-04-02T14:50:42.180 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 2612_fsm 2024-04-02T14:50:42.180 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2612_vm 2024-04-02T14:50:42.180 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 2613 2024-04-02T14:50:42.180 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2615 2024-04-02T14:50:42.180 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 2615_fsm 2024-04-02T14:50:42.180 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2615_vm 2024-04-02T14:50:42.180 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2616 2024-04-02T14:50:42.180 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 2616_fsm 2024-04-02T14:50:42.180 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2616_vm 2024-04-02T14:50:42.181 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 114688 Apr 2 14:35 2617 2024-04-02T14:50:42.181 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 2617_fsm 2024-04-02T14:50:42.181 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2617_vm 2024-04-02T14:50:42.181 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 98304 Apr 2 14:35 2618 2024-04-02T14:50:42.181 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 2618_fsm 2024-04-02T14:50:42.181 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2618_vm 2024-04-02T14:50:42.181 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 147456 Apr 2 14:35 2619 2024-04-02T14:50:42.181 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 2619_fsm 2024-04-02T14:50:42.181 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2619_vm 2024-04-02T14:50:42.181 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 2620 2024-04-02T14:50:42.181 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2650 2024-04-02T14:50:42.181 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2651 2024-04-02T14:50:42.182 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2652 2024-04-02T14:50:42.182 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 2 14:35 2653 2024-04-02T14:50:42.182 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 2 14:35 2654 2024-04-02T14:50:42.182 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 2 14:35 2655 2024-04-02T14:50:42.182 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2656 2024-04-02T14:50:42.182 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2657 2024-04-02T14:50:42.182 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 122880 Apr 2 14:35 2658 2024-04-02T14:50:42.182 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 81920 Apr 2 14:35 2659 2024-04-02T14:50:42.182 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2660 2024-04-02T14:50:42.182 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2661 2024-04-02T14:50:42.182 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 2 14:35 2662 2024-04-02T14:50:42.182 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Apr 2 14:35 2663 2024-04-02T14:50:42.182 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2664 2024-04-02T14:50:42.183 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2665 2024-04-02T14:50:42.183 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2666 2024-04-02T14:50:42.183 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2667 2024-04-02T14:50:42.183 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2668 2024-04-02T14:50:42.183 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2669 2024-04-02T14:50:42.183 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2670 2024-04-02T14:50:42.183 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 278528 Apr 2 14:35 2673 2024-04-02T14:50:42.183 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 352256 Apr 2 14:35 2674 2024-04-02T14:50:42.183 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 196608 Apr 2 14:35 2675 2024-04-02T14:50:42.183 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2678 2024-04-02T14:50:42.183 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2679 2024-04-02T14:50:42.183 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2680 2024-04-02T14:50:42.183 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2681 2024-04-02T14:50:42.183 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2682 2024-04-02T14:50:42.184 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2683 2024-04-02T14:50:42.184 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2684 2024-04-02T14:50:42.184 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2685 2024-04-02T14:50:42.184 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2686 2024-04-02T14:50:42.184 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2687 2024-04-02T14:50:42.184 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 2 14:35 2688 2024-04-02T14:50:42.184 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Apr 2 14:35 2689 2024-04-02T14:50:42.184 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 90112 Apr 2 14:35 2690 2024-04-02T14:50:42.184 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 253952 Apr 2 14:35 2691 2024-04-02T14:50:42.184 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2692 2024-04-02T14:50:42.184 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2693 2024-04-02T14:50:42.184 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2696 2024-04-02T14:50:42.184 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2699 2024-04-02T14:50:42.184 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2701 2024-04-02T14:50:42.184 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2702 2024-04-02T14:50:42.184 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 2 14:35 2703 2024-04-02T14:50:42.185 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 2 14:35 2704 2024-04-02T14:50:42.185 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2753 2024-04-02T14:50:42.185 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 2753_fsm 2024-04-02T14:50:42.185 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2753_vm 2024-04-02T14:50:42.185 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2754 2024-04-02T14:50:42.185 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2755 2024-04-02T14:50:42.185 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 2 14:35 2756 2024-04-02T14:50:42.185 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 2 14:35 2757 2024-04-02T14:50:42.185 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 2830 2024-04-02T14:50:42.185 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2831 2024-04-02T14:50:42.185 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 2832 2024-04-02T14:50:42.185 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2833 2024-04-02T14:50:42.185 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 2834 2024-04-02T14:50:42.185 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2835 2024-04-02T14:50:42.186 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 2836 2024-04-02T14:50:42.186 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2837 2024-04-02T14:50:42.186 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 491520 Apr 2 14:35 2838 2024-04-02T14:50:42.186 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 2838_fsm 2024-04-02T14:50:42.186 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2838_vm 2024-04-02T14:50:42.186 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2839 2024-04-02T14:50:42.186 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 2840 2024-04-02T14:50:42.186 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 2840_fsm 2024-04-02T14:50:42.186 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2840_vm 2024-04-02T14:50:42.186 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2841 2024-04-02T14:50:42.186 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 2995 2024-04-02T14:50:42.186 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2996 2024-04-02T14:50:42.186 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3079 2024-04-02T14:50:42.186 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 3079_fsm 2024-04-02T14:50:42.187 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3079_vm 2024-04-02T14:50:42.187 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 3080 2024-04-02T14:50:42.187 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 3081 2024-04-02T14:50:42.187 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 2 14:35 3085 2024-04-02T14:50:42.187 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 3118 2024-04-02T14:50:42.187 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3119 2024-04-02T14:50:42.187 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 57344 Apr 2 14:35 3164 2024-04-02T14:50:42.187 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 3256 2024-04-02T14:50:42.187 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3257 2024-04-02T14:50:42.187 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3258 2024-04-02T14:50:42.187 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 3350 2024-04-02T14:50:42.187 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3351 2024-04-02T14:50:42.187 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3379 2024-04-02T14:50:42.187 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3380 2024-04-02T14:50:42.187 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 3381 2024-04-02T14:50:42.187 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 3394 2024-04-02T14:50:42.188 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 3394_fsm 2024-04-02T14:50:42.188 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3394_vm 2024-04-02T14:50:42.188 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 3395 2024-04-02T14:50:42.188 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 3429 2024-04-02T14:50:42.188 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 3430 2024-04-02T14:50:42.188 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3431 2024-04-02T14:50:42.188 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3433 2024-04-02T14:50:42.188 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 3439 2024-04-02T14:50:42.188 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3440 2024-04-02T14:50:42.188 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 2 14:35 3455 2024-04-02T14:50:42.188 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 221184 Apr 2 14:35 3456 2024-04-02T14:50:42.188 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 3456_fsm 2024-04-02T14:50:42.188 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3456_vm 2024-04-02T14:50:42.188 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 3466 2024-04-02T14:50:42.188 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3467 2024-04-02T14:50:42.189 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3468 2024-04-02T14:50:42.189 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 3501 2024-04-02T14:50:42.189 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3502 2024-04-02T14:50:42.189 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3503 2024-04-02T14:50:42.189 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3534 2024-04-02T14:50:42.189 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3541 2024-04-02T14:50:42.189 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 3541_fsm 2024-04-02T14:50:42.189 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3541_vm 2024-04-02T14:50:42.189 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 3542 2024-04-02T14:50:42.189 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3574 2024-04-02T14:50:42.189 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3575 2024-04-02T14:50:42.189 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 3576 2024-04-02T14:50:42.189 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 3596 2024-04-02T14:50:42.189 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3597 2024-04-02T14:50:42.189 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 3598 2024-04-02T14:50:42.190 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3599 2024-04-02T14:50:42.190 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3600 2024-04-02T14:50:42.190 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 3600_fsm 2024-04-02T14:50:42.190 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3600_vm 2024-04-02T14:50:42.190 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3601 2024-04-02T14:50:42.190 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 3601_fsm 2024-04-02T14:50:42.190 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3601_vm 2024-04-02T14:50:42.190 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3602 2024-04-02T14:50:42.190 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 3602_fsm 2024-04-02T14:50:42.190 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3602_vm 2024-04-02T14:50:42.190 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 3603 2024-04-02T14:50:42.190 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 3603_fsm 2024-04-02T14:50:42.190 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3603_vm 2024-04-02T14:50:42.190 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 3604 2024-04-02T14:50:42.190 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 3605 2024-04-02T14:50:42.191 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 3606 2024-04-02T14:50:42.191 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 3607 2024-04-02T14:50:42.191 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 3608 2024-04-02T14:50:42.191 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 2 14:35 3609 2024-04-02T14:50:42.191 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 3712 2024-04-02T14:50:42.191 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3764 2024-04-02T14:50:42.191 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 3764_fsm 2024-04-02T14:50:42.191 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3764_vm 2024-04-02T14:50:42.191 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 3766 2024-04-02T14:50:42.191 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 3767 2024-04-02T14:50:42.191 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3997 2024-04-02T14:50:42.191 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 4143 2024-04-02T14:50:42.191 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 4144 2024-04-02T14:50:42.191 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 4145 2024-04-02T14:50:42.192 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 4146 2024-04-02T14:50:42.192 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 4147 2024-04-02T14:50:42.192 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 4148 2024-04-02T14:50:42.192 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 4149 2024-04-02T14:50:42.192 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 4150 2024-04-02T14:50:42.192 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 4151 2024-04-02T14:50:42.192 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 4152 2024-04-02T14:50:42.192 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 4153 2024-04-02T14:50:42.192 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 4154 2024-04-02T14:50:42.192 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 4155 2024-04-02T14:50:42.192 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 4156 2024-04-02T14:50:42.192 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 4157 2024-04-02T14:50:42.192 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 4158 2024-04-02T14:50:42.192 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 4159 2024-04-02T14:50:42.192 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 4160 2024-04-02T14:50:42.192 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 4161 2024-04-02T14:50:42.193 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 4162 2024-04-02T14:50:42.193 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 4163 2024-04-02T14:50:42.193 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 4164 2024-04-02T14:50:42.193 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 4165 2024-04-02T14:50:42.193 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 4166 2024-04-02T14:50:42.193 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 4167 2024-04-02T14:50:42.193 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 4168 2024-04-02T14:50:42.193 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 4169 2024-04-02T14:50:42.193 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 4170 2024-04-02T14:50:42.193 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 4171 2024-04-02T14:50:42.193 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 4172 2024-04-02T14:50:42.204 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 4173 2024-04-02T14:50:42.205 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 4174 2024-04-02T14:50:42.205 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 5002 2024-04-02T14:50:42.205 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 548 2024-04-02T14:50:42.205 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 549 2024-04-02T14:50:42.205 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 6102 2024-04-02T14:50:42.205 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 6104 2024-04-02T14:50:42.205 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 6106 2024-04-02T14:50:42.205 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 6110 2024-04-02T14:50:42.205 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 6111 2024-04-02T14:50:42.205 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 6112 2024-04-02T14:50:42.205 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 6113 2024-04-02T14:50:42.205 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 6117 2024-04-02T14:50:42.205 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 826 2024-04-02T14:50:42.205 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 827 2024-04-02T14:50:42.205 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 828 2024-04-02T14:50:42.205 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 512 Apr 2 14:35 pg_filenode.map 2024-04-02T14:50:42.206 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Apr 2 14:35 PG_VERSION 2024-04-02T14:50:42.206 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-02T14:50:42.206 INFO:teuthology.orchestra.run.smithi090.stdout:/tmp/cephfs/postgres/data/base/13494: 2024-04-02T14:50:42.206 INFO:teuthology.orchestra.run.smithi090.stdout:total 604806 2024-04-02T14:50:42.206 INFO:teuthology.orchestra.run.smithi090.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 305 Apr 2 14:39 . 2024-04-02T14:50:42.206 INFO:teuthology.orchestra.run.smithi090.stdout:drwx------. 6 postgres postgres system_u:object_r:postgresql_db_t:s0 4 Apr 2 14:35 .. 2024-04-02T14:50:42.206 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 112 2024-04-02T14:50:42.206 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 113 2024-04-02T14:50:42.206 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 81920 Apr 2 14:35 1247 2024-04-02T14:50:42.206 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 1247_fsm 2024-04-02T14:50:42.206 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 1247_vm 2024-04-02T14:50:42.206 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 434176 Apr 2 14:35 1249 2024-04-02T14:50:42.206 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 1249_fsm 2024-04-02T14:50:42.206 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 1249_vm 2024-04-02T14:50:42.206 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 663552 Apr 2 14:35 1255 2024-04-02T14:50:42.207 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 1255_fsm 2024-04-02T14:50:42.207 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 1255_vm 2024-04-02T14:50:42.207 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 106496 Apr 2 14:50 1259 2024-04-02T14:50:42.207 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 1259_fsm 2024-04-02T14:50:42.207 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 1259_vm 2024-04-02T14:50:42.207 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 65536 Apr 2 14:35 13344 2024-04-02T14:50:42.207 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 13344_fsm 2024-04-02T14:50:42.207 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 13344_vm 2024-04-02T14:50:42.207 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 13346 2024-04-02T14:50:42.207 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 13348 2024-04-02T14:50:42.207 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 13349 2024-04-02T14:50:42.207 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 13349_fsm 2024-04-02T14:50:42.207 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 13349_vm 2024-04-02T14:50:42.207 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 13351 2024-04-02T14:50:42.207 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 13353 2024-04-02T14:50:42.207 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 13354 2024-04-02T14:50:42.208 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 13354_fsm 2024-04-02T14:50:42.208 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 13354_vm 2024-04-02T14:50:42.208 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 13356 2024-04-02T14:50:42.208 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 13358 2024-04-02T14:50:42.208 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 13359 2024-04-02T14:50:42.208 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 13359_fsm 2024-04-02T14:50:42.208 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 13359_vm 2024-04-02T14:50:42.208 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 13361 2024-04-02T14:50:42.208 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 13363 2024-04-02T14:50:42.208 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 1417 2024-04-02T14:50:42.208 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 1418 2024-04-02T14:50:42.208 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 443621376 Apr 2 14:50 16396 2024-04-02T14:50:42.208 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 131072 Apr 2 14:50 16396_fsm 2024-04-02T14:50:42.208 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:38 16396_vm 2024-04-02T14:50:42.208 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 1310720 Apr 2 14:50 16397 2024-04-02T14:50:42.208 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:50 16397_fsm 2024-04-02T14:50:42.208 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:50 16397_vm 2024-04-02T14:50:42.209 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 1867776 Apr 2 14:50 16399 2024-04-02T14:50:42.209 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:50 16399_fsm 2024-04-02T14:50:42.209 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:50 16399_vm 2024-04-02T14:50:42.209 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:50 16400 2024-04-02T14:50:42.209 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 57344 Apr 2 14:50 16402 2024-04-02T14:50:42.209 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 71901184 Apr 2 14:50 16404 2024-04-02T14:50:42.209 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 92127232 Apr 2 14:50 16406 2024-04-02T14:50:42.209 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 2 14:50 16406_fsm 2024-04-02T14:50:42.209 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:49 16406_vm 2024-04-02T14:50:42.209 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 174 2024-04-02T14:50:42.209 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 175 2024-04-02T14:50:42.209 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2187 2024-04-02T14:50:42.209 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 2224 2024-04-02T14:50:42.209 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 2328 2024-04-02T14:50:42.209 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 2336 2024-04-02T14:50:42.209 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2337 2024-04-02T14:50:42.210 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2579 2024-04-02T14:50:42.210 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2600 2024-04-02T14:50:42.210 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 2600_fsm 2024-04-02T14:50:42.210 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2600_vm 2024-04-02T14:50:42.210 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2601 2024-04-02T14:50:42.210 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 2601_fsm 2024-04-02T14:50:42.210 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2601_vm 2024-04-02T14:50:42.210 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Apr 2 14:35 2602 2024-04-02T14:50:42.210 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 2602_fsm 2024-04-02T14:50:42.210 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2602_vm 2024-04-02T14:50:42.210 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 2 14:35 2603 2024-04-02T14:50:42.210 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 2603_fsm 2024-04-02T14:50:42.210 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2603_vm 2024-04-02T14:50:42.210 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 2604 2024-04-02T14:50:42.211 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2605 2024-04-02T14:50:42.211 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 2605_fsm 2024-04-02T14:50:42.211 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2605_vm 2024-04-02T14:50:42.211 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2606 2024-04-02T14:50:42.211 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 2606_fsm 2024-04-02T14:50:42.211 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2606_vm 2024-04-02T14:50:42.211 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2607 2024-04-02T14:50:42.211 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 2607_fsm 2024-04-02T14:50:42.211 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2607_vm 2024-04-02T14:50:42.211 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 466944 Apr 2 14:35 2608 2024-04-02T14:50:42.211 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 2608_fsm 2024-04-02T14:50:42.211 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2608_vm 2024-04-02T14:50:42.211 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 344064 Apr 2 14:35 2609 2024-04-02T14:50:42.211 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 2609_fsm 2024-04-02T14:50:42.212 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2609_vm 2024-04-02T14:50:42.212 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 2 14:35 2610 2024-04-02T14:50:42.212 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 2610_fsm 2024-04-02T14:50:42.212 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2610_vm 2024-04-02T14:50:42.212 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 2611 2024-04-02T14:50:42.212 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2612 2024-04-02T14:50:42.212 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 2612_fsm 2024-04-02T14:50:42.212 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2612_vm 2024-04-02T14:50:42.212 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 2613 2024-04-02T14:50:42.212 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2615 2024-04-02T14:50:42.212 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 2615_fsm 2024-04-02T14:50:42.212 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2615_vm 2024-04-02T14:50:42.212 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2616 2024-04-02T14:50:42.212 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 2616_fsm 2024-04-02T14:50:42.212 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2616_vm 2024-04-02T14:50:42.212 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 114688 Apr 2 14:35 2617 2024-04-02T14:50:42.213 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 2617_fsm 2024-04-02T14:50:42.213 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2617_vm 2024-04-02T14:50:42.213 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 98304 Apr 2 14:35 2618 2024-04-02T14:50:42.213 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 2618_fsm 2024-04-02T14:50:42.213 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2618_vm 2024-04-02T14:50:42.213 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 172032 Apr 2 14:50 2619 2024-04-02T14:50:42.213 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:40 2619_fsm 2024-04-02T14:50:42.213 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2619_vm 2024-04-02T14:50:42.213 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 2620 2024-04-02T14:50:42.213 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2650 2024-04-02T14:50:42.213 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2651 2024-04-02T14:50:42.213 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2652 2024-04-02T14:50:42.213 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 2 14:35 2653 2024-04-02T14:50:42.213 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 2 14:35 2654 2024-04-02T14:50:42.213 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 2 14:35 2655 2024-04-02T14:50:42.213 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2656 2024-04-02T14:50:42.213 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2657 2024-04-02T14:50:42.214 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 122880 Apr 2 14:35 2658 2024-04-02T14:50:42.214 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 81920 Apr 2 14:35 2659 2024-04-02T14:50:42.214 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2660 2024-04-02T14:50:42.214 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2661 2024-04-02T14:50:42.214 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 2 14:35 2662 2024-04-02T14:50:42.214 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Apr 2 14:35 2663 2024-04-02T14:50:42.214 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2664 2024-04-02T14:50:42.214 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2665 2024-04-02T14:50:42.214 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2666 2024-04-02T14:50:42.214 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2667 2024-04-02T14:50:42.214 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2668 2024-04-02T14:50:42.214 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2669 2024-04-02T14:50:42.214 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2670 2024-04-02T14:50:42.214 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 278528 Apr 2 14:35 2673 2024-04-02T14:50:42.214 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 352256 Apr 2 14:35 2674 2024-04-02T14:50:42.214 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 196608 Apr 2 14:35 2675 2024-04-02T14:50:42.215 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2678 2024-04-02T14:50:42.215 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2679 2024-04-02T14:50:42.215 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2680 2024-04-02T14:50:42.215 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2681 2024-04-02T14:50:42.215 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2682 2024-04-02T14:50:42.215 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2683 2024-04-02T14:50:42.215 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2684 2024-04-02T14:50:42.215 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2685 2024-04-02T14:50:42.215 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2686 2024-04-02T14:50:42.215 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2687 2024-04-02T14:50:42.215 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 2 14:35 2688 2024-04-02T14:50:42.215 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Apr 2 14:35 2689 2024-04-02T14:50:42.215 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 90112 Apr 2 14:35 2690 2024-04-02T14:50:42.215 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 253952 Apr 2 14:35 2691 2024-04-02T14:50:42.216 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2692 2024-04-02T14:50:42.216 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2693 2024-04-02T14:50:42.216 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 2 14:41 2696 2024-04-02T14:50:42.216 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2699 2024-04-02T14:50:42.216 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2701 2024-04-02T14:50:42.216 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2702 2024-04-02T14:50:42.216 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 2 14:35 2703 2024-04-02T14:50:42.216 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 2 14:35 2704 2024-04-02T14:50:42.216 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2753 2024-04-02T14:50:42.216 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 2753_fsm 2024-04-02T14:50:42.216 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2753_vm 2024-04-02T14:50:42.216 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2754 2024-04-02T14:50:42.216 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2755 2024-04-02T14:50:42.216 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 2 14:35 2756 2024-04-02T14:50:42.216 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 2 14:35 2757 2024-04-02T14:50:42.217 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 2830 2024-04-02T14:50:42.217 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2831 2024-04-02T14:50:42.217 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 2832 2024-04-02T14:50:42.217 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2833 2024-04-02T14:50:42.217 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 2834 2024-04-02T14:50:42.217 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2835 2024-04-02T14:50:42.217 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 2836 2024-04-02T14:50:42.217 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2837 2024-04-02T14:50:42.217 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 491520 Apr 2 14:35 2838 2024-04-02T14:50:42.217 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 2838_fsm 2024-04-02T14:50:42.217 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2838_vm 2024-04-02T14:50:42.217 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2839 2024-04-02T14:50:42.217 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 2840 2024-04-02T14:50:42.217 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 2840_fsm 2024-04-02T14:50:42.217 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2840_vm 2024-04-02T14:50:42.217 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2841 2024-04-02T14:50:42.218 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 2995 2024-04-02T14:50:42.218 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2996 2024-04-02T14:50:42.218 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3079 2024-04-02T14:50:42.218 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 3079_fsm 2024-04-02T14:50:42.218 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3079_vm 2024-04-02T14:50:42.218 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 3080 2024-04-02T14:50:42.218 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 3081 2024-04-02T14:50:42.218 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 2 14:35 3085 2024-04-02T14:50:42.218 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 3118 2024-04-02T14:50:42.218 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3119 2024-04-02T14:50:42.218 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 57344 Apr 2 14:35 3164 2024-04-02T14:50:42.218 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 3256 2024-04-02T14:50:42.218 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3257 2024-04-02T14:50:42.218 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3258 2024-04-02T14:50:42.218 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 3350 2024-04-02T14:50:42.218 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3351 2024-04-02T14:50:42.219 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3379 2024-04-02T14:50:42.219 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3380 2024-04-02T14:50:42.219 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 3381 2024-04-02T14:50:42.219 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 3394 2024-04-02T14:50:42.219 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 3394_fsm 2024-04-02T14:50:42.219 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3394_vm 2024-04-02T14:50:42.219 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 3395 2024-04-02T14:50:42.219 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 3429 2024-04-02T14:50:42.219 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 3430 2024-04-02T14:50:42.219 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3431 2024-04-02T14:50:42.219 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3433 2024-04-02T14:50:42.219 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 3439 2024-04-02T14:50:42.219 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3440 2024-04-02T14:50:42.219 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 2 14:35 3455 2024-04-02T14:50:42.219 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 221184 Apr 2 14:35 3456 2024-04-02T14:50:42.219 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 3456_fsm 2024-04-02T14:50:42.220 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3456_vm 2024-04-02T14:50:42.220 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 3466 2024-04-02T14:50:42.220 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3467 2024-04-02T14:50:42.220 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3468 2024-04-02T14:50:42.220 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 3501 2024-04-02T14:50:42.220 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3502 2024-04-02T14:50:42.220 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3503 2024-04-02T14:50:42.220 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3534 2024-04-02T14:50:42.220 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3541 2024-04-02T14:50:42.220 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 3541_fsm 2024-04-02T14:50:42.220 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3541_vm 2024-04-02T14:50:42.220 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 3542 2024-04-02T14:50:42.220 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3574 2024-04-02T14:50:42.220 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3575 2024-04-02T14:50:42.220 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 3576 2024-04-02T14:50:42.220 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 3596 2024-04-02T14:50:42.221 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3597 2024-04-02T14:50:42.221 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 3598 2024-04-02T14:50:42.221 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3599 2024-04-02T14:50:42.221 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3600 2024-04-02T14:50:42.221 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 3600_fsm 2024-04-02T14:50:42.221 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3600_vm 2024-04-02T14:50:42.221 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3601 2024-04-02T14:50:42.221 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 3601_fsm 2024-04-02T14:50:42.221 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3601_vm 2024-04-02T14:50:42.221 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3602 2024-04-02T14:50:42.221 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 3602_fsm 2024-04-02T14:50:42.221 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3602_vm 2024-04-02T14:50:42.221 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 3603 2024-04-02T14:50:42.221 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 3603_fsm 2024-04-02T14:50:42.221 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3603_vm 2024-04-02T14:50:42.221 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 3604 2024-04-02T14:50:42.222 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 3605 2024-04-02T14:50:42.222 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 3606 2024-04-02T14:50:42.222 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 3607 2024-04-02T14:50:42.222 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 3608 2024-04-02T14:50:42.222 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 2 14:35 3609 2024-04-02T14:50:42.222 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 3712 2024-04-02T14:50:42.222 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3764 2024-04-02T14:50:42.222 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 3764_fsm 2024-04-02T14:50:42.222 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3764_vm 2024-04-02T14:50:42.222 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 3766 2024-04-02T14:50:42.222 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 3767 2024-04-02T14:50:42.222 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3997 2024-04-02T14:50:42.222 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 4143 2024-04-02T14:50:42.222 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 4144 2024-04-02T14:50:42.222 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 4145 2024-04-02T14:50:42.222 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 4146 2024-04-02T14:50:42.222 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 4147 2024-04-02T14:50:42.223 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 4148 2024-04-02T14:50:42.223 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 4149 2024-04-02T14:50:42.223 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 4150 2024-04-02T14:50:42.223 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 4151 2024-04-02T14:50:42.223 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 4152 2024-04-02T14:50:42.223 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 4153 2024-04-02T14:50:42.223 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 4154 2024-04-02T14:50:42.223 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 4155 2024-04-02T14:50:42.223 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 4156 2024-04-02T14:50:42.223 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 4157 2024-04-02T14:50:42.223 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 4158 2024-04-02T14:50:42.223 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 4159 2024-04-02T14:50:42.224 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 4160 2024-04-02T14:50:42.224 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 4161 2024-04-02T14:50:42.224 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 4162 2024-04-02T14:50:42.224 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 4163 2024-04-02T14:50:42.224 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 4164 2024-04-02T14:50:42.224 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 4165 2024-04-02T14:50:42.224 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 4166 2024-04-02T14:50:42.224 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 4167 2024-04-02T14:50:42.224 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 4168 2024-04-02T14:50:42.224 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 4169 2024-04-02T14:50:42.224 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 4170 2024-04-02T14:50:42.224 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 4171 2024-04-02T14:50:42.225 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 4172 2024-04-02T14:50:42.225 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 4173 2024-04-02T14:50:42.225 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 4174 2024-04-02T14:50:42.225 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 5002 2024-04-02T14:50:42.225 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 548 2024-04-02T14:50:42.225 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 549 2024-04-02T14:50:42.225 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 6102 2024-04-02T14:50:42.225 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 6104 2024-04-02T14:50:42.225 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 6106 2024-04-02T14:50:42.225 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 6110 2024-04-02T14:50:42.225 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 6111 2024-04-02T14:50:42.225 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 6112 2024-04-02T14:50:42.225 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 6113 2024-04-02T14:50:42.225 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 6117 2024-04-02T14:50:42.225 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 826 2024-04-02T14:50:42.225 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 827 2024-04-02T14:50:42.225 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 828 2024-04-02T14:50:42.226 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 512 Apr 2 14:35 pg_filenode.map 2024-04-02T14:50:42.226 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 151596 Apr 2 14:35 pg_internal.init 2024-04-02T14:50:42.226 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Apr 2 14:35 PG_VERSION 2024-04-02T14:50:42.226 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-02T14:50:42.226 INFO:teuthology.orchestra.run.smithi090.stdout:/tmp/cephfs/postgres/data/base/pgsql_tmp: 2024-04-02T14:50:42.226 INFO:teuthology.orchestra.run.smithi090.stdout:total 0 2024-04-02T14:50:42.226 INFO:teuthology.orchestra.run.smithi090.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 . 2024-04-02T14:50:42.226 INFO:teuthology.orchestra.run.smithi090.stdout:drwx------. 6 postgres postgres system_u:object_r:postgresql_db_t:s0 4 Apr 2 14:35 .. 2024-04-02T14:50:42.226 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-02T14:50:42.226 INFO:teuthology.orchestra.run.smithi090.stdout:/tmp/cephfs/postgres/data/global: 2024-04-02T14:50:42.226 INFO:teuthology.orchestra.run.smithi090.stdout:total 560 2024-04-02T14:50:42.226 INFO:teuthology.orchestra.run.smithi090.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 58 Apr 2 14:35 . 2024-04-02T14:50:42.226 INFO:teuthology.orchestra.run.smithi090.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Apr 2 14:50 .. 2024-04-02T14:50:42.226 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 1213 2024-04-02T14:50:42.226 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 1213_fsm 2024-04-02T14:50:42.226 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 1213_vm 2024-04-02T14:50:42.226 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 1214 2024-04-02T14:50:42.227 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 1214_fsm 2024-04-02T14:50:42.227 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 1214_vm 2024-04-02T14:50:42.227 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 1232 2024-04-02T14:50:42.227 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 1233 2024-04-02T14:50:42.227 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 1260 2024-04-02T14:50:42.227 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 1260_fsm 2024-04-02T14:50:42.227 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 1260_vm 2024-04-02T14:50:42.227 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 1261 2024-04-02T14:50:42.227 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 1261_fsm 2024-04-02T14:50:42.227 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 1261_vm 2024-04-02T14:50:42.227 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 1262 2024-04-02T14:50:42.227 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 1262_fsm 2024-04-02T14:50:42.227 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 1262_vm 2024-04-02T14:50:42.227 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2396 2024-04-02T14:50:42.227 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 2 14:35 2396_fsm 2024-04-02T14:50:42.227 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2396_vm 2024-04-02T14:50:42.228 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2397 2024-04-02T14:50:42.228 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2671 2024-04-02T14:50:42.228 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2672 2024-04-02T14:50:42.228 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2676 2024-04-02T14:50:42.228 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2677 2024-04-02T14:50:42.228 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2694 2024-04-02T14:50:42.228 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2695 2024-04-02T14:50:42.228 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2697 2024-04-02T14:50:42.228 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 2 14:35 2698 2024-04-02T14:50:42.228 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 2846 2024-04-02T14:50:42.228 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2847 2024-04-02T14:50:42.228 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 2964 2024-04-02T14:50:42.228 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2965 2024-04-02T14:50:42.228 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 2966 2024-04-02T14:50:42.228 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 2967 2024-04-02T14:50:42.228 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 3592 2024-04-02T14:50:42.229 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 3593 2024-04-02T14:50:42.229 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 4060 2024-04-02T14:50:42.229 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 4061 2024-04-02T14:50:42.229 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 4175 2024-04-02T14:50:42.229 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 4176 2024-04-02T14:50:42.229 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 4177 2024-04-02T14:50:42.229 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 4178 2024-04-02T14:50:42.229 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 4181 2024-04-02T14:50:42.229 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 4182 2024-04-02T14:50:42.229 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 4183 2024-04-02T14:50:42.229 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 4184 2024-04-02T14:50:42.229 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 4185 2024-04-02T14:50:42.229 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 4186 2024-04-02T14:50:42.229 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 6000 2024-04-02T14:50:42.229 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 6001 2024-04-02T14:50:42.230 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 6002 2024-04-02T14:50:42.230 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 6100 2024-04-02T14:50:42.230 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 6114 2024-04-02T14:50:42.230 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 6115 2024-04-02T14:50:42.230 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:50 pg_control 2024-04-02T14:50:42.230 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 512 Apr 2 14:35 pg_filenode.map 2024-04-02T14:50:42.230 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 23144 Apr 2 14:35 pg_internal.init 2024-04-02T14:50:42.230 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-02T14:50:42.230 INFO:teuthology.orchestra.run.smithi090.stdout:/tmp/cephfs/postgres/data/log: 2024-04-02T14:50:42.230 INFO:teuthology.orchestra.run.smithi090.stdout:total 2 2024-04-02T14:50:42.230 INFO:teuthology.orchestra.run.smithi090.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Apr 2 14:35 . 2024-04-02T14:50:42.230 INFO:teuthology.orchestra.run.smithi090.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Apr 2 14:50 .. 2024-04-02T14:50:42.230 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 1118 Apr 2 14:50 postgresql-Tue.log 2024-04-02T14:50:42.230 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-02T14:50:42.230 INFO:teuthology.orchestra.run.smithi090.stdout:/tmp/cephfs/postgres/data/pg_commit_ts: 2024-04-02T14:50:42.231 INFO:teuthology.orchestra.run.smithi090.stdout:total 0 2024-04-02T14:50:42.231 INFO:teuthology.orchestra.run.smithi090.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 . 2024-04-02T14:50:42.231 INFO:teuthology.orchestra.run.smithi090.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Apr 2 14:50 .. 2024-04-02T14:50:42.231 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-02T14:50:42.231 INFO:teuthology.orchestra.run.smithi090.stdout:/tmp/cephfs/postgres/data/pg_dynshmem: 2024-04-02T14:50:42.231 INFO:teuthology.orchestra.run.smithi090.stdout:total 0 2024-04-02T14:50:42.231 INFO:teuthology.orchestra.run.smithi090.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 . 2024-04-02T14:50:42.231 INFO:teuthology.orchestra.run.smithi090.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Apr 2 14:50 .. 2024-04-02T14:50:42.231 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-02T14:50:42.231 INFO:teuthology.orchestra.run.smithi090.stdout:/tmp/cephfs/postgres/data/pg_logical: 2024-04-02T14:50:42.231 INFO:teuthology.orchestra.run.smithi090.stdout:total 1 2024-04-02T14:50:42.231 INFO:teuthology.orchestra.run.smithi090.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Apr 2 14:50 . 2024-04-02T14:50:42.231 INFO:teuthology.orchestra.run.smithi090.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Apr 2 14:50 .. 2024-04-02T14:50:42.231 INFO:teuthology.orchestra.run.smithi090.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 mappings 2024-04-02T14:50:42.231 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8 Apr 2 14:50 replorigin_checkpoint 2024-04-02T14:50:42.231 INFO:teuthology.orchestra.run.smithi090.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 snapshots 2024-04-02T14:50:42.232 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-02T14:50:42.232 INFO:teuthology.orchestra.run.smithi090.stdout:/tmp/cephfs/postgres/data/pg_logical/mappings: 2024-04-02T14:50:42.232 INFO:teuthology.orchestra.run.smithi090.stdout:total 0 2024-04-02T14:50:42.232 INFO:teuthology.orchestra.run.smithi090.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 . 2024-04-02T14:50:42.232 INFO:teuthology.orchestra.run.smithi090.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Apr 2 14:50 .. 2024-04-02T14:50:42.232 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-02T14:50:42.232 INFO:teuthology.orchestra.run.smithi090.stdout:/tmp/cephfs/postgres/data/pg_logical/snapshots: 2024-04-02T14:50:42.232 INFO:teuthology.orchestra.run.smithi090.stdout:total 0 2024-04-02T14:50:42.232 INFO:teuthology.orchestra.run.smithi090.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 . 2024-04-02T14:50:42.232 INFO:teuthology.orchestra.run.smithi090.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Apr 2 14:50 .. 2024-04-02T14:50:42.232 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-02T14:50:42.232 INFO:teuthology.orchestra.run.smithi090.stdout:/tmp/cephfs/postgres/data/pg_multixact: 2024-04-02T14:50:42.232 INFO:teuthology.orchestra.run.smithi090.stdout:total 0 2024-04-02T14:50:42.232 INFO:teuthology.orchestra.run.smithi090.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 2 Apr 2 14:35 . 2024-04-02T14:50:42.233 INFO:teuthology.orchestra.run.smithi090.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Apr 2 14:50 .. 2024-04-02T14:50:42.233 INFO:teuthology.orchestra.run.smithi090.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Apr 2 14:35 members 2024-04-02T14:50:42.233 INFO:teuthology.orchestra.run.smithi090.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Apr 2 14:35 offsets 2024-04-02T14:50:42.233 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-02T14:50:42.233 INFO:teuthology.orchestra.run.smithi090.stdout:/tmp/cephfs/postgres/data/pg_multixact/members: 2024-04-02T14:50:42.233 INFO:teuthology.orchestra.run.smithi090.stdout:total 8 2024-04-02T14:50:42.233 INFO:teuthology.orchestra.run.smithi090.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Apr 2 14:35 . 2024-04-02T14:50:42.233 INFO:teuthology.orchestra.run.smithi090.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 2 Apr 2 14:35 .. 2024-04-02T14:50:42.233 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:35 0000 2024-04-02T14:50:42.233 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-02T14:50:42.233 INFO:teuthology.orchestra.run.smithi090.stdout:/tmp/cephfs/postgres/data/pg_multixact/offsets: 2024-04-02T14:50:42.233 INFO:teuthology.orchestra.run.smithi090.stdout:total 8 2024-04-02T14:50:42.233 INFO:teuthology.orchestra.run.smithi090.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Apr 2 14:35 . 2024-04-02T14:50:42.233 INFO:teuthology.orchestra.run.smithi090.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 2 Apr 2 14:35 .. 2024-04-02T14:50:42.233 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 2 14:38 0000 2024-04-02T14:50:42.233 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-02T14:50:42.234 INFO:teuthology.orchestra.run.smithi090.stdout:/tmp/cephfs/postgres/data/pg_notify: 2024-04-02T14:50:42.234 INFO:teuthology.orchestra.run.smithi090.stdout:total 0 2024-04-02T14:50:42.234 INFO:teuthology.orchestra.run.smithi090.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 . 2024-04-02T14:50:42.234 INFO:teuthology.orchestra.run.smithi090.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Apr 2 14:50 .. 2024-04-02T14:50:42.234 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-02T14:50:42.234 INFO:teuthology.orchestra.run.smithi090.stdout:/tmp/cephfs/postgres/data/pg_replslot: 2024-04-02T14:50:42.234 INFO:teuthology.orchestra.run.smithi090.stdout:total 0 2024-04-02T14:50:42.234 INFO:teuthology.orchestra.run.smithi090.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 . 2024-04-02T14:50:42.235 INFO:teuthology.orchestra.run.smithi090.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Apr 2 14:50 .. 2024-04-02T14:50:42.235 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-02T14:50:42.235 INFO:teuthology.orchestra.run.smithi090.stdout:/tmp/cephfs/postgres/data/pg_serial: 2024-04-02T14:50:42.235 INFO:teuthology.orchestra.run.smithi090.stdout:total 0 2024-04-02T14:50:42.235 INFO:teuthology.orchestra.run.smithi090.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 . 2024-04-02T14:50:42.235 INFO:teuthology.orchestra.run.smithi090.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Apr 2 14:50 .. 2024-04-02T14:50:42.235 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-02T14:50:42.235 INFO:teuthology.orchestra.run.smithi090.stdout:/tmp/cephfs/postgres/data/pg_snapshots: 2024-04-02T14:50:42.235 INFO:teuthology.orchestra.run.smithi090.stdout:total 0 2024-04-02T14:50:42.235 INFO:teuthology.orchestra.run.smithi090.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 . 2024-04-02T14:50:42.235 INFO:teuthology.orchestra.run.smithi090.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Apr 2 14:50 .. 2024-04-02T14:50:42.235 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-02T14:50:42.236 INFO:teuthology.orchestra.run.smithi090.stdout:/tmp/cephfs/postgres/data/pg_stat: 2024-04-02T14:50:42.236 INFO:teuthology.orchestra.run.smithi090.stdout:total 15 2024-04-02T14:50:42.236 INFO:teuthology.orchestra.run.smithi090.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Apr 2 14:50 . 2024-04-02T14:50:42.236 INFO:teuthology.orchestra.run.smithi090.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Apr 2 14:50 .. 2024-04-02T14:50:42.236 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 1952 Apr 2 14:50 db_0.stat 2024-04-02T14:50:42.236 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 10979 Apr 2 14:50 db_13494.stat 2024-04-02T14:50:42.236 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 1151 Apr 2 14:50 global.stat 2024-04-02T14:50:42.236 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-02T14:50:42.236 INFO:teuthology.orchestra.run.smithi090.stdout:/tmp/cephfs/postgres/data/pg_stat_tmp: 2024-04-02T14:50:42.236 INFO:teuthology.orchestra.run.smithi090.stdout:total 0 2024-04-02T14:50:42.236 INFO:teuthology.orchestra.run.smithi090.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:50 . 2024-04-02T14:50:42.236 INFO:teuthology.orchestra.run.smithi090.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Apr 2 14:50 .. 2024-04-02T14:50:42.236 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-02T14:50:42.236 INFO:teuthology.orchestra.run.smithi090.stdout:/tmp/cephfs/postgres/data/pg_subtrans: 2024-04-02T14:50:42.237 INFO:teuthology.orchestra.run.smithi090.stdout:total 240 2024-04-02T14:50:42.237 INFO:teuthology.orchestra.run.smithi090.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Apr 2 14:50 . 2024-04-02T14:50:42.237 INFO:teuthology.orchestra.run.smithi090.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Apr 2 14:50 .. 2024-04-02T14:50:42.237 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 245760 Apr 2 14:50 001A 2024-04-02T14:50:42.237 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-02T14:50:42.237 INFO:teuthology.orchestra.run.smithi090.stdout:/tmp/cephfs/postgres/data/pg_tblspc: 2024-04-02T14:50:42.237 INFO:teuthology.orchestra.run.smithi090.stdout:total 0 2024-04-02T14:50:42.237 INFO:teuthology.orchestra.run.smithi090.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 . 2024-04-02T14:50:42.237 INFO:teuthology.orchestra.run.smithi090.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Apr 2 14:50 .. 2024-04-02T14:50:42.237 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-02T14:50:42.237 INFO:teuthology.orchestra.run.smithi090.stdout:/tmp/cephfs/postgres/data/pg_twophase: 2024-04-02T14:50:42.237 INFO:teuthology.orchestra.run.smithi090.stdout:total 0 2024-04-02T14:50:42.237 INFO:teuthology.orchestra.run.smithi090.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 . 2024-04-02T14:50:42.237 INFO:teuthology.orchestra.run.smithi090.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Apr 2 14:50 .. 2024-04-02T14:50:42.237 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-02T14:50:42.238 INFO:teuthology.orchestra.run.smithi090.stdout:/tmp/cephfs/postgres/data/pg_wal: 2024-04-02T14:50:42.238 INFO:teuthology.orchestra.run.smithi090.stdout:total 1048576 2024-04-02T14:50:42.238 INFO:teuthology.orchestra.run.smithi090.stdout:drwx------. 3 postgres postgres system_u:object_r:postgresql_db_t:s0 65 Apr 2 14:50 . 2024-04-02T14:50:42.238 INFO:teuthology.orchestra.run.smithi090.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Apr 2 14:50 .. 2024-04-02T14:50:42.238 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 2 14:50 00000001000000000000009E 2024-04-02T14:50:42.238 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 2 14:42 00000001000000000000009F 2024-04-02T14:50:42.238 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 2 14:45 0000000100000000000000A0 2024-04-02T14:50:42.238 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 2 14:43 0000000100000000000000A1 2024-04-02T14:50:42.238 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 2 14:42 0000000100000000000000A2 2024-04-02T14:50:42.238 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 2 14:43 0000000100000000000000A3 2024-04-02T14:50:42.238 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 2 14:44 0000000100000000000000A4 2024-04-02T14:50:42.238 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 2 14:42 0000000100000000000000A5 2024-04-02T14:50:42.238 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 2 14:43 0000000100000000000000A6 2024-04-02T14:50:42.238 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 2 14:42 0000000100000000000000A7 2024-04-02T14:50:42.238 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 2 14:42 0000000100000000000000A8 2024-04-02T14:50:42.239 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 2 14:43 0000000100000000000000A9 2024-04-02T14:50:42.239 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 2 14:44 0000000100000000000000AA 2024-04-02T14:50:42.239 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 2 14:44 0000000100000000000000AB 2024-04-02T14:50:42.239 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 2 14:42 0000000100000000000000AC 2024-04-02T14:50:42.239 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 2 14:43 0000000100000000000000AD 2024-04-02T14:50:42.239 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 2 14:42 0000000100000000000000AE 2024-04-02T14:50:42.239 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 2 14:43 0000000100000000000000AF 2024-04-02T14:50:42.239 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 2 14:43 0000000100000000000000B0 2024-04-02T14:50:42.239 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 2 14:42 0000000100000000000000B1 2024-04-02T14:50:42.239 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 2 14:43 0000000100000000000000B2 2024-04-02T14:50:42.239 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 2 14:43 0000000100000000000000B3 2024-04-02T14:50:42.239 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 2 14:42 0000000100000000000000B4 2024-04-02T14:50:42.239 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 2 14:43 0000000100000000000000B5 2024-04-02T14:50:42.239 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 2 14:42 0000000100000000000000B6 2024-04-02T14:50:42.239 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 2 14:46 0000000100000000000000B7 2024-04-02T14:50:42.239 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 2 14:44 0000000100000000000000B8 2024-04-02T14:50:42.240 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 2 14:47 0000000100000000000000B9 2024-04-02T14:50:42.240 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 2 14:46 0000000100000000000000BA 2024-04-02T14:50:42.240 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 2 14:45 0000000100000000000000BB 2024-04-02T14:50:42.240 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 2 14:42 0000000100000000000000BC 2024-04-02T14:50:42.240 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 2 14:48 0000000100000000000000BD 2024-04-02T14:50:42.240 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 2 14:47 0000000100000000000000BE 2024-04-02T14:50:42.240 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 2 14:48 0000000100000000000000BF 2024-04-02T14:50:42.240 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 2 14:47 0000000100000000000000C0 2024-04-02T14:50:42.240 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 2 14:49 0000000100000000000000C1 2024-04-02T14:50:42.240 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 2 14:49 0000000100000000000000C2 2024-04-02T14:50:42.240 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 2 14:47 0000000100000000000000C3 2024-04-02T14:50:42.240 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 2 14:48 0000000100000000000000C4 2024-04-02T14:50:42.240 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 2 14:48 0000000100000000000000C5 2024-04-02T14:50:42.240 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 2 14:47 0000000100000000000000C6 2024-04-02T14:50:42.241 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 2 14:48 0000000100000000000000C7 2024-04-02T14:50:42.241 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 2 14:48 0000000100000000000000C8 2024-04-02T14:50:42.241 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 2 14:47 0000000100000000000000C9 2024-04-02T14:50:42.241 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 2 14:49 0000000100000000000000CA 2024-04-02T14:50:42.241 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 2 14:48 0000000100000000000000CB 2024-04-02T14:50:42.241 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 2 14:49 0000000100000000000000CC 2024-04-02T14:50:42.241 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 2 14:48 0000000100000000000000CD 2024-04-02T14:50:42.241 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 2 14:47 0000000100000000000000CE 2024-04-02T14:50:42.241 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 2 14:50 0000000100000000000000CF 2024-04-02T14:50:42.242 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 2 14:47 0000000100000000000000D0 2024-04-02T14:50:42.242 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 2 14:48 0000000100000000000000D1 2024-04-02T14:50:42.242 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 2 14:47 0000000100000000000000D2 2024-04-02T14:50:42.242 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 2 14:47 0000000100000000000000D3 2024-04-02T14:50:42.242 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 2 14:48 0000000100000000000000D4 2024-04-02T14:50:42.242 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 2 14:48 0000000100000000000000D5 2024-04-02T14:50:42.242 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 2 14:48 0000000100000000000000D6 2024-04-02T14:50:42.242 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 2 14:49 0000000100000000000000D7 2024-04-02T14:50:42.242 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 2 14:47 0000000100000000000000D8 2024-04-02T14:50:42.242 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 2 14:49 0000000100000000000000D9 2024-04-02T14:50:42.242 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 2 14:48 0000000100000000000000DA 2024-04-02T14:50:42.243 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 2 14:48 0000000100000000000000DB 2024-04-02T14:50:42.243 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 2 14:48 0000000100000000000000DC 2024-04-02T14:50:42.243 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 2 14:48 0000000100000000000000DD 2024-04-02T14:50:42.243 INFO:teuthology.orchestra.run.smithi090.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 archive_status 2024-04-02T14:50:42.243 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-02T14:50:42.243 INFO:teuthology.orchestra.run.smithi090.stdout:/tmp/cephfs/postgres/data/pg_wal/archive_status: 2024-04-02T14:50:42.243 INFO:teuthology.orchestra.run.smithi090.stdout:total 0 2024-04-02T14:50:42.243 INFO:teuthology.orchestra.run.smithi090.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 2 14:35 . 2024-04-02T14:50:42.243 INFO:teuthology.orchestra.run.smithi090.stdout:drwx------. 3 postgres postgres system_u:object_r:postgresql_db_t:s0 65 Apr 2 14:50 .. 2024-04-02T14:50:42.243 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-02T14:50:42.243 INFO:teuthology.orchestra.run.smithi090.stdout:/tmp/cephfs/postgres/data/pg_xact: 2024-04-02T14:50:42.244 INFO:teuthology.orchestra.run.smithi090.stdout:total 432 2024-04-02T14:50:42.244 INFO:teuthology.orchestra.run.smithi090.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 2 Apr 2 14:42 . 2024-04-02T14:50:42.244 INFO:teuthology.orchestra.run.smithi090.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Apr 2 14:50 .. 2024-04-02T14:50:42.244 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 262144 Apr 2 14:42 0000 2024-04-02T14:50:42.244 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 180224 Apr 2 14:50 0001 2024-04-02T14:50:42.246 DEBUG:teuthology.run_tasks:Unwinding manager background_exec 2024-04-02T14:50:42.259 INFO:teuthology.task.background_exec:Stopping background command on ubuntu@smithi090.front.sepia.ceph.com 2024-04-02T14:50:42.286 DEBUG:teuthology.run_tasks:Unwinding manager full_sequential_finally 2024-04-02T14:50:42.307 INFO:teuthology.task.full_sequential_finally:In full_sequential_finally, running task exec... 2024-04-02T14:50:42.307 INFO:teuthology.task.exec:Executing custom commands... 2024-04-02T14:50:42.307 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi090.front.sepia.ceph.com 2024-04-02T14:50:42.307 DEBUG:teuthology.orchestra.run.smithi090:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > END_TIME' 2024-04-02T14:50:42.429 DEBUG:teuthology.orchestra.run.smithi090:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi' 2024-04-02T14:50:42.504 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:50:42 smithi090 ceph-mon[31674]: pgmap v495: 97 pgs: 97 active+clean; 4.3 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 15 MiB/s wr, 1.53k op/s 2024-04-02T14:50:42.523 DEBUG:teuthology.orchestra.run.smithi090:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-04-02T14:50:42.602 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:50:42 smithi092 ceph-mon[36497]: pgmap v495: 97 pgs: 97 active+clean; 4.3 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 15 MiB/s wr, 1.53k op/s 2024-04-02T14:50:42.678 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:50:42 smithi177 ceph-mon[36431]: pgmap v495: 97 pgs: 97 active+clean; 4.3 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 15 MiB/s wr, 1.53k op/s 2024-04-02T14:50:42.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:42.729+0000 7fcdfb400640 1 -- 172.21.15.90:0/1005615667 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcdf406a840 msgr2=0x7fcdf406ac80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:42.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:42.729+0000 7fcdfb400640 1 --2- 172.21.15.90:0/1005615667 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcdf406a840 0x7fcdf406ac80 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7fcde8009900 tx=0x7fcde802bcd0 comp rx=0 tx=0).stop 2024-04-02T14:50:42.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:42.729+0000 7fcdfb400640 1 -- 172.21.15.90:0/1005615667 shutdown_connections 2024-04-02T14:50:42.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:42.729+0000 7fcdfb400640 1 --2- 172.21.15.90:0/1005615667 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcdf410e2c0 0x7fcdf4110730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:42.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:42.729+0000 7fcdfb400640 1 --2- 172.21.15.90:0/1005615667 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcdf406a840 0x7fcdf406ac80 unknown :-1 s=CLOSED pgs=1036 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:42.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:42.729+0000 7fcdfb400640 1 --2- 172.21.15.90:0/1005615667 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcdf4102a60 0x7fcdf406a300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:42.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:42.729+0000 7fcdfb400640 1 -- 172.21.15.90:0/1005615667 >> 172.21.15.90:0/1005615667 conn(0x7fcdf40777d0 msgr2=0x7fcdf4077be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:50:42.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:42.729+0000 7fcdfb400640 1 -- 172.21.15.90:0/1005615667 shutdown_connections 2024-04-02T14:50:42.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:42.729+0000 7fcdfb400640 1 -- 172.21.15.90:0/1005615667 wait complete. 2024-04-02T14:50:42.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:42.730+0000 7fcdfb400640 1 Processor -- start 2024-04-02T14:50:42.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:42.730+0000 7fcdfb400640 1 -- start start 2024-04-02T14:50:42.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:42.730+0000 7fcdfb400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcdf406a840 0x7fcdf419a440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:42.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:42.730+0000 7fcdfb400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcdf4102a60 0x7fcdf419a980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:42.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:42.730+0000 7fcdfb400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcdf410e2c0 0x7fcdf419ed00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:42.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:42.730+0000 7fcdfb400640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fcdf41140c0 con 0x7fcdf4102a60 2024-04-02T14:50:42.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:42.730+0000 7fcdfb400640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fcdf4113f40 con 0x7fcdf410e2c0 2024-04-02T14:50:42.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:42.730+0000 7fcdfb400640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fcdf4114240 con 0x7fcdf406a840 2024-04-02T14:50:42.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:42.731+0000 7fcdf3e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcdf4102a60 0x7fcdf419a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:42.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:42.731+0000 7fcdf8a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcdf406a840 0x7fcdf419a440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:42.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:42.731+0000 7fcdf3e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcdf4102a60 0x7fcdf419a980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:42634/0 (socket says 172.21.15.90:42634) 2024-04-02T14:50:42.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:42.731+0000 7fcdf3e00640 1 -- 172.21.15.90:0/670665420 learned_addr learned my addr 172.21.15.90:0/670665420 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:50:42.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:42.731+0000 7fcdf9400640 1 --2- 172.21.15.90:0/670665420 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcdf410e2c0 0x7fcdf419ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:42.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:42.731+0000 7fcdf3e00640 1 -- 172.21.15.90:0/670665420 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcdf406a840 msgr2=0x7fcdf419a440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:42.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:42.731+0000 7fcdf3e00640 1 --2- 172.21.15.90:0/670665420 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcdf406a840 0x7fcdf419a440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:42.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:42.731+0000 7fcdf3e00640 1 -- 172.21.15.90:0/670665420 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcdf410e2c0 msgr2=0x7fcdf419ed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:42.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:42.731+0000 7fcdf3e00640 1 --2- 172.21.15.90:0/670665420 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcdf410e2c0 0x7fcdf419ed00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:42.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:42.731+0000 7fcdf3e00640 1 -- 172.21.15.90:0/670665420 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcdf419f430 con 0x7fcdf4102a60 2024-04-02T14:50:42.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:42.731+0000 7fcdf9400640 1 --2- 172.21.15.90:0/670665420 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcdf410e2c0 0x7fcdf419ed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:50:42.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:42.732+0000 7fcdf3e00640 1 --2- 172.21.15.90:0/670665420 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcdf4102a60 0x7fcdf419a980 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fcde8000c00 tx=0x7fcde8004360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:50:42.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:42.732+0000 7fcdf1600640 1 -- 172.21.15.90:0/670665420 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcde8042020 con 0x7fcdf4102a60 2024-04-02T14:50:42.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:42.732+0000 7fcdf1600640 1 -- 172.21.15.90:0/670665420 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcde80054e0 con 0x7fcdf4102a60 2024-04-02T14:50:42.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:42.732+0000 7fcdfb400640 1 -- 172.21.15.90:0/670665420 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcdf419f6c0 con 0x7fcdf4102a60 2024-04-02T14:50:42.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:42.732+0000 7fcdf1600640 1 -- 172.21.15.90:0/670665420 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcde8003af0 con 0x7fcdf4102a60 2024-04-02T14:50:42.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:42.732+0000 7fcdfb400640 1 -- 172.21.15.90:0/670665420 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcdf41a6fb0 con 0x7fcdf4102a60 2024-04-02T14:50:42.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:42.733+0000 7fcdfb400640 1 -- 172.21.15.90:0/670665420 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcdf4100ff0 con 0x7fcdf4102a60 2024-04-02T14:50:42.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:42.735+0000 7fcdf1600640 1 -- 172.21.15.90:0/670665420 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fcde8045020 con 0x7fcdf4102a60 2024-04-02T14:50:42.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:42.736+0000 7fcdf1600640 1 --2- 172.21.15.90:0/670665420 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fcddc0a3dc0 0x7fcddc0a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:42.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:42.736+0000 7fcdf1600640 1 -- 172.21.15.90:0/670665420 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 8550+0+0 (secure 0 0 0) 0x7fcde80f2350 con 0x7fcdf4102a60 2024-04-02T14:50:42.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:42.737+0000 7fcdf8a00640 1 --2- 172.21.15.90:0/670665420 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fcddc0a3dc0 0x7fcddc0a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:42.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:42.737+0000 7fcdf8a00640 1 --2- 172.21.15.90:0/670665420 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fcddc0a3dc0 0x7fcddc0a6280 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fcde0009bd0 tx=0x7fcde0005f90 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:50:42.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:42.741+0000 7fcdf1600640 1 -- 172.21.15.90:0/670665420 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7fcde804c020 con 0x7fcdf4102a60 2024-04-02T14:50:42.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:42.967+0000 7fcdfb400640 1 -- 172.21.15.90:0/670665420 --> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) -- 0x7fcdf4100350 con 0x7fcddc0a3dc0 2024-04-02T14:50:42.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:42.981+0000 7fcdf1600640 1 -- 172.21.15.90:0/670665420 <== mgr.25289 v2:172.21.15.90:6800/3904993754 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+350 (secure 0 0 0) 0x7fcdf4100350 con 0x7fcddc0a3dc0 2024-04-02T14:50:42.974 INFO:teuthology.orchestra.run.smithi090.stdout:{"fs": "cephfs", "subvol": null, "group": null, "path": "/", "rel_path": "/", "schedule": "1m", "retention": {"m": 6, "h": 3}, "start": "2024-04-02T00:00:00", "created": "2024-04-02T14:35:02", "first": "2024-04-02T14:36:00", "last": "2024-04-02T14:50:00", "last_pruned": "2024-04-02T14:50:01", "created_count": 15, "pruned_count": 9, "active": true} 2024-04-02T14:50:42.977 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:42.984+0000 7fcdfb400640 1 -- 172.21.15.90:0/670665420 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fcddc0a3dc0 msgr2=0x7fcddc0a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:42.977 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:42.984+0000 7fcdfb400640 1 --2- 172.21.15.90:0/670665420 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fcddc0a3dc0 0x7fcddc0a6280 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fcde0009bd0 tx=0x7fcde0005f90 comp rx=0 tx=0).stop 2024-04-02T14:50:42.977 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:42.984+0000 7fcdfb400640 1 -- 172.21.15.90:0/670665420 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcdf4102a60 msgr2=0x7fcdf419a980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:42.977 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:42.984+0000 7fcdfb400640 1 --2- 172.21.15.90:0/670665420 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcdf4102a60 0x7fcdf419a980 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fcde8000c00 tx=0x7fcde8004360 comp rx=0 tx=0).stop 2024-04-02T14:50:42.978 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:42.984+0000 7fcdfb400640 1 -- 172.21.15.90:0/670665420 shutdown_connections 2024-04-02T14:50:42.978 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:42.984+0000 7fcdfb400640 1 --2- 172.21.15.90:0/670665420 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fcddc0a3dc0 0x7fcddc0a6280 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:42.978 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:42.984+0000 7fcdfb400640 1 --2- 172.21.15.90:0/670665420 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcdf410e2c0 0x7fcdf419ed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:42.978 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:42.984+0000 7fcdfb400640 1 --2- 172.21.15.90:0/670665420 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcdf4102a60 0x7fcdf419a980 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:42.978 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:42.984+0000 7fcdfb400640 1 --2- 172.21.15.90:0/670665420 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcdf406a840 0x7fcdf419a440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:42.978 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:42.984+0000 7fcdfb400640 1 -- 172.21.15.90:0/670665420 >> 172.21.15.90:0/670665420 conn(0x7fcdf40777d0 msgr2=0x7fcdf40ffc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:50:42.978 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:42.984+0000 7fcdfb400640 1 -- 172.21.15.90:0/670665420 shutdown_connections 2024-04-02T14:50:42.978 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:42.984+0000 7fcdfb400640 1 -- 172.21.15.90:0/670665420 wait complete. 2024-04-02T14:50:42.996 DEBUG:teuthology.orchestra.run.smithi090:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2024-04-02T14:50:43.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:43.205+0000 7f422b200640 1 -- 172.21.15.90:0/1149900417 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4224108040 msgr2=0x7f4224108480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:43.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:43.205+0000 7f422b200640 1 --2- 172.21.15.90:0/1149900417 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4224108040 0x7f4224108480 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f4214009930 tx=0x7f421402bc20 comp rx=0 tx=0).stop 2024-04-02T14:50:43.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:43.205+0000 7f422b200640 1 -- 172.21.15.90:0/1149900417 shutdown_connections 2024-04-02T14:50:43.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:43.205+0000 7f422b200640 1 --2- 172.21.15.90:0/1149900417 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f42241089c0 0x7f4224114dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:43.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:43.205+0000 7f422b200640 1 --2- 172.21.15.90:0/1149900417 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4224108040 0x7f4224108480 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:43.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:43.205+0000 7f422b200640 1 --2- 172.21.15.90:0/1149900417 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f422410dfa0 0x7f422410e380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:43.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:43.205+0000 7f422b200640 1 -- 172.21.15.90:0/1149900417 >> 172.21.15.90:0/1149900417 conn(0x7f42240778e0 msgr2=0x7f4224106920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:50:43.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:43.205+0000 7f422b200640 1 -- 172.21.15.90:0/1149900417 shutdown_connections 2024-04-02T14:50:43.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:43.205+0000 7f422b200640 1 -- 172.21.15.90:0/1149900417 wait complete. 2024-04-02T14:50:43.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:43.206+0000 7f422b200640 1 Processor -- start 2024-04-02T14:50:43.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:43.206+0000 7f422b200640 1 -- start start 2024-04-02T14:50:43.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:43.207+0000 7f422b200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4224108040 0x7f42241a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:43.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:43.207+0000 7f422b200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f42241089c0 0x7f42241a4ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:43.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:43.207+0000 7f422b200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f422410dfa0 0x7f422419e640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:43.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:43.207+0000 7f422b200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f42241182c0 con 0x7f4224108040 2024-04-02T14:50:43.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:43.207+0000 7f422b200640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f4224118140 con 0x7f42241089c0 2024-04-02T14:50:43.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:43.207+0000 7f422b200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f4224118440 con 0x7f422410dfa0 2024-04-02T14:50:43.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:43.207+0000 7f4229e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4224108040 0x7f42241a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:43.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:43.207+0000 7f4229e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4224108040 0x7f42241a4580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:42648/0 (socket says 172.21.15.90:42648) 2024-04-02T14:50:43.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:43.207+0000 7f4229e00640 1 -- 172.21.15.90:0/1600751995 learned_addr learned my addr 172.21.15.90:0/1600751995 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:50:43.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:43.207+0000 7f422a800640 1 --2- 172.21.15.90:0/1600751995 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f422410dfa0 0x7f422419e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:43.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:43.207+0000 7f4229400640 1 --2- 172.21.15.90:0/1600751995 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f42241089c0 0x7f42241a4ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:43.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:43.207+0000 7f4229e00640 1 -- 172.21.15.90:0/1600751995 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f422410dfa0 msgr2=0x7f422419e640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:43.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:43.207+0000 7f4229e00640 1 --2- 172.21.15.90:0/1600751995 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f422410dfa0 0x7f422419e640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:43.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:43.207+0000 7f4229e00640 1 -- 172.21.15.90:0/1600751995 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f42241089c0 msgr2=0x7f42241a4ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:43.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:43.207+0000 7f4229e00640 1 --2- 172.21.15.90:0/1600751995 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f42241089c0 0x7f42241a4ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:43.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:43.207+0000 7f4229e00640 1 -- 172.21.15.90:0/1600751995 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f422419ef00 con 0x7f4224108040 2024-04-02T14:50:43.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:43.207+0000 7f4229400640 1 --2- 172.21.15.90:0/1600751995 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f42241089c0 0x7f42241a4ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:50:43.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:43.208+0000 7f4229e00640 1 --2- 172.21.15.90:0/1600751995 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4224108040 0x7f42241a4580 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f421800e9f0 tx=0x7f421800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:50:43.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:43.208+0000 7f421ea00640 1 -- 172.21.15.90:0/1600751995 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f421800cd80 con 0x7f4224108040 2024-04-02T14:50:43.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:43.208+0000 7f421ea00640 1 -- 172.21.15.90:0/1600751995 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4218005500 con 0x7f4224108040 2024-04-02T14:50:43.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:43.208+0000 7f421ea00640 1 -- 172.21.15.90:0/1600751995 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4218010690 con 0x7f4224108040 2024-04-02T14:50:43.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:43.208+0000 7f422b200640 1 -- 172.21.15.90:0/1600751995 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f422419f1f0 con 0x7f4224108040 2024-04-02T14:50:43.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:43.208+0000 7f422b200640 1 -- 172.21.15.90:0/1600751995 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42241ab500 con 0x7f4224108040 2024-04-02T14:50:43.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:43.209+0000 7f422b200640 1 -- 172.21.15.90:0/1600751995 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4224109640 con 0x7f4224108040 2024-04-02T14:50:43.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:43.211+0000 7f421ea00640 1 -- 172.21.15.90:0/1600751995 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f42180046c0 con 0x7f4224108040 2024-04-02T14:50:43.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:43.213+0000 7f421ea00640 1 --2- 172.21.15.90:0/1600751995 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f420c0a3d10 0x7f420c0a61d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:43.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:43.213+0000 7f421ea00640 1 -- 172.21.15.90:0/1600751995 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 8550+0+0 (secure 0 0 0) 0x7f42180d0cb0 con 0x7f4224108040 2024-04-02T14:50:43.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:43.213+0000 7f4229400640 1 --2- 172.21.15.90:0/1600751995 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f420c0a3d10 0x7f420c0a61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:43.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:43.214+0000 7f4229400640 1 --2- 172.21.15.90:0/1600751995 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f420c0a3d10 0x7f420c0a61d0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f4214006010 tx=0x7f421402c6c0 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:50:43.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:43.217+0000 7f421ea00640 1 -- 172.21.15.90:0/1600751995 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7f421809d2d0 con 0x7f4224108040 2024-04-02T14:50:43.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:43.444+0000 7f422b200640 1 -- 172.21.15.90:0/1600751995 --> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) -- 0x7f422419f420 con 0x7f420c0a3d10 2024-04-02T14:50:43.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:43.457+0000 7f421ea00640 1 -- 172.21.15.90:0/1600751995 <== mgr.25289 v2:172.21.15.90:6800/3904993754 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+10 (secure 0 0 0) 0x7f422419f420 con 0x7f420c0a3d10 2024-04-02T14:50:43.451 INFO:teuthology.orchestra.run.smithi090.stdout:/ 1m 6m3h 2024-04-02T14:50:43.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:43.460+0000 7f422b200640 1 -- 172.21.15.90:0/1600751995 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f420c0a3d10 msgr2=0x7f420c0a61d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:43.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:43.460+0000 7f422b200640 1 --2- 172.21.15.90:0/1600751995 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f420c0a3d10 0x7f420c0a61d0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f4214006010 tx=0x7f421402c6c0 comp rx=0 tx=0).stop 2024-04-02T14:50:43.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:43.460+0000 7f422b200640 1 -- 172.21.15.90:0/1600751995 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4224108040 msgr2=0x7f42241a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:43.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:43.460+0000 7f422b200640 1 --2- 172.21.15.90:0/1600751995 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4224108040 0x7f42241a4580 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f421800e9f0 tx=0x7f421800edf0 comp rx=0 tx=0).stop 2024-04-02T14:50:43.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:43.460+0000 7f422b200640 1 -- 172.21.15.90:0/1600751995 shutdown_connections 2024-04-02T14:50:43.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:43.460+0000 7f422b200640 1 --2- 172.21.15.90:0/1600751995 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f420c0a3d10 0x7f420c0a61d0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:43.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:43.460+0000 7f422b200640 1 --2- 172.21.15.90:0/1600751995 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f422410dfa0 0x7f422419e640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:43.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:43.460+0000 7f422b200640 1 --2- 172.21.15.90:0/1600751995 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f42241089c0 0x7f42241a4ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:43.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:43.460+0000 7f422b200640 1 --2- 172.21.15.90:0/1600751995 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4224108040 0x7f42241a4580 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:43.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:43.460+0000 7f422b200640 1 -- 172.21.15.90:0/1600751995 >> 172.21.15.90:0/1600751995 conn(0x7f42240778e0 msgr2=0x7f4224078b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:50:43.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:43.461+0000 7f422b200640 1 -- 172.21.15.90:0/1600751995 shutdown_connections 2024-04-02T14:50:43.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:43.461+0000 7f422b200640 1 -- 172.21.15.90:0/1600751995 wait complete. 2024-04-02T14:50:43.472 DEBUG:teuthology.run_tasks:Unwinding manager fwd_scrub 2024-04-02T14:50:43.483 INFO:tasks.fwd_scrub:joining ForwardScrubbers 2024-04-02T14:50:44.602 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:50:44 smithi092 ceph-mon[36497]: pgmap v496: 97 pgs: 97 active+clean; 4.3 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 12 MiB/s wr, 1.25k op/s 2024-04-02T14:50:44.602 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:50:44 smithi092 ceph-mon[36497]: from='client.18972 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T14:50:44.678 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:50:44 smithi177 ceph-mon[36431]: pgmap v496: 97 pgs: 97 active+clean; 4.3 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 12 MiB/s wr, 1.25k op/s 2024-04-02T14:50:44.678 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:50:44 smithi177 ceph-mon[36431]: from='client.18972 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T14:50:44.680 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:50:44 smithi090 ceph-mon[31674]: pgmap v496: 97 pgs: 97 active+clean; 4.3 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 12 MiB/s wr, 1.25k op/s 2024-04-02T14:50:44.680 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:50:44 smithi090 ceph-mon[31674]: from='client.18972 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T14:50:45.602 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:50:45 smithi092 ceph-mon[36497]: from='client.18978 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T14:50:45.678 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:50:45 smithi177 ceph-mon[36431]: from='client.18978 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T14:50:45.680 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:50:45 smithi090 ceph-mon[31674]: from='client.18978 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-02T14:50:46.044 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-02T14:50:46.045 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-02T14:50:46.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.272+0000 7f196d200640 1 -- 172.21.15.90:0/1238702305 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1968108310 msgr2=0x7f19681086f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:46.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.272+0000 7f196d200640 1 --2- 172.21.15.90:0/1238702305 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1968108310 0x7f19681086f0 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7f1958009ca0 tx=0x7f195802c100 comp rx=0 tx=0).stop 2024-04-02T14:50:46.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.272+0000 7f196d200640 1 -- 172.21.15.90:0/1238702305 shutdown_connections 2024-04-02T14:50:46.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.272+0000 7f196d200640 1 --2- 172.21.15.90:0/1238702305 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f196810d3e0 0x7f1968114fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:46.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.272+0000 7f196d200640 1 --2- 172.21.15.90:0/1238702305 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1968108cc0 0x7f196810cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:46.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.272+0000 7f196d200640 1 --2- 172.21.15.90:0/1238702305 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1968108310 0x7f19681086f0 unknown :-1 s=CLOSED pgs=1037 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:46.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.272+0000 7f196d200640 1 -- 172.21.15.90:0/1238702305 >> 172.21.15.90:0/1238702305 conn(0x7f1968076040 msgr2=0x7f1968078460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:50:46.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.272+0000 7f196d200640 1 -- 172.21.15.90:0/1238702305 shutdown_connections 2024-04-02T14:50:46.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.272+0000 7f196d200640 1 -- 172.21.15.90:0/1238702305 wait complete. 2024-04-02T14:50:46.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.273+0000 7f196d200640 1 Processor -- start 2024-04-02T14:50:46.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.273+0000 7f196d200640 1 -- start start 2024-04-02T14:50:46.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.274+0000 7f196d200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1968108310 0x7f19681a4670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:46.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.274+0000 7f196d200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f196810d3e0 0x7f19681a4bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:46.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.274+0000 7f196d200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f196819e7e0 0x7f196819ec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:46.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.274+0000 7f196d200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f19681184a0 con 0x7f1968108310 2024-04-02T14:50:46.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.274+0000 7f196d200640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f1968118320 con 0x7f196810d3e0 2024-04-02T14:50:46.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.274+0000 7f196d200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f1968118620 con 0x7f196819e7e0 2024-04-02T14:50:46.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.274+0000 7f1965e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1968108310 0x7f19681a4670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:46.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.274+0000 7f195d400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f196810d3e0 0x7f19681a4bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:46.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.274+0000 7f195d400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f196810d3e0 0x7f19681a4bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:40412/0 (socket says 172.21.15.90:40412) 2024-04-02T14:50:46.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.274+0000 7f1965e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1968108310 0x7f19681a4670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:37740/0 (socket says 172.21.15.90:37740) 2024-04-02T14:50:46.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.274+0000 7f195d400640 1 -- 172.21.15.90:0/2016864006 learned_addr learned my addr 172.21.15.90:0/2016864006 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:50:46.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.274+0000 7f1966800640 1 --2- 172.21.15.90:0/2016864006 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f196819e7e0 0x7f196819ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:46.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.275+0000 7f195d400640 1 -- 172.21.15.90:0/2016864006 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f196819e7e0 msgr2=0x7f196819ec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:46.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.275+0000 7f195d400640 1 --2- 172.21.15.90:0/2016864006 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f196819e7e0 0x7f196819ec90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:46.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.275+0000 7f195d400640 1 -- 172.21.15.90:0/2016864006 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1968108310 msgr2=0x7f19681a4670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:46.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.275+0000 7f195d400640 1 --2- 172.21.15.90:0/2016864006 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1968108310 0x7f19681a4670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:46.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.275+0000 7f195d400640 1 -- 172.21.15.90:0/2016864006 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f196819f370 con 0x7f196810d3e0 2024-04-02T14:50:46.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.275+0000 7f1966800640 1 --2- 172.21.15.90:0/2016864006 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f196819e7e0 0x7f196819ec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:50:46.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.275+0000 7f195d400640 1 --2- 172.21.15.90:0/2016864006 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f196810d3e0 0x7f19681a4bb0 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7f195000e9c0 tx=0x7f195000ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:50:46.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.275+0000 7f195f400640 1 -- 172.21.15.90:0/2016864006 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f195000cd80 con 0x7f196810d3e0 2024-04-02T14:50:46.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.275+0000 7f195f400640 1 -- 172.21.15.90:0/2016864006 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1950005500 con 0x7f196810d3e0 2024-04-02T14:50:46.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.275+0000 7f195f400640 1 -- 172.21.15.90:0/2016864006 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1950010690 con 0x7f196810d3e0 2024-04-02T14:50:46.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.275+0000 7f196d200640 1 -- 172.21.15.90:0/2016864006 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f196819f660 con 0x7f196810d3e0 2024-04-02T14:50:46.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.275+0000 7f196d200640 1 -- 172.21.15.90:0/2016864006 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19681ab7f0 con 0x7f196810d3e0 2024-04-02T14:50:46.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.278+0000 7f196ff8a740 1 Processor -- start 2024-04-02T14:50:46.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.279+0000 7f196ff8a740 1 -- start start 2024-04-02T14:50:46.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.279+0000 7f195f400640 1 -- 172.21.15.90:0/2016864006 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f19500046c0 con 0x7f196810d3e0 2024-04-02T14:50:46.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.279+0000 7f196ff8a740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556c8971c7b0 0x556c8971cb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:46.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.279+0000 7f196ff8a740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556c8971d0d0 0x556c89725770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:46.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.279+0000 7f196ff8a740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556c89725cb0 0x556c897280a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:46.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.279+0000 7f196ff8a740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x556c89680770 con 0x556c89725cb0 2024-04-02T14:50:46.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.279+0000 7f196ff8a740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x556c89628700 con 0x556c8971d0d0 2024-04-02T14:50:46.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.279+0000 7f196ff8a740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x556c8967c720 con 0x556c8971c7b0 2024-04-02T14:50:46.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.279+0000 7f1965e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556c8971d0d0 0x556c89725770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:46.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.279+0000 7f195d400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556c89725cb0 0x556c897280a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:46.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.279+0000 7f1965e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556c8971d0d0 0x556c89725770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:40416/0 (socket says 172.21.15.90:40416) 2024-04-02T14:50:46.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.279+0000 7f1965e00640 1 -- 172.21.15.90:0/3372329468 learned_addr learned my addr 172.21.15.90:0/3372329468 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:50:46.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.279+0000 7f1966800640 1 --2- 172.21.15.90:0/3372329468 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556c8971c7b0 0x556c8971cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:46.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.280+0000 7f1965e00640 1 -- 172.21.15.90:0/3372329468 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556c8971c7b0 msgr2=0x556c8971cb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:50:46.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.280+0000 7f1965e00640 1 --2- 172.21.15.90:0/3372329468 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556c8971c7b0 0x556c8971cb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:46.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.280+0000 7f1965e00640 1 -- 172.21.15.90:0/3372329468 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556c89725cb0 msgr2=0x556c897280a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:50:46.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.280+0000 7f1965e00640 1 --2- 172.21.15.90:0/3372329468 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556c89725cb0 0x556c897280a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:46.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.280+0000 7f1965e00640 1 -- 172.21.15.90:0/3372329468 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556c8945e7d0 con 0x556c8971d0d0 2024-04-02T14:50:46.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.280+0000 7f1966800640 1 --2- 172.21.15.90:0/3372329468 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556c8971c7b0 0x556c8971cb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:50:46.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.280+0000 7f1965e00640 1 --2- 172.21.15.90:0/3372329468 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556c8971d0d0 0x556c89725770 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7f1958004af0 tx=0x7f1958005fc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ca9fdffbe1d8132d server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:50:46.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.280+0000 7f1957400640 1 -- 172.21.15.90:0/3372329468 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1958043070 con 0x556c8971d0d0 2024-04-02T14:50:46.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.281+0000 7f195f400640 1 --2- 172.21.15.90:0/2016864006 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f194c0a3da0 0x7f194c0a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:46.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.281+0000 7f1957400640 1 -- 172.21.15.90:0/3372329468 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1958003970 con 0x556c8971d0d0 2024-04-02T14:50:46.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.281+0000 7f1957400640 1 -- 172.21.15.90:0/3372329468 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1958039040 con 0x556c8971d0d0 2024-04-02T14:50:46.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.281+0000 7f195f400640 1 -- 172.21.15.90:0/2016864006 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 8550+0+0 (secure 0 0 0) 0x7f19500d0c20 con 0x7f196810d3e0 2024-04-02T14:50:46.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.281+0000 7f1966800640 1 --2- 172.21.15.90:0/2016864006 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f194c0a3da0 0x7f194c0a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:46.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.281+0000 7f196ff8a740 1 -- 172.21.15.90:0/3372329468 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556c8971d0d0 msgr2=0x556c89725770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:46.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.281+0000 7f196ff8a740 1 --2- 172.21.15.90:0/3372329468 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556c8971d0d0 0x556c89725770 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7f1958004af0 tx=0x7f1958005fc0 comp rx=0 tx=0).stop 2024-04-02T14:50:46.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.281+0000 7f196ff8a740 1 -- 172.21.15.90:0/3372329468 shutdown_connections 2024-04-02T14:50:46.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.281+0000 7f196ff8a740 1 --2- 172.21.15.90:0/3372329468 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556c89725cb0 0x556c897280a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:46.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.281+0000 7f196ff8a740 1 --2- 172.21.15.90:0/3372329468 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556c8971d0d0 0x556c89725770 unknown :-1 s=CLOSED pgs=1039 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:46.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.281+0000 7f196ff8a740 1 --2- 172.21.15.90:0/3372329468 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556c8971c7b0 0x556c8971cb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:46.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.281+0000 7f196ff8a740 1 -- 172.21.15.90:0/3372329468 >> 172.21.15.90:0/3372329468 conn(0x556c89716ee0 msgr2=0x556c897172c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:50:46.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.282+0000 7f196ff8a740 1 -- 172.21.15.90:0/3372329468 shutdown_connections 2024-04-02T14:50:46.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.282+0000 7f1966800640 1 --2- 172.21.15.90:0/2016864006 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f194c0a3da0 0x7f194c0a6260 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f1960009e80 tx=0x7f1960005f90 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:50:46.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.282+0000 7f196ff8a740 1 -- 172.21.15.90:0/3372329468 wait complete. 2024-04-02T14:50:46.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.282+0000 7f196ff8a740 1 -- start start 2024-04-02T14:50:46.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.282+0000 7f196ff8a740 1 Processor -- start 2024-04-02T14:50:46.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.283+0000 7f196ff8a740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556c8971c7b0 0x556c89722c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:46.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.283+0000 7f196ff8a740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556c8971d0d0 0x556c897231d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:46.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.283+0000 7f196ff8a740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556c89725cb0 0x556c89723710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:46.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.283+0000 7f196ff8a740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x556c89728a40 con 0x556c8971d0d0 2024-04-02T14:50:46.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.283+0000 7f196ff8a740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x556c8963bac0 con 0x556c8971c7b0 2024-04-02T14:50:46.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.283+0000 7f196ff8a740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x556c89728bc0 con 0x556c89725cb0 2024-04-02T14:50:46.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.283+0000 7f196ff8a740 10 client.0 _ll_register_callbacks cb 0x556c8963e180 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:50:46.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.283+0000 7f195d400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556c8971d0d0 0x556c897231d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:46.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.283+0000 7f195d400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556c8971d0d0 0x556c897231d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:37750/0 (socket says 172.21.15.90:37750) 2024-04-02T14:50:46.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.283+0000 7f195d400640 1 -- 172.21.15.90:0/4041274670 learned_addr learned my addr 172.21.15.90:0/4041274670 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:50:46.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.283+0000 7f1965e00640 1 --2- 172.21.15.90:0/4041274670 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556c8971c7b0 0x556c89722c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:46.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.283+0000 7f1966800640 1 --2- 172.21.15.90:0/4041274670 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556c89725cb0 0x556c89723710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:46.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.284+0000 7f195d400640 1 -- 172.21.15.90:0/4041274670 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556c89725cb0 msgr2=0x556c89723710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:50:46.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.284+0000 7f195d400640 1 --2- 172.21.15.90:0/4041274670 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556c89725cb0 0x556c89723710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:46.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.284+0000 7f195d400640 1 -- 172.21.15.90:0/4041274670 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556c8971c7b0 msgr2=0x556c89722c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:50:46.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.284+0000 7f195d400640 1 --2- 172.21.15.90:0/4041274670 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556c8971c7b0 0x556c89722c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:46.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.284+0000 7f195d400640 1 -- 172.21.15.90:0/4041274670 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556c893943f0 con 0x556c8971d0d0 2024-04-02T14:50:46.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.284+0000 7f1966800640 1 --2- 172.21.15.90:0/4041274670 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556c89725cb0 0x556c89723710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:50:46.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.284+0000 7f1965e00640 1 --2- 172.21.15.90:0/4041274670 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556c8971c7b0 0x556c89722c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:50:46.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.284+0000 7f195d400640 1 --2- 172.21.15.90:0/4041274670 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556c8971d0d0 0x556c897231d0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f195009c840 tx=0x7f19500d4200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d4c4a7062277280 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:50:46.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.284+0000 7f194be00640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-02T14:50:46.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.284+0000 7f194be00640 1 -- 172.21.15.90:0/4041274670 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f195009c910 con 0x556c8971d0d0 2024-04-02T14:50:46.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.284+0000 7f194be00640 1 -- 172.21.15.90:0/4041274670 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 8550+0+0 (secure 0 0 0) 0x7f19500a5320 con 0x556c8971d0d0 2024-04-02T14:50:46.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.284+0000 7f196ff8a740 1 -- 172.21.15.90:0/4041274670 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1958003970 con 0x556c8971d0d0 2024-04-02T14:50:46.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.284+0000 7f194be00640 1 -- 172.21.15.90:0/4041274670 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f195009c4a0 con 0x556c8971d0d0 2024-04-02T14:50:46.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.284+0000 7f196ff8a740 10 client.18990 fetch_fsmap learned FSMap version 27 2024-04-02T14:50:46.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.284+0000 7f196ff8a740 1 -- 172.21.15.90:0/4041274670 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x556c893540f0 con 0x556c8971d0d0 2024-04-02T14:50:46.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.285+0000 7f194be00640 1 -- 172.21.15.90:0/4041274670 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1958003970 con 0x556c8971d0d0 2024-04-02T14:50:46.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.285+0000 7f194be00640 1 -- 172.21.15.90:0/4041274670 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f195009bf90 con 0x556c8971d0d0 2024-04-02T14:50:46.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.285+0000 7f196ff8a740 10 client.18990 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:50:46.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.285+0000 7f196ff8a740 10 client.18990 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:50:46.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.285+0000 7f196ff8a740 20 client.18990 populate_metadata read hostname 'smithi090' 2024-04-02T14:50:46.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.285+0000 7f196ff8a740 1 --2- 172.21.15.90:0/4041274670 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x556c89735c00 0x556c89738010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:46.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.285+0000 7f196ff8a740 4 client.18990 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:50:46.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.285+0000 7f196ff8a740 1 -- 172.21.15.90:0/4041274670 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556c89621630 con 0x556c89735c00 2024-04-02T14:50:46.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.285+0000 7f1965e00640 1 --2- 172.21.15.90:0/4041274670 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x556c89735c00 0x556c89738010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:46.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.286+0000 7f1965e00640 1 --2- 172.21.15.90:0/4041274670 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x556c89735c00 0x556c89738010 crc :-1 s=READY pgs=396 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=69f346549d5146d9 server_cookie=be8827ccbfbc38e3 in_seq=0 out_seq=0 2024-04-02T14:50:46.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.286+0000 7f194be00640 10 client.18990 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:50:46.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.287+0000 7f194be00640 1 -- 172.21.15.90:0/4041274670 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x556c89621630 con 0x556c89735c00 2024-04-02T14:50:46.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.287+0000 7f194be00640 10 client.18990 handle_command_reply: tid=0 2024-04-02T14:50:46.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.287+0000 7f196ff8a740 1 client.18990 shutdown 2024-04-02T14:50:46.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.287+0000 7f196ff8a740 1 -- 172.21.15.90:0/4041274670 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556c8971d0d0 msgr2=0x556c897231d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:46.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.287+0000 7f196ff8a740 1 --2- 172.21.15.90:0/4041274670 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556c8971d0d0 0x556c897231d0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f195009c840 tx=0x7f19500d4200 comp rx=0 tx=0).stop 2024-04-02T14:50:46.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.287+0000 7f196ff8a740 1 -- 172.21.15.90:0/4041274670 shutdown_connections 2024-04-02T14:50:46.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.287+0000 7f196ff8a740 1 --2- 172.21.15.90:0/4041274670 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x556c89735c00 0x556c89738010 crc :-1 s=READY pgs=396 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:46.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.287+0000 7f196ff8a740 1 --2- 172.21.15.90:0/4041274670 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556c89725cb0 0x556c89723710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:46.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.287+0000 7f196ff8a740 1 --2- 172.21.15.90:0/4041274670 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556c8971d0d0 0x556c897231d0 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:46.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.287+0000 7f194be00640 0 client.18990 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:50:46.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.287+0000 7f196ff8a740 1 --2- 172.21.15.90:0/4041274670 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556c8971c7b0 0x556c89722c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:46.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.287+0000 7f196ff8a740 1 -- 172.21.15.90:0/4041274670 >> 172.21.15.90:0/4041274670 conn(0x556c89716ee0 msgr2=0x556c89718c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:50:46.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.287+0000 7f196ff8a740 1 -- 172.21.15.90:0/4041274670 shutdown_connections 2024-04-02T14:50:46.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.287+0000 7f196ff8a740 1 -- 172.21.15.90:0/4041274670 wait complete. 2024-04-02T14:50:46.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.287+0000 7f196ff8a740 20 client.18990 trim_cache size 0 max 16384 2024-04-02T14:50:46.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.306+0000 7f196ff8a740 1 Processor -- start 2024-04-02T14:50:46.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.306+0000 7f196ff8a740 1 -- start start 2024-04-02T14:50:46.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.306+0000 7f196ff8a740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556c8971c7b0 0x556c897253c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:46.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.306+0000 7f196ff8a740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556c8971d0d0 0x556c89721bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:46.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.306+0000 7f196ff8a740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556c89725cb0 0x556c89731960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:46.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.306+0000 7f196ff8a740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f19681b3ea0 con 0x556c8971d0d0 2024-04-02T14:50:46.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.306+0000 7f196ff8a740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f19681b59e0 con 0x556c89725cb0 2024-04-02T14:50:46.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.306+0000 7f196ff8a740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f19681b4570 con 0x556c8971c7b0 2024-04-02T14:50:46.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.307+0000 7f195d400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556c8971d0d0 0x556c89721bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:46.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.307+0000 7f1965e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556c8971c7b0 0x556c897253c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:46.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.307+0000 7f195d400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556c8971d0d0 0x556c89721bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:37752/0 (socket says 172.21.15.90:37752) 2024-04-02T14:50:46.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.307+0000 7f195d400640 1 -- 172.21.15.90:0/2274472954 learned_addr learned my addr 172.21.15.90:0/2274472954 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:50:46.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.307+0000 7f1965e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556c8971c7b0 0x556c897253c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:50876/0 (socket says 172.21.15.90:50876) 2024-04-02T14:50:46.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.307+0000 7f1966800640 1 --2- 172.21.15.90:0/2274472954 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556c89725cb0 0x556c89731960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:46.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.307+0000 7f195d400640 1 -- 172.21.15.90:0/2274472954 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556c8971c7b0 msgr2=0x556c897253c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:50:46.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.307+0000 7f195d400640 1 --2- 172.21.15.90:0/2274472954 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556c8971c7b0 0x556c897253c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:46.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.307+0000 7f195d400640 1 -- 172.21.15.90:0/2274472954 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556c89725cb0 msgr2=0x556c89731960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:50:46.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.307+0000 7f195d400640 1 --2- 172.21.15.90:0/2274472954 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556c89725cb0 0x556c89731960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:46.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.307+0000 7f195d400640 1 -- 172.21.15.90:0/2274472954 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556c89728980 con 0x556c8971d0d0 2024-04-02T14:50:46.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.307+0000 7f1965e00640 1 --2- 172.21.15.90:0/2274472954 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556c8971c7b0 0x556c897253c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:50:46.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.308+0000 7f195d400640 1 --2- 172.21.15.90:0/2274472954 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556c8971d0d0 0x556c89721bd0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f19500a5280 tx=0x7f19500d43c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8a933fad684f01a server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:50:46.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.308+0000 7f1956a00640 1 -- 172.21.15.90:0/2274472954 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f195009c680 con 0x556c8971d0d0 2024-04-02T14:50:46.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.308+0000 7f1956a00640 1 -- 172.21.15.90:0/2274472954 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f195009d240 con 0x556c8971d0d0 2024-04-02T14:50:46.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.308+0000 7f1956a00640 1 -- 172.21.15.90:0/2274472954 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19500a57b0 con 0x556c8971d0d0 2024-04-02T14:50:46.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.308+0000 7f196ff8a740 1 -- 172.21.15.90:0/2274472954 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556c8971d0d0 msgr2=0x556c89721bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:46.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.308+0000 7f196ff8a740 1 --2- 172.21.15.90:0/2274472954 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556c8971d0d0 0x556c89721bd0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f19500a5280 tx=0x7f19500d43c0 comp rx=0 tx=0).stop 2024-04-02T14:50:46.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.309+0000 7f196ff8a740 1 -- 172.21.15.90:0/2274472954 shutdown_connections 2024-04-02T14:50:46.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.309+0000 7f196ff8a740 1 --2- 172.21.15.90:0/2274472954 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556c89725cb0 0x556c89731960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:46.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.309+0000 7f196ff8a740 1 --2- 172.21.15.90:0/2274472954 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556c8971d0d0 0x556c89721bd0 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:46.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.309+0000 7f196ff8a740 1 --2- 172.21.15.90:0/2274472954 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556c8971c7b0 0x556c897253c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:46.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.309+0000 7f196ff8a740 1 -- 172.21.15.90:0/2274472954 >> 172.21.15.90:0/2274472954 conn(0x556c89716ee0 msgr2=0x556c897234e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:50:46.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.309+0000 7f196ff8a740 1 -- 172.21.15.90:0/2274472954 shutdown_connections 2024-04-02T14:50:46.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.309+0000 7f196ff8a740 1 -- 172.21.15.90:0/2274472954 wait complete. 2024-04-02T14:50:46.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.309+0000 7f196ff8a740 1 -- start start 2024-04-02T14:50:46.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.309+0000 7f196ff8a740 1 Processor -- start 2024-04-02T14:50:46.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.310+0000 7f196ff8a740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556c8971c7b0 0x556c8971f860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:46.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.310+0000 7f196ff8a740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556c8971d0d0 0x556c8971fda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:46.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.310+0000 7f196ff8a740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556c89725cb0 0x556c897202e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:46.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.310+0000 7f196ff8a740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f19681b5310 con 0x556c8971d0d0 2024-04-02T14:50:46.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.310+0000 7f196ff8a740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f19681b4c40 con 0x556c8971c7b0 2024-04-02T14:50:46.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.310+0000 7f196ff8a740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f19681b1ea0 con 0x556c89725cb0 2024-04-02T14:50:46.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.310+0000 7f196ff8a740 10 client.0 _ll_register_callbacks cb 0x7f1950009790 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:50:46.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.310+0000 7f195d400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556c8971d0d0 0x556c8971fda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:46.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.310+0000 7f195d400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556c8971d0d0 0x556c8971fda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:37754/0 (socket says 172.21.15.90:37754) 2024-04-02T14:50:46.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.310+0000 7f195d400640 1 -- 172.21.15.90:0/1545862554 learned_addr learned my addr 172.21.15.90:0/1545862554 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:50:46.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.310+0000 7f1966800640 1 --2- 172.21.15.90:0/1545862554 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556c89725cb0 0x556c897202e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:46.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.310+0000 7f1965e00640 1 --2- 172.21.15.90:0/1545862554 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556c8971c7b0 0x556c8971f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:46.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.310+0000 7f195d400640 1 -- 172.21.15.90:0/1545862554 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556c89725cb0 msgr2=0x556c897202e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:50:46.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.310+0000 7f195d400640 1 --2- 172.21.15.90:0/1545862554 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556c89725cb0 0x556c897202e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:46.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.310+0000 7f195d400640 1 -- 172.21.15.90:0/1545862554 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556c8971c7b0 msgr2=0x556c8971f860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:50:46.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.310+0000 7f195d400640 1 --2- 172.21.15.90:0/1545862554 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556c8971c7b0 0x556c8971f860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:46.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.310+0000 7f195d400640 1 -- 172.21.15.90:0/1545862554 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556c8979b150 con 0x556c8971d0d0 2024-04-02T14:50:46.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.311+0000 7f195d400640 1 --2- 172.21.15.90:0/1545862554 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556c8971d0d0 0x556c8971fda0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f195009adf0 tx=0x7f19500ce3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5c6369a394085a server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:50:46.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.311+0000 7f194be00640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-02T14:50:46.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.311+0000 7f194be00640 1 -- 172.21.15.90:0/1545862554 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f195009c680 con 0x556c8971d0d0 2024-04-02T14:50:46.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.311+0000 7f194be00640 1 -- 172.21.15.90:0/1545862554 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 8550+0+0 (secure 0 0 0) 0x7f19500a5520 con 0x556c8971d0d0 2024-04-02T14:50:46.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.311+0000 7f196ff8a740 1 -- 172.21.15.90:0/1545862554 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f195009d240 con 0x556c8971d0d0 2024-04-02T14:50:46.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.311+0000 7f194be00640 1 -- 172.21.15.90:0/1545862554 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f195009c910 con 0x556c8971d0d0 2024-04-02T14:50:46.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.311+0000 7f196ff8a740 10 client.18996 fetch_fsmap learned FSMap version 27 2024-04-02T14:50:46.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.311+0000 7f196ff8a740 1 -- 172.21.15.90:0/1545862554 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x556c8979c360 con 0x556c8971d0d0 2024-04-02T14:50:46.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.311+0000 7f194be00640 1 -- 172.21.15.90:0/1545862554 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f195009d240 con 0x556c8971d0d0 2024-04-02T14:50:46.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.312+0000 7f194be00640 1 -- 172.21.15.90:0/1545862554 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f195009bc00 con 0x556c8971d0d0 2024-04-02T14:50:46.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.312+0000 7f196ff8a740 10 client.18996 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:50:46.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.312+0000 7f196ff8a740 10 client.18996 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:50:46.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.312+0000 7f196ff8a740 20 client.18996 populate_metadata read hostname 'smithi090' 2024-04-02T14:50:46.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.312+0000 7f196ff8a740 1 --2- 172.21.15.90:0/1545862554 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x556c8979c650 0x556c8979ea70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:46.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.312+0000 7f196ff8a740 4 client.18996 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-02T14:50:46.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.312+0000 7f196ff8a740 1 -- 172.21.15.90:0/1545862554 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "scrub status"}) -- 0x556c8968dcb0 con 0x556c8979c650 2024-04-02T14:50:46.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.312+0000 7f1965e00640 1 --2- 172.21.15.90:0/1545862554 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x556c8979c650 0x556c8979ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:46.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.313+0000 7f1965e00640 1 --2- 172.21.15.90:0/1545862554 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x556c8979c650 0x556c8979ea70 crc :-1 s=READY pgs=397 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=755140b0fd9906bf server_cookie=e288b1f2ee964219 in_seq=0 out_seq=0 2024-04-02T14:50:46.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.313+0000 7f194be00640 10 client.18996 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:50:46.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.313+0000 7f194be00640 1 -- 172.21.15.90:0/1545862554 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x556c8968dcb0 con 0x556c8979c650 2024-04-02T14:50:46.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.313+0000 7f194be00640 10 client.18996 handle_command_reply: tid=0 2024-04-02T14:50:46.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.313+0000 7f196ff8a740 1 client.18996 shutdown 2024-04-02T14:50:46.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.313+0000 7f196ff8a740 1 -- 172.21.15.90:0/1545862554 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556c8971d0d0 msgr2=0x556c8971fda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:46.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.313+0000 7f196ff8a740 1 --2- 172.21.15.90:0/1545862554 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556c8971d0d0 0x556c8971fda0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f195009adf0 tx=0x7f19500ce3f0 comp rx=0 tx=0).stop 2024-04-02T14:50:46.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.313+0000 7f196ff8a740 1 -- 172.21.15.90:0/1545862554 shutdown_connections 2024-04-02T14:50:46.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.313+0000 7f196ff8a740 1 --2- 172.21.15.90:0/1545862554 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x556c8979c650 0x556c8979ea70 crc :-1 s=READY pgs=397 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:46.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.313+0000 7f196ff8a740 1 --2- 172.21.15.90:0/1545862554 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556c89725cb0 0x556c897202e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:46.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.313+0000 7f196ff8a740 1 --2- 172.21.15.90:0/1545862554 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x556c8971d0d0 0x556c8971fda0 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:46.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.313+0000 7f196ff8a740 1 --2- 172.21.15.90:0/1545862554 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x556c8971c7b0 0x556c8971f860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:46.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.313+0000 7f194be00640 0 client.18996 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:50:46.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.313+0000 7f196ff8a740 1 -- 172.21.15.90:0/1545862554 >> 172.21.15.90:0/1545862554 conn(0x556c89716ee0 msgr2=0x556c89722c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:50:46.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.313+0000 7f196ff8a740 1 -- 172.21.15.90:0/1545862554 shutdown_connections 2024-04-02T14:50:46.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.313+0000 7f196ff8a740 1 -- 172.21.15.90:0/1545862554 wait complete. 2024-04-02T14:50:46.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.313+0000 7f196ff8a740 20 client.18996 trim_cache size 0 max 16384 2024-04-02T14:50:46.315 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-02T14:50:46.316 INFO:teuthology.orchestra.run.smithi090.stdout: "status": "no active scrubs running", 2024-04-02T14:50:46.316 INFO:teuthology.orchestra.run.smithi090.stdout: "scrubs": {} 2024-04-02T14:50:46.316 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-02T14:50:46.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.314+0000 7f196d200640 1 -- 172.21.15.90:0/2016864006 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f194c0a3da0 msgr2=0x7f194c0a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:46.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.314+0000 7f196d200640 1 --2- 172.21.15.90:0/2016864006 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f194c0a3da0 0x7f194c0a6260 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f1960009e80 tx=0x7f1960005f90 comp rx=0 tx=0).stop 2024-04-02T14:50:46.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.314+0000 7f196d200640 1 -- 172.21.15.90:0/2016864006 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f196810d3e0 msgr2=0x7f19681a4bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:46.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.314+0000 7f196d200640 1 --2- 172.21.15.90:0/2016864006 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f196810d3e0 0x7f19681a4bb0 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7f195000e9c0 tx=0x7f195000ee80 comp rx=0 tx=0).stop 2024-04-02T14:50:46.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.314+0000 7f196d200640 1 -- 172.21.15.90:0/2016864006 shutdown_connections 2024-04-02T14:50:46.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.314+0000 7f196d200640 1 --2- 172.21.15.90:0/2016864006 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f194c0a3da0 0x7f194c0a6260 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:46.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.314+0000 7f196d200640 1 --2- 172.21.15.90:0/2016864006 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f196819e7e0 0x7f196819ec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:46.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.314+0000 7f196d200640 1 --2- 172.21.15.90:0/2016864006 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f196810d3e0 0x7f19681a4bb0 unknown :-1 s=CLOSED pgs=1038 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:46.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.314+0000 7f196d200640 1 --2- 172.21.15.90:0/2016864006 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1968108310 0x7f19681a4670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:46.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.314+0000 7f196d200640 1 -- 172.21.15.90:0/2016864006 >> 172.21.15.90:0/2016864006 conn(0x7f1968076040 msgr2=0x7f19680783a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:50:46.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.314+0000 7f196d200640 1 -- 172.21.15.90:0/2016864006 shutdown_connections 2024-04-02T14:50:46.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.314+0000 7f196d200640 1 -- 172.21.15.90:0/2016864006 wait complete. 2024-04-02T14:50:46.325 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-02T14:50:46.325 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T14:50:46.404 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:50:46 smithi090 ceph-mon[31674]: pgmap v497: 97 pgs: 97 active+clean; 4.3 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 16 MiB/s wr, 1.59k op/s 2024-04-02T14:50:46.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.546+0000 7ff48ca00640 1 -- 172.21.15.90:0/1503603529 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff488108ff0 msgr2=0x7ff488110bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:46.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.546+0000 7ff48ca00640 1 --2- 172.21.15.90:0/1503603529 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff488108ff0 0x7ff488110bc0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7ff480009ca0 tx=0x7ff48002c100 comp rx=0 tx=0).stop 2024-04-02T14:50:46.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.546+0000 7ff48ca00640 1 -- 172.21.15.90:0/1503603529 shutdown_connections 2024-04-02T14:50:46.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.546+0000 7ff48ca00640 1 --2- 172.21.15.90:0/1503603529 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff488108ff0 0x7ff488110bc0 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:46.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.546+0000 7ff48ca00640 1 --2- 172.21.15.90:0/1503603529 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff4881048d0 0x7ff488108910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:46.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.546+0000 7ff48ca00640 1 --2- 172.21.15.90:0/1503603529 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff488103f20 0x7ff488104300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:46.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.546+0000 7ff48ca00640 1 -- 172.21.15.90:0/1503603529 >> 172.21.15.90:0/1503603529 conn(0x7ff4880ff780 msgr2=0x7ff488101ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:50:46.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.546+0000 7ff48ca00640 1 -- 172.21.15.90:0/1503603529 shutdown_connections 2024-04-02T14:50:46.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.547+0000 7ff48ca00640 1 -- 172.21.15.90:0/1503603529 wait complete. 2024-04-02T14:50:46.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.547+0000 7ff48ca00640 1 Processor -- start 2024-04-02T14:50:46.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.548+0000 7ff48ca00640 1 -- start start 2024-04-02T14:50:46.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.548+0000 7ff48ca00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff488103f20 0x7ff48819a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:46.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.548+0000 7ff48ca00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff488108ff0 0x7ff48819ab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:46.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.548+0000 7ff48ca00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff48819eea0 0x7ff48819f350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:46.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.548+0000 7ff48ca00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7ff4881140a0 con 0x7ff48819eea0 2024-04-02T14:50:46.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.548+0000 7ff48ca00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7ff488113f20 con 0x7ff488103f20 2024-04-02T14:50:46.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.548+0000 7ff48ca00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7ff488114220 con 0x7ff488108ff0 2024-04-02T14:50:46.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.548+0000 7ff485e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff488103f20 0x7ff48819a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:46.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.548+0000 7ff485e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff488103f20 0x7ff48819a5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:40448/0 (socket says 172.21.15.90:40448) 2024-04-02T14:50:46.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.548+0000 7ff485e00640 1 -- 172.21.15.90:0/3065706362 learned_addr learned my addr 172.21.15.90:0/3065706362 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:50:46.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.548+0000 7ff486800640 1 --2- 172.21.15.90:0/3065706362 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff48819eea0 0x7ff48819f350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:46.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.548+0000 7ff485400640 1 --2- 172.21.15.90:0/3065706362 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff488108ff0 0x7ff48819ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:46.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.549+0000 7ff485e00640 1 -- 172.21.15.90:0/3065706362 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff488108ff0 msgr2=0x7ff48819ab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:46.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.549+0000 7ff485e00640 1 --2- 172.21.15.90:0/3065706362 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff488108ff0 0x7ff48819ab20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:46.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.549+0000 7ff485e00640 1 -- 172.21.15.90:0/3065706362 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff48819eea0 msgr2=0x7ff48819f350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:46.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.549+0000 7ff485e00640 1 --2- 172.21.15.90:0/3065706362 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff48819eea0 0x7ff48819f350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:46.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.549+0000 7ff485e00640 1 -- 172.21.15.90:0/3065706362 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff48819f990 con 0x7ff488103f20 2024-04-02T14:50:46.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.549+0000 7ff485e00640 1 --2- 172.21.15.90:0/3065706362 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff488103f20 0x7ff48819a5e0 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7ff47800e9f0 tx=0x7ff47800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:50:46.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.549+0000 7ff47ea00640 1 -- 172.21.15.90:0/3065706362 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff47800cd80 con 0x7ff488103f20 2024-04-02T14:50:46.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.549+0000 7ff48ca00640 1 -- 172.21.15.90:0/3065706362 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff48819fc80 con 0x7ff488103f20 2024-04-02T14:50:46.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.549+0000 7ff47ea00640 1 -- 172.21.15.90:0/3065706362 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff478010910 con 0x7ff488103f20 2024-04-02T14:50:46.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.549+0000 7ff47ea00640 1 -- 172.21.15.90:0/3065706362 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4780188f0 con 0x7ff488103f20 2024-04-02T14:50:46.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.549+0000 7ff48ca00640 1 -- 172.21.15.90:0/3065706362 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4881a7470 con 0x7ff488103f20 2024-04-02T14:50:46.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.550+0000 7ff48ca00640 1 -- 172.21.15.90:0/3065706362 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff488068d40 con 0x7ff488103f20 2024-04-02T14:50:46.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.552+0000 7ff47ea00640 1 -- 172.21.15.90:0/3065706362 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7ff478010ac0 con 0x7ff488103f20 2024-04-02T14:50:46.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.553+0000 7ff47ea00640 1 --2- 172.21.15.90:0/3065706362 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7ff46c0a3d20 0x7ff46c0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:46.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.554+0000 7ff47ea00640 1 -- 172.21.15.90:0/3065706362 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 8550+0+0 (secure 0 0 0) 0x7ff4780d0c20 con 0x7ff488103f20 2024-04-02T14:50:46.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.554+0000 7ff485400640 1 --2- 172.21.15.90:0/3065706362 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7ff46c0a3d20 0x7ff46c0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:46.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.554+0000 7ff485400640 1 --2- 172.21.15.90:0/3065706362 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7ff46c0a3d20 0x7ff46c0a61e0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7ff488108970 tx=0x7ff470005f90 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:50:46.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.558+0000 7ff47ea00640 1 -- 172.21.15.90:0/3065706362 <== mon.2 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7ff47809cbf0 con 0x7ff488103f20 2024-04-02T14:50:46.602 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:50:46 smithi092 ceph-mon[36497]: pgmap v497: 97 pgs: 97 active+clean; 4.3 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 16 MiB/s wr, 1.59k op/s 2024-04-02T14:50:46.678 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:50:46 smithi177 ceph-mon[36431]: pgmap v497: 97 pgs: 97 active+clean; 4.3 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 16 MiB/s wr, 1.59k op/s 2024-04-02T14:50:46.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.810+0000 7ff48ca00640 1 -- 172.21.15.90:0/3065706362 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff48810c2e0 con 0x7ff488103f20 2024-04-02T14:50:46.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.812+0000 7ff47ea00640 1 -- 172.21.15.90:0/3065706362 <== mon.2 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11927 (secure 0 0 0) 0x7ff4780a0ef0 con 0x7ff488103f20 2024-04-02T14:50:46.805 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-02T14:50:46.805 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14412,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6837/2852160309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6836","nonce":2852160309},{"type":"v1","addr":"172.21.15.90:6837","nonce":2852160309}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14478,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6841/1919376858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6840","nonce":1919376858},{"type":"v1","addr":"172.21.15.90:6841","nonce":1919376858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-02T14:32:22.319083+0000","modified":"2024-04-02T14:34:52.266152+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24590,"mds_1":24560,"mds_2":24547,"mds_3":24539,"mds_4":24523},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24454":{"gid":24454,"name":"l","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6833/1564319431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":1564319431},{"type":"v1","addr":"172.21.15.177:6833","nonce":1564319431}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.92:6833/1570337372","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":1570337372},{"type":"v1","addr":"172.21.15.92:6833","nonce":1570337372}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"j","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.90:6835/1277124891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6834","nonce":1277124891},{"type":"v1","addr":"172.21.15.90:6835","nonce":1277124891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.92:6835/860721164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":860721164},{"type":"v1","addr":"172.21.15.92:6835","nonce":860721164}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24511":{"gid":24511,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6835/3177244988","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":3177244988},{"type":"v1","addr":"172.21.15.177:6835","nonce":3177244988}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"c","rank":4,"incarnation":25,"state":"up:active","state_seq":9,"addr":"172.21.15.177:6837/4223414858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":4223414858},{"type":"v1","addr":"172.21.15.177:6837","nonce":4223414858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24539":{"gid":24539,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":8,"addr":"172.21.15.92:6837/1270635939","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":1270635939},{"type":"v1","addr":"172.21.15.92:6837","nonce":1270635939}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24547":{"gid":24547,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":7,"addr":"172.21.15.177:6839/664895663","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":664895663},{"type":"v1","addr":"172.21.15.177:6839","nonce":664895663}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"g","rank":1,"incarnation":19,"state":"up:active","state_seq":5,"addr":"172.21.15.90:6839/1498915870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6838","nonce":1498915870},{"type":"v1","addr":"172.21.15.90:6839","nonce":1498915870}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24590":{"gid":24590,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.92:6839/1054888219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":1054888219},{"type":"v1","addr":"172.21.15.92:6839","nonce":1054888219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24590,"qdb_cluster":[24547,24590,24523,24539,24560]},"id":1}]} 2024-04-02T14:50:46.805 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 27 2024-04-02T14:50:46.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.814+0000 7ff48ca00640 1 -- 172.21.15.90:0/3065706362 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7ff46c0a3d20 msgr2=0x7ff46c0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:46.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.814+0000 7ff48ca00640 1 --2- 172.21.15.90:0/3065706362 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7ff46c0a3d20 0x7ff46c0a61e0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7ff488108970 tx=0x7ff470005f90 comp rx=0 tx=0).stop 2024-04-02T14:50:46.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.814+0000 7ff48ca00640 1 -- 172.21.15.90:0/3065706362 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff488103f20 msgr2=0x7ff48819a5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:46.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.814+0000 7ff48ca00640 1 --2- 172.21.15.90:0/3065706362 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff488103f20 0x7ff48819a5e0 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7ff47800e9f0 tx=0x7ff47800eeb0 comp rx=0 tx=0).stop 2024-04-02T14:50:46.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.814+0000 7ff48ca00640 1 -- 172.21.15.90:0/3065706362 shutdown_connections 2024-04-02T14:50:46.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.814+0000 7ff48ca00640 1 --2- 172.21.15.90:0/3065706362 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7ff46c0a3d20 0x7ff46c0a61e0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:46.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.814+0000 7ff48ca00640 1 --2- 172.21.15.90:0/3065706362 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff48819eea0 0x7ff48819f350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:46.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.814+0000 7ff48ca00640 1 --2- 172.21.15.90:0/3065706362 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff488108ff0 0x7ff48819ab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:46.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.814+0000 7ff48ca00640 1 --2- 172.21.15.90:0/3065706362 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff488103f20 0x7ff48819a5e0 unknown :-1 s=CLOSED pgs=1040 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:46.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.814+0000 7ff48ca00640 1 -- 172.21.15.90:0/3065706362 >> 172.21.15.90:0/3065706362 conn(0x7ff4880ff780 msgr2=0x7ff488100280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:50:46.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.814+0000 7ff48ca00640 1 -- 172.21.15.90:0/3065706362 shutdown_connections 2024-04-02T14:50:46.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:46.814+0000 7ff48ca00640 1 -- 172.21.15.90:0/3065706362 wait complete. 2024-04-02T14:50:46.826 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-04-02T14:50:47.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.046+0000 7f4d25200640 1 -- 172.21.15.90:0/3330663314 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4d20108fe0 msgr2=0x7f4d20110bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:47.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.046+0000 7f4d25200640 1 --2- 172.21.15.90:0/3330663314 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4d20108fe0 0x7f4d20110bb0 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f4d18009900 tx=0x7f4d1802bc40 comp rx=0 tx=0).stop 2024-04-02T14:50:47.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.046+0000 7f4d25200640 1 -- 172.21.15.90:0/3330663314 shutdown_connections 2024-04-02T14:50:47.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.046+0000 7f4d25200640 1 --2- 172.21.15.90:0/3330663314 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4d20108fe0 0x7f4d20110bb0 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:47.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.046+0000 7f4d25200640 1 --2- 172.21.15.90:0/3330663314 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4d201048c0 0x7f4d20108900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:47.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.046+0000 7f4d25200640 1 --2- 172.21.15.90:0/3330663314 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4d20103f10 0x7f4d201042f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:47.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.046+0000 7f4d25200640 1 -- 172.21.15.90:0/3330663314 >> 172.21.15.90:0/3330663314 conn(0x7f4d200ff750 msgr2=0x7f4d20101b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:50:47.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.047+0000 7f4d25200640 1 -- 172.21.15.90:0/3330663314 shutdown_connections 2024-04-02T14:50:47.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.047+0000 7f4d25200640 1 -- 172.21.15.90:0/3330663314 wait complete. 2024-04-02T14:50:47.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.048+0000 7f4d25200640 1 Processor -- start 2024-04-02T14:50:47.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.048+0000 7f4d25200640 1 -- start start 2024-04-02T14:50:47.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.048+0000 7f4d25200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4d20103f10 0x7f4d2019a610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:47.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.048+0000 7f4d25200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4d20108fe0 0x7f4d2019ab50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:47.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.048+0000 7f4d25200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4d2019ee80 0x7f4d2019f330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:47.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.048+0000 7f4d25200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f4d20114090 con 0x7f4d20108fe0 2024-04-02T14:50:47.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.048+0000 7f4d25200640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f4d20113f10 con 0x7f4d2019ee80 2024-04-02T14:50:47.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.048+0000 7f4d25200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f4d20114210 con 0x7f4d20103f10 2024-04-02T14:50:47.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.048+0000 7f4d15400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4d20108fe0 0x7f4d2019ab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:47.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.049+0000 7f4d15400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4d20108fe0 0x7f4d2019ab50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:37782/0 (socket says 172.21.15.90:37782) 2024-04-02T14:50:47.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.049+0000 7f4d15400640 1 -- 172.21.15.90:0/973464103 learned_addr learned my addr 172.21.15.90:0/973464103 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:50:47.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.049+0000 7f4d1de00640 1 --2- 172.21.15.90:0/973464103 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4d20103f10 0x7f4d2019a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:47.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.049+0000 7f4d1e800640 1 --2- 172.21.15.90:0/973464103 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4d2019ee80 0x7f4d2019f330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:47.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.049+0000 7f4d15400640 1 -- 172.21.15.90:0/973464103 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4d20103f10 msgr2=0x7f4d2019a610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:47.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.049+0000 7f4d15400640 1 --2- 172.21.15.90:0/973464103 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4d20103f10 0x7f4d2019a610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:47.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.049+0000 7f4d15400640 1 -- 172.21.15.90:0/973464103 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4d2019ee80 msgr2=0x7f4d2019f330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:47.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.049+0000 7f4d15400640 1 --2- 172.21.15.90:0/973464103 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4d2019ee80 0x7f4d2019f330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:47.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.049+0000 7f4d15400640 1 -- 172.21.15.90:0/973464103 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d2019f970 con 0x7f4d20108fe0 2024-04-02T14:50:47.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.049+0000 7f4d15400640 1 --2- 172.21.15.90:0/973464103 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4d20108fe0 0x7f4d2019ab50 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f4d0800e9a0 tx=0x7f4d0800ee60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:50:47.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.049+0000 7f4d17400640 1 -- 172.21.15.90:0/973464103 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d0800cd80 con 0x7f4d20108fe0 2024-04-02T14:50:47.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.050+0000 7f4d25200640 1 -- 172.21.15.90:0/973464103 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d2019fc60 con 0x7f4d20108fe0 2024-04-02T14:50:47.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.050+0000 7f4d17400640 1 -- 172.21.15.90:0/973464103 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4d080035f0 con 0x7f4d20108fe0 2024-04-02T14:50:47.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.050+0000 7f4d17400640 1 -- 172.21.15.90:0/973464103 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d08010670 con 0x7f4d20108fe0 2024-04-02T14:50:47.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.050+0000 7f4d25200640 1 -- 172.21.15.90:0/973464103 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d201a7450 con 0x7f4d20108fe0 2024-04-02T14:50:47.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.053+0000 7f4d17400640 1 -- 172.21.15.90:0/973464103 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f4d080037a0 con 0x7f4d20108fe0 2024-04-02T14:50:47.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.053+0000 7f4d27e67740 1 Processor -- start 2024-04-02T14:50:47.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.053+0000 7f4d27e67740 1 -- start start 2024-04-02T14:50:47.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.054+0000 7f4d27e67740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e5f2d969f0 0x55e5f2d95740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:47.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.054+0000 7f4d27e67740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e5f2d95d10 0x55e5f2d96190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:47.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.054+0000 7f4d27e67740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e5f2d9f080 0x55e5f2da1470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:47.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.054+0000 7f4d27e67740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55e5f2cd6400 con 0x55e5f2d95d10 2024-04-02T14:50:47.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.054+0000 7f4d27e67740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55e5f2cbaf10 con 0x55e5f2d969f0 2024-04-02T14:50:47.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.054+0000 7f4d27e67740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55e5f2cfa6f0 con 0x55e5f2d9f080 2024-04-02T14:50:47.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.054+0000 7f4d1de00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e5f2d95d10 0x55e5f2d96190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:47.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.054+0000 7f4d1de00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e5f2d95d10 0x55e5f2d96190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:37784/0 (socket says 172.21.15.90:37784) 2024-04-02T14:50:47.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.054+0000 7f4d1de00640 1 -- 172.21.15.90:0/1608754541 learned_addr learned my addr 172.21.15.90:0/1608754541 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:50:47.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.054+0000 7f4d1e800640 1 --2- 172.21.15.90:0/1608754541 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e5f2d969f0 0x55e5f2d95740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:47.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.054+0000 7f4d15400640 1 --2- 172.21.15.90:0/1608754541 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e5f2d9f080 0x55e5f2da1470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:47.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.054+0000 7f4d1de00640 1 -- 172.21.15.90:0/1608754541 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e5f2d9f080 msgr2=0x55e5f2da1470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:50:47.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.054+0000 7f4d1de00640 1 --2- 172.21.15.90:0/1608754541 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e5f2d9f080 0x55e5f2da1470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:47.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.054+0000 7f4d1de00640 1 -- 172.21.15.90:0/1608754541 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e5f2d969f0 msgr2=0x55e5f2d95740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:50:47.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.054+0000 7f4d1de00640 1 --2- 172.21.15.90:0/1608754541 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e5f2d969f0 0x55e5f2d95740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:47.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.054+0000 7f4d1de00640 1 -- 172.21.15.90:0/1608754541 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e5f2a0e3f0 con 0x55e5f2d95d10 2024-04-02T14:50:47.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.055+0000 7f4d17400640 1 --2- 172.21.15.90:0/973464103 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f4d040a3d20 0x7f4d040a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:47.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.055+0000 7f4d1de00640 1 --2- 172.21.15.90:0/1608754541 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e5f2d95d10 0x55e5f2d96190 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f4d100114f0 tx=0x7f4d100119b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d78a40798cf83d33 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:50:47.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.055+0000 7f4d17400640 1 -- 172.21.15.90:0/973464103 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 8550+0+0 (secure 0 0 0) 0x7f4d080d0a00 con 0x7f4d20108fe0 2024-04-02T14:50:47.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.055+0000 7f4d1e800640 1 --2- 172.21.15.90:0/973464103 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f4d040a3d20 0x7f4d040a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:47.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.055+0000 7f4d0f400640 1 -- 172.21.15.90:0/1608754541 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d10004d20 con 0x55e5f2d95d10 2024-04-02T14:50:47.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.055+0000 7f4d0f400640 1 -- 172.21.15.90:0/1608754541 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4d10004050 con 0x55e5f2d95d10 2024-04-02T14:50:47.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.055+0000 7f4d0f400640 1 -- 172.21.15.90:0/1608754541 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d1000f760 con 0x55e5f2d95d10 2024-04-02T14:50:47.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.056+0000 7f4d27e67740 1 -- 172.21.15.90:0/1608754541 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e5f2d95d10 msgr2=0x55e5f2d96190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:47.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.056+0000 7f4d1e800640 1 --2- 172.21.15.90:0/973464103 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f4d040a3d20 0x7f4d040a61e0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f4d18005ab0 tx=0x7f4d1802e040 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:50:47.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.056+0000 7f4d27e67740 1 --2- 172.21.15.90:0/1608754541 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e5f2d95d10 0x55e5f2d96190 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f4d100114f0 tx=0x7f4d100119b0 comp rx=0 tx=0).stop 2024-04-02T14:50:47.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.056+0000 7f4d27e67740 1 -- 172.21.15.90:0/1608754541 shutdown_connections 2024-04-02T14:50:47.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.056+0000 7f4d27e67740 1 --2- 172.21.15.90:0/1608754541 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e5f2d9f080 0x55e5f2da1470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:47.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.056+0000 7f4d27e67740 1 --2- 172.21.15.90:0/1608754541 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e5f2d95d10 0x55e5f2d96190 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:47.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.056+0000 7f4d27e67740 1 --2- 172.21.15.90:0/1608754541 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e5f2d969f0 0x55e5f2d95740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:47.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.056+0000 7f4d27e67740 1 -- 172.21.15.90:0/1608754541 >> 172.21.15.90:0/1608754541 conn(0x55e5f2d5ca50 msgr2=0x55e5f2d90760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:50:47.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.056+0000 7f4d27e67740 1 -- 172.21.15.90:0/1608754541 shutdown_connections 2024-04-02T14:50:47.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.056+0000 7f4d27e67740 1 -- 172.21.15.90:0/1608754541 wait complete. 2024-04-02T14:50:47.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.057+0000 7f4d27e67740 1 -- start start 2024-04-02T14:50:47.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.057+0000 7f4d27e67740 1 Processor -- start 2024-04-02T14:50:47.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.057+0000 7f4d27e67740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e5f2d95d10 0x55e5f2d9c010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:47.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.057+0000 7f4d27e67740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e5f2d969f0 0x55e5f2d9c550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:47.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.057+0000 7f4d27e67740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e5f2d9f080 0x55e5f2d9ca90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:47.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.057+0000 7f4d27e67740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55e5f2cf66a0 con 0x55e5f2d95d10 2024-04-02T14:50:47.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.057+0000 7f4d27e67740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55e5f2ca2980 con 0x55e5f2d969f0 2024-04-02T14:50:47.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.057+0000 7f4d27e67740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55e5f2cb5cb0 con 0x55e5f2d9f080 2024-04-02T14:50:47.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.057+0000 7f4d27e67740 10 client.0 _ll_register_callbacks cb 0x55e5f2cb56a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:50:47.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.058+0000 7f4d1de00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e5f2d95d10 0x55e5f2d9c010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:47.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.058+0000 7f4d1de00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e5f2d95d10 0x55e5f2d9c010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:37798/0 (socket says 172.21.15.90:37798) 2024-04-02T14:50:47.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.058+0000 7f4d1de00640 1 -- 172.21.15.90:0/3664795274 learned_addr learned my addr 172.21.15.90:0/3664795274 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:50:47.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.058+0000 7f4d15400640 1 --2- 172.21.15.90:0/3664795274 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e5f2d969f0 0x55e5f2d9c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:47.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.058+0000 7f4d1e800640 1 --2- 172.21.15.90:0/3664795274 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e5f2d9f080 0x55e5f2d9ca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:47.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.058+0000 7f4d1de00640 1 -- 172.21.15.90:0/3664795274 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e5f2d9f080 msgr2=0x55e5f2d9ca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:50:47.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.058+0000 7f4d1de00640 1 --2- 172.21.15.90:0/3664795274 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e5f2d9f080 0x55e5f2d9ca90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:47.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.058+0000 7f4d1de00640 1 -- 172.21.15.90:0/3664795274 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e5f2d969f0 msgr2=0x55e5f2d9c550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:50:47.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.058+0000 7f4d1de00640 1 --2- 172.21.15.90:0/3664795274 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e5f2d969f0 0x55e5f2d9c550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:47.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.058+0000 7f4d1de00640 1 -- 172.21.15.90:0/3664795274 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e5f2ad87d0 con 0x55e5f2d95d10 2024-04-02T14:50:47.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.058+0000 7f4d1de00640 1 --2- 172.21.15.90:0/3664795274 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e5f2d95d10 0x55e5f2d9c010 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f4d10011eb0 tx=0x7f4d10004cf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e716e4cc03b64625 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:50:47.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.058+0000 7f4d03e00640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-02T14:50:47.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.059+0000 7f4d03e00640 1 -- 172.21.15.90:0/3664795274 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d10028020 con 0x55e5f2d95d10 2024-04-02T14:50:47.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.059+0000 7f4d03e00640 1 -- 172.21.15.90:0/3664795274 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 8550+0+0 (secure 0 0 0) 0x7f4d10004410 con 0x55e5f2d95d10 2024-04-02T14:50:47.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.059+0000 7f4d27e67740 1 -- 172.21.15.90:0/3664795274 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4d10004050 con 0x55e5f2d95d10 2024-04-02T14:50:47.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.059+0000 7f4d03e00640 1 -- 172.21.15.90:0/3664795274 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f4d10033050 con 0x55e5f2d95d10 2024-04-02T14:50:47.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.059+0000 7f4d27e67740 10 client.19014 fetch_fsmap learned FSMap version 27 2024-04-02T14:50:47.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.059+0000 7f4d27e67740 1 -- 172.21.15.90:0/3664795274 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55e5f2dadeb0 con 0x55e5f2d95d10 2024-04-02T14:50:47.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.059+0000 7f4d03e00640 1 -- 172.21.15.90:0/3664795274 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4d10004050 con 0x55e5f2d95d10 2024-04-02T14:50:47.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.059+0000 7f4d03e00640 1 -- 172.21.15.90:0/3664795274 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f4d10005040 con 0x55e5f2d95d10 2024-04-02T14:50:47.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.060+0000 7f4d27e67740 10 client.19014 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:50:47.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.060+0000 7f4d27e67740 10 client.19014 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.c 2024-04-02T14:50:47.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.060+0000 7f4d27e67740 20 client.19014 populate_metadata read hostname 'smithi090' 2024-04-02T14:50:47.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.060+0000 7f4d27e67740 1 --2- 172.21.15.90:0/3664795274 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x55e5f2daf260 0x55e5f2db1650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:47.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.060+0000 7f4d27e67740 4 client.19014 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:50:47.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.060+0000 7f4d27e67740 1 -- 172.21.15.90:0/3664795274 --> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e5f2cba040 con 0x55e5f2daf260 2024-04-02T14:50:47.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.060+0000 7f4d15400640 1 --2- 172.21.15.90:0/3664795274 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x55e5f2daf260 0x55e5f2db1650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:47.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.061+0000 7f4d15400640 1 --2- 172.21.15.90:0/3664795274 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x55e5f2daf260 0x55e5f2db1650 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=caef8b013d6311f0 server_cookie=96f4050a8121419 in_seq=0 out_seq=0 2024-04-02T14:50:47.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.061+0000 7f4d03e00640 10 client.19014 ms_handle_connect on v2:172.21.15.177:6836/4223414858 2024-04-02T14:50:47.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.061+0000 7f4d03e00640 1 -- 172.21.15.90:0/3664795274 <== mds.4 v2:172.21.15.177:6836/4223414858 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e5f2cba040 con 0x55e5f2daf260 2024-04-02T14:50:47.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.061+0000 7f4d03e00640 10 client.19014 handle_command_reply: tid=0 2024-04-02T14:50:47.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.061+0000 7f4d27e67740 1 client.19014 shutdown 2024-04-02T14:50:47.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.062+0000 7f4d27e67740 1 -- 172.21.15.90:0/3664795274 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e5f2d95d10 msgr2=0x55e5f2d9c010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:47.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.062+0000 7f4d27e67740 1 --2- 172.21.15.90:0/3664795274 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e5f2d95d10 0x55e5f2d9c010 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f4d10011eb0 tx=0x7f4d10004cf0 comp rx=0 tx=0).stop 2024-04-02T14:50:47.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.062+0000 7f4d27e67740 1 -- 172.21.15.90:0/3664795274 shutdown_connections 2024-04-02T14:50:47.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.062+0000 7f4d27e67740 1 --2- 172.21.15.90:0/3664795274 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e5f2d9f080 0x55e5f2d9ca90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:47.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.062+0000 7f4d27e67740 1 --2- 172.21.15.90:0/3664795274 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e5f2d969f0 0x55e5f2d9c550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:47.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.062+0000 7f4d27e67740 1 --2- 172.21.15.90:0/3664795274 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x55e5f2daf260 0x55e5f2db1650 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:47.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.062+0000 7f4d27e67740 1 --2- 172.21.15.90:0/3664795274 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e5f2d95d10 0x55e5f2d9c010 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:47.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.062+0000 7f4d03e00640 0 client.19014 ms_handle_reset on v2:172.21.15.177:6836/4223414858 2024-04-02T14:50:47.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.062+0000 7f4d27e67740 1 -- 172.21.15.90:0/3664795274 >> 172.21.15.90:0/3664795274 conn(0x55e5f2d5ca50 msgr2=0x55e5f2d924b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:50:47.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.062+0000 7f4d27e67740 1 -- 172.21.15.90:0/3664795274 shutdown_connections 2024-04-02T14:50:47.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.062+0000 7f4d27e67740 1 -- 172.21.15.90:0/3664795274 wait complete. 2024-04-02T14:50:47.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.062+0000 7f4d27e67740 20 client.19014 trim_cache size 0 max 16384 2024-04-02T14:50:47.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.076+0000 7f4d27e67740 1 Processor -- start 2024-04-02T14:50:47.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.077+0000 7f4d27e67740 1 -- start start 2024-04-02T14:50:47.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.077+0000 7f4d27e67740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e5f2d95d10 0x55e5f2d905c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:47.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.077+0000 7f4d27e67740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e5f2d969f0 0x55e5f2daaaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:47.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.077+0000 7f4d27e67740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e5f2d9f080 0x55e5f2d8fae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:47.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.077+0000 7f4d27e67740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f4d201afb50 con 0x55e5f2d95d10 2024-04-02T14:50:47.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.077+0000 7f4d27e67740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f4d201b1690 con 0x55e5f2d969f0 2024-04-02T14:50:47.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.077+0000 7f4d27e67740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f4d201b0220 con 0x55e5f2d9f080 2024-04-02T14:50:47.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.077+0000 7f4d1de00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e5f2d95d10 0x55e5f2d905c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:47.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.077+0000 7f4d1de00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e5f2d95d10 0x55e5f2d905c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:37802/0 (socket says 172.21.15.90:37802) 2024-04-02T14:50:47.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.077+0000 7f4d1de00640 1 -- 172.21.15.90:0/1502264388 learned_addr learned my addr 172.21.15.90:0/1502264388 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:50:47.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.077+0000 7f4d1e800640 1 --2- 172.21.15.90:0/1502264388 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e5f2d9f080 0x55e5f2d8fae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:47.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.077+0000 7f4d15400640 1 --2- 172.21.15.90:0/1502264388 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e5f2d969f0 0x55e5f2daaaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:47.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.077+0000 7f4d1de00640 1 -- 172.21.15.90:0/1502264388 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e5f2d9f080 msgr2=0x55e5f2d8fae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:50:47.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.077+0000 7f4d1de00640 1 --2- 172.21.15.90:0/1502264388 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e5f2d9f080 0x55e5f2d8fae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:47.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.077+0000 7f4d1de00640 1 -- 172.21.15.90:0/1502264388 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e5f2d969f0 msgr2=0x55e5f2daaaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:50:47.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.078+0000 7f4d1de00640 1 --2- 172.21.15.90:0/1502264388 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e5f2d969f0 0x55e5f2daaaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:47.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.078+0000 7f4d1de00640 1 -- 172.21.15.90:0/1502264388 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e5f2d9cf70 con 0x55e5f2d95d10 2024-04-02T14:50:47.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.078+0000 7f4d1e800640 1 --2- 172.21.15.90:0/1502264388 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e5f2d9f080 0x55e5f2d8fae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:50:47.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.078+0000 7f4d15400640 1 --2- 172.21.15.90:0/1502264388 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e5f2d969f0 0x55e5f2daaaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:50:47.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.078+0000 7f4d1de00640 1 --2- 172.21.15.90:0/1502264388 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e5f2d95d10 0x55e5f2d905c0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f4d10012fd0 tx=0x7f4d10009d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=840ac8162f836b58 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:50:47.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.078+0000 7f4d0ea00640 1 -- 172.21.15.90:0/1502264388 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d10028070 con 0x55e5f2d95d10 2024-04-02T14:50:47.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.078+0000 7f4d0ea00640 1 -- 172.21.15.90:0/1502264388 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4d100056d0 con 0x55e5f2d95d10 2024-04-02T14:50:47.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.078+0000 7f4d0ea00640 1 -- 172.21.15.90:0/1502264388 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d1000f560 con 0x55e5f2d95d10 2024-04-02T14:50:47.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.079+0000 7f4d27e67740 1 -- 172.21.15.90:0/1502264388 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e5f2d95d10 msgr2=0x55e5f2d905c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:47.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.079+0000 7f4d27e67740 1 --2- 172.21.15.90:0/1502264388 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e5f2d95d10 0x55e5f2d905c0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f4d10012fd0 tx=0x7f4d10009d40 comp rx=0 tx=0).stop 2024-04-02T14:50:47.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.079+0000 7f4d27e67740 1 -- 172.21.15.90:0/1502264388 shutdown_connections 2024-04-02T14:50:47.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.079+0000 7f4d27e67740 1 --2- 172.21.15.90:0/1502264388 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e5f2d9f080 0x55e5f2d8fae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:47.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.079+0000 7f4d27e67740 1 --2- 172.21.15.90:0/1502264388 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e5f2d969f0 0x55e5f2daaaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:47.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.079+0000 7f4d27e67740 1 --2- 172.21.15.90:0/1502264388 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e5f2d95d10 0x55e5f2d905c0 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:47.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.079+0000 7f4d27e67740 1 -- 172.21.15.90:0/1502264388 >> 172.21.15.90:0/1502264388 conn(0x55e5f2d5ca50 msgr2=0x55e5f2d9c860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:50:47.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.079+0000 7f4d27e67740 1 -- 172.21.15.90:0/1502264388 shutdown_connections 2024-04-02T14:50:47.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.079+0000 7f4d27e67740 1 -- 172.21.15.90:0/1502264388 wait complete. 2024-04-02T14:50:47.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.079+0000 7f4d27e67740 1 -- start start 2024-04-02T14:50:47.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.080+0000 7f4d27e67740 1 Processor -- start 2024-04-02T14:50:47.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.080+0000 7f4d27e67740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e5f2d95d10 0x55e5f2da7180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:47.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.080+0000 7f4d27e67740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e5f2d969f0 0x55e5f2da2540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:47.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.080+0000 7f4d27e67740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e5f2d9f080 0x55e5f2da2a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:47.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.080+0000 7f4d27e67740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f4d201b0fc0 con 0x55e5f2d95d10 2024-04-02T14:50:47.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.080+0000 7f4d27e67740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f4d201b08f0 con 0x55e5f2d969f0 2024-04-02T14:50:47.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.080+0000 7f4d27e67740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f4d201adb50 con 0x55e5f2d9f080 2024-04-02T14:50:47.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.080+0000 7f4d27e67740 10 client.0 _ll_register_callbacks cb 0x7f4d10003e10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:50:47.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.080+0000 7f4d1de00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e5f2d95d10 0x55e5f2da7180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:47.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.080+0000 7f4d1de00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e5f2d95d10 0x55e5f2da7180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:37810/0 (socket says 172.21.15.90:37810) 2024-04-02T14:50:47.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.080+0000 7f4d1de00640 1 -- 172.21.15.90:0/726686784 learned_addr learned my addr 172.21.15.90:0/726686784 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:50:47.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.080+0000 7f4d15400640 1 --2- 172.21.15.90:0/726686784 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e5f2d969f0 0x55e5f2da2540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:47.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.080+0000 7f4d1e800640 1 --2- 172.21.15.90:0/726686784 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e5f2d9f080 0x55e5f2da2a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:47.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.081+0000 7f4d15400640 1 -- 172.21.15.90:0/726686784 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e5f2d9f080 msgr2=0x55e5f2da2a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:50:47.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.081+0000 7f4d15400640 1 --2- 172.21.15.90:0/726686784 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e5f2d9f080 0x55e5f2da2a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:47.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.081+0000 7f4d15400640 1 -- 172.21.15.90:0/726686784 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e5f2d95d10 msgr2=0x55e5f2da7180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:50:47.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.081+0000 7f4d15400640 1 --2- 172.21.15.90:0/726686784 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e5f2d95d10 0x55e5f2da7180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:47.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.081+0000 7f4d15400640 1 -- 172.21.15.90:0/726686784 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e5f2da3040 con 0x55e5f2d969f0 2024-04-02T14:50:47.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.081+0000 7f4d1de00640 1 --2- 172.21.15.90:0/726686784 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e5f2d95d10 0x55e5f2da7180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:50:47.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.081+0000 7f4d15400640 1 --2- 172.21.15.90:0/726686784 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e5f2d969f0 0x55e5f2da2540 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7f4d080ce130 tx=0x7f4d08000bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=de1afd2a835d6219 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:50:47.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.081+0000 7f4d03e00640 10 client.0 ms_handle_connect on v2:172.21.15.92:3300/0 2024-04-02T14:50:47.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.081+0000 7f4d03e00640 1 -- 172.21.15.90:0/726686784 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d080d7070 con 0x55e5f2d969f0 2024-04-02T14:50:47.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.081+0000 7f4d27e67740 1 -- 172.21.15.90:0/726686784 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4d100056d0 con 0x55e5f2d969f0 2024-04-02T14:50:47.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.081+0000 7f4d03e00640 1 -- 172.21.15.90:0/726686784 <== mon.2 v2:172.21.15.92:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 8550+0+0 (secure 0 0 0) 0x7f4d080ce3b0 con 0x55e5f2d969f0 2024-04-02T14:50:47.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.081+0000 7f4d03e00640 1 -- 172.21.15.90:0/726686784 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f4d0809c980 con 0x55e5f2d969f0 2024-04-02T14:50:47.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.081+0000 7f4d27e67740 10 client.30347 fetch_fsmap learned FSMap version 27 2024-04-02T14:50:47.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.081+0000 7f4d27e67740 1 -- 172.21.15.90:0/726686784 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55e5f2e1dd10 con 0x55e5f2d969f0 2024-04-02T14:50:47.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.082+0000 7f4d03e00640 1 -- 172.21.15.90:0/726686784 <== mon.2 v2:172.21.15.92:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4d100056d0 con 0x55e5f2d969f0 2024-04-02T14:50:47.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.082+0000 7f4d03e00640 1 -- 172.21.15.90:0/726686784 <== mon.2 v2:172.21.15.92:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f4d080870b0 con 0x55e5f2d969f0 2024-04-02T14:50:47.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.082+0000 7f4d27e67740 10 client.30347 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:50:47.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.082+0000 7f4d27e67740 10 client.30347 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.c 2024-04-02T14:50:47.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.082+0000 7f4d27e67740 20 client.30347 populate_metadata read hostname 'smithi090' 2024-04-02T14:50:47.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.082+0000 7f4d27e67740 1 --2- 172.21.15.90:0/726686784 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x55e5f2e1e3e0 0x55e5f2e207d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:47.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.082+0000 7f4d27e67740 4 client.30347 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-02T14:50:47.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.082+0000 7f4d27e67740 1 -- 172.21.15.90:0/726686784 --> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55e5f2d07c30 con 0x55e5f2e1e3e0 2024-04-02T14:50:47.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.082+0000 7f4d1de00640 1 --2- 172.21.15.90:0/726686784 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x55e5f2e1e3e0 0x55e5f2e207d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:47.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.083+0000 7f4d1de00640 1 --2- 172.21.15.90:0/726686784 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x55e5f2e1e3e0 0x55e5f2e207d0 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=ae5478de9e478bd3 server_cookie=df8beaaa378a8a94 in_seq=0 out_seq=0 2024-04-02T14:50:47.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.083+0000 7f4d03e00640 10 client.30347 ms_handle_connect on v2:172.21.15.177:6836/4223414858 2024-04-02T14:50:47.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.083+0000 7f4d03e00640 1 -- 172.21.15.90:0/726686784 <== mds.4 v2:172.21.15.177:6836/4223414858 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55e5f2d07c30 con 0x55e5f2e1e3e0 2024-04-02T14:50:47.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.083+0000 7f4d03e00640 10 client.30347 handle_command_reply: tid=0 2024-04-02T14:50:47.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.083+0000 7f4d27e67740 1 client.30347 shutdown 2024-04-02T14:50:47.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.083+0000 7f4d27e67740 1 -- 172.21.15.90:0/726686784 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e5f2d969f0 msgr2=0x55e5f2da2540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:47.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.083+0000 7f4d27e67740 1 --2- 172.21.15.90:0/726686784 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e5f2d969f0 0x55e5f2da2540 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7f4d080ce130 tx=0x7f4d08000bc0 comp rx=0 tx=0).stop 2024-04-02T14:50:47.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.084+0000 7f4d27e67740 1 -- 172.21.15.90:0/726686784 shutdown_connections 2024-04-02T14:50:47.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.084+0000 7f4d27e67740 1 --2- 172.21.15.90:0/726686784 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e5f2d9f080 0x55e5f2da2a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:47.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.084+0000 7f4d27e67740 1 --2- 172.21.15.90:0/726686784 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e5f2d969f0 0x55e5f2da2540 unknown :-1 s=CLOSED pgs=1041 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:47.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.084+0000 7f4d27e67740 1 --2- 172.21.15.90:0/726686784 >> [v2:172.21.15.177:6836/4223414858,v1:172.21.15.177:6837/4223414858] conn(0x55e5f2e1e3e0 0x55e5f2e207d0 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:47.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.084+0000 7f4d27e67740 1 --2- 172.21.15.90:0/726686784 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55e5f2d95d10 0x55e5f2da7180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:47.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.084+0000 7f4d03e00640 0 client.30347 ms_handle_reset on v2:172.21.15.177:6836/4223414858 2024-04-02T14:50:47.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.084+0000 7f4d27e67740 1 -- 172.21.15.90:0/726686784 >> 172.21.15.90:0/726686784 conn(0x55e5f2d5ca50 msgr2=0x55e5f2da9ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:50:47.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.084+0000 7f4d27e67740 1 -- 172.21.15.90:0/726686784 shutdown_connections 2024-04-02T14:50:47.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.084+0000 7f4d27e67740 1 -- 172.21.15.90:0/726686784 wait complete. 2024-04-02T14:50:47.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.084+0000 7f4d27e67740 20 client.30347 trim_cache size 0 max 16384 2024-04-02T14:50:47.081 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-02T14:50:47.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.084+0000 7f4d25200640 1 -- 172.21.15.90:0/973464103 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f4d040a3d20 msgr2=0x7f4d040a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:47.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.084+0000 7f4d25200640 1 --2- 172.21.15.90:0/973464103 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f4d040a3d20 0x7f4d040a61e0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f4d18005ab0 tx=0x7f4d1802e040 comp rx=0 tx=0).stop 2024-04-02T14:50:47.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.084+0000 7f4d25200640 1 -- 172.21.15.90:0/973464103 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4d20108fe0 msgr2=0x7f4d2019ab50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:47.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.084+0000 7f4d25200640 1 --2- 172.21.15.90:0/973464103 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4d20108fe0 0x7f4d2019ab50 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f4d0800e9a0 tx=0x7f4d0800ee60 comp rx=0 tx=0).stop 2024-04-02T14:50:47.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.085+0000 7f4d25200640 1 -- 172.21.15.90:0/973464103 shutdown_connections 2024-04-02T14:50:47.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.085+0000 7f4d25200640 1 --2- 172.21.15.90:0/973464103 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f4d040a3d20 0x7f4d040a61e0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:47.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.085+0000 7f4d25200640 1 --2- 172.21.15.90:0/973464103 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4d2019ee80 0x7f4d2019f330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:47.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.085+0000 7f4d25200640 1 --2- 172.21.15.90:0/973464103 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4d20108fe0 0x7f4d2019ab50 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:47.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.085+0000 7f4d25200640 1 --2- 172.21.15.90:0/973464103 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4d20103f10 0x7f4d2019a610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:47.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.085+0000 7f4d25200640 1 -- 172.21.15.90:0/973464103 >> 172.21.15.90:0/973464103 conn(0x7f4d200ff750 msgr2=0x7f4d20101a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:50:47.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.085+0000 7f4d25200640 1 -- 172.21.15.90:0/973464103 shutdown_connections 2024-04-02T14:50:47.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.085+0000 7f4d25200640 1 -- 172.21.15.90:0/973464103 wait complete. 2024-04-02T14:50:47.095 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-04-02T14:50:47.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.308+0000 7f7ce2200640 1 -- 172.21.15.90:0/3332548786 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7cdc108ec0 msgr2=0x7f7cdc110a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:47.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.308+0000 7f7ce2200640 1 --2- 172.21.15.90:0/3332548786 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7cdc108ec0 0x7f7cdc110a90 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7f7cd4009ca0 tx=0x7f7cd402c0f0 comp rx=0 tx=0).stop 2024-04-02T14:50:47.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.308+0000 7f7ce2200640 1 -- 172.21.15.90:0/3332548786 shutdown_connections 2024-04-02T14:50:47.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.308+0000 7f7ce2200640 1 --2- 172.21.15.90:0/3332548786 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7cdc108ec0 0x7f7cdc110a90 unknown :-1 s=CLOSED pgs=1042 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:47.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.308+0000 7f7ce2200640 1 --2- 172.21.15.90:0/3332548786 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7cdc1047a0 0x7f7cdc1087e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:47.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.308+0000 7f7ce2200640 1 --2- 172.21.15.90:0/3332548786 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7cdc103df0 0x7f7cdc1041d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:47.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.308+0000 7f7ce2200640 1 -- 172.21.15.90:0/3332548786 >> 172.21.15.90:0/3332548786 conn(0x7f7cdc0ff630 msgr2=0x7f7cdc101a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:50:47.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.308+0000 7f7ce2200640 1 -- 172.21.15.90:0/3332548786 shutdown_connections 2024-04-02T14:50:47.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.308+0000 7f7ce2200640 1 -- 172.21.15.90:0/3332548786 wait complete. 2024-04-02T14:50:47.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.309+0000 7f7ce2200640 1 Processor -- start 2024-04-02T14:50:47.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.310+0000 7f7ce2200640 1 -- start start 2024-04-02T14:50:47.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.310+0000 7f7ce2200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7cdc103df0 0x7f7cdc19a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:47.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.310+0000 7f7ce2200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7cdc108ec0 0x7f7cdc19a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:47.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.310+0000 7f7ce2200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7cdc19ed20 0x7f7cdc19f1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:47.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.310+0000 7f7ce2200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f7cdc113f70 con 0x7f7cdc103df0 2024-04-02T14:50:47.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.310+0000 7f7ce2200640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f7cdc113df0 con 0x7f7cdc108ec0 2024-04-02T14:50:47.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.310+0000 7f7ce2200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f7cdc1140f0 con 0x7f7cdc19ed20 2024-04-02T14:50:47.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.310+0000 7f7ce0e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7cdc103df0 0x7f7cdc19a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:47.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.310+0000 7f7cdbe00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7cdc108ec0 0x7f7cdc19a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:47.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.310+0000 7f7cdbe00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7cdc108ec0 0x7f7cdc19a9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:40512/0 (socket says 172.21.15.90:40512) 2024-04-02T14:50:47.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.310+0000 7f7ce0e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7cdc103df0 0x7f7cdc19a4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:37834/0 (socket says 172.21.15.90:37834) 2024-04-02T14:50:47.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.310+0000 7f7cdbe00640 1 -- 172.21.15.90:0/2139667720 learned_addr learned my addr 172.21.15.90:0/2139667720 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:50:47.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.310+0000 7f7ce1800640 1 --2- 172.21.15.90:0/2139667720 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7cdc19ed20 0x7f7cdc19f1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:47.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.311+0000 7f7cdbe00640 1 -- 172.21.15.90:0/2139667720 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7cdc19ed20 msgr2=0x7f7cdc19f1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:47.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.311+0000 7f7cdbe00640 1 --2- 172.21.15.90:0/2139667720 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7cdc19ed20 0x7f7cdc19f1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:47.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.311+0000 7f7cdbe00640 1 -- 172.21.15.90:0/2139667720 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7cdc103df0 msgr2=0x7f7cdc19a4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:47.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.311+0000 7f7cdbe00640 1 --2- 172.21.15.90:0/2139667720 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7cdc103df0 0x7f7cdc19a4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:47.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.311+0000 7f7cdbe00640 1 -- 172.21.15.90:0/2139667720 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7cdc19f810 con 0x7f7cdc108ec0 2024-04-02T14:50:47.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.311+0000 7f7cdbe00640 1 --2- 172.21.15.90:0/2139667720 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7cdc108ec0 0x7f7cdc19a9f0 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7f7cc800e9c0 tx=0x7f7cc800ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:50:47.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.311+0000 7f7cd9600640 1 -- 172.21.15.90:0/2139667720 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cc800cd80 con 0x7f7cdc108ec0 2024-04-02T14:50:47.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.311+0000 7f7ce2200640 1 -- 172.21.15.90:0/2139667720 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7cdc19fb00 con 0x7f7cdc108ec0 2024-04-02T14:50:47.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.311+0000 7f7cd9600640 1 -- 172.21.15.90:0/2139667720 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7cc8005500 con 0x7f7cdc108ec0 2024-04-02T14:50:47.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.311+0000 7f7cd9600640 1 -- 172.21.15.90:0/2139667720 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cc8010690 con 0x7f7cdc108ec0 2024-04-02T14:50:47.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.311+0000 7f7ce2200640 1 -- 172.21.15.90:0/2139667720 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7cdc1a7270 con 0x7f7cdc108ec0 2024-04-02T14:50:47.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.315+0000 7f7ce65da740 1 Processor -- start 2024-04-02T14:50:47.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.316+0000 7f7ce65da740 1 -- start start 2024-04-02T14:50:47.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.316+0000 7f7ce65da740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555fac74dba0 0x555fac74c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:47.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.316+0000 7f7ce65da740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555fac74cec0 0x555fac74d340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:47.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.316+0000 7f7ce65da740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555fac756230 0x555fac758620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:47.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.316+0000 7f7ce65da740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x555fac68d400 con 0x555fac74dba0 2024-04-02T14:50:47.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.316+0000 7f7ce65da740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x555fac671f10 con 0x555fac756230 2024-04-02T14:50:47.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.316+0000 7f7ce65da740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x555fac6b16f0 con 0x555fac74cec0 2024-04-02T14:50:47.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.316+0000 7f7ce1800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555fac74dba0 0x555fac74c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:47.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.316+0000 7f7ce1800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555fac74dba0 0x555fac74c8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:37842/0 (socket says 172.21.15.90:37842) 2024-04-02T14:50:47.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.316+0000 7f7ce1800640 1 -- 172.21.15.90:0/364849789 learned_addr learned my addr 172.21.15.90:0/364849789 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:50:47.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.316+0000 7f7ce0e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555fac74cec0 0x555fac74d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:47.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.317+0000 7f7cd9600640 1 -- 172.21.15.90:0/2139667720 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f7cc80046c0 con 0x7f7cdc108ec0 2024-04-02T14:50:47.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.317+0000 7f7ce1800640 1 -- 172.21.15.90:0/364849789 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555fac74cec0 msgr2=0x555fac74d340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:50:47.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.317+0000 7f7ce1800640 1 --2- 172.21.15.90:0/364849789 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555fac74cec0 0x555fac74d340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:47.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.317+0000 7f7ce1800640 1 -- 172.21.15.90:0/364849789 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555fac756230 msgr2=0x555fac758620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:50:47.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.317+0000 7f7ce1800640 1 --2- 172.21.15.90:0/364849789 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555fac756230 0x555fac758620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:47.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.317+0000 7f7ce1800640 1 -- 172.21.15.90:0/364849789 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555fac3c53f0 con 0x555fac74dba0 2024-04-02T14:50:47.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.317+0000 7f7ce0e00640 1 --2- 172.21.15.90:0/364849789 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555fac74cec0 0x555fac74d340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:50:47.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.317+0000 7f7ce1800640 1 --2- 172.21.15.90:0/364849789 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555fac74dba0 0x555fac74c8f0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f7cd402c700 tx=0x7f7cd4005f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f93ded473c55fef7 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:50:47.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.317+0000 7f7cd2000640 1 -- 172.21.15.90:0/364849789 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cd4043070 con 0x555fac74dba0 2024-04-02T14:50:47.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.318+0000 7f7cd2000640 1 -- 172.21.15.90:0/364849789 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7cd402cca0 con 0x555fac74dba0 2024-04-02T14:50:47.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.318+0000 7f7cd2000640 1 -- 172.21.15.90:0/364849789 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cd4039040 con 0x555fac74dba0 2024-04-02T14:50:47.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.318+0000 7f7cd9600640 1 --2- 172.21.15.90:0/2139667720 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f7cc00a3da0 0x7f7cc00a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:47.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.318+0000 7f7ce65da740 1 -- 172.21.15.90:0/364849789 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555fac74dba0 msgr2=0x555fac74c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:47.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.318+0000 7f7ce65da740 1 --2- 172.21.15.90:0/364849789 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555fac74dba0 0x555fac74c8f0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f7cd402c700 tx=0x7f7cd4005f80 comp rx=0 tx=0).stop 2024-04-02T14:50:47.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.318+0000 7f7cd9600640 1 -- 172.21.15.90:0/2139667720 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 8550+0+0 (secure 0 0 0) 0x7f7cc80d0a90 con 0x7f7cdc108ec0 2024-04-02T14:50:47.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.318+0000 7f7ce0e00640 1 --2- 172.21.15.90:0/2139667720 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f7cc00a3da0 0x7f7cc00a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:47.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.318+0000 7f7ce65da740 1 -- 172.21.15.90:0/364849789 shutdown_connections 2024-04-02T14:50:47.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.318+0000 7f7ce65da740 1 --2- 172.21.15.90:0/364849789 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555fac756230 0x555fac758620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:47.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.318+0000 7f7ce65da740 1 --2- 172.21.15.90:0/364849789 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555fac74cec0 0x555fac74d340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:47.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.318+0000 7f7ce65da740 1 --2- 172.21.15.90:0/364849789 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555fac74dba0 0x555fac74c8f0 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:47.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.318+0000 7f7ce65da740 1 -- 172.21.15.90:0/364849789 >> 172.21.15.90:0/364849789 conn(0x555fac713a50 msgr2=0x555fac747910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:50:47.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.319+0000 7f7ce65da740 1 -- 172.21.15.90:0/364849789 shutdown_connections 2024-04-02T14:50:47.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.319+0000 7f7ce65da740 1 -- 172.21.15.90:0/364849789 wait complete. 2024-04-02T14:50:47.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.319+0000 7f7ce65da740 1 -- start start 2024-04-02T14:50:47.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.319+0000 7f7ce0e00640 1 --2- 172.21.15.90:0/2139667720 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f7cc00a3da0 0x7f7cc00a6260 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x555fac74e0f0 tx=0x7f7ccc005f90 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:50:47.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.319+0000 7f7ce65da740 1 Processor -- start 2024-04-02T14:50:47.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.320+0000 7f7ce65da740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555fac74cec0 0x555fac753210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:47.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.320+0000 7f7ce65da740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555fac74dba0 0x555fac753750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:47.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.320+0000 7f7ce65da740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555fac756230 0x555fac753c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:47.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.320+0000 7f7ce65da740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x555fac6ad6a0 con 0x555fac74dba0 2024-04-02T14:50:47.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.320+0000 7f7ce65da740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x555fac659980 con 0x555fac756230 2024-04-02T14:50:47.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.320+0000 7f7ce65da740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x555fac66ccb0 con 0x555fac74cec0 2024-04-02T14:50:47.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.320+0000 7f7ce65da740 10 client.0 _ll_register_callbacks cb 0x555fac66c6a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:50:47.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.320+0000 7f7cdbe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555fac74dba0 0x555fac753750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:47.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.320+0000 7f7cdbe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555fac74dba0 0x555fac753750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:37846/0 (socket says 172.21.15.90:37846) 2024-04-02T14:50:47.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.320+0000 7f7cdbe00640 1 -- 172.21.15.90:0/1175244146 learned_addr learned my addr 172.21.15.90:0/1175244146 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:50:47.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.320+0000 7f7ce0e00640 1 --2- 172.21.15.90:0/1175244146 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555fac74cec0 0x555fac753210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:47.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.320+0000 7f7ce1800640 1 --2- 172.21.15.90:0/1175244146 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555fac756230 0x555fac753c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:47.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.320+0000 7f7cdbe00640 1 -- 172.21.15.90:0/1175244146 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555fac74cec0 msgr2=0x555fac753210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:50:47.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.320+0000 7f7cdbe00640 1 --2- 172.21.15.90:0/1175244146 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555fac74cec0 0x555fac753210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:47.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.320+0000 7f7cdbe00640 1 -- 172.21.15.90:0/1175244146 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555fac756230 msgr2=0x555fac753c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:50:47.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.320+0000 7f7cdbe00640 1 --2- 172.21.15.90:0/1175244146 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555fac756230 0x555fac753c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:47.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.320+0000 7f7cdbe00640 1 -- 172.21.15.90:0/1175244146 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555fac48f7d0 con 0x555fac74dba0 2024-04-02T14:50:47.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.321+0000 7f7ce1800640 1 --2- 172.21.15.90:0/1175244146 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555fac756230 0x555fac753c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:50:47.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.321+0000 7f7cdbe00640 1 --2- 172.21.15.90:0/1175244146 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555fac74dba0 0x555fac753750 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f7cc80a5150 tx=0x7f7cc809c610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6497e8334fbce72b server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:50:47.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.321+0000 7f7cc6a00640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-02T14:50:47.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.321+0000 7f7cc6a00640 1 -- 172.21.15.90:0/1175244146 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cc80bbdd0 con 0x555fac74dba0 2024-04-02T14:50:47.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.321+0000 7f7cc6a00640 1 -- 172.21.15.90:0/1175244146 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 8550+0+0 (secure 0 0 0) 0x7f7cc80a51f0 con 0x555fac74dba0 2024-04-02T14:50:47.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.321+0000 7f7ce65da740 1 -- 172.21.15.90:0/1175244146 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7cd402cca0 con 0x555fac74dba0 2024-04-02T14:50:47.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.321+0000 7f7cc6a00640 1 -- 172.21.15.90:0/1175244146 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f7cc809ca10 con 0x555fac74dba0 2024-04-02T14:50:47.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.321+0000 7f7ce65da740 10 client.19032 fetch_fsmap learned FSMap version 27 2024-04-02T14:50:47.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.321+0000 7f7ce65da740 1 -- 172.21.15.90:0/1175244146 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x555fac7650f0 con 0x555fac74dba0 2024-04-02T14:50:47.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.322+0000 7f7cc6a00640 1 -- 172.21.15.90:0/1175244146 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7cd402cca0 con 0x555fac74dba0 2024-04-02T14:50:47.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.322+0000 7f7cc6a00640 1 -- 172.21.15.90:0/1175244146 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f7cc809be60 con 0x555fac74dba0 2024-04-02T14:50:47.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.322+0000 7f7ce65da740 10 client.19032 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:50:47.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.322+0000 7f7ce65da740 10 client.19032 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-04-02T14:50:47.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.322+0000 7f7ce65da740 20 client.19032 populate_metadata read hostname 'smithi090' 2024-04-02T14:50:47.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.322+0000 7f7ce65da740 1 --2- 172.21.15.90:0/1175244146 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x555fac7665b0 0x555fac7689a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:47.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.322+0000 7f7ce65da740 4 client.19032 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:50:47.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.322+0000 7f7ce65da740 1 -- 172.21.15.90:0/1175244146 --> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555fac671040 con 0x555fac7665b0 2024-04-02T14:50:47.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.322+0000 7f7ce0e00640 1 --2- 172.21.15.90:0/1175244146 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x555fac7665b0 0x555fac7689a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:47.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.323+0000 7f7ce0e00640 1 --2- 172.21.15.90:0/1175244146 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x555fac7665b0 0x555fac7689a0 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=29d7e1b310e684f6 server_cookie=28b0eb3ccb84a206 in_seq=0 out_seq=0 2024-04-02T14:50:47.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.323+0000 7f7cc6a00640 10 client.19032 ms_handle_connect on v2:172.21.15.92:6836/1270635939 2024-04-02T14:50:47.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.324+0000 7f7cc6a00640 1 -- 172.21.15.90:0/1175244146 <== mds.3 v2:172.21.15.92:6836/1270635939 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x555fac671040 con 0x555fac7665b0 2024-04-02T14:50:47.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.324+0000 7f7cc6a00640 10 client.19032 handle_command_reply: tid=0 2024-04-02T14:50:47.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.324+0000 7f7ce65da740 1 client.19032 shutdown 2024-04-02T14:50:47.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.324+0000 7f7ce65da740 1 -- 172.21.15.90:0/1175244146 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555fac74dba0 msgr2=0x555fac753750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:47.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.324+0000 7f7ce65da740 1 --2- 172.21.15.90:0/1175244146 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555fac74dba0 0x555fac753750 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f7cc80a5150 tx=0x7f7cc809c610 comp rx=0 tx=0).stop 2024-04-02T14:50:47.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.324+0000 7f7ce65da740 1 -- 172.21.15.90:0/1175244146 shutdown_connections 2024-04-02T14:50:47.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.324+0000 7f7ce65da740 1 --2- 172.21.15.90:0/1175244146 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x555fac7665b0 0x555fac7689a0 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:47.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.324+0000 7f7ce65da740 1 --2- 172.21.15.90:0/1175244146 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555fac756230 0x555fac753c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:47.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.324+0000 7f7ce65da740 1 --2- 172.21.15.90:0/1175244146 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555fac74dba0 0x555fac753750 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:47.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.324+0000 7f7cc6a00640 0 client.19032 ms_handle_reset on v2:172.21.15.92:6836/1270635939 2024-04-02T14:50:47.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.324+0000 7f7ce65da740 1 --2- 172.21.15.90:0/1175244146 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555fac74cec0 0x555fac753210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:47.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.324+0000 7f7ce65da740 1 -- 172.21.15.90:0/1175244146 >> 172.21.15.90:0/1175244146 conn(0x555fac713a50 msgr2=0x555fac749660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:50:47.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.324+0000 7f7ce65da740 1 -- 172.21.15.90:0/1175244146 shutdown_connections 2024-04-02T14:50:47.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.324+0000 7f7ce65da740 1 -- 172.21.15.90:0/1175244146 wait complete. 2024-04-02T14:50:47.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.324+0000 7f7ce65da740 20 client.19032 trim_cache size 0 max 16384 2024-04-02T14:50:47.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.339+0000 7f7ce65da740 1 Processor -- start 2024-04-02T14:50:47.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.339+0000 7f7ce65da740 1 -- start start 2024-04-02T14:50:47.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.339+0000 7f7ce65da740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555fac74cec0 0x555fac755940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:47.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.339+0000 7f7ce65da740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555fac74dba0 0x555fac761c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:47.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.339+0000 7f7ce65da740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555fac756230 0x555fac7523b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:47.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.339+0000 7f7ce65da740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f7cdc1af7e0 con 0x555fac756230 2024-04-02T14:50:47.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.339+0000 7f7ce65da740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f7cdc1b1320 con 0x555fac74dba0 2024-04-02T14:50:47.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.339+0000 7f7ce65da740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f7cdc1afeb0 con 0x555fac74cec0 2024-04-02T14:50:47.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.340+0000 7f7ce1800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555fac756230 0x555fac7523b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:47.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.340+0000 7f7ce1800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555fac756230 0x555fac7523b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:37858/0 (socket says 172.21.15.90:37858) 2024-04-02T14:50:47.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.340+0000 7f7ce1800640 1 -- 172.21.15.90:0/182420209 learned_addr learned my addr 172.21.15.90:0/182420209 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:50:47.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.340+0000 7f7ce0e00640 1 --2- 172.21.15.90:0/182420209 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555fac74cec0 0x555fac755940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:47.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.340+0000 7f7cdbe00640 1 --2- 172.21.15.90:0/182420209 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555fac74dba0 0x555fac761c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:47.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.340+0000 7f7ce1800640 1 -- 172.21.15.90:0/182420209 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555fac74cec0 msgr2=0x555fac755940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:50:47.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.340+0000 7f7ce1800640 1 --2- 172.21.15.90:0/182420209 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555fac74cec0 0x555fac755940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:47.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.340+0000 7f7ce1800640 1 -- 172.21.15.90:0/182420209 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555fac74dba0 msgr2=0x555fac761c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:50:47.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.340+0000 7f7ce1800640 1 --2- 172.21.15.90:0/182420209 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555fac74dba0 0x555fac761c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:47.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.340+0000 7f7ce1800640 1 -- 172.21.15.90:0/182420209 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555fac74bef0 con 0x555fac756230 2024-04-02T14:50:47.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.340+0000 7f7ce1800640 1 --2- 172.21.15.90:0/182420209 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555fac756230 0x555fac7523b0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f7cd4009bf0 tx=0x7f7cd402c770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8a0867ae33a8ab7 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:50:47.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.341+0000 7f7cd1600640 1 -- 172.21.15.90:0/182420209 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cd404c070 con 0x555fac756230 2024-04-02T14:50:47.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.341+0000 7f7cd1600640 1 -- 172.21.15.90:0/182420209 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7cd4003970 con 0x555fac756230 2024-04-02T14:50:47.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.341+0000 7f7cd1600640 1 -- 172.21.15.90:0/182420209 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cd4039050 con 0x555fac756230 2024-04-02T14:50:47.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.342+0000 7f7ce65da740 1 -- 172.21.15.90:0/182420209 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555fac756230 msgr2=0x555fac7523b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:47.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.342+0000 7f7ce65da740 1 --2- 172.21.15.90:0/182420209 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555fac756230 0x555fac7523b0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f7cd4009bf0 tx=0x7f7cd402c770 comp rx=0 tx=0).stop 2024-04-02T14:50:47.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.342+0000 7f7ce65da740 1 -- 172.21.15.90:0/182420209 shutdown_connections 2024-04-02T14:50:47.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.342+0000 7f7ce65da740 1 --2- 172.21.15.90:0/182420209 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555fac756230 0x555fac7523b0 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:47.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.342+0000 7f7ce65da740 1 --2- 172.21.15.90:0/182420209 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555fac74dba0 0x555fac761c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:47.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.342+0000 7f7ce65da740 1 --2- 172.21.15.90:0/182420209 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555fac74cec0 0x555fac755940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:47.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.342+0000 7f7ce65da740 1 -- 172.21.15.90:0/182420209 >> 172.21.15.90:0/182420209 conn(0x555fac713a50 msgr2=0x555fac748d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:50:47.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.342+0000 7f7ce65da740 1 -- 172.21.15.90:0/182420209 shutdown_connections 2024-04-02T14:50:47.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.342+0000 7f7ce65da740 1 -- 172.21.15.90:0/182420209 wait complete. 2024-04-02T14:50:47.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.342+0000 7f7ce65da740 1 -- start start 2024-04-02T14:50:47.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.342+0000 7f7ce65da740 1 Processor -- start 2024-04-02T14:50:47.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.343+0000 7f7ce65da740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555fac74cec0 0x555fac75e650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:47.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.343+0000 7f7ce65da740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555fac74dba0 0x555fac7504d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:47.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.343+0000 7f7ce65da740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555fac756230 0x555fac750b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:47.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.343+0000 7f7ce65da740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f7cdc1b0c50 con 0x555fac74dba0 2024-04-02T14:50:47.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.343+0000 7f7ce65da740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f7cdc1b0580 con 0x555fac756230 2024-04-02T14:50:47.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.343+0000 7f7ce65da740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f7cdc1ad7e0 con 0x555fac74cec0 2024-04-02T14:50:47.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.343+0000 7f7ce65da740 10 client.0 _ll_register_callbacks cb 0x7f7cc8098d20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:50:47.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.343+0000 7f7cdbe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555fac74dba0 0x555fac7504d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:47.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.343+0000 7f7cdbe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555fac74dba0 0x555fac7504d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:37874/0 (socket says 172.21.15.90:37874) 2024-04-02T14:50:47.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.343+0000 7f7cdbe00640 1 -- 172.21.15.90:0/3236457683 learned_addr learned my addr 172.21.15.90:0/3236457683 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:50:47.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.343+0000 7f7ce1800640 1 --2- 172.21.15.90:0/3236457683 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555fac756230 0x555fac750b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:47.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.343+0000 7f7ce0e00640 1 --2- 172.21.15.90:0/3236457683 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555fac74cec0 0x555fac75e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:47.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.343+0000 7f7cdbe00640 1 -- 172.21.15.90:0/3236457683 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555fac74cec0 msgr2=0x555fac75e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:50:47.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.343+0000 7f7cdbe00640 1 --2- 172.21.15.90:0/3236457683 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555fac74cec0 0x555fac75e650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:47.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.343+0000 7f7cdbe00640 1 -- 172.21.15.90:0/3236457683 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555fac756230 msgr2=0x555fac750b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:50:47.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.343+0000 7f7cdbe00640 1 --2- 172.21.15.90:0/3236457683 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555fac756230 0x555fac750b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:47.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.343+0000 7f7cdbe00640 1 -- 172.21.15.90:0/3236457683 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555fac7512d0 con 0x555fac74dba0 2024-04-02T14:50:47.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.343+0000 7f7ce1800640 1 --2- 172.21.15.90:0/3236457683 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555fac756230 0x555fac750b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:50:47.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.343+0000 7f7ce0e00640 1 --2- 172.21.15.90:0/3236457683 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555fac74cec0 0x555fac75e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:50:47.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.344+0000 7f7cdbe00640 1 --2- 172.21.15.90:0/3236457683 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555fac74dba0 0x555fac7504d0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f7cc800c6a0 tx=0x7f7cc8004b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f814607423683d3 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:50:47.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.344+0000 7f7cc6a00640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-02T14:50:47.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.344+0000 7f7cc6a00640 1 -- 172.21.15.90:0/3236457683 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cc80a53f0 con 0x555fac74dba0 2024-04-02T14:50:47.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.344+0000 7f7cc6a00640 1 -- 172.21.15.90:0/3236457683 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 8550+0+0 (secure 0 0 0) 0x7f7cc80870b0 con 0x555fac74dba0 2024-04-02T14:50:47.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.344+0000 7f7ce65da740 1 -- 172.21.15.90:0/3236457683 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7cd4003970 con 0x555fac74dba0 2024-04-02T14:50:47.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.344+0000 7f7cc6a00640 1 -- 172.21.15.90:0/3236457683 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f7cc80a5680 con 0x555fac74dba0 2024-04-02T14:50:47.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.344+0000 7f7ce65da740 10 client.19038 fetch_fsmap learned FSMap version 27 2024-04-02T14:50:47.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.344+0000 7f7ce65da740 1 -- 172.21.15.90:0/3236457683 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x555fac7ccb50 con 0x555fac74dba0 2024-04-02T14:50:47.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.344+0000 7f7cc6a00640 1 -- 172.21.15.90:0/3236457683 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7cd4003970 con 0x555fac74dba0 2024-04-02T14:50:47.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.345+0000 7f7cc6a00640 1 -- 172.21.15.90:0/3236457683 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f7cc8082050 con 0x555fac74dba0 2024-04-02T14:50:47.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.345+0000 7f7ce65da740 10 client.19038 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:50:47.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.345+0000 7f7ce65da740 10 client.19038 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-04-02T14:50:47.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.345+0000 7f7ce65da740 20 client.19038 populate_metadata read hostname 'smithi090' 2024-04-02T14:50:47.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.345+0000 7f7ce65da740 1 --2- 172.21.15.90:0/3236457683 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x555fac7cd1f0 0x555fac7cf610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:47.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.345+0000 7f7ce65da740 4 client.19038 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-02T14:50:47.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.345+0000 7f7ce65da740 1 -- 172.21.15.90:0/3236457683 --> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] -- command(tid 0: {"prefix": "damage ls"}) -- 0x555fac6bec30 con 0x555fac7cd1f0 2024-04-02T14:50:47.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.345+0000 7f7ce0e00640 1 --2- 172.21.15.90:0/3236457683 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x555fac7cd1f0 0x555fac7cf610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:47.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.345+0000 7f7ce0e00640 1 --2- 172.21.15.90:0/3236457683 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x555fac7cd1f0 0x555fac7cf610 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=d58d2fb47a3c9337 server_cookie=f3ba4d6b497c2fb7 in_seq=0 out_seq=0 2024-04-02T14:50:47.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.345+0000 7f7cc6a00640 10 client.19038 ms_handle_connect on v2:172.21.15.92:6836/1270635939 2024-04-02T14:50:47.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.346+0000 7f7cc6a00640 1 -- 172.21.15.90:0/3236457683 <== mds.3 v2:172.21.15.92:6836/1270635939 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x555fac6bec30 con 0x555fac7cd1f0 2024-04-02T14:50:47.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.346+0000 7f7cc6a00640 10 client.19038 handle_command_reply: tid=0 2024-04-02T14:50:47.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.346+0000 7f7ce65da740 1 client.19038 shutdown 2024-04-02T14:50:47.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.346+0000 7f7ce65da740 1 -- 172.21.15.90:0/3236457683 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555fac74dba0 msgr2=0x555fac7504d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:47.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.346+0000 7f7ce65da740 1 --2- 172.21.15.90:0/3236457683 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555fac74dba0 0x555fac7504d0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f7cc800c6a0 tx=0x7f7cc8004b30 comp rx=0 tx=0).stop 2024-04-02T14:50:47.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.346+0000 7f7ce65da740 1 -- 172.21.15.90:0/3236457683 shutdown_connections 2024-04-02T14:50:47.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.346+0000 7f7ce65da740 1 --2- 172.21.15.90:0/3236457683 >> [v2:172.21.15.92:6836/1270635939,v1:172.21.15.92:6837/1270635939] conn(0x555fac7cd1f0 0x555fac7cf610 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:47.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.346+0000 7f7ce65da740 1 --2- 172.21.15.90:0/3236457683 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x555fac756230 0x555fac750b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:47.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.346+0000 7f7ce65da740 1 --2- 172.21.15.90:0/3236457683 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x555fac74dba0 0x555fac7504d0 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:47.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.346+0000 7f7cc6a00640 0 client.19038 ms_handle_reset on v2:172.21.15.92:6836/1270635939 2024-04-02T14:50:47.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.346+0000 7f7ce65da740 1 --2- 172.21.15.90:0/3236457683 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555fac74cec0 0x555fac75e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:47.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.346+0000 7f7ce65da740 1 -- 172.21.15.90:0/3236457683 >> 172.21.15.90:0/3236457683 conn(0x555fac713a50 msgr2=0x555fac749580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:50:47.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.346+0000 7f7ce65da740 1 -- 172.21.15.90:0/3236457683 shutdown_connections 2024-04-02T14:50:47.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.346+0000 7f7ce65da740 1 -- 172.21.15.90:0/3236457683 wait complete. 2024-04-02T14:50:47.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.346+0000 7f7ce65da740 20 client.19038 trim_cache size 0 max 16384 2024-04-02T14:50:47.344 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-02T14:50:47.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.347+0000 7f7ce2200640 1 -- 172.21.15.90:0/2139667720 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f7cc00a3da0 msgr2=0x7f7cc00a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:47.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.347+0000 7f7ce2200640 1 --2- 172.21.15.90:0/2139667720 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f7cc00a3da0 0x7f7cc00a6260 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x555fac74e0f0 tx=0x7f7ccc005f90 comp rx=0 tx=0).stop 2024-04-02T14:50:47.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.347+0000 7f7ce2200640 1 -- 172.21.15.90:0/2139667720 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7cdc108ec0 msgr2=0x7f7cdc19a9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:47.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.347+0000 7f7ce2200640 1 --2- 172.21.15.90:0/2139667720 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7cdc108ec0 0x7f7cdc19a9f0 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7f7cc800e9c0 tx=0x7f7cc800ee80 comp rx=0 tx=0).stop 2024-04-02T14:50:47.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.347+0000 7f7ce2200640 1 -- 172.21.15.90:0/2139667720 shutdown_connections 2024-04-02T14:50:47.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.347+0000 7f7ce2200640 1 --2- 172.21.15.90:0/2139667720 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f7cc00a3da0 0x7f7cc00a6260 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:47.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.347+0000 7f7ce2200640 1 --2- 172.21.15.90:0/2139667720 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7cdc19ed20 0x7f7cdc19f1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:47.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.347+0000 7f7ce2200640 1 --2- 172.21.15.90:0/2139667720 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7cdc108ec0 0x7f7cdc19a9f0 unknown :-1 s=CLOSED pgs=1043 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:47.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.347+0000 7f7ce2200640 1 --2- 172.21.15.90:0/2139667720 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7cdc103df0 0x7f7cdc19a4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:47.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.347+0000 7f7ce2200640 1 -- 172.21.15.90:0/2139667720 >> 172.21.15.90:0/2139667720 conn(0x7f7cdc0ff630 msgr2=0x7f7cdc101900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:50:47.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.347+0000 7f7ce2200640 1 -- 172.21.15.90:0/2139667720 shutdown_connections 2024-04-02T14:50:47.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.347+0000 7f7ce2200640 1 -- 172.21.15.90:0/2139667720 wait complete. 2024-04-02T14:50:47.355 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:50:47 smithi090 ceph-mon[31674]: from='client.? 172.21.15.90:0/3065706362' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-02T14:50:47.358 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-02T14:50:47.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.570+0000 7f835d400640 1 -- 172.21.15.90:0/232811968 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8358101970 msgr2=0x7f8358101df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:47.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.570+0000 7f835d400640 1 --2- 172.21.15.90:0/232811968 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8358101970 0x7f8358101df0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f8340009930 tx=0x7f834002bc30 comp rx=0 tx=0).stop 2024-04-02T14:50:47.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.570+0000 7f835d400640 1 -- 172.21.15.90:0/232811968 shutdown_connections 2024-04-02T14:50:47.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.570+0000 7f835d400640 1 --2- 172.21.15.90:0/232811968 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8358102330 0x7f835810e840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:47.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.570+0000 7f835d400640 1 --2- 172.21.15.90:0/232811968 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8358101970 0x7f8358101df0 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:47.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.570+0000 7f835d400640 1 --2- 172.21.15.90:0/232811968 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8358107980 0x7f8358107d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:47.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.570+0000 7f835d400640 1 -- 172.21.15.90:0/232811968 >> 172.21.15.90:0/232811968 conn(0x7f83580fd4a0 msgr2=0x7f83580ff8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:50:47.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.570+0000 7f835d400640 1 -- 172.21.15.90:0/232811968 shutdown_connections 2024-04-02T14:50:47.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.570+0000 7f835d400640 1 -- 172.21.15.90:0/232811968 wait complete. 2024-04-02T14:50:47.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.571+0000 7f835d400640 1 Processor -- start 2024-04-02T14:50:47.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.572+0000 7f835d400640 1 -- start start 2024-04-02T14:50:47.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.572+0000 7f835d400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8358101970 0x7f8358198310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:47.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.572+0000 7f835d400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8358102330 0x7f8358198850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:47.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.572+0000 7f835d400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8358107980 0x7f835819cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:47.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.572+0000 7f835d400640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f8358111d90 con 0x7f8358101970 2024-04-02T14:50:47.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.572+0000 7f835d400640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f8358111c10 con 0x7f8358107980 2024-04-02T14:50:47.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.572+0000 7f835d400640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f8358111f10 con 0x7f8358102330 2024-04-02T14:50:47.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.572+0000 7f8357e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8358101970 0x7f8358198310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:47.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.572+0000 7f8357e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8358101970 0x7f8358198310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:37888/0 (socket says 172.21.15.90:37888) 2024-04-02T14:50:47.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.572+0000 7f8357e00640 1 -- 172.21.15.90:0/3249196995 learned_addr learned my addr 172.21.15.90:0/3249196995 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:50:47.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.572+0000 7f835ca00640 1 --2- 172.21.15.90:0/3249196995 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8358107980 0x7f835819cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:47.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.572+0000 7f8357400640 1 --2- 172.21.15.90:0/3249196995 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8358102330 0x7f8358198850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:47.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.573+0000 7f8357e00640 1 -- 172.21.15.90:0/3249196995 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8358102330 msgr2=0x7f8358198850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:47.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.573+0000 7f8357e00640 1 --2- 172.21.15.90:0/3249196995 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8358102330 0x7f8358198850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:47.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.573+0000 7f8357e00640 1 -- 172.21.15.90:0/3249196995 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8358107980 msgr2=0x7f835819cbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:47.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.573+0000 7f8357e00640 1 --2- 172.21.15.90:0/3249196995 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8358107980 0x7f835819cbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:47.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.573+0000 7f8357e00640 1 -- 172.21.15.90:0/3249196995 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f835819d300 con 0x7f8358101970 2024-04-02T14:50:47.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.573+0000 7f8357e00640 1 --2- 172.21.15.90:0/3249196995 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8358101970 0x7f8358198310 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f834800eb90 tx=0x7f834800ef90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:50:47.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.573+0000 7f8354c00640 1 -- 172.21.15.90:0/3249196995 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8348004b20 con 0x7f8358101970 2024-04-02T14:50:47.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.573+0000 7f8354c00640 1 -- 172.21.15.90:0/3249196995 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8348004cc0 con 0x7f8358101970 2024-04-02T14:50:47.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.573+0000 7f835d400640 1 -- 172.21.15.90:0/3249196995 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f835819d5f0 con 0x7f8358101970 2024-04-02T14:50:47.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.573+0000 7f8354c00640 1 -- 172.21.15.90:0/3249196995 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8348010720 con 0x7f8358101970 2024-04-02T14:50:47.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.574+0000 7f835d400640 1 -- 172.21.15.90:0/3249196995 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83581a4e80 con 0x7f8358101970 2024-04-02T14:50:47.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.577+0000 7f83617c9740 1 Processor -- start 2024-04-02T14:50:47.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.577+0000 7f83617c9740 1 -- start start 2024-04-02T14:50:47.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.577+0000 7f8354c00640 1 -- 172.21.15.90:0/3249196995 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f8348005020 con 0x7f8358101970 2024-04-02T14:50:47.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.577+0000 7f83617c9740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a3153b3ba0 0x55a3153b28f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:47.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.577+0000 7f83617c9740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a3153b2ec0 0x55a3153b3340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:47.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.577+0000 7f83617c9740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a3153c0640 0x55a3153c2a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:47.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.577+0000 7f83617c9740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55a3152f3400 con 0x55a3153c0640 2024-04-02T14:50:47.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.577+0000 7f83617c9740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55a3152d7f10 con 0x55a3153b3ba0 2024-04-02T14:50:47.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.577+0000 7f83617c9740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55a3153176f0 con 0x55a3153b2ec0 2024-04-02T14:50:47.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.577+0000 7f835ca00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a3153b3ba0 0x55a3153b28f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:47.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.577+0000 7f8357400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a3153c0640 0x55a3153c2a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:47.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.577+0000 7f835ca00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a3153b3ba0 0x55a3153b28f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:40574/0 (socket says 172.21.15.90:40574) 2024-04-02T14:50:47.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.577+0000 7f835ca00640 1 -- 172.21.15.90:0/312203582 learned_addr learned my addr 172.21.15.90:0/312203582 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:50:47.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.577+0000 7f8357e00640 1 --2- 172.21.15.90:0/312203582 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a3153b2ec0 0x55a3153b3340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:47.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.578+0000 7f835ca00640 1 -- 172.21.15.90:0/312203582 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a3153b2ec0 msgr2=0x55a3153b3340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:50:47.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.578+0000 7f835ca00640 1 --2- 172.21.15.90:0/312203582 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a3153b2ec0 0x55a3153b3340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:47.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.578+0000 7f835ca00640 1 -- 172.21.15.90:0/312203582 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a3153c0640 msgr2=0x55a3153c2a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:50:47.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.578+0000 7f835ca00640 1 --2- 172.21.15.90:0/312203582 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a3153c0640 0x55a3153c2a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:47.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.578+0000 7f835ca00640 1 -- 172.21.15.90:0/312203582 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a31502b3f0 con 0x55a3153b3ba0 2024-04-02T14:50:47.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.578+0000 7f8357e00640 1 --2- 172.21.15.90:0/312203582 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a3153b2ec0 0x55a3153b3340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:50:47.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.578+0000 7f835ca00640 1 --2- 172.21.15.90:0/312203582 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a3153b3ba0 0x55a3153b28f0 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7f8350009c90 tx=0x7f8350011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2912266928e782c7 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:50:47.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.578+0000 7f834d200640 1 -- 172.21.15.90:0/312203582 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8350004230 con 0x55a3153b3ba0 2024-04-02T14:50:47.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.578+0000 7f834d200640 1 -- 172.21.15.90:0/312203582 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8350009e40 con 0x55a3153b3ba0 2024-04-02T14:50:47.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.578+0000 7f834d200640 1 -- 172.21.15.90:0/312203582 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8350019760 con 0x55a3153b3ba0 2024-04-02T14:50:47.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.579+0000 7f8354c00640 1 --2- 172.21.15.90:0/3249196995 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f833c0a3d20 0x7f833c0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:47.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.579+0000 7f8357e00640 1 --2- 172.21.15.90:0/3249196995 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f833c0a3d20 0x7f833c0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:47.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.579+0000 7f8354c00640 1 -- 172.21.15.90:0/3249196995 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 8550+0+0 (secure 0 0 0) 0x7f83480d0de0 con 0x7f8358101970 2024-04-02T14:50:47.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.579+0000 7f83617c9740 1 -- 172.21.15.90:0/312203582 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a3153b3ba0 msgr2=0x55a3153b28f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:47.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.579+0000 7f83617c9740 1 --2- 172.21.15.90:0/312203582 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a3153b3ba0 0x55a3153b28f0 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7f8350009c90 tx=0x7f8350011810 comp rx=0 tx=0).stop 2024-04-02T14:50:47.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.579+0000 7f8357e00640 1 --2- 172.21.15.90:0/3249196995 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f833c0a3d20 0x7f833c0a61e0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x55a3153b40f0 tx=0x7f83480058d0 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:50:47.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.579+0000 7f83617c9740 1 -- 172.21.15.90:0/312203582 shutdown_connections 2024-04-02T14:50:47.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.579+0000 7f83617c9740 1 --2- 172.21.15.90:0/312203582 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a3153c0640 0x55a3153c2a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:47.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.579+0000 7f83617c9740 1 --2- 172.21.15.90:0/312203582 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a3153b2ec0 0x55a3153b3340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:47.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.579+0000 7f83617c9740 1 --2- 172.21.15.90:0/312203582 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a3153b3ba0 0x55a3153b28f0 unknown :-1 s=CLOSED pgs=1044 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:47.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.579+0000 7f83617c9740 1 -- 172.21.15.90:0/312203582 >> 172.21.15.90:0/312203582 conn(0x55a315379a50 msgr2=0x55a3153ad910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:50:47.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.580+0000 7f83617c9740 1 -- 172.21.15.90:0/312203582 shutdown_connections 2024-04-02T14:50:47.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.580+0000 7f83617c9740 1 -- 172.21.15.90:0/312203582 wait complete. 2024-04-02T14:50:47.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.580+0000 7f83617c9740 1 -- start start 2024-04-02T14:50:47.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.580+0000 7f83617c9740 1 Processor -- start 2024-04-02T14:50:47.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.581+0000 7f83617c9740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a3153b2ec0 0x55a3153bd5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:47.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.581+0000 7f83617c9740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a3153b3ba0 0x55a3153bdae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:47.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.581+0000 7f83617c9740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a3153c0640 0x55a3153be020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:47.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.581+0000 7f83617c9740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55a3153136a0 con 0x55a3153c0640 2024-04-02T14:50:47.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.581+0000 7f83617c9740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55a3152bf980 con 0x55a3153b2ec0 2024-04-02T14:50:47.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.581+0000 7f83617c9740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55a3152d2cb0 con 0x55a3153b3ba0 2024-04-02T14:50:47.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.581+0000 7f83617c9740 10 client.0 _ll_register_callbacks cb 0x55a3152d26a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:50:47.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.581+0000 7f835ca00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a3153c0640 0x55a3153be020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:47.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.581+0000 7f835ca00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a3153c0640 0x55a3153be020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:37908/0 (socket says 172.21.15.90:37908) 2024-04-02T14:50:47.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.581+0000 7f835ca00640 1 -- 172.21.15.90:0/3508819806 learned_addr learned my addr 172.21.15.90:0/3508819806 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:50:47.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.581+0000 7f835ca00640 1 --2- 172.21.15.90:0/3508819806 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a3153b2ec0 0x55a3153bd5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:47.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.581+0000 7f8357400640 1 --2- 172.21.15.90:0/3508819806 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a3153b3ba0 0x55a3153bdae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:47.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.581+0000 7f835ca00640 1 -- 172.21.15.90:0/3508819806 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a3153b3ba0 msgr2=0x55a3153bdae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:50:47.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.581+0000 7f835ca00640 1 --2- 172.21.15.90:0/3508819806 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a3153b3ba0 0x55a3153bdae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:47.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.581+0000 7f835ca00640 1 -- 172.21.15.90:0/3508819806 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a3153b2ec0 msgr2=0x55a3153bd5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:50:47.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.581+0000 7f835ca00640 1 --2- 172.21.15.90:0/3508819806 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a3153b2ec0 0x55a3153bd5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:47.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.581+0000 7f835ca00640 1 -- 172.21.15.90:0/3508819806 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a3150f57d0 con 0x55a3153c0640 2024-04-02T14:50:47.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.582+0000 7f835ca00640 1 --2- 172.21.15.90:0/3508819806 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a3153c0640 0x55a3153be020 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f83500045d0 tx=0x7f83500193e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d071f1b30fa1dd0b server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:50:47.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.582+0000 7f8345600640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-02T14:50:47.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.582+0000 7f8345600640 1 -- 172.21.15.90:0/3508819806 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f835002d060 con 0x55a3153c0640 2024-04-02T14:50:47.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.582+0000 7f83617c9740 1 -- 172.21.15.90:0/3508819806 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55a3152d7040 con 0x55a3153c0640 2024-04-02T14:50:47.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.582+0000 7f8345600640 1 -- 172.21.15.90:0/3508819806 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 8550+0+0 (secure 0 0 0) 0x7f8350005110 con 0x55a3153c0640 2024-04-02T14:50:47.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.582+0000 7f8345600640 1 -- 172.21.15.90:0/3508819806 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f8350005310 con 0x55a3153c0640 2024-04-02T14:50:47.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.582+0000 7f83617c9740 10 client.19050 fetch_fsmap learned FSMap version 27 2024-04-02T14:50:47.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.582+0000 7f83617c9740 1 -- 172.21.15.90:0/3508819806 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55a3153cf470 con 0x55a3153c0640 2024-04-02T14:50:47.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.582+0000 7f8345600640 1 -- 172.21.15.90:0/3508819806 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55a3152d7040 con 0x55a3153c0640 2024-04-02T14:50:47.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.583+0000 7f8345600640 1 -- 172.21.15.90:0/3508819806 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f8350018070 con 0x55a3153c0640 2024-04-02T14:50:47.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.583+0000 7f83617c9740 10 client.19050 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:50:47.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.583+0000 7f83617c9740 10 client.19050 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-04-02T14:50:47.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.583+0000 7f83617c9740 20 client.19050 populate_metadata read hostname 'smithi090' 2024-04-02T14:50:47.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.583+0000 7f83617c9740 1 --2- 172.21.15.90:0/3508819806 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x55a3153d0880 0x55a3153d2c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:47.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.583+0000 7f83617c9740 4 client.19050 mds_command: new command op to 24547 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:50:47.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.583+0000 7f83617c9740 1 -- 172.21.15.90:0/3508819806 --> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a315324c30 con 0x55a3153d0880 2024-04-02T14:50:47.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.583+0000 7f8357400640 1 --2- 172.21.15.90:0/3508819806 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x55a3153d0880 0x55a3153d2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:47.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.584+0000 7f8357400640 1 --2- 172.21.15.90:0/3508819806 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x55a3153d0880 0x55a3153d2c70 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=a5affb65f28694a1 server_cookie=cd4553df4ff90027 in_seq=0 out_seq=0 2024-04-02T14:50:47.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.584+0000 7f8345600640 10 client.19050 ms_handle_connect on v2:172.21.15.177:6838/664895663 2024-04-02T14:50:47.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.585+0000 7f8345600640 1 -- 172.21.15.90:0/3508819806 <== mds.2 v2:172.21.15.177:6838/664895663 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a315324c30 con 0x55a3153d0880 2024-04-02T14:50:47.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.585+0000 7f8345600640 10 client.19050 handle_command_reply: tid=0 2024-04-02T14:50:47.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.585+0000 7f83617c9740 1 client.19050 shutdown 2024-04-02T14:50:47.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.585+0000 7f83617c9740 1 -- 172.21.15.90:0/3508819806 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a3153c0640 msgr2=0x55a3153be020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:47.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.585+0000 7f83617c9740 1 --2- 172.21.15.90:0/3508819806 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a3153c0640 0x55a3153be020 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f83500045d0 tx=0x7f83500193e0 comp rx=0 tx=0).stop 2024-04-02T14:50:47.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.585+0000 7f83617c9740 1 -- 172.21.15.90:0/3508819806 shutdown_connections 2024-04-02T14:50:47.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.585+0000 7f83617c9740 1 --2- 172.21.15.90:0/3508819806 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x55a3153d0880 0x55a3153d2c70 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:47.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.585+0000 7f83617c9740 1 --2- 172.21.15.90:0/3508819806 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a3153c0640 0x55a3153be020 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:47.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.585+0000 7f83617c9740 1 --2- 172.21.15.90:0/3508819806 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a3153b3ba0 0x55a3153bdae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:47.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.585+0000 7f8345600640 0 client.19050 ms_handle_reset on v2:172.21.15.177:6838/664895663 2024-04-02T14:50:47.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.585+0000 7f83617c9740 1 --2- 172.21.15.90:0/3508819806 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a3153b2ec0 0x55a3153bd5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:47.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.585+0000 7f83617c9740 1 -- 172.21.15.90:0/3508819806 >> 172.21.15.90:0/3508819806 conn(0x55a315379a50 msgr2=0x55a3153af660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:50:47.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.585+0000 7f83617c9740 1 -- 172.21.15.90:0/3508819806 shutdown_connections 2024-04-02T14:50:47.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.585+0000 7f83617c9740 1 -- 172.21.15.90:0/3508819806 wait complete. 2024-04-02T14:50:47.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.585+0000 7f83617c9740 20 client.19050 trim_cache size 0 max 16384 2024-04-02T14:50:47.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.600+0000 7f83617c9740 1 Processor -- start 2024-04-02T14:50:47.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.600+0000 7f83617c9740 1 -- start start 2024-04-02T14:50:47.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.600+0000 7f83617c9740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a3153b2ec0 0x55a3153bfcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:47.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.600+0000 7f83617c9740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a3153b3ba0 0x55a3153cc0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:47.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.600+0000 7f83617c9740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a3153c0640 0x55a3153d83b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:47.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.600+0000 7f83617c9740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f83581ab5c0 con 0x55a3153b3ba0 2024-04-02T14:50:47.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.600+0000 7f83617c9740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f83581ad100 con 0x55a3153c0640 2024-04-02T14:50:47.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.600+0000 7f83617c9740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f83581abc90 con 0x55a3153b2ec0 2024-04-02T14:50:47.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.600+0000 7f8357400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a3153b3ba0 0x55a3153cc0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:47.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.600+0000 7f835ca00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a3153b2ec0 0x55a3153bfcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:47.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.600+0000 7f8357400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a3153b3ba0 0x55a3153cc0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:37916/0 (socket says 172.21.15.90:37916) 2024-04-02T14:50:47.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.600+0000 7f8357400640 1 -- 172.21.15.90:0/2082372203 learned_addr learned my addr 172.21.15.90:0/2082372203 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:50:47.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.600+0000 7f835ca00640 1 --2- 172.21.15.90:0/2082372203 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a3153c0640 0x55a3153d83b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:47.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.601+0000 7f8357400640 1 -- 172.21.15.90:0/2082372203 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a3153b2ec0 msgr2=0x55a3153bfcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:50:47.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.601+0000 7f8357400640 1 --2- 172.21.15.90:0/2082372203 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a3153b2ec0 0x55a3153bfcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:47.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.601+0000 7f8357400640 1 -- 172.21.15.90:0/2082372203 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a3153c0640 msgr2=0x55a3153d83b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:50:47.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.601+0000 7f8357400640 1 --2- 172.21.15.90:0/2082372203 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a3153c0640 0x55a3153d83b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:47.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.601+0000 7f8357400640 1 -- 172.21.15.90:0/2082372203 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a3153b1ef0 con 0x55a3153b3ba0 2024-04-02T14:50:47.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.601+0000 7f8357400640 1 --2- 172.21.15.90:0/2082372203 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a3153b3ba0 0x55a3153cc0e0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f8340036040 tx=0x7f834002e040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46bee3dca1c90267 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:50:47.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.601+0000 7f8347e00640 1 -- 172.21.15.90:0/2082372203 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8340004380 con 0x55a3153b3ba0 2024-04-02T14:50:47.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.601+0000 7f8347e00640 1 -- 172.21.15.90:0/2082372203 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f834002cd60 con 0x55a3153b3ba0 2024-04-02T14:50:47.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.601+0000 7f8347e00640 1 -- 172.21.15.90:0/2082372203 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8340004920 con 0x55a3153b3ba0 2024-04-02T14:50:47.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.602+0000 7f83617c9740 1 -- 172.21.15.90:0/2082372203 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a3153b3ba0 msgr2=0x55a3153cc0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:47.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.602+0000 7f83617c9740 1 --2- 172.21.15.90:0/2082372203 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a3153b3ba0 0x55a3153cc0e0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f8340036040 tx=0x7f834002e040 comp rx=0 tx=0).stop 2024-04-02T14:50:47.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.602+0000 7f83617c9740 1 -- 172.21.15.90:0/2082372203 shutdown_connections 2024-04-02T14:50:47.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.602+0000 7f83617c9740 1 --2- 172.21.15.90:0/2082372203 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a3153c0640 0x55a3153d83b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:47.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.602+0000 7f83617c9740 1 --2- 172.21.15.90:0/2082372203 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a3153b3ba0 0x55a3153cc0e0 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:47.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.602+0000 7f83617c9740 1 --2- 172.21.15.90:0/2082372203 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a3153b2ec0 0x55a3153bfcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:47.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.602+0000 7f83617c9740 1 -- 172.21.15.90:0/2082372203 >> 172.21.15.90:0/2082372203 conn(0x55a315379a50 msgr2=0x55a3153acaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:50:47.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.602+0000 7f83617c9740 1 -- 172.21.15.90:0/2082372203 shutdown_connections 2024-04-02T14:50:47.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.602+0000 7f83617c9740 1 -- 172.21.15.90:0/2082372203 wait complete. 2024-04-02T14:50:47.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.602+0000 7f83617c9740 1 -- start start 2024-04-02T14:50:47.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.603+0000 7f83617c9740 1 Processor -- start 2024-04-02T14:50:47.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.603+0000 7f83617c9740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a3153b2ec0 0x55a3153ba860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:47.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.603+0000 7f83617c9740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a3153b3ba0 0x55a3153bada0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:47.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.603+0000 7f83617c9740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a3153c0640 0x55a3153bb2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:47.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.603+0000 7f83617c9740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f83581aca30 con 0x55a3153b2ec0 2024-04-02T14:50:47.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.603+0000 7f83617c9740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f83581ac360 con 0x55a3153c0640 2024-04-02T14:50:47.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.603+0000 7f83617c9740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f83581a95c0 con 0x55a3153b3ba0 2024-04-02T14:50:47.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.603+0000 7f83617c9740 10 client.0 _ll_register_callbacks cb 0x7f8350003680 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:50:47.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.603+0000 7f835ca00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a3153b2ec0 0x55a3153ba860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:47.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.603+0000 7f835ca00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a3153b2ec0 0x55a3153ba860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:37922/0 (socket says 172.21.15.90:37922) 2024-04-02T14:50:47.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.603+0000 7f835ca00640 1 -- 172.21.15.90:0/2289513061 learned_addr learned my addr 172.21.15.90:0/2289513061 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:50:47.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.603+0000 7f8357400640 1 --2- 172.21.15.90:0/2289513061 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a3153b3ba0 0x55a3153bada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:47.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.603+0000 7f835ca00640 1 --2- 172.21.15.90:0/2289513061 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a3153c0640 0x55a3153bb2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:47.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.604+0000 7f8357400640 1 -- 172.21.15.90:0/2289513061 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a3153c0640 msgr2=0x55a3153bb2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:50:47.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.604+0000 7f8357400640 1 --2- 172.21.15.90:0/2289513061 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a3153c0640 0x55a3153bb2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:47.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.604+0000 7f8357400640 1 -- 172.21.15.90:0/2289513061 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a3153b2ec0 msgr2=0x55a3153ba860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:50:47.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.604+0000 7f8357400640 1 --2- 172.21.15.90:0/2289513061 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a3153b2ec0 0x55a3153ba860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:47.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.604+0000 7f8357400640 1 -- 172.21.15.90:0/2289513061 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a3153c8610 con 0x55a3153b3ba0 2024-04-02T14:50:47.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.604+0000 7f835ca00640 1 --2- 172.21.15.90:0/2289513061 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a3153b2ec0 0x55a3153ba860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:50:47.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.604+0000 7f8357400640 1 --2- 172.21.15.90:0/2289513061 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a3153b3ba0 0x55a3153bada0 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x55a3153bd0c0 tx=0x7f834002c880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2c784164e43a471c server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:50:47.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.604+0000 7f8345600640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:50:47.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.604+0000 7f8345600640 1 -- 172.21.15.90:0/2289513061 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8340055030 con 0x55a3153b3ba0 2024-04-02T14:50:47.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.604+0000 7f83617c9740 1 -- 172.21.15.90:0/2289513061 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f834002cd60 con 0x55a3153b3ba0 2024-04-02T14:50:47.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.604+0000 7f8345600640 1 -- 172.21.15.90:0/2289513061 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 8550+0+0 (secure 0 0 0) 0x7f8340004180 con 0x55a3153b3ba0 2024-04-02T14:50:47.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.605+0000 7f8345600640 1 -- 172.21.15.90:0/2289513061 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f8340004380 con 0x55a3153b3ba0 2024-04-02T14:50:47.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.605+0000 7f83617c9740 10 client.30175 fetch_fsmap learned FSMap version 27 2024-04-02T14:50:47.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.605+0000 7f83617c9740 1 -- 172.21.15.90:0/2289513061 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55a3153c1680 con 0x55a3153b3ba0 2024-04-02T14:50:47.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.605+0000 7f8345600640 1 -- 172.21.15.90:0/2289513061 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f834002cd60 con 0x55a3153b3ba0 2024-04-02T14:50:47.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.605+0000 7f8345600640 1 -- 172.21.15.90:0/2289513061 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f8340004f50 con 0x55a3153b3ba0 2024-04-02T14:50:47.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.605+0000 7f83617c9740 10 client.30175 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:50:47.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.605+0000 7f83617c9740 10 client.30175 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-04-02T14:50:47.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.605+0000 7f83617c9740 20 client.30175 populate_metadata read hostname 'smithi090' 2024-04-02T14:50:47.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.605+0000 7f83617c9740 1 --2- 172.21.15.90:0/2289513061 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x55a31543b410 0x55a31543d800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:47.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.605+0000 7f83617c9740 4 client.30175 mds_command: new command op to 24547 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-02T14:50:47.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.605+0000 7f83617c9740 1 -- 172.21.15.90:0/2289513061 --> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55a31531ef80 con 0x55a31543b410 2024-04-02T14:50:47.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.605+0000 7f835ca00640 1 --2- 172.21.15.90:0/2289513061 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x55a31543b410 0x55a31543d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:47.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.606+0000 7f835ca00640 1 --2- 172.21.15.90:0/2289513061 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x55a31543b410 0x55a31543d800 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=cbb48585428508e2 server_cookie=ba4600e92ab68c42 in_seq=0 out_seq=0 2024-04-02T14:50:47.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.606+0000 7f8345600640 10 client.30175 ms_handle_connect on v2:172.21.15.177:6838/664895663 2024-04-02T14:50:47.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.606+0000 7f8345600640 1 -- 172.21.15.90:0/2289513061 <== mds.2 v2:172.21.15.177:6838/664895663 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55a31531ef80 con 0x55a31543b410 2024-04-02T14:50:47.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.606+0000 7f8345600640 10 client.30175 handle_command_reply: tid=0 2024-04-02T14:50:47.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.606+0000 7f83617c9740 1 client.30175 shutdown 2024-04-02T14:50:47.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.606+0000 7f83617c9740 1 -- 172.21.15.90:0/2289513061 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a3153b3ba0 msgr2=0x55a3153bada0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:47.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.606+0000 7f83617c9740 1 --2- 172.21.15.90:0/2289513061 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a3153b3ba0 0x55a3153bada0 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x55a3153bd0c0 tx=0x7f834002c880 comp rx=0 tx=0).stop 2024-04-02T14:50:47.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.606+0000 7f83617c9740 1 -- 172.21.15.90:0/2289513061 shutdown_connections 2024-04-02T14:50:47.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.606+0000 7f83617c9740 1 --2- 172.21.15.90:0/2289513061 >> [v2:172.21.15.177:6838/664895663,v1:172.21.15.177:6839/664895663] conn(0x55a31543b410 0x55a31543d800 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:47.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.606+0000 7f83617c9740 1 --2- 172.21.15.90:0/2289513061 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55a3153c0640 0x55a3153bb2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:47.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.606+0000 7f8345600640 0 client.30175 ms_handle_reset on v2:172.21.15.177:6838/664895663 2024-04-02T14:50:47.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.606+0000 7f83617c9740 1 --2- 172.21.15.90:0/2289513061 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a3153b3ba0 0x55a3153bada0 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:47.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.606+0000 7f83617c9740 1 --2- 172.21.15.90:0/2289513061 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55a3153b2ec0 0x55a3153ba860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:47.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.606+0000 7f83617c9740 1 -- 172.21.15.90:0/2289513061 >> 172.21.15.90:0/2289513061 conn(0x55a315379a50 msgr2=0x55a3153bd560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:50:47.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.607+0000 7f83617c9740 1 -- 172.21.15.90:0/2289513061 shutdown_connections 2024-04-02T14:50:47.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.607+0000 7f83617c9740 1 -- 172.21.15.90:0/2289513061 wait complete. 2024-04-02T14:50:47.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.607+0000 7f83617c9740 20 client.30175 trim_cache size 0 max 16384 2024-04-02T14:50:47.604 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-02T14:50:47.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.607+0000 7f835d400640 1 -- 172.21.15.90:0/3249196995 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f833c0a3d20 msgr2=0x7f833c0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:47.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.607+0000 7f835d400640 1 --2- 172.21.15.90:0/3249196995 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f833c0a3d20 0x7f833c0a61e0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x55a3153b40f0 tx=0x7f83480058d0 comp rx=0 tx=0).stop 2024-04-02T14:50:47.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.607+0000 7f835d400640 1 -- 172.21.15.90:0/3249196995 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8358101970 msgr2=0x7f8358198310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:47.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.607+0000 7f835d400640 1 --2- 172.21.15.90:0/3249196995 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8358101970 0x7f8358198310 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f834800eb90 tx=0x7f834800ef90 comp rx=0 tx=0).stop 2024-04-02T14:50:47.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.607+0000 7f835d400640 1 -- 172.21.15.90:0/3249196995 shutdown_connections 2024-04-02T14:50:47.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.607+0000 7f835d400640 1 --2- 172.21.15.90:0/3249196995 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f833c0a3d20 0x7f833c0a61e0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:47.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.607+0000 7f835d400640 1 --2- 172.21.15.90:0/3249196995 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8358107980 0x7f835819cbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:47.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.607+0000 7f835d400640 1 --2- 172.21.15.90:0/3249196995 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8358102330 0x7f8358198850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:47.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.607+0000 7f835d400640 1 --2- 172.21.15.90:0/3249196995 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8358101970 0x7f8358198310 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:47.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.607+0000 7f835d400640 1 -- 172.21.15.90:0/3249196995 >> 172.21.15.90:0/3249196995 conn(0x7f83580fd4a0 msgr2=0x7f83580fdce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:50:47.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.608+0000 7f835d400640 1 -- 172.21.15.90:0/3249196995 shutdown_connections 2024-04-02T14:50:47.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.608+0000 7f835d400640 1 -- 172.21.15.90:0/3249196995 wait complete. 2024-04-02T14:50:47.606 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:50:47 smithi092 ceph-mon[36497]: from='client.? 172.21.15.90:0/3065706362' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-02T14:50:47.619 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-02T14:50:47.678 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:50:47 smithi177 ceph-mon[36431]: from='client.? 172.21.15.90:0/3065706362' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-02T14:50:47.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.839+0000 7f728b200640 1 -- 172.21.15.90:0/1627386820 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7284076950 msgr2=0x7f7284112400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:47.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.839+0000 7f728b200640 1 --2- 172.21.15.90:0/1627386820 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7284076950 0x7f7284112400 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7f726c009930 tx=0x7f726c02bc20 comp rx=0 tx=0).stop 2024-04-02T14:50:47.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.839+0000 7f728b200640 1 -- 172.21.15.90:0/1627386820 shutdown_connections 2024-04-02T14:50:47.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.839+0000 7f728b200640 1 --2- 172.21.15.90:0/1627386820 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7284112940 0x7f7284114d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:47.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.839+0000 7f728b200640 1 --2- 172.21.15.90:0/1627386820 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7284076950 0x7f7284112400 unknown :-1 s=CLOSED pgs=1045 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:47.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.839+0000 7f728b200640 1 --2- 172.21.15.90:0/1627386820 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7284076030 0x7f7284076410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:47.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.839+0000 7f728b200640 1 -- 172.21.15.90:0/1627386820 >> 172.21.15.90:0/1627386820 conn(0x7f72840ff7b0 msgr2=0x7f7284101bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:50:47.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.839+0000 7f728b200640 1 -- 172.21.15.90:0/1627386820 shutdown_connections 2024-04-02T14:50:47.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.839+0000 7f728b200640 1 -- 172.21.15.90:0/1627386820 wait complete. 2024-04-02T14:50:47.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.840+0000 7f728b200640 1 Processor -- start 2024-04-02T14:50:47.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.841+0000 7f728b200640 1 -- start start 2024-04-02T14:50:47.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.841+0000 7f728b200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7284076030 0x7f72841a46c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:47.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.841+0000 7f728b200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7284076950 0x7f72841a4c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:47.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.841+0000 7f728b200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7284112940 0x7f728419e830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:47.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.841+0000 7f728b200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f72841184b0 con 0x7f7284076030 2024-04-02T14:50:47.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.841+0000 7f728b200640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f7284118330 con 0x7f7284112940 2024-04-02T14:50:47.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.841+0000 7f728b200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f7284118630 con 0x7f7284076950 2024-04-02T14:50:47.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.841+0000 7f7283e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7284076030 0x7f72841a46c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:47.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.841+0000 7f7283e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7284076030 0x7f72841a46c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:37944/0 (socket says 172.21.15.90:37944) 2024-04-02T14:50:47.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.841+0000 7f7283e00640 1 -- 172.21.15.90:0/4044948735 learned_addr learned my addr 172.21.15.90:0/4044948735 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:50:47.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.841+0000 7f7289200640 1 --2- 172.21.15.90:0/4044948735 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7284112940 0x7f728419e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:47.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.841+0000 7f7283400640 1 --2- 172.21.15.90:0/4044948735 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7284076950 0x7f72841a4c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:47.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.842+0000 7f7283e00640 1 -- 172.21.15.90:0/4044948735 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7284076950 msgr2=0x7f72841a4c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:47.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.842+0000 7f7283e00640 1 --2- 172.21.15.90:0/4044948735 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7284076950 0x7f72841a4c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:47.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.842+0000 7f7283e00640 1 -- 172.21.15.90:0/4044948735 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7284112940 msgr2=0x7f728419e830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:47.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.842+0000 7f7283e00640 1 --2- 172.21.15.90:0/4044948735 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7284112940 0x7f728419e830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:47.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.842+0000 7f7283e00640 1 -- 172.21.15.90:0/4044948735 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f728419f000 con 0x7f7284076030 2024-04-02T14:50:47.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.842+0000 7f7289200640 1 --2- 172.21.15.90:0/4044948735 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7284112940 0x7f728419e830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:50:47.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.842+0000 7f7283e00640 1 --2- 172.21.15.90:0/4044948735 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7284076030 0x7f72841a46c0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f727400e9f0 tx=0x7f727400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:50:47.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.842+0000 7f7280c00640 1 -- 172.21.15.90:0/4044948735 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f727400cd80 con 0x7f7284076030 2024-04-02T14:50:47.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.842+0000 7f728b200640 1 -- 172.21.15.90:0/4044948735 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f728419f2f0 con 0x7f7284076030 2024-04-02T14:50:47.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.842+0000 7f7280c00640 1 -- 172.21.15.90:0/4044948735 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7274005500 con 0x7f7284076030 2024-04-02T14:50:47.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.842+0000 7f7280c00640 1 -- 172.21.15.90:0/4044948735 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7274010690 con 0x7f7284076030 2024-04-02T14:50:47.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.843+0000 7f728b200640 1 -- 172.21.15.90:0/4044948735 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72841ab510 con 0x7f7284076030 2024-04-02T14:50:47.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.846+0000 7f728e033740 1 Processor -- start 2024-04-02T14:50:47.840 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.846+0000 7f7280c00640 1 -- 172.21.15.90:0/4044948735 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f72740046c0 con 0x7f7284076030 2024-04-02T14:50:47.840 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.846+0000 7f728e033740 1 -- start start 2024-04-02T14:50:47.840 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.846+0000 7f728e033740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564e4ee969f0 0x564e4ee95740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:47.840 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.846+0000 7f728e033740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564e4ee95d10 0x564e4ee96190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:47.840 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.846+0000 7f728e033740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564e4ee9f080 0x564e4eea1470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:47.840 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.846+0000 7f728e033740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x564e4edd6400 con 0x564e4ee969f0 2024-04-02T14:50:47.840 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.846+0000 7f728e033740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x564e4edbaf10 con 0x564e4ee95d10 2024-04-02T14:50:47.840 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.846+0000 7f728e033740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x564e4edfa6f0 con 0x564e4ee9f080 2024-04-02T14:50:47.840 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.846+0000 7f7289200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564e4ee969f0 0x564e4ee95740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:47.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.847+0000 7f7283e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564e4ee95d10 0x564e4ee96190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:47.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.847+0000 7f7289200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564e4ee969f0 0x564e4ee95740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:37946/0 (socket says 172.21.15.90:37946) 2024-04-02T14:50:47.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.847+0000 7f7283400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564e4ee9f080 0x564e4eea1470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:47.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.847+0000 7f7289200640 1 -- 172.21.15.90:0/3431522871 learned_addr learned my addr 172.21.15.90:0/3431522871 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:50:47.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.847+0000 7f7289200640 1 -- 172.21.15.90:0/3431522871 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564e4ee9f080 msgr2=0x564e4eea1470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:50:47.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.847+0000 7f7289200640 1 --2- 172.21.15.90:0/3431522871 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564e4ee9f080 0x564e4eea1470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:47.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.847+0000 7f7289200640 1 -- 172.21.15.90:0/3431522871 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564e4ee95d10 msgr2=0x564e4ee96190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:50:47.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.847+0000 7f7289200640 1 --2- 172.21.15.90:0/3431522871 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564e4ee95d10 0x564e4ee96190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:47.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.847+0000 7f7289200640 1 -- 172.21.15.90:0/3431522871 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564e4eb0e3f0 con 0x564e4ee969f0 2024-04-02T14:50:47.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.847+0000 7f7283e00640 1 --2- 172.21.15.90:0/3431522871 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564e4ee95d10 0x564e4ee96190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:50:47.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.847+0000 7f7289200640 1 --2- 172.21.15.90:0/3431522871 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564e4ee969f0 0x564e4ee95740 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f727c011760 tx=0x7f727c011c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d12178de9493afd server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:50:47.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.848+0000 7f7279600640 1 -- 172.21.15.90:0/3431522871 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f727c004d20 con 0x564e4ee969f0 2024-04-02T14:50:47.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.848+0000 7f7279600640 1 -- 172.21.15.90:0/3431522871 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f727c00f9a0 con 0x564e4ee969f0 2024-04-02T14:50:47.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.848+0000 7f7279600640 1 -- 172.21.15.90:0/3431522871 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f727c004530 con 0x564e4ee969f0 2024-04-02T14:50:47.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.848+0000 7f7280c00640 1 --2- 172.21.15.90:0/4044948735 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f72680a3d20 0x7f72680a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:47.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.848+0000 7f7280c00640 1 -- 172.21.15.90:0/4044948735 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 8550+0+0 (secure 0 0 0) 0x7f72740d0c90 con 0x7f7284076030 2024-04-02T14:50:47.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.848+0000 7f7283e00640 1 --2- 172.21.15.90:0/4044948735 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f72680a3d20 0x7f72680a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:47.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.848+0000 7f728e033740 1 -- 172.21.15.90:0/3431522871 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564e4ee969f0 msgr2=0x564e4ee95740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:47.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.848+0000 7f728e033740 1 --2- 172.21.15.90:0/3431522871 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564e4ee969f0 0x564e4ee95740 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f727c011760 tx=0x7f727c011c20 comp rx=0 tx=0).stop 2024-04-02T14:50:47.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.848+0000 7f728e033740 1 -- 172.21.15.90:0/3431522871 shutdown_connections 2024-04-02T14:50:47.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.848+0000 7f728e033740 1 --2- 172.21.15.90:0/3431522871 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564e4ee9f080 0x564e4eea1470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:47.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.848+0000 7f728e033740 1 --2- 172.21.15.90:0/3431522871 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564e4ee95d10 0x564e4ee96190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:47.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.848+0000 7f728e033740 1 --2- 172.21.15.90:0/3431522871 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564e4ee969f0 0x564e4ee95740 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:47.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.848+0000 7f728e033740 1 -- 172.21.15.90:0/3431522871 >> 172.21.15.90:0/3431522871 conn(0x564e4ee5ca50 msgr2=0x564e4ee90760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:50:47.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.849+0000 7f728e033740 1 -- 172.21.15.90:0/3431522871 shutdown_connections 2024-04-02T14:50:47.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.849+0000 7f7283e00640 1 --2- 172.21.15.90:0/4044948735 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f72680a3d20 0x7f72680a61e0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x564e4ee96f40 tx=0x7f72740d4740 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:50:47.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.849+0000 7f728e033740 1 -- 172.21.15.90:0/3431522871 wait complete. 2024-04-02T14:50:47.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.849+0000 7f728e033740 1 -- start start 2024-04-02T14:50:47.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.849+0000 7f728e033740 1 Processor -- start 2024-04-02T14:50:47.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.850+0000 7f728e033740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564e4ee95d10 0x564e4ee9c070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:47.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.850+0000 7f728e033740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564e4ee969f0 0x564e4ee9c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:47.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.850+0000 7f728e033740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564e4ee9f080 0x564e4ee9caf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:47.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.850+0000 7f728e033740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x564e4edf66a0 con 0x564e4ee95d10 2024-04-02T14:50:47.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.850+0000 7f728e033740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x564e4eda2980 con 0x564e4ee969f0 2024-04-02T14:50:47.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.850+0000 7f728e033740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x564e4edb5cb0 con 0x564e4ee9f080 2024-04-02T14:50:47.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.850+0000 7f728e033740 10 client.0 _ll_register_callbacks cb 0x564e4edb56a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:50:47.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.850+0000 7f7289200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564e4ee95d10 0x564e4ee9c070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:47.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.850+0000 7f7289200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564e4ee95d10 0x564e4ee9c070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:37950/0 (socket says 172.21.15.90:37950) 2024-04-02T14:50:47.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.850+0000 7f7289200640 1 -- 172.21.15.90:0/3209020421 learned_addr learned my addr 172.21.15.90:0/3209020421 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:50:47.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.850+0000 7f7283400640 1 --2- 172.21.15.90:0/3209020421 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564e4ee969f0 0x564e4ee9c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:47.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.850+0000 7f7289200640 1 --2- 172.21.15.90:0/3209020421 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564e4ee9f080 0x564e4ee9caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:47.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.851+0000 7f7289200640 1 -- 172.21.15.90:0/3209020421 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564e4ee9f080 msgr2=0x564e4ee9caf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:50:47.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.851+0000 7f7289200640 1 --2- 172.21.15.90:0/3209020421 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564e4ee9f080 0x564e4ee9caf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:47.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.851+0000 7f7289200640 1 -- 172.21.15.90:0/3209020421 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564e4ee969f0 msgr2=0x564e4ee9c5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:50:47.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.851+0000 7f7289200640 1 --2- 172.21.15.90:0/3209020421 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564e4ee969f0 0x564e4ee9c5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:47.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.851+0000 7f7289200640 1 -- 172.21.15.90:0/3209020421 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564e4ebd87d0 con 0x564e4ee95d10 2024-04-02T14:50:47.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.851+0000 7f7289200640 1 --2- 172.21.15.90:0/3209020421 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564e4ee95d10 0x564e4ee9c070 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f727c005c10 tx=0x7f727c009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ceb6817d251c4d9 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:50:47.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.851+0000 7f7272000640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-02T14:50:47.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.851+0000 7f7272000640 1 -- 172.21.15.90:0/3209020421 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f727c005000 con 0x564e4ee95d10 2024-04-02T14:50:47.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.851+0000 7f7272000640 1 -- 172.21.15.90:0/3209020421 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 8550+0+0 (secure 0 0 0) 0x7f727c0051a0 con 0x564e4ee95d10 2024-04-02T14:50:47.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.851+0000 7f728e033740 1 -- 172.21.15.90:0/3209020421 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f727c00f9a0 con 0x564e4ee95d10 2024-04-02T14:50:47.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.851+0000 7f7272000640 1 -- 172.21.15.90:0/3209020421 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f727c0053a0 con 0x564e4ee95d10 2024-04-02T14:50:47.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.851+0000 7f728e033740 10 client.19068 fetch_fsmap learned FSMap version 27 2024-04-02T14:50:47.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.851+0000 7f728e033740 1 -- 172.21.15.90:0/3209020421 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x564e4eeadf20 con 0x564e4ee95d10 2024-04-02T14:50:47.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.852+0000 7f7272000640 1 -- 172.21.15.90:0/3209020421 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f727c00f9a0 con 0x564e4ee95d10 2024-04-02T14:50:47.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.852+0000 7f7272000640 1 -- 172.21.15.90:0/3209020421 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f727c018070 con 0x564e4ee95d10 2024-04-02T14:50:47.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.852+0000 7f728e033740 10 client.19068 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:50:47.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.852+0000 7f728e033740 10 client.19068 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2024-04-02T14:50:47.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.852+0000 7f728e033740 20 client.19068 populate_metadata read hostname 'smithi090' 2024-04-02T14:50:47.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.852+0000 7f728e033740 1 --2- 172.21.15.90:0/3209020421 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x564e4eeaf320 0x564e4eeb1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:47.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.852+0000 7f728e033740 4 client.19068 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:50:47.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.852+0000 7f728e033740 1 -- 172.21.15.90:0/3209020421 --> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564e4edba040 con 0x564e4eeaf320 2024-04-02T14:50:47.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.852+0000 7f7283400640 1 --2- 172.21.15.90:0/3209020421 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x564e4eeaf320 0x564e4eeb1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:47.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.853+0000 7f7283400640 1 --2- 172.21.15.90:0/3209020421 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x564e4eeaf320 0x564e4eeb1710 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=e60d7916e4df421d server_cookie=a0b10491088436ef in_seq=0 out_seq=0 2024-04-02T14:50:47.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.853+0000 7f7272000640 10 client.19068 ms_handle_connect on v2:172.21.15.90:6838/1498915870 2024-04-02T14:50:47.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.854+0000 7f7272000640 1 -- 172.21.15.90:0/3209020421 <== mds.1 v2:172.21.15.90:6838/1498915870 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x564e4edba040 con 0x564e4eeaf320 2024-04-02T14:50:47.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.854+0000 7f7272000640 10 client.19068 handle_command_reply: tid=0 2024-04-02T14:50:47.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.854+0000 7f728e033740 1 client.19068 shutdown 2024-04-02T14:50:47.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.854+0000 7f728e033740 1 -- 172.21.15.90:0/3209020421 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564e4ee95d10 msgr2=0x564e4ee9c070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:47.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.854+0000 7f728e033740 1 --2- 172.21.15.90:0/3209020421 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564e4ee95d10 0x564e4ee9c070 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f727c005c10 tx=0x7f727c009f90 comp rx=0 tx=0).stop 2024-04-02T14:50:47.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.854+0000 7f728e033740 1 -- 172.21.15.90:0/3209020421 shutdown_connections 2024-04-02T14:50:47.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.854+0000 7f728e033740 1 --2- 172.21.15.90:0/3209020421 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x564e4eeaf320 0x564e4eeb1710 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:47.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.854+0000 7f728e033740 1 --2- 172.21.15.90:0/3209020421 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564e4ee9f080 0x564e4ee9caf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:47.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.854+0000 7f728e033740 1 --2- 172.21.15.90:0/3209020421 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564e4ee969f0 0x564e4ee9c5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:47.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.854+0000 7f728e033740 1 --2- 172.21.15.90:0/3209020421 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564e4ee95d10 0x564e4ee9c070 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:47.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.854+0000 7f7272000640 0 client.19068 ms_handle_reset on v2:172.21.15.90:6838/1498915870 2024-04-02T14:50:47.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.854+0000 7f728e033740 1 -- 172.21.15.90:0/3209020421 >> 172.21.15.90:0/3209020421 conn(0x564e4ee5ca50 msgr2=0x564e4ee924b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:50:47.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.854+0000 7f728e033740 1 -- 172.21.15.90:0/3209020421 shutdown_connections 2024-04-02T14:50:47.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.854+0000 7f728e033740 1 -- 172.21.15.90:0/3209020421 wait complete. 2024-04-02T14:50:47.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.855+0000 7f728e033740 20 client.19068 trim_cache size 0 max 16384 2024-04-02T14:50:47.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.869+0000 7f728e033740 1 Processor -- start 2024-04-02T14:50:47.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.869+0000 7f728e033740 1 -- start start 2024-04-02T14:50:47.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.869+0000 7f728e033740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564e4ee95d10 0x564e4eeaaaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:47.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.869+0000 7f728e033740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564e4ee969f0 0x564e4ee9b950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:47.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.869+0000 7f728e033740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564e4ee9f080 0x564e4ee8b7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:47.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.869+0000 7f728e033740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f72841b3f70 con 0x564e4ee95d10 2024-04-02T14:50:47.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.869+0000 7f728e033740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f72841b5ab0 con 0x564e4ee9f080 2024-04-02T14:50:47.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.869+0000 7f728e033740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f72841b4640 con 0x564e4ee969f0 2024-04-02T14:50:47.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.870+0000 7f7289200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564e4ee95d10 0x564e4eeaaaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:47.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.870+0000 7f7289200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564e4ee95d10 0x564e4eeaaaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:37966/0 (socket says 172.21.15.90:37966) 2024-04-02T14:50:47.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.870+0000 7f7289200640 1 -- 172.21.15.90:0/156513567 learned_addr learned my addr 172.21.15.90:0/156513567 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:50:47.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.870+0000 7f7283400640 1 --2- 172.21.15.90:0/156513567 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564e4ee969f0 0x564e4ee9b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:47.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.870+0000 7f7289200640 1 -- 172.21.15.90:0/156513567 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564e4ee969f0 msgr2=0x564e4ee9b950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:50:47.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.870+0000 7f7289200640 1 --2- 172.21.15.90:0/156513567 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564e4ee969f0 0x564e4ee9b950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:47.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.870+0000 7f7289200640 1 -- 172.21.15.90:0/156513567 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564e4ee9f080 msgr2=0x564e4ee8b7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:50:47.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.870+0000 7f7289200640 1 --2- 172.21.15.90:0/156513567 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564e4ee9f080 0x564e4ee8b7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:47.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.870+0000 7f7289200640 1 -- 172.21.15.90:0/156513567 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564e4ee90970 con 0x564e4ee95d10 2024-04-02T14:50:47.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.870+0000 7f7289200640 1 --2- 172.21.15.90:0/156513567 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564e4ee95d10 0x564e4eeaaaf0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f727c002230 tx=0x7f727c032e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9290a22260c8e75b server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:50:47.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.871+0000 7f7278c00640 1 -- 172.21.15.90:0/156513567 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f727c00f9a0 con 0x564e4ee95d10 2024-04-02T14:50:47.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.871+0000 7f7278c00640 1 -- 172.21.15.90:0/156513567 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f727c01f040 con 0x564e4ee95d10 2024-04-02T14:50:47.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.871+0000 7f7278c00640 1 -- 172.21.15.90:0/156513567 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f727c032480 con 0x564e4ee95d10 2024-04-02T14:50:47.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.871+0000 7f728e033740 1 -- 172.21.15.90:0/156513567 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564e4ee95d10 msgr2=0x564e4eeaaaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:47.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.871+0000 7f728e033740 1 --2- 172.21.15.90:0/156513567 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564e4ee95d10 0x564e4eeaaaf0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f727c002230 tx=0x7f727c032e80 comp rx=0 tx=0).stop 2024-04-02T14:50:47.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.871+0000 7f728e033740 1 -- 172.21.15.90:0/156513567 shutdown_connections 2024-04-02T14:50:47.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.871+0000 7f728e033740 1 --2- 172.21.15.90:0/156513567 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564e4ee9f080 0x564e4ee8b7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:47.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.871+0000 7f728e033740 1 --2- 172.21.15.90:0/156513567 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564e4ee969f0 0x564e4ee9b950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:47.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.871+0000 7f728e033740 1 --2- 172.21.15.90:0/156513567 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564e4ee95d10 0x564e4eeaaaf0 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:47.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.871+0000 7f728e033740 1 -- 172.21.15.90:0/156513567 >> 172.21.15.90:0/156513567 conn(0x564e4ee5ca50 msgr2=0x564e4ee8f8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:50:47.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.871+0000 7f728e033740 1 -- 172.21.15.90:0/156513567 shutdown_connections 2024-04-02T14:50:47.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.871+0000 7f728e033740 1 -- 172.21.15.90:0/156513567 wait complete. 2024-04-02T14:50:47.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.872+0000 7f728e033740 1 -- start start 2024-04-02T14:50:47.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.872+0000 7f728e033740 1 Processor -- start 2024-04-02T14:50:47.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.872+0000 7f728e033740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564e4ee95d10 0x564e4ee9f460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:47.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.872+0000 7f728e033740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564e4ee969f0 0x564e4ee9f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:47.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.872+0000 7f728e033740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564e4ee9f080 0x564e4ee9fee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:47.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.872+0000 7f728e033740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f72841b53e0 con 0x564e4ee95d10 2024-04-02T14:50:47.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.872+0000 7f728e033740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f72841b4d10 con 0x564e4ee9f080 2024-04-02T14:50:47.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.872+0000 7f728e033740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f72841b1f70 con 0x564e4ee969f0 2024-04-02T14:50:47.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.872+0000 7f728e033740 10 client.0 _ll_register_callbacks cb 0x7f727c011660 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:50:47.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.873+0000 7f7283400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564e4ee969f0 0x564e4ee9f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:47.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.873+0000 7f7289200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564e4ee95d10 0x564e4ee9f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:47.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.873+0000 7f7283400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564e4ee969f0 0x564e4ee9f9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:51162/0 (socket says 172.21.15.90:51162) 2024-04-02T14:50:47.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.873+0000 7f7283400640 1 -- 172.21.15.90:0/3575293344 learned_addr learned my addr 172.21.15.90:0/3575293344 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:50:47.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.873+0000 7f7289200640 1 --2- 172.21.15.90:0/3575293344 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564e4ee9f080 0x564e4ee9fee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:47.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.873+0000 7f7289200640 1 -- 172.21.15.90:0/3575293344 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564e4ee969f0 msgr2=0x564e4ee9f9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:50:47.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.873+0000 7f7289200640 1 --2- 172.21.15.90:0/3575293344 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564e4ee969f0 0x564e4ee9f9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:47.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.873+0000 7f7289200640 1 -- 172.21.15.90:0/3575293344 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564e4ee9f080 msgr2=0x564e4ee9fee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:50:47.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.873+0000 7f7289200640 1 --2- 172.21.15.90:0/3575293344 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564e4ee9f080 0x564e4ee9fee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:47.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.873+0000 7f7289200640 1 -- 172.21.15.90:0/3575293344 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564e4eea62a0 con 0x564e4ee95d10 2024-04-02T14:50:47.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.873+0000 7f7289200640 1 --2- 172.21.15.90:0/3575293344 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564e4ee95d10 0x564e4ee9f460 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f727c000f80 tx=0x7f727c005bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=728f936915f1151b server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:50:47.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.873+0000 7f7272000640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-02T14:50:47.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.873+0000 7f7272000640 1 -- 172.21.15.90:0/3575293344 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f727c005540 con 0x564e4ee95d10 2024-04-02T14:50:47.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.873+0000 7f7272000640 1 -- 172.21.15.90:0/3575293344 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 8550+0+0 (secure 0 0 0) 0x7f727c0056e0 con 0x564e4ee95d10 2024-04-02T14:50:47.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.873+0000 7f728e033740 1 -- 172.21.15.90:0/3575293344 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f727c01f040 con 0x564e4ee95d10 2024-04-02T14:50:47.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.874+0000 7f7272000640 1 -- 172.21.15.90:0/3575293344 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f727c005900 con 0x564e4ee95d10 2024-04-02T14:50:47.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.874+0000 7f728e033740 10 client.19074 fetch_fsmap learned FSMap version 27 2024-04-02T14:50:47.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.874+0000 7f728e033740 1 -- 172.21.15.90:0/3575293344 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x564e4eea75a0 con 0x564e4ee95d10 2024-04-02T14:50:47.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.874+0000 7f7272000640 1 -- 172.21.15.90:0/3575293344 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f727c01f040 con 0x564e4ee95d10 2024-04-02T14:50:47.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.874+0000 7f7272000640 1 -- 172.21.15.90:0/3575293344 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f727c018070 con 0x564e4ee95d10 2024-04-02T14:50:47.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.874+0000 7f728e033740 10 client.19074 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:50:47.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.874+0000 7f728e033740 10 client.19074 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2024-04-02T14:50:47.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.874+0000 7f728e033740 20 client.19074 populate_metadata read hostname 'smithi090' 2024-04-02T14:50:47.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.874+0000 7f728e033740 1 --2- 172.21.15.90:0/3575293344 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x564e4ef157b0 0x564e4ef17ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:47.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.874+0000 7f728e033740 4 client.19074 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-02T14:50:47.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.874+0000 7f728e033740 1 -- 172.21.15.90:0/3575293344 --> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] -- command(tid 0: {"prefix": "damage ls"}) -- 0x564e4ee07c30 con 0x564e4ef157b0 2024-04-02T14:50:47.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.875+0000 7f7283400640 1 --2- 172.21.15.90:0/3575293344 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x564e4ef157b0 0x564e4ef17ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:47.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.875+0000 7f7283400640 1 --2- 172.21.15.90:0/3575293344 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x564e4ef157b0 0x564e4ef17ba0 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=94d8d9337dd6879c server_cookie=850f57cc505f18ff in_seq=0 out_seq=0 2024-04-02T14:50:47.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.875+0000 7f7272000640 10 client.19074 ms_handle_connect on v2:172.21.15.90:6838/1498915870 2024-04-02T14:50:47.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.875+0000 7f7272000640 1 -- 172.21.15.90:0/3575293344 <== mds.1 v2:172.21.15.90:6838/1498915870 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x564e4ee07c30 con 0x564e4ef157b0 2024-04-02T14:50:47.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.875+0000 7f7272000640 10 client.19074 handle_command_reply: tid=0 2024-04-02T14:50:47.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.875+0000 7f728e033740 1 client.19074 shutdown 2024-04-02T14:50:47.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.876+0000 7f728e033740 1 -- 172.21.15.90:0/3575293344 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564e4ee95d10 msgr2=0x564e4ee9f460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:47.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.876+0000 7f728e033740 1 --2- 172.21.15.90:0/3575293344 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564e4ee95d10 0x564e4ee9f460 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f727c000f80 tx=0x7f727c005bd0 comp rx=0 tx=0).stop 2024-04-02T14:50:47.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.876+0000 7f728e033740 1 -- 172.21.15.90:0/3575293344 shutdown_connections 2024-04-02T14:50:47.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.876+0000 7f728e033740 1 --2- 172.21.15.90:0/3575293344 >> [v2:172.21.15.90:6838/1498915870,v1:172.21.15.90:6839/1498915870] conn(0x564e4ef157b0 0x564e4ef17ba0 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:47.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.876+0000 7f728e033740 1 --2- 172.21.15.90:0/3575293344 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x564e4ee9f080 0x564e4ee9fee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:47.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.876+0000 7f728e033740 1 --2- 172.21.15.90:0/3575293344 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564e4ee969f0 0x564e4ee9f9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:47.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.876+0000 7f7272000640 0 client.19074 ms_handle_reset on v2:172.21.15.90:6838/1498915870 2024-04-02T14:50:47.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.876+0000 7f728e033740 1 --2- 172.21.15.90:0/3575293344 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x564e4ee95d10 0x564e4ee9f460 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:47.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.876+0000 7f728e033740 1 -- 172.21.15.90:0/3575293344 >> 172.21.15.90:0/3575293344 conn(0x564e4ee5ca50 msgr2=0x564e4ee9c030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:50:47.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.876+0000 7f728e033740 1 -- 172.21.15.90:0/3575293344 shutdown_connections 2024-04-02T14:50:47.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.876+0000 7f728e033740 1 -- 172.21.15.90:0/3575293344 wait complete. 2024-04-02T14:50:47.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.876+0000 7f728e033740 20 client.19074 trim_cache size 0 max 16384 2024-04-02T14:50:47.874 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-02T14:50:47.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.877+0000 7f728b200640 1 -- 172.21.15.90:0/4044948735 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f72680a3d20 msgr2=0x7f72680a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:47.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.877+0000 7f728b200640 1 --2- 172.21.15.90:0/4044948735 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f72680a3d20 0x7f72680a61e0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x564e4ee96f40 tx=0x7f72740d4740 comp rx=0 tx=0).stop 2024-04-02T14:50:47.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.877+0000 7f728b200640 1 -- 172.21.15.90:0/4044948735 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7284076030 msgr2=0x7f72841a46c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:47.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.877+0000 7f728b200640 1 --2- 172.21.15.90:0/4044948735 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7284076030 0x7f72841a46c0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f727400e9f0 tx=0x7f727400edf0 comp rx=0 tx=0).stop 2024-04-02T14:50:47.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.877+0000 7f728b200640 1 -- 172.21.15.90:0/4044948735 shutdown_connections 2024-04-02T14:50:47.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.877+0000 7f728b200640 1 --2- 172.21.15.90:0/4044948735 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f72680a3d20 0x7f72680a61e0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:47.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.877+0000 7f728b200640 1 --2- 172.21.15.90:0/4044948735 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7284112940 0x7f728419e830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:47.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.877+0000 7f728b200640 1 --2- 172.21.15.90:0/4044948735 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7284076950 0x7f72841a4c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:47.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.877+0000 7f728b200640 1 --2- 172.21.15.90:0/4044948735 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7284076030 0x7f72841a46c0 secure :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f727400e9f0 tx=0x7f727400edf0 comp rx=0 tx=0).stop 2024-04-02T14:50:47.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.877+0000 7f728b200640 1 -- 172.21.15.90:0/4044948735 >> 172.21.15.90:0/4044948735 conn(0x7f72840ff7b0 msgr2=0x7f7284101ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:50:47.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.877+0000 7f728b200640 1 -- 172.21.15.90:0/4044948735 shutdown_connections 2024-04-02T14:50:47.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:47.877+0000 7f728b200640 1 -- 172.21.15.90:0/4044948735 wait complete. 2024-04-02T14:50:47.887 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-02T14:50:48.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.097+0000 7f1e62000640 1 -- 172.21.15.90:0/3530972476 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1e5c1082b0 msgr2=0x7f1e5c108690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:48.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.097+0000 7f1e62000640 1 --2- 172.21.15.90:0/3530972476 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1e5c1082b0 0x7f1e5c108690 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f1e50009900 tx=0x7f1e5002bc40 comp rx=0 tx=0).stop 2024-04-02T14:50:48.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.098+0000 7f1e62000640 1 -- 172.21.15.90:0/3530972476 shutdown_connections 2024-04-02T14:50:48.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.098+0000 7f1e62000640 1 --2- 172.21.15.90:0/3530972476 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1e5c10d300 0x7f1e5c114ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:48.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.098+0000 7f1e62000640 1 --2- 172.21.15.90:0/3530972476 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1e5c108c60 0x7f1e5c10cc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:48.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.098+0000 7f1e62000640 1 --2- 172.21.15.90:0/3530972476 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1e5c1082b0 0x7f1e5c108690 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:48.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.098+0000 7f1e62000640 1 -- 172.21.15.90:0/3530972476 >> 172.21.15.90:0/3530972476 conn(0x7f1e5c077920 msgr2=0x7f1e5c106920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:50:48.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.098+0000 7f1e62000640 1 -- 172.21.15.90:0/3530972476 shutdown_connections 2024-04-02T14:50:48.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.098+0000 7f1e62000640 1 -- 172.21.15.90:0/3530972476 wait complete. 2024-04-02T14:50:48.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.099+0000 7f1e62000640 1 Processor -- start 2024-04-02T14:50:48.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.099+0000 7f1e62000640 1 -- start start 2024-04-02T14:50:48.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.099+0000 7f1e62000640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1e5c1082b0 0x7f1e5c1a4600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:48.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.099+0000 7f1e62000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1e5c10d300 0x7f1e5c1a4b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:48.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.099+0000 7f1e62000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1e5c19e710 0x7f1e5c19eb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:48.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.099+0000 7f1e62000640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f1e5c1183d0 con 0x7f1e5c10d300 2024-04-02T14:50:48.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.099+0000 7f1e62000640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f1e5c118250 con 0x7f1e5c19e710 2024-04-02T14:50:48.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.099+0000 7f1e62000640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f1e5c118550 con 0x7f1e5c1082b0 2024-04-02T14:50:48.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.100+0000 7f1e5be00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1e5c10d300 0x7f1e5c1a4b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:48.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.100+0000 7f1e5be00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1e5c10d300 0x7f1e5c1a4b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:37992/0 (socket says 172.21.15.90:37992) 2024-04-02T14:50:48.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.100+0000 7f1e5be00640 1 -- 172.21.15.90:0/2315235813 learned_addr learned my addr 172.21.15.90:0/2315235813 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:50:48.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.100+0000 7f1e60c00640 1 --2- 172.21.15.90:0/2315235813 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1e5c1082b0 0x7f1e5c1a4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:48.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.100+0000 7f1e61600640 1 --2- 172.21.15.90:0/2315235813 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1e5c19e710 0x7f1e5c19eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:48.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.100+0000 7f1e5be00640 1 -- 172.21.15.90:0/2315235813 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1e5c1082b0 msgr2=0x7f1e5c1a4600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:48.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.100+0000 7f1e5be00640 1 --2- 172.21.15.90:0/2315235813 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1e5c1082b0 0x7f1e5c1a4600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:48.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.100+0000 7f1e5be00640 1 -- 172.21.15.90:0/2315235813 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1e5c19e710 msgr2=0x7f1e5c19eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:48.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.100+0000 7f1e5be00640 1 --2- 172.21.15.90:0/2315235813 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1e5c19e710 0x7f1e5c19eb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:48.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.100+0000 7f1e5be00640 1 -- 172.21.15.90:0/2315235813 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e5c19f310 con 0x7f1e5c10d300 2024-04-02T14:50:48.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.100+0000 7f1e5be00640 1 --2- 172.21.15.90:0/2315235813 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1e5c10d300 0x7f1e5c1a4b40 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f1e4800e9c0 tx=0x7f1e4800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:50:48.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.101+0000 7f1e59600640 1 -- 172.21.15.90:0/2315235813 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e4800ccc0 con 0x7f1e5c10d300 2024-04-02T14:50:48.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.101+0000 7f1e59600640 1 -- 172.21.15.90:0/2315235813 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1e48005500 con 0x7f1e5c10d300 2024-04-02T14:50:48.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.101+0000 7f1e62000640 1 -- 172.21.15.90:0/2315235813 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e5c1ab3a0 con 0x7f1e5c10d300 2024-04-02T14:50:48.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.101+0000 7f1e59600640 1 -- 172.21.15.90:0/2315235813 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e48010690 con 0x7f1e5c10d300 2024-04-02T14:50:48.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.101+0000 7f1e62000640 1 -- 172.21.15.90:0/2315235813 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e5c1ab750 con 0x7f1e5c10d300 2024-04-02T14:50:48.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.104+0000 7f1e663d8740 1 Processor -- start 2024-04-02T14:50:48.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.104+0000 7f1e59600640 1 -- 172.21.15.90:0/2315235813 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f1e48004e80 con 0x7f1e5c10d300 2024-04-02T14:50:48.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.104+0000 7f1e663d8740 1 -- start start 2024-04-02T14:50:48.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.104+0000 7f1e663d8740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d5937daba0 0x55d5937d98f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:48.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.104+0000 7f1e663d8740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d5937d9ec0 0x55d5937da340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:48.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.104+0000 7f1e663d8740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d5937e3230 0x55d5937e5620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:48.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.104+0000 7f1e663d8740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55d59371a400 con 0x55d5937daba0 2024-04-02T14:50:48.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.104+0000 7f1e663d8740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55d5936fef10 con 0x55d5937d9ec0 2024-04-02T14:50:48.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.104+0000 7f1e663d8740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55d59373e6f0 con 0x55d5937e3230 2024-04-02T14:50:48.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.105+0000 7f1e61600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d5937daba0 0x55d5937d98f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:48.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.105+0000 7f1e61600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d5937daba0 0x55d5937d98f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:37998/0 (socket says 172.21.15.90:37998) 2024-04-02T14:50:48.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.105+0000 7f1e61600640 1 -- 172.21.15.90:0/3327011419 learned_addr learned my addr 172.21.15.90:0/3327011419 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:50:48.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.105+0000 7f1e5be00640 1 --2- 172.21.15.90:0/3327011419 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d5937e3230 0x55d5937e5620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:48.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.105+0000 7f1e60c00640 1 --2- 172.21.15.90:0/3327011419 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d5937d9ec0 0x55d5937da340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:48.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.105+0000 7f1e61600640 1 -- 172.21.15.90:0/3327011419 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d5937e3230 msgr2=0x55d5937e5620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:50:48.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.105+0000 7f1e61600640 1 --2- 172.21.15.90:0/3327011419 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d5937e3230 0x55d5937e5620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:48.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.105+0000 7f1e61600640 1 -- 172.21.15.90:0/3327011419 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d5937d9ec0 msgr2=0x55d5937da340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:50:48.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.105+0000 7f1e61600640 1 --2- 172.21.15.90:0/3327011419 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d5937d9ec0 0x55d5937da340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:48.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.105+0000 7f1e61600640 1 -- 172.21.15.90:0/3327011419 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d5934523f0 con 0x55d5937daba0 2024-04-02T14:50:48.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.105+0000 7f1e60c00640 1 --2- 172.21.15.90:0/3327011419 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d5937d9ec0 0x55d5937da340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:50:48.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.105+0000 7f1e5be00640 1 --2- 172.21.15.90:0/3327011419 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d5937e3230 0x55d5937e5620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:50:48.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.106+0000 7f1e61600640 1 --2- 172.21.15.90:0/3327011419 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d5937daba0 0x55d5937d98f0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f1e54009f60 tx=0x7f1e54009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7526748d0ac3982b server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:50:48.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.106+0000 7f1e59600640 1 --2- 172.21.15.90:0/2315235813 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f1e440a3d20 0x7f1e440a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:48.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.106+0000 7f1e59600640 1 -- 172.21.15.90:0/2315235813 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 8550+0+0 (secure 0 0 0) 0x7f1e480d0bb0 con 0x7f1e5c10d300 2024-04-02T14:50:48.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.106+0000 7f1e4dc00640 1 -- 172.21.15.90:0/3327011419 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e54016070 con 0x55d5937daba0 2024-04-02T14:50:48.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.106+0000 7f1e4dc00640 1 -- 172.21.15.90:0/3327011419 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1e54004d20 con 0x55d5937daba0 2024-04-02T14:50:48.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.106+0000 7f1e4dc00640 1 -- 172.21.15.90:0/3327011419 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e54012950 con 0x55d5937daba0 2024-04-02T14:50:48.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.106+0000 7f1e60c00640 1 --2- 172.21.15.90:0/2315235813 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f1e440a3d20 0x7f1e440a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:48.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.107+0000 7f1e663d8740 1 -- 172.21.15.90:0/3327011419 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d5937daba0 msgr2=0x55d5937d98f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:48.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.107+0000 7f1e60c00640 1 --2- 172.21.15.90:0/2315235813 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f1e440a3d20 0x7f1e440a61e0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f1e50006010 tx=0x7f1e50005dd0 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:50:48.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.107+0000 7f1e663d8740 1 --2- 172.21.15.90:0/3327011419 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d5937daba0 0x55d5937d98f0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f1e54009f60 tx=0x7f1e54009f90 comp rx=0 tx=0).stop 2024-04-02T14:50:48.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.107+0000 7f1e663d8740 1 -- 172.21.15.90:0/3327011419 shutdown_connections 2024-04-02T14:50:48.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.107+0000 7f1e663d8740 1 --2- 172.21.15.90:0/3327011419 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d5937e3230 0x55d5937e5620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:48.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.107+0000 7f1e663d8740 1 --2- 172.21.15.90:0/3327011419 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d5937d9ec0 0x55d5937da340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:48.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.107+0000 7f1e663d8740 1 --2- 172.21.15.90:0/3327011419 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d5937daba0 0x55d5937d98f0 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:48.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.107+0000 7f1e663d8740 1 -- 172.21.15.90:0/3327011419 >> 172.21.15.90:0/3327011419 conn(0x55d5937a0a50 msgr2=0x55d5937d4910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:50:48.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.107+0000 7f1e663d8740 1 -- 172.21.15.90:0/3327011419 shutdown_connections 2024-04-02T14:50:48.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.107+0000 7f1e663d8740 1 -- 172.21.15.90:0/3327011419 wait complete. 2024-04-02T14:50:48.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.107+0000 7f1e663d8740 1 -- start start 2024-04-02T14:50:48.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.108+0000 7f1e663d8740 1 Processor -- start 2024-04-02T14:50:48.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.108+0000 7f1e663d8740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d5937d9ec0 0x55d5937e01d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:48.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.108+0000 7f1e663d8740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d5937daba0 0x55d5937e0710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:48.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.108+0000 7f1e663d8740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d5937e3230 0x55d5937e0c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:48.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.108+0000 7f1e663d8740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55d59373a6a0 con 0x55d5937d9ec0 2024-04-02T14:50:48.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.108+0000 7f1e663d8740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x55d5936e6980 con 0x55d5937e3230 2024-04-02T14:50:48.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.108+0000 7f1e663d8740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55d5936f9cb0 con 0x55d5937daba0 2024-04-02T14:50:48.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.108+0000 7f1e663d8740 10 client.0 _ll_register_callbacks cb 0x55d5936f96a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:50:48.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.108+0000 7f1e5be00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d5937daba0 0x55d5937e0710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:48.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.108+0000 7f1e5be00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d5937daba0 0x55d5937e0710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:51208/0 (socket says 172.21.15.90:51208) 2024-04-02T14:50:48.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.108+0000 7f1e5be00640 1 -- 172.21.15.90:0/4055289121 learned_addr learned my addr 172.21.15.90:0/4055289121 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:50:48.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.108+0000 7f1e60c00640 1 --2- 172.21.15.90:0/4055289121 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d5937d9ec0 0x55d5937e01d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:48.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.108+0000 7f1e61600640 1 --2- 172.21.15.90:0/4055289121 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d5937e3230 0x55d5937e0c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:48.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.109+0000 7f1e5be00640 1 -- 172.21.15.90:0/4055289121 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d5937e3230 msgr2=0x55d5937e0c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:50:48.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.109+0000 7f1e5be00640 1 --2- 172.21.15.90:0/4055289121 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d5937e3230 0x55d5937e0c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:48.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.109+0000 7f1e5be00640 1 -- 172.21.15.90:0/4055289121 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d5937d9ec0 msgr2=0x55d5937e01d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:50:48.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.109+0000 7f1e5be00640 1 --2- 172.21.15.90:0/4055289121 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d5937d9ec0 0x55d5937e01d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:48.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.109+0000 7f1e5be00640 1 -- 172.21.15.90:0/4055289121 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d59351c7d0 con 0x55d5937daba0 2024-04-02T14:50:48.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.109+0000 7f1e61600640 1 --2- 172.21.15.90:0/4055289121 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d5937e3230 0x55d5937e0c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:50:48.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.109+0000 7f1e5be00640 1 --2- 172.21.15.90:0/4055289121 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d5937daba0 0x55d5937e0710 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f1e48004650 tx=0x7f1e4809c9c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c451854676c942a server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:50:48.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.109+0000 7f1e42000640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-02T14:50:48.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.109+0000 7f1e42000640 1 -- 172.21.15.90:0/4055289121 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e4809de60 con 0x55d5937daba0 2024-04-02T14:50:48.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.109+0000 7f1e42000640 1 -- 172.21.15.90:0/4055289121 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 8550+0+0 (secure 0 0 0) 0x7f1e480991a0 con 0x55d5937daba0 2024-04-02T14:50:48.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.109+0000 7f1e663d8740 1 -- 172.21.15.90:0/4055289121 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1e54004d20 con 0x55d5937daba0 2024-04-02T14:50:48.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.109+0000 7f1e42000640 1 -- 172.21.15.90:0/4055289121 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f1e4809d230 con 0x55d5937daba0 2024-04-02T14:50:48.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.109+0000 7f1e663d8740 10 client.30205 fetch_fsmap learned FSMap version 27 2024-04-02T14:50:48.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.110+0000 7f1e663d8740 1 -- 172.21.15.90:0/4055289121 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55d5937f20d0 con 0x55d5937daba0 2024-04-02T14:50:48.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.110+0000 7f1e42000640 1 -- 172.21.15.90:0/4055289121 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1e54004d20 con 0x55d5937daba0 2024-04-02T14:50:48.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.110+0000 7f1e42000640 1 -- 172.21.15.90:0/4055289121 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f1e48005170 con 0x55d5937daba0 2024-04-02T14:50:48.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.110+0000 7f1e663d8740 10 client.30205 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:50:48.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.110+0000 7f1e663d8740 10 client.30205 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:50:48.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.110+0000 7f1e663d8740 20 client.30205 populate_metadata read hostname 'smithi090' 2024-04-02T14:50:48.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.110+0000 7f1e663d8740 1 --2- 172.21.15.90:0/4055289121 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55d5937f35e0 0x55d5937f59d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:48.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.110+0000 7f1e663d8740 4 client.30205 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-02T14:50:48.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.110+0000 7f1e663d8740 1 -- 172.21.15.90:0/4055289121 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d5936fe040 con 0x55d5937f35e0 2024-04-02T14:50:48.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.110+0000 7f1e60c00640 1 --2- 172.21.15.90:0/4055289121 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55d5937f35e0 0x55d5937f59d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:48.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.111+0000 7f1e60c00640 1 --2- 172.21.15.90:0/4055289121 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55d5937f35e0 0x55d5937f59d0 crc :-1 s=READY pgs=398 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d230544afd1e26c5 server_cookie=80e512057a721433 in_seq=0 out_seq=0 2024-04-02T14:50:48.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.111+0000 7f1e42000640 10 client.30205 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:50:48.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.112+0000 7f1e42000640 1 -- 172.21.15.90:0/4055289121 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d5936fe040 con 0x55d5937f35e0 2024-04-02T14:50:48.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.112+0000 7f1e42000640 10 client.30205 handle_command_reply: tid=0 2024-04-02T14:50:48.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.112+0000 7f1e663d8740 1 client.30205 shutdown 2024-04-02T14:50:48.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.112+0000 7f1e663d8740 1 -- 172.21.15.90:0/4055289121 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d5937daba0 msgr2=0x55d5937e0710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:48.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.112+0000 7f1e663d8740 1 --2- 172.21.15.90:0/4055289121 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d5937daba0 0x55d5937e0710 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f1e48004650 tx=0x7f1e4809c9c0 comp rx=0 tx=0).stop 2024-04-02T14:50:48.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.112+0000 7f1e663d8740 1 -- 172.21.15.90:0/4055289121 shutdown_connections 2024-04-02T14:50:48.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.112+0000 7f1e663d8740 1 --2- 172.21.15.90:0/4055289121 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55d5937f35e0 0x55d5937f59d0 crc :-1 s=READY pgs=398 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:48.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.112+0000 7f1e663d8740 1 --2- 172.21.15.90:0/4055289121 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d5937e3230 0x55d5937e0c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:48.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.112+0000 7f1e663d8740 1 --2- 172.21.15.90:0/4055289121 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d5937daba0 0x55d5937e0710 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:48.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.112+0000 7f1e42000640 0 client.30205 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:50:48.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.112+0000 7f1e663d8740 1 --2- 172.21.15.90:0/4055289121 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d5937d9ec0 0x55d5937e01d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:48.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.112+0000 7f1e663d8740 1 -- 172.21.15.90:0/4055289121 >> 172.21.15.90:0/4055289121 conn(0x55d5937a0a50 msgr2=0x55d5937d6660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:50:48.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.112+0000 7f1e663d8740 1 -- 172.21.15.90:0/4055289121 shutdown_connections 2024-04-02T14:50:48.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.112+0000 7f1e663d8740 1 -- 172.21.15.90:0/4055289121 wait complete. 2024-04-02T14:50:48.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.112+0000 7f1e663d8740 20 client.30205 trim_cache size 0 max 16384 2024-04-02T14:50:48.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.127+0000 7f1e663d8740 1 Processor -- start 2024-04-02T14:50:48.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.127+0000 7f1e663d8740 1 -- start start 2024-04-02T14:50:48.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.127+0000 7f1e663d8740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d5937d9ec0 0x55d5937eeca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:48.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.127+0000 7f1e663d8740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d5937daba0 0x55d5937df980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:48.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.127+0000 7f1e663d8740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d5937e3230 0x55d5937e2000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:48.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.127+0000 7f1e663d8740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f1e5c1b3cd0 con 0x55d5937d9ec0 2024-04-02T14:50:48.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.127+0000 7f1e663d8740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f1e5c1b5810 con 0x55d5937daba0 2024-04-02T14:50:48.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.127+0000 7f1e663d8740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f1e5c1b43a0 con 0x55d5937e3230 2024-04-02T14:50:48.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.128+0000 7f1e60c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d5937d9ec0 0x55d5937eeca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:48.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.128+0000 7f1e60c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d5937d9ec0 0x55d5937eeca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38016/0 (socket says 172.21.15.90:38016) 2024-04-02T14:50:48.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.128+0000 7f1e60c00640 1 -- 172.21.15.90:0/1717888621 learned_addr learned my addr 172.21.15.90:0/1717888621 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:50:48.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.128+0000 7f1e5be00640 1 --2- 172.21.15.90:0/1717888621 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d5937daba0 0x55d5937df980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:48.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.128+0000 7f1e61600640 1 --2- 172.21.15.90:0/1717888621 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d5937e3230 0x55d5937e2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:48.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.128+0000 7f1e60c00640 1 -- 172.21.15.90:0/1717888621 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d5937e3230 msgr2=0x55d5937e2000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:50:48.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.128+0000 7f1e60c00640 1 --2- 172.21.15.90:0/1717888621 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d5937e3230 0x55d5937e2000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:48.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.128+0000 7f1e60c00640 1 -- 172.21.15.90:0/1717888621 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d5937daba0 msgr2=0x55d5937df980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:50:48.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.128+0000 7f1e60c00640 1 --2- 172.21.15.90:0/1717888621 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d5937daba0 0x55d5937df980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:48.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.128+0000 7f1e60c00640 1 -- 172.21.15.90:0/1717888621 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d5937f6030 con 0x55d5937d9ec0 2024-04-02T14:50:48.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.128+0000 7f1e60c00640 1 --2- 172.21.15.90:0/1717888621 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d5937d9ec0 0x55d5937eeca0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f1e5002c180 tx=0x7f1e5002e050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8573d274582e85f6 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:50:48.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.129+0000 7f1e4d200640 1 -- 172.21.15.90:0/1717888621 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e50004300 con 0x55d5937d9ec0 2024-04-02T14:50:48.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.129+0000 7f1e4d200640 1 -- 172.21.15.90:0/1717888621 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1e50059070 con 0x55d5937d9ec0 2024-04-02T14:50:48.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.129+0000 7f1e4d200640 1 -- 172.21.15.90:0/1717888621 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e50004ec0 con 0x55d5937d9ec0 2024-04-02T14:50:48.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.129+0000 7f1e663d8740 1 -- 172.21.15.90:0/1717888621 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d5937d9ec0 msgr2=0x55d5937eeca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:48.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.129+0000 7f1e663d8740 1 --2- 172.21.15.90:0/1717888621 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d5937d9ec0 0x55d5937eeca0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f1e5002c180 tx=0x7f1e5002e050 comp rx=0 tx=0).stop 2024-04-02T14:50:48.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.129+0000 7f1e663d8740 1 -- 172.21.15.90:0/1717888621 shutdown_connections 2024-04-02T14:50:48.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.129+0000 7f1e663d8740 1 --2- 172.21.15.90:0/1717888621 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d5937e3230 0x55d5937e2000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:48.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.129+0000 7f1e663d8740 1 --2- 172.21.15.90:0/1717888621 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d5937daba0 0x55d5937df980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:48.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.129+0000 7f1e663d8740 1 --2- 172.21.15.90:0/1717888621 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d5937d9ec0 0x55d5937eeca0 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:48.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.129+0000 7f1e663d8740 1 -- 172.21.15.90:0/1717888621 >> 172.21.15.90:0/1717888621 conn(0x55d5937a0a50 msgr2=0x55d5937e18f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:50:48.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.130+0000 7f1e663d8740 1 -- 172.21.15.90:0/1717888621 shutdown_connections 2024-04-02T14:50:48.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.130+0000 7f1e663d8740 1 -- 172.21.15.90:0/1717888621 wait complete. 2024-04-02T14:50:48.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.130+0000 7f1e663d8740 1 -- start start 2024-04-02T14:50:48.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.130+0000 7f1e663d8740 1 Processor -- start 2024-04-02T14:50:48.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.131+0000 7f1e663d8740 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d5937d9ec0 0x55d5937eb500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:48.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.131+0000 7f1e663d8740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d5937daba0 0x55d5937dd450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:48.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.131+0000 7f1e663d8740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d5937e3230 0x55d5937dd990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:48.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.131+0000 7f1e663d8740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f1e5c1b5140 con 0x55d5937daba0 2024-04-02T14:50:48.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.131+0000 7f1e663d8740 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f1e5c1b4a70 con 0x55d5937d9ec0 2024-04-02T14:50:48.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.131+0000 7f1e663d8740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f1e5c1b1cd0 con 0x55d5937e3230 2024-04-02T14:50:48.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.131+0000 7f1e663d8740 10 client.0 _ll_register_callbacks cb 0x7f1e480d4400 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-02T14:50:48.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.131+0000 7f1e5be00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d5937daba0 0x55d5937dd450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:48.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.131+0000 7f1e5be00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d5937daba0 0x55d5937dd450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38026/0 (socket says 172.21.15.90:38026) 2024-04-02T14:50:48.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.131+0000 7f1e60c00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d5937d9ec0 0x55d5937eb500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:48.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.131+0000 7f1e5be00640 1 -- 172.21.15.90:0/1356794110 learned_addr learned my addr 172.21.15.90:0/1356794110 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:50:48.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.131+0000 7f1e61600640 1 --2- 172.21.15.90:0/1356794110 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d5937e3230 0x55d5937dd990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:48.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.131+0000 7f1e5be00640 1 -- 172.21.15.90:0/1356794110 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d5937e3230 msgr2=0x55d5937dd990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:50:48.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.131+0000 7f1e5be00640 1 --2- 172.21.15.90:0/1356794110 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d5937e3230 0x55d5937dd990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:48.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.131+0000 7f1e5be00640 1 -- 172.21.15.90:0/1356794110 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d5937d9ec0 msgr2=0x55d5937eb500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-02T14:50:48.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.131+0000 7f1e5be00640 1 --2- 172.21.15.90:0/1356794110 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d5937d9ec0 0x55d5937eb500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:48.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.131+0000 7f1e5be00640 1 -- 172.21.15.90:0/1356794110 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d5937d8ef0 con 0x55d5937daba0 2024-04-02T14:50:48.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.131+0000 7f1e60c00640 1 --2- 172.21.15.90:0/1356794110 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d5937d9ec0 0x55d5937eb500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:50:48.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.132+0000 7f1e5be00640 1 --2- 172.21.15.90:0/1356794110 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d5937daba0 0x55d5937dd450 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f1e48005760 tx=0x7f1e480ce800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f07b88423a72f1e3 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:50:48.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.132+0000 7f1e42000640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-04-02T14:50:48.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.132+0000 7f1e42000640 1 -- 172.21.15.90:0/1356794110 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e4809de60 con 0x55d5937daba0 2024-04-02T14:50:48.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.132+0000 7f1e42000640 1 -- 172.21.15.90:0/1356794110 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 8550+0+0 (secure 0 0 0) 0x7f1e4809bf80 con 0x55d5937daba0 2024-04-02T14:50:48.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.132+0000 7f1e663d8740 1 -- 172.21.15.90:0/1356794110 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1e50059070 con 0x55d5937daba0 2024-04-02T14:50:48.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.132+0000 7f1e42000640 1 -- 172.21.15.90:0/1356794110 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f1e4809c490 con 0x55d5937daba0 2024-04-02T14:50:48.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.132+0000 7f1e663d8740 10 client.19092 fetch_fsmap learned FSMap version 27 2024-04-02T14:50:48.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.132+0000 7f1e663d8740 1 -- 172.21.15.90:0/1356794110 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55d5937e49a0 con 0x55d5937daba0 2024-04-02T14:50:48.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.132+0000 7f1e42000640 1 -- 172.21.15.90:0/1356794110 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1e50059070 con 0x55d5937daba0 2024-04-02T14:50:48.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.132+0000 7f1e42000640 1 -- 172.21.15.90:0/1356794110 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f1e4809b9f0 con 0x55d5937daba0 2024-04-02T14:50:48.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.132+0000 7f1e663d8740 10 client.19092 fetch_fsmap finished waiting for FSMap version 27 2024-04-02T14:50:48.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.132+0000 7f1e663d8740 10 client.19092 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-02T14:50:48.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.132+0000 7f1e663d8740 20 client.19092 populate_metadata read hostname 'smithi090' 2024-04-02T14:50:48.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.133+0000 7f1e663d8740 1 --2- 172.21.15.90:0/1356794110 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55d593859e40 0x55d59385c260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:48.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.133+0000 7f1e663d8740 4 client.19092 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-02T14:50:48.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.133+0000 7f1e663d8740 1 -- 172.21.15.90:0/1356794110 --> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55d59374bc30 con 0x55d593859e40 2024-04-02T14:50:48.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.133+0000 7f1e60c00640 1 --2- 172.21.15.90:0/1356794110 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55d593859e40 0x55d59385c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:48.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.133+0000 7f1e60c00640 1 --2- 172.21.15.90:0/1356794110 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55d593859e40 0x55d59385c260 crc :-1 s=READY pgs=399 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=81bbabe9893a7e9d server_cookie=dfc242429612e661 in_seq=0 out_seq=0 2024-04-02T14:50:48.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.133+0000 7f1e42000640 10 client.19092 ms_handle_connect on v2:172.21.15.92:6838/1054888219 2024-04-02T14:50:48.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.134+0000 7f1e42000640 1 -- 172.21.15.90:0/1356794110 <== mds.0 v2:172.21.15.92:6838/1054888219 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55d59374bc30 con 0x55d593859e40 2024-04-02T14:50:48.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.134+0000 7f1e42000640 10 client.19092 handle_command_reply: tid=0 2024-04-02T14:50:48.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.134+0000 7f1e663d8740 1 client.19092 shutdown 2024-04-02T14:50:48.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.134+0000 7f1e663d8740 1 -- 172.21.15.90:0/1356794110 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d5937daba0 msgr2=0x55d5937dd450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:48.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.134+0000 7f1e663d8740 1 --2- 172.21.15.90:0/1356794110 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d5937daba0 0x55d5937dd450 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f1e48005760 tx=0x7f1e480ce800 comp rx=0 tx=0).stop 2024-04-02T14:50:48.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.134+0000 7f1e663d8740 1 -- 172.21.15.90:0/1356794110 shutdown_connections 2024-04-02T14:50:48.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.134+0000 7f1e663d8740 1 --2- 172.21.15.90:0/1356794110 >> [v2:172.21.15.92:6838/1054888219,v1:172.21.15.92:6839/1054888219] conn(0x55d593859e40 0x55d59385c260 crc :-1 s=READY pgs=399 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:48.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.134+0000 7f1e663d8740 1 --2- 172.21.15.90:0/1356794110 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55d5937e3230 0x55d5937dd990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:48.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.134+0000 7f1e663d8740 1 --2- 172.21.15.90:0/1356794110 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55d5937daba0 0x55d5937dd450 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:48.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.134+0000 7f1e663d8740 1 --2- 172.21.15.90:0/1356794110 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55d5937d9ec0 0x55d5937eb500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:48.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.134+0000 7f1e42000640 0 client.19092 ms_handle_reset on v2:172.21.15.92:6838/1054888219 2024-04-02T14:50:48.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.134+0000 7f1e663d8740 1 -- 172.21.15.90:0/1356794110 >> 172.21.15.90:0/1356794110 conn(0x55d5937a0a50 msgr2=0x55d5937e2110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:50:48.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.134+0000 7f1e663d8740 1 -- 172.21.15.90:0/1356794110 shutdown_connections 2024-04-02T14:50:48.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.134+0000 7f1e663d8740 1 -- 172.21.15.90:0/1356794110 wait complete. 2024-04-02T14:50:48.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.134+0000 7f1e663d8740 20 client.19092 trim_cache size 0 max 16384 2024-04-02T14:50:48.132 INFO:teuthology.orchestra.run.smithi090.stdout:[] 2024-04-02T14:50:48.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.135+0000 7f1e62000640 1 -- 172.21.15.90:0/2315235813 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f1e440a3d20 msgr2=0x7f1e440a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:48.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.135+0000 7f1e62000640 1 --2- 172.21.15.90:0/2315235813 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f1e440a3d20 0x7f1e440a61e0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f1e50006010 tx=0x7f1e50005dd0 comp rx=0 tx=0).stop 2024-04-02T14:50:48.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.135+0000 7f1e62000640 1 -- 172.21.15.90:0/2315235813 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1e5c10d300 msgr2=0x7f1e5c1a4b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:48.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.135+0000 7f1e62000640 1 --2- 172.21.15.90:0/2315235813 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1e5c10d300 0x7f1e5c1a4b40 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f1e4800e9c0 tx=0x7f1e4800ee80 comp rx=0 tx=0).stop 2024-04-02T14:50:48.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.135+0000 7f1e62000640 1 -- 172.21.15.90:0/2315235813 shutdown_connections 2024-04-02T14:50:48.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.135+0000 7f1e62000640 1 --2- 172.21.15.90:0/2315235813 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f1e440a3d20 0x7f1e440a61e0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:48.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.135+0000 7f1e62000640 1 --2- 172.21.15.90:0/2315235813 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1e5c19e710 0x7f1e5c19eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:48.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.135+0000 7f1e62000640 1 --2- 172.21.15.90:0/2315235813 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1e5c10d300 0x7f1e5c1a4b40 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:48.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.135+0000 7f1e62000640 1 --2- 172.21.15.90:0/2315235813 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1e5c1082b0 0x7f1e5c1a4600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:48.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.135+0000 7f1e62000640 1 -- 172.21.15.90:0/2315235813 >> 172.21.15.90:0/2315235813 conn(0x7f1e5c077920 msgr2=0x7f1e5c079550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:50:48.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.135+0000 7f1e62000640 1 -- 172.21.15.90:0/2315235813 shutdown_connections 2024-04-02T14:50:48.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:48.135+0000 7f1e62000640 1 -- 172.21.15.90:0/2315235813 wait complete. 2024-04-02T14:50:48.430 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:50:48 smithi090 ceph-mon[31674]: pgmap v498: 97 pgs: 97 active+clean; 4.3 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 190 KiB/s rd, 15 MiB/s wr, 1.55k op/s 2024-04-02T14:50:48.602 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:50:48 smithi092 ceph-mon[36497]: pgmap v498: 97 pgs: 97 active+clean; 4.3 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 190 KiB/s rd, 15 MiB/s wr, 1.55k op/s 2024-04-02T14:50:48.678 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:50:48 smithi177 ceph-mon[36431]: pgmap v498: 97 pgs: 97 active+clean; 4.3 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 190 KiB/s rd, 15 MiB/s wr, 1.55k op/s 2024-04-02T14:50:49.147 INFO:tasks.fwd_scrub.fs.[cephfs]:end scrubbing fs: cephfs 2024-04-02T14:50:49.147 INFO:tasks.fwd_scrub:done joining 2024-04-02T14:50:49.147 DEBUG:teuthology.run_tasks:Unwinding manager check-counter 2024-04-02T14:50:49.163 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T14:50:49.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:49.393+0000 7f81b3e00640 1 -- 172.21.15.90:0/2249370442 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f81ac074850 msgr2=0x7f81ac10de20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:49.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:49.393+0000 7f81b3e00640 1 --2- 172.21.15.90:0/2249370442 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f81ac074850 0x7f81ac10de20 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f8198009900 tx=0x7f819802bcd0 comp rx=0 tx=0).stop 2024-04-02T14:50:49.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:49.393+0000 7f81b3e00640 1 -- 172.21.15.90:0/2249370442 shutdown_connections 2024-04-02T14:50:49.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:49.393+0000 7f81b3e00640 1 --2- 172.21.15.90:0/2249370442 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f81ac10e360 0x7f81ac1107b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:49.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:49.393+0000 7f81b3e00640 1 --2- 172.21.15.90:0/2249370442 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f81ac074850 0x7f81ac10de20 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:49.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:49.393+0000 7f81b3e00640 1 --2- 172.21.15.90:0/2249370442 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f81ac073f30 0x7f81ac074310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:49.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:49.393+0000 7f81b3e00640 1 -- 172.21.15.90:0/2249370442 >> 172.21.15.90:0/2249370442 conn(0x7f81ac0fd5d0 msgr2=0x7f81ac0ff9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:50:49.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:49.393+0000 7f81b3e00640 1 -- 172.21.15.90:0/2249370442 shutdown_connections 2024-04-02T14:50:49.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:49.393+0000 7f81b3e00640 1 -- 172.21.15.90:0/2249370442 wait complete. 2024-04-02T14:50:49.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:49.394+0000 7f81b3e00640 1 Processor -- start 2024-04-02T14:50:49.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:49.394+0000 7f81b3e00640 1 -- start start 2024-04-02T14:50:49.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:49.395+0000 7f81b3e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f81ac073f30 0x7f81ac19a640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:49.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:49.395+0000 7f81b3e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f81ac074850 0x7f81ac19ab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:49.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:49.395+0000 7f81b3e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f81ac10e360 0x7f81ac19ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:49.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:49.395+0000 7f81b3e00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f81ac114120 con 0x7f81ac073f30 2024-04-02T14:50:49.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:49.395+0000 7f81b3e00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f81ac113fa0 con 0x7f81ac074850 2024-04-02T14:50:49.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:49.395+0000 7f81b3e00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f81ac1142a0 con 0x7f81ac10e360 2024-04-02T14:50:49.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:49.395+0000 7f81b1400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f81ac073f30 0x7f81ac19a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:49.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:49.395+0000 7f81b1400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f81ac073f30 0x7f81ac19a640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38044/0 (socket says 172.21.15.90:38044) 2024-04-02T14:50:49.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:49.395+0000 7f81b1400640 1 -- 172.21.15.90:0/2688082082 learned_addr learned my addr 172.21.15.90:0/2688082082 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:50:49.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:49.395+0000 7f81b1e00640 1 --2- 172.21.15.90:0/2688082082 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f81ac10e360 0x7f81ac19ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:49.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:49.395+0000 7f81b0a00640 1 --2- 172.21.15.90:0/2688082082 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f81ac074850 0x7f81ac19ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:49.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:49.395+0000 7f81b1400640 1 -- 172.21.15.90:0/2688082082 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f81ac10e360 msgr2=0x7f81ac19ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:49.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:49.395+0000 7f81b1400640 1 --2- 172.21.15.90:0/2688082082 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f81ac10e360 0x7f81ac19ef00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:49.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:49.395+0000 7f81b1400640 1 -- 172.21.15.90:0/2688082082 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f81ac074850 msgr2=0x7f81ac19ab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:49.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:49.395+0000 7f81b1400640 1 --2- 172.21.15.90:0/2688082082 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f81ac074850 0x7f81ac19ab80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:49.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:49.395+0000 7f81b1400640 1 -- 172.21.15.90:0/2688082082 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81ac19f630 con 0x7f81ac073f30 2024-04-02T14:50:49.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:49.396+0000 7f81b1400640 1 --2- 172.21.15.90:0/2688082082 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f81ac073f30 0x7f81ac19a640 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f81a000d950 tx=0x7f81a000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:50:49.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:49.396+0000 7f81aa000640 1 -- 172.21.15.90:0/2688082082 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81a0017030 con 0x7f81ac073f30 2024-04-02T14:50:49.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:49.396+0000 7f81aa000640 1 -- 172.21.15.90:0/2688082082 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f81a0004020 con 0x7f81ac073f30 2024-04-02T14:50:49.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:49.396+0000 7f81b3e00640 1 -- 172.21.15.90:0/2688082082 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f81ac19f920 con 0x7f81ac073f30 2024-04-02T14:50:49.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:49.396+0000 7f81aa000640 1 -- 172.21.15.90:0/2688082082 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81a00047a0 con 0x7f81ac073f30 2024-04-02T14:50:49.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:49.396+0000 7f81b3e00640 1 -- 172.21.15.90:0/2688082082 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81ac1a71b0 con 0x7f81ac073f30 2024-04-02T14:50:49.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:49.397+0000 7f81b3e00640 1 -- 172.21.15.90:0/2688082082 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f81ac068060 con 0x7f81ac073f30 2024-04-02T14:50:49.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:49.399+0000 7f81aa000640 1 -- 172.21.15.90:0/2688082082 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f81a00041d0 con 0x7f81ac073f30 2024-04-02T14:50:49.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:49.401+0000 7f81aa000640 1 --2- 172.21.15.90:0/2688082082 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f81940a3d20 0x7f81940a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:49.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:49.401+0000 7f81aa000640 1 -- 172.21.15.90:0/2688082082 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 8550+0+0 (secure 0 0 0) 0x7f81a00d01d0 con 0x7f81ac073f30 2024-04-02T14:50:49.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:49.401+0000 7f81b0a00640 1 --2- 172.21.15.90:0/2688082082 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f81940a3d20 0x7f81940a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:49.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:49.402+0000 7f81b0a00640 1 --2- 172.21.15.90:0/2688082082 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f81940a3d20 0x7f81940a61e0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f8198005ab0 tx=0x7f8198005a20 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:50:49.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:49.405+0000 7f81aa000640 1 -- 172.21.15.90:0/2688082082 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7f81a009c7f0 con 0x7f81ac073f30 2024-04-02T14:50:49.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:49.609+0000 7f81b3e00640 1 -- 172.21.15.90:0/2688082082 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f81ac1094e0 con 0x7f81ac073f30 2024-04-02T14:50:49.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:49.611+0000 7f81aa000640 1 -- 172.21.15.90:0/2688082082 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v109) ==== 74+0+23118 (secure 0 0 0) 0x7f81a009c0f0 con 0x7f81ac073f30 2024-04-02T14:50:49.605 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-02T14:50:49.605 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":109,"fsid":"6449448a-f0fc-11ee-b647-cb9ed24678a4","created":"2024-04-02T14:23:16.530201+0000","modified":"2024-04-02T14:50:15.172960+0000","last_up_change":"2024-04-02T14:31:23.819732+0000","last_in_change":"2024-04-02T14:31:02.824284+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T14:27:16.758622+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-02T14:32:20.016331+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-02T14:32:21.294367+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"108","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":25,"snap_epoch":108,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a1a6a856-ab44-44bd-bc5e-45c718a70cd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":3635087609},{"type":"v1","addr":"172.21.15.90:6803","nonce":3635087609}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":3635087609},{"type":"v1","addr":"172.21.15.90:6805","nonce":3635087609}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":3635087609},{"type":"v1","addr":"172.21.15.90:6809","nonce":3635087609}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":3635087609},{"type":"v1","addr":"172.21.15.90:6807","nonce":3635087609}]},"public_addr":"172.21.15.90:6803/3635087609","cluster_addr":"172.21.15.90:6805/3635087609","heartbeat_back_addr":"172.21.15.90:6809/3635087609","heartbeat_front_addr":"172.21.15.90:6807/3635087609","state":["exists","up"]},{"osd":1,"uuid":"124139cd-5793-463e-9607-9d9c10164183","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":1994536059},{"type":"v1","addr":"172.21.15.92:6801","nonce":1994536059}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":1994536059},{"type":"v1","addr":"172.21.15.92:6803","nonce":1994536059}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":1994536059},{"type":"v1","addr":"172.21.15.92:6807","nonce":1994536059}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":1994536059},{"type":"v1","addr":"172.21.15.92:6805","nonce":1994536059}]},"public_addr":"172.21.15.92:6801/1994536059","cluster_addr":"172.21.15.92:6803/1994536059","heartbeat_back_addr":"172.21.15.92:6807/1994536059","heartbeat_front_addr":"172.21.15.92:6805/1994536059","state":["exists","up"]},{"osd":2,"uuid":"a61fbef2-9dca-4875-a814-020fff0db0e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":1358472800},{"type":"v1","addr":"172.21.15.177:6801","nonce":1358472800}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":1358472800},{"type":"v1","addr":"172.21.15.177:6803","nonce":1358472800}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":1358472800},{"type":"v1","addr":"172.21.15.177:6807","nonce":1358472800}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":1358472800},{"type":"v1","addr":"172.21.15.177:6805","nonce":1358472800}]},"public_addr":"172.21.15.177:6801/1358472800","cluster_addr":"172.21.15.177:6803/1358472800","heartbeat_back_addr":"172.21.15.177:6807/1358472800","heartbeat_front_addr":"172.21.15.177:6805/1358472800","state":["exists","up"]},{"osd":3,"uuid":"b175a2f4-7859-4eb0-bc1f-fba8c2b92a2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":3114811216},{"type":"v1","addr":"172.21.15.90:6811","nonce":3114811216}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":3114811216},{"type":"v1","addr":"172.21.15.90:6813","nonce":3114811216}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":3114811216},{"type":"v1","addr":"172.21.15.90:6817","nonce":3114811216}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":3114811216},{"type":"v1","addr":"172.21.15.90:6815","nonce":3114811216}]},"public_addr":"172.21.15.90:6811/3114811216","cluster_addr":"172.21.15.90:6813/3114811216","heartbeat_back_addr":"172.21.15.90:6817/3114811216","heartbeat_front_addr":"172.21.15.90:6815/3114811216","state":["exists","up"]},{"osd":4,"uuid":"38cc68ea-c109-4d8d-b77d-1c6f9b56c875","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":2987208879},{"type":"v1","addr":"172.21.15.92:6809","nonce":2987208879}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6810","nonce":2987208879},{"type":"v1","addr":"172.21.15.92:6811","nonce":2987208879}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6814","nonce":2987208879},{"type":"v1","addr":"172.21.15.92:6815","nonce":2987208879}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":2987208879},{"type":"v1","addr":"172.21.15.92:6813","nonce":2987208879}]},"public_addr":"172.21.15.92:6809/2987208879","cluster_addr":"172.21.15.92:6811/2987208879","heartbeat_back_addr":"172.21.15.92:6815/2987208879","heartbeat_front_addr":"172.21.15.92:6813/2987208879","state":["exists","up"]},{"osd":5,"uuid":"2347232e-ac91-466f-8ac9-496f397679ba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":3486820237},{"type":"v1","addr":"172.21.15.177:6809","nonce":3486820237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":3486820237},{"type":"v1","addr":"172.21.15.177:6811","nonce":3486820237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":3486820237},{"type":"v1","addr":"172.21.15.177:6815","nonce":3486820237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":3486820237},{"type":"v1","addr":"172.21.15.177:6813","nonce":3486820237}]},"public_addr":"172.21.15.177:6809/3486820237","cluster_addr":"172.21.15.177:6811/3486820237","heartbeat_back_addr":"172.21.15.177:6815/3486820237","heartbeat_front_addr":"172.21.15.177:6813/3486820237","state":["exists","up"]},{"osd":6,"uuid":"aa72392f-0498-44db-a8d2-eccc89a64871","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":2579716443},{"type":"v1","addr":"172.21.15.90:6819","nonce":2579716443}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6820","nonce":2579716443},{"type":"v1","addr":"172.21.15.90:6821","nonce":2579716443}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":2579716443},{"type":"v1","addr":"172.21.15.90:6825","nonce":2579716443}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2579716443},{"type":"v1","addr":"172.21.15.90:6823","nonce":2579716443}]},"public_addr":"172.21.15.90:6819/2579716443","cluster_addr":"172.21.15.90:6821/2579716443","heartbeat_back_addr":"172.21.15.90:6825/2579716443","heartbeat_front_addr":"172.21.15.90:6823/2579716443","state":["exists","up"]},{"osd":7,"uuid":"931bd105-bc14-4a1e-bfe4-c7ad58fa4b73","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":3371649139},{"type":"v1","addr":"172.21.15.92:6817","nonce":3371649139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":3371649139},{"type":"v1","addr":"172.21.15.92:6819","nonce":3371649139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":3371649139},{"type":"v1","addr":"172.21.15.92:6823","nonce":3371649139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6820","nonce":3371649139},{"type":"v1","addr":"172.21.15.92:6821","nonce":3371649139}]},"public_addr":"172.21.15.92:6817/3371649139","cluster_addr":"172.21.15.92:6819/3371649139","heartbeat_back_addr":"172.21.15.92:6823/3371649139","heartbeat_front_addr":"172.21.15.92:6821/3371649139","state":["exists","up"]},{"osd":8,"uuid":"82624ec2-b382-4886-9f40-7f5f376de3ce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":1459053034},{"type":"v1","addr":"172.21.15.177:6817","nonce":1459053034}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":1459053034},{"type":"v1","addr":"172.21.15.177:6819","nonce":1459053034}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":1459053034},{"type":"v1","addr":"172.21.15.177:6823","nonce":1459053034}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":1459053034},{"type":"v1","addr":"172.21.15.177:6821","nonce":1459053034}]},"public_addr":"172.21.15.177:6817/1459053034","cluster_addr":"172.21.15.177:6819/1459053034","heartbeat_back_addr":"172.21.15.177:6823/1459053034","heartbeat_front_addr":"172.21.15.177:6821/1459053034","state":["exists","up"]},{"osd":9,"uuid":"fbe36e48-3468-42b2-8094-e16552f3cb16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":2592604332},{"type":"v1","addr":"172.21.15.90:6827","nonce":2592604332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":2592604332},{"type":"v1","addr":"172.21.15.90:6829","nonce":2592604332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6832","nonce":2592604332},{"type":"v1","addr":"172.21.15.90:6833","nonce":2592604332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":2592604332},{"type":"v1","addr":"172.21.15.90:6831","nonce":2592604332}]},"public_addr":"172.21.15.90:6827/2592604332","cluster_addr":"172.21.15.90:6829/2592604332","heartbeat_back_addr":"172.21.15.90:6833/2592604332","heartbeat_front_addr":"172.21.15.90:6831/2592604332","state":["exists","up"]},{"osd":10,"uuid":"9240df75-e7e2-47f1-8aae-534b70b3c210","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":1594153142},{"type":"v1","addr":"172.21.15.92:6825","nonce":1594153142}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6826","nonce":1594153142},{"type":"v1","addr":"172.21.15.92:6827","nonce":1594153142}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":1594153142},{"type":"v1","addr":"172.21.15.92:6831","nonce":1594153142}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":1594153142},{"type":"v1","addr":"172.21.15.92:6829","nonce":1594153142}]},"public_addr":"172.21.15.92:6825/1594153142","cluster_addr":"172.21.15.92:6827/1594153142","heartbeat_back_addr":"172.21.15.92:6831/1594153142","heartbeat_front_addr":"172.21.15.92:6829/1594153142","state":["exists","up"]},{"osd":11,"uuid":"83cba9fa-b745-4df1-adda-e86837d4b91a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":83,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1622582126},{"type":"v1","addr":"172.21.15.177:6825","nonce":1622582126}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":1622582126},{"type":"v1","addr":"172.21.15.177:6827","nonce":1622582126}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":1622582126},{"type":"v1","addr":"172.21.15.177:6831","nonce":1622582126}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1622582126},{"type":"v1","addr":"172.21.15.177:6829","nonce":1622582126}]},"public_addr":"172.21.15.177:6825/1622582126","cluster_addr":"172.21.15.177:6827/1622582126","heartbeat_back_addr":"172.21.15.177:6831/1622582126","heartbeat_front_addr":"172.21.15.177:6829/1622582126","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:26:24.290365+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:26:49.405988+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:27:13.639196+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:27:45.603746+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:28:11.019126+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:28:35.084163+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:29:08.064626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:29:33.266133+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:29:57.381816+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:30:30.286847+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:30:56.226279+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:31:20.809611+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":7,"to":11}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":8,"to":4}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.11","mappings":[{"from":4,"to":8},{"from":5,"to":9}]},{"pgid":"3.21","mappings":[{"from":5,"to":9}]},{"pgid":"3.23","mappings":[{"from":4,"to":7}]},{"pgid":"3.3a","mappings":[{"from":4,"to":6}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.90:0/2805994468":"2024-04-03T14:34:52.235215+0000","172.21.15.90:0/1431307002":"2024-04-03T14:34:52.235215+0000","172.21.15.90:6801/4266054817":"2024-04-03T14:34:52.235215+0000","172.21.15.90:0/314641601":"2024-04-03T14:24:34.595561+0000","172.21.15.90:0/1400895868":"2024-04-03T14:24:34.595561+0000","172.21.15.90:6801/129436436":"2024-04-03T14:24:34.595561+0000","172.21.15.90:0/1335673697":"2024-04-03T14:34:52.235215+0000","172.21.15.90:6801/2372398598":"2024-04-03T14:24:00.362147+0000","172.21.15.90:0/278569338":"2024-04-03T14:24:34.595561+0000","172.21.15.90:0/2321464594":"2024-04-03T14:24:00.362147+0000","172.21.15.90:6800/4266054817":"2024-04-03T14:34:52.235215+0000","172.21.15.90:6800/2372398598":"2024-04-03T14:24:00.362147+0000","172.21.15.90:0/2962656672":"2024-04-03T14:24:00.362147+0000","172.21.15.90:0/126693610":"2024-04-03T14:24:00.362147+0000","172.21.15.90:0/1757385857":"2024-04-03T14:34:52.235215+0000","172.21.15.90:6800/129436436":"2024-04-03T14:24:34.595561+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":25,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T14:50:49.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:49.613+0000 7f81b3e00640 1 -- 172.21.15.90:0/2688082082 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f81940a3d20 msgr2=0x7f81940a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:49.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:49.613+0000 7f81b3e00640 1 --2- 172.21.15.90:0/2688082082 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f81940a3d20 0x7f81940a61e0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f8198005ab0 tx=0x7f8198005a20 comp rx=0 tx=0).stop 2024-04-02T14:50:49.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:49.613+0000 7f81b3e00640 1 -- 172.21.15.90:0/2688082082 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f81ac073f30 msgr2=0x7f81ac19a640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:49.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:49.613+0000 7f81b3e00640 1 --2- 172.21.15.90:0/2688082082 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f81ac073f30 0x7f81ac19a640 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f81a000d950 tx=0x7f81a000de10 comp rx=0 tx=0).stop 2024-04-02T14:50:49.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:49.613+0000 7f81b3e00640 1 -- 172.21.15.90:0/2688082082 shutdown_connections 2024-04-02T14:50:49.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:49.613+0000 7f81b3e00640 1 --2- 172.21.15.90:0/2688082082 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f81940a3d20 0x7f81940a61e0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:49.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:49.613+0000 7f81b3e00640 1 --2- 172.21.15.90:0/2688082082 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f81ac10e360 0x7f81ac19ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:49.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:49.614+0000 7f81b3e00640 1 --2- 172.21.15.90:0/2688082082 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f81ac074850 0x7f81ac19ab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:49.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:49.614+0000 7f81b3e00640 1 --2- 172.21.15.90:0/2688082082 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f81ac073f30 0x7f81ac19a640 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:49.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:49.614+0000 7f81b3e00640 1 -- 172.21.15.90:0/2688082082 >> 172.21.15.90:0/2688082082 conn(0x7f81ac0fd5d0 msgr2=0x7f81ac0fe300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:50:49.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:49.614+0000 7f81b3e00640 1 -- 172.21.15.90:0/2688082082 shutdown_connections 2024-04-02T14:50:49.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:49.614+0000 7f81b3e00640 1 -- 172.21.15.90:0/2688082082 wait complete. 2024-04-02T14:50:49.628 INFO:tasks.check_counter.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-02T14:27:16.758622+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-02T14:32:20.016331+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-02T14:32:21.294367+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '108', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 25, 'snap_epoch': 108, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-02T14:50:49.628 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-02T14:50:49.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:49.852+0000 7f2a87800640 1 -- 172.21.15.90:0/1820889543 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2a80108a80 msgr2=0x7f2a80114f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:49.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:49.852+0000 7f2a87800640 1 --2- 172.21.15.90:0/1820889543 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2a80108a80 0x7f2a80114f40 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7f2a78009ca0 tx=0x7f2a7802c180 comp rx=0 tx=0).stop 2024-04-02T14:50:49.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:49.852+0000 7f2a87800640 1 -- 172.21.15.90:0/1820889543 shutdown_connections 2024-04-02T14:50:49.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:49.852+0000 7f2a87800640 1 --2- 172.21.15.90:0/1820889543 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2a80108a80 0x7f2a80114f40 unknown :-1 s=CLOSED pgs=1046 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:49.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:49.852+0000 7f2a87800640 1 --2- 172.21.15.90:0/1820889543 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2a801080e0 0x7f2a80108540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:49.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:49.852+0000 7f2a87800640 1 --2- 172.21.15.90:0/1820889543 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2a8010e1a0 0x7f2a8010e580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:49.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:49.852+0000 7f2a87800640 1 -- 172.21.15.90:0/1820889543 >> 172.21.15.90:0/1820889543 conn(0x7f2a800778d0 msgr2=0x7f2a80106920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:50:49.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:49.852+0000 7f2a87800640 1 -- 172.21.15.90:0/1820889543 shutdown_connections 2024-04-02T14:50:49.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:49.853+0000 7f2a87800640 1 -- 172.21.15.90:0/1820889543 wait complete. 2024-04-02T14:50:49.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:49.854+0000 7f2a87800640 1 Processor -- start 2024-04-02T14:50:49.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:49.854+0000 7f2a87800640 1 -- start start 2024-04-02T14:50:49.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:49.854+0000 7f2a87800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2a801080e0 0x7f2a801a4600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:49.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:49.854+0000 7f2a87800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2a80108a80 0x7f2a801a4b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:49.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:49.854+0000 7f2a87800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2a8010e1a0 0x7f2a8019e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:49.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:49.854+0000 7f2a87800640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f2a80118480 con 0x7f2a80108a80 2024-04-02T14:50:49.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:49.854+0000 7f2a87800640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f2a80118300 con 0x7f2a8010e1a0 2024-04-02T14:50:49.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:49.854+0000 7f2a87800640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f2a80118600 con 0x7f2a801080e0 2024-04-02T14:50:49.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:49.854+0000 7f2a77e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2a80108a80 0x7f2a801a4b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:49.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:49.855+0000 7f2a77e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2a80108a80 0x7f2a801a4b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38058/0 (socket says 172.21.15.90:38058) 2024-04-02T14:50:49.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:49.855+0000 7f2a77e00640 1 -- 172.21.15.90:0/840273506 learned_addr learned my addr 172.21.15.90:0/840273506 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:50:49.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:49.855+0000 7f2a84e00640 1 --2- 172.21.15.90:0/840273506 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2a801080e0 0x7f2a801a4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:49.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:49.855+0000 7f2a85800640 1 --2- 172.21.15.90:0/840273506 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2a8010e1a0 0x7f2a8019e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:49.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:49.855+0000 7f2a77e00640 1 -- 172.21.15.90:0/840273506 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2a801080e0 msgr2=0x7f2a801a4600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:49.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:49.855+0000 7f2a77e00640 1 --2- 172.21.15.90:0/840273506 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2a801080e0 0x7f2a801a4600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:49.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:49.855+0000 7f2a77e00640 1 -- 172.21.15.90:0/840273506 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2a8010e1a0 msgr2=0x7f2a8019e6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:49.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:49.855+0000 7f2a77e00640 1 --2- 172.21.15.90:0/840273506 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2a8010e1a0 0x7f2a8019e6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:49.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:49.855+0000 7f2a77e00640 1 -- 172.21.15.90:0/840273506 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a8019ef80 con 0x7f2a80108a80 2024-04-02T14:50:49.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:49.855+0000 7f2a77e00640 1 --2- 172.21.15.90:0/840273506 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2a80108a80 0x7f2a801a4b40 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f2a6800e9f0 tx=0x7f2a6800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:50:49.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:49.855+0000 7f2a7e000640 1 -- 172.21.15.90:0/840273506 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a6800cd80 con 0x7f2a80108a80 2024-04-02T14:50:49.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:49.856+0000 7f2a7e000640 1 -- 172.21.15.90:0/840273506 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2a68005500 con 0x7f2a80108a80 2024-04-02T14:50:49.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:49.856+0000 7f2a7e000640 1 -- 172.21.15.90:0/840273506 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a68010690 con 0x7f2a80108a80 2024-04-02T14:50:49.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:49.856+0000 7f2a87800640 1 -- 172.21.15.90:0/840273506 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a8019f270 con 0x7f2a80108a80 2024-04-02T14:50:49.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:49.856+0000 7f2a87800640 1 -- 172.21.15.90:0/840273506 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a801ab580 con 0x7f2a80108a80 2024-04-02T14:50:49.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:49.857+0000 7f2a87800640 1 -- 172.21.15.90:0/840273506 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2a801097d0 con 0x7f2a80108a80 2024-04-02T14:50:49.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:49.859+0000 7f2a7e000640 1 -- 172.21.15.90:0/840273506 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f2a680046c0 con 0x7f2a80108a80 2024-04-02T14:50:49.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:49.860+0000 7f2a7e000640 1 --2- 172.21.15.90:0/840273506 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f2a640a3d20 0x7f2a640a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:49.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:49.860+0000 7f2a7e000640 1 -- 172.21.15.90:0/840273506 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 8550+0+0 (secure 0 0 0) 0x7f2a680d0c70 con 0x7f2a80108a80 2024-04-02T14:50:49.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:49.860+0000 7f2a84e00640 1 --2- 172.21.15.90:0/840273506 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f2a640a3d20 0x7f2a640a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:49.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:49.861+0000 7f2a84e00640 1 --2- 172.21.15.90:0/840273506 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f2a640a3d20 0x7f2a640a61e0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f2a70009bd0 tx=0x7f2a70005f90 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:50:49.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:49.865+0000 7f2a7e000640 1 -- 172.21.15.90:0/840273506 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7f2a6809d290 con 0x7f2a80108a80 2024-04-02T14:50:50.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:50.067+0000 7f2a87800640 1 -- 172.21.15.90:0/840273506 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f2a8019f4a0 con 0x7f2a80108a80 2024-04-02T14:50:50.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:50.068+0000 7f2a7e000640 1 -- 172.21.15.90:0/840273506 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v109) ==== 93+0+10 (secure 0 0 0) 0x7f2a6809cb90 con 0x7f2a80108a80 2024-04-02T14:50:50.061 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 1 2024-04-02T14:50:50.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:50.070+0000 7f2a87800640 1 -- 172.21.15.90:0/840273506 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f2a640a3d20 msgr2=0x7f2a640a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:50.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:50.070+0000 7f2a87800640 1 --2- 172.21.15.90:0/840273506 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f2a640a3d20 0x7f2a640a61e0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f2a70009bd0 tx=0x7f2a70005f90 comp rx=0 tx=0).stop 2024-04-02T14:50:50.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:50.070+0000 7f2a87800640 1 -- 172.21.15.90:0/840273506 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2a80108a80 msgr2=0x7f2a801a4b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:50.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:50.070+0000 7f2a87800640 1 --2- 172.21.15.90:0/840273506 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2a80108a80 0x7f2a801a4b40 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f2a6800e9f0 tx=0x7f2a6800edf0 comp rx=0 tx=0).stop 2024-04-02T14:50:50.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:50.070+0000 7f2a87800640 1 -- 172.21.15.90:0/840273506 shutdown_connections 2024-04-02T14:50:50.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:50.070+0000 7f2a87800640 1 --2- 172.21.15.90:0/840273506 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f2a640a3d20 0x7f2a640a61e0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:50.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:50.070+0000 7f2a87800640 1 --2- 172.21.15.90:0/840273506 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2a8010e1a0 0x7f2a8019e6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:50.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:50.070+0000 7f2a87800640 1 --2- 172.21.15.90:0/840273506 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2a80108a80 0x7f2a801a4b40 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:50.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:50.070+0000 7f2a87800640 1 --2- 172.21.15.90:0/840273506 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2a801080e0 0x7f2a801a4600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:50.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:50.070+0000 7f2a87800640 1 -- 172.21.15.90:0/840273506 >> 172.21.15.90:0/840273506 conn(0x7f2a800778d0 msgr2=0x7f2a80078720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:50:50.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:50.070+0000 7f2a87800640 1 -- 172.21.15.90:0/840273506 shutdown_connections 2024-04-02T14:50:50.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:50.070+0000 7f2a87800640 1 -- 172.21.15.90:0/840273506 wait complete. 2024-04-02T14:50:50.083 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-02T14:50:50.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:50.303+0000 7f55bc200640 1 -- 172.21.15.90:0/1927101159 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f55b4108310 msgr2=0x7f55b41086f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:50.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:50.303+0000 7f55bc200640 1 --2- 172.21.15.90:0/1927101159 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f55b4108310 0x7f55b41086f0 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f55a8009860 tx=0x7f55a802bb70 comp rx=0 tx=0).stop 2024-04-02T14:50:50.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:50.304+0000 7f55bc200640 1 -- 172.21.15.90:0/1927101159 shutdown_connections 2024-04-02T14:50:50.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:50.304+0000 7f55bc200640 1 --2- 172.21.15.90:0/1927101159 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f55b410d3e0 0x7f55b4114fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:50.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:50.304+0000 7f55bc200640 1 --2- 172.21.15.90:0/1927101159 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f55b4108cc0 0x7f55b410cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:50.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:50.304+0000 7f55bc200640 1 --2- 172.21.15.90:0/1927101159 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f55b4108310 0x7f55b41086f0 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:50.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:50.304+0000 7f55bc200640 1 -- 172.21.15.90:0/1927101159 >> 172.21.15.90:0/1927101159 conn(0x7f55b4076040 msgr2=0x7f55b4078460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:50:50.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:50.304+0000 7f55bc200640 1 -- 172.21.15.90:0/1927101159 shutdown_connections 2024-04-02T14:50:50.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:50.304+0000 7f55bc200640 1 -- 172.21.15.90:0/1927101159 wait complete. 2024-04-02T14:50:50.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:50.305+0000 7f55bc200640 1 Processor -- start 2024-04-02T14:50:50.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:50.305+0000 7f55bc200640 1 -- start start 2024-04-02T14:50:50.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:50.305+0000 7f55bc200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f55b4108310 0x7f55b41a4640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:50.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:50.305+0000 7f55bc200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f55b410d3e0 0x7f55b41a4b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:50.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:50.305+0000 7f55bc200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f55b419e800 0x7f55b419ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:50.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:50.305+0000 7f55bc200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f55b41184a0 con 0x7f55b410d3e0 2024-04-02T14:50:50.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:50.305+0000 7f55bc200640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f55b4118320 con 0x7f55b4108310 2024-04-02T14:50:50.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:50.305+0000 7f55bc200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f55b4118620 con 0x7f55b419e800 2024-04-02T14:50:50.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:50.306+0000 7f55b8e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f55b410d3e0 0x7f55b41a4b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:50.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:50.306+0000 7f55b9800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f55b4108310 0x7f55b41a4640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:50.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:50.306+0000 7f55ba200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f55b419e800 0x7f55b419ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:50.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:50.306+0000 7f55b8e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f55b410d3e0 0x7f55b41a4b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38074/0 (socket says 172.21.15.90:38074) 2024-04-02T14:50:50.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:50.306+0000 7f55ba200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f55b419e800 0x7f55b419ecb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:51270/0 (socket says 172.21.15.90:51270) 2024-04-02T14:50:50.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:50.306+0000 7f55b8e00640 1 -- 172.21.15.90:0/3896688807 learned_addr learned my addr 172.21.15.90:0/3896688807 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:50:50.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:50.306+0000 7f55b9800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f55b4108310 0x7f55b41a4640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:40764/0 (socket says 172.21.15.90:40764) 2024-04-02T14:50:50.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:50.306+0000 7f55b9800640 1 -- 172.21.15.90:0/3896688807 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f55b419e800 msgr2=0x7f55b419ecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:50.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:50.306+0000 7f55b9800640 1 --2- 172.21.15.90:0/3896688807 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f55b419e800 0x7f55b419ecb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:50.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:50.306+0000 7f55b9800640 1 -- 172.21.15.90:0/3896688807 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f55b410d3e0 msgr2=0x7f55b41a4b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:50.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:50.306+0000 7f55b9800640 1 --2- 172.21.15.90:0/3896688807 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f55b410d3e0 0x7f55b41a4b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:50.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:50.306+0000 7f55b9800640 1 -- 172.21.15.90:0/3896688807 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55b419f390 con 0x7f55b4108310 2024-04-02T14:50:50.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:50.306+0000 7f55ba200640 1 --2- 172.21.15.90:0/3896688807 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f55b419e800 0x7f55b419ecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:50:50.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:50.306+0000 7f55b8e00640 1 --2- 172.21.15.90:0/3896688807 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f55b410d3e0 0x7f55b41a4b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:50:50.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:50.306+0000 7f55b9800640 1 --2- 172.21.15.90:0/3896688807 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f55b4108310 0x7f55b41a4640 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7f55a8037040 tx=0x7f55a80045b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:50:50.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:50.307+0000 7f55b2000640 1 -- 172.21.15.90:0/3896688807 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55a8041020 con 0x7f55b4108310 2024-04-02T14:50:50.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:50.307+0000 7f55bc200640 1 -- 172.21.15.90:0/3896688807 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f55b419f620 con 0x7f55b4108310 2024-04-02T14:50:50.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:50.307+0000 7f55b2000640 1 -- 172.21.15.90:0/3896688807 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f55a802c9f0 con 0x7f55b4108310 2024-04-02T14:50:50.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:50.307+0000 7f55b2000640 1 -- 172.21.15.90:0/3896688807 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55a8039040 con 0x7f55b4108310 2024-04-02T14:50:50.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:50.307+0000 7f55bc200640 1 -- 172.21.15.90:0/3896688807 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55b41ab760 con 0x7f55b4108310 2024-04-02T14:50:50.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:50.308+0000 7f55bc200640 1 -- 172.21.15.90:0/3896688807 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f55b41097e0 con 0x7f55b4108310 2024-04-02T14:50:50.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:50.310+0000 7f55b2000640 1 -- 172.21.15.90:0/3896688807 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f55a8044020 con 0x7f55b4108310 2024-04-02T14:50:50.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:50.311+0000 7f55b2000640 1 --2- 172.21.15.90:0/3896688807 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f559c0a3d20 0x7f559c0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:50.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:50.311+0000 7f55b2000640 1 -- 172.21.15.90:0/3896688807 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 8550+0+0 (secure 0 0 0) 0x7f55a80f22d0 con 0x7f55b4108310 2024-04-02T14:50:50.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:50.312+0000 7f55b8e00640 1 --2- 172.21.15.90:0/3896688807 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f559c0a3d20 0x7f559c0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:50.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:50.312+0000 7f55b8e00640 1 --2- 172.21.15.90:0/3896688807 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f559c0a3d20 0x7f559c0a61e0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f55a00096c0 tx=0x7f55a0009260 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:50:50.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:50.314+0000 7f55b2000640 1 -- 172.21.15.90:0/3896688807 <== mon.2 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7f55a804b020 con 0x7f55b4108310 2024-04-02T14:50:50.430 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:50:50 smithi090 ceph-mon[31674]: pgmap v499: 97 pgs: 97 active+clean; 4.3 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 174 KiB/s rd, 13 MiB/s wr, 1.31k op/s 2024-04-02T14:50:50.430 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:50:50 smithi090 ceph-mon[31674]: from='client.? 172.21.15.90:0/2688082082' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-02T14:50:50.430 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:50:50 smithi090 ceph-mon[31674]: from='client.? 172.21.15.90:0/840273506' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-02T14:50:50.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:50.518+0000 7f55bc200640 1 -- 172.21.15.90:0/3896688807 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f55b4108cc0 con 0x7f55b4108310 2024-04-02T14:50:50.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:50.519+0000 7f55b2000640 1 -- 172.21.15.90:0/3896688807 <== mon.2 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v109) ==== 104+0+11 (secure 0 0 0) 0x7f55a80bea60 con 0x7f55b4108310 2024-04-02T14:50:50.512 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-04-02T14:50:50.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:50.521+0000 7f55bc200640 1 -- 172.21.15.90:0/3896688807 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f559c0a3d20 msgr2=0x7f559c0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:50.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:50.521+0000 7f55bc200640 1 --2- 172.21.15.90:0/3896688807 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f559c0a3d20 0x7f559c0a61e0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f55a00096c0 tx=0x7f55a0009260 comp rx=0 tx=0).stop 2024-04-02T14:50:50.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:50.521+0000 7f55bc200640 1 -- 172.21.15.90:0/3896688807 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f55b4108310 msgr2=0x7f55b41a4640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:50.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:50.521+0000 7f55bc200640 1 --2- 172.21.15.90:0/3896688807 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f55b4108310 0x7f55b41a4640 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7f55a8037040 tx=0x7f55a80045b0 comp rx=0 tx=0).stop 2024-04-02T14:50:50.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:50.522+0000 7f55bc200640 1 -- 172.21.15.90:0/3896688807 shutdown_connections 2024-04-02T14:50:50.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:50.522+0000 7f55bc200640 1 --2- 172.21.15.90:0/3896688807 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f559c0a3d20 0x7f559c0a61e0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:50.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:50.522+0000 7f55bc200640 1 --2- 172.21.15.90:0/3896688807 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f55b419e800 0x7f55b419ecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:50.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:50.522+0000 7f55bc200640 1 --2- 172.21.15.90:0/3896688807 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f55b410d3e0 0x7f55b41a4b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:50.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:50.522+0000 7f55bc200640 1 --2- 172.21.15.90:0/3896688807 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f55b4108310 0x7f55b41a4640 unknown :-1 s=CLOSED pgs=1047 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:50.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:50.522+0000 7f55bc200640 1 -- 172.21.15.90:0/3896688807 >> 172.21.15.90:0/3896688807 conn(0x7f55b4076040 msgr2=0x7f55b40783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:50:50.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:50.522+0000 7f55bc200640 1 -- 172.21.15.90:0/3896688807 shutdown_connections 2024-04-02T14:50:50.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:50.522+0000 7f55bc200640 1 -- 172.21.15.90:0/3896688807 wait complete. 2024-04-02T14:50:50.535 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-02T14:50:50.602 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:50:50 smithi092 ceph-mon[36497]: pgmap v499: 97 pgs: 97 active+clean; 4.3 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 174 KiB/s rd, 13 MiB/s wr, 1.31k op/s 2024-04-02T14:50:50.602 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:50:50 smithi092 ceph-mon[36497]: from='client.? 172.21.15.90:0/2688082082' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-02T14:50:50.602 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:50:50 smithi092 ceph-mon[36497]: from='client.? 172.21.15.90:0/840273506' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-02T14:50:50.678 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:50:50 smithi177 ceph-mon[36431]: pgmap v499: 97 pgs: 97 active+clean; 4.3 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 174 KiB/s rd, 13 MiB/s wr, 1.31k op/s 2024-04-02T14:50:50.678 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:50:50 smithi177 ceph-mon[36431]: from='client.? 172.21.15.90:0/2688082082' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-02T14:50:50.678 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:50:50 smithi177 ceph-mon[36431]: from='client.? 172.21.15.90:0/840273506' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-02T14:50:50.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:50.755+0000 7f7187800640 1 -- 172.21.15.90:0/553065784 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f718010e2d0 msgr2=0x7f7180110720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:50.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:50.755+0000 7f7187800640 1 --2- 172.21.15.90:0/553065784 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f718010e2d0 0x7f7180110720 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f7178009ca0 tx=0x7f717802c180 comp rx=0 tx=0).stop 2024-04-02T14:50:50.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:50.755+0000 7f7187800640 1 -- 172.21.15.90:0/553065784 shutdown_connections 2024-04-02T14:50:50.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:50.755+0000 7f7187800640 1 --2- 172.21.15.90:0/553065784 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f718010e2d0 0x7f7180110720 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:50.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:50.755+0000 7f7187800640 1 --2- 172.21.15.90:0/553065784 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7180074850 0x7f718010dd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:50.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:50.755+0000 7f7187800640 1 --2- 172.21.15.90:0/553065784 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7180073f30 0x7f7180074310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:50.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:50.755+0000 7f7187800640 1 -- 172.21.15.90:0/553065784 >> 172.21.15.90:0/553065784 conn(0x7f7180078f00 msgr2=0x7f7180079310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:50:50.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:50.755+0000 7f7187800640 1 -- 172.21.15.90:0/553065784 shutdown_connections 2024-04-02T14:50:50.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:50.755+0000 7f7187800640 1 -- 172.21.15.90:0/553065784 wait complete. 2024-04-02T14:50:50.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:50.756+0000 7f7187800640 1 Processor -- start 2024-04-02T14:50:50.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:50.756+0000 7f7187800640 1 -- start start 2024-04-02T14:50:50.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:50.756+0000 7f7187800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7180073f30 0x7f718019a560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:50.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:50.756+0000 7f7187800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7180074850 0x7f718019aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:50.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:50.756+0000 7f7187800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f718010e2d0 0x7f718019ee20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:50.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:50.756+0000 7f7187800640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f7180114060 con 0x7f7180073f30 2024-04-02T14:50:50.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:50.756+0000 7f7187800640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f7180113ee0 con 0x7f718010e2d0 2024-04-02T14:50:50.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:50.756+0000 7f7187800640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f71801141e0 con 0x7f7180074850 2024-04-02T14:50:50.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:50.757+0000 7f7184e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7180073f30 0x7f718019a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:50.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:50.757+0000 7f7177e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7180074850 0x7f718019aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:50.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:50.757+0000 7f7184e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7180073f30 0x7f718019a560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38096/0 (socket says 172.21.15.90:38096) 2024-04-02T14:50:50.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:50.757+0000 7f7177e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7180074850 0x7f718019aaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:51276/0 (socket says 172.21.15.90:51276) 2024-04-02T14:50:50.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:50.757+0000 7f7184e00640 1 -- 172.21.15.90:0/865775243 learned_addr learned my addr 172.21.15.90:0/865775243 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:50:50.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:50.757+0000 7f7185800640 1 --2- 172.21.15.90:0/865775243 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f718010e2d0 0x7f718019ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:50.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:50.757+0000 7f7184e00640 1 -- 172.21.15.90:0/865775243 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7180074850 msgr2=0x7f718019aaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:50.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:50.757+0000 7f7184e00640 1 --2- 172.21.15.90:0/865775243 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7180074850 0x7f718019aaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:50.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:50.757+0000 7f7184e00640 1 -- 172.21.15.90:0/865775243 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f718010e2d0 msgr2=0x7f718019ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:50.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:50.757+0000 7f7184e00640 1 --2- 172.21.15.90:0/865775243 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f718010e2d0 0x7f718019ee20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:50.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:50.757+0000 7f7184e00640 1 -- 172.21.15.90:0/865775243 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f718019f550 con 0x7f7180073f30 2024-04-02T14:50:50.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:50.757+0000 7f7177e00640 1 --2- 172.21.15.90:0/865775243 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7180074850 0x7f718019aaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:50:50.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:50.757+0000 7f7184e00640 1 --2- 172.21.15.90:0/865775243 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7180073f30 0x7f718019a560 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f717000e9f0 tx=0x7f717000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:50:50.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:50.758+0000 7f717e000640 1 -- 172.21.15.90:0/865775243 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f717000cd80 con 0x7f7180073f30 2024-04-02T14:50:50.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:50.758+0000 7f717e000640 1 -- 172.21.15.90:0/865775243 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7170010910 con 0x7f7180073f30 2024-04-02T14:50:50.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:50.758+0000 7f7187800640 1 -- 172.21.15.90:0/865775243 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f718019f840 con 0x7f7180073f30 2024-04-02T14:50:50.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:50.758+0000 7f717e000640 1 -- 172.21.15.90:0/865775243 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71700188f0 con 0x7f7180073f30 2024-04-02T14:50:50.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:50.758+0000 7f7187800640 1 -- 172.21.15.90:0/865775243 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f71801a70d0 con 0x7f7180073f30 2024-04-02T14:50:50.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:50.759+0000 7f7187800640 1 -- 172.21.15.90:0/865775243 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7180068d40 con 0x7f7180073f30 2024-04-02T14:50:50.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:50.761+0000 7f717e000640 1 -- 172.21.15.90:0/865775243 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f7170010ac0 con 0x7f7180073f30 2024-04-02T14:50:50.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:50.763+0000 7f717e000640 1 --2- 172.21.15.90:0/865775243 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f71640a3d20 0x7f71640a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:50.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:50.763+0000 7f717e000640 1 -- 172.21.15.90:0/865775243 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 8550+0+0 (secure 0 0 0) 0x7f71700d0c10 con 0x7f7180073f30 2024-04-02T14:50:50.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:50.763+0000 7f7177e00640 1 --2- 172.21.15.90:0/865775243 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f71640a3d20 0x7f71640a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:50.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:50.764+0000 7f7177e00640 1 --2- 172.21.15.90:0/865775243 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f71640a3d20 0x7f71640a61e0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f7180101400 tx=0x7f7168005f90 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:50:50.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:50.768+0000 7f717e000640 1 -- 172.21.15.90:0/865775243 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7f717009cbe0 con 0x7f7180073f30 2024-04-02T14:50:50.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:50.957+0000 7f7187800640 1 -- 172.21.15.90:0/865775243 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f718002d370 con 0x7f7180073f30 2024-04-02T14:50:50.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:50.958+0000 7f717e000640 1 -- 172.21.15.90:0/865775243 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v109) ==== 100+0+11 (secure 0 0 0) 0x7f71700a0ee0 con 0x7f7180073f30 2024-04-02T14:50:50.951 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 64 2024-04-02T14:50:50.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:50.960+0000 7f7187800640 1 -- 172.21.15.90:0/865775243 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f71640a3d20 msgr2=0x7f71640a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:50.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:50.960+0000 7f7187800640 1 --2- 172.21.15.90:0/865775243 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f71640a3d20 0x7f71640a61e0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f7180101400 tx=0x7f7168005f90 comp rx=0 tx=0).stop 2024-04-02T14:50:50.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:50.960+0000 7f7187800640 1 -- 172.21.15.90:0/865775243 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7180073f30 msgr2=0x7f718019a560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:50.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:50.960+0000 7f7187800640 1 --2- 172.21.15.90:0/865775243 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7180073f30 0x7f718019a560 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f717000e9f0 tx=0x7f717000eeb0 comp rx=0 tx=0).stop 2024-04-02T14:50:50.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:50.961+0000 7f7187800640 1 -- 172.21.15.90:0/865775243 shutdown_connections 2024-04-02T14:50:50.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:50.961+0000 7f7187800640 1 --2- 172.21.15.90:0/865775243 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f71640a3d20 0x7f71640a61e0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:50.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:50.961+0000 7f7187800640 1 --2- 172.21.15.90:0/865775243 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f718010e2d0 0x7f718019ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:50.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:50.961+0000 7f7187800640 1 --2- 172.21.15.90:0/865775243 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7180074850 0x7f718019aaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:50.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:50.961+0000 7f7187800640 1 --2- 172.21.15.90:0/865775243 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7180073f30 0x7f718019a560 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:50.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:50.961+0000 7f7187800640 1 -- 172.21.15.90:0/865775243 >> 172.21.15.90:0/865775243 conn(0x7f7180078f00 msgr2=0x7f71800fe9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:50:50.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:50.961+0000 7f7187800640 1 -- 172.21.15.90:0/865775243 shutdown_connections 2024-04-02T14:50:50.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:50.961+0000 7f7187800640 1 -- 172.21.15.90:0/865775243 wait complete. 2024-04-02T14:50:50.973 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json-pretty 2024-04-02T14:50:51.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:51.180+0000 7f3ec9a00640 1 -- 172.21.15.90:0/2803722477 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3ec40747d0 msgr2=0x7f3ec410de40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:51.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:51.180+0000 7f3ec9a00640 1 --2- 172.21.15.90:0/2803722477 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3ec40747d0 0x7f3ec410de40 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f3eb0009ca0 tx=0x7f3eb002c180 comp rx=0 tx=0).stop 2024-04-02T14:50:51.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:51.180+0000 7f3ec9a00640 1 -- 172.21.15.90:0/2803722477 shutdown_connections 2024-04-02T14:50:51.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:51.180+0000 7f3ec9a00640 1 --2- 172.21.15.90:0/2803722477 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3ec410e380 0x7f3ec4110770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:51.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:51.180+0000 7f3ec9a00640 1 --2- 172.21.15.90:0/2803722477 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3ec40747d0 0x7f3ec410de40 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:51.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:51.180+0000 7f3ec9a00640 1 --2- 172.21.15.90:0/2803722477 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3ec4073eb0 0x7f3ec4074290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:51.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:51.180+0000 7f3ec9a00640 1 -- 172.21.15.90:0/2803722477 >> 172.21.15.90:0/2803722477 conn(0x7f3ec40fd590 msgr2=0x7f3ec40ff9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:50:51.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:51.180+0000 7f3ec9a00640 1 -- 172.21.15.90:0/2803722477 shutdown_connections 2024-04-02T14:50:51.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:51.180+0000 7f3ec9a00640 1 -- 172.21.15.90:0/2803722477 wait complete. 2024-04-02T14:50:51.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:51.181+0000 7f3ec9a00640 1 Processor -- start 2024-04-02T14:50:51.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:51.181+0000 7f3ec9a00640 1 -- start start 2024-04-02T14:50:51.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:51.182+0000 7f3ec9a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3ec4073eb0 0x7f3ec419a630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:51.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:51.182+0000 7f3ec9a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3ec40747d0 0x7f3ec419ab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:51.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:51.182+0000 7f3ec9a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3ec410e380 0x7f3ec419eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:51.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:51.182+0000 7f3ec9a00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f3ec4114060 con 0x7f3ec4073eb0 2024-04-02T14:50:51.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:51.182+0000 7f3ec9a00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f3ec4113ee0 con 0x7f3ec40747d0 2024-04-02T14:50:51.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:51.182+0000 7f3ec9a00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f3ec41141e0 con 0x7f3ec410e380 2024-04-02T14:50:51.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:51.182+0000 7f3ec2a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3ec4073eb0 0x7f3ec419a630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:51.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:51.182+0000 7f3ec2a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3ec4073eb0 0x7f3ec419a630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38118/0 (socket says 172.21.15.90:38118) 2024-04-02T14:50:51.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:51.182+0000 7f3ec2000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3ec40747d0 0x7f3ec419ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:51.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:51.182+0000 7f3ec2a00640 1 -- 172.21.15.90:0/1181514577 learned_addr learned my addr 172.21.15.90:0/1181514577 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:50:51.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:51.182+0000 7f3ec3400640 1 --2- 172.21.15.90:0/1181514577 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3ec410e380 0x7f3ec419eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:51.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:51.182+0000 7f3ec2a00640 1 -- 172.21.15.90:0/1181514577 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3ec410e380 msgr2=0x7f3ec419eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:51.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:51.182+0000 7f3ec2a00640 1 --2- 172.21.15.90:0/1181514577 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3ec410e380 0x7f3ec419eef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:51.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:51.182+0000 7f3ec2a00640 1 -- 172.21.15.90:0/1181514577 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3ec40747d0 msgr2=0x7f3ec419ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:51.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:51.182+0000 7f3ec2a00640 1 --2- 172.21.15.90:0/1181514577 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3ec40747d0 0x7f3ec419ab70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:51.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:51.182+0000 7f3ec2a00640 1 -- 172.21.15.90:0/1181514577 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ec419f620 con 0x7f3ec4073eb0 2024-04-02T14:50:51.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:51.182+0000 7f3ec3400640 1 --2- 172.21.15.90:0/1181514577 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3ec410e380 0x7f3ec419eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:50:51.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:51.183+0000 7f3ec2a00640 1 --2- 172.21.15.90:0/1181514577 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3ec4073eb0 0x7f3ec419a630 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f3eb400ea60 tx=0x7f3eb400ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:50:51.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:51.183+0000 7f3ebb400640 1 -- 172.21.15.90:0/1181514577 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3eb400cde0 con 0x7f3ec4073eb0 2024-04-02T14:50:51.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:51.183+0000 7f3ec9a00640 1 -- 172.21.15.90:0/1181514577 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ec419f910 con 0x7f3ec4073eb0 2024-04-02T14:50:51.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:51.183+0000 7f3ebb400640 1 -- 172.21.15.90:0/1181514577 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3eb40035f0 con 0x7f3ec4073eb0 2024-04-02T14:50:51.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:51.183+0000 7f3ebb400640 1 -- 172.21.15.90:0/1181514577 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3eb4010670 con 0x7f3ec4073eb0 2024-04-02T14:50:51.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:51.183+0000 7f3ec9a00640 1 -- 172.21.15.90:0/1181514577 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ec41a71a0 con 0x7f3ec4073eb0 2024-04-02T14:50:51.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:51.184+0000 7f3ec9a00640 1 -- 172.21.15.90:0/1181514577 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3ec40688f0 con 0x7f3ec4073eb0 2024-04-02T14:50:51.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:51.186+0000 7f3ebb400640 1 -- 172.21.15.90:0/1181514577 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f3eb40037a0 con 0x7f3ec4073eb0 2024-04-02T14:50:51.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:51.187+0000 7f3ebb400640 1 --2- 172.21.15.90:0/1181514577 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f3ea80a3d20 0x7f3ea80a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:50:51.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:51.187+0000 7f3ebb400640 1 -- 172.21.15.90:0/1181514577 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 8550+0+0 (secure 0 0 0) 0x7f3eb40d0bf0 con 0x7f3ec4073eb0 2024-04-02T14:50:51.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:51.188+0000 7f3ec2000640 1 --2- 172.21.15.90:0/1181514577 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f3ea80a3d20 0x7f3ea80a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:50:51.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:51.188+0000 7f3ec2000640 1 --2- 172.21.15.90:0/1181514577 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f3ea80a3d20 0x7f3ea80a61e0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f3eb0004af0 tx=0x7f3eb0005c50 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:50:51.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:51.192+0000 7f3ebb400640 1 -- 172.21.15.90:0/1181514577 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7f3eb409d210 con 0x7f3ec4073eb0 2024-04-02T14:50:51.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:51.441+0000 7f3ec9a00640 1 -- 172.21.15.90:0/1181514577 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json-pretty"} v 0) -- 0x7f3ec4109560 con 0x7f3ec4073eb0 2024-04-02T14:50:51.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:51.458+0000 7f3ebb400640 1 -- 172.21.15.90:0/1181514577 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json-pretty"}]=0 v26) ==== 81+0+746059 (secure 0 0 0) 0x7f3eb409cb10 con 0x7f3ec4073eb0 2024-04-02T14:50:51.462 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-02T14:50:51.462 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-02T14:50:51.462 INFO:teuthology.orchestra.run.smithi090.stdout: "epoch": 26, 2024-04-02T14:50:51.462 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.463 INFO:teuthology.orchestra.run.smithi090.stdout: "active_gid": 25289, 2024-04-02T14:50:51.463 INFO:teuthology.orchestra.run.smithi090.stdout: "active_name": "x", 2024-04-02T14:50:51.463 INFO:teuthology.orchestra.run.smithi090.stdout: "active_addrs": { 2024-04-02T14:50:51.463 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-02T14:50:51.463 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-02T14:50:51.463 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2024-04-02T14:50:51.463 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6800", 2024-04-02T14:50:51.463 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 3904993754 2024-04-02T14:50:51.463 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.463 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-02T14:50:51.463 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-02T14:50:51.463 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6801", 2024-04-02T14:50:51.463 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 3904993754 2024-04-02T14:50:51.463 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:51.464 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-02T14:50:51.464 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.464 INFO:teuthology.orchestra.run.smithi090.stdout: "active_addr": "172.21.15.90:6801/3904993754", 2024-04-02T14:50:51.464 INFO:teuthology.orchestra.run.smithi090.stdout: "active_change": "2024-04-02T14:34:52.235668+0000", 2024-04-02T14:50:51.464 INFO:teuthology.orchestra.run.smithi090.stdout: "active_mgr_features": 4540701547738038271, 2024-04-02T14:50:51.464 INFO:teuthology.orchestra.run.smithi090.stdout: "available": true, 2024-04-02T14:50:51.464 INFO:teuthology.orchestra.run.smithi090.stdout: "standbys": [ 2024-04-02T14:50:51.464 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-02T14:50:51.464 INFO:teuthology.orchestra.run.smithi090.stdout: "gid": 25264, 2024-04-02T14:50:51.464 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "z", 2024-04-02T14:50:51.464 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr_features": 4540701547738038271, 2024-04-02T14:50:51.464 INFO:teuthology.orchestra.run.smithi090.stdout: "available_modules": [ 2024-04-02T14:50:51.464 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-02T14:50:51.464 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "alerts", 2024-04-02T14:50:51.464 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-02T14:50:51.465 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-02T14:50:51.465 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-02T14:50:51.465 INFO:teuthology.orchestra.run.smithi090.stdout: "interval": { 2024-04-02T14:50:51.465 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "interval", 2024-04-02T14:50:51.465 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "secs", 2024-04-02T14:50:51.465 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.465 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.465 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "60", 2024-04-02T14:50:51.465 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.465 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.465 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.465 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "How frequently to reexamine health status", 2024-04-02T14:50:51.465 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.465 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.465 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.466 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.466 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-02T14:50:51.466 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-02T14:50:51.466 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.466 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.466 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.466 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:51.466 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.466 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.466 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:51.466 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:51.466 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:51.466 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:51.466 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:51.466 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:51.467 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:51.467 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:51.467 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.467 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.467 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.467 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.467 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.467 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-02T14:50:51.467 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-02T14:50:51.467 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:51.467 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.467 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.467 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:51.467 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.467 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.467 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.468 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.468 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.468 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.468 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.468 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.468 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-02T14:50:51.468 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-02T14:50:51.468 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.468 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.468 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.468 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-02T14:50:51.468 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.468 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.468 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:51.468 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:51.468 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:51.469 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:51.469 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:51.469 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:51.469 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:51.469 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:51.469 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.469 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.469 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.469 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.469 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.469 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-02T14:50:51.469 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-02T14:50:51.469 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:51.469 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.469 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.469 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:51.470 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.470 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.470 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.470 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.470 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.470 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.470 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.470 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.470 INFO:teuthology.orchestra.run.smithi090.stdout: "smtp_destination": { 2024-04-02T14:50:51.470 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "smtp_destination", 2024-04-02T14:50:51.470 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.470 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.470 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.470 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:51.470 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.471 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.471 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.471 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Email address to send alerts to", 2024-04-02T14:50:51.471 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.471 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.471 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.471 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.471 INFO:teuthology.orchestra.run.smithi090.stdout: "smtp_from_name": { 2024-04-02T14:50:51.471 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "smtp_from_name", 2024-04-02T14:50:51.471 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.471 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.471 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.471 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "Ceph", 2024-04-02T14:50:51.471 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.472 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.472 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.472 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Email From: name", 2024-04-02T14:50:51.472 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.472 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.472 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.472 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.472 INFO:teuthology.orchestra.run.smithi090.stdout: "smtp_host": { 2024-04-02T14:50:51.472 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "smtp_host", 2024-04-02T14:50:51.472 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.472 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.472 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.472 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:51.472 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.472 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.473 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.473 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "SMTP server", 2024-04-02T14:50:51.473 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.473 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.473 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.473 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.473 INFO:teuthology.orchestra.run.smithi090.stdout: "smtp_password": { 2024-04-02T14:50:51.473 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "smtp_password", 2024-04-02T14:50:51.473 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.473 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.473 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.473 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:51.473 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.473 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.473 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.474 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Password to authenticate with", 2024-04-02T14:50:51.474 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.474 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.474 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.474 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.474 INFO:teuthology.orchestra.run.smithi090.stdout: "smtp_port": { 2024-04-02T14:50:51.474 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "smtp_port", 2024-04-02T14:50:51.474 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-02T14:50:51.474 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.474 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.474 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "465", 2024-04-02T14:50:51.474 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.474 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.474 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.474 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "SMTP port", 2024-04-02T14:50:51.475 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.475 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.475 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.475 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.475 INFO:teuthology.orchestra.run.smithi090.stdout: "smtp_sender": { 2024-04-02T14:50:51.475 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "smtp_sender", 2024-04-02T14:50:51.475 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.475 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.475 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.475 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:51.475 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.475 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.475 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.475 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "SMTP envelope sender", 2024-04-02T14:50:51.475 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.475 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.476 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.476 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.476 INFO:teuthology.orchestra.run.smithi090.stdout: "smtp_ssl": { 2024-04-02T14:50:51.476 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "smtp_ssl", 2024-04-02T14:50:51.476 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:51.476 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.476 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.476 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-02T14:50:51.476 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.476 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.476 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.476 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Use SSL to connect to SMTP server", 2024-04-02T14:50:51.476 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.477 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.477 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.477 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.477 INFO:teuthology.orchestra.run.smithi090.stdout: "smtp_user": { 2024-04-02T14:50:51.477 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "smtp_user", 2024-04-02T14:50:51.477 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.477 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.477 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.477 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:51.477 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.477 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.477 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.477 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "User to authenticate as", 2024-04-02T14:50:51.477 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.478 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.478 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.478 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:51.478 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:51.478 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.478 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-02T14:50:51.478 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "balancer", 2024-04-02T14:50:51.478 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-02T14:50:51.478 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-02T14:50:51.478 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-02T14:50:51.478 INFO:teuthology.orchestra.run.smithi090.stdout: "active": { 2024-04-02T14:50:51.478 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "active", 2024-04-02T14:50:51.478 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:51.478 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.479 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.479 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-02T14:50:51.479 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.479 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.479 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.479 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "automatically balance PGs across cluster", 2024-04-02T14:50:51.479 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.479 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.479 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.479 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.479 INFO:teuthology.orchestra.run.smithi090.stdout: "begin_time": { 2024-04-02T14:50:51.479 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "begin_time", 2024-04-02T14:50:51.479 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.479 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.479 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.480 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "0000", 2024-04-02T14:50:51.480 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.480 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.480 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.480 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "beginning time of day to automatically balance", 2024-04-02T14:50:51.480 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-02T14:50:51.480 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.480 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.480 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.480 INFO:teuthology.orchestra.run.smithi090.stdout: "begin_weekday": { 2024-04-02T14:50:51.480 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "begin_weekday", 2024-04-02T14:50:51.480 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "uint", 2024-04-02T14:50:51.480 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.480 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.480 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "0", 2024-04-02T14:50:51.481 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "0", 2024-04-02T14:50:51.481 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "6", 2024-04-02T14:50:51.481 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.481 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-04-02T14:50:51.481 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-02T14:50:51.481 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.481 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.481 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.481 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_compat_max_iterations": { 2024-04-02T14:50:51.481 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "crush_compat_max_iterations", 2024-04-02T14:50:51.481 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "uint", 2024-04-02T14:50:51.481 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.481 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.481 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "25", 2024-04-02T14:50:51.481 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "1", 2024-04-02T14:50:51.481 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "250", 2024-04-02T14:50:51.481 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.482 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-04-02T14:50:51.482 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.482 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.482 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.482 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.482 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_compat_metrics": { 2024-04-02T14:50:51.482 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "crush_compat_metrics", 2024-04-02T14:50:51.482 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.482 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.482 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.482 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "pgs,objects,bytes", 2024-04-02T14:50:51.482 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.482 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.482 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.483 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-04-02T14:50:51.483 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-04-02T14:50:51.483 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.483 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.483 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.483 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_compat_step": { 2024-04-02T14:50:51.483 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "crush_compat_step", 2024-04-02T14:50:51.483 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "float", 2024-04-02T14:50:51.483 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.483 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.483 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "0.5", 2024-04-02T14:50:51.483 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "0.001", 2024-04-02T14:50:51.483 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "0.999", 2024-04-02T14:50:51.483 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.483 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "aggressiveness of optimization", 2024-04-02T14:50:51.483 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-04-02T14:50:51.484 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.484 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.484 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.484 INFO:teuthology.orchestra.run.smithi090.stdout: "end_time": { 2024-04-02T14:50:51.484 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "end_time", 2024-04-02T14:50:51.484 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.484 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.484 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.484 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "2359", 2024-04-02T14:50:51.484 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.484 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.484 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.484 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "ending time of day to automatically balance", 2024-04-02T14:50:51.484 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-02T14:50:51.484 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.484 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.485 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.485 INFO:teuthology.orchestra.run.smithi090.stdout: "end_weekday": { 2024-04-02T14:50:51.485 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "end_weekday", 2024-04-02T14:50:51.485 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "uint", 2024-04-02T14:50:51.485 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.485 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.485 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "0", 2024-04-02T14:50:51.485 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "0", 2024-04-02T14:50:51.485 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "6", 2024-04-02T14:50:51.485 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.485 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-04-02T14:50:51.485 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-02T14:50:51.485 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.485 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.485 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.486 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-02T14:50:51.486 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-02T14:50:51.486 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.486 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.486 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.486 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:51.486 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.486 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.486 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:51.486 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:51.486 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:51.486 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:51.486 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:51.486 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:51.487 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:51.487 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:51.487 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.487 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.487 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.487 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.487 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.487 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-02T14:50:51.487 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-02T14:50:51.487 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:51.487 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.487 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.487 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:51.487 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.488 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.488 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.488 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.488 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.488 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.488 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.488 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.488 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-02T14:50:51.488 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-02T14:50:51.488 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.488 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.488 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.488 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-02T14:50:51.488 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.488 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.488 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:51.489 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:51.489 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:51.489 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:51.489 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:51.489 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:51.489 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:51.489 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:51.489 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.489 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.489 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.489 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.489 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.489 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-02T14:50:51.489 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-02T14:50:51.489 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:51.490 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.490 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.490 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:51.490 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.490 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.490 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.490 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.490 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.490 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.490 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.490 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.490 INFO:teuthology.orchestra.run.smithi090.stdout: "min_score": { 2024-04-02T14:50:51.490 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "min_score", 2024-04-02T14:50:51.490 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "float", 2024-04-02T14:50:51.491 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.491 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.491 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "0", 2024-04-02T14:50:51.491 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.491 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.491 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.491 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-04-02T14:50:51.491 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.491 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.491 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.491 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.491 INFO:teuthology.orchestra.run.smithi090.stdout: "mode": { 2024-04-02T14:50:51.491 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "mode", 2024-04-02T14:50:51.491 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.491 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.492 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.492 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "upmap", 2024-04-02T14:50:51.492 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.492 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.492 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:51.492 INFO:teuthology.orchestra.run.smithi090.stdout: "crush-compat", 2024-04-02T14:50:51.492 INFO:teuthology.orchestra.run.smithi090.stdout: "none", 2024-04-02T14:50:51.492 INFO:teuthology.orchestra.run.smithi090.stdout: "read", 2024-04-02T14:50:51.492 INFO:teuthology.orchestra.run.smithi090.stdout: "upmap", 2024-04-02T14:50:51.492 INFO:teuthology.orchestra.run.smithi090.stdout: "upmap-read" 2024-04-02T14:50:51.492 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:51.492 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Balancer mode", 2024-04-02T14:50:51.492 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.492 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.493 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.493 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.493 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_ids": { 2024-04-02T14:50:51.493 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "pool_ids", 2024-04-02T14:50:51.493 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.493 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.493 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.493 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:51.493 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.493 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.493 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.493 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-04-02T14:50:51.493 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.493 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.493 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.493 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.494 INFO:teuthology.orchestra.run.smithi090.stdout: "sleep_interval": { 2024-04-02T14:50:51.494 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "sleep_interval", 2024-04-02T14:50:51.494 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "secs", 2024-04-02T14:50:51.494 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.494 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.494 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "60", 2024-04-02T14:50:51.494 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.494 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.494 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.494 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-04-02T14:50:51.494 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.494 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.494 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.494 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.495 INFO:teuthology.orchestra.run.smithi090.stdout: "upmap_max_deviation": { 2024-04-02T14:50:51.495 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "upmap_max_deviation", 2024-04-02T14:50:51.495 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-02T14:50:51.495 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.495 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.495 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "5", 2024-04-02T14:50:51.495 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "1", 2024-04-02T14:50:51.495 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.495 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.495 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "deviation below which no optimization is attempted", 2024-04-02T14:50:51.495 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-04-02T14:50:51.495 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.495 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.495 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.495 INFO:teuthology.orchestra.run.smithi090.stdout: "upmap_max_optimizations": { 2024-04-02T14:50:51.495 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "upmap_max_optimizations", 2024-04-02T14:50:51.496 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "uint", 2024-04-02T14:50:51.496 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.496 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.496 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "10", 2024-04-02T14:50:51.496 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.496 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.496 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.496 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-04-02T14:50:51.496 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.496 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.496 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.496 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:51.496 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:51.496 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.496 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-02T14:50:51.496 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "cephadm", 2024-04-02T14:50:51.497 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-02T14:50:51.497 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-02T14:50:51.497 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-02T14:50:51.497 INFO:teuthology.orchestra.run.smithi090.stdout: "agent_down_multiplier": { 2024-04-02T14:50:51.497 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "agent_down_multiplier", 2024-04-02T14:50:51.497 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "float", 2024-04-02T14:50:51.497 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.497 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.497 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "3.0", 2024-04-02T14:50:51.497 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.497 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.497 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.497 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-04-02T14:50:51.497 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.497 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.497 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.498 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.498 INFO:teuthology.orchestra.run.smithi090.stdout: "agent_refresh_rate": { 2024-04-02T14:50:51.498 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "agent_refresh_rate", 2024-04-02T14:50:51.498 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "secs", 2024-04-02T14:50:51.498 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.498 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.498 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "20", 2024-04-02T14:50:51.498 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.498 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.498 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.498 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-04-02T14:50:51.498 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.498 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.498 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.498 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.499 INFO:teuthology.orchestra.run.smithi090.stdout: "agent_starting_port": { 2024-04-02T14:50:51.499 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "agent_starting_port", 2024-04-02T14:50:51.499 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-02T14:50:51.499 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.499 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.499 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "4721", 2024-04-02T14:50:51.499 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.499 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.499 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.499 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-04-02T14:50:51.499 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.499 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.499 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.499 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.500 INFO:teuthology.orchestra.run.smithi090.stdout: "allow_ptrace": { 2024-04-02T14:50:51.500 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "allow_ptrace", 2024-04-02T14:50:51.500 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:51.500 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.500 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.500 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:51.500 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.500 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.500 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.500 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-04-02T14:50:51.500 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-04-02T14:50:51.500 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.500 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.500 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.500 INFO:teuthology.orchestra.run.smithi090.stdout: "autotune_interval": { 2024-04-02T14:50:51.500 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "autotune_interval", 2024-04-02T14:50:51.501 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "secs", 2024-04-02T14:50:51.501 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.501 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.501 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "600", 2024-04-02T14:50:51.501 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.501 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.501 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.501 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "how frequently to autotune daemon memory", 2024-04-02T14:50:51.501 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.501 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.501 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.501 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.501 INFO:teuthology.orchestra.run.smithi090.stdout: "autotune_memory_target_ratio": { 2024-04-02T14:50:51.501 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "autotune_memory_target_ratio", 2024-04-02T14:50:51.501 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "float", 2024-04-02T14:50:51.502 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.502 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.502 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "0.7", 2024-04-02T14:50:51.502 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.502 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.502 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.502 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-04-02T14:50:51.502 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.502 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.502 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.502 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.502 INFO:teuthology.orchestra.run.smithi090.stdout: "cephadm_log_destination": { 2024-04-02T14:50:51.502 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "cephadm_log_destination", 2024-04-02T14:50:51.502 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.503 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.503 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.503 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:51.503 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.503 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.503 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:51.503 INFO:teuthology.orchestra.run.smithi090.stdout: "file", 2024-04-02T14:50:51.503 INFO:teuthology.orchestra.run.smithi090.stdout: "file,syslog", 2024-04-02T14:50:51.503 INFO:teuthology.orchestra.run.smithi090.stdout: "syslog" 2024-04-02T14:50:51.503 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:51.503 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-04-02T14:50:51.503 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.503 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.503 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.503 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.504 INFO:teuthology.orchestra.run.smithi090.stdout: "cgroups_split": { 2024-04-02T14:50:51.504 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "cgroups_split", 2024-04-02T14:50:51.504 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:51.504 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.504 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.504 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-02T14:50:51.504 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.504 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.504 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.504 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-04-02T14:50:51.504 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.504 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.504 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.504 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.504 INFO:teuthology.orchestra.run.smithi090.stdout: "config_checks_enabled": { 2024-04-02T14:50:51.504 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "config_checks_enabled", 2024-04-02T14:50:51.505 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:51.505 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.505 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.505 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:51.505 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.505 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.505 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.505 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-04-02T14:50:51.505 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.505 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.505 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.505 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.505 INFO:teuthology.orchestra.run.smithi090.stdout: "config_dashboard": { 2024-04-02T14:50:51.505 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "config_dashboard", 2024-04-02T14:50:51.505 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:51.505 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.505 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.506 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-02T14:50:51.506 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.506 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.506 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.506 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-04-02T14:50:51.506 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.506 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.506 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.506 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.506 INFO:teuthology.orchestra.run.smithi090.stdout: "container_image_alertmanager": { 2024-04-02T14:50:51.506 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "container_image_alertmanager", 2024-04-02T14:50:51.506 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.506 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.506 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.507 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-04-02T14:50:51.507 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.507 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.507 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.507 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Prometheus container image", 2024-04-02T14:50:51.507 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.507 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.507 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.507 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.507 INFO:teuthology.orchestra.run.smithi090.stdout: "container_image_base": { 2024-04-02T14:50:51.507 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "container_image_base", 2024-04-02T14:50:51.507 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.507 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.507 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.507 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "quay.io/ceph/ceph", 2024-04-02T14:50:51.507 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.508 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.508 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.508 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Container image name, without the tag", 2024-04-02T14:50:51.508 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.508 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.508 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.508 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.508 INFO:teuthology.orchestra.run.smithi090.stdout: "container_image_elasticsearch": { 2024-04-02T14:50:51.508 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "container_image_elasticsearch", 2024-04-02T14:50:51.508 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.508 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.508 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.508 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-04-02T14:50:51.508 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.508 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.508 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.509 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "elasticsearch container image", 2024-04-02T14:50:51.509 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.509 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.509 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.509 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.509 INFO:teuthology.orchestra.run.smithi090.stdout: "container_image_grafana": { 2024-04-02T14:50:51.509 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "container_image_grafana", 2024-04-02T14:50:51.509 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.509 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.509 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.509 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-04-02T14:50:51.509 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.509 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.509 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.509 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Prometheus container image", 2024-04-02T14:50:51.509 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.509 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.510 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.510 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.510 INFO:teuthology.orchestra.run.smithi090.stdout: "container_image_haproxy": { 2024-04-02T14:50:51.510 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "container_image_haproxy", 2024-04-02T14:50:51.510 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.510 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.510 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.510 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-04-02T14:50:51.510 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.510 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.510 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.510 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "HAproxy container image", 2024-04-02T14:50:51.510 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.511 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.511 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.511 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.511 INFO:teuthology.orchestra.run.smithi090.stdout: "container_image_jaeger_agent": { 2024-04-02T14:50:51.511 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "container_image_jaeger_agent", 2024-04-02T14:50:51.511 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.511 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.511 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.511 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-04-02T14:50:51.511 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.511 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.511 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.511 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Jaeger agent container image", 2024-04-02T14:50:51.511 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.511 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.511 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.512 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.512 INFO:teuthology.orchestra.run.smithi090.stdout: "container_image_jaeger_collector": { 2024-04-02T14:50:51.512 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "container_image_jaeger_collector", 2024-04-02T14:50:51.512 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.512 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.512 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.512 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-04-02T14:50:51.512 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.512 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.512 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.512 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Jaeger collector container image", 2024-04-02T14:50:51.512 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.512 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.512 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.512 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.512 INFO:teuthology.orchestra.run.smithi090.stdout: "container_image_jaeger_query": { 2024-04-02T14:50:51.513 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "container_image_jaeger_query", 2024-04-02T14:50:51.513 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.513 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.513 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.513 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-04-02T14:50:51.513 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.513 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.513 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.513 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Jaeger query container image", 2024-04-02T14:50:51.513 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.513 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.513 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.513 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.513 INFO:teuthology.orchestra.run.smithi090.stdout: "container_image_keepalived": { 2024-04-02T14:50:51.513 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "container_image_keepalived", 2024-04-02T14:50:51.514 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.514 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.514 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.514 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-04-02T14:50:51.514 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.514 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.514 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.514 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Keepalived container image", 2024-04-02T14:50:51.514 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.514 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.514 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.514 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.514 INFO:teuthology.orchestra.run.smithi090.stdout: "container_image_loki": { 2024-04-02T14:50:51.514 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "container_image_loki", 2024-04-02T14:50:51.515 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.515 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.515 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.515 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-04-02T14:50:51.515 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.515 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.515 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.515 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Loki container image", 2024-04-02T14:50:51.515 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.515 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.515 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.515 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.515 INFO:teuthology.orchestra.run.smithi090.stdout: "container_image_node_exporter": { 2024-04-02T14:50:51.515 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "container_image_node_exporter", 2024-04-02T14:50:51.515 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.516 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.516 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.516 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-04-02T14:50:51.516 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.516 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.516 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.516 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Prometheus container image", 2024-04-02T14:50:51.516 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.516 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.516 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.516 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.516 INFO:teuthology.orchestra.run.smithi090.stdout: "container_image_nvmeof": { 2024-04-02T14:50:51.516 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "container_image_nvmeof", 2024-04-02T14:50:51.516 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.516 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.516 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.517 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "quay.io/ceph/nvmeof:1.0.0", 2024-04-02T14:50:51.517 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.517 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.517 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.517 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Nvme-of container image", 2024-04-02T14:50:51.517 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.517 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.517 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.517 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.517 INFO:teuthology.orchestra.run.smithi090.stdout: "container_image_prometheus": { 2024-04-02T14:50:51.517 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "container_image_prometheus", 2024-04-02T14:50:51.517 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.517 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.517 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.517 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-04-02T14:50:51.518 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.518 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.518 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.518 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Prometheus container image", 2024-04-02T14:50:51.518 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.518 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.518 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.518 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.518 INFO:teuthology.orchestra.run.smithi090.stdout: "container_image_promtail": { 2024-04-02T14:50:51.518 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "container_image_promtail", 2024-04-02T14:50:51.518 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.518 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.518 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.518 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-04-02T14:50:51.519 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.519 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.519 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.519 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Promtail container image", 2024-04-02T14:50:51.519 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.519 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.519 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.519 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.519 INFO:teuthology.orchestra.run.smithi090.stdout: "container_image_samba": { 2024-04-02T14:50:51.519 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "container_image_samba", 2024-04-02T14:50:51.519 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.519 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.519 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.519 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-04-02T14:50:51.520 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.520 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.520 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.520 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Samba/SMB container image", 2024-04-02T14:50:51.520 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.520 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.520 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.520 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.520 INFO:teuthology.orchestra.run.smithi090.stdout: "container_image_snmp_gateway": { 2024-04-02T14:50:51.520 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "container_image_snmp_gateway", 2024-04-02T14:50:51.520 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.520 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.520 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.520 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-04-02T14:50:51.520 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.520 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.520 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.521 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "SNMP Gateway container image", 2024-04-02T14:50:51.521 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.521 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.521 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.521 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.521 INFO:teuthology.orchestra.run.smithi090.stdout: "container_init": { 2024-04-02T14:50:51.521 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "container_init", 2024-04-02T14:50:51.521 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:51.521 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.521 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.521 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-02T14:50:51.521 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.521 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.521 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.522 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Run podman/docker with `--init`", 2024-04-02T14:50:51.522 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.522 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.522 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.522 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.522 INFO:teuthology.orchestra.run.smithi090.stdout: "daemon_cache_timeout": { 2024-04-02T14:50:51.522 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "daemon_cache_timeout", 2024-04-02T14:50:51.522 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "secs", 2024-04-02T14:50:51.522 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.522 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.522 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "600", 2024-04-02T14:50:51.522 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.522 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.522 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.523 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-04-02T14:50:51.523 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.523 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.523 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.523 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.523 INFO:teuthology.orchestra.run.smithi090.stdout: "default_cephadm_command_timeout": { 2024-04-02T14:50:51.523 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "default_cephadm_command_timeout", 2024-04-02T14:50:51.523 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-02T14:50:51.523 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.523 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.523 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "900", 2024-04-02T14:50:51.523 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.523 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.523 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.523 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-04-02T14:50:51.524 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.524 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.524 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.524 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.524 INFO:teuthology.orchestra.run.smithi090.stdout: "default_registry": { 2024-04-02T14:50:51.524 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "default_registry", 2024-04-02T14:50:51.524 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.524 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.524 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.524 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "docker.io", 2024-04-02T14:50:51.524 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.524 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.524 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.524 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-04-02T14:50:51.524 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.524 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.525 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.525 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.525 INFO:teuthology.orchestra.run.smithi090.stdout: "device_cache_timeout": { 2024-04-02T14:50:51.525 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "device_cache_timeout", 2024-04-02T14:50:51.525 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "secs", 2024-04-02T14:50:51.525 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.525 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.525 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "1800", 2024-04-02T14:50:51.525 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.525 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.525 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.525 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "seconds to cache device inventory", 2024-04-02T14:50:51.525 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.525 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.525 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.526 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.526 INFO:teuthology.orchestra.run.smithi090.stdout: "device_enhanced_scan": { 2024-04-02T14:50:51.526 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "device_enhanced_scan", 2024-04-02T14:50:51.526 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:51.526 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.526 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.526 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:51.526 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.526 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.526 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.526 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Use libstoragemgmt during device scans", 2024-04-02T14:50:51.526 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.526 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.526 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.526 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.527 INFO:teuthology.orchestra.run.smithi090.stdout: "facts_cache_timeout": { 2024-04-02T14:50:51.527 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "facts_cache_timeout", 2024-04-02T14:50:51.527 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "secs", 2024-04-02T14:50:51.527 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.527 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.527 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "60", 2024-04-02T14:50:51.527 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.527 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.527 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.527 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "seconds to cache host facts data", 2024-04-02T14:50:51.527 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.527 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.527 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.527 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.527 INFO:teuthology.orchestra.run.smithi090.stdout: "grafana_dashboards_path": { 2024-04-02T14:50:51.527 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "grafana_dashboards_path", 2024-04-02T14:50:51.528 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.528 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.528 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.528 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-04-02T14:50:51.528 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.528 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.528 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.528 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-04-02T14:50:51.528 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.528 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.528 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.528 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.528 INFO:teuthology.orchestra.run.smithi090.stdout: "host_check_interval": { 2024-04-02T14:50:51.528 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "host_check_interval", 2024-04-02T14:50:51.528 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "secs", 2024-04-02T14:50:51.529 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.529 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.529 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "600", 2024-04-02T14:50:51.529 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.529 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.529 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.529 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "how frequently to perform a host check", 2024-04-02T14:50:51.529 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.529 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.529 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.529 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.529 INFO:teuthology.orchestra.run.smithi090.stdout: "hw_monitoring": { 2024-04-02T14:50:51.529 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "hw_monitoring", 2024-04-02T14:50:51.529 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:51.529 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.530 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.530 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:51.530 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.530 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.530 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.530 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-04-02T14:50:51.530 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.530 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.530 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.530 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.530 INFO:teuthology.orchestra.run.smithi090.stdout: "inventory_list_all": { 2024-04-02T14:50:51.530 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "inventory_list_all", 2024-04-02T14:50:51.530 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:51.530 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.530 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.531 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:51.531 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.531 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.531 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.531 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-04-02T14:50:51.531 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.531 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.531 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.531 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.531 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-02T14:50:51.531 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-02T14:50:51.531 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.531 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.531 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.531 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:51.532 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.532 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.532 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:51.532 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:51.532 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:51.532 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:51.532 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:51.532 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:51.532 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:51.532 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:51.532 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.532 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.532 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.532 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.532 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.533 INFO:teuthology.orchestra.run.smithi090.stdout: "log_refresh_metadata": { 2024-04-02T14:50:51.533 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_refresh_metadata", 2024-04-02T14:50:51.533 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:51.533 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.533 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.533 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:51.533 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.533 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.533 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.533 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-04-02T14:50:51.533 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.533 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.533 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.533 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.534 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-02T14:50:51.534 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-02T14:50:51.534 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:51.534 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.534 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.534 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-02T14:50:51.534 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.534 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.534 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.534 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-04-02T14:50:51.534 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.534 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.534 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.534 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.535 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-02T14:50:51.535 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-02T14:50:51.535 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.535 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.535 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.535 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-02T14:50:51.535 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.535 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.535 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:51.535 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:51.535 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:51.535 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:51.535 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:51.535 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:51.535 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:51.536 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:51.536 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.536 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.536 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.536 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.536 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.536 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-02T14:50:51.536 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-02T14:50:51.536 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:51.536 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.536 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.536 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:51.536 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.536 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.536 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.537 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.537 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.537 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.537 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.537 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.537 INFO:teuthology.orchestra.run.smithi090.stdout: "manage_etc_ceph_ceph_conf": { 2024-04-02T14:50:51.537 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-04-02T14:50:51.537 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:51.537 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.537 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.537 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:51.537 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.537 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.537 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.537 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-04-02T14:50:51.537 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.538 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.538 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.538 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.538 INFO:teuthology.orchestra.run.smithi090.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-04-02T14:50:51.538 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-04-02T14:50:51.538 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.538 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.538 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.538 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "*", 2024-04-02T14:50:51.538 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.538 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.538 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.538 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-04-02T14:50:51.538 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.539 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.539 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.539 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.539 INFO:teuthology.orchestra.run.smithi090.stdout: "max_count_per_host": { 2024-04-02T14:50:51.539 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "max_count_per_host", 2024-04-02T14:50:51.539 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-02T14:50:51.539 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.539 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.539 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "10", 2024-04-02T14:50:51.539 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.539 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.539 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.539 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "max number of daemons per service per host", 2024-04-02T14:50:51.539 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.539 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.539 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.540 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.540 INFO:teuthology.orchestra.run.smithi090.stdout: "max_osd_draining_count": { 2024-04-02T14:50:51.540 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "max_osd_draining_count", 2024-04-02T14:50:51.540 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-02T14:50:51.540 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.540 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.540 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "10", 2024-04-02T14:50:51.540 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.540 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.540 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.540 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-04-02T14:50:51.540 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.540 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.540 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.540 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.540 INFO:teuthology.orchestra.run.smithi090.stdout: "migration_current": { 2024-04-02T14:50:51.540 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "migration_current", 2024-04-02T14:50:51.541 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-02T14:50:51.541 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.541 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.541 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:51.541 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.541 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.541 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.541 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "internal - do not modify", 2024-04-02T14:50:51.541 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.541 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.541 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.541 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.541 INFO:teuthology.orchestra.run.smithi090.stdout: "mode": { 2024-04-02T14:50:51.541 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "mode", 2024-04-02T14:50:51.541 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.542 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.542 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.542 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "root", 2024-04-02T14:50:51.542 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.542 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.542 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:51.542 INFO:teuthology.orchestra.run.smithi090.stdout: "cephadm-package", 2024-04-02T14:50:51.542 INFO:teuthology.orchestra.run.smithi090.stdout: "root" 2024-04-02T14:50:51.542 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:51.542 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "mode for remote execution of cephadm", 2024-04-02T14:50:51.542 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.542 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.542 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.542 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.543 INFO:teuthology.orchestra.run.smithi090.stdout: "oob_default_addr": { 2024-04-02T14:50:51.543 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "oob_default_addr", 2024-04-02T14:50:51.543 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.543 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.543 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.543 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "169.254.1.1", 2024-04-02T14:50:51.543 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.543 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.543 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.543 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Default address for RedFish API (oob management).", 2024-04-02T14:50:51.543 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.543 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.543 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.543 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.543 INFO:teuthology.orchestra.run.smithi090.stdout: "prometheus_alerts_path": { 2024-04-02T14:50:51.544 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "prometheus_alerts_path", 2024-04-02T14:50:51.544 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.544 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.544 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.544 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-04-02T14:50:51.544 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.544 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.544 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.544 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-04-02T14:50:51.544 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.544 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.544 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.544 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.544 INFO:teuthology.orchestra.run.smithi090.stdout: "registry_insecure": { 2024-04-02T14:50:51.545 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "registry_insecure", 2024-04-02T14:50:51.545 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:51.545 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.545 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.545 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:51.545 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.545 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.545 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.545 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-04-02T14:50:51.545 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.545 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.545 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.545 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.545 INFO:teuthology.orchestra.run.smithi090.stdout: "registry_password": { 2024-04-02T14:50:51.546 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "registry_password", 2024-04-02T14:50:51.546 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.546 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.546 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.546 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:51.546 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.546 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.546 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.546 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-04-02T14:50:51.546 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.546 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.546 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.546 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.546 INFO:teuthology.orchestra.run.smithi090.stdout: "registry_url": { 2024-04-02T14:50:51.547 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "registry_url", 2024-04-02T14:50:51.547 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.547 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.547 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.547 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:51.547 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.547 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.547 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.547 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-04-02T14:50:51.547 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.547 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.547 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.547 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.547 INFO:teuthology.orchestra.run.smithi090.stdout: "registry_username": { 2024-04-02T14:50:51.547 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "registry_username", 2024-04-02T14:50:51.547 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.548 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.548 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.548 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:51.548 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.548 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.548 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.548 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-04-02T14:50:51.548 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.548 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.548 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.548 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.548 INFO:teuthology.orchestra.run.smithi090.stdout: "secure_monitoring_stack": { 2024-04-02T14:50:51.548 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "secure_monitoring_stack", 2024-04-02T14:50:51.548 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:51.548 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.549 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.549 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:51.549 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.549 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.549 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.549 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-04-02T14:50:51.549 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.549 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.549 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.549 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.549 INFO:teuthology.orchestra.run.smithi090.stdout: "service_discovery_port": { 2024-04-02T14:50:51.549 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "service_discovery_port", 2024-04-02T14:50:51.549 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-02T14:50:51.549 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.549 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.549 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "8765", 2024-04-02T14:50:51.550 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.550 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.550 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.550 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "cephadm service discovery port", 2024-04-02T14:50:51.550 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.550 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.550 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.550 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.550 INFO:teuthology.orchestra.run.smithi090.stdout: "ssh_config_file": { 2024-04-02T14:50:51.550 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "ssh_config_file", 2024-04-02T14:50:51.550 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.550 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.550 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.550 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:51.550 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.551 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.551 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.551 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-04-02T14:50:51.551 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.551 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.551 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.551 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.551 INFO:teuthology.orchestra.run.smithi090.stdout: "use_agent": { 2024-04-02T14:50:51.551 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "use_agent", 2024-04-02T14:50:51.551 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:51.551 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.551 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.551 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:51.551 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.551 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.551 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.552 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-04-02T14:50:51.552 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.552 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.552 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.552 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.552 INFO:teuthology.orchestra.run.smithi090.stdout: "use_repo_digest": { 2024-04-02T14:50:51.552 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "use_repo_digest", 2024-04-02T14:50:51.552 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:51.552 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.552 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.552 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-02T14:50:51.552 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.552 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.552 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.552 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-04-02T14:50:51.552 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.553 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.553 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.553 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.553 INFO:teuthology.orchestra.run.smithi090.stdout: "warn_on_failed_host_check": { 2024-04-02T14:50:51.553 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "warn_on_failed_host_check", 2024-04-02T14:50:51.553 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:51.553 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.553 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.553 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-02T14:50:51.553 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.553 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.553 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.553 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "raise a health warning if the host check fails", 2024-04-02T14:50:51.553 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.553 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.553 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.554 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.554 INFO:teuthology.orchestra.run.smithi090.stdout: "warn_on_stray_daemons": { 2024-04-02T14:50:51.554 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "warn_on_stray_daemons", 2024-04-02T14:50:51.554 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:51.554 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.554 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.554 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-02T14:50:51.554 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.554 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.554 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.554 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-04-02T14:50:51.554 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.554 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.554 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.555 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.555 INFO:teuthology.orchestra.run.smithi090.stdout: "warn_on_stray_hosts": { 2024-04-02T14:50:51.555 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "warn_on_stray_hosts", 2024-04-02T14:50:51.555 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:51.555 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.555 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.555 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-02T14:50:51.555 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.555 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.555 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.555 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-04-02T14:50:51.555 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.555 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.555 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.555 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:51.555 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:51.555 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.556 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-02T14:50:51.556 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "crash", 2024-04-02T14:50:51.556 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-02T14:50:51.556 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-02T14:50:51.556 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-02T14:50:51.556 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-02T14:50:51.556 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-02T14:50:51.556 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.556 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.556 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.556 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:51.556 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.556 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.556 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:51.556 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:51.557 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:51.557 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:51.557 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:51.557 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:51.557 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:51.557 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:51.557 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.557 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.557 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.557 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.557 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.557 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-02T14:50:51.557 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-02T14:50:51.557 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:51.557 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.557 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.558 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:51.558 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.558 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.558 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.558 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.558 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.558 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.558 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.558 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.558 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-02T14:50:51.558 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-02T14:50:51.558 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.558 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.558 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.559 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-02T14:50:51.559 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.559 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.559 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:51.559 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:51.559 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:51.559 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:51.559 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:51.559 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:51.559 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:51.559 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:51.559 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.559 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.559 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.559 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.559 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.560 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-02T14:50:51.560 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-02T14:50:51.560 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:51.560 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.560 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.560 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:51.560 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.560 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.560 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.560 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.560 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.560 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.560 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.560 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.560 INFO:teuthology.orchestra.run.smithi090.stdout: "retain_interval": { 2024-04-02T14:50:51.561 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "retain_interval", 2024-04-02T14:50:51.561 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "secs", 2024-04-02T14:50:51.561 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.561 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.561 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "31536000", 2024-04-02T14:50:51.561 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.561 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.561 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.561 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "how long to retain crashes before pruning them", 2024-04-02T14:50:51.561 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.561 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.561 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.561 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.561 INFO:teuthology.orchestra.run.smithi090.stdout: "warn_recent_interval": { 2024-04-02T14:50:51.561 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "warn_recent_interval", 2024-04-02T14:50:51.561 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "secs", 2024-04-02T14:50:51.562 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.562 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.562 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "1209600", 2024-04-02T14:50:51.562 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.562 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.562 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.562 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "time interval in which to warn about recent crashes", 2024-04-02T14:50:51.562 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.562 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.562 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.562 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:51.562 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:51.562 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.562 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-02T14:50:51.562 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "dashboard", 2024-04-02T14:50:51.563 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-02T14:50:51.563 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-02T14:50:51.563 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-02T14:50:51.563 INFO:teuthology.orchestra.run.smithi090.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-04-02T14:50:51.563 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-04-02T14:50:51.563 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-02T14:50:51.563 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.563 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.563 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "10", 2024-04-02T14:50:51.563 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.563 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.563 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.563 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.563 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.563 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.563 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.563 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.564 INFO:teuthology.orchestra.run.smithi090.stdout: "ALERTMANAGER_API_HOST": { 2024-04-02T14:50:51.564 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "ALERTMANAGER_API_HOST", 2024-04-02T14:50:51.564 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.564 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.564 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.564 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:51.564 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.564 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.564 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.564 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.564 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.564 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.564 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.564 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.564 INFO:teuthology.orchestra.run.smithi090.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-04-02T14:50:51.565 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-04-02T14:50:51.565 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:51.565 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.565 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.565 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-02T14:50:51.565 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.565 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.565 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.565 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.565 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.565 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.565 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.565 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.565 INFO:teuthology.orchestra.run.smithi090.stdout: "AUDIT_API_ENABLED": { 2024-04-02T14:50:51.565 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "AUDIT_API_ENABLED", 2024-04-02T14:50:51.565 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:51.566 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.566 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.566 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:51.566 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.566 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.566 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.566 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.566 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.566 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.566 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.566 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.566 INFO:teuthology.orchestra.run.smithi090.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-04-02T14:50:51.566 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-04-02T14:50:51.566 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:51.566 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.567 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.567 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-02T14:50:51.567 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.567 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.567 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.567 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.567 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.567 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.567 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.567 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.567 INFO:teuthology.orchestra.run.smithi090.stdout: "ENABLE_BROWSABLE_API": { 2024-04-02T14:50:51.567 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "ENABLE_BROWSABLE_API", 2024-04-02T14:50:51.567 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:51.567 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.567 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.567 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-02T14:50:51.568 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.568 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.568 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.568 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.568 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.568 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.568 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.568 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.568 INFO:teuthology.orchestra.run.smithi090.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-04-02T14:50:51.568 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-04-02T14:50:51.568 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:51.568 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.568 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.568 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-02T14:50:51.568 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.569 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.569 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.569 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.569 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.569 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.569 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.569 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.569 INFO:teuthology.orchestra.run.smithi090.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-04-02T14:50:51.569 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-04-02T14:50:51.569 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:51.569 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.569 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.569 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-02T14:50:51.569 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.569 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.569 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.570 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.570 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.570 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.570 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.570 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.570 INFO:teuthology.orchestra.run.smithi090.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-04-02T14:50:51.570 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-04-02T14:50:51.570 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:51.570 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.570 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.570 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-02T14:50:51.570 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.570 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.570 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.570 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.571 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.571 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.571 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.571 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.571 INFO:teuthology.orchestra.run.smithi090.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-04-02T14:50:51.571 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-04-02T14:50:51.571 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:51.571 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.571 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.571 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-02T14:50:51.571 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.571 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.571 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.571 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.571 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.571 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.572 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.572 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.572 INFO:teuthology.orchestra.run.smithi090.stdout: "FEATURE_TOGGLE_NFS": { 2024-04-02T14:50:51.572 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-04-02T14:50:51.572 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:51.572 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.572 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.572 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-02T14:50:51.572 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.572 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.572 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.572 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.572 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.572 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.572 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.572 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.573 INFO:teuthology.orchestra.run.smithi090.stdout: "FEATURE_TOGGLE_RBD": { 2024-04-02T14:50:51.573 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-04-02T14:50:51.573 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:51.573 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.573 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.573 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-02T14:50:51.573 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.573 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.573 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.573 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.573 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.573 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.573 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.573 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.573 INFO:teuthology.orchestra.run.smithi090.stdout: "FEATURE_TOGGLE_RGW": { 2024-04-02T14:50:51.574 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-04-02T14:50:51.574 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:51.574 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.574 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.574 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-02T14:50:51.574 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.574 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.574 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.574 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.574 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.574 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.574 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.574 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.574 INFO:teuthology.orchestra.run.smithi090.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-04-02T14:50:51.574 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-04-02T14:50:51.575 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.575 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.575 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.575 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:51.575 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.575 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.575 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.575 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.575 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.575 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.575 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.575 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.575 INFO:teuthology.orchestra.run.smithi090.stdout: "GRAFANA_API_PASSWORD": { 2024-04-02T14:50:51.575 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "GRAFANA_API_PASSWORD", 2024-04-02T14:50:51.575 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.575 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.576 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.576 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "admin", 2024-04-02T14:50:51.576 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.576 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.576 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.576 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.576 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.576 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.576 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.576 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.576 INFO:teuthology.orchestra.run.smithi090.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-04-02T14:50:51.576 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-04-02T14:50:51.576 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:51.576 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.576 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.576 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-02T14:50:51.577 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.577 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.577 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.577 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.577 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.577 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.577 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.577 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.577 INFO:teuthology.orchestra.run.smithi090.stdout: "GRAFANA_API_URL": { 2024-04-02T14:50:51.577 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "GRAFANA_API_URL", 2024-04-02T14:50:51.577 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.577 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.577 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.577 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:51.577 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.577 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.578 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.578 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.578 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.578 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.578 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.578 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.578 INFO:teuthology.orchestra.run.smithi090.stdout: "GRAFANA_API_USERNAME": { 2024-04-02T14:50:51.578 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "GRAFANA_API_USERNAME", 2024-04-02T14:50:51.578 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.578 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.578 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.578 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "admin", 2024-04-02T14:50:51.578 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.578 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.579 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.579 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.579 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.579 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.579 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.579 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.579 INFO:teuthology.orchestra.run.smithi090.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-04-02T14:50:51.579 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-04-02T14:50:51.579 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.579 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.579 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.579 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:51.579 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.579 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.579 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.579 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.580 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.580 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.580 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.580 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.580 INFO:teuthology.orchestra.run.smithi090.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-04-02T14:50:51.580 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-04-02T14:50:51.580 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:51.580 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.580 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.580 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:51.580 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.580 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.580 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.580 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.580 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.581 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.581 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.581 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.581 INFO:teuthology.orchestra.run.smithi090.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-04-02T14:50:51.581 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-04-02T14:50:51.581 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:51.581 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.581 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.581 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-02T14:50:51.581 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.581 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.581 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.581 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.581 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.581 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.581 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.582 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.582 INFO:teuthology.orchestra.run.smithi090.stdout: "ISSUE_TRACKER_API_KEY": { 2024-04-02T14:50:51.582 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-04-02T14:50:51.582 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.582 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.582 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.582 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:51.582 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.582 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.582 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.582 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.582 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.582 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.582 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.583 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.583 INFO:teuthology.orchestra.run.smithi090.stdout: "MANAGED_BY_CLUSTERS": { 2024-04-02T14:50:51.583 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "MANAGED_BY_CLUSTERS", 2024-04-02T14:50:51.583 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.583 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.583 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.583 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "[]", 2024-04-02T14:50:51.583 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.583 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.583 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.583 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.583 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.583 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.583 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.583 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.584 INFO:teuthology.orchestra.run.smithi090.stdout: "MULTICLUSTER_CONFIG": { 2024-04-02T14:50:51.584 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "MULTICLUSTER_CONFIG", 2024-04-02T14:50:51.584 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.584 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.584 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.584 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "{}", 2024-04-02T14:50:51.584 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.584 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.584 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.584 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.584 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.584 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.584 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.584 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.584 INFO:teuthology.orchestra.run.smithi090.stdout: "PROMETHEUS_API_HOST": { 2024-04-02T14:50:51.584 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "PROMETHEUS_API_HOST", 2024-04-02T14:50:51.584 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.585 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.585 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.585 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:51.585 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.585 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.585 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.585 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.585 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.585 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.585 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.585 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.585 INFO:teuthology.orchestra.run.smithi090.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-04-02T14:50:51.585 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-04-02T14:50:51.585 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:51.585 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.586 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.586 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-02T14:50:51.586 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.586 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.586 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.586 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.586 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.586 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.586 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.586 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.586 INFO:teuthology.orchestra.run.smithi090.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-04-02T14:50:51.586 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-04-02T14:50:51.586 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:51.586 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.586 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.587 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:51.587 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.587 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.587 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.587 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.587 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.587 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.587 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.587 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.587 INFO:teuthology.orchestra.run.smithi090.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-04-02T14:50:51.587 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-04-02T14:50:51.587 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:51.587 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.587 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.587 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:51.587 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.588 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.588 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.588 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.588 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.588 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.588 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.588 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.588 INFO:teuthology.orchestra.run.smithi090.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-04-02T14:50:51.588 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-04-02T14:50:51.588 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:51.588 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.588 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.588 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-02T14:50:51.588 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.588 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.588 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.589 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.589 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.589 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.589 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.589 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.589 INFO:teuthology.orchestra.run.smithi090.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-04-02T14:50:51.589 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-04-02T14:50:51.589 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:51.589 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.589 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.589 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-02T14:50:51.589 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.589 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.589 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.589 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.589 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.590 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.590 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.590 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.590 INFO:teuthology.orchestra.run.smithi090.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-04-02T14:50:51.590 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-04-02T14:50:51.590 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:51.590 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.590 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.590 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:51.590 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.590 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.590 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.590 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.590 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.590 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.591 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.591 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.591 INFO:teuthology.orchestra.run.smithi090.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-04-02T14:50:51.591 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-04-02T14:50:51.591 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:51.591 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.591 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.591 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:51.591 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.591 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.591 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.591 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.591 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.591 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.591 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.591 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.592 INFO:teuthology.orchestra.run.smithi090.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-04-02T14:50:51.592 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-04-02T14:50:51.592 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:51.592 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.592 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.592 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:51.592 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.592 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.592 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.592 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.592 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.592 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.592 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.592 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.593 INFO:teuthology.orchestra.run.smithi090.stdout: "PWD_POLICY_ENABLED": { 2024-04-02T14:50:51.593 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "PWD_POLICY_ENABLED", 2024-04-02T14:50:51.593 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:51.593 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.593 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.593 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-02T14:50:51.593 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.593 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.593 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.593 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.593 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.593 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.593 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.593 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.593 INFO:teuthology.orchestra.run.smithi090.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-04-02T14:50:51.593 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-04-02T14:50:51.594 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.594 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.594 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.594 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-04-02T14:50:51.594 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.594 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.594 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.594 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.594 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.594 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.594 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.594 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.594 INFO:teuthology.orchestra.run.smithi090.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-04-02T14:50:51.594 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-04-02T14:50:51.594 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-02T14:50:51.595 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.595 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.595 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "10", 2024-04-02T14:50:51.595 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.595 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.595 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.595 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.595 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.595 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.595 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.595 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.595 INFO:teuthology.orchestra.run.smithi090.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-04-02T14:50:51.595 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-04-02T14:50:51.595 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-02T14:50:51.595 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.596 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.596 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "8", 2024-04-02T14:50:51.596 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.596 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.596 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.596 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.596 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.596 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.596 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.596 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.596 INFO:teuthology.orchestra.run.smithi090.stdout: "REST_REQUESTS_TIMEOUT": { 2024-04-02T14:50:51.596 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-04-02T14:50:51.596 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-02T14:50:51.596 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.596 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.596 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "45", 2024-04-02T14:50:51.597 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.597 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.597 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.597 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.597 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.597 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.597 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.597 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.597 INFO:teuthology.orchestra.run.smithi090.stdout: "RGW_API_ACCESS_KEY": { 2024-04-02T14:50:51.597 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "RGW_API_ACCESS_KEY", 2024-04-02T14:50:51.597 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.597 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.597 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.597 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:51.597 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.597 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.597 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.598 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.598 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.598 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.598 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.598 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.598 INFO:teuthology.orchestra.run.smithi090.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-04-02T14:50:51.598 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-04-02T14:50:51.598 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.598 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.598 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.598 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "admin", 2024-04-02T14:50:51.598 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.598 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.598 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.598 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.599 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.599 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.599 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.599 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.599 INFO:teuthology.orchestra.run.smithi090.stdout: "RGW_API_SECRET_KEY": { 2024-04-02T14:50:51.599 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "RGW_API_SECRET_KEY", 2024-04-02T14:50:51.599 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.599 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.599 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.599 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:51.599 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.599 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.599 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.599 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.599 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.599 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.600 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.600 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.600 INFO:teuthology.orchestra.run.smithi090.stdout: "RGW_API_SSL_VERIFY": { 2024-04-02T14:50:51.600 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "RGW_API_SSL_VERIFY", 2024-04-02T14:50:51.600 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:51.600 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.600 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.600 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-02T14:50:51.600 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.600 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.600 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.600 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.600 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.600 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.600 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.600 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.601 INFO:teuthology.orchestra.run.smithi090.stdout: "UNSAFE_TLS_v1_2": { 2024-04-02T14:50:51.601 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "UNSAFE_TLS_v1_2", 2024-04-02T14:50:51.601 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:51.601 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.601 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.601 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:51.601 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.601 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.601 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.601 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.601 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.601 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.601 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.601 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.601 INFO:teuthology.orchestra.run.smithi090.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-04-02T14:50:51.601 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-04-02T14:50:51.602 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-02T14:50:51.602 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.602 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.602 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "0", 2024-04-02T14:50:51.602 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.602 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.602 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.602 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.602 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.602 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.602 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.602 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.602 INFO:teuthology.orchestra.run.smithi090.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-04-02T14:50:51.602 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-04-02T14:50:51.602 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-02T14:50:51.603 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.603 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.603 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "10", 2024-04-02T14:50:51.603 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.603 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.603 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.603 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.603 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.603 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.603 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.603 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.603 INFO:teuthology.orchestra.run.smithi090.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-04-02T14:50:51.603 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-04-02T14:50:51.603 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-02T14:50:51.603 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.604 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.604 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "5", 2024-04-02T14:50:51.604 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.604 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.604 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.604 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.604 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.604 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.604 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.604 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.604 INFO:teuthology.orchestra.run.smithi090.stdout: "cross_origin_url": { 2024-04-02T14:50:51.604 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "cross_origin_url", 2024-04-02T14:50:51.604 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.604 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.604 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.604 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:51.605 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.605 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.605 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.605 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.605 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.605 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.605 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.605 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.605 INFO:teuthology.orchestra.run.smithi090.stdout: "crt_file": { 2024-04-02T14:50:51.605 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "crt_file", 2024-04-02T14:50:51.605 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.605 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.605 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.605 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:51.605 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.605 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.606 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.606 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.606 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.606 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.606 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.606 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.606 INFO:teuthology.orchestra.run.smithi090.stdout: "debug": { 2024-04-02T14:50:51.606 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "debug", 2024-04-02T14:50:51.606 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:51.606 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.606 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.606 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:51.606 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.606 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.607 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.607 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Enable/disable debug options", 2024-04-02T14:50:51.607 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.607 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.607 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.607 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.607 INFO:teuthology.orchestra.run.smithi090.stdout: "jwt_token_ttl": { 2024-04-02T14:50:51.607 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "jwt_token_ttl", 2024-04-02T14:50:51.607 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-02T14:50:51.607 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.607 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.607 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "28800", 2024-04-02T14:50:51.607 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.607 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.607 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.607 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.607 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.608 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.608 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.608 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.608 INFO:teuthology.orchestra.run.smithi090.stdout: "key_file": { 2024-04-02T14:50:51.608 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "key_file", 2024-04-02T14:50:51.608 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.608 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.608 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.608 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:51.608 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.608 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.608 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.608 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.608 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.608 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.608 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.609 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.609 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-02T14:50:51.609 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-02T14:50:51.609 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.609 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.609 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.609 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:51.609 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.609 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.609 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:51.609 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:51.609 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:51.609 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:51.609 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:51.609 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:51.609 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:51.610 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:51.610 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.610 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.610 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.610 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.610 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.610 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-02T14:50:51.610 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-02T14:50:51.610 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:51.610 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.610 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.610 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:51.610 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.610 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.610 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.610 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.611 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.611 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.611 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.611 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.611 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-02T14:50:51.611 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-02T14:50:51.611 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.611 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.611 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.611 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-02T14:50:51.611 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.611 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.611 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:51.611 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:51.611 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:51.611 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:51.612 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:51.612 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:51.612 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:51.612 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:51.612 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.612 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.612 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.612 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.612 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.612 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-02T14:50:51.612 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-02T14:50:51.612 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:51.612 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.612 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.612 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:51.612 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.612 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.613 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.613 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.613 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.613 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.613 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.613 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.613 INFO:teuthology.orchestra.run.smithi090.stdout: "motd": { 2024-04-02T14:50:51.613 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "motd", 2024-04-02T14:50:51.613 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.613 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.613 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.613 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:51.613 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.613 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.613 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.613 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "The message of the day", 2024-04-02T14:50:51.614 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.614 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.614 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.614 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.614 INFO:teuthology.orchestra.run.smithi090.stdout: "redirect_resolve_ip_addr": { 2024-04-02T14:50:51.614 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "redirect_resolve_ip_addr", 2024-04-02T14:50:51.614 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:51.614 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.614 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.614 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:51.614 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.614 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.614 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.614 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.614 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.614 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.615 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.615 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.615 INFO:teuthology.orchestra.run.smithi090.stdout: "server_addr": { 2024-04-02T14:50:51.615 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "server_addr", 2024-04-02T14:50:51.615 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.615 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.615 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.615 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "::", 2024-04-02T14:50:51.615 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.615 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.615 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.615 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.615 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.615 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.616 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.616 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.616 INFO:teuthology.orchestra.run.smithi090.stdout: "server_port": { 2024-04-02T14:50:51.616 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "server_port", 2024-04-02T14:50:51.616 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-02T14:50:51.616 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.616 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.616 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "8080", 2024-04-02T14:50:51.616 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.616 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.616 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.616 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.616 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.616 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.616 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.616 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.617 INFO:teuthology.orchestra.run.smithi090.stdout: "ssl": { 2024-04-02T14:50:51.617 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "ssl", 2024-04-02T14:50:51.617 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:51.617 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.617 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.617 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-02T14:50:51.617 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.617 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.617 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.617 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.617 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.617 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.617 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.617 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.617 INFO:teuthology.orchestra.run.smithi090.stdout: "ssl_server_port": { 2024-04-02T14:50:51.618 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "ssl_server_port", 2024-04-02T14:50:51.618 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-02T14:50:51.618 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.618 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.618 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "8443", 2024-04-02T14:50:51.618 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.618 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.618 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.618 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.618 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.618 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.618 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.618 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.618 INFO:teuthology.orchestra.run.smithi090.stdout: "standby_behaviour": { 2024-04-02T14:50:51.618 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "standby_behaviour", 2024-04-02T14:50:51.619 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.619 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.619 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.619 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "redirect", 2024-04-02T14:50:51.619 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.619 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.619 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:51.619 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:51.619 INFO:teuthology.orchestra.run.smithi090.stdout: "redirect" 2024-04-02T14:50:51.619 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:51.619 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.619 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.619 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.619 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.619 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.620 INFO:teuthology.orchestra.run.smithi090.stdout: "standby_error_status_code": { 2024-04-02T14:50:51.620 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "standby_error_status_code", 2024-04-02T14:50:51.620 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-02T14:50:51.620 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.620 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.620 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "500", 2024-04-02T14:50:51.620 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "400", 2024-04-02T14:50:51.620 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "599", 2024-04-02T14:50:51.620 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.620 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.620 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.620 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.620 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.620 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.620 INFO:teuthology.orchestra.run.smithi090.stdout: "url_prefix": { 2024-04-02T14:50:51.621 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "url_prefix", 2024-04-02T14:50:51.621 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.621 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.621 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.621 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:51.621 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.621 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.621 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.621 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.621 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.621 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.621 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.621 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:51.621 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:51.621 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.622 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-02T14:50:51.622 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "devicehealth", 2024-04-02T14:50:51.622 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-02T14:50:51.622 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-02T14:50:51.622 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-02T14:50:51.622 INFO:teuthology.orchestra.run.smithi090.stdout: "enable_monitoring": { 2024-04-02T14:50:51.622 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "enable_monitoring", 2024-04-02T14:50:51.622 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:51.622 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.622 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.622 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-02T14:50:51.622 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.622 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.622 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.623 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "monitor device health metrics", 2024-04-02T14:50:51.623 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.623 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.623 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.623 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.623 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-02T14:50:51.623 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-02T14:50:51.623 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.623 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.623 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.623 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:51.623 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.623 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.623 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:51.623 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:51.623 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:51.624 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:51.624 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:51.624 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:51.624 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:51.624 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:51.624 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.624 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.624 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.624 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.624 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.624 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-02T14:50:51.624 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-02T14:50:51.624 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:51.624 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.624 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.624 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:51.625 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.625 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.625 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.625 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.625 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.625 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.625 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.625 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.625 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-02T14:50:51.625 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-02T14:50:51.625 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.625 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.625 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.625 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-02T14:50:51.625 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.625 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.626 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:51.626 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:51.626 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:51.626 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:51.626 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:51.626 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:51.626 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:51.626 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:51.626 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.626 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.626 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.626 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.626 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.626 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-02T14:50:51.627 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-02T14:50:51.627 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:51.627 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.627 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.627 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:51.627 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.627 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.627 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.627 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.627 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.627 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.627 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.630 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.630 INFO:teuthology.orchestra.run.smithi090.stdout: "mark_out_threshold": { 2024-04-02T14:50:51.630 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "mark_out_threshold", 2024-04-02T14:50:51.630 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "secs", 2024-04-02T14:50:51.630 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.630 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.630 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "2419200", 2024-04-02T14:50:51.631 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.631 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.631 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.631 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-04-02T14:50:51.631 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.631 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.631 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.631 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.631 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_name": { 2024-04-02T14:50:51.631 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "pool_name", 2024-04-02T14:50:51.631 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.631 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.631 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.632 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "device_health_metrics", 2024-04-02T14:50:51.632 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.632 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.632 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.632 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "name of pool in which to store device health metrics", 2024-04-02T14:50:51.632 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.632 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.632 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.632 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.632 INFO:teuthology.orchestra.run.smithi090.stdout: "retention_period": { 2024-04-02T14:50:51.632 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "retention_period", 2024-04-02T14:50:51.632 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "secs", 2024-04-02T14:50:51.632 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.632 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.632 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "15552000", 2024-04-02T14:50:51.633 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.633 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.633 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.633 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "how long to retain device health metrics", 2024-04-02T14:50:51.633 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.633 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.633 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.633 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.633 INFO:teuthology.orchestra.run.smithi090.stdout: "scrape_frequency": { 2024-04-02T14:50:51.633 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "scrape_frequency", 2024-04-02T14:50:51.633 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "secs", 2024-04-02T14:50:51.633 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.633 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.633 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "86400", 2024-04-02T14:50:51.634 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.634 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.634 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.634 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "how frequently to scrape device health metrics", 2024-04-02T14:50:51.634 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.634 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.634 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.634 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.634 INFO:teuthology.orchestra.run.smithi090.stdout: "self_heal": { 2024-04-02T14:50:51.634 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "self_heal", 2024-04-02T14:50:51.634 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:51.634 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.634 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.634 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-02T14:50:51.634 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.635 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.635 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.635 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-04-02T14:50:51.635 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.635 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.635 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.635 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.635 INFO:teuthology.orchestra.run.smithi090.stdout: "sleep_interval": { 2024-04-02T14:50:51.635 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "sleep_interval", 2024-04-02T14:50:51.635 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "secs", 2024-04-02T14:50:51.635 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.635 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.635 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "600", 2024-04-02T14:50:51.635 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.636 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.636 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.636 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "how frequently to wake up and check device health", 2024-04-02T14:50:51.636 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.636 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.636 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.636 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.636 INFO:teuthology.orchestra.run.smithi090.stdout: "warn_threshold": { 2024-04-02T14:50:51.636 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "warn_threshold", 2024-04-02T14:50:51.636 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "secs", 2024-04-02T14:50:51.636 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.636 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.636 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "7257600", 2024-04-02T14:50:51.636 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.637 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.637 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.637 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-04-02T14:50:51.637 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.637 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.637 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.637 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:51.637 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:51.637 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.637 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-02T14:50:51.637 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "diskprediction_local", 2024-04-02T14:50:51.637 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-02T14:50:51.637 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-02T14:50:51.637 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-02T14:50:51.637 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-02T14:50:51.638 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-02T14:50:51.638 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.638 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.638 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.638 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:51.638 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.638 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.638 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:51.638 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:51.638 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:51.638 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:51.638 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:51.638 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:51.639 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:51.639 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:51.639 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.639 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.639 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.639 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.639 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.639 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-02T14:50:51.639 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-02T14:50:51.639 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:51.639 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.639 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.639 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:51.639 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.640 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.640 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.640 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.640 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.640 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.640 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.640 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.640 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-02T14:50:51.640 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-02T14:50:51.640 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.640 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.640 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.640 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-02T14:50:51.640 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.640 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.640 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:51.641 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:51.641 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:51.641 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:51.641 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:51.641 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:51.641 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:51.641 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:51.641 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.641 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.641 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.641 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.641 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.641 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-02T14:50:51.641 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-02T14:50:51.642 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:51.642 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.642 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.642 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:51.642 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.642 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.642 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.642 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.642 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.642 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.642 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.642 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.642 INFO:teuthology.orchestra.run.smithi090.stdout: "predict_interval": { 2024-04-02T14:50:51.642 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "predict_interval", 2024-04-02T14:50:51.642 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.643 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.643 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.643 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "86400", 2024-04-02T14:50:51.643 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.643 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.643 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.643 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.643 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.643 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.643 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.643 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.643 INFO:teuthology.orchestra.run.smithi090.stdout: "predictor_model": { 2024-04-02T14:50:51.643 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "predictor_model", 2024-04-02T14:50:51.643 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.643 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.643 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.644 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "prophetstor", 2024-04-02T14:50:51.644 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.644 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.644 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.644 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.644 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.644 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.644 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.644 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.644 INFO:teuthology.orchestra.run.smithi090.stdout: "sleep_interval": { 2024-04-02T14:50:51.644 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "sleep_interval", 2024-04-02T14:50:51.644 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.644 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.644 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.644 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "600", 2024-04-02T14:50:51.645 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.645 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.645 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.645 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.645 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.645 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.645 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.645 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:51.645 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:51.645 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.645 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-02T14:50:51.645 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "influx", 2024-04-02T14:50:51.645 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": false, 2024-04-02T14:50:51.645 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "influxdb python module not found", 2024-04-02T14:50:51.645 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-02T14:50:51.645 INFO:teuthology.orchestra.run.smithi090.stdout: "batch_size": { 2024-04-02T14:50:51.646 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "batch_size", 2024-04-02T14:50:51.646 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-02T14:50:51.646 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.646 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.646 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "5000", 2024-04-02T14:50:51.646 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.646 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.646 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.646 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-04-02T14:50:51.646 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.646 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.646 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.646 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.646 INFO:teuthology.orchestra.run.smithi090.stdout: "database": { 2024-04-02T14:50:51.646 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "database", 2024-04-02T14:50:51.646 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.647 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.647 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.647 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "ceph", 2024-04-02T14:50:51.647 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.647 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.647 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.647 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-04-02T14:50:51.647 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.647 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.647 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.647 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.647 INFO:teuthology.orchestra.run.smithi090.stdout: "hostname": { 2024-04-02T14:50:51.647 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "hostname", 2024-04-02T14:50:51.647 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.647 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.647 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.648 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:51.648 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.648 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.648 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.648 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "InfluxDB server hostname", 2024-04-02T14:50:51.648 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.648 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.648 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.648 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.648 INFO:teuthology.orchestra.run.smithi090.stdout: "interval": { 2024-04-02T14:50:51.648 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "interval", 2024-04-02T14:50:51.648 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "secs", 2024-04-02T14:50:51.648 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.648 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.648 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "30", 2024-04-02T14:50:51.648 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "5", 2024-04-02T14:50:51.648 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.649 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.649 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-04-02T14:50:51.649 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.649 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.649 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.649 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.649 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-02T14:50:51.649 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-02T14:50:51.649 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.649 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.649 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.649 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:51.649 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.649 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.649 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:51.649 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:51.649 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:51.650 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:51.650 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:51.650 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:51.650 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:51.650 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:51.650 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.650 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.650 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.650 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.650 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.650 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-02T14:50:51.650 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-02T14:50:51.650 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:51.650 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.651 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.651 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:51.651 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.651 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.651 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.651 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.651 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.651 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.651 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.651 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.651 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-02T14:50:51.651 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-02T14:50:51.651 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.651 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.651 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.652 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-02T14:50:51.652 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.652 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.652 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:51.652 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:51.652 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:51.652 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:51.652 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:51.652 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:51.652 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:51.652 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:51.652 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.652 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.652 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.653 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.653 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.653 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-02T14:50:51.653 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-02T14:50:51.653 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:51.653 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.653 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.653 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:51.653 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.653 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.653 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.653 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.653 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.653 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.653 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.653 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.653 INFO:teuthology.orchestra.run.smithi090.stdout: "password": { 2024-04-02T14:50:51.654 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "password", 2024-04-02T14:50:51.654 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.654 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.654 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.654 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:51.654 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.654 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.654 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.654 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "password of InfluxDB server user", 2024-04-02T14:50:51.654 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.654 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.654 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.654 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.654 INFO:teuthology.orchestra.run.smithi090.stdout: "port": { 2024-04-02T14:50:51.654 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "port", 2024-04-02T14:50:51.654 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-02T14:50:51.655 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.655 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.655 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "8086", 2024-04-02T14:50:51.655 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.655 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.655 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.655 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "InfluxDB server port", 2024-04-02T14:50:51.655 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.655 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.655 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.655 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.655 INFO:teuthology.orchestra.run.smithi090.stdout: "ssl": { 2024-04-02T14:50:51.655 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "ssl", 2024-04-02T14:50:51.655 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.655 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.655 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.656 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "false", 2024-04-02T14:50:51.656 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.656 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.656 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.656 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-04-02T14:50:51.656 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.656 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.656 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.656 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.656 INFO:teuthology.orchestra.run.smithi090.stdout: "threads": { 2024-04-02T14:50:51.656 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "threads", 2024-04-02T14:50:51.656 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-02T14:50:51.656 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.656 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.656 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "5", 2024-04-02T14:50:51.657 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "1", 2024-04-02T14:50:51.657 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "32", 2024-04-02T14:50:51.657 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.657 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-04-02T14:50:51.657 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.657 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.657 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.657 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.657 INFO:teuthology.orchestra.run.smithi090.stdout: "username": { 2024-04-02T14:50:51.657 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "username", 2024-04-02T14:50:51.657 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.657 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.657 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.657 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:51.657 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.657 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.658 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.658 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "username of InfluxDB server user", 2024-04-02T14:50:51.658 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.658 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.658 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.658 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.658 INFO:teuthology.orchestra.run.smithi090.stdout: "verify_ssl": { 2024-04-02T14:50:51.658 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "verify_ssl", 2024-04-02T14:50:51.658 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.658 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.658 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.658 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "true", 2024-04-02T14:50:51.658 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.658 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.658 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.659 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-04-02T14:50:51.659 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.659 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.659 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.659 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:51.659 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:51.659 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.659 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-02T14:50:51.659 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "insights", 2024-04-02T14:50:51.659 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-02T14:50:51.659 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-02T14:50:51.659 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-02T14:50:51.659 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-02T14:50:51.659 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-02T14:50:51.659 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.660 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.660 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.660 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:51.660 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.660 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.660 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:51.660 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:51.660 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:51.660 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:51.660 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:51.660 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:51.660 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:51.660 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:51.660 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.660 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.661 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.661 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.661 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.661 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-02T14:50:51.661 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-02T14:50:51.661 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:51.661 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.661 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.661 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:51.661 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.661 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.661 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.661 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.661 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.661 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.661 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.662 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.662 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-02T14:50:51.662 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-02T14:50:51.662 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.662 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.662 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.662 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-02T14:50:51.662 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.662 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.662 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:51.662 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:51.662 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:51.662 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:51.662 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:51.662 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:51.663 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:51.663 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:51.663 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.663 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.663 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.663 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.663 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.663 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-02T14:50:51.663 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-02T14:50:51.663 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:51.663 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.663 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.663 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:51.663 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.664 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.664 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.664 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.664 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.664 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.664 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.664 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:51.664 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:51.664 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.664 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-02T14:50:51.664 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "iostat", 2024-04-02T14:50:51.664 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-02T14:50:51.664 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-02T14:50:51.664 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-02T14:50:51.664 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-02T14:50:51.664 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-02T14:50:51.665 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.665 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.665 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.665 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:51.665 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.665 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.665 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:51.665 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:51.665 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:51.665 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:51.665 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:51.665 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:51.665 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:51.665 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:51.665 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.666 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.666 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.666 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.666 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.666 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-02T14:50:51.666 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-02T14:50:51.666 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:51.666 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.666 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.666 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:51.666 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.666 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.666 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.666 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.666 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.667 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.667 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.667 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.667 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-02T14:50:51.667 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-02T14:50:51.667 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.667 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.667 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.667 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-02T14:50:51.667 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.667 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.667 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:51.667 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:51.667 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:51.667 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:51.667 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:51.668 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:51.668 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:51.668 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:51.668 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.668 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.668 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.668 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.668 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.668 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-02T14:50:51.668 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-02T14:50:51.668 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:51.668 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.668 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.669 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:51.669 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.669 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.669 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.669 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.669 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.669 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.669 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.669 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:51.669 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:51.669 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.669 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-02T14:50:51.669 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "k8sevents", 2024-04-02T14:50:51.669 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-02T14:50:51.670 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-02T14:50:51.670 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-02T14:50:51.670 INFO:teuthology.orchestra.run.smithi090.stdout: "ceph_event_retention_days": { 2024-04-02T14:50:51.670 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "ceph_event_retention_days", 2024-04-02T14:50:51.670 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-02T14:50:51.670 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.670 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.670 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "7", 2024-04-02T14:50:51.670 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.670 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.670 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.670 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Days to hold ceph event information within local cache", 2024-04-02T14:50:51.671 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.671 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.671 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.671 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.671 INFO:teuthology.orchestra.run.smithi090.stdout: "config_check_secs": { 2024-04-02T14:50:51.671 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "config_check_secs", 2024-04-02T14:50:51.671 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-02T14:50:51.671 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.671 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.671 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "10", 2024-04-02T14:50:51.671 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "10", 2024-04-02T14:50:51.671 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.671 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.671 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-04-02T14:50:51.671 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.672 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.672 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.672 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.672 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-02T14:50:51.672 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-02T14:50:51.672 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.672 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.672 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.672 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:51.672 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.672 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.672 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:51.672 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:51.672 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:51.672 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:51.673 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:51.673 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:51.673 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:51.673 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:51.673 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.673 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.673 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.673 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.673 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.673 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-02T14:50:51.673 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-02T14:50:51.673 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:51.673 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.673 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.674 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:51.674 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.674 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.674 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.674 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.674 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.674 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.674 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.674 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.674 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-02T14:50:51.674 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-02T14:50:51.674 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.674 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.675 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.675 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-02T14:50:51.675 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.675 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.675 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:51.675 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:51.675 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:51.675 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:51.675 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:51.675 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:51.675 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:51.675 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:51.675 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.675 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.675 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.675 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.676 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.676 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-02T14:50:51.676 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-02T14:50:51.676 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:51.676 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.676 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.676 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:51.676 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.676 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.676 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.676 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.676 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.676 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.676 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.676 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:51.677 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:51.677 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.677 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-02T14:50:51.677 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "localpool", 2024-04-02T14:50:51.677 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-02T14:50:51.677 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-02T14:50:51.677 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-02T14:50:51.677 INFO:teuthology.orchestra.run.smithi090.stdout: "failure_domain": { 2024-04-02T14:50:51.677 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "failure_domain", 2024-04-02T14:50:51.677 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.677 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.677 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.677 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "host", 2024-04-02T14:50:51.677 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.678 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.678 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.678 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "failure domain for any created local pool", 2024-04-02T14:50:51.678 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-04-02T14:50:51.678 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.678 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.678 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.678 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-02T14:50:51.678 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-02T14:50:51.678 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.678 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.678 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.678 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:51.679 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.679 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.679 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:51.679 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:51.679 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:51.679 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:51.679 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:51.679 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:51.679 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:51.679 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:51.679 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.679 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.679 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.679 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.679 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.679 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-02T14:50:51.679 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-02T14:50:51.680 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:51.680 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.680 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.680 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:51.680 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.680 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.680 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.680 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.680 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.680 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.680 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.680 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.680 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-02T14:50:51.680 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-02T14:50:51.680 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.680 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.681 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.681 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-02T14:50:51.681 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.681 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.681 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:51.681 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:51.681 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:51.681 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:51.681 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:51.681 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:51.681 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:51.681 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:51.681 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.681 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.682 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.682 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.682 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.682 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-02T14:50:51.682 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-02T14:50:51.682 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:51.682 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.682 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.682 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:51.682 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.682 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.682 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.682 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.682 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.683 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.683 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.683 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.683 INFO:teuthology.orchestra.run.smithi090.stdout: "min_size": { 2024-04-02T14:50:51.683 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "min_size", 2024-04-02T14:50:51.683 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-02T14:50:51.683 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.683 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.683 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:51.683 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.683 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.683 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.683 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "default min_size for any created local pool", 2024-04-02T14:50:51.683 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-04-02T14:50:51.683 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.684 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.684 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.684 INFO:teuthology.orchestra.run.smithi090.stdout: "num_rep": { 2024-04-02T14:50:51.684 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "num_rep", 2024-04-02T14:50:51.684 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-02T14:50:51.684 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.684 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.684 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "3", 2024-04-02T14:50:51.684 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.684 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.684 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.684 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "default replica count for any created local pool", 2024-04-02T14:50:51.684 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.684 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.684 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.685 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.685 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num": { 2024-04-02T14:50:51.685 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "pg_num", 2024-04-02T14:50:51.685 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-02T14:50:51.685 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.685 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.685 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "128", 2024-04-02T14:50:51.685 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.685 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.685 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.685 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "default pg_num for any created local pool", 2024-04-02T14:50:51.685 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.685 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.685 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.686 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.686 INFO:teuthology.orchestra.run.smithi090.stdout: "prefix": { 2024-04-02T14:50:51.686 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "prefix", 2024-04-02T14:50:51.686 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.686 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.686 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.686 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:51.686 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.686 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.686 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.686 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "name prefix for any created local pool", 2024-04-02T14:50:51.686 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.686 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.687 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.687 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.687 INFO:teuthology.orchestra.run.smithi090.stdout: "subtree": { 2024-04-02T14:50:51.687 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "subtree", 2024-04-02T14:50:51.687 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.687 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.687 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.687 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "rack", 2024-04-02T14:50:51.687 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.687 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.687 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.687 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "CRUSH level for which to create a local pool", 2024-04-02T14:50:51.687 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-04-02T14:50:51.687 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.688 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.688 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:51.688 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:51.688 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.688 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-02T14:50:51.688 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "mds_autoscaler", 2024-04-02T14:50:51.688 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-02T14:50:51.688 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-02T14:50:51.688 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-02T14:50:51.688 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-02T14:50:51.688 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-02T14:50:51.688 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.688 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.688 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.689 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:51.689 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.689 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.689 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:51.689 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:51.689 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:51.689 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:51.689 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:51.689 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:51.689 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:51.689 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:51.689 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.689 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.689 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.689 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.690 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.690 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-02T14:50:51.690 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-02T14:50:51.690 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:51.690 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.690 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.690 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:51.690 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.690 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.690 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.690 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.690 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.690 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.690 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.691 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.691 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-02T14:50:51.691 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-02T14:50:51.691 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.691 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.691 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.691 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-02T14:50:51.691 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.691 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.691 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:51.691 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:51.691 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:51.691 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:51.691 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:51.692 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:51.692 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:51.692 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:51.692 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.692 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.692 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.692 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.692 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.692 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-02T14:50:51.692 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-02T14:50:51.692 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:51.692 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.692 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.692 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:51.693 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.693 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.693 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.693 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.693 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.693 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.693 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.693 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:51.693 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:51.693 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.693 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-02T14:50:51.693 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "mirroring", 2024-04-02T14:50:51.693 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-02T14:50:51.693 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-02T14:50:51.693 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-02T14:50:51.694 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-02T14:50:51.694 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-02T14:50:51.694 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.694 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.694 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.694 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:51.694 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.694 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.694 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:51.694 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:51.694 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:51.694 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:51.694 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:51.694 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:51.694 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:51.695 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:51.695 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.695 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.695 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.695 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.695 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.695 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-02T14:50:51.695 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-02T14:50:51.695 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:51.695 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.695 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.695 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:51.695 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.696 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.696 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.696 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.696 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.696 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.696 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.696 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.696 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-02T14:50:51.696 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-02T14:50:51.696 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.696 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.696 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.696 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-02T14:50:51.696 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.696 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.697 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:51.697 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:51.697 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:51.697 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:51.697 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:51.697 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:51.697 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:51.697 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:51.697 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.697 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.697 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.697 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.697 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.697 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-02T14:50:51.698 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-02T14:50:51.698 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:51.698 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.698 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.698 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:51.698 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.698 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.698 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.698 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.698 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.698 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.698 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.698 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:51.698 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:51.699 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.699 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-02T14:50:51.699 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "nfs", 2024-04-02T14:50:51.699 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-02T14:50:51.699 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-02T14:50:51.699 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-02T14:50:51.699 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-02T14:50:51.699 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-02T14:50:51.699 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.699 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.699 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.699 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:51.699 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.699 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.700 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:51.700 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:51.700 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:51.700 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:51.700 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:51.700 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:51.700 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:51.700 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:51.700 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.700 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.700 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.700 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.700 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.700 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-02T14:50:51.700 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-02T14:50:51.701 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:51.701 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.701 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.701 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:51.701 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.701 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.701 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.701 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.701 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.701 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.701 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.701 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.701 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-02T14:50:51.701 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-02T14:50:51.702 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.702 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.702 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.702 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-02T14:50:51.702 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.702 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.702 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:51.702 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:51.702 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:51.702 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:51.702 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:51.702 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:51.702 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:51.702 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:51.702 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.703 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.703 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.703 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.703 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.703 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-02T14:50:51.703 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-02T14:50:51.703 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:51.703 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.703 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.703 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:51.703 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.703 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.703 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.703 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.703 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.704 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.704 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.704 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:51.704 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:51.704 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.704 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-02T14:50:51.704 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "orchestrator", 2024-04-02T14:50:51.704 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-02T14:50:51.704 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-02T14:50:51.704 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-02T14:50:51.704 INFO:teuthology.orchestra.run.smithi090.stdout: "fail_fs": { 2024-04-02T14:50:51.704 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "fail_fs", 2024-04-02T14:50:51.704 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:51.704 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.705 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.705 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:51.705 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.705 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.705 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.705 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-04-02T14:50:51.705 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.705 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.705 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.705 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.705 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-02T14:50:51.705 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-02T14:50:51.705 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.705 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.705 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.706 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:51.706 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.706 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.706 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:51.706 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:51.706 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:51.706 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:51.706 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:51.706 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:51.706 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:51.706 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:51.706 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.706 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.706 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.706 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.707 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.707 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-02T14:50:51.707 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-02T14:50:51.707 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:51.707 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.707 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.707 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:51.707 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.707 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.707 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.707 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.707 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.707 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.707 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.708 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.708 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-02T14:50:51.708 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-02T14:50:51.708 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.708 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.708 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.708 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-02T14:50:51.708 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.708 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.708 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:51.708 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:51.708 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:51.708 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:51.709 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:51.709 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:51.709 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:51.709 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:51.709 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.709 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.709 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.709 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.709 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.709 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-02T14:50:51.709 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-02T14:50:51.709 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:51.709 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.709 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.709 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:51.710 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.710 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.710 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.710 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.710 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.710 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.710 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.710 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.710 INFO:teuthology.orchestra.run.smithi090.stdout: "orchestrator": { 2024-04-02T14:50:51.710 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "orchestrator", 2024-04-02T14:50:51.710 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.710 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.710 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.711 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:51.711 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.711 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.711 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:51.711 INFO:teuthology.orchestra.run.smithi090.stdout: "cephadm", 2024-04-02T14:50:51.711 INFO:teuthology.orchestra.run.smithi090.stdout: "rook", 2024-04-02T14:50:51.711 INFO:teuthology.orchestra.run.smithi090.stdout: "test_orchestrator" 2024-04-02T14:50:51.711 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:51.711 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Orchestrator backend", 2024-04-02T14:50:51.711 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.711 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.711 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.711 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:51.711 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:51.712 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.712 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-02T14:50:51.712 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "osd_perf_query", 2024-04-02T14:50:51.712 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-02T14:50:51.712 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-02T14:50:51.712 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-02T14:50:51.712 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-02T14:50:51.712 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-02T14:50:51.712 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.712 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.712 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.712 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:51.712 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.712 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.712 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:51.713 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:51.713 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:51.713 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:51.713 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:51.713 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:51.713 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:51.713 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:51.713 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.713 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.713 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.713 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.713 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.713 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-02T14:50:51.713 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-02T14:50:51.714 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:51.714 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.714 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.714 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:51.714 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.714 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.714 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.714 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.714 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.714 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.714 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.714 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.714 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-02T14:50:51.714 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-02T14:50:51.715 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.715 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.715 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.715 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-02T14:50:51.715 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.715 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.715 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:51.715 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:51.715 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:51.715 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:51.715 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:51.715 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:51.715 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:51.715 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:51.715 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.716 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.716 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.716 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.716 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.716 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-02T14:50:51.716 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-02T14:50:51.716 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:51.716 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.716 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.716 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:51.716 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.716 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.716 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.716 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.717 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.717 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.717 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.717 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:51.717 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:51.717 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.717 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-02T14:50:51.717 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "osd_support", 2024-04-02T14:50:51.717 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-02T14:50:51.717 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-02T14:50:51.717 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-02T14:50:51.717 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-02T14:50:51.717 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-02T14:50:51.717 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.717 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.717 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.718 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:51.718 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.718 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.718 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:51.718 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:51.718 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:51.718 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:51.718 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:51.718 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:51.718 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:51.718 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:51.718 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.718 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.718 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.719 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.719 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.719 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-02T14:50:51.719 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-02T14:50:51.719 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:51.719 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.719 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.719 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:51.719 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.719 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.719 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.719 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.719 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.719 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.719 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.720 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.720 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-02T14:50:51.720 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-02T14:50:51.720 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.720 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.720 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.720 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-02T14:50:51.720 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.720 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.720 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:51.720 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:51.720 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:51.720 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:51.720 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:51.720 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:51.721 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:51.721 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:51.721 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.721 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.721 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.721 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.721 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.721 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-02T14:50:51.721 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-02T14:50:51.721 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:51.721 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.721 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.721 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:51.721 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.722 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.722 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.722 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.722 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.722 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.722 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.722 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:51.722 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:51.722 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.722 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-02T14:50:51.722 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "pg_autoscaler", 2024-04-02T14:50:51.722 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-02T14:50:51.722 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-02T14:50:51.722 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-02T14:50:51.723 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-02T14:50:51.723 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-02T14:50:51.723 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.723 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.723 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.723 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:51.723 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.723 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.723 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:51.723 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:51.723 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:51.723 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:51.723 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:51.723 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:51.724 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:51.724 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:51.724 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.724 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.724 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.724 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.724 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.724 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-02T14:50:51.724 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-02T14:50:51.724 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:51.724 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.724 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.724 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:51.724 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.724 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.725 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.725 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.725 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.725 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.725 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.725 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.725 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-02T14:50:51.725 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-02T14:50:51.725 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.725 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.725 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.725 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-02T14:50:51.725 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.725 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.725 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:51.726 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:51.726 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:51.726 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:51.726 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:51.726 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:51.726 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:51.726 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:51.726 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.726 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.726 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.726 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.726 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.726 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-02T14:50:51.726 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-02T14:50:51.727 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:51.727 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.727 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.727 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:51.727 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.727 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.727 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.727 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.727 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.727 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.727 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.727 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.727 INFO:teuthology.orchestra.run.smithi090.stdout: "sleep_interval": { 2024-04-02T14:50:51.727 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "sleep_interval", 2024-04-02T14:50:51.727 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "secs", 2024-04-02T14:50:51.728 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.728 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.728 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "60", 2024-04-02T14:50:51.728 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.728 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.728 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.728 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.728 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.728 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.728 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.728 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.728 INFO:teuthology.orchestra.run.smithi090.stdout: "threshold": { 2024-04-02T14:50:51.728 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "threshold", 2024-04-02T14:50:51.728 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "float", 2024-04-02T14:50:51.729 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.729 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.729 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "3.0", 2024-04-02T14:50:51.729 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "1.0", 2024-04-02T14:50:51.729 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.729 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.729 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "scaling threshold", 2024-04-02T14:50:51.729 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-04-02T14:50:51.729 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.729 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.729 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:51.729 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:51.729 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.729 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-02T14:50:51.730 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "progress", 2024-04-02T14:50:51.730 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-02T14:50:51.730 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-02T14:50:51.730 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-02T14:50:51.730 INFO:teuthology.orchestra.run.smithi090.stdout: "allow_pg_recovery_event": { 2024-04-02T14:50:51.730 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "allow_pg_recovery_event", 2024-04-02T14:50:51.730 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:51.730 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.730 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.730 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:51.730 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.730 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.730 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.731 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "allow the module to show pg recovery progress", 2024-04-02T14:50:51.731 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.731 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.731 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.731 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.731 INFO:teuthology.orchestra.run.smithi090.stdout: "enabled": { 2024-04-02T14:50:51.731 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "enabled", 2024-04-02T14:50:51.731 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:51.731 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.731 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.731 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-02T14:50:51.731 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.731 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.731 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.731 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.732 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.732 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.732 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.732 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.732 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-02T14:50:51.732 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-02T14:50:51.732 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.732 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.732 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.732 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:51.732 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.732 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.732 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:51.732 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:51.732 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:51.733 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:51.733 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:51.733 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:51.733 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:51.733 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:51.733 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.733 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.733 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.733 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.733 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.733 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-02T14:50:51.733 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-02T14:50:51.733 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:51.733 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.734 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.734 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:51.734 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.734 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.734 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.734 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.734 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.734 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.734 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.734 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.734 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-02T14:50:51.734 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-02T14:50:51.734 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.734 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.735 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.735 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-02T14:50:51.735 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.735 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.735 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:51.735 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:51.735 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:51.735 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:51.735 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:51.735 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:51.735 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:51.735 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:51.735 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.735 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.735 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.736 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.736 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.736 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-02T14:50:51.736 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-02T14:50:51.736 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:51.736 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.736 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.736 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:51.736 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.736 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.736 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.736 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.736 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.736 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.736 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.737 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.737 INFO:teuthology.orchestra.run.smithi090.stdout: "max_completed_events": { 2024-04-02T14:50:51.737 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "max_completed_events", 2024-04-02T14:50:51.737 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-02T14:50:51.737 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.737 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.737 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "50", 2024-04-02T14:50:51.737 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.737 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.737 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.737 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "number of past completed events to remember", 2024-04-02T14:50:51.737 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.737 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.737 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.737 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.738 INFO:teuthology.orchestra.run.smithi090.stdout: "sleep_interval": { 2024-04-02T14:50:51.738 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "sleep_interval", 2024-04-02T14:50:51.738 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "secs", 2024-04-02T14:50:51.738 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.738 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.738 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "5", 2024-04-02T14:50:51.738 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.738 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.738 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.738 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "how long the module is going to sleep", 2024-04-02T14:50:51.738 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.738 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.738 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.738 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:51.739 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:51.739 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.739 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-02T14:50:51.739 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "prometheus", 2024-04-02T14:50:51.739 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-02T14:50:51.739 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-02T14:50:51.739 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-02T14:50:51.739 INFO:teuthology.orchestra.run.smithi090.stdout: "cache": { 2024-04-02T14:50:51.739 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "cache", 2024-04-02T14:50:51.739 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:51.739 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.739 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.739 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-02T14:50:51.739 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.739 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.740 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.740 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.740 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.740 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.740 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.740 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.740 INFO:teuthology.orchestra.run.smithi090.stdout: "exclude_perf_counters": { 2024-04-02T14:50:51.740 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "exclude_perf_counters", 2024-04-02T14:50:51.740 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:51.740 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.740 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.740 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-02T14:50:51.740 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.741 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.741 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.741 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-04-02T14:50:51.741 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-04-02T14:50:51.741 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.741 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.741 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.741 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-02T14:50:51.741 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-02T14:50:51.741 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.741 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.741 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.741 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:51.741 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.742 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.742 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:51.742 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:51.742 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:51.742 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:51.742 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:51.742 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:51.742 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:51.742 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:51.742 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.742 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.742 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.742 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.743 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.743 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-02T14:50:51.743 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-02T14:50:51.743 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:51.743 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.743 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.743 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:51.743 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.743 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.743 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.743 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.743 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.743 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.744 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.744 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.744 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-02T14:50:51.744 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-02T14:50:51.744 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.744 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.744 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.744 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-02T14:50:51.744 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.744 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.744 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:51.744 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:51.744 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:51.745 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:51.745 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:51.745 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:51.745 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:51.745 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:51.745 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.745 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.745 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.745 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.745 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.745 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-02T14:50:51.745 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-02T14:50:51.745 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:51.745 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.745 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.746 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:51.746 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.746 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.746 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.746 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.746 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.746 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.746 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.746 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.746 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_stats_pools": { 2024-04-02T14:50:51.746 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "rbd_stats_pools", 2024-04-02T14:50:51.746 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.746 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.747 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.747 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:51.747 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.747 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.747 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.747 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.747 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.747 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.747 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.747 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.747 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_stats_pools_refresh_interval": { 2024-04-02T14:50:51.747 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-04-02T14:50:51.747 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-02T14:50:51.747 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.748 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.748 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "300", 2024-04-02T14:50:51.748 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.748 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.748 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.748 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.748 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.748 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.748 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.748 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.748 INFO:teuthology.orchestra.run.smithi090.stdout: "scrape_interval": { 2024-04-02T14:50:51.748 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "scrape_interval", 2024-04-02T14:50:51.749 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "float", 2024-04-02T14:50:51.749 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.749 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.749 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "15.0", 2024-04-02T14:50:51.749 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.749 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.749 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.749 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.749 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.749 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.749 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.749 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.750 INFO:teuthology.orchestra.run.smithi090.stdout: "server_addr": { 2024-04-02T14:50:51.750 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "server_addr", 2024-04-02T14:50:51.750 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.750 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.750 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.750 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "::", 2024-04-02T14:50:51.750 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.750 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.750 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.750 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-04-02T14:50:51.750 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.750 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.750 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.750 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.751 INFO:teuthology.orchestra.run.smithi090.stdout: "server_port": { 2024-04-02T14:50:51.751 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "server_port", 2024-04-02T14:50:51.751 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-02T14:50:51.751 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.751 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.751 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "9283", 2024-04-02T14:50:51.751 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.751 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.751 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.751 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-04-02T14:50:51.751 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.751 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.751 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.751 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.752 INFO:teuthology.orchestra.run.smithi090.stdout: "stale_cache_strategy": { 2024-04-02T14:50:51.752 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "stale_cache_strategy", 2024-04-02T14:50:51.752 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.752 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.752 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.752 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "log", 2024-04-02T14:50:51.752 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.752 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.752 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.752 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.752 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.752 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.752 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.753 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.753 INFO:teuthology.orchestra.run.smithi090.stdout: "standby_behaviour": { 2024-04-02T14:50:51.753 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "standby_behaviour", 2024-04-02T14:50:51.753 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.753 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.753 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.753 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "default", 2024-04-02T14:50:51.753 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.753 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.753 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:51.753 INFO:teuthology.orchestra.run.smithi090.stdout: "default", 2024-04-02T14:50:51.753 INFO:teuthology.orchestra.run.smithi090.stdout: "error" 2024-04-02T14:50:51.753 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:51.753 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.753 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.754 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.754 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.754 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.754 INFO:teuthology.orchestra.run.smithi090.stdout: "standby_error_status_code": { 2024-04-02T14:50:51.754 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "standby_error_status_code", 2024-04-02T14:50:51.754 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-02T14:50:51.754 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.754 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.754 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "500", 2024-04-02T14:50:51.754 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "400", 2024-04-02T14:50:51.754 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "599", 2024-04-02T14:50:51.754 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.754 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.755 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.755 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.755 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.755 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:51.755 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:51.755 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.755 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-02T14:50:51.755 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "rbd_support", 2024-04-02T14:50:51.755 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-02T14:50:51.755 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-02T14:50:51.755 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-02T14:50:51.755 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-02T14:50:51.755 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-02T14:50:51.755 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.755 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.756 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.756 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:51.756 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.756 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.756 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:51.756 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:51.756 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:51.756 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:51.756 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:51.756 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:51.756 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:51.756 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:51.756 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.757 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.757 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.757 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.757 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.757 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-02T14:50:51.757 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-02T14:50:51.757 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:51.757 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.757 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.757 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:51.757 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.757 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.757 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.757 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.758 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.758 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.758 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.758 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.758 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-02T14:50:51.758 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-02T14:50:51.758 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.758 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.758 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.758 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-02T14:50:51.758 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.758 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.758 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:51.759 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:51.759 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:51.759 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:51.759 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:51.759 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:51.759 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:51.759 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:51.759 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.759 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.759 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.759 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.759 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.759 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-02T14:50:51.759 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-02T14:50:51.760 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:51.760 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.760 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.760 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:51.760 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.760 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.760 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.760 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.760 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.760 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.760 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.760 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.760 INFO:teuthology.orchestra.run.smithi090.stdout: "max_concurrent_snap_create": { 2024-04-02T14:50:51.761 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "max_concurrent_snap_create", 2024-04-02T14:50:51.761 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-02T14:50:51.761 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.761 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.761 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "10", 2024-04-02T14:50:51.761 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.761 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.761 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.761 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.761 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.761 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.761 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.761 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.761 INFO:teuthology.orchestra.run.smithi090.stdout: "mirror_snapshot_schedule": { 2024-04-02T14:50:51.761 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "mirror_snapshot_schedule", 2024-04-02T14:50:51.762 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.762 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.762 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.762 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:51.762 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.762 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.762 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.762 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.762 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.762 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.762 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.762 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.762 INFO:teuthology.orchestra.run.smithi090.stdout: "trash_purge_schedule": { 2024-04-02T14:50:51.763 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "trash_purge_schedule", 2024-04-02T14:50:51.763 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.763 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.763 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.763 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:51.763 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.763 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.763 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.763 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.763 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.763 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.763 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.764 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:51.764 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:51.764 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.764 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-02T14:50:51.764 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "restful", 2024-04-02T14:50:51.764 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-02T14:50:51.764 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-02T14:50:51.764 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-02T14:50:51.764 INFO:teuthology.orchestra.run.smithi090.stdout: "enable_auth": { 2024-04-02T14:50:51.764 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "enable_auth", 2024-04-02T14:50:51.764 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:51.764 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.764 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.765 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-02T14:50:51.765 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.765 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.765 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.765 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.765 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.765 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.765 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.765 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.765 INFO:teuthology.orchestra.run.smithi090.stdout: "key_file": { 2024-04-02T14:50:51.765 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "key_file", 2024-04-02T14:50:51.765 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.765 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.765 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.766 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:51.766 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.766 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.766 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.766 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.766 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.766 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.766 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.766 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.766 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-02T14:50:51.766 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-02T14:50:51.766 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.766 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.766 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.767 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:51.767 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.767 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.767 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:51.767 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:51.767 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:51.767 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:51.767 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:51.767 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:51.767 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:51.767 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:51.767 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.767 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.768 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.768 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.768 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.768 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-02T14:50:51.768 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-02T14:50:51.768 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:51.768 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.768 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.768 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:51.768 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.768 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.768 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.768 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.768 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.769 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.769 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.769 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.769 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-02T14:50:51.769 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-02T14:50:51.769 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.769 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.769 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.769 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-02T14:50:51.769 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.769 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.769 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:51.769 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:51.769 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:51.770 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:51.770 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:51.770 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:51.770 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:51.770 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:51.770 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.770 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.770 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.770 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.770 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.770 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-02T14:50:51.770 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-02T14:50:51.770 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:51.771 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.771 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.771 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:51.771 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.771 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.771 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.771 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.771 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.771 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.771 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.771 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.771 INFO:teuthology.orchestra.run.smithi090.stdout: "server_addr": { 2024-04-02T14:50:51.771 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "server_addr", 2024-04-02T14:50:51.771 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.772 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.772 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.772 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:51.772 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.772 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.772 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.772 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.772 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.772 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.772 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.772 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.772 INFO:teuthology.orchestra.run.smithi090.stdout: "server_port": { 2024-04-02T14:50:51.772 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "server_port", 2024-04-02T14:50:51.772 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.773 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.773 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.773 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:51.773 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.773 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.773 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.773 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.773 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.773 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.773 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.773 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:51.773 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:51.773 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.773 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-02T14:50:51.774 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "rgw", 2024-04-02T14:50:51.774 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-02T14:50:51.774 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-02T14:50:51.774 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-02T14:50:51.774 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-02T14:50:51.774 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-02T14:50:51.774 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.774 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.774 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.774 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:51.774 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.774 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.774 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:51.774 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:51.775 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:51.775 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:51.775 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:51.775 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:51.775 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:51.775 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:51.775 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.775 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.775 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.775 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.775 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.775 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-02T14:50:51.775 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-02T14:50:51.775 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:51.775 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.776 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.776 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:51.776 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.776 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.776 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.776 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.776 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.776 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.776 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.776 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.776 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-02T14:50:51.776 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-02T14:50:51.776 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.776 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.776 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.776 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-02T14:50:51.777 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.777 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.777 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:51.777 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:51.777 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:51.777 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:51.777 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:51.777 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:51.777 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:51.777 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:51.777 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.777 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.777 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.777 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.777 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.778 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-02T14:50:51.778 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-02T14:50:51.778 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:51.778 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.778 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.778 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:51.778 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.778 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.778 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.778 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.778 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.778 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.778 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.778 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:51.778 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:51.779 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.779 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-02T14:50:51.779 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "rook", 2024-04-02T14:50:51.779 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-02T14:50:51.779 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-02T14:50:51.779 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-02T14:50:51.779 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-02T14:50:51.779 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-02T14:50:51.779 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.779 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.779 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.779 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:51.779 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.779 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.779 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:51.780 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:51.780 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:51.780 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:51.780 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:51.780 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:51.780 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:51.780 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:51.780 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.780 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.780 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.780 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.780 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.780 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-02T14:50:51.780 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-02T14:50:51.780 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:51.780 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.781 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.781 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:51.781 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.781 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.781 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.781 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.781 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.781 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.781 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.781 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.781 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-02T14:50:51.781 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-02T14:50:51.781 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.781 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.781 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.782 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-02T14:50:51.782 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.782 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.782 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:51.782 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:51.782 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:51.782 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:51.782 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:51.782 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:51.782 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:51.782 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:51.782 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.782 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.782 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.783 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.783 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.783 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-02T14:50:51.783 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-02T14:50:51.783 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:51.783 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.783 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.783 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:51.783 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.783 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.783 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.783 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.783 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.783 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.783 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.784 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.784 INFO:teuthology.orchestra.run.smithi090.stdout: "storage_class": { 2024-04-02T14:50:51.784 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "storage_class", 2024-04-02T14:50:51.784 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.784 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.784 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.784 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "local", 2024-04-02T14:50:51.784 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.784 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.784 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.784 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-04-02T14:50:51.784 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.784 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.784 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.784 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:51.784 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:51.784 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.785 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-02T14:50:51.785 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "selftest", 2024-04-02T14:50:51.785 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-02T14:50:51.785 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-02T14:50:51.785 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-02T14:50:51.785 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-02T14:50:51.785 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-02T14:50:51.785 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.785 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.785 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.785 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:51.785 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.785 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.785 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:51.785 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:51.786 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:51.786 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:51.786 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:51.786 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:51.786 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:51.786 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:51.786 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.786 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.786 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.786 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.786 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.786 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-02T14:50:51.786 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-02T14:50:51.787 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:51.787 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.787 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.787 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:51.787 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.787 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.787 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.787 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.787 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.787 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.787 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.787 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.787 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-02T14:50:51.787 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-02T14:50:51.788 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.788 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.788 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.788 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-02T14:50:51.788 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.788 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.788 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:51.788 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:51.788 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:51.788 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:51.788 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:51.788 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:51.788 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:51.788 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:51.789 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.789 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.789 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.789 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.789 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.789 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-02T14:50:51.789 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-02T14:50:51.789 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:51.789 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.789 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.789 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:51.789 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.789 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.789 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.789 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.790 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.790 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.790 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.790 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.790 INFO:teuthology.orchestra.run.smithi090.stdout: "roption1": { 2024-04-02T14:50:51.790 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "roption1", 2024-04-02T14:50:51.790 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.790 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.790 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.790 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:51.790 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.790 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.790 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.791 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.791 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.791 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.791 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.791 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.791 INFO:teuthology.orchestra.run.smithi090.stdout: "roption2": { 2024-04-02T14:50:51.791 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "roption2", 2024-04-02T14:50:51.791 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.791 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.791 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.791 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "xyz", 2024-04-02T14:50:51.791 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.791 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.791 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.791 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.792 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.792 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.792 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.792 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.792 INFO:teuthology.orchestra.run.smithi090.stdout: "rwoption1": { 2024-04-02T14:50:51.792 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "rwoption1", 2024-04-02T14:50:51.792 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.792 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.792 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.792 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:51.792 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.792 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.792 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.792 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.792 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.793 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.793 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.793 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.793 INFO:teuthology.orchestra.run.smithi090.stdout: "rwoption2": { 2024-04-02T14:50:51.793 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "rwoption2", 2024-04-02T14:50:51.793 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-02T14:50:51.793 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.793 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.793 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:51.793 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.793 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.793 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.793 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.793 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.793 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.794 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.794 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.794 INFO:teuthology.orchestra.run.smithi090.stdout: "rwoption3": { 2024-04-02T14:50:51.794 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "rwoption3", 2024-04-02T14:50:51.794 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "float", 2024-04-02T14:50:51.794 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.794 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.794 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:51.794 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.794 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.794 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.794 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.794 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.795 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.795 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.795 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.795 INFO:teuthology.orchestra.run.smithi090.stdout: "rwoption4": { 2024-04-02T14:50:51.795 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "rwoption4", 2024-04-02T14:50:51.795 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.795 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.795 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.795 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:51.795 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.795 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.795 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.795 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.795 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.796 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.796 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.796 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.796 INFO:teuthology.orchestra.run.smithi090.stdout: "rwoption5": { 2024-04-02T14:50:51.796 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "rwoption5", 2024-04-02T14:50:51.796 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:51.796 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.796 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.796 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:51.796 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.796 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.796 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.796 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.796 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.796 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.797 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.797 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.797 INFO:teuthology.orchestra.run.smithi090.stdout: "rwoption6": { 2024-04-02T14:50:51.797 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "rwoption6", 2024-04-02T14:50:51.797 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:51.797 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.797 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.797 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-02T14:50:51.797 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.797 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.797 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.797 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.797 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.797 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.797 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.798 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.798 INFO:teuthology.orchestra.run.smithi090.stdout: "rwoption7": { 2024-04-02T14:50:51.798 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "rwoption7", 2024-04-02T14:50:51.798 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-02T14:50:51.798 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.798 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.798 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:51.798 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "1", 2024-04-02T14:50:51.798 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "42", 2024-04-02T14:50:51.798 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.798 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.798 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.798 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.798 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.799 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.799 INFO:teuthology.orchestra.run.smithi090.stdout: "testkey": { 2024-04-02T14:50:51.799 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "testkey", 2024-04-02T14:50:51.799 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.799 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.799 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.799 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:51.799 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.799 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.799 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.799 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.799 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.799 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.799 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.800 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.800 INFO:teuthology.orchestra.run.smithi090.stdout: "testlkey": { 2024-04-02T14:50:51.800 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "testlkey", 2024-04-02T14:50:51.800 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.800 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.800 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.800 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:51.800 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.800 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.800 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.800 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.800 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.800 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.800 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.800 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.801 INFO:teuthology.orchestra.run.smithi090.stdout: "testnewline": { 2024-04-02T14:50:51.801 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "testnewline", 2024-04-02T14:50:51.801 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.801 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.801 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.801 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:51.801 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.801 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.801 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.801 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.801 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.801 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.801 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.801 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:51.801 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:51.802 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.802 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-02T14:50:51.802 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "snap_schedule", 2024-04-02T14:50:51.802 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-02T14:50:51.802 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-02T14:50:51.802 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-02T14:50:51.802 INFO:teuthology.orchestra.run.smithi090.stdout: "allow_m_granularity": { 2024-04-02T14:50:51.802 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "allow_m_granularity", 2024-04-02T14:50:51.802 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:51.802 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.802 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.802 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:51.802 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.802 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.802 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.803 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "allow minute scheduled snapshots", 2024-04-02T14:50:51.803 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.803 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.803 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.803 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.803 INFO:teuthology.orchestra.run.smithi090.stdout: "dump_on_update": { 2024-04-02T14:50:51.803 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "dump_on_update", 2024-04-02T14:50:51.803 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:51.803 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.803 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.803 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:51.803 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.803 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.803 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.804 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "dump database to debug log on update", 2024-04-02T14:50:51.804 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.804 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.804 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.804 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.804 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-02T14:50:51.804 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-02T14:50:51.804 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.804 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.804 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.804 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:51.804 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.804 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.804 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:51.805 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:51.805 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:51.805 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:51.805 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:51.805 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:51.805 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:51.805 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:51.805 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.805 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.805 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.805 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.805 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.805 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-02T14:50:51.805 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-02T14:50:51.806 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:51.806 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.806 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.806 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:51.806 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.806 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.806 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.806 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.806 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.806 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.806 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.806 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.806 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-02T14:50:51.807 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-02T14:50:51.807 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.807 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.807 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.807 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-02T14:50:51.807 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.807 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.807 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:51.807 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:51.807 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:51.807 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:51.807 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:51.807 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:51.807 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:51.807 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:51.808 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.808 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.808 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.808 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.808 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.808 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-02T14:50:51.808 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-02T14:50:51.808 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:51.808 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.808 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.808 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:51.808 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.808 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.808 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.809 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.809 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.809 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.809 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.809 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:51.809 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:51.809 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.809 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-02T14:50:51.809 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "stats", 2024-04-02T14:50:51.809 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-02T14:50:51.809 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-02T14:50:51.809 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-02T14:50:51.809 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-02T14:50:51.809 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-02T14:50:51.809 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.810 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.810 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.810 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:51.810 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.810 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.810 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:51.810 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:51.810 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:51.810 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:51.810 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:51.810 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:51.810 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:51.810 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:51.810 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.810 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.811 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.811 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.811 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.811 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-02T14:50:51.811 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-02T14:50:51.811 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:51.811 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.811 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.811 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:51.811 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.811 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.811 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.811 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.811 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.811 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.812 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.812 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.812 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-02T14:50:51.812 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-02T14:50:51.812 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.812 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.812 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.812 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-02T14:50:51.812 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.812 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.812 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:51.812 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:51.812 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:51.812 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:51.812 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:51.812 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:51.813 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:51.813 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:51.813 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.813 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.813 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.813 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.813 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.813 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-02T14:50:51.813 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-02T14:50:51.813 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:51.813 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.813 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.813 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:51.813 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.813 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.814 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.814 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.814 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.814 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.814 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.814 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:51.814 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:51.814 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.814 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-02T14:50:51.814 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "status", 2024-04-02T14:50:51.814 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-02T14:50:51.814 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-02T14:50:51.814 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-02T14:50:51.814 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-02T14:50:51.814 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-02T14:50:51.815 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.815 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.815 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.815 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:51.815 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.815 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.815 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:51.815 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:51.815 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:51.815 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:51.815 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:51.815 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:51.815 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:51.815 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:51.815 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.815 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.816 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.816 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.816 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.816 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-02T14:50:51.816 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-02T14:50:51.816 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:51.816 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.816 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.816 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:51.816 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.816 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.816 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.816 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.816 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.816 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.817 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.817 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.817 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-02T14:50:51.817 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-02T14:50:51.817 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.817 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.817 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.817 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-02T14:50:51.817 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.817 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.817 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:51.817 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:51.817 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:51.817 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:51.818 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:51.818 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:51.818 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:51.818 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:51.818 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.818 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.818 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.818 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.818 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.818 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-02T14:50:51.818 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-02T14:50:51.818 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:51.818 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.818 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.819 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:51.819 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.819 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.819 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.819 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.819 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.819 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.819 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.819 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:51.819 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:51.819 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.819 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-02T14:50:51.819 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "telegraf", 2024-04-02T14:50:51.819 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-02T14:50:51.819 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-02T14:50:51.820 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-02T14:50:51.820 INFO:teuthology.orchestra.run.smithi090.stdout: "address": { 2024-04-02T14:50:51.820 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "address", 2024-04-02T14:50:51.820 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.820 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.820 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.820 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-04-02T14:50:51.820 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.820 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.820 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.820 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.820 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.820 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.820 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.821 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.821 INFO:teuthology.orchestra.run.smithi090.stdout: "interval": { 2024-04-02T14:50:51.821 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "interval", 2024-04-02T14:50:51.821 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "secs", 2024-04-02T14:50:51.821 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.821 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.821 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "15", 2024-04-02T14:50:51.821 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.821 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.821 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.821 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.821 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.821 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.821 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.821 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.821 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-02T14:50:51.822 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-02T14:50:51.822 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.822 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.822 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.822 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:51.822 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.822 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.822 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:51.822 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:51.822 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:51.822 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:51.822 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:51.822 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:51.823 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:51.823 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:51.823 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.823 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.823 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.823 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.823 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.823 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-02T14:50:51.823 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-02T14:50:51.823 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:51.823 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.823 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.823 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:51.823 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.823 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.824 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.824 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.824 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.824 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.824 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.824 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.824 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-02T14:50:51.824 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-02T14:50:51.824 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.824 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.824 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.824 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-02T14:50:51.824 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.824 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.824 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:51.825 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:51.825 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:51.825 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:51.825 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:51.825 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:51.825 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:51.825 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:51.825 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.825 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.825 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.825 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.825 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.825 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-02T14:50:51.825 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-02T14:50:51.825 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:51.826 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.826 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.826 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:51.826 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.826 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.826 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.826 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.826 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.826 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.826 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.826 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:51.826 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:51.826 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.826 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-02T14:50:51.826 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "telemetry", 2024-04-02T14:50:51.827 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-02T14:50:51.827 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-02T14:50:51.827 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-02T14:50:51.827 INFO:teuthology.orchestra.run.smithi090.stdout: "channel_basic": { 2024-04-02T14:50:51.827 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "channel_basic", 2024-04-02T14:50:51.827 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:51.827 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.827 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.827 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-02T14:50:51.827 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.827 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.827 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.827 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Share basic cluster information (size, version)", 2024-04-02T14:50:51.827 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.828 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.828 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.828 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.828 INFO:teuthology.orchestra.run.smithi090.stdout: "channel_crash": { 2024-04-02T14:50:51.828 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "channel_crash", 2024-04-02T14:50:51.828 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:51.828 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.828 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.828 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-02T14:50:51.828 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.828 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.828 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.828 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-04-02T14:50:51.828 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.829 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.829 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.829 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.829 INFO:teuthology.orchestra.run.smithi090.stdout: "channel_device": { 2024-04-02T14:50:51.829 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "channel_device", 2024-04-02T14:50:51.829 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:51.829 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.829 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.829 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-02T14:50:51.829 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.829 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.829 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.829 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-04-02T14:50:51.829 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.830 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.830 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.830 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.830 INFO:teuthology.orchestra.run.smithi090.stdout: "channel_ident": { 2024-04-02T14:50:51.830 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "channel_ident", 2024-04-02T14:50:51.830 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:51.830 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.830 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.830 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:51.830 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.830 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.830 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.830 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-04-02T14:50:51.831 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.831 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.831 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.831 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.831 INFO:teuthology.orchestra.run.smithi090.stdout: "channel_perf": { 2024-04-02T14:50:51.831 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "channel_perf", 2024-04-02T14:50:51.831 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:51.831 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.831 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.831 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:51.831 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.831 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.831 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.831 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Share various performance metrics of a cluster", 2024-04-02T14:50:51.832 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.834 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.834 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.834 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.834 INFO:teuthology.orchestra.run.smithi090.stdout: "contact": { 2024-04-02T14:50:51.834 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "contact", 2024-04-02T14:50:51.834 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.834 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.835 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.835 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:51.835 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.835 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.835 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.835 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.835 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.835 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.835 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.835 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.835 INFO:teuthology.orchestra.run.smithi090.stdout: "description": { 2024-04-02T14:50:51.835 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "description", 2024-04-02T14:50:51.835 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.836 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.836 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.836 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:51.836 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.836 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.836 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.836 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.836 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.836 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.836 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.836 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.836 INFO:teuthology.orchestra.run.smithi090.stdout: "device_url": { 2024-04-02T14:50:51.836 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "device_url", 2024-04-02T14:50:51.837 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.837 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.837 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.837 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-04-02T14:50:51.837 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.837 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.837 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.837 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.837 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.837 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.837 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.837 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.837 INFO:teuthology.orchestra.run.smithi090.stdout: "enabled": { 2024-04-02T14:50:51.837 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "enabled", 2024-04-02T14:50:51.838 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:51.838 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.838 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.838 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:51.838 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.838 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.838 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.838 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.838 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.838 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.838 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.838 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.838 INFO:teuthology.orchestra.run.smithi090.stdout: "interval": { 2024-04-02T14:50:51.838 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "interval", 2024-04-02T14:50:51.839 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-02T14:50:51.839 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.839 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.839 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "24", 2024-04-02T14:50:51.839 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "8", 2024-04-02T14:50:51.839 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.839 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.839 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.839 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.839 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.839 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.839 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.839 INFO:teuthology.orchestra.run.smithi090.stdout: "last_opt_revision": { 2024-04-02T14:50:51.839 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "last_opt_revision", 2024-04-02T14:50:51.840 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-02T14:50:51.840 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.840 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.840 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "1", 2024-04-02T14:50:51.840 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.840 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.840 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.840 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.840 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.840 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.840 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.840 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.840 INFO:teuthology.orchestra.run.smithi090.stdout: "leaderboard": { 2024-04-02T14:50:51.841 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "leaderboard", 2024-04-02T14:50:51.841 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:51.841 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.841 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.841 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:51.841 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.841 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.841 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.841 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.841 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.841 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.841 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.841 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.841 INFO:teuthology.orchestra.run.smithi090.stdout: "leaderboard_description": { 2024-04-02T14:50:51.841 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "leaderboard_description", 2024-04-02T14:50:51.842 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.842 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.842 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.842 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:51.842 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.842 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.842 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.842 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.842 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.842 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.842 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.842 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.842 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-02T14:50:51.843 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-02T14:50:51.843 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.843 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.843 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.843 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:51.843 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.843 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.843 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:51.843 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:51.843 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:51.843 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:51.843 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:51.843 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:51.843 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:51.843 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:51.844 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.844 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.844 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.844 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.844 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.844 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-02T14:50:51.844 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-02T14:50:51.844 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:51.844 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.844 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.844 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:51.844 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.844 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.844 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.845 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.845 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.845 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.845 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.845 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.845 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-02T14:50:51.845 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-02T14:50:51.845 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.845 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.845 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.845 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-02T14:50:51.845 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.845 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.846 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:51.846 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:51.846 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:51.846 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:51.846 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:51.846 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:51.846 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:51.846 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:51.846 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.846 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.846 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.846 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.846 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.847 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-02T14:50:51.847 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-02T14:50:51.847 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:51.847 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.847 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.847 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:51.847 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.847 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.847 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.847 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.847 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.847 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.847 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.847 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.848 INFO:teuthology.orchestra.run.smithi090.stdout: "organization": { 2024-04-02T14:50:51.848 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "organization", 2024-04-02T14:50:51.848 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.848 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.848 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.848 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:51.848 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.848 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.848 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.848 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.848 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.848 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.848 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.849 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.849 INFO:teuthology.orchestra.run.smithi090.stdout: "proxy": { 2024-04-02T14:50:51.849 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "proxy", 2024-04-02T14:50:51.849 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.849 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.849 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.849 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:51.849 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.849 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.849 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.849 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.849 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.849 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.849 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.850 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.850 INFO:teuthology.orchestra.run.smithi090.stdout: "url": { 2024-04-02T14:50:51.850 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "url", 2024-04-02T14:50:51.850 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.850 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.850 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.850 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-04-02T14:50:51.850 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.850 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.850 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.850 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.850 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.850 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.850 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.851 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:51.851 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:51.851 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.851 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-02T14:50:51.851 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "test_orchestrator", 2024-04-02T14:50:51.851 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-02T14:50:51.851 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-02T14:50:51.851 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-02T14:50:51.851 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-02T14:50:51.851 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-02T14:50:51.851 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.851 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.852 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.852 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:51.852 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.852 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.852 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:51.852 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:51.852 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:51.852 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:51.852 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:51.852 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:51.852 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:51.852 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:51.852 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.852 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.853 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.853 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.853 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.853 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-02T14:50:51.853 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-02T14:50:51.853 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:51.853 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.853 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.853 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:51.853 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.853 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.853 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.853 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.854 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.854 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.854 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.854 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.854 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-02T14:50:51.854 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-02T14:50:51.854 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.854 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.854 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.854 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-02T14:50:51.854 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.854 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.854 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:51.854 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:51.854 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:51.855 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:51.855 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:51.855 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:51.855 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:51.855 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:51.855 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.855 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.855 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.855 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.855 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.855 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-02T14:50:51.855 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-02T14:50:51.855 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:51.855 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.856 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.856 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:51.856 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.856 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.856 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.856 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.856 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.856 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.856 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.856 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:51.856 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:51.856 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.856 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-02T14:50:51.856 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "volumes", 2024-04-02T14:50:51.857 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-02T14:50:51.857 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-02T14:50:51.857 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-02T14:50:51.857 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-02T14:50:51.857 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-02T14:50:51.857 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.857 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.857 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.857 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:51.857 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.857 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.857 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:51.857 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:51.857 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:51.857 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:51.857 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:51.858 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:51.858 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:51.858 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:51.858 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.858 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.858 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.858 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.858 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.858 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-02T14:50:51.858 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-02T14:50:51.858 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:51.858 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.858 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.859 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:51.859 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.859 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.859 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.859 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.859 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.859 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.859 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.859 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.859 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-02T14:50:51.859 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-02T14:50:51.859 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.859 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.859 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.859 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-02T14:50:51.860 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.860 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.860 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:51.860 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:51.860 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:51.860 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:51.860 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:51.860 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:51.860 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:51.860 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:51.860 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.860 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.860 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.860 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.861 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.861 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-02T14:50:51.861 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-02T14:50:51.861 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:51.861 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.861 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.861 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:51.861 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.861 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.861 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.861 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.861 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.861 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.861 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.862 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.862 INFO:teuthology.orchestra.run.smithi090.stdout: "max_concurrent_clones": { 2024-04-02T14:50:51.862 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "max_concurrent_clones", 2024-04-02T14:50:51.862 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-02T14:50:51.862 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.862 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.862 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "4", 2024-04-02T14:50:51.862 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.862 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.862 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.862 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Number of asynchronous cloner threads", 2024-04-02T14:50:51.862 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.862 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.862 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.863 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.863 INFO:teuthology.orchestra.run.smithi090.stdout: "periodic_async_work": { 2024-04-02T14:50:51.863 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "periodic_async_work", 2024-04-02T14:50:51.863 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:51.863 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.863 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.863 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:51.863 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.863 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.863 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.863 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Periodically check for async work", 2024-04-02T14:50:51.863 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.863 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.863 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.863 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.864 INFO:teuthology.orchestra.run.smithi090.stdout: "snapshot_clone_delay": { 2024-04-02T14:50:51.864 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "snapshot_clone_delay", 2024-04-02T14:50:51.864 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-02T14:50:51.864 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.864 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.864 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "0", 2024-04-02T14:50:51.864 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.864 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.864 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.864 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-04-02T14:50:51.864 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.864 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.864 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.864 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.864 INFO:teuthology.orchestra.run.smithi090.stdout: "snapshot_clone_no_wait": { 2024-04-02T14:50:51.864 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "snapshot_clone_no_wait", 2024-04-02T14:50:51.865 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:51.865 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.865 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.865 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-02T14:50:51.865 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.865 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.865 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.865 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-04-02T14:50:51.865 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.865 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.865 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.865 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:51.865 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:51.865 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.866 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-02T14:50:51.866 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "zabbix", 2024-04-02T14:50:51.866 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-02T14:50:51.866 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-02T14:50:51.866 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-02T14:50:51.866 INFO:teuthology.orchestra.run.smithi090.stdout: "discovery_interval": { 2024-04-02T14:50:51.866 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "discovery_interval", 2024-04-02T14:50:51.866 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "uint", 2024-04-02T14:50:51.866 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.866 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.866 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "100", 2024-04-02T14:50:51.866 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.866 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.867 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.867 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.867 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.867 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.867 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.867 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.867 INFO:teuthology.orchestra.run.smithi090.stdout: "identifier": { 2024-04-02T14:50:51.867 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "identifier", 2024-04-02T14:50:51.867 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.867 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.867 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.867 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:51.867 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.867 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.867 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.867 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.867 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.868 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.868 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.868 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.868 INFO:teuthology.orchestra.run.smithi090.stdout: "interval": { 2024-04-02T14:50:51.868 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "interval", 2024-04-02T14:50:51.868 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "secs", 2024-04-02T14:50:51.868 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.868 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.868 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "60", 2024-04-02T14:50:51.868 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.868 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.868 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.868 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.868 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.868 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.868 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.869 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.869 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-02T14:50:51.869 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-02T14:50:51.869 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.869 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.869 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.869 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:51.869 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.869 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.869 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:51.869 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:51.869 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:51.869 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:51.869 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:51.869 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:51.869 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:51.870 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:51.870 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.870 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.870 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.870 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.870 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.870 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-02T14:50:51.870 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-02T14:50:51.870 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:51.870 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.870 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.870 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:51.870 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.870 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.871 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.871 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.871 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.871 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.871 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.871 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.871 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-02T14:50:51.871 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-02T14:50:51.871 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.871 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.871 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.871 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-02T14:50:51.871 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.871 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.871 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:51.872 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:51.872 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:51.872 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:51.872 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:51.872 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:51.872 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:51.872 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:51.872 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.872 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.872 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.872 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.872 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.872 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-02T14:50:51.872 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-02T14:50:51.872 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:51.873 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.873 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.873 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:51.873 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.873 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.873 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.873 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.873 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.873 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.873 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.873 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.873 INFO:teuthology.orchestra.run.smithi090.stdout: "zabbix_host": { 2024-04-02T14:50:51.873 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "zabbix_host", 2024-04-02T14:50:51.873 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.873 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.873 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.874 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:51.874 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.874 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.874 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.874 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.874 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.874 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.874 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.874 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.874 INFO:teuthology.orchestra.run.smithi090.stdout: "zabbix_port": { 2024-04-02T14:50:51.874 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "zabbix_port", 2024-04-02T14:50:51.874 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-02T14:50:51.874 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.874 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.875 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "10051", 2024-04-02T14:50:51.875 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.875 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.875 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.875 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.875 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.875 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.875 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.875 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.875 INFO:teuthology.orchestra.run.smithi090.stdout: "zabbix_sender": { 2024-04-02T14:50:51.875 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "zabbix_sender", 2024-04-02T14:50:51.875 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.875 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.875 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.876 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-04-02T14:50:51.876 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.876 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.876 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.876 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.876 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.876 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.876 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.876 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:51.876 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:51.876 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:51.876 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-02T14:50:51.876 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.877 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-02T14:50:51.877 INFO:teuthology.orchestra.run.smithi090.stdout: "gid": 25283, 2024-04-02T14:50:51.877 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "y", 2024-04-02T14:50:51.877 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr_features": 4540701547738038271, 2024-04-02T14:50:51.877 INFO:teuthology.orchestra.run.smithi090.stdout: "available_modules": [ 2024-04-02T14:50:51.877 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-02T14:50:51.877 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "alerts", 2024-04-02T14:50:51.877 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-02T14:50:51.877 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-02T14:50:51.877 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-02T14:50:51.877 INFO:teuthology.orchestra.run.smithi090.stdout: "interval": { 2024-04-02T14:50:51.877 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "interval", 2024-04-02T14:50:51.877 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "secs", 2024-04-02T14:50:51.877 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.877 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.878 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "60", 2024-04-02T14:50:51.878 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.878 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.878 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.878 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "How frequently to reexamine health status", 2024-04-02T14:50:51.878 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.878 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.878 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.878 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.878 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-02T14:50:51.878 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-02T14:50:51.878 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.878 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.878 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.879 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:51.879 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.879 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.879 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:51.879 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:51.879 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:51.879 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:51.879 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:51.879 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:51.879 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:51.879 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:51.879 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.879 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.879 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.880 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.880 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.880 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-02T14:50:51.880 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-02T14:50:51.880 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:51.880 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.880 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.880 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:51.880 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.880 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.880 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.880 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.880 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.880 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.881 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.881 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.881 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-02T14:50:51.881 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-02T14:50:51.881 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.881 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.881 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.881 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-02T14:50:51.881 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.881 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.881 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:51.881 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:51.881 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:51.881 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:51.882 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:51.882 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:51.882 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:51.882 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:51.882 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.882 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.882 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.882 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.882 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.882 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-02T14:50:51.882 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-02T14:50:51.882 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:51.882 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.882 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.882 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:51.883 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.883 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.883 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.883 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.883 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.883 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.883 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.883 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.883 INFO:teuthology.orchestra.run.smithi090.stdout: "smtp_destination": { 2024-04-02T14:50:51.883 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "smtp_destination", 2024-04-02T14:50:51.883 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.883 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.883 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.883 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:51.884 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.884 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.884 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.884 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Email address to send alerts to", 2024-04-02T14:50:51.884 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.884 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.884 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.884 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.884 INFO:teuthology.orchestra.run.smithi090.stdout: "smtp_from_name": { 2024-04-02T14:50:51.884 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "smtp_from_name", 2024-04-02T14:50:51.884 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.884 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.884 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.884 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "Ceph", 2024-04-02T14:50:51.884 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.885 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.885 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.885 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Email From: name", 2024-04-02T14:50:51.885 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.885 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.885 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.885 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.885 INFO:teuthology.orchestra.run.smithi090.stdout: "smtp_host": { 2024-04-02T14:50:51.885 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "smtp_host", 2024-04-02T14:50:51.885 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.885 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.885 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.885 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:51.885 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.885 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.885 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.885 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "SMTP server", 2024-04-02T14:50:51.886 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.886 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.886 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.886 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.886 INFO:teuthology.orchestra.run.smithi090.stdout: "smtp_password": { 2024-04-02T14:50:51.886 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "smtp_password", 2024-04-02T14:50:51.886 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.886 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.886 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.886 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:51.886 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.886 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.886 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.886 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Password to authenticate with", 2024-04-02T14:50:51.887 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.887 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.887 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.887 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.887 INFO:teuthology.orchestra.run.smithi090.stdout: "smtp_port": { 2024-04-02T14:50:51.887 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "smtp_port", 2024-04-02T14:50:51.887 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-02T14:50:51.887 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.887 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.887 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "465", 2024-04-02T14:50:51.887 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.887 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.887 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.887 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "SMTP port", 2024-04-02T14:50:51.888 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.888 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.888 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.888 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.888 INFO:teuthology.orchestra.run.smithi090.stdout: "smtp_sender": { 2024-04-02T14:50:51.888 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "smtp_sender", 2024-04-02T14:50:51.888 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.888 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.888 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.888 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:51.888 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.888 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.888 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.888 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "SMTP envelope sender", 2024-04-02T14:50:51.888 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.889 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.889 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.889 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.889 INFO:teuthology.orchestra.run.smithi090.stdout: "smtp_ssl": { 2024-04-02T14:50:51.889 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "smtp_ssl", 2024-04-02T14:50:51.889 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:51.889 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.889 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.889 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-02T14:50:51.889 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.889 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.889 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.889 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Use SSL to connect to SMTP server", 2024-04-02T14:50:51.889 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.889 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.890 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.890 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.890 INFO:teuthology.orchestra.run.smithi090.stdout: "smtp_user": { 2024-04-02T14:50:51.890 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "smtp_user", 2024-04-02T14:50:51.890 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.890 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.890 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.890 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:51.890 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.890 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.890 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.890 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "User to authenticate as", 2024-04-02T14:50:51.890 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.890 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.890 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.890 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:51.891 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:51.891 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.891 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-02T14:50:51.891 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "balancer", 2024-04-02T14:50:51.891 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-02T14:50:51.891 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-02T14:50:51.891 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-02T14:50:51.891 INFO:teuthology.orchestra.run.smithi090.stdout: "active": { 2024-04-02T14:50:51.891 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "active", 2024-04-02T14:50:51.891 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:51.891 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.891 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.891 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-02T14:50:51.891 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.892 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.892 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.892 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "automatically balance PGs across cluster", 2024-04-02T14:50:51.892 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.892 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.892 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.892 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.892 INFO:teuthology.orchestra.run.smithi090.stdout: "begin_time": { 2024-04-02T14:50:51.892 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "begin_time", 2024-04-02T14:50:51.892 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.892 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.892 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.892 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "0000", 2024-04-02T14:50:51.892 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.892 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.892 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.892 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "beginning time of day to automatically balance", 2024-04-02T14:50:51.893 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-02T14:50:51.893 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.893 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.893 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.893 INFO:teuthology.orchestra.run.smithi090.stdout: "begin_weekday": { 2024-04-02T14:50:51.893 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "begin_weekday", 2024-04-02T14:50:51.893 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "uint", 2024-04-02T14:50:51.893 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.893 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.893 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "0", 2024-04-02T14:50:51.893 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "0", 2024-04-02T14:50:51.893 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "6", 2024-04-02T14:50:51.893 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.893 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-04-02T14:50:51.893 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-02T14:50:51.893 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.893 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.894 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.894 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_compat_max_iterations": { 2024-04-02T14:50:51.894 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "crush_compat_max_iterations", 2024-04-02T14:50:51.894 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "uint", 2024-04-02T14:50:51.894 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.894 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.894 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "25", 2024-04-02T14:50:51.894 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "1", 2024-04-02T14:50:51.894 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "250", 2024-04-02T14:50:51.894 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.894 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-04-02T14:50:51.894 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.894 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.894 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.894 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.895 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_compat_metrics": { 2024-04-02T14:50:51.895 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "crush_compat_metrics", 2024-04-02T14:50:51.895 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.895 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.895 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.895 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "pgs,objects,bytes", 2024-04-02T14:50:51.895 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.895 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.895 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.895 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-04-02T14:50:51.895 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-04-02T14:50:51.895 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.895 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.895 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.895 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_compat_step": { 2024-04-02T14:50:51.895 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "crush_compat_step", 2024-04-02T14:50:51.896 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "float", 2024-04-02T14:50:51.896 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.896 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.896 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "0.5", 2024-04-02T14:50:51.896 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "0.001", 2024-04-02T14:50:51.896 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "0.999", 2024-04-02T14:50:51.896 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.896 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "aggressiveness of optimization", 2024-04-02T14:50:51.896 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-04-02T14:50:51.896 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.896 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.896 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.896 INFO:teuthology.orchestra.run.smithi090.stdout: "end_time": { 2024-04-02T14:50:51.896 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "end_time", 2024-04-02T14:50:51.896 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.897 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.897 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.897 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "2359", 2024-04-02T14:50:51.897 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.897 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.897 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.897 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "ending time of day to automatically balance", 2024-04-02T14:50:51.897 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-02T14:50:51.897 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.897 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.897 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.897 INFO:teuthology.orchestra.run.smithi090.stdout: "end_weekday": { 2024-04-02T14:50:51.897 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "end_weekday", 2024-04-02T14:50:51.897 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "uint", 2024-04-02T14:50:51.897 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.897 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.897 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "0", 2024-04-02T14:50:51.898 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "0", 2024-04-02T14:50:51.898 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "6", 2024-04-02T14:50:51.898 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.898 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-04-02T14:50:51.898 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-02T14:50:51.898 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.898 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.898 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.898 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-02T14:50:51.898 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-02T14:50:51.898 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.898 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.898 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.898 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:51.899 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.899 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.899 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:51.899 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:51.899 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:51.899 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:51.899 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:51.899 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:51.899 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:51.899 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:51.899 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.899 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.899 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.899 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.899 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.900 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-02T14:50:51.900 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-02T14:50:51.900 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:51.900 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.900 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.900 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:51.900 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.900 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.900 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.900 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.900 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.900 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.900 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.901 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.901 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-02T14:50:51.901 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-02T14:50:51.901 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.901 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.901 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.901 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-02T14:50:51.901 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.901 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.901 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:51.901 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:51.901 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:51.901 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:51.901 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:51.901 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:51.902 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:51.902 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:51.902 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.902 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.902 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.902 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.902 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.902 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-02T14:50:51.902 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-02T14:50:51.902 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:51.902 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.902 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.902 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:51.902 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.902 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.903 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.903 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.903 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.903 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.903 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.903 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.903 INFO:teuthology.orchestra.run.smithi090.stdout: "min_score": { 2024-04-02T14:50:51.903 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "min_score", 2024-04-02T14:50:51.903 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "float", 2024-04-02T14:50:51.903 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.903 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.903 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "0", 2024-04-02T14:50:51.903 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.903 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.903 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.903 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-04-02T14:50:51.903 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.904 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.904 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.904 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.904 INFO:teuthology.orchestra.run.smithi090.stdout: "mode": { 2024-04-02T14:50:51.904 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "mode", 2024-04-02T14:50:51.904 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.904 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.904 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.904 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "upmap", 2024-04-02T14:50:51.904 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.904 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.904 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:51.904 INFO:teuthology.orchestra.run.smithi090.stdout: "crush-compat", 2024-04-02T14:50:51.904 INFO:teuthology.orchestra.run.smithi090.stdout: "none", 2024-04-02T14:50:51.904 INFO:teuthology.orchestra.run.smithi090.stdout: "read", 2024-04-02T14:50:51.904 INFO:teuthology.orchestra.run.smithi090.stdout: "upmap", 2024-04-02T14:50:51.904 INFO:teuthology.orchestra.run.smithi090.stdout: "upmap-read" 2024-04-02T14:50:51.905 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:51.905 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Balancer mode", 2024-04-02T14:50:51.905 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.905 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.905 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.905 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.905 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_ids": { 2024-04-02T14:50:51.905 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "pool_ids", 2024-04-02T14:50:51.905 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.905 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.905 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.905 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:51.905 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.905 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.905 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.905 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-04-02T14:50:51.905 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.906 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.906 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.906 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.906 INFO:teuthology.orchestra.run.smithi090.stdout: "sleep_interval": { 2024-04-02T14:50:51.906 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "sleep_interval", 2024-04-02T14:50:51.906 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "secs", 2024-04-02T14:50:51.906 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.906 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.906 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "60", 2024-04-02T14:50:51.906 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.906 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.906 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.906 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-04-02T14:50:51.906 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.906 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.906 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.907 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.907 INFO:teuthology.orchestra.run.smithi090.stdout: "upmap_max_deviation": { 2024-04-02T14:50:51.907 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "upmap_max_deviation", 2024-04-02T14:50:51.907 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-02T14:50:51.907 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.907 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.907 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "5", 2024-04-02T14:50:51.907 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "1", 2024-04-02T14:50:51.907 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.907 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.907 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "deviation below which no optimization is attempted", 2024-04-02T14:50:51.907 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-04-02T14:50:51.907 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.907 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.907 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.907 INFO:teuthology.orchestra.run.smithi090.stdout: "upmap_max_optimizations": { 2024-04-02T14:50:51.908 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "upmap_max_optimizations", 2024-04-02T14:50:51.908 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "uint", 2024-04-02T14:50:51.908 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.908 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.908 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "10", 2024-04-02T14:50:51.908 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.908 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.908 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.908 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-04-02T14:50:51.908 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.908 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.908 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.908 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:51.908 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:51.908 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.908 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-02T14:50:51.908 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "cephadm", 2024-04-02T14:50:51.909 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-02T14:50:51.909 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-02T14:50:51.909 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-02T14:50:51.909 INFO:teuthology.orchestra.run.smithi090.stdout: "agent_down_multiplier": { 2024-04-02T14:50:51.909 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "agent_down_multiplier", 2024-04-02T14:50:51.909 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "float", 2024-04-02T14:50:51.909 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.909 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.909 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "3.0", 2024-04-02T14:50:51.909 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.909 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.909 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.909 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-04-02T14:50:51.909 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.909 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.909 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.910 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.910 INFO:teuthology.orchestra.run.smithi090.stdout: "agent_refresh_rate": { 2024-04-02T14:50:51.910 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "agent_refresh_rate", 2024-04-02T14:50:51.910 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "secs", 2024-04-02T14:50:51.910 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.910 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.910 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "20", 2024-04-02T14:50:51.910 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.910 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.910 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.910 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-04-02T14:50:51.910 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.910 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.910 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.910 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.910 INFO:teuthology.orchestra.run.smithi090.stdout: "agent_starting_port": { 2024-04-02T14:50:51.911 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "agent_starting_port", 2024-04-02T14:50:51.911 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-02T14:50:51.911 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.911 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.911 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "4721", 2024-04-02T14:50:51.911 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.911 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.911 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.911 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-04-02T14:50:51.911 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.911 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.911 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.911 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.911 INFO:teuthology.orchestra.run.smithi090.stdout: "allow_ptrace": { 2024-04-02T14:50:51.911 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "allow_ptrace", 2024-04-02T14:50:51.911 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:51.912 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.912 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.912 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:51.912 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.912 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.912 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.912 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-04-02T14:50:51.912 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-04-02T14:50:51.912 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.912 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.912 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.912 INFO:teuthology.orchestra.run.smithi090.stdout: "autotune_interval": { 2024-04-02T14:50:51.912 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "autotune_interval", 2024-04-02T14:50:51.912 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "secs", 2024-04-02T14:50:51.912 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.912 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.912 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "600", 2024-04-02T14:50:51.913 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.913 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.913 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.913 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "how frequently to autotune daemon memory", 2024-04-02T14:50:51.913 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.913 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.913 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.913 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.913 INFO:teuthology.orchestra.run.smithi090.stdout: "autotune_memory_target_ratio": { 2024-04-02T14:50:51.913 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "autotune_memory_target_ratio", 2024-04-02T14:50:51.913 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "float", 2024-04-02T14:50:51.913 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.913 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.913 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "0.7", 2024-04-02T14:50:51.913 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.913 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.913 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.914 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-04-02T14:50:51.914 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.914 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.914 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.914 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.914 INFO:teuthology.orchestra.run.smithi090.stdout: "cephadm_log_destination": { 2024-04-02T14:50:51.914 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "cephadm_log_destination", 2024-04-02T14:50:51.914 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.914 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.914 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.914 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:51.914 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.914 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.914 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:51.914 INFO:teuthology.orchestra.run.smithi090.stdout: "file", 2024-04-02T14:50:51.915 INFO:teuthology.orchestra.run.smithi090.stdout: "file,syslog", 2024-04-02T14:50:51.915 INFO:teuthology.orchestra.run.smithi090.stdout: "syslog" 2024-04-02T14:50:51.915 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:51.915 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-04-02T14:50:51.915 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.915 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.915 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.915 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.915 INFO:teuthology.orchestra.run.smithi090.stdout: "cgroups_split": { 2024-04-02T14:50:51.915 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "cgroups_split", 2024-04-02T14:50:51.915 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:51.915 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.915 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.915 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-02T14:50:51.915 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.915 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.916 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.916 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-04-02T14:50:51.916 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.916 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.916 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.916 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.916 INFO:teuthology.orchestra.run.smithi090.stdout: "config_checks_enabled": { 2024-04-02T14:50:51.916 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "config_checks_enabled", 2024-04-02T14:50:51.916 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:51.916 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.916 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.916 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:51.916 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.916 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.916 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.917 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-04-02T14:50:51.917 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.917 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.917 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.917 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.917 INFO:teuthology.orchestra.run.smithi090.stdout: "config_dashboard": { 2024-04-02T14:50:51.917 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "config_dashboard", 2024-04-02T14:50:51.917 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:51.917 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.917 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.917 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-02T14:50:51.917 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.917 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.917 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.918 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-04-02T14:50:51.918 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.918 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.918 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.918 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.918 INFO:teuthology.orchestra.run.smithi090.stdout: "container_image_alertmanager": { 2024-04-02T14:50:51.918 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "container_image_alertmanager", 2024-04-02T14:50:51.918 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.918 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.918 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.918 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-04-02T14:50:51.918 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.918 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.918 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.919 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Prometheus container image", 2024-04-02T14:50:51.919 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.919 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.919 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.919 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.919 INFO:teuthology.orchestra.run.smithi090.stdout: "container_image_base": { 2024-04-02T14:50:51.919 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "container_image_base", 2024-04-02T14:50:51.919 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.919 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.919 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.919 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "quay.io/ceph/ceph", 2024-04-02T14:50:51.919 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.919 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.920 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.920 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Container image name, without the tag", 2024-04-02T14:50:51.920 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.920 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.920 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.920 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.920 INFO:teuthology.orchestra.run.smithi090.stdout: "container_image_elasticsearch": { 2024-04-02T14:50:51.920 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "container_image_elasticsearch", 2024-04-02T14:50:51.920 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.920 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.920 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.920 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-04-02T14:50:51.920 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.920 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.921 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.921 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "elasticsearch container image", 2024-04-02T14:50:51.921 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.921 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.921 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.921 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.921 INFO:teuthology.orchestra.run.smithi090.stdout: "container_image_grafana": { 2024-04-02T14:50:51.921 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "container_image_grafana", 2024-04-02T14:50:51.921 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.921 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.921 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.921 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-04-02T14:50:51.921 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.921 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.921 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.922 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Prometheus container image", 2024-04-02T14:50:51.922 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.922 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.922 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.922 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.922 INFO:teuthology.orchestra.run.smithi090.stdout: "container_image_haproxy": { 2024-04-02T14:50:51.922 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "container_image_haproxy", 2024-04-02T14:50:51.922 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.922 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.922 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.922 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-04-02T14:50:51.922 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.922 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.923 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.923 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "HAproxy container image", 2024-04-02T14:50:51.923 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.923 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.923 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.923 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.923 INFO:teuthology.orchestra.run.smithi090.stdout: "container_image_jaeger_agent": { 2024-04-02T14:50:51.923 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "container_image_jaeger_agent", 2024-04-02T14:50:51.923 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.923 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.923 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.923 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-04-02T14:50:51.923 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.923 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.924 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.924 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Jaeger agent container image", 2024-04-02T14:50:51.924 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.924 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.924 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.924 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.924 INFO:teuthology.orchestra.run.smithi090.stdout: "container_image_jaeger_collector": { 2024-04-02T14:50:51.924 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "container_image_jaeger_collector", 2024-04-02T14:50:51.924 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.924 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.924 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.924 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-04-02T14:50:51.924 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.924 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.925 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.925 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Jaeger collector container image", 2024-04-02T14:50:51.925 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.925 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.925 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.925 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.925 INFO:teuthology.orchestra.run.smithi090.stdout: "container_image_jaeger_query": { 2024-04-02T14:50:51.925 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "container_image_jaeger_query", 2024-04-02T14:50:51.925 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.925 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.925 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.925 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-04-02T14:50:51.925 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.926 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.926 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.926 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Jaeger query container image", 2024-04-02T14:50:51.926 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.926 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.926 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.926 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.926 INFO:teuthology.orchestra.run.smithi090.stdout: "container_image_keepalived": { 2024-04-02T14:50:51.926 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "container_image_keepalived", 2024-04-02T14:50:51.926 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.926 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.926 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.926 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-04-02T14:50:51.926 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.927 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.927 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.927 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Keepalived container image", 2024-04-02T14:50:51.927 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.927 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.927 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.927 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.927 INFO:teuthology.orchestra.run.smithi090.stdout: "container_image_loki": { 2024-04-02T14:50:51.927 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "container_image_loki", 2024-04-02T14:50:51.927 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.927 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.927 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.927 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-04-02T14:50:51.927 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.928 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.928 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.928 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Loki container image", 2024-04-02T14:50:51.928 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.928 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.928 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.928 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.928 INFO:teuthology.orchestra.run.smithi090.stdout: "container_image_node_exporter": { 2024-04-02T14:50:51.928 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "container_image_node_exporter", 2024-04-02T14:50:51.928 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.928 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.928 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.928 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-04-02T14:50:51.929 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.929 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.929 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.929 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Prometheus container image", 2024-04-02T14:50:51.929 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.929 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.929 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.929 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.929 INFO:teuthology.orchestra.run.smithi090.stdout: "container_image_nvmeof": { 2024-04-02T14:50:51.929 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "container_image_nvmeof", 2024-04-02T14:50:51.929 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.929 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.929 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.929 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "quay.io/ceph/nvmeof:1.0.0", 2024-04-02T14:50:51.929 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.929 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.930 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.930 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Nvme-of container image", 2024-04-02T14:50:51.930 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.930 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.930 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.930 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.930 INFO:teuthology.orchestra.run.smithi090.stdout: "container_image_prometheus": { 2024-04-02T14:50:51.930 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "container_image_prometheus", 2024-04-02T14:50:51.930 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.930 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.930 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.930 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-04-02T14:50:51.930 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.930 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.931 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.931 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Prometheus container image", 2024-04-02T14:50:51.931 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.931 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.931 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.931 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.931 INFO:teuthology.orchestra.run.smithi090.stdout: "container_image_promtail": { 2024-04-02T14:50:51.931 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "container_image_promtail", 2024-04-02T14:50:51.931 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.931 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.931 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.931 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-04-02T14:50:51.931 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.931 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.931 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.932 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Promtail container image", 2024-04-02T14:50:51.932 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.932 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.932 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.932 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.932 INFO:teuthology.orchestra.run.smithi090.stdout: "container_image_samba": { 2024-04-02T14:50:51.932 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "container_image_samba", 2024-04-02T14:50:51.932 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.932 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.932 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.932 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-04-02T14:50:51.932 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.932 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.932 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.933 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Samba/SMB container image", 2024-04-02T14:50:51.933 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.933 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.933 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.933 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.933 INFO:teuthology.orchestra.run.smithi090.stdout: "container_image_snmp_gateway": { 2024-04-02T14:50:51.933 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "container_image_snmp_gateway", 2024-04-02T14:50:51.933 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.933 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.933 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.933 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-04-02T14:50:51.933 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.933 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.933 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.933 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "SNMP Gateway container image", 2024-04-02T14:50:51.934 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.934 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.934 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.934 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.934 INFO:teuthology.orchestra.run.smithi090.stdout: "container_init": { 2024-04-02T14:50:51.934 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "container_init", 2024-04-02T14:50:51.934 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:51.934 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.934 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.934 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-02T14:50:51.934 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.934 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.934 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.934 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Run podman/docker with `--init`", 2024-04-02T14:50:51.935 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.935 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.935 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.935 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.935 INFO:teuthology.orchestra.run.smithi090.stdout: "daemon_cache_timeout": { 2024-04-02T14:50:51.935 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "daemon_cache_timeout", 2024-04-02T14:50:51.935 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "secs", 2024-04-02T14:50:51.935 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.935 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.935 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "600", 2024-04-02T14:50:51.935 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.935 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.935 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.935 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-04-02T14:50:51.935 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.936 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.936 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.936 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.936 INFO:teuthology.orchestra.run.smithi090.stdout: "default_cephadm_command_timeout": { 2024-04-02T14:50:51.936 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "default_cephadm_command_timeout", 2024-04-02T14:50:51.936 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-02T14:50:51.936 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.936 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.936 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "900", 2024-04-02T14:50:51.936 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.936 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.936 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.936 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-04-02T14:50:51.936 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.936 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.937 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.937 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.937 INFO:teuthology.orchestra.run.smithi090.stdout: "default_registry": { 2024-04-02T14:50:51.937 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "default_registry", 2024-04-02T14:50:51.937 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.937 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.937 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.937 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "docker.io", 2024-04-02T14:50:51.937 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.937 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.937 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.937 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-04-02T14:50:51.937 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.937 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.937 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.938 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.938 INFO:teuthology.orchestra.run.smithi090.stdout: "device_cache_timeout": { 2024-04-02T14:50:51.938 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "device_cache_timeout", 2024-04-02T14:50:51.938 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "secs", 2024-04-02T14:50:51.938 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.938 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.938 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "1800", 2024-04-02T14:50:51.938 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.938 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.938 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.938 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "seconds to cache device inventory", 2024-04-02T14:50:51.938 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.938 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.938 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.938 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.938 INFO:teuthology.orchestra.run.smithi090.stdout: "device_enhanced_scan": { 2024-04-02T14:50:51.939 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "device_enhanced_scan", 2024-04-02T14:50:51.939 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:51.939 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.939 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.939 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:51.939 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.939 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.939 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.939 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Use libstoragemgmt during device scans", 2024-04-02T14:50:51.939 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.939 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.939 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.939 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.939 INFO:teuthology.orchestra.run.smithi090.stdout: "facts_cache_timeout": { 2024-04-02T14:50:51.939 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "facts_cache_timeout", 2024-04-02T14:50:51.940 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "secs", 2024-04-02T14:50:51.940 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.940 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.940 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "60", 2024-04-02T14:50:51.940 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.940 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.940 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.940 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "seconds to cache host facts data", 2024-04-02T14:50:51.940 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.940 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.940 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.940 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.940 INFO:teuthology.orchestra.run.smithi090.stdout: "grafana_dashboards_path": { 2024-04-02T14:50:51.940 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "grafana_dashboards_path", 2024-04-02T14:50:51.941 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.941 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.941 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.941 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-04-02T14:50:51.941 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.941 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.941 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.941 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-04-02T14:50:51.941 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.941 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.941 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.941 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.941 INFO:teuthology.orchestra.run.smithi090.stdout: "host_check_interval": { 2024-04-02T14:50:51.941 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "host_check_interval", 2024-04-02T14:50:51.941 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "secs", 2024-04-02T14:50:51.942 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.942 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.942 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "600", 2024-04-02T14:50:51.942 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.942 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.942 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.942 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "how frequently to perform a host check", 2024-04-02T14:50:51.942 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.942 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.942 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.942 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.942 INFO:teuthology.orchestra.run.smithi090.stdout: "hw_monitoring": { 2024-04-02T14:50:51.942 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "hw_monitoring", 2024-04-02T14:50:51.942 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:51.943 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.943 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.943 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:51.943 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.943 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.943 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.943 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-04-02T14:50:51.943 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.943 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.943 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.943 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.943 INFO:teuthology.orchestra.run.smithi090.stdout: "inventory_list_all": { 2024-04-02T14:50:51.943 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "inventory_list_all", 2024-04-02T14:50:51.943 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:51.943 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.944 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.944 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:51.944 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.944 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.944 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.944 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-04-02T14:50:51.944 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.944 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.944 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.944 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.944 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-02T14:50:51.944 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-02T14:50:51.944 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.944 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.944 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.945 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:51.945 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.945 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.945 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:51.945 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:51.945 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:51.945 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:51.945 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:51.945 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:51.945 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:51.945 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:51.945 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.945 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.945 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.946 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.946 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.946 INFO:teuthology.orchestra.run.smithi090.stdout: "log_refresh_metadata": { 2024-04-02T14:50:51.946 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_refresh_metadata", 2024-04-02T14:50:51.946 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:51.946 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.946 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.946 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:51.946 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.946 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.946 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.946 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-04-02T14:50:51.947 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.947 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.947 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.947 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.947 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-02T14:50:51.947 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-02T14:50:51.947 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:51.947 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.947 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.947 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-02T14:50:51.947 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.947 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.947 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.948 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-04-02T14:50:51.948 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.948 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.948 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.948 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.948 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-02T14:50:51.948 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-02T14:50:51.948 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.948 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.948 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.948 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-02T14:50:51.948 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.948 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.948 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:51.949 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:51.949 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:51.949 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:51.949 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:51.949 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:51.949 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:51.949 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:51.949 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.949 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.949 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.949 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.949 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.949 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-02T14:50:51.950 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-02T14:50:51.950 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:51.950 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.950 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.950 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:51.950 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.950 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.950 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.950 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.950 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.950 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.950 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.950 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.951 INFO:teuthology.orchestra.run.smithi090.stdout: "manage_etc_ceph_ceph_conf": { 2024-04-02T14:50:51.951 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-04-02T14:50:51.951 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:51.951 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.951 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.951 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:51.951 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.951 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.951 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.951 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-04-02T14:50:51.951 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.951 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.951 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.951 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.952 INFO:teuthology.orchestra.run.smithi090.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-04-02T14:50:51.952 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-04-02T14:50:51.952 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.952 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.952 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.952 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "*", 2024-04-02T14:50:51.952 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.952 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.952 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.952 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-04-02T14:50:51.952 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.952 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.952 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.952 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.952 INFO:teuthology.orchestra.run.smithi090.stdout: "max_count_per_host": { 2024-04-02T14:50:51.953 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "max_count_per_host", 2024-04-02T14:50:51.953 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-02T14:50:51.953 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.953 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.953 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "10", 2024-04-02T14:50:51.953 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.953 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.953 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.953 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "max number of daemons per service per host", 2024-04-02T14:50:51.953 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.953 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.953 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.953 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.953 INFO:teuthology.orchestra.run.smithi090.stdout: "max_osd_draining_count": { 2024-04-02T14:50:51.954 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "max_osd_draining_count", 2024-04-02T14:50:51.954 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-02T14:50:51.954 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.954 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.954 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "10", 2024-04-02T14:50:51.954 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.954 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.954 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.954 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-04-02T14:50:51.954 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.954 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.954 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.954 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.954 INFO:teuthology.orchestra.run.smithi090.stdout: "migration_current": { 2024-04-02T14:50:51.955 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "migration_current", 2024-04-02T14:50:51.955 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-02T14:50:51.955 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.955 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.955 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:51.955 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.955 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.955 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.955 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "internal - do not modify", 2024-04-02T14:50:51.955 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.955 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.955 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.955 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.955 INFO:teuthology.orchestra.run.smithi090.stdout: "mode": { 2024-04-02T14:50:51.955 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "mode", 2024-04-02T14:50:51.956 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.956 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.956 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.956 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "root", 2024-04-02T14:50:51.956 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.956 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.956 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:51.956 INFO:teuthology.orchestra.run.smithi090.stdout: "cephadm-package", 2024-04-02T14:50:51.956 INFO:teuthology.orchestra.run.smithi090.stdout: "root" 2024-04-02T14:50:51.956 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:51.956 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "mode for remote execution of cephadm", 2024-04-02T14:50:51.956 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.956 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.956 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.956 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.956 INFO:teuthology.orchestra.run.smithi090.stdout: "oob_default_addr": { 2024-04-02T14:50:51.956 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "oob_default_addr", 2024-04-02T14:50:51.957 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.957 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.957 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.957 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "169.254.1.1", 2024-04-02T14:50:51.957 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.957 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.957 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.957 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Default address for RedFish API (oob management).", 2024-04-02T14:50:51.957 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.957 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.957 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.957 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.957 INFO:teuthology.orchestra.run.smithi090.stdout: "prometheus_alerts_path": { 2024-04-02T14:50:51.957 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "prometheus_alerts_path", 2024-04-02T14:50:51.958 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.958 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.958 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.958 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-04-02T14:50:51.958 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.958 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.958 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.958 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-04-02T14:50:51.958 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.958 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.958 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.958 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.958 INFO:teuthology.orchestra.run.smithi090.stdout: "registry_insecure": { 2024-04-02T14:50:51.958 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "registry_insecure", 2024-04-02T14:50:51.959 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:51.959 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.959 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.959 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:51.959 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.959 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.959 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.959 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-04-02T14:50:51.959 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.959 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.959 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.959 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.959 INFO:teuthology.orchestra.run.smithi090.stdout: "registry_password": { 2024-04-02T14:50:51.959 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "registry_password", 2024-04-02T14:50:51.959 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.959 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.959 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.960 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:51.960 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.960 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.960 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.960 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-04-02T14:50:51.960 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.960 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.960 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.960 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.960 INFO:teuthology.orchestra.run.smithi090.stdout: "registry_url": { 2024-04-02T14:50:51.960 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "registry_url", 2024-04-02T14:50:51.960 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.960 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.961 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.961 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:51.961 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.961 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.961 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.961 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-04-02T14:50:51.961 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.961 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.961 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.961 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.961 INFO:teuthology.orchestra.run.smithi090.stdout: "registry_username": { 2024-04-02T14:50:51.961 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "registry_username", 2024-04-02T14:50:51.961 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.961 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.961 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.962 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:51.962 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.962 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.962 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.962 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-04-02T14:50:51.962 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.962 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.962 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.962 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.962 INFO:teuthology.orchestra.run.smithi090.stdout: "secure_monitoring_stack": { 2024-04-02T14:50:51.962 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "secure_monitoring_stack", 2024-04-02T14:50:51.962 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:51.962 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.962 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.962 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:51.963 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.963 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.963 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.963 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-04-02T14:50:51.963 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.963 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.963 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.963 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.963 INFO:teuthology.orchestra.run.smithi090.stdout: "service_discovery_port": { 2024-04-02T14:50:51.963 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "service_discovery_port", 2024-04-02T14:50:51.963 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-02T14:50:51.963 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.963 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.963 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "8765", 2024-04-02T14:50:51.963 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.964 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.964 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.964 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "cephadm service discovery port", 2024-04-02T14:50:51.964 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.964 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.964 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.964 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.964 INFO:teuthology.orchestra.run.smithi090.stdout: "ssh_config_file": { 2024-04-02T14:50:51.964 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "ssh_config_file", 2024-04-02T14:50:51.964 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.964 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.964 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.964 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:51.964 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.965 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.965 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.965 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-04-02T14:50:51.965 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.965 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.965 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.965 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.965 INFO:teuthology.orchestra.run.smithi090.stdout: "use_agent": { 2024-04-02T14:50:51.965 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "use_agent", 2024-04-02T14:50:51.965 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:51.965 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.965 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.965 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:51.965 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.965 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.965 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.966 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-04-02T14:50:51.966 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.966 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.966 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.966 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.966 INFO:teuthology.orchestra.run.smithi090.stdout: "use_repo_digest": { 2024-04-02T14:50:51.966 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "use_repo_digest", 2024-04-02T14:50:51.966 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:51.966 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.966 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.966 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-02T14:50:51.966 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.966 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.966 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.966 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-04-02T14:50:51.967 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.967 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.967 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.967 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.967 INFO:teuthology.orchestra.run.smithi090.stdout: "warn_on_failed_host_check": { 2024-04-02T14:50:51.967 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "warn_on_failed_host_check", 2024-04-02T14:50:51.967 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:51.967 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.967 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.967 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-02T14:50:51.967 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.967 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.967 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.967 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "raise a health warning if the host check fails", 2024-04-02T14:50:51.967 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.968 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.968 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.968 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.968 INFO:teuthology.orchestra.run.smithi090.stdout: "warn_on_stray_daemons": { 2024-04-02T14:50:51.968 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "warn_on_stray_daemons", 2024-04-02T14:50:51.968 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:51.968 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.968 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.968 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-02T14:50:51.968 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.968 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.968 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.968 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-04-02T14:50:51.968 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.968 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.969 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.969 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.969 INFO:teuthology.orchestra.run.smithi090.stdout: "warn_on_stray_hosts": { 2024-04-02T14:50:51.969 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "warn_on_stray_hosts", 2024-04-02T14:50:51.969 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:51.969 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.969 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.969 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-02T14:50:51.969 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.969 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.969 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.969 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-04-02T14:50:51.969 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.969 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.969 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.970 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:51.970 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:51.970 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.970 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-02T14:50:51.970 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "crash", 2024-04-02T14:50:51.970 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-02T14:50:51.970 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-02T14:50:51.970 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-02T14:50:51.970 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-02T14:50:51.970 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-02T14:50:51.970 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.971 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.971 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.971 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:51.971 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.971 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.971 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:51.971 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:51.971 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:51.971 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:51.971 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:51.971 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:51.971 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:51.971 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:51.971 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.971 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.971 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.972 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.972 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.972 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-02T14:50:51.972 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-02T14:50:51.972 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:51.972 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.972 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.972 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:51.972 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.972 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.972 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.972 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.972 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.973 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.973 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.973 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.973 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-02T14:50:51.973 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-02T14:50:51.973 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.973 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.973 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.973 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-02T14:50:51.973 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.973 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.973 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:51.973 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:51.974 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:51.974 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:51.974 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:51.974 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:51.974 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:51.974 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:51.974 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.974 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.974 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.974 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.974 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.974 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-02T14:50:51.974 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-02T14:50:51.974 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:51.975 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.975 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.975 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:51.975 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.975 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.975 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.975 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.975 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.975 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.975 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.975 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.975 INFO:teuthology.orchestra.run.smithi090.stdout: "retain_interval": { 2024-04-02T14:50:51.975 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "retain_interval", 2024-04-02T14:50:51.975 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "secs", 2024-04-02T14:50:51.976 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.976 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.976 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "31536000", 2024-04-02T14:50:51.976 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.976 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.976 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.976 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "how long to retain crashes before pruning them", 2024-04-02T14:50:51.976 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.976 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.976 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.976 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.976 INFO:teuthology.orchestra.run.smithi090.stdout: "warn_recent_interval": { 2024-04-02T14:50:51.976 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "warn_recent_interval", 2024-04-02T14:50:51.976 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "secs", 2024-04-02T14:50:51.977 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.977 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:51.977 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "1209600", 2024-04-02T14:50:51.977 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.977 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.977 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.977 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "time interval in which to warn about recent crashes", 2024-04-02T14:50:51.977 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.977 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.977 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.977 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:51.977 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:51.977 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.977 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-02T14:50:51.977 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "dashboard", 2024-04-02T14:50:51.977 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-02T14:50:51.978 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-02T14:50:51.978 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-02T14:50:51.978 INFO:teuthology.orchestra.run.smithi090.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-04-02T14:50:51.978 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-04-02T14:50:51.978 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-02T14:50:51.978 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.978 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.978 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "10", 2024-04-02T14:50:51.978 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.978 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.978 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.978 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.978 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.978 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.979 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.979 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.979 INFO:teuthology.orchestra.run.smithi090.stdout: "ALERTMANAGER_API_HOST": { 2024-04-02T14:50:51.979 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "ALERTMANAGER_API_HOST", 2024-04-02T14:50:51.979 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.979 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.979 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.979 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:51.979 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.979 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.979 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.979 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.979 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.979 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.980 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.980 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.980 INFO:teuthology.orchestra.run.smithi090.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-04-02T14:50:51.980 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-04-02T14:50:51.980 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:51.980 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.980 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.980 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-02T14:50:51.980 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.980 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.980 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.980 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.980 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.980 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.980 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.980 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.981 INFO:teuthology.orchestra.run.smithi090.stdout: "AUDIT_API_ENABLED": { 2024-04-02T14:50:51.981 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "AUDIT_API_ENABLED", 2024-04-02T14:50:51.981 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:51.981 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.981 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.981 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:51.981 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.981 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.981 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.981 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.981 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.981 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.981 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.982 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.982 INFO:teuthology.orchestra.run.smithi090.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-04-02T14:50:51.982 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-04-02T14:50:51.982 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:51.982 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.982 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.982 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-02T14:50:51.982 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.982 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.982 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.982 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.982 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.983 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.983 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.983 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.983 INFO:teuthology.orchestra.run.smithi090.stdout: "ENABLE_BROWSABLE_API": { 2024-04-02T14:50:51.983 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "ENABLE_BROWSABLE_API", 2024-04-02T14:50:51.983 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:51.983 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.983 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.983 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-02T14:50:51.983 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.983 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.983 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.983 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.983 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.984 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.984 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.984 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.984 INFO:teuthology.orchestra.run.smithi090.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-04-02T14:50:51.984 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-04-02T14:50:51.984 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:51.984 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.984 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.984 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-02T14:50:51.984 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.984 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.984 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.984 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.984 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.985 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.985 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.985 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.985 INFO:teuthology.orchestra.run.smithi090.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-04-02T14:50:51.985 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-04-02T14:50:51.985 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:51.985 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.985 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.985 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-02T14:50:51.985 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.985 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.985 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.985 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.986 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.986 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.986 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.986 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.986 INFO:teuthology.orchestra.run.smithi090.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-04-02T14:50:51.986 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-04-02T14:50:51.986 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:51.986 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.986 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.986 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-02T14:50:51.986 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.986 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.986 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.986 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.987 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.987 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.987 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.987 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.987 INFO:teuthology.orchestra.run.smithi090.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-04-02T14:50:51.987 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-04-02T14:50:51.987 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:51.987 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.987 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.987 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-02T14:50:51.987 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.987 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.987 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.987 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.988 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.988 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.988 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.988 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.988 INFO:teuthology.orchestra.run.smithi090.stdout: "FEATURE_TOGGLE_NFS": { 2024-04-02T14:50:51.988 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-04-02T14:50:51.988 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:51.988 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.988 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.988 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-02T14:50:51.988 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.988 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.988 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.988 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.988 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.989 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.989 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.989 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.989 INFO:teuthology.orchestra.run.smithi090.stdout: "FEATURE_TOGGLE_RBD": { 2024-04-02T14:50:51.989 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-04-02T14:50:51.989 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:51.989 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.989 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.989 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-02T14:50:51.989 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.989 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.989 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.989 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.989 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.989 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.990 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.990 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.990 INFO:teuthology.orchestra.run.smithi090.stdout: "FEATURE_TOGGLE_RGW": { 2024-04-02T14:50:51.990 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-04-02T14:50:51.990 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:51.990 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.990 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.990 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-02T14:50:51.990 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.990 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.990 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.990 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.990 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.990 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.991 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.991 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.991 INFO:teuthology.orchestra.run.smithi090.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-04-02T14:50:51.991 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-04-02T14:50:51.991 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.991 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.991 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.991 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:51.991 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.991 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.991 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.991 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.991 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.991 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.991 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.992 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.992 INFO:teuthology.orchestra.run.smithi090.stdout: "GRAFANA_API_PASSWORD": { 2024-04-02T14:50:51.992 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "GRAFANA_API_PASSWORD", 2024-04-02T14:50:51.992 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.992 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.992 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.992 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "admin", 2024-04-02T14:50:51.992 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.992 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.992 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.992 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.992 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.993 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.993 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.993 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.993 INFO:teuthology.orchestra.run.smithi090.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-04-02T14:50:51.993 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-04-02T14:50:51.993 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:51.993 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.993 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.993 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-02T14:50:51.993 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.993 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.993 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.993 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.993 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.993 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.994 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.994 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.994 INFO:teuthology.orchestra.run.smithi090.stdout: "GRAFANA_API_URL": { 2024-04-02T14:50:51.994 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "GRAFANA_API_URL", 2024-04-02T14:50:51.994 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.994 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.994 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.994 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:51.994 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.994 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.994 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.994 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.994 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.994 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.995 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.995 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.995 INFO:teuthology.orchestra.run.smithi090.stdout: "GRAFANA_API_USERNAME": { 2024-04-02T14:50:51.995 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "GRAFANA_API_USERNAME", 2024-04-02T14:50:51.995 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.995 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.995 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.995 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "admin", 2024-04-02T14:50:51.995 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.995 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.995 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.995 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.995 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.996 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.996 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.996 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.996 INFO:teuthology.orchestra.run.smithi090.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-04-02T14:50:51.996 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-04-02T14:50:51.996 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.996 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.996 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.996 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:51.996 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.996 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.996 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.996 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.996 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.997 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.997 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.997 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.997 INFO:teuthology.orchestra.run.smithi090.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-04-02T14:50:51.997 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-04-02T14:50:51.997 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:51.997 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.997 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.997 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:51.997 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.997 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.997 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.997 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.997 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.997 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.998 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.998 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.998 INFO:teuthology.orchestra.run.smithi090.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-04-02T14:50:51.998 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-04-02T14:50:51.998 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:51.998 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.998 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.998 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-02T14:50:51.998 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.998 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.998 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.998 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.998 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:51.999 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:51.999 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:51.999 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:51.999 INFO:teuthology.orchestra.run.smithi090.stdout: "ISSUE_TRACKER_API_KEY": { 2024-04-02T14:50:51.999 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-04-02T14:50:51.999 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:51.999 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:51.999 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:51.999 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:51.999 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:51.999 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:51.999 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:51.999 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:51.999 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.000 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.000 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.000 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.000 INFO:teuthology.orchestra.run.smithi090.stdout: "MANAGED_BY_CLUSTERS": { 2024-04-02T14:50:52.000 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "MANAGED_BY_CLUSTERS", 2024-04-02T14:50:52.000 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.000 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.000 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.000 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "[]", 2024-04-02T14:50:52.000 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.000 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.000 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.000 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.000 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.001 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.001 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.001 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.001 INFO:teuthology.orchestra.run.smithi090.stdout: "MULTICLUSTER_CONFIG": { 2024-04-02T14:50:52.001 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "MULTICLUSTER_CONFIG", 2024-04-02T14:50:52.001 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.001 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.001 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.001 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "{}", 2024-04-02T14:50:52.001 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.001 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.001 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.001 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.001 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.001 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.002 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.002 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.002 INFO:teuthology.orchestra.run.smithi090.stdout: "PROMETHEUS_API_HOST": { 2024-04-02T14:50:52.002 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "PROMETHEUS_API_HOST", 2024-04-02T14:50:52.002 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.002 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.002 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.002 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:52.002 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.002 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.002 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.002 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.002 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.003 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.003 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.003 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.003 INFO:teuthology.orchestra.run.smithi090.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-04-02T14:50:52.003 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-04-02T14:50:52.003 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.003 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.003 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.003 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-02T14:50:52.003 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.003 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.003 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.003 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.003 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.003 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.004 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.004 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.004 INFO:teuthology.orchestra.run.smithi090.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-04-02T14:50:52.004 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-04-02T14:50:52.004 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.004 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.004 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.004 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:52.004 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.004 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.004 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.004 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.004 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.004 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.004 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.004 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.005 INFO:teuthology.orchestra.run.smithi090.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-04-02T14:50:52.005 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-04-02T14:50:52.005 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.005 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.005 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.005 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:52.005 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.005 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.005 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.005 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.005 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.005 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.005 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.005 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.005 INFO:teuthology.orchestra.run.smithi090.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-04-02T14:50:52.006 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-04-02T14:50:52.006 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.006 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.006 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.006 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-02T14:50:52.006 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.006 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.006 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.006 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.006 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.006 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.006 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.006 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.006 INFO:teuthology.orchestra.run.smithi090.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-04-02T14:50:52.006 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-04-02T14:50:52.006 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.007 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.007 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.007 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-02T14:50:52.007 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.007 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.007 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.007 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.007 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.007 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.007 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.007 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.007 INFO:teuthology.orchestra.run.smithi090.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-04-02T14:50:52.007 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-04-02T14:50:52.007 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.007 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.007 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.008 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:52.008 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.008 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.008 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.008 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.008 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.008 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.008 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.008 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.008 INFO:teuthology.orchestra.run.smithi090.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-04-02T14:50:52.008 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-04-02T14:50:52.008 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.008 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.008 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.008 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:52.009 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.009 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.009 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.009 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.009 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.009 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.009 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.009 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.009 INFO:teuthology.orchestra.run.smithi090.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-04-02T14:50:52.009 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-04-02T14:50:52.009 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.009 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.009 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.009 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:52.009 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.009 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.010 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.010 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.010 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.010 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.010 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.010 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.010 INFO:teuthology.orchestra.run.smithi090.stdout: "PWD_POLICY_ENABLED": { 2024-04-02T14:50:52.010 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "PWD_POLICY_ENABLED", 2024-04-02T14:50:52.010 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.010 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.010 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.010 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-02T14:50:52.010 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.010 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.010 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.010 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.011 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.011 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.011 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.011 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.011 INFO:teuthology.orchestra.run.smithi090.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-04-02T14:50:52.011 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-04-02T14:50:52.011 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.011 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.011 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.011 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-04-02T14:50:52.011 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.011 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.011 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.011 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.011 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.011 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.012 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.012 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.012 INFO:teuthology.orchestra.run.smithi090.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-04-02T14:50:52.012 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-04-02T14:50:52.012 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-02T14:50:52.012 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.012 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.012 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "10", 2024-04-02T14:50:52.012 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.012 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.012 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.012 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.012 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.012 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.012 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.012 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.012 INFO:teuthology.orchestra.run.smithi090.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-04-02T14:50:52.013 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-04-02T14:50:52.013 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-02T14:50:52.013 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.013 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.013 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "8", 2024-04-02T14:50:52.013 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.013 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.013 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.013 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.013 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.013 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.013 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.013 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.013 INFO:teuthology.orchestra.run.smithi090.stdout: "REST_REQUESTS_TIMEOUT": { 2024-04-02T14:50:52.013 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-04-02T14:50:52.013 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-02T14:50:52.013 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.014 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.014 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "45", 2024-04-02T14:50:52.014 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.014 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.014 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.014 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.014 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.014 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.014 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.014 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.014 INFO:teuthology.orchestra.run.smithi090.stdout: "RGW_API_ACCESS_KEY": { 2024-04-02T14:50:52.014 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "RGW_API_ACCESS_KEY", 2024-04-02T14:50:52.014 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.014 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.014 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.014 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:52.014 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.015 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.015 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.015 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.015 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.015 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.015 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.015 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.015 INFO:teuthology.orchestra.run.smithi090.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-04-02T14:50:52.015 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-04-02T14:50:52.015 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.015 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.015 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.015 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "admin", 2024-04-02T14:50:52.015 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.015 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.015 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.016 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.016 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.016 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.016 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.016 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.016 INFO:teuthology.orchestra.run.smithi090.stdout: "RGW_API_SECRET_KEY": { 2024-04-02T14:50:52.016 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "RGW_API_SECRET_KEY", 2024-04-02T14:50:52.016 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.016 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.016 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.016 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:52.016 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.016 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.016 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.016 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.016 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.016 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.016 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.017 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.017 INFO:teuthology.orchestra.run.smithi090.stdout: "RGW_API_SSL_VERIFY": { 2024-04-02T14:50:52.017 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "RGW_API_SSL_VERIFY", 2024-04-02T14:50:52.017 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.017 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.017 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.017 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-02T14:50:52.017 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.017 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.017 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.017 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.017 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.017 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.017 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.017 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.017 INFO:teuthology.orchestra.run.smithi090.stdout: "UNSAFE_TLS_v1_2": { 2024-04-02T14:50:52.018 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "UNSAFE_TLS_v1_2", 2024-04-02T14:50:52.018 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.018 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.018 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.018 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:52.018 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.018 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.018 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.018 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.018 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.018 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.018 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.018 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.018 INFO:teuthology.orchestra.run.smithi090.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-04-02T14:50:52.018 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-04-02T14:50:52.018 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-02T14:50:52.019 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.019 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.019 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "0", 2024-04-02T14:50:52.019 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.019 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.019 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.019 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.019 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.019 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.019 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.019 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.019 INFO:teuthology.orchestra.run.smithi090.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-04-02T14:50:52.019 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-04-02T14:50:52.019 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-02T14:50:52.019 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.019 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.020 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "10", 2024-04-02T14:50:52.020 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.020 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.020 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.020 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.020 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.020 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.020 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.020 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.020 INFO:teuthology.orchestra.run.smithi090.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-04-02T14:50:52.020 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-04-02T14:50:52.020 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-02T14:50:52.020 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.020 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.020 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "5", 2024-04-02T14:50:52.021 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.021 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.021 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.021 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.021 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.021 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.021 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.021 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.021 INFO:teuthology.orchestra.run.smithi090.stdout: "cross_origin_url": { 2024-04-02T14:50:52.021 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "cross_origin_url", 2024-04-02T14:50:52.021 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.021 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.021 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.021 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:52.021 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.021 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.022 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.022 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.022 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.022 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.022 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.022 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.022 INFO:teuthology.orchestra.run.smithi090.stdout: "crt_file": { 2024-04-02T14:50:52.022 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "crt_file", 2024-04-02T14:50:52.022 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.022 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.022 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.022 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:52.022 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.023 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.023 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.023 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.023 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.023 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.023 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.023 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.023 INFO:teuthology.orchestra.run.smithi090.stdout: "debug": { 2024-04-02T14:50:52.023 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "debug", 2024-04-02T14:50:52.023 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.023 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.023 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.023 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:52.023 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.023 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.023 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.023 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Enable/disable debug options", 2024-04-02T14:50:52.024 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.024 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.024 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.024 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.024 INFO:teuthology.orchestra.run.smithi090.stdout: "jwt_token_ttl": { 2024-04-02T14:50:52.024 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "jwt_token_ttl", 2024-04-02T14:50:52.024 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-02T14:50:52.024 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.024 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.024 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "28800", 2024-04-02T14:50:52.024 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.024 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.024 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.024 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.024 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.024 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.024 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.025 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.025 INFO:teuthology.orchestra.run.smithi090.stdout: "key_file": { 2024-04-02T14:50:52.025 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "key_file", 2024-04-02T14:50:52.025 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.025 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.025 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.025 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:52.025 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.025 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.025 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.025 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.025 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.025 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.025 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.025 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.025 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-02T14:50:52.025 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-02T14:50:52.026 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.026 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.026 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.026 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:52.026 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.026 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.026 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:52.026 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:52.026 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:52.026 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:52.026 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:52.026 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:52.026 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:52.026 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:52.026 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.027 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.027 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.027 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.027 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.027 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-02T14:50:52.027 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-02T14:50:52.027 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.027 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.027 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.027 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:52.027 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.027 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.027 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.027 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.027 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.028 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.028 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.028 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.028 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-02T14:50:52.028 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-02T14:50:52.028 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.028 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.028 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.028 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-02T14:50:52.028 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.028 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.028 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:52.028 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:52.028 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:52.028 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:52.029 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:52.029 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:52.029 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:52.029 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:52.029 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.029 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.029 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.029 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.029 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.029 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-02T14:50:52.029 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-02T14:50:52.029 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.029 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.029 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.029 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:52.029 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.030 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.030 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.030 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.030 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.030 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.030 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.030 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.030 INFO:teuthology.orchestra.run.smithi090.stdout: "motd": { 2024-04-02T14:50:52.030 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "motd", 2024-04-02T14:50:52.030 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.030 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.030 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.030 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:52.030 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.031 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.031 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.031 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "The message of the day", 2024-04-02T14:50:52.031 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.031 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.031 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.031 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.031 INFO:teuthology.orchestra.run.smithi090.stdout: "redirect_resolve_ip_addr": { 2024-04-02T14:50:52.031 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "redirect_resolve_ip_addr", 2024-04-02T14:50:52.031 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.031 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.031 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.031 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:52.031 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.032 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.032 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.032 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.032 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.032 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.032 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.032 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.032 INFO:teuthology.orchestra.run.smithi090.stdout: "server_addr": { 2024-04-02T14:50:52.032 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "server_addr", 2024-04-02T14:50:52.032 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.032 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.032 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.032 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "::", 2024-04-02T14:50:52.032 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.033 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.033 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.033 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.033 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.033 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.033 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.033 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.033 INFO:teuthology.orchestra.run.smithi090.stdout: "server_port": { 2024-04-02T14:50:52.033 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "server_port", 2024-04-02T14:50:52.033 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-02T14:50:52.033 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.033 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.033 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "8080", 2024-04-02T14:50:52.033 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.033 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.034 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.034 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.034 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.034 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.034 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.034 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.034 INFO:teuthology.orchestra.run.smithi090.stdout: "ssl": { 2024-04-02T14:50:52.034 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "ssl", 2024-04-02T14:50:52.034 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.034 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.034 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.034 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-02T14:50:52.034 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.034 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.035 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.035 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.035 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.035 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.038 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.038 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.038 INFO:teuthology.orchestra.run.smithi090.stdout: "ssl_server_port": { 2024-04-02T14:50:52.038 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "ssl_server_port", 2024-04-02T14:50:52.039 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-02T14:50:52.039 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.039 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.039 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "8443", 2024-04-02T14:50:52.039 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.039 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.039 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.039 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.039 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.039 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.039 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.039 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.039 INFO:teuthology.orchestra.run.smithi090.stdout: "standby_behaviour": { 2024-04-02T14:50:52.039 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "standby_behaviour", 2024-04-02T14:50:52.040 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.040 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.040 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.040 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "redirect", 2024-04-02T14:50:52.040 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.040 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.040 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:52.040 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:52.040 INFO:teuthology.orchestra.run.smithi090.stdout: "redirect" 2024-04-02T14:50:52.040 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:52.040 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.040 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.040 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.040 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.041 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.041 INFO:teuthology.orchestra.run.smithi090.stdout: "standby_error_status_code": { 2024-04-02T14:50:52.041 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "standby_error_status_code", 2024-04-02T14:50:52.041 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-02T14:50:52.041 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.041 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.041 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "500", 2024-04-02T14:50:52.041 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "400", 2024-04-02T14:50:52.041 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "599", 2024-04-02T14:50:52.041 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.041 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.041 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.041 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.041 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.041 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.042 INFO:teuthology.orchestra.run.smithi090.stdout: "url_prefix": { 2024-04-02T14:50:52.042 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "url_prefix", 2024-04-02T14:50:52.042 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.042 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.042 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.042 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:52.042 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.042 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.042 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.042 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.042 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.042 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.042 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.042 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:52.042 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:52.043 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.043 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-02T14:50:52.043 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "devicehealth", 2024-04-02T14:50:52.043 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-02T14:50:52.043 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-02T14:50:52.043 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-02T14:50:52.043 INFO:teuthology.orchestra.run.smithi090.stdout: "enable_monitoring": { 2024-04-02T14:50:52.043 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "enable_monitoring", 2024-04-02T14:50:52.043 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.043 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.043 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.043 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-02T14:50:52.043 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.043 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.043 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.044 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "monitor device health metrics", 2024-04-02T14:50:52.044 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.044 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.044 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.044 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.044 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-02T14:50:52.044 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-02T14:50:52.044 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.044 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.044 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.044 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:52.044 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.044 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.045 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:52.045 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:52.045 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:52.045 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:52.045 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:52.045 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:52.045 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:52.045 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:52.045 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.045 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.045 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.045 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.045 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.045 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-02T14:50:52.045 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-02T14:50:52.046 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.046 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.046 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.046 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:52.046 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.046 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.046 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.046 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.046 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.046 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.046 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.046 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.046 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-02T14:50:52.047 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-02T14:50:52.047 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.047 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.047 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.047 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-02T14:50:52.047 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.047 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.047 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:52.047 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:52.047 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:52.047 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:52.047 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:52.047 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:52.047 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:52.047 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:52.048 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.048 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.048 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.048 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.048 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.048 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-02T14:50:52.048 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-02T14:50:52.048 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.048 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.048 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.048 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:52.048 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.048 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.048 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.048 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.049 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.049 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.049 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.049 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.049 INFO:teuthology.orchestra.run.smithi090.stdout: "mark_out_threshold": { 2024-04-02T14:50:52.049 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "mark_out_threshold", 2024-04-02T14:50:52.049 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "secs", 2024-04-02T14:50:52.049 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.049 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.049 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "2419200", 2024-04-02T14:50:52.049 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.049 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.049 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.050 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-04-02T14:50:52.050 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.050 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.050 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.050 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.050 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_name": { 2024-04-02T14:50:52.050 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "pool_name", 2024-04-02T14:50:52.050 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.050 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.050 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.050 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "device_health_metrics", 2024-04-02T14:50:52.050 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.050 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.051 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.051 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "name of pool in which to store device health metrics", 2024-04-02T14:50:52.051 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.051 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.051 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.051 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.051 INFO:teuthology.orchestra.run.smithi090.stdout: "retention_period": { 2024-04-02T14:50:52.051 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "retention_period", 2024-04-02T14:50:52.051 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "secs", 2024-04-02T14:50:52.051 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.051 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.051 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "15552000", 2024-04-02T14:50:52.051 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.051 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.051 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.052 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "how long to retain device health metrics", 2024-04-02T14:50:52.052 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.052 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.052 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.052 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.052 INFO:teuthology.orchestra.run.smithi090.stdout: "scrape_frequency": { 2024-04-02T14:50:52.052 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "scrape_frequency", 2024-04-02T14:50:52.052 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "secs", 2024-04-02T14:50:52.052 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.052 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.052 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "86400", 2024-04-02T14:50:52.052 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.052 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.052 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.053 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "how frequently to scrape device health metrics", 2024-04-02T14:50:52.053 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.053 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.053 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.053 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.053 INFO:teuthology.orchestra.run.smithi090.stdout: "self_heal": { 2024-04-02T14:50:52.053 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "self_heal", 2024-04-02T14:50:52.053 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.053 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.053 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.053 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-02T14:50:52.053 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.053 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.053 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.054 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-04-02T14:50:52.054 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.054 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.054 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.054 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.054 INFO:teuthology.orchestra.run.smithi090.stdout: "sleep_interval": { 2024-04-02T14:50:52.054 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "sleep_interval", 2024-04-02T14:50:52.054 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "secs", 2024-04-02T14:50:52.054 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.054 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.054 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "600", 2024-04-02T14:50:52.054 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.054 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.054 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.055 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "how frequently to wake up and check device health", 2024-04-02T14:50:52.055 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.055 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.055 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.055 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.055 INFO:teuthology.orchestra.run.smithi090.stdout: "warn_threshold": { 2024-04-02T14:50:52.055 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "warn_threshold", 2024-04-02T14:50:52.055 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "secs", 2024-04-02T14:50:52.055 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.055 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.055 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "7257600", 2024-04-02T14:50:52.055 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.055 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.056 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.056 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-04-02T14:50:52.056 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.056 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.056 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.056 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:52.056 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:52.056 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.056 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-02T14:50:52.056 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "diskprediction_local", 2024-04-02T14:50:52.056 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-02T14:50:52.056 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-02T14:50:52.056 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-02T14:50:52.056 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-02T14:50:52.056 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-02T14:50:52.057 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.057 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.057 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.057 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:52.057 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.057 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.057 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:52.057 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:52.057 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:52.057 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:52.057 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:52.057 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:52.057 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:52.057 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:52.057 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.057 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.058 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.058 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.058 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.058 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-02T14:50:52.058 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-02T14:50:52.058 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.058 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.058 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.058 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:52.058 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.058 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.058 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.058 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.058 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.058 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.059 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.059 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.059 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-02T14:50:52.059 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-02T14:50:52.059 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.059 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.059 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.059 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-02T14:50:52.059 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.059 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.059 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:52.059 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:52.059 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:52.059 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:52.060 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:52.060 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:52.060 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:52.060 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:52.060 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.060 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.060 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.060 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.060 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.060 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-02T14:50:52.060 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-02T14:50:52.060 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.060 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.060 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.061 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:52.061 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.061 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.061 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.061 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.061 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.061 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.061 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.061 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.061 INFO:teuthology.orchestra.run.smithi090.stdout: "predict_interval": { 2024-04-02T14:50:52.061 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "predict_interval", 2024-04-02T14:50:52.061 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.061 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.061 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.062 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "86400", 2024-04-02T14:50:52.062 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.062 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.062 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.062 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.062 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.062 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.062 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.062 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.062 INFO:teuthology.orchestra.run.smithi090.stdout: "predictor_model": { 2024-04-02T14:50:52.062 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "predictor_model", 2024-04-02T14:50:52.062 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.062 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.063 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.063 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "prophetstor", 2024-04-02T14:50:52.063 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.063 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.063 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.063 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.063 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.063 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.063 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.063 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.063 INFO:teuthology.orchestra.run.smithi090.stdout: "sleep_interval": { 2024-04-02T14:50:52.063 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "sleep_interval", 2024-04-02T14:50:52.063 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.063 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.063 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.063 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "600", 2024-04-02T14:50:52.064 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.064 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.064 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.064 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.064 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.064 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.064 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.064 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:52.064 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:52.064 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.064 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-02T14:50:52.064 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "influx", 2024-04-02T14:50:52.064 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": false, 2024-04-02T14:50:52.064 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "influxdb python module not found", 2024-04-02T14:50:52.064 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-02T14:50:52.065 INFO:teuthology.orchestra.run.smithi090.stdout: "batch_size": { 2024-04-02T14:50:52.065 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "batch_size", 2024-04-02T14:50:52.065 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-02T14:50:52.065 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.065 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.065 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "5000", 2024-04-02T14:50:52.065 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.065 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.065 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.065 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-04-02T14:50:52.065 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.065 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.065 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.066 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.066 INFO:teuthology.orchestra.run.smithi090.stdout: "database": { 2024-04-02T14:50:52.066 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "database", 2024-04-02T14:50:52.066 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.066 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.066 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.066 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "ceph", 2024-04-02T14:50:52.066 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.066 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.066 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.066 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-04-02T14:50:52.066 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.066 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.066 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.067 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.067 INFO:teuthology.orchestra.run.smithi090.stdout: "hostname": { 2024-04-02T14:50:52.067 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "hostname", 2024-04-02T14:50:52.067 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.067 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.067 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.067 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:52.067 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.067 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.067 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.067 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "InfluxDB server hostname", 2024-04-02T14:50:52.067 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.068 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.068 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.068 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.068 INFO:teuthology.orchestra.run.smithi090.stdout: "interval": { 2024-04-02T14:50:52.068 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "interval", 2024-04-02T14:50:52.068 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "secs", 2024-04-02T14:50:52.068 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.068 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.068 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "30", 2024-04-02T14:50:52.068 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "5", 2024-04-02T14:50:52.068 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.068 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.068 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-04-02T14:50:52.068 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.069 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.069 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.069 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.069 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-02T14:50:52.069 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-02T14:50:52.069 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.069 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.069 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.069 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:52.069 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.069 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.069 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:52.069 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:52.070 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:52.070 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:52.070 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:52.070 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:52.070 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:52.070 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:52.070 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.070 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.070 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.070 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.070 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.070 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-02T14:50:52.070 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-02T14:50:52.070 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.071 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.071 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.071 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:52.071 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.071 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.071 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.071 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.071 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.071 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.071 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.071 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.071 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-02T14:50:52.071 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-02T14:50:52.071 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.072 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.072 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.072 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-02T14:50:52.072 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.072 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.072 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:52.072 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:52.072 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:52.072 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:52.072 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:52.072 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:52.072 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:52.072 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:52.072 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.073 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.073 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.073 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.073 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.073 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-02T14:50:52.073 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-02T14:50:52.073 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.073 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.073 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.073 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:52.073 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.073 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.073 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.073 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.073 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.074 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.074 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.074 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.074 INFO:teuthology.orchestra.run.smithi090.stdout: "password": { 2024-04-02T14:50:52.074 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "password", 2024-04-02T14:50:52.074 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.074 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.074 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.074 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:52.074 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.074 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.074 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.074 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "password of InfluxDB server user", 2024-04-02T14:50:52.074 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.075 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.075 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.075 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.075 INFO:teuthology.orchestra.run.smithi090.stdout: "port": { 2024-04-02T14:50:52.075 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "port", 2024-04-02T14:50:52.075 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-02T14:50:52.075 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.075 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.075 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "8086", 2024-04-02T14:50:52.075 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.075 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.075 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.075 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "InfluxDB server port", 2024-04-02T14:50:52.075 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.075 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.076 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.076 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.076 INFO:teuthology.orchestra.run.smithi090.stdout: "ssl": { 2024-04-02T14:50:52.076 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "ssl", 2024-04-02T14:50:52.076 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.076 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.076 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.076 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "false", 2024-04-02T14:50:52.076 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.076 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.076 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.076 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-04-02T14:50:52.076 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.077 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.077 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.077 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.077 INFO:teuthology.orchestra.run.smithi090.stdout: "threads": { 2024-04-02T14:50:52.077 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "threads", 2024-04-02T14:50:52.077 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-02T14:50:52.077 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.077 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.077 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "5", 2024-04-02T14:50:52.077 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "1", 2024-04-02T14:50:52.077 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "32", 2024-04-02T14:50:52.077 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.077 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-04-02T14:50:52.077 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.077 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.078 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.078 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.078 INFO:teuthology.orchestra.run.smithi090.stdout: "username": { 2024-04-02T14:50:52.078 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "username", 2024-04-02T14:50:52.078 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.078 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.078 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.078 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:52.078 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.078 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.078 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.078 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "username of InfluxDB server user", 2024-04-02T14:50:52.078 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.078 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.079 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.079 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.079 INFO:teuthology.orchestra.run.smithi090.stdout: "verify_ssl": { 2024-04-02T14:50:52.079 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "verify_ssl", 2024-04-02T14:50:52.079 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.079 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.079 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.079 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "true", 2024-04-02T14:50:52.079 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.079 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.079 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.079 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-04-02T14:50:52.079 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.079 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.079 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.079 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:52.080 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:52.080 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.080 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-02T14:50:52.080 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "insights", 2024-04-02T14:50:52.080 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-02T14:50:52.080 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-02T14:50:52.080 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-02T14:50:52.080 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-02T14:50:52.080 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-02T14:50:52.080 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.080 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.080 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.080 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:52.080 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.081 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.081 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:52.081 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:52.081 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:52.081 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:52.081 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:52.081 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:52.081 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:52.081 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:52.081 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.081 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.081 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.081 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.081 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.081 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-02T14:50:52.081 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-02T14:50:52.082 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.082 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.082 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.082 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:52.082 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.082 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.082 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.082 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.082 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.082 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.082 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.082 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.082 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-02T14:50:52.083 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-02T14:50:52.083 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.083 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.083 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.083 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-02T14:50:52.083 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.083 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.083 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:52.083 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:52.083 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:52.083 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:52.083 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:52.083 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:52.083 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:52.083 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:52.084 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.084 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.084 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.084 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.084 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.084 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-02T14:50:52.084 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-02T14:50:52.084 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.084 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.084 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.084 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:52.084 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.084 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.084 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.084 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.085 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.085 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.085 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.085 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:52.085 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:52.085 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.085 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-02T14:50:52.085 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "iostat", 2024-04-02T14:50:52.085 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-02T14:50:52.085 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-02T14:50:52.085 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-02T14:50:52.085 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-02T14:50:52.085 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-02T14:50:52.085 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.086 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.086 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.086 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:52.086 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.086 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.086 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:52.086 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:52.086 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:52.086 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:52.086 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:52.086 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:52.086 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:52.086 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:52.086 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.086 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.087 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.087 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.087 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.087 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-02T14:50:52.087 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-02T14:50:52.087 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.087 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.087 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.087 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:52.087 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.087 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.087 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.087 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.087 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.088 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.088 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.088 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.088 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-02T14:50:52.088 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-02T14:50:52.088 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.088 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.088 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.088 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-02T14:50:52.088 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.088 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.088 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:52.088 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:52.088 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:52.088 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:52.089 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:52.089 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:52.089 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:52.089 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:52.089 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.089 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.089 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.089 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.089 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.089 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-02T14:50:52.089 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-02T14:50:52.089 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.089 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.089 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.089 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:52.090 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.090 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.090 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.090 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.090 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.090 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.090 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.090 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:52.090 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:52.090 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.090 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-02T14:50:52.090 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "k8sevents", 2024-04-02T14:50:52.090 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-02T14:50:52.091 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-02T14:50:52.091 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-02T14:50:52.091 INFO:teuthology.orchestra.run.smithi090.stdout: "ceph_event_retention_days": { 2024-04-02T14:50:52.091 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "ceph_event_retention_days", 2024-04-02T14:50:52.091 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-02T14:50:52.091 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.091 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.091 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "7", 2024-04-02T14:50:52.091 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.091 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.091 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.091 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Days to hold ceph event information within local cache", 2024-04-02T14:50:52.091 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.091 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.091 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.092 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.092 INFO:teuthology.orchestra.run.smithi090.stdout: "config_check_secs": { 2024-04-02T14:50:52.092 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "config_check_secs", 2024-04-02T14:50:52.092 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-02T14:50:52.092 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.092 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.092 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "10", 2024-04-02T14:50:52.092 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "10", 2024-04-02T14:50:52.092 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.092 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.092 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-04-02T14:50:52.092 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.092 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.093 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.093 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.093 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-02T14:50:52.093 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-02T14:50:52.093 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.093 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.093 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.093 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:52.093 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.093 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.093 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:52.093 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:52.093 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:52.093 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:52.094 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:52.094 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:52.094 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:52.094 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:52.094 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.094 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.094 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.094 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.094 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.094 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-02T14:50:52.094 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-02T14:50:52.094 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.094 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.094 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.095 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:52.095 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.095 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.095 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.095 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.095 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.095 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.095 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.095 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.095 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-02T14:50:52.095 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-02T14:50:52.095 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.095 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.095 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.096 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-02T14:50:52.096 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.096 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.096 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:52.096 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:52.096 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:52.096 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:52.096 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:52.096 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:52.096 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:52.096 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:52.096 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.096 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.096 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.097 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.097 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.097 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-02T14:50:52.097 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-02T14:50:52.097 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.097 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.097 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.097 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:52.097 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.097 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.097 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.097 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.097 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.097 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.097 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.098 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:52.098 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:52.098 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.098 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-02T14:50:52.098 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "localpool", 2024-04-02T14:50:52.098 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-02T14:50:52.098 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-02T14:50:52.098 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-02T14:50:52.098 INFO:teuthology.orchestra.run.smithi090.stdout: "failure_domain": { 2024-04-02T14:50:52.098 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "failure_domain", 2024-04-02T14:50:52.098 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.098 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.098 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.098 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "host", 2024-04-02T14:50:52.099 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.099 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.099 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.099 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "failure domain for any created local pool", 2024-04-02T14:50:52.099 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-04-02T14:50:52.099 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.099 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.099 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.099 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-02T14:50:52.099 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-02T14:50:52.099 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.099 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.099 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.099 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:52.100 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.100 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.100 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:52.100 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:52.100 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:52.100 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:52.100 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:52.100 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:52.100 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:52.100 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:52.100 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.100 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.100 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.100 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.100 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.101 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-02T14:50:52.101 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-02T14:50:52.101 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.101 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.101 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.101 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:52.101 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.101 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.101 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.101 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.102 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.102 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.102 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.102 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.102 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-02T14:50:52.102 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-02T14:50:52.102 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.102 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.102 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.102 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-02T14:50:52.102 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.102 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.102 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:52.102 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:52.103 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:52.103 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:52.103 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:52.103 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:52.103 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:52.103 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:52.103 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.103 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.103 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.103 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.103 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.103 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-02T14:50:52.103 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-02T14:50:52.104 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.104 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.104 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.104 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:52.104 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.104 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.104 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.104 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.104 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.104 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.104 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.104 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.104 INFO:teuthology.orchestra.run.smithi090.stdout: "min_size": { 2024-04-02T14:50:52.104 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "min_size", 2024-04-02T14:50:52.105 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-02T14:50:52.105 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.105 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.105 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:52.105 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.105 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.105 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.105 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "default min_size for any created local pool", 2024-04-02T14:50:52.105 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-04-02T14:50:52.105 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.105 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.105 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.105 INFO:teuthology.orchestra.run.smithi090.stdout: "num_rep": { 2024-04-02T14:50:52.105 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "num_rep", 2024-04-02T14:50:52.105 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-02T14:50:52.106 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.106 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.106 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "3", 2024-04-02T14:50:52.106 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.106 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.106 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.106 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "default replica count for any created local pool", 2024-04-02T14:50:52.106 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.106 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.106 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.106 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.106 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num": { 2024-04-02T14:50:52.106 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "pg_num", 2024-04-02T14:50:52.106 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-02T14:50:52.107 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.107 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.107 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "128", 2024-04-02T14:50:52.107 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.107 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.107 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.107 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "default pg_num for any created local pool", 2024-04-02T14:50:52.107 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.107 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.107 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.107 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.107 INFO:teuthology.orchestra.run.smithi090.stdout: "prefix": { 2024-04-02T14:50:52.107 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "prefix", 2024-04-02T14:50:52.107 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.108 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.108 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.108 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:52.108 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.108 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.108 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.108 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "name prefix for any created local pool", 2024-04-02T14:50:52.108 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.108 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.108 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.108 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.108 INFO:teuthology.orchestra.run.smithi090.stdout: "subtree": { 2024-04-02T14:50:52.108 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "subtree", 2024-04-02T14:50:52.108 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.108 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.108 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.109 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "rack", 2024-04-02T14:50:52.109 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.109 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.109 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.109 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "CRUSH level for which to create a local pool", 2024-04-02T14:50:52.109 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-04-02T14:50:52.109 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.109 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.109 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:52.109 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:52.109 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.109 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-02T14:50:52.109 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "mds_autoscaler", 2024-04-02T14:50:52.109 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-02T14:50:52.109 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-02T14:50:52.110 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-02T14:50:52.110 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-02T14:50:52.110 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-02T14:50:52.110 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.110 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.110 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.110 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:52.110 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.110 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.110 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:52.110 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:52.110 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:52.110 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:52.110 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:52.111 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:52.111 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:52.111 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:52.111 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.111 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.111 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.111 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.111 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.111 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-02T14:50:52.111 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-02T14:50:52.111 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.111 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.111 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.111 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:52.111 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.111 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.112 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.112 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.112 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.112 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.112 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.112 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.112 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-02T14:50:52.112 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-02T14:50:52.112 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.112 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.112 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.112 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-02T14:50:52.112 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.112 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.113 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:52.113 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:52.113 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:52.113 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:52.113 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:52.113 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:52.113 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:52.113 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:52.113 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.113 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.113 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.113 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.113 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.113 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-02T14:50:52.113 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-02T14:50:52.114 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.114 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.114 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.114 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:52.114 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.114 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.114 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.114 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.114 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.114 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.114 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.114 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:52.114 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:52.114 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.115 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-02T14:50:52.115 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "mirroring", 2024-04-02T14:50:52.115 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-02T14:50:52.115 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-02T14:50:52.115 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-02T14:50:52.115 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-02T14:50:52.115 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-02T14:50:52.115 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.115 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.115 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.115 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:52.115 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.115 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.115 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:52.116 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:52.116 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:52.116 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:52.116 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:52.116 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:52.116 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:52.116 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:52.116 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.116 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.116 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.116 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.116 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.116 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-02T14:50:52.116 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-02T14:50:52.116 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.116 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.117 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.117 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:52.117 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.117 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.117 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.117 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.117 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.117 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.117 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.117 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.117 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-02T14:50:52.117 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-02T14:50:52.117 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.117 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.117 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.118 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-02T14:50:52.118 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.118 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.118 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:52.118 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:52.118 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:52.118 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:52.118 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:52.118 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:52.118 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:52.118 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:52.118 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.118 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.118 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.118 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.119 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.119 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-02T14:50:52.119 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-02T14:50:52.119 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.119 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.119 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.119 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:52.119 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.119 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.119 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.119 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.119 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.119 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.120 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.120 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:52.120 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:52.120 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.120 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-02T14:50:52.120 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "nfs", 2024-04-02T14:50:52.120 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-02T14:50:52.120 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-02T14:50:52.120 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-02T14:50:52.120 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-02T14:50:52.120 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-02T14:50:52.120 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.120 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.120 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.121 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:52.121 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.121 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.121 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:52.121 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:52.121 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:52.121 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:52.121 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:52.121 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:52.121 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:52.121 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:52.121 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.121 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.121 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.121 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.121 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.121 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-02T14:50:52.122 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-02T14:50:52.122 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.122 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.122 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.122 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:52.122 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.122 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.122 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.122 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.122 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.122 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.122 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.122 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.122 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-02T14:50:52.123 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-02T14:50:52.123 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.123 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.123 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.123 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-02T14:50:52.123 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.123 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.123 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:52.123 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:52.123 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:52.123 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:52.123 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:52.123 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:52.123 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:52.124 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:52.124 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.124 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.124 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.124 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.124 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.124 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-02T14:50:52.124 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-02T14:50:52.124 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.124 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.124 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.124 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:52.124 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.124 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.124 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.124 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.124 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.125 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.125 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.125 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:52.125 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:52.125 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.125 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-02T14:50:52.125 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "orchestrator", 2024-04-02T14:50:52.125 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-02T14:50:52.125 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-02T14:50:52.125 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-02T14:50:52.125 INFO:teuthology.orchestra.run.smithi090.stdout: "fail_fs": { 2024-04-02T14:50:52.125 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "fail_fs", 2024-04-02T14:50:52.125 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.126 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.126 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.126 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:52.126 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.126 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.126 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.126 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-04-02T14:50:52.126 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.126 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.126 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.126 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.126 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-02T14:50:52.126 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-02T14:50:52.126 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.126 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.127 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.127 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:52.127 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.127 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.127 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:52.127 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:52.127 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:52.127 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:52.127 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:52.127 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:52.127 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:52.127 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:52.127 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.127 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.128 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.128 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.128 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.128 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-02T14:50:52.128 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-02T14:50:52.128 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.128 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.128 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.128 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:52.128 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.128 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.128 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.128 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.129 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.129 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.129 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.129 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.129 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-02T14:50:52.129 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-02T14:50:52.129 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.129 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.129 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.129 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-02T14:50:52.129 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.129 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.129 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:52.129 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:52.130 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:52.130 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:52.130 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:52.130 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:52.130 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:52.130 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:52.130 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.130 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.130 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.130 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.130 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.130 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-02T14:50:52.130 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-02T14:50:52.130 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.131 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.131 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.131 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:52.131 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.131 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.131 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.131 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.131 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.131 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.131 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.131 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.131 INFO:teuthology.orchestra.run.smithi090.stdout: "orchestrator": { 2024-04-02T14:50:52.131 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "orchestrator", 2024-04-02T14:50:52.132 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.132 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.132 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.132 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:52.132 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.132 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.132 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:52.132 INFO:teuthology.orchestra.run.smithi090.stdout: "cephadm", 2024-04-02T14:50:52.132 INFO:teuthology.orchestra.run.smithi090.stdout: "rook", 2024-04-02T14:50:52.132 INFO:teuthology.orchestra.run.smithi090.stdout: "test_orchestrator" 2024-04-02T14:50:52.132 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:52.132 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Orchestrator backend", 2024-04-02T14:50:52.132 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.132 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.133 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.133 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:52.133 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:52.133 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.133 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-02T14:50:52.133 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "osd_perf_query", 2024-04-02T14:50:52.133 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-02T14:50:52.133 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-02T14:50:52.133 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-02T14:50:52.133 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-02T14:50:52.133 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-02T14:50:52.133 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.133 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.133 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.134 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:52.134 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.134 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.134 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:52.134 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:52.134 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:52.134 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:52.134 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:52.134 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:52.134 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:52.134 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:52.134 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.134 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.134 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.134 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.134 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.135 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-02T14:50:52.135 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-02T14:50:52.135 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.135 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.135 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.135 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:52.135 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.135 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.135 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.135 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.135 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.135 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.135 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.136 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.136 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-02T14:50:52.136 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-02T14:50:52.136 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.136 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.136 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.136 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-02T14:50:52.136 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.136 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.136 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:52.136 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:52.136 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:52.136 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:52.136 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:52.137 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:52.137 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:52.137 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:52.137 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.137 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.137 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.137 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.137 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.137 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-02T14:50:52.137 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-02T14:50:52.137 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.137 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.137 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.137 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:52.137 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.137 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.138 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.138 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.138 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.138 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.138 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.138 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:52.138 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:52.138 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.138 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-02T14:50:52.138 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "osd_support", 2024-04-02T14:50:52.138 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-02T14:50:52.138 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-02T14:50:52.138 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-02T14:50:52.139 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-02T14:50:52.139 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-02T14:50:52.139 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.139 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.139 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.139 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:52.139 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.139 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.139 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:52.139 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:52.139 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:52.139 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:52.139 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:52.139 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:52.139 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:52.139 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:52.139 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.140 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.140 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.140 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.140 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.140 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-02T14:50:52.140 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-02T14:50:52.140 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.140 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.140 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.140 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:52.140 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.140 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.140 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.141 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.141 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.141 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.141 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.141 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.141 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-02T14:50:52.141 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-02T14:50:52.141 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.141 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.141 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.141 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-02T14:50:52.141 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.141 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.141 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:52.141 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:52.142 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:52.142 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:52.142 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:52.142 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:52.142 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:52.142 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:52.142 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.142 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.142 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.142 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.142 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.142 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-02T14:50:52.142 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-02T14:50:52.143 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.143 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.143 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.143 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:52.143 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.143 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.143 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.143 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.143 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.143 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.143 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.143 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:52.143 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:52.143 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.144 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-02T14:50:52.144 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "pg_autoscaler", 2024-04-02T14:50:52.144 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-02T14:50:52.144 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-02T14:50:52.144 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-02T14:50:52.144 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-02T14:50:52.144 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-02T14:50:52.144 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.144 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.144 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.144 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:52.144 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.144 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.144 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:52.144 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:52.145 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:52.145 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:52.145 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:52.145 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:52.145 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:52.145 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:52.145 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.145 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.145 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.145 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.145 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.145 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-02T14:50:52.145 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-02T14:50:52.145 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.145 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.146 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.146 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:52.146 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.146 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.146 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.146 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.146 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.146 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.146 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.146 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.146 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-02T14:50:52.146 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-02T14:50:52.146 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.147 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.147 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.147 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-02T14:50:52.147 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.147 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.147 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:52.147 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:52.147 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:52.147 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:52.147 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:52.147 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:52.147 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:52.147 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:52.147 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.147 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.148 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.148 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.148 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.148 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-02T14:50:52.148 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-02T14:50:52.148 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.148 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.148 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.148 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:52.148 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.148 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.148 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.148 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.148 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.148 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.149 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.149 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.149 INFO:teuthology.orchestra.run.smithi090.stdout: "sleep_interval": { 2024-04-02T14:50:52.149 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "sleep_interval", 2024-04-02T14:50:52.149 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "secs", 2024-04-02T14:50:52.149 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.149 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.149 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "60", 2024-04-02T14:50:52.149 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.149 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.149 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.149 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.149 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.149 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.149 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.150 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.150 INFO:teuthology.orchestra.run.smithi090.stdout: "threshold": { 2024-04-02T14:50:52.150 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "threshold", 2024-04-02T14:50:52.150 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "float", 2024-04-02T14:50:52.150 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.150 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.150 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "3.0", 2024-04-02T14:50:52.150 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "1.0", 2024-04-02T14:50:52.150 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.150 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.150 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "scaling threshold", 2024-04-02T14:50:52.150 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-04-02T14:50:52.150 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.150 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.151 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:52.151 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:52.151 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.151 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-02T14:50:52.151 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "progress", 2024-04-02T14:50:52.151 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-02T14:50:52.151 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-02T14:50:52.151 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-02T14:50:52.151 INFO:teuthology.orchestra.run.smithi090.stdout: "allow_pg_recovery_event": { 2024-04-02T14:50:52.151 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "allow_pg_recovery_event", 2024-04-02T14:50:52.151 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.151 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.151 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.151 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:52.151 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.151 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.152 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.152 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "allow the module to show pg recovery progress", 2024-04-02T14:50:52.152 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.152 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.152 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.152 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.152 INFO:teuthology.orchestra.run.smithi090.stdout: "enabled": { 2024-04-02T14:50:52.152 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "enabled", 2024-04-02T14:50:52.152 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.152 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.152 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.152 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-02T14:50:52.152 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.152 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.152 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.153 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.153 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.153 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.153 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.153 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.153 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-02T14:50:52.153 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-02T14:50:52.153 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.153 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.153 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.153 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:52.153 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.153 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.153 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:52.153 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:52.153 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:52.154 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:52.154 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:52.154 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:52.154 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:52.154 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:52.154 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.154 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.154 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.154 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.154 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.154 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-02T14:50:52.154 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-02T14:50:52.154 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.154 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.155 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.155 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:52.155 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.155 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.155 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.155 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.155 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.155 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.155 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.155 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.155 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-02T14:50:52.155 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-02T14:50:52.155 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.155 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.156 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.156 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-02T14:50:52.156 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.156 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.156 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:52.156 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:52.156 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:52.156 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:52.156 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:52.156 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:52.156 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:52.156 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:52.156 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.156 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.156 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.156 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.157 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.157 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-02T14:50:52.157 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-02T14:50:52.157 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.157 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.157 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.157 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:52.157 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.157 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.157 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.157 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.157 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.157 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.157 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.157 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.158 INFO:teuthology.orchestra.run.smithi090.stdout: "max_completed_events": { 2024-04-02T14:50:52.158 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "max_completed_events", 2024-04-02T14:50:52.158 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-02T14:50:52.158 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.158 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.158 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "50", 2024-04-02T14:50:52.158 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.158 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.158 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.158 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "number of past completed events to remember", 2024-04-02T14:50:52.158 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.158 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.158 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.158 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.159 INFO:teuthology.orchestra.run.smithi090.stdout: "sleep_interval": { 2024-04-02T14:50:52.159 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "sleep_interval", 2024-04-02T14:50:52.159 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "secs", 2024-04-02T14:50:52.159 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.159 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.159 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "5", 2024-04-02T14:50:52.159 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.159 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.159 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.159 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "how long the module is going to sleep", 2024-04-02T14:50:52.159 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.159 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.159 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.159 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:52.159 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:52.160 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.160 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-02T14:50:52.160 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "prometheus", 2024-04-02T14:50:52.160 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-02T14:50:52.160 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-02T14:50:52.160 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-02T14:50:52.160 INFO:teuthology.orchestra.run.smithi090.stdout: "cache": { 2024-04-02T14:50:52.160 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "cache", 2024-04-02T14:50:52.160 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.160 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.160 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.160 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-02T14:50:52.160 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.160 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.160 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.161 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.161 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.161 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.161 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.161 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.161 INFO:teuthology.orchestra.run.smithi090.stdout: "exclude_perf_counters": { 2024-04-02T14:50:52.161 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "exclude_perf_counters", 2024-04-02T14:50:52.161 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.161 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.161 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.161 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-02T14:50:52.161 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.161 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.161 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.161 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-04-02T14:50:52.162 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-04-02T14:50:52.162 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.162 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.162 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.162 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-02T14:50:52.162 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-02T14:50:52.162 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.162 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.162 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.162 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:52.162 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.162 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.162 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:52.162 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:52.162 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:52.163 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:52.163 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:52.163 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:52.163 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:52.163 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:52.163 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.163 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.163 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.163 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.163 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.163 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-02T14:50:52.163 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-02T14:50:52.163 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.163 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.163 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.164 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:52.164 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.164 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.164 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.164 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.164 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.164 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.164 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.164 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.164 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-02T14:50:52.164 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-02T14:50:52.164 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.164 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.164 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.164 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-02T14:50:52.165 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.165 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.165 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:52.165 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:52.165 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:52.165 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:52.165 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:52.165 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:52.165 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:52.165 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:52.165 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.165 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.165 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.165 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.165 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.165 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-02T14:50:52.166 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-02T14:50:52.166 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.166 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.166 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.166 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:52.166 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.166 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.166 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.166 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.166 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.166 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.166 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.166 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.166 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_stats_pools": { 2024-04-02T14:50:52.167 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "rbd_stats_pools", 2024-04-02T14:50:52.167 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.167 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.167 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.167 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:52.167 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.167 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.167 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.167 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.167 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.167 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.167 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.167 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.167 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_stats_pools_refresh_interval": { 2024-04-02T14:50:52.168 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-04-02T14:50:52.168 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-02T14:50:52.168 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.168 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.168 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "300", 2024-04-02T14:50:52.168 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.168 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.168 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.168 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.168 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.168 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.168 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.168 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.168 INFO:teuthology.orchestra.run.smithi090.stdout: "scrape_interval": { 2024-04-02T14:50:52.168 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "scrape_interval", 2024-04-02T14:50:52.169 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "float", 2024-04-02T14:50:52.169 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.169 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.169 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "15.0", 2024-04-02T14:50:52.169 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.169 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.169 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.169 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.169 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.169 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.169 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.169 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.169 INFO:teuthology.orchestra.run.smithi090.stdout: "server_addr": { 2024-04-02T14:50:52.169 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "server_addr", 2024-04-02T14:50:52.169 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.170 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.170 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.170 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "::", 2024-04-02T14:50:52.170 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.170 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.170 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.170 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-04-02T14:50:52.170 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.170 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.170 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.170 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.170 INFO:teuthology.orchestra.run.smithi090.stdout: "server_port": { 2024-04-02T14:50:52.170 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "server_port", 2024-04-02T14:50:52.170 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-02T14:50:52.170 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.171 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.171 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "9283", 2024-04-02T14:50:52.171 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.171 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.171 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.171 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-04-02T14:50:52.171 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.171 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.171 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.171 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.171 INFO:teuthology.orchestra.run.smithi090.stdout: "stale_cache_strategy": { 2024-04-02T14:50:52.171 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "stale_cache_strategy", 2024-04-02T14:50:52.171 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.171 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.171 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.172 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "log", 2024-04-02T14:50:52.172 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.172 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.172 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.172 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.172 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.172 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.172 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.172 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.172 INFO:teuthology.orchestra.run.smithi090.stdout: "standby_behaviour": { 2024-04-02T14:50:52.172 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "standby_behaviour", 2024-04-02T14:50:52.172 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.172 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.172 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.173 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "default", 2024-04-02T14:50:52.173 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.173 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.173 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:52.173 INFO:teuthology.orchestra.run.smithi090.stdout: "default", 2024-04-02T14:50:52.173 INFO:teuthology.orchestra.run.smithi090.stdout: "error" 2024-04-02T14:50:52.173 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:52.173 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.173 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.173 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.173 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.173 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.173 INFO:teuthology.orchestra.run.smithi090.stdout: "standby_error_status_code": { 2024-04-02T14:50:52.173 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "standby_error_status_code", 2024-04-02T14:50:52.173 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-02T14:50:52.173 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.174 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.174 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "500", 2024-04-02T14:50:52.174 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "400", 2024-04-02T14:50:52.174 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "599", 2024-04-02T14:50:52.174 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.174 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.174 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.174 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.174 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.174 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:52.174 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:52.174 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.174 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-02T14:50:52.174 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "rbd_support", 2024-04-02T14:50:52.174 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-02T14:50:52.175 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-02T14:50:52.175 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-02T14:50:52.175 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-02T14:50:52.175 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-02T14:50:52.175 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.175 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.175 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.175 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:52.175 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.175 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.175 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:52.175 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:52.175 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:52.175 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:52.175 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:52.176 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:52.176 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:52.176 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:52.176 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.176 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.176 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.176 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.176 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.176 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-02T14:50:52.176 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-02T14:50:52.176 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.176 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.176 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.176 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:52.176 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.177 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.177 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.177 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.177 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.177 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.177 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.177 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.177 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-02T14:50:52.177 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-02T14:50:52.177 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.177 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.177 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.177 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-02T14:50:52.177 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.177 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.178 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:52.178 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:52.178 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:52.178 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:52.178 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:52.178 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:52.178 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:52.178 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:52.178 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.178 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.178 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.178 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.178 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.178 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-02T14:50:52.178 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-02T14:50:52.179 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.179 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.179 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.179 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:52.179 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.179 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.179 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.179 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.179 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.179 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.179 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.179 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.179 INFO:teuthology.orchestra.run.smithi090.stdout: "max_concurrent_snap_create": { 2024-04-02T14:50:52.180 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "max_concurrent_snap_create", 2024-04-02T14:50:52.180 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-02T14:50:52.180 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.180 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.180 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "10", 2024-04-02T14:50:52.180 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.180 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.180 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.180 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.180 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.180 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.180 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.180 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.180 INFO:teuthology.orchestra.run.smithi090.stdout: "mirror_snapshot_schedule": { 2024-04-02T14:50:52.180 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "mirror_snapshot_schedule", 2024-04-02T14:50:52.181 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.181 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.181 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.181 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:52.181 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.181 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.181 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.181 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.181 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.181 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.181 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.181 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.181 INFO:teuthology.orchestra.run.smithi090.stdout: "trash_purge_schedule": { 2024-04-02T14:50:52.181 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "trash_purge_schedule", 2024-04-02T14:50:52.181 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.181 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.182 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.182 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:52.182 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.182 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.182 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.182 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.182 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.182 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.182 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.182 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:52.182 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:52.182 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.182 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-02T14:50:52.183 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "restful", 2024-04-02T14:50:52.183 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-02T14:50:52.183 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-02T14:50:52.183 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-02T14:50:52.183 INFO:teuthology.orchestra.run.smithi090.stdout: "enable_auth": { 2024-04-02T14:50:52.183 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "enable_auth", 2024-04-02T14:50:52.183 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.183 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.183 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.183 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-02T14:50:52.183 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.183 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.183 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.183 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.183 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.184 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.184 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.184 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.184 INFO:teuthology.orchestra.run.smithi090.stdout: "key_file": { 2024-04-02T14:50:52.184 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "key_file", 2024-04-02T14:50:52.184 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.184 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.184 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.184 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:52.184 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.184 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.184 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.184 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.184 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.184 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.184 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.185 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.185 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-02T14:50:52.185 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-02T14:50:52.185 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.185 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.185 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.185 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:52.185 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.185 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.185 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:52.185 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:52.185 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:52.185 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:52.185 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:52.186 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:52.186 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:52.186 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:52.186 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.186 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.186 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.186 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.186 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.186 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-02T14:50:52.186 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-02T14:50:52.186 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.186 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.186 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.186 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:52.187 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.187 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.187 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.187 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.187 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.187 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.187 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.187 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.187 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-02T14:50:52.187 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-02T14:50:52.187 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.187 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.187 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.188 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-02T14:50:52.188 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.188 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.188 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:52.188 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:52.188 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:52.188 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:52.188 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:52.188 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:52.188 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:52.188 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:52.188 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.188 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.188 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.188 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.189 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.189 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-02T14:50:52.189 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-02T14:50:52.189 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.189 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.189 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.189 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:52.189 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.189 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.189 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.189 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.189 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.189 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.189 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.190 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.190 INFO:teuthology.orchestra.run.smithi090.stdout: "server_addr": { 2024-04-02T14:50:52.190 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "server_addr", 2024-04-02T14:50:52.190 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.190 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.190 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.190 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:52.190 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.190 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.190 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.190 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.190 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.190 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.190 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.191 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.191 INFO:teuthology.orchestra.run.smithi090.stdout: "server_port": { 2024-04-02T14:50:52.191 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "server_port", 2024-04-02T14:50:52.191 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.191 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.191 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.191 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:52.191 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.191 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.191 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.191 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.191 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.191 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.191 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.192 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:52.192 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:52.192 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.192 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-02T14:50:52.192 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "rgw", 2024-04-02T14:50:52.192 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-02T14:50:52.192 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-02T14:50:52.192 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-02T14:50:52.192 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-02T14:50:52.192 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-02T14:50:52.192 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.192 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.192 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.192 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:52.193 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.193 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.193 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:52.193 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:52.193 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:52.193 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:52.193 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:52.193 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:52.193 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:52.193 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:52.193 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.193 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.193 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.193 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.194 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.194 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-02T14:50:52.194 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-02T14:50:52.194 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.194 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.194 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.194 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:52.194 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.194 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.194 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.194 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.194 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.194 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.194 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.195 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.195 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-02T14:50:52.195 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-02T14:50:52.195 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.195 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.195 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.195 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-02T14:50:52.195 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.195 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.195 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:52.195 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:52.195 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:52.195 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:52.195 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:52.195 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:52.196 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:52.196 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:52.196 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.196 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.196 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.196 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.196 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.196 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-02T14:50:52.196 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-02T14:50:52.196 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.196 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.196 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.196 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:52.196 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.197 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.197 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.197 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.197 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.197 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.197 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.197 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:52.197 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:52.197 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.197 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-02T14:50:52.197 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "rook", 2024-04-02T14:50:52.197 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-02T14:50:52.197 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-02T14:50:52.197 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-02T14:50:52.198 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-02T14:50:52.198 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-02T14:50:52.198 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.198 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.198 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.198 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:52.198 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.198 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.198 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:52.198 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:52.198 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:52.198 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:52.198 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:52.198 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:52.199 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:52.199 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:52.199 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.199 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.199 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.199 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.199 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.199 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-02T14:50:52.199 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-02T14:50:52.199 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.199 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.199 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.199 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:52.199 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.199 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.200 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.200 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.200 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.200 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.200 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.200 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.200 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-02T14:50:52.200 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-02T14:50:52.200 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.200 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.200 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.200 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-02T14:50:52.200 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.200 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.201 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:52.201 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:52.201 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:52.201 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:52.201 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:52.201 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:52.201 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:52.201 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:52.201 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.201 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.201 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.201 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.201 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.201 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-02T14:50:52.202 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-02T14:50:52.202 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.202 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.202 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.202 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:52.202 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.202 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.202 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.202 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.202 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.202 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.202 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.202 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.203 INFO:teuthology.orchestra.run.smithi090.stdout: "storage_class": { 2024-04-02T14:50:52.203 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "storage_class", 2024-04-02T14:50:52.203 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.203 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.203 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.203 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "local", 2024-04-02T14:50:52.203 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.203 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.203 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.203 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-04-02T14:50:52.203 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.203 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.203 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.203 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:52.203 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:52.203 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.204 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-02T14:50:52.204 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "selftest", 2024-04-02T14:50:52.204 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-02T14:50:52.204 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-02T14:50:52.204 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-02T14:50:52.204 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-02T14:50:52.204 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-02T14:50:52.204 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.204 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.204 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.204 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:52.204 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.204 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.204 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:52.204 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:52.204 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:52.205 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:52.205 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:52.205 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:52.205 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:52.205 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:52.205 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.205 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.205 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.205 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.205 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.205 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-02T14:50:52.205 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-02T14:50:52.205 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.205 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.205 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.205 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:52.206 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.206 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.206 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.206 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.206 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.206 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.206 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.206 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.206 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-02T14:50:52.206 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-02T14:50:52.206 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.206 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.206 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.206 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-02T14:50:52.206 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.206 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.207 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:52.207 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:52.207 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:52.207 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:52.207 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:52.207 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:52.207 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:52.207 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:52.207 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.207 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.207 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.207 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.207 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.207 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-02T14:50:52.207 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-02T14:50:52.208 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.208 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.208 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.208 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:52.208 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.208 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.208 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.208 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.208 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.208 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.208 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.208 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.208 INFO:teuthology.orchestra.run.smithi090.stdout: "roption1": { 2024-04-02T14:50:52.208 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "roption1", 2024-04-02T14:50:52.208 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.208 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.209 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.209 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:52.209 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.209 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.209 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.209 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.209 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.209 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.209 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.209 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.209 INFO:teuthology.orchestra.run.smithi090.stdout: "roption2": { 2024-04-02T14:50:52.209 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "roption2", 2024-04-02T14:50:52.209 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.209 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.209 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.210 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "xyz", 2024-04-02T14:50:52.210 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.210 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.210 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.210 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.210 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.210 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.210 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.210 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.210 INFO:teuthology.orchestra.run.smithi090.stdout: "rwoption1": { 2024-04-02T14:50:52.210 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "rwoption1", 2024-04-02T14:50:52.210 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.210 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.210 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.210 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:52.210 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.211 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.211 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.211 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.211 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.211 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.211 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.211 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.211 INFO:teuthology.orchestra.run.smithi090.stdout: "rwoption2": { 2024-04-02T14:50:52.211 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "rwoption2", 2024-04-02T14:50:52.211 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-02T14:50:52.211 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.211 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.211 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:52.211 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.211 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.212 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.212 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.212 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.212 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.212 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.212 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.212 INFO:teuthology.orchestra.run.smithi090.stdout: "rwoption3": { 2024-04-02T14:50:52.212 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "rwoption3", 2024-04-02T14:50:52.212 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "float", 2024-04-02T14:50:52.212 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.212 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.212 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:52.212 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.212 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.212 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.212 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.213 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.213 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.213 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.213 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.213 INFO:teuthology.orchestra.run.smithi090.stdout: "rwoption4": { 2024-04-02T14:50:52.213 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "rwoption4", 2024-04-02T14:50:52.213 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.213 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.213 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.213 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:52.213 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.213 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.213 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.213 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.213 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.213 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.213 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.214 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.214 INFO:teuthology.orchestra.run.smithi090.stdout: "rwoption5": { 2024-04-02T14:50:52.214 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "rwoption5", 2024-04-02T14:50:52.214 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.214 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.214 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.214 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:52.214 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.214 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.214 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.214 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.214 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.214 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.214 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.214 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.214 INFO:teuthology.orchestra.run.smithi090.stdout: "rwoption6": { 2024-04-02T14:50:52.215 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "rwoption6", 2024-04-02T14:50:52.215 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.215 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.215 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.215 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-02T14:50:52.215 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.215 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.215 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.215 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.215 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.215 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.215 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.215 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.215 INFO:teuthology.orchestra.run.smithi090.stdout: "rwoption7": { 2024-04-02T14:50:52.215 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "rwoption7", 2024-04-02T14:50:52.215 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-02T14:50:52.216 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.216 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.216 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:52.216 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "1", 2024-04-02T14:50:52.216 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "42", 2024-04-02T14:50:52.216 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.216 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.216 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.216 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.216 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.216 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.216 INFO:teuthology.orchestra.run.smithi090.stdout: "testkey": { 2024-04-02T14:50:52.216 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "testkey", 2024-04-02T14:50:52.216 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.216 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.216 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.216 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:52.217 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.217 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.217 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.217 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.217 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.217 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.217 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.217 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.217 INFO:teuthology.orchestra.run.smithi090.stdout: "testlkey": { 2024-04-02T14:50:52.217 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "testlkey", 2024-04-02T14:50:52.217 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.217 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.217 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.217 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:52.217 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.217 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.218 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.218 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.218 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.218 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.218 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.218 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.218 INFO:teuthology.orchestra.run.smithi090.stdout: "testnewline": { 2024-04-02T14:50:52.218 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "testnewline", 2024-04-02T14:50:52.218 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.218 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.218 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.218 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:52.218 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.218 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.218 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.218 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.219 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.219 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.219 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.219 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:52.219 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:52.219 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.219 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-02T14:50:52.219 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "snap_schedule", 2024-04-02T14:50:52.219 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-02T14:50:52.219 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-02T14:50:52.219 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-02T14:50:52.219 INFO:teuthology.orchestra.run.smithi090.stdout: "allow_m_granularity": { 2024-04-02T14:50:52.219 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "allow_m_granularity", 2024-04-02T14:50:52.219 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.220 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.220 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.220 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:52.220 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.220 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.220 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.220 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "allow minute scheduled snapshots", 2024-04-02T14:50:52.220 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.220 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.220 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.220 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.220 INFO:teuthology.orchestra.run.smithi090.stdout: "dump_on_update": { 2024-04-02T14:50:52.220 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "dump_on_update", 2024-04-02T14:50:52.220 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.220 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.221 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.221 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:52.221 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.221 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.221 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.221 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "dump database to debug log on update", 2024-04-02T14:50:52.221 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.221 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.221 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.221 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.221 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-02T14:50:52.221 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-02T14:50:52.221 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.221 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.221 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.221 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:52.222 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.222 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.222 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:52.222 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:52.222 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:52.222 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:52.222 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:52.222 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:52.222 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:52.222 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:52.222 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.222 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.222 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.222 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.222 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.222 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-02T14:50:52.223 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-02T14:50:52.223 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.223 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.223 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.223 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:52.223 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.223 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.223 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.223 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.223 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.223 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.223 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.223 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.223 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-02T14:50:52.223 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-02T14:50:52.224 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.224 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.224 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.224 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-02T14:50:52.224 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.224 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.224 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:52.224 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:52.224 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:52.224 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:52.224 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:52.224 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:52.224 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:52.224 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:52.224 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.224 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.225 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.225 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.225 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.225 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-02T14:50:52.225 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-02T14:50:52.225 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.225 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.225 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.225 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:52.225 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.225 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.225 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.225 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.225 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.225 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.225 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.226 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:52.226 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:52.226 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.226 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-02T14:50:52.226 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "stats", 2024-04-02T14:50:52.226 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-02T14:50:52.226 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-02T14:50:52.226 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-02T14:50:52.226 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-02T14:50:52.226 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-02T14:50:52.226 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.226 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.226 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.226 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:52.226 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.226 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.227 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:52.227 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:52.227 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:52.227 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:52.227 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:52.227 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:52.227 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:52.227 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:52.227 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.227 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.227 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.227 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.227 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.227 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-02T14:50:52.228 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-02T14:50:52.228 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.228 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.228 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.228 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:52.228 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.228 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.228 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.228 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.228 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.228 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.228 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.228 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.228 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-02T14:50:52.228 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-02T14:50:52.228 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.228 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.229 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.229 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-02T14:50:52.229 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.229 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.229 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:52.229 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:52.229 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:52.229 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:52.229 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:52.229 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:52.229 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:52.229 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:52.229 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.229 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.229 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.229 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.230 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.230 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-02T14:50:52.230 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-02T14:50:52.230 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.230 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.230 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.230 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:52.230 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.230 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.230 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.230 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.230 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.230 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.230 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.230 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:52.230 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:52.231 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.231 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-02T14:50:52.231 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "status", 2024-04-02T14:50:52.231 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-02T14:50:52.231 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-02T14:50:52.231 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-02T14:50:52.231 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-02T14:50:52.231 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-02T14:50:52.231 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.231 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.231 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.231 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:52.231 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.231 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.231 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:52.231 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:52.231 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:52.232 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:52.232 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:52.232 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:52.232 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:52.232 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:52.232 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.232 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.232 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.232 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.232 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.232 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-02T14:50:52.232 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-02T14:50:52.232 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.232 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.232 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.233 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:52.233 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.233 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.233 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.233 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.233 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.233 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.233 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.233 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.233 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-02T14:50:52.233 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-02T14:50:52.233 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.233 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.233 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.233 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-02T14:50:52.234 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.234 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.234 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:52.234 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:52.234 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:52.234 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:52.234 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:52.234 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:52.234 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:52.234 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:52.234 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.234 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.235 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.235 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.235 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.235 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-02T14:50:52.235 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-02T14:50:52.235 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.235 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.235 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.235 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:52.235 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.235 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.235 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.235 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.235 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.236 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.236 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.236 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:52.236 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:52.236 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.236 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-02T14:50:52.236 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "telegraf", 2024-04-02T14:50:52.236 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-02T14:50:52.236 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-02T14:50:52.236 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-02T14:50:52.236 INFO:teuthology.orchestra.run.smithi090.stdout: "address": { 2024-04-02T14:50:52.236 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "address", 2024-04-02T14:50:52.236 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.236 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.236 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.236 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-04-02T14:50:52.237 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.237 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.237 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.237 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.237 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.237 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.237 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.237 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.237 INFO:teuthology.orchestra.run.smithi090.stdout: "interval": { 2024-04-02T14:50:52.237 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "interval", 2024-04-02T14:50:52.237 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "secs", 2024-04-02T14:50:52.237 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.237 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.237 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "15", 2024-04-02T14:50:52.237 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.238 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.238 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.238 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.238 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.238 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.238 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.238 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.238 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-02T14:50:52.238 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-02T14:50:52.238 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.238 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.238 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.238 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:52.238 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.238 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.239 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:52.239 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:52.239 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:52.239 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:52.239 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:52.239 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:52.239 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:52.239 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:52.239 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.239 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.242 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.242 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.242 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.242 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-02T14:50:52.242 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-02T14:50:52.242 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.242 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.242 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.242 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:52.243 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.243 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.243 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.243 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.243 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.243 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.243 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.243 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.243 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-02T14:50:52.243 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-02T14:50:52.243 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.243 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.243 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.243 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-02T14:50:52.243 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.244 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.244 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:52.244 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:52.244 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:52.244 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:52.244 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:52.244 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:52.244 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:52.244 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:52.244 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.244 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.244 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.244 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.244 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.244 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-02T14:50:52.244 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-02T14:50:52.245 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.245 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.245 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.245 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:52.245 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.245 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.245 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.245 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.245 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.245 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.245 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.245 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:52.245 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:52.245 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.245 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-02T14:50:52.246 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "telemetry", 2024-04-02T14:50:52.246 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-02T14:50:52.246 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-02T14:50:52.246 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-02T14:50:52.246 INFO:teuthology.orchestra.run.smithi090.stdout: "channel_basic": { 2024-04-02T14:50:52.246 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "channel_basic", 2024-04-02T14:50:52.246 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.246 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.246 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.246 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-02T14:50:52.246 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.246 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.246 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.246 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Share basic cluster information (size, version)", 2024-04-02T14:50:52.247 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.247 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.247 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.247 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.247 INFO:teuthology.orchestra.run.smithi090.stdout: "channel_crash": { 2024-04-02T14:50:52.247 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "channel_crash", 2024-04-02T14:50:52.247 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.247 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.247 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.247 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-02T14:50:52.247 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.247 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.247 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.247 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-04-02T14:50:52.247 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.247 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.248 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.248 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.248 INFO:teuthology.orchestra.run.smithi090.stdout: "channel_device": { 2024-04-02T14:50:52.248 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "channel_device", 2024-04-02T14:50:52.248 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.248 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.248 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.248 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-02T14:50:52.248 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.248 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.248 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.248 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-04-02T14:50:52.248 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.248 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.248 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.249 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.249 INFO:teuthology.orchestra.run.smithi090.stdout: "channel_ident": { 2024-04-02T14:50:52.249 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "channel_ident", 2024-04-02T14:50:52.249 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.249 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.249 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.249 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:52.249 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.249 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.249 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.249 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-04-02T14:50:52.249 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.249 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.249 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.249 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.249 INFO:teuthology.orchestra.run.smithi090.stdout: "channel_perf": { 2024-04-02T14:50:52.249 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "channel_perf", 2024-04-02T14:50:52.250 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.250 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.250 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.250 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:52.250 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.250 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.250 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.250 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Share various performance metrics of a cluster", 2024-04-02T14:50:52.250 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.250 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.250 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.250 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.250 INFO:teuthology.orchestra.run.smithi090.stdout: "contact": { 2024-04-02T14:50:52.250 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "contact", 2024-04-02T14:50:52.250 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.251 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.251 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.251 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:52.251 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.251 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.251 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.251 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.251 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.251 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.251 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.251 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.251 INFO:teuthology.orchestra.run.smithi090.stdout: "description": { 2024-04-02T14:50:52.251 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "description", 2024-04-02T14:50:52.251 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.251 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.251 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.251 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:52.252 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.252 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.252 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.252 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.252 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.252 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.252 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.252 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.252 INFO:teuthology.orchestra.run.smithi090.stdout: "device_url": { 2024-04-02T14:50:52.252 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "device_url", 2024-04-02T14:50:52.252 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.252 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.252 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.252 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-04-02T14:50:52.253 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.253 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.253 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.253 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.253 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.253 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.253 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.253 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.253 INFO:teuthology.orchestra.run.smithi090.stdout: "enabled": { 2024-04-02T14:50:52.253 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "enabled", 2024-04-02T14:50:52.253 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.253 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.253 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.253 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:52.253 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.253 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.254 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.254 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.254 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.254 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.254 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.254 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.254 INFO:teuthology.orchestra.run.smithi090.stdout: "interval": { 2024-04-02T14:50:52.254 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "interval", 2024-04-02T14:50:52.254 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-02T14:50:52.254 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.254 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.254 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "24", 2024-04-02T14:50:52.254 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "8", 2024-04-02T14:50:52.254 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.254 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.254 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.255 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.255 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.255 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.255 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.255 INFO:teuthology.orchestra.run.smithi090.stdout: "last_opt_revision": { 2024-04-02T14:50:52.255 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "last_opt_revision", 2024-04-02T14:50:52.255 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-02T14:50:52.255 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.255 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.255 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "1", 2024-04-02T14:50:52.255 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.255 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.255 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.255 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.255 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.255 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.255 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.256 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.256 INFO:teuthology.orchestra.run.smithi090.stdout: "leaderboard": { 2024-04-02T14:50:52.256 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "leaderboard", 2024-04-02T14:50:52.256 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.256 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.256 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.256 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:52.256 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.256 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.256 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.256 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.256 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.256 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.256 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.256 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.256 INFO:teuthology.orchestra.run.smithi090.stdout: "leaderboard_description": { 2024-04-02T14:50:52.257 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "leaderboard_description", 2024-04-02T14:50:52.257 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.257 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.257 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.257 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:52.257 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.257 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.257 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.257 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.257 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.257 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.257 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.257 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.257 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-02T14:50:52.257 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-02T14:50:52.257 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.257 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.258 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.258 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:52.258 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.258 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.258 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:52.258 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:52.258 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:52.258 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:52.258 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:52.258 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:52.258 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:52.258 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:52.258 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.258 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.258 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.258 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.259 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.259 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-02T14:50:52.259 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-02T14:50:52.259 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.259 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.259 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.259 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:52.259 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.259 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.259 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.259 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.259 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.259 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.259 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.259 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.259 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-02T14:50:52.260 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-02T14:50:52.260 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.260 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.260 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.260 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-02T14:50:52.260 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.260 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.260 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:52.260 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:52.260 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:52.260 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:52.260 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:52.260 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:52.260 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:52.260 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:52.260 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.261 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.261 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.261 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.261 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.261 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-02T14:50:52.261 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-02T14:50:52.261 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.261 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.261 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.261 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:52.261 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.261 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.261 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.261 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.261 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.262 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.262 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.262 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.262 INFO:teuthology.orchestra.run.smithi090.stdout: "organization": { 2024-04-02T14:50:52.262 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "organization", 2024-04-02T14:50:52.262 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.262 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.262 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.262 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:52.262 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.262 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.262 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.262 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.262 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.262 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.262 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.263 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.263 INFO:teuthology.orchestra.run.smithi090.stdout: "proxy": { 2024-04-02T14:50:52.263 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "proxy", 2024-04-02T14:50:52.263 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.263 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.263 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.263 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:52.263 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.263 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.263 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.263 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.263 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.263 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.263 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.263 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.263 INFO:teuthology.orchestra.run.smithi090.stdout: "url": { 2024-04-02T14:50:52.264 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "url", 2024-04-02T14:50:52.264 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.264 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.264 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.264 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-04-02T14:50:52.264 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.264 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.264 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.264 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.264 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.264 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.264 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.264 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:52.264 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:52.264 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.264 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-02T14:50:52.264 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "test_orchestrator", 2024-04-02T14:50:52.265 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-02T14:50:52.265 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-02T14:50:52.265 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-02T14:50:52.265 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-02T14:50:52.265 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-02T14:50:52.265 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.265 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.265 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.265 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:52.265 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.265 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.265 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:52.265 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:52.265 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:52.265 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:52.265 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:52.266 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:52.266 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:52.266 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:52.266 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.266 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.266 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.266 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.266 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.266 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-02T14:50:52.266 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-02T14:50:52.266 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.266 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.266 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.266 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:52.266 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.266 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.267 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.267 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.267 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.267 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.267 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.267 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.267 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-02T14:50:52.267 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-02T14:50:52.267 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.267 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.267 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.267 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-02T14:50:52.267 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.267 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.267 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:52.267 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:52.268 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:52.268 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:52.268 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:52.268 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:52.268 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:52.268 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:52.268 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.268 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.268 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.268 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.268 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.268 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-02T14:50:52.268 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-02T14:50:52.268 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.268 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.269 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.269 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:52.269 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.269 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.269 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.269 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.269 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.269 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.269 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.269 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:52.269 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:52.269 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.269 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-02T14:50:52.269 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "volumes", 2024-04-02T14:50:52.269 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-02T14:50:52.269 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-02T14:50:52.270 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-02T14:50:52.270 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-02T14:50:52.270 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-02T14:50:52.270 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.270 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.270 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.270 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:52.270 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.270 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.270 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:52.270 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:52.270 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:52.270 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:52.270 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:52.270 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:52.271 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:52.271 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:52.271 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.271 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.271 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.271 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.271 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.271 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-02T14:50:52.271 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-02T14:50:52.271 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.271 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.271 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.271 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:52.271 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.272 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.272 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.272 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.272 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.272 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.272 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.272 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.272 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-02T14:50:52.272 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-02T14:50:52.272 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.272 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.272 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.272 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-02T14:50:52.272 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.273 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.273 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:52.273 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:52.273 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:52.273 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:52.273 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:52.273 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:52.273 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:52.273 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:52.273 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.273 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.273 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.273 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.273 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.274 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-02T14:50:52.274 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-02T14:50:52.274 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.274 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.274 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.274 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:52.274 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.274 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.274 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.274 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.274 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.274 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.274 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.274 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.275 INFO:teuthology.orchestra.run.smithi090.stdout: "max_concurrent_clones": { 2024-04-02T14:50:52.275 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "max_concurrent_clones", 2024-04-02T14:50:52.275 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-02T14:50:52.275 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.275 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.275 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "4", 2024-04-02T14:50:52.275 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.275 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.275 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.275 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Number of asynchronous cloner threads", 2024-04-02T14:50:52.275 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.275 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.275 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.275 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.275 INFO:teuthology.orchestra.run.smithi090.stdout: "periodic_async_work": { 2024-04-02T14:50:52.275 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "periodic_async_work", 2024-04-02T14:50:52.276 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.276 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.276 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.276 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:52.276 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.276 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.276 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.276 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Periodically check for async work", 2024-04-02T14:50:52.276 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.276 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.276 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.276 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.276 INFO:teuthology.orchestra.run.smithi090.stdout: "snapshot_clone_delay": { 2024-04-02T14:50:52.276 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "snapshot_clone_delay", 2024-04-02T14:50:52.276 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-02T14:50:52.277 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.277 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.277 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "0", 2024-04-02T14:50:52.277 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.277 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.277 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.277 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-04-02T14:50:52.277 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.277 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.277 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.277 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.277 INFO:teuthology.orchestra.run.smithi090.stdout: "snapshot_clone_no_wait": { 2024-04-02T14:50:52.277 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "snapshot_clone_no_wait", 2024-04-02T14:50:52.277 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.277 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.278 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.278 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-02T14:50:52.278 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.278 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.278 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.278 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-04-02T14:50:52.278 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.278 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.278 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.278 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:52.278 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:52.278 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.278 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-02T14:50:52.278 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "zabbix", 2024-04-02T14:50:52.279 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-02T14:50:52.279 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-02T14:50:52.279 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-02T14:50:52.279 INFO:teuthology.orchestra.run.smithi090.stdout: "discovery_interval": { 2024-04-02T14:50:52.279 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "discovery_interval", 2024-04-02T14:50:52.279 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "uint", 2024-04-02T14:50:52.279 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.279 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.279 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "100", 2024-04-02T14:50:52.279 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.279 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.279 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.279 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.279 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.279 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.279 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.280 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.280 INFO:teuthology.orchestra.run.smithi090.stdout: "identifier": { 2024-04-02T14:50:52.280 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "identifier", 2024-04-02T14:50:52.280 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.280 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.280 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.280 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:52.280 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.280 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.280 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.280 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.280 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.280 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.280 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.280 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.281 INFO:teuthology.orchestra.run.smithi090.stdout: "interval": { 2024-04-02T14:50:52.281 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "interval", 2024-04-02T14:50:52.281 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "secs", 2024-04-02T14:50:52.281 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.281 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.281 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "60", 2024-04-02T14:50:52.281 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.281 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.281 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.281 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.281 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.281 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.281 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.281 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.281 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-02T14:50:52.282 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-02T14:50:52.282 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.282 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.282 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.282 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:52.282 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.282 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.282 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:52.282 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:52.282 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:52.282 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:52.282 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:52.282 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:52.282 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:52.283 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:52.283 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.283 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.283 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.283 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.283 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.283 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-02T14:50:52.283 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-02T14:50:52.283 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.283 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.283 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.283 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:52.283 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.283 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.284 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.284 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.284 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.284 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.284 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.284 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.284 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-02T14:50:52.284 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-02T14:50:52.284 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.284 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.284 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.284 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-02T14:50:52.284 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.284 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.284 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:52.285 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:52.285 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:52.285 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:52.285 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:52.285 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:52.285 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:52.285 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:52.285 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.285 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.285 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.285 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.285 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.285 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-02T14:50:52.285 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-02T14:50:52.285 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.285 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.286 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.286 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:52.286 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.286 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.286 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.286 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.286 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.286 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.286 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.286 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.286 INFO:teuthology.orchestra.run.smithi090.stdout: "zabbix_host": { 2024-04-02T14:50:52.286 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "zabbix_host", 2024-04-02T14:50:52.286 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.286 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.286 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.287 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:52.287 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.287 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.287 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.287 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.287 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.287 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.287 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.287 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.287 INFO:teuthology.orchestra.run.smithi090.stdout: "zabbix_port": { 2024-04-02T14:50:52.287 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "zabbix_port", 2024-04-02T14:50:52.287 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-02T14:50:52.287 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.287 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.287 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "10051", 2024-04-02T14:50:52.287 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.287 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.288 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.288 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.288 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.288 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.288 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.288 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.288 INFO:teuthology.orchestra.run.smithi090.stdout: "zabbix_sender": { 2024-04-02T14:50:52.288 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "zabbix_sender", 2024-04-02T14:50:52.288 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.288 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.288 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.288 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-04-02T14:50:52.288 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.288 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.288 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.288 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.289 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.289 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.289 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.289 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:52.289 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:52.289 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:52.289 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-02T14:50:52.289 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:52.289 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:52.289 INFO:teuthology.orchestra.run.smithi090.stdout: "modules": [ 2024-04-02T14:50:52.289 INFO:teuthology.orchestra.run.smithi090.stdout: "cephadm", 2024-04-02T14:50:52.289 INFO:teuthology.orchestra.run.smithi090.stdout: "dashboard", 2024-04-02T14:50:52.289 INFO:teuthology.orchestra.run.smithi090.stdout: "iostat", 2024-04-02T14:50:52.289 INFO:teuthology.orchestra.run.smithi090.stdout: "nfs", 2024-04-02T14:50:52.289 INFO:teuthology.orchestra.run.smithi090.stdout: "restful", 2024-04-02T14:50:52.289 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_schedule" 2024-04-02T14:50:52.289 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:52.290 INFO:teuthology.orchestra.run.smithi090.stdout: "available_modules": [ 2024-04-02T14:50:52.290 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-02T14:50:52.290 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "alerts", 2024-04-02T14:50:52.290 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-02T14:50:52.290 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-02T14:50:52.290 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-02T14:50:52.290 INFO:teuthology.orchestra.run.smithi090.stdout: "interval": { 2024-04-02T14:50:52.290 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "interval", 2024-04-02T14:50:52.290 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "secs", 2024-04-02T14:50:52.290 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.290 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.290 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "60", 2024-04-02T14:50:52.290 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.290 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.290 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.291 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "How frequently to reexamine health status", 2024-04-02T14:50:52.291 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.291 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.291 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.291 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.291 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-02T14:50:52.291 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-02T14:50:52.291 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.291 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.291 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.291 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:52.291 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.291 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.291 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:52.291 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:52.291 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:52.291 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:52.292 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:52.292 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:52.292 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:52.292 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:52.292 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.292 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.292 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.292 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.292 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.292 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-02T14:50:52.292 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-02T14:50:52.292 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.292 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.292 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.292 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:52.292 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.293 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.293 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.293 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.293 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.293 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.293 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.293 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.293 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-02T14:50:52.293 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-02T14:50:52.293 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.293 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.293 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.293 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-02T14:50:52.293 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.293 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.293 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:52.293 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:52.294 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:52.294 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:52.294 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:52.294 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:52.294 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:52.294 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:52.294 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.294 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.294 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.294 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.294 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.294 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-02T14:50:52.294 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-02T14:50:52.294 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.295 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.295 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.295 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:52.295 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.295 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.295 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.295 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.295 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.295 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.295 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.295 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.295 INFO:teuthology.orchestra.run.smithi090.stdout: "smtp_destination": { 2024-04-02T14:50:52.295 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "smtp_destination", 2024-04-02T14:50:52.295 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.295 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.295 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.295 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:52.296 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.296 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.296 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.296 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Email address to send alerts to", 2024-04-02T14:50:52.296 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.296 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.296 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.296 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.296 INFO:teuthology.orchestra.run.smithi090.stdout: "smtp_from_name": { 2024-04-02T14:50:52.296 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "smtp_from_name", 2024-04-02T14:50:52.296 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.296 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.296 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.296 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "Ceph", 2024-04-02T14:50:52.296 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.296 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.296 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.297 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Email From: name", 2024-04-02T14:50:52.297 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.297 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.297 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.297 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.297 INFO:teuthology.orchestra.run.smithi090.stdout: "smtp_host": { 2024-04-02T14:50:52.297 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "smtp_host", 2024-04-02T14:50:52.297 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.297 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.297 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.297 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:52.297 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.297 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.297 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.297 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "SMTP server", 2024-04-02T14:50:52.297 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.297 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.298 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.298 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.298 INFO:teuthology.orchestra.run.smithi090.stdout: "smtp_password": { 2024-04-02T14:50:52.298 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "smtp_password", 2024-04-02T14:50:52.298 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.298 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.298 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.298 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:52.298 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.298 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.298 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.298 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Password to authenticate with", 2024-04-02T14:50:52.298 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.298 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.299 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.299 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.299 INFO:teuthology.orchestra.run.smithi090.stdout: "smtp_port": { 2024-04-02T14:50:52.299 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "smtp_port", 2024-04-02T14:50:52.299 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-02T14:50:52.299 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.299 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.299 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "465", 2024-04-02T14:50:52.299 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.299 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.299 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.299 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "SMTP port", 2024-04-02T14:50:52.299 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.299 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.299 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.299 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.300 INFO:teuthology.orchestra.run.smithi090.stdout: "smtp_sender": { 2024-04-02T14:50:52.300 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "smtp_sender", 2024-04-02T14:50:52.300 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.300 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.300 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.300 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:52.300 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.300 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.300 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.300 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "SMTP envelope sender", 2024-04-02T14:50:52.300 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.300 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.300 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.300 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.300 INFO:teuthology.orchestra.run.smithi090.stdout: "smtp_ssl": { 2024-04-02T14:50:52.300 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "smtp_ssl", 2024-04-02T14:50:52.301 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.301 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.301 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.301 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-02T14:50:52.301 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.301 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.301 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.301 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Use SSL to connect to SMTP server", 2024-04-02T14:50:52.301 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.301 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.301 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.301 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.301 INFO:teuthology.orchestra.run.smithi090.stdout: "smtp_user": { 2024-04-02T14:50:52.301 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "smtp_user", 2024-04-02T14:50:52.301 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.302 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.302 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.302 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:52.302 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.302 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.302 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.302 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "User to authenticate as", 2024-04-02T14:50:52.302 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.302 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.302 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.302 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:52.302 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:52.302 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.302 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-02T14:50:52.302 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "balancer", 2024-04-02T14:50:52.303 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-02T14:50:52.303 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-02T14:50:52.303 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-02T14:50:52.303 INFO:teuthology.orchestra.run.smithi090.stdout: "active": { 2024-04-02T14:50:52.303 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "active", 2024-04-02T14:50:52.303 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.303 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.303 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.303 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-02T14:50:52.303 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.303 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.303 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.303 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "automatically balance PGs across cluster", 2024-04-02T14:50:52.303 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.303 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.304 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.304 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.304 INFO:teuthology.orchestra.run.smithi090.stdout: "begin_time": { 2024-04-02T14:50:52.304 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "begin_time", 2024-04-02T14:50:52.304 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.304 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.304 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.304 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "0000", 2024-04-02T14:50:52.304 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.304 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.304 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.304 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "beginning time of day to automatically balance", 2024-04-02T14:50:52.304 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-02T14:50:52.304 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.304 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.304 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.304 INFO:teuthology.orchestra.run.smithi090.stdout: "begin_weekday": { 2024-04-02T14:50:52.305 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "begin_weekday", 2024-04-02T14:50:52.305 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "uint", 2024-04-02T14:50:52.305 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.305 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.305 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "0", 2024-04-02T14:50:52.305 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "0", 2024-04-02T14:50:52.305 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "6", 2024-04-02T14:50:52.305 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.305 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-04-02T14:50:52.305 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-02T14:50:52.305 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.305 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.305 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.305 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_compat_max_iterations": { 2024-04-02T14:50:52.305 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "crush_compat_max_iterations", 2024-04-02T14:50:52.306 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "uint", 2024-04-02T14:50:52.306 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.306 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.306 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "25", 2024-04-02T14:50:52.306 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "1", 2024-04-02T14:50:52.306 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "250", 2024-04-02T14:50:52.306 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.306 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-04-02T14:50:52.306 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.306 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.306 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.306 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.306 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_compat_metrics": { 2024-04-02T14:50:52.306 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "crush_compat_metrics", 2024-04-02T14:50:52.306 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.307 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.307 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.307 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "pgs,objects,bytes", 2024-04-02T14:50:52.307 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.307 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.307 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.307 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-04-02T14:50:52.307 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-04-02T14:50:52.307 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.307 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.307 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.307 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_compat_step": { 2024-04-02T14:50:52.307 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "crush_compat_step", 2024-04-02T14:50:52.308 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "float", 2024-04-02T14:50:52.308 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.308 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.308 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "0.5", 2024-04-02T14:50:52.308 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "0.001", 2024-04-02T14:50:52.308 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "0.999", 2024-04-02T14:50:52.308 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.308 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "aggressiveness of optimization", 2024-04-02T14:50:52.308 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-04-02T14:50:52.308 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.308 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.308 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.308 INFO:teuthology.orchestra.run.smithi090.stdout: "end_time": { 2024-04-02T14:50:52.308 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "end_time", 2024-04-02T14:50:52.308 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.308 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.309 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.309 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "2359", 2024-04-02T14:50:52.309 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.309 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.309 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.309 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "ending time of day to automatically balance", 2024-04-02T14:50:52.309 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-02T14:50:52.309 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.309 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.309 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.309 INFO:teuthology.orchestra.run.smithi090.stdout: "end_weekday": { 2024-04-02T14:50:52.309 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "end_weekday", 2024-04-02T14:50:52.309 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "uint", 2024-04-02T14:50:52.309 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.309 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.309 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "0", 2024-04-02T14:50:52.309 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "0", 2024-04-02T14:50:52.310 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "6", 2024-04-02T14:50:52.310 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.310 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-04-02T14:50:52.310 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-02T14:50:52.310 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.310 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.310 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.310 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-02T14:50:52.310 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-02T14:50:52.310 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.310 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.310 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.310 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:52.310 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.310 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.311 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:52.311 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:52.311 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:52.311 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:52.311 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:52.311 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:52.311 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:52.311 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:52.311 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.311 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.311 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.311 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.311 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.311 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-02T14:50:52.311 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-02T14:50:52.311 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.312 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.312 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.312 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:52.312 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.312 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.312 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.312 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.312 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.312 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.312 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.312 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.312 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-02T14:50:52.312 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-02T14:50:52.312 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.313 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.313 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.313 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-02T14:50:52.313 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.313 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.313 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:52.313 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:52.313 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:52.313 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:52.313 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:52.313 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:52.313 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:52.313 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:52.313 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.313 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.314 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.314 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.314 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.314 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-02T14:50:52.314 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-02T14:50:52.314 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.314 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.314 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.314 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:52.314 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.314 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.314 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.314 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.314 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.315 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.315 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.315 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.315 INFO:teuthology.orchestra.run.smithi090.stdout: "min_score": { 2024-04-02T14:50:52.315 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "min_score", 2024-04-02T14:50:52.315 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "float", 2024-04-02T14:50:52.315 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.315 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.315 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "0", 2024-04-02T14:50:52.315 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.315 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.315 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.315 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-04-02T14:50:52.315 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.315 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.315 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.316 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.316 INFO:teuthology.orchestra.run.smithi090.stdout: "mode": { 2024-04-02T14:50:52.316 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "mode", 2024-04-02T14:50:52.316 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.316 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.316 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.316 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "upmap", 2024-04-02T14:50:52.316 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.316 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.316 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:52.316 INFO:teuthology.orchestra.run.smithi090.stdout: "crush-compat", 2024-04-02T14:50:52.316 INFO:teuthology.orchestra.run.smithi090.stdout: "none", 2024-04-02T14:50:52.316 INFO:teuthology.orchestra.run.smithi090.stdout: "read", 2024-04-02T14:50:52.316 INFO:teuthology.orchestra.run.smithi090.stdout: "upmap", 2024-04-02T14:50:52.317 INFO:teuthology.orchestra.run.smithi090.stdout: "upmap-read" 2024-04-02T14:50:52.317 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:52.317 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Balancer mode", 2024-04-02T14:50:52.317 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.317 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.317 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.317 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.317 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_ids": { 2024-04-02T14:50:52.317 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "pool_ids", 2024-04-02T14:50:52.317 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.317 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.317 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.317 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:52.317 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.317 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.318 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.318 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-04-02T14:50:52.318 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.318 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.318 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.318 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.318 INFO:teuthology.orchestra.run.smithi090.stdout: "sleep_interval": { 2024-04-02T14:50:52.318 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "sleep_interval", 2024-04-02T14:50:52.318 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "secs", 2024-04-02T14:50:52.318 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.318 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.318 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "60", 2024-04-02T14:50:52.318 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.318 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.319 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.319 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-04-02T14:50:52.319 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.319 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.319 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.319 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.319 INFO:teuthology.orchestra.run.smithi090.stdout: "upmap_max_deviation": { 2024-04-02T14:50:52.319 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "upmap_max_deviation", 2024-04-02T14:50:52.319 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-02T14:50:52.319 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.319 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.319 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "5", 2024-04-02T14:50:52.319 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "1", 2024-04-02T14:50:52.319 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.319 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.319 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "deviation below which no optimization is attempted", 2024-04-02T14:50:52.320 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-04-02T14:50:52.320 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.320 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.320 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.320 INFO:teuthology.orchestra.run.smithi090.stdout: "upmap_max_optimizations": { 2024-04-02T14:50:52.320 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "upmap_max_optimizations", 2024-04-02T14:50:52.320 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "uint", 2024-04-02T14:50:52.320 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.320 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.320 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "10", 2024-04-02T14:50:52.320 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.320 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.320 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.320 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-04-02T14:50:52.320 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.320 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.321 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.321 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:52.321 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:52.321 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.321 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-02T14:50:52.321 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "cephadm", 2024-04-02T14:50:52.321 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-02T14:50:52.321 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-02T14:50:52.321 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-02T14:50:52.321 INFO:teuthology.orchestra.run.smithi090.stdout: "agent_down_multiplier": { 2024-04-02T14:50:52.321 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "agent_down_multiplier", 2024-04-02T14:50:52.321 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "float", 2024-04-02T14:50:52.321 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.321 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.321 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "3.0", 2024-04-02T14:50:52.321 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.321 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.322 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.322 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-04-02T14:50:52.322 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.322 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.322 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.322 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.322 INFO:teuthology.orchestra.run.smithi090.stdout: "agent_refresh_rate": { 2024-04-02T14:50:52.322 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "agent_refresh_rate", 2024-04-02T14:50:52.322 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "secs", 2024-04-02T14:50:52.322 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.322 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.322 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "20", 2024-04-02T14:50:52.322 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.322 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.322 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.323 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-04-02T14:50:52.323 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.323 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.323 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.323 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.323 INFO:teuthology.orchestra.run.smithi090.stdout: "agent_starting_port": { 2024-04-02T14:50:52.323 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "agent_starting_port", 2024-04-02T14:50:52.323 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-02T14:50:52.323 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.323 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.323 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "4721", 2024-04-02T14:50:52.323 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.323 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.323 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.323 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-04-02T14:50:52.324 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.324 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.324 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.324 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.324 INFO:teuthology.orchestra.run.smithi090.stdout: "allow_ptrace": { 2024-04-02T14:50:52.324 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "allow_ptrace", 2024-04-02T14:50:52.324 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.324 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.324 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.324 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:52.324 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.324 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.324 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.324 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-04-02T14:50:52.324 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-04-02T14:50:52.324 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.325 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.325 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.325 INFO:teuthology.orchestra.run.smithi090.stdout: "autotune_interval": { 2024-04-02T14:50:52.325 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "autotune_interval", 2024-04-02T14:50:52.325 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "secs", 2024-04-02T14:50:52.325 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.325 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.325 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "600", 2024-04-02T14:50:52.325 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.325 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.325 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.325 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "how frequently to autotune daemon memory", 2024-04-02T14:50:52.325 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.325 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.325 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.325 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.326 INFO:teuthology.orchestra.run.smithi090.stdout: "autotune_memory_target_ratio": { 2024-04-02T14:50:52.326 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "autotune_memory_target_ratio", 2024-04-02T14:50:52.326 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "float", 2024-04-02T14:50:52.326 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.326 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.326 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "0.7", 2024-04-02T14:50:52.326 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.326 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.326 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.326 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-04-02T14:50:52.326 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.326 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.326 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.326 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.326 INFO:teuthology.orchestra.run.smithi090.stdout: "cephadm_log_destination": { 2024-04-02T14:50:52.327 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "cephadm_log_destination", 2024-04-02T14:50:52.327 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.327 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.327 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.327 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:52.327 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.327 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.327 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:52.327 INFO:teuthology.orchestra.run.smithi090.stdout: "file", 2024-04-02T14:50:52.327 INFO:teuthology.orchestra.run.smithi090.stdout: "file,syslog", 2024-04-02T14:50:52.327 INFO:teuthology.orchestra.run.smithi090.stdout: "syslog" 2024-04-02T14:50:52.327 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:52.327 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-04-02T14:50:52.327 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.327 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.327 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.327 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.328 INFO:teuthology.orchestra.run.smithi090.stdout: "cgroups_split": { 2024-04-02T14:50:52.328 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "cgroups_split", 2024-04-02T14:50:52.328 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.328 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.328 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.328 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-02T14:50:52.328 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.328 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.328 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.328 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-04-02T14:50:52.328 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.328 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.328 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.328 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.328 INFO:teuthology.orchestra.run.smithi090.stdout: "config_checks_enabled": { 2024-04-02T14:50:52.329 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "config_checks_enabled", 2024-04-02T14:50:52.329 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.329 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.329 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.329 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:52.329 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.329 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.329 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.329 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-04-02T14:50:52.329 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.329 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.329 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.329 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.329 INFO:teuthology.orchestra.run.smithi090.stdout: "config_dashboard": { 2024-04-02T14:50:52.329 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "config_dashboard", 2024-04-02T14:50:52.329 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.330 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.330 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.330 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-02T14:50:52.330 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.330 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.330 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.330 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-04-02T14:50:52.330 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.330 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.330 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.330 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.330 INFO:teuthology.orchestra.run.smithi090.stdout: "container_image_alertmanager": { 2024-04-02T14:50:52.330 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "container_image_alertmanager", 2024-04-02T14:50:52.330 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.330 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.331 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.331 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-04-02T14:50:52.331 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.331 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.331 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.331 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Prometheus container image", 2024-04-02T14:50:52.331 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.331 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.331 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.331 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.331 INFO:teuthology.orchestra.run.smithi090.stdout: "container_image_base": { 2024-04-02T14:50:52.331 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "container_image_base", 2024-04-02T14:50:52.331 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.331 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.331 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.331 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "quay.io/ceph/ceph", 2024-04-02T14:50:52.331 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.332 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.332 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.332 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Container image name, without the tag", 2024-04-02T14:50:52.332 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.332 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.332 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.332 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.332 INFO:teuthology.orchestra.run.smithi090.stdout: "container_image_elasticsearch": { 2024-04-02T14:50:52.332 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "container_image_elasticsearch", 2024-04-02T14:50:52.332 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.332 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.332 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.332 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-04-02T14:50:52.332 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.332 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.333 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.333 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "elasticsearch container image", 2024-04-02T14:50:52.333 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.333 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.333 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.333 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.333 INFO:teuthology.orchestra.run.smithi090.stdout: "container_image_grafana": { 2024-04-02T14:50:52.333 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "container_image_grafana", 2024-04-02T14:50:52.333 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.333 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.333 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.333 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-04-02T14:50:52.333 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.333 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.333 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.333 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Prometheus container image", 2024-04-02T14:50:52.333 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.334 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.334 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.334 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.334 INFO:teuthology.orchestra.run.smithi090.stdout: "container_image_haproxy": { 2024-04-02T14:50:52.334 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "container_image_haproxy", 2024-04-02T14:50:52.334 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.334 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.334 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.334 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-04-02T14:50:52.334 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.334 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.334 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.334 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "HAproxy container image", 2024-04-02T14:50:52.334 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.334 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.335 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.335 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.335 INFO:teuthology.orchestra.run.smithi090.stdout: "container_image_jaeger_agent": { 2024-04-02T14:50:52.335 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "container_image_jaeger_agent", 2024-04-02T14:50:52.335 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.335 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.335 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.335 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-04-02T14:50:52.335 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.335 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.335 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.335 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Jaeger agent container image", 2024-04-02T14:50:52.335 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.335 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.335 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.336 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.336 INFO:teuthology.orchestra.run.smithi090.stdout: "container_image_jaeger_collector": { 2024-04-02T14:50:52.336 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "container_image_jaeger_collector", 2024-04-02T14:50:52.336 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.336 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.336 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.336 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-04-02T14:50:52.336 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.336 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.336 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.336 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Jaeger collector container image", 2024-04-02T14:50:52.336 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.336 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.336 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.336 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.337 INFO:teuthology.orchestra.run.smithi090.stdout: "container_image_jaeger_query": { 2024-04-02T14:50:52.337 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "container_image_jaeger_query", 2024-04-02T14:50:52.337 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.337 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.337 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.337 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-04-02T14:50:52.337 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.337 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.337 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.337 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Jaeger query container image", 2024-04-02T14:50:52.337 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.337 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.337 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.337 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.337 INFO:teuthology.orchestra.run.smithi090.stdout: "container_image_keepalived": { 2024-04-02T14:50:52.337 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "container_image_keepalived", 2024-04-02T14:50:52.337 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.338 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.338 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.338 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-04-02T14:50:52.338 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.338 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.338 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.338 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Keepalived container image", 2024-04-02T14:50:52.338 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.338 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.338 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.338 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.338 INFO:teuthology.orchestra.run.smithi090.stdout: "container_image_loki": { 2024-04-02T14:50:52.338 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "container_image_loki", 2024-04-02T14:50:52.338 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.338 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.339 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.339 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-04-02T14:50:52.339 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.339 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.339 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.339 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Loki container image", 2024-04-02T14:50:52.339 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.339 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.339 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.339 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.339 INFO:teuthology.orchestra.run.smithi090.stdout: "container_image_node_exporter": { 2024-04-02T14:50:52.339 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "container_image_node_exporter", 2024-04-02T14:50:52.339 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.339 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.339 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.339 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-04-02T14:50:52.340 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.340 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.340 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.340 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Prometheus container image", 2024-04-02T14:50:52.340 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.340 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.340 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.340 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.340 INFO:teuthology.orchestra.run.smithi090.stdout: "container_image_nvmeof": { 2024-04-02T14:50:52.340 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "container_image_nvmeof", 2024-04-02T14:50:52.340 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.340 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.340 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.340 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "quay.io/ceph/nvmeof:1.0.0", 2024-04-02T14:50:52.340 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.340 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.341 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.341 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Nvme-of container image", 2024-04-02T14:50:52.341 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.341 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.341 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.341 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.341 INFO:teuthology.orchestra.run.smithi090.stdout: "container_image_prometheus": { 2024-04-02T14:50:52.341 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "container_image_prometheus", 2024-04-02T14:50:52.341 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.341 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.341 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.341 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-04-02T14:50:52.341 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.341 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.341 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.341 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Prometheus container image", 2024-04-02T14:50:52.341 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.342 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.342 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.342 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.342 INFO:teuthology.orchestra.run.smithi090.stdout: "container_image_promtail": { 2024-04-02T14:50:52.342 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "container_image_promtail", 2024-04-02T14:50:52.342 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.342 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.342 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.342 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-04-02T14:50:52.342 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.342 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.342 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.342 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Promtail container image", 2024-04-02T14:50:52.342 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.343 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.343 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.343 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.343 INFO:teuthology.orchestra.run.smithi090.stdout: "container_image_samba": { 2024-04-02T14:50:52.343 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "container_image_samba", 2024-04-02T14:50:52.343 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.343 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.343 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.343 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-04-02T14:50:52.343 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.343 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.343 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.343 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Samba/SMB container image", 2024-04-02T14:50:52.343 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.343 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.344 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.344 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.344 INFO:teuthology.orchestra.run.smithi090.stdout: "container_image_snmp_gateway": { 2024-04-02T14:50:52.344 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "container_image_snmp_gateway", 2024-04-02T14:50:52.344 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.344 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.344 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.344 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-04-02T14:50:52.344 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.344 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.344 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.344 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "SNMP Gateway container image", 2024-04-02T14:50:52.344 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.344 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.344 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.344 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.344 INFO:teuthology.orchestra.run.smithi090.stdout: "container_init": { 2024-04-02T14:50:52.345 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "container_init", 2024-04-02T14:50:52.345 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.345 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.345 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.345 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-02T14:50:52.345 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.345 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.345 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.345 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Run podman/docker with `--init`", 2024-04-02T14:50:52.345 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.345 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.345 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.345 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.345 INFO:teuthology.orchestra.run.smithi090.stdout: "daemon_cache_timeout": { 2024-04-02T14:50:52.345 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "daemon_cache_timeout", 2024-04-02T14:50:52.345 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "secs", 2024-04-02T14:50:52.346 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.346 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.346 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "600", 2024-04-02T14:50:52.346 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.346 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.346 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.346 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-04-02T14:50:52.346 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.346 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.346 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.346 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.346 INFO:teuthology.orchestra.run.smithi090.stdout: "default_cephadm_command_timeout": { 2024-04-02T14:50:52.346 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "default_cephadm_command_timeout", 2024-04-02T14:50:52.346 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-02T14:50:52.346 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.347 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.347 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "900", 2024-04-02T14:50:52.347 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.347 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.347 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.347 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-04-02T14:50:52.347 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.347 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.347 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.347 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.347 INFO:teuthology.orchestra.run.smithi090.stdout: "default_registry": { 2024-04-02T14:50:52.347 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "default_registry", 2024-04-02T14:50:52.347 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.347 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.347 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.347 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "docker.io", 2024-04-02T14:50:52.347 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.348 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.348 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.348 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-04-02T14:50:52.348 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.348 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.348 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.348 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.348 INFO:teuthology.orchestra.run.smithi090.stdout: "device_cache_timeout": { 2024-04-02T14:50:52.348 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "device_cache_timeout", 2024-04-02T14:50:52.348 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "secs", 2024-04-02T14:50:52.348 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.348 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.348 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "1800", 2024-04-02T14:50:52.348 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.348 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.349 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.349 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "seconds to cache device inventory", 2024-04-02T14:50:52.349 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.349 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.349 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.349 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.349 INFO:teuthology.orchestra.run.smithi090.stdout: "device_enhanced_scan": { 2024-04-02T14:50:52.349 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "device_enhanced_scan", 2024-04-02T14:50:52.349 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.349 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.349 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.349 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:52.349 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.349 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.349 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.349 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Use libstoragemgmt during device scans", 2024-04-02T14:50:52.349 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.350 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.350 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.350 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.350 INFO:teuthology.orchestra.run.smithi090.stdout: "facts_cache_timeout": { 2024-04-02T14:50:52.350 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "facts_cache_timeout", 2024-04-02T14:50:52.350 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "secs", 2024-04-02T14:50:52.350 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.350 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.350 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "60", 2024-04-02T14:50:52.350 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.350 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.350 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.350 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "seconds to cache host facts data", 2024-04-02T14:50:52.350 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.350 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.350 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.351 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.351 INFO:teuthology.orchestra.run.smithi090.stdout: "grafana_dashboards_path": { 2024-04-02T14:50:52.351 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "grafana_dashboards_path", 2024-04-02T14:50:52.351 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.351 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.351 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.351 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-04-02T14:50:52.351 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.351 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.351 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.351 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-04-02T14:50:52.351 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.351 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.351 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.351 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.352 INFO:teuthology.orchestra.run.smithi090.stdout: "host_check_interval": { 2024-04-02T14:50:52.352 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "host_check_interval", 2024-04-02T14:50:52.352 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "secs", 2024-04-02T14:50:52.352 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.352 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.352 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "600", 2024-04-02T14:50:52.352 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.352 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.352 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.352 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "how frequently to perform a host check", 2024-04-02T14:50:52.352 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.352 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.352 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.352 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.353 INFO:teuthology.orchestra.run.smithi090.stdout: "hw_monitoring": { 2024-04-02T14:50:52.353 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "hw_monitoring", 2024-04-02T14:50:52.353 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.353 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.353 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.353 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:52.353 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.353 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.353 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.353 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-04-02T14:50:52.353 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.353 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.353 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.353 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.353 INFO:teuthology.orchestra.run.smithi090.stdout: "inventory_list_all": { 2024-04-02T14:50:52.353 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "inventory_list_all", 2024-04-02T14:50:52.354 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.354 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.354 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.354 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:52.354 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.354 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.354 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.354 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-04-02T14:50:52.354 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.354 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.354 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.354 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.354 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-02T14:50:52.354 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-02T14:50:52.354 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.355 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.355 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.355 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:52.355 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.355 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.355 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:52.355 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:52.355 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:52.355 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:52.355 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:52.355 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:52.355 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:52.355 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:52.355 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.355 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.355 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.356 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.356 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.356 INFO:teuthology.orchestra.run.smithi090.stdout: "log_refresh_metadata": { 2024-04-02T14:50:52.356 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_refresh_metadata", 2024-04-02T14:50:52.356 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.356 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.356 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.356 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:52.356 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.356 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.356 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.356 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-04-02T14:50:52.356 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.356 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.356 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.356 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.356 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-02T14:50:52.357 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-02T14:50:52.357 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.357 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.357 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.357 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-02T14:50:52.357 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.357 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.357 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.357 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-04-02T14:50:52.357 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.357 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.357 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.357 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.357 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-02T14:50:52.357 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-02T14:50:52.357 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.357 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.358 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.358 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-02T14:50:52.358 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.358 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.358 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:52.358 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:52.358 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:52.358 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:52.358 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:52.358 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:52.358 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:52.358 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:52.358 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.358 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.358 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.359 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.359 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.359 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-02T14:50:52.359 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-02T14:50:52.359 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.359 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.359 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.359 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:52.359 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.359 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.359 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.359 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.359 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.359 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.359 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.360 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.360 INFO:teuthology.orchestra.run.smithi090.stdout: "manage_etc_ceph_ceph_conf": { 2024-04-02T14:50:52.360 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-04-02T14:50:52.360 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.360 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.360 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.360 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:52.360 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.360 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.360 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.360 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-04-02T14:50:52.360 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.360 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.360 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.360 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.361 INFO:teuthology.orchestra.run.smithi090.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-04-02T14:50:52.361 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-04-02T14:50:52.361 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.361 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.361 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.361 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "*", 2024-04-02T14:50:52.361 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.361 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.361 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.361 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-04-02T14:50:52.361 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.361 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.361 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.361 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.361 INFO:teuthology.orchestra.run.smithi090.stdout: "max_count_per_host": { 2024-04-02T14:50:52.361 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "max_count_per_host", 2024-04-02T14:50:52.361 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-02T14:50:52.362 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.362 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.362 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "10", 2024-04-02T14:50:52.362 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.362 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.362 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.362 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "max number of daemons per service per host", 2024-04-02T14:50:52.362 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.362 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.362 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.362 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.362 INFO:teuthology.orchestra.run.smithi090.stdout: "max_osd_draining_count": { 2024-04-02T14:50:52.362 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "max_osd_draining_count", 2024-04-02T14:50:52.362 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-02T14:50:52.363 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.363 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.363 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "10", 2024-04-02T14:50:52.363 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.363 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.363 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.363 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-04-02T14:50:52.363 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.363 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.363 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.363 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.363 INFO:teuthology.orchestra.run.smithi090.stdout: "migration_current": { 2024-04-02T14:50:52.363 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "migration_current", 2024-04-02T14:50:52.363 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-02T14:50:52.363 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.364 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.364 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:52.364 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.364 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.364 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.364 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "internal - do not modify", 2024-04-02T14:50:52.364 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.364 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.364 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.364 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.364 INFO:teuthology.orchestra.run.smithi090.stdout: "mode": { 2024-04-02T14:50:52.364 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "mode", 2024-04-02T14:50:52.364 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.364 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.364 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.364 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "root", 2024-04-02T14:50:52.364 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.365 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.365 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:52.365 INFO:teuthology.orchestra.run.smithi090.stdout: "cephadm-package", 2024-04-02T14:50:52.365 INFO:teuthology.orchestra.run.smithi090.stdout: "root" 2024-04-02T14:50:52.365 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:52.365 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "mode for remote execution of cephadm", 2024-04-02T14:50:52.365 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.365 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.365 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.365 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.365 INFO:teuthology.orchestra.run.smithi090.stdout: "oob_default_addr": { 2024-04-02T14:50:52.365 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "oob_default_addr", 2024-04-02T14:50:52.365 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.365 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.365 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.365 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "169.254.1.1", 2024-04-02T14:50:52.366 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.366 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.366 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.366 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Default address for RedFish API (oob management).", 2024-04-02T14:50:52.366 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.366 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.366 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.366 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.366 INFO:teuthology.orchestra.run.smithi090.stdout: "prometheus_alerts_path": { 2024-04-02T14:50:52.366 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "prometheus_alerts_path", 2024-04-02T14:50:52.366 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.366 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.366 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.366 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-04-02T14:50:52.367 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.367 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.367 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.367 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-04-02T14:50:52.367 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.367 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.367 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.367 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.367 INFO:teuthology.orchestra.run.smithi090.stdout: "registry_insecure": { 2024-04-02T14:50:52.367 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "registry_insecure", 2024-04-02T14:50:52.367 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.367 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.367 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.367 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:52.367 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.367 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.367 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.368 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-04-02T14:50:52.368 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.368 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.368 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.368 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.368 INFO:teuthology.orchestra.run.smithi090.stdout: "registry_password": { 2024-04-02T14:50:52.368 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "registry_password", 2024-04-02T14:50:52.368 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.368 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.368 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.368 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:52.368 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.368 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.368 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.368 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-04-02T14:50:52.369 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.369 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.369 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.369 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.369 INFO:teuthology.orchestra.run.smithi090.stdout: "registry_url": { 2024-04-02T14:50:52.369 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "registry_url", 2024-04-02T14:50:52.369 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.369 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.369 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.369 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:52.369 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.369 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.369 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.369 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-04-02T14:50:52.369 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.369 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.369 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.370 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.370 INFO:teuthology.orchestra.run.smithi090.stdout: "registry_username": { 2024-04-02T14:50:52.370 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "registry_username", 2024-04-02T14:50:52.370 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.370 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.370 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.370 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:52.370 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.370 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.370 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.370 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-04-02T14:50:52.370 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.370 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.370 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.370 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.371 INFO:teuthology.orchestra.run.smithi090.stdout: "secure_monitoring_stack": { 2024-04-02T14:50:52.371 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "secure_monitoring_stack", 2024-04-02T14:50:52.371 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.371 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.371 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.371 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:52.371 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.371 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.371 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.371 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-04-02T14:50:52.371 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.371 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.371 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.371 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.371 INFO:teuthology.orchestra.run.smithi090.stdout: "service_discovery_port": { 2024-04-02T14:50:52.371 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "service_discovery_port", 2024-04-02T14:50:52.372 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-02T14:50:52.372 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.372 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.372 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "8765", 2024-04-02T14:50:52.372 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.372 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.372 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.372 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "cephadm service discovery port", 2024-04-02T14:50:52.372 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.372 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.372 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.372 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.372 INFO:teuthology.orchestra.run.smithi090.stdout: "ssh_config_file": { 2024-04-02T14:50:52.372 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "ssh_config_file", 2024-04-02T14:50:52.373 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.373 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.373 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.373 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:52.373 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.373 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.373 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.373 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-04-02T14:50:52.373 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.373 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.373 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.373 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.373 INFO:teuthology.orchestra.run.smithi090.stdout: "use_agent": { 2024-04-02T14:50:52.373 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "use_agent", 2024-04-02T14:50:52.373 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.373 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.373 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.374 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:52.374 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.374 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.374 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.374 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-04-02T14:50:52.374 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.374 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.374 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.374 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.374 INFO:teuthology.orchestra.run.smithi090.stdout: "use_repo_digest": { 2024-04-02T14:50:52.374 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "use_repo_digest", 2024-04-02T14:50:52.374 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.374 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.374 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.374 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-02T14:50:52.375 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.375 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.375 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.375 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-04-02T14:50:52.375 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.375 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.375 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.375 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.375 INFO:teuthology.orchestra.run.smithi090.stdout: "warn_on_failed_host_check": { 2024-04-02T14:50:52.375 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "warn_on_failed_host_check", 2024-04-02T14:50:52.375 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.375 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.375 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.375 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-02T14:50:52.375 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.375 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.376 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.376 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "raise a health warning if the host check fails", 2024-04-02T14:50:52.376 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.376 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.376 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.376 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.376 INFO:teuthology.orchestra.run.smithi090.stdout: "warn_on_stray_daemons": { 2024-04-02T14:50:52.376 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "warn_on_stray_daemons", 2024-04-02T14:50:52.376 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.376 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.376 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.376 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-02T14:50:52.376 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.376 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.376 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.376 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-04-02T14:50:52.376 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.377 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.377 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.377 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.377 INFO:teuthology.orchestra.run.smithi090.stdout: "warn_on_stray_hosts": { 2024-04-02T14:50:52.377 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "warn_on_stray_hosts", 2024-04-02T14:50:52.377 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.377 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.377 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.377 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-02T14:50:52.377 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.377 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.377 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.377 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-04-02T14:50:52.377 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.377 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.377 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.377 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:52.378 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:52.378 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.378 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-02T14:50:52.378 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "crash", 2024-04-02T14:50:52.378 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-02T14:50:52.378 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-02T14:50:52.378 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-02T14:50:52.378 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-02T14:50:52.378 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-02T14:50:52.378 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.378 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.378 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.378 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:52.378 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.379 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.379 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:52.379 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:52.379 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:52.379 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:52.379 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:52.379 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:52.379 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:52.379 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:52.379 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.379 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.379 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.379 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.379 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.379 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-02T14:50:52.380 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-02T14:50:52.380 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.380 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.380 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.380 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:52.380 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.380 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.380 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.380 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.380 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.380 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.380 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.380 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.380 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-02T14:50:52.380 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-02T14:50:52.381 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.381 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.381 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.381 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-02T14:50:52.381 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.381 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.381 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:52.381 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:52.381 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:52.381 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:52.381 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:52.381 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:52.381 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:52.381 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:52.381 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.381 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.382 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.382 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.382 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.382 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-02T14:50:52.382 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-02T14:50:52.382 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.382 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.382 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.382 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:52.382 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.382 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.382 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.382 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.382 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.382 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.383 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.383 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.383 INFO:teuthology.orchestra.run.smithi090.stdout: "retain_interval": { 2024-04-02T14:50:52.383 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "retain_interval", 2024-04-02T14:50:52.383 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "secs", 2024-04-02T14:50:52.383 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.383 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.383 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "31536000", 2024-04-02T14:50:52.383 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.383 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.383 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.383 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "how long to retain crashes before pruning them", 2024-04-02T14:50:52.383 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.383 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.383 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.383 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.383 INFO:teuthology.orchestra.run.smithi090.stdout: "warn_recent_interval": { 2024-04-02T14:50:52.384 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "warn_recent_interval", 2024-04-02T14:50:52.384 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "secs", 2024-04-02T14:50:52.384 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.384 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.384 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "1209600", 2024-04-02T14:50:52.384 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.384 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.384 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.384 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "time interval in which to warn about recent crashes", 2024-04-02T14:50:52.384 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.384 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.384 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.384 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:52.384 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:52.384 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.384 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-02T14:50:52.385 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "dashboard", 2024-04-02T14:50:52.385 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-02T14:50:52.385 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-02T14:50:52.385 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-02T14:50:52.385 INFO:teuthology.orchestra.run.smithi090.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-04-02T14:50:52.385 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-04-02T14:50:52.385 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-02T14:50:52.385 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.385 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.385 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "10", 2024-04-02T14:50:52.385 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.385 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.385 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.385 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.385 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.385 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.386 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.386 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.386 INFO:teuthology.orchestra.run.smithi090.stdout: "ALERTMANAGER_API_HOST": { 2024-04-02T14:50:52.386 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "ALERTMANAGER_API_HOST", 2024-04-02T14:50:52.386 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.386 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.386 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.386 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:52.386 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.386 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.386 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.386 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.386 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.386 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.386 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.386 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.387 INFO:teuthology.orchestra.run.smithi090.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-04-02T14:50:52.387 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-04-02T14:50:52.387 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.387 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.387 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.387 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-02T14:50:52.387 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.387 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.387 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.387 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.387 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.387 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.387 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.387 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.387 INFO:teuthology.orchestra.run.smithi090.stdout: "AUDIT_API_ENABLED": { 2024-04-02T14:50:52.387 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "AUDIT_API_ENABLED", 2024-04-02T14:50:52.387 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.388 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.388 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.388 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:52.388 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.388 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.388 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.388 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.388 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.388 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.388 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.388 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.388 INFO:teuthology.orchestra.run.smithi090.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-04-02T14:50:52.388 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-04-02T14:50:52.388 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.388 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.388 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.388 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-02T14:50:52.389 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.389 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.389 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.389 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.389 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.389 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.389 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.389 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.389 INFO:teuthology.orchestra.run.smithi090.stdout: "ENABLE_BROWSABLE_API": { 2024-04-02T14:50:52.389 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "ENABLE_BROWSABLE_API", 2024-04-02T14:50:52.389 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.389 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.389 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.389 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-02T14:50:52.389 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.389 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.389 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.390 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.390 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.390 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.390 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.390 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.390 INFO:teuthology.orchestra.run.smithi090.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-04-02T14:50:52.390 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-04-02T14:50:52.390 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.390 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.390 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.390 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-02T14:50:52.390 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.390 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.390 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.390 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.390 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.391 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.391 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.391 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.391 INFO:teuthology.orchestra.run.smithi090.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-04-02T14:50:52.391 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-04-02T14:50:52.391 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.391 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.391 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.391 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-02T14:50:52.391 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.391 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.391 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.391 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.391 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.391 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.391 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.391 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.392 INFO:teuthology.orchestra.run.smithi090.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-04-02T14:50:52.392 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-04-02T14:50:52.392 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.392 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.392 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.392 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-02T14:50:52.392 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.392 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.392 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.392 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.392 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.392 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.392 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.392 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.392 INFO:teuthology.orchestra.run.smithi090.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-04-02T14:50:52.392 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-04-02T14:50:52.393 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.393 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.393 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.393 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-02T14:50:52.393 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.393 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.393 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.393 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.393 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.393 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.393 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.393 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.393 INFO:teuthology.orchestra.run.smithi090.stdout: "FEATURE_TOGGLE_NFS": { 2024-04-02T14:50:52.393 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-04-02T14:50:52.393 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.394 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.394 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.394 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-02T14:50:52.394 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.394 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.394 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.394 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.394 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.394 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.394 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.394 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.394 INFO:teuthology.orchestra.run.smithi090.stdout: "FEATURE_TOGGLE_RBD": { 2024-04-02T14:50:52.394 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-04-02T14:50:52.394 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.394 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.395 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.395 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-02T14:50:52.395 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.395 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.395 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.395 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.395 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.395 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.395 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.395 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.395 INFO:teuthology.orchestra.run.smithi090.stdout: "FEATURE_TOGGLE_RGW": { 2024-04-02T14:50:52.395 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-04-02T14:50:52.395 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.395 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.395 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.395 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-02T14:50:52.396 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.396 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.396 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.396 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.396 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.396 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.396 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.396 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.396 INFO:teuthology.orchestra.run.smithi090.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-04-02T14:50:52.396 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-04-02T14:50:52.396 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.396 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.396 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.396 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:52.396 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.396 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.396 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.397 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.397 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.397 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.397 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.397 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.397 INFO:teuthology.orchestra.run.smithi090.stdout: "GRAFANA_API_PASSWORD": { 2024-04-02T14:50:52.397 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "GRAFANA_API_PASSWORD", 2024-04-02T14:50:52.397 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.397 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.397 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.397 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "admin", 2024-04-02T14:50:52.397 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.397 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.397 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.397 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.397 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.398 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.398 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.398 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.398 INFO:teuthology.orchestra.run.smithi090.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-04-02T14:50:52.398 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-04-02T14:50:52.398 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.398 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.398 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.398 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-02T14:50:52.398 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.398 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.398 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.398 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.398 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.398 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.399 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.399 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.399 INFO:teuthology.orchestra.run.smithi090.stdout: "GRAFANA_API_URL": { 2024-04-02T14:50:52.399 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "GRAFANA_API_URL", 2024-04-02T14:50:52.399 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.399 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.399 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.399 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:52.399 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.399 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.399 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.399 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.399 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.399 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.399 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.399 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.399 INFO:teuthology.orchestra.run.smithi090.stdout: "GRAFANA_API_USERNAME": { 2024-04-02T14:50:52.400 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "GRAFANA_API_USERNAME", 2024-04-02T14:50:52.400 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.400 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.400 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.400 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "admin", 2024-04-02T14:50:52.400 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.400 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.400 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.400 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.400 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.400 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.400 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.400 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.400 INFO:teuthology.orchestra.run.smithi090.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-04-02T14:50:52.400 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-04-02T14:50:52.400 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.401 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.401 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.401 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:52.401 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.401 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.401 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.401 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.401 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.401 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.401 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.401 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.401 INFO:teuthology.orchestra.run.smithi090.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-04-02T14:50:52.401 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-04-02T14:50:52.401 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.401 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.401 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.402 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:52.402 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.402 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.402 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.402 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.402 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.402 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.402 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.402 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.402 INFO:teuthology.orchestra.run.smithi090.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-04-02T14:50:52.402 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-04-02T14:50:52.402 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.402 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.402 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.403 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-02T14:50:52.403 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.403 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.403 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.403 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.403 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.403 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.403 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.403 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.403 INFO:teuthology.orchestra.run.smithi090.stdout: "ISSUE_TRACKER_API_KEY": { 2024-04-02T14:50:52.403 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-04-02T14:50:52.403 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.403 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.403 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.403 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:52.404 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.404 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.404 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.404 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.404 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.404 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.404 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.404 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.404 INFO:teuthology.orchestra.run.smithi090.stdout: "MANAGED_BY_CLUSTERS": { 2024-04-02T14:50:52.404 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "MANAGED_BY_CLUSTERS", 2024-04-02T14:50:52.404 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.404 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.404 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.404 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "[]", 2024-04-02T14:50:52.404 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.404 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.405 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.405 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.405 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.405 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.405 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.405 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.405 INFO:teuthology.orchestra.run.smithi090.stdout: "MULTICLUSTER_CONFIG": { 2024-04-02T14:50:52.405 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "MULTICLUSTER_CONFIG", 2024-04-02T14:50:52.405 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.405 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.405 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.405 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "{}", 2024-04-02T14:50:52.405 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.405 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.405 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.406 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.406 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.406 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.406 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.406 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.406 INFO:teuthology.orchestra.run.smithi090.stdout: "PROMETHEUS_API_HOST": { 2024-04-02T14:50:52.406 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "PROMETHEUS_API_HOST", 2024-04-02T14:50:52.406 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.406 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.406 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.406 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:52.406 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.406 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.406 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.406 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.406 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.407 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.407 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.407 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.407 INFO:teuthology.orchestra.run.smithi090.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-04-02T14:50:52.407 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-04-02T14:50:52.407 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.407 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.407 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.407 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-02T14:50:52.407 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.407 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.407 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.407 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.407 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.407 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.407 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.408 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.408 INFO:teuthology.orchestra.run.smithi090.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-04-02T14:50:52.408 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-04-02T14:50:52.408 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.408 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.408 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.408 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:52.408 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.408 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.408 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.408 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.408 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.408 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.408 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.408 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.408 INFO:teuthology.orchestra.run.smithi090.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-04-02T14:50:52.409 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-04-02T14:50:52.409 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.409 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.409 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.409 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:52.409 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.409 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.409 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.409 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.409 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.409 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.409 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.409 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.409 INFO:teuthology.orchestra.run.smithi090.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-04-02T14:50:52.409 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-04-02T14:50:52.409 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.410 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.410 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.410 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-02T14:50:52.410 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.410 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.410 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.410 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.410 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.410 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.410 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.410 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.410 INFO:teuthology.orchestra.run.smithi090.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-04-02T14:50:52.410 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-04-02T14:50:52.410 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.410 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.411 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.411 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-02T14:50:52.411 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.411 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.411 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.411 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.411 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.411 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.411 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.411 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.411 INFO:teuthology.orchestra.run.smithi090.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-04-02T14:50:52.411 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-04-02T14:50:52.411 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.411 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.411 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.411 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:52.411 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.412 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.412 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.412 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.412 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.412 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.412 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.412 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.412 INFO:teuthology.orchestra.run.smithi090.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-04-02T14:50:52.412 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-04-02T14:50:52.412 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.412 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.412 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.412 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:52.412 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.412 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.412 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.413 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.413 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.413 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.413 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.413 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.413 INFO:teuthology.orchestra.run.smithi090.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-04-02T14:50:52.413 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-04-02T14:50:52.413 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.413 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.413 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.413 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:52.413 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.413 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.413 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.413 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.413 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.414 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.414 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.414 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.414 INFO:teuthology.orchestra.run.smithi090.stdout: "PWD_POLICY_ENABLED": { 2024-04-02T14:50:52.414 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "PWD_POLICY_ENABLED", 2024-04-02T14:50:52.414 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.414 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.414 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.414 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-02T14:50:52.414 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.414 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.414 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.414 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.414 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.414 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.414 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.415 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.415 INFO:teuthology.orchestra.run.smithi090.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-04-02T14:50:52.415 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-04-02T14:50:52.415 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.415 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.415 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.415 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-04-02T14:50:52.415 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.415 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.415 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.415 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.415 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.415 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.415 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.415 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.415 INFO:teuthology.orchestra.run.smithi090.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-04-02T14:50:52.416 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-04-02T14:50:52.416 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-02T14:50:52.416 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.416 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.416 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "10", 2024-04-02T14:50:52.416 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.416 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.416 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.416 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.416 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.416 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.416 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.416 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.416 INFO:teuthology.orchestra.run.smithi090.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-04-02T14:50:52.416 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-04-02T14:50:52.416 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-02T14:50:52.417 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.417 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.417 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "8", 2024-04-02T14:50:52.417 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.417 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.417 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.417 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.417 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.417 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.417 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.417 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.417 INFO:teuthology.orchestra.run.smithi090.stdout: "REST_REQUESTS_TIMEOUT": { 2024-04-02T14:50:52.417 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-04-02T14:50:52.417 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-02T14:50:52.418 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.418 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.418 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "45", 2024-04-02T14:50:52.418 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.418 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.418 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.418 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.418 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.418 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.418 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.418 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.418 INFO:teuthology.orchestra.run.smithi090.stdout: "RGW_API_ACCESS_KEY": { 2024-04-02T14:50:52.418 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "RGW_API_ACCESS_KEY", 2024-04-02T14:50:52.418 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.418 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.419 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.419 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:52.419 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.419 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.419 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.419 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.419 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.419 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.419 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.419 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.419 INFO:teuthology.orchestra.run.smithi090.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-04-02T14:50:52.419 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-04-02T14:50:52.419 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.419 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.419 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.419 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "admin", 2024-04-02T14:50:52.420 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.420 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.420 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.420 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.420 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.420 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.420 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.420 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.420 INFO:teuthology.orchestra.run.smithi090.stdout: "RGW_API_SECRET_KEY": { 2024-04-02T14:50:52.420 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "RGW_API_SECRET_KEY", 2024-04-02T14:50:52.420 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.420 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.420 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.420 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:52.420 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.420 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.420 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.421 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.421 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.421 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.421 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.421 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.421 INFO:teuthology.orchestra.run.smithi090.stdout: "RGW_API_SSL_VERIFY": { 2024-04-02T14:50:52.421 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "RGW_API_SSL_VERIFY", 2024-04-02T14:50:52.421 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.421 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.421 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.421 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-02T14:50:52.421 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.421 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.421 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.421 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.422 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.422 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.422 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.422 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.422 INFO:teuthology.orchestra.run.smithi090.stdout: "UNSAFE_TLS_v1_2": { 2024-04-02T14:50:52.422 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "UNSAFE_TLS_v1_2", 2024-04-02T14:50:52.422 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.422 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.422 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.422 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:52.422 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.422 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.422 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.422 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.423 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.423 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.423 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.423 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.423 INFO:teuthology.orchestra.run.smithi090.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-04-02T14:50:52.423 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-04-02T14:50:52.423 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-02T14:50:52.423 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.423 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.423 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "0", 2024-04-02T14:50:52.423 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.423 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.423 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.423 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.423 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.424 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.424 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.424 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.424 INFO:teuthology.orchestra.run.smithi090.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-04-02T14:50:52.424 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-04-02T14:50:52.424 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-02T14:50:52.424 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.424 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.424 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "10", 2024-04-02T14:50:52.424 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.424 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.424 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.424 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.424 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.424 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.424 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.425 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.425 INFO:teuthology.orchestra.run.smithi090.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-04-02T14:50:52.425 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-04-02T14:50:52.425 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-02T14:50:52.425 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.425 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.425 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "5", 2024-04-02T14:50:52.425 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.425 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.425 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.425 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.425 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.425 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.425 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.425 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.426 INFO:teuthology.orchestra.run.smithi090.stdout: "cross_origin_url": { 2024-04-02T14:50:52.426 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "cross_origin_url", 2024-04-02T14:50:52.426 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.426 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.426 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.426 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:52.426 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.426 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.426 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.426 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.426 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.426 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.426 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.426 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.427 INFO:teuthology.orchestra.run.smithi090.stdout: "crt_file": { 2024-04-02T14:50:52.427 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "crt_file", 2024-04-02T14:50:52.427 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.427 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.427 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.427 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:52.427 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.427 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.427 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.427 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.427 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.427 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.427 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.427 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.427 INFO:teuthology.orchestra.run.smithi090.stdout: "debug": { 2024-04-02T14:50:52.427 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "debug", 2024-04-02T14:50:52.427 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.428 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.428 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.428 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:52.428 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.428 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.428 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.428 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Enable/disable debug options", 2024-04-02T14:50:52.428 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.428 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.428 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.428 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.428 INFO:teuthology.orchestra.run.smithi090.stdout: "jwt_token_ttl": { 2024-04-02T14:50:52.428 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "jwt_token_ttl", 2024-04-02T14:50:52.428 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-02T14:50:52.428 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.428 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.429 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "28800", 2024-04-02T14:50:52.429 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.429 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.429 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.429 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.429 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.429 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.429 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.429 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.429 INFO:teuthology.orchestra.run.smithi090.stdout: "key_file": { 2024-04-02T14:50:52.429 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "key_file", 2024-04-02T14:50:52.429 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.429 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.429 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.429 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:52.429 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.429 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.430 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.430 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.430 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.430 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.430 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.430 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.430 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-02T14:50:52.430 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-02T14:50:52.430 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.430 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.430 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.430 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:52.430 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.430 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.430 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:52.431 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:52.431 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:52.431 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:52.431 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:52.431 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:52.431 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:52.431 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:52.431 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.431 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.431 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.431 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.431 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.431 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-02T14:50:52.431 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-02T14:50:52.431 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.431 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.431 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.432 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:52.432 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.432 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.432 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.432 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.432 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.432 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.432 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.432 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.432 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-02T14:50:52.432 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-02T14:50:52.432 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.432 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.432 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.432 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-02T14:50:52.433 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.433 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.433 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:52.433 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:52.433 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:52.433 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:52.433 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:52.433 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:52.433 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:52.433 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:52.433 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.433 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.433 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.433 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.433 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.433 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-02T14:50:52.433 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-02T14:50:52.434 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.434 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.434 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.434 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:52.434 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.434 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.434 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.434 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.434 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.434 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.434 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.434 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.434 INFO:teuthology.orchestra.run.smithi090.stdout: "motd": { 2024-04-02T14:50:52.434 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "motd", 2024-04-02T14:50:52.434 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.435 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.435 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.435 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:52.435 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.435 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.435 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.435 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "The message of the day", 2024-04-02T14:50:52.435 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.435 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.435 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.435 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.435 INFO:teuthology.orchestra.run.smithi090.stdout: "redirect_resolve_ip_addr": { 2024-04-02T14:50:52.435 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "redirect_resolve_ip_addr", 2024-04-02T14:50:52.435 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.435 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.435 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.435 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:52.436 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.436 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.436 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.436 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.436 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.436 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.436 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.436 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.436 INFO:teuthology.orchestra.run.smithi090.stdout: "server_addr": { 2024-04-02T14:50:52.436 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "server_addr", 2024-04-02T14:50:52.436 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.436 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.436 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.436 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "::", 2024-04-02T14:50:52.436 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.436 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.437 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.437 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.437 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.437 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.437 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.437 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.437 INFO:teuthology.orchestra.run.smithi090.stdout: "server_port": { 2024-04-02T14:50:52.437 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "server_port", 2024-04-02T14:50:52.437 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-02T14:50:52.437 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.437 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.437 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "8080", 2024-04-02T14:50:52.437 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.437 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.437 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.437 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.438 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.438 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.438 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.438 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.438 INFO:teuthology.orchestra.run.smithi090.stdout: "ssl": { 2024-04-02T14:50:52.438 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "ssl", 2024-04-02T14:50:52.438 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.438 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.438 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.438 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-02T14:50:52.438 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.438 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.438 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.438 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.439 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.439 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.439 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.439 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.439 INFO:teuthology.orchestra.run.smithi090.stdout: "ssl_server_port": { 2024-04-02T14:50:52.439 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "ssl_server_port", 2024-04-02T14:50:52.439 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-02T14:50:52.439 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.439 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.439 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "8443", 2024-04-02T14:50:52.439 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.439 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.439 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.439 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.439 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.440 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.440 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.440 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.440 INFO:teuthology.orchestra.run.smithi090.stdout: "standby_behaviour": { 2024-04-02T14:50:52.440 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "standby_behaviour", 2024-04-02T14:50:52.440 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.440 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.440 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.440 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "redirect", 2024-04-02T14:50:52.440 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.440 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.440 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:52.440 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:52.440 INFO:teuthology.orchestra.run.smithi090.stdout: "redirect" 2024-04-02T14:50:52.440 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:52.440 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.440 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.441 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.441 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.441 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.441 INFO:teuthology.orchestra.run.smithi090.stdout: "standby_error_status_code": { 2024-04-02T14:50:52.441 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "standby_error_status_code", 2024-04-02T14:50:52.441 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-02T14:50:52.441 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.441 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.441 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "500", 2024-04-02T14:50:52.441 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "400", 2024-04-02T14:50:52.441 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "599", 2024-04-02T14:50:52.441 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.441 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.441 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.441 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.441 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.442 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.442 INFO:teuthology.orchestra.run.smithi090.stdout: "url_prefix": { 2024-04-02T14:50:52.442 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "url_prefix", 2024-04-02T14:50:52.442 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.442 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.442 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.442 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:52.442 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.442 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.442 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.442 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.442 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.442 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.442 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.443 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:52.443 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:52.443 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.443 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-02T14:50:52.443 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "devicehealth", 2024-04-02T14:50:52.443 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-02T14:50:52.443 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-02T14:50:52.443 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-02T14:50:52.443 INFO:teuthology.orchestra.run.smithi090.stdout: "enable_monitoring": { 2024-04-02T14:50:52.443 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "enable_monitoring", 2024-04-02T14:50:52.443 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.443 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.443 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.443 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-02T14:50:52.443 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.444 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.444 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.444 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "monitor device health metrics", 2024-04-02T14:50:52.444 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.444 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.444 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.444 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.444 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-02T14:50:52.444 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-02T14:50:52.444 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.444 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.444 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.444 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:52.444 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.444 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.445 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:52.445 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:52.445 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:52.445 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:52.445 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:52.445 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:52.445 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:52.445 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:52.445 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.445 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.445 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.445 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.445 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.445 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-02T14:50:52.445 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-02T14:50:52.445 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.446 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.446 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.446 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:52.446 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.446 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.446 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.446 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.446 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.446 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.446 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.446 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.446 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-02T14:50:52.446 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-02T14:50:52.446 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.446 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.447 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.447 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-02T14:50:52.447 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.447 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.447 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:52.447 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:52.447 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:52.447 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:52.447 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:52.447 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:52.447 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:52.447 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:52.447 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.447 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.447 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.447 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.448 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.448 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-02T14:50:52.448 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-02T14:50:52.448 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.448 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.448 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.448 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:52.448 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.448 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.448 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.448 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.448 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.448 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.448 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.448 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.448 INFO:teuthology.orchestra.run.smithi090.stdout: "mark_out_threshold": { 2024-04-02T14:50:52.449 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "mark_out_threshold", 2024-04-02T14:50:52.449 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "secs", 2024-04-02T14:50:52.449 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.449 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.449 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "2419200", 2024-04-02T14:50:52.449 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.449 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.449 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.449 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-04-02T14:50:52.449 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.449 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.449 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.449 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.449 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_name": { 2024-04-02T14:50:52.449 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "pool_name", 2024-04-02T14:50:52.449 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.449 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.450 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.450 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "device_health_metrics", 2024-04-02T14:50:52.450 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.450 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.450 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.450 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "name of pool in which to store device health metrics", 2024-04-02T14:50:52.450 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.450 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.450 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.450 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.450 INFO:teuthology.orchestra.run.smithi090.stdout: "retention_period": { 2024-04-02T14:50:52.450 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "retention_period", 2024-04-02T14:50:52.450 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "secs", 2024-04-02T14:50:52.450 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.450 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.451 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "15552000", 2024-04-02T14:50:52.451 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.451 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.451 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.451 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "how long to retain device health metrics", 2024-04-02T14:50:52.451 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.451 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.451 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.451 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.451 INFO:teuthology.orchestra.run.smithi090.stdout: "scrape_frequency": { 2024-04-02T14:50:52.451 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "scrape_frequency", 2024-04-02T14:50:52.451 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "secs", 2024-04-02T14:50:52.452 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.452 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.452 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "86400", 2024-04-02T14:50:52.452 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.452 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.452 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.452 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "how frequently to scrape device health metrics", 2024-04-02T14:50:52.452 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.452 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.452 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.452 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.452 INFO:teuthology.orchestra.run.smithi090.stdout: "self_heal": { 2024-04-02T14:50:52.452 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "self_heal", 2024-04-02T14:50:52.452 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.452 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.452 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.453 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-02T14:50:52.453 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.453 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.453 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.453 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-04-02T14:50:52.453 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.453 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.453 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.453 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.453 INFO:teuthology.orchestra.run.smithi090.stdout: "sleep_interval": { 2024-04-02T14:50:52.453 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "sleep_interval", 2024-04-02T14:50:52.453 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "secs", 2024-04-02T14:50:52.453 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.453 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.453 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "600", 2024-04-02T14:50:52.453 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.454 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.454 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.454 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "how frequently to wake up and check device health", 2024-04-02T14:50:52.454 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.454 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.454 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.454 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.454 INFO:teuthology.orchestra.run.smithi090.stdout: "warn_threshold": { 2024-04-02T14:50:52.454 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "warn_threshold", 2024-04-02T14:50:52.454 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "secs", 2024-04-02T14:50:52.454 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.454 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.454 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "7257600", 2024-04-02T14:50:52.454 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.455 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.455 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.455 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-04-02T14:50:52.455 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.455 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.455 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.455 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:52.455 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:52.455 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.455 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-02T14:50:52.455 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "diskprediction_local", 2024-04-02T14:50:52.455 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-02T14:50:52.455 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-02T14:50:52.455 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-02T14:50:52.455 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-02T14:50:52.455 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-02T14:50:52.455 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.456 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.456 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.456 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:52.456 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.456 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.456 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:52.456 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:52.456 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:52.456 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:52.456 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:52.456 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:52.456 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:52.456 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:52.456 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.456 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.456 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.457 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.457 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.457 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-02T14:50:52.457 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-02T14:50:52.457 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.457 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.457 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.457 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:52.457 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.457 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.457 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.457 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.457 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.457 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.457 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.457 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.457 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-02T14:50:52.458 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-02T14:50:52.458 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.458 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.458 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.458 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-02T14:50:52.458 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.458 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.458 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:52.458 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:52.458 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:52.458 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:52.458 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:52.458 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:52.458 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:52.458 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:52.459 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.459 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.459 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.459 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.459 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.459 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-02T14:50:52.459 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-02T14:50:52.459 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.459 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.459 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.459 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:52.459 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.459 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.459 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.459 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.460 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.460 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.460 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.460 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.460 INFO:teuthology.orchestra.run.smithi090.stdout: "predict_interval": { 2024-04-02T14:50:52.460 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "predict_interval", 2024-04-02T14:50:52.460 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.460 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.460 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.460 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "86400", 2024-04-02T14:50:52.460 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.460 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.460 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.460 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.460 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.460 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.461 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.461 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.461 INFO:teuthology.orchestra.run.smithi090.stdout: "predictor_model": { 2024-04-02T14:50:52.461 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "predictor_model", 2024-04-02T14:50:52.461 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.461 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.461 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.461 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "prophetstor", 2024-04-02T14:50:52.461 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.461 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.461 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.461 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.461 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.461 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.461 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.461 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.462 INFO:teuthology.orchestra.run.smithi090.stdout: "sleep_interval": { 2024-04-02T14:50:52.462 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "sleep_interval", 2024-04-02T14:50:52.462 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.462 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.462 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.462 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "600", 2024-04-02T14:50:52.462 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.462 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.462 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.462 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.462 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.462 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.462 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.462 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:52.463 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:52.463 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.463 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-02T14:50:52.463 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "influx", 2024-04-02T14:50:52.463 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": false, 2024-04-02T14:50:52.463 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "influxdb python module not found", 2024-04-02T14:50:52.463 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-02T14:50:52.463 INFO:teuthology.orchestra.run.smithi090.stdout: "batch_size": { 2024-04-02T14:50:52.463 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "batch_size", 2024-04-02T14:50:52.463 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-02T14:50:52.463 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.463 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.463 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "5000", 2024-04-02T14:50:52.463 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.463 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.463 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.463 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-04-02T14:50:52.464 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.464 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.464 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.464 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.464 INFO:teuthology.orchestra.run.smithi090.stdout: "database": { 2024-04-02T14:50:52.464 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "database", 2024-04-02T14:50:52.464 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.464 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.464 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.464 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "ceph", 2024-04-02T14:50:52.464 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.464 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.464 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.464 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-04-02T14:50:52.464 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.465 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.465 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.465 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.465 INFO:teuthology.orchestra.run.smithi090.stdout: "hostname": { 2024-04-02T14:50:52.465 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "hostname", 2024-04-02T14:50:52.465 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.465 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.465 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.465 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:52.465 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.465 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.465 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.465 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "InfluxDB server hostname", 2024-04-02T14:50:52.465 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.465 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.465 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.466 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.466 INFO:teuthology.orchestra.run.smithi090.stdout: "interval": { 2024-04-02T14:50:52.466 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "interval", 2024-04-02T14:50:52.466 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "secs", 2024-04-02T14:50:52.466 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.466 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.466 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "30", 2024-04-02T14:50:52.466 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "5", 2024-04-02T14:50:52.466 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.466 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.466 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-04-02T14:50:52.466 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.466 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.466 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.466 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.467 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-02T14:50:52.467 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-02T14:50:52.467 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.467 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.467 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.467 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:52.467 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.467 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.467 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:52.467 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:52.467 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:52.467 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:52.467 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:52.467 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:52.467 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:52.467 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:52.467 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.468 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.468 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.468 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.468 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.468 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-02T14:50:52.468 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-02T14:50:52.468 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.468 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.468 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.468 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:52.468 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.468 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.468 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.468 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.468 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.468 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.469 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.469 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.469 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-02T14:50:52.469 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-02T14:50:52.469 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.469 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.469 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.469 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-02T14:50:52.469 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.469 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.469 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:52.469 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:52.469 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:52.469 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:52.469 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:52.469 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:52.469 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:52.470 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:52.470 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.470 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.470 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.470 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.470 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.470 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-02T14:50:52.470 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-02T14:50:52.470 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.470 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.470 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.470 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:52.470 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.470 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.470 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.471 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.471 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.471 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.471 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.471 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.471 INFO:teuthology.orchestra.run.smithi090.stdout: "password": { 2024-04-02T14:50:52.471 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "password", 2024-04-02T14:50:52.471 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.471 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.471 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.471 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:52.471 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.471 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.471 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.471 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "password of InfluxDB server user", 2024-04-02T14:50:52.471 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.471 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.472 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.472 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.472 INFO:teuthology.orchestra.run.smithi090.stdout: "port": { 2024-04-02T14:50:52.472 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "port", 2024-04-02T14:50:52.472 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-02T14:50:52.472 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.472 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.472 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "8086", 2024-04-02T14:50:52.472 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.472 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.472 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.472 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "InfluxDB server port", 2024-04-02T14:50:52.472 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.472 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.472 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.473 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.473 INFO:teuthology.orchestra.run.smithi090.stdout: "ssl": { 2024-04-02T14:50:52.473 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "ssl", 2024-04-02T14:50:52.473 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.473 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.473 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.473 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "false", 2024-04-02T14:50:52.473 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.473 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.473 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.473 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-04-02T14:50:52.473 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.473 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.473 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.473 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.473 INFO:teuthology.orchestra.run.smithi090.stdout: "threads": { 2024-04-02T14:50:52.474 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "threads", 2024-04-02T14:50:52.474 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-02T14:50:52.474 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.474 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.474 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "5", 2024-04-02T14:50:52.474 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "1", 2024-04-02T14:50:52.474 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "32", 2024-04-02T14:50:52.474 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.474 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-04-02T14:50:52.474 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.474 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.474 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.474 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.474 INFO:teuthology.orchestra.run.smithi090.stdout: "username": { 2024-04-02T14:50:52.475 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "username", 2024-04-02T14:50:52.475 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.475 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.475 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.475 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:52.475 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.475 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.475 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.475 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "username of InfluxDB server user", 2024-04-02T14:50:52.475 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.475 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.475 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.475 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.475 INFO:teuthology.orchestra.run.smithi090.stdout: "verify_ssl": { 2024-04-02T14:50:52.475 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "verify_ssl", 2024-04-02T14:50:52.476 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.476 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.476 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.476 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "true", 2024-04-02T14:50:52.476 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.476 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.476 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.476 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-04-02T14:50:52.476 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.476 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.476 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.476 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:52.476 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:52.476 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.476 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-02T14:50:52.476 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "insights", 2024-04-02T14:50:52.477 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-02T14:50:52.477 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-02T14:50:52.477 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-02T14:50:52.477 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-02T14:50:52.477 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-02T14:50:52.477 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.477 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.477 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.477 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:52.477 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.477 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.477 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:52.477 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:52.477 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:52.477 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:52.477 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:52.478 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:52.478 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:52.478 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:52.478 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.478 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.478 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.478 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.478 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.478 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-02T14:50:52.478 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-02T14:50:52.478 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.478 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.478 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.478 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:52.479 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.479 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.479 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.479 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.479 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.479 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.479 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.479 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.479 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-02T14:50:52.479 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-02T14:50:52.479 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.479 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.479 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.479 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-02T14:50:52.480 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.480 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.480 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:52.480 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:52.480 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:52.480 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:52.480 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:52.480 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:52.480 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:52.480 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:52.480 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.480 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.480 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.480 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.480 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.481 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-02T14:50:52.481 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-02T14:50:52.481 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.481 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.481 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.481 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:52.481 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.481 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.481 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.481 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.481 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.481 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.481 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.481 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:52.481 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:52.481 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.482 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-02T14:50:52.482 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "iostat", 2024-04-02T14:50:52.482 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-02T14:50:52.482 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-02T14:50:52.482 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-02T14:50:52.482 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-02T14:50:52.482 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-02T14:50:52.482 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.482 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.482 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.482 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:52.482 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.482 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.482 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:52.483 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:52.483 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:52.483 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:52.483 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:52.483 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:52.483 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:52.483 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:52.483 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.483 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.483 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.483 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.483 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.483 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-02T14:50:52.483 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-02T14:50:52.483 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.484 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.484 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.484 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:52.484 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.484 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.484 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.484 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.484 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.484 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.484 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.484 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.484 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-02T14:50:52.484 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-02T14:50:52.484 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.485 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.485 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.485 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-02T14:50:52.485 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.485 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.485 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:52.485 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:52.485 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:52.485 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:52.485 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:52.485 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:52.485 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:52.485 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:52.486 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.486 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.486 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.486 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.486 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.486 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-02T14:50:52.486 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-02T14:50:52.486 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.487 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.487 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.487 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:52.487 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.487 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.487 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.487 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.487 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.487 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.487 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.487 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:52.488 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:52.488 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.488 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-02T14:50:52.488 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "k8sevents", 2024-04-02T14:50:52.488 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-02T14:50:52.488 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-02T14:50:52.488 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-02T14:50:52.488 INFO:teuthology.orchestra.run.smithi090.stdout: "ceph_event_retention_days": { 2024-04-02T14:50:52.488 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "ceph_event_retention_days", 2024-04-02T14:50:52.488 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-02T14:50:52.488 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.488 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.488 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "7", 2024-04-02T14:50:52.488 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.489 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.489 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.489 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Days to hold ceph event information within local cache", 2024-04-02T14:50:52.489 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.489 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.489 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.489 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.489 INFO:teuthology.orchestra.run.smithi090.stdout: "config_check_secs": { 2024-04-02T14:50:52.489 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "config_check_secs", 2024-04-02T14:50:52.489 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-02T14:50:52.489 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.489 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.489 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "10", 2024-04-02T14:50:52.489 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "10", 2024-04-02T14:50:52.489 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.490 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.490 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-04-02T14:50:52.490 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.490 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.490 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.490 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.490 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-02T14:50:52.490 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-02T14:50:52.490 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.490 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.490 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.490 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:52.490 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.490 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.490 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:52.491 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:52.491 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:52.491 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:52.491 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:52.491 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:52.491 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:52.491 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:52.491 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.491 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.491 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.491 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.491 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.491 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-02T14:50:52.491 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-02T14:50:52.491 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.492 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.492 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.492 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:52.492 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.492 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.492 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.492 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.492 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.492 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.492 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.492 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.492 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-02T14:50:52.492 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-02T14:50:52.492 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.492 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.493 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.493 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-02T14:50:52.493 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.493 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.493 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:52.493 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:52.493 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:52.493 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:52.493 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:52.493 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:52.493 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:52.493 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:52.493 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.493 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.493 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.494 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.494 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.494 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-02T14:50:52.494 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-02T14:50:52.494 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.494 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.494 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.494 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:52.494 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.494 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.494 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.494 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.494 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.494 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.494 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.495 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:52.495 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:52.495 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.495 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-02T14:50:52.495 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "localpool", 2024-04-02T14:50:52.495 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-02T14:50:52.495 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-02T14:50:52.495 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-02T14:50:52.495 INFO:teuthology.orchestra.run.smithi090.stdout: "failure_domain": { 2024-04-02T14:50:52.495 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "failure_domain", 2024-04-02T14:50:52.495 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.495 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.495 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.495 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "host", 2024-04-02T14:50:52.495 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.496 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.496 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.496 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "failure domain for any created local pool", 2024-04-02T14:50:52.496 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-04-02T14:50:52.496 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.496 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.496 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.496 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-02T14:50:52.496 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-02T14:50:52.496 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.496 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.496 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.496 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:52.496 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.496 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.496 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:52.497 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:52.497 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:52.497 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:52.497 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:52.497 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:52.497 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:52.497 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:52.497 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.497 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.497 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.497 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.497 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.497 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-02T14:50:52.497 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-02T14:50:52.497 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.498 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.498 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.498 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:52.498 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.498 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.498 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.498 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.498 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.498 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.498 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.498 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.498 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-02T14:50:52.498 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-02T14:50:52.498 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.499 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.499 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.499 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-02T14:50:52.499 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.499 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.499 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:52.499 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:52.499 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:52.499 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:52.499 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:52.499 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:52.499 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:52.499 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:52.499 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.499 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.500 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.500 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.500 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.500 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-02T14:50:52.500 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-02T14:50:52.500 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.500 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.500 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.500 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:52.500 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.500 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.500 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.500 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.500 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.500 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.501 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.501 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.501 INFO:teuthology.orchestra.run.smithi090.stdout: "min_size": { 2024-04-02T14:50:52.501 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "min_size", 2024-04-02T14:50:52.501 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-02T14:50:52.501 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.501 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.501 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:52.501 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.501 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.501 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.501 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "default min_size for any created local pool", 2024-04-02T14:50:52.501 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-04-02T14:50:52.501 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.501 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.502 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.502 INFO:teuthology.orchestra.run.smithi090.stdout: "num_rep": { 2024-04-02T14:50:52.502 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "num_rep", 2024-04-02T14:50:52.502 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-02T14:50:52.502 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.502 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.502 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "3", 2024-04-02T14:50:52.502 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.502 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.502 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.502 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "default replica count for any created local pool", 2024-04-02T14:50:52.502 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.502 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.502 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.503 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.503 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num": { 2024-04-02T14:50:52.503 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "pg_num", 2024-04-02T14:50:52.503 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-02T14:50:52.503 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.503 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.503 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "128", 2024-04-02T14:50:52.503 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.503 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.503 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.503 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "default pg_num for any created local pool", 2024-04-02T14:50:52.503 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.503 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.503 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.503 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.504 INFO:teuthology.orchestra.run.smithi090.stdout: "prefix": { 2024-04-02T14:50:52.504 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "prefix", 2024-04-02T14:50:52.504 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.504 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.504 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.504 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:52.504 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.504 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.504 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.504 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "name prefix for any created local pool", 2024-04-02T14:50:52.504 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.504 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.504 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.504 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.504 INFO:teuthology.orchestra.run.smithi090.stdout: "subtree": { 2024-04-02T14:50:52.504 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "subtree", 2024-04-02T14:50:52.505 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.505 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.505 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.505 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "rack", 2024-04-02T14:50:52.505 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.505 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.505 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.505 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "CRUSH level for which to create a local pool", 2024-04-02T14:50:52.505 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-04-02T14:50:52.505 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.505 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.505 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:52.505 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:52.505 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.505 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-02T14:50:52.506 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "mds_autoscaler", 2024-04-02T14:50:52.506 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-02T14:50:52.506 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-02T14:50:52.506 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-02T14:50:52.506 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-02T14:50:52.506 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-02T14:50:52.506 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.506 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.506 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.506 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:52.506 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.506 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.506 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:52.506 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:52.507 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:52.507 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:52.507 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:52.507 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:52.507 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:52.507 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:52.507 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.507 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.507 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.507 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.507 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.507 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-02T14:50:52.507 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-02T14:50:52.507 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.507 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.507 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.508 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:52.508 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.508 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.508 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.508 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.508 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.508 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.508 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.508 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.508 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-02T14:50:52.508 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-02T14:50:52.508 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.508 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.508 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.508 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-02T14:50:52.509 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.509 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.509 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:52.509 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:52.509 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:52.509 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:52.509 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:52.509 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:52.509 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:52.509 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:52.509 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.509 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.509 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.509 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.509 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.510 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-02T14:50:52.510 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-02T14:50:52.510 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.510 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.510 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.510 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:52.510 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.510 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.510 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.510 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.510 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.510 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.510 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.510 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:52.510 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:52.511 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.511 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-02T14:50:52.511 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "mirroring", 2024-04-02T14:50:52.511 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-02T14:50:52.511 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-02T14:50:52.511 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-02T14:50:52.511 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-02T14:50:52.511 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-02T14:50:52.511 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.511 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.511 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.511 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:52.511 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.511 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.511 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:52.511 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:52.512 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:52.512 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:52.512 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:52.512 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:52.512 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:52.512 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:52.512 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.512 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.512 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.512 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.512 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.512 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-02T14:50:52.512 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-02T14:50:52.512 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.512 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.513 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.513 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:52.513 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.513 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.513 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.513 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.513 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.513 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.513 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.513 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.513 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-02T14:50:52.513 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-02T14:50:52.513 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.513 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.513 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.514 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-02T14:50:52.514 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.514 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.514 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:52.514 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:52.514 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:52.514 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:52.514 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:52.514 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:52.514 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:52.514 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:52.514 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.514 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.514 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.515 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.515 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.515 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-02T14:50:52.515 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-02T14:50:52.515 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.515 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.515 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.515 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:52.515 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.515 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.515 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.515 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.515 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.515 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.515 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.515 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:52.516 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:52.516 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.516 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-02T14:50:52.516 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "nfs", 2024-04-02T14:50:52.516 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-02T14:50:52.516 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-02T14:50:52.516 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-02T14:50:52.516 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-02T14:50:52.516 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-02T14:50:52.516 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.516 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.516 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.516 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:52.516 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.517 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.517 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:52.517 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:52.517 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:52.517 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:52.517 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:52.517 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:52.517 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:52.517 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:52.517 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.517 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.517 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.517 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.517 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.517 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-02T14:50:52.517 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-02T14:50:52.518 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.518 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.518 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.518 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:52.518 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.518 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.518 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.518 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.518 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.518 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.518 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.518 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.518 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-02T14:50:52.518 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-02T14:50:52.519 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.519 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.519 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.519 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-02T14:50:52.519 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.519 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.519 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:52.519 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:52.519 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:52.519 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:52.519 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:52.519 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:52.519 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:52.519 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:52.520 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.520 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.520 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.520 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.520 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.520 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-02T14:50:52.520 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-02T14:50:52.520 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.520 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.520 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.520 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:52.520 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.520 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.520 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.520 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.521 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.521 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.521 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.521 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:52.521 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:52.521 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.521 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-02T14:50:52.521 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "orchestrator", 2024-04-02T14:50:52.521 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-02T14:50:52.521 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-02T14:50:52.521 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-02T14:50:52.521 INFO:teuthology.orchestra.run.smithi090.stdout: "fail_fs": { 2024-04-02T14:50:52.521 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "fail_fs", 2024-04-02T14:50:52.521 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.522 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.522 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.522 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:52.522 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.522 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.522 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.522 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-04-02T14:50:52.522 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.522 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.522 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.522 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.522 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-02T14:50:52.522 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-02T14:50:52.522 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.523 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.523 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.523 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:52.523 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.523 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.523 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:52.523 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:52.523 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:52.523 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:52.523 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:52.523 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:52.523 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:52.523 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:52.523 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.523 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.524 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.524 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.524 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.524 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-02T14:50:52.524 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-02T14:50:52.524 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.524 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.524 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.524 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:52.524 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.524 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.524 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.524 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.524 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.524 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.525 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.525 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.525 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-02T14:50:52.525 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-02T14:50:52.525 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.525 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.525 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.525 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-02T14:50:52.525 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.525 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.525 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:52.525 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:52.525 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:52.525 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:52.526 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:52.526 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:52.526 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:52.526 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:52.526 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.526 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.526 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.526 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.526 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.526 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-02T14:50:52.526 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-02T14:50:52.526 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.526 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.526 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.527 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:52.527 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.527 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.527 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.527 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.527 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.527 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.527 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.527 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.527 INFO:teuthology.orchestra.run.smithi090.stdout: "orchestrator": { 2024-04-02T14:50:52.527 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "orchestrator", 2024-04-02T14:50:52.527 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.527 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.527 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.527 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:52.528 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.528 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.528 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:52.528 INFO:teuthology.orchestra.run.smithi090.stdout: "cephadm", 2024-04-02T14:50:52.528 INFO:teuthology.orchestra.run.smithi090.stdout: "rook", 2024-04-02T14:50:52.528 INFO:teuthology.orchestra.run.smithi090.stdout: "test_orchestrator" 2024-04-02T14:50:52.528 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:52.528 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Orchestrator backend", 2024-04-02T14:50:52.528 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.528 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.528 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.528 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:52.528 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:52.528 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.528 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-02T14:50:52.529 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "osd_perf_query", 2024-04-02T14:50:52.529 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-02T14:50:52.529 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-02T14:50:52.529 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-02T14:50:52.529 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-02T14:50:52.529 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-02T14:50:52.529 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.529 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.529 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.529 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:52.529 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.529 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.529 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:52.529 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:52.530 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:52.530 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:52.530 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:52.530 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:52.530 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:52.530 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:52.530 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.530 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.530 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.530 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.530 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.530 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-02T14:50:52.530 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-02T14:50:52.530 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.531 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.531 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.531 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:52.531 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.531 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.531 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.531 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.531 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.531 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.531 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.531 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.531 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-02T14:50:52.531 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-02T14:50:52.531 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.531 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.532 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.532 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-02T14:50:52.532 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.532 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.532 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:52.532 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:52.532 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:52.532 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:52.532 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:52.532 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:52.532 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:52.532 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:52.532 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.532 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.533 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.533 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.533 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.533 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-02T14:50:52.533 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-02T14:50:52.533 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.533 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.533 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.533 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:52.533 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.533 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.533 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.533 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.533 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.533 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.533 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.534 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:52.534 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:52.534 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.534 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-02T14:50:52.534 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "osd_support", 2024-04-02T14:50:52.534 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-02T14:50:52.534 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-02T14:50:52.534 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-02T14:50:52.534 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-02T14:50:52.534 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-02T14:50:52.534 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.534 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.534 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.534 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:52.535 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.535 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.535 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:52.535 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:52.535 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:52.535 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:52.535 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:52.535 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:52.535 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:52.535 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:52.535 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.535 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.535 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.535 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.535 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.536 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-02T14:50:52.536 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-02T14:50:52.536 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.536 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.536 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.536 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:52.536 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.536 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.536 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.536 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.536 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.536 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.536 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.536 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.536 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-02T14:50:52.536 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-02T14:50:52.537 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.537 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.537 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.537 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-02T14:50:52.537 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.537 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.537 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:52.537 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:52.537 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:52.537 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:52.537 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:52.537 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:52.537 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:52.537 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:52.537 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.537 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.538 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.538 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.538 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.538 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-02T14:50:52.538 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-02T14:50:52.538 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.538 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.538 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.538 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:52.538 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.538 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.538 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.538 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.538 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.538 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.539 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.539 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:52.539 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:52.539 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.539 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-02T14:50:52.539 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "pg_autoscaler", 2024-04-02T14:50:52.539 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-02T14:50:52.539 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-02T14:50:52.539 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-02T14:50:52.539 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-02T14:50:52.539 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-02T14:50:52.539 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.539 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.539 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.539 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:52.540 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.540 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.540 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:52.540 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:52.540 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:52.540 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:52.540 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:52.540 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:52.540 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:52.540 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:52.540 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.540 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.540 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.540 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.540 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.540 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-02T14:50:52.541 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-02T14:50:52.541 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.541 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.541 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.541 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:52.541 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.541 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.541 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.541 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.541 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.541 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.541 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.541 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.541 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-02T14:50:52.541 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-02T14:50:52.541 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.542 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.542 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.542 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-02T14:50:52.542 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.542 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.542 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:52.542 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:52.542 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:52.542 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:52.542 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:52.542 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:52.542 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:52.542 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:52.542 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.542 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.543 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.543 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.543 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.543 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-02T14:50:52.543 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-02T14:50:52.543 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.543 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.543 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.543 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:52.543 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.543 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.543 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.543 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.543 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.543 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.544 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.544 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.544 INFO:teuthology.orchestra.run.smithi090.stdout: "sleep_interval": { 2024-04-02T14:50:52.544 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "sleep_interval", 2024-04-02T14:50:52.544 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "secs", 2024-04-02T14:50:52.544 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.544 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.544 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "60", 2024-04-02T14:50:52.544 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.544 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.544 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.544 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.544 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.544 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.544 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.545 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.545 INFO:teuthology.orchestra.run.smithi090.stdout: "threshold": { 2024-04-02T14:50:52.545 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "threshold", 2024-04-02T14:50:52.545 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "float", 2024-04-02T14:50:52.545 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.545 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.545 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "3.0", 2024-04-02T14:50:52.545 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "1.0", 2024-04-02T14:50:52.545 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.545 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.545 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "scaling threshold", 2024-04-02T14:50:52.545 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-04-02T14:50:52.545 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.545 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.546 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:52.546 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:52.546 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.546 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-02T14:50:52.546 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "progress", 2024-04-02T14:50:52.546 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-02T14:50:52.546 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-02T14:50:52.546 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-02T14:50:52.546 INFO:teuthology.orchestra.run.smithi090.stdout: "allow_pg_recovery_event": { 2024-04-02T14:50:52.546 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "allow_pg_recovery_event", 2024-04-02T14:50:52.546 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.546 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.546 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.546 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:52.546 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.547 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.547 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.547 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "allow the module to show pg recovery progress", 2024-04-02T14:50:52.547 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.547 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.547 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.547 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.547 INFO:teuthology.orchestra.run.smithi090.stdout: "enabled": { 2024-04-02T14:50:52.547 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "enabled", 2024-04-02T14:50:52.547 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.547 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.547 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.547 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-02T14:50:52.547 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.547 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.548 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.548 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.548 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.548 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.548 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.548 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.548 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-02T14:50:52.548 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-02T14:50:52.548 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.548 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.548 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.548 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:52.548 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.548 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.548 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:52.549 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:52.549 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:52.549 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:52.549 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:52.549 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:52.549 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:52.549 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:52.549 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.549 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.549 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.549 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.549 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.549 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-02T14:50:52.549 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-02T14:50:52.550 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.550 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.550 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.550 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:52.550 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.550 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.550 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.550 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.550 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.550 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.550 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.550 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.550 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-02T14:50:52.550 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-02T14:50:52.551 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.551 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.551 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.551 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-02T14:50:52.551 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.551 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.551 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:52.551 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:52.551 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:52.551 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:52.551 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:52.551 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:52.551 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:52.551 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:52.552 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.552 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.552 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.552 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.552 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.552 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-02T14:50:52.552 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-02T14:50:52.552 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.552 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.552 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.552 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:52.552 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.552 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.552 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.552 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.553 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.553 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.553 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.553 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.553 INFO:teuthology.orchestra.run.smithi090.stdout: "max_completed_events": { 2024-04-02T14:50:52.553 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "max_completed_events", 2024-04-02T14:50:52.553 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-02T14:50:52.553 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.553 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.553 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "50", 2024-04-02T14:50:52.553 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.553 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.553 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.553 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "number of past completed events to remember", 2024-04-02T14:50:52.553 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.554 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.554 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.554 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.554 INFO:teuthology.orchestra.run.smithi090.stdout: "sleep_interval": { 2024-04-02T14:50:52.554 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "sleep_interval", 2024-04-02T14:50:52.554 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "secs", 2024-04-02T14:50:52.554 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.554 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.554 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "5", 2024-04-02T14:50:52.554 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.554 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.554 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.554 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "how long the module is going to sleep", 2024-04-02T14:50:52.554 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.554 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.555 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.555 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:52.555 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:52.555 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.555 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-02T14:50:52.555 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "prometheus", 2024-04-02T14:50:52.555 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-02T14:50:52.555 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-02T14:50:52.555 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-02T14:50:52.555 INFO:teuthology.orchestra.run.smithi090.stdout: "cache": { 2024-04-02T14:50:52.555 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "cache", 2024-04-02T14:50:52.555 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.555 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.555 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.555 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-02T14:50:52.556 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.556 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.556 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.556 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.556 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.556 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.556 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.556 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.556 INFO:teuthology.orchestra.run.smithi090.stdout: "exclude_perf_counters": { 2024-04-02T14:50:52.556 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "exclude_perf_counters", 2024-04-02T14:50:52.556 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.556 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.556 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.556 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-02T14:50:52.556 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.556 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.557 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.557 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-04-02T14:50:52.557 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-04-02T14:50:52.557 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.557 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.557 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.557 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-02T14:50:52.557 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-02T14:50:52.557 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.557 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.557 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.557 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:52.557 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.557 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.558 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:52.558 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:52.558 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:52.558 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:52.558 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:52.558 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:52.558 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:52.558 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:52.558 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.558 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.558 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.558 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.558 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.558 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-02T14:50:52.558 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-02T14:50:52.559 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.559 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.559 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.559 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:52.559 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.559 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.559 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.559 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.559 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.559 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.559 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.559 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.559 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-02T14:50:52.559 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-02T14:50:52.559 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.559 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.560 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.560 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-02T14:50:52.560 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.560 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.560 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:52.560 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:52.560 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:52.560 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:52.560 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:52.560 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:52.560 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:52.560 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:52.560 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.560 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.560 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.561 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.561 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.561 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-02T14:50:52.561 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-02T14:50:52.561 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.561 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.561 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.561 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:52.561 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.561 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.561 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.561 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.561 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.562 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.562 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.562 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.562 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_stats_pools": { 2024-04-02T14:50:52.562 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "rbd_stats_pools", 2024-04-02T14:50:52.562 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.562 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.562 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.562 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:52.562 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.562 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.562 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.562 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.562 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.563 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.563 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.563 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.563 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_stats_pools_refresh_interval": { 2024-04-02T14:50:52.563 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-04-02T14:50:52.563 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-02T14:50:52.563 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.563 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.563 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "300", 2024-04-02T14:50:52.563 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.563 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.563 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.563 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.563 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.563 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.564 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.564 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.564 INFO:teuthology.orchestra.run.smithi090.stdout: "scrape_interval": { 2024-04-02T14:50:52.564 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "scrape_interval", 2024-04-02T14:50:52.564 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "float", 2024-04-02T14:50:52.564 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.564 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.564 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "15.0", 2024-04-02T14:50:52.564 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.564 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.564 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.564 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.564 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.564 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.564 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.565 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.565 INFO:teuthology.orchestra.run.smithi090.stdout: "server_addr": { 2024-04-02T14:50:52.565 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "server_addr", 2024-04-02T14:50:52.565 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.565 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.565 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.565 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "::", 2024-04-02T14:50:52.565 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.565 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.565 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.565 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-04-02T14:50:52.565 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.565 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.565 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.565 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.565 INFO:teuthology.orchestra.run.smithi090.stdout: "server_port": { 2024-04-02T14:50:52.566 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "server_port", 2024-04-02T14:50:52.566 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-02T14:50:52.566 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.566 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.566 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "9283", 2024-04-02T14:50:52.566 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.566 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.566 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.566 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-04-02T14:50:52.566 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.566 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.566 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.566 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.566 INFO:teuthology.orchestra.run.smithi090.stdout: "stale_cache_strategy": { 2024-04-02T14:50:52.566 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "stale_cache_strategy", 2024-04-02T14:50:52.567 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.567 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.567 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.567 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "log", 2024-04-02T14:50:52.567 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.567 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.567 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.567 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.567 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.567 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.567 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.567 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.567 INFO:teuthology.orchestra.run.smithi090.stdout: "standby_behaviour": { 2024-04-02T14:50:52.567 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "standby_behaviour", 2024-04-02T14:50:52.567 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.567 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.568 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.568 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "default", 2024-04-02T14:50:52.568 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.568 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.568 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:52.568 INFO:teuthology.orchestra.run.smithi090.stdout: "default", 2024-04-02T14:50:52.568 INFO:teuthology.orchestra.run.smithi090.stdout: "error" 2024-04-02T14:50:52.568 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:52.568 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.568 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.568 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.568 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.568 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.568 INFO:teuthology.orchestra.run.smithi090.stdout: "standby_error_status_code": { 2024-04-02T14:50:52.568 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "standby_error_status_code", 2024-04-02T14:50:52.568 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-02T14:50:52.569 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.569 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.569 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "500", 2024-04-02T14:50:52.569 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "400", 2024-04-02T14:50:52.569 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "599", 2024-04-02T14:50:52.569 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.569 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.569 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.569 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.569 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.569 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:52.569 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:52.569 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.569 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-02T14:50:52.569 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "rbd_support", 2024-04-02T14:50:52.569 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-02T14:50:52.570 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-02T14:50:52.570 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-02T14:50:52.570 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-02T14:50:52.570 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-02T14:50:52.570 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.570 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.570 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.570 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:52.570 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.570 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.570 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:52.570 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:52.570 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:52.570 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:52.571 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:52.571 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:52.571 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:52.571 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:52.571 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.571 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.571 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.571 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.571 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.571 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-02T14:50:52.571 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-02T14:50:52.571 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.571 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.571 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.571 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:52.571 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.571 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.572 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.572 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.572 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.572 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.572 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.572 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.572 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-02T14:50:52.572 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-02T14:50:52.572 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.572 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.572 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.572 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-02T14:50:52.572 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.572 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.572 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:52.573 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:52.573 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:52.573 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:52.573 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:52.573 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:52.573 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:52.573 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:52.573 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.573 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.573 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.573 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.573 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.573 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-02T14:50:52.573 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-02T14:50:52.573 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.574 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.574 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.574 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:52.574 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.574 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.574 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.574 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.574 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.574 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.574 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.574 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.574 INFO:teuthology.orchestra.run.smithi090.stdout: "max_concurrent_snap_create": { 2024-04-02T14:50:52.574 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "max_concurrent_snap_create", 2024-04-02T14:50:52.574 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-02T14:50:52.574 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.575 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.575 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "10", 2024-04-02T14:50:52.575 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.575 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.575 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.575 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.575 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.575 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.575 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.575 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.575 INFO:teuthology.orchestra.run.smithi090.stdout: "mirror_snapshot_schedule": { 2024-04-02T14:50:52.575 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "mirror_snapshot_schedule", 2024-04-02T14:50:52.575 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.575 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.575 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.575 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:52.576 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.576 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.576 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.576 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.576 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.576 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.576 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.576 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.576 INFO:teuthology.orchestra.run.smithi090.stdout: "trash_purge_schedule": { 2024-04-02T14:50:52.576 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "trash_purge_schedule", 2024-04-02T14:50:52.576 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.576 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.576 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.576 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:52.576 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.577 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.577 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.577 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.577 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.577 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.577 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.577 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:52.577 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:52.577 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.577 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-02T14:50:52.577 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "restful", 2024-04-02T14:50:52.577 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-02T14:50:52.577 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-02T14:50:52.577 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-02T14:50:52.577 INFO:teuthology.orchestra.run.smithi090.stdout: "enable_auth": { 2024-04-02T14:50:52.578 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "enable_auth", 2024-04-02T14:50:52.578 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.578 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.578 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.578 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-02T14:50:52.578 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.578 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.578 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.578 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.578 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.578 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.578 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.578 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.578 INFO:teuthology.orchestra.run.smithi090.stdout: "key_file": { 2024-04-02T14:50:52.578 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "key_file", 2024-04-02T14:50:52.579 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.579 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.579 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.579 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:52.579 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.579 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.579 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.579 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.579 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.579 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.579 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.579 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.579 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-02T14:50:52.579 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-02T14:50:52.579 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.580 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.580 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.580 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:52.580 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.580 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.580 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:52.580 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:52.580 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:52.580 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:52.580 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:52.580 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:52.580 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:52.580 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:52.580 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.580 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.581 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.581 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.581 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.581 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-02T14:50:52.581 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-02T14:50:52.581 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.581 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.581 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.581 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:52.581 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.581 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.581 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.581 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.581 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.581 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.582 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.582 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.582 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-02T14:50:52.582 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-02T14:50:52.582 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.582 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.582 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.582 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-02T14:50:52.582 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.582 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.582 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:52.582 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:52.582 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:52.582 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:52.582 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:52.583 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:52.583 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:52.583 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:52.583 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.583 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.583 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.583 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.583 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.583 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-02T14:50:52.583 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-02T14:50:52.583 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.583 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.583 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.583 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:52.583 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.584 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.584 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.584 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.584 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.584 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.584 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.584 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.584 INFO:teuthology.orchestra.run.smithi090.stdout: "server_addr": { 2024-04-02T14:50:52.584 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "server_addr", 2024-04-02T14:50:52.584 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.584 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.584 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.584 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:52.584 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.584 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.585 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.585 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.585 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.585 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.585 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.585 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.585 INFO:teuthology.orchestra.run.smithi090.stdout: "server_port": { 2024-04-02T14:50:52.585 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "server_port", 2024-04-02T14:50:52.585 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.585 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.585 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.585 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:52.585 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.585 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.585 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.585 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.586 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.586 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.586 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.586 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:52.586 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:52.586 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.586 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-02T14:50:52.586 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "rgw", 2024-04-02T14:50:52.586 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-02T14:50:52.586 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-02T14:50:52.586 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-02T14:50:52.586 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-02T14:50:52.586 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-02T14:50:52.586 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.586 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.587 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.587 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:52.587 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.587 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.587 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:52.587 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:52.587 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:52.587 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:52.587 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:52.587 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:52.587 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:52.587 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:52.587 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.587 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.587 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.587 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.588 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.588 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-02T14:50:52.588 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-02T14:50:52.588 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.588 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.588 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.588 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:52.588 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.588 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.588 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.588 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.588 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.588 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.588 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.588 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.588 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-02T14:50:52.589 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-02T14:50:52.589 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.589 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.589 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.589 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-02T14:50:52.589 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.589 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.589 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:52.589 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:52.589 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:52.589 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:52.589 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:52.589 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:52.589 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:52.589 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:52.589 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.590 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.590 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.590 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.590 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.590 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-02T14:50:52.590 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-02T14:50:52.590 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.590 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.590 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.590 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:52.590 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.590 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.590 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.590 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.590 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.591 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.591 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.591 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:52.591 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:52.591 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.591 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-02T14:50:52.591 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "rook", 2024-04-02T14:50:52.591 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-02T14:50:52.591 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-02T14:50:52.591 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-02T14:50:52.591 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-02T14:50:52.591 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-02T14:50:52.591 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.591 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.591 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.592 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:52.592 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.592 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.592 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:52.592 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:52.592 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:52.592 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:52.592 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:52.592 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:52.592 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:52.592 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:52.592 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.592 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.592 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.592 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.593 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.593 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-02T14:50:52.593 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-02T14:50:52.593 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.593 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.593 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.593 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:52.593 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.593 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.593 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.593 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.593 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.593 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.593 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.593 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.594 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-02T14:50:52.594 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-02T14:50:52.594 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.594 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.594 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.594 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-02T14:50:52.594 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.594 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.594 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:52.594 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:52.594 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:52.594 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:52.594 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:52.594 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:52.594 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:52.595 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:52.595 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.595 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.595 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.595 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.595 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.595 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-02T14:50:52.595 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-02T14:50:52.595 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.595 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.595 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.595 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:52.595 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.595 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.595 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.595 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.596 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.596 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.596 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.596 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.596 INFO:teuthology.orchestra.run.smithi090.stdout: "storage_class": { 2024-04-02T14:50:52.596 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "storage_class", 2024-04-02T14:50:52.596 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.596 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.596 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.596 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "local", 2024-04-02T14:50:52.596 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.596 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.596 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.596 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-04-02T14:50:52.596 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.597 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.597 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.597 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:52.597 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:52.597 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.597 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-02T14:50:52.597 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "selftest", 2024-04-02T14:50:52.597 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-02T14:50:52.597 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-02T14:50:52.597 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-02T14:50:52.597 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-02T14:50:52.597 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-02T14:50:52.597 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.597 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.597 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.597 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:52.598 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.598 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.598 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:52.598 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:52.598 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:52.598 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:52.598 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:52.598 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:52.598 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:52.598 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:52.598 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.598 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.598 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.598 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.599 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.599 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-02T14:50:52.599 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-02T14:50:52.599 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.599 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.599 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.599 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:52.599 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.599 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.599 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.599 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.599 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.599 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.599 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.599 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.599 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-02T14:50:52.600 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-02T14:50:52.600 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.600 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.600 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.600 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-02T14:50:52.600 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.600 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.600 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:52.600 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:52.600 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:52.600 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:52.600 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:52.600 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:52.600 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:52.600 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:52.601 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.601 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.601 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.601 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.601 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.601 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-02T14:50:52.601 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-02T14:50:52.601 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.601 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.601 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.601 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:52.601 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.601 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.601 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.601 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.602 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.602 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.602 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.602 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.602 INFO:teuthology.orchestra.run.smithi090.stdout: "roption1": { 2024-04-02T14:50:52.602 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "roption1", 2024-04-02T14:50:52.602 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.602 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.602 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.602 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:52.602 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.602 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.602 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.602 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.603 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.603 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.603 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.603 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.603 INFO:teuthology.orchestra.run.smithi090.stdout: "roption2": { 2024-04-02T14:50:52.603 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "roption2", 2024-04-02T14:50:52.603 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.603 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.603 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.603 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "xyz", 2024-04-02T14:50:52.603 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.603 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.603 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.603 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.603 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.604 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.604 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.604 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.604 INFO:teuthology.orchestra.run.smithi090.stdout: "rwoption1": { 2024-04-02T14:50:52.604 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "rwoption1", 2024-04-02T14:50:52.604 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.604 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.604 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.604 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:52.604 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.604 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.604 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.604 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.604 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.604 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.604 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.605 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.605 INFO:teuthology.orchestra.run.smithi090.stdout: "rwoption2": { 2024-04-02T14:50:52.605 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "rwoption2", 2024-04-02T14:50:52.605 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-02T14:50:52.605 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.605 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.605 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:52.605 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.605 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.605 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.605 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.605 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.605 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.605 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.605 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.605 INFO:teuthology.orchestra.run.smithi090.stdout: "rwoption3": { 2024-04-02T14:50:52.606 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "rwoption3", 2024-04-02T14:50:52.606 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "float", 2024-04-02T14:50:52.606 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.606 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.606 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:52.606 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.606 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.606 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.606 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.606 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.606 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.606 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.606 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.606 INFO:teuthology.orchestra.run.smithi090.stdout: "rwoption4": { 2024-04-02T14:50:52.606 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "rwoption4", 2024-04-02T14:50:52.607 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.607 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.607 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.607 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:52.607 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.607 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.607 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.607 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.607 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.607 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.607 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.607 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.607 INFO:teuthology.orchestra.run.smithi090.stdout: "rwoption5": { 2024-04-02T14:50:52.607 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "rwoption5", 2024-04-02T14:50:52.607 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.607 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.608 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.608 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:52.608 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.608 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.608 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.608 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.608 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.608 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.608 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.608 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.608 INFO:teuthology.orchestra.run.smithi090.stdout: "rwoption6": { 2024-04-02T14:50:52.608 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "rwoption6", 2024-04-02T14:50:52.608 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.608 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.608 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.609 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-02T14:50:52.609 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.609 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.609 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.609 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.609 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.609 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.609 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.609 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.609 INFO:teuthology.orchestra.run.smithi090.stdout: "rwoption7": { 2024-04-02T14:50:52.609 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "rwoption7", 2024-04-02T14:50:52.609 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-02T14:50:52.609 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.609 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.610 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:52.610 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "1", 2024-04-02T14:50:52.610 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "42", 2024-04-02T14:50:52.610 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.610 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.610 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.610 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.610 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.610 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.610 INFO:teuthology.orchestra.run.smithi090.stdout: "testkey": { 2024-04-02T14:50:52.610 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "testkey", 2024-04-02T14:50:52.610 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.610 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.610 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.611 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:52.611 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.611 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.611 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.611 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.611 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.611 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.611 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.611 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.611 INFO:teuthology.orchestra.run.smithi090.stdout: "testlkey": { 2024-04-02T14:50:52.611 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "testlkey", 2024-04-02T14:50:52.611 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.611 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.611 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.611 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:52.611 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.612 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.612 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.612 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.612 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.612 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.612 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.612 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.612 INFO:teuthology.orchestra.run.smithi090.stdout: "testnewline": { 2024-04-02T14:50:52.612 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "testnewline", 2024-04-02T14:50:52.612 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.612 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.612 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.612 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:52.612 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.613 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.613 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.613 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.613 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.613 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.613 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.613 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:52.613 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:52.613 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.613 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-02T14:50:52.613 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "snap_schedule", 2024-04-02T14:50:52.613 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-02T14:50:52.613 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-02T14:50:52.613 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-02T14:50:52.613 INFO:teuthology.orchestra.run.smithi090.stdout: "allow_m_granularity": { 2024-04-02T14:50:52.614 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "allow_m_granularity", 2024-04-02T14:50:52.614 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.614 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.614 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.614 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:52.614 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.614 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.614 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.614 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "allow minute scheduled snapshots", 2024-04-02T14:50:52.614 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.614 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.614 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.614 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.614 INFO:teuthology.orchestra.run.smithi090.stdout: "dump_on_update": { 2024-04-02T14:50:52.614 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "dump_on_update", 2024-04-02T14:50:52.615 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.615 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.615 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.615 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:52.615 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.615 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.615 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.615 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "dump database to debug log on update", 2024-04-02T14:50:52.615 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.615 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.615 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.615 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.615 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-02T14:50:52.615 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-02T14:50:52.615 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.616 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.616 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.616 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:52.616 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.616 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.616 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:52.616 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:52.616 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:52.616 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:52.616 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:52.616 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:52.616 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:52.616 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:52.616 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.616 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.616 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.617 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.617 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.617 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-02T14:50:52.617 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-02T14:50:52.617 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.617 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.617 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.617 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:52.617 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.617 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.617 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.617 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.617 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.617 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.617 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.617 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.618 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-02T14:50:52.618 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-02T14:50:52.618 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.618 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.618 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.618 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-02T14:50:52.618 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.618 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.618 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:52.618 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:52.618 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:52.618 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:52.618 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:52.618 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:52.618 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:52.619 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:52.619 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.619 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.619 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.619 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.619 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.619 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-02T14:50:52.619 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-02T14:50:52.619 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.619 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.619 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.619 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:52.619 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.619 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.619 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.619 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.620 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.620 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.620 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.620 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:52.620 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:52.620 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.620 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-02T14:50:52.620 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "stats", 2024-04-02T14:50:52.620 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-02T14:50:52.620 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-02T14:50:52.620 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-02T14:50:52.620 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-02T14:50:52.620 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-02T14:50:52.620 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.620 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.620 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.621 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:52.621 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.621 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.621 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:52.621 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:52.621 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:52.621 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:52.621 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:52.621 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:52.621 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:52.621 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:52.621 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.621 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.621 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.621 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.621 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.622 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-02T14:50:52.622 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-02T14:50:52.622 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.622 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.622 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.622 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:52.622 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.622 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.622 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.622 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.622 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.622 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.622 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.622 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.623 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-02T14:50:52.623 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-02T14:50:52.623 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.623 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.623 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.623 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-02T14:50:52.623 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.623 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.623 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:52.623 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:52.623 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:52.623 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:52.623 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:52.623 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:52.623 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:52.623 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:52.624 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.624 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.624 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.624 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.624 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.624 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-02T14:50:52.624 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-02T14:50:52.624 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.624 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.624 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.624 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:52.624 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.624 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.624 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.624 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.625 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.625 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.625 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.625 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:52.625 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:52.625 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.625 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-02T14:50:52.625 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "status", 2024-04-02T14:50:52.625 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-02T14:50:52.625 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-02T14:50:52.625 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-02T14:50:52.625 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-02T14:50:52.625 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-02T14:50:52.625 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.625 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.626 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.626 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:52.626 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.626 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.626 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:52.626 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:52.626 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:52.626 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:52.626 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:52.626 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:52.626 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:52.626 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:52.626 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.626 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.627 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.627 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.627 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.627 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-02T14:50:52.627 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-02T14:50:52.627 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.627 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.627 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.627 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:52.627 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.627 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.627 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.627 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.627 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.627 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.627 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.628 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.628 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-02T14:50:52.628 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-02T14:50:52.628 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.628 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.628 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.628 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-02T14:50:52.628 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.628 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.628 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:52.628 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:52.628 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:52.628 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:52.628 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:52.628 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:52.628 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:52.629 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:52.629 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.629 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.629 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.629 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.629 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.629 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-02T14:50:52.629 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-02T14:50:52.629 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.629 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.629 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.629 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:52.629 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.629 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.629 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.630 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.630 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.630 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.630 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.630 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:52.630 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:52.630 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.630 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-02T14:50:52.630 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "telegraf", 2024-04-02T14:50:52.630 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-02T14:50:52.630 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-02T14:50:52.630 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-02T14:50:52.630 INFO:teuthology.orchestra.run.smithi090.stdout: "address": { 2024-04-02T14:50:52.630 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "address", 2024-04-02T14:50:52.630 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.631 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.631 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.631 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-04-02T14:50:52.631 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.631 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.631 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.631 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.631 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.631 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.631 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.631 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.631 INFO:teuthology.orchestra.run.smithi090.stdout: "interval": { 2024-04-02T14:50:52.631 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "interval", 2024-04-02T14:50:52.631 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "secs", 2024-04-02T14:50:52.631 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.632 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.632 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "15", 2024-04-02T14:50:52.632 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.632 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.632 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.632 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.632 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.632 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.632 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.632 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.632 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-02T14:50:52.632 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-02T14:50:52.632 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.632 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.632 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.633 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:52.633 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.633 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.633 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:52.633 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:52.633 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:52.633 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:52.633 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:52.633 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:52.633 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:52.633 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:52.633 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.633 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.633 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.633 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.633 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.634 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-02T14:50:52.634 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-02T14:50:52.634 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.634 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.634 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.634 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:52.634 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.634 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.634 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.634 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.634 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.634 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.634 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.634 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.634 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-02T14:50:52.635 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-02T14:50:52.635 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.635 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.635 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.635 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-02T14:50:52.635 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.635 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.635 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:52.635 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:52.635 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:52.635 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:52.635 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:52.635 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:52.635 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:52.635 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:52.636 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.636 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.636 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.636 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.636 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.636 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-02T14:50:52.636 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-02T14:50:52.636 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.636 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.636 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.636 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:52.636 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.636 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.636 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.636 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.636 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.637 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.637 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.637 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:52.637 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:52.637 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.637 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-02T14:50:52.637 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "telemetry", 2024-04-02T14:50:52.637 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-02T14:50:52.637 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-02T14:50:52.637 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-02T14:50:52.637 INFO:teuthology.orchestra.run.smithi090.stdout: "channel_basic": { 2024-04-02T14:50:52.637 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "channel_basic", 2024-04-02T14:50:52.637 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.637 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.637 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.638 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-02T14:50:52.638 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.638 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.638 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.638 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Share basic cluster information (size, version)", 2024-04-02T14:50:52.638 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.638 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.638 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.638 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.638 INFO:teuthology.orchestra.run.smithi090.stdout: "channel_crash": { 2024-04-02T14:50:52.638 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "channel_crash", 2024-04-02T14:50:52.638 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.638 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.638 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.639 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-02T14:50:52.639 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.639 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.639 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.639 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-04-02T14:50:52.639 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.639 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.639 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.639 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.639 INFO:teuthology.orchestra.run.smithi090.stdout: "channel_device": { 2024-04-02T14:50:52.639 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "channel_device", 2024-04-02T14:50:52.639 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.639 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.639 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.639 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-02T14:50:52.640 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.640 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.640 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.640 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-04-02T14:50:52.640 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.640 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.640 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.640 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.640 INFO:teuthology.orchestra.run.smithi090.stdout: "channel_ident": { 2024-04-02T14:50:52.640 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "channel_ident", 2024-04-02T14:50:52.640 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.640 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.640 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.640 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:52.640 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.641 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.641 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.641 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-04-02T14:50:52.641 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.641 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.641 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.641 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.641 INFO:teuthology.orchestra.run.smithi090.stdout: "channel_perf": { 2024-04-02T14:50:52.641 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "channel_perf", 2024-04-02T14:50:52.641 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.641 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.641 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.641 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:52.641 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.641 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.642 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.642 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Share various performance metrics of a cluster", 2024-04-02T14:50:52.642 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.642 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.642 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.642 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.642 INFO:teuthology.orchestra.run.smithi090.stdout: "contact": { 2024-04-02T14:50:52.642 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "contact", 2024-04-02T14:50:52.642 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.642 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.642 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.642 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:52.642 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.642 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.643 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.643 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.643 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.643 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.643 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.643 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.643 INFO:teuthology.orchestra.run.smithi090.stdout: "description": { 2024-04-02T14:50:52.643 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "description", 2024-04-02T14:50:52.643 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.643 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.643 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.643 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:52.643 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.643 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.643 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.643 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.644 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.644 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.644 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.644 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.644 INFO:teuthology.orchestra.run.smithi090.stdout: "device_url": { 2024-04-02T14:50:52.644 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "device_url", 2024-04-02T14:50:52.644 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.644 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.644 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.644 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-04-02T14:50:52.644 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.644 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.644 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.644 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.645 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.645 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.645 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.645 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.645 INFO:teuthology.orchestra.run.smithi090.stdout: "enabled": { 2024-04-02T14:50:52.645 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "enabled", 2024-04-02T14:50:52.645 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.645 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.645 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.645 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:52.645 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.645 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.645 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.645 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.645 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.646 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.646 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.646 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.646 INFO:teuthology.orchestra.run.smithi090.stdout: "interval": { 2024-04-02T14:50:52.646 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "interval", 2024-04-02T14:50:52.646 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-02T14:50:52.646 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.646 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.646 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "24", 2024-04-02T14:50:52.646 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "8", 2024-04-02T14:50:52.646 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.646 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.646 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.647 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.647 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.647 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.647 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.647 INFO:teuthology.orchestra.run.smithi090.stdout: "last_opt_revision": { 2024-04-02T14:50:52.647 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "last_opt_revision", 2024-04-02T14:50:52.647 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-02T14:50:52.647 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.647 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.647 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "1", 2024-04-02T14:50:52.647 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.647 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.647 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.647 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.647 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.648 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.648 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.648 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.648 INFO:teuthology.orchestra.run.smithi090.stdout: "leaderboard": { 2024-04-02T14:50:52.648 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "leaderboard", 2024-04-02T14:50:52.648 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.648 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.648 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.648 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:52.648 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.648 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.648 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.648 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.648 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.648 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.648 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.649 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.649 INFO:teuthology.orchestra.run.smithi090.stdout: "leaderboard_description": { 2024-04-02T14:50:52.649 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "leaderboard_description", 2024-04-02T14:50:52.649 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.649 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.649 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.649 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:52.649 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.649 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.649 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.649 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.649 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.649 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.649 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.649 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.649 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-02T14:50:52.650 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-02T14:50:52.650 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.650 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.650 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.650 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:52.650 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.650 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.650 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:52.650 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:52.650 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:52.650 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:52.650 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:52.650 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:52.650 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:52.651 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:52.651 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.651 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.651 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.651 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.651 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.651 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-02T14:50:52.651 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-02T14:50:52.651 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.651 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.651 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.651 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:52.651 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.651 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.651 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.651 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.652 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.652 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.654 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.654 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.654 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-02T14:50:52.654 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-02T14:50:52.654 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.654 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.654 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.654 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-02T14:50:52.654 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.654 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.655 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:52.655 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:52.655 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:52.655 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:52.655 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:52.655 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:52.655 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:52.655 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:52.655 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.655 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.655 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.655 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.655 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.655 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-02T14:50:52.655 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-02T14:50:52.655 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.656 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.656 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.656 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:52.656 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.656 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.656 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.656 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.656 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.656 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.656 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.656 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.656 INFO:teuthology.orchestra.run.smithi090.stdout: "organization": { 2024-04-02T14:50:52.656 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "organization", 2024-04-02T14:50:52.656 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.656 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.657 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.657 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:52.657 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.657 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.657 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.657 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.657 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.657 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.657 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.657 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.657 INFO:teuthology.orchestra.run.smithi090.stdout: "proxy": { 2024-04-02T14:50:52.657 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "proxy", 2024-04-02T14:50:52.657 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.657 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.657 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.657 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:52.658 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.658 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.658 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.658 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.658 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.658 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.658 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.658 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.658 INFO:teuthology.orchestra.run.smithi090.stdout: "url": { 2024-04-02T14:50:52.658 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "url", 2024-04-02T14:50:52.658 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.658 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.658 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.658 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-04-02T14:50:52.659 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.659 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.659 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.659 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.659 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.659 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.659 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.659 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:52.659 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:52.659 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.659 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-02T14:50:52.659 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "test_orchestrator", 2024-04-02T14:50:52.659 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-02T14:50:52.659 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-02T14:50:52.659 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-02T14:50:52.660 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-02T14:50:52.660 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-02T14:50:52.660 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.660 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.660 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.660 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:52.660 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.660 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.660 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:52.660 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:52.660 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:52.660 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:52.660 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:52.660 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:52.660 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:52.661 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:52.661 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.661 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.661 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.661 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.661 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.661 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-02T14:50:52.661 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-02T14:50:52.661 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.661 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.661 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.661 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:52.661 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.661 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.661 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.662 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.662 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.662 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.662 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.662 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.662 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-02T14:50:52.662 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-02T14:50:52.662 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.662 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.662 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.662 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-02T14:50:52.662 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.662 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.662 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:52.662 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:52.663 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:52.663 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:52.663 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:52.663 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:52.663 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:52.663 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:52.663 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.663 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.663 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.663 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.663 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.663 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-02T14:50:52.663 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-02T14:50:52.663 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.664 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.664 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.664 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:52.664 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.664 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.664 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.664 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.664 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.664 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.664 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.664 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:52.664 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:52.664 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.664 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-02T14:50:52.664 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "volumes", 2024-04-02T14:50:52.664 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-02T14:50:52.665 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-02T14:50:52.665 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-02T14:50:52.665 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-02T14:50:52.665 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-02T14:50:52.665 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.665 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.665 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.665 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:52.665 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.665 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.665 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:52.665 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:52.665 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:52.665 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:52.665 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:52.666 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:52.666 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:52.666 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:52.666 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.666 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.666 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.666 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.666 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.666 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-02T14:50:52.666 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-02T14:50:52.666 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.666 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.666 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.666 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:52.667 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.667 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.667 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.667 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.667 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.667 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.667 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.667 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.667 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-02T14:50:52.667 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-02T14:50:52.667 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.667 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.667 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.667 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-02T14:50:52.667 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.667 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.668 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:52.668 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:52.668 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:52.668 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:52.668 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:52.668 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:52.668 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:52.668 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:52.668 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.668 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.668 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.668 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.668 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.668 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-02T14:50:52.668 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-02T14:50:52.668 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.668 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.669 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.669 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:52.669 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.669 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.669 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.669 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.669 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.669 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.669 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.669 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.669 INFO:teuthology.orchestra.run.smithi090.stdout: "max_concurrent_clones": { 2024-04-02T14:50:52.669 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "max_concurrent_clones", 2024-04-02T14:50:52.669 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-02T14:50:52.669 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.669 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.669 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "4", 2024-04-02T14:50:52.670 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.670 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.670 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.670 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Number of asynchronous cloner threads", 2024-04-02T14:50:52.670 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.670 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.670 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.670 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.670 INFO:teuthology.orchestra.run.smithi090.stdout: "periodic_async_work": { 2024-04-02T14:50:52.670 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "periodic_async_work", 2024-04-02T14:50:52.670 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.670 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.670 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.670 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:52.670 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.671 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.671 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.671 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Periodically check for async work", 2024-04-02T14:50:52.671 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.671 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.671 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.671 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.671 INFO:teuthology.orchestra.run.smithi090.stdout: "snapshot_clone_delay": { 2024-04-02T14:50:52.671 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "snapshot_clone_delay", 2024-04-02T14:50:52.671 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-02T14:50:52.671 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.671 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.671 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "0", 2024-04-02T14:50:52.671 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.671 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.672 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.672 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-04-02T14:50:52.672 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.672 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.672 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.672 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.672 INFO:teuthology.orchestra.run.smithi090.stdout: "snapshot_clone_no_wait": { 2024-04-02T14:50:52.672 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "snapshot_clone_no_wait", 2024-04-02T14:50:52.672 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.672 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.672 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.672 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "True", 2024-04-02T14:50:52.672 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.672 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.672 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.673 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-04-02T14:50:52.673 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.673 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.673 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.673 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:52.673 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:52.673 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.673 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-02T14:50:52.673 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "zabbix", 2024-04-02T14:50:52.673 INFO:teuthology.orchestra.run.smithi090.stdout: "can_run": true, 2024-04-02T14:50:52.673 INFO:teuthology.orchestra.run.smithi090.stdout: "error_string": "", 2024-04-02T14:50:52.673 INFO:teuthology.orchestra.run.smithi090.stdout: "module_options": { 2024-04-02T14:50:52.673 INFO:teuthology.orchestra.run.smithi090.stdout: "discovery_interval": { 2024-04-02T14:50:52.673 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "discovery_interval", 2024-04-02T14:50:52.673 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "uint", 2024-04-02T14:50:52.673 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.674 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.674 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "100", 2024-04-02T14:50:52.674 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.674 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.674 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.674 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.674 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.674 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.674 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.674 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.674 INFO:teuthology.orchestra.run.smithi090.stdout: "identifier": { 2024-04-02T14:50:52.674 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "identifier", 2024-04-02T14:50:52.674 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.674 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.675 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.675 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:52.675 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.675 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.675 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.675 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.675 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.675 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.675 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.675 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.675 INFO:teuthology.orchestra.run.smithi090.stdout: "interval": { 2024-04-02T14:50:52.675 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "interval", 2024-04-02T14:50:52.675 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "secs", 2024-04-02T14:50:52.675 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.676 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.676 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "60", 2024-04-02T14:50:52.676 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.676 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.676 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.676 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.676 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.676 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.676 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.676 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.676 INFO:teuthology.orchestra.run.smithi090.stdout: "log_level": { 2024-04-02T14:50:52.676 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_level", 2024-04-02T14:50:52.676 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.676 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.676 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.676 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:52.677 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.677 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.677 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:52.677 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:52.677 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:52.677 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:52.677 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:52.677 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:52.677 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:52.677 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:52.677 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.677 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.677 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.677 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.677 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.678 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster": { 2024-04-02T14:50:52.678 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster", 2024-04-02T14:50:52.678 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.678 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.678 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.678 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:52.678 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.678 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.678 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.678 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.678 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.678 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.678 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.678 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.678 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_cluster_level": { 2024-04-02T14:50:52.679 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_cluster_level", 2024-04-02T14:50:52.679 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.679 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.679 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.679 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "info", 2024-04-02T14:50:52.679 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.679 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.679 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [ 2024-04-02T14:50:52.679 INFO:teuthology.orchestra.run.smithi090.stdout: "", 2024-04-02T14:50:52.679 INFO:teuthology.orchestra.run.smithi090.stdout: "critical", 2024-04-02T14:50:52.679 INFO:teuthology.orchestra.run.smithi090.stdout: "debug", 2024-04-02T14:50:52.679 INFO:teuthology.orchestra.run.smithi090.stdout: "error", 2024-04-02T14:50:52.679 INFO:teuthology.orchestra.run.smithi090.stdout: "info", 2024-04-02T14:50:52.679 INFO:teuthology.orchestra.run.smithi090.stdout: "warning" 2024-04-02T14:50:52.679 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:52.679 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.680 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.680 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.680 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.680 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.680 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": { 2024-04-02T14:50:52.680 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "log_to_file", 2024-04-02T14:50:52.680 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bool", 2024-04-02T14:50:52.680 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.680 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-02T14:50:52.680 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "False", 2024-04-02T14:50:52.680 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.680 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.680 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.680 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.680 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.681 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.681 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.681 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.681 INFO:teuthology.orchestra.run.smithi090.stdout: "zabbix_host": { 2024-04-02T14:50:52.681 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "zabbix_host", 2024-04-02T14:50:52.681 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.681 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.681 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.681 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "", 2024-04-02T14:50:52.681 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.681 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.681 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.681 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.681 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.681 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.682 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.682 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.682 INFO:teuthology.orchestra.run.smithi090.stdout: "zabbix_port": { 2024-04-02T14:50:52.682 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "zabbix_port", 2024-04-02T14:50:52.682 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "int", 2024-04-02T14:50:52.682 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.682 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.682 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "10051", 2024-04-02T14:50:52.682 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.682 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.682 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.682 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.682 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.682 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.683 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.683 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.683 INFO:teuthology.orchestra.run.smithi090.stdout: "zabbix_sender": { 2024-04-02T14:50:52.683 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "zabbix_sender", 2024-04-02T14:50:52.683 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "str", 2024-04-02T14:50:52.683 INFO:teuthology.orchestra.run.smithi090.stdout: "level": "advanced", 2024-04-02T14:50:52.683 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 0, 2024-04-02T14:50:52.683 INFO:teuthology.orchestra.run.smithi090.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-04-02T14:50:52.683 INFO:teuthology.orchestra.run.smithi090.stdout: "min": "", 2024-04-02T14:50:52.683 INFO:teuthology.orchestra.run.smithi090.stdout: "max": "", 2024-04-02T14:50:52.683 INFO:teuthology.orchestra.run.smithi090.stdout: "enum_allowed": [], 2024-04-02T14:50:52.683 INFO:teuthology.orchestra.run.smithi090.stdout: "desc": "", 2024-04-02T14:50:52.683 INFO:teuthology.orchestra.run.smithi090.stdout: "long_desc": "", 2024-04-02T14:50:52.683 INFO:teuthology.orchestra.run.smithi090.stdout: "tags": [], 2024-04-02T14:50:52.683 INFO:teuthology.orchestra.run.smithi090.stdout: "see_also": [] 2024-04-02T14:50:52.683 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:52.684 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:52.684 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:52.684 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:52.684 INFO:teuthology.orchestra.run.smithi090.stdout: "services": { 2024-04-02T14:50:52.684 INFO:teuthology.orchestra.run.smithi090.stdout: "dashboard": "https://172.21.15.90:8443/" 2024-04-02T14:50:52.684 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.684 INFO:teuthology.orchestra.run.smithi090.stdout: "always_on_modules": { 2024-04-02T14:50:52.684 INFO:teuthology.orchestra.run.smithi090.stdout: "octopus": [ 2024-04-02T14:50:52.684 INFO:teuthology.orchestra.run.smithi090.stdout: "balancer", 2024-04-02T14:50:52.684 INFO:teuthology.orchestra.run.smithi090.stdout: "crash", 2024-04-02T14:50:52.684 INFO:teuthology.orchestra.run.smithi090.stdout: "devicehealth", 2024-04-02T14:50:52.684 INFO:teuthology.orchestra.run.smithi090.stdout: "orchestrator", 2024-04-02T14:50:52.684 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_autoscaler", 2024-04-02T14:50:52.684 INFO:teuthology.orchestra.run.smithi090.stdout: "progress", 2024-04-02T14:50:52.684 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_support", 2024-04-02T14:50:52.684 INFO:teuthology.orchestra.run.smithi090.stdout: "status", 2024-04-02T14:50:52.685 INFO:teuthology.orchestra.run.smithi090.stdout: "telemetry", 2024-04-02T14:50:52.685 INFO:teuthology.orchestra.run.smithi090.stdout: "volumes" 2024-04-02T14:50:52.685 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:52.685 INFO:teuthology.orchestra.run.smithi090.stdout: "pacific": [ 2024-04-02T14:50:52.685 INFO:teuthology.orchestra.run.smithi090.stdout: "balancer", 2024-04-02T14:50:52.685 INFO:teuthology.orchestra.run.smithi090.stdout: "crash", 2024-04-02T14:50:52.685 INFO:teuthology.orchestra.run.smithi090.stdout: "devicehealth", 2024-04-02T14:50:52.685 INFO:teuthology.orchestra.run.smithi090.stdout: "orchestrator", 2024-04-02T14:50:52.685 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_autoscaler", 2024-04-02T14:50:52.685 INFO:teuthology.orchestra.run.smithi090.stdout: "progress", 2024-04-02T14:50:52.685 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_support", 2024-04-02T14:50:52.685 INFO:teuthology.orchestra.run.smithi090.stdout: "status", 2024-04-02T14:50:52.685 INFO:teuthology.orchestra.run.smithi090.stdout: "telemetry", 2024-04-02T14:50:52.685 INFO:teuthology.orchestra.run.smithi090.stdout: "volumes" 2024-04-02T14:50:52.685 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:52.686 INFO:teuthology.orchestra.run.smithi090.stdout: "quincy": [ 2024-04-02T14:50:52.686 INFO:teuthology.orchestra.run.smithi090.stdout: "balancer", 2024-04-02T14:50:52.686 INFO:teuthology.orchestra.run.smithi090.stdout: "crash", 2024-04-02T14:50:52.686 INFO:teuthology.orchestra.run.smithi090.stdout: "devicehealth", 2024-04-02T14:50:52.686 INFO:teuthology.orchestra.run.smithi090.stdout: "orchestrator", 2024-04-02T14:50:52.686 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_autoscaler", 2024-04-02T14:50:52.686 INFO:teuthology.orchestra.run.smithi090.stdout: "progress", 2024-04-02T14:50:52.686 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_support", 2024-04-02T14:50:52.686 INFO:teuthology.orchestra.run.smithi090.stdout: "status", 2024-04-02T14:50:52.686 INFO:teuthology.orchestra.run.smithi090.stdout: "telemetry", 2024-04-02T14:50:52.686 INFO:teuthology.orchestra.run.smithi090.stdout: "volumes" 2024-04-02T14:50:52.686 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:52.686 INFO:teuthology.orchestra.run.smithi090.stdout: "reef": [ 2024-04-02T14:50:52.686 INFO:teuthology.orchestra.run.smithi090.stdout: "balancer", 2024-04-02T14:50:52.686 INFO:teuthology.orchestra.run.smithi090.stdout: "crash", 2024-04-02T14:50:52.686 INFO:teuthology.orchestra.run.smithi090.stdout: "devicehealth", 2024-04-02T14:50:52.687 INFO:teuthology.orchestra.run.smithi090.stdout: "orchestrator", 2024-04-02T14:50:52.687 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_autoscaler", 2024-04-02T14:50:52.687 INFO:teuthology.orchestra.run.smithi090.stdout: "progress", 2024-04-02T14:50:52.687 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_support", 2024-04-02T14:50:52.687 INFO:teuthology.orchestra.run.smithi090.stdout: "status", 2024-04-02T14:50:52.687 INFO:teuthology.orchestra.run.smithi090.stdout: "telemetry", 2024-04-02T14:50:52.687 INFO:teuthology.orchestra.run.smithi090.stdout: "volumes" 2024-04-02T14:50:52.687 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-02T14:50:52.687 INFO:teuthology.orchestra.run.smithi090.stdout: "squid": [ 2024-04-02T14:50:52.687 INFO:teuthology.orchestra.run.smithi090.stdout: "balancer", 2024-04-02T14:50:52.687 INFO:teuthology.orchestra.run.smithi090.stdout: "crash", 2024-04-02T14:50:52.687 INFO:teuthology.orchestra.run.smithi090.stdout: "devicehealth", 2024-04-02T14:50:52.687 INFO:teuthology.orchestra.run.smithi090.stdout: "orchestrator", 2024-04-02T14:50:52.687 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_autoscaler", 2024-04-02T14:50:52.687 INFO:teuthology.orchestra.run.smithi090.stdout: "progress", 2024-04-02T14:50:52.687 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_support", 2024-04-02T14:50:52.688 INFO:teuthology.orchestra.run.smithi090.stdout: "status", 2024-04-02T14:50:52.688 INFO:teuthology.orchestra.run.smithi090.stdout: "telemetry", 2024-04-02T14:50:52.688 INFO:teuthology.orchestra.run.smithi090.stdout: "volumes" 2024-04-02T14:50:52.688 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-02T14:50:52.688 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.688 INFO:teuthology.orchestra.run.smithi090.stdout: "last_failure_osd_epoch": 91, 2024-04-02T14:50:52.688 INFO:teuthology.orchestra.run.smithi090.stdout: "active_clients": [ 2024-04-02T14:50:52.688 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-02T14:50:52.688 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "devicehealth", 2024-04-02T14:50:52.688 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-02T14:50:52.688 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-02T14:50:52.688 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2024-04-02T14:50:52.688 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:0", 2024-04-02T14:50:52.688 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 358685899 2024-04-02T14:50:52.688 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:52.688 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-02T14:50:52.689 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.689 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-02T14:50:52.689 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "libcephsqlite", 2024-04-02T14:50:52.689 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-02T14:50:52.689 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-02T14:50:52.689 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2024-04-02T14:50:52.689 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:0", 2024-04-02T14:50:52.689 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 731838965 2024-04-02T14:50:52.689 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:52.689 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-02T14:50:52.689 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.689 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-02T14:50:52.689 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "rbd_support", 2024-04-02T14:50:52.689 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-02T14:50:52.689 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-02T14:50:52.689 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2024-04-02T14:50:52.689 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:0", 2024-04-02T14:50:52.690 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 236223048 2024-04-02T14:50:52.690 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:52.690 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-02T14:50:52.690 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.690 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-02T14:50:52.690 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "snap_schedule", 2024-04-02T14:50:52.690 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-02T14:50:52.690 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-02T14:50:52.690 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2024-04-02T14:50:52.690 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:0", 2024-04-02T14:50:52.690 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 516602713 2024-04-02T14:50:52.690 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:52.690 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-02T14:50:52.690 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.690 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-02T14:50:52.691 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "snap_schedule", 2024-04-02T14:50:52.691 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-02T14:50:52.691 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-02T14:50:52.691 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2024-04-02T14:50:52.691 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:0", 2024-04-02T14:50:52.691 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 2730331774 2024-04-02T14:50:52.691 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:52.691 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-02T14:50:52.691 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:52.691 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-02T14:50:52.691 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "volumes", 2024-04-02T14:50:52.691 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-02T14:50:52.691 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-02T14:50:52.691 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2024-04-02T14:50:52.691 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:0", 2024-04-02T14:50:52.691 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 2440445516 2024-04-02T14:50:52.691 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:52.692 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-02T14:50:52.692 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:52.692 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-02T14:50:52.692 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-02T14:50:52.693 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:50:51 smithi177 ceph-mon[36431]: from='client.? 172.21.15.90:0/3896688807' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-02T14:50:52.694 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:50:51 smithi177 ceph-mon[36431]: from='client.? 172.21.15.90:0/865775243' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-02T14:50:52.694 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:50:52 smithi177 ceph-mon[36431]: pgmap v500: 97 pgs: 97 active+clean; 4.3 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 184 KiB/s rd, 13 MiB/s wr, 1.32k op/s 2024-04-02T14:50:52.694 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:50:52 smithi177 ceph-mon[36431]: from='client.? 172.21.15.90:0/1181514577' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-04-02T14:50:52.694 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:50:51 smithi092 ceph-mon[36497]: from='client.? 172.21.15.90:0/3896688807' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-02T14:50:52.694 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:50:51 smithi092 ceph-mon[36497]: from='client.? 172.21.15.90:0/865775243' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-02T14:50:52.694 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:50:52 smithi092 ceph-mon[36497]: pgmap v500: 97 pgs: 97 active+clean; 4.3 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 184 KiB/s rd, 13 MiB/s wr, 1.32k op/s 2024-04-02T14:50:52.694 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:50:52 smithi092 ceph-mon[36497]: from='client.? 172.21.15.90:0/1181514577' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-04-02T14:50:52.695 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:51.466+0000 7f3ec9a00640 1 -- 172.21.15.90:0/1181514577 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f3ea80a3d20 msgr2=0x7f3ea80a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:52.695 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:51.466+0000 7f3ec9a00640 1 --2- 172.21.15.90:0/1181514577 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f3ea80a3d20 0x7f3ea80a61e0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f3eb0004af0 tx=0x7f3eb0005c50 comp rx=0 tx=0).stop 2024-04-02T14:50:52.695 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:51.466+0000 7f3ec9a00640 1 -- 172.21.15.90:0/1181514577 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3ec4073eb0 msgr2=0x7f3ec419a630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:50:52.695 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:51.466+0000 7f3ec9a00640 1 --2- 172.21.15.90:0/1181514577 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3ec4073eb0 0x7f3ec419a630 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f3eb400ea60 tx=0x7f3eb400ef20 comp rx=0 tx=0).stop 2024-04-02T14:50:52.695 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:51.466+0000 7f3ec9a00640 1 -- 172.21.15.90:0/1181514577 shutdown_connections 2024-04-02T14:50:52.695 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:51.466+0000 7f3ec9a00640 1 --2- 172.21.15.90:0/1181514577 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f3ea80a3d20 0x7f3ea80a61e0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:52.695 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:51.466+0000 7f3ec9a00640 1 --2- 172.21.15.90:0/1181514577 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3ec410e380 0x7f3ec419eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:52.695 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:51.466+0000 7f3ec9a00640 1 --2- 172.21.15.90:0/1181514577 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3ec40747d0 0x7f3ec419ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:52.695 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:51.466+0000 7f3ec9a00640 1 --2- 172.21.15.90:0/1181514577 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3ec4073eb0 0x7f3ec419a630 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:50:52.695 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:51.466+0000 7f3ec9a00640 1 -- 172.21.15.90:0/1181514577 >> 172.21.15.90:0/1181514577 conn(0x7f3ec40fd590 msgr2=0x7f3ec40fee20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:50:52.695 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:51.467+0000 7f3ec9a00640 1 -- 172.21.15.90:0/1181514577 shutdown_connections 2024-04-02T14:50:52.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:50:51.467+0000 7f3ec9a00640 1 -- 172.21.15.90:0/1181514577 wait complete. 2024-04-02T14:50:52.701 DEBUG:tasks.check_counter:Getting stats from a 2024-04-02T14:50:52.702 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:191d963ac9de7322da8aff20b567b3fa458e2ef6 shell --fsid 6449448a-f0fc-11ee-b647-cb9ed24678a4 -- ceph daemon mds.a perf dump 2024-04-02T14:50:52.702 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:50:51 smithi090 ceph-mon[31674]: from='client.? 172.21.15.90:0/3896688807' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-02T14:50:52.702 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:50:51 smithi090 ceph-mon[31674]: from='client.? 172.21.15.90:0/865775243' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-02T14:50:52.702 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:50:52 smithi090 ceph-mon[31674]: pgmap v500: 97 pgs: 97 active+clean; 4.3 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 184 KiB/s rd, 13 MiB/s wr, 1.32k op/s 2024-04-02T14:50:52.702 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:50:52 smithi090 ceph-mon[31674]: from='client.? 172.21.15.90:0/1181514577' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-04-02T14:50:53.148 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T14:50:54.602 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:50:54 smithi092 ceph-mon[36497]: pgmap v501: 97 pgs: 97 active+clean; 4.3 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 5.2 MiB/s wr, 510 op/s 2024-04-02T14:50:54.649 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:50:54 smithi090 ceph-mon[31674]: pgmap v501: 97 pgs: 97 active+clean; 4.3 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 5.2 MiB/s wr, 510 op/s 2024-04-02T14:50:54.678 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:50:54 smithi177 ceph-mon[36431]: pgmap v501: 97 pgs: 97 active+clean; 4.3 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 5.2 MiB/s wr, 510 op/s 2024-04-02T14:50:55.488 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-02T14:50:55.489 INFO:teuthology.orchestra.run.smithi090.stdout: "AsyncMessenger::Worker-0": { 2024-04-02T14:50:55.489 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_recv_messages": 0, 2024-04-02T14:50:55.489 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_send_messages": 3, 2024-04-02T14:50:55.489 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_recv_bytes": 0, 2024-04-02T14:50:55.489 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_send_bytes": 0, 2024-04-02T14:50:55.489 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_created_connections": 5, 2024-04-02T14:50:55.489 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_active_connections": 0, 2024-04-02T14:50:55.489 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_running_total_time": 0.004429038, 2024-04-02T14:50:55.489 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_running_send_time": 0.000000000, 2024-04-02T14:50:55.489 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_running_recv_time": 0.001253830, 2024-04-02T14:50:55.489 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-02T14:50:55.490 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_send_messages_queue_lat": { 2024-04-02T14:50:55.490 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 0, 2024-04-02T14:50:55.490 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.000000000, 2024-04-02T14:50:55.490 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000000000 2024-04-02T14:50:55.490 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:55.490 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_handle_ack_lat": { 2024-04-02T14:50:55.490 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 0, 2024-04-02T14:50:55.490 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.000000000, 2024-04-02T14:50:55.490 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000000000 2024-04-02T14:50:55.490 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:55.490 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-02T14:50:55.490 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_send_encrypted_bytes": 0 2024-04-02T14:50:55.490 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:55.491 INFO:teuthology.orchestra.run.smithi090.stdout: "AsyncMessenger::Worker-1": { 2024-04-02T14:50:55.491 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_recv_messages": 3, 2024-04-02T14:50:55.491 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_send_messages": 230, 2024-04-02T14:50:55.491 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_recv_bytes": 480, 2024-04-02T14:50:55.491 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_send_bytes": 276432, 2024-04-02T14:50:55.491 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_created_connections": 7, 2024-04-02T14:50:55.491 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_active_connections": 3, 2024-04-02T14:50:55.491 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_running_total_time": 0.041997984, 2024-04-02T14:50:55.491 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_running_send_time": 0.030458833, 2024-04-02T14:50:55.491 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_running_recv_time": 0.005401573, 2024-04-02T14:50:55.491 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-02T14:50:55.491 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_send_messages_queue_lat": { 2024-04-02T14:50:55.491 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 226, 2024-04-02T14:50:55.491 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.045010414, 2024-04-02T14:50:55.492 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000199161 2024-04-02T14:50:55.492 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:55.492 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_handle_ack_lat": { 2024-04-02T14:50:55.492 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 0, 2024-04-02T14:50:55.492 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.000000000, 2024-04-02T14:50:55.492 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000000000 2024-04-02T14:50:55.492 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:55.492 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_recv_encrypted_bytes": 480, 2024-04-02T14:50:55.492 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_send_encrypted_bytes": 276432 2024-04-02T14:50:55.492 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:55.492 INFO:teuthology.orchestra.run.smithi090.stdout: "AsyncMessenger::Worker-2": { 2024-04-02T14:50:55.492 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_recv_messages": 316, 2024-04-02T14:50:55.492 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_send_messages": 302, 2024-04-02T14:50:55.492 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_recv_bytes": 1759104, 2024-04-02T14:50:55.493 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_send_bytes": 185184, 2024-04-02T14:50:55.493 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_created_connections": 3, 2024-04-02T14:50:55.493 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_active_connections": 1, 2024-04-02T14:50:55.493 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_running_total_time": 0.103899004, 2024-04-02T14:50:55.493 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_running_send_time": 0.040146990, 2024-04-02T14:50:55.493 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_running_recv_time": 0.058819298, 2024-04-02T14:50:55.493 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_running_fast_dispatch_time": 0.013122873, 2024-04-02T14:50:55.493 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_send_messages_queue_lat": { 2024-04-02T14:50:55.493 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 302, 2024-04-02T14:50:55.493 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.068465551, 2024-04-02T14:50:55.493 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000226707 2024-04-02T14:50:55.493 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:55.493 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_handle_ack_lat": { 2024-04-02T14:50:55.493 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 0, 2024-04-02T14:50:55.494 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.000000000, 2024-04-02T14:50:55.494 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000000000 2024-04-02T14:50:55.494 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:55.494 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_recv_encrypted_bytes": 1759104, 2024-04-02T14:50:55.494 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_send_encrypted_bytes": 185184 2024-04-02T14:50:55.494 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:55.494 INFO:teuthology.orchestra.run.smithi090.stdout: "cct": { 2024-04-02T14:50:55.494 INFO:teuthology.orchestra.run.smithi090.stdout: "total_workers": 0, 2024-04-02T14:50:55.494 INFO:teuthology.orchestra.run.smithi090.stdout: "unhealthy_workers": 0 2024-04-02T14:50:55.494 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:55.494 INFO:teuthology.orchestra.run.smithi090.stdout: "mempool": { 2024-04-02T14:50:55.494 INFO:teuthology.orchestra.run.smithi090.stdout: "bloom_filter_bytes": 0, 2024-04-02T14:50:55.494 INFO:teuthology.orchestra.run.smithi090.stdout: "bloom_filter_items": 0, 2024-04-02T14:50:55.495 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_alloc_bytes": 0, 2024-04-02T14:50:55.495 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_alloc_items": 0, 2024-04-02T14:50:55.495 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_cache_data_bytes": 0, 2024-04-02T14:50:55.495 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_cache_data_items": 0, 2024-04-02T14:50:55.495 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-02T14:50:55.495 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_cache_onode_items": 0, 2024-04-02T14:50:55.495 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-02T14:50:55.495 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_cache_meta_items": 0, 2024-04-02T14:50:55.495 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_cache_other_bytes": 0, 2024-04-02T14:50:55.495 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_cache_other_items": 0, 2024-04-02T14:50:55.495 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-02T14:50:55.495 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_cache_buffer_items": 0, 2024-04-02T14:50:55.495 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_extent_bytes": 0, 2024-04-02T14:50:55.496 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_extent_items": 0, 2024-04-02T14:50:55.496 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_blob_bytes": 0, 2024-04-02T14:50:55.496 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_blob_items": 0, 2024-04-02T14:50:55.496 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-02T14:50:55.496 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_shared_blob_items": 0, 2024-04-02T14:50:55.496 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-02T14:50:55.496 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_inline_bl_items": 0, 2024-04-02T14:50:55.496 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_fsck_bytes": 0, 2024-04-02T14:50:55.496 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_fsck_items": 0, 2024-04-02T14:50:55.496 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_txc_bytes": 0, 2024-04-02T14:50:55.496 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_txc_items": 0, 2024-04-02T14:50:55.496 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-02T14:50:55.496 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_writing_deferred_items": 0, 2024-04-02T14:50:55.496 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_writing_bytes": 0, 2024-04-02T14:50:55.497 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_writing_items": 0, 2024-04-02T14:50:55.497 INFO:teuthology.orchestra.run.smithi090.stdout: "bluefs_bytes": 0, 2024-04-02T14:50:55.497 INFO:teuthology.orchestra.run.smithi090.stdout: "bluefs_items": 0, 2024-04-02T14:50:55.497 INFO:teuthology.orchestra.run.smithi090.stdout: "bluefs_file_reader_bytes": 0, 2024-04-02T14:50:55.497 INFO:teuthology.orchestra.run.smithi090.stdout: "bluefs_file_reader_items": 0, 2024-04-02T14:50:55.497 INFO:teuthology.orchestra.run.smithi090.stdout: "bluefs_file_writer_bytes": 0, 2024-04-02T14:50:55.497 INFO:teuthology.orchestra.run.smithi090.stdout: "bluefs_file_writer_items": 0, 2024-04-02T14:50:55.497 INFO:teuthology.orchestra.run.smithi090.stdout: "buffer_anon_bytes": 149011, 2024-04-02T14:50:55.497 INFO:teuthology.orchestra.run.smithi090.stdout: "buffer_anon_items": 30, 2024-04-02T14:50:55.497 INFO:teuthology.orchestra.run.smithi090.stdout: "buffer_meta_bytes": 0, 2024-04-02T14:50:55.497 INFO:teuthology.orchestra.run.smithi090.stdout: "buffer_meta_items": 0, 2024-04-02T14:50:55.497 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_bytes": 0, 2024-04-02T14:50:55.497 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_items": 0, 2024-04-02T14:50:55.497 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_mapbl_bytes": 0, 2024-04-02T14:50:55.497 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_mapbl_items": 0, 2024-04-02T14:50:55.498 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_pglog_bytes": 0, 2024-04-02T14:50:55.498 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_pglog_items": 0, 2024-04-02T14:50:55.498 INFO:teuthology.orchestra.run.smithi090.stdout: "osdmap_bytes": 0, 2024-04-02T14:50:55.498 INFO:teuthology.orchestra.run.smithi090.stdout: "osdmap_items": 0, 2024-04-02T14:50:55.498 INFO:teuthology.orchestra.run.smithi090.stdout: "osdmap_mapping_bytes": 0, 2024-04-02T14:50:55.498 INFO:teuthology.orchestra.run.smithi090.stdout: "osdmap_mapping_items": 0, 2024-04-02T14:50:55.498 INFO:teuthology.orchestra.run.smithi090.stdout: "pgmap_bytes": 0, 2024-04-02T14:50:55.498 INFO:teuthology.orchestra.run.smithi090.stdout: "pgmap_items": 0, 2024-04-02T14:50:55.498 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_co_bytes": 928, 2024-04-02T14:50:55.498 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_co_items": 2, 2024-04-02T14:50:55.498 INFO:teuthology.orchestra.run.smithi090.stdout: "unittest_1_bytes": 0, 2024-04-02T14:50:55.498 INFO:teuthology.orchestra.run.smithi090.stdout: "unittest_1_items": 0, 2024-04-02T14:50:55.498 INFO:teuthology.orchestra.run.smithi090.stdout: "unittest_2_bytes": 0, 2024-04-02T14:50:55.499 INFO:teuthology.orchestra.run.smithi090.stdout: "unittest_2_items": 0 2024-04-02T14:50:55.499 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:55.499 INFO:teuthology.orchestra.run.smithi090.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-02T14:50:55.499 INFO:teuthology.orchestra.run.smithi090.stdout: "val": 0, 2024-04-02T14:50:55.499 INFO:teuthology.orchestra.run.smithi090.stdout: "max": 104857600, 2024-04-02T14:50:55.499 INFO:teuthology.orchestra.run.smithi090.stdout: "get_started": 0, 2024-04-02T14:50:55.499 INFO:teuthology.orchestra.run.smithi090.stdout: "get": 315, 2024-04-02T14:50:55.499 INFO:teuthology.orchestra.run.smithi090.stdout: "get_sum": 1712022, 2024-04-02T14:50:55.499 INFO:teuthology.orchestra.run.smithi090.stdout: "get_or_fail_fail": 0, 2024-04-02T14:50:55.499 INFO:teuthology.orchestra.run.smithi090.stdout: "get_or_fail_success": 315, 2024-04-02T14:50:55.499 INFO:teuthology.orchestra.run.smithi090.stdout: "take": 0, 2024-04-02T14:50:55.499 INFO:teuthology.orchestra.run.smithi090.stdout: "take_sum": 0, 2024-04-02T14:50:55.499 INFO:teuthology.orchestra.run.smithi090.stdout: "put": 315, 2024-04-02T14:50:55.499 INFO:teuthology.orchestra.run.smithi090.stdout: "put_sum": 1712022, 2024-04-02T14:50:55.500 INFO:teuthology.orchestra.run.smithi090.stdout: "wait": { 2024-04-02T14:50:55.500 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 0, 2024-04-02T14:50:55.500 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.000000000, 2024-04-02T14:50:55.500 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000000000 2024-04-02T14:50:55.500 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:55.500 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:55.500 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-02T14:50:56.310 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.a 2024-04-02T14:50:56.311 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.a 2024-04-02T14:50:56.311 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.a 2024-04-02T14:50:56.311 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.a 2024-04-02T14:50:56.311 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.a 2024-04-02T14:50:56.311 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.a 2024-04-02T14:50:56.311 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.a 2024-04-02T14:50:56.311 DEBUG:tasks.check_counter:Getting stats from d 2024-04-02T14:50:56.311 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:191d963ac9de7322da8aff20b567b3fa458e2ef6 shell --fsid 6449448a-f0fc-11ee-b647-cb9ed24678a4 -- ceph daemon mds.d perf dump 2024-04-02T14:50:56.602 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:50:56 smithi092 ceph-mon[36497]: pgmap v502: 97 pgs: 97 active+clean; 4.3 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 164 KiB/s rd, 5.2 MiB/s wr, 514 op/s 2024-04-02T14:50:56.678 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:50:56 smithi177 ceph-mon[36431]: pgmap v502: 97 pgs: 97 active+clean; 4.3 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 164 KiB/s rd, 5.2 MiB/s wr, 514 op/s 2024-04-02T14:50:56.679 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:50:56 smithi090 ceph-mon[31674]: pgmap v502: 97 pgs: 97 active+clean; 4.3 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 164 KiB/s rd, 5.2 MiB/s wr, 514 op/s 2024-04-02T14:50:56.730 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T14:50:57.678 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:50:57 smithi177 ceph-mon[36431]: pgmap v503: 97 pgs: 97 active+clean; 4.3 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 164 KiB/s rd, 1.8 MiB/s wr, 178 op/s 2024-04-02T14:50:57.680 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:50:57 smithi090 ceph-mon[31674]: pgmap v503: 97 pgs: 97 active+clean; 4.3 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 164 KiB/s rd, 1.8 MiB/s wr, 178 op/s 2024-04-02T14:50:57.852 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:50:57 smithi092 ceph-mon[36497]: pgmap v503: 97 pgs: 97 active+clean; 4.3 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 164 KiB/s rd, 1.8 MiB/s wr, 178 op/s 2024-04-02T14:50:58.963 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-02T14:50:58.963 INFO:teuthology.orchestra.run.smithi090.stdout: "AsyncMessenger::Worker-0": { 2024-04-02T14:50:58.963 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_recv_messages": 0, 2024-04-02T14:50:58.963 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_send_messages": 2, 2024-04-02T14:50:58.963 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_recv_bytes": 0, 2024-04-02T14:50:58.963 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_send_bytes": 0, 2024-04-02T14:50:58.963 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_created_connections": 4, 2024-04-02T14:50:58.964 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_active_connections": 0, 2024-04-02T14:50:58.964 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_running_total_time": 0.003631762, 2024-04-02T14:50:58.964 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_running_send_time": 0.000000000, 2024-04-02T14:50:58.964 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_running_recv_time": 0.000887780, 2024-04-02T14:50:58.964 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-02T14:50:58.964 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_send_messages_queue_lat": { 2024-04-02T14:50:58.964 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 0, 2024-04-02T14:50:58.964 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.000000000, 2024-04-02T14:50:58.964 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000000000 2024-04-02T14:50:58.964 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:58.964 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_handle_ack_lat": { 2024-04-02T14:50:58.964 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 0, 2024-04-02T14:50:58.965 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.000000000, 2024-04-02T14:50:58.965 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000000000 2024-04-02T14:50:58.965 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:58.965 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-02T14:50:58.965 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_send_encrypted_bytes": 0 2024-04-02T14:50:58.965 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:58.965 INFO:teuthology.orchestra.run.smithi090.stdout: "AsyncMessenger::Worker-1": { 2024-04-02T14:50:58.965 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_recv_messages": 316, 2024-04-02T14:50:58.965 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_send_messages": 305, 2024-04-02T14:50:58.965 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_recv_bytes": 1616928, 2024-04-02T14:50:58.965 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_send_bytes": 188560, 2024-04-02T14:50:58.965 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_created_connections": 2, 2024-04-02T14:50:58.966 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_active_connections": 1, 2024-04-02T14:50:58.966 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_running_total_time": 0.111826789, 2024-04-02T14:50:58.966 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_running_send_time": 0.041931451, 2024-04-02T14:50:58.966 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_running_recv_time": 0.071639043, 2024-04-02T14:50:58.966 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_running_fast_dispatch_time": 0.011476974, 2024-04-02T14:50:58.966 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_send_messages_queue_lat": { 2024-04-02T14:50:58.966 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 304, 2024-04-02T14:50:58.966 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.046283657, 2024-04-02T14:50:58.966 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000152248 2024-04-02T14:50:58.966 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:58.966 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_handle_ack_lat": { 2024-04-02T14:50:58.966 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 0, 2024-04-02T14:50:58.967 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.000000000, 2024-04-02T14:50:58.967 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000000000 2024-04-02T14:50:58.967 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:58.967 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_recv_encrypted_bytes": 1616928, 2024-04-02T14:50:58.967 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_send_encrypted_bytes": 188560 2024-04-02T14:50:58.967 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:58.967 INFO:teuthology.orchestra.run.smithi090.stdout: "AsyncMessenger::Worker-2": { 2024-04-02T14:50:58.967 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_recv_messages": 7, 2024-04-02T14:50:58.967 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_send_messages": 237, 2024-04-02T14:50:58.967 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_recv_bytes": 3552, 2024-04-02T14:50:58.967 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_send_bytes": 275664, 2024-04-02T14:50:58.967 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_created_connections": 6, 2024-04-02T14:50:58.968 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_active_connections": 3, 2024-04-02T14:50:58.968 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_running_total_time": 0.042432608, 2024-04-02T14:50:58.968 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_running_send_time": 0.030614136, 2024-04-02T14:50:58.968 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_running_recv_time": 0.006661787, 2024-04-02T14:50:58.968 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-02T14:50:58.968 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_send_messages_queue_lat": { 2024-04-02T14:50:58.968 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 235, 2024-04-02T14:50:58.968 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.045088191, 2024-04-02T14:50:58.968 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000191864 2024-04-02T14:50:58.968 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:58.968 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_handle_ack_lat": { 2024-04-02T14:50:58.968 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 0, 2024-04-02T14:50:58.968 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.000000000, 2024-04-02T14:50:58.969 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000000000 2024-04-02T14:50:58.969 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:58.969 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_recv_encrypted_bytes": 3552, 2024-04-02T14:50:58.969 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_send_encrypted_bytes": 275664 2024-04-02T14:50:58.969 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:58.969 INFO:teuthology.orchestra.run.smithi090.stdout: "cct": { 2024-04-02T14:50:58.969 INFO:teuthology.orchestra.run.smithi090.stdout: "total_workers": 0, 2024-04-02T14:50:58.969 INFO:teuthology.orchestra.run.smithi090.stdout: "unhealthy_workers": 0 2024-04-02T14:50:58.969 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:58.969 INFO:teuthology.orchestra.run.smithi090.stdout: "mempool": { 2024-04-02T14:50:58.969 INFO:teuthology.orchestra.run.smithi090.stdout: "bloom_filter_bytes": 0, 2024-04-02T14:50:58.970 INFO:teuthology.orchestra.run.smithi090.stdout: "bloom_filter_items": 0, 2024-04-02T14:50:58.970 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_alloc_bytes": 0, 2024-04-02T14:50:58.970 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_alloc_items": 0, 2024-04-02T14:50:58.970 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_cache_data_bytes": 0, 2024-04-02T14:50:58.970 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_cache_data_items": 0, 2024-04-02T14:50:58.970 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-02T14:50:58.970 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_cache_onode_items": 0, 2024-04-02T14:50:58.970 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-02T14:50:58.970 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_cache_meta_items": 0, 2024-04-02T14:50:58.970 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_cache_other_bytes": 0, 2024-04-02T14:50:58.970 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_cache_other_items": 0, 2024-04-02T14:50:58.970 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-02T14:50:58.970 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_cache_buffer_items": 0, 2024-04-02T14:50:58.970 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_extent_bytes": 0, 2024-04-02T14:50:58.971 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_extent_items": 0, 2024-04-02T14:50:58.971 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_blob_bytes": 0, 2024-04-02T14:50:58.971 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_blob_items": 0, 2024-04-02T14:50:58.971 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-02T14:50:58.971 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_shared_blob_items": 0, 2024-04-02T14:50:58.971 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-02T14:50:58.971 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_inline_bl_items": 0, 2024-04-02T14:50:58.971 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_fsck_bytes": 0, 2024-04-02T14:50:58.971 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_fsck_items": 0, 2024-04-02T14:50:58.971 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_txc_bytes": 0, 2024-04-02T14:50:58.971 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_txc_items": 0, 2024-04-02T14:50:58.971 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-02T14:50:58.971 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_writing_deferred_items": 0, 2024-04-02T14:50:58.971 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_writing_bytes": 0, 2024-04-02T14:50:58.972 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_writing_items": 0, 2024-04-02T14:50:58.972 INFO:teuthology.orchestra.run.smithi090.stdout: "bluefs_bytes": 0, 2024-04-02T14:50:58.972 INFO:teuthology.orchestra.run.smithi090.stdout: "bluefs_items": 0, 2024-04-02T14:50:58.972 INFO:teuthology.orchestra.run.smithi090.stdout: "bluefs_file_reader_bytes": 0, 2024-04-02T14:50:58.972 INFO:teuthology.orchestra.run.smithi090.stdout: "bluefs_file_reader_items": 0, 2024-04-02T14:50:58.972 INFO:teuthology.orchestra.run.smithi090.stdout: "bluefs_file_writer_bytes": 0, 2024-04-02T14:50:58.972 INFO:teuthology.orchestra.run.smithi090.stdout: "bluefs_file_writer_items": 0, 2024-04-02T14:50:58.972 INFO:teuthology.orchestra.run.smithi090.stdout: "buffer_anon_bytes": 147930, 2024-04-02T14:50:58.972 INFO:teuthology.orchestra.run.smithi090.stdout: "buffer_anon_items": 28, 2024-04-02T14:50:58.972 INFO:teuthology.orchestra.run.smithi090.stdout: "buffer_meta_bytes": 0, 2024-04-02T14:50:58.972 INFO:teuthology.orchestra.run.smithi090.stdout: "buffer_meta_items": 0, 2024-04-02T14:50:58.972 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_bytes": 0, 2024-04-02T14:50:58.972 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_items": 0, 2024-04-02T14:50:58.972 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_mapbl_bytes": 0, 2024-04-02T14:50:58.972 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_mapbl_items": 0, 2024-04-02T14:50:58.973 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_pglog_bytes": 0, 2024-04-02T14:50:58.973 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_pglog_items": 0, 2024-04-02T14:50:58.973 INFO:teuthology.orchestra.run.smithi090.stdout: "osdmap_bytes": 0, 2024-04-02T14:50:58.973 INFO:teuthology.orchestra.run.smithi090.stdout: "osdmap_items": 0, 2024-04-02T14:50:58.973 INFO:teuthology.orchestra.run.smithi090.stdout: "osdmap_mapping_bytes": 0, 2024-04-02T14:50:58.973 INFO:teuthology.orchestra.run.smithi090.stdout: "osdmap_mapping_items": 0, 2024-04-02T14:50:58.973 INFO:teuthology.orchestra.run.smithi090.stdout: "pgmap_bytes": 0, 2024-04-02T14:50:58.973 INFO:teuthology.orchestra.run.smithi090.stdout: "pgmap_items": 0, 2024-04-02T14:50:58.973 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_co_bytes": 928, 2024-04-02T14:50:58.973 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_co_items": 2, 2024-04-02T14:50:58.973 INFO:teuthology.orchestra.run.smithi090.stdout: "unittest_1_bytes": 0, 2024-04-02T14:50:58.973 INFO:teuthology.orchestra.run.smithi090.stdout: "unittest_1_items": 0, 2024-04-02T14:50:58.973 INFO:teuthology.orchestra.run.smithi090.stdout: "unittest_2_bytes": 0, 2024-04-02T14:50:58.973 INFO:teuthology.orchestra.run.smithi090.stdout: "unittest_2_items": 0 2024-04-02T14:50:58.973 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:50:58.974 INFO:teuthology.orchestra.run.smithi090.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-02T14:50:58.974 INFO:teuthology.orchestra.run.smithi090.stdout: "val": 0, 2024-04-02T14:50:58.974 INFO:teuthology.orchestra.run.smithi090.stdout: "max": 104857600, 2024-04-02T14:50:58.974 INFO:teuthology.orchestra.run.smithi090.stdout: "get_started": 0, 2024-04-02T14:50:58.974 INFO:teuthology.orchestra.run.smithi090.stdout: "get": 319, 2024-04-02T14:50:58.974 INFO:teuthology.orchestra.run.smithi090.stdout: "get_sum": 1572330, 2024-04-02T14:50:58.974 INFO:teuthology.orchestra.run.smithi090.stdout: "get_or_fail_fail": 0, 2024-04-02T14:50:58.974 INFO:teuthology.orchestra.run.smithi090.stdout: "get_or_fail_success": 319, 2024-04-02T14:50:58.974 INFO:teuthology.orchestra.run.smithi090.stdout: "take": 0, 2024-04-02T14:50:58.974 INFO:teuthology.orchestra.run.smithi090.stdout: "take_sum": 0, 2024-04-02T14:50:58.974 INFO:teuthology.orchestra.run.smithi090.stdout: "put": 319, 2024-04-02T14:50:58.974 INFO:teuthology.orchestra.run.smithi090.stdout: "put_sum": 1572330, 2024-04-02T14:50:58.974 INFO:teuthology.orchestra.run.smithi090.stdout: "wait": { 2024-04-02T14:50:58.975 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 0, 2024-04-02T14:50:58.975 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.000000000, 2024-04-02T14:50:58.975 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000000000 2024-04-02T14:50:58.975 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:58.975 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:50:58.975 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-02T14:50:59.795 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.d 2024-04-02T14:50:59.795 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.d 2024-04-02T14:50:59.795 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.d 2024-04-02T14:50:59.795 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.d 2024-04-02T14:50:59.795 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.d 2024-04-02T14:50:59.795 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.d 2024-04-02T14:50:59.795 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.d 2024-04-02T14:50:59.796 DEBUG:tasks.check_counter:Getting stats from g 2024-04-02T14:50:59.796 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:191d963ac9de7322da8aff20b567b3fa458e2ef6 shell --fsid 6449448a-f0fc-11ee-b647-cb9ed24678a4 -- ceph daemon mds.g perf dump 2024-04-02T14:51:00.102 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:50:59 smithi092 ceph-mon[36497]: pgmap v504: 97 pgs: 97 active+clean; 4.3 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 14 KiB/s wr, 11 op/s 2024-04-02T14:51:00.178 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:50:59 smithi177 ceph-mon[36431]: pgmap v504: 97 pgs: 97 active+clean; 4.3 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 14 KiB/s wr, 11 op/s 2024-04-02T14:51:00.179 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:50:59 smithi090 ceph-mon[31674]: pgmap v504: 97 pgs: 97 active+clean; 4.3 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 14 KiB/s wr, 11 op/s 2024-04-02T14:51:00.257 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T14:51:02.102 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:51:01 smithi092 ceph-mon[36497]: pgmap v505: 97 pgs: 97 active+clean; 4.3 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 33 KiB/s wr, 18 op/s 2024-04-02T14:51:02.178 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:51:01 smithi177 ceph-mon[36431]: pgmap v505: 97 pgs: 97 active+clean; 4.3 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 33 KiB/s wr, 18 op/s 2024-04-02T14:51:02.180 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:51:01 smithi090 ceph-mon[31674]: pgmap v505: 97 pgs: 97 active+clean; 4.3 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 33 KiB/s wr, 18 op/s 2024-04-02T14:51:02.416 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-02T14:51:02.416 INFO:teuthology.orchestra.run.smithi090.stdout: "AsyncMessenger::Worker-0": { 2024-04-02T14:51:02.417 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_recv_messages": 44, 2024-04-02T14:51:02.417 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_send_messages": 241, 2024-04-02T14:51:02.417 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_recv_bytes": 12209, 2024-04-02T14:51:02.417 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_send_bytes": 631633, 2024-04-02T14:51:02.417 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_created_connections": 21, 2024-04-02T14:51:02.417 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_active_connections": 4, 2024-04-02T14:51:02.417 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_running_total_time": 0.064019381, 2024-04-02T14:51:02.417 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_running_send_time": 0.029277677, 2024-04-02T14:51:02.417 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_running_recv_time": 0.013089614, 2024-04-02T14:51:02.417 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_running_fast_dispatch_time": 0.000990321, 2024-04-02T14:51:02.417 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_send_messages_queue_lat": { 2024-04-02T14:51:02.417 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 237, 2024-04-02T14:51:02.417 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.066702269, 2024-04-02T14:51:02.417 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000281444 2024-04-02T14:51:02.418 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:51:02.418 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_handle_ack_lat": { 2024-04-02T14:51:02.418 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 15, 2024-04-02T14:51:02.418 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.000002499, 2024-04-02T14:51:02.418 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000000166 2024-04-02T14:51:02.418 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:51:02.418 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_recv_encrypted_bytes": 4320, 2024-04-02T14:51:02.418 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_send_encrypted_bytes": 524000 2024-04-02T14:51:02.418 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:51:02.418 INFO:teuthology.orchestra.run.smithi090.stdout: "AsyncMessenger::Worker-1": { 2024-04-02T14:51:02.418 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_recv_messages": 3790, 2024-04-02T14:51:02.418 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_send_messages": 1957, 2024-04-02T14:51:02.419 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_recv_bytes": 1028506, 2024-04-02T14:51:02.419 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_send_bytes": 1169209, 2024-04-02T14:51:02.419 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_created_connections": 52, 2024-04-02T14:51:02.419 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_active_connections": 4, 2024-04-02T14:51:02.419 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_running_total_time": 0.411384283, 2024-04-02T14:51:02.419 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_running_send_time": 0.149531389, 2024-04-02T14:51:02.419 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_running_recv_time": 0.419085639, 2024-04-02T14:51:02.419 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_running_fast_dispatch_time": 0.008765529, 2024-04-02T14:51:02.419 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_send_messages_queue_lat": { 2024-04-02T14:51:02.419 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 1957, 2024-04-02T14:51:02.419 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.365704222, 2024-04-02T14:51:02.419 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000186869 2024-04-02T14:51:02.419 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:51:02.419 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_handle_ack_lat": { 2024-04-02T14:51:02.419 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 5260, 2024-04-02T14:51:02.420 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.003003417, 2024-04-02T14:51:02.420 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000000570 2024-04-02T14:51:02.420 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:51:02.420 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_recv_encrypted_bytes": 1088, 2024-04-02T14:51:02.420 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_send_encrypted_bytes": 137648 2024-04-02T14:51:02.420 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:51:02.420 INFO:teuthology.orchestra.run.smithi090.stdout: "AsyncMessenger::Worker-2": { 2024-04-02T14:51:02.420 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_recv_messages": 486, 2024-04-02T14:51:02.420 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_send_messages": 493, 2024-04-02T14:51:02.420 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_recv_bytes": 1711273, 2024-04-02T14:51:02.420 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_send_bytes": 365233, 2024-04-02T14:51:02.420 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_created_connections": 19, 2024-04-02T14:51:02.420 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_active_connections": 6, 2024-04-02T14:51:02.420 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_running_total_time": 0.209953729, 2024-04-02T14:51:02.420 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_running_send_time": 0.089743010, 2024-04-02T14:51:02.421 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_running_recv_time": 0.138179356, 2024-04-02T14:51:02.421 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_running_fast_dispatch_time": 0.011725856, 2024-04-02T14:51:02.421 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_send_messages_queue_lat": { 2024-04-02T14:51:02.421 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 492, 2024-04-02T14:51:02.421 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.086612556, 2024-04-02T14:51:02.421 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000176041 2024-04-02T14:51:02.421 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:51:02.421 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_handle_ack_lat": { 2024-04-02T14:51:02.421 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 228, 2024-04-02T14:51:02.421 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.000137567, 2024-04-02T14:51:02.421 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000000603 2024-04-02T14:51:02.421 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:51:02.421 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_recv_encrypted_bytes": 1707312, 2024-04-02T14:51:02.422 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_send_encrypted_bytes": 215504 2024-04-02T14:51:02.422 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:51:02.422 INFO:teuthology.orchestra.run.smithi090.stdout: "cct": { 2024-04-02T14:51:02.422 INFO:teuthology.orchestra.run.smithi090.stdout: "total_workers": 1, 2024-04-02T14:51:02.422 INFO:teuthology.orchestra.run.smithi090.stdout: "unhealthy_workers": 0 2024-04-02T14:51:02.422 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:51:02.422 INFO:teuthology.orchestra.run.smithi090.stdout: "finisher-MDSRank": { 2024-04-02T14:51:02.422 INFO:teuthology.orchestra.run.smithi090.stdout: "queue_len": 0, 2024-04-02T14:51:02.422 INFO:teuthology.orchestra.run.smithi090.stdout: "complete_latency": { 2024-04-02T14:51:02.422 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 53, 2024-04-02T14:51:02.422 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.002323933, 2024-04-02T14:51:02.422 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000043847 2024-04-02T14:51:02.422 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:51:02.423 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:51:02.423 INFO:teuthology.orchestra.run.smithi090.stdout: "finisher-PurgeQueue": { 2024-04-02T14:51:02.423 INFO:teuthology.orchestra.run.smithi090.stdout: "queue_len": 0, 2024-04-02T14:51:02.423 INFO:teuthology.orchestra.run.smithi090.stdout: "complete_latency": { 2024-04-02T14:51:02.423 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 2, 2024-04-02T14:51:02.423 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.000007157, 2024-04-02T14:51:02.423 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000003578 2024-04-02T14:51:02.423 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:51:02.423 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:51:02.423 INFO:teuthology.orchestra.run.smithi090.stdout: "mds": { 2024-04-02T14:51:02.423 INFO:teuthology.orchestra.run.smithi090.stdout: "request": 82, 2024-04-02T14:51:02.423 INFO:teuthology.orchestra.run.smithi090.stdout: "reply": 0, 2024-04-02T14:51:02.423 INFO:teuthology.orchestra.run.smithi090.stdout: "reply_latency": { 2024-04-02T14:51:02.423 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 0, 2024-04-02T14:51:02.424 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.000000000, 2024-04-02T14:51:02.424 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000000000 2024-04-02T14:51:02.424 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:51:02.424 INFO:teuthology.orchestra.run.smithi090.stdout: "slow_reply": 0, 2024-04-02T14:51:02.424 INFO:teuthology.orchestra.run.smithi090.stdout: "forward": 82, 2024-04-02T14:51:02.424 INFO:teuthology.orchestra.run.smithi090.stdout: "dir_fetch_complete": 0, 2024-04-02T14:51:02.424 INFO:teuthology.orchestra.run.smithi090.stdout: "dir_fetch_keys": 0, 2024-04-02T14:51:02.424 INFO:teuthology.orchestra.run.smithi090.stdout: "dir_commit": 11, 2024-04-02T14:51:02.424 INFO:teuthology.orchestra.run.smithi090.stdout: "dir_split": 0, 2024-04-02T14:51:02.424 INFO:teuthology.orchestra.run.smithi090.stdout: "dir_merge": 0, 2024-04-02T14:51:02.424 INFO:teuthology.orchestra.run.smithi090.stdout: "inodes": 34, 2024-04-02T14:51:02.424 INFO:teuthology.orchestra.run.smithi090.stdout: "inodes_top": 11, 2024-04-02T14:51:02.424 INFO:teuthology.orchestra.run.smithi090.stdout: "inodes_bottom": 23, 2024-04-02T14:51:02.424 INFO:teuthology.orchestra.run.smithi090.stdout: "inodes_pin_tail": 0, 2024-04-02T14:51:02.424 INFO:teuthology.orchestra.run.smithi090.stdout: "inodes_pinned": 18, 2024-04-02T14:51:02.425 INFO:teuthology.orchestra.run.smithi090.stdout: "inodes_expired": 4, 2024-04-02T14:51:02.425 INFO:teuthology.orchestra.run.smithi090.stdout: "inodes_with_caps": 0, 2024-04-02T14:51:02.425 INFO:teuthology.orchestra.run.smithi090.stdout: "caps": 0, 2024-04-02T14:51:02.425 INFO:teuthology.orchestra.run.smithi090.stdout: "subtrees": 3, 2024-04-02T14:51:02.425 INFO:teuthology.orchestra.run.smithi090.stdout: "traverse": 145, 2024-04-02T14:51:02.425 INFO:teuthology.orchestra.run.smithi090.stdout: "traverse_hit": 42, 2024-04-02T14:51:02.425 INFO:teuthology.orchestra.run.smithi090.stdout: "traverse_forward": 82, 2024-04-02T14:51:02.425 INFO:teuthology.orchestra.run.smithi090.stdout: "traverse_discover": 21, 2024-04-02T14:51:02.425 INFO:teuthology.orchestra.run.smithi090.stdout: "traverse_dir_fetch": 0, 2024-04-02T14:51:02.425 INFO:teuthology.orchestra.run.smithi090.stdout: "traverse_remote_ino": 0, 2024-04-02T14:51:02.425 INFO:teuthology.orchestra.run.smithi090.stdout: "traverse_lock": 0, 2024-04-02T14:51:02.425 INFO:teuthology.orchestra.run.smithi090.stdout: "load_cent": 0, 2024-04-02T14:51:02.425 INFO:teuthology.orchestra.run.smithi090.stdout: "q": 0, 2024-04-02T14:51:02.425 INFO:teuthology.orchestra.run.smithi090.stdout: "exported": 0, 2024-04-02T14:51:02.426 INFO:teuthology.orchestra.run.smithi090.stdout: "exported_inodes": 0, 2024-04-02T14:51:02.426 INFO:teuthology.orchestra.run.smithi090.stdout: "imported": 0, 2024-04-02T14:51:02.426 INFO:teuthology.orchestra.run.smithi090.stdout: "imported_inodes": 0, 2024-04-02T14:51:02.426 INFO:teuthology.orchestra.run.smithi090.stdout: "openino_dir_fetch": 0, 2024-04-02T14:51:02.426 INFO:teuthology.orchestra.run.smithi090.stdout: "openino_backtrace_fetch": 0, 2024-04-02T14:51:02.426 INFO:teuthology.orchestra.run.smithi090.stdout: "openino_peer_discover": 0, 2024-04-02T14:51:02.426 INFO:teuthology.orchestra.run.smithi090.stdout: "root_rfiles": 1026, 2024-04-02T14:51:02.426 INFO:teuthology.orchestra.run.smithi090.stdout: "root_rbytes": 1709677761, 2024-04-02T14:51:02.426 INFO:teuthology.orchestra.run.smithi090.stdout: "root_rsnaps": 6, 2024-04-02T14:51:02.426 INFO:teuthology.orchestra.run.smithi090.stdout: "scrub_backtrace_fetch": 0, 2024-04-02T14:51:02.426 INFO:teuthology.orchestra.run.smithi090.stdout: "scrub_set_tag": 0, 2024-04-02T14:51:02.426 INFO:teuthology.orchestra.run.smithi090.stdout: "scrub_backtrace_repaired": 0, 2024-04-02T14:51:02.426 INFO:teuthology.orchestra.run.smithi090.stdout: "scrub_inotable_repaired": 0, 2024-04-02T14:51:02.426 INFO:teuthology.orchestra.run.smithi090.stdout: "scrub_dir_inodes": 0, 2024-04-02T14:51:02.427 INFO:teuthology.orchestra.run.smithi090.stdout: "scrub_dir_base_inodes": 0, 2024-04-02T14:51:02.427 INFO:teuthology.orchestra.run.smithi090.stdout: "scrub_dirfrag_rstats": 0, 2024-04-02T14:51:02.427 INFO:teuthology.orchestra.run.smithi090.stdout: "scrub_file_inodes": 0, 2024-04-02T14:51:02.427 INFO:teuthology.orchestra.run.smithi090.stdout: "handle_inode_file_caps": 0, 2024-04-02T14:51:02.427 INFO:teuthology.orchestra.run.smithi090.stdout: "ceph_cap_op_revoke": 0, 2024-04-02T14:51:02.427 INFO:teuthology.orchestra.run.smithi090.stdout: "ceph_cap_op_grant": 0, 2024-04-02T14:51:02.427 INFO:teuthology.orchestra.run.smithi090.stdout: "ceph_cap_op_trunc": 0, 2024-04-02T14:51:02.427 INFO:teuthology.orchestra.run.smithi090.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-02T14:51:02.427 INFO:teuthology.orchestra.run.smithi090.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-02T14:51:02.427 INFO:teuthology.orchestra.run.smithi090.stdout: "handle_client_caps": 0, 2024-04-02T14:51:02.427 INFO:teuthology.orchestra.run.smithi090.stdout: "handle_client_caps_dirty": 0, 2024-04-02T14:51:02.427 INFO:teuthology.orchestra.run.smithi090.stdout: "handle_client_cap_release": 0, 2024-04-02T14:51:02.427 INFO:teuthology.orchestra.run.smithi090.stdout: "process_request_cap_release": 0 2024-04-02T14:51:02.427 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:51:02.427 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_cache": { 2024-04-02T14:51:02.428 INFO:teuthology.orchestra.run.smithi090.stdout: "dir_update": 0, 2024-04-02T14:51:02.428 INFO:teuthology.orchestra.run.smithi090.stdout: "dir_update_receipt": 105, 2024-04-02T14:51:02.428 INFO:teuthology.orchestra.run.smithi090.stdout: "dir_try_discover": 63, 2024-04-02T14:51:02.428 INFO:teuthology.orchestra.run.smithi090.stdout: "dir_send_discover": 63, 2024-04-02T14:51:02.428 INFO:teuthology.orchestra.run.smithi090.stdout: "dir_handle_discover": 0, 2024-04-02T14:51:02.428 INFO:teuthology.orchestra.run.smithi090.stdout: "num_strays": 0, 2024-04-02T14:51:02.428 INFO:teuthology.orchestra.run.smithi090.stdout: "num_strays_delayed": 0, 2024-04-02T14:51:02.428 INFO:teuthology.orchestra.run.smithi090.stdout: "num_strays_enqueuing": 0, 2024-04-02T14:51:02.428 INFO:teuthology.orchestra.run.smithi090.stdout: "strays_created": 0, 2024-04-02T14:51:02.428 INFO:teuthology.orchestra.run.smithi090.stdout: "strays_enqueued": 0, 2024-04-02T14:51:02.428 INFO:teuthology.orchestra.run.smithi090.stdout: "strays_reintegrated": 0, 2024-04-02T14:51:02.428 INFO:teuthology.orchestra.run.smithi090.stdout: "strays_migrated": 0, 2024-04-02T14:51:02.428 INFO:teuthology.orchestra.run.smithi090.stdout: "num_recovering_processing": 0, 2024-04-02T14:51:02.428 INFO:teuthology.orchestra.run.smithi090.stdout: "num_recovering_enqueued": 0, 2024-04-02T14:51:02.428 INFO:teuthology.orchestra.run.smithi090.stdout: "num_recovering_prioritized": 0, 2024-04-02T14:51:02.429 INFO:teuthology.orchestra.run.smithi090.stdout: "recovery_started": 0, 2024-04-02T14:51:02.429 INFO:teuthology.orchestra.run.smithi090.stdout: "recovery_completed": 0, 2024-04-02T14:51:02.429 INFO:teuthology.orchestra.run.smithi090.stdout: "ireq_quiesce_path": 0, 2024-04-02T14:51:02.429 INFO:teuthology.orchestra.run.smithi090.stdout: "ireq_quiesce_inode": 0, 2024-04-02T14:51:02.429 INFO:teuthology.orchestra.run.smithi090.stdout: "ireq_enqueue_scrub": 0, 2024-04-02T14:51:02.429 INFO:teuthology.orchestra.run.smithi090.stdout: "ireq_exportdir": 0, 2024-04-02T14:51:02.429 INFO:teuthology.orchestra.run.smithi090.stdout: "ireq_flush": 0, 2024-04-02T14:51:02.429 INFO:teuthology.orchestra.run.smithi090.stdout: "ireq_fragmentdir": 0, 2024-04-02T14:51:02.429 INFO:teuthology.orchestra.run.smithi090.stdout: "ireq_fragstats": 0, 2024-04-02T14:51:02.429 INFO:teuthology.orchestra.run.smithi090.stdout: "ireq_inodestats": 0 2024-04-02T14:51:02.429 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:51:02.429 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_log": { 2024-04-02T14:51:02.429 INFO:teuthology.orchestra.run.smithi090.stdout: "evlrg": 0, 2024-04-02T14:51:02.429 INFO:teuthology.orchestra.run.smithi090.stdout: "evadd": 7, 2024-04-02T14:51:02.429 INFO:teuthology.orchestra.run.smithi090.stdout: "evex": 0, 2024-04-02T14:51:02.430 INFO:teuthology.orchestra.run.smithi090.stdout: "evtrm": 0, 2024-04-02T14:51:02.430 INFO:teuthology.orchestra.run.smithi090.stdout: "ev": 7, 2024-04-02T14:51:02.430 INFO:teuthology.orchestra.run.smithi090.stdout: "evexg": 0, 2024-04-02T14:51:02.430 INFO:teuthology.orchestra.run.smithi090.stdout: "evexd": 0, 2024-04-02T14:51:02.430 INFO:teuthology.orchestra.run.smithi090.stdout: "segadd": 2, 2024-04-02T14:51:02.430 INFO:teuthology.orchestra.run.smithi090.stdout: "segex": 0, 2024-04-02T14:51:02.430 INFO:teuthology.orchestra.run.smithi090.stdout: "segtrm": 0, 2024-04-02T14:51:02.430 INFO:teuthology.orchestra.run.smithi090.stdout: "seg": 2, 2024-04-02T14:51:02.430 INFO:teuthology.orchestra.run.smithi090.stdout: "segmjr": 2, 2024-04-02T14:51:02.430 INFO:teuthology.orchestra.run.smithi090.stdout: "segexg": 0, 2024-04-02T14:51:02.430 INFO:teuthology.orchestra.run.smithi090.stdout: "segexd": 0, 2024-04-02T14:51:02.430 INFO:teuthology.orchestra.run.smithi090.stdout: "expos": 4194304, 2024-04-02T14:51:02.430 INFO:teuthology.orchestra.run.smithi090.stdout: "wrpos": 4196262, 2024-04-02T14:51:02.430 INFO:teuthology.orchestra.run.smithi090.stdout: "rdpos": 0, 2024-04-02T14:51:02.430 INFO:teuthology.orchestra.run.smithi090.stdout: "jlat": { 2024-04-02T14:51:02.431 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 6, 2024-04-02T14:51:02.431 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.012202394, 2024-04-02T14:51:02.431 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.002033732 2024-04-02T14:51:02.431 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:51:02.431 INFO:teuthology.orchestra.run.smithi090.stdout: "replayed": 0 2024-04-02T14:51:02.431 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:51:02.431 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_mem": { 2024-04-02T14:51:02.431 INFO:teuthology.orchestra.run.smithi090.stdout: "ino": 38, 2024-04-02T14:51:02.431 INFO:teuthology.orchestra.run.smithi090.stdout: "ino+": 40, 2024-04-02T14:51:02.431 INFO:teuthology.orchestra.run.smithi090.stdout: "ino-": 2, 2024-04-02T14:51:02.431 INFO:teuthology.orchestra.run.smithi090.stdout: "dir": 65, 2024-04-02T14:51:02.431 INFO:teuthology.orchestra.run.smithi090.stdout: "dir+": 69, 2024-04-02T14:51:02.431 INFO:teuthology.orchestra.run.smithi090.stdout: "dir-": 4, 2024-04-02T14:51:02.431 INFO:teuthology.orchestra.run.smithi090.stdout: "dn": 34, 2024-04-02T14:51:02.431 INFO:teuthology.orchestra.run.smithi090.stdout: "dn+": 38, 2024-04-02T14:51:02.432 INFO:teuthology.orchestra.run.smithi090.stdout: "dn-": 4, 2024-04-02T14:51:02.432 INFO:teuthology.orchestra.run.smithi090.stdout: "cap": 0, 2024-04-02T14:51:02.432 INFO:teuthology.orchestra.run.smithi090.stdout: "cap+": 0, 2024-04-02T14:51:02.432 INFO:teuthology.orchestra.run.smithi090.stdout: "cap-": 0, 2024-04-02T14:51:02.432 INFO:teuthology.orchestra.run.smithi090.stdout: "rss": 52864, 2024-04-02T14:51:02.432 INFO:teuthology.orchestra.run.smithi090.stdout: "heap": 223608 2024-04-02T14:51:02.432 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:51:02.432 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_server": { 2024-04-02T14:51:02.432 INFO:teuthology.orchestra.run.smithi090.stdout: "dispatch_client_request": 82, 2024-04-02T14:51:02.432 INFO:teuthology.orchestra.run.smithi090.stdout: "dispatch_server_request": 0, 2024-04-02T14:51:02.432 INFO:teuthology.orchestra.run.smithi090.stdout: "handle_client_request": 82, 2024-04-02T14:51:02.432 INFO:teuthology.orchestra.run.smithi090.stdout: "handle_client_session": 21, 2024-04-02T14:51:02.432 INFO:teuthology.orchestra.run.smithi090.stdout: "handle_peer_request": 0, 2024-04-02T14:51:02.432 INFO:teuthology.orchestra.run.smithi090.stdout: "req_create_latency": { 2024-04-02T14:51:02.432 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 0, 2024-04-02T14:51:02.433 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.000000000, 2024-04-02T14:51:02.433 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000000000 2024-04-02T14:51:02.433 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:51:02.433 INFO:teuthology.orchestra.run.smithi090.stdout: "req_getattr_latency": { 2024-04-02T14:51:02.433 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 0, 2024-04-02T14:51:02.433 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.000000000, 2024-04-02T14:51:02.433 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000000000 2024-04-02T14:51:02.433 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:51:02.433 INFO:teuthology.orchestra.run.smithi090.stdout: "req_getfilelock_latency": { 2024-04-02T14:51:02.433 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 0, 2024-04-02T14:51:02.433 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.000000000, 2024-04-02T14:51:02.433 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000000000 2024-04-02T14:51:02.433 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:51:02.433 INFO:teuthology.orchestra.run.smithi090.stdout: "req_link_latency": { 2024-04-02T14:51:02.433 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 0, 2024-04-02T14:51:02.433 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.000000000, 2024-04-02T14:51:02.434 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000000000 2024-04-02T14:51:02.434 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:51:02.434 INFO:teuthology.orchestra.run.smithi090.stdout: "req_lookup_latency": { 2024-04-02T14:51:02.434 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 0, 2024-04-02T14:51:02.434 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.000000000, 2024-04-02T14:51:02.434 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000000000 2024-04-02T14:51:02.434 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:51:02.434 INFO:teuthology.orchestra.run.smithi090.stdout: "req_lookuphash_latency": { 2024-04-02T14:51:02.434 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 0, 2024-04-02T14:51:02.434 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.000000000, 2024-04-02T14:51:02.434 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000000000 2024-04-02T14:51:02.434 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:51:02.434 INFO:teuthology.orchestra.run.smithi090.stdout: "req_lookupino_latency": { 2024-04-02T14:51:02.434 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 0, 2024-04-02T14:51:02.435 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.000000000, 2024-04-02T14:51:02.435 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000000000 2024-04-02T14:51:02.435 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:51:02.435 INFO:teuthology.orchestra.run.smithi090.stdout: "req_lookupname_latency": { 2024-04-02T14:51:02.435 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 0, 2024-04-02T14:51:02.435 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.000000000, 2024-04-02T14:51:02.435 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000000000 2024-04-02T14:51:02.435 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:51:02.435 INFO:teuthology.orchestra.run.smithi090.stdout: "req_lookupparent_latency": { 2024-04-02T14:51:02.435 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 0, 2024-04-02T14:51:02.435 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.000000000, 2024-04-02T14:51:02.435 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000000000 2024-04-02T14:51:02.435 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:51:02.435 INFO:teuthology.orchestra.run.smithi090.stdout: "req_lookupsnap_latency": { 2024-04-02T14:51:02.435 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 0, 2024-04-02T14:51:02.435 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.000000000, 2024-04-02T14:51:02.436 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000000000 2024-04-02T14:51:02.436 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:51:02.436 INFO:teuthology.orchestra.run.smithi090.stdout: "req_lssnap_latency": { 2024-04-02T14:51:02.436 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 0, 2024-04-02T14:51:02.436 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.000000000, 2024-04-02T14:51:02.436 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000000000 2024-04-02T14:51:02.436 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:51:02.436 INFO:teuthology.orchestra.run.smithi090.stdout: "req_mkdir_latency": { 2024-04-02T14:51:02.436 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 0, 2024-04-02T14:51:02.437 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.000000000, 2024-04-02T14:51:02.437 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000000000 2024-04-02T14:51:02.437 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:51:02.437 INFO:teuthology.orchestra.run.smithi090.stdout: "req_mknod_latency": { 2024-04-02T14:51:02.437 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 0, 2024-04-02T14:51:02.437 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.000000000, 2024-04-02T14:51:02.437 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000000000 2024-04-02T14:51:02.437 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:51:02.437 INFO:teuthology.orchestra.run.smithi090.stdout: "req_mksnap_latency": { 2024-04-02T14:51:02.437 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 0, 2024-04-02T14:51:02.437 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.000000000, 2024-04-02T14:51:02.437 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000000000 2024-04-02T14:51:02.437 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:51:02.437 INFO:teuthology.orchestra.run.smithi090.stdout: "req_open_latency": { 2024-04-02T14:51:02.438 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 0, 2024-04-02T14:51:02.438 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.000000000, 2024-04-02T14:51:02.438 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000000000 2024-04-02T14:51:02.438 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:51:02.438 INFO:teuthology.orchestra.run.smithi090.stdout: "req_readdir_latency": { 2024-04-02T14:51:02.438 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 0, 2024-04-02T14:51:02.438 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.000000000, 2024-04-02T14:51:02.438 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000000000 2024-04-02T14:51:02.438 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:51:02.438 INFO:teuthology.orchestra.run.smithi090.stdout: "req_rename_latency": { 2024-04-02T14:51:02.438 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 0, 2024-04-02T14:51:02.438 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.000000000, 2024-04-02T14:51:02.438 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000000000 2024-04-02T14:51:02.438 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:51:02.439 INFO:teuthology.orchestra.run.smithi090.stdout: "req_renamesnap_latency": { 2024-04-02T14:51:02.439 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 0, 2024-04-02T14:51:02.439 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.000000000, 2024-04-02T14:51:02.439 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000000000 2024-04-02T14:51:02.439 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:51:02.439 INFO:teuthology.orchestra.run.smithi090.stdout: "req_snapdiff_latency": { 2024-04-02T14:51:02.439 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 0, 2024-04-02T14:51:02.439 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.000000000, 2024-04-02T14:51:02.439 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000000000 2024-04-02T14:51:02.439 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:51:02.439 INFO:teuthology.orchestra.run.smithi090.stdout: "req_rmdir_latency": { 2024-04-02T14:51:02.439 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 0, 2024-04-02T14:51:02.439 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.000000000, 2024-04-02T14:51:02.439 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000000000 2024-04-02T14:51:02.439 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:51:02.439 INFO:teuthology.orchestra.run.smithi090.stdout: "req_rmsnap_latency": { 2024-04-02T14:51:02.440 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 0, 2024-04-02T14:51:02.440 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.000000000, 2024-04-02T14:51:02.440 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000000000 2024-04-02T14:51:02.440 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:51:02.440 INFO:teuthology.orchestra.run.smithi090.stdout: "req_rmxattr_latency": { 2024-04-02T14:51:02.440 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 0, 2024-04-02T14:51:02.440 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.000000000, 2024-04-02T14:51:02.440 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000000000 2024-04-02T14:51:02.440 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:51:02.440 INFO:teuthology.orchestra.run.smithi090.stdout: "req_setattr_latency": { 2024-04-02T14:51:02.440 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 0, 2024-04-02T14:51:02.440 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.000000000, 2024-04-02T14:51:02.440 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000000000 2024-04-02T14:51:02.440 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:51:02.440 INFO:teuthology.orchestra.run.smithi090.stdout: "req_setdirlayout_latency": { 2024-04-02T14:51:02.441 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 0, 2024-04-02T14:51:02.441 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.000000000, 2024-04-02T14:51:02.441 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000000000 2024-04-02T14:51:02.441 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:51:02.441 INFO:teuthology.orchestra.run.smithi090.stdout: "req_setfilelock_latency": { 2024-04-02T14:51:02.441 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 0, 2024-04-02T14:51:02.441 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.000000000, 2024-04-02T14:51:02.441 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000000000 2024-04-02T14:51:02.441 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:51:02.441 INFO:teuthology.orchestra.run.smithi090.stdout: "req_setlayout_latency": { 2024-04-02T14:51:02.441 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 0, 2024-04-02T14:51:02.441 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.000000000, 2024-04-02T14:51:02.441 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000000000 2024-04-02T14:51:02.441 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:51:02.441 INFO:teuthology.orchestra.run.smithi090.stdout: "req_setxattr_latency": { 2024-04-02T14:51:02.442 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 0, 2024-04-02T14:51:02.442 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.000000000, 2024-04-02T14:51:02.442 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000000000 2024-04-02T14:51:02.442 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:51:02.442 INFO:teuthology.orchestra.run.smithi090.stdout: "req_symlink_latency": { 2024-04-02T14:51:02.442 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 0, 2024-04-02T14:51:02.442 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.000000000, 2024-04-02T14:51:02.442 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000000000 2024-04-02T14:51:02.442 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:51:02.442 INFO:teuthology.orchestra.run.smithi090.stdout: "req_unlink_latency": { 2024-04-02T14:51:02.442 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 0, 2024-04-02T14:51:02.442 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.000000000, 2024-04-02T14:51:02.442 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000000000 2024-04-02T14:51:02.442 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:51:02.443 INFO:teuthology.orchestra.run.smithi090.stdout: "cap_revoke_eviction": 0, 2024-04-02T14:51:02.443 INFO:teuthology.orchestra.run.smithi090.stdout: "cap_acquisition_throttle": 0, 2024-04-02T14:51:02.443 INFO:teuthology.orchestra.run.smithi090.stdout: "req_getvxattr_latency": { 2024-04-02T14:51:02.443 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 0, 2024-04-02T14:51:02.443 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.000000000, 2024-04-02T14:51:02.443 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000000000 2024-04-02T14:51:02.443 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:51:02.443 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:51:02.443 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_sessions": { 2024-04-02T14:51:02.443 INFO:teuthology.orchestra.run.smithi090.stdout: "session_count": 1, 2024-04-02T14:51:02.443 INFO:teuthology.orchestra.run.smithi090.stdout: "session_add": 3, 2024-04-02T14:51:02.443 INFO:teuthology.orchestra.run.smithi090.stdout: "session_remove": 2, 2024-04-02T14:51:02.443 INFO:teuthology.orchestra.run.smithi090.stdout: "sessions_open": 1, 2024-04-02T14:51:02.443 INFO:teuthology.orchestra.run.smithi090.stdout: "sessions_stale": 0, 2024-04-02T14:51:02.444 INFO:teuthology.orchestra.run.smithi090.stdout: "total_load": 0, 2024-04-02T14:51:02.444 INFO:teuthology.orchestra.run.smithi090.stdout: "average_load": 0, 2024-04-02T14:51:02.444 INFO:teuthology.orchestra.run.smithi090.stdout: "avg_session_uptime": 941, 2024-04-02T14:51:02.444 INFO:teuthology.orchestra.run.smithi090.stdout: "mdthresh_evicted": 0 2024-04-02T14:51:02.444 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:51:02.444 INFO:teuthology.orchestra.run.smithi090.stdout: "mempool": { 2024-04-02T14:51:02.444 INFO:teuthology.orchestra.run.smithi090.stdout: "bloom_filter_bytes": 0, 2024-04-02T14:51:02.444 INFO:teuthology.orchestra.run.smithi090.stdout: "bloom_filter_items": 0, 2024-04-02T14:51:02.444 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_alloc_bytes": 0, 2024-04-02T14:51:02.444 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_alloc_items": 0, 2024-04-02T14:51:02.444 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_cache_data_bytes": 0, 2024-04-02T14:51:02.444 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_cache_data_items": 0, 2024-04-02T14:51:02.444 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-02T14:51:02.444 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_cache_onode_items": 0, 2024-04-02T14:51:02.444 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-02T14:51:02.444 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_cache_meta_items": 0, 2024-04-02T14:51:02.445 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_cache_other_bytes": 0, 2024-04-02T14:51:02.445 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_cache_other_items": 0, 2024-04-02T14:51:02.445 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-02T14:51:02.445 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_cache_buffer_items": 0, 2024-04-02T14:51:02.445 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_extent_bytes": 0, 2024-04-02T14:51:02.445 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_extent_items": 0, 2024-04-02T14:51:02.445 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_blob_bytes": 0, 2024-04-02T14:51:02.445 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_blob_items": 0, 2024-04-02T14:51:02.445 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-02T14:51:02.445 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_shared_blob_items": 0, 2024-04-02T14:51:02.445 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-02T14:51:02.445 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_inline_bl_items": 0, 2024-04-02T14:51:02.445 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_fsck_bytes": 0, 2024-04-02T14:51:02.445 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_fsck_items": 0, 2024-04-02T14:51:02.445 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_txc_bytes": 0, 2024-04-02T14:51:02.446 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_txc_items": 0, 2024-04-02T14:51:02.446 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-02T14:51:02.446 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_writing_deferred_items": 0, 2024-04-02T14:51:02.446 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_writing_bytes": 0, 2024-04-02T14:51:02.446 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_writing_items": 0, 2024-04-02T14:51:02.446 INFO:teuthology.orchestra.run.smithi090.stdout: "bluefs_bytes": 0, 2024-04-02T14:51:02.446 INFO:teuthology.orchestra.run.smithi090.stdout: "bluefs_items": 0, 2024-04-02T14:51:02.446 INFO:teuthology.orchestra.run.smithi090.stdout: "bluefs_file_reader_bytes": 0, 2024-04-02T14:51:02.446 INFO:teuthology.orchestra.run.smithi090.stdout: "bluefs_file_reader_items": 0, 2024-04-02T14:51:02.446 INFO:teuthology.orchestra.run.smithi090.stdout: "bluefs_file_writer_bytes": 0, 2024-04-02T14:51:02.446 INFO:teuthology.orchestra.run.smithi090.stdout: "bluefs_file_writer_items": 0, 2024-04-02T14:51:02.446 INFO:teuthology.orchestra.run.smithi090.stdout: "buffer_anon_bytes": 243899, 2024-04-02T14:51:02.446 INFO:teuthology.orchestra.run.smithi090.stdout: "buffer_anon_items": 118, 2024-04-02T14:51:02.446 INFO:teuthology.orchestra.run.smithi090.stdout: "buffer_meta_bytes": 0, 2024-04-02T14:51:02.446 INFO:teuthology.orchestra.run.smithi090.stdout: "buffer_meta_items": 0, 2024-04-02T14:51:02.447 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_bytes": 0, 2024-04-02T14:51:02.447 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_items": 0, 2024-04-02T14:51:02.447 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_mapbl_bytes": 0, 2024-04-02T14:51:02.447 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_mapbl_items": 0, 2024-04-02T14:51:02.447 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_pglog_bytes": 0, 2024-04-02T14:51:02.447 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_pglog_items": 0, 2024-04-02T14:51:02.447 INFO:teuthology.orchestra.run.smithi090.stdout: "osdmap_bytes": 8136, 2024-04-02T14:51:02.447 INFO:teuthology.orchestra.run.smithi090.stdout: "osdmap_items": 170, 2024-04-02T14:51:02.447 INFO:teuthology.orchestra.run.smithi090.stdout: "osdmap_mapping_bytes": 0, 2024-04-02T14:51:02.447 INFO:teuthology.orchestra.run.smithi090.stdout: "osdmap_mapping_items": 0, 2024-04-02T14:51:02.447 INFO:teuthology.orchestra.run.smithi090.stdout: "pgmap_bytes": 0, 2024-04-02T14:51:02.447 INFO:teuthology.orchestra.run.smithi090.stdout: "pgmap_items": 0, 2024-04-02T14:51:02.447 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_co_bytes": 200417, 2024-04-02T14:51:02.447 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_co_items": 994, 2024-04-02T14:51:02.447 INFO:teuthology.orchestra.run.smithi090.stdout: "unittest_1_bytes": 0, 2024-04-02T14:51:02.447 INFO:teuthology.orchestra.run.smithi090.stdout: "unittest_1_items": 0, 2024-04-02T14:51:02.448 INFO:teuthology.orchestra.run.smithi090.stdout: "unittest_2_bytes": 0, 2024-04-02T14:51:02.448 INFO:teuthology.orchestra.run.smithi090.stdout: "unittest_2_items": 0 2024-04-02T14:51:02.448 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:51:02.448 INFO:teuthology.orchestra.run.smithi090.stdout: "objecter": { 2024-04-02T14:51:02.448 INFO:teuthology.orchestra.run.smithi090.stdout: "op_active": 0, 2024-04-02T14:51:02.448 INFO:teuthology.orchestra.run.smithi090.stdout: "op_laggy": 0, 2024-04-02T14:51:02.448 INFO:teuthology.orchestra.run.smithi090.stdout: "op_send": 44, 2024-04-02T14:51:02.448 INFO:teuthology.orchestra.run.smithi090.stdout: "op_send_bytes": 12440, 2024-04-02T14:51:02.448 INFO:teuthology.orchestra.run.smithi090.stdout: "op_resend": 27, 2024-04-02T14:51:02.448 INFO:teuthology.orchestra.run.smithi090.stdout: "op_reply": 44, 2024-04-02T14:51:02.448 INFO:teuthology.orchestra.run.smithi090.stdout: "op_latency": { 2024-04-02T14:51:02.448 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 44, 2024-04-02T14:51:02.448 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.264995818, 2024-04-02T14:51:02.448 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.006022632 2024-04-02T14:51:02.448 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:51:02.449 INFO:teuthology.orchestra.run.smithi090.stdout: "op_inflight": 0, 2024-04-02T14:51:02.449 INFO:teuthology.orchestra.run.smithi090.stdout: "oplen_avg": { 2024-04-02T14:51:02.449 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 44, 2024-04-02T14:51:02.449 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 66 2024-04-02T14:51:02.449 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:51:02.449 INFO:teuthology.orchestra.run.smithi090.stdout: "op": 44, 2024-04-02T14:51:02.449 INFO:teuthology.orchestra.run.smithi090.stdout: "op_r": 0, 2024-04-02T14:51:02.449 INFO:teuthology.orchestra.run.smithi090.stdout: "op_w": 44, 2024-04-02T14:51:02.449 INFO:teuthology.orchestra.run.smithi090.stdout: "op_rmw": 0, 2024-04-02T14:51:02.449 INFO:teuthology.orchestra.run.smithi090.stdout: "op_pg": 0, 2024-04-02T14:51:02.449 INFO:teuthology.orchestra.run.smithi090.stdout: "osdop_stat": 0, 2024-04-02T14:51:02.449 INFO:teuthology.orchestra.run.smithi090.stdout: "osdop_create": 10, 2024-04-02T14:51:02.449 INFO:teuthology.orchestra.run.smithi090.stdout: "osdop_read": 0, 2024-04-02T14:51:02.449 INFO:teuthology.orchestra.run.smithi090.stdout: "osdop_write": 6, 2024-04-02T14:51:02.449 INFO:teuthology.orchestra.run.smithi090.stdout: "osdop_writefull": 9, 2024-04-02T14:51:02.450 INFO:teuthology.orchestra.run.smithi090.stdout: "osdop_writesame": 0, 2024-04-02T14:51:02.450 INFO:teuthology.orchestra.run.smithi090.stdout: "osdop_append": 0, 2024-04-02T14:51:02.450 INFO:teuthology.orchestra.run.smithi090.stdout: "osdop_zero": 0, 2024-04-02T14:51:02.450 INFO:teuthology.orchestra.run.smithi090.stdout: "osdop_truncate": 0, 2024-04-02T14:51:02.450 INFO:teuthology.orchestra.run.smithi090.stdout: "osdop_delete": 6, 2024-04-02T14:51:02.450 INFO:teuthology.orchestra.run.smithi090.stdout: "osdop_mapext": 0, 2024-04-02T14:51:02.450 INFO:teuthology.orchestra.run.smithi090.stdout: "osdop_sparse_read": 0, 2024-04-02T14:51:02.450 INFO:teuthology.orchestra.run.smithi090.stdout: "osdop_clonerange": 0, 2024-04-02T14:51:02.450 INFO:teuthology.orchestra.run.smithi090.stdout: "osdop_getxattr": 0, 2024-04-02T14:51:02.450 INFO:teuthology.orchestra.run.smithi090.stdout: "osdop_setxattr": 20, 2024-04-02T14:51:02.450 INFO:teuthology.orchestra.run.smithi090.stdout: "osdop_cmpxattr": 0, 2024-04-02T14:51:02.450 INFO:teuthology.orchestra.run.smithi090.stdout: "osdop_rmxattr": 0, 2024-04-02T14:51:02.450 INFO:teuthology.orchestra.run.smithi090.stdout: "osdop_resetxattrs": 0, 2024-04-02T14:51:02.450 INFO:teuthology.orchestra.run.smithi090.stdout: "osdop_call": 0, 2024-04-02T14:51:02.450 INFO:teuthology.orchestra.run.smithi090.stdout: "osdop_watch": 0, 2024-04-02T14:51:02.451 INFO:teuthology.orchestra.run.smithi090.stdout: "osdop_notify": 0, 2024-04-02T14:51:02.451 INFO:teuthology.orchestra.run.smithi090.stdout: "osdop_src_cmpxattr": 0, 2024-04-02T14:51:02.451 INFO:teuthology.orchestra.run.smithi090.stdout: "osdop_pgls": 0, 2024-04-02T14:51:02.451 INFO:teuthology.orchestra.run.smithi090.stdout: "osdop_pgls_filter": 0, 2024-04-02T14:51:02.451 INFO:teuthology.orchestra.run.smithi090.stdout: "osdop_other": 0, 2024-04-02T14:51:02.451 INFO:teuthology.orchestra.run.smithi090.stdout: "linger_active": 0, 2024-04-02T14:51:02.451 INFO:teuthology.orchestra.run.smithi090.stdout: "linger_send": 0, 2024-04-02T14:51:02.451 INFO:teuthology.orchestra.run.smithi090.stdout: "linger_resend": 0, 2024-04-02T14:51:02.451 INFO:teuthology.orchestra.run.smithi090.stdout: "linger_ping": 0, 2024-04-02T14:51:02.451 INFO:teuthology.orchestra.run.smithi090.stdout: "poolop_active": 0, 2024-04-02T14:51:02.451 INFO:teuthology.orchestra.run.smithi090.stdout: "poolop_send": 0, 2024-04-02T14:51:02.451 INFO:teuthology.orchestra.run.smithi090.stdout: "poolop_resend": 0, 2024-04-02T14:51:02.451 INFO:teuthology.orchestra.run.smithi090.stdout: "poolstat_active": 0, 2024-04-02T14:51:02.451 INFO:teuthology.orchestra.run.smithi090.stdout: "poolstat_send": 0, 2024-04-02T14:51:02.451 INFO:teuthology.orchestra.run.smithi090.stdout: "poolstat_resend": 0, 2024-04-02T14:51:02.452 INFO:teuthology.orchestra.run.smithi090.stdout: "statfs_active": 0, 2024-04-02T14:51:02.452 INFO:teuthology.orchestra.run.smithi090.stdout: "statfs_send": 0, 2024-04-02T14:51:02.452 INFO:teuthology.orchestra.run.smithi090.stdout: "statfs_resend": 0, 2024-04-02T14:51:02.452 INFO:teuthology.orchestra.run.smithi090.stdout: "command_active": 0, 2024-04-02T14:51:02.452 INFO:teuthology.orchestra.run.smithi090.stdout: "command_send": 0, 2024-04-02T14:51:02.452 INFO:teuthology.orchestra.run.smithi090.stdout: "command_resend": 0, 2024-04-02T14:51:02.452 INFO:teuthology.orchestra.run.smithi090.stdout: "map_epoch": 109, 2024-04-02T14:51:02.452 INFO:teuthology.orchestra.run.smithi090.stdout: "map_full": 0, 2024-04-02T14:51:02.452 INFO:teuthology.orchestra.run.smithi090.stdout: "map_inc": 21, 2024-04-02T14:51:02.452 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_sessions": 9, 2024-04-02T14:51:02.452 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_session_open": 15, 2024-04-02T14:51:02.452 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_session_close": 6, 2024-04-02T14:51:02.452 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_laggy": 0, 2024-04-02T14:51:02.452 INFO:teuthology.orchestra.run.smithi090.stdout: "omap_wr": 14, 2024-04-02T14:51:02.453 INFO:teuthology.orchestra.run.smithi090.stdout: "omap_rd": 0, 2024-04-02T14:51:02.453 INFO:teuthology.orchestra.run.smithi090.stdout: "omap_del": 1 2024-04-02T14:51:02.453 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:51:02.453 INFO:teuthology.orchestra.run.smithi090.stdout: "oft": { 2024-04-02T14:51:02.453 INFO:teuthology.orchestra.run.smithi090.stdout: "omap_total_objs": 1, 2024-04-02T14:51:02.453 INFO:teuthology.orchestra.run.smithi090.stdout: "omap_total_kv_pairs": 0, 2024-04-02T14:51:02.453 INFO:teuthology.orchestra.run.smithi090.stdout: "omap_total_updates": 0, 2024-04-02T14:51:02.453 INFO:teuthology.orchestra.run.smithi090.stdout: "omap_total_removes": 0 2024-04-02T14:51:02.453 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:51:02.453 INFO:teuthology.orchestra.run.smithi090.stdout: "osd-slow-ops": { 2024-04-02T14:51:02.453 INFO:teuthology.orchestra.run.smithi090.stdout: "slow_ops_count": 0 2024-04-02T14:51:02.453 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:51:02.453 INFO:teuthology.orchestra.run.smithi090.stdout: "purge_queue": { 2024-04-02T14:51:02.453 INFO:teuthology.orchestra.run.smithi090.stdout: "pq_executing_ops": 0, 2024-04-02T14:51:02.453 INFO:teuthology.orchestra.run.smithi090.stdout: "pq_executing_ops_high_water": 0, 2024-04-02T14:51:02.454 INFO:teuthology.orchestra.run.smithi090.stdout: "pq_executing": 0, 2024-04-02T14:51:02.454 INFO:teuthology.orchestra.run.smithi090.stdout: "pq_executing_high_water": 0, 2024-04-02T14:51:02.454 INFO:teuthology.orchestra.run.smithi090.stdout: "pq_executed_ops": 0, 2024-04-02T14:51:02.454 INFO:teuthology.orchestra.run.smithi090.stdout: "pq_executed": 0, 2024-04-02T14:51:02.454 INFO:teuthology.orchestra.run.smithi090.stdout: "pq_item_in_journal": 0 2024-04-02T14:51:02.454 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:51:02.454 INFO:teuthology.orchestra.run.smithi090.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-02T14:51:02.454 INFO:teuthology.orchestra.run.smithi090.stdout: "val": 0, 2024-04-02T14:51:02.454 INFO:teuthology.orchestra.run.smithi090.stdout: "max": 104857600, 2024-04-02T14:51:02.454 INFO:teuthology.orchestra.run.smithi090.stdout: "get_started": 0, 2024-04-02T14:51:02.454 INFO:teuthology.orchestra.run.smithi090.stdout: "get": 4316, 2024-04-02T14:51:02.454 INFO:teuthology.orchestra.run.smithi090.stdout: "get_sum": 2336889, 2024-04-02T14:51:02.454 INFO:teuthology.orchestra.run.smithi090.stdout: "get_or_fail_fail": 0, 2024-04-02T14:51:02.454 INFO:teuthology.orchestra.run.smithi090.stdout: "get_or_fail_success": 4316, 2024-04-02T14:51:02.455 INFO:teuthology.orchestra.run.smithi090.stdout: "take": 0, 2024-04-02T14:51:02.455 INFO:teuthology.orchestra.run.smithi090.stdout: "take_sum": 0, 2024-04-02T14:51:02.455 INFO:teuthology.orchestra.run.smithi090.stdout: "put": 4316, 2024-04-02T14:51:02.455 INFO:teuthology.orchestra.run.smithi090.stdout: "put_sum": 2336889, 2024-04-02T14:51:02.455 INFO:teuthology.orchestra.run.smithi090.stdout: "wait": { 2024-04-02T14:51:02.455 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 0, 2024-04-02T14:51:02.455 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.000000000, 2024-04-02T14:51:02.455 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000000000 2024-04-02T14:51:02.455 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:51:02.455 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:51:02.455 INFO:teuthology.orchestra.run.smithi090.stdout: "throttle-objecter_bytes": { 2024-04-02T14:51:02.455 INFO:teuthology.orchestra.run.smithi090.stdout: "val": 0, 2024-04-02T14:51:02.455 INFO:teuthology.orchestra.run.smithi090.stdout: "max": 104857600, 2024-04-02T14:51:02.455 INFO:teuthology.orchestra.run.smithi090.stdout: "get_started": 0, 2024-04-02T14:51:02.455 INFO:teuthology.orchestra.run.smithi090.stdout: "get": 0, 2024-04-02T14:51:02.456 INFO:teuthology.orchestra.run.smithi090.stdout: "get_sum": 0, 2024-04-02T14:51:02.456 INFO:teuthology.orchestra.run.smithi090.stdout: "get_or_fail_fail": 0, 2024-04-02T14:51:02.456 INFO:teuthology.orchestra.run.smithi090.stdout: "get_or_fail_success": 0, 2024-04-02T14:51:02.456 INFO:teuthology.orchestra.run.smithi090.stdout: "take": 44, 2024-04-02T14:51:02.456 INFO:teuthology.orchestra.run.smithi090.stdout: "take_sum": 12440, 2024-04-02T14:51:02.456 INFO:teuthology.orchestra.run.smithi090.stdout: "put": 44, 2024-04-02T14:51:02.456 INFO:teuthology.orchestra.run.smithi090.stdout: "put_sum": 12440, 2024-04-02T14:51:02.456 INFO:teuthology.orchestra.run.smithi090.stdout: "wait": { 2024-04-02T14:51:02.456 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 0, 2024-04-02T14:51:02.456 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.000000000, 2024-04-02T14:51:02.456 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000000000 2024-04-02T14:51:02.456 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:51:02.456 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:51:02.456 INFO:teuthology.orchestra.run.smithi090.stdout: "throttle-objecter_ops": { 2024-04-02T14:51:02.457 INFO:teuthology.orchestra.run.smithi090.stdout: "val": 0, 2024-04-02T14:51:02.457 INFO:teuthology.orchestra.run.smithi090.stdout: "max": 1024, 2024-04-02T14:51:02.457 INFO:teuthology.orchestra.run.smithi090.stdout: "get_started": 0, 2024-04-02T14:51:02.457 INFO:teuthology.orchestra.run.smithi090.stdout: "get": 0, 2024-04-02T14:51:02.457 INFO:teuthology.orchestra.run.smithi090.stdout: "get_sum": 0, 2024-04-02T14:51:02.457 INFO:teuthology.orchestra.run.smithi090.stdout: "get_or_fail_fail": 0, 2024-04-02T14:51:02.457 INFO:teuthology.orchestra.run.smithi090.stdout: "get_or_fail_success": 0, 2024-04-02T14:51:02.457 INFO:teuthology.orchestra.run.smithi090.stdout: "take": 44, 2024-04-02T14:51:02.457 INFO:teuthology.orchestra.run.smithi090.stdout: "take_sum": 44, 2024-04-02T14:51:02.457 INFO:teuthology.orchestra.run.smithi090.stdout: "put": 44, 2024-04-02T14:51:02.457 INFO:teuthology.orchestra.run.smithi090.stdout: "put_sum": 44, 2024-04-02T14:51:02.457 INFO:teuthology.orchestra.run.smithi090.stdout: "wait": { 2024-04-02T14:51:02.457 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 0, 2024-04-02T14:51:02.457 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.000000000, 2024-04-02T14:51:02.457 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000000000 2024-04-02T14:51:02.457 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:51:02.458 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:51:02.458 INFO:teuthology.orchestra.run.smithi090.stdout: "throttle-write_buf_throttle": { 2024-04-02T14:51:02.458 INFO:teuthology.orchestra.run.smithi090.stdout: "val": 0, 2024-04-02T14:51:02.458 INFO:teuthology.orchestra.run.smithi090.stdout: "max": 3758096384, 2024-04-02T14:51:02.458 INFO:teuthology.orchestra.run.smithi090.stdout: "get_started": 0, 2024-04-02T14:51:02.458 INFO:teuthology.orchestra.run.smithi090.stdout: "get": 0, 2024-04-02T14:51:02.458 INFO:teuthology.orchestra.run.smithi090.stdout: "get_sum": 0, 2024-04-02T14:51:02.458 INFO:teuthology.orchestra.run.smithi090.stdout: "get_or_fail_fail": 0, 2024-04-02T14:51:02.458 INFO:teuthology.orchestra.run.smithi090.stdout: "get_or_fail_success": 0, 2024-04-02T14:51:02.458 INFO:teuthology.orchestra.run.smithi090.stdout: "take": 0, 2024-04-02T14:51:02.458 INFO:teuthology.orchestra.run.smithi090.stdout: "take_sum": 0, 2024-04-02T14:51:02.458 INFO:teuthology.orchestra.run.smithi090.stdout: "put": 0, 2024-04-02T14:51:02.458 INFO:teuthology.orchestra.run.smithi090.stdout: "put_sum": 0, 2024-04-02T14:51:02.458 INFO:teuthology.orchestra.run.smithi090.stdout: "wait": { 2024-04-02T14:51:02.459 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 0, 2024-04-02T14:51:02.459 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.000000000, 2024-04-02T14:51:02.459 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000000000 2024-04-02T14:51:02.459 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:51:02.459 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:51:02.459 INFO:teuthology.orchestra.run.smithi090.stdout: "throttle-write_buf_throttle-0x55a1377fc500": { 2024-04-02T14:51:02.459 INFO:teuthology.orchestra.run.smithi090.stdout: "val": 0, 2024-04-02T14:51:02.459 INFO:teuthology.orchestra.run.smithi090.stdout: "max": 3758096384, 2024-04-02T14:51:02.459 INFO:teuthology.orchestra.run.smithi090.stdout: "get_started": 0, 2024-04-02T14:51:02.459 INFO:teuthology.orchestra.run.smithi090.stdout: "get": 7, 2024-04-02T14:51:02.459 INFO:teuthology.orchestra.run.smithi090.stdout: "get_sum": 1958, 2024-04-02T14:51:02.459 INFO:teuthology.orchestra.run.smithi090.stdout: "get_or_fail_fail": 0, 2024-04-02T14:51:02.459 INFO:teuthology.orchestra.run.smithi090.stdout: "get_or_fail_success": 7, 2024-04-02T14:51:02.459 INFO:teuthology.orchestra.run.smithi090.stdout: "take": 0, 2024-04-02T14:51:02.459 INFO:teuthology.orchestra.run.smithi090.stdout: "take_sum": 0, 2024-04-02T14:51:02.460 INFO:teuthology.orchestra.run.smithi090.stdout: "put": 6, 2024-04-02T14:51:02.460 INFO:teuthology.orchestra.run.smithi090.stdout: "put_sum": 1958, 2024-04-02T14:51:02.460 INFO:teuthology.orchestra.run.smithi090.stdout: "wait": { 2024-04-02T14:51:02.460 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 0, 2024-04-02T14:51:02.460 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.000000000, 2024-04-02T14:51:02.460 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000000000 2024-04-02T14:51:02.460 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:51:02.460 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:51:02.460 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-02T14:51:03.476 INFO:tasks.check_counter:Daemon mds.g mds.exported=0 2024-04-02T14:51:03.476 INFO:tasks.check_counter:Daemon mds.g mds.imported=0 2024-04-02T14:51:03.476 INFO:tasks.check_counter:Daemon mds.g mds_cache.dir_update=0 2024-04-02T14:51:03.476 INFO:tasks.check_counter:Daemon mds.g mds_cache.dir_update_receipt=105 2024-04-02T14:51:03.476 INFO:tasks.check_counter:Daemon mds.g mds.root_rsnaps=6 2024-04-02T14:51:03.476 INFO:tasks.check_counter:Daemon mds.g mds_server.req_mksnap_latency.avgcount=0 2024-04-02T14:51:03.476 INFO:tasks.check_counter:Daemon mds.g mds_server.req_rmsnap_latency.avgcount=0 2024-04-02T14:51:03.476 DEBUG:tasks.check_counter:Getting stats from j 2024-04-02T14:51:03.477 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:191d963ac9de7322da8aff20b567b3fa458e2ef6 shell --fsid 6449448a-f0fc-11ee-b647-cb9ed24678a4 -- ceph daemon mds.j perf dump 2024-04-02T14:51:03.937 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-02T14:51:04.089 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:51:03 smithi090 ceph-mon[31674]: pgmap v506: 97 pgs: 97 active+clean; 4.3 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 23 KiB/s wr, 12 op/s 2024-04-02T14:51:04.102 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:51:03 smithi092 ceph-mon[36497]: pgmap v506: 97 pgs: 97 active+clean; 4.3 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 23 KiB/s wr, 12 op/s 2024-04-02T14:51:04.178 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:51:03 smithi177 ceph-mon[36431]: pgmap v506: 97 pgs: 97 active+clean; 4.3 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 23 KiB/s wr, 12 op/s 2024-04-02T14:51:05.911 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:51:05 smithi090 ceph-mon[31674]: pgmap v507: 97 pgs: 97 active+clean; 4.3 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 87 KiB/s wr, 32 op/s 2024-04-02T14:51:06.102 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:51:05 smithi092 ceph-mon[36497]: pgmap v507: 97 pgs: 97 active+clean; 4.3 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 87 KiB/s wr, 32 op/s 2024-04-02T14:51:06.123 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-02T14:51:06.123 INFO:teuthology.orchestra.run.smithi090.stdout: "AsyncMessenger::Worker-0": { 2024-04-02T14:51:06.123 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_recv_messages": 1118, 2024-04-02T14:51:06.124 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_send_messages": 1118, 2024-04-02T14:51:06.124 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_recv_bytes": 298880, 2024-04-02T14:51:06.124 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_send_bytes": 354764, 2024-04-02T14:51:06.124 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_created_connections": 7, 2024-04-02T14:51:06.124 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_active_connections": 2, 2024-04-02T14:51:06.124 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_running_total_time": 0.171249636, 2024-04-02T14:51:06.124 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_running_send_time": 0.054707136, 2024-04-02T14:51:06.124 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_running_recv_time": 0.041174333, 2024-04-02T14:51:06.124 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_running_fast_dispatch_time": 0.054767141, 2024-04-02T14:51:06.124 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_send_messages_queue_lat": { 2024-04-02T14:51:06.124 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 1117, 2024-04-02T14:51:06.124 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.141414589, 2024-04-02T14:51:06.124 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000126602 2024-04-02T14:51:06.124 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:51:06.125 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_handle_ack_lat": { 2024-04-02T14:51:06.125 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 0, 2024-04-02T14:51:06.125 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.000000000, 2024-04-02T14:51:06.125 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000000000 2024-04-02T14:51:06.125 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:51:06.125 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_recv_encrypted_bytes": 3072, 2024-04-02T14:51:06.125 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_send_encrypted_bytes": 448 2024-04-02T14:51:06.125 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:51:06.125 INFO:teuthology.orchestra.run.smithi090.stdout: "AsyncMessenger::Worker-1": { 2024-04-02T14:51:06.125 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_recv_messages": 1387, 2024-04-02T14:51:06.125 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_send_messages": 1375, 2024-04-02T14:51:06.125 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_recv_bytes": 1909708, 2024-04-02T14:51:06.125 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_send_bytes": 526051, 2024-04-02T14:51:06.125 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_created_connections": 4, 2024-04-02T14:51:06.125 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_active_connections": 3, 2024-04-02T14:51:06.126 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_running_total_time": 0.311422739, 2024-04-02T14:51:06.126 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_running_send_time": 0.101558758, 2024-04-02T14:51:06.126 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_running_recv_time": 0.153560114, 2024-04-02T14:51:06.126 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_running_fast_dispatch_time": 0.082453565, 2024-04-02T14:51:06.126 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_send_messages_queue_lat": { 2024-04-02T14:51:06.126 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 1374, 2024-04-02T14:51:06.126 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.191774588, 2024-04-02T14:51:06.126 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000139573 2024-04-02T14:51:06.126 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:51:06.126 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_handle_ack_lat": { 2024-04-02T14:51:06.126 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 0, 2024-04-02T14:51:06.126 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.000000000, 2024-04-02T14:51:06.126 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000000000 2024-04-02T14:51:06.127 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:51:06.127 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_recv_encrypted_bytes": 1653600, 2024-04-02T14:51:06.127 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_send_encrypted_bytes": 195008 2024-04-02T14:51:06.127 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:51:06.127 INFO:teuthology.orchestra.run.smithi090.stdout: "AsyncMessenger::Worker-2": { 2024-04-02T14:51:06.127 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_recv_messages": 1106, 2024-04-02T14:51:06.127 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_send_messages": 1339, 2024-04-02T14:51:06.127 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_recv_bytes": 371264, 2024-04-02T14:51:06.127 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_send_bytes": 923856, 2024-04-02T14:51:06.127 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_created_connections": 6, 2024-04-02T14:51:06.127 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_active_connections": 4, 2024-04-02T14:51:06.127 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_running_total_time": 0.275021621, 2024-04-02T14:51:06.127 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_running_send_time": 0.141108676, 2024-04-02T14:51:06.127 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_running_recv_time": 0.050136328, 2024-04-02T14:51:06.128 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_running_fast_dispatch_time": 0.061517668, 2024-04-02T14:51:06.128 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_send_messages_queue_lat": { 2024-04-02T14:51:06.128 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 1336, 2024-04-02T14:51:06.128 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.279885217, 2024-04-02T14:51:06.128 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000209494 2024-04-02T14:51:06.128 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:51:06.128 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_handle_ack_lat": { 2024-04-02T14:51:06.128 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 0, 2024-04-02T14:51:06.128 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.000000000, 2024-04-02T14:51:06.128 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000000000 2024-04-02T14:51:06.128 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:51:06.128 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_recv_encrypted_bytes": 320, 2024-04-02T14:51:06.128 INFO:teuthology.orchestra.run.smithi090.stdout: "msgr_send_encrypted_bytes": 572784 2024-04-02T14:51:06.128 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:51:06.128 INFO:teuthology.orchestra.run.smithi090.stdout: "cct": { 2024-04-02T14:51:06.129 INFO:teuthology.orchestra.run.smithi090.stdout: "total_workers": 1, 2024-04-02T14:51:06.129 INFO:teuthology.orchestra.run.smithi090.stdout: "unhealthy_workers": 0 2024-04-02T14:51:06.129 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:51:06.129 INFO:teuthology.orchestra.run.smithi090.stdout: "finisher-MDSRank": { 2024-04-02T14:51:06.129 INFO:teuthology.orchestra.run.smithi090.stdout: "queue_len": 0, 2024-04-02T14:51:06.129 INFO:teuthology.orchestra.run.smithi090.stdout: "complete_latency": { 2024-04-02T14:51:06.129 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 5367, 2024-04-02T14:51:06.129 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.234145400, 2024-04-02T14:51:06.129 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000043626 2024-04-02T14:51:06.129 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:51:06.129 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:51:06.129 INFO:teuthology.orchestra.run.smithi090.stdout: "finisher-PurgeQueue": { 2024-04-02T14:51:06.129 INFO:teuthology.orchestra.run.smithi090.stdout: "queue_len": 0, 2024-04-02T14:51:06.129 INFO:teuthology.orchestra.run.smithi090.stdout: "complete_latency": { 2024-04-02T14:51:06.130 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 0, 2024-04-02T14:51:06.130 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.000000000, 2024-04-02T14:51:06.130 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000000000 2024-04-02T14:51:06.130 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:51:06.130 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:51:06.130 INFO:teuthology.orchestra.run.smithi090.stdout: "mds": { 2024-04-02T14:51:06.130 INFO:teuthology.orchestra.run.smithi090.stdout: "request": 0, 2024-04-02T14:51:06.130 INFO:teuthology.orchestra.run.smithi090.stdout: "reply": 0, 2024-04-02T14:51:06.130 INFO:teuthology.orchestra.run.smithi090.stdout: "reply_latency": { 2024-04-02T14:51:06.130 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 0, 2024-04-02T14:51:06.130 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.000000000, 2024-04-02T14:51:06.130 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000000000 2024-04-02T14:51:06.130 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:51:06.131 INFO:teuthology.orchestra.run.smithi090.stdout: "slow_reply": 0, 2024-04-02T14:51:06.131 INFO:teuthology.orchestra.run.smithi090.stdout: "forward": 0, 2024-04-02T14:51:06.131 INFO:teuthology.orchestra.run.smithi090.stdout: "dir_fetch_complete": 0, 2024-04-02T14:51:06.131 INFO:teuthology.orchestra.run.smithi090.stdout: "dir_fetch_keys": 0, 2024-04-02T14:51:06.131 INFO:teuthology.orchestra.run.smithi090.stdout: "dir_commit": 0, 2024-04-02T14:51:06.131 INFO:teuthology.orchestra.run.smithi090.stdout: "dir_split": 0, 2024-04-02T14:51:06.131 INFO:teuthology.orchestra.run.smithi090.stdout: "dir_merge": 0, 2024-04-02T14:51:06.132 INFO:teuthology.orchestra.run.smithi090.stdout: "inodes": 0, 2024-04-02T14:51:06.132 INFO:teuthology.orchestra.run.smithi090.stdout: "inodes_top": 0, 2024-04-02T14:51:06.132 INFO:teuthology.orchestra.run.smithi090.stdout: "inodes_bottom": 0, 2024-04-02T14:51:06.132 INFO:teuthology.orchestra.run.smithi090.stdout: "inodes_pin_tail": 0, 2024-04-02T14:51:06.132 INFO:teuthology.orchestra.run.smithi090.stdout: "inodes_pinned": 0, 2024-04-02T14:51:06.132 INFO:teuthology.orchestra.run.smithi090.stdout: "inodes_expired": 0, 2024-04-02T14:51:06.132 INFO:teuthology.orchestra.run.smithi090.stdout: "inodes_with_caps": 0, 2024-04-02T14:51:06.132 INFO:teuthology.orchestra.run.smithi090.stdout: "caps": 0, 2024-04-02T14:51:06.132 INFO:teuthology.orchestra.run.smithi090.stdout: "subtrees": 1, 2024-04-02T14:51:06.132 INFO:teuthology.orchestra.run.smithi090.stdout: "traverse": 0, 2024-04-02T14:51:06.133 INFO:teuthology.orchestra.run.smithi090.stdout: "traverse_hit": 0, 2024-04-02T14:51:06.133 INFO:teuthology.orchestra.run.smithi090.stdout: "traverse_forward": 0, 2024-04-02T14:51:06.133 INFO:teuthology.orchestra.run.smithi090.stdout: "traverse_discover": 0, 2024-04-02T14:51:06.133 INFO:teuthology.orchestra.run.smithi090.stdout: "traverse_dir_fetch": 0, 2024-04-02T14:51:06.133 INFO:teuthology.orchestra.run.smithi090.stdout: "traverse_remote_ino": 0, 2024-04-02T14:51:06.133 INFO:teuthology.orchestra.run.smithi090.stdout: "traverse_lock": 0, 2024-04-02T14:51:06.133 INFO:teuthology.orchestra.run.smithi090.stdout: "load_cent": 0, 2024-04-02T14:51:06.133 INFO:teuthology.orchestra.run.smithi090.stdout: "q": 0, 2024-04-02T14:51:06.133 INFO:teuthology.orchestra.run.smithi090.stdout: "exported": 0, 2024-04-02T14:51:06.133 INFO:teuthology.orchestra.run.smithi090.stdout: "exported_inodes": 0, 2024-04-02T14:51:06.133 INFO:teuthology.orchestra.run.smithi090.stdout: "imported": 0, 2024-04-02T14:51:06.133 INFO:teuthology.orchestra.run.smithi090.stdout: "imported_inodes": 0, 2024-04-02T14:51:06.134 INFO:teuthology.orchestra.run.smithi090.stdout: "openino_dir_fetch": 0, 2024-04-02T14:51:06.134 INFO:teuthology.orchestra.run.smithi090.stdout: "openino_backtrace_fetch": 0, 2024-04-02T14:51:06.134 INFO:teuthology.orchestra.run.smithi090.stdout: "openino_peer_discover": 0, 2024-04-02T14:51:06.134 INFO:teuthology.orchestra.run.smithi090.stdout: "root_rfiles": 0, 2024-04-02T14:51:06.134 INFO:teuthology.orchestra.run.smithi090.stdout: "root_rbytes": 0, 2024-04-02T14:51:06.134 INFO:teuthology.orchestra.run.smithi090.stdout: "root_rsnaps": 0, 2024-04-02T14:51:06.134 INFO:teuthology.orchestra.run.smithi090.stdout: "scrub_backtrace_fetch": 0, 2024-04-02T14:51:06.134 INFO:teuthology.orchestra.run.smithi090.stdout: "scrub_set_tag": 0, 2024-04-02T14:51:06.134 INFO:teuthology.orchestra.run.smithi090.stdout: "scrub_backtrace_repaired": 0, 2024-04-02T14:51:06.134 INFO:teuthology.orchestra.run.smithi090.stdout: "scrub_inotable_repaired": 0, 2024-04-02T14:51:06.134 INFO:teuthology.orchestra.run.smithi090.stdout: "scrub_dir_inodes": 0, 2024-04-02T14:51:06.134 INFO:teuthology.orchestra.run.smithi090.stdout: "scrub_dir_base_inodes": 0, 2024-04-02T14:51:06.134 INFO:teuthology.orchestra.run.smithi090.stdout: "scrub_dirfrag_rstats": 0, 2024-04-02T14:51:06.134 INFO:teuthology.orchestra.run.smithi090.stdout: "scrub_file_inodes": 0, 2024-04-02T14:51:06.134 INFO:teuthology.orchestra.run.smithi090.stdout: "handle_inode_file_caps": 0, 2024-04-02T14:51:06.135 INFO:teuthology.orchestra.run.smithi090.stdout: "ceph_cap_op_revoke": 0, 2024-04-02T14:51:06.135 INFO:teuthology.orchestra.run.smithi090.stdout: "ceph_cap_op_grant": 0, 2024-04-02T14:51:06.135 INFO:teuthology.orchestra.run.smithi090.stdout: "ceph_cap_op_trunc": 0, 2024-04-02T14:51:06.135 INFO:teuthology.orchestra.run.smithi090.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-02T14:51:06.135 INFO:teuthology.orchestra.run.smithi090.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-02T14:51:06.135 INFO:teuthology.orchestra.run.smithi090.stdout: "handle_client_caps": 0, 2024-04-02T14:51:06.135 INFO:teuthology.orchestra.run.smithi090.stdout: "handle_client_caps_dirty": 0, 2024-04-02T14:51:06.135 INFO:teuthology.orchestra.run.smithi090.stdout: "handle_client_cap_release": 0, 2024-04-02T14:51:06.135 INFO:teuthology.orchestra.run.smithi090.stdout: "process_request_cap_release": 0 2024-04-02T14:51:06.135 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:51:06.135 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_cache": { 2024-04-02T14:51:06.135 INFO:teuthology.orchestra.run.smithi090.stdout: "dir_update": 0, 2024-04-02T14:51:06.135 INFO:teuthology.orchestra.run.smithi090.stdout: "dir_update_receipt": 0, 2024-04-02T14:51:06.135 INFO:teuthology.orchestra.run.smithi090.stdout: "dir_try_discover": 0, 2024-04-02T14:51:06.135 INFO:teuthology.orchestra.run.smithi090.stdout: "dir_send_discover": 0, 2024-04-02T14:51:06.136 INFO:teuthology.orchestra.run.smithi090.stdout: "dir_handle_discover": 0, 2024-04-02T14:51:06.136 INFO:teuthology.orchestra.run.smithi090.stdout: "num_strays": 0, 2024-04-02T14:51:06.136 INFO:teuthology.orchestra.run.smithi090.stdout: "num_strays_delayed": 0, 2024-04-02T14:51:06.136 INFO:teuthology.orchestra.run.smithi090.stdout: "num_strays_enqueuing": 0, 2024-04-02T14:51:06.136 INFO:teuthology.orchestra.run.smithi090.stdout: "strays_created": 0, 2024-04-02T14:51:06.136 INFO:teuthology.orchestra.run.smithi090.stdout: "strays_enqueued": 0, 2024-04-02T14:51:06.136 INFO:teuthology.orchestra.run.smithi090.stdout: "strays_reintegrated": 0, 2024-04-02T14:51:06.136 INFO:teuthology.orchestra.run.smithi090.stdout: "strays_migrated": 0, 2024-04-02T14:51:06.136 INFO:teuthology.orchestra.run.smithi090.stdout: "num_recovering_processing": 0, 2024-04-02T14:51:06.136 INFO:teuthology.orchestra.run.smithi090.stdout: "num_recovering_enqueued": 0, 2024-04-02T14:51:06.136 INFO:teuthology.orchestra.run.smithi090.stdout: "num_recovering_prioritized": 0, 2024-04-02T14:51:06.136 INFO:teuthology.orchestra.run.smithi090.stdout: "recovery_started": 0, 2024-04-02T14:51:06.136 INFO:teuthology.orchestra.run.smithi090.stdout: "recovery_completed": 0, 2024-04-02T14:51:06.136 INFO:teuthology.orchestra.run.smithi090.stdout: "ireq_quiesce_path": 0, 2024-04-02T14:51:06.136 INFO:teuthology.orchestra.run.smithi090.stdout: "ireq_quiesce_inode": 0, 2024-04-02T14:51:06.137 INFO:teuthology.orchestra.run.smithi090.stdout: "ireq_enqueue_scrub": 0, 2024-04-02T14:51:06.137 INFO:teuthology.orchestra.run.smithi090.stdout: "ireq_exportdir": 0, 2024-04-02T14:51:06.137 INFO:teuthology.orchestra.run.smithi090.stdout: "ireq_flush": 0, 2024-04-02T14:51:06.137 INFO:teuthology.orchestra.run.smithi090.stdout: "ireq_fragmentdir": 0, 2024-04-02T14:51:06.137 INFO:teuthology.orchestra.run.smithi090.stdout: "ireq_fragstats": 0, 2024-04-02T14:51:06.137 INFO:teuthology.orchestra.run.smithi090.stdout: "ireq_inodestats": 0 2024-04-02T14:51:06.137 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:51:06.137 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_log": { 2024-04-02T14:51:06.137 INFO:teuthology.orchestra.run.smithi090.stdout: "evlrg": 0, 2024-04-02T14:51:06.137 INFO:teuthology.orchestra.run.smithi090.stdout: "evadd": 0, 2024-04-02T14:51:06.137 INFO:teuthology.orchestra.run.smithi090.stdout: "evex": 0, 2024-04-02T14:51:06.137 INFO:teuthology.orchestra.run.smithi090.stdout: "evtrm": 0, 2024-04-02T14:51:06.137 INFO:teuthology.orchestra.run.smithi090.stdout: "ev": 12, 2024-04-02T14:51:06.137 INFO:teuthology.orchestra.run.smithi090.stdout: "evexg": 0, 2024-04-02T14:51:06.137 INFO:teuthology.orchestra.run.smithi090.stdout: "evexd": 0, 2024-04-02T14:51:06.137 INFO:teuthology.orchestra.run.smithi090.stdout: "segadd": 0, 2024-04-02T14:51:06.138 INFO:teuthology.orchestra.run.smithi090.stdout: "segex": 0, 2024-04-02T14:51:06.138 INFO:teuthology.orchestra.run.smithi090.stdout: "segtrm": 0, 2024-04-02T14:51:06.138 INFO:teuthology.orchestra.run.smithi090.stdout: "seg": 2, 2024-04-02T14:51:06.138 INFO:teuthology.orchestra.run.smithi090.stdout: "segmjr": 2, 2024-04-02T14:51:06.138 INFO:teuthology.orchestra.run.smithi090.stdout: "segexg": 0, 2024-04-02T14:51:06.138 INFO:teuthology.orchestra.run.smithi090.stdout: "segexd": 0, 2024-04-02T14:51:06.138 INFO:teuthology.orchestra.run.smithi090.stdout: "expos": 4194304, 2024-04-02T14:51:06.138 INFO:teuthology.orchestra.run.smithi090.stdout: "wrpos": 4197717, 2024-04-02T14:51:06.138 INFO:teuthology.orchestra.run.smithi090.stdout: "rdpos": 4197576, 2024-04-02T14:51:06.138 INFO:teuthology.orchestra.run.smithi090.stdout: "jlat": { 2024-04-02T14:51:06.138 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 0, 2024-04-02T14:51:06.138 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.000000000, 2024-04-02T14:51:06.138 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000000000 2024-04-02T14:51:06.138 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:51:06.139 INFO:teuthology.orchestra.run.smithi090.stdout: "replayed": 12 2024-04-02T14:51:06.139 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:51:06.139 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_mem": { 2024-04-02T14:51:06.139 INFO:teuthology.orchestra.run.smithi090.stdout: "ino": 3, 2024-04-02T14:51:06.139 INFO:teuthology.orchestra.run.smithi090.stdout: "ino+": 3, 2024-04-02T14:51:06.139 INFO:teuthology.orchestra.run.smithi090.stdout: "ino-": 0, 2024-04-02T14:51:06.139 INFO:teuthology.orchestra.run.smithi090.stdout: "dir": 1, 2024-04-02T14:51:06.139 INFO:teuthology.orchestra.run.smithi090.stdout: "dir+": 1, 2024-04-02T14:51:06.139 INFO:teuthology.orchestra.run.smithi090.stdout: "dir-": 0, 2024-04-02T14:51:06.139 INFO:teuthology.orchestra.run.smithi090.stdout: "dn": 0, 2024-04-02T14:51:06.139 INFO:teuthology.orchestra.run.smithi090.stdout: "dn+": 0, 2024-04-02T14:51:06.139 INFO:teuthology.orchestra.run.smithi090.stdout: "dn-": 0, 2024-04-02T14:51:06.139 INFO:teuthology.orchestra.run.smithi090.stdout: "cap": 0, 2024-04-02T14:51:06.139 INFO:teuthology.orchestra.run.smithi090.stdout: "cap+": 0, 2024-04-02T14:51:06.139 INFO:teuthology.orchestra.run.smithi090.stdout: "cap-": 0, 2024-04-02T14:51:06.139 INFO:teuthology.orchestra.run.smithi090.stdout: "rss": 48896, 2024-04-02T14:51:06.140 INFO:teuthology.orchestra.run.smithi090.stdout: "heap": 215416 2024-04-02T14:51:06.140 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:51:06.140 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_server": { 2024-04-02T14:51:06.140 INFO:teuthology.orchestra.run.smithi090.stdout: "dispatch_client_request": 0, 2024-04-02T14:51:06.140 INFO:teuthology.orchestra.run.smithi090.stdout: "dispatch_server_request": 0, 2024-04-02T14:51:06.140 INFO:teuthology.orchestra.run.smithi090.stdout: "handle_client_request": 0, 2024-04-02T14:51:06.140 INFO:teuthology.orchestra.run.smithi090.stdout: "handle_client_session": 0, 2024-04-02T14:51:06.140 INFO:teuthology.orchestra.run.smithi090.stdout: "handle_peer_request": 0, 2024-04-02T14:51:06.140 INFO:teuthology.orchestra.run.smithi090.stdout: "req_create_latency": { 2024-04-02T14:51:06.140 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 0, 2024-04-02T14:51:06.140 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.000000000, 2024-04-02T14:51:06.140 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000000000 2024-04-02T14:51:06.140 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:51:06.140 INFO:teuthology.orchestra.run.smithi090.stdout: "req_getattr_latency": { 2024-04-02T14:51:06.140 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 0, 2024-04-02T14:51:06.140 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.000000000, 2024-04-02T14:51:06.141 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000000000 2024-04-02T14:51:06.141 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:51:06.141 INFO:teuthology.orchestra.run.smithi090.stdout: "req_getfilelock_latency": { 2024-04-02T14:51:06.141 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 0, 2024-04-02T14:51:06.141 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.000000000, 2024-04-02T14:51:06.141 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000000000 2024-04-02T14:51:06.141 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:51:06.141 INFO:teuthology.orchestra.run.smithi090.stdout: "req_link_latency": { 2024-04-02T14:51:06.141 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 0, 2024-04-02T14:51:06.141 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.000000000, 2024-04-02T14:51:06.141 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000000000 2024-04-02T14:51:06.141 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:51:06.141 INFO:teuthology.orchestra.run.smithi090.stdout: "req_lookup_latency": { 2024-04-02T14:51:06.141 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 0, 2024-04-02T14:51:06.141 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.000000000, 2024-04-02T14:51:06.142 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000000000 2024-04-02T14:51:06.142 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:51:06.142 INFO:teuthology.orchestra.run.smithi090.stdout: "req_lookuphash_latency": { 2024-04-02T14:51:06.142 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 0, 2024-04-02T14:51:06.142 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.000000000, 2024-04-02T14:51:06.142 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000000000 2024-04-02T14:51:06.142 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:51:06.142 INFO:teuthology.orchestra.run.smithi090.stdout: "req_lookupino_latency": { 2024-04-02T14:51:06.142 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 0, 2024-04-02T14:51:06.142 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.000000000, 2024-04-02T14:51:06.143 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000000000 2024-04-02T14:51:06.143 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:51:06.143 INFO:teuthology.orchestra.run.smithi090.stdout: "req_lookupname_latency": { 2024-04-02T14:51:06.143 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 0, 2024-04-02T14:51:06.143 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.000000000, 2024-04-02T14:51:06.143 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000000000 2024-04-02T14:51:06.143 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:51:06.143 INFO:teuthology.orchestra.run.smithi090.stdout: "req_lookupparent_latency": { 2024-04-02T14:51:06.143 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 0, 2024-04-02T14:51:06.143 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.000000000, 2024-04-02T14:51:06.143 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000000000 2024-04-02T14:51:06.143 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:51:06.143 INFO:teuthology.orchestra.run.smithi090.stdout: "req_lookupsnap_latency": { 2024-04-02T14:51:06.143 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 0, 2024-04-02T14:51:06.143 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.000000000, 2024-04-02T14:51:06.144 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000000000 2024-04-02T14:51:06.144 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:51:06.144 INFO:teuthology.orchestra.run.smithi090.stdout: "req_lssnap_latency": { 2024-04-02T14:51:06.144 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 0, 2024-04-02T14:51:06.144 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.000000000, 2024-04-02T14:51:06.144 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000000000 2024-04-02T14:51:06.144 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:51:06.144 INFO:teuthology.orchestra.run.smithi090.stdout: "req_mkdir_latency": { 2024-04-02T14:51:06.144 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 0, 2024-04-02T14:51:06.145 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.000000000, 2024-04-02T14:51:06.145 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000000000 2024-04-02T14:51:06.145 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:51:06.145 INFO:teuthology.orchestra.run.smithi090.stdout: "req_mknod_latency": { 2024-04-02T14:51:06.145 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 0, 2024-04-02T14:51:06.145 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.000000000, 2024-04-02T14:51:06.145 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000000000 2024-04-02T14:51:06.145 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:51:06.145 INFO:teuthology.orchestra.run.smithi090.stdout: "req_mksnap_latency": { 2024-04-02T14:51:06.145 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 0, 2024-04-02T14:51:06.146 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.000000000, 2024-04-02T14:51:06.146 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000000000 2024-04-02T14:51:06.146 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:51:06.146 INFO:teuthology.orchestra.run.smithi090.stdout: "req_open_latency": { 2024-04-02T14:51:06.146 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 0, 2024-04-02T14:51:06.146 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.000000000, 2024-04-02T14:51:06.146 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000000000 2024-04-02T14:51:06.146 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:51:06.146 INFO:teuthology.orchestra.run.smithi090.stdout: "req_readdir_latency": { 2024-04-02T14:51:06.146 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 0, 2024-04-02T14:51:06.146 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.000000000, 2024-04-02T14:51:06.146 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000000000 2024-04-02T14:51:06.146 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:51:06.147 INFO:teuthology.orchestra.run.smithi090.stdout: "req_rename_latency": { 2024-04-02T14:51:06.147 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 0, 2024-04-02T14:51:06.147 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.000000000, 2024-04-02T14:51:06.147 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000000000 2024-04-02T14:51:06.147 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:51:06.147 INFO:teuthology.orchestra.run.smithi090.stdout: "req_renamesnap_latency": { 2024-04-02T14:51:06.147 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 0, 2024-04-02T14:51:06.147 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.000000000, 2024-04-02T14:51:06.147 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000000000 2024-04-02T14:51:06.147 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:51:06.147 INFO:teuthology.orchestra.run.smithi090.stdout: "req_snapdiff_latency": { 2024-04-02T14:51:06.147 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 0, 2024-04-02T14:51:06.147 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.000000000, 2024-04-02T14:51:06.147 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000000000 2024-04-02T14:51:06.147 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:51:06.147 INFO:teuthology.orchestra.run.smithi090.stdout: "req_rmdir_latency": { 2024-04-02T14:51:06.148 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 0, 2024-04-02T14:51:06.148 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.000000000, 2024-04-02T14:51:06.148 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000000000 2024-04-02T14:51:06.148 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:51:06.148 INFO:teuthology.orchestra.run.smithi090.stdout: "req_rmsnap_latency": { 2024-04-02T14:51:06.148 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 0, 2024-04-02T14:51:06.148 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.000000000, 2024-04-02T14:51:06.148 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000000000 2024-04-02T14:51:06.148 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:51:06.148 INFO:teuthology.orchestra.run.smithi090.stdout: "req_rmxattr_latency": { 2024-04-02T14:51:06.148 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 0, 2024-04-02T14:51:06.148 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.000000000, 2024-04-02T14:51:06.148 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000000000 2024-04-02T14:51:06.148 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:51:06.148 INFO:teuthology.orchestra.run.smithi090.stdout: "req_setattr_latency": { 2024-04-02T14:51:06.149 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 0, 2024-04-02T14:51:06.149 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.000000000, 2024-04-02T14:51:06.149 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000000000 2024-04-02T14:51:06.149 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:51:06.149 INFO:teuthology.orchestra.run.smithi090.stdout: "req_setdirlayout_latency": { 2024-04-02T14:51:06.149 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 0, 2024-04-02T14:51:06.149 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.000000000, 2024-04-02T14:51:06.149 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000000000 2024-04-02T14:51:06.149 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:51:06.149 INFO:teuthology.orchestra.run.smithi090.stdout: "req_setfilelock_latency": { 2024-04-02T14:51:06.149 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 0, 2024-04-02T14:51:06.149 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.000000000, 2024-04-02T14:51:06.149 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000000000 2024-04-02T14:51:06.149 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:51:06.149 INFO:teuthology.orchestra.run.smithi090.stdout: "req_setlayout_latency": { 2024-04-02T14:51:06.150 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 0, 2024-04-02T14:51:06.150 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.000000000, 2024-04-02T14:51:06.150 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000000000 2024-04-02T14:51:06.150 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:51:06.150 INFO:teuthology.orchestra.run.smithi090.stdout: "req_setxattr_latency": { 2024-04-02T14:51:06.150 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 0, 2024-04-02T14:51:06.150 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.000000000, 2024-04-02T14:51:06.150 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000000000 2024-04-02T14:51:06.150 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:51:06.150 INFO:teuthology.orchestra.run.smithi090.stdout: "req_symlink_latency": { 2024-04-02T14:51:06.150 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 0, 2024-04-02T14:51:06.150 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.000000000, 2024-04-02T14:51:06.150 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000000000 2024-04-02T14:51:06.150 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:51:06.150 INFO:teuthology.orchestra.run.smithi090.stdout: "req_unlink_latency": { 2024-04-02T14:51:06.151 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 0, 2024-04-02T14:51:06.151 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.000000000, 2024-04-02T14:51:06.151 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000000000 2024-04-02T14:51:06.151 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:51:06.151 INFO:teuthology.orchestra.run.smithi090.stdout: "cap_revoke_eviction": 0, 2024-04-02T14:51:06.151 INFO:teuthology.orchestra.run.smithi090.stdout: "cap_acquisition_throttle": 0, 2024-04-02T14:51:06.151 INFO:teuthology.orchestra.run.smithi090.stdout: "req_getvxattr_latency": { 2024-04-02T14:51:06.151 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 0, 2024-04-02T14:51:06.151 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.000000000, 2024-04-02T14:51:06.151 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000000000 2024-04-02T14:51:06.151 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:51:06.151 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:51:06.151 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_sessions": { 2024-04-02T14:51:06.151 INFO:teuthology.orchestra.run.smithi090.stdout: "session_count": 2, 2024-04-02T14:51:06.151 INFO:teuthology.orchestra.run.smithi090.stdout: "session_add": 6, 2024-04-02T14:51:06.152 INFO:teuthology.orchestra.run.smithi090.stdout: "session_remove": 4, 2024-04-02T14:51:06.152 INFO:teuthology.orchestra.run.smithi090.stdout: "sessions_open": 5, 2024-04-02T14:51:06.152 INFO:teuthology.orchestra.run.smithi090.stdout: "sessions_stale": 0, 2024-04-02T14:51:06.152 INFO:teuthology.orchestra.run.smithi090.stdout: "total_load": 0, 2024-04-02T14:51:06.152 INFO:teuthology.orchestra.run.smithi090.stdout: "average_load": 0, 2024-04-02T14:51:06.152 INFO:teuthology.orchestra.run.smithi090.stdout: "avg_session_uptime": 4301, 2024-04-02T14:51:06.152 INFO:teuthology.orchestra.run.smithi090.stdout: "mdthresh_evicted": 0 2024-04-02T14:51:06.152 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:51:06.152 INFO:teuthology.orchestra.run.smithi090.stdout: "mempool": { 2024-04-02T14:51:06.152 INFO:teuthology.orchestra.run.smithi090.stdout: "bloom_filter_bytes": 0, 2024-04-02T14:51:06.152 INFO:teuthology.orchestra.run.smithi090.stdout: "bloom_filter_items": 0, 2024-04-02T14:51:06.152 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_alloc_bytes": 0, 2024-04-02T14:51:06.152 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_alloc_items": 0, 2024-04-02T14:51:06.152 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_cache_data_bytes": 0, 2024-04-02T14:51:06.153 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_cache_data_items": 0, 2024-04-02T14:51:06.153 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-02T14:51:06.153 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_cache_onode_items": 0, 2024-04-02T14:51:06.153 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-02T14:51:06.153 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_cache_meta_items": 0, 2024-04-02T14:51:06.153 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_cache_other_bytes": 0, 2024-04-02T14:51:06.153 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_cache_other_items": 0, 2024-04-02T14:51:06.153 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-02T14:51:06.153 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_cache_buffer_items": 0, 2024-04-02T14:51:06.153 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_extent_bytes": 0, 2024-04-02T14:51:06.154 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_extent_items": 0, 2024-04-02T14:51:06.154 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_blob_bytes": 0, 2024-04-02T14:51:06.154 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_blob_items": 0, 2024-04-02T14:51:06.154 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-02T14:51:06.154 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_shared_blob_items": 0, 2024-04-02T14:51:06.154 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-02T14:51:06.154 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_inline_bl_items": 0, 2024-04-02T14:51:06.154 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_fsck_bytes": 0, 2024-04-02T14:51:06.154 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_fsck_items": 0, 2024-04-02T14:51:06.154 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_txc_bytes": 0, 2024-04-02T14:51:06.154 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_txc_items": 0, 2024-04-02T14:51:06.154 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-02T14:51:06.154 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_writing_deferred_items": 0, 2024-04-02T14:51:06.154 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_writing_bytes": 0, 2024-04-02T14:51:06.155 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_writing_items": 0, 2024-04-02T14:51:06.155 INFO:teuthology.orchestra.run.smithi090.stdout: "bluefs_bytes": 0, 2024-04-02T14:51:06.155 INFO:teuthology.orchestra.run.smithi090.stdout: "bluefs_items": 0, 2024-04-02T14:51:06.155 INFO:teuthology.orchestra.run.smithi090.stdout: "bluefs_file_reader_bytes": 0, 2024-04-02T14:51:06.155 INFO:teuthology.orchestra.run.smithi090.stdout: "bluefs_file_reader_items": 0, 2024-04-02T14:51:06.155 INFO:teuthology.orchestra.run.smithi090.stdout: "bluefs_file_writer_bytes": 0, 2024-04-02T14:51:06.155 INFO:teuthology.orchestra.run.smithi090.stdout: "bluefs_file_writer_items": 0, 2024-04-02T14:51:06.155 INFO:teuthology.orchestra.run.smithi090.stdout: "buffer_anon_bytes": 190945, 2024-04-02T14:51:06.155 INFO:teuthology.orchestra.run.smithi090.stdout: "buffer_anon_items": 54, 2024-04-02T14:51:06.155 INFO:teuthology.orchestra.run.smithi090.stdout: "buffer_meta_bytes": 0, 2024-04-02T14:51:06.155 INFO:teuthology.orchestra.run.smithi090.stdout: "buffer_meta_items": 0, 2024-04-02T14:51:06.155 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_bytes": 0, 2024-04-02T14:51:06.155 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_items": 0, 2024-04-02T14:51:06.155 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_mapbl_bytes": 0, 2024-04-02T14:51:06.156 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_mapbl_items": 0, 2024-04-02T14:51:06.156 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_pglog_bytes": 0, 2024-04-02T14:51:06.156 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_pglog_items": 0, 2024-04-02T14:51:06.156 INFO:teuthology.orchestra.run.smithi090.stdout: "osdmap_bytes": 8136, 2024-04-02T14:51:06.156 INFO:teuthology.orchestra.run.smithi090.stdout: "osdmap_items": 170, 2024-04-02T14:51:06.156 INFO:teuthology.orchestra.run.smithi090.stdout: "osdmap_mapping_bytes": 0, 2024-04-02T14:51:06.156 INFO:teuthology.orchestra.run.smithi090.stdout: "osdmap_mapping_items": 0, 2024-04-02T14:51:06.156 INFO:teuthology.orchestra.run.smithi090.stdout: "pgmap_bytes": 0, 2024-04-02T14:51:06.156 INFO:teuthology.orchestra.run.smithi090.stdout: "pgmap_items": 0, 2024-04-02T14:51:06.156 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_co_bytes": 8464, 2024-04-02T14:51:06.156 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_co_items": 13, 2024-04-02T14:51:06.156 INFO:teuthology.orchestra.run.smithi090.stdout: "unittest_1_bytes": 0, 2024-04-02T14:51:06.156 INFO:teuthology.orchestra.run.smithi090.stdout: "unittest_1_items": 0, 2024-04-02T14:51:06.156 INFO:teuthology.orchestra.run.smithi090.stdout: "unittest_2_bytes": 0, 2024-04-02T14:51:06.156 INFO:teuthology.orchestra.run.smithi090.stdout: "unittest_2_items": 0 2024-04-02T14:51:06.157 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:51:06.157 INFO:teuthology.orchestra.run.smithi090.stdout: "objecter": { 2024-04-02T14:51:06.157 INFO:teuthology.orchestra.run.smithi090.stdout: "op_active": 0, 2024-04-02T14:51:06.157 INFO:teuthology.orchestra.run.smithi090.stdout: "op_laggy": 0, 2024-04-02T14:51:06.157 INFO:teuthology.orchestra.run.smithi090.stdout: "op_send": 3259, 2024-04-02T14:51:06.157 INFO:teuthology.orchestra.run.smithi090.stdout: "op_send_bytes": 21, 2024-04-02T14:51:06.157 INFO:teuthology.orchestra.run.smithi090.stdout: "op_resend": 3, 2024-04-02T14:51:06.157 INFO:teuthology.orchestra.run.smithi090.stdout: "op_reply": 3259, 2024-04-02T14:51:06.157 INFO:teuthology.orchestra.run.smithi090.stdout: "op_latency": { 2024-04-02T14:51:06.157 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 3259, 2024-04-02T14:51:06.157 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 13.130798879, 2024-04-02T14:51:06.157 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.004029088 2024-04-02T14:51:06.157 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:51:06.157 INFO:teuthology.orchestra.run.smithi090.stdout: "op_inflight": 0, 2024-04-02T14:51:06.157 INFO:teuthology.orchestra.run.smithi090.stdout: "oplen_avg": { 2024-04-02T14:51:06.158 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 3259, 2024-04-02T14:51:06.158 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 3260 2024-04-02T14:51:06.158 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:51:06.158 INFO:teuthology.orchestra.run.smithi090.stdout: "op": 3259, 2024-04-02T14:51:06.158 INFO:teuthology.orchestra.run.smithi090.stdout: "op_r": 3259, 2024-04-02T14:51:06.158 INFO:teuthology.orchestra.run.smithi090.stdout: "op_w": 0, 2024-04-02T14:51:06.158 INFO:teuthology.orchestra.run.smithi090.stdout: "op_rmw": 0, 2024-04-02T14:51:06.158 INFO:teuthology.orchestra.run.smithi090.stdout: "op_pg": 0, 2024-04-02T14:51:06.158 INFO:teuthology.orchestra.run.smithi090.stdout: "osdop_stat": 2143, 2024-04-02T14:51:06.158 INFO:teuthology.orchestra.run.smithi090.stdout: "osdop_create": 0, 2024-04-02T14:51:06.158 INFO:teuthology.orchestra.run.smithi090.stdout: "osdop_read": 1114, 2024-04-02T14:51:06.158 INFO:teuthology.orchestra.run.smithi090.stdout: "osdop_write": 0, 2024-04-02T14:51:06.158 INFO:teuthology.orchestra.run.smithi090.stdout: "osdop_writefull": 0, 2024-04-02T14:51:06.159 INFO:teuthology.orchestra.run.smithi090.stdout: "osdop_writesame": 0, 2024-04-02T14:51:06.159 INFO:teuthology.orchestra.run.smithi090.stdout: "osdop_append": 0, 2024-04-02T14:51:06.159 INFO:teuthology.orchestra.run.smithi090.stdout: "osdop_zero": 0, 2024-04-02T14:51:06.159 INFO:teuthology.orchestra.run.smithi090.stdout: "osdop_truncate": 0, 2024-04-02T14:51:06.159 INFO:teuthology.orchestra.run.smithi090.stdout: "osdop_delete": 0, 2024-04-02T14:51:06.159 INFO:teuthology.orchestra.run.smithi090.stdout: "osdop_mapext": 0, 2024-04-02T14:51:06.159 INFO:teuthology.orchestra.run.smithi090.stdout: "osdop_sparse_read": 0, 2024-04-02T14:51:06.159 INFO:teuthology.orchestra.run.smithi090.stdout: "osdop_clonerange": 0, 2024-04-02T14:51:06.159 INFO:teuthology.orchestra.run.smithi090.stdout: "osdop_getxattr": 1, 2024-04-02T14:51:06.159 INFO:teuthology.orchestra.run.smithi090.stdout: "osdop_setxattr": 0, 2024-04-02T14:51:06.159 INFO:teuthology.orchestra.run.smithi090.stdout: "osdop_cmpxattr": 0, 2024-04-02T14:51:06.159 INFO:teuthology.orchestra.run.smithi090.stdout: "osdop_rmxattr": 0, 2024-04-02T14:51:06.159 INFO:teuthology.orchestra.run.smithi090.stdout: "osdop_resetxattrs": 0, 2024-04-02T14:51:06.160 INFO:teuthology.orchestra.run.smithi090.stdout: "osdop_call": 0, 2024-04-02T14:51:06.160 INFO:teuthology.orchestra.run.smithi090.stdout: "osdop_watch": 0, 2024-04-02T14:51:06.160 INFO:teuthology.orchestra.run.smithi090.stdout: "osdop_notify": 0, 2024-04-02T14:51:06.160 INFO:teuthology.orchestra.run.smithi090.stdout: "osdop_src_cmpxattr": 0, 2024-04-02T14:51:06.160 INFO:teuthology.orchestra.run.smithi090.stdout: "osdop_pgls": 0, 2024-04-02T14:51:06.160 INFO:teuthology.orchestra.run.smithi090.stdout: "osdop_pgls_filter": 0, 2024-04-02T14:51:06.160 INFO:teuthology.orchestra.run.smithi090.stdout: "osdop_other": 0, 2024-04-02T14:51:06.160 INFO:teuthology.orchestra.run.smithi090.stdout: "linger_active": 0, 2024-04-02T14:51:06.160 INFO:teuthology.orchestra.run.smithi090.stdout: "linger_send": 0, 2024-04-02T14:51:06.160 INFO:teuthology.orchestra.run.smithi090.stdout: "linger_resend": 0, 2024-04-02T14:51:06.160 INFO:teuthology.orchestra.run.smithi090.stdout: "linger_ping": 0, 2024-04-02T14:51:06.160 INFO:teuthology.orchestra.run.smithi090.stdout: "poolop_active": 0, 2024-04-02T14:51:06.160 INFO:teuthology.orchestra.run.smithi090.stdout: "poolop_send": 0, 2024-04-02T14:51:06.160 INFO:teuthology.orchestra.run.smithi090.stdout: "poolop_resend": 0, 2024-04-02T14:51:06.161 INFO:teuthology.orchestra.run.smithi090.stdout: "poolstat_active": 0, 2024-04-02T14:51:06.161 INFO:teuthology.orchestra.run.smithi090.stdout: "poolstat_send": 0, 2024-04-02T14:51:06.161 INFO:teuthology.orchestra.run.smithi090.stdout: "poolstat_resend": 0, 2024-04-02T14:51:06.161 INFO:teuthology.orchestra.run.smithi090.stdout: "statfs_active": 0, 2024-04-02T14:51:06.161 INFO:teuthology.orchestra.run.smithi090.stdout: "statfs_send": 0, 2024-04-02T14:51:06.161 INFO:teuthology.orchestra.run.smithi090.stdout: "statfs_resend": 0, 2024-04-02T14:51:06.161 INFO:teuthology.orchestra.run.smithi090.stdout: "command_active": 0, 2024-04-02T14:51:06.161 INFO:teuthology.orchestra.run.smithi090.stdout: "command_send": 0, 2024-04-02T14:51:06.161 INFO:teuthology.orchestra.run.smithi090.stdout: "command_resend": 0, 2024-04-02T14:51:06.161 INFO:teuthology.orchestra.run.smithi090.stdout: "map_epoch": 109, 2024-04-02T14:51:06.161 INFO:teuthology.orchestra.run.smithi090.stdout: "map_full": 0, 2024-04-02T14:51:06.161 INFO:teuthology.orchestra.run.smithi090.stdout: "map_inc": 21, 2024-04-02T14:51:06.161 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_sessions": 5, 2024-04-02T14:51:06.161 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_session_open": 6, 2024-04-02T14:51:06.161 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_session_close": 1, 2024-04-02T14:51:06.162 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_laggy": 0, 2024-04-02T14:51:06.162 INFO:teuthology.orchestra.run.smithi090.stdout: "omap_wr": 0, 2024-04-02T14:51:06.162 INFO:teuthology.orchestra.run.smithi090.stdout: "omap_rd": 2, 2024-04-02T14:51:06.162 INFO:teuthology.orchestra.run.smithi090.stdout: "omap_del": 0 2024-04-02T14:51:06.162 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:51:06.162 INFO:teuthology.orchestra.run.smithi090.stdout: "oft": { 2024-04-02T14:51:06.162 INFO:teuthology.orchestra.run.smithi090.stdout: "omap_total_objs": 0, 2024-04-02T14:51:06.162 INFO:teuthology.orchestra.run.smithi090.stdout: "omap_total_kv_pairs": 0, 2024-04-02T14:51:06.162 INFO:teuthology.orchestra.run.smithi090.stdout: "omap_total_updates": 0, 2024-04-02T14:51:06.162 INFO:teuthology.orchestra.run.smithi090.stdout: "omap_total_removes": 0 2024-04-02T14:51:06.162 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:51:06.162 INFO:teuthology.orchestra.run.smithi090.stdout: "osd-slow-ops": { 2024-04-02T14:51:06.162 INFO:teuthology.orchestra.run.smithi090.stdout: "slow_ops_count": 0 2024-04-02T14:51:06.163 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:51:06.163 INFO:teuthology.orchestra.run.smithi090.stdout: "purge_queue": { 2024-04-02T14:51:06.163 INFO:teuthology.orchestra.run.smithi090.stdout: "pq_executing_ops": 0, 2024-04-02T14:51:06.163 INFO:teuthology.orchestra.run.smithi090.stdout: "pq_executing_ops_high_water": 0, 2024-04-02T14:51:06.163 INFO:teuthology.orchestra.run.smithi090.stdout: "pq_executing": 0, 2024-04-02T14:51:06.163 INFO:teuthology.orchestra.run.smithi090.stdout: "pq_executing_high_water": 0, 2024-04-02T14:51:06.163 INFO:teuthology.orchestra.run.smithi090.stdout: "pq_executed_ops": 0, 2024-04-02T14:51:06.163 INFO:teuthology.orchestra.run.smithi090.stdout: "pq_executed": 0, 2024-04-02T14:51:06.163 INFO:teuthology.orchestra.run.smithi090.stdout: "pq_item_in_journal": 0 2024-04-02T14:51:06.163 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:51:06.163 INFO:teuthology.orchestra.run.smithi090.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-02T14:51:06.164 INFO:teuthology.orchestra.run.smithi090.stdout: "val": 0, 2024-04-02T14:51:06.164 INFO:teuthology.orchestra.run.smithi090.stdout: "max": 104857600, 2024-04-02T14:51:06.164 INFO:teuthology.orchestra.run.smithi090.stdout: "get_started": 0, 2024-04-02T14:51:06.164 INFO:teuthology.orchestra.run.smithi090.stdout: "get": 3607, 2024-04-02T14:51:06.164 INFO:teuthology.orchestra.run.smithi090.stdout: "get_sum": 2234414, 2024-04-02T14:51:06.164 INFO:teuthology.orchestra.run.smithi090.stdout: "get_or_fail_fail": 0, 2024-04-02T14:51:06.164 INFO:teuthology.orchestra.run.smithi090.stdout: "get_or_fail_success": 3607, 2024-04-02T14:51:06.164 INFO:teuthology.orchestra.run.smithi090.stdout: "take": 0, 2024-04-02T14:51:06.164 INFO:teuthology.orchestra.run.smithi090.stdout: "take_sum": 0, 2024-04-02T14:51:06.164 INFO:teuthology.orchestra.run.smithi090.stdout: "put": 3607, 2024-04-02T14:51:06.164 INFO:teuthology.orchestra.run.smithi090.stdout: "put_sum": 2234414, 2024-04-02T14:51:06.164 INFO:teuthology.orchestra.run.smithi090.stdout: "wait": { 2024-04-02T14:51:06.165 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 0, 2024-04-02T14:51:06.165 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.000000000, 2024-04-02T14:51:06.165 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000000000 2024-04-02T14:51:06.165 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:51:06.165 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:51:06.165 INFO:teuthology.orchestra.run.smithi090.stdout: "throttle-objecter_bytes": { 2024-04-02T14:51:06.165 INFO:teuthology.orchestra.run.smithi090.stdout: "val": 0, 2024-04-02T14:51:06.165 INFO:teuthology.orchestra.run.smithi090.stdout: "max": 104857600, 2024-04-02T14:51:06.165 INFO:teuthology.orchestra.run.smithi090.stdout: "get_started": 0, 2024-04-02T14:51:06.165 INFO:teuthology.orchestra.run.smithi090.stdout: "get": 0, 2024-04-02T14:51:06.165 INFO:teuthology.orchestra.run.smithi090.stdout: "get_sum": 0, 2024-04-02T14:51:06.165 INFO:teuthology.orchestra.run.smithi090.stdout: "get_or_fail_fail": 0, 2024-04-02T14:51:06.165 INFO:teuthology.orchestra.run.smithi090.stdout: "get_or_fail_success": 0, 2024-04-02T14:51:06.165 INFO:teuthology.orchestra.run.smithi090.stdout: "take": 3259, 2024-04-02T14:51:06.165 INFO:teuthology.orchestra.run.smithi090.stdout: "take_sum": 3418, 2024-04-02T14:51:06.165 INFO:teuthology.orchestra.run.smithi090.stdout: "put": 3259, 2024-04-02T14:51:06.166 INFO:teuthology.orchestra.run.smithi090.stdout: "put_sum": 3418, 2024-04-02T14:51:06.166 INFO:teuthology.orchestra.run.smithi090.stdout: "wait": { 2024-04-02T14:51:06.166 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 0, 2024-04-02T14:51:06.166 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.000000000, 2024-04-02T14:51:06.166 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000000000 2024-04-02T14:51:06.166 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:51:06.166 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:51:06.166 INFO:teuthology.orchestra.run.smithi090.stdout: "throttle-objecter_ops": { 2024-04-02T14:51:06.166 INFO:teuthology.orchestra.run.smithi090.stdout: "val": 0, 2024-04-02T14:51:06.166 INFO:teuthology.orchestra.run.smithi090.stdout: "max": 1024, 2024-04-02T14:51:06.166 INFO:teuthology.orchestra.run.smithi090.stdout: "get_started": 0, 2024-04-02T14:51:06.166 INFO:teuthology.orchestra.run.smithi090.stdout: "get": 0, 2024-04-02T14:51:06.166 INFO:teuthology.orchestra.run.smithi090.stdout: "get_sum": 0, 2024-04-02T14:51:06.166 INFO:teuthology.orchestra.run.smithi090.stdout: "get_or_fail_fail": 0, 2024-04-02T14:51:06.167 INFO:teuthology.orchestra.run.smithi090.stdout: "get_or_fail_success": 0, 2024-04-02T14:51:06.167 INFO:teuthology.orchestra.run.smithi090.stdout: "take": 3259, 2024-04-02T14:51:06.167 INFO:teuthology.orchestra.run.smithi090.stdout: "take_sum": 3259, 2024-04-02T14:51:06.167 INFO:teuthology.orchestra.run.smithi090.stdout: "put": 3259, 2024-04-02T14:51:06.167 INFO:teuthology.orchestra.run.smithi090.stdout: "put_sum": 3259, 2024-04-02T14:51:06.167 INFO:teuthology.orchestra.run.smithi090.stdout: "wait": { 2024-04-02T14:51:06.167 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 0, 2024-04-02T14:51:06.167 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.000000000, 2024-04-02T14:51:06.167 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000000000 2024-04-02T14:51:06.167 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:51:06.167 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:51:06.167 INFO:teuthology.orchestra.run.smithi090.stdout: "throttle-write_buf_throttle": { 2024-04-02T14:51:06.167 INFO:teuthology.orchestra.run.smithi090.stdout: "val": 0, 2024-04-02T14:51:06.167 INFO:teuthology.orchestra.run.smithi090.stdout: "max": 3758096384, 2024-04-02T14:51:06.168 INFO:teuthology.orchestra.run.smithi090.stdout: "get_started": 0, 2024-04-02T14:51:06.168 INFO:teuthology.orchestra.run.smithi090.stdout: "get": 0, 2024-04-02T14:51:06.168 INFO:teuthology.orchestra.run.smithi090.stdout: "get_sum": 0, 2024-04-02T14:51:06.168 INFO:teuthology.orchestra.run.smithi090.stdout: "get_or_fail_fail": 0, 2024-04-02T14:51:06.168 INFO:teuthology.orchestra.run.smithi090.stdout: "get_or_fail_success": 0, 2024-04-02T14:51:06.168 INFO:teuthology.orchestra.run.smithi090.stdout: "take": 0, 2024-04-02T14:51:06.168 INFO:teuthology.orchestra.run.smithi090.stdout: "take_sum": 0, 2024-04-02T14:51:06.168 INFO:teuthology.orchestra.run.smithi090.stdout: "put": 0, 2024-04-02T14:51:06.168 INFO:teuthology.orchestra.run.smithi090.stdout: "put_sum": 0, 2024-04-02T14:51:06.168 INFO:teuthology.orchestra.run.smithi090.stdout: "wait": { 2024-04-02T14:51:06.168 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 0, 2024-04-02T14:51:06.168 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.000000000, 2024-04-02T14:51:06.168 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000000000 2024-04-02T14:51:06.168 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:51:06.168 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-02T14:51:06.168 INFO:teuthology.orchestra.run.smithi090.stdout: "throttle-write_buf_throttle-0x55e14b6d6780": { 2024-04-02T14:51:06.169 INFO:teuthology.orchestra.run.smithi090.stdout: "val": 0, 2024-04-02T14:51:06.169 INFO:teuthology.orchestra.run.smithi090.stdout: "max": 3758096384, 2024-04-02T14:51:06.169 INFO:teuthology.orchestra.run.smithi090.stdout: "get_started": 0, 2024-04-02T14:51:06.169 INFO:teuthology.orchestra.run.smithi090.stdout: "get": 0, 2024-04-02T14:51:06.169 INFO:teuthology.orchestra.run.smithi090.stdout: "get_sum": 0, 2024-04-02T14:51:06.169 INFO:teuthology.orchestra.run.smithi090.stdout: "get_or_fail_fail": 0, 2024-04-02T14:51:06.169 INFO:teuthology.orchestra.run.smithi090.stdout: "get_or_fail_success": 0, 2024-04-02T14:51:06.169 INFO:teuthology.orchestra.run.smithi090.stdout: "take": 0, 2024-04-02T14:51:06.169 INFO:teuthology.orchestra.run.smithi090.stdout: "take_sum": 0, 2024-04-02T14:51:06.169 INFO:teuthology.orchestra.run.smithi090.stdout: "put": 0, 2024-04-02T14:51:06.169 INFO:teuthology.orchestra.run.smithi090.stdout: "put_sum": 0, 2024-04-02T14:51:06.169 INFO:teuthology.orchestra.run.smithi090.stdout: "wait": { 2024-04-02T14:51:06.169 INFO:teuthology.orchestra.run.smithi090.stdout: "avgcount": 0, 2024-04-02T14:51:06.169 INFO:teuthology.orchestra.run.smithi090.stdout: "sum": 0.000000000, 2024-04-02T14:51:06.169 INFO:teuthology.orchestra.run.smithi090.stdout: "avgtime": 0.000000000 2024-04-02T14:51:06.170 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:51:06.170 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-02T14:51:06.170 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-02T14:51:06.177 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:51:05 smithi177 ceph-mon[36431]: pgmap v507: 97 pgs: 97 active+clean; 4.3 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 87 KiB/s wr, 32 op/s 2024-04-02T14:51:07.234 INFO:tasks.check_counter:Daemon mds.j mds.exported=0 2024-04-02T14:51:07.234 INFO:tasks.check_counter:Daemon mds.j mds.imported=0 2024-04-02T14:51:07.234 INFO:tasks.check_counter:Daemon mds.j mds_cache.dir_update=0 2024-04-02T14:51:07.234 INFO:tasks.check_counter:Daemon mds.j mds_cache.dir_update_receipt=0 2024-04-02T14:51:07.234 INFO:tasks.check_counter:Daemon mds.j mds.root_rsnaps=0 2024-04-02T14:51:07.234 INFO:tasks.check_counter:Daemon mds.j mds_server.req_mksnap_latency.avgcount=0 2024-04-02T14:51:07.234 INFO:tasks.check_counter:Daemon mds.j mds_server.req_rmsnap_latency.avgcount=0 2024-04-02T14:51:07.234 DEBUG:tasks.check_counter:Getting stats from b 2024-04-02T14:51:07.235 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:191d963ac9de7322da8aff20b567b3fa458e2ef6 shell --fsid 6449448a-f0fc-11ee-b647-cb9ed24678a4 -- ceph daemon mds.b perf dump 2024-04-02T14:51:07.482 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/mon.b/config 2024-04-02T14:51:07.856 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:51:07 smithi092 ceph-mon[36497]: pgmap v508: 97 pgs: 97 active+clean; 4.3 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 89 KiB/s wr, 30 op/s 2024-04-02T14:51:07.856 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:51:07 smithi092 ceph-mon[36497]: osdmap e110: 12 total, 12 up, 12 in 2024-04-02T14:51:08.178 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:51:07 smithi177 ceph-mon[36431]: pgmap v508: 97 pgs: 97 active+clean; 4.3 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 89 KiB/s wr, 30 op/s 2024-04-02T14:51:08.178 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:51:07 smithi177 ceph-mon[36431]: osdmap e110: 12 total, 12 up, 12 in 2024-04-02T14:51:08.180 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:51:07 smithi090 ceph-mon[31674]: pgmap v508: 97 pgs: 97 active+clean; 4.3 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 89 KiB/s wr, 30 op/s 2024-04-02T14:51:08.180 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:51:07 smithi090 ceph-mon[31674]: osdmap e110: 12 total, 12 up, 12 in 2024-04-02T14:51:09.304 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-04-02T14:51:09.304 INFO:teuthology.orchestra.run.smithi092.stdout: "AsyncMessenger::Worker-0": { 2024-04-02T14:51:09.304 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_recv_messages": 16940, 2024-04-02T14:51:09.304 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_send_messages": 20890, 2024-04-02T14:51:09.304 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_recv_bytes": 8046982, 2024-04-02T14:51:09.305 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_send_bytes": 25541501, 2024-04-02T14:51:09.305 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_created_connections": 23, 2024-04-02T14:51:09.305 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_active_connections": 4, 2024-04-02T14:51:09.305 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_running_total_time": 1.005618524, 2024-04-02T14:51:09.305 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_running_send_time": 0.348683720, 2024-04-02T14:51:09.305 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_running_recv_time": 0.301933864, 2024-04-02T14:51:09.305 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_running_fast_dispatch_time": 0.268124418, 2024-04-02T14:51:09.305 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_send_messages_queue_lat": { 2024-04-02T14:51:09.305 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 20889, 2024-04-02T14:51:09.305 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 1.502502609, 2024-04-02T14:51:09.305 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000071927 2024-04-02T14:51:09.305 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:09.305 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_handle_ack_lat": { 2024-04-02T14:51:09.305 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 10586, 2024-04-02T14:51:09.305 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.001786385, 2024-04-02T14:51:09.306 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000168 2024-04-02T14:51:09.306 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:09.306 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_recv_encrypted_bytes": 3232, 2024-04-02T14:51:09.306 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_send_encrypted_bytes": 443088 2024-04-02T14:51:09.306 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:09.306 INFO:teuthology.orchestra.run.smithi092.stdout: "AsyncMessenger::Worker-1": { 2024-04-02T14:51:09.306 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_recv_messages": 27054, 2024-04-02T14:51:09.306 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_send_messages": 28868, 2024-04-02T14:51:09.306 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_recv_bytes": 14243581, 2024-04-02T14:51:09.306 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_send_bytes": 34726687, 2024-04-02T14:51:09.306 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_created_connections": 18, 2024-04-02T14:51:09.306 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_active_connections": 5, 2024-04-02T14:51:09.306 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_running_total_time": 1.558009046, 2024-04-02T14:51:09.306 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_running_send_time": 0.573282559, 2024-04-02T14:51:09.307 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_running_recv_time": 0.723406656, 2024-04-02T14:51:09.307 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_running_fast_dispatch_time": 0.335563821, 2024-04-02T14:51:09.307 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_send_messages_queue_lat": { 2024-04-02T14:51:09.307 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 28867, 2024-04-02T14:51:09.307 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 2.752861519, 2024-04-02T14:51:09.307 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000095363 2024-04-02T14:51:09.307 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:09.307 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_handle_ack_lat": { 2024-04-02T14:51:09.307 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 20118, 2024-04-02T14:51:09.307 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.002738856, 2024-04-02T14:51:09.307 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000136 2024-04-02T14:51:09.307 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:09.307 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_recv_encrypted_bytes": 5670896, 2024-04-02T14:51:09.307 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_send_encrypted_bytes": 7517152 2024-04-02T14:51:09.307 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:09.308 INFO:teuthology.orchestra.run.smithi092.stdout: "AsyncMessenger::Worker-2": { 2024-04-02T14:51:09.308 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_recv_messages": 12961, 2024-04-02T14:51:09.308 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_send_messages": 15710, 2024-04-02T14:51:09.308 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_recv_bytes": 5792596, 2024-04-02T14:51:09.308 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_send_bytes": 18550217, 2024-04-02T14:51:09.308 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_created_connections": 361, 2024-04-02T14:51:09.308 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_active_connections": 6, 2024-04-02T14:51:09.308 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_running_total_time": 1.024714774, 2024-04-02T14:51:09.308 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_running_send_time": 0.347814989, 2024-04-02T14:51:09.308 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_running_recv_time": 0.399419354, 2024-04-02T14:51:09.308 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_running_fast_dispatch_time": 0.237077808, 2024-04-02T14:51:09.308 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_send_messages_queue_lat": { 2024-04-02T14:51:09.308 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 15707, 2024-04-02T14:51:09.308 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 1.349617255, 2024-04-02T14:51:09.308 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000085924 2024-04-02T14:51:09.308 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:09.309 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_handle_ack_lat": { 2024-04-02T14:51:09.309 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 11850, 2024-04-02T14:51:09.309 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.002055223, 2024-04-02T14:51:09.309 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000173 2024-04-02T14:51:09.309 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:09.309 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_recv_encrypted_bytes": 570000, 2024-04-02T14:51:09.309 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_send_encrypted_bytes": 376000 2024-04-02T14:51:09.309 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:09.309 INFO:teuthology.orchestra.run.smithi092.stdout: "cct": { 2024-04-02T14:51:09.309 INFO:teuthology.orchestra.run.smithi092.stdout: "total_workers": 1, 2024-04-02T14:51:09.309 INFO:teuthology.orchestra.run.smithi092.stdout: "unhealthy_workers": 0 2024-04-02T14:51:09.309 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:09.309 INFO:teuthology.orchestra.run.smithi092.stdout: "finisher-MDSRank": { 2024-04-02T14:51:09.309 INFO:teuthology.orchestra.run.smithi092.stdout: "queue_len": 0, 2024-04-02T14:51:09.309 INFO:teuthology.orchestra.run.smithi092.stdout: "complete_latency": { 2024-04-02T14:51:09.309 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 29808, 2024-04-02T14:51:09.309 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 8.505496950, 2024-04-02T14:51:09.310 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000285342 2024-04-02T14:51:09.310 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-02T14:51:09.310 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:09.310 INFO:teuthology.orchestra.run.smithi092.stdout: "finisher-PurgeQueue": { 2024-04-02T14:51:09.310 INFO:teuthology.orchestra.run.smithi092.stdout: "queue_len": 0, 2024-04-02T14:51:09.310 INFO:teuthology.orchestra.run.smithi092.stdout: "complete_latency": { 2024-04-02T14:51:09.310 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 1007, 2024-04-02T14:51:09.310 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.036683646, 2024-04-02T14:51:09.310 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000036428 2024-04-02T14:51:09.310 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-02T14:51:09.310 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:09.310 INFO:teuthology.orchestra.run.smithi092.stdout: "mds": { 2024-04-02T14:51:09.310 INFO:teuthology.orchestra.run.smithi092.stdout: "request": 4263, 2024-04-02T14:51:09.310 INFO:teuthology.orchestra.run.smithi092.stdout: "reply": 4263, 2024-04-02T14:51:09.310 INFO:teuthology.orchestra.run.smithi092.stdout: "reply_latency": { 2024-04-02T14:51:09.311 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 4263, 2024-04-02T14:51:09.311 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 12.941576548, 2024-04-02T14:51:09.311 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.003035790 2024-04-02T14:51:09.311 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:09.311 INFO:teuthology.orchestra.run.smithi092.stdout: "slow_reply": 0, 2024-04-02T14:51:09.311 INFO:teuthology.orchestra.run.smithi092.stdout: "forward": 0, 2024-04-02T14:51:09.311 INFO:teuthology.orchestra.run.smithi092.stdout: "dir_fetch_complete": 27, 2024-04-02T14:51:09.311 INFO:teuthology.orchestra.run.smithi092.stdout: "dir_fetch_keys": 27, 2024-04-02T14:51:09.311 INFO:teuthology.orchestra.run.smithi092.stdout: "dir_commit": 1471, 2024-04-02T14:51:09.311 INFO:teuthology.orchestra.run.smithi092.stdout: "dir_split": 4, 2024-04-02T14:51:09.311 INFO:teuthology.orchestra.run.smithi092.stdout: "dir_merge": 0, 2024-04-02T14:51:09.311 INFO:teuthology.orchestra.run.smithi092.stdout: "inodes": 1711, 2024-04-02T14:51:09.311 INFO:teuthology.orchestra.run.smithi092.stdout: "inodes_top": 360, 2024-04-02T14:51:09.311 INFO:teuthology.orchestra.run.smithi092.stdout: "inodes_bottom": 1351, 2024-04-02T14:51:09.311 INFO:teuthology.orchestra.run.smithi092.stdout: "inodes_pin_tail": 0, 2024-04-02T14:51:09.312 INFO:teuthology.orchestra.run.smithi092.stdout: "inodes_pinned": 1197, 2024-04-02T14:51:09.312 INFO:teuthology.orchestra.run.smithi092.stdout: "inodes_expired": 0, 2024-04-02T14:51:09.312 INFO:teuthology.orchestra.run.smithi092.stdout: "inodes_with_caps": 1053, 2024-04-02T14:51:09.312 INFO:teuthology.orchestra.run.smithi092.stdout: "caps": 1054, 2024-04-02T14:51:09.312 INFO:teuthology.orchestra.run.smithi092.stdout: "subtrees": 2, 2024-04-02T14:51:09.312 INFO:teuthology.orchestra.run.smithi092.stdout: "traverse": 7297, 2024-04-02T14:51:09.312 INFO:teuthology.orchestra.run.smithi092.stdout: "traverse_hit": 5956, 2024-04-02T14:51:09.312 INFO:teuthology.orchestra.run.smithi092.stdout: "traverse_forward": 0, 2024-04-02T14:51:09.312 INFO:teuthology.orchestra.run.smithi092.stdout: "traverse_discover": 0, 2024-04-02T14:51:09.312 INFO:teuthology.orchestra.run.smithi092.stdout: "traverse_dir_fetch": 0, 2024-04-02T14:51:09.312 INFO:teuthology.orchestra.run.smithi092.stdout: "traverse_remote_ino": 0, 2024-04-02T14:51:09.312 INFO:teuthology.orchestra.run.smithi092.stdout: "traverse_lock": 360, 2024-04-02T14:51:09.312 INFO:teuthology.orchestra.run.smithi092.stdout: "load_cent": 0, 2024-04-02T14:51:09.312 INFO:teuthology.orchestra.run.smithi092.stdout: "q": 0, 2024-04-02T14:51:09.312 INFO:teuthology.orchestra.run.smithi092.stdout: "exported": 0, 2024-04-02T14:51:09.313 INFO:teuthology.orchestra.run.smithi092.stdout: "exported_inodes": 0, 2024-04-02T14:51:09.313 INFO:teuthology.orchestra.run.smithi092.stdout: "imported": 0, 2024-04-02T14:51:09.313 INFO:teuthology.orchestra.run.smithi092.stdout: "imported_inodes": 0, 2024-04-02T14:51:09.313 INFO:teuthology.orchestra.run.smithi092.stdout: "openino_dir_fetch": 0, 2024-04-02T14:51:09.313 INFO:teuthology.orchestra.run.smithi092.stdout: "openino_backtrace_fetch": 28613, 2024-04-02T14:51:09.313 INFO:teuthology.orchestra.run.smithi092.stdout: "openino_peer_discover": 0, 2024-04-02T14:51:09.313 INFO:teuthology.orchestra.run.smithi092.stdout: "root_rfiles": 1026, 2024-04-02T14:51:09.313 INFO:teuthology.orchestra.run.smithi092.stdout: "root_rbytes": 1710054593, 2024-04-02T14:51:09.313 INFO:teuthology.orchestra.run.smithi092.stdout: "root_rsnaps": 6, 2024-04-02T14:51:09.313 INFO:teuthology.orchestra.run.smithi092.stdout: "scrub_backtrace_fetch": 28613, 2024-04-02T14:51:09.313 INFO:teuthology.orchestra.run.smithi092.stdout: "scrub_set_tag": 0, 2024-04-02T14:51:09.313 INFO:teuthology.orchestra.run.smithi092.stdout: "scrub_backtrace_repaired": 0, 2024-04-02T14:51:09.313 INFO:teuthology.orchestra.run.smithi092.stdout: "scrub_inotable_repaired": 0, 2024-04-02T14:51:09.313 INFO:teuthology.orchestra.run.smithi092.stdout: "scrub_dir_inodes": 979, 2024-04-02T14:51:09.313 INFO:teuthology.orchestra.run.smithi092.stdout: "scrub_dir_base_inodes": 28, 2024-04-02T14:51:09.313 INFO:teuthology.orchestra.run.smithi092.stdout: "scrub_dirfrag_rstats": 951, 2024-04-02T14:51:09.314 INFO:teuthology.orchestra.run.smithi092.stdout: "scrub_file_inodes": 27634, 2024-04-02T14:51:09.314 INFO:teuthology.orchestra.run.smithi092.stdout: "handle_inode_file_caps": 0, 2024-04-02T14:51:09.314 INFO:teuthology.orchestra.run.smithi092.stdout: "ceph_cap_op_revoke": 771, 2024-04-02T14:51:09.314 INFO:teuthology.orchestra.run.smithi092.stdout: "ceph_cap_op_grant": 3081, 2024-04-02T14:51:09.314 INFO:teuthology.orchestra.run.smithi092.stdout: "ceph_cap_op_trunc": 1, 2024-04-02T14:51:09.314 INFO:teuthology.orchestra.run.smithi092.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-02T14:51:09.314 INFO:teuthology.orchestra.run.smithi092.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-02T14:51:09.314 INFO:teuthology.orchestra.run.smithi092.stdout: "handle_client_caps": 4759, 2024-04-02T14:51:09.314 INFO:teuthology.orchestra.run.smithi092.stdout: "handle_client_caps_dirty": 2392, 2024-04-02T14:51:09.314 INFO:teuthology.orchestra.run.smithi092.stdout: "handle_client_cap_release": 209, 2024-04-02T14:51:09.314 INFO:teuthology.orchestra.run.smithi092.stdout: "process_request_cap_release": 1244 2024-04-02T14:51:09.314 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:09.314 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_cache": { 2024-04-02T14:51:09.314 INFO:teuthology.orchestra.run.smithi092.stdout: "dir_update": 163, 2024-04-02T14:51:09.314 INFO:teuthology.orchestra.run.smithi092.stdout: "dir_update_receipt": 0, 2024-04-02T14:51:09.315 INFO:teuthology.orchestra.run.smithi092.stdout: "dir_try_discover": 0, 2024-04-02T14:51:09.315 INFO:teuthology.orchestra.run.smithi092.stdout: "dir_send_discover": 0, 2024-04-02T14:51:09.315 INFO:teuthology.orchestra.run.smithi092.stdout: "dir_handle_discover": 246, 2024-04-02T14:51:09.315 INFO:teuthology.orchestra.run.smithi092.stdout: "num_strays": 0, 2024-04-02T14:51:09.315 INFO:teuthology.orchestra.run.smithi092.stdout: "num_strays_delayed": 0, 2024-04-02T14:51:09.315 INFO:teuthology.orchestra.run.smithi092.stdout: "num_strays_enqueuing": 0, 2024-04-02T14:51:09.315 INFO:teuthology.orchestra.run.smithi092.stdout: "strays_created": 497, 2024-04-02T14:51:09.315 INFO:teuthology.orchestra.run.smithi092.stdout: "strays_enqueued": 276, 2024-04-02T14:51:09.315 INFO:teuthology.orchestra.run.smithi092.stdout: "strays_reintegrated": 221, 2024-04-02T14:51:09.315 INFO:teuthology.orchestra.run.smithi092.stdout: "strays_migrated": 0, 2024-04-02T14:51:09.315 INFO:teuthology.orchestra.run.smithi092.stdout: "num_recovering_processing": 0, 2024-04-02T14:51:09.315 INFO:teuthology.orchestra.run.smithi092.stdout: "num_recovering_enqueued": 0, 2024-04-02T14:51:09.315 INFO:teuthology.orchestra.run.smithi092.stdout: "num_recovering_prioritized": 0, 2024-04-02T14:51:09.315 INFO:teuthology.orchestra.run.smithi092.stdout: "recovery_started": 0, 2024-04-02T14:51:09.315 INFO:teuthology.orchestra.run.smithi092.stdout: "recovery_completed": 0, 2024-04-02T14:51:09.316 INFO:teuthology.orchestra.run.smithi092.stdout: "ireq_quiesce_path": 0, 2024-04-02T14:51:09.316 INFO:teuthology.orchestra.run.smithi092.stdout: "ireq_quiesce_inode": 0, 2024-04-02T14:51:09.316 INFO:teuthology.orchestra.run.smithi092.stdout: "ireq_enqueue_scrub": 28, 2024-04-02T14:51:09.316 INFO:teuthology.orchestra.run.smithi092.stdout: "ireq_exportdir": 0, 2024-04-02T14:51:09.316 INFO:teuthology.orchestra.run.smithi092.stdout: "ireq_flush": 0, 2024-04-02T14:51:09.316 INFO:teuthology.orchestra.run.smithi092.stdout: "ireq_fragmentdir": 4, 2024-04-02T14:51:09.316 INFO:teuthology.orchestra.run.smithi092.stdout: "ireq_fragstats": 0, 2024-04-02T14:51:09.316 INFO:teuthology.orchestra.run.smithi092.stdout: "ireq_inodestats": 0 2024-04-02T14:51:09.316 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:09.316 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_log": { 2024-04-02T14:51:09.316 INFO:teuthology.orchestra.run.smithi092.stdout: "evlrg": 0, 2024-04-02T14:51:09.316 INFO:teuthology.orchestra.run.smithi092.stdout: "evadd": 8408, 2024-04-02T14:51:09.316 INFO:teuthology.orchestra.run.smithi092.stdout: "evex": 8109, 2024-04-02T14:51:09.316 INFO:teuthology.orchestra.run.smithi092.stdout: "evtrm": 8109, 2024-04-02T14:51:09.316 INFO:teuthology.orchestra.run.smithi092.stdout: "ev": 299, 2024-04-02T14:51:09.316 INFO:teuthology.orchestra.run.smithi092.stdout: "evexg": 0, 2024-04-02T14:51:09.317 INFO:teuthology.orchestra.run.smithi092.stdout: "evexd": 42, 2024-04-02T14:51:09.317 INFO:teuthology.orchestra.run.smithi092.stdout: "segadd": 82, 2024-04-02T14:51:09.317 INFO:teuthology.orchestra.run.smithi092.stdout: "segex": 81, 2024-04-02T14:51:09.317 INFO:teuthology.orchestra.run.smithi092.stdout: "segtrm": 81, 2024-04-02T14:51:09.317 INFO:teuthology.orchestra.run.smithi092.stdout: "seg": 1, 2024-04-02T14:51:09.317 INFO:teuthology.orchestra.run.smithi092.stdout: "segmjr": 1, 2024-04-02T14:51:09.317 INFO:teuthology.orchestra.run.smithi092.stdout: "segexg": 0, 2024-04-02T14:51:09.317 INFO:teuthology.orchestra.run.smithi092.stdout: "segexd": 1, 2024-04-02T14:51:09.317 INFO:teuthology.orchestra.run.smithi092.stdout: "expos": 47317489, 2024-04-02T14:51:09.317 INFO:teuthology.orchestra.run.smithi092.stdout: "wrpos": 49699677, 2024-04-02T14:51:09.317 INFO:teuthology.orchestra.run.smithi092.stdout: "rdpos": 0, 2024-04-02T14:51:09.317 INFO:teuthology.orchestra.run.smithi092.stdout: "jlat": { 2024-04-02T14:51:09.317 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 3306, 2024-04-02T14:51:09.317 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 79.672371218, 2024-04-02T14:51:09.317 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.024099325 2024-04-02T14:51:09.317 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:09.318 INFO:teuthology.orchestra.run.smithi092.stdout: "replayed": 0 2024-04-02T14:51:09.318 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:09.318 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_mem": { 2024-04-02T14:51:09.318 INFO:teuthology.orchestra.run.smithi092.stdout: "ino": 1530, 2024-04-02T14:51:09.318 INFO:teuthology.orchestra.run.smithi092.stdout: "ino+": 1834, 2024-04-02T14:51:09.318 INFO:teuthology.orchestra.run.smithi092.stdout: "ino-": 304, 2024-04-02T14:51:09.318 INFO:teuthology.orchestra.run.smithi092.stdout: "dir": 74, 2024-04-02T14:51:09.318 INFO:teuthology.orchestra.run.smithi092.stdout: "dir+": 108, 2024-04-02T14:51:09.318 INFO:teuthology.orchestra.run.smithi092.stdout: "dir-": 34, 2024-04-02T14:51:09.318 INFO:teuthology.orchestra.run.smithi092.stdout: "dn": 1711, 2024-04-02T14:51:09.318 INFO:teuthology.orchestra.run.smithi092.stdout: "dn+": 2645, 2024-04-02T14:51:09.318 INFO:teuthology.orchestra.run.smithi092.stdout: "dn-": 934, 2024-04-02T14:51:09.318 INFO:teuthology.orchestra.run.smithi092.stdout: "cap": 1054, 2024-04-02T14:51:09.318 INFO:teuthology.orchestra.run.smithi092.stdout: "cap+": 1552, 2024-04-02T14:51:09.318 INFO:teuthology.orchestra.run.smithi092.stdout: "cap-": 498, 2024-04-02T14:51:09.319 INFO:teuthology.orchestra.run.smithi092.stdout: "rss": 74976, 2024-04-02T14:51:09.319 INFO:teuthology.orchestra.run.smithi092.stdout: "heap": 231800 2024-04-02T14:51:09.319 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:09.319 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_server": { 2024-04-02T14:51:09.319 INFO:teuthology.orchestra.run.smithi092.stdout: "dispatch_client_request": 5305, 2024-04-02T14:51:09.319 INFO:teuthology.orchestra.run.smithi092.stdout: "dispatch_server_request": 1, 2024-04-02T14:51:09.319 INFO:teuthology.orchestra.run.smithi092.stdout: "handle_client_request": 4263, 2024-04-02T14:51:09.319 INFO:teuthology.orchestra.run.smithi092.stdout: "handle_client_session": 488, 2024-04-02T14:51:09.319 INFO:teuthology.orchestra.run.smithi092.stdout: "handle_peer_request": 2, 2024-04-02T14:51:09.319 INFO:teuthology.orchestra.run.smithi092.stdout: "req_create_latency": { 2024-04-02T14:51:09.319 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 1303, 2024-04-02T14:51:09.319 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 7.098692020, 2024-04-02T14:51:09.319 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.005447960 2024-04-02T14:51:09.319 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:09.319 INFO:teuthology.orchestra.run.smithi092.stdout: "req_getattr_latency": { 2024-04-02T14:51:09.319 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 190, 2024-04-02T14:51:09.319 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.077163791, 2024-04-02T14:51:09.320 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000406125 2024-04-02T14:51:09.320 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:09.320 INFO:teuthology.orchestra.run.smithi092.stdout: "req_getfilelock_latency": { 2024-04-02T14:51:09.320 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2024-04-02T14:51:09.320 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2024-04-02T14:51:09.320 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2024-04-02T14:51:09.320 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:09.320 INFO:teuthology.orchestra.run.smithi092.stdout: "req_link_latency": { 2024-04-02T14:51:09.320 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 221, 2024-04-02T14:51:09.320 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.487187222, 2024-04-02T14:51:09.320 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.002204467 2024-04-02T14:51:09.320 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:09.320 INFO:teuthology.orchestra.run.smithi092.stdout: "req_lookup_latency": { 2024-04-02T14:51:09.320 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 1124, 2024-04-02T14:51:09.320 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 1.027363139, 2024-04-02T14:51:09.320 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000914024 2024-04-02T14:51:09.321 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:09.321 INFO:teuthology.orchestra.run.smithi092.stdout: "req_lookuphash_latency": { 2024-04-02T14:51:09.321 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2024-04-02T14:51:09.321 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2024-04-02T14:51:09.321 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2024-04-02T14:51:09.321 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:09.321 INFO:teuthology.orchestra.run.smithi092.stdout: "req_lookupino_latency": { 2024-04-02T14:51:09.321 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 1, 2024-04-02T14:51:09.321 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000219416, 2024-04-02T14:51:09.321 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000219416 2024-04-02T14:51:09.321 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:09.321 INFO:teuthology.orchestra.run.smithi092.stdout: "req_lookupname_latency": { 2024-04-02T14:51:09.321 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2024-04-02T14:51:09.321 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2024-04-02T14:51:09.321 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2024-04-02T14:51:09.322 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:09.322 INFO:teuthology.orchestra.run.smithi092.stdout: "req_lookupparent_latency": { 2024-04-02T14:51:09.322 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2024-04-02T14:51:09.322 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2024-04-02T14:51:09.322 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2024-04-02T14:51:09.322 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:09.322 INFO:teuthology.orchestra.run.smithi092.stdout: "req_lookupsnap_latency": { 2024-04-02T14:51:09.322 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2024-04-02T14:51:09.322 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2024-04-02T14:51:09.322 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2024-04-02T14:51:09.322 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:09.323 INFO:teuthology.orchestra.run.smithi092.stdout: "req_lssnap_latency": { 2024-04-02T14:51:09.323 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 16, 2024-04-02T14:51:09.323 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.002815688, 2024-04-02T14:51:09.323 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000175980 2024-04-02T14:51:09.323 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:09.323 INFO:teuthology.orchestra.run.smithi092.stdout: "req_mkdir_latency": { 2024-04-02T14:51:09.323 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 37, 2024-04-02T14:51:09.323 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.688977655, 2024-04-02T14:51:09.323 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.018621017 2024-04-02T14:51:09.323 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:09.324 INFO:teuthology.orchestra.run.smithi092.stdout: "req_mknod_latency": { 2024-04-02T14:51:09.324 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2024-04-02T14:51:09.324 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2024-04-02T14:51:09.324 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2024-04-02T14:51:09.324 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:09.324 INFO:teuthology.orchestra.run.smithi092.stdout: "req_mksnap_latency": { 2024-04-02T14:51:09.324 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 16, 2024-04-02T14:51:09.324 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.149040040, 2024-04-02T14:51:09.324 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.009315002 2024-04-02T14:51:09.324 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:09.324 INFO:teuthology.orchestra.run.smithi092.stdout: "req_open_latency": { 2024-04-02T14:51:09.324 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 509, 2024-04-02T14:51:09.324 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.228322907, 2024-04-02T14:51:09.324 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000448571 2024-04-02T14:51:09.325 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:09.325 INFO:teuthology.orchestra.run.smithi092.stdout: "req_readdir_latency": { 2024-04-02T14:51:09.325 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 94, 2024-04-02T14:51:09.325 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.183186410, 2024-04-02T14:51:09.325 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.001948791 2024-04-02T14:51:09.325 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:09.325 INFO:teuthology.orchestra.run.smithi092.stdout: "req_rename_latency": { 2024-04-02T14:51:09.325 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 464, 2024-04-02T14:51:09.325 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 2.365630426, 2024-04-02T14:51:09.325 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.005098341 2024-04-02T14:51:09.325 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:09.325 INFO:teuthology.orchestra.run.smithi092.stdout: "req_renamesnap_latency": { 2024-04-02T14:51:09.325 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2024-04-02T14:51:09.325 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2024-04-02T14:51:09.325 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2024-04-02T14:51:09.325 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:09.326 INFO:teuthology.orchestra.run.smithi092.stdout: "req_snapdiff_latency": { 2024-04-02T14:51:09.326 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2024-04-02T14:51:09.326 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2024-04-02T14:51:09.326 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2024-04-02T14:51:09.326 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:09.326 INFO:teuthology.orchestra.run.smithi092.stdout: "req_rmdir_latency": { 2024-04-02T14:51:09.326 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 2, 2024-04-02T14:51:09.326 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.019671379, 2024-04-02T14:51:09.326 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.009835689 2024-04-02T14:51:09.326 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:09.326 INFO:teuthology.orchestra.run.smithi092.stdout: "req_rmsnap_latency": { 2024-04-02T14:51:09.326 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 10, 2024-04-02T14:51:09.326 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.082604646, 2024-04-02T14:51:09.326 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.008260464 2024-04-02T14:51:09.326 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:09.327 INFO:teuthology.orchestra.run.smithi092.stdout: "req_rmxattr_latency": { 2024-04-02T14:51:09.327 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2024-04-02T14:51:09.327 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2024-04-02T14:51:09.327 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2024-04-02T14:51:09.327 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:09.327 INFO:teuthology.orchestra.run.smithi092.stdout: "req_setattr_latency": { 2024-04-02T14:51:09.327 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 2, 2024-04-02T14:51:09.327 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.007955090, 2024-04-02T14:51:09.327 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.003977545 2024-04-02T14:51:09.327 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:09.327 INFO:teuthology.orchestra.run.smithi092.stdout: "req_setdirlayout_latency": { 2024-04-02T14:51:09.327 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2024-04-02T14:51:09.327 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2024-04-02T14:51:09.327 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2024-04-02T14:51:09.327 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:09.328 INFO:teuthology.orchestra.run.smithi092.stdout: "req_setfilelock_latency": { 2024-04-02T14:51:09.328 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2024-04-02T14:51:09.328 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2024-04-02T14:51:09.328 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2024-04-02T14:51:09.328 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:09.328 INFO:teuthology.orchestra.run.smithi092.stdout: "req_setlayout_latency": { 2024-04-02T14:51:09.328 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2024-04-02T14:51:09.328 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2024-04-02T14:51:09.328 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2024-04-02T14:51:09.328 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:09.328 INFO:teuthology.orchestra.run.smithi092.stdout: "req_setxattr_latency": { 2024-04-02T14:51:09.328 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 8, 2024-04-02T14:51:09.328 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.022112404, 2024-04-02T14:51:09.328 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.002764050 2024-04-02T14:51:09.328 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:09.329 INFO:teuthology.orchestra.run.smithi092.stdout: "req_symlink_latency": { 2024-04-02T14:51:09.329 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2024-04-02T14:51:09.329 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2024-04-02T14:51:09.329 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2024-04-02T14:51:09.329 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:09.329 INFO:teuthology.orchestra.run.smithi092.stdout: "req_unlink_latency": { 2024-04-02T14:51:09.329 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 266, 2024-04-02T14:51:09.329 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.500634315, 2024-04-02T14:51:09.329 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.001882083 2024-04-02T14:51:09.329 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:09.329 INFO:teuthology.orchestra.run.smithi092.stdout: "cap_revoke_eviction": 0, 2024-04-02T14:51:09.329 INFO:teuthology.orchestra.run.smithi092.stdout: "cap_acquisition_throttle": 0, 2024-04-02T14:51:09.329 INFO:teuthology.orchestra.run.smithi092.stdout: "req_getvxattr_latency": { 2024-04-02T14:51:09.329 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2024-04-02T14:51:09.329 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2024-04-02T14:51:09.330 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2024-04-02T14:51:09.330 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-02T14:51:09.330 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:09.330 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_sessions": { 2024-04-02T14:51:09.330 INFO:teuthology.orchestra.run.smithi092.stdout: "session_count": 2, 2024-04-02T14:51:09.330 INFO:teuthology.orchestra.run.smithi092.stdout: "session_add": 6, 2024-04-02T14:51:09.330 INFO:teuthology.orchestra.run.smithi092.stdout: "session_remove": 4, 2024-04-02T14:51:09.330 INFO:teuthology.orchestra.run.smithi092.stdout: "sessions_open": 2, 2024-04-02T14:51:09.330 INFO:teuthology.orchestra.run.smithi092.stdout: "sessions_stale": 0, 2024-04-02T14:51:09.330 INFO:teuthology.orchestra.run.smithi092.stdout: "total_load": 590, 2024-04-02T14:51:09.330 INFO:teuthology.orchestra.run.smithi092.stdout: "average_load": 295, 2024-04-02T14:51:09.330 INFO:teuthology.orchestra.run.smithi092.stdout: "avg_session_uptime": 941, 2024-04-02T14:51:09.330 INFO:teuthology.orchestra.run.smithi092.stdout: "mdthresh_evicted": 0 2024-04-02T14:51:09.330 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:09.331 INFO:teuthology.orchestra.run.smithi092.stdout: "mempool": { 2024-04-02T14:51:09.331 INFO:teuthology.orchestra.run.smithi092.stdout: "bloom_filter_bytes": 0, 2024-04-02T14:51:09.331 INFO:teuthology.orchestra.run.smithi092.stdout: "bloom_filter_items": 0, 2024-04-02T14:51:09.331 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_alloc_bytes": 0, 2024-04-02T14:51:09.331 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_alloc_items": 0, 2024-04-02T14:51:09.331 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_cache_data_bytes": 0, 2024-04-02T14:51:09.331 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_cache_data_items": 0, 2024-04-02T14:51:09.331 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-02T14:51:09.331 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_cache_onode_items": 0, 2024-04-02T14:51:09.331 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-02T14:51:09.331 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_cache_meta_items": 0, 2024-04-02T14:51:09.331 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_cache_other_bytes": 0, 2024-04-02T14:51:09.331 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_cache_other_items": 0, 2024-04-02T14:51:09.331 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-02T14:51:09.332 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_cache_buffer_items": 0, 2024-04-02T14:51:09.332 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_extent_bytes": 0, 2024-04-02T14:51:09.332 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_extent_items": 0, 2024-04-02T14:51:09.332 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_blob_bytes": 0, 2024-04-02T14:51:09.332 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_blob_items": 0, 2024-04-02T14:51:09.332 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-02T14:51:09.332 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_shared_blob_items": 0, 2024-04-02T14:51:09.332 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-02T14:51:09.332 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_inline_bl_items": 0, 2024-04-02T14:51:09.332 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_fsck_bytes": 0, 2024-04-02T14:51:09.332 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_fsck_items": 0, 2024-04-02T14:51:09.332 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_txc_bytes": 0, 2024-04-02T14:51:09.332 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_txc_items": 0, 2024-04-02T14:51:09.332 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-02T14:51:09.332 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_writing_deferred_items": 0, 2024-04-02T14:51:09.333 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_writing_bytes": 0, 2024-04-02T14:51:09.333 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_writing_items": 0, 2024-04-02T14:51:09.333 INFO:teuthology.orchestra.run.smithi092.stdout: "bluefs_bytes": 0, 2024-04-02T14:51:09.333 INFO:teuthology.orchestra.run.smithi092.stdout: "bluefs_items": 0, 2024-04-02T14:51:09.333 INFO:teuthology.orchestra.run.smithi092.stdout: "bluefs_file_reader_bytes": 0, 2024-04-02T14:51:09.333 INFO:teuthology.orchestra.run.smithi092.stdout: "bluefs_file_reader_items": 0, 2024-04-02T14:51:09.333 INFO:teuthology.orchestra.run.smithi092.stdout: "bluefs_file_writer_bytes": 0, 2024-04-02T14:51:09.333 INFO:teuthology.orchestra.run.smithi092.stdout: "bluefs_file_writer_items": 0, 2024-04-02T14:51:09.333 INFO:teuthology.orchestra.run.smithi092.stdout: "buffer_anon_bytes": 991471, 2024-04-02T14:51:09.333 INFO:teuthology.orchestra.run.smithi092.stdout: "buffer_anon_items": 1429, 2024-04-02T14:51:09.333 INFO:teuthology.orchestra.run.smithi092.stdout: "buffer_meta_bytes": 792, 2024-04-02T14:51:09.333 INFO:teuthology.orchestra.run.smithi092.stdout: "buffer_meta_items": 9, 2024-04-02T14:51:09.333 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_bytes": 0, 2024-04-02T14:51:09.333 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_items": 0, 2024-04-02T14:51:09.334 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_mapbl_bytes": 0, 2024-04-02T14:51:09.334 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_mapbl_items": 0, 2024-04-02T14:51:09.334 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_pglog_bytes": 0, 2024-04-02T14:51:09.334 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_pglog_items": 0, 2024-04-02T14:51:09.334 INFO:teuthology.orchestra.run.smithi092.stdout: "osdmap_bytes": 8136, 2024-04-02T14:51:09.334 INFO:teuthology.orchestra.run.smithi092.stdout: "osdmap_items": 170, 2024-04-02T14:51:09.334 INFO:teuthology.orchestra.run.smithi092.stdout: "osdmap_mapping_bytes": 0, 2024-04-02T14:51:09.334 INFO:teuthology.orchestra.run.smithi092.stdout: "osdmap_mapping_items": 0, 2024-04-02T14:51:09.334 INFO:teuthology.orchestra.run.smithi092.stdout: "pgmap_bytes": 0, 2024-04-02T14:51:09.334 INFO:teuthology.orchestra.run.smithi092.stdout: "pgmap_items": 0, 2024-04-02T14:51:09.334 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_co_bytes": 5562356, 2024-04-02T14:51:09.334 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_co_items": 101799, 2024-04-02T14:51:09.334 INFO:teuthology.orchestra.run.smithi092.stdout: "unittest_1_bytes": 0, 2024-04-02T14:51:09.334 INFO:teuthology.orchestra.run.smithi092.stdout: "unittest_1_items": 0, 2024-04-02T14:51:09.334 INFO:teuthology.orchestra.run.smithi092.stdout: "unittest_2_bytes": 0, 2024-04-02T14:51:09.334 INFO:teuthology.orchestra.run.smithi092.stdout: "unittest_2_items": 0 2024-04-02T14:51:09.335 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:09.335 INFO:teuthology.orchestra.run.smithi092.stdout: "objecter": { 2024-04-02T14:51:09.335 INFO:teuthology.orchestra.run.smithi092.stdout: "op_active": 0, 2024-04-02T14:51:09.335 INFO:teuthology.orchestra.run.smithi092.stdout: "op_laggy": 0, 2024-04-02T14:51:09.335 INFO:teuthology.orchestra.run.smithi092.stdout: "op_send": 36955, 2024-04-02T14:51:09.335 INFO:teuthology.orchestra.run.smithi092.stdout: "op_send_bytes": 52792272, 2024-04-02T14:51:09.335 INFO:teuthology.orchestra.run.smithi092.stdout: "op_resend": 37, 2024-04-02T14:51:09.335 INFO:teuthology.orchestra.run.smithi092.stdout: "op_reply": 36955, 2024-04-02T14:51:09.335 INFO:teuthology.orchestra.run.smithi092.stdout: "op_latency": { 2024-04-02T14:51:09.335 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 36955, 2024-04-02T14:51:09.335 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 217.487670513, 2024-04-02T14:51:09.335 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.005885202 2024-04-02T14:51:09.335 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:09.336 INFO:teuthology.orchestra.run.smithi092.stdout: "op_inflight": 0, 2024-04-02T14:51:09.336 INFO:teuthology.orchestra.run.smithi092.stdout: "oplen_avg": { 2024-04-02T14:51:09.336 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 36955, 2024-04-02T14:51:09.336 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 43420 2024-04-02T14:51:09.336 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:09.336 INFO:teuthology.orchestra.run.smithi092.stdout: "op": 36955, 2024-04-02T14:51:09.336 INFO:teuthology.orchestra.run.smithi092.stdout: "op_r": 28843, 2024-04-02T14:51:09.336 INFO:teuthology.orchestra.run.smithi092.stdout: "op_w": 8112, 2024-04-02T14:51:09.336 INFO:teuthology.orchestra.run.smithi092.stdout: "op_rmw": 0, 2024-04-02T14:51:09.336 INFO:teuthology.orchestra.run.smithi092.stdout: "op_pg": 0, 2024-04-02T14:51:09.336 INFO:teuthology.orchestra.run.smithi092.stdout: "osdop_stat": 1390, 2024-04-02T14:51:09.336 INFO:teuthology.orchestra.run.smithi092.stdout: "osdop_create": 1508, 2024-04-02T14:51:09.336 INFO:teuthology.orchestra.run.smithi092.stdout: "osdop_read": 148, 2024-04-02T14:51:09.336 INFO:teuthology.orchestra.run.smithi092.stdout: "osdop_write": 3426, 2024-04-02T14:51:09.336 INFO:teuthology.orchestra.run.smithi092.stdout: "osdop_writefull": 396, 2024-04-02T14:51:09.337 INFO:teuthology.orchestra.run.smithi092.stdout: "osdop_writesame": 0, 2024-04-02T14:51:09.337 INFO:teuthology.orchestra.run.smithi092.stdout: "osdop_append": 0, 2024-04-02T14:51:09.337 INFO:teuthology.orchestra.run.smithi092.stdout: "osdop_zero": 0, 2024-04-02T14:51:09.337 INFO:teuthology.orchestra.run.smithi092.stdout: "osdop_truncate": 4, 2024-04-02T14:51:09.337 INFO:teuthology.orchestra.run.smithi092.stdout: "osdop_delete": 822, 2024-04-02T14:51:09.337 INFO:teuthology.orchestra.run.smithi092.stdout: "osdop_mapext": 0, 2024-04-02T14:51:09.337 INFO:teuthology.orchestra.run.smithi092.stdout: "osdop_sparse_read": 0, 2024-04-02T14:51:09.337 INFO:teuthology.orchestra.run.smithi092.stdout: "osdop_clonerange": 0, 2024-04-02T14:51:09.337 INFO:teuthology.orchestra.run.smithi092.stdout: "osdop_getxattr": 28695, 2024-04-02T14:51:09.337 INFO:teuthology.orchestra.run.smithi092.stdout: "osdop_setxattr": 3016, 2024-04-02T14:51:09.337 INFO:teuthology.orchestra.run.smithi092.stdout: "osdop_cmpxattr": 0, 2024-04-02T14:51:09.337 INFO:teuthology.orchestra.run.smithi092.stdout: "osdop_rmxattr": 0, 2024-04-02T14:51:09.337 INFO:teuthology.orchestra.run.smithi092.stdout: "osdop_resetxattrs": 0, 2024-04-02T14:51:09.337 INFO:teuthology.orchestra.run.smithi092.stdout: "osdop_call": 0, 2024-04-02T14:51:09.337 INFO:teuthology.orchestra.run.smithi092.stdout: "osdop_watch": 0, 2024-04-02T14:51:09.338 INFO:teuthology.orchestra.run.smithi092.stdout: "osdop_notify": 0, 2024-04-02T14:51:09.338 INFO:teuthology.orchestra.run.smithi092.stdout: "osdop_src_cmpxattr": 0, 2024-04-02T14:51:09.338 INFO:teuthology.orchestra.run.smithi092.stdout: "osdop_pgls": 0, 2024-04-02T14:51:09.338 INFO:teuthology.orchestra.run.smithi092.stdout: "osdop_pgls_filter": 0, 2024-04-02T14:51:09.338 INFO:teuthology.orchestra.run.smithi092.stdout: "osdop_other": 1, 2024-04-02T14:51:09.338 INFO:teuthology.orchestra.run.smithi092.stdout: "linger_active": 0, 2024-04-02T14:51:09.338 INFO:teuthology.orchestra.run.smithi092.stdout: "linger_send": 0, 2024-04-02T14:51:09.338 INFO:teuthology.orchestra.run.smithi092.stdout: "linger_resend": 0, 2024-04-02T14:51:09.338 INFO:teuthology.orchestra.run.smithi092.stdout: "linger_ping": 0, 2024-04-02T14:51:09.338 INFO:teuthology.orchestra.run.smithi092.stdout: "poolop_active": 0, 2024-04-02T14:51:09.338 INFO:teuthology.orchestra.run.smithi092.stdout: "poolop_send": 0, 2024-04-02T14:51:09.338 INFO:teuthology.orchestra.run.smithi092.stdout: "poolop_resend": 0, 2024-04-02T14:51:09.338 INFO:teuthology.orchestra.run.smithi092.stdout: "poolstat_active": 0, 2024-04-02T14:51:09.338 INFO:teuthology.orchestra.run.smithi092.stdout: "poolstat_send": 0, 2024-04-02T14:51:09.339 INFO:teuthology.orchestra.run.smithi092.stdout: "poolstat_resend": 0, 2024-04-02T14:51:09.339 INFO:teuthology.orchestra.run.smithi092.stdout: "statfs_active": 0, 2024-04-02T14:51:09.339 INFO:teuthology.orchestra.run.smithi092.stdout: "statfs_send": 0, 2024-04-02T14:51:09.339 INFO:teuthology.orchestra.run.smithi092.stdout: "statfs_resend": 0, 2024-04-02T14:51:09.339 INFO:teuthology.orchestra.run.smithi092.stdout: "command_active": 0, 2024-04-02T14:51:09.339 INFO:teuthology.orchestra.run.smithi092.stdout: "command_send": 0, 2024-04-02T14:51:09.339 INFO:teuthology.orchestra.run.smithi092.stdout: "command_resend": 0, 2024-04-02T14:51:09.339 INFO:teuthology.orchestra.run.smithi092.stdout: "map_epoch": 110, 2024-04-02T14:51:09.339 INFO:teuthology.orchestra.run.smithi092.stdout: "map_full": 0, 2024-04-02T14:51:09.339 INFO:teuthology.orchestra.run.smithi092.stdout: "map_inc": 23, 2024-04-02T14:51:09.339 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_sessions": 12, 2024-04-02T14:51:09.339 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_session_open": 12, 2024-04-02T14:51:09.339 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_session_close": 0, 2024-04-02T14:51:09.339 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_laggy": 0, 2024-04-02T14:51:09.340 INFO:teuthology.orchestra.run.smithi092.stdout: "omap_wr": 3614, 2024-04-02T14:51:09.340 INFO:teuthology.orchestra.run.smithi092.stdout: "omap_rd": 108, 2024-04-02T14:51:09.340 INFO:teuthology.orchestra.run.smithi092.stdout: "omap_del": 292 2024-04-02T14:51:09.340 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:09.340 INFO:teuthology.orchestra.run.smithi092.stdout: "oft": { 2024-04-02T14:51:09.340 INFO:teuthology.orchestra.run.smithi092.stdout: "omap_total_objs": 1, 2024-04-02T14:51:09.340 INFO:teuthology.orchestra.run.smithi092.stdout: "omap_total_kv_pairs": 0, 2024-04-02T14:51:09.340 INFO:teuthology.orchestra.run.smithi092.stdout: "omap_total_updates": 0, 2024-04-02T14:51:09.340 INFO:teuthology.orchestra.run.smithi092.stdout: "omap_total_removes": 0 2024-04-02T14:51:09.340 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:09.340 INFO:teuthology.orchestra.run.smithi092.stdout: "osd-slow-ops": { 2024-04-02T14:51:09.340 INFO:teuthology.orchestra.run.smithi092.stdout: "slow_ops_count": 1 2024-04-02T14:51:09.340 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:09.340 INFO:teuthology.orchestra.run.smithi092.stdout: "purge_queue": { 2024-04-02T14:51:09.340 INFO:teuthology.orchestra.run.smithi092.stdout: "pq_executing_ops": 0, 2024-04-02T14:51:09.341 INFO:teuthology.orchestra.run.smithi092.stdout: "pq_executing_ops_high_water": 8, 2024-04-02T14:51:09.341 INFO:teuthology.orchestra.run.smithi092.stdout: "pq_executing": 0, 2024-04-02T14:51:09.341 INFO:teuthology.orchestra.run.smithi092.stdout: "pq_executing_high_water": 6, 2024-04-02T14:51:09.341 INFO:teuthology.orchestra.run.smithi092.stdout: "pq_executed_ops": 290, 2024-04-02T14:51:09.341 INFO:teuthology.orchestra.run.smithi092.stdout: "pq_executed": 276, 2024-04-02T14:51:09.341 INFO:teuthology.orchestra.run.smithi092.stdout: "pq_item_in_journal": 0 2024-04-02T14:51:09.341 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:09.341 INFO:teuthology.orchestra.run.smithi092.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-02T14:51:09.341 INFO:teuthology.orchestra.run.smithi092.stdout: "val": 0, 2024-04-02T14:51:09.341 INFO:teuthology.orchestra.run.smithi092.stdout: "max": 104857600, 2024-04-02T14:51:09.341 INFO:teuthology.orchestra.run.smithi092.stdout: "get_started": 0, 2024-04-02T14:51:09.341 INFO:teuthology.orchestra.run.smithi092.stdout: "get": 56951, 2024-04-02T14:51:09.341 INFO:teuthology.orchestra.run.smithi092.stdout: "get_sum": 22394122, 2024-04-02T14:51:09.341 INFO:teuthology.orchestra.run.smithi092.stdout: "get_or_fail_fail": 0, 2024-04-02T14:51:09.341 INFO:teuthology.orchestra.run.smithi092.stdout: "get_or_fail_success": 56951, 2024-04-02T14:51:09.342 INFO:teuthology.orchestra.run.smithi092.stdout: "take": 0, 2024-04-02T14:51:09.342 INFO:teuthology.orchestra.run.smithi092.stdout: "take_sum": 0, 2024-04-02T14:51:09.342 INFO:teuthology.orchestra.run.smithi092.stdout: "put": 56951, 2024-04-02T14:51:09.342 INFO:teuthology.orchestra.run.smithi092.stdout: "put_sum": 22394122, 2024-04-02T14:51:09.342 INFO:teuthology.orchestra.run.smithi092.stdout: "wait": { 2024-04-02T14:51:09.342 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2024-04-02T14:51:09.342 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2024-04-02T14:51:09.342 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2024-04-02T14:51:09.342 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-02T14:51:09.342 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:09.342 INFO:teuthology.orchestra.run.smithi092.stdout: "throttle-objecter_bytes": { 2024-04-02T14:51:09.342 INFO:teuthology.orchestra.run.smithi092.stdout: "val": 0, 2024-04-02T14:51:09.342 INFO:teuthology.orchestra.run.smithi092.stdout: "max": 104857600, 2024-04-02T14:51:09.342 INFO:teuthology.orchestra.run.smithi092.stdout: "get_started": 0, 2024-04-02T14:51:09.343 INFO:teuthology.orchestra.run.smithi092.stdout: "get": 0, 2024-04-02T14:51:09.343 INFO:teuthology.orchestra.run.smithi092.stdout: "get_sum": 0, 2024-04-02T14:51:09.343 INFO:teuthology.orchestra.run.smithi092.stdout: "get_or_fail_fail": 0, 2024-04-02T14:51:09.343 INFO:teuthology.orchestra.run.smithi092.stdout: "get_or_fail_success": 0, 2024-04-02T14:51:09.343 INFO:teuthology.orchestra.run.smithi092.stdout: "take": 36955, 2024-04-02T14:51:09.343 INFO:teuthology.orchestra.run.smithi092.stdout: "take_sum": 52820776, 2024-04-02T14:51:09.343 INFO:teuthology.orchestra.run.smithi092.stdout: "put": 36955, 2024-04-02T14:51:09.343 INFO:teuthology.orchestra.run.smithi092.stdout: "put_sum": 52820776, 2024-04-02T14:51:09.343 INFO:teuthology.orchestra.run.smithi092.stdout: "wait": { 2024-04-02T14:51:09.343 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2024-04-02T14:51:09.343 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2024-04-02T14:51:09.343 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2024-04-02T14:51:09.343 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-02T14:51:09.343 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:09.343 INFO:teuthology.orchestra.run.smithi092.stdout: "throttle-objecter_ops": { 2024-04-02T14:51:09.344 INFO:teuthology.orchestra.run.smithi092.stdout: "val": 0, 2024-04-02T14:51:09.344 INFO:teuthology.orchestra.run.smithi092.stdout: "max": 1024, 2024-04-02T14:51:09.344 INFO:teuthology.orchestra.run.smithi092.stdout: "get_started": 0, 2024-04-02T14:51:09.344 INFO:teuthology.orchestra.run.smithi092.stdout: "get": 0, 2024-04-02T14:51:09.344 INFO:teuthology.orchestra.run.smithi092.stdout: "get_sum": 0, 2024-04-02T14:51:09.344 INFO:teuthology.orchestra.run.smithi092.stdout: "get_or_fail_fail": 0, 2024-04-02T14:51:09.344 INFO:teuthology.orchestra.run.smithi092.stdout: "get_or_fail_success": 0, 2024-04-02T14:51:09.344 INFO:teuthology.orchestra.run.smithi092.stdout: "take": 36955, 2024-04-02T14:51:09.344 INFO:teuthology.orchestra.run.smithi092.stdout: "take_sum": 36955, 2024-04-02T14:51:09.344 INFO:teuthology.orchestra.run.smithi092.stdout: "put": 36955, 2024-04-02T14:51:09.344 INFO:teuthology.orchestra.run.smithi092.stdout: "put_sum": 36955, 2024-04-02T14:51:09.344 INFO:teuthology.orchestra.run.smithi092.stdout: "wait": { 2024-04-02T14:51:09.344 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2024-04-02T14:51:09.344 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2024-04-02T14:51:09.344 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2024-04-02T14:51:09.345 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-02T14:51:09.345 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:09.345 INFO:teuthology.orchestra.run.smithi092.stdout: "throttle-write_buf_throttle": { 2024-04-02T14:51:09.345 INFO:teuthology.orchestra.run.smithi092.stdout: "val": 0, 2024-04-02T14:51:09.345 INFO:teuthology.orchestra.run.smithi092.stdout: "max": 3758096384, 2024-04-02T14:51:09.345 INFO:teuthology.orchestra.run.smithi092.stdout: "get_started": 0, 2024-04-02T14:51:09.345 INFO:teuthology.orchestra.run.smithi092.stdout: "get": 276, 2024-04-02T14:51:09.345 INFO:teuthology.orchestra.run.smithi092.stdout: "get_sum": 29044, 2024-04-02T14:51:09.345 INFO:teuthology.orchestra.run.smithi092.stdout: "get_or_fail_fail": 0, 2024-04-02T14:51:09.345 INFO:teuthology.orchestra.run.smithi092.stdout: "get_or_fail_success": 276, 2024-04-02T14:51:09.345 INFO:teuthology.orchestra.run.smithi092.stdout: "take": 0, 2024-04-02T14:51:09.345 INFO:teuthology.orchestra.run.smithi092.stdout: "take_sum": 0, 2024-04-02T14:51:09.345 INFO:teuthology.orchestra.run.smithi092.stdout: "put": 120, 2024-04-02T14:51:09.345 INFO:teuthology.orchestra.run.smithi092.stdout: "put_sum": 29044, 2024-04-02T14:51:09.345 INFO:teuthology.orchestra.run.smithi092.stdout: "wait": { 2024-04-02T14:51:09.346 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2024-04-02T14:51:09.346 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2024-04-02T14:51:09.346 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2024-04-02T14:51:09.346 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-02T14:51:09.346 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:09.346 INFO:teuthology.orchestra.run.smithi092.stdout: "throttle-write_buf_throttle-0x561817cc4500": { 2024-04-02T14:51:09.346 INFO:teuthology.orchestra.run.smithi092.stdout: "val": 112, 2024-04-02T14:51:09.346 INFO:teuthology.orchestra.run.smithi092.stdout: "max": 3758096384, 2024-04-02T14:51:09.346 INFO:teuthology.orchestra.run.smithi092.stdout: "get_started": 0, 2024-04-02T14:51:09.346 INFO:teuthology.orchestra.run.smithi092.stdout: "get": 8408, 2024-04-02T14:51:09.346 INFO:teuthology.orchestra.run.smithi092.stdout: "get_sum": 45505373, 2024-04-02T14:51:09.346 INFO:teuthology.orchestra.run.smithi092.stdout: "get_or_fail_fail": 0, 2024-04-02T14:51:09.346 INFO:teuthology.orchestra.run.smithi092.stdout: "get_or_fail_success": 8408, 2024-04-02T14:51:09.346 INFO:teuthology.orchestra.run.smithi092.stdout: "take": 0, 2024-04-02T14:51:09.347 INFO:teuthology.orchestra.run.smithi092.stdout: "take_sum": 0, 2024-04-02T14:51:09.347 INFO:teuthology.orchestra.run.smithi092.stdout: "put": 3306, 2024-04-02T14:51:09.347 INFO:teuthology.orchestra.run.smithi092.stdout: "put_sum": 45505261, 2024-04-02T14:51:09.347 INFO:teuthology.orchestra.run.smithi092.stdout: "wait": { 2024-04-02T14:51:09.347 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2024-04-02T14:51:09.347 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2024-04-02T14:51:09.347 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2024-04-02T14:51:09.347 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-02T14:51:09.347 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-02T14:51:09.347 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-04-02T14:51:10.102 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:51:09 smithi092 ceph-mon[36497]: pgmap v510: 97 pgs: 97 active+clean; 4.3 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 104 KiB/s wr, 34 op/s 2024-04-02T14:51:10.176 INFO:tasks.check_counter:Daemon mds.b mds.exported=0 2024-04-02T14:51:10.176 INFO:tasks.check_counter:Daemon mds.b mds.imported=0 2024-04-02T14:51:10.176 INFO:tasks.check_counter:Daemon mds.b mds_cache.dir_update=163 2024-04-02T14:51:10.176 INFO:tasks.check_counter:Daemon mds.b mds_cache.dir_update_receipt=0 2024-04-02T14:51:10.176 INFO:tasks.check_counter:Daemon mds.b mds.root_rsnaps=6 2024-04-02T14:51:10.176 INFO:tasks.check_counter:Daemon mds.b mds_server.req_mksnap_latency.avgcount=16 2024-04-02T14:51:10.176 INFO:tasks.check_counter:Daemon mds.b mds_server.req_rmsnap_latency.avgcount=10 2024-04-02T14:51:10.176 DEBUG:tasks.check_counter:Getting stats from e 2024-04-02T14:51:10.176 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:191d963ac9de7322da8aff20b567b3fa458e2ef6 shell --fsid 6449448a-f0fc-11ee-b647-cb9ed24678a4 -- ceph daemon mds.e perf dump 2024-04-02T14:51:10.177 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:51:09 smithi177 ceph-mon[36431]: pgmap v510: 97 pgs: 97 active+clean; 4.3 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 104 KiB/s wr, 34 op/s 2024-04-02T14:51:10.179 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:51:09 smithi090 ceph-mon[31674]: pgmap v510: 97 pgs: 97 active+clean; 4.3 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 104 KiB/s wr, 34 op/s 2024-04-02T14:51:10.439 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/mon.b/config 2024-04-02T14:51:11.921 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:51:11 smithi092 ceph-mon[36497]: pgmap v511: 97 pgs: 40 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 35 active+clean; 4.1 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 89 KiB/s wr, 32 op/s 2024-04-02T14:51:12.177 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-04-02T14:51:12.178 INFO:teuthology.orchestra.run.smithi092.stdout: "AsyncMessenger::Worker-0": { 2024-04-02T14:51:12.178 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_recv_messages": 3306, 2024-04-02T14:51:12.178 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_send_messages": 3306, 2024-04-02T14:51:12.178 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_recv_bytes": 936665, 2024-04-02T14:51:12.178 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_send_bytes": 1050491, 2024-04-02T14:51:12.178 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_created_connections": 6, 2024-04-02T14:51:12.178 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_active_connections": 2, 2024-04-02T14:51:12.178 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_running_total_time": 0.203266089, 2024-04-02T14:51:12.178 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_running_send_time": 0.062357945, 2024-04-02T14:51:12.178 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_running_recv_time": 0.050101594, 2024-04-02T14:51:12.178 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_running_fast_dispatch_time": 0.075878281, 2024-04-02T14:51:12.179 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_send_messages_queue_lat": { 2024-04-02T14:51:12.179 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 3305, 2024-04-02T14:51:12.179 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.087507307, 2024-04-02T14:51:12.179 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000026477 2024-04-02T14:51:12.179 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:12.179 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_handle_ack_lat": { 2024-04-02T14:51:12.179 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2024-04-02T14:51:12.179 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2024-04-02T14:51:12.179 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2024-04-02T14:51:12.179 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:12.179 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_recv_encrypted_bytes": 3072, 2024-04-02T14:51:12.179 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_send_encrypted_bytes": 448 2024-04-02T14:51:12.179 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:12.180 INFO:teuthology.orchestra.run.smithi092.stdout: "AsyncMessenger::Worker-1": { 2024-04-02T14:51:12.180 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_recv_messages": 349, 2024-04-02T14:51:12.180 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_send_messages": 337, 2024-04-02T14:51:12.180 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_recv_bytes": 1655020, 2024-04-02T14:51:12.180 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_send_bytes": 195882, 2024-04-02T14:51:12.180 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_created_connections": 3, 2024-04-02T14:51:12.180 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_active_connections": 2, 2024-04-02T14:51:12.180 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_running_total_time": 0.052233457, 2024-04-02T14:51:12.180 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_running_send_time": 0.015974877, 2024-04-02T14:51:12.180 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_running_recv_time": 0.038631961, 2024-04-02T14:51:12.180 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_running_fast_dispatch_time": 0.006844357, 2024-04-02T14:51:12.180 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_send_messages_queue_lat": { 2024-04-02T14:51:12.180 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 336, 2024-04-02T14:51:12.181 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.016151933, 2024-04-02T14:51:12.181 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000048071 2024-04-02T14:51:12.181 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:12.181 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_handle_ack_lat": { 2024-04-02T14:51:12.181 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2024-04-02T14:51:12.181 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2024-04-02T14:51:12.181 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2024-04-02T14:51:12.181 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:12.181 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_recv_encrypted_bytes": 1654464, 2024-04-02T14:51:12.181 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_send_encrypted_bytes": 195184 2024-04-02T14:51:12.181 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:12.181 INFO:teuthology.orchestra.run.smithi092.stdout: "AsyncMessenger::Worker-2": { 2024-04-02T14:51:12.181 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_recv_messages": 3, 2024-04-02T14:51:12.182 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_send_messages": 236, 2024-04-02T14:51:12.182 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_recv_bytes": 588, 2024-04-02T14:51:12.182 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_send_bytes": 576334, 2024-04-02T14:51:12.182 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_created_connections": 7, 2024-04-02T14:51:12.182 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_active_connections": 4, 2024-04-02T14:51:12.182 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_running_total_time": 0.012121444, 2024-04-02T14:51:12.182 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_running_send_time": 0.007956908, 2024-04-02T14:51:12.182 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_running_recv_time": 0.001939618, 2024-04-02T14:51:12.182 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_running_fast_dispatch_time": 0.000015970, 2024-04-02T14:51:12.182 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_send_messages_queue_lat": { 2024-04-02T14:51:12.182 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 233, 2024-04-02T14:51:12.182 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.017439946, 2024-04-02T14:51:12.182 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000074849 2024-04-02T14:51:12.182 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:12.183 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_handle_ack_lat": { 2024-04-02T14:51:12.183 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2024-04-02T14:51:12.183 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2024-04-02T14:51:12.183 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2024-04-02T14:51:12.183 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:12.183 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_recv_encrypted_bytes": 320, 2024-04-02T14:51:12.183 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_send_encrypted_bytes": 576016 2024-04-02T14:51:12.183 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:12.183 INFO:teuthology.orchestra.run.smithi092.stdout: "cct": { 2024-04-02T14:51:12.183 INFO:teuthology.orchestra.run.smithi092.stdout: "total_workers": 1, 2024-04-02T14:51:12.183 INFO:teuthology.orchestra.run.smithi092.stdout: "unhealthy_workers": 0 2024-04-02T14:51:12.183 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:12.183 INFO:teuthology.orchestra.run.smithi092.stdout: "finisher-MDSRank": { 2024-04-02T14:51:12.183 INFO:teuthology.orchestra.run.smithi092.stdout: "queue_len": 0, 2024-04-02T14:51:12.183 INFO:teuthology.orchestra.run.smithi092.stdout: "complete_latency": { 2024-04-02T14:51:12.184 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 5457, 2024-04-02T14:51:12.184 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.112005767, 2024-04-02T14:51:12.184 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000020525 2024-04-02T14:51:12.184 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-02T14:51:12.184 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:12.184 INFO:teuthology.orchestra.run.smithi092.stdout: "finisher-PurgeQueue": { 2024-04-02T14:51:12.184 INFO:teuthology.orchestra.run.smithi092.stdout: "queue_len": 0, 2024-04-02T14:51:12.184 INFO:teuthology.orchestra.run.smithi092.stdout: "complete_latency": { 2024-04-02T14:51:12.184 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2024-04-02T14:51:12.184 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2024-04-02T14:51:12.184 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2024-04-02T14:51:12.184 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-02T14:51:12.184 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:12.184 INFO:teuthology.orchestra.run.smithi092.stdout: "mds": { 2024-04-02T14:51:12.184 INFO:teuthology.orchestra.run.smithi092.stdout: "request": 0, 2024-04-02T14:51:12.185 INFO:teuthology.orchestra.run.smithi092.stdout: "reply": 0, 2024-04-02T14:51:12.185 INFO:teuthology.orchestra.run.smithi092.stdout: "reply_latency": { 2024-04-02T14:51:12.185 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2024-04-02T14:51:12.185 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2024-04-02T14:51:12.185 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2024-04-02T14:51:12.185 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:12.185 INFO:teuthology.orchestra.run.smithi092.stdout: "slow_reply": 0, 2024-04-02T14:51:12.185 INFO:teuthology.orchestra.run.smithi092.stdout: "forward": 0, 2024-04-02T14:51:12.185 INFO:teuthology.orchestra.run.smithi092.stdout: "dir_fetch_complete": 0, 2024-04-02T14:51:12.185 INFO:teuthology.orchestra.run.smithi092.stdout: "dir_fetch_keys": 0, 2024-04-02T14:51:12.185 INFO:teuthology.orchestra.run.smithi092.stdout: "dir_commit": 0, 2024-04-02T14:51:12.185 INFO:teuthology.orchestra.run.smithi092.stdout: "dir_split": 0, 2024-04-02T14:51:12.185 INFO:teuthology.orchestra.run.smithi092.stdout: "dir_merge": 0, 2024-04-02T14:51:12.185 INFO:teuthology.orchestra.run.smithi092.stdout: "inodes": 0, 2024-04-02T14:51:12.186 INFO:teuthology.orchestra.run.smithi092.stdout: "inodes_top": 0, 2024-04-02T14:51:12.186 INFO:teuthology.orchestra.run.smithi092.stdout: "inodes_bottom": 0, 2024-04-02T14:51:12.186 INFO:teuthology.orchestra.run.smithi092.stdout: "inodes_pin_tail": 0, 2024-04-02T14:51:12.186 INFO:teuthology.orchestra.run.smithi092.stdout: "inodes_pinned": 0, 2024-04-02T14:51:12.186 INFO:teuthology.orchestra.run.smithi092.stdout: "inodes_expired": 0, 2024-04-02T14:51:12.186 INFO:teuthology.orchestra.run.smithi092.stdout: "inodes_with_caps": 0, 2024-04-02T14:51:12.186 INFO:teuthology.orchestra.run.smithi092.stdout: "caps": 0, 2024-04-02T14:51:12.186 INFO:teuthology.orchestra.run.smithi092.stdout: "subtrees": 1, 2024-04-02T14:51:12.186 INFO:teuthology.orchestra.run.smithi092.stdout: "traverse": 0, 2024-04-02T14:51:12.186 INFO:teuthology.orchestra.run.smithi092.stdout: "traverse_hit": 0, 2024-04-02T14:51:12.186 INFO:teuthology.orchestra.run.smithi092.stdout: "traverse_forward": 0, 2024-04-02T14:51:12.186 INFO:teuthology.orchestra.run.smithi092.stdout: "traverse_discover": 0, 2024-04-02T14:51:12.186 INFO:teuthology.orchestra.run.smithi092.stdout: "traverse_dir_fetch": 0, 2024-04-02T14:51:12.186 INFO:teuthology.orchestra.run.smithi092.stdout: "traverse_remote_ino": 0, 2024-04-02T14:51:12.186 INFO:teuthology.orchestra.run.smithi092.stdout: "traverse_lock": 0, 2024-04-02T14:51:12.187 INFO:teuthology.orchestra.run.smithi092.stdout: "load_cent": 0, 2024-04-02T14:51:12.187 INFO:teuthology.orchestra.run.smithi092.stdout: "q": 0, 2024-04-02T14:51:12.187 INFO:teuthology.orchestra.run.smithi092.stdout: "exported": 0, 2024-04-02T14:51:12.187 INFO:teuthology.orchestra.run.smithi092.stdout: "exported_inodes": 0, 2024-04-02T14:51:12.187 INFO:teuthology.orchestra.run.smithi092.stdout: "imported": 0, 2024-04-02T14:51:12.187 INFO:teuthology.orchestra.run.smithi092.stdout: "imported_inodes": 0, 2024-04-02T14:51:12.187 INFO:teuthology.orchestra.run.smithi092.stdout: "openino_dir_fetch": 0, 2024-04-02T14:51:12.187 INFO:teuthology.orchestra.run.smithi092.stdout: "openino_backtrace_fetch": 0, 2024-04-02T14:51:12.187 INFO:teuthology.orchestra.run.smithi092.stdout: "openino_peer_discover": 0, 2024-04-02T14:51:12.187 INFO:teuthology.orchestra.run.smithi092.stdout: "root_rfiles": 0, 2024-04-02T14:51:12.187 INFO:teuthology.orchestra.run.smithi092.stdout: "root_rbytes": 0, 2024-04-02T14:51:12.187 INFO:teuthology.orchestra.run.smithi092.stdout: "root_rsnaps": 0, 2024-04-02T14:51:12.187 INFO:teuthology.orchestra.run.smithi092.stdout: "scrub_backtrace_fetch": 0, 2024-04-02T14:51:12.187 INFO:teuthology.orchestra.run.smithi092.stdout: "scrub_set_tag": 0, 2024-04-02T14:51:12.188 INFO:teuthology.orchestra.run.smithi092.stdout: "scrub_backtrace_repaired": 0, 2024-04-02T14:51:12.188 INFO:teuthology.orchestra.run.smithi092.stdout: "scrub_inotable_repaired": 0, 2024-04-02T14:51:12.188 INFO:teuthology.orchestra.run.smithi092.stdout: "scrub_dir_inodes": 0, 2024-04-02T14:51:12.188 INFO:teuthology.orchestra.run.smithi092.stdout: "scrub_dir_base_inodes": 0, 2024-04-02T14:51:12.188 INFO:teuthology.orchestra.run.smithi092.stdout: "scrub_dirfrag_rstats": 0, 2024-04-02T14:51:12.188 INFO:teuthology.orchestra.run.smithi092.stdout: "scrub_file_inodes": 0, 2024-04-02T14:51:12.188 INFO:teuthology.orchestra.run.smithi092.stdout: "handle_inode_file_caps": 0, 2024-04-02T14:51:12.188 INFO:teuthology.orchestra.run.smithi092.stdout: "ceph_cap_op_revoke": 0, 2024-04-02T14:51:12.188 INFO:teuthology.orchestra.run.smithi092.stdout: "ceph_cap_op_grant": 0, 2024-04-02T14:51:12.188 INFO:teuthology.orchestra.run.smithi092.stdout: "ceph_cap_op_trunc": 0, 2024-04-02T14:51:12.188 INFO:teuthology.orchestra.run.smithi092.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-02T14:51:12.188 INFO:teuthology.orchestra.run.smithi092.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-02T14:51:12.188 INFO:teuthology.orchestra.run.smithi092.stdout: "handle_client_caps": 0, 2024-04-02T14:51:12.188 INFO:teuthology.orchestra.run.smithi092.stdout: "handle_client_caps_dirty": 0, 2024-04-02T14:51:12.189 INFO:teuthology.orchestra.run.smithi092.stdout: "handle_client_cap_release": 0, 2024-04-02T14:51:12.189 INFO:teuthology.orchestra.run.smithi092.stdout: "process_request_cap_release": 0 2024-04-02T14:51:12.189 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:12.189 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_cache": { 2024-04-02T14:51:12.189 INFO:teuthology.orchestra.run.smithi092.stdout: "dir_update": 0, 2024-04-02T14:51:12.189 INFO:teuthology.orchestra.run.smithi092.stdout: "dir_update_receipt": 0, 2024-04-02T14:51:12.189 INFO:teuthology.orchestra.run.smithi092.stdout: "dir_try_discover": 0, 2024-04-02T14:51:12.189 INFO:teuthology.orchestra.run.smithi092.stdout: "dir_send_discover": 0, 2024-04-02T14:51:12.189 INFO:teuthology.orchestra.run.smithi092.stdout: "dir_handle_discover": 0, 2024-04-02T14:51:12.189 INFO:teuthology.orchestra.run.smithi092.stdout: "num_strays": 0, 2024-04-02T14:51:12.189 INFO:teuthology.orchestra.run.smithi092.stdout: "num_strays_delayed": 0, 2024-04-02T14:51:12.189 INFO:teuthology.orchestra.run.smithi092.stdout: "num_strays_enqueuing": 0, 2024-04-02T14:51:12.189 INFO:teuthology.orchestra.run.smithi092.stdout: "strays_created": 0, 2024-04-02T14:51:12.189 INFO:teuthology.orchestra.run.smithi092.stdout: "strays_enqueued": 0, 2024-04-02T14:51:12.190 INFO:teuthology.orchestra.run.smithi092.stdout: "strays_reintegrated": 0, 2024-04-02T14:51:12.190 INFO:teuthology.orchestra.run.smithi092.stdout: "strays_migrated": 0, 2024-04-02T14:51:12.190 INFO:teuthology.orchestra.run.smithi092.stdout: "num_recovering_processing": 0, 2024-04-02T14:51:12.190 INFO:teuthology.orchestra.run.smithi092.stdout: "num_recovering_enqueued": 0, 2024-04-02T14:51:12.190 INFO:teuthology.orchestra.run.smithi092.stdout: "num_recovering_prioritized": 0, 2024-04-02T14:51:12.190 INFO:teuthology.orchestra.run.smithi092.stdout: "recovery_started": 0, 2024-04-02T14:51:12.190 INFO:teuthology.orchestra.run.smithi092.stdout: "recovery_completed": 0, 2024-04-02T14:51:12.190 INFO:teuthology.orchestra.run.smithi092.stdout: "ireq_quiesce_path": 0, 2024-04-02T14:51:12.190 INFO:teuthology.orchestra.run.smithi092.stdout: "ireq_quiesce_inode": 0, 2024-04-02T14:51:12.190 INFO:teuthology.orchestra.run.smithi092.stdout: "ireq_enqueue_scrub": 0, 2024-04-02T14:51:12.190 INFO:teuthology.orchestra.run.smithi092.stdout: "ireq_exportdir": 0, 2024-04-02T14:51:12.190 INFO:teuthology.orchestra.run.smithi092.stdout: "ireq_flush": 0, 2024-04-02T14:51:12.190 INFO:teuthology.orchestra.run.smithi092.stdout: "ireq_fragmentdir": 0, 2024-04-02T14:51:12.191 INFO:teuthology.orchestra.run.smithi092.stdout: "ireq_fragstats": 0, 2024-04-02T14:51:12.191 INFO:teuthology.orchestra.run.smithi092.stdout: "ireq_inodestats": 0 2024-04-02T14:51:12.191 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:12.191 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_log": { 2024-04-02T14:51:12.191 INFO:teuthology.orchestra.run.smithi092.stdout: "evlrg": 0, 2024-04-02T14:51:12.191 INFO:teuthology.orchestra.run.smithi092.stdout: "evadd": 0, 2024-04-02T14:51:12.191 INFO:teuthology.orchestra.run.smithi092.stdout: "evex": 0, 2024-04-02T14:51:12.191 INFO:teuthology.orchestra.run.smithi092.stdout: "evtrm": 0, 2024-04-02T14:51:12.191 INFO:teuthology.orchestra.run.smithi092.stdout: "ev": 7, 2024-04-02T14:51:12.191 INFO:teuthology.orchestra.run.smithi092.stdout: "evexg": 0, 2024-04-02T14:51:12.191 INFO:teuthology.orchestra.run.smithi092.stdout: "evexd": 0, 2024-04-02T14:51:12.191 INFO:teuthology.orchestra.run.smithi092.stdout: "segadd": 0, 2024-04-02T14:51:12.191 INFO:teuthology.orchestra.run.smithi092.stdout: "segex": 0, 2024-04-02T14:51:12.192 INFO:teuthology.orchestra.run.smithi092.stdout: "segtrm": 0, 2024-04-02T14:51:12.192 INFO:teuthology.orchestra.run.smithi092.stdout: "seg": 2, 2024-04-02T14:51:12.192 INFO:teuthology.orchestra.run.smithi092.stdout: "segmjr": 2, 2024-04-02T14:51:12.192 INFO:teuthology.orchestra.run.smithi092.stdout: "segexg": 0, 2024-04-02T14:51:12.192 INFO:teuthology.orchestra.run.smithi092.stdout: "segexd": 0, 2024-04-02T14:51:12.192 INFO:teuthology.orchestra.run.smithi092.stdout: "expos": 4194304, 2024-04-02T14:51:12.192 INFO:teuthology.orchestra.run.smithi092.stdout: "wrpos": 4196262, 2024-04-02T14:51:12.192 INFO:teuthology.orchestra.run.smithi092.stdout: "rdpos": 4196121, 2024-04-02T14:51:12.192 INFO:teuthology.orchestra.run.smithi092.stdout: "jlat": { 2024-04-02T14:51:12.192 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2024-04-02T14:51:12.192 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2024-04-02T14:51:12.192 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2024-04-02T14:51:12.192 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:12.192 INFO:teuthology.orchestra.run.smithi092.stdout: "replayed": 7 2024-04-02T14:51:12.192 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:12.193 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_mem": { 2024-04-02T14:51:12.193 INFO:teuthology.orchestra.run.smithi092.stdout: "ino": 3, 2024-04-02T14:51:12.193 INFO:teuthology.orchestra.run.smithi092.stdout: "ino+": 3, 2024-04-02T14:51:12.193 INFO:teuthology.orchestra.run.smithi092.stdout: "ino-": 0, 2024-04-02T14:51:12.193 INFO:teuthology.orchestra.run.smithi092.stdout: "dir": 1, 2024-04-02T14:51:12.193 INFO:teuthology.orchestra.run.smithi092.stdout: "dir+": 1, 2024-04-02T14:51:12.193 INFO:teuthology.orchestra.run.smithi092.stdout: "dir-": 0, 2024-04-02T14:51:12.193 INFO:teuthology.orchestra.run.smithi092.stdout: "dn": 0, 2024-04-02T14:51:12.193 INFO:teuthology.orchestra.run.smithi092.stdout: "dn+": 0, 2024-04-02T14:51:12.193 INFO:teuthology.orchestra.run.smithi092.stdout: "dn-": 0, 2024-04-02T14:51:12.193 INFO:teuthology.orchestra.run.smithi092.stdout: "cap": 0, 2024-04-02T14:51:12.193 INFO:teuthology.orchestra.run.smithi092.stdout: "cap+": 0, 2024-04-02T14:51:12.193 INFO:teuthology.orchestra.run.smithi092.stdout: "cap-": 0, 2024-04-02T14:51:12.193 INFO:teuthology.orchestra.run.smithi092.stdout: "rss": 48308, 2024-04-02T14:51:12.194 INFO:teuthology.orchestra.run.smithi092.stdout: "heap": 215416 2024-04-02T14:51:12.194 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:12.194 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_server": { 2024-04-02T14:51:12.194 INFO:teuthology.orchestra.run.smithi092.stdout: "dispatch_client_request": 0, 2024-04-02T14:51:12.194 INFO:teuthology.orchestra.run.smithi092.stdout: "dispatch_server_request": 0, 2024-04-02T14:51:12.194 INFO:teuthology.orchestra.run.smithi092.stdout: "handle_client_request": 0, 2024-04-02T14:51:12.194 INFO:teuthology.orchestra.run.smithi092.stdout: "handle_client_session": 0, 2024-04-02T14:51:12.194 INFO:teuthology.orchestra.run.smithi092.stdout: "handle_peer_request": 0, 2024-04-02T14:51:12.194 INFO:teuthology.orchestra.run.smithi092.stdout: "req_create_latency": { 2024-04-02T14:51:12.194 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2024-04-02T14:51:12.194 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2024-04-02T14:51:12.194 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2024-04-02T14:51:12.194 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:12.194 INFO:teuthology.orchestra.run.smithi092.stdout: "req_getattr_latency": { 2024-04-02T14:51:12.195 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2024-04-02T14:51:12.195 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2024-04-02T14:51:12.195 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2024-04-02T14:51:12.195 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:12.195 INFO:teuthology.orchestra.run.smithi092.stdout: "req_getfilelock_latency": { 2024-04-02T14:51:12.195 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2024-04-02T14:51:12.195 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2024-04-02T14:51:12.195 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2024-04-02T14:51:12.195 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:12.195 INFO:teuthology.orchestra.run.smithi092.stdout: "req_link_latency": { 2024-04-02T14:51:12.195 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2024-04-02T14:51:12.195 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2024-04-02T14:51:12.195 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2024-04-02T14:51:12.195 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:12.195 INFO:teuthology.orchestra.run.smithi092.stdout: "req_lookup_latency": { 2024-04-02T14:51:12.196 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2024-04-02T14:51:12.196 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2024-04-02T14:51:12.196 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2024-04-02T14:51:12.196 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:12.196 INFO:teuthology.orchestra.run.smithi092.stdout: "req_lookuphash_latency": { 2024-04-02T14:51:12.196 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2024-04-02T14:51:12.196 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2024-04-02T14:51:12.196 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2024-04-02T14:51:12.196 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:12.196 INFO:teuthology.orchestra.run.smithi092.stdout: "req_lookupino_latency": { 2024-04-02T14:51:12.196 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2024-04-02T14:51:12.196 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2024-04-02T14:51:12.196 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2024-04-02T14:51:12.196 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:12.196 INFO:teuthology.orchestra.run.smithi092.stdout: "req_lookupname_latency": { 2024-04-02T14:51:12.197 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2024-04-02T14:51:12.197 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2024-04-02T14:51:12.197 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2024-04-02T14:51:12.197 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:12.197 INFO:teuthology.orchestra.run.smithi092.stdout: "req_lookupparent_latency": { 2024-04-02T14:51:12.197 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2024-04-02T14:51:12.197 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2024-04-02T14:51:12.197 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2024-04-02T14:51:12.197 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:12.197 INFO:teuthology.orchestra.run.smithi092.stdout: "req_lookupsnap_latency": { 2024-04-02T14:51:12.197 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2024-04-02T14:51:12.197 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2024-04-02T14:51:12.197 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2024-04-02T14:51:12.197 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:12.198 INFO:teuthology.orchestra.run.smithi092.stdout: "req_lssnap_latency": { 2024-04-02T14:51:12.198 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2024-04-02T14:51:12.198 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2024-04-02T14:51:12.198 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2024-04-02T14:51:12.198 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:12.198 INFO:teuthology.orchestra.run.smithi092.stdout: "req_mkdir_latency": { 2024-04-02T14:51:12.198 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2024-04-02T14:51:12.199 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:51:11 smithi090 ceph-mon[31674]: pgmap v511: 97 pgs: 40 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 35 active+clean; 4.1 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 89 KiB/s wr, 32 op/s 2024-04-02T14:51:12.199 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:51:11 smithi177 ceph-mon[36431]: pgmap v511: 97 pgs: 40 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 35 active+clean; 4.1 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 89 KiB/s wr, 32 op/s 2024-04-02T14:51:12.199 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2024-04-02T14:51:12.199 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2024-04-02T14:51:12.199 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:12.200 INFO:teuthology.orchestra.run.smithi092.stdout: "req_mknod_latency": { 2024-04-02T14:51:12.200 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2024-04-02T14:51:12.200 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2024-04-02T14:51:12.200 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2024-04-02T14:51:12.200 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:12.200 INFO:teuthology.orchestra.run.smithi092.stdout: "req_mksnap_latency": { 2024-04-02T14:51:12.200 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2024-04-02T14:51:12.200 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2024-04-02T14:51:12.200 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2024-04-02T14:51:12.200 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:12.200 INFO:teuthology.orchestra.run.smithi092.stdout: "req_open_latency": { 2024-04-02T14:51:12.200 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2024-04-02T14:51:12.200 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2024-04-02T14:51:12.201 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2024-04-02T14:51:12.201 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:12.201 INFO:teuthology.orchestra.run.smithi092.stdout: "req_readdir_latency": { 2024-04-02T14:51:12.201 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2024-04-02T14:51:12.201 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2024-04-02T14:51:12.201 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2024-04-02T14:51:12.201 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:12.201 INFO:teuthology.orchestra.run.smithi092.stdout: "req_rename_latency": { 2024-04-02T14:51:12.201 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2024-04-02T14:51:12.201 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2024-04-02T14:51:12.201 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2024-04-02T14:51:12.201 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:12.201 INFO:teuthology.orchestra.run.smithi092.stdout: "req_renamesnap_latency": { 2024-04-02T14:51:12.201 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2024-04-02T14:51:12.201 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2024-04-02T14:51:12.202 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2024-04-02T14:51:12.202 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:12.202 INFO:teuthology.orchestra.run.smithi092.stdout: "req_snapdiff_latency": { 2024-04-02T14:51:12.202 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2024-04-02T14:51:12.202 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2024-04-02T14:51:12.202 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2024-04-02T14:51:12.202 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:12.202 INFO:teuthology.orchestra.run.smithi092.stdout: "req_rmdir_latency": { 2024-04-02T14:51:12.202 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2024-04-02T14:51:12.202 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2024-04-02T14:51:12.202 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2024-04-02T14:51:12.202 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:12.202 INFO:teuthology.orchestra.run.smithi092.stdout: "req_rmsnap_latency": { 2024-04-02T14:51:12.202 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2024-04-02T14:51:12.203 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2024-04-02T14:51:12.203 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2024-04-02T14:51:12.203 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:12.203 INFO:teuthology.orchestra.run.smithi092.stdout: "req_rmxattr_latency": { 2024-04-02T14:51:12.203 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2024-04-02T14:51:12.203 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2024-04-02T14:51:12.203 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2024-04-02T14:51:12.203 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:12.203 INFO:teuthology.orchestra.run.smithi092.stdout: "req_setattr_latency": { 2024-04-02T14:51:12.203 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2024-04-02T14:51:12.203 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2024-04-02T14:51:12.203 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2024-04-02T14:51:12.203 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:12.204 INFO:teuthology.orchestra.run.smithi092.stdout: "req_setdirlayout_latency": { 2024-04-02T14:51:12.204 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2024-04-02T14:51:12.204 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2024-04-02T14:51:12.204 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2024-04-02T14:51:12.204 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:12.204 INFO:teuthology.orchestra.run.smithi092.stdout: "req_setfilelock_latency": { 2024-04-02T14:51:12.204 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2024-04-02T14:51:12.204 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2024-04-02T14:51:12.204 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2024-04-02T14:51:12.204 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:12.204 INFO:teuthology.orchestra.run.smithi092.stdout: "req_setlayout_latency": { 2024-04-02T14:51:12.204 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2024-04-02T14:51:12.204 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2024-04-02T14:51:12.204 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2024-04-02T14:51:12.205 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:12.205 INFO:teuthology.orchestra.run.smithi092.stdout: "req_setxattr_latency": { 2024-04-02T14:51:12.205 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2024-04-02T14:51:12.205 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2024-04-02T14:51:12.205 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2024-04-02T14:51:12.205 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:12.205 INFO:teuthology.orchestra.run.smithi092.stdout: "req_symlink_latency": { 2024-04-02T14:51:12.205 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2024-04-02T14:51:12.205 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2024-04-02T14:51:12.205 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2024-04-02T14:51:12.205 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:12.205 INFO:teuthology.orchestra.run.smithi092.stdout: "req_unlink_latency": { 2024-04-02T14:51:12.205 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2024-04-02T14:51:12.205 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2024-04-02T14:51:12.206 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2024-04-02T14:51:12.206 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:12.206 INFO:teuthology.orchestra.run.smithi092.stdout: "cap_revoke_eviction": 0, 2024-04-02T14:51:12.206 INFO:teuthology.orchestra.run.smithi092.stdout: "cap_acquisition_throttle": 0, 2024-04-02T14:51:12.206 INFO:teuthology.orchestra.run.smithi092.stdout: "req_getvxattr_latency": { 2024-04-02T14:51:12.206 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2024-04-02T14:51:12.206 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2024-04-02T14:51:12.206 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2024-04-02T14:51:12.206 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-02T14:51:12.206 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:12.206 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_sessions": { 2024-04-02T14:51:12.206 INFO:teuthology.orchestra.run.smithi092.stdout: "session_count": 1, 2024-04-02T14:51:12.206 INFO:teuthology.orchestra.run.smithi092.stdout: "session_add": 3, 2024-04-02T14:51:12.207 INFO:teuthology.orchestra.run.smithi092.stdout: "session_remove": 2, 2024-04-02T14:51:12.207 INFO:teuthology.orchestra.run.smithi092.stdout: "sessions_open": 2, 2024-04-02T14:51:12.207 INFO:teuthology.orchestra.run.smithi092.stdout: "sessions_stale": 0, 2024-04-02T14:51:12.207 INFO:teuthology.orchestra.run.smithi092.stdout: "total_load": 0, 2024-04-02T14:51:12.207 INFO:teuthology.orchestra.run.smithi092.stdout: "average_load": 0, 2024-04-02T14:51:12.207 INFO:teuthology.orchestra.run.smithi092.stdout: "avg_session_uptime": 4358, 2024-04-02T14:51:12.207 INFO:teuthology.orchestra.run.smithi092.stdout: "mdthresh_evicted": 0 2024-04-02T14:51:12.207 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:12.207 INFO:teuthology.orchestra.run.smithi092.stdout: "mempool": { 2024-04-02T14:51:12.207 INFO:teuthology.orchestra.run.smithi092.stdout: "bloom_filter_bytes": 0, 2024-04-02T14:51:12.207 INFO:teuthology.orchestra.run.smithi092.stdout: "bloom_filter_items": 0, 2024-04-02T14:51:12.207 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_alloc_bytes": 0, 2024-04-02T14:51:12.207 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_alloc_items": 0, 2024-04-02T14:51:12.207 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_cache_data_bytes": 0, 2024-04-02T14:51:12.208 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_cache_data_items": 0, 2024-04-02T14:51:12.208 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-02T14:51:12.208 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_cache_onode_items": 0, 2024-04-02T14:51:12.208 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-02T14:51:12.208 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_cache_meta_items": 0, 2024-04-02T14:51:12.208 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_cache_other_bytes": 0, 2024-04-02T14:51:12.208 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_cache_other_items": 0, 2024-04-02T14:51:12.208 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-02T14:51:12.208 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_cache_buffer_items": 0, 2024-04-02T14:51:12.208 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_extent_bytes": 0, 2024-04-02T14:51:12.208 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_extent_items": 0, 2024-04-02T14:51:12.208 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_blob_bytes": 0, 2024-04-02T14:51:12.208 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_blob_items": 0, 2024-04-02T14:51:12.208 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-02T14:51:12.208 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_shared_blob_items": 0, 2024-04-02T14:51:12.209 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-02T14:51:12.209 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_inline_bl_items": 0, 2024-04-02T14:51:12.209 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_fsck_bytes": 0, 2024-04-02T14:51:12.209 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_fsck_items": 0, 2024-04-02T14:51:12.209 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_txc_bytes": 0, 2024-04-02T14:51:12.209 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_txc_items": 0, 2024-04-02T14:51:12.209 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-02T14:51:12.209 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_writing_deferred_items": 0, 2024-04-02T14:51:12.209 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_writing_bytes": 0, 2024-04-02T14:51:12.209 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_writing_items": 0, 2024-04-02T14:51:12.209 INFO:teuthology.orchestra.run.smithi092.stdout: "bluefs_bytes": 0, 2024-04-02T14:51:12.209 INFO:teuthology.orchestra.run.smithi092.stdout: "bluefs_items": 0, 2024-04-02T14:51:12.209 INFO:teuthology.orchestra.run.smithi092.stdout: "bluefs_file_reader_bytes": 0, 2024-04-02T14:51:12.209 INFO:teuthology.orchestra.run.smithi092.stdout: "bluefs_file_reader_items": 0, 2024-04-02T14:51:12.210 INFO:teuthology.orchestra.run.smithi092.stdout: "bluefs_file_writer_bytes": 0, 2024-04-02T14:51:12.210 INFO:teuthology.orchestra.run.smithi092.stdout: "bluefs_file_writer_items": 0, 2024-04-02T14:51:12.210 INFO:teuthology.orchestra.run.smithi092.stdout: "buffer_anon_bytes": 182115, 2024-04-02T14:51:12.210 INFO:teuthology.orchestra.run.smithi092.stdout: "buffer_anon_items": 50, 2024-04-02T14:51:12.210 INFO:teuthology.orchestra.run.smithi092.stdout: "buffer_meta_bytes": 0, 2024-04-02T14:51:12.210 INFO:teuthology.orchestra.run.smithi092.stdout: "buffer_meta_items": 0, 2024-04-02T14:51:12.210 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_bytes": 0, 2024-04-02T14:51:12.210 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_items": 0, 2024-04-02T14:51:12.210 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_mapbl_bytes": 0, 2024-04-02T14:51:12.210 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_mapbl_items": 0, 2024-04-02T14:51:12.210 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_pglog_bytes": 0, 2024-04-02T14:51:12.210 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_pglog_items": 0, 2024-04-02T14:51:12.210 INFO:teuthology.orchestra.run.smithi092.stdout: "osdmap_bytes": 8256, 2024-04-02T14:51:12.210 INFO:teuthology.orchestra.run.smithi092.stdout: "osdmap_items": 173, 2024-04-02T14:51:12.211 INFO:teuthology.orchestra.run.smithi092.stdout: "osdmap_mapping_bytes": 0, 2024-04-02T14:51:12.211 INFO:teuthology.orchestra.run.smithi092.stdout: "osdmap_mapping_items": 0, 2024-04-02T14:51:12.211 INFO:teuthology.orchestra.run.smithi092.stdout: "pgmap_bytes": 0, 2024-04-02T14:51:12.211 INFO:teuthology.orchestra.run.smithi092.stdout: "pgmap_items": 0, 2024-04-02T14:51:12.211 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_co_bytes": 8464, 2024-04-02T14:51:12.211 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_co_items": 13, 2024-04-02T14:51:12.211 INFO:teuthology.orchestra.run.smithi092.stdout: "unittest_1_bytes": 0, 2024-04-02T14:51:12.211 INFO:teuthology.orchestra.run.smithi092.stdout: "unittest_1_items": 0, 2024-04-02T14:51:12.211 INFO:teuthology.orchestra.run.smithi092.stdout: "unittest_2_bytes": 0, 2024-04-02T14:51:12.211 INFO:teuthology.orchestra.run.smithi092.stdout: "unittest_2_items": 0 2024-04-02T14:51:12.211 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:12.211 INFO:teuthology.orchestra.run.smithi092.stdout: "objecter": { 2024-04-02T14:51:12.211 INFO:teuthology.orchestra.run.smithi092.stdout: "op_active": 0, 2024-04-02T14:51:12.211 INFO:teuthology.orchestra.run.smithi092.stdout: "op_laggy": 0, 2024-04-02T14:51:12.211 INFO:teuthology.orchestra.run.smithi092.stdout: "op_send": 3305, 2024-04-02T14:51:12.212 INFO:teuthology.orchestra.run.smithi092.stdout: "op_send_bytes": 21, 2024-04-02T14:51:12.212 INFO:teuthology.orchestra.run.smithi092.stdout: "op_resend": 3, 2024-04-02T14:51:12.212 INFO:teuthology.orchestra.run.smithi092.stdout: "op_reply": 3305, 2024-04-02T14:51:12.212 INFO:teuthology.orchestra.run.smithi092.stdout: "op_latency": { 2024-04-02T14:51:12.212 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 3305, 2024-04-02T14:51:12.212 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 3.203907067, 2024-04-02T14:51:12.212 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000969412 2024-04-02T14:51:12.212 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:12.212 INFO:teuthology.orchestra.run.smithi092.stdout: "op_inflight": 0, 2024-04-02T14:51:12.212 INFO:teuthology.orchestra.run.smithi092.stdout: "oplen_avg": { 2024-04-02T14:51:12.212 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 3305, 2024-04-02T14:51:12.212 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 3306 2024-04-02T14:51:12.212 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:12.212 INFO:teuthology.orchestra.run.smithi092.stdout: "op": 3305, 2024-04-02T14:51:12.212 INFO:teuthology.orchestra.run.smithi092.stdout: "op_r": 3305, 2024-04-02T14:51:12.213 INFO:teuthology.orchestra.run.smithi092.stdout: "op_w": 0, 2024-04-02T14:51:12.213 INFO:teuthology.orchestra.run.smithi092.stdout: "op_rmw": 0, 2024-04-02T14:51:12.213 INFO:teuthology.orchestra.run.smithi092.stdout: "op_pg": 0, 2024-04-02T14:51:12.213 INFO:teuthology.orchestra.run.smithi092.stdout: "osdop_stat": 2175, 2024-04-02T14:51:12.213 INFO:teuthology.orchestra.run.smithi092.stdout: "osdop_create": 0, 2024-04-02T14:51:12.213 INFO:teuthology.orchestra.run.smithi092.stdout: "osdop_read": 1128, 2024-04-02T14:51:12.213 INFO:teuthology.orchestra.run.smithi092.stdout: "osdop_write": 0, 2024-04-02T14:51:12.213 INFO:teuthology.orchestra.run.smithi092.stdout: "osdop_writefull": 0, 2024-04-02T14:51:12.213 INFO:teuthology.orchestra.run.smithi092.stdout: "osdop_writesame": 0, 2024-04-02T14:51:12.213 INFO:teuthology.orchestra.run.smithi092.stdout: "osdop_append": 0, 2024-04-02T14:51:12.213 INFO:teuthology.orchestra.run.smithi092.stdout: "osdop_zero": 0, 2024-04-02T14:51:12.213 INFO:teuthology.orchestra.run.smithi092.stdout: "osdop_truncate": 0, 2024-04-02T14:51:12.213 INFO:teuthology.orchestra.run.smithi092.stdout: "osdop_delete": 0, 2024-04-02T14:51:12.213 INFO:teuthology.orchestra.run.smithi092.stdout: "osdop_mapext": 0, 2024-04-02T14:51:12.213 INFO:teuthology.orchestra.run.smithi092.stdout: "osdop_sparse_read": 0, 2024-04-02T14:51:12.214 INFO:teuthology.orchestra.run.smithi092.stdout: "osdop_clonerange": 0, 2024-04-02T14:51:12.214 INFO:teuthology.orchestra.run.smithi092.stdout: "osdop_getxattr": 1, 2024-04-02T14:51:12.214 INFO:teuthology.orchestra.run.smithi092.stdout: "osdop_setxattr": 0, 2024-04-02T14:51:12.214 INFO:teuthology.orchestra.run.smithi092.stdout: "osdop_cmpxattr": 0, 2024-04-02T14:51:12.214 INFO:teuthology.orchestra.run.smithi092.stdout: "osdop_rmxattr": 0, 2024-04-02T14:51:12.214 INFO:teuthology.orchestra.run.smithi092.stdout: "osdop_resetxattrs": 0, 2024-04-02T14:51:12.214 INFO:teuthology.orchestra.run.smithi092.stdout: "osdop_call": 0, 2024-04-02T14:51:12.214 INFO:teuthology.orchestra.run.smithi092.stdout: "osdop_watch": 0, 2024-04-02T14:51:12.214 INFO:teuthology.orchestra.run.smithi092.stdout: "osdop_notify": 0, 2024-04-02T14:51:12.214 INFO:teuthology.orchestra.run.smithi092.stdout: "osdop_src_cmpxattr": 0, 2024-04-02T14:51:12.214 INFO:teuthology.orchestra.run.smithi092.stdout: "osdop_pgls": 0, 2024-04-02T14:51:12.214 INFO:teuthology.orchestra.run.smithi092.stdout: "osdop_pgls_filter": 0, 2024-04-02T14:51:12.214 INFO:teuthology.orchestra.run.smithi092.stdout: "osdop_other": 0, 2024-04-02T14:51:12.214 INFO:teuthology.orchestra.run.smithi092.stdout: "linger_active": 0, 2024-04-02T14:51:12.215 INFO:teuthology.orchestra.run.smithi092.stdout: "linger_send": 0, 2024-04-02T14:51:12.215 INFO:teuthology.orchestra.run.smithi092.stdout: "linger_resend": 0, 2024-04-02T14:51:12.215 INFO:teuthology.orchestra.run.smithi092.stdout: "linger_ping": 0, 2024-04-02T14:51:12.215 INFO:teuthology.orchestra.run.smithi092.stdout: "poolop_active": 0, 2024-04-02T14:51:12.215 INFO:teuthology.orchestra.run.smithi092.stdout: "poolop_send": 0, 2024-04-02T14:51:12.215 INFO:teuthology.orchestra.run.smithi092.stdout: "poolop_resend": 0, 2024-04-02T14:51:12.215 INFO:teuthology.orchestra.run.smithi092.stdout: "poolstat_active": 0, 2024-04-02T14:51:12.215 INFO:teuthology.orchestra.run.smithi092.stdout: "poolstat_send": 0, 2024-04-02T14:51:12.215 INFO:teuthology.orchestra.run.smithi092.stdout: "poolstat_resend": 0, 2024-04-02T14:51:12.215 INFO:teuthology.orchestra.run.smithi092.stdout: "statfs_active": 0, 2024-04-02T14:51:12.215 INFO:teuthology.orchestra.run.smithi092.stdout: "statfs_send": 0, 2024-04-02T14:51:12.215 INFO:teuthology.orchestra.run.smithi092.stdout: "statfs_resend": 0, 2024-04-02T14:51:12.215 INFO:teuthology.orchestra.run.smithi092.stdout: "command_active": 0, 2024-04-02T14:51:12.215 INFO:teuthology.orchestra.run.smithi092.stdout: "command_send": 0, 2024-04-02T14:51:12.215 INFO:teuthology.orchestra.run.smithi092.stdout: "command_resend": 0, 2024-04-02T14:51:12.215 INFO:teuthology.orchestra.run.smithi092.stdout: "map_epoch": 110, 2024-04-02T14:51:12.216 INFO:teuthology.orchestra.run.smithi092.stdout: "map_full": 0, 2024-04-02T14:51:12.216 INFO:teuthology.orchestra.run.smithi092.stdout: "map_inc": 22, 2024-04-02T14:51:12.216 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_sessions": 4, 2024-04-02T14:51:12.216 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_session_open": 5, 2024-04-02T14:51:12.216 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_session_close": 1, 2024-04-02T14:51:12.216 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_laggy": 0, 2024-04-02T14:51:12.216 INFO:teuthology.orchestra.run.smithi092.stdout: "omap_wr": 0, 2024-04-02T14:51:12.216 INFO:teuthology.orchestra.run.smithi092.stdout: "omap_rd": 2, 2024-04-02T14:51:12.216 INFO:teuthology.orchestra.run.smithi092.stdout: "omap_del": 0 2024-04-02T14:51:12.216 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:12.216 INFO:teuthology.orchestra.run.smithi092.stdout: "oft": { 2024-04-02T14:51:12.216 INFO:teuthology.orchestra.run.smithi092.stdout: "omap_total_objs": 0, 2024-04-02T14:51:12.216 INFO:teuthology.orchestra.run.smithi092.stdout: "omap_total_kv_pairs": 0, 2024-04-02T14:51:12.216 INFO:teuthology.orchestra.run.smithi092.stdout: "omap_total_updates": 0, 2024-04-02T14:51:12.217 INFO:teuthology.orchestra.run.smithi092.stdout: "omap_total_removes": 0 2024-04-02T14:51:12.217 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:12.217 INFO:teuthology.orchestra.run.smithi092.stdout: "osd-slow-ops": { 2024-04-02T14:51:12.217 INFO:teuthology.orchestra.run.smithi092.stdout: "slow_ops_count": 0 2024-04-02T14:51:12.217 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:12.217 INFO:teuthology.orchestra.run.smithi092.stdout: "purge_queue": { 2024-04-02T14:51:12.217 INFO:teuthology.orchestra.run.smithi092.stdout: "pq_executing_ops": 0, 2024-04-02T14:51:12.217 INFO:teuthology.orchestra.run.smithi092.stdout: "pq_executing_ops_high_water": 0, 2024-04-02T14:51:12.217 INFO:teuthology.orchestra.run.smithi092.stdout: "pq_executing": 0, 2024-04-02T14:51:12.217 INFO:teuthology.orchestra.run.smithi092.stdout: "pq_executing_high_water": 0, 2024-04-02T14:51:12.217 INFO:teuthology.orchestra.run.smithi092.stdout: "pq_executed_ops": 0, 2024-04-02T14:51:12.217 INFO:teuthology.orchestra.run.smithi092.stdout: "pq_executed": 0, 2024-04-02T14:51:12.217 INFO:teuthology.orchestra.run.smithi092.stdout: "pq_item_in_journal": 0 2024-04-02T14:51:12.217 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:12.217 INFO:teuthology.orchestra.run.smithi092.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-02T14:51:12.218 INFO:teuthology.orchestra.run.smithi092.stdout: "val": 0, 2024-04-02T14:51:12.218 INFO:teuthology.orchestra.run.smithi092.stdout: "max": 104857600, 2024-04-02T14:51:12.218 INFO:teuthology.orchestra.run.smithi092.stdout: "get_started": 0, 2024-04-02T14:51:12.218 INFO:teuthology.orchestra.run.smithi092.stdout: "get": 3654, 2024-04-02T14:51:12.218 INFO:teuthology.orchestra.run.smithi092.stdout: "get_sum": 2242554, 2024-04-02T14:51:12.218 INFO:teuthology.orchestra.run.smithi092.stdout: "get_or_fail_fail": 0, 2024-04-02T14:51:12.218 INFO:teuthology.orchestra.run.smithi092.stdout: "get_or_fail_success": 3654, 2024-04-02T14:51:12.218 INFO:teuthology.orchestra.run.smithi092.stdout: "take": 0, 2024-04-02T14:51:12.218 INFO:teuthology.orchestra.run.smithi092.stdout: "take_sum": 0, 2024-04-02T14:51:12.218 INFO:teuthology.orchestra.run.smithi092.stdout: "put": 3654, 2024-04-02T14:51:12.218 INFO:teuthology.orchestra.run.smithi092.stdout: "put_sum": 2242554, 2024-04-02T14:51:12.218 INFO:teuthology.orchestra.run.smithi092.stdout: "wait": { 2024-04-02T14:51:12.218 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2024-04-02T14:51:12.218 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2024-04-02T14:51:12.218 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2024-04-02T14:51:12.219 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-02T14:51:12.219 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:12.219 INFO:teuthology.orchestra.run.smithi092.stdout: "throttle-objecter_bytes": { 2024-04-02T14:51:12.219 INFO:teuthology.orchestra.run.smithi092.stdout: "val": 0, 2024-04-02T14:51:12.219 INFO:teuthology.orchestra.run.smithi092.stdout: "max": 104857600, 2024-04-02T14:51:12.219 INFO:teuthology.orchestra.run.smithi092.stdout: "get_started": 0, 2024-04-02T14:51:12.219 INFO:teuthology.orchestra.run.smithi092.stdout: "get": 0, 2024-04-02T14:51:12.219 INFO:teuthology.orchestra.run.smithi092.stdout: "get_sum": 0, 2024-04-02T14:51:12.219 INFO:teuthology.orchestra.run.smithi092.stdout: "get_or_fail_fail": 0, 2024-04-02T14:51:12.219 INFO:teuthology.orchestra.run.smithi092.stdout: "get_or_fail_success": 0, 2024-04-02T14:51:12.219 INFO:teuthology.orchestra.run.smithi092.stdout: "take": 3305, 2024-04-02T14:51:12.219 INFO:teuthology.orchestra.run.smithi092.stdout: "take_sum": 1963, 2024-04-02T14:51:12.219 INFO:teuthology.orchestra.run.smithi092.stdout: "put": 3305, 2024-04-02T14:51:12.219 INFO:teuthology.orchestra.run.smithi092.stdout: "put_sum": 1963, 2024-04-02T14:51:12.220 INFO:teuthology.orchestra.run.smithi092.stdout: "wait": { 2024-04-02T14:51:12.220 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2024-04-02T14:51:12.220 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2024-04-02T14:51:12.220 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2024-04-02T14:51:12.220 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-02T14:51:12.220 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:12.220 INFO:teuthology.orchestra.run.smithi092.stdout: "throttle-objecter_ops": { 2024-04-02T14:51:12.220 INFO:teuthology.orchestra.run.smithi092.stdout: "val": 0, 2024-04-02T14:51:12.220 INFO:teuthology.orchestra.run.smithi092.stdout: "max": 1024, 2024-04-02T14:51:12.220 INFO:teuthology.orchestra.run.smithi092.stdout: "get_started": 0, 2024-04-02T14:51:12.220 INFO:teuthology.orchestra.run.smithi092.stdout: "get": 0, 2024-04-02T14:51:12.220 INFO:teuthology.orchestra.run.smithi092.stdout: "get_sum": 0, 2024-04-02T14:51:12.220 INFO:teuthology.orchestra.run.smithi092.stdout: "get_or_fail_fail": 0, 2024-04-02T14:51:12.220 INFO:teuthology.orchestra.run.smithi092.stdout: "get_or_fail_success": 0, 2024-04-02T14:51:12.220 INFO:teuthology.orchestra.run.smithi092.stdout: "take": 3305, 2024-04-02T14:51:12.221 INFO:teuthology.orchestra.run.smithi092.stdout: "take_sum": 3305, 2024-04-02T14:51:12.221 INFO:teuthology.orchestra.run.smithi092.stdout: "put": 3305, 2024-04-02T14:51:12.221 INFO:teuthology.orchestra.run.smithi092.stdout: "put_sum": 3305, 2024-04-02T14:51:12.221 INFO:teuthology.orchestra.run.smithi092.stdout: "wait": { 2024-04-02T14:51:12.221 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2024-04-02T14:51:12.221 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2024-04-02T14:51:12.221 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2024-04-02T14:51:12.221 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-02T14:51:12.221 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:12.221 INFO:teuthology.orchestra.run.smithi092.stdout: "throttle-write_buf_throttle": { 2024-04-02T14:51:12.221 INFO:teuthology.orchestra.run.smithi092.stdout: "val": 0, 2024-04-02T14:51:12.221 INFO:teuthology.orchestra.run.smithi092.stdout: "max": 3758096384, 2024-04-02T14:51:12.221 INFO:teuthology.orchestra.run.smithi092.stdout: "get_started": 0, 2024-04-02T14:51:12.221 INFO:teuthology.orchestra.run.smithi092.stdout: "get": 0, 2024-04-02T14:51:12.221 INFO:teuthology.orchestra.run.smithi092.stdout: "get_sum": 0, 2024-04-02T14:51:12.222 INFO:teuthology.orchestra.run.smithi092.stdout: "get_or_fail_fail": 0, 2024-04-02T14:51:12.222 INFO:teuthology.orchestra.run.smithi092.stdout: "get_or_fail_success": 0, 2024-04-02T14:51:12.222 INFO:teuthology.orchestra.run.smithi092.stdout: "take": 0, 2024-04-02T14:51:12.222 INFO:teuthology.orchestra.run.smithi092.stdout: "take_sum": 0, 2024-04-02T14:51:12.222 INFO:teuthology.orchestra.run.smithi092.stdout: "put": 0, 2024-04-02T14:51:12.222 INFO:teuthology.orchestra.run.smithi092.stdout: "put_sum": 0, 2024-04-02T14:51:12.222 INFO:teuthology.orchestra.run.smithi092.stdout: "wait": { 2024-04-02T14:51:12.222 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2024-04-02T14:51:12.222 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2024-04-02T14:51:12.222 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2024-04-02T14:51:12.222 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-02T14:51:12.222 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:12.222 INFO:teuthology.orchestra.run.smithi092.stdout: "throttle-write_buf_throttle-0x56302fbe46e0": { 2024-04-02T14:51:12.222 INFO:teuthology.orchestra.run.smithi092.stdout: "val": 0, 2024-04-02T14:51:12.223 INFO:teuthology.orchestra.run.smithi092.stdout: "max": 3758096384, 2024-04-02T14:51:12.223 INFO:teuthology.orchestra.run.smithi092.stdout: "get_started": 0, 2024-04-02T14:51:12.223 INFO:teuthology.orchestra.run.smithi092.stdout: "get": 0, 2024-04-02T14:51:12.223 INFO:teuthology.orchestra.run.smithi092.stdout: "get_sum": 0, 2024-04-02T14:51:12.223 INFO:teuthology.orchestra.run.smithi092.stdout: "get_or_fail_fail": 0, 2024-04-02T14:51:12.223 INFO:teuthology.orchestra.run.smithi092.stdout: "get_or_fail_success": 0, 2024-04-02T14:51:12.223 INFO:teuthology.orchestra.run.smithi092.stdout: "take": 0, 2024-04-02T14:51:12.223 INFO:teuthology.orchestra.run.smithi092.stdout: "take_sum": 0, 2024-04-02T14:51:12.223 INFO:teuthology.orchestra.run.smithi092.stdout: "put": 0, 2024-04-02T14:51:12.223 INFO:teuthology.orchestra.run.smithi092.stdout: "put_sum": 0, 2024-04-02T14:51:12.223 INFO:teuthology.orchestra.run.smithi092.stdout: "wait": { 2024-04-02T14:51:12.223 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2024-04-02T14:51:12.223 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2024-04-02T14:51:12.223 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2024-04-02T14:51:12.223 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-02T14:51:12.224 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-02T14:51:12.224 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-04-02T14:51:12.993 INFO:tasks.check_counter:Daemon mds.e mds.exported=0 2024-04-02T14:51:12.994 INFO:tasks.check_counter:Daemon mds.e mds.imported=0 2024-04-02T14:51:12.994 INFO:tasks.check_counter:Daemon mds.e mds_cache.dir_update=0 2024-04-02T14:51:12.994 INFO:tasks.check_counter:Daemon mds.e mds_cache.dir_update_receipt=0 2024-04-02T14:51:12.994 INFO:tasks.check_counter:Daemon mds.e mds.root_rsnaps=0 2024-04-02T14:51:12.994 INFO:tasks.check_counter:Daemon mds.e mds_server.req_mksnap_latency.avgcount=0 2024-04-02T14:51:12.994 INFO:tasks.check_counter:Daemon mds.e mds_server.req_rmsnap_latency.avgcount=0 2024-04-02T14:51:12.994 DEBUG:tasks.check_counter:Getting stats from h 2024-04-02T14:51:12.994 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:191d963ac9de7322da8aff20b567b3fa458e2ef6 shell --fsid 6449448a-f0fc-11ee-b647-cb9ed24678a4 -- ceph daemon mds.h perf dump 2024-04-02T14:51:13.233 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/mon.b/config 2024-04-02T14:51:14.093 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:51:13 smithi092 ceph-mon[36497]: pgmap v512: 97 pgs: 40 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 35 active+clean; 4.1 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 89 KiB/s wr, 32 op/s 2024-04-02T14:51:14.178 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:51:13 smithi177 ceph-mon[36431]: pgmap v512: 97 pgs: 40 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 35 active+clean; 4.1 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 89 KiB/s wr, 32 op/s 2024-04-02T14:51:14.179 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:51:13 smithi090 ceph-mon[31674]: pgmap v512: 97 pgs: 40 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 35 active+clean; 4.1 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 89 KiB/s wr, 32 op/s 2024-04-02T14:51:15.043 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-04-02T14:51:15.043 INFO:teuthology.orchestra.run.smithi092.stdout: "AsyncMessenger::Worker-0": { 2024-04-02T14:51:15.043 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_recv_messages": 3784, 2024-04-02T14:51:15.043 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_send_messages": 1930, 2024-04-02T14:51:15.043 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_recv_bytes": 1029730, 2024-04-02T14:51:15.043 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_send_bytes": 759234, 2024-04-02T14:51:15.043 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_created_connections": 11, 2024-04-02T14:51:15.044 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_active_connections": 3, 2024-04-02T14:51:15.044 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_running_total_time": 0.190833428, 2024-04-02T14:51:15.044 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_running_send_time": 0.084269287, 2024-04-02T14:51:15.044 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_running_recv_time": 0.132412365, 2024-04-02T14:51:15.044 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_running_fast_dispatch_time": 0.005169637, 2024-04-02T14:51:15.044 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_send_messages_queue_lat": { 2024-04-02T14:51:15.044 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 1933, 2024-04-02T14:51:15.044 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.095677839, 2024-04-02T14:51:15.044 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000049497 2024-04-02T14:51:15.044 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:15.044 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_handle_ack_lat": { 2024-04-02T14:51:15.044 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 5306, 2024-04-02T14:51:15.044 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000983771, 2024-04-02T14:51:15.044 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000185 2024-04-02T14:51:15.044 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:15.045 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_recv_encrypted_bytes": 5280, 2024-04-02T14:51:15.045 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_send_encrypted_bytes": 9600 2024-04-02T14:51:15.045 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:15.045 INFO:teuthology.orchestra.run.smithi092.stdout: "AsyncMessenger::Worker-1": { 2024-04-02T14:51:15.045 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_recv_messages": 76, 2024-04-02T14:51:15.045 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_send_messages": 114, 2024-04-02T14:51:15.045 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_recv_bytes": 13270, 2024-04-02T14:51:15.045 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_send_bytes": 855966, 2024-04-02T14:51:15.045 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_created_connections": 67, 2024-04-02T14:51:15.045 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_active_connections": 6, 2024-04-02T14:51:15.045 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_running_total_time": 0.023802347, 2024-04-02T14:51:15.045 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_running_send_time": 0.003252932, 2024-04-02T14:51:15.045 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_running_recv_time": 0.018335779, 2024-04-02T14:51:15.045 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_running_fast_dispatch_time": 0.000275840, 2024-04-02T14:51:15.045 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_send_messages_queue_lat": { 2024-04-02T14:51:15.045 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 111, 2024-04-02T14:51:15.046 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.016778491, 2024-04-02T14:51:15.046 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000151157 2024-04-02T14:51:15.046 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:15.046 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_handle_ack_lat": { 2024-04-02T14:51:15.046 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 69, 2024-04-02T14:51:15.046 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000008801, 2024-04-02T14:51:15.046 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000127 2024-04-02T14:51:15.046 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:15.046 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_recv_encrypted_bytes": 2176, 2024-04-02T14:51:15.046 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_send_encrypted_bytes": 217280 2024-04-02T14:51:15.046 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:15.046 INFO:teuthology.orchestra.run.smithi092.stdout: "AsyncMessenger::Worker-2": { 2024-04-02T14:51:15.046 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_recv_messages": 473, 2024-04-02T14:51:15.046 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_send_messages": 685, 2024-04-02T14:51:15.046 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_recv_bytes": 1699668, 2024-04-02T14:51:15.046 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_send_bytes": 714804, 2024-04-02T14:51:15.047 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_created_connections": 12, 2024-04-02T14:51:15.047 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_active_connections": 5, 2024-04-02T14:51:15.047 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_running_total_time": 0.075291109, 2024-04-02T14:51:15.047 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_running_send_time": 0.026846285, 2024-04-02T14:51:15.047 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_running_recv_time": 0.052086018, 2024-04-02T14:51:15.047 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_running_fast_dispatch_time": 0.006810869, 2024-04-02T14:51:15.047 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_send_messages_queue_lat": { 2024-04-02T14:51:15.047 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 680, 2024-04-02T14:51:15.047 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.054580363, 2024-04-02T14:51:15.047 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000080265 2024-04-02T14:51:15.047 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:15.047 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_handle_ack_lat": { 2024-04-02T14:51:15.047 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 196, 2024-04-02T14:51:15.047 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000031545, 2024-04-02T14:51:15.047 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000160 2024-04-02T14:51:15.047 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:15.048 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_recv_encrypted_bytes": 1697488, 2024-04-02T14:51:15.048 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_send_encrypted_bytes": 658720 2024-04-02T14:51:15.048 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:15.048 INFO:teuthology.orchestra.run.smithi092.stdout: "cct": { 2024-04-02T14:51:15.048 INFO:teuthology.orchestra.run.smithi092.stdout: "total_workers": 1, 2024-04-02T14:51:15.048 INFO:teuthology.orchestra.run.smithi092.stdout: "unhealthy_workers": 0 2024-04-02T14:51:15.048 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:15.048 INFO:teuthology.orchestra.run.smithi092.stdout: "finisher-MDSRank": { 2024-04-02T14:51:15.048 INFO:teuthology.orchestra.run.smithi092.stdout: "queue_len": 0, 2024-04-02T14:51:15.048 INFO:teuthology.orchestra.run.smithi092.stdout: "complete_latency": { 2024-04-02T14:51:15.048 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 61, 2024-04-02T14:51:15.048 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.001226466, 2024-04-02T14:51:15.048 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000020106 2024-04-02T14:51:15.048 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-02T14:51:15.048 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:15.048 INFO:teuthology.orchestra.run.smithi092.stdout: "finisher-PurgeQueue": { 2024-04-02T14:51:15.049 INFO:teuthology.orchestra.run.smithi092.stdout: "queue_len": 0, 2024-04-02T14:51:15.049 INFO:teuthology.orchestra.run.smithi092.stdout: "complete_latency": { 2024-04-02T14:51:15.049 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 2, 2024-04-02T14:51:15.049 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000005738, 2024-04-02T14:51:15.049 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000002869 2024-04-02T14:51:15.049 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-02T14:51:15.049 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:15.049 INFO:teuthology.orchestra.run.smithi092.stdout: "mds": { 2024-04-02T14:51:15.049 INFO:teuthology.orchestra.run.smithi092.stdout: "request": 70, 2024-04-02T14:51:15.049 INFO:teuthology.orchestra.run.smithi092.stdout: "reply": 0, 2024-04-02T14:51:15.049 INFO:teuthology.orchestra.run.smithi092.stdout: "reply_latency": { 2024-04-02T14:51:15.049 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2024-04-02T14:51:15.049 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2024-04-02T14:51:15.049 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2024-04-02T14:51:15.049 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:15.049 INFO:teuthology.orchestra.run.smithi092.stdout: "slow_reply": 0, 2024-04-02T14:51:15.050 INFO:teuthology.orchestra.run.smithi092.stdout: "forward": 70, 2024-04-02T14:51:15.050 INFO:teuthology.orchestra.run.smithi092.stdout: "dir_fetch_complete": 0, 2024-04-02T14:51:15.050 INFO:teuthology.orchestra.run.smithi092.stdout: "dir_fetch_keys": 0, 2024-04-02T14:51:15.050 INFO:teuthology.orchestra.run.smithi092.stdout: "dir_commit": 11, 2024-04-02T14:51:15.050 INFO:teuthology.orchestra.run.smithi092.stdout: "dir_split": 0, 2024-04-02T14:51:15.050 INFO:teuthology.orchestra.run.smithi092.stdout: "dir_merge": 0, 2024-04-02T14:51:15.050 INFO:teuthology.orchestra.run.smithi092.stdout: "inodes": 33, 2024-04-02T14:51:15.050 INFO:teuthology.orchestra.run.smithi092.stdout: "inodes_top": 10, 2024-04-02T14:51:15.050 INFO:teuthology.orchestra.run.smithi092.stdout: "inodes_bottom": 23, 2024-04-02T14:51:15.050 INFO:teuthology.orchestra.run.smithi092.stdout: "inodes_pin_tail": 0, 2024-04-02T14:51:15.050 INFO:teuthology.orchestra.run.smithi092.stdout: "inodes_pinned": 18, 2024-04-02T14:51:15.050 INFO:teuthology.orchestra.run.smithi092.stdout: "inodes_expired": 4, 2024-04-02T14:51:15.050 INFO:teuthology.orchestra.run.smithi092.stdout: "inodes_with_caps": 0, 2024-04-02T14:51:15.050 INFO:teuthology.orchestra.run.smithi092.stdout: "caps": 0, 2024-04-02T14:51:15.050 INFO:teuthology.orchestra.run.smithi092.stdout: "subtrees": 3, 2024-04-02T14:51:15.050 INFO:teuthology.orchestra.run.smithi092.stdout: "traverse": 129, 2024-04-02T14:51:15.050 INFO:teuthology.orchestra.run.smithi092.stdout: "traverse_hit": 40, 2024-04-02T14:51:15.051 INFO:teuthology.orchestra.run.smithi092.stdout: "traverse_forward": 70, 2024-04-02T14:51:15.051 INFO:teuthology.orchestra.run.smithi092.stdout: "traverse_discover": 19, 2024-04-02T14:51:15.051 INFO:teuthology.orchestra.run.smithi092.stdout: "traverse_dir_fetch": 0, 2024-04-02T14:51:15.051 INFO:teuthology.orchestra.run.smithi092.stdout: "traverse_remote_ino": 0, 2024-04-02T14:51:15.051 INFO:teuthology.orchestra.run.smithi092.stdout: "traverse_lock": 0, 2024-04-02T14:51:15.051 INFO:teuthology.orchestra.run.smithi092.stdout: "load_cent": 0, 2024-04-02T14:51:15.051 INFO:teuthology.orchestra.run.smithi092.stdout: "q": 0, 2024-04-02T14:51:15.051 INFO:teuthology.orchestra.run.smithi092.stdout: "exported": 0, 2024-04-02T14:51:15.051 INFO:teuthology.orchestra.run.smithi092.stdout: "exported_inodes": 0, 2024-04-02T14:51:15.051 INFO:teuthology.orchestra.run.smithi092.stdout: "imported": 0, 2024-04-02T14:51:15.051 INFO:teuthology.orchestra.run.smithi092.stdout: "imported_inodes": 0, 2024-04-02T14:51:15.051 INFO:teuthology.orchestra.run.smithi092.stdout: "openino_dir_fetch": 0, 2024-04-02T14:51:15.051 INFO:teuthology.orchestra.run.smithi092.stdout: "openino_backtrace_fetch": 0, 2024-04-02T14:51:15.051 INFO:teuthology.orchestra.run.smithi092.stdout: "openino_peer_discover": 0, 2024-04-02T14:51:15.051 INFO:teuthology.orchestra.run.smithi092.stdout: "root_rfiles": 1026, 2024-04-02T14:51:15.052 INFO:teuthology.orchestra.run.smithi092.stdout: "root_rbytes": 1709677761, 2024-04-02T14:51:15.052 INFO:teuthology.orchestra.run.smithi092.stdout: "root_rsnaps": 6, 2024-04-02T14:51:15.052 INFO:teuthology.orchestra.run.smithi092.stdout: "scrub_backtrace_fetch": 0, 2024-04-02T14:51:15.052 INFO:teuthology.orchestra.run.smithi092.stdout: "scrub_set_tag": 0, 2024-04-02T14:51:15.052 INFO:teuthology.orchestra.run.smithi092.stdout: "scrub_backtrace_repaired": 0, 2024-04-02T14:51:15.052 INFO:teuthology.orchestra.run.smithi092.stdout: "scrub_inotable_repaired": 0, 2024-04-02T14:51:15.052 INFO:teuthology.orchestra.run.smithi092.stdout: "scrub_dir_inodes": 0, 2024-04-02T14:51:15.052 INFO:teuthology.orchestra.run.smithi092.stdout: "scrub_dir_base_inodes": 0, 2024-04-02T14:51:15.052 INFO:teuthology.orchestra.run.smithi092.stdout: "scrub_dirfrag_rstats": 0, 2024-04-02T14:51:15.052 INFO:teuthology.orchestra.run.smithi092.stdout: "scrub_file_inodes": 0, 2024-04-02T14:51:15.052 INFO:teuthology.orchestra.run.smithi092.stdout: "handle_inode_file_caps": 0, 2024-04-02T14:51:15.052 INFO:teuthology.orchestra.run.smithi092.stdout: "ceph_cap_op_revoke": 0, 2024-04-02T14:51:15.052 INFO:teuthology.orchestra.run.smithi092.stdout: "ceph_cap_op_grant": 0, 2024-04-02T14:51:15.052 INFO:teuthology.orchestra.run.smithi092.stdout: "ceph_cap_op_trunc": 0, 2024-04-02T14:51:15.053 INFO:teuthology.orchestra.run.smithi092.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-02T14:51:15.053 INFO:teuthology.orchestra.run.smithi092.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-02T14:51:15.053 INFO:teuthology.orchestra.run.smithi092.stdout: "handle_client_caps": 0, 2024-04-02T14:51:15.053 INFO:teuthology.orchestra.run.smithi092.stdout: "handle_client_caps_dirty": 0, 2024-04-02T14:51:15.053 INFO:teuthology.orchestra.run.smithi092.stdout: "handle_client_cap_release": 0, 2024-04-02T14:51:15.053 INFO:teuthology.orchestra.run.smithi092.stdout: "process_request_cap_release": 0 2024-04-02T14:51:15.053 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:15.053 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_cache": { 2024-04-02T14:51:15.053 INFO:teuthology.orchestra.run.smithi092.stdout: "dir_update": 0, 2024-04-02T14:51:15.053 INFO:teuthology.orchestra.run.smithi092.stdout: "dir_update_receipt": 99, 2024-04-02T14:51:15.053 INFO:teuthology.orchestra.run.smithi092.stdout: "dir_try_discover": 59, 2024-04-02T14:51:15.053 INFO:teuthology.orchestra.run.smithi092.stdout: "dir_send_discover": 60, 2024-04-02T14:51:15.053 INFO:teuthology.orchestra.run.smithi092.stdout: "dir_handle_discover": 0, 2024-04-02T14:51:15.053 INFO:teuthology.orchestra.run.smithi092.stdout: "num_strays": 0, 2024-04-02T14:51:15.053 INFO:teuthology.orchestra.run.smithi092.stdout: "num_strays_delayed": 0, 2024-04-02T14:51:15.053 INFO:teuthology.orchestra.run.smithi092.stdout: "num_strays_enqueuing": 0, 2024-04-02T14:51:15.054 INFO:teuthology.orchestra.run.smithi092.stdout: "strays_created": 0, 2024-04-02T14:51:15.054 INFO:teuthology.orchestra.run.smithi092.stdout: "strays_enqueued": 0, 2024-04-02T14:51:15.054 INFO:teuthology.orchestra.run.smithi092.stdout: "strays_reintegrated": 0, 2024-04-02T14:51:15.054 INFO:teuthology.orchestra.run.smithi092.stdout: "strays_migrated": 0, 2024-04-02T14:51:15.054 INFO:teuthology.orchestra.run.smithi092.stdout: "num_recovering_processing": 0, 2024-04-02T14:51:15.054 INFO:teuthology.orchestra.run.smithi092.stdout: "num_recovering_enqueued": 0, 2024-04-02T14:51:15.054 INFO:teuthology.orchestra.run.smithi092.stdout: "num_recovering_prioritized": 0, 2024-04-02T14:51:15.054 INFO:teuthology.orchestra.run.smithi092.stdout: "recovery_started": 0, 2024-04-02T14:51:15.054 INFO:teuthology.orchestra.run.smithi092.stdout: "recovery_completed": 0, 2024-04-02T14:51:15.054 INFO:teuthology.orchestra.run.smithi092.stdout: "ireq_quiesce_path": 0, 2024-04-02T14:51:15.054 INFO:teuthology.orchestra.run.smithi092.stdout: "ireq_quiesce_inode": 0, 2024-04-02T14:51:15.054 INFO:teuthology.orchestra.run.smithi092.stdout: "ireq_enqueue_scrub": 0, 2024-04-02T14:51:15.054 INFO:teuthology.orchestra.run.smithi092.stdout: "ireq_exportdir": 0, 2024-04-02T14:51:15.054 INFO:teuthology.orchestra.run.smithi092.stdout: "ireq_flush": 0, 2024-04-02T14:51:15.055 INFO:teuthology.orchestra.run.smithi092.stdout: "ireq_fragmentdir": 0, 2024-04-02T14:51:15.055 INFO:teuthology.orchestra.run.smithi092.stdout: "ireq_fragstats": 0, 2024-04-02T14:51:15.055 INFO:teuthology.orchestra.run.smithi092.stdout: "ireq_inodestats": 0 2024-04-02T14:51:15.055 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:15.055 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_log": { 2024-04-02T14:51:15.055 INFO:teuthology.orchestra.run.smithi092.stdout: "evlrg": 0, 2024-04-02T14:51:15.055 INFO:teuthology.orchestra.run.smithi092.stdout: "evadd": 8, 2024-04-02T14:51:15.055 INFO:teuthology.orchestra.run.smithi092.stdout: "evex": 0, 2024-04-02T14:51:15.055 INFO:teuthology.orchestra.run.smithi092.stdout: "evtrm": 0, 2024-04-02T14:51:15.055 INFO:teuthology.orchestra.run.smithi092.stdout: "ev": 8, 2024-04-02T14:51:15.055 INFO:teuthology.orchestra.run.smithi092.stdout: "evexg": 0, 2024-04-02T14:51:15.055 INFO:teuthology.orchestra.run.smithi092.stdout: "evexd": 0, 2024-04-02T14:51:15.055 INFO:teuthology.orchestra.run.smithi092.stdout: "segadd": 2, 2024-04-02T14:51:15.055 INFO:teuthology.orchestra.run.smithi092.stdout: "segex": 0, 2024-04-02T14:51:15.055 INFO:teuthology.orchestra.run.smithi092.stdout: "segtrm": 0, 2024-04-02T14:51:15.056 INFO:teuthology.orchestra.run.smithi092.stdout: "seg": 2, 2024-04-02T14:51:15.056 INFO:teuthology.orchestra.run.smithi092.stdout: "segmjr": 2, 2024-04-02T14:51:15.056 INFO:teuthology.orchestra.run.smithi092.stdout: "segexg": 0, 2024-04-02T14:51:15.056 INFO:teuthology.orchestra.run.smithi092.stdout: "segexd": 0, 2024-04-02T14:51:15.056 INFO:teuthology.orchestra.run.smithi092.stdout: "expos": 4194304, 2024-04-02T14:51:15.056 INFO:teuthology.orchestra.run.smithi092.stdout: "wrpos": 4196653, 2024-04-02T14:51:15.056 INFO:teuthology.orchestra.run.smithi092.stdout: "rdpos": 0, 2024-04-02T14:51:15.056 INFO:teuthology.orchestra.run.smithi092.stdout: "jlat": { 2024-04-02T14:51:15.056 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 7, 2024-04-02T14:51:15.056 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.029986466, 2024-04-02T14:51:15.056 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.004283780 2024-04-02T14:51:15.056 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:15.056 INFO:teuthology.orchestra.run.smithi092.stdout: "replayed": 0 2024-04-02T14:51:15.056 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:15.056 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_mem": { 2024-04-02T14:51:15.056 INFO:teuthology.orchestra.run.smithi092.stdout: "ino": 37, 2024-04-02T14:51:15.057 INFO:teuthology.orchestra.run.smithi092.stdout: "ino+": 39, 2024-04-02T14:51:15.057 INFO:teuthology.orchestra.run.smithi092.stdout: "ino-": 2, 2024-04-02T14:51:15.057 INFO:teuthology.orchestra.run.smithi092.stdout: "dir": 64, 2024-04-02T14:51:15.057 INFO:teuthology.orchestra.run.smithi092.stdout: "dir+": 67, 2024-04-02T14:51:15.057 INFO:teuthology.orchestra.run.smithi092.stdout: "dir-": 3, 2024-04-02T14:51:15.057 INFO:teuthology.orchestra.run.smithi092.stdout: "dn": 33, 2024-04-02T14:51:15.057 INFO:teuthology.orchestra.run.smithi092.stdout: "dn+": 37, 2024-04-02T14:51:15.057 INFO:teuthology.orchestra.run.smithi092.stdout: "dn-": 4, 2024-04-02T14:51:15.057 INFO:teuthology.orchestra.run.smithi092.stdout: "cap": 0, 2024-04-02T14:51:15.057 INFO:teuthology.orchestra.run.smithi092.stdout: "cap+": 0, 2024-04-02T14:51:15.057 INFO:teuthology.orchestra.run.smithi092.stdout: "cap-": 0, 2024-04-02T14:51:15.057 INFO:teuthology.orchestra.run.smithi092.stdout: "rss": 54040, 2024-04-02T14:51:15.057 INFO:teuthology.orchestra.run.smithi092.stdout: "heap": 223608 2024-04-02T14:51:15.057 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:15.057 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_server": { 2024-04-02T14:51:15.058 INFO:teuthology.orchestra.run.smithi092.stdout: "dispatch_client_request": 70, 2024-04-02T14:51:15.058 INFO:teuthology.orchestra.run.smithi092.stdout: "dispatch_server_request": 0, 2024-04-02T14:51:15.058 INFO:teuthology.orchestra.run.smithi092.stdout: "handle_client_request": 70, 2024-04-02T14:51:15.058 INFO:teuthology.orchestra.run.smithi092.stdout: "handle_client_session": 33, 2024-04-02T14:51:15.058 INFO:teuthology.orchestra.run.smithi092.stdout: "handle_peer_request": 0, 2024-04-02T14:51:15.058 INFO:teuthology.orchestra.run.smithi092.stdout: "req_create_latency": { 2024-04-02T14:51:15.058 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2024-04-02T14:51:15.058 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2024-04-02T14:51:15.058 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2024-04-02T14:51:15.058 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:15.058 INFO:teuthology.orchestra.run.smithi092.stdout: "req_getattr_latency": { 2024-04-02T14:51:15.058 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2024-04-02T14:51:15.058 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2024-04-02T14:51:15.058 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2024-04-02T14:51:15.058 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:15.059 INFO:teuthology.orchestra.run.smithi092.stdout: "req_getfilelock_latency": { 2024-04-02T14:51:15.059 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2024-04-02T14:51:15.059 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2024-04-02T14:51:15.059 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2024-04-02T14:51:15.059 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:15.059 INFO:teuthology.orchestra.run.smithi092.stdout: "req_link_latency": { 2024-04-02T14:51:15.059 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2024-04-02T14:51:15.059 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2024-04-02T14:51:15.059 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2024-04-02T14:51:15.059 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:15.059 INFO:teuthology.orchestra.run.smithi092.stdout: "req_lookup_latency": { 2024-04-02T14:51:15.059 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2024-04-02T14:51:15.059 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2024-04-02T14:51:15.059 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2024-04-02T14:51:15.059 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:15.060 INFO:teuthology.orchestra.run.smithi092.stdout: "req_lookuphash_latency": { 2024-04-02T14:51:15.060 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2024-04-02T14:51:15.060 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2024-04-02T14:51:15.060 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2024-04-02T14:51:15.060 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:15.060 INFO:teuthology.orchestra.run.smithi092.stdout: "req_lookupino_latency": { 2024-04-02T14:51:15.060 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2024-04-02T14:51:15.060 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2024-04-02T14:51:15.060 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2024-04-02T14:51:15.060 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:15.060 INFO:teuthology.orchestra.run.smithi092.stdout: "req_lookupname_latency": { 2024-04-02T14:51:15.060 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2024-04-02T14:51:15.060 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2024-04-02T14:51:15.060 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2024-04-02T14:51:15.060 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:15.061 INFO:teuthology.orchestra.run.smithi092.stdout: "req_lookupparent_latency": { 2024-04-02T14:51:15.061 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2024-04-02T14:51:15.061 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2024-04-02T14:51:15.061 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2024-04-02T14:51:15.061 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:15.061 INFO:teuthology.orchestra.run.smithi092.stdout: "req_lookupsnap_latency": { 2024-04-02T14:51:15.061 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2024-04-02T14:51:15.061 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2024-04-02T14:51:15.061 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2024-04-02T14:51:15.061 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:15.061 INFO:teuthology.orchestra.run.smithi092.stdout: "req_lssnap_latency": { 2024-04-02T14:51:15.061 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2024-04-02T14:51:15.061 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2024-04-02T14:51:15.061 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2024-04-02T14:51:15.061 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:15.062 INFO:teuthology.orchestra.run.smithi092.stdout: "req_mkdir_latency": { 2024-04-02T14:51:15.062 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2024-04-02T14:51:15.062 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2024-04-02T14:51:15.062 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2024-04-02T14:51:15.062 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:15.062 INFO:teuthology.orchestra.run.smithi092.stdout: "req_mknod_latency": { 2024-04-02T14:51:15.063 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2024-04-02T14:51:15.063 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2024-04-02T14:51:15.063 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2024-04-02T14:51:15.063 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:15.063 INFO:teuthology.orchestra.run.smithi092.stdout: "req_mksnap_latency": { 2024-04-02T14:51:15.063 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2024-04-02T14:51:15.063 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2024-04-02T14:51:15.063 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2024-04-02T14:51:15.063 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:15.063 INFO:teuthology.orchestra.run.smithi092.stdout: "req_open_latency": { 2024-04-02T14:51:15.063 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2024-04-02T14:51:15.063 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2024-04-02T14:51:15.063 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2024-04-02T14:51:15.063 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:15.064 INFO:teuthology.orchestra.run.smithi092.stdout: "req_readdir_latency": { 2024-04-02T14:51:15.064 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2024-04-02T14:51:15.064 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2024-04-02T14:51:15.064 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2024-04-02T14:51:15.064 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:15.064 INFO:teuthology.orchestra.run.smithi092.stdout: "req_rename_latency": { 2024-04-02T14:51:15.064 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2024-04-02T14:51:15.064 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2024-04-02T14:51:15.064 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2024-04-02T14:51:15.064 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:15.064 INFO:teuthology.orchestra.run.smithi092.stdout: "req_renamesnap_latency": { 2024-04-02T14:51:15.064 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2024-04-02T14:51:15.064 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2024-04-02T14:51:15.064 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2024-04-02T14:51:15.064 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:15.064 INFO:teuthology.orchestra.run.smithi092.stdout: "req_snapdiff_latency": { 2024-04-02T14:51:15.065 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2024-04-02T14:51:15.065 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2024-04-02T14:51:15.065 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2024-04-02T14:51:15.065 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:15.065 INFO:teuthology.orchestra.run.smithi092.stdout: "req_rmdir_latency": { 2024-04-02T14:51:15.065 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2024-04-02T14:51:15.065 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2024-04-02T14:51:15.065 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2024-04-02T14:51:15.065 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:15.065 INFO:teuthology.orchestra.run.smithi092.stdout: "req_rmsnap_latency": { 2024-04-02T14:51:15.065 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2024-04-02T14:51:15.065 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2024-04-02T14:51:15.065 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2024-04-02T14:51:15.065 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:15.065 INFO:teuthology.orchestra.run.smithi092.stdout: "req_rmxattr_latency": { 2024-04-02T14:51:15.066 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2024-04-02T14:51:15.066 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2024-04-02T14:51:15.066 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2024-04-02T14:51:15.066 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:15.066 INFO:teuthology.orchestra.run.smithi092.stdout: "req_setattr_latency": { 2024-04-02T14:51:15.066 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2024-04-02T14:51:15.066 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2024-04-02T14:51:15.066 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2024-04-02T14:51:15.066 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:15.066 INFO:teuthology.orchestra.run.smithi092.stdout: "req_setdirlayout_latency": { 2024-04-02T14:51:15.066 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2024-04-02T14:51:15.066 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2024-04-02T14:51:15.066 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2024-04-02T14:51:15.066 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:15.066 INFO:teuthology.orchestra.run.smithi092.stdout: "req_setfilelock_latency": { 2024-04-02T14:51:15.067 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2024-04-02T14:51:15.067 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2024-04-02T14:51:15.067 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2024-04-02T14:51:15.067 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:15.067 INFO:teuthology.orchestra.run.smithi092.stdout: "req_setlayout_latency": { 2024-04-02T14:51:15.067 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2024-04-02T14:51:15.067 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2024-04-02T14:51:15.067 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2024-04-02T14:51:15.067 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:15.067 INFO:teuthology.orchestra.run.smithi092.stdout: "req_setxattr_latency": { 2024-04-02T14:51:15.067 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2024-04-02T14:51:15.067 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2024-04-02T14:51:15.067 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2024-04-02T14:51:15.067 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:15.067 INFO:teuthology.orchestra.run.smithi092.stdout: "req_symlink_latency": { 2024-04-02T14:51:15.068 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2024-04-02T14:51:15.068 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2024-04-02T14:51:15.068 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2024-04-02T14:51:15.068 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:15.068 INFO:teuthology.orchestra.run.smithi092.stdout: "req_unlink_latency": { 2024-04-02T14:51:15.068 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2024-04-02T14:51:15.068 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2024-04-02T14:51:15.068 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2024-04-02T14:51:15.068 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:15.068 INFO:teuthology.orchestra.run.smithi092.stdout: "cap_revoke_eviction": 0, 2024-04-02T14:51:15.068 INFO:teuthology.orchestra.run.smithi092.stdout: "cap_acquisition_throttle": 0, 2024-04-02T14:51:15.068 INFO:teuthology.orchestra.run.smithi092.stdout: "req_getvxattr_latency": { 2024-04-02T14:51:15.068 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2024-04-02T14:51:15.068 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2024-04-02T14:51:15.068 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2024-04-02T14:51:15.068 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-02T14:51:15.069 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:15.069 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_sessions": { 2024-04-02T14:51:15.069 INFO:teuthology.orchestra.run.smithi092.stdout: "session_count": 2, 2024-04-02T14:51:15.069 INFO:teuthology.orchestra.run.smithi092.stdout: "session_add": 4, 2024-04-02T14:51:15.069 INFO:teuthology.orchestra.run.smithi092.stdout: "session_remove": 2, 2024-04-02T14:51:15.069 INFO:teuthology.orchestra.run.smithi092.stdout: "sessions_open": 2, 2024-04-02T14:51:15.069 INFO:teuthology.orchestra.run.smithi092.stdout: "sessions_stale": 0, 2024-04-02T14:51:15.069 INFO:teuthology.orchestra.run.smithi092.stdout: "total_load": 0, 2024-04-02T14:51:15.069 INFO:teuthology.orchestra.run.smithi092.stdout: "average_load": 0, 2024-04-02T14:51:15.069 INFO:teuthology.orchestra.run.smithi092.stdout: "avg_session_uptime": 933, 2024-04-02T14:51:15.069 INFO:teuthology.orchestra.run.smithi092.stdout: "mdthresh_evicted": 0 2024-04-02T14:51:15.069 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:15.069 INFO:teuthology.orchestra.run.smithi092.stdout: "mempool": { 2024-04-02T14:51:15.069 INFO:teuthology.orchestra.run.smithi092.stdout: "bloom_filter_bytes": 0, 2024-04-02T14:51:15.069 INFO:teuthology.orchestra.run.smithi092.stdout: "bloom_filter_items": 0, 2024-04-02T14:51:15.070 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_alloc_bytes": 0, 2024-04-02T14:51:15.070 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_alloc_items": 0, 2024-04-02T14:51:15.070 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_cache_data_bytes": 0, 2024-04-02T14:51:15.070 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_cache_data_items": 0, 2024-04-02T14:51:15.070 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-02T14:51:15.070 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_cache_onode_items": 0, 2024-04-02T14:51:15.070 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-02T14:51:15.070 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_cache_meta_items": 0, 2024-04-02T14:51:15.070 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_cache_other_bytes": 0, 2024-04-02T14:51:15.070 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_cache_other_items": 0, 2024-04-02T14:51:15.070 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-02T14:51:15.070 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_cache_buffer_items": 0, 2024-04-02T14:51:15.070 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_extent_bytes": 0, 2024-04-02T14:51:15.070 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_extent_items": 0, 2024-04-02T14:51:15.070 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_blob_bytes": 0, 2024-04-02T14:51:15.071 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_blob_items": 0, 2024-04-02T14:51:15.071 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-02T14:51:15.071 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_shared_blob_items": 0, 2024-04-02T14:51:15.071 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-02T14:51:15.071 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_inline_bl_items": 0, 2024-04-02T14:51:15.071 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_fsck_bytes": 0, 2024-04-02T14:51:15.071 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_fsck_items": 0, 2024-04-02T14:51:15.071 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_txc_bytes": 0, 2024-04-02T14:51:15.071 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_txc_items": 0, 2024-04-02T14:51:15.071 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-02T14:51:15.071 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_writing_deferred_items": 0, 2024-04-02T14:51:15.071 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_writing_bytes": 0, 2024-04-02T14:51:15.071 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_writing_items": 0, 2024-04-02T14:51:15.072 INFO:teuthology.orchestra.run.smithi092.stdout: "bluefs_bytes": 0, 2024-04-02T14:51:15.072 INFO:teuthology.orchestra.run.smithi092.stdout: "bluefs_items": 0, 2024-04-02T14:51:15.072 INFO:teuthology.orchestra.run.smithi092.stdout: "bluefs_file_reader_bytes": 0, 2024-04-02T14:51:15.072 INFO:teuthology.orchestra.run.smithi092.stdout: "bluefs_file_reader_items": 0, 2024-04-02T14:51:15.072 INFO:teuthology.orchestra.run.smithi092.stdout: "bluefs_file_writer_bytes": 0, 2024-04-02T14:51:15.072 INFO:teuthology.orchestra.run.smithi092.stdout: "bluefs_file_writer_items": 0, 2024-04-02T14:51:15.072 INFO:teuthology.orchestra.run.smithi092.stdout: "buffer_anon_bytes": 236043, 2024-04-02T14:51:15.072 INFO:teuthology.orchestra.run.smithi092.stdout: "buffer_anon_items": 117, 2024-04-02T14:51:15.072 INFO:teuthology.orchestra.run.smithi092.stdout: "buffer_meta_bytes": 0, 2024-04-02T14:51:15.072 INFO:teuthology.orchestra.run.smithi092.stdout: "buffer_meta_items": 0, 2024-04-02T14:51:15.072 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_bytes": 0, 2024-04-02T14:51:15.072 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_items": 0, 2024-04-02T14:51:15.072 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_mapbl_bytes": 0, 2024-04-02T14:51:15.072 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_mapbl_items": 0, 2024-04-02T14:51:15.073 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_pglog_bytes": 0, 2024-04-02T14:51:15.073 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_pglog_items": 0, 2024-04-02T14:51:15.073 INFO:teuthology.orchestra.run.smithi092.stdout: "osdmap_bytes": 8256, 2024-04-02T14:51:15.073 INFO:teuthology.orchestra.run.smithi092.stdout: "osdmap_items": 173, 2024-04-02T14:51:15.073 INFO:teuthology.orchestra.run.smithi092.stdout: "osdmap_mapping_bytes": 0, 2024-04-02T14:51:15.073 INFO:teuthology.orchestra.run.smithi092.stdout: "osdmap_mapping_items": 0, 2024-04-02T14:51:15.073 INFO:teuthology.orchestra.run.smithi092.stdout: "pgmap_bytes": 0, 2024-04-02T14:51:15.073 INFO:teuthology.orchestra.run.smithi092.stdout: "pgmap_items": 0, 2024-04-02T14:51:15.073 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_co_bytes": 196217, 2024-04-02T14:51:15.073 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_co_items": 978, 2024-04-02T14:51:15.073 INFO:teuthology.orchestra.run.smithi092.stdout: "unittest_1_bytes": 0, 2024-04-02T14:51:15.073 INFO:teuthology.orchestra.run.smithi092.stdout: "unittest_1_items": 0, 2024-04-02T14:51:15.073 INFO:teuthology.orchestra.run.smithi092.stdout: "unittest_2_bytes": 0, 2024-04-02T14:51:15.073 INFO:teuthology.orchestra.run.smithi092.stdout: "unittest_2_items": 0 2024-04-02T14:51:15.073 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:15.073 INFO:teuthology.orchestra.run.smithi092.stdout: "objecter": { 2024-04-02T14:51:15.074 INFO:teuthology.orchestra.run.smithi092.stdout: "op_active": 0, 2024-04-02T14:51:15.074 INFO:teuthology.orchestra.run.smithi092.stdout: "op_laggy": 0, 2024-04-02T14:51:15.074 INFO:teuthology.orchestra.run.smithi092.stdout: "op_send": 45, 2024-04-02T14:51:15.074 INFO:teuthology.orchestra.run.smithi092.stdout: "op_send_bytes": 12831, 2024-04-02T14:51:15.074 INFO:teuthology.orchestra.run.smithi092.stdout: "op_resend": 33, 2024-04-02T14:51:15.074 INFO:teuthology.orchestra.run.smithi092.stdout: "op_reply": 45, 2024-04-02T14:51:15.074 INFO:teuthology.orchestra.run.smithi092.stdout: "op_latency": { 2024-04-02T14:51:15.074 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 45, 2024-04-02T14:51:15.074 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.310990927, 2024-04-02T14:51:15.074 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.006910909 2024-04-02T14:51:15.074 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:15.074 INFO:teuthology.orchestra.run.smithi092.stdout: "op_inflight": 0, 2024-04-02T14:51:15.074 INFO:teuthology.orchestra.run.smithi092.stdout: "oplen_avg": { 2024-04-02T14:51:15.074 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 45, 2024-04-02T14:51:15.074 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 67 2024-04-02T14:51:15.075 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:15.075 INFO:teuthology.orchestra.run.smithi092.stdout: "op": 45, 2024-04-02T14:51:15.075 INFO:teuthology.orchestra.run.smithi092.stdout: "op_r": 0, 2024-04-02T14:51:15.075 INFO:teuthology.orchestra.run.smithi092.stdout: "op_w": 45, 2024-04-02T14:51:15.075 INFO:teuthology.orchestra.run.smithi092.stdout: "op_rmw": 0, 2024-04-02T14:51:15.075 INFO:teuthology.orchestra.run.smithi092.stdout: "op_pg": 0, 2024-04-02T14:51:15.075 INFO:teuthology.orchestra.run.smithi092.stdout: "osdop_stat": 0, 2024-04-02T14:51:15.075 INFO:teuthology.orchestra.run.smithi092.stdout: "osdop_create": 10, 2024-04-02T14:51:15.075 INFO:teuthology.orchestra.run.smithi092.stdout: "osdop_read": 0, 2024-04-02T14:51:15.075 INFO:teuthology.orchestra.run.smithi092.stdout: "osdop_write": 7, 2024-04-02T14:51:15.075 INFO:teuthology.orchestra.run.smithi092.stdout: "osdop_writefull": 9, 2024-04-02T14:51:15.075 INFO:teuthology.orchestra.run.smithi092.stdout: "osdop_writesame": 0, 2024-04-02T14:51:15.075 INFO:teuthology.orchestra.run.smithi092.stdout: "osdop_append": 0, 2024-04-02T14:51:15.075 INFO:teuthology.orchestra.run.smithi092.stdout: "osdop_zero": 0, 2024-04-02T14:51:15.075 INFO:teuthology.orchestra.run.smithi092.stdout: "osdop_truncate": 0, 2024-04-02T14:51:15.076 INFO:teuthology.orchestra.run.smithi092.stdout: "osdop_delete": 6, 2024-04-02T14:51:15.076 INFO:teuthology.orchestra.run.smithi092.stdout: "osdop_mapext": 0, 2024-04-02T14:51:15.076 INFO:teuthology.orchestra.run.smithi092.stdout: "osdop_sparse_read": 0, 2024-04-02T14:51:15.076 INFO:teuthology.orchestra.run.smithi092.stdout: "osdop_clonerange": 0, 2024-04-02T14:51:15.076 INFO:teuthology.orchestra.run.smithi092.stdout: "osdop_getxattr": 0, 2024-04-02T14:51:15.076 INFO:teuthology.orchestra.run.smithi092.stdout: "osdop_setxattr": 20, 2024-04-02T14:51:15.076 INFO:teuthology.orchestra.run.smithi092.stdout: "osdop_cmpxattr": 0, 2024-04-02T14:51:15.076 INFO:teuthology.orchestra.run.smithi092.stdout: "osdop_rmxattr": 0, 2024-04-02T14:51:15.076 INFO:teuthology.orchestra.run.smithi092.stdout: "osdop_resetxattrs": 0, 2024-04-02T14:51:15.076 INFO:teuthology.orchestra.run.smithi092.stdout: "osdop_call": 0, 2024-04-02T14:51:15.076 INFO:teuthology.orchestra.run.smithi092.stdout: "osdop_watch": 0, 2024-04-02T14:51:15.076 INFO:teuthology.orchestra.run.smithi092.stdout: "osdop_notify": 0, 2024-04-02T14:51:15.076 INFO:teuthology.orchestra.run.smithi092.stdout: "osdop_src_cmpxattr": 0, 2024-04-02T14:51:15.076 INFO:teuthology.orchestra.run.smithi092.stdout: "osdop_pgls": 0, 2024-04-02T14:51:15.076 INFO:teuthology.orchestra.run.smithi092.stdout: "osdop_pgls_filter": 0, 2024-04-02T14:51:15.077 INFO:teuthology.orchestra.run.smithi092.stdout: "osdop_other": 0, 2024-04-02T14:51:15.077 INFO:teuthology.orchestra.run.smithi092.stdout: "linger_active": 0, 2024-04-02T14:51:15.077 INFO:teuthology.orchestra.run.smithi092.stdout: "linger_send": 0, 2024-04-02T14:51:15.077 INFO:teuthology.orchestra.run.smithi092.stdout: "linger_resend": 0, 2024-04-02T14:51:15.077 INFO:teuthology.orchestra.run.smithi092.stdout: "linger_ping": 0, 2024-04-02T14:51:15.077 INFO:teuthology.orchestra.run.smithi092.stdout: "poolop_active": 0, 2024-04-02T14:51:15.077 INFO:teuthology.orchestra.run.smithi092.stdout: "poolop_send": 0, 2024-04-02T14:51:15.077 INFO:teuthology.orchestra.run.smithi092.stdout: "poolop_resend": 0, 2024-04-02T14:51:15.077 INFO:teuthology.orchestra.run.smithi092.stdout: "poolstat_active": 0, 2024-04-02T14:51:15.077 INFO:teuthology.orchestra.run.smithi092.stdout: "poolstat_send": 0, 2024-04-02T14:51:15.077 INFO:teuthology.orchestra.run.smithi092.stdout: "poolstat_resend": 0, 2024-04-02T14:51:15.077 INFO:teuthology.orchestra.run.smithi092.stdout: "statfs_active": 0, 2024-04-02T14:51:15.077 INFO:teuthology.orchestra.run.smithi092.stdout: "statfs_send": 0, 2024-04-02T14:51:15.077 INFO:teuthology.orchestra.run.smithi092.stdout: "statfs_resend": 0, 2024-04-02T14:51:15.077 INFO:teuthology.orchestra.run.smithi092.stdout: "command_active": 0, 2024-04-02T14:51:15.078 INFO:teuthology.orchestra.run.smithi092.stdout: "command_send": 0, 2024-04-02T14:51:15.078 INFO:teuthology.orchestra.run.smithi092.stdout: "command_resend": 0, 2024-04-02T14:51:15.078 INFO:teuthology.orchestra.run.smithi092.stdout: "map_epoch": 110, 2024-04-02T14:51:15.078 INFO:teuthology.orchestra.run.smithi092.stdout: "map_full": 0, 2024-04-02T14:51:15.078 INFO:teuthology.orchestra.run.smithi092.stdout: "map_inc": 22, 2024-04-02T14:51:15.078 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_sessions": 8, 2024-04-02T14:51:15.078 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_session_open": 12, 2024-04-02T14:51:15.078 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_session_close": 4, 2024-04-02T14:51:15.078 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_laggy": 0, 2024-04-02T14:51:15.078 INFO:teuthology.orchestra.run.smithi092.stdout: "omap_wr": 14, 2024-04-02T14:51:15.078 INFO:teuthology.orchestra.run.smithi092.stdout: "omap_rd": 0, 2024-04-02T14:51:15.078 INFO:teuthology.orchestra.run.smithi092.stdout: "omap_del": 1 2024-04-02T14:51:15.078 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:15.078 INFO:teuthology.orchestra.run.smithi092.stdout: "oft": { 2024-04-02T14:51:15.078 INFO:teuthology.orchestra.run.smithi092.stdout: "omap_total_objs": 1, 2024-04-02T14:51:15.079 INFO:teuthology.orchestra.run.smithi092.stdout: "omap_total_kv_pairs": 0, 2024-04-02T14:51:15.079 INFO:teuthology.orchestra.run.smithi092.stdout: "omap_total_updates": 0, 2024-04-02T14:51:15.079 INFO:teuthology.orchestra.run.smithi092.stdout: "omap_total_removes": 0 2024-04-02T14:51:15.079 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:15.079 INFO:teuthology.orchestra.run.smithi092.stdout: "osd-slow-ops": { 2024-04-02T14:51:15.079 INFO:teuthology.orchestra.run.smithi092.stdout: "slow_ops_count": 0 2024-04-02T14:51:15.079 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:15.079 INFO:teuthology.orchestra.run.smithi092.stdout: "purge_queue": { 2024-04-02T14:51:15.079 INFO:teuthology.orchestra.run.smithi092.stdout: "pq_executing_ops": 0, 2024-04-02T14:51:15.079 INFO:teuthology.orchestra.run.smithi092.stdout: "pq_executing_ops_high_water": 0, 2024-04-02T14:51:15.079 INFO:teuthology.orchestra.run.smithi092.stdout: "pq_executing": 0, 2024-04-02T14:51:15.079 INFO:teuthology.orchestra.run.smithi092.stdout: "pq_executing_high_water": 0, 2024-04-02T14:51:15.079 INFO:teuthology.orchestra.run.smithi092.stdout: "pq_executed_ops": 0, 2024-04-02T14:51:15.079 INFO:teuthology.orchestra.run.smithi092.stdout: "pq_executed": 0, 2024-04-02T14:51:15.079 INFO:teuthology.orchestra.run.smithi092.stdout: "pq_item_in_journal": 0 2024-04-02T14:51:15.079 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:15.080 INFO:teuthology.orchestra.run.smithi092.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-02T14:51:15.080 INFO:teuthology.orchestra.run.smithi092.stdout: "val": 0, 2024-04-02T14:51:15.080 INFO:teuthology.orchestra.run.smithi092.stdout: "max": 104857600, 2024-04-02T14:51:15.080 INFO:teuthology.orchestra.run.smithi092.stdout: "get_started": 0, 2024-04-02T14:51:15.080 INFO:teuthology.orchestra.run.smithi092.stdout: "get": 4329, 2024-04-02T14:51:15.080 INFO:teuthology.orchestra.run.smithi092.stdout: "get_sum": 2326035, 2024-04-02T14:51:15.080 INFO:teuthology.orchestra.run.smithi092.stdout: "get_or_fail_fail": 0, 2024-04-02T14:51:15.080 INFO:teuthology.orchestra.run.smithi092.stdout: "get_or_fail_success": 4329, 2024-04-02T14:51:15.080 INFO:teuthology.orchestra.run.smithi092.stdout: "take": 0, 2024-04-02T14:51:15.080 INFO:teuthology.orchestra.run.smithi092.stdout: "take_sum": 0, 2024-04-02T14:51:15.080 INFO:teuthology.orchestra.run.smithi092.stdout: "put": 4329, 2024-04-02T14:51:15.080 INFO:teuthology.orchestra.run.smithi092.stdout: "put_sum": 2326035, 2024-04-02T14:51:15.080 INFO:teuthology.orchestra.run.smithi092.stdout: "wait": { 2024-04-02T14:51:15.080 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2024-04-02T14:51:15.080 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2024-04-02T14:51:15.081 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2024-04-02T14:51:15.081 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-02T14:51:15.081 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:15.081 INFO:teuthology.orchestra.run.smithi092.stdout: "throttle-objecter_bytes": { 2024-04-02T14:51:15.081 INFO:teuthology.orchestra.run.smithi092.stdout: "val": 0, 2024-04-02T14:51:15.081 INFO:teuthology.orchestra.run.smithi092.stdout: "max": 104857600, 2024-04-02T14:51:15.081 INFO:teuthology.orchestra.run.smithi092.stdout: "get_started": 0, 2024-04-02T14:51:15.081 INFO:teuthology.orchestra.run.smithi092.stdout: "get": 0, 2024-04-02T14:51:15.081 INFO:teuthology.orchestra.run.smithi092.stdout: "get_sum": 0, 2024-04-02T14:51:15.081 INFO:teuthology.orchestra.run.smithi092.stdout: "get_or_fail_fail": 0, 2024-04-02T14:51:15.081 INFO:teuthology.orchestra.run.smithi092.stdout: "get_or_fail_success": 0, 2024-04-02T14:51:15.081 INFO:teuthology.orchestra.run.smithi092.stdout: "take": 45, 2024-04-02T14:51:15.081 INFO:teuthology.orchestra.run.smithi092.stdout: "take_sum": 12831, 2024-04-02T14:51:15.081 INFO:teuthology.orchestra.run.smithi092.stdout: "put": 45, 2024-04-02T14:51:15.081 INFO:teuthology.orchestra.run.smithi092.stdout: "put_sum": 12831, 2024-04-02T14:51:15.081 INFO:teuthology.orchestra.run.smithi092.stdout: "wait": { 2024-04-02T14:51:15.082 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2024-04-02T14:51:15.082 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2024-04-02T14:51:15.082 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2024-04-02T14:51:15.082 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-02T14:51:15.082 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:15.082 INFO:teuthology.orchestra.run.smithi092.stdout: "throttle-objecter_ops": { 2024-04-02T14:51:15.082 INFO:teuthology.orchestra.run.smithi092.stdout: "val": 0, 2024-04-02T14:51:15.082 INFO:teuthology.orchestra.run.smithi092.stdout: "max": 1024, 2024-04-02T14:51:15.082 INFO:teuthology.orchestra.run.smithi092.stdout: "get_started": 0, 2024-04-02T14:51:15.082 INFO:teuthology.orchestra.run.smithi092.stdout: "get": 0, 2024-04-02T14:51:15.082 INFO:teuthology.orchestra.run.smithi092.stdout: "get_sum": 0, 2024-04-02T14:51:15.082 INFO:teuthology.orchestra.run.smithi092.stdout: "get_or_fail_fail": 0, 2024-04-02T14:51:15.082 INFO:teuthology.orchestra.run.smithi092.stdout: "get_or_fail_success": 0, 2024-04-02T14:51:15.082 INFO:teuthology.orchestra.run.smithi092.stdout: "take": 45, 2024-04-02T14:51:15.083 INFO:teuthology.orchestra.run.smithi092.stdout: "take_sum": 45, 2024-04-02T14:51:15.083 INFO:teuthology.orchestra.run.smithi092.stdout: "put": 45, 2024-04-02T14:51:15.083 INFO:teuthology.orchestra.run.smithi092.stdout: "put_sum": 45, 2024-04-02T14:51:15.083 INFO:teuthology.orchestra.run.smithi092.stdout: "wait": { 2024-04-02T14:51:15.083 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2024-04-02T14:51:15.083 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2024-04-02T14:51:15.083 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2024-04-02T14:51:15.083 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-02T14:51:15.083 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:15.083 INFO:teuthology.orchestra.run.smithi092.stdout: "throttle-write_buf_throttle": { 2024-04-02T14:51:15.083 INFO:teuthology.orchestra.run.smithi092.stdout: "val": 0, 2024-04-02T14:51:15.083 INFO:teuthology.orchestra.run.smithi092.stdout: "max": 3758096384, 2024-04-02T14:51:15.083 INFO:teuthology.orchestra.run.smithi092.stdout: "get_started": 0, 2024-04-02T14:51:15.083 INFO:teuthology.orchestra.run.smithi092.stdout: "get": 0, 2024-04-02T14:51:15.083 INFO:teuthology.orchestra.run.smithi092.stdout: "get_sum": 0, 2024-04-02T14:51:15.083 INFO:teuthology.orchestra.run.smithi092.stdout: "get_or_fail_fail": 0, 2024-04-02T14:51:15.084 INFO:teuthology.orchestra.run.smithi092.stdout: "get_or_fail_success": 0, 2024-04-02T14:51:15.084 INFO:teuthology.orchestra.run.smithi092.stdout: "take": 0, 2024-04-02T14:51:15.084 INFO:teuthology.orchestra.run.smithi092.stdout: "take_sum": 0, 2024-04-02T14:51:15.084 INFO:teuthology.orchestra.run.smithi092.stdout: "put": 0, 2024-04-02T14:51:15.084 INFO:teuthology.orchestra.run.smithi092.stdout: "put_sum": 0, 2024-04-02T14:51:15.084 INFO:teuthology.orchestra.run.smithi092.stdout: "wait": { 2024-04-02T14:51:15.084 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2024-04-02T14:51:15.084 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2024-04-02T14:51:15.084 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2024-04-02T14:51:15.084 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-02T14:51:15.084 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:15.084 INFO:teuthology.orchestra.run.smithi092.stdout: "throttle-write_buf_throttle-0x5619e132e500": { 2024-04-02T14:51:15.084 INFO:teuthology.orchestra.run.smithi092.stdout: "val": 0, 2024-04-02T14:51:15.084 INFO:teuthology.orchestra.run.smithi092.stdout: "max": 3758096384, 2024-04-02T14:51:15.084 INFO:teuthology.orchestra.run.smithi092.stdout: "get_started": 0, 2024-04-02T14:51:15.084 INFO:teuthology.orchestra.run.smithi092.stdout: "get": 8, 2024-04-02T14:51:15.085 INFO:teuthology.orchestra.run.smithi092.stdout: "get_sum": 2349, 2024-04-02T14:51:15.085 INFO:teuthology.orchestra.run.smithi092.stdout: "get_or_fail_fail": 0, 2024-04-02T14:51:15.085 INFO:teuthology.orchestra.run.smithi092.stdout: "get_or_fail_success": 8, 2024-04-02T14:51:15.085 INFO:teuthology.orchestra.run.smithi092.stdout: "take": 0, 2024-04-02T14:51:15.085 INFO:teuthology.orchestra.run.smithi092.stdout: "take_sum": 0, 2024-04-02T14:51:15.085 INFO:teuthology.orchestra.run.smithi092.stdout: "put": 7, 2024-04-02T14:51:15.085 INFO:teuthology.orchestra.run.smithi092.stdout: "put_sum": 2349, 2024-04-02T14:51:15.085 INFO:teuthology.orchestra.run.smithi092.stdout: "wait": { 2024-04-02T14:51:15.085 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2024-04-02T14:51:15.085 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2024-04-02T14:51:15.085 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2024-04-02T14:51:15.085 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-02T14:51:15.085 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-02T14:51:15.085 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-04-02T14:51:15.806 INFO:tasks.check_counter:Daemon mds.h mds.exported=0 2024-04-02T14:51:15.806 INFO:tasks.check_counter:Daemon mds.h mds.imported=0 2024-04-02T14:51:15.806 INFO:tasks.check_counter:Daemon mds.h mds_cache.dir_update=0 2024-04-02T14:51:15.806 INFO:tasks.check_counter:Daemon mds.h mds_cache.dir_update_receipt=99 2024-04-02T14:51:15.806 INFO:tasks.check_counter:Daemon mds.h mds.root_rsnaps=6 2024-04-02T14:51:15.806 INFO:tasks.check_counter:Daemon mds.h mds_server.req_mksnap_latency.avgcount=0 2024-04-02T14:51:15.806 INFO:tasks.check_counter:Daemon mds.h mds_server.req_rmsnap_latency.avgcount=0 2024-04-02T14:51:15.806 DEBUG:tasks.check_counter:Getting stats from k 2024-04-02T14:51:15.807 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:191d963ac9de7322da8aff20b567b3fa458e2ef6 shell --fsid 6449448a-f0fc-11ee-b647-cb9ed24678a4 -- ceph daemon mds.k perf dump 2024-04-02T14:51:16.046 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/mon.b/config 2024-04-02T14:51:16.102 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:51:15 smithi092 ceph-mon[36497]: pgmap v513: 97 pgs: 21 active+clean+snaptrim_wait, 8 active+clean+snaptrim, 68 active+clean; 4.0 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 14 KiB/s wr, 11 op/s 2024-04-02T14:51:16.178 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:51:15 smithi177 ceph-mon[36431]: pgmap v513: 97 pgs: 21 active+clean+snaptrim_wait, 8 active+clean+snaptrim, 68 active+clean; 4.0 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 14 KiB/s wr, 11 op/s 2024-04-02T14:51:16.179 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:51:15 smithi090 ceph-mon[31674]: pgmap v513: 97 pgs: 21 active+clean+snaptrim_wait, 8 active+clean+snaptrim, 68 active+clean; 4.0 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 14 KiB/s wr, 11 op/s 2024-04-02T14:51:17.807 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-04-02T14:51:17.808 INFO:teuthology.orchestra.run.smithi092.stdout: "AsyncMessenger::Worker-0": { 2024-04-02T14:51:17.808 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_recv_messages": 2257, 2024-04-02T14:51:17.808 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_send_messages": 2259, 2024-04-02T14:51:17.808 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_recv_bytes": 676856, 2024-04-02T14:51:17.808 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_send_bytes": 717727, 2024-04-02T14:51:17.808 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_created_connections": 6, 2024-04-02T14:51:17.808 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_active_connections": 2, 2024-04-02T14:51:17.808 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_running_total_time": 0.167424055, 2024-04-02T14:51:17.808 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_running_send_time": 0.057939130, 2024-04-02T14:51:17.808 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_running_recv_time": 0.041947997, 2024-04-02T14:51:17.808 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_running_fast_dispatch_time": 0.052801633, 2024-04-02T14:51:17.808 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_send_messages_queue_lat": { 2024-04-02T14:51:17.809 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 2257, 2024-04-02T14:51:17.809 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.098055941, 2024-04-02T14:51:17.809 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000043445 2024-04-02T14:51:17.809 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:17.809 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_handle_ack_lat": { 2024-04-02T14:51:17.809 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2024-04-02T14:51:17.809 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2024-04-02T14:51:17.809 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2024-04-02T14:51:17.809 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:17.809 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-02T14:51:17.809 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_send_encrypted_bytes": 0 2024-04-02T14:51:17.809 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:17.809 INFO:teuthology.orchestra.run.smithi092.stdout: "AsyncMessenger::Worker-1": { 2024-04-02T14:51:17.809 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_recv_messages": 1336, 2024-04-02T14:51:17.810 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_send_messages": 1324, 2024-04-02T14:51:17.810 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_recv_bytes": 1898514, 2024-04-02T14:51:17.810 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_send_bytes": 511042, 2024-04-02T14:51:17.810 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_created_connections": 4, 2024-04-02T14:51:17.810 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_active_connections": 3, 2024-04-02T14:51:17.810 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_running_total_time": 0.123776625, 2024-04-02T14:51:17.810 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_running_send_time": 0.034175646, 2024-04-02T14:51:17.810 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_running_recv_time": 0.056256317, 2024-04-02T14:51:17.810 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_running_fast_dispatch_time": 0.037012909, 2024-04-02T14:51:17.810 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_send_messages_queue_lat": { 2024-04-02T14:51:17.810 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 1323, 2024-04-02T14:51:17.810 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.052783547, 2024-04-02T14:51:17.811 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000039896 2024-04-02T14:51:17.811 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:17.811 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_handle_ack_lat": { 2024-04-02T14:51:17.811 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2024-04-02T14:51:17.811 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2024-04-02T14:51:17.811 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2024-04-02T14:51:17.811 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:17.811 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_recv_encrypted_bytes": 1655552, 2024-04-02T14:51:17.811 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_send_encrypted_bytes": 197744 2024-04-02T14:51:17.811 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:17.811 INFO:teuthology.orchestra.run.smithi092.stdout: "AsyncMessenger::Worker-2": { 2024-04-02T14:51:17.811 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_recv_messages": 7, 2024-04-02T14:51:17.811 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_send_messages": 241, 2024-04-02T14:51:17.811 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_recv_bytes": 3660, 2024-04-02T14:51:17.811 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_send_bytes": 578926, 2024-04-02T14:51:17.812 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_created_connections": 7, 2024-04-02T14:51:17.812 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_active_connections": 4, 2024-04-02T14:51:17.812 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_running_total_time": 0.012142052, 2024-04-02T14:51:17.812 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_running_send_time": 0.007706621, 2024-04-02T14:51:17.812 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_running_recv_time": 0.002340855, 2024-04-02T14:51:17.812 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_running_fast_dispatch_time": 0.000015316, 2024-04-02T14:51:17.812 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_send_messages_queue_lat": { 2024-04-02T14:51:17.812 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 239, 2024-04-02T14:51:17.812 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.021708140, 2024-04-02T14:51:17.812 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000090829 2024-04-02T14:51:17.812 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:17.812 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_handle_ack_lat": { 2024-04-02T14:51:17.812 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2024-04-02T14:51:17.812 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2024-04-02T14:51:17.812 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2024-04-02T14:51:17.813 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:17.813 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_recv_encrypted_bytes": 3392, 2024-04-02T14:51:17.813 INFO:teuthology.orchestra.run.smithi092.stdout: "msgr_send_encrypted_bytes": 578608 2024-04-02T14:51:17.813 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:17.813 INFO:teuthology.orchestra.run.smithi092.stdout: "cct": { 2024-04-02T14:51:17.813 INFO:teuthology.orchestra.run.smithi092.stdout: "total_workers": 1, 2024-04-02T14:51:17.813 INFO:teuthology.orchestra.run.smithi092.stdout: "unhealthy_workers": 0 2024-04-02T14:51:17.813 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:17.813 INFO:teuthology.orchestra.run.smithi092.stdout: "finisher-MDSRank": { 2024-04-02T14:51:17.813 INFO:teuthology.orchestra.run.smithi092.stdout: "queue_len": 0, 2024-04-02T14:51:17.813 INFO:teuthology.orchestra.run.smithi092.stdout: "complete_latency": { 2024-04-02T14:51:17.813 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 5397, 2024-04-02T14:51:17.813 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.121189180, 2024-04-02T14:51:17.814 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000022454 2024-04-02T14:51:17.814 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-02T14:51:17.814 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:17.814 INFO:teuthology.orchestra.run.smithi092.stdout: "finisher-PurgeQueue": { 2024-04-02T14:51:17.814 INFO:teuthology.orchestra.run.smithi092.stdout: "queue_len": 0, 2024-04-02T14:51:17.814 INFO:teuthology.orchestra.run.smithi092.stdout: "complete_latency": { 2024-04-02T14:51:17.814 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2024-04-02T14:51:17.814 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2024-04-02T14:51:17.814 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2024-04-02T14:51:17.814 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-02T14:51:17.814 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:17.814 INFO:teuthology.orchestra.run.smithi092.stdout: "mds": { 2024-04-02T14:51:17.814 INFO:teuthology.orchestra.run.smithi092.stdout: "request": 0, 2024-04-02T14:51:17.814 INFO:teuthology.orchestra.run.smithi092.stdout: "reply": 0, 2024-04-02T14:51:17.815 INFO:teuthology.orchestra.run.smithi092.stdout: "reply_latency": { 2024-04-02T14:51:17.815 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2024-04-02T14:51:17.815 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2024-04-02T14:51:17.815 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2024-04-02T14:51:17.815 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:17.815 INFO:teuthology.orchestra.run.smithi092.stdout: "slow_reply": 0, 2024-04-02T14:51:17.815 INFO:teuthology.orchestra.run.smithi092.stdout: "forward": 0, 2024-04-02T14:51:17.815 INFO:teuthology.orchestra.run.smithi092.stdout: "dir_fetch_complete": 0, 2024-04-02T14:51:17.815 INFO:teuthology.orchestra.run.smithi092.stdout: "dir_fetch_keys": 0, 2024-04-02T14:51:17.815 INFO:teuthology.orchestra.run.smithi092.stdout: "dir_commit": 0, 2024-04-02T14:51:17.815 INFO:teuthology.orchestra.run.smithi092.stdout: "dir_split": 0, 2024-04-02T14:51:17.815 INFO:teuthology.orchestra.run.smithi092.stdout: "dir_merge": 0, 2024-04-02T14:51:17.815 INFO:teuthology.orchestra.run.smithi092.stdout: "inodes": 0, 2024-04-02T14:51:17.816 INFO:teuthology.orchestra.run.smithi092.stdout: "inodes_top": 0, 2024-04-02T14:51:17.816 INFO:teuthology.orchestra.run.smithi092.stdout: "inodes_bottom": 0, 2024-04-02T14:51:17.816 INFO:teuthology.orchestra.run.smithi092.stdout: "inodes_pin_tail": 0, 2024-04-02T14:51:17.816 INFO:teuthology.orchestra.run.smithi092.stdout: "inodes_pinned": 0, 2024-04-02T14:51:17.816 INFO:teuthology.orchestra.run.smithi092.stdout: "inodes_expired": 0, 2024-04-02T14:51:17.816 INFO:teuthology.orchestra.run.smithi092.stdout: "inodes_with_caps": 0, 2024-04-02T14:51:17.816 INFO:teuthology.orchestra.run.smithi092.stdout: "caps": 0, 2024-04-02T14:51:17.816 INFO:teuthology.orchestra.run.smithi092.stdout: "subtrees": 1, 2024-04-02T14:51:17.816 INFO:teuthology.orchestra.run.smithi092.stdout: "traverse": 0, 2024-04-02T14:51:17.816 INFO:teuthology.orchestra.run.smithi092.stdout: "traverse_hit": 0, 2024-04-02T14:51:17.816 INFO:teuthology.orchestra.run.smithi092.stdout: "traverse_forward": 0, 2024-04-02T14:51:17.816 INFO:teuthology.orchestra.run.smithi092.stdout: "traverse_discover": 0, 2024-04-02T14:51:17.816 INFO:teuthology.orchestra.run.smithi092.stdout: "traverse_dir_fetch": 0, 2024-04-02T14:51:17.816 INFO:teuthology.orchestra.run.smithi092.stdout: "traverse_remote_ino": 0, 2024-04-02T14:51:17.816 INFO:teuthology.orchestra.run.smithi092.stdout: "traverse_lock": 0, 2024-04-02T14:51:17.816 INFO:teuthology.orchestra.run.smithi092.stdout: "load_cent": 0, 2024-04-02T14:51:17.817 INFO:teuthology.orchestra.run.smithi092.stdout: "q": 0, 2024-04-02T14:51:17.817 INFO:teuthology.orchestra.run.smithi092.stdout: "exported": 0, 2024-04-02T14:51:17.817 INFO:teuthology.orchestra.run.smithi092.stdout: "exported_inodes": 0, 2024-04-02T14:51:17.817 INFO:teuthology.orchestra.run.smithi092.stdout: "imported": 0, 2024-04-02T14:51:17.817 INFO:teuthology.orchestra.run.smithi092.stdout: "imported_inodes": 0, 2024-04-02T14:51:17.817 INFO:teuthology.orchestra.run.smithi092.stdout: "openino_dir_fetch": 0, 2024-04-02T14:51:17.817 INFO:teuthology.orchestra.run.smithi092.stdout: "openino_backtrace_fetch": 0, 2024-04-02T14:51:17.817 INFO:teuthology.orchestra.run.smithi092.stdout: "openino_peer_discover": 0, 2024-04-02T14:51:17.817 INFO:teuthology.orchestra.run.smithi092.stdout: "root_rfiles": 0, 2024-04-02T14:51:17.817 INFO:teuthology.orchestra.run.smithi092.stdout: "root_rbytes": 0, 2024-04-02T14:51:17.817 INFO:teuthology.orchestra.run.smithi092.stdout: "root_rsnaps": 0, 2024-04-02T14:51:17.817 INFO:teuthology.orchestra.run.smithi092.stdout: "scrub_backtrace_fetch": 0, 2024-04-02T14:51:17.817 INFO:teuthology.orchestra.run.smithi092.stdout: "scrub_set_tag": 0, 2024-04-02T14:51:17.817 INFO:teuthology.orchestra.run.smithi092.stdout: "scrub_backtrace_repaired": 0, 2024-04-02T14:51:17.817 INFO:teuthology.orchestra.run.smithi092.stdout: "scrub_inotable_repaired": 0, 2024-04-02T14:51:17.818 INFO:teuthology.orchestra.run.smithi092.stdout: "scrub_dir_inodes": 0, 2024-04-02T14:51:17.818 INFO:teuthology.orchestra.run.smithi092.stdout: "scrub_dir_base_inodes": 0, 2024-04-02T14:51:17.818 INFO:teuthology.orchestra.run.smithi092.stdout: "scrub_dirfrag_rstats": 0, 2024-04-02T14:51:17.818 INFO:teuthology.orchestra.run.smithi092.stdout: "scrub_file_inodes": 0, 2024-04-02T14:51:17.818 INFO:teuthology.orchestra.run.smithi092.stdout: "handle_inode_file_caps": 0, 2024-04-02T14:51:17.818 INFO:teuthology.orchestra.run.smithi092.stdout: "ceph_cap_op_revoke": 0, 2024-04-02T14:51:17.818 INFO:teuthology.orchestra.run.smithi092.stdout: "ceph_cap_op_grant": 0, 2024-04-02T14:51:17.818 INFO:teuthology.orchestra.run.smithi092.stdout: "ceph_cap_op_trunc": 0, 2024-04-02T14:51:17.818 INFO:teuthology.orchestra.run.smithi092.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-02T14:51:17.818 INFO:teuthology.orchestra.run.smithi092.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-02T14:51:17.818 INFO:teuthology.orchestra.run.smithi092.stdout: "handle_client_caps": 0, 2024-04-02T14:51:17.818 INFO:teuthology.orchestra.run.smithi092.stdout: "handle_client_caps_dirty": 0, 2024-04-02T14:51:17.818 INFO:teuthology.orchestra.run.smithi092.stdout: "handle_client_cap_release": 0, 2024-04-02T14:51:17.818 INFO:teuthology.orchestra.run.smithi092.stdout: "process_request_cap_release": 0 2024-04-02T14:51:17.818 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:17.819 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_cache": { 2024-04-02T14:51:17.819 INFO:teuthology.orchestra.run.smithi092.stdout: "dir_update": 0, 2024-04-02T14:51:17.819 INFO:teuthology.orchestra.run.smithi092.stdout: "dir_update_receipt": 0, 2024-04-02T14:51:17.819 INFO:teuthology.orchestra.run.smithi092.stdout: "dir_try_discover": 0, 2024-04-02T14:51:17.819 INFO:teuthology.orchestra.run.smithi092.stdout: "dir_send_discover": 0, 2024-04-02T14:51:17.819 INFO:teuthology.orchestra.run.smithi092.stdout: "dir_handle_discover": 0, 2024-04-02T14:51:17.819 INFO:teuthology.orchestra.run.smithi092.stdout: "num_strays": 0, 2024-04-02T14:51:17.819 INFO:teuthology.orchestra.run.smithi092.stdout: "num_strays_delayed": 0, 2024-04-02T14:51:17.819 INFO:teuthology.orchestra.run.smithi092.stdout: "num_strays_enqueuing": 0, 2024-04-02T14:51:17.819 INFO:teuthology.orchestra.run.smithi092.stdout: "strays_created": 0, 2024-04-02T14:51:17.819 INFO:teuthology.orchestra.run.smithi092.stdout: "strays_enqueued": 0, 2024-04-02T14:51:17.819 INFO:teuthology.orchestra.run.smithi092.stdout: "strays_reintegrated": 0, 2024-04-02T14:51:17.819 INFO:teuthology.orchestra.run.smithi092.stdout: "strays_migrated": 0, 2024-04-02T14:51:17.819 INFO:teuthology.orchestra.run.smithi092.stdout: "num_recovering_processing": 0, 2024-04-02T14:51:17.819 INFO:teuthology.orchestra.run.smithi092.stdout: "num_recovering_enqueued": 0, 2024-04-02T14:51:17.819 INFO:teuthology.orchestra.run.smithi092.stdout: "num_recovering_prioritized": 0, 2024-04-02T14:51:17.820 INFO:teuthology.orchestra.run.smithi092.stdout: "recovery_started": 0, 2024-04-02T14:51:17.820 INFO:teuthology.orchestra.run.smithi092.stdout: "recovery_completed": 0, 2024-04-02T14:51:17.820 INFO:teuthology.orchestra.run.smithi092.stdout: "ireq_quiesce_path": 0, 2024-04-02T14:51:17.820 INFO:teuthology.orchestra.run.smithi092.stdout: "ireq_quiesce_inode": 0, 2024-04-02T14:51:17.820 INFO:teuthology.orchestra.run.smithi092.stdout: "ireq_enqueue_scrub": 0, 2024-04-02T14:51:17.820 INFO:teuthology.orchestra.run.smithi092.stdout: "ireq_exportdir": 0, 2024-04-02T14:51:17.820 INFO:teuthology.orchestra.run.smithi092.stdout: "ireq_flush": 0, 2024-04-02T14:51:17.820 INFO:teuthology.orchestra.run.smithi092.stdout: "ireq_fragmentdir": 0, 2024-04-02T14:51:17.820 INFO:teuthology.orchestra.run.smithi092.stdout: "ireq_fragstats": 0, 2024-04-02T14:51:17.820 INFO:teuthology.orchestra.run.smithi092.stdout: "ireq_inodestats": 0 2024-04-02T14:51:17.820 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:17.820 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_log": { 2024-04-02T14:51:17.820 INFO:teuthology.orchestra.run.smithi092.stdout: "evlrg": 0, 2024-04-02T14:51:17.820 INFO:teuthology.orchestra.run.smithi092.stdout: "evadd": 0, 2024-04-02T14:51:17.820 INFO:teuthology.orchestra.run.smithi092.stdout: "evex": 0, 2024-04-02T14:51:17.821 INFO:teuthology.orchestra.run.smithi092.stdout: "evtrm": 0, 2024-04-02T14:51:17.821 INFO:teuthology.orchestra.run.smithi092.stdout: "ev": 8, 2024-04-02T14:51:17.821 INFO:teuthology.orchestra.run.smithi092.stdout: "evexg": 0, 2024-04-02T14:51:17.821 INFO:teuthology.orchestra.run.smithi092.stdout: "evexd": 0, 2024-04-02T14:51:17.821 INFO:teuthology.orchestra.run.smithi092.stdout: "segadd": 0, 2024-04-02T14:51:17.821 INFO:teuthology.orchestra.run.smithi092.stdout: "segex": 0, 2024-04-02T14:51:17.821 INFO:teuthology.orchestra.run.smithi092.stdout: "segtrm": 0, 2024-04-02T14:51:17.821 INFO:teuthology.orchestra.run.smithi092.stdout: "seg": 2, 2024-04-02T14:51:17.821 INFO:teuthology.orchestra.run.smithi092.stdout: "segmjr": 2, 2024-04-02T14:51:17.821 INFO:teuthology.orchestra.run.smithi092.stdout: "segexg": 0, 2024-04-02T14:51:17.821 INFO:teuthology.orchestra.run.smithi092.stdout: "segexd": 0, 2024-04-02T14:51:17.821 INFO:teuthology.orchestra.run.smithi092.stdout: "expos": 4194304, 2024-04-02T14:51:17.821 INFO:teuthology.orchestra.run.smithi092.stdout: "wrpos": 4196653, 2024-04-02T14:51:17.821 INFO:teuthology.orchestra.run.smithi092.stdout: "rdpos": 4196512, 2024-04-02T14:51:17.821 INFO:teuthology.orchestra.run.smithi092.stdout: "jlat": { 2024-04-02T14:51:17.821 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2024-04-02T14:51:17.822 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2024-04-02T14:51:17.822 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2024-04-02T14:51:17.822 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:17.822 INFO:teuthology.orchestra.run.smithi092.stdout: "replayed": 8 2024-04-02T14:51:17.822 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:17.822 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_mem": { 2024-04-02T14:51:17.822 INFO:teuthology.orchestra.run.smithi092.stdout: "ino": 3, 2024-04-02T14:51:17.822 INFO:teuthology.orchestra.run.smithi092.stdout: "ino+": 3, 2024-04-02T14:51:17.822 INFO:teuthology.orchestra.run.smithi092.stdout: "ino-": 0, 2024-04-02T14:51:17.822 INFO:teuthology.orchestra.run.smithi092.stdout: "dir": 1, 2024-04-02T14:51:17.822 INFO:teuthology.orchestra.run.smithi092.stdout: "dir+": 1, 2024-04-02T14:51:17.822 INFO:teuthology.orchestra.run.smithi092.stdout: "dir-": 0, 2024-04-02T14:51:17.822 INFO:teuthology.orchestra.run.smithi092.stdout: "dn": 0, 2024-04-02T14:51:17.822 INFO:teuthology.orchestra.run.smithi092.stdout: "dn+": 0, 2024-04-02T14:51:17.822 INFO:teuthology.orchestra.run.smithi092.stdout: "dn-": 0, 2024-04-02T14:51:17.823 INFO:teuthology.orchestra.run.smithi092.stdout: "cap": 0, 2024-04-02T14:51:17.823 INFO:teuthology.orchestra.run.smithi092.stdout: "cap+": 0, 2024-04-02T14:51:17.823 INFO:teuthology.orchestra.run.smithi092.stdout: "cap-": 0, 2024-04-02T14:51:17.823 INFO:teuthology.orchestra.run.smithi092.stdout: "rss": 52020, 2024-04-02T14:51:17.823 INFO:teuthology.orchestra.run.smithi092.stdout: "heap": 215416 2024-04-02T14:51:17.823 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:17.823 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_server": { 2024-04-02T14:51:17.823 INFO:teuthology.orchestra.run.smithi092.stdout: "dispatch_client_request": 0, 2024-04-02T14:51:17.823 INFO:teuthology.orchestra.run.smithi092.stdout: "dispatch_server_request": 0, 2024-04-02T14:51:17.823 INFO:teuthology.orchestra.run.smithi092.stdout: "handle_client_request": 0, 2024-04-02T14:51:17.823 INFO:teuthology.orchestra.run.smithi092.stdout: "handle_client_session": 0, 2024-04-02T14:51:17.823 INFO:teuthology.orchestra.run.smithi092.stdout: "handle_peer_request": 0, 2024-04-02T14:51:17.823 INFO:teuthology.orchestra.run.smithi092.stdout: "req_create_latency": { 2024-04-02T14:51:17.823 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2024-04-02T14:51:17.824 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2024-04-02T14:51:17.824 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2024-04-02T14:51:17.824 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:17.824 INFO:teuthology.orchestra.run.smithi092.stdout: "req_getattr_latency": { 2024-04-02T14:51:17.824 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2024-04-02T14:51:17.824 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2024-04-02T14:51:17.824 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2024-04-02T14:51:17.824 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:17.824 INFO:teuthology.orchestra.run.smithi092.stdout: "req_getfilelock_latency": { 2024-04-02T14:51:17.824 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2024-04-02T14:51:17.824 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2024-04-02T14:51:17.824 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2024-04-02T14:51:17.825 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:17.825 INFO:teuthology.orchestra.run.smithi092.stdout: "req_link_latency": { 2024-04-02T14:51:17.825 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2024-04-02T14:51:17.825 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2024-04-02T14:51:17.825 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2024-04-02T14:51:17.825 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:17.825 INFO:teuthology.orchestra.run.smithi092.stdout: "req_lookup_latency": { 2024-04-02T14:51:17.825 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2024-04-02T14:51:17.825 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2024-04-02T14:51:17.825 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2024-04-02T14:51:17.825 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:17.825 INFO:teuthology.orchestra.run.smithi092.stdout: "req_lookuphash_latency": { 2024-04-02T14:51:17.825 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2024-04-02T14:51:17.825 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2024-04-02T14:51:17.825 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2024-04-02T14:51:17.826 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:17.826 INFO:teuthology.orchestra.run.smithi092.stdout: "req_lookupino_latency": { 2024-04-02T14:51:17.826 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2024-04-02T14:51:17.826 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2024-04-02T14:51:17.826 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2024-04-02T14:51:17.826 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:17.826 INFO:teuthology.orchestra.run.smithi092.stdout: "req_lookupname_latency": { 2024-04-02T14:51:17.826 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2024-04-02T14:51:17.826 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2024-04-02T14:51:17.826 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2024-04-02T14:51:17.826 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:17.826 INFO:teuthology.orchestra.run.smithi092.stdout: "req_lookupparent_latency": { 2024-04-02T14:51:17.826 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2024-04-02T14:51:17.826 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2024-04-02T14:51:17.827 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2024-04-02T14:51:17.827 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:17.827 INFO:teuthology.orchestra.run.smithi092.stdout: "req_lookupsnap_latency": { 2024-04-02T14:51:17.827 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2024-04-02T14:51:17.827 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2024-04-02T14:51:17.827 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2024-04-02T14:51:17.827 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:17.827 INFO:teuthology.orchestra.run.smithi092.stdout: "req_lssnap_latency": { 2024-04-02T14:51:17.827 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2024-04-02T14:51:17.827 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2024-04-02T14:51:17.828 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2024-04-02T14:51:17.828 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:17.828 INFO:teuthology.orchestra.run.smithi092.stdout: "req_mkdir_latency": { 2024-04-02T14:51:17.828 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2024-04-02T14:51:17.828 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2024-04-02T14:51:17.829 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2024-04-02T14:51:17.829 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:17.829 INFO:teuthology.orchestra.run.smithi092.stdout: "req_mknod_latency": { 2024-04-02T14:51:17.829 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2024-04-02T14:51:17.829 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2024-04-02T14:51:17.829 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2024-04-02T14:51:17.829 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:17.829 INFO:teuthology.orchestra.run.smithi092.stdout: "req_mksnap_latency": { 2024-04-02T14:51:17.829 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2024-04-02T14:51:17.829 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2024-04-02T14:51:17.829 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2024-04-02T14:51:17.829 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:17.829 INFO:teuthology.orchestra.run.smithi092.stdout: "req_open_latency": { 2024-04-02T14:51:17.830 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2024-04-02T14:51:17.830 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2024-04-02T14:51:17.830 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2024-04-02T14:51:17.830 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:17.830 INFO:teuthology.orchestra.run.smithi092.stdout: "req_readdir_latency": { 2024-04-02T14:51:17.830 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2024-04-02T14:51:17.830 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2024-04-02T14:51:17.830 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2024-04-02T14:51:17.830 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:17.830 INFO:teuthology.orchestra.run.smithi092.stdout: "req_rename_latency": { 2024-04-02T14:51:17.830 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2024-04-02T14:51:17.830 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2024-04-02T14:51:17.830 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2024-04-02T14:51:17.831 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:17.831 INFO:teuthology.orchestra.run.smithi092.stdout: "req_renamesnap_latency": { 2024-04-02T14:51:17.831 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2024-04-02T14:51:17.831 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2024-04-02T14:51:17.831 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2024-04-02T14:51:17.831 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:17.831 INFO:teuthology.orchestra.run.smithi092.stdout: "req_snapdiff_latency": { 2024-04-02T14:51:17.831 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2024-04-02T14:51:17.831 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2024-04-02T14:51:17.831 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2024-04-02T14:51:17.831 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:17.831 INFO:teuthology.orchestra.run.smithi092.stdout: "req_rmdir_latency": { 2024-04-02T14:51:17.831 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2024-04-02T14:51:17.832 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2024-04-02T14:51:17.832 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2024-04-02T14:51:17.832 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:17.832 INFO:teuthology.orchestra.run.smithi092.stdout: "req_rmsnap_latency": { 2024-04-02T14:51:17.832 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2024-04-02T14:51:17.832 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2024-04-02T14:51:17.832 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2024-04-02T14:51:17.832 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:17.832 INFO:teuthology.orchestra.run.smithi092.stdout: "req_rmxattr_latency": { 2024-04-02T14:51:17.832 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2024-04-02T14:51:17.832 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2024-04-02T14:51:17.832 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2024-04-02T14:51:17.832 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:17.832 INFO:teuthology.orchestra.run.smithi092.stdout: "req_setattr_latency": { 2024-04-02T14:51:17.833 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2024-04-02T14:51:17.833 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2024-04-02T14:51:17.833 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2024-04-02T14:51:17.833 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:17.833 INFO:teuthology.orchestra.run.smithi092.stdout: "req_setdirlayout_latency": { 2024-04-02T14:51:17.833 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2024-04-02T14:51:17.833 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2024-04-02T14:51:17.833 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2024-04-02T14:51:17.833 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:17.833 INFO:teuthology.orchestra.run.smithi092.stdout: "req_setfilelock_latency": { 2024-04-02T14:51:17.833 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2024-04-02T14:51:17.833 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2024-04-02T14:51:17.833 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2024-04-02T14:51:17.833 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:17.834 INFO:teuthology.orchestra.run.smithi092.stdout: "req_setlayout_latency": { 2024-04-02T14:51:17.834 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2024-04-02T14:51:17.834 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2024-04-02T14:51:17.834 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2024-04-02T14:51:17.834 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:17.834 INFO:teuthology.orchestra.run.smithi092.stdout: "req_setxattr_latency": { 2024-04-02T14:51:17.834 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2024-04-02T14:51:17.834 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2024-04-02T14:51:17.834 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2024-04-02T14:51:17.834 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:17.834 INFO:teuthology.orchestra.run.smithi092.stdout: "req_symlink_latency": { 2024-04-02T14:51:17.834 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2024-04-02T14:51:17.834 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2024-04-02T14:51:17.834 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2024-04-02T14:51:17.835 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:17.835 INFO:teuthology.orchestra.run.smithi092.stdout: "req_unlink_latency": { 2024-04-02T14:51:17.835 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2024-04-02T14:51:17.835 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2024-04-02T14:51:17.835 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2024-04-02T14:51:17.835 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:17.835 INFO:teuthology.orchestra.run.smithi092.stdout: "cap_revoke_eviction": 0, 2024-04-02T14:51:17.835 INFO:teuthology.orchestra.run.smithi092.stdout: "cap_acquisition_throttle": 0, 2024-04-02T14:51:17.835 INFO:teuthology.orchestra.run.smithi092.stdout: "req_getvxattr_latency": { 2024-04-02T14:51:17.835 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2024-04-02T14:51:17.835 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2024-04-02T14:51:17.835 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2024-04-02T14:51:17.835 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-02T14:51:17.835 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:17.835 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_sessions": { 2024-04-02T14:51:17.836 INFO:teuthology.orchestra.run.smithi092.stdout: "session_count": 2, 2024-04-02T14:51:17.836 INFO:teuthology.orchestra.run.smithi092.stdout: "session_add": 4, 2024-04-02T14:51:17.836 INFO:teuthology.orchestra.run.smithi092.stdout: "session_remove": 2, 2024-04-02T14:51:17.836 INFO:teuthology.orchestra.run.smithi092.stdout: "sessions_open": 4, 2024-04-02T14:51:17.836 INFO:teuthology.orchestra.run.smithi092.stdout: "sessions_stale": 0, 2024-04-02T14:51:17.836 INFO:teuthology.orchestra.run.smithi092.stdout: "total_load": 0, 2024-04-02T14:51:17.836 INFO:teuthology.orchestra.run.smithi092.stdout: "average_load": 0, 2024-04-02T14:51:17.836 INFO:teuthology.orchestra.run.smithi092.stdout: "avg_session_uptime": 4362, 2024-04-02T14:51:17.836 INFO:teuthology.orchestra.run.smithi092.stdout: "mdthresh_evicted": 0 2024-04-02T14:51:17.836 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:17.836 INFO:teuthology.orchestra.run.smithi092.stdout: "mempool": { 2024-04-02T14:51:17.836 INFO:teuthology.orchestra.run.smithi092.stdout: "bloom_filter_bytes": 0, 2024-04-02T14:51:17.836 INFO:teuthology.orchestra.run.smithi092.stdout: "bloom_filter_items": 0, 2024-04-02T14:51:17.837 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_alloc_bytes": 0, 2024-04-02T14:51:17.837 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_alloc_items": 0, 2024-04-02T14:51:17.837 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_cache_data_bytes": 0, 2024-04-02T14:51:17.837 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_cache_data_items": 0, 2024-04-02T14:51:17.837 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-02T14:51:17.837 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_cache_onode_items": 0, 2024-04-02T14:51:17.837 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-02T14:51:17.837 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_cache_meta_items": 0, 2024-04-02T14:51:17.837 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_cache_other_bytes": 0, 2024-04-02T14:51:17.837 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_cache_other_items": 0, 2024-04-02T14:51:17.837 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-02T14:51:17.837 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_cache_buffer_items": 0, 2024-04-02T14:51:17.837 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_extent_bytes": 0, 2024-04-02T14:51:17.837 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_extent_items": 0, 2024-04-02T14:51:17.837 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_blob_bytes": 0, 2024-04-02T14:51:17.838 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_blob_items": 0, 2024-04-02T14:51:17.838 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-02T14:51:17.838 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_shared_blob_items": 0, 2024-04-02T14:51:17.838 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-02T14:51:17.838 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_inline_bl_items": 0, 2024-04-02T14:51:17.838 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_fsck_bytes": 0, 2024-04-02T14:51:17.838 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_fsck_items": 0, 2024-04-02T14:51:17.838 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_txc_bytes": 0, 2024-04-02T14:51:17.838 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_txc_items": 0, 2024-04-02T14:51:17.838 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-02T14:51:17.838 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_writing_deferred_items": 0, 2024-04-02T14:51:17.838 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_writing_bytes": 0, 2024-04-02T14:51:17.838 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_writing_items": 0, 2024-04-02T14:51:17.838 INFO:teuthology.orchestra.run.smithi092.stdout: "bluefs_bytes": 0, 2024-04-02T14:51:17.838 INFO:teuthology.orchestra.run.smithi092.stdout: "bluefs_items": 0, 2024-04-02T14:51:17.839 INFO:teuthology.orchestra.run.smithi092.stdout: "bluefs_file_reader_bytes": 0, 2024-04-02T14:51:17.839 INFO:teuthology.orchestra.run.smithi092.stdout: "bluefs_file_reader_items": 0, 2024-04-02T14:51:17.839 INFO:teuthology.orchestra.run.smithi092.stdout: "bluefs_file_writer_bytes": 0, 2024-04-02T14:51:17.839 INFO:teuthology.orchestra.run.smithi092.stdout: "bluefs_file_writer_items": 0, 2024-04-02T14:51:17.839 INFO:teuthology.orchestra.run.smithi092.stdout: "buffer_anon_bytes": 190963, 2024-04-02T14:51:17.839 INFO:teuthology.orchestra.run.smithi092.stdout: "buffer_anon_items": 55, 2024-04-02T14:51:17.839 INFO:teuthology.orchestra.run.smithi092.stdout: "buffer_meta_bytes": 0, 2024-04-02T14:51:17.839 INFO:teuthology.orchestra.run.smithi092.stdout: "buffer_meta_items": 0, 2024-04-02T14:51:17.839 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_bytes": 0, 2024-04-02T14:51:17.839 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_items": 0, 2024-04-02T14:51:17.839 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_mapbl_bytes": 0, 2024-04-02T14:51:17.839 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_mapbl_items": 0, 2024-04-02T14:51:17.839 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_pglog_bytes": 0, 2024-04-02T14:51:17.839 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_pglog_items": 0, 2024-04-02T14:51:17.839 INFO:teuthology.orchestra.run.smithi092.stdout: "osdmap_bytes": 8256, 2024-04-02T14:51:17.839 INFO:teuthology.orchestra.run.smithi092.stdout: "osdmap_items": 173, 2024-04-02T14:51:17.840 INFO:teuthology.orchestra.run.smithi092.stdout: "osdmap_mapping_bytes": 0, 2024-04-02T14:51:17.840 INFO:teuthology.orchestra.run.smithi092.stdout: "osdmap_mapping_items": 0, 2024-04-02T14:51:17.840 INFO:teuthology.orchestra.run.smithi092.stdout: "pgmap_bytes": 0, 2024-04-02T14:51:17.840 INFO:teuthology.orchestra.run.smithi092.stdout: "pgmap_items": 0, 2024-04-02T14:51:17.840 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_co_bytes": 8464, 2024-04-02T14:51:17.840 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_co_items": 13, 2024-04-02T14:51:17.840 INFO:teuthology.orchestra.run.smithi092.stdout: "unittest_1_bytes": 0, 2024-04-02T14:51:17.840 INFO:teuthology.orchestra.run.smithi092.stdout: "unittest_1_items": 0, 2024-04-02T14:51:17.840 INFO:teuthology.orchestra.run.smithi092.stdout: "unittest_2_bytes": 0, 2024-04-02T14:51:17.840 INFO:teuthology.orchestra.run.smithi092.stdout: "unittest_2_items": 0 2024-04-02T14:51:17.840 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:17.840 INFO:teuthology.orchestra.run.smithi092.stdout: "objecter": { 2024-04-02T14:51:17.840 INFO:teuthology.orchestra.run.smithi092.stdout: "op_active": 0, 2024-04-02T14:51:17.840 INFO:teuthology.orchestra.run.smithi092.stdout: "op_laggy": 0, 2024-04-02T14:51:17.840 INFO:teuthology.orchestra.run.smithi092.stdout: "op_send": 3243, 2024-04-02T14:51:17.840 INFO:teuthology.orchestra.run.smithi092.stdout: "op_send_bytes": 21, 2024-04-02T14:51:17.840 INFO:teuthology.orchestra.run.smithi092.stdout: "op_resend": 3, 2024-04-02T14:51:17.841 INFO:teuthology.orchestra.run.smithi092.stdout: "op_reply": 3243, 2024-04-02T14:51:17.841 INFO:teuthology.orchestra.run.smithi092.stdout: "op_latency": { 2024-04-02T14:51:17.841 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 3243, 2024-04-02T14:51:17.841 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 3.995884052, 2024-04-02T14:51:17.841 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.001232156 2024-04-02T14:51:17.841 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:17.841 INFO:teuthology.orchestra.run.smithi092.stdout: "op_inflight": 0, 2024-04-02T14:51:17.841 INFO:teuthology.orchestra.run.smithi092.stdout: "oplen_avg": { 2024-04-02T14:51:17.841 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 3243, 2024-04-02T14:51:17.841 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 3244 2024-04-02T14:51:17.841 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:17.841 INFO:teuthology.orchestra.run.smithi092.stdout: "op": 3243, 2024-04-02T14:51:17.841 INFO:teuthology.orchestra.run.smithi092.stdout: "op_r": 3243, 2024-04-02T14:51:17.841 INFO:teuthology.orchestra.run.smithi092.stdout: "op_w": 0, 2024-04-02T14:51:17.841 INFO:teuthology.orchestra.run.smithi092.stdout: "op_rmw": 0, 2024-04-02T14:51:17.841 INFO:teuthology.orchestra.run.smithi092.stdout: "op_pg": 0, 2024-04-02T14:51:17.842 INFO:teuthology.orchestra.run.smithi092.stdout: "osdop_stat": 2107, 2024-04-02T14:51:17.842 INFO:teuthology.orchestra.run.smithi092.stdout: "osdop_create": 0, 2024-04-02T14:51:17.842 INFO:teuthology.orchestra.run.smithi092.stdout: "osdop_read": 1134, 2024-04-02T14:51:17.842 INFO:teuthology.orchestra.run.smithi092.stdout: "osdop_write": 0, 2024-04-02T14:51:17.842 INFO:teuthology.orchestra.run.smithi092.stdout: "osdop_writefull": 0, 2024-04-02T14:51:17.842 INFO:teuthology.orchestra.run.smithi092.stdout: "osdop_writesame": 0, 2024-04-02T14:51:17.842 INFO:teuthology.orchestra.run.smithi092.stdout: "osdop_append": 0, 2024-04-02T14:51:17.842 INFO:teuthology.orchestra.run.smithi092.stdout: "osdop_zero": 0, 2024-04-02T14:51:17.842 INFO:teuthology.orchestra.run.smithi092.stdout: "osdop_truncate": 0, 2024-04-02T14:51:17.842 INFO:teuthology.orchestra.run.smithi092.stdout: "osdop_delete": 0, 2024-04-02T14:51:17.842 INFO:teuthology.orchestra.run.smithi092.stdout: "osdop_mapext": 0, 2024-04-02T14:51:17.842 INFO:teuthology.orchestra.run.smithi092.stdout: "osdop_sparse_read": 0, 2024-04-02T14:51:17.842 INFO:teuthology.orchestra.run.smithi092.stdout: "osdop_clonerange": 0, 2024-04-02T14:51:17.842 INFO:teuthology.orchestra.run.smithi092.stdout: "osdop_getxattr": 1, 2024-04-02T14:51:17.842 INFO:teuthology.orchestra.run.smithi092.stdout: "osdop_setxattr": 0, 2024-04-02T14:51:17.843 INFO:teuthology.orchestra.run.smithi092.stdout: "osdop_cmpxattr": 0, 2024-04-02T14:51:17.843 INFO:teuthology.orchestra.run.smithi092.stdout: "osdop_rmxattr": 0, 2024-04-02T14:51:17.843 INFO:teuthology.orchestra.run.smithi092.stdout: "osdop_resetxattrs": 0, 2024-04-02T14:51:17.843 INFO:teuthology.orchestra.run.smithi092.stdout: "osdop_call": 0, 2024-04-02T14:51:17.843 INFO:teuthology.orchestra.run.smithi092.stdout: "osdop_watch": 0, 2024-04-02T14:51:17.843 INFO:teuthology.orchestra.run.smithi092.stdout: "osdop_notify": 0, 2024-04-02T14:51:17.843 INFO:teuthology.orchestra.run.smithi092.stdout: "osdop_src_cmpxattr": 0, 2024-04-02T14:51:17.843 INFO:teuthology.orchestra.run.smithi092.stdout: "osdop_pgls": 0, 2024-04-02T14:51:17.843 INFO:teuthology.orchestra.run.smithi092.stdout: "osdop_pgls_filter": 0, 2024-04-02T14:51:17.843 INFO:teuthology.orchestra.run.smithi092.stdout: "osdop_other": 0, 2024-04-02T14:51:17.843 INFO:teuthology.orchestra.run.smithi092.stdout: "linger_active": 0, 2024-04-02T14:51:17.843 INFO:teuthology.orchestra.run.smithi092.stdout: "linger_send": 0, 2024-04-02T14:51:17.843 INFO:teuthology.orchestra.run.smithi092.stdout: "linger_resend": 0, 2024-04-02T14:51:17.843 INFO:teuthology.orchestra.run.smithi092.stdout: "linger_ping": 0, 2024-04-02T14:51:17.843 INFO:teuthology.orchestra.run.smithi092.stdout: "poolop_active": 0, 2024-04-02T14:51:17.843 INFO:teuthology.orchestra.run.smithi092.stdout: "poolop_send": 0, 2024-04-02T14:51:17.844 INFO:teuthology.orchestra.run.smithi092.stdout: "poolop_resend": 0, 2024-04-02T14:51:17.844 INFO:teuthology.orchestra.run.smithi092.stdout: "poolstat_active": 0, 2024-04-02T14:51:17.844 INFO:teuthology.orchestra.run.smithi092.stdout: "poolstat_send": 0, 2024-04-02T14:51:17.844 INFO:teuthology.orchestra.run.smithi092.stdout: "poolstat_resend": 0, 2024-04-02T14:51:17.844 INFO:teuthology.orchestra.run.smithi092.stdout: "statfs_active": 0, 2024-04-02T14:51:17.844 INFO:teuthology.orchestra.run.smithi092.stdout: "statfs_send": 0, 2024-04-02T14:51:17.844 INFO:teuthology.orchestra.run.smithi092.stdout: "statfs_resend": 0, 2024-04-02T14:51:17.844 INFO:teuthology.orchestra.run.smithi092.stdout: "command_active": 0, 2024-04-02T14:51:17.844 INFO:teuthology.orchestra.run.smithi092.stdout: "command_send": 0, 2024-04-02T14:51:17.844 INFO:teuthology.orchestra.run.smithi092.stdout: "command_resend": 0, 2024-04-02T14:51:17.844 INFO:teuthology.orchestra.run.smithi092.stdout: "map_epoch": 110, 2024-04-02T14:51:17.844 INFO:teuthology.orchestra.run.smithi092.stdout: "map_full": 0, 2024-04-02T14:51:17.844 INFO:teuthology.orchestra.run.smithi092.stdout: "map_inc": 22, 2024-04-02T14:51:17.844 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_sessions": 5, 2024-04-02T14:51:17.844 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_session_open": 6, 2024-04-02T14:51:17.845 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_session_close": 1, 2024-04-02T14:51:17.845 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_laggy": 0, 2024-04-02T14:51:17.845 INFO:teuthology.orchestra.run.smithi092.stdout: "omap_wr": 0, 2024-04-02T14:51:17.845 INFO:teuthology.orchestra.run.smithi092.stdout: "omap_rd": 2, 2024-04-02T14:51:17.845 INFO:teuthology.orchestra.run.smithi092.stdout: "omap_del": 0 2024-04-02T14:51:17.845 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:17.845 INFO:teuthology.orchestra.run.smithi092.stdout: "oft": { 2024-04-02T14:51:17.845 INFO:teuthology.orchestra.run.smithi092.stdout: "omap_total_objs": 0, 2024-04-02T14:51:17.845 INFO:teuthology.orchestra.run.smithi092.stdout: "omap_total_kv_pairs": 0, 2024-04-02T14:51:17.845 INFO:teuthology.orchestra.run.smithi092.stdout: "omap_total_updates": 0, 2024-04-02T14:51:17.845 INFO:teuthology.orchestra.run.smithi092.stdout: "omap_total_removes": 0 2024-04-02T14:51:17.845 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:17.845 INFO:teuthology.orchestra.run.smithi092.stdout: "osd-slow-ops": { 2024-04-02T14:51:17.845 INFO:teuthology.orchestra.run.smithi092.stdout: "slow_ops_count": 0 2024-04-02T14:51:17.845 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:17.845 INFO:teuthology.orchestra.run.smithi092.stdout: "purge_queue": { 2024-04-02T14:51:17.845 INFO:teuthology.orchestra.run.smithi092.stdout: "pq_executing_ops": 0, 2024-04-02T14:51:17.846 INFO:teuthology.orchestra.run.smithi092.stdout: "pq_executing_ops_high_water": 0, 2024-04-02T14:51:17.846 INFO:teuthology.orchestra.run.smithi092.stdout: "pq_executing": 0, 2024-04-02T14:51:17.846 INFO:teuthology.orchestra.run.smithi092.stdout: "pq_executing_high_water": 0, 2024-04-02T14:51:17.846 INFO:teuthology.orchestra.run.smithi092.stdout: "pq_executed_ops": 0, 2024-04-02T14:51:17.846 INFO:teuthology.orchestra.run.smithi092.stdout: "pq_executed": 0, 2024-04-02T14:51:17.846 INFO:teuthology.orchestra.run.smithi092.stdout: "pq_item_in_journal": 0 2024-04-02T14:51:17.846 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:17.846 INFO:teuthology.orchestra.run.smithi092.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-02T14:51:17.846 INFO:teuthology.orchestra.run.smithi092.stdout: "val": 0, 2024-04-02T14:51:17.846 INFO:teuthology.orchestra.run.smithi092.stdout: "max": 104857600, 2024-04-02T14:51:17.846 INFO:teuthology.orchestra.run.smithi092.stdout: "get_started": 0, 2024-04-02T14:51:17.846 INFO:teuthology.orchestra.run.smithi092.stdout: "get": 3596, 2024-04-02T14:51:17.846 INFO:teuthology.orchestra.run.smithi092.stdout: "get_sum": 2234323, 2024-04-02T14:51:17.846 INFO:teuthology.orchestra.run.smithi092.stdout: "get_or_fail_fail": 0, 2024-04-02T14:51:17.846 INFO:teuthology.orchestra.run.smithi092.stdout: "get_or_fail_success": 3596, 2024-04-02T14:51:17.847 INFO:teuthology.orchestra.run.smithi092.stdout: "take": 0, 2024-04-02T14:51:17.847 INFO:teuthology.orchestra.run.smithi092.stdout: "take_sum": 0, 2024-04-02T14:51:17.847 INFO:teuthology.orchestra.run.smithi092.stdout: "put": 3596, 2024-04-02T14:51:17.847 INFO:teuthology.orchestra.run.smithi092.stdout: "put_sum": 2234323, 2024-04-02T14:51:17.847 INFO:teuthology.orchestra.run.smithi092.stdout: "wait": { 2024-04-02T14:51:17.847 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2024-04-02T14:51:17.847 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2024-04-02T14:51:17.847 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2024-04-02T14:51:17.847 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-02T14:51:17.847 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:17.847 INFO:teuthology.orchestra.run.smithi092.stdout: "throttle-objecter_bytes": { 2024-04-02T14:51:17.847 INFO:teuthology.orchestra.run.smithi092.stdout: "val": 0, 2024-04-02T14:51:17.847 INFO:teuthology.orchestra.run.smithi092.stdout: "max": 104857600, 2024-04-02T14:51:17.847 INFO:teuthology.orchestra.run.smithi092.stdout: "get_started": 0, 2024-04-02T14:51:17.847 INFO:teuthology.orchestra.run.smithi092.stdout: "get": 0, 2024-04-02T14:51:17.847 INFO:teuthology.orchestra.run.smithi092.stdout: "get_sum": 0, 2024-04-02T14:51:17.847 INFO:teuthology.orchestra.run.smithi092.stdout: "get_or_fail_fail": 0, 2024-04-02T14:51:17.848 INFO:teuthology.orchestra.run.smithi092.stdout: "get_or_fail_success": 0, 2024-04-02T14:51:17.848 INFO:teuthology.orchestra.run.smithi092.stdout: "take": 3243, 2024-04-02T14:51:17.848 INFO:teuthology.orchestra.run.smithi092.stdout: "take_sum": 2354, 2024-04-02T14:51:17.848 INFO:teuthology.orchestra.run.smithi092.stdout: "put": 3243, 2024-04-02T14:51:17.848 INFO:teuthology.orchestra.run.smithi092.stdout: "put_sum": 2354, 2024-04-02T14:51:17.848 INFO:teuthology.orchestra.run.smithi092.stdout: "wait": { 2024-04-02T14:51:17.848 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2024-04-02T14:51:17.848 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2024-04-02T14:51:17.848 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2024-04-02T14:51:17.848 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-02T14:51:17.848 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:17.848 INFO:teuthology.orchestra.run.smithi092.stdout: "throttle-objecter_ops": { 2024-04-02T14:51:17.848 INFO:teuthology.orchestra.run.smithi092.stdout: "val": 0, 2024-04-02T14:51:17.848 INFO:teuthology.orchestra.run.smithi092.stdout: "max": 1024, 2024-04-02T14:51:17.848 INFO:teuthology.orchestra.run.smithi092.stdout: "get_started": 0, 2024-04-02T14:51:17.848 INFO:teuthology.orchestra.run.smithi092.stdout: "get": 0, 2024-04-02T14:51:17.848 INFO:teuthology.orchestra.run.smithi092.stdout: "get_sum": 0, 2024-04-02T14:51:17.849 INFO:teuthology.orchestra.run.smithi092.stdout: "get_or_fail_fail": 0, 2024-04-02T14:51:17.849 INFO:teuthology.orchestra.run.smithi092.stdout: "get_or_fail_success": 0, 2024-04-02T14:51:17.849 INFO:teuthology.orchestra.run.smithi092.stdout: "take": 3243, 2024-04-02T14:51:17.849 INFO:teuthology.orchestra.run.smithi092.stdout: "take_sum": 3243, 2024-04-02T14:51:17.849 INFO:teuthology.orchestra.run.smithi092.stdout: "put": 3243, 2024-04-02T14:51:17.849 INFO:teuthology.orchestra.run.smithi092.stdout: "put_sum": 3243, 2024-04-02T14:51:17.849 INFO:teuthology.orchestra.run.smithi092.stdout: "wait": { 2024-04-02T14:51:17.849 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2024-04-02T14:51:17.849 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2024-04-02T14:51:17.849 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2024-04-02T14:51:17.849 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-02T14:51:17.849 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:17.849 INFO:teuthology.orchestra.run.smithi092.stdout: "throttle-write_buf_throttle": { 2024-04-02T14:51:17.849 INFO:teuthology.orchestra.run.smithi092.stdout: "val": 0, 2024-04-02T14:51:17.849 INFO:teuthology.orchestra.run.smithi092.stdout: "max": 3758096384, 2024-04-02T14:51:17.849 INFO:teuthology.orchestra.run.smithi092.stdout: "get_started": 0, 2024-04-02T14:51:17.849 INFO:teuthology.orchestra.run.smithi092.stdout: "get": 0, 2024-04-02T14:51:17.850 INFO:teuthology.orchestra.run.smithi092.stdout: "get_sum": 0, 2024-04-02T14:51:17.850 INFO:teuthology.orchestra.run.smithi092.stdout: "get_or_fail_fail": 0, 2024-04-02T14:51:17.850 INFO:teuthology.orchestra.run.smithi092.stdout: "get_or_fail_success": 0, 2024-04-02T14:51:17.850 INFO:teuthology.orchestra.run.smithi092.stdout: "take": 0, 2024-04-02T14:51:17.850 INFO:teuthology.orchestra.run.smithi092.stdout: "take_sum": 0, 2024-04-02T14:51:17.850 INFO:teuthology.orchestra.run.smithi092.stdout: "put": 0, 2024-04-02T14:51:17.850 INFO:teuthology.orchestra.run.smithi092.stdout: "put_sum": 0, 2024-04-02T14:51:17.850 INFO:teuthology.orchestra.run.smithi092.stdout: "wait": { 2024-04-02T14:51:17.850 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2024-04-02T14:51:17.850 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2024-04-02T14:51:17.850 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2024-04-02T14:51:17.850 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-02T14:51:17.850 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-02T14:51:17.850 INFO:teuthology.orchestra.run.smithi092.stdout: "throttle-write_buf_throttle-0x55fb74f786e0": { 2024-04-02T14:51:17.850 INFO:teuthology.orchestra.run.smithi092.stdout: "val": 0, 2024-04-02T14:51:17.851 INFO:teuthology.orchestra.run.smithi092.stdout: "max": 3758096384, 2024-04-02T14:51:17.851 INFO:teuthology.orchestra.run.smithi092.stdout: "get_started": 0, 2024-04-02T14:51:17.851 INFO:teuthology.orchestra.run.smithi092.stdout: "get": 0, 2024-04-02T14:51:17.851 INFO:teuthology.orchestra.run.smithi092.stdout: "get_sum": 0, 2024-04-02T14:51:17.851 INFO:teuthology.orchestra.run.smithi092.stdout: "get_or_fail_fail": 0, 2024-04-02T14:51:17.851 INFO:teuthology.orchestra.run.smithi092.stdout: "get_or_fail_success": 0, 2024-04-02T14:51:17.851 INFO:teuthology.orchestra.run.smithi092.stdout: "take": 0, 2024-04-02T14:51:17.851 INFO:teuthology.orchestra.run.smithi092.stdout: "take_sum": 0, 2024-04-02T14:51:17.851 INFO:teuthology.orchestra.run.smithi092.stdout: "put": 0, 2024-04-02T14:51:17.851 INFO:teuthology.orchestra.run.smithi092.stdout: "put_sum": 0, 2024-04-02T14:51:17.851 INFO:teuthology.orchestra.run.smithi092.stdout: "wait": { 2024-04-02T14:51:17.851 INFO:teuthology.orchestra.run.smithi092.stdout: "avgcount": 0, 2024-04-02T14:51:17.851 INFO:teuthology.orchestra.run.smithi092.stdout: "sum": 0.000000000, 2024-04-02T14:51:17.851 INFO:teuthology.orchestra.run.smithi092.stdout: "avgtime": 0.000000000 2024-04-02T14:51:17.851 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-02T14:51:17.851 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-02T14:51:17.852 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-04-02T14:51:18.101 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:51:17 smithi092 ceph-mon[36497]: pgmap v514: 97 pgs: 97 active+clean; 3.9 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 18 KiB/s wr, 11 op/s 2024-04-02T14:51:18.178 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:51:17 smithi177 ceph-mon[36431]: pgmap v514: 97 pgs: 97 active+clean; 3.9 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 18 KiB/s wr, 11 op/s 2024-04-02T14:51:18.179 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:51:17 smithi090 ceph-mon[31674]: pgmap v514: 97 pgs: 97 active+clean; 3.9 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 18 KiB/s wr, 11 op/s 2024-04-02T14:51:18.637 INFO:tasks.check_counter:Daemon mds.k mds.exported=0 2024-04-02T14:51:18.637 INFO:tasks.check_counter:Daemon mds.k mds.imported=0 2024-04-02T14:51:18.637 INFO:tasks.check_counter:Daemon mds.k mds_cache.dir_update=0 2024-04-02T14:51:18.637 INFO:tasks.check_counter:Daemon mds.k mds_cache.dir_update_receipt=0 2024-04-02T14:51:18.637 INFO:tasks.check_counter:Daemon mds.k mds.root_rsnaps=0 2024-04-02T14:51:18.637 INFO:tasks.check_counter:Daemon mds.k mds_server.req_mksnap_latency.avgcount=0 2024-04-02T14:51:18.637 INFO:tasks.check_counter:Daemon mds.k mds_server.req_rmsnap_latency.avgcount=0 2024-04-02T14:51:18.637 DEBUG:tasks.check_counter:Getting stats from c 2024-04-02T14:51:18.637 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:191d963ac9de7322da8aff20b567b3fa458e2ef6 shell --fsid 6449448a-f0fc-11ee-b647-cb9ed24678a4 -- ceph daemon mds.c perf dump 2024-04-02T14:51:18.888 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/mon.c/config 2024-04-02T14:51:20.102 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:51:19 smithi092 ceph-mon[36497]: pgmap v515: 97 pgs: 97 active+clean; 3.9 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 15 KiB/s wr, 9 op/s 2024-04-02T14:51:20.124 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:51:19 smithi177 ceph-mon[36431]: pgmap v515: 97 pgs: 97 active+clean; 3.9 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 15 KiB/s wr, 9 op/s 2024-04-02T14:51:20.179 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:51:19 smithi090 ceph-mon[31674]: pgmap v515: 97 pgs: 97 active+clean; 3.9 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 15 KiB/s wr, 9 op/s 2024-04-02T14:51:20.682 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-04-02T14:51:20.682 INFO:teuthology.orchestra.run.smithi177.stdout: "AsyncMessenger::Worker-0": { 2024-04-02T14:51:20.682 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_messages": 31, 2024-04-02T14:51:20.682 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_messages": 89, 2024-04-02T14:51:20.682 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_bytes": 9110, 2024-04-02T14:51:20.682 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_bytes": 69900, 2024-04-02T14:51:20.683 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_created_connections": 14, 2024-04-02T14:51:20.683 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_active_connections": 4, 2024-04-02T14:51:20.683 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_total_time": 0.018981134, 2024-04-02T14:51:20.683 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_send_time": 0.003681994, 2024-04-02T14:51:20.683 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_recv_time": 0.009276884, 2024-04-02T14:51:20.683 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_fast_dispatch_time": 0.000319723, 2024-04-02T14:51:20.683 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_messages_queue_lat": { 2024-04-02T14:51:20.683 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 83, 2024-04-02T14:51:20.683 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.014091105, 2024-04-02T14:51:20.683 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000169772 2024-04-02T14:51:20.683 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:20.684 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_handle_ack_lat": { 2024-04-02T14:51:20.684 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 65, 2024-04-02T14:51:20.684 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000014217, 2024-04-02T14:51:20.684 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000218 2024-04-02T14:51:20.684 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:20.684 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_encrypted_bytes": 5088, 2024-04-02T14:51:20.684 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_encrypted_bytes": 15792 2024-04-02T14:51:20.684 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:20.684 INFO:teuthology.orchestra.run.smithi177.stdout: "AsyncMessenger::Worker-1": { 2024-04-02T14:51:20.684 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_messages": 4136, 2024-04-02T14:51:20.684 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_messages": 2235, 2024-04-02T14:51:20.684 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_bytes": 2682331, 2024-04-02T14:51:20.685 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_bytes": 742516, 2024-04-02T14:51:20.685 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_created_connections": 9, 2024-04-02T14:51:20.685 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_active_connections": 4, 2024-04-02T14:51:20.685 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_total_time": 0.212117147, 2024-04-02T14:51:20.685 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_send_time": 0.076921753, 2024-04-02T14:51:20.685 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_recv_time": 0.162060299, 2024-04-02T14:51:20.685 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_fast_dispatch_time": 0.012963555, 2024-04-02T14:51:20.685 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_messages_queue_lat": { 2024-04-02T14:51:20.685 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 2238, 2024-04-02T14:51:20.685 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.090778813, 2024-04-02T14:51:20.685 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000040562 2024-04-02T14:51:20.685 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:20.685 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_handle_ack_lat": { 2024-04-02T14:51:20.685 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 5315, 2024-04-02T14:51:20.685 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000806161, 2024-04-02T14:51:20.686 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000151 2024-04-02T14:51:20.686 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:20.686 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_encrypted_bytes": 1663824, 2024-04-02T14:51:20.686 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_encrypted_bytes": 200800 2024-04-02T14:51:20.686 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:20.686 INFO:teuthology.orchestra.run.smithi177.stdout: "AsyncMessenger::Worker-2": { 2024-04-02T14:51:20.686 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_messages": 78, 2024-04-02T14:51:20.686 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_messages": 310, 2024-04-02T14:51:20.686 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_bytes": 15857, 2024-04-02T14:51:20.686 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_bytes": 1244268, 2024-04-02T14:51:20.686 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_created_connections": 66, 2024-04-02T14:51:20.686 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_active_connections": 8, 2024-04-02T14:51:20.686 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_total_time": 0.032765002, 2024-04-02T14:51:20.687 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_send_time": 0.008750342, 2024-04-02T14:51:20.687 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_recv_time": 0.019895521, 2024-04-02T14:51:20.687 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_fast_dispatch_time": 0.000553179, 2024-04-02T14:51:20.687 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_messages_queue_lat": { 2024-04-02T14:51:20.687 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 308, 2024-04-02T14:51:20.687 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.029690371, 2024-04-02T14:51:20.687 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000096397 2024-04-02T14:51:20.687 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:20.687 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_handle_ack_lat": { 2024-04-02T14:51:20.687 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 56, 2024-04-02T14:51:20.687 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000007106, 2024-04-02T14:51:20.687 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000126 2024-04-02T14:51:20.688 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:20.688 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_encrypted_bytes": 3392, 2024-04-02T14:51:20.688 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_encrypted_bytes": 579872 2024-04-02T14:51:20.688 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:20.688 INFO:teuthology.orchestra.run.smithi177.stdout: "cct": { 2024-04-02T14:51:20.688 INFO:teuthology.orchestra.run.smithi177.stdout: "total_workers": 1, 2024-04-02T14:51:20.688 INFO:teuthology.orchestra.run.smithi177.stdout: "unhealthy_workers": 0 2024-04-02T14:51:20.688 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:20.688 INFO:teuthology.orchestra.run.smithi177.stdout: "finisher-MDSRank": { 2024-04-02T14:51:20.688 INFO:teuthology.orchestra.run.smithi177.stdout: "queue_len": 0, 2024-04-02T14:51:20.688 INFO:teuthology.orchestra.run.smithi177.stdout: "complete_latency": { 2024-04-02T14:51:20.688 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 40, 2024-04-02T14:51:20.688 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.001448465, 2024-04-02T14:51:20.688 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000036211 2024-04-02T14:51:20.689 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-04-02T14:51:20.689 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:20.689 INFO:teuthology.orchestra.run.smithi177.stdout: "finisher-PurgeQueue": { 2024-04-02T14:51:20.689 INFO:teuthology.orchestra.run.smithi177.stdout: "queue_len": 0, 2024-04-02T14:51:20.689 INFO:teuthology.orchestra.run.smithi177.stdout: "complete_latency": { 2024-04-02T14:51:20.689 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 2, 2024-04-02T14:51:20.689 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000007629, 2024-04-02T14:51:20.689 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000003814 2024-04-02T14:51:20.689 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-04-02T14:51:20.689 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:20.689 INFO:teuthology.orchestra.run.smithi177.stdout: "mds": { 2024-04-02T14:51:20.689 INFO:teuthology.orchestra.run.smithi177.stdout: "request": 1, 2024-04-02T14:51:20.689 INFO:teuthology.orchestra.run.smithi177.stdout: "reply": 1, 2024-04-02T14:51:20.689 INFO:teuthology.orchestra.run.smithi177.stdout: "reply_latency": { 2024-04-02T14:51:20.689 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 1, 2024-04-02T14:51:20.689 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000160002, 2024-04-02T14:51:20.690 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000160002 2024-04-02T14:51:20.690 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:20.690 INFO:teuthology.orchestra.run.smithi177.stdout: "slow_reply": 0, 2024-04-02T14:51:20.690 INFO:teuthology.orchestra.run.smithi177.stdout: "forward": 0, 2024-04-02T14:51:20.690 INFO:teuthology.orchestra.run.smithi177.stdout: "dir_fetch_complete": 0, 2024-04-02T14:51:20.690 INFO:teuthology.orchestra.run.smithi177.stdout: "dir_fetch_keys": 0, 2024-04-02T14:51:20.690 INFO:teuthology.orchestra.run.smithi177.stdout: "dir_commit": 11, 2024-04-02T14:51:20.690 INFO:teuthology.orchestra.run.smithi177.stdout: "dir_split": 0, 2024-04-02T14:51:20.690 INFO:teuthology.orchestra.run.smithi177.stdout: "dir_merge": 0, 2024-04-02T14:51:20.690 INFO:teuthology.orchestra.run.smithi177.stdout: "inodes": 33, 2024-04-02T14:51:20.690 INFO:teuthology.orchestra.run.smithi177.stdout: "inodes_top": 10, 2024-04-02T14:51:20.690 INFO:teuthology.orchestra.run.smithi177.stdout: "inodes_bottom": 23, 2024-04-02T14:51:20.690 INFO:teuthology.orchestra.run.smithi177.stdout: "inodes_pin_tail": 0, 2024-04-02T14:51:20.690 INFO:teuthology.orchestra.run.smithi177.stdout: "inodes_pinned": 18, 2024-04-02T14:51:20.691 INFO:teuthology.orchestra.run.smithi177.stdout: "inodes_expired": 4, 2024-04-02T14:51:20.691 INFO:teuthology.orchestra.run.smithi177.stdout: "inodes_with_caps": 1, 2024-04-02T14:51:20.691 INFO:teuthology.orchestra.run.smithi177.stdout: "caps": 1, 2024-04-02T14:51:20.691 INFO:teuthology.orchestra.run.smithi177.stdout: "subtrees": 3, 2024-04-02T14:51:20.691 INFO:teuthology.orchestra.run.smithi177.stdout: "traverse": 60, 2024-04-02T14:51:20.691 INFO:teuthology.orchestra.run.smithi177.stdout: "traverse_hit": 40, 2024-04-02T14:51:20.691 INFO:teuthology.orchestra.run.smithi177.stdout: "traverse_forward": 0, 2024-04-02T14:51:20.691 INFO:teuthology.orchestra.run.smithi177.stdout: "traverse_discover": 20, 2024-04-02T14:51:20.691 INFO:teuthology.orchestra.run.smithi177.stdout: "traverse_dir_fetch": 0, 2024-04-02T14:51:20.691 INFO:teuthology.orchestra.run.smithi177.stdout: "traverse_remote_ino": 0, 2024-04-02T14:51:20.691 INFO:teuthology.orchestra.run.smithi177.stdout: "traverse_lock": 0, 2024-04-02T14:51:20.691 INFO:teuthology.orchestra.run.smithi177.stdout: "load_cent": 0, 2024-04-02T14:51:20.691 INFO:teuthology.orchestra.run.smithi177.stdout: "q": 0, 2024-04-02T14:51:20.691 INFO:teuthology.orchestra.run.smithi177.stdout: "exported": 0, 2024-04-02T14:51:20.692 INFO:teuthology.orchestra.run.smithi177.stdout: "exported_inodes": 0, 2024-04-02T14:51:20.692 INFO:teuthology.orchestra.run.smithi177.stdout: "imported": 0, 2024-04-02T14:51:20.692 INFO:teuthology.orchestra.run.smithi177.stdout: "imported_inodes": 0, 2024-04-02T14:51:20.692 INFO:teuthology.orchestra.run.smithi177.stdout: "openino_dir_fetch": 0, 2024-04-02T14:51:20.692 INFO:teuthology.orchestra.run.smithi177.stdout: "openino_backtrace_fetch": 0, 2024-04-02T14:51:20.692 INFO:teuthology.orchestra.run.smithi177.stdout: "openino_peer_discover": 0, 2024-04-02T14:51:20.692 INFO:teuthology.orchestra.run.smithi177.stdout: "root_rfiles": 1026, 2024-04-02T14:51:20.692 INFO:teuthology.orchestra.run.smithi177.stdout: "root_rbytes": 1709677761, 2024-04-02T14:51:20.692 INFO:teuthology.orchestra.run.smithi177.stdout: "root_rsnaps": 6, 2024-04-02T14:51:20.692 INFO:teuthology.orchestra.run.smithi177.stdout: "scrub_backtrace_fetch": 0, 2024-04-02T14:51:20.692 INFO:teuthology.orchestra.run.smithi177.stdout: "scrub_set_tag": 0, 2024-04-02T14:51:20.692 INFO:teuthology.orchestra.run.smithi177.stdout: "scrub_backtrace_repaired": 0, 2024-04-02T14:51:20.692 INFO:teuthology.orchestra.run.smithi177.stdout: "scrub_inotable_repaired": 0, 2024-04-02T14:51:20.692 INFO:teuthology.orchestra.run.smithi177.stdout: "scrub_dir_inodes": 0, 2024-04-02T14:51:20.693 INFO:teuthology.orchestra.run.smithi177.stdout: "scrub_dir_base_inodes": 0, 2024-04-02T14:51:20.693 INFO:teuthology.orchestra.run.smithi177.stdout: "scrub_dirfrag_rstats": 0, 2024-04-02T14:51:20.693 INFO:teuthology.orchestra.run.smithi177.stdout: "scrub_file_inodes": 0, 2024-04-02T14:51:20.693 INFO:teuthology.orchestra.run.smithi177.stdout: "handle_inode_file_caps": 0, 2024-04-02T14:51:20.693 INFO:teuthology.orchestra.run.smithi177.stdout: "ceph_cap_op_revoke": 0, 2024-04-02T14:51:20.693 INFO:teuthology.orchestra.run.smithi177.stdout: "ceph_cap_op_grant": 0, 2024-04-02T14:51:20.693 INFO:teuthology.orchestra.run.smithi177.stdout: "ceph_cap_op_trunc": 0, 2024-04-02T14:51:20.693 INFO:teuthology.orchestra.run.smithi177.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-02T14:51:20.693 INFO:teuthology.orchestra.run.smithi177.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-02T14:51:20.693 INFO:teuthology.orchestra.run.smithi177.stdout: "handle_client_caps": 0, 2024-04-02T14:51:20.693 INFO:teuthology.orchestra.run.smithi177.stdout: "handle_client_caps_dirty": 0, 2024-04-02T14:51:20.693 INFO:teuthology.orchestra.run.smithi177.stdout: "handle_client_cap_release": 0, 2024-04-02T14:51:20.693 INFO:teuthology.orchestra.run.smithi177.stdout: "process_request_cap_release": 0 2024-04-02T14:51:20.693 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:20.694 INFO:teuthology.orchestra.run.smithi177.stdout: "mds_cache": { 2024-04-02T14:51:20.694 INFO:teuthology.orchestra.run.smithi177.stdout: "dir_update": 0, 2024-04-02T14:51:20.694 INFO:teuthology.orchestra.run.smithi177.stdout: "dir_update_receipt": 100, 2024-04-02T14:51:20.694 INFO:teuthology.orchestra.run.smithi177.stdout: "dir_try_discover": 60, 2024-04-02T14:51:20.694 INFO:teuthology.orchestra.run.smithi177.stdout: "dir_send_discover": 60, 2024-04-02T14:51:20.694 INFO:teuthology.orchestra.run.smithi177.stdout: "dir_handle_discover": 0, 2024-04-02T14:51:20.694 INFO:teuthology.orchestra.run.smithi177.stdout: "num_strays": 0, 2024-04-02T14:51:20.694 INFO:teuthology.orchestra.run.smithi177.stdout: "num_strays_delayed": 0, 2024-04-02T14:51:20.694 INFO:teuthology.orchestra.run.smithi177.stdout: "num_strays_enqueuing": 0, 2024-04-02T14:51:20.694 INFO:teuthology.orchestra.run.smithi177.stdout: "strays_created": 0, 2024-04-02T14:51:20.694 INFO:teuthology.orchestra.run.smithi177.stdout: "strays_enqueued": 0, 2024-04-02T14:51:20.694 INFO:teuthology.orchestra.run.smithi177.stdout: "strays_reintegrated": 0, 2024-04-02T14:51:20.694 INFO:teuthology.orchestra.run.smithi177.stdout: "strays_migrated": 0, 2024-04-02T14:51:20.694 INFO:teuthology.orchestra.run.smithi177.stdout: "num_recovering_processing": 0, 2024-04-02T14:51:20.695 INFO:teuthology.orchestra.run.smithi177.stdout: "num_recovering_enqueued": 0, 2024-04-02T14:51:20.695 INFO:teuthology.orchestra.run.smithi177.stdout: "num_recovering_prioritized": 0, 2024-04-02T14:51:20.695 INFO:teuthology.orchestra.run.smithi177.stdout: "recovery_started": 0, 2024-04-02T14:51:20.695 INFO:teuthology.orchestra.run.smithi177.stdout: "recovery_completed": 0, 2024-04-02T14:51:20.695 INFO:teuthology.orchestra.run.smithi177.stdout: "ireq_quiesce_path": 0, 2024-04-02T14:51:20.695 INFO:teuthology.orchestra.run.smithi177.stdout: "ireq_quiesce_inode": 0, 2024-04-02T14:51:20.695 INFO:teuthology.orchestra.run.smithi177.stdout: "ireq_enqueue_scrub": 0, 2024-04-02T14:51:20.695 INFO:teuthology.orchestra.run.smithi177.stdout: "ireq_exportdir": 0, 2024-04-02T14:51:20.695 INFO:teuthology.orchestra.run.smithi177.stdout: "ireq_flush": 0, 2024-04-02T14:51:20.695 INFO:teuthology.orchestra.run.smithi177.stdout: "ireq_fragmentdir": 0, 2024-04-02T14:51:20.695 INFO:teuthology.orchestra.run.smithi177.stdout: "ireq_fragstats": 0, 2024-04-02T14:51:20.695 INFO:teuthology.orchestra.run.smithi177.stdout: "ireq_inodestats": 0 2024-04-02T14:51:20.695 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:20.695 INFO:teuthology.orchestra.run.smithi177.stdout: "mds_log": { 2024-04-02T14:51:20.696 INFO:teuthology.orchestra.run.smithi177.stdout: "evlrg": 0, 2024-04-02T14:51:20.696 INFO:teuthology.orchestra.run.smithi177.stdout: "evadd": 3, 2024-04-02T14:51:20.696 INFO:teuthology.orchestra.run.smithi177.stdout: "evex": 0, 2024-04-02T14:51:20.696 INFO:teuthology.orchestra.run.smithi177.stdout: "evtrm": 0, 2024-04-02T14:51:20.696 INFO:teuthology.orchestra.run.smithi177.stdout: "ev": 3, 2024-04-02T14:51:20.696 INFO:teuthology.orchestra.run.smithi177.stdout: "evexg": 0, 2024-04-02T14:51:20.696 INFO:teuthology.orchestra.run.smithi177.stdout: "evexd": 0, 2024-04-02T14:51:20.696 INFO:teuthology.orchestra.run.smithi177.stdout: "segadd": 2, 2024-04-02T14:51:20.696 INFO:teuthology.orchestra.run.smithi177.stdout: "segex": 0, 2024-04-02T14:51:20.696 INFO:teuthology.orchestra.run.smithi177.stdout: "segtrm": 0, 2024-04-02T14:51:20.696 INFO:teuthology.orchestra.run.smithi177.stdout: "seg": 2, 2024-04-02T14:51:20.696 INFO:teuthology.orchestra.run.smithi177.stdout: "segmjr": 2, 2024-04-02T14:51:20.696 INFO:teuthology.orchestra.run.smithi177.stdout: "segexg": 0, 2024-04-02T14:51:20.696 INFO:teuthology.orchestra.run.smithi177.stdout: "segexd": 0, 2024-04-02T14:51:20.696 INFO:teuthology.orchestra.run.smithi177.stdout: "expos": 4194304, 2024-04-02T14:51:20.697 INFO:teuthology.orchestra.run.smithi177.stdout: "wrpos": 4195198, 2024-04-02T14:51:20.697 INFO:teuthology.orchestra.run.smithi177.stdout: "rdpos": 0, 2024-04-02T14:51:20.697 INFO:teuthology.orchestra.run.smithi177.stdout: "jlat": { 2024-04-02T14:51:20.697 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 2, 2024-04-02T14:51:20.697 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.004458293, 2024-04-02T14:51:20.697 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.002229146 2024-04-02T14:51:20.697 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:20.697 INFO:teuthology.orchestra.run.smithi177.stdout: "replayed": 0 2024-04-02T14:51:20.697 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:20.697 INFO:teuthology.orchestra.run.smithi177.stdout: "mds_mem": { 2024-04-02T14:51:20.697 INFO:teuthology.orchestra.run.smithi177.stdout: "ino": 37, 2024-04-02T14:51:20.697 INFO:teuthology.orchestra.run.smithi177.stdout: "ino+": 39, 2024-04-02T14:51:20.697 INFO:teuthology.orchestra.run.smithi177.stdout: "ino-": 2, 2024-04-02T14:51:20.697 INFO:teuthology.orchestra.run.smithi177.stdout: "dir": 64, 2024-04-02T14:51:20.697 INFO:teuthology.orchestra.run.smithi177.stdout: "dir+": 67, 2024-04-02T14:51:20.698 INFO:teuthology.orchestra.run.smithi177.stdout: "dir-": 3, 2024-04-02T14:51:20.698 INFO:teuthology.orchestra.run.smithi177.stdout: "dn": 33, 2024-04-02T14:51:20.698 INFO:teuthology.orchestra.run.smithi177.stdout: "dn+": 37, 2024-04-02T14:51:20.698 INFO:teuthology.orchestra.run.smithi177.stdout: "dn-": 4, 2024-04-02T14:51:20.698 INFO:teuthology.orchestra.run.smithi177.stdout: "cap": 1, 2024-04-02T14:51:20.698 INFO:teuthology.orchestra.run.smithi177.stdout: "cap+": 1, 2024-04-02T14:51:20.698 INFO:teuthology.orchestra.run.smithi177.stdout: "cap-": 0, 2024-04-02T14:51:20.698 INFO:teuthology.orchestra.run.smithi177.stdout: "rss": 53548, 2024-04-02T14:51:20.698 INFO:teuthology.orchestra.run.smithi177.stdout: "heap": 223608 2024-04-02T14:51:20.698 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:20.698 INFO:teuthology.orchestra.run.smithi177.stdout: "mds_server": { 2024-04-02T14:51:20.698 INFO:teuthology.orchestra.run.smithi177.stdout: "dispatch_client_request": 1, 2024-04-02T14:51:20.698 INFO:teuthology.orchestra.run.smithi177.stdout: "dispatch_server_request": 0, 2024-04-02T14:51:20.699 INFO:teuthology.orchestra.run.smithi177.stdout: "handle_client_request": 1, 2024-04-02T14:51:20.699 INFO:teuthology.orchestra.run.smithi177.stdout: "handle_client_session": 13, 2024-04-02T14:51:20.699 INFO:teuthology.orchestra.run.smithi177.stdout: "handle_peer_request": 0, 2024-04-02T14:51:20.699 INFO:teuthology.orchestra.run.smithi177.stdout: "req_create_latency": { 2024-04-02T14:51:20.699 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-02T14:51:20.699 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-02T14:51:20.699 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-02T14:51:20.699 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:20.699 INFO:teuthology.orchestra.run.smithi177.stdout: "req_getattr_latency": { 2024-04-02T14:51:20.699 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-02T14:51:20.699 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-02T14:51:20.699 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-02T14:51:20.699 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:20.699 INFO:teuthology.orchestra.run.smithi177.stdout: "req_getfilelock_latency": { 2024-04-02T14:51:20.699 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-02T14:51:20.700 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-02T14:51:20.700 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-02T14:51:20.700 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:20.700 INFO:teuthology.orchestra.run.smithi177.stdout: "req_link_latency": { 2024-04-02T14:51:20.700 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-02T14:51:20.700 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-02T14:51:20.700 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-02T14:51:20.700 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:20.700 INFO:teuthology.orchestra.run.smithi177.stdout: "req_lookup_latency": { 2024-04-02T14:51:20.700 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-02T14:51:20.700 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-02T14:51:20.700 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-02T14:51:20.700 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:20.700 INFO:teuthology.orchestra.run.smithi177.stdout: "req_lookuphash_latency": { 2024-04-02T14:51:20.700 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-02T14:51:20.700 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-02T14:51:20.700 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-02T14:51:20.701 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:20.701 INFO:teuthology.orchestra.run.smithi177.stdout: "req_lookupino_latency": { 2024-04-02T14:51:20.701 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 1, 2024-04-02T14:51:20.701 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000160002, 2024-04-02T14:51:20.701 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000160002 2024-04-02T14:51:20.701 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:20.701 INFO:teuthology.orchestra.run.smithi177.stdout: "req_lookupname_latency": { 2024-04-02T14:51:20.701 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-02T14:51:20.701 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-02T14:51:20.701 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-02T14:51:20.701 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:20.701 INFO:teuthology.orchestra.run.smithi177.stdout: "req_lookupparent_latency": { 2024-04-02T14:51:20.701 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-02T14:51:20.701 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-02T14:51:20.701 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-02T14:51:20.701 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:20.702 INFO:teuthology.orchestra.run.smithi177.stdout: "req_lookupsnap_latency": { 2024-04-02T14:51:20.702 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-02T14:51:20.702 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-02T14:51:20.702 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-02T14:51:20.702 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:20.702 INFO:teuthology.orchestra.run.smithi177.stdout: "req_lssnap_latency": { 2024-04-02T14:51:20.702 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-02T14:51:20.702 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-02T14:51:20.702 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-02T14:51:20.702 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:20.702 INFO:teuthology.orchestra.run.smithi177.stdout: "req_mkdir_latency": { 2024-04-02T14:51:20.702 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-02T14:51:20.702 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-02T14:51:20.702 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-02T14:51:20.703 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:20.703 INFO:teuthology.orchestra.run.smithi177.stdout: "req_mknod_latency": { 2024-04-02T14:51:20.703 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-02T14:51:20.703 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-02T14:51:20.703 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-02T14:51:20.703 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:20.703 INFO:teuthology.orchestra.run.smithi177.stdout: "req_mksnap_latency": { 2024-04-02T14:51:20.703 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-02T14:51:20.703 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-02T14:51:20.703 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-02T14:51:20.703 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:20.703 INFO:teuthology.orchestra.run.smithi177.stdout: "req_open_latency": { 2024-04-02T14:51:20.703 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-02T14:51:20.703 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-02T14:51:20.703 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-02T14:51:20.703 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:20.703 INFO:teuthology.orchestra.run.smithi177.stdout: "req_readdir_latency": { 2024-04-02T14:51:20.704 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-02T14:51:20.704 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-02T14:51:20.704 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-02T14:51:20.704 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:20.704 INFO:teuthology.orchestra.run.smithi177.stdout: "req_rename_latency": { 2024-04-02T14:51:20.704 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-02T14:51:20.704 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-02T14:51:20.704 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-02T14:51:20.704 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:20.704 INFO:teuthology.orchestra.run.smithi177.stdout: "req_renamesnap_latency": { 2024-04-02T14:51:20.704 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-02T14:51:20.704 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-02T14:51:20.704 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-02T14:51:20.704 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:20.705 INFO:teuthology.orchestra.run.smithi177.stdout: "req_snapdiff_latency": { 2024-04-02T14:51:20.705 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-02T14:51:20.705 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-02T14:51:20.705 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-02T14:51:20.705 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:20.705 INFO:teuthology.orchestra.run.smithi177.stdout: "req_rmdir_latency": { 2024-04-02T14:51:20.705 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-02T14:51:20.705 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-02T14:51:20.705 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-02T14:51:20.705 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:20.705 INFO:teuthology.orchestra.run.smithi177.stdout: "req_rmsnap_latency": { 2024-04-02T14:51:20.705 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-02T14:51:20.705 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-02T14:51:20.705 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-02T14:51:20.706 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:20.706 INFO:teuthology.orchestra.run.smithi177.stdout: "req_rmxattr_latency": { 2024-04-02T14:51:20.706 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-02T14:51:20.706 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-02T14:51:20.706 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-02T14:51:20.706 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:20.706 INFO:teuthology.orchestra.run.smithi177.stdout: "req_setattr_latency": { 2024-04-02T14:51:20.706 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-02T14:51:20.706 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-02T14:51:20.706 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-02T14:51:20.706 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:20.706 INFO:teuthology.orchestra.run.smithi177.stdout: "req_setdirlayout_latency": { 2024-04-02T14:51:20.707 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-02T14:51:20.707 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-02T14:51:20.707 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-02T14:51:20.707 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:20.707 INFO:teuthology.orchestra.run.smithi177.stdout: "req_setfilelock_latency": { 2024-04-02T14:51:20.707 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-02T14:51:20.707 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-02T14:51:20.707 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-02T14:51:20.707 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:20.707 INFO:teuthology.orchestra.run.smithi177.stdout: "req_setlayout_latency": { 2024-04-02T14:51:20.707 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-02T14:51:20.707 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-02T14:51:20.707 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-02T14:51:20.708 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:20.708 INFO:teuthology.orchestra.run.smithi177.stdout: "req_setxattr_latency": { 2024-04-02T14:51:20.708 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-02T14:51:20.708 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-02T14:51:20.708 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-02T14:51:20.708 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:20.708 INFO:teuthology.orchestra.run.smithi177.stdout: "req_symlink_latency": { 2024-04-02T14:51:20.708 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-02T14:51:20.708 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-02T14:51:20.708 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-02T14:51:20.708 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:20.708 INFO:teuthology.orchestra.run.smithi177.stdout: "req_unlink_latency": { 2024-04-02T14:51:20.708 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-02T14:51:20.708 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-02T14:51:20.709 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-02T14:51:20.709 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:20.709 INFO:teuthology.orchestra.run.smithi177.stdout: "cap_revoke_eviction": 0, 2024-04-02T14:51:20.709 INFO:teuthology.orchestra.run.smithi177.stdout: "cap_acquisition_throttle": 0, 2024-04-02T14:51:20.709 INFO:teuthology.orchestra.run.smithi177.stdout: "req_getvxattr_latency": { 2024-04-02T14:51:20.709 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-02T14:51:20.709 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-02T14:51:20.709 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-02T14:51:20.709 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-04-02T14:51:20.709 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:20.709 INFO:teuthology.orchestra.run.smithi177.stdout: "mds_sessions": { 2024-04-02T14:51:20.709 INFO:teuthology.orchestra.run.smithi177.stdout: "session_count": 1, 2024-04-02T14:51:20.709 INFO:teuthology.orchestra.run.smithi177.stdout: "session_add": 1, 2024-04-02T14:51:20.709 INFO:teuthology.orchestra.run.smithi177.stdout: "session_remove": 0, 2024-04-02T14:51:20.710 INFO:teuthology.orchestra.run.smithi177.stdout: "sessions_open": 1, 2024-04-02T14:51:20.710 INFO:teuthology.orchestra.run.smithi177.stdout: "sessions_stale": 0, 2024-04-02T14:51:20.710 INFO:teuthology.orchestra.run.smithi177.stdout: "total_load": 1, 2024-04-02T14:51:20.710 INFO:teuthology.orchestra.run.smithi177.stdout: "average_load": 1, 2024-04-02T14:51:20.710 INFO:teuthology.orchestra.run.smithi177.stdout: "avg_session_uptime": 971, 2024-04-02T14:51:20.710 INFO:teuthology.orchestra.run.smithi177.stdout: "mdthresh_evicted": 0 2024-04-02T14:51:20.710 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:20.710 INFO:teuthology.orchestra.run.smithi177.stdout: "mempool": { 2024-04-02T14:51:20.710 INFO:teuthology.orchestra.run.smithi177.stdout: "bloom_filter_bytes": 0, 2024-04-02T14:51:20.710 INFO:teuthology.orchestra.run.smithi177.stdout: "bloom_filter_items": 0, 2024-04-02T14:51:20.710 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_alloc_bytes": 0, 2024-04-02T14:51:20.710 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_alloc_items": 0, 2024-04-02T14:51:20.710 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_data_bytes": 0, 2024-04-02T14:51:20.711 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_data_items": 0, 2024-04-02T14:51:20.711 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-02T14:51:20.711 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_onode_items": 0, 2024-04-02T14:51:20.711 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-02T14:51:20.711 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_meta_items": 0, 2024-04-02T14:51:20.711 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_other_bytes": 0, 2024-04-02T14:51:20.711 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_other_items": 0, 2024-04-02T14:51:20.711 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-02T14:51:20.711 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_buffer_items": 0, 2024-04-02T14:51:20.711 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_extent_bytes": 0, 2024-04-02T14:51:20.711 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_extent_items": 0, 2024-04-02T14:51:20.711 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_blob_bytes": 0, 2024-04-02T14:51:20.711 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_blob_items": 0, 2024-04-02T14:51:20.711 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-02T14:51:20.711 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_shared_blob_items": 0, 2024-04-02T14:51:20.712 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-02T14:51:20.712 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_inline_bl_items": 0, 2024-04-02T14:51:20.712 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_fsck_bytes": 0, 2024-04-02T14:51:20.712 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_fsck_items": 0, 2024-04-02T14:51:20.712 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_txc_bytes": 0, 2024-04-02T14:51:20.712 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_txc_items": 0, 2024-04-02T14:51:20.712 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-02T14:51:20.712 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_writing_deferred_items": 0, 2024-04-02T14:51:20.712 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_writing_bytes": 0, 2024-04-02T14:51:20.712 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_writing_items": 0, 2024-04-02T14:51:20.712 INFO:teuthology.orchestra.run.smithi177.stdout: "bluefs_bytes": 0, 2024-04-02T14:51:20.712 INFO:teuthology.orchestra.run.smithi177.stdout: "bluefs_items": 0, 2024-04-02T14:51:20.712 INFO:teuthology.orchestra.run.smithi177.stdout: "bluefs_file_reader_bytes": 0, 2024-04-02T14:51:20.712 INFO:teuthology.orchestra.run.smithi177.stdout: "bluefs_file_reader_items": 0, 2024-04-02T14:51:20.712 INFO:teuthology.orchestra.run.smithi177.stdout: "bluefs_file_writer_bytes": 0, 2024-04-02T14:51:20.713 INFO:teuthology.orchestra.run.smithi177.stdout: "bluefs_file_writer_items": 0, 2024-04-02T14:51:20.713 INFO:teuthology.orchestra.run.smithi177.stdout: "buffer_anon_bytes": 264251, 2024-04-02T14:51:20.713 INFO:teuthology.orchestra.run.smithi177.stdout: "buffer_anon_items": 109, 2024-04-02T14:51:20.713 INFO:teuthology.orchestra.run.smithi177.stdout: "buffer_meta_bytes": 0, 2024-04-02T14:51:20.713 INFO:teuthology.orchestra.run.smithi177.stdout: "buffer_meta_items": 0, 2024-04-02T14:51:20.713 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_bytes": 0, 2024-04-02T14:51:20.713 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_items": 0, 2024-04-02T14:51:20.713 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_mapbl_bytes": 0, 2024-04-02T14:51:20.713 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_mapbl_items": 0, 2024-04-02T14:51:20.713 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_pglog_bytes": 0, 2024-04-02T14:51:20.713 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_pglog_items": 0, 2024-04-02T14:51:20.713 INFO:teuthology.orchestra.run.smithi177.stdout: "osdmap_bytes": 8256, 2024-04-02T14:51:20.714 INFO:teuthology.orchestra.run.smithi177.stdout: "osdmap_items": 173, 2024-04-02T14:51:20.714 INFO:teuthology.orchestra.run.smithi177.stdout: "osdmap_mapping_bytes": 0, 2024-04-02T14:51:20.714 INFO:teuthology.orchestra.run.smithi177.stdout: "osdmap_mapping_items": 0, 2024-04-02T14:51:20.714 INFO:teuthology.orchestra.run.smithi177.stdout: "pgmap_bytes": 0, 2024-04-02T14:51:20.714 INFO:teuthology.orchestra.run.smithi177.stdout: "pgmap_items": 0, 2024-04-02T14:51:20.715 INFO:teuthology.orchestra.run.smithi177.stdout: "mds_co_bytes": 196593, 2024-04-02T14:51:20.715 INFO:teuthology.orchestra.run.smithi177.stdout: "mds_co_items": 981, 2024-04-02T14:51:20.715 INFO:teuthology.orchestra.run.smithi177.stdout: "unittest_1_bytes": 0, 2024-04-02T14:51:20.715 INFO:teuthology.orchestra.run.smithi177.stdout: "unittest_1_items": 0, 2024-04-02T14:51:20.715 INFO:teuthology.orchestra.run.smithi177.stdout: "unittest_2_bytes": 0, 2024-04-02T14:51:20.715 INFO:teuthology.orchestra.run.smithi177.stdout: "unittest_2_items": 0 2024-04-02T14:51:20.716 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:20.716 INFO:teuthology.orchestra.run.smithi177.stdout: "objecter": { 2024-04-02T14:51:20.716 INFO:teuthology.orchestra.run.smithi177.stdout: "op_active": 0, 2024-04-02T14:51:20.716 INFO:teuthology.orchestra.run.smithi177.stdout: "op_laggy": 0, 2024-04-02T14:51:20.716 INFO:teuthology.orchestra.run.smithi177.stdout: "op_send": 38, 2024-04-02T14:51:20.716 INFO:teuthology.orchestra.run.smithi177.stdout: "op_send_bytes": 11317, 2024-04-02T14:51:20.717 INFO:teuthology.orchestra.run.smithi177.stdout: "op_resend": 33, 2024-04-02T14:51:20.717 INFO:teuthology.orchestra.run.smithi177.stdout: "op_reply": 38, 2024-04-02T14:51:20.717 INFO:teuthology.orchestra.run.smithi177.stdout: "op_latency": { 2024-04-02T14:51:20.717 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 38, 2024-04-02T14:51:20.718 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.201994512, 2024-04-02T14:51:20.718 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.005315645 2024-04-02T14:51:20.718 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:20.718 INFO:teuthology.orchestra.run.smithi177.stdout: "op_inflight": 0, 2024-04-02T14:51:20.718 INFO:teuthology.orchestra.run.smithi177.stdout: "oplen_avg": { 2024-04-02T14:51:20.719 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 38, 2024-04-02T14:51:20.719 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 61 2024-04-02T14:51:20.719 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:20.719 INFO:teuthology.orchestra.run.smithi177.stdout: "op": 38, 2024-04-02T14:51:20.719 INFO:teuthology.orchestra.run.smithi177.stdout: "op_r": 0, 2024-04-02T14:51:20.720 INFO:teuthology.orchestra.run.smithi177.stdout: "op_w": 38, 2024-04-02T14:51:20.720 INFO:teuthology.orchestra.run.smithi177.stdout: "op_rmw": 0, 2024-04-02T14:51:20.720 INFO:teuthology.orchestra.run.smithi177.stdout: "op_pg": 0, 2024-04-02T14:51:20.720 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_stat": 0, 2024-04-02T14:51:20.720 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_create": 10, 2024-04-02T14:51:20.720 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_read": 0, 2024-04-02T14:51:20.721 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_write": 2, 2024-04-02T14:51:20.721 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_writefull": 6, 2024-04-02T14:51:20.721 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_writesame": 0, 2024-04-02T14:51:20.721 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_append": 0, 2024-04-02T14:51:20.721 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_zero": 0, 2024-04-02T14:51:20.722 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_truncate": 0, 2024-04-02T14:51:20.722 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_delete": 6, 2024-04-02T14:51:20.722 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_mapext": 0, 2024-04-02T14:51:20.722 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_sparse_read": 0, 2024-04-02T14:51:20.722 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_clonerange": 0, 2024-04-02T14:51:20.722 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_getxattr": 0, 2024-04-02T14:51:20.723 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_setxattr": 20, 2024-04-02T14:51:20.723 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_cmpxattr": 0, 2024-04-02T14:51:20.723 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_rmxattr": 0, 2024-04-02T14:51:20.723 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_resetxattrs": 0, 2024-04-02T14:51:20.723 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_call": 0, 2024-04-02T14:51:20.724 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_watch": 0, 2024-04-02T14:51:20.724 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_notify": 0, 2024-04-02T14:51:20.724 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_src_cmpxattr": 0, 2024-04-02T14:51:20.724 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_pgls": 0, 2024-04-02T14:51:20.724 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_pgls_filter": 0, 2024-04-02T14:51:20.724 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_other": 0, 2024-04-02T14:51:20.725 INFO:teuthology.orchestra.run.smithi177.stdout: "linger_active": 0, 2024-04-02T14:51:20.725 INFO:teuthology.orchestra.run.smithi177.stdout: "linger_send": 0, 2024-04-02T14:51:20.725 INFO:teuthology.orchestra.run.smithi177.stdout: "linger_resend": 0, 2024-04-02T14:51:20.725 INFO:teuthology.orchestra.run.smithi177.stdout: "linger_ping": 0, 2024-04-02T14:51:20.725 INFO:teuthology.orchestra.run.smithi177.stdout: "poolop_active": 0, 2024-04-02T14:51:20.725 INFO:teuthology.orchestra.run.smithi177.stdout: "poolop_send": 0, 2024-04-02T14:51:20.726 INFO:teuthology.orchestra.run.smithi177.stdout: "poolop_resend": 0, 2024-04-02T14:51:20.726 INFO:teuthology.orchestra.run.smithi177.stdout: "poolstat_active": 0, 2024-04-02T14:51:20.726 INFO:teuthology.orchestra.run.smithi177.stdout: "poolstat_send": 0, 2024-04-02T14:51:20.726 INFO:teuthology.orchestra.run.smithi177.stdout: "poolstat_resend": 0, 2024-04-02T14:51:20.726 INFO:teuthology.orchestra.run.smithi177.stdout: "statfs_active": 0, 2024-04-02T14:51:20.727 INFO:teuthology.orchestra.run.smithi177.stdout: "statfs_send": 0, 2024-04-02T14:51:20.727 INFO:teuthology.orchestra.run.smithi177.stdout: "statfs_resend": 0, 2024-04-02T14:51:20.727 INFO:teuthology.orchestra.run.smithi177.stdout: "command_active": 0, 2024-04-02T14:51:20.727 INFO:teuthology.orchestra.run.smithi177.stdout: "command_send": 0, 2024-04-02T14:51:20.727 INFO:teuthology.orchestra.run.smithi177.stdout: "command_resend": 0, 2024-04-02T14:51:20.727 INFO:teuthology.orchestra.run.smithi177.stdout: "map_epoch": 111, 2024-04-02T14:51:20.728 INFO:teuthology.orchestra.run.smithi177.stdout: "map_full": 0, 2024-04-02T14:51:20.728 INFO:teuthology.orchestra.run.smithi177.stdout: "map_inc": 23, 2024-04-02T14:51:20.728 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_sessions": 9, 2024-04-02T14:51:20.728 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_session_open": 15, 2024-04-02T14:51:20.728 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_session_close": 6, 2024-04-02T14:51:20.728 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_laggy": 0, 2024-04-02T14:51:20.729 INFO:teuthology.orchestra.run.smithi177.stdout: "omap_wr": 16, 2024-04-02T14:51:20.729 INFO:teuthology.orchestra.run.smithi177.stdout: "omap_rd": 0, 2024-04-02T14:51:20.729 INFO:teuthology.orchestra.run.smithi177.stdout: "omap_del": 1 2024-04-02T14:51:20.729 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:20.729 INFO:teuthology.orchestra.run.smithi177.stdout: "oft": { 2024-04-02T14:51:20.729 INFO:teuthology.orchestra.run.smithi177.stdout: "omap_total_objs": 1, 2024-04-02T14:51:20.730 INFO:teuthology.orchestra.run.smithi177.stdout: "omap_total_kv_pairs": 0, 2024-04-02T14:51:20.730 INFO:teuthology.orchestra.run.smithi177.stdout: "omap_total_updates": 0, 2024-04-02T14:51:20.730 INFO:teuthology.orchestra.run.smithi177.stdout: "omap_total_removes": 0 2024-04-02T14:51:20.730 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:20.730 INFO:teuthology.orchestra.run.smithi177.stdout: "osd-slow-ops": { 2024-04-02T14:51:20.731 INFO:teuthology.orchestra.run.smithi177.stdout: "slow_ops_count": 0 2024-04-02T14:51:20.731 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:20.731 INFO:teuthology.orchestra.run.smithi177.stdout: "purge_queue": { 2024-04-02T14:51:20.731 INFO:teuthology.orchestra.run.smithi177.stdout: "pq_executing_ops": 0, 2024-04-02T14:51:20.731 INFO:teuthology.orchestra.run.smithi177.stdout: "pq_executing_ops_high_water": 0, 2024-04-02T14:51:20.731 INFO:teuthology.orchestra.run.smithi177.stdout: "pq_executing": 0, 2024-04-02T14:51:20.732 INFO:teuthology.orchestra.run.smithi177.stdout: "pq_executing_high_water": 0, 2024-04-02T14:51:20.732 INFO:teuthology.orchestra.run.smithi177.stdout: "pq_executed_ops": 0, 2024-04-02T14:51:20.732 INFO:teuthology.orchestra.run.smithi177.stdout: "pq_executed": 0, 2024-04-02T14:51:20.732 INFO:teuthology.orchestra.run.smithi177.stdout: "pq_item_in_journal": 0 2024-04-02T14:51:20.732 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:20.732 INFO:teuthology.orchestra.run.smithi177.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-02T14:51:20.733 INFO:teuthology.orchestra.run.smithi177.stdout: "val": 0, 2024-04-02T14:51:20.733 INFO:teuthology.orchestra.run.smithi177.stdout: "max": 104857600, 2024-04-02T14:51:20.733 INFO:teuthology.orchestra.run.smithi177.stdout: "get_started": 0, 2024-04-02T14:51:20.733 INFO:teuthology.orchestra.run.smithi177.stdout: "get": 4241, 2024-04-02T14:51:20.733 INFO:teuthology.orchestra.run.smithi177.stdout: "get_sum": 2302524, 2024-04-02T14:51:20.733 INFO:teuthology.orchestra.run.smithi177.stdout: "get_or_fail_fail": 0, 2024-04-02T14:51:20.734 INFO:teuthology.orchestra.run.smithi177.stdout: "get_or_fail_success": 4241, 2024-04-02T14:51:20.734 INFO:teuthology.orchestra.run.smithi177.stdout: "take": 0, 2024-04-02T14:51:20.734 INFO:teuthology.orchestra.run.smithi177.stdout: "take_sum": 0, 2024-04-02T14:51:20.734 INFO:teuthology.orchestra.run.smithi177.stdout: "put": 4241, 2024-04-02T14:51:20.734 INFO:teuthology.orchestra.run.smithi177.stdout: "put_sum": 2302524, 2024-04-02T14:51:20.735 INFO:teuthology.orchestra.run.smithi177.stdout: "wait": { 2024-04-02T14:51:20.735 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-02T14:51:20.735 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-02T14:51:20.735 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-02T14:51:20.735 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-04-02T14:51:20.735 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:20.736 INFO:teuthology.orchestra.run.smithi177.stdout: "throttle-objecter_bytes": { 2024-04-02T14:51:20.736 INFO:teuthology.orchestra.run.smithi177.stdout: "val": 0, 2024-04-02T14:51:20.736 INFO:teuthology.orchestra.run.smithi177.stdout: "max": 104857600, 2024-04-02T14:51:20.736 INFO:teuthology.orchestra.run.smithi177.stdout: "get_started": 0, 2024-04-02T14:51:20.736 INFO:teuthology.orchestra.run.smithi177.stdout: "get": 0, 2024-04-02T14:51:20.736 INFO:teuthology.orchestra.run.smithi177.stdout: "get_sum": 0, 2024-04-02T14:51:20.737 INFO:teuthology.orchestra.run.smithi177.stdout: "get_or_fail_fail": 0, 2024-04-02T14:51:20.737 INFO:teuthology.orchestra.run.smithi177.stdout: "get_or_fail_success": 0, 2024-04-02T14:51:20.737 INFO:teuthology.orchestra.run.smithi177.stdout: "take": 38, 2024-04-02T14:51:20.737 INFO:teuthology.orchestra.run.smithi177.stdout: "take_sum": 11317, 2024-04-02T14:51:20.737 INFO:teuthology.orchestra.run.smithi177.stdout: "put": 38, 2024-04-02T14:51:20.737 INFO:teuthology.orchestra.run.smithi177.stdout: "put_sum": 11317, 2024-04-02T14:51:20.738 INFO:teuthology.orchestra.run.smithi177.stdout: "wait": { 2024-04-02T14:51:20.739 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-02T14:51:20.739 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-02T14:51:20.739 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-02T14:51:20.739 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-04-02T14:51:20.739 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:20.739 INFO:teuthology.orchestra.run.smithi177.stdout: "throttle-objecter_ops": { 2024-04-02T14:51:20.740 INFO:teuthology.orchestra.run.smithi177.stdout: "val": 0, 2024-04-02T14:51:20.740 INFO:teuthology.orchestra.run.smithi177.stdout: "max": 1024, 2024-04-02T14:51:20.740 INFO:teuthology.orchestra.run.smithi177.stdout: "get_started": 0, 2024-04-02T14:51:20.740 INFO:teuthology.orchestra.run.smithi177.stdout: "get": 0, 2024-04-02T14:51:20.740 INFO:teuthology.orchestra.run.smithi177.stdout: "get_sum": 0, 2024-04-02T14:51:20.740 INFO:teuthology.orchestra.run.smithi177.stdout: "get_or_fail_fail": 0, 2024-04-02T14:51:20.741 INFO:teuthology.orchestra.run.smithi177.stdout: "get_or_fail_success": 0, 2024-04-02T14:51:20.741 INFO:teuthology.orchestra.run.smithi177.stdout: "take": 38, 2024-04-02T14:51:20.741 INFO:teuthology.orchestra.run.smithi177.stdout: "take_sum": 38, 2024-04-02T14:51:20.741 INFO:teuthology.orchestra.run.smithi177.stdout: "put": 38, 2024-04-02T14:51:20.741 INFO:teuthology.orchestra.run.smithi177.stdout: "put_sum": 38, 2024-04-02T14:51:20.742 INFO:teuthology.orchestra.run.smithi177.stdout: "wait": { 2024-04-02T14:51:20.742 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-02T14:51:20.742 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-02T14:51:20.742 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-02T14:51:20.742 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-04-02T14:51:20.742 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:20.743 INFO:teuthology.orchestra.run.smithi177.stdout: "throttle-write_buf_throttle": { 2024-04-02T14:51:20.743 INFO:teuthology.orchestra.run.smithi177.stdout: "val": 0, 2024-04-02T14:51:20.743 INFO:teuthology.orchestra.run.smithi177.stdout: "max": 3758096384, 2024-04-02T14:51:20.743 INFO:teuthology.orchestra.run.smithi177.stdout: "get_started": 0, 2024-04-02T14:51:20.743 INFO:teuthology.orchestra.run.smithi177.stdout: "get": 0, 2024-04-02T14:51:20.743 INFO:teuthology.orchestra.run.smithi177.stdout: "get_sum": 0, 2024-04-02T14:51:20.744 INFO:teuthology.orchestra.run.smithi177.stdout: "get_or_fail_fail": 0, 2024-04-02T14:51:20.744 INFO:teuthology.orchestra.run.smithi177.stdout: "get_or_fail_success": 0, 2024-04-02T14:51:20.744 INFO:teuthology.orchestra.run.smithi177.stdout: "take": 0, 2024-04-02T14:51:20.744 INFO:teuthology.orchestra.run.smithi177.stdout: "take_sum": 0, 2024-04-02T14:51:20.744 INFO:teuthology.orchestra.run.smithi177.stdout: "put": 0, 2024-04-02T14:51:20.744 INFO:teuthology.orchestra.run.smithi177.stdout: "put_sum": 0, 2024-04-02T14:51:20.745 INFO:teuthology.orchestra.run.smithi177.stdout: "wait": { 2024-04-02T14:51:20.745 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-02T14:51:20.745 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-02T14:51:20.745 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-02T14:51:20.745 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-04-02T14:51:20.745 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:20.746 INFO:teuthology.orchestra.run.smithi177.stdout: "throttle-write_buf_throttle-0x556691574500": { 2024-04-02T14:51:20.746 INFO:teuthology.orchestra.run.smithi177.stdout: "val": 0, 2024-04-02T14:51:20.746 INFO:teuthology.orchestra.run.smithi177.stdout: "max": 3758096384, 2024-04-02T14:51:20.746 INFO:teuthology.orchestra.run.smithi177.stdout: "get_started": 0, 2024-04-02T14:51:20.746 INFO:teuthology.orchestra.run.smithi177.stdout: "get": 3, 2024-04-02T14:51:20.747 INFO:teuthology.orchestra.run.smithi177.stdout: "get_sum": 894, 2024-04-02T14:51:20.747 INFO:teuthology.orchestra.run.smithi177.stdout: "get_or_fail_fail": 0, 2024-04-02T14:51:20.747 INFO:teuthology.orchestra.run.smithi177.stdout: "get_or_fail_success": 3, 2024-04-02T14:51:20.747 INFO:teuthology.orchestra.run.smithi177.stdout: "take": 0, 2024-04-02T14:51:20.747 INFO:teuthology.orchestra.run.smithi177.stdout: "take_sum": 0, 2024-04-02T14:51:20.747 INFO:teuthology.orchestra.run.smithi177.stdout: "put": 2, 2024-04-02T14:51:20.747 INFO:teuthology.orchestra.run.smithi177.stdout: "put_sum": 894, 2024-04-02T14:51:20.748 INFO:teuthology.orchestra.run.smithi177.stdout: "wait": { 2024-04-02T14:51:20.748 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-02T14:51:20.748 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-02T14:51:20.748 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-02T14:51:20.748 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-04-02T14:51:20.748 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-04-02T14:51:20.749 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-04-02T14:51:21.356 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:51:21 smithi090 ceph-mon[31674]: osdmap e111: 12 total, 12 up, 12 in 2024-04-02T14:51:21.368 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:51:21 smithi177 ceph-mon[36431]: osdmap e111: 12 total, 12 up, 12 in 2024-04-02T14:51:21.400 INFO:tasks.check_counter:Daemon mds.c mds.exported=0 2024-04-02T14:51:21.400 INFO:tasks.check_counter:Daemon mds.c mds.imported=0 2024-04-02T14:51:21.400 INFO:tasks.check_counter:Daemon mds.c mds_cache.dir_update=0 2024-04-02T14:51:21.400 INFO:tasks.check_counter:Daemon mds.c mds_cache.dir_update_receipt=100 2024-04-02T14:51:21.400 INFO:tasks.check_counter:Daemon mds.c mds.root_rsnaps=6 2024-04-02T14:51:21.400 INFO:tasks.check_counter:Daemon mds.c mds_server.req_mksnap_latency.avgcount=0 2024-04-02T14:51:21.400 INFO:tasks.check_counter:Daemon mds.c mds_server.req_rmsnap_latency.avgcount=0 2024-04-02T14:51:21.400 DEBUG:tasks.check_counter:Getting stats from f 2024-04-02T14:51:21.400 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:191d963ac9de7322da8aff20b567b3fa458e2ef6 shell --fsid 6449448a-f0fc-11ee-b647-cb9ed24678a4 -- ceph daemon mds.f perf dump 2024-04-02T14:51:21.601 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:51:21 smithi092 ceph-mon[36497]: osdmap e111: 12 total, 12 up, 12 in 2024-04-02T14:51:21.648 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/mon.c/config 2024-04-02T14:51:22.427 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:51:22 smithi177 ceph-mon[36431]: pgmap v517: 97 pgs: 97 active+clean; 3.9 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 12 KiB/s wr, 11 op/s 2024-04-02T14:51:22.577 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:51:22 smithi090 ceph-mon[31674]: pgmap v517: 97 pgs: 97 active+clean; 3.9 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 12 KiB/s wr, 11 op/s 2024-04-02T14:51:22.602 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:51:22 smithi092 ceph-mon[36497]: pgmap v517: 97 pgs: 97 active+clean; 3.9 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 12 KiB/s wr, 11 op/s 2024-04-02T14:51:23.303 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-04-02T14:51:23.303 INFO:teuthology.orchestra.run.smithi177.stdout: "AsyncMessenger::Worker-0": { 2024-04-02T14:51:23.303 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_messages": 2273, 2024-04-02T14:51:23.304 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_messages": 2258, 2024-04-02T14:51:23.304 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_bytes": 14820848, 2024-04-02T14:51:23.304 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_bytes": 806858, 2024-04-02T14:51:23.304 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_created_connections": 6, 2024-04-02T14:51:23.304 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_active_connections": 2, 2024-04-02T14:51:23.304 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_total_time": 0.229160370, 2024-04-02T14:51:23.304 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_send_time": 0.065642767, 2024-04-02T14:51:23.304 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_recv_time": 0.091840284, 2024-04-02T14:51:23.304 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_fast_dispatch_time": 0.065817633, 2024-04-02T14:51:23.304 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_messages_queue_lat": { 2024-04-02T14:51:23.304 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 2258, 2024-04-02T14:51:23.304 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.085648885, 2024-04-02T14:51:23.304 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000037931 2024-04-02T14:51:23.304 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:23.304 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_handle_ack_lat": { 2024-04-02T14:51:23.305 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-02T14:51:23.305 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-02T14:51:23.305 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-02T14:51:23.305 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:23.305 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_encrypted_bytes": 1658704, 2024-04-02T14:51:23.305 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_encrypted_bytes": 197248 2024-04-02T14:51:23.305 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:23.305 INFO:teuthology.orchestra.run.smithi177.stdout: "AsyncMessenger::Worker-1": { 2024-04-02T14:51:23.305 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_messages": 1116, 2024-04-02T14:51:23.305 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_messages": 1349, 2024-04-02T14:51:23.305 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_bytes": 20796334, 2024-04-02T14:51:23.305 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_bytes": 931808, 2024-04-02T14:51:23.305 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_created_connections": 8, 2024-04-02T14:51:23.305 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_active_connections": 6, 2024-04-02T14:51:23.305 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_total_time": 0.100985350, 2024-04-02T14:51:23.306 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_send_time": 0.031059739, 2024-04-02T14:51:23.306 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_recv_time": 0.033758179, 2024-04-02T14:51:23.306 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_fast_dispatch_time": 0.027450491, 2024-04-02T14:51:23.306 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_messages_queue_lat": { 2024-04-02T14:51:23.306 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 1346, 2024-04-02T14:51:23.306 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.067878630, 2024-04-02T14:51:23.306 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000050429 2024-04-02T14:51:23.306 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:23.306 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_handle_ack_lat": { 2024-04-02T14:51:23.306 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-02T14:51:23.306 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-02T14:51:23.306 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-02T14:51:23.306 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:23.306 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_encrypted_bytes": 320, 2024-04-02T14:51:23.307 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_encrypted_bytes": 577808 2024-04-02T14:51:23.307 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:23.307 INFO:teuthology.orchestra.run.smithi177.stdout: "AsyncMessenger::Worker-2": { 2024-04-02T14:51:23.307 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_messages": 754, 2024-04-02T14:51:23.307 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_messages": 756, 2024-04-02T14:51:23.307 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_bytes": 12773366, 2024-04-02T14:51:23.307 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_bytes": 239779, 2024-04-02T14:51:23.307 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_created_connections": 5, 2024-04-02T14:51:23.307 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_active_connections": 4, 2024-04-02T14:51:23.307 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_total_time": 0.066268578, 2024-04-02T14:51:23.307 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_send_time": 0.019384704, 2024-04-02T14:51:23.307 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_recv_time": 0.021763702, 2024-04-02T14:51:23.307 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_fast_dispatch_time": 0.019697703, 2024-04-02T14:51:23.307 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_messages_queue_lat": { 2024-04-02T14:51:23.308 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 754, 2024-04-02T14:51:23.308 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.036167711, 2024-04-02T14:51:23.308 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000047967 2024-04-02T14:51:23.308 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:23.308 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_handle_ack_lat": { 2024-04-02T14:51:23.308 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-02T14:51:23.308 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-02T14:51:23.308 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-02T14:51:23.308 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:23.308 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-02T14:51:23.308 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_encrypted_bytes": 0 2024-04-02T14:51:23.308 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:23.308 INFO:teuthology.orchestra.run.smithi177.stdout: "cct": { 2024-04-02T14:51:23.308 INFO:teuthology.orchestra.run.smithi177.stdout: "total_workers": 1, 2024-04-02T14:51:23.308 INFO:teuthology.orchestra.run.smithi177.stdout: "unhealthy_workers": 0 2024-04-02T14:51:23.308 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:23.309 INFO:teuthology.orchestra.run.smithi177.stdout: "finisher-MDSRank": { 2024-04-02T14:51:23.309 INFO:teuthology.orchestra.run.smithi177.stdout: "queue_len": 0, 2024-04-02T14:51:23.309 INFO:teuthology.orchestra.run.smithi177.stdout: "complete_latency": { 2024-04-02T14:51:23.309 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 6396, 2024-04-02T14:51:23.309 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.196897077, 2024-04-02T14:51:23.309 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000030784 2024-04-02T14:51:23.309 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-04-02T14:51:23.309 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:23.309 INFO:teuthology.orchestra.run.smithi177.stdout: "finisher-PurgeQueue": { 2024-04-02T14:51:23.309 INFO:teuthology.orchestra.run.smithi177.stdout: "queue_len": 0, 2024-04-02T14:51:23.309 INFO:teuthology.orchestra.run.smithi177.stdout: "complete_latency": { 2024-04-02T14:51:23.309 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-02T14:51:23.309 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-02T14:51:23.309 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-02T14:51:23.310 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-04-02T14:51:23.310 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:23.310 INFO:teuthology.orchestra.run.smithi177.stdout: "mds": { 2024-04-02T14:51:23.310 INFO:teuthology.orchestra.run.smithi177.stdout: "request": 0, 2024-04-02T14:51:23.310 INFO:teuthology.orchestra.run.smithi177.stdout: "reply": 0, 2024-04-02T14:51:23.310 INFO:teuthology.orchestra.run.smithi177.stdout: "reply_latency": { 2024-04-02T14:51:23.310 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-02T14:51:23.310 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-02T14:51:23.310 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-02T14:51:23.310 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:23.310 INFO:teuthology.orchestra.run.smithi177.stdout: "slow_reply": 0, 2024-04-02T14:51:23.310 INFO:teuthology.orchestra.run.smithi177.stdout: "forward": 0, 2024-04-02T14:51:23.310 INFO:teuthology.orchestra.run.smithi177.stdout: "dir_fetch_complete": 0, 2024-04-02T14:51:23.310 INFO:teuthology.orchestra.run.smithi177.stdout: "dir_fetch_keys": 0, 2024-04-02T14:51:23.311 INFO:teuthology.orchestra.run.smithi177.stdout: "dir_commit": 0, 2024-04-02T14:51:23.311 INFO:teuthology.orchestra.run.smithi177.stdout: "dir_split": 0, 2024-04-02T14:51:23.311 INFO:teuthology.orchestra.run.smithi177.stdout: "dir_merge": 0, 2024-04-02T14:51:23.311 INFO:teuthology.orchestra.run.smithi177.stdout: "inodes": 2318, 2024-04-02T14:51:23.311 INFO:teuthology.orchestra.run.smithi177.stdout: "inodes_top": 1463, 2024-04-02T14:51:23.311 INFO:teuthology.orchestra.run.smithi177.stdout: "inodes_bottom": 855, 2024-04-02T14:51:23.311 INFO:teuthology.orchestra.run.smithi177.stdout: "inodes_pin_tail": 0, 2024-04-02T14:51:23.311 INFO:teuthology.orchestra.run.smithi177.stdout: "inodes_pinned": 228, 2024-04-02T14:51:23.311 INFO:teuthology.orchestra.run.smithi177.stdout: "inodes_expired": 0, 2024-04-02T14:51:23.311 INFO:teuthology.orchestra.run.smithi177.stdout: "inodes_with_caps": 0, 2024-04-02T14:51:23.311 INFO:teuthology.orchestra.run.smithi177.stdout: "caps": 0, 2024-04-02T14:51:23.311 INFO:teuthology.orchestra.run.smithi177.stdout: "subtrees": 2, 2024-04-02T14:51:23.311 INFO:teuthology.orchestra.run.smithi177.stdout: "traverse": 0, 2024-04-02T14:51:23.311 INFO:teuthology.orchestra.run.smithi177.stdout: "traverse_hit": 0, 2024-04-02T14:51:23.311 INFO:teuthology.orchestra.run.smithi177.stdout: "traverse_forward": 0, 2024-04-02T14:51:23.312 INFO:teuthology.orchestra.run.smithi177.stdout: "traverse_discover": 0, 2024-04-02T14:51:23.312 INFO:teuthology.orchestra.run.smithi177.stdout: "traverse_dir_fetch": 0, 2024-04-02T14:51:23.312 INFO:teuthology.orchestra.run.smithi177.stdout: "traverse_remote_ino": 0, 2024-04-02T14:51:23.312 INFO:teuthology.orchestra.run.smithi177.stdout: "traverse_lock": 0, 2024-04-02T14:51:23.312 INFO:teuthology.orchestra.run.smithi177.stdout: "load_cent": 0, 2024-04-02T14:51:23.312 INFO:teuthology.orchestra.run.smithi177.stdout: "q": 0, 2024-04-02T14:51:23.312 INFO:teuthology.orchestra.run.smithi177.stdout: "exported": 0, 2024-04-02T14:51:23.312 INFO:teuthology.orchestra.run.smithi177.stdout: "exported_inodes": 0, 2024-04-02T14:51:23.312 INFO:teuthology.orchestra.run.smithi177.stdout: "imported": 0, 2024-04-02T14:51:23.312 INFO:teuthology.orchestra.run.smithi177.stdout: "imported_inodes": 0, 2024-04-02T14:51:23.312 INFO:teuthology.orchestra.run.smithi177.stdout: "openino_dir_fetch": 0, 2024-04-02T14:51:23.312 INFO:teuthology.orchestra.run.smithi177.stdout: "openino_backtrace_fetch": 0, 2024-04-02T14:51:23.312 INFO:teuthology.orchestra.run.smithi177.stdout: "openino_peer_discover": 0, 2024-04-02T14:51:23.312 INFO:teuthology.orchestra.run.smithi177.stdout: "root_rfiles": 1026, 2024-04-02T14:51:23.312 INFO:teuthology.orchestra.run.smithi177.stdout: "root_rbytes": 1710314837, 2024-04-02T14:51:23.313 INFO:teuthology.orchestra.run.smithi177.stdout: "root_rsnaps": 6, 2024-04-02T14:51:23.313 INFO:teuthology.orchestra.run.smithi177.stdout: "scrub_backtrace_fetch": 0, 2024-04-02T14:51:23.313 INFO:teuthology.orchestra.run.smithi177.stdout: "scrub_set_tag": 0, 2024-04-02T14:51:23.313 INFO:teuthology.orchestra.run.smithi177.stdout: "scrub_backtrace_repaired": 0, 2024-04-02T14:51:23.313 INFO:teuthology.orchestra.run.smithi177.stdout: "scrub_inotable_repaired": 0, 2024-04-02T14:51:23.313 INFO:teuthology.orchestra.run.smithi177.stdout: "scrub_dir_inodes": 0, 2024-04-02T14:51:23.313 INFO:teuthology.orchestra.run.smithi177.stdout: "scrub_dir_base_inodes": 0, 2024-04-02T14:51:23.313 INFO:teuthology.orchestra.run.smithi177.stdout: "scrub_dirfrag_rstats": 0, 2024-04-02T14:51:23.313 INFO:teuthology.orchestra.run.smithi177.stdout: "scrub_file_inodes": 0, 2024-04-02T14:51:23.313 INFO:teuthology.orchestra.run.smithi177.stdout: "handle_inode_file_caps": 0, 2024-04-02T14:51:23.313 INFO:teuthology.orchestra.run.smithi177.stdout: "ceph_cap_op_revoke": 0, 2024-04-02T14:51:23.313 INFO:teuthology.orchestra.run.smithi177.stdout: "ceph_cap_op_grant": 0, 2024-04-02T14:51:23.313 INFO:teuthology.orchestra.run.smithi177.stdout: "ceph_cap_op_trunc": 0, 2024-04-02T14:51:23.313 INFO:teuthology.orchestra.run.smithi177.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-02T14:51:23.313 INFO:teuthology.orchestra.run.smithi177.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-02T14:51:23.313 INFO:teuthology.orchestra.run.smithi177.stdout: "handle_client_caps": 0, 2024-04-02T14:51:23.314 INFO:teuthology.orchestra.run.smithi177.stdout: "handle_client_caps_dirty": 0, 2024-04-02T14:51:23.314 INFO:teuthology.orchestra.run.smithi177.stdout: "handle_client_cap_release": 0, 2024-04-02T14:51:23.314 INFO:teuthology.orchestra.run.smithi177.stdout: "process_request_cap_release": 0 2024-04-02T14:51:23.314 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:23.314 INFO:teuthology.orchestra.run.smithi177.stdout: "mds_cache": { 2024-04-02T14:51:23.314 INFO:teuthology.orchestra.run.smithi177.stdout: "dir_update": 0, 2024-04-02T14:51:23.314 INFO:teuthology.orchestra.run.smithi177.stdout: "dir_update_receipt": 0, 2024-04-02T14:51:23.314 INFO:teuthology.orchestra.run.smithi177.stdout: "dir_try_discover": 0, 2024-04-02T14:51:23.314 INFO:teuthology.orchestra.run.smithi177.stdout: "dir_send_discover": 0, 2024-04-02T14:51:23.314 INFO:teuthology.orchestra.run.smithi177.stdout: "dir_handle_discover": 0, 2024-04-02T14:51:23.314 INFO:teuthology.orchestra.run.smithi177.stdout: "num_strays": 0, 2024-04-02T14:51:23.314 INFO:teuthology.orchestra.run.smithi177.stdout: "num_strays_delayed": 0, 2024-04-02T14:51:23.314 INFO:teuthology.orchestra.run.smithi177.stdout: "num_strays_enqueuing": 0, 2024-04-02T14:51:23.314 INFO:teuthology.orchestra.run.smithi177.stdout: "strays_created": 0, 2024-04-02T14:51:23.315 INFO:teuthology.orchestra.run.smithi177.stdout: "strays_enqueued": 0, 2024-04-02T14:51:23.315 INFO:teuthology.orchestra.run.smithi177.stdout: "strays_reintegrated": 0, 2024-04-02T14:51:23.315 INFO:teuthology.orchestra.run.smithi177.stdout: "strays_migrated": 0, 2024-04-02T14:51:23.315 INFO:teuthology.orchestra.run.smithi177.stdout: "num_recovering_processing": 0, 2024-04-02T14:51:23.315 INFO:teuthology.orchestra.run.smithi177.stdout: "num_recovering_enqueued": 0, 2024-04-02T14:51:23.315 INFO:teuthology.orchestra.run.smithi177.stdout: "num_recovering_prioritized": 0, 2024-04-02T14:51:23.315 INFO:teuthology.orchestra.run.smithi177.stdout: "recovery_started": 0, 2024-04-02T14:51:23.315 INFO:teuthology.orchestra.run.smithi177.stdout: "recovery_completed": 0, 2024-04-02T14:51:23.315 INFO:teuthology.orchestra.run.smithi177.stdout: "ireq_quiesce_path": 0, 2024-04-02T14:51:23.315 INFO:teuthology.orchestra.run.smithi177.stdout: "ireq_quiesce_inode": 0, 2024-04-02T14:51:23.315 INFO:teuthology.orchestra.run.smithi177.stdout: "ireq_enqueue_scrub": 0, 2024-04-02T14:51:23.315 INFO:teuthology.orchestra.run.smithi177.stdout: "ireq_exportdir": 0, 2024-04-02T14:51:23.315 INFO:teuthology.orchestra.run.smithi177.stdout: "ireq_flush": 0, 2024-04-02T14:51:23.315 INFO:teuthology.orchestra.run.smithi177.stdout: "ireq_fragmentdir": 0, 2024-04-02T14:51:23.315 INFO:teuthology.orchestra.run.smithi177.stdout: "ireq_fragstats": 0, 2024-04-02T14:51:23.316 INFO:teuthology.orchestra.run.smithi177.stdout: "ireq_inodestats": 0 2024-04-02T14:51:23.316 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:23.316 INFO:teuthology.orchestra.run.smithi177.stdout: "mds_log": { 2024-04-02T14:51:23.316 INFO:teuthology.orchestra.run.smithi177.stdout: "evlrg": 0, 2024-04-02T14:51:23.316 INFO:teuthology.orchestra.run.smithi177.stdout: "evadd": 0, 2024-04-02T14:51:23.316 INFO:teuthology.orchestra.run.smithi177.stdout: "evex": 0, 2024-04-02T14:51:23.316 INFO:teuthology.orchestra.run.smithi177.stdout: "evtrm": 0, 2024-04-02T14:51:23.316 INFO:teuthology.orchestra.run.smithi177.stdout: "ev": 8419, 2024-04-02T14:51:23.316 INFO:teuthology.orchestra.run.smithi177.stdout: "evexg": 0, 2024-04-02T14:51:23.316 INFO:teuthology.orchestra.run.smithi177.stdout: "evexd": 0, 2024-04-02T14:51:23.316 INFO:teuthology.orchestra.run.smithi177.stdout: "segadd": 0, 2024-04-02T14:51:23.316 INFO:teuthology.orchestra.run.smithi177.stdout: "segex": 0, 2024-04-02T14:51:23.316 INFO:teuthology.orchestra.run.smithi177.stdout: "segtrm": 0, 2024-04-02T14:51:23.316 INFO:teuthology.orchestra.run.smithi177.stdout: "seg": 2, 2024-04-02T14:51:23.316 INFO:teuthology.orchestra.run.smithi177.stdout: "segmjr": 71, 2024-04-02T14:51:23.317 INFO:teuthology.orchestra.run.smithi177.stdout: "segexg": 0, 2024-04-02T14:51:23.317 INFO:teuthology.orchestra.run.smithi177.stdout: "segexd": 0, 2024-04-02T14:51:23.317 INFO:teuthology.orchestra.run.smithi177.stdout: "expos": 47317489, 2024-04-02T14:51:23.317 INFO:teuthology.orchestra.run.smithi177.stdout: "wrpos": 49837701, 2024-04-02T14:51:23.317 INFO:teuthology.orchestra.run.smithi177.stdout: "rdpos": 49807160, 2024-04-02T14:51:23.317 INFO:teuthology.orchestra.run.smithi177.stdout: "jlat": { 2024-04-02T14:51:23.317 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-02T14:51:23.317 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-02T14:51:23.317 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-02T14:51:23.317 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:23.317 INFO:teuthology.orchestra.run.smithi177.stdout: "replayed": 8419 2024-04-02T14:51:23.317 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:23.317 INFO:teuthology.orchestra.run.smithi177.stdout: "mds_mem": { 2024-04-02T14:51:23.317 INFO:teuthology.orchestra.run.smithi177.stdout: "ino": 1533, 2024-04-02T14:51:23.317 INFO:teuthology.orchestra.run.smithi177.stdout: "ino+": 1809, 2024-04-02T14:51:23.317 INFO:teuthology.orchestra.run.smithi177.stdout: "ino-": 276, 2024-04-02T14:51:23.318 INFO:teuthology.orchestra.run.smithi177.stdout: "dir": 75, 2024-04-02T14:51:23.318 INFO:teuthology.orchestra.run.smithi177.stdout: "dir+": 81, 2024-04-02T14:51:23.318 INFO:teuthology.orchestra.run.smithi177.stdout: "dir-": 6, 2024-04-02T14:51:23.318 INFO:teuthology.orchestra.run.smithi177.stdout: "dn": 2318, 2024-04-02T14:51:23.318 INFO:teuthology.orchestra.run.smithi177.stdout: "dn+": 2322, 2024-04-02T14:51:23.318 INFO:teuthology.orchestra.run.smithi177.stdout: "dn-": 4, 2024-04-02T14:51:23.318 INFO:teuthology.orchestra.run.smithi177.stdout: "cap": 0, 2024-04-02T14:51:23.318 INFO:teuthology.orchestra.run.smithi177.stdout: "cap+": 0, 2024-04-02T14:51:23.318 INFO:teuthology.orchestra.run.smithi177.stdout: "cap-": 0, 2024-04-02T14:51:23.318 INFO:teuthology.orchestra.run.smithi177.stdout: "rss": 57120, 2024-04-02T14:51:23.318 INFO:teuthology.orchestra.run.smithi177.stdout: "heap": 215416 2024-04-02T14:51:23.318 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:23.318 INFO:teuthology.orchestra.run.smithi177.stdout: "mds_server": { 2024-04-02T14:51:23.318 INFO:teuthology.orchestra.run.smithi177.stdout: "dispatch_client_request": 0, 2024-04-02T14:51:23.319 INFO:teuthology.orchestra.run.smithi177.stdout: "dispatch_server_request": 0, 2024-04-02T14:51:23.319 INFO:teuthology.orchestra.run.smithi177.stdout: "handle_client_request": 0, 2024-04-02T14:51:23.319 INFO:teuthology.orchestra.run.smithi177.stdout: "handle_client_session": 0, 2024-04-02T14:51:23.319 INFO:teuthology.orchestra.run.smithi177.stdout: "handle_peer_request": 0, 2024-04-02T14:51:23.319 INFO:teuthology.orchestra.run.smithi177.stdout: "req_create_latency": { 2024-04-02T14:51:23.319 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-02T14:51:23.319 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-02T14:51:23.319 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-02T14:51:23.319 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:23.319 INFO:teuthology.orchestra.run.smithi177.stdout: "req_getattr_latency": { 2024-04-02T14:51:23.319 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-02T14:51:23.319 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-02T14:51:23.319 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-02T14:51:23.319 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:23.319 INFO:teuthology.orchestra.run.smithi177.stdout: "req_getfilelock_latency": { 2024-04-02T14:51:23.320 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-02T14:51:23.320 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-02T14:51:23.320 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-02T14:51:23.320 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:23.320 INFO:teuthology.orchestra.run.smithi177.stdout: "req_link_latency": { 2024-04-02T14:51:23.320 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-02T14:51:23.320 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-02T14:51:23.320 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-02T14:51:23.320 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:23.320 INFO:teuthology.orchestra.run.smithi177.stdout: "req_lookup_latency": { 2024-04-02T14:51:23.320 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-02T14:51:23.320 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-02T14:51:23.320 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-02T14:51:23.320 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:23.320 INFO:teuthology.orchestra.run.smithi177.stdout: "req_lookuphash_latency": { 2024-04-02T14:51:23.320 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-02T14:51:23.321 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-02T14:51:23.321 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-02T14:51:23.321 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:23.321 INFO:teuthology.orchestra.run.smithi177.stdout: "req_lookupino_latency": { 2024-04-02T14:51:23.321 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-02T14:51:23.321 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-02T14:51:23.321 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-02T14:51:23.321 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:23.321 INFO:teuthology.orchestra.run.smithi177.stdout: "req_lookupname_latency": { 2024-04-02T14:51:23.321 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-02T14:51:23.321 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-02T14:51:23.321 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-02T14:51:23.321 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:23.321 INFO:teuthology.orchestra.run.smithi177.stdout: "req_lookupparent_latency": { 2024-04-02T14:51:23.322 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-02T14:51:23.322 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-02T14:51:23.322 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-02T14:51:23.322 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:23.322 INFO:teuthology.orchestra.run.smithi177.stdout: "req_lookupsnap_latency": { 2024-04-02T14:51:23.322 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-02T14:51:23.322 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-02T14:51:23.322 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-02T14:51:23.322 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:23.322 INFO:teuthology.orchestra.run.smithi177.stdout: "req_lssnap_latency": { 2024-04-02T14:51:23.322 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-02T14:51:23.322 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-02T14:51:23.322 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-02T14:51:23.322 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:23.323 INFO:teuthology.orchestra.run.smithi177.stdout: "req_mkdir_latency": { 2024-04-02T14:51:23.323 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-02T14:51:23.323 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-02T14:51:23.323 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-02T14:51:23.323 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:23.323 INFO:teuthology.orchestra.run.smithi177.stdout: "req_mknod_latency": { 2024-04-02T14:51:23.324 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-02T14:51:23.324 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-02T14:51:23.324 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-02T14:51:23.324 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:23.324 INFO:teuthology.orchestra.run.smithi177.stdout: "req_mksnap_latency": { 2024-04-02T14:51:23.324 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-02T14:51:23.324 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-02T14:51:23.324 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-02T14:51:23.324 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:23.324 INFO:teuthology.orchestra.run.smithi177.stdout: "req_open_latency": { 2024-04-02T14:51:23.324 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-02T14:51:23.324 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-02T14:51:23.324 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-02T14:51:23.324 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:23.325 INFO:teuthology.orchestra.run.smithi177.stdout: "req_readdir_latency": { 2024-04-02T14:51:23.325 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-02T14:51:23.325 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-02T14:51:23.325 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-02T14:51:23.325 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:23.325 INFO:teuthology.orchestra.run.smithi177.stdout: "req_rename_latency": { 2024-04-02T14:51:23.325 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-02T14:51:23.325 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-02T14:51:23.325 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-02T14:51:23.325 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:23.325 INFO:teuthology.orchestra.run.smithi177.stdout: "req_renamesnap_latency": { 2024-04-02T14:51:23.325 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-02T14:51:23.325 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-02T14:51:23.325 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-02T14:51:23.326 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:23.326 INFO:teuthology.orchestra.run.smithi177.stdout: "req_snapdiff_latency": { 2024-04-02T14:51:23.326 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-02T14:51:23.326 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-02T14:51:23.326 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-02T14:51:23.326 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:23.326 INFO:teuthology.orchestra.run.smithi177.stdout: "req_rmdir_latency": { 2024-04-02T14:51:23.326 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-02T14:51:23.326 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-02T14:51:23.326 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-02T14:51:23.326 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:23.326 INFO:teuthology.orchestra.run.smithi177.stdout: "req_rmsnap_latency": { 2024-04-02T14:51:23.326 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-02T14:51:23.326 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-02T14:51:23.327 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-02T14:51:23.327 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:23.327 INFO:teuthology.orchestra.run.smithi177.stdout: "req_rmxattr_latency": { 2024-04-02T14:51:23.327 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-02T14:51:23.327 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-02T14:51:23.327 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-02T14:51:23.327 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:23.327 INFO:teuthology.orchestra.run.smithi177.stdout: "req_setattr_latency": { 2024-04-02T14:51:23.327 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-02T14:51:23.327 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-02T14:51:23.327 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-02T14:51:23.327 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:23.327 INFO:teuthology.orchestra.run.smithi177.stdout: "req_setdirlayout_latency": { 2024-04-02T14:51:23.327 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-02T14:51:23.328 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-02T14:51:23.328 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-02T14:51:23.328 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:23.328 INFO:teuthology.orchestra.run.smithi177.stdout: "req_setfilelock_latency": { 2024-04-02T14:51:23.328 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-02T14:51:23.328 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-02T14:51:23.328 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-02T14:51:23.328 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:23.328 INFO:teuthology.orchestra.run.smithi177.stdout: "req_setlayout_latency": { 2024-04-02T14:51:23.328 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-02T14:51:23.328 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-02T14:51:23.328 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-02T14:51:23.328 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:23.328 INFO:teuthology.orchestra.run.smithi177.stdout: "req_setxattr_latency": { 2024-04-02T14:51:23.328 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-02T14:51:23.328 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-02T14:51:23.329 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-02T14:51:23.329 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:23.329 INFO:teuthology.orchestra.run.smithi177.stdout: "req_symlink_latency": { 2024-04-02T14:51:23.329 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-02T14:51:23.329 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-02T14:51:23.329 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-02T14:51:23.329 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:23.329 INFO:teuthology.orchestra.run.smithi177.stdout: "req_unlink_latency": { 2024-04-02T14:51:23.329 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-02T14:51:23.329 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-02T14:51:23.329 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-02T14:51:23.329 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:23.329 INFO:teuthology.orchestra.run.smithi177.stdout: "cap_revoke_eviction": 0, 2024-04-02T14:51:23.329 INFO:teuthology.orchestra.run.smithi177.stdout: "cap_acquisition_throttle": 0, 2024-04-02T14:51:23.329 INFO:teuthology.orchestra.run.smithi177.stdout: "req_getvxattr_latency": { 2024-04-02T14:51:23.330 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-02T14:51:23.330 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-02T14:51:23.330 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-02T14:51:23.330 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-04-02T14:51:23.330 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:23.330 INFO:teuthology.orchestra.run.smithi177.stdout: "mds_sessions": { 2024-04-02T14:51:23.330 INFO:teuthology.orchestra.run.smithi177.stdout: "session_count": 2, 2024-04-02T14:51:23.330 INFO:teuthology.orchestra.run.smithi177.stdout: "session_add": 6, 2024-04-02T14:51:23.330 INFO:teuthology.orchestra.run.smithi177.stdout: "session_remove": 4, 2024-04-02T14:51:23.330 INFO:teuthology.orchestra.run.smithi177.stdout: "sessions_open": 5, 2024-04-02T14:51:23.330 INFO:teuthology.orchestra.run.smithi177.stdout: "sessions_stale": 0, 2024-04-02T14:51:23.330 INFO:teuthology.orchestra.run.smithi177.stdout: "total_load": 0, 2024-04-02T14:51:23.330 INFO:teuthology.orchestra.run.smithi177.stdout: "average_load": 0, 2024-04-02T14:51:23.330 INFO:teuthology.orchestra.run.smithi177.stdout: "avg_session_uptime": 5251, 2024-04-02T14:51:23.331 INFO:teuthology.orchestra.run.smithi177.stdout: "mdthresh_evicted": 0 2024-04-02T14:51:23.331 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:23.331 INFO:teuthology.orchestra.run.smithi177.stdout: "mempool": { 2024-04-02T14:51:23.331 INFO:teuthology.orchestra.run.smithi177.stdout: "bloom_filter_bytes": 0, 2024-04-02T14:51:23.331 INFO:teuthology.orchestra.run.smithi177.stdout: "bloom_filter_items": 0, 2024-04-02T14:51:23.331 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_alloc_bytes": 0, 2024-04-02T14:51:23.331 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_alloc_items": 0, 2024-04-02T14:51:23.331 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_data_bytes": 0, 2024-04-02T14:51:23.331 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_data_items": 0, 2024-04-02T14:51:23.331 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-02T14:51:23.331 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_onode_items": 0, 2024-04-02T14:51:23.331 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-02T14:51:23.331 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_meta_items": 0, 2024-04-02T14:51:23.331 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_other_bytes": 0, 2024-04-02T14:51:23.331 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_other_items": 0, 2024-04-02T14:51:23.332 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-02T14:51:23.332 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_buffer_items": 0, 2024-04-02T14:51:23.332 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_extent_bytes": 0, 2024-04-02T14:51:23.332 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_extent_items": 0, 2024-04-02T14:51:23.332 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_blob_bytes": 0, 2024-04-02T14:51:23.332 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_blob_items": 0, 2024-04-02T14:51:23.332 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-02T14:51:23.332 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_shared_blob_items": 0, 2024-04-02T14:51:23.332 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-02T14:51:23.332 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_inline_bl_items": 0, 2024-04-02T14:51:23.332 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_fsck_bytes": 0, 2024-04-02T14:51:23.332 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_fsck_items": 0, 2024-04-02T14:51:23.332 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_txc_bytes": 0, 2024-04-02T14:51:23.332 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_txc_items": 0, 2024-04-02T14:51:23.332 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-02T14:51:23.333 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_writing_deferred_items": 0, 2024-04-02T14:51:23.333 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_writing_bytes": 0, 2024-04-02T14:51:23.333 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_writing_items": 0, 2024-04-02T14:51:23.333 INFO:teuthology.orchestra.run.smithi177.stdout: "bluefs_bytes": 0, 2024-04-02T14:51:23.333 INFO:teuthology.orchestra.run.smithi177.stdout: "bluefs_items": 0, 2024-04-02T14:51:23.333 INFO:teuthology.orchestra.run.smithi177.stdout: "bluefs_file_reader_bytes": 0, 2024-04-02T14:51:23.333 INFO:teuthology.orchestra.run.smithi177.stdout: "bluefs_file_reader_items": 0, 2024-04-02T14:51:23.333 INFO:teuthology.orchestra.run.smithi177.stdout: "bluefs_file_writer_bytes": 0, 2024-04-02T14:51:23.333 INFO:teuthology.orchestra.run.smithi177.stdout: "bluefs_file_writer_items": 0, 2024-04-02T14:51:23.333 INFO:teuthology.orchestra.run.smithi177.stdout: "buffer_anon_bytes": 282788, 2024-04-02T14:51:23.333 INFO:teuthology.orchestra.run.smithi177.stdout: "buffer_anon_items": 1875, 2024-04-02T14:51:23.333 INFO:teuthology.orchestra.run.smithi177.stdout: "buffer_meta_bytes": 0, 2024-04-02T14:51:23.333 INFO:teuthology.orchestra.run.smithi177.stdout: "buffer_meta_items": 0, 2024-04-02T14:51:23.333 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_bytes": 0, 2024-04-02T14:51:23.333 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_items": 0, 2024-04-02T14:51:23.333 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_mapbl_bytes": 0, 2024-04-02T14:51:23.334 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_mapbl_items": 0, 2024-04-02T14:51:23.334 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_pglog_bytes": 0, 2024-04-02T14:51:23.334 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_pglog_items": 0, 2024-04-02T14:51:23.334 INFO:teuthology.orchestra.run.smithi177.stdout: "osdmap_bytes": 8136, 2024-04-02T14:51:23.334 INFO:teuthology.orchestra.run.smithi177.stdout: "osdmap_items": 170, 2024-04-02T14:51:23.334 INFO:teuthology.orchestra.run.smithi177.stdout: "osdmap_mapping_bytes": 0, 2024-04-02T14:51:23.334 INFO:teuthology.orchestra.run.smithi177.stdout: "osdmap_mapping_items": 0, 2024-04-02T14:51:23.334 INFO:teuthology.orchestra.run.smithi177.stdout: "pgmap_bytes": 0, 2024-04-02T14:51:23.334 INFO:teuthology.orchestra.run.smithi177.stdout: "pgmap_items": 0, 2024-04-02T14:51:23.334 INFO:teuthology.orchestra.run.smithi177.stdout: "mds_co_bytes": 5567193, 2024-04-02T14:51:23.334 INFO:teuthology.orchestra.run.smithi177.stdout: "mds_co_items": 100757, 2024-04-02T14:51:23.334 INFO:teuthology.orchestra.run.smithi177.stdout: "unittest_1_bytes": 0, 2024-04-02T14:51:23.334 INFO:teuthology.orchestra.run.smithi177.stdout: "unittest_1_items": 0, 2024-04-02T14:51:23.334 INFO:teuthology.orchestra.run.smithi177.stdout: "unittest_2_bytes": 0, 2024-04-02T14:51:23.335 INFO:teuthology.orchestra.run.smithi177.stdout: "unittest_2_items": 0 2024-04-02T14:51:23.335 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:23.335 INFO:teuthology.orchestra.run.smithi177.stdout: "objecter": { 2024-04-02T14:51:23.335 INFO:teuthology.orchestra.run.smithi177.stdout: "op_active": 0, 2024-04-02T14:51:23.335 INFO:teuthology.orchestra.run.smithi177.stdout: "op_laggy": 0, 2024-04-02T14:51:23.335 INFO:teuthology.orchestra.run.smithi177.stdout: "op_send": 3784, 2024-04-02T14:51:23.335 INFO:teuthology.orchestra.run.smithi177.stdout: "op_send_bytes": 26, 2024-04-02T14:51:23.335 INFO:teuthology.orchestra.run.smithi177.stdout: "op_resend": 4, 2024-04-02T14:51:23.335 INFO:teuthology.orchestra.run.smithi177.stdout: "op_reply": 3784, 2024-04-02T14:51:23.335 INFO:teuthology.orchestra.run.smithi177.stdout: "op_latency": { 2024-04-02T14:51:23.335 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 3784, 2024-04-02T14:51:23.335 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 15.204596676, 2024-04-02T14:51:23.335 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.004018128 2024-04-02T14:51:23.335 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:23.336 INFO:teuthology.orchestra.run.smithi177.stdout: "op_inflight": 0, 2024-04-02T14:51:23.336 INFO:teuthology.orchestra.run.smithi177.stdout: "oplen_avg": { 2024-04-02T14:51:23.336 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 3784, 2024-04-02T14:51:23.336 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 3785 2024-04-02T14:51:23.336 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:23.336 INFO:teuthology.orchestra.run.smithi177.stdout: "op": 3784, 2024-04-02T14:51:23.336 INFO:teuthology.orchestra.run.smithi177.stdout: "op_r": 3784, 2024-04-02T14:51:23.336 INFO:teuthology.orchestra.run.smithi177.stdout: "op_w": 0, 2024-04-02T14:51:23.336 INFO:teuthology.orchestra.run.smithi177.stdout: "op_rmw": 0, 2024-04-02T14:51:23.336 INFO:teuthology.orchestra.run.smithi177.stdout: "op_pg": 0, 2024-04-02T14:51:23.336 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_stat": 2222, 2024-04-02T14:51:23.336 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_create": 0, 2024-04-02T14:51:23.336 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_read": 1559, 2024-04-02T14:51:23.336 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_write": 0, 2024-04-02T14:51:23.336 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_writefull": 0, 2024-04-02T14:51:23.337 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_writesame": 0, 2024-04-02T14:51:23.337 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_append": 0, 2024-04-02T14:51:23.337 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_zero": 0, 2024-04-02T14:51:23.337 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_truncate": 0, 2024-04-02T14:51:23.337 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_delete": 0, 2024-04-02T14:51:23.337 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_mapext": 0, 2024-04-02T14:51:23.337 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_sparse_read": 0, 2024-04-02T14:51:23.337 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_clonerange": 0, 2024-04-02T14:51:23.337 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_getxattr": 2, 2024-04-02T14:51:23.337 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_setxattr": 0, 2024-04-02T14:51:23.337 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_cmpxattr": 0, 2024-04-02T14:51:23.337 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_rmxattr": 0, 2024-04-02T14:51:23.337 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_resetxattrs": 0, 2024-04-02T14:51:23.337 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_call": 0, 2024-04-02T14:51:23.337 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_watch": 0, 2024-04-02T14:51:23.338 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_notify": 0, 2024-04-02T14:51:23.338 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_src_cmpxattr": 0, 2024-04-02T14:51:23.338 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_pgls": 0, 2024-04-02T14:51:23.338 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_pgls_filter": 0, 2024-04-02T14:51:23.338 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_other": 0, 2024-04-02T14:51:23.338 INFO:teuthology.orchestra.run.smithi177.stdout: "linger_active": 0, 2024-04-02T14:51:23.338 INFO:teuthology.orchestra.run.smithi177.stdout: "linger_send": 0, 2024-04-02T14:51:23.338 INFO:teuthology.orchestra.run.smithi177.stdout: "linger_resend": 0, 2024-04-02T14:51:23.338 INFO:teuthology.orchestra.run.smithi177.stdout: "linger_ping": 0, 2024-04-02T14:51:23.338 INFO:teuthology.orchestra.run.smithi177.stdout: "poolop_active": 0, 2024-04-02T14:51:23.338 INFO:teuthology.orchestra.run.smithi177.stdout: "poolop_send": 0, 2024-04-02T14:51:23.338 INFO:teuthology.orchestra.run.smithi177.stdout: "poolop_resend": 0, 2024-04-02T14:51:23.338 INFO:teuthology.orchestra.run.smithi177.stdout: "poolstat_active": 0, 2024-04-02T14:51:23.338 INFO:teuthology.orchestra.run.smithi177.stdout: "poolstat_send": 0, 2024-04-02T14:51:23.338 INFO:teuthology.orchestra.run.smithi177.stdout: "poolstat_resend": 0, 2024-04-02T14:51:23.339 INFO:teuthology.orchestra.run.smithi177.stdout: "statfs_active": 0, 2024-04-02T14:51:23.339 INFO:teuthology.orchestra.run.smithi177.stdout: "statfs_send": 0, 2024-04-02T14:51:23.339 INFO:teuthology.orchestra.run.smithi177.stdout: "statfs_resend": 0, 2024-04-02T14:51:23.339 INFO:teuthology.orchestra.run.smithi177.stdout: "command_active": 0, 2024-04-02T14:51:23.339 INFO:teuthology.orchestra.run.smithi177.stdout: "command_send": 0, 2024-04-02T14:51:23.339 INFO:teuthology.orchestra.run.smithi177.stdout: "command_resend": 0, 2024-04-02T14:51:23.339 INFO:teuthology.orchestra.run.smithi177.stdout: "map_epoch": 111, 2024-04-02T14:51:23.339 INFO:teuthology.orchestra.run.smithi177.stdout: "map_full": 0, 2024-04-02T14:51:23.339 INFO:teuthology.orchestra.run.smithi177.stdout: "map_inc": 23, 2024-04-02T14:51:23.339 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_sessions": 8, 2024-04-02T14:51:23.339 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_session_open": 8, 2024-04-02T14:51:23.339 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_session_close": 0, 2024-04-02T14:51:23.339 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_laggy": 0, 2024-04-02T14:51:23.339 INFO:teuthology.orchestra.run.smithi177.stdout: "omap_wr": 0, 2024-04-02T14:51:23.339 INFO:teuthology.orchestra.run.smithi177.stdout: "omap_rd": 2, 2024-04-02T14:51:23.339 INFO:teuthology.orchestra.run.smithi177.stdout: "omap_del": 0 2024-04-02T14:51:23.340 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:23.340 INFO:teuthology.orchestra.run.smithi177.stdout: "oft": { 2024-04-02T14:51:23.340 INFO:teuthology.orchestra.run.smithi177.stdout: "omap_total_objs": 0, 2024-04-02T14:51:23.340 INFO:teuthology.orchestra.run.smithi177.stdout: "omap_total_kv_pairs": 0, 2024-04-02T14:51:23.340 INFO:teuthology.orchestra.run.smithi177.stdout: "omap_total_updates": 0, 2024-04-02T14:51:23.340 INFO:teuthology.orchestra.run.smithi177.stdout: "omap_total_removes": 0 2024-04-02T14:51:23.340 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:23.340 INFO:teuthology.orchestra.run.smithi177.stdout: "osd-slow-ops": { 2024-04-02T14:51:23.340 INFO:teuthology.orchestra.run.smithi177.stdout: "slow_ops_count": 0 2024-04-02T14:51:23.340 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:23.340 INFO:teuthology.orchestra.run.smithi177.stdout: "purge_queue": { 2024-04-02T14:51:23.340 INFO:teuthology.orchestra.run.smithi177.stdout: "pq_executing_ops": 0, 2024-04-02T14:51:23.340 INFO:teuthology.orchestra.run.smithi177.stdout: "pq_executing_ops_high_water": 0, 2024-04-02T14:51:23.340 INFO:teuthology.orchestra.run.smithi177.stdout: "pq_executing": 0, 2024-04-02T14:51:23.340 INFO:teuthology.orchestra.run.smithi177.stdout: "pq_executing_high_water": 0, 2024-04-02T14:51:23.341 INFO:teuthology.orchestra.run.smithi177.stdout: "pq_executed_ops": 0, 2024-04-02T14:51:23.341 INFO:teuthology.orchestra.run.smithi177.stdout: "pq_executed": 0, 2024-04-02T14:51:23.341 INFO:teuthology.orchestra.run.smithi177.stdout: "pq_item_in_journal": 0 2024-04-02T14:51:23.341 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:23.341 INFO:teuthology.orchestra.run.smithi177.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-02T14:51:23.341 INFO:teuthology.orchestra.run.smithi177.stdout: "val": 0, 2024-04-02T14:51:23.341 INFO:teuthology.orchestra.run.smithi177.stdout: "max": 104857600, 2024-04-02T14:51:23.341 INFO:teuthology.orchestra.run.smithi177.stdout: "get_started": 0, 2024-04-02T14:51:23.341 INFO:teuthology.orchestra.run.smithi177.stdout: "get": 4139, 2024-04-02T14:51:23.341 INFO:teuthology.orchestra.run.smithi177.stdout: "get_sum": 47996987, 2024-04-02T14:51:23.341 INFO:teuthology.orchestra.run.smithi177.stdout: "get_or_fail_fail": 0, 2024-04-02T14:51:23.341 INFO:teuthology.orchestra.run.smithi177.stdout: "get_or_fail_success": 4139, 2024-04-02T14:51:23.341 INFO:teuthology.orchestra.run.smithi177.stdout: "take": 0, 2024-04-02T14:51:23.341 INFO:teuthology.orchestra.run.smithi177.stdout: "take_sum": 0, 2024-04-02T14:51:23.341 INFO:teuthology.orchestra.run.smithi177.stdout: "put": 4139, 2024-04-02T14:51:23.342 INFO:teuthology.orchestra.run.smithi177.stdout: "put_sum": 47996987, 2024-04-02T14:51:23.342 INFO:teuthology.orchestra.run.smithi177.stdout: "wait": { 2024-04-02T14:51:23.342 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-02T14:51:23.342 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-02T14:51:23.342 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-02T14:51:23.342 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-04-02T14:51:23.342 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:23.342 INFO:teuthology.orchestra.run.smithi177.stdout: "throttle-objecter_bytes": { 2024-04-02T14:51:23.342 INFO:teuthology.orchestra.run.smithi177.stdout: "val": 0, 2024-04-02T14:51:23.342 INFO:teuthology.orchestra.run.smithi177.stdout: "max": 104857600, 2024-04-02T14:51:23.342 INFO:teuthology.orchestra.run.smithi177.stdout: "get_started": 0, 2024-04-02T14:51:23.342 INFO:teuthology.orchestra.run.smithi177.stdout: "get": 0, 2024-04-02T14:51:23.342 INFO:teuthology.orchestra.run.smithi177.stdout: "get_sum": 0, 2024-04-02T14:51:23.342 INFO:teuthology.orchestra.run.smithi177.stdout: "get_or_fail_fail": 0, 2024-04-02T14:51:23.342 INFO:teuthology.orchestra.run.smithi177.stdout: "get_or_fail_success": 0, 2024-04-02T14:51:23.343 INFO:teuthology.orchestra.run.smithi177.stdout: "take": 3784, 2024-04-02T14:51:23.343 INFO:teuthology.orchestra.run.smithi177.stdout: "take_sum": 45678453, 2024-04-02T14:51:23.343 INFO:teuthology.orchestra.run.smithi177.stdout: "put": 3784, 2024-04-02T14:51:23.343 INFO:teuthology.orchestra.run.smithi177.stdout: "put_sum": 45678453, 2024-04-02T14:51:23.343 INFO:teuthology.orchestra.run.smithi177.stdout: "wait": { 2024-04-02T14:51:23.343 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-02T14:51:23.343 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-02T14:51:23.343 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-02T14:51:23.343 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-04-02T14:51:23.343 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:23.343 INFO:teuthology.orchestra.run.smithi177.stdout: "throttle-objecter_ops": { 2024-04-02T14:51:23.343 INFO:teuthology.orchestra.run.smithi177.stdout: "val": 0, 2024-04-02T14:51:23.343 INFO:teuthology.orchestra.run.smithi177.stdout: "max": 1024, 2024-04-02T14:51:23.343 INFO:teuthology.orchestra.run.smithi177.stdout: "get_started": 0, 2024-04-02T14:51:23.344 INFO:teuthology.orchestra.run.smithi177.stdout: "get": 0, 2024-04-02T14:51:23.344 INFO:teuthology.orchestra.run.smithi177.stdout: "get_sum": 0, 2024-04-02T14:51:23.344 INFO:teuthology.orchestra.run.smithi177.stdout: "get_or_fail_fail": 0, 2024-04-02T14:51:23.344 INFO:teuthology.orchestra.run.smithi177.stdout: "get_or_fail_success": 0, 2024-04-02T14:51:23.344 INFO:teuthology.orchestra.run.smithi177.stdout: "take": 3784, 2024-04-02T14:51:23.344 INFO:teuthology.orchestra.run.smithi177.stdout: "take_sum": 3784, 2024-04-02T14:51:23.344 INFO:teuthology.orchestra.run.smithi177.stdout: "put": 3784, 2024-04-02T14:51:23.344 INFO:teuthology.orchestra.run.smithi177.stdout: "put_sum": 3784, 2024-04-02T14:51:23.344 INFO:teuthology.orchestra.run.smithi177.stdout: "wait": { 2024-04-02T14:51:23.344 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-02T14:51:23.344 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-02T14:51:23.344 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-02T14:51:23.344 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-04-02T14:51:23.344 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:23.344 INFO:teuthology.orchestra.run.smithi177.stdout: "throttle-write_buf_throttle": { 2024-04-02T14:51:23.344 INFO:teuthology.orchestra.run.smithi177.stdout: "val": 0, 2024-04-02T14:51:23.345 INFO:teuthology.orchestra.run.smithi177.stdout: "max": 3758096384, 2024-04-02T14:51:23.345 INFO:teuthology.orchestra.run.smithi177.stdout: "get_started": 0, 2024-04-02T14:51:23.345 INFO:teuthology.orchestra.run.smithi177.stdout: "get": 0, 2024-04-02T14:51:23.345 INFO:teuthology.orchestra.run.smithi177.stdout: "get_sum": 0, 2024-04-02T14:51:23.345 INFO:teuthology.orchestra.run.smithi177.stdout: "get_or_fail_fail": 0, 2024-04-02T14:51:23.345 INFO:teuthology.orchestra.run.smithi177.stdout: "get_or_fail_success": 0, 2024-04-02T14:51:23.345 INFO:teuthology.orchestra.run.smithi177.stdout: "take": 0, 2024-04-02T14:51:23.345 INFO:teuthology.orchestra.run.smithi177.stdout: "take_sum": 0, 2024-04-02T14:51:23.345 INFO:teuthology.orchestra.run.smithi177.stdout: "put": 0, 2024-04-02T14:51:23.345 INFO:teuthology.orchestra.run.smithi177.stdout: "put_sum": 0, 2024-04-02T14:51:23.345 INFO:teuthology.orchestra.run.smithi177.stdout: "wait": { 2024-04-02T14:51:23.345 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-02T14:51:23.345 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-02T14:51:23.345 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-02T14:51:23.345 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-04-02T14:51:23.346 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:23.346 INFO:teuthology.orchestra.run.smithi177.stdout: "throttle-write_buf_throttle-0x55e687c466e0": { 2024-04-02T14:51:23.346 INFO:teuthology.orchestra.run.smithi177.stdout: "val": 0, 2024-04-02T14:51:23.346 INFO:teuthology.orchestra.run.smithi177.stdout: "max": 3758096384, 2024-04-02T14:51:23.346 INFO:teuthology.orchestra.run.smithi177.stdout: "get_started": 0, 2024-04-02T14:51:23.346 INFO:teuthology.orchestra.run.smithi177.stdout: "get": 0, 2024-04-02T14:51:23.346 INFO:teuthology.orchestra.run.smithi177.stdout: "get_sum": 0, 2024-04-02T14:51:23.346 INFO:teuthology.orchestra.run.smithi177.stdout: "get_or_fail_fail": 0, 2024-04-02T14:51:23.346 INFO:teuthology.orchestra.run.smithi177.stdout: "get_or_fail_success": 0, 2024-04-02T14:51:23.346 INFO:teuthology.orchestra.run.smithi177.stdout: "take": 0, 2024-04-02T14:51:23.346 INFO:teuthology.orchestra.run.smithi177.stdout: "take_sum": 0, 2024-04-02T14:51:23.346 INFO:teuthology.orchestra.run.smithi177.stdout: "put": 0, 2024-04-02T14:51:23.346 INFO:teuthology.orchestra.run.smithi177.stdout: "put_sum": 0, 2024-04-02T14:51:23.346 INFO:teuthology.orchestra.run.smithi177.stdout: "wait": { 2024-04-02T14:51:23.346 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-02T14:51:23.347 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-02T14:51:23.347 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-02T14:51:23.347 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-04-02T14:51:23.347 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-04-02T14:51:23.347 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-04-02T14:51:24.126 INFO:tasks.check_counter:Daemon mds.f mds.exported=0 2024-04-02T14:51:24.126 INFO:tasks.check_counter:Daemon mds.f mds.imported=0 2024-04-02T14:51:24.126 INFO:tasks.check_counter:Daemon mds.f mds_cache.dir_update=0 2024-04-02T14:51:24.126 INFO:tasks.check_counter:Daemon mds.f mds_cache.dir_update_receipt=0 2024-04-02T14:51:24.126 INFO:tasks.check_counter:Daemon mds.f mds.root_rsnaps=6 2024-04-02T14:51:24.126 INFO:tasks.check_counter:Daemon mds.f mds_server.req_mksnap_latency.avgcount=0 2024-04-02T14:51:24.126 INFO:tasks.check_counter:Daemon mds.f mds_server.req_rmsnap_latency.avgcount=0 2024-04-02T14:51:24.126 DEBUG:tasks.check_counter:Getting stats from i 2024-04-02T14:51:24.127 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:191d963ac9de7322da8aff20b567b3fa458e2ef6 shell --fsid 6449448a-f0fc-11ee-b647-cb9ed24678a4 -- ceph daemon mds.i perf dump 2024-04-02T14:51:24.366 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/mon.c/config 2024-04-02T14:51:24.427 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:51:24 smithi177 ceph-mon[36431]: pgmap v518: 97 pgs: 97 active+clean; 3.9 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 12 KiB/s wr, 11 op/s 2024-04-02T14:51:24.601 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:51:24 smithi092 ceph-mon[36497]: pgmap v518: 97 pgs: 97 active+clean; 3.9 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 12 KiB/s wr, 11 op/s 2024-04-02T14:51:24.679 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:51:24 smithi090 ceph-mon[31674]: pgmap v518: 97 pgs: 97 active+clean; 3.9 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 12 KiB/s wr, 11 op/s 2024-04-02T14:51:26.184 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-04-02T14:51:26.185 INFO:teuthology.orchestra.run.smithi177.stdout: "AsyncMessenger::Worker-0": { 2024-04-02T14:51:26.185 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_messages": 3893, 2024-04-02T14:51:26.185 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_messages": 2037, 2024-04-02T14:51:26.185 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_bytes": 1066853, 2024-04-02T14:51:26.185 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_bytes": 786404, 2024-04-02T14:51:26.185 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_created_connections": 9, 2024-04-02T14:51:26.185 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_active_connections": 2, 2024-04-02T14:51:26.185 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_total_time": 0.190844474, 2024-04-02T14:51:26.185 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_send_time": 0.068575755, 2024-04-02T14:51:26.185 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_recv_time": 0.147767712, 2024-04-02T14:51:26.185 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_fast_dispatch_time": 0.006785226, 2024-04-02T14:51:26.185 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_messages_queue_lat": { 2024-04-02T14:51:26.185 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 2039, 2024-04-02T14:51:26.185 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.082414061, 2024-04-02T14:51:26.186 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000040418 2024-04-02T14:51:26.186 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:26.186 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_handle_ack_lat": { 2024-04-02T14:51:26.186 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 5595, 2024-04-02T14:51:26.186 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000927998, 2024-04-02T14:51:26.186 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000165 2024-04-02T14:51:26.186 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:26.186 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_encrypted_bytes": 31376, 2024-04-02T14:51:26.186 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_encrypted_bytes": 20208 2024-04-02T14:51:26.186 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:26.186 INFO:teuthology.orchestra.run.smithi177.stdout: "AsyncMessenger::Worker-1": { 2024-04-02T14:51:26.186 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_messages": 425, 2024-04-02T14:51:26.186 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_messages": 657, 2024-04-02T14:51:26.186 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_bytes": 1687403, 2024-04-02T14:51:26.187 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_bytes": 700612, 2024-04-02T14:51:26.187 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_created_connections": 10, 2024-04-02T14:51:26.187 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_active_connections": 5, 2024-04-02T14:51:26.187 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_total_time": 0.067244806, 2024-04-02T14:51:26.187 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_send_time": 0.025481228, 2024-04-02T14:51:26.187 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_recv_time": 0.042637533, 2024-04-02T14:51:26.187 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_fast_dispatch_time": 0.007298438, 2024-04-02T14:51:26.187 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_messages_queue_lat": { 2024-04-02T14:51:26.187 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 657, 2024-04-02T14:51:26.187 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.044274710, 2024-04-02T14:51:26.187 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000067389 2024-04-02T14:51:26.187 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:26.187 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_handle_ack_lat": { 2024-04-02T14:51:26.187 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 69, 2024-04-02T14:51:26.187 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000012180, 2024-04-02T14:51:26.188 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000176 2024-04-02T14:51:26.188 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:26.188 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_encrypted_bytes": 1682448, 2024-04-02T14:51:26.188 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_encrypted_bytes": 663712 2024-04-02T14:51:26.188 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:26.188 INFO:teuthology.orchestra.run.smithi177.stdout: "AsyncMessenger::Worker-2": { 2024-04-02T14:51:26.188 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_messages": 92, 2024-04-02T14:51:26.188 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_messages": 136, 2024-04-02T14:51:26.188 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_bytes": 17453, 2024-04-02T14:51:26.188 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_bytes": 829435, 2024-04-02T14:51:26.188 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_created_connections": 73, 2024-04-02T14:51:26.188 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_active_connections": 8, 2024-04-02T14:51:26.188 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_total_time": 0.027995326, 2024-04-02T14:51:26.188 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_send_time": 0.003746541, 2024-04-02T14:51:26.188 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_recv_time": 0.021932590, 2024-04-02T14:51:26.189 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_fast_dispatch_time": 0.000119281, 2024-04-02T14:51:26.189 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_messages_queue_lat": { 2024-04-02T14:51:26.189 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 129, 2024-04-02T14:51:26.189 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.008206625, 2024-04-02T14:51:26.189 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000063617 2024-04-02T14:51:26.189 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:26.189 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_handle_ack_lat": { 2024-04-02T14:51:26.189 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 113, 2024-04-02T14:51:26.189 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000013456, 2024-04-02T14:51:26.189 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000119 2024-04-02T14:51:26.189 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:26.189 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_encrypted_bytes": 7680, 2024-04-02T14:51:26.189 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_encrypted_bytes": 146384 2024-04-02T14:51:26.189 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:26.189 INFO:teuthology.orchestra.run.smithi177.stdout: "cct": { 2024-04-02T14:51:26.189 INFO:teuthology.orchestra.run.smithi177.stdout: "total_workers": 1, 2024-04-02T14:51:26.190 INFO:teuthology.orchestra.run.smithi177.stdout: "unhealthy_workers": 0 2024-04-02T14:51:26.190 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:26.190 INFO:teuthology.orchestra.run.smithi177.stdout: "finisher-MDSRank": { 2024-04-02T14:51:26.190 INFO:teuthology.orchestra.run.smithi177.stdout: "queue_len": 0, 2024-04-02T14:51:26.190 INFO:teuthology.orchestra.run.smithi177.stdout: "complete_latency": { 2024-04-02T14:51:26.190 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 72, 2024-04-02T14:51:26.190 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.002275135, 2024-04-02T14:51:26.190 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000031599 2024-04-02T14:51:26.190 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-04-02T14:51:26.190 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:26.190 INFO:teuthology.orchestra.run.smithi177.stdout: "finisher-PurgeQueue": { 2024-04-02T14:51:26.190 INFO:teuthology.orchestra.run.smithi177.stdout: "queue_len": 0, 2024-04-02T14:51:26.190 INFO:teuthology.orchestra.run.smithi177.stdout: "complete_latency": { 2024-04-02T14:51:26.190 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 2, 2024-04-02T14:51:26.191 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000008610, 2024-04-02T14:51:26.191 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000004305 2024-04-02T14:51:26.191 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-04-02T14:51:26.191 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:26.191 INFO:teuthology.orchestra.run.smithi177.stdout: "mds": { 2024-04-02T14:51:26.191 INFO:teuthology.orchestra.run.smithi177.stdout: "request": 97, 2024-04-02T14:51:26.191 INFO:teuthology.orchestra.run.smithi177.stdout: "reply": 10, 2024-04-02T14:51:26.191 INFO:teuthology.orchestra.run.smithi177.stdout: "reply_latency": { 2024-04-02T14:51:26.191 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 10, 2024-04-02T14:51:26.191 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.004223544, 2024-04-02T14:51:26.191 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000422354 2024-04-02T14:51:26.191 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:26.191 INFO:teuthology.orchestra.run.smithi177.stdout: "slow_reply": 0, 2024-04-02T14:51:26.191 INFO:teuthology.orchestra.run.smithi177.stdout: "forward": 87, 2024-04-02T14:51:26.191 INFO:teuthology.orchestra.run.smithi177.stdout: "dir_fetch_complete": 0, 2024-04-02T14:51:26.191 INFO:teuthology.orchestra.run.smithi177.stdout: "dir_fetch_keys": 0, 2024-04-02T14:51:26.192 INFO:teuthology.orchestra.run.smithi177.stdout: "dir_commit": 11, 2024-04-02T14:51:26.192 INFO:teuthology.orchestra.run.smithi177.stdout: "dir_split": 0, 2024-04-02T14:51:26.192 INFO:teuthology.orchestra.run.smithi177.stdout: "dir_merge": 0, 2024-04-02T14:51:26.192 INFO:teuthology.orchestra.run.smithi177.stdout: "inodes": 34, 2024-04-02T14:51:26.192 INFO:teuthology.orchestra.run.smithi177.stdout: "inodes_top": 11, 2024-04-02T14:51:26.192 INFO:teuthology.orchestra.run.smithi177.stdout: "inodes_bottom": 23, 2024-04-02T14:51:26.192 INFO:teuthology.orchestra.run.smithi177.stdout: "inodes_pin_tail": 0, 2024-04-02T14:51:26.192 INFO:teuthology.orchestra.run.smithi177.stdout: "inodes_pinned": 18, 2024-04-02T14:51:26.192 INFO:teuthology.orchestra.run.smithi177.stdout: "inodes_expired": 5, 2024-04-02T14:51:26.192 INFO:teuthology.orchestra.run.smithi177.stdout: "inodes_with_caps": 1, 2024-04-02T14:51:26.192 INFO:teuthology.orchestra.run.smithi177.stdout: "caps": 1, 2024-04-02T14:51:26.192 INFO:teuthology.orchestra.run.smithi177.stdout: "subtrees": 3, 2024-04-02T14:51:26.192 INFO:teuthology.orchestra.run.smithi177.stdout: "traverse": 166, 2024-04-02T14:51:26.192 INFO:teuthology.orchestra.run.smithi177.stdout: "traverse_hit": 59, 2024-04-02T14:51:26.192 INFO:teuthology.orchestra.run.smithi177.stdout: "traverse_forward": 81, 2024-04-02T14:51:26.193 INFO:teuthology.orchestra.run.smithi177.stdout: "traverse_discover": 21, 2024-04-02T14:51:26.193 INFO:teuthology.orchestra.run.smithi177.stdout: "traverse_dir_fetch": 0, 2024-04-02T14:51:26.193 INFO:teuthology.orchestra.run.smithi177.stdout: "traverse_remote_ino": 0, 2024-04-02T14:51:26.193 INFO:teuthology.orchestra.run.smithi177.stdout: "traverse_lock": 0, 2024-04-02T14:51:26.193 INFO:teuthology.orchestra.run.smithi177.stdout: "load_cent": 0, 2024-04-02T14:51:26.193 INFO:teuthology.orchestra.run.smithi177.stdout: "q": 0, 2024-04-02T14:51:26.193 INFO:teuthology.orchestra.run.smithi177.stdout: "exported": 0, 2024-04-02T14:51:26.193 INFO:teuthology.orchestra.run.smithi177.stdout: "exported_inodes": 0, 2024-04-02T14:51:26.193 INFO:teuthology.orchestra.run.smithi177.stdout: "imported": 0, 2024-04-02T14:51:26.193 INFO:teuthology.orchestra.run.smithi177.stdout: "imported_inodes": 0, 2024-04-02T14:51:26.193 INFO:teuthology.orchestra.run.smithi177.stdout: "openino_dir_fetch": 0, 2024-04-02T14:51:26.193 INFO:teuthology.orchestra.run.smithi177.stdout: "openino_backtrace_fetch": 0, 2024-04-02T14:51:26.193 INFO:teuthology.orchestra.run.smithi177.stdout: "openino_peer_discover": 0, 2024-04-02T14:51:26.193 INFO:teuthology.orchestra.run.smithi177.stdout: "root_rfiles": 1026, 2024-04-02T14:51:26.193 INFO:teuthology.orchestra.run.smithi177.stdout: "root_rbytes": 1709677761, 2024-04-02T14:51:26.193 INFO:teuthology.orchestra.run.smithi177.stdout: "root_rsnaps": 6, 2024-04-02T14:51:26.193 INFO:teuthology.orchestra.run.smithi177.stdout: "scrub_backtrace_fetch": 0, 2024-04-02T14:51:26.194 INFO:teuthology.orchestra.run.smithi177.stdout: "scrub_set_tag": 0, 2024-04-02T14:51:26.194 INFO:teuthology.orchestra.run.smithi177.stdout: "scrub_backtrace_repaired": 0, 2024-04-02T14:51:26.194 INFO:teuthology.orchestra.run.smithi177.stdout: "scrub_inotable_repaired": 0, 2024-04-02T14:51:26.194 INFO:teuthology.orchestra.run.smithi177.stdout: "scrub_dir_inodes": 0, 2024-04-02T14:51:26.194 INFO:teuthology.orchestra.run.smithi177.stdout: "scrub_dir_base_inodes": 0, 2024-04-02T14:51:26.194 INFO:teuthology.orchestra.run.smithi177.stdout: "scrub_dirfrag_rstats": 0, 2024-04-02T14:51:26.194 INFO:teuthology.orchestra.run.smithi177.stdout: "scrub_file_inodes": 0, 2024-04-02T14:51:26.194 INFO:teuthology.orchestra.run.smithi177.stdout: "handle_inode_file_caps": 0, 2024-04-02T14:51:26.194 INFO:teuthology.orchestra.run.smithi177.stdout: "ceph_cap_op_revoke": 0, 2024-04-02T14:51:26.194 INFO:teuthology.orchestra.run.smithi177.stdout: "ceph_cap_op_grant": 0, 2024-04-02T14:51:26.194 INFO:teuthology.orchestra.run.smithi177.stdout: "ceph_cap_op_trunc": 0, 2024-04-02T14:51:26.194 INFO:teuthology.orchestra.run.smithi177.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-02T14:51:26.194 INFO:teuthology.orchestra.run.smithi177.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-02T14:51:26.194 INFO:teuthology.orchestra.run.smithi177.stdout: "handle_client_caps": 0, 2024-04-02T14:51:26.194 INFO:teuthology.orchestra.run.smithi177.stdout: "handle_client_caps_dirty": 0, 2024-04-02T14:51:26.195 INFO:teuthology.orchestra.run.smithi177.stdout: "handle_client_cap_release": 0, 2024-04-02T14:51:26.195 INFO:teuthology.orchestra.run.smithi177.stdout: "process_request_cap_release": 2 2024-04-02T14:51:26.195 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:26.195 INFO:teuthology.orchestra.run.smithi177.stdout: "mds_cache": { 2024-04-02T14:51:26.195 INFO:teuthology.orchestra.run.smithi177.stdout: "dir_update": 0, 2024-04-02T14:51:26.195 INFO:teuthology.orchestra.run.smithi177.stdout: "dir_update_receipt": 102, 2024-04-02T14:51:26.195 INFO:teuthology.orchestra.run.smithi177.stdout: "dir_try_discover": 61, 2024-04-02T14:51:26.195 INFO:teuthology.orchestra.run.smithi177.stdout: "dir_send_discover": 63, 2024-04-02T14:51:26.195 INFO:teuthology.orchestra.run.smithi177.stdout: "dir_handle_discover": 0, 2024-04-02T14:51:26.195 INFO:teuthology.orchestra.run.smithi177.stdout: "num_strays": 0, 2024-04-02T14:51:26.195 INFO:teuthology.orchestra.run.smithi177.stdout: "num_strays_delayed": 0, 2024-04-02T14:51:26.195 INFO:teuthology.orchestra.run.smithi177.stdout: "num_strays_enqueuing": 0, 2024-04-02T14:51:26.195 INFO:teuthology.orchestra.run.smithi177.stdout: "strays_created": 0, 2024-04-02T14:51:26.195 INFO:teuthology.orchestra.run.smithi177.stdout: "strays_enqueued": 0, 2024-04-02T14:51:26.195 INFO:teuthology.orchestra.run.smithi177.stdout: "strays_reintegrated": 0, 2024-04-02T14:51:26.195 INFO:teuthology.orchestra.run.smithi177.stdout: "strays_migrated": 0, 2024-04-02T14:51:26.196 INFO:teuthology.orchestra.run.smithi177.stdout: "num_recovering_processing": 0, 2024-04-02T14:51:26.196 INFO:teuthology.orchestra.run.smithi177.stdout: "num_recovering_enqueued": 0, 2024-04-02T14:51:26.196 INFO:teuthology.orchestra.run.smithi177.stdout: "num_recovering_prioritized": 0, 2024-04-02T14:51:26.196 INFO:teuthology.orchestra.run.smithi177.stdout: "recovery_started": 0, 2024-04-02T14:51:26.196 INFO:teuthology.orchestra.run.smithi177.stdout: "recovery_completed": 0, 2024-04-02T14:51:26.196 INFO:teuthology.orchestra.run.smithi177.stdout: "ireq_quiesce_path": 0, 2024-04-02T14:51:26.196 INFO:teuthology.orchestra.run.smithi177.stdout: "ireq_quiesce_inode": 0, 2024-04-02T14:51:26.196 INFO:teuthology.orchestra.run.smithi177.stdout: "ireq_enqueue_scrub": 0, 2024-04-02T14:51:26.196 INFO:teuthology.orchestra.run.smithi177.stdout: "ireq_exportdir": 0, 2024-04-02T14:51:26.196 INFO:teuthology.orchestra.run.smithi177.stdout: "ireq_flush": 0, 2024-04-02T14:51:26.196 INFO:teuthology.orchestra.run.smithi177.stdout: "ireq_fragmentdir": 0, 2024-04-02T14:51:26.196 INFO:teuthology.orchestra.run.smithi177.stdout: "ireq_fragstats": 0, 2024-04-02T14:51:26.196 INFO:teuthology.orchestra.run.smithi177.stdout: "ireq_inodestats": 0 2024-04-02T14:51:26.196 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:26.196 INFO:teuthology.orchestra.run.smithi177.stdout: "mds_log": { 2024-04-02T14:51:26.196 INFO:teuthology.orchestra.run.smithi177.stdout: "evlrg": 0, 2024-04-02T14:51:26.196 INFO:teuthology.orchestra.run.smithi177.stdout: "evadd": 12, 2024-04-02T14:51:26.197 INFO:teuthology.orchestra.run.smithi177.stdout: "evex": 0, 2024-04-02T14:51:26.197 INFO:teuthology.orchestra.run.smithi177.stdout: "evtrm": 0, 2024-04-02T14:51:26.197 INFO:teuthology.orchestra.run.smithi177.stdout: "ev": 12, 2024-04-02T14:51:26.197 INFO:teuthology.orchestra.run.smithi177.stdout: "evexg": 0, 2024-04-02T14:51:26.197 INFO:teuthology.orchestra.run.smithi177.stdout: "evexd": 0, 2024-04-02T14:51:26.197 INFO:teuthology.orchestra.run.smithi177.stdout: "segadd": 2, 2024-04-02T14:51:26.197 INFO:teuthology.orchestra.run.smithi177.stdout: "segex": 0, 2024-04-02T14:51:26.197 INFO:teuthology.orchestra.run.smithi177.stdout: "segtrm": 0, 2024-04-02T14:51:26.197 INFO:teuthology.orchestra.run.smithi177.stdout: "seg": 2, 2024-04-02T14:51:26.197 INFO:teuthology.orchestra.run.smithi177.stdout: "segmjr": 2, 2024-04-02T14:51:26.197 INFO:teuthology.orchestra.run.smithi177.stdout: "segexg": 0, 2024-04-02T14:51:26.197 INFO:teuthology.orchestra.run.smithi177.stdout: "segexd": 0, 2024-04-02T14:51:26.197 INFO:teuthology.orchestra.run.smithi177.stdout: "expos": 4194304, 2024-04-02T14:51:26.197 INFO:teuthology.orchestra.run.smithi177.stdout: "wrpos": 4197717, 2024-04-02T14:51:26.197 INFO:teuthology.orchestra.run.smithi177.stdout: "rdpos": 0, 2024-04-02T14:51:26.197 INFO:teuthology.orchestra.run.smithi177.stdout: "jlat": { 2024-04-02T14:51:26.198 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 11, 2024-04-02T14:51:26.198 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.019230884, 2024-04-02T14:51:26.198 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.001748262 2024-04-02T14:51:26.198 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:26.198 INFO:teuthology.orchestra.run.smithi177.stdout: "replayed": 0 2024-04-02T14:51:26.198 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:26.198 INFO:teuthology.orchestra.run.smithi177.stdout: "mds_mem": { 2024-04-02T14:51:26.198 INFO:teuthology.orchestra.run.smithi177.stdout: "ino": 38, 2024-04-02T14:51:26.198 INFO:teuthology.orchestra.run.smithi177.stdout: "ino+": 40, 2024-04-02T14:51:26.198 INFO:teuthology.orchestra.run.smithi177.stdout: "ino-": 2, 2024-04-02T14:51:26.198 INFO:teuthology.orchestra.run.smithi177.stdout: "dir": 65, 2024-04-02T14:51:26.198 INFO:teuthology.orchestra.run.smithi177.stdout: "dir+": 68, 2024-04-02T14:51:26.198 INFO:teuthology.orchestra.run.smithi177.stdout: "dir-": 3, 2024-04-02T14:51:26.198 INFO:teuthology.orchestra.run.smithi177.stdout: "dn": 34, 2024-04-02T14:51:26.198 INFO:teuthology.orchestra.run.smithi177.stdout: "dn+": 39, 2024-04-02T14:51:26.198 INFO:teuthology.orchestra.run.smithi177.stdout: "dn-": 5, 2024-04-02T14:51:26.199 INFO:teuthology.orchestra.run.smithi177.stdout: "cap": 1, 2024-04-02T14:51:26.199 INFO:teuthology.orchestra.run.smithi177.stdout: "cap+": 7, 2024-04-02T14:51:26.199 INFO:teuthology.orchestra.run.smithi177.stdout: "cap-": 6, 2024-04-02T14:51:26.199 INFO:teuthology.orchestra.run.smithi177.stdout: "rss": 53864, 2024-04-02T14:51:26.199 INFO:teuthology.orchestra.run.smithi177.stdout: "heap": 223608 2024-04-02T14:51:26.199 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:26.199 INFO:teuthology.orchestra.run.smithi177.stdout: "mds_server": { 2024-04-02T14:51:26.199 INFO:teuthology.orchestra.run.smithi177.stdout: "dispatch_client_request": 101, 2024-04-02T14:51:26.199 INFO:teuthology.orchestra.run.smithi177.stdout: "dispatch_server_request": 0, 2024-04-02T14:51:26.199 INFO:teuthology.orchestra.run.smithi177.stdout: "handle_client_request": 97, 2024-04-02T14:51:26.199 INFO:teuthology.orchestra.run.smithi177.stdout: "handle_client_session": 41, 2024-04-02T14:51:26.199 INFO:teuthology.orchestra.run.smithi177.stdout: "handle_peer_request": 1, 2024-04-02T14:51:26.199 INFO:teuthology.orchestra.run.smithi177.stdout: "req_create_latency": { 2024-04-02T14:51:26.199 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-02T14:51:26.199 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-02T14:51:26.200 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-02T14:51:26.200 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:26.200 INFO:teuthology.orchestra.run.smithi177.stdout: "req_getattr_latency": { 2024-04-02T14:51:26.200 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 6, 2024-04-02T14:51:26.200 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.003295489, 2024-04-02T14:51:26.200 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000549248 2024-04-02T14:51:26.200 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:26.200 INFO:teuthology.orchestra.run.smithi177.stdout: "req_getfilelock_latency": { 2024-04-02T14:51:26.200 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-02T14:51:26.200 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-02T14:51:26.200 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-02T14:51:26.200 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:26.200 INFO:teuthology.orchestra.run.smithi177.stdout: "req_link_latency": { 2024-04-02T14:51:26.200 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-02T14:51:26.200 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-02T14:51:26.200 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-02T14:51:26.201 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:26.201 INFO:teuthology.orchestra.run.smithi177.stdout: "req_lookup_latency": { 2024-04-02T14:51:26.201 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 4, 2024-04-02T14:51:26.201 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000928055, 2024-04-02T14:51:26.201 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000232013 2024-04-02T14:51:26.201 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:26.201 INFO:teuthology.orchestra.run.smithi177.stdout: "req_lookuphash_latency": { 2024-04-02T14:51:26.201 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-02T14:51:26.201 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-02T14:51:26.201 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-02T14:51:26.201 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:26.201 INFO:teuthology.orchestra.run.smithi177.stdout: "req_lookupino_latency": { 2024-04-02T14:51:26.201 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-02T14:51:26.202 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-02T14:51:26.202 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-02T14:51:26.202 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:26.202 INFO:teuthology.orchestra.run.smithi177.stdout: "req_lookupname_latency": { 2024-04-02T14:51:26.202 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-02T14:51:26.202 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-02T14:51:26.202 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-02T14:51:26.202 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:26.202 INFO:teuthology.orchestra.run.smithi177.stdout: "req_lookupparent_latency": { 2024-04-02T14:51:26.202 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-02T14:51:26.202 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-02T14:51:26.202 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-02T14:51:26.202 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:26.203 INFO:teuthology.orchestra.run.smithi177.stdout: "req_lookupsnap_latency": { 2024-04-02T14:51:26.203 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-02T14:51:26.203 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-02T14:51:26.203 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-02T14:51:26.203 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:26.203 INFO:teuthology.orchestra.run.smithi177.stdout: "req_lssnap_latency": { 2024-04-02T14:51:26.203 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-02T14:51:26.203 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-02T14:51:26.203 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-02T14:51:26.203 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:26.204 INFO:teuthology.orchestra.run.smithi177.stdout: "req_mkdir_latency": { 2024-04-02T14:51:26.204 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-02T14:51:26.204 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-02T14:51:26.204 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-02T14:51:26.204 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:26.204 INFO:teuthology.orchestra.run.smithi177.stdout: "req_mknod_latency": { 2024-04-02T14:51:26.204 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-02T14:51:26.204 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-02T14:51:26.204 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-02T14:51:26.204 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:26.205 INFO:teuthology.orchestra.run.smithi177.stdout: "req_mksnap_latency": { 2024-04-02T14:51:26.205 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-02T14:51:26.205 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-02T14:51:26.205 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-02T14:51:26.205 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:26.205 INFO:teuthology.orchestra.run.smithi177.stdout: "req_open_latency": { 2024-04-02T14:51:26.205 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-02T14:51:26.205 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-02T14:51:26.205 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-02T14:51:26.205 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:26.205 INFO:teuthology.orchestra.run.smithi177.stdout: "req_readdir_latency": { 2024-04-02T14:51:26.205 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-02T14:51:26.205 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-02T14:51:26.206 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-02T14:51:26.206 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:26.206 INFO:teuthology.orchestra.run.smithi177.stdout: "req_rename_latency": { 2024-04-02T14:51:26.206 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-02T14:51:26.206 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-02T14:51:26.206 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-02T14:51:26.206 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:26.206 INFO:teuthology.orchestra.run.smithi177.stdout: "req_renamesnap_latency": { 2024-04-02T14:51:26.206 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-02T14:51:26.206 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-02T14:51:26.206 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-02T14:51:26.206 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:26.206 INFO:teuthology.orchestra.run.smithi177.stdout: "req_snapdiff_latency": { 2024-04-02T14:51:26.206 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-02T14:51:26.206 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-02T14:51:26.207 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-02T14:51:26.207 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:26.207 INFO:teuthology.orchestra.run.smithi177.stdout: "req_rmdir_latency": { 2024-04-02T14:51:26.207 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-02T14:51:26.207 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-02T14:51:26.207 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-02T14:51:26.207 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:26.207 INFO:teuthology.orchestra.run.smithi177.stdout: "req_rmsnap_latency": { 2024-04-02T14:51:26.207 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-02T14:51:26.207 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-02T14:51:26.207 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-02T14:51:26.207 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:26.207 INFO:teuthology.orchestra.run.smithi177.stdout: "req_rmxattr_latency": { 2024-04-02T14:51:26.207 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-02T14:51:26.207 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-02T14:51:26.207 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-02T14:51:26.208 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:26.208 INFO:teuthology.orchestra.run.smithi177.stdout: "req_setattr_latency": { 2024-04-02T14:51:26.208 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-02T14:51:26.208 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-02T14:51:26.208 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-02T14:51:26.208 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:26.208 INFO:teuthology.orchestra.run.smithi177.stdout: "req_setdirlayout_latency": { 2024-04-02T14:51:26.208 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-02T14:51:26.208 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-02T14:51:26.208 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-02T14:51:26.208 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:26.208 INFO:teuthology.orchestra.run.smithi177.stdout: "req_setfilelock_latency": { 2024-04-02T14:51:26.208 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-02T14:51:26.208 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-02T14:51:26.208 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-02T14:51:26.209 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:26.209 INFO:teuthology.orchestra.run.smithi177.stdout: "req_setlayout_latency": { 2024-04-02T14:51:26.209 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-02T14:51:26.209 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-02T14:51:26.209 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-02T14:51:26.209 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:26.209 INFO:teuthology.orchestra.run.smithi177.stdout: "req_setxattr_latency": { 2024-04-02T14:51:26.209 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-02T14:51:26.209 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-02T14:51:26.209 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-02T14:51:26.209 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:26.209 INFO:teuthology.orchestra.run.smithi177.stdout: "req_symlink_latency": { 2024-04-02T14:51:26.209 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-02T14:51:26.209 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-02T14:51:26.209 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-02T14:51:26.209 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:26.210 INFO:teuthology.orchestra.run.smithi177.stdout: "req_unlink_latency": { 2024-04-02T14:51:26.210 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-02T14:51:26.210 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-02T14:51:26.210 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-02T14:51:26.210 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:26.210 INFO:teuthology.orchestra.run.smithi177.stdout: "cap_revoke_eviction": 0, 2024-04-02T14:51:26.210 INFO:teuthology.orchestra.run.smithi177.stdout: "cap_acquisition_throttle": 0, 2024-04-02T14:51:26.210 INFO:teuthology.orchestra.run.smithi177.stdout: "req_getvxattr_latency": { 2024-04-02T14:51:26.210 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-02T14:51:26.210 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-02T14:51:26.210 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-02T14:51:26.210 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-04-02T14:51:26.210 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:26.210 INFO:teuthology.orchestra.run.smithi177.stdout: "mds_sessions": { 2024-04-02T14:51:26.210 INFO:teuthology.orchestra.run.smithi177.stdout: "session_count": 2, 2024-04-02T14:51:26.211 INFO:teuthology.orchestra.run.smithi177.stdout: "session_add": 6, 2024-04-02T14:51:26.211 INFO:teuthology.orchestra.run.smithi177.stdout: "session_remove": 4, 2024-04-02T14:51:26.211 INFO:teuthology.orchestra.run.smithi177.stdout: "sessions_open": 2, 2024-04-02T14:51:26.211 INFO:teuthology.orchestra.run.smithi177.stdout: "sessions_stale": 0, 2024-04-02T14:51:26.211 INFO:teuthology.orchestra.run.smithi177.stdout: "total_load": 5, 2024-04-02T14:51:26.211 INFO:teuthology.orchestra.run.smithi177.stdout: "average_load": 1, 2024-04-02T14:51:26.211 INFO:teuthology.orchestra.run.smithi177.stdout: "avg_session_uptime": 941, 2024-04-02T14:51:26.211 INFO:teuthology.orchestra.run.smithi177.stdout: "mdthresh_evicted": 0 2024-04-02T14:51:26.211 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:26.211 INFO:teuthology.orchestra.run.smithi177.stdout: "mempool": { 2024-04-02T14:51:26.211 INFO:teuthology.orchestra.run.smithi177.stdout: "bloom_filter_bytes": 0, 2024-04-02T14:51:26.211 INFO:teuthology.orchestra.run.smithi177.stdout: "bloom_filter_items": 0, 2024-04-02T14:51:26.211 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_alloc_bytes": 0, 2024-04-02T14:51:26.211 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_alloc_items": 0, 2024-04-02T14:51:26.211 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_data_bytes": 0, 2024-04-02T14:51:26.211 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_data_items": 0, 2024-04-02T14:51:26.211 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-02T14:51:26.212 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_onode_items": 0, 2024-04-02T14:51:26.212 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-02T14:51:26.212 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_meta_items": 0, 2024-04-02T14:51:26.212 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_other_bytes": 0, 2024-04-02T14:51:26.212 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_other_items": 0, 2024-04-02T14:51:26.212 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-02T14:51:26.212 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_buffer_items": 0, 2024-04-02T14:51:26.212 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_extent_bytes": 0, 2024-04-02T14:51:26.212 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_extent_items": 0, 2024-04-02T14:51:26.212 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_blob_bytes": 0, 2024-04-02T14:51:26.212 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_blob_items": 0, 2024-04-02T14:51:26.212 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-02T14:51:26.212 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_shared_blob_items": 0, 2024-04-02T14:51:26.212 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-02T14:51:26.212 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_inline_bl_items": 0, 2024-04-02T14:51:26.213 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_fsck_bytes": 0, 2024-04-02T14:51:26.213 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_fsck_items": 0, 2024-04-02T14:51:26.213 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_txc_bytes": 0, 2024-04-02T14:51:26.213 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_txc_items": 0, 2024-04-02T14:51:26.213 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-02T14:51:26.213 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_writing_deferred_items": 0, 2024-04-02T14:51:26.213 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_writing_bytes": 0, 2024-04-02T14:51:26.213 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_writing_items": 0, 2024-04-02T14:51:26.213 INFO:teuthology.orchestra.run.smithi177.stdout: "bluefs_bytes": 0, 2024-04-02T14:51:26.213 INFO:teuthology.orchestra.run.smithi177.stdout: "bluefs_items": 0, 2024-04-02T14:51:26.213 INFO:teuthology.orchestra.run.smithi177.stdout: "bluefs_file_reader_bytes": 0, 2024-04-02T14:51:26.213 INFO:teuthology.orchestra.run.smithi177.stdout: "bluefs_file_reader_items": 0, 2024-04-02T14:51:26.213 INFO:teuthology.orchestra.run.smithi177.stdout: "bluefs_file_writer_bytes": 0, 2024-04-02T14:51:26.213 INFO:teuthology.orchestra.run.smithi177.stdout: "bluefs_file_writer_items": 0, 2024-04-02T14:51:26.214 INFO:teuthology.orchestra.run.smithi177.stdout: "buffer_anon_bytes": 244203, 2024-04-02T14:51:26.214 INFO:teuthology.orchestra.run.smithi177.stdout: "buffer_anon_items": 121, 2024-04-02T14:51:26.214 INFO:teuthology.orchestra.run.smithi177.stdout: "buffer_meta_bytes": 0, 2024-04-02T14:51:26.214 INFO:teuthology.orchestra.run.smithi177.stdout: "buffer_meta_items": 0, 2024-04-02T14:51:26.214 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_bytes": 0, 2024-04-02T14:51:26.214 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_items": 0, 2024-04-02T14:51:26.214 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_mapbl_bytes": 0, 2024-04-02T14:51:26.214 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_mapbl_items": 0, 2024-04-02T14:51:26.214 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_pglog_bytes": 0, 2024-04-02T14:51:26.214 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_pglog_items": 0, 2024-04-02T14:51:26.214 INFO:teuthology.orchestra.run.smithi177.stdout: "osdmap_bytes": 8136, 2024-04-02T14:51:26.214 INFO:teuthology.orchestra.run.smithi177.stdout: "osdmap_items": 170, 2024-04-02T14:51:26.214 INFO:teuthology.orchestra.run.smithi177.stdout: "osdmap_mapping_bytes": 0, 2024-04-02T14:51:26.214 INFO:teuthology.orchestra.run.smithi177.stdout: "osdmap_mapping_items": 0, 2024-04-02T14:51:26.214 INFO:teuthology.orchestra.run.smithi177.stdout: "pgmap_bytes": 0, 2024-04-02T14:51:26.215 INFO:teuthology.orchestra.run.smithi177.stdout: "pgmap_items": 0, 2024-04-02T14:51:26.215 INFO:teuthology.orchestra.run.smithi177.stdout: "mds_co_bytes": 200873, 2024-04-02T14:51:26.215 INFO:teuthology.orchestra.run.smithi177.stdout: "mds_co_items": 1007, 2024-04-02T14:51:26.215 INFO:teuthology.orchestra.run.smithi177.stdout: "unittest_1_bytes": 0, 2024-04-02T14:51:26.215 INFO:teuthology.orchestra.run.smithi177.stdout: "unittest_1_items": 0, 2024-04-02T14:51:26.215 INFO:teuthology.orchestra.run.smithi177.stdout: "unittest_2_bytes": 0, 2024-04-02T14:51:26.215 INFO:teuthology.orchestra.run.smithi177.stdout: "unittest_2_items": 0 2024-04-02T14:51:26.215 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:26.215 INFO:teuthology.orchestra.run.smithi177.stdout: "objecter": { 2024-04-02T14:51:26.215 INFO:teuthology.orchestra.run.smithi177.stdout: "op_active": 0, 2024-04-02T14:51:26.215 INFO:teuthology.orchestra.run.smithi177.stdout: "op_laggy": 0, 2024-04-02T14:51:26.215 INFO:teuthology.orchestra.run.smithi177.stdout: "op_send": 53, 2024-04-02T14:51:26.215 INFO:teuthology.orchestra.run.smithi177.stdout: "op_send_bytes": 14256, 2024-04-02T14:51:26.215 INFO:teuthology.orchestra.run.smithi177.stdout: "op_resend": 33, 2024-04-02T14:51:26.215 INFO:teuthology.orchestra.run.smithi177.stdout: "op_reply": 53, 2024-04-02T14:51:26.216 INFO:teuthology.orchestra.run.smithi177.stdout: "op_latency": { 2024-04-02T14:51:26.216 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 53, 2024-04-02T14:51:26.216 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.372989856, 2024-04-02T14:51:26.216 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.007037544 2024-04-02T14:51:26.216 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:26.216 INFO:teuthology.orchestra.run.smithi177.stdout: "op_inflight": 0, 2024-04-02T14:51:26.216 INFO:teuthology.orchestra.run.smithi177.stdout: "oplen_avg": { 2024-04-02T14:51:26.216 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 53, 2024-04-02T14:51:26.216 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 78 2024-04-02T14:51:26.216 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:26.216 INFO:teuthology.orchestra.run.smithi177.stdout: "op": 53, 2024-04-02T14:51:26.216 INFO:teuthology.orchestra.run.smithi177.stdout: "op_r": 0, 2024-04-02T14:51:26.216 INFO:teuthology.orchestra.run.smithi177.stdout: "op_w": 53, 2024-04-02T14:51:26.216 INFO:teuthology.orchestra.run.smithi177.stdout: "op_rmw": 0, 2024-04-02T14:51:26.216 INFO:teuthology.orchestra.run.smithi177.stdout: "op_pg": 0, 2024-04-02T14:51:26.217 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_stat": 0, 2024-04-02T14:51:26.217 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_create": 10, 2024-04-02T14:51:26.217 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_read": 0, 2024-04-02T14:51:26.217 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_write": 11, 2024-04-02T14:51:26.217 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_writefull": 10, 2024-04-02T14:51:26.217 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_writesame": 0, 2024-04-02T14:51:26.217 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_append": 0, 2024-04-02T14:51:26.217 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_zero": 0, 2024-04-02T14:51:26.217 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_truncate": 0, 2024-04-02T14:51:26.217 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_delete": 6, 2024-04-02T14:51:26.217 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_mapext": 0, 2024-04-02T14:51:26.217 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_sparse_read": 0, 2024-04-02T14:51:26.217 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_clonerange": 0, 2024-04-02T14:51:26.217 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_getxattr": 0, 2024-04-02T14:51:26.217 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_setxattr": 20, 2024-04-02T14:51:26.217 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_cmpxattr": 0, 2024-04-02T14:51:26.218 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_rmxattr": 0, 2024-04-02T14:51:26.218 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_resetxattrs": 0, 2024-04-02T14:51:26.218 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_call": 0, 2024-04-02T14:51:26.218 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_watch": 0, 2024-04-02T14:51:26.218 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_notify": 0, 2024-04-02T14:51:26.218 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_src_cmpxattr": 0, 2024-04-02T14:51:26.218 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_pgls": 0, 2024-04-02T14:51:26.218 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_pgls_filter": 0, 2024-04-02T14:51:26.218 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_other": 0, 2024-04-02T14:51:26.218 INFO:teuthology.orchestra.run.smithi177.stdout: "linger_active": 0, 2024-04-02T14:51:26.218 INFO:teuthology.orchestra.run.smithi177.stdout: "linger_send": 0, 2024-04-02T14:51:26.218 INFO:teuthology.orchestra.run.smithi177.stdout: "linger_resend": 0, 2024-04-02T14:51:26.218 INFO:teuthology.orchestra.run.smithi177.stdout: "linger_ping": 0, 2024-04-02T14:51:26.218 INFO:teuthology.orchestra.run.smithi177.stdout: "poolop_active": 0, 2024-04-02T14:51:26.218 INFO:teuthology.orchestra.run.smithi177.stdout: "poolop_send": 0, 2024-04-02T14:51:26.218 INFO:teuthology.orchestra.run.smithi177.stdout: "poolop_resend": 0, 2024-04-02T14:51:26.219 INFO:teuthology.orchestra.run.smithi177.stdout: "poolstat_active": 0, 2024-04-02T14:51:26.219 INFO:teuthology.orchestra.run.smithi177.stdout: "poolstat_send": 0, 2024-04-02T14:51:26.219 INFO:teuthology.orchestra.run.smithi177.stdout: "poolstat_resend": 0, 2024-04-02T14:51:26.219 INFO:teuthology.orchestra.run.smithi177.stdout: "statfs_active": 0, 2024-04-02T14:51:26.219 INFO:teuthology.orchestra.run.smithi177.stdout: "statfs_send": 0, 2024-04-02T14:51:26.219 INFO:teuthology.orchestra.run.smithi177.stdout: "statfs_resend": 0, 2024-04-02T14:51:26.219 INFO:teuthology.orchestra.run.smithi177.stdout: "command_active": 0, 2024-04-02T14:51:26.219 INFO:teuthology.orchestra.run.smithi177.stdout: "command_send": 0, 2024-04-02T14:51:26.219 INFO:teuthology.orchestra.run.smithi177.stdout: "command_resend": 0, 2024-04-02T14:51:26.219 INFO:teuthology.orchestra.run.smithi177.stdout: "map_epoch": 111, 2024-04-02T14:51:26.219 INFO:teuthology.orchestra.run.smithi177.stdout: "map_full": 0, 2024-04-02T14:51:26.219 INFO:teuthology.orchestra.run.smithi177.stdout: "map_inc": 23, 2024-04-02T14:51:26.219 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_sessions": 8, 2024-04-02T14:51:26.219 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_session_open": 13, 2024-04-02T14:51:26.219 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_session_close": 5, 2024-04-02T14:51:26.219 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_laggy": 0, 2024-04-02T14:51:26.220 INFO:teuthology.orchestra.run.smithi177.stdout: "omap_wr": 19, 2024-04-02T14:51:26.220 INFO:teuthology.orchestra.run.smithi177.stdout: "omap_rd": 0, 2024-04-02T14:51:26.220 INFO:teuthology.orchestra.run.smithi177.stdout: "omap_del": 2 2024-04-02T14:51:26.220 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:26.220 INFO:teuthology.orchestra.run.smithi177.stdout: "oft": { 2024-04-02T14:51:26.220 INFO:teuthology.orchestra.run.smithi177.stdout: "omap_total_objs": 1, 2024-04-02T14:51:26.220 INFO:teuthology.orchestra.run.smithi177.stdout: "omap_total_kv_pairs": 0, 2024-04-02T14:51:26.220 INFO:teuthology.orchestra.run.smithi177.stdout: "omap_total_updates": 0, 2024-04-02T14:51:26.220 INFO:teuthology.orchestra.run.smithi177.stdout: "omap_total_removes": 0 2024-04-02T14:51:26.220 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:26.220 INFO:teuthology.orchestra.run.smithi177.stdout: "osd-slow-ops": { 2024-04-02T14:51:26.220 INFO:teuthology.orchestra.run.smithi177.stdout: "slow_ops_count": 0 2024-04-02T14:51:26.220 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:26.220 INFO:teuthology.orchestra.run.smithi177.stdout: "purge_queue": { 2024-04-02T14:51:26.220 INFO:teuthology.orchestra.run.smithi177.stdout: "pq_executing_ops": 0, 2024-04-02T14:51:26.220 INFO:teuthology.orchestra.run.smithi177.stdout: "pq_executing_ops_high_water": 0, 2024-04-02T14:51:26.221 INFO:teuthology.orchestra.run.smithi177.stdout: "pq_executing": 0, 2024-04-02T14:51:26.221 INFO:teuthology.orchestra.run.smithi177.stdout: "pq_executing_high_water": 0, 2024-04-02T14:51:26.221 INFO:teuthology.orchestra.run.smithi177.stdout: "pq_executed_ops": 0, 2024-04-02T14:51:26.221 INFO:teuthology.orchestra.run.smithi177.stdout: "pq_executed": 0, 2024-04-02T14:51:26.221 INFO:teuthology.orchestra.run.smithi177.stdout: "pq_item_in_journal": 0 2024-04-02T14:51:26.221 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:26.221 INFO:teuthology.orchestra.run.smithi177.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-02T14:51:26.221 INFO:teuthology.orchestra.run.smithi177.stdout: "val": 0, 2024-04-02T14:51:26.221 INFO:teuthology.orchestra.run.smithi177.stdout: "max": 104857600, 2024-04-02T14:51:26.221 INFO:teuthology.orchestra.run.smithi177.stdout: "get_started": 0, 2024-04-02T14:51:26.221 INFO:teuthology.orchestra.run.smithi177.stdout: "get": 4406, 2024-04-02T14:51:26.221 INFO:teuthology.orchestra.run.smithi177.stdout: "get_sum": 2346401, 2024-04-02T14:51:26.221 INFO:teuthology.orchestra.run.smithi177.stdout: "get_or_fail_fail": 0, 2024-04-02T14:51:26.221 INFO:teuthology.orchestra.run.smithi177.stdout: "get_or_fail_success": 4406, 2024-04-02T14:51:26.221 INFO:teuthology.orchestra.run.smithi177.stdout: "take": 0, 2024-04-02T14:51:26.221 INFO:teuthology.orchestra.run.smithi177.stdout: "take_sum": 0, 2024-04-02T14:51:26.222 INFO:teuthology.orchestra.run.smithi177.stdout: "put": 4406, 2024-04-02T14:51:26.222 INFO:teuthology.orchestra.run.smithi177.stdout: "put_sum": 2346401, 2024-04-02T14:51:26.222 INFO:teuthology.orchestra.run.smithi177.stdout: "wait": { 2024-04-02T14:51:26.222 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-02T14:51:26.222 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-02T14:51:26.222 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-02T14:51:26.222 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-04-02T14:51:26.222 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:26.222 INFO:teuthology.orchestra.run.smithi177.stdout: "throttle-objecter_bytes": { 2024-04-02T14:51:26.222 INFO:teuthology.orchestra.run.smithi177.stdout: "val": 0, 2024-04-02T14:51:26.222 INFO:teuthology.orchestra.run.smithi177.stdout: "max": 104857600, 2024-04-02T14:51:26.222 INFO:teuthology.orchestra.run.smithi177.stdout: "get_started": 0, 2024-04-02T14:51:26.222 INFO:teuthology.orchestra.run.smithi177.stdout: "get": 0, 2024-04-02T14:51:26.222 INFO:teuthology.orchestra.run.smithi177.stdout: "get_sum": 0, 2024-04-02T14:51:26.222 INFO:teuthology.orchestra.run.smithi177.stdout: "get_or_fail_fail": 0, 2024-04-02T14:51:26.222 INFO:teuthology.orchestra.run.smithi177.stdout: "get_or_fail_success": 0, 2024-04-02T14:51:26.223 INFO:teuthology.orchestra.run.smithi177.stdout: "take": 53, 2024-04-02T14:51:26.223 INFO:teuthology.orchestra.run.smithi177.stdout: "take_sum": 14256, 2024-04-02T14:51:26.223 INFO:teuthology.orchestra.run.smithi177.stdout: "put": 53, 2024-04-02T14:51:26.223 INFO:teuthology.orchestra.run.smithi177.stdout: "put_sum": 14256, 2024-04-02T14:51:26.223 INFO:teuthology.orchestra.run.smithi177.stdout: "wait": { 2024-04-02T14:51:26.223 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-02T14:51:26.223 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-02T14:51:26.223 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-02T14:51:26.223 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-04-02T14:51:26.223 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:26.223 INFO:teuthology.orchestra.run.smithi177.stdout: "throttle-objecter_ops": { 2024-04-02T14:51:26.223 INFO:teuthology.orchestra.run.smithi177.stdout: "val": 0, 2024-04-02T14:51:26.223 INFO:teuthology.orchestra.run.smithi177.stdout: "max": 1024, 2024-04-02T14:51:26.223 INFO:teuthology.orchestra.run.smithi177.stdout: "get_started": 0, 2024-04-02T14:51:26.223 INFO:teuthology.orchestra.run.smithi177.stdout: "get": 0, 2024-04-02T14:51:26.223 INFO:teuthology.orchestra.run.smithi177.stdout: "get_sum": 0, 2024-04-02T14:51:26.224 INFO:teuthology.orchestra.run.smithi177.stdout: "get_or_fail_fail": 0, 2024-04-02T14:51:26.224 INFO:teuthology.orchestra.run.smithi177.stdout: "get_or_fail_success": 0, 2024-04-02T14:51:26.224 INFO:teuthology.orchestra.run.smithi177.stdout: "take": 53, 2024-04-02T14:51:26.224 INFO:teuthology.orchestra.run.smithi177.stdout: "take_sum": 53, 2024-04-02T14:51:26.224 INFO:teuthology.orchestra.run.smithi177.stdout: "put": 53, 2024-04-02T14:51:26.224 INFO:teuthology.orchestra.run.smithi177.stdout: "put_sum": 53, 2024-04-02T14:51:26.224 INFO:teuthology.orchestra.run.smithi177.stdout: "wait": { 2024-04-02T14:51:26.224 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-02T14:51:26.224 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-02T14:51:26.224 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-02T14:51:26.224 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-04-02T14:51:26.224 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:26.224 INFO:teuthology.orchestra.run.smithi177.stdout: "throttle-write_buf_throttle": { 2024-04-02T14:51:26.224 INFO:teuthology.orchestra.run.smithi177.stdout: "val": 0, 2024-04-02T14:51:26.224 INFO:teuthology.orchestra.run.smithi177.stdout: "max": 3758096384, 2024-04-02T14:51:26.224 INFO:teuthology.orchestra.run.smithi177.stdout: "get_started": 0, 2024-04-02T14:51:26.225 INFO:teuthology.orchestra.run.smithi177.stdout: "get": 0, 2024-04-02T14:51:26.225 INFO:teuthology.orchestra.run.smithi177.stdout: "get_sum": 0, 2024-04-02T14:51:26.225 INFO:teuthology.orchestra.run.smithi177.stdout: "get_or_fail_fail": 0, 2024-04-02T14:51:26.225 INFO:teuthology.orchestra.run.smithi177.stdout: "get_or_fail_success": 0, 2024-04-02T14:51:26.225 INFO:teuthology.orchestra.run.smithi177.stdout: "take": 0, 2024-04-02T14:51:26.225 INFO:teuthology.orchestra.run.smithi177.stdout: "take_sum": 0, 2024-04-02T14:51:26.225 INFO:teuthology.orchestra.run.smithi177.stdout: "put": 0, 2024-04-02T14:51:26.225 INFO:teuthology.orchestra.run.smithi177.stdout: "put_sum": 0, 2024-04-02T14:51:26.225 INFO:teuthology.orchestra.run.smithi177.stdout: "wait": { 2024-04-02T14:51:26.225 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-02T14:51:26.225 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-02T14:51:26.225 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-02T14:51:26.225 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-04-02T14:51:26.225 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:26.225 INFO:teuthology.orchestra.run.smithi177.stdout: "throttle-write_buf_throttle-0x5647916fe500": { 2024-04-02T14:51:26.225 INFO:teuthology.orchestra.run.smithi177.stdout: "val": 0, 2024-04-02T14:51:26.225 INFO:teuthology.orchestra.run.smithi177.stdout: "max": 3758096384, 2024-04-02T14:51:26.226 INFO:teuthology.orchestra.run.smithi177.stdout: "get_started": 0, 2024-04-02T14:51:26.226 INFO:teuthology.orchestra.run.smithi177.stdout: "get": 12, 2024-04-02T14:51:26.226 INFO:teuthology.orchestra.run.smithi177.stdout: "get_sum": 3413, 2024-04-02T14:51:26.226 INFO:teuthology.orchestra.run.smithi177.stdout: "get_or_fail_fail": 0, 2024-04-02T14:51:26.226 INFO:teuthology.orchestra.run.smithi177.stdout: "get_or_fail_success": 12, 2024-04-02T14:51:26.226 INFO:teuthology.orchestra.run.smithi177.stdout: "take": 0, 2024-04-02T14:51:26.226 INFO:teuthology.orchestra.run.smithi177.stdout: "take_sum": 0, 2024-04-02T14:51:26.226 INFO:teuthology.orchestra.run.smithi177.stdout: "put": 11, 2024-04-02T14:51:26.226 INFO:teuthology.orchestra.run.smithi177.stdout: "put_sum": 3413, 2024-04-02T14:51:26.226 INFO:teuthology.orchestra.run.smithi177.stdout: "wait": { 2024-04-02T14:51:26.226 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-02T14:51:26.226 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-02T14:51:26.226 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-02T14:51:26.226 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-04-02T14:51:26.226 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-04-02T14:51:26.226 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-04-02T14:51:26.427 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:51:26 smithi177 ceph-mon[36431]: pgmap v519: 97 pgs: 97 active+clean; 3.9 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 11 KiB/s wr, 10 op/s 2024-04-02T14:51:26.601 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:51:26 smithi092 ceph-mon[36497]: pgmap v519: 97 pgs: 97 active+clean; 3.9 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 11 KiB/s wr, 10 op/s 2024-04-02T14:51:26.679 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:51:26 smithi090 ceph-mon[31674]: pgmap v519: 97 pgs: 97 active+clean; 3.9 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 11 KiB/s wr, 10 op/s 2024-04-02T14:51:27.025 INFO:tasks.check_counter:Daemon mds.i mds.exported=0 2024-04-02T14:51:27.025 INFO:tasks.check_counter:Daemon mds.i mds.imported=0 2024-04-02T14:51:27.025 INFO:tasks.check_counter:Daemon mds.i mds_cache.dir_update=0 2024-04-02T14:51:27.025 INFO:tasks.check_counter:Daemon mds.i mds_cache.dir_update_receipt=102 2024-04-02T14:51:27.025 INFO:tasks.check_counter:Daemon mds.i mds.root_rsnaps=6 2024-04-02T14:51:27.025 INFO:tasks.check_counter:Daemon mds.i mds_server.req_mksnap_latency.avgcount=0 2024-04-02T14:51:27.025 INFO:tasks.check_counter:Daemon mds.i mds_server.req_rmsnap_latency.avgcount=0 2024-04-02T14:51:27.025 DEBUG:tasks.check_counter:Getting stats from l 2024-04-02T14:51:27.025 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:191d963ac9de7322da8aff20b567b3fa458e2ef6 shell --fsid 6449448a-f0fc-11ee-b647-cb9ed24678a4 -- ceph daemon mds.l perf dump 2024-04-02T14:51:27.262 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/mon.c/config 2024-04-02T14:51:28.412 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:51:28 smithi177 ceph-mon[36431]: pgmap v520: 97 pgs: 97 active+clean; 3.9 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 5.7 KiB/s wr, 10 op/s 2024-04-02T14:51:28.601 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:51:28 smithi092 ceph-mon[36497]: pgmap v520: 97 pgs: 97 active+clean; 3.9 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 5.7 KiB/s wr, 10 op/s 2024-04-02T14:51:28.679 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:51:28 smithi090 ceph-mon[31674]: pgmap v520: 97 pgs: 97 active+clean; 3.9 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 5.7 KiB/s wr, 10 op/s 2024-04-02T14:51:28.946 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-04-02T14:51:28.946 INFO:teuthology.orchestra.run.smithi177.stdout: "AsyncMessenger::Worker-0": { 2024-04-02T14:51:28.946 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_messages": 1134, 2024-04-02T14:51:28.946 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_messages": 1133, 2024-04-02T14:51:28.947 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_bytes": 382779, 2024-04-02T14:51:28.947 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_bytes": 359471, 2024-04-02T14:51:28.947 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_created_connections": 7, 2024-04-02T14:51:28.947 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_active_connections": 2, 2024-04-02T14:51:28.947 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_total_time": 0.116514994, 2024-04-02T14:51:28.947 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_send_time": 0.033327122, 2024-04-02T14:51:28.947 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_recv_time": 0.032619063, 2024-04-02T14:51:28.947 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_fast_dispatch_time": 0.039487445, 2024-04-02T14:51:28.947 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_messages_queue_lat": { 2024-04-02T14:51:28.947 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 1132, 2024-04-02T14:51:28.947 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.058282577, 2024-04-02T14:51:28.947 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000051486 2024-04-02T14:51:28.947 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:28.947 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_handle_ack_lat": { 2024-04-02T14:51:28.947 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-02T14:51:28.948 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-02T14:51:28.948 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-02T14:51:28.948 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:28.948 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_encrypted_bytes": 3072, 2024-04-02T14:51:28.948 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_encrypted_bytes": 448 2024-04-02T14:51:28.948 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:28.948 INFO:teuthology.orchestra.run.smithi177.stdout: "AsyncMessenger::Worker-1": { 2024-04-02T14:51:28.948 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_messages": 2460, 2024-04-02T14:51:28.948 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_messages": 2448, 2024-04-02T14:51:28.948 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_bytes": 2193668, 2024-04-02T14:51:28.948 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_bytes": 869287, 2024-04-02T14:51:28.948 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_created_connections": 4, 2024-04-02T14:51:28.948 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_active_connections": 3, 2024-04-02T14:51:28.948 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_total_time": 0.193083077, 2024-04-02T14:51:28.949 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_send_time": 0.057874383, 2024-04-02T14:51:28.949 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_recv_time": 0.077338903, 2024-04-02T14:51:28.949 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_fast_dispatch_time": 0.058099314, 2024-04-02T14:51:28.949 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_messages_queue_lat": { 2024-04-02T14:51:28.949 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 2447, 2024-04-02T14:51:28.949 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.105906035, 2024-04-02T14:51:28.949 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000043279 2024-04-02T14:51:28.949 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:28.949 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_handle_ack_lat": { 2024-04-02T14:51:28.949 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-02T14:51:28.949 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-02T14:51:28.949 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-02T14:51:28.949 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:28.949 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_encrypted_bytes": 1656832, 2024-04-02T14:51:28.949 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_encrypted_bytes": 199840 2024-04-02T14:51:28.950 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:28.950 INFO:teuthology.orchestra.run.smithi177.stdout: "AsyncMessenger::Worker-2": { 2024-04-02T14:51:28.950 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_messages": 7, 2024-04-02T14:51:28.950 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_messages": 248, 2024-04-02T14:51:28.950 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_bytes": 1988, 2024-04-02T14:51:28.950 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_bytes": 750741, 2024-04-02T14:51:28.950 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_created_connections": 10, 2024-04-02T14:51:28.950 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_active_connections": 5, 2024-04-02T14:51:28.950 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_total_time": 0.013793304, 2024-04-02T14:51:28.950 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_send_time": 0.008358043, 2024-04-02T14:51:28.950 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_recv_time": 0.002756227, 2024-04-02T14:51:28.950 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_fast_dispatch_time": 0.000084195, 2024-04-02T14:51:28.950 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_messages_queue_lat": { 2024-04-02T14:51:28.950 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 245, 2024-04-02T14:51:28.950 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.028234957, 2024-04-02T14:51:28.951 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000115244 2024-04-02T14:51:28.951 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:28.951 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_handle_ack_lat": { 2024-04-02T14:51:28.951 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-02T14:51:28.951 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-02T14:51:28.951 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-02T14:51:28.951 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:28.951 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_encrypted_bytes": 640, 2024-04-02T14:51:28.951 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_encrypted_bytes": 749776 2024-04-02T14:51:28.951 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:28.951 INFO:teuthology.orchestra.run.smithi177.stdout: "cct": { 2024-04-02T14:51:28.951 INFO:teuthology.orchestra.run.smithi177.stdout: "total_workers": 1, 2024-04-02T14:51:28.951 INFO:teuthology.orchestra.run.smithi177.stdout: "unhealthy_workers": 0 2024-04-02T14:51:28.951 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:28.952 INFO:teuthology.orchestra.run.smithi177.stdout: "finisher-MDSRank": { 2024-04-02T14:51:28.952 INFO:teuthology.orchestra.run.smithi177.stdout: "queue_len": 0, 2024-04-02T14:51:28.952 INFO:teuthology.orchestra.run.smithi177.stdout: "complete_latency": { 2024-04-02T14:51:28.952 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 5482, 2024-04-02T14:51:28.952 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.133826429, 2024-04-02T14:51:28.952 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000024411 2024-04-02T14:51:28.952 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-04-02T14:51:28.952 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:28.952 INFO:teuthology.orchestra.run.smithi177.stdout: "finisher-PurgeQueue": { 2024-04-02T14:51:28.952 INFO:teuthology.orchestra.run.smithi177.stdout: "queue_len": 0, 2024-04-02T14:51:28.952 INFO:teuthology.orchestra.run.smithi177.stdout: "complete_latency": { 2024-04-02T14:51:28.952 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-02T14:51:28.952 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-02T14:51:28.952 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-02T14:51:28.952 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-04-02T14:51:28.953 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:28.953 INFO:teuthology.orchestra.run.smithi177.stdout: "mds": { 2024-04-02T14:51:28.953 INFO:teuthology.orchestra.run.smithi177.stdout: "request": 0, 2024-04-02T14:51:28.953 INFO:teuthology.orchestra.run.smithi177.stdout: "reply": 0, 2024-04-02T14:51:28.953 INFO:teuthology.orchestra.run.smithi177.stdout: "reply_latency": { 2024-04-02T14:51:28.953 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-02T14:51:28.953 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-02T14:51:28.953 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-02T14:51:28.953 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:28.953 INFO:teuthology.orchestra.run.smithi177.stdout: "slow_reply": 0, 2024-04-02T14:51:28.953 INFO:teuthology.orchestra.run.smithi177.stdout: "forward": 0, 2024-04-02T14:51:28.953 INFO:teuthology.orchestra.run.smithi177.stdout: "dir_fetch_complete": 0, 2024-04-02T14:51:28.953 INFO:teuthology.orchestra.run.smithi177.stdout: "dir_fetch_keys": 0, 2024-04-02T14:51:28.953 INFO:teuthology.orchestra.run.smithi177.stdout: "dir_commit": 0, 2024-04-02T14:51:28.953 INFO:teuthology.orchestra.run.smithi177.stdout: "dir_split": 0, 2024-04-02T14:51:28.954 INFO:teuthology.orchestra.run.smithi177.stdout: "dir_merge": 0, 2024-04-02T14:51:28.954 INFO:teuthology.orchestra.run.smithi177.stdout: "inodes": 0, 2024-04-02T14:51:28.954 INFO:teuthology.orchestra.run.smithi177.stdout: "inodes_top": 0, 2024-04-02T14:51:28.954 INFO:teuthology.orchestra.run.smithi177.stdout: "inodes_bottom": 0, 2024-04-02T14:51:28.954 INFO:teuthology.orchestra.run.smithi177.stdout: "inodes_pin_tail": 0, 2024-04-02T14:51:28.954 INFO:teuthology.orchestra.run.smithi177.stdout: "inodes_pinned": 0, 2024-04-02T14:51:28.954 INFO:teuthology.orchestra.run.smithi177.stdout: "inodes_expired": 0, 2024-04-02T14:51:28.954 INFO:teuthology.orchestra.run.smithi177.stdout: "inodes_with_caps": 0, 2024-04-02T14:51:28.954 INFO:teuthology.orchestra.run.smithi177.stdout: "caps": 0, 2024-04-02T14:51:28.954 INFO:teuthology.orchestra.run.smithi177.stdout: "subtrees": 1, 2024-04-02T14:51:28.954 INFO:teuthology.orchestra.run.smithi177.stdout: "traverse": 0, 2024-04-02T14:51:28.954 INFO:teuthology.orchestra.run.smithi177.stdout: "traverse_hit": 0, 2024-04-02T14:51:28.954 INFO:teuthology.orchestra.run.smithi177.stdout: "traverse_forward": 0, 2024-04-02T14:51:28.954 INFO:teuthology.orchestra.run.smithi177.stdout: "traverse_discover": 0, 2024-04-02T14:51:28.955 INFO:teuthology.orchestra.run.smithi177.stdout: "traverse_dir_fetch": 0, 2024-04-02T14:51:28.955 INFO:teuthology.orchestra.run.smithi177.stdout: "traverse_remote_ino": 0, 2024-04-02T14:51:28.955 INFO:teuthology.orchestra.run.smithi177.stdout: "traverse_lock": 0, 2024-04-02T14:51:28.955 INFO:teuthology.orchestra.run.smithi177.stdout: "load_cent": 0, 2024-04-02T14:51:28.955 INFO:teuthology.orchestra.run.smithi177.stdout: "q": 0, 2024-04-02T14:51:28.955 INFO:teuthology.orchestra.run.smithi177.stdout: "exported": 0, 2024-04-02T14:51:28.955 INFO:teuthology.orchestra.run.smithi177.stdout: "exported_inodes": 0, 2024-04-02T14:51:28.955 INFO:teuthology.orchestra.run.smithi177.stdout: "imported": 0, 2024-04-02T14:51:28.955 INFO:teuthology.orchestra.run.smithi177.stdout: "imported_inodes": 0, 2024-04-02T14:51:28.955 INFO:teuthology.orchestra.run.smithi177.stdout: "openino_dir_fetch": 0, 2024-04-02T14:51:28.955 INFO:teuthology.orchestra.run.smithi177.stdout: "openino_backtrace_fetch": 0, 2024-04-02T14:51:28.955 INFO:teuthology.orchestra.run.smithi177.stdout: "openino_peer_discover": 0, 2024-04-02T14:51:28.955 INFO:teuthology.orchestra.run.smithi177.stdout: "root_rfiles": 0, 2024-04-02T14:51:28.955 INFO:teuthology.orchestra.run.smithi177.stdout: "root_rbytes": 0, 2024-04-02T14:51:28.955 INFO:teuthology.orchestra.run.smithi177.stdout: "root_rsnaps": 0, 2024-04-02T14:51:28.955 INFO:teuthology.orchestra.run.smithi177.stdout: "scrub_backtrace_fetch": 0, 2024-04-02T14:51:28.956 INFO:teuthology.orchestra.run.smithi177.stdout: "scrub_set_tag": 0, 2024-04-02T14:51:28.956 INFO:teuthology.orchestra.run.smithi177.stdout: "scrub_backtrace_repaired": 0, 2024-04-02T14:51:28.956 INFO:teuthology.orchestra.run.smithi177.stdout: "scrub_inotable_repaired": 0, 2024-04-02T14:51:28.956 INFO:teuthology.orchestra.run.smithi177.stdout: "scrub_dir_inodes": 0, 2024-04-02T14:51:28.956 INFO:teuthology.orchestra.run.smithi177.stdout: "scrub_dir_base_inodes": 0, 2024-04-02T14:51:28.956 INFO:teuthology.orchestra.run.smithi177.stdout: "scrub_dirfrag_rstats": 0, 2024-04-02T14:51:28.956 INFO:teuthology.orchestra.run.smithi177.stdout: "scrub_file_inodes": 0, 2024-04-02T14:51:28.956 INFO:teuthology.orchestra.run.smithi177.stdout: "handle_inode_file_caps": 0, 2024-04-02T14:51:28.956 INFO:teuthology.orchestra.run.smithi177.stdout: "ceph_cap_op_revoke": 0, 2024-04-02T14:51:28.956 INFO:teuthology.orchestra.run.smithi177.stdout: "ceph_cap_op_grant": 0, 2024-04-02T14:51:28.956 INFO:teuthology.orchestra.run.smithi177.stdout: "ceph_cap_op_trunc": 0, 2024-04-02T14:51:28.956 INFO:teuthology.orchestra.run.smithi177.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-02T14:51:28.956 INFO:teuthology.orchestra.run.smithi177.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-02T14:51:28.956 INFO:teuthology.orchestra.run.smithi177.stdout: "handle_client_caps": 0, 2024-04-02T14:51:28.956 INFO:teuthology.orchestra.run.smithi177.stdout: "handle_client_caps_dirty": 0, 2024-04-02T14:51:28.957 INFO:teuthology.orchestra.run.smithi177.stdout: "handle_client_cap_release": 0, 2024-04-02T14:51:28.957 INFO:teuthology.orchestra.run.smithi177.stdout: "process_request_cap_release": 0 2024-04-02T14:51:28.957 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:28.957 INFO:teuthology.orchestra.run.smithi177.stdout: "mds_cache": { 2024-04-02T14:51:28.957 INFO:teuthology.orchestra.run.smithi177.stdout: "dir_update": 0, 2024-04-02T14:51:28.957 INFO:teuthology.orchestra.run.smithi177.stdout: "dir_update_receipt": 0, 2024-04-02T14:51:28.957 INFO:teuthology.orchestra.run.smithi177.stdout: "dir_try_discover": 0, 2024-04-02T14:51:28.957 INFO:teuthology.orchestra.run.smithi177.stdout: "dir_send_discover": 0, 2024-04-02T14:51:28.957 INFO:teuthology.orchestra.run.smithi177.stdout: "dir_handle_discover": 0, 2024-04-02T14:51:28.957 INFO:teuthology.orchestra.run.smithi177.stdout: "num_strays": 0, 2024-04-02T14:51:28.957 INFO:teuthology.orchestra.run.smithi177.stdout: "num_strays_delayed": 0, 2024-04-02T14:51:28.957 INFO:teuthology.orchestra.run.smithi177.stdout: "num_strays_enqueuing": 0, 2024-04-02T14:51:28.957 INFO:teuthology.orchestra.run.smithi177.stdout: "strays_created": 0, 2024-04-02T14:51:28.957 INFO:teuthology.orchestra.run.smithi177.stdout: "strays_enqueued": 0, 2024-04-02T14:51:28.957 INFO:teuthology.orchestra.run.smithi177.stdout: "strays_reintegrated": 0, 2024-04-02T14:51:28.958 INFO:teuthology.orchestra.run.smithi177.stdout: "strays_migrated": 0, 2024-04-02T14:51:28.958 INFO:teuthology.orchestra.run.smithi177.stdout: "num_recovering_processing": 0, 2024-04-02T14:51:28.958 INFO:teuthology.orchestra.run.smithi177.stdout: "num_recovering_enqueued": 0, 2024-04-02T14:51:28.958 INFO:teuthology.orchestra.run.smithi177.stdout: "num_recovering_prioritized": 0, 2024-04-02T14:51:28.958 INFO:teuthology.orchestra.run.smithi177.stdout: "recovery_started": 0, 2024-04-02T14:51:28.958 INFO:teuthology.orchestra.run.smithi177.stdout: "recovery_completed": 0, 2024-04-02T14:51:28.958 INFO:teuthology.orchestra.run.smithi177.stdout: "ireq_quiesce_path": 0, 2024-04-02T14:51:28.958 INFO:teuthology.orchestra.run.smithi177.stdout: "ireq_quiesce_inode": 0, 2024-04-02T14:51:28.958 INFO:teuthology.orchestra.run.smithi177.stdout: "ireq_enqueue_scrub": 0, 2024-04-02T14:51:28.958 INFO:teuthology.orchestra.run.smithi177.stdout: "ireq_exportdir": 0, 2024-04-02T14:51:28.958 INFO:teuthology.orchestra.run.smithi177.stdout: "ireq_flush": 0, 2024-04-02T14:51:28.958 INFO:teuthology.orchestra.run.smithi177.stdout: "ireq_fragmentdir": 0, 2024-04-02T14:51:28.958 INFO:teuthology.orchestra.run.smithi177.stdout: "ireq_fragstats": 0, 2024-04-02T14:51:28.958 INFO:teuthology.orchestra.run.smithi177.stdout: "ireq_inodestats": 0 2024-04-02T14:51:28.959 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:28.959 INFO:teuthology.orchestra.run.smithi177.stdout: "mds_log": { 2024-04-02T14:51:28.959 INFO:teuthology.orchestra.run.smithi177.stdout: "evlrg": 0, 2024-04-02T14:51:28.959 INFO:teuthology.orchestra.run.smithi177.stdout: "evadd": 0, 2024-04-02T14:51:28.959 INFO:teuthology.orchestra.run.smithi177.stdout: "evex": 0, 2024-04-02T14:51:28.959 INFO:teuthology.orchestra.run.smithi177.stdout: "evtrm": 0, 2024-04-02T14:51:28.959 INFO:teuthology.orchestra.run.smithi177.stdout: "ev": 3, 2024-04-02T14:51:28.959 INFO:teuthology.orchestra.run.smithi177.stdout: "evexg": 0, 2024-04-02T14:51:28.959 INFO:teuthology.orchestra.run.smithi177.stdout: "evexd": 0, 2024-04-02T14:51:28.959 INFO:teuthology.orchestra.run.smithi177.stdout: "segadd": 0, 2024-04-02T14:51:28.959 INFO:teuthology.orchestra.run.smithi177.stdout: "segex": 0, 2024-04-02T14:51:28.959 INFO:teuthology.orchestra.run.smithi177.stdout: "segtrm": 0, 2024-04-02T14:51:28.959 INFO:teuthology.orchestra.run.smithi177.stdout: "seg": 2, 2024-04-02T14:51:28.959 INFO:teuthology.orchestra.run.smithi177.stdout: "segmjr": 2, 2024-04-02T14:51:28.959 INFO:teuthology.orchestra.run.smithi177.stdout: "segexg": 0, 2024-04-02T14:51:28.960 INFO:teuthology.orchestra.run.smithi177.stdout: "segexd": 0, 2024-04-02T14:51:28.960 INFO:teuthology.orchestra.run.smithi177.stdout: "expos": 4194304, 2024-04-02T14:51:28.960 INFO:teuthology.orchestra.run.smithi177.stdout: "wrpos": 4195198, 2024-04-02T14:51:28.960 INFO:teuthology.orchestra.run.smithi177.stdout: "rdpos": 4194892, 2024-04-02T14:51:28.960 INFO:teuthology.orchestra.run.smithi177.stdout: "jlat": { 2024-04-02T14:51:28.960 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-02T14:51:28.960 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-02T14:51:28.960 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-02T14:51:28.960 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:28.960 INFO:teuthology.orchestra.run.smithi177.stdout: "replayed": 3 2024-04-02T14:51:28.960 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:28.960 INFO:teuthology.orchestra.run.smithi177.stdout: "mds_mem": { 2024-04-02T14:51:28.960 INFO:teuthology.orchestra.run.smithi177.stdout: "ino": 3, 2024-04-02T14:51:28.960 INFO:teuthology.orchestra.run.smithi177.stdout: "ino+": 3, 2024-04-02T14:51:28.960 INFO:teuthology.orchestra.run.smithi177.stdout: "ino-": 0, 2024-04-02T14:51:28.960 INFO:teuthology.orchestra.run.smithi177.stdout: "dir": 1, 2024-04-02T14:51:28.961 INFO:teuthology.orchestra.run.smithi177.stdout: "dir+": 1, 2024-04-02T14:51:28.961 INFO:teuthology.orchestra.run.smithi177.stdout: "dir-": 0, 2024-04-02T14:51:28.961 INFO:teuthology.orchestra.run.smithi177.stdout: "dn": 0, 2024-04-02T14:51:28.961 INFO:teuthology.orchestra.run.smithi177.stdout: "dn+": 0, 2024-04-02T14:51:28.961 INFO:teuthology.orchestra.run.smithi177.stdout: "dn-": 0, 2024-04-02T14:51:28.961 INFO:teuthology.orchestra.run.smithi177.stdout: "cap": 0, 2024-04-02T14:51:28.961 INFO:teuthology.orchestra.run.smithi177.stdout: "cap+": 0, 2024-04-02T14:51:28.961 INFO:teuthology.orchestra.run.smithi177.stdout: "cap-": 0, 2024-04-02T14:51:28.961 INFO:teuthology.orchestra.run.smithi177.stdout: "rss": 50724, 2024-04-02T14:51:28.961 INFO:teuthology.orchestra.run.smithi177.stdout: "heap": 215416 2024-04-02T14:51:28.961 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:28.961 INFO:teuthology.orchestra.run.smithi177.stdout: "mds_server": { 2024-04-02T14:51:28.961 INFO:teuthology.orchestra.run.smithi177.stdout: "dispatch_client_request": 0, 2024-04-02T14:51:28.961 INFO:teuthology.orchestra.run.smithi177.stdout: "dispatch_server_request": 0, 2024-04-02T14:51:28.962 INFO:teuthology.orchestra.run.smithi177.stdout: "handle_client_request": 0, 2024-04-02T14:51:28.962 INFO:teuthology.orchestra.run.smithi177.stdout: "handle_client_session": 0, 2024-04-02T14:51:28.962 INFO:teuthology.orchestra.run.smithi177.stdout: "handle_peer_request": 0, 2024-04-02T14:51:28.962 INFO:teuthology.orchestra.run.smithi177.stdout: "req_create_latency": { 2024-04-02T14:51:28.962 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-02T14:51:28.962 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-02T14:51:28.962 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-02T14:51:28.962 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:28.962 INFO:teuthology.orchestra.run.smithi177.stdout: "req_getattr_latency": { 2024-04-02T14:51:28.962 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-02T14:51:28.962 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-02T14:51:28.962 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-02T14:51:28.962 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:28.962 INFO:teuthology.orchestra.run.smithi177.stdout: "req_getfilelock_latency": { 2024-04-02T14:51:28.963 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-02T14:51:28.963 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-02T14:51:28.963 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-02T14:51:28.963 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:28.963 INFO:teuthology.orchestra.run.smithi177.stdout: "req_link_latency": { 2024-04-02T14:51:28.963 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-02T14:51:28.963 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-02T14:51:28.963 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-02T14:51:28.963 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:28.963 INFO:teuthology.orchestra.run.smithi177.stdout: "req_lookup_latency": { 2024-04-02T14:51:28.963 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-02T14:51:28.963 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-02T14:51:28.963 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-02T14:51:28.963 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:28.963 INFO:teuthology.orchestra.run.smithi177.stdout: "req_lookuphash_latency": { 2024-04-02T14:51:28.964 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-02T14:51:28.964 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-02T14:51:28.964 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-02T14:51:28.964 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:28.964 INFO:teuthology.orchestra.run.smithi177.stdout: "req_lookupino_latency": { 2024-04-02T14:51:28.964 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-02T14:51:28.964 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-02T14:51:28.964 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-02T14:51:28.964 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:28.964 INFO:teuthology.orchestra.run.smithi177.stdout: "req_lookupname_latency": { 2024-04-02T14:51:28.964 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-02T14:51:28.964 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-02T14:51:28.964 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-02T14:51:28.964 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:28.964 INFO:teuthology.orchestra.run.smithi177.stdout: "req_lookupparent_latency": { 2024-04-02T14:51:28.965 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-02T14:51:28.965 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-02T14:51:28.965 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-02T14:51:28.965 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:28.965 INFO:teuthology.orchestra.run.smithi177.stdout: "req_lookupsnap_latency": { 2024-04-02T14:51:28.965 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-02T14:51:28.965 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-02T14:51:28.965 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-02T14:51:28.965 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:28.965 INFO:teuthology.orchestra.run.smithi177.stdout: "req_lssnap_latency": { 2024-04-02T14:51:28.965 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-02T14:51:28.965 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-02T14:51:28.965 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-02T14:51:28.965 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:28.965 INFO:teuthology.orchestra.run.smithi177.stdout: "req_mkdir_latency": { 2024-04-02T14:51:28.966 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-02T14:51:28.966 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-02T14:51:28.966 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-02T14:51:28.966 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:28.966 INFO:teuthology.orchestra.run.smithi177.stdout: "req_mknod_latency": { 2024-04-02T14:51:28.966 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-02T14:51:28.967 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-02T14:51:28.967 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-02T14:51:28.967 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:28.967 INFO:teuthology.orchestra.run.smithi177.stdout: "req_mksnap_latency": { 2024-04-02T14:51:28.967 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-02T14:51:28.967 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-02T14:51:28.967 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-02T14:51:28.967 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:28.967 INFO:teuthology.orchestra.run.smithi177.stdout: "req_open_latency": { 2024-04-02T14:51:28.967 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-02T14:51:28.967 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-02T14:51:28.967 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-02T14:51:28.967 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:28.967 INFO:teuthology.orchestra.run.smithi177.stdout: "req_readdir_latency": { 2024-04-02T14:51:28.967 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-02T14:51:28.968 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-02T14:51:28.968 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-02T14:51:28.968 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:28.968 INFO:teuthology.orchestra.run.smithi177.stdout: "req_rename_latency": { 2024-04-02T14:51:28.968 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-02T14:51:28.968 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-02T14:51:28.968 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-02T14:51:28.968 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:28.968 INFO:teuthology.orchestra.run.smithi177.stdout: "req_renamesnap_latency": { 2024-04-02T14:51:28.968 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-02T14:51:28.968 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-02T14:51:28.969 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-02T14:51:28.969 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:28.969 INFO:teuthology.orchestra.run.smithi177.stdout: "req_snapdiff_latency": { 2024-04-02T14:51:28.969 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-02T14:51:28.969 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-02T14:51:28.969 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-02T14:51:28.969 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:28.969 INFO:teuthology.orchestra.run.smithi177.stdout: "req_rmdir_latency": { 2024-04-02T14:51:28.969 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-02T14:51:28.969 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-02T14:51:28.969 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-02T14:51:28.969 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:28.969 INFO:teuthology.orchestra.run.smithi177.stdout: "req_rmsnap_latency": { 2024-04-02T14:51:28.969 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-02T14:51:28.970 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-02T14:51:28.970 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-02T14:51:28.970 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:28.970 INFO:teuthology.orchestra.run.smithi177.stdout: "req_rmxattr_latency": { 2024-04-02T14:51:28.970 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-02T14:51:28.970 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-02T14:51:28.970 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-02T14:51:28.970 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:28.970 INFO:teuthology.orchestra.run.smithi177.stdout: "req_setattr_latency": { 2024-04-02T14:51:28.970 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-02T14:51:28.970 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-02T14:51:28.970 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-02T14:51:28.970 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:28.970 INFO:teuthology.orchestra.run.smithi177.stdout: "req_setdirlayout_latency": { 2024-04-02T14:51:28.971 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-02T14:51:28.971 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-02T14:51:28.971 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-02T14:51:28.971 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:28.971 INFO:teuthology.orchestra.run.smithi177.stdout: "req_setfilelock_latency": { 2024-04-02T14:51:28.971 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-02T14:51:28.971 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-02T14:51:28.971 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-02T14:51:28.971 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:28.971 INFO:teuthology.orchestra.run.smithi177.stdout: "req_setlayout_latency": { 2024-04-02T14:51:28.971 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-02T14:51:28.971 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-02T14:51:28.971 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-02T14:51:28.971 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:28.971 INFO:teuthology.orchestra.run.smithi177.stdout: "req_setxattr_latency": { 2024-04-02T14:51:28.972 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-02T14:51:28.972 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-02T14:51:28.972 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-02T14:51:28.972 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:28.972 INFO:teuthology.orchestra.run.smithi177.stdout: "req_symlink_latency": { 2024-04-02T14:51:28.972 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-02T14:51:28.972 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-02T14:51:28.972 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-02T14:51:28.972 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:28.972 INFO:teuthology.orchestra.run.smithi177.stdout: "req_unlink_latency": { 2024-04-02T14:51:28.972 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-02T14:51:28.972 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-02T14:51:28.972 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-02T14:51:28.972 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:28.972 INFO:teuthology.orchestra.run.smithi177.stdout: "cap_revoke_eviction": 0, 2024-04-02T14:51:28.973 INFO:teuthology.orchestra.run.smithi177.stdout: "cap_acquisition_throttle": 0, 2024-04-02T14:51:28.973 INFO:teuthology.orchestra.run.smithi177.stdout: "req_getvxattr_latency": { 2024-04-02T14:51:28.973 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-02T14:51:28.973 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-02T14:51:28.973 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-02T14:51:28.973 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-04-02T14:51:28.973 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:28.973 INFO:teuthology.orchestra.run.smithi177.stdout: "mds_sessions": { 2024-04-02T14:51:28.973 INFO:teuthology.orchestra.run.smithi177.stdout: "session_count": 1, 2024-04-02T14:51:28.973 INFO:teuthology.orchestra.run.smithi177.stdout: "session_add": 1, 2024-04-02T14:51:28.973 INFO:teuthology.orchestra.run.smithi177.stdout: "session_remove": 0, 2024-04-02T14:51:28.973 INFO:teuthology.orchestra.run.smithi177.stdout: "sessions_open": 1, 2024-04-02T14:51:28.973 INFO:teuthology.orchestra.run.smithi177.stdout: "sessions_stale": 0, 2024-04-02T14:51:28.973 INFO:teuthology.orchestra.run.smithi177.stdout: "total_load": 0, 2024-04-02T14:51:28.973 INFO:teuthology.orchestra.run.smithi177.stdout: "average_load": 0, 2024-04-02T14:51:28.973 INFO:teuthology.orchestra.run.smithi177.stdout: "avg_session_uptime": 2697, 2024-04-02T14:51:28.974 INFO:teuthology.orchestra.run.smithi177.stdout: "mdthresh_evicted": 0 2024-04-02T14:51:28.974 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:28.974 INFO:teuthology.orchestra.run.smithi177.stdout: "mempool": { 2024-04-02T14:51:28.974 INFO:teuthology.orchestra.run.smithi177.stdout: "bloom_filter_bytes": 0, 2024-04-02T14:51:28.974 INFO:teuthology.orchestra.run.smithi177.stdout: "bloom_filter_items": 0, 2024-04-02T14:51:28.974 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_alloc_bytes": 0, 2024-04-02T14:51:28.974 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_alloc_items": 0, 2024-04-02T14:51:28.974 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_data_bytes": 0, 2024-04-02T14:51:28.974 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_data_items": 0, 2024-04-02T14:51:28.974 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-02T14:51:28.974 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_onode_items": 0, 2024-04-02T14:51:28.974 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-02T14:51:28.974 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_meta_items": 0, 2024-04-02T14:51:28.974 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_other_bytes": 0, 2024-04-02T14:51:28.975 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_other_items": 0, 2024-04-02T14:51:28.975 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-02T14:51:28.975 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_buffer_items": 0, 2024-04-02T14:51:28.975 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_extent_bytes": 0, 2024-04-02T14:51:28.975 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_extent_items": 0, 2024-04-02T14:51:28.975 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_blob_bytes": 0, 2024-04-02T14:51:28.975 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_blob_items": 0, 2024-04-02T14:51:28.975 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-02T14:51:28.975 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_shared_blob_items": 0, 2024-04-02T14:51:28.975 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-02T14:51:28.975 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_inline_bl_items": 0, 2024-04-02T14:51:28.975 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_fsck_bytes": 0, 2024-04-02T14:51:28.975 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_fsck_items": 0, 2024-04-02T14:51:28.975 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_txc_bytes": 0, 2024-04-02T14:51:28.975 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_txc_items": 0, 2024-04-02T14:51:28.975 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-02T14:51:28.976 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_writing_deferred_items": 0, 2024-04-02T14:51:28.976 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_writing_bytes": 0, 2024-04-02T14:51:28.976 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_writing_items": 0, 2024-04-02T14:51:28.976 INFO:teuthology.orchestra.run.smithi177.stdout: "bluefs_bytes": 0, 2024-04-02T14:51:28.976 INFO:teuthology.orchestra.run.smithi177.stdout: "bluefs_items": 0, 2024-04-02T14:51:28.976 INFO:teuthology.orchestra.run.smithi177.stdout: "bluefs_file_reader_bytes": 0, 2024-04-02T14:51:28.976 INFO:teuthology.orchestra.run.smithi177.stdout: "bluefs_file_reader_items": 0, 2024-04-02T14:51:28.976 INFO:teuthology.orchestra.run.smithi177.stdout: "bluefs_file_writer_bytes": 0, 2024-04-02T14:51:28.976 INFO:teuthology.orchestra.run.smithi177.stdout: "bluefs_file_writer_items": 0, 2024-04-02T14:51:28.976 INFO:teuthology.orchestra.run.smithi177.stdout: "buffer_anon_bytes": 198219, 2024-04-02T14:51:28.976 INFO:teuthology.orchestra.run.smithi177.stdout: "buffer_anon_items": 56, 2024-04-02T14:51:28.976 INFO:teuthology.orchestra.run.smithi177.stdout: "buffer_meta_bytes": 0, 2024-04-02T14:51:28.976 INFO:teuthology.orchestra.run.smithi177.stdout: "buffer_meta_items": 0, 2024-04-02T14:51:28.976 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_bytes": 0, 2024-04-02T14:51:28.976 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_items": 0, 2024-04-02T14:51:28.977 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_mapbl_bytes": 0, 2024-04-02T14:51:28.977 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_mapbl_items": 0, 2024-04-02T14:51:28.977 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_pglog_bytes": 0, 2024-04-02T14:51:28.977 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_pglog_items": 0, 2024-04-02T14:51:28.977 INFO:teuthology.orchestra.run.smithi177.stdout: "osdmap_bytes": 8136, 2024-04-02T14:51:28.977 INFO:teuthology.orchestra.run.smithi177.stdout: "osdmap_items": 170, 2024-04-02T14:51:28.977 INFO:teuthology.orchestra.run.smithi177.stdout: "osdmap_mapping_bytes": 0, 2024-04-02T14:51:28.977 INFO:teuthology.orchestra.run.smithi177.stdout: "osdmap_mapping_items": 0, 2024-04-02T14:51:28.977 INFO:teuthology.orchestra.run.smithi177.stdout: "pgmap_bytes": 0, 2024-04-02T14:51:28.977 INFO:teuthology.orchestra.run.smithi177.stdout: "pgmap_items": 0, 2024-04-02T14:51:28.977 INFO:teuthology.orchestra.run.smithi177.stdout: "mds_co_bytes": 8464, 2024-04-02T14:51:28.977 INFO:teuthology.orchestra.run.smithi177.stdout: "mds_co_items": 13, 2024-04-02T14:51:28.977 INFO:teuthology.orchestra.run.smithi177.stdout: "unittest_1_bytes": 0, 2024-04-02T14:51:28.977 INFO:teuthology.orchestra.run.smithi177.stdout: "unittest_1_items": 0, 2024-04-02T14:51:28.977 INFO:teuthology.orchestra.run.smithi177.stdout: "unittest_2_bytes": 0, 2024-04-02T14:51:28.978 INFO:teuthology.orchestra.run.smithi177.stdout: "unittest_2_items": 0 2024-04-02T14:51:28.978 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:28.978 INFO:teuthology.orchestra.run.smithi177.stdout: "objecter": { 2024-04-02T14:51:28.978 INFO:teuthology.orchestra.run.smithi177.stdout: "op_active": 0, 2024-04-02T14:51:28.978 INFO:teuthology.orchestra.run.smithi177.stdout: "op_laggy": 0, 2024-04-02T14:51:28.978 INFO:teuthology.orchestra.run.smithi177.stdout: "op_send": 3237, 2024-04-02T14:51:28.978 INFO:teuthology.orchestra.run.smithi177.stdout: "op_send_bytes": 21, 2024-04-02T14:51:28.978 INFO:teuthology.orchestra.run.smithi177.stdout: "op_resend": 3, 2024-04-02T14:51:28.978 INFO:teuthology.orchestra.run.smithi177.stdout: "op_reply": 3237, 2024-04-02T14:51:28.978 INFO:teuthology.orchestra.run.smithi177.stdout: "op_latency": { 2024-04-02T14:51:28.978 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 3237, 2024-04-02T14:51:28.978 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 12.560666915, 2024-04-02T14:51:28.978 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.003880341 2024-04-02T14:51:28.978 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:28.978 INFO:teuthology.orchestra.run.smithi177.stdout: "op_inflight": 0, 2024-04-02T14:51:28.979 INFO:teuthology.orchestra.run.smithi177.stdout: "oplen_avg": { 2024-04-02T14:51:28.979 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 3237, 2024-04-02T14:51:28.979 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 3238 2024-04-02T14:51:28.979 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:28.979 INFO:teuthology.orchestra.run.smithi177.stdout: "op": 3237, 2024-04-02T14:51:28.979 INFO:teuthology.orchestra.run.smithi177.stdout: "op_r": 3237, 2024-04-02T14:51:28.979 INFO:teuthology.orchestra.run.smithi177.stdout: "op_w": 0, 2024-04-02T14:51:28.979 INFO:teuthology.orchestra.run.smithi177.stdout: "op_rmw": 0, 2024-04-02T14:51:28.979 INFO:teuthology.orchestra.run.smithi177.stdout: "op_pg": 0, 2024-04-02T14:51:28.979 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_stat": 2102, 2024-04-02T14:51:28.979 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_create": 0, 2024-04-02T14:51:28.979 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_read": 1133, 2024-04-02T14:51:28.979 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_write": 0, 2024-04-02T14:51:28.979 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_writefull": 0, 2024-04-02T14:51:28.979 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_writesame": 0, 2024-04-02T14:51:28.980 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_append": 0, 2024-04-02T14:51:28.980 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_zero": 0, 2024-04-02T14:51:28.980 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_truncate": 0, 2024-04-02T14:51:28.980 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_delete": 0, 2024-04-02T14:51:28.980 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_mapext": 0, 2024-04-02T14:51:28.980 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_sparse_read": 0, 2024-04-02T14:51:28.980 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_clonerange": 0, 2024-04-02T14:51:28.980 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_getxattr": 1, 2024-04-02T14:51:28.980 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_setxattr": 0, 2024-04-02T14:51:28.980 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_cmpxattr": 0, 2024-04-02T14:51:28.980 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_rmxattr": 0, 2024-04-02T14:51:28.980 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_resetxattrs": 0, 2024-04-02T14:51:28.980 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_call": 0, 2024-04-02T14:51:28.980 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_watch": 0, 2024-04-02T14:51:28.981 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_notify": 0, 2024-04-02T14:51:28.981 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_src_cmpxattr": 0, 2024-04-02T14:51:28.981 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_pgls": 0, 2024-04-02T14:51:28.981 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_pgls_filter": 0, 2024-04-02T14:51:28.981 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_other": 0, 2024-04-02T14:51:28.981 INFO:teuthology.orchestra.run.smithi177.stdout: "linger_active": 0, 2024-04-02T14:51:28.981 INFO:teuthology.orchestra.run.smithi177.stdout: "linger_send": 0, 2024-04-02T14:51:28.981 INFO:teuthology.orchestra.run.smithi177.stdout: "linger_resend": 0, 2024-04-02T14:51:28.981 INFO:teuthology.orchestra.run.smithi177.stdout: "linger_ping": 0, 2024-04-02T14:51:28.981 INFO:teuthology.orchestra.run.smithi177.stdout: "poolop_active": 0, 2024-04-02T14:51:28.981 INFO:teuthology.orchestra.run.smithi177.stdout: "poolop_send": 0, 2024-04-02T14:51:28.981 INFO:teuthology.orchestra.run.smithi177.stdout: "poolop_resend": 0, 2024-04-02T14:51:28.981 INFO:teuthology.orchestra.run.smithi177.stdout: "poolstat_active": 0, 2024-04-02T14:51:28.981 INFO:teuthology.orchestra.run.smithi177.stdout: "poolstat_send": 0, 2024-04-02T14:51:28.981 INFO:teuthology.orchestra.run.smithi177.stdout: "poolstat_resend": 0, 2024-04-02T14:51:28.982 INFO:teuthology.orchestra.run.smithi177.stdout: "statfs_active": 0, 2024-04-02T14:51:28.982 INFO:teuthology.orchestra.run.smithi177.stdout: "statfs_send": 0, 2024-04-02T14:51:28.982 INFO:teuthology.orchestra.run.smithi177.stdout: "statfs_resend": 0, 2024-04-02T14:51:28.982 INFO:teuthology.orchestra.run.smithi177.stdout: "command_active": 0, 2024-04-02T14:51:28.982 INFO:teuthology.orchestra.run.smithi177.stdout: "command_send": 0, 2024-04-02T14:51:28.982 INFO:teuthology.orchestra.run.smithi177.stdout: "command_resend": 0, 2024-04-02T14:51:28.982 INFO:teuthology.orchestra.run.smithi177.stdout: "map_epoch": 111, 2024-04-02T14:51:28.982 INFO:teuthology.orchestra.run.smithi177.stdout: "map_full": 0, 2024-04-02T14:51:28.982 INFO:teuthology.orchestra.run.smithi177.stdout: "map_inc": 23, 2024-04-02T14:51:28.982 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_sessions": 6, 2024-04-02T14:51:28.982 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_session_open": 8, 2024-04-02T14:51:28.982 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_session_close": 2, 2024-04-02T14:51:28.982 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_laggy": 0, 2024-04-02T14:51:28.982 INFO:teuthology.orchestra.run.smithi177.stdout: "omap_wr": 0, 2024-04-02T14:51:28.982 INFO:teuthology.orchestra.run.smithi177.stdout: "omap_rd": 2, 2024-04-02T14:51:28.983 INFO:teuthology.orchestra.run.smithi177.stdout: "omap_del": 0 2024-04-02T14:51:28.983 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:28.983 INFO:teuthology.orchestra.run.smithi177.stdout: "oft": { 2024-04-02T14:51:28.983 INFO:teuthology.orchestra.run.smithi177.stdout: "omap_total_objs": 0, 2024-04-02T14:51:28.983 INFO:teuthology.orchestra.run.smithi177.stdout: "omap_total_kv_pairs": 0, 2024-04-02T14:51:28.983 INFO:teuthology.orchestra.run.smithi177.stdout: "omap_total_updates": 0, 2024-04-02T14:51:28.983 INFO:teuthology.orchestra.run.smithi177.stdout: "omap_total_removes": 0 2024-04-02T14:51:28.983 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:28.983 INFO:teuthology.orchestra.run.smithi177.stdout: "osd-slow-ops": { 2024-04-02T14:51:28.983 INFO:teuthology.orchestra.run.smithi177.stdout: "slow_ops_count": 0 2024-04-02T14:51:28.983 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:28.983 INFO:teuthology.orchestra.run.smithi177.stdout: "purge_queue": { 2024-04-02T14:51:28.983 INFO:teuthology.orchestra.run.smithi177.stdout: "pq_executing_ops": 0, 2024-04-02T14:51:28.983 INFO:teuthology.orchestra.run.smithi177.stdout: "pq_executing_ops_high_water": 0, 2024-04-02T14:51:28.983 INFO:teuthology.orchestra.run.smithi177.stdout: "pq_executing": 0, 2024-04-02T14:51:28.984 INFO:teuthology.orchestra.run.smithi177.stdout: "pq_executing_high_water": 0, 2024-04-02T14:51:28.984 INFO:teuthology.orchestra.run.smithi177.stdout: "pq_executed_ops": 0, 2024-04-02T14:51:28.984 INFO:teuthology.orchestra.run.smithi177.stdout: "pq_executed": 0, 2024-04-02T14:51:28.984 INFO:teuthology.orchestra.run.smithi177.stdout: "pq_item_in_journal": 0 2024-04-02T14:51:28.984 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:28.984 INFO:teuthology.orchestra.run.smithi177.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-02T14:51:28.984 INFO:teuthology.orchestra.run.smithi177.stdout: "val": 0, 2024-04-02T14:51:28.984 INFO:teuthology.orchestra.run.smithi177.stdout: "max": 104857600, 2024-04-02T14:51:28.984 INFO:teuthology.orchestra.run.smithi177.stdout: "get_started": 0, 2024-04-02T14:51:28.984 INFO:teuthology.orchestra.run.smithi177.stdout: "get": 3597, 2024-04-02T14:51:28.984 INFO:teuthology.orchestra.run.smithi177.stdout: "get_sum": 2233354, 2024-04-02T14:51:28.984 INFO:teuthology.orchestra.run.smithi177.stdout: "get_or_fail_fail": 0, 2024-04-02T14:51:28.984 INFO:teuthology.orchestra.run.smithi177.stdout: "get_or_fail_success": 3597, 2024-04-02T14:51:28.984 INFO:teuthology.orchestra.run.smithi177.stdout: "take": 0, 2024-04-02T14:51:28.984 INFO:teuthology.orchestra.run.smithi177.stdout: "take_sum": 0, 2024-04-02T14:51:28.984 INFO:teuthology.orchestra.run.smithi177.stdout: "put": 3597, 2024-04-02T14:51:28.985 INFO:teuthology.orchestra.run.smithi177.stdout: "put_sum": 2233354, 2024-04-02T14:51:28.985 INFO:teuthology.orchestra.run.smithi177.stdout: "wait": { 2024-04-02T14:51:28.985 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-02T14:51:28.985 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-02T14:51:28.985 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-02T14:51:28.985 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-04-02T14:51:28.985 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:28.985 INFO:teuthology.orchestra.run.smithi177.stdout: "throttle-objecter_bytes": { 2024-04-02T14:51:28.985 INFO:teuthology.orchestra.run.smithi177.stdout: "val": 0, 2024-04-02T14:51:28.985 INFO:teuthology.orchestra.run.smithi177.stdout: "max": 104857600, 2024-04-02T14:51:28.985 INFO:teuthology.orchestra.run.smithi177.stdout: "get_started": 0, 2024-04-02T14:51:28.985 INFO:teuthology.orchestra.run.smithi177.stdout: "get": 0, 2024-04-02T14:51:28.985 INFO:teuthology.orchestra.run.smithi177.stdout: "get_sum": 0, 2024-04-02T14:51:28.985 INFO:teuthology.orchestra.run.smithi177.stdout: "get_or_fail_fail": 0, 2024-04-02T14:51:28.986 INFO:teuthology.orchestra.run.smithi177.stdout: "get_or_fail_success": 0, 2024-04-02T14:51:28.986 INFO:teuthology.orchestra.run.smithi177.stdout: "take": 3237, 2024-04-02T14:51:28.986 INFO:teuthology.orchestra.run.smithi177.stdout: "take_sum": 899, 2024-04-02T14:51:28.986 INFO:teuthology.orchestra.run.smithi177.stdout: "put": 3237, 2024-04-02T14:51:28.986 INFO:teuthology.orchestra.run.smithi177.stdout: "put_sum": 899, 2024-04-02T14:51:28.986 INFO:teuthology.orchestra.run.smithi177.stdout: "wait": { 2024-04-02T14:51:28.986 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-02T14:51:28.986 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-02T14:51:28.986 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-02T14:51:28.986 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-04-02T14:51:28.986 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:28.986 INFO:teuthology.orchestra.run.smithi177.stdout: "throttle-objecter_ops": { 2024-04-02T14:51:28.986 INFO:teuthology.orchestra.run.smithi177.stdout: "val": 0, 2024-04-02T14:51:28.986 INFO:teuthology.orchestra.run.smithi177.stdout: "max": 1024, 2024-04-02T14:51:28.986 INFO:teuthology.orchestra.run.smithi177.stdout: "get_started": 0, 2024-04-02T14:51:28.987 INFO:teuthology.orchestra.run.smithi177.stdout: "get": 0, 2024-04-02T14:51:28.987 INFO:teuthology.orchestra.run.smithi177.stdout: "get_sum": 0, 2024-04-02T14:51:28.987 INFO:teuthology.orchestra.run.smithi177.stdout: "get_or_fail_fail": 0, 2024-04-02T14:51:28.987 INFO:teuthology.orchestra.run.smithi177.stdout: "get_or_fail_success": 0, 2024-04-02T14:51:28.987 INFO:teuthology.orchestra.run.smithi177.stdout: "take": 3237, 2024-04-02T14:51:28.987 INFO:teuthology.orchestra.run.smithi177.stdout: "take_sum": 3237, 2024-04-02T14:51:28.987 INFO:teuthology.orchestra.run.smithi177.stdout: "put": 3237, 2024-04-02T14:51:28.987 INFO:teuthology.orchestra.run.smithi177.stdout: "put_sum": 3237, 2024-04-02T14:51:28.987 INFO:teuthology.orchestra.run.smithi177.stdout: "wait": { 2024-04-02T14:51:28.987 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-02T14:51:28.987 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-02T14:51:28.987 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-02T14:51:28.987 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-04-02T14:51:28.987 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:28.987 INFO:teuthology.orchestra.run.smithi177.stdout: "throttle-write_buf_throttle": { 2024-04-02T14:51:28.987 INFO:teuthology.orchestra.run.smithi177.stdout: "val": 0, 2024-04-02T14:51:28.988 INFO:teuthology.orchestra.run.smithi177.stdout: "max": 3758096384, 2024-04-02T14:51:28.988 INFO:teuthology.orchestra.run.smithi177.stdout: "get_started": 0, 2024-04-02T14:51:28.988 INFO:teuthology.orchestra.run.smithi177.stdout: "get": 0, 2024-04-02T14:51:28.988 INFO:teuthology.orchestra.run.smithi177.stdout: "get_sum": 0, 2024-04-02T14:51:28.988 INFO:teuthology.orchestra.run.smithi177.stdout: "get_or_fail_fail": 0, 2024-04-02T14:51:28.988 INFO:teuthology.orchestra.run.smithi177.stdout: "get_or_fail_success": 0, 2024-04-02T14:51:28.988 INFO:teuthology.orchestra.run.smithi177.stdout: "take": 0, 2024-04-02T14:51:28.988 INFO:teuthology.orchestra.run.smithi177.stdout: "take_sum": 0, 2024-04-02T14:51:28.988 INFO:teuthology.orchestra.run.smithi177.stdout: "put": 0, 2024-04-02T14:51:28.988 INFO:teuthology.orchestra.run.smithi177.stdout: "put_sum": 0, 2024-04-02T14:51:28.988 INFO:teuthology.orchestra.run.smithi177.stdout: "wait": { 2024-04-02T14:51:28.988 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-02T14:51:28.988 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-02T14:51:28.988 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-02T14:51:28.988 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-04-02T14:51:28.989 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-02T14:51:28.989 INFO:teuthology.orchestra.run.smithi177.stdout: "throttle-write_buf_throttle-0x5622da3986e0": { 2024-04-02T14:51:28.989 INFO:teuthology.orchestra.run.smithi177.stdout: "val": 0, 2024-04-02T14:51:28.989 INFO:teuthology.orchestra.run.smithi177.stdout: "max": 3758096384, 2024-04-02T14:51:28.989 INFO:teuthology.orchestra.run.smithi177.stdout: "get_started": 0, 2024-04-02T14:51:28.989 INFO:teuthology.orchestra.run.smithi177.stdout: "get": 0, 2024-04-02T14:51:28.989 INFO:teuthology.orchestra.run.smithi177.stdout: "get_sum": 0, 2024-04-02T14:51:28.989 INFO:teuthology.orchestra.run.smithi177.stdout: "get_or_fail_fail": 0, 2024-04-02T14:51:28.989 INFO:teuthology.orchestra.run.smithi177.stdout: "get_or_fail_success": 0, 2024-04-02T14:51:28.989 INFO:teuthology.orchestra.run.smithi177.stdout: "take": 0, 2024-04-02T14:51:28.989 INFO:teuthology.orchestra.run.smithi177.stdout: "take_sum": 0, 2024-04-02T14:51:28.989 INFO:teuthology.orchestra.run.smithi177.stdout: "put": 0, 2024-04-02T14:51:28.989 INFO:teuthology.orchestra.run.smithi177.stdout: "put_sum": 0, 2024-04-02T14:51:28.989 INFO:teuthology.orchestra.run.smithi177.stdout: "wait": { 2024-04-02T14:51:28.989 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-02T14:51:28.989 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-02T14:51:28.990 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-02T14:51:28.990 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-04-02T14:51:28.990 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-04-02T14:51:28.990 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-04-02T14:51:29.775 INFO:tasks.check_counter:Daemon mds.l mds.exported=0 2024-04-02T14:51:29.776 INFO:tasks.check_counter:Daemon mds.l mds.imported=0 2024-04-02T14:51:29.776 INFO:tasks.check_counter:Daemon mds.l mds_cache.dir_update=0 2024-04-02T14:51:29.776 INFO:tasks.check_counter:Daemon mds.l mds_cache.dir_update_receipt=0 2024-04-02T14:51:29.776 INFO:tasks.check_counter:Daemon mds.l mds.root_rsnaps=0 2024-04-02T14:51:29.776 INFO:tasks.check_counter:Daemon mds.l mds_server.req_mksnap_latency.avgcount=0 2024-04-02T14:51:29.776 INFO:tasks.check_counter:Daemon mds.l mds_server.req_rmsnap_latency.avgcount=0 2024-04-02T14:51:29.776 ERROR:teuthology.run_tasks:Manager failed: check-counter Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/__init__.py", line 132, in __exit__ self.end() File "/home/teuthworker/src/git.ceph.com_ceph-c_191d963ac9de7322da8aff20b567b3fa458e2ef6/qa/tasks/check_counter.py", line 125, in end raise RuntimeError("The following counters failed to be set " RuntimeError: The following counters failed to be set on mds daemons: {'mds.exported', 'mds.imported'} 2024-04-02T14:51:29.777 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-04-02T14:51:29.788 INFO:tasks.kclient:Unmounting kernel clients... 2024-04-02T14:51:29.788 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-04-02T14:51:29.788 DEBUG:teuthology.orchestra.run.smithi090:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T14:51:29.850 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-04-02T14:51:29.850 DEBUG:teuthology.orchestra.run.smithi090:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T14:51:29.904 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T14:51:30.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:30.137+0000 7f576f800640 1 -- 172.21.15.90:0/1131180338 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5768108cd0 msgr2=0x7f576810cd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:51:30.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:30.137+0000 7f576f800640 1 --2- 172.21.15.90:0/1131180338 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5768108cd0 0x7f576810cd10 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f5754009900 tx=0x7f575402bc30 comp rx=0 tx=0).stop 2024-04-02T14:51:30.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:30.137+0000 7f576f800640 1 -- 172.21.15.90:0/1131180338 shutdown_connections 2024-04-02T14:51:30.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:30.137+0000 7f576f800640 1 --2- 172.21.15.90:0/1131180338 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f576810d3f0 0x7f5768114fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:51:30.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:30.137+0000 7f576f800640 1 --2- 172.21.15.90:0/1131180338 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5768108cd0 0x7f576810cd10 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:51:30.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:30.137+0000 7f576f800640 1 --2- 172.21.15.90:0/1131180338 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5768108320 0x7f5768108700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:51:30.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:30.137+0000 7f576f800640 1 -- 172.21.15.90:0/1131180338 >> 172.21.15.90:0/1131180338 conn(0x7f5768076050 msgr2=0x7f5768078470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:51:30.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:30.137+0000 7f576f800640 1 -- 172.21.15.90:0/1131180338 shutdown_connections 2024-04-02T14:51:30.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:30.137+0000 7f576f800640 1 -- 172.21.15.90:0/1131180338 wait complete. 2024-04-02T14:51:30.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:30.138+0000 7f576f800640 1 Processor -- start 2024-04-02T14:51:30.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:30.138+0000 7f576f800640 1 -- start start 2024-04-02T14:51:30.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:30.139+0000 7f576f800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5768108320 0x7f57681a46c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:51:30.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:30.139+0000 7f576f800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f576810d3f0 0x7f57681a4c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:51:30.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:30.139+0000 7f576f800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f576819e830 0x7f576819ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:51:30.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:30.139+0000 7f576f800640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f57681184d0 con 0x7f576810d3f0 2024-04-02T14:51:30.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:30.139+0000 7f576f800640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f5768118350 con 0x7f576819e830 2024-04-02T14:51:30.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:30.139+0000 7f576f800640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f5768118650 con 0x7f5768108320 2024-04-02T14:51:30.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:30.139+0000 7f5767e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f576810d3f0 0x7f57681a4c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:51:30.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:30.139+0000 7f5767e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f576810d3f0 0x7f57681a4c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34990/0 (socket says 172.21.15.90:34990) 2024-04-02T14:51:30.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:30.139+0000 7f5767e00640 1 -- 172.21.15.90:0/2623247184 learned_addr learned my addr 172.21.15.90:0/2623247184 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:51:30.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:30.139+0000 7f576ce00640 1 --2- 172.21.15.90:0/2623247184 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5768108320 0x7f57681a46c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:51:30.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:30.139+0000 7f576d800640 1 --2- 172.21.15.90:0/2623247184 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f576819e830 0x7f576819ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:51:30.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:30.139+0000 7f5767e00640 1 -- 172.21.15.90:0/2623247184 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5768108320 msgr2=0x7f57681a46c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:51:30.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:30.139+0000 7f5767e00640 1 --2- 172.21.15.90:0/2623247184 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5768108320 0x7f57681a46c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:51:30.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:30.139+0000 7f5767e00640 1 -- 172.21.15.90:0/2623247184 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f576819e830 msgr2=0x7f576819ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:51:30.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:30.139+0000 7f5767e00640 1 --2- 172.21.15.90:0/2623247184 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f576819e830 0x7f576819ece0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:51:30.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:30.139+0000 7f5767e00640 1 -- 172.21.15.90:0/2623247184 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f576819f3c0 con 0x7f576810d3f0 2024-04-02T14:51:30.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:30.140+0000 7f5767e00640 1 --2- 172.21.15.90:0/2623247184 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f576810d3f0 0x7f57681a4c00 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f575402c240 tx=0x7f5754004360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:51:30.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:30.140+0000 7f5765600640 1 -- 172.21.15.90:0/2623247184 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5754043020 con 0x7f576810d3f0 2024-04-02T14:51:30.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:30.140+0000 7f5765600640 1 -- 172.21.15.90:0/2623247184 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f57540054e0 con 0x7f576810d3f0 2024-04-02T14:51:30.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:30.140+0000 7f576f800640 1 -- 172.21.15.90:0/2623247184 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f576819f650 con 0x7f576810d3f0 2024-04-02T14:51:30.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:30.140+0000 7f5765600640 1 -- 172.21.15.90:0/2623247184 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5754003b80 con 0x7f576810d3f0 2024-04-02T14:51:30.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:30.140+0000 7f576f800640 1 -- 172.21.15.90:0/2623247184 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57681ab830 con 0x7f576810d3f0 2024-04-02T14:51:30.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:30.142+0000 7f576f800640 1 -- 172.21.15.90:0/2623247184 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f57681097f0 con 0x7f576810d3f0 2024-04-02T14:51:30.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:30.143+0000 7f5765600640 1 -- 172.21.15.90:0/2623247184 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f5754046020 con 0x7f576810d3f0 2024-04-02T14:51:30.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:30.145+0000 7f5765600640 1 --2- 172.21.15.90:0/2623247184 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f57500a3d20 0x7f57500a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:51:30.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:30.146+0000 7f5765600640 1 -- 172.21.15.90:0/2623247184 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 8550+0+0 (secure 0 0 0) 0x7f57540f2b90 con 0x7f576810d3f0 2024-04-02T14:51:30.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:30.146+0000 7f576ce00640 1 --2- 172.21.15.90:0/2623247184 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f57500a3d20 0x7f57500a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:51:30.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:30.146+0000 7f576ce00640 1 --2- 172.21.15.90:0/2623247184 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f57500a3d20 0x7f57500a61e0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f575c009bd0 tx=0x7f575c005f90 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:51:30.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:30.147+0000 7f5765600640 1 -- 172.21.15.90:0/2623247184 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7f575404d020 con 0x7f576810d3f0 2024-04-02T14:51:30.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:30.351+0000 7f576f800640 1 -- 172.21.15.90:0/2623247184 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f5768110620 con 0x7f576810d3f0 2024-04-02T14:51:30.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:30.356+0000 7f5765600640 1 -- 172.21.15.90:0/2623247184 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v111) ==== 74+0+23118 (secure 0 0 0) 0x7f57540bf2c0 con 0x7f576810d3f0 2024-04-02T14:51:30.348 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-02T14:51:30.349 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":111,"fsid":"6449448a-f0fc-11ee-b647-cb9ed24678a4","created":"2024-04-02T14:23:16.530201+0000","modified":"2024-04-02T14:51:20.219915+0000","last_up_change":"2024-04-02T14:31:23.819732+0000","last_in_change":"2024-04-02T14:31:02.824284+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T14:27:16.758622+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-02T14:32:20.016331+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-02T14:32:21.294367+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"110","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":27,"snap_epoch":110,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a1a6a856-ab44-44bd-bc5e-45c718a70cd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":3635087609},{"type":"v1","addr":"172.21.15.90:6803","nonce":3635087609}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":3635087609},{"type":"v1","addr":"172.21.15.90:6805","nonce":3635087609}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":3635087609},{"type":"v1","addr":"172.21.15.90:6809","nonce":3635087609}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":3635087609},{"type":"v1","addr":"172.21.15.90:6807","nonce":3635087609}]},"public_addr":"172.21.15.90:6803/3635087609","cluster_addr":"172.21.15.90:6805/3635087609","heartbeat_back_addr":"172.21.15.90:6809/3635087609","heartbeat_front_addr":"172.21.15.90:6807/3635087609","state":["exists","up"]},{"osd":1,"uuid":"124139cd-5793-463e-9607-9d9c10164183","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":1994536059},{"type":"v1","addr":"172.21.15.92:6801","nonce":1994536059}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":1994536059},{"type":"v1","addr":"172.21.15.92:6803","nonce":1994536059}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":1994536059},{"type":"v1","addr":"172.21.15.92:6807","nonce":1994536059}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":1994536059},{"type":"v1","addr":"172.21.15.92:6805","nonce":1994536059}]},"public_addr":"172.21.15.92:6801/1994536059","cluster_addr":"172.21.15.92:6803/1994536059","heartbeat_back_addr":"172.21.15.92:6807/1994536059","heartbeat_front_addr":"172.21.15.92:6805/1994536059","state":["exists","up"]},{"osd":2,"uuid":"a61fbef2-9dca-4875-a814-020fff0db0e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":1358472800},{"type":"v1","addr":"172.21.15.177:6801","nonce":1358472800}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":1358472800},{"type":"v1","addr":"172.21.15.177:6803","nonce":1358472800}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":1358472800},{"type":"v1","addr":"172.21.15.177:6807","nonce":1358472800}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":1358472800},{"type":"v1","addr":"172.21.15.177:6805","nonce":1358472800}]},"public_addr":"172.21.15.177:6801/1358472800","cluster_addr":"172.21.15.177:6803/1358472800","heartbeat_back_addr":"172.21.15.177:6807/1358472800","heartbeat_front_addr":"172.21.15.177:6805/1358472800","state":["exists","up"]},{"osd":3,"uuid":"b175a2f4-7859-4eb0-bc1f-fba8c2b92a2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":3114811216},{"type":"v1","addr":"172.21.15.90:6811","nonce":3114811216}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":3114811216},{"type":"v1","addr":"172.21.15.90:6813","nonce":3114811216}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":3114811216},{"type":"v1","addr":"172.21.15.90:6817","nonce":3114811216}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":3114811216},{"type":"v1","addr":"172.21.15.90:6815","nonce":3114811216}]},"public_addr":"172.21.15.90:6811/3114811216","cluster_addr":"172.21.15.90:6813/3114811216","heartbeat_back_addr":"172.21.15.90:6817/3114811216","heartbeat_front_addr":"172.21.15.90:6815/3114811216","state":["exists","up"]},{"osd":4,"uuid":"38cc68ea-c109-4d8d-b77d-1c6f9b56c875","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":2987208879},{"type":"v1","addr":"172.21.15.92:6809","nonce":2987208879}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6810","nonce":2987208879},{"type":"v1","addr":"172.21.15.92:6811","nonce":2987208879}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6814","nonce":2987208879},{"type":"v1","addr":"172.21.15.92:6815","nonce":2987208879}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":2987208879},{"type":"v1","addr":"172.21.15.92:6813","nonce":2987208879}]},"public_addr":"172.21.15.92:6809/2987208879","cluster_addr":"172.21.15.92:6811/2987208879","heartbeat_back_addr":"172.21.15.92:6815/2987208879","heartbeat_front_addr":"172.21.15.92:6813/2987208879","state":["exists","up"]},{"osd":5,"uuid":"2347232e-ac91-466f-8ac9-496f397679ba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":3486820237},{"type":"v1","addr":"172.21.15.177:6809","nonce":3486820237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":3486820237},{"type":"v1","addr":"172.21.15.177:6811","nonce":3486820237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":3486820237},{"type":"v1","addr":"172.21.15.177:6815","nonce":3486820237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":3486820237},{"type":"v1","addr":"172.21.15.177:6813","nonce":3486820237}]},"public_addr":"172.21.15.177:6809/3486820237","cluster_addr":"172.21.15.177:6811/3486820237","heartbeat_back_addr":"172.21.15.177:6815/3486820237","heartbeat_front_addr":"172.21.15.177:6813/3486820237","state":["exists","up"]},{"osd":6,"uuid":"aa72392f-0498-44db-a8d2-eccc89a64871","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":2579716443},{"type":"v1","addr":"172.21.15.90:6819","nonce":2579716443}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6820","nonce":2579716443},{"type":"v1","addr":"172.21.15.90:6821","nonce":2579716443}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":2579716443},{"type":"v1","addr":"172.21.15.90:6825","nonce":2579716443}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2579716443},{"type":"v1","addr":"172.21.15.90:6823","nonce":2579716443}]},"public_addr":"172.21.15.90:6819/2579716443","cluster_addr":"172.21.15.90:6821/2579716443","heartbeat_back_addr":"172.21.15.90:6825/2579716443","heartbeat_front_addr":"172.21.15.90:6823/2579716443","state":["exists","up"]},{"osd":7,"uuid":"931bd105-bc14-4a1e-bfe4-c7ad58fa4b73","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":3371649139},{"type":"v1","addr":"172.21.15.92:6817","nonce":3371649139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":3371649139},{"type":"v1","addr":"172.21.15.92:6819","nonce":3371649139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":3371649139},{"type":"v1","addr":"172.21.15.92:6823","nonce":3371649139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6820","nonce":3371649139},{"type":"v1","addr":"172.21.15.92:6821","nonce":3371649139}]},"public_addr":"172.21.15.92:6817/3371649139","cluster_addr":"172.21.15.92:6819/3371649139","heartbeat_back_addr":"172.21.15.92:6823/3371649139","heartbeat_front_addr":"172.21.15.92:6821/3371649139","state":["exists","up"]},{"osd":8,"uuid":"82624ec2-b382-4886-9f40-7f5f376de3ce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":1459053034},{"type":"v1","addr":"172.21.15.177:6817","nonce":1459053034}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":1459053034},{"type":"v1","addr":"172.21.15.177:6819","nonce":1459053034}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":1459053034},{"type":"v1","addr":"172.21.15.177:6823","nonce":1459053034}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":1459053034},{"type":"v1","addr":"172.21.15.177:6821","nonce":1459053034}]},"public_addr":"172.21.15.177:6817/1459053034","cluster_addr":"172.21.15.177:6819/1459053034","heartbeat_back_addr":"172.21.15.177:6823/1459053034","heartbeat_front_addr":"172.21.15.177:6821/1459053034","state":["exists","up"]},{"osd":9,"uuid":"fbe36e48-3468-42b2-8094-e16552f3cb16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":2592604332},{"type":"v1","addr":"172.21.15.90:6827","nonce":2592604332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":2592604332},{"type":"v1","addr":"172.21.15.90:6829","nonce":2592604332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6832","nonce":2592604332},{"type":"v1","addr":"172.21.15.90:6833","nonce":2592604332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":2592604332},{"type":"v1","addr":"172.21.15.90:6831","nonce":2592604332}]},"public_addr":"172.21.15.90:6827/2592604332","cluster_addr":"172.21.15.90:6829/2592604332","heartbeat_back_addr":"172.21.15.90:6833/2592604332","heartbeat_front_addr":"172.21.15.90:6831/2592604332","state":["exists","up"]},{"osd":10,"uuid":"9240df75-e7e2-47f1-8aae-534b70b3c210","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":1594153142},{"type":"v1","addr":"172.21.15.92:6825","nonce":1594153142}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6826","nonce":1594153142},{"type":"v1","addr":"172.21.15.92:6827","nonce":1594153142}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":1594153142},{"type":"v1","addr":"172.21.15.92:6831","nonce":1594153142}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":1594153142},{"type":"v1","addr":"172.21.15.92:6829","nonce":1594153142}]},"public_addr":"172.21.15.92:6825/1594153142","cluster_addr":"172.21.15.92:6827/1594153142","heartbeat_back_addr":"172.21.15.92:6831/1594153142","heartbeat_front_addr":"172.21.15.92:6829/1594153142","state":["exists","up"]},{"osd":11,"uuid":"83cba9fa-b745-4df1-adda-e86837d4b91a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":83,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1622582126},{"type":"v1","addr":"172.21.15.177:6825","nonce":1622582126}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":1622582126},{"type":"v1","addr":"172.21.15.177:6827","nonce":1622582126}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":1622582126},{"type":"v1","addr":"172.21.15.177:6831","nonce":1622582126}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1622582126},{"type":"v1","addr":"172.21.15.177:6829","nonce":1622582126}]},"public_addr":"172.21.15.177:6825/1622582126","cluster_addr":"172.21.15.177:6827/1622582126","heartbeat_back_addr":"172.21.15.177:6831/1622582126","heartbeat_front_addr":"172.21.15.177:6829/1622582126","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:26:24.290365+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:26:49.405988+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:27:13.639196+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:27:45.603746+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:28:11.019126+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:28:35.084163+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:29:08.064626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:29:33.266133+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:29:57.381816+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:30:30.286847+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:30:56.226279+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:31:20.809611+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":7,"to":11}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":8,"to":4}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.11","mappings":[{"from":4,"to":8},{"from":5,"to":9}]},{"pgid":"3.21","mappings":[{"from":5,"to":9}]},{"pgid":"3.23","mappings":[{"from":4,"to":7}]},{"pgid":"3.3a","mappings":[{"from":4,"to":6}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.90:0/2805994468":"2024-04-03T14:34:52.235215+0000","172.21.15.90:0/1431307002":"2024-04-03T14:34:52.235215+0000","172.21.15.90:6801/4266054817":"2024-04-03T14:34:52.235215+0000","172.21.15.90:0/314641601":"2024-04-03T14:24:34.595561+0000","172.21.15.90:0/1400895868":"2024-04-03T14:24:34.595561+0000","172.21.15.90:6801/129436436":"2024-04-03T14:24:34.595561+0000","172.21.15.90:0/1335673697":"2024-04-03T14:34:52.235215+0000","172.21.15.90:6801/2372398598":"2024-04-03T14:24:00.362147+0000","172.21.15.90:0/278569338":"2024-04-03T14:24:34.595561+0000","172.21.15.90:0/2321464594":"2024-04-03T14:24:00.362147+0000","172.21.15.90:6800/4266054817":"2024-04-03T14:34:52.235215+0000","172.21.15.90:6800/2372398598":"2024-04-03T14:24:00.362147+0000","172.21.15.90:0/2962656672":"2024-04-03T14:24:00.362147+0000","172.21.15.90:0/126693610":"2024-04-03T14:24:00.362147+0000","172.21.15.90:0/1757385857":"2024-04-03T14:34:52.235215+0000","172.21.15.90:6800/129436436":"2024-04-03T14:24:34.595561+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":14,"length":1},{"begin":27,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T14:51:30.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:30.359+0000 7f576f800640 1 -- 172.21.15.90:0/2623247184 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f57500a3d20 msgr2=0x7f57500a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:51:30.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:30.359+0000 7f576f800640 1 --2- 172.21.15.90:0/2623247184 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f57500a3d20 0x7f57500a61e0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f575c009bd0 tx=0x7f575c005f90 comp rx=0 tx=0).stop 2024-04-02T14:51:30.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:30.359+0000 7f576f800640 1 -- 172.21.15.90:0/2623247184 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f576810d3f0 msgr2=0x7f57681a4c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:51:30.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:30.359+0000 7f576f800640 1 --2- 172.21.15.90:0/2623247184 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f576810d3f0 0x7f57681a4c00 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f575402c240 tx=0x7f5754004360 comp rx=0 tx=0).stop 2024-04-02T14:51:30.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:30.359+0000 7f576f800640 1 -- 172.21.15.90:0/2623247184 shutdown_connections 2024-04-02T14:51:30.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:30.359+0000 7f576f800640 1 --2- 172.21.15.90:0/2623247184 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f57500a3d20 0x7f57500a61e0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:51:30.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:30.359+0000 7f576f800640 1 --2- 172.21.15.90:0/2623247184 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f576819e830 0x7f576819ece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:51:30.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:30.359+0000 7f576f800640 1 --2- 172.21.15.90:0/2623247184 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f576810d3f0 0x7f57681a4c00 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:51:30.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:30.359+0000 7f576f800640 1 --2- 172.21.15.90:0/2623247184 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5768108320 0x7f57681a46c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:51:30.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:30.359+0000 7f576f800640 1 -- 172.21.15.90:0/2623247184 >> 172.21.15.90:0/2623247184 conn(0x7f5768076050 msgr2=0x7f57680783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:51:30.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:30.359+0000 7f576f800640 1 -- 172.21.15.90:0/2623247184 shutdown_connections 2024-04-02T14:51:30.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:30.359+0000 7f576f800640 1 -- 172.21.15.90:0/2623247184 wait complete. 2024-04-02T14:51:30.371 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-02T14:27:16.758622+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-02T14:32:20.016331+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-02T14:32:21.294367+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '110', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 27, 'snap_epoch': 110, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-02T14:51:30.372 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-02T14:51:30.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:30.593+0000 7ff011a00640 1 -- 172.21.15.90:0/691938539 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff00c1048d0 msgr2=0x7ff00c108910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:51:30.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:30.593+0000 7ff011a00640 1 --2- 172.21.15.90:0/691938539 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff00c1048d0 0x7ff00c108910 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7feff4009ca0 tx=0x7feff402c100 comp rx=0 tx=0).stop 2024-04-02T14:51:30.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:30.593+0000 7ff011a00640 1 -- 172.21.15.90:0/691938539 shutdown_connections 2024-04-02T14:51:30.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:30.593+0000 7ff011a00640 1 --2- 172.21.15.90:0/691938539 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff00c108ff0 0x7ff00c110bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:51:30.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:30.593+0000 7ff011a00640 1 --2- 172.21.15.90:0/691938539 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff00c1048d0 0x7ff00c108910 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:51:30.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:30.593+0000 7ff011a00640 1 --2- 172.21.15.90:0/691938539 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff00c103f20 0x7ff00c104300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:51:30.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:30.593+0000 7ff011a00640 1 -- 172.21.15.90:0/691938539 >> 172.21.15.90:0/691938539 conn(0x7ff00c0ff780 msgr2=0x7ff00c101ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:51:30.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:30.593+0000 7ff011a00640 1 -- 172.21.15.90:0/691938539 shutdown_connections 2024-04-02T14:51:30.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:30.593+0000 7ff011a00640 1 -- 172.21.15.90:0/691938539 wait complete. 2024-04-02T14:51:30.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:30.595+0000 7ff011a00640 1 Processor -- start 2024-04-02T14:51:30.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:30.595+0000 7ff011a00640 1 -- start start 2024-04-02T14:51:30.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:30.595+0000 7ff011a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff00c103f20 0x7ff00c19a600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:51:30.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:30.595+0000 7ff011a00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff00c108ff0 0x7ff00c19ab40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:51:30.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:30.595+0000 7ff011a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff00c19ee70 0x7ff00c19f320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:51:30.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:30.595+0000 7ff011a00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7ff00c1140a0 con 0x7ff00c103f20 2024-04-02T14:51:30.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:30.595+0000 7ff011a00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7ff00c113f20 con 0x7ff00c108ff0 2024-04-02T14:51:30.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:30.595+0000 7ff011a00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7ff00c114220 con 0x7ff00c19ee70 2024-04-02T14:51:30.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:30.596+0000 7ff00aa00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff00c103f20 0x7ff00c19a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:51:30.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:30.596+0000 7ff00b400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff00c19ee70 0x7ff00c19f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:51:30.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:30.596+0000 7ff00a000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff00c108ff0 0x7ff00c19ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:51:30.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:30.596+0000 7ff00b400640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff00c19ee70 0x7ff00c19f320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:51736/0 (socket says 172.21.15.90:51736) 2024-04-02T14:51:30.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:30.596+0000 7ff00b400640 1 -- 172.21.15.90:0/882655103 learned_addr learned my addr 172.21.15.90:0/882655103 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:51:30.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:30.596+0000 7ff00aa00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff00c103f20 0x7ff00c19a600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:35018/0 (socket says 172.21.15.90:35018) 2024-04-02T14:51:30.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:30.596+0000 7ff00b400640 1 -- 172.21.15.90:0/882655103 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff00c108ff0 msgr2=0x7ff00c19ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:51:30.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:30.596+0000 7ff00b400640 1 --2- 172.21.15.90:0/882655103 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff00c108ff0 0x7ff00c19ab40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:51:30.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:30.596+0000 7ff00b400640 1 -- 172.21.15.90:0/882655103 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff00c103f20 msgr2=0x7ff00c19a600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:51:30.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:30.596+0000 7ff00b400640 1 --2- 172.21.15.90:0/882655103 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff00c103f20 0x7ff00c19a600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:51:30.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:30.596+0000 7ff00b400640 1 -- 172.21.15.90:0/882655103 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff00c19f960 con 0x7ff00c19ee70 2024-04-02T14:51:30.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:30.596+0000 7ff00a000640 1 --2- 172.21.15.90:0/882655103 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff00c108ff0 0x7ff00c19ab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:51:30.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:30.596+0000 7ff00b400640 1 --2- 172.21.15.90:0/882655103 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff00c19ee70 0x7ff00c19f320 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7feffc00e970 tx=0x7feffc00ed70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:51:30.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:30.597+0000 7ff003400640 1 -- 172.21.15.90:0/882655103 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feffc00ccc0 con 0x7ff00c19ee70 2024-04-02T14:51:30.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:30.597+0000 7ff003400640 1 -- 172.21.15.90:0/882655103 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feffc00ce60 con 0x7ff00c19ee70 2024-04-02T14:51:30.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:30.597+0000 7ff003400640 1 -- 172.21.15.90:0/882655103 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feffc010490 con 0x7ff00c19ee70 2024-04-02T14:51:30.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:30.597+0000 7ff011a00640 1 -- 172.21.15.90:0/882655103 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff00c19fc50 con 0x7ff00c19ee70 2024-04-02T14:51:30.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:30.597+0000 7ff011a00640 1 -- 172.21.15.90:0/882655103 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff00c1a73c0 con 0x7ff00c19ee70 2024-04-02T14:51:30.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:30.598+0000 7ff011a00640 1 -- 172.21.15.90:0/882655103 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff00c068d40 con 0x7ff00c19ee70 2024-04-02T14:51:30.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:30.600+0000 7ff003400640 1 -- 172.21.15.90:0/882655103 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7feffc005020 con 0x7ff00c19ee70 2024-04-02T14:51:30.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:30.601+0000 7ff003400640 1 --2- 172.21.15.90:0/882655103 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fefec0a3d20 0x7fefec0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:51:30.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:30.601+0000 7ff003400640 1 -- 172.21.15.90:0/882655103 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 8550+0+0 (secure 0 0 0) 0x7feffc0d0c00 con 0x7ff00c19ee70 2024-04-02T14:51:30.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:30.602+0000 7ff00aa00640 1 --2- 172.21.15.90:0/882655103 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fefec0a3d20 0x7fefec0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:51:30.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:30.602+0000 7ff00aa00640 1 --2- 172.21.15.90:0/882655103 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fefec0a3d20 0x7fefec0a61e0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7ff004009bd0 tx=0x7ff004005f90 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:51:30.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:30.604+0000 7ff003400640 1 -- 172.21.15.90:0/882655103 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7feffc09d220 con 0x7ff00c19ee70 2024-04-02T14:51:30.601 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:51:30 smithi092 ceph-mon[36497]: pgmap v521: 97 pgs: 97 active+clean; 3.9 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 5.7 KiB/s wr, 10 op/s 2024-04-02T14:51:30.678 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:51:30 smithi177 ceph-mon[36431]: pgmap v521: 97 pgs: 97 active+clean; 3.9 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 5.7 KiB/s wr, 10 op/s 2024-04-02T14:51:30.679 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:51:30 smithi090 ceph-mon[31674]: pgmap v521: 97 pgs: 97 active+clean; 3.9 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 5.7 KiB/s wr, 10 op/s 2024-04-02T14:51:30.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:30.809+0000 7ff011a00640 1 -- 172.21.15.90:0/882655103 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7ff00c1048d0 con 0x7ff00c19ee70 2024-04-02T14:51:30.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:30.810+0000 7ff003400640 1 -- 172.21.15.90:0/882655103 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v111) ==== 93+0+10 (secure 0 0 0) 0x7feffc09cb20 con 0x7ff00c19ee70 2024-04-02T14:51:30.802 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 1 2024-04-02T14:51:30.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:30.812+0000 7ff011a00640 1 -- 172.21.15.90:0/882655103 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fefec0a3d20 msgr2=0x7fefec0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:51:30.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:30.812+0000 7ff011a00640 1 --2- 172.21.15.90:0/882655103 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fefec0a3d20 0x7fefec0a61e0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7ff004009bd0 tx=0x7ff004005f90 comp rx=0 tx=0).stop 2024-04-02T14:51:30.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:30.812+0000 7ff011a00640 1 -- 172.21.15.90:0/882655103 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff00c19ee70 msgr2=0x7ff00c19f320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:51:30.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:30.812+0000 7ff011a00640 1 --2- 172.21.15.90:0/882655103 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff00c19ee70 0x7ff00c19f320 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7feffc00e970 tx=0x7feffc00ed70 comp rx=0 tx=0).stop 2024-04-02T14:51:30.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:30.812+0000 7ff011a00640 1 -- 172.21.15.90:0/882655103 shutdown_connections 2024-04-02T14:51:30.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:30.812+0000 7ff011a00640 1 --2- 172.21.15.90:0/882655103 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fefec0a3d20 0x7fefec0a61e0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:51:30.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:30.812+0000 7ff011a00640 1 --2- 172.21.15.90:0/882655103 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff00c19ee70 0x7ff00c19f320 unknown :-1 s=CLOSED pgs=855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:51:30.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:30.812+0000 7ff011a00640 1 --2- 172.21.15.90:0/882655103 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff00c108ff0 0x7ff00c19ab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:51:30.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:30.812+0000 7ff011a00640 1 --2- 172.21.15.90:0/882655103 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff00c103f20 0x7ff00c19a600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:51:30.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:30.812+0000 7ff011a00640 1 -- 172.21.15.90:0/882655103 >> 172.21.15.90:0/882655103 conn(0x7ff00c0ff780 msgr2=0x7ff00c101ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:51:30.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:30.813+0000 7ff011a00640 1 -- 172.21.15.90:0/882655103 shutdown_connections 2024-04-02T14:51:30.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:30.813+0000 7ff011a00640 1 -- 172.21.15.90:0/882655103 wait complete. 2024-04-02T14:51:30.823 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-02T14:51:31.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:31.034+0000 7f03bac00640 1 -- 172.21.15.90:0/349869066 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f03b4103f10 msgr2=0x7f03b41042f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:51:31.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:31.034+0000 7f03bac00640 1 --2- 172.21.15.90:0/349869066 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f03b4103f10 0x7f03b41042f0 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7f03a4009900 tx=0x7f03a402bc30 comp rx=0 tx=0).stop 2024-04-02T14:51:31.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:31.034+0000 7f03bac00640 1 -- 172.21.15.90:0/349869066 shutdown_connections 2024-04-02T14:51:31.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:31.034+0000 7f03bac00640 1 --2- 172.21.15.90:0/349869066 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f03b4108fe0 0x7f03b4110bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:51:31.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:31.034+0000 7f03bac00640 1 --2- 172.21.15.90:0/349869066 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f03b41048c0 0x7f03b4108900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:51:31.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:31.034+0000 7f03bac00640 1 --2- 172.21.15.90:0/349869066 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f03b4103f10 0x7f03b41042f0 unknown :-1 s=CLOSED pgs=1048 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:51:31.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:31.034+0000 7f03bac00640 1 -- 172.21.15.90:0/349869066 >> 172.21.15.90:0/349869066 conn(0x7f03b40ff750 msgr2=0x7f03b4101b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:51:31.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:31.034+0000 7f03bac00640 1 -- 172.21.15.90:0/349869066 shutdown_connections 2024-04-02T14:51:31.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:31.034+0000 7f03bac00640 1 -- 172.21.15.90:0/349869066 wait complete. 2024-04-02T14:51:31.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:31.035+0000 7f03bac00640 1 Processor -- start 2024-04-02T14:51:31.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:31.035+0000 7f03bac00640 1 -- start start 2024-04-02T14:51:31.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:31.036+0000 7f03bac00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f03b4103f10 0x7f03b419a5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:51:31.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:31.036+0000 7f03bac00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f03b4108fe0 0x7f03b419ab10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:51:31.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:31.036+0000 7f03bac00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f03b419ee90 0x7f03b419f340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:51:31.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:31.036+0000 7f03bac00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f03b4114090 con 0x7f03b419ee90 2024-04-02T14:51:31.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:31.036+0000 7f03bac00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f03b4113f10 con 0x7f03b4108fe0 2024-04-02T14:51:31.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:31.036+0000 7f03bac00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f03b4114210 con 0x7f03b4103f10 2024-04-02T14:51:31.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:31.036+0000 7f03b3e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f03b4103f10 0x7f03b419a5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:51:31.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:31.036+0000 7f03ab400640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f03b4108fe0 0x7f03b419ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:51:31.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:31.036+0000 7f03b8c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f03b419ee90 0x7f03b419f340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:51:31.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:31.036+0000 7f03b3e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f03b4103f10 0x7f03b419a5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:51758/0 (socket says 172.21.15.90:51758) 2024-04-02T14:51:31.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:31.036+0000 7f03b3e00640 1 -- 172.21.15.90:0/3785006154 learned_addr learned my addr 172.21.15.90:0/3785006154 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:51:31.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:31.036+0000 7f03ab400640 1 --2- 172.21.15.90:0/3785006154 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f03b4108fe0 0x7f03b419ab10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:55958/0 (socket says 172.21.15.90:55958) 2024-04-02T14:51:31.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:31.036+0000 7f03ab400640 1 -- 172.21.15.90:0/3785006154 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f03b4103f10 msgr2=0x7f03b419a5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:51:31.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:31.036+0000 7f03ab400640 1 --2- 172.21.15.90:0/3785006154 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f03b4103f10 0x7f03b419a5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:51:31.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:31.036+0000 7f03ab400640 1 -- 172.21.15.90:0/3785006154 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f03b419ee90 msgr2=0x7f03b419f340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:51:31.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:31.036+0000 7f03ab400640 1 --2- 172.21.15.90:0/3785006154 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f03b419ee90 0x7f03b419f340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:51:31.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:31.036+0000 7f03ab400640 1 -- 172.21.15.90:0/3785006154 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03b419f980 con 0x7f03b4108fe0 2024-04-02T14:51:31.031 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:31.036+0000 7f03b8c00640 1 --2- 172.21.15.90:0/3785006154 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f03b419ee90 0x7f03b419f340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:51:31.031 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:31.037+0000 7f03ab400640 1 --2- 172.21.15.90:0/3785006154 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f03b4108fe0 0x7f03b419ab10 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7f039c00e9c0 tx=0x7f039c00ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:51:31.031 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:31.037+0000 7f03b1600640 1 -- 172.21.15.90:0/3785006154 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f039c00ccc0 con 0x7f03b4108fe0 2024-04-02T14:51:31.031 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:31.037+0000 7f03b1600640 1 -- 172.21.15.90:0/3785006154 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f039c010910 con 0x7f03b4108fe0 2024-04-02T14:51:31.031 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:31.037+0000 7f03bac00640 1 -- 172.21.15.90:0/3785006154 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03b419fc70 con 0x7f03b4108fe0 2024-04-02T14:51:31.031 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:31.037+0000 7f03b1600640 1 -- 172.21.15.90:0/3785006154 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f039c0188c0 con 0x7f03b4108fe0 2024-04-02T14:51:31.031 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:31.037+0000 7f03bac00640 1 -- 172.21.15.90:0/3785006154 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03b41a7460 con 0x7f03b4108fe0 2024-04-02T14:51:31.031 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:31.038+0000 7f03bac00640 1 -- 172.21.15.90:0/3785006154 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f03b4068d40 con 0x7f03b4108fe0 2024-04-02T14:51:31.032 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:31.040+0000 7f03b1600640 1 -- 172.21.15.90:0/3785006154 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f039c010ac0 con 0x7f03b4108fe0 2024-04-02T14:51:31.033 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:31.042+0000 7f03b1600640 1 --2- 172.21.15.90:0/3785006154 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f03980a3d20 0x7f03980a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:51:31.034 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:31.042+0000 7f03b1600640 1 -- 172.21.15.90:0/3785006154 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 8550+0+0 (secure 0 0 0) 0x7f039c0d10e0 con 0x7f03b4108fe0 2024-04-02T14:51:31.034 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:31.042+0000 7f03b3e00640 1 --2- 172.21.15.90:0/3785006154 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f03980a3d20 0x7f03980a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:51:31.034 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:31.043+0000 7f03b3e00640 1 --2- 172.21.15.90:0/3785006154 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f03980a3d20 0x7f03980a61e0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f03a402cfd0 tx=0x7f03a40058d0 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:51:31.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:31.044+0000 7f03b1600640 1 -- 172.21.15.90:0/3785006154 <== mon.2 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7f039c09d0b0 con 0x7f03b4108fe0 2024-04-02T14:51:31.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:31.245+0000 7f03bac00640 1 -- 172.21.15.90:0/3785006154 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f03b41048c0 con 0x7f03b4108fe0 2024-04-02T14:51:31.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:31.246+0000 7f03b1600640 1 -- 172.21.15.90:0/3785006154 <== mon.2 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v111) ==== 104+0+11 (secure 0 0 0) 0x7f039c0a13b0 con 0x7f03b4108fe0 2024-04-02T14:51:31.238 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-04-02T14:51:31.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:31.249+0000 7f03bac00640 1 -- 172.21.15.90:0/3785006154 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f03980a3d20 msgr2=0x7f03980a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:51:31.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:31.249+0000 7f03bac00640 1 --2- 172.21.15.90:0/3785006154 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f03980a3d20 0x7f03980a61e0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f03a402cfd0 tx=0x7f03a40058d0 comp rx=0 tx=0).stop 2024-04-02T14:51:31.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:31.249+0000 7f03bac00640 1 -- 172.21.15.90:0/3785006154 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f03b4108fe0 msgr2=0x7f03b419ab10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:51:31.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:31.249+0000 7f03bac00640 1 --2- 172.21.15.90:0/3785006154 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f03b4108fe0 0x7f03b419ab10 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7f039c00e9c0 tx=0x7f039c00ee80 comp rx=0 tx=0).stop 2024-04-02T14:51:31.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:31.249+0000 7f03bac00640 1 -- 172.21.15.90:0/3785006154 shutdown_connections 2024-04-02T14:51:31.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:31.249+0000 7f03bac00640 1 --2- 172.21.15.90:0/3785006154 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f03980a3d20 0x7f03980a61e0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:51:31.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:31.249+0000 7f03bac00640 1 --2- 172.21.15.90:0/3785006154 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f03b419ee90 0x7f03b419f340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:51:31.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:31.249+0000 7f03bac00640 1 --2- 172.21.15.90:0/3785006154 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f03b4108fe0 0x7f03b419ab10 unknown :-1 s=CLOSED pgs=1049 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:51:31.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:31.249+0000 7f03bac00640 1 --2- 172.21.15.90:0/3785006154 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f03b4103f10 0x7f03b419a5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:51:31.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:31.249+0000 7f03bac00640 1 -- 172.21.15.90:0/3785006154 >> 172.21.15.90:0/3785006154 conn(0x7f03b40ff750 msgr2=0x7f03b4100a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:51:31.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:31.249+0000 7f03bac00640 1 -- 172.21.15.90:0/3785006154 shutdown_connections 2024-04-02T14:51:31.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:31.249+0000 7f03bac00640 1 -- 172.21.15.90:0/3785006154 wait complete. 2024-04-02T14:51:31.262 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-02T14:51:31.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:31.496+0000 7faf6ac00640 1 -- 172.21.15.90:0/1427035477 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faf641048c0 msgr2=0x7faf64108900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:51:31.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:31.496+0000 7faf6ac00640 1 --2- 172.21.15.90:0/1427035477 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faf641048c0 0x7faf64108900 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7faf4c009900 tx=0x7faf4c02bc40 comp rx=0 tx=0).stop 2024-04-02T14:51:31.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:31.496+0000 7faf6ac00640 1 -- 172.21.15.90:0/1427035477 shutdown_connections 2024-04-02T14:51:31.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:31.496+0000 7faf6ac00640 1 --2- 172.21.15.90:0/1427035477 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7faf64108fe0 0x7faf64110bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:51:31.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:31.496+0000 7faf6ac00640 1 --2- 172.21.15.90:0/1427035477 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faf641048c0 0x7faf64108900 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:51:31.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:31.496+0000 7faf6ac00640 1 --2- 172.21.15.90:0/1427035477 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faf64103f10 0x7faf641042f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:51:31.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:31.496+0000 7faf6ac00640 1 -- 172.21.15.90:0/1427035477 >> 172.21.15.90:0/1427035477 conn(0x7faf640ff750 msgr2=0x7faf64101b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:51:31.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:31.496+0000 7faf6ac00640 1 -- 172.21.15.90:0/1427035477 shutdown_connections 2024-04-02T14:51:31.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:31.496+0000 7faf6ac00640 1 -- 172.21.15.90:0/1427035477 wait complete. 2024-04-02T14:51:31.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:31.497+0000 7faf6ac00640 1 Processor -- start 2024-04-02T14:51:31.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:31.497+0000 7faf6ac00640 1 -- start start 2024-04-02T14:51:31.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:31.498+0000 7faf6ac00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faf64103f10 0x7faf6419a5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:51:31.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:31.498+0000 7faf6ac00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7faf64108fe0 0x7faf6419ab00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:51:31.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:31.498+0000 7faf6ac00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faf6419ee80 0x7faf6419f330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:51:31.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:31.498+0000 7faf6ac00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7faf641140a0 con 0x7faf64103f10 2024-04-02T14:51:31.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:31.498+0000 7faf6ac00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7faf64113f20 con 0x7faf64108fe0 2024-04-02T14:51:31.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:31.498+0000 7faf6ac00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7faf64114220 con 0x7faf6419ee80 2024-04-02T14:51:31.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:31.498+0000 7faf63e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faf64103f10 0x7faf6419a5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:51:31.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:31.498+0000 7faf63e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faf64103f10 0x7faf6419a5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:35058/0 (socket says 172.21.15.90:35058) 2024-04-02T14:51:31.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:31.498+0000 7faf63e00640 1 -- 172.21.15.90:0/1345876550 learned_addr learned my addr 172.21.15.90:0/1345876550 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:51:31.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:31.498+0000 7faf68c00640 1 --2- 172.21.15.90:0/1345876550 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faf6419ee80 0x7faf6419f330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:51:31.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:31.498+0000 7faf63400640 1 --2- 172.21.15.90:0/1345876550 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7faf64108fe0 0x7faf6419ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:51:31.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:31.498+0000 7faf63e00640 1 -- 172.21.15.90:0/1345876550 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faf6419ee80 msgr2=0x7faf6419f330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:51:31.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:31.498+0000 7faf63e00640 1 --2- 172.21.15.90:0/1345876550 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faf6419ee80 0x7faf6419f330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:51:31.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:31.498+0000 7faf63e00640 1 -- 172.21.15.90:0/1345876550 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7faf64108fe0 msgr2=0x7faf6419ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:51:31.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:31.498+0000 7faf63e00640 1 --2- 172.21.15.90:0/1345876550 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7faf64108fe0 0x7faf6419ab00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:51:31.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:31.498+0000 7faf63e00640 1 -- 172.21.15.90:0/1345876550 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf6419f970 con 0x7faf64103f10 2024-04-02T14:51:31.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:31.499+0000 7faf63400640 1 --2- 172.21.15.90:0/1345876550 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7faf64108fe0 0x7faf6419ab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:51:31.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:31.499+0000 7faf63e00640 1 --2- 172.21.15.90:0/1345876550 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faf64103f10 0x7faf6419a5c0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7faf5c009d50 tx=0x7faf5c00c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:51:31.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:31.499+0000 7faf60c00640 1 -- 172.21.15.90:0/1345876550 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf5c003b70 con 0x7faf64103f10 2024-04-02T14:51:31.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:31.499+0000 7faf60c00640 1 -- 172.21.15.90:0/1345876550 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faf5c003d10 con 0x7faf64103f10 2024-04-02T14:51:31.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:31.499+0000 7faf6ac00640 1 -- 172.21.15.90:0/1345876550 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faf6419fc60 con 0x7faf64103f10 2024-04-02T14:51:31.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:31.499+0000 7faf60c00640 1 -- 172.21.15.90:0/1345876550 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf5c010a20 con 0x7faf64103f10 2024-04-02T14:51:31.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:31.499+0000 7faf6ac00640 1 -- 172.21.15.90:0/1345876550 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faf641a7470 con 0x7faf64103f10 2024-04-02T14:51:31.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:31.500+0000 7faf6ac00640 1 -- 172.21.15.90:0/1345876550 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faf64068d40 con 0x7faf64103f10 2024-04-02T14:51:31.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:31.502+0000 7faf60c00640 1 -- 172.21.15.90:0/1345876550 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7faf5c003ec0 con 0x7faf64103f10 2024-04-02T14:51:31.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:31.504+0000 7faf60c00640 1 --2- 172.21.15.90:0/1345876550 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7faf480a3da0 0x7faf480a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:51:31.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:31.504+0000 7faf60c00640 1 -- 172.21.15.90:0/1345876550 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 8550+0+0 (secure 0 0 0) 0x7faf5c0d0f30 con 0x7faf64103f10 2024-04-02T14:51:31.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:31.504+0000 7faf63400640 1 --2- 172.21.15.90:0/1345876550 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7faf480a3da0 0x7faf480a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:51:31.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:31.505+0000 7faf63400640 1 --2- 172.21.15.90:0/1345876550 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7faf480a3da0 0x7faf480a6260 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7faf4c02c250 tx=0x7faf4c005fd0 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:51:31.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:31.508+0000 7faf60c00640 1 -- 172.21.15.90:0/1345876550 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7faf5c09d550 con 0x7faf64103f10 2024-04-02T14:51:31.601 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:51:31 smithi092 ceph-mon[36497]: from='client.? 172.21.15.90:0/2623247184' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-02T14:51:31.601 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:51:31 smithi092 ceph-mon[36497]: from='client.? 172.21.15.90:0/882655103' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-02T14:51:31.602 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:51:31 smithi092 ceph-mon[36497]: from='client.? 172.21.15.90:0/3785006154' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-02T14:51:31.677 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:51:31 smithi177 ceph-mon[36431]: from='client.? 172.21.15.90:0/2623247184' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-02T14:51:31.678 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:51:31 smithi177 ceph-mon[36431]: from='client.? 172.21.15.90:0/882655103' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-02T14:51:31.678 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:51:31 smithi177 ceph-mon[36431]: from='client.? 172.21.15.90:0/3785006154' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-02T14:51:31.679 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:51:31 smithi090 ceph-mon[31674]: from='client.? 172.21.15.90:0/2623247184' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-02T14:51:31.679 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:51:31 smithi090 ceph-mon[31674]: from='client.? 172.21.15.90:0/882655103' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-02T14:51:31.679 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:51:31 smithi090 ceph-mon[31674]: from='client.? 172.21.15.90:0/3785006154' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-02T14:51:31.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:31.711+0000 7faf6ac00640 1 -- 172.21.15.90:0/1345876550 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7faf641048c0 con 0x7faf64103f10 2024-04-02T14:51:31.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:31.712+0000 7faf60c00640 1 -- 172.21.15.90:0/1345876550 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v111) ==== 100+0+11 (secure 0 0 0) 0x7faf5c09ce50 con 0x7faf64103f10 2024-04-02T14:51:31.704 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 64 2024-04-02T14:51:31.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:31.714+0000 7faf6ac00640 1 -- 172.21.15.90:0/1345876550 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7faf480a3da0 msgr2=0x7faf480a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:51:31.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:31.714+0000 7faf6ac00640 1 --2- 172.21.15.90:0/1345876550 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7faf480a3da0 0x7faf480a6260 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7faf4c02c250 tx=0x7faf4c005fd0 comp rx=0 tx=0).stop 2024-04-02T14:51:31.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:31.714+0000 7faf6ac00640 1 -- 172.21.15.90:0/1345876550 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faf64103f10 msgr2=0x7faf6419a5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:51:31.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:31.714+0000 7faf6ac00640 1 --2- 172.21.15.90:0/1345876550 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faf64103f10 0x7faf6419a5c0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7faf5c009d50 tx=0x7faf5c00c6a0 comp rx=0 tx=0).stop 2024-04-02T14:51:31.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:31.714+0000 7faf6ac00640 1 -- 172.21.15.90:0/1345876550 shutdown_connections 2024-04-02T14:51:31.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:31.714+0000 7faf6ac00640 1 --2- 172.21.15.90:0/1345876550 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7faf480a3da0 0x7faf480a6260 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:51:31.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:31.714+0000 7faf6ac00640 1 --2- 172.21.15.90:0/1345876550 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faf6419ee80 0x7faf6419f330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:51:31.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:31.714+0000 7faf6ac00640 1 --2- 172.21.15.90:0/1345876550 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7faf64108fe0 0x7faf6419ab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:51:31.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:31.715+0000 7faf6ac00640 1 --2- 172.21.15.90:0/1345876550 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faf64103f10 0x7faf6419a5c0 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:51:31.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:31.715+0000 7faf6ac00640 1 -- 172.21.15.90:0/1345876550 >> 172.21.15.90:0/1345876550 conn(0x7faf640ff750 msgr2=0x7faf64101ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:51:31.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:31.715+0000 7faf6ac00640 1 -- 172.21.15.90:0/1345876550 shutdown_connections 2024-04-02T14:51:31.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:31.715+0000 7faf6ac00640 1 -- 172.21.15.90:0/1345876550 wait complete. 2024-04-02T14:51:31.726 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-02T14:51:31.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:31.948+0000 7f24edc00640 1 -- 172.21.15.90:0/2554919237 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f24e80747f0 msgr2=0x7f24e810ddf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:51:31.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:31.948+0000 7f24edc00640 1 --2- 172.21.15.90:0/2554919237 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f24e80747f0 0x7f24e810ddf0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f24d4009900 tx=0x7f24d402bcc0 comp rx=0 tx=0).stop 2024-04-02T14:51:31.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:31.948+0000 7f24edc00640 1 -- 172.21.15.90:0/2554919237 shutdown_connections 2024-04-02T14:51:31.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:31.948+0000 7f24edc00640 1 --2- 172.21.15.90:0/2554919237 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f24e810e330 0x7f24e8110740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:51:31.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:31.948+0000 7f24edc00640 1 --2- 172.21.15.90:0/2554919237 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f24e80747f0 0x7f24e810ddf0 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:51:31.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:31.948+0000 7f24edc00640 1 --2- 172.21.15.90:0/2554919237 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f24e8073ed0 0x7f24e80742b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:51:31.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:31.948+0000 7f24edc00640 1 -- 172.21.15.90:0/2554919237 >> 172.21.15.90:0/2554919237 conn(0x7f24e8078f00 msgr2=0x7f24e8079310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:51:31.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:31.948+0000 7f24edc00640 1 -- 172.21.15.90:0/2554919237 shutdown_connections 2024-04-02T14:51:31.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:31.948+0000 7f24edc00640 1 -- 172.21.15.90:0/2554919237 wait complete. 2024-04-02T14:51:31.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:31.949+0000 7f24edc00640 1 Processor -- start 2024-04-02T14:51:31.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:31.949+0000 7f24edc00640 1 -- start start 2024-04-02T14:51:31.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:31.950+0000 7f24edc00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f24e8073ed0 0x7f24e819a650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:51:31.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:31.950+0000 7f24edc00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f24e80747f0 0x7f24e819ab90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:51:31.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:31.950+0000 7f24edc00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f24e810e330 0x7f24e819eec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:51:31.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:31.950+0000 7f24edc00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f24e8114090 con 0x7f24e8073ed0 2024-04-02T14:51:31.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:31.950+0000 7f24edc00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f24e8113f10 con 0x7f24e80747f0 2024-04-02T14:51:31.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:31.950+0000 7f24edc00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f24e8114210 con 0x7f24e810e330 2024-04-02T14:51:31.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:31.950+0000 7f24e6a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f24e8073ed0 0x7f24e819a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:51:31.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:31.950+0000 7f24e6a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f24e8073ed0 0x7f24e819a650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:35086/0 (socket says 172.21.15.90:35086) 2024-04-02T14:51:31.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:31.950+0000 7f24e6a00640 1 -- 172.21.15.90:0/4174740252 learned_addr learned my addr 172.21.15.90:0/4174740252 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:51:31.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:31.950+0000 7f24e6000640 1 --2- 172.21.15.90:0/4174740252 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f24e80747f0 0x7f24e819ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:51:31.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:31.950+0000 7f24e7400640 1 --2- 172.21.15.90:0/4174740252 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f24e810e330 0x7f24e819eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:51:31.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:31.951+0000 7f24e6a00640 1 -- 172.21.15.90:0/4174740252 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f24e810e330 msgr2=0x7f24e819eec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:51:31.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:31.951+0000 7f24e6a00640 1 --2- 172.21.15.90:0/4174740252 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f24e810e330 0x7f24e819eec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:51:31.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:31.951+0000 7f24e6a00640 1 -- 172.21.15.90:0/4174740252 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f24e80747f0 msgr2=0x7f24e819ab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:51:31.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:31.951+0000 7f24e6a00640 1 --2- 172.21.15.90:0/4174740252 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f24e80747f0 0x7f24e819ab90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:51:31.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:31.951+0000 7f24e6a00640 1 -- 172.21.15.90:0/4174740252 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24e819f5f0 con 0x7f24e8073ed0 2024-04-02T14:51:31.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:31.951+0000 7f24e6a00640 1 --2- 172.21.15.90:0/4174740252 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f24e8073ed0 0x7f24e819a650 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f24d800d950 tx=0x7f24d800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:51:31.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:31.951+0000 7f24df400640 1 -- 172.21.15.90:0/4174740252 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24d8017070 con 0x7f24e8073ed0 2024-04-02T14:51:31.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:31.951+0000 7f24df400640 1 -- 172.21.15.90:0/4174740252 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f24d8009c30 con 0x7f24e8073ed0 2024-04-02T14:51:31.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:31.951+0000 7f24edc00640 1 -- 172.21.15.90:0/4174740252 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f24e819f8e0 con 0x7f24e8073ed0 2024-04-02T14:51:31.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:31.951+0000 7f24df400640 1 -- 172.21.15.90:0/4174740252 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24d800f6b0 con 0x7f24e8073ed0 2024-04-02T14:51:31.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:31.951+0000 7f24edc00640 1 -- 172.21.15.90:0/4174740252 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24e81a7170 con 0x7f24e8073ed0 2024-04-02T14:51:31.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:31.953+0000 7f24edc00640 1 -- 172.21.15.90:0/4174740252 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f24e8068d40 con 0x7f24e8073ed0 2024-04-02T14:51:31.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:31.955+0000 7f24df400640 1 -- 172.21.15.90:0/4174740252 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f24d80109d0 con 0x7f24e8073ed0 2024-04-02T14:51:31.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:31.957+0000 7f24df400640 1 --2- 172.21.15.90:0/4174740252 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f24cc0a3d20 0x7f24cc0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:51:31.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:31.957+0000 7f24df400640 1 -- 172.21.15.90:0/4174740252 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 8550+0+0 (secure 0 0 0) 0x7f24d80d0410 con 0x7f24e8073ed0 2024-04-02T14:51:31.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:31.957+0000 7f24e6000640 1 --2- 172.21.15.90:0/4174740252 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f24cc0a3d20 0x7f24cc0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:51:31.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:31.958+0000 7f24e6000640 1 --2- 172.21.15.90:0/4174740252 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f24cc0a3d20 0x7f24cc0a61e0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f24d4009230 tx=0x7f24d402df90 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:51:31.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:31.959+0000 7f24df400640 1 -- 172.21.15.90:0/4174740252 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7f24d8009de0 con 0x7f24e8073ed0 2024-04-02T14:51:32.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:32.176+0000 7f24edc00640 1 -- 172.21.15.90:0/4174740252 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f24e819b950 con 0x7f24e8073ed0 2024-04-02T14:51:32.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:32.177+0000 7f24df400640 1 -- 172.21.15.90:0/4174740252 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 16 entries v111) ==== 81+0+941 (secure 0 0 0) 0x7f24e819b950 con 0x7f24e8073ed0 2024-04-02T14:51:32.169 INFO:teuthology.orchestra.run.smithi090.stdout:172.21.15.90:0/2805994468 2024-04-03T14:34:52.235215+0000 2024-04-02T14:51:32.169 INFO:teuthology.orchestra.run.smithi090.stdout:172.21.15.90:0/1431307002 2024-04-03T14:34:52.235215+0000 2024-04-02T14:51:32.169 INFO:teuthology.orchestra.run.smithi090.stdout:172.21.15.90:6801/4266054817 2024-04-03T14:34:52.235215+0000 2024-04-02T14:51:32.170 INFO:teuthology.orchestra.run.smithi090.stdout:172.21.15.90:0/314641601 2024-04-03T14:24:34.595561+0000 2024-04-02T14:51:32.170 INFO:teuthology.orchestra.run.smithi090.stdout:172.21.15.90:0/1400895868 2024-04-03T14:24:34.595561+0000 2024-04-02T14:51:32.170 INFO:teuthology.orchestra.run.smithi090.stdout:172.21.15.90:6801/129436436 2024-04-03T14:24:34.595561+0000 2024-04-02T14:51:32.170 INFO:teuthology.orchestra.run.smithi090.stdout:172.21.15.90:0/1335673697 2024-04-03T14:34:52.235215+0000 2024-04-02T14:51:32.170 INFO:teuthology.orchestra.run.smithi090.stdout:172.21.15.90:6801/2372398598 2024-04-03T14:24:00.362147+0000 2024-04-02T14:51:32.170 INFO:teuthology.orchestra.run.smithi090.stdout:172.21.15.90:0/278569338 2024-04-03T14:24:34.595561+0000 2024-04-02T14:51:32.171 INFO:teuthology.orchestra.run.smithi090.stdout:172.21.15.90:0/2321464594 2024-04-03T14:24:00.362147+0000 2024-04-02T14:51:32.171 INFO:teuthology.orchestra.run.smithi090.stdout:172.21.15.90:6800/4266054817 2024-04-03T14:34:52.235215+0000 2024-04-02T14:51:32.171 INFO:teuthology.orchestra.run.smithi090.stdout:172.21.15.90:6800/2372398598 2024-04-03T14:24:00.362147+0000 2024-04-02T14:51:32.171 INFO:teuthology.orchestra.run.smithi090.stdout:172.21.15.90:0/2962656672 2024-04-03T14:24:00.362147+0000 2024-04-02T14:51:32.171 INFO:teuthology.orchestra.run.smithi090.stdout:172.21.15.90:0/126693610 2024-04-03T14:24:00.362147+0000 2024-04-02T14:51:32.171 INFO:teuthology.orchestra.run.smithi090.stdout:172.21.15.90:0/1757385857 2024-04-03T14:34:52.235215+0000 2024-04-02T14:51:32.171 INFO:teuthology.orchestra.run.smithi090.stdout:172.21.15.90:6800/129436436 2024-04-03T14:24:34.595561+0000 2024-04-02T14:51:32.172 INFO:teuthology.orchestra.run.smithi090.stderr:listed 16 entries 2024-04-02T14:51:32.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:32.180+0000 7f24edc00640 1 -- 172.21.15.90:0/4174740252 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f24cc0a3d20 msgr2=0x7f24cc0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:51:32.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:32.180+0000 7f24edc00640 1 --2- 172.21.15.90:0/4174740252 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f24cc0a3d20 0x7f24cc0a61e0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f24d4009230 tx=0x7f24d402df90 comp rx=0 tx=0).stop 2024-04-02T14:51:32.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:32.180+0000 7f24edc00640 1 -- 172.21.15.90:0/4174740252 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f24e8073ed0 msgr2=0x7f24e819a650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:51:32.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:32.180+0000 7f24edc00640 1 --2- 172.21.15.90:0/4174740252 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f24e8073ed0 0x7f24e819a650 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f24d800d950 tx=0x7f24d800de10 comp rx=0 tx=0).stop 2024-04-02T14:51:32.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:32.180+0000 7f24edc00640 1 -- 172.21.15.90:0/4174740252 shutdown_connections 2024-04-02T14:51:32.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:32.180+0000 7f24edc00640 1 --2- 172.21.15.90:0/4174740252 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f24cc0a3d20 0x7f24cc0a61e0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:51:32.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:32.180+0000 7f24edc00640 1 --2- 172.21.15.90:0/4174740252 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f24e810e330 0x7f24e819eec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:51:32.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:32.180+0000 7f24edc00640 1 --2- 172.21.15.90:0/4174740252 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f24e80747f0 0x7f24e819ab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:51:32.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:32.180+0000 7f24edc00640 1 --2- 172.21.15.90:0/4174740252 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f24e8073ed0 0x7f24e819a650 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:51:32.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:32.180+0000 7f24edc00640 1 -- 172.21.15.90:0/4174740252 >> 172.21.15.90:0/4174740252 conn(0x7f24e8078f00 msgr2=0x7f24e80fea60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:51:32.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:32.180+0000 7f24edc00640 1 -- 172.21.15.90:0/4174740252 shutdown_connections 2024-04-02T14:51:32.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:32.180+0000 7f24edc00640 1 -- 172.21.15.90:0/4174740252 wait complete. 2024-04-02T14:51:32.192 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-04-02T14:51:32.192 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T14:51:32.192 DEBUG:teuthology.orchestra.run.smithi090:> sudo umount /tmp/cephfs 2024-04-02T14:51:32.274 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:51:32 smithi090 ceph-mon[31674]: pgmap v522: 97 pgs: 97 active+clean; 3.9 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 7.1 KiB/s wr, 11 op/s 2024-04-02T14:51:32.274 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:51:32 smithi090 ceph-mon[31674]: from='client.? 172.21.15.90:0/1345876550' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-02T14:51:32.274 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:51:32 smithi090 ceph-mon[31674]: from='client.? 172.21.15.90:0/4174740252' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-02T14:51:32.428 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:51:32 smithi177 ceph-mon[36431]: pgmap v522: 97 pgs: 97 active+clean; 3.9 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 7.1 KiB/s wr, 11 op/s 2024-04-02T14:51:32.428 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:51:32 smithi177 ceph-mon[36431]: from='client.? 172.21.15.90:0/1345876550' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-02T14:51:32.428 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:51:32 smithi177 ceph-mon[36431]: from='client.? 172.21.15.90:0/4174740252' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-02T14:51:32.583 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi090.front.sepia.ceph.com 2024-04-02T14:51:32.583 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T14:51:32.583 DEBUG:teuthology.orchestra.run.smithi090:> (cd /home/ubuntu/cephtest && exec rmdir -- /tmp/cephfs) 2024-04-02T14:51:32.601 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:51:32 smithi092 ceph-mon[36497]: pgmap v522: 97 pgs: 97 active+clean; 3.9 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 7.1 KiB/s wr, 11 op/s 2024-04-02T14:51:32.601 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:51:32 smithi092 ceph-mon[36497]: from='client.? 172.21.15.90:0/1345876550' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-02T14:51:32.601 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:51:32 smithi092 ceph-mon[36497]: from='client.? 172.21.15.90:0/4174740252' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-02T14:51:32.661 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T14:51:32.661 DEBUG:teuthology.orchestra.run.smithi090:> ip netns list 2024-04-02T14:51:32.734 INFO:teuthology.orchestra.run.smithi090.stdout:ceph-ns--tmp-cephfs (id: 0) 2024-04-02T14:51:32.735 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T14:51:32.735 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip netns delete ceph-ns--tmp-cephfs 2024-04-02T14:51:32.814 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T14:51:32.815 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip link delete ceph-brx 2024-04-02T14:51:32.929 INFO:tasks.kclient:Unmounting kernel clients... 2024-04-02T14:51:32.929 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-04-02T14:51:32.929 DEBUG:teuthology.orchestra.run.smithi090:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-02T14:51:32.968 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053414,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498340k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249168k,nr_inodes=812292,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/sda1 /var/lib/containers/storage/overlay ext4 rw,seclabel,relatime 0 0 overlay /var/lib/containers/storage/overlay/ab91d77d325f14ee55344a833d76897f744555ee0454255a0656bf5c690d7ed5/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/ZZK3FN43RUUW7X45KOCINWXQGY:/var/lib/containers/storage/overlay/l/VUEIBWGEL32L7ZVDV3ZWKSHPG3,upperdir=/var/lib/containers/storage/overlay/ab91d77d325f14ee55344a833d76897f744555ee0454255a0656bf5c690d7ed5/diff,workdir=/var/lib/containers/storage/overlay/ab91d77d325f14ee55344a833d76897f744555ee0454255a0656bf5c690d7ed5/work,redirect_dir=on,uuid=on,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/91bbfdc327cd8f1feb1933b3344608b38362b4529edf84c68aa5395b0f8e06f6/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/ZZK3FN43RUUW7X45KOCINWXQGY:/var/lib/containers/storage/overlay/l/VUEIBWGEL32L7ZVDV3ZWKSHPG3,upperdir=/var/lib/containers/storage/overlay/91bbfdc327cd8f1feb1933b3344608b38362b4529edf84c68aa5395b0f8e06f6/diff,workdir=/var/lib/containers/storage/overlay/91bbfdc327cd8f1feb1933b3344608b38362b4529edf84c68aa5395b0f8e06f6/work,redirect_dir=on,uuid=on,metacopy=on,volatile 0 0 tmpfs /run/user/0 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249168k,nr_inodes=812292,mode=700 0 0 binfmt_misc /proc/sys/fs/binfmt_misc binfmt_misc rw,nosuid,nodev,noexec,relatime 0 0 overlay /var/lib/containers/storage/overlay/fbb079a948e3898f52bbc1ceb5674fcd0424e6076f22a111b73cffc9fdb668f7/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/ZZK3FN43RUUW7X45KOCINWXQGY:/var/lib/containers/storage/overlay/l/VUEIBWGEL32L7ZVDV3ZWKSHPG3,upperdir=/var/lib/containers/storage/overlay/fbb079a948e3898f52bbc1ceb5674fcd0424e6076f22a111b73cffc9fdb668f7/diff,workdir=/var/lib/containers/storage/overlay/fbb079a948e3898f52bbc1ceb5674fcd0424e6076f22a111b73cffc9fdb668f7/work,redirect_dir=on,uuid=on,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/b379da3c294ad5f6fa09d3cabdfbabd48d748029af5e39e3cecbac55d75b25f0/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/ZZK3FN43RUUW7X45KOCINWXQGY:/var/lib/containers/storage/overlay/l/VUEIBWGEL32L7ZVDV3ZWKSHPG3,upperdir=/var/lib/containers/storage/overlay/b379da3c294ad5f6fa09d3cabdfbabd48d748029af5e39e3cecbac55d75b25f0/diff,workdir=/var/lib/containers/storage/overlay/b379da3c294ad5f6fa09d3cabdfbabd48d748029af5e39e3cecbac55d75b25f0/work,redirect_dir=on,uuid=on,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/d77c9540230e9666dcd56874c5a951eee164e36f202681825d0c575e9845fe07/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/ZZK3FN43RUUW7X45KOCINWXQGY:/var/lib/containers/storage/overlay/l/VUEIBWGEL32L7ZVDV3ZWKSHPG3,upperdir=/var/lib/containers/storage/overlay/d77c9540230e9666dcd56874c5a951eee164e36f202681825d0c575e9845fe07/diff,workdir=/var/lib/containers/storage/overlay/d77c9540230e9666dcd56874c5a951eee164e36f202681825d0c575e9845fe07/work,redirect_dir=on,uuid=on,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/f0efbc0f8f23ff7a9fead10a1fe3da3fd363fc91a00854f2408a47c6aea696d3/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/ZZK3FN43RUUW7X45KOCINWXQGY:/var/lib/containers/storage/overlay/l/VUEIBWGEL32L7ZVDV3ZWKSHPG3,upperdir=/var/lib/containers/storage/overlay/f0efbc0f8f23ff7a9fead10a1fe3da3fd363fc91a00854f2408a47c6aea696d3/diff,workdir=/var/lib/containers/storage/overlay/f0efbc0f8f23ff7a9fead10a1fe3da3fd363fc91a00854f2408a47c6aea696d3/work,redirect_dir=on,uuid=on,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/c206b9e8556cc4adf4ff4422a8f5ac1a10545f0e4c18c0ff6fffcc1129b23775/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/ZZK3FN43RUUW7X45KOCINWXQGY:/var/lib/containers/storage/overlay/l/VUEIBWGEL32L7ZVDV3ZWKSHPG3,upperdir=/var/lib/containers/storage/overlay/c206b9e8556cc4adf4ff4422a8f5ac1a10545f0e4c18c0ff6fffcc1129b23775/diff,workdir=/var/lib/containers/storage/overlay/c206b9e8556cc4adf4ff4422a8f5ac1a10545f0e4c18c0ff6fffcc1129b23775/work,redirect_dir=on,uuid=on,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/9f4622c8481686308c8bdddd65673158947dc5abfe5adb84d4f30ee34cb3920e/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/ZZK3FN43RUUW7X45KOCINWXQGY:/var/lib/containers/storage/overlay/l/VUEIBWGEL32L7ZVDV3ZWKSHPG3,upperdir=/var/lib/containers/storage/overlay/9f4622c8481686308c8bdddd65673158947dc5abfe5adb84d4f30ee34cb3920e/diff,workdir=/var/lib/containers/storage/overlay/9f4622c8481686308c8bdddd65673158947dc5abfe5adb84d4f30ee34cb3920e/work,redirect_dir=on,uuid=on,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/cec6d267d3525930ff9e80af99610eb26a3204c318dd4f8eea78e6404d476ad3/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/ZZK3FN43RUUW7X45KOCINWXQGY:/var/lib/containers/storage/overlay/l/VUEIBWGEL32L7ZVDV3ZWKSHPG3,upperdir=/var/lib/containers/storage/overlay/cec6d267d3525930ff9e80af99610eb26a3204c318dd4f8eea78e6404d476ad3/diff,workdir=/var/lib/containers/storage/overlay/cec6d267d3525930ff9e80af99610eb26a3204c318dd4f8eea78e6404d476ad3/work,redirect_dir=on,uuid=on,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/d0a710ace8912bd2ed4310247630da4fb5f5fc39ed2db8e939c0bf5adcf56535/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/ZZK3FN43RUUW7X45KOCINWXQGY:/var/lib/containers/storage/overlay/l/VUEIBWGEL32L7ZVDV3ZWKSHPG3,upperdir=/var/lib/containers/storage/overlay/d0a710ace8912bd2ed4310247630da4fb5f5fc39ed2db8e939c0bf5adcf56535/diff,workdir=/var/lib/containers/storage/overlay/d0a710ace8912bd2ed4310247630da4fb5f5fc39ed2db8e939c0bf5adcf56535/work,redirect_dir=on,uuid=on,metacopy=on,volatile 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498340k,nr_inodes=819200,mode=755 0 0 2024-04-02T14:51:32.968 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T14:51:32.968 DEBUG:teuthology.orchestra.run.smithi090:> ip netns list 2024-04-02T14:51:33.055 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-02T14:51:33.055 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip link delete ceph-brx 2024-04-02T14:51:33.184 INFO:teuthology.orchestra.run.smithi090.stderr:Cannot find device "ceph-brx" 2024-04-02T14:51:33.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T14:51:33.189 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-04-02T14:51:33.199 INFO:tasks.cephadm:Teardown begin 2024-04-02T14:51:33.199 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-02T14:51:33.199 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T14:51:33.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:33.432+0000 7f3733c00640 1 -- 172.21.15.90:0/4074220839 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f372c103f00 msgr2=0x7f372c1042e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:51:33.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:33.432+0000 7f3733c00640 1 --2- 172.21.15.90:0/4074220839 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f372c103f00 0x7f372c1042e0 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f3720009900 tx=0x7f372002bc30 comp rx=0 tx=0).stop 2024-04-02T14:51:33.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:33.432+0000 7f3733c00640 1 -- 172.21.15.90:0/4074220839 shutdown_connections 2024-04-02T14:51:33.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:33.432+0000 7f3733c00640 1 --2- 172.21.15.90:0/4074220839 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f372c108fd0 0x7f372c110ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:51:33.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:33.432+0000 7f3733c00640 1 --2- 172.21.15.90:0/4074220839 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f372c1048b0 0x7f372c1088f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:51:33.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:33.432+0000 7f3733c00640 1 --2- 172.21.15.90:0/4074220839 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f372c103f00 0x7f372c1042e0 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:51:33.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:33.432+0000 7f3733c00640 1 -- 172.21.15.90:0/4074220839 >> 172.21.15.90:0/4074220839 conn(0x7f372c0ff760 msgr2=0x7f372c101b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:51:33.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:33.432+0000 7f3733c00640 1 -- 172.21.15.90:0/4074220839 shutdown_connections 2024-04-02T14:51:33.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:33.432+0000 7f3733c00640 1 -- 172.21.15.90:0/4074220839 wait complete. 2024-04-02T14:51:33.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:33.433+0000 7f3733c00640 1 Processor -- start 2024-04-02T14:51:33.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:33.433+0000 7f3733c00640 1 -- start start 2024-04-02T14:51:33.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:33.433+0000 7f3733c00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f372c103f00 0x7f372c19a5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:51:33.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:33.433+0000 7f3733c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f372c108fd0 0x7f372c19ab00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:51:33.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:33.433+0000 7f3733c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f372c19ee80 0x7f372c19f330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:51:33.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:33.433+0000 7f3733c00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f372c114080 con 0x7f372c19ee80 2024-04-02T14:51:33.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:33.433+0000 7f3733c00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f372c113f00 con 0x7f372c103f00 2024-04-02T14:51:33.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:33.433+0000 7f3733c00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f372c114200 con 0x7f372c108fd0 2024-04-02T14:51:33.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:33.433+0000 7f372be00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f372c108fd0 0x7f372c19ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:51:33.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:33.433+0000 7f3731200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f372c103f00 0x7f372c19a5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:51:33.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:33.434+0000 7f372be00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f372c108fd0 0x7f372c19ab00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:51804/0 (socket says 172.21.15.90:51804) 2024-04-02T14:51:33.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:33.434+0000 7f372be00640 1 -- 172.21.15.90:0/360431697 learned_addr learned my addr 172.21.15.90:0/360431697 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:51:33.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:33.434+0000 7f3731c00640 1 --2- 172.21.15.90:0/360431697 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f372c19ee80 0x7f372c19f330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:51:33.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:33.434+0000 7f3731200640 1 -- 172.21.15.90:0/360431697 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f372c108fd0 msgr2=0x7f372c19ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:51:33.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:33.434+0000 7f3731200640 1 --2- 172.21.15.90:0/360431697 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f372c108fd0 0x7f372c19ab00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:51:33.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:33.434+0000 7f3731200640 1 -- 172.21.15.90:0/360431697 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f372c19ee80 msgr2=0x7f372c19f330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:51:33.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:33.434+0000 7f3731200640 1 --2- 172.21.15.90:0/360431697 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f372c19ee80 0x7f372c19f330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:51:33.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:33.434+0000 7f3731200640 1 -- 172.21.15.90:0/360431697 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f372c19f970 con 0x7f372c103f00 2024-04-02T14:51:33.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:33.434+0000 7f3731c00640 1 --2- 172.21.15.90:0/360431697 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f372c19ee80 0x7f372c19f330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:51:33.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:33.434+0000 7f3731200640 1 --2- 172.21.15.90:0/360431697 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f372c103f00 0x7f372c19a5c0 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7f3720036040 tx=0x7f372002cb20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:51:33.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:33.434+0000 7f3729600640 1 -- 172.21.15.90:0/360431697 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3720041070 con 0x7f372c103f00 2024-04-02T14:51:33.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:33.435+0000 7f3729600640 1 -- 172.21.15.90:0/360431697 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3720004450 con 0x7f372c103f00 2024-04-02T14:51:33.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:33.435+0000 7f3733c00640 1 -- 172.21.15.90:0/360431697 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f372c19fc00 con 0x7f372c103f00 2024-04-02T14:51:33.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:33.435+0000 7f3729600640 1 -- 172.21.15.90:0/360431697 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3720003d90 con 0x7f372c103f00 2024-04-02T14:51:33.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:33.435+0000 7f3733c00640 1 -- 172.21.15.90:0/360431697 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f372c1a73c0 con 0x7f372c103f00 2024-04-02T14:51:33.429 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:33.436+0000 7f3733c00640 1 -- 172.21.15.90:0/360431697 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f372c068d40 con 0x7f372c103f00 2024-04-02T14:51:33.429 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:33.438+0000 7f3729600640 1 -- 172.21.15.90:0/360431697 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f3720044070 con 0x7f372c103f00 2024-04-02T14:51:33.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:33.439+0000 7f3729600640 1 --2- 172.21.15.90:0/360431697 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f37140a3d20 0x7f37140a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:51:33.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:33.439+0000 7f3729600640 1 -- 172.21.15.90:0/360431697 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 8550+0+0 (secure 0 0 0) 0x7f37200f23e0 con 0x7f372c103f00 2024-04-02T14:51:33.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:33.439+0000 7f372be00640 1 --2- 172.21.15.90:0/360431697 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f37140a3d20 0x7f37140a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:51:33.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:33.440+0000 7f372be00640 1 --2- 172.21.15.90:0/360431697 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f37140a3d20 0x7f37140a61e0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f37180095f0 tx=0x7f3718005f90 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:51:33.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:33.441+0000 7f3729600640 1 -- 172.21.15.90:0/360431697 <== mon.2 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7f37200beb70 con 0x7f372c103f00 2024-04-02T14:51:33.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:33.679+0000 7f3733c00640 1 -- 172.21.15.90:0/360431697 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f372c19b910 con 0x7f372c103f00 2024-04-02T14:51:33.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:33.680+0000 7f3729600640 1 -- 172.21.15.90:0/360431697 <== mon.2 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11927 (secure 0 0 0) 0x7f37200be470 con 0x7f372c103f00 2024-04-02T14:51:33.672 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-02T14:51:33.672 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14412,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6837/2852160309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6836","nonce":2852160309},{"type":"v1","addr":"172.21.15.90:6837","nonce":2852160309}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14478,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6841/1919376858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6840","nonce":1919376858},{"type":"v1","addr":"172.21.15.90:6841","nonce":1919376858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-02T14:32:22.319083+0000","modified":"2024-04-02T14:34:52.266152+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24590,"mds_1":24560,"mds_2":24547,"mds_3":24539,"mds_4":24523},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24454":{"gid":24454,"name":"l","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6833/1564319431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":1564319431},{"type":"v1","addr":"172.21.15.177:6833","nonce":1564319431}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.92:6833/1570337372","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":1570337372},{"type":"v1","addr":"172.21.15.92:6833","nonce":1570337372}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"j","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.90:6835/1277124891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6834","nonce":1277124891},{"type":"v1","addr":"172.21.15.90:6835","nonce":1277124891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.92:6835/860721164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":860721164},{"type":"v1","addr":"172.21.15.92:6835","nonce":860721164}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24511":{"gid":24511,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6835/3177244988","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":3177244988},{"type":"v1","addr":"172.21.15.177:6835","nonce":3177244988}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"c","rank":4,"incarnation":25,"state":"up:active","state_seq":9,"addr":"172.21.15.177:6837/4223414858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":4223414858},{"type":"v1","addr":"172.21.15.177:6837","nonce":4223414858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24539":{"gid":24539,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":8,"addr":"172.21.15.92:6837/1270635939","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":1270635939},{"type":"v1","addr":"172.21.15.92:6837","nonce":1270635939}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24547":{"gid":24547,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":7,"addr":"172.21.15.177:6839/664895663","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":664895663},{"type":"v1","addr":"172.21.15.177:6839","nonce":664895663}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"g","rank":1,"incarnation":19,"state":"up:active","state_seq":5,"addr":"172.21.15.90:6839/1498915870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6838","nonce":1498915870},{"type":"v1","addr":"172.21.15.90:6839","nonce":1498915870}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24590":{"gid":24590,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.92:6839/1054888219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":1054888219},{"type":"v1","addr":"172.21.15.92:6839","nonce":1054888219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24590,"qdb_cluster":[24547,24590,24523,24539,24560]},"id":1}]} 2024-04-02T14:51:33.673 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 27 2024-04-02T14:51:33.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:33.682+0000 7f3733c00640 1 -- 172.21.15.90:0/360431697 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f37140a3d20 msgr2=0x7f37140a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:51:33.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:33.682+0000 7f3733c00640 1 --2- 172.21.15.90:0/360431697 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f37140a3d20 0x7f37140a61e0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f37180095f0 tx=0x7f3718005f90 comp rx=0 tx=0).stop 2024-04-02T14:51:33.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:33.682+0000 7f3733c00640 1 -- 172.21.15.90:0/360431697 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f372c103f00 msgr2=0x7f372c19a5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:51:33.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:33.682+0000 7f3733c00640 1 --2- 172.21.15.90:0/360431697 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f372c103f00 0x7f372c19a5c0 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7f3720036040 tx=0x7f372002cb20 comp rx=0 tx=0).stop 2024-04-02T14:51:33.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:33.682+0000 7f3733c00640 1 -- 172.21.15.90:0/360431697 shutdown_connections 2024-04-02T14:51:33.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:33.682+0000 7f3733c00640 1 --2- 172.21.15.90:0/360431697 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f37140a3d20 0x7f37140a61e0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:51:33.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:33.682+0000 7f3733c00640 1 --2- 172.21.15.90:0/360431697 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f372c19ee80 0x7f372c19f330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:51:33.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:33.682+0000 7f3733c00640 1 --2- 172.21.15.90:0/360431697 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f372c108fd0 0x7f372c19ab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:51:33.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:33.682+0000 7f3733c00640 1 --2- 172.21.15.90:0/360431697 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f372c103f00 0x7f372c19a5c0 unknown :-1 s=CLOSED pgs=1050 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:51:33.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:33.682+0000 7f3733c00640 1 -- 172.21.15.90:0/360431697 >> 172.21.15.90:0/360431697 conn(0x7f372c0ff760 msgr2=0x7f372c100aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:51:33.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:33.683+0000 7f3733c00640 1 -- 172.21.15.90:0/360431697 shutdown_connections 2024-04-02T14:51:33.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:33.683+0000 7f3733c00640 1 -- 172.21.15.90:0/360431697 wait complete. 2024-04-02T14:51:33.694 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-02T14:51:33.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:33.893+0000 7fe6c3800640 1 -- 172.21.15.90:0/1360408791 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe6bc102d30 msgr2=0x7fe6bc103110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:51:33.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:33.893+0000 7fe6c3800640 1 --2- 172.21.15.90:0/1360408791 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe6bc102d30 0x7fe6bc103110 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fe6b0009900 tx=0x7fe6b002bcd0 comp rx=0 tx=0).stop 2024-04-02T14:51:33.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:33.893+0000 7fe6c3800640 1 -- 172.21.15.90:0/1360408791 shutdown_connections 2024-04-02T14:51:33.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:33.893+0000 7fe6c3800640 1 --2- 172.21.15.90:0/1360408791 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe6bc10c100 0x7fe6bc10e530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:51:33.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:33.893+0000 7fe6c3800640 1 --2- 172.21.15.90:0/1360408791 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe6bc103650 0x7fe6bc10bb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:51:33.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:33.893+0000 7fe6c3800640 1 --2- 172.21.15.90:0/1360408791 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe6bc102d30 0x7fe6bc103110 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:51:33.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:33.893+0000 7fe6c3800640 1 -- 172.21.15.90:0/1360408791 >> 172.21.15.90:0/1360408791 conn(0x7fe6bc076d80 msgr2=0x7fe6bc077190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:51:33.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:33.893+0000 7fe6c3800640 1 -- 172.21.15.90:0/1360408791 shutdown_connections 2024-04-02T14:51:33.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:33.894+0000 7fe6c3800640 1 -- 172.21.15.90:0/1360408791 wait complete. 2024-04-02T14:51:33.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:33.895+0000 7fe6c3800640 1 Processor -- start 2024-04-02T14:51:33.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:33.895+0000 7fe6c3800640 1 -- start start 2024-04-02T14:51:33.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:33.895+0000 7fe6c3800640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe6bc102d30 0x7fe6bc19a590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:51:33.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:33.895+0000 7fe6c3800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe6bc103650 0x7fe6bc19aad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:51:33.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:33.895+0000 7fe6c3800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe6bc10c100 0x7fe6bc19ee50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:51:33.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:33.895+0000 7fe6c3800640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fe6bc111e90 con 0x7fe6bc103650 2024-04-02T14:51:33.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:33.895+0000 7fe6c3800640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7fe6bc111d10 con 0x7fe6bc102d30 2024-04-02T14:51:33.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:33.895+0000 7fe6c3800640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fe6bc112010 con 0x7fe6bc10c100 2024-04-02T14:51:33.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:33.895+0000 7fe6c0e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe6bc102d30 0x7fe6bc19a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:51:33.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:33.895+0000 7fe6c0e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe6bc102d30 0x7fe6bc19a590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:43298/0 (socket says 172.21.15.90:43298) 2024-04-02T14:51:33.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:33.895+0000 7fe6c1800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe6bc10c100 0x7fe6bc19ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:51:33.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:33.895+0000 7fe6c0e00640 1 -- 172.21.15.90:0/3379483538 learned_addr learned my addr 172.21.15.90:0/3379483538 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:51:33.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:33.896+0000 7fe6bbe00640 1 --2- 172.21.15.90:0/3379483538 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe6bc103650 0x7fe6bc19aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:51:33.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:33.896+0000 7fe6c0e00640 1 -- 172.21.15.90:0/3379483538 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe6bc10c100 msgr2=0x7fe6bc19ee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:51:33.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:33.896+0000 7fe6c0e00640 1 --2- 172.21.15.90:0/3379483538 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe6bc10c100 0x7fe6bc19ee50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:51:33.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:33.896+0000 7fe6c0e00640 1 -- 172.21.15.90:0/3379483538 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe6bc103650 msgr2=0x7fe6bc19aad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:51:33.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:33.896+0000 7fe6c0e00640 1 --2- 172.21.15.90:0/3379483538 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe6bc103650 0x7fe6bc19aad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:51:33.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:33.896+0000 7fe6c0e00640 1 -- 172.21.15.90:0/3379483538 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6bc19f580 con 0x7fe6bc102d30 2024-04-02T14:51:33.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:33.896+0000 7fe6c0e00640 1 --2- 172.21.15.90:0/3379483538 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe6bc102d30 0x7fe6bc19a590 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7fe6b0005940 tx=0x7fe6b002cbd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:51:33.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:33.897+0000 7fe6b9600640 1 -- 172.21.15.90:0/3379483538 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6b0041070 con 0x7fe6bc102d30 2024-04-02T14:51:33.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:33.897+0000 7fe6b9600640 1 -- 172.21.15.90:0/3379483538 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe6b0004450 con 0x7fe6bc102d30 2024-04-02T14:51:33.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:33.897+0000 7fe6b9600640 1 -- 172.21.15.90:0/3379483538 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6b0003d80 con 0x7fe6bc102d30 2024-04-02T14:51:33.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:33.897+0000 7fe6c3800640 1 -- 172.21.15.90:0/3379483538 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe6bc19f810 con 0x7fe6bc102d30 2024-04-02T14:51:33.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:33.897+0000 7fe6c3800640 1 -- 172.21.15.90:0/3379483538 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6bc1a7100 con 0x7fe6bc102d30 2024-04-02T14:51:33.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:33.898+0000 7fe6c3800640 1 -- 172.21.15.90:0/3379483538 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe6bc069980 con 0x7fe6bc102d30 2024-04-02T14:51:33.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:33.900+0000 7fe6b9600640 1 -- 172.21.15.90:0/3379483538 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fe6b0044070 con 0x7fe6bc102d30 2024-04-02T14:51:33.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:33.901+0000 7fe6b9600640 1 --2- 172.21.15.90:0/3379483538 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fe6a40a3da0 0x7fe6a40a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:51:33.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:33.901+0000 7fe6b9600640 1 -- 172.21.15.90:0/3379483538 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 8550+0+0 (secure 0 0 0) 0x7fe6b00f21d0 con 0x7fe6bc102d30 2024-04-02T14:51:33.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:33.901+0000 7fe6bbe00640 1 --2- 172.21.15.90:0/3379483538 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fe6a40a3da0 0x7fe6a40a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:51:33.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:33.902+0000 7fe6bbe00640 1 --2- 172.21.15.90:0/3379483538 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fe6a40a3da0 0x7fe6a40a6260 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fe6bc1029a0 tx=0x7fe6a8005f90 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:51:33.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:33.905+0000 7fe6b9600640 1 -- 172.21.15.90:0/3379483538 <== mon.2 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7fe6b00be2b0 con 0x7fe6bc102d30 2024-04-02T14:51:34.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:34.192+0000 7fe6c3800640 1 -- 172.21.15.90:0/3379483538 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe6bc06a950 con 0x7fe6bc102d30 2024-04-02T14:51:34.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:34.193+0000 7fe6b9600640 1 -- 172.21.15.90:0/3379483538 <== mon.2 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11927 (secure 0 0 0) 0x7fe6b00b8e20 con 0x7fe6bc102d30 2024-04-02T14:51:34.185 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-02T14:51:34.185 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14412,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6837/2852160309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6836","nonce":2852160309},{"type":"v1","addr":"172.21.15.90:6837","nonce":2852160309}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14478,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6841/1919376858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6840","nonce":1919376858},{"type":"v1","addr":"172.21.15.90:6841","nonce":1919376858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-02T14:32:22.319083+0000","modified":"2024-04-02T14:34:52.266152+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24590,"mds_1":24560,"mds_2":24547,"mds_3":24539,"mds_4":24523},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24454":{"gid":24454,"name":"l","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6833/1564319431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":1564319431},{"type":"v1","addr":"172.21.15.177:6833","nonce":1564319431}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.92:6833/1570337372","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":1570337372},{"type":"v1","addr":"172.21.15.92:6833","nonce":1570337372}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"j","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.90:6835/1277124891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6834","nonce":1277124891},{"type":"v1","addr":"172.21.15.90:6835","nonce":1277124891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.92:6835/860721164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":860721164},{"type":"v1","addr":"172.21.15.92:6835","nonce":860721164}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24511":{"gid":24511,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6835/3177244988","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":3177244988},{"type":"v1","addr":"172.21.15.177:6835","nonce":3177244988}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"c","rank":4,"incarnation":25,"state":"up:active","state_seq":9,"addr":"172.21.15.177:6837/4223414858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":4223414858},{"type":"v1","addr":"172.21.15.177:6837","nonce":4223414858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24539":{"gid":24539,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":8,"addr":"172.21.15.92:6837/1270635939","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":1270635939},{"type":"v1","addr":"172.21.15.92:6837","nonce":1270635939}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24547":{"gid":24547,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":7,"addr":"172.21.15.177:6839/664895663","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":664895663},{"type":"v1","addr":"172.21.15.177:6839","nonce":664895663}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"g","rank":1,"incarnation":19,"state":"up:active","state_seq":5,"addr":"172.21.15.90:6839/1498915870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6838","nonce":1498915870},{"type":"v1","addr":"172.21.15.90:6839","nonce":1498915870}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24590":{"gid":24590,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.92:6839/1054888219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":1054888219},{"type":"v1","addr":"172.21.15.92:6839","nonce":1054888219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24590,"qdb_cluster":[24547,24590,24523,24539,24560]},"id":1}]} 2024-04-02T14:51:34.186 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 27 2024-04-02T14:51:34.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:34.195+0000 7fe6c3800640 1 -- 172.21.15.90:0/3379483538 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fe6a40a3da0 msgr2=0x7fe6a40a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:51:34.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:34.195+0000 7fe6c3800640 1 --2- 172.21.15.90:0/3379483538 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fe6a40a3da0 0x7fe6a40a6260 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fe6bc1029a0 tx=0x7fe6a8005f90 comp rx=0 tx=0).stop 2024-04-02T14:51:34.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:34.195+0000 7fe6c3800640 1 -- 172.21.15.90:0/3379483538 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe6bc102d30 msgr2=0x7fe6bc19a590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:51:34.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:34.195+0000 7fe6c3800640 1 --2- 172.21.15.90:0/3379483538 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe6bc102d30 0x7fe6bc19a590 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7fe6b0005940 tx=0x7fe6b002cbd0 comp rx=0 tx=0).stop 2024-04-02T14:51:34.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:34.196+0000 7fe6c3800640 1 -- 172.21.15.90:0/3379483538 shutdown_connections 2024-04-02T14:51:34.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:34.196+0000 7fe6c3800640 1 --2- 172.21.15.90:0/3379483538 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7fe6a40a3da0 0x7fe6a40a6260 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:51:34.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:34.196+0000 7fe6c3800640 1 --2- 172.21.15.90:0/3379483538 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe6bc10c100 0x7fe6bc19ee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:51:34.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:34.196+0000 7fe6c3800640 1 --2- 172.21.15.90:0/3379483538 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe6bc103650 0x7fe6bc19aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:51:34.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:34.196+0000 7fe6c3800640 1 --2- 172.21.15.90:0/3379483538 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe6bc102d30 0x7fe6bc19a590 unknown :-1 s=CLOSED pgs=1051 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:51:34.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:34.196+0000 7fe6c3800640 1 -- 172.21.15.90:0/3379483538 >> 172.21.15.90:0/3379483538 conn(0x7fe6bc076d80 msgr2=0x7fe6bc0fd710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:51:34.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:34.196+0000 7fe6c3800640 1 -- 172.21.15.90:0/3379483538 shutdown_connections 2024-04-02T14:51:34.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:34.196+0000 7fe6c3800640 1 -- 172.21.15.90:0/3379483538 wait complete. 2024-04-02T14:51:34.212 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T14:51:34.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:34.419+0000 7faf39600640 1 -- 172.21.15.90:0/2697524171 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7faf3406a0d0 msgr2=0x7faf3406a550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:51:34.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:34.419+0000 7faf39600640 1 --2- 172.21.15.90:0/2697524171 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7faf3406a0d0 0x7faf3406a550 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7faf1c009900 tx=0x7faf1c02bc30 comp rx=0 tx=0).stop 2024-04-02T14:51:34.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:34.419+0000 7faf39600640 1 -- 172.21.15.90:0/2697524171 shutdown_connections 2024-04-02T14:51:34.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:34.419+0000 7faf39600640 1 --2- 172.21.15.90:0/2697524171 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faf3406aa90 0x7faf3410e940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:51:34.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:34.419+0000 7faf39600640 1 --2- 172.21.15.90:0/2697524171 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7faf3406a0d0 0x7faf3406a550 unknown :-1 s=CLOSED pgs=1052 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:51:34.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:34.419+0000 7faf39600640 1 --2- 172.21.15.90:0/2697524171 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faf34107c50 0x7faf34108030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:51:34.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:34.419+0000 7faf39600640 1 -- 172.21.15.90:0/2697524171 >> 172.21.15.90:0/2697524171 conn(0x7faf340779e0 msgr2=0x7faf34077df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:51:34.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:34.419+0000 7faf39600640 1 -- 172.21.15.90:0/2697524171 shutdown_connections 2024-04-02T14:51:34.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:34.419+0000 7faf39600640 1 -- 172.21.15.90:0/2697524171 wait complete. 2024-04-02T14:51:34.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:34.420+0000 7faf39600640 1 Processor -- start 2024-04-02T14:51:34.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:34.420+0000 7faf39600640 1 -- start start 2024-04-02T14:51:34.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:34.420+0000 7faf39600640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faf3406a0d0 0x7faf34198350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:51:34.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:34.421+0000 7faf39600640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7faf3406aa90 0x7faf34198890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:51:34.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:34.421+0000 7faf39600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faf34107c50 0x7faf3419cc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:51:34.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:34.421+0000 7faf39600640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7faf34111ed0 con 0x7faf34107c50 2024-04-02T14:51:34.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:34.421+0000 7faf39600640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7faf34111d50 con 0x7faf3406aa90 2024-04-02T14:51:34.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:34.421+0000 7faf39600640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7faf34112050 con 0x7faf3406a0d0 2024-04-02T14:51:34.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:34.421+0000 7faf32a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faf3406a0d0 0x7faf34198350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:51:34.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:34.421+0000 7faf32000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7faf3406aa90 0x7faf34198890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:51:34.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:34.421+0000 7faf32a00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faf3406a0d0 0x7faf34198350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:48084/0 (socket says 172.21.15.90:48084) 2024-04-02T14:51:34.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:34.421+0000 7faf32a00640 1 -- 172.21.15.90:0/49961366 learned_addr learned my addr 172.21.15.90:0/49961366 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:51:34.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:34.421+0000 7faf33400640 1 --2- 172.21.15.90:0/49961366 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faf34107c50 0x7faf3419cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:51:34.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:34.421+0000 7faf32000640 1 -- 172.21.15.90:0/49961366 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faf3406a0d0 msgr2=0x7faf34198350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:51:34.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:34.421+0000 7faf32000640 1 --2- 172.21.15.90:0/49961366 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faf3406a0d0 0x7faf34198350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:51:34.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:34.421+0000 7faf32000640 1 -- 172.21.15.90:0/49961366 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faf34107c50 msgr2=0x7faf3419cc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:51:34.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:34.421+0000 7faf32000640 1 --2- 172.21.15.90:0/49961366 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faf34107c50 0x7faf3419cc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:51:34.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:34.421+0000 7faf32000640 1 -- 172.21.15.90:0/49961366 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf3419d340 con 0x7faf3406aa90 2024-04-02T14:51:34.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:34.421+0000 7faf32000640 1 --2- 172.21.15.90:0/49961366 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7faf3406aa90 0x7faf34198890 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7faf1c02c170 tx=0x7faf1c004360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:51:34.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:34.422+0000 7faf2b400640 1 -- 172.21.15.90:0/49961366 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf1c043020 con 0x7faf3406aa90 2024-04-02T14:51:34.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:34.422+0000 7faf2b400640 1 -- 172.21.15.90:0/49961366 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faf1c0054e0 con 0x7faf3406aa90 2024-04-02T14:51:34.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:34.422+0000 7faf2b400640 1 -- 172.21.15.90:0/49961366 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf1c003b80 con 0x7faf3406aa90 2024-04-02T14:51:34.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:34.422+0000 7faf39600640 1 -- 172.21.15.90:0/49961366 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faf3419d5d0 con 0x7faf3406aa90 2024-04-02T14:51:34.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:34.422+0000 7faf39600640 1 -- 172.21.15.90:0/49961366 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faf341a4ec0 con 0x7faf3406aa90 2024-04-02T14:51:34.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:34.423+0000 7faf39600640 1 -- 172.21.15.90:0/49961366 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faf341031d0 con 0x7faf3406aa90 2024-04-02T14:51:34.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:34.425+0000 7faf2b400640 1 -- 172.21.15.90:0/49961366 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7faf1c046020 con 0x7faf3406aa90 2024-04-02T14:51:34.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:34.426+0000 7faf2b400640 1 --2- 172.21.15.90:0/49961366 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7faf180a3da0 0x7faf180a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:51:34.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:34.426+0000 7faf2b400640 1 -- 172.21.15.90:0/49961366 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 8550+0+0 (secure 0 0 0) 0x7faf1c0f2670 con 0x7faf3406aa90 2024-04-02T14:51:34.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:34.426+0000 7faf32a00640 1 --2- 172.21.15.90:0/49961366 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7faf180a3da0 0x7faf180a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:51:34.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:34.427+0000 7faf32a00640 1 --2- 172.21.15.90:0/49961366 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7faf180a3da0 0x7faf180a6260 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7faf24009fd0 tx=0x7faf24009500 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:51:34.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:34.430+0000 7faf2b400640 1 -- 172.21.15.90:0/49961366 <== mon.2 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7faf1c04d020 con 0x7faf3406aa90 2024-04-02T14:51:34.542 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:51:34 smithi177 ceph-mon[36431]: pgmap v523: 97 pgs: 97 active+clean; 3.9 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 KiB/s rd, 4.0 KiB/s wr, 8 op/s 2024-04-02T14:51:34.542 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:51:34 smithi177 ceph-mon[36431]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T14:51:34.542 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:51:34 smithi177 ceph-mon[36431]: from='client.? 172.21.15.90:0/360431697' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-02T14:51:34.542 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:51:34 smithi177 ceph-mon[36431]: from='client.? 172.21.15.90:0/3379483538' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-02T14:51:34.601 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:51:34 smithi092 ceph-mon[36497]: pgmap v523: 97 pgs: 97 active+clean; 3.9 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 KiB/s rd, 4.0 KiB/s wr, 8 op/s 2024-04-02T14:51:34.602 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:51:34 smithi092 ceph-mon[36497]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T14:51:34.602 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:51:34 smithi092 ceph-mon[36497]: from='client.? 172.21.15.90:0/360431697' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-02T14:51:34.602 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:51:34 smithi092 ceph-mon[36497]: from='client.? 172.21.15.90:0/3379483538' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-02T14:51:34.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:34.628+0000 7faf39600640 1 -- 172.21.15.90:0/49961366 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7faf34109fe0 con 0x7faf3406aa90 2024-04-02T14:51:34.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:34.630+0000 7faf2b400640 1 -- 172.21.15.90:0/49961366 <== mon.2 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v111) ==== 74+0+23118 (secure 0 0 0) 0x7faf1c0beda0 con 0x7faf3406aa90 2024-04-02T14:51:34.623 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-02T14:51:34.623 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":111,"fsid":"6449448a-f0fc-11ee-b647-cb9ed24678a4","created":"2024-04-02T14:23:16.530201+0000","modified":"2024-04-02T14:51:20.219915+0000","last_up_change":"2024-04-02T14:31:23.819732+0000","last_in_change":"2024-04-02T14:31:02.824284+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T14:27:16.758622+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-02T14:32:20.016331+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-02T14:32:21.294367+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"110","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":27,"snap_epoch":110,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a1a6a856-ab44-44bd-bc5e-45c718a70cd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":3635087609},{"type":"v1","addr":"172.21.15.90:6803","nonce":3635087609}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":3635087609},{"type":"v1","addr":"172.21.15.90:6805","nonce":3635087609}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":3635087609},{"type":"v1","addr":"172.21.15.90:6809","nonce":3635087609}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":3635087609},{"type":"v1","addr":"172.21.15.90:6807","nonce":3635087609}]},"public_addr":"172.21.15.90:6803/3635087609","cluster_addr":"172.21.15.90:6805/3635087609","heartbeat_back_addr":"172.21.15.90:6809/3635087609","heartbeat_front_addr":"172.21.15.90:6807/3635087609","state":["exists","up"]},{"osd":1,"uuid":"124139cd-5793-463e-9607-9d9c10164183","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":1994536059},{"type":"v1","addr":"172.21.15.92:6801","nonce":1994536059}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":1994536059},{"type":"v1","addr":"172.21.15.92:6803","nonce":1994536059}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":1994536059},{"type":"v1","addr":"172.21.15.92:6807","nonce":1994536059}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":1994536059},{"type":"v1","addr":"172.21.15.92:6805","nonce":1994536059}]},"public_addr":"172.21.15.92:6801/1994536059","cluster_addr":"172.21.15.92:6803/1994536059","heartbeat_back_addr":"172.21.15.92:6807/1994536059","heartbeat_front_addr":"172.21.15.92:6805/1994536059","state":["exists","up"]},{"osd":2,"uuid":"a61fbef2-9dca-4875-a814-020fff0db0e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":1358472800},{"type":"v1","addr":"172.21.15.177:6801","nonce":1358472800}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":1358472800},{"type":"v1","addr":"172.21.15.177:6803","nonce":1358472800}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":1358472800},{"type":"v1","addr":"172.21.15.177:6807","nonce":1358472800}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":1358472800},{"type":"v1","addr":"172.21.15.177:6805","nonce":1358472800}]},"public_addr":"172.21.15.177:6801/1358472800","cluster_addr":"172.21.15.177:6803/1358472800","heartbeat_back_addr":"172.21.15.177:6807/1358472800","heartbeat_front_addr":"172.21.15.177:6805/1358472800","state":["exists","up"]},{"osd":3,"uuid":"b175a2f4-7859-4eb0-bc1f-fba8c2b92a2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":3114811216},{"type":"v1","addr":"172.21.15.90:6811","nonce":3114811216}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":3114811216},{"type":"v1","addr":"172.21.15.90:6813","nonce":3114811216}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":3114811216},{"type":"v1","addr":"172.21.15.90:6817","nonce":3114811216}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":3114811216},{"type":"v1","addr":"172.21.15.90:6815","nonce":3114811216}]},"public_addr":"172.21.15.90:6811/3114811216","cluster_addr":"172.21.15.90:6813/3114811216","heartbeat_back_addr":"172.21.15.90:6817/3114811216","heartbeat_front_addr":"172.21.15.90:6815/3114811216","state":["exists","up"]},{"osd":4,"uuid":"38cc68ea-c109-4d8d-b77d-1c6f9b56c875","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":2987208879},{"type":"v1","addr":"172.21.15.92:6809","nonce":2987208879}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6810","nonce":2987208879},{"type":"v1","addr":"172.21.15.92:6811","nonce":2987208879}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6814","nonce":2987208879},{"type":"v1","addr":"172.21.15.92:6815","nonce":2987208879}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":2987208879},{"type":"v1","addr":"172.21.15.92:6813","nonce":2987208879}]},"public_addr":"172.21.15.92:6809/2987208879","cluster_addr":"172.21.15.92:6811/2987208879","heartbeat_back_addr":"172.21.15.92:6815/2987208879","heartbeat_front_addr":"172.21.15.92:6813/2987208879","state":["exists","up"]},{"osd":5,"uuid":"2347232e-ac91-466f-8ac9-496f397679ba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":3486820237},{"type":"v1","addr":"172.21.15.177:6809","nonce":3486820237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":3486820237},{"type":"v1","addr":"172.21.15.177:6811","nonce":3486820237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":3486820237},{"type":"v1","addr":"172.21.15.177:6815","nonce":3486820237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":3486820237},{"type":"v1","addr":"172.21.15.177:6813","nonce":3486820237}]},"public_addr":"172.21.15.177:6809/3486820237","cluster_addr":"172.21.15.177:6811/3486820237","heartbeat_back_addr":"172.21.15.177:6815/3486820237","heartbeat_front_addr":"172.21.15.177:6813/3486820237","state":["exists","up"]},{"osd":6,"uuid":"aa72392f-0498-44db-a8d2-eccc89a64871","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":2579716443},{"type":"v1","addr":"172.21.15.90:6819","nonce":2579716443}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6820","nonce":2579716443},{"type":"v1","addr":"172.21.15.90:6821","nonce":2579716443}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":2579716443},{"type":"v1","addr":"172.21.15.90:6825","nonce":2579716443}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2579716443},{"type":"v1","addr":"172.21.15.90:6823","nonce":2579716443}]},"public_addr":"172.21.15.90:6819/2579716443","cluster_addr":"172.21.15.90:6821/2579716443","heartbeat_back_addr":"172.21.15.90:6825/2579716443","heartbeat_front_addr":"172.21.15.90:6823/2579716443","state":["exists","up"]},{"osd":7,"uuid":"931bd105-bc14-4a1e-bfe4-c7ad58fa4b73","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":3371649139},{"type":"v1","addr":"172.21.15.92:6817","nonce":3371649139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":3371649139},{"type":"v1","addr":"172.21.15.92:6819","nonce":3371649139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":3371649139},{"type":"v1","addr":"172.21.15.92:6823","nonce":3371649139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6820","nonce":3371649139},{"type":"v1","addr":"172.21.15.92:6821","nonce":3371649139}]},"public_addr":"172.21.15.92:6817/3371649139","cluster_addr":"172.21.15.92:6819/3371649139","heartbeat_back_addr":"172.21.15.92:6823/3371649139","heartbeat_front_addr":"172.21.15.92:6821/3371649139","state":["exists","up"]},{"osd":8,"uuid":"82624ec2-b382-4886-9f40-7f5f376de3ce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":1459053034},{"type":"v1","addr":"172.21.15.177:6817","nonce":1459053034}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":1459053034},{"type":"v1","addr":"172.21.15.177:6819","nonce":1459053034}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":1459053034},{"type":"v1","addr":"172.21.15.177:6823","nonce":1459053034}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":1459053034},{"type":"v1","addr":"172.21.15.177:6821","nonce":1459053034}]},"public_addr":"172.21.15.177:6817/1459053034","cluster_addr":"172.21.15.177:6819/1459053034","heartbeat_back_addr":"172.21.15.177:6823/1459053034","heartbeat_front_addr":"172.21.15.177:6821/1459053034","state":["exists","up"]},{"osd":9,"uuid":"fbe36e48-3468-42b2-8094-e16552f3cb16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":2592604332},{"type":"v1","addr":"172.21.15.90:6827","nonce":2592604332}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":2592604332},{"type":"v1","addr":"172.21.15.90:6829","nonce":2592604332}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6832","nonce":2592604332},{"type":"v1","addr":"172.21.15.90:6833","nonce":2592604332}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":2592604332},{"type":"v1","addr":"172.21.15.90:6831","nonce":2592604332}]},"public_addr":"172.21.15.90:6827/2592604332","cluster_addr":"172.21.15.90:6829/2592604332","heartbeat_back_addr":"172.21.15.90:6833/2592604332","heartbeat_front_addr":"172.21.15.90:6831/2592604332","state":["exists","up"]},{"osd":10,"uuid":"9240df75-e7e2-47f1-8aae-534b70b3c210","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":1594153142},{"type":"v1","addr":"172.21.15.92:6825","nonce":1594153142}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6826","nonce":1594153142},{"type":"v1","addr":"172.21.15.92:6827","nonce":1594153142}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":1594153142},{"type":"v1","addr":"172.21.15.92:6831","nonce":1594153142}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":1594153142},{"type":"v1","addr":"172.21.15.92:6829","nonce":1594153142}]},"public_addr":"172.21.15.92:6825/1594153142","cluster_addr":"172.21.15.92:6827/1594153142","heartbeat_back_addr":"172.21.15.92:6831/1594153142","heartbeat_front_addr":"172.21.15.92:6829/1594153142","state":["exists","up"]},{"osd":11,"uuid":"83cba9fa-b745-4df1-adda-e86837d4b91a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":83,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1622582126},{"type":"v1","addr":"172.21.15.177:6825","nonce":1622582126}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":1622582126},{"type":"v1","addr":"172.21.15.177:6827","nonce":1622582126}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":1622582126},{"type":"v1","addr":"172.21.15.177:6831","nonce":1622582126}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1622582126},{"type":"v1","addr":"172.21.15.177:6829","nonce":1622582126}]},"public_addr":"172.21.15.177:6825/1622582126","cluster_addr":"172.21.15.177:6827/1622582126","heartbeat_back_addr":"172.21.15.177:6831/1622582126","heartbeat_front_addr":"172.21.15.177:6829/1622582126","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:26:24.290365+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:26:49.405988+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:27:13.639196+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:27:45.603746+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:28:11.019126+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:28:35.084163+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:29:08.064626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:29:33.266133+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:29:57.381816+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:30:30.286847+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:30:56.226279+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-02T14:31:20.809611+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":7,"to":11}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":8,"to":4}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.11","mappings":[{"from":4,"to":8},{"from":5,"to":9}]},{"pgid":"3.21","mappings":[{"from":5,"to":9}]},{"pgid":"3.23","mappings":[{"from":4,"to":7}]},{"pgid":"3.3a","mappings":[{"from":4,"to":6}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.90:0/2805994468":"2024-04-03T14:34:52.235215+0000","172.21.15.90:0/1431307002":"2024-04-03T14:34:52.235215+0000","172.21.15.90:6801/4266054817":"2024-04-03T14:34:52.235215+0000","172.21.15.90:0/314641601":"2024-04-03T14:24:34.595561+0000","172.21.15.90:0/1400895868":"2024-04-03T14:24:34.595561+0000","172.21.15.90:6801/129436436":"2024-04-03T14:24:34.595561+0000","172.21.15.90:0/1335673697":"2024-04-03T14:34:52.235215+0000","172.21.15.90:6801/2372398598":"2024-04-03T14:24:00.362147+0000","172.21.15.90:0/278569338":"2024-04-03T14:24:34.595561+0000","172.21.15.90:0/2321464594":"2024-04-03T14:24:00.362147+0000","172.21.15.90:6800/4266054817":"2024-04-03T14:34:52.235215+0000","172.21.15.90:6800/2372398598":"2024-04-03T14:24:00.362147+0000","172.21.15.90:0/2962656672":"2024-04-03T14:24:00.362147+0000","172.21.15.90:0/126693610":"2024-04-03T14:24:00.362147+0000","172.21.15.90:0/1757385857":"2024-04-03T14:34:52.235215+0000","172.21.15.90:6800/129436436":"2024-04-03T14:24:34.595561+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":14,"length":1},{"begin":27,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T14:51:34.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:34.633+0000 7faf39600640 1 -- 172.21.15.90:0/49961366 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7faf180a3da0 msgr2=0x7faf180a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:51:34.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:34.633+0000 7faf39600640 1 --2- 172.21.15.90:0/49961366 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7faf180a3da0 0x7faf180a6260 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7faf24009fd0 tx=0x7faf24009500 comp rx=0 tx=0).stop 2024-04-02T14:51:34.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:34.633+0000 7faf39600640 1 -- 172.21.15.90:0/49961366 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7faf3406aa90 msgr2=0x7faf34198890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:51:34.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:34.633+0000 7faf39600640 1 --2- 172.21.15.90:0/49961366 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7faf3406aa90 0x7faf34198890 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7faf1c02c170 tx=0x7faf1c004360 comp rx=0 tx=0).stop 2024-04-02T14:51:34.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:34.633+0000 7faf39600640 1 -- 172.21.15.90:0/49961366 shutdown_connections 2024-04-02T14:51:34.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:34.633+0000 7faf39600640 1 --2- 172.21.15.90:0/49961366 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7faf180a3da0 0x7faf180a6260 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:51:34.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:34.633+0000 7faf39600640 1 --2- 172.21.15.90:0/49961366 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faf34107c50 0x7faf3419cc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:51:34.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:34.633+0000 7faf39600640 1 --2- 172.21.15.90:0/49961366 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7faf3406aa90 0x7faf34198890 unknown :-1 s=CLOSED pgs=1053 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:51:34.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:34.633+0000 7faf39600640 1 --2- 172.21.15.90:0/49961366 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faf3406a0d0 0x7faf34198350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:51:34.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:34.633+0000 7faf39600640 1 -- 172.21.15.90:0/49961366 >> 172.21.15.90:0/49961366 conn(0x7faf340779e0 msgr2=0x7faf340ff820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:51:34.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:34.633+0000 7faf39600640 1 -- 172.21.15.90:0/49961366 shutdown_connections 2024-04-02T14:51:34.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:34.633+0000 7faf39600640 1 -- 172.21.15.90:0/49961366 wait complete. 2024-04-02T14:51:34.641 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:51:34 smithi090 ceph-mon[31674]: pgmap v523: 97 pgs: 97 active+clean; 3.9 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 KiB/s rd, 4.0 KiB/s wr, 8 op/s 2024-04-02T14:51:34.641 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:51:34 smithi090 ceph-mon[31674]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-02T14:51:34.641 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:51:34 smithi090 ceph-mon[31674]: from='client.? 172.21.15.90:0/360431697' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-02T14:51:34.641 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:51:34 smithi090 ceph-mon[31674]: from='client.? 172.21.15.90:0/3379483538' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-02T14:51:34.645 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-02T14:51:34.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:34.863+0000 7ffb91200640 1 -- 172.21.15.90:0/1792273879 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffb8c108c60 msgr2=0x7ffb8c10cc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:51:34.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:34.863+0000 7ffb91200640 1 --2- 172.21.15.90:0/1792273879 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffb8c108c60 0x7ffb8c10cc20 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7ffb78009900 tx=0x7ffb7802bc40 comp rx=0 tx=0).stop 2024-04-02T14:51:34.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:34.863+0000 7ffb91200640 1 -- 172.21.15.90:0/1792273879 shutdown_connections 2024-04-02T14:51:34.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:34.863+0000 7ffb91200640 1 --2- 172.21.15.90:0/1792273879 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffb8c10d300 0x7ffb8c114ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:51:34.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:34.863+0000 7ffb91200640 1 --2- 172.21.15.90:0/1792273879 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffb8c108c60 0x7ffb8c10cc20 unknown :-1 s=CLOSED pgs=857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:51:34.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:34.863+0000 7ffb91200640 1 --2- 172.21.15.90:0/1792273879 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ffb8c1082b0 0x7ffb8c108690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:51:34.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:34.863+0000 7ffb91200640 1 -- 172.21.15.90:0/1792273879 >> 172.21.15.90:0/1792273879 conn(0x7ffb8c077920 msgr2=0x7ffb8c106920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:51:34.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:34.863+0000 7ffb91200640 1 -- 172.21.15.90:0/1792273879 shutdown_connections 2024-04-02T14:51:34.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:34.863+0000 7ffb91200640 1 -- 172.21.15.90:0/1792273879 wait complete. 2024-04-02T14:51:34.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:34.864+0000 7ffb91200640 1 Processor -- start 2024-04-02T14:51:34.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:34.864+0000 7ffb91200640 1 -- start start 2024-04-02T14:51:34.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:34.864+0000 7ffb91200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ffb8c1082b0 0x7ffb8c1a46a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:51:34.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:34.864+0000 7ffb91200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffb8c10d300 0x7ffb8c1a4be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:51:34.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:34.864+0000 7ffb91200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffb8c19e760 0x7ffb8c19ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:51:34.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:34.864+0000 7ffb91200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7ffb8c1183e0 con 0x7ffb8c19e760 2024-04-02T14:51:34.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:34.864+0000 7ffb91200640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7ffb8c118260 con 0x7ffb8c1082b0 2024-04-02T14:51:34.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:34.864+0000 7ffb91200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7ffb8c118560 con 0x7ffb8c10d300 2024-04-02T14:51:34.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:34.865+0000 7ffb8aa00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffb8c10d300 0x7ffb8c1a4be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:51:34.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:34.865+0000 7ffb8aa00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffb8c10d300 0x7ffb8c1a4be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:48088/0 (socket says 172.21.15.90:48088) 2024-04-02T14:51:34.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:34.865+0000 7ffb8aa00640 1 -- 172.21.15.90:0/871700795 learned_addr learned my addr 172.21.15.90:0/871700795 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:51:34.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:34.865+0000 7ffb8b400640 1 --2- 172.21.15.90:0/871700795 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ffb8c1082b0 0x7ffb8c1a46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:51:34.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:34.865+0000 7ffb8be00640 1 --2- 172.21.15.90:0/871700795 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffb8c19e760 0x7ffb8c19ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:51:34.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:34.865+0000 7ffb8aa00640 1 -- 172.21.15.90:0/871700795 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ffb8c1082b0 msgr2=0x7ffb8c1a46a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:51:34.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:34.865+0000 7ffb8aa00640 1 --2- 172.21.15.90:0/871700795 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ffb8c1082b0 0x7ffb8c1a46a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:51:34.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:34.865+0000 7ffb8aa00640 1 -- 172.21.15.90:0/871700795 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffb8c19e760 msgr2=0x7ffb8c19ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:51:34.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:34.865+0000 7ffb8aa00640 1 --2- 172.21.15.90:0/871700795 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffb8c19e760 0x7ffb8c19ebe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:51:34.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:34.865+0000 7ffb8aa00640 1 -- 172.21.15.90:0/871700795 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb8c19f360 con 0x7ffb8c10d300 2024-04-02T14:51:34.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:34.865+0000 7ffb8aa00640 1 --2- 172.21.15.90:0/871700795 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffb8c10d300 0x7ffb8c1a4be0 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7ffb7802c250 tx=0x7ffb780042f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:51:34.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:34.866+0000 7ffb83e00640 1 -- 172.21.15.90:0/871700795 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb78041070 con 0x7ffb8c10d300 2024-04-02T14:51:34.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:34.866+0000 7ffb83e00640 1 -- 172.21.15.90:0/871700795 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffb7802cca0 con 0x7ffb8c10d300 2024-04-02T14:51:34.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:34.866+0000 7ffb83e00640 1 -- 172.21.15.90:0/871700795 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb78003cf0 con 0x7ffb8c10d300 2024-04-02T14:51:34.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:34.866+0000 7ffb91200640 1 -- 172.21.15.90:0/871700795 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffb8c19f5f0 con 0x7ffb8c10d300 2024-04-02T14:51:34.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:34.866+0000 7ffb91200640 1 -- 172.21.15.90:0/871700795 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffb8c1ab780 con 0x7ffb8c10d300 2024-04-02T14:51:34.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:34.867+0000 7ffb91200640 1 -- 172.21.15.90:0/871700795 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffb8c109700 con 0x7ffb8c10d300 2024-04-02T14:51:34.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:34.869+0000 7ffb83e00640 1 -- 172.21.15.90:0/871700795 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7ffb78044070 con 0x7ffb8c10d300 2024-04-02T14:51:34.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:34.870+0000 7ffb83e00640 1 --2- 172.21.15.90:0/871700795 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7ffb700a3da0 0x7ffb700a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:51:34.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:34.870+0000 7ffb83e00640 1 -- 172.21.15.90:0/871700795 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 8550+0+0 (secure 0 0 0) 0x7ffb780f22d0 con 0x7ffb8c10d300 2024-04-02T14:51:34.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:34.870+0000 7ffb8b400640 1 --2- 172.21.15.90:0/871700795 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7ffb700a3da0 0x7ffb700a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:51:34.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:34.871+0000 7ffb8b400640 1 --2- 172.21.15.90:0/871700795 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7ffb700a3da0 0x7ffb700a6260 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7ffb7c009bd0 tx=0x7ffb7c005f90 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:51:34.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:34.872+0000 7ffb83e00640 1 -- 172.21.15.90:0/871700795 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7ffb780be3b0 con 0x7ffb8c10d300 2024-04-02T14:51:35.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:35.120+0000 7ffb91200640 1 -- 172.21.15.90:0/871700795 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7ffb8c110520 con 0x7ffb8c10d300 2024-04-02T14:51:35.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:35.130+0000 7ffb83e00640 1 -- 172.21.15.90:0/871700795 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v28) ==== 161+0+0 (secure 0 0 0) 0x7ffb780c26b0 con 0x7ffb8c10d300 2024-04-02T14:51:35.122 INFO:teuthology.orchestra.run.smithi090.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-02T14:51:35.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:35.133+0000 7ffb91200640 1 -- 172.21.15.90:0/871700795 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7ffb700a3da0 msgr2=0x7ffb700a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:51:35.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:35.133+0000 7ffb91200640 1 --2- 172.21.15.90:0/871700795 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7ffb700a3da0 0x7ffb700a6260 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7ffb7c009bd0 tx=0x7ffb7c005f90 comp rx=0 tx=0).stop 2024-04-02T14:51:35.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:35.133+0000 7ffb91200640 1 -- 172.21.15.90:0/871700795 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffb8c10d300 msgr2=0x7ffb8c1a4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:51:35.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:35.133+0000 7ffb91200640 1 --2- 172.21.15.90:0/871700795 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffb8c10d300 0x7ffb8c1a4be0 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7ffb7802c250 tx=0x7ffb780042f0 comp rx=0 tx=0).stop 2024-04-02T14:51:35.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:35.133+0000 7ffb91200640 1 -- 172.21.15.90:0/871700795 shutdown_connections 2024-04-02T14:51:35.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:35.133+0000 7ffb91200640 1 --2- 172.21.15.90:0/871700795 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7ffb700a3da0 0x7ffb700a6260 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:51:35.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:35.133+0000 7ffb91200640 1 --2- 172.21.15.90:0/871700795 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffb8c19e760 0x7ffb8c19ebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:51:35.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:35.133+0000 7ffb91200640 1 --2- 172.21.15.90:0/871700795 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffb8c10d300 0x7ffb8c1a4be0 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:51:35.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:35.133+0000 7ffb91200640 1 --2- 172.21.15.90:0/871700795 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ffb8c1082b0 0x7ffb8c1a46a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:51:35.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:35.133+0000 7ffb91200640 1 -- 172.21.15.90:0/871700795 >> 172.21.15.90:0/871700795 conn(0x7ffb8c077920 msgr2=0x7ffb8c0794e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:51:35.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:35.133+0000 7ffb91200640 1 -- 172.21.15.90:0/871700795 shutdown_connections 2024-04-02T14:51:35.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:35.133+0000 7ffb91200640 1 -- 172.21.15.90:0/871700795 wait complete. 2024-04-02T14:51:35.146 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-04-02T14:51:35.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:35.340+0000 7f3c84200640 1 -- 172.21.15.90:0/2011377458 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3c7c0747f0 msgr2=0x7f3c7c10ddf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:51:35.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:35.340+0000 7f3c84200640 1 --2- 172.21.15.90:0/2011377458 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3c7c0747f0 0x7f3c7c10ddf0 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f3c68009860 tx=0x7f3c6802bc10 comp rx=0 tx=0).stop 2024-04-02T14:51:35.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:35.340+0000 7f3c84200640 1 -- 172.21.15.90:0/2011377458 shutdown_connections 2024-04-02T14:51:35.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:35.340+0000 7f3c84200640 1 --2- 172.21.15.90:0/2011377458 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3c7c10e330 0x7f3c7c110740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:51:35.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:35.340+0000 7f3c84200640 1 --2- 172.21.15.90:0/2011377458 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3c7c0747f0 0x7f3c7c10ddf0 unknown :-1 s=CLOSED pgs=869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:51:35.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:35.340+0000 7f3c84200640 1 --2- 172.21.15.90:0/2011377458 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3c7c073ed0 0x7f3c7c0742b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:51:35.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:35.340+0000 7f3c84200640 1 -- 172.21.15.90:0/2011377458 >> 172.21.15.90:0/2011377458 conn(0x7f3c7c078f00 msgr2=0x7f3c7c079310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:51:35.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:35.340+0000 7f3c84200640 1 -- 172.21.15.90:0/2011377458 shutdown_connections 2024-04-02T14:51:35.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:35.340+0000 7f3c84200640 1 -- 172.21.15.90:0/2011377458 wait complete. 2024-04-02T14:51:35.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:35.341+0000 7f3c84200640 1 Processor -- start 2024-04-02T14:51:35.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:35.341+0000 7f3c84200640 1 -- start start 2024-04-02T14:51:35.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:35.342+0000 7f3c84200640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3c7c073ed0 0x7f3c7c19a650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:51:35.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:35.342+0000 7f3c84200640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3c7c0747f0 0x7f3c7c19ab90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:51:35.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:35.342+0000 7f3c84200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3c7c10e330 0x7f3c7c19eec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:51:35.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:35.342+0000 7f3c84200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f3c7c114090 con 0x7f3c7c10e330 2024-04-02T14:51:35.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:35.342+0000 7f3c84200640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f3c7c113f10 con 0x7f3c7c0747f0 2024-04-02T14:51:35.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:35.342+0000 7f3c84200640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f3c7c114210 con 0x7f3c7c073ed0 2024-04-02T14:51:35.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:35.342+0000 7f3c81800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3c7c073ed0 0x7f3c7c19a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:51:35.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:35.342+0000 7f3c81800640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3c7c073ed0 0x7f3c7c19a650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.90:48118/0 (socket says 172.21.15.90:48118) 2024-04-02T14:51:35.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:35.342+0000 7f3c80e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3c7c0747f0 0x7f3c7c19ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:51:35.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:35.342+0000 7f3c81800640 1 -- 172.21.15.90:0/1503803052 learned_addr learned my addr 172.21.15.90:0/1503803052 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:51:35.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:35.342+0000 7f3c82200640 1 --2- 172.21.15.90:0/1503803052 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3c7c10e330 0x7f3c7c19eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:51:35.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:35.342+0000 7f3c81800640 1 -- 172.21.15.90:0/1503803052 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3c7c0747f0 msgr2=0x7f3c7c19ab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:51:35.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:35.342+0000 7f3c81800640 1 --2- 172.21.15.90:0/1503803052 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3c7c0747f0 0x7f3c7c19ab90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:51:35.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:35.342+0000 7f3c81800640 1 -- 172.21.15.90:0/1503803052 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3c7c10e330 msgr2=0x7f3c7c19eec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:51:35.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:35.342+0000 7f3c81800640 1 --2- 172.21.15.90:0/1503803052 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3c7c10e330 0x7f3c7c19eec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:51:35.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:35.342+0000 7f3c81800640 1 -- 172.21.15.90:0/1503803052 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c7c19f5f0 con 0x7f3c7c073ed0 2024-04-02T14:51:35.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:35.342+0000 7f3c82200640 1 --2- 172.21.15.90:0/1503803052 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3c7c10e330 0x7f3c7c19eec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:51:35.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:35.342+0000 7f3c80e00640 1 --2- 172.21.15.90:0/1503803052 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3c7c0747f0 0x7f3c7c19ab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:51:35.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:35.343+0000 7f3c81800640 1 --2- 172.21.15.90:0/1503803052 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3c7c073ed0 0x7f3c7c19a650 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f3c7000e9f0 tx=0x7f3c7000eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:51:35.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:35.343+0000 7f3c7a000640 1 -- 172.21.15.90:0/1503803052 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c7000cd80 con 0x7f3c7c073ed0 2024-04-02T14:51:35.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:35.343+0000 7f3c7a000640 1 -- 172.21.15.90:0/1503803052 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3c70010910 con 0x7f3c7c073ed0 2024-04-02T14:51:35.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:35.343+0000 7f3c84200640 1 -- 172.21.15.90:0/1503803052 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3c7c19f8e0 con 0x7f3c7c073ed0 2024-04-02T14:51:35.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:35.343+0000 7f3c7a000640 1 -- 172.21.15.90:0/1503803052 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c70018990 con 0x7f3c7c073ed0 2024-04-02T14:51:35.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:35.343+0000 7f3c84200640 1 -- 172.21.15.90:0/1503803052 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3c7c1a7170 con 0x7f3c7c073ed0 2024-04-02T14:51:35.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:35.344+0000 7f3c84200640 1 -- 172.21.15.90:0/1503803052 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3c7c068d40 con 0x7f3c7c073ed0 2024-04-02T14:51:35.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:35.346+0000 7f3c7a000640 1 -- 172.21.15.90:0/1503803052 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f3c70010ac0 con 0x7f3c7c073ed0 2024-04-02T14:51:35.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:35.347+0000 7f3c7a000640 1 --2- 172.21.15.90:0/1503803052 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f3c640a3da0 0x7f3c640a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:51:35.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:35.347+0000 7f3c7a000640 1 -- 172.21.15.90:0/1503803052 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(112..112 src has 1..112) ==== 8936+0+0 (secure 0 0 0) 0x7f3c700d0d30 con 0x7f3c7c073ed0 2024-04-02T14:51:35.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:35.347+0000 7f3c80e00640 1 --2- 172.21.15.90:0/1503803052 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f3c640a3da0 0x7f3c640a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:51:35.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:35.348+0000 7f3c80e00640 1 --2- 172.21.15.90:0/1503803052 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f3c640a3da0 0x7f3c640a6260 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f3c6802c220 tx=0x7f3c6802e040 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:51:35.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:35.351+0000 7f3c7a000640 1 -- 172.21.15.90:0/1503803052 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7f3c7009d1d0 con 0x7f3c7c073ed0 2024-04-02T14:51:35.428 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:51:35 smithi177 ceph-mon[36431]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:51:35.428 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:51:35 smithi177 ceph-mon[36431]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T14:51:35.428 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:51:35 smithi177 ceph-mon[36431]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' 2024-04-02T14:51:35.428 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:51:35 smithi177 ceph-mon[36431]: from='client.? 172.21.15.90:0/49961366' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-02T14:51:35.428 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:51:35 smithi177 ceph-mon[36431]: from='client.? 172.21.15.90:0/871700795' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-02T14:51:35.428 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:51:35 smithi177 ceph-mon[36431]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-02T14:51:35.428 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:51:35 smithi177 ceph-mon[36431]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-02T14:51:35.428 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:51:35 smithi177 ceph-mon[36431]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-02T14:51:35.428 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:51:35 smithi177 ceph-mon[36431]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-04-02T14:51:35.428 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:51:35 smithi177 ceph-mon[36431]: osdmap e112: 12 total, 12 up, 12 in 2024-04-02T14:51:35.428 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:51:35 smithi177 ceph-mon[36431]: fsmap cephfs:0/5 2 up:standby, 5 failed 2024-04-02T14:51:35.429 INFO:journalctl@ceph.mds.f.smithi177.stdout:Apr 02 14:51:35 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mds-f[66068]: ignoring --setuser ceph since I am not root 2024-04-02T14:51:35.429 INFO:journalctl@ceph.mds.f.smithi177.stdout:Apr 02 14:51:35 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mds-f[66068]: ignoring --setgroup ceph since I am not root 2024-04-02T14:51:35.429 INFO:journalctl@ceph.mds.f.smithi177.stdout:Apr 02 14:51:35 smithi177 ceph-mds[66072]: -- 172.21.15.177:0/1080646265 <== mon.1 v2:172.21.15.177:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x564ef18d5a40 con 0x564ef18a8c00 2024-04-02T14:51:35.429 INFO:journalctl@ceph.mds.f.smithi177.stdout:Apr 02 14:51:35 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mds-f[66068]: starting mds.f at 2024-04-02T14:51:35.429 INFO:journalctl@ceph.mds.f.smithi177.stdout:Apr 02 14:51:35 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mds-f[66068]: 2024-04-02T14:51:35.172+0000 7f91a7152700 -1 PosixStack listen unable to listen on v2:0.0.0.0:6832/0: (98) Address already in use 2024-04-02T14:51:35.429 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 02 14:51:35 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mds-c[66371]: ignoring --setuser ceph since I am not root 2024-04-02T14:51:35.429 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 02 14:51:35 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mds-c[66371]: ignoring --setgroup ceph since I am not root 2024-04-02T14:51:35.430 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 02 14:51:35 smithi177 ceph-mds[66375]: -- 172.21.15.177:0/2743740009 <== mon.1 v2:172.21.15.177:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x562fd3e83a40 con 0x562fd3e57000 2024-04-02T14:51:35.430 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 02 14:51:35 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mds-c[66371]: starting mds.c at 2024-04-02T14:51:35.430 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 02 14:51:35 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mds-i[66682]: ignoring --setuser ceph since I am not root 2024-04-02T14:51:35.430 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 02 14:51:35 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mds-i[66682]: ignoring --setgroup ceph since I am not root 2024-04-02T14:51:35.430 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 02 14:51:35 smithi177 ceph-mds[66686]: -- 172.21.15.177:0/137868405 <== mon.1 v2:172.21.15.177:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55fda35eda40 con 0x55fda35c0800 2024-04-02T14:51:35.430 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 02 14:51:35 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mds-i[66682]: starting mds.i at 2024-04-02T14:51:35.430 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 02 14:51:35 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mds-l[65756]: ignoring --setuser ceph since I am not root 2024-04-02T14:51:35.431 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 02 14:51:35 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mds-l[65756]: ignoring --setgroup ceph since I am not root 2024-04-02T14:51:35.431 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 02 14:51:35 smithi177 ceph-mds[65760]: -- 172.21.15.177:0/2468298363 <== mon.1 v2:172.21.15.177:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55f18f39fa40 con 0x55f18f373000 2024-04-02T14:51:35.431 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 02 14:51:35 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mds-l[65756]: starting mds.l at 2024-04-02T14:51:35.431 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:51:35 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mon-a[31670]: 2024-04-02T14:51:35.122+0000 7fa7b0c00700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-02T14:51:35.431 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:51:35 smithi090 ceph-mon[31674]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:51:35.431 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:51:35 smithi090 ceph-mon[31674]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T14:51:35.431 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:51:35 smithi090 ceph-mon[31674]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' 2024-04-02T14:51:35.431 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:51:35 smithi090 ceph-mon[31674]: from='client.? 172.21.15.90:0/49961366' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-02T14:51:35.431 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:51:35 smithi090 ceph-mon[31674]: from='client.? 172.21.15.90:0/871700795' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-02T14:51:35.432 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:51:35 smithi090 ceph-mon[31674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-02T14:51:35.432 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:51:35 smithi090 ceph-mon[31674]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-02T14:51:35.432 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:51:35 smithi090 ceph-mon[31674]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-02T14:51:35.432 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:51:35 smithi090 ceph-mon[31674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-04-02T14:51:35.432 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:51:35 smithi090 ceph-mon[31674]: osdmap e112: 12 total, 12 up, 12 in 2024-04-02T14:51:35.432 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:51:35 smithi090 ceph-mon[31674]: fsmap cephfs:0/5 2 up:standby, 5 failed 2024-04-02T14:51:35.432 INFO:journalctl@ceph.mds.j.smithi090.stdout:Apr 02 14:51:35 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mds-j[72161]: ignoring --setuser ceph since I am not root 2024-04-02T14:51:35.432 INFO:journalctl@ceph.mds.j.smithi090.stdout:Apr 02 14:51:35 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mds-j[72161]: ignoring --setgroup ceph since I am not root 2024-04-02T14:51:35.432 INFO:journalctl@ceph.mds.j.smithi090.stdout:Apr 02 14:51:35 smithi090 ceph-mds[72165]: -- 172.21.15.90:0/2821732099 <== mon.0 v2:172.21.15.90:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x560f134d7a40 con 0x560f134aac00 2024-04-02T14:51:35.432 INFO:journalctl@ceph.mds.j.smithi090.stdout:Apr 02 14:51:35 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mds-j[72161]: starting mds.j at 2024-04-02T14:51:35.433 INFO:journalctl@ceph.mds.g.smithi090.stdout:Apr 02 14:51:35 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mds-g[73104]: ignoring --setuser ceph since I am not root 2024-04-02T14:51:35.433 INFO:journalctl@ceph.mds.g.smithi090.stdout:Apr 02 14:51:35 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mds-g[73104]: ignoring --setgroup ceph since I am not root 2024-04-02T14:51:35.433 INFO:journalctl@ceph.mds.g.smithi090.stdout:Apr 02 14:51:35 smithi090 ceph-mds[73123]: -- 172.21.15.90:0/3824031977 <== mon.2 v2:172.21.15.92:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56331dc9fa40 con 0x56331dc73000 2024-04-02T14:51:35.433 INFO:journalctl@ceph.mds.g.smithi090.stdout:Apr 02 14:51:35 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mds-g[73104]: starting mds.g at 2024-04-02T14:51:35.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:35.594+0000 7f3c84200640 1 -- 172.21.15.90:0/1503803052 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f3c7c1094b0 con 0x7f3c7c073ed0 2024-04-02T14:51:35.601 INFO:journalctl@ceph.mds.b.smithi092.stdout:Apr 02 14:51:35 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mds-b[67422]: ignoring --setuser ceph since I am not root 2024-04-02T14:51:35.602 INFO:journalctl@ceph.mds.b.smithi092.stdout:Apr 02 14:51:35 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mds-b[67422]: ignoring --setgroup ceph since I am not root 2024-04-02T14:51:35.602 INFO:journalctl@ceph.mds.b.smithi092.stdout:Apr 02 14:51:35 smithi092 ceph-mds[67426]: -- 172.21.15.92:0/2433988994 <== mon.2 v2:172.21.15.92:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55d1519d9a40 con 0x55d1519acc00 2024-04-02T14:51:35.602 INFO:journalctl@ceph.mds.b.smithi092.stdout:Apr 02 14:51:35 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mds-b[67422]: starting mds.b at 2024-04-02T14:51:35.602 INFO:journalctl@ceph.mds.k.smithi092.stdout:Apr 02 14:51:35 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mds-k[66466]: ignoring --setuser ceph since I am not root 2024-04-02T14:51:35.602 INFO:journalctl@ceph.mds.k.smithi092.stdout:Apr 02 14:51:35 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mds-k[66466]: ignoring --setgroup ceph since I am not root 2024-04-02T14:51:35.602 INFO:journalctl@ceph.mds.k.smithi092.stdout:Apr 02 14:51:35 smithi092 ceph-mds[66470]: -- 172.21.15.92:0/1223537185 <== mon.2 v2:172.21.15.92:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55fb1fca1a40 con 0x55fb1fc74c00 2024-04-02T14:51:35.602 INFO:journalctl@ceph.mds.k.smithi092.stdout:Apr 02 14:51:35 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mds-k[66466]: starting mds.k at 2024-04-02T14:51:35.603 INFO:journalctl@ceph.mds.e.smithi092.stdout:Apr 02 14:51:35 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mds-e[66791]: ignoring --setuser ceph since I am not root 2024-04-02T14:51:35.603 INFO:journalctl@ceph.mds.e.smithi092.stdout:Apr 02 14:51:35 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mds-e[66791]: ignoring --setgroup ceph since I am not root 2024-04-02T14:51:35.603 INFO:journalctl@ceph.mds.e.smithi092.stdout:Apr 02 14:51:35 smithi092 ceph-mds[66795]: -- 172.21.15.92:0/2599443855 <== mon.2 v2:172.21.15.92:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55a4bdfd7a40 con 0x55a4bdfaac00 2024-04-02T14:51:35.603 INFO:journalctl@ceph.mds.e.smithi092.stdout:Apr 02 14:51:35 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mds-e[66791]: starting mds.e at 2024-04-02T14:51:35.603 INFO:journalctl@ceph.mds.e.smithi092.stdout:Apr 02 14:51:35 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mds-e[66791]: 2024-04-02T14:51:35.171+0000 7f486920d700 -1 PosixStack listen unable to listen on v2:0.0.0.0:6832/0: (98) Address already in use 2024-04-02T14:51:35.603 INFO:journalctl@ceph.mds.h.smithi092.stdout:Apr 02 14:51:35 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mds-h[67113]: ignoring --setuser ceph since I am not root 2024-04-02T14:51:35.603 INFO:journalctl@ceph.mds.h.smithi092.stdout:Apr 02 14:51:35 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mds-h[67113]: ignoring --setgroup ceph since I am not root 2024-04-02T14:51:35.603 INFO:journalctl@ceph.mds.h.smithi092.stdout:Apr 02 14:51:35 smithi092 ceph-mds[67117]: -- 172.21.15.92:0/4267141443 <== mon.1 v2:172.21.15.177:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x557fea87da40 con 0x557fea850c00 2024-04-02T14:51:35.603 INFO:journalctl@ceph.mds.h.smithi092.stdout:Apr 02 14:51:35 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mds-h[67113]: starting mds.h at 2024-04-02T14:51:35.604 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:51:35 smithi092 ceph-mon[36497]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-02T14:51:35.604 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:51:35 smithi092 ceph-mon[36497]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-02T14:51:35.604 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:51:35 smithi092 ceph-mon[36497]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' 2024-04-02T14:51:35.604 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:51:35 smithi092 ceph-mon[36497]: from='client.? 172.21.15.90:0/49961366' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-02T14:51:35.604 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:51:35 smithi092 ceph-mon[36497]: from='client.? 172.21.15.90:0/871700795' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-02T14:51:35.604 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:51:35 smithi092 ceph-mon[36497]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-02T14:51:35.604 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:51:35 smithi092 ceph-mon[36497]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-02T14:51:35.604 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:51:35 smithi092 ceph-mon[36497]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-02T14:51:35.604 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:51:35 smithi092 ceph-mon[36497]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-04-02T14:51:35.604 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:51:35 smithi092 ceph-mon[36497]: osdmap e112: 12 total, 12 up, 12 in 2024-04-02T14:51:35.604 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:51:35 smithi092 ceph-mon[36497]: fsmap cephfs:0/5 2 up:standby, 5 failed 2024-04-02T14:51:36.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:36.132+0000 7f3c7a000640 1 -- 172.21.15.90:0/1503803052 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v29) ==== 104+0+0 (secure 0 0 0) 0x7f3c7009cad0 con 0x7f3c7c073ed0 2024-04-02T14:51:36.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:36.135+0000 7f3c84200640 1 -- 172.21.15.90:0/1503803052 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f3c640a3da0 msgr2=0x7f3c640a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:51:36.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:36.135+0000 7f3c84200640 1 --2- 172.21.15.90:0/1503803052 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f3c640a3da0 0x7f3c640a6260 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f3c6802c220 tx=0x7f3c6802e040 comp rx=0 tx=0).stop 2024-04-02T14:51:36.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:36.135+0000 7f3c84200640 1 -- 172.21.15.90:0/1503803052 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3c7c073ed0 msgr2=0x7f3c7c19a650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:51:36.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:36.135+0000 7f3c84200640 1 --2- 172.21.15.90:0/1503803052 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3c7c073ed0 0x7f3c7c19a650 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f3c7000e9f0 tx=0x7f3c7000eeb0 comp rx=0 tx=0).stop 2024-04-02T14:51:36.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:36.136+0000 7f3c84200640 1 -- 172.21.15.90:0/1503803052 shutdown_connections 2024-04-02T14:51:36.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:36.136+0000 7f3c84200640 1 --2- 172.21.15.90:0/1503803052 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f3c640a3da0 0x7f3c640a6260 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:51:36.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:36.136+0000 7f3c84200640 1 --2- 172.21.15.90:0/1503803052 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3c7c10e330 0x7f3c7c19eec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:51:36.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:36.136+0000 7f3c84200640 1 --2- 172.21.15.90:0/1503803052 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3c7c0747f0 0x7f3c7c19ab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:51:36.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:36.136+0000 7f3c84200640 1 --2- 172.21.15.90:0/1503803052 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3c7c073ed0 0x7f3c7c19a650 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:51:36.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:36.136+0000 7f3c84200640 1 -- 172.21.15.90:0/1503803052 >> 172.21.15.90:0/1503803052 conn(0x7f3c7c078f00 msgr2=0x7f3c7c0fea60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:51:36.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:36.136+0000 7f3c84200640 1 -- 172.21.15.90:0/1503803052 shutdown_connections 2024-04-02T14:51:36.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:36.136+0000 7f3c84200640 1 -- 172.21.15.90:0/1503803052 wait complete. 2024-04-02T14:51:36.155 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-04-02T14:51:36.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:36.375+0000 7f0681c00640 1 -- 172.21.15.90:0/1225822607 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f067c108310 msgr2=0x7f067c1086f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:51:36.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:36.375+0000 7f0681c00640 1 --2- 172.21.15.90:0/1225822607 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f067c108310 0x7f067c1086f0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f066c009900 tx=0x7f066c02bc30 comp rx=0 tx=0).stop 2024-04-02T14:51:36.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:36.375+0000 7f0681c00640 1 -- 172.21.15.90:0/1225822607 shutdown_connections 2024-04-02T14:51:36.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:36.375+0000 7f0681c00640 1 --2- 172.21.15.90:0/1225822607 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f067c10d3e0 0x7f067c114fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:51:36.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:36.375+0000 7f0681c00640 1 --2- 172.21.15.90:0/1225822607 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f067c108cc0 0x7f067c10cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:51:36.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:36.375+0000 7f0681c00640 1 --2- 172.21.15.90:0/1225822607 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f067c108310 0x7f067c1086f0 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:51:36.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:36.375+0000 7f0681c00640 1 -- 172.21.15.90:0/1225822607 >> 172.21.15.90:0/1225822607 conn(0x7f067c076040 msgr2=0x7f067c078460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:51:36.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:36.375+0000 7f0681c00640 1 -- 172.21.15.90:0/1225822607 shutdown_connections 2024-04-02T14:51:36.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:36.375+0000 7f0681c00640 1 -- 172.21.15.90:0/1225822607 wait complete. 2024-04-02T14:51:36.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:36.376+0000 7f0681c00640 1 Processor -- start 2024-04-02T14:51:36.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:36.377+0000 7f0681c00640 1 -- start start 2024-04-02T14:51:36.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:36.377+0000 7f0681c00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f067c108310 0x7f067c1a46b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:51:36.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:36.377+0000 7f0681c00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f067c10d3e0 0x7f067c1a4bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:51:36.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:36.377+0000 7f0681c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f067c19e820 0x7f067c19ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:51:36.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:36.377+0000 7f0681c00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f067c1184e0 con 0x7f067c19e820 2024-04-02T14:51:36.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:36.377+0000 7f0681c00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f067c118360 con 0x7f067c108310 2024-04-02T14:51:36.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:36.377+0000 7f0681c00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f067c118660 con 0x7f067c10d3e0 2024-04-02T14:51:36.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:36.377+0000 7f067b400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f067c19e820 0x7f067c19ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:51:36.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:36.377+0000 7f067b400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f067c19e820 0x7f067c19ecd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:59510/0 (socket says 172.21.15.90:59510) 2024-04-02T14:51:36.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:36.377+0000 7f067b400640 1 -- 172.21.15.90:0/2771266443 learned_addr learned my addr 172.21.15.90:0/2771266443 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:51:36.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:36.377+0000 7f067a000640 1 --2- 172.21.15.90:0/2771266443 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f067c10d3e0 0x7f067c1a4bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:51:36.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:36.377+0000 7f067aa00640 1 --2- 172.21.15.90:0/2771266443 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f067c108310 0x7f067c1a46b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:51:36.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:36.378+0000 7f067b400640 1 -- 172.21.15.90:0/2771266443 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f067c10d3e0 msgr2=0x7f067c1a4bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:51:36.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:36.378+0000 7f067b400640 1 --2- 172.21.15.90:0/2771266443 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f067c10d3e0 0x7f067c1a4bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:51:36.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:36.378+0000 7f067b400640 1 -- 172.21.15.90:0/2771266443 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f067c108310 msgr2=0x7f067c1a46b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:51:36.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:36.378+0000 7f067b400640 1 --2- 172.21.15.90:0/2771266443 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f067c108310 0x7f067c1a46b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:51:36.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:36.378+0000 7f067b400640 1 -- 172.21.15.90:0/2771266443 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f067c19f3b0 con 0x7f067c19e820 2024-04-02T14:51:36.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:36.378+0000 7f067aa00640 1 --2- 172.21.15.90:0/2771266443 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f067c108310 0x7f067c1a46b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:51:36.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:36.378+0000 7f067b400640 1 --2- 172.21.15.90:0/2771266443 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f067c19e820 0x7f067c19ecd0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f067400eb10 tx=0x7f067400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:51:36.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:36.378+0000 7f0673400640 1 -- 172.21.15.90:0/2771266443 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0674004b20 con 0x7f067c19e820 2024-04-02T14:51:36.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:36.378+0000 7f0681c00640 1 -- 172.21.15.90:0/2771266443 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f067c19f6a0 con 0x7f067c19e820 2024-04-02T14:51:36.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:36.378+0000 7f0673400640 1 -- 172.21.15.90:0/2771266443 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0674004cc0 con 0x7f067c19e820 2024-04-02T14:51:36.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:36.378+0000 7f0673400640 1 -- 172.21.15.90:0/2771266443 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0674010760 con 0x7f067c19e820 2024-04-02T14:51:36.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:36.378+0000 7f0681c00640 1 -- 172.21.15.90:0/2771266443 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f067c1ab830 con 0x7f067c19e820 2024-04-02T14:51:36.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:36.379+0000 7f0681c00640 1 -- 172.21.15.90:0/2771266443 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f067c1097e0 con 0x7f067c19e820 2024-04-02T14:51:36.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:36.381+0000 7f0673400640 1 -- 172.21.15.90:0/2771266443 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f0674005020 con 0x7f067c19e820 2024-04-02T14:51:36.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:36.382+0000 7f0673400640 1 --2- 172.21.15.90:0/2771266443 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f06600a3d20 0x7f06600a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:51:36.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:36.382+0000 7f0673400640 1 -- 172.21.15.90:0/2771266443 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(112..112 src has 1..112) ==== 8936+0+0 (secure 0 0 0) 0x7f06740d0fb0 con 0x7f067c19e820 2024-04-02T14:51:36.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:36.383+0000 7f067aa00640 1 --2- 172.21.15.90:0/2771266443 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f06600a3d20 0x7f06600a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:51:36.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:36.383+0000 7f067aa00640 1 --2- 172.21.15.90:0/2771266443 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f06600a3d20 0x7f06600a61e0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f066c006010 tx=0x7f066c02df90 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:51:36.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:36.387+0000 7f0673400640 1 -- 172.21.15.90:0/2771266443 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7f067c19f6a0 con 0x7f067c19e820 2024-04-02T14:51:36.430 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:51:36 smithi090 ceph-mon[31674]: pgmap v524: 97 pgs: 97 active+clean; 3.9 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 25 KiB/s wr, 12 op/s 2024-04-02T14:51:36.430 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:51:36 smithi090 ceph-mon[31674]: from='client.? 172.21.15.90:0/1503803052' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-02T14:51:36.430 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:51:36 smithi090 ceph-mon[31674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-02T14:51:36.430 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:51:36 smithi090 ceph-mon[31674]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-02T14:51:36.431 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:51:36 smithi090 ceph-mon[31674]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-02T14:51:36.431 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:51:36 smithi090 ceph-mon[31674]: Cluster is now healthy 2024-04-02T14:51:36.431 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:51:36 smithi090 ceph-mon[31674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-04-02T14:51:36.431 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:51:36 smithi090 ceph-mon[31674]: mds.? [v2:172.21.15.177:6837/2726474392,v1:172.21.15.177:6839/2726474392] up:boot 2024-04-02T14:51:36.431 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:51:36 smithi090 ceph-mon[31674]: mds.? [v2:172.21.15.92:6838/3262723786,v1:172.21.15.92:6839/3262723786] up:boot 2024-04-02T14:51:36.431 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:51:36 smithi090 ceph-mon[31674]: mds.? [v2:172.21.15.92:6832/1814519403,v1:172.21.15.92:6834/1814519403] up:boot 2024-04-02T14:51:36.431 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:51:36 smithi090 ceph-mon[31674]: mds.? [v2:172.21.15.92:6833/635519828,v1:172.21.15.92:6835/635519828] up:boot 2024-04-02T14:51:36.431 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:51:36 smithi090 ceph-mon[31674]: mds.? [v2:172.21.15.177:6833/921106846,v1:172.21.15.177:6835/921106846] up:boot 2024-04-02T14:51:36.431 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:51:36 smithi090 ceph-mon[31674]: mds.? [v2:172.21.15.177:6832/18549368,v1:172.21.15.177:6834/18549368] up:boot 2024-04-02T14:51:36.431 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:51:36 smithi090 ceph-mon[31674]: mds.? [v2:172.21.15.177:6836/3919018090,v1:172.21.15.177:6838/3919018090] up:boot 2024-04-02T14:51:36.431 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:51:36 smithi090 ceph-mon[31674]: mds.? [v2:172.21.15.92:6836/2863617386,v1:172.21.15.92:6837/2863617386] up:boot 2024-04-02T14:51:36.431 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:51:36 smithi090 ceph-mon[31674]: mds.? [v2:172.21.15.90:6834/2143040567,v1:172.21.15.90:6835/2143040567] up:boot 2024-04-02T14:51:36.431 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:51:36 smithi090 ceph-mon[31674]: mds.? [v2:172.21.15.90:6838/2699191907,v1:172.21.15.90:6839/2699191907] up:boot 2024-04-02T14:51:36.431 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:51:36 smithi090 ceph-mon[31674]: fsmap 12 up:standby 2024-04-02T14:51:36.431 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:51:36 smithi090 ceph-mon[31674]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-02T14:51:36.432 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:51:36 smithi090 ceph-mon[31674]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-02T14:51:36.432 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:51:36 smithi090 ceph-mon[31674]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-02T14:51:36.432 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:51:36 smithi090 ceph-mon[31674]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-02T14:51:36.432 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:51:36 smithi090 ceph-mon[31674]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-02T14:51:36.432 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:51:36 smithi090 ceph-mon[31674]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-02T14:51:36.432 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:51:36 smithi090 ceph-mon[31674]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-02T14:51:36.432 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:51:36 smithi090 ceph-mon[31674]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-02T14:51:36.432 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:51:36 smithi090 ceph-mon[31674]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-02T14:51:36.432 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:51:36 smithi090 ceph-mon[31674]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-02T14:51:36.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:36.585+0000 7f0681c00640 1 -- 172.21.15.90:0/2771266443 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f067c068d40 con 0x7f067c19e820 2024-04-02T14:51:36.602 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:51:36 smithi092 ceph-mon[36497]: pgmap v524: 97 pgs: 97 active+clean; 3.9 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 25 KiB/s wr, 12 op/s 2024-04-02T14:51:36.602 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:51:36 smithi092 ceph-mon[36497]: from='client.? 172.21.15.90:0/1503803052' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-02T14:51:36.602 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:51:36 smithi092 ceph-mon[36497]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-02T14:51:36.602 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:51:36 smithi092 ceph-mon[36497]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-02T14:51:36.602 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:51:36 smithi092 ceph-mon[36497]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-02T14:51:36.602 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:51:36 smithi092 ceph-mon[36497]: Cluster is now healthy 2024-04-02T14:51:36.602 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:51:36 smithi092 ceph-mon[36497]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-04-02T14:51:36.602 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:51:36 smithi092 ceph-mon[36497]: mds.? [v2:172.21.15.177:6837/2726474392,v1:172.21.15.177:6839/2726474392] up:boot 2024-04-02T14:51:36.602 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:51:36 smithi092 ceph-mon[36497]: mds.? [v2:172.21.15.92:6838/3262723786,v1:172.21.15.92:6839/3262723786] up:boot 2024-04-02T14:51:36.602 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:51:36 smithi092 ceph-mon[36497]: mds.? [v2:172.21.15.92:6832/1814519403,v1:172.21.15.92:6834/1814519403] up:boot 2024-04-02T14:51:36.603 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:51:36 smithi092 ceph-mon[36497]: mds.? [v2:172.21.15.92:6833/635519828,v1:172.21.15.92:6835/635519828] up:boot 2024-04-02T14:51:36.603 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:51:36 smithi092 ceph-mon[36497]: mds.? [v2:172.21.15.177:6833/921106846,v1:172.21.15.177:6835/921106846] up:boot 2024-04-02T14:51:36.603 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:51:36 smithi092 ceph-mon[36497]: mds.? [v2:172.21.15.177:6832/18549368,v1:172.21.15.177:6834/18549368] up:boot 2024-04-02T14:51:36.603 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:51:36 smithi092 ceph-mon[36497]: mds.? [v2:172.21.15.177:6836/3919018090,v1:172.21.15.177:6838/3919018090] up:boot 2024-04-02T14:51:36.603 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:51:36 smithi092 ceph-mon[36497]: mds.? [v2:172.21.15.92:6836/2863617386,v1:172.21.15.92:6837/2863617386] up:boot 2024-04-02T14:51:36.603 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:51:36 smithi092 ceph-mon[36497]: mds.? [v2:172.21.15.90:6834/2143040567,v1:172.21.15.90:6835/2143040567] up:boot 2024-04-02T14:51:36.603 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:51:36 smithi092 ceph-mon[36497]: mds.? [v2:172.21.15.90:6838/2699191907,v1:172.21.15.90:6839/2699191907] up:boot 2024-04-02T14:51:36.603 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:51:36 smithi092 ceph-mon[36497]: fsmap 12 up:standby 2024-04-02T14:51:36.603 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:51:36 smithi092 ceph-mon[36497]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-02T14:51:36.603 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:51:36 smithi092 ceph-mon[36497]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-02T14:51:36.603 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:51:36 smithi092 ceph-mon[36497]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-02T14:51:36.603 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:51:36 smithi092 ceph-mon[36497]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-02T14:51:36.603 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:51:36 smithi092 ceph-mon[36497]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-02T14:51:36.604 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:51:36 smithi092 ceph-mon[36497]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-02T14:51:36.604 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:51:36 smithi092 ceph-mon[36497]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-02T14:51:36.604 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:51:36 smithi092 ceph-mon[36497]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-02T14:51:36.604 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:51:36 smithi092 ceph-mon[36497]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-02T14:51:36.604 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:51:36 smithi092 ceph-mon[36497]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-02T14:51:36.678 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:51:36 smithi177 ceph-mon[36431]: pgmap v524: 97 pgs: 97 active+clean; 3.9 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 25 KiB/s wr, 12 op/s 2024-04-02T14:51:36.678 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:51:36 smithi177 ceph-mon[36431]: from='client.? 172.21.15.90:0/1503803052' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-02T14:51:36.678 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:51:36 smithi177 ceph-mon[36431]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-02T14:51:36.678 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:51:36 smithi177 ceph-mon[36431]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-02T14:51:36.678 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:51:36 smithi177 ceph-mon[36431]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-02T14:51:36.678 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:51:36 smithi177 ceph-mon[36431]: Cluster is now healthy 2024-04-02T14:51:36.678 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:51:36 smithi177 ceph-mon[36431]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-04-02T14:51:36.678 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:51:36 smithi177 ceph-mon[36431]: mds.? [v2:172.21.15.177:6837/2726474392,v1:172.21.15.177:6839/2726474392] up:boot 2024-04-02T14:51:36.678 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:51:36 smithi177 ceph-mon[36431]: mds.? [v2:172.21.15.92:6838/3262723786,v1:172.21.15.92:6839/3262723786] up:boot 2024-04-02T14:51:36.678 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:51:36 smithi177 ceph-mon[36431]: mds.? [v2:172.21.15.92:6832/1814519403,v1:172.21.15.92:6834/1814519403] up:boot 2024-04-02T14:51:36.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:51:36 smithi177 ceph-mon[36431]: mds.? [v2:172.21.15.92:6833/635519828,v1:172.21.15.92:6835/635519828] up:boot 2024-04-02T14:51:36.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:51:36 smithi177 ceph-mon[36431]: mds.? [v2:172.21.15.177:6833/921106846,v1:172.21.15.177:6835/921106846] up:boot 2024-04-02T14:51:36.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:51:36 smithi177 ceph-mon[36431]: mds.? [v2:172.21.15.177:6832/18549368,v1:172.21.15.177:6834/18549368] up:boot 2024-04-02T14:51:36.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:51:36 smithi177 ceph-mon[36431]: mds.? [v2:172.21.15.177:6836/3919018090,v1:172.21.15.177:6838/3919018090] up:boot 2024-04-02T14:51:36.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:51:36 smithi177 ceph-mon[36431]: mds.? [v2:172.21.15.92:6836/2863617386,v1:172.21.15.92:6837/2863617386] up:boot 2024-04-02T14:51:36.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:51:36 smithi177 ceph-mon[36431]: mds.? [v2:172.21.15.90:6834/2143040567,v1:172.21.15.90:6835/2143040567] up:boot 2024-04-02T14:51:36.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:51:36 smithi177 ceph-mon[36431]: mds.? [v2:172.21.15.90:6838/2699191907,v1:172.21.15.90:6839/2699191907] up:boot 2024-04-02T14:51:36.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:51:36 smithi177 ceph-mon[36431]: fsmap 12 up:standby 2024-04-02T14:51:36.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:51:36 smithi177 ceph-mon[36431]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-02T14:51:36.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:51:36 smithi177 ceph-mon[36431]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-02T14:51:36.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:51:36 smithi177 ceph-mon[36431]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-02T14:51:36.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:51:36 smithi177 ceph-mon[36431]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-02T14:51:36.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:51:36 smithi177 ceph-mon[36431]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-02T14:51:36.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:51:36 smithi177 ceph-mon[36431]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-02T14:51:36.679 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:51:36 smithi177 ceph-mon[36431]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-02T14:51:36.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:51:36 smithi177 ceph-mon[36431]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-02T14:51:36.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:51:36 smithi177 ceph-mon[36431]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-02T14:51:36.680 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:51:36 smithi177 ceph-mon[36431]: from='mgr.25289 172.21.15.90:0/1895822338' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-02T14:51:37.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:37.291+0000 7f0673400640 1 -- 172.21.15.90:0/2771266443 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v113) ==== 181+0+0 (secure 0 0 0) 0x7f067c068d40 con 0x7f067c19e820 2024-04-02T14:51:37.284 INFO:teuthology.orchestra.run.smithi090.stderr:pool 'cephfs_metadata' removed 2024-04-02T14:51:37.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:37.295+0000 7f0681c00640 1 -- 172.21.15.90:0/2771266443 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f06600a3d20 msgr2=0x7f06600a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:51:37.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:37.295+0000 7f0681c00640 1 --2- 172.21.15.90:0/2771266443 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f06600a3d20 0x7f06600a61e0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f066c006010 tx=0x7f066c02df90 comp rx=0 tx=0).stop 2024-04-02T14:51:37.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:37.295+0000 7f0681c00640 1 -- 172.21.15.90:0/2771266443 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f067c19e820 msgr2=0x7f067c19ecd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:51:37.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:37.296+0000 7f0681c00640 1 --2- 172.21.15.90:0/2771266443 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f067c19e820 0x7f067c19ecd0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f067400eb10 tx=0x7f067400c2d0 comp rx=0 tx=0).stop 2024-04-02T14:51:37.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:37.297+0000 7f0681c00640 1 -- 172.21.15.90:0/2771266443 shutdown_connections 2024-04-02T14:51:37.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:37.297+0000 7f0681c00640 1 --2- 172.21.15.90:0/2771266443 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f06600a3d20 0x7f06600a61e0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:51:37.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:37.297+0000 7f0681c00640 1 --2- 172.21.15.90:0/2771266443 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f067c19e820 0x7f067c19ecd0 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:51:37.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:37.297+0000 7f0681c00640 1 --2- 172.21.15.90:0/2771266443 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f067c10d3e0 0x7f067c1a4bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:51:37.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:37.297+0000 7f0681c00640 1 --2- 172.21.15.90:0/2771266443 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f067c108310 0x7f067c1a46b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:51:37.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:37.297+0000 7f0681c00640 1 -- 172.21.15.90:0/2771266443 >> 172.21.15.90:0/2771266443 conn(0x7f067c076040 msgr2=0x7f067c0783a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:51:37.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:37.298+0000 7f0681c00640 1 -- 172.21.15.90:0/2771266443 shutdown_connections 2024-04-02T14:51:37.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:37.298+0000 7f0681c00640 1 -- 172.21.15.90:0/2771266443 wait complete. 2024-04-02T14:51:37.315 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-04-02T14:51:37.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:37.522+0000 7f1801e00640 1 -- 172.21.15.90:0/1664008859 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f17fc103f10 msgr2=0x7f17fc1042f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:51:37.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:37.522+0000 7f1801e00640 1 --2- 172.21.15.90:0/1664008859 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f17fc103f10 0x7f17fc1042f0 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7f17ec009900 tx=0x7f17ec02bc30 comp rx=0 tx=0).stop 2024-04-02T14:51:37.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:37.522+0000 7f1801e00640 1 -- 172.21.15.90:0/1664008859 shutdown_connections 2024-04-02T14:51:37.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:37.522+0000 7f1801e00640 1 --2- 172.21.15.90:0/1664008859 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f17fc108fe0 0x7f17fc110bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:51:37.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:37.522+0000 7f1801e00640 1 --2- 172.21.15.90:0/1664008859 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f17fc1048c0 0x7f17fc108900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:51:37.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:37.522+0000 7f1801e00640 1 --2- 172.21.15.90:0/1664008859 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f17fc103f10 0x7f17fc1042f0 unknown :-1 s=CLOSED pgs=1062 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:51:37.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:37.522+0000 7f1801e00640 1 -- 172.21.15.90:0/1664008859 >> 172.21.15.90:0/1664008859 conn(0x7f17fc0ff750 msgr2=0x7f17fc101b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:51:37.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:37.522+0000 7f1801e00640 1 -- 172.21.15.90:0/1664008859 shutdown_connections 2024-04-02T14:51:37.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:37.522+0000 7f1801e00640 1 -- 172.21.15.90:0/1664008859 wait complete. 2024-04-02T14:51:37.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:37.523+0000 7f1801e00640 1 Processor -- start 2024-04-02T14:51:37.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:37.524+0000 7f1801e00640 1 -- start start 2024-04-02T14:51:37.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:37.524+0000 7f1801e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f17fc103f10 0x7f17fc19a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:51:37.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:37.524+0000 7f1801e00640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f17fc108fe0 0x7f17fc19abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:51:37.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:37.524+0000 7f1801e00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f17fc19ef10 0x7f17fc19f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:51:37.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:37.524+0000 7f1801e00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f17fc1140e0 con 0x7f17fc103f10 2024-04-02T14:51:37.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:37.524+0000 7f1801e00640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 -- 0x7f17fc113f60 con 0x7f17fc108fe0 2024-04-02T14:51:37.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:37.524+0000 7f1801e00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f17fc114260 con 0x7f17fc19ef10 2024-04-02T14:51:37.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:37.524+0000 7f17faa00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f17fc103f10 0x7f17fc19a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:51:37.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:37.524+0000 7f17fa000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f17fc108fe0 0x7f17fc19abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:51:37.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:37.524+0000 7f17fa000640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f17fc108fe0 0x7f17fc19abe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.90:43388/0 (socket says 172.21.15.90:43388) 2024-04-02T14:51:37.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:37.524+0000 7f17fa000640 1 -- 172.21.15.90:0/3235082515 learned_addr learned my addr 172.21.15.90:0/3235082515 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-04-02T14:51:37.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:37.524+0000 7f17fb400640 1 --2- 172.21.15.90:0/3235082515 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f17fc19ef10 0x7f17fc19f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:51:37.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:37.524+0000 7f17fa000640 1 -- 172.21.15.90:0/3235082515 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f17fc19ef10 msgr2=0x7f17fc19f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:51:37.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:37.524+0000 7f17fa000640 1 --2- 172.21.15.90:0/3235082515 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f17fc19ef10 0x7f17fc19f3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:51:37.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:37.525+0000 7f17fa000640 1 -- 172.21.15.90:0/3235082515 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f17fc103f10 msgr2=0x7f17fc19a6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:51:37.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:37.525+0000 7f17fa000640 1 --2- 172.21.15.90:0/3235082515 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f17fc103f10 0x7f17fc19a6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:51:37.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:37.525+0000 7f17fa000640 1 -- 172.21.15.90:0/3235082515 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17fc19fa00 con 0x7f17fc108fe0 2024-04-02T14:51:37.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:37.525+0000 7f17fb400640 1 --2- 172.21.15.90:0/3235082515 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f17fc19ef10 0x7f17fc19f3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-02T14:51:37.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:37.525+0000 7f17faa00640 1 --2- 172.21.15.90:0/3235082515 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f17fc103f10 0x7f17fc19a6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-02T14:51:37.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:37.525+0000 7f17fa000640 1 --2- 172.21.15.90:0/3235082515 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f17fc108fe0 0x7f17fc19abe0 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7f17e800e9a0 tx=0x7f17e800ee60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:51:37.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:37.525+0000 7f1800a00640 1 -- 172.21.15.90:0/3235082515 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17e800cd80 con 0x7f17fc108fe0 2024-04-02T14:51:37.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:37.525+0000 7f1801e00640 1 -- 172.21.15.90:0/3235082515 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17fc19fcf0 con 0x7f17fc108fe0 2024-04-02T14:51:37.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:37.525+0000 7f1800a00640 1 -- 172.21.15.90:0/3235082515 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f17e80035f0 con 0x7f17fc108fe0 2024-04-02T14:51:37.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:37.525+0000 7f1800a00640 1 -- 172.21.15.90:0/3235082515 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17e8010670 con 0x7f17fc108fe0 2024-04-02T14:51:37.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:37.525+0000 7f1801e00640 1 -- 172.21.15.90:0/3235082515 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17fc1a74e0 con 0x7f17fc108fe0 2024-04-02T14:51:37.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:37.526+0000 7f1801e00640 1 -- 172.21.15.90:0/3235082515 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f17fc068d40 con 0x7f17fc108fe0 2024-04-02T14:51:37.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:37.529+0000 7f1800a00640 1 -- 172.21.15.90:0/3235082515 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f17e80037a0 con 0x7f17fc108fe0 2024-04-02T14:51:37.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:37.531+0000 7f1800a00640 1 --2- 172.21.15.90:0/3235082515 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f17e00a3d20 0x7f17e00a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-02T14:51:37.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:37.531+0000 7f1800a00640 1 -- 172.21.15.90:0/3235082515 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 8382+0+0 (secure 0 0 0) 0x7f17e80d0c20 con 0x7f17fc108fe0 2024-04-02T14:51:37.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:37.531+0000 7f17faa00640 1 --2- 172.21.15.90:0/3235082515 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f17e00a3d20 0x7f17e00a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-02T14:51:37.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:37.531+0000 7f17faa00640 1 --2- 172.21.15.90:0/3235082515 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f17e00a3d20 0x7f17e00a61e0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f17ec000c00 tx=0x7f17ec02df90 comp rx=0 tx=0).ready entity=mgr.25289 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-02T14:51:37.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:37.532+0000 7f1800a00640 1 -- 172.21.15.90:0/3235082515 <== mon.2 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188173 (secure 0 0 0) 0x7f17e809d2f0 con 0x7f17fc108fe0 2024-04-02T14:51:37.601 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:51:37 smithi092 ceph-mon[36497]: from='client.? 172.21.15.90:0/2771266443' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-02T14:51:37.677 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:51:37 smithi177 ceph-mon[36431]: from='client.? 172.21.15.90:0/2771266443' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-02T14:51:37.679 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:51:37 smithi090 ceph-mon[31674]: from='client.? 172.21.15.90:0/2771266443' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-02T14:51:37.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:37.737+0000 7f1801e00640 1 -- 172.21.15.90:0/3235082515 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f17fc19b9a0 con 0x7f17fc108fe0 2024-04-02T14:51:38.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:38.308+0000 7f1800a00640 1 -- 172.21.15.90:0/3235082515 <== mon.2 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v114) ==== 169+0+0 (secure 0 0 0) 0x7f17e809cbf0 con 0x7f17fc108fe0 2024-04-02T14:51:38.301 INFO:teuthology.orchestra.run.smithi090.stderr:pool 'cephfs_data' removed 2024-04-02T14:51:38.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:38.313+0000 7f1801e00640 1 -- 172.21.15.90:0/3235082515 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f17e00a3d20 msgr2=0x7f17e00a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:51:38.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:38.313+0000 7f1801e00640 1 --2- 172.21.15.90:0/3235082515 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f17e00a3d20 0x7f17e00a61e0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f17ec000c00 tx=0x7f17ec02df90 comp rx=0 tx=0).stop 2024-04-02T14:51:38.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:38.313+0000 7f1801e00640 1 -- 172.21.15.90:0/3235082515 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f17fc108fe0 msgr2=0x7f17fc19abe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-02T14:51:38.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:38.313+0000 7f1801e00640 1 --2- 172.21.15.90:0/3235082515 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f17fc108fe0 0x7f17fc19abe0 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7f17e800e9a0 tx=0x7f17e800ee60 comp rx=0 tx=0).stop 2024-04-02T14:51:38.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:38.315+0000 7f1801e00640 1 -- 172.21.15.90:0/3235082515 shutdown_connections 2024-04-02T14:51:38.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:38.315+0000 7f1801e00640 1 --2- 172.21.15.90:0/3235082515 >> [v2:172.21.15.90:6800/3904993754,v1:172.21.15.90:6801/3904993754] conn(0x7f17e00a3d20 0x7f17e00a61e0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:51:38.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:38.315+0000 7f1801e00640 1 --2- 172.21.15.90:0/3235082515 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f17fc19ef10 0x7f17fc19f3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:51:38.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:38.315+0000 7f1801e00640 1 --2- 172.21.15.90:0/3235082515 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f17fc108fe0 0x7f17fc19abe0 unknown :-1 s=CLOSED pgs=1063 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:51:38.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:38.315+0000 7f1801e00640 1 --2- 172.21.15.90:0/3235082515 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f17fc103f10 0x7f17fc19a6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-02T14:51:38.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:38.315+0000 7f1801e00640 1 -- 172.21.15.90:0/3235082515 >> 172.21.15.90:0/3235082515 conn(0x7f17fc0ff750 msgr2=0x7f17fc101ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-02T14:51:38.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:38.316+0000 7f1801e00640 1 -- 172.21.15.90:0/3235082515 shutdown_connections 2024-04-02T14:51:38.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-02T14:51:38.316+0000 7f1801e00640 1 -- 172.21.15.90:0/3235082515 wait complete. 2024-04-02T14:51:38.342 DEBUG:teuthology.orchestra.run.smithi090:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-02T14:51:38.454 DEBUG:teuthology.orchestra.run.smithi092:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-02T14:51:38.485 DEBUG:teuthology.orchestra.run.smithi177:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-02T14:51:38.514 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-04-02T14:51:38.514 DEBUG:teuthology.orchestra.run.smithi090:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-02T14:51:38.560 DEBUG:teuthology.orchestra.run.smithi092:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-02T14:51:38.579 DEBUG:teuthology.orchestra.run.smithi177:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-02T14:51:38.600 INFO:tasks.cephadm:Stopping all daemons... 2024-04-02T14:51:38.600 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-04-02T14:51:38.600 DEBUG:teuthology.orchestra.run.smithi090:> sudo systemctl stop ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4@mon.a 2024-04-02T14:51:38.602 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:51:38 smithi092 ceph-mon[36497]: pgmap v526: 97 pgs: 97 active+clean; 3.9 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 74 KiB/s wr, 12 op/s 2024-04-02T14:51:38.602 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:51:38 smithi092 ceph-mon[36497]: from='client.? 172.21.15.90:0/2771266443' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-04-02T14:51:38.602 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:51:38 smithi092 ceph-mon[36497]: osdmap e113: 12 total, 12 up, 12 in 2024-04-02T14:51:38.602 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:51:38 smithi092 ceph-mon[36497]: from='client.? 172.21.15.90:0/3235082515' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-02T14:51:38.602 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:51:38 smithi092 ceph-mon[36497]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-02T14:51:38.661 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:51:38 smithi090 ceph-mon[31674]: pgmap v526: 97 pgs: 97 active+clean; 3.9 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 74 KiB/s wr, 12 op/s 2024-04-02T14:51:38.661 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:51:38 smithi090 ceph-mon[31674]: from='client.? 172.21.15.90:0/2771266443' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-04-02T14:51:38.661 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:51:38 smithi090 ceph-mon[31674]: osdmap e113: 12 total, 12 up, 12 in 2024-04-02T14:51:38.661 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:51:38 smithi090 ceph-mon[31674]: from='client.? 172.21.15.90:0/3235082515' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-02T14:51:38.661 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:51:38 smithi090 ceph-mon[31674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-02T14:51:38.677 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:51:38 smithi177 ceph-mon[36431]: pgmap v526: 97 pgs: 97 active+clean; 3.9 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 74 KiB/s wr, 12 op/s 2024-04-02T14:51:38.677 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:51:38 smithi177 ceph-mon[36431]: from='client.? 172.21.15.90:0/2771266443' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-04-02T14:51:38.677 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:51:38 smithi177 ceph-mon[36431]: osdmap e113: 12 total, 12 up, 12 in 2024-04-02T14:51:38.677 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:51:38 smithi177 ceph-mon[36431]: from='client.? 172.21.15.90:0/3235082515' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-02T14:51:38.677 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:51:38 smithi177 ceph-mon[36431]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-02T14:51:38.929 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:51:38 smithi090 systemd[1]: Stopping Ceph mon.a for 6449448a-f0fc-11ee-b647-cb9ed24678a4... 2024-04-02T14:51:39.385 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:51:39 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mon-a[31670]: 2024-04-02T14:51:39.103+0000 7fa7b7a00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-02T14:51:39.385 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:51:39 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mon-a[31670]: 2024-04-02T14:51:39.103+0000 7fa7b7a00700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-04-02T14:51:39.386 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:51:39 smithi090 podman[110368]: 2024-04-02 14:51:39.134798237 +0000 UTC m=+0.192013050 container died c279381a76c0b9cad3ae8fc1060f16ff48ffbeccb15d933f53df963923db1eec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:191d963ac9de7322da8aff20b567b3fa458e2ef6, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=wip-batrick-testing-20240402.004512-191d963, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240326, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-04-02T14:51:39.679 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:51:39 smithi090 podman[110368]: 2024-04-02 14:51:39.532601047 +0000 UTC m=+0.589815872 container cleanup c279381a76c0b9cad3ae8fc1060f16ff48ffbeccb15d933f53df963923db1eec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:191d963ac9de7322da8aff20b567b3fa458e2ef6, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mon-a, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240402.004512-191d963, GIT_CLEAN=True, org.label-schema.build-date=20240326, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-02T14:51:39.679 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:51:39 smithi090 bash[110368]: ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mon-a 2024-04-02T14:51:40.099 INFO:journalctl@ceph.mon.a.smithi090.stdout:Apr 02 14:51:39 smithi090 podman[110381]: 2024-04-02 14:51:39.751083501 +0000 UTC m=+0.603582477 container remove c279381a76c0b9cad3ae8fc1060f16ff48ffbeccb15d933f53df963923db1eec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:191d963ac9de7322da8aff20b567b3fa458e2ef6, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mon-a, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240326, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240402.004512-191d963, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-04-02T14:51:40.125 DEBUG:teuthology.orchestra.run.smithi090:> sudo pkill -f 'journalctl -f -n 0 -u ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4@mon.a.service' 2024-04-02T14:51:40.250 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-02T14:51:40.478 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-04-02T14:51:40.478 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2024-04-02T14:51:40.479 DEBUG:teuthology.orchestra.run.smithi092:> sudo systemctl stop ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4@mon.b 2024-04-02T14:51:40.781 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:51:40 smithi092 systemd[1]: Stopping Ceph mon.b for 6449448a-f0fc-11ee-b647-cb9ed24678a4... 2024-04-02T14:51:41.101 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:51:40 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mon-b[36493]: 2024-04-02T14:51:40.788+0000 7ff1b3a81700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-02T14:51:41.102 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:51:40 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mon-b[36493]: 2024-04-02T14:51:40.788+0000 7ff1b3a81700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-04-02T14:51:41.102 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:51:40 smithi092 podman[73642]: 2024-04-02 14:51:40.869967465 +0000 UTC m=+0.198340718 container died 458efc42a6e4bec9b661b760c07f6dfc0677b57b56da11233fcc325fada6fa52 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mon-b, RELEASE=wip-batrick-testing-20240402.004512-191d963, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240326, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, io.buildah.version=1.33.5) 2024-04-02T14:51:41.601 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:51:41 smithi092 podman[73642]: 2024-04-02 14:51:41.213981668 +0000 UTC m=+0.542354883 container cleanup 458efc42a6e4bec9b661b760c07f6dfc0677b57b56da11233fcc325fada6fa52 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mon-b, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5, org.label-schema.build-date=20240326, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, RELEASE=wip-batrick-testing-20240402.004512-191d963, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-02T14:51:41.601 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:51:41 smithi092 bash[73642]: ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mon-b 2024-04-02T14:51:41.602 INFO:journalctl@ceph.mon.b.smithi092.stdout:Apr 02 14:51:41 smithi092 podman[73656]: 2024-04-02 14:51:41.381800428 +0000 UTC m=+0.507031511 container remove 458efc42a6e4bec9b661b760c07f6dfc0677b57b56da11233fcc325fada6fa52 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mon-b, RELEASE=wip-batrick-testing-20240402.004512-191d963, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240326, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-04-02T14:51:41.624 DEBUG:teuthology.orchestra.run.smithi092:> sudo pkill -f 'journalctl -f -n 0 -u ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4@mon.b.service' 2024-04-02T14:51:41.671 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-02T14:51:41.671 INFO:tasks.cephadm.mon.c:Stopped mon.b 2024-04-02T14:51:41.671 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2024-04-02T14:51:41.671 DEBUG:teuthology.orchestra.run.smithi177:> sudo systemctl stop ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4@mon.c 2024-04-02T14:51:41.981 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:51:41 smithi177 systemd[1]: Stopping Ceph mon.c for 6449448a-f0fc-11ee-b647-cb9ed24678a4... 2024-04-02T14:51:42.282 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:51:41 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mon-c[36419]: 2024-04-02T14:51:41.988+0000 7fbda021e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-02T14:51:42.282 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:51:41 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mon-c[36419]: 2024-04-02T14:51:41.988+0000 7fbda021e700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-04-02T14:51:42.282 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:51:42 smithi177 podman[73334]: 2024-04-02 14:51:42.029966985 +0000 UTC m=+0.162747014 container died caa84bb1a070cfe74a5016e64d8314759ba36a3063035b9aec22e9a11b8d3669 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mon-c, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240326, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240402.004512-191d963, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-04-02T14:51:42.649 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:51:42 smithi177 podman[73334]: 2024-04-02 14:51:42.439452418 +0000 UTC m=+0.572232436 container cleanup caa84bb1a070cfe74a5016e64d8314759ba36a3063035b9aec22e9a11b8d3669 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mon-c, ceph=True, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240402.004512-191d963, org.label-schema.build-date=20240326) 2024-04-02T14:51:42.649 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 02 14:51:42 smithi177 bash[73334]: ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mon-c 2024-04-02T14:51:42.787 DEBUG:teuthology.orchestra.run.smithi177:> sudo pkill -f 'journalctl -f -n 0 -u ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4@mon.c.service' 2024-04-02T14:51:42.849 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-02T14:51:42.849 INFO:tasks.cephadm.mon.c:Stopped mon.c 2024-04-02T14:51:42.850 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-04-02T14:51:42.850 DEBUG:teuthology.orchestra.run.smithi090:> sudo systemctl stop ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4@mgr.x 2024-04-02T14:51:43.373 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 02 14:51:42 smithi090 systemd[1]: Stopping Ceph mgr.x for 6449448a-f0fc-11ee-b647-cb9ed24678a4... 2024-04-02T14:51:43.680 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 02 14:51:43 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-x[31910]: 2024-04-02T14:51:43.377+0000 7ff694c00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.x -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-02T14:51:43.680 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 02 14:51:43 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-x[31910]: 2024-04-02T14:51:43.377+0000 7ff694c00700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-04-02T14:51:43.680 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 02 14:51:43 smithi090 podman[110486]: 2024-04-02 14:51:43.472300088 +0000 UTC m=+0.232935536 container died 3c48d7f60a2ef43864ecbcab9b9a093ebc25d4c6aeca6d8313f5c0a7d172988d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:191d963ac9de7322da8aff20b567b3fa458e2ef6, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-x, org.label-schema.build-date=20240326, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=wip-batrick-testing-20240402.004512-191d963, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-02T14:51:44.180 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 02 14:51:43 smithi090 podman[110486]: 2024-04-02 14:51:43.964902596 +0000 UTC m=+0.725538014 container cleanup 3c48d7f60a2ef43864ecbcab9b9a093ebc25d4c6aeca6d8313f5c0a7d172988d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:191d963ac9de7322da8aff20b567b3fa458e2ef6, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-x, RELEASE=wip-batrick-testing-20240402.004512-191d963, org.label-schema.build-date=20240326, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-02T14:51:44.180 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 02 14:51:43 smithi090 bash[110486]: ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-x 2024-04-02T14:51:44.607 INFO:journalctl@ceph.mgr.x.smithi090.stdout:Apr 02 14:51:44 smithi090 podman[110500]: 2024-04-02 14:51:44.191242764 +0000 UTC m=+0.700331144 container remove 3c48d7f60a2ef43864ecbcab9b9a093ebc25d4c6aeca6d8313f5c0a7d172988d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:191d963ac9de7322da8aff20b567b3fa458e2ef6, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-x, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240326, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240402.004512-191d963, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-04-02T14:51:44.633 DEBUG:teuthology.orchestra.run.smithi090:> sudo pkill -f 'journalctl -f -n 0 -u ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4@mgr.x.service' 2024-04-02T14:51:44.733 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-02T14:51:44.733 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-04-02T14:51:44.733 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-04-02T14:51:44.734 DEBUG:teuthology.orchestra.run.smithi092:> sudo systemctl stop ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4@mgr.y 2024-04-02T14:51:45.038 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 02 14:51:44 smithi092 systemd[1]: Stopping Ceph mgr.y for 6449448a-f0fc-11ee-b647-cb9ed24678a4... 2024-04-02T14:51:45.313 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 02 14:51:45 smithi092 podman[73763]: 2024-04-02 14:51:45.046696355 +0000 UTC m=+0.124177307 container died f24422517204d21896612990dfe79732e99158d6e42b90160a73f59da125b809 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-y, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240326, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240402.004512-191d963, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-04-02T14:51:45.601 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 02 14:51:45 smithi092 podman[73763]: 2024-04-02 14:51:45.451312494 +0000 UTC m=+0.528793438 container cleanup f24422517204d21896612990dfe79732e99158d6e42b90160a73f59da125b809 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-y, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240402.004512-191d963, io.buildah.version=1.33.5, org.label-schema.build-date=20240326, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-04-02T14:51:45.602 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 02 14:51:45 smithi092 bash[73763]: ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-y 2024-04-02T14:51:45.929 DEBUG:teuthology.orchestra.run.smithi092:> sudo pkill -f 'journalctl -f -n 0 -u ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4@mgr.y.service' 2024-04-02T14:51:45.968 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-02T14:51:45.968 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-04-02T14:51:45.969 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2024-04-02T14:51:45.969 DEBUG:teuthology.orchestra.run.smithi177:> sudo systemctl stop ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4@mgr.z 2024-04-02T14:51:46.281 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 02 14:51:46 smithi177 systemd[1]: Stopping Ceph mgr.z for 6449448a-f0fc-11ee-b647-cb9ed24678a4... 2024-04-02T14:51:46.593 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 02 14:51:46 smithi177 podman[73456]: 2024-04-02 14:51:46.289413725 +0000 UTC m=+0.131070786 container died 83b9170d642e06d1e26f425bdd3090175ca9e6ab53c0b8164a1f0d0d0c73c003 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-z, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240326, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240402.004512-191d963) 2024-04-02T14:51:46.928 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 02 14:51:46 smithi177 podman[73456]: 2024-04-02 14:51:46.601869528 +0000 UTC m=+0.443526595 container cleanup 83b9170d642e06d1e26f425bdd3090175ca9e6ab53c0b8164a1f0d0d0c73c003 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-z, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240402.004512-191d963, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240326, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-04-02T14:51:46.928 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 02 14:51:46 smithi177 bash[73456]: ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-z 2024-04-02T14:51:46.928 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 02 14:51:46 smithi177 podman[73471]: 2024-04-02 14:51:46.836637409 +0000 UTC m=+0.544162701 container remove 83b9170d642e06d1e26f425bdd3090175ca9e6ab53c0b8164a1f0d0d0c73c003 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mgr-z, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240326, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240402.004512-191d963, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-04-02T14:51:46.928 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 02 14:51:46 smithi177 systemd[1]: ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4@mgr.z.service: Main process exited, code=exited, status=143/n/a 2024-04-02T14:51:47.063 DEBUG:teuthology.orchestra.run.smithi177:> sudo pkill -f 'journalctl -f -n 0 -u ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4@mgr.z.service' 2024-04-02T14:51:47.099 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-02T14:51:47.099 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2024-04-02T14:51:47.099 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-04-02T14:51:47.099 DEBUG:teuthology.orchestra.run.smithi090:> sudo systemctl stop ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4@osd.0 2024-04-02T14:51:47.429 INFO:journalctl@ceph.osd.0.smithi090.stdout:Apr 02 14:51:47 smithi090 systemd[1]: Stopping Ceph osd.0 for 6449448a-f0fc-11ee-b647-cb9ed24678a4... 2024-04-02T14:51:47.929 INFO:journalctl@ceph.osd.0.smithi090.stdout:Apr 02 14:51:47 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-0[46353]: 2024-04-02T14:51:47.656+0000 7f822a000700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-02T14:51:47.930 INFO:journalctl@ceph.osd.0.smithi090.stdout:Apr 02 14:51:47 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-0[46353]: 2024-04-02T14:51:47.657+0000 7f822a000700 -1 osd.0 114 *** Got signal Terminated *** 2024-04-02T14:51:47.930 INFO:journalctl@ceph.osd.0.smithi090.stdout:Apr 02 14:51:47 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-0[46353]: 2024-04-02T14:51:47.657+0000 7f822a000700 -1 osd.0 114 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-02T14:52:13.679 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:52:13 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:52:13.370+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:49.133761+0000 front 2024-04-02T14:51:49.134172+0000 (oldest deadline 2024-04-02T14:52:13.233870+0000) 2024-04-02T14:52:13.851 INFO:journalctl@ceph.osd.1.smithi092.stdout:Apr 02 14:52:13 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-1[41091]: 2024-04-02T14:52:13.524+0000 7fa1a4619700 -1 osd.1 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:51.100791+0000 front 2024-04-02T14:51:51.100477+0000 (oldest deadline 2024-04-02T14:52:12.800313+0000) 2024-04-02T14:52:13.927 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:52:13 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:52:13.673+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:50.397138+0000 front 2024-04-02T14:51:50.397096+0000 (oldest deadline 2024-04-02T14:52:12.697036+0000) 2024-04-02T14:52:14.180 INFO:journalctl@ceph.osd.6.smithi090.stdout:Apr 02 14:52:13 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-6[59804]: 2024-04-02T14:52:13.868+0000 7fab55400700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:47.904942+0000 front 2024-04-02T14:51:47.905629+0000 (oldest deadline 2024-04-02T14:52:13.805157+0000) 2024-04-02T14:52:14.180 INFO:journalctl@ceph.osd.3.smithi090.stdout:Apr 02 14:52:13 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-3[53141]: 2024-04-02T14:52:13.941+0000 7fb518e00700 -1 osd.3 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:50.237494+0000 front 2024-04-02T14:51:50.237543+0000 (oldest deadline 2024-04-02T14:52:13.737537+0000) 2024-04-02T14:52:14.497 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:52:14 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:52:14.229+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:50.925796+0000 front 2024-04-02T14:51:50.925668+0000 (oldest deadline 2024-04-02T14:52:13.225431+0000) 2024-04-02T14:52:14.680 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:52:14 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:52:14.333+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:49.133761+0000 front 2024-04-02T14:51:49.134172+0000 (oldest deadline 2024-04-02T14:52:13.233870+0000) 2024-04-02T14:52:14.851 INFO:journalctl@ceph.osd.1.smithi092.stdout:Apr 02 14:52:14 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-1[41091]: 2024-04-02T14:52:14.505+0000 7fa1a4619700 -1 osd.1 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:51.100791+0000 front 2024-04-02T14:51:51.100477+0000 (oldest deadline 2024-04-02T14:52:12.800313+0000) 2024-04-02T14:52:14.927 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:52:14 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:52:14.669+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:50.397138+0000 front 2024-04-02T14:51:50.397096+0000 (oldest deadline 2024-04-02T14:52:12.697036+0000) 2024-04-02T14:52:15.179 INFO:journalctl@ceph.osd.3.smithi090.stdout:Apr 02 14:52:14 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-3[53141]: 2024-04-02T14:52:14.908+0000 7fb518e00700 -1 osd.3 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:50.237494+0000 front 2024-04-02T14:51:50.237543+0000 (oldest deadline 2024-04-02T14:52:13.737537+0000) 2024-04-02T14:52:15.179 INFO:journalctl@ceph.osd.0.smithi090.stdout:Apr 02 14:52:15 smithi090 podman[110606]: 2024-04-02 14:52:15.004618842 +0000 UTC m=+27.516448446 container died 9c4567e6e8f05c9874e0950b935b09d27ee6e4ded5f1b9a83b58420a4574b319 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-0, RELEASE=wip-batrick-testing-20240402.004512-191d963, ceph=True, org.label-schema.build-date=20240326, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_BRANCH=HEAD) 2024-04-02T14:52:15.180 INFO:journalctl@ceph.osd.6.smithi090.stdout:Apr 02 14:52:14 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-6[59804]: 2024-04-02T14:52:14.828+0000 7fab55400700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:47.904942+0000 front 2024-04-02T14:51:47.905629+0000 (oldest deadline 2024-04-02T14:52:13.805157+0000) 2024-04-02T14:52:15.641 INFO:journalctl@ceph.osd.0.smithi090.stdout:Apr 02 14:52:15 smithi090 podman[110606]: 2024-04-02 14:52:15.464110682 +0000 UTC m=+27.975940288 container cleanup 9c4567e6e8f05c9874e0950b935b09d27ee6e4ded5f1b9a83b58420a4574b319 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-0, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240402.004512-191d963, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240326, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-04-02T14:52:15.641 INFO:journalctl@ceph.osd.0.smithi090.stdout:Apr 02 14:52:15 smithi090 bash[110606]: ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-0 2024-04-02T14:52:15.929 INFO:journalctl@ceph.osd.0.smithi090.stdout:Apr 02 14:52:15 smithi090 podman[110638]: 2024-04-02 14:52:15.648981286 +0000 UTC m=+0.636334777 container remove 9c4567e6e8f05c9874e0950b935b09d27ee6e4ded5f1b9a83b58420a4574b319 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240326, RELEASE=wip-batrick-testing-20240402.004512-191d963, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-04-02T14:52:16.706 INFO:journalctl@ceph.osd.0.smithi090.stdout:Apr 02 14:52:16 smithi090 podman[110726]: 2024-04-02 14:52:16.347936426 +0000 UTC m=+0.036049683 image pull f7301e070545fc23d8e5b7b052a2c15e832a0e57102f1acad8e4f3583709d23f quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534 2024-04-02T14:52:16.706 INFO:journalctl@ceph.osd.0.smithi090.stdout:Apr 02 14:52:16 smithi090 podman[110726]: 2024-04-02 14:52:16.450036273 +0000 UTC m=+0.138149551 container create a4fd7b68597128a805e3720859652e69d9cec2a3e72a56690349ea68aae70469 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-0-deactivate, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240402.004512-191d963, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240326) 2024-04-02T14:52:17.143 INFO:journalctl@ceph.osd.0.smithi090.stdout:Apr 02 14:52:16 smithi090 podman[110726]: 2024-04-02 14:52:16.712191732 +0000 UTC m=+0.400305016 container init a4fd7b68597128a805e3720859652e69d9cec2a3e72a56690349ea68aae70469 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-0-deactivate, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240402.004512-191d963, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5, org.label-schema.build-date=20240326, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-04-02T14:52:17.143 INFO:journalctl@ceph.osd.0.smithi090.stdout:Apr 02 14:52:16 smithi090 podman[110726]: 2024-04-02 14:52:16.723729818 +0000 UTC m=+0.411843070 container start a4fd7b68597128a805e3720859652e69d9cec2a3e72a56690349ea68aae70469 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-0-deactivate, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, RELEASE=wip-batrick-testing-20240402.004512-191d963, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240326, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-04-02T14:52:17.143 INFO:journalctl@ceph.osd.0.smithi090.stdout:Apr 02 14:52:16 smithi090 podman[110726]: 2024-04-02 14:52:16.802642394 +0000 UTC m=+0.490755702 container attach a4fd7b68597128a805e3720859652e69d9cec2a3e72a56690349ea68aae70469 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-0-deactivate, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240326, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=wip-batrick-testing-20240402.004512-191d963, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-04-02T14:52:17.143 INFO:journalctl@ceph.osd.0.smithi090.stdout:Apr 02 14:52:17 smithi090 podman[110726]: 2024-04-02 14:52:17.150712639 +0000 UTC m=+0.838825880 container died a4fd7b68597128a805e3720859652e69d9cec2a3e72a56690349ea68aae70469 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-0-deactivate, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240326, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240402.004512-191d963, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-04-02T14:52:17.797 DEBUG:teuthology.orchestra.run.smithi090:> sudo pkill -f 'journalctl -f -n 0 -u ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4@osd.0.service' 2024-04-02T14:52:17.916 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-02T14:52:17.916 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-04-02T14:52:17.916 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-04-02T14:52:17.916 DEBUG:teuthology.orchestra.run.smithi092:> sudo systemctl stop ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4@osd.1 2024-04-02T14:52:18.311 INFO:journalctl@ceph.osd.1.smithi092.stdout:Apr 02 14:52:17 smithi092 systemd[1]: Stopping Ceph osd.1 for 6449448a-f0fc-11ee-b647-cb9ed24678a4... 2024-04-02T14:52:18.591 INFO:journalctl@ceph.osd.1.smithi092.stdout:Apr 02 14:52:18 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-1[41091]: 2024-04-02T14:52:18.318+0000 7fa1ac02d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-02T14:52:18.591 INFO:journalctl@ceph.osd.1.smithi092.stdout:Apr 02 14:52:18 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-1[41091]: 2024-04-02T14:52:18.318+0000 7fa1ac02d700 -1 osd.1 114 *** Got signal Terminated *** 2024-04-02T14:52:18.591 INFO:journalctl@ceph.osd.1.smithi092.stdout:Apr 02 14:52:18 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-1[41091]: 2024-04-02T14:52:18.318+0000 7fa1ac02d700 -1 osd.1 114 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-02T14:52:25.186 INFO:journalctl@ceph.osd.1.smithi092.stdout:Apr 02 14:52:24 smithi092 podman[73901]: 2024-04-02 14:52:24.917668417 +0000 UTC m=+6.807659704 container died 238e77d289fb247c6b20bea04e3c84ad502931ccad985e2fb493bb253fe137b9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-1, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=wip-batrick-testing-20240402.004512-191d963, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240326, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-04-02T14:52:25.553 INFO:journalctl@ceph.osd.1.smithi092.stdout:Apr 02 14:52:25 smithi092 podman[73901]: 2024-04-02 14:52:25.28521133 +0000 UTC m=+7.175202590 container cleanup 238e77d289fb247c6b20bea04e3c84ad502931ccad985e2fb493bb253fe137b9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-1, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240402.004512-191d963, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240326, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5) 2024-04-02T14:52:25.553 INFO:journalctl@ceph.osd.1.smithi092.stdout:Apr 02 14:52:25 smithi092 bash[73901]: ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-1 2024-04-02T14:52:25.851 INFO:journalctl@ceph.osd.1.smithi092.stdout:Apr 02 14:52:25 smithi092 podman[73932]: 2024-04-02 14:52:25.562108551 +0000 UTC m=+0.636782516 container remove 238e77d289fb247c6b20bea04e3c84ad502931ccad985e2fb493bb253fe137b9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-1, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240326, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240402.004512-191d963, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-04-02T14:52:26.148 INFO:journalctl@ceph.osd.1.smithi092.stdout:Apr 02 14:52:25 smithi092 podman[74023]: 2024-04-02 14:52:25.794475712 +0000 UTC m=+0.017635915 image pull f7301e070545fc23d8e5b7b052a2c15e832a0e57102f1acad8e4f3583709d23f quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534 2024-04-02T14:52:26.148 INFO:journalctl@ceph.osd.1.smithi092.stdout:Apr 02 14:52:25 smithi092 podman[74023]: 2024-04-02 14:52:25.937913293 +0000 UTC m=+0.161073491 container create 08d46582db5489cb36bba3d5e7b5fd5f45f63d9bac2cf7c572f5ae964d531e37 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-1-deactivate, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240402.004512-191d963, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240326, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-04-02T14:52:26.443 INFO:journalctl@ceph.osd.1.smithi092.stdout:Apr 02 14:52:26 smithi092 podman[74023]: 2024-04-02 14:52:26.157153797 +0000 UTC m=+0.380314002 container init 08d46582db5489cb36bba3d5e7b5fd5f45f63d9bac2cf7c572f5ae964d531e37 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-1-deactivate, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240402.004512-191d963, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240326, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-04-02T14:52:26.443 INFO:journalctl@ceph.osd.1.smithi092.stdout:Apr 02 14:52:26 smithi092 podman[74023]: 2024-04-02 14:52:26.160319768 +0000 UTC m=+0.383479974 container start 08d46582db5489cb36bba3d5e7b5fd5f45f63d9bac2cf7c572f5ae964d531e37 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-1-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240326, RELEASE=wip-batrick-testing-20240402.004512-191d963) 2024-04-02T14:52:26.443 INFO:journalctl@ceph.osd.1.smithi092.stdout:Apr 02 14:52:26 smithi092 podman[74023]: 2024-04-02 14:52:26.231951986 +0000 UTC m=+0.455112201 container attach 08d46582db5489cb36bba3d5e7b5fd5f45f63d9bac2cf7c572f5ae964d531e37 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-1-deactivate, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240402.004512-191d963, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240326, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-04-02T14:52:26.743 INFO:journalctl@ceph.osd.1.smithi092.stdout:Apr 02 14:52:26 smithi092 podman[74023]: 2024-04-02 14:52:26.451196563 +0000 UTC m=+0.674356767 container died 08d46582db5489cb36bba3d5e7b5fd5f45f63d9bac2cf7c572f5ae964d531e37 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-1-deactivate, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240326, RELEASE=wip-batrick-testing-20240402.004512-191d963, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-04-02T14:52:27.016 DEBUG:teuthology.orchestra.run.smithi092:> sudo pkill -f 'journalctl -f -n 0 -u ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4@osd.1.service' 2024-04-02T14:52:27.057 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-02T14:52:27.057 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-04-02T14:52:27.057 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-04-02T14:52:27.057 DEBUG:teuthology.orchestra.run.smithi177:> sudo systemctl stop ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4@osd.2 2024-04-02T14:52:27.361 INFO:journalctl@ceph.osd.2.smithi177.stdout:Apr 02 14:52:27 smithi177 systemd[1]: Stopping Ceph osd.2 for 6449448a-f0fc-11ee-b647-cb9ed24678a4... 2024-04-02T14:52:27.677 INFO:journalctl@ceph.osd.2.smithi177.stdout:Apr 02 14:52:27 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-2[41565]: 2024-04-02T14:52:27.369+0000 7f0c97370700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-02T14:52:27.677 INFO:journalctl@ceph.osd.2.smithi177.stdout:Apr 02 14:52:27 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-2[41565]: 2024-04-02T14:52:27.369+0000 7f0c97370700 -1 osd.2 114 *** Got signal Terminated *** 2024-04-02T14:52:27.677 INFO:journalctl@ceph.osd.2.smithi177.stdout:Apr 02 14:52:27 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-2[41565]: 2024-04-02T14:52:27.369+0000 7f0c97370700 -1 osd.2 114 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-02T14:52:33.327 INFO:journalctl@ceph.osd.2.smithi177.stdout:Apr 02 14:52:33 smithi177 podman[73594]: 2024-04-02 14:52:33.057760156 +0000 UTC m=+5.793998456 container died e85618ec2a597a0f4695d8bfb82dbdd90e661e88aa929aec81327a1a62fba4e4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240402.004512-191d963, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240326, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-04-02T14:52:33.677 INFO:journalctl@ceph.osd.2.smithi177.stdout:Apr 02 14:52:33 smithi177 podman[73594]: 2024-04-02 14:52:33.410164792 +0000 UTC m=+6.146403091 container cleanup e85618ec2a597a0f4695d8bfb82dbdd90e661e88aa929aec81327a1a62fba4e4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240402.004512-191d963, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240326) 2024-04-02T14:52:33.677 INFO:journalctl@ceph.osd.2.smithi177.stdout:Apr 02 14:52:33 smithi177 bash[73594]: ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-2 2024-04-02T14:52:34.003 INFO:journalctl@ceph.osd.2.smithi177.stdout:Apr 02 14:52:33 smithi177 podman[73611]: 2024-04-02 14:52:33.686392518 +0000 UTC m=+0.625516189 container remove e85618ec2a597a0f4695d8bfb82dbdd90e661e88aa929aec81327a1a62fba4e4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-2, RELEASE=wip-batrick-testing-20240402.004512-191d963, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240326, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-04-02T14:52:34.281 INFO:journalctl@ceph.osd.2.smithi177.stdout:Apr 02 14:52:34 smithi177 podman[73702]: 2024-04-02 14:52:34.012088388 +0000 UTC m=+0.113090953 container create d0f3e676377841e6f04682a3758b950f68af23f0a7f020f253b993fc9f364597 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-2-deactivate, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240402.004512-191d963, ceph=True, org.label-schema.build-date=20240326, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-04-02T14:52:34.281 INFO:journalctl@ceph.osd.2.smithi177.stdout:Apr 02 14:52:34 smithi177 podman[73702]: 2024-04-02 14:52:33.916748828 +0000 UTC m=+0.017751363 image pull f7301e070545fc23d8e5b7b052a2c15e832a0e57102f1acad8e4f3583709d23f quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534 2024-04-02T14:52:34.281 INFO:journalctl@ceph.osd.2.smithi177.stdout:Apr 02 14:52:34 smithi177 podman[73702]: 2024-04-02 14:52:34.206903262 +0000 UTC m=+0.307905792 container init d0f3e676377841e6f04682a3758b950f68af23f0a7f020f253b993fc9f364597 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-2-deactivate, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240402.004512-191d963, io.buildah.version=1.33.5, org.label-schema.build-date=20240326, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=) 2024-04-02T14:52:34.281 INFO:journalctl@ceph.osd.2.smithi177.stdout:Apr 02 14:52:34 smithi177 podman[73702]: 2024-04-02 14:52:34.210466026 +0000 UTC m=+0.311468552 container start d0f3e676377841e6f04682a3758b950f68af23f0a7f020f253b993fc9f364597 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-2-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240402.004512-191d963, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240326, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-04-02T14:52:34.281 INFO:journalctl@ceph.osd.2.smithi177.stdout:Apr 02 14:52:34 smithi177 podman[73702]: 2024-04-02 14:52:34.289235894 +0000 UTC m=+0.390238420 container attach d0f3e676377841e6f04682a3758b950f68af23f0a7f020f253b993fc9f364597 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-2-deactivate, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240326, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, RELEASE=wip-batrick-testing-20240402.004512-191d963, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-02T14:52:34.677 INFO:journalctl@ceph.osd.2.smithi177.stdout:Apr 02 14:52:34 smithi177 podman[73702]: 2024-04-02 14:52:34.492137604 +0000 UTC m=+0.593140120 container died d0f3e676377841e6f04682a3758b950f68af23f0a7f020f253b993fc9f364597 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-2-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240326, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240402.004512-191d963, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True) 2024-04-02T14:52:35.046 DEBUG:teuthology.orchestra.run.smithi177:> sudo pkill -f 'journalctl -f -n 0 -u ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4@osd.2.service' 2024-04-02T14:52:35.089 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-02T14:52:35.089 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-04-02T14:52:35.089 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-04-02T14:52:35.089 DEBUG:teuthology.orchestra.run.smithi090:> sudo systemctl stop ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4@osd.3 2024-04-02T14:52:35.429 INFO:journalctl@ceph.osd.3.smithi090.stdout:Apr 02 14:52:35 smithi090 systemd[1]: Stopping Ceph osd.3 for 6449448a-f0fc-11ee-b647-cb9ed24678a4... 2024-04-02T14:52:36.179 INFO:journalctl@ceph.osd.3.smithi090.stdout:Apr 02 14:52:35 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-3[53141]: 2024-04-02T14:52:35.701+0000 7fb51d600700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-02T14:52:36.179 INFO:journalctl@ceph.osd.3.smithi090.stdout:Apr 02 14:52:35 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-3[53141]: 2024-04-02T14:52:35.701+0000 7fb51d600700 -1 osd.3 114 *** Got signal Terminated *** 2024-04-02T14:52:36.179 INFO:journalctl@ceph.osd.3.smithi090.stdout:Apr 02 14:52:35 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-3[53141]: 2024-04-02T14:52:35.701+0000 7fb51d600700 -1 osd.3 114 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-02T14:52:36.677 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:52:36 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:52:36.534+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:49.133761+0000 front 2024-04-02T14:51:49.134172+0000 (oldest deadline 2024-04-02T14:52:35.841983+0000) 2024-04-02T14:52:37.798 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:52:37 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:52:37.547+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:49.133761+0000 front 2024-04-02T14:51:49.134172+0000 (oldest deadline 2024-04-02T14:52:35.841983+0000) 2024-04-02T14:52:38.177 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:52:37 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:52:37.910+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:50.397138+0000 front 2024-04-02T14:51:50.397096+0000 (oldest deadline 2024-04-02T14:52:37.200213+0000) 2024-04-02T14:52:38.179 INFO:journalctl@ceph.osd.3.smithi090.stdout:Apr 02 14:52:37 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-3[53141]: 2024-04-02T14:52:37.806+0000 7fb518e00700 -1 osd.3 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:50.237494+0000 front 2024-04-02T14:51:50.237543+0000 (oldest deadline 2024-04-02T14:52:37.447729+0000) 2024-04-02T14:52:38.677 INFO:journalctl@ceph.osd.5.smithi177.stdout:Apr 02 14:52:38 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-5[48118]: 2024-04-02T14:52:38.409+0000 7f34e2cdd700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:50.261330+0000 front 2024-04-02T14:51:50.261317+0000 (oldest deadline 2024-04-02T14:52:37.664335+0000) 2024-04-02T14:52:38.807 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:52:38 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:52:38.558+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:49.133761+0000 front 2024-04-02T14:51:49.134172+0000 (oldest deadline 2024-04-02T14:52:35.841983+0000) 2024-04-02T14:52:39.177 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:52:38 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:52:38.870+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:50.397138+0000 front 2024-04-02T14:51:50.397096+0000 (oldest deadline 2024-04-02T14:52:37.200213+0000) 2024-04-02T14:52:39.179 INFO:journalctl@ceph.osd.3.smithi090.stdout:Apr 02 14:52:38 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-3[53141]: 2024-04-02T14:52:38.815+0000 7fb518e00700 -1 osd.3 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:50.237494+0000 front 2024-04-02T14:51:50.237543+0000 (oldest deadline 2024-04-02T14:52:37.447729+0000) 2024-04-02T14:52:39.677 INFO:journalctl@ceph.osd.5.smithi177.stdout:Apr 02 14:52:39 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-5[48118]: 2024-04-02T14:52:39.390+0000 7f34e2cdd700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:50.261330+0000 front 2024-04-02T14:51:50.261317+0000 (oldest deadline 2024-04-02T14:52:37.664335+0000) 2024-04-02T14:52:39.929 INFO:journalctl@ceph.osd.3.smithi090.stdout:Apr 02 14:52:39 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-3[53141]: 2024-04-02T14:52:39.840+0000 7fb518e00700 -1 osd.3 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:50.237494+0000 front 2024-04-02T14:51:50.237543+0000 (oldest deadline 2024-04-02T14:52:37.447729+0000) 2024-04-02T14:52:39.929 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:52:39 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:52:39.605+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:49.133761+0000 front 2024-04-02T14:51:49.134172+0000 (oldest deadline 2024-04-02T14:52:35.841983+0000) 2024-04-02T14:52:40.177 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:52:39 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:52:39.916+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:50.397138+0000 front 2024-04-02T14:51:50.397096+0000 (oldest deadline 2024-04-02T14:52:37.200213+0000) 2024-04-02T14:52:40.677 INFO:journalctl@ceph.osd.5.smithi177.stdout:Apr 02 14:52:40 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-5[48118]: 2024-04-02T14:52:40.402+0000 7f34e2cdd700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:50.261330+0000 front 2024-04-02T14:51:50.261317+0000 (oldest deadline 2024-04-02T14:52:37.664335+0000) 2024-04-02T14:52:40.929 INFO:journalctl@ceph.osd.6.smithi090.stdout:Apr 02 14:52:40 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-6[59804]: 2024-04-02T14:52:40.800+0000 7fab55400700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:47.904942+0000 front 2024-04-02T14:51:47.905629+0000 (oldest deadline 2024-04-02T14:52:40.210630+0000) 2024-04-02T14:52:40.929 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:52:40 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:52:40.611+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:49.133761+0000 front 2024-04-02T14:51:49.134172+0000 (oldest deadline 2024-04-02T14:52:35.841983+0000) 2024-04-02T14:52:41.351 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:52:40 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:52:40.927+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:50.925796+0000 front 2024-04-02T14:51:50.925668+0000 (oldest deadline 2024-04-02T14:52:40.128984+0000) 2024-04-02T14:52:41.416 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:52:40 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:52:40.948+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:50.397138+0000 front 2024-04-02T14:51:50.397096+0000 (oldest deadline 2024-04-02T14:52:37.200213+0000) 2024-04-02T14:52:41.677 INFO:journalctl@ceph.osd.5.smithi177.stdout:Apr 02 14:52:41 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-5[48118]: 2024-04-02T14:52:41.424+0000 7f34e2cdd700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:50.261330+0000 front 2024-04-02T14:51:50.261317+0000 (oldest deadline 2024-04-02T14:52:37.664335+0000) 2024-04-02T14:52:41.930 INFO:journalctl@ceph.osd.3.smithi090.stdout:Apr 02 14:52:41 smithi090 podman[110832]: 2024-04-02 14:52:41.661362147 +0000 UTC m=+6.139274677 container died 33ad45052eb459422fabb5d9d9ce2706465dc3b0379ae33ece0feaa1cfe3138c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-3, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240326, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=wip-batrick-testing-20240402.004512-191d963, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-02T14:52:41.930 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:52:41 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:52:41.630+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:49.133761+0000 front 2024-04-02T14:51:49.134172+0000 (oldest deadline 2024-04-02T14:52:35.841983+0000) 2024-04-02T14:52:41.930 INFO:journalctl@ceph.osd.6.smithi090.stdout:Apr 02 14:52:41 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-6[59804]: 2024-04-02T14:52:41.816+0000 7fab55400700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:47.904942+0000 front 2024-04-02T14:51:47.905629+0000 (oldest deadline 2024-04-02T14:52:40.210630+0000) 2024-04-02T14:52:42.257 INFO:journalctl@ceph.osd.3.smithi090.stdout:Apr 02 14:52:42 smithi090 podman[110832]: 2024-04-02 14:52:42.105419399 +0000 UTC m=+6.583331925 container cleanup 33ad45052eb459422fabb5d9d9ce2706465dc3b0379ae33ece0feaa1cfe3138c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-3, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240402.004512-191d963, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240326, CEPH_POINT_RELEASE=) 2024-04-02T14:52:42.257 INFO:journalctl@ceph.osd.3.smithi090.stdout:Apr 02 14:52:42 smithi090 bash[110832]: ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-3 2024-04-02T14:52:42.351 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:52:41 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:52:41.883+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:50.925796+0000 front 2024-04-02T14:51:50.925668+0000 (oldest deadline 2024-04-02T14:52:40.128984+0000) 2024-04-02T14:52:42.427 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:52:41 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:52:41.935+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:50.397138+0000 front 2024-04-02T14:51:50.397096+0000 (oldest deadline 2024-04-02T14:52:37.200213+0000) 2024-04-02T14:52:42.609 INFO:journalctl@ceph.osd.3.smithi090.stdout:Apr 02 14:52:42 smithi090 podman[110863]: 2024-04-02 14:52:42.264855463 +0000 UTC m=+0.597252054 container remove 33ad45052eb459422fabb5d9d9ce2706465dc3b0379ae33ece0feaa1cfe3138c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-3, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240402.004512-191d963, ceph=True, org.label-schema.build-date=20240326, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux ) 2024-04-02T14:52:42.927 INFO:journalctl@ceph.osd.5.smithi177.stdout:Apr 02 14:52:42 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-5[48118]: 2024-04-02T14:52:42.451+0000 7f34e2cdd700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:50.261330+0000 front 2024-04-02T14:51:50.261317+0000 (oldest deadline 2024-04-02T14:52:37.664335+0000) 2024-04-02T14:52:42.929 INFO:journalctl@ceph.osd.6.smithi090.stdout:Apr 02 14:52:42 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-6[59804]: 2024-04-02T14:52:42.790+0000 7fab55400700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:47.904942+0000 front 2024-04-02T14:51:47.905629+0000 (oldest deadline 2024-04-02T14:52:40.210630+0000) 2024-04-02T14:52:42.929 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:52:42 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:52:42.616+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:49.133761+0000 front 2024-04-02T14:51:49.134172+0000 (oldest deadline 2024-04-02T14:52:35.841983+0000) 2024-04-02T14:52:43.101 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:52:42 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:52:42.859+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:50.925796+0000 front 2024-04-02T14:51:50.925668+0000 (oldest deadline 2024-04-02T14:52:40.128984+0000) 2024-04-02T14:52:43.353 INFO:journalctl@ceph.osd.3.smithi090.stdout:Apr 02 14:52:43 smithi090 podman[110951]: 2024-04-02 14:52:42.971164228 +0000 UTC m=+0.036830505 image pull f7301e070545fc23d8e5b7b052a2c15e832a0e57102f1acad8e4f3583709d23f quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534 2024-04-02T14:52:43.354 INFO:journalctl@ceph.osd.3.smithi090.stdout:Apr 02 14:52:43 smithi090 podman[110951]: 2024-04-02 14:52:43.08275646 +0000 UTC m=+0.148422722 container create 3a5e27d1eff983090059b567a405b7bcd69554ba056a4b71d103ee9385c6a90d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-3-deactivate, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240326, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240402.004512-191d963, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-04-02T14:52:43.354 INFO:journalctl@ceph.osd.3.smithi090.stdout:Apr 02 14:52:43 smithi090 podman[110951]: 2024-04-02 14:52:43.361511767 +0000 UTC m=+0.427178036 container init 3a5e27d1eff983090059b567a405b7bcd69554ba056a4b71d103ee9385c6a90d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-3-deactivate, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240326, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240402.004512-191d963, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-04-02T14:52:43.427 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:52:42 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:52:42.979+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:50.397138+0000 front 2024-04-02T14:51:50.397096+0000 (oldest deadline 2024-04-02T14:52:37.200213+0000) 2024-04-02T14:52:43.646 INFO:journalctl@ceph.osd.3.smithi090.stdout:Apr 02 14:52:43 smithi090 podman[110951]: 2024-04-02 14:52:43.373811469 +0000 UTC m=+0.439477723 container start 3a5e27d1eff983090059b567a405b7bcd69554ba056a4b71d103ee9385c6a90d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-3-deactivate, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240326, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=wip-batrick-testing-20240402.004512-191d963, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-02T14:52:43.646 INFO:journalctl@ceph.osd.3.smithi090.stdout:Apr 02 14:52:43 smithi090 podman[110951]: 2024-04-02 14:52:43.435411365 +0000 UTC m=+0.501077615 container attach 3a5e27d1eff983090059b567a405b7bcd69554ba056a4b71d103ee9385c6a90d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-3-deactivate, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240326, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240402.004512-191d963, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-02T14:52:43.927 INFO:journalctl@ceph.osd.5.smithi177.stdout:Apr 02 14:52:43 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-5[48118]: 2024-04-02T14:52:43.438+0000 7f34e2cdd700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:50.261330+0000 front 2024-04-02T14:51:50.261317+0000 (oldest deadline 2024-04-02T14:52:37.664335+0000) 2024-04-02T14:52:43.929 INFO:journalctl@ceph.osd.6.smithi090.stdout:Apr 02 14:52:43 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-6[59804]: 2024-04-02T14:52:43.770+0000 7fab55400700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:47.904942+0000 front 2024-04-02T14:51:47.905629+0000 (oldest deadline 2024-04-02T14:52:40.210630+0000) 2024-04-02T14:52:43.929 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:52:43 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:52:43.654+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:49.133761+0000 front 2024-04-02T14:51:49.134172+0000 (oldest deadline 2024-04-02T14:52:35.841983+0000) 2024-04-02T14:52:43.929 INFO:journalctl@ceph.osd.3.smithi090.stdout:Apr 02 14:52:43 smithi090 podman[110951]: 2024-04-02 14:52:43.813210469 +0000 UTC m=+0.878876710 container died 3a5e27d1eff983090059b567a405b7bcd69554ba056a4b71d103ee9385c6a90d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-3-deactivate, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240402.004512-191d963, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240326, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-04-02T14:52:44.101 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:52:43 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:52:43.852+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:50.925796+0000 front 2024-04-02T14:51:50.925668+0000 (oldest deadline 2024-04-02T14:52:40.128984+0000) 2024-04-02T14:52:44.427 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:52:44 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:52:44.006+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:50.397138+0000 front 2024-04-02T14:51:50.397096+0000 (oldest deadline 2024-04-02T14:52:37.200213+0000) 2024-04-02T14:52:44.462 DEBUG:teuthology.orchestra.run.smithi090:> sudo pkill -f 'journalctl -f -n 0 -u ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4@osd.3.service' 2024-04-02T14:52:44.584 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-02T14:52:44.584 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-04-02T14:52:44.584 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-04-02T14:52:44.584 DEBUG:teuthology.orchestra.run.smithi092:> sudo systemctl stop ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4@osd.4 2024-04-02T14:52:44.679 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:52:44 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:52:44.616+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:49.133761+0000 front 2024-04-02T14:51:49.134172+0000 (oldest deadline 2024-04-02T14:52:35.841983+0000) 2024-04-02T14:52:44.884 INFO:journalctl@ceph.osd.4.smithi092.stdout:Apr 02 14:52:44 smithi092 systemd[1]: Stopping Ceph osd.4 for 6449448a-f0fc-11ee-b647-cb9ed24678a4... 2024-04-02T14:52:44.927 INFO:journalctl@ceph.osd.5.smithi177.stdout:Apr 02 14:52:44 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-5[48118]: 2024-04-02T14:52:44.444+0000 7f34e2cdd700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:50.261330+0000 front 2024-04-02T14:51:50.261317+0000 (oldest deadline 2024-04-02T14:52:37.664335+0000) 2024-04-02T14:52:45.179 INFO:journalctl@ceph.osd.6.smithi090.stdout:Apr 02 14:52:44 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-6[59804]: 2024-04-02T14:52:44.745+0000 7fab55400700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:47.904942+0000 front 2024-04-02T14:51:47.905629+0000 (oldest deadline 2024-04-02T14:52:40.210630+0000) 2024-04-02T14:52:45.351 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:52:44 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:52:44.891+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:50.925796+0000 front 2024-04-02T14:51:50.925668+0000 (oldest deadline 2024-04-02T14:52:40.128984+0000) 2024-04-02T14:52:45.351 INFO:journalctl@ceph.osd.4.smithi092.stdout:Apr 02 14:52:44 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-4[47625]: 2024-04-02T14:52:44.893+0000 7f30777ba700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-02T14:52:45.351 INFO:journalctl@ceph.osd.4.smithi092.stdout:Apr 02 14:52:44 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-4[47625]: 2024-04-02T14:52:44.893+0000 7f30777ba700 -1 osd.4 114 *** Got signal Terminated *** 2024-04-02T14:52:45.351 INFO:journalctl@ceph.osd.4.smithi092.stdout:Apr 02 14:52:44 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-4[47625]: 2024-04-02T14:52:44.893+0000 7f30777ba700 -1 osd.4 114 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-02T14:52:45.427 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:52:44 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:52:44.973+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:50.397138+0000 front 2024-04-02T14:51:50.397096+0000 (oldest deadline 2024-04-02T14:52:37.200213+0000) 2024-04-02T14:52:45.927 INFO:journalctl@ceph.osd.5.smithi177.stdout:Apr 02 14:52:45 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-5[48118]: 2024-04-02T14:52:45.488+0000 7f34e2cdd700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:50.261330+0000 front 2024-04-02T14:51:50.261317+0000 (oldest deadline 2024-04-02T14:52:37.664335+0000) 2024-04-02T14:52:45.929 INFO:journalctl@ceph.osd.6.smithi090.stdout:Apr 02 14:52:45 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-6[59804]: 2024-04-02T14:52:45.736+0000 7fab55400700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:47.904942+0000 front 2024-04-02T14:51:47.905629+0000 (oldest deadline 2024-04-02T14:52:40.210630+0000) 2024-04-02T14:52:45.929 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:52:45 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:52:45.645+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:49.133761+0000 front 2024-04-02T14:51:49.134172+0000 (oldest deadline 2024-04-02T14:52:35.841983+0000) 2024-04-02T14:52:46.101 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:52:45 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:52:45.849+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:50.925796+0000 front 2024-04-02T14:51:50.925668+0000 (oldest deadline 2024-04-02T14:52:40.128984+0000) 2024-04-02T14:52:46.101 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:52:45 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:52:45.849+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:20.129325+0000 front 2024-04-02T14:52:20.129293+0000 (oldest deadline 2024-04-02T14:52:45.429431+0000) 2024-04-02T14:52:46.101 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 02 14:52:45 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-7[54644]: 2024-04-02T14:52:45.920+0000 7f0bb6c43700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:22.855156+0000 front 2024-04-02T14:52:22.855176+0000 (oldest deadline 2024-04-02T14:52:45.755296+0000) 2024-04-02T14:52:46.427 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:52:45 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:52:45.983+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:50.397138+0000 front 2024-04-02T14:51:50.397096+0000 (oldest deadline 2024-04-02T14:52:37.200213+0000) 2024-04-02T14:52:46.427 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:52:45 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:52:45.983+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:22.501322+0000 front 2024-04-02T14:52:22.501346+0000 (oldest deadline 2024-04-02T14:52:45.401451+0000) 2024-04-02T14:52:46.927 INFO:journalctl@ceph.osd.5.smithi177.stdout:Apr 02 14:52:46 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-5[48118]: 2024-04-02T14:52:46.458+0000 7f34e2cdd700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:50.261330+0000 front 2024-04-02T14:51:50.261317+0000 (oldest deadline 2024-04-02T14:52:37.664335+0000) 2024-04-02T14:52:46.927 INFO:journalctl@ceph.osd.5.smithi177.stdout:Apr 02 14:52:46 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-5[48118]: 2024-04-02T14:52:46.458+0000 7f34e2cdd700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:20.466010+0000 front 2024-04-02T14:52:20.466157+0000 (oldest deadline 2024-04-02T14:52:46.366121+0000) 2024-04-02T14:52:46.929 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:52:46 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:52:46.615+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:49.133761+0000 front 2024-04-02T14:51:49.134172+0000 (oldest deadline 2024-04-02T14:52:35.841983+0000) 2024-04-02T14:52:46.929 INFO:journalctl@ceph.osd.6.smithi090.stdout:Apr 02 14:52:46 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-6[59804]: 2024-04-02T14:52:46.703+0000 7fab55400700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:47.904942+0000 front 2024-04-02T14:51:47.905629+0000 (oldest deadline 2024-04-02T14:52:40.210630+0000) 2024-04-02T14:52:47.101 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 02 14:52:46 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-7[54644]: 2024-04-02T14:52:46.887+0000 7f0bb6c43700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:22.855156+0000 front 2024-04-02T14:52:22.855176+0000 (oldest deadline 2024-04-02T14:52:45.755296+0000) 2024-04-02T14:52:47.101 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:52:46 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:52:46.857+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:50.925796+0000 front 2024-04-02T14:51:50.925668+0000 (oldest deadline 2024-04-02T14:52:40.128984+0000) 2024-04-02T14:52:47.101 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:52:46 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:52:46.857+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:20.129325+0000 front 2024-04-02T14:52:20.129293+0000 (oldest deadline 2024-04-02T14:52:45.429431+0000) 2024-04-02T14:52:47.420 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:52:46 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:52:46.960+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:50.397138+0000 front 2024-04-02T14:51:50.397096+0000 (oldest deadline 2024-04-02T14:52:37.200213+0000) 2024-04-02T14:52:47.420 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:52:46 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:52:46.960+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:22.501322+0000 front 2024-04-02T14:52:22.501346+0000 (oldest deadline 2024-04-02T14:52:45.401451+0000) 2024-04-02T14:52:47.677 INFO:journalctl@ceph.osd.5.smithi177.stdout:Apr 02 14:52:47 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-5[48118]: 2024-04-02T14:52:47.428+0000 7f34e2cdd700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:50.261330+0000 front 2024-04-02T14:51:50.261317+0000 (oldest deadline 2024-04-02T14:52:37.664335+0000) 2024-04-02T14:52:47.677 INFO:journalctl@ceph.osd.5.smithi177.stdout:Apr 02 14:52:47 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-5[48118]: 2024-04-02T14:52:47.428+0000 7f34e2cdd700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:20.466010+0000 front 2024-04-02T14:52:20.466157+0000 (oldest deadline 2024-04-02T14:52:46.366121+0000) 2024-04-02T14:52:47.680 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:52:47 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:52:47.589+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:49.133761+0000 front 2024-04-02T14:51:49.134172+0000 (oldest deadline 2024-04-02T14:52:35.841983+0000) 2024-04-02T14:52:47.680 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:52:47 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:52:47.589+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:21.645685+0000 front 2024-04-02T14:52:21.645805+0000 (oldest deadline 2024-04-02T14:52:46.945889+0000) 2024-04-02T14:52:48.004 INFO:journalctl@ceph.osd.6.smithi090.stdout:Apr 02 14:52:47 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-6[59804]: 2024-04-02T14:52:47.689+0000 7fab55400700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:47.904942+0000 front 2024-04-02T14:51:47.905629+0000 (oldest deadline 2024-04-02T14:52:40.210630+0000) 2024-04-02T14:52:48.351 INFO:journalctl@ceph.osd.4.smithi092.stdout:Apr 02 14:52:48 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-4[47625]: 2024-04-02T14:52:48.084+0000 7f306fda6700 -1 osd.4 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:22.222182+0000 front 2024-04-02T14:52:22.222312+0000 (oldest deadline 2024-04-02T14:52:47.522372+0000) 2024-04-02T14:52:48.351 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 02 14:52:47 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-7[54644]: 2024-04-02T14:52:47.922+0000 7f0bb6c43700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:22.855156+0000 front 2024-04-02T14:52:22.855176+0000 (oldest deadline 2024-04-02T14:52:45.755296+0000) 2024-04-02T14:52:48.351 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:52:47 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:52:47.883+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:50.925796+0000 front 2024-04-02T14:51:50.925668+0000 (oldest deadline 2024-04-02T14:52:40.128984+0000) 2024-04-02T14:52:48.351 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:52:47 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:52:47.883+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:20.129325+0000 front 2024-04-02T14:52:20.129293+0000 (oldest deadline 2024-04-02T14:52:45.429431+0000) 2024-04-02T14:52:48.372 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:52:47 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:52:47.995+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:50.397138+0000 front 2024-04-02T14:51:50.397096+0000 (oldest deadline 2024-04-02T14:52:37.200213+0000) 2024-04-02T14:52:48.372 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:52:47 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:52:47.995+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:22.501322+0000 front 2024-04-02T14:52:22.501346+0000 (oldest deadline 2024-04-02T14:52:45.401451+0000) 2024-04-02T14:52:48.677 INFO:journalctl@ceph.osd.5.smithi177.stdout:Apr 02 14:52:48 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-5[48118]: 2024-04-02T14:52:48.380+0000 7f34e2cdd700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:50.261330+0000 front 2024-04-02T14:51:50.261317+0000 (oldest deadline 2024-04-02T14:52:37.664335+0000) 2024-04-02T14:52:48.677 INFO:journalctl@ceph.osd.5.smithi177.stdout:Apr 02 14:52:48 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-5[48118]: 2024-04-02T14:52:48.380+0000 7f34e2cdd700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:20.466010+0000 front 2024-04-02T14:52:20.466157+0000 (oldest deadline 2024-04-02T14:52:46.366121+0000) 2024-04-02T14:52:48.677 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 02 14:52:48 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-8[54680]: 2024-04-02T14:52:48.612+0000 7f64f23c8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:23.288870+0000 front 2024-04-02T14:52:23.288907+0000 (oldest deadline 2024-04-02T14:52:48.588938+0000) 2024-04-02T14:52:48.929 INFO:journalctl@ceph.osd.6.smithi090.stdout:Apr 02 14:52:48 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-6[59804]: 2024-04-02T14:52:48.682+0000 7fab55400700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:47.904942+0000 front 2024-04-02T14:51:47.905629+0000 (oldest deadline 2024-04-02T14:52:40.210630+0000) 2024-04-02T14:52:48.929 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:52:48 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:52:48.540+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:49.133761+0000 front 2024-04-02T14:51:49.134172+0000 (oldest deadline 2024-04-02T14:52:35.841983+0000) 2024-04-02T14:52:48.929 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:52:48 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:52:48.540+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:21.645685+0000 front 2024-04-02T14:52:21.645805+0000 (oldest deadline 2024-04-02T14:52:46.945889+0000) 2024-04-02T14:52:49.351 INFO:journalctl@ceph.osd.4.smithi092.stdout:Apr 02 14:52:49 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-4[47625]: 2024-04-02T14:52:49.072+0000 7f306fda6700 -1 osd.4 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:22.222182+0000 front 2024-04-02T14:52:22.222312+0000 (oldest deadline 2024-04-02T14:52:47.522372+0000) 2024-04-02T14:52:49.351 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 02 14:52:48 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-7[54644]: 2024-04-02T14:52:48.962+0000 7f0bb6c43700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:22.855156+0000 front 2024-04-02T14:52:22.855176+0000 (oldest deadline 2024-04-02T14:52:45.755296+0000) 2024-04-02T14:52:49.351 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:52:48 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:52:48.882+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:50.925796+0000 front 2024-04-02T14:51:50.925668+0000 (oldest deadline 2024-04-02T14:52:40.128984+0000) 2024-04-02T14:52:49.351 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:52:48 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:52:48.882+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:20.129325+0000 front 2024-04-02T14:52:20.129293+0000 (oldest deadline 2024-04-02T14:52:45.429431+0000) 2024-04-02T14:52:49.355 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:52:48 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:52:48.990+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:50.397138+0000 front 2024-04-02T14:51:50.397096+0000 (oldest deadline 2024-04-02T14:52:37.200213+0000) 2024-04-02T14:52:49.355 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:52:48 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:52:48.990+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:22.501322+0000 front 2024-04-02T14:52:22.501346+0000 (oldest deadline 2024-04-02T14:52:45.401451+0000) 2024-04-02T14:52:49.640 INFO:journalctl@ceph.osd.5.smithi177.stdout:Apr 02 14:52:49 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-5[48118]: 2024-04-02T14:52:49.363+0000 7f34e2cdd700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:50.261330+0000 front 2024-04-02T14:51:50.261317+0000 (oldest deadline 2024-04-02T14:52:37.664335+0000) 2024-04-02T14:52:49.641 INFO:journalctl@ceph.osd.5.smithi177.stdout:Apr 02 14:52:49 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-5[48118]: 2024-04-02T14:52:49.363+0000 7f34e2cdd700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:20.466010+0000 front 2024-04-02T14:52:20.466157+0000 (oldest deadline 2024-04-02T14:52:46.366121+0000) 2024-04-02T14:52:49.927 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 02 14:52:49 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-8[54680]: 2024-04-02T14:52:49.648+0000 7f64f23c8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:23.288870+0000 front 2024-04-02T14:52:23.288907+0000 (oldest deadline 2024-04-02T14:52:48.588938+0000) 2024-04-02T14:52:49.929 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:52:49 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:52:49.512+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:49.133761+0000 front 2024-04-02T14:51:49.134172+0000 (oldest deadline 2024-04-02T14:52:35.841983+0000) 2024-04-02T14:52:49.929 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:52:49 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:52:49.512+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:21.645685+0000 front 2024-04-02T14:52:21.645805+0000 (oldest deadline 2024-04-02T14:52:46.945889+0000) 2024-04-02T14:52:49.929 INFO:journalctl@ceph.osd.6.smithi090.stdout:Apr 02 14:52:49 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-6[59804]: 2024-04-02T14:52:49.660+0000 7fab55400700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:47.904942+0000 front 2024-04-02T14:51:47.905629+0000 (oldest deadline 2024-04-02T14:52:40.210630+0000) 2024-04-02T14:52:49.929 INFO:journalctl@ceph.osd.6.smithi090.stdout:Apr 02 14:52:49 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-6[59804]: 2024-04-02T14:52:49.660+0000 7fab55400700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:20.211746+0000 front 2024-04-02T14:52:20.211924+0000 (oldest deadline 2024-04-02T14:52:49.613199+0000) 2024-04-02T14:52:50.351 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 02 14:52:49 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-7[54644]: 2024-04-02T14:52:49.959+0000 7f0bb6c43700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:22.855156+0000 front 2024-04-02T14:52:22.855176+0000 (oldest deadline 2024-04-02T14:52:45.755296+0000) 2024-04-02T14:52:50.351 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:52:49 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:52:49.882+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:50.925796+0000 front 2024-04-02T14:51:50.925668+0000 (oldest deadline 2024-04-02T14:52:40.128984+0000) 2024-04-02T14:52:50.351 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:52:49 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:52:49.882+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:20.129325+0000 front 2024-04-02T14:52:20.129293+0000 (oldest deadline 2024-04-02T14:52:45.429431+0000) 2024-04-02T14:52:50.376 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:52:50 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:52:50.028+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:50.397138+0000 front 2024-04-02T14:51:50.397096+0000 (oldest deadline 2024-04-02T14:52:37.200213+0000) 2024-04-02T14:52:50.377 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:52:50 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:52:50.028+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:22.501322+0000 front 2024-04-02T14:52:22.501346+0000 (oldest deadline 2024-04-02T14:52:45.401451+0000) 2024-04-02T14:52:50.636 INFO:journalctl@ceph.osd.4.smithi092.stdout:Apr 02 14:52:50 smithi092 podman[74129]: 2024-04-02 14:52:50.387037862 +0000 UTC m=+5.600678676 container died 3f4278c9110f3c4d0c792b4f2fd580d85f24f09476610c52fa6355cdb8b632f2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-4, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240326, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240402.004512-191d963, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-02T14:52:50.653 INFO:journalctl@ceph.osd.5.smithi177.stdout:Apr 02 14:52:50 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-5[48118]: 2024-04-02T14:52:50.384+0000 7f34e2cdd700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:50.261330+0000 front 2024-04-02T14:51:50.261317+0000 (oldest deadline 2024-04-02T14:52:37.664335+0000) 2024-04-02T14:52:50.654 INFO:journalctl@ceph.osd.5.smithi177.stdout:Apr 02 14:52:50 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-5[48118]: 2024-04-02T14:52:50.384+0000 7f34e2cdd700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:20.466010+0000 front 2024-04-02T14:52:20.466157+0000 (oldest deadline 2024-04-02T14:52:46.366121+0000) 2024-04-02T14:52:50.927 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 02 14:52:50 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-8[54680]: 2024-04-02T14:52:50.662+0000 7f64f23c8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:23.288870+0000 front 2024-04-02T14:52:23.288907+0000 (oldest deadline 2024-04-02T14:52:48.588938+0000) 2024-04-02T14:52:50.928 INFO:journalctl@ceph.osd.4.smithi092.stdout:Apr 02 14:52:50 smithi092 podman[74129]: 2024-04-02 14:52:50.735993005 +0000 UTC m=+5.949633816 container cleanup 3f4278c9110f3c4d0c792b4f2fd580d85f24f09476610c52fa6355cdb8b632f2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-4, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240326, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=wip-batrick-testing-20240402.004512-191d963, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-04-02T14:52:50.928 INFO:journalctl@ceph.osd.4.smithi092.stdout:Apr 02 14:52:50 smithi092 bash[74129]: ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-4 2024-04-02T14:52:50.928 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:52:50 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:52:50.861+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:50.925796+0000 front 2024-04-02T14:51:50.925668+0000 (oldest deadline 2024-04-02T14:52:40.128984+0000) 2024-04-02T14:52:50.928 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:52:50 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:52:50.861+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:20.129325+0000 front 2024-04-02T14:52:20.129293+0000 (oldest deadline 2024-04-02T14:52:45.429431+0000) 2024-04-02T14:52:50.929 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:52:50 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:52:50.474+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:49.133761+0000 front 2024-04-02T14:51:49.134172+0000 (oldest deadline 2024-04-02T14:52:35.841983+0000) 2024-04-02T14:52:50.929 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:52:50 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:52:50.474+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:21.645685+0000 front 2024-04-02T14:52:21.645805+0000 (oldest deadline 2024-04-02T14:52:46.945889+0000) 2024-04-02T14:52:50.929 INFO:journalctl@ceph.osd.6.smithi090.stdout:Apr 02 14:52:50 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-6[59804]: 2024-04-02T14:52:50.653+0000 7fab55400700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:47.904942+0000 front 2024-04-02T14:51:47.905629+0000 (oldest deadline 2024-04-02T14:52:40.210630+0000) 2024-04-02T14:52:50.929 INFO:journalctl@ceph.osd.6.smithi090.stdout:Apr 02 14:52:50 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-6[59804]: 2024-04-02T14:52:50.653+0000 7fab55400700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:20.211746+0000 front 2024-04-02T14:52:20.211924+0000 (oldest deadline 2024-04-02T14:52:49.613199+0000) 2024-04-02T14:52:51.331 INFO:journalctl@ceph.osd.4.smithi092.stdout:Apr 02 14:52:51 smithi092 podman[74146]: 2024-04-02 14:52:51.029202632 +0000 UTC m=+0.647101749 container remove 3f4278c9110f3c4d0c792b4f2fd580d85f24f09476610c52fa6355cdb8b632f2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-4, RELEASE=wip-batrick-testing-20240402.004512-191d963, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240326, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-04-02T14:52:51.331 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 02 14:52:50 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-7[54644]: 2024-04-02T14:52:50.936+0000 7f0bb6c43700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:22.855156+0000 front 2024-04-02T14:52:22.855176+0000 (oldest deadline 2024-04-02T14:52:45.755296+0000) 2024-04-02T14:52:51.382 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:52:51 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:52:51.043+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:50.397138+0000 front 2024-04-02T14:51:50.397096+0000 (oldest deadline 2024-04-02T14:52:37.200213+0000) 2024-04-02T14:52:51.382 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:52:51 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:52:51.043+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:22.501322+0000 front 2024-04-02T14:52:22.501346+0000 (oldest deadline 2024-04-02T14:52:45.401451+0000) 2024-04-02T14:52:51.601 INFO:journalctl@ceph.osd.4.smithi092.stdout:Apr 02 14:52:51 smithi092 podman[74241]: 2024-04-02 14:52:51.243017427 +0000 UTC m=+0.017839618 image pull f7301e070545fc23d8e5b7b052a2c15e832a0e57102f1acad8e4f3583709d23f quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534 2024-04-02T14:52:51.601 INFO:journalctl@ceph.osd.4.smithi092.stdout:Apr 02 14:52:51 smithi092 podman[74241]: 2024-04-02 14:52:51.371854395 +0000 UTC m=+0.146676628 container create 83eb5998c8d4e860dd70dd8034371853495d4776a2c7427b1a58beb2a7f316aa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-4-deactivate, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240402.004512-191d963, org.label-schema.build-date=20240326, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-04-02T14:52:51.601 INFO:journalctl@ceph.osd.4.smithi092.stdout:Apr 02 14:52:51 smithi092 podman[74241]: 2024-04-02 14:52:51.565978987 +0000 UTC m=+0.340801177 container init 83eb5998c8d4e860dd70dd8034371853495d4776a2c7427b1a58beb2a7f316aa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-4-deactivate, RELEASE=wip-batrick-testing-20240402.004512-191d963, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240326, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-04-02T14:52:51.601 INFO:journalctl@ceph.osd.4.smithi092.stdout:Apr 02 14:52:51 smithi092 podman[74241]: 2024-04-02 14:52:51.569003642 +0000 UTC m=+0.343825839 container start 83eb5998c8d4e860dd70dd8034371853495d4776a2c7427b1a58beb2a7f316aa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-4-deactivate, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240402.004512-191d963, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240326, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-04-02T14:52:51.677 INFO:journalctl@ceph.osd.5.smithi177.stdout:Apr 02 14:52:51 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-5[48118]: 2024-04-02T14:52:51.390+0000 7f34e2cdd700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:50.261330+0000 front 2024-04-02T14:51:50.261317+0000 (oldest deadline 2024-04-02T14:52:37.664335+0000) 2024-04-02T14:52:51.677 INFO:journalctl@ceph.osd.5.smithi177.stdout:Apr 02 14:52:51 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-5[48118]: 2024-04-02T14:52:51.390+0000 7f34e2cdd700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:20.466010+0000 front 2024-04-02T14:52:20.466157+0000 (oldest deadline 2024-04-02T14:52:46.366121+0000) 2024-04-02T14:52:51.927 INFO:journalctl@ceph.osd.4.smithi092.stdout:Apr 02 14:52:51 smithi092 podman[74241]: 2024-04-02 14:52:51.640913088 +0000 UTC m=+0.415735292 container attach 83eb5998c8d4e860dd70dd8034371853495d4776a2c7427b1a58beb2a7f316aa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-4-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240326, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=wip-batrick-testing-20240402.004512-191d963, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-04-02T14:52:51.927 INFO:journalctl@ceph.osd.4.smithi092.stdout:Apr 02 14:52:51 smithi092 podman[74241]: 2024-04-02 14:52:51.857641703 +0000 UTC m=+0.632463894 container died 83eb5998c8d4e860dd70dd8034371853495d4776a2c7427b1a58beb2a7f316aa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-4-deactivate, RELEASE=wip-batrick-testing-20240402.004512-191d963, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240326, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-04-02T14:52:51.928 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 02 14:52:51 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-7[54644]: 2024-04-02T14:52:51.934+0000 7f0bb6c43700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:22.855156+0000 front 2024-04-02T14:52:22.855176+0000 (oldest deadline 2024-04-02T14:52:45.755296+0000) 2024-04-02T14:52:51.928 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:52:51 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:52:51.817+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:50.925796+0000 front 2024-04-02T14:51:50.925668+0000 (oldest deadline 2024-04-02T14:52:40.128984+0000) 2024-04-02T14:52:51.928 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:52:51 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:52:51.817+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:20.129325+0000 front 2024-04-02T14:52:20.129293+0000 (oldest deadline 2024-04-02T14:52:45.429431+0000) 2024-04-02T14:52:51.929 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:52:51 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:52:51.477+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:49.133761+0000 front 2024-04-02T14:51:49.134172+0000 (oldest deadline 2024-04-02T14:52:35.841983+0000) 2024-04-02T14:52:51.929 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:52:51 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:52:51.477+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:21.645685+0000 front 2024-04-02T14:52:21.645805+0000 (oldest deadline 2024-04-02T14:52:46.945889+0000) 2024-04-02T14:52:51.930 INFO:journalctl@ceph.osd.6.smithi090.stdout:Apr 02 14:52:51 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-6[59804]: 2024-04-02T14:52:51.703+0000 7fab55400700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:47.904942+0000 front 2024-04-02T14:51:47.905629+0000 (oldest deadline 2024-04-02T14:52:40.210630+0000) 2024-04-02T14:52:51.930 INFO:journalctl@ceph.osd.6.smithi090.stdout:Apr 02 14:52:51 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-6[59804]: 2024-04-02T14:52:51.703+0000 7fab55400700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:20.211746+0000 front 2024-04-02T14:52:20.211924+0000 (oldest deadline 2024-04-02T14:52:49.613199+0000) 2024-04-02T14:52:52.032 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 02 14:52:51 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-8[54680]: 2024-04-02T14:52:51.701+0000 7f64f23c8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:23.288870+0000 front 2024-04-02T14:52:23.288907+0000 (oldest deadline 2024-04-02T14:52:48.588938+0000) 2024-04-02T14:52:52.402 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:52:52 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:52:52.040+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:50.397138+0000 front 2024-04-02T14:51:50.397096+0000 (oldest deadline 2024-04-02T14:52:37.200213+0000) 2024-04-02T14:52:52.402 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:52:52 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:52:52.040+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:22.501322+0000 front 2024-04-02T14:52:22.501346+0000 (oldest deadline 2024-04-02T14:52:45.401451+0000) 2024-04-02T14:52:52.410 DEBUG:teuthology.orchestra.run.smithi092:> sudo pkill -f 'journalctl -f -n 0 -u ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4@osd.4.service' 2024-04-02T14:52:52.449 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-02T14:52:52.449 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-04-02T14:52:52.449 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-04-02T14:52:52.449 DEBUG:teuthology.orchestra.run.smithi177:> sudo systemctl stop ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4@osd.5 2024-04-02T14:52:52.671 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 02 14:52:52 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-8[54680]: 2024-04-02T14:52:52.678+0000 7f64f23c8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:23.288870+0000 front 2024-04-02T14:52:23.288907+0000 (oldest deadline 2024-04-02T14:52:48.588938+0000) 2024-04-02T14:52:52.672 INFO:journalctl@ceph.osd.5.smithi177.stdout:Apr 02 14:52:52 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-5[48118]: 2024-04-02T14:52:52.409+0000 7f34e2cdd700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:50.261330+0000 front 2024-04-02T14:51:50.261317+0000 (oldest deadline 2024-04-02T14:52:37.664335+0000) 2024-04-02T14:52:52.672 INFO:journalctl@ceph.osd.5.smithi177.stdout:Apr 02 14:52:52 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-5[48118]: 2024-04-02T14:52:52.409+0000 7f34e2cdd700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:20.466010+0000 front 2024-04-02T14:52:20.466157+0000 (oldest deadline 2024-04-02T14:52:46.366121+0000) 2024-04-02T14:52:52.672 INFO:journalctl@ceph.osd.5.smithi177.stdout:Apr 02 14:52:52 smithi177 systemd[1]: Stopping Ceph osd.5 for 6449448a-f0fc-11ee-b647-cb9ed24678a4... 2024-04-02T14:52:52.679 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:52:52 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:52:52.428+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:49.133761+0000 front 2024-04-02T14:51:49.134172+0000 (oldest deadline 2024-04-02T14:52:35.841983+0000) 2024-04-02T14:52:52.679 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:52:52 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:52:52.428+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:21.645685+0000 front 2024-04-02T14:52:21.645805+0000 (oldest deadline 2024-04-02T14:52:46.945889+0000) 2024-04-02T14:52:52.927 INFO:journalctl@ceph.osd.5.smithi177.stdout:Apr 02 14:52:52 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-5[48118]: 2024-04-02T14:52:52.769+0000 7f34e66e9700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-02T14:52:52.927 INFO:journalctl@ceph.osd.5.smithi177.stdout:Apr 02 14:52:52 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-5[48118]: 2024-04-02T14:52:52.769+0000 7f34e66e9700 -1 osd.5 114 *** Got signal Terminated *** 2024-04-02T14:52:52.927 INFO:journalctl@ceph.osd.5.smithi177.stdout:Apr 02 14:52:52 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-5[48118]: 2024-04-02T14:52:52.769+0000 7f34e66e9700 -1 osd.5 114 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-02T14:52:53.179 INFO:journalctl@ceph.osd.6.smithi090.stdout:Apr 02 14:52:52 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-6[59804]: 2024-04-02T14:52:52.696+0000 7fab55400700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:47.904942+0000 front 2024-04-02T14:51:47.905629+0000 (oldest deadline 2024-04-02T14:52:40.210630+0000) 2024-04-02T14:52:53.179 INFO:journalctl@ceph.osd.6.smithi090.stdout:Apr 02 14:52:52 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-6[59804]: 2024-04-02T14:52:52.696+0000 7fab55400700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:20.211746+0000 front 2024-04-02T14:52:20.211924+0000 (oldest deadline 2024-04-02T14:52:49.613199+0000) 2024-04-02T14:52:53.351 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 02 14:52:52 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-7[54644]: 2024-04-02T14:52:52.972+0000 7f0bb6c43700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:22.855156+0000 front 2024-04-02T14:52:22.855176+0000 (oldest deadline 2024-04-02T14:52:45.755296+0000) 2024-04-02T14:52:53.351 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:52:52 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:52:52.867+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:50.925796+0000 front 2024-04-02T14:51:50.925668+0000 (oldest deadline 2024-04-02T14:52:40.128984+0000) 2024-04-02T14:52:53.351 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:52:52 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:52:52.867+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:20.129325+0000 front 2024-04-02T14:52:20.129293+0000 (oldest deadline 2024-04-02T14:52:45.429431+0000) 2024-04-02T14:52:53.396 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:52:53 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:52:53.056+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:50.397138+0000 front 2024-04-02T14:51:50.397096+0000 (oldest deadline 2024-04-02T14:52:37.200213+0000) 2024-04-02T14:52:53.396 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:52:53 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:52:53.056+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:22.501322+0000 front 2024-04-02T14:52:22.501346+0000 (oldest deadline 2024-04-02T14:52:45.401451+0000) 2024-04-02T14:52:53.677 INFO:journalctl@ceph.osd.5.smithi177.stdout:Apr 02 14:52:53 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-5[48118]: 2024-04-02T14:52:53.404+0000 7f34e2cdd700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:50.261330+0000 front 2024-04-02T14:51:50.261317+0000 (oldest deadline 2024-04-02T14:52:37.664335+0000) 2024-04-02T14:52:53.677 INFO:journalctl@ceph.osd.5.smithi177.stdout:Apr 02 14:52:53 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-5[48118]: 2024-04-02T14:52:53.404+0000 7f34e2cdd700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:20.466010+0000 front 2024-04-02T14:52:20.466157+0000 (oldest deadline 2024-04-02T14:52:46.366121+0000) 2024-04-02T14:52:53.677 INFO:journalctl@ceph.osd.5.smithi177.stdout:Apr 02 14:52:53 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-5[48118]: 2024-04-02T14:52:53.404+0000 7f34e2cdd700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:29.267006+0000 front 2024-04-02T14:52:29.267027+0000 (oldest deadline 2024-04-02T14:52:53.367088+0000) 2024-04-02T14:52:53.679 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:52:53 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:52:53.401+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:49.133761+0000 front 2024-04-02T14:51:49.134172+0000 (oldest deadline 2024-04-02T14:52:35.841983+0000) 2024-04-02T14:52:53.679 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:52:53 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:52:53.401+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:21.645685+0000 front 2024-04-02T14:52:21.645805+0000 (oldest deadline 2024-04-02T14:52:46.945889+0000) 2024-04-02T14:52:54.008 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 02 14:52:53 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-8[54680]: 2024-04-02T14:52:53.691+0000 7f64f23c8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:23.288870+0000 front 2024-04-02T14:52:23.288907+0000 (oldest deadline 2024-04-02T14:52:48.588938+0000) 2024-04-02T14:52:54.008 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 02 14:52:53 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-8[54680]: 2024-04-02T14:52:53.691+0000 7f64f23c8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:29.090126+0000 front 2024-04-02T14:52:29.090177+0000 (oldest deadline 2024-04-02T14:52:53.190215+0000) 2024-04-02T14:52:54.179 INFO:journalctl@ceph.osd.6.smithi090.stdout:Apr 02 14:52:53 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-6[59804]: 2024-04-02T14:52:53.701+0000 7fab55400700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:47.904942+0000 front 2024-04-02T14:51:47.905629+0000 (oldest deadline 2024-04-02T14:52:40.210630+0000) 2024-04-02T14:52:54.179 INFO:journalctl@ceph.osd.6.smithi090.stdout:Apr 02 14:52:53 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-6[59804]: 2024-04-02T14:52:53.701+0000 7fab55400700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:20.211746+0000 front 2024-04-02T14:52:20.211924+0000 (oldest deadline 2024-04-02T14:52:49.613199+0000) 2024-04-02T14:52:54.351 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 02 14:52:53 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-7[54644]: 2024-04-02T14:52:53.971+0000 7f0bb6c43700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:22.855156+0000 front 2024-04-02T14:52:22.855176+0000 (oldest deadline 2024-04-02T14:52:45.755296+0000) 2024-04-02T14:52:54.351 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 02 14:52:53 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-7[54644]: 2024-04-02T14:52:53.971+0000 7f0bb6c43700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:32.156922+0000 front 2024-04-02T14:52:32.157038+0000 (oldest deadline 2024-04-02T14:52:53.856857+0000) 2024-04-02T14:52:54.351 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:52:53 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:52:53.913+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:50.925796+0000 front 2024-04-02T14:51:50.925668+0000 (oldest deadline 2024-04-02T14:52:40.128984+0000) 2024-04-02T14:52:54.352 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:52:53 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:52:53.913+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:20.129325+0000 front 2024-04-02T14:52:20.129293+0000 (oldest deadline 2024-04-02T14:52:45.429431+0000) 2024-04-02T14:52:54.352 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:52:53 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:52:53.913+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:27.631081+0000 front 2024-04-02T14:52:27.631094+0000 (oldest deadline 2024-04-02T14:52:53.531189+0000) 2024-04-02T14:52:54.410 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:52:54 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:52:54.016+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:50.397138+0000 front 2024-04-02T14:51:50.397096+0000 (oldest deadline 2024-04-02T14:52:37.200213+0000) 2024-04-02T14:52:54.410 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:52:54 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:52:54.016+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:22.501322+0000 front 2024-04-02T14:52:22.501346+0000 (oldest deadline 2024-04-02T14:52:45.401451+0000) 2024-04-02T14:52:54.675 INFO:journalctl@ceph.osd.5.smithi177.stdout:Apr 02 14:52:54 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-5[48118]: 2024-04-02T14:52:54.418+0000 7f34e2cdd700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:50.261330+0000 front 2024-04-02T14:51:50.261317+0000 (oldest deadline 2024-04-02T14:52:37.664335+0000) 2024-04-02T14:52:54.675 INFO:journalctl@ceph.osd.5.smithi177.stdout:Apr 02 14:52:54 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-5[48118]: 2024-04-02T14:52:54.418+0000 7f34e2cdd700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:20.466010+0000 front 2024-04-02T14:52:20.466157+0000 (oldest deadline 2024-04-02T14:52:46.366121+0000) 2024-04-02T14:52:54.675 INFO:journalctl@ceph.osd.5.smithi177.stdout:Apr 02 14:52:54 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-5[48118]: 2024-04-02T14:52:54.418+0000 7f34e2cdd700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:29.267006+0000 front 2024-04-02T14:52:29.267027+0000 (oldest deadline 2024-04-02T14:52:53.367088+0000) 2024-04-02T14:52:54.679 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:52:54 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:52:54.418+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:49.133761+0000 front 2024-04-02T14:51:49.134172+0000 (oldest deadline 2024-04-02T14:52:35.841983+0000) 2024-04-02T14:52:54.679 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:52:54 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:52:54.418+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:21.645685+0000 front 2024-04-02T14:52:21.645805+0000 (oldest deadline 2024-04-02T14:52:46.945889+0000) 2024-04-02T14:52:54.927 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 02 14:52:54 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-8[54680]: 2024-04-02T14:52:54.683+0000 7f64f23c8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:23.288870+0000 front 2024-04-02T14:52:23.288907+0000 (oldest deadline 2024-04-02T14:52:48.588938+0000) 2024-04-02T14:52:54.927 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 02 14:52:54 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-8[54680]: 2024-04-02T14:52:54.683+0000 7f64f23c8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:29.090126+0000 front 2024-04-02T14:52:29.090177+0000 (oldest deadline 2024-04-02T14:52:53.190215+0000) 2024-04-02T14:52:55.179 INFO:journalctl@ceph.osd.6.smithi090.stdout:Apr 02 14:52:54 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-6[59804]: 2024-04-02T14:52:54.714+0000 7fab55400700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:47.904942+0000 front 2024-04-02T14:51:47.905629+0000 (oldest deadline 2024-04-02T14:52:40.210630+0000) 2024-04-02T14:52:55.179 INFO:journalctl@ceph.osd.6.smithi090.stdout:Apr 02 14:52:54 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-6[59804]: 2024-04-02T14:52:54.714+0000 7fab55400700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:20.211746+0000 front 2024-04-02T14:52:20.211924+0000 (oldest deadline 2024-04-02T14:52:49.613199+0000) 2024-04-02T14:52:55.179 INFO:journalctl@ceph.osd.6.smithi090.stdout:Apr 02 14:52:54 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-6[59804]: 2024-04-02T14:52:54.714+0000 7fab55400700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:29.614242+0000 front 2024-04-02T14:52:29.614310+0000 (oldest deadline 2024-04-02T14:52:53.714498+0000) 2024-04-02T14:52:55.351 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 02 14:52:55 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-7[54644]: 2024-04-02T14:52:55.021+0000 7f0bb6c43700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:22.855156+0000 front 2024-04-02T14:52:22.855176+0000 (oldest deadline 2024-04-02T14:52:45.755296+0000) 2024-04-02T14:52:55.351 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 02 14:52:55 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-7[54644]: 2024-04-02T14:52:55.021+0000 7f0bb6c43700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:32.156922+0000 front 2024-04-02T14:52:32.157038+0000 (oldest deadline 2024-04-02T14:52:53.856857+0000) 2024-04-02T14:52:55.351 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:52:54 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:52:54.868+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:50.925796+0000 front 2024-04-02T14:51:50.925668+0000 (oldest deadline 2024-04-02T14:52:40.128984+0000) 2024-04-02T14:52:55.351 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:52:54 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:52:54.868+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:20.129325+0000 front 2024-04-02T14:52:20.129293+0000 (oldest deadline 2024-04-02T14:52:45.429431+0000) 2024-04-02T14:52:55.351 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:52:54 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:52:54.868+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:27.631081+0000 front 2024-04-02T14:52:27.631094+0000 (oldest deadline 2024-04-02T14:52:53.531189+0000) 2024-04-02T14:52:55.427 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:52:54 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:52:54.966+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:50.397138+0000 front 2024-04-02T14:51:50.397096+0000 (oldest deadline 2024-04-02T14:52:37.200213+0000) 2024-04-02T14:52:55.427 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:52:54 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:52:54.966+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:22.501322+0000 front 2024-04-02T14:52:22.501346+0000 (oldest deadline 2024-04-02T14:52:45.401451+0000) 2024-04-02T14:52:55.673 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:52:55 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:52:55.379+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:49.133761+0000 front 2024-04-02T14:51:49.134172+0000 (oldest deadline 2024-04-02T14:52:35.841983+0000) 2024-04-02T14:52:55.673 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:52:55 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:52:55.379+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:21.645685+0000 front 2024-04-02T14:52:21.645805+0000 (oldest deadline 2024-04-02T14:52:46.945889+0000) 2024-04-02T14:52:55.673 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:52:55 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:52:55.379+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:31.047892+0000 front 2024-04-02T14:52:31.047845+0000 (oldest deadline 2024-04-02T14:52:55.148007+0000) 2024-04-02T14:52:55.693 INFO:journalctl@ceph.osd.5.smithi177.stdout:Apr 02 14:52:55 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-5[48118]: 2024-04-02T14:52:55.436+0000 7f34e2cdd700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:50.261330+0000 front 2024-04-02T14:51:50.261317+0000 (oldest deadline 2024-04-02T14:52:37.664335+0000) 2024-04-02T14:52:55.693 INFO:journalctl@ceph.osd.5.smithi177.stdout:Apr 02 14:52:55 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-5[48118]: 2024-04-02T14:52:55.436+0000 7f34e2cdd700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:20.466010+0000 front 2024-04-02T14:52:20.466157+0000 (oldest deadline 2024-04-02T14:52:46.366121+0000) 2024-04-02T14:52:55.693 INFO:journalctl@ceph.osd.5.smithi177.stdout:Apr 02 14:52:55 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-5[48118]: 2024-04-02T14:52:55.436+0000 7f34e2cdd700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:29.267006+0000 front 2024-04-02T14:52:29.267027+0000 (oldest deadline 2024-04-02T14:52:53.367088+0000) 2024-04-02T14:52:55.929 INFO:journalctl@ceph.osd.6.smithi090.stdout:Apr 02 14:52:55 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-6[59804]: 2024-04-02T14:52:55.680+0000 7fab55400700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:47.904942+0000 front 2024-04-02T14:51:47.905629+0000 (oldest deadline 2024-04-02T14:52:40.210630+0000) 2024-04-02T14:52:55.929 INFO:journalctl@ceph.osd.6.smithi090.stdout:Apr 02 14:52:55 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-6[59804]: 2024-04-02T14:52:55.680+0000 7fab55400700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:20.211746+0000 front 2024-04-02T14:52:20.211924+0000 (oldest deadline 2024-04-02T14:52:49.613199+0000) 2024-04-02T14:52:55.929 INFO:journalctl@ceph.osd.6.smithi090.stdout:Apr 02 14:52:55 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-6[59804]: 2024-04-02T14:52:55.680+0000 7fab55400700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:29.614242+0000 front 2024-04-02T14:52:29.614310+0000 (oldest deadline 2024-04-02T14:52:53.714498+0000) 2024-04-02T14:52:55.993 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 02 14:52:55 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-8[54680]: 2024-04-02T14:52:55.701+0000 7f64f23c8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:23.288870+0000 front 2024-04-02T14:52:23.288907+0000 (oldest deadline 2024-04-02T14:52:48.588938+0000) 2024-04-02T14:52:55.993 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 02 14:52:55 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-8[54680]: 2024-04-02T14:52:55.701+0000 7f64f23c8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:29.090126+0000 front 2024-04-02T14:52:29.090177+0000 (oldest deadline 2024-04-02T14:52:53.190215+0000) 2024-04-02T14:52:56.101 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 02 14:52:56 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-7[54644]: 2024-04-02T14:52:56.048+0000 7f0bb6c43700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:22.855156+0000 front 2024-04-02T14:52:22.855176+0000 (oldest deadline 2024-04-02T14:52:45.755296+0000) 2024-04-02T14:52:56.101 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 02 14:52:56 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-7[54644]: 2024-04-02T14:52:56.048+0000 7f0bb6c43700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:32.156922+0000 front 2024-04-02T14:52:32.157038+0000 (oldest deadline 2024-04-02T14:52:53.856857+0000) 2024-04-02T14:52:56.101 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:52:55 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:52:55.837+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:50.925796+0000 front 2024-04-02T14:51:50.925668+0000 (oldest deadline 2024-04-02T14:52:40.128984+0000) 2024-04-02T14:52:56.101 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:52:55 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:52:55.837+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:20.129325+0000 front 2024-04-02T14:52:20.129293+0000 (oldest deadline 2024-04-02T14:52:45.429431+0000) 2024-04-02T14:52:56.101 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:52:55 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:52:55.837+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:27.631081+0000 front 2024-04-02T14:52:27.631094+0000 (oldest deadline 2024-04-02T14:52:53.531189+0000) 2024-04-02T14:52:56.427 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:52:56 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:52:56.001+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:50.397138+0000 front 2024-04-02T14:51:50.397096+0000 (oldest deadline 2024-04-02T14:52:37.200213+0000) 2024-04-02T14:52:56.427 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:52:56 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:52:56.001+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:22.501322+0000 front 2024-04-02T14:52:22.501346+0000 (oldest deadline 2024-04-02T14:52:45.401451+0000) 2024-04-02T14:52:56.679 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:52:56 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:52:56.367+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:49.133761+0000 front 2024-04-02T14:51:49.134172+0000 (oldest deadline 2024-04-02T14:52:35.841983+0000) 2024-04-02T14:52:56.679 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:52:56 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:52:56.367+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:21.645685+0000 front 2024-04-02T14:52:21.645805+0000 (oldest deadline 2024-04-02T14:52:46.945889+0000) 2024-04-02T14:52:56.679 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:52:56 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:52:56.367+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:31.047892+0000 front 2024-04-02T14:52:31.047845+0000 (oldest deadline 2024-04-02T14:52:55.148007+0000) 2024-04-02T14:52:56.927 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 02 14:52:56 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-8[54680]: 2024-04-02T14:52:56.693+0000 7f64f23c8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:23.288870+0000 front 2024-04-02T14:52:23.288907+0000 (oldest deadline 2024-04-02T14:52:48.588938+0000) 2024-04-02T14:52:56.927 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 02 14:52:56 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-8[54680]: 2024-04-02T14:52:56.693+0000 7f64f23c8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:29.090126+0000 front 2024-04-02T14:52:29.090177+0000 (oldest deadline 2024-04-02T14:52:53.190215+0000) 2024-04-02T14:52:56.927 INFO:journalctl@ceph.osd.5.smithi177.stdout:Apr 02 14:52:56 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-5[48118]: 2024-04-02T14:52:56.456+0000 7f34e2cdd700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:50.261330+0000 front 2024-04-02T14:51:50.261317+0000 (oldest deadline 2024-04-02T14:52:37.664335+0000) 2024-04-02T14:52:56.927 INFO:journalctl@ceph.osd.5.smithi177.stdout:Apr 02 14:52:56 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-5[48118]: 2024-04-02T14:52:56.456+0000 7f34e2cdd700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:20.466010+0000 front 2024-04-02T14:52:20.466157+0000 (oldest deadline 2024-04-02T14:52:46.366121+0000) 2024-04-02T14:52:56.928 INFO:journalctl@ceph.osd.5.smithi177.stdout:Apr 02 14:52:56 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-5[48118]: 2024-04-02T14:52:56.456+0000 7f34e2cdd700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:29.267006+0000 front 2024-04-02T14:52:29.267027+0000 (oldest deadline 2024-04-02T14:52:53.367088+0000) 2024-04-02T14:52:57.101 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 02 14:52:57 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-7[54644]: 2024-04-02T14:52:57.070+0000 7f0bb6c43700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:22.855156+0000 front 2024-04-02T14:52:22.855176+0000 (oldest deadline 2024-04-02T14:52:45.755296+0000) 2024-04-02T14:52:57.101 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 02 14:52:57 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-7[54644]: 2024-04-02T14:52:57.070+0000 7f0bb6c43700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:32.156922+0000 front 2024-04-02T14:52:32.157038+0000 (oldest deadline 2024-04-02T14:52:53.856857+0000) 2024-04-02T14:52:57.101 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:52:56 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:52:56.837+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:50.925796+0000 front 2024-04-02T14:51:50.925668+0000 (oldest deadline 2024-04-02T14:52:40.128984+0000) 2024-04-02T14:52:57.101 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:52:56 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:52:56.837+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:20.129325+0000 front 2024-04-02T14:52:20.129293+0000 (oldest deadline 2024-04-02T14:52:45.429431+0000) 2024-04-02T14:52:57.101 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:52:56 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:52:56.837+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:27.631081+0000 front 2024-04-02T14:52:27.631094+0000 (oldest deadline 2024-04-02T14:52:53.531189+0000) 2024-04-02T14:52:57.179 INFO:journalctl@ceph.osd.6.smithi090.stdout:Apr 02 14:52:56 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-6[59804]: 2024-04-02T14:52:56.730+0000 7fab55400700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:47.904942+0000 front 2024-04-02T14:51:47.905629+0000 (oldest deadline 2024-04-02T14:52:40.210630+0000) 2024-04-02T14:52:57.179 INFO:journalctl@ceph.osd.6.smithi090.stdout:Apr 02 14:52:56 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-6[59804]: 2024-04-02T14:52:56.730+0000 7fab55400700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:20.211746+0000 front 2024-04-02T14:52:20.211924+0000 (oldest deadline 2024-04-02T14:52:49.613199+0000) 2024-04-02T14:52:57.179 INFO:journalctl@ceph.osd.6.smithi090.stdout:Apr 02 14:52:56 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-6[59804]: 2024-04-02T14:52:56.730+0000 7fab55400700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:29.614242+0000 front 2024-04-02T14:52:29.614310+0000 (oldest deadline 2024-04-02T14:52:53.714498+0000) 2024-04-02T14:52:57.427 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:52:56 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:52:56.997+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:50.397138+0000 front 2024-04-02T14:51:50.397096+0000 (oldest deadline 2024-04-02T14:52:37.200213+0000) 2024-04-02T14:52:57.427 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:52:56 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:52:56.997+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:22.501322+0000 front 2024-04-02T14:52:22.501346+0000 (oldest deadline 2024-04-02T14:52:45.401451+0000) 2024-04-02T14:52:57.679 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:52:57 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:52:57.380+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:49.133761+0000 front 2024-04-02T14:51:49.134172+0000 (oldest deadline 2024-04-02T14:52:35.841983+0000) 2024-04-02T14:52:57.679 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:52:57 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:52:57.380+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:21.645685+0000 front 2024-04-02T14:52:21.645805+0000 (oldest deadline 2024-04-02T14:52:46.945889+0000) 2024-04-02T14:52:57.679 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:52:57 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:52:57.380+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:31.047892+0000 front 2024-04-02T14:52:31.047845+0000 (oldest deadline 2024-04-02T14:52:55.148007+0000) 2024-04-02T14:52:57.927 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 02 14:52:57 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-8[54680]: 2024-04-02T14:52:57.650+0000 7f64f23c8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:23.288870+0000 front 2024-04-02T14:52:23.288907+0000 (oldest deadline 2024-04-02T14:52:48.588938+0000) 2024-04-02T14:52:57.927 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 02 14:52:57 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-8[54680]: 2024-04-02T14:52:57.650+0000 7f64f23c8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:29.090126+0000 front 2024-04-02T14:52:29.090177+0000 (oldest deadline 2024-04-02T14:52:53.190215+0000) 2024-04-02T14:52:57.927 INFO:journalctl@ceph.osd.5.smithi177.stdout:Apr 02 14:52:57 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-5[48118]: 2024-04-02T14:52:57.459+0000 7f34e2cdd700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:50.261330+0000 front 2024-04-02T14:51:50.261317+0000 (oldest deadline 2024-04-02T14:52:37.664335+0000) 2024-04-02T14:52:57.927 INFO:journalctl@ceph.osd.5.smithi177.stdout:Apr 02 14:52:57 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-5[48118]: 2024-04-02T14:52:57.459+0000 7f34e2cdd700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:20.466010+0000 front 2024-04-02T14:52:20.466157+0000 (oldest deadline 2024-04-02T14:52:46.366121+0000) 2024-04-02T14:52:57.928 INFO:journalctl@ceph.osd.5.smithi177.stdout:Apr 02 14:52:57 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-5[48118]: 2024-04-02T14:52:57.459+0000 7f34e2cdd700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:29.267006+0000 front 2024-04-02T14:52:29.267027+0000 (oldest deadline 2024-04-02T14:52:53.367088+0000) 2024-04-02T14:52:58.075 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:52:57 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:52:57.799+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:50.925796+0000 front 2024-04-02T14:51:50.925668+0000 (oldest deadline 2024-04-02T14:52:40.128984+0000) 2024-04-02T14:52:58.075 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:52:57 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:52:57.799+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:20.129325+0000 front 2024-04-02T14:52:20.129293+0000 (oldest deadline 2024-04-02T14:52:45.429431+0000) 2024-04-02T14:52:58.075 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:52:57 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:52:57.799+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:27.631081+0000 front 2024-04-02T14:52:27.631094+0000 (oldest deadline 2024-04-02T14:52:53.531189+0000) 2024-04-02T14:52:58.179 INFO:journalctl@ceph.osd.6.smithi090.stdout:Apr 02 14:52:57 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-6[59804]: 2024-04-02T14:52:57.729+0000 7fab55400700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:47.904942+0000 front 2024-04-02T14:51:47.905629+0000 (oldest deadline 2024-04-02T14:52:40.210630+0000) 2024-04-02T14:52:58.179 INFO:journalctl@ceph.osd.6.smithi090.stdout:Apr 02 14:52:57 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-6[59804]: 2024-04-02T14:52:57.729+0000 7fab55400700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:20.211746+0000 front 2024-04-02T14:52:20.211924+0000 (oldest deadline 2024-04-02T14:52:49.613199+0000) 2024-04-02T14:52:58.179 INFO:journalctl@ceph.osd.6.smithi090.stdout:Apr 02 14:52:57 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-6[59804]: 2024-04-02T14:52:57.729+0000 7fab55400700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:29.614242+0000 front 2024-04-02T14:52:29.614310+0000 (oldest deadline 2024-04-02T14:52:53.714498+0000) 2024-04-02T14:52:58.351 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 02 14:52:58 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-7[54644]: 2024-04-02T14:52:58.083+0000 7f0bb6c43700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:22.855156+0000 front 2024-04-02T14:52:22.855176+0000 (oldest deadline 2024-04-02T14:52:45.755296+0000) 2024-04-02T14:52:58.351 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 02 14:52:58 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-7[54644]: 2024-04-02T14:52:58.083+0000 7f0bb6c43700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:32.156922+0000 front 2024-04-02T14:52:32.157038+0000 (oldest deadline 2024-04-02T14:52:53.856857+0000) 2024-04-02T14:52:58.396 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:52:58 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:52:58.016+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:50.397138+0000 front 2024-04-02T14:51:50.397096+0000 (oldest deadline 2024-04-02T14:52:37.200213+0000) 2024-04-02T14:52:58.396 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:52:58 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:52:58.016+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:22.501322+0000 front 2024-04-02T14:52:22.501346+0000 (oldest deadline 2024-04-02T14:52:45.401451+0000) 2024-04-02T14:52:58.396 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:52:58 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:52:58.016+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:28.302204+0000 front 2024-04-02T14:52:28.302332+0000 (oldest deadline 2024-04-02T14:52:57.102868+0000) 2024-04-02T14:52:58.661 INFO:journalctl@ceph.osd.5.smithi177.stdout:Apr 02 14:52:58 smithi177 podman[73820]: 2024-04-02 14:52:58.40493562 +0000 UTC m=+5.749109928 container died 31f658b60a05fb9cbf23d6dcd99f4845b45078a80dd7da261414779fb0bc8fd5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240402.004512-191d963, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240326) 2024-04-02T14:52:58.679 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:52:58 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:52:58.416+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:49.133761+0000 front 2024-04-02T14:51:49.134172+0000 (oldest deadline 2024-04-02T14:52:35.841983+0000) 2024-04-02T14:52:58.679 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:52:58 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:52:58.416+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:21.645685+0000 front 2024-04-02T14:52:21.645805+0000 (oldest deadline 2024-04-02T14:52:46.945889+0000) 2024-04-02T14:52:58.679 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:52:58 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:52:58.416+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:31.047892+0000 front 2024-04-02T14:52:31.047845+0000 (oldest deadline 2024-04-02T14:52:55.148007+0000) 2024-04-02T14:52:58.927 INFO:journalctl@ceph.osd.5.smithi177.stdout:Apr 02 14:52:58 smithi177 podman[73820]: 2024-04-02 14:52:58.761030613 +0000 UTC m=+6.105204915 container cleanup 31f658b60a05fb9cbf23d6dcd99f4845b45078a80dd7da261414779fb0bc8fd5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-5, RELEASE=wip-batrick-testing-20240402.004512-191d963, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240326, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0) 2024-04-02T14:52:58.927 INFO:journalctl@ceph.osd.5.smithi177.stdout:Apr 02 14:52:58 smithi177 bash[73820]: ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-5 2024-04-02T14:52:58.927 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 02 14:52:58 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-8[54680]: 2024-04-02T14:52:58.669+0000 7f64f23c8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:23.288870+0000 front 2024-04-02T14:52:23.288907+0000 (oldest deadline 2024-04-02T14:52:48.588938+0000) 2024-04-02T14:52:58.927 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 02 14:52:58 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-8[54680]: 2024-04-02T14:52:58.669+0000 7f64f23c8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:29.090126+0000 front 2024-04-02T14:52:29.090177+0000 (oldest deadline 2024-04-02T14:52:53.190215+0000) 2024-04-02T14:52:59.101 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 02 14:52:59 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-7[54644]: 2024-04-02T14:52:59.090+0000 7f0bb6c43700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:22.855156+0000 front 2024-04-02T14:52:22.855176+0000 (oldest deadline 2024-04-02T14:52:45.755296+0000) 2024-04-02T14:52:59.101 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 02 14:52:59 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-7[54644]: 2024-04-02T14:52:59.090+0000 7f0bb6c43700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:32.156922+0000 front 2024-04-02T14:52:32.157038+0000 (oldest deadline 2024-04-02T14:52:53.856857+0000) 2024-04-02T14:52:59.101 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:52:58 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:52:58.846+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:50.925796+0000 front 2024-04-02T14:51:50.925668+0000 (oldest deadline 2024-04-02T14:52:40.128984+0000) 2024-04-02T14:52:59.101 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:52:58 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:52:58.846+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:20.129325+0000 front 2024-04-02T14:52:20.129293+0000 (oldest deadline 2024-04-02T14:52:45.429431+0000) 2024-04-02T14:52:59.101 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:52:58 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:52:58.846+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:27.631081+0000 front 2024-04-02T14:52:27.631094+0000 (oldest deadline 2024-04-02T14:52:53.531189+0000) 2024-04-02T14:52:59.179 INFO:journalctl@ceph.osd.6.smithi090.stdout:Apr 02 14:52:58 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-6[59804]: 2024-04-02T14:52:58.772+0000 7fab55400700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:47.904942+0000 front 2024-04-02T14:51:47.905629+0000 (oldest deadline 2024-04-02T14:52:40.210630+0000) 2024-04-02T14:52:59.179 INFO:journalctl@ceph.osd.6.smithi090.stdout:Apr 02 14:52:58 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-6[59804]: 2024-04-02T14:52:58.772+0000 7fab55400700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:20.211746+0000 front 2024-04-02T14:52:20.211924+0000 (oldest deadline 2024-04-02T14:52:49.613199+0000) 2024-04-02T14:52:59.179 INFO:journalctl@ceph.osd.6.smithi090.stdout:Apr 02 14:52:58 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-6[59804]: 2024-04-02T14:52:58.772+0000 7fab55400700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:29.614242+0000 front 2024-04-02T14:52:29.614310+0000 (oldest deadline 2024-04-02T14:52:53.714498+0000) 2024-04-02T14:52:59.363 INFO:journalctl@ceph.osd.5.smithi177.stdout:Apr 02 14:52:59 smithi177 podman[73837]: 2024-04-02 14:52:59.054392383 +0000 UTC m=+0.642755432 container remove 31f658b60a05fb9cbf23d6dcd99f4845b45078a80dd7da261414779fb0bc8fd5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240326, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240402.004512-191d963, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-04-02T14:52:59.363 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:52:59 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:52:59.008+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:50.397138+0000 front 2024-04-02T14:51:50.397096+0000 (oldest deadline 2024-04-02T14:52:37.200213+0000) 2024-04-02T14:52:59.364 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:52:59 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:52:59.008+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:22.501322+0000 front 2024-04-02T14:52:22.501346+0000 (oldest deadline 2024-04-02T14:52:45.401451+0000) 2024-04-02T14:52:59.364 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:52:59 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:52:59.008+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:28.302204+0000 front 2024-04-02T14:52:28.302332+0000 (oldest deadline 2024-04-02T14:52:57.102868+0000) 2024-04-02T14:52:59.650 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 02 14:52:59 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-8[54680]: 2024-04-02T14:52:59.620+0000 7f64f23c8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:23.288870+0000 front 2024-04-02T14:52:23.288907+0000 (oldest deadline 2024-04-02T14:52:48.588938+0000) 2024-04-02T14:52:59.650 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 02 14:52:59 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-8[54680]: 2024-04-02T14:52:59.620+0000 7f64f23c8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:29.090126+0000 front 2024-04-02T14:52:29.090177+0000 (oldest deadline 2024-04-02T14:52:53.190215+0000) 2024-04-02T14:52:59.651 INFO:journalctl@ceph.osd.5.smithi177.stdout:Apr 02 14:52:59 smithi177 podman[73925]: 2024-04-02 14:52:59.27493934 +0000 UTC m=+0.017850714 image pull f7301e070545fc23d8e5b7b052a2c15e832a0e57102f1acad8e4f3583709d23f quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534 2024-04-02T14:52:59.651 INFO:journalctl@ceph.osd.5.smithi177.stdout:Apr 02 14:52:59 smithi177 podman[73925]: 2024-04-02 14:52:59.385822447 +0000 UTC m=+0.128733864 container create 0256c7c5e97a162f134f4177f682c7c2cf312c4437d4db9704584a32ac760d0f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-5-deactivate, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240402.004512-191d963, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240326, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-04-02T14:52:59.651 INFO:journalctl@ceph.osd.5.smithi177.stdout:Apr 02 14:52:59 smithi177 podman[73925]: 2024-04-02 14:52:59.584355178 +0000 UTC m=+0.327266553 container init 0256c7c5e97a162f134f4177f682c7c2cf312c4437d4db9704584a32ac760d0f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-5-deactivate, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=wip-batrick-testing-20240402.004512-191d963, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240326) 2024-04-02T14:52:59.651 INFO:journalctl@ceph.osd.5.smithi177.stdout:Apr 02 14:52:59 smithi177 podman[73925]: 2024-04-02 14:52:59.587496574 +0000 UTC m=+0.330407941 container start 0256c7c5e97a162f134f4177f682c7c2cf312c4437d4db9704584a32ac760d0f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-5-deactivate, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240326, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240402.004512-191d963, GIT_CLEAN=True, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux ) 2024-04-02T14:52:59.651 INFO:journalctl@ceph.osd.5.smithi177.stdout:Apr 02 14:52:59 smithi177 podman[73925]: 2024-04-02 14:52:59.659228953 +0000 UTC m=+0.402140331 container attach 0256c7c5e97a162f134f4177f682c7c2cf312c4437d4db9704584a32ac760d0f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-5-deactivate, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240326, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=wip-batrick-testing-20240402.004512-191d963, io.buildah.version=1.33.5) 2024-04-02T14:52:59.679 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:52:59 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:52:59.429+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:49.133761+0000 front 2024-04-02T14:51:49.134172+0000 (oldest deadline 2024-04-02T14:52:35.841983+0000) 2024-04-02T14:52:59.679 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:52:59 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:52:59.429+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:21.645685+0000 front 2024-04-02T14:52:21.645805+0000 (oldest deadline 2024-04-02T14:52:46.945889+0000) 2024-04-02T14:52:59.679 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:52:59 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:52:59.429+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:31.047892+0000 front 2024-04-02T14:52:31.047845+0000 (oldest deadline 2024-04-02T14:52:55.148007+0000) 2024-04-02T14:52:59.927 INFO:journalctl@ceph.osd.5.smithi177.stdout:Apr 02 14:52:59 smithi177 podman[73925]: 2024-04-02 14:52:59.890120928 +0000 UTC m=+0.633032297 container died 0256c7c5e97a162f134f4177f682c7c2cf312c4437d4db9704584a32ac760d0f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-5-deactivate, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240326, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240402.004512-191d963, io.buildah.version=1.33.5) 2024-04-02T14:53:00.097 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:52:59 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:52:59.843+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:50.925796+0000 front 2024-04-02T14:51:50.925668+0000 (oldest deadline 2024-04-02T14:52:40.128984+0000) 2024-04-02T14:53:00.097 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:52:59 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:52:59.843+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:20.129325+0000 front 2024-04-02T14:52:20.129293+0000 (oldest deadline 2024-04-02T14:52:45.429431+0000) 2024-04-02T14:53:00.097 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:52:59 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:52:59.843+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:27.631081+0000 front 2024-04-02T14:52:27.631094+0000 (oldest deadline 2024-04-02T14:52:53.531189+0000) 2024-04-02T14:53:00.179 INFO:journalctl@ceph.osd.6.smithi090.stdout:Apr 02 14:52:59 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-6[59804]: 2024-04-02T14:52:59.739+0000 7fab55400700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:47.904942+0000 front 2024-04-02T14:51:47.905629+0000 (oldest deadline 2024-04-02T14:52:40.210630+0000) 2024-04-02T14:53:00.179 INFO:journalctl@ceph.osd.6.smithi090.stdout:Apr 02 14:52:59 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-6[59804]: 2024-04-02T14:52:59.739+0000 7fab55400700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:20.211746+0000 front 2024-04-02T14:52:20.211924+0000 (oldest deadline 2024-04-02T14:52:49.613199+0000) 2024-04-02T14:53:00.179 INFO:journalctl@ceph.osd.6.smithi090.stdout:Apr 02 14:52:59 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-6[59804]: 2024-04-02T14:52:59.739+0000 7fab55400700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:29.614242+0000 front 2024-04-02T14:52:29.614310+0000 (oldest deadline 2024-04-02T14:52:53.714498+0000) 2024-04-02T14:53:00.351 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 02 14:53:00 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-7[54644]: 2024-04-02T14:53:00.105+0000 7f0bb6c43700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:22.855156+0000 front 2024-04-02T14:52:22.855176+0000 (oldest deadline 2024-04-02T14:52:45.755296+0000) 2024-04-02T14:53:00.351 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 02 14:53:00 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-7[54644]: 2024-04-02T14:53:00.105+0000 7f0bb6c43700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:32.156922+0000 front 2024-04-02T14:52:32.157038+0000 (oldest deadline 2024-04-02T14:52:53.856857+0000) 2024-04-02T14:53:00.420 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:52:59 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:52:59.965+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:50.397138+0000 front 2024-04-02T14:51:50.397096+0000 (oldest deadline 2024-04-02T14:52:37.200213+0000) 2024-04-02T14:53:00.420 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:52:59 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:52:59.965+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:22.501322+0000 front 2024-04-02T14:52:22.501346+0000 (oldest deadline 2024-04-02T14:52:45.401451+0000) 2024-04-02T14:53:00.420 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:52:59 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:52:59.965+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:28.302204+0000 front 2024-04-02T14:52:28.302332+0000 (oldest deadline 2024-04-02T14:52:57.102868+0000) 2024-04-02T14:53:00.435 DEBUG:teuthology.orchestra.run.smithi177:> sudo pkill -f 'journalctl -f -n 0 -u ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4@osd.5.service' 2024-04-02T14:53:00.475 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-02T14:53:00.475 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-04-02T14:53:00.475 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-04-02T14:53:00.476 DEBUG:teuthology.orchestra.run.smithi090:> sudo systemctl stop ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4@osd.6 2024-04-02T14:53:00.677 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 02 14:53:00 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-8[54680]: 2024-04-02T14:53:00.662+0000 7f64f23c8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:23.288870+0000 front 2024-04-02T14:52:23.288907+0000 (oldest deadline 2024-04-02T14:52:48.588938+0000) 2024-04-02T14:53:00.677 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 02 14:53:00 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-8[54680]: 2024-04-02T14:53:00.662+0000 7f64f23c8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:29.090126+0000 front 2024-04-02T14:52:29.090177+0000 (oldest deadline 2024-04-02T14:52:53.190215+0000) 2024-04-02T14:53:00.740 INFO:journalctl@ceph.osd.6.smithi090.stdout:Apr 02 14:53:00 smithi090 systemd[1]: Stopping Ceph osd.6 for 6449448a-f0fc-11ee-b647-cb9ed24678a4... 2024-04-02T14:53:00.740 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:53:00 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:53:00.463+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:49.133761+0000 front 2024-04-02T14:51:49.134172+0000 (oldest deadline 2024-04-02T14:52:35.841983+0000) 2024-04-02T14:53:00.740 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:53:00 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:53:00.463+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:21.645685+0000 front 2024-04-02T14:52:21.645805+0000 (oldest deadline 2024-04-02T14:52:46.945889+0000) 2024-04-02T14:53:00.740 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:53:00 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:53:00.463+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:31.047892+0000 front 2024-04-02T14:52:31.047845+0000 (oldest deadline 2024-04-02T14:52:55.148007+0000) 2024-04-02T14:53:01.025 INFO:journalctl@ceph.osd.6.smithi090.stdout:Apr 02 14:53:00 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-6[59804]: 2024-04-02T14:53:00.746+0000 7fab55400700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:47.904942+0000 front 2024-04-02T14:51:47.905629+0000 (oldest deadline 2024-04-02T14:52:40.210630+0000) 2024-04-02T14:53:01.026 INFO:journalctl@ceph.osd.6.smithi090.stdout:Apr 02 14:53:00 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-6[59804]: 2024-04-02T14:53:00.746+0000 7fab55400700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:20.211746+0000 front 2024-04-02T14:52:20.211924+0000 (oldest deadline 2024-04-02T14:52:49.613199+0000) 2024-04-02T14:53:01.026 INFO:journalctl@ceph.osd.6.smithi090.stdout:Apr 02 14:53:00 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-6[59804]: 2024-04-02T14:53:00.746+0000 7fab55400700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:29.614242+0000 front 2024-04-02T14:52:29.614310+0000 (oldest deadline 2024-04-02T14:52:53.714498+0000) 2024-04-02T14:53:01.026 INFO:journalctl@ceph.osd.6.smithi090.stdout:Apr 02 14:53:01 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-6[59804]: 2024-04-02T14:53:01.032+0000 7fab5ec00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-02T14:53:01.026 INFO:journalctl@ceph.osd.6.smithi090.stdout:Apr 02 14:53:01 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-6[59804]: 2024-04-02T14:53:01.032+0000 7fab5ec00700 -1 osd.6 114 *** Got signal Terminated *** 2024-04-02T14:53:01.026 INFO:journalctl@ceph.osd.6.smithi090.stdout:Apr 02 14:53:01 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-6[59804]: 2024-04-02T14:53:01.032+0000 7fab5ec00700 -1 osd.6 114 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-02T14:53:01.177 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:00 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:00.932+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:50.397138+0000 front 2024-04-02T14:51:50.397096+0000 (oldest deadline 2024-04-02T14:52:37.200213+0000) 2024-04-02T14:53:01.177 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:00 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:00.932+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:22.501322+0000 front 2024-04-02T14:52:22.501346+0000 (oldest deadline 2024-04-02T14:52:45.401451+0000) 2024-04-02T14:53:01.177 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:00 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:00.932+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:28.302204+0000 front 2024-04-02T14:52:28.302332+0000 (oldest deadline 2024-04-02T14:52:57.102868+0000) 2024-04-02T14:53:01.351 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 02 14:53:01 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-7[54644]: 2024-04-02T14:53:01.095+0000 7f0bb6c43700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:22.855156+0000 front 2024-04-02T14:52:22.855176+0000 (oldest deadline 2024-04-02T14:52:45.755296+0000) 2024-04-02T14:53:01.351 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 02 14:53:01 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-7[54644]: 2024-04-02T14:53:01.095+0000 7f0bb6c43700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:32.156922+0000 front 2024-04-02T14:52:32.157038+0000 (oldest deadline 2024-04-02T14:52:53.856857+0000) 2024-04-02T14:53:01.351 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:00 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:00.876+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:50.925796+0000 front 2024-04-02T14:51:50.925668+0000 (oldest deadline 2024-04-02T14:52:40.128984+0000) 2024-04-02T14:53:01.351 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:00 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:00.876+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:20.129325+0000 front 2024-04-02T14:52:20.129293+0000 (oldest deadline 2024-04-02T14:52:45.429431+0000) 2024-04-02T14:53:01.351 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:00 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:00.876+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:27.631081+0000 front 2024-04-02T14:52:27.631094+0000 (oldest deadline 2024-04-02T14:52:53.531189+0000) 2024-04-02T14:53:01.929 INFO:journalctl@ceph.osd.6.smithi090.stdout:Apr 02 14:53:01 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-6[59804]: 2024-04-02T14:53:01.712+0000 7fab55400700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:47.904942+0000 front 2024-04-02T14:51:47.905629+0000 (oldest deadline 2024-04-02T14:52:40.210630+0000) 2024-04-02T14:53:01.929 INFO:journalctl@ceph.osd.6.smithi090.stdout:Apr 02 14:53:01 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-6[59804]: 2024-04-02T14:53:01.712+0000 7fab55400700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:20.211746+0000 front 2024-04-02T14:52:20.211924+0000 (oldest deadline 2024-04-02T14:52:49.613199+0000) 2024-04-02T14:53:01.929 INFO:journalctl@ceph.osd.6.smithi090.stdout:Apr 02 14:53:01 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-6[59804]: 2024-04-02T14:53:01.712+0000 7fab55400700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:29.614242+0000 front 2024-04-02T14:52:29.614310+0000 (oldest deadline 2024-04-02T14:52:53.714498+0000) 2024-04-02T14:53:01.930 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:53:01 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:53:01.486+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:49.133761+0000 front 2024-04-02T14:51:49.134172+0000 (oldest deadline 2024-04-02T14:52:35.841983+0000) 2024-04-02T14:53:01.930 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:53:01 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:53:01.486+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:21.645685+0000 front 2024-04-02T14:52:21.645805+0000 (oldest deadline 2024-04-02T14:52:46.945889+0000) 2024-04-02T14:53:01.930 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:53:01 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:53:01.486+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:31.047892+0000 front 2024-04-02T14:52:31.047845+0000 (oldest deadline 2024-04-02T14:52:55.148007+0000) 2024-04-02T14:53:01.963 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 02 14:53:01 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-8[54680]: 2024-04-02T14:53:01.706+0000 7f64f23c8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:23.288870+0000 front 2024-04-02T14:52:23.288907+0000 (oldest deadline 2024-04-02T14:52:48.588938+0000) 2024-04-02T14:53:01.963 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 02 14:53:01 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-8[54680]: 2024-04-02T14:53:01.706+0000 7f64f23c8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:29.090126+0000 front 2024-04-02T14:52:29.090177+0000 (oldest deadline 2024-04-02T14:52:53.190215+0000) 2024-04-02T14:53:02.351 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 02 14:53:02 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-7[54644]: 2024-04-02T14:53:02.105+0000 7f0bb6c43700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:22.855156+0000 front 2024-04-02T14:52:22.855176+0000 (oldest deadline 2024-04-02T14:52:45.755296+0000) 2024-04-02T14:53:02.351 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 02 14:53:02 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-7[54644]: 2024-04-02T14:53:02.105+0000 7f0bb6c43700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:32.156922+0000 front 2024-04-02T14:52:32.157038+0000 (oldest deadline 2024-04-02T14:52:53.856857+0000) 2024-04-02T14:53:02.351 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:01 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:01.886+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:50.925796+0000 front 2024-04-02T14:51:50.925668+0000 (oldest deadline 2024-04-02T14:52:40.128984+0000) 2024-04-02T14:53:02.351 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:01 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:01.886+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:20.129325+0000 front 2024-04-02T14:52:20.129293+0000 (oldest deadline 2024-04-02T14:52:45.429431+0000) 2024-04-02T14:53:02.351 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:01 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:01.886+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:27.631081+0000 front 2024-04-02T14:52:27.631094+0000 (oldest deadline 2024-04-02T14:52:53.531189+0000) 2024-04-02T14:53:02.427 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:01 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:01.970+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:50.397138+0000 front 2024-04-02T14:51:50.397096+0000 (oldest deadline 2024-04-02T14:52:37.200213+0000) 2024-04-02T14:53:02.427 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:01 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:01.970+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:22.501322+0000 front 2024-04-02T14:52:22.501346+0000 (oldest deadline 2024-04-02T14:52:45.401451+0000) 2024-04-02T14:53:02.427 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:01 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:01.970+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:28.302204+0000 front 2024-04-02T14:52:28.302332+0000 (oldest deadline 2024-04-02T14:52:57.102868+0000) 2024-04-02T14:53:02.717 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:53:02 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:53:02.448+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:49.133761+0000 front 2024-04-02T14:51:49.134172+0000 (oldest deadline 2024-04-02T14:52:35.841983+0000) 2024-04-02T14:53:02.717 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:53:02 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:53:02.448+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:21.645685+0000 front 2024-04-02T14:52:21.645805+0000 (oldest deadline 2024-04-02T14:52:46.945889+0000) 2024-04-02T14:53:02.717 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:53:02 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:53:02.448+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:31.047892+0000 front 2024-04-02T14:52:31.047845+0000 (oldest deadline 2024-04-02T14:52:55.148007+0000) 2024-04-02T14:53:02.967 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 02 14:53:02 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-8[54680]: 2024-04-02T14:53:02.712+0000 7f64f23c8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:23.288870+0000 front 2024-04-02T14:52:23.288907+0000 (oldest deadline 2024-04-02T14:52:48.588938+0000) 2024-04-02T14:53:02.967 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 02 14:53:02 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-8[54680]: 2024-04-02T14:53:02.712+0000 7f64f23c8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:29.090126+0000 front 2024-04-02T14:52:29.090177+0000 (oldest deadline 2024-04-02T14:52:53.190215+0000) 2024-04-02T14:53:03.179 INFO:journalctl@ceph.osd.6.smithi090.stdout:Apr 02 14:53:02 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-6[59804]: 2024-04-02T14:53:02.724+0000 7fab55400700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:47.904942+0000 front 2024-04-02T14:51:47.905629+0000 (oldest deadline 2024-04-02T14:52:40.210630+0000) 2024-04-02T14:53:03.179 INFO:journalctl@ceph.osd.6.smithi090.stdout:Apr 02 14:53:02 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-6[59804]: 2024-04-02T14:53:02.724+0000 7fab55400700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:20.211746+0000 front 2024-04-02T14:52:20.211924+0000 (oldest deadline 2024-04-02T14:52:49.613199+0000) 2024-04-02T14:53:03.179 INFO:journalctl@ceph.osd.6.smithi090.stdout:Apr 02 14:53:02 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-6[59804]: 2024-04-02T14:53:02.724+0000 7fab55400700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:29.614242+0000 front 2024-04-02T14:52:29.614310+0000 (oldest deadline 2024-04-02T14:52:53.714498+0000) 2024-04-02T14:53:03.351 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 02 14:53:03 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-7[54644]: 2024-04-02T14:53:03.068+0000 7f0bb6c43700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:22.855156+0000 front 2024-04-02T14:52:22.855176+0000 (oldest deadline 2024-04-02T14:52:45.755296+0000) 2024-04-02T14:53:03.351 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 02 14:53:03 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-7[54644]: 2024-04-02T14:53:03.068+0000 7f0bb6c43700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:32.156922+0000 front 2024-04-02T14:52:32.157038+0000 (oldest deadline 2024-04-02T14:52:53.856857+0000) 2024-04-02T14:53:03.352 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:02 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:02.892+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:50.925796+0000 front 2024-04-02T14:51:50.925668+0000 (oldest deadline 2024-04-02T14:52:40.128984+0000) 2024-04-02T14:53:03.352 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:02 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:02.892+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:20.129325+0000 front 2024-04-02T14:52:20.129293+0000 (oldest deadline 2024-04-02T14:52:45.429431+0000) 2024-04-02T14:53:03.352 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:02 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:02.892+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:27.631081+0000 front 2024-04-02T14:52:27.631094+0000 (oldest deadline 2024-04-02T14:52:53.531189+0000) 2024-04-02T14:53:03.427 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:02 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:02.975+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:50.397138+0000 front 2024-04-02T14:51:50.397096+0000 (oldest deadline 2024-04-02T14:52:37.200213+0000) 2024-04-02T14:53:03.427 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:02 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:02.975+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:22.501322+0000 front 2024-04-02T14:52:22.501346+0000 (oldest deadline 2024-04-02T14:52:45.401451+0000) 2024-04-02T14:53:03.427 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:02 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:02.975+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:28.302204+0000 front 2024-04-02T14:52:28.302332+0000 (oldest deadline 2024-04-02T14:52:57.102868+0000) 2024-04-02T14:53:03.679 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:53:03 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:53:03.404+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:49.133761+0000 front 2024-04-02T14:51:49.134172+0000 (oldest deadline 2024-04-02T14:52:35.841983+0000) 2024-04-02T14:53:03.679 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:53:03 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:53:03.404+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:21.645685+0000 front 2024-04-02T14:52:21.645805+0000 (oldest deadline 2024-04-02T14:52:46.945889+0000) 2024-04-02T14:53:03.679 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:53:03 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:53:03.404+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:31.047892+0000 front 2024-04-02T14:52:31.047845+0000 (oldest deadline 2024-04-02T14:52:55.148007+0000) 2024-04-02T14:53:03.984 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 02 14:53:03 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-8[54680]: 2024-04-02T14:53:03.741+0000 7f64f23c8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:23.288870+0000 front 2024-04-02T14:52:23.288907+0000 (oldest deadline 2024-04-02T14:52:48.588938+0000) 2024-04-02T14:53:03.984 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 02 14:53:03 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-8[54680]: 2024-04-02T14:53:03.741+0000 7f64f23c8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:29.090126+0000 front 2024-04-02T14:52:29.090177+0000 (oldest deadline 2024-04-02T14:52:53.190215+0000) 2024-04-02T14:53:03.984 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 02 14:53:03 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-8[54680]: 2024-04-02T14:53:03.741+0000 7f64f23c8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-02T14:52:38.391857+0000 front 2024-04-02T14:52:38.391952+0000 (oldest deadline 2024-04-02T14:53:03.091726+0000) 2024-04-02T14:53:04.179 INFO:journalctl@ceph.osd.6.smithi090.stdout:Apr 02 14:53:03 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-6[59804]: 2024-04-02T14:53:03.721+0000 7fab55400700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:47.904942+0000 front 2024-04-02T14:51:47.905629+0000 (oldest deadline 2024-04-02T14:52:40.210630+0000) 2024-04-02T14:53:04.179 INFO:journalctl@ceph.osd.6.smithi090.stdout:Apr 02 14:53:03 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-6[59804]: 2024-04-02T14:53:03.721+0000 7fab55400700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:20.211746+0000 front 2024-04-02T14:52:20.211924+0000 (oldest deadline 2024-04-02T14:52:49.613199+0000) 2024-04-02T14:53:04.179 INFO:journalctl@ceph.osd.6.smithi090.stdout:Apr 02 14:53:03 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-6[59804]: 2024-04-02T14:53:03.721+0000 7fab55400700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:29.614242+0000 front 2024-04-02T14:52:29.614310+0000 (oldest deadline 2024-04-02T14:52:53.714498+0000) 2024-04-02T14:53:04.351 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 02 14:53:04 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-7[54644]: 2024-04-02T14:53:04.088+0000 7f0bb6c43700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:22.855156+0000 front 2024-04-02T14:52:22.855176+0000 (oldest deadline 2024-04-02T14:52:45.755296+0000) 2024-04-02T14:53:04.351 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 02 14:53:04 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-7[54644]: 2024-04-02T14:53:04.088+0000 7f0bb6c43700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:32.156922+0000 front 2024-04-02T14:52:32.157038+0000 (oldest deadline 2024-04-02T14:52:53.856857+0000) 2024-04-02T14:53:04.351 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:03 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:03.928+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:50.925796+0000 front 2024-04-02T14:51:50.925668+0000 (oldest deadline 2024-04-02T14:52:40.128984+0000) 2024-04-02T14:53:04.351 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:03 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:03.928+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:20.129325+0000 front 2024-04-02T14:52:20.129293+0000 (oldest deadline 2024-04-02T14:52:45.429431+0000) 2024-04-02T14:53:04.351 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:03 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:03.928+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:27.631081+0000 front 2024-04-02T14:52:27.631094+0000 (oldest deadline 2024-04-02T14:52:53.531189+0000) 2024-04-02T14:53:04.427 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:03 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:03.993+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:50.397138+0000 front 2024-04-02T14:51:50.397096+0000 (oldest deadline 2024-04-02T14:52:37.200213+0000) 2024-04-02T14:53:04.427 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:03 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:03.993+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:22.501322+0000 front 2024-04-02T14:52:22.501346+0000 (oldest deadline 2024-04-02T14:52:45.401451+0000) 2024-04-02T14:53:04.427 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:03 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:03.993+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:28.302204+0000 front 2024-04-02T14:52:28.302332+0000 (oldest deadline 2024-04-02T14:52:57.102868+0000) 2024-04-02T14:53:04.679 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:53:04 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:53:04.402+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:49.133761+0000 front 2024-04-02T14:51:49.134172+0000 (oldest deadline 2024-04-02T14:52:35.841983+0000) 2024-04-02T14:53:04.679 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:53:04 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:53:04.402+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:21.645685+0000 front 2024-04-02T14:52:21.645805+0000 (oldest deadline 2024-04-02T14:52:46.945889+0000) 2024-04-02T14:53:04.679 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:53:04 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:53:04.402+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:31.047892+0000 front 2024-04-02T14:52:31.047845+0000 (oldest deadline 2024-04-02T14:52:55.148007+0000) 2024-04-02T14:53:05.028 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 02 14:53:04 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-8[54680]: 2024-04-02T14:53:04.758+0000 7f64f23c8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:23.288870+0000 front 2024-04-02T14:52:23.288907+0000 (oldest deadline 2024-04-02T14:52:48.588938+0000) 2024-04-02T14:53:05.028 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 02 14:53:04 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-8[54680]: 2024-04-02T14:53:04.758+0000 7f64f23c8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:29.090126+0000 front 2024-04-02T14:52:29.090177+0000 (oldest deadline 2024-04-02T14:52:53.190215+0000) 2024-04-02T14:53:05.028 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 02 14:53:04 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-8[54680]: 2024-04-02T14:53:04.758+0000 7f64f23c8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-02T14:52:38.391857+0000 front 2024-04-02T14:52:38.391952+0000 (oldest deadline 2024-04-02T14:53:03.091726+0000) 2024-04-02T14:53:05.179 INFO:journalctl@ceph.osd.6.smithi090.stdout:Apr 02 14:53:04 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-6[59804]: 2024-04-02T14:53:04.687+0000 7fab55400700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:47.904942+0000 front 2024-04-02T14:51:47.905629+0000 (oldest deadline 2024-04-02T14:52:40.210630+0000) 2024-04-02T14:53:05.179 INFO:journalctl@ceph.osd.6.smithi090.stdout:Apr 02 14:53:04 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-6[59804]: 2024-04-02T14:53:04.687+0000 7fab55400700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:20.211746+0000 front 2024-04-02T14:52:20.211924+0000 (oldest deadline 2024-04-02T14:52:49.613199+0000) 2024-04-02T14:53:05.179 INFO:journalctl@ceph.osd.6.smithi090.stdout:Apr 02 14:53:04 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-6[59804]: 2024-04-02T14:53:04.687+0000 7fab55400700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:29.614242+0000 front 2024-04-02T14:52:29.614310+0000 (oldest deadline 2024-04-02T14:52:53.714498+0000) 2024-04-02T14:53:05.351 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 02 14:53:05 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-7[54644]: 2024-04-02T14:53:05.069+0000 7f0bb6c43700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:22.855156+0000 front 2024-04-02T14:52:22.855176+0000 (oldest deadline 2024-04-02T14:52:45.755296+0000) 2024-04-02T14:53:05.351 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 02 14:53:05 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-7[54644]: 2024-04-02T14:53:05.069+0000 7f0bb6c43700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:32.156922+0000 front 2024-04-02T14:52:32.157038+0000 (oldest deadline 2024-04-02T14:52:53.856857+0000) 2024-04-02T14:53:05.351 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:04 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:04.897+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:50.925796+0000 front 2024-04-02T14:51:50.925668+0000 (oldest deadline 2024-04-02T14:52:40.128984+0000) 2024-04-02T14:53:05.351 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:04 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:04.897+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:20.129325+0000 front 2024-04-02T14:52:20.129293+0000 (oldest deadline 2024-04-02T14:52:45.429431+0000) 2024-04-02T14:53:05.351 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:04 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:04.897+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:27.631081+0000 front 2024-04-02T14:52:27.631094+0000 (oldest deadline 2024-04-02T14:52:53.531189+0000) 2024-04-02T14:53:05.351 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:04 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:04.897+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-02T14:52:39.933034+0000 front 2024-04-02T14:52:39.933064+0000 (oldest deadline 2024-04-02T14:53:04.632770+0000) 2024-04-02T14:53:05.427 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:05 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:05.036+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:50.397138+0000 front 2024-04-02T14:51:50.397096+0000 (oldest deadline 2024-04-02T14:52:37.200213+0000) 2024-04-02T14:53:05.427 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:05 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:05.036+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:22.501322+0000 front 2024-04-02T14:52:22.501346+0000 (oldest deadline 2024-04-02T14:52:45.401451+0000) 2024-04-02T14:53:05.427 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:05 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:05.036+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:28.302204+0000 front 2024-04-02T14:52:28.302332+0000 (oldest deadline 2024-04-02T14:52:57.102868+0000) 2024-04-02T14:53:05.427 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:05 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:05.036+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-02T14:52:38.804314+0000 front 2024-04-02T14:52:38.804365+0000 (oldest deadline 2024-04-02T14:53:04.104362+0000) 2024-04-02T14:53:05.653 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:53:05 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:53:05.359+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:49.133761+0000 front 2024-04-02T14:51:49.134172+0000 (oldest deadline 2024-04-02T14:52:35.841983+0000) 2024-04-02T14:53:05.653 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:53:05 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:53:05.359+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:21.645685+0000 front 2024-04-02T14:52:21.645805+0000 (oldest deadline 2024-04-02T14:52:46.945889+0000) 2024-04-02T14:53:05.653 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:53:05 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:53:05.359+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:31.047892+0000 front 2024-04-02T14:52:31.047845+0000 (oldest deadline 2024-04-02T14:52:55.148007+0000) 2024-04-02T14:53:05.653 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:53:05 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:53:05.359+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-02T14:52:39.151607+0000 front 2024-04-02T14:52:39.151703+0000 (oldest deadline 2024-04-02T14:53:04.451174+0000) 2024-04-02T14:53:05.929 INFO:journalctl@ceph.osd.6.smithi090.stdout:Apr 02 14:53:05 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-6[59804]: 2024-04-02T14:53:05.660+0000 7fab55400700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:47.904942+0000 front 2024-04-02T14:51:47.905629+0000 (oldest deadline 2024-04-02T14:52:40.210630+0000) 2024-04-02T14:53:05.929 INFO:journalctl@ceph.osd.6.smithi090.stdout:Apr 02 14:53:05 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-6[59804]: 2024-04-02T14:53:05.660+0000 7fab55400700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:20.211746+0000 front 2024-04-02T14:52:20.211924+0000 (oldest deadline 2024-04-02T14:52:49.613199+0000) 2024-04-02T14:53:05.929 INFO:journalctl@ceph.osd.6.smithi090.stdout:Apr 02 14:53:05 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-6[59804]: 2024-04-02T14:53:05.660+0000 7fab55400700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:29.614242+0000 front 2024-04-02T14:52:29.614310+0000 (oldest deadline 2024-04-02T14:52:53.714498+0000) 2024-04-02T14:53:05.929 INFO:journalctl@ceph.osd.6.smithi090.stdout:Apr 02 14:53:05 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-6[59804]: 2024-04-02T14:53:05.660+0000 7fab55400700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-02T14:52:39.616674+0000 front 2024-04-02T14:52:39.616936+0000 (oldest deadline 2024-04-02T14:53:04.916358+0000) 2024-04-02T14:53:06.019 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 02 14:53:05 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-8[54680]: 2024-04-02T14:53:05.713+0000 7f64f23c8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:23.288870+0000 front 2024-04-02T14:52:23.288907+0000 (oldest deadline 2024-04-02T14:52:48.588938+0000) 2024-04-02T14:53:06.019 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 02 14:53:05 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-8[54680]: 2024-04-02T14:53:05.713+0000 7f64f23c8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:29.090126+0000 front 2024-04-02T14:52:29.090177+0000 (oldest deadline 2024-04-02T14:52:53.190215+0000) 2024-04-02T14:53:06.019 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 02 14:53:05 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-8[54680]: 2024-04-02T14:53:05.713+0000 7f64f23c8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-02T14:52:38.391857+0000 front 2024-04-02T14:52:38.391952+0000 (oldest deadline 2024-04-02T14:53:03.091726+0000) 2024-04-02T14:53:06.101 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 02 14:53:06 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-7[54644]: 2024-04-02T14:53:06.091+0000 7f0bb6c43700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:22.855156+0000 front 2024-04-02T14:52:22.855176+0000 (oldest deadline 2024-04-02T14:52:45.755296+0000) 2024-04-02T14:53:06.101 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 02 14:53:06 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-7[54644]: 2024-04-02T14:53:06.091+0000 7f0bb6c43700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:32.156922+0000 front 2024-04-02T14:52:32.157038+0000 (oldest deadline 2024-04-02T14:52:53.856857+0000) 2024-04-02T14:53:06.101 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 02 14:53:06 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-7[54644]: 2024-04-02T14:53:06.091+0000 7f0bb6c43700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-02T14:52:40.259022+0000 front 2024-04-02T14:52:40.258912+0000 (oldest deadline 2024-04-02T14:53:05.558692+0000) 2024-04-02T14:53:06.102 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:05 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:05.857+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:50.925796+0000 front 2024-04-02T14:51:50.925668+0000 (oldest deadline 2024-04-02T14:52:40.128984+0000) 2024-04-02T14:53:06.102 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:05 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:05.857+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:20.129325+0000 front 2024-04-02T14:52:20.129293+0000 (oldest deadline 2024-04-02T14:52:45.429431+0000) 2024-04-02T14:53:06.102 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:05 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:05.857+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:27.631081+0000 front 2024-04-02T14:52:27.631094+0000 (oldest deadline 2024-04-02T14:52:53.531189+0000) 2024-04-02T14:53:06.102 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:05 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:05.857+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-02T14:52:39.933034+0000 front 2024-04-02T14:52:39.933064+0000 (oldest deadline 2024-04-02T14:53:04.632770+0000) 2024-04-02T14:53:06.427 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:06 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:06.026+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:50.397138+0000 front 2024-04-02T14:51:50.397096+0000 (oldest deadline 2024-04-02T14:52:37.200213+0000) 2024-04-02T14:53:06.427 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:06 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:06.026+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:22.501322+0000 front 2024-04-02T14:52:22.501346+0000 (oldest deadline 2024-04-02T14:52:45.401451+0000) 2024-04-02T14:53:06.427 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:06 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:06.026+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:28.302204+0000 front 2024-04-02T14:52:28.302332+0000 (oldest deadline 2024-04-02T14:52:57.102868+0000) 2024-04-02T14:53:06.427 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:06 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:06.026+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-02T14:52:38.804314+0000 front 2024-04-02T14:52:38.804365+0000 (oldest deadline 2024-04-02T14:53:04.104362+0000) 2024-04-02T14:53:06.557 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:53:06 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:53:06.333+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:49.133761+0000 front 2024-04-02T14:51:49.134172+0000 (oldest deadline 2024-04-02T14:52:35.841983+0000) 2024-04-02T14:53:06.557 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:53:06 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:53:06.333+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:21.645685+0000 front 2024-04-02T14:52:21.645805+0000 (oldest deadline 2024-04-02T14:52:46.945889+0000) 2024-04-02T14:53:06.558 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:53:06 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:53:06.333+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:31.047892+0000 front 2024-04-02T14:52:31.047845+0000 (oldest deadline 2024-04-02T14:52:55.148007+0000) 2024-04-02T14:53:06.558 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:53:06 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:53:06.333+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-02T14:52:39.151607+0000 front 2024-04-02T14:52:39.151703+0000 (oldest deadline 2024-04-02T14:53:04.451174+0000) 2024-04-02T14:53:06.974 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 02 14:53:06 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-8[54680]: 2024-04-02T14:53:06.729+0000 7f64f23c8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:23.288870+0000 front 2024-04-02T14:52:23.288907+0000 (oldest deadline 2024-04-02T14:52:48.588938+0000) 2024-04-02T14:53:06.974 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 02 14:53:06 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-8[54680]: 2024-04-02T14:53:06.729+0000 7f64f23c8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:29.090126+0000 front 2024-04-02T14:52:29.090177+0000 (oldest deadline 2024-04-02T14:52:53.190215+0000) 2024-04-02T14:53:06.974 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 02 14:53:06 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-8[54680]: 2024-04-02T14:53:06.729+0000 7f64f23c8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-02T14:52:38.391857+0000 front 2024-04-02T14:52:38.391952+0000 (oldest deadline 2024-04-02T14:53:03.091726+0000) 2024-04-02T14:53:07.101 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 02 14:53:07 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-7[54644]: 2024-04-02T14:53:07.047+0000 7f0bb6c43700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:22.855156+0000 front 2024-04-02T14:52:22.855176+0000 (oldest deadline 2024-04-02T14:52:45.755296+0000) 2024-04-02T14:53:07.101 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 02 14:53:07 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-7[54644]: 2024-04-02T14:53:07.047+0000 7f0bb6c43700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:32.156922+0000 front 2024-04-02T14:52:32.157038+0000 (oldest deadline 2024-04-02T14:52:53.856857+0000) 2024-04-02T14:53:07.101 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 02 14:53:07 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-7[54644]: 2024-04-02T14:53:07.047+0000 7f0bb6c43700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-02T14:52:40.259022+0000 front 2024-04-02T14:52:40.258912+0000 (oldest deadline 2024-04-02T14:53:05.558692+0000) 2024-04-02T14:53:07.101 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:06 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:06.820+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:50.925796+0000 front 2024-04-02T14:51:50.925668+0000 (oldest deadline 2024-04-02T14:52:40.128984+0000) 2024-04-02T14:53:07.101 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:06 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:06.820+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:20.129325+0000 front 2024-04-02T14:52:20.129293+0000 (oldest deadline 2024-04-02T14:52:45.429431+0000) 2024-04-02T14:53:07.101 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:06 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:06.820+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:27.631081+0000 front 2024-04-02T14:52:27.631094+0000 (oldest deadline 2024-04-02T14:52:53.531189+0000) 2024-04-02T14:53:07.102 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:06 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:06.820+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-02T14:52:39.933034+0000 front 2024-04-02T14:52:39.933064+0000 (oldest deadline 2024-04-02T14:53:04.632770+0000) 2024-04-02T14:53:07.178 INFO:journalctl@ceph.osd.6.smithi090.stdout:Apr 02 14:53:07 smithi090 podman[111056]: 2024-04-02 14:53:07.029233083 +0000 UTC m=+6.121309022 container died 16eea31106b90928606d9242a3dcf89abd7f1a7973eee99c9b23c38a77b64d6b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-6, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=wip-batrick-testing-20240402.004512-191d963, org.label-schema.build-date=20240326, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-04-02T14:53:07.427 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:06 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:06.982+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:50.397138+0000 front 2024-04-02T14:51:50.397096+0000 (oldest deadline 2024-04-02T14:52:37.200213+0000) 2024-04-02T14:53:07.427 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:06 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:06.982+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:22.501322+0000 front 2024-04-02T14:52:22.501346+0000 (oldest deadline 2024-04-02T14:52:45.401451+0000) 2024-04-02T14:53:07.427 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:06 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:06.982+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:28.302204+0000 front 2024-04-02T14:52:28.302332+0000 (oldest deadline 2024-04-02T14:52:57.102868+0000) 2024-04-02T14:53:07.428 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:06 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:06.982+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-02T14:52:38.804314+0000 front 2024-04-02T14:52:38.804365+0000 (oldest deadline 2024-04-02T14:53:04.104362+0000) 2024-04-02T14:53:07.638 INFO:journalctl@ceph.osd.6.smithi090.stdout:Apr 02 14:53:07 smithi090 podman[111056]: 2024-04-02 14:53:07.486534973 +0000 UTC m=+6.578610902 container cleanup 16eea31106b90928606d9242a3dcf89abd7f1a7973eee99c9b23c38a77b64d6b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-6, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=wip-batrick-testing-20240402.004512-191d963, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240326, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-02T14:53:07.638 INFO:journalctl@ceph.osd.6.smithi090.stdout:Apr 02 14:53:07 smithi090 bash[111056]: ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-6 2024-04-02T14:53:07.638 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:53:07 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:53:07.370+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:49.133761+0000 front 2024-04-02T14:51:49.134172+0000 (oldest deadline 2024-04-02T14:52:35.841983+0000) 2024-04-02T14:53:07.638 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:53:07 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:53:07.370+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:21.645685+0000 front 2024-04-02T14:52:21.645805+0000 (oldest deadline 2024-04-02T14:52:46.945889+0000) 2024-04-02T14:53:07.638 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:53:07 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:53:07.370+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:31.047892+0000 front 2024-04-02T14:52:31.047845+0000 (oldest deadline 2024-04-02T14:52:55.148007+0000) 2024-04-02T14:53:07.638 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:53:07 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:53:07.370+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-02T14:52:39.151607+0000 front 2024-04-02T14:52:39.151703+0000 (oldest deadline 2024-04-02T14:53:04.451174+0000) 2024-04-02T14:53:07.773 INFO:journalctl@ceph.osd.6.smithi090.stdout:Apr 02 14:53:07 smithi090 podman[111082]: 2024-04-02 14:53:07.646056506 +0000 UTC m=+0.601180218 container remove 16eea31106b90928606d9242a3dcf89abd7f1a7973eee99c9b23c38a77b64d6b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-6, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240326, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240402.004512-191d963, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0) 2024-04-02T14:53:08.101 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 02 14:53:08 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-7[54644]: 2024-04-02T14:53:08.041+0000 7f0bb6c43700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:22.855156+0000 front 2024-04-02T14:52:22.855176+0000 (oldest deadline 2024-04-02T14:52:45.755296+0000) 2024-04-02T14:53:08.101 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 02 14:53:08 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-7[54644]: 2024-04-02T14:53:08.041+0000 7f0bb6c43700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:32.156922+0000 front 2024-04-02T14:52:32.157038+0000 (oldest deadline 2024-04-02T14:52:53.856857+0000) 2024-04-02T14:53:08.101 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 02 14:53:08 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-7[54644]: 2024-04-02T14:53:08.041+0000 7f0bb6c43700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-02T14:52:40.259022+0000 front 2024-04-02T14:52:40.258912+0000 (oldest deadline 2024-04-02T14:53:05.558692+0000) 2024-04-02T14:53:08.101 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:07 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:07.823+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:50.925796+0000 front 2024-04-02T14:51:50.925668+0000 (oldest deadline 2024-04-02T14:52:40.128984+0000) 2024-04-02T14:53:08.101 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:07 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:07.823+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:20.129325+0000 front 2024-04-02T14:52:20.129293+0000 (oldest deadline 2024-04-02T14:52:45.429431+0000) 2024-04-02T14:53:08.101 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:07 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:07.823+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:27.631081+0000 front 2024-04-02T14:52:27.631094+0000 (oldest deadline 2024-04-02T14:52:53.531189+0000) 2024-04-02T14:53:08.101 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:07 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:07.823+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-02T14:52:39.933034+0000 front 2024-04-02T14:52:39.933064+0000 (oldest deadline 2024-04-02T14:53:04.632770+0000) 2024-04-02T14:53:08.177 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 02 14:53:07 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-8[54680]: 2024-04-02T14:53:07.754+0000 7f64f23c8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:23.288870+0000 front 2024-04-02T14:52:23.288907+0000 (oldest deadline 2024-04-02T14:52:48.588938+0000) 2024-04-02T14:53:08.177 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 02 14:53:07 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-8[54680]: 2024-04-02T14:53:07.754+0000 7f64f23c8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:29.090126+0000 front 2024-04-02T14:52:29.090177+0000 (oldest deadline 2024-04-02T14:52:53.190215+0000) 2024-04-02T14:53:08.177 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 02 14:53:07 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-8[54680]: 2024-04-02T14:53:07.754+0000 7f64f23c8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-02T14:52:38.391857+0000 front 2024-04-02T14:52:38.391952+0000 (oldest deadline 2024-04-02T14:53:03.091726+0000) 2024-04-02T14:53:08.177 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:07 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:07.958+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:50.397138+0000 front 2024-04-02T14:51:50.397096+0000 (oldest deadline 2024-04-02T14:52:37.200213+0000) 2024-04-02T14:53:08.177 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:07 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:07.958+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:22.501322+0000 front 2024-04-02T14:52:22.501346+0000 (oldest deadline 2024-04-02T14:52:45.401451+0000) 2024-04-02T14:53:08.177 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:07 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:07.958+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:28.302204+0000 front 2024-04-02T14:52:28.302332+0000 (oldest deadline 2024-04-02T14:52:57.102868+0000) 2024-04-02T14:53:08.177 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:07 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:07.958+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-02T14:52:38.804314+0000 front 2024-04-02T14:52:38.804365+0000 (oldest deadline 2024-04-02T14:53:04.104362+0000) 2024-04-02T14:53:08.628 INFO:journalctl@ceph.osd.6.smithi090.stdout:Apr 02 14:53:08 smithi090 podman[111174]: 2024-04-02 14:53:08.223846835 +0000 UTC m=+0.032764278 image pull f7301e070545fc23d8e5b7b052a2c15e832a0e57102f1acad8e4f3583709d23f quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534 2024-04-02T14:53:08.628 INFO:journalctl@ceph.osd.6.smithi090.stdout:Apr 02 14:53:08 smithi090 podman[111174]: 2024-04-02 14:53:08.331925579 +0000 UTC m=+0.140843045 container create 1860692154afa5114909f946ba2d8a4a3e657202c90ba50714798c4f8032b636 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-6-deactivate, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240326, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=wip-batrick-testing-20240402.004512-191d963, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD) 2024-04-02T14:53:08.628 INFO:journalctl@ceph.osd.6.smithi090.stdout:Apr 02 14:53:08 smithi090 podman[111174]: 2024-04-02 14:53:08.635867354 +0000 UTC m=+0.444784822 container init 1860692154afa5114909f946ba2d8a4a3e657202c90ba50714798c4f8032b636 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-6-deactivate, RELEASE=wip-batrick-testing-20240402.004512-191d963, org.label-schema.build-date=20240326, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2) 2024-04-02T14:53:08.629 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:53:08 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:53:08.366+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:49.133761+0000 front 2024-04-02T14:51:49.134172+0000 (oldest deadline 2024-04-02T14:52:35.841983+0000) 2024-04-02T14:53:08.629 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:53:08 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:53:08.366+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:21.645685+0000 front 2024-04-02T14:52:21.645805+0000 (oldest deadline 2024-04-02T14:52:46.945889+0000) 2024-04-02T14:53:08.629 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:53:08 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:53:08.366+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:31.047892+0000 front 2024-04-02T14:52:31.047845+0000 (oldest deadline 2024-04-02T14:52:55.148007+0000) 2024-04-02T14:53:08.629 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:53:08 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:53:08.366+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-02T14:52:39.151607+0000 front 2024-04-02T14:52:39.151703+0000 (oldest deadline 2024-04-02T14:53:04.451174+0000) 2024-04-02T14:53:08.929 INFO:journalctl@ceph.osd.6.smithi090.stdout:Apr 02 14:53:08 smithi090 podman[111174]: 2024-04-02 14:53:08.646718584 +0000 UTC m=+0.455636076 container start 1860692154afa5114909f946ba2d8a4a3e657202c90ba50714798c4f8032b636 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-6-deactivate, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240326, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240402.004512-191d963, maintainer=Guillaume Abrioux ) 2024-04-02T14:53:08.929 INFO:journalctl@ceph.osd.6.smithi090.stdout:Apr 02 14:53:08 smithi090 podman[111174]: 2024-04-02 14:53:08.718009937 +0000 UTC m=+0.526927469 container attach 1860692154afa5114909f946ba2d8a4a3e657202c90ba50714798c4f8032b636 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-6-deactivate, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=wip-batrick-testing-20240402.004512-191d963, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240326, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-04-02T14:53:09.101 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 02 14:53:09 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-7[54644]: 2024-04-02T14:53:09.018+0000 7f0bb6c43700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:22.855156+0000 front 2024-04-02T14:52:22.855176+0000 (oldest deadline 2024-04-02T14:52:45.755296+0000) 2024-04-02T14:53:09.101 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 02 14:53:09 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-7[54644]: 2024-04-02T14:53:09.018+0000 7f0bb6c43700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:32.156922+0000 front 2024-04-02T14:52:32.157038+0000 (oldest deadline 2024-04-02T14:52:53.856857+0000) 2024-04-02T14:53:09.101 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 02 14:53:09 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-7[54644]: 2024-04-02T14:53:09.018+0000 7f0bb6c43700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-02T14:52:40.259022+0000 front 2024-04-02T14:52:40.258912+0000 (oldest deadline 2024-04-02T14:53:05.558692+0000) 2024-04-02T14:53:09.101 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:08 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:08.825+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:50.925796+0000 front 2024-04-02T14:51:50.925668+0000 (oldest deadline 2024-04-02T14:52:40.128984+0000) 2024-04-02T14:53:09.101 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:08 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:08.825+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:20.129325+0000 front 2024-04-02T14:52:20.129293+0000 (oldest deadline 2024-04-02T14:52:45.429431+0000) 2024-04-02T14:53:09.102 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:08 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:08.825+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:27.631081+0000 front 2024-04-02T14:52:27.631094+0000 (oldest deadline 2024-04-02T14:52:53.531189+0000) 2024-04-02T14:53:09.102 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:08 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:08.825+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-02T14:52:39.933034+0000 front 2024-04-02T14:52:39.933064+0000 (oldest deadline 2024-04-02T14:53:04.632770+0000) 2024-04-02T14:53:09.177 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 02 14:53:08 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-8[54680]: 2024-04-02T14:53:08.746+0000 7f64f23c8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:23.288870+0000 front 2024-04-02T14:52:23.288907+0000 (oldest deadline 2024-04-02T14:52:48.588938+0000) 2024-04-02T14:53:09.177 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 02 14:53:08 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-8[54680]: 2024-04-02T14:53:08.746+0000 7f64f23c8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:29.090126+0000 front 2024-04-02T14:52:29.090177+0000 (oldest deadline 2024-04-02T14:52:53.190215+0000) 2024-04-02T14:53:09.177 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 02 14:53:08 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-8[54680]: 2024-04-02T14:53:08.746+0000 7f64f23c8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-02T14:52:38.391857+0000 front 2024-04-02T14:52:38.391952+0000 (oldest deadline 2024-04-02T14:53:03.091726+0000) 2024-04-02T14:53:09.177 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:08 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:08.933+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:50.397138+0000 front 2024-04-02T14:51:50.397096+0000 (oldest deadline 2024-04-02T14:52:37.200213+0000) 2024-04-02T14:53:09.177 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:08 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:08.933+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:22.501322+0000 front 2024-04-02T14:52:22.501346+0000 (oldest deadline 2024-04-02T14:52:45.401451+0000) 2024-04-02T14:53:09.177 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:08 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:08.933+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:28.302204+0000 front 2024-04-02T14:52:28.302332+0000 (oldest deadline 2024-04-02T14:52:57.102868+0000) 2024-04-02T14:53:09.177 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:08 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:08.933+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-02T14:52:38.804314+0000 front 2024-04-02T14:52:38.804365+0000 (oldest deadline 2024-04-02T14:53:04.104362+0000) 2024-04-02T14:53:09.313 INFO:journalctl@ceph.osd.6.smithi090.stdout:Apr 02 14:53:09 smithi090 podman[111174]: 2024-04-02 14:53:09.050602414 +0000 UTC m=+0.859519903 container died 1860692154afa5114909f946ba2d8a4a3e657202c90ba50714798c4f8032b636 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-6-deactivate, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240326, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240402.004512-191d963) 2024-04-02T14:53:09.664 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:53:09 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:53:09.319+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:49.133761+0000 front 2024-04-02T14:51:49.134172+0000 (oldest deadline 2024-04-02T14:52:35.841983+0000) 2024-04-02T14:53:09.664 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:53:09 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:53:09.319+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:21.645685+0000 front 2024-04-02T14:52:21.645805+0000 (oldest deadline 2024-04-02T14:52:46.945889+0000) 2024-04-02T14:53:09.664 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:53:09 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:53:09.319+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:31.047892+0000 front 2024-04-02T14:52:31.047845+0000 (oldest deadline 2024-04-02T14:52:55.148007+0000) 2024-04-02T14:53:09.664 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:53:09 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:53:09.319+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-02T14:52:39.151607+0000 front 2024-04-02T14:52:39.151703+0000 (oldest deadline 2024-04-02T14:53:04.451174+0000) 2024-04-02T14:53:09.717 DEBUG:teuthology.orchestra.run.smithi090:> sudo pkill -f 'journalctl -f -n 0 -u ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4@osd.6.service' 2024-04-02T14:53:09.839 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-02T14:53:09.840 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-04-02T14:53:09.840 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-04-02T14:53:09.840 DEBUG:teuthology.orchestra.run.smithi092:> sudo systemctl stop ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4@osd.7 2024-04-02T14:53:10.101 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 02 14:53:09 smithi092 systemd[1]: Stopping Ceph osd.7 for 6449448a-f0fc-11ee-b647-cb9ed24678a4... 2024-04-02T14:53:10.101 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 02 14:53:10 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-7[54644]: 2024-04-02T14:53:10.021+0000 7f0bb6c43700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:22.855156+0000 front 2024-04-02T14:52:22.855176+0000 (oldest deadline 2024-04-02T14:52:45.755296+0000) 2024-04-02T14:53:10.101 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 02 14:53:10 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-7[54644]: 2024-04-02T14:53:10.021+0000 7f0bb6c43700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:32.156922+0000 front 2024-04-02T14:52:32.157038+0000 (oldest deadline 2024-04-02T14:52:53.856857+0000) 2024-04-02T14:53:10.101 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 02 14:53:10 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-7[54644]: 2024-04-02T14:53:10.021+0000 7f0bb6c43700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-02T14:52:40.259022+0000 front 2024-04-02T14:52:40.258912+0000 (oldest deadline 2024-04-02T14:53:05.558692+0000) 2024-04-02T14:53:10.101 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:09 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:09.830+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:50.925796+0000 front 2024-04-02T14:51:50.925668+0000 (oldest deadline 2024-04-02T14:52:40.128984+0000) 2024-04-02T14:53:10.102 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:09 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:09.830+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:20.129325+0000 front 2024-04-02T14:52:20.129293+0000 (oldest deadline 2024-04-02T14:52:45.429431+0000) 2024-04-02T14:53:10.102 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:09 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:09.830+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:27.631081+0000 front 2024-04-02T14:52:27.631094+0000 (oldest deadline 2024-04-02T14:52:53.531189+0000) 2024-04-02T14:53:10.102 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:09 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:09.830+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-02T14:52:39.933034+0000 front 2024-04-02T14:52:39.933064+0000 (oldest deadline 2024-04-02T14:53:04.632770+0000) 2024-04-02T14:53:10.177 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 02 14:53:09 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-8[54680]: 2024-04-02T14:53:09.786+0000 7f64f23c8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:23.288870+0000 front 2024-04-02T14:52:23.288907+0000 (oldest deadline 2024-04-02T14:52:48.588938+0000) 2024-04-02T14:53:10.177 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 02 14:53:09 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-8[54680]: 2024-04-02T14:53:09.786+0000 7f64f23c8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:29.090126+0000 front 2024-04-02T14:52:29.090177+0000 (oldest deadline 2024-04-02T14:52:53.190215+0000) 2024-04-02T14:53:10.177 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 02 14:53:09 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-8[54680]: 2024-04-02T14:53:09.786+0000 7f64f23c8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-02T14:52:38.391857+0000 front 2024-04-02T14:52:38.391952+0000 (oldest deadline 2024-04-02T14:53:03.091726+0000) 2024-04-02T14:53:10.177 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:09 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:09.977+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:50.397138+0000 front 2024-04-02T14:51:50.397096+0000 (oldest deadline 2024-04-02T14:52:37.200213+0000) 2024-04-02T14:53:10.177 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:09 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:09.977+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:22.501322+0000 front 2024-04-02T14:52:22.501346+0000 (oldest deadline 2024-04-02T14:52:45.401451+0000) 2024-04-02T14:53:10.177 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:09 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:09.977+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:28.302204+0000 front 2024-04-02T14:52:28.302332+0000 (oldest deadline 2024-04-02T14:52:57.102868+0000) 2024-04-02T14:53:10.178 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:09 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:09.977+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-02T14:52:38.804314+0000 front 2024-04-02T14:52:38.804365+0000 (oldest deadline 2024-04-02T14:53:04.104362+0000) 2024-04-02T14:53:10.601 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 02 14:53:10 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-7[54644]: 2024-04-02T14:53:10.152+0000 7f0bbe657700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-02T14:53:10.601 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 02 14:53:10 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-7[54644]: 2024-04-02T14:53:10.152+0000 7f0bbe657700 -1 osd.7 114 *** Got signal Terminated *** 2024-04-02T14:53:10.601 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 02 14:53:10 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-7[54644]: 2024-04-02T14:53:10.152+0000 7f0bbe657700 -1 osd.7 114 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-02T14:53:10.679 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:53:10 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:53:10.302+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:49.133761+0000 front 2024-04-02T14:51:49.134172+0000 (oldest deadline 2024-04-02T14:52:35.841983+0000) 2024-04-02T14:53:10.679 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:53:10 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:53:10.302+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:21.645685+0000 front 2024-04-02T14:52:21.645805+0000 (oldest deadline 2024-04-02T14:52:46.945889+0000) 2024-04-02T14:53:10.679 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:53:10 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:53:10.302+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:31.047892+0000 front 2024-04-02T14:52:31.047845+0000 (oldest deadline 2024-04-02T14:52:55.148007+0000) 2024-04-02T14:53:10.679 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:53:10 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:53:10.302+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-02T14:52:39.151607+0000 front 2024-04-02T14:52:39.151703+0000 (oldest deadline 2024-04-02T14:53:04.451174+0000) 2024-04-02T14:53:11.033 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:10 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:10.791+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:50.925796+0000 front 2024-04-02T14:51:50.925668+0000 (oldest deadline 2024-04-02T14:52:40.128984+0000) 2024-04-02T14:53:11.033 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:10 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:10.791+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:20.129325+0000 front 2024-04-02T14:52:20.129293+0000 (oldest deadline 2024-04-02T14:52:45.429431+0000) 2024-04-02T14:53:11.033 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:10 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:10.791+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:27.631081+0000 front 2024-04-02T14:52:27.631094+0000 (oldest deadline 2024-04-02T14:52:53.531189+0000) 2024-04-02T14:53:11.033 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:10 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:10.791+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-02T14:52:39.933034+0000 front 2024-04-02T14:52:39.933064+0000 (oldest deadline 2024-04-02T14:53:04.632770+0000) 2024-04-02T14:53:11.177 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 02 14:53:10 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-8[54680]: 2024-04-02T14:53:10.776+0000 7f64f23c8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:23.288870+0000 front 2024-04-02T14:52:23.288907+0000 (oldest deadline 2024-04-02T14:52:48.588938+0000) 2024-04-02T14:53:11.177 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 02 14:53:10 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-8[54680]: 2024-04-02T14:53:10.776+0000 7f64f23c8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:29.090126+0000 front 2024-04-02T14:52:29.090177+0000 (oldest deadline 2024-04-02T14:52:53.190215+0000) 2024-04-02T14:53:11.177 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 02 14:53:10 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-8[54680]: 2024-04-02T14:53:10.776+0000 7f64f23c8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-02T14:52:38.391857+0000 front 2024-04-02T14:52:38.391952+0000 (oldest deadline 2024-04-02T14:53:03.091726+0000) 2024-04-02T14:53:11.177 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 02 14:53:10 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-8[54680]: 2024-04-02T14:53:10.776+0000 7f64f23c8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.92:6812 osd.4 since back 2024-04-02T14:52:47.693204+0000 front 2024-04-02T14:52:47.693149+0000 (oldest deadline 2024-04-02T14:53:10.593118+0000) 2024-04-02T14:53:11.178 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:10 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:10.948+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:50.397138+0000 front 2024-04-02T14:51:50.397096+0000 (oldest deadline 2024-04-02T14:52:37.200213+0000) 2024-04-02T14:53:11.178 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:10 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:10.948+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:22.501322+0000 front 2024-04-02T14:52:22.501346+0000 (oldest deadline 2024-04-02T14:52:45.401451+0000) 2024-04-02T14:53:11.178 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:10 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:10.948+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:28.302204+0000 front 2024-04-02T14:52:28.302332+0000 (oldest deadline 2024-04-02T14:52:57.102868+0000) 2024-04-02T14:53:11.178 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:10 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:10.948+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-02T14:52:38.804314+0000 front 2024-04-02T14:52:38.804365+0000 (oldest deadline 2024-04-02T14:53:04.104362+0000) 2024-04-02T14:53:11.178 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:10 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:10.948+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.92:6812 osd.4 since back 2024-04-02T14:52:47.605500+0000 front 2024-04-02T14:52:47.605485+0000 (oldest deadline 2024-04-02T14:53:10.505428+0000) 2024-04-02T14:53:11.351 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 02 14:53:11 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-7[54644]: 2024-04-02T14:53:11.041+0000 7f0bb6c43700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:22.855156+0000 front 2024-04-02T14:52:22.855176+0000 (oldest deadline 2024-04-02T14:52:45.755296+0000) 2024-04-02T14:53:11.351 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 02 14:53:11 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-7[54644]: 2024-04-02T14:53:11.041+0000 7f0bb6c43700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:32.156922+0000 front 2024-04-02T14:52:32.157038+0000 (oldest deadline 2024-04-02T14:52:53.856857+0000) 2024-04-02T14:53:11.351 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 02 14:53:11 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-7[54644]: 2024-04-02T14:53:11.041+0000 7f0bb6c43700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-02T14:52:40.259022+0000 front 2024-04-02T14:52:40.258912+0000 (oldest deadline 2024-04-02T14:53:05.558692+0000) 2024-04-02T14:53:11.351 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 02 14:53:11 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-7[54644]: 2024-04-02T14:53:11.041+0000 7f0bb6c43700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.92:6812 osd.4 since back 2024-04-02T14:52:45.559190+0000 front 2024-04-02T14:52:45.559221+0000 (oldest deadline 2024-04-02T14:53:10.859288+0000) 2024-04-02T14:53:11.679 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:53:11 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:53:11.317+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:49.133761+0000 front 2024-04-02T14:51:49.134172+0000 (oldest deadline 2024-04-02T14:52:35.841983+0000) 2024-04-02T14:53:11.679 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:53:11 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:53:11.317+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:21.645685+0000 front 2024-04-02T14:52:21.645805+0000 (oldest deadline 2024-04-02T14:52:46.945889+0000) 2024-04-02T14:53:11.679 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:53:11 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:53:11.317+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:31.047892+0000 front 2024-04-02T14:52:31.047845+0000 (oldest deadline 2024-04-02T14:52:55.148007+0000) 2024-04-02T14:53:11.679 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:53:11 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:53:11.317+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-02T14:52:39.151607+0000 front 2024-04-02T14:52:39.151703+0000 (oldest deadline 2024-04-02T14:53:04.451174+0000) 2024-04-02T14:53:12.101 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 02 14:53:12 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-7[54644]: 2024-04-02T14:53:12.066+0000 7f0bb6c43700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:22.855156+0000 front 2024-04-02T14:52:22.855176+0000 (oldest deadline 2024-04-02T14:52:45.755296+0000) 2024-04-02T14:53:12.101 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 02 14:53:12 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-7[54644]: 2024-04-02T14:53:12.066+0000 7f0bb6c43700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:32.156922+0000 front 2024-04-02T14:52:32.157038+0000 (oldest deadline 2024-04-02T14:52:53.856857+0000) 2024-04-02T14:53:12.101 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 02 14:53:12 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-7[54644]: 2024-04-02T14:53:12.066+0000 7f0bb6c43700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-02T14:52:40.259022+0000 front 2024-04-02T14:52:40.258912+0000 (oldest deadline 2024-04-02T14:53:05.558692+0000) 2024-04-02T14:53:12.102 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 02 14:53:12 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-7[54644]: 2024-04-02T14:53:12.066+0000 7f0bb6c43700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.92:6812 osd.4 since back 2024-04-02T14:52:45.559190+0000 front 2024-04-02T14:52:45.559221+0000 (oldest deadline 2024-04-02T14:53:10.859288+0000) 2024-04-02T14:53:12.102 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:11 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:11.826+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:50.925796+0000 front 2024-04-02T14:51:50.925668+0000 (oldest deadline 2024-04-02T14:52:40.128984+0000) 2024-04-02T14:53:12.102 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:11 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:11.826+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:20.129325+0000 front 2024-04-02T14:52:20.129293+0000 (oldest deadline 2024-04-02T14:52:45.429431+0000) 2024-04-02T14:53:12.102 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:11 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:11.826+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:27.631081+0000 front 2024-04-02T14:52:27.631094+0000 (oldest deadline 2024-04-02T14:52:53.531189+0000) 2024-04-02T14:53:12.102 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:11 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:11.826+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-02T14:52:39.933034+0000 front 2024-04-02T14:52:39.933064+0000 (oldest deadline 2024-04-02T14:53:04.632770+0000) 2024-04-02T14:53:12.177 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 02 14:53:11 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-8[54680]: 2024-04-02T14:53:11.772+0000 7f64f23c8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:23.288870+0000 front 2024-04-02T14:52:23.288907+0000 (oldest deadline 2024-04-02T14:52:48.588938+0000) 2024-04-02T14:53:12.177 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 02 14:53:11 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-8[54680]: 2024-04-02T14:53:11.772+0000 7f64f23c8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:29.090126+0000 front 2024-04-02T14:52:29.090177+0000 (oldest deadline 2024-04-02T14:52:53.190215+0000) 2024-04-02T14:53:12.177 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 02 14:53:11 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-8[54680]: 2024-04-02T14:53:11.772+0000 7f64f23c8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-02T14:52:38.391857+0000 front 2024-04-02T14:52:38.391952+0000 (oldest deadline 2024-04-02T14:53:03.091726+0000) 2024-04-02T14:53:12.178 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 02 14:53:11 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-8[54680]: 2024-04-02T14:53:11.772+0000 7f64f23c8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.92:6812 osd.4 since back 2024-04-02T14:52:47.693204+0000 front 2024-04-02T14:52:47.693149+0000 (oldest deadline 2024-04-02T14:53:10.593118+0000) 2024-04-02T14:53:12.178 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:11 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:11.990+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:50.397138+0000 front 2024-04-02T14:51:50.397096+0000 (oldest deadline 2024-04-02T14:52:37.200213+0000) 2024-04-02T14:53:12.178 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:11 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:11.990+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:22.501322+0000 front 2024-04-02T14:52:22.501346+0000 (oldest deadline 2024-04-02T14:52:45.401451+0000) 2024-04-02T14:53:12.178 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:11 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:11.990+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:28.302204+0000 front 2024-04-02T14:52:28.302332+0000 (oldest deadline 2024-04-02T14:52:57.102868+0000) 2024-04-02T14:53:12.178 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:11 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:11.990+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-02T14:52:38.804314+0000 front 2024-04-02T14:52:38.804365+0000 (oldest deadline 2024-04-02T14:53:04.104362+0000) 2024-04-02T14:53:12.178 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:11 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:11.990+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.92:6812 osd.4 since back 2024-04-02T14:52:47.605500+0000 front 2024-04-02T14:52:47.605485+0000 (oldest deadline 2024-04-02T14:53:10.505428+0000) 2024-04-02T14:53:12.679 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:53:12 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:53:12.365+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:49.133761+0000 front 2024-04-02T14:51:49.134172+0000 (oldest deadline 2024-04-02T14:52:35.841983+0000) 2024-04-02T14:53:12.837 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:53:12 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:53:12.365+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:21.645685+0000 front 2024-04-02T14:52:21.645805+0000 (oldest deadline 2024-04-02T14:52:46.945889+0000) 2024-04-02T14:53:12.837 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:53:12 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:53:12.365+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:31.047892+0000 front 2024-04-02T14:52:31.047845+0000 (oldest deadline 2024-04-02T14:52:55.148007+0000) 2024-04-02T14:53:12.837 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:53:12 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:53:12.365+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-02T14:52:39.151607+0000 front 2024-04-02T14:52:39.151703+0000 (oldest deadline 2024-04-02T14:53:04.451174+0000) 2024-04-02T14:53:13.005 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 02 14:53:12 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-8[54680]: 2024-04-02T14:53:12.740+0000 7f64f23c8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:23.288870+0000 front 2024-04-02T14:52:23.288907+0000 (oldest deadline 2024-04-02T14:52:48.588938+0000) 2024-04-02T14:53:13.005 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 02 14:53:12 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-8[54680]: 2024-04-02T14:53:12.740+0000 7f64f23c8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:29.090126+0000 front 2024-04-02T14:52:29.090177+0000 (oldest deadline 2024-04-02T14:52:53.190215+0000) 2024-04-02T14:53:13.005 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 02 14:53:12 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-8[54680]: 2024-04-02T14:53:12.740+0000 7f64f23c8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-02T14:52:38.391857+0000 front 2024-04-02T14:52:38.391952+0000 (oldest deadline 2024-04-02T14:53:03.091726+0000) 2024-04-02T14:53:13.005 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 02 14:53:12 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-8[54680]: 2024-04-02T14:53:12.740+0000 7f64f23c8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.92:6812 osd.4 since back 2024-04-02T14:52:47.693204+0000 front 2024-04-02T14:52:47.693149+0000 (oldest deadline 2024-04-02T14:53:10.593118+0000) 2024-04-02T14:53:13.101 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 02 14:53:13 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-7[54644]: 2024-04-02T14:53:13.053+0000 7f0bb6c43700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:22.855156+0000 front 2024-04-02T14:52:22.855176+0000 (oldest deadline 2024-04-02T14:52:45.755296+0000) 2024-04-02T14:53:13.101 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 02 14:53:13 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-7[54644]: 2024-04-02T14:53:13.053+0000 7f0bb6c43700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:32.156922+0000 front 2024-04-02T14:52:32.157038+0000 (oldest deadline 2024-04-02T14:52:53.856857+0000) 2024-04-02T14:53:13.101 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 02 14:53:13 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-7[54644]: 2024-04-02T14:53:13.053+0000 7f0bb6c43700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-02T14:52:40.259022+0000 front 2024-04-02T14:52:40.258912+0000 (oldest deadline 2024-04-02T14:53:05.558692+0000) 2024-04-02T14:53:13.101 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 02 14:53:13 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-7[54644]: 2024-04-02T14:53:13.053+0000 7f0bb6c43700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.92:6812 osd.4 since back 2024-04-02T14:52:45.559190+0000 front 2024-04-02T14:52:45.559221+0000 (oldest deadline 2024-04-02T14:53:10.859288+0000) 2024-04-02T14:53:13.101 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:12 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:12.834+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:50.925796+0000 front 2024-04-02T14:51:50.925668+0000 (oldest deadline 2024-04-02T14:52:40.128984+0000) 2024-04-02T14:53:13.101 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:12 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:12.834+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:20.129325+0000 front 2024-04-02T14:52:20.129293+0000 (oldest deadline 2024-04-02T14:52:45.429431+0000) 2024-04-02T14:53:13.101 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:12 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:12.834+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:27.631081+0000 front 2024-04-02T14:52:27.631094+0000 (oldest deadline 2024-04-02T14:52:53.531189+0000) 2024-04-02T14:53:13.102 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:12 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:12.834+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-02T14:52:39.933034+0000 front 2024-04-02T14:52:39.933064+0000 (oldest deadline 2024-04-02T14:53:04.632770+0000) 2024-04-02T14:53:13.427 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:13 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:13.013+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:50.397138+0000 front 2024-04-02T14:51:50.397096+0000 (oldest deadline 2024-04-02T14:52:37.200213+0000) 2024-04-02T14:53:13.427 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:13 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:13.013+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:22.501322+0000 front 2024-04-02T14:52:22.501346+0000 (oldest deadline 2024-04-02T14:52:45.401451+0000) 2024-04-02T14:53:13.427 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:13 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:13.013+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:28.302204+0000 front 2024-04-02T14:52:28.302332+0000 (oldest deadline 2024-04-02T14:52:57.102868+0000) 2024-04-02T14:53:13.427 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:13 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:13.013+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-02T14:52:38.804314+0000 front 2024-04-02T14:52:38.804365+0000 (oldest deadline 2024-04-02T14:53:04.104362+0000) 2024-04-02T14:53:13.428 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:13 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:13.013+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.92:6812 osd.4 since back 2024-04-02T14:52:47.605500+0000 front 2024-04-02T14:52:47.605485+0000 (oldest deadline 2024-04-02T14:53:10.505428+0000) 2024-04-02T14:53:13.679 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:53:13 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:53:13.320+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:49.133761+0000 front 2024-04-02T14:51:49.134172+0000 (oldest deadline 2024-04-02T14:52:35.841983+0000) 2024-04-02T14:53:13.679 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:53:13 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:53:13.320+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:21.645685+0000 front 2024-04-02T14:52:21.645805+0000 (oldest deadline 2024-04-02T14:52:46.945889+0000) 2024-04-02T14:53:13.679 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:53:13 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:53:13.320+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:31.047892+0000 front 2024-04-02T14:52:31.047845+0000 (oldest deadline 2024-04-02T14:52:55.148007+0000) 2024-04-02T14:53:13.679 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:53:13 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:53:13.320+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-02T14:52:39.151607+0000 front 2024-04-02T14:52:39.151703+0000 (oldest deadline 2024-04-02T14:53:04.451174+0000) 2024-04-02T14:53:13.679 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:53:13 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:53:13.320+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.92:6812 osd.4 since back 2024-04-02T14:52:48.355136+0000 front 2024-04-02T14:52:48.355090+0000 (oldest deadline 2024-04-02T14:53:13.055332+0000) 2024-04-02T14:53:13.964 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 02 14:53:13 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-8[54680]: 2024-04-02T14:53:13.695+0000 7f64f23c8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:23.288870+0000 front 2024-04-02T14:52:23.288907+0000 (oldest deadline 2024-04-02T14:52:48.588938+0000) 2024-04-02T14:53:13.964 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 02 14:53:13 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-8[54680]: 2024-04-02T14:53:13.695+0000 7f64f23c8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:29.090126+0000 front 2024-04-02T14:52:29.090177+0000 (oldest deadline 2024-04-02T14:52:53.190215+0000) 2024-04-02T14:53:13.964 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 02 14:53:13 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-8[54680]: 2024-04-02T14:53:13.695+0000 7f64f23c8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-02T14:52:38.391857+0000 front 2024-04-02T14:52:38.391952+0000 (oldest deadline 2024-04-02T14:53:03.091726+0000) 2024-04-02T14:53:13.964 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 02 14:53:13 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-8[54680]: 2024-04-02T14:53:13.695+0000 7f64f23c8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.92:6812 osd.4 since back 2024-04-02T14:52:47.693204+0000 front 2024-04-02T14:52:47.693149+0000 (oldest deadline 2024-04-02T14:53:10.593118+0000) 2024-04-02T14:53:14.275 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:13 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:13.972+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:50.397138+0000 front 2024-04-02T14:51:50.397096+0000 (oldest deadline 2024-04-02T14:52:37.200213+0000) 2024-04-02T14:53:14.275 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:13 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:13.972+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:22.501322+0000 front 2024-04-02T14:52:22.501346+0000 (oldest deadline 2024-04-02T14:52:45.401451+0000) 2024-04-02T14:53:14.275 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:13 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:13.972+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:28.302204+0000 front 2024-04-02T14:52:28.302332+0000 (oldest deadline 2024-04-02T14:52:57.102868+0000) 2024-04-02T14:53:14.275 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:13 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:13.972+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-02T14:52:38.804314+0000 front 2024-04-02T14:52:38.804365+0000 (oldest deadline 2024-04-02T14:53:04.104362+0000) 2024-04-02T14:53:14.276 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:13 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:13.972+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.92:6812 osd.4 since back 2024-04-02T14:52:47.605500+0000 front 2024-04-02T14:52:47.605485+0000 (oldest deadline 2024-04-02T14:53:10.505428+0000) 2024-04-02T14:53:14.351 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 02 14:53:14 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-7[54644]: 2024-04-02T14:53:14.071+0000 7f0bb6c43700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:22.855156+0000 front 2024-04-02T14:52:22.855176+0000 (oldest deadline 2024-04-02T14:52:45.755296+0000) 2024-04-02T14:53:14.351 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 02 14:53:14 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-7[54644]: 2024-04-02T14:53:14.071+0000 7f0bb6c43700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:32.156922+0000 front 2024-04-02T14:52:32.157038+0000 (oldest deadline 2024-04-02T14:52:53.856857+0000) 2024-04-02T14:53:14.351 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 02 14:53:14 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-7[54644]: 2024-04-02T14:53:14.071+0000 7f0bb6c43700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-02T14:52:40.259022+0000 front 2024-04-02T14:52:40.258912+0000 (oldest deadline 2024-04-02T14:53:05.558692+0000) 2024-04-02T14:53:14.351 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 02 14:53:14 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-7[54644]: 2024-04-02T14:53:14.071+0000 7f0bb6c43700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.92:6812 osd.4 since back 2024-04-02T14:52:45.559190+0000 front 2024-04-02T14:52:45.559221+0000 (oldest deadline 2024-04-02T14:53:10.859288+0000) 2024-04-02T14:53:14.352 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:13 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:13.867+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:50.925796+0000 front 2024-04-02T14:51:50.925668+0000 (oldest deadline 2024-04-02T14:52:40.128984+0000) 2024-04-02T14:53:14.352 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:13 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:13.867+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:20.129325+0000 front 2024-04-02T14:52:20.129293+0000 (oldest deadline 2024-04-02T14:52:45.429431+0000) 2024-04-02T14:53:14.352 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:13 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:13.867+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:27.631081+0000 front 2024-04-02T14:52:27.631094+0000 (oldest deadline 2024-04-02T14:52:53.531189+0000) 2024-04-02T14:53:14.352 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:13 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:13.867+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-02T14:52:39.933034+0000 front 2024-04-02T14:52:39.933064+0000 (oldest deadline 2024-04-02T14:53:04.632770+0000) 2024-04-02T14:53:14.679 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:53:14 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:53:14.295+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:49.133761+0000 front 2024-04-02T14:51:49.134172+0000 (oldest deadline 2024-04-02T14:52:35.841983+0000) 2024-04-02T14:53:14.679 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:53:14 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:53:14.295+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:21.645685+0000 front 2024-04-02T14:52:21.645805+0000 (oldest deadline 2024-04-02T14:52:46.945889+0000) 2024-04-02T14:53:14.679 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:53:14 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:53:14.295+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:31.047892+0000 front 2024-04-02T14:52:31.047845+0000 (oldest deadline 2024-04-02T14:52:55.148007+0000) 2024-04-02T14:53:14.679 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:53:14 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:53:14.295+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-02T14:52:39.151607+0000 front 2024-04-02T14:52:39.151703+0000 (oldest deadline 2024-04-02T14:53:04.451174+0000) 2024-04-02T14:53:14.680 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:53:14 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:53:14.295+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.92:6812 osd.4 since back 2024-04-02T14:52:48.355136+0000 front 2024-04-02T14:52:48.355090+0000 (oldest deadline 2024-04-02T14:53:13.055332+0000) 2024-04-02T14:53:14.927 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 02 14:53:14 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-8[54680]: 2024-04-02T14:53:14.646+0000 7f64f23c8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:23.288870+0000 front 2024-04-02T14:52:23.288907+0000 (oldest deadline 2024-04-02T14:52:48.588938+0000) 2024-04-02T14:53:14.927 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 02 14:53:14 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-8[54680]: 2024-04-02T14:53:14.646+0000 7f64f23c8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:29.090126+0000 front 2024-04-02T14:52:29.090177+0000 (oldest deadline 2024-04-02T14:52:53.190215+0000) 2024-04-02T14:53:14.927 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 02 14:53:14 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-8[54680]: 2024-04-02T14:53:14.646+0000 7f64f23c8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-02T14:52:38.391857+0000 front 2024-04-02T14:52:38.391952+0000 (oldest deadline 2024-04-02T14:53:03.091726+0000) 2024-04-02T14:53:14.927 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 02 14:53:14 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-8[54680]: 2024-04-02T14:53:14.646+0000 7f64f23c8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.92:6812 osd.4 since back 2024-04-02T14:52:47.693204+0000 front 2024-04-02T14:52:47.693149+0000 (oldest deadline 2024-04-02T14:53:10.593118+0000) 2024-04-02T14:53:15.084 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:14 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:14.824+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:50.925796+0000 front 2024-04-02T14:51:50.925668+0000 (oldest deadline 2024-04-02T14:52:40.128984+0000) 2024-04-02T14:53:15.085 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:14 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:14.824+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:20.129325+0000 front 2024-04-02T14:52:20.129293+0000 (oldest deadline 2024-04-02T14:52:45.429431+0000) 2024-04-02T14:53:15.085 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:14 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:14.824+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:27.631081+0000 front 2024-04-02T14:52:27.631094+0000 (oldest deadline 2024-04-02T14:52:53.531189+0000) 2024-04-02T14:53:15.085 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:14 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:14.824+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-02T14:52:39.933034+0000 front 2024-04-02T14:52:39.933064+0000 (oldest deadline 2024-04-02T14:53:04.632770+0000) 2024-04-02T14:53:15.085 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:14 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:14.824+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.92:6812 osd.4 since back 2024-04-02T14:52:48.133729+0000 front 2024-04-02T14:52:48.133748+0000 (oldest deadline 2024-04-02T14:53:14.033931+0000) 2024-04-02T14:53:15.200 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:14 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:14.953+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:50.397138+0000 front 2024-04-02T14:51:50.397096+0000 (oldest deadline 2024-04-02T14:52:37.200213+0000) 2024-04-02T14:53:15.200 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:14 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:14.953+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:22.501322+0000 front 2024-04-02T14:52:22.501346+0000 (oldest deadline 2024-04-02T14:52:45.401451+0000) 2024-04-02T14:53:15.200 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:14 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:14.953+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:28.302204+0000 front 2024-04-02T14:52:28.302332+0000 (oldest deadline 2024-04-02T14:52:57.102868+0000) 2024-04-02T14:53:15.200 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:14 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:14.953+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-02T14:52:38.804314+0000 front 2024-04-02T14:52:38.804365+0000 (oldest deadline 2024-04-02T14:53:04.104362+0000) 2024-04-02T14:53:15.200 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:14 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:14.953+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.92:6812 osd.4 since back 2024-04-02T14:52:47.605500+0000 front 2024-04-02T14:52:47.605485+0000 (oldest deadline 2024-04-02T14:53:10.505428+0000) 2024-04-02T14:53:15.351 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 02 14:53:15 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-7[54644]: 2024-04-02T14:53:15.093+0000 7f0bb6c43700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:22.855156+0000 front 2024-04-02T14:52:22.855176+0000 (oldest deadline 2024-04-02T14:52:45.755296+0000) 2024-04-02T14:53:15.351 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 02 14:53:15 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-7[54644]: 2024-04-02T14:53:15.093+0000 7f0bb6c43700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:32.156922+0000 front 2024-04-02T14:52:32.157038+0000 (oldest deadline 2024-04-02T14:52:53.856857+0000) 2024-04-02T14:53:15.351 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 02 14:53:15 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-7[54644]: 2024-04-02T14:53:15.093+0000 7f0bb6c43700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-02T14:52:40.259022+0000 front 2024-04-02T14:52:40.258912+0000 (oldest deadline 2024-04-02T14:53:05.558692+0000) 2024-04-02T14:53:15.351 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 02 14:53:15 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-7[54644]: 2024-04-02T14:53:15.093+0000 7f0bb6c43700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.92:6812 osd.4 since back 2024-04-02T14:52:45.559190+0000 front 2024-04-02T14:52:45.559221+0000 (oldest deadline 2024-04-02T14:53:10.859288+0000) 2024-04-02T14:53:15.679 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:53:15 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:53:15.244+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:49.133761+0000 front 2024-04-02T14:51:49.134172+0000 (oldest deadline 2024-04-02T14:52:35.841983+0000) 2024-04-02T14:53:15.679 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:53:15 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:53:15.244+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:21.645685+0000 front 2024-04-02T14:52:21.645805+0000 (oldest deadline 2024-04-02T14:52:46.945889+0000) 2024-04-02T14:53:15.679 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:53:15 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:53:15.244+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:31.047892+0000 front 2024-04-02T14:52:31.047845+0000 (oldest deadline 2024-04-02T14:52:55.148007+0000) 2024-04-02T14:53:15.679 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:53:15 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:53:15.244+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-02T14:52:39.151607+0000 front 2024-04-02T14:52:39.151703+0000 (oldest deadline 2024-04-02T14:53:04.451174+0000) 2024-04-02T14:53:15.679 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:53:15 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:53:15.244+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.92:6812 osd.4 since back 2024-04-02T14:52:48.355136+0000 front 2024-04-02T14:52:48.355090+0000 (oldest deadline 2024-04-02T14:53:13.055332+0000) 2024-04-02T14:53:15.930 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 02 14:53:15 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-8[54680]: 2024-04-02T14:53:15.638+0000 7f64f23c8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:23.288870+0000 front 2024-04-02T14:52:23.288907+0000 (oldest deadline 2024-04-02T14:52:48.588938+0000) 2024-04-02T14:53:15.930 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 02 14:53:15 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-8[54680]: 2024-04-02T14:53:15.638+0000 7f64f23c8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:29.090126+0000 front 2024-04-02T14:52:29.090177+0000 (oldest deadline 2024-04-02T14:52:53.190215+0000) 2024-04-02T14:53:15.930 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 02 14:53:15 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-8[54680]: 2024-04-02T14:53:15.638+0000 7f64f23c8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-02T14:52:38.391857+0000 front 2024-04-02T14:52:38.391952+0000 (oldest deadline 2024-04-02T14:53:03.091726+0000) 2024-04-02T14:53:15.930 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 02 14:53:15 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-8[54680]: 2024-04-02T14:53:15.638+0000 7f64f23c8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.92:6812 osd.4 since back 2024-04-02T14:52:47.693204+0000 front 2024-04-02T14:52:47.693149+0000 (oldest deadline 2024-04-02T14:53:10.593118+0000) 2024-04-02T14:53:16.044 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 02 14:53:15 smithi092 podman[74346]: 2024-04-02 14:53:15.773405732 +0000 UTC m=+5.733895251 container died b6b4bdf8b23e6d269cc631ac14f0c62f4684a3c975cf5e7d905d0186d5f23636 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-7, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240326, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240402.004512-191d963, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-04-02T14:53:16.045 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:15 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:15.808+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:50.925796+0000 front 2024-04-02T14:51:50.925668+0000 (oldest deadline 2024-04-02T14:52:40.128984+0000) 2024-04-02T14:53:16.045 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:15 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:15.808+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:20.129325+0000 front 2024-04-02T14:52:20.129293+0000 (oldest deadline 2024-04-02T14:52:45.429431+0000) 2024-04-02T14:53:16.045 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:15 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:15.808+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:27.631081+0000 front 2024-04-02T14:52:27.631094+0000 (oldest deadline 2024-04-02T14:52:53.531189+0000) 2024-04-02T14:53:16.045 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:15 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:15.808+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-02T14:52:39.933034+0000 front 2024-04-02T14:52:39.933064+0000 (oldest deadline 2024-04-02T14:53:04.632770+0000) 2024-04-02T14:53:16.045 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:15 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:15.808+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.92:6812 osd.4 since back 2024-04-02T14:52:48.133729+0000 front 2024-04-02T14:52:48.133748+0000 (oldest deadline 2024-04-02T14:53:14.033931+0000) 2024-04-02T14:53:16.351 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 02 14:53:16 smithi092 podman[74346]: 2024-04-02 14:53:16.152663506 +0000 UTC m=+6.113153033 container cleanup b6b4bdf8b23e6d269cc631ac14f0c62f4684a3c975cf5e7d905d0186d5f23636 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-7, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=wip-batrick-testing-20240402.004512-191d963, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240326) 2024-04-02T14:53:16.351 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 02 14:53:16 smithi092 bash[74346]: ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-7 2024-04-02T14:53:16.427 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:15 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:15.961+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:50.397138+0000 front 2024-04-02T14:51:50.397096+0000 (oldest deadline 2024-04-02T14:52:37.200213+0000) 2024-04-02T14:53:16.427 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:15 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:15.961+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:22.501322+0000 front 2024-04-02T14:52:22.501346+0000 (oldest deadline 2024-04-02T14:52:45.401451+0000) 2024-04-02T14:53:16.427 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:15 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:15.961+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:28.302204+0000 front 2024-04-02T14:52:28.302332+0000 (oldest deadline 2024-04-02T14:52:57.102868+0000) 2024-04-02T14:53:16.427 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:15 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:15.961+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-02T14:52:38.804314+0000 front 2024-04-02T14:52:38.804365+0000 (oldest deadline 2024-04-02T14:53:04.104362+0000) 2024-04-02T14:53:16.427 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:15 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:15.961+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.92:6812 osd.4 since back 2024-04-02T14:52:47.605500+0000 front 2024-04-02T14:52:47.605485+0000 (oldest deadline 2024-04-02T14:53:10.505428+0000) 2024-04-02T14:53:16.679 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:53:16 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:53:16.232+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:49.133761+0000 front 2024-04-02T14:51:49.134172+0000 (oldest deadline 2024-04-02T14:52:35.841983+0000) 2024-04-02T14:53:16.679 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:53:16 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:53:16.232+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:21.645685+0000 front 2024-04-02T14:52:21.645805+0000 (oldest deadline 2024-04-02T14:52:46.945889+0000) 2024-04-02T14:53:16.679 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:53:16 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:53:16.232+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:31.047892+0000 front 2024-04-02T14:52:31.047845+0000 (oldest deadline 2024-04-02T14:52:55.148007+0000) 2024-04-02T14:53:16.679 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:53:16 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:53:16.232+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-02T14:52:39.151607+0000 front 2024-04-02T14:52:39.151703+0000 (oldest deadline 2024-04-02T14:53:04.451174+0000) 2024-04-02T14:53:16.679 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:53:16 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:53:16.232+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.92:6812 osd.4 since back 2024-04-02T14:52:48.355136+0000 front 2024-04-02T14:52:48.355090+0000 (oldest deadline 2024-04-02T14:53:13.055332+0000) 2024-04-02T14:53:16.796 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 02 14:53:16 smithi092 podman[74363]: 2024-04-02 14:53:16.454195196 +0000 UTC m=+0.684176166 container remove b6b4bdf8b23e6d269cc631ac14f0c62f4684a3c975cf5e7d905d0186d5f23636 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-7, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240402.004512-191d963, org.label-schema.build-date=20240326, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-04-02T14:53:16.927 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 02 14:53:16 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-8[54680]: 2024-04-02T14:53:16.610+0000 7f64f23c8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:23.288870+0000 front 2024-04-02T14:52:23.288907+0000 (oldest deadline 2024-04-02T14:52:48.588938+0000) 2024-04-02T14:53:16.927 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 02 14:53:16 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-8[54680]: 2024-04-02T14:53:16.610+0000 7f64f23c8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:29.090126+0000 front 2024-04-02T14:52:29.090177+0000 (oldest deadline 2024-04-02T14:52:53.190215+0000) 2024-04-02T14:53:16.927 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 02 14:53:16 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-8[54680]: 2024-04-02T14:53:16.610+0000 7f64f23c8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-02T14:52:38.391857+0000 front 2024-04-02T14:52:38.391952+0000 (oldest deadline 2024-04-02T14:53:03.091726+0000) 2024-04-02T14:53:16.927 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 02 14:53:16 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-8[54680]: 2024-04-02T14:53:16.610+0000 7f64f23c8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.92:6812 osd.4 since back 2024-04-02T14:52:47.693204+0000 front 2024-04-02T14:52:47.693149+0000 (oldest deadline 2024-04-02T14:53:10.593118+0000) 2024-04-02T14:53:17.073 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 02 14:53:16 smithi092 podman[74453]: 2024-04-02 14:53:16.805038207 +0000 UTC m=+0.110330766 container create 102c45ae79198db2e47db5201e8c98e60939818bee03542e8986c071e38d7a16 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-7-deactivate, RELEASE=wip-batrick-testing-20240402.004512-191d963, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240326) 2024-04-02T14:53:17.073 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 02 14:53:16 smithi092 podman[74453]: 2024-04-02 14:53:16.712668819 +0000 UTC m=+0.017961390 image pull f7301e070545fc23d8e5b7b052a2c15e832a0e57102f1acad8e4f3583709d23f quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534 2024-04-02T14:53:17.073 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 02 14:53:17 smithi092 podman[74453]: 2024-04-02 14:53:17.007357604 +0000 UTC m=+0.312650177 container init 102c45ae79198db2e47db5201e8c98e60939818bee03542e8986c071e38d7a16 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-7-deactivate, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=wip-batrick-testing-20240402.004512-191d963, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240326) 2024-04-02T14:53:17.073 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 02 14:53:17 smithi092 podman[74453]: 2024-04-02 14:53:17.010714512 +0000 UTC m=+0.316007082 container start 102c45ae79198db2e47db5201e8c98e60939818bee03542e8986c071e38d7a16 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-7-deactivate, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240326, RELEASE=wip-batrick-testing-20240402.004512-191d963, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-04-02T14:53:17.073 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 02 14:53:17 smithi092 podman[74453]: 2024-04-02 14:53:17.08218329 +0000 UTC m=+0.387475863 container attach 102c45ae79198db2e47db5201e8c98e60939818bee03542e8986c071e38d7a16 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-7-deactivate, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=wip-batrick-testing-20240402.004512-191d963, io.buildah.version=1.33.5, org.label-schema.build-date=20240326, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True) 2024-04-02T14:53:17.074 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:16 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:16.804+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:50.925796+0000 front 2024-04-02T14:51:50.925668+0000 (oldest deadline 2024-04-02T14:52:40.128984+0000) 2024-04-02T14:53:17.074 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:16 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:16.804+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:20.129325+0000 front 2024-04-02T14:52:20.129293+0000 (oldest deadline 2024-04-02T14:52:45.429431+0000) 2024-04-02T14:53:17.074 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:16 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:16.804+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:27.631081+0000 front 2024-04-02T14:52:27.631094+0000 (oldest deadline 2024-04-02T14:52:53.531189+0000) 2024-04-02T14:53:17.074 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:16 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:16.804+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-02T14:52:39.933034+0000 front 2024-04-02T14:52:39.933064+0000 (oldest deadline 2024-04-02T14:53:04.632770+0000) 2024-04-02T14:53:17.074 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:16 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:16.804+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.92:6812 osd.4 since back 2024-04-02T14:52:48.133729+0000 front 2024-04-02T14:52:48.133748+0000 (oldest deadline 2024-04-02T14:53:14.033931+0000) 2024-04-02T14:53:17.351 INFO:journalctl@ceph.osd.7.smithi092.stdout:Apr 02 14:53:17 smithi092 podman[74453]: 2024-04-02 14:53:17.299548116 +0000 UTC m=+0.604840684 container died 102c45ae79198db2e47db5201e8c98e60939818bee03542e8986c071e38d7a16 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-7-deactivate, org.label-schema.build-date=20240326, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240402.004512-191d963) 2024-04-02T14:53:17.427 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:17 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:17.003+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:50.397138+0000 front 2024-04-02T14:51:50.397096+0000 (oldest deadline 2024-04-02T14:52:37.200213+0000) 2024-04-02T14:53:17.427 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:17 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:17.003+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:22.501322+0000 front 2024-04-02T14:52:22.501346+0000 (oldest deadline 2024-04-02T14:52:45.401451+0000) 2024-04-02T14:53:17.427 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:17 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:17.003+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:28.302204+0000 front 2024-04-02T14:52:28.302332+0000 (oldest deadline 2024-04-02T14:52:57.102868+0000) 2024-04-02T14:53:17.427 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:17 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:17.003+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-02T14:52:38.804314+0000 front 2024-04-02T14:52:38.804365+0000 (oldest deadline 2024-04-02T14:53:04.104362+0000) 2024-04-02T14:53:17.427 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:17 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:17.003+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.92:6812 osd.4 since back 2024-04-02T14:52:47.605500+0000 front 2024-04-02T14:52:47.605485+0000 (oldest deadline 2024-04-02T14:53:10.505428+0000) 2024-04-02T14:53:17.679 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:53:17 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:53:17.204+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:49.133761+0000 front 2024-04-02T14:51:49.134172+0000 (oldest deadline 2024-04-02T14:52:35.841983+0000) 2024-04-02T14:53:17.679 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:53:17 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:53:17.204+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:21.645685+0000 front 2024-04-02T14:52:21.645805+0000 (oldest deadline 2024-04-02T14:52:46.945889+0000) 2024-04-02T14:53:17.679 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:53:17 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:53:17.204+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:31.047892+0000 front 2024-04-02T14:52:31.047845+0000 (oldest deadline 2024-04-02T14:52:55.148007+0000) 2024-04-02T14:53:17.679 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:53:17 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:53:17.204+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-02T14:52:39.151607+0000 front 2024-04-02T14:52:39.151703+0000 (oldest deadline 2024-04-02T14:53:04.451174+0000) 2024-04-02T14:53:17.679 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:53:17 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:53:17.204+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.92:6812 osd.4 since back 2024-04-02T14:52:48.355136+0000 front 2024-04-02T14:52:48.355090+0000 (oldest deadline 2024-04-02T14:53:13.055332+0000) 2024-04-02T14:53:17.851 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:17 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:17.797+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:50.925796+0000 front 2024-04-02T14:51:50.925668+0000 (oldest deadline 2024-04-02T14:52:40.128984+0000) 2024-04-02T14:53:17.851 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:17 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:17.797+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:20.129325+0000 front 2024-04-02T14:52:20.129293+0000 (oldest deadline 2024-04-02T14:52:45.429431+0000) 2024-04-02T14:53:17.851 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:17 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:17.797+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:27.631081+0000 front 2024-04-02T14:52:27.631094+0000 (oldest deadline 2024-04-02T14:52:53.531189+0000) 2024-04-02T14:53:17.851 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:17 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:17.797+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-02T14:52:39.933034+0000 front 2024-04-02T14:52:39.933064+0000 (oldest deadline 2024-04-02T14:53:04.632770+0000) 2024-04-02T14:53:17.852 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:17 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:17.797+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.92:6812 osd.4 since back 2024-04-02T14:52:48.133729+0000 front 2024-04-02T14:52:48.133748+0000 (oldest deadline 2024-04-02T14:53:14.033931+0000) 2024-04-02T14:53:17.877 DEBUG:teuthology.orchestra.run.smithi092:> sudo pkill -f 'journalctl -f -n 0 -u ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4@osd.7.service' 2024-04-02T14:53:17.919 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-02T14:53:17.919 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-04-02T14:53:17.919 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2024-04-02T14:53:17.920 DEBUG:teuthology.orchestra.run.smithi177:> sudo systemctl stop ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4@osd.8 2024-04-02T14:53:17.926 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 02 14:53:17 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-8[54680]: 2024-04-02T14:53:17.580+0000 7f64f23c8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:23.288870+0000 front 2024-04-02T14:52:23.288907+0000 (oldest deadline 2024-04-02T14:52:48.588938+0000) 2024-04-02T14:53:17.927 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 02 14:53:17 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-8[54680]: 2024-04-02T14:53:17.580+0000 7f64f23c8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:29.090126+0000 front 2024-04-02T14:52:29.090177+0000 (oldest deadline 2024-04-02T14:52:53.190215+0000) 2024-04-02T14:53:17.927 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 02 14:53:17 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-8[54680]: 2024-04-02T14:53:17.580+0000 7f64f23c8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-02T14:52:38.391857+0000 front 2024-04-02T14:52:38.391952+0000 (oldest deadline 2024-04-02T14:53:03.091726+0000) 2024-04-02T14:53:17.927 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 02 14:53:17 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-8[54680]: 2024-04-02T14:53:17.580+0000 7f64f23c8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.92:6812 osd.4 since back 2024-04-02T14:52:47.693204+0000 front 2024-04-02T14:52:47.693149+0000 (oldest deadline 2024-04-02T14:53:10.593118+0000) 2024-04-02T14:53:18.213 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 02 14:53:17 smithi177 systemd[1]: Stopping Ceph osd.8 for 6449448a-f0fc-11ee-b647-cb9ed24678a4... 2024-04-02T14:53:18.214 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:18 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:18.009+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:50.397138+0000 front 2024-04-02T14:51:50.397096+0000 (oldest deadline 2024-04-02T14:52:37.200213+0000) 2024-04-02T14:53:18.214 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:18 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:18.009+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:22.501322+0000 front 2024-04-02T14:52:22.501346+0000 (oldest deadline 2024-04-02T14:52:45.401451+0000) 2024-04-02T14:53:18.214 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:18 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:18.009+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:28.302204+0000 front 2024-04-02T14:52:28.302332+0000 (oldest deadline 2024-04-02T14:52:57.102868+0000) 2024-04-02T14:53:18.214 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:18 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:18.009+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-02T14:52:38.804314+0000 front 2024-04-02T14:52:38.804365+0000 (oldest deadline 2024-04-02T14:53:04.104362+0000) 2024-04-02T14:53:18.214 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:18 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:18.009+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.92:6812 osd.4 since back 2024-04-02T14:52:47.605500+0000 front 2024-04-02T14:52:47.605485+0000 (oldest deadline 2024-04-02T14:53:10.505428+0000) 2024-04-02T14:53:18.594 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 02 14:53:18 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-8[54680]: 2024-04-02T14:53:18.221+0000 7f64f9ddc700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-02T14:53:18.594 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 02 14:53:18 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-8[54680]: 2024-04-02T14:53:18.221+0000 7f64f9ddc700 -1 osd.8 114 *** Got signal Terminated *** 2024-04-02T14:53:18.594 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 02 14:53:18 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-8[54680]: 2024-04-02T14:53:18.221+0000 7f64f9ddc700 -1 osd.8 114 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-02T14:53:18.679 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:53:18 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:53:18.244+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:49.133761+0000 front 2024-04-02T14:51:49.134172+0000 (oldest deadline 2024-04-02T14:52:35.841983+0000) 2024-04-02T14:53:18.679 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:53:18 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:53:18.244+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:21.645685+0000 front 2024-04-02T14:52:21.645805+0000 (oldest deadline 2024-04-02T14:52:46.945889+0000) 2024-04-02T14:53:18.679 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:53:18 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:53:18.244+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:31.047892+0000 front 2024-04-02T14:52:31.047845+0000 (oldest deadline 2024-04-02T14:52:55.148007+0000) 2024-04-02T14:53:18.679 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:53:18 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:53:18.244+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-02T14:52:39.151607+0000 front 2024-04-02T14:52:39.151703+0000 (oldest deadline 2024-04-02T14:53:04.451174+0000) 2024-04-02T14:53:18.679 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:53:18 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:53:18.244+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.92:6812 osd.4 since back 2024-04-02T14:52:48.355136+0000 front 2024-04-02T14:52:48.355090+0000 (oldest deadline 2024-04-02T14:53:13.055332+0000) 2024-04-02T14:53:18.927 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 02 14:53:18 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-8[54680]: 2024-04-02T14:53:18.601+0000 7f64f23c8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:23.288870+0000 front 2024-04-02T14:52:23.288907+0000 (oldest deadline 2024-04-02T14:52:48.588938+0000) 2024-04-02T14:53:18.927 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 02 14:53:18 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-8[54680]: 2024-04-02T14:53:18.601+0000 7f64f23c8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:29.090126+0000 front 2024-04-02T14:52:29.090177+0000 (oldest deadline 2024-04-02T14:52:53.190215+0000) 2024-04-02T14:53:18.927 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 02 14:53:18 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-8[54680]: 2024-04-02T14:53:18.601+0000 7f64f23c8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-02T14:52:38.391857+0000 front 2024-04-02T14:52:38.391952+0000 (oldest deadline 2024-04-02T14:53:03.091726+0000) 2024-04-02T14:53:18.927 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 02 14:53:18 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-8[54680]: 2024-04-02T14:53:18.601+0000 7f64f23c8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.92:6812 osd.4 since back 2024-04-02T14:52:47.693204+0000 front 2024-04-02T14:52:47.693149+0000 (oldest deadline 2024-04-02T14:53:10.593118+0000) 2024-04-02T14:53:19.101 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:18 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:18.773+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:50.925796+0000 front 2024-04-02T14:51:50.925668+0000 (oldest deadline 2024-04-02T14:52:40.128984+0000) 2024-04-02T14:53:19.101 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:18 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:18.773+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:20.129325+0000 front 2024-04-02T14:52:20.129293+0000 (oldest deadline 2024-04-02T14:52:45.429431+0000) 2024-04-02T14:53:19.101 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:18 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:18.773+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:27.631081+0000 front 2024-04-02T14:52:27.631094+0000 (oldest deadline 2024-04-02T14:52:53.531189+0000) 2024-04-02T14:53:19.101 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:18 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:18.773+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-02T14:52:39.933034+0000 front 2024-04-02T14:52:39.933064+0000 (oldest deadline 2024-04-02T14:53:04.632770+0000) 2024-04-02T14:53:19.101 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:18 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:18.773+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.92:6812 osd.4 since back 2024-04-02T14:52:48.133729+0000 front 2024-04-02T14:52:48.133748+0000 (oldest deadline 2024-04-02T14:53:14.033931+0000) 2024-04-02T14:53:19.427 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:19 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:19.039+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:50.397138+0000 front 2024-04-02T14:51:50.397096+0000 (oldest deadline 2024-04-02T14:52:37.200213+0000) 2024-04-02T14:53:19.427 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:19 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:19.039+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:22.501322+0000 front 2024-04-02T14:52:22.501346+0000 (oldest deadline 2024-04-02T14:52:45.401451+0000) 2024-04-02T14:53:19.427 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:19 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:19.039+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:28.302204+0000 front 2024-04-02T14:52:28.302332+0000 (oldest deadline 2024-04-02T14:52:57.102868+0000) 2024-04-02T14:53:19.427 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:19 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:19.039+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-02T14:52:38.804314+0000 front 2024-04-02T14:52:38.804365+0000 (oldest deadline 2024-04-02T14:53:04.104362+0000) 2024-04-02T14:53:19.427 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:19 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:19.039+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.92:6812 osd.4 since back 2024-04-02T14:52:47.605500+0000 front 2024-04-02T14:52:47.605485+0000 (oldest deadline 2024-04-02T14:53:10.505428+0000) 2024-04-02T14:53:19.429 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:53:19 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:53:19.277+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:49.133761+0000 front 2024-04-02T14:51:49.134172+0000 (oldest deadline 2024-04-02T14:52:35.841983+0000) 2024-04-02T14:53:19.429 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:53:19 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:53:19.277+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:21.645685+0000 front 2024-04-02T14:52:21.645805+0000 (oldest deadline 2024-04-02T14:52:46.945889+0000) 2024-04-02T14:53:19.429 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:53:19 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:53:19.277+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:31.047892+0000 front 2024-04-02T14:52:31.047845+0000 (oldest deadline 2024-04-02T14:52:55.148007+0000) 2024-04-02T14:53:19.429 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:53:19 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:53:19.277+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-02T14:52:39.151607+0000 front 2024-04-02T14:52:39.151703+0000 (oldest deadline 2024-04-02T14:53:04.451174+0000) 2024-04-02T14:53:19.429 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:53:19 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:53:19.277+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.92:6812 osd.4 since back 2024-04-02T14:52:48.355136+0000 front 2024-04-02T14:52:48.355090+0000 (oldest deadline 2024-04-02T14:53:13.055332+0000) 2024-04-02T14:53:19.927 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 02 14:53:19 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-8[54680]: 2024-04-02T14:53:19.583+0000 7f64f23c8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:23.288870+0000 front 2024-04-02T14:52:23.288907+0000 (oldest deadline 2024-04-02T14:52:48.588938+0000) 2024-04-02T14:53:19.927 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 02 14:53:19 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-8[54680]: 2024-04-02T14:53:19.583+0000 7f64f23c8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:29.090126+0000 front 2024-04-02T14:52:29.090177+0000 (oldest deadline 2024-04-02T14:52:53.190215+0000) 2024-04-02T14:53:19.927 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 02 14:53:19 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-8[54680]: 2024-04-02T14:53:19.583+0000 7f64f23c8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-02T14:52:38.391857+0000 front 2024-04-02T14:52:38.391952+0000 (oldest deadline 2024-04-02T14:53:03.091726+0000) 2024-04-02T14:53:19.927 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 02 14:53:19 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-8[54680]: 2024-04-02T14:53:19.583+0000 7f64f23c8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.92:6812 osd.4 since back 2024-04-02T14:52:47.693204+0000 front 2024-04-02T14:52:47.693149+0000 (oldest deadline 2024-04-02T14:53:10.593118+0000) 2024-04-02T14:53:20.101 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:19 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:19.754+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:50.925796+0000 front 2024-04-02T14:51:50.925668+0000 (oldest deadline 2024-04-02T14:52:40.128984+0000) 2024-04-02T14:53:20.101 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:19 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:19.754+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:20.129325+0000 front 2024-04-02T14:52:20.129293+0000 (oldest deadline 2024-04-02T14:52:45.429431+0000) 2024-04-02T14:53:20.101 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:19 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:19.754+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:27.631081+0000 front 2024-04-02T14:52:27.631094+0000 (oldest deadline 2024-04-02T14:52:53.531189+0000) 2024-04-02T14:53:20.101 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:19 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:19.754+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-02T14:52:39.933034+0000 front 2024-04-02T14:52:39.933064+0000 (oldest deadline 2024-04-02T14:53:04.632770+0000) 2024-04-02T14:53:20.101 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:19 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:19.754+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.92:6812 osd.4 since back 2024-04-02T14:52:48.133729+0000 front 2024-04-02T14:52:48.133748+0000 (oldest deadline 2024-04-02T14:53:14.033931+0000) 2024-04-02T14:53:20.427 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:20 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:20.053+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:50.397138+0000 front 2024-04-02T14:51:50.397096+0000 (oldest deadline 2024-04-02T14:52:37.200213+0000) 2024-04-02T14:53:20.427 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:20 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:20.053+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:22.501322+0000 front 2024-04-02T14:52:22.501346+0000 (oldest deadline 2024-04-02T14:52:45.401451+0000) 2024-04-02T14:53:20.427 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:20 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:20.053+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:28.302204+0000 front 2024-04-02T14:52:28.302332+0000 (oldest deadline 2024-04-02T14:52:57.102868+0000) 2024-04-02T14:53:20.427 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:20 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:20.053+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-02T14:52:38.804314+0000 front 2024-04-02T14:52:38.804365+0000 (oldest deadline 2024-04-02T14:53:04.104362+0000) 2024-04-02T14:53:20.427 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:20 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:20.053+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.92:6812 osd.4 since back 2024-04-02T14:52:47.605500+0000 front 2024-04-02T14:52:47.605485+0000 (oldest deadline 2024-04-02T14:53:10.505428+0000) 2024-04-02T14:53:20.679 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:53:20 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:53:20.311+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:49.133761+0000 front 2024-04-02T14:51:49.134172+0000 (oldest deadline 2024-04-02T14:52:35.841983+0000) 2024-04-02T14:53:20.679 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:53:20 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:53:20.311+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:21.645685+0000 front 2024-04-02T14:52:21.645805+0000 (oldest deadline 2024-04-02T14:52:46.945889+0000) 2024-04-02T14:53:20.679 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:53:20 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:53:20.311+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:31.047892+0000 front 2024-04-02T14:52:31.047845+0000 (oldest deadline 2024-04-02T14:52:55.148007+0000) 2024-04-02T14:53:20.679 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:53:20 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:53:20.311+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-02T14:52:39.151607+0000 front 2024-04-02T14:52:39.151703+0000 (oldest deadline 2024-04-02T14:53:04.451174+0000) 2024-04-02T14:53:20.679 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:53:20 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:53:20.311+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.92:6812 osd.4 since back 2024-04-02T14:52:48.355136+0000 front 2024-04-02T14:52:48.355090+0000 (oldest deadline 2024-04-02T14:53:13.055332+0000) 2024-04-02T14:53:20.927 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 02 14:53:20 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-8[54680]: 2024-04-02T14:53:20.591+0000 7f64f23c8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:23.288870+0000 front 2024-04-02T14:52:23.288907+0000 (oldest deadline 2024-04-02T14:52:48.588938+0000) 2024-04-02T14:53:20.927 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 02 14:53:20 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-8[54680]: 2024-04-02T14:53:20.591+0000 7f64f23c8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:29.090126+0000 front 2024-04-02T14:52:29.090177+0000 (oldest deadline 2024-04-02T14:52:53.190215+0000) 2024-04-02T14:53:20.927 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 02 14:53:20 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-8[54680]: 2024-04-02T14:53:20.591+0000 7f64f23c8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-02T14:52:38.391857+0000 front 2024-04-02T14:52:38.391952+0000 (oldest deadline 2024-04-02T14:53:03.091726+0000) 2024-04-02T14:53:20.927 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 02 14:53:20 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-8[54680]: 2024-04-02T14:53:20.591+0000 7f64f23c8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.92:6812 osd.4 since back 2024-04-02T14:52:47.693204+0000 front 2024-04-02T14:52:47.693149+0000 (oldest deadline 2024-04-02T14:53:10.593118+0000) 2024-04-02T14:53:20.927 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 02 14:53:20 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-8[54680]: 2024-04-02T14:53:20.591+0000 7f64f23c8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-04-02T14:52:56.994449+0000 front 2024-04-02T14:52:56.994480+0000 (oldest deadline 2024-04-02T14:53:20.494617+0000) 2024-04-02T14:53:21.101 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:20 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:20.788+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:50.925796+0000 front 2024-04-02T14:51:50.925668+0000 (oldest deadline 2024-04-02T14:52:40.128984+0000) 2024-04-02T14:53:21.101 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:20 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:20.788+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:20.129325+0000 front 2024-04-02T14:52:20.129293+0000 (oldest deadline 2024-04-02T14:52:45.429431+0000) 2024-04-02T14:53:21.101 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:20 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:20.788+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:27.631081+0000 front 2024-04-02T14:52:27.631094+0000 (oldest deadline 2024-04-02T14:52:53.531189+0000) 2024-04-02T14:53:21.101 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:20 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:20.788+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-02T14:52:39.933034+0000 front 2024-04-02T14:52:39.933064+0000 (oldest deadline 2024-04-02T14:53:04.632770+0000) 2024-04-02T14:53:21.101 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:20 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:20.788+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.92:6812 osd.4 since back 2024-04-02T14:52:48.133729+0000 front 2024-04-02T14:52:48.133748+0000 (oldest deadline 2024-04-02T14:53:14.033931+0000) 2024-04-02T14:53:21.427 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:21 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:21.023+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:50.397138+0000 front 2024-04-02T14:51:50.397096+0000 (oldest deadline 2024-04-02T14:52:37.200213+0000) 2024-04-02T14:53:21.427 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:21 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:21.023+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:22.501322+0000 front 2024-04-02T14:52:22.501346+0000 (oldest deadline 2024-04-02T14:52:45.401451+0000) 2024-04-02T14:53:21.427 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:21 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:21.023+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:28.302204+0000 front 2024-04-02T14:52:28.302332+0000 (oldest deadline 2024-04-02T14:52:57.102868+0000) 2024-04-02T14:53:21.427 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:21 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:21.023+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-02T14:52:38.804314+0000 front 2024-04-02T14:52:38.804365+0000 (oldest deadline 2024-04-02T14:53:04.104362+0000) 2024-04-02T14:53:21.427 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:21 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:21.023+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.92:6812 osd.4 since back 2024-04-02T14:52:47.605500+0000 front 2024-04-02T14:52:47.605485+0000 (oldest deadline 2024-04-02T14:53:10.505428+0000) 2024-04-02T14:53:21.427 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:21 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:21.023+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-04-02T14:52:56.306832+0000 front 2024-04-02T14:52:56.306765+0000 (oldest deadline 2024-04-02T14:53:20.907329+0000) 2024-04-02T14:53:21.679 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:53:21 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:53:21.335+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:49.133761+0000 front 2024-04-02T14:51:49.134172+0000 (oldest deadline 2024-04-02T14:52:35.841983+0000) 2024-04-02T14:53:21.679 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:53:21 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:53:21.335+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:21.645685+0000 front 2024-04-02T14:52:21.645805+0000 (oldest deadline 2024-04-02T14:52:46.945889+0000) 2024-04-02T14:53:21.679 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:53:21 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:53:21.335+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:31.047892+0000 front 2024-04-02T14:52:31.047845+0000 (oldest deadline 2024-04-02T14:52:55.148007+0000) 2024-04-02T14:53:21.679 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:53:21 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:53:21.335+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-02T14:52:39.151607+0000 front 2024-04-02T14:52:39.151703+0000 (oldest deadline 2024-04-02T14:53:04.451174+0000) 2024-04-02T14:53:21.679 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:53:21 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:53:21.335+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.92:6812 osd.4 since back 2024-04-02T14:52:48.355136+0000 front 2024-04-02T14:52:48.355090+0000 (oldest deadline 2024-04-02T14:53:13.055332+0000) 2024-04-02T14:53:21.927 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 02 14:53:21 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-8[54680]: 2024-04-02T14:53:21.615+0000 7f64f23c8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:23.288870+0000 front 2024-04-02T14:52:23.288907+0000 (oldest deadline 2024-04-02T14:52:48.588938+0000) 2024-04-02T14:53:21.927 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 02 14:53:21 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-8[54680]: 2024-04-02T14:53:21.615+0000 7f64f23c8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:29.090126+0000 front 2024-04-02T14:52:29.090177+0000 (oldest deadline 2024-04-02T14:52:53.190215+0000) 2024-04-02T14:53:21.927 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 02 14:53:21 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-8[54680]: 2024-04-02T14:53:21.615+0000 7f64f23c8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-02T14:52:38.391857+0000 front 2024-04-02T14:52:38.391952+0000 (oldest deadline 2024-04-02T14:53:03.091726+0000) 2024-04-02T14:53:21.927 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 02 14:53:21 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-8[54680]: 2024-04-02T14:53:21.615+0000 7f64f23c8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.92:6812 osd.4 since back 2024-04-02T14:52:47.693204+0000 front 2024-04-02T14:52:47.693149+0000 (oldest deadline 2024-04-02T14:53:10.593118+0000) 2024-04-02T14:53:21.927 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 02 14:53:21 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-8[54680]: 2024-04-02T14:53:21.615+0000 7f64f23c8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-04-02T14:52:56.994449+0000 front 2024-04-02T14:52:56.994480+0000 (oldest deadline 2024-04-02T14:53:20.494617+0000) 2024-04-02T14:53:22.101 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:21 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:21.766+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:50.925796+0000 front 2024-04-02T14:51:50.925668+0000 (oldest deadline 2024-04-02T14:52:40.128984+0000) 2024-04-02T14:53:22.101 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:21 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:21.766+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:20.129325+0000 front 2024-04-02T14:52:20.129293+0000 (oldest deadline 2024-04-02T14:52:45.429431+0000) 2024-04-02T14:53:22.101 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:21 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:21.766+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:27.631081+0000 front 2024-04-02T14:52:27.631094+0000 (oldest deadline 2024-04-02T14:52:53.531189+0000) 2024-04-02T14:53:22.101 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:21 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:21.766+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-02T14:52:39.933034+0000 front 2024-04-02T14:52:39.933064+0000 (oldest deadline 2024-04-02T14:53:04.632770+0000) 2024-04-02T14:53:22.101 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:21 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:21.766+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.92:6812 osd.4 since back 2024-04-02T14:52:48.133729+0000 front 2024-04-02T14:52:48.133748+0000 (oldest deadline 2024-04-02T14:53:14.033931+0000) 2024-04-02T14:53:22.427 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:22 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:22.062+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:50.397138+0000 front 2024-04-02T14:51:50.397096+0000 (oldest deadline 2024-04-02T14:52:37.200213+0000) 2024-04-02T14:53:22.427 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:22 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:22.062+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:22.501322+0000 front 2024-04-02T14:52:22.501346+0000 (oldest deadline 2024-04-02T14:52:45.401451+0000) 2024-04-02T14:53:22.427 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:22 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:22.062+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:28.302204+0000 front 2024-04-02T14:52:28.302332+0000 (oldest deadline 2024-04-02T14:52:57.102868+0000) 2024-04-02T14:53:22.427 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:22 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:22.062+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-02T14:52:38.804314+0000 front 2024-04-02T14:52:38.804365+0000 (oldest deadline 2024-04-02T14:53:04.104362+0000) 2024-04-02T14:53:22.428 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:22 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:22.062+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.92:6812 osd.4 since back 2024-04-02T14:52:47.605500+0000 front 2024-04-02T14:52:47.605485+0000 (oldest deadline 2024-04-02T14:53:10.505428+0000) 2024-04-02T14:53:22.428 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:22 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:22.062+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-04-02T14:52:56.306832+0000 front 2024-04-02T14:52:56.306765+0000 (oldest deadline 2024-04-02T14:53:20.907329+0000) 2024-04-02T14:53:22.682 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:53:22 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:53:22.308+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:49.133761+0000 front 2024-04-02T14:51:49.134172+0000 (oldest deadline 2024-04-02T14:52:35.841983+0000) 2024-04-02T14:53:22.682 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:53:22 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:53:22.308+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:21.645685+0000 front 2024-04-02T14:52:21.645805+0000 (oldest deadline 2024-04-02T14:52:46.945889+0000) 2024-04-02T14:53:22.682 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:53:22 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:53:22.308+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:31.047892+0000 front 2024-04-02T14:52:31.047845+0000 (oldest deadline 2024-04-02T14:52:55.148007+0000) 2024-04-02T14:53:22.682 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:53:22 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:53:22.308+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-02T14:52:39.151607+0000 front 2024-04-02T14:52:39.151703+0000 (oldest deadline 2024-04-02T14:53:04.451174+0000) 2024-04-02T14:53:22.682 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:53:22 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:53:22.308+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.92:6812 osd.4 since back 2024-04-02T14:52:48.355136+0000 front 2024-04-02T14:52:48.355090+0000 (oldest deadline 2024-04-02T14:53:13.055332+0000) 2024-04-02T14:53:22.927 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 02 14:53:22 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-8[54680]: 2024-04-02T14:53:22.620+0000 7f64f23c8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:23.288870+0000 front 2024-04-02T14:52:23.288907+0000 (oldest deadline 2024-04-02T14:52:48.588938+0000) 2024-04-02T14:53:22.927 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 02 14:53:22 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-8[54680]: 2024-04-02T14:53:22.620+0000 7f64f23c8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:29.090126+0000 front 2024-04-02T14:52:29.090177+0000 (oldest deadline 2024-04-02T14:52:53.190215+0000) 2024-04-02T14:53:22.927 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 02 14:53:22 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-8[54680]: 2024-04-02T14:53:22.620+0000 7f64f23c8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-02T14:52:38.391857+0000 front 2024-04-02T14:52:38.391952+0000 (oldest deadline 2024-04-02T14:53:03.091726+0000) 2024-04-02T14:53:22.927 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 02 14:53:22 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-8[54680]: 2024-04-02T14:53:22.620+0000 7f64f23c8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.92:6812 osd.4 since back 2024-04-02T14:52:47.693204+0000 front 2024-04-02T14:52:47.693149+0000 (oldest deadline 2024-04-02T14:53:10.593118+0000) 2024-04-02T14:53:22.927 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 02 14:53:22 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-8[54680]: 2024-04-02T14:53:22.620+0000 7f64f23c8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-04-02T14:52:56.994449+0000 front 2024-04-02T14:52:56.994480+0000 (oldest deadline 2024-04-02T14:53:20.494617+0000) 2024-04-02T14:53:23.101 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:22 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:22.799+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:50.925796+0000 front 2024-04-02T14:51:50.925668+0000 (oldest deadline 2024-04-02T14:52:40.128984+0000) 2024-04-02T14:53:23.101 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:22 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:22.799+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:20.129325+0000 front 2024-04-02T14:52:20.129293+0000 (oldest deadline 2024-04-02T14:52:45.429431+0000) 2024-04-02T14:53:23.101 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:22 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:22.799+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:27.631081+0000 front 2024-04-02T14:52:27.631094+0000 (oldest deadline 2024-04-02T14:52:53.531189+0000) 2024-04-02T14:53:23.101 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:22 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:22.799+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-02T14:52:39.933034+0000 front 2024-04-02T14:52:39.933064+0000 (oldest deadline 2024-04-02T14:53:04.632770+0000) 2024-04-02T14:53:23.101 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:22 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:22.799+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.92:6812 osd.4 since back 2024-04-02T14:52:48.133729+0000 front 2024-04-02T14:52:48.133748+0000 (oldest deadline 2024-04-02T14:53:14.033931+0000) 2024-04-02T14:53:23.427 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:23 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:23.060+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:50.397138+0000 front 2024-04-02T14:51:50.397096+0000 (oldest deadline 2024-04-02T14:52:37.200213+0000) 2024-04-02T14:53:23.427 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:23 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:23.060+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:22.501322+0000 front 2024-04-02T14:52:22.501346+0000 (oldest deadline 2024-04-02T14:52:45.401451+0000) 2024-04-02T14:53:23.427 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:23 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:23.060+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:28.302204+0000 front 2024-04-02T14:52:28.302332+0000 (oldest deadline 2024-04-02T14:52:57.102868+0000) 2024-04-02T14:53:23.427 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:23 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:23.060+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-02T14:52:38.804314+0000 front 2024-04-02T14:52:38.804365+0000 (oldest deadline 2024-04-02T14:53:04.104362+0000) 2024-04-02T14:53:23.427 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:23 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:23.060+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.92:6812 osd.4 since back 2024-04-02T14:52:47.605500+0000 front 2024-04-02T14:52:47.605485+0000 (oldest deadline 2024-04-02T14:53:10.505428+0000) 2024-04-02T14:53:23.427 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:23 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:23.060+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-04-02T14:52:56.306832+0000 front 2024-04-02T14:52:56.306765+0000 (oldest deadline 2024-04-02T14:53:20.907329+0000) 2024-04-02T14:53:23.679 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:53:23 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:53:23.263+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:49.133761+0000 front 2024-04-02T14:51:49.134172+0000 (oldest deadline 2024-04-02T14:52:35.841983+0000) 2024-04-02T14:53:23.679 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:53:23 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:53:23.263+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:21.645685+0000 front 2024-04-02T14:52:21.645805+0000 (oldest deadline 2024-04-02T14:52:46.945889+0000) 2024-04-02T14:53:23.679 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:53:23 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:53:23.263+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:31.047892+0000 front 2024-04-02T14:52:31.047845+0000 (oldest deadline 2024-04-02T14:52:55.148007+0000) 2024-04-02T14:53:23.679 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:53:23 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:53:23.263+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-02T14:52:39.151607+0000 front 2024-04-02T14:52:39.151703+0000 (oldest deadline 2024-04-02T14:53:04.451174+0000) 2024-04-02T14:53:23.679 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:53:23 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:53:23.263+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.92:6812 osd.4 since back 2024-04-02T14:52:48.355136+0000 front 2024-04-02T14:52:48.355090+0000 (oldest deadline 2024-04-02T14:53:13.055332+0000) 2024-04-02T14:53:23.680 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:53:23 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:53:23.263+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-04-02T14:52:57.757258+0000 front 2024-04-02T14:52:57.757335+0000 (oldest deadline 2024-04-02T14:53:23.057240+0000) 2024-04-02T14:53:24.037 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 02 14:53:23 smithi177 podman[74043]: 2024-04-02 14:53:23.706402632 +0000 UTC m=+5.581206726 container died 8c3e437d3784083f36fe0b20108654c1c03f96e958961ad67717e6afa0b0feb3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-8, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240326, org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240402.004512-191d963) 2024-04-02T14:53:24.101 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:23 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:23.802+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:50.925796+0000 front 2024-04-02T14:51:50.925668+0000 (oldest deadline 2024-04-02T14:52:40.128984+0000) 2024-04-02T14:53:24.101 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:23 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:23.802+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:20.129325+0000 front 2024-04-02T14:52:20.129293+0000 (oldest deadline 2024-04-02T14:52:45.429431+0000) 2024-04-02T14:53:24.101 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:23 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:23.802+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:27.631081+0000 front 2024-04-02T14:52:27.631094+0000 (oldest deadline 2024-04-02T14:52:53.531189+0000) 2024-04-02T14:53:24.101 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:23 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:23.802+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-02T14:52:39.933034+0000 front 2024-04-02T14:52:39.933064+0000 (oldest deadline 2024-04-02T14:53:04.632770+0000) 2024-04-02T14:53:24.101 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:23 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:23.802+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.92:6812 osd.4 since back 2024-04-02T14:52:48.133729+0000 front 2024-04-02T14:52:48.133748+0000 (oldest deadline 2024-04-02T14:53:14.033931+0000) 2024-04-02T14:53:24.101 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:23 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:23.802+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-04-02T14:52:54.034568+0000 front 2024-04-02T14:52:54.034525+0000 (oldest deadline 2024-04-02T14:53:23.434953+0000) 2024-04-02T14:53:24.427 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 02 14:53:24 smithi177 podman[74043]: 2024-04-02 14:53:24.0464174 +0000 UTC m=+5.921221488 container cleanup 8c3e437d3784083f36fe0b20108654c1c03f96e958961ad67717e6afa0b0feb3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-8, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240326, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=wip-batrick-testing-20240402.004512-191d963) 2024-04-02T14:53:24.427 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 02 14:53:24 smithi177 bash[74043]: ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-8 2024-04-02T14:53:24.427 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 02 14:53:24 smithi177 podman[74060]: 2024-04-02 14:53:24.264317729 +0000 UTC m=+0.556464795 container remove 8c3e437d3784083f36fe0b20108654c1c03f96e958961ad67717e6afa0b0feb3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-8, RELEASE=wip-batrick-testing-20240402.004512-191d963, org.label-schema.build-date=20240326, io.buildah.version=1.33.5, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-04-02T14:53:24.427 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:24 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:24.104+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:50.397138+0000 front 2024-04-02T14:51:50.397096+0000 (oldest deadline 2024-04-02T14:52:37.200213+0000) 2024-04-02T14:53:24.427 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:24 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:24.104+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:22.501322+0000 front 2024-04-02T14:52:22.501346+0000 (oldest deadline 2024-04-02T14:52:45.401451+0000) 2024-04-02T14:53:24.427 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:24 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:24.104+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:28.302204+0000 front 2024-04-02T14:52:28.302332+0000 (oldest deadline 2024-04-02T14:52:57.102868+0000) 2024-04-02T14:53:24.427 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:24 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:24.104+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-02T14:52:38.804314+0000 front 2024-04-02T14:52:38.804365+0000 (oldest deadline 2024-04-02T14:53:04.104362+0000) 2024-04-02T14:53:24.427 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:24 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:24.104+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.92:6812 osd.4 since back 2024-04-02T14:52:47.605500+0000 front 2024-04-02T14:52:47.605485+0000 (oldest deadline 2024-04-02T14:53:10.505428+0000) 2024-04-02T14:53:24.428 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:24 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:24.104+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-04-02T14:52:56.306832+0000 front 2024-04-02T14:52:56.306765+0000 (oldest deadline 2024-04-02T14:53:20.907329+0000) 2024-04-02T14:53:24.679 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:53:24 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:53:24.307+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:49.133761+0000 front 2024-04-02T14:51:49.134172+0000 (oldest deadline 2024-04-02T14:52:35.841983+0000) 2024-04-02T14:53:24.679 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:53:24 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:53:24.307+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:21.645685+0000 front 2024-04-02T14:52:21.645805+0000 (oldest deadline 2024-04-02T14:52:46.945889+0000) 2024-04-02T14:53:24.679 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:53:24 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:53:24.307+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:31.047892+0000 front 2024-04-02T14:52:31.047845+0000 (oldest deadline 2024-04-02T14:52:55.148007+0000) 2024-04-02T14:53:24.679 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:53:24 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:53:24.307+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-02T14:52:39.151607+0000 front 2024-04-02T14:52:39.151703+0000 (oldest deadline 2024-04-02T14:53:04.451174+0000) 2024-04-02T14:53:24.679 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:53:24 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:53:24.307+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.92:6812 osd.4 since back 2024-04-02T14:52:48.355136+0000 front 2024-04-02T14:52:48.355090+0000 (oldest deadline 2024-04-02T14:53:13.055332+0000) 2024-04-02T14:53:24.679 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:53:24 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:53:24.307+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-04-02T14:52:57.757258+0000 front 2024-04-02T14:52:57.757335+0000 (oldest deadline 2024-04-02T14:53:23.057240+0000) 2024-04-02T14:53:25.000 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 02 14:53:24 smithi177 podman[74153]: 2024-04-02 14:53:24.599717079 +0000 UTC m=+0.017918156 image pull f7301e070545fc23d8e5b7b052a2c15e832a0e57102f1acad8e4f3583709d23f quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534 2024-04-02T14:53:25.001 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 02 14:53:24 smithi177 podman[74153]: 2024-04-02 14:53:24.715258257 +0000 UTC m=+0.133459312 container create 49128bd59b6cf58117eca845b77fea80511bbb2b76567a11986387862552e19b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-8-deactivate, org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240402.004512-191d963, org.label-schema.license=GPLv2, org.label-schema.build-date=20240326, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-02T14:53:25.001 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 02 14:53:24 smithi177 podman[74153]: 2024-04-02 14:53:24.926285911 +0000 UTC m=+0.344486981 container init 49128bd59b6cf58117eca845b77fea80511bbb2b76567a11986387862552e19b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-8-deactivate, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240402.004512-191d963, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240326, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-04-02T14:53:25.001 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 02 14:53:24 smithi177 podman[74153]: 2024-04-02 14:53:24.929414945 +0000 UTC m=+0.347616006 container start 49128bd59b6cf58117eca845b77fea80511bbb2b76567a11986387862552e19b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-8-deactivate, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240402.004512-191d963, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240326, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-02T14:53:25.001 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 02 14:53:25 smithi177 podman[74153]: 2024-04-02 14:53:25.00946838 +0000 UTC m=+0.427669449 container attach 49128bd59b6cf58117eca845b77fea80511bbb2b76567a11986387862552e19b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-8-deactivate, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240326, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240402.004512-191d963) 2024-04-02T14:53:25.101 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:24 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:24.781+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:50.925796+0000 front 2024-04-02T14:51:50.925668+0000 (oldest deadline 2024-04-02T14:52:40.128984+0000) 2024-04-02T14:53:25.101 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:24 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:24.781+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:20.129325+0000 front 2024-04-02T14:52:20.129293+0000 (oldest deadline 2024-04-02T14:52:45.429431+0000) 2024-04-02T14:53:25.101 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:24 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:24.781+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:27.631081+0000 front 2024-04-02T14:52:27.631094+0000 (oldest deadline 2024-04-02T14:52:53.531189+0000) 2024-04-02T14:53:25.101 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:24 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:24.781+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-02T14:52:39.933034+0000 front 2024-04-02T14:52:39.933064+0000 (oldest deadline 2024-04-02T14:53:04.632770+0000) 2024-04-02T14:53:25.101 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:24 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:24.781+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.92:6812 osd.4 since back 2024-04-02T14:52:48.133729+0000 front 2024-04-02T14:52:48.133748+0000 (oldest deadline 2024-04-02T14:53:14.033931+0000) 2024-04-02T14:53:25.101 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:24 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:24.781+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-04-02T14:52:54.034568+0000 front 2024-04-02T14:52:54.034525+0000 (oldest deadline 2024-04-02T14:53:23.434953+0000) 2024-04-02T14:53:25.427 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 02 14:53:25 smithi177 podman[74153]: 2024-04-02 14:53:25.246084634 +0000 UTC m=+0.664285704 container died 49128bd59b6cf58117eca845b77fea80511bbb2b76567a11986387862552e19b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-8-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240326, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240402.004512-191d963, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-04-02T14:53:25.427 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:25 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:25.078+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:50.397138+0000 front 2024-04-02T14:51:50.397096+0000 (oldest deadline 2024-04-02T14:52:37.200213+0000) 2024-04-02T14:53:25.427 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:25 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:25.078+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:22.501322+0000 front 2024-04-02T14:52:22.501346+0000 (oldest deadline 2024-04-02T14:52:45.401451+0000) 2024-04-02T14:53:25.427 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:25 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:25.078+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:28.302204+0000 front 2024-04-02T14:52:28.302332+0000 (oldest deadline 2024-04-02T14:52:57.102868+0000) 2024-04-02T14:53:25.428 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:25 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:25.078+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-02T14:52:38.804314+0000 front 2024-04-02T14:52:38.804365+0000 (oldest deadline 2024-04-02T14:53:04.104362+0000) 2024-04-02T14:53:25.428 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:25 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:25.078+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.92:6812 osd.4 since back 2024-04-02T14:52:47.605500+0000 front 2024-04-02T14:52:47.605485+0000 (oldest deadline 2024-04-02T14:53:10.505428+0000) 2024-04-02T14:53:25.428 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:25 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:25.078+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-04-02T14:52:56.306832+0000 front 2024-04-02T14:52:56.306765+0000 (oldest deadline 2024-04-02T14:53:20.907329+0000) 2024-04-02T14:53:25.679 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:53:25 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:53:25.305+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:49.133761+0000 front 2024-04-02T14:51:49.134172+0000 (oldest deadline 2024-04-02T14:52:35.841983+0000) 2024-04-02T14:53:25.679 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:53:25 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:53:25.305+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:21.645685+0000 front 2024-04-02T14:52:21.645805+0000 (oldest deadline 2024-04-02T14:52:46.945889+0000) 2024-04-02T14:53:25.679 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:53:25 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:53:25.305+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:31.047892+0000 front 2024-04-02T14:52:31.047845+0000 (oldest deadline 2024-04-02T14:52:55.148007+0000) 2024-04-02T14:53:25.679 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:53:25 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:53:25.305+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-02T14:52:39.151607+0000 front 2024-04-02T14:52:39.151703+0000 (oldest deadline 2024-04-02T14:53:04.451174+0000) 2024-04-02T14:53:25.680 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:53:25 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:53:25.305+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.92:6812 osd.4 since back 2024-04-02T14:52:48.355136+0000 front 2024-04-02T14:52:48.355090+0000 (oldest deadline 2024-04-02T14:53:13.055332+0000) 2024-04-02T14:53:25.680 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:53:25 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:53:25.305+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-04-02T14:52:57.757258+0000 front 2024-04-02T14:52:57.757335+0000 (oldest deadline 2024-04-02T14:53:23.057240+0000) 2024-04-02T14:53:25.797 DEBUG:teuthology.orchestra.run.smithi177:> sudo pkill -f 'journalctl -f -n 0 -u ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4@osd.8.service' 2024-04-02T14:53:25.878 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-02T14:53:25.878 INFO:tasks.cephadm.osd.8:Stopped osd.8 2024-04-02T14:53:25.879 INFO:tasks.cephadm.osd.9:Stopping osd.9... 2024-04-02T14:53:25.879 DEBUG:teuthology.orchestra.run.smithi090:> sudo systemctl stop ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4@osd.9 2024-04-02T14:53:26.101 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:25 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:25.770+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:50.925796+0000 front 2024-04-02T14:51:50.925668+0000 (oldest deadline 2024-04-02T14:52:40.128984+0000) 2024-04-02T14:53:26.101 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:25 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:25.770+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:20.129325+0000 front 2024-04-02T14:52:20.129293+0000 (oldest deadline 2024-04-02T14:52:45.429431+0000) 2024-04-02T14:53:26.101 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:25 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:25.770+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:27.631081+0000 front 2024-04-02T14:52:27.631094+0000 (oldest deadline 2024-04-02T14:52:53.531189+0000) 2024-04-02T14:53:26.101 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:25 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:25.770+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-02T14:52:39.933034+0000 front 2024-04-02T14:52:39.933064+0000 (oldest deadline 2024-04-02T14:53:04.632770+0000) 2024-04-02T14:53:26.101 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:25 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:25.770+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.92:6812 osd.4 since back 2024-04-02T14:52:48.133729+0000 front 2024-04-02T14:52:48.133748+0000 (oldest deadline 2024-04-02T14:53:14.033931+0000) 2024-04-02T14:53:26.101 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:25 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:25.770+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-04-02T14:52:54.034568+0000 front 2024-04-02T14:52:54.034525+0000 (oldest deadline 2024-04-02T14:53:23.434953+0000) 2024-04-02T14:53:26.282 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:53:26 smithi090 systemd[1]: Stopping Ceph osd.9 for 6449448a-f0fc-11ee-b647-cb9ed24678a4... 2024-04-02T14:53:26.427 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:26 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:26.081+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:50.397138+0000 front 2024-04-02T14:51:50.397096+0000 (oldest deadline 2024-04-02T14:52:37.200213+0000) 2024-04-02T14:53:26.427 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:26 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:26.081+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:22.501322+0000 front 2024-04-02T14:52:22.501346+0000 (oldest deadline 2024-04-02T14:52:45.401451+0000) 2024-04-02T14:53:26.427 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:26 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:26.081+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:28.302204+0000 front 2024-04-02T14:52:28.302332+0000 (oldest deadline 2024-04-02T14:52:57.102868+0000) 2024-04-02T14:53:26.427 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:26 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:26.081+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-02T14:52:38.804314+0000 front 2024-04-02T14:52:38.804365+0000 (oldest deadline 2024-04-02T14:53:04.104362+0000) 2024-04-02T14:53:26.427 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:26 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:26.081+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.92:6812 osd.4 since back 2024-04-02T14:52:47.605500+0000 front 2024-04-02T14:52:47.605485+0000 (oldest deadline 2024-04-02T14:53:10.505428+0000) 2024-04-02T14:53:26.427 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:26 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:26.081+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-04-02T14:52:56.306832+0000 front 2024-04-02T14:52:56.306765+0000 (oldest deadline 2024-04-02T14:53:20.907329+0000) 2024-04-02T14:53:26.680 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:53:26 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:53:26.290+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:49.133761+0000 front 2024-04-02T14:51:49.134172+0000 (oldest deadline 2024-04-02T14:52:35.841983+0000) 2024-04-02T14:53:26.680 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:53:26 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:53:26.290+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:21.645685+0000 front 2024-04-02T14:52:21.645805+0000 (oldest deadline 2024-04-02T14:52:46.945889+0000) 2024-04-02T14:53:26.680 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:53:26 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:53:26.290+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:31.047892+0000 front 2024-04-02T14:52:31.047845+0000 (oldest deadline 2024-04-02T14:52:55.148007+0000) 2024-04-02T14:53:26.680 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:53:26 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:53:26.290+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-02T14:52:39.151607+0000 front 2024-04-02T14:52:39.151703+0000 (oldest deadline 2024-04-02T14:53:04.451174+0000) 2024-04-02T14:53:26.680 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:53:26 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:53:26.290+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.92:6812 osd.4 since back 2024-04-02T14:52:48.355136+0000 front 2024-04-02T14:52:48.355090+0000 (oldest deadline 2024-04-02T14:53:13.055332+0000) 2024-04-02T14:53:26.680 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:53:26 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:53:26.290+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-04-02T14:52:57.757258+0000 front 2024-04-02T14:52:57.757335+0000 (oldest deadline 2024-04-02T14:53:23.057240+0000) 2024-04-02T14:53:26.680 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:53:26 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:53:26.500+0000 7fc1aa200700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.9 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-02T14:53:26.680 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:53:26 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:53:26.500+0000 7fc1aa200700 -1 osd.9 114 *** Got signal Terminated *** 2024-04-02T14:53:26.680 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:53:26 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:53:26.500+0000 7fc1aa200700 -1 osd.9 114 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-02T14:53:27.101 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:26 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:26.787+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:50.925796+0000 front 2024-04-02T14:51:50.925668+0000 (oldest deadline 2024-04-02T14:52:40.128984+0000) 2024-04-02T14:53:27.101 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:26 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:26.787+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:20.129325+0000 front 2024-04-02T14:52:20.129293+0000 (oldest deadline 2024-04-02T14:52:45.429431+0000) 2024-04-02T14:53:27.101 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:26 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:26.787+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:27.631081+0000 front 2024-04-02T14:52:27.631094+0000 (oldest deadline 2024-04-02T14:52:53.531189+0000) 2024-04-02T14:53:27.101 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:26 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:26.787+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-02T14:52:39.933034+0000 front 2024-04-02T14:52:39.933064+0000 (oldest deadline 2024-04-02T14:53:04.632770+0000) 2024-04-02T14:53:27.101 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:26 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:26.787+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.92:6812 osd.4 since back 2024-04-02T14:52:48.133729+0000 front 2024-04-02T14:52:48.133748+0000 (oldest deadline 2024-04-02T14:53:14.033931+0000) 2024-04-02T14:53:27.101 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:26 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:26.787+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-04-02T14:52:54.034568+0000 front 2024-04-02T14:52:54.034525+0000 (oldest deadline 2024-04-02T14:53:23.434953+0000) 2024-04-02T14:53:27.427 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:27 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:27.069+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:50.397138+0000 front 2024-04-02T14:51:50.397096+0000 (oldest deadline 2024-04-02T14:52:37.200213+0000) 2024-04-02T14:53:27.427 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:27 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:27.069+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:22.501322+0000 front 2024-04-02T14:52:22.501346+0000 (oldest deadline 2024-04-02T14:52:45.401451+0000) 2024-04-02T14:53:27.427 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:27 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:27.069+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:28.302204+0000 front 2024-04-02T14:52:28.302332+0000 (oldest deadline 2024-04-02T14:52:57.102868+0000) 2024-04-02T14:53:27.427 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:27 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:27.069+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-02T14:52:38.804314+0000 front 2024-04-02T14:52:38.804365+0000 (oldest deadline 2024-04-02T14:53:04.104362+0000) 2024-04-02T14:53:27.427 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:27 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:27.069+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.92:6812 osd.4 since back 2024-04-02T14:52:47.605500+0000 front 2024-04-02T14:52:47.605485+0000 (oldest deadline 2024-04-02T14:53:10.505428+0000) 2024-04-02T14:53:27.427 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:27 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:27.069+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-04-02T14:52:56.306832+0000 front 2024-04-02T14:52:56.306765+0000 (oldest deadline 2024-04-02T14:53:20.907329+0000) 2024-04-02T14:53:27.679 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:53:27 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:53:27.316+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:49.133761+0000 front 2024-04-02T14:51:49.134172+0000 (oldest deadline 2024-04-02T14:52:35.841983+0000) 2024-04-02T14:53:27.679 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:53:27 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:53:27.316+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:21.645685+0000 front 2024-04-02T14:52:21.645805+0000 (oldest deadline 2024-04-02T14:52:46.945889+0000) 2024-04-02T14:53:27.679 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:53:27 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:53:27.316+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:31.047892+0000 front 2024-04-02T14:52:31.047845+0000 (oldest deadline 2024-04-02T14:52:55.148007+0000) 2024-04-02T14:53:27.680 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:53:27 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:53:27.316+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-02T14:52:39.151607+0000 front 2024-04-02T14:52:39.151703+0000 (oldest deadline 2024-04-02T14:53:04.451174+0000) 2024-04-02T14:53:27.680 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:53:27 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:53:27.316+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.92:6812 osd.4 since back 2024-04-02T14:52:48.355136+0000 front 2024-04-02T14:52:48.355090+0000 (oldest deadline 2024-04-02T14:53:13.055332+0000) 2024-04-02T14:53:27.680 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:53:27 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:53:27.316+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-04-02T14:52:57.757258+0000 front 2024-04-02T14:52:57.757335+0000 (oldest deadline 2024-04-02T14:53:23.057240+0000) 2024-04-02T14:53:28.101 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:27 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:27.820+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:50.925796+0000 front 2024-04-02T14:51:50.925668+0000 (oldest deadline 2024-04-02T14:52:40.128984+0000) 2024-04-02T14:53:28.101 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:27 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:27.820+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:20.129325+0000 front 2024-04-02T14:52:20.129293+0000 (oldest deadline 2024-04-02T14:52:45.429431+0000) 2024-04-02T14:53:28.101 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:27 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:27.820+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:27.631081+0000 front 2024-04-02T14:52:27.631094+0000 (oldest deadline 2024-04-02T14:52:53.531189+0000) 2024-04-02T14:53:28.101 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:27 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:27.820+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-02T14:52:39.933034+0000 front 2024-04-02T14:52:39.933064+0000 (oldest deadline 2024-04-02T14:53:04.632770+0000) 2024-04-02T14:53:28.101 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:27 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:27.820+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.92:6812 osd.4 since back 2024-04-02T14:52:48.133729+0000 front 2024-04-02T14:52:48.133748+0000 (oldest deadline 2024-04-02T14:53:14.033931+0000) 2024-04-02T14:53:28.101 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:27 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:27.820+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-04-02T14:52:54.034568+0000 front 2024-04-02T14:52:54.034525+0000 (oldest deadline 2024-04-02T14:53:23.434953+0000) 2024-04-02T14:53:28.101 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:27 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:27.820+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.90:6822 osd.6 since back 2024-04-02T14:53:05.736111+0000 front 2024-04-02T14:53:05.736207+0000 (oldest deadline 2024-04-02T14:53:27.435911+0000) 2024-04-02T14:53:28.427 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:28 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:28.087+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:50.397138+0000 front 2024-04-02T14:51:50.397096+0000 (oldest deadline 2024-04-02T14:52:37.200213+0000) 2024-04-02T14:53:28.427 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:28 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:28.087+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:22.501322+0000 front 2024-04-02T14:52:22.501346+0000 (oldest deadline 2024-04-02T14:52:45.401451+0000) 2024-04-02T14:53:28.427 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:28 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:28.087+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:28.302204+0000 front 2024-04-02T14:52:28.302332+0000 (oldest deadline 2024-04-02T14:52:57.102868+0000) 2024-04-02T14:53:28.427 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:28 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:28.087+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-02T14:52:38.804314+0000 front 2024-04-02T14:52:38.804365+0000 (oldest deadline 2024-04-02T14:53:04.104362+0000) 2024-04-02T14:53:28.427 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:28 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:28.087+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.92:6812 osd.4 since back 2024-04-02T14:52:47.605500+0000 front 2024-04-02T14:52:47.605485+0000 (oldest deadline 2024-04-02T14:53:10.505428+0000) 2024-04-02T14:53:28.427 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:28 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:28.087+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-04-02T14:52:56.306832+0000 front 2024-04-02T14:52:56.306765+0000 (oldest deadline 2024-04-02T14:53:20.907329+0000) 2024-04-02T14:53:28.679 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:53:28 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:53:28.318+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:49.133761+0000 front 2024-04-02T14:51:49.134172+0000 (oldest deadline 2024-04-02T14:52:35.841983+0000) 2024-04-02T14:53:28.679 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:53:28 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:53:28.318+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:21.645685+0000 front 2024-04-02T14:52:21.645805+0000 (oldest deadline 2024-04-02T14:52:46.945889+0000) 2024-04-02T14:53:28.679 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:53:28 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:53:28.318+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:31.047892+0000 front 2024-04-02T14:52:31.047845+0000 (oldest deadline 2024-04-02T14:52:55.148007+0000) 2024-04-02T14:53:28.680 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:53:28 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:53:28.318+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-02T14:52:39.151607+0000 front 2024-04-02T14:52:39.151703+0000 (oldest deadline 2024-04-02T14:53:04.451174+0000) 2024-04-02T14:53:28.680 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:53:28 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:53:28.318+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.92:6812 osd.4 since back 2024-04-02T14:52:48.355136+0000 front 2024-04-02T14:52:48.355090+0000 (oldest deadline 2024-04-02T14:53:13.055332+0000) 2024-04-02T14:53:28.680 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:53:28 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:53:28.318+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-04-02T14:52:57.757258+0000 front 2024-04-02T14:52:57.757335+0000 (oldest deadline 2024-04-02T14:53:23.057240+0000) 2024-04-02T14:53:28.680 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:53:28 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:53:28.318+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.90:6822 osd.6 since back 2024-04-02T14:53:04.660419+0000 front 2024-04-02T14:53:04.660560+0000 (oldest deadline 2024-04-02T14:53:28.160367+0000) 2024-04-02T14:53:29.101 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:28 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:28.800+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:50.925796+0000 front 2024-04-02T14:51:50.925668+0000 (oldest deadline 2024-04-02T14:52:40.128984+0000) 2024-04-02T14:53:29.101 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:28 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:28.800+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:20.129325+0000 front 2024-04-02T14:52:20.129293+0000 (oldest deadline 2024-04-02T14:52:45.429431+0000) 2024-04-02T14:53:29.101 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:28 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:28.800+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:27.631081+0000 front 2024-04-02T14:52:27.631094+0000 (oldest deadline 2024-04-02T14:52:53.531189+0000) 2024-04-02T14:53:29.101 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:28 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:28.800+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-02T14:52:39.933034+0000 front 2024-04-02T14:52:39.933064+0000 (oldest deadline 2024-04-02T14:53:04.632770+0000) 2024-04-02T14:53:29.101 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:28 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:28.800+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.92:6812 osd.4 since back 2024-04-02T14:52:48.133729+0000 front 2024-04-02T14:52:48.133748+0000 (oldest deadline 2024-04-02T14:53:14.033931+0000) 2024-04-02T14:53:29.101 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:28 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:28.800+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-04-02T14:52:54.034568+0000 front 2024-04-02T14:52:54.034525+0000 (oldest deadline 2024-04-02T14:53:23.434953+0000) 2024-04-02T14:53:29.101 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:28 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:28.800+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.90:6822 osd.6 since back 2024-04-02T14:53:05.736111+0000 front 2024-04-02T14:53:05.736207+0000 (oldest deadline 2024-04-02T14:53:27.435911+0000) 2024-04-02T14:53:29.427 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:29 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:29.038+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:50.397138+0000 front 2024-04-02T14:51:50.397096+0000 (oldest deadline 2024-04-02T14:52:37.200213+0000) 2024-04-02T14:53:29.427 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:29 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:29.038+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:22.501322+0000 front 2024-04-02T14:52:22.501346+0000 (oldest deadline 2024-04-02T14:52:45.401451+0000) 2024-04-02T14:53:29.427 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:29 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:29.038+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:28.302204+0000 front 2024-04-02T14:52:28.302332+0000 (oldest deadline 2024-04-02T14:52:57.102868+0000) 2024-04-02T14:53:29.427 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:29 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:29.038+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-02T14:52:38.804314+0000 front 2024-04-02T14:52:38.804365+0000 (oldest deadline 2024-04-02T14:53:04.104362+0000) 2024-04-02T14:53:29.427 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:29 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:29.038+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.92:6812 osd.4 since back 2024-04-02T14:52:47.605500+0000 front 2024-04-02T14:52:47.605485+0000 (oldest deadline 2024-04-02T14:53:10.505428+0000) 2024-04-02T14:53:29.427 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:29 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:29.038+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-04-02T14:52:56.306832+0000 front 2024-04-02T14:52:56.306765+0000 (oldest deadline 2024-04-02T14:53:20.907329+0000) 2024-04-02T14:53:29.679 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:53:29 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:53:29.352+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:49.133761+0000 front 2024-04-02T14:51:49.134172+0000 (oldest deadline 2024-04-02T14:52:35.841983+0000) 2024-04-02T14:53:29.679 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:53:29 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:53:29.352+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:21.645685+0000 front 2024-04-02T14:52:21.645805+0000 (oldest deadline 2024-04-02T14:52:46.945889+0000) 2024-04-02T14:53:29.680 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:53:29 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:53:29.352+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:31.047892+0000 front 2024-04-02T14:52:31.047845+0000 (oldest deadline 2024-04-02T14:52:55.148007+0000) 2024-04-02T14:53:29.680 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:53:29 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:53:29.352+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-02T14:52:39.151607+0000 front 2024-04-02T14:52:39.151703+0000 (oldest deadline 2024-04-02T14:53:04.451174+0000) 2024-04-02T14:53:29.680 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:53:29 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:53:29.352+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.92:6812 osd.4 since back 2024-04-02T14:52:48.355136+0000 front 2024-04-02T14:52:48.355090+0000 (oldest deadline 2024-04-02T14:53:13.055332+0000) 2024-04-02T14:53:29.680 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:53:29 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:53:29.352+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-04-02T14:52:57.757258+0000 front 2024-04-02T14:52:57.757335+0000 (oldest deadline 2024-04-02T14:53:23.057240+0000) 2024-04-02T14:53:29.680 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:53:29 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:53:29.352+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.90:6822 osd.6 since back 2024-04-02T14:53:04.660419+0000 front 2024-04-02T14:53:04.660560+0000 (oldest deadline 2024-04-02T14:53:28.160367+0000) 2024-04-02T14:53:30.101 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:29 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:29.785+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:50.925796+0000 front 2024-04-02T14:51:50.925668+0000 (oldest deadline 2024-04-02T14:52:40.128984+0000) 2024-04-02T14:53:30.101 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:29 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:29.785+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:20.129325+0000 front 2024-04-02T14:52:20.129293+0000 (oldest deadline 2024-04-02T14:52:45.429431+0000) 2024-04-02T14:53:30.101 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:29 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:29.785+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:27.631081+0000 front 2024-04-02T14:52:27.631094+0000 (oldest deadline 2024-04-02T14:52:53.531189+0000) 2024-04-02T14:53:30.101 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:29 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:29.785+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-02T14:52:39.933034+0000 front 2024-04-02T14:52:39.933064+0000 (oldest deadline 2024-04-02T14:53:04.632770+0000) 2024-04-02T14:53:30.101 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:29 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:29.785+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.92:6812 osd.4 since back 2024-04-02T14:52:48.133729+0000 front 2024-04-02T14:52:48.133748+0000 (oldest deadline 2024-04-02T14:53:14.033931+0000) 2024-04-02T14:53:30.101 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:29 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:29.785+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-04-02T14:52:54.034568+0000 front 2024-04-02T14:52:54.034525+0000 (oldest deadline 2024-04-02T14:53:23.434953+0000) 2024-04-02T14:53:30.102 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:29 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:29.785+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.90:6822 osd.6 since back 2024-04-02T14:53:05.736111+0000 front 2024-04-02T14:53:05.736207+0000 (oldest deadline 2024-04-02T14:53:27.435911+0000) 2024-04-02T14:53:30.427 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:30 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:30.014+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:50.397138+0000 front 2024-04-02T14:51:50.397096+0000 (oldest deadline 2024-04-02T14:52:37.200213+0000) 2024-04-02T14:53:30.427 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:30 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:30.014+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:22.501322+0000 front 2024-04-02T14:52:22.501346+0000 (oldest deadline 2024-04-02T14:52:45.401451+0000) 2024-04-02T14:53:30.427 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:30 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:30.014+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:28.302204+0000 front 2024-04-02T14:52:28.302332+0000 (oldest deadline 2024-04-02T14:52:57.102868+0000) 2024-04-02T14:53:30.427 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:30 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:30.014+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-02T14:52:38.804314+0000 front 2024-04-02T14:52:38.804365+0000 (oldest deadline 2024-04-02T14:53:04.104362+0000) 2024-04-02T14:53:30.427 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:30 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:30.014+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.92:6812 osd.4 since back 2024-04-02T14:52:47.605500+0000 front 2024-04-02T14:52:47.605485+0000 (oldest deadline 2024-04-02T14:53:10.505428+0000) 2024-04-02T14:53:30.427 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:30 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:30.014+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-04-02T14:52:56.306832+0000 front 2024-04-02T14:52:56.306765+0000 (oldest deadline 2024-04-02T14:53:20.907329+0000) 2024-04-02T14:53:30.679 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:53:30 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:53:30.396+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:49.133761+0000 front 2024-04-02T14:51:49.134172+0000 (oldest deadline 2024-04-02T14:52:35.841983+0000) 2024-04-02T14:53:30.679 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:53:30 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:53:30.396+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:21.645685+0000 front 2024-04-02T14:52:21.645805+0000 (oldest deadline 2024-04-02T14:52:46.945889+0000) 2024-04-02T14:53:30.679 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:53:30 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:53:30.396+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:31.047892+0000 front 2024-04-02T14:52:31.047845+0000 (oldest deadline 2024-04-02T14:52:55.148007+0000) 2024-04-02T14:53:30.679 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:53:30 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:53:30.396+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-02T14:52:39.151607+0000 front 2024-04-02T14:52:39.151703+0000 (oldest deadline 2024-04-02T14:53:04.451174+0000) 2024-04-02T14:53:30.680 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:53:30 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:53:30.396+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.92:6812 osd.4 since back 2024-04-02T14:52:48.355136+0000 front 2024-04-02T14:52:48.355090+0000 (oldest deadline 2024-04-02T14:53:13.055332+0000) 2024-04-02T14:53:30.680 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:53:30 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:53:30.396+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-04-02T14:52:57.757258+0000 front 2024-04-02T14:52:57.757335+0000 (oldest deadline 2024-04-02T14:53:23.057240+0000) 2024-04-02T14:53:30.680 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:53:30 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:53:30.396+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.90:6822 osd.6 since back 2024-04-02T14:53:04.660419+0000 front 2024-04-02T14:53:04.660560+0000 (oldest deadline 2024-04-02T14:53:28.160367+0000) 2024-04-02T14:53:31.101 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:30 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:30.785+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:50.925796+0000 front 2024-04-02T14:51:50.925668+0000 (oldest deadline 2024-04-02T14:52:40.128984+0000) 2024-04-02T14:53:31.101 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:30 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:30.785+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:20.129325+0000 front 2024-04-02T14:52:20.129293+0000 (oldest deadline 2024-04-02T14:52:45.429431+0000) 2024-04-02T14:53:31.101 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:30 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:30.785+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:27.631081+0000 front 2024-04-02T14:52:27.631094+0000 (oldest deadline 2024-04-02T14:52:53.531189+0000) 2024-04-02T14:53:31.101 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:30 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:30.785+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-02T14:52:39.933034+0000 front 2024-04-02T14:52:39.933064+0000 (oldest deadline 2024-04-02T14:53:04.632770+0000) 2024-04-02T14:53:31.101 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:30 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:30.785+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.92:6812 osd.4 since back 2024-04-02T14:52:48.133729+0000 front 2024-04-02T14:52:48.133748+0000 (oldest deadline 2024-04-02T14:53:14.033931+0000) 2024-04-02T14:53:31.101 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:30 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:30.785+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-04-02T14:52:54.034568+0000 front 2024-04-02T14:52:54.034525+0000 (oldest deadline 2024-04-02T14:53:23.434953+0000) 2024-04-02T14:53:31.101 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:30 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:30.785+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.90:6822 osd.6 since back 2024-04-02T14:53:05.736111+0000 front 2024-04-02T14:53:05.736207+0000 (oldest deadline 2024-04-02T14:53:27.435911+0000) 2024-04-02T14:53:31.427 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:31 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:31.012+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:50.397138+0000 front 2024-04-02T14:51:50.397096+0000 (oldest deadline 2024-04-02T14:52:37.200213+0000) 2024-04-02T14:53:31.427 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:31 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:31.012+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:22.501322+0000 front 2024-04-02T14:52:22.501346+0000 (oldest deadline 2024-04-02T14:52:45.401451+0000) 2024-04-02T14:53:31.427 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:31 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:31.012+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:28.302204+0000 front 2024-04-02T14:52:28.302332+0000 (oldest deadline 2024-04-02T14:52:57.102868+0000) 2024-04-02T14:53:31.427 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:31 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:31.012+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-02T14:52:38.804314+0000 front 2024-04-02T14:52:38.804365+0000 (oldest deadline 2024-04-02T14:53:04.104362+0000) 2024-04-02T14:53:31.427 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:31 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:31.012+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.92:6812 osd.4 since back 2024-04-02T14:52:47.605500+0000 front 2024-04-02T14:52:47.605485+0000 (oldest deadline 2024-04-02T14:53:10.505428+0000) 2024-04-02T14:53:31.427 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:31 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:31.012+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-04-02T14:52:56.306832+0000 front 2024-04-02T14:52:56.306765+0000 (oldest deadline 2024-04-02T14:53:20.907329+0000) 2024-04-02T14:53:31.679 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:53:31 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:53:31.431+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:49.133761+0000 front 2024-04-02T14:51:49.134172+0000 (oldest deadline 2024-04-02T14:52:35.841983+0000) 2024-04-02T14:53:31.679 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:53:31 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:53:31.431+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:21.645685+0000 front 2024-04-02T14:52:21.645805+0000 (oldest deadline 2024-04-02T14:52:46.945889+0000) 2024-04-02T14:53:31.679 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:53:31 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:53:31.431+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:31.047892+0000 front 2024-04-02T14:52:31.047845+0000 (oldest deadline 2024-04-02T14:52:55.148007+0000) 2024-04-02T14:53:31.680 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:53:31 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:53:31.431+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-02T14:52:39.151607+0000 front 2024-04-02T14:52:39.151703+0000 (oldest deadline 2024-04-02T14:53:04.451174+0000) 2024-04-02T14:53:31.680 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:53:31 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:53:31.431+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.92:6812 osd.4 since back 2024-04-02T14:52:48.355136+0000 front 2024-04-02T14:52:48.355090+0000 (oldest deadline 2024-04-02T14:53:13.055332+0000) 2024-04-02T14:53:31.680 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:53:31 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:53:31.431+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-04-02T14:52:57.757258+0000 front 2024-04-02T14:52:57.757335+0000 (oldest deadline 2024-04-02T14:53:23.057240+0000) 2024-04-02T14:53:31.680 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:53:31 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9[66387]: 2024-04-02T14:53:31.432+0000 7fc1a0a00700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.90:6822 osd.6 since back 2024-04-02T14:53:04.660419+0000 front 2024-04-02T14:53:04.660560+0000 (oldest deadline 2024-04-02T14:53:28.160367+0000) 2024-04-02T14:53:32.101 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:31 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:31.767+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:50.925796+0000 front 2024-04-02T14:51:50.925668+0000 (oldest deadline 2024-04-02T14:52:40.128984+0000) 2024-04-02T14:53:32.101 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:31 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:31.767+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:20.129325+0000 front 2024-04-02T14:52:20.129293+0000 (oldest deadline 2024-04-02T14:52:45.429431+0000) 2024-04-02T14:53:32.101 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:31 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:31.767+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:27.631081+0000 front 2024-04-02T14:52:27.631094+0000 (oldest deadline 2024-04-02T14:52:53.531189+0000) 2024-04-02T14:53:32.101 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:31 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:31.767+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-02T14:52:39.933034+0000 front 2024-04-02T14:52:39.933064+0000 (oldest deadline 2024-04-02T14:53:04.632770+0000) 2024-04-02T14:53:32.101 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:31 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:31.767+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.92:6812 osd.4 since back 2024-04-02T14:52:48.133729+0000 front 2024-04-02T14:52:48.133748+0000 (oldest deadline 2024-04-02T14:53:14.033931+0000) 2024-04-02T14:53:32.102 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:31 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:31.767+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-04-02T14:52:54.034568+0000 front 2024-04-02T14:52:54.034525+0000 (oldest deadline 2024-04-02T14:53:23.434953+0000) 2024-04-02T14:53:32.102 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:31 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:31.767+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.90:6822 osd.6 since back 2024-04-02T14:53:05.736111+0000 front 2024-04-02T14:53:05.736207+0000 (oldest deadline 2024-04-02T14:53:27.435911+0000) 2024-04-02T14:53:32.427 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:32 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:32.062+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:50.397138+0000 front 2024-04-02T14:51:50.397096+0000 (oldest deadline 2024-04-02T14:52:37.200213+0000) 2024-04-02T14:53:32.427 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:32 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:32.062+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:22.501322+0000 front 2024-04-02T14:52:22.501346+0000 (oldest deadline 2024-04-02T14:52:45.401451+0000) 2024-04-02T14:53:32.427 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:32 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:32.062+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:28.302204+0000 front 2024-04-02T14:52:28.302332+0000 (oldest deadline 2024-04-02T14:52:57.102868+0000) 2024-04-02T14:53:32.427 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:32 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:32.062+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-02T14:52:38.804314+0000 front 2024-04-02T14:52:38.804365+0000 (oldest deadline 2024-04-02T14:53:04.104362+0000) 2024-04-02T14:53:32.427 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:32 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:32.062+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.92:6812 osd.4 since back 2024-04-02T14:52:47.605500+0000 front 2024-04-02T14:52:47.605485+0000 (oldest deadline 2024-04-02T14:53:10.505428+0000) 2024-04-02T14:53:32.427 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:32 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:32.062+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-04-02T14:52:56.306832+0000 front 2024-04-02T14:52:56.306765+0000 (oldest deadline 2024-04-02T14:53:20.907329+0000) 2024-04-02T14:53:32.679 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:53:32 smithi090 podman[111279]: 2024-04-02 14:53:32.403554128 +0000 UTC m=+6.103903932 container died 28fa8627dda549594b5fe987a14ddc3748c135214b00ae4b7dc32e52c75c7a14 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240402.004512-191d963, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240326, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-02T14:53:33.101 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:32 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:32.722+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:50.925796+0000 front 2024-04-02T14:51:50.925668+0000 (oldest deadline 2024-04-02T14:52:40.128984+0000) 2024-04-02T14:53:33.101 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:32 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:32.722+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:20.129325+0000 front 2024-04-02T14:52:20.129293+0000 (oldest deadline 2024-04-02T14:52:45.429431+0000) 2024-04-02T14:53:33.101 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:32 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:32.722+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:27.631081+0000 front 2024-04-02T14:52:27.631094+0000 (oldest deadline 2024-04-02T14:52:53.531189+0000) 2024-04-02T14:53:33.101 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:32 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:32.722+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-02T14:52:39.933034+0000 front 2024-04-02T14:52:39.933064+0000 (oldest deadline 2024-04-02T14:53:04.632770+0000) 2024-04-02T14:53:33.101 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:32 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:32.722+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.92:6812 osd.4 since back 2024-04-02T14:52:48.133729+0000 front 2024-04-02T14:52:48.133748+0000 (oldest deadline 2024-04-02T14:53:14.033931+0000) 2024-04-02T14:53:33.101 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:32 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:32.722+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-04-02T14:52:54.034568+0000 front 2024-04-02T14:52:54.034525+0000 (oldest deadline 2024-04-02T14:53:23.434953+0000) 2024-04-02T14:53:33.101 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:32 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:32.722+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.90:6822 osd.6 since back 2024-04-02T14:53:05.736111+0000 front 2024-04-02T14:53:05.736207+0000 (oldest deadline 2024-04-02T14:53:27.435911+0000) 2024-04-02T14:53:33.153 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:53:32 smithi090 podman[111279]: 2024-04-02 14:53:32.869087252 +0000 UTC m=+6.569437093 container cleanup 28fa8627dda549594b5fe987a14ddc3748c135214b00ae4b7dc32e52c75c7a14 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240326, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, RELEASE=wip-batrick-testing-20240402.004512-191d963) 2024-04-02T14:53:33.154 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:53:32 smithi090 bash[111279]: ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9 2024-04-02T14:53:33.427 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:33 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:33.033+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:50.397138+0000 front 2024-04-02T14:51:50.397096+0000 (oldest deadline 2024-04-02T14:52:37.200213+0000) 2024-04-02T14:53:33.427 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:33 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:33.033+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:22.501322+0000 front 2024-04-02T14:52:22.501346+0000 (oldest deadline 2024-04-02T14:52:45.401451+0000) 2024-04-02T14:53:33.427 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:33 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:33.033+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:28.302204+0000 front 2024-04-02T14:52:28.302332+0000 (oldest deadline 2024-04-02T14:52:57.102868+0000) 2024-04-02T14:53:33.427 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:33 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:33.033+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-02T14:52:38.804314+0000 front 2024-04-02T14:52:38.804365+0000 (oldest deadline 2024-04-02T14:53:04.104362+0000) 2024-04-02T14:53:33.427 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:33 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:33.033+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.92:6812 osd.4 since back 2024-04-02T14:52:47.605500+0000 front 2024-04-02T14:52:47.605485+0000 (oldest deadline 2024-04-02T14:53:10.505428+0000) 2024-04-02T14:53:33.427 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:33 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:33.033+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-04-02T14:52:56.306832+0000 front 2024-04-02T14:52:56.306765+0000 (oldest deadline 2024-04-02T14:53:20.907329+0000) 2024-04-02T14:53:33.428 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:53:33 smithi090 podman[111299]: 2024-04-02 14:53:33.162107037 +0000 UTC m=+0.750051047 container remove 28fa8627dda549594b5fe987a14ddc3748c135214b00ae4b7dc32e52c75c7a14 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, org.label-schema.build-date=20240326, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240402.004512-191d963, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-02T14:53:34.026 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:53:33 smithi090 podman[111388]: 2024-04-02 14:53:33.648195033 +0000 UTC m=+0.031938789 image pull f7301e070545fc23d8e5b7b052a2c15e832a0e57102f1acad8e4f3583709d23f quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534 2024-04-02T14:53:34.026 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:53:33 smithi090 podman[111388]: 2024-04-02 14:53:33.746320299 +0000 UTC m=+0.130064097 container create 633f21bc1fd69c17e221e4ac4297754dfea29491382ac1d0ac5d66c5e4cc938d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9-deactivate, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240326, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=wip-batrick-testing-20240402.004512-191d963, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-04-02T14:53:34.026 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:53:34 smithi090 podman[111388]: 2024-04-02 14:53:34.034417362 +0000 UTC m=+0.418161220 container init 633f21bc1fd69c17e221e4ac4297754dfea29491382ac1d0ac5d66c5e4cc938d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9-deactivate, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240326, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, RELEASE=wip-batrick-testing-20240402.004512-191d963, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-04-02T14:53:34.101 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:33 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:33.717+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:50.925796+0000 front 2024-04-02T14:51:50.925668+0000 (oldest deadline 2024-04-02T14:52:40.128984+0000) 2024-04-02T14:53:34.101 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:33 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:33.717+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:20.129325+0000 front 2024-04-02T14:52:20.129293+0000 (oldest deadline 2024-04-02T14:52:45.429431+0000) 2024-04-02T14:53:34.101 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:33 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:33.717+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:27.631081+0000 front 2024-04-02T14:52:27.631094+0000 (oldest deadline 2024-04-02T14:52:53.531189+0000) 2024-04-02T14:53:34.101 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:33 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:33.717+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-02T14:52:39.933034+0000 front 2024-04-02T14:52:39.933064+0000 (oldest deadline 2024-04-02T14:53:04.632770+0000) 2024-04-02T14:53:34.101 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:33 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:33.717+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.92:6812 osd.4 since back 2024-04-02T14:52:48.133729+0000 front 2024-04-02T14:52:48.133748+0000 (oldest deadline 2024-04-02T14:53:14.033931+0000) 2024-04-02T14:53:34.101 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:33 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:33.717+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-04-02T14:52:54.034568+0000 front 2024-04-02T14:52:54.034525+0000 (oldest deadline 2024-04-02T14:53:23.434953+0000) 2024-04-02T14:53:34.101 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:33 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:33.717+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.90:6822 osd.6 since back 2024-04-02T14:53:05.736111+0000 front 2024-04-02T14:53:05.736207+0000 (oldest deadline 2024-04-02T14:53:27.435911+0000) 2024-04-02T14:53:34.428 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:33 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:33.985+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:50.397138+0000 front 2024-04-02T14:51:50.397096+0000 (oldest deadline 2024-04-02T14:52:37.200213+0000) 2024-04-02T14:53:34.428 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:33 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:33.985+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:22.501322+0000 front 2024-04-02T14:52:22.501346+0000 (oldest deadline 2024-04-02T14:52:45.401451+0000) 2024-04-02T14:53:34.428 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:33 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:33.985+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:28.302204+0000 front 2024-04-02T14:52:28.302332+0000 (oldest deadline 2024-04-02T14:52:57.102868+0000) 2024-04-02T14:53:34.428 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:33 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:33.985+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-02T14:52:38.804314+0000 front 2024-04-02T14:52:38.804365+0000 (oldest deadline 2024-04-02T14:53:04.104362+0000) 2024-04-02T14:53:34.428 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:33 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:33.985+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.92:6812 osd.4 since back 2024-04-02T14:52:47.605500+0000 front 2024-04-02T14:52:47.605485+0000 (oldest deadline 2024-04-02T14:53:10.505428+0000) 2024-04-02T14:53:34.428 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:33 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:33.985+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-04-02T14:52:56.306832+0000 front 2024-04-02T14:52:56.306765+0000 (oldest deadline 2024-04-02T14:53:20.907329+0000) 2024-04-02T14:53:34.429 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:53:34 smithi090 podman[111388]: 2024-04-02 14:53:34.045986071 +0000 UTC m=+0.429729834 container start 633f21bc1fd69c17e221e4ac4297754dfea29491382ac1d0ac5d66c5e4cc938d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240402.004512-191d963, org.label-schema.build-date=20240326, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux ) 2024-04-02T14:53:34.429 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:53:34 smithi090 podman[111388]: 2024-04-02 14:53:34.116838615 +0000 UTC m=+0.500582453 container attach 633f21bc1fd69c17e221e4ac4297754dfea29491382ac1d0ac5d66c5e4cc938d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9-deactivate, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240326, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240402.004512-191d963, org.label-schema.vendor=CentOS) 2024-04-02T14:53:34.815 INFO:journalctl@ceph.osd.9.smithi090.stdout:Apr 02 14:53:34 smithi090 podman[111388]: 2024-04-02 14:53:34.479083756 +0000 UTC m=+0.862827519 container died 633f21bc1fd69c17e221e4ac4297754dfea29491382ac1d0ac5d66c5e4cc938d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-9-deactivate, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240326, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240402.004512-191d963, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-04-02T14:53:35.101 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:34 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:34.712+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:50.925796+0000 front 2024-04-02T14:51:50.925668+0000 (oldest deadline 2024-04-02T14:52:40.128984+0000) 2024-04-02T14:53:35.101 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:34 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:34.712+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:20.129325+0000 front 2024-04-02T14:52:20.129293+0000 (oldest deadline 2024-04-02T14:52:45.429431+0000) 2024-04-02T14:53:35.101 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:34 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:34.712+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:27.631081+0000 front 2024-04-02T14:52:27.631094+0000 (oldest deadline 2024-04-02T14:52:53.531189+0000) 2024-04-02T14:53:35.101 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:34 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:34.712+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-02T14:52:39.933034+0000 front 2024-04-02T14:52:39.933064+0000 (oldest deadline 2024-04-02T14:53:04.632770+0000) 2024-04-02T14:53:35.101 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:34 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:34.712+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.92:6812 osd.4 since back 2024-04-02T14:52:48.133729+0000 front 2024-04-02T14:52:48.133748+0000 (oldest deadline 2024-04-02T14:53:14.033931+0000) 2024-04-02T14:53:35.101 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:34 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:34.712+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-04-02T14:52:54.034568+0000 front 2024-04-02T14:52:54.034525+0000 (oldest deadline 2024-04-02T14:53:23.434953+0000) 2024-04-02T14:53:35.101 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:34 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:34.712+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.90:6822 osd.6 since back 2024-04-02T14:53:05.736111+0000 front 2024-04-02T14:53:05.736207+0000 (oldest deadline 2024-04-02T14:53:27.435911+0000) 2024-04-02T14:53:35.191 DEBUG:teuthology.orchestra.run.smithi090:> sudo pkill -f 'journalctl -f -n 0 -u ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4@osd.9.service' 2024-04-02T14:53:35.306 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:34 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:34.962+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:50.397138+0000 front 2024-04-02T14:51:50.397096+0000 (oldest deadline 2024-04-02T14:52:37.200213+0000) 2024-04-02T14:53:35.307 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:34 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:34.962+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:22.501322+0000 front 2024-04-02T14:52:22.501346+0000 (oldest deadline 2024-04-02T14:52:45.401451+0000) 2024-04-02T14:53:35.307 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:34 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:34.962+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:28.302204+0000 front 2024-04-02T14:52:28.302332+0000 (oldest deadline 2024-04-02T14:52:57.102868+0000) 2024-04-02T14:53:35.307 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:34 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:34.962+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-02T14:52:38.804314+0000 front 2024-04-02T14:52:38.804365+0000 (oldest deadline 2024-04-02T14:53:04.104362+0000) 2024-04-02T14:53:35.307 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:34 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:34.962+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.92:6812 osd.4 since back 2024-04-02T14:52:47.605500+0000 front 2024-04-02T14:52:47.605485+0000 (oldest deadline 2024-04-02T14:53:10.505428+0000) 2024-04-02T14:53:35.307 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:34 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:34.962+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-04-02T14:52:56.306832+0000 front 2024-04-02T14:52:56.306765+0000 (oldest deadline 2024-04-02T14:53:20.907329+0000) 2024-04-02T14:53:35.311 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-02T14:53:35.311 INFO:tasks.cephadm.osd.9:Stopped osd.9 2024-04-02T14:53:35.311 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2024-04-02T14:53:35.311 DEBUG:teuthology.orchestra.run.smithi092:> sudo systemctl stop ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4@osd.10 2024-04-02T14:53:35.601 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:35 smithi092 systemd[1]: Stopping Ceph osd.10 for 6449448a-f0fc-11ee-b647-cb9ed24678a4... 2024-04-02T14:53:36.101 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:35 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:35.636+0000 7f6ae1d5d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-02T14:53:36.101 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:35 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:35.636+0000 7f6ae1d5d700 -1 osd.10 114 *** Got signal Terminated *** 2024-04-02T14:53:36.101 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:35 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:35.636+0000 7f6ae1d5d700 -1 osd.10 114 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-02T14:53:36.101 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:35 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:35.756+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:50.925796+0000 front 2024-04-02T14:51:50.925668+0000 (oldest deadline 2024-04-02T14:52:40.128984+0000) 2024-04-02T14:53:36.101 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:35 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:35.756+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:20.129325+0000 front 2024-04-02T14:52:20.129293+0000 (oldest deadline 2024-04-02T14:52:45.429431+0000) 2024-04-02T14:53:36.101 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:35 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:35.756+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:27.631081+0000 front 2024-04-02T14:52:27.631094+0000 (oldest deadline 2024-04-02T14:52:53.531189+0000) 2024-04-02T14:53:36.101 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:35 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:35.756+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-02T14:52:39.933034+0000 front 2024-04-02T14:52:39.933064+0000 (oldest deadline 2024-04-02T14:53:04.632770+0000) 2024-04-02T14:53:36.101 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:35 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:35.756+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.92:6812 osd.4 since back 2024-04-02T14:52:48.133729+0000 front 2024-04-02T14:52:48.133748+0000 (oldest deadline 2024-04-02T14:53:14.033931+0000) 2024-04-02T14:53:36.102 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:35 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:35.756+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-04-02T14:52:54.034568+0000 front 2024-04-02T14:52:54.034525+0000 (oldest deadline 2024-04-02T14:53:23.434953+0000) 2024-04-02T14:53:36.102 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:35 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:35.756+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.90:6822 osd.6 since back 2024-04-02T14:53:05.736111+0000 front 2024-04-02T14:53:05.736207+0000 (oldest deadline 2024-04-02T14:53:27.435911+0000) 2024-04-02T14:53:36.177 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:35 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:35.923+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:50.397138+0000 front 2024-04-02T14:51:50.397096+0000 (oldest deadline 2024-04-02T14:52:37.200213+0000) 2024-04-02T14:53:36.177 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:35 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:35.923+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:22.501322+0000 front 2024-04-02T14:52:22.501346+0000 (oldest deadline 2024-04-02T14:52:45.401451+0000) 2024-04-02T14:53:36.177 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:35 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:35.923+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:28.302204+0000 front 2024-04-02T14:52:28.302332+0000 (oldest deadline 2024-04-02T14:52:57.102868+0000) 2024-04-02T14:53:36.177 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:35 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:35.923+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-02T14:52:38.804314+0000 front 2024-04-02T14:52:38.804365+0000 (oldest deadline 2024-04-02T14:53:04.104362+0000) 2024-04-02T14:53:36.177 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:35 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:35.923+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.92:6812 osd.4 since back 2024-04-02T14:52:47.605500+0000 front 2024-04-02T14:52:47.605485+0000 (oldest deadline 2024-04-02T14:53:10.505428+0000) 2024-04-02T14:53:36.177 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:35 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:35.923+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-04-02T14:52:56.306832+0000 front 2024-04-02T14:52:56.306765+0000 (oldest deadline 2024-04-02T14:53:20.907329+0000) 2024-04-02T14:53:37.101 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:36 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:36.769+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:50.925796+0000 front 2024-04-02T14:51:50.925668+0000 (oldest deadline 2024-04-02T14:52:40.128984+0000) 2024-04-02T14:53:37.101 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:36 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:36.769+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:20.129325+0000 front 2024-04-02T14:52:20.129293+0000 (oldest deadline 2024-04-02T14:52:45.429431+0000) 2024-04-02T14:53:37.101 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:36 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:36.769+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:27.631081+0000 front 2024-04-02T14:52:27.631094+0000 (oldest deadline 2024-04-02T14:52:53.531189+0000) 2024-04-02T14:53:37.101 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:36 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:36.769+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-02T14:52:39.933034+0000 front 2024-04-02T14:52:39.933064+0000 (oldest deadline 2024-04-02T14:53:04.632770+0000) 2024-04-02T14:53:37.101 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:36 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:36.769+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.92:6812 osd.4 since back 2024-04-02T14:52:48.133729+0000 front 2024-04-02T14:52:48.133748+0000 (oldest deadline 2024-04-02T14:53:14.033931+0000) 2024-04-02T14:53:37.101 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:36 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:36.769+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-04-02T14:52:54.034568+0000 front 2024-04-02T14:52:54.034525+0000 (oldest deadline 2024-04-02T14:53:23.434953+0000) 2024-04-02T14:53:37.101 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:36 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:36.769+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.90:6822 osd.6 since back 2024-04-02T14:53:05.736111+0000 front 2024-04-02T14:53:05.736207+0000 (oldest deadline 2024-04-02T14:53:27.435911+0000) 2024-04-02T14:53:37.177 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:36 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:36.886+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:50.397138+0000 front 2024-04-02T14:51:50.397096+0000 (oldest deadline 2024-04-02T14:52:37.200213+0000) 2024-04-02T14:53:37.177 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:36 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:36.886+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:22.501322+0000 front 2024-04-02T14:52:22.501346+0000 (oldest deadline 2024-04-02T14:52:45.401451+0000) 2024-04-02T14:53:37.177 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:36 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:36.886+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:28.302204+0000 front 2024-04-02T14:52:28.302332+0000 (oldest deadline 2024-04-02T14:52:57.102868+0000) 2024-04-02T14:53:37.177 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:36 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:36.886+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-02T14:52:38.804314+0000 front 2024-04-02T14:52:38.804365+0000 (oldest deadline 2024-04-02T14:53:04.104362+0000) 2024-04-02T14:53:37.177 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:36 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:36.886+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.92:6812 osd.4 since back 2024-04-02T14:52:47.605500+0000 front 2024-04-02T14:52:47.605485+0000 (oldest deadline 2024-04-02T14:53:10.505428+0000) 2024-04-02T14:53:37.177 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:36 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:36.886+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-04-02T14:52:56.306832+0000 front 2024-04-02T14:52:56.306765+0000 (oldest deadline 2024-04-02T14:53:20.907329+0000) 2024-04-02T14:53:38.101 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:37 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:37.787+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:50.925796+0000 front 2024-04-02T14:51:50.925668+0000 (oldest deadline 2024-04-02T14:52:40.128984+0000) 2024-04-02T14:53:38.101 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:37 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:37.787+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:20.129325+0000 front 2024-04-02T14:52:20.129293+0000 (oldest deadline 2024-04-02T14:52:45.429431+0000) 2024-04-02T14:53:38.101 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:37 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:37.787+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:27.631081+0000 front 2024-04-02T14:52:27.631094+0000 (oldest deadline 2024-04-02T14:52:53.531189+0000) 2024-04-02T14:53:38.101 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:37 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:37.787+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-02T14:52:39.933034+0000 front 2024-04-02T14:52:39.933064+0000 (oldest deadline 2024-04-02T14:53:04.632770+0000) 2024-04-02T14:53:38.101 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:37 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:37.787+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.92:6812 osd.4 since back 2024-04-02T14:52:48.133729+0000 front 2024-04-02T14:52:48.133748+0000 (oldest deadline 2024-04-02T14:53:14.033931+0000) 2024-04-02T14:53:38.101 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:37 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:37.787+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-04-02T14:52:54.034568+0000 front 2024-04-02T14:52:54.034525+0000 (oldest deadline 2024-04-02T14:53:23.434953+0000) 2024-04-02T14:53:38.101 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:37 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:37.787+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.90:6822 osd.6 since back 2024-04-02T14:53:05.736111+0000 front 2024-04-02T14:53:05.736207+0000 (oldest deadline 2024-04-02T14:53:27.435911+0000) 2024-04-02T14:53:38.177 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:37 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:37.894+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:50.397138+0000 front 2024-04-02T14:51:50.397096+0000 (oldest deadline 2024-04-02T14:52:37.200213+0000) 2024-04-02T14:53:38.177 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:37 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:37.894+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:22.501322+0000 front 2024-04-02T14:52:22.501346+0000 (oldest deadline 2024-04-02T14:52:45.401451+0000) 2024-04-02T14:53:38.177 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:37 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:37.894+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:28.302204+0000 front 2024-04-02T14:52:28.302332+0000 (oldest deadline 2024-04-02T14:52:57.102868+0000) 2024-04-02T14:53:38.177 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:37 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:37.894+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-02T14:52:38.804314+0000 front 2024-04-02T14:52:38.804365+0000 (oldest deadline 2024-04-02T14:53:04.104362+0000) 2024-04-02T14:53:38.177 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:37 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:37.894+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.92:6812 osd.4 since back 2024-04-02T14:52:47.605500+0000 front 2024-04-02T14:52:47.605485+0000 (oldest deadline 2024-04-02T14:53:10.505428+0000) 2024-04-02T14:53:38.177 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:37 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:37.894+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-04-02T14:52:56.306832+0000 front 2024-04-02T14:52:56.306765+0000 (oldest deadline 2024-04-02T14:53:20.907329+0000) 2024-04-02T14:53:39.101 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:38 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:38.771+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:50.925796+0000 front 2024-04-02T14:51:50.925668+0000 (oldest deadline 2024-04-02T14:52:40.128984+0000) 2024-04-02T14:53:39.101 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:38 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:38.771+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:20.129325+0000 front 2024-04-02T14:52:20.129293+0000 (oldest deadline 2024-04-02T14:52:45.429431+0000) 2024-04-02T14:53:39.101 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:38 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:38.771+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:27.631081+0000 front 2024-04-02T14:52:27.631094+0000 (oldest deadline 2024-04-02T14:52:53.531189+0000) 2024-04-02T14:53:39.101 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:38 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:38.771+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-02T14:52:39.933034+0000 front 2024-04-02T14:52:39.933064+0000 (oldest deadline 2024-04-02T14:53:04.632770+0000) 2024-04-02T14:53:39.101 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:38 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:38.771+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.92:6812 osd.4 since back 2024-04-02T14:52:48.133729+0000 front 2024-04-02T14:52:48.133748+0000 (oldest deadline 2024-04-02T14:53:14.033931+0000) 2024-04-02T14:53:39.101 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:38 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:38.771+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-04-02T14:52:54.034568+0000 front 2024-04-02T14:52:54.034525+0000 (oldest deadline 2024-04-02T14:53:23.434953+0000) 2024-04-02T14:53:39.101 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:38 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:38.771+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.90:6822 osd.6 since back 2024-04-02T14:53:05.736111+0000 front 2024-04-02T14:53:05.736207+0000 (oldest deadline 2024-04-02T14:53:27.435911+0000) 2024-04-02T14:53:39.427 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:38 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:38.943+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:50.397138+0000 front 2024-04-02T14:51:50.397096+0000 (oldest deadline 2024-04-02T14:52:37.200213+0000) 2024-04-02T14:53:39.427 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:38 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:38.943+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:22.501322+0000 front 2024-04-02T14:52:22.501346+0000 (oldest deadline 2024-04-02T14:52:45.401451+0000) 2024-04-02T14:53:39.427 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:38 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:38.943+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:28.302204+0000 front 2024-04-02T14:52:28.302332+0000 (oldest deadline 2024-04-02T14:52:57.102868+0000) 2024-04-02T14:53:39.427 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:38 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:38.943+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-02T14:52:38.804314+0000 front 2024-04-02T14:52:38.804365+0000 (oldest deadline 2024-04-02T14:53:04.104362+0000) 2024-04-02T14:53:39.427 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:38 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:38.943+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.92:6812 osd.4 since back 2024-04-02T14:52:47.605500+0000 front 2024-04-02T14:52:47.605485+0000 (oldest deadline 2024-04-02T14:53:10.505428+0000) 2024-04-02T14:53:39.427 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:38 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:38.943+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-04-02T14:52:56.306832+0000 front 2024-04-02T14:52:56.306765+0000 (oldest deadline 2024-04-02T14:53:20.907329+0000) 2024-04-02T14:53:40.101 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:39 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:39.756+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:50.925796+0000 front 2024-04-02T14:51:50.925668+0000 (oldest deadline 2024-04-02T14:52:40.128984+0000) 2024-04-02T14:53:40.101 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:39 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:39.756+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:20.129325+0000 front 2024-04-02T14:52:20.129293+0000 (oldest deadline 2024-04-02T14:52:45.429431+0000) 2024-04-02T14:53:40.101 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:39 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:39.756+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:27.631081+0000 front 2024-04-02T14:52:27.631094+0000 (oldest deadline 2024-04-02T14:52:53.531189+0000) 2024-04-02T14:53:40.101 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:39 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:39.756+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-02T14:52:39.933034+0000 front 2024-04-02T14:52:39.933064+0000 (oldest deadline 2024-04-02T14:53:04.632770+0000) 2024-04-02T14:53:40.101 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:39 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:39.756+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.92:6812 osd.4 since back 2024-04-02T14:52:48.133729+0000 front 2024-04-02T14:52:48.133748+0000 (oldest deadline 2024-04-02T14:53:14.033931+0000) 2024-04-02T14:53:40.101 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:39 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:39.756+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-04-02T14:52:54.034568+0000 front 2024-04-02T14:52:54.034525+0000 (oldest deadline 2024-04-02T14:53:23.434953+0000) 2024-04-02T14:53:40.101 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:39 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10[61701]: 2024-04-02T14:53:39.756+0000 7f6ada349700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.90:6822 osd.6 since back 2024-04-02T14:53:05.736111+0000 front 2024-04-02T14:53:05.736207+0000 (oldest deadline 2024-04-02T14:53:27.435911+0000) 2024-04-02T14:53:40.427 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:39 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:39.987+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:50.397138+0000 front 2024-04-02T14:51:50.397096+0000 (oldest deadline 2024-04-02T14:52:37.200213+0000) 2024-04-02T14:53:40.427 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:39 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:39.987+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:22.501322+0000 front 2024-04-02T14:52:22.501346+0000 (oldest deadline 2024-04-02T14:52:45.401451+0000) 2024-04-02T14:53:40.427 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:39 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:39.987+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:28.302204+0000 front 2024-04-02T14:52:28.302332+0000 (oldest deadline 2024-04-02T14:52:57.102868+0000) 2024-04-02T14:53:40.427 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:39 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:39.987+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-02T14:52:38.804314+0000 front 2024-04-02T14:52:38.804365+0000 (oldest deadline 2024-04-02T14:53:04.104362+0000) 2024-04-02T14:53:40.427 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:39 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:39.987+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.92:6812 osd.4 since back 2024-04-02T14:52:47.605500+0000 front 2024-04-02T14:52:47.605485+0000 (oldest deadline 2024-04-02T14:53:10.505428+0000) 2024-04-02T14:53:40.427 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:39 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:39.987+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-04-02T14:52:56.306832+0000 front 2024-04-02T14:52:56.306765+0000 (oldest deadline 2024-04-02T14:53:20.907329+0000) 2024-04-02T14:53:41.427 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:40 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:40.987+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:50.397138+0000 front 2024-04-02T14:51:50.397096+0000 (oldest deadline 2024-04-02T14:52:37.200213+0000) 2024-04-02T14:53:41.427 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:40 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:40.987+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:22.501322+0000 front 2024-04-02T14:52:22.501346+0000 (oldest deadline 2024-04-02T14:52:45.401451+0000) 2024-04-02T14:53:41.427 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:40 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:40.987+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:28.302204+0000 front 2024-04-02T14:52:28.302332+0000 (oldest deadline 2024-04-02T14:52:57.102868+0000) 2024-04-02T14:53:41.427 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:40 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:40.987+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-02T14:52:38.804314+0000 front 2024-04-02T14:52:38.804365+0000 (oldest deadline 2024-04-02T14:53:04.104362+0000) 2024-04-02T14:53:41.427 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:40 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:40.987+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.92:6812 osd.4 since back 2024-04-02T14:52:47.605500+0000 front 2024-04-02T14:52:47.605485+0000 (oldest deadline 2024-04-02T14:53:10.505428+0000) 2024-04-02T14:53:41.427 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:40 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:40.987+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-04-02T14:52:56.306832+0000 front 2024-04-02T14:52:56.306765+0000 (oldest deadline 2024-04-02T14:53:20.907329+0000) 2024-04-02T14:53:41.427 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:40 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:40.987+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.92:6820 osd.7 since back 2024-04-02T14:53:14.909553+0000 front 2024-04-02T14:53:14.909583+0000 (oldest deadline 2024-04-02T14:53:40.809508+0000) 2024-04-02T14:53:41.600 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:41 smithi092 podman[74571]: 2024-04-02 14:53:41.35635598 +0000 UTC m=+5.832609094 container died 04dd1c26ccbbbe2599007a28fd9e5c83cbed5d63bddd1850695ff92dbe3615b9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240402.004512-191d963, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240326, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True) 2024-04-02T14:53:42.101 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:41 smithi092 podman[74571]: 2024-04-02 14:53:41.618745736 +0000 UTC m=+6.094998852 container cleanup 04dd1c26ccbbbe2599007a28fd9e5c83cbed5d63bddd1850695ff92dbe3615b9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240402.004512-191d963, ceph=True, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240326) 2024-04-02T14:53:42.101 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:41 smithi092 bash[74571]: ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10 2024-04-02T14:53:42.101 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:41 smithi092 podman[74590]: 2024-04-02 14:53:41.795087398 +0000 UTC m=+0.433844892 container remove 04dd1c26ccbbbe2599007a28fd9e5c83cbed5d63bddd1850695ff92dbe3615b9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240402.004512-191d963, ceph=True, org.label-schema.build-date=20240326, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-04-02T14:53:42.427 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:41 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:41.959+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:50.397138+0000 front 2024-04-02T14:51:50.397096+0000 (oldest deadline 2024-04-02T14:52:37.200213+0000) 2024-04-02T14:53:42.427 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:41 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:41.959+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:22.501322+0000 front 2024-04-02T14:52:22.501346+0000 (oldest deadline 2024-04-02T14:52:45.401451+0000) 2024-04-02T14:53:42.427 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:41 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:41.959+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:28.302204+0000 front 2024-04-02T14:52:28.302332+0000 (oldest deadline 2024-04-02T14:52:57.102868+0000) 2024-04-02T14:53:42.427 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:41 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:41.959+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-02T14:52:38.804314+0000 front 2024-04-02T14:52:38.804365+0000 (oldest deadline 2024-04-02T14:53:04.104362+0000) 2024-04-02T14:53:42.427 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:41 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:41.959+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.92:6812 osd.4 since back 2024-04-02T14:52:47.605500+0000 front 2024-04-02T14:52:47.605485+0000 (oldest deadline 2024-04-02T14:53:10.505428+0000) 2024-04-02T14:53:42.427 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:41 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:41.959+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-04-02T14:52:56.306832+0000 front 2024-04-02T14:52:56.306765+0000 (oldest deadline 2024-04-02T14:53:20.907329+0000) 2024-04-02T14:53:42.427 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:41 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:41.959+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.92:6820 osd.7 since back 2024-04-02T14:53:14.909553+0000 front 2024-04-02T14:53:14.909583+0000 (oldest deadline 2024-04-02T14:53:40.809508+0000) 2024-04-02T14:53:42.481 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:42 smithi092 podman[74677]: 2024-04-02 14:53:42.12091222 +0000 UTC m=+0.017815341 image pull f7301e070545fc23d8e5b7b052a2c15e832a0e57102f1acad8e4f3583709d23f quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534 2024-04-02T14:53:42.481 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:42 smithi092 podman[74677]: 2024-04-02 14:53:42.221081005 +0000 UTC m=+0.117984168 container create fc1af78fce995c6c302fbb1f368d2f426ce57c5f6b025dc4d2ba0733ec086dd0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240402.004512-191d963, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240326, GIT_CLEAN=True) 2024-04-02T14:53:42.481 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:42 smithi092 podman[74677]: 2024-04-02 14:53:42.414889754 +0000 UTC m=+0.311792880 container init fc1af78fce995c6c302fbb1f368d2f426ce57c5f6b025dc4d2ba0733ec086dd0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10-deactivate, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240326, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240402.004512-191d963, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-04-02T14:53:42.481 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:42 smithi092 podman[74677]: 2024-04-02 14:53:42.417946367 +0000 UTC m=+0.314849482 container start fc1af78fce995c6c302fbb1f368d2f426ce57c5f6b025dc4d2ba0733ec086dd0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10-deactivate, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240326, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240402.004512-191d963, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-04-02T14:53:42.481 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:42 smithi092 podman[74677]: 2024-04-02 14:53:42.489779978 +0000 UTC m=+0.386683127 container attach fc1af78fce995c6c302fbb1f368d2f426ce57c5f6b025dc4d2ba0733ec086dd0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240402.004512-191d963, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240326, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-04-02T14:53:42.851 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:42 smithi092 podman[74677]: 2024-04-02 14:53:42.707979612 +0000 UTC m=+0.604882734 container died fc1af78fce995c6c302fbb1f368d2f426ce57c5f6b025dc4d2ba0733ec086dd0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240326, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240402.004512-191d963, GIT_BRANCH=HEAD) 2024-04-02T14:53:43.225 DEBUG:teuthology.orchestra.run.smithi092:> sudo pkill -f 'journalctl -f -n 0 -u ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4@osd.10.service' 2024-04-02T14:53:43.249 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:43 smithi092 podman[74703]: 2024-04-02 14:53:43.218196911 +0000 UTC m=+0.505577790 container remove fc1af78fce995c6c302fbb1f368d2f426ce57c5f6b025dc4d2ba0733ec086dd0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-10-deactivate, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240402.004512-191d963, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240326, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-04-02T14:53:43.249 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:43 smithi092 systemd[1]: ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4@osd.10.service: Deactivated successfully. 2024-04-02T14:53:43.249 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:43 smithi092 systemd[1]: Stopped Ceph osd.10 for 6449448a-f0fc-11ee-b647-cb9ed24678a4. 2024-04-02T14:53:43.249 INFO:journalctl@ceph.osd.10.smithi092.stdout:Apr 02 14:53:43 smithi092 systemd[1]: ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4@osd.10.service: Consumed 7min 55.587s CPU time. 2024-04-02T14:53:43.264 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-02T14:53:43.264 INFO:tasks.cephadm.osd.10:Stopped osd.10 2024-04-02T14:53:43.264 INFO:tasks.cephadm.osd.11:Stopping osd.11... 2024-04-02T14:53:43.264 DEBUG:teuthology.orchestra.run.smithi177:> sudo systemctl stop ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4@osd.11 2024-04-02T14:53:43.294 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:42 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:42.955+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:50.397138+0000 front 2024-04-02T14:51:50.397096+0000 (oldest deadline 2024-04-02T14:52:37.200213+0000) 2024-04-02T14:53:43.294 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:42 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:42.955+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:22.501322+0000 front 2024-04-02T14:52:22.501346+0000 (oldest deadline 2024-04-02T14:52:45.401451+0000) 2024-04-02T14:53:43.294 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:42 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:42.955+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:28.302204+0000 front 2024-04-02T14:52:28.302332+0000 (oldest deadline 2024-04-02T14:52:57.102868+0000) 2024-04-02T14:53:43.294 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:42 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:42.955+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-02T14:52:38.804314+0000 front 2024-04-02T14:52:38.804365+0000 (oldest deadline 2024-04-02T14:53:04.104362+0000) 2024-04-02T14:53:43.294 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:42 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:42.955+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.92:6812 osd.4 since back 2024-04-02T14:52:47.605500+0000 front 2024-04-02T14:52:47.605485+0000 (oldest deadline 2024-04-02T14:53:10.505428+0000) 2024-04-02T14:53:43.294 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:42 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:42.955+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-04-02T14:52:56.306832+0000 front 2024-04-02T14:52:56.306765+0000 (oldest deadline 2024-04-02T14:53:20.907329+0000) 2024-04-02T14:53:43.294 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:42 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:42.955+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.92:6820 osd.7 since back 2024-04-02T14:53:14.909553+0000 front 2024-04-02T14:53:14.909583+0000 (oldest deadline 2024-04-02T14:53:40.809508+0000) 2024-04-02T14:53:43.565 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:43 smithi177 systemd[1]: Stopping Ceph osd.11 for 6449448a-f0fc-11ee-b647-cb9ed24678a4... 2024-04-02T14:53:43.926 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:43 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:43.573+0000 7fba08b3d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-02T14:53:43.927 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:43 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:43.573+0000 7fba08b3d700 -1 osd.11 114 *** Got signal Terminated *** 2024-04-02T14:53:43.927 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:43 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:43.573+0000 7fba08b3d700 -1 osd.11 114 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-02T14:53:44.427 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:43 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:43.982+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:50.397138+0000 front 2024-04-02T14:51:50.397096+0000 (oldest deadline 2024-04-02T14:52:37.200213+0000) 2024-04-02T14:53:44.427 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:43 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:43.982+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:22.501322+0000 front 2024-04-02T14:52:22.501346+0000 (oldest deadline 2024-04-02T14:52:45.401451+0000) 2024-04-02T14:53:44.427 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:43 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:43.982+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:28.302204+0000 front 2024-04-02T14:52:28.302332+0000 (oldest deadline 2024-04-02T14:52:57.102868+0000) 2024-04-02T14:53:44.427 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:43 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:43.982+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-02T14:52:38.804314+0000 front 2024-04-02T14:52:38.804365+0000 (oldest deadline 2024-04-02T14:53:04.104362+0000) 2024-04-02T14:53:44.427 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:43 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:43.982+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.92:6812 osd.4 since back 2024-04-02T14:52:47.605500+0000 front 2024-04-02T14:52:47.605485+0000 (oldest deadline 2024-04-02T14:53:10.505428+0000) 2024-04-02T14:53:44.427 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:43 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:43.982+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-04-02T14:52:56.306832+0000 front 2024-04-02T14:52:56.306765+0000 (oldest deadline 2024-04-02T14:53:20.907329+0000) 2024-04-02T14:53:44.427 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:43 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:43.982+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.92:6820 osd.7 since back 2024-04-02T14:53:14.909553+0000 front 2024-04-02T14:53:14.909583+0000 (oldest deadline 2024-04-02T14:53:40.809508+0000) 2024-04-02T14:53:45.427 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:44 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:44.991+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:50.397138+0000 front 2024-04-02T14:51:50.397096+0000 (oldest deadline 2024-04-02T14:52:37.200213+0000) 2024-04-02T14:53:45.427 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:44 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:44.991+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:22.501322+0000 front 2024-04-02T14:52:22.501346+0000 (oldest deadline 2024-04-02T14:52:45.401451+0000) 2024-04-02T14:53:45.427 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:44 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:44.991+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:28.302204+0000 front 2024-04-02T14:52:28.302332+0000 (oldest deadline 2024-04-02T14:52:57.102868+0000) 2024-04-02T14:53:45.427 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:44 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:44.991+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-02T14:52:38.804314+0000 front 2024-04-02T14:52:38.804365+0000 (oldest deadline 2024-04-02T14:53:04.104362+0000) 2024-04-02T14:53:45.427 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:44 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:44.991+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.92:6812 osd.4 since back 2024-04-02T14:52:47.605500+0000 front 2024-04-02T14:52:47.605485+0000 (oldest deadline 2024-04-02T14:53:10.505428+0000) 2024-04-02T14:53:45.427 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:44 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:44.991+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-04-02T14:52:56.306832+0000 front 2024-04-02T14:52:56.306765+0000 (oldest deadline 2024-04-02T14:53:20.907329+0000) 2024-04-02T14:53:45.427 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:44 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:44.991+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.92:6820 osd.7 since back 2024-04-02T14:53:14.909553+0000 front 2024-04-02T14:53:14.909583+0000 (oldest deadline 2024-04-02T14:53:40.809508+0000) 2024-04-02T14:53:46.427 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:45 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:45.948+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:50.397138+0000 front 2024-04-02T14:51:50.397096+0000 (oldest deadline 2024-04-02T14:52:37.200213+0000) 2024-04-02T14:53:46.427 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:45 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:45.948+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:22.501322+0000 front 2024-04-02T14:52:22.501346+0000 (oldest deadline 2024-04-02T14:52:45.401451+0000) 2024-04-02T14:53:46.427 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:45 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:45.948+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:28.302204+0000 front 2024-04-02T14:52:28.302332+0000 (oldest deadline 2024-04-02T14:52:57.102868+0000) 2024-04-02T14:53:46.427 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:45 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:45.948+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-02T14:52:38.804314+0000 front 2024-04-02T14:52:38.804365+0000 (oldest deadline 2024-04-02T14:53:04.104362+0000) 2024-04-02T14:53:46.427 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:45 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:45.948+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.92:6812 osd.4 since back 2024-04-02T14:52:47.605500+0000 front 2024-04-02T14:52:47.605485+0000 (oldest deadline 2024-04-02T14:53:10.505428+0000) 2024-04-02T14:53:46.427 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:45 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:45.948+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-04-02T14:52:56.306832+0000 front 2024-04-02T14:52:56.306765+0000 (oldest deadline 2024-04-02T14:53:20.907329+0000) 2024-04-02T14:53:46.427 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:45 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:45.948+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.92:6820 osd.7 since back 2024-04-02T14:53:14.909553+0000 front 2024-04-02T14:53:14.909583+0000 (oldest deadline 2024-04-02T14:53:40.809508+0000) 2024-04-02T14:53:46.427 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:45 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:45.948+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.177:6820 osd.8 since back 2024-04-02T14:53:20.809913+0000 front 2024-04-02T14:53:20.809943+0000 (oldest deadline 2024-04-02T14:53:45.509933+0000) 2024-04-02T14:53:47.177 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:46 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:46.918+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:50.397138+0000 front 2024-04-02T14:51:50.397096+0000 (oldest deadline 2024-04-02T14:52:37.200213+0000) 2024-04-02T14:53:47.177 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:46 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:46.919+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:22.501322+0000 front 2024-04-02T14:52:22.501346+0000 (oldest deadline 2024-04-02T14:52:45.401451+0000) 2024-04-02T14:53:47.178 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:46 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:46.919+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:28.302204+0000 front 2024-04-02T14:52:28.302332+0000 (oldest deadline 2024-04-02T14:52:57.102868+0000) 2024-04-02T14:53:47.178 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:46 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:46.919+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-02T14:52:38.804314+0000 front 2024-04-02T14:52:38.804365+0000 (oldest deadline 2024-04-02T14:53:04.104362+0000) 2024-04-02T14:53:47.178 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:46 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:46.919+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.92:6812 osd.4 since back 2024-04-02T14:52:47.605500+0000 front 2024-04-02T14:52:47.605485+0000 (oldest deadline 2024-04-02T14:53:10.505428+0000) 2024-04-02T14:53:47.178 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:46 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:46.919+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-04-02T14:52:56.306832+0000 front 2024-04-02T14:52:56.306765+0000 (oldest deadline 2024-04-02T14:53:20.907329+0000) 2024-04-02T14:53:47.178 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:46 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:46.919+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.92:6820 osd.7 since back 2024-04-02T14:53:14.909553+0000 front 2024-04-02T14:53:14.909583+0000 (oldest deadline 2024-04-02T14:53:40.809508+0000) 2024-04-02T14:53:47.178 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:46 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:46.919+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.177:6820 osd.8 since back 2024-04-02T14:53:20.809913+0000 front 2024-04-02T14:53:20.809943+0000 (oldest deadline 2024-04-02T14:53:45.509933+0000) 2024-04-02T14:53:48.177 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:47 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:47.885+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-04-02T14:51:50.397138+0000 front 2024-04-02T14:51:50.397096+0000 (oldest deadline 2024-04-02T14:52:37.200213+0000) 2024-04-02T14:53:48.177 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:47 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:47.885+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2024-04-02T14:52:22.501322+0000 front 2024-04-02T14:52:22.501346+0000 (oldest deadline 2024-04-02T14:52:45.401451+0000) 2024-04-02T14:53:48.177 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:47 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:47.885+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-02T14:52:28.302204+0000 front 2024-04-02T14:52:28.302332+0000 (oldest deadline 2024-04-02T14:52:57.102868+0000) 2024-04-02T14:53:48.177 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:47 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:47.885+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.90:6814 osd.3 since back 2024-04-02T14:52:38.804314+0000 front 2024-04-02T14:52:38.804365+0000 (oldest deadline 2024-04-02T14:53:04.104362+0000) 2024-04-02T14:53:48.177 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:47 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:47.885+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.92:6812 osd.4 since back 2024-04-02T14:52:47.605500+0000 front 2024-04-02T14:52:47.605485+0000 (oldest deadline 2024-04-02T14:53:10.505428+0000) 2024-04-02T14:53:48.177 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:47 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:47.885+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-04-02T14:52:56.306832+0000 front 2024-04-02T14:52:56.306765+0000 (oldest deadline 2024-04-02T14:53:20.907329+0000) 2024-04-02T14:53:48.177 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:47 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:47.885+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.92:6820 osd.7 since back 2024-04-02T14:53:14.909553+0000 front 2024-04-02T14:53:14.909583+0000 (oldest deadline 2024-04-02T14:53:40.809508+0000) 2024-04-02T14:53:48.177 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:47 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11[61338]: 2024-04-02T14:53:47.885+0000 7fba01129700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.177:6820 osd.8 since back 2024-04-02T14:53:20.809913+0000 front 2024-04-02T14:53:20.809943+0000 (oldest deadline 2024-04-02T14:53:45.509933+0000) 2024-04-02T14:53:49.347 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:49 smithi177 podman[74262]: 2024-04-02 14:53:49.09863654 +0000 UTC m=+5.627116357 container died f9f4927883d60006080304d75cd5fa4007aa15747118d3a55367952f293c99a1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11, RELEASE=wip-batrick-testing-20240402.004512-191d963, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240326, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-02T14:53:49.677 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:49 smithi177 podman[74262]: 2024-04-02 14:53:49.455797416 +0000 UTC m=+5.984277233 container cleanup f9f4927883d60006080304d75cd5fa4007aa15747118d3a55367952f293c99a1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11, org.label-schema.build-date=20240326, RELEASE=wip-batrick-testing-20240402.004512-191d963, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-04-02T14:53:49.677 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:49 smithi177 bash[74262]: ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11 2024-04-02T14:53:50.052 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:49 smithi177 podman[74280]: 2024-04-02 14:53:49.73306406 +0000 UTC m=+0.632610706 container remove f9f4927883d60006080304d75cd5fa4007aa15747118d3a55367952f293c99a1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240326, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240402.004512-191d963, GIT_BRANCH=HEAD) 2024-04-02T14:53:50.368 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:50 smithi177 podman[74376]: 2024-04-02 14:53:49.964293746 +0000 UTC m=+0.017549536 image pull f7301e070545fc23d8e5b7b052a2c15e832a0e57102f1acad8e4f3583709d23f quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534 2024-04-02T14:53:50.368 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:50 smithi177 podman[74376]: 2024-04-02 14:53:50.067102832 +0000 UTC m=+0.120358653 container create e09c7a22003beec089b47ad9f19d38d6d39535c81bfce58c0cb122a8dc9ae4e2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11-deactivate, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240326, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=wip-batrick-testing-20240402.004512-191d963, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-04-02T14:53:50.368 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:50 smithi177 podman[74376]: 2024-04-02 14:53:50.286255186 +0000 UTC m=+0.339510977 container init e09c7a22003beec089b47ad9f19d38d6d39535c81bfce58c0cb122a8dc9ae4e2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11-deactivate, RELEASE=wip-batrick-testing-20240402.004512-191d963, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240326, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, org.label-schema.vendor=CentOS) 2024-04-02T14:53:50.368 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:50 smithi177 podman[74376]: 2024-04-02 14:53:50.289600703 +0000 UTC m=+0.342856492 container start e09c7a22003beec089b47ad9f19d38d6d39535c81bfce58c0cb122a8dc9ae4e2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11-deactivate, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240326, ceph=True, RELEASE=wip-batrick-testing-20240402.004512-191d963, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-04-02T14:53:50.369 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:50 smithi177 podman[74376]: 2024-04-02 14:53:50.377363938 +0000 UTC m=+0.430619722 container attach e09c7a22003beec089b47ad9f19d38d6d39535c81bfce58c0cb122a8dc9ae4e2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11-deactivate, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240326, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240402.004512-191d963) 2024-04-02T14:53:50.677 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:50 smithi177 podman[74376]: 2024-04-02 14:53:50.574311249 +0000 UTC m=+0.627567070 container died e09c7a22003beec089b47ad9f19d38d6d39535c81bfce58c0cb122a8dc9ae4e2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11-deactivate, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240402.004512-191d963, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240326, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-04-02T14:53:51.158 DEBUG:teuthology.orchestra.run.smithi177:> sudo pkill -f 'journalctl -f -n 0 -u ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4@osd.11.service' 2024-04-02T14:53:51.177 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:51 smithi177 podman[74401]: 2024-04-02 14:53:51.13900247 +0000 UTC m=+0.556148051 container remove e09c7a22003beec089b47ad9f19d38d6d39535c81bfce58c0cb122a8dc9ae4e2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-osd-11-deactivate, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240326, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240402.004512-191d963, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-04-02T14:53:51.177 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:51 smithi177 systemd[1]: ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4@osd.11.service: Deactivated successfully. 2024-04-02T14:53:51.177 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:51 smithi177 systemd[1]: Stopped Ceph osd.11 for 6449448a-f0fc-11ee-b647-cb9ed24678a4. 2024-04-02T14:53:51.177 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 02 14:53:51 smithi177 systemd[1]: ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4@osd.11.service: Consumed 8min 5.861s CPU time. 2024-04-02T14:53:51.217 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-02T14:53:51.217 INFO:tasks.cephadm.osd.11:Stopped osd.11 2024-04-02T14:53:51.217 INFO:tasks.cephadm.mds.a:Stopping mds.a... 2024-04-02T14:53:51.217 DEBUG:teuthology.orchestra.run.smithi090:> sudo systemctl stop ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4@mds.a 2024-04-02T14:53:51.679 INFO:journalctl@ceph.mds.a.smithi090.stdout:Apr 02 14:53:51 smithi090 systemd[1]: Stopping Ceph mds.a for 6449448a-f0fc-11ee-b647-cb9ed24678a4... 2024-04-02T14:53:52.179 INFO:journalctl@ceph.mds.a.smithi090.stdout:Apr 02 14:53:51 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mds-a[73483]: 2024-04-02T14:53:51.795+0000 7f9108e00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-02T14:53:52.180 INFO:journalctl@ceph.mds.a.smithi090.stdout:Apr 02 14:53:51 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mds-a[73483]: 2024-04-02T14:53:51.795+0000 7f9108e00700 -1 mds.a *** got signal Terminated *** 2024-04-02T14:53:54.271 INFO:journalctl@ceph.mds.a.smithi090.stdout:Apr 02 14:53:54 smithi090 podman[111506]: 2024-04-02 14:53:54.015861588 +0000 UTC m=+2.341699732 container died a132ebe3f49248be2bff7507d037d9c01ee94a0e1cb386bc50535158667b8536 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mds-a, GIT_BRANCH=HEAD, org.label-schema.build-date=20240326, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240402.004512-191d963, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-04-02T14:53:54.577 INFO:journalctl@ceph.mds.a.smithi090.stdout:Apr 02 14:53:54 smithi090 podman[111506]: 2024-04-02 14:53:54.425619405 +0000 UTC m=+2.751457539 container cleanup a132ebe3f49248be2bff7507d037d9c01ee94a0e1cb386bc50535158667b8536 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mds-a, org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240326, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240402.004512-191d963, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-04-02T14:53:54.577 INFO:journalctl@ceph.mds.a.smithi090.stdout:Apr 02 14:53:54 smithi090 bash[111506]: ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mds-a 2024-04-02T14:53:54.928 INFO:journalctl@ceph.mds.a.smithi090.stdout:Apr 02 14:53:54 smithi090 podman[111520]: 2024-04-02 14:53:54.585030504 +0000 UTC m=+0.556670957 container remove a132ebe3f49248be2bff7507d037d9c01ee94a0e1cb386bc50535158667b8536 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mds-a, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240326, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=wip-batrick-testing-20240402.004512-191d963) 2024-04-02T14:53:55.075 DEBUG:teuthology.orchestra.run.smithi090:> sudo pkill -f 'journalctl -f -n 0 -u ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4@mds.a.service' 2024-04-02T14:53:55.195 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-02T14:53:55.195 INFO:tasks.cephadm.mds.a:Stopped mds.a 2024-04-02T14:53:55.196 INFO:tasks.cephadm.mds.d:Stopping mds.d... 2024-04-02T14:53:55.196 DEBUG:teuthology.orchestra.run.smithi090:> sudo systemctl stop ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4@mds.d 2024-04-02T14:53:55.678 INFO:journalctl@ceph.mds.d.smithi090.stdout:Apr 02 14:53:55 smithi090 systemd[1]: Stopping Ceph mds.d for 6449448a-f0fc-11ee-b647-cb9ed24678a4... 2024-04-02T14:53:56.178 INFO:journalctl@ceph.mds.d.smithi090.stdout:Apr 02 14:53:55 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mds-d[72503]: 2024-04-02T14:53:55.760+0000 7fdbd7000700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.d -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-02T14:53:56.179 INFO:journalctl@ceph.mds.d.smithi090.stdout:Apr 02 14:53:55 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mds-d[72503]: 2024-04-02T14:53:55.760+0000 7fdbd7000700 -1 mds.d *** got signal Terminated *** 2024-04-02T14:54:00.043 INFO:journalctl@ceph.mds.d.smithi090.stdout:Apr 02 14:53:59 smithi090 podman[111626]: 2024-04-02 14:53:59.779817801 +0000 UTC m=+4.150269031 container died 675ab21b436c576e67e74eedf917205f2b15bff912fea39abee86a79c66cf35a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mds-d, org.label-schema.build-date=20240326, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240402.004512-191d963) 2024-04-02T14:54:00.356 INFO:journalctl@ceph.mds.d.smithi090.stdout:Apr 02 14:54:00 smithi090 podman[111626]: 2024-04-02 14:54:00.204997818 +0000 UTC m=+4.575449099 container cleanup 675ab21b436c576e67e74eedf917205f2b15bff912fea39abee86a79c66cf35a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mds-d, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240402.004512-191d963, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240326) 2024-04-02T14:54:00.356 INFO:journalctl@ceph.mds.d.smithi090.stdout:Apr 02 14:54:00 smithi090 bash[111626]: ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mds-d 2024-04-02T14:54:00.678 INFO:journalctl@ceph.mds.d.smithi090.stdout:Apr 02 14:54:00 smithi090 podman[111651]: 2024-04-02 14:54:00.364494277 +0000 UTC m=+0.573669504 container remove 675ab21b436c576e67e74eedf917205f2b15bff912fea39abee86a79c66cf35a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mds-d, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240402.004512-191d963, org.label-schema.build-date=20240326, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-02T14:54:00.825 DEBUG:teuthology.orchestra.run.smithi090:> sudo pkill -f 'journalctl -f -n 0 -u ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4@mds.d.service' 2024-04-02T14:54:00.943 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-02T14:54:00.943 INFO:tasks.cephadm.mds.d:Stopped mds.d 2024-04-02T14:54:00.944 INFO:tasks.cephadm.mds.g:Stopping mds.g... 2024-04-02T14:54:00.944 DEBUG:teuthology.orchestra.run.smithi090:> sudo systemctl stop ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4@mds.g 2024-04-02T14:54:01.178 INFO:journalctl@ceph.mds.g.smithi090.stdout:Apr 02 14:54:01 smithi090 systemd[1]: Stopping Ceph mds.g for 6449448a-f0fc-11ee-b647-cb9ed24678a4... 2024-04-02T14:54:01.929 INFO:journalctl@ceph.mds.g.smithi090.stdout:Apr 02 14:54:01 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mds-g[73104]: 2024-04-02T14:54:01.465+0000 7fe8ad200700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.g -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-02T14:54:01.929 INFO:journalctl@ceph.mds.g.smithi090.stdout:Apr 02 14:54:01 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mds-g[73104]: 2024-04-02T14:54:01.465+0000 7fe8ad200700 -1 mds.g *** got signal Terminated *** 2024-04-02T14:54:03.499 INFO:journalctl@ceph.mds.g.smithi090.stdout:Apr 02 14:54:03 smithi090 podman[111758]: 2024-04-02 14:54:03.229094163 +0000 UTC m=+1.889167216 container died 6aff0442039416e961ffd68fd6dad56c48c5e4663165ee6520f5165b663dd27b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mds-g, RELEASE=wip-batrick-testing-20240402.004512-191d963, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240326) 2024-04-02T14:54:03.820 INFO:journalctl@ceph.mds.g.smithi090.stdout:Apr 02 14:54:03 smithi090 podman[111758]: 2024-04-02 14:54:03.659941046 +0000 UTC m=+2.320014092 container cleanup 6aff0442039416e961ffd68fd6dad56c48c5e4663165ee6520f5165b663dd27b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mds-g, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240326, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=wip-batrick-testing-20240402.004512-191d963, CEPH_POINT_RELEASE=, ceph=True) 2024-04-02T14:54:03.820 INFO:journalctl@ceph.mds.g.smithi090.stdout:Apr 02 14:54:03 smithi090 bash[111758]: ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mds-g 2024-04-02T14:54:04.178 INFO:journalctl@ceph.mds.g.smithi090.stdout:Apr 02 14:54:03 smithi090 podman[111772]: 2024-04-02 14:54:03.828291937 +0000 UTC m=+0.583768568 container remove 6aff0442039416e961ffd68fd6dad56c48c5e4663165ee6520f5165b663dd27b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mds-g, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240326, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240402.004512-191d963) 2024-04-02T14:54:04.287 DEBUG:teuthology.orchestra.run.smithi090:> sudo pkill -f 'journalctl -f -n 0 -u ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4@mds.g.service' 2024-04-02T14:54:04.405 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-02T14:54:04.405 INFO:tasks.cephadm.mds.g:Stopped mds.g 2024-04-02T14:54:04.405 INFO:tasks.cephadm.mds.j:Stopping mds.j... 2024-04-02T14:54:04.405 DEBUG:teuthology.orchestra.run.smithi090:> sudo systemctl stop ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4@mds.j 2024-04-02T14:54:04.928 INFO:journalctl@ceph.mds.j.smithi090.stdout:Apr 02 14:54:04 smithi090 systemd[1]: Stopping Ceph mds.j for 6449448a-f0fc-11ee-b647-cb9ed24678a4... 2024-04-02T14:54:05.428 INFO:journalctl@ceph.mds.j.smithi090.stdout:Apr 02 14:54:04 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mds-j[72161]: 2024-04-02T14:54:04.994+0000 7fb1aec00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.j -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-02T14:54:05.429 INFO:journalctl@ceph.mds.j.smithi090.stdout:Apr 02 14:54:04 smithi090 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mds-j[72161]: 2024-04-02T14:54:04.994+0000 7fb1aec00700 -1 mds.j *** got signal Terminated *** 2024-04-02T14:54:07.519 INFO:journalctl@ceph.mds.j.smithi090.stdout:Apr 02 14:54:07 smithi090 podman[111876]: 2024-04-02 14:54:07.248119834 +0000 UTC m=+2.374693400 container died d0bfbc2f5a235c81910098034061e6dfc083cd7b5b700ffd017863330a77e1d0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mds-j, org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240326, RELEASE=wip-batrick-testing-20240402.004512-191d963, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-02T14:54:07.873 INFO:journalctl@ceph.mds.j.smithi090.stdout:Apr 02 14:54:07 smithi090 podman[111876]: 2024-04-02 14:54:07.639563603 +0000 UTC m=+2.766137183 container cleanup d0bfbc2f5a235c81910098034061e6dfc083cd7b5b700ffd017863330a77e1d0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mds-j, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240402.004512-191d963, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240326, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5) 2024-04-02T14:54:07.873 INFO:journalctl@ceph.mds.j.smithi090.stdout:Apr 02 14:54:07 smithi090 bash[111876]: ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mds-j 2024-04-02T14:54:08.178 INFO:journalctl@ceph.mds.j.smithi090.stdout:Apr 02 14:54:07 smithi090 podman[111890]: 2024-04-02 14:54:07.882204248 +0000 UTC m=+0.620548237 container remove d0bfbc2f5a235c81910098034061e6dfc083cd7b5b700ffd017863330a77e1d0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mds-j, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240402.004512-191d963, ceph=True, org.label-schema.build-date=20240326, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-04-02T14:54:08.318 DEBUG:teuthology.orchestra.run.smithi090:> sudo pkill -f 'journalctl -f -n 0 -u ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4@mds.j.service' 2024-04-02T14:54:08.426 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-02T14:54:08.426 INFO:tasks.cephadm.mds.j:Stopped mds.j 2024-04-02T14:54:08.426 INFO:tasks.cephadm.mds.b:Stopping mds.b... 2024-04-02T14:54:08.426 DEBUG:teuthology.orchestra.run.smithi092:> sudo systemctl stop ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4@mds.b 2024-04-02T14:54:08.724 INFO:journalctl@ceph.mds.b.smithi092.stdout:Apr 02 14:54:08 smithi092 systemd[1]: Stopping Ceph mds.b for 6449448a-f0fc-11ee-b647-cb9ed24678a4... 2024-04-02T14:54:08.996 INFO:journalctl@ceph.mds.b.smithi092.stdout:Apr 02 14:54:08 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mds-b[67422]: 2024-04-02T14:54:08.732+0000 7ff765637700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-02T14:54:08.997 INFO:journalctl@ceph.mds.b.smithi092.stdout:Apr 02 14:54:08 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mds-b[67422]: 2024-04-02T14:54:08.732+0000 7ff765637700 -1 mds.b *** got signal Terminated *** 2024-04-02T14:54:11.549 INFO:journalctl@ceph.mds.b.smithi092.stdout:Apr 02 14:54:11 smithi092 podman[74783]: 2024-04-02 14:54:11.20338335 +0000 UTC m=+2.577264776 container died 1f09a480542acfd0be9c6d797a51dc98d110671d4ba1fec86d0c0fdc6ee6a18d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mds-b, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240402.004512-191d963, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240326, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-04-02T14:54:11.799 INFO:journalctl@ceph.mds.b.smithi092.stdout:Apr 02 14:54:11 smithi092 podman[74783]: 2024-04-02 14:54:11.640935108 +0000 UTC m=+3.014816505 container cleanup 1f09a480542acfd0be9c6d797a51dc98d110671d4ba1fec86d0c0fdc6ee6a18d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mds-b, org.label-schema.build-date=20240326, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240402.004512-191d963, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5) 2024-04-02T14:54:11.800 INFO:journalctl@ceph.mds.b.smithi092.stdout:Apr 02 14:54:11 smithi092 bash[74783]: ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mds-b 2024-04-02T14:54:12.044 DEBUG:teuthology.orchestra.run.smithi092:> sudo pkill -f 'journalctl -f -n 0 -u ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4@mds.b.service' 2024-04-02T14:54:12.068 INFO:journalctl@ceph.mds.b.smithi092.stdout:Apr 02 14:54:11 smithi092 podman[74809]: 2024-04-02 14:54:11.80879079 +0000 UTC m=+0.604371752 container remove 1f09a480542acfd0be9c6d797a51dc98d110671d4ba1fec86d0c0fdc6ee6a18d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mds-b, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240326, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, RELEASE=wip-batrick-testing-20240402.004512-191d963) 2024-04-02T14:54:12.068 INFO:journalctl@ceph.mds.b.smithi092.stdout:Apr 02 14:54:12 smithi092 systemd[1]: ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4@mds.b.service: Deactivated successfully. 2024-04-02T14:54:12.069 INFO:journalctl@ceph.mds.b.smithi092.stdout:Apr 02 14:54:12 smithi092 systemd[1]: Stopped Ceph mds.b for 6449448a-f0fc-11ee-b647-cb9ed24678a4. 2024-04-02T14:54:12.069 INFO:journalctl@ceph.mds.b.smithi092.stdout:Apr 02 14:54:12 smithi092 systemd[1]: ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4@mds.b.service: Consumed 35.461s CPU time. 2024-04-02T14:54:12.082 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-02T14:54:12.082 INFO:tasks.cephadm.mds.b:Stopped mds.b 2024-04-02T14:54:12.082 INFO:tasks.cephadm.mds.e:Stopping mds.e... 2024-04-02T14:54:12.082 DEBUG:teuthology.orchestra.run.smithi092:> sudo systemctl stop ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4@mds.e 2024-04-02T14:54:12.350 INFO:journalctl@ceph.mds.e.smithi092.stdout:Apr 02 14:54:12 smithi092 systemd[1]: Stopping Ceph mds.e for 6449448a-f0fc-11ee-b647-cb9ed24678a4... 2024-04-02T14:54:12.851 INFO:journalctl@ceph.mds.e.smithi092.stdout:Apr 02 14:54:12 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mds-e[66791]: 2024-04-02T14:54:12.389+0000 7f4867209700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.e -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-02T14:54:12.851 INFO:journalctl@ceph.mds.e.smithi092.stdout:Apr 02 14:54:12 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mds-e[66791]: 2024-04-02T14:54:12.389+0000 7f4867209700 -1 mds.e *** got signal Terminated *** 2024-04-02T14:54:15.444 INFO:journalctl@ceph.mds.e.smithi092.stdout:Apr 02 14:54:15 smithi092 podman[74916]: 2024-04-02 14:54:15.18985291 +0000 UTC m=+2.882652076 container died bf500f992b9b82e9bc3794a57e60d72326a20177c60a2a9b1ac24698e5759f1d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mds-e, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240402.004512-191d963, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240326, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-02T14:54:15.786 INFO:journalctl@ceph.mds.e.smithi092.stdout:Apr 02 14:54:15 smithi092 podman[74916]: 2024-04-02 14:54:15.544197557 +0000 UTC m=+3.236996730 container cleanup bf500f992b9b82e9bc3794a57e60d72326a20177c60a2a9b1ac24698e5759f1d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mds-e, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240326, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=wip-batrick-testing-20240402.004512-191d963, maintainer=Guillaume Abrioux ) 2024-04-02T14:54:15.786 INFO:journalctl@ceph.mds.e.smithi092.stdout:Apr 02 14:54:15 smithi092 bash[74916]: ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mds-e 2024-04-02T14:54:15.949 DEBUG:teuthology.orchestra.run.smithi092:> sudo pkill -f 'journalctl -f -n 0 -u ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4@mds.e.service' 2024-04-02T14:54:15.985 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-02T14:54:15.985 INFO:tasks.cephadm.mds.e:Stopped mds.e 2024-04-02T14:54:15.985 INFO:tasks.cephadm.mds.h:Stopping mds.h... 2024-04-02T14:54:15.985 DEBUG:teuthology.orchestra.run.smithi092:> sudo systemctl stop ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4@mds.h 2024-04-02T14:54:16.350 INFO:journalctl@ceph.mds.h.smithi092.stdout:Apr 02 14:54:16 smithi092 systemd[1]: Stopping Ceph mds.h for 6449448a-f0fc-11ee-b647-cb9ed24678a4... 2024-04-02T14:54:16.350 INFO:journalctl@ceph.mds.h.smithi092.stdout:Apr 02 14:54:16 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mds-h[67113]: 2024-04-02T14:54:16.305+0000 7fc6e5eee700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.h -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-02T14:54:16.351 INFO:journalctl@ceph.mds.h.smithi092.stdout:Apr 02 14:54:16 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mds-h[67113]: 2024-04-02T14:54:16.305+0000 7fc6e5eee700 -1 mds.h *** got signal Terminated *** 2024-04-02T14:54:19.432 INFO:journalctl@ceph.mds.h.smithi092.stdout:Apr 02 14:54:19 smithi092 podman[75035]: 2024-04-02 14:54:19.187989419 +0000 UTC m=+2.980414002 container died 1d4fecb3a6534db2a0288f7540ff8facc76d62efb44248ebfbac325a378023f2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mds-h, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240402.004512-191d963, org.label-schema.build-date=20240326, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-04-02T14:54:19.839 INFO:journalctl@ceph.mds.h.smithi092.stdout:Apr 02 14:54:19 smithi092 podman[75035]: 2024-04-02 14:54:19.60589348 +0000 UTC m=+3.398318051 container cleanup 1d4fecb3a6534db2a0288f7540ff8facc76d62efb44248ebfbac325a378023f2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mds-h, RELEASE=wip-batrick-testing-20240402.004512-191d963, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240326, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5) 2024-04-02T14:54:19.839 INFO:journalctl@ceph.mds.h.smithi092.stdout:Apr 02 14:54:19 smithi092 bash[75035]: ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mds-h 2024-04-02T14:54:19.968 DEBUG:teuthology.orchestra.run.smithi092:> sudo pkill -f 'journalctl -f -n 0 -u ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4@mds.h.service' 2024-04-02T14:54:20.006 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-02T14:54:20.007 INFO:tasks.cephadm.mds.h:Stopped mds.h 2024-04-02T14:54:20.007 INFO:tasks.cephadm.mds.k:Stopping mds.k... 2024-04-02T14:54:20.007 DEBUG:teuthology.orchestra.run.smithi092:> sudo systemctl stop ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4@mds.k 2024-04-02T14:54:20.342 INFO:journalctl@ceph.mds.k.smithi092.stdout:Apr 02 14:54:20 smithi092 systemd[1]: Stopping Ceph mds.k for 6449448a-f0fc-11ee-b647-cb9ed24678a4... 2024-04-02T14:54:20.600 INFO:journalctl@ceph.mds.k.smithi092.stdout:Apr 02 14:54:20 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mds-k[66466]: 2024-04-02T14:54:20.351+0000 7fa045782700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.k -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-02T14:54:20.600 INFO:journalctl@ceph.mds.k.smithi092.stdout:Apr 02 14:54:20 smithi092 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mds-k[66466]: 2024-04-02T14:54:20.351+0000 7fa045782700 -1 mds.k *** got signal Terminated *** 2024-04-02T14:54:23.559 INFO:journalctl@ceph.mds.k.smithi092.stdout:Apr 02 14:54:23 smithi092 podman[75157]: 2024-04-02 14:54:23.200631009 +0000 UTC m=+2.967341960 container died e4972a161fe41e632a181beea3c83297e14ab752718bd75694072d3b571eb5b1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mds-k, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240326, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=wip-batrick-testing-20240402.004512-191d963, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-04-02T14:54:23.850 INFO:journalctl@ceph.mds.k.smithi092.stdout:Apr 02 14:54:23 smithi092 podman[75157]: 2024-04-02 14:54:23.651219192 +0000 UTC m=+3.417930136 container cleanup e4972a161fe41e632a181beea3c83297e14ab752718bd75694072d3b571eb5b1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mds-k, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240326, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=wip-batrick-testing-20240402.004512-191d963, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-04-02T14:54:23.850 INFO:journalctl@ceph.mds.k.smithi092.stdout:Apr 02 14:54:23 smithi092 bash[75157]: ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mds-k 2024-04-02T14:54:24.111 DEBUG:teuthology.orchestra.run.smithi092:> sudo pkill -f 'journalctl -f -n 0 -u ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4@mds.k.service' 2024-04-02T14:54:24.146 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-02T14:54:24.146 INFO:tasks.cephadm.mds.k:Stopped mds.k 2024-04-02T14:54:24.146 INFO:tasks.cephadm.mds.c:Stopping mds.c... 2024-04-02T14:54:24.146 DEBUG:teuthology.orchestra.run.smithi177:> sudo systemctl stop ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4@mds.c 2024-04-02T14:54:24.426 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 02 14:54:24 smithi177 systemd[1]: Stopping Ceph mds.c for 6449448a-f0fc-11ee-b647-cb9ed24678a4... 2024-04-02T14:54:24.687 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 02 14:54:24 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mds-c[66371]: 2024-04-02T14:54:24.445+0000 7f74692c3700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-02T14:54:24.687 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 02 14:54:24 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mds-c[66371]: 2024-04-02T14:54:24.445+0000 7f74692c3700 -1 mds.c *** got signal Terminated *** 2024-04-02T14:54:27.503 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 02 14:54:27 smithi177 podman[74481]: 2024-04-02 14:54:27.193067729 +0000 UTC m=+2.851052459 container died ab5bb25e316a74a8b503c0000f6bfce72e9a2438a34696c0ed85f1f14404585b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mds-c, org.label-schema.build-date=20240326, org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240402.004512-191d963, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-04-02T14:54:27.778 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 02 14:54:27 smithi177 podman[74481]: 2024-04-02 14:54:27.603190588 +0000 UTC m=+3.261175309 container cleanup ab5bb25e316a74a8b503c0000f6bfce72e9a2438a34696c0ed85f1f14404585b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mds-c, GIT_BRANCH=HEAD, ceph=True, RELEASE=wip-batrick-testing-20240402.004512-191d963, io.buildah.version=1.33.5, org.label-schema.build-date=20240326, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-04-02T14:54:27.778 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 02 14:54:27 smithi177 bash[74481]: ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mds-c 2024-04-02T14:54:28.021 DEBUG:teuthology.orchestra.run.smithi177:> sudo pkill -f 'journalctl -f -n 0 -u ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4@mds.c.service' 2024-04-02T14:54:28.047 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 02 14:54:27 smithi177 podman[74507]: 2024-04-02 14:54:27.787714318 +0000 UTC m=+0.593417411 container remove ab5bb25e316a74a8b503c0000f6bfce72e9a2438a34696c0ed85f1f14404585b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mds-c, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240326, RELEASE=wip-batrick-testing-20240402.004512-191d963, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-04-02T14:54:28.047 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 02 14:54:28 smithi177 systemd[1]: ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4@mds.c.service: Deactivated successfully. 2024-04-02T14:54:28.047 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 02 14:54:28 smithi177 systemd[1]: Stopped Ceph mds.c for 6449448a-f0fc-11ee-b647-cb9ed24678a4. 2024-04-02T14:54:28.047 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 02 14:54:28 smithi177 systemd[1]: ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4@mds.c.service: Consumed 2.544s CPU time. 2024-04-02T14:54:28.060 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-02T14:54:28.061 INFO:tasks.cephadm.mds.c:Stopped mds.c 2024-04-02T14:54:28.061 INFO:tasks.cephadm.mds.f:Stopping mds.f... 2024-04-02T14:54:28.061 DEBUG:teuthology.orchestra.run.smithi177:> sudo systemctl stop ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4@mds.f 2024-04-02T14:54:28.381 INFO:journalctl@ceph.mds.f.smithi177.stdout:Apr 02 14:54:28 smithi177 systemd[1]: Stopping Ceph mds.f for 6449448a-f0fc-11ee-b647-cb9ed24678a4... 2024-04-02T14:54:28.676 INFO:journalctl@ceph.mds.f.smithi177.stdout:Apr 02 14:54:28 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mds-f[66068]: 2024-04-02T14:54:28.390+0000 7f91a514e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.f -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-02T14:54:28.677 INFO:journalctl@ceph.mds.f.smithi177.stdout:Apr 02 14:54:28 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mds-f[66068]: 2024-04-02T14:54:28.390+0000 7f91a514e700 -1 mds.f *** got signal Terminated *** 2024-04-02T14:54:31.456 INFO:journalctl@ceph.mds.f.smithi177.stdout:Apr 02 14:54:31 smithi177 podman[74615]: 2024-04-02 14:54:31.196654051 +0000 UTC m=+2.910688813 container died 20637d31e333ede5df8ec43de524ab8a118131f4c03a6423c8e9c29d4ee77265 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mds-f, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240326, ceph=True, RELEASE=wip-batrick-testing-20240402.004512-191d963, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-04-02T14:54:31.798 INFO:journalctl@ceph.mds.f.smithi177.stdout:Apr 02 14:54:31 smithi177 podman[74615]: 2024-04-02 14:54:31.54014808 +0000 UTC m=+3.254182837 container cleanup 20637d31e333ede5df8ec43de524ab8a118131f4c03a6423c8e9c29d4ee77265 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mds-f, ceph=True, org.label-schema.build-date=20240326, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240402.004512-191d963, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-04-02T14:54:31.798 INFO:journalctl@ceph.mds.f.smithi177.stdout:Apr 02 14:54:31 smithi177 bash[74615]: ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mds-f 2024-04-02T14:54:31.951 DEBUG:teuthology.orchestra.run.smithi177:> sudo pkill -f 'journalctl -f -n 0 -u ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4@mds.f.service' 2024-04-02T14:54:31.988 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-02T14:54:31.988 INFO:tasks.cephadm.mds.f:Stopped mds.f 2024-04-02T14:54:31.988 INFO:tasks.cephadm.mds.i:Stopping mds.i... 2024-04-02T14:54:31.989 DEBUG:teuthology.orchestra.run.smithi177:> sudo systemctl stop ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4@mds.i 2024-04-02T14:54:32.317 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 02 14:54:32 smithi177 systemd[1]: Stopping Ceph mds.i for 6449448a-f0fc-11ee-b647-cb9ed24678a4... 2024-04-02T14:54:32.676 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 02 14:54:32 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mds-i[66682]: 2024-04-02T14:54:32.325+0000 7f19fedee700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.i -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-02T14:54:32.677 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 02 14:54:32 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mds-i[66682]: 2024-04-02T14:54:32.325+0000 7f19fedee700 -1 mds.i *** got signal Terminated *** 2024-04-02T14:54:35.426 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 02 14:54:35 smithi177 podman[74739]: 2024-04-02 14:54:35.20515596 +0000 UTC m=+2.989500599 container died 689f6bf561b195e380da9b1fc9af69ee836d5c3417fdd6c4402e0a127d113595 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mds-i, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240326, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240402.004512-191d963, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-04-02T14:54:35.809 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 02 14:54:35 smithi177 podman[74739]: 2024-04-02 14:54:35.634529664 +0000 UTC m=+3.418874295 container cleanup 689f6bf561b195e380da9b1fc9af69ee836d5c3417fdd6c4402e0a127d113595 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mds-i, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240326, RELEASE=wip-batrick-testing-20240402.004512-191d963, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-04-02T14:54:35.810 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 02 14:54:35 smithi177 bash[74739]: ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mds-i 2024-04-02T14:54:36.030 DEBUG:teuthology.orchestra.run.smithi177:> sudo pkill -f 'journalctl -f -n 0 -u ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4@mds.i.service' 2024-04-02T14:54:36.065 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-02T14:54:36.065 INFO:tasks.cephadm.mds.i:Stopped mds.i 2024-04-02T14:54:36.066 INFO:tasks.cephadm.mds.l:Stopping mds.l... 2024-04-02T14:54:36.066 DEBUG:teuthology.orchestra.run.smithi177:> sudo systemctl stop ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4@mds.l 2024-04-02T14:54:36.387 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 02 14:54:36 smithi177 systemd[1]: Stopping Ceph mds.l for 6449448a-f0fc-11ee-b647-cb9ed24678a4... 2024-04-02T14:54:36.676 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 02 14:54:36 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mds-l[65756]: 2024-04-02T14:54:36.396+0000 7f8a4e7c0700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.l -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-02T14:54:36.676 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 02 14:54:36 smithi177 ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mds-l[65756]: 2024-04-02T14:54:36.396+0000 7f8a4e7c0700 -1 mds.l *** got signal Terminated *** 2024-04-02T14:54:39.521 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 02 14:54:39 smithi177 podman[74860]: 2024-04-02 14:54:39.206108539 +0000 UTC m=+2.909844262 container died 4bd8c1a305ce27556da2de18974dfa1b668caa4851466acf21193c539d78da79 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mds-l, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240402.004512-191d963, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240326, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-04-02T14:54:39.821 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 02 14:54:39 smithi177 podman[74860]: 2024-04-02 14:54:39.613023491 +0000 UTC m=+3.316759200 container cleanup 4bd8c1a305ce27556da2de18974dfa1b668caa4851466acf21193c539d78da79 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:453f87f941362961d3118868553a25727c89e5b7c0629cbb7db5be9ea34ce534, name=ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mds-l, org.label-schema.build-date=20240326, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240402.004512-191d963, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2) 2024-04-02T14:54:39.821 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 02 14:54:39 smithi177 bash[74860]: ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4-mds-l 2024-04-02T14:54:40.030 DEBUG:teuthology.orchestra.run.smithi177:> sudo pkill -f 'journalctl -f -n 0 -u ceph-6449448a-f0fc-11ee-b647-cb9ed24678a4@mds.l.service' 2024-04-02T14:54:40.064 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-02T14:54:40.065 INFO:tasks.cephadm.mds.l:Stopped mds.l 2024-04-02T14:54:40.065 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 6449448a-f0fc-11ee-b647-cb9ed24678a4 --force --keep-logs 2024-04-02T14:54:40.391 INFO:teuthology.orchestra.run.smithi090.stdout:Deleting cluster with fsid: 6449448a-f0fc-11ee-b647-cb9ed24678a4 2024-04-02T14:54:45.392 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 6449448a-f0fc-11ee-b647-cb9ed24678a4 --force --keep-logs 2024-04-02T14:54:45.583 INFO:teuthology.orchestra.run.smithi092.stdout:Deleting cluster with fsid: 6449448a-f0fc-11ee-b647-cb9ed24678a4 2024-04-02T14:54:48.134 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 6449448a-f0fc-11ee-b647-cb9ed24678a4 --force --keep-logs 2024-04-02T14:54:48.326 INFO:teuthology.orchestra.run.smithi177.stdout:Deleting cluster with fsid: 6449448a-f0fc-11ee-b647-cb9ed24678a4 2024-04-02T14:54:50.774 DEBUG:teuthology.orchestra.run.smithi090:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-02T14:54:50.876 DEBUG:teuthology.orchestra.run.smithi092:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-02T14:54:50.911 DEBUG:teuthology.orchestra.run.smithi177:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-02T14:54:50.940 INFO:tasks.cephadm:Archiving crash dumps... 2024-04-02T14:54:50.942 DEBUG:teuthology.misc:Transferring archived files from smithi090:/var/lib/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/crash to /home/teuthworker/archive/pdonnell-2024-04-02_11:52:43-fs-wip-batrick-testing-20240402.004512-distro-default-smithi/7635633/remote/smithi090/crash 2024-04-02T14:54:50.943 DEBUG:teuthology.orchestra.run.smithi090:> sudo tar c -f - -C /var/lib/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/crash -- . 2024-04-02T14:54:51.081 INFO:teuthology.orchestra.run.smithi090.stderr:tar: /var/lib/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/crash: Cannot open: No such file or directory 2024-04-02T14:54:51.081 INFO:teuthology.orchestra.run.smithi090.stderr:tar: Error is not recoverable: exiting now 2024-04-02T14:54:51.087 DEBUG:teuthology.misc:Transferring archived files from smithi092:/var/lib/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/crash to /home/teuthworker/archive/pdonnell-2024-04-02_11:52:43-fs-wip-batrick-testing-20240402.004512-distro-default-smithi/7635633/remote/smithi092/crash 2024-04-02T14:54:51.088 DEBUG:teuthology.orchestra.run.smithi092:> sudo tar c -f - -C /var/lib/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/crash -- . 2024-04-02T14:54:51.134 INFO:teuthology.orchestra.run.smithi092.stderr:tar: /var/lib/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/crash: Cannot open: No such file or directory 2024-04-02T14:54:51.134 INFO:teuthology.orchestra.run.smithi092.stderr:tar: Error is not recoverable: exiting now 2024-04-02T14:54:51.136 DEBUG:teuthology.misc:Transferring archived files from smithi177:/var/lib/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/crash to /home/teuthworker/archive/pdonnell-2024-04-02_11:52:43-fs-wip-batrick-testing-20240402.004512-distro-default-smithi/7635633/remote/smithi177/crash 2024-04-02T14:54:51.137 DEBUG:teuthology.orchestra.run.smithi177:> sudo tar c -f - -C /var/lib/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/crash -- . 2024-04-02T14:54:51.171 INFO:teuthology.orchestra.run.smithi177.stderr:tar: /var/lib/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/crash: Cannot open: No such file or directory 2024-04-02T14:54:51.171 INFO:teuthology.orchestra.run.smithi177.stderr:tar: Error is not recoverable: exiting now 2024-04-02T14:54:51.172 INFO:tasks.cephadm:Checking cluster log for badness... 2024-04-02T14:54:51.172 DEBUG:teuthology.orchestra.run.smithi090:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | egrep -v 'slow metadata IO' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2024-04-02T14:54:51.278 INFO:tasks.cephadm:Compressing logs... 2024-04-02T14:54:51.278 DEBUG:teuthology.orchestra.run.smithi090:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-02T14:54:51.283 DEBUG:teuthology.orchestra.run.smithi092:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-02T14:54:51.285 DEBUG:teuthology.orchestra.run.smithi177:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-02T14:54:51.315 INFO:teuthology.orchestra.run.smithi177.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-02T14:54:51.315 INFO:teuthology.orchestra.run.smithi177.stderr:: No such file or directory 2024-04-02T14:54:51.315 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/ceph-mds.l.log 2024-04-02T14:54:51.315 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose/var/log/ceph/cephadm.log: -- /var/log/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/ceph.audit.log 2024-04-02T14:54:51.315 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/ceph-mds.l.log: gzip -5 --verbose -- /var/log/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/ceph-mds.i.log 2024-04-02T14:54:51.315 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/ceph-osd.11.log 2024-04-02T14:54:51.315 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/ceph-mgr.z.log 2024-04-02T14:54:51.316 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/ceph-mds.i.log: /var/log/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/ceph-osd.11.log: gzip -5 --verbose -- /var/log/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/ceph-osd.8.log 2024-04-02T14:54:51.316 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/ceph-mgr.z.log: gzip -5 --verbose 92.4% -- /var/log/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/ceph-osd.5.log 2024-04-02T14:54:51.316 INFO:teuthology.orchestra.run.smithi177.stderr: -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-02T14:54:51.316 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/ceph-osd.8.log: gzip -5 --verbose -- /var/log/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/ceph-mds.f.log 2024-04-02T14:54:51.316 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/ceph.cephadm.log 2024-04-02T14:54:51.320 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/ceph-mds.f.log: 90.4% -- replaced with /var/log/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/ceph.audit.log.gz 2024-04-02T14:54:51.320 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/ceph.log 2024-04-02T14:54:51.320 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/ceph.cephadm.log: 83.6% -- replaced with /var/log/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/ceph.cephadm.log.gz 2024-04-02T14:54:51.320 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/ceph-osd.2.log 2024-04-02T14:54:51.324 INFO:teuthology.orchestra.run.smithi092.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-02T14:54:51.325 INFO:teuthology.orchestra.run.smithi092.stderr:: No such file or directory 2024-04-02T14:54:51.325 INFO:teuthology.orchestra.run.smithi092.stderr:gzip -5 --verbose -- /var/log/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/ceph-osd.10.log 2024-04-02T14:54:51.325 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/ceph.audit.log 2024-04-02T14:54:51.325 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/ceph-osd.10.log: gzip -5 --verbose -- /var/log/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/ceph-mds.b.log 2024-04-02T14:54:51.325 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/ceph-osd.4.log 2024-04-02T14:54:51.325 INFO:teuthology.orchestra.run.smithi092.stderr: 92.1% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-02T14:54:51.325 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/ceph-mds.b.log: gzip -5 --verbose -- /var/log/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/ceph-osd.1.log 2024-04-02T14:54:51.325 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/ceph-mds.k.log 2024-04-02T14:54:51.325 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/ceph-mds.h.log 2024-04-02T14:54:51.325 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/ceph-mds.k.log: gzip -5 --verbose -- /var/log/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/ceph-osd.7.log 2024-04-02T14:54:51.325 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/ceph-mds.h.log: gzip -5 --verbose -- /var/log/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/ceph-mds.e.log 2024-04-02T14:54:51.325 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/ceph-osd.7.log: 90.4% -- replaced with /var/log/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/ceph.audit.log.gz 2024-04-02T14:54:51.325 INFO:teuthology.orchestra.run.smithi092.stderr:gzip -5 --verbose -- /var/log/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/ceph-mgr.y.log 2024-04-02T14:54:51.326 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/ceph.log: 86.9% -- replaced with /var/log/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/ceph.log.gz 2024-04-02T14:54:51.326 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/ceph-mds.c.log 2024-04-02T14:54:51.326 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/ceph-osd.2.log: 92.2% -- replaced with /var/log/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/ceph-mgr.z.log.gz 2024-04-02T14:54:51.326 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/ceph-volume.log 2024-04-02T14:54:51.332 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/ceph-mds.e.log: gzip -5 --verbose -- /var/log/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/ceph.cephadm.log 2024-04-02T14:54:51.338 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/ceph-mds.c.log: gzip -5 --verbose -- /var/log/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/ceph-mon.c.log 2024-04-02T14:54:51.343 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/ceph.log 2024-04-02T14:54:51.344 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/ceph.cephadm.log: 83.6% -- replaced with /var/log/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/ceph.cephadm.log.gz 2024-04-02T14:54:51.351 INFO:teuthology.orchestra.run.smithi090.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-04-02T14:54:51.351 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-02T14:54:51.352 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/ceph-osd.6.log 2024-04-02T14:54:51.353 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/ceph-mon.a.log 2024-04-02T14:54:51.354 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/ceph.audit.log 2024-04-02T14:54:51.356 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/ceph-mds.g.log 2024-04-02T14:54:51.356 INFO:teuthology.orchestra.run.smithi092.stderr:gzip -5 --verbose -- /var/log/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/ceph-volume.log 2024-04-02T14:54:51.357 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/ceph.log: 92.2% -- replaced with /var/log/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/ceph-mgr.y.log.gz 2024-04-02T14:54:51.357 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/ceph-mds.j.log 2024-04-02T14:54:51.359 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/ceph-mds.g.log: gzip -5 --verbose -- /var/log/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/ceph-osd.3.log 2024-04-02T14:54:51.360 INFO:teuthology.orchestra.run.smithi092.stderr: 86.9% -- replaced with /var/log/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/ceph.log.gz 2024-04-02T14:54:51.361 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/ceph-mds.j.log: gzip -5 --verbose -- /var/log/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/ceph-mgr.x.log 2024-04-02T14:54:51.361 INFO:teuthology.orchestra.run.smithi090.stderr: 93.2% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-02T14:54:51.362 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/ceph-osd.0.log 2024-04-02T14:54:51.362 INFO:teuthology.orchestra.run.smithi090.stderr: 90.2% -- replaced with /var/log/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/ceph.audit.log.gz 2024-04-02T14:54:51.363 INFO:teuthology.orchestra.run.smithi092.stderr:gzip -5 --verbose -- /var/log/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/ceph-mon.b.log 2024-04-02T14:54:51.363 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/ceph.cephadm.log 2024-04-02T14:54:51.365 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/ceph-volume.log: /var/log/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/ceph-mon.c.log: 90.9% -- replaced with /var/log/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/ceph-volume.log.gz 2024-04-02T14:54:51.366 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/ceph-osd.0.log: /var/log/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/ceph.log 2024-04-02T14:54:51.369 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/ceph.log: 84.1%gzip -5 --verbose -- /var/log/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/ceph-volume.log 2024-04-02T14:54:51.369 INFO:teuthology.orchestra.run.smithi090.stderr: -- replaced with /var/log/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/ceph.cephadm.log.gz 2024-04-02T14:54:51.372 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/ceph-osd.9.log 2024-04-02T14:54:51.379 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/ceph-volume.log: 86.9%/var/log/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/ceph-osd.9.log: gzip -5 --verbose -- /var/log/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/ceph-mds.d.log 2024-04-02T14:54:51.380 INFO:teuthology.orchestra.run.smithi090.stderr: -- replaced with /var/log/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/ceph.log.gz 2024-04-02T14:54:51.382 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/ceph-mds.a.log 2024-04-02T14:54:51.393 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/ceph-volume.log: /var/log/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/ceph-mon.b.log: 90.9% -- replaced with /var/log/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/ceph-volume.log.gz 2024-04-02T14:54:51.399 INFO:teuthology.orchestra.run.smithi177.stderr: 94.6% -- replaced with /var/log/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/ceph-mds.l.log.gz 2024-04-02T14:54:51.400 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/ceph-mds.d.log: /var/log/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/ceph-mds.a.log: 94.2% 94.3% -- replaced with /var/log/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/ceph-mds.d.log.gz 2024-04-02T14:54:51.403 INFO:teuthology.orchestra.run.smithi090.stderr: -- replaced with /var/log/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/ceph-mds.a.log.gz 2024-04-02T14:54:51.407 INFO:teuthology.orchestra.run.smithi090.stderr: 90.8% -- replaced with /var/log/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/ceph-volume.log.gz 2024-04-02T14:54:51.413 INFO:teuthology.orchestra.run.smithi092.stderr: 94.7% -- replaced with /var/log/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/ceph-mds.e.log.gz 2024-04-02T14:54:51.421 INFO:teuthology.orchestra.run.smithi092.stderr: 94.6% -- replaced with /var/log/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/ceph-mds.k.log.gz 2024-04-02T14:54:51.459 INFO:teuthology.orchestra.run.smithi090.stderr: 94.5% -- replaced with /var/log/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/ceph-mds.j.log.gz 2024-04-02T14:54:51.491 INFO:teuthology.orchestra.run.smithi092.stderr: 92.6% -- replaced with /var/log/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/ceph-mds.h.log.gz 2024-04-02T14:54:51.492 INFO:teuthology.orchestra.run.smithi177.stderr: 92.7% -- replaced with /var/log/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/ceph-mds.c.log.gz 2024-04-02T14:54:51.517 INFO:teuthology.orchestra.run.smithi177.stderr: 92.6% -- replaced with /var/log/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/ceph-mds.i.log.gz 2024-04-02T14:54:51.567 INFO:teuthology.orchestra.run.smithi090.stderr: 92.6% -- replaced with /var/log/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/ceph-mds.g.log.gz 2024-04-02T14:54:51.863 INFO:teuthology.orchestra.run.smithi090.stderr: 90.3% -- replaced with /var/log/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/ceph-mgr.x.log.gz 2024-04-02T14:54:51.894 INFO:teuthology.orchestra.run.smithi092.stderr: 91.8% -- replaced with /var/log/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/ceph-mon.b.log.gz 2024-04-02T14:54:52.062 INFO:teuthology.orchestra.run.smithi177.stderr: 91.8% -- replaced with /var/log/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/ceph-mon.c.log.gz 2024-04-02T14:54:52.111 INFO:teuthology.orchestra.run.smithi177.stderr: 94.0% -- replaced with /var/log/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/ceph-mds.f.log.gz 2024-04-02T14:54:53.003 INFO:teuthology.orchestra.run.smithi090.stderr: 90.3% -- replaced with /var/log/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/ceph-mon.a.log.gz 2024-04-02T14:55:02.818 INFO:teuthology.orchestra.run.smithi092.stderr: 94.0% -- replaced with /var/log/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/ceph-mds.b.log.gz 2024-04-02T14:55:03.175 INFO:teuthology.orchestra.run.smithi092.stderr: 94.0% -- replaced with /var/log/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/ceph-osd.4.log.gz 2024-04-02T14:55:04.450 INFO:teuthology.orchestra.run.smithi177.stderr: 94.2% -- replaced with /var/log/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/ceph-osd.8.log.gz 2024-04-02T14:55:05.113 INFO:teuthology.orchestra.run.smithi090.stderr: 93.9% -- replaced with /var/log/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/ceph-osd.9.log.gz 2024-04-02T14:55:06.909 INFO:teuthology.orchestra.run.smithi177.stderr: 94.1% -- replaced with /var/log/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/ceph-osd.5.log.gz 2024-04-02T14:55:09.214 INFO:teuthology.orchestra.run.smithi177.stderr: 93.8% -- replaced with /var/log/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/ceph-osd.2.log.gz 2024-04-02T14:55:11.058 INFO:teuthology.orchestra.run.smithi090.stderr: 93.9% -- replaced with /var/log/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/ceph-osd.6.log.gz 2024-04-02T14:55:11.583 INFO:teuthology.orchestra.run.smithi177.stderr: 94.3% -- replaced with /var/log/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/ceph-osd.11.log.gz 2024-04-02T14:55:11.585 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-04-02T14:55:11.585 INFO:teuthology.orchestra.run.smithi177.stderr:real 0m20.286s 2024-04-02T14:55:11.585 INFO:teuthology.orchestra.run.smithi177.stderr:user 1m6.090s 2024-04-02T14:55:11.585 INFO:teuthology.orchestra.run.smithi177.stderr:sys 0m2.569s 2024-04-02T14:55:12.585 INFO:teuthology.orchestra.run.smithi092.stderr: 93.8% -- replaced with /var/log/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/ceph-osd.10.log.gz 2024-04-02T14:55:13.019 INFO:teuthology.orchestra.run.smithi092.stderr: 94.0% -- replaced with /var/log/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/ceph-osd.1.log.gz 2024-04-02T14:55:15.815 INFO:teuthology.orchestra.run.smithi090.stderr: 93.9% -- replaced with /var/log/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/ceph-osd.0.log.gz 2024-04-02T14:55:15.947 INFO:teuthology.orchestra.run.smithi090.stderr: 93.9% -- replaced with /var/log/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/ceph-osd.3.log.gz 2024-04-02T14:55:15.953 INFO:teuthology.orchestra.run.smithi090.stderr: 2024-04-02T14:55:15.953 INFO:teuthology.orchestra.run.smithi090.stderr:real 0m24.637s 2024-04-02T14:55:15.953 INFO:teuthology.orchestra.run.smithi090.stderr:user 1m18.214s 2024-04-02T14:55:15.953 INFO:teuthology.orchestra.run.smithi090.stderr:sys 0m6.817s 2024-04-02T14:55:20.752 INFO:teuthology.orchestra.run.smithi092.stderr: 94.0% -- replaced with /var/log/ceph/6449448a-f0fc-11ee-b647-cb9ed24678a4/ceph-osd.7.log.gz 2024-04-02T14:55:20.754 INFO:teuthology.orchestra.run.smithi092.stderr: 2024-04-02T14:55:20.754 INFO:teuthology.orchestra.run.smithi092.stderr:real 0m29.455s 2024-04-02T14:55:20.754 INFO:teuthology.orchestra.run.smithi092.stderr:user 1m33.135s 2024-04-02T14:55:20.754 INFO:teuthology.orchestra.run.smithi092.stderr:sys 0m3.329s 2024-04-02T14:55:20.755 INFO:tasks.cephadm:Archiving logs... 2024-04-02T14:55:20.755 DEBUG:teuthology.misc:Transferring archived files from smithi090:/var/log/ceph to /home/teuthworker/archive/pdonnell-2024-04-02_11:52:43-fs-wip-batrick-testing-20240402.004512-distro-default-smithi/7635633/remote/smithi090/log 2024-04-02T14:55:20.756 DEBUG:teuthology.orchestra.run.smithi090:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-02T14:55:32.375 DEBUG:teuthology.misc:Transferring archived files from smithi092:/var/log/ceph to /home/teuthworker/archive/pdonnell-2024-04-02_11:52:43-fs-wip-batrick-testing-20240402.004512-distro-default-smithi/7635633/remote/smithi092/log 2024-04-02T14:55:32.376 DEBUG:teuthology.orchestra.run.smithi092:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-02T14:55:45.098 DEBUG:teuthology.misc:Transferring archived files from smithi177:/var/log/ceph to /home/teuthworker/archive/pdonnell-2024-04-02_11:52:43-fs-wip-batrick-testing-20240402.004512-distro-default-smithi/7635633/remote/smithi177/log 2024-04-02T14:55:45.100 DEBUG:teuthology.orchestra.run.smithi177:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-02T14:55:54.496 INFO:tasks.cephadm:Removing cluster... 2024-04-02T14:55:54.497 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 6449448a-f0fc-11ee-b647-cb9ed24678a4 --force 2024-04-02T14:55:54.821 INFO:teuthology.orchestra.run.smithi090.stdout:Deleting cluster with fsid: 6449448a-f0fc-11ee-b647-cb9ed24678a4 2024-04-02T14:55:55.560 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 6449448a-f0fc-11ee-b647-cb9ed24678a4 --force 2024-04-02T14:55:55.759 INFO:teuthology.orchestra.run.smithi092.stdout:Deleting cluster with fsid: 6449448a-f0fc-11ee-b647-cb9ed24678a4 2024-04-02T14:55:56.131 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 6449448a-f0fc-11ee-b647-cb9ed24678a4 --force 2024-04-02T14:55:56.323 INFO:teuthology.orchestra.run.smithi177.stdout:Deleting cluster with fsid: 6449448a-f0fc-11ee-b647-cb9ed24678a4 2024-04-02T14:55:56.667 INFO:tasks.cephadm:Removing cephadm ... 2024-04-02T14:55:56.667 DEBUG:teuthology.orchestra.run.smithi090:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-02T14:55:56.725 DEBUG:teuthology.orchestra.run.smithi092:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-02T14:55:56.741 DEBUG:teuthology.orchestra.run.smithi177:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-02T14:55:56.756 INFO:tasks.cephadm:Teardown complete 2024-04-02T14:55:56.757 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-04-02T14:55:56.769 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-04-02T14:55:56.769 DEBUG:teuthology.orchestra.run.smithi090:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-02T14:55:56.777 DEBUG:teuthology.orchestra.run.smithi092:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-02T14:55:56.784 DEBUG:teuthology.orchestra.run.smithi177:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-02T14:55:56.940 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-02T14:55:56.952 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-02T14:55:56.952 DEBUG:teuthology.orchestra.run.smithi090:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-02T14:55:56.960 DEBUG:teuthology.orchestra.run.smithi092:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-02T14:55:56.962 DEBUG:teuthology.orchestra.run.smithi177:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-02T14:55:56.974 INFO:teuthology.orchestra.run.smithi092.stderr:bash: line 1: ntpq: command not found 2024-04-02T14:55:56.976 INFO:teuthology.orchestra.run.smithi177.stderr:bash: line 1: ntpq: command not found 2024-04-02T14:55:56.979 INFO:teuthology.orchestra.run.smithi092.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-02T14:55:56.979 INFO:teuthology.orchestra.run.smithi092.stdout:=============================================================================== 2024-04-02T14:55:56.979 INFO:teuthology.orchestra.run.smithi092.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 18 -636us[ -630us] +/- 91ms 2024-04-02T14:55:56.979 INFO:teuthology.orchestra.run.smithi092.stdout:^* hv02.front.sepia.ceph.com 3 6 377 18 -1306us[-1301us] +/- 79ms 2024-04-02T14:55:56.979 INFO:teuthology.orchestra.run.smithi092.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 19 +1635us[+1640us] +/- 69ms 2024-04-02T14:55:56.980 INFO:teuthology.orchestra.run.smithi092.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-04-02T14:55:56.981 INFO:teuthology.orchestra.run.smithi177.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-02T14:55:56.981 INFO:teuthology.orchestra.run.smithi177.stdout:=============================================================================== 2024-04-02T14:55:56.981 INFO:teuthology.orchestra.run.smithi177.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 15 -591us[ -591us] +/- 91ms 2024-04-02T14:55:56.981 INFO:teuthology.orchestra.run.smithi177.stdout:^* hv02.front.sepia.ceph.com 3 6 377 16 -1265us[-1227us] +/- 79ms 2024-04-02T14:55:56.981 INFO:teuthology.orchestra.run.smithi177.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 16 +1658us[+1658us] +/- 69ms 2024-04-02T14:55:56.981 INFO:teuthology.orchestra.run.smithi177.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-04-02T14:55:57.004 INFO:teuthology.orchestra.run.smithi090.stderr:bash: line 1: ntpq: command not found 2024-04-02T14:55:57.014 INFO:teuthology.orchestra.run.smithi090.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-02T14:55:57.014 INFO:teuthology.orchestra.run.smithi090.stdout:=============================================================================== 2024-04-02T14:55:57.014 INFO:teuthology.orchestra.run.smithi090.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 17 -555us[ -530us] +/- 91ms 2024-04-02T14:55:57.015 INFO:teuthology.orchestra.run.smithi090.stdout:^* hv02.front.sepia.ceph.com 3 6 377 17 -1223us[-1197us] +/- 79ms 2024-04-02T14:55:57.015 INFO:teuthology.orchestra.run.smithi090.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 19 +1685us[+1710us] +/- 69ms 2024-04-02T14:55:57.015 INFO:teuthology.orchestra.run.smithi090.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-04-02T14:55:57.016 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-02T14:55:57.035 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-02T14:55:57.036 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-02T14:55:57.134 DEBUG:teuthology.orchestra.run.smithi090:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-02T14:55:57.374 DEBUG:teuthology.orchestra.run.smithi092:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-02T14:55:57.450 DEBUG:teuthology.orchestra.run.smithi177:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-02T14:55:57.524 DEBUG:teuthology.orchestra.run.smithi090:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-02T14:55:57.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T14:55:57.627 DEBUG:teuthology.orchestra.run.smithi092:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-02T14:55:57.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T14:55:57.658 DEBUG:teuthology.orchestra.run.smithi177:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-02T14:55:57.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T14:55:57.689 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-02T14:55:57.700 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-02T14:55:57.809 INFO:teuthology.task.internal:Duration was 2520.543982 seconds 2024-04-02T14:55:57.810 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-02T14:55:57.961 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-02T14:55:57.962 DEBUG:teuthology.orchestra.run.smithi090:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-02T14:55:57.970 DEBUG:teuthology.orchestra.run.smithi092:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-02T14:55:57.972 DEBUG:teuthology.orchestra.run.smithi177:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-02T14:55:58.015 INFO:teuthology.orchestra.run.smithi092.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-02T14:55:58.016 INFO:teuthology.orchestra.run.smithi177.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-02T14:55:58.132 INFO:teuthology.orchestra.run.smithi090.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-02T14:55:58.530 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-02T14:55:58.531 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi090.front.sepia.ceph.com 2024-04-02T14:55:58.531 DEBUG:teuthology.orchestra.run.smithi090:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-02T14:55:58.591 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi092.front.sepia.ceph.com 2024-04-02T14:55:58.591 DEBUG:teuthology.orchestra.run.smithi092:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-02T14:55:58.617 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi177.front.sepia.ceph.com 2024-04-02T14:55:58.617 DEBUG:teuthology.orchestra.run.smithi177:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-02T14:55:58.644 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-02T14:55:58.644 DEBUG:teuthology.orchestra.run.smithi090:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-02T14:55:58.648 DEBUG:teuthology.orchestra.run.smithi092:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-02T14:55:58.660 DEBUG:teuthology.orchestra.run.smithi177:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-02T14:55:58.725 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-02T14:55:58.725 DEBUG:teuthology.orchestra.run.smithi090:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-02T14:55:58.730 DEBUG:teuthology.orchestra.run.smithi092:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-02T14:55:58.732 DEBUG:teuthology.orchestra.run.smithi177:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-02T14:55:59.225 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-02T14:55:59.237 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-02T14:55:59.237 DEBUG:teuthology.orchestra.run.smithi090:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-02T14:55:59.317 DEBUG:teuthology.orchestra.run.smithi092:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-02T14:55:59.344 DEBUG:teuthology.orchestra.run.smithi177:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-02T14:55:59.372 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-02T14:55:59.384 DEBUG:teuthology.orchestra.run.smithi090:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-02T14:55:59.392 DEBUG:teuthology.orchestra.run.smithi092:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-02T14:55:59.394 DEBUG:teuthology.orchestra.run.smithi177:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-02T14:55:59.418 INFO:teuthology.orchestra.run.smithi092.stdout:kernel.core_pattern = core 2024-04-02T14:55:59.438 INFO:teuthology.orchestra.run.smithi177.stdout:kernel.core_pattern = core 2024-04-02T14:55:59.484 INFO:teuthology.orchestra.run.smithi090.stdout:kernel.core_pattern = core 2024-04-02T14:55:59.537 DEBUG:teuthology.orchestra.run.smithi090:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-02T14:55:59.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T14:55:59.582 DEBUG:teuthology.orchestra.run.smithi092:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-02T14:55:59.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T14:55:59.596 DEBUG:teuthology.orchestra.run.smithi177:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-02T14:55:59.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T14:55:59.610 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-02T14:55:59.622 INFO:teuthology.task.internal:Transferring archived files... 2024-04-02T14:55:59.624 DEBUG:teuthology.misc:Transferring archived files from smithi090:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2024-04-02_11:52:43-fs-wip-batrick-testing-20240402.004512-distro-default-smithi/7635633/remote/smithi090 2024-04-02T14:55:59.624 DEBUG:teuthology.orchestra.run.smithi090:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-02T14:55:59.740 DEBUG:teuthology.misc:Transferring archived files from smithi092:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2024-04-02_11:52:43-fs-wip-batrick-testing-20240402.004512-distro-default-smithi/7635633/remote/smithi092 2024-04-02T14:55:59.740 DEBUG:teuthology.orchestra.run.smithi092:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-02T14:55:59.778 DEBUG:teuthology.misc:Transferring archived files from smithi177:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2024-04-02_11:52:43-fs-wip-batrick-testing-20240402.004512-distro-default-smithi/7635633/remote/smithi177 2024-04-02T14:55:59.778 DEBUG:teuthology.orchestra.run.smithi177:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-02T14:55:59.817 INFO:teuthology.task.internal:Removing archive directory... 2024-04-02T14:55:59.817 DEBUG:teuthology.orchestra.run.smithi090:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-02T14:55:59.825 DEBUG:teuthology.orchestra.run.smithi092:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-02T14:55:59.827 DEBUG:teuthology.orchestra.run.smithi177:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-02T14:55:59.876 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-02T14:55:59.890 INFO:teuthology.task.internal:Not uploading archives. 2024-04-02T14:55:59.891 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-02T14:55:59.976 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-02T14:55:59.976 DEBUG:teuthology.orchestra.run.smithi090:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-02T14:55:59.983 DEBUG:teuthology.orchestra.run.smithi092:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-02T14:55:59.986 DEBUG:teuthology.orchestra.run.smithi177:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-02T14:55:59.998 INFO:teuthology.orchestra.run.smithi092.stdout: 262162 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 2 14:55 /home/ubuntu/cephtest 2024-04-02T14:56:00.000 INFO:teuthology.orchestra.run.smithi177.stdout: 262162 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 2 14:55 /home/ubuntu/cephtest 2024-04-02T14:56:00.033 INFO:teuthology.orchestra.run.smithi090.stdout: 393229 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 2 14:55 /home/ubuntu/cephtest 2024-04-02T14:56:00.036 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-02T14:56:00.069 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-02T14:56:00.319 INFO:teuthology.nuke:Checking targets against current locks 2024-04-02T14:56:00.344 DEBUG:teuthology.nuke:shortname: smithi090 2024-04-02T14:56:00.344 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-02T14:56:00.367 DEBUG:teuthology.nuke:shortname: smithi092 2024-04-02T14:56:00.368 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-02T14:56:00.372 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi090.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-04-02_11:52:43-fs-wip-batrick-testing-20240402.004512-distro-default-smithi/7635633', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-02 14:02:37.469537', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJS93EoA4MChx/q67iYrMT1Obfm2I07Gk+q4/6Jmlvf6klUQ3J3AS7yZ+1/ts6xGTOkpkbdUpkV0fP816IS0OMk='} 2024-04-02T14:56:00.392 DEBUG:teuthology.nuke:shortname: smithi177 2024-04-02T14:56:00.392 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-02T14:56:00.397 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi092.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-04-02_11:52:43-fs-wip-batrick-testing-20240402.004512-distro-default-smithi/7635633', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-02 14:02:37.468340', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJuCtV9TCEQvasOugmWQqmXiTqfkE60iZ0BdPbwHiSHnR8IrHgqtyalK87AAGA5CgQnquCASauo24il75OhQFCE='} 2024-04-02T14:56:00.424 INFO:teuthology.orchestra.console.smithi090:Power off 2024-04-02T14:56:00.424 DEBUG:teuthology.orchestra.console.smithi090:pexpect command: ipmitool -H smithi090.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-02T14:56:00.437 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi177.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-04-02_11:52:43-fs-wip-batrick-testing-20240402.004512-distro-default-smithi/7635633', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-02 14:02:37.466927', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOhQVdK2fqKEdXq4G1AcWvfCJbdxAhzafRar5kO/1BGBcupxYJjyADu9+jEC99Q1L3EYwhf8AN8DNkr+Ql/uR6o='} 2024-04-02T14:56:00.447 DEBUG:teuthology.orchestra.console.smithi090:power off output: Chassis Power Control: Down/Off 2024-04-02T14:56:00.447 DEBUG:teuthology.orchestra.console.smithi090:pexpect command: ipmitool -H smithi090.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-02T14:56:00.469 INFO:teuthology.orchestra.console.smithi092:Power off 2024-04-02T14:56:00.469 DEBUG:teuthology.orchestra.console.smithi092:pexpect command: ipmitool -H smithi092.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-02T14:56:00.484 DEBUG:teuthology.orchestra.console.smithi090:check power output: Chassis Power is on 2024-04-02T14:56:00.492 DEBUG:teuthology.orchestra.console.smithi092:power off output: Chassis Power Control: Down/Off 2024-04-02T14:56:00.492 DEBUG:teuthology.orchestra.console.smithi092:pexpect command: ipmitool -H smithi092.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-02T14:56:00.509 INFO:teuthology.orchestra.console.smithi177:Power off 2024-04-02T14:56:00.510 DEBUG:teuthology.orchestra.console.smithi177:pexpect command: ipmitool -H smithi177.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-02T14:56:00.521 DEBUG:teuthology.orchestra.console.smithi092:check power output: Chassis Power is on 2024-04-02T14:56:00.535 DEBUG:teuthology.orchestra.console.smithi177:power off output: Chassis Power Control: Down/Off 2024-04-02T14:56:00.536 DEBUG:teuthology.orchestra.console.smithi177:pexpect command: ipmitool -H smithi177.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-02T14:56:00.560 DEBUG:teuthology.orchestra.console.smithi177:check power output: Chassis Power is on 2024-04-02T14:56:04.485 DEBUG:teuthology.orchestra.console.smithi090:pexpect command: ipmitool -H smithi090.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-02T14:56:04.523 DEBUG:teuthology.orchestra.console.smithi092:pexpect command: ipmitool -H smithi092.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-02T14:56:04.561 DEBUG:teuthology.orchestra.console.smithi177:pexpect command: ipmitool -H smithi177.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-02T14:56:04.595 DEBUG:teuthology.orchestra.console.smithi090:check power output: Chassis Power is on 2024-04-02T14:56:04.633 DEBUG:teuthology.orchestra.console.smithi092:check power output: Chassis Power is on 2024-04-02T14:56:04.671 DEBUG:teuthology.orchestra.console.smithi177:check power output: Chassis Power is on 2024-04-02T14:56:08.598 DEBUG:teuthology.orchestra.console.smithi090:pexpect command: ipmitool -H smithi090.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-02T14:56:08.634 DEBUG:teuthology.orchestra.console.smithi092:pexpect command: ipmitool -H smithi092.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-02T14:56:08.672 DEBUG:teuthology.orchestra.console.smithi177:pexpect command: ipmitool -H smithi177.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-02T14:56:08.708 DEBUG:teuthology.orchestra.console.smithi090:check power output: Chassis Power is off 2024-04-02T14:56:08.743 DEBUG:teuthology.orchestra.console.smithi092:check power output: Chassis Power is on 2024-04-02T14:56:08.781 DEBUG:teuthology.orchestra.console.smithi177:check power output: Chassis Power is off 2024-04-02T14:56:08.810 INFO:teuthology.orchestra.console.smithi090:Power off completed 2024-04-02T14:56:08.882 INFO:teuthology.orchestra.console.smithi177:Power off completed 2024-04-02T14:56:12.746 DEBUG:teuthology.orchestra.console.smithi092:pexpect command: ipmitool -H smithi092.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-02T14:56:12.859 DEBUG:teuthology.orchestra.console.smithi092:check power output: Chassis Power is off 2024-04-02T14:56:12.960 INFO:teuthology.orchestra.console.smithi092:Power off completed 2024-04-02T14:56:13.065 INFO:teuthology.run:Summary data: client.0-kernel-sha1: 1df0c7e8a2354c42d738e5454ba83ea7332deac2 description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/postgres}} duration: 2520.543982028961 failure_reason: 'The following counters failed to be set on mds daemons: {''mds.exported'', ''mds.imported''}' flavor: default owner: scheduled_pdonnell@teuthology status: fail success: false 2024-04-02T14:56:13.066 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-02T14:56:13.145 INFO:teuthology.run:FAIL